- yaml-import is now logged via schema.Logger
- iremoved relative imports - not used in other mailu modules
- removed develepment comments
- added Mailconfig.check method to check for duplicate domain names
- converted .format() to .format_map() where possible
- switched to yaml multiline dump for dkim_key
- converted dkim_key import from regex to string functions
- automatically unhide/unexclude explicitly specified attributes on dump
- use field order when loading to stabilize import
- fail when using 'hash_password' without 'password'
- fixed logging of dkim_key
- fixed pruning and deleting of lists
- modified error messages
- added debug flag and two verbosity levels
- idna.encode does not encode upper-case letters,
so .lower() has to be called on value not on result
- split email-address on '@' only once
- converted '*'.format(*) to f-strings
- added docstrings
- removed from_dict method
- code cleanup/style (list concat, exceptions, return&else, line-length)
- added TODO comments on possible future changes
- revived original config-update function for backwards compability
- renamed config-dump to config-export to be in line with config-import
- converted '*'.format(*) to f-strings
- converted string-concatenation to f-strings
renamed single letter variables (m => match)
renamed classmethod arguments to cls (model)
removed shadowing of variables (hash, context)
shortened unneeded lambda functions (id)
converted type ... is to isinstance(...)
removed unneded imports (flask)
Updated ConfigManager to only modify app.config and not replace it.
Swagger does not play well, when app.config is not a real dict and
it is not necessary to keep ConfigManager around after init.
Also added "API" flag to config (default: disabled).
this enables usage of e.g. docker swarm secrets instead of exposing the
passwords directly via environment variables
just use DB_PW_FILE and SECRET_KEY_FILE instead of DB_PW and SECRET_KEY
- stabilized CommaSeparatedList by sorting values
- CommaSeparatedList can now handle list and set input
- from_dict now handles mapped keys
- from_dict now handles null values
- class Domain: handle dkim-key None correctly
- class User: delete obsolete keys after converting
- class Alias: now uses Email._dict_input
1399: Remove SPF type SPF record #1394 r=mergify[bot] a=bladeswords
As mentioned in #1394 - In accordance with RFC 7208, offer only TXT RRs for SPF.
Agree with @Nebukadneza - but not sure how to go about telling people to remove the old record...
## What type of PR?
Documentation
## What does this PR do?
Removes the recommendation to add a SPF RR for SPF records, as this is no longer RFC complaint and often causes issues to maintain two records.
### Related issue(s)
- closes#1394
## Prerequistes
None
1417: docker-compose exec needs a -T flag if no TTY is allocated r=mergify[bot] a=ofthesun9
This flag is missing in 00_create_users.sh and is failing the tests on travis arm architecture
## What type of PR?
This PR is an enhancement/bugfix needed to allow usage of travis to test and deploy on arm platform
Before the PR, tests are failing with the msg: "the input device is not a TTY"
## What does this PR do?
This PR add -T flag for the docker-compose exec occurences found in 00_create_users.sh
Co-authored-by: bladeswords <bladeswords@users.noreply.github.com>
Co-authored-by: Dario Ernst <dario@kanojo.de>
Co-authored-by: ofthesun9 <olivier@ofthesun.net>
1389: Prefer specific alias over wildcard, regardless of case r=mergify[bot] a=Nebukadneza
## What type of PR?
bug-fix
## What does this PR do?
Since direct addresses (not aliases) are case-insensitive since a while,
it makes sense for aliases to behave the same. Up until now, a wildcard
alias could trump a alias not-matching-the-case of the incoming address.
This clarifies this behavior.
## Notes
I realize that the if-hell down there isn’t nice. What it is, however, is quite clear and easy to read. I’m hoping that if anyone ever gets confused in the future, this will make the current behavior transparent. For me, that was more important than a minimal amount of statements/branches …
### Related issue(s)
closes#1387
## Prerequistes
- [x] Unless it's docs or a minor change: add [changelog](https://mailu.io/master/contributors/guide.html#changelog) entry file.
Co-authored-by: Dario Ernst <github@kanojo.de>
1296: fetchmail: print unhandled exceptions, but don't crash r=Nebukadneza a=Al2Klimov
fixes#1295
1322: Bump validators from 0.12.5 to 0.12.6 in /core/admin r=Nebukadneza a=dependabot[bot]
Bumps [validators](https://github.com/kvesteri/validators) from 0.12.5 to 0.12.6.
<details>
<summary>Changelog</summary>
*Sourced from [validators's changelog](https://github.com/kvesteri/validators/blob/master/CHANGES.rst).*
> 0.12.6 (2019-05-08)
> ^^^^^^^^^^^^^^^^^^^
>
> - Fixed domain validator for single character domains ([#118](https://github-redirect.dependabot.com/kvesteri/validators/issues/118), pull request courtesy kingbuzzman)
</details>
<details>
<summary>Commits</summary>
- See full diff in [compare view](https://github.com/kvesteri/validators/commits)
</details>
<br />
[](https://help.github.com/articles/configuring-automated-security-fixes)
Dependabot will resolve any conflicts with this PR as long as you don't alter it yourself. You can also trigger a rebase manually by commenting `@dependabot rebase`.
[//]: # (dependabot-automerge-start)
[//]: # (dependabot-automerge-end)
---
<details>
<summary>Dependabot commands and options</summary>
<br />
You can trigger Dependabot actions by commenting on this PR:
- `@dependabot rebase` will rebase this PR
- `@dependabot recreate` will recreate this PR, overwriting any edits that have been made to it
- `@dependabot merge` will merge this PR after your CI passes on it
- `@dependabot squash and merge` will squash and merge this PR after your CI passes on it
- `@dependabot cancel merge` will cancel a previously requested merge and block automerging
- `@dependabot reopen` will reopen this PR if it is closed
- `@dependabot ignore this [patch|minor|major] version` will close this PR and stop Dependabot creating any more for this minor/major version (unless you reopen the PR or upgrade to it yourself)
- `@dependabot ignore this dependency` will close this PR and stop Dependabot creating any more for this dependency (unless you reopen the PR or upgrade to it yourself)
- `@dependabot use these labels` will set the current labels as the default for future PRs for this repo and language
- `@dependabot use these reviewers` will set the current reviewers as the default for future PRs for this repo and language
- `@dependabot use these assignees` will set the current assignees as the default for future PRs for this repo and language
- `@dependabot use this milestone` will set the current milestone as the default for future PRs for this repo and language
You can disable automated security fix PRs for this repo from the [Security Alerts page](https://github.com/Mailu/Mailu/network/alerts).
</details>
1337: Add IPv6 to allow_nets r=Nebukadneza a=PhilRW
Roundcube was not connecting to sieve with IPv6 enabled.
Fixes#1336
1358: Add port to relay if it contains a colon r=Nebukadneza a=PhilRW
## What type of PR?
enhancement
## What does this PR do?
Allows relaying domains to non-standard SMTP ports by appending `:port` to the destination host/IP. E.g., `mx1.internal:2525`
### Related issue(s)
Closes#1357
## Prerequistes
Before we can consider review and merge, please make sure the following list is done and checked.
If an entry in not applicable, you can check it or remove it from the list.
- [x] In case of feature or enhancement: documentation updated accordingly
- [x] Unless it's docs or a minor change: add [changelog](https://mailu.io/master/contributors/guide.html#changelog) entry file.
Co-authored-by: Alexander A. Klimov <grandmaster@al2klimov.de>
Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com>
Co-authored-by: Philip Rosenberg-Watt <p.rosenberg-watt@cablelabs.com>
As mentioned in #1394 - In accordance with RFC 7208, offer only TXT RRs for SPF.
Agree with @Nebukadneza - but not sure how to go about telling people to remove the old record...
1359: Refactor the rate limiting code r=mergify[bot] a=kaiyou
## What type of PR?
Enhancement
## What does this PR do?
Rate limiting was already redesigned to use Python limits. This
introduced some unexpected behavior, including the fact that only
one criteria is supported per limiter. Docs and setup utility are
updated with this in mind.
Also, the code was made more generic, so limiters can be delivered
for something else than authentication. Authentication-specific
code was moved directly to the authentication routine.
### Related issue(s)
No specific issue.
## Prerequistes
Before we can consider review and merge, please make sure the following list is done and checked.
If an entry in not applicable, you can check it or remove it from the list.
- [x] In case of feature or enhancement: documentation updated accordingly
- [x] Unless it's docs or a minor change: add [changelog](https://mailu.io/master/contributors/guide.html#changelog) entry file.
Co-authored-by: kaiyou <pierre@jaury.eu>
1349: Add support for SRS, related to #328 r=mergify[bot] a=kaiyou
## What type of PR?
Feature
## What does this PR do?
It implements SRS using a Python SRS library.
### Related issue(s)
- closes#328
## Prerequistes
Before we can consider review and merge, please make sure the following list is done and checked.
If an entry in not applicable, you can check it or remove it from the list.
- [x] In case of feature or enhancement: documentation updated accordingly
- [x] Unless it's docs or a minor change: add [changelog](https://mailu.io/master/contributors/guide.html#changelog) entry file.
Co-authored-by: kaiyou <pierre@jaury.eu>
Since direct addresses (not aliases) are case-insensitive since a while,
it makes sense for aliases to behave the same. Up until now, a wildcard
alias could trump a alias not-matching-the-case of the incoming address.
This clarifies this behavior.
closes#1387
Rate limiting was already redesigned to use Python limits. This
introduced some unexpected behavior, including the fact that only
one criteria is supported per limiter. Docs and setup utility are
updated with this in mind.
Also, the code was made more generic, so limiters can be delivered
for something else than authentication. Authentication-specific
code was moved directly to the authentication routine.
1278: Limiter implementation r=kaiyou a=micw
## What type of PR?
(Feature, enhancement, bug-fix, documentation)
## What does this PR do?
Adds a custom limter based on the "limits" lirary that counts up on failed auths only
### Related issue(s)
- closes#1195
- closes#634
## Prerequistes
- [X] Unless it's docs or a minor change: add [changelog](https://mailu.io/master/contributors/guide.html#changelog) entry file.
Co-authored-by: Michael Wyraz <michael@wyraz.de>
Co-authored-by: micw <michael@wyraz.de>
1316: Fix the encoding of incoming user email and password r=mergify[bot] a=kaiyou
## What type of PR?
Bug fix
## What does this PR do?
As described in the changes, RFC2616 states that header should be considered ISO8859-1 in HTTP, which obviously nginx does not really care about when forwarding the password from SMTP authentication to the backend. Hence, we need to encode-then-decode passwords to get the proper value in case a special char is in there.
### Related issue(s)
- This fixes#1139
- This is also related to #1281
- This is also related to #1139
## Prerequistes
Before we can consider review and merge, please make sure the following list is done and checked.
If an entry in not applicable, you can check it or remove it from the list.
- [x] In case of feature or enhancement: documentation updated accordingly
- [x] Unless it's docs or a minor change: add [changelog](https://mailu.io/master/contributors/guide.html#changelog) entry file.
Co-authored-by: kaiyou <pierre@jaury.eu>
1299: Don't remove the address extension in postfix r=mergify[bot] a=RobertMe
## What type of PR?
Bugfix
## What does this PR do?
Currently when the mail address is looked up by Postfix (using the admin
part) the address extension is removed. This is due to the address
extension being removed to look up the user, and afterwards returning
the users mail address. But by not returning the mail address including
the address extension it also isn't part anymore in the LMTP
communication to Dovecot. So Dovecot doesn't know about the extension,
and in turn the address extension can't be used in Sieve mail filtering.
This change fixes that by returning the original address by just
concatinating the "localpart" and domain again when the user is found.
### Related issue(s)
Fixes#982
## Prerequistes
Before we can consider review and merge, please make sure the following list is done and checked.
If an entry in not applicable, you can check it or remove it from the list.
- [ ] In case of feature or enhancement: documentation updated accordingly
- [ ] Unless it's docs or a minor change: add [changelog](https://mailu.io/master/contributors/guide.html#changelog) entry file.
Co-authored-by: Robert Meijers <robert.meijers@gmail.com>
Currently when the mail address is looked up by Postfix (using the admin
part) the address extension is removed. This is due to the address
extension being removed to look up the user, and afterwards returning
the users mail address. But by not returning the mail address including
the address extension it also isn't part anymore in the LMTP
communication to Dovecot. So Dovecot doesn't know about the extension,
and in turn the address extension can't be used in Sieve mail filtering.
This change fixes that by returning the original address by just
concatinating the "localpart" and domain again when the user is found.
Fixes#982
985: Permit raspberry pi (and other architectures) builds r=mergify[bot] a=abondis
## What type of PR?
Enhancement
## What does this PR do?
Add an option to select base images and permit building for different CPU architectures.
### Related issue(s)
N/A
## Prerequistes
- [X] documentation updated accordingly
- [x] Unless it's docs or a minor change: add [changelog](https://mailu.io/master/contributors/guide.html#changelog) entry file.
Co-authored-by: Aurélien Bondis <aurelien.bondis@gmail.com>
Co-authored-by: Aurelien <aurelien.bondis@gmail.com>
1213: Bump werkzeug from 0.15.2 to 0.15.3 in /core/admin r=mergify[bot] a=dependabot[bot]
Bumps [werkzeug](https://github.com/pallets/werkzeug) from 0.15.2 to 0.15.3.
<details>
<summary>Release notes</summary>
*Sourced from [werkzeug's releases](https://github.com/pallets/werkzeug/releases).*
> ## 0.15.3
> * Blog: https://palletsprojects.com/blog/werkzeug-0-15-3-released/
> * Changes: https://werkzeug.palletsprojects.com/en/0.15.x/changes/#version-0-15-3
>
</details>
<details>
<summary>Changelog</summary>
*Sourced from [werkzeug's changelog](https://github.com/pallets/werkzeug/blob/master/CHANGES.rst).*
> Version 0.15.3
> --------------
>
> Released 2019-05-14
>
> - Properly handle multi-line header folding in development server in
> Python 2.7. (:issue:`1080`)
> - Restore the ``response`` argument to :exc:`~exceptions.Unauthorized`.
> (:pr:`1527`)
> - :exc:`~exceptions.Unauthorized` doesn't add the ``WWW-Authenticate``
> header if ``www_authenticate`` is not given. (:issue:`1516`)
> - The default URL converter correctly encodes bytes to string rather
> than representing them with ``b''``. (:issue:`1502`)
> - Fix the filename format string in
> :class:`~middleware.profiler.ProfilerMiddleware` to correctly handle
> float values. (:issue:`1511`)
> - Update :class:`~middleware.lint.LintMiddleware` to work on Python 3.
> (:issue:`1510`)
> - The debugger detects cycles in chained exceptions and does not time
> out in that case. (:issue:`1536`)
> - When running the development server in Docker, the debugger security
> pin is now unique per container.
</details>
<details>
<summary>Commits</summary>
- [`9b1123a`](9b1123a779) release version 0.15.3
- [`00bc43b`](00bc43b167) unique debugger pin in Docker containers
- [`2cbdf2b`](2cbdf2b022) Merge pull request [#1542](https://github-redirect.dependabot.com/pallets/werkzeug/issues/1542) from asottile/exceptions_arent_always_hashable
- [`0e669f6`](0e669f6be5) Fix unhashable exception types
- [`bdc17e4`](bdc17e4cd1) Merge pull request [#1540](https://github-redirect.dependabot.com/pallets/werkzeug/issues/1540) from pallets/break-tb-cycle
- [`44e38c2`](44e38c2985) break cycle in chained exceptions
- [`777500b`](777500b646) Merge pull request [#1518](https://github-redirect.dependabot.com/pallets/werkzeug/issues/1518) from NiklasMM/fix/1510_lint-middleware-python3-compa...
- [`e00c7c2`](e00c7c2ced) Make LintMiddleware Python 3 compatible and add tests
- [`d590cc7`](d590cc7cf2) Merge pull request [#1539](https://github-redirect.dependabot.com/pallets/werkzeug/issues/1539) from pallets/profiler-format
- [`0388fc9`](0388fc95e6) update filename_format for ProfilerMiddleware.
- Additional commits viewable in [compare view](https://github.com/pallets/werkzeug/compare/0.15.2...0.15.3)
</details>
<br />
[](https://help.github.com/articles/configuring-automated-security-fixes)
Dependabot will resolve any conflicts with this PR as long as you don't alter it yourself. You can also trigger a rebase manually by commenting `@dependabot rebase`.
[//]: # (dependabot-automerge-start)
[//]: # (dependabot-automerge-end)
---
<details>
<summary>Dependabot commands and options</summary>
<br />
You can trigger Dependabot actions by commenting on this PR:
- `@dependabot rebase` will rebase this PR
- `@dependabot recreate` will recreate this PR, overwriting any edits that have been made to it
- `@dependabot merge` will merge this PR after your CI passes on it
- `@dependabot squash and merge` will squash and merge this PR after your CI passes on it
- `@dependabot cancel merge` will cancel a previously requested merge and block automerging
- `@dependabot reopen` will reopen this PR if it is closed
- `@dependabot ignore this [patch|minor|major] version` will close this PR and stop Dependabot creating any more for this minor/major version (unless you reopen the PR or upgrade to it yourself)
- `@dependabot ignore this dependency` will close this PR and stop Dependabot creating any more for this dependency (unless you reopen the PR or upgrade to it yourself)
- `@dependabot use these labels` will set the current labels as the default for future PRs for this repo and language
- `@dependabot use these reviewers` will set the current reviewers as the default for future PRs for this repo and language
- `@dependabot use these assignees` will set the current assignees as the default for future PRs for this repo and language
- `@dependabot use this milestone` will set the current milestone as the default for future PRs for this repo and language
You can disable automated security fix PRs for this repo from the [Security Alerts page](https://github.com/Mailu/Mailu/network/alerts).
</details>
Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com>
1211: Split HOST_ANTISPAM in HOST_ANTISPAM_MILTER and HOST_ANTISPAM_WEBUI r=mergify[bot] a=micw
## What type of PR?
bug-fix
## What does this PR do?
Fixes#1190 by separating HOST_ANTISPAM into HOST_ANTISPAM_MILTER and HOST_ANTISPAM_WEBUI
### Related issue(s)
- closes#1190
- closes#1150
## Prerequistes
Before we can consider review and merge, please make sure the following list is done and checked.
If an entry in not applicable, you can check it or remove it from the list.
- [ ] In case of feature or enhancement: documentation updated accordingly
- [x] Unless it's docs or a minor change: add [changelog](https://mailu.io/master/contributors/guide.html#changelog) entry file.
Co-authored-by: Michael Wyraz <michael@wyraz.de>
1198: Enable access log of admin service only for log levels of INFO and finer r=muhlemmer a=micw
## What type of PR?
bug fix
## What does this PR do?
### Related issue(s)
- closes#1197
## Prerequistes
- [x] Unless it's docs or a minor change: add [changelog](https://mailu.io/master/contributors/guide.html#changelog) entry file.
1204: Add initial admin account to kubernetes example r=muhlemmer a=micw
## What type of PR?
documentation
## What does this PR do?
Add INITIAL_ADMIN_* example to kubernetes configmap.yaml
### Related issue(s)
- closes#1201
## Prerequistes
Before we can consider review and merge, please make sure the following list is done and checked.
If an entry in not applicable, you can check it or remove it from the list.
- docs example only
1207: Add Japanese translation r=muhlemmer a=IchikawaYukko
## What type of PR?
Translation
## What does this PR do?
Provide completed Japanese translation.
### Related issue(s)
None
## Prerequistes
Before we can consider review and merge, please make sure the following list is done and checked.
If an entry in not applicable, you can check it or remove it from the list.
- [*] In case of feature or enhancement: documentation updated accordingly
- [*] Unless it's docs or a minor change: add [changelog](https://mailu.io/master/contributors/guide.html#changelog) entry file.
1208: Persist mailqueue r=muhlemmer a=micw
## What type of PR?
bug-fix
## What does this PR do?
Makes postfix mailqueue presistent (for docker, swarm and kubernetes)
### Related issue(s)
- closes#1161
## Prerequistes
Before we can consider review and merge, please make sure the following list is done and checked.
If an entry in not applicable, you can check it or remove it from the list.
- [x] Unless it's docs or a minor change: add [changelog](https://mailu.io/master/contributors/guide.html#changelog) entry file.
Co-authored-by: Michael Wyraz <michael@wyraz.de>
Co-authored-by: Michael Wyraz <michael.wyraz@evermind.de>
Co-authored-by: U-COREI3-3120M\市川ゆり子 <ichikawayuriko@yahoo.co.jp>
Co-authored-by: micw <michael@wyraz.de>
1196: Change default password scheme to PBKDF2 (#1194) r=mergify[bot] a=hoellen
## What type of PR?
enhancement
## What does this PR do?
This PR change the default password scheme to PBKDF2. It is already changed in some places (e.g. [docs/compose/.env](https://github.com/Mailu/Mailu/blob/master/docs/compose/.env#L142)).
### Related issue(s)
closes#1194
## Prerequistes
Before we can consider review and merge, please make sure the following list is done and checked.
If an entry in not applicable, you can check it or remove it from the list.
- [x] In case of feature or enhancement: documentation updated accordingly
- [x] Unless it's docs or a minor change: add [changelog](https://mailu.io/master/contributors/guide.html#changelog) entry file.
Co-authored-by: hoellen <dev@hoellen.eu>
940: Fix host variables r=mergify[bot] a=ionutfilip
## What type of PR?
bug-fix
## What does this PR do?
### Related issue(s)
- Fix#884
## Prerequistes
Before we can consider review and merge, please make sure the following list is done and checked.
If an entry in not applicable, you can check it or remove it from the list.
- [x] In case of feature or enhancement: documentation updated accordingly
- [x] Unless it's docs or a minor change: place entry in the [changelog](CHANGELOG.md), under the latest un-released version.
Co-authored-by: Ionut Filip <ionut.philip@gmail.com>
Co-authored-by: Tim Möhlmann <muhlemmer@gmail.com>
883: Admin create user enhancement r=mergify[bot] a=cr1st1p
## What type of PR?
Enhancement
## What does this PR do?
It allows the admin docker image to also create the admin user.
The idea is that in my kubernetes setup, I do not want to manually do anything, as such, I need a way for the admin user to also be created automatically without me getting inside the pod.
So I had to change the manage.py function that creates the user to allow different 'modes' (me, I'll be using 'ifmissing') and also start.py to call that functionality if appropriate environment variables are present.
So now, in my Deployment, I add 3 more environment variables and I get the admin user created, IF not already present.
### Related issue(s)
none?
## Prerequistes
Before we can consider review and merge, please make sure the following list is done and checked.
If an entry in not applicable, you can check it or remove it from the list.
- [x] In case of feature or enhancement: documentation updated accordingly
- [x] Unless it's docs or a minor change: place entry in the [changelog](CHANGELOG.md), under the latest un-released version.
Co-authored-by: cristi <cristi.posoiu@gmail.com>
Co-authored-by: cr1st1p <cristi.posoiu@gmail.com>
Co-authored-by: Tim Möhlmann <muhlemmer@gmail.com>
1052: Upgrade alpine to 3.10 r=mergify[bot] a=Nebukadneza
## What type of PR?
enhancement / bug-fix
## What does this PR do?
Upgrade the alpine base image to 3.10 and clean up ensuing problems. Also directly uses postfix foreground-running with stdout logging.
### Related issue(s)
closes#1049closes#1051
Note: This is a duplicate effort of #1050#1039 …
## Prerequistes
- [X] In case of feature or enhancement: documentation updated accordingly
- [X] Unless it's docs or a minor change: add [changelog](https://mailu.io/master/contributors/guide.html#changelog) entry file.
With these images, I have tested manually:
- Email receive to user
- on main domain
- on additional domain
- via an alias on main-domain
- via an alias on additional domain
- via catchall
- Email reject
- of eicar testfiles
- Email sending
- Fetchmail from legacy POP
- Front LE certificates
- Simple overriding for postfix (only postfix.cf), nginx, dovecot, rspamd
- Creating a CalDAV calendar and CardDAV entry using davx5
I have not (yet) tested:
- certdumper
- databases other than sqlite
Todo:
- [x] Get rid of podop at `git+https://github.com/Nebukadneza/Podop.git@fix_py37` once https://github.com/Mailu/Podop/pull/6 is merged
- [x] Bend Mailustart back to original repository
- [x] Test more (? also help wanted ?)
Co-authored-by: Thomas Sänger <thomas@gecko.space>
Co-authored-by: Dario Ernst <dario@kanojo.de>
Co-authored-by: Reto Glauser <git@blinkeye.ch>
Jaume Barber
lub