2058: Implement versioning for CI/CD workflow. r=mergify[bot] a=Diman0
## What type of PR?
Feature!
## What does this PR do?
This PR introduces 3 things
- Add versioning (tagging) for branch x.y (1.8). E.g. 1.8.0, 1.8.1 etc.
- docker repo will contain x.y (latest) and x.y.z (pinned version) images.
- The X.Y.Z tag is incremented automatically. E.g. if 1.8.0 already exists, then the next merge on 1.8 will result in the new tag 1.8.1 being used.
- Make the version available in the image.
- For X.Y and X.Y.Z write the version (X.Y.Z) into /version on the image and add a label with version=X.Y.Z
- This means that the latest X.Y image shows the pinned version (X.Y.Z e.g. 1.8.1) it was based on. Via the tag X.Y.Z you can see the commit hash that triggered the built.
- For master write the commit hash into /version on the image and add a label with version={commit hash}
- Automatic releases. For x.y triggered builts (e.g. merge on 1.9) do a new github release for the pinned x.y.z (e.g. 1.9.2).
- Release shows a static message (see RELEASE_TEMPLATE.md) that explains how to reach the newsfragments folder and change the branch to the tag (x.y.z) mentioned in the release. Now you can get the changelog by reading all newsfragment files in this folder.
This PR does not change anything to our workflow (what we (human persons) do). Our processes are still exactly the same. The above introduced logic is automatic. When we backport to X.Y all the magic for creating the pinned version X.Y.Z is handled by the CI/CD workflow.
### Related issue(s)
- closes#1182
## Prerequisites
Before we can consider review and merge, please make sure the following list is done and checked.
If an entry in not applicable, you can check it or remove it from the list.
- [x] In case of feature or enhancement: documentation updated accordingly
- [x] Unless it's docs or a minor change: add [changelog](https://mailu.io/master/contributors/workflow.html#changelog) entry file.
## Testing
Suggested testing steps. This should cover all situations including BORS. It does require that you use your own docker repo or temporarily create a new one.
Suggested testing steps.
1. Create new github repo.
2. Add the required docker secrets to the project (see beginning of CI.yml for the secret names), DOCKER_UN, DOCKER_PW, DOCKER_ORG, DOCKER_ORG_TESTS.
3. Clone the project.
4. Copy the contents of the PR to the cloned project.
5. Push to your new github repo.
6. Now master images are built. Check that images with tag master are pushed to your docker repo
7. Check with docker inspect nginx:master that it has the label version={commit hash}.
8. Run an image, run `docker-compose exec <name> cat /version`. Note that /version also contains the pinned version. For master the pinned version is the commit hash.
9. Create branch 1.8.
10. Push branch 1.8 to repo.
11. Note that tags 1.8 and 1.8.0 are built and pushed to docker repo
12. Inspect label and /version. Note that 1.8 and 1.8.0 both show version 1.8.0.
13. Push another commit to branch 1.8.
14. Note that tags 1.8 and 1.8.1 are built and pushed to docker repo
15. Inspect label and /version. Note that 1.8 and 1.8.1 both show version 1.8.1.
16. Let's check BORS stuff.
17. Create branch testing.
18. Push the commit with the exact commit text (IMPORTANT!!): `Try #1234:`'.
19. Note that images are built and pushed for tag `pr-1234`.
20. Inspect label and /version. Note that the version is `pr-1234`.
20. Create branch staging.
21. Push the commit with commit text: `Merge #1234`.
22. Note that this image is not pushed to docker (as expected).
but you could also check the GH repo and docker repo I used:
https://github.com/Diman0/Mailu_Forkhttps://hub.docker.com/r/diman/rainloop/tags
Co-authored-by: Dimitri Huisman <diman@huisman.xyz>
2076: fix the default for DEFER_ON_TLS_ERROR r=mergify[bot] a=nextgens
## What type of PR?
bug-fix
## What does this PR do?
The default wasn't set anywhere
Co-authored-by: Florent Daigniere <nextgens@freenetproject.org>
- Moved run to bottom of Dockerfile to allow using unmodified / cached states.
- Simplified bash code in deploy.sh.
- Improved the large bash one-liner in CI.yml. It could not handle >9 for 1.x.
2044: Vault/rspamd: don't return any key for relayed domains r=mergify[bot] a=nextgens
## What type of PR?
enhancement
## What does this PR
Don't return any key for relayed domains. We may want to revisit this (ARC signing)... but in the meantime it saves from a scary message in rspamd.
```signing failure: cannot request data from the vault url: /internal/rspamd/vault/v1/dkim/ ...```
Co-authored-by: Florent Daigniere <nextgens@freenetproject.org>
2042: Add MESSAGE_RATELIMIT_EXEMPTION r=mergify[bot] a=nextgens
## What type of PR?
Enhancement
## What does this PR do?
Add a new knob called ```MESSAGE_RATELIMIT_EXEMPTION```.
### Related issue(s)
- #1774
## Prerequisites
Before we can consider review and merge, please make sure the following list is done and checked.
If an entry in not applicable, you can check it or remove it from the list.
- [x] In case of feature or enhancement: documentation updated accordingly
- [ ] Unless it's docs or a minor change: add [changelog](https://mailu.io/master/contributors/workflow.html#changelog) entry file.
Co-authored-by: Florent Daigniere <nextgens@freenetproject.org>
2034: Add timezone to containers r=mergify[bot] a=DjVinnii
## What type of PR?
Enhancement
## What does this PR do?
This PR adds the tzdata package so that the environment variable `TZ` can be used to set the timezone of containers.
### Related issue(s)
- closes#1154
## Prerequisites
Before we can consider review and merge, please make sure the following list is done and checked.
If an entry in not applicable, you can check it or remove it from the list.
- [x] In case of feature or enhancement: documentation updated accordingly
- [x] Unless it's docs or a minor change: add [changelog](https://mailu.io/master/contributors/workflow.html#changelog) entry file.
Co-authored-by: DjVinnii <vincentkling@msn.com>
We may want to revisit this (ARC signing)... but in the meantime
it saves from a scary message in rspamd
signing failure: cannot request data from the vault url: /internal/rspamd/vault/v1/dkim/ ...
this is working fine, but introduces a sqlalchemy warning
when using config-import:
/app/mailu/schemas.py:822:
SAWarning: Identity map already had an identity for (...),
replacing it with newly flushed object.
Are there load operations occurring inside of an event handler
within the flush?
ConfigManager should not replace app.config - this is causing trouble
with some other flask modules (swagger).
Updated ConfigManager to only modify app.config and not replace it.
2036: round display of range inputs to 2 decimals r=mergify[bot] a=ghostwheel42
## What type of PR?
small fix
## What does this PR do?
rounds display of range inputs to 2 decimals
### Related issue(s)
- small fix to #1966
## Prerequisites
Before we can consider review and merge, please make sure the following list is done and checked.
If an entry in not applicable, you can check it or remove it from the list.
- [X] In case of feature or enhancement: documentation updated accordingly
- [X] Unless it's docs or a minor change: add [changelog](https://mailu.io/master/contributors/workflow.html#changelog) entry file.
Co-authored-by: Alexander Graf <ghostwheel42@users.noreply.github.com>
2027: Make logs more quiet r=mergify[bot] a=nextgens
## What type of PR?
enhancement
## What does this PR do?
It silences various useless log messages in front, specifically:
```
Oct 30 03:11:04 instance-20210109-1612 docker-front[1963]: 127.0.0.1 - - [30/Oct/2021:03:11:04 +0000] "GET /health HTTP/1.1" 301 162 "-" "curl/7.78.0"
Oct 30 03:11:04 instance-20210109-1612 docker-front[1963]: 127.0.0.1 - - [30/Oct/2021:03:11:04 +0000] "GET /health HTTP/2.0" 204 0 "-" "curl/7.78.0"
Oct 30 03:11:04 instance-20210109-1612 docker-front[1963]: 2021/10/30 03:11:04 [info] 476302#476302: *2622679 client 127.0.0.1 closed keepalive connection
Oct 30 03:13:02 instance-20210109-1612 docker-front[1963]: 127.0.0.1 - - [30/Oct/2021:03:13:02 +0000] "GET /auth/email HTTP/1.0" 200 0 "-" "-"
```
`@micw` has requested it for k8s
2030: Fix RELAYNETS r=mergify[bot] a=nextgens
## What type of PR?
bug-fix
## What does this PR do?
RELAYNETS should be comma separated like everything else; rspamd should also be aware of what is considered "trusted".
I am not sure whether ```local_networks``` is the right configuration option for it though
- close#360
Co-authored-by: Florent Daigniere <nextgens@freenetproject.org>
1987: Enhancement to the rate limits r=mergify[bot] a=nextgens
## What type of PR?
enhancement
## What does this PR do?
Turn the rate-limiters into something useful (that won't fire for no reason).
- fix rate-limiting on /webdav/
- it changes the rate-limiting behaviour from limiting a single IP address to a subnet of a reasonable size (/24 on v4 and /56 on v6 both are now configurable) : AUTH_RATELIMIT_IP / AUTH_RATELIMIT_IP_V4_MASK / AUTH_RATELIMIT_IP_V6_MASK
- It ensures we only use IP-based rate-limits for attempts on accounts that do not exist
- it creates a new rate limit preventing attackers from targetting a specific user account (separate from what's above) : AUTH_RATELIMIT_USER
- it introduces a rate limiting exemption mechanism whereby, upon authentication, users will see their source-ip address being exempt for a specific amount of time AUTH_RATELIMIT_EXEMPTION_LENGTH. A similar mechanism is available for web-based sessions (see below)
- It introduces in AUTH_RATELIMIT_EXEMPTION a comma separated list of network CIDRs that will be exempt from both types of rate limiting
- it implements device-tokens, as described on https://owasp.org/www-community/Slow_Down_Online_Guessing_Attacks_with_Device_Cookies to ensure that genuine users aren't locked-out by a malicious attacker abusing the rate-limit feature.
Things that could be improved include:
- the IP-based rate limiter flags attempts against "non-existing" accounts: it could go further and flag the number of unique non-existing accounts attempted (to prevent the case of a user making a typo in his MUA configuration)
- the IP address exemption mechanism doesn't pin the exemption to a specific username: any real user can trivially bypass the rate limits (and attempt to brute-force someone else's account)
### Related issue(s)
- close#1926
- close#1745
- close#1915
## Prerequisites
Before we can consider review and merge, please make sure the following list is done and checked.
If an entry in not applicable, you can check it or remove it from the list.
- [x] In case of feature or enhancement: documentation updated accordingly
- [x] Unless it's docs or a minor change: add [changelog](https://mailu.io/master/contributors/workflow.html#changelog) entry file.
Co-authored-by: Florent Daigniere <nextgens@freenetproject.org>
Co-authored-by: Diman0 <diman@huisman.xyz>
Co-authored-by: Florent Daigniere <nextgens@users.noreply.github.com>
2018: show dmarc record for report domain in domain details r=mergify[bot] a=ghostwheel42
## What type of PR?
documentation
## What does this PR do?
show dmarc record for report domain in domain details
### Related issue(s)
closes#1382
## Prerequisites
- [X] In case of feature or enhancement: documentation updated accordingly
- [X] Unless it's docs or a minor change: add [changelog](https://mailu.io/master/contributors/workflow.html#changelog) entry file.
Co-authored-by: Alexander Graf <ghostwheel42@users.noreply.github.com>
1986: Document how to setup client autoconfig r=mergify[bot] a=nextgens
## What type of PR?
enhancement
## What does this PR do?
Document how to setup autoconfig. This works with most open-source MUAs (thunderbird, evolution, ...)
We could go further than that by providing dynamic configuration (issue an auth token for each MUA request)... but it won't work unless a new DNS entry (and matching certificate) is created.
### Related issue(s)
- #224
## Prerequisites
Before we can consider review and merge, please make sure the following list is done and checked.
If an entry in not applicable, you can check it or remove it from the list.
- [x] In case of feature or enhancement: documentation updated accordingly
- [x] Unless it's docs or a minor change: add [changelog](https://mailu.io/master/contributors/workflow.html#changelog) entry file.
2014: Update Chinese translation r=mergify[bot] a=qy117121
## What type of PR?
translation
## What does this PR do?
Update Chinese translation. Use `zh` instead of `zh_CN`.
### Related issue(s)
none
## Prerequistes
Before we can consider review and merge, please make sure the following list is done and checked.
If an entry in not applicable, you can check it or remove it from the list.
- [X] In case of feature or enhancement: documentation updated accordingly
- [X] Unless it's docs or a minor change: add [changelog](https://mailu.io/master/contributors/workflow.html#changelog) entry file.
Co-authored-by: Florent Daigniere <nextgens@freenetproject.org>
Co-authored-by: qy117121 <mixuan121@gmail.com>
Co-authored-by: Alexander Graf <ghostwheel42@users.noreply.github.com>
2017: rspamd: get dkim keys via REST API instead of filesystem r=mergify[bot] a=ghostwheel42
## What type of PR?
enhancement
## What does this PR do?
rspamd now uses hashicorp's vault api v1 to get dkim keys and selectors for a domain.
this allows future enhancement (multiple keys) without reconfiguring and restarting rspamd.
it also makes mounting the /dkim volume into the rspamd container unnecessary.
### Related issue(s)
- improves and closes#2012
- allows to implement key rotation using multiple selectors (see #1700)
- allows to implement dkim for alternate domains (see #1519)
- fixes and closes#1345 (selector transmitted by admin container is used)
- closes#1179 (no keys on disk)
- allows to implement key rotation from the outside (ie. via a helper script talking to some dns provider's api) (see #547)
## Prerequisites
- [X] In case of feature or enhancement: documentation updated accordingly
- [X] Unless it's docs or a minor change: add [changelog](https://mailu.io/master/contributors/workflow.html#changelog) entry file.
Co-authored-by: Alexander Graf <ghostwheel42@users.noreply.github.com>
1997: Prevent traceback when using non-email in login r=mergify[bot] a=ghostwheel42
There's a traceback when the username used to log via SMTPAUTH
in is not an email address:
=== before ===
```
[...] ERROR in app: Exception on /internal/auth/email [GET]
Traceback (most recent call last):
File "/usr/lib/python3.9/site-packages/sqlalchemy/engine/base.py", line 1179, in _execute_context
context = constructor(dialect, self, conn, *args)
File "/usr/lib/python3.9/site-packages/sqlalchemy/engine/default.py", line 719, in _init_compiled
param.append(processors[key](compiled_params[key]))
File "/usr/lib/python3.9/site-packages/sqlalchemy/sql/type_api.py", line 1201, in process
return process_param(value, dialect)
File "/app/mailu/models.py", line 60, in process_bind_param
localpart, domain_name = value.lower().rsplit('`@',` 1)
ValueError: not enough values to unpack (expected 2, got 1)
[...]
[parameters: [{'%(140657157923216 param)s': 'foobar'}]]
```
=== after ===
```
[...] WARNING in nginx: Invalid user 'foobar': (builtins.ValueError) invalid email address (no "`@")`
```
## What type of PR?
enhancement
## What does this PR do?
replace traceback (ERROR) with error message (WARNING)
### Related issue(s)
## Prerequisites
Before we can consider review and merge, please make sure the following list is done and checked.
If an entry in not applicable, you can check it or remove it from the list.
- [ ] In case of feature or enhancement: documentation updated accordingly
- [ ] Unless it's docs or a minor change: add [changelog](https://mailu.io/master/contributors/workflow.html#changelog) entry file.
Co-authored-by: Alexander Graf <ghostwheel42@users.noreply.github.com>
2007: allow sending emails as user+detail@domain.tld r=mergify[bot] a=ghostwheel42
## What type of PR?
bug-fix or enhancement
## What does this PR do?
Allows sending emails with an added "+detail" in the local part.
### Related issue(s)
closes#1948
## Prerequisites
Before we can consider review and merge, please make sure the following list is done and checked.
If an entry in not applicable, you can check it or remove it from the list.
- [ ] In case of feature or enhancement: documentation updated accordingly
- [X] Unless it's docs or a minor change: add [changelog](https://mailu.io/master/contributors/workflow.html#changelog) entry file.
Co-authored-by: Alexander Graf <ghostwheel42@users.noreply.github.com>
Co-authored-by: root <ghostwheel42@users.noreply.github.com>
Co-authored-by: Florent Daigniere <nextgens@freenetproject.org>
There's a traceback when the username used to log via SMTPAUTH
in is not an email address:
=== before ===
```
[...] ERROR in app: Exception on /internal/auth/email [GET]
Traceback (most recent call last):
File "/usr/lib/python3.9/site-packages/sqlalchemy/engine/base.py", line 1179, in _execute_context
context = constructor(dialect, self, conn, *args)
File "/usr/lib/python3.9/site-packages/sqlalchemy/engine/default.py", line 719, in _init_compiled
param.append(processors[key](compiled_params[key]))
File "/usr/lib/python3.9/site-packages/sqlalchemy/sql/type_api.py", line 1201, in process
return process_param(value, dialect)
File "/app/mailu/models.py", line 60, in process_bind_param
localpart, domain_name = value.lower().rsplit('@', 1)
ValueError: not enough values to unpack (expected 2, got 1)
[...]
[parameters: [{'%(140657157923216 param)s': 'foobar'}]]
```
=== after ===
```
[...] WARNING in nginx: Invalid user 'foobar': (builtins.ValueError) invalid email address (no "@")
```
1966: AdminLTE3 optimizations & compression and caching r=mergify[bot] a=ghostwheel42
## What type of PR?
enhancement, bugfix
## What does this PR do?
Optimization and cleanup of styles and javascript code for AdminLTE 3
Adds caching headers, gzip and robots.txt to nginx.
### Related issue(s)
Makes #1800 even better. Thanks to `@DjVinnii` and `@Diman0` for the good work.
Closes#1905
## Prerequistes
Before we can consider review and merge, please make sure the following list is done and checked.
If an entry in not applicable, you can check it or remove it from the list.
- [X] In case of feature or enhancement: documentation updated accordingly
- [X] Unless it's docs or a minor change: add [changelog](https://mailu.io/master/contributors/workflow.html#changelog) entry file.
Co-authored-by: Alexander Graf <ghostwheel42@users.noreply.github.com>
Co-authored-by: Dimitri Huisman <diman@huisman.xyz>
1971: Updated Polish translation. r=mergify[bot] a=ghostwheel42
## What type of PR?
translation
## What does this PR do?
Update polish translation. Used `pl/LC_MESSAGES/messages.po` from PR #1751 created by `@martys71`
Part of Discussion of 1.9 roadmap #1930
### Related issue(s)
closes#1751
## Prerequistes
Before we can consider review and merge, please make sure the following list is done and checked.
If an entry in not applicable, you can check it or remove it from the list.
- [X] In case of feature or enhancement: documentation updated accordingly
- [X] Unless it's docs or a minor change: add [changelog](https://mailu.io/master/contributors/workflow.html#changelog) entry file.
Co-authored-by: Alexander Graf <ghostwheel42@users.noreply.github.com>
1975: Replace traceback with error message when creating initial admin user r=mergify[bot] a=ghostwheel42
## What type of PR?
small enhancement
## What does this PR do?
when creating the admin user via cli a traceback is shown when this user is already present in the database.
This is confusing users. I've replaced the traceback with an error message.
### Related issue(s)
#1921
## Prerequisites
Before we can consider review and merge, please make sure the following list is done and checked.
If an entry in not applicable, you can check it or remove it from the list.
- [X] In case of feature or enhancement: documentation updated accordingly
- [X] Unless it's docs or a minor change: add [changelog](https://mailu.io/master/contributors/workflow.html#changelog) entry file.
Co-authored-by: Alexander Graf <ghostwheel42@users.noreply.github.com>
1968: optimize handle_authentication r=mergify[bot] a=ghostwheel42
## What type of PR?
bug-fix
## What does this PR do?
catch utf-8 decoding errors and log a warning in handle_authentication instead of writing a traceback into the log.
### Related issue(s)
closes#1361
## Prerequistes
Before we can consider review and merge, please make sure the following list is done and checked.
If an entry in not applicable, you can check it or remove it from the list.
- [X] In case of feature or enhancement: documentation updated accordingly
- [X] Unless it's docs or a minor change: add [changelog](https://mailu.io/master/contributors/workflow.html#changelog) entry file.
Co-authored-by: Alexander Graf <ghostwheel42@users.noreply.github.com>
1967: fix 1789: ensure that nginx resolves ipv4 addresses r=mergify[bot] a=nextgens
## What type of PR?
bug-fix
## What does this PR do?
This fixes ipv6 enabled setup by disabling it. If you were using SUBNET6 in your configuration, odds are it's broken since gunicorn isn't bound on an on an ipv6 enabled socket.
Should we backport this?
### Related issue(s)
- close#1789
- close#1802
Co-authored-by: Florent Daigniere <nextgens@freenetproject.org>
- fixed copy of qemu-arm-static for alpine
- added 'set -eu' safeguard
- silenced npm update notification
- added color to webpack call
- changed Admin-LTE default blue
(core/admin/Dockerfile)
- AdminLTE 3 style tweaks
(core/admin/assets/app.css)
(core/admin/mailu/ui/templates/base.html)
(core/admin/mailu/ui/templates/sidebar.html)
- localized datatables
(core/admin/Dockerfile)
(core/admin/assets/app.js)
(core/admin/package.json)
- moved external javascript code to vendor.js
(core/admin/assets/app.js)
(core/admin/assets/vendor.js)
(core/admin/webpack.config.js)
- added mailu logo
(core/admin/assets/app.js)
(core/admin/assets/app.css)
(core/admin/assets/mailu.png)
- moved all inline javascript to app.js
(core/admin/assets/app.js)
(core/admin/mailu/ui/templates/domain/create.html)
(core/admin/mailu/ui/templates/user/create.html)
- added iframe display of rspamd page
(core/admin/assets/app.js)
(core/admin/mailu/ui/views/base.py)
(core/admin/mailu/ui/templates/sidebar.html)
(core/admin/mailu/ui/templates/antispam.html)
- updated language-selector to display full language names and use post
(core/admin/assets/app.js)
(core/admin/mailu/__init__.py)
(core/admin/mailu/utils.py)
(core/admin/mailu/ui/views/languages.py)
- added fieldset to group and en/disable input fields
(core/admin/assets/app.js)
(core/admin/mailu/ui/templates/macros.html)
(core/admin/mailu/ui/templates/user/settings.html)
(core/admin/mailu/ui/templates/user/reply.html)
- added clipboard copy buttons
(core/admin/assets/app.js)
(core/admin/assets/vendor.js)
(core/admin/mailu/ui/templates/macros.html)
(core/admin/mailu/ui/templates/domain/details.html)
- cleaned external javascript imports
(core/admin/assets/vendor.js)
- pre-split first hostname for further use
(core/admin/mailu/__init__.py)
(core/admin/mailu/models.py)
(core/admin/mailu/ui/templates/client.html)
(core/admin/mailu/ui/templates/domain/signup.html)
- cache dns_* properties of domain object (immutable during runtime)
(core/admin/mailu/models.py)
(core/admin/mailu/ui/templates/domain/details.html)
- fixed and splitted dns_dkim property of domain object (space missing)
- added autoconfig and tlsa properties to domain object
(core/admin/mailu/models.py)
- suppressed extra vertical spacing in jinja2 templates
- improved accessibility for screen reader
(core/admin/mailu/ui/templates/**.html)
- deleted unused/broken /user/forward route
(core/admin/mailu/ui/templates/user/forward.html)
(core/admin/mailu/ui/views/users.py)
- updated gunicorn to 20.1.0 to get rid of buffering error at startup
(core/admin/requirements-prod.txt)
- switched webpack to production mode
(core/admin/webpack.config.js)
- added css and javascript minimization
- added pre-compression of assets (gzip)
(core/admin/webpack.config.js)
(core/admin/package.json)
- removed obsolte dependencies
- switched from node-sass to dart-sass
(core/admin/package.json)
- changed startup cleaning message from error to info
(core/admin/mailu/utils.py)
- move client config to "my account" section when logged in
(core/admin/mailu/ui/templates/sidebar.html)
1800: AdminLTE 3 r=mergify[bot] a=DjVinnii
## What type of PR?
Enhancement
## What does this PR do?
This PR implements AdminLTE 3 for the admin interface. It also includes the implementation of DataTables and a language selector.
### Related issue(s)
- closes: #1567
- closes: #1764
## Prerequistes
- [x] Unless it's docs or a minor change: add [changelog](https://mailu.io/master/contributors/guide.html#changelog) entry file.
Co-authored-by: Vincent Kling <vincentkling@msn.com>
Co-authored-by: DjVinnii <vincentkling@msn.com>
Co-authored-by: Dimitri Huisman <52963853+Diman0@users.noreply.github.com>
Co-authored-by: Diman0 <diman@huisman.xyz>
Co-authored-by: Dimitri Huisman <diman@huisman.xyz>
This will default to True and defer emails that fail even "loose"
validation of DANE or MTA-STS
It should work most of the time but if it doesn't and you would rather
see your emails delivered, you can turn it off.
1873: Completed Hebrew translation r=mergify[bot] a=yarons
The Hebrew translation is incomplete so I've completed it.
Co-authored-by: Yaron Shahrabani <sh.yaron@gmail.com>
1939: Ensure that we don't do multiple DNS lookups in the sieve script r=mergify[bot] a=nextgens
## What type of PR?
bug-fix
## What does this PR do?
It ensures that DNS lookups don't introduce inconsistent state. We may want to go further and actually check the return codes of rspamc too.
I haven't tested it but it should work.
### Related issue(s)
- #1938
Co-authored-by: Florent Daigniere <nextgens@freenetproject.org>
1916: Ratelimit outgoing emails per user r=mergify[bot] a=nextgens
## What type of PR?
Feature
## What does this PR do?
A conflict-free version of #1360 implementing per-user sender limits
### Related issue(s)
- close#1360
- close#1031
- close#1774
## Prerequistes
Before we can consider review and merge, please make sure the following list is done and checked.
If an entry in not applicable, you can check it or remove it from the list.
- [x] In case of feature or enhancement: documentation updated accordingly
- [x] Unless it's docs or a minor change: add [changelog](https://mailu.io/master/contributors/workflow.html#changelog) entry file.
Co-authored-by: Florent Daigniere <nextgens@freenetproject.org>
Co-authored-by: Dimitri Huisman <diman@huisman.xyz>
Adds a 'none' env option to WEBROOT_REDIRECT so that no `location /`
configuration is written to nginx.conf.
This is useful for setting up Mailu and Mailman where we override the
root to proxy to the mailing list server instead. Without this change
the nginx container will not start, or for 1.7 users can set their
WEBMAIL_PATH to / with no webmail to get the same results.
This fix means that future users don't have to choose between webmail
and a root override and makes the configuration intention clear.
I have found a list of the top100 email destinations online and ran them
through a script to ensure that all of their MX servers had valid
configuration... this is the result
1656: Add ability to set no WEBROOT_REDIRECT to Nginx r=mergify[bot] a=DavidFair
## What type of PR?
Enhancement / Documentation
## What does this PR do?
From commit:
---
Adds a 'none' env option to WEBROOT_REDIRECT so that no `location /`
configuration is written to nginx.conf.
This is useful for setting up Mailu and Mailman where we override the
root to proxy to the mailing list server instead. Without this change
the nginx container will not start, or for 1.7 users can set their
WEBMAIL_PATH to / with no webmail to get the same results.
This fix means that future users don't have to choose between webmail
and a root override and makes the configuration intention clear.
---
I've also added bullet points to break up a long flowing sentence in `configuration.rst` - it should be a bit easier to read now
### Related issue(s)
No Related Issue - I just jumped to a PR
## Prerequistes
Before we can consider review and merge, please make sure the following list is done and checked.
If an entry in not applicable, you can check it or remove it from the list.
- [x] In case of feature or enhancement: documentation updated accordingly
@ Maintainers - Is this worthy of the changelog, it's useful to know about but I imagine the number of people it affects is equally minimal?
- [ ] Unless it's docs or a minor change: add [changelog](https://mailu.io/master/contributors/guide.html#changelog) entry file.
Co-authored-by: David Fairbrother <DavidFair@users.noreply.github.com>
1901: treat localpart case insensitive again r=mergify[bot] a=ghostwheel42
## What type of PR?
bug-fix
## What does this PR do?
fixes error introduced by #1604 where the localpart of an email address was handled case sensitive.
this screwed things up at various other places.
### Related issue(s)
closes#1895closes#1900
Co-authored-by: Alexander Graf <ghostwheel42@users.noreply.github.com>
1856: update asset builder dependencies r=mergify[bot] a=ghostwheel42
## What type of PR?
update asset builder dependencies
## What does this PR do?
only include needed dependencies to build mailu assets with nodejs v8
### Related issue(s)
update dependencies as discussed in #1829
Co-authored-by: Alexander Graf <ghostwheel42@users.noreply.github.com>
1896: save dkim key after creation r=mergify[bot] a=ghostwheel42
## What type of PR?
bug-fix
## What does this PR do?
saves generated dkim key after creation vi web ui.
after the model change the domain object needs to be added and flushed via sqlalchemy.
### Related issue(s)
closes#1892
Co-authored-by: Alexander Graf <ghostwheel42@users.noreply.github.com>
1888: Use threads in gunicorn rather than workers/processes r=mergify[bot] a=nextgens
## What type of PR?
enhancement
## What does this PR do?
This ensures that we share the auth-cache... will enable memory savings
and may improve performances when a higher number of cores is available
"smarter default"
Co-authored-by: Florent Daigniere <nextgens@freenetproject.org>
1885: fix 1884: always lookup a FQDN r=mergify[bot] a=nextgens
## What type of PR?
bugfix
## What does this PR do?
Fix bug #1884. Ensure that we avoid the musl resolver bug by always looking up a FQDN
### Related issue(s)
- closes#1884
Co-authored-by: Florent Daigniere <nextgens@freenetproject.org>
1890: fix Email class in model.py r=mergify[bot] a=ghostwheel42
## What type of PR?
bug-fix
## What does this PR do?
fixes class Email - keep email, localpart and domain in sync.
### Related issue(s)
closes#1878
Co-authored-by: Alexander Graf <ghostwheel42@users.noreply.github.com>
This ensures that we share the auth-cache... will enable memory savings
and may improve performances when a higher number of cores is available
"smarter default"
1857: disable startdate when autoreply is disabled r=mergify[bot] a=ghostwheel42
## What type of PR?
bug-fix
## What does this PR do?
disable the reply startdate field when autoreply is disabled
Co-authored-by: Alexander Graf <ghostwheel42@users.noreply.github.com>
1791: Enhanced session handling r=mergify[bot] a=ghostwheel42
## What type of PR?
bug-fix
## What does this PR do?
- replaces flask_kvsession and simplekv with a mailu-specific session store
- call cleanup_sessions before first request and not on startup.
this allows to run cmdline actions without redis (and makes it faster)
- allow running without redis for debugging purposes by setting MEMORY_SESSIONS to True
- don't sign session id, as it has plenty of entropy (as suggested by nextgens)
- adds method to prune a user's sessions
### Related issue(s)
- enhances and close#1787
Co-authored-by: Alexander Graf <ghostwheel42@users.noreply.github.com>
1754: centralize Webmail authentication behind the admin panel (SSO) r=mergify[bot] a=nextgens
## What type of PR?
Enhancement: it centralizes the authentication of webmails to the admin interface.
## What does this PR do?
It implements the glue required for webmails to do SSO using the admin interface.
One of the main advantages of centralizing things this way is that it reduces significantly the attack surface available to an unauthenticated attacker (no webmail access until there is a valid Flask session).
Others include the ability to implement 2FA down the line and rate-limit things as required.
### Related issue(s)
- #783
## Prerequistes
Before we can consider review and merge, please make sure the following list is done and checked.
If an entry in not applicable, you can check it or remove it from the list.
- [x] In case of feature or enhancement: documentation updated accordingly
- [x] Unless it's docs or a minor change: add [changelog](https://mailu.io/master/contributors/guide.html#changelog) entry file.
Co-authored-by: Florent Daigniere <nextgens@freenetproject.org>
1758: Implement a simpler credential cache (alternative to #1755) r=mergify[bot] a=nextgens
## What type of PR?
Feature: it implements a credential cache to speedup authentication requests.
## What does this PR do?
Credentials are stored in cold-storage using a slow, salted/iterated hash function to prevent offline bruteforce attacks. This creates a performance bottleneck for no valid reason (see the
rationale/long version on https://github.com/Mailu/Mailu/issues/1194#issuecomment-762115549).
The new credential cache makes things fast again.
This is the simpler version of #1755 (with no new dependencies)
### Related issue(s)
- close#1411
- close#1194
- close#1755
## Prerequistes
Before we can consider review and merge, please make sure the following list is done and checked.
If an entry in not applicable, you can check it or remove it from the list.
- [x] In case of feature or enhancement: documentation updated accordingly
- [x] Unless it's docs or a minor change: add [changelog](https://mailu.io/master/contributors/guide.html#changelog) entry file.
1776: optimize generation of transport nexthop r=mergify[bot] a=ghostwheel42
## What type of PR?
bug-fix and enhancement.
## What does this PR do?
Possibly there should be more input validation when editing a relay, but for now this tries to make the best out of the existing "smtp" attribute while maintaining backwards compatibility. When relay is empty, the transport's nexthop is the MX of the relayed domain to fix#1588
```
RELAY NEXTHOP TRANSPORT
empty use MX of relay domain smtp:domain
:port use MX of relay domain and use port smtp:domain:port
target resolve A/AAAA of target smtp:[target]
target:port resolve A/AAAA of target and use port smtp:[target]:port
mx:target resolve MX of target smtp:target
mx:target:port resolve MX of target and use port smtp:target:port
lmtp:target resolve A/AAAA of target lmtp:target
lmtp:target:port resolve A/AAAA of target and use port lmtp:target:port
target can also be an IPv4 or IPv6 address (an IPv6 address must be enclosed in []: [2001:DB8::]).
```
When there is proper input validation and existing database entries are migrated this function can be made much shorter again.
### Related issue(s)
- closes#1588
- closes#1815
## Prerequistes
Before we can consider review and merge, please make sure the following list is done and checked.
If an entry in not applicable, you can check it or remove it from the list.
- [x] In case of feature or enhancement: documentation updated accordingly
- [X] Unless it's docs or a minor change: add [changelog](https://mailu.io/master/contributors/guide.html#changelog) entry file.
Co-authored-by: Florent Daigniere <nextgens@freenetproject.org>
Co-authored-by: Alexander Graf <ghostwheel42@users.noreply.github.com>
1746: DNS records for client autoconfiguration (RFC6186) r=Diman0 a=nextgens
## What type of PR?
Feature
## What does this PR do?
Add instructions on how to configure rfc6186 DNS records for client autoconfiguration
### Related issue(s)
- #224
- #498
## Prerequistes
Before we can consider review and merge, please make sure the following list is done and checked.
If an entry in not applicable, you can check it or remove it from the list.
- [x ] Unless it's docs or a minor change: add [changelog](https://mailu.io/master/contributors/guide.html#changelog) entry file.
Co-authored-by: Florent Daigniere <nextgens@freenetproject.org>
1694: update compression algorithms for current dovecot r=nextgens a=lub
## What type of PR?
enhancement
## What does this PR do?
This adds additional compression algorithms in accordance with
https://doc.dovecot.org/configuration_manual/zlib_plugin/
### Related issue(s)
## Prerequistes
Before we can consider review and merge, please make sure the following list is done and checked.
If an entry in not applicable, you can check it or remove it from the list.
- [x] In case of feature or enhancement: documentation updated accordingly
- [x] Unless it's docs or a minor change: add [changelog](https://mailu.io/master/contributors/guide.html#changelog) entry file.
Co-authored-by: lub <git@lubiland.de>
1649: Update docs/reverse.rst with Traefik v2+ info r=mergify[bot] a=patryk-tech
## What type of PR?
Documentation
## What does this PR do?
Adds information about using Traefik v2+ as a reverse proxy.
### Related issue(s)
Closes#1503
## Prerequistes
Before we can consider review and merge, please make sure the following list is done and checked.
If an entry in not applicable, you can check it or remove it from the list.
- [x] In case of feature or enhancement: documentation updated accordingly
- [x] Unless it's docs or a minor change: add [changelog](https://mailu.io/master/contributors/guide.html#changelog) entry file.
1673: Remove rspamd unused env var from start script r=mergify[bot] a=cbachert
## What type of PR?
Cleanup
## What does this PR do?
Remove unused environment variable FRONT_ADDRESS in rspamd. FRONT_ADDRESS references were removed with commit 8172f3e in PR #727 like mentioned in chat https://matrix.to/#/!MINuyJjJSrfowljYCK:tedomum.net/$160401946364NGNmI:imninja.net?via=huisman.xyz&via=matrix.org&via=imninja.net
```
Mailu$ grep -r "FRONT_ADDRESS" core/rspamd/
core/rspamd/start.py:os.environ["FRONT_ADDRESS"] = system.get_host_address_from_environment("FRONT", "front")
```
### Related issue(s)
N/A
## Prerequistes
- [x] Documentation updated accordingly: No documentation to update
- [x] Add to changelog: Minor change
Co-authored-by: Patryk Tech <git@patryk.tech>
Co-authored-by: cbachert <cbachert@users.noreply.github.com>
- the session key is now generated using
- a hash of the uid seeded by the apps secret_key (size: SESSION_KEY_BITS)
- a random token (size: 128 bits)
- the session's creation time (size: 32 bits)
- redis server side sessions are now refreshed after 1/2 the session lifetime
even if not modified
- the cookie is also updated if necessary
This can be used to delete all sessions belonging to a user/login.
For no it just iterates over all sessions.
This could be enhanced by using a prefix for and deleting by prefix.
- call cleanup_sessions on first kvstore access
this allows to run cmdline actions without redis (and makes it faster)
- Allow development using DictStore by setting REDIS_ADDRESS to the empty string in env
- don't sign 64bit random session id as suggested by nextgens
1785: Fix bug #1660 (don't replace nested headers) r=mergify[bot] a=nextgens
## What type of PR?
bug-fix
## What does this PR do?
Don't replace nested headers (typically in forwarded/attached emails). This will ensure we don't break cryptographic signatures.
### Related issue(s)
- close#1660
## Prerequistes
Before we can consider review and merge, please make sure the following list is done and checked.
If an entry in not applicable, you can check it or remove it from the list.
- [x] Unless it's docs or a minor change: add [changelog](https://mailu.io/master/contributors/guide.html#changelog) entry file.
Co-authored-by: Florent Daigniere <nextgens@freenetproject.org>
1783: Switch to server-side sessions r=mergify[bot] a=nextgens
## What type of PR?
bug-fix
## What does this PR do?
It simplifies session management.
- it ensures that sessions will eventually expire (*)
- it implements some mitigation against session-fixation attacks
- it switches from client-side to server-side sessions (in Redis)
It doesn't prevent us from (re)-implementing a "remember_me" type of feature if that's considered useful by some.
Co-authored-by: Florent Daigniere <nextgens@freenetproject.org>
1610: add option to enforce inbound starttls r=mergify[bot] a=lub
## What type of PR?
Feature
## What does this PR do?
It implements a check in the auth_http handler to check for Auth-SSL == on and otherwise returns a 530 starttls error.
If INBOUND_TLS_ENFORCE is not set the behaviour is still the same as before, so existing installations should be unaffected.
Although there is a small difference to e.g. smtpd_tls_security_level of Postfix.
Postfix already throws a 530 after mail from, but this solution only throws it after rcpt to. auth_http is only the request after rcpt to, so it's not possible to do it earlier.
### Related issue(s)
#1328 is kinda related, although this PR doesn't solve the issue that the headers will still display ESMTP instead of ESMTPS
## Prerequistes
Before we can consider review and merge, please make sure the following list is done and checked.
If an entry in not applicable, you can check it or remove it from the list.
- [x] In case of feature or enhancement: documentation updated accordingly
- [x] Unless it's docs or a minor change: add [changelog](https://mailu.io/master/contributors/guide.html#changelog) entry file.
Co-authored-by: lub <git@lubiland.de>
1638: Remove the username from the milter_headers r=mergify[bot] a=githtz
Rspamd adds the name of the authenticated user by default. Setting add_smtp_user to false prevents the login to be leaked.
## What type of PR?
Enhancement
## What does this PR do?
This PR prevents the user login to be leaked in sent emails (for example using an alias)
### Related issue(s)
Closes https://github.com/Mailu/Mailu/issues/1465
## Prerequistes
Before we can consider review and merge, please make sure the following list is done and checked.
If an entry in not applicable, you can check it or remove it from the list.
- [x] Unless it's docs or a minor change: add [changelog](https://mailu.io/master/contributors/guide.html#changelog) entry file.
Co-authored-by: anrc <15327800+githtz@users.noreply.github.com>
- remove PASSWORD_SCHEME altogether
- introduce CREDENTIAL_ROUNDS
- migrate all old hashes to the current format
- auto-detect/enable all hash types that passlib supports
- upgrade passlib to 1.7.4 (see #1706: ldap_salted_sha512 support)
1765: Set sensible cookie flags on the admin app r=mergify[bot] a=nextgens
## What type of PR?
Bugfix
## What does this PR do?
It sets the right flags on the session cookie issued by the admin app.
This should probably be backported as the lack of secure flag on TLS-enabled setup is a high risk vulnerability.
SameSite is hardening / helps against CSRF on modern browsers
HTTPOnly is hardening / helps reduce the impact of XSS
Co-authored-by: Florent Daigniere <nextgens@freenetproject.org>
1763: show flash messages again r=mergify[bot] a=lub
## What type of PR?
bug-fix
## What does this PR do?
This basically restores the behaviour, that got removed in
ecdf0c25b3 during refactoring.
### Related issue(s)
- noticed it while reviewing #1756
## Prerequistes
Before we can consider review and merge, please make sure the following list is done and checked.
If an entry in not applicable, you can check it or remove it from the list.
- [-] In case of feature or enhancement: documentation updated accordingly
- [-] Unless it's docs or a minor change: add [changelog](https://mailu.io/master/contributors/guide.html#changelog) entry file.
Co-authored-by: lub <git@lubiland.de>
1618: add OCSP stapling to nginx.conf r=mergify[bot] a=lub
It's not added in tls.conf, because apparently the mail ssl module
doesnt' support OCSP stapling.
https://nginx.org/en/docs/http/ngx_http_ssl_module.html#ssl_stapling
^ exists
https://nginx.org/en/docs/mail/ngx_mail_ssl_module.html#ssl_stapling
^ missing
When the configured certificate doesn't have OCSP information, it'll
just log a warning during startup.
## What type of PR?
enhancement
## What does this PR do?
It enables OCSP stapling for the http server. OCSP stapling reduces roundtrips for the client and reduces load on OCSP responders.
### Related issue(s)
- fixes #1616
## Prerequistes
Before we can consider review and merge, please make sure the following list is done and checked.
If an entry in not applicable, you can check it or remove it from the list.
- [x] Unless it's docs or a minor change: add [changelog](https://mailu.io/master/contributors/guide.html#changelog) entry file.
Co-authored-by: lub <git@lubiland.de>
1607: _FILE variables for Docker swarm secrets r=mergify[bot] a=lub
## What type of PR?
enhancement
## What does this PR do?
This PR enables usage of DB_PW_FILE and SECRET_KEY_FILE instead of DB_PW and SECRET_KEY to load these values from files instead of supplying them directly. That way it's possible to use Docker secrets.
### Related issue(s)
## Prerequistes
Before we can consider review and merge, please make sure the following list is done and checked.
If an entry in not applicable, you can check it or remove it from the list.
- [x] In case of feature or enhancement: documentation updated accordingly
- [x] Unless it's docs or a minor change: add [changelog](https://mailu.io/master/contributors/guide.html#changelog) entry file.
Co-authored-by: lub <git@lubiland.de>
With `CHUNKING`set as a capability, nginx advertises this capability to
clients at a stage where the SMTP dialog does not seem to be forwarded
to the proxy-target (postfix) yet. Nginx' SMTP parser itself does not
support the `BDAT` command issued as part of a chunke-d dialog. This makes
Nginx respond with a `250 2.0.0 OK` and close the connection, after the
mail-data got sent by the client — without forwarding this to the
proxy-target.
With this, users mail can be lost.
Furthermore, when a user uses a sieve filter to forward mail, dovecot
sometimes chunks the forwarded mail when sending it through `front`.
These forwards then fail.
Removing `CHUNKING` from the capabilities fixes this behavior.
- yaml-import is now logged via schema.Logger
- iremoved relative imports - not used in other mailu modules
- removed develepment comments
- added Mailconfig.check method to check for duplicate domain names
- converted .format() to .format_map() where possible
- switched to yaml multiline dump for dkim_key
- converted dkim_key import from regex to string functions
- automatically unhide/unexclude explicitly specified attributes on dump
- use field order when loading to stabilize import
- fail when using 'hash_password' without 'password'
- fixed logging of dkim_key
- fixed pruning and deleting of lists
- modified error messages
- added debug flag and two verbosity levels
- idna.encode does not encode upper-case letters,
so .lower() has to be called on value not on result
- split email-address on '@' only once
- converted '*'.format(*) to f-strings
- added docstrings
- removed from_dict method
- code cleanup/style (list concat, exceptions, return&else, line-length)
- added TODO comments on possible future changes
- revived original config-update function for backwards compability
- renamed config-dump to config-export to be in line with config-import
- converted '*'.format(*) to f-strings
- converted string-concatenation to f-strings
renamed single letter variables (m => match)
renamed classmethod arguments to cls (model)
removed shadowing of variables (hash, context)
shortened unneeded lambda functions (id)
converted type ... is to isinstance(...)
removed unneded imports (flask)
Updated ConfigManager to only modify app.config and not replace it.
Swagger does not play well, when app.config is not a real dict and
it is not necessary to keep ConfigManager around after init.
Also added "API" flag to config (default: disabled).
1654: Ensure that the rendered file ends with newline in order to make `pos… r=mergify[bot] a=tremlin
## What type of PR?
Bugfix
## What does this PR do?
This fixes#1580
### Related issue(s)
- closes#1580
## Prerequistes
Before we can consider review and merge, please make sure the following list is done and checked.
If an entry in not applicable, you can check it or remove it from the list.
Co-authored-by: Thomas Rehn <thomas.rehn@initos.com>
Adds a 'none' env option to WEBROOT_REDIRECT so that no `location /`
configuration is written to nginx.conf.
This is useful for setting up Mailu and Mailman where we override the
root to proxy to the mailing list server instead. Without this change
the nginx container will not start, or for 1.7 users can set their
WEBMAIL_PATH to / with no webmail to get the same results.
This fix means that future users don't have to choose between webmail
and a root override and makes the configuration intention clear.
1611: Adds own server on port 80 for letsencrypt and redirect r=mergify[bot] a=elektro-wolle
## What type of PR?
Bugfix
## What does this PR do?
Handle letsencrypt route to `.well-known` by own server configuration within nginx.
### Related issue(s)
closes#1564
## Prerequistes
Before we can consider review and merge, please make sure the following list is done and checked.
If an entry in not applicable, you can check it or remove it from the list.
- [x] Unless it's docs or a minor change: add [changelog](https://mailu.io/master/contributors/guide.html#changelog) entry file.
Co-authored-by: Wolfgang Jung <w.jung@polyas.de>
1635: Add support for AUTH LOGIN authentication mechanism for relaying emai… r=mergify[bot] a=Diman0
…l via smart hosts.
## What type of PR?
Feature
## What does this PR do?
This PR adds support to postfix for AUTH LOGIN authentication mechanism. This enables using smart hosts which only offer AUTH LOGIN.
### Related issue(s)
- Auto close an issue like: closes#1633
## Prerequistes
Before we can consider review and merge, please make sure the following list is done and checked.
If an entry in not applicable, you can check it or remove it from the list.
- [n/a] In case of feature or enhancement: documentation updated accordingly
- [x] Unless it's docs or a minor change: add [changelog](https://mailu.io/master/contributors/guide.html#changelog) entry file.
Co-authored-by: Dimitri Huisman <diman@huisman.xyz>
Because basically every MTA out there uses opportunistic TLS _in
the best case_, it's actually counter productive to use such strict
settings.
The alternative to a handshake error is often an unencrypted submission,
which is basically the opposite of what strict ssl_protocols and
ssl_ciphers tries to achieve.
Even big and established providers like Amazon SES are incompatible with the current
settings.
This reverts commit 2ddf46ad2b.
this enables usage of e.g. docker swarm secrets instead of exposing the
passwords directly via environment variables
just use DB_PW_FILE and SECRET_KEY_FILE instead of DB_PW and SECRET_KEY
- stabilized CommaSeparatedList by sorting values
- CommaSeparatedList can now handle list and set input
- from_dict now handles mapped keys
- from_dict now handles null values
- class Domain: handle dkim-key None correctly
- class User: delete obsolete keys after converting
- class Alias: now uses Email._dict_input
Florent Daigniere
lub