2111: Preparations for 1.9 release r=mergify[bot] a=Diman0
## What type of PR?
Preparations for 1.9 release.
## What does this PR do?
All changes required for the 1.9 release. This PR does not trigger the 1.9 release yet. For that we only have to create a 1.9 branch after this PR has been merged.
Please double check all the documentation. Feel free to directly commit to this branch any spelling errors you see.
After this is merged, I only have to create the 1.9 branch and update the infra project to release 1.9.
### Related issue(s)
- closes#1930
## Prerequisites
Before we can consider review and merge, please make sure the following list is done and checked.
If an entry in not applicable, you can check it or remove it from the list.
- [x] In case of feature or enhancement: documentation updated accordingly
- [x] Unless it's docs or a minor change: add [changelog](https://mailu.io/master/contributors/workflow.html#changelog) entry file.
Co-authored-by: Dimitri Huisman <diman@huisman.xyz>
Co-authored-by: Florent Daigniere <nextgens@freenetproject.org>
Co-authored-by: Alexander Graf <ghostwheel42@users.noreply.github.com>
For full details see the [releases page](https://mailu.io/1.9/releases.html)
Warning, the helm-chart repo is not in sync yet with the new Mailu 1.9 release. If you use helm-chart (kubernetes), we advise to stick to version 1.8.
Upgrade should run fine as long as you generate a new compose or stack
configuration and upgrade your mailu.env.
There are some changes to the configuration overrides. Override files are now mounted read-only into the containers.
The Dovecot and Postfix overrides are moved in their own sub-directory.
If there are local override files, they will need to be moved from overrides/ to overrides/dovecot and overrides/postfix/.
See https://mailu.io/1.8/faq.html#how-can-i-override-settings for all the mappings.
If you use a reverse proxy in front of Mailu, it is vital to configure the newly introduced env variables REAL_IP_HEADER and REAL_IP_FROM.
These settings tell Mailu that the HTTP header with the remote client IP address from the reverse proxy can be trusted.
For more information see the [configuration reference](https://mailu.io/1.9/configuration.html#advanced-settings).
One major change for the docker compose file is that the antispam container needs a fixed hostname [#1837](https://github.com/Mailu/Mailu/issues/1837).
This is handled when you regenerate the docker-compose file. A fixed hostname is required to retain rspamd history.
This is also handled in the helm-chart repo.
Improvements have been made to protect again session-fixation attacks.
To be fully protected, it is required to change your SECRET_KEY in Mailu.env after upgrading.
A new SECRET_KEY is generated when you recreate your docker-compose.yml & mailu.env file via setup.mailu.io.
The SECRET_KEY is an uppercase alphanumeric string of length 16. You can manually create such a string via
After changing mailu.env, it is required to recreate all containers for the changes to be propagated.
Please note that the shipped image for PostgreSQL database is deprecated.
We advise to switch to an external PostgreSQL database server.
Please note that the shipped image for PostgreSQL database is fully deprecated now. To migrate to the official PostgreSQL image, you can follow our guide [here](https://mailu.io/master/database.html#mailu-postgresql)
1.9.0 - 2021-12-28
- Features: Make the rate limit apply to a subnet rather than a specific IP (/24 for v4 and /56 for v6) ([#116](https://github.com/Mailu/Mailu/issues/116))
- Features: Add instructions on how to create DNS records for email client auto-configuration (RFC6186 style) ([#224](https://github.com/Mailu/Mailu/issues/224))
- Features: Remove the Received header with PRIMARY_HOSTNAME [PUBLIC_IP] ([#466](https://github.com/Mailu/Mailu/issues/466))
- Features: Centralize the authentication of webmails behind the admin interface ([#783](https://github.com/Mailu/Mailu/issues/783))
- Features: Add sending quotas per user ([#1031](https://github.com/Mailu/Mailu/issues/1031))
- Features: Allow specific users to send emails from any address using the WILDCARD_SENDERS setting ([#1096](https://github.com/Mailu/Mailu/issues/1096))
- Features: Use semantic versioning for building releases.
- Add versioning (tagging) for branch x.y (1.8). E.g. 1.8.0, 1.8.1 etc.
- docker repo will contain x.y (latest) and x.y.z (pinned version) images.
- The X.Y.Z tag is incremented automatically. E.g. if 1.8.0 already exists, then the next merge on 1.8 will result in the new tag 1.8.1 being used.
- Make the version available in the image.
- For X.Y and X.Y.Z write the version (X.Y.Z) into /version on the image and add a label with version=X.Y.Z
- This means that the latest X.Y image shows the pinned version (X.Y.Z e.g. 1.8.1) it was based on. Via the tag X.Y.Z you can see the commit hash that triggered the built.
- For master write the commit hash into /version on the image and add a label with version={commit hash}
- Automatic releases. For x.y triggered builts (e.g. merge on 1.9) do a new github release for the pinned x.y.z (e.g. 1.9.2).
- Release shows a static message (see RELEASE_TEMPLATE.md) that explains how to reach the newsfragments folder and change the branch to the tag (x.y.z) mentioned in the release. Now you can get the changelog by reading all newsfragment files in this folder. ([#1182](https://github.com/Mailu/Mailu/issues/1182))
- Features: Add a credential cache to speedup authentication requests. ([#1194](https://github.com/Mailu/Mailu/issues/1194))
- Features: Introduces postfix logging via syslog with these features:
- stdout logging still enabled
- internal test request log messages (healthcheck) are filtered out by rsyslog
- optional logging to file via POSTFIX_LOG_FILE env variable
To use logging to file configure in mailu.env
- ``POSTFIX_LOG_FILE``: The file to log the mail log to ([#1441](https://github.com/Mailu/Mailu/issues/1441))
- Features: Make smtp_tls_policy_maps easily configurable ([#1558](https://github.com/Mailu/Mailu/issues/1558))
- Features: Implement a language selector for the admin interface. ([#1567](https://github.com/Mailu/Mailu/issues/1567))
- Features: Add cli commands config-import and config-export ([#1604](https://github.com/Mailu/Mailu/issues/1604))
- Features: Implement SECRET_KEY_FILE and DB_PW_FILE variables for usage with Docker secrets. ([#1607](https://github.com/Mailu/Mailu/issues/1607))
- Features: Add possibility to enforce inbound STARTTLS via INBOUND_TLS_LEVEL=true ([#1610](https://github.com/Mailu/Mailu/issues/1610))
- Features: Refactor the rate limiter to ensure that it performs as intented. ([#1612](https://github.com/Mailu/Mailu/issues/1612))
- Features: Enable OCSP stapling for the http server within nginx. ([#1618](https://github.com/Mailu/Mailu/issues/1618))
- Features: Enable support of all hash types passlib supports. ([#1662](https://github.com/Mailu/Mailu/issues/1662))
- Features: Support configuring lz4 and zstd compression for dovecot. ([#1694](https://github.com/Mailu/Mailu/issues/1694))
- Features: Switch to bcrypt_sha256, replace PASSWORD_SCHEME with CREDENTIAL_ROUNDS and dynamically update existing hashes on first login ([#1753](https://github.com/Mailu/Mailu/issues/1753))
- Features: Implement AdminLTE 3 for the admin interface. ([#1764](https://github.com/Mailu/Mailu/issues/1764))
- Features: Implement MTA-STS and DANE validation. Introduce DEFER_ON_TLS_ERROR (default: True) to harden or loosen the policy enforcement. ([#1798](https://github.com/Mailu/Mailu/issues/1798))
- Features: Remove cyrus-sasl-plain as it's not packaged by alpine anymore. SASL-login is still available and used when relaying. ([#1851](https://github.com/Mailu/Mailu/issues/1851))
- Features: Hebrew translation has been completed. ([#1873](https://github.com/Mailu/Mailu/issues/1873))
- Features: Log authentication attempts on the admin portal ([#1926](https://github.com/Mailu/Mailu/issues/1926))
- Features: AdminLTE3 design optimizations, asset compression and caching
- fixed copy of qemu-arm-static for alpine
- added 'set -eu' safeguard
- silenced npm update notification
- added color to webpack call
- changed Admin-LTE default blue
- AdminLTE 3 style tweaks
- localized datatables
- moved external javascript code to vendor.js
- added mailu logo
- moved all inline javascript to app.js
- added iframe display of rspamd page
- updated language-selector to display full language names and use post
- added fieldset to group and en/disable input fields
- added clipboard copy buttons
- cleaned external javascript imports
- pre-split first hostname for further use
- cache dns_* properties of domain object (immutable during runtime)
- fixed and splitted dns_dkim property of domain object (space missing)
- added autoconfig and tlsa properties to domain object
- suppressed extra vertical spacing in jinja2 templates
- improved accessibility for screen reader
- deleted unused/broken /user/forward route
- updated gunicorn to 20.1.0 to get rid of buffering error at startup
- switched webpack to production mode
- added css and javascript minimization
- added pre-compression of assets (gzip)
- removed obsolete dependencies
- switched from node-sass to dart-sass
- changed startup cleaning message from error to info
- move client config to "my account" section when logged in ([#1966](https://github.com/Mailu/Mailu/issues/1966))
- Features: Remove Mailu PostgreSQL. It is fully deprecated. No images will be built anymore and it cannot be selected in the setup utility.
The roundcube database flavour (e.g. PostgreSQL or SQLite) can now be selected indepently of the Mailu (Admin) database flavour.
- Bugfixes: RELAYNETS should be a comma separated list of networks ([#360](https://github.com/Mailu/Mailu/issues/360))
- Bugfixes: Fix rate-limiting on /webdav/ ([#1194](https://github.com/Mailu/Mailu/issues/1194))
- Bugfixes: Fixed fetchmail losing track of fetched emails upon container recreation.
The relevant fetchmail files are now retained in the /data folder (in the fetchmail image).
See the docker-compose.yml file for the relevant volume mapping.
If you already had your own mapping, you must double check the volume mapping and take action. ([#1223](https://github.com/Mailu/Mailu/issues/1223))
- Bugfixes: Ensure that the podop socket is always owned by the postfix user (wasn't the case when build using non-standard base images... typically for arm64) ([#1294](https://github.com/Mailu/Mailu/issues/1294))
- Bugfixes: Fix "extract_host_port" function to support containers with custom / dynamic ports ([#1669](https://github.com/Mailu/Mailu/issues/1669))
- Bugfixes: Fix CVE-2021-23240, CVE-2021-3156 and CVE-2021-23239 for postgresql
by force-upgrading sudo. ([#1760](https://github.com/Mailu/Mailu/issues/1760))
- Bugfixes: Fix roundcube environment configuration for databases ([#1831](https://github.com/Mailu/Mailu/issues/1831))
- Bugfixes: Alpine has removed support for btree and hash in postfix... please use lmdb instead ([#1917](https://github.com/Mailu/Mailu/issues/1917))
- Bugfixes: Webmail and Radicale (webdav) were not useable with domains with special characters such as umlauts.
Webmail and radicale now use punycode for logging in.
Punycode was not used in the HTTP headers. This resulted in illegal non-ASCII HTTP headers. ([#1952](https://github.com/Mailu/Mailu/issues/1952))
- Bugfixes: Ensure that we do not trust the source-ip address set in headers if REAL_IP_HEADER isn't set. If you are using Mailu behind a reverse proxy, please ensure that you do read the documentation. ([#1960](https://github.com/Mailu/Mailu/issues/1960))
- Bugfixes: Reverse proxy documentation has been updated to reflect new security hardening from PR#1959.
If you do not set the configuration parameters in Mailu what reverse proxy header to trust,
then Mailu will not have access to the real ip address of the connecting client.
This means that rate limiting will not properly work. You can also not use fail2ban.
It is very important to configure this when using a reverse proxy. ([#1962](https://github.com/Mailu/Mailu/issues/1962))
- Bugfixes: Fixed roundcube sso login not working. ([#1990](https://github.com/Mailu/Mailu/issues/1990))
- Bugfixes: The DB_PORT and ROUNDCUBE_DB_PORT environment variables were not actually used. They are removed from the documentation. For using different ports you can already use the notation host:port . ([#2073](https://github.com/Mailu/Mailu/issues/2073))
- Bugfixes: Ensure that webmail tokens expire in sync with sessions ([#2080](https://github.com/Mailu/Mailu/issues/2080))
- Bugfixes: Introduce SESSION_TIMEOUT (1h) and PERMANENT_SESSION_LIFETIME (30d) ([#2094](https://github.com/Mailu/Mailu/issues/2094))
- Improved Documentation: Document hardware requirements when using clamav.
Clamav requires **at least** 2GB of memory.
This 2Gb does not entail any other software running on the box.
So in total you require at least 3GB of memory and 1GB swap when antivirus is enabled. ([#470](https://github.com/Mailu/Mailu/issues/470))
- Improved Documentation: Added documentation for how to switch the database back-end used by Mailu.
Added documentation for migrating from the deprecated Mailu PostgreSQL image to a different PostgreSQL database. ([#1037](https://github.com/Mailu/Mailu/issues/1037))
- Improved Documentation: Add documentation for Traefik 2 in Reverse Proxy ([#1503](https://github.com/Mailu/Mailu/issues/1503))
This is a new automatic release of Mailu. The new version can be seen in the tag name.
The main version X.Y (e.g. 1.8) will always reflect the latest version of the branch. To update your Mailu installation simply pull the latest images `docker-compose pull && docker-compose up -d`.
The pinned version X.Y.Z (e.g. 1.8.1) is not updated. It is pinned to the commit that was used for creating this release. You can use a pinned version to make sure your Mailu installation is not suddenly updated when recreating containers. The pinned version allows the user to manually update. It also allows to go back to a previous pinned version.
The main version X.Y (e.g. 1.9) will always reflect the latest version of the branch. To update your Mailu installation simply pull the latest images `docker-compose pull && docker-compose up -d`.
The pinned version X.Y.Z (e.g. 1.9.1) is not updated. It is pinned to the commit that was used for creating this release. You can use a pinned version to make sure your Mailu installation is not suddenly updated when recreating containers. The pinned version allows the user to manually update. It also allows to go back to a previous pinned version.
To check what was changed:
- Go to https://github.com/Mailu/Mailu/tree/master/towncrier/newsfragments
- Change the branch to the tag of this release.
- Read the news fragment files to check what was changed.
The release notes of the original release can be accessed via menu item 'Release notes' on [mailu.io](https://mailu.io/).
The release notes of the original release can be accessed via menu item 'Release notes' on [mailu.io](https://mailu.io/).
@ -570,24 +570,32 @@ This will generate the DKIM and DMARC entries for you.
*Issue reference:*`102`_.
.._Fail2Ban:
Do you support Fail2Ban?
````````````````````````
Fail2Ban is not included in Mailu. Fail2Ban needs to modify the host's IP tables in order to
ban the addresses. We consider such a program should be run on the host system and not
inside a container. The ``front`` container does use authentication rate limiting to slow
down brute force attacks.
down brute force attacks. The same applies to login attempts via the single sign on page.
We *do* provide a possibility to export the logs from the ``front`` service to the host.
We *do* provide a possibility to export the logs from the ``front`` service and ``Admin`` service to the host.
The ``front`` container logs failed logon attempts on SMTP, IMAP and POP3.
The ``Admin``container logs failed logon attempt on the single sign on page.
For this you need to set ``LOG_DRIVER=journald`` or ``syslog``, depending on the log
manager of the host. You will need to setup the proper Regex in the Fail2Ban configuration.
Below an example how to do so. Be aware that webmail authentication appears to come from the
Docker network, so don't ban those addresses!
Below an example how to do so.
If you use a reverse proxy in front of Mailu, it is vital to set the environment variables REAL_IP_HEADER and REAL_IP_FROM.
Without these environment variables, Mailu will not trust the remote client IP passed on by the reverse proxy and as a result your reverse proxy will be banned.
See the :ref:`[configuration reference <reverse_proxy_headers>` for more information.
Assuming you have a working Fail2Ban installation on the host running your Docker containers,
follow these steps:
1. In the mailu docker-compose set the logging driver of the front container to journald
1. In the mailu docker-compose set the logging driver of the front container to journald; and set the tag to mailu-front
..code-block:: bash
@ -621,7 +629,41 @@ follow these steps:
The above will block flagged IPs for a week, you can of course change it to you needs.
4. Add the /etc/fail2ban/action.d/docker-action.conf
4. In the mailu docker-compose set the logging driver of the Admin container to journald; and set the tag to mailu-admin
..code-block:: bash
logging:
driver: journald
options:
tag: mailu-admin
5. Add the /etc/fail2ban/filter.d/bad-auth-sso.conf
..code-block:: bash
# Fail2Ban configuration file
[Definition]
failregex = .* Login failed for .+ from <HOST>.
ignoreregex =
journalmatch = CONTAINER_TAG=mailu-admin
6. Add the /etc/fail2ban/jail.d/bad-auth-sso.conf
..code-block:: bash
[bad-auth-sso]
enabled = true
backend = systemd
filter = bad-auth-sso
bantime = 604800
findtime = 300
maxretry = 10
action = docker-action
The above will block flagged IPs for a week, you can of course change it to you needs.
7. Add the /etc/fail2ban/action.d/docker-action.conf
..code-block:: bash
@ -643,7 +685,7 @@ The above will block flagged IPs for a week, you can of course change it to you
Using DOCKER-USER chain ensures that the blocked IPs are processed in the correct order with Docker. See more in: https://docs.docker.com/network/iptables/
5. Configure and restart the Fail2Ban service
8. Configure and restart the Fail2Ban service
Make sure Fail2Ban is started after the Docker service by adding a partial override which appends this to the existing configuration.
Mailu 1.9 is available now. The helm-chart repo is not in sync yet with the new Mailu 1.9 release. If you use helm-chart (kubernetes), we advise to stick to version 1.8 for now.
See the section `Upgrading` for important information in regard to upgrading to Mailu 1.9.
Highlights
````````````````````````````````
Quite a lot of new features have been implemented. Of these new features we'd like to highlight these:
Security
^^^^^^^^
A far amount of work went in this release; In no particular order:
- outbound SMTP connections from Mailu are now enjoying some protection against active attackers thanks to DANE and MTA-STS support. Specific policies can be configured for specific destinations thanks to ``tls_policy_maps`` and configuring your system to publish a policy has been documented in the FAQ.
- outbound emails can now be rate-limited (to mitigate SPAM in case an account is taken over)
- long term storage of passwords has been rethought to enable stronger protection against offline attackers (switch to iterated and salted SHA+bcrypt) while enabling much better performance (credential cache). Please encourage your users to use tokens where appropriate and keep in mind that existing hashes will be converted on first use to the new format.
- session handling has been reworked from the grounds up: they have been switched from client side (cookies) to server-side, unified (SSO, expiry, lifetime) accross all web-facing applications and some mitigations against session fixation have been implemented.
- rate limiting has seen many improvements: It is now deployed on **all** entry points (SMTP/IMAP/POP3/WEB/WEBMAIL) and configured to defeat both password bruteforces (thanks to a limit against total number of failed attempts against an account over a period) and password spraying (thanks to a limit for each client on the total number of non-existing accounts that can be queried). Exemption mechanisms have been put in place (device tokens, dynamic IP whitelists) to ensure that genuine clients and users won't be affected by default and the default configuration thought to fit most usecases.
- if you use letsencrypt, Mailu is now configured to offer both RSA and ECC certificates to clients; It will OSCP stapple its replies where appropriate
Updated Admin interface
^^^^^^^^^^^^^^^^^^^^^^^
The Web Administration interface makes use of AdminLTE. The AdminLTE2 technology has been upgraded to AdminLTE3. This cost a lot of effort due to the changes between AdminLTE2 and AdminLTE3.
As a result the webpage looks more modern. All tables now have a filter and columns that can be sorted. If you have many users or domains, this will be a very welcome new feature!
A language selector has been added. On the login page and in the Web Admin Interface, the language selector can be accessed in the top right.
Import/Export command on steroids
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
The Mailu command line has been enhanced with the new config-export and config-import command.
**Everything** that can be configured in the Mailu Web Administration Interface can now be exported and imported via yaml files.
So via YAML files, you can now bulk configure a complete new installation, without the need to access the Mailu Web Administration Interface.
It is also possible to create new users or import new users (with password hashes) using the config-import.
With this new command it is very easy to switch to a different database management system for the Mailu database. Simply dump your configuration to yaml file.
After setting up your new Mailu system with the different DBMS, you can import the yaml file with all Mailu configuration.
For more information, see the :ref:`Mailu command line <config-export>` page.
New SSO login page
^^^^^^^^^^^^^^^^^^
A new single sign on login page is introduced which handles logins for the Mailu Web Administration Interface and webmail. It has enabled a drastic attack-surface reduction and will enable us to add support for two factor authentication in the future.
All failed login attempts are now logged to the Admin service, significantly simplifying the deployment of solutions such as Fail2ban.
See the :ref:`updated Fail2Ban documentation <Fail2Ban>` for more information.
Semantic versioning
^^^^^^^^^^^^^^^^^^^
From Mailu 1.9, we will use semantic versioning. First we only had x.y (e.g. 1.9) releases. For every update to an existing version, we will create an additional x.y.z (e.g. 1.9.1) release.
- The X.Y (1.9) tag will always feature the latest version.
- The X.Y.Z (1.9.1) tag is a pinned version. This release is not updated. You can use this to update in a controlled manner. At a convenient time, you can choose to switch to a newer version (e.g 1.9.2). The X.Y.Z tag is incremented automatically when an update is pushed for the X.Y release.
The images now also contain the release it was built for.
- Every docker image will have a docker label with the version.
- Every docker image will have the file /version with the same version information.
- Master images will contain the commit hash that initiated the built of the image.
- X.Y and X.Y.Z images will have the X.Y.Z version that triggered the built.
On the github project we will automatically create releases for each X.Y.Z release. Via this release you can check what commit hash the tag is assigned to.
With this improvement in our CI/CD workflow, it is possible to be notified when an update is released via github releases. It is also possible to use pinned versions to update in a controlled manner.
New Functionality & Improvements
````````````````````````````````
For a list of all the changes (including bug fixes) refer to `CHANGELOG.md` in the root folder of the Mailu github project.
A short summary of the new features:
- Roundcube and Rainloop have been updated.
- All dependencies have been updated to the latest security update
- AdminLTE (used by Admin service) is updated to AdminLTE3.
- Much improved rate limiting.
- Rate limiting small subnets instead of single IP addresses.
- Rate limiting for accounts that do not exist.
- Rate limiting for existing accounts (failed logon attempts).
- Device-tokens are introduced to ensure genuine users are not locked out
- Domain details page is enhanced with DNS client auto-configuration (RFC6186) entries.
- Centralize the authentication of webmails behind the admin interface.
- The new single sign on page opens up the possiblity to introduce 2 factor authentication in the future.
- Add sending quotas per user (configured in mailu.env). This determines how many emails each user can send every day.
- Allow specific users to send emails from any address using the WILDCARD_SENDERS setting (mailu.env.).
- Use semantic versioning for building releases.
- Internal improviments to improve performance of authentication requests.
- Introduded a language selector for the Admin interface.
- Add cli commands config-import and config-export for importing/exporting Mailu config via YAML.
- Enable support of all hash types passlib supports.
- Switch to bcrypt_sha256 (stronger hashing of passwords in Mailu database)/
- Introduce MTA-STS and DANE validation.
- Added Hebrew translation.
- Log authentication attempts on the admin portal. Fail2ban can now be used to monitor login attempts on Admin/Webmail.
- Remove Mailu PostgreSQL.
- Admin/Webmail sessions expire now. This can be tweakers via mailu.env.
Upgrading
`````````
Upgrade should run fine as long as you generate a new compose or stack configuration and upgrade your mailu.env.
If you use a reverse proxy in front of Mailu, it is vital to configure the newly introduced environment variables `REAL_IP_HEADER`` and `REAL_IP_FROM`.
These settings tell Mailu that the HTTP header with the remote client IP address from the reverse proxy can be trusted.
For more information see the :ref:`configuration reference <reverse_proxy_headers>`.
If you use Fail2Ban, you configure Fail2Ban to monitor failed logon attempts for the web-facing frontend (Admin/Webmail). See the :ref:`updated Fail2Ban documentation <Fail2Ban>` for more information.
Please note that the shipped image for the PostgreSQL database is fully deprecated now.
To migrate to the official PostgreSQL image, you can follow our :ref:`migration guide <migrate_mailu_postgresql>`.
Ensure that we do not trust the source-ip address set in headers if REAL_IP_HEADER isn't set. If you are using Mailu behind a reverse proxy, please ensure that you do read the documentation.
The DB_PORT and ROUNDCUBE_DB_PORT environment variables were not actually used. They are removed from the documentation. For using different ports you can already use the notation host:port .
bors[bot]
3 years ago
committed by