Merge remote-tracking branch 'upstream/master' into import-export

.mergify.yml

@@ -35,18 +35,18 @@ pull_request_rules:
       comment:
         message: bors r+
 
-  - name: Backport to 1.7 branch
+  - name: Backport to 1.8  branch
     conditions:
       - base=master
       - label=type/backport
     actions:
       backport:
         branches:
-          - '1.7'
+          - '1.8'
 
   - name: remove outdated reviews
     conditions:
-      - base~=^(master|1.7)$
+      - base~=^(master|1.8)$
     actions:
       dismiss_reviews:
         approved: True

.travis.yml

@@ -5,9 +5,10 @@ branches:
     - '1.5'
     - '1.6'
     - '1.7'
+    - '1.8'
     - master
     # version tags, e.g. 1.7.1
-    - /^1\.[567]\.\d+$/
+    - /^1\.[5678]\.\d+$/
     # pre-releases, e.g. 1.8-pre1
     - /^1\.8-pre\d+$/
     # test branches, e.g. test-debian

AUTHORS.md

@@ -25,3 +25,6 @@ Other contributors:
  - [Tim Mohlmann](https://github.com/muhlemmer) - [Contributions](https://github.com/Mailu/Mailu/commits?author=muhlemmer)
  - [Ionut Filip](https://github.com/ionutfilip) - [Contributions](https://github.com/Mailu/Mailu/commits?author=ionutfilip)
  - [Ichikawa Yuriko](https://github.com/IchikawaYukko) - [Contributions](https://github.com/Mailu/Mailu/commits?author=IchikawaYukko) Japanese translation
+ - [Dimitri Huisman](https://github.com/Diman0) - [Contributions](https://github.com/Mailu/Mailu/commits?author=Diman0)
+ - [lub](https://github.com/lub) - [Contributions](https://github.com/Mailu/Mailu/commits?author=lub)
+ - [Dario Ernst](https://github.com/Nebukadneza) - [Contributions](https://github.com/Mailu/Mailu/commits?author=Nebukadneza)

CHANGELOG.md

@@ -1,11 +1,53 @@
 Changelog
 =========
 
-Notable changes to this project are documented in the current file. For more
+Upgrade should run fine as long as you generate a new compose or stack
-details about individual changes, see the Git log. You should read this before
+configuration and upgrade your mailu.env.
-upgrading Mailu as some changes will include useful notes.
+
+Please note that the current 1.8 is what we call a "soft release": It’s there for everyone to see and use, but to limit possible user-impact of this very big release, it’s not yet the default in the setup-utility for new users. When upgrading, please treat it with some care, and be sure to always have backups!
+
+There are some changes to the configuration overrides. Override files are now mounted read-only into the containers.
+The Dovecot and Postfix overrides are moved in their own sub-directory.
+If there are local override files, they will need to be moved from overrides/ to overrides/dovecot and overrides/postfix/.
+See https://mailu.io/1.8/faq.html#how-can-i-override-settings for all the mappings.
+
+Please not that the shipped image for PostgreSQL database is deprecated.
+We advise to switch to an external database server.
 
 <!-- TOWNCRIER -->
+v1.8.0 - 2020-09-28
+--------------------
+
+- Features: Add support for backward-forwarding using SRS ([#328](https://github.com/Mailu/Mailu/issues/328))
+- Features: Add options to support different architectures builds ([#985](https://github.com/Mailu/Mailu/issues/985))
+- Features: Add support for Traefik v2 certificate dumping ([#1011](https://github.com/Mailu/Mailu/issues/1011))
+- Features: Resolve hosts to IPs if only HOST_* is set. If *_ADDRESS is set, leave it unresolved. ([#1113](https://github.com/Mailu/Mailu/issues/1113))
+- Features: - Use nginx as http endpoint on kubernetes to simplify ingress ([#1158](https://github.com/Mailu/Mailu/issues/1158))
+- Features: Advertise correct mail capabilities through the front-container, this also enables support for PIPELINING in mail-protocols and IMAP IDLE which is a (potential) performance gain. ([#1160](https://github.com/Mailu/Mailu/issues/1160))
+- Features: Change default password scheme to PBKDF2 ([#1194](https://github.com/Mailu/Mailu/issues/1194))
+- Features: Enable access log of admin service only for log levels of INFO and finer ([#1197](https://github.com/Mailu/Mailu/issues/1197))
+- Features: japanese loca is now available ([#1207](https://github.com/Mailu/Mailu/issues/1207))
+- Features: Allow to reject virus mails by setting ANTIVITUS_ACTION=reject ([#1259](https://github.com/Mailu/Mailu/issues/1259))
+- Features: Update roundcube to 1.4.0 and enable the new elastic skin ([#1267](https://github.com/Mailu/Mailu/issues/1267))
+- Features: The roundcube container does support mysql now (no setup integration yet) ([#1268](https://github.com/Mailu/Mailu/issues/1268))
+- Features: Added CardDAV-Plugin for webmail roundcube. ([#1298](https://github.com/Mailu/Mailu/issues/1298))
+- Features: Allow users to use server-sided full-text-search again by adding the dovecot fts-xapian plugin ([#1320](https://github.com/Mailu/Mailu/issues/1320))
+- Features: Relay a domain to a nonstandard SMTP port by adding ":<port_num>" to the remote hostname or IP address. ([#1357](https://github.com/Mailu/Mailu/issues/1357))
+- Features: Introduce option to disable dovecot full-text-search by an enviroment variable. ([#1538](https://github.com/Mailu/Mailu/issues/1538))
+- Features: Add support for AUTH LOGIN authentication mechanism for relaying email via smart hosts. ([#1635](https://github.com/Mailu/Mailu/issues/1635))
+- Bugfixes: Use pip package for radicale to fix failing builds caused by [alpine]upstream package rebuild against different python version ([#1255](https://github.com/Mailu/Mailu/issues/1255))
+- Bugfixes: Disable Health checks on swarm mode ([#1289](https://github.com/Mailu/Mailu/issues/1289))
+- Bugfixes: Show SPF records in accordance with RFC 7208: Previously we instructed admins to create SPF and TXT records, where only TXT records are correct now. !! Attention !! You need to manually remove the SPF-typed records and keep only TXT in your DNS. ([#1394](https://github.com/Mailu/Mailu/issues/1394))
+- Bugfixes: Cover relearning messages when moving bewteen Ham and Spam status ([#1438](https://github.com/Mailu/Mailu/issues/1438))
+- Bugfixes: Defining POSTMASTER through setup tool apply also to DMARC_RUA and DMARC_RUF settings ([#1463](https://github.com/Mailu/Mailu/issues/1463))
+- Bugfixes: Check postfix mailqueue permissions before start-up ([#1486](https://github.com/Mailu/Mailu/issues/1486))
+- Bugfixes: Fixes certbot renewal ([#1564](https://github.com/Mailu/Mailu/issues/1564))
+- Improved Documentation: Added documentation that describes how spam filtering works in Mailu. ([#1167](https://github.com/Mailu/Mailu/issues/1167))
+- Improved Documentation: Add documentation for the web administration interface. ([#1590](https://github.com/Mailu/Mailu/issues/1590))
+- Deprecations and Removals: Dovecot: Delete obsolete data volume ([#1221](https://github.com/Mailu/Mailu/issues/1221))
+- Misc:  ([#508](https://github.com/Mailu/Mailu/issues/508), [#1098](https://github.com/Mailu/Mailu/issues/1098), [#1214](https://github.com/Mailu/Mailu/issues/1214), [#1308](https://github.com/Mailu/Mailu/issues/1308), [#1444](https://github.com/Mailu/Mailu/issues/1444), [#1512](https://github.com/Mailu/Mailu/issues/1512))
+
+
 v1.7.0 - 2019-08-22
 -------------------
 

core/admin/mailu/configuration.py

@@ -24,7 +24,7 @@ DEFAULT_CONFIG = {
     'SQLALCHEMY_TRACK_MODIFICATIONS': False,
     # Statistics management
     'INSTANCE_ID_PATH': '/data/instance',
-    'STATS_ENDPOINT': '0.{}.stats.mailu.io',
+    'STATS_ENDPOINT': '18.{}.stats.mailu.io',
     # Common configuration variables
     'SECRET_KEY': 'changeMe',
     'DOMAIN': 'mailu.io',

docs/conf.py

@@ -41,6 +41,7 @@ html_context = {
         ('1.5', '/1.5/'),
         ('1.6', '/1.6/'),
         ('1.7', '/1.7/'),
+        ('1.8', '/1.8/'),
         ('master', '/master/')
     ],
     'conf_py_path': '/docs/'

docs/releases.rst

@@ -1,15 +1,54 @@
 Release notes
 =============
 
-Mailu 1.8 - tbd
+Mailu 1.8 - 2020-10-02
 ----------------------
 
+Release 1.8 has come a long way again. Due to corona the project slowed down to a crawl. Fortunately new contributors have joined the team what enabled us to still release Mailu 1.8 this year.
+
+Please note that the current 1.8 is what we call a "soft release": It’s there for everyone to see and use, but to limit possible user-impact of this very big release, it’s not yet the default in the setup-utility for new users. When upgrading, please treat it with some care, and be sure to always have backups!
+
+For a list of all changes refer to `CHANGELOG.md` in the root folder of the Mailu github project. Please read the 'Override location changes' section. It contains important information for the people who use the overrides folder.
+
+New Functionality & Improvements
+````````````````````````````````
+
+Here’s a short summary of new features:
+
+- Full-text-search is back after having been disabled for a while due to nasty bugs. It can still be disabled via the mailu.env file.
+- Tons of documentation improvements, especially geared towards new users.
+- (Experimental) support for different architectures, such as ARM.
+- Improvements around webmails, such as CardDAV, GPG and a new skin for an updated roundcube, and support for MySQL for it. Updated Rainloop, too.
+- Improvements around relaying, such as AUTH LOGIN and non-standard port support.
+- Update to alpine:3.12 as baseimage for most containers.
+- Setup warns users about compose-IPv6 deployments which have caused open relays in the past.
+- Improved handling of upper-vs-lowercase aliases and user-addresses.
+- Improved rate-limiting system.
+- Support for SRS.
+- Japanese localisation is now available.
+
+Upgrading
+`````````
+
+Upgrade should run fine as long as you generate a new compose or stack
+configuration and upgrade your mailu.env.
+
+Please not that the shipped image for PostgreSQL database is deprecated.
+The shipped image for PostgreSQL is not maintained anymore from release 1.8.
+We recommend switching to an external database as soon as possible.
+
 Override location changes
-`````````````````````````
+^^^^^^^^^^^^^^^^^^^^^^^^^
 
 If you have regenerated the Docker compose and environment files, there are some changes to the configuration overrides.
 Override files are now mounted read-only into the containers. The Dovecot and Postfix overrides are moved in their own sub-directory. If there are local override files, they will need to be moved from ``overrides/`` to ``overrides/dovecot`` and ``overrides/postfix/``.
 
+Update your DNS SPF Records
+^^^^^^^^^^^^^^^^^^^^^^^^^^^
+
+It has become known that the SPF DNS records generated by the admin interface are not completely standard compliant anymore. Please check the DNS records for your domains and compare them to what the new admin-interface instructs you to use. In most cases, this should be a simple copy-paste operation for you ….
+
+
 Mailu 1.7 - 2019-08-22
 ----------------------
 

docs/setup.rst

@@ -38,7 +38,7 @@ Pick a Mailu version
 
 Mailu is shipped in multiple versions.
 
-- ``1.7`` features the most recent stable version for Mailu. This is the
+- ``1.8`` features the most recent stable version for Mailu. This is the
   recommended build for new setups, old setups should migrate when possible.
 
 - ``1.0``, ``1.1``, and other version branches feature old versions of Mailu

setup/flavors/stack/docker-compose.yml

@@ -29,7 +29,7 @@ services:
       - "{{ root }}/certs:/certs"
       - "{{ root }}/overrides/nginx:/overrides:ro"
     deploy:
-      replicas: {{ front_replicas }}
+      replicas: 1
 
   admin:
     image: ${DOCKER_ORG:-mailu}/${DOCKER_PREFIX:-}admin:${MAILU_VERSION:-{{ version }}}
@@ -42,7 +42,7 @@ services:
       - "{{ root }}/data:/data"
       - "{{ root }}/dkim:/dkim"
     deploy:
-      replicas: {{ admin_replicas }}
+      replicas: 1
     healthcheck:
       disable: true
 
@@ -53,7 +53,7 @@ services:
       - "{{ root }}/mail:/mail"
       - "{{ root }}/overrides/dovecot:/overrides:ro"
     deploy:
-      replicas: {{ imap_replicas }}
+      replicas: 1
     healthcheck:
       disable: true
 
@@ -64,7 +64,7 @@ services:
       - "{{ root }}/mailqueue:/queue"
       - "{{ root }}/overrides/postfix:/overrides:ro"
     deploy:
-      replicas: {{ smtp_replicas }}
+      replicas: 1
     healthcheck:
       disable: true
 

setup/server.py

@@ -53,7 +53,10 @@ def build_app(path):
 
     @app.context_processor
     def app_context():
-        return dict(versions=os.getenv("VERSIONS","master").split(','))
+        return dict(
+            versions=os.getenv("VERSIONS","master").split(','), 
+            stable_version = os.getenv("stable_version", "master")
+        )
 
     prefix_bp = flask.Blueprint(version, __name__)
     prefix_bp.jinja_loader = jinja2.ChoiceLoader([

setup/static/render.js

@@ -51,7 +51,12 @@ $(document).ready(function() {
 		} else if (this.value == 'postgresql') {
 			$("#postgres_db").show();
 			$("#external_db").hide();
-			$("#external_db").prop('checked', false);
+			$("#external_psql").prop('checked', true);
+			$("#external_db").show();
+			$("#db_user").prop('required',true);
+			$("#db_pw").prop('required',true);
+			$("#db_url").prop('required',true);
+			$("#db_name").prop('required',true);
 		} else if (this.value == 'mysql') {
 			$("#postgres_db").hide();
 			$("#external_db").show();

setup/templates/base.html

@@ -8,13 +8,21 @@
   <h1>Mailu configuration</h1>
   <p>
     Version
-    <select onchange="window.location.href=this.value;" class="btn btn-primary dropdown-toggle">
+    <select id=version_select onchange="window.location.href=this.value;" class="btn btn-primary dropdown-toggle">
       {% for module in versions %}
-      <option value="/{{ module }}" {% if module == version %}selected{% endif %}>{{ module }}</option>
+      <option value="/{{ module }}" {% if module == version %}selected {% endif %}>{{ module }}</option>
       {% endfor %}
     </select>
   </p>
 
+  {% if version != stable_version %}
+  {% call macros.panel("danger", "You have not selected the stable version") %}
+You have not selected the stable version. The stable version is {{ stable_version }}.
+The selected version can be used for testing and reporting bugs.
+For production scenarios we recommend to use the stable version.
+  {% endcall %}
+  {% endif %}
+
   {% block page %}{% endblock %}
 
 </div>

setup/templates/steps/config.html

@@ -1,9 +1,11 @@
 {% if flavor == "stack" %}
 {% call macros.panel("danger", "Docker stack / swarm is experimental") %}
 Setup is capable of generating a somewhat decent docker-compose.yml,
-for the docker stack flavor. However its usage is for advanced users an is experimental.
+for the docker stack flavor. However its usage is for advanced users only and is experimental.
-Expect many challenges is shared mail storage and fail-over scenarios! Some user experiences
+Expect many challenges such as shared mail storage and fail-over scenarios! Some user experiences
 have been <a href="https://github.com/Mailu/Mailu/blob/master/docs/swarm/master/README.md">shared on GitHub.</a>
+For this reason also think very hard about using a replica count higher than 1. This cannot be used with the default config.
+Manual post-configuration is required for using a replica count higher than 1.
 {% endcall %}
 {% endif %}
 

setup/templates/steps/database.html

@@ -11,13 +11,16 @@
 	<p></p>
 	<div id="postgres_db" style="display: none">
 		<div class="form-check">
-			<input class="form-check-input" type="radio" name="postgresql" id="internal_psql" value="internal" checked>
+			<input class="form-check-input" type="radio" name="postgresql" id="internal_psql" value="internal">
 			<label class="form-check-label" for="internal_psql">
-				Use the Postgresql from Mailu
+				Use the PostgreSQL image from Mailu. Warning, this image is deprecated.
+                                It will be removed from Mailu 1.9. Existing users can still upgrade to Mailu 1.8 using this image.
+                                With the release of Mailu 1.9,
+                                we will offer a mechanism for migrating your data to another database management system.
 			</label>
 		</div>
 		<div class="form-check">
-			<input class="form-check-input" type="radio" name="postgresql" value="external" id="external_psql" >
+			<input class="form-check-input" type="radio" name="postgresql" value="external" id="external_psql">
 			<label class="form-check-label" for="external_psql">
 				I want to connect to an external database
 			</label>

setup/templates/steps/stack/04_replicas.html

@@ -1,28 +0,0 @@
-{% call macros.panel("info", "Step 5 - Number of replicas for containers") %}
-<p>Select number of replicas for containers</p>
-
-<div class="form-group">
-  <input class="form-control" type="number" name="front_replicas" min="1" required value="1"
-  		style="width: 6%; display: inline;">
-  <label>Front</label>
-</div>
-
-<div class="form-group">
-  <input class="form-control" type="number" name="admin_replicas" min="1" required value="1"
-  		style="width: 6%; display: inline;">
-  <label>Admin</label>
-</div>
-
-<div class="form-group">
-  <input class="form-control" type="number" name="imap_replicas" min="1" required value="1"
-  		style="width: 6%; display: inline;">
-  <label>IMAP</label>
-</div>
-
-<div class="form-group">
-  <input class="form-control" type="number" name=smtp_replicas min="1" required value="1"
-  		style="width: 6%; display: inline;">
-  <label>SMTP</label>
-</div>
-
-{% endcall %}

towncrier/newsfragments/1011.feature

@@ -1 +0,0 @@
-Add support for Traefik v2 certificate dumping

towncrier/newsfragments/1098.misc

@@ -1 +0,0 @@
-Ignore newlines and comment-lines in postfix overrides - this means you can now make your override configfiles much more readable.

towncrier/newsfragments/1113.feature

@@ -1 +0,0 @@
-Resolve hosts to IPs if only HOST_* is set. If *_ADDRESS is set, leave it unresolved.

towncrier/newsfragments/1158.feature

@@ -1 +0,0 @@
-- Use nginx as http endpoint on kubernetes to simplify ingress

towncrier/newsfragments/1160.feature

@@ -1 +0,0 @@
-Advertise correct mail capabilities through the front-container, this also enables support for PIPELINING in mail-protocols and IMAP IDLE which is a (potential) performance gain.

towncrier/newsfragments/1167.doc

@@ -1 +0,0 @@
-Added documentation that describes how spam filtering works in Mailu.

towncrier/newsfragments/1194.feature

@@ -1 +0,0 @@
-Change default password scheme to PBKDF2

towncrier/newsfragments/1197.feature

@@ -1 +0,0 @@
-Enable access log of admin service only for log levels of INFO and finer

towncrier/newsfragments/1207.feature

@@ -1 +0,0 @@
-japanese loca is now available

towncrier/newsfragments/1214.misc

@@ -1 +0,0 @@
-Move rspamd, fetchmail and unbound services into core and optional

towncrier/newsfragments/1221.removal

@@ -1 +0,0 @@
-Dovecot: Delete obsolete data volume

towncrier/newsfragments/1255.bugfix

@@ -1 +0,0 @@
-Use pip package for radicale to fix failing builds caused by [alpine]upstream package rebuild against different python version

towncrier/newsfragments/1259.feature

@@ -1 +0,0 @@
-Allow to reject virus mails by setting ANTIVITUS_ACTION=reject

towncrier/newsfragments/1267.feature

@@ -1 +0,0 @@
-Update roundcube to 1.4.0 and enable the new elastic skin

towncrier/newsfragments/1268.feature

@@ -1 +0,0 @@
-The roundcube container does support mysql now (no setup integration yet)

towncrier/newsfragments/1289.bugfix

@@ -1 +0,0 @@
-Disable Health checks on swarm mode

towncrier/newsfragments/1298.feature

@@ -1 +0,0 @@
-Added CardDAV-Plugin for webmail roundcube.

towncrier/newsfragments/1308.misc

@@ -1 +0,0 @@
-Use Redis 5.0 in kubernetes manifests

towncrier/newsfragments/1320.feature

@@ -1 +0,0 @@
-Allow users to use server-sided full-text-search again by adding the dovecot fts-xapian plugin

towncrier/newsfragments/1357.feature

@@ -1 +0,0 @@
-Relay a domain to a nonstandard SMTP port by adding ":<port_num>" to the remote hostname or IP address.

towncrier/newsfragments/1394.bugfix

@@ -1 +0,0 @@
-Show SPF records in accordance with RFC 7208: Previously we instructed admins to create SPF and TXT records, where only TXT records are correct now. !! Attention !! You need to manually remove the SPF-typed records and keep only TXT in your DNS.

towncrier/newsfragments/1438.bugfix

@@ -1 +0,0 @@
-Cover relearning messages when moving bewteen Ham and Spam status

towncrier/newsfragments/1444.misc

@@ -1 +0,0 @@
-Harden security by making certain configuration files read-only. Moves Postfix and Dovecot overrides an independed sub-directories.

towncrier/newsfragments/1463.bugfix

@@ -1 +0,0 @@
-Defining POSTMASTER through setup tool apply also to DMARC_RUA and DMARC_RUF settings

towncrier/newsfragments/1486.bugfix

@@ -1 +0,0 @@
-Check postfix mailqueue permissions before start-up 

towncrier/newsfragments/1512.misc

@@ -1 +0,0 @@
-Use Radicale 3.x for webdav service

towncrier/newsfragments/1538.feature

@@ -1 +0,0 @@
-Introduce option to disable dovecot full-text-search by an enviroment variable. 

towncrier/newsfragments/1564.bugfix

@@ -1 +0,0 @@
-Fixes certbot renewal

towncrier/newsfragments/1590.doc

@@ -1 +0,0 @@
-Add documentation for the web administration interface.

towncrier/newsfragments/1635.feature

@@ -1 +0,0 @@
-Add support for AUTH LOGIN authentication mechanism for relaying email via smart hosts.

towncrier/newsfragments/328.feature

@@ -1 +0,0 @@
-Add support for backward-forwarding using SRS

towncrier/newsfragments/508.misc

@@ -1 +0,0 @@
-Enable sponsorship

towncrier/newsfragments/985.feature

@@ -1 +0,0 @@
-Add options to support different architectures builds