From cc9a6b05a80de3669fddfd1fb34f95a0ee14ffaf Mon Sep 17 00:00:00 2001 From: =?UTF-8?q?Tim=20M=C3=B6hlmann?= Date: Mon, 14 Oct 2019 22:13:05 +0300 Subject: [PATCH 01/44] RFC: Mailu directory structure --- design/mailu-directory-structure.md | 199 ++++++++++++++++++++++++++++ 1 file changed, 199 insertions(+) create mode 100644 design/mailu-directory-structure.md diff --git a/design/mailu-directory-structure.md b/design/mailu-directory-structure.md new file mode 100644 index 00000000..ad75eeaa --- /dev/null +++ b/design/mailu-directory-structure.md @@ -0,0 +1,199 @@ +# RFC: Mailu directory structure +The current layout of the Mailu directory structure can be improved to allow for easier replicated deployments, like Docker Swarm and Kubernetes. Please read https://usrpro.com/publications/mailu-persistent-storage/ for the background motivation of this RFC. + +## Scope +This document only describes the re-arrangement of the `$ROOT` Mailu filesystem as-is. This means moving around of current files and directories. The linked article also proposes more advanced improvements. Those are not included in this RFC and need to be evaluated and implemented independently. However, the changes proposed in this document should make such improvements easier. + +Currently some services are wrongfully sharing mountpoints or have unused volumes declared. Those are also taken care of in this RFC. + +## Compatibility +If these changes were to be accepted, it will break compatibility with previous Mailu version `<=1.7`. As such, we will propably increment to version `2.0`. + +## Root +The root of the Mailu filesystem is located at `/mailu` by default. For simplicity we will assume this location throughout the document. Within `/mailu` we will aim to define 3 main sub-directories: + +### Config + +- Path: `/mailu/config/` + +Small config bearing files, sometimes shared between multiple services. The performance and storage needs for this filesystem are low. Availability is important for correct functioning of the mail server. No file locking issues are expected from concurrent access. A basic (and redundant) filesystem should suffice. + +#### Dovecot + +- Old path: `/mailu/overrides` (shared with postfix, nginx and rspamd) +- New Path `/mailu/config/dovecot` + +Dovecot configuration overrides. + +#### Postfix + +- Old path: `/mailu/overrides` (shared with dovecot, nginx and rspamd) +- New Path `/mailu/config/posfix` + +Postfix configuration overrides. + +#### Rspamd + +- Old path: `/mailu/overrides/rspamd` +- New path: `/mailu/config/rspamd` + +RSpamD configuration overrides. + +#### Rainloop + +- Old path: `/mailu/webmail/_data_/_default_/storage` (part of `/mailu/webmail` mountpoint, shared with Roundcube) +- New path: `/mailu/config/rainloop` + +User specific configs. The remaining files under the old `/mailu/webmail` don't need to be persistent. Except for `AddressBook.sqlite`, see `/mailu/data`. + +#### Roundcube + +- Old path: `/mailu/webmail/gpg` (part of `/mailu/webmail` mountpoint, shared with Rainloop) +- New path: `/mailu/config/roundcube/gpg` + +User configured GPG keys. + +#### Redis + +- Old path: `/mailu/redis` +- New path: `/mailu/config/redis` + +Holds `dump.rdb` for data restoration. Although technically a database, Redis works from memory. The dump file is only written to every minute and read from during start. Hence it fits better in the replicated config directory filesystem. + +#### Share + +- Path: `/mailu/config/share/` + +Shared configuration between different services + +##### DKIM + +- Old path: `/mailu/dkim/` +- New path: `/mailu/config/share/dkim` + +DKIM private keys store. Read/write access by Admin. Read only access by rSpamD. + +##### Certs + +- Old path: `/mailu/certs` +- New path: `/mailu/config/share/certs` (Proposal in anticipation of the RFC outcome at: https://github.com/Mailu/Mailu/issues/1222.) + +TLS certificates. Write access from `nginx` in case `TLS_FLAVOR=letsencrypt`. Or write access from `treafik-certdumper` or any other tool obtaining certificates. + +`letsencrypt` setting is not compatible with replicated setups. Multiple instances would disrupt the ACME challenge verification and cause race conditions on requesting certificates. In such cases certificates will need to be provided by other tools. + +If RFC issue #1222 is accepted, Dovecot will need read-only access to the certificates. + +### Data + +- Path: `/mailu/data/` + +Database files, like SQLite or PostgreSQL files. Databases don't perform well on network filesystems as they depend heavily on file locking and full controll on the database files. Making it unfit for concurrent access from multiple hosts. This directory should always live on a local filesystem. This makes it only usable in `docker-compose` deployments. Usage of this directory should be avoided in Kubernetes and Docker Swarm deployments. Some services will need to be improved to allow for this. + +#### admin data + +- Old path: `/mailu/data/` +- New path: `/mailu/data/admin/` (mount point on `admin` directory) + +Holds `main.db` SQLite database file holding domains, users, aliases etcetera. Read/write access only by admin. Can be avoided by using a remote DB server like PostgreSQL, MySQL or MariaDB. + +Also holds `instance` for unique statistics transmission. Removing of this file is proposed in RFC [issue 129](https://github.com/Mailu/Mailu/issues/1219). + +This move is needed in order to be able to mount the directory without exposing data files from other services into admin. + +#### rspamd + +- Old path: `/mailu/filter` (shared with ClamAV) +- New path: `/mailu/data/rspamd` + +Storage of Bayes and Fuzzy learning SQLite databases and caches. As future optimization we should look into moving all this into Redis. + +#### Rainloop + +- Old path: `/mailu/webmail/_data_/_default_/AddressBook.sqlite` (part of `/mailu/webmail` mountpoint, shared with Roundcube) +- New path: `/mailu/data/rainloop/AddressBook.sqlite` (mount on `rainloop` directory) + +Addressbook SQLite file. For future replicated deployments this might better be configured to use an external DB. + +For this modification, the `AddressBook.sqlite` will need to be moved to a different directory inside the container. + +#### Roundcube + +- Old path: `/mailu/webmail/roundcube.db` (part of `/mailu/webmail` mountpoint, shared with Rainloop) +- New path: `/mailu/data/roundcube/roundcube.db` (mount on `roundcube` directory) + +User settings SQLite database file for roundcube. For future replicated deployments this might better be configured to use an external DB. + +For this modification, the `rouncube.db` file will need to be moved to a different directory inside the container. + +### Mail + +- Path: `/mailu/mail` (unmodified) + +User mail, managed my Dovecot IMAP server. In replicated deployments, this filesystem needs to be shared over all IMAP server instances. It should be high performant and capable of propgating file locks. Storage size is proportional to the users and their quotas. Old versions of NFS are known to be buggy with file locking. Also Samaba or CIFS should be avoided. + +In the old situation, Maildir indexes are stored on the same volume. However, they need not to be persistent and should be located on a voletile filesystem instead. This allows better performance on network filesystems. + +### Local + +- Path: `/mailu/local` (new) + +Persistent storage not suitable for replication. In `docker-compose` deployments it lives inside `/mailu` and in replicated deployments it should live somewhere on the local host machine. + +#### Mailqueue + +- Old path: `/mailu/mailqueue` +- New path: `/mailu/local/mailqueue` + +The SMTP mailqueue should be persistant, as per SMTP spec it is not allowed to loose mail. However, persistance should be local only for performance reasons and the possibility to replicate Postfix servers. In setups like Docker Swarm and Kubernetes, admins should take care that Postfix is always restarted on same hosts in order to empty any remaining queue after a crash. + +#### ClamAV + +- Old path: `/mailu/filters` (shared with rSpamD) +- New path: `/mailu/local/clamav` + +Virus definitions do not need to be replicated, as they can be easily pulled in when ClamAV instances migrate to other nodes. Persistance does allow for some bandwith and time saving if ClamAV would be restarted on a previously used node (in case of updates or similar cases). Local only storage also prevents `freshclam` race conditions. + +## Conclusion + +The final layout of the Mailu filesystem will look like: + +```` +/mailu +├── config +│   ├── dovecot +│   ├── postfix +│   ├── rainloop +│   ├── redis +│   ├── roundcube +│   │   └── gpg +│   ├── rspamd +│   └── share +│   ├── certs +│   └── dkim +├── data +│   ├── admin +│   ├── rainloop +│   ├── roundcube +│   └── rspamd +├── local +│   ├── clamav +│   └── mailqueue +└── mail +```` + +Where in replicated environments: + +- `/mailu/config/`: should be a small, low performant and shared filesystem. +- `/mailu/data`: should be avoided. More work will need to be done to configure external DB servers for relevant services. Ideally, this directory should only exist on docker-compose deployments. +- `/mailu/local/`: Should exist only on local file systems of worker nodes. +- `/mailu/mail`: A distributed filesystem with sufficient performance and storage requirements to hold and process all user mailboxes. Ideally only Maildir without indexes. + +### Implementing + +The works to implement this changes should happen outside the `master` branch. Inclusion into `master` can only be accepted if: + +1. `docker-compose.yml` from setup reflects this changes correctly. +2. Kubernetes documentation is updated. +3. Legacy `docker-compose.yml` is either updated or deleted. +4. A clear data migration guide is written. \ No newline at end of file From 598b2df5a02e6e5532f86c936812a7bd23030a9a Mon Sep 17 00:00:00 2001 From: Alexander Graf Date: Tue, 2 Nov 2021 13:04:40 +0100 Subject: [PATCH 02/44] update wtforms --- core/admin/requirements-prod.txt | 4 ++-- 1 file changed, 2 insertions(+), 2 deletions(-) diff --git a/core/admin/requirements-prod.txt b/core/admin/requirements-prod.txt index 3406122d..506cf050 100644 --- a/core/admin/requirements-prod.txt +++ b/core/admin/requirements-prod.txt @@ -52,5 +52,5 @@ tenacity==5.0.4 validators==0.12.6 visitor==0.1.3 Werkzeug==0.15.5 -WTForms==2.2.1 -WTForms-Components==0.10.4 +WTForms==2.3.3 +WTForms-Components==0.10.5 From 80be3506daddf5d075c93e94b1521321baa369af Mon Sep 17 00:00:00 2001 From: Alexander Graf Date: Tue, 2 Nov 2021 13:32:12 +0100 Subject: [PATCH 03/44] upgrade pip. completed reqs via pip freeze --- core/admin/Dockerfile | 3 ++- core/admin/requirements-prod.txt | 27 ++++++++++++++++++++++++--- 2 files changed, 26 insertions(+), 4 deletions(-) diff --git a/core/admin/Dockerfile b/core/admin/Dockerfile index cdc426c6..95a46fa2 100644 --- a/core/admin/Dockerfile +++ b/core/admin/Dockerfile @@ -35,7 +35,8 @@ COPY requirements-prod.txt requirements.txt RUN set -eu \ && apk add --no-cache libressl curl postgresql-libs mariadb-connector-c \ && apk add --no-cache --virtual build-dep libressl-dev libffi-dev python3-dev build-base postgresql-dev mariadb-connector-c-dev cargo \ - && pip3 install -r requirements.txt \ + && pip install --upgrade pip \ + && pip install -r requirements.txt \ && apk del --no-cache build-dep COPY --from=assets static ./mailu/ui/static diff --git a/core/admin/requirements-prod.txt b/core/admin/requirements-prod.txt index 506cf050..38717145 100644 --- a/core/admin/requirements-prod.txt +++ b/core/admin/requirements-prod.txt @@ -1,14 +1,23 @@ alembic==1.0.10 +appdirs==1.4.4 asn1crypto==0.24.0 Babel==2.6.0 bcrypt==3.1.6 blinker==1.4 +CacheControl==0.12.6 +certifi==2020.12.5 cffi==1.12.3 +chardet==4.0.0 Click==7.0 +colorama==0.4.4 +contextlib2==0.6.0 cryptography==3.4.7 decorator==4.4.0 +distlib==0.3.1 +distro==1.5.0 dnspython==1.16.0 dominate==2.3.5 +email-validator==1.1.3 Flask==1.0.2 Flask-Babel==0.12.2 Flask-Bootstrap==3.3.7.1 @@ -21,36 +30,48 @@ Flask-Script==2.0.6 Flask-SQLAlchemy==2.4.0 Flask-WTF==0.14.2 gunicorn==20.1.0 +html5lib==1.1 idna==2.8 infinity==1.4 intervals==0.8.1 itsdangerous==1.1.0 Jinja2==2.11.3 limits==1.3 +lockfile==0.12.2 Mako==1.0.9 MarkupSafe==1.1.1 -mysqlclient==1.4.2.post1 marshmallow==3.10.0 marshmallow-sqlalchemy==0.24.1 +msgpack==1.0.2 +mysqlclient==1.4.2.post1 +ordered-set==4.0.2 +packaging==20.9 passlib==1.7.4 +pep517==0.10.0 +progress==1.5 psycopg2==2.8.2 pycparser==2.19 Pygments==2.8.1 pyOpenSSL==20.0.1 +pyparsing==2.4.7 python-dateutil==2.8.0 python-editor==1.0.4 pytz==2019.1 PyYAML==5.4.1 redis==3.2.1 -#alpine3:12 provides six==1.15.0 -#six==1.12.0 +requests==2.25.1 +retrying==1.3.3 +six==1.15.0 socrate==0.1.1 SQLAlchemy==1.3.3 srslib==0.1.4 tabulate==0.8.3 tenacity==5.0.4 +toml==0.10.2 +urllib3==1.26.5 validators==0.12.6 visitor==0.1.3 +webencodings==0.5.1 Werkzeug==0.15.5 WTForms==2.3.3 WTForms-Components==0.10.5 From 8d90a74624631745ea7f4964a4f7b55a5c27ad46 Mon Sep 17 00:00:00 2001 From: Alexander Graf Date: Tue, 2 Nov 2021 15:39:41 +0100 Subject: [PATCH 04/44] update werkzeug to 1.x --- core/admin/mailu/debug.py | 4 ++-- core/admin/mailu/utils.py | 4 ++-- core/admin/requirements-prod.txt | 6 +++--- 3 files changed, 7 insertions(+), 7 deletions(-) diff --git a/core/admin/mailu/debug.py b/core/admin/mailu/debug.py index 7677901b..4d63f3c5 100644 --- a/core/admin/mailu/debug.py +++ b/core/admin/mailu/debug.py @@ -1,6 +1,6 @@ import flask_debugtoolbar -from werkzeug.contrib import profiler as werkzeug_profiler +from werkzeug.middleware.profiler import ProfilerMiddleware # Debugging toolbar @@ -10,7 +10,7 @@ toolbar = flask_debugtoolbar.DebugToolbarExtension() # Profiler class Profiler(object): def init_app(self, app): - app.wsgi_app = werkzeug_profiler.ProfilerMiddleware( + app.wsgi_app = ProfilerMiddleware( app.wsgi_app, restrictions=[30] ) diff --git a/core/admin/mailu/utils.py b/core/admin/mailu/utils.py index dda927b0..eddd6848 100644 --- a/core/admin/mailu/utils.py +++ b/core/admin/mailu/utils.py @@ -28,7 +28,7 @@ import redis from flask.sessions import SessionMixin, SessionInterface from itsdangerous.encoding import want_bytes from werkzeug.datastructures import CallbackDict -from werkzeug.contrib import fixers +from werkzeug.middleware.proxy_fix import ProxyFix # Login configuration login = flask_login.LoginManager() @@ -109,7 +109,7 @@ class PrefixMiddleware(object): return self.app(environ, start_response) def init_app(self, app): - self.app = fixers.ProxyFix(app.wsgi_app, x_for=1, x_proto=1) + self.app = ProxyFix(app.wsgi_app, x_for=1, x_proto=1) app.wsgi_app = self proxy = PrefixMiddleware() diff --git a/core/admin/requirements-prod.txt b/core/admin/requirements-prod.txt index 38717145..673c4445 100644 --- a/core/admin/requirements-prod.txt +++ b/core/admin/requirements-prod.txt @@ -19,7 +19,7 @@ dnspython==1.16.0 dominate==2.3.5 email-validator==1.1.3 Flask==1.0.2 -Flask-Babel==0.12.2 +Flask-Babel==1.0.0 Flask-Bootstrap==3.3.7.1 Flask-DebugToolbar==0.10.1 Flask-Limiter==1.0.1 @@ -28,7 +28,7 @@ flask-marshmallow==0.14.0 Flask-Migrate==2.4.0 Flask-Script==2.0.6 Flask-SQLAlchemy==2.4.0 -Flask-WTF==0.14.2 +Flask-WTF==0.15.1 gunicorn==20.1.0 html5lib==1.1 idna==2.8 @@ -72,6 +72,6 @@ urllib3==1.26.5 validators==0.12.6 visitor==0.1.3 webencodings==0.5.1 -Werkzeug==0.15.5 +Werkzeug==1.0.1 WTForms==2.3.3 WTForms-Components==0.10.5 From 23d0cd04660abdb4ce9f4ebfb8b3f52070fa4c25 Mon Sep 17 00:00:00 2001 From: Alexander Graf Date: Tue, 2 Nov 2021 15:55:20 +0100 Subject: [PATCH 05/44] update tabluate. fix audit.py and include in container --- core/admin/Dockerfile | 1 + core/admin/audit.py | 28 +++++++++++++++------------- core/admin/requirements-prod.txt | 2 +- 3 files changed, 17 insertions(+), 14 deletions(-) mode change 100644 => 100755 core/admin/audit.py diff --git a/core/admin/Dockerfile b/core/admin/Dockerfile index 95a46fa2..8adf2ca5 100644 --- a/core/admin/Dockerfile +++ b/core/admin/Dockerfile @@ -43,6 +43,7 @@ COPY --from=assets static ./mailu/ui/static COPY mailu ./mailu COPY migrations ./migrations COPY start.py /start.py +COPY audit.py /audit.py RUN pybabel compile -d mailu/translations diff --git a/core/admin/audit.py b/core/admin/audit.py old mode 100644 new mode 100755 index db105ff4..79365286 --- a/core/admin/audit.py +++ b/core/admin/audit.py @@ -1,14 +1,17 @@ -from mailu import app +#!/usr/bin/python3 import sys import tabulate +import mailu +app = mailu.create_app() + # Known endpoints without permissions known_missing_permissions = [ - "index", - "static", "bootstrap.static", - "admin.static", "admin.login" + 'index', + 'static', 'bootstrap.static', + 'admin.static', 'admin.login' ] @@ -16,7 +19,7 @@ known_missing_permissions = [ missing_permissions = [] permissions = {} for endpoint, function in app.view_functions.items(): - audit = function.__dict__.get("_audit_permissions") + audit = function.__dict__.get('_audit_permissions') if audit: handler, args = audit if args: @@ -28,16 +31,15 @@ for endpoint, function in app.view_functions.items(): elif endpoint not in known_missing_permissions: missing_permissions.append(endpoint) - -# Fail if any endpoint is missing a permission check -if missing_permissions: - print("The following endpoints are missing permission checks:") - print(missing_permissions.join(",")) - sys.exit(1) - - # Display the permissions table print(tabulate.tabulate([ [route, *permissions[route.endpoint]] for route in app.url_map.iter_rules() if route.endpoint in permissions ])) + +# Warn if any endpoint is missing a permission check +if missing_permissions: + print() + print('The following endpoints are missing permission checks:') + print(','.join(missing_permissions)) + diff --git a/core/admin/requirements-prod.txt b/core/admin/requirements-prod.txt index 673c4445..79075f96 100644 --- a/core/admin/requirements-prod.txt +++ b/core/admin/requirements-prod.txt @@ -65,7 +65,7 @@ six==1.15.0 socrate==0.1.1 SQLAlchemy==1.3.3 srslib==0.1.4 -tabulate==0.8.3 +tabulate==0.8.9 tenacity==5.0.4 toml==0.10.2 urllib3==1.26.5 From 771b2d1112d846ddde6e5f7f54f787280c9a0c04 Mon Sep 17 00:00:00 2001 From: Alexander Graf Date: Tue, 2 Nov 2021 16:21:31 +0100 Subject: [PATCH 06/44] duh --- core/admin/audit.py | 2 ++ 1 file changed, 2 insertions(+) diff --git a/core/admin/audit.py b/core/admin/audit.py index 79365286..60583f83 100755 --- a/core/admin/audit.py +++ b/core/admin/audit.py @@ -3,6 +3,8 @@ import sys import tabulate +sys.path[0:0] = ['/app'] + import mailu app = mailu.create_app() From dcbe55f0620395532a767d1a69647154d886a943 Mon Sep 17 00:00:00 2001 From: Alexander Graf Date: Tue, 2 Nov 2021 16:28:37 +0100 Subject: [PATCH 07/44] updated crypto --- core/admin/requirements-prod.txt | 13 ++++++------- core/admin/requirements.txt | 1 - 2 files changed, 6 insertions(+), 8 deletions(-) diff --git a/core/admin/requirements-prod.txt b/core/admin/requirements-prod.txt index 79075f96..9bd131fb 100644 --- a/core/admin/requirements-prod.txt +++ b/core/admin/requirements-prod.txt @@ -1,18 +1,17 @@ alembic==1.0.10 appdirs==1.4.4 -asn1crypto==0.24.0 Babel==2.6.0 -bcrypt==3.1.6 +bcrypt==3.2.0 blinker==1.4 CacheControl==0.12.6 certifi==2020.12.5 -cffi==1.12.3 +cffi==1.15.0 chardet==4.0.0 Click==7.0 colorama==0.4.4 contextlib2==0.6.0 -cryptography==3.4.7 -decorator==4.4.0 +cryptography==35.0.0 +decorator==5.1.0 distlib==0.3.1 distro==1.5.0 dnspython==1.16.0 @@ -50,9 +49,9 @@ passlib==1.7.4 pep517==0.10.0 progress==1.5 psycopg2==2.8.2 -pycparser==2.19 +pycparser==2.20 Pygments==2.8.1 -pyOpenSSL==20.0.1 +pyOpenSSL==21.0.0 pyparsing==2.4.7 python-dateutil==2.8.0 python-editor==1.0.4 diff --git a/core/admin/requirements.txt b/core/admin/requirements.txt index e1de3b01..6cfa8a35 100644 --- a/core/admin/requirements.txt +++ b/core/admin/requirements.txt @@ -18,7 +18,6 @@ PyYAML PyOpenSSL Pygments dnspython -bcrypt tenacity mysqlclient psycopg2 From 40cdff491192cf8135e9c7818d490a39ce0f5fe0 Mon Sep 17 00:00:00 2001 From: Alexander Graf Date: Tue, 2 Nov 2021 16:49:25 +0100 Subject: [PATCH 08/44] updated dnspython --- core/admin/mailu/models.py | 3 ++- core/admin/mailu/ui/views/domains.py | 1 - core/admin/mailu/utils.py | 6 +++++- core/admin/requirements-prod.txt | 2 +- 4 files changed, 8 insertions(+), 4 deletions(-) diff --git a/core/admin/mailu/models.py b/core/admin/mailu/models.py index f5fe3b5e..300d8d8c 100644 --- a/core/admin/mailu/models.py +++ b/core/admin/mailu/models.py @@ -19,7 +19,8 @@ import os import hmac import smtplib import idna -import dns +import dns.resolver +import dns.exception from flask import current_app as app from sqlalchemy.ext import declarative diff --git a/core/admin/mailu/ui/views/domains.py b/core/admin/mailu/ui/views/domains.py index f394ce7d..a48bb154 100644 --- a/core/admin/mailu/ui/views/domains.py +++ b/core/admin/mailu/ui/views/domains.py @@ -5,7 +5,6 @@ from flask import current_app as app import flask import flask_login import wtforms_components -import dns.resolver @ui.route('/domain', methods=['GET']) diff --git a/core/admin/mailu/utils.py b/core/admin/mailu/utils.py index eddd6848..d9cc1894 100644 --- a/core/admin/mailu/utils.py +++ b/core/admin/mailu/utils.py @@ -6,8 +6,12 @@ try: except ImportError: import pickle -import dns import dns.resolver +import dns.exception +import dns.flags +import dns.rdtypes +import dns.rdatatype +import dns.rdataclass import hmac import secrets diff --git a/core/admin/requirements-prod.txt b/core/admin/requirements-prod.txt index 9bd131fb..ecb0b8ed 100644 --- a/core/admin/requirements-prod.txt +++ b/core/admin/requirements-prod.txt @@ -14,7 +14,7 @@ cryptography==35.0.0 decorator==5.1.0 distlib==0.3.1 distro==1.5.0 -dnspython==1.16.0 +dnspython==2.1.0 dominate==2.3.5 email-validator==1.1.3 Flask==1.0.2 From 3ac1b3d86ce2452a87ff1a27de94f597773fff0e Mon Sep 17 00:00:00 2001 From: Alexander Graf Date: Tue, 2 Nov 2021 17:02:54 +0100 Subject: [PATCH 09/44] update pyyaml and pygments --- core/admin/requirements-prod.txt | 4 ++-- 1 file changed, 2 insertions(+), 2 deletions(-) diff --git a/core/admin/requirements-prod.txt b/core/admin/requirements-prod.txt index ecb0b8ed..159a3c3c 100644 --- a/core/admin/requirements-prod.txt +++ b/core/admin/requirements-prod.txt @@ -50,13 +50,13 @@ pep517==0.10.0 progress==1.5 psycopg2==2.8.2 pycparser==2.20 -Pygments==2.8.1 +Pygments==2.10.0 pyOpenSSL==21.0.0 pyparsing==2.4.7 python-dateutil==2.8.0 python-editor==1.0.4 pytz==2019.1 -PyYAML==5.4.1 +PyYAML==6.0 redis==3.2.1 requests==2.25.1 retrying==1.3.3 From 8ad8cde0e27b62679e924fc7b4ba64ed7c53cb67 Mon Sep 17 00:00:00 2001 From: Alexander Graf Date: Tue, 2 Nov 2021 17:06:28 +0100 Subject: [PATCH 10/44] removed some obsolete requirements --- core/admin/requirements-prod.txt | 2 -- core/admin/requirements.txt | 1 - 2 files changed, 3 deletions(-) diff --git a/core/admin/requirements-prod.txt b/core/admin/requirements-prod.txt index 159a3c3c..9d734d94 100644 --- a/core/admin/requirements-prod.txt +++ b/core/admin/requirements-prod.txt @@ -53,8 +53,6 @@ pycparser==2.20 Pygments==2.10.0 pyOpenSSL==21.0.0 pyparsing==2.4.7 -python-dateutil==2.8.0 -python-editor==1.0.4 pytz==2019.1 PyYAML==6.0 redis==3.2.1 diff --git a/core/admin/requirements.txt b/core/admin/requirements.txt index 6cfa8a35..65130a8c 100644 --- a/core/admin/requirements.txt +++ b/core/admin/requirements.txt @@ -20,7 +20,6 @@ Pygments dnspython tenacity mysqlclient -psycopg2 idna srslib marshmallow From d8efd3057c85a397dd7c77f81d59ae4d91475841 Mon Sep 17 00:00:00 2001 From: Alexander Graf Date: Tue, 2 Nov 2021 17:52:25 +0100 Subject: [PATCH 11/44] updated idna --- core/admin/requirements-prod.txt | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/core/admin/requirements-prod.txt b/core/admin/requirements-prod.txt index 9d734d94..649eac49 100644 --- a/core/admin/requirements-prod.txt +++ b/core/admin/requirements-prod.txt @@ -30,7 +30,7 @@ Flask-SQLAlchemy==2.4.0 Flask-WTF==0.15.1 gunicorn==20.1.0 html5lib==1.1 -idna==2.8 +idna==3.2 infinity==1.4 intervals==0.8.1 itsdangerous==1.1.0 From ef19869cdef45829c8b52276d58ddab4e72085ad Mon Sep 17 00:00:00 2001 From: Alexander Graf Date: Tue, 2 Nov 2021 18:06:26 +0100 Subject: [PATCH 12/44] updated redis --- core/admin/requirements-prod.txt | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/core/admin/requirements-prod.txt b/core/admin/requirements-prod.txt index 649eac49..40447e79 100644 --- a/core/admin/requirements-prod.txt +++ b/core/admin/requirements-prod.txt @@ -55,7 +55,7 @@ pyOpenSSL==21.0.0 pyparsing==2.4.7 pytz==2019.1 PyYAML==6.0 -redis==3.2.1 +redis==3.5.3 requests==2.25.1 retrying==1.3.3 six==1.15.0 From 866741bcbeaae33cf819bd0bebe2517ad9317008 Mon Sep 17 00:00:00 2001 From: Alexander Graf Date: Tue, 2 Nov 2021 19:22:58 +0100 Subject: [PATCH 13/44] updated WTForms-Components deps --- core/admin/requirements-prod.txt | 6 +++--- 1 file changed, 3 insertions(+), 3 deletions(-) diff --git a/core/admin/requirements-prod.txt b/core/admin/requirements-prod.txt index 40447e79..d88c71ed 100644 --- a/core/admin/requirements-prod.txt +++ b/core/admin/requirements-prod.txt @@ -31,8 +31,8 @@ Flask-WTF==0.15.1 gunicorn==20.1.0 html5lib==1.1 idna==3.2 -infinity==1.4 -intervals==0.8.1 +infinity==1.5 +intervals==0.9.2 itsdangerous==1.1.0 Jinja2==2.11.3 limits==1.3 @@ -66,7 +66,7 @@ tabulate==0.8.9 tenacity==5.0.4 toml==0.10.2 urllib3==1.26.5 -validators==0.12.6 +validators==0.18.2 visitor==0.1.3 webencodings==0.5.1 Werkzeug==1.0.1 From aca1e136484d73ea9c978a97563220bf1e633280 Mon Sep 17 00:00:00 2001 From: Alexander Graf Date: Tue, 2 Nov 2021 20:47:53 +0100 Subject: [PATCH 14/44] update socrate - will be removed later --- core/admin/requirements-prod.txt | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/core/admin/requirements-prod.txt b/core/admin/requirements-prod.txt index d88c71ed..3883c1c6 100644 --- a/core/admin/requirements-prod.txt +++ b/core/admin/requirements-prod.txt @@ -59,7 +59,7 @@ redis==3.5.3 requests==2.25.1 retrying==1.3.3 six==1.15.0 -socrate==0.1.1 +socrate==0.2.0 SQLAlchemy==1.3.3 srslib==0.1.4 tabulate==0.8.9 From f1d7bedd1b5465ac4fd86f7c4f380b257eee9059 Mon Sep 17 00:00:00 2001 From: Alexander Graf Date: Wed, 3 Nov 2021 19:54:15 +0100 Subject: [PATCH 15/44] fix display of range inputs (again) --- core/admin/assets/app.css | 5 +++++ core/admin/assets/app.js | 4 +++- 2 files changed, 8 insertions(+), 1 deletion(-) diff --git a/core/admin/assets/app.css b/core/admin/assets/app.css index 3886b5c1..84644900 100644 --- a/core/admin/assets/app.css +++ b/core/admin/assets/app.css @@ -52,3 +52,8 @@ fieldset:disabled .form-control:disabled { .select2-container--default .select2-selection--multiple .select2-selection__choice { color: black; } + +/* range input spacing */ +.input-group-text { + margin-right: 1em; +} diff --git a/core/admin/assets/app.js b/core/admin/assets/app.js index 54602d1f..e18228eb 100644 --- a/core/admin/assets/app.js +++ b/core/admin/assets/app.js @@ -43,7 +43,9 @@ $('document').ready(function() { var infinity = $(this).data('infinity'); var step = $(this).attr('step'); $(this).on('input', function() { - value_element.text((infinity && this.value == 0) ? '∞' : (this.value/step).toFixed(2)); + var num = (infinity && this.value == 0) ? '∞' : (this.value/step).toFixed(2); + if (num.endsWith('.00')) num = num.substr(0, num.length - 3); + value_element.text(num); }).trigger('input'); } }); From abc4112242d24ee6d6e4dd976ee51cae720df228 Mon Sep 17 00:00:00 2001 From: Alexander Graf Date: Wed, 3 Nov 2021 20:12:20 +0100 Subject: [PATCH 16/44] updated Werkzeug, Click and Flask-Migrate --- core/admin/requirements-prod.txt | 6 +++--- 1 file changed, 3 insertions(+), 3 deletions(-) diff --git a/core/admin/requirements-prod.txt b/core/admin/requirements-prod.txt index 3883c1c6..4a209fba 100644 --- a/core/admin/requirements-prod.txt +++ b/core/admin/requirements-prod.txt @@ -7,7 +7,7 @@ CacheControl==0.12.6 certifi==2020.12.5 cffi==1.15.0 chardet==4.0.0 -Click==7.0 +click==8.0.3 colorama==0.4.4 contextlib2==0.6.0 cryptography==35.0.0 @@ -24,7 +24,7 @@ Flask-DebugToolbar==0.10.1 Flask-Limiter==1.0.1 Flask-Login==0.4.1 flask-marshmallow==0.14.0 -Flask-Migrate==2.4.0 +Flask-Migrate==3.1.0 Flask-Script==2.0.6 Flask-SQLAlchemy==2.4.0 Flask-WTF==0.15.1 @@ -69,6 +69,6 @@ urllib3==1.26.5 validators==0.18.2 visitor==0.1.3 webencodings==0.5.1 -Werkzeug==1.0.1 +Werkzeug==2.0.2 WTForms==2.3.3 WTForms-Components==0.10.5 From 26fb108a3fb435d097883901470781726757f9f4 Mon Sep 17 00:00:00 2001 From: Alexander Graf Date: Wed, 3 Nov 2021 20:22:47 +0100 Subject: [PATCH 17/44] updated Flask-Login --- core/admin/mailu/utils.py | 2 +- core/admin/requirements-prod.txt | 2 +- 2 files changed, 2 insertions(+), 2 deletions(-) diff --git a/core/admin/mailu/utils.py b/core/admin/mailu/utils.py index 972546d6..8ae028bf 100644 --- a/core/admin/mailu/utils.py +++ b/core/admin/mailu/utils.py @@ -269,7 +269,7 @@ class MailuSession(CallbackDict, SessionMixin): # set uid from dict data if self._uid is None: - self._uid = self.app.session_config.gen_uid(self.get('user_id', '')) + self._uid = self.app.session_config.gen_uid(self.get('_user_id', '')) # create new session id for new or regenerated sessions and force setting the cookie if self._sid is None: diff --git a/core/admin/requirements-prod.txt b/core/admin/requirements-prod.txt index 4a209fba..bb94e4b5 100644 --- a/core/admin/requirements-prod.txt +++ b/core/admin/requirements-prod.txt @@ -22,7 +22,7 @@ Flask-Babel==1.0.0 Flask-Bootstrap==3.3.7.1 Flask-DebugToolbar==0.10.1 Flask-Limiter==1.0.1 -Flask-Login==0.4.1 +Flask-Login==0.5.0 flask-marshmallow==0.14.0 Flask-Migrate==3.1.0 Flask-Script==2.0.6 From 8b15820b014f73bf41aaffca3914ee8207082114 Mon Sep 17 00:00:00 2001 From: Alexander Graf Date: Wed, 3 Nov 2021 20:35:05 +0100 Subject: [PATCH 18/44] fix sso login button spacing --- core/admin/mailu/ui/templates/macros.html | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/core/admin/mailu/ui/templates/macros.html b/core/admin/mailu/ui/templates/macros.html index 5143b697..75de4e0e 100644 --- a/core/admin/mailu/ui/templates/macros.html +++ b/core/admin/mailu/ui/templates/macros.html @@ -22,7 +22,7 @@ {%- if spacing %} {%- set width = (12 / fields|length)|int %} {%- else %} - {%- set width = 0 %} + {%- set width = 2 %} {% endif %}
From 833ccb5544816e5321a4066bb1102ed144c02384 Mon Sep 17 00:00:00 2001 From: Alexander Graf Date: Wed, 3 Nov 2021 20:38:00 +0100 Subject: [PATCH 19/44] reload page using GET when selecting language --- core/admin/assets/app.js | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/core/admin/assets/app.js b/core/admin/assets/app.js index e18228eb..0926578d 100644 --- a/core/admin/assets/app.js +++ b/core/admin/assets/app.js @@ -18,7 +18,7 @@ $('document').ready(function() { $.post({ url: $(this).attr('href'), success: function() { - location.reload(); + window.location = window.location.href; }, }); }); From f613205fe16488fcc1b7658bfff35eb186a543fa Mon Sep 17 00:00:00 2001 From: Alexander Graf Date: Wed, 3 Nov 2021 21:30:34 +0100 Subject: [PATCH 20/44] update tenacity --- core/admin/migrations/env.py | 69 ++++++++++++++++---------------- core/admin/requirements-prod.txt | 2 +- 2 files changed, 36 insertions(+), 35 deletions(-) diff --git a/core/admin/migrations/env.py b/core/admin/migrations/env.py index 3e45bb18..65a4e471 100755 --- a/core/admin/migrations/env.py +++ b/core/admin/migrations/env.py @@ -1,10 +1,12 @@ -from __future__ import with_statement +import logging +import tenacity + from alembic import context from sqlalchemy import engine_from_config, pool from logging.config import fileConfig -import logging -import tenacity -from tenacity import retry + +from flask import current_app +from mailu import models # this is the Alembic Config object, which provides # access to the values within the .ini file in use. @@ -17,20 +19,12 @@ logger = logging.getLogger('alembic.env') # add your model's MetaData object here # for 'autogenerate' support -# from myapp import mymodel -# target_metadata = mymodel.Base.metadata -from flask import current_app -config.set_main_option('sqlalchemy.url', - current_app.config.get('SQLALCHEMY_DATABASE_URI')) -#target_metadata = current_app.extensions['migrate'].db.metadata -from mailu import models +config.set_main_option( + 'sqlalchemy.url', + current_app.config.get('SQLALCHEMY_DATABASE_URI') +) target_metadata = models.Base.metadata -# other values from the config, defined by the needs of env.py, -# can be acquired: -# my_important_option = config.get_main_option("my_important_option") -# ... etc. - def run_migrations_offline(): """Run migrations in 'offline' mode. @@ -44,7 +38,7 @@ def run_migrations_offline(): script output. """ - url = config.get_main_option("sqlalchemy.url") + url = config.get_main_option('sqlalchemy.url') context.configure(url=url) with context.begin_transaction(): @@ -69,28 +63,35 @@ def run_migrations_online(): directives[:] = [] logger.info('No changes in schema detected.') - engine = engine_from_config(config.get_section(config.config_ini_section), - prefix='sqlalchemy.', - poolclass=pool.NullPool) + engine = engine_from_config( + config.get_section(config.config_ini_section), + prefix = 'sqlalchemy.', + poolclass = pool.NullPool + ) - connection = tenacity.Retrying( - stop=tenacity.stop_after_attempt(100), - wait=tenacity.wait_random(min=2, max=5), - before=tenacity.before_log(logging.getLogger("tenacity.retry"), logging.DEBUG), - before_sleep=tenacity.before_sleep_log(logging.getLogger("tenacity.retry"), logging.INFO), - after=tenacity.after_log(logging.getLogger("tenacity.retry"), logging.DEBUG) - ).call(engine.connect) + @tenacity.retry( + stop = tenacity.stop_after_attempt(100), + wait = tenacity.wait_random(min=2, max=5), + before = tenacity.before_log(logging.getLogger('tenacity.retry'), logging.DEBUG), + before_sleep = tenacity.before_sleep_log(logging.getLogger('tenacity.retry'), logging.INFO), + after = tenacity.after_log(logging.getLogger('tenacity.retry'), logging.DEBUG) + ) + def try_connect(db): + return db.connect() - context.configure(connection=connection, - target_metadata=target_metadata, - process_revision_directives=process_revision_directives, - **current_app.extensions['migrate'].configure_args) + with try_connect(engine) as connection: + + context.configure( + connection = connection, + target_metadata = target_metadata, + process_revision_directives = process_revision_directives, + **current_app.extensions['migrate'].configure_args + ) - try: with context.begin_transaction(): context.run_migrations() - finally: - connection.close() + + connection.close() if context.is_offline_mode(): run_migrations_offline() diff --git a/core/admin/requirements-prod.txt b/core/admin/requirements-prod.txt index bb94e4b5..3d08e2ff 100644 --- a/core/admin/requirements-prod.txt +++ b/core/admin/requirements-prod.txt @@ -63,7 +63,7 @@ socrate==0.2.0 SQLAlchemy==1.3.3 srslib==0.1.4 tabulate==0.8.9 -tenacity==5.0.4 +tenacity==8.0.1 toml==0.10.2 urllib3==1.26.5 validators==0.18.2 From 5238b00f0b0358700a16c52e77f35f1759cf6437 Mon Sep 17 00:00:00 2001 From: Alexander Graf Date: Wed, 3 Nov 2021 21:33:39 +0100 Subject: [PATCH 21/44] update alembic --- core/admin/requirements-prod.txt | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/core/admin/requirements-prod.txt b/core/admin/requirements-prod.txt index 3d08e2ff..4b629d5d 100644 --- a/core/admin/requirements-prod.txt +++ b/core/admin/requirements-prod.txt @@ -1,4 +1,4 @@ -alembic==1.0.10 +alembic==1.7.4 appdirs==1.4.4 Babel==2.6.0 bcrypt==3.2.0 From 56f65d724d1982fa0c3cb9a5444812328d283094 Mon Sep 17 00:00:00 2001 From: Alexander Graf Date: Wed, 3 Nov 2021 21:52:59 +0100 Subject: [PATCH 22/44] update babel --- core/admin/mailu/__init__.py | 5 +++++ core/admin/mailu/ui/templates/domain/list.html | 4 ++-- core/admin/mailu/utils.py | 3 +-- core/admin/requirements-prod.txt | 6 +++--- 4 files changed, 11 insertions(+), 7 deletions(-) diff --git a/core/admin/mailu/__init__.py b/core/admin/mailu/__init__.py index e4024e47..b49bdd67 100644 --- a/core/admin/mailu/__init__.py +++ b/core/admin/mailu/__init__.py @@ -57,6 +57,11 @@ def create_app_from_config(config): config = app.config, ) + # Jinja filter + @app.template_filter() + def format_date(value): + return utils.flask_babel.format_date(value) if value else '' + # Import views from mailu import ui, internal, sso app.register_blueprint(ui.ui, url_prefix=app.config['WEB_ADMIN']) diff --git a/core/admin/mailu/ui/templates/domain/list.html b/core/admin/mailu/ui/templates/domain/list.html index 6f6bc467..61c09151 100644 --- a/core/admin/mailu/ui/templates/domain/list.html +++ b/core/admin/mailu/ui/templates/domain/list.html @@ -46,8 +46,8 @@ {{ domain.users | count }} / {{ '∞' if domain.max_users == -1 else domain.max_users }} {{ domain.aliases | count }} / {{ '∞' if domain.max_aliases == -1 else domain.max_aliases }} {{ domain.comment or '' }} - {{ domain.created_at }} - {{ domain.updated_at or '' }} + {{ domain.created_at | format_date }} + {{ domain.updated_at | format_date }} {%- endfor %} diff --git a/core/admin/mailu/utils.py b/core/admin/mailu/utils.py index 8ae028bf..024c487f 100644 --- a/core/admin/mailu/utils.py +++ b/core/admin/mailu/utils.py @@ -18,10 +18,9 @@ import secrets import time from multiprocessing import Value - from mailu import limiter - from flask import current_app as app + import flask import flask_login import flask_migrate diff --git a/core/admin/requirements-prod.txt b/core/admin/requirements-prod.txt index 4b629d5d..ccd2b83d 100644 --- a/core/admin/requirements-prod.txt +++ b/core/admin/requirements-prod.txt @@ -1,6 +1,6 @@ alembic==1.7.4 appdirs==1.4.4 -Babel==2.6.0 +Babel==2.9.1 bcrypt==3.2.0 blinker==1.4 CacheControl==0.12.6 @@ -18,7 +18,7 @@ dnspython==2.1.0 dominate==2.3.5 email-validator==1.1.3 Flask==1.0.2 -Flask-Babel==1.0.0 +Flask-Babel==2.0.0 Flask-Bootstrap==3.3.7.1 Flask-DebugToolbar==0.10.1 Flask-Limiter==1.0.1 @@ -53,7 +53,7 @@ pycparser==2.20 Pygments==2.10.0 pyOpenSSL==21.0.0 pyparsing==2.4.7 -pytz==2019.1 +pytz==2021.3 PyYAML==6.0 redis==3.5.3 requests==2.25.1 From 87884213c466472a98cbd19e988b65c9190fea62 Mon Sep 17 00:00:00 2001 From: Alexander Graf Date: Wed, 3 Nov 2021 22:03:51 +0100 Subject: [PATCH 23/44] update misc helper libs --- core/admin/requirements-prod.txt | 14 +++++++------- 1 file changed, 7 insertions(+), 7 deletions(-) diff --git a/core/admin/requirements-prod.txt b/core/admin/requirements-prod.txt index ccd2b83d..282d00c3 100644 --- a/core/admin/requirements-prod.txt +++ b/core/admin/requirements-prod.txt @@ -15,12 +15,12 @@ decorator==5.1.0 distlib==0.3.1 distro==1.5.0 dnspython==2.1.0 -dominate==2.3.5 +dominate==2.6.0 email-validator==1.1.3 Flask==1.0.2 Flask-Babel==2.0.0 Flask-Bootstrap==3.3.7.1 -Flask-DebugToolbar==0.10.1 +Flask-DebugToolbar==0.11.0 Flask-Limiter==1.0.1 Flask-Login==0.5.0 flask-marshmallow==0.14.0 @@ -33,12 +33,12 @@ html5lib==1.1 idna==3.2 infinity==1.5 intervals==0.9.2 -itsdangerous==1.1.0 -Jinja2==2.11.3 -limits==1.3 +itsdangerous==2.0.1 +Jinja2==3.0.2 +limits==1.5.1 lockfile==0.12.2 -Mako==1.0.9 -MarkupSafe==1.1.1 +Mako==1.1.5 +MarkupSafe==2.0.1 marshmallow==3.10.0 marshmallow-sqlalchemy==0.24.1 msgpack==1.0.2 From ffd99c3fa8ee6c3ad590973c4c5ebdbe0172474d Mon Sep 17 00:00:00 2001 From: Alexander Graf Date: Wed, 3 Nov 2021 22:21:26 +0100 Subject: [PATCH 24/44] updated flask ConfigManager should not replace app.config - this is causing trouble with some other flask modules (swagger). Updated ConfigManager to only modify app.config and not replace it. --- core/admin/mailu/__init__.py | 2 +- core/admin/mailu/configuration.py | 48 +++++++++---------------------- core/admin/requirements-prod.txt | 2 +- 3 files changed, 16 insertions(+), 36 deletions(-) diff --git a/core/admin/mailu/__init__.py b/core/admin/mailu/__init__.py index b49bdd67..2f1b8469 100644 --- a/core/admin/mailu/__init__.py +++ b/core/admin/mailu/__init__.py @@ -33,7 +33,7 @@ def create_app_from_config(config): app.srs_key = hmac.new(bytearray(app.secret_key, 'utf-8'), bytearray('SRS_KEY', 'utf-8'), 'sha256').digest() # Initialize list of translations - config.translations = { + app.config.translations = { str(locale): locale for locale in sorted( utils.babel.list_translations(), diff --git a/core/admin/mailu/configuration.py b/core/admin/mailu/configuration.py index 9829f798..96127ee2 100644 --- a/core/admin/mailu/configuration.py +++ b/core/admin/mailu/configuration.py @@ -88,7 +88,7 @@ DEFAULT_CONFIG = { 'POD_ADDRESS_RANGE': None } -class ConfigManager(dict): +class ConfigManager: """ Naive configuration manager that uses environment only """ @@ -103,19 +103,16 @@ class ConfigManager(dict): def get_host_address(self, name): # if MYSERVICE_ADDRESS is defined, use this - if '{}_ADDRESS'.format(name) in os.environ: - return os.environ.get('{}_ADDRESS'.format(name)) + if f'{name}_ADDRESS' in os.environ: + return os.environ.get(f'{name}_ADDRESS') # otherwise use the host name and resolve it - return system.resolve_address(self.config['HOST_{}'.format(name)]) + return system.resolve_address(self.config[f'HOST_{name}']) def resolve_hosts(self): - self.config["IMAP_ADDRESS"] = self.get_host_address("IMAP") - self.config["POP3_ADDRESS"] = self.get_host_address("POP3") - self.config["AUTHSMTP_ADDRESS"] = self.get_host_address("AUTHSMTP") - self.config["SMTP_ADDRESS"] = self.get_host_address("SMTP") - self.config["REDIS_ADDRESS"] = self.get_host_address("REDIS") - if self.config["WEBMAIL"] != "none": - self.config["WEBMAIL_ADDRESS"] = self.get_host_address("WEBMAIL") + for key in ['IMAP', 'POP3', 'AUTHSMTP', 'SMTP', 'REDIS']: + self.config[f'{key}_ADDRESS'] = self.get_host_address(key) + if self.config['WEBMAIL'] != 'none': + self.config['WEBMAIL_ADDRESS'] = self.get_host_address('WEBMAIL') def __get_env(self, key, value): key_file = key + "_FILE" @@ -134,6 +131,7 @@ class ConfigManager(dict): return value def init_app(self, app): + # get current app config self.config.update(app.config) # get environment variables self.config.update({ @@ -147,9 +145,9 @@ class ConfigManager(dict): template = self.DB_TEMPLATES[self.config['DB_FLAVOR']] self.config['SQLALCHEMY_DATABASE_URI'] = template.format(**self.config) - self.config['RATELIMIT_STORAGE_URL'] = 'redis://{0}/2'.format(self.config['REDIS_ADDRESS']) - self.config['QUOTA_STORAGE_URL'] = 'redis://{0}/1'.format(self.config['REDIS_ADDRESS']) - self.config['SESSION_STORAGE_URL'] = 'redis://{0}/3'.format(self.config['REDIS_ADDRESS']) + self.config['RATELIMIT_STORAGE_URL'] = f'redis://{self.config["REDIS_ADDRESS"]}/2' + self.config['QUOTA_STORAGE_URL'] = f'redis://{self.config["REDIS_ADDRESS"]}/1' + self.config['SESSION_STORAGE_URL'] = f'redis://{self.config["REDIS_ADDRESS"]}/3' self.config['SESSION_COOKIE_SAMESITE'] = 'Strict' self.config['SESSION_COOKIE_HTTPONLY'] = True self.config['PERMANENT_SESSION_LIFETIME'] = timedelta(hours=int(self.config['SESSION_LIFETIME'])) @@ -157,25 +155,7 @@ class ConfigManager(dict): self.config['AUTH_RATELIMIT_EXEMPTION'] = set(ipaddress.ip_network(cidr, False) for cidr in (cidr.strip() for cidr in self.config['AUTH_RATELIMIT_EXEMPTION'].split(',')) if cidr) self.config['HOSTNAMES'] = ','.join(hostnames) self.config['HOSTNAME'] = hostnames[0] - # update the app config itself - app.config = self - def setdefault(self, key, value): - if key not in self.config: - self.config[key] = value - return self.config[key] + # update the app config + app.config.update(self.config) - def get(self, *args): - return self.config.get(*args) - - def keys(self): - return self.config.keys() - - def __getitem__(self, key): - return self.config.get(key) - - def __setitem__(self, key, value): - self.config[key] = value - - def __contains__(self, key): - return key in self.config diff --git a/core/admin/requirements-prod.txt b/core/admin/requirements-prod.txt index 282d00c3..ce7d4017 100644 --- a/core/admin/requirements-prod.txt +++ b/core/admin/requirements-prod.txt @@ -17,7 +17,7 @@ distro==1.5.0 dnspython==2.1.0 dominate==2.6.0 email-validator==1.1.3 -Flask==1.0.2 +Flask==2.0.2 Flask-Babel==2.0.0 Flask-Bootstrap==3.3.7.1 Flask-DebugToolbar==0.11.0 From 85d86d415624f2d3c335a1922527f92886d9fd3f Mon Sep 17 00:00:00 2001 From: Alexander Graf Date: Wed, 3 Nov 2021 22:55:26 +0100 Subject: [PATCH 25/44] some more libs updated --- core/admin/requirements-prod.txt | 28 ++++++++++++++-------------- 1 file changed, 14 insertions(+), 14 deletions(-) diff --git a/core/admin/requirements-prod.txt b/core/admin/requirements-prod.txt index ce7d4017..331239ba 100644 --- a/core/admin/requirements-prod.txt +++ b/core/admin/requirements-prod.txt @@ -3,17 +3,17 @@ appdirs==1.4.4 Babel==2.9.1 bcrypt==3.2.0 blinker==1.4 -CacheControl==0.12.6 -certifi==2020.12.5 +CacheControl==0.12.9 +certifi==2021.10.8 cffi==1.15.0 chardet==4.0.0 click==8.0.3 colorama==0.4.4 -contextlib2==0.6.0 +contextlib2==21.6.0 cryptography==35.0.0 decorator==5.1.0 -distlib==0.3.1 -distro==1.5.0 +# distlib==0.3.1 +# distro==1.5.0 dnspython==2.1.0 dominate==2.6.0 email-validator==1.1.3 @@ -21,7 +21,7 @@ Flask==2.0.2 Flask-Babel==2.0.0 Flask-Bootstrap==3.3.7.1 Flask-DebugToolbar==0.11.0 -Flask-Limiter==1.0.1 +Flask-Limiter==1.4 Flask-Login==0.5.0 flask-marshmallow==0.14.0 Flask-Migrate==3.1.0 @@ -30,7 +30,7 @@ Flask-SQLAlchemy==2.4.0 Flask-WTF==0.15.1 gunicorn==20.1.0 html5lib==1.1 -idna==3.2 +idna==3.3 infinity==1.5 intervals==0.9.2 itsdangerous==2.0.1 @@ -44,28 +44,28 @@ marshmallow-sqlalchemy==0.24.1 msgpack==1.0.2 mysqlclient==1.4.2.post1 ordered-set==4.0.2 -packaging==20.9 +# packaging==20.9 passlib==1.7.4 -pep517==0.10.0 -progress==1.5 +# pep517==0.10.0 +progress==1.6 psycopg2==2.8.2 pycparser==2.20 Pygments==2.10.0 pyOpenSSL==21.0.0 -pyparsing==2.4.7 +pyparsing==3.0.4 pytz==2021.3 PyYAML==6.0 redis==3.5.3 -requests==2.25.1 +requests==2.26.0 retrying==1.3.3 -six==1.15.0 +# six==1.15.0 socrate==0.2.0 SQLAlchemy==1.3.3 srslib==0.1.4 tabulate==0.8.9 tenacity==8.0.1 toml==0.10.2 -urllib3==1.26.5 +urllib3==1.26.7 validators==0.18.2 visitor==0.1.3 webencodings==0.5.1 From 4669374b9ec951c081777fe4284edc69cb4be0a2 Mon Sep 17 00:00:00 2001 From: Alexander Graf Date: Wed, 3 Nov 2021 22:55:41 +0100 Subject: [PATCH 26/44] use python wheels --- core/admin/Dockerfile | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/core/admin/Dockerfile b/core/admin/Dockerfile index 5c1227e5..cf911d82 100644 --- a/core/admin/Dockerfile +++ b/core/admin/Dockerfile @@ -25,7 +25,7 @@ COPY --from=balenalib/rpi-alpine:3.14 /usr/bin/qemu-arm-static /usr/bin/qemu-arm # python3 shared with most images RUN set -eu \ - && apk add --no-cache python3 py3-pip git bash \ + && apk add --no-cache python3 py3-pip py3-wheel git bash \ && pip3 install --upgrade pip RUN mkdir -p /app From 73ab4327c29b3e5f65e5c26b5f86418612daa972 Mon Sep 17 00:00:00 2001 From: Alexander Graf Date: Wed, 3 Nov 2021 22:57:07 +0100 Subject: [PATCH 27/44] updated database libraries (sqlalchemy etc.) this is working fine, but introduces a sqlalchemy warning when using config-import: /app/mailu/schemas.py:822: SAWarning: Identity map already had an identity for (...), replacing it with newly flushed object. Are there load operations occurring inside of an event handler within the flush? --- core/admin/mailu/models.py | 16 ++++++++-------- core/admin/mailu/schemas.py | 5 +++++ core/admin/requirements-prod.txt | 13 +++++++------ 3 files changed, 20 insertions(+), 14 deletions(-) diff --git a/core/admin/mailu/models.py b/core/admin/mailu/models.py index 300d8d8c..697e4df7 100644 --- a/core/admin/mailu/models.py +++ b/core/admin/mailu/models.py @@ -39,6 +39,8 @@ class IdnaDomain(db.TypeDecorator): """ impl = db.String(80) + cache_ok = True + python_type = str def process_bind_param(self, value, dialect): """ encode unicode domain name to punycode """ @@ -48,13 +50,13 @@ class IdnaDomain(db.TypeDecorator): """ decode punycode domain name to unicode """ return idna.decode(value) - python_type = str - class IdnaEmail(db.TypeDecorator): """ Stores a Unicode string in it's IDNA representation (ASCII only) """ impl = db.String(255) + cache_ok = True + python_type = str def process_bind_param(self, value, dialect): """ encode unicode domain part of email address to punycode """ @@ -70,13 +72,13 @@ class IdnaEmail(db.TypeDecorator): localpart, domain_name = value.rsplit('@', 1) return f'{localpart}@{idna.decode(domain_name)}' - python_type = str - class CommaSeparatedList(db.TypeDecorator): """ Stores a list as a comma-separated string, compatible with Postfix. """ impl = db.String + cache_ok = True + python_type = list def process_bind_param(self, value, dialect): """ join list of items to comma separated string """ @@ -91,13 +93,13 @@ class CommaSeparatedList(db.TypeDecorator): """ split comma separated string to list """ return list(filter(bool, (item.strip() for item in value.split(',')))) if value else [] - python_type = list - class JSONEncoded(db.TypeDecorator): """ Represents an immutable structure as a json-encoded string. """ impl = db.String + cache_ok = True + python_type = str def process_bind_param(self, value, dialect): """ encode data as json """ @@ -107,8 +109,6 @@ class JSONEncoded(db.TypeDecorator): """ decode json to data """ return json.loads(value) if value else None - python_type = str - class Base(db.Model): """ Base class for all models """ diff --git a/core/admin/mailu/schemas.py b/core/admin/mailu/schemas.py index 191d01ac..00cbf464 100644 --- a/core/admin/mailu/schemas.py +++ b/core/admin/mailu/schemas.py @@ -145,6 +145,11 @@ class Logger: if history.has_changes() and history.deleted: before = history.deleted[-1] after = getattr(target, attr.key) + # we don't have ordered lists + if isinstance(before, list): + before = set(before) + if isinstance(after, list): + after = set(after) # TODO: this can be removed when comment is not nullable in model if attr.key == 'comment' and not before and not after: pass diff --git a/core/admin/requirements-prod.txt b/core/admin/requirements-prod.txt index 331239ba..d6c7aca3 100644 --- a/core/admin/requirements-prod.txt +++ b/core/admin/requirements-prod.txt @@ -26,8 +26,9 @@ Flask-Login==0.5.0 flask-marshmallow==0.14.0 Flask-Migrate==3.1.0 Flask-Script==2.0.6 -Flask-SQLAlchemy==2.4.0 +Flask-SQLAlchemy==2.5.1 Flask-WTF==0.15.1 +greenlet==1.1.2 gunicorn==20.1.0 html5lib==1.1 idna==3.3 @@ -39,16 +40,16 @@ limits==1.5.1 lockfile==0.12.2 Mako==1.1.5 MarkupSafe==2.0.1 -marshmallow==3.10.0 -marshmallow-sqlalchemy==0.24.1 +marshmallow==3.14.0 +marshmallow-sqlalchemy==0.26.1 msgpack==1.0.2 -mysqlclient==1.4.2.post1 +mysqlclient==2.0.3 ordered-set==4.0.2 # packaging==20.9 passlib==1.7.4 # pep517==0.10.0 progress==1.6 -psycopg2==2.8.2 +psycopg2==2.9.1 pycparser==2.20 Pygments==2.10.0 pyOpenSSL==21.0.0 @@ -60,7 +61,7 @@ requests==2.26.0 retrying==1.3.3 # six==1.15.0 socrate==0.2.0 -SQLAlchemy==1.3.3 +SQLAlchemy==1.4.26 srslib==0.1.4 tabulate==0.8.9 tenacity==8.0.1 From 97e79a973f1ebd1e3d586cb318f0a7484cc4313e Mon Sep 17 00:00:00 2001 From: Alexander Graf Date: Thu, 4 Nov 2021 08:32:53 +0100 Subject: [PATCH 28/44] fix sso login button spacing again --- core/admin/mailu/sso/templates/form_sso.html | 2 +- core/admin/mailu/sso/views/base.py | 1 + core/admin/mailu/ui/templates/macros.html | 14 ++++++++------ 3 files changed, 10 insertions(+), 7 deletions(-) diff --git a/core/admin/mailu/sso/templates/form_sso.html b/core/admin/mailu/sso/templates/form_sso.html index b14e7600..d2451597 100644 --- a/core/admin/mailu/sso/templates/form_sso.html +++ b/core/admin/mailu/sso/templates/form_sso.html @@ -5,7 +5,7 @@
{{ macros.form_field(form.email) }} {{ macros.form_field(form.pw) }} - {{ macros.form_fields(fields, label=False, class="btn btn-default", spacing=False) }} + {{ macros.form_fields(fields, label=False, class="btn btn-default") }}
{%- endcall %} {%- endblock %} diff --git a/core/admin/mailu/sso/views/base.py b/core/admin/mailu/sso/views/base.py index fbee52a7..e948aeb7 100644 --- a/core/admin/mailu/sso/views/base.py +++ b/core/admin/mailu/sso/views/base.py @@ -19,6 +19,7 @@ def login(): fields.append(form.submitAdmin) if str(app.config["WEBMAIL"]).upper() != "NONE": fields.append(form.submitWebmail) + fields = [fields] if form.validate_on_submit(): if form.submitAdmin.data: diff --git a/core/admin/mailu/ui/templates/macros.html b/core/admin/mailu/ui/templates/macros.html index 75de4e0e..3b7e4c1d 100644 --- a/core/admin/mailu/ui/templates/macros.html +++ b/core/admin/mailu/ui/templates/macros.html @@ -18,17 +18,19 @@ {%- endif %} {%- endmacro %} -{%- macro form_fields(fields, prepend='', append='', label=True, spacing=True) %} - {%- if spacing %} +{%- macro form_fields(fields, prepend='', append='', label=True) %} {%- set width = (12 / fields|length)|int %} - {%- else %} - {%- set width = 2 %} - {% endif %}
{%- for field in fields %}
- {{ form_individual_field(field, prepend=prepend, append=append, label=label, **kwargs) }} + {%- if field is iterable %} + {%- for subfield in field %} + {{ form_individual_field(subfield, prepend=prepend, append=append, label=label, **kwargs) }} + {%- endfor %} + {%- else %} + {{ form_individual_field(field, prepend=prepend, append=append, label=label, **kwargs) }} + {%- endif %}
{%- endfor %}
From 5714b4f4b0326ee808d0fb4242a1d92ac913d4ae Mon Sep 17 00:00:00 2001 From: Florent Daigniere Date: Sat, 6 Nov 2021 10:05:52 +0100 Subject: [PATCH 29/44] introduce MESSAGE_RATELIMIT_EXEMPTION --- core/admin/mailu/configuration.py | 1 + core/admin/mailu/internal/views/postfix.py | 2 ++ docs/configuration.rst | 8 +++++--- 3 files changed, 8 insertions(+), 3 deletions(-) diff --git a/core/admin/mailu/configuration.py b/core/admin/mailu/configuration.py index 9829f798..d395073d 100644 --- a/core/admin/mailu/configuration.py +++ b/core/admin/mailu/configuration.py @@ -54,6 +54,7 @@ DEFAULT_CONFIG = { 'DKIM_PATH': '/dkim/{domain}.{selector}.key', 'DEFAULT_QUOTA': 1000000000, 'MESSAGE_RATELIMIT': '200/day', + 'MESSAGE_RATELIMIT_EXEMPTION': '', 'RECIPIENT_DELIMITER': '', # Web settings 'SITENAME': 'Mailu', diff --git a/core/admin/mailu/internal/views/postfix.py b/core/admin/mailu/internal/views/postfix.py index ab965967..2664f968 100644 --- a/core/admin/mailu/internal/views/postfix.py +++ b/core/admin/mailu/internal/views/postfix.py @@ -149,6 +149,8 @@ def postfix_sender_login(sender): def postfix_sender_rate(sender): """ Rate limit outbound emails per sender login """ + if sender in [s for s in flask.current_app.config.get('MESSAGE_RATELIMIT_EXEMPTION', '').lower().replace(' ', '').split(',') if s]: + flask.abort(404) user = models.User.get(sender) or flask.abort(404) return flask.abort(404) if user.sender_limiter.hit() else flask.jsonify("450 4.2.1 You are sending too many emails too fast.") diff --git a/docs/configuration.rst b/docs/configuration.rst index fa574415..39680fbd 100644 --- a/docs/configuration.rst +++ b/docs/configuration.rst @@ -69,9 +69,11 @@ The ``MESSAGE_SIZE_LIMIT`` is the maximum size of a single email. It should not be too low to avoid dropping legitimate emails and should not be too high to avoid filling the disks with large junk emails. -The ``MESSAGE_RATELIMIT`` is the limit of messages a single user can send. This is -meant to fight outbound spam in case of compromised or malicious account on the -server. +The ``MESSAGE_RATELIMIT`` (default: 200/day) is the maximum number of messages +a single user can send. ``MESSAGE_RATELIMIT_EXEMPTION`` contains a comma delimited +list of user email addresses that are exempted from any restriction. Those +settings are meant to reduce outbound spam in case of compromised or malicious +account on the server. The ``RELAYNETS`` (default: unset) is a comma delimited list of network addresses for which mail is relayed for with no authentication required. This should be From 6c6b0b161caa31c2ba4ce45e715d0324ebf22e41 Mon Sep 17 00:00:00 2001 From: Florent Daigniere Date: Sat, 6 Nov 2021 10:45:59 +0100 Subject: [PATCH 30/44] Set the right flags on the rate_limit cookie --- core/admin/mailu/sso/views/base.py | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/core/admin/mailu/sso/views/base.py b/core/admin/mailu/sso/views/base.py index fbee52a7..c11c588a 100644 --- a/core/admin/mailu/sso/views/base.py +++ b/core/admin/mailu/sso/views/base.py @@ -38,7 +38,7 @@ def login(): flask.session.regenerate() flask_login.login_user(user) response = flask.redirect(destination) - response.set_cookie('rate_limit', utils.limiter.device_cookie(username), max_age=31536000, path=flask.url_for('sso.login')) + response.set_cookie('rate_limit', utils.limiter.device_cookie(username), max_age=31536000, path=flask.url_for('sso.login'), secure=True, httponly=True) flask.current_app.logger.info(f'Login succeeded for {username} from {client_ip}.') return response else: From bbef4bee2763011af11a14568c4d85c54a73557e Mon Sep 17 00:00:00 2001 From: Florent Daigniere Date: Sun, 7 Nov 2021 12:20:31 +0100 Subject: [PATCH 31/44] Don't return any key for relayed domains We may want to revisit this (ARC signing)... but in the meantime it saves from a scary message in rspamd signing failure: cannot request data from the vault url: /internal/rspamd/vault/v1/dkim/ ... --- core/admin/mailu/internal/views/rspamd.py | 5 +++++ 1 file changed, 5 insertions(+) diff --git a/core/admin/mailu/internal/views/rspamd.py b/core/admin/mailu/internal/views/rspamd.py index 8551eb8f..123ec4e2 100644 --- a/core/admin/mailu/internal/views/rspamd.py +++ b/core/admin/mailu/internal/views/rspamd.py @@ -14,6 +14,11 @@ def vault_error(*messages, status=404): @internal.route("/rspamd/vault/v1/dkim/", methods=['GET']) def rspamd_dkim_key(domain_name): + models.Relay.query.get(domain_name) and return flask.jsonify({ + 'data': { + 'selectors': [] + } + }) domain = models.Domain.query.get(domain_name) or flask.abort(vault_error('unknown domain')) key = domain.dkim_key or flask.abort(vault_error('no dkim key', status=400)) return flask.jsonify({ From dc6e970a7f668a9030aff44100682b3a17dc6346 Mon Sep 17 00:00:00 2001 From: Florent Daigniere Date: Sun, 7 Nov 2021 12:41:29 +0100 Subject: [PATCH 32/44] handle HTTP too --- core/admin/mailu/sso/views/base.py | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/core/admin/mailu/sso/views/base.py b/core/admin/mailu/sso/views/base.py index c11c588a..831949e7 100644 --- a/core/admin/mailu/sso/views/base.py +++ b/core/admin/mailu/sso/views/base.py @@ -38,7 +38,7 @@ def login(): flask.session.regenerate() flask_login.login_user(user) response = flask.redirect(destination) - response.set_cookie('rate_limit', utils.limiter.device_cookie(username), max_age=31536000, path=flask.url_for('sso.login'), secure=True, httponly=True) + response.set_cookie('rate_limit', utils.limiter.device_cookie(username), max_age=31536000, path=flask.url_for('sso.login'), secure=app.config['SESSION_COOKIE_SECURE'], httponly=True) flask.current_app.logger.info(f'Login succeeded for {username} from {client_ip}.') return response else: From b68033eb43fd249a9092df3f536ffa551de156a5 Mon Sep 17 00:00:00 2001 From: Florent Daigniere Date: Mon, 8 Nov 2021 09:23:24 +0100 Subject: [PATCH 33/44] only parse it once --- core/admin/mailu/configuration.py | 1 + core/admin/mailu/internal/views/postfix.py | 2 +- 2 files changed, 2 insertions(+), 1 deletion(-) diff --git a/core/admin/mailu/configuration.py b/core/admin/mailu/configuration.py index d395073d..a997a8c7 100644 --- a/core/admin/mailu/configuration.py +++ b/core/admin/mailu/configuration.py @@ -156,6 +156,7 @@ class ConfigManager(dict): self.config['PERMANENT_SESSION_LIFETIME'] = timedelta(hours=int(self.config['SESSION_LIFETIME'])) hostnames = [host.strip() for host in self.config['HOSTNAMES'].split(',')] self.config['AUTH_RATELIMIT_EXEMPTION'] = set(ipaddress.ip_network(cidr, False) for cidr in (cidr.strip() for cidr in self.config['AUTH_RATELIMIT_EXEMPTION'].split(',')) if cidr) + self.config['MESSAGE_RATELIMIT_EXEMPTION'] = set([s for s in self.config['MESSAGE_RATELIMIT_EXEMPTION'].lower().replace(' ', '').split(',') if s]) self.config['HOSTNAMES'] = ','.join(hostnames) self.config['HOSTNAME'] = hostnames[0] # update the app config itself diff --git a/core/admin/mailu/internal/views/postfix.py b/core/admin/mailu/internal/views/postfix.py index 2664f968..ed951943 100644 --- a/core/admin/mailu/internal/views/postfix.py +++ b/core/admin/mailu/internal/views/postfix.py @@ -149,7 +149,7 @@ def postfix_sender_login(sender): def postfix_sender_rate(sender): """ Rate limit outbound emails per sender login """ - if sender in [s for s in flask.current_app.config.get('MESSAGE_RATELIMIT_EXEMPTION', '').lower().replace(' ', '').split(',') if s]: + if sender in flask.current_app.config['MESSAGE_RATELIMIT_EXEMPTION']: flask.abort(404) user = models.User.get(sender) or flask.abort(404) return flask.abort(404) if user.sender_limiter.hit() else flask.jsonify("450 4.2.1 You are sending too many emails too fast.") From 6bf1a178b94bbfe17da0f0073ab4553ada399c01 Mon Sep 17 00:00:00 2001 From: Florent Daigniere Date: Mon, 8 Nov 2021 09:34:02 +0100 Subject: [PATCH 34/44] Go with ghostwheel42's suggestion --- core/admin/mailu/internal/views/rspamd.py | 20 ++++++-------------- 1 file changed, 6 insertions(+), 14 deletions(-) diff --git a/core/admin/mailu/internal/views/rspamd.py b/core/admin/mailu/internal/views/rspamd.py index 123ec4e2..458dbb81 100644 --- a/core/admin/mailu/internal/views/rspamd.py +++ b/core/admin/mailu/internal/views/rspamd.py @@ -14,22 +14,14 @@ def vault_error(*messages, status=404): @internal.route("/rspamd/vault/v1/dkim/", methods=['GET']) def rspamd_dkim_key(domain_name): - models.Relay.query.get(domain_name) and return flask.jsonify({ - 'data': { - 'selectors': [] - } - }) - domain = models.Domain.query.get(domain_name) or flask.abort(vault_error('unknown domain')) - key = domain.dkim_key or flask.abort(vault_error('no dkim key', status=400)) - return flask.jsonify({ - 'data': { - 'selectors': [ + selectors = [] + if domain := models.Domain.query.get(domain_name): + if key := domain.dkim_key: + selectors.append( { 'domain' : domain.name, 'key' : key.decode('utf8'), 'selector': flask.current_app.config.get('DKIM_SELECTOR', 'dkim'), } - ] - } - }) - + ) + return flask.jsonify({'data': {'selectors': selectors}}) From c48e00ee2664a8d74d9797706d465029cd358605 Mon Sep 17 00:00:00 2001 From: Till Skrodzki Date: Tue, 9 Nov 2021 12:22:53 +0100 Subject: [PATCH 35/44] Do not call .split() on RELAYNETS if not specified --- core/postfix/conf/main.cf | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/core/postfix/conf/main.cf b/core/postfix/conf/main.cf index 5fffc330..8ac646da 100644 --- a/core/postfix/conf/main.cf +++ b/core/postfix/conf/main.cf @@ -17,7 +17,7 @@ queue_directory = /queue message_size_limit = {{ MESSAGE_SIZE_LIMIT }} # Relayed networks -mynetworks = 127.0.0.1/32 [::1]/128 {{ SUBNET }} {{ RELAYNETS.split(",") | join(' ') }} +mynetworks = 127.0.0.1/32 [::1]/128 {{ SUBNET }} {% if RELAYNETS %}{{ RELAYNETS.split(",") | join(' ') }}{% endif %} # Empty alias list to override the configuration variable and disable NIS alias_maps = From 2404cf2e3ddf0456fe2cc1a399d6b1738c0798ad Mon Sep 17 00:00:00 2001 From: Dimitri Huisman Date: Tue, 9 Nov 2021 14:10:04 +0000 Subject: [PATCH 36/44] Fix for issue #1223 --- optional/fetchmail/Dockerfile | 3 +++ optional/fetchmail/fetchmail.py | 1 + setup/flavors/compose/docker-compose.yml | 2 ++ setup/flavors/stack/docker-compose.yml | 2 +- towncrier/newsfragments/1223.bugfix | 4 ++++ 5 files changed, 11 insertions(+), 1 deletion(-) create mode 100644 towncrier/newsfragments/1223.bugfix diff --git a/optional/fetchmail/Dockerfile b/optional/fetchmail/Dockerfile index 995ec48f..0b3d4375 100644 --- a/optional/fetchmail/Dockerfile +++ b/optional/fetchmail/Dockerfile @@ -12,6 +12,9 @@ RUN apk add --no-cache \ RUN apk add --no-cache fetchmail ca-certificates openssl \ && pip3 install requests +RUN mkdir -p /data +RUN chown fetchmail:fetchmail /data + COPY fetchmail.py /fetchmail.py USER fetchmail diff --git a/optional/fetchmail/fetchmail.py b/optional/fetchmail/fetchmail.py index 4be3c2bd..3752c395 100755 --- a/optional/fetchmail/fetchmail.py +++ b/optional/fetchmail/fetchmail.py @@ -13,6 +13,7 @@ import traceback FETCHMAIL = """ fetchmail -N \ + --idfile /data/.fetchids \ --sslcertck --sslcertpath /etc/ssl/certs \ -f {} """ diff --git a/setup/flavors/compose/docker-compose.yml b/setup/flavors/compose/docker-compose.yml index 2675a2ab..5cdd6129 100644 --- a/setup/flavors/compose/docker-compose.yml +++ b/setup/flavors/compose/docker-compose.yml @@ -129,6 +129,8 @@ services: image: ${DOCKER_ORG:-mailu}/${DOCKER_PREFIX:-}fetchmail:${MAILU_VERSION:-{{ version }}} restart: always env_file: {{ env }} + volumes: + - "{{ root }}/fetchmail:/data" {% if resolver_enabled %} depends_on: - resolver diff --git a/setup/flavors/stack/docker-compose.yml b/setup/flavors/stack/docker-compose.yml index 24afa9f3..ae4f0022 100644 --- a/setup/flavors/stack/docker-compose.yml +++ b/setup/flavors/stack/docker-compose.yml @@ -110,7 +110,7 @@ services: image: ${DOCKER_ORG:-mailu}/${DOCKER_PREFIX:-}fetchmail:${MAILU_VERSION:-{{ version }}} env_file: {{ env }} volumes: - - "{{ root }}/data:/data" + - "{{ root }}/fetchmail:/data" deploy: replicas: 1 healthcheck: diff --git a/towncrier/newsfragments/1223.bugfix b/towncrier/newsfragments/1223.bugfix new file mode 100644 index 00000000..3c23d1a4 --- /dev/null +++ b/towncrier/newsfragments/1223.bugfix @@ -0,0 +1,4 @@ +Fixed fetchmail losing track of fetched emails upon container recreation. +The relevant fetchmail files are now retained in the /data folder (in the fetchmail image). +See the docker-compose.yml file for the relevant volume mapping. +If you already had your own mapping, you must double check the volume mapping and take action. From 92e65b33e03ba0bc881db28a7412489c572b732e Mon Sep 17 00:00:00 2001 From: Dimitri Huisman Date: Wed, 10 Nov 2021 09:07:03 +0000 Subject: [PATCH 37/44] Configure fetchmail to use idfile to keep track of messages. Run fetchmail as root. This is unfortunately required because all files are owned by root in the mailu data folder. In the future we must switch all images to running all all processes with a non-root user. --- optional/fetchmail/Dockerfile | 5 +---- optional/fetchmail/fetchmail.py | 2 +- 2 files changed, 2 insertions(+), 5 deletions(-) diff --git a/optional/fetchmail/Dockerfile b/optional/fetchmail/Dockerfile index 0b3d4375..068a5dce 100644 --- a/optional/fetchmail/Dockerfile +++ b/optional/fetchmail/Dockerfile @@ -13,10 +13,7 @@ RUN apk add --no-cache fetchmail ca-certificates openssl \ && pip3 install requests RUN mkdir -p /data -RUN chown fetchmail:fetchmail /data COPY fetchmail.py /fetchmail.py -USER fetchmail - -CMD ["/fetchmail.py"] +CMD ["/fetchmail.py"] \ No newline at end of file diff --git a/optional/fetchmail/fetchmail.py b/optional/fetchmail/fetchmail.py index 3752c395..f3b65fc0 100755 --- a/optional/fetchmail/fetchmail.py +++ b/optional/fetchmail/fetchmail.py @@ -13,7 +13,7 @@ import traceback FETCHMAIL = """ fetchmail -N \ - --idfile /data/.fetchids \ + --idfile /data/.fetchids --uidl \ --sslcertck --sslcertpath /etc/ssl/certs \ -f {} """ From c81aa67dfa9c8d01cb9d55a7bd8c216518d74f98 Mon Sep 17 00:00:00 2001 From: Dimitri Huisman Date: Wed, 10 Nov 2021 13:08:33 +0000 Subject: [PATCH 38/44] Use a better location for storing the fetchmail data. --- setup/flavors/compose/docker-compose.yml | 2 +- setup/flavors/stack/docker-compose.yml | 2 +- 2 files changed, 2 insertions(+), 2 deletions(-) diff --git a/setup/flavors/compose/docker-compose.yml b/setup/flavors/compose/docker-compose.yml index 5cdd6129..18a881b8 100644 --- a/setup/flavors/compose/docker-compose.yml +++ b/setup/flavors/compose/docker-compose.yml @@ -130,7 +130,7 @@ services: restart: always env_file: {{ env }} volumes: - - "{{ root }}/fetchmail:/data" + - "{{ root }}/data/fetchmail:/data" {% if resolver_enabled %} depends_on: - resolver diff --git a/setup/flavors/stack/docker-compose.yml b/setup/flavors/stack/docker-compose.yml index ae4f0022..0c744d7e 100644 --- a/setup/flavors/stack/docker-compose.yml +++ b/setup/flavors/stack/docker-compose.yml @@ -110,7 +110,7 @@ services: image: ${DOCKER_ORG:-mailu}/${DOCKER_PREFIX:-}fetchmail:${MAILU_VERSION:-{{ version }}} env_file: {{ env }} volumes: - - "{{ root }}/fetchmail:/data" + - "{{ root }}/data/fetchmail:/data" deploy: replicas: 1 healthcheck: From fd5bdc86501562645d540e44358a787a55ebe542 Mon Sep 17 00:00:00 2001 From: Alexander Graf Date: Thu, 11 Nov 2021 12:20:52 +0100 Subject: [PATCH 39/44] added localized date output --- core/admin/mailu/__init__.py | 6 +++++- core/admin/mailu/ui/templates/alias/list.html | 4 ++-- core/admin/mailu/ui/templates/alternative/list.html | 4 +++- core/admin/mailu/ui/templates/fetch/list.html | 6 +++--- core/admin/mailu/ui/templates/relay/list.html | 4 ++-- core/admin/mailu/ui/templates/token/list.html | 4 +++- core/admin/mailu/ui/templates/user/list.html | 4 ++-- 7 files changed, 20 insertions(+), 12 deletions(-) diff --git a/core/admin/mailu/__init__.py b/core/admin/mailu/__init__.py index 2f1b8469..fe1f376c 100644 --- a/core/admin/mailu/__init__.py +++ b/core/admin/mailu/__init__.py @@ -57,11 +57,15 @@ def create_app_from_config(config): config = app.config, ) - # Jinja filter + # Jinja filters @app.template_filter() def format_date(value): return utils.flask_babel.format_date(value) if value else '' + @app.template_filter() + def format_datetime(value): + return utils.flask_babel.format_datetime(value) if value else '' + # Import views from mailu import ui, internal, sso app.register_blueprint(ui.ui, url_prefix=app.config['WEB_ADMIN']) diff --git a/core/admin/mailu/ui/templates/alias/list.html b/core/admin/mailu/ui/templates/alias/list.html index 0b784d52..6b52165e 100644 --- a/core/admin/mailu/ui/templates/alias/list.html +++ b/core/admin/mailu/ui/templates/alias/list.html @@ -34,8 +34,8 @@ {{ alias }} {{ alias.destination|join(', ') or '-' }} {{ alias.comment or '' }} - {{ alias.created_at }} - {{ alias.updated_at or '' }} + {{ alias.created_at | format_date }} + {{ alias.updated_at | format_date }} {%- endfor %} diff --git a/core/admin/mailu/ui/templates/alternative/list.html b/core/admin/mailu/ui/templates/alternative/list.html index b56cd751..4ca9f3c8 100644 --- a/core/admin/mailu/ui/templates/alternative/list.html +++ b/core/admin/mailu/ui/templates/alternative/list.html @@ -19,6 +19,7 @@ {% trans %}Actions{% endtrans %} {% trans %}Name{% endtrans %} {% trans %}Created{% endtrans %} + {% trans %}Last edit{% endtrans %} @@ -28,7 +29,8 @@ {{ alternative }} - {{ alternative.created_at }} + {{ alternative.created_at | format_date }} + {{ alternative.updated_at | format_date }} {%- endfor %} diff --git a/core/admin/mailu/ui/templates/fetch/list.html b/core/admin/mailu/ui/templates/fetch/list.html index d9374fc6..60b214de 100644 --- a/core/admin/mailu/ui/templates/fetch/list.html +++ b/core/admin/mailu/ui/templates/fetch/list.html @@ -36,10 +36,10 @@ {{ fetch.protocol }}{{ 's' if fetch.tls else '' }}://{{ fetch.host }}:{{ fetch.port }} {{ fetch.username }} {% if fetch.keep %}{% trans %}yes{% endtrans %}{% else %}{% trans %}no{% endtrans %}{% endif %} - {{ fetch.last_check or '-' }} + {{ fetch.last_check | format_datetime or '-' }} {{ fetch.error or '-' }} - {{ fetch.created_at }} - {{ fetch.updated_at or '' }} + {{ fetch.created_at | format_date }} + {{ fetch.updated_at | format_date }} {%- endfor %} diff --git a/core/admin/mailu/ui/templates/relay/list.html b/core/admin/mailu/ui/templates/relay/list.html index 07838273..80bc5338 100644 --- a/core/admin/mailu/ui/templates/relay/list.html +++ b/core/admin/mailu/ui/templates/relay/list.html @@ -32,8 +32,8 @@ {{ relay.name }} {{ relay.smtp or '-' }} {{ relay.comment or '' }} - {{ relay.created_at }} - {{ relay.updated_at or '' }} + {{ relay.created_at | format_date }} + {{ relay.updated_at | format_date }} {%- endfor %} diff --git a/core/admin/mailu/ui/templates/token/list.html b/core/admin/mailu/ui/templates/token/list.html index c3cc9b5c..d7c48737 100644 --- a/core/admin/mailu/ui/templates/token/list.html +++ b/core/admin/mailu/ui/templates/token/list.html @@ -20,6 +20,7 @@ {% trans %}Comment{% endtrans %} {% trans %}Authorized IP{% endtrans %} {% trans %}Created{% endtrans %} + {% trans %}Last edit{% endtrans %} @@ -30,7 +31,8 @@ {{ token.comment }} {{ token.ip or "any" }} - {{ token.created_at }} + {{ token.created_at | format_date }} + {{ token.updated_at | format_date }} {%- endfor %} diff --git a/core/admin/mailu/ui/templates/user/list.html b/core/admin/mailu/ui/templates/user/list.html index 59a06ea7..7faddab5 100644 --- a/core/admin/mailu/ui/templates/user/list.html +++ b/core/admin/mailu/ui/templates/user/list.html @@ -45,8 +45,8 @@ {{ user.quota_bytes_used | filesizeformat }} / {{ (user.quota_bytes | filesizeformat) if user.quota_bytes else '∞' }} {{ user.comment or '-' }} - {{ user.created_at }} - {{ user.updated_at or '' }} + {{ user.created_at | format_date }} + {{ user.updated_at | format_date }} {%- endfor %} From cf7914d050687d72606a0157cdf73c7fab785f56 Mon Sep 17 00:00:00 2001 From: Alexander Graf Date: Thu, 11 Nov 2021 16:00:00 +0100 Subject: [PATCH 40/44] fixed field iteration --- core/admin/mailu/ui/templates/macros.html | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/core/admin/mailu/ui/templates/macros.html b/core/admin/mailu/ui/templates/macros.html index 3b7e4c1d..0da069a9 100644 --- a/core/admin/mailu/ui/templates/macros.html +++ b/core/admin/mailu/ui/templates/macros.html @@ -24,7 +24,7 @@
{%- for field in fields %}
- {%- if field is iterable %} + {%- if field.__class__.__name__ == 'list' %} {%- for subfield in field %} {{ form_individual_field(subfield, prepend=prepend, append=append, label=label, **kwargs) }} {%- endfor %} From 84a5514a97c564c23faa36714e33750899fdf28a Mon Sep 17 00:00:00 2001 From: Alexander Graf Date: Fri, 12 Nov 2021 12:19:45 +0100 Subject: [PATCH 41/44] fixed auto reply form --- core/admin/assets/app.js | 4 ++-- 1 file changed, 2 insertions(+), 2 deletions(-) diff --git a/core/admin/assets/app.js b/core/admin/assets/app.js index 0926578d..03ea6215 100644 --- a/core/admin/assets/app.js +++ b/core/admin/assets/app.js @@ -28,10 +28,10 @@ $('document').ready(function() { var fieldset = $(this).parents('fieldset'); if (this.checked) { fieldset.removeAttr('disabled'); - fieldset.find('input').not(this).removeAttr('disabled'); + fieldset.find('input,textarea').not(this).removeAttr('disabled'); } else { fieldset.attr('disabled', ''); - fieldset.find('input').not(this).attr('disabled', ''); + fieldset.find('input,textarea').not(this).attr('disabled', ''); } }); From c3dd7330cb9f4b90bc0897fe48556761f5c1c66d Mon Sep 17 00:00:00 2001 From: Dimitri Huisman Date: Fri, 12 Nov 2021 13:30:31 +0000 Subject: [PATCH 42/44] Update reverse proxy documentation (see #1962). --- docs/configuration.rst | 4 +- docs/reverse.rst | 95 +++++++++++++---------------- towncrier/newsfragments/1962.bugfix | 4 ++ 3 files changed, 51 insertions(+), 52 deletions(-) create mode 100644 towncrier/newsfragments/1962.bugfix diff --git a/docs/configuration.rst b/docs/configuration.rst index ccf49058..8f363a63 100644 --- a/docs/configuration.rst +++ b/docs/configuration.rst @@ -194,7 +194,9 @@ The ``LETSENCRYPT_SHORTCHAIN`` (default: False) setting controls whether we send .. _`android handsets older than 7.1.1`: https://community.letsencrypt.org/t/production-chain-changes/150739 -The ``REAL_IP_HEADER`` (default: unset) and ``REAL_IP_FROM`` (default: unset) settings controls whether HTTP headers such as ``X-Forwarded-For`` or ``X-Real-IP`` should be trusted. The former should be the name of the HTTP header to extract the client IP address from and the later a comma separated list of IP addresses designing which proxies to trust. If you are using Mailu behind a reverse proxy, you should set both. Setting the former without the later introduces a security vulnerability allowing a potential attacker to spoof his source address. +.. _reverse_proxy_headers: + +The ``REAL_IP_HEADER`` (default: unset) and ``REAL_IP_FROM`` (default: unset) settings controls whether HTTP headers such as ``X-Forwarded-For`` or ``X-Real-IP`` should be trusted. The former should be the name of the HTTP header to extract the client IP address from and the later a comma separated list of IP addresses designating which proxies to trust. If you are using Mailu behind a reverse proxy, you should set both. Setting the former without the later introduces a security vulnerability allowing a potential attacker to spoof his source address. The ``TZ`` sets the timezone Mailu will use. The timezone naming convention usually uses a ``Region/City`` format. See `TZ database name`_ for a list of valid timezones This defaults to ``Etc/UTC``. Warning: if you are observing different timestamps in your log files you should change your hosts timezone to UTC instead of changing TZ to your local timezone. Using UTC allows easy log correlation with remote MTAs. diff --git a/docs/reverse.rst b/docs/reverse.rst index c6b98e4a..6cde55f2 100644 --- a/docs/reverse.rst +++ b/docs/reverse.rst @@ -11,7 +11,10 @@ There are basically three options, from the most to the least recommended one: - `use Traefik in another container as central system-reverse-proxy`_ - `override Mailu Web frontend configuration`_ -All options will require that you modify the ``docker-compose.yml`` file. +All options will require that you modify the ``docker-compose.yml`` and ``mailu.env`` file. + +Mailu must also be configured with the information what header is used by the reverse proxy for passing the remote client IP. +This is configured in the mailu.env file. See the :ref:`configuration reference ` for more information. Have Mailu Web frontend listen locally -------------------------------------- @@ -43,10 +46,19 @@ Then on your own frontend, point to these local ports. In practice, you only nee # [...] here goes your standard configuration location / { + proxy_set_header Host $host; + proxy_set_header X-Real-IP $remote_addr proxy_pass https://localhost:8443; } } +.. code-block:: docker + + #mailu.env file + REAL_IP_HEADER=X-Real-IP + REAL_IP_FROM=x.x.x.x,y.y.y.y.y + #x.x.x.x,y.y.y.y.y is the static IP address your reverse proxy uses for connecting to Mailu. + Because the admin interface is served as ``/admin``, the Webmail as ``/webmail``, the single sign on page as ``/sso``, webdav as ``/webdav`` and the static files endpoint as ``/static``, you may also want to use a single virtual host and serve other applications (still Nginx): .. code-block:: nginx @@ -55,8 +67,9 @@ Because the admin interface is served as ``/admin``, the Webmail as ``/webmail`` # [...] here goes your standard configuration location ~ ^/(admin|sso|static|webdav|webmail)/ { - proxy_pass https://localhost:8443; - proxy_set_header Host $http_host; + proxy_set_header Host $host; + proxy_set_header X-Real-IP $remote_addr + proxy_pass https://localhost:8443; } location /main_app { @@ -76,6 +89,13 @@ Because the admin interface is served as ``/admin``, the Webmail as ``/webmail`` } } +.. code-block:: docker + + #mailu.env file + REAL_IP_HEADER=X-Real-IP + REAL_IP_FROM=x.x.x.x,y.y.y.y.y + #x.x.x.x,y.y.y.y.y is the static IP address your reverse proxy uses for connecting to Mailu. + Finally, you might want to serve the admin interface on a separate virtual host but not expose the admin container directly (have your own HTTPS virtual hosts on top of Mailu, one public for the Webmail and one internal for administration for instance). Here is an example configuration : @@ -88,6 +108,8 @@ Here is an example configuration : # [...] here goes your standard configuration location /webmail { + proxy_set_header Host $host; + proxy_set_header X-Real-IP $remote_addr proxy_pass https://localhost:8443/webmail; } } @@ -98,12 +120,21 @@ Here is an example configuration : # [...] here goes your standard configuration location /admin { + proxy_set_header Host $host; + proxy_set_header X-Real-IP $remote_addr proxy_pass https://localhost:8443/admin; proxy_set_header Host $http_host; } } +.. code-block:: docker + + #mailu.env file + REAL_IP_HEADER=X-Real-IP + REAL_IP_FROM=x.x.x.x,y.y.y.y.y + #x.x.x.x,y.y.y.y.y is the static IP address your reverse proxy uses for connecting to Mailu. + Depending on how you access the front server, you might want to add a ``proxy_redirect`` directive to your ``location`` blocks: .. code-block:: nginx @@ -151,7 +182,16 @@ If your Traefik is configured to automatically request certificates from *letsen and this is the ``DOMAIN`` in your ``.env``? To support that use-case, Traefik can request ``SANs`` for your domain. The configuration for this will depend on your Traefik version. ----- +Mailu must also be configured with the information what header is used by the reverse proxy for passing the remote client IP. This is configured in mailu.env: + +.. code-block:: docker + + #mailu.env file + REAL_IP_HEADER=X-Real-IP + REAL_IP_FROM=x.x.x.x,y.y.y.y.y + #x.x.x.x,y.y.y.y.y is the static IP address your reverse proxy uses for connecting to Mailu. + +For more information see the :ref:`configuration reference ` for more information. Traefik 2.x using labels configuration ~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~ @@ -179,53 +219,6 @@ Of course, be sure to define the Certificate Resolver ``foo`` in the static conf Alternatively, you can define SANs in the Traefik static configuration using routers, or in the static configuration using entrypoints. Refer to the Traefik documentation for more details. -Traefik 1.x with TOML configuration -~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~ - -Lets add something like - -.. code-block:: yaml - - [acme] - [[acme.domains]] - main = "your.example.com" # this is the same as $TRAEFIK_DOMAIN! - sans = ["mail.your.example.com", "webmail.your.example.com", "smtp.your.example.com"] - -to your ``traefik.toml``. - ----- - -You might need to clear your ``acme.json``, if a certificate for one of these domains already exists. - -You will need some solution which dumps the certificates in ``acme.json``, so you can include them in the ``mailu/front`` container. -One such example is ``mailu/traefik-certdumper``, which has been adapted for use in Mailu. You can add it to your ``docker-compose.yml`` like: - -.. code-block:: yaml - - certdumper: - restart: always - image: mailu/traefik-certdumper:$VERSION - environment: - # Make sure this is the same as the main=-domain in traefik.toml - # !!! Also don’t forget to add "TRAEFIK_DOMAIN=[...]" to your .env! - - DOMAIN=$TRAEFIK_DOMAIN - volumes: - # Folder, which contains the acme.json - - "/data/traefik:/traefik" - # Folder, where cert.pem and key.pem will be written - - "/data/mailu/certs:/output" - - -Assuming you have ``volume-mounted`` your ``acme.json`` put to ``/data/traefik`` on your host. The dumper will then write out ``/data/mailu/certs/cert.pem`` and ``/data/mailu/certs/key.pem`` whenever ``acme.json`` is updated. -Yay! Now let’s mount this to our ``front`` container like: - -.. code-block:: yaml - - volumes: - - /data/mailu/certs:/certs - -This works, because we set ``TLS_FLAVOR=mail``, which picks up the key-certificate pair (e.g., ``cert.pem`` and ``key.pem``) from the certs folder in the root path (``/certs/``). - .. _`Traefik`: https://traefik.io/ Override Mailu configuration diff --git a/towncrier/newsfragments/1962.bugfix b/towncrier/newsfragments/1962.bugfix new file mode 100644 index 00000000..423271a9 --- /dev/null +++ b/towncrier/newsfragments/1962.bugfix @@ -0,0 +1,4 @@ +Reverse proxy documentation has been updated to reflect new security hardening from PR#1959. +If you do not set the required header, Mailu will not have access to the real ip address of the connecting client. +This means that rate limiting will not properly work. You can also not use fail2ban. +It is very important to set this header when using a reverse proxy. \ No newline at end of file From 5911ee605658328bbba375ecfab17bcd63bce54e Mon Sep 17 00:00:00 2001 From: Dimitri Huisman Date: Sat, 13 Nov 2021 14:35:23 +0000 Subject: [PATCH 43/44] Reworded changelog that it is very important to set the new configuration parameters --- towncrier/newsfragments/1962.bugfix | 5 +++-- 1 file changed, 3 insertions(+), 2 deletions(-) diff --git a/towncrier/newsfragments/1962.bugfix b/towncrier/newsfragments/1962.bugfix index 423271a9..70b2aa72 100644 --- a/towncrier/newsfragments/1962.bugfix +++ b/towncrier/newsfragments/1962.bugfix @@ -1,4 +1,5 @@ Reverse proxy documentation has been updated to reflect new security hardening from PR#1959. -If you do not set the required header, Mailu will not have access to the real ip address of the connecting client. +If you do not set the configuration parameters in Mailu what reverse proxy header to trust, +then Mailu will not have access to the real ip address of the connecting client. This means that rate limiting will not properly work. You can also not use fail2ban. -It is very important to set this header when using a reverse proxy. \ No newline at end of file +It is very important to configure this when using a reverse proxy. From d7d02152bbc55f1d26dca7c245d3224786a2ed4a Mon Sep 17 00:00:00 2001 From: Dimitri Huisman Date: Sat, 13 Nov 2021 14:40:22 +0000 Subject: [PATCH 44/44] Make fetchid file not hidden. --- optional/fetchmail/fetchmail.py | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/optional/fetchmail/fetchmail.py b/optional/fetchmail/fetchmail.py index f3b65fc0..5459de59 100755 --- a/optional/fetchmail/fetchmail.py +++ b/optional/fetchmail/fetchmail.py @@ -13,7 +13,7 @@ import traceback FETCHMAIL = """ fetchmail -N \ - --idfile /data/.fetchids --uidl \ + --idfile /data/fetchids --uidl \ --sslcertck --sslcertpath /etc/ssl/certs \ -f {} """