1961: Implement MTA-STS and DANE validation r=mergify[bot] a=nextgens

## What type of PR?

Feature

## What does this PR do?

Implement MTA-STS: the tls_policy_map will now be auto-configured based on the policies published by the various domains. A FAQ entry has been added to document how to publish a policy using Mailu.

As configured by default there is no persistence. If we want persistence we can have either sqlite3 (with a db in the mailqueue) or redis...

This also introduces a DEFER_ON_TLS_ERROR (default: True) setting that will harden policy enforcement and defer emails that shouldn't be delivered. Turn it off if you never want to set an override.

### Related issue(s)
- closes #1798
- closes #707 

## Prerequistes
Before we can consider review and merge, please make sure the following list is done and checked.
If an entry in not applicable, you can check it or remove it from the list.

- [x] In case of feature or enhancement: documentation updated accordingly
- [x] Unless it's docs or a minor change: add [changelog](https://mailu.io/master/contributors/workflow.html#changelog) entry file.


Co-authored-by: Florent Daigniere <nextgens@freenetproject.org>
master
bors[bot] 3 years ago committed by GitHub
commit e38844cfcd
No known key found for this signature in database
GPG Key ID: 4AEE18F83AFDEB23

@ -22,7 +22,7 @@ Main features include:
- **Web access**, multiple Webmails and administration interface - **Web access**, multiple Webmails and administration interface
- **User features**, aliases, auto-reply, auto-forward, fetched accounts - **User features**, aliases, auto-reply, auto-forward, fetched accounts
- **Admin features**, global admins, announcements, per-domain delegation, quotas - **Admin features**, global admins, announcements, per-domain delegation, quotas
- **Security**, enforced TLS, Letsencrypt!, outgoing DKIM, anti-virus scanner - **Security**, enforced TLS, DANE, MTA-STS, Letsencrypt!, outgoing DKIM, anti-virus scanner
- **Antispam**, auto-learn, greylisting, DMARC and SPF - **Antispam**, auto-learn, greylisting, DMARC and SPF
- **Freedom**, all FOSS components, no tracker included - **Freedom**, all FOSS components, no tracker included

@ -35,6 +35,7 @@ DEFAULT_CONFIG = {
'WILDCARD_SENDERS': '', 'WILDCARD_SENDERS': '',
'TLS_FLAVOR': 'cert', 'TLS_FLAVOR': 'cert',
'INBOUND_TLS_ENFORCE': False, 'INBOUND_TLS_ENFORCE': False,
'DEFER_ON_TLS_ERROR': True,
'AUTH_RATELIMIT': '1000/minute;10000/hour', 'AUTH_RATELIMIT': '1000/minute;10000/hour',
'AUTH_RATELIMIT_SUBNET': False, 'AUTH_RATELIMIT_SUBNET': False,
'DISABLE_STATISTICS': False, 'DISABLE_STATISTICS': False,

@ -7,6 +7,9 @@ import idna
import re import re
import srslib import srslib
@internal.route("/postfix/dane/<domain_name>")
def postfix_dane_map(domain_name):
return flask.jsonify('dane-only') if utils.has_dane_record(domain_name) else flask.abort(404)
@internal.route("/postfix/domain/<domain_name>") @internal.route("/postfix/domain/<domain_name>")
def postfix_mailbox_domain(domain_name): def postfix_mailbox_domain(domain_name):

@ -6,6 +6,9 @@ try:
except ImportError: except ImportError:
import pickle import pickle
import dns
import dns.resolver
import hmac import hmac
import secrets import secrets
import time import time
@ -25,7 +28,6 @@ from itsdangerous.encoding import want_bytes
from werkzeug.datastructures import CallbackDict from werkzeug.datastructures import CallbackDict
from werkzeug.contrib import fixers from werkzeug.contrib import fixers
# Login configuration # Login configuration
login = flask_login.LoginManager() login = flask_login.LoginManager()
login.login_view = "ui.login" login.login_view = "ui.login"
@ -37,6 +39,34 @@ def handle_needs_login():
flask.url_for('ui.login', next=flask.request.endpoint) flask.url_for('ui.login', next=flask.request.endpoint)
) )
# DNS stub configured to do DNSSEC enabled queries
resolver = dns.resolver.Resolver()
resolver.use_edns(0, 0, 1232)
resolver.flags = dns.flags.AD | dns.flags.RD
def has_dane_record(domain, timeout=10):
try:
result = resolver.query(f'_25._tcp.{domain}', dns.rdatatype.TLSA,dns.rdataclass.IN, lifetime=timeout)
if result.response.flags & dns.flags.AD:
for record in result:
if isinstance(record, dns.rdtypes.ANY.TLSA.TLSA):
record.validate()
if record.usage in [2,3] and record.selector in [0,1] and record.mtype in [0,1,2]:
return True
except dns.resolver.NoNameservers:
# If the DNSSEC data is invalid and the DNS resolver is DNSSEC enabled
# we will receive this non-specific exception. The safe behaviour is to
# accept to defer the email.
flask.current_app.logger.warn(f'Unable to lookup the TLSA record for {domain}. Is the DNSSEC zone okay on https://dnsviz.net/d/{domain}/dnssec/?')
return app.config['DEFER_ON_TLS_ERROR']
except dns.exception.Timeout:
flask.current_app.logger.warn(f'Timeout while resolving the TLSA record for {domain} ({timeout}s).')
except dns.resolver.NXDOMAIN:
pass # this is expected, not TLSA record is fine
except Exception as e:
flask.current_app.logger.error(f'Error while looking up the TLSA record for {domain} {e}')
pass
# Rate limiter # Rate limiter
limiter = limiter.LimitWraperFactory() limiter = limiter.LimitWraperFactory()

@ -12,10 +12,15 @@ RUN pip3 install socrate==0.2.0
RUN pip3 install "podop>0.2.5" RUN pip3 install "podop>0.2.5"
# Image specific layers under this line # Image specific layers under this line
RUN apk add --no-cache --virtual .build-deps gcc musl-dev python3-dev
RUN pip3 install --no-binary :all: postfix-mta-sts-resolver==1.0.1
RUN apk del .build-deps gcc musl-dev python3-dev
RUN apk add --no-cache postfix postfix-pcre cyrus-sasl-login RUN apk add --no-cache postfix postfix-pcre cyrus-sasl-login
COPY conf /conf COPY conf /conf
COPY start.py /start.py COPY start.py /start.py
COPY mta-sts-daemon.yml /etc/
EXPOSE 25/tcp 10025/tcp EXPOSE 25/tcp 10025/tcp
VOLUME ["/queue"] VOLUME ["/queue"]

@ -58,13 +58,17 @@ tls_ssl_options = NO_COMPRESSION, NO_TICKET
# 2. not all will have and up-to-date TLS stack. # 2. not all will have and up-to-date TLS stack.
smtp_tls_mandatory_protocols = !SSLv2, !SSLv3 smtp_tls_mandatory_protocols = !SSLv2, !SSLv3
smtp_tls_protocols =!SSLv2,!SSLv3 smtp_tls_protocols =!SSLv2,!SSLv3
smtp_tls_security_level = {{ OUTBOUND_TLS_LEVEL|default('may') }} smtp_tls_security_level = {{ OUTBOUND_TLS_LEVEL|default('dane') }}
smtp_tls_policy_maps=lmdb:/etc/postfix/tls_policy.map smtp_tls_dane_insecure_mx_policy = {% if DEFER_ON_TLS_ERROR == 'false' %}may{% else %}dane{% endif %}
smtp_tls_policy_maps=lmdb:/etc/postfix/tls_policy.map, ${podop}dane, socketmap:unix:/tmp/mta-sts.socket:postfix
smtp_tls_CApath = /etc/ssl/certs smtp_tls_CApath = /etc/ssl/certs
smtp_tls_session_cache_database = lmdb:/dev/shm/postfix/smtp_scache smtp_tls_session_cache_database = lmdb:/dev/shm/postfix/smtp_scache
smtpd_tls_session_cache_database = lmdb:/dev/shm/postfix/smtpd_scache smtpd_tls_session_cache_database = lmdb:/dev/shm/postfix/smtpd_scache
smtp_host_lookup = dns smtp_host_lookup = dns
smtp_dns_support_level = dnssec smtp_dns_support_level = dnssec
delay_warning_time = 5m
smtp_tls_loglevel = 1
notify_classes = resource, software, delay
############### ###############
# Virtual # Virtual

@ -0,0 +1,10 @@
path: "/tmp/mta-sts.socket"
mode: 0600
shutdown_timeout: 20
cache:
type: internal
options:
cache_size: 10000
default_zone:
strict_testing: {{ DEFER_ON_TLS_ERROR |default('true') }}
timeout: 4

@ -21,6 +21,7 @@ def start_podop():
run_server(0, "postfix", "/tmp/podop.socket", [ run_server(0, "postfix", "/tmp/podop.socket", [
("transport", "url", url + "transport/§"), ("transport", "url", url + "transport/§"),
("alias", "url", url + "alias/§"), ("alias", "url", url + "alias/§"),
("dane", "url", url + "dane/§"),
("domain", "url", url + "domain/§"), ("domain", "url", url + "domain/§"),
("mailbox", "url", url + "mailbox/§"), ("mailbox", "url", url + "mailbox/§"),
("recipientmap", "url", url + "recipient/map/§"), ("recipientmap", "url", url + "recipient/map/§"),
@ -30,6 +31,12 @@ def start_podop():
("senderrate", "url", url + "sender/rate/§") ("senderrate", "url", url + "sender/rate/§")
]) ])
def start_mta_sts_daemon():
os.chmod("/root/", 0o755) # read access to /root/.netrc required
os.setuid(getpwnam('postfix').pw_uid)
from postfix_mta_sts_resolver import daemon
daemon.main()
def is_valid_postconf_line(line): def is_valid_postconf_line(line):
return not line.startswith("#") \ return not line.startswith("#") \
and not line == '' and not line == ''
@ -68,10 +75,12 @@ for map_file in glob.glob("/overrides/*.map"):
os.system("postmap {}".format(destination)) os.system("postmap {}".format(destination))
os.remove(destination) os.remove(destination)
if not os.path.exists("/etc/postfix/tls_policy.map.db"): if os.path.exists("/overrides/mta-sts-daemon.yml"):
with open("/etc/postfix/tls_policy.map", "w") as f: shutil.copyfile("/overrides/mta-sts-daemon.yml", "/etc/mta-sts-daemon.yml")
for domain in ['gmail.com', 'yahoo.com', 'hotmail.com', 'aol.com', 'outlook.com', 'comcast.net', 'icloud.com', 'msn.com', 'hotmail.co.uk', 'live.com', 'yahoo.co.in', 'me.com', 'mail.ru', 'cox.net', 'yahoo.co.uk', 'verizon.net', 'ymail.com', 'hotmail.it', 'kw.com', 'yahoo.com.tw', 'mac.com', 'live.se', 'live.nl', 'yahoo.com.br', 'googlemail.com', 'libero.it', 'web.de', 'allstate.com', 'btinternet.com', 'online.no', 'yahoo.com.au', 'live.dk', 'earthlink.net', 'yahoo.fr', 'yahoo.it', 'gmx.de', 'hotmail.fr', 'shawinc.com', 'yahoo.de', 'moe.edu.sg', 'naver.com', 'bigpond.com', 'statefarm.com', 'remax.net', 'rocketmail.com', 'live.no', 'yahoo.ca', 'bigpond.net.au', 'hotmail.se', 'gmx.at', 'live.co.uk', 'mail.com', 'yahoo.in', 'yandex.ru', 'qq.com', 'charter.net', 'indeedemail.com', 'alice.it', 'hotmail.de', 'bluewin.ch', 'optonline.net', 'wp.pl', 'yahoo.es', 'hotmail.no', 'pindotmedia.com', 'orange.fr', 'live.it', 'yahoo.co.id', 'yahoo.no', 'hotmail.es', 'morganstanley.com', 'wellsfargo.com', 'wanadoo.fr', 'facebook.com', 'yahoo.se', 'fema.dhs.gov', 'rogers.com', 'yahoo.com.hk', 'live.com.au', 'nic.in', 'nab.com.au', 'ubs.com', 'shaw.ca', 'umich.edu', 'westpac.com.au', 'yahoo.com.mx', 'yahoo.com.sg', 'farmersagent.com', 'yahoo.dk', 'dhs.gov']: conf.jinja("/etc/mta-sts-daemon.yml", os.environ, "/etc/mta-sts-daemon.yml")
f.write(f'{domain}\tsecure\n')
if not os.path.exists("/etc/postfix/tls_policy.map.lmdb"):
open("/etc/postfix/tls_policy.map", "a").close()
os.system("postmap /etc/postfix/tls_policy.map") os.system("postmap /etc/postfix/tls_policy.map")
if "RELAYUSER" in os.environ: if "RELAYUSER" in os.environ:
@ -81,6 +90,7 @@ if "RELAYUSER" in os.environ:
# Run Podop and Postfix # Run Podop and Postfix
multiprocessing.Process(target=start_podop).start() multiprocessing.Process(target=start_podop).start()
multiprocessing.Process(target=start_mta_sts_daemon).start()
os.system("/usr/libexec/postfix/post-install meta_directory=/etc/postfix create-missing") os.system("/usr/libexec/postfix/post-install meta_directory=/etc/postfix create-missing")
# Before starting postfix, we need to check permissions on /queue # Before starting postfix, we need to check permissions on /queue
# in the event that postfix,postdrop id have changed # in the event that postfix,postdrop id have changed

@ -72,8 +72,12 @@ mail in following format: ``[HOST]:PORT``.
``RELAYUSER`` and ``RELAYPASSWORD`` can be used when authentication is needed. ``RELAYUSER`` and ``RELAYPASSWORD`` can be used when authentication is needed.
By default postfix uses "opportunistic TLS" for outbound mail. This can be changed By default postfix uses "opportunistic TLS" for outbound mail. This can be changed
by setting ``OUTBOUND_TLS_LEVEL`` to ``encrypt`` or ``secure``. This setting is highly recommended by setting ``OUTBOUND_TLS_LEVEL`` to ``encrypt`` or ``secure``. This setting is
if you are using a relayhost that supports TLS. highly recommended if you are using a relayhost that supports TLS but discouraged
otherwise. ``DEFER_ON_TLS_ERROR`` (default: True) controls whether incomplete
policies (DANE without DNSSEC or "testing" MTA-STS policies) will be taken into
account and whether emails will be defered if the additional checks enforced by
those policies fail.
Similarily by default nginx uses "opportunistic TLS" for inbound mail. This can be changed Similarily by default nginx uses "opportunistic TLS" for inbound mail. This can be changed
by setting ``INBOUND_TLS_ENFORCE`` to ``True``. Please note that this is forbidden for by setting ``INBOUND_TLS_ENFORCE`` to ``True``. Please note that this is forbidden for

@ -369,6 +369,31 @@ How do I use webdav (radicale)?
.. _`575`: https://github.com/Mailu/Mailu/issues/575 .. _`575`: https://github.com/Mailu/Mailu/issues/575
.. _`1591`: https://github.com/Mailu/Mailu/issues/1591 .. _`1591`: https://github.com/Mailu/Mailu/issues/1591
How do I setup a MTA-STS policy?
````````````````````````````````
Mailu can serve an `MTA-STS policy`_; To configure it you will need to:
1. add ``mta-sts.example.com`` to the ``HOSTNAMES`` configuration variable (and ensure that a valid SSL certificate is available for it; this may mean restarting your smtp container)
2. configure an override with the policy itself; for example, your ``overrides/nginx/mta-sts.conf`` could read:
.. code-block:: bash
location ^~ /.well-known/mta-sts.txt {
return 200 "version: STSv1
mode: enforce
max_age: 1296000
mx: mailu.example.com\r\n";
}
3. setup the appropriate DNS/CNAME record (``mta-sts.example.com`` -> ``mailu.example.com``) and DNS/TXT record (``_mta-sts.example.com`` -> ``v=STSv1; id=1``) paying attention to the ``TTL`` as this is used by MTA-STS.
*issue reference:* `1798`_.
.. _`1798`: https://github.com/Mailu/Mailu/issues/1798
.. _`MTA-STS policy`: https://datatracker.ietf.org/doc/html/rfc8461
Technical issues Technical issues
---------------- ----------------
@ -397,6 +422,22 @@ Any mail related connection is proxied by nginx. Therefore the SMTP Banner is al
.. _`1368`: https://github.com/Mailu/Mailu/issues/1368 .. _`1368`: https://github.com/Mailu/Mailu/issues/1368
My emails are getting defered, what can I do?
`````````````````````````````````````````````
Emails are asynchronous and it's not abnormal for them to be defered sometimes. That being said, Mailu enforces secure connections where possible using DANE and MTA-STS, both of which have the potential to delay indefinitely delivery if something is misconfigured.
If delivery to a specific domain fails because their DANE records are invalid or their TLS configuration inadequate (expired certificate, ...), you can assist delivery by downgrading the security level for that domain by creating an override at ``overrides/postfix/tls_policy.map`` as follow:
.. code-block:: bash
domain.example.com may
domain.example.org encrypt
The syntax and options are as described in `postfix's documentation`_. Re-creating the smtp container will be required for changes to take effect.
.. _`postfix's documentation`: http://www.postfix.org/postconf.5.html#smtp_tls_policy_maps
403 - Access Denied Errors 403 - Access Denied Errors
--------------------------- ---------------------------

@ -0,0 +1 @@
Implement MTA-STS and DANE validation. Introduce DEFER_ON_TLS_ERROR (default: True) to harden or loosen the policy enforcement.
Loading…
Cancel
Save