Merge #1961

1961: Implement MTA-STS and DANE validation r=mergify[bot] a=nextgens

## What type of PR?

Feature

## What does this PR do?

Implement MTA-STS: the tls_policy_map will now be auto-configured based on the policies published by the various domains. A FAQ entry has been added to document how to publish a policy using Mailu.

As configured by default there is no persistence. If we want persistence we can have either sqlite3 (with a db in the mailqueue) or redis...

This also introduces a DEFER_ON_TLS_ERROR (default: True) setting that will harden policy enforcement and defer emails that shouldn't be delivered. Turn it off if you never want to set an override.

### Related issue(s)
- closes #1798
- closes #707 

## Prerequistes
Before we can consider review and merge, please make sure the following list is done and checked.
If an entry in not applicable, you can check it or remove it from the list.

- [x] In case of feature or enhancement: documentation updated accordingly
- [x] Unless it's docs or a minor change: add [changelog](https://mailu.io/master/contributors/workflow.html#changelog) entry file.


Co-authored-by: Florent Daigniere <nextgens@freenetproject.org>

README.md

@@ -22,7 +22,7 @@ Main features include:
 - **Web access**, multiple Webmails and administration interface
 - **User features**, aliases, auto-reply, auto-forward, fetched accounts
 - **Admin features**, global admins, announcements, per-domain delegation, quotas
-- **Security**, enforced TLS, Letsencrypt!, outgoing DKIM, anti-virus scanner
+- **Security**, enforced TLS, DANE, MTA-STS, Letsencrypt!, outgoing DKIM, anti-virus scanner
 - **Antispam**, auto-learn, greylisting, DMARC and SPF
 - **Freedom**, all FOSS components, no tracker included
 

core/admin/mailu/configuration.py

@@ -35,6 +35,7 @@ DEFAULT_CONFIG = {
     'WILDCARD_SENDERS': '',
     'TLS_FLAVOR': 'cert',
     'INBOUND_TLS_ENFORCE': False,
+    'DEFER_ON_TLS_ERROR': True,
     'AUTH_RATELIMIT': '1000/minute;10000/hour',
     'AUTH_RATELIMIT_SUBNET': False,
     'DISABLE_STATISTICS': False,

core/admin/mailu/internal/views/postfix.py

@@ -7,6 +7,9 @@ import idna
 import re
 import srslib
 
+@internal.route("/postfix/dane/<domain_name>")
+def postfix_dane_map(domain_name):
+    return flask.jsonify('dane-only') if utils.has_dane_record(domain_name) else flask.abort(404)
 
 @internal.route("/postfix/domain/<domain_name>")
 def postfix_mailbox_domain(domain_name):

core/admin/mailu/utils.py

@@ -6,6 +6,9 @@ try:
 except ImportError:
     import pickle
 
+import dns
+import dns.resolver
+
 import hmac
 import secrets
 import time
@@ -25,7 +28,6 @@ from itsdangerous.encoding import want_bytes
 from werkzeug.datastructures import CallbackDict
 from werkzeug.contrib import fixers
 
-
 # Login configuration
 login = flask_login.LoginManager()
 login.login_view = "ui.login"
@@ -37,6 +39,34 @@ def handle_needs_login():
         flask.url_for('ui.login', next=flask.request.endpoint)
     )
 
+# DNS stub configured to do DNSSEC enabled queries
+resolver = dns.resolver.Resolver()
+resolver.use_edns(0, 0, 1232)
+resolver.flags = dns.flags.AD | dns.flags.RD
+
+def has_dane_record(domain, timeout=10):
+    try:
+        result = resolver.query(f'_25._tcp.{domain}', dns.rdatatype.TLSA,dns.rdataclass.IN, lifetime=timeout)
+        if result.response.flags & dns.flags.AD:
+            for record in result:
+                if isinstance(record, dns.rdtypes.ANY.TLSA.TLSA):
+                    record.validate()
+                    if record.usage in [2,3] and record.selector in [0,1] and record.mtype in [0,1,2]:
+                        return True
+    except dns.resolver.NoNameservers:
+        # If the DNSSEC data is invalid and the DNS resolver is DNSSEC enabled
+        # we will receive this non-specific exception. The safe behaviour is to
+        # accept to defer the email.
+        flask.current_app.logger.warn(f'Unable to lookup the TLSA record for {domain}. Is the DNSSEC zone okay on https://dnsviz.net/d/{domain}/dnssec/?')
+        return app.config['DEFER_ON_TLS_ERROR']
+    except dns.exception.Timeout:
+        flask.current_app.logger.warn(f'Timeout while resolving the TLSA record for {domain} ({timeout}s).')
+    except dns.resolver.NXDOMAIN:
+        pass # this is expected, not TLSA record is fine
+    except Exception as e:
+        flask.current_app.logger.error(f'Error while looking up the TLSA record for {domain} {e}')
+        pass
+
 # Rate limiter
 limiter = limiter.LimitWraperFactory()
 

core/postfix/Dockerfile

@@ -12,10 +12,15 @@ RUN pip3 install socrate==0.2.0
 RUN pip3 install "podop>0.2.5"
 
 # Image specific layers under this line
+RUN apk add --no-cache --virtual .build-deps gcc musl-dev python3-dev
+RUN pip3 install --no-binary :all: postfix-mta-sts-resolver==1.0.1
+RUN apk del .build-deps gcc musl-dev python3-dev
+
 RUN apk add --no-cache postfix postfix-pcre cyrus-sasl-login
 
 COPY conf /conf
 COPY start.py /start.py
+COPY mta-sts-daemon.yml /etc/
 
 EXPOSE 25/tcp 10025/tcp
 VOLUME ["/queue"]

core/postfix/conf/main.cf

@@ -58,13 +58,17 @@ tls_ssl_options = NO_COMPRESSION, NO_TICKET
 # 2. not all will have and up-to-date TLS stack.
 smtp_tls_mandatory_protocols = !SSLv2, !SSLv3
 smtp_tls_protocols =!SSLv2,!SSLv3
-smtp_tls_security_level = {{ OUTBOUND_TLS_LEVEL|default('may') }}
+smtp_tls_security_level = {{ OUTBOUND_TLS_LEVEL|default('dane') }}
-smtp_tls_policy_maps=lmdb:/etc/postfix/tls_policy.map
+smtp_tls_dane_insecure_mx_policy = {% if DEFER_ON_TLS_ERROR == 'false' %}may{% else %}dane{% endif %}
+smtp_tls_policy_maps=lmdb:/etc/postfix/tls_policy.map, ${podop}dane, socketmap:unix:/tmp/mta-sts.socket:postfix
 smtp_tls_CApath = /etc/ssl/certs
 smtp_tls_session_cache_database = lmdb:/dev/shm/postfix/smtp_scache
 smtpd_tls_session_cache_database = lmdb:/dev/shm/postfix/smtpd_scache
 smtp_host_lookup = dns
 smtp_dns_support_level = dnssec
+delay_warning_time = 5m
+smtp_tls_loglevel = 1
+notify_classes = resource, software, delay
 
 ###############
 # Virtual

core/postfix/mta-sts-daemon.yml

@@ -0,0 +1,10 @@
+path: "/tmp/mta-sts.socket"
+mode: 0600
+shutdown_timeout: 20
+cache:
+  type: internal
+  options:
+    cache_size: 10000
+default_zone:
+  strict_testing: {{ DEFER_ON_TLS_ERROR |default('true') }}
+  timeout: 4

core/postfix/start.py

@@ -21,6 +21,7 @@ def start_podop():
     run_server(0, "postfix", "/tmp/podop.socket", [
         ("transport", "url", url + "transport/§"),
         ("alias", "url", url + "alias/§"),
+        ("dane", "url", url + "dane/§"),
         ("domain", "url", url + "domain/§"),
         ("mailbox", "url", url + "mailbox/§"),
         ("recipientmap", "url", url + "recipient/map/§"),
@@ -30,6 +31,12 @@ def start_podop():
         ("senderrate", "url", url + "sender/rate/§")
     ])
 
+def start_mta_sts_daemon():
+    os.chmod("/root/", 0o755) # read access to /root/.netrc required
+    os.setuid(getpwnam('postfix').pw_uid)
+    from postfix_mta_sts_resolver import daemon
+    daemon.main()
+
 def is_valid_postconf_line(line):
     return not line.startswith("#") \
             and not line == ''
@@ -68,10 +75,12 @@ for map_file in glob.glob("/overrides/*.map"):
     os.system("postmap {}".format(destination))
     os.remove(destination)
 
-if not os.path.exists("/etc/postfix/tls_policy.map.db"):
+if os.path.exists("/overrides/mta-sts-daemon.yml"):
-    with open("/etc/postfix/tls_policy.map", "w") as f:
+    shutil.copyfile("/overrides/mta-sts-daemon.yml", "/etc/mta-sts-daemon.yml")
-        for domain in ['gmail.com', 'yahoo.com', 'hotmail.com', 'aol.com', 'outlook.com', 'comcast.net', 'icloud.com', 'msn.com', 'hotmail.co.uk', 'live.com', 'yahoo.co.in', 'me.com', 'mail.ru', 'cox.net', 'yahoo.co.uk', 'verizon.net', 'ymail.com', 'hotmail.it', 'kw.com', 'yahoo.com.tw', 'mac.com', 'live.se', 'live.nl', 'yahoo.com.br', 'googlemail.com', 'libero.it', 'web.de', 'allstate.com', 'btinternet.com', 'online.no', 'yahoo.com.au', 'live.dk', 'earthlink.net', 'yahoo.fr', 'yahoo.it', 'gmx.de', 'hotmail.fr', 'shawinc.com', 'yahoo.de', 'moe.edu.sg', 'naver.com', 'bigpond.com', 'statefarm.com', 'remax.net', 'rocketmail.com', 'live.no', 'yahoo.ca', 'bigpond.net.au', 'hotmail.se', 'gmx.at', 'live.co.uk', 'mail.com', 'yahoo.in', 'yandex.ru', 'qq.com', 'charter.net', 'indeedemail.com', 'alice.it', 'hotmail.de', 'bluewin.ch', 'optonline.net', 'wp.pl', 'yahoo.es', 'hotmail.no', 'pindotmedia.com', 'orange.fr', 'live.it', 'yahoo.co.id', 'yahoo.no', 'hotmail.es', 'morganstanley.com', 'wellsfargo.com', 'wanadoo.fr', 'facebook.com', 'yahoo.se', 'fema.dhs.gov', 'rogers.com', 'yahoo.com.hk', 'live.com.au', 'nic.in', 'nab.com.au', 'ubs.com', 'shaw.ca', 'umich.edu', 'westpac.com.au', 'yahoo.com.mx', 'yahoo.com.sg', 'farmersagent.com', 'yahoo.dk', 'dhs.gov']:
+conf.jinja("/etc/mta-sts-daemon.yml", os.environ, "/etc/mta-sts-daemon.yml")
-            f.write(f'{domain}\tsecure\n')
+
+if not os.path.exists("/etc/postfix/tls_policy.map.lmdb"):
+    open("/etc/postfix/tls_policy.map", "a").close()
     os.system("postmap /etc/postfix/tls_policy.map")
 
 if "RELAYUSER" in os.environ:
@@ -81,6 +90,7 @@ if "RELAYUSER" in os.environ:
 
 # Run Podop and Postfix
 multiprocessing.Process(target=start_podop).start()
+multiprocessing.Process(target=start_mta_sts_daemon).start()
 os.system("/usr/libexec/postfix/post-install meta_directory=/etc/postfix create-missing")
 # Before starting postfix, we need to check permissions on /queue
 # in the event that postfix,postdrop id have changed

docs/configuration.rst

@@ -72,8 +72,12 @@ mail in following format: ``[HOST]:PORT``.
 ``RELAYUSER`` and ``RELAYPASSWORD`` can be used when authentication is needed.
 
 By default postfix uses "opportunistic TLS" for outbound mail. This can be changed
-by setting ``OUTBOUND_TLS_LEVEL`` to ``encrypt`` or ``secure``. This setting is highly recommended
+by setting ``OUTBOUND_TLS_LEVEL`` to ``encrypt`` or ``secure``. This setting is
-if you are using a relayhost that supports TLS.
+highly recommended if you are using a relayhost that supports TLS but discouraged
+otherwise. ``DEFER_ON_TLS_ERROR`` (default: True) controls whether incomplete
+policies (DANE without DNSSEC or "testing" MTA-STS policies) will be taken into
+account and whether emails will be defered if the additional checks enforced by
+those policies fail.
 
 Similarily by default nginx uses "opportunistic TLS" for inbound mail. This can be changed
 by setting ``INBOUND_TLS_ENFORCE`` to ``True``. Please note that this is forbidden for

docs/faq.rst

@@ -369,6 +369,31 @@ How do I use webdav (radicale)?
 .. _`575`: https://github.com/Mailu/Mailu/issues/575
 .. _`1591`: https://github.com/Mailu/Mailu/issues/1591
 
+How do I setup a MTA-STS policy?
+````````````````````````````````
+
+Mailu can serve an `MTA-STS policy`_; To configure it you will need to:
+
+1. add ``mta-sts.example.com`` to the ``HOSTNAMES`` configuration variable (and ensure that a valid SSL certificate is available for it; this may mean restarting your smtp container)
+
+2. configure an override with the policy itself; for example, your ``overrides/nginx/mta-sts.conf`` could read:
+
+.. code-block:: bash
+
+   location ^~ /.well-known/mta-sts.txt {
+   return 200 "version: STSv1
+   mode: enforce
+   max_age: 1296000
+   mx: mailu.example.com\r\n";
+   }
+
+3. setup the appropriate DNS/CNAME record (``mta-sts.example.com`` -> ``mailu.example.com``) and DNS/TXT record (``_mta-sts.example.com`` -> ``v=STSv1; id=1``) paying attention to the ``TTL`` as this is used by MTA-STS.
+
+*issue reference:* `1798`_.
+
+.. _`1798`: https://github.com/Mailu/Mailu/issues/1798
+.. _`MTA-STS policy`: https://datatracker.ietf.org/doc/html/rfc8461
+
 Technical issues
 ----------------
 
@@ -397,6 +422,22 @@ Any mail related connection is proxied by nginx. Therefore the SMTP Banner is al
 
 .. _`1368`: https://github.com/Mailu/Mailu/issues/1368
 
+My emails are getting defered, what can I do?
+`````````````````````````````````````````````
+
+Emails are asynchronous and it's not abnormal for them to be defered sometimes. That being said, Mailu enforces secure connections where possible using DANE and MTA-STS, both of which have the potential to delay indefinitely delivery if something is misconfigured.
+
+If delivery to a specific domain fails because their DANE records are invalid or their TLS configuration inadequate (expired certificate, ...), you can assist delivery by downgrading the security level for that domain by creating an override at ``overrides/postfix/tls_policy.map`` as follow:
+
+.. code-block:: bash
+
+   domain.example.com   may
+   domain.example.org   encrypt
+
+The syntax and options are as described in `postfix's documentation`_. Re-creating the smtp container will be required for changes to take effect.
+
+.. _`postfix's documentation`: http://www.postfix.org/postconf.5.html#smtp_tls_policy_maps
+
 403 - Access Denied Errors
 ---------------------------
 

towncrier/newsfragments/1798.feature

@@ -0,0 +1 @@
+Implement MTA-STS and DANE validation. Introduce DEFER_ON_TLS_ERROR (default: True) to harden or loosen the policy enforcement.