Merge #1758 #1776

1758: Implement a simpler credential cache (alternative to #1755) r=mergify[bot] a=nextgens

## What type of PR?

Feature: it implements a credential cache to speedup authentication requests.

## What does this PR do?

Credentials are stored in cold-storage using a slow, salted/iterated hash function to prevent offline bruteforce attacks. This creates a performance bottleneck for no valid reason (see the
rationale/long version on https://github.com/Mailu/Mailu/issues/1194#issuecomment-762115549).

The new credential cache makes things fast again.

This is the simpler version of #1755 (with no new dependencies)

### Related issue(s)
- close #1411
- close #1194 
- close #1755

## Prerequistes
Before we can consider review and merge, please make sure the following list is done and checked.
If an entry in not applicable, you can check it or remove it from the list.

- [x] In case of feature or enhancement: documentation updated accordingly
- [x] Unless it's docs or a minor change: add [changelog](https://mailu.io/master/contributors/guide.html#changelog) entry file.


1776: optimize generation of transport nexthop r=mergify[bot] a=ghostwheel42

## What type of PR?

bug-fix and enhancement.

## What does this PR do?

Possibly there should be more input validation when editing a relay, but for now this tries to make the best out of the existing "smtp" attribute while maintaining backwards compatibility. When relay is empty, the transport's nexthop is the MX of the relayed domain to fix #1588 

```
RELAY			NEXTHOP						TRANSPORT
empty			use MX of relay domain				smtp:domain
:port			use MX of relay domain and use port	smtp:domain:port
target			resolve A/AAAA of target			smtp:[target]
target:port		resolve A/AAAA of target and use port	smtp:[target]:port
mx:target		resolve MX of target				smtp:target
mx:target:port	resolve MX of target and use port	smtp:target:port
lmtp:target		resolve A/AAAA of target			lmtp:target
lmtp:target:port	resolve A/AAAA of target and use port	lmtp:target:port

target can also be an IPv4 or IPv6 address (an IPv6 address must be enclosed in []: [2001:DB8::]).
```

When there is proper input validation and existing database entries are migrated this function can be made much shorter again.

### Related issue(s)
- closes #1588 
- closes #1815 

## Prerequistes
Before we can consider review and merge, please make sure the following list is done and checked.
If an entry in not applicable, you can check it or remove it from the list.

- [x] In case of feature or enhancement: documentation updated accordingly
- [X] Unless it's docs or a minor change: add [changelog](https://mailu.io/master/contributors/guide.html#changelog) entry file.


Co-authored-by: Florent Daigniere <nextgens@freenetproject.org>
Co-authored-by: Alexander Graf <ghostwheel42@users.noreply.github.com>

core/admin/mailu/internal/views/postfix.py

@@ -2,6 +2,7 @@ from mailu import models
 from mailu.internal import internal
 
 import flask
+import idna
 import re
 import srslib
 
@@ -35,13 +36,67 @@ def postfix_alias_map(alias):
 def postfix_transport(email):
     if email == '*' or re.match("(^|.*@)\[.*\]$", email):
         return flask.abort(404)
-    localpart, domain_name = models.Email.resolve_domain(email)
+    _, domain_name = models.Email.resolve_domain(email)
     relay = models.Relay.query.get(domain_name) or flask.abort(404)
-    ret = "smtp:[{0}]".format(relay.smtp)
+    target = relay.smtp.lower()
-    if ":" in relay.smtp:
+    port = None
-        split = relay.smtp.split(':')
+    use_lmtp = False
-        ret = "smtp:[{0}]:{1}".format(split[0], split[1])
+    use_mx = False
-    return flask.jsonify(ret)
+    # strip prefixes mx: and lmtp:
+    if target.startswith('mx:'):
+        target = target[3:]
+        use_mx = True
+    elif target.startswith('lmtp:'):
+        target = target[5:]
+        use_lmtp = True
+    # split host:port or [host]:port
+    if target.startswith('['):
+        if use_mx or ']' not in target:
+            # invalid target (mx: and [] or missing ])
+            flask.abort(400)
+        host, rest = target[1:].split(']', 1)
+        if rest.startswith(':'):
+            port = rest[1:]
+        elif rest:
+            # invalid target (rest should be :port)
+            flask.abort(400)
+    else:
+        if ':' in target:
+            host, port = target.rsplit(':', 1)
+        else:
+            host = target
+    # default for empty host part is mx:domain
+    if not host:
+        if not use_lmtp:
+            host = relay.name.lower()
+            use_mx = True
+        else:
+            # lmtp: needs a host part
+            flask.abort(400)
+    # detect ipv6 address or encode host
+    if ':' in host:
+        host = f'ipv6:{host}'
+    else:
+        try:
+            host = idna.encode(host).decode('ascii')
+        except idna.IDNAError:
+            # invalid host (fqdn not encodable)
+            flask.abort(400)
+    # validate port
+    if port is not None:
+        try:
+            port = int(port, 10)
+        except ValueError:
+            # invalid port (should be numeric)
+            flask.abort(400)
+    # create transport
+    transport = 'lmtp' if use_lmtp else 'smtp'
+    # use [] when not using MX lookups or host is an ipv6 address
+    if host.startswith('ipv6:') or (not use_lmtp and not use_mx):
+        host = f'[{host}]'
+    # create port suffix
+    port = '' if port is None else f':{port}'
+    return flask.jsonify(f'{transport}:{host}{port}')
 
 
 @internal.route("/postfix/recipient/map/<path:recipient>")

core/admin/mailu/models.py

@@ -305,6 +305,7 @@ class User(Base, Email):
     """
     __tablename__ = "user"
     _ctx = None
+    _credential_cache = {}
 
     domain = db.relationship(Domain,
         backref=db.backref('users', cascade='all, delete-orphan'))
@@ -382,6 +383,17 @@ class User(Base, Email):
         return User._ctx
 
     def check_password(self, password):
+        cache_result = self._credential_cache.get(self.get_id())
+        current_salt = self.password.split('$')[3] if len(self.password.split('$')) == 5 else None
+        if cache_result and current_salt:
+            cache_salt, cache_hash = cache_result
+            if cache_salt == current_salt:
+                return hash.pbkdf2_sha256.verify(password, cache_hash)
+            else:
+                # the cache is local per gunicorn; the password has changed
+                # so the local cache can be invalidated
+                del self._credential_cache[self.get_id()]
+
         reference = self.password
         # strip {scheme} if that's something mailu has added
         # passlib will identify *crypt based hashes just fine
@@ -396,6 +408,17 @@ class User(Base, Email):
             self.password = new_hash
             db.session.add(self)
             db.session.commit()
+
+        if result:
+            """The credential cache uses a low number of rounds to be fast.
+While it's not meant to be persisted to cold-storage, no additional measures
+are taken to ensure it isn't (mlock(), encrypted swap, ...) on the basis that
+we have little control over GC and string interning anyways.
+
+ An attacker that can dump the process' memory is likely to find credentials
+in clear-text regardless of the presence of the cache.
+            """
+            self._credential_cache[self.get_id()] = (self.password.split('$')[3], hash.pbkdf2_sha256.using(rounds=1).hash(password))
         return result
 
     def set_password(self, password, hash_scheme=None, raw=False):

docs/webadministration.rst

@@ -1,7 +1,7 @@
 Web administration interface
 ============================
 
-The web administration interface is the main website for maintaining your Mailu installation. 
+The web administration interface is the main website for maintaining your Mailu installation.
 For brevity the web administration interface will now be mentioned as admin gui.
 It offers the following configuration options:
 
@@ -30,13 +30,13 @@ It offers the following configuration options:
 * Configure all email domains served by Mailu, including:
 
     * generating dkim and dmarc keys for a domain.
-    
+
     * view email domain information on how to configure your SPF, DMARC, DKIM and MX dns records for an email domain.
-    
+
     * Add new email domains.
-    
+
     * For existing domains, configure users, quotas, aliases, administrators and alternative domain names.
-    
+
 * access the webmail site.
 
 * lookup settings for configuring your email client.
@@ -49,7 +49,7 @@ The admin GUI is by default accessed via the URL `https://<my domain>/admin`, wh
 To login the admin GUI enter the email address and password of an user.
 
 Only global administrator users have access to all configuration settings and the Rspamd webgui. Other users will be presented with settings for only their account, and domains they are managers of.
-To create a user who is a global administrator for a new installation, the Mailu.env file can be adapted. 
+To create a user who is a global administrator for a new installation, the Mailu.env file can be adapted.
 For more information see the section 'Admin account - automatic creation' in :ref:`the configuration reference <admin_account>`.
 
 The following sections are only accessible for global administrators:
@@ -69,7 +69,7 @@ The following sections are only accessible for global administrators:
 
 Settings
 --------
-After logging in the web administration interface, the settings page is loaded. 
+After logging in the web administration interface, the settings page is loaded.
 On the settings page the settings of the currently logged in user can be changed.
 Changes are saved and effective immediately after clicking the Save Settings button at the bottom of the page.
 
@@ -77,27 +77,27 @@ Changes are saved and effective immediately after clicking the Save Settings but
 Display name
 ````````````
 
-On the settings page the displayed name can be changed of the logged in user. 
+On the settings page the displayed name can be changed of the logged in user.
 This display name is only used within the web administration interface.
 
 
 Antispam
 ````````
 
-Under the section `Antispam` the spam filter can be enabled or disabled for the logged in user. By default the spam filter is enabled. 
+Under the section `Antispam` the spam filter can be enabled or disabled for the logged in user. By default the spam filter is enabled.
 
 When the spam filter is disabled, all received email messages will go to the inbox folder of the logged in user.
 The exception to this rule, are email messages with an extremely high spam score. These email messages are always rejected by Rspamd.
 
 When the spam filter is enabled, received email messages will be moved to the logged in user's inbox folder or junk folder depending on the user defined spam filter tolerance.
 
-The user defined spam filter tolerance determines when an email is classified as ham (moved to the inbox folder) or spam (moved to the junk folder). 
+The user defined spam filter tolerance determines when an email is classified as ham (moved to the inbox folder) or spam (moved to the junk folder).
-The default value is 80%. The lower the spam filter tolerance, the more false positives (ham classified as spam). The higher the spam filter tolerance, the more false negatives (spam classified as ham). 
+The default value is 80%. The lower the spam filter tolerance, the more false positives (ham classified as spam). The higher the spam filter tolerance, the more false negatives (spam classified as ham).
 For more information see the :ref:`antispam documentation <antispam_howto>`.
 
 Auto-forward
 `````````````
-Under the section `Auto-forward`, the automatic forwarding of received email messages can be enabled. When enabled, all received email messages are forwarded to the specified email address. 
+Under the section `Auto-forward`, the automatic forwarding of received email messages can be enabled. When enabled, all received email messages are forwarded to the specified email address.
 
 The option "Keep a copy of the emails" can be ticked, to keep a copy of the received email message in the inbox folder.
 
@@ -107,7 +107,7 @@ In the destination textbox, the email addresses can be entered for automatic for
 Update password
 ---------------
 
-On the `update password` page, the password of the logged in user can be changed. Changes are effective immediately. 
+On the `update password` page, the password of the logged in user can be changed. Changes are effective immediately.
 
 
 .. _webadministration_auto-reply:
@@ -117,7 +117,7 @@ Auto-reply
 
 On the `auto-reply` page, automatic replies can be configured. This is also known as out of office (ooo) or out of facility (oof) replies.
 
-To enable automatic replies tick the checkbox 'Enable automatic reply'. 
+To enable automatic replies tick the checkbox 'Enable automatic reply'.
 
 Under Reply subject the email subject for automatic replies can be configured. When a reply subject is entered, this subject will be used for the automatic reply.
 
@@ -130,12 +130,12 @@ E.g. if the email subject of the received email message is "how are you?", then
 Fetched accounts
 ----------------
 
-This page is only available when the Fetchmail container is part of your Mailu deployment. 
+This page is only available when the Fetchmail container is part of your Mailu deployment.
 Fetchmail can be enabled when creating the docker-compose.yml file with the setup utility (https://setup.mailu.io).
 
 On the `fetched accounts` page you can configure email accounts from which email messages will be retrieved.
-Only unread email messages are retrieved from the specified email account. 
+Only unread email messages are retrieved from the specified email account.
-By default Fetchmail will retrieve email messages every 10 minutes. This can be changed in the Mailu.env file. 
+By default Fetchmail will retrieve email messages every 10 minutes. This can be changed in the Mailu.env file.
 For more information on changing the polling interval see :ref:`the configuration reference <fetchmail>`.
 
 
@@ -149,7 +149,7 @@ You can add a fetched account by clicking on the `Add an account` button on the
 
 * Enable TLS. Tick this setting if the email server requires TLS/SSL instead of STARTTLS.
 
-* Username. The user name for logging in to the email server. Normally this is the email address or the email address' local-part (the part before @). 
+* Username. The user name for logging in to the email server. Normally this is the email address or the email address' local-part (the part before @).
 
 * Password. The password for logging in to the email server.
 
@@ -166,8 +166,8 @@ The purpose of an authentication token is to create a unique and strong password
 The application will use this authentication token instead of the logged in user's password for sending/receiving email.
 This allows safe access to the logged in user's email account. At any moment, the authentication token can be deleted so that the application has no access to the logged in user's email account anymore.
 
-By clicking on the New token button on the top right of the page, a new authentication token can be created. On this page the generated authentication token will only be displayed once. 
+By clicking on the New token button on the top right of the page, a new authentication token can be created. On this page the generated authentication token will only be displayed once.
-After saving the application token it is not possible anymore to view the unique password. 
+After saving the application token it is not possible anymore to view the unique password.
 
 The comment field can be used to enter a description for the authentication token. For example the name of the application the application token is created for.
 
@@ -198,9 +198,9 @@ A global administrator can change `any setting` in the admin GUI. Be careful tha
 Relayed domains
 ---------------
 
-On the `relayed domains list` page, destination domains can be added that Mailu will relay email messages for without authentication. 
+On the `relayed domains list` page, destination domains can be added that Mailu will relay email messages for without authentication.
-This means that for these destination domains, other email clients or email servers can send email via Mailu unauthenticated via port 25 to this destination domain. 
+This means that for these destination domains, other email clients or email servers can send email via Mailu unauthenticated via port 25 to this destination domain.
-For example if the destination domain example.com is added. Any emails to example.com (john@example.com) will be relayed to example.com. 
+For example if the destination domain example.com is added. Any emails to example.com (john@example.com) will be relayed to example.com.
 Example scenario's are:
 
 * relay domain from a backup server.
@@ -212,30 +212,37 @@ Example scenario's are:
 
 On the new relayed domain page the following options can be entered for a new relayed domain:
 
-* Relayed domain name. The domain name that is relayed. Email messages addressed to this domain (To: John@example.com), will be forwarded to this domain. 
+* Relayed domain name. The domain name that is relayed. Email messages addressed to this domain (To: John@example.com), will be forwarded to this domain.
-  No authentication is required. 
+  No authentication is required.
 
-* Remote host (optional). The SMPT server that will be used for relaying the email message.
+* Remote host (optional). The host that will be used for relaying the email message.
-  When this field is blank, the Mailu server will directly send the email message to the relayed domain.
+  When this field is blank, the Mailu server will directly send the email message to the mail server  of the relayed domain.
-  As value can be entered either a hostname or IP address of the SMPT server. 
+  When a remote host is specified it can be prefixed by ``mx:`` or ``lmtp:`` and followed by a port number: ``:port``).
-  By default port 25 is used. To use a different port append ":port number" to the Remote Host. For example:
+
-  123.45.67.90:2525.
+  ================  =====================================  =========================
+  Remote host       Description                            postfix transport:nexthop
+  ================  =====================================  =========================
+  empty             use MX of relay domain                 smtp:domain
+  :port             use MX of relay domain and use port    smtp:domain:port
+  target            resolve A/AAAA of target               smtp:[target]
+  target:port       resolve A/AAAA of target and use port  smtp:[target]:port
+  mx:target         resolve MX of target                   smtp:target
+  mx:target:port    resolve MX of target and use port      smtp:target:port
+  lmtp:target       resolve A/AAAA of target               lmtp:target
+  lmtp:target:port  resolve A/AAAA of target and use port  lmtp:target:port
+  ================  =====================================  =========================
+
+  `target` can also be an IPv4 or IPv6 address (an IPv6 address must be enclosed in []: ``[2001:DB8::]``).
 
 * Comment. A text field where a comment can be entered to describe the entry.
 
 Changes are effective immediately after clicking the Save button.
 
-NOTE: Due to bug `1588`_ email messages fail to be relayed if no Remote Host is configured. 
-As a workaround the HOSTNAME or IP Address of the SMPT server of the relayed domain can be entered as Remote Host.
-Please note that no MX lookup is performed when entering a hostname as Remote Host. You can use the MX lookup on mxtoolbox.com to find the hostname and IP Address of the SMTP server.
-
-.. _`1588`: https://github.com/Mailu/Mailu/issues/1588
-
 Antispam
 --------
 
 The menu item Antispam opens the Rspamd webgui. For more information how spam filtering works in Mailu see the :ref:`Spam filtering page <antispam_howto_block>`.
-The spam filtering page also contains a section that describes how to create a local blacklist for blocking email messages from specific domains. 
+The spam filtering page also contains a section that describes how to create a local blacklist for blocking email messages from specific domains.
 The Rspamd webgui offers basic functions for setting metric actions, scores, viewing statistics and learning.
 
 The following settings are not persisent and are *lost* when the Antispam container is recreated or restarted:
@@ -266,31 +273,31 @@ On the `Mail domains` page all the domains served by Mailu are configured. Via t
 Details
 ```````
 
-This page is also accessible for domain managers. On the details page all DNS settings are displayed for configuring your DNS server. It contains information on what to configure as MX record and SPF record. On this page it is also possible to (re-)generate the keys for DKIM and DMARC. The option for generating keys for DKIM and DMARC is only available for global administrators.  After generating the keys for DKIM and DMARC, this page will also show the DNS records for configuring the DKIM/DMARC records on the DNS server. 
+This page is also accessible for domain managers. On the details page all DNS settings are displayed for configuring your DNS server. It contains information on what to configure as MX record and SPF record. On this page it is also possible to (re-)generate the keys for DKIM and DMARC. The option for generating keys for DKIM and DMARC is only available for global administrators.  After generating the keys for DKIM and DMARC, this page will also show the DNS records for configuring the DKIM/DMARC records on the DNS server.
 
 
 Edit
-```` 
+````
 
-This page is only accessible for global administrators. On the edit page, the global settings for the domain can be changed.  
+This page is only accessible for global administrators. On the edit page, the global settings for the domain can be changed.
 
 * Maximum user count. The maximum amount of users that can be created under this domain. Once this limit is reached it is not possible anymore to add users to the domain; and it is also not possible for users to self-register.
-    
+
 * Maximum alias count. The maximum amount of aliases that can be created for an email account.
-    
+
 * Maximum user quota. The maximum amount of quota that can be assigned to a user. When creating or editing a user, this sets the limit on the maximum amount of quota that can be assigned to the user.
-    
+
-* Enable sign-up. When this option is ticked, self-registration is enabled. When the Admin GUI is accessed, in the menu list the option Signup becomes available. 
+* Enable sign-up. When this option is ticked, self-registration is enabled. When the Admin GUI is accessed, in the menu list the option Signup becomes available.
-  Obviously this menu item is only visible when signed out. On the Signup page a user can create an email account. 
+  Obviously this menu item is only visible when signed out. On the Signup page a user can create an email account.
-  If your Admin GUI is available to the public internet, this means your Mailu installation basically becomes a free email provider. 
+  If your Admin GUI is available to the public internet, this means your Mailu installation basically becomes a free email provider.
   Use this option with care!
-  
+
 * Comment. Description for the domain. This description is visible on the parent domains list page.
 
 Delete
 ``````
 
-This page is only accessible for global administrators. This page allows you to delete the domain. The Admin GUI will ask for confirmation if the domain must be really deleted. 
+This page is only accessible for global administrators. This page allows you to delete the domain. The Admin GUI will ask for confirmation if the domain must be really deleted.
 
 
 Users
@@ -326,7 +333,7 @@ For adding a new user the following options can be configured.
 
 * Enabled. Tick this checkbox to enable the user account. When an user is disabled, the user is unable to login to the Admin GUI or webmail or access his email via IMAP/POP3 or send mail.
   The email inbox of the user is still retained. This option can be used to temporarily suspend an user account.
-  
+
 * Quota. The maximum quota for the user's email box.
 
 * Allow IMAP access. When ticked, allows email retrieval via the IMAP protocol.
@@ -337,7 +344,7 @@ For adding a new user the following options can be configured.
 Aliases
 ```````
 
-This page is also accessible for domain managers. On the aliases page, aliases can be added for email addresses. An alias is a way to disguise another email address. 
+This page is also accessible for domain managers. On the aliases page, aliases can be added for email addresses. An alias is a way to disguise another email address.
 Everything sent to an alias email address is actually received in the primary email account's inbox of the destination email address.
 Aliases can diversify a single email account without having to create multiple email addresses (users).
 It is also possible to add multiple email addresses to the destination field. All incoming mails will be sent to each users inbox in this case.
@@ -348,11 +355,11 @@ The following options are available when adding an alias:
 
 * Use SQL LIKE Syntax (e.g. for catch-all aliases). When this option is ticked, you can use SQL LIKE syntax as alias.
   The SQL LIKE syntax is used to match text values against a pattern using wildcards. There are two wildcards that can be used with SQL LIKE syntax:
-    
+
     * % - The percent sign represents zero, one, or multiple characters
     * _ - The underscore represents a single character
-    
+
-  Examples are:    
+  Examples are:
     * a% - Finds any values that start with "a"
     * %a - Finds any values that end with "a"
     * %or% - Finds any values that have "or" in any position
@@ -369,7 +376,7 @@ The following options are available when adding an alias:
 Managers
 ````````
 
-This page is also accessible for domain managers. On the `managers list` page, managers can be added for the domain and can be deleted. 
+This page is also accessible for domain managers. On the `managers list` page, managers can be added for the domain and can be deleted.
 Managers have access to configuration settings of the domain.
 On the `add manager` page you can click on the manager email text box to access a drop down list of users that can be made a manager of the domain.
 
@@ -377,11 +384,11 @@ On the `add manager` page you can click on the manager email text box to access
 Alternatives
 ````````````
 
-This page is only accessible for global administrators. On the alternatives page, alternative domains can be added for the domain. 
+This page is only accessible for global administrators. On the alternatives page, alternative domains can be added for the domain.
 An alternative domain acts as a copy of a given domain.
-Everything sent to an alternative domain, is actually received in the domain the alternative is created for. 
+Everything sent to an alternative domain, is actually received in the domain the alternative is created for.
-This allows you to receive emails for multiple domains while using a single domain. 
+This allows you to receive emails for multiple domains while using a single domain.
-For example if the main domain has the email address user@example.com, and the alternative domain is mymail.com, 
+For example if the main domain has the email address user@example.com, and the alternative domain is mymail.com,
 then email send to user@mymail.com will end up in the email box of user@example.com.
 
 New domain
@@ -392,16 +399,16 @@ This page is only accessible for global administrators. Via this page a new doma
 * domain name. The name of the domain.
 
 * Maximum user count. The maximum amount of users that can be created under this domain. Once this limit is reached it is not possible anymore to add users to the domain; and it is also not possible for users to self-register.
-    
+
 * Maximum alias count. The maximum amount of aliases that can be made for an email account.
-    
+
 * Maximum user quota. The maximum amount of quota that can be assigned to a user. When creating or editing a user, this sets the limit on the maximum amount of quota that can be assigned to the user.
-    
+
-* Enable sign-up. When this option is ticked, self-registration is enabled. When the Admin GUI is accessed, in the menu list the option Signup becomes available. 
+* Enable sign-up. When this option is ticked, self-registration is enabled. When the Admin GUI is accessed, in the menu list the option Signup becomes available.
-  Obviously this menu item is only visible when signed out. On the Signup page a user can create an email account. 
+  Obviously this menu item is only visible when signed out. On the Signup page a user can create an email account.
-  If your Admin GUI is available to the public internet, this means your Mailu installation basically becomes a free email provider. 
+  If your Admin GUI is available to the public internet, this means your Mailu installation basically becomes a free email provider.
   Use this option with care!
-  
+
 * Comment. Description for the domain. This description is visible on the parent domains list page.
 
 
@@ -414,7 +421,7 @@ The menu item `Webmail` opens the webmail page. This option is only available if
 Client setup
 ------------
 
-The menu item `Client setup` shows all settings for configuring your email client for connecting to Mailu. 
+The menu item `Client setup` shows all settings for configuring your email client for connecting to Mailu.
 
 
 Website

towncrier/newsfragments/1194.feature

@@ -0,0 +1 @@
+Add a credential cache to speedup authentication requests.