Merge #1296 #1322 #1337 #1358

1296: fetchmail: print unhandled exceptions, but don't crash r=Nebukadneza a=Al2Klimov

fixes #1295

1322: Bump validators from 0.12.5 to 0.12.6 in /core/admin r=Nebukadneza a=dependabot[bot]

Bumps [validators](https://github.com/kvesteri/validators) from 0.12.5 to 0.12.6.
<details>
<summary>Changelog</summary>

*Sourced from [validators's changelog](https://github.com/kvesteri/validators/blob/master/CHANGES.rst).*

> 0.12.6 (2019-05-08)
> ^^^^^^^^^^^^^^^^^^^
> 
> - Fixed domain validator for single character domains ([#118](https://github-redirect.dependabot.com/kvesteri/validators/issues/118), pull request courtesy kingbuzzman)
</details>
<details>
<summary>Commits</summary>

- See full diff in [compare view](https://github.com/kvesteri/validators/commits)
</details>
<br />

[![Dependabot compatibility score](https://dependabot-badges.githubapp.com/badges/compatibility_score?dependency-name=validators&package-manager=pip&previous-version=0.12.5&new-version=0.12.6)](https://help.github.com/articles/configuring-automated-security-fixes)

Dependabot will resolve any conflicts with this PR as long as you don't alter it yourself. You can also trigger a rebase manually by commenting `@dependabot rebase`.

[//]: # (dependabot-automerge-start)
[//]: # (dependabot-automerge-end)

---

<details>
<summary>Dependabot commands and options</summary>
<br />

You can trigger Dependabot actions by commenting on this PR:
- `@dependabot rebase` will rebase this PR
- `@dependabot recreate` will recreate this PR, overwriting any edits that have been made to it
- `@dependabot merge` will merge this PR after your CI passes on it
- `@dependabot squash and merge` will squash and merge this PR after your CI passes on it
- `@dependabot cancel merge` will cancel a previously requested merge and block automerging
- `@dependabot reopen` will reopen this PR if it is closed
- `@dependabot ignore this [patch|minor|major] version` will close this PR and stop Dependabot creating any more for this minor/major version (unless you reopen the PR or upgrade to it yourself)
- `@dependabot ignore this dependency` will close this PR and stop Dependabot creating any more for this dependency (unless you reopen the PR or upgrade to it yourself)
- `@dependabot use these labels` will set the current labels as the default for future PRs for this repo and language
- `@dependabot use these reviewers` will set the current reviewers as the default for future PRs for this repo and language
- `@dependabot use these assignees` will set the current assignees as the default for future PRs for this repo and language
- `@dependabot use this milestone` will set the current milestone as the default for future PRs for this repo and language

You can disable automated security fix PRs for this repo from the [Security Alerts page](https://github.com/Mailu/Mailu/network/alerts).

</details>

1337: Add IPv6 to allow_nets r=Nebukadneza a=PhilRW

Roundcube was not connecting to sieve with IPv6 enabled.

Fixes #1336

1358: Add port to relay if it contains a colon r=Nebukadneza a=PhilRW

## What type of PR?

enhancement

## What does this PR do?

Allows relaying domains to non-standard SMTP ports by appending `:port` to the destination host/IP. E.g., `mx1.internal:2525`

### Related issue(s)

Closes #1357 


## Prerequistes
Before we can consider review and merge, please make sure the following list is done and checked.
If an entry in not applicable, you can check it or remove it from the list.

- [x] In case of feature or enhancement: documentation updated accordingly
- [x] Unless it's docs or a minor change: add [changelog](https://mailu.io/master/contributors/guide.html#changelog) entry file.


Co-authored-by: Alexander A. Klimov <grandmaster@al2klimov.de>
Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com>
Co-authored-by: Philip Rosenberg-Watt <p.rosenberg-watt@cablelabs.com>

core/admin/mailu/configuration.py

@@ -67,6 +67,7 @@ DEFAULT_CONFIG = {
     'HOST_REDIS': 'redis',
     'HOST_FRONT': 'front',
     'SUBNET': '192.168.203.0/24',
+    'SUBNET6': None,
     'POD_ADDRESS_RANGE': None
 }
 

core/admin/mailu/internal/views/dovecot.py

@@ -11,6 +11,8 @@ def dovecot_passdb_dict(user_email):
     user = models.User.query.get(user_email) or flask.abort(404)
     allow_nets = []
     allow_nets.append(app.config["SUBNET"])
+    if app.config["SUBNET6"]:
+        allow_nets.append(app.config["SUBNET6"])
     if app.config["POD_ADDRESS_RANGE"]:
         allow_nets.append(app.config["POD_ADDRESS_RANGE"])
     return flask.jsonify({

core/admin/mailu/internal/views/postfix.py

@@ -37,7 +37,11 @@ def postfix_transport(email):
         return flask.abort(404)
     localpart, domain_name = models.Email.resolve_domain(email)
     relay = models.Relay.query.get(domain_name) or flask.abort(404)
-    return flask.jsonify("smtp:[{}]".format(relay.smtp))
+    ret = "smtp:[{0}]".format(relay.smtp)
+    if ":" in relay.smtp:
+        split = relay.smtp.split(':')
+        ret = "smtp:[{0}]:{1}".format(split[0], split[1])
+    return flask.jsonify(ret)
 
 
 @internal.route("/postfix/recipient/map/<path:recipient>")

core/admin/requirements-prod.txt

@@ -44,7 +44,7 @@ SQLAlchemy==1.3.3
 srslib==0.1.4
 tabulate==0.8.3
 tenacity==5.0.4
-validators==0.12.5
+validators==0.12.6
 visitor==0.1.3
 Werkzeug==0.15.3
 WTForms==2.2.1

optional/fetchmail/fetchmail.py

@@ -8,6 +8,7 @@ import subprocess
 import re
 import requests
 import sys
+import traceback
 
 
 FETCHMAIL = """
@@ -45,47 +46,50 @@ def fetchmail(fetchmailrc):
 
 
 def run(debug):
-    fetches = requests.get("http://admin/internal/fetch").json()
-    smtphost, smtpport = extract_host_port(os.environ.get("HOST_SMTP", "smtp"), None)
-    if smtpport is None:
-        smtphostport = smtphost
-    else:
-        smtphostport = "%s/%d" % (smtphost, smtpport)
-    for fetch in fetches:
-        fetchmailrc = ""
-        options = "options antispam 501, 504, 550, 553, 554"
-        options += " sslmode wrapped" if fetch["tls"] else ""
-        options += " keep" if fetch["keep"] else " fetchall"
-        fetchmailrc += RC_LINE.format(
-            user_email=escape_rc_string(fetch["user_email"]),
-            protocol=fetch["protocol"],
-            host=escape_rc_string(fetch["host"]),
-            port=fetch["port"],
-            smtphost=smtphostport,
-            username=escape_rc_string(fetch["username"]),
-            password=escape_rc_string(fetch["password"]),
-            options=options
-        )
-        if debug:
-            print(fetchmailrc)
-        try:
-            print(fetchmail(fetchmailrc))
-            error_message = ""
-        except subprocess.CalledProcessError as error:
-            error_message = error.output.decode("utf8")
-            # No mail is not an error
-            if not error_message.startswith("fetchmail: No mail"):
-                print(error_message)
-            user_info = "for %s at %s" % (fetch["user_email"], fetch["host"])
-            # Number of messages seen is not a error as well
-            if ("messages" in error_message and
-                    "(seen " in error_message and
-                    user_info in error_message):
-                print(error_message)
-        finally:
-            requests.post("http://admin/internal/fetch/{}".format(fetch["id"]),
-                json=error_message.split("\n")[0]
+    try:
+        fetches = requests.get("http://admin/internal/fetch").json()
+        smtphost, smtpport = extract_host_port(os.environ.get("HOST_SMTP", "smtp"), None)
+        if smtpport is None:
+            smtphostport = smtphost
+        else:
+            smtphostport = "%s/%d" % (smtphost, smtpport)
+        for fetch in fetches:
+            fetchmailrc = ""
+            options = "options antispam 501, 504, 550, 553, 554"
+            options += " sslmode wrapped" if fetch["tls"] else ""
+            options += " keep" if fetch["keep"] else " fetchall"
+            fetchmailrc += RC_LINE.format(
+                user_email=escape_rc_string(fetch["user_email"]),
+                protocol=fetch["protocol"],
+                host=escape_rc_string(fetch["host"]),
+                port=fetch["port"],
+                smtphost=smtphostport,
+                username=escape_rc_string(fetch["username"]),
+                password=escape_rc_string(fetch["password"]),
+                options=options
             )
+            if debug:
+                print(fetchmailrc)
+            try:
+                print(fetchmail(fetchmailrc))
+                error_message = ""
+            except subprocess.CalledProcessError as error:
+                error_message = error.output.decode("utf8")
+                # No mail is not an error
+                if not error_message.startswith("fetchmail: No mail"):
+                    print(error_message)
+                user_info = "for %s at %s" % (fetch["user_email"], fetch["host"])
+                # Number of messages seen is not a error as well
+                if ("messages" in error_message and
+                        "(seen " in error_message and
+                        user_info in error_message):
+                    print(error_message)
+            finally:
+                requests.post("http://admin/internal/fetch/{}".format(fetch["id"]),
+                    json=error_message.split("\n")[0]
+                )
+    except Exception:
+        traceback.print_exc()
 
 
 if __name__ == "__main__":

towncrier/newsfragments/1357.feature

@@ -0,0 +1 @@
+Relay a domain to a nonstandard SMTP port by adding ":<port_num>" to the remote hostname or IP address.