Merge #1441 #2090

1441: Rsyslog logging for postfix r=mergify[bot] a=micw


## What type of PR?

enhancement

## What does this PR do?
Changes postfix logging from stdout to rsyslog:
* stdout logging still enabled
* internal test request log messages are filtered out by rsyslog
* optional logging to file via POSTFIX_LOG_FILE env variable

## Prerequistes
Before we can consider review and merge, please make sure the following list is done and checked.
If an entry in not applicable, you can check it or remove it from the list.

- [x] In case of feature or enhancement: documentation updated accordingly
- [x] Unless it's docs or a minor change: add [changelog](https://mailu.io/master/contributors/guide.html#changelog) entry file.


2090: fix 2086 r=mergify[bot] a=nextgens

## What type of PR?

bug-fix

## What does this PR do?

Fix a bug I've introduced in ae8db08bd

### Related issue(s)
- close #2086

Co-authored-by: Michael Wyraz <michael@wyraz.de>
Co-authored-by: Dimitri Huisman <diman@huisman.xyz>
Co-authored-by: Dimitri Huisman <52963853+Diman0@users.noreply.github.com>
Co-authored-by: Florent Daigniere <nextgens@freenetproject.org>

core/dovecot/conf/ham.script

@@ -1,6 +1,6 @@
 #!/bin/bash
-
+{% set hostname,port = ANTISPAM_WEBUI_ADDRESS.split(':') %}
-RSPAMD_HOST="$(getent hosts {{ ANTISPAM_WEBUI_ADDRESS }}|cut -d\  -f1)"
+RSPAMD_HOST="$(getent hosts {{ hostname }}|cut -d\  -f1):{{ port }}"
 if [[ $? -ne 0 ]]
 then
 	echo "Failed to lookup {{ ANTISPAM_WEBUI_ADDRESS }}" >&2

core/dovecot/conf/spam.script

@@ -1,6 +1,6 @@
 #!/bin/bash
-
+{% set hostname,port = ANTISPAM_WEBUI_ADDRESS.split(':') %}
-RSPAMD_HOST="$(getent hosts {{ ANTISPAM_WEBUI_ADDRESS }}|cut -d\  -f1)"
+RSPAMD_HOST="$(getent hosts {{ hostname }}|cut -d\  -f1):{{ port }}"
 if [[ $? -ne 0 ]]
 then
 	echo "Failed to lookup {{ ANTISPAM_WEBUI_ADDRESS }}" >&2

core/postfix/Dockerfile

@@ -23,7 +23,7 @@ RUN apk add --no-cache --virtual .build-deps gcc musl-dev python3-dev
 RUN pip3 install --no-binary :all: postfix-mta-sts-resolver==1.0.1
 RUN apk del .build-deps gcc musl-dev python3-dev
 
-RUN apk add --no-cache postfix postfix-pcre cyrus-sasl-login
+RUN apk add --no-cache postfix postfix-pcre cyrus-sasl-login rsyslog logrotate
 
 COPY conf /conf
 COPY start.py /start.py

core/postfix/conf/logrotate.conf

@@ -0,0 +1,11 @@
+{{POSTFIX_LOG_FILE}} {
+weekly
+rotate 52
+nocompress
+extension log
+create 0644 root root
+      postrotate
+          /bin/kill -HUP $(cat /run/rsyslogd.pid)
+          postfix reload
+      endscript
+}

core/postfix/conf/main.cf

@@ -2,9 +2,6 @@
 # General
 ###############
 
-# Logging configuration
-maillog_file = /dev/stdout
-
 # Main domain and hostname
 mydomain = {{ DOMAIN }}
 myhostname = {{ HOSTNAMES.split(",")[0] }}

core/postfix/conf/rsyslog.conf

@@ -0,0 +1,40 @@
+# rsyslog configuration file
+#
+# For more information see /usr/share/doc/rsyslog-*/rsyslog_conf.html
+# or latest version online at http://www.rsyslog.com/doc/rsyslog_conf.html
+# If you experience problems, see http://www.rsyslog.com/doc/troubleshoot.html
+
+
+#### Global directives ####
+
+# Sets the directory that rsyslog uses for work files.
+$WorkDirectory /var/lib/rsyslog
+
+# Sets default permissions for all log files.
+$FileOwner root
+$FileGroup adm
+$FileCreateMode 0640
+$DirCreateMode 0755
+$Umask 0022
+
+# Reduce repeating messages (default off).
+$RepeatedMsgReduction on
+
+
+#### Modules ####
+
+# Provides support for local system logging (e.g. via logger command).
+module(load="imuxsock")
+
+#### Rules ####
+
+# Discard messages from local test requests
+:msg, contains, "connect from localhost[127.0.0.1]"  ~
+
+{% if POSTFIX_LOG_FILE %}
+# Log mail logs to file
+mail.*							-{{POSTFIX_LOG_FILE}}
+{% endif %}
+
+# Log mail logs to stdout
+mail.*							-/dev/stdout

core/postfix/start.py

@@ -47,6 +47,8 @@ os.environ["FRONT_ADDRESS"] = system.get_host_address_from_environment("FRONT",
 os.environ["ADMIN_ADDRESS"] = system.get_host_address_from_environment("ADMIN", "admin")
 os.environ["ANTISPAM_MILTER_ADDRESS"] = system.get_host_address_from_environment("ANTISPAM_MILTER", "antispam:11332")
 os.environ["LMTP_ADDRESS"] = system.get_host_address_from_environment("LMTP", "imap:2525")
+os.environ["POSTFIX_LOG_SYSLOG"] = os.environ.get("POSTFIX_LOG_SYSLOG","local")
+os.environ["POSTFIX_LOG_FILE"] = os.environ.get("POSTFIX_LOG_FILE", "")
 
 for postfix_file in glob.glob("/conf/*.cf"):
     conf.jinja(postfix_file, os.environ, os.path.join("/etc/postfix", os.path.basename(postfix_file)))
@@ -81,6 +83,16 @@ if "RELAYUSER" in os.environ:
     conf.jinja("/conf/sasl_passwd", os.environ, path)
     os.system("postmap {}".format(path))
 
+# Configure and start local rsyslog server
+conf.jinja("/conf/rsyslog.conf", os.environ, "/etc/rsyslog.conf")
+os.system("/usr/sbin/rsyslogd -n &")
+# Configure logrotate and start crond
+if os.environ["POSTFIX_LOG_FILE"] != "":
+    conf.jinja("/conf/logrotate.conf", os.environ, "/etc/logrotate.d/postfix.conf")
+    os.system("/usr/sbin/crond")
+    if os.path.exists("/overrides/logrotate.conf"):
+        shutil.copyfile("/overrides/logrotate.conf", "/etc/logrotate.d/postfix.conf")
+
 # Run Podop and Postfix
 multiprocessing.Process(target=start_podop).start()
 multiprocessing.Process(target=start_mta_sts_daemon).start()

docs/configuration.rst

@@ -262,3 +262,17 @@ The roundcube service stores configurations in a database.
 - ``ROUNDCUBE_DB_PW``: the database password for roundcube service. (when not ``sqlite``)
 - ``ROUNDCUBE_DB_USER``: the database user for roundcube service. (when not ``sqlite``)
 - ``ROUNDCUBE_DB_NAME``: the database name for roundcube service. (when not ``sqlite``)
+
+
+Mail log settings
+-----------------
+
+By default, all services log directly to stdout/stderr. Logs can be collected by any docker log processing solution.
+
+Postfix writes the logs to a syslog server which logs to stdout. This is used to filter out messages from the healthcheck.
+In some situations, a separate mail log is required (e.g. for legal reasons). The syslog server can be configured to write log files to a volume. It can be configured with the following option:
+
+- ``POSTFIX_LOG_FILE``: The file to log the mail log to. When enabled, the syslog server will also log to stdout.
+
+When ``POSTFIX_LOG_FILE`` is enabled, the logrotate program will automatically rotate the logs every week and keep 52 logs.
+To override the logrotate configuration, create the file logrotate.conf with the desired configuration in the :ref:`Postfix overrides folder<override-label>`.

docs/faq.rst

@@ -263,6 +263,7 @@ correct syntax. The following file names will be taken as override configuration
    - All ``$ROOT/overrides/postfix/*.map`` files
    - For both ``postfix.cf`` and ``postfix.master``, you need to put one configuration per line, as they are fed line-by-line
      to postfix.
+   - ``logrotate.conf`` as ``$ROOT/overrides/postfix/logrotate.conf`` - Replaces the logrotate.conf file used for rotating ``POSTFIX_LOG_FILE``.
 - `Dovecot`_ - ``dovecot.conf`` in dovecot sub-directory;
 - `Nginx`_ - All ``*.conf`` files in the ``nginx`` sub-directory;
 - `Rspamd`_ - All files in the ``rspamd`` sub-directory.

towncrier/newsfragments/1441.feature

@@ -0,0 +1,6 @@
+Introduces postfix logging via syslog with these features:
+- stdout logging still enabled
+- internal test request log messages (healthcheck) are filtered out by rsyslog
+- optional logging to file via POSTFIX_LOG_FILE env variable
+To use logging to file configure in mailu.env
+- ``POSTFIX_LOG_FILE``: The file to log the mail log to