Merge #2328

2328: Feature: Configurable default spam threshold used for new users r=mergify[bot] a=enginefeeder101

## What type of PR?

Feature

## What does this PR do?

This PR adds functionality to set a custom default spam threshold
for new users. The environment variable ``DEFAULT_SPAM_THRESHOLD`` is
used for this purpose. When not set, it defaults back to 80%, as the
default value was before.

If ``DEFAULT_SPAM_THRESHOLD`` is set to a value that Python cannot
parse as an integer, a ValueError is thrown. There is no error handling
for that case built-in. Should that be done?

## Prerequisites
Before we can consider review and merge, please make sure the following list is done and checked.
If an entry in not applicable, you can check it or remove it from the list.

- [x] In case of feature or enhancement: documentation updated accordingly
- [x] Unless it's docs or a minor change: add [changelog](https://mailu.io/master/contributors/workflow.html#changelog) entry file.


Co-authored-by: enginefeeder101 <enginefeeder101@users.noreply.github.com>
Co-authored-by: Dimitri Huisman <diman@huisman.xyz>

core/admin/mailu/configuration.py

@@ -76,6 +76,7 @@ DEFAULT_CONFIG = {
     'CREDENTIAL_ROUNDS': 12,
     'TLS_PERMISSIVE': True,
     'TZ': 'Etc/UTC',
+    'DEFAULT_SPAM_THRESHOLD': 80,
     # Host settings
     'HOST_IMAP': 'imap',
     'HOST_LMTP': 'imap:2525',
@@ -163,6 +164,7 @@ class ConfigManager:
         self.config['MESSAGE_RATELIMIT_EXEMPTION'] = set([s for s in self.config['MESSAGE_RATELIMIT_EXEMPTION'].lower().replace(' ', '').split(',') if s])
         self.config['HOSTNAMES'] = ','.join(hostnames)
         self.config['HOSTNAME'] = hostnames[0]
+        self.config['DEFAULT_SPAM_THRESHOLD'] = int(self.config['DEFAULT_SPAM_THRESHOLD'])
 
         # update the app config
         app.config.update(self.config)

core/admin/mailu/models.py

@@ -518,7 +518,7 @@ class User(Base, Email):
     displayed_name = db.Column(db.String(160), nullable=False, default='')
     spam_enabled = db.Column(db.Boolean, nullable=False, default=True)
     spam_mark_as_read = db.Column(db.Boolean, nullable=False, default=True)
-    spam_threshold = db.Column(db.Integer, nullable=False, default=80)
+    spam_threshold = db.Column(db.Integer, nullable=False, default=lambda:int(app.config.get("DEFAULT_SPAM_THRESHOLD", 80)))
 
     # Flask-login attributes
     is_authenticated = True

docs/antispam.rst

@@ -21,28 +21,30 @@ Rspamd rejects non-compliant email messages and email messages that contain viru
 1. When an email message is received or send by Postfix, it is scanned by Rspamd. If the email message receives a spam score between 6 and 15, a header is added to the email message with the spam score. This is an example of an email header for a spam score of 14::
 
     X-Spamd-Bar: ++++++++++++++
-    X-Spam-Level: ************** 
+    X-Spam-Level: **************
     Authentication-Results: test.mailu.io;
        dkim=pass header.d=example.com header.s=mailing header.b=ABCDE;
        dkim=pass header.d=example.com header.s=mailing header.b=ABCDE;
        dmarc=pass (policy=none) header.from=eventim.de;
        spf=pass (test.mailu.io: domain of return@example.com designates 11.22.33.44 as permitted sender) smtp.mailfrom=return@example.com
-    X-Spam: Yes    
+    X-Spam: Yes
- 
+
 2. Dovecot is then responsible for classifying the email message to the Junk folder based on user preferences. It works as following:
 
    * In the administration web interface, under settings under Antispam 'Enable spam filter' must be ticked. If this option is disabled, then all email messages will automatically go to the inbox folder. Except for email messages with a score of 15 or higher, as these email messages are rejected by Rspamd.
-   
+
    * In the administration web interface, under settings under Antispam, the user defined spam filter tolerance must be configured. The default value is 80%. The lower the spam filter tolerance, the more false positives (ham classified as spam). The user can change this setting to finetune when an email message is classified as spam.
 
    * Dovecot extracts the X-Spam-Level email header from the email message and converts the spam score (0 - 15) to a 0 - 100 percent scale. This spam score is compared with the user defined spam filter tolerance. If the spam score is lower than the user defined spam filter tolerance, then the email message is accepted. In logic:
-     
+
      If <spam_score in percent> is greater than <spam filter tolerance>, then move the email message to the spam folder and mark the email message as read.
-   
+
    For example if the user defined spam filter tolerance is set to 80%(default) and the spam score of an email message is 10:
-      
+
    66% (10/15) is less than 80%, so the email is classified as ham. This email message will go to the inbox folder. If the user wants email messages with a score of 10 (66%) to be classified as spam, then the user defined spam filter tolerance can be lowered to 65% in the administration web interface.
-   
+
+   The default spam filter tolerance used for new users can be configured using the environment variable ``DEFAULT_SPAM_THRESHOLD``. See also :ref:`common_cfg` in the configuration reference.
+
 .. image:: assets/screenshots/SpamFiltering.png
 
 The location in the administration web interface where the spam filter and spam filter tolerance can be configured.
@@ -61,7 +63,7 @@ If you already have an existing mailbox and want Mailu to learn them all as ham
   rspamc -h antispam:11334 -P mailu -f 13 fuzzy_add /mail/user\@example.com/.Ham_Learn/cur/
   rspamc -h antispam:11334 -P mailu learn_ham /mail/user\@example.com/.Ham_Learn/cur/
 
-This should learn every file located in the ``Ham_Learn`` folder from user@example.com 
+This should learn every file located in the ``Ham_Learn`` folder from user@example.com
 
 Likewise, to learn all messages within the folder ``Spam_Learn`` as spam messages :
 
@@ -77,7 +79,7 @@ Likewise, to learn all messages within the folder ``Spam_Learn`` as spam message
 How can I block emails from a domain?
 -------------------------------------
 
-Via the multimap filter it is possible to block emails from a sender domain. See the `official rspamd documentation`_ for more information. A local blacklist that contains the domains to be blocked can be configured via the :ref:`Rspamd overrides folder <override-label>`. 
+Via the multimap filter it is possible to block emails from a sender domain. See the `official rspamd documentation`_ for more information. A local blacklist that contains the domains to be blocked can be configured via the :ref:`Rspamd overrides folder <override-label>`.
 
 The following steps have to be taken to configure an additional symbol (rule) that uses the multimap filter to block emails from sender domain.
 
@@ -99,7 +101,7 @@ The following steps have to be taken to configure an additional symbol (rule) th
     }
 
    Note the "action = "reject";" line. This is a so-called pre-filter. No further filters/rules are processed when a pre-filter is used. If you omit this line, then the configured score will be added to the total score of the email message. Depending on the end-score after processing all rules, a verdict is made. To override this, you can add the action line. When this symbol (rule) is fired, then this action is immediately taken and no further processing occurs. You can use the following actions:
-  
+
    * discard: drop an email message, but return success for sender (should be used merely in special cases)
 
    * reject: reject the email message. This enables the actual blocking of mails from the domain.
@@ -112,9 +114,9 @@ The following steps have to be taken to configure an additional symbol (rule) th
 
    To move an email message to the Junk (Spam) folder, a score of 15 can be used in combination with the action "add header".
    The above example configuration will reject all emails send from domains that are listed in '/etc/rspamd/override.d/blacklist.inc'.
-   
 
-2. In the Rspamd overrides folder create a map that contains the domains to be blocked. You can use # to add comments. 
+
+2. In the Rspamd overrides folder create a map that contains the domains to be blocked. You can use # to add comments.
    Create the file /mailu/overrides/rspamd/blacklist.inc with the following contents:
 
    .. code-block:: bash
@@ -130,12 +132,12 @@ The following steps have to be taken to configure an additional symbol (rule) th
      docker-compose scale antispam=0
      docker-compose scale antispam=1
 
-4. (Optional) Check if the custom symbol is loaded. To access the Rspamd webgui, log in the Mailu administration web interface with a user that is an administrator and go to Antispam. In Rspamd webgui go to tab Symbols. Change the group drop-down box to local_bl. The following additional rule will be listed. 
+4. (Optional) Check if the custom symbol is loaded. To access the Rspamd webgui, log in the Mailu administration web interface with a user that is an administrator and go to Antispam. In Rspamd webgui go to tab Symbols. Change the group drop-down box to local_bl. The following additional rule will be listed.
 
    .. image:: assets/screenshots/RspamdSymbolBlacklist.png
 
    The symbol is only displayed if the symbol has no pre-filter (action= line) configured. Changes made in this screen are not saved to the configuration file.
-  
+
 5. Check if the map is available. In rspamd webgui to to configuration. A map is available with the path:
    /etc/rspamd/override.d/blacklist.inc	Senders domain part is on the local blacklist
 

docs/configuration.rst

@@ -63,6 +63,8 @@ there is a good way to disable rate limiting altogether.
 The ``TLS_FLAVOR`` sets how Mailu handles TLS connections. Setting this value to
 ``notls`` will cause Mailu not to serve any web content! More on :ref:`tls_flavor`.
 
+The ``DEFAULT_SPAM_THRESHOLD`` (default: 80) is the default spam tolerance used when creating a new user.
+
 Mail settings
 -------------
 
@@ -168,7 +170,7 @@ To have the account created automatically, you just need to define a few environ
 - ``INITIAL_ADMIN_DOMAIN``: the domain appendix: Most probably identical to the ``DOMAIN`` variable.
 - ``INITIAL_ADMIN_PW``: the admin password.
 - ``INITIAL_ADMIN_MODE``: use one of the options below for configuring how the admin account must be created:
-  
+
   - ``create``: (default) creates a new admin account and raises an exception when it already exists.
   - ``ifmissing``: creates a new admin account when the admin account does not exist.
   - ``update``: creates a new admin account when it does not exist, or update the password of an existing admin account.
@@ -187,6 +189,8 @@ An example:
 
 Depending on your particular deployment you most probably will want to change the default.
 
+.. _advanced_cfg:
+
 Advanced settings
 -----------------
 

setup/flavors/compose/mailu.env

@@ -173,6 +173,9 @@ LOG_LEVEL=WARNING
 # Timezone for the Mailu containers. See this link for all possible values https://en.wikipedia.org/wiki/List_of_tz_database_time_zones
 TZ=Etc/UTC
 
+# Default spam threshold used for new users
+DEFAULT_SPAM_THRESHOLD=80
+
 ###################################
 # Database settings
 ###################################

towncrier/newsfragments/2328.feature

@@ -0,0 +1 @@
+Configurable default spam threshold used for new users