2403: Feature: switch CI/CD from build to buildx r=mergify[bot] a=Diman0
## What type of PR?
Feature and enhancement
## What does this PR do?
Switch from docker build to buildx for CI/CD.
- The main workflow file has been optimised and simplified.
- Images are built in parallel when building locally resulting in much faster build times.
- The github action workflow is about 50% faster.
- Arm images are built as well. These images are not tested due to restrictions of github actions (no arm runners). The tags of the images have -arm appended to it. The arm images are built on merge on master and release branch (x.y). They do not influence the normal CI/CD workflow used for bors (for PR) and real releases (merge on master and branch x.y for x86_64).
- Arm images (and normal x86_64 images) can also be built locally.
- Reusable workflow is introduced for building, testing and deploying the images. This allows the workflow to be reused for other purposes in the future.
- Workflow can be manually triggered. This allows forked Mailu projects to also use the workflow for building images.
The main workflow makes use of github actions cache to store the cache layer. This layer is used to quickly rebuilt the images in the testing step and deploy step.
Unfortunately the building the arm images fails sometimes due to timeouts. Sometimes the connection to github actions cache is very slow. Restarting the workflow from the last failed step resolves this. I have not observed this with the normal build.
Just as previous time, you can use a forked project for testing the changes (https://github.com/Diman0/Mailu_Fork). You should still have owner access. I have created branch 1.11 for testing. You can see I already push 4 times to branch 1.11 (current version is 1.11.3).
### Related issue(s)
- Mention an issue like: #001
- closes#2383
- closes#1830
- closes#1200
## Prerequisites
Before we can consider review and merge, please make sure the following list is done and checked.
If an entry in not applicable, you can check it or remove it from the list.
- [x] In case of feature or enhancement: documentation updated accordingly
- [x] Unless it's docs or a minor change: add [changelog](https://mailu.io/master/contributors/workflow.html#changelog) entry file.
Co-authored-by: Dimitri Huisman <diman@huisman.xyz>
2221: Add support for custom NGINX config r=mergify[bot] a=easybe
## What type of PR?
enhancement
## What does this PR do?
Add support for custom NGINX config. Including *.conf files in /etc/nginx/conf.d same as the default NGINX configuration gives the user more flexibility.
### Related issue(s)
## Prerequisites
Before we can consider review and merge, please make sure the following list is done and checked.
If an entry in not applicable, you can check it or remove it from the list.
- [ ] In case of feature or enhancement: documentation updated accordingly
- [x] Unless it's docs or a minor change: add [changelog](https://mailu.io/master/contributors/workflow.html#changelog) entry file.
Co-authored-by: Ezra Buehler <ezra@easyb.ch>
2357: Switch to ffdhe3072 to enable RFC 7919 r=mergify[bot] a=nextgens
## What type of PR?
enhancement
## What does this PR do?
The idea being:
- it's a "nothing up my sleeves" group
- it may help shave off some bytes of the SSL handshake; That being said, I doubt that clients that are modern enough to support this RFC won't offer an EC kex
https://raw.githubusercontent.com/internetstandards/dhe_groups/master/ffdhe3072.pem
## Prerequisites
Before we can consider review and merge, please make sure the following list is done and checked.
If an entry in not applicable, you can check it or remove it from the list.
- [ ] In case of feature or enhancement: documentation updated accordingly
- [x] Unless it's docs or a minor change: add [changelog](https://mailu.io/master/contributors/workflow.html#changelog) entry file.
Co-authored-by: Florent Daigniere <nextgens@freenetproject.org>
2404: Forwarding emails option in user settings did not support 1 letter do… r=mergify[bot] a=Diman0
…mains.
## What type of PR?
Bug-fix
## What does this PR do?
Forwarding emails option in user setting did not support 1 letter domains. The regex for checking the validity of multiple email addresses string has been modified to allow 1 letter domains and to allow 1 letter local part.
### Related issue(s)
- closes#2402
## Prerequisites
Before we can consider review and merge, please make sure the following list is done and checked.
If an entry in not applicable, you can check it or remove it from the list.
- [n/a] In case of feature or enhancement: documentation updated accordingly
- [x] Unless it's docs or a minor change: add [changelog](https://mailu.io/master/contributors/workflow.html#changelog) entry file.
Co-authored-by: Dimitri Huisman <diman@huisman.xyz>
2397: Fix resolving alias addresses for postfix when target is a punycode domain r=mergify[bot] a=ghostwheel42
## What type of PR?
bug-fix
## What does this PR do?
- fix splitting of localpart in resolve_destination
- idna-enode domain-part of email addresses before returning to postfix
### Related issue(s)
- closes#2393
Co-authored-by: Alexander Graf <ghostwheel42@users.noreply.github.com>
- The main workflow file has been optimised and simplified.
- Images are built in parallel when building locally resulting in faster build times.
- The github action workflow is about 50% faster.
- Arm images are built as well. These images are not tested due to restrictions of github actions (no arm runners). The tags of the images have -arm appended to it.
- Arm images can also be built locally.
- Reusable workflow is introduced for building, testing and deploying the images.
This allows the workflow to be reused for other purposes in the future.
- Workflow can be manually triggered. This allows forked Mailu projects to also use the workflow for building images.
Remove unneeded IF statement in /admin block in nginx.conf of front.
Fix contributions made to Dockerfile, add missing trailing \ and add back curl
Change healthcheck to monitoring page of fpm. Now we check nginx and fpm.
This is required for passing rDNS/ptr information to postfix.
The mail proxy uses the resolver info for passing XCLIENT info.
See http://nginx.org/en/docs/mail/ngx_mail_proxy_module.html#xclient
Without this info rspamd will flag all messages with DHFILTER_HOSTNAME_UNKNOWN due to the missing rDNS/ptr info.
Per requested changes added the ``DEFAULT_SPAM_THRESHOLD`` to the main
application configuration dictionary in ``configuration.py`` and updated
``models.py`` accordingly.
No error handling is added, as that was not required.
This commit adds functionality to set a custom default spam threshold
for new users. The environment variable ``DEFAULT_SPAM_THRESHOLD`` can
be used for this purpose. When not set, it defaults back to 80%, as the
default value was before
If ``DEFAULT_SPAM_THRESHOLD`` is set to a value that Python cannot
parse as an integer, a ValueError is thrown. There is no error handling
for that case built-in.
2325: postfix: wrap IPv6 CIDRs in square brackets for RELAYNETS r=mergify[bot] a=pommi
## What type of PR?
bug-fix
## What does this PR do?
This PR wraps IPv6 CIDRs in the `RELAYNETS` environment variable in square brackets for the postfix configuration.
The `RELAYNETS` environment variable is used for configuring both postfix `mynetworks` and rspamd `local_networks`. Postfix requires IPv6 addresses to be wrapped in square brackets (eg. `[2001:db8::]/64`).
When an IPv6 address is not wrapped in square brackets in the postfix configuration for `mynetworks` it results in this error while processing an incoming email from an IPv6 sender:
```
postfix/smtpd[340]: warning: 2001:db8::/64 is unavailable. unsupported dictionary type: 2001
postfix/smtpd[340]: warning: smtpd_client_event_limit_exceptions: 2001:db8::/64: table lookup problem
```
The sender sees an error and the incoming email is refused:
```
451 4.3.0 <unknown[2001:xxx:xxx:xxx:xxx:xxx:xxx:xxx]>: Temporary lookup failure
```
I tried to work around this issue by wrapping the IPv6 CIDR in square brackets in the `RELAYNETS` environment variable, but it segfaults rspamd, because it can't deal with this non-standard IPv6 notation used by postfix:
```
kernel: [4305632.603704] rspamd[1954299]: segfault at 0 ip 00007fb848983871 sp 00007ffe02cc6d1
8 error 4 in ld-musl-x86_64.so.1[7fb848948000+48000]
```
### Related issue(s)
- #2293
- #2272
## Prerequisites
Before we can consider review and merge, please make sure the following list is done and checked.
If an entry in not applicable, you can check it or remove it from the list.
**No changelog or documentation necessary for this minor change.**
Co-authored-by: Pim van den Berg <pim@nethuis.nl>
The RELAYNETS environment variable is used for configuring both postfix
`mynetworks` and rspamd `local_networks`. Postfix requires IPv6
addresses to be wrapped in square brackets (eg. [2001:db8::]/64).
2310: Update deprecated rspamd config option r=mergify[bot] a=henniaufmrenni
## What type of PR?
Configuration update
## What does this PR do?
This is just a small config update to get rid of the following warning message:
`lua; antivirus.lua:109: CLAM_VIRUS [clamav]: Using attachments_only is deprecated. Please use scan_mime_parts = true instead`
As per the rspamd documentation https://rspamd.com/doc/modules/antivirus.html
> attachments_only = true; # Before 1.8.1
> scan_mime_parts = true; # After 1.8.1
The currently used version of rspamd is 3.1.
## Prerequisites
Before we can consider review and merge, please make sure the following list is done and checked.
If an entry in not applicable, you can check it or remove it from the list.
- [x] In case of feature or enhancement: documentation updated accordingly
- [x] Unless it's docs or a minor change: add [changelog](https://mailu.io/master/contributors/workflow.html#changelog) entry file.
Co-authored-by: henniaufmrenni <henniaufmrenni@keinvergessen.org>
2338: Update X-XSS-Protection to current recommendation r=mergify[bot] a=AvverbioPronome
See:
- https://developer.mozilla.org/en-US/docs/Web/HTTP/Headers/X-XSS-Protection and
- https://cheatsheetseries.owasp.org/cheatsheets/HTTP_Headers_Cheat_Sheet.html#x-xss-protection
## What type of PR?
Slight enhancement
## What does this PR do?
This PR turns off the XSS auditor in the few browsers that still have one.
## Prerequisites
Before we can consider review and merge, please make sure the following list is done and checked.
If an entry in not applicable, you can check it or remove it from the list.
- [ ?] In case of feature or enhancement: documentation updated accordingly
- [x ] Unless it's docs or a minor change: add [changelog](https://mailu.io/master/contributors/workflow.html#changelog) entry file.
Co-authored-by: Giuseppe C <1191978+AvverbioPronome@users.noreply.github.com>
Co-authored-by: Your Name <you@example.com>
2323: Fix Postfix FileExistsError on startup r=mergify[bot] a=Pumba98
## What type of PR?
bug-fix
## What does this PR do?
I'm running mailu with the mailu helm-chart on kubernetes. Sometimes when a Pod restarts I get the following error during startup:
```
Process Process-1:
Traceback (most recent call last):
File "/usr/lib/python3.9/multiprocessing/process.py", line 315, in _bootstrap
self.run()
File "/usr/lib/python3.9/multiprocessing/process.py", line 108, in run
self._target(*self._args, **self._kwargs)
File "/start.py", line 18, in start_podop
os.mkdir('/dev/shm/postfix',mode=0o700)
FileExistsError: [Errno 17] File exists: '/dev/shm/postfix'
INFO:MAIN:MTA-STS daemon starting...
```
But that does not prevent the container startup. When mails arrive it will fail with something like:
```
postfix/trivial-rewrite[94979]: warning: connect to /tmp/podop.socket: No such file or directory
postfix/trivial-rewrite[94979]: warning: table socketmap:unix:/tmp/podop.socket:transport lookup error: No such file or directory
postfix/trivial-rewrite[94979]: warning: socketmap:unix:/tmp/podop.socket:transport lookup error for "*"
```
I'm running this quick fix now since almost two months without problems. Maybe you got a better approach how to solve this, but this works fine for me.
### Related issue(s)
- none
## Prerequisites
Before we can consider review and merge, please make sure the following list is done and checked.
If an entry in not applicable, you can check it or remove it from the list.
<!--
- [ ] In case of feature or enhancement: documentation updated accordingly
- [ ] Unless it's docs or a minor change: add [changelog](https://mailu.io/master/contributors/workflow.html#changelog) entry file.
--->
**No changelog or documentation necessary for this minor change.**
Co-authored-by: Pumba98 <mail@pumba98.de>
Co-authored-by: Florent Daigniere <nextgens@users.noreply.github.com>
This gets rid of the following error message:
lua; antivirus.lua:109: CLAM_VIRUS [clamav]: Using attachments_only is deprecated. Please use scan_mime_parts = true instead
As per the rspamd documentation https://rspamd.com/doc/modules/antivirus.html
attachments_only = true; # Before 1.8.1
scan_mime_parts = true; # After 1.8.1
The currently used version is rspamd 3.1.
2302: Update alpine-linux to 3.14.5 - Zlib security FIX r=mergify[bot] a=willofr
## What type of PR?
Security fix
## What does this PR do?
Update alpine docker image to alpine-3.14.5
- closes#2291
## Prerequisites
Before we can consider review and merge, please make sure the following list is done and checked.
If an entry in not applicable, you can check it or remove it from the list.
- [x] Unless it's docs or a minor change: add [changelog](https://mailu.io/master/contributors/workflow.html#changelog) entry file.
Co-authored-by: Will <will@packer-output-c8fcfb40-3d93-4475-8f87-e14a9dd683b6>
2276: Autoconfig of email clients r=mergify[bot] a=nextgens
## What type of PR?
Feature
## What does this PR do?
It provides auto-configuration templates for email clients and encourages them to use implicit TLS (see https://nostarttls.secvuln.info/)
There are numerous caveats:
- it will only work if suitable DNS records are created and certificates obtained (autoconfig, autodiscover, ...)
- the mobileconfig file isn't signed
- the credentials will be prompted... we could/should provision a token on each request instead
- it currently doesn't advertise caldav
- it's IMAP only
### Related issue(s)
- close#224
## Prerequisites
Before we can consider review and merge, please make sure the following list is done and checked.
If an entry in not applicable, you can check it or remove it from the list.
- [x] In case of feature or enhancement: documentation updated accordingly
- [x] Unless it's docs or a minor change: add [changelog](https://mailu.io/master/contributors/workflow.html#changelog) entry file.
Co-authored-by: Florent Daigniere <nextgens@freenetproject.org>
2281: Update alpine-linux to 3.14.4 - OpenSSL security FIX r=mergify[bot] a=willofr
## What type of PR?
Security fix
## What does this PR do?
Update Dockerfiles to use alpine-linux 3.14.4 which contains a security fix for openssl
https://alpinelinux.org/posts/Alpine-3.12.10-3.13.8-3.14.4-released.html
## Prerequisites
Before we can consider review and merge, please make sure the following list is done and checked.
If an entry in not applicable, you can check it or remove it from the list.
- [x] Unless it's docs or a minor change: add [changelog](https://mailu.io/master/contributors/workflow.html#changelog) entry file.
2285: Update names of language json files r=mergify[bot] a=ghostwheel42
## What type of PR?
bug-fix
## What does this PR do?
language json files of datatables i18n have been renamed
this updates the mappings to the current names
2286: Fix typo in Traefik reverse proxy docs r=mergify[bot] a=ghostwheel42
Slight typo in the Traefik reverse proxy docs. Found through running into the issue on my own instance.
## What type of PR?
documentation
## What does this PR do?
Adds #2282 to master
2287: Fix typo in docs: cert not certs r=mergify[bot] a=ghostwheel42
## What type of PR?
documentation
## What does this PR do?
just a typo
Co-authored-by: Will <will@packer-output-c8fcfb40-3d93-4475-8f87-e14a9dd683b6>
Co-authored-by: willofr <willofr@users.noreply.github.com>
Co-authored-by: Alexander Graf <ghostwheel42@users.noreply.github.com>
Co-authored-by: DAHPr0gram3r <cbillwork02@gmail.com>
2284: Fixing AUTH_RATELIMIT_IP not working on imap/pop3/smtp r=mergify[bot] a=fischerscode
#2283
## What type of PR?
bug-fix
## What does this PR do?
This fixes AUTH_RATELIMIT_IP not working on imap/pop3/smtp.
### Related issue(s)
closes#2283
## Prerequisites
Before we can consider review and merge, please make sure the following list is done and checked.
If an entry in not applicable, you can check it or remove it from the list.
- [x] Unless it's docs or a minor change: add [changelog](https://mailu.io/master/contributors/workflow.html#changelog) entry file.
Co-authored-by: Maximilian Fischer <github@maaeps.de>
Co-authored-by: Alexander Graf <ghostwheel42@users.noreply.github.com>
2255: Create a polite and turtle delivery queue to accommodate destinations that expect emails to be sent slowly r=mergify[bot] a=nextgens
## What type of PR?
Feature
## What does this PR do?
Create a polite and turtle delivery queue to accommodate destinations that expect emails to be sent slowly
### Related issue(s)
- closes#2213
## Prerequisites
Before we can consider review and merge, please make sure the following list is done and checked.
If an entry in not applicable, you can check it or remove it from the list.
- [x] In case of feature or enhancement: documentation updated accordingly
- [x] Unless it's docs or a minor change: add [changelog](https://mailu.io/master/contributors/workflow.html#changelog) entry file.
Co-authored-by: Florent Daigniere <nextgens@freenetproject.org>
2210: Add input validation for domain creation r=mergify[bot] a=0pc0deFR
## What type of PR?
bug-fix
## What does this PR do?
This patch add the input validation for domain creation.
### Related issue(s)
- Mention an issue like: #1817
- Auto close an issue like: closes#1817
Co-authored-by: Kevin Falcoz <0pc0defr@gmail.com>
Co-authored-by: Dimitri Huisman <diman@huisman.xyz>
2254: Send ISRG_X1 on port 25, make DANE pin that r=mergify[bot] a=nextgens
## What type of PR?
bug-fix
## What does this PR do?
Ensure we send ISRG_X1 in the handshake on port 25 (non-interactive, size doesn't really matter).
Update the DANE pin to reflect the change.
I am not sure whether we will need to add --preferred-chain= in the future; This may be the case when letsencrypt decides to use X2/the ECDSA chain
This needs to be tested on a letsencrypt account that isn't mine (I'm opted in for the alternate cert chains)
### Related issue(s)
- closes#2138
## Prerequisites
Before we can consider review and merge, please make sure the following list is done and checked.
If an entry in not applicable, you can check it or remove it from the list.
There's already a towncrier news for it
Co-authored-by: Florent Daigniere <nextgens@freenetproject.org>
2211: Ensure we use IMAP IDLE like it's supposed to work r=mergify[bot] a=nextgens
## What type of PR?
enhancement
## What does this PR do?
Increase IMAP IDLE time from 2min to 29mins: this should massively help reduce network traffic & increase battery life of clients
See https://peterkieser.com/2011/03/25/androids-k-9-mail-battery-life-and-dovecots-push-imap/
## Prerequisites
Before we can consider review and merge, please make sure the following list is done and checked.
If an entry in not applicable, you can check it or remove it from the list.
- [ ] In case of feature or enhancement: documentation updated accordingly
- [x] Unless it's docs or a minor change: add [changelog](https://mailu.io/master/contributors/workflow.html#changelog) entry file.
Co-authored-by: Florent Daigniere <nextgens@freenetproject.org>
Co-authored-by: Florent Daigniere <nextgens@users.noreply.github.com>
2144: Enable unbound by default, warn if the DNS resolver doesn't work r=mergify[bot] a=nextgens
## What type of PR?
bug-fix
## What does this PR do?
Enable unbound by default, warn if the DNS resolver doesn't work
### Related issue(s)
- close#2135
## Prerequisites
Before we can consider review and merge, please make sure the following list is done and checked.
If an entry in not applicable, you can check it or remove it from the list.
- [ ] In case of feature or enhancement: documentation updated accordingly
- [x] Unless it's docs or a minor change: add [changelog](https://mailu.io/master/contributors/workflow.html#changelog) entry file.
Co-authored-by: Florent Daigniere <nextgens@freenetproject.org>
Co-authored-by: Florent Daigniere <nextgens@users.noreply.github.com>
2150: fix 2145: exceptions may be thrown when login is invalid or rate-limits exceeded r=mergify[bot] a=nextgens
## What type of PR?
bug-fix
## What does this PR do?
Exceptions may be thrown when login is invalid or rate-limits exceeded for those running very recent builds of 1.9
For some reason I haven't caught it while testing #2130... that's when it was introduced.
### Related issue(s)
- close#2145
- close#2146
- #2130
Co-authored-by: Florent Daigniere <nextgens@freenetproject.org>
2099: update Dockerfile to alpine 3.14.3 r=mergify[bot] a=willofr
## What type of PR?
Security fix
## What does this PR do?
Updated the Dockerfile to use the latest alpine version 3.14.3 where several CVEs have been fixed: https://alpinelinux.org/posts/Alpine-3.14.3-released.html
New images successfully built on my test env.
### Related issue(s)
None
## Prerequisites
Before we can consider review and merge, please make sure the following list is done and checked.
If an entry in not applicable, you can check it or remove it from the list.
- [x] In case of feature or enhancement: documentation updated accordingly
- [x] Unless it's docs or a minor change: add [changelog](https://mailu.io/master/contributors/workflow.html#changelog) entry file.
Co-authored-by: Will <will@packer-output-c8fcfb40-3d93-4475-8f87-e14a9dd683b6>
Co-authored-by: willofr <willofr@users.noreply.github.com>
2116: fix 2114: redirect old path r=mergify[bot] a=nextgens
## What type of PR?
bug-fix
## What does this PR do?
Old paths may still be cached in browsers, it's easy enough to redirect them
### Related issue(s)
- close#2114
Co-authored-by: Florent Daigniere <nextgens@freenetproject.org>
2094: Sessions tweaks r=mergify[bot] a=nextgens
## What type of PR?
bug-fix
## What does this PR do?
- Make all sessions permanent, introduce SESSION_TIMEOUT and PERMANENT_SESSION_LIFETIME.
- Prevent the creation of a session before there is a login attempt
- Ensure that webmail tokens are in sync with sessions
### Related issue(s)
- close#2080
## Prerequisites
Before we can consider review and merge, please make sure the following list is done and checked.
If an entry in not applicable, you can check it or remove it from the list.
- [x] In case of feature or enhancement: documentation updated accordingly
- [x] Unless it's docs or a minor change: add [changelog](https://mailu.io/master/contributors/workflow.html#changelog) entry file.
Co-authored-by: Florent Daigniere <nextgens@freenetproject.org>
Co-authored-by: Dimitri Huisman <diman@huisman.xyz>
1441: Rsyslog logging for postfix r=mergify[bot] a=micw
## What type of PR?
enhancement
## What does this PR do?
Changes postfix logging from stdout to rsyslog:
* stdout logging still enabled
* internal test request log messages are filtered out by rsyslog
* optional logging to file via POSTFIX_LOG_FILE env variable
## Prerequistes
Before we can consider review and merge, please make sure the following list is done and checked.
If an entry in not applicable, you can check it or remove it from the list.
- [x] In case of feature or enhancement: documentation updated accordingly
- [x] Unless it's docs or a minor change: add [changelog](https://mailu.io/master/contributors/guide.html#changelog) entry file.
2090: fix 2086 r=mergify[bot] a=nextgens
## What type of PR?
bug-fix
## What does this PR do?
Fix a bug I've introduced in ae8db08bd
### Related issue(s)
- close#2086
Co-authored-by: Michael Wyraz <michael@wyraz.de>
Co-authored-by: Dimitri Huisman <diman@huisman.xyz>
Co-authored-by: Dimitri Huisman <52963853+Diman0@users.noreply.github.com>
Co-authored-by: Florent Daigniere <nextgens@freenetproject.org>
2072: use dovecot-fts-xapian from alpine package r=mergify[bot] a=willofr
## What type of PR?
enhancement
## What does this PR do?
use dovecot-fts-xapian from alpine packages repository (newer) instead of compiling an older version from source
see https://pkgs.alpinelinux.org/package/edge/community/x86/dovecot-fts-xapian
### Related issue(s)
No
## Prerequisites
Before we can consider review and merge, please make sure the following list is done and checked.
If an entry in not applicable, you can check it or remove it from the list.
- [x] In case of feature or enhancement: documentation updated accordingly
- [x] Unless it's docs or a minor change: add [changelog](https://mailu.io/master/contributors/workflow.html#changelog) entry file.
Co-authored-by: willofr <willofr@users.noreply.github.com>
2084: Fix#2078 (login to webmail did not work when WEB_WEBMAIL=/ was set) r=mergify[bot] a=Diman0
## What type of PR?
bug-fix
## What does this PR do?
It fixes#2078. Login from SSO page to webmail did not work if WEB_WEBMAIL=/ was set in mailu.env.
I tested that it works with
- WEB_WEBMAIL=/webmail
- WEB_WEBMAIL=/
### Related issue(s)
- closes#2078
## Prerequisites
Before we can consider review and merge, please make sure the following list is done and checked.
If an entry in not applicable, you can check it or remove it from the list.
- [x] n/a In case of feature or enhancement: documentation updated accordingly
- [x] Unless it's docs or a minor change: add [changelog](https://mailu.io/master/contributors/workflow.html#changelog) entry file.
Co-authored-by: Dimitri Huisman <diman@huisman.xyz>
Co-authored-by: Florent Daigniere <nextgens@users.noreply.github.com>
2058: Implement versioning for CI/CD workflow. r=mergify[bot] a=Diman0
## What type of PR?
Feature!
## What does this PR do?
This PR introduces 3 things
- Add versioning (tagging) for branch x.y (1.8). E.g. 1.8.0, 1.8.1 etc.
- docker repo will contain x.y (latest) and x.y.z (pinned version) images.
- The X.Y.Z tag is incremented automatically. E.g. if 1.8.0 already exists, then the next merge on 1.8 will result in the new tag 1.8.1 being used.
- Make the version available in the image.
- For X.Y and X.Y.Z write the version (X.Y.Z) into /version on the image and add a label with version=X.Y.Z
- This means that the latest X.Y image shows the pinned version (X.Y.Z e.g. 1.8.1) it was based on. Via the tag X.Y.Z you can see the commit hash that triggered the built.
- For master write the commit hash into /version on the image and add a label with version={commit hash}
- Automatic releases. For x.y triggered builts (e.g. merge on 1.9) do a new github release for the pinned x.y.z (e.g. 1.9.2).
- Release shows a static message (see RELEASE_TEMPLATE.md) that explains how to reach the newsfragments folder and change the branch to the tag (x.y.z) mentioned in the release. Now you can get the changelog by reading all newsfragment files in this folder.
This PR does not change anything to our workflow (what we (human persons) do). Our processes are still exactly the same. The above introduced logic is automatic. When we backport to X.Y all the magic for creating the pinned version X.Y.Z is handled by the CI/CD workflow.
### Related issue(s)
- closes#1182
## Prerequisites
Before we can consider review and merge, please make sure the following list is done and checked.
If an entry in not applicable, you can check it or remove it from the list.
- [x] In case of feature or enhancement: documentation updated accordingly
- [x] Unless it's docs or a minor change: add [changelog](https://mailu.io/master/contributors/workflow.html#changelog) entry file.
## Testing
Suggested testing steps. This should cover all situations including BORS. It does require that you use your own docker repo or temporarily create a new one.
Suggested testing steps.
1. Create new github repo.
2. Add the required docker secrets to the project (see beginning of CI.yml for the secret names), DOCKER_UN, DOCKER_PW, DOCKER_ORG, DOCKER_ORG_TESTS.
3. Clone the project.
4. Copy the contents of the PR to the cloned project.
5. Push to your new github repo.
6. Now master images are built. Check that images with tag master are pushed to your docker repo
7. Check with docker inspect nginx:master that it has the label version={commit hash}.
8. Run an image, run `docker-compose exec <name> cat /version`. Note that /version also contains the pinned version. For master the pinned version is the commit hash.
9. Create branch 1.8.
10. Push branch 1.8 to repo.
11. Note that tags 1.8 and 1.8.0 are built and pushed to docker repo
12. Inspect label and /version. Note that 1.8 and 1.8.0 both show version 1.8.0.
13. Push another commit to branch 1.8.
14. Note that tags 1.8 and 1.8.1 are built and pushed to docker repo
15. Inspect label and /version. Note that 1.8 and 1.8.1 both show version 1.8.1.
16. Let's check BORS stuff.
17. Create branch testing.
18. Push the commit with the exact commit text (IMPORTANT!!): `Try #1234:`'.
19. Note that images are built and pushed for tag `pr-1234`.
20. Inspect label and /version. Note that the version is `pr-1234`.
20. Create branch staging.
21. Push the commit with commit text: `Merge #1234`.
22. Note that this image is not pushed to docker (as expected).
but you could also check the GH repo and docker repo I used:
https://github.com/Diman0/Mailu_Forkhttps://hub.docker.com/r/diman/rainloop/tags
Co-authored-by: Dimitri Huisman <diman@huisman.xyz>
2076: fix the default for DEFER_ON_TLS_ERROR r=mergify[bot] a=nextgens
## What type of PR?
bug-fix
## What does this PR do?
The default wasn't set anywhere
Co-authored-by: Florent Daigniere <nextgens@freenetproject.org>
- Moved run to bottom of Dockerfile to allow using unmodified / cached states.
- Simplified bash code in deploy.sh.
- Improved the large bash one-liner in CI.yml. It could not handle >9 for 1.x.
2044: Vault/rspamd: don't return any key for relayed domains r=mergify[bot] a=nextgens
## What type of PR?
enhancement
## What does this PR
Don't return any key for relayed domains. We may want to revisit this (ARC signing)... but in the meantime it saves from a scary message in rspamd.
```signing failure: cannot request data from the vault url: /internal/rspamd/vault/v1/dkim/ ...```
Co-authored-by: Florent Daigniere <nextgens@freenetproject.org>
2042: Add MESSAGE_RATELIMIT_EXEMPTION r=mergify[bot] a=nextgens
## What type of PR?
Enhancement
## What does this PR do?
Add a new knob called ```MESSAGE_RATELIMIT_EXEMPTION```.
### Related issue(s)
- #1774
## Prerequisites
Before we can consider review and merge, please make sure the following list is done and checked.
If an entry in not applicable, you can check it or remove it from the list.
- [x] In case of feature or enhancement: documentation updated accordingly
- [ ] Unless it's docs or a minor change: add [changelog](https://mailu.io/master/contributors/workflow.html#changelog) entry file.
Co-authored-by: Florent Daigniere <nextgens@freenetproject.org>
2034: Add timezone to containers r=mergify[bot] a=DjVinnii
## What type of PR?
Enhancement
## What does this PR do?
This PR adds the tzdata package so that the environment variable `TZ` can be used to set the timezone of containers.
### Related issue(s)
- closes#1154
## Prerequisites
Before we can consider review and merge, please make sure the following list is done and checked.
If an entry in not applicable, you can check it or remove it from the list.
- [x] In case of feature or enhancement: documentation updated accordingly
- [x] Unless it's docs or a minor change: add [changelog](https://mailu.io/master/contributors/workflow.html#changelog) entry file.
Co-authored-by: DjVinnii <vincentkling@msn.com>
We may want to revisit this (ARC signing)... but in the meantime
it saves from a scary message in rspamd
signing failure: cannot request data from the vault url: /internal/rspamd/vault/v1/dkim/ ...
this is working fine, but introduces a sqlalchemy warning
when using config-import:
/app/mailu/schemas.py:822:
SAWarning: Identity map already had an identity for (...),
replacing it with newly flushed object.
Are there load operations occurring inside of an event handler
within the flush?
ConfigManager should not replace app.config - this is causing trouble
with some other flask modules (swagger).
Updated ConfigManager to only modify app.config and not replace it.
2036: round display of range inputs to 2 decimals r=mergify[bot] a=ghostwheel42
## What type of PR?
small fix
## What does this PR do?
rounds display of range inputs to 2 decimals
### Related issue(s)
- small fix to #1966
## Prerequisites
Before we can consider review and merge, please make sure the following list is done and checked.
If an entry in not applicable, you can check it or remove it from the list.
- [X] In case of feature or enhancement: documentation updated accordingly
- [X] Unless it's docs or a minor change: add [changelog](https://mailu.io/master/contributors/workflow.html#changelog) entry file.
Co-authored-by: Alexander Graf <ghostwheel42@users.noreply.github.com>
2027: Make logs more quiet r=mergify[bot] a=nextgens
## What type of PR?
enhancement
## What does this PR do?
It silences various useless log messages in front, specifically:
```
Oct 30 03:11:04 instance-20210109-1612 docker-front[1963]: 127.0.0.1 - - [30/Oct/2021:03:11:04 +0000] "GET /health HTTP/1.1" 301 162 "-" "curl/7.78.0"
Oct 30 03:11:04 instance-20210109-1612 docker-front[1963]: 127.0.0.1 - - [30/Oct/2021:03:11:04 +0000] "GET /health HTTP/2.0" 204 0 "-" "curl/7.78.0"
Oct 30 03:11:04 instance-20210109-1612 docker-front[1963]: 2021/10/30 03:11:04 [info] 476302#476302: *2622679 client 127.0.0.1 closed keepalive connection
Oct 30 03:13:02 instance-20210109-1612 docker-front[1963]: 127.0.0.1 - - [30/Oct/2021:03:13:02 +0000] "GET /auth/email HTTP/1.0" 200 0 "-" "-"
```
`@micw` has requested it for k8s
2030: Fix RELAYNETS r=mergify[bot] a=nextgens
## What type of PR?
bug-fix
## What does this PR do?
RELAYNETS should be comma separated like everything else; rspamd should also be aware of what is considered "trusted".
I am not sure whether ```local_networks``` is the right configuration option for it though
- close#360
Co-authored-by: Florent Daigniere <nextgens@freenetproject.org>
1987: Enhancement to the rate limits r=mergify[bot] a=nextgens
## What type of PR?
enhancement
## What does this PR do?
Turn the rate-limiters into something useful (that won't fire for no reason).
- fix rate-limiting on /webdav/
- it changes the rate-limiting behaviour from limiting a single IP address to a subnet of a reasonable size (/24 on v4 and /56 on v6 both are now configurable) : AUTH_RATELIMIT_IP / AUTH_RATELIMIT_IP_V4_MASK / AUTH_RATELIMIT_IP_V6_MASK
- It ensures we only use IP-based rate-limits for attempts on accounts that do not exist
- it creates a new rate limit preventing attackers from targetting a specific user account (separate from what's above) : AUTH_RATELIMIT_USER
- it introduces a rate limiting exemption mechanism whereby, upon authentication, users will see their source-ip address being exempt for a specific amount of time AUTH_RATELIMIT_EXEMPTION_LENGTH. A similar mechanism is available for web-based sessions (see below)
- It introduces in AUTH_RATELIMIT_EXEMPTION a comma separated list of network CIDRs that will be exempt from both types of rate limiting
- it implements device-tokens, as described on https://owasp.org/www-community/Slow_Down_Online_Guessing_Attacks_with_Device_Cookies to ensure that genuine users aren't locked-out by a malicious attacker abusing the rate-limit feature.
Things that could be improved include:
- the IP-based rate limiter flags attempts against "non-existing" accounts: it could go further and flag the number of unique non-existing accounts attempted (to prevent the case of a user making a typo in his MUA configuration)
- the IP address exemption mechanism doesn't pin the exemption to a specific username: any real user can trivially bypass the rate limits (and attempt to brute-force someone else's account)
### Related issue(s)
- close#1926
- close#1745
- close#1915
## Prerequisites
Before we can consider review and merge, please make sure the following list is done and checked.
If an entry in not applicable, you can check it or remove it from the list.
- [x] In case of feature or enhancement: documentation updated accordingly
- [x] Unless it's docs or a minor change: add [changelog](https://mailu.io/master/contributors/workflow.html#changelog) entry file.
Co-authored-by: Florent Daigniere <nextgens@freenetproject.org>
Co-authored-by: Diman0 <diman@huisman.xyz>
Co-authored-by: Florent Daigniere <nextgens@users.noreply.github.com>
2018: show dmarc record for report domain in domain details r=mergify[bot] a=ghostwheel42
## What type of PR?
documentation
## What does this PR do?
show dmarc record for report domain in domain details
### Related issue(s)
closes#1382
## Prerequisites
- [X] In case of feature or enhancement: documentation updated accordingly
- [X] Unless it's docs or a minor change: add [changelog](https://mailu.io/master/contributors/workflow.html#changelog) entry file.
Co-authored-by: Alexander Graf <ghostwheel42@users.noreply.github.com>
1986: Document how to setup client autoconfig r=mergify[bot] a=nextgens
## What type of PR?
enhancement
## What does this PR do?
Document how to setup autoconfig. This works with most open-source MUAs (thunderbird, evolution, ...)
We could go further than that by providing dynamic configuration (issue an auth token for each MUA request)... but it won't work unless a new DNS entry (and matching certificate) is created.
### Related issue(s)
- #224
## Prerequisites
Before we can consider review and merge, please make sure the following list is done and checked.
If an entry in not applicable, you can check it or remove it from the list.
- [x] In case of feature or enhancement: documentation updated accordingly
- [x] Unless it's docs or a minor change: add [changelog](https://mailu.io/master/contributors/workflow.html#changelog) entry file.
2014: Update Chinese translation r=mergify[bot] a=qy117121
## What type of PR?
translation
## What does this PR do?
Update Chinese translation. Use `zh` instead of `zh_CN`.
### Related issue(s)
none
## Prerequistes
Before we can consider review and merge, please make sure the following list is done and checked.
If an entry in not applicable, you can check it or remove it from the list.
- [X] In case of feature or enhancement: documentation updated accordingly
- [X] Unless it's docs or a minor change: add [changelog](https://mailu.io/master/contributors/workflow.html#changelog) entry file.
Co-authored-by: Florent Daigniere <nextgens@freenetproject.org>
Co-authored-by: qy117121 <mixuan121@gmail.com>
Co-authored-by: Alexander Graf <ghostwheel42@users.noreply.github.com>
2017: rspamd: get dkim keys via REST API instead of filesystem r=mergify[bot] a=ghostwheel42
## What type of PR?
enhancement
## What does this PR do?
rspamd now uses hashicorp's vault api v1 to get dkim keys and selectors for a domain.
this allows future enhancement (multiple keys) without reconfiguring and restarting rspamd.
it also makes mounting the /dkim volume into the rspamd container unnecessary.
### Related issue(s)
- improves and closes#2012
- allows to implement key rotation using multiple selectors (see #1700)
- allows to implement dkim for alternate domains (see #1519)
- fixes and closes#1345 (selector transmitted by admin container is used)
- closes#1179 (no keys on disk)
- allows to implement key rotation from the outside (ie. via a helper script talking to some dns provider's api) (see #547)
## Prerequisites
- [X] In case of feature or enhancement: documentation updated accordingly
- [X] Unless it's docs or a minor change: add [changelog](https://mailu.io/master/contributors/workflow.html#changelog) entry file.
Co-authored-by: Alexander Graf <ghostwheel42@users.noreply.github.com>
1997: Prevent traceback when using non-email in login r=mergify[bot] a=ghostwheel42
There's a traceback when the username used to log via SMTPAUTH
in is not an email address:
=== before ===
```
[...] ERROR in app: Exception on /internal/auth/email [GET]
Traceback (most recent call last):
File "/usr/lib/python3.9/site-packages/sqlalchemy/engine/base.py", line 1179, in _execute_context
context = constructor(dialect, self, conn, *args)
File "/usr/lib/python3.9/site-packages/sqlalchemy/engine/default.py", line 719, in _init_compiled
param.append(processors[key](compiled_params[key]))
File "/usr/lib/python3.9/site-packages/sqlalchemy/sql/type_api.py", line 1201, in process
return process_param(value, dialect)
File "/app/mailu/models.py", line 60, in process_bind_param
localpart, domain_name = value.lower().rsplit('`@',` 1)
ValueError: not enough values to unpack (expected 2, got 1)
[...]
[parameters: [{'%(140657157923216 param)s': 'foobar'}]]
```
=== after ===
```
[...] WARNING in nginx: Invalid user 'foobar': (builtins.ValueError) invalid email address (no "`@")`
```
## What type of PR?
enhancement
## What does this PR do?
replace traceback (ERROR) with error message (WARNING)
### Related issue(s)
## Prerequisites
Before we can consider review and merge, please make sure the following list is done and checked.
If an entry in not applicable, you can check it or remove it from the list.
- [ ] In case of feature or enhancement: documentation updated accordingly
- [ ] Unless it's docs or a minor change: add [changelog](https://mailu.io/master/contributors/workflow.html#changelog) entry file.
Co-authored-by: Alexander Graf <ghostwheel42@users.noreply.github.com>
2007: allow sending emails as user+detail@domain.tld r=mergify[bot] a=ghostwheel42
## What type of PR?
bug-fix or enhancement
## What does this PR do?
Allows sending emails with an added "+detail" in the local part.
### Related issue(s)
closes#1948
## Prerequisites
Before we can consider review and merge, please make sure the following list is done and checked.
If an entry in not applicable, you can check it or remove it from the list.
- [ ] In case of feature or enhancement: documentation updated accordingly
- [X] Unless it's docs or a minor change: add [changelog](https://mailu.io/master/contributors/workflow.html#changelog) entry file.
Co-authored-by: Alexander Graf <ghostwheel42@users.noreply.github.com>
Co-authored-by: root <ghostwheel42@users.noreply.github.com>
Co-authored-by: Florent Daigniere <nextgens@freenetproject.org>
Florent Daigniere