1852: Improve ci r=mergify[bot] a=Diman0
## What type of PR?
Enhancement for CI.
## What does this PR do?
Moved logic for configuring the tag for images for testing branch (bors try) and building the images, from deploy.sh (deploy job) to the Build job. Now the images are only built once during the build job. This saves about ~10 minutes for the CI workflow for bors try. Now for all situations (bors try, bors r+ and merge with master) should take around 20 minutes.
I did not create yet another newsfragment since it is an internal enhancement.
### Related issue(s)
## Prerequistes
Before we can consider review and merge, please make sure the following list is done and checked.
If an entry in not applicable, you can check it or remove it from the list.
- [x] In case of feature or enhancement: documentation updated accordingly
- [x] Unless it's docs or a minor change: add [changelog](https://mailu.io/master/contributors/guide.html#changelog) entry file.
Co-authored-by: Dimitri Huisman <diman@huisman.xyz>
1853: Update dependencies r=mergify[bot] a=ghostwheel42
## What type of PR?
security update python dependencies
## What does this PR do?
updates cryptography, Jinja2, pyOpenSSL and PyYAML to current versions
### Related issue(s)
security updates as discussed in #1829
Co-authored-by: Alexander Graf <ghostwheel42@users.noreply.github.com>
1791: Enhanced session handling r=mergify[bot] a=ghostwheel42
## What type of PR?
bug-fix
## What does this PR do?
- replaces flask_kvsession and simplekv with a mailu-specific session store
- call cleanup_sessions before first request and not on startup.
this allows to run cmdline actions without redis (and makes it faster)
- allow running without redis for debugging purposes by setting MEMORY_SESSIONS to True
- don't sign session id, as it has plenty of entropy (as suggested by nextgens)
- adds method to prune a user's sessions
### Related issue(s)
- enhances and close#1787
Co-authored-by: Alexander Graf <ghostwheel42@users.noreply.github.com>
1846: fix newsfragment of #1841 r=mergify[bot] a=ghostwheel42
## What type of PR?
bug-fix
## What does this PR do?
moves newsfragment to correct location.
Co-authored-by: Alexander Graf <ghostwheel42@users.noreply.github.com>
1841: Update version of roundcube webmail and carddav plugin. r=mergify[bot] a=ghostwheel42
## What type of PR?
Security-update for roundcube-webmailer.
## What does this PR do?
This PR updates the Roundcube webmail to the latest version, also updates the roundcube carddav-plugin to a new version.
- roundcube 1.4.11
- carddav 4.1.2
### Related issue(s)
This PR superseeds PR #1739
## Prerequistes
Before we can consider review and merge, please make sure the following list is done and checked.
If an entry in not applicable, you can check it or remove it from the list.
- [ ] In case of feature or enhancement: documentation updated accordingly
- [X] Unless it's docs or a minor change: add [changelog](https://mailu.io/master/contributors/guide.html#changelog) entry file.
Co-authored-by: Alexander Graf <ghostwheel42@users.noreply.github.com>
1843: fix bugs in model and schema introduced by #1604 r=mergify[bot] a=ghostwheel42
## What type of PR?
bug fix. linter complained about "== True" but with sqlalchemy this is correct
## What does this PR do?
### Related issue(s)
closes#1842
## Prerequistes
Before we can consider review and merge, please make sure the following list is done and checked.
If an entry in not applicable, you can check it or remove it from the list.
- [ ] In case of feature or enhancement: documentation updated accordingly
- [ ] Unless it's docs or a minor change: add [changelog](https://mailu.io/master/contributors/guide.html#changelog) entry file.
Co-authored-by: Alexander Graf <ghostwheel42@users.noreply.github.com>
1604: Added CLI command to export and import the configuration r=mergify[bot] a=ghostwheel42
## What type of PR?
enhancement
## What does this PR do?
This PR adds a `config-dump` cli command and updates the `config-update` cli command to
handle all possible items and parameters.
This was done by adding generic **to_dict** and **from_dict** methods to the Base model, so it should be quite future-proof.
The changes to `config-update` are backwards-compatible to the old command.
I've only removed the undocumented yaml-section _managers_ - managers can now be defined in the _users_ section.
The YAML now looks like this:
```
aliases:
- email: email@example.com
destination:
- address@example.com
domains:
- name: example.com
alternatives:
- alternative.tld
relays:
- name: relay.example.com
smtp: mx.example.com
users:
- email: postmaster@example.com
displayed_name: 'Postmaster'
enable_imap: true
enable_pop: false
enabled: true
forward_destination:
- address@remote.example.com
forward_enabled: true
forward_keep: true
global_admin: true
manager_of:
- example.com
password: '{BLF-CRYPT}$2b$12$...'
spam_enabled: true
spam_threshold: 80
```
### Related issue(s)
- Closes issue #1377
## Prerequistes
Before we can consider review and merge, please make sure the following list is done and checked.
If an entry in not applicable, you can check it or remove it from the list.
- [X] In case of feature or enhancement: documentation updated accordingly
- [X] Unless it's docs or a minor change: add [changelog](https://mailu.io/master/contributors/guide.html#changelog) entry file.
Co-authored-by: Alexander Graf <ghostwheel42@users.noreply.github.com>
1754: centralize Webmail authentication behind the admin panel (SSO) r=mergify[bot] a=nextgens
## What type of PR?
Enhancement: it centralizes the authentication of webmails to the admin interface.
## What does this PR do?
It implements the glue required for webmails to do SSO using the admin interface.
One of the main advantages of centralizing things this way is that it reduces significantly the attack surface available to an unauthenticated attacker (no webmail access until there is a valid Flask session).
Others include the ability to implement 2FA down the line and rate-limit things as required.
### Related issue(s)
- #783
## Prerequistes
Before we can consider review and merge, please make sure the following list is done and checked.
If an entry in not applicable, you can check it or remove it from the list.
- [x] In case of feature or enhancement: documentation updated accordingly
- [x] Unless it's docs or a minor change: add [changelog](https://mailu.io/master/contributors/guide.html#changelog) entry file.
Co-authored-by: Florent Daigniere <nextgens@freenetproject.org>
1758: Implement a simpler credential cache (alternative to #1755) r=mergify[bot] a=nextgens
## What type of PR?
Feature: it implements a credential cache to speedup authentication requests.
## What does this PR do?
Credentials are stored in cold-storage using a slow, salted/iterated hash function to prevent offline bruteforce attacks. This creates a performance bottleneck for no valid reason (see the
rationale/long version on https://github.com/Mailu/Mailu/issues/1194#issuecomment-762115549).
The new credential cache makes things fast again.
This is the simpler version of #1755 (with no new dependencies)
### Related issue(s)
- close#1411
- close#1194
- close#1755
## Prerequistes
Before we can consider review and merge, please make sure the following list is done and checked.
If an entry in not applicable, you can check it or remove it from the list.
- [x] In case of feature or enhancement: documentation updated accordingly
- [x] Unless it's docs or a minor change: add [changelog](https://mailu.io/master/contributors/guide.html#changelog) entry file.
1776: optimize generation of transport nexthop r=mergify[bot] a=ghostwheel42
## What type of PR?
bug-fix and enhancement.
## What does this PR do?
Possibly there should be more input validation when editing a relay, but for now this tries to make the best out of the existing "smtp" attribute while maintaining backwards compatibility. When relay is empty, the transport's nexthop is the MX of the relayed domain to fix#1588
```
RELAY NEXTHOP TRANSPORT
empty use MX of relay domain smtp:domain
:port use MX of relay domain and use port smtp:domain:port
target resolve A/AAAA of target smtp:[target]
target:port resolve A/AAAA of target and use port smtp:[target]:port
mx:target resolve MX of target smtp:target
mx:target:port resolve MX of target and use port smtp:target:port
lmtp:target resolve A/AAAA of target lmtp:target
lmtp:target:port resolve A/AAAA of target and use port lmtp:target:port
target can also be an IPv4 or IPv6 address (an IPv6 address must be enclosed in []: [2001:DB8::]).
```
When there is proper input validation and existing database entries are migrated this function can be made much shorter again.
### Related issue(s)
- closes#1588
- closes#1815
## Prerequistes
Before we can consider review and merge, please make sure the following list is done and checked.
If an entry in not applicable, you can check it or remove it from the list.
- [x] In case of feature or enhancement: documentation updated accordingly
- [X] Unless it's docs or a minor change: add [changelog](https://mailu.io/master/contributors/guide.html#changelog) entry file.
Co-authored-by: Florent Daigniere <nextgens@freenetproject.org>
Co-authored-by: Alexander Graf <ghostwheel42@users.noreply.github.com>
1831: Fix roundcube database env configuration r=mergify[bot] a=parisni
## What type of PR?
bug-fix
## What does this PR do?
Both roundcube and mailu admin website can be backed by postgres/mysql. Before this PR, the `DB_FLAVOR` is shared by both services. However, the other roundcube dedicated DB params are prefixed with `ROUNDCUBE_`. (eg: `ROUNDCUBE_DB_NAME`)
There is no reason to share the DB_FLAVOR for both: This PR makes them be considered independently to make things clear and avoid bugs.
Also, the roundcube_db_flavor and db_flavor are made separated in this PR. However for simplicity, the template generator bind them : roundcube_db_flavor = db_flavor. This makes the template generator UI more simple. I considered most of the time people want to have both roundcube and mailu share the same RDBMS.
Also, AFAIK the internal postgresql service is deprecated and will be removed in 1.9. This is why this PR does not integrate roundcube in postgres when the internal DB is choosen: in case of internal postgres, the roundcube is backed with sqlite.
Both documentation and setup website have been updated accordingly.
### Related issue(s)
- Auto close an issue like: closes#1648#1471
## Prerequistes
Before we can consider review and merge, please make sure the following list is done and checked.
If an entry in not applicable, you can check it or remove it from the list.
- [X] In case of feature or enhancement: documentation updated accordingly
- [X] Unless it's docs or a minor change: add [changelog](https://mailu.io/master/contributors/guide.html#changelog) entry file.
Co-authored-by: parisni <nicolas.paris@riseup.net>
Co-authored-by: Nicolas Paris <nicolas.paris@riseup.net>
1837: Set static hostname for antispam to preserve history. r=mergify[bot] a=Diman0
## What type of PR?
bug-fix
## What does this PR do?
It addresses bug #561. Rspamd stores the history in redis. The key for storing the history contains the hostname. On recreation of the docker container, the hostname changes and for this reason the rspamd history is lost. Setting a fixed hostname resolves this. Upon recreating the antispam container the history is retained.
### Related issue(s)
- Closes#561
## Prerequistes
Before we can consider review and merge, please make sure the following list is done and checked.
If an entry in not applicable, you can check it or remove it from the list.
- [x] In case of feature or enhancement: documentation updated accordingly
- [x] Unless it's docs or a minor change: add [changelog](https://mailu.io/master/contributors/guide.html#changelog) entry file.
Co-authored-by: Dimitri Huisman <diman@huisman.xyz>
1836: Test ci parallel r=Diman0 a=Diman0
## What type of PR?
enhancement
## What does this PR do?
Changes CI workflow to run all tests in parellel. After performing some tests (see #1830 ), I determined that using actions/cache to only cache a tar.gz. file with all build images and use this for all parallel tests is the fasted solution.
With Travis builds took ~30 minutes. Now each build runs for a maximum of 20 minutes (bors test and merge on master).
Bors r+ runs take about ~16/17 minutes.
### Related issue(s)
- Auto close an issue like: closes#1830
## Prerequistes
Before we can consider review and merge, please make sure the following list is done and checked.
If an entry in not applicable, you can check it or remove it from the list.
- [x] In case of feature or enhancement: documentation updated accordingly
- [x] Unless it's docs or a minor change: add [changelog](https://mailu.io/master/contributors/guide.html#changelog) entry file.
Co-authored-by: Dimitri Huisman <diman@huisman.xyz>
Co-authored-by: Dimitri Huisman <52963853+Diman0@users.noreply.github.com>
1746: DNS records for client autoconfiguration (RFC6186) r=Diman0 a=nextgens
## What type of PR?
Feature
## What does this PR do?
Add instructions on how to configure rfc6186 DNS records for client autoconfiguration
### Related issue(s)
- #224
- #498
## Prerequistes
Before we can consider review and merge, please make sure the following list is done and checked.
If an entry in not applicable, you can check it or remove it from the list.
- [x ] Unless it's docs or a minor change: add [changelog](https://mailu.io/master/contributors/guide.html#changelog) entry file.
Co-authored-by: Florent Daigniere <nextgens@freenetproject.org>
Workflow file only triggers if it resides in the branch. If an old PR based on the mailu repo without CI.yml is tested, then the workflow run will not trigger. The merged commit on TESTING/STAGING branch does not contain the required CI.yml workflow file after all. In these cases simply run the workflow manually on the TESTING or STAGING branch,
1694: update compression algorithms for current dovecot r=nextgens a=lub
## What type of PR?
enhancement
## What does this PR do?
This adds additional compression algorithms in accordance with
https://doc.dovecot.org/configuration_manual/zlib_plugin/
### Related issue(s)
## Prerequistes
Before we can consider review and merge, please make sure the following list is done and checked.
If an entry in not applicable, you can check it or remove it from the list.
- [x] In case of feature or enhancement: documentation updated accordingly
- [x] Unless it's docs or a minor change: add [changelog](https://mailu.io/master/contributors/guide.html#changelog) entry file.
Co-authored-by: lub <git@lubiland.de>
1806: Reflect override settings for postfix r=mergify[bot] a=ineiti
## What type of PR?
(documentation)
## What does this PR do?
Fixes path in the documentation setting, and also adds how to change postfix.cf
Co-authored-by: Linus Gasser <linus.gasser@epfl.ch>
1649: Update docs/reverse.rst with Traefik v2+ info r=mergify[bot] a=patryk-tech
## What type of PR?
Documentation
## What does this PR do?
Adds information about using Traefik v2+ as a reverse proxy.
### Related issue(s)
Closes#1503
## Prerequistes
Before we can consider review and merge, please make sure the following list is done and checked.
If an entry in not applicable, you can check it or remove it from the list.
- [x] In case of feature or enhancement: documentation updated accordingly
- [x] Unless it's docs or a minor change: add [changelog](https://mailu.io/master/contributors/guide.html#changelog) entry file.
1673: Remove rspamd unused env var from start script r=mergify[bot] a=cbachert
## What type of PR?
Cleanup
## What does this PR do?
Remove unused environment variable FRONT_ADDRESS in rspamd. FRONT_ADDRESS references were removed with commit 8172f3e in PR #727 like mentioned in chat https://matrix.to/#/!MINuyJjJSrfowljYCK:tedomum.net/$160401946364NGNmI:imninja.net?via=huisman.xyz&via=matrix.org&via=imninja.net
```
Mailu$ grep -r "FRONT_ADDRESS" core/rspamd/
core/rspamd/start.py:os.environ["FRONT_ADDRESS"] = system.get_host_address_from_environment("FRONT", "front")
```
### Related issue(s)
N/A
## Prerequistes
- [x] Documentation updated accordingly: No documentation to update
- [x] Add to changelog: Minor change
Co-authored-by: Patryk Tech <git@patryk.tech>
Co-authored-by: cbachert <cbachert@users.noreply.github.com>
bors[bot]