1791: Enhanced session handling r=mergify[bot] a=ghostwheel42
## What type of PR?
bug-fix
## What does this PR do?
- replaces flask_kvsession and simplekv with a mailu-specific session store
- call cleanup_sessions before first request and not on startup.
this allows to run cmdline actions without redis (and makes it faster)
- allow running without redis for debugging purposes by setting MEMORY_SESSIONS to True
- don't sign session id, as it has plenty of entropy (as suggested by nextgens)
- adds method to prune a user's sessions
### Related issue(s)
- enhances and close#1787
Co-authored-by: Alexander Graf <ghostwheel42@users.noreply.github.com>
1754: centralize Webmail authentication behind the admin panel (SSO) r=mergify[bot] a=nextgens
## What type of PR?
Enhancement: it centralizes the authentication of webmails to the admin interface.
## What does this PR do?
It implements the glue required for webmails to do SSO using the admin interface.
One of the main advantages of centralizing things this way is that it reduces significantly the attack surface available to an unauthenticated attacker (no webmail access until there is a valid Flask session).
Others include the ability to implement 2FA down the line and rate-limit things as required.
### Related issue(s)
- #783
## Prerequistes
Before we can consider review and merge, please make sure the following list is done and checked.
If an entry in not applicable, you can check it or remove it from the list.
- [x] In case of feature or enhancement: documentation updated accordingly
- [x] Unless it's docs or a minor change: add [changelog](https://mailu.io/master/contributors/guide.html#changelog) entry file.
Co-authored-by: Florent Daigniere <nextgens@freenetproject.org>
1758: Implement a simpler credential cache (alternative to #1755) r=mergify[bot] a=nextgens
## What type of PR?
Feature: it implements a credential cache to speedup authentication requests.
## What does this PR do?
Credentials are stored in cold-storage using a slow, salted/iterated hash function to prevent offline bruteforce attacks. This creates a performance bottleneck for no valid reason (see the
rationale/long version on https://github.com/Mailu/Mailu/issues/1194#issuecomment-762115549).
The new credential cache makes things fast again.
This is the simpler version of #1755 (with no new dependencies)
### Related issue(s)
- close#1411
- close#1194
- close#1755
## Prerequistes
Before we can consider review and merge, please make sure the following list is done and checked.
If an entry in not applicable, you can check it or remove it from the list.
- [x] In case of feature or enhancement: documentation updated accordingly
- [x] Unless it's docs or a minor change: add [changelog](https://mailu.io/master/contributors/guide.html#changelog) entry file.
1776: optimize generation of transport nexthop r=mergify[bot] a=ghostwheel42
## What type of PR?
bug-fix and enhancement.
## What does this PR do?
Possibly there should be more input validation when editing a relay, but for now this tries to make the best out of the existing "smtp" attribute while maintaining backwards compatibility. When relay is empty, the transport's nexthop is the MX of the relayed domain to fix#1588
```
RELAY NEXTHOP TRANSPORT
empty use MX of relay domain smtp:domain
:port use MX of relay domain and use port smtp:domain:port
target resolve A/AAAA of target smtp:[target]
target:port resolve A/AAAA of target and use port smtp:[target]:port
mx:target resolve MX of target smtp:target
mx:target:port resolve MX of target and use port smtp:target:port
lmtp:target resolve A/AAAA of target lmtp:target
lmtp:target:port resolve A/AAAA of target and use port lmtp:target:port
target can also be an IPv4 or IPv6 address (an IPv6 address must be enclosed in []: [2001:DB8::]).
```
When there is proper input validation and existing database entries are migrated this function can be made much shorter again.
### Related issue(s)
- closes#1588
- closes#1815
## Prerequistes
Before we can consider review and merge, please make sure the following list is done and checked.
If an entry in not applicable, you can check it or remove it from the list.
- [x] In case of feature or enhancement: documentation updated accordingly
- [X] Unless it's docs or a minor change: add [changelog](https://mailu.io/master/contributors/guide.html#changelog) entry file.
Co-authored-by: Florent Daigniere <nextgens@freenetproject.org>
Co-authored-by: Alexander Graf <ghostwheel42@users.noreply.github.com>
1746: DNS records for client autoconfiguration (RFC6186) r=Diman0 a=nextgens
## What type of PR?
Feature
## What does this PR do?
Add instructions on how to configure rfc6186 DNS records for client autoconfiguration
### Related issue(s)
- #224
- #498
## Prerequistes
Before we can consider review and merge, please make sure the following list is done and checked.
If an entry in not applicable, you can check it or remove it from the list.
- [x ] Unless it's docs or a minor change: add [changelog](https://mailu.io/master/contributors/guide.html#changelog) entry file.
Co-authored-by: Florent Daigniere <nextgens@freenetproject.org>
- the session key is now generated using
- a hash of the uid seeded by the apps secret_key (size: SESSION_KEY_BITS)
- a random token (size: 128 bits)
- the session's creation time (size: 32 bits)
- redis server side sessions are now refreshed after 1/2 the session lifetime
even if not modified
- the cookie is also updated if necessary
This can be used to delete all sessions belonging to a user/login.
For no it just iterates over all sessions.
This could be enhanced by using a prefix for and deleting by prefix.
- call cleanup_sessions on first kvstore access
this allows to run cmdline actions without redis (and makes it faster)
- Allow development using DictStore by setting REDIS_ADDRESS to the empty string in env
- don't sign 64bit random session id as suggested by nextgens
1783: Switch to server-side sessions r=mergify[bot] a=nextgens
## What type of PR?
bug-fix
## What does this PR do?
It simplifies session management.
- it ensures that sessions will eventually expire (*)
- it implements some mitigation against session-fixation attacks
- it switches from client-side to server-side sessions (in Redis)
It doesn't prevent us from (re)-implementing a "remember_me" type of feature if that's considered useful by some.
Co-authored-by: Florent Daigniere <nextgens@freenetproject.org>
- remove PASSWORD_SCHEME altogether
- introduce CREDENTIAL_ROUNDS
- migrate all old hashes to the current format
- auto-detect/enable all hash types that passlib supports
- upgrade passlib to 1.7.4 (see #1706: ldap_salted_sha512 support)
1765: Set sensible cookie flags on the admin app r=mergify[bot] a=nextgens
## What type of PR?
Bugfix
## What does this PR do?
It sets the right flags on the session cookie issued by the admin app.
This should probably be backported as the lack of secure flag on TLS-enabled setup is a high risk vulnerability.
SameSite is hardening / helps against CSRF on modern browsers
HTTPOnly is hardening / helps reduce the impact of XSS
Co-authored-by: Florent Daigniere <nextgens@freenetproject.org>
1763: show flash messages again r=mergify[bot] a=lub
## What type of PR?
bug-fix
## What does this PR do?
This basically restores the behaviour, that got removed in
ecdf0c25b3 during refactoring.
### Related issue(s)
- noticed it while reviewing #1756
## Prerequistes
Before we can consider review and merge, please make sure the following list is done and checked.
If an entry in not applicable, you can check it or remove it from the list.
- [-] In case of feature or enhancement: documentation updated accordingly
- [-] Unless it's docs or a minor change: add [changelog](https://mailu.io/master/contributors/guide.html#changelog) entry file.
Co-authored-by: lub <git@lubiland.de>
1607: _FILE variables for Docker swarm secrets r=mergify[bot] a=lub
## What type of PR?
enhancement
## What does this PR do?
This PR enables usage of DB_PW_FILE and SECRET_KEY_FILE instead of DB_PW and SECRET_KEY to load these values from files instead of supplying them directly. That way it's possible to use Docker secrets.
### Related issue(s)
## Prerequistes
Before we can consider review and merge, please make sure the following list is done and checked.
If an entry in not applicable, you can check it or remove it from the list.
- [x] In case of feature or enhancement: documentation updated accordingly
- [x] Unless it's docs or a minor change: add [changelog](https://mailu.io/master/contributors/guide.html#changelog) entry file.
Co-authored-by: lub <git@lubiland.de>
- yaml-import is now logged via schema.Logger
- iremoved relative imports - not used in other mailu modules
- removed develepment comments
- added Mailconfig.check method to check for duplicate domain names
- converted .format() to .format_map() where possible
- switched to yaml multiline dump for dkim_key
- converted dkim_key import from regex to string functions
- automatically unhide/unexclude explicitly specified attributes on dump
- use field order when loading to stabilize import
- fail when using 'hash_password' without 'password'
- fixed logging of dkim_key
- fixed pruning and deleting of lists
- modified error messages
- added debug flag and two verbosity levels
- idna.encode does not encode upper-case letters,
so .lower() has to be called on value not on result
- split email-address on '@' only once
- converted '*'.format(*) to f-strings
- added docstrings
- removed from_dict method
- code cleanup/style (list concat, exceptions, return&else, line-length)
- added TODO comments on possible future changes
- revived original config-update function for backwards compability
- renamed config-dump to config-export to be in line with config-import
- converted '*'.format(*) to f-strings
- converted string-concatenation to f-strings
renamed single letter variables (m => match)
renamed classmethod arguments to cls (model)
removed shadowing of variables (hash, context)
shortened unneeded lambda functions (id)
converted type ... is to isinstance(...)
removed unneded imports (flask)
Updated ConfigManager to only modify app.config and not replace it.
Swagger does not play well, when app.config is not a real dict and
it is not necessary to keep ConfigManager around after init.
Also added "API" flag to config (default: disabled).
this enables usage of e.g. docker swarm secrets instead of exposing the
passwords directly via environment variables
just use DB_PW_FILE and SECRET_KEY_FILE instead of DB_PW and SECRET_KEY
- stabilized CommaSeparatedList by sorting values
- CommaSeparatedList can now handle list and set input
- from_dict now handles mapped keys
- from_dict now handles null values
- class Domain: handle dkim-key None correctly
- class User: delete obsolete keys after converting
- class Alias: now uses Email._dict_input
1399: Remove SPF type SPF record #1394 r=mergify[bot] a=bladeswords
As mentioned in #1394 - In accordance with RFC 7208, offer only TXT RRs for SPF.
Agree with @Nebukadneza - but not sure how to go about telling people to remove the old record...
## What type of PR?
Documentation
## What does this PR do?
Removes the recommendation to add a SPF RR for SPF records, as this is no longer RFC complaint and often causes issues to maintain two records.
### Related issue(s)
- closes#1394
## Prerequistes
None
1417: docker-compose exec needs a -T flag if no TTY is allocated r=mergify[bot] a=ofthesun9
This flag is missing in 00_create_users.sh and is failing the tests on travis arm architecture
## What type of PR?
This PR is an enhancement/bugfix needed to allow usage of travis to test and deploy on arm platform
Before the PR, tests are failing with the msg: "the input device is not a TTY"
## What does this PR do?
This PR add -T flag for the docker-compose exec occurences found in 00_create_users.sh
Co-authored-by: bladeswords <bladeswords@users.noreply.github.com>
Co-authored-by: Dario Ernst <dario@kanojo.de>
Co-authored-by: ofthesun9 <olivier@ofthesun.net>
1389: Prefer specific alias over wildcard, regardless of case r=mergify[bot] a=Nebukadneza
## What type of PR?
bug-fix
## What does this PR do?
Since direct addresses (not aliases) are case-insensitive since a while,
it makes sense for aliases to behave the same. Up until now, a wildcard
alias could trump a alias not-matching-the-case of the incoming address.
This clarifies this behavior.
## Notes
I realize that the if-hell down there isn’t nice. What it is, however, is quite clear and easy to read. I’m hoping that if anyone ever gets confused in the future, this will make the current behavior transparent. For me, that was more important than a minimal amount of statements/branches …
### Related issue(s)
closes#1387
## Prerequistes
- [x] Unless it's docs or a minor change: add [changelog](https://mailu.io/master/contributors/guide.html#changelog) entry file.
Co-authored-by: Dario Ernst <github@kanojo.de>
1296: fetchmail: print unhandled exceptions, but don't crash r=Nebukadneza a=Al2Klimov
fixes#1295
1322: Bump validators from 0.12.5 to 0.12.6 in /core/admin r=Nebukadneza a=dependabot[bot]
Bumps [validators](https://github.com/kvesteri/validators) from 0.12.5 to 0.12.6.
<details>
<summary>Changelog</summary>
*Sourced from [validators's changelog](https://github.com/kvesteri/validators/blob/master/CHANGES.rst).*
> 0.12.6 (2019-05-08)
> ^^^^^^^^^^^^^^^^^^^
>
> - Fixed domain validator for single character domains ([#118](https://github-redirect.dependabot.com/kvesteri/validators/issues/118), pull request courtesy kingbuzzman)
</details>
<details>
<summary>Commits</summary>
- See full diff in [compare view](https://github.com/kvesteri/validators/commits)
</details>
<br />
[](https://help.github.com/articles/configuring-automated-security-fixes)
Dependabot will resolve any conflicts with this PR as long as you don't alter it yourself. You can also trigger a rebase manually by commenting `@dependabot rebase`.
[//]: # (dependabot-automerge-start)
[//]: # (dependabot-automerge-end)
---
<details>
<summary>Dependabot commands and options</summary>
<br />
You can trigger Dependabot actions by commenting on this PR:
- `@dependabot rebase` will rebase this PR
- `@dependabot recreate` will recreate this PR, overwriting any edits that have been made to it
- `@dependabot merge` will merge this PR after your CI passes on it
- `@dependabot squash and merge` will squash and merge this PR after your CI passes on it
- `@dependabot cancel merge` will cancel a previously requested merge and block automerging
- `@dependabot reopen` will reopen this PR if it is closed
- `@dependabot ignore this [patch|minor|major] version` will close this PR and stop Dependabot creating any more for this minor/major version (unless you reopen the PR or upgrade to it yourself)
- `@dependabot ignore this dependency` will close this PR and stop Dependabot creating any more for this dependency (unless you reopen the PR or upgrade to it yourself)
- `@dependabot use these labels` will set the current labels as the default for future PRs for this repo and language
- `@dependabot use these reviewers` will set the current reviewers as the default for future PRs for this repo and language
- `@dependabot use these assignees` will set the current assignees as the default for future PRs for this repo and language
- `@dependabot use this milestone` will set the current milestone as the default for future PRs for this repo and language
You can disable automated security fix PRs for this repo from the [Security Alerts page](https://github.com/Mailu/Mailu/network/alerts).
</details>
1337: Add IPv6 to allow_nets r=Nebukadneza a=PhilRW
Roundcube was not connecting to sieve with IPv6 enabled.
Fixes#1336
1358: Add port to relay if it contains a colon r=Nebukadneza a=PhilRW
## What type of PR?
enhancement
## What does this PR do?
Allows relaying domains to non-standard SMTP ports by appending `:port` to the destination host/IP. E.g., `mx1.internal:2525`
### Related issue(s)
Closes#1357
## Prerequistes
Before we can consider review and merge, please make sure the following list is done and checked.
If an entry in not applicable, you can check it or remove it from the list.
- [x] In case of feature or enhancement: documentation updated accordingly
- [x] Unless it's docs or a minor change: add [changelog](https://mailu.io/master/contributors/guide.html#changelog) entry file.
Co-authored-by: Alexander A. Klimov <grandmaster@al2klimov.de>
Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com>
Co-authored-by: Philip Rosenberg-Watt <p.rosenberg-watt@cablelabs.com>
As mentioned in #1394 - In accordance with RFC 7208, offer only TXT RRs for SPF.
Agree with @Nebukadneza - but not sure how to go about telling people to remove the old record...
1359: Refactor the rate limiting code r=mergify[bot] a=kaiyou
## What type of PR?
Enhancement
## What does this PR do?
Rate limiting was already redesigned to use Python limits. This
introduced some unexpected behavior, including the fact that only
one criteria is supported per limiter. Docs and setup utility are
updated with this in mind.
Also, the code was made more generic, so limiters can be delivered
for something else than authentication. Authentication-specific
code was moved directly to the authentication routine.
### Related issue(s)
No specific issue.
## Prerequistes
Before we can consider review and merge, please make sure the following list is done and checked.
If an entry in not applicable, you can check it or remove it from the list.
- [x] In case of feature or enhancement: documentation updated accordingly
- [x] Unless it's docs or a minor change: add [changelog](https://mailu.io/master/contributors/guide.html#changelog) entry file.
Co-authored-by: kaiyou <pierre@jaury.eu>
1349: Add support for SRS, related to #328 r=mergify[bot] a=kaiyou
## What type of PR?
Feature
## What does this PR do?
It implements SRS using a Python SRS library.
### Related issue(s)
- closes#328
## Prerequistes
Before we can consider review and merge, please make sure the following list is done and checked.
If an entry in not applicable, you can check it or remove it from the list.
- [x] In case of feature or enhancement: documentation updated accordingly
- [x] Unless it's docs or a minor change: add [changelog](https://mailu.io/master/contributors/guide.html#changelog) entry file.
Co-authored-by: kaiyou <pierre@jaury.eu>
Since direct addresses (not aliases) are case-insensitive since a while,
it makes sense for aliases to behave the same. Up until now, a wildcard
alias could trump a alias not-matching-the-case of the incoming address.
This clarifies this behavior.
closes#1387
Rate limiting was already redesigned to use Python limits. This
introduced some unexpected behavior, including the fact that only
one criteria is supported per limiter. Docs and setup utility are
updated with this in mind.
Also, the code was made more generic, so limiters can be delivered
for something else than authentication. Authentication-specific
code was moved directly to the authentication routine.
1278: Limiter implementation r=kaiyou a=micw
## What type of PR?
(Feature, enhancement, bug-fix, documentation)
## What does this PR do?
Adds a custom limter based on the "limits" lirary that counts up on failed auths only
### Related issue(s)
- closes#1195
- closes#634
## Prerequistes
- [X] Unless it's docs or a minor change: add [changelog](https://mailu.io/master/contributors/guide.html#changelog) entry file.
Co-authored-by: Michael Wyraz <michael@wyraz.de>
Co-authored-by: micw <michael@wyraz.de>
1316: Fix the encoding of incoming user email and password r=mergify[bot] a=kaiyou
## What type of PR?
Bug fix
## What does this PR do?
As described in the changes, RFC2616 states that header should be considered ISO8859-1 in HTTP, which obviously nginx does not really care about when forwarding the password from SMTP authentication to the backend. Hence, we need to encode-then-decode passwords to get the proper value in case a special char is in there.
### Related issue(s)
- This fixes#1139
- This is also related to #1281
- This is also related to #1139
## Prerequistes
Before we can consider review and merge, please make sure the following list is done and checked.
If an entry in not applicable, you can check it or remove it from the list.
- [x] In case of feature or enhancement: documentation updated accordingly
- [x] Unless it's docs or a minor change: add [changelog](https://mailu.io/master/contributors/guide.html#changelog) entry file.
Co-authored-by: kaiyou <pierre@jaury.eu>
1299: Don't remove the address extension in postfix r=mergify[bot] a=RobertMe
## What type of PR?
Bugfix
## What does this PR do?
Currently when the mail address is looked up by Postfix (using the admin
part) the address extension is removed. This is due to the address
extension being removed to look up the user, and afterwards returning
the users mail address. But by not returning the mail address including
the address extension it also isn't part anymore in the LMTP
communication to Dovecot. So Dovecot doesn't know about the extension,
and in turn the address extension can't be used in Sieve mail filtering.
This change fixes that by returning the original address by just
concatinating the "localpart" and domain again when the user is found.
### Related issue(s)
Fixes#982
## Prerequistes
Before we can consider review and merge, please make sure the following list is done and checked.
If an entry in not applicable, you can check it or remove it from the list.
- [ ] In case of feature or enhancement: documentation updated accordingly
- [ ] Unless it's docs or a minor change: add [changelog](https://mailu.io/master/contributors/guide.html#changelog) entry file.
Co-authored-by: Robert Meijers <robert.meijers@gmail.com>
Currently when the mail address is looked up by Postfix (using the admin
part) the address extension is removed. This is due to the address
extension being removed to look up the user, and afterwards returning
the users mail address. But by not returning the mail address including
the address extension it also isn't part anymore in the LMTP
communication to Dovecot. So Dovecot doesn't know about the extension,
and in turn the address extension can't be used in Sieve mail filtering.
This change fixes that by returning the original address by just
concatinating the "localpart" and domain again when the user is found.
Fixes#982
Alexander Graf
lub