2058: Implement versioning for CI/CD workflow. r=mergify[bot] a=Diman0
## What type of PR?
Feature!
## What does this PR do?
This PR introduces 3 things
- Add versioning (tagging) for branch x.y (1.8). E.g. 1.8.0, 1.8.1 etc.
- docker repo will contain x.y (latest) and x.y.z (pinned version) images.
- The X.Y.Z tag is incremented automatically. E.g. if 1.8.0 already exists, then the next merge on 1.8 will result in the new tag 1.8.1 being used.
- Make the version available in the image.
- For X.Y and X.Y.Z write the version (X.Y.Z) into /version on the image and add a label with version=X.Y.Z
- This means that the latest X.Y image shows the pinned version (X.Y.Z e.g. 1.8.1) it was based on. Via the tag X.Y.Z you can see the commit hash that triggered the built.
- For master write the commit hash into /version on the image and add a label with version={commit hash}
- Automatic releases. For x.y triggered builts (e.g. merge on 1.9) do a new github release for the pinned x.y.z (e.g. 1.9.2).
- Release shows a static message (see RELEASE_TEMPLATE.md) that explains how to reach the newsfragments folder and change the branch to the tag (x.y.z) mentioned in the release. Now you can get the changelog by reading all newsfragment files in this folder.
This PR does not change anything to our workflow (what we (human persons) do). Our processes are still exactly the same. The above introduced logic is automatic. When we backport to X.Y all the magic for creating the pinned version X.Y.Z is handled by the CI/CD workflow.
### Related issue(s)
- closes#1182
## Prerequisites
Before we can consider review and merge, please make sure the following list is done and checked.
If an entry in not applicable, you can check it or remove it from the list.
- [x] In case of feature or enhancement: documentation updated accordingly
- [x] Unless it's docs or a minor change: add [changelog](https://mailu.io/master/contributors/workflow.html#changelog) entry file.
## Testing
Suggested testing steps. This should cover all situations including BORS. It does require that you use your own docker repo or temporarily create a new one.
Suggested testing steps.
1. Create new github repo.
2. Add the required docker secrets to the project (see beginning of CI.yml for the secret names), DOCKER_UN, DOCKER_PW, DOCKER_ORG, DOCKER_ORG_TESTS.
3. Clone the project.
4. Copy the contents of the PR to the cloned project.
5. Push to your new github repo.
6. Now master images are built. Check that images with tag master are pushed to your docker repo
7. Check with docker inspect nginx:master that it has the label version={commit hash}.
8. Run an image, run `docker-compose exec <name> cat /version`. Note that /version also contains the pinned version. For master the pinned version is the commit hash.
9. Create branch 1.8.
10. Push branch 1.8 to repo.
11. Note that tags 1.8 and 1.8.0 are built and pushed to docker repo
12. Inspect label and /version. Note that 1.8 and 1.8.0 both show version 1.8.0.
13. Push another commit to branch 1.8.
14. Note that tags 1.8 and 1.8.1 are built and pushed to docker repo
15. Inspect label and /version. Note that 1.8 and 1.8.1 both show version 1.8.1.
16. Let's check BORS stuff.
17. Create branch testing.
18. Push the commit with the exact commit text (IMPORTANT!!): `Try #1234:`'.
19. Note that images are built and pushed for tag `pr-1234`.
20. Inspect label and /version. Note that the version is `pr-1234`.
20. Create branch staging.
21. Push the commit with commit text: `Merge #1234`.
22. Note that this image is not pushed to docker (as expected).
but you could also check the GH repo and docker repo I used:
https://github.com/Diman0/Mailu_Forkhttps://hub.docker.com/r/diman/rainloop/tags
Co-authored-by: Dimitri Huisman <diman@huisman.xyz>
2076: fix the default for DEFER_ON_TLS_ERROR r=mergify[bot] a=nextgens
## What type of PR?
bug-fix
## What does this PR do?
The default wasn't set anywhere
Co-authored-by: Florent Daigniere <nextgens@freenetproject.org>
2083: Fix Webmail token check. Fix Auth-Port for Webmail. #2079 r=mergify[bot] a=Diman0
## What type of PR?
Bug fix
## What does this PR do?
Fixes issues #2079 and #2081.
### Related issue(s)
- closes#2079
- closes#2081
## Prerequisites
Before we can consider review and merge, please make sure the following list is done and checked.
If an entry in not applicable, you can check it or remove it from the list.
- [x] n/a In case of feature or enhancement: documentation updated accordingly
- [x] Unless it's docs or a minor change: add [changelog](https://mailu.io/master/contributors/workflow.html#changelog) entry file.
Co-authored-by: Dimitri Huisman <diman@huisman.xyz>
2071: Reduce logging level r=mergify[bot] a=nextgens
## What type of PR?
bug-fix
## What does this PR do?
Reduce the logging level associated with TLSA record lookup. I've been running master in prod for a few months now and one of the common messages is:
```
[2021-11-23 08:53:29,884] ERROR in utils: Error while looking up the TLSA record for .fr A DNS label is empty.
[2021-11-23 08:53:30,630] ERROR in utils: Error while looking up the TLSA record for .co.uk A DNS label is empty.
[2021-11-23 08:53:30,636] ERROR in utils: Error while looking up the TLSA record for .uk A DNS label is empty.
[2021-11-23 08:58:16,264] ERROR in utils: Error while looking up the TLSA record for .net A DNS label is empty.
[2021-11-23 08:58:17,059] ERROR in utils: Error while looking up the TLSA record for .com A DNS label is empty.
[2021-11-23 09:04:04,597] ERROR in utils: Error while looking up the TLSA record for .org A DNS label is empty.
```
There is no point in having them at all, so let's mute them.
Another (but that arguably is still worth having):
```
[2021-11-23 12:52:46,231] ERROR in utils: Error while looking up the TLSA record for frenger.com The DNS response does not contain an answer to the question: _25._tcp.frenger.com. IN TLSA
[2021-11-24 08:52:57,794] ERROR in utils: Error while looking up the TLSA record for numericable.fr The DNS response does not contain an answer to the question: _25._tcp.numericable.fr. IN TLSA
[2021-11-24 08:52:58,687] ERROR in utils: Error while looking up the TLSA record for neuf.fr The DNS response does not contain an answer to the question: _25._tcp.neuf.fr. IN TLSA
```
For that one I have reduced the severity it's logged at.
Keep in mind that the default action is "pass": this means that we won't impose "dane-only". There will be a test for MTA-STS and then a fallback to "dane" (where postfix will make its own determination as of what those DNS errors should dictate).
Co-authored-by: Florent Daigniere <nextgens@freenetproject.org>
2064: Documentation for switching database-backend and for migrating from Mailu PostgreSQL r=mergify[bot] a=Diman0
## What type of PR?
Documentation
## What does this PR do?
Added documentation for how to switch the database back-end used by Mailu.
Added documentation for migrating from the deprecated Mailu PostgreSQL image to a different PostgreSQL database.
### Related issue(s)
- closes#1037
- closes#1216
- closes#1675
## Prerequisites
Before we can consider review and merge, please make sure the following list is done and checked.
If an entry in not applicable, you can check it or remove it from the list.
- [x] In case of feature or enhancement: documentation updated accordingly
- [x] Unless it's docs or a minor change: add [changelog](https://mailu.io/master/contributors/workflow.html#changelog) entry file.
Co-authored-by: Dimitri Huisman <diman@huisman.xyz>
2066: Upgrade rspamd r=mergify[bot] a=nextgens
## What type of PR?
enhancement
## What does this PR do?
Upgrade rspamd to a version that hopefully won't segfault on arm
### Related issue(s)
- #1200
Co-authored-by: Florent Daigniere <nextgens@freenetproject.org>
2065: Update stale bot with clearer message why an issue is marked stale. r=mergify[bot] a=Diman0
## What type of PR?
enhancement
## What does this PR do?
Update the message from stale bot to provide more info about
- why the issue is marked stale
- after how many days it is marked stale
- when the issue will be closed automatically
- how to remove the stale label.
- stalebot only acts upon user support issues (issues with a label are excluded). Explain how to reach the matrix channel for user support.
### Related issue(s)
- #1582
## Prerequisites
Before we can consider review and merge, please make sure the following list is done and checked.
If an entry in not applicable, you can check it or remove it from the list.
- [n/a] In case of feature or enhancement: documentation updated accordingly
- [n/a] Unless it's docs or a minor change: add [changelog](https://mailu.io/master/contributors/workflow.html#changelog) entry file.
Co-authored-by: Dimitri Huisman <diman@huisman.xyz>
1982: Change memory requirements r=mergify[bot] a=teadur
Running with ClamAV requires atleast 3GB of memory otherwise ClamAV updates fail and fill the disk https://github.com/Mailu/Mailu/issues/470
## What type of PR?
documentation
## What does this PR do?
### Related issue(s)
- Information from #470
## Prerequisites
Before we can consider review and merge, please make sure the following list is done and checked.
If an entry in not applicable, you can check it or remove it from the list.
Co-authored-by: Georg <teadur@users.noreply.github.com>
Co-authored-by: Dimitri Huisman <diman@huisman.xyz>
2063: fixed ipv6 access-control r=mergify[bot] a=ghostwheel42
## What type of PR?
bug-fix
## What does this PR do?
fixes access-control for SUBNET6
Co-authored-by: Alexander Graf <ghostwheel42@users.noreply.github.com>
- Moved run to bottom of Dockerfile to allow using unmodified / cached states.
- Simplified bash code in deploy.sh.
- Improved the large bash one-liner in CI.yml. It could not handle >9 for 1.x.
2054: Testing images are pushed to DOCKER_ORG_TESTS again. r=mergify[bot] a=Diman0
## What type of PR?
Bug fix
## What does this PR do?
Fixes CI workflow. Testing images ( *:pr-xxxx) where pushed to DOCKER_ORG (mailu) instead of DOCKER_ORG_TESTS (mailuci). Images for testing (branch testing) are pushed to mailuci again.
Co-authored-by: Dimitri Huisman <diman@huisman.xyz>
2056: Passlib r=mergify[bot] a=ghostwheel42
## What type of PR?
minor bug-fix
## What does this PR do?
compiles list of schemes using an iterator. will not fail when `scrypt` is not present in registry.
### Related issue(s)
updates #1753
## Prerequisites
Before we can consider review and merge, please make sure the following list is done and checked.
If an entry in not applicable, you can check it or remove it from the list.
- [ ] In case of feature or enhancement: documentation updated accordingly
- [ ] Unless it's docs or a minor change: add [changelog](https://mailu.io/master/contributors/workflow.html#changelog) entry file.
Co-authored-by: Alexander Graf <ghostwheel42@users.noreply.github.com>
2052: Update reverse proxy documentation (see #1962). r=mergify[bot] a=Diman0
## What type of PR?
Bug-fix / documentation
## What does this PR do?
PR #1959 introduces functionality that Mailu must be told what header to trust from a reverse proxy. This PR updates the documentation that for a reverse proxy a header must be configured for passing the remote client IP.
And that in mailu.env file you must configure what header is used by the reverse proxy and what the IP address is of this reverse proxy.
### Related issue(s)
- Auto close an issue like: closes#1962
## Prerequisites
Before we can consider review and merge, please make sure the following list is done and checked.
If an entry in not applicable, you can check it or remove it from the list.
- [x] In case of feature or enhancement: documentation updated accordingly
- [x] Unless it's docs or a minor change: add [changelog](https://mailu.io/master/contributors/workflow.html#changelog) entry file.
Co-authored-by: Dimitri Huisman <diman@huisman.xyz>
2049: fix for issue 1223 (fetchmail persistence idfile) r=mergify[bot] a=Diman0
## What type of PR?
bug-fix
## What does this PR do?
It introduces a new data folder (/mailu/fetchmail) that will hold the idfile. The file that is used by fetchmail to keep track of what messages where retrieved. Recreating the fetchmail container does not result in all messages being retrieved again. It also configurs fetchmail to actually create this file (--uidl).
It changes fetchmail to run as root. For now this is required, because the mailu data folder (/mailu) is owned by root. In the future we must change all images at the same time, to run without root and use a mailu folder that is not owned by root. That is out of scope for this PR.
### Related issue(s)
- closes#1223
## Prerequisites
- [x] In case of feature or enhancement: documentation updated accordingly
- [x] Unless it's docs or a minor change: add [changelog](https://mailu.io/master/contributors/workflow.html#changelog) entry file.
Co-authored-by: Dimitri Huisman <diman@huisman.xyz>
2037: update python dependencies of admin container r=mergify[bot] a=ghostwheel42
## What type of PR?
updates python dependencies of admin container
## What does this PR do?
### Related issue(s)
## Prerequisites
Before we can consider review and merge, please make sure the following list is done and checked.
If an entry in not applicable, you can check it or remove it from the list.
- [X] In case of feature or enhancement: documentation updated accordingly
- [X] Unless it's docs or a minor change: add [changelog](https://mailu.io/master/contributors/workflow.html#changelog) entry file.
Co-authored-by: Alexander Graf <ghostwheel42@users.noreply.github.com>
1224: RFC: Mailu directory structure r=mergify[bot] a=muhlemmer
## What type of PR?
RFC / design documentation
## What does this PR do?
Describes a proposal to restructure the `/mailu` directories to allow for easier and more clear configuration in replicated environments.
It proposes the following layout:
````
/mailu
├── config
│ ├── dovecot
│ ├── postfix
│ ├── rainloop
│ ├── redis
│ ├── roundcube
│ │ └── gpg
│ ├── rspamd
│ └── share
│ ├── certs
│ └── dkim
├── data
│ ├── admin
│ ├── rainloop
│ ├── roundcube
│ └── rspamd
├── local
│ ├── clamav
│ └── mailqueue
└── mail
````
Where in replicated environments:
- `/mailu/config/`: should be a small, low performant and shared filesystem.
- `/mailu/data`: should be avoided. More work will need to be done to configure external DB servers for relevant services. Ideally, this directory should only exist on docker-compose deployments.
- `/mailu/local/`: Should exist only on local file systems of worker nodes.
- `/mailu/mail`: A distributed filesystem with sufficient performance and storage requirements to hold and process all user mailboxes. Ideally only Maildir without indexes.
Co-authored-by: Tim Möhlmann <muhlemmer@gmail.com>
Run fetchmail as root. This is unfortunately required because
all files are owned by root in the mailu data folder.
In the future we must switch all images to running all
all processes with a non-root user.
2047: Do not call .split() on RELAYNETS if not specified r=mergify[bot] a=Grennith
## What type of PR?
bug-fix
## What does this PR do?
The call to {{ RELAYNETS.split(",") | join(' ') }} when starting postfix breaks if RELAYNETS has not been specified using the environmental variables.
## Prerequisites
Before we can consider review and merge, please make sure the following list is done and checked.
If an entry in not applicable, you can check it or remove it from the list.
Co-authored-by: Till Skrodzki <till@mueskro.de>
2044: Vault/rspamd: don't return any key for relayed domains r=mergify[bot] a=nextgens
## What type of PR?
enhancement
## What does this PR
Don't return any key for relayed domains. We may want to revisit this (ARC signing)... but in the meantime it saves from a scary message in rspamd.
```signing failure: cannot request data from the vault url: /internal/rspamd/vault/v1/dkim/ ...```
Co-authored-by: Florent Daigniere <nextgens@freenetproject.org>
2042: Add MESSAGE_RATELIMIT_EXEMPTION r=mergify[bot] a=nextgens
## What type of PR?
Enhancement
## What does this PR do?
Add a new knob called ```MESSAGE_RATELIMIT_EXEMPTION```.
### Related issue(s)
- #1774
## Prerequisites
Before we can consider review and merge, please make sure the following list is done and checked.
If an entry in not applicable, you can check it or remove it from the list.
- [x] In case of feature or enhancement: documentation updated accordingly
- [ ] Unless it's docs or a minor change: add [changelog](https://mailu.io/master/contributors/workflow.html#changelog) entry file.
Co-authored-by: Florent Daigniere <nextgens@freenetproject.org>
bors[bot]