2613: Enhance network segregation r=nextgens a=nextgens
## What type of PR?
enhancement
## What does this PR do?
- put radicale and webmail on their own network: this is done for security: that way they have no privileged access anywhere (no access to redis, no access to XCLIENT, ...)
- remove the EXPOSE statements from the dockerfiles. These ports are for internal comms and are not meant to be exposed in any way to the outside world.
### Related issue(s)
- #2611
## Prerequisites
Before we can consider review and merge, please make sure the following list is done and checked.
If an entry in not applicable, you can check it or remove it from the list.
- [ ] In case of feature or enhancement: documentation updated accordingly
- [ ] Unless it's docs or a minor change: add [changelog](https://mailu.io/master/contributors/workflow.html#changelog) entry file.
Co-authored-by: Florent Daigniere <nextgens@freenetproject.org>
2624: Move runtime environment variables to the end r=nextgens a=ghostwheel42
## What type of PR?
bug-fix
## What does this PR do?
This moves the environment variables used at runtime from the system to the base image.
It's a workaround for a strange build issue observed when building with hardened malloc enabled.
Co-authored-by: Alexander Graf <ghostwheel42@users.noreply.github.com>
2464: Introduce RESTful API r=mergify[bot] a=Diman0
## What type of PR?
Feature
## What does this PR do?
Introduces a RESTful API for changing the complete Mailu config.
Anything that can be configured in the web administration interface, can also be configured via the Mailu RESTful API.
Via the swagger.json endpoint the complete OpenAPI specification can be retrieved.
Via the endpoint swaggerui, a web client is available which shows all the endpoints, data models and allows you to submit requests.
See docs/api.rst and docs/configuration.rst for details for enabling it.
### Related issue(s)
- closes#445
## Prerequisites
Before we can consider review and merge, please make sure the following list is done and checked.
If an entry in not applicable, you can check it or remove it from the list.
- [x] In case of feature or enhancement: documentation updated accordingly
- [x] Unless it's docs or a minor change: add [changelog](https://mailu.io/master/contributors/workflow.html#changelog) entry file.
Co-authored-by: Alexander Graf <ghostwheel42@users.noreply.github.com>
Co-authored-by: Dimitri Huisman <diman@huisman.xyz>
2623: Fix ipv6 subnet for xclient_hosts r=mergify[bot] a=ghostwheel42
## What type of PR?
bug-fix
## What does this PR do?
this puts the ipv6 prefix into square brackets in the xclient_hosts configuration.
strictly speaking putting the square brackets also around the netmask is not correct, but it's okay for postfix
this will be cleaned when all configuration variables and normalizations are moved to the base container.
Co-authored-by: Alexander Graf <ghostwheel42@users.noreply.github.com>
2622: Fix smtplib.LMTP wrong argument name: ip -> host r=mergify[bot] a=zozzz
## What type of PR?
bug-fix
## What does this PR do?
Bug fix when sending welcome message.
### Related issue(s)
- Mention an issue like: -
- Auto close an issue like: closes -
## Prerequisites
Before we can consider review and merge, please make sure the following list is done and checked.
If an entry in not applicable, you can check it or remove it from the list.
- [ ] In case of feature or enhancement: documentation updated accordingly
- [ ] Unless it's docs or a minor change: add [changelog](https://mailu.io/master/contributors/workflow.html#changelog) entry file.
Co-authored-by: Vetési Zoltán <zozzz@trigon.hu>
2621: Upgrade to snuffleupagus 0.9.0 r=mergify[bot] a=nextgens
## What type of PR?
enhancement
## What does this PR do?
Upgrade to snuffleupagus 0.9.0. This has a better way to deal with unserialize() and a better compatibility with PHP8.2
### Related issue(s)
## Prerequisites
Before we can consider review and merge, please make sure the following list is done and checked.
If an entry in not applicable, you can check it or remove it from the list.
- [ ] In case of feature or enhancement: documentation updated accordingly
- [x] Unless it's docs or a minor change: add [changelog](https://mailu.io/master/contributors/workflow.html#changelog) entry file.
Co-authored-by: Florent Daigniere <nextgens@freenetproject.org>
2617: doh r=nextgens a=nextgens
## What type of PR?
bug-fix
## What does this PR do?
### Related issue(s)
## Prerequisites
Before we can consider review and merge, please make sure the following list is done and checked.
If an entry in not applicable, you can check it or remove it from the list.
- [ ] In case of feature or enhancement: documentation updated accordingly
- [ ] Unless it's docs or a minor change: add [changelog](https://mailu.io/master/contributors/workflow.html#changelog) entry file.
Co-authored-by: Florent Daigniere <nextgens@freenetproject.org>
2615: Fix snappymail r=mergify[bot] a=nextgens
## What type of PR?
bug-fix
## What does this PR do?
This was broken in #2611. The other is non-consequential security-wise as nginx filters XHOST... but it's worth fixing regardless.
### Related issue(s)
- #2580
- #2611
## Prerequisites
Before we can consider review and merge, please make sure the following list is done and checked.
If an entry in not applicable, you can check it or remove it from the list.
- [ ] In case of feature or enhancement: documentation updated accordingly
- [ ] Unless it's docs or a minor change: add [changelog](https://mailu.io/master/contributors/workflow.html#changelog) entry file.
Co-authored-by: Florent Daigniere <nextgens@freenetproject.org>
2612: Remove duplicated 'actionstart = ' in fail2ban conf. r=mergify[bot] a=lucassith
In fail2ban example configuration for ipset `option #2`, there is a duplicated string which makes the ipset and fail2ban fail to create the set.
Fail2ban will never ban any ip due to this error.
## What type of PR?
documentation
Co-authored-by: Łukasz Sitarski <Lucassith@gmail.com>
In fail2ban example configuration for ipset option, there was a duplicated string which makes the ipset and fail2ban fail to create the set.
Fail2ban will never ban any ip due to this error.
2611: Fix authenticated submission r=mergify[bot] a=nextgens
## What type of PR?
bug-fix
## What does this PR do?
Don't talk haproxy to postfix; it's more headaches than it is currently worth.
### Related issue(s)
- #2603
## Prerequisites
Before we can consider review and merge, please make sure the following list is done and checked.
If an entry in not applicable, you can check it or remove it from the list.
- [ ] In case of feature or enhancement: documentation updated accordingly
- [x] Unless it's docs or a minor change: add [changelog](https://mailu.io/master/contributors/workflow.html#changelog) entry file.
Co-authored-by: Florent Daigniere <nextgens@freenetproject.org>
2603: Enable HAPROXY protocol on SUBNET r=mergify[bot] a=nextgens
## What type of PR?
bug-fix
## What does this PR do?
- Enable HAPROXY in between front and imap: With this we avoid running into the limitations of ``mail_max_userip_connections`` and the logfiles reflect the real IP.
- Enable HAPROXY in between front and smtp: with this postfix and rspamd are aware of whether TLS was used or not on the last hop. In practice this won't work as nginx doesn't send PROTO yet.
- Discard redundant log messages from postfix
With all of this, not only are the logs easier to understand but ``doveadm who`` also works as one would expect.
### Related issue(s)
- closes#894
- #1328
- closes#1364
- #1705
## Prerequisites
Before we can consider review and merge, please make sure the following list is done and checked.
If an entry in not applicable, you can check it or remove it from the list.
- [ ] In case of feature or enhancement: documentation updated accordingly
- [x] Unless it's docs or a minor change: add [changelog](https://mailu.io/master/contributors/workflow.html#changelog) entry file.
Co-authored-by: Florent Daigniere <nextgens@freenetproject.org>
2596: db.String without length cause an error in migration for MySQL DB r=mergify[bot] a=csthiang
## What type of PR?
bug-fix
## What does this PR do?
For MySQL, `db.String` requires a length because db.String gets translated to `VARCHAR` in MySQL and `VARCHAR` requires a length. I was considering adding a length to it but since the affected fields were used to store CommaSeparatedList and json-encoded string, I have a feeling it can be quite large in the future. `db.Text` seems to fit into this use case but please correct me if I am wrong.
This actually affects a DB migration with the following error:
```
File "/app/venv/bin/flask", line 8, in <module>
sys.exit(main())
File "/app/venv/lib/python3.10/site-packages/flask/cli.py", line 1047, in main
cli.main()
File "/app/venv/lib/python3.10/site-packages/click/core.py", line 1055, in main
rv = self.invoke(ctx)
File "/app/venv/lib/python3.10/site-packages/click/core.py", line 1657, in invoke
return _process_result(sub_ctx.command.invoke(sub_ctx))
File "/app/venv/lib/python3.10/site-packages/click/core.py", line 1657, in invoke
return _process_result(sub_ctx.command.invoke(sub_ctx))
File "/app/venv/lib/python3.10/site-packages/click/core.py", line 1404, in invoke
return ctx.invoke(self.callback, **ctx.params)
File "/app/venv/lib/python3.10/site-packages/click/core.py", line 760, in invoke
return __callback(*args, **kwargs)
File "/app/venv/lib/python3.10/site-packages/click/decorators.py", line 26, in new_func
return f(get_current_context(), *args, **kwargs)
File "/app/venv/lib/python3.10/site-packages/flask/cli.py", line 357, in decorator
return __ctx.invoke(f, *args, **kwargs)
File "/app/venv/lib/python3.10/site-packages/click/core.py", line 760, in invoke
return __callback(*args, **kwargs)
File "/app/venv/lib/python3.10/site-packages/flask_migrate/cli.py", line 149, in upgrade
_upgrade(directory, revision, sql, tag, x_arg)
File "/app/venv/lib/python3.10/site-packages/flask_migrate/__init__.py", line 98, in wrapped
f(*args, **kwargs)
File "/app/venv/lib/python3.10/site-packages/flask_migrate/__init__.py", line 185, in upgrade
command.upgrade(config, revision, sql=sql, tag=tag)
File "/app/venv/lib/python3.10/site-packages/alembic/command.py", line 322, in upgrade
script.run_env()
File "/app/venv/lib/python3.10/site-packages/alembic/script/base.py", line 569, in run_env
util.load_python_file(self.dir, "env.py")
File "/app/venv/lib/python3.10/site-packages/alembic/util/pyfiles.py", line 94, in load_python_file
module = load_module_py(module_id, path)
File "/app/venv/lib/python3.10/site-packages/alembic/util/pyfiles.py", line 110, in load_module_py
spec.loader.exec_module(module) # type: ignore
File "<frozen importlib._bootstrap_external>", line 883, in exec_module
File "<frozen importlib._bootstrap>", line 241, in _call_with_frames_removed
File "/app/migrations/env.py", line 99, in <module>
run_migrations_online()
File "/app/migrations/env.py", line 92, in run_migrations_online
context.run_migrations()
File "<string>", line 8, in run_migrations
File "/app/venv/lib/python3.10/site-packages/alembic/runtime/environment.py", line 853, in run_migrations
self.get_context().run_migrations(**kw)
File "/app/venv/lib/python3.10/site-packages/alembic/runtime/migration.py", line 623, in run_migrations
step.migration_fn(**kw)
File "/app/migrations/versions/f4f0f89e0047_.py", line 18, in upgrade
with op.batch_alter_table('fetch') as batch:
File "/usr/lib/python3.10/contextlib.py", line 142, in __exit__
next(self.gen)
File "/app/venv/lib/python3.10/site-packages/alembic/operations/base.py", line 381, in batch_alter_table
impl.flush()
File "/app/venv/lib/python3.10/site-packages/alembic/operations/batch.py", line 111, in flush
fn(*arg, **kw)
File "/app/venv/lib/python3.10/site-packages/alembic/ddl/impl.py", line 322, in add_column
self._exec(base.AddColumn(table_name, column, schema=schema))
File "/app/venv/lib/python3.10/site-packages/alembic/ddl/impl.py", line 195, in _exec
return conn.execute(construct, multiparams)
File "/app/venv/lib/python3.10/site-packages/sqlalchemy/engine/base.py", line 1380, in execute
return meth(self, multiparams, params, _EMPTY_EXECUTION_OPTS)
File "/app/venv/lib/python3.10/site-packages/sqlalchemy/sql/ddl.py", line 80, in _execute_on_connection
return connection._execute_ddl(
File "/app/venv/lib/python3.10/site-packages/sqlalchemy/engine/base.py", line 1469, in _execute_ddl
compiled = ddl.compile(
File "/app/venv/lib/python3.10/site-packages/sqlalchemy/sql/elements.py", line 502, in compile
return self._compiler(dialect, **kw)
File "/app/venv/lib/python3.10/site-packages/sqlalchemy/sql/ddl.py", line 32, in _compiler
return dialect.ddl_compiler(dialect, self, **kw)
File "/app/venv/lib/python3.10/site-packages/sqlalchemy/sql/compiler.py", line 463, in __init__
self.string = self.process(self.statement, **compile_kwargs)
File "/app/venv/lib/python3.10/site-packages/sqlalchemy/sql/compiler.py", line 498, in process
return obj._compiler_dispatch(self, **kwargs)
File "/app/venv/lib/python3.10/site-packages/sqlalchemy/ext/compiler.py", line 548, in <lambda>
lambda *arg, **kw: existing(*arg, **kw),
File "/app/venv/lib/python3.10/site-packages/sqlalchemy/ext/compiler.py", line 604, in __call__
expr = fn(element, compiler, **kw)
File "/app/venv/lib/python3.10/site-packages/alembic/ddl/base.py", line 190, in visit_add_column
add_column(compiler, element.column, **kw),
File "/app/venv/lib/python3.10/site-packages/alembic/ddl/base.py", line 330, in add_column
text = "ADD COLUMN %s" % compiler.get_column_specification(column, **kw)
File "/app/venv/lib/python3.10/site-packages/sqlalchemy/dialects/mysql/base.py", line 1714, in get_column_specification
self.dialect.type_compiler.process(
File "/app/venv/lib/python3.10/site-packages/sqlalchemy/sql/compiler.py", line 532, in process
return type_._compiler_dispatch(self, **kw)
File "/app/venv/lib/python3.10/site-packages/sqlalchemy/sql/visitors.py", line 82, in _compiler_dispatch
return meth(self, **kw)
File "/app/venv/lib/python3.10/site-packages/sqlalchemy/sql/compiler.py", line 5028, in visit_type_decorator
return self.process(type_.type_engine(self.dialect), **kw)
File "/app/venv/lib/python3.10/site-packages/sqlalchemy/sql/compiler.py", line 532, in process
return type_._compiler_dispatch(self, **kw)
File "/app/venv/lib/python3.10/site-packages/sqlalchemy/sql/visitors.py", line 82, in _compiler_dispatch
return meth(self, **kw)
File "/app/venv/lib/python3.10/site-packages/sqlalchemy/sql/compiler.py", line 5006, in visit_string
return self.visit_VARCHAR(type_, **kw)
File "/app/venv/lib/python3.10/site-packages/sqlalchemy/dialects/mysql/base.py", line 2214, in visit_VARCHAR
raise exc.CompileError(
sqlalchemy.exc.CompileError: VARCHAR requires a length on dialect mysql
[2022-12-22 09:23:12 +0000] [17] [INFO] Starting gunicorn 20.1.0
[2022-12-22 09:23:12 +0000] [17] [INFO] Listening at: http://0.0.0.0:80 (17)
[2022-12-22 09:23:12 +0000] [17] [INFO] Using worker: gthread
[2022-12-22 09:23:12 +0000] [18] [INFO] Booting worker with pid: 18
```
### Related issue(s)
none
## Prerequisites
Before we can consider review and merge, please make sure the following list is done and checked.
If an entry in not applicable, you can check it or remove it from the list.
- [x] In case of feature or enhancement: documentation updated accordingly
- [x] Unless it's docs or a minor change: add [changelog](https://mailu.io/master/contributors/workflow.html#changelog) entry file.
Co-authored-by: Johnson Thiang <jthiang@pop-os.localdomain>
Co-authored-by: Florent Daigniere <nextgens@users.noreply.github.com>
2607: Update python dependencies as suggested by dependabot r=mergify[bot] a=ghostwheel42
## What type of PR?
bug-fix
## What does this PR do?
Update dependencies to silence dependabot (vulnerabilities are probably not exploitable)
Only the certifi upgrade could be backported.
Co-authored-by: Alexander Graf <ghostwheel42@users.noreply.github.com>
This is not perfect...
- dovecot now complains about waitpid/finding a new process
- postfix is still regularly pinging rspamd / his milter and that
generates a few lines worth of logs each time.
2605: Reduce the SSL session caches from 50m each to 3m each r=nextgens a=nextgens
## What type of PR?
enhancement
## What does this PR do?
Reduce the SSL session caches from 50m each to 3m each. This should be good for 12k sessions (within 1day, see http://nginx.org/en/docs/http/ngx_http_ssl_module.html#ssl_session_cache and our ssl_session_timeout) for each cache and will help reduce memory usage.
### Related issue(s)
## Prerequisites
Before we can consider review and merge, please make sure the following list is done and checked.
If an entry in not applicable, you can check it or remove it from the list.
- [ ] In case of feature or enhancement: documentation updated accordingly
- [x] Unless it's docs or a minor change: add [changelog](https://mailu.io/master/contributors/workflow.html#changelog) entry file.
Co-authored-by: Florent Daigniere <nextgens@freenetproject.org>
With this we avoid running into the limitations of
mail_max_userip_connections (see #894 amd #1364) and the
logfiles as well as ``doveadm who`` give an accurate picture.
2601: Fix creation of deep structures using import in update mode r=mergify[bot] a=ghostwheel42
## What type of PR?
bug-fix
## What does this PR do?
Fixes creation of deep structures (ie user with fetch) when using config-import in update mode.
### Related issue(s)
- closes#2493
## Prerequisites
Before we can consider review and merge, please make sure the following list is done and checked.
If an entry in not applicable, you can check it or remove it from the list.
- [ ] In case of feature or enhancement: documentation updated accordingly
- [ ] Unless it's docs or a minor change: add [changelog](https://mailu.io/master/contributors/workflow.html#changelog) entry file.
Co-authored-by: Alexander Graf <ghostwheel42@users.noreply.github.com>
Alexander Graf