1664 Commits (120a7e83681667a8d1c8599d1c88227ce7f73e9f)

Author SHA1 Message Date
Alexander Graf 842be9b7c3
Skip listen to v6 when SUBNET6 is not set 2 years ago
Alexander Graf 1ad1d8d95d
Rewrite generation of gunicorn cmdline 2 years ago
Chris Schäpers 35331a4295
Make gunicorn IPv6 conditional
Only listen on [::]:80 in case SUBNET6 is defined, otherwise do the normal :80
2 years ago
Chris 9f6848110a
Make gunicorn listen on ipv6 2 years ago
bors[bot] db2a490256
Merge #2633
2633: Don't apply antispoof rules on locally generated emails r=mergify[bot] a=nextgens

## What type of PR?

bug-fix

## What does this PR do?

Don't apply antispoof rules on locally generated emails; This was breaking the auto-responder and sieve rules.

### Related issue(s)


## Prerequisites
Before we can consider review and merge, please make sure the following list is done and checked.
If an entry in not applicable, you can check it or remove it from the list.

- [ ] In case of feature or enhancement: documentation updated accordingly
- [x] Unless it's docs or a minor change: add [changelog](https://mailu.io/master/contributors/workflow.html#changelog) entry file.


Co-authored-by: Florent Daigniere <nextgens@freenetproject.org>
2 years ago
Florent Daigniere 46f05cb651 Merge branch 'master' of https://github.com/Mailu/Mailu into reduce-logging 2 years ago
Florent Daigniere 36623188b5 Don't apply antispoof rules on locally generated emails 2 years ago
bors[bot] 179c624116
Merge #2631
2631: Restful api finishing touches r=mergify[bot] a=Diman0

## What type of PR?

enhancement

## What does this PR do?
Some finishing touches for the restful api.

- Make the API configurable via the setup utility.  
  - Configured exactly the same as the ADMIN and WEBMAIL. 
- We have a single config (API) that configures whether it is exposed (via front). Just like ADMIN. The API is always reachable by directly connecting to the admin container.
- API_TOKEN does not enable/disable the API anymore. When it is not configured, an error is returned (via the internet browser) that the API_TOKEN must be configured in mailu.env.
- Fix some small bugs in the setup utility ( selecting none in the dropdown boxes, now correctly changes the config)
- Update Flask-RestX to 1.0.5. This resolves the deprecation warnings introduced by Flask-RestX.

### Related issue(s)

## Prerequisites
Before we can consider review and merge, please make sure the following list is done and checked.
If an entry in not applicable, you can check it or remove it from the list.

- [x] In case of feature or enhancement: documentation updated accordingly
- [x] Unless it's docs or a minor change: add [changelog](https://mailu.io/master/contributors/workflow.html#changelog) entry file.


Co-authored-by: Dimitri Huisman <diman@huisman.xyz>
2 years ago
bors[bot] 43e500faf5
Merge #2628
2628: Set default for FETCHMAIL_ENABLED r=mergify[bot] a=ghostwheel42

## What type of PR?

bug-fix

## What does this PR do?

Set the default for FETCHMAIL_ENABLED to true in the admin container.
This keeps existing functionality for people upgrading without re-creating the `mailu.env`.


Co-authored-by: Alexander Graf <ghostwheel42@users.noreply.github.com>
2 years ago
Dimitri Huisman 18b900699c
Bump version of Flask-RESTX to 1.0.5.
This resolves all deprecation warnings caused by Flask-RESTX.
2 years ago
Dimitri Huisman d6e7314f05
Make API configurable via the setup utility
Fix some small bugs in the setup utility
Improve documentation on the API.
2 years ago
Alexander Graf c4ca1cffaf
Set default for FETCHMAIL_ENABLED 2 years ago
Alexander Graf 5c968256e6
Really fix creation of deep structures using import in update mode 2 years ago
bors[bot] 151601744f
Merge #2627
2627: Add SUBNET6 to places where SUBNET is used r=nextgens a=ghostwheel42

## What type of PR?

bug-fix

## What does this PR do?

Also add SUBNET6 where SUBNET is used.

Co-authored-by: Alexander Graf <ghostwheel42@users.noreply.github.com>
2 years ago
bors[bot] 6d994525c4
Merge #2625
2625: Disable fetchmail r=mergify[bot] a=ghostwheel42

## What type of PR?

bug-fix

## What does this PR do?

Only show "fetched accounts" button in user list when fetchmail feature is enabled.


Co-authored-by: Alexander Graf <ghostwheel42@users.noreply.github.com>
2 years ago
Alexander Graf 10562233ca
Add SUBNET6 to places where SUBNET is used 2 years ago
bors[bot] 7e60ba4e98
Merge #2613
2613: Enhance network segregation r=nextgens a=nextgens

## What type of PR?

enhancement

## What does this PR do?

- put radicale and webmail on their own network: this is done for security: that way they have no privileged access anywhere (no access to redis, no access to XCLIENT, ...)
- remove the EXPOSE statements from the dockerfiles. These ports are for internal comms and are not meant to be exposed in any way to the outside world.

### Related issue(s)
- #2611

## Prerequisites
Before we can consider review and merge, please make sure the following list is done and checked.
If an entry in not applicable, you can check it or remove it from the list.

- [ ] In case of feature or enhancement: documentation updated accordingly
- [ ] Unless it's docs or a minor change: add [changelog](https://mailu.io/master/contributors/workflow.html#changelog) entry file.


Co-authored-by: Florent Daigniere <nextgens@freenetproject.org>
2 years ago
Alexander Graf 1697da6e23
Disable "Fetched accounts" button in user list. 2 years ago
bors[bot] dae9e9242b
Merge #2624
2624: Move runtime environment variables to the end r=nextgens a=ghostwheel42

## What type of PR?

bug-fix

## What does this PR do?

This moves the environment variables used at runtime from the system to the base image.
It's a workaround for a strange build issue observed when building with hardened malloc enabled.



Co-authored-by: Alexander Graf <ghostwheel42@users.noreply.github.com>
2 years ago
bors[bot] bbf0ac5d47
Merge #2464
2464: Introduce RESTful API r=mergify[bot] a=Diman0

## What type of PR?

Feature

## What does this PR do?
Introduces a RESTful API for changing the complete Mailu config.
Anything that can be configured in the web administration interface, can also be configured via the Mailu RESTful API.

Via the swagger.json endpoint the complete OpenAPI specification can be retrieved.
Via the endpoint swaggerui, a web client is available which shows all the endpoints, data models and allows you to submit requests.

See docs/api.rst and docs/configuration.rst for details for enabling it.

### Related issue(s)
- closes #445 

## Prerequisites
Before we can consider review and merge, please make sure the following list is done and checked.
If an entry in not applicable, you can check it or remove it from the list.

- [x] In case of feature or enhancement: documentation updated accordingly
- [x] Unless it's docs or a minor change: add [changelog](https://mailu.io/master/contributors/workflow.html#changelog) entry file.


Co-authored-by: Alexander Graf <ghostwheel42@users.noreply.github.com>
Co-authored-by: Dimitri Huisman <diman@huisman.xyz>
2 years ago
Alexander Graf 712679b4d8
Duh 2 years ago
Alexander Graf d558be20f6
Move runtime environment variables to the end 2 years ago
Alexander Graf 3b08b113bf
Fix ipv6 subnet for xclient_hosts 2 years ago
Dimitri Huisman b0569035ae
Change PUT method to PATCH method.
This better reflects what the interface does.
2 years ago
Vetési Zoltán e76e857ae7 Fix smtplib.LMTP wrong argument name: ip -> host 2 years ago
Florent Daigniere 052f8e41ba Upgrade to snuffleupagus 0.9.0 2 years ago
Florent Daigniere ee6975b109 doh 2 years ago
Dimitri Huisman bcceac359d
Merge branch 'apiv1' of https://github.com/ghostwheel42/Mailu into feature-445-restful-api-ghostwheel 2 years ago
Florent Daigniere 9d555b0eec Don't expose any port (suggestion from ghost) 2 years ago
Florent Daigniere e85a2a7e99 Step1: expose managesieve, make the webmails use it 2 years ago
Florent Daigniere b263db72df Restrict XHOST to where useful 2 years ago
Florent Daigniere 4d80c95c41 Fix authentication submission
Don't talk haproxy to postfix; it's more headaches than it is currently
worth.
2 years ago
bors[bot] bba6c5bb88
Merge #2603
2603: Enable HAPROXY protocol on SUBNET r=mergify[bot] a=nextgens

## What type of PR?

bug-fix

## What does this PR do?

- Enable HAPROXY in between front and imap: With this we avoid running into the limitations of  ``mail_max_userip_connections`` and the logfiles reflect the real IP.
- Enable HAPROXY in between front and smtp: with this postfix and rspamd are aware of whether TLS was used or not on the last hop. In practice this won't work as nginx doesn't send PROTO yet.
- Discard redundant log messages from postfix

With all of this, not only are the logs easier to understand but ``doveadm who`` also works as one would expect.

### Related issue(s)
- closes #894
- #1328
- closes #1364
- #1705

## Prerequisites
Before we can consider review and merge, please make sure the following list is done and checked.
If an entry in not applicable, you can check it or remove it from the list.

- [ ] In case of feature or enhancement: documentation updated accordingly
- [x] Unless it's docs or a minor change: add [changelog](https://mailu.io/master/contributors/workflow.html#changelog) entry file.


Co-authored-by: Florent Daigniere <nextgens@freenetproject.org>
2 years ago
bors[bot] 52c17411bd
Merge #2596
2596: db.String without length cause an error in migration for MySQL DB r=mergify[bot] a=csthiang

## What type of PR?

bug-fix

## What does this PR do?

For MySQL, `db.String` requires a length because db.String gets translated to `VARCHAR` in MySQL and `VARCHAR` requires a length. I was considering adding a length to it but since the affected fields were used to store CommaSeparatedList and json-encoded string, I have a feeling it can be quite large in the future. `db.Text` seems to fit into this use case but please correct me if I am wrong.

This actually affects a DB migration with the following error:

```
  File "/app/venv/bin/flask", line 8, in <module>
    sys.exit(main())
  File "/app/venv/lib/python3.10/site-packages/flask/cli.py", line 1047, in main
    cli.main()
  File "/app/venv/lib/python3.10/site-packages/click/core.py", line 1055, in main
    rv = self.invoke(ctx)
  File "/app/venv/lib/python3.10/site-packages/click/core.py", line 1657, in invoke
    return _process_result(sub_ctx.command.invoke(sub_ctx))
  File "/app/venv/lib/python3.10/site-packages/click/core.py", line 1657, in invoke
    return _process_result(sub_ctx.command.invoke(sub_ctx))
  File "/app/venv/lib/python3.10/site-packages/click/core.py", line 1404, in invoke
    return ctx.invoke(self.callback, **ctx.params)
  File "/app/venv/lib/python3.10/site-packages/click/core.py", line 760, in invoke
    return __callback(*args, **kwargs)
  File "/app/venv/lib/python3.10/site-packages/click/decorators.py", line 26, in new_func
    return f(get_current_context(), *args, **kwargs)
  File "/app/venv/lib/python3.10/site-packages/flask/cli.py", line 357, in decorator
    return __ctx.invoke(f, *args, **kwargs)
  File "/app/venv/lib/python3.10/site-packages/click/core.py", line 760, in invoke
    return __callback(*args, **kwargs)
  File "/app/venv/lib/python3.10/site-packages/flask_migrate/cli.py", line 149, in upgrade
    _upgrade(directory, revision, sql, tag, x_arg)
  File "/app/venv/lib/python3.10/site-packages/flask_migrate/__init__.py", line 98, in wrapped
    f(*args, **kwargs)
  File "/app/venv/lib/python3.10/site-packages/flask_migrate/__init__.py", line 185, in upgrade
    command.upgrade(config, revision, sql=sql, tag=tag)
  File "/app/venv/lib/python3.10/site-packages/alembic/command.py", line 322, in upgrade
    script.run_env()
  File "/app/venv/lib/python3.10/site-packages/alembic/script/base.py", line 569, in run_env
    util.load_python_file(self.dir, "env.py")
  File "/app/venv/lib/python3.10/site-packages/alembic/util/pyfiles.py", line 94, in load_python_file
    module = load_module_py(module_id, path)
  File "/app/venv/lib/python3.10/site-packages/alembic/util/pyfiles.py", line 110, in load_module_py
    spec.loader.exec_module(module)  # type: ignore
  File "<frozen importlib._bootstrap_external>", line 883, in exec_module
  File "<frozen importlib._bootstrap>", line 241, in _call_with_frames_removed
  File "/app/migrations/env.py", line 99, in <module>
    run_migrations_online()
  File "/app/migrations/env.py", line 92, in run_migrations_online
    context.run_migrations()
  File "<string>", line 8, in run_migrations
  File "/app/venv/lib/python3.10/site-packages/alembic/runtime/environment.py", line 853, in run_migrations
    self.get_context().run_migrations(**kw)
  File "/app/venv/lib/python3.10/site-packages/alembic/runtime/migration.py", line 623, in run_migrations
    step.migration_fn(**kw)
  File "/app/migrations/versions/f4f0f89e0047_.py", line 18, in upgrade
    with op.batch_alter_table('fetch') as batch:
  File "/usr/lib/python3.10/contextlib.py", line 142, in __exit__
    next(self.gen)
  File "/app/venv/lib/python3.10/site-packages/alembic/operations/base.py", line 381, in batch_alter_table
    impl.flush()
  File "/app/venv/lib/python3.10/site-packages/alembic/operations/batch.py", line 111, in flush
    fn(*arg, **kw)
  File "/app/venv/lib/python3.10/site-packages/alembic/ddl/impl.py", line 322, in add_column
    self._exec(base.AddColumn(table_name, column, schema=schema))
  File "/app/venv/lib/python3.10/site-packages/alembic/ddl/impl.py", line 195, in _exec
    return conn.execute(construct, multiparams)
  File "/app/venv/lib/python3.10/site-packages/sqlalchemy/engine/base.py", line 1380, in execute
    return meth(self, multiparams, params, _EMPTY_EXECUTION_OPTS)
  File "/app/venv/lib/python3.10/site-packages/sqlalchemy/sql/ddl.py", line 80, in _execute_on_connection
    return connection._execute_ddl(
  File "/app/venv/lib/python3.10/site-packages/sqlalchemy/engine/base.py", line 1469, in _execute_ddl
    compiled = ddl.compile(
  File "/app/venv/lib/python3.10/site-packages/sqlalchemy/sql/elements.py", line 502, in compile
    return self._compiler(dialect, **kw)
  File "/app/venv/lib/python3.10/site-packages/sqlalchemy/sql/ddl.py", line 32, in _compiler
    return dialect.ddl_compiler(dialect, self, **kw)
  File "/app/venv/lib/python3.10/site-packages/sqlalchemy/sql/compiler.py", line 463, in __init__
    self.string = self.process(self.statement, **compile_kwargs)
  File "/app/venv/lib/python3.10/site-packages/sqlalchemy/sql/compiler.py", line 498, in process
    return obj._compiler_dispatch(self, **kwargs)
  File "/app/venv/lib/python3.10/site-packages/sqlalchemy/ext/compiler.py", line 548, in <lambda>
    lambda *arg, **kw: existing(*arg, **kw),
  File "/app/venv/lib/python3.10/site-packages/sqlalchemy/ext/compiler.py", line 604, in __call__
    expr = fn(element, compiler, **kw)
  File "/app/venv/lib/python3.10/site-packages/alembic/ddl/base.py", line 190, in visit_add_column
    add_column(compiler, element.column, **kw),
  File "/app/venv/lib/python3.10/site-packages/alembic/ddl/base.py", line 330, in add_column
    text = "ADD COLUMN %s" % compiler.get_column_specification(column, **kw)
  File "/app/venv/lib/python3.10/site-packages/sqlalchemy/dialects/mysql/base.py", line 1714, in get_column_specification
    self.dialect.type_compiler.process(
  File "/app/venv/lib/python3.10/site-packages/sqlalchemy/sql/compiler.py", line 532, in process
    return type_._compiler_dispatch(self, **kw)
  File "/app/venv/lib/python3.10/site-packages/sqlalchemy/sql/visitors.py", line 82, in _compiler_dispatch
    return meth(self, **kw)
  File "/app/venv/lib/python3.10/site-packages/sqlalchemy/sql/compiler.py", line 5028, in visit_type_decorator
    return self.process(type_.type_engine(self.dialect), **kw)
  File "/app/venv/lib/python3.10/site-packages/sqlalchemy/sql/compiler.py", line 532, in process
    return type_._compiler_dispatch(self, **kw)
  File "/app/venv/lib/python3.10/site-packages/sqlalchemy/sql/visitors.py", line 82, in _compiler_dispatch
    return meth(self, **kw)
  File "/app/venv/lib/python3.10/site-packages/sqlalchemy/sql/compiler.py", line 5006, in visit_string
    return self.visit_VARCHAR(type_, **kw)
  File "/app/venv/lib/python3.10/site-packages/sqlalchemy/dialects/mysql/base.py", line 2214, in visit_VARCHAR
    raise exc.CompileError(
sqlalchemy.exc.CompileError: VARCHAR requires a length on dialect mysql
[2022-12-22 09:23:12 +0000] [17] [INFO] Starting gunicorn 20.1.0
[2022-12-22 09:23:12 +0000] [17] [INFO] Listening at: http://0.0.0.0:80 (17)
[2022-12-22 09:23:12 +0000] [17] [INFO] Using worker: gthread
[2022-12-22 09:23:12 +0000] [18] [INFO] Booting worker with pid: 18
```

### Related issue(s)
none

## Prerequisites
Before we can consider review and merge, please make sure the following list is done and checked.
If an entry in not applicable, you can check it or remove it from the list.

- [x] In case of feature or enhancement: documentation updated accordingly
- [x] Unless it's docs or a minor change: add [changelog](https://mailu.io/master/contributors/workflow.html#changelog) entry file.


Co-authored-by: Johnson Thiang <jthiang@pop-os.localdomain>
Co-authored-by: Florent Daigniere <nextgens@users.noreply.github.com>
2 years ago
Florent Daigniere ca44ccbe1c
Use the size other implementations default to 2 years ago
Alexander Graf 6f71ea833b
Update python dependencies as suggested by dependabot 2 years ago
Florent Daigniere edd303f54d Modify the healtchecks to make them disapear from the logs.
This is not perfect...
- dovecot now complains about waitpid/finding a new process
- postfix is still regularly pinging rspamd / his milter and that
generates a few lines worth of logs each time.
2 years ago
Florent Daigniere cd107182c1 comment 2 years ago
Florent Daigniere 8539344331 Reduce nginx ssl_session_cache to 3m each 2 years ago
Florent Daigniere 36b3a9f4fb Will fix it in another PR 2 years ago
Florent Daigniere 83ea708490 fix healthcheck 2 years ago
Florent Daigniere 7a2d06401a Tweak postfix logging 2 years ago
Florent Daigniere 55c1e55529 Same for front-smtp
This should enable postfix to have visibility on TLS usage and fix the
following: #1705
2 years ago
Florent Daigniere 4ae0d7d768 Enable HAPROXY protocol in between front and imap
With this we avoid running into the limitations of
 mail_max_userip_connections (see #894 amd #1364) and the
 logfiles as well as ``doveadm who`` give an accurate picture.
2 years ago
Alexander Graf be40781394
Add default for WEB_API, re-add flask-restx to deps, remove whitespace 2 years ago
Alexander Graf 84ebab2cb4
Fix creation of deep structures using import in update mode 2 years ago
Dimitri Huisman 3cb8358090
Process review comments PR#2464
- When visiting root of WEB_API, the swaggerui is shown
- simplify the condition for endpoint WEB_API
2 years ago
Dimitri Huisman 39b0d44079
Use first() instead of all() for better performance
Actually return all data for Get user
Remove non-used code
2 years ago
Dimitri Huisman f9b26bd934
Update User with newly introduced allow spoofing field 2 years ago
Dimitri Huisman 6347c18f8a
Process review comments (PR2464) 2 years ago