4158 Commits (8eb1542f64075b0c7e6bc88f8dca6ada3387aa8a)
 

Author SHA1 Message Date
Florent Daigniere 4d80c95c41 Fix authentication submission
Don't talk haproxy to postfix; it's more headaches than it is currently
worth.
2 years ago
bors[bot] bba6c5bb88
Merge #2603
2603: Enable HAPROXY protocol on SUBNET r=mergify[bot] a=nextgens

## What type of PR?

bug-fix

## What does this PR do?

- Enable HAPROXY in between front and imap: With this we avoid running into the limitations of  ``mail_max_userip_connections`` and the logfiles reflect the real IP.
- Enable HAPROXY in between front and smtp: with this postfix and rspamd are aware of whether TLS was used or not on the last hop. In practice this won't work as nginx doesn't send PROTO yet.
- Discard redundant log messages from postfix

With all of this, not only are the logs easier to understand but ``doveadm who`` also works as one would expect.

### Related issue(s)
- closes #894
- #1328
- closes #1364
- #1705

## Prerequisites
Before we can consider review and merge, please make sure the following list is done and checked.
If an entry in not applicable, you can check it or remove it from the list.

- [ ] In case of feature or enhancement: documentation updated accordingly
- [x] Unless it's docs or a minor change: add [changelog](https://mailu.io/master/contributors/workflow.html#changelog) entry file.


Co-authored-by: Florent Daigniere <nextgens@freenetproject.org>
2 years ago
bors[bot] 52c17411bd
Merge #2596
2596: db.String without length cause an error in migration for MySQL DB r=mergify[bot] a=csthiang

## What type of PR?

bug-fix

## What does this PR do?

For MySQL, `db.String` requires a length because db.String gets translated to `VARCHAR` in MySQL and `VARCHAR` requires a length. I was considering adding a length to it but since the affected fields were used to store CommaSeparatedList and json-encoded string, I have a feeling it can be quite large in the future. `db.Text` seems to fit into this use case but please correct me if I am wrong.

This actually affects a DB migration with the following error:

```
  File "/app/venv/bin/flask", line 8, in <module>
    sys.exit(main())
  File "/app/venv/lib/python3.10/site-packages/flask/cli.py", line 1047, in main
    cli.main()
  File "/app/venv/lib/python3.10/site-packages/click/core.py", line 1055, in main
    rv = self.invoke(ctx)
  File "/app/venv/lib/python3.10/site-packages/click/core.py", line 1657, in invoke
    return _process_result(sub_ctx.command.invoke(sub_ctx))
  File "/app/venv/lib/python3.10/site-packages/click/core.py", line 1657, in invoke
    return _process_result(sub_ctx.command.invoke(sub_ctx))
  File "/app/venv/lib/python3.10/site-packages/click/core.py", line 1404, in invoke
    return ctx.invoke(self.callback, **ctx.params)
  File "/app/venv/lib/python3.10/site-packages/click/core.py", line 760, in invoke
    return __callback(*args, **kwargs)
  File "/app/venv/lib/python3.10/site-packages/click/decorators.py", line 26, in new_func
    return f(get_current_context(), *args, **kwargs)
  File "/app/venv/lib/python3.10/site-packages/flask/cli.py", line 357, in decorator
    return __ctx.invoke(f, *args, **kwargs)
  File "/app/venv/lib/python3.10/site-packages/click/core.py", line 760, in invoke
    return __callback(*args, **kwargs)
  File "/app/venv/lib/python3.10/site-packages/flask_migrate/cli.py", line 149, in upgrade
    _upgrade(directory, revision, sql, tag, x_arg)
  File "/app/venv/lib/python3.10/site-packages/flask_migrate/__init__.py", line 98, in wrapped
    f(*args, **kwargs)
  File "/app/venv/lib/python3.10/site-packages/flask_migrate/__init__.py", line 185, in upgrade
    command.upgrade(config, revision, sql=sql, tag=tag)
  File "/app/venv/lib/python3.10/site-packages/alembic/command.py", line 322, in upgrade
    script.run_env()
  File "/app/venv/lib/python3.10/site-packages/alembic/script/base.py", line 569, in run_env
    util.load_python_file(self.dir, "env.py")
  File "/app/venv/lib/python3.10/site-packages/alembic/util/pyfiles.py", line 94, in load_python_file
    module = load_module_py(module_id, path)
  File "/app/venv/lib/python3.10/site-packages/alembic/util/pyfiles.py", line 110, in load_module_py
    spec.loader.exec_module(module)  # type: ignore
  File "<frozen importlib._bootstrap_external>", line 883, in exec_module
  File "<frozen importlib._bootstrap>", line 241, in _call_with_frames_removed
  File "/app/migrations/env.py", line 99, in <module>
    run_migrations_online()
  File "/app/migrations/env.py", line 92, in run_migrations_online
    context.run_migrations()
  File "<string>", line 8, in run_migrations
  File "/app/venv/lib/python3.10/site-packages/alembic/runtime/environment.py", line 853, in run_migrations
    self.get_context().run_migrations(**kw)
  File "/app/venv/lib/python3.10/site-packages/alembic/runtime/migration.py", line 623, in run_migrations
    step.migration_fn(**kw)
  File "/app/migrations/versions/f4f0f89e0047_.py", line 18, in upgrade
    with op.batch_alter_table('fetch') as batch:
  File "/usr/lib/python3.10/contextlib.py", line 142, in __exit__
    next(self.gen)
  File "/app/venv/lib/python3.10/site-packages/alembic/operations/base.py", line 381, in batch_alter_table
    impl.flush()
  File "/app/venv/lib/python3.10/site-packages/alembic/operations/batch.py", line 111, in flush
    fn(*arg, **kw)
  File "/app/venv/lib/python3.10/site-packages/alembic/ddl/impl.py", line 322, in add_column
    self._exec(base.AddColumn(table_name, column, schema=schema))
  File "/app/venv/lib/python3.10/site-packages/alembic/ddl/impl.py", line 195, in _exec
    return conn.execute(construct, multiparams)
  File "/app/venv/lib/python3.10/site-packages/sqlalchemy/engine/base.py", line 1380, in execute
    return meth(self, multiparams, params, _EMPTY_EXECUTION_OPTS)
  File "/app/venv/lib/python3.10/site-packages/sqlalchemy/sql/ddl.py", line 80, in _execute_on_connection
    return connection._execute_ddl(
  File "/app/venv/lib/python3.10/site-packages/sqlalchemy/engine/base.py", line 1469, in _execute_ddl
    compiled = ddl.compile(
  File "/app/venv/lib/python3.10/site-packages/sqlalchemy/sql/elements.py", line 502, in compile
    return self._compiler(dialect, **kw)
  File "/app/venv/lib/python3.10/site-packages/sqlalchemy/sql/ddl.py", line 32, in _compiler
    return dialect.ddl_compiler(dialect, self, **kw)
  File "/app/venv/lib/python3.10/site-packages/sqlalchemy/sql/compiler.py", line 463, in __init__
    self.string = self.process(self.statement, **compile_kwargs)
  File "/app/venv/lib/python3.10/site-packages/sqlalchemy/sql/compiler.py", line 498, in process
    return obj._compiler_dispatch(self, **kwargs)
  File "/app/venv/lib/python3.10/site-packages/sqlalchemy/ext/compiler.py", line 548, in <lambda>
    lambda *arg, **kw: existing(*arg, **kw),
  File "/app/venv/lib/python3.10/site-packages/sqlalchemy/ext/compiler.py", line 604, in __call__
    expr = fn(element, compiler, **kw)
  File "/app/venv/lib/python3.10/site-packages/alembic/ddl/base.py", line 190, in visit_add_column
    add_column(compiler, element.column, **kw),
  File "/app/venv/lib/python3.10/site-packages/alembic/ddl/base.py", line 330, in add_column
    text = "ADD COLUMN %s" % compiler.get_column_specification(column, **kw)
  File "/app/venv/lib/python3.10/site-packages/sqlalchemy/dialects/mysql/base.py", line 1714, in get_column_specification
    self.dialect.type_compiler.process(
  File "/app/venv/lib/python3.10/site-packages/sqlalchemy/sql/compiler.py", line 532, in process
    return type_._compiler_dispatch(self, **kw)
  File "/app/venv/lib/python3.10/site-packages/sqlalchemy/sql/visitors.py", line 82, in _compiler_dispatch
    return meth(self, **kw)
  File "/app/venv/lib/python3.10/site-packages/sqlalchemy/sql/compiler.py", line 5028, in visit_type_decorator
    return self.process(type_.type_engine(self.dialect), **kw)
  File "/app/venv/lib/python3.10/site-packages/sqlalchemy/sql/compiler.py", line 532, in process
    return type_._compiler_dispatch(self, **kw)
  File "/app/venv/lib/python3.10/site-packages/sqlalchemy/sql/visitors.py", line 82, in _compiler_dispatch
    return meth(self, **kw)
  File "/app/venv/lib/python3.10/site-packages/sqlalchemy/sql/compiler.py", line 5006, in visit_string
    return self.visit_VARCHAR(type_, **kw)
  File "/app/venv/lib/python3.10/site-packages/sqlalchemy/dialects/mysql/base.py", line 2214, in visit_VARCHAR
    raise exc.CompileError(
sqlalchemy.exc.CompileError: VARCHAR requires a length on dialect mysql
[2022-12-22 09:23:12 +0000] [17] [INFO] Starting gunicorn 20.1.0
[2022-12-22 09:23:12 +0000] [17] [INFO] Listening at: http://0.0.0.0:80 (17)
[2022-12-22 09:23:12 +0000] [17] [INFO] Using worker: gthread
[2022-12-22 09:23:12 +0000] [18] [INFO] Booting worker with pid: 18
```

### Related issue(s)
none

## Prerequisites
Before we can consider review and merge, please make sure the following list is done and checked.
If an entry in not applicable, you can check it or remove it from the list.

- [x] In case of feature or enhancement: documentation updated accordingly
- [x] Unless it's docs or a minor change: add [changelog](https://mailu.io/master/contributors/workflow.html#changelog) entry file.


Co-authored-by: Johnson Thiang <jthiang@pop-os.localdomain>
Co-authored-by: Florent Daigniere <nextgens@users.noreply.github.com>
2 years ago
Florent Daigniere ca44ccbe1c
Use the size other implementations default to 2 years ago
bors[bot] fe2b0bedb7
Merge #2607
2607: Update python dependencies as suggested by dependabot r=mergify[bot] a=ghostwheel42

## What type of PR?

bug-fix

## What does this PR do?

Update dependencies to silence dependabot (vulnerabilities are probably not exploitable)
Only the certifi upgrade could be backported.


Co-authored-by: Alexander Graf <ghostwheel42@users.noreply.github.com>
2 years ago
Alexander Graf 6f71ea833b
Update python dependencies as suggested by dependabot 2 years ago
Florent Daigniere edd303f54d Modify the healtchecks to make them disapear from the logs.
This is not perfect...
- dovecot now complains about waitpid/finding a new process
- postfix is still regularly pinging rspamd / his milter and that
generates a few lines worth of logs each time.
2 years ago
bors[bot] 874e58348f
Merge #2605
2605: Reduce the SSL session caches from 50m each to 3m each r=nextgens a=nextgens

## What type of PR?

enhancement

## What does this PR do?

Reduce the SSL session caches from 50m each to 3m each. This should be good for 12k sessions (within 1day, see http://nginx.org/en/docs/http/ngx_http_ssl_module.html#ssl_session_cache and our ssl_session_timeout) for each cache and will help reduce memory usage.

### Related issue(s)

## Prerequisites
Before we can consider review and merge, please make sure the following list is done and checked.
If an entry in not applicable, you can check it or remove it from the list.

- [ ] In case of feature or enhancement: documentation updated accordingly
- [x] Unless it's docs or a minor change: add [changelog](https://mailu.io/master/contributors/workflow.html#changelog) entry file.


Co-authored-by: Florent Daigniere <nextgens@freenetproject.org>
2 years ago
Florent Daigniere cd107182c1 comment 2 years ago
Florent Daigniere 8539344331 Reduce nginx ssl_session_cache to 3m each 2 years ago
Florent Daigniere 36b3a9f4fb Will fix it in another PR 2 years ago
Florent Daigniere 83ea708490 fix healthcheck 2 years ago
Florent Daigniere 7a2d06401a Tweak postfix logging 2 years ago
Florent Daigniere 163261d951 Towncrier 2 years ago
Florent Daigniere 55c1e55529 Same for front-smtp
This should enable postfix to have visibility on TLS usage and fix the
following: #1705
2 years ago
Florent Daigniere 4ae0d7d768 Enable HAPROXY protocol in between front and imap
With this we avoid running into the limitations of
 mail_max_userip_connections (see #894 amd #1364) and the
 logfiles as well as ``doveadm who`` give an accurate picture.
2 years ago
bors[bot] c729954b4a
Merge #2601
2601: Fix creation of deep structures using import in update mode r=mergify[bot] a=ghostwheel42

## What type of PR?

bug-fix

## What does this PR do?

Fixes creation of deep structures (ie user with fetch) when using config-import in update mode.

### Related issue(s)

- closes #2493

## Prerequisites

Before we can consider review and merge, please make sure the following list is done and checked.
If an entry in not applicable, you can check it or remove it from the list.

- [ ] In case of feature or enhancement: documentation updated accordingly
- [ ] Unless it's docs or a minor change: add [changelog](https://mailu.io/master/contributors/workflow.html#changelog) entry file.


Co-authored-by: Alexander Graf <ghostwheel42@users.noreply.github.com>
2 years ago
Alexander Graf be40781394
Add default for WEB_API, re-add flask-restx to deps, remove whitespace 2 years ago
Alexander Graf 2f0f46c8fa
Add towncrier 2 years ago
Alexander Graf 84ebab2cb4
Fix creation of deep structures using import in update mode 2 years ago
Dimitri Huisman 3cb8358090
Process review comments PR#2464
- When visiting root of WEB_API, the swaggerui is shown
- simplify the condition for endpoint WEB_API
2 years ago
Dimitri Huisman 39b0d44079
Use first() instead of all() for better performance
Actually return all data for Get user
Remove non-used code
2 years ago
Dimitri Huisman f9b26bd934
Update User with newly introduced allow spoofing field 2 years ago
Dimitri Huisman 6347c18f8a
Process review comments (PR2464) 2 years ago
Dimitri Huisman 61d092922c
Process review comments (PR2464) 2 years ago
Dimitri Huisman afb224e796
Update password hash description for user API endpoint 2 years ago
Dimitri Huisman d4e5db5084
Remove unneeded comment 2 years ago
Dimitri Huisman 46d07ec236
Fix syntax styling api documentation. 2 years ago
Dimitri Huisman 67c423d61f
Add URL for accessing swaggerui to documentation 2 years ago
Dimitri Huisman 7a36f6bbb9
Use hmac.compare_digest to prevent timing attacks. 2 years ago
Dimitri Huisman 5c9cdfe1de
Introduction of the Mailu RESTful API.
Anything that can be configured in the web administration interface,
can also be configured via the Mailu RESTful API.
See the section Advanced configuration in the configuration reference
for the relevant settings in mailu.env for enabling the API.
(API, WEB_API, API_TOKEN).
2 years ago
Alexander Graf 866ad89dfc
first try at api using flask-restx & marshmallow 2 years ago
Alexander Graf c30944404d
Add "API" flag to config (default: disabled) 2 years ago
bors[bot] e9175da586
Merge #2598
2598: drop privs better r=mergify[bot] a=nextgens

## What type of PR?

bug-fix

## What does this PR do?

Without this we may see the following:
```
Initializing database
PHP Deprecated:  Return type of zipdownload_mbox_filter::filter($in, $out, &$consumed, $closing) should either be compatible with php_user_filter::filter($in, $out, &$consumed, bool $closing): int, or the #[\ReturnTypeWillChange] attribute should be used to temporarily suppress the notice in /var/www/roundcube/plugins/zipdownload/zipdownload.php on line 405
PHP Fatal error:  [snuffleupagus][0.0.0.0][readonly_exec][drop] Attempted execution of a writable file (/var/www/roundcube/plugins/mailu/mailu.php) in /var/www/roundcube/program/lib/Roundcube/rcube_plugin_api.php on line 204
Fatal error: Please check the Roundcube error log and/or server error logs for more information.
```

This has been confirmed to fix it.

### Related issue(s)

## Prerequisites
Before we can consider review and merge, please make sure the following list is done and checked.
If an entry in not applicable, you can check it or remove it from the list.

- [ ] In case of feature or enhancement: documentation updated accordingly
- [ ] Unless it's docs or a minor change: add [changelog](https://mailu.io/master/contributors/workflow.html#changelog) entry file.


Co-authored-by: Florent Daigniere <nextgens@freenetproject.org>
2 years ago
Florent Daigniere 108958cabb drop privs better 2 years ago
bors[bot] 8d2bd6d9ff
Merge #2528
2528: Implement #2510: oletools integration r=mergify[bot] a=nextgens

## What type of PR?

Feature

## What does this PR do?

OLETools now flags documents with macros and rejects suspicious ones. We also block executable file extensions by default (but don't perform inspection in archives: you can tell users to zip-up whatever needs sending).

### Related issue(s)
- closes #2510
- closes #2511

## Prerequisites
Before we can consider review and merge, please make sure the following list is done and checked.
If an entry in not applicable, you can check it or remove it from the list.

- [ ] In case of feature or enhancement: documentation updated accordingly
- [x] Unless it's docs or a minor change: add [changelog](https://mailu.io/master/contributors/workflow.html#changelog) entry file.


Co-authored-by: Florent Daigniere <nextgens@freenetproject.org>
Co-authored-by: Dimitri Huisman <diman@huisman.xyz>
2 years ago
Dimitri Huisman 6d87fa423c
Mention you must restart rspamd for the changes to take effect. 2 years ago
Dimitri Huisman 33497c8e31
Small extra clarification for new documentation 2 years ago
bors[bot] 8461a11ff4
Merge #2588
2588: IMAP folder names may contain characters outside of \w: [a-zA-Z0-9] r=mergify[bot] a=nextgens

## What type of PR?

bug-fix

## What does this PR do?

IMAP folder names may contain characters outside of \w: [a-zA-Z0-9]. Typically it may be subfolders...

I have also simplified the regexp since we strip spaces the line below.

This is used for "external accounts"/fetchmail.

### Related issue(s)

## Prerequisites
Before we can consider review and merge, please make sure the following list is done and checked.
If an entry in not applicable, you can check it or remove it from the list.

- [ ] In case of feature or enhancement: documentation updated accordingly
- [ ] Unless it's docs or a minor change: add [changelog](https://mailu.io/master/contributors/workflow.html#changelog) entry file.


Co-authored-by: Florent Daigniere <nextgens@freenetproject.org>
2 years ago
bors[bot] caa27ede4b
Merge #2593
2593: Drop postfix rsyslog localhost messages with IPv6 address r=mergify[bot] a=UbiquitousBear

## What type of PR?


Enhancement

## What does this PR do?

### Related issue(s)
#2594


## Prerequisites
Before we can consider review and merge, please make sure the following list is done and checked.
If an entry in not applicable, you can check it or remove it from the list.

- [ ] In case of feature or enhancement: documentation updated accordingly
- [x] Unless it's docs or a minor change: add [changelog](https://mailu.io/master/contributors/workflow.html#changelog) entry file.


Co-authored-by: Shamil Nunhuck <shamil@shamil.co.uk>
2 years ago
Johnson Thiang bd20ef04cc change field type to db.text 2 years ago
Shamil Nunhuck 5264a3070b Added missing towncrier newsfragments 2 years ago
Shamil Nunhuck 7225cb0d3e
Drop rsyslog localhost messages with IPv6 address 2 years ago
bors[bot] 23b09518db
Merge #2591
2591: Add button to mailu-admin in roundcube task menu r=mergify[bot] a=ghostwheel42

## What type of PR?

feature

## What does this PR do?

Adds a button to the roundcube interface. This button gets you back to the admin interface.

### Related issue(s)
- Replaces  #2367


Co-authored-by: Alexander Graf <ghostwheel42@users.noreply.github.com>
2 years ago
Alexander Graf 15ba442477
Duh #2 2 years ago
Alexander Graf 5a99ab316d
Duh 2 years ago
Alexander Graf 373488148b
Remove useless style for larry skin 2 years ago
Alexander Graf 36a567c783
Add towncrier 2 years ago
Alexander Graf c38e6aae4e
Add button to mailu-admin in roundcube task menu 2 years ago
Florent Daigniere 6370d03f80 merge snafu 2 years ago