From cc9a6b05a80de3669fddfd1fb34f95a0ee14ffaf Mon Sep 17 00:00:00 2001 From: =?UTF-8?q?Tim=20M=C3=B6hlmann?= Date: Mon, 14 Oct 2019 22:13:05 +0300 Subject: [PATCH 001/596] RFC: Mailu directory structure --- design/mailu-directory-structure.md | 199 ++++++++++++++++++++++++++++ 1 file changed, 199 insertions(+) create mode 100644 design/mailu-directory-structure.md diff --git a/design/mailu-directory-structure.md b/design/mailu-directory-structure.md new file mode 100644 index 00000000..ad75eeaa --- /dev/null +++ b/design/mailu-directory-structure.md @@ -0,0 +1,199 @@ +# RFC: Mailu directory structure +The current layout of the Mailu directory structure can be improved to allow for easier replicated deployments, like Docker Swarm and Kubernetes. Please read https://usrpro.com/publications/mailu-persistent-storage/ for the background motivation of this RFC. + +## Scope +This document only describes the re-arrangement of the `$ROOT` Mailu filesystem as-is. This means moving around of current files and directories. The linked article also proposes more advanced improvements. Those are not included in this RFC and need to be evaluated and implemented independently. However, the changes proposed in this document should make such improvements easier. + +Currently some services are wrongfully sharing mountpoints or have unused volumes declared. Those are also taken care of in this RFC. + +## Compatibility +If these changes were to be accepted, it will break compatibility with previous Mailu version `<=1.7`. As such, we will propably increment to version `2.0`. + +## Root +The root of the Mailu filesystem is located at `/mailu` by default. For simplicity we will assume this location throughout the document. Within `/mailu` we will aim to define 3 main sub-directories: + +### Config + +- Path: `/mailu/config/` + +Small config bearing files, sometimes shared between multiple services. The performance and storage needs for this filesystem are low. Availability is important for correct functioning of the mail server. No file locking issues are expected from concurrent access. A basic (and redundant) filesystem should suffice. + +#### Dovecot + +- Old path: `/mailu/overrides` (shared with postfix, nginx and rspamd) +- New Path `/mailu/config/dovecot` + +Dovecot configuration overrides. + +#### Postfix + +- Old path: `/mailu/overrides` (shared with dovecot, nginx and rspamd) +- New Path `/mailu/config/posfix` + +Postfix configuration overrides. + +#### Rspamd + +- Old path: `/mailu/overrides/rspamd` +- New path: `/mailu/config/rspamd` + +RSpamD configuration overrides. + +#### Rainloop + +- Old path: `/mailu/webmail/_data_/_default_/storage` (part of `/mailu/webmail` mountpoint, shared with Roundcube) +- New path: `/mailu/config/rainloop` + +User specific configs. The remaining files under the old `/mailu/webmail` don't need to be persistent. Except for `AddressBook.sqlite`, see `/mailu/data`. + +#### Roundcube + +- Old path: `/mailu/webmail/gpg` (part of `/mailu/webmail` mountpoint, shared with Rainloop) +- New path: `/mailu/config/roundcube/gpg` + +User configured GPG keys. + +#### Redis + +- Old path: `/mailu/redis` +- New path: `/mailu/config/redis` + +Holds `dump.rdb` for data restoration. Although technically a database, Redis works from memory. The dump file is only written to every minute and read from during start. Hence it fits better in the replicated config directory filesystem. + +#### Share + +- Path: `/mailu/config/share/` + +Shared configuration between different services + +##### DKIM + +- Old path: `/mailu/dkim/` +- New path: `/mailu/config/share/dkim` + +DKIM private keys store. Read/write access by Admin. Read only access by rSpamD. + +##### Certs + +- Old path: `/mailu/certs` +- New path: `/mailu/config/share/certs` (Proposal in anticipation of the RFC outcome at: https://github.com/Mailu/Mailu/issues/1222.) + +TLS certificates. Write access from `nginx` in case `TLS_FLAVOR=letsencrypt`. Or write access from `treafik-certdumper` or any other tool obtaining certificates. + +`letsencrypt` setting is not compatible with replicated setups. Multiple instances would disrupt the ACME challenge verification and cause race conditions on requesting certificates. In such cases certificates will need to be provided by other tools. + +If RFC issue #1222 is accepted, Dovecot will need read-only access to the certificates. + +### Data + +- Path: `/mailu/data/` + +Database files, like SQLite or PostgreSQL files. Databases don't perform well on network filesystems as they depend heavily on file locking and full controll on the database files. Making it unfit for concurrent access from multiple hosts. This directory should always live on a local filesystem. This makes it only usable in `docker-compose` deployments. Usage of this directory should be avoided in Kubernetes and Docker Swarm deployments. Some services will need to be improved to allow for this. + +#### admin data + +- Old path: `/mailu/data/` +- New path: `/mailu/data/admin/` (mount point on `admin` directory) + +Holds `main.db` SQLite database file holding domains, users, aliases etcetera. Read/write access only by admin. Can be avoided by using a remote DB server like PostgreSQL, MySQL or MariaDB. + +Also holds `instance` for unique statistics transmission. Removing of this file is proposed in RFC [issue 129](https://github.com/Mailu/Mailu/issues/1219). + +This move is needed in order to be able to mount the directory without exposing data files from other services into admin. + +#### rspamd + +- Old path: `/mailu/filter` (shared with ClamAV) +- New path: `/mailu/data/rspamd` + +Storage of Bayes and Fuzzy learning SQLite databases and caches. As future optimization we should look into moving all this into Redis. + +#### Rainloop + +- Old path: `/mailu/webmail/_data_/_default_/AddressBook.sqlite` (part of `/mailu/webmail` mountpoint, shared with Roundcube) +- New path: `/mailu/data/rainloop/AddressBook.sqlite` (mount on `rainloop` directory) + +Addressbook SQLite file. For future replicated deployments this might better be configured to use an external DB. + +For this modification, the `AddressBook.sqlite` will need to be moved to a different directory inside the container. + +#### Roundcube + +- Old path: `/mailu/webmail/roundcube.db` (part of `/mailu/webmail` mountpoint, shared with Rainloop) +- New path: `/mailu/data/roundcube/roundcube.db` (mount on `roundcube` directory) + +User settings SQLite database file for roundcube. For future replicated deployments this might better be configured to use an external DB. + +For this modification, the `rouncube.db` file will need to be moved to a different directory inside the container. + +### Mail + +- Path: `/mailu/mail` (unmodified) + +User mail, managed my Dovecot IMAP server. In replicated deployments, this filesystem needs to be shared over all IMAP server instances. It should be high performant and capable of propgating file locks. Storage size is proportional to the users and their quotas. Old versions of NFS are known to be buggy with file locking. Also Samaba or CIFS should be avoided. + +In the old situation, Maildir indexes are stored on the same volume. However, they need not to be persistent and should be located on a voletile filesystem instead. This allows better performance on network filesystems. + +### Local + +- Path: `/mailu/local` (new) + +Persistent storage not suitable for replication. In `docker-compose` deployments it lives inside `/mailu` and in replicated deployments it should live somewhere on the local host machine. + +#### Mailqueue + +- Old path: `/mailu/mailqueue` +- New path: `/mailu/local/mailqueue` + +The SMTP mailqueue should be persistant, as per SMTP spec it is not allowed to loose mail. However, persistance should be local only for performance reasons and the possibility to replicate Postfix servers. In setups like Docker Swarm and Kubernetes, admins should take care that Postfix is always restarted on same hosts in order to empty any remaining queue after a crash. + +#### ClamAV + +- Old path: `/mailu/filters` (shared with rSpamD) +- New path: `/mailu/local/clamav` + +Virus definitions do not need to be replicated, as they can be easily pulled in when ClamAV instances migrate to other nodes. Persistance does allow for some bandwith and time saving if ClamAV would be restarted on a previously used node (in case of updates or similar cases). Local only storage also prevents `freshclam` race conditions. + +## Conclusion + +The final layout of the Mailu filesystem will look like: + +```` +/mailu +├── config +│   ├── dovecot +│   ├── postfix +│   ├── rainloop +│   ├── redis +│   ├── roundcube +│   │   └── gpg +│   ├── rspamd +│   └── share +│   ├── certs +│   └── dkim +├── data +│   ├── admin +│   ├── rainloop +│   ├── roundcube +│   └── rspamd +├── local +│   ├── clamav +│   └── mailqueue +└── mail +```` + +Where in replicated environments: + +- `/mailu/config/`: should be a small, low performant and shared filesystem. +- `/mailu/data`: should be avoided. More work will need to be done to configure external DB servers for relevant services. Ideally, this directory should only exist on docker-compose deployments. +- `/mailu/local/`: Should exist only on local file systems of worker nodes. +- `/mailu/mail`: A distributed filesystem with sufficient performance and storage requirements to hold and process all user mailboxes. Ideally only Maildir without indexes. + +### Implementing + +The works to implement this changes should happen outside the `master` branch. Inclusion into `master` can only be accepted if: + +1. `docker-compose.yml` from setup reflects this changes correctly. +2. Kubernetes documentation is updated. +3. Legacy `docker-compose.yml` is either updated or deleted. +4. A clear data migration guide is written. \ No newline at end of file From 5dfccdafe9e4bc62dbcbab166a8f62b256af8af2 Mon Sep 17 00:00:00 2001 From: Alexander Graf Date: Wed, 26 Aug 2020 11:11:23 +0200 Subject: [PATCH 002/596] fixed some minor typos, removed unused variable --- core/admin/mailu/models.py | 10 +++++----- 1 file changed, 5 insertions(+), 5 deletions(-) diff --git a/core/admin/mailu/models.py b/core/admin/mailu/models.py index bbc00f2d..6c9a2ab1 100644 --- a/core/admin/mailu/models.py +++ b/core/admin/mailu/models.py @@ -65,10 +65,10 @@ class CommaSeparatedList(db.TypeDecorator): def process_bind_param(self, value, dialect): if type(value) is not list: - raise TypeError("Shoud be a list") + raise TypeError("Should be a list") for item in value: if "," in item: - raise ValueError("No item should contain a comma") + raise ValueError("Item must not contain a comma") return ",".join(value) def process_result_value(self, value, dialect): @@ -76,7 +76,7 @@ class CommaSeparatedList(db.TypeDecorator): class JSONEncoded(db.TypeDecorator): - """Represents an immutable structure as a json-encoded string. + """ Represents an immutable structure as a json-encoded string. """ impl = db.String @@ -172,7 +172,7 @@ class Domain(Base): str(rset).split()[-1][:-1] in hostnames for rset in dns.resolver.query(self.name, 'MX') ) - except Exception as e: + except Exception: return False def __str__(self): @@ -503,7 +503,7 @@ class Token(Base): class Fetch(Base): - """ A fetched account is a repote POP/IMAP account fetched into a local + """ A fetched account is a remote POP/IMAP account fetched into a local account. """ __tablename__ = "fetch" From c26ddd3c68b41b35395850a9157d61b83655ac0a Mon Sep 17 00:00:00 2001 From: Alexander Graf Date: Wed, 26 Aug 2020 11:19:01 +0200 Subject: [PATCH 003/596] fixed user's destination property self.forward_destination is a list (and not string) --- core/admin/mailu/models.py | 6 +++--- 1 file changed, 3 insertions(+), 3 deletions(-) diff --git a/core/admin/mailu/models.py b/core/admin/mailu/models.py index 6c9a2ab1..0a447758 100644 --- a/core/admin/mailu/models.py +++ b/core/admin/mailu/models.py @@ -346,10 +346,10 @@ class User(Base, Email): @property def destination(self): if self.forward_enabled: - result = self.forward_destination + result = list(self.forward_destination) if self.forward_keep: - result += ',' + self.email - return result + result.append(self.email) + return ','.join(result) else: return self.email From 5c0efe82cf96f73a8c15e130051f0ccbe91a4c5a Mon Sep 17 00:00:00 2001 From: Alexander Graf Date: Wed, 26 Aug 2020 11:27:38 +0200 Subject: [PATCH 004/596] implemented config_update and config_dump enhanced data model with to_dict and from_dict methods added config_dump function to manage command config_update now uses new data model methods --- core/admin/mailu/manage.py | 263 ++++++++++++++++---------------- core/admin/mailu/models.py | 301 ++++++++++++++++++++++++++++++++++++- 2 files changed, 431 insertions(+), 133 deletions(-) diff --git a/core/admin/mailu/manage.py b/core/admin/mailu/manage.py index 62f214d3..bf0148df 100644 --- a/core/admin/mailu/manage.py +++ b/core/admin/mailu/manage.py @@ -8,6 +8,8 @@ import os import socket import uuid import click +import yaml +import sys db = models.db @@ -169,146 +171,147 @@ def user_import(localpart, domain_name, password_hash, hash_scheme = None): db.session.commit() +yaml_sections = [ + ('domains', models.Domain), + ('relays', models.Relay), + ('users', models.User), + ('aliases', models.Alias), +# ('config', models.Config), +] + @mailu.command() -@click.option('-v', '--verbose') -@click.option('-d', '--delete-objects') +@click.option('-v', '--verbose', is_flag=True) +@click.option('-d', '--delete-objects', is_flag=True) +@click.option('-n', '--dry-run', is_flag=True) @flask_cli.with_appcontext -def config_update(verbose=False, delete_objects=False): +def config_update(verbose=False, delete_objects=False, dry_run=False, file=None): """sync configuration with data from YAML-formatted stdin""" - import yaml - import sys - new_config = yaml.safe_load(sys.stdin) - # print new_config - domains = new_config.get('domains', []) - tracked_domains = set() - for domain_config in domains: - if verbose: - print(str(domain_config)) - domain_name = domain_config['name'] - max_users = domain_config.get('max_users', -1) - max_aliases = domain_config.get('max_aliases', -1) - max_quota_bytes = domain_config.get('max_quota_bytes', 0) - tracked_domains.add(domain_name) - domain = models.Domain.query.get(domain_name) - if not domain: - domain = models.Domain(name=domain_name, - max_users=max_users, - max_aliases=max_aliases, - max_quota_bytes=max_quota_bytes) - db.session.add(domain) - print("Added " + str(domain_config)) - else: - domain.max_users = max_users - domain.max_aliases = max_aliases - domain.max_quota_bytes = max_quota_bytes - db.session.add(domain) - print("Updated " + str(domain_config)) - users = new_config.get('users', []) - tracked_users = set() - user_optional_params = ('comment', 'quota_bytes', 'global_admin', - 'enable_imap', 'enable_pop', 'forward_enabled', - 'forward_destination', 'reply_enabled', - 'reply_subject', 'reply_body', 'displayed_name', - 'spam_enabled', 'email', 'spam_threshold') - for user_config in users: - if verbose: - print(str(user_config)) - localpart = user_config['localpart'] - domain_name = user_config['domain'] - password_hash = user_config.get('password_hash', None) - hash_scheme = user_config.get('hash_scheme', None) - domain = models.Domain.query.get(domain_name) - email = '{0}@{1}'.format(localpart, domain_name) - optional_params = {} - for k in user_optional_params: - if k in user_config: - optional_params[k] = user_config[k] - if not domain: - domain = models.Domain(name=domain_name) - db.session.add(domain) - user = models.User.query.get(email) - tracked_users.add(email) - tracked_domains.add(domain_name) - if not user: - user = models.User( - localpart=localpart, - domain=domain, - **optional_params - ) - else: - for k in optional_params: - setattr(user, k, optional_params[k]) - user.set_password(password_hash, hash_scheme=hash_scheme, raw=True) - db.session.add(user) + out = (lambda *args: print('(DRY RUN)', *args)) if dry_run else print - aliases = new_config.get('aliases', []) - tracked_aliases = set() - for alias_config in aliases: - if verbose: - print(str(alias_config)) - localpart = alias_config['localpart'] - domain_name = alias_config['domain'] - if type(alias_config['destination']) is str: - destination = alias_config['destination'].split(',') - else: - destination = alias_config['destination'] - wildcard = alias_config.get('wildcard', False) - domain = models.Domain.query.get(domain_name) - email = '{0}@{1}'.format(localpart, domain_name) - if not domain: - domain = models.Domain(name=domain_name) - db.session.add(domain) - alias = models.Alias.query.get(email) - tracked_aliases.add(email) - tracked_domains.add(domain_name) - if not alias: - alias = models.Alias( - localpart=localpart, - domain=domain, - wildcard=wildcard, - destination=destination, - email=email - ) - else: - alias.destination = destination - alias.wildcard = wildcard - db.session.add(alias) + try: + new_config = yaml.safe_load(sys.stdin) + except (yaml.scanner.ScannerError, yaml.parser.ParserError) as reason: + out(f'[ERROR] Invalid yaml: {reason}') + sys.exit(1) + else: + if type(new_config) is str: + out(f'[ERROR] Invalid yaml: {new_config!r}') + sys.exit(1) + elif new_config is None or not len(new_config): + out('[ERROR] Empty yaml: Please pipe yaml into stdin') + sys.exit(1) - db.session.commit() + error = False + tracked = {} + for section, model in yaml_sections: - managers = new_config.get('managers', []) - # tracked_managers=set() - for manager_config in managers: - if verbose: - print(str(manager_config)) - domain_name = manager_config['domain'] - user_name = manager_config['user'] - domain = models.Domain.query.get(domain_name) - manageruser = models.User.query.get(user_name + '@' + domain_name) - if manageruser not in domain.managers: - domain.managers.append(manageruser) - db.session.add(domain) + items = new_config.get(section) + if items is None: + if delete_objects: + out(f'[ERROR] Invalid yaml: Section "{section}" is missing') + error = True + break + else: + continue - db.session.commit() + del new_config[section] + if type(items) is not list: + out(f'[ERROR] Section "{section}" must be a list, not {items.__class__.__name__}') + error = True + break + elif not items: + continue + + # create items + for data in items: + + if verbose: + out(f'Handling {model.__table__} data: {data!r}') + + try: + changed = model.from_dict(data, delete_objects) + except Exception as reason: + out(f'[ERROR] {reason.args[0]} in data: {data}') + error = True + break + + for item, created in changed: + + if created is True: + # flush newly created item + db.session.add(item) + db.session.flush() + if verbose: + out(f'Added {item!r}: {item.to_dict()}') + else: + out(f'Added {item!r}') + + elif len(created): + # modified instance + if verbose: + for key, old, new in created: + out(f'Updated {key!r} of {item!r}: {old!r} -> {new!r}') + else: + out(f'Updated {item!r}: {", ".join(sorted([kon[0] for kon in created]))}') + + # track primary key of all items + tracked.setdefault(item.__class__, set()).update(set([item._dict_pval()])) + + if error: + break + + # on error: stop early + if error: + out('An error occured. Not committing changes.') + db.session.rollback() + sys.exit(1) + + # are there sections left in new_config? + if new_config: + out(f'[ERROR] Unknown section(s) in yaml: {", ".join(sorted(new_config.keys()))}') + error = True + + # test for conflicting domains + domains = set() + for model, items in tracked.items(): + if model in (models.Domain, models.Alternative, models.Relay): + if domains & items: + for domain in domains & items: + out(f'[ERROR] Duplicate domain name used: {domain}') + error = True + domains.update(items) + + # delete items not tracked if delete_objects: - for user in db.session.query(models.User).all(): - if not (user.email in tracked_users): - if verbose: - print("Deleting user: " + str(user.email)) - db.session.delete(user) - for alias in db.session.query(models.Alias).all(): - if not (alias.email in tracked_aliases): - if verbose: - print("Deleting alias: " + str(alias.email)) - db.session.delete(alias) - for domain in db.session.query(models.Domain).all(): - if not (domain.name in tracked_domains): - if verbose: - print("Deleting domain: " + str(domain.name)) - db.session.delete(domain) - db.session.commit() + for model, items in tracked.items(): + for item in model.query.all(): + if not item._dict_pval() in items: + out(f'Deleted {item!r} {item}') + db.session.delete(item) + + # don't commit when running dry + if dry_run: + db.session.rollback() + else: + db.session.commit() + + +@mailu.command() +@click.option('-v', '--verbose', is_flag=True) +@click.option('-s', '--secrets', is_flag=True) +@flask_cli.with_appcontext +def config_dump(verbose=False, secrets=False): + """dump configuration as YAML-formatted data to stdout""" + + config = {} + for section, model in yaml_sections: + dump = [item.to_dict(verbose, secrets) for item in model.query.all()] + if len(dump): + config[section] = dump + + yaml.dump(config, sys.stdout, default_flow_style=False, allow_unicode=True) @mailu.command() diff --git a/core/admin/mailu/models.py b/core/admin/mailu/models.py index 0a447758..fde4d6f1 100644 --- a/core/admin/mailu/models.py +++ b/core/admin/mailu/models.py @@ -5,6 +5,7 @@ from passlib import context, hash from datetime import datetime, date from email.mime import text from flask import current_app as app +from textwrap import wrap import flask_sqlalchemy import sqlalchemy @@ -15,6 +16,8 @@ import glob import smtplib import idna import dns +import json +import itertools db = flask_sqlalchemy.SQLAlchemy() @@ -32,6 +35,7 @@ class IdnaDomain(db.TypeDecorator): def process_result_value(self, value, dialect): return idna.decode(value) + python_type = str class IdnaEmail(db.TypeDecorator): """ Stores a Unicode string in it's IDNA representation (ASCII only) @@ -56,6 +60,7 @@ class IdnaEmail(db.TypeDecorator): idna.decode(domain_name), ) + python_type = str class CommaSeparatedList(db.TypeDecorator): """ Stores a list as a comma-separated string, compatible with Postfix. @@ -74,6 +79,7 @@ class CommaSeparatedList(db.TypeDecorator): def process_result_value(self, value, dialect): return list(filter(bool, value.split(","))) if value else [] + python_type = list class JSONEncoded(db.TypeDecorator): """ Represents an immutable structure as a json-encoded string. @@ -87,6 +93,7 @@ class JSONEncoded(db.TypeDecorator): def process_result_value(self, value, dialect): return json.loads(value) if value else None + python_type = str class Base(db.Model): """ Base class for all models @@ -105,6 +112,219 @@ class Base(db.Model): updated_at = db.Column(db.Date, nullable=True, onupdate=date.today) comment = db.Column(db.String(255), nullable=True) + @classmethod + def _dict_pkey(model): + return model.__mapper__.primary_key[0].name + + def _dict_pval(self): + return getattr(self, self._dict_pkey()) + + def to_dict(self, full=False, include_secrets=False, recursed=False, hide=None): + """ Return a dictionary representation of this model. + """ + + if recursed and not getattr(self, '_dict_recurse', False): + return str(self) + + hide = set(hide or []) | {'created_at', 'updated_at'} + if hasattr(self, '_dict_hide'): + hide |= self._dict_hide + + secret = set() + if not include_secrets and hasattr(self, '_dict_secret'): + secret |= self._dict_secret + + convert = getattr(self, '_dict_output', {}) + + res = {} + + for key in itertools.chain(self.__table__.columns.keys(), getattr(self, '_dict_show', [])): + if key in hide: + continue + if key in self.__table__.columns: + default = self.__table__.columns[key].default + if isinstance(default, sqlalchemy.sql.schema.ColumnDefault): + default = default.arg + else: + default = None + value = getattr(self, key) + if full or ((default or value) and value != default): + if key in secret: + value = '' + elif value is not None and key in convert: + value = convert[key](value) + res[key] = value + + for key in self.__mapper__.relationships.keys(): + if key in hide: + continue + if self.__mapper__.relationships[key].uselist: + items = getattr(self, key) + if self.__mapper__.relationships[key].query_class is not None: + if hasattr(items, 'all'): + items = items.all() + if full or len(items): + if key in secret: + res[key] = '' + else: + res[key] = [item.to_dict(full, include_secrets, True) for item in items] + else: + value = getattr(self, key) + if full or value is not None: + if key in secret: + res[key] = '' + else: + res[key] = value.to_dict(full, include_secrets, True) + + return res + + @classmethod + def from_dict(model, data, delete=False): + + changed = [] + + pkey = model._dict_pkey() + + # handle "primary key" only + if type(data) is not dict: + data = {pkey: data} + + # modify input data + if hasattr(model, '_dict_input'): + try: + model._dict_input(data) + except Exception as reason: + raise ValueError(f'{reason}', model, None, data) + + # check for primary key (if not recursed) + if not getattr(model, '_dict_recurse', False): + if not pkey in data: + raise KeyError(f'primary key {model.__table__}.{pkey} is missing', model, pkey, data) + + # check data keys and values + for key, value in data.items(): + + # check key + if not hasattr(model, key): + raise KeyError(f'unknown key {model.__table__}.{key}', model, key, data) + + # check value type + col = model.__mapper__.columns.get(key) + if col is not None: + if not type(value) is col.type.python_type: + raise TypeError(f'{model.__table__}.{key} {value!r} has invalid type {type(value).__name__!r}', model, key, data) + else: + rel = model.__mapper__.relationships.get(key) + if rel is None: + itype = getattr(model, '_dict_types', {}).get(key) + if itype is not None: + if type(value) is not itype: + raise TypeError(f'{model.__table__}.{key} {value!r} has invalid type {type(value).__name__!r}', model, key, data) + else: + raise NotImplementedError(f'type not defined for {model.__table__}.{key}') + + # handle relationships + if key in model.__mapper__.relationships: + rel_model = model.__mapper__.relationships[key].argument + if not isinstance(rel_model, sqlalchemy.orm.Mapper): + add = rel_model.from_dict(value, delete) + assert len(add) == 1 + item, updated = add[0] + changed.append((item, updated)) + data[key] = item + + # create or update item? + item = model.query.get(data[pkey]) if pkey in data else None + if item is None: + # create item + + # check for mandatory keys + missing = getattr(model, '_dict_mandatory', set()) - set(data.keys()) + if missing: + raise ValueError(f'mandatory key(s) {", ".join(sorted(missing))} for {model.__table__} missing', model, missing, data) + + changed.append((model(**data), True)) + + else: + # update item + + updated = [] + for key, value in data.items(): + + # skip primary key + if key == pkey: + continue + + if key in model.__mapper__.relationships: + # update relationship + rel_model = model.__mapper__.relationships[key].argument + if isinstance(rel_model, sqlalchemy.orm.Mapper): + rel_model = rel_model.class_ + # add (and create) referenced items + cur = getattr(item, key) + old = sorted(cur, key=lambda i:id(i)) + new = [] + for rel_data in value: + # get or create related item + add = rel_model.from_dict(rel_data, delete) + assert len(add) == 1 + rel_item, rel_updated = add[0] + changed.append((rel_item, rel_updated)) + if rel_item not in cur: + cur.append(rel_item) + new.append(rel_item) + + # delete referenced items missing in yaml + rel_pkey = rel_model._dict_pkey() + new_data = list([i.to_dict(True, True, True, [rel_pkey]) for i in new]) + for rel_item in old: + if rel_item not in new: + # check if item with same data exists to stabilze import without primary key + rel_data = rel_item.to_dict(True, True, True, [rel_pkey]) + try: + same_idx = new_data.index(rel_data) + except ValueError: + same = None + else: + same = new[same_idx] + + if same is None: + # delete items missing in new + if delete: + cur.remove(rel_item) + else: + new.append(rel_item) + else: + # swap found item with same data with newly created item + new.append(rel_item) + new_data.append(rel_data) + new.remove(same) + del new_data[same_idx] + for i, (ch_item, ch_update) in enumerate(changed): + if ch_item is same: + changed[i] = (rel_item, []) + db.session.flush() + db.session.delete(ch_item) + break + + # remember changes + new = sorted(new, key=lambda i:id(i)) + if new != old: + updated.append((key, old, new)) + + else: + # update key + old = getattr(item, key) + if type(old) is list and not delete: + value = old + value + if value != old: + updated.append((key, old, value)) + setattr(item, key, value) + + changed.append((item, updated)) + + return changed + # Many-to-many association table for domain managers managers = db.Table('manager', Base.metadata, @@ -126,6 +346,27 @@ class Domain(Base): """ __tablename__ = "domain" + _dict_hide = {'users', 'managers', 'aliases'} + _dict_show = {'dkim_key'} + _dict_secret = {'dkim_key'} + _dict_types = {'dkim_key': bytes} + _dict_output = {'dkim_key': lambda v: v.decode('utf-8').strip().split('\n')[1:-1]} + @staticmethod + def _dict_input(data): + key = data.get('dkim_key') + if key is not None: + key = data['dkim_key'] + if type(key) is list: + key = ''.join(key) + if type(key) is str: + key = ''.join(key.strip().split()) + if key.startswith('-----BEGIN PRIVATE KEY-----'): + key = key[25:] + if key.endswith('-----END PRIVATE KEY-----'): + key = key[:-23] + key = '\n'.join(wrap(key, 64)) + data['dkim_key'] = f'-----BEGIN PRIVATE KEY-----\n{key}\n-----END PRIVATE KEY-----\n'.encode('ascii') + name = db.Column(IdnaDomain, primary_key=True, nullable=False) managers = db.relationship('User', secondary=managers, backref=db.backref('manager_of'), lazy='dynamic') @@ -208,6 +449,8 @@ class Relay(Base): __tablename__ = "relay" + _dict_mandatory = {'smtp'} + name = db.Column(IdnaDomain, primary_key=True, nullable=False) smtp = db.Column(db.String(80), nullable=True) @@ -221,6 +464,16 @@ class Email(object): localpart = db.Column(db.String(80), nullable=False) + @staticmethod + def _dict_input(data): + if 'email' in data: + if 'localpart' in data or 'domain' in data: + raise ValueError('ambigous key email and localpart/domain') + elif type(data['email']) is str: + data['localpart'], data['domain'] = data['email'].rsplit('@', 1) + else: + data['email'] = f"{data['localpart']}@{data['domain']}" + @declarative.declared_attr def domain_name(cls): return db.Column(IdnaDomain, db.ForeignKey(Domain.name), @@ -306,6 +559,28 @@ class User(Base, Email): """ __tablename__ = "user" + _dict_hide = {'domain_name', 'domain', 'localpart', 'quota_bytes_used'} + _dict_mandatory = {'localpart', 'domain', 'password'} + @classmethod + def _dict_input(cls, data): + Email._dict_input(data) + # handle password + if 'password' in data: + if 'password_hash' in data or 'hash_scheme' in data: + raise ValueError('ambigous key password and password_hash/hash_scheme') + # check (hashed) password + password = data['password'] + if password.startswith('{') and '}' in password: + scheme = password[1:password.index('}')] + if scheme not in cls.scheme_dict: + raise ValueError(f'invalid password scheme {scheme!r}') + else: + raise ValueError(f'invalid hashed password {password!r}') + elif 'password_hash' in data and 'hash_scheme' in data: + if data['hash_scheme'] not in cls.scheme_dict: + raise ValueError(f'invalid password scheme {scheme!r}') + data['password'] = '{'+data['hash_scheme']+'}'+ data['password_hash'] + domain = db.relationship(Domain, backref=db.backref('users', cascade='all, delete-orphan')) password = db.Column(db.String(255), nullable=False) @@ -431,6 +706,14 @@ class Alias(Base, Email): """ __tablename__ = "alias" + _dict_hide = {'domain_name', 'domain', 'localpart'} + @staticmethod + def _dict_input(data): + # handle comma delimited string for backwards compability + dst = data.get('destination') + if type(dst) is str: + data['destination'] = list([adr.strip() for adr in dst.split(',')]) + domain = db.relationship(Domain, backref=db.backref('aliases', cascade='all, delete-orphan')) wildcard = db.Column(db.Boolean(), nullable=False, default=False) @@ -484,6 +767,10 @@ class Token(Base): """ __tablename__ = "token" + _dict_recurse = True + _dict_hide = {'user', 'user_email'} + _dict_mandatory = {'password'} + id = db.Column(db.Integer(), primary_key=True) user_email = db.Column(db.String(255), db.ForeignKey(User.email), nullable=False) @@ -499,7 +786,7 @@ class Token(Base): self.password = hash.sha256_crypt.using(rounds=1000).hash(password) def __str__(self): - return self.comment + return self.comment or self.ip class Fetch(Base): @@ -508,6 +795,11 @@ class Fetch(Base): """ __tablename__ = "fetch" + _dict_recurse = True + _dict_hide = {'user_email', 'user', 'last_check', 'error'} + _dict_mandatory = {'protocol', 'host', 'port', 'username', 'password'} + _dict_secret = {'password'} + id = db.Column(db.Integer(), primary_key=True) user_email = db.Column(db.String(255), db.ForeignKey(User.email), nullable=False) @@ -516,9 +808,12 @@ class Fetch(Base): protocol = db.Column(db.Enum('imap', 'pop3'), nullable=False) host = db.Column(db.String(255), nullable=False) port = db.Column(db.Integer(), nullable=False) - tls = db.Column(db.Boolean(), nullable=False) + tls = db.Column(db.Boolean(), nullable=False, default=False) username = db.Column(db.String(255), nullable=False) password = db.Column(db.String(255), nullable=False) - keep = db.Column(db.Boolean(), nullable=False) + keep = db.Column(db.Boolean(), nullable=False, default=False) last_check = db.Column(db.DateTime, nullable=True) error = db.Column(db.String(1023), nullable=True) + + def __str__(self): + return f'{self.protocol}{"s" if self.tls else ""}://{self.username}@{self.host}:{self.port}' From 190e7a709b6e3bb134ba8086e54796f14d42a05c Mon Sep 17 00:00:00 2001 From: Alexander Graf Date: Wed, 26 Aug 2020 23:14:27 +0200 Subject: [PATCH 005/596] renamed config-dump option --verbose to --full --- core/admin/mailu/manage.py | 6 +++--- 1 file changed, 3 insertions(+), 3 deletions(-) diff --git a/core/admin/mailu/manage.py b/core/admin/mailu/manage.py index bf0148df..f74d24b5 100644 --- a/core/admin/mailu/manage.py +++ b/core/admin/mailu/manage.py @@ -299,15 +299,15 @@ def config_update(verbose=False, delete_objects=False, dry_run=False, file=None) @mailu.command() -@click.option('-v', '--verbose', is_flag=True) +@click.option('-f', '--full', is_flag=True) @click.option('-s', '--secrets', is_flag=True) @flask_cli.with_appcontext -def config_dump(verbose=False, secrets=False): +def config_dump(full=False, secrets=False): """dump configuration as YAML-formatted data to stdout""" config = {} for section, model in yaml_sections: - dump = [item.to_dict(verbose, secrets) for item in model.query.all()] + dump = [item.to_dict(full, secrets) for item in model.query.all()] if len(dump): config[section] = dump From 69ccf791d27dabfe1b8943d837923d5b2ba78d72 Mon Sep 17 00:00:00 2001 From: Alexander Graf Date: Wed, 26 Aug 2020 23:16:37 +0200 Subject: [PATCH 006/596] fixed data import via from_dict - stabilized CommaSeparatedList by sorting values - CommaSeparatedList can now handle list and set input - from_dict now handles mapped keys - from_dict now handles null values - class Domain: handle dkim-key None correctly - class User: delete obsolete keys after converting - class Alias: now uses Email._dict_input --- core/admin/mailu/models.py | 205 ++++++++++++++++++++----------------- 1 file changed, 113 insertions(+), 92 deletions(-) diff --git a/core/admin/mailu/models.py b/core/admin/mailu/models.py index fde4d6f1..cbafc6a4 100644 --- a/core/admin/mailu/models.py +++ b/core/admin/mailu/models.py @@ -69,12 +69,12 @@ class CommaSeparatedList(db.TypeDecorator): impl = db.String def process_bind_param(self, value, dialect): - if type(value) is not list: + if not isinstance(value, (list, set)): raise TypeError("Should be a list") for item in value: if "," in item: raise ValueError("Item must not contain a comma") - return ",".join(value) + return ",".join(sorted(value)) def process_result_value(self, value, dialect): return list(filter(bool, value.split(","))) if value else [] @@ -205,13 +205,13 @@ class Base(db.Model): for key, value in data.items(): # check key - if not hasattr(model, key): + if not hasattr(model, key) and not key in model.__mapper__.relationships: raise KeyError(f'unknown key {model.__table__}.{key}', model, key, data) # check value type col = model.__mapper__.columns.get(key) if col is not None: - if not type(value) is col.type.python_type: + if not ((value is None and col.nullable) or (type(value) is col.type.python_type)): raise TypeError(f'{model.__table__}.{key} {value!r} has invalid type {type(value).__name__!r}', model, key, data) else: rel = model.__mapper__.relationships.get(key) @@ -229,99 +229,115 @@ class Base(db.Model): if not isinstance(rel_model, sqlalchemy.orm.Mapper): add = rel_model.from_dict(value, delete) assert len(add) == 1 - item, updated = add[0] - changed.append((item, updated)) - data[key] = item + rel_item, updated = add[0] + changed.append((rel_item, updated)) + data[key] = rel_item - # create or update item? + # create item if necessary + created = False item = model.query.get(data[pkey]) if pkey in data else None if item is None: - # create item # check for mandatory keys missing = getattr(model, '_dict_mandatory', set()) - set(data.keys()) if missing: raise ValueError(f'mandatory key(s) {", ".join(sorted(missing))} for {model.__table__} missing', model, missing, data) - changed.append((model(**data), True)) - - else: - # update item - - updated = [] - for key, value in data.items(): - - # skip primary key - if key == pkey: - continue - + # remove mapped relationships from data + mapped = {} + for key in list(data.keys()): if key in model.__mapper__.relationships: - # update relationship - rel_model = model.__mapper__.relationships[key].argument - if isinstance(rel_model, sqlalchemy.orm.Mapper): - rel_model = rel_model.class_ - # add (and create) referenced items - cur = getattr(item, key) - old = sorted(cur, key=lambda i:id(i)) - new = [] - for rel_data in value: - # get or create related item - add = rel_model.from_dict(rel_data, delete) - assert len(add) == 1 - rel_item, rel_updated = add[0] - changed.append((rel_item, rel_updated)) - if rel_item not in cur: - cur.append(rel_item) - new.append(rel_item) + if isinstance(model.__mapper__.relationships[key].argument, sqlalchemy.orm.Mapper): + mapped[key] = data[key] + del data[key] - # delete referenced items missing in yaml - rel_pkey = rel_model._dict_pkey() - new_data = list([i.to_dict(True, True, True, [rel_pkey]) for i in new]) - for rel_item in old: - if rel_item not in new: - # check if item with same data exists to stabilze import without primary key - rel_data = rel_item.to_dict(True, True, True, [rel_pkey]) - try: - same_idx = new_data.index(rel_data) - except ValueError: - same = None - else: - same = new[same_idx] + # create new item + item = model(**data) + created = True - if same is None: - # delete items missing in new - if delete: - cur.remove(rel_item) - else: - new.append(rel_item) + # and update mapped relationships (below) + data = mapped + + # update item + updated = [] + for key, value in data.items(): + + # skip primary key + if key == pkey: + continue + + if key in model.__mapper__.relationships: + # update relationship + rel_model = model.__mapper__.relationships[key].argument + if isinstance(rel_model, sqlalchemy.orm.Mapper): + rel_model = rel_model.class_ + # add (and create) referenced items + cur = getattr(item, key) + old = sorted(cur, key=lambda i:id(i)) + new = [] + for rel_data in value: + # get or create related item + add = rel_model.from_dict(rel_data, delete) + assert len(add) == 1 + rel_item, rel_updated = add[0] + changed.append((rel_item, rel_updated)) + if rel_item not in cur: + cur.append(rel_item) + new.append(rel_item) + + # delete referenced items missing in yaml + rel_pkey = rel_model._dict_pkey() + new_data = list([i.to_dict(True, True, True, [rel_pkey]) for i in new]) + for rel_item in old: + if rel_item not in new: + # check if item with same data exists to stabilze import without primary key + rel_data = rel_item.to_dict(True, True, True, [rel_pkey]) + try: + same_idx = new_data.index(rel_data) + except ValueError: + same = None + else: + same = new[same_idx] + + if same is None: + # delete items missing in new + if delete: + cur.remove(rel_item) else: - # swap found item with same data with newly created item new.append(rel_item) - new_data.append(rel_data) - new.remove(same) - del new_data[same_idx] - for i, (ch_item, ch_update) in enumerate(changed): - if ch_item is same: - changed[i] = (rel_item, []) - db.session.flush() - db.session.delete(ch_item) - break + else: + # swap found item with same data with newly created item + new.append(rel_item) + new_data.append(rel_data) + new.remove(same) + del new_data[same_idx] + for i, (ch_item, ch_update) in enumerate(changed): + if ch_item is same: + changed[i] = (rel_item, []) + db.session.flush() + db.session.delete(ch_item) + break - # remember changes - new = sorted(new, key=lambda i:id(i)) - if new != old: - updated.append((key, old, new)) + # remember changes + new = sorted(new, key=lambda i:id(i)) + if new != old: + updated.append((key, old, new)) - else: - # update key - old = getattr(item, key) - if type(old) is list and not delete: - value = old + value - if value != old: - updated.append((key, old, value)) - setattr(item, key, value) + else: + # update key + old = getattr(item, key) + if type(old) is list: + # deduplicate list value + assert type(value) is list + value = set(value) + old = set(old) + if not delete: + value = old | value + if value != old: + updated.append((key, old, value)) + setattr(item, key, value) - changed.append((item, updated)) + changed.append((item, created if created else updated)) return changed @@ -353,19 +369,21 @@ class Domain(Base): _dict_output = {'dkim_key': lambda v: v.decode('utf-8').strip().split('\n')[1:-1]} @staticmethod def _dict_input(data): - key = data.get('dkim_key') - if key is not None: + if 'dkim_key' in data: key = data['dkim_key'] - if type(key) is list: - key = ''.join(key) - if type(key) is str: - key = ''.join(key.strip().split()) - if key.startswith('-----BEGIN PRIVATE KEY-----'): - key = key[25:] - if key.endswith('-----END PRIVATE KEY-----'): - key = key[:-23] - key = '\n'.join(wrap(key, 64)) - data['dkim_key'] = f'-----BEGIN PRIVATE KEY-----\n{key}\n-----END PRIVATE KEY-----\n'.encode('ascii') + if key is None: + del data['dkim_key'] + else: + if type(key) is list: + key = ''.join(key) + if type(key) is str: + key = ''.join(key.strip().split()) + if key.startswith('-----BEGIN PRIVATE KEY-----'): + key = key[25:] + if key.endswith('-----END PRIVATE KEY-----'): + key = key[:-23] + key = '\n'.join(wrap(key, 64)) + data['dkim_key'] = f'-----BEGIN PRIVATE KEY-----\n{key}\n-----END PRIVATE KEY-----\n'.encode('ascii') name = db.Column(IdnaDomain, primary_key=True, nullable=False) managers = db.relationship('User', secondary=managers, @@ -580,6 +598,8 @@ class User(Base, Email): if data['hash_scheme'] not in cls.scheme_dict: raise ValueError(f'invalid password scheme {scheme!r}') data['password'] = '{'+data['hash_scheme']+'}'+ data['password_hash'] + del data['hash_scheme'] + del data['password_hash'] domain = db.relationship(Domain, backref=db.backref('users', cascade='all, delete-orphan')) @@ -709,6 +729,7 @@ class Alias(Base, Email): _dict_hide = {'domain_name', 'domain', 'localpart'} @staticmethod def _dict_input(data): + Email._dict_input(data) # handle comma delimited string for backwards compability dst = data.get('destination') if type(dst) is str: From 3a5a15a5e6326cee38b018fcad6d05ab92c0485a Mon Sep 17 00:00:00 2001 From: Alexander Graf Date: Wed, 26 Aug 2020 23:23:03 +0200 Subject: [PATCH 007/596] updated documentation and changelog added some documentation for cli commands config-dump and config-update --- CHANGELOG.md | 2 + docs/cli.rst | 127 ++++++++++++++++++++++++++++++++++++++------------- 2 files changed, 98 insertions(+), 31 deletions(-) diff --git a/CHANGELOG.md b/CHANGELOG.md index 7ccde4bc..1c20d9b2 100644 --- a/CHANGELOG.md +++ b/CHANGELOG.md @@ -28,6 +28,7 @@ deprecated before 1.8.0, you can switch to an external database server by then. - Features: use HTTP/1.1 for proxyied connections ([#1070](https://github.com/Mailu/Mailu/issues/1070)) - Features: Update Rainloop to 1.13.0 ([#1071](https://github.com/Mailu/Mailu/issues/1071)) - Features: Use python package socrate instead of Mailustart ([#1082](https://github.com/Mailu/Mailu/issues/1082)) +- Features: Added cli command config-dump ([#1377](https://github.com/Mailu/Mailu/issues/1377)) - Bugfixes: Use ldez/traefik-certs-dumper in our certificate dumper to have a more robust solution ([#820](https://github.com/Mailu/Mailu/issues/820)) - Bugfixes: Make aliases optionally case-insensitive: After attempting to resolve an alias in its preserved case, also attempt to match it case-insensitively ([#867](https://github.com/Mailu/Mailu/issues/867)) - Bugfixes: Fix HOST_* variable usage ([#884](https://github.com/Mailu/Mailu/issues/884)) @@ -47,6 +48,7 @@ deprecated before 1.8.0, you can switch to an external database server by then. - Enhancement: Create an Authentication Token with IPv6 address restriction ([#829](https://github.com/Mailu/Mailu/issues/829)) - Enhancement: Automatically create admin user on container startup if given appropriate environment variables - Enhancement: Missing wildcard option in alias flask command ([#869](https://github.com/Mailu/Mailu/issues/869)) +- Enhancement: Cli command config-update now updates all objects and parameters ([#1377](https://github.com/Mailu/Mailu/issues/1377)) v1.6.0 - 2019-01-18 ------------------- diff --git a/docs/cli.rst b/docs/cli.rst index a9cff41c..7476e676 100644 --- a/docs/cli.rst +++ b/docs/cli.rst @@ -10,6 +10,7 @@ Managing users and aliases can be done from CLI using commands: * user * user-import * user-delete +* config-dump * config-update alias @@ -62,60 +63,124 @@ primary difference with simple `user` command is that password is being imported docker-compose run --rm admin flask mailu user-import myuser example.net '$6$51ebe0cb9f1dab48effa2a0ad8660cb489b445936b9ffd812a0b8f46bca66dd549fea530ce' 'SHA512-CRYPT' user-delete ------------- +----------- .. code-block:: bash docker-compose exec admin flask mailu user-delete foo@example.net +config-dump +----------- + +The purpose of this command is to dump domain-, relay-, alias- and user-configuration to a YAML template. +If you want to export non-hashed secrets you have to add the ``--secrets`` option. +Only non-default options are dumped. If you want to dump all options use ``--full``. + +.. code-block:: bash + + docker-compose exec admin flask mailu config-dump > mail-config.yml + config-update ------------- -The sole purpose of this command is for importing users/aliases in bulk and synchronizing DB entries with external YAML template: +The purpose of this command is for importing domain-, relay-, alias- and user-configuration in bulk and synchronizing DB entries with an external YAML template. .. code-block:: bash - cat mail-config.yml | docker-compose exec -T admin flask mailu config-update --delete-objects + docker exec -i $(docker-compose ps -q admin) flask mailu config-update -nvd < mail-config.yml -where mail-config.yml looks like: +*(The current version of docker-compose exec does not pass stdin correctly, so you have to user docker exec)* -.. code-block:: bash +mail-config.yml looks like: + +.. code-block:: yaml + + domains: + - name: example.com + alternatives: + - alternative.example.com users: - - localpart: foo - domain: example.com + - email: foo@example.com password_hash: klkjhumnzxcjkajahsdqweqqwr hash_scheme: MD5-CRYPT aliases: - - localpart: alias1 - domain: example.com + - email: alias1@example.com destination: "user1@example.com,user2@example.com" -without ``--delete-object`` option config-update will only add/update new values but will *not* remove any entries missing in provided YAML input. + relays: + - name: relay.example.com + comment: test + smtp: mx.example.com -Users ------ +You can use ``--dry-run`` to test your YAML without omitting any changes to the database. +With ``--verbose`` config-update will show exactly what it changes in the database. +Without ``--delete-object`` option config-update will only add/update new values but will *not* remove any entries missing in provided YAML input. -following are additional parameters that could be defined for users: +This is a complete YAML template with all additional parameters that could be defined: -* comment -* quota_bytes -* global_admin -* enable_imap -* enable_pop -* forward_enabled -* forward_destination -* reply_enabled -* reply_subject -* reply_body -* displayed_name -* spam_enabled -* spam_threshold +.. code-block:: yaml -Alias ------ + aliases: + - email: email@example.com + comment: '' + destination: + - address@example.com + wildcard: false + + domains: + - name: example.com + alternatives: + - alternative.tld + comment: '' + dkim_key: '' + max_aliases: -1 + max_quota_bytes: 0 + max_users: -1 + signup_enabled: false + + relays: + - name: relay.example.com + comment: '' + smtp: mx.example.com + + users: + - email: postmaster@example.com + comment: '' + displayed_name: 'Postmaster' + enable_imap: true + enable_pop: false + enabled: true + fetches: + - id: 1 + comment: 'test fetch' + username: fetch-user + host: other.example.com + password: 'secret' + port: 993 + protocol: imap + tls: true + keep: true + forward_destination: + - address@remote.example.com + forward_enabled: true + forward_keep: true + global_admin: true + manager_of: + - example.com + password: '{BLF-CRYPT}$2b$12$...' + quota_bytes: 1000000000 + reply_body: '' + reply_enabled: false + reply_enddate: 2999-12-31 + reply_startdate: 1900-01-01 + reply_subject: '' + spam_enabled: true + spam_threshold: 80 + tokens: + - id: 1 + comment: email-client + ip: 192.168.1.1 + password: '$5$rounds=1000$...' -additional fields: - -* wildcard From 0cf91f35a45b7272a1b9b5cb0d8760ab4cb0bdfe Mon Sep 17 00:00:00 2001 From: Alexander Graf Date: Thu, 27 Aug 2020 16:08:15 +0200 Subject: [PATCH 008/596] moved change log entry to towncrier --- CHANGELOG.md | 2 -- towncrier/newsfragments/1604.feature | 1 + 2 files changed, 1 insertion(+), 2 deletions(-) create mode 100644 towncrier/newsfragments/1604.feature diff --git a/CHANGELOG.md b/CHANGELOG.md index 1c20d9b2..7ccde4bc 100644 --- a/CHANGELOG.md +++ b/CHANGELOG.md @@ -28,7 +28,6 @@ deprecated before 1.8.0, you can switch to an external database server by then. - Features: use HTTP/1.1 for proxyied connections ([#1070](https://github.com/Mailu/Mailu/issues/1070)) - Features: Update Rainloop to 1.13.0 ([#1071](https://github.com/Mailu/Mailu/issues/1071)) - Features: Use python package socrate instead of Mailustart ([#1082](https://github.com/Mailu/Mailu/issues/1082)) -- Features: Added cli command config-dump ([#1377](https://github.com/Mailu/Mailu/issues/1377)) - Bugfixes: Use ldez/traefik-certs-dumper in our certificate dumper to have a more robust solution ([#820](https://github.com/Mailu/Mailu/issues/820)) - Bugfixes: Make aliases optionally case-insensitive: After attempting to resolve an alias in its preserved case, also attempt to match it case-insensitively ([#867](https://github.com/Mailu/Mailu/issues/867)) - Bugfixes: Fix HOST_* variable usage ([#884](https://github.com/Mailu/Mailu/issues/884)) @@ -48,7 +47,6 @@ deprecated before 1.8.0, you can switch to an external database server by then. - Enhancement: Create an Authentication Token with IPv6 address restriction ([#829](https://github.com/Mailu/Mailu/issues/829)) - Enhancement: Automatically create admin user on container startup if given appropriate environment variables - Enhancement: Missing wildcard option in alias flask command ([#869](https://github.com/Mailu/Mailu/issues/869)) -- Enhancement: Cli command config-update now updates all objects and parameters ([#1377](https://github.com/Mailu/Mailu/issues/1377)) v1.6.0 - 2019-01-18 ------------------- diff --git a/towncrier/newsfragments/1604.feature b/towncrier/newsfragments/1604.feature new file mode 100644 index 00000000..06ee0beb --- /dev/null +++ b/towncrier/newsfragments/1604.feature @@ -0,0 +1 @@ +Added cli command config-dump and enhanced config-update From 85de70212904755a5bc6ef756cfe615dec7ff0bb Mon Sep 17 00:00:00 2001 From: Alexander Graf Date: Thu, 27 Aug 2020 16:10:53 +0200 Subject: [PATCH 009/596] small typo. Change 'omitting' to 'commiting' --- docs/cli.rst | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/docs/cli.rst b/docs/cli.rst index 7476e676..64a4b17a 100644 --- a/docs/cli.rst +++ b/docs/cli.rst @@ -114,7 +114,7 @@ mail-config.yml looks like: comment: test smtp: mx.example.com -You can use ``--dry-run`` to test your YAML without omitting any changes to the database. +You can use ``--dry-run`` to test your YAML without comitting any changes to the database. With ``--verbose`` config-update will show exactly what it changes in the database. Without ``--delete-object`` option config-update will only add/update new values but will *not* remove any entries missing in provided YAML input. From ffbeabeb6f685fa3939bbc1ef1266ef8c7ae511c Mon Sep 17 00:00:00 2001 From: Alexander Graf Date: Thu, 27 Aug 2020 22:02:20 +0200 Subject: [PATCH 010/596] updated test to use --verbose flag --verbose (or -v) is now a flag and not an option --- tests/compose/core/02_forward_test.sh | 4 ++-- tests/compose/core/03_alias_test.sh | 4 ++-- tests/compose/core/04_reply_test.sh | 4 ++-- 3 files changed, 6 insertions(+), 6 deletions(-) diff --git a/tests/compose/core/02_forward_test.sh b/tests/compose/core/02_forward_test.sh index 595820cf..651e027c 100755 --- a/tests/compose/core/02_forward_test.sh +++ b/tests/compose/core/02_forward_test.sh @@ -1,4 +1,4 @@ -cat << EOF | docker-compose -f tests/compose/core/docker-compose.yml exec -T admin flask mailu config-update -v 1 +cat << EOF | docker-compose -f tests/compose/core/docker-compose.yml exec -T admin flask mailu config-update --verbose users: - localpart: forwardinguser password_hash: "\$1\$F2OStvi1\$Q8hBIHkdJpJkJn/TrMIZ9/" @@ -10,7 +10,7 @@ EOF python3 tests/forward_test.py -cat << EOF | docker-compose -f tests/compose/core/docker-compose.yml exec -T admin flask mailu config-update -v 1 +cat << EOF | docker-compose -f tests/compose/core/docker-compose.yml exec -T admin flask mailu config-update --verbose users: - localpart: forwardinguser password_hash: "\$1\$F2OStvi1\$Q8hBIHkdJpJkJn/TrMIZ9/" diff --git a/tests/compose/core/03_alias_test.sh b/tests/compose/core/03_alias_test.sh index dce1918a..2d40903a 100755 --- a/tests/compose/core/03_alias_test.sh +++ b/tests/compose/core/03_alias_test.sh @@ -1,4 +1,4 @@ -cat << EOF | docker-compose -f tests/compose/core/docker-compose.yml exec -T admin flask mailu config-update -v 1 +cat << EOF | docker-compose -f tests/compose/core/docker-compose.yml exec -T admin flask mailu config-update --verbose aliases: - localpart: alltheusers domain: mailu.io @@ -7,6 +7,6 @@ EOF python3 tests/alias_test.py -cat << EOF | docker-compose -f tests/compose/core/docker-compose.yml exec -T admin flask mailu config-update -v 1 +cat << EOF | docker-compose -f tests/compose/core/docker-compose.yml exec -T admin flask mailu config-update --verbose aliases: [] EOF diff --git a/tests/compose/core/04_reply_test.sh b/tests/compose/core/04_reply_test.sh index 83c114f6..7615a0f8 100755 --- a/tests/compose/core/04_reply_test.sh +++ b/tests/compose/core/04_reply_test.sh @@ -1,4 +1,4 @@ -cat << EOF | docker-compose -f tests/compose/core/docker-compose.yml exec -T admin flask mailu config-update -v 1 +cat << EOF | docker-compose -f tests/compose/core/docker-compose.yml exec -T admin flask mailu config-update --verbose users: - localpart: replyuser password_hash: "\$1\$F2OStvi1\$Q8hBIHkdJpJkJn/TrMIZ9/" @@ -11,7 +11,7 @@ EOF python3 tests/reply_test.py -cat << EOF | docker-compose -f tests/compose/core/docker-compose.yml exec -T admin flask mailu config-update -v 1 +cat << EOF | docker-compose -f tests/compose/core/docker-compose.yml exec -T admin flask mailu config-update --verbose users: - localpart: replyuser password_hash: "\$1\$F2OStvi1\$Q8hBIHkdJpJkJn/TrMIZ9/" From 02cfe326d3226f8dfffeda445c3de736b64288d7 Mon Sep 17 00:00:00 2001 From: lub Date: Sun, 30 Aug 2020 01:04:36 +0200 Subject: [PATCH 011/596] support using files for SECRET_KEY and DB_PW this enables usage of e.g. docker swarm secrets instead of exposing the passwords directly via environment variables just use DB_PW_FILE and SECRET_KEY_FILE instead of DB_PW and SECRET_KEY --- core/admin/mailu/configuration.py | 11 ++++++++++- webmails/roundcube/config.inc.php | 2 +- 2 files changed, 11 insertions(+), 2 deletions(-) diff --git a/core/admin/mailu/configuration.py b/core/admin/mailu/configuration.py index 66b0b832..eacf7803 100644 --- a/core/admin/mailu/configuration.py +++ b/core/admin/mailu/configuration.py @@ -100,6 +100,15 @@ class ConfigManager(dict): if self.config["WEBMAIL"] != "none": self.config["WEBMAIL_ADDRESS"] = self.get_host_address("WEBMAIL") + def __get_env(self, key, value): + key_file = key + "_FILE" + if key_file in os.environ: + with open(os.environ.get(key_file)) as file: + value_from_file = file.read() + return value_from_file.strip() + else: + return os.environ.get(key, value) + def __coerce_value(self, value): if isinstance(value, str) and value.lower() in ('true','yes'): return True @@ -111,7 +120,7 @@ class ConfigManager(dict): self.config.update(app.config) # get environment variables self.config.update({ - key: self.__coerce_value(os.environ.get(key, value)) + key: self.__coerce_value(self.__get_env(key, value)) for key, value in DEFAULT_CONFIG.items() }) self.resolve_hosts() diff --git a/webmails/roundcube/config.inc.php b/webmails/roundcube/config.inc.php index eb40047a..627b96a7 100644 --- a/webmails/roundcube/config.inc.php +++ b/webmails/roundcube/config.inc.php @@ -5,7 +5,7 @@ $config = array(); // Generals $config['db_dsnw'] = getenv('DB_DSNW');; $config['temp_dir'] = '/tmp/'; -$config['des_key'] = getenv('SECRET_KEY'); +$config['des_key'] = getenv('SECRET_KEY') ? getenv('SECRET_KEY') : trim(file_get_contents(getenv('SECRET_KEY_FILE'))); $config['cipher_method'] = 'AES-256-CBC'; $config['identities_level'] = 0; $config['reply_all_mode'] = 1; From 426355f6b861058a1f489f791f23c2ab44d6ed60 Mon Sep 17 00:00:00 2001 From: lub Date: Sun, 30 Aug 2020 01:14:47 +0200 Subject: [PATCH 012/596] add some docs about _FILE variables --- docs/swarm/master/README.md | 3 +++ 1 file changed, 3 insertions(+) diff --git a/docs/swarm/master/README.md b/docs/swarm/master/README.md index 58723c33..42e742da 100644 --- a/docs/swarm/master/README.md +++ b/docs/swarm/master/README.md @@ -106,6 +106,9 @@ As a side effect of this ingress mode "feature", make sure that the ingress subn - front and webmail are scalable (pending POD_ADDRESS_RANGE is used), although the let's encrypt magic might not like it (race condidtion ? or risk to be banned by let's encrypt server if too many front containers attemps to renew the certs at the same time) - redis, antispam, antivirus, fetchmail, admin, webdav have not been tested (hence replicas=1 in the following docker-compose.yml file) +## Docker secrets +There are DB_PW_FILE and SECRET_KEY_FILE environment variables available to specify files for these variables. These can be used to configure Docker secrets instead of writing the values directly into the `docker-compose.yml` or `mailu.env`. + ## Variable substitution and docker-compose.yml The docker stack deploy command doesn't support variable substitution in the .yml file itself. As a consequence, we cannot simply use ``` docker stack deploy -c docker.compose.yml mailu ``` From 714fa044e08288d14ba02b016b5aca2114e68faa Mon Sep 17 00:00:00 2001 From: lub Date: Sun, 30 Aug 2020 01:19:42 +0200 Subject: [PATCH 013/596] add towncrier for #1607 --- towncrier/newsfragments/1607.feature | 1 + 1 file changed, 1 insertion(+) create mode 100644 towncrier/newsfragments/1607.feature diff --git a/towncrier/newsfragments/1607.feature b/towncrier/newsfragments/1607.feature new file mode 100644 index 00000000..de9f0895 --- /dev/null +++ b/towncrier/newsfragments/1607.feature @@ -0,0 +1 @@ +Implement SECRET_KEY_FILE and DB_PW_FILE variables for usage with Docker secrets. From f0f873ffe7f176d480dc9a80cb68242f1fe8a69d Mon Sep 17 00:00:00 2001 From: lub Date: Tue, 1 Sep 2020 21:48:09 +0200 Subject: [PATCH 014/596] add option to enforce inbound starttls --- core/admin/mailu/configuration.py | 1 + core/admin/mailu/internal/nginx.py | 30 ++++++++++++++++++++++++------ docs/configuration.rst | 7 +++++++ 3 files changed, 32 insertions(+), 6 deletions(-) diff --git a/core/admin/mailu/configuration.py b/core/admin/mailu/configuration.py index 66b0b832..7fcd1ee7 100644 --- a/core/admin/mailu/configuration.py +++ b/core/admin/mailu/configuration.py @@ -31,6 +31,7 @@ DEFAULT_CONFIG = { 'HOSTNAMES': 'mail.mailu.io,alternative.mailu.io,yetanother.mailu.io', 'POSTMASTER': 'postmaster', 'TLS_FLAVOR': 'cert', + 'INBOUND_TLS_ENFORCE': False, 'AUTH_RATELIMIT': '10/minute;1000/hour', 'AUTH_RATELIMIT_SUBNET': True, 'DISABLE_STATISTICS': False, diff --git a/core/admin/mailu/internal/nginx.py b/core/admin/mailu/internal/nginx.py index fa127584..a7771f91 100644 --- a/core/admin/mailu/internal/nginx.py +++ b/core/admin/mailu/internal/nginx.py @@ -17,6 +17,9 @@ STATUSES = { "smtp": "535 5.7.8", "pop3": "-ERR Authentication failed" }), + "encryption": ("Must issue a STARTTLS command first", { + "smtp": "530 5.7.0" + }), } @@ -28,12 +31,27 @@ def handle_authentication(headers): protocol = headers["Auth-Protocol"] # Incoming mail, no authentication if method == "none" and protocol == "smtp": - server, port = get_server(headers["Auth-Protocol"], False) - return { - "Auth-Status": "OK", - "Auth-Server": server, - "Auth-Port": port - } + server, port = get_server(protocol, False) + if app.config["INBOUND_TLS_ENFORCE"]: + if "Auth-SSl" in headers and headers["Auth-SSL"] == "on": + return { + "Auth-Status": "OK", + "Auth-Server": server, + "Auth-Port": port + } + else: + status, code = get_status(protocol, "encryption") + return { + "Auth-Status": status, + "Auth-Error-Code" : code, + "Auth-Wait": 0 + } + else: + return { + "Auth-Status": "OK", + "Auth-Server": server, + "Auth-Port": port + } # Authenticated user elif method == "plain": server, port = get_server(headers["Auth-Protocol"], True) diff --git a/docs/configuration.rst b/docs/configuration.rst index 4b211925..3438b7fe 100644 --- a/docs/configuration.rst +++ b/docs/configuration.rst @@ -73,6 +73,13 @@ By default postfix uses "opportunistic TLS" for outbound mail. This can be chang by setting ``OUTBOUND_TLS_LEVEL`` to ``encrypt``. This setting is highly recommended if you are a relayhost that supports TLS. +Similarily by default nginx uses "opportunistic TLS" for inbound mail. This can be changed +by setting ``INBOUND_TLS_ENFORCE`` to ``True``. Please note that this is forbidden for +internet facing hosts according to e.g. `RFC 3207`_ , because this prevents MTAs without STARTTLS +support or e.g. mismatching TLS versions to deliver emails to Mailu. + +.. _`RFC 3207`: https://tools.ietf.org/html/rfc3207 + The ``FETCHMAIL_DELAY`` is a delay (in seconds) for the fetchmail service to go and fetch new email if available. Do not use too short delays if you do not want to be blacklisted by external services, but not too long delays if you From d348477efc933e5f2d1fa85c027b7d7fb2ccf623 Mon Sep 17 00:00:00 2001 From: lub Date: Tue, 1 Sep 2020 21:50:21 +0200 Subject: [PATCH 015/596] add towncrier for 1610 --- towncrier/newsfragments/1610.feature | 1 + 1 file changed, 1 insertion(+) create mode 100644 towncrier/newsfragments/1610.feature diff --git a/towncrier/newsfragments/1610.feature b/towncrier/newsfragments/1610.feature new file mode 100644 index 00000000..b56ac332 --- /dev/null +++ b/towncrier/newsfragments/1610.feature @@ -0,0 +1 @@ +Add possibility to enforce inbound STARTTLS via INBOUND_TLS_LEVEL=true From 05e2af180256b6a9387cd437d023c8d3f968a651 Mon Sep 17 00:00:00 2001 From: lub Date: Wed, 2 Sep 2020 15:16:10 +0200 Subject: [PATCH 016/596] fix small typo in Auth-SSL --- core/admin/mailu/internal/nginx.py | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/core/admin/mailu/internal/nginx.py b/core/admin/mailu/internal/nginx.py index a7771f91..e1f8cb7a 100644 --- a/core/admin/mailu/internal/nginx.py +++ b/core/admin/mailu/internal/nginx.py @@ -33,7 +33,7 @@ def handle_authentication(headers): if method == "none" and protocol == "smtp": server, port = get_server(protocol, False) if app.config["INBOUND_TLS_ENFORCE"]: - if "Auth-SSl" in headers and headers["Auth-SSL"] == "on": + if "Auth-SSL" in headers and headers["Auth-SSL"] == "on": return { "Auth-Status": "OK", "Auth-Server": server, From 9d2327b0f1b62de4f57e0c4367434572ae85c46a Mon Sep 17 00:00:00 2001 From: Alexander Graf Date: Fri, 4 Sep 2020 12:32:51 +0200 Subject: [PATCH 017/596] add space for more human readable indentation add a newline before main sections add some spaces to indent --- core/admin/mailu/manage.py | 12 +++++++++++- 1 file changed, 11 insertions(+), 1 deletion(-) diff --git a/core/admin/mailu/manage.py b/core/admin/mailu/manage.py index f74d24b5..c36f45c0 100644 --- a/core/admin/mailu/manage.py +++ b/core/admin/mailu/manage.py @@ -305,13 +305,23 @@ def config_update(verbose=False, delete_objects=False, dry_run=False, file=None) def config_dump(full=False, secrets=False): """dump configuration as YAML-formatted data to stdout""" + class spacedDumper(yaml.Dumper): + + def write_line_break(self, data=None): + super().write_line_break(data) + if len(self.indents) == 1: + super().write_line_break() + + def increase_indent(self, flow=False, indentless=False): + return super().increase_indent(flow, False) + config = {} for section, model in yaml_sections: dump = [item.to_dict(full, secrets) for item in model.query.all()] if len(dump): config[section] = dump - yaml.dump(config, sys.stdout, default_flow_style=False, allow_unicode=True) + yaml.dump(config, sys.stdout, Dumper=spacedDumper, default_flow_style=False, allow_unicode=True) @mailu.command() From 8e14aa80ee2ea6546bd659a03813db44bd9f0024 Mon Sep 17 00:00:00 2001 From: Alexander Graf Date: Fri, 4 Sep 2020 12:57:40 +0200 Subject: [PATCH 018/596] documented options and added help text --- core/admin/mailu/manage.py | 10 +++++----- docs/cli.rst | 40 +++++++++++++++++++++++++++++++++----- 2 files changed, 40 insertions(+), 10 deletions(-) diff --git a/core/admin/mailu/manage.py b/core/admin/mailu/manage.py index c36f45c0..b25ab568 100644 --- a/core/admin/mailu/manage.py +++ b/core/admin/mailu/manage.py @@ -180,9 +180,9 @@ yaml_sections = [ ] @mailu.command() -@click.option('-v', '--verbose', is_flag=True) -@click.option('-d', '--delete-objects', is_flag=True) -@click.option('-n', '--dry-run', is_flag=True) +@click.option('-v', '--verbose', is_flag=True, help='Increase verbosity') +@click.option('-d', '--delete-objects', is_flag=True, help='Remove objects not included in yaml') +@click.option('-n', '--dry-run', is_flag=True, help='Perform a trial run with no changes made') @flask_cli.with_appcontext def config_update(verbose=False, delete_objects=False, dry_run=False, file=None): """sync configuration with data from YAML-formatted stdin""" @@ -299,8 +299,8 @@ def config_update(verbose=False, delete_objects=False, dry_run=False, file=None) @mailu.command() -@click.option('-f', '--full', is_flag=True) -@click.option('-s', '--secrets', is_flag=True) +@click.option('-f', '--full', is_flag=True, help='Include default attributes') +@click.option('-s', '--secrets', is_flag=True, help='Include secrets (plain-text / not hashed)') @flask_cli.with_appcontext def config_dump(full=False, secrets=False): """dump configuration as YAML-formatted data to stdout""" diff --git a/docs/cli.rst b/docs/cli.rst index 64a4b17a..0036e504 100644 --- a/docs/cli.rst +++ b/docs/cli.rst @@ -73,8 +73,22 @@ config-dump ----------- The purpose of this command is to dump domain-, relay-, alias- and user-configuration to a YAML template. -If you want to export non-hashed secrets you have to add the ``--secrets`` option. -Only non-default options are dumped. If you want to dump all options use ``--full``. + +.. code-block:: bash + + # docker-compose exec admin flask mailu config-dump --help + + Usage: flask mailu config-dump [OPTIONS] + + dump configuration as YAML-formatted data to stdout + + Options: + -f, --full Include default attributes + -s, --secrets Include secrets (plain-text / not hashed) + --help Show this message and exit. + +If you want to export secrets (plain-text / not hashed) you have to add the ``--secrets`` option. +Only non-default attributes are dumped. If you want to dump all attributes use ``--full``. .. code-block:: bash @@ -85,13 +99,29 @@ config-update The purpose of this command is for importing domain-, relay-, alias- and user-configuration in bulk and synchronizing DB entries with an external YAML template. +.. code-block:: bash + + # docker-compose exec admin flask mailu config-update --help + + Usage: flask mailu config-update [OPTIONS] + + sync configuration with data from YAML-formatted stdin + + Options: + -v, --verbose Increase verbosity + -d, --delete-objects Remove objects not included in yaml + -n, --dry-run Perform a trial run with no changes made + --help Show this message and exit. + + +The current version of docker-compose exec does not pass stdin correctly, so you have to user docker exec instead: + .. code-block:: bash docker exec -i $(docker-compose ps -q admin) flask mailu config-update -nvd < mail-config.yml -*(The current version of docker-compose exec does not pass stdin correctly, so you have to user docker exec)* -mail-config.yml looks like: +mail-config.yml looks like this: .. code-block:: yaml @@ -116,7 +146,7 @@ mail-config.yml looks like: You can use ``--dry-run`` to test your YAML without comitting any changes to the database. With ``--verbose`` config-update will show exactly what it changes in the database. -Without ``--delete-object`` option config-update will only add/update new values but will *not* remove any entries missing in provided YAML input. +Without ``--delete-object`` option config-update will only add/update changed values but will *not* remove any entries missing in provided YAML input. This is a complete YAML template with all additional parameters that could be defined: From 66db1f8fd075a0160915814b2a9d945f123efcf4 Mon Sep 17 00:00:00 2001 From: lub Date: Sat, 12 Sep 2020 01:32:03 +0200 Subject: [PATCH 019/596] add OCSP stapling to nginx.conf It's not added in tls.conf, because apparently the mail ssl module doesnt' support OCSP stapling. https://nginx.org/en/docs/http/ngx_http_ssl_module.html#ssl_stapling ^ exists https://nginx.org/en/docs/mail/ngx_mail_ssl_module.html#ssl_stapling ^ missing When the configured certificate doesn't have OCSP information, it'll just log a warning during startup. --- core/nginx/conf/nginx.conf | 2 ++ 1 file changed, 2 insertions(+) diff --git a/core/nginx/conf/nginx.conf b/core/nginx/conf/nginx.conf index 46db324f..bea822a5 100644 --- a/core/nginx/conf/nginx.conf +++ b/core/nginx/conf/nginx.conf @@ -58,6 +58,8 @@ http { listen [::]:443 ssl http2; include /etc/nginx/tls.conf; + ssl_stapling on; + ssl_stapling_verify on; ssl_session_cache shared:SSLHTTP:50m; add_header Strict-Transport-Security 'max-age=31536000'; From e8b67470800f7b9c0a808b9d0f71b5bd034e892f Mon Sep 17 00:00:00 2001 From: lub Date: Sat, 12 Sep 2020 01:38:37 +0200 Subject: [PATCH 020/596] add newsfragemnt for #1618 --- towncrier/newsfragments/1618.feature | 1 + 1 file changed, 1 insertion(+) create mode 100644 towncrier/newsfragments/1618.feature diff --git a/towncrier/newsfragments/1618.feature b/towncrier/newsfragments/1618.feature new file mode 100644 index 00000000..443f2b5c --- /dev/null +++ b/towncrier/newsfragments/1618.feature @@ -0,0 +1 @@ +Enable OCSP stapling for the http server within nginx. From 5e32447f07a2148cc1800a49ce7c9c7a19049d31 Mon Sep 17 00:00:00 2001 From: Jon Wilson Date: Mon, 21 Sep 2020 15:06:43 +0100 Subject: [PATCH 021/596] Change unbound logfile to the empty string This is defined to send log messages to stderr, which is what we want - fixes #1536 ("Could not open logfile /dev/stdout: Permission denied") --- optional/unbound/unbound.conf | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/optional/unbound/unbound.conf b/optional/unbound/unbound.conf index 8abd4325..6c8fc64d 100644 --- a/optional/unbound/unbound.conf +++ b/optional/unbound/unbound.conf @@ -2,7 +2,7 @@ server: verbosity: 1 interface: 0.0.0.0 interface: ::0 - logfile: /dev/stdout + logfile: "" do-ip4: yes do-ip6: yes do-udp: yes From 59bc4f7aea9da3f3ccf9ceca9a07af825de9e4af Mon Sep 17 00:00:00 2001 From: anrc <15327800+githtz@users.noreply.github.com> Date: Thu, 24 Sep 2020 13:16:25 +0200 Subject: [PATCH 022/596] Remove the username from the milter_headers Rspamd adds the name of the authenticated user by default. Setting add_smtp_user to false prevents the login to be leaked. --- core/rspamd/conf/milter_headers.conf | 3 +++ 1 file changed, 3 insertions(+) diff --git a/core/rspamd/conf/milter_headers.conf b/core/rspamd/conf/milter_headers.conf index cb680cfa..7da200df 100644 --- a/core/rspamd/conf/milter_headers.conf +++ b/core/rspamd/conf/milter_headers.conf @@ -5,6 +5,9 @@ skip_authenticated = false; use = ["x-spamd-bar", "x-spam-level", "x-virus", "authentication-results"]; routines { + authentication-results { + add_smtp_user = false; + } x-virus { symbols = ["CLAM_VIRUS", "FPROT_VIRUS", "JUST_EICAR"]; } From 3f037e2f08acdf078064e4e03fc1c2b46551b27a Mon Sep 17 00:00:00 2001 From: anrc <15327800+githtz@users.noreply.github.com> Date: Thu, 24 Sep 2020 16:53:42 +0200 Subject: [PATCH 023/596] Add changelog --- towncrier/newsfragments/1638.fix | 1 + 1 file changed, 1 insertion(+) create mode 100644 towncrier/newsfragments/1638.fix diff --git a/towncrier/newsfragments/1638.fix b/towncrier/newsfragments/1638.fix new file mode 100644 index 00000000..9a87e41e --- /dev/null +++ b/towncrier/newsfragments/1638.fix @@ -0,0 +1 @@ +Hide the login of the user in sent emails From ef71bc04cb27f6ed0779584afcc8690fbf20048a Mon Sep 17 00:00:00 2001 From: Patryk Tech Date: Thu, 1 Oct 2020 13:51:06 +0300 Subject: [PATCH 024/596] Update docs/reverse.rst with Traefik v2+ info --- docs/reverse.rst | 41 ++++++++++++++++++++++++++++++-- towncrier/newsfragments/1503.doc | 1 + 2 files changed, 40 insertions(+), 2 deletions(-) create mode 100644 towncrier/newsfragments/1503.doc diff --git a/docs/reverse.rst b/docs/reverse.rst index 9a7a7dc0..f3b3e7bd 100644 --- a/docs/reverse.rst +++ b/docs/reverse.rst @@ -154,7 +154,40 @@ Add the respective Traefik labels for your domain/configuration, like If your Traefik is configured to automatically request certificates from *letsencrypt*, then you’ll have a certificate for ``mail.your.doma.in`` now. However, ``mail.your.doma.in`` might only be the location where you want the Mailu web-interfaces to live — your mail should be sent/received from ``your.doma.in``, and this is the ``DOMAIN`` in your ``.env``? -To support that use-case, Traefik can request ``SANs`` for your domain. Lets add something like +To support that use-case, Traefik can request ``SANs`` for your domain. The configuration for this will depend on your Traefik version. + +---- + +Traefik 2.x using labels configuration +~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~ + +Add the appropriate labels for your domain(s) to the ``front`` container in ``docker-compose.yml``. + +.. code-block:: yaml + + services: + front: + labels: + # Enable TLS + - "traefik.http.routers.mailu-secure.tls" + # Your main domain + - "traefik.http.routers.mailu-secure.tls.domains[0].main=your.doma.in" + # Optional SANs for your main domain + - "traefik.http.routers.mailu-secure.tls.domains[0].sans=mail.your.doma.in,webmail.your.doma.in,smtp.your.doma.in" + # Optionally add other domains + - "traefik.http.routers.mailu-secure.tls.domains[1].main=mail.other.doma.in" + - "traefik.http.routers.mailu-secure.tls.domains[1].sans=mail2.other.doma.in,mail3.other.doma.in" + # Your ACME certificate resolver + - "traefik.http.routers.mailu-secure.tls.certResolver=foo" + +Of course, be sure to define the Certificate Resolver ``foo`` in the static configuration as well. + +Alternatively, you can define SANs in the Traefik static configuration using routers, or in the static configuration using entrypoints. Refer to the Traefik documentation for more details. + +Traefik 1.x with TOML configuration +~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~ + +Lets add something like .. code-block:: yaml @@ -163,7 +196,11 @@ To support that use-case, Traefik can request ``SANs`` for your domain. Lets add main = "your.doma.in" # this is the same as $TRAEFIK_DOMAIN! sans = ["mail.your.doma.in", "webmail.your.doma.in", "smtp.your.doma.in"] -to your ``traefik.toml``. You might need to clear your ``acme.json``, if a certificate for one of these domains already exists. +to your ``traefik.toml``. + +---- + +You might need to clear your ``acme.json``, if a certificate for one of these domains already exists. You will need some solution which dumps the certificates in ``acme.json``, so you can include them in the ``mailu/front`` container. One such example is ``mailu/traefik-certdumper``, which has been adapted for use in Mailu. You can add it to your ``docker-compose.yml`` like: diff --git a/towncrier/newsfragments/1503.doc b/towncrier/newsfragments/1503.doc new file mode 100644 index 00000000..9c59feb7 --- /dev/null +++ b/towncrier/newsfragments/1503.doc @@ -0,0 +1 @@ +Add documentation for Traefik 2 in Reverse Proxy \ No newline at end of file From e7caff9811512dc47bddcd9aa11ce983254e7c40 Mon Sep 17 00:00:00 2001 From: David Fairbrother Date: Mon, 5 Oct 2020 15:13:07 +0100 Subject: [PATCH 025/596] Add ability to set no WEBROOT_REDIRECT to Nginx Adds a 'none' env option to WEBROOT_REDIRECT so that no `location /` configuration is written to nginx.conf. This is useful for setting up Mailu and Mailman where we override the root to proxy to the mailing list server instead. Without this change the nginx container will not start, or for 1.7 users can set their WEBMAIL_PATH to / with no webmail to get the same results. This fix means that future users don't have to choose between webmail and a root override and makes the configuration intention clear. --- core/nginx/conf/nginx.conf | 2 +- docs/configuration.rst | 15 ++++++++++----- docs/faq.rst | 2 ++ 3 files changed, 13 insertions(+), 6 deletions(-) diff --git a/core/nginx/conf/nginx.conf b/core/nginx/conf/nginx.conf index 8f6eaa0d..f672c7a3 100644 --- a/core/nginx/conf/nginx.conf +++ b/core/nginx/conf/nginx.conf @@ -115,7 +115,7 @@ http { include /overrides/*.conf; # Actual logic - {% if WEB_WEBMAIL != '/' %} + {% if WEB_WEBMAIL != '/' and WEBROOT_REDIRECT != 'none' %} location / { {% if WEBROOT_REDIRECT %} try_files $uri {{ WEBROOT_REDIRECT }}; diff --git a/docs/configuration.rst b/docs/configuration.rst index 5ff3546a..c2c55190 100644 --- a/docs/configuration.rst +++ b/docs/configuration.rst @@ -92,14 +92,19 @@ the localpart for DMARC rua and ruf email addresses. Full-text search is enabled for IMAP is enabled by default. This feature can be disabled (e.g. for performance reasons) by setting the optional variable ``FULL_TEXT_SEARCH`` to ``off``. +.. _web_settings: + Web settings ------------ -The ``WEB_ADMIN`` contains the path to the main admin interface, while -``WEB_WEBMAIL`` contains the path to the Web email client. -The ``WEBROOT_REDIRECT`` redirects all non-found queries to the set path. -An empty ``WEBROOT_REDIRECT`` value disables redirecting and enables classic -behavior of a 404 result when not found. +- ``WEB_ADMIN`` contains the path to the main admin interface + +- ``WEB_WEBMAIL`` contains the path to the Web email client. + +- ``WEBROOT_REDIRECT`` redirects all non-found queries to the set path. + An empty ``WEBROOT_REDIRECT`` value disables redirecting and enables classic behavior of a 404 result when not found. + Alternatively, ``WEBROOT_REDIRECT`` can be set to ``none`` if you are using an Nginx override for ``location /``. + All three options need a leading slash (``/``) to work. .. note:: ``WEBROOT_REDIRECT`` has to point to a valid path on the webserver. diff --git a/docs/faq.rst b/docs/faq.rst index b292cd05..296cd59f 100644 --- a/docs/faq.rst +++ b/docs/faq.rst @@ -262,6 +262,8 @@ correct syntax. The following file names will be taken as override configuration - `Nginx`_ - All ``*.conf`` files in the ``nginx`` sub-directory; - `Rspamd`_ - All files in the ``rspamd`` sub-directory. +To override the root location (``/``) in Nginx ``WEBROOT_REDIRECT`` needs to be set to ``none`` in the env file (see :ref:`web settings `). + *Issue reference:* `206`_, `1368`_. I want to integrate Nextcloud 15 (and newer) with Mailu From 45cdcbdab9635e2b63d8862898187e7a0c9b13b5 Mon Sep 17 00:00:00 2001 From: Ciprian Pascu Date: Sun, 18 Oct 2020 19:27:50 +0300 Subject: [PATCH 026/596] Update front.yaml To match the other kubernetes objects' version --- docs/kubernetes/mailu/front.yaml | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/docs/kubernetes/mailu/front.yaml b/docs/kubernetes/mailu/front.yaml index a1d5acb2..2fba1026 100644 --- a/docs/kubernetes/mailu/front.yaml +++ b/docs/kubernetes/mailu/front.yaml @@ -1,4 +1,4 @@ -apiVersion: apps/v1beta2 +apiVersion: apps/v1 kind: DaemonSet metadata: name: mailu-front From acc728109bc94e9d996985c582cdfc96ba8ecde9 Mon Sep 17 00:00:00 2001 From: Alexander Graf Date: Sat, 24 Oct 2020 22:31:13 +0200 Subject: [PATCH 027/596] validate dkim keys and allow removal --- core/admin/mailu/models.py | 41 +++++++++++++++++++++++++------------- 1 file changed, 27 insertions(+), 14 deletions(-) diff --git a/core/admin/mailu/models.py b/core/admin/mailu/models.py index cbafc6a4..0dc82bf4 100644 --- a/core/admin/mailu/models.py +++ b/core/admin/mailu/models.py @@ -218,7 +218,7 @@ class Base(db.Model): if rel is None: itype = getattr(model, '_dict_types', {}).get(key) if itype is not None: - if type(value) is not itype: + if not isinstance(value, itype): raise TypeError(f'{model.__table__}.{key} {value!r} has invalid type {type(value).__name__!r}', model, key, data) else: raise NotImplementedError(f'type not defined for {model.__table__}.{key}') @@ -365,25 +365,34 @@ class Domain(Base): _dict_hide = {'users', 'managers', 'aliases'} _dict_show = {'dkim_key'} _dict_secret = {'dkim_key'} - _dict_types = {'dkim_key': bytes} + _dict_types = {'dkim_key': (bytes, type(None))} _dict_output = {'dkim_key': lambda v: v.decode('utf-8').strip().split('\n')[1:-1]} @staticmethod def _dict_input(data): if 'dkim_key' in data: key = data['dkim_key'] - if key is None: - del data['dkim_key'] - else: + if key is not None: if type(key) is list: key = ''.join(key) if type(key) is str: - key = ''.join(key.strip().split()) - if key.startswith('-----BEGIN PRIVATE KEY-----'): - key = key[25:] - if key.endswith('-----END PRIVATE KEY-----'): - key = key[:-23] - key = '\n'.join(wrap(key, 64)) - data['dkim_key'] = f'-----BEGIN PRIVATE KEY-----\n{key}\n-----END PRIVATE KEY-----\n'.encode('ascii') + key = ''.join(key.strip().split()) # removes all whitespace + if key: + m = re.match('^-----BEGIN (RSA )?PRIVATE KEY-----', key) + if m is not None: + key = key[m.end():] + m = re.search('-----END (RSA )?PRIVATE KEY-----$', key) + if m is not None: + key = key[:m.start()] + key = '\n'.join(wrap(key, 64)) + key = f'-----BEGIN PRIVATE KEY-----\n{key}\n-----END PRIVATE KEY-----\n'.encode('ascii') + try: + dkim.strip_key(key) + except: + raise ValueError('invalid dkim key') + else: + data['dkim_key'] = key + else: + data['dkim_key'] = None name = db.Column(IdnaDomain, primary_key=True, nullable=False) managers = db.relationship('User', secondary=managers, @@ -405,8 +414,12 @@ class Domain(Base): def dkim_key(self, value): file_path = app.config["DKIM_PATH"].format( domain=self.name, selector=app.config["DKIM_SELECTOR"]) - with open(file_path, "wb") as handle: - handle.write(value) + if value is None: + if os.path.exists(file_path): + os.unlink(file_path) + else: + with open(file_path, "wb") as handle: + handle.write(value) @property def dkim_publickey(self): From c46f9328f700b12cd5dda4fbe0c270d54e584ae2 Mon Sep 17 00:00:00 2001 From: Alexander Graf Date: Sat, 24 Oct 2020 22:31:26 +0200 Subject: [PATCH 028/596] also dump dkim_publickey. allow key generation. --- core/admin/mailu/models.py | 8 +++++--- 1 file changed, 5 insertions(+), 3 deletions(-) diff --git a/core/admin/mailu/models.py b/core/admin/mailu/models.py index 0dc82bf4..a9cc8d16 100644 --- a/core/admin/mailu/models.py +++ b/core/admin/mailu/models.py @@ -363,7 +363,7 @@ class Domain(Base): __tablename__ = "domain" _dict_hide = {'users', 'managers', 'aliases'} - _dict_show = {'dkim_key'} + _dict_show = {'dkim_key', 'dkim_publickey'} _dict_secret = {'dkim_key'} _dict_types = {'dkim_key': (bytes, type(None))} _dict_output = {'dkim_key': lambda v: v.decode('utf-8').strip().split('\n')[1:-1]} @@ -376,7 +376,9 @@ class Domain(Base): key = ''.join(key) if type(key) is str: key = ''.join(key.strip().split()) # removes all whitespace - if key: + if key == 'generate': + data['dkim_key'] = dkim.gen_key() + elif key: m = re.match('^-----BEGIN (RSA )?PRIVATE KEY-----', key) if m is not None: key = key[m.end():] @@ -390,7 +392,7 @@ class Domain(Base): except: raise ValueError('invalid dkim key') else: - data['dkim_key'] = key + data['dkim_key'] = key else: data['dkim_key'] = None From 500967b2f585bca74a53b15a49986293f16e1750 Mon Sep 17 00:00:00 2001 From: Alexander Graf Date: Sat, 24 Oct 2020 22:31:29 +0200 Subject: [PATCH 029/596] ignore dkim_publickey when updating config --- core/admin/mailu/models.py | 10 +++++++--- 1 file changed, 7 insertions(+), 3 deletions(-) diff --git a/core/admin/mailu/models.py b/core/admin/mailu/models.py index a9cc8d16..6c82baa0 100644 --- a/core/admin/mailu/models.py +++ b/core/admin/mailu/models.py @@ -202,13 +202,14 @@ class Base(db.Model): raise KeyError(f'primary key {model.__table__}.{pkey} is missing', model, pkey, data) # check data keys and values - for key, value in data.items(): + for key in list(data.keys()): # check key if not hasattr(model, key) and not key in model.__mapper__.relationships: raise KeyError(f'unknown key {model.__table__}.{key}', model, key, data) # check value type + value = data[key] col = model.__mapper__.columns.get(key) if col is not None: if not ((value is None and col.nullable) or (type(value) is col.type.python_type)): @@ -218,7 +219,10 @@ class Base(db.Model): if rel is None: itype = getattr(model, '_dict_types', {}).get(key) if itype is not None: - if not isinstance(value, itype): + if not itype: # empty tuple => ignore value + del data[key] + continue + elif not isinstance(value, itype): raise TypeError(f'{model.__table__}.{key} {value!r} has invalid type {type(value).__name__!r}', model, key, data) else: raise NotImplementedError(f'type not defined for {model.__table__}.{key}') @@ -365,7 +369,7 @@ class Domain(Base): _dict_hide = {'users', 'managers', 'aliases'} _dict_show = {'dkim_key', 'dkim_publickey'} _dict_secret = {'dkim_key'} - _dict_types = {'dkim_key': (bytes, type(None))} + _dict_types = {'dkim_key': (bytes, type(None)), 'dkim_publickey': tuple()} _dict_output = {'dkim_key': lambda v: v.decode('utf-8').strip().split('\n')[1:-1]} @staticmethod def _dict_input(data): From 2a5c46c890bffac107d00f13cf6acbb9047e59e8 Mon Sep 17 00:00:00 2001 From: Alexander Graf Date: Sat, 24 Oct 2020 22:31:31 +0200 Subject: [PATCH 030/596] Allow to dump only selected sections --- core/admin/mailu/manage.py | 23 +++++++++++++++++------ 1 file changed, 17 insertions(+), 6 deletions(-) diff --git a/core/admin/mailu/manage.py b/core/admin/mailu/manage.py index b25ab568..44be509c 100644 --- a/core/admin/mailu/manage.py +++ b/core/admin/mailu/manage.py @@ -300,10 +300,13 @@ def config_update(verbose=False, delete_objects=False, dry_run=False, file=None) @mailu.command() @click.option('-f', '--full', is_flag=True, help='Include default attributes') -@click.option('-s', '--secrets', is_flag=True, help='Include secrets (plain-text / not hashed)') +@click.option('-s', '--secrets', is_flag=True, help='Include secrets (dkim-key, plain-text / not hashed)') +@click.argument('sections', nargs=-1) @flask_cli.with_appcontext -def config_dump(full=False, secrets=False): - """dump configuration as YAML-formatted data to stdout""" +def config_dump(full=False, secrets=False, sections=None): + """dump configuration as YAML-formatted data to stdout + valid SECTIONS are: domains, relays, users, aliases + """ class spacedDumper(yaml.Dumper): @@ -315,11 +318,19 @@ def config_dump(full=False, secrets=False): def increase_indent(self, flow=False, indentless=False): return super().increase_indent(flow, False) + if sections: + check = dict(yaml_sections) + for section in sections: + if section not in check: + print(f'[ERROR] Invalid section: {section}') + return 1 + config = {} for section, model in yaml_sections: - dump = [item.to_dict(full, secrets) for item in model.query.all()] - if len(dump): - config[section] = dump + if not sections or section in sections: + dump = [item.to_dict(full, secrets) for item in model.query.all()] + if len(dump): + config[section] = dump yaml.dump(config, sys.stdout, Dumper=spacedDumper, default_flow_style=False, allow_unicode=True) From adc9c70c3e5b0f2f5c37ae298fd92d5d903c6d94 Mon Sep 17 00:00:00 2001 From: Alexander Graf Date: Sat, 24 Oct 2020 22:31:32 +0200 Subject: [PATCH 031/596] added dump option to dump dns data of domains --- core/admin/mailu/manage.py | 14 +++++--- core/admin/mailu/models.py | 72 ++++++++++++++++++++++++++++++-------- 2 files changed, 68 insertions(+), 18 deletions(-) diff --git a/core/admin/mailu/manage.py b/core/admin/mailu/manage.py index 44be509c..42d5b5f7 100644 --- a/core/admin/mailu/manage.py +++ b/core/admin/mailu/manage.py @@ -301,11 +301,13 @@ def config_update(verbose=False, delete_objects=False, dry_run=False, file=None) @mailu.command() @click.option('-f', '--full', is_flag=True, help='Include default attributes') @click.option('-s', '--secrets', is_flag=True, help='Include secrets (dkim-key, plain-text / not hashed)') +@click.option('-d', '--dns', is_flag=True, help='Include dns records') @click.argument('sections', nargs=-1) @flask_cli.with_appcontext -def config_dump(full=False, secrets=False, sections=None): - """dump configuration as YAML-formatted data to stdout - valid SECTIONS are: domains, relays, users, aliases +def config_dump(full=False, secrets=False, dns=False, sections=None): + """dump configuration as YAML-formatted data to stdout + + SECTIONS can be: domains, relays, users, aliases """ class spacedDumper(yaml.Dumper): @@ -325,10 +327,14 @@ def config_dump(full=False, secrets=False, sections=None): print(f'[ERROR] Invalid section: {section}') return 1 + extra = [] + if dns: + extra.append('dns') + config = {} for section, model in yaml_sections: if not sections or section in sections: - dump = [item.to_dict(full, secrets) for item in model.query.all()] + dump = [item.to_dict(full, secrets, extra) for item in model.query.all()] if len(dump): config[section] = dump diff --git a/core/admin/mailu/models.py b/core/admin/mailu/models.py index 6c82baa0..79ce49ab 100644 --- a/core/admin/mailu/models.py +++ b/core/admin/mailu/models.py @@ -119,7 +119,7 @@ class Base(db.Model): def _dict_pval(self): return getattr(self, self._dict_pkey()) - def to_dict(self, full=False, include_secrets=False, recursed=False, hide=None): + def to_dict(self, full=False, include_secrets=False, include_extra=None, recursed=False, hide=None): """ Return a dictionary representation of this model. """ @@ -136,9 +136,17 @@ class Base(db.Model): convert = getattr(self, '_dict_output', {}) + extra_keys = getattr(self, '_dict_extra', {}) + if include_extra is None: + include_extra = [] + res = {} - for key in itertools.chain(self.__table__.columns.keys(), getattr(self, '_dict_show', [])): + for key in itertools.chain( + self.__table__.columns.keys(), + getattr(self, '_dict_show', []), + *[extra_keys.get(extra, []) for extra in include_extra] + ): if key in hide: continue if key in self.__table__.columns: @@ -167,14 +175,14 @@ class Base(db.Model): if key in secret: res[key] = '' else: - res[key] = [item.to_dict(full, include_secrets, True) for item in items] + res[key] = [item.to_dict(full, include_secrets, include_extra, True) for item in items] else: value = getattr(self, key) if full or value is not None: if key in secret: res[key] = '' else: - res[key] = value.to_dict(full, include_secrets, True) + res[key] = value.to_dict(full, include_secrets, include_extra, True) return res @@ -219,7 +227,7 @@ class Base(db.Model): if rel is None: itype = getattr(model, '_dict_types', {}).get(key) if itype is not None: - if not itype: # empty tuple => ignore value + if itype is False: # ignore value del data[key] continue elif not isinstance(value, itype): @@ -291,11 +299,11 @@ class Base(db.Model): # delete referenced items missing in yaml rel_pkey = rel_model._dict_pkey() - new_data = list([i.to_dict(True, True, True, [rel_pkey]) for i in new]) + new_data = list([i.to_dict(True, True, None, True, [rel_pkey]) for i in new]) for rel_item in old: if rel_item not in new: # check if item with same data exists to stabilze import without primary key - rel_data = rel_item.to_dict(True, True, True, [rel_pkey]) + rel_data = rel_item.to_dict(True, True, None, True, [rel_pkey]) try: same_idx = new_data.index(rel_data) except ValueError: @@ -367,10 +375,18 @@ class Domain(Base): __tablename__ = "domain" _dict_hide = {'users', 'managers', 'aliases'} - _dict_show = {'dkim_key', 'dkim_publickey'} + _dict_show = {'dkim_key'} + _dict_extra = {'dns':{'dkim_publickey', 'dns_mx', 'dns_spf', 'dns_dkim', 'dns_dmarc'}} _dict_secret = {'dkim_key'} - _dict_types = {'dkim_key': (bytes, type(None)), 'dkim_publickey': tuple()} - _dict_output = {'dkim_key': lambda v: v.decode('utf-8').strip().split('\n')[1:-1]} + _dict_types = { + 'dkim_key': (bytes, type(None)), + 'dkim_publickey': False, + 'dns_mx': False, + 'dns_spf': False, + 'dns_dkim': False, + 'dns_dmarc': False, + } + _dict_output = {'dkim_key': lambda key: key.decode('utf-8').strip().split('\n')[1:-1]} @staticmethod def _dict_input(data): if 'dkim_key' in data: @@ -408,18 +424,46 @@ class Domain(Base): max_quota_bytes = db.Column(db.BigInteger(), nullable=False, default=0) signup_enabled = db.Column(db.Boolean(), nullable=False, default=False) + def _dkim_file(self): + return app.config["DKIM_PATH"].format( + domain=self.name, selector=app.config["DKIM_SELECTOR"]) + + @property + def dns_mx(self): + hostname = app.config['HOSTNAMES'].split(',')[0] + return f'{self.name}. 600 IN MX 10 {hostname}.' + + @property + def dns_spf(self): + hostname = app.config['HOSTNAMES'].split(',')[0] + return f'{self.name}. 600 IN TXT "v=spf1 mx a:{hostname} ~all"' + + @property + def dns_dkim(self): + if os.path.exists(self._dkim_file()): + selector = app.config['DKIM_SELECTOR'] + return f'{selector}._domainkey.{self.name}. 600 IN TXT "v=DKIM1; k=rsa; p={self.dkim_publickey}"' + + @property + def dns_dmarc(self): + if os.path.exists(self._dkim_file()): + domain = app.config['DOMAIN'] + rua = app.config['DMARC_RUA'] + rua = f' rua=mailto:{rua}@{domain};' if rua else '' + ruf = app.config['DMARC_RUF'] + ruf = f' ruf=mailto:{ruf}@{domain};' if ruf else '' + return f'_dmarc.{self.name}. 600 IN TXT "v=DMARC1; p=reject;{rua}{ruf} adkim=s; aspf=s"' + @property def dkim_key(self): - file_path = app.config["DKIM_PATH"].format( - domain=self.name, selector=app.config["DKIM_SELECTOR"]) + file_path = self._dkim_file() if os.path.exists(file_path): with open(file_path, "rb") as handle: return handle.read() @dkim_key.setter def dkim_key(self, value): - file_path = app.config["DKIM_PATH"].format( - domain=self.name, selector=app.config["DKIM_SELECTOR"]) + file_path = self._dkim_file() if value is None: if os.path.exists(file_path): os.unlink(file_path) From 0a907a744e1027f8e2b79b6fce54fc03e27b69f8 Mon Sep 17 00:00:00 2001 From: Alexander Graf Date: Sat, 24 Oct 2020 22:32:08 +0200 Subject: [PATCH 032/596] updated documentation for config-dump --- docs/cli.rst | 13 ++++++++++--- 1 file changed, 10 insertions(+), 3 deletions(-) diff --git a/docs/cli.rst b/docs/cli.rst index 0036e504..1b2ed14f 100644 --- a/docs/cli.rst +++ b/docs/cli.rst @@ -78,22 +78,29 @@ The purpose of this command is to dump domain-, relay-, alias- and user-configur # docker-compose exec admin flask mailu config-dump --help - Usage: flask mailu config-dump [OPTIONS] + Usage: flask mailu config-dump [OPTIONS] [SECTIONS]... dump configuration as YAML-formatted data to stdout + SECTIONS can be: domains, relays, users, aliases + Options: -f, --full Include default attributes - -s, --secrets Include secrets (plain-text / not hashed) + -s, --secrets Include secrets (dkim-key, plain-text / not hashed) + -d, --dns Include dns records --help Show this message and exit. -If you want to export secrets (plain-text / not hashed) you have to add the ``--secrets`` option. +If you want to export secrets (dkim-keys, plain-text / not hashed) you have to add the ``--secrets`` option. Only non-default attributes are dumped. If you want to dump all attributes use ``--full``. +To include dns records (mx, spf, dkim and dmarc) add the ``--dns`` option. +Unless you specify some sections all sections are dumped by default. .. code-block:: bash docker-compose exec admin flask mailu config-dump > mail-config.yml + docker-compose exec admin flask mailu config-dump --dns domains + config-update ------------- From 32f6a23a95de6d5ee2d64eecc452469967289fdf Mon Sep 17 00:00:00 2001 From: cbachert Date: Fri, 30 Oct 2020 17:12:34 +0000 Subject: [PATCH 033/596] Remove rspamd unused env var from start script Environment variable FRONT_ADDRESS is unused in rspamd FRONT_ADDRESS references were removed with commit 8172f3e in PR #727 --- core/rspamd/start.py | 1 - 1 file changed, 1 deletion(-) diff --git a/core/rspamd/start.py b/core/rspamd/start.py index bde708f2..e2e72bcb 100755 --- a/core/rspamd/start.py +++ b/core/rspamd/start.py @@ -10,7 +10,6 @@ log.basicConfig(stream=sys.stderr, level=os.environ.get("LOG_LEVEL", "WARNING")) # Actual startup script -os.environ["FRONT_ADDRESS"] = system.get_host_address_from_environment("FRONT", "front") os.environ["REDIS_ADDRESS"] = system.get_host_address_from_environment("REDIS", "redis") if os.environ.get("ANTIVIRUS") == 'clamav': From d63ca857b415036dcd7e72ff2ea26cc8993f27ea Mon Sep 17 00:00:00 2001 From: lub Date: Sat, 21 Nov 2020 00:55:28 +0100 Subject: [PATCH 034/596] update rainloop to php 7.4 --- webmails/rainloop/Dockerfile | 4 ++-- 1 file changed, 2 insertions(+), 2 deletions(-) diff --git a/webmails/rainloop/Dockerfile b/webmails/rainloop/Dockerfile index c67c7496..ee040f25 100644 --- a/webmails/rainloop/Dockerfile +++ b/webmails/rainloop/Dockerfile @@ -2,10 +2,10 @@ ARG ARCH="" ARG QEMU=other # NOTE: only add file if building for arm -FROM ${ARCH}php:7.3-apache as build_arm +FROM ${ARCH}php:7.4-apache as build_arm ONBUILD COPY --from=balenalib/rpi-alpine:3.10 /usr/bin/qemu-arm-static /usr/bin/qemu-arm-static -FROM ${ARCH}php:7.3-apache as build_other +FROM ${ARCH}php:7.4-apache as build_other FROM build_${QEMU} #Shared layer between rainloop and roundcube From 8dd5dac3ed6adecdd129aef42680ce3a04cd7413 Mon Sep 17 00:00:00 2001 From: lub Date: Sat, 21 Nov 2020 00:55:38 +0100 Subject: [PATCH 035/596] update roundcube to php 7.4 --- webmails/roundcube/Dockerfile | 4 ++-- 1 file changed, 2 insertions(+), 2 deletions(-) diff --git a/webmails/roundcube/Dockerfile b/webmails/roundcube/Dockerfile index 79b911b0..1c303342 100644 --- a/webmails/roundcube/Dockerfile +++ b/webmails/roundcube/Dockerfile @@ -1,10 +1,10 @@ # NOTE: only add file if building for arm ARG ARCH="" ARG QEMU=other -FROM ${ARCH}php:7.3-apache as build_arm +FROM ${ARCH}php:7.4-apache as build_arm ONBUILD COPY --from=balenalib/rpi-alpine:3.10 /usr/bin/qemu-arm-static /usr/bin/qemu-arm-static -FROM ${ARCH}php:7.3-apache as build_other +FROM ${ARCH}php:7.4-apache as build_other FROM build_${QEMU} #Shared layer between rainloop and roundcube From 98a6ffb497e0e368808614219c6b099dcc828c0b Mon Sep 17 00:00:00 2001 From: lub Date: Thu, 17 Sep 2020 19:33:55 +0200 Subject: [PATCH 036/596] add compression via xz and lz4 --- core/dovecot/conf/dovecot.conf | 2 +- docs/compose/.env | 2 +- setup/flavors/compose/mailu.env | 2 +- tests/compose/core/mailu.env | 2 +- tests/compose/fetchmail/mailu.env | 2 +- tests/compose/filters/mailu.env | 2 +- tests/compose/rainloop/mailu.env | 2 +- tests/compose/roundcube/mailu.env | 2 +- tests/compose/webdav/mailu.env | 2 +- 9 files changed, 9 insertions(+), 9 deletions(-) diff --git a/core/dovecot/conf/dovecot.conf b/core/dovecot/conf/dovecot.conf index 81811cdb..ab5cb43a 100644 --- a/core/dovecot/conf/dovecot.conf +++ b/core/dovecot/conf/dovecot.conf @@ -50,7 +50,7 @@ plugin { fts_autoindex_exclude = \Trash {% endif %} - {% if COMPRESSION in [ 'gz', 'bz2' ] %} + {% if COMPRESSION in [ 'gz', 'bz2', 'xz', 'lz4' ] %} zlib_save = {{ COMPRESSION }} {% endif %} diff --git a/docs/compose/.env b/docs/compose/.env index 7f91c270..b4a8b218 100644 --- a/docs/compose/.env +++ b/docs/compose/.env @@ -97,7 +97,7 @@ WELCOME_SUBJECT=Welcome to your new email account WELCOME_BODY=Welcome to your new email account, if you can read this, then it is configured properly! # Maildir Compression -# choose compression-method, default: none (value: bz2, gz) +# choose compression-method, default: none (value: gz, bz2, xz, lz4) COMPRESSION= # change compression-level, default: 6 (value: 1-9) COMPRESSION_LEVEL= diff --git a/setup/flavors/compose/mailu.env b/setup/flavors/compose/mailu.env index 44452e36..04148b40 100644 --- a/setup/flavors/compose/mailu.env +++ b/setup/flavors/compose/mailu.env @@ -86,7 +86,7 @@ WELCOME_SUBJECT={{ welcome_subject or 'Welcome to your new email account' }} WELCOME_BODY={{ welcome_body or 'Welcome to your new email account, if you can read this, then it is configured properly!' }} # Maildir Compression -# choose compression-method, default: none (value: bz2, gz) +# choose compression-method, default: none (value: gz, bz2, xz, lz4) COMPRESSION={{ compression }} # change compression-level, default: 6 (value: 1-9) COMPRESSION_LEVEL={{ compression_level }} diff --git a/tests/compose/core/mailu.env b/tests/compose/core/mailu.env index b13e57c5..edea6a5c 100644 --- a/tests/compose/core/mailu.env +++ b/tests/compose/core/mailu.env @@ -92,7 +92,7 @@ DMARC_RUF=admin # Maildir Compression -# choose compression-method, default: none (value: bz2, gz) +# choose compression-method, default: none (value: gz, bz2, xz, lz4) COMPRESSION= # change compression-level, default: 6 (value: 1-9) COMPRESSION_LEVEL= diff --git a/tests/compose/fetchmail/mailu.env b/tests/compose/fetchmail/mailu.env index 636a09a9..4a53ec46 100644 --- a/tests/compose/fetchmail/mailu.env +++ b/tests/compose/fetchmail/mailu.env @@ -92,7 +92,7 @@ DMARC_RUF=admin # Maildir Compression -# choose compression-method, default: none (value: bz2, gz) +# choose compression-method, default: none (value: gz, bz2, xz, lz4) COMPRESSION= # change compression-level, default: 6 (value: 1-9) COMPRESSION_LEVEL= diff --git a/tests/compose/filters/mailu.env b/tests/compose/filters/mailu.env index b6d5ca8f..1106deb0 100644 --- a/tests/compose/filters/mailu.env +++ b/tests/compose/filters/mailu.env @@ -92,7 +92,7 @@ DMARC_RUF=admin # Maildir Compression -# choose compression-method, default: none (value: bz2, gz) +# choose compression-method, default: none (value: gz, bz2, xz, lz4) COMPRESSION= # change compression-level, default: 6 (value: 1-9) COMPRESSION_LEVEL= diff --git a/tests/compose/rainloop/mailu.env b/tests/compose/rainloop/mailu.env index 9c31c8bb..d02b98f2 100644 --- a/tests/compose/rainloop/mailu.env +++ b/tests/compose/rainloop/mailu.env @@ -92,7 +92,7 @@ DMARC_RUF=admin # Maildir Compression -# choose compression-method, default: none (value: bz2, gz) +# choose compression-method, default: none (value: gz, bz2, xz, lz4) COMPRESSION= # change compression-level, default: 6 (value: 1-9) COMPRESSION_LEVEL= diff --git a/tests/compose/roundcube/mailu.env b/tests/compose/roundcube/mailu.env index dc503268..e1005487 100644 --- a/tests/compose/roundcube/mailu.env +++ b/tests/compose/roundcube/mailu.env @@ -92,7 +92,7 @@ DMARC_RUF=admin # Maildir Compression -# choose compression-method, default: none (value: bz2, gz) +# choose compression-method, default: none (value: gz, bz2, xz, lz4) COMPRESSION= # change compression-level, default: 6 (value: 1-9) COMPRESSION_LEVEL= diff --git a/tests/compose/webdav/mailu.env b/tests/compose/webdav/mailu.env index 90fb25b1..58b9810a 100644 --- a/tests/compose/webdav/mailu.env +++ b/tests/compose/webdav/mailu.env @@ -92,7 +92,7 @@ DMARC_RUF=admin # Maildir Compression -# choose compression-method, default: none (value: bz2, gz) +# choose compression-method, default: none (value: gz, bz2, xz, lz4) COMPRESSION= # change compression-level, default: 6 (value: 1-9) COMPRESSION_LEVEL= From 2606ace1dfddcc32c00b640a95300b9a0c820635 Mon Sep 17 00:00:00 2001 From: lub Date: Sat, 21 Nov 2020 12:39:42 +0100 Subject: [PATCH 037/596] add changelog for #1694 --- towncrier/newsfragments/1694.feature | 1 + 1 file changed, 1 insertion(+) create mode 100644 towncrier/newsfragments/1694.feature diff --git a/towncrier/newsfragments/1694.feature b/towncrier/newsfragments/1694.feature new file mode 100644 index 00000000..41548707 --- /dev/null +++ b/towncrier/newsfragments/1694.feature @@ -0,0 +1 @@ +Support configuring xz and lz4 compression for dovecot. From 0051b93077530c9950973ce2a50fea725a2b110e Mon Sep 17 00:00:00 2001 From: Alexander Graf Date: Wed, 16 Dec 2020 22:39:50 +0100 Subject: [PATCH 038/596] removed unused variable --- core/admin/mailu/models.py | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/core/admin/mailu/models.py b/core/admin/mailu/models.py index 79ce49ab..ab9a8329 100644 --- a/core/admin/mailu/models.py +++ b/core/admin/mailu/models.py @@ -323,7 +323,7 @@ class Base(db.Model): new_data.append(rel_data) new.remove(same) del new_data[same_idx] - for i, (ch_item, ch_update) in enumerate(changed): + for i, (ch_item, _) in enumerate(changed): if ch_item is same: changed[i] = (rel_item, []) db.session.flush() From 3064a1dcff573f6dea99a90cc5b545bc6c3d2670 Mon Sep 17 00:00:00 2001 From: Alexander Graf Date: Sun, 20 Dec 2020 23:38:55 +0100 Subject: [PATCH 039/596] removed call to (undefined) cli --- core/admin/mailu/manage.py | 4 ---- 1 file changed, 4 deletions(-) diff --git a/core/admin/mailu/manage.py b/core/admin/mailu/manage.py index 42d5b5f7..f70c5c85 100644 --- a/core/admin/mailu/manage.py +++ b/core/admin/mailu/manage.py @@ -416,7 +416,3 @@ def setmanager(domain_name, user_name='manager'): domain.managers.append(manageruser) db.session.add(domain) db.session.commit() - - -if __name__ == '__main__': - cli() From 0a594aaa2ca9ca1964d03e914e636a7be5296e10 Mon Sep 17 00:00:00 2001 From: Alexander Graf Date: Sun, 20 Dec 2020 23:45:27 +0100 Subject: [PATCH 040/596] cosmetic changes --- core/admin/mailu/models.py | 83 ++++++++++++++++++++------------------ 1 file changed, 44 insertions(+), 39 deletions(-) diff --git a/core/admin/mailu/models.py b/core/admin/mailu/models.py index ab9a8329..a752000e 100644 --- a/core/admin/mailu/models.py +++ b/core/admin/mailu/models.py @@ -30,7 +30,7 @@ class IdnaDomain(db.TypeDecorator): impl = db.String(80) def process_bind_param(self, value, dialect): - return idna.encode(value).decode("ascii").lower() + return idna.encode(value).decode('ascii').lower() def process_result_value(self, value, dialect): return idna.decode(value) @@ -46,7 +46,7 @@ class IdnaEmail(db.TypeDecorator): def process_bind_param(self, value, dialect): try: localpart, domain_name = value.split('@') - return "{0}@{1}".format( + return '{0}@{1}'.format( localpart, idna.encode(domain_name).decode('ascii'), ).lower() @@ -55,7 +55,7 @@ class IdnaEmail(db.TypeDecorator): def process_result_value(self, value, dialect): localpart, domain_name = value.split('@') - return "{0}@{1}".format( + return '{0}@{1}'.format( localpart, idna.decode(domain_name), ) @@ -70,14 +70,14 @@ class CommaSeparatedList(db.TypeDecorator): def process_bind_param(self, value, dialect): if not isinstance(value, (list, set)): - raise TypeError("Should be a list") + raise TypeError('Should be a list') for item in value: - if "," in item: - raise ValueError("Item must not contain a comma") - return ",".join(sorted(value)) + if ',' in item: + raise ValueError('Item must not contain a comma') + return ','.join(sorted(value)) def process_result_value(self, value, dialect): - return list(filter(bool, value.split(","))) if value else [] + return list(filter(bool, value.split(','))) if value else [] python_type = list @@ -103,8 +103,8 @@ class Base(db.Model): metadata = sqlalchemy.schema.MetaData( naming_convention={ - "fk": "%(table_name)s_%(column_0_name)s_fkey", - "pk": "%(table_name)s_pkey" + 'fk': '%(table_name)s_%(column_0_name)s_fkey', + 'pk': '%(table_name)s_pkey' } ) @@ -227,7 +227,7 @@ class Base(db.Model): if rel is None: itype = getattr(model, '_dict_types', {}).get(key) if itype is not None: - if itype is False: # ignore value + if itype is False: # ignore value. TODO: emit warning? del data[key] continue elif not isinstance(value, itype): @@ -372,7 +372,8 @@ class Config(Base): class Domain(Base): """ A DNS domain that has mail addresses associated to it. """ - __tablename__ = "domain" + + __tablename__ = 'domain' _dict_hide = {'users', 'managers', 'aliases'} _dict_show = {'dkim_key'} @@ -425,8 +426,8 @@ class Domain(Base): signup_enabled = db.Column(db.Boolean(), nullable=False, default=False) def _dkim_file(self): - return app.config["DKIM_PATH"].format( - domain=self.name, selector=app.config["DKIM_SELECTOR"]) + return app.config['DKIM_PATH'].format( + domain=self.name, selector=app.config['DKIM_SELECTOR']) @property def dns_mx(self): @@ -458,7 +459,7 @@ class Domain(Base): def dkim_key(self): file_path = self._dkim_file() if os.path.exists(file_path): - with open(file_path, "rb") as handle: + with open(file_path, 'rb') as handle: return handle.read() @dkim_key.setter @@ -468,14 +469,14 @@ class Domain(Base): if os.path.exists(file_path): os.unlink(file_path) else: - with open(file_path, "wb") as handle: + with open(file_path, 'wb') as handle: handle.write(value) @property def dkim_publickey(self): dkim_key = self.dkim_key if dkim_key: - return dkim.strip_key(self.dkim_key).decode("utf8") + return dkim.strip_key(self.dkim_key).decode('utf8') def generate_dkim_key(self): self.dkim_key = dkim.gen_key() @@ -512,7 +513,7 @@ class Alternative(Base): The name "domain alias" was avoided to prevent some confusion. """ - __tablename__ = "alternative" + __tablename__ = 'alternative' name = db.Column(IdnaDomain, primary_key=True, nullable=False) domain_name = db.Column(IdnaDomain, db.ForeignKey(Domain.name)) @@ -528,7 +529,7 @@ class Relay(Base): The domain is either relayed publicly or through a specified SMTP host. """ - __tablename__ = "relay" + __tablename__ = 'relay' _dict_mandatory = {'smtp'} @@ -553,7 +554,7 @@ class Email(object): elif type(data['email']) is str: data['localpart'], data['domain'] = data['email'].rsplit('@', 1) else: - data['email'] = f"{data['localpart']}@{data['domain']}" + data['email'] = f'{data["localpart"]}@{data["domain"]}' @declarative.declared_attr def domain_name(cls): @@ -565,9 +566,9 @@ class Email(object): # especially when the mail server is reading the database. @declarative.declared_attr def email(cls): - updater = lambda context: "{0}@{1}".format( - context.current_parameters["localpart"], - context.current_parameters["domain_name"], + updater = lambda context: '{0}@{1}'.format( + context.current_parameters['localpart'], + context.current_parameters['domain_name'], ) return db.Column(IdnaEmail, primary_key=True, nullable=False, @@ -576,12 +577,12 @@ class Email(object): def sendmail(self, subject, body): """ Send an email to the address. """ - from_address = "{0}@{1}".format( + from_address = '{0}@{1}'.format( app.config['POSTMASTER'], idna.encode(app.config['DOMAIN']).decode('ascii'), ) with smtplib.SMTP(app.config['HOST_AUTHSMTP'], port=10025) as smtp: - to_address = "{0}@{1}".format( + to_address = '{0}@{1}'.format( self.localpart, idna.encode(self.domain_name).decode('ascii'), ) @@ -638,7 +639,8 @@ class Email(object): class User(Base, Email): """ A user is an email address that has a password to access a mailbox. """ - __tablename__ = "user" + + __tablename__ = 'user' _dict_hide = {'domain_name', 'domain', 'localpart', 'quota_bytes_used'} _dict_mandatory = {'localpart', 'domain', 'password'} @@ -689,7 +691,7 @@ class User(Base, Email): default=date(2999, 12, 31)) # Settings - displayed_name = db.Column(db.String(160), nullable=False, default="") + displayed_name = db.Column(db.String(160), nullable=False, default='') spam_enabled = db.Column(db.Boolean(), nullable=False, default=True) spam_threshold = db.Column(db.Integer(), nullable=False, default=80) @@ -720,12 +722,12 @@ class User(Base, Email): self.reply_enddate > now ) - scheme_dict = {'PBKDF2': "pbkdf2_sha512", - 'BLF-CRYPT': "bcrypt", - 'SHA512-CRYPT': "sha512_crypt", - 'SHA256-CRYPT': "sha256_crypt", - 'MD5-CRYPT': "md5_crypt", - 'CRYPT': "des_crypt"} + scheme_dict = {'PBKDF2': 'pbkdf2_sha512', + 'BLF-CRYPT': 'bcrypt', + 'SHA512-CRYPT': 'sha512_crypt', + 'SHA256-CRYPT': 'sha256_crypt', + 'MD5-CRYPT': 'md5_crypt', + 'CRYPT': 'des_crypt'} def get_password_context(self): return context.CryptContext( @@ -745,7 +747,7 @@ class User(Base, Email): def set_password(self, password, hash_scheme=None, raw=False): """Set password for user with specified encryption scheme - @password: plain text password to encrypt (if raw == True the hash itself) + @password: plain text password to encrypt (if raw == True the hash itself) """ if hash_scheme is None: hash_scheme = app.config['PASSWORD_SCHEME'] @@ -787,7 +789,8 @@ class User(Base, Email): class Alias(Base, Email): """ An alias is an email address that redirects to some destination. """ - __tablename__ = "alias" + + __tablename__ = 'alias' _dict_hide = {'domain_name', 'domain', 'localpart'} @staticmethod @@ -813,7 +816,7 @@ class Alias(Base, Email): cls.localpart == localpart ), sqlalchemy.and_( cls.wildcard == True, - sqlalchemy.bindparam("l", localpart).like(cls.localpart) + sqlalchemy.bindparam('l', localpart).like(cls.localpart) ) ) ) @@ -828,7 +831,7 @@ class Alias(Base, Email): sqlalchemy.func.lower(cls.localpart) == localpart_lower ), sqlalchemy.and_( cls.wildcard == True, - sqlalchemy.bindparam("l", localpart_lower).like(sqlalchemy.func.lower(cls.localpart)) + sqlalchemy.bindparam('l', localpart_lower).like(sqlalchemy.func.lower(cls.localpart)) ) ) ) @@ -849,7 +852,8 @@ class Alias(Base, Email): class Token(Base): """ A token is an application password for a given user. """ - __tablename__ = "token" + + __tablename__ = 'token' _dict_recurse = True _dict_hide = {'user', 'user_email'} @@ -877,7 +881,8 @@ class Fetch(Base): """ A fetched account is a remote POP/IMAP account fetched into a local account. """ - __tablename__ = "fetch" + + __tablename__ = 'fetch' _dict_recurse = True _dict_hide = {'user_email', 'user', 'last_check', 'error'} From 815f47667bbd940c136175f73b1b10a8157c9075 Mon Sep 17 00:00:00 2001 From: Alexander Graf Date: Sun, 20 Dec 2020 23:49:42 +0100 Subject: [PATCH 041/596] update dkim-key on commit only --- core/admin/mailu/models.py | 42 ++++++++++++++++++++++++++++---------- 1 file changed, 31 insertions(+), 11 deletions(-) diff --git a/core/admin/mailu/models.py b/core/admin/mailu/models.py index a752000e..d38c0ad9 100644 --- a/core/admin/mailu/models.py +++ b/core/admin/mailu/models.py @@ -369,6 +369,21 @@ class Config(Base): value = db.Column(JSONEncoded) +@sqlalchemy.event.listens_for(db.session, 'after_commit') +def store_dkim_key(session): + """ Store DKIM key on commit + """ + + for obj in session.identity_map.values(): + if isinstance(obj, Domain): + if obj._dkim_key_changed: + file_path = obj._dkim_file() + if obj._dkim_key: + with open(file_path, 'wb') as handle: + handle.write(obj._dkim_key) + elif os.path.exists(file_path): + os.unlink(file_path) + class Domain(Base): """ A DNS domain that has mail addresses associated to it. """ @@ -424,6 +439,9 @@ class Domain(Base): max_aliases = db.Column(db.Integer, nullable=False, default=-1) max_quota_bytes = db.Column(db.BigInteger(), nullable=False, default=0) signup_enabled = db.Column(db.Boolean(), nullable=False, default=False) + + _dkim_key = None + _dkim_key_changed = False def _dkim_file(self): return app.config['DKIM_PATH'].format( @@ -457,26 +475,28 @@ class Domain(Base): @property def dkim_key(self): - file_path = self._dkim_file() - if os.path.exists(file_path): - with open(file_path, 'rb') as handle: - return handle.read() + if self._dkim_key is None: + file_path = self._dkim_file() + if os.path.exists(file_path): + with open(file_path, 'rb') as handle: + self._dkim_key = handle.read() + else: + self._dkim_key = b'' + return self._dkim_key if self._dkim_key else None @dkim_key.setter def dkim_key(self, value): - file_path = self._dkim_file() + old_key = self.dkim_key if value is None: - if os.path.exists(file_path): - os.unlink(file_path) - else: - with open(file_path, 'wb') as handle: - handle.write(value) + value = b'' + self._dkim_key_changed = value != old_key + self._dkim_key = value @property def dkim_publickey(self): dkim_key = self.dkim_key if dkim_key: - return dkim.strip_key(self.dkim_key).decode('utf8') + return dkim.strip_key(dkim_key).decode('utf8') def generate_dkim_key(self): self.dkim_key = dkim.gen_key() From 3b35180b41ee298735946d3fbc3bab34f686ccbd Mon Sep 17 00:00:00 2001 From: Alexander Graf Date: Sun, 20 Dec 2020 23:50:26 +0100 Subject: [PATCH 042/596] cosmetic changes --- core/admin/mailu/models.py | 8 ++++---- 1 file changed, 4 insertions(+), 4 deletions(-) diff --git a/core/admin/mailu/models.py b/core/admin/mailu/models.py index d38c0ad9..13ebce60 100644 --- a/core/admin/mailu/models.py +++ b/core/admin/mailu/models.py @@ -70,7 +70,7 @@ class CommaSeparatedList(db.TypeDecorator): def process_bind_param(self, value, dialect): if not isinstance(value, (list, set)): - raise TypeError('Should be a list') + raise TypeError('Must be a list') for item in value: if ',' in item: raise ValueError('Item must not contain a comma') @@ -792,9 +792,9 @@ class User(Base, Email): return emails def send_welcome(self): - if app.config["WELCOME"]: - self.sendmail(app.config["WELCOME_SUBJECT"], - app.config["WELCOME_BODY"]) + if app.config['WELCOME']: + self.sendmail(app.config['WELCOME_SUBJECT'], + app.config['WELCOME_BODY']) @classmethod def get(cls, email): From 0b14fefb1acef5b7f8dccce56a1f94c8ad3854e3 Mon Sep 17 00:00:00 2001 From: =?UTF-8?q?Tomasz=20W=C3=B3jcik?= Date: Mon, 21 Dec 2020 00:19:27 +0100 Subject: [PATCH 043/596] fix typo in faq.rst --- docs/faq.rst | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/docs/faq.rst b/docs/faq.rst index b292cd05..4037800a 100644 --- a/docs/faq.rst +++ b/docs/faq.rst @@ -129,7 +129,7 @@ So when you have something like this: - The admin interface generates ``MX`` and ``SPF`` examples which point to the first entry of ``HOSTNAMES`` but these are only examples. You can modify them to use any other ``HOSTNAMES`` entry. -You're mail service will be reachable for IMAP, POP3, SMTP and Webmail at the addresses: +Your mail service will be reachable for IMAP, POP3, SMTP and Webmail at the addresses: - mail.example.com - mail.foo.com From bee0261dd8dd802e79ade31bc9e3a98f8c8a55b9 Mon Sep 17 00:00:00 2001 From: Stephan Holl Date: Wed, 23 Dec 2020 18:48:11 +0100 Subject: [PATCH 044/596] Add details for postfix-overrides --- docs/faq.rst | 5 ++++- 1 file changed, 4 insertions(+), 1 deletion(-) diff --git a/docs/faq.rst b/docs/faq.rst index b292cd05..0a171bc9 100644 --- a/docs/faq.rst +++ b/docs/faq.rst @@ -257,7 +257,10 @@ Postfix, Dovecot, Nginx and Rspamd support overriding configuration files. Overr ``$ROOT/overrides``. Please refer to the official documentation of those programs for the correct syntax. The following file names will be taken as override configuration: -- `Postfix`_ - ``postfix.cf`` in postfix sub-directory; +- `Postfix`_ : + - ``postfix.cf`` as ``/overrides/postfix.cf`` + - ``master.cf`` as ``/overrides/postfix.master`` + - All ``/overrides/*.map`` files - `Dovecot`_ - ``dovecot.conf`` in dovecot sub-directory; - `Nginx`_ - All ``*.conf`` files in the ``nginx`` sub-directory; - `Rspamd`_ - All files in the ``rspamd`` sub-directory. From 6ba40bc0d79e82d55b34694fd62093d88c470617 Mon Sep 17 00:00:00 2001 From: Stephan Holl Date: Wed, 23 Dec 2020 18:53:56 +0100 Subject: [PATCH 045/596] Add newsfragment --- towncrier/newsfragments/1712.misc | 1 + 1 file changed, 1 insertion(+) create mode 100644 towncrier/newsfragments/1712.misc diff --git a/towncrier/newsfragments/1712.misc b/towncrier/newsfragments/1712.misc new file mode 100644 index 00000000..57c5a3b8 --- /dev/null +++ b/towncrier/newsfragments/1712.misc @@ -0,0 +1 @@ +This adds more details about the postfix-override possibilities (fixes #1628) From a1a527f201c25ab553ea3d41f37c70a390abe340 Mon Sep 17 00:00:00 2001 From: Grace <30454698+c4lliope@users.noreply.github.com> Date: Mon, 4 Jan 2021 02:37:48 -0500 Subject: [PATCH 046/596] Warn people off of the documentation's K8s recipe. Based on a discussion on Matrix (2020.12.19), the helm charts are the way to go. --- docs/kubernetes/mailu/index.rst | 4 ++++ 1 file changed, 4 insertions(+) diff --git a/docs/kubernetes/mailu/index.rst b/docs/kubernetes/mailu/index.rst index 5d3502a7..0af3942e 100644 --- a/docs/kubernetes/mailu/index.rst +++ b/docs/kubernetes/mailu/index.rst @@ -3,6 +3,10 @@ Kubernetes setup ================ +> Hold up! +> These instructions are not recommended for setting up Mailu in a production Kubernetes environment. +> Please see [the Helm Chart documentation](https://github.com/Mailu/helm-charts/blob/master/mailu/README.md). + Prequisites ----------- From 7229c89de1e1aec152f275d6e55a2077a810e40e Mon Sep 17 00:00:00 2001 From: Alexander Graf Date: Wed, 6 Jan 2021 16:31:03 +0100 Subject: [PATCH 047/596] ConfigManager should not replace app.config Updated ConfigManager to only modify app.config and not replace it. Swagger does not play well, when app.config is not a real dict and it is not necessary to keep ConfigManager around after init. Also added "API" flag to config (default: disabled). --- core/admin/mailu/configuration.py | 48 +++++++++---------------------- 1 file changed, 14 insertions(+), 34 deletions(-) diff --git a/core/admin/mailu/configuration.py b/core/admin/mailu/configuration.py index 2cf6a478..a4d3c069 100644 --- a/core/admin/mailu/configuration.py +++ b/core/admin/mailu/configuration.py @@ -51,6 +51,7 @@ DEFAULT_CONFIG = { 'WEBMAIL': 'none', 'RECAPTCHA_PUBLIC_KEY': '', 'RECAPTCHA_PRIVATE_KEY': '', + 'API': False, # Advanced settings 'PASSWORD_SCHEME': 'PBKDF2', 'LOG_LEVEL': 'WARNING', @@ -71,7 +72,7 @@ DEFAULT_CONFIG = { 'POD_ADDRESS_RANGE': None } -class ConfigManager(dict): +class ConfigManager: """ Naive configuration manager that uses environment only """ @@ -86,19 +87,16 @@ class ConfigManager(dict): def get_host_address(self, name): # if MYSERVICE_ADDRESS is defined, use this - if '{}_ADDRESS'.format(name) in os.environ: - return os.environ.get('{}_ADDRESS'.format(name)) + if f'{name}_ADDRESS' in os.environ: + return os.environ.get(f'{name}_ADDRESS') # otherwise use the host name and resolve it - return system.resolve_address(self.config['HOST_{}'.format(name)]) + return system.resolve_address(self.config[f'HOST_{name}']) def resolve_hosts(self): - self.config["IMAP_ADDRESS"] = self.get_host_address("IMAP") - self.config["POP3_ADDRESS"] = self.get_host_address("POP3") - self.config["AUTHSMTP_ADDRESS"] = self.get_host_address("AUTHSMTP") - self.config["SMTP_ADDRESS"] = self.get_host_address("SMTP") - self.config["REDIS_ADDRESS"] = self.get_host_address("REDIS") - if self.config["WEBMAIL"] != "none": - self.config["WEBMAIL_ADDRESS"] = self.get_host_address("WEBMAIL") + for key in ['IMAP', 'POP3', 'AUTHSMTP', 'SMTP', 'REDIS']: + self.config[f'{key}_ADDRESS'] = self.get_host_address(key) + if self.config['WEBMAIL'] != 'none': + self.config['WEBMAIL_ADDRESS'] = self.get_host_address('WEBMAIL') def __coerce_value(self, value): if isinstance(value, str) and value.lower() in ('true','yes'): @@ -108,6 +106,7 @@ class ConfigManager(dict): return value def init_app(self, app): + # get current app config self.config.update(app.config) # get environment variables self.config.update({ @@ -121,27 +120,8 @@ class ConfigManager(dict): template = self.DB_TEMPLATES[self.config['DB_FLAVOR']] self.config['SQLALCHEMY_DATABASE_URI'] = template.format(**self.config) - self.config['RATELIMIT_STORAGE_URL'] = 'redis://{0}/2'.format(self.config['REDIS_ADDRESS']) - self.config['QUOTA_STORAGE_URL'] = 'redis://{0}/1'.format(self.config['REDIS_ADDRESS']) - # update the app config itself - app.config = self + self.config['RATELIMIT_STORAGE_URL'] = f'redis://{self.config["REDIS_ADDRESS"]}/2' + self.config['QUOTA_STORAGE_URL'] = f'redis://{self.config["REDIS_ADDRESS"]}/1' - def setdefault(self, key, value): - if key not in self.config: - self.config[key] = value - return self.config[key] - - def get(self, *args): - return self.config.get(*args) - - def keys(self): - return self.config.keys() - - def __getitem__(self, key): - return self.config.get(key) - - def __setitem__(self, key, value): - self.config[key] = value - - def __contains__(self, key): - return key in self.config + # update the app config + app.config.update(self.config) From 4c258f5a6b8ba082f016cf186884b42e8a3d8549 Mon Sep 17 00:00:00 2001 From: Alexander Graf Date: Wed, 6 Jan 2021 16:45:55 +0100 Subject: [PATCH 048/596] cosmetic changes & make linter happy renamed single letter variables (m => match) renamed classmethod arguments to cls (model) removed shadowing of variables (hash, context) shortened unneeded lambda functions (id) converted type ... is to isinstance(...) removed unneded imports (flask) --- core/admin/mailu/manage.py | 33 ++++++------ core/admin/mailu/models.py | 107 ++++++++++++++++++------------------- 2 files changed, 69 insertions(+), 71 deletions(-) diff --git a/core/admin/mailu/manage.py b/core/admin/mailu/manage.py index f70c5c85..c07ca2b7 100644 --- a/core/admin/mailu/manage.py +++ b/core/admin/mailu/manage.py @@ -1,9 +1,8 @@ from mailu import models from flask import current_app as app -from flask import cli as flask_cli +from flask.cli import FlaskGroup, with_appcontext -import flask import os import socket import uuid @@ -15,14 +14,14 @@ import sys db = models.db -@click.group() -def mailu(cls=flask_cli.FlaskGroup): +@click.group(cls=FlaskGroup) +def mailu(): """ Mailu command line """ @mailu.command() -@flask_cli.with_appcontext +@with_appcontext def advertise(): """ Advertise this server against statistic services. """ @@ -45,7 +44,7 @@ def advertise(): @click.argument('domain_name') @click.argument('password') @click.option('-m', '--mode') -@flask_cli.with_appcontext +@with_appcontext def admin(localpart, domain_name, password, mode='create'): """ Create an admin user 'mode' can be: @@ -89,7 +88,7 @@ def admin(localpart, domain_name, password, mode='create'): @click.argument('domain_name') @click.argument('password') @click.argument('hash_scheme', required=False) -@flask_cli.with_appcontext +@with_appcontext def user(localpart, domain_name, password, hash_scheme=None): """ Create a user """ @@ -114,7 +113,7 @@ def user(localpart, domain_name, password, hash_scheme=None): @click.argument('domain_name') @click.argument('password') @click.argument('hash_scheme', required=False) -@flask_cli.with_appcontext +@with_appcontext def password(localpart, domain_name, password, hash_scheme=None): """ Change the password of an user """ @@ -134,7 +133,7 @@ def password(localpart, domain_name, password, hash_scheme=None): @click.option('-u', '--max-users') @click.option('-a', '--max-aliases') @click.option('-q', '--max-quota-bytes') -@flask_cli.with_appcontext +@with_appcontext def domain(domain_name, max_users=-1, max_aliases=-1, max_quota_bytes=0): """ Create a domain """ @@ -151,7 +150,7 @@ def domain(domain_name, max_users=-1, max_aliases=-1, max_quota_bytes=0): @click.argument('domain_name') @click.argument('password_hash') @click.argument('hash_scheme') -@flask_cli.with_appcontext +@with_appcontext def user_import(localpart, domain_name, password_hash, hash_scheme = None): """ Import a user along with password hash. """ @@ -183,7 +182,7 @@ yaml_sections = [ @click.option('-v', '--verbose', is_flag=True, help='Increase verbosity') @click.option('-d', '--delete-objects', is_flag=True, help='Remove objects not included in yaml') @click.option('-n', '--dry-run', is_flag=True, help='Perform a trial run with no changes made') -@flask_cli.with_appcontext +@with_appcontext def config_update(verbose=False, delete_objects=False, dry_run=False, file=None): """sync configuration with data from YAML-formatted stdin""" @@ -303,7 +302,7 @@ def config_update(verbose=False, delete_objects=False, dry_run=False, file=None) @click.option('-s', '--secrets', is_flag=True, help='Include secrets (dkim-key, plain-text / not hashed)') @click.option('-d', '--dns', is_flag=True, help='Include dns records') @click.argument('sections', nargs=-1) -@flask_cli.with_appcontext +@with_appcontext def config_dump(full=False, secrets=False, dns=False, sections=None): """dump configuration as YAML-formatted data to stdout @@ -343,7 +342,7 @@ def config_dump(full=False, secrets=False, dns=False, sections=None): @mailu.command() @click.argument('email') -@flask_cli.with_appcontext +@with_appcontext def user_delete(email): """delete user""" user = models.User.query.get(email) @@ -354,7 +353,7 @@ def user_delete(email): @mailu.command() @click.argument('email') -@flask_cli.with_appcontext +@with_appcontext def alias_delete(email): """delete alias""" alias = models.Alias.query.get(email) @@ -368,7 +367,7 @@ def alias_delete(email): @click.argument('domain_name') @click.argument('destination') @click.option('-w', '--wildcard', is_flag=True) -@flask_cli.with_appcontext +@with_appcontext def alias(localpart, domain_name, destination, wildcard=False): """ Create an alias """ @@ -392,7 +391,7 @@ def alias(localpart, domain_name, destination, wildcard=False): @click.argument('max_users') @click.argument('max_aliases') @click.argument('max_quota_bytes') -@flask_cli.with_appcontext +@with_appcontext def setlimits(domain_name, max_users, max_aliases, max_quota_bytes): """ Set domain limits """ @@ -407,7 +406,7 @@ def setlimits(domain_name, max_users, max_aliases, max_quota_bytes): @mailu.command() @click.argument('domain_name') @click.argument('user_name') -@flask_cli.with_appcontext +@with_appcontext def setmanager(domain_name, user_name='manager'): """ Make a user manager of a domain """ diff --git a/core/admin/mailu/models.py b/core/admin/mailu/models.py index 13ebce60..3bf92244 100644 --- a/core/admin/mailu/models.py +++ b/core/admin/mailu/models.py @@ -1,7 +1,6 @@ from mailu import dkim from sqlalchemy.ext import declarative -from passlib import context, hash from datetime import datetime, date from email.mime import text from flask import current_app as app @@ -12,6 +11,7 @@ import sqlalchemy import re import time import os +import passlib import glob import smtplib import idna @@ -113,8 +113,8 @@ class Base(db.Model): comment = db.Column(db.String(255), nullable=True) @classmethod - def _dict_pkey(model): - return model.__mapper__.primary_key[0].name + def _dict_pkey(cls): + return cls.__mapper__.primary_key[0].name def _dict_pval(self): return getattr(self, self._dict_pkey()) @@ -187,57 +187,57 @@ class Base(db.Model): return res @classmethod - def from_dict(model, data, delete=False): + def from_dict(cls, data, delete=False): changed = [] - pkey = model._dict_pkey() + pkey = cls._dict_pkey() # handle "primary key" only - if type(data) is not dict: + if isinstance(data, dict): data = {pkey: data} # modify input data - if hasattr(model, '_dict_input'): + if hasattr(cls, '_dict_input'): try: - model._dict_input(data) + cls._dict_input(data) except Exception as reason: - raise ValueError(f'{reason}', model, None, data) + raise ValueError(f'{reason}', cls, None, data) # check for primary key (if not recursed) - if not getattr(model, '_dict_recurse', False): + if not getattr(cls, '_dict_recurse', False): if not pkey in data: - raise KeyError(f'primary key {model.__table__}.{pkey} is missing', model, pkey, data) + raise KeyError(f'primary key {cls.__table__}.{pkey} is missing', cls, pkey, data) # check data keys and values for key in list(data.keys()): # check key - if not hasattr(model, key) and not key in model.__mapper__.relationships: - raise KeyError(f'unknown key {model.__table__}.{key}', model, key, data) + if not hasattr(cls, key) and not key in cls.__mapper__.relationships: + raise KeyError(f'unknown key {cls.__table__}.{key}', cls, key, data) # check value type value = data[key] - col = model.__mapper__.columns.get(key) + col = cls.__mapper__.columns.get(key) if col is not None: - if not ((value is None and col.nullable) or (type(value) is col.type.python_type)): - raise TypeError(f'{model.__table__}.{key} {value!r} has invalid type {type(value).__name__!r}', model, key, data) + if not ((value is None and col.nullable) or (isinstance(value, col.type.python_type))): + raise TypeError(f'{cls.__table__}.{key} {value!r} has invalid type {type(value).__name__!r}', cls, key, data) else: - rel = model.__mapper__.relationships.get(key) + rel = cls.__mapper__.relationships.get(key) if rel is None: - itype = getattr(model, '_dict_types', {}).get(key) + itype = getattr(cls, '_dict_types', {}).get(key) if itype is not None: if itype is False: # ignore value. TODO: emit warning? del data[key] continue elif not isinstance(value, itype): - raise TypeError(f'{model.__table__}.{key} {value!r} has invalid type {type(value).__name__!r}', model, key, data) + raise TypeError(f'{cls.__table__}.{key} {value!r} has invalid type {type(value).__name__!r}', cls, key, data) else: - raise NotImplementedError(f'type not defined for {model.__table__}.{key}') + raise NotImplementedError(f'type not defined for {cls.__table__}.{key}') # handle relationships - if key in model.__mapper__.relationships: - rel_model = model.__mapper__.relationships[key].argument + if key in cls.__mapper__.relationships: + rel_model = cls.__mapper__.relationships[key].argument if not isinstance(rel_model, sqlalchemy.orm.Mapper): add = rel_model.from_dict(value, delete) assert len(add) == 1 @@ -247,24 +247,24 @@ class Base(db.Model): # create item if necessary created = False - item = model.query.get(data[pkey]) if pkey in data else None + item = cls.query.get(data[pkey]) if pkey in data else None if item is None: # check for mandatory keys - missing = getattr(model, '_dict_mandatory', set()) - set(data.keys()) + missing = getattr(cls, '_dict_mandatory', set()) - set(data.keys()) if missing: - raise ValueError(f'mandatory key(s) {", ".join(sorted(missing))} for {model.__table__} missing', model, missing, data) + raise ValueError(f'mandatory key(s) {", ".join(sorted(missing))} for {cls.__table__} missing', cls, missing, data) # remove mapped relationships from data mapped = {} for key in list(data.keys()): - if key in model.__mapper__.relationships: - if isinstance(model.__mapper__.relationships[key].argument, sqlalchemy.orm.Mapper): + if key in cls.__mapper__.relationships: + if isinstance(cls.__mapper__.relationships[key].argument, sqlalchemy.orm.Mapper): mapped[key] = data[key] del data[key] # create new item - item = model(**data) + item = cls(**data) created = True # and update mapped relationships (below) @@ -278,14 +278,14 @@ class Base(db.Model): if key == pkey: continue - if key in model.__mapper__.relationships: + if key in cls.__mapper__.relationships: # update relationship - rel_model = model.__mapper__.relationships[key].argument + rel_model = cls.__mapper__.relationships[key].argument if isinstance(rel_model, sqlalchemy.orm.Mapper): rel_model = rel_model.class_ # add (and create) referenced items cur = getattr(item, key) - old = sorted(cur, key=lambda i:id(i)) + old = sorted(cur, key=id) new = [] for rel_data in value: # get or create related item @@ -331,16 +331,16 @@ class Base(db.Model): break # remember changes - new = sorted(new, key=lambda i:id(i)) + new = sorted(new, key=id) if new != old: updated.append((key, old, new)) else: # update key old = getattr(item, key) - if type(old) is list: + if isinstance(old, list): # deduplicate list value - assert type(value) is list + assert isinstance(value, list) value = set(value) old = set(old) if not delete: @@ -408,19 +408,19 @@ class Domain(Base): if 'dkim_key' in data: key = data['dkim_key'] if key is not None: - if type(key) is list: + if isinstance(key, list): key = ''.join(key) - if type(key) is str: + if isinstance(key, str): key = ''.join(key.strip().split()) # removes all whitespace if key == 'generate': data['dkim_key'] = dkim.gen_key() elif key: - m = re.match('^-----BEGIN (RSA )?PRIVATE KEY-----', key) - if m is not None: - key = key[m.end():] - m = re.search('-----END (RSA )?PRIVATE KEY-----$', key) - if m is not None: - key = key[:m.start()] + match = re.match('^-----BEGIN (RSA )?PRIVATE KEY-----', key) + if match is not None: + key = key[match.end():] + match = re.search('-----END (RSA )?PRIVATE KEY-----$', key) + if match is not None: + key = key[:match.start()] key = '\n'.join(wrap(key, 64)) key = f'-----BEGIN PRIVATE KEY-----\n{key}\n-----END PRIVATE KEY-----\n'.encode('ascii') try: @@ -428,7 +428,7 @@ class Domain(Base): except: raise ValueError('invalid dkim key') else: - data['dkim_key'] = key + data['dkim_key'] = key else: data['dkim_key'] = None @@ -505,8 +505,7 @@ class Domain(Base): for email in self.users + self.aliases: if email.localpart == localpart: return True - else: - return False + return False def check_mx(self): try: @@ -519,7 +518,7 @@ class Domain(Base): return False def __str__(self): - return self.name + return str(self.name) def __eq__(self, other): try: @@ -541,7 +540,7 @@ class Alternative(Base): backref=db.backref('alternatives', cascade='all, delete-orphan')) def __str__(self): - return self.name + return str(self.name) class Relay(Base): @@ -557,7 +556,7 @@ class Relay(Base): smtp = db.Column(db.String(80), nullable=True) def __str__(self): - return self.name + return str(self.name) class Email(object): @@ -571,7 +570,7 @@ class Email(object): if 'email' in data: if 'localpart' in data or 'domain' in data: raise ValueError('ambigous key email and localpart/domain') - elif type(data['email']) is str: + elif isinstance(data['email'], str): data['localpart'], data['domain'] = data['email'].rsplit('@', 1) else: data['email'] = f'{data["localpart"]}@{data["domain"]}' @@ -653,7 +652,7 @@ class Email(object): return pure_alias.destination def __str__(self): - return self.email + return str(self.email) class User(Base, Email): @@ -750,7 +749,7 @@ class User(Base, Email): 'CRYPT': 'des_crypt'} def get_password_context(self): - return context.CryptContext( + return passlib.context.CryptContext( schemes=self.scheme_dict.values(), default=self.scheme_dict[app.config['PASSWORD_SCHEME']], ) @@ -818,7 +817,7 @@ class Alias(Base, Email): Email._dict_input(data) # handle comma delimited string for backwards compability dst = data.get('destination') - if type(dst) is str: + if isinstance(dst, str): data['destination'] = list([adr.strip() for adr in dst.split(',')]) domain = db.relationship(Domain, @@ -888,10 +887,10 @@ class Token(Base): ip = db.Column(db.String(255)) def check_password(self, password): - return hash.sha256_crypt.verify(password, self.password) + return passlib.hash.sha256_crypt.verify(password, self.password) def set_password(self, password): - self.password = hash.sha256_crypt.using(rounds=1000).hash(password) + self.password = passlib.hash.sha256_crypt.using(rounds=1000).hash(password) def __str__(self): return self.comment or self.ip From 6629aa3ff824c57b1c752fdf7a7e7ea87423bc5e Mon Sep 17 00:00:00 2001 From: Alexander Graf Date: Wed, 6 Jan 2021 17:05:21 +0100 Subject: [PATCH 049/596] first try at api using flask-restx & marshmallow --- core/admin/mailu/__init__.py | 6 +- core/admin/mailu/api/__init__.py | 38 +++++ core/admin/mailu/api/common.py | 8 + core/admin/mailu/api/v1/__init__.py | 27 ++++ core/admin/mailu/api/v1/domains.py | 183 +++++++++++++++++++++++ core/admin/mailu/manage.py | 33 +++-- core/admin/mailu/schemas.py | 222 ++++++++++++++++++++++++++++ 7 files changed, 502 insertions(+), 15 deletions(-) create mode 100644 core/admin/mailu/api/__init__.py create mode 100644 core/admin/mailu/api/common.py create mode 100644 core/admin/mailu/api/v1/__init__.py create mode 100644 core/admin/mailu/api/v1/domains.py create mode 100644 core/admin/mailu/schemas.py diff --git a/core/admin/mailu/__init__.py b/core/admin/mailu/__init__.py index 4de3e580..9ab90add 100644 --- a/core/admin/mailu/__init__.py +++ b/core/admin/mailu/__init__.py @@ -41,15 +41,17 @@ def create_app_from_config(config): ) # Import views - from mailu import ui, internal + from mailu import ui, internal, api app.register_blueprint(ui.ui, url_prefix='/ui') app.register_blueprint(internal.internal, url_prefix='/internal') + if app.config.get('API'): + api.register(app) return app def create_app(): - """ Create a new application based on the config module + """ Create a new application based on the config module """ config = configuration.ConfigManager() return create_app_from_config(config) diff --git a/core/admin/mailu/api/__init__.py b/core/admin/mailu/api/__init__.py new file mode 100644 index 00000000..6e7d6386 --- /dev/null +++ b/core/admin/mailu/api/__init__.py @@ -0,0 +1,38 @@ +from flask import redirect, url_for + +# import api version(s) +from . import v1 + +# api +ROOT='/api' +ACTIVE=v1 + +# patch url for swaggerui static assets +from flask_restx.apidoc import apidoc +apidoc.static_url_path = f'{ROOT}/swaggerui' + +def register(app): + + # register api bluprint(s) + app.register_blueprint(v1.blueprint, url_prefix=f'{ROOT}/v{int(v1.VERSION)}') + + # add redirect to current api version + @app.route(f'{ROOT}/') + def redir(): + return redirect(url_for(f'{ACTIVE.blueprint.name}.root')) + + # swagger ui config + app.config.SWAGGER_UI_DOC_EXPANSION = 'list' + app.config.SWAGGER_UI_OPERATION_ID = True + app.config.SWAGGER_UI_REQUEST_DURATION = True + + # TODO: remove patch of static assets for debugging + import os + if 'STATIC_ASSETS' in os.environ: + app.blueprints['ui'].static_folder = os.environ['STATIC_ASSETS'] + +# TODO: authentication via username + password +# TODO: authentication via api token +# TODO: api access for all users (via token) +# TODO: use permissions from "manager_of" +# TODO: switch to marshmallow, as parser is deprecated. use flask_accepts? diff --git a/core/admin/mailu/api/common.py b/core/admin/mailu/api/common.py new file mode 100644 index 00000000..700835ac --- /dev/null +++ b/core/admin/mailu/api/common.py @@ -0,0 +1,8 @@ +from .. import models + +def fqdn_in_use(*names): + for name in names: + for model in models.Domain, models.Alternative, models.Relay: + if model.query.get(name): + return model + return None diff --git a/core/admin/mailu/api/v1/__init__.py b/core/admin/mailu/api/v1/__init__.py new file mode 100644 index 00000000..c6de6fa4 --- /dev/null +++ b/core/admin/mailu/api/v1/__init__.py @@ -0,0 +1,27 @@ +from flask import Blueprint +from flask_restx import Api, fields + +VERSION = 1.0 + +blueprint = Blueprint(f'api_v{int(VERSION)}', __name__) + +api = Api( + blueprint, version=f'{VERSION:.1f}', + title='Mailu API', default_label='Mailu', + validate=True +) + +response_fields = api.model('Response', { + 'code': fields.Integer, + 'message': fields.String, +}) + +error_fields = api.model('Error', { + 'errors': fields.Nested(api.model('Error_Key', { + 'key': fields.String, + 'message':fields.String + })), + 'message': fields.String, +}) + +from . import domains diff --git a/core/admin/mailu/api/v1/domains.py b/core/admin/mailu/api/v1/domains.py new file mode 100644 index 00000000..7afd26b9 --- /dev/null +++ b/core/admin/mailu/api/v1/domains.py @@ -0,0 +1,183 @@ +from flask_restx import Resource, fields, abort + +from . import api, response_fields, error_fields +from .. import common +from ... import models + +db = models.db + +dom = api.namespace('domain', description='Domain operations') +alt = api.namespace('alternative', description='Alternative operations') + +domain_fields = api.model('Domain', { + 'name': fields.String(description='FQDN', example='example.com', required=True), + 'comment': fields.String(description='a comment'), + 'max_users': fields.Integer(description='maximum number of users', min=-1, default=-1), + 'max_aliases': fields.Integer(description='maximum number of aliases', min=-1, default=-1), + 'max_quota_bytes': fields.Integer(description='maximum quota for mailbox', min=0), + 'signup_enabled': fields.Boolean(description='allow signup'), +# 'dkim_key': fields.String, + 'alternatives': fields.List(fields.String(attribute='name', description='FQDN', example='example.com')), +}) +# TODO - name ist required on creation but immutable on change +# TODO - name and alteranatives need to be checked to be a fqdn (regex) + +domain_parser = api.parser() +domain_parser.add_argument('max_users', type=int, help='maximum number of users') +# TODO ... add more (or use marshmallow) + +alternative_fields = api.model('Domain', { + 'name': fields.String(description='alternative FQDN', example='example.com', required=True), + 'domain': fields.String(description='domain FQDN', example='example.com', required=True), + 'dkim_key': fields.String, +}) +# TODO: domain and name are not always required and can't be changed + + +@dom.route('') +class Domains(Resource): + + @dom.doc('list_domain') + @dom.marshal_with(domain_fields, as_list=True, skip_none=True, mask=['dkim_key']) + def get(self): + """ List domains """ + return models.Domain.query.all() + + @dom.doc('create_domain') + @dom.expect(domain_fields) + @dom.response(200, 'Success', response_fields) + @dom.response(400, 'Input validation exception', error_fields) + @dom.response(409, 'Duplicate domain name', error_fields) + def post(self): + """ Create a new domain """ + data = api.payload + if common.fqdn_in_use(data['name']): + abort(409, f'Duplicate domain name {data["name"]!r}', errors={ + 'name': data['name'], + }) + for item, created in models.Domain.from_dict(data): + if not created: + abort(409, f'Duplicate domain name {item.name!r}', errors={ + 'alternatives': item.name, + }) + db.session.add(item) + db.session.commit() + +@dom.route('/') +class Domain(Resource): + + @dom.doc('get_domain') + @dom.response(200, 'Success', domain_fields) + @dom.response(404, 'Domain not found') + @dom.marshal_with(domain_fields) + def get(self, name): + """ Find domain by name """ + domain = models.Domain.query.get(name) + if not domain: + abort(404) + return domain + + @dom.doc('update_domain') + @dom.expect(domain_fields) + @dom.response(200, 'Success', response_fields) + @dom.response(400, 'Input validation exception', error_fields) + @dom.response(404, 'Domain not found') + def put(self, name): + """ Update an existing domain """ + domain = models.Domain.query.get(name) + if not domain: + abort(404) + data = api.payload + data['name'] = name + for item, created in models.Domain.from_dict(data): + if created is True: + db.session.add(item) + db.session.commit() + + @dom.doc('modify_domain') + @dom.expect(domain_parser) + @dom.response(200, 'Success', response_fields) + @dom.response(400, 'Input validation exception', error_fields) + @dom.response(404, 'Domain not found') + def post(self, name=None): + """ Updates domain with form data """ + domain = models.Domain.query.get(name) + if not domain: + abort(404) + data = dict(domain_parser.parse_args()) + data['name'] = name + for item, created in models.Domain.from_dict(data): + if created is True: + db.session.add(item) + # TODO: flush? + db.session.commit() + + @dom.doc('delete_domain') + @dom.response(200, 'Success', response_fields) + @dom.response(404, 'Domain not found') + def delete(self, name=None): + """ Delete domain """ + domain = models.Domain.query.get(name) + if not domain: + abort(404) + db.session.delete(domain) + db.session.commit() + + +# @dom.route('//alternative') +# @alt.route('') +# class Alternatives(Resource): + +# @alt.doc('alternatives_list') +# @alt.marshal_with(alternative_fields, as_list=True, skip_none=True, mask=['dkim_key']) +# def get(self, name=None): +# """ List alternatives (of domain) """ +# if name is None: +# return models.Alternative.query.all() +# else: +# return models.Alternative.query.filter_by(domain_name = name).all() + +# @alt.doc('alternative_create') +# @alt.expect(alternative_fields) +# @alt.response(200, 'Success', response_fields) +# @alt.response(400, 'Input validation exception', error_fields) +# @alt.response(404, 'Domain not found') +# @alt.response(409, 'Duplicate domain name', error_fields) +# def post(self, name=None): +# """ Create new alternative (for domain) """ +# # abort(501) +# data = api.payload +# if name is not None: +# data['name'] = name +# domain = models.Domain.query.get(name) +# if not domain: +# abort(404) +# if common.fqdn_in_use(data['name']): +# abort(409, f'Duplicate domain name {data["name"]!r}', errors={ +# 'name': data['name'], +# }) +# for item, created in models.Alternative.from_dict(data): +# # TODO: handle creation of domain +# if not created: +# abort(409, f'Duplicate domain name {item.name!r}', errors={ +# 'alternatives': item.name, +# }) +# # db.session.add(item) +# # db.session.commit() + +# @dom.route('//alternative/') +# @alt.route('/') +# class Alternative(Resource): +# def get(self, name, alt=None): +# """ Find alternative (of domain) """ +# abort(501) +# def put(self, name, alt=None): +# """ Update alternative (of domain) """ +# abort(501) +# def post(self, name, alt=None): +# """ Update alternative (of domain) with form data """ +# abort(501) +# def delete(self, name, alt=None): +# """ Delete alternative (for domain) """ +# abort(501) + diff --git a/core/admin/mailu/manage.py b/core/admin/mailu/manage.py index c07ca2b7..569c161c 100644 --- a/core/admin/mailu/manage.py +++ b/core/admin/mailu/manage.py @@ -1,4 +1,5 @@ from mailu import models +from .schemas import schemas from flask import current_app as app from flask.cli import FlaskGroup, with_appcontext @@ -320,24 +321,30 @@ def config_dump(full=False, secrets=False, dns=False, sections=None): return super().increase_indent(flow, False) if sections: - check = dict(yaml_sections) for section in sections: - if section not in check: + if section not in schemas: print(f'[ERROR] Invalid section: {section}') return 1 + else: + sections = sorted(schemas.keys()) - extra = [] - if dns: - extra.append('dns') +# TODO: create World Schema and dump only this with Word.dumps ? - config = {} - for section, model in yaml_sections: - if not sections or section in sections: - dump = [item.to_dict(full, secrets, extra) for item in model.query.all()] - if len(dump): - config[section] = dump - - yaml.dump(config, sys.stdout, Dumper=spacedDumper, default_flow_style=False, allow_unicode=True) + for section in sections: + schema = schemas[section](many=True) + schema.context.update({ + 'full': full, + 'secrets': secrets, + 'dns': dns, + }) + yaml.dump( + {section: schema.dump(schema.Meta.model.query.all())}, + sys.stdout, + Dumper=spacedDumper, + default_flow_style=False, + allow_unicode=True + ) + sys.stdout.write('\n') @mailu.command() diff --git a/core/admin/mailu/schemas.py b/core/admin/mailu/schemas.py new file mode 100644 index 00000000..20aa98c9 --- /dev/null +++ b/core/admin/mailu/schemas.py @@ -0,0 +1,222 @@ +import marshmallow +import flask_marshmallow + +from . import models + + +ma = flask_marshmallow.Marshmallow() + + +class BaseSchema(ma.SQLAlchemyAutoSchema): + + SKIP_IF = { + 'comment': {'', None} + } + + @marshmallow.post_dump + def remove_skip_values(self, data, many, **kwargs): + print(repr(data), self.context) + return { + key: value for key, value in data.items() + if key not in self.SKIP_IF or value not in self.SKIP_IF[key] + } + +class BaseMeta: + exclude = ['created_at', 'updated_at'] + + +class DomainSchema(ma.SQLAlchemyAutoSchema): + class Meta: + model = models.Domain + + # _dict_hide = {'users', 'managers', 'aliases'} + # _dict_show = {'dkim_key'} + # _dict_extra = {'dns':{'dkim_publickey', 'dns_mx', 'dns_spf', 'dns_dkim', 'dns_dmarc'}} + # _dict_secret = {'dkim_key'} + # _dict_types = { + # 'dkim_key': (bytes, type(None)), + # 'dkim_publickey': False, + # 'dns_mx': False, + # 'dns_spf': False, + # 'dns_dkim': False, + # 'dns_dmarc': False, + # } + # _dict_output = {'dkim_key': lambda key: key.decode('utf-8').strip().split('\n')[1:-1]} + # @staticmethod + # def _dict_input(data): + # if 'dkim_key' in data: + # key = data['dkim_key'] + # if key is not None: + # if type(key) is list: + # key = ''.join(key) + # if type(key) is str: + # key = ''.join(key.strip().split()) # removes all whitespace + # if key == 'generate': + # data['dkim_key'] = dkim.gen_key() + # elif key: + # m = re.match('^-----BEGIN (RSA )?PRIVATE KEY-----', key) + # if m is not None: + # key = key[m.end():] + # m = re.search('-----END (RSA )?PRIVATE KEY-----$', key) + # if m is not None: + # key = key[:m.start()] + # key = '\n'.join(wrap(key, 64)) + # key = f'-----BEGIN PRIVATE KEY-----\n{key}\n-----END PRIVATE KEY-----\n'.encode('ascii') + # try: + # dkim.strip_key(key) + # except: + # raise ValueError('invalid dkim key') + # else: + # data['dkim_key'] = key + # else: + # data['dkim_key'] = None + + # name = db.Column(IdnaDomain, primary_key=True, nullable=False) + # managers = db.relationship('User', secondary=managers, + # backref=db.backref('manager_of'), lazy='dynamic') + # max_users = db.Column(db.Integer, nullable=False, default=-1) + # max_aliases = db.Column(db.Integer, nullable=False, default=-1) + # max_quota_bytes = db.Column(db.BigInteger(), nullable=False, default=0) + # signup_enabled = db.Column(db.Boolean(), nullable=False, default=False) + + +class UserSchema(ma.SQLAlchemyAutoSchema): + class Meta: + model = models.User + + # _dict_hide = {'domain_name', 'domain', 'localpart', 'quota_bytes_used'} + # _dict_mandatory = {'localpart', 'domain', 'password'} + # @classmethod + # def _dict_input(cls, data): + # Email._dict_input(data) + # # handle password + # if 'password' in data: + # if 'password_hash' in data or 'hash_scheme' in data: + # raise ValueError('ambigous key password and password_hash/hash_scheme') + # # check (hashed) password + # password = data['password'] + # if password.startswith('{') and '}' in password: + # scheme = password[1:password.index('}')] + # if scheme not in cls.scheme_dict: + # raise ValueError(f'invalid password scheme {scheme!r}') + # else: + # raise ValueError(f'invalid hashed password {password!r}') + # elif 'password_hash' in data and 'hash_scheme' in data: + # if data['hash_scheme'] not in cls.scheme_dict: + # raise ValueError(f'invalid password scheme {scheme!r}') + # data['password'] = '{'+data['hash_scheme']+'}'+ data['password_hash'] + # del data['hash_scheme'] + # del data['password_hash'] + + # domain = db.relationship(Domain, + # backref=db.backref('users', cascade='all, delete-orphan')) + # password = db.Column(db.String(255), nullable=False) + # quota_bytes = db.Column(db.BigInteger(), nullable=False, default=10**9) + # quota_bytes_used = db.Column(db.BigInteger(), nullable=False, default=0) + # global_admin = db.Column(db.Boolean(), nullable=False, default=False) + # enabled = db.Column(db.Boolean(), nullable=False, default=True) + + # # Features + # enable_imap = db.Column(db.Boolean(), nullable=False, default=True) + # enable_pop = db.Column(db.Boolean(), nullable=False, default=True) + + # # Filters + # forward_enabled = db.Column(db.Boolean(), nullable=False, default=False) + # forward_destination = db.Column(CommaSeparatedList(), nullable=True, default=[]) + # forward_keep = db.Column(db.Boolean(), nullable=False, default=True) + # reply_enabled = db.Column(db.Boolean(), nullable=False, default=False) + # reply_subject = db.Column(db.String(255), nullable=True, default=None) + # reply_body = db.Column(db.Text(), nullable=True, default=None) + # reply_startdate = db.Column(db.Date, nullable=False, + # default=date(1900, 1, 1)) + # reply_enddate = db.Column(db.Date, nullable=False, + # default=date(2999, 12, 31)) + + # # Settings + # displayed_name = db.Column(db.String(160), nullable=False, default='') + # spam_enabled = db.Column(db.Boolean(), nullable=False, default=True) + # spam_threshold = db.Column(db.Integer(), nullable=False, default=80) + +class AliasSchema(BaseSchema): + class Meta(BaseMeta): + model = models.Alias + exclude = BaseMeta.exclude + ['localpart'] + # TODO look for good way to exclude secrets, unverbose and defaults + + # _dict_hide = {'domain_name', 'domain', 'localpart'} + # @staticmethod + # def _dict_input(data): + # Email._dict_input(data) + # # handle comma delimited string for backwards compability + # dst = data.get('destination') + # if type(dst) is str: + # data['destination'] = list([adr.strip() for adr in dst.split(',')]) + + +class TokenSchema(ma.SQLAlchemyAutoSchema): + class Meta: + model = models.Token + + # _dict_recurse = True + # _dict_hide = {'user', 'user_email'} + # _dict_mandatory = {'password'} + + # id = db.Column(db.Integer(), primary_key=True) + # user_email = db.Column(db.String(255), db.ForeignKey(User.email), + # nullable=False) + # user = db.relationship(User, + # backref=db.backref('tokens', cascade='all, delete-orphan')) + # password = db.Column(db.String(255), nullable=False) + # ip = db.Column(db.String(255)) + + +class FetchSchema(ma.SQLAlchemyAutoSchema): + class Meta: + model = models.Fetch + + # _dict_recurse = True + # _dict_hide = {'user_email', 'user', 'last_check', 'error'} + # _dict_mandatory = {'protocol', 'host', 'port', 'username', 'password'} + # _dict_secret = {'password'} + + # id = db.Column(db.Integer(), primary_key=True) + # user_email = db.Column(db.String(255), db.ForeignKey(User.email), + # nullable=False) + # user = db.relationship(User, + # backref=db.backref('fetches', cascade='all, delete-orphan')) + # protocol = db.Column(db.Enum('imap', 'pop3'), nullable=False) + # host = db.Column(db.String(255), nullable=False) + # port = db.Column(db.Integer(), nullable=False) + # tls = db.Column(db.Boolean(), nullable=False, default=False) + # username = db.Column(db.String(255), nullable=False) + # password = db.Column(db.String(255), nullable=False) + # keep = db.Column(db.Boolean(), nullable=False, default=False) + # last_check = db.Column(db.DateTime, nullable=True) + # error = db.Column(db.String(1023), nullable=True) + + +class ConfigSchema(ma.SQLAlchemySchema): + class Meta: + model = models.Config +# created_at = ma.auto_field(dump_only=True) +# updated_at = ma.auto_field(dump_only=True) + comment = ma.auto_field() + name = ma.auto_field(required=True) + value = ma.auto_field(required=True) + +class RelaySchema(BaseSchema): + class Meta(BaseMeta): + model = models.Relay +# created_at = ma.auto_field(dump_only=True) +# updated_at = ma.auto_field(dump_only=True) +# comment = ma.auto_field() +# name = ma.auto_field(required=True) +# smtp = ma.auto_field(required=True) + +schemas = { + 'domains': DomainSchema, + 'relays': RelaySchema, + 'users': UserSchema, + 'aliases': AliasSchema, +# 'config': ConfigSchema, +} From b3f8dacdadcc562c34205fb9c8e2fe95c90394e8 Mon Sep 17 00:00:00 2001 From: Alexander Graf Date: Fri, 8 Jan 2021 14:17:28 +0100 Subject: [PATCH 050/596] add docstrings and make linter happy --- core/admin/mailu/api/__init__.py | 13 ++++++++++--- core/admin/mailu/api/common.py | 6 ++++++ core/admin/mailu/api/v1/__init__.py | 5 +++++ core/admin/mailu/api/v1/domains.py | 24 ++++++++++++++---------- 4 files changed, 35 insertions(+), 13 deletions(-) diff --git a/core/admin/mailu/api/__init__.py b/core/admin/mailu/api/__init__.py index 6e7d6386..a4d8689f 100644 --- a/core/admin/mailu/api/__init__.py +++ b/core/admin/mailu/api/__init__.py @@ -1,4 +1,11 @@ +""" +Mailu API +""" + +import os + from flask import redirect, url_for +from flask_restx.apidoc import apidoc # import api version(s) from . import v1 @@ -8,10 +15,11 @@ ROOT='/api' ACTIVE=v1 # patch url for swaggerui static assets -from flask_restx.apidoc import apidoc apidoc.static_url_path = f'{ROOT}/swaggerui' def register(app): + """ Register api blueprint in flask app + """ # register api bluprint(s) app.register_blueprint(v1.blueprint, url_prefix=f'{ROOT}/v{int(v1.VERSION)}') @@ -26,8 +34,7 @@ def register(app): app.config.SWAGGER_UI_OPERATION_ID = True app.config.SWAGGER_UI_REQUEST_DURATION = True - # TODO: remove patch of static assets for debugging - import os + # TODO: remove patch of static asset location if 'STATIC_ASSETS' in os.environ: app.blueprints['ui'].static_folder = os.environ['STATIC_ASSETS'] diff --git a/core/admin/mailu/api/common.py b/core/admin/mailu/api/common.py index 700835ac..000bb2d6 100644 --- a/core/admin/mailu/api/common.py +++ b/core/admin/mailu/api/common.py @@ -1,6 +1,12 @@ +""" +Common functions for all API versions +""" + from .. import models def fqdn_in_use(*names): + """ Checks if fqdn is used + """ for name in names: for model in models.Domain, models.Alternative, models.Relay: if model.query.get(name): diff --git a/core/admin/mailu/api/v1/__init__.py b/core/admin/mailu/api/v1/__init__.py index c6de6fa4..c4379074 100644 --- a/core/admin/mailu/api/v1/__init__.py +++ b/core/admin/mailu/api/v1/__init__.py @@ -1,3 +1,7 @@ +""" +API Blueprint +""" + from flask import Blueprint from flask_restx import Api, fields @@ -24,4 +28,5 @@ error_fields = api.model('Error', { 'message': fields.String, }) +# import api namespaces (below field defs to avoid circular reference) from . import domains diff --git a/core/admin/mailu/api/v1/domains.py b/core/admin/mailu/api/v1/domains.py index 7afd26b9..82ed1fde 100644 --- a/core/admin/mailu/api/v1/domains.py +++ b/core/admin/mailu/api/v1/domains.py @@ -1,3 +1,7 @@ +""" +API: domain +""" + from flask_restx import Resource, fields, abort from . import api, response_fields, error_fields @@ -9,6 +13,7 @@ db = models.db dom = api.namespace('domain', description='Domain operations') alt = api.namespace('alternative', description='Alternative operations') +# TODO: use marshmallow domain_fields = api.model('Domain', { 'name': fields.String(description='FQDN', example='example.com', required=True), 'comment': fields.String(description='a comment'), @@ -19,12 +24,12 @@ domain_fields = api.model('Domain', { # 'dkim_key': fields.String, 'alternatives': fields.List(fields.String(attribute='name', description='FQDN', example='example.com')), }) -# TODO - name ist required on creation but immutable on change -# TODO - name and alteranatives need to be checked to be a fqdn (regex) +# TODO: name ist required on creation but immutable on change +# TODO: name and alteranatives need to be checked to be a fqdn (regex) domain_parser = api.parser() domain_parser.add_argument('max_users', type=int, help='maximum number of users') -# TODO ... add more (or use marshmallow) +# TODO: ... add more (or use marshmallow) alternative_fields = api.model('Domain', { 'name': fields.String(description='alternative FQDN', example='example.com', required=True), @@ -65,7 +70,7 @@ class Domains(Resource): @dom.route('/') class Domain(Resource): - + @dom.doc('get_domain') @dom.response(200, 'Success', domain_fields) @dom.response(404, 'Domain not found') @@ -76,7 +81,7 @@ class Domain(Resource): if not domain: abort(404) return domain - + @dom.doc('update_domain') @dom.expect(domain_fields) @dom.response(200, 'Success', response_fields) @@ -109,7 +114,7 @@ class Domain(Resource): for item, created in models.Domain.from_dict(data): if created is True: db.session.add(item) - # TODO: flush? + # TODO: flush? db.session.commit() @dom.doc('delete_domain') @@ -123,11 +128,11 @@ class Domain(Resource): db.session.delete(domain) db.session.commit() - + # @dom.route('//alternative') # @alt.route('') # class Alternatives(Resource): - + # @alt.doc('alternatives_list') # @alt.marshal_with(alternative_fields, as_list=True, skip_none=True, mask=['dkim_key']) # def get(self, name=None): @@ -136,7 +141,7 @@ class Domain(Resource): # return models.Alternative.query.all() # else: # return models.Alternative.query.filter_by(domain_name = name).all() - + # @alt.doc('alternative_create') # @alt.expect(alternative_fields) # @alt.response(200, 'Success', response_fields) @@ -180,4 +185,3 @@ class Domain(Resource): # def delete(self, name, alt=None): # """ Delete alternative (for domain) """ # abort(501) - From 82cf0d843f70584e7d0779bb58088e295193c63b Mon Sep 17 00:00:00 2001 From: Alexander Graf Date: Fri, 8 Jan 2021 14:22:11 +0100 Subject: [PATCH 051/596] fix sqlalchemy column definitions --- core/admin/mailu/models.py | 45 +++++++++++++++++++------------------- 1 file changed, 23 insertions(+), 22 deletions(-) diff --git a/core/admin/mailu/models.py b/core/admin/mailu/models.py index 3bf92244..625d1fd6 100644 --- a/core/admin/mailu/models.py +++ b/core/admin/mailu/models.py @@ -369,6 +369,7 @@ class Config(Base): value = db.Column(JSONEncoded) +# TODO: use sqlalchemy.event.listen() on a store method of object? @sqlalchemy.event.listens_for(db.session, 'after_commit') def store_dkim_key(session): """ Store DKIM key on commit @@ -437,8 +438,8 @@ class Domain(Base): backref=db.backref('manager_of'), lazy='dynamic') max_users = db.Column(db.Integer, nullable=False, default=-1) max_aliases = db.Column(db.Integer, nullable=False, default=-1) - max_quota_bytes = db.Column(db.BigInteger(), nullable=False, default=0) - signup_enabled = db.Column(db.Boolean(), nullable=False, default=False) + max_quota_bytes = db.Column(db.BigInteger, nullable=False, default=0) + signup_enabled = db.Column(db.Boolean, nullable=False, default=False) _dkim_key = None _dkim_key_changed = False @@ -688,22 +689,22 @@ class User(Base, Email): domain = db.relationship(Domain, backref=db.backref('users', cascade='all, delete-orphan')) password = db.Column(db.String(255), nullable=False) - quota_bytes = db.Column(db.BigInteger(), nullable=False, default=10**9) - quota_bytes_used = db.Column(db.BigInteger(), nullable=False, default=0) - global_admin = db.Column(db.Boolean(), nullable=False, default=False) - enabled = db.Column(db.Boolean(), nullable=False, default=True) + quota_bytes = db.Column(db.BigInteger, nullable=False, default=10**9) + quota_bytes_used = db.Column(db.BigInteger, nullable=False, default=0) + global_admin = db.Column(db.Boolean, nullable=False, default=False) + enabled = db.Column(db.Boolean, nullable=False, default=True) # Features - enable_imap = db.Column(db.Boolean(), nullable=False, default=True) - enable_pop = db.Column(db.Boolean(), nullable=False, default=True) + enable_imap = db.Column(db.Boolean, nullable=False, default=True) + enable_pop = db.Column(db.Boolean, nullable=False, default=True) # Filters - forward_enabled = db.Column(db.Boolean(), nullable=False, default=False) - forward_destination = db.Column(CommaSeparatedList(), nullable=True, default=[]) - forward_keep = db.Column(db.Boolean(), nullable=False, default=True) - reply_enabled = db.Column(db.Boolean(), nullable=False, default=False) + forward_enabled = db.Column(db.Boolean, nullable=False, default=False) + forward_destination = db.Column(CommaSeparatedList, nullable=True, default=list) + forward_keep = db.Column(db.Boolean, nullable=False, default=True) + reply_enabled = db.Column(db.Boolean, nullable=False, default=False) reply_subject = db.Column(db.String(255), nullable=True, default=None) - reply_body = db.Column(db.Text(), nullable=True, default=None) + reply_body = db.Column(db.Text, nullable=True, default=None) reply_startdate = db.Column(db.Date, nullable=False, default=date(1900, 1, 1)) reply_enddate = db.Column(db.Date, nullable=False, @@ -711,8 +712,8 @@ class User(Base, Email): # Settings displayed_name = db.Column(db.String(160), nullable=False, default='') - spam_enabled = db.Column(db.Boolean(), nullable=False, default=True) - spam_threshold = db.Column(db.Integer(), nullable=False, default=80) + spam_enabled = db.Column(db.Boolean, nullable=False, default=True) + spam_threshold = db.Column(db.Integer, nullable=False, default=80) # Flask-login attributes is_authenticated = True @@ -822,8 +823,8 @@ class Alias(Base, Email): domain = db.relationship(Domain, backref=db.backref('aliases', cascade='all, delete-orphan')) - wildcard = db.Column(db.Boolean(), nullable=False, default=False) - destination = db.Column(CommaSeparatedList, nullable=False, default=[]) + wildcard = db.Column(db.Boolean, nullable=False, default=False) + destination = db.Column(CommaSeparatedList, nullable=False, default=list) @classmethod def resolve(cls, localpart, domain_name): @@ -878,7 +879,7 @@ class Token(Base): _dict_hide = {'user', 'user_email'} _dict_mandatory = {'password'} - id = db.Column(db.Integer(), primary_key=True) + id = db.Column(db.Integer, primary_key=True) user_email = db.Column(db.String(255), db.ForeignKey(User.email), nullable=False) user = db.relationship(User, @@ -908,18 +909,18 @@ class Fetch(Base): _dict_mandatory = {'protocol', 'host', 'port', 'username', 'password'} _dict_secret = {'password'} - id = db.Column(db.Integer(), primary_key=True) + id = db.Column(db.Integer, primary_key=True) user_email = db.Column(db.String(255), db.ForeignKey(User.email), nullable=False) user = db.relationship(User, backref=db.backref('fetches', cascade='all, delete-orphan')) protocol = db.Column(db.Enum('imap', 'pop3'), nullable=False) host = db.Column(db.String(255), nullable=False) - port = db.Column(db.Integer(), nullable=False) - tls = db.Column(db.Boolean(), nullable=False, default=False) + port = db.Column(db.Integer, nullable=False) + tls = db.Column(db.Boolean, nullable=False, default=False) username = db.Column(db.String(255), nullable=False) password = db.Column(db.String(255), nullable=False) - keep = db.Column(db.Boolean(), nullable=False, default=False) + keep = db.Column(db.Boolean, nullable=False, default=False) last_check = db.Column(db.DateTime, nullable=True) error = db.Column(db.String(1023), nullable=True) From dc42d375e262d9d1d4f34364dbd68c50e5fd452f Mon Sep 17 00:00:00 2001 From: Alexander Graf Date: Fri, 8 Jan 2021 14:22:59 +0100 Subject: [PATCH 052/596] added filtering of keys and default value --- core/admin/mailu/schemas.py | 88 +++++++++++++++++++++++-------------- 1 file changed, 55 insertions(+), 33 deletions(-) diff --git a/core/admin/mailu/schemas.py b/core/admin/mailu/schemas.py index 20aa98c9..c2a3d03d 100644 --- a/core/admin/mailu/schemas.py +++ b/core/admin/mailu/schemas.py @@ -1,4 +1,5 @@ import marshmallow +import sqlalchemy import flask_marshmallow from . import models @@ -6,27 +7,51 @@ from . import models ma = flask_marshmallow.Marshmallow() - +import collections class BaseSchema(ma.SQLAlchemyAutoSchema): - SKIP_IF = { - 'comment': {'', None} - } + class Meta: + base_hide_always = {'created_at', 'updated_at'} + base_hide_secrets = set() + base_hide_by_value = { +# 'comment': {'', None} + } @marshmallow.post_dump def remove_skip_values(self, data, many, **kwargs): - print(repr(data), self.context) +# print(repr(data), self.context) + + # always hide + hide_by_key = self.Meta.base_hide_always | set(getattr(self.Meta, 'hide_always', ())) + + # hide secrets + if not self.context.get('secrets'): + hide_by_key |= self.Meta.base_hide_secrets + hide_by_key |= set(getattr(self.Meta, 'hide_secrets', ())) + + # hide by value + hide_by_value = self.Meta.base_hide_by_value | getattr(self.Meta, 'hide_by_value', {}) + + # hide defaults + if not self.context.get('full'): + for column in self.Meta.model.__table__.columns: +# print(column.name, column.default.arg if isinstance(column.default, sqlalchemy.sql.schema.ColumnDefault) else column.default) +# alias.destiantion has default [] - is this okay. how to check it? + if column.name not in hide_by_key: + hide_by_value.setdefault(column.name, set()).add(None if column.default is None else column.default.arg) + return { key: value for key, value in data.items() - if key not in self.SKIP_IF or value not in self.SKIP_IF[key] + if + not isinstance(value, collections.Hashable) + or( + key not in hide_by_key + and + (key not in hide_by_value or value not in hide_by_value[key])) } -class BaseMeta: - exclude = ['created_at', 'updated_at'] - - -class DomainSchema(ma.SQLAlchemyAutoSchema): - class Meta: +class DomainSchema(BaseSchema): + class Meta(BaseSchema.Meta): model = models.Domain # _dict_hide = {'users', 'managers', 'aliases'} @@ -80,8 +105,8 @@ class DomainSchema(ma.SQLAlchemyAutoSchema): # signup_enabled = db.Column(db.Boolean(), nullable=False, default=False) -class UserSchema(ma.SQLAlchemyAutoSchema): - class Meta: +class UserSchema(BaseSchema): + class Meta(BaseSchema.Meta): model = models.User # _dict_hide = {'domain_name', 'domain', 'localpart', 'quota_bytes_used'} @@ -138,12 +163,14 @@ class UserSchema(ma.SQLAlchemyAutoSchema): # spam_threshold = db.Column(db.Integer(), nullable=False, default=80) class AliasSchema(BaseSchema): - class Meta(BaseMeta): + class Meta(BaseSchema.Meta): model = models.Alias - exclude = BaseMeta.exclude + ['localpart'] - # TODO look for good way to exclude secrets, unverbose and defaults + hide_always = {'localpart'} + hide_secrets = {'wildcard'} + hide_by_value = { + 'destination': set([]) # always hide empty lists?! + } - # _dict_hide = {'domain_name', 'domain', 'localpart'} # @staticmethod # def _dict_input(data): # Email._dict_input(data) @@ -153,8 +180,8 @@ class AliasSchema(BaseSchema): # data['destination'] = list([adr.strip() for adr in dst.split(',')]) -class TokenSchema(ma.SQLAlchemyAutoSchema): - class Meta: +class TokenSchema(BaseSchema): + class Meta(BaseSchema.Meta): model = models.Token # _dict_recurse = True @@ -170,8 +197,8 @@ class TokenSchema(ma.SQLAlchemyAutoSchema): # ip = db.Column(db.String(255)) -class FetchSchema(ma.SQLAlchemyAutoSchema): - class Meta: +class FetchSchema(BaseSchema): + class Meta(BaseSchema.Meta): model = models.Fetch # _dict_recurse = True @@ -195,23 +222,18 @@ class FetchSchema(ma.SQLAlchemyAutoSchema): # error = db.Column(db.String(1023), nullable=True) -class ConfigSchema(ma.SQLAlchemySchema): - class Meta: +class ConfigSchema(BaseSchema): + class Meta(BaseSchema.Meta): model = models.Config -# created_at = ma.auto_field(dump_only=True) -# updated_at = ma.auto_field(dump_only=True) - comment = ma.auto_field() +# TODO: how to mark keys as "required" while unserializing (in certain use cases/API) name = ma.auto_field(required=True) value = ma.auto_field(required=True) + class RelaySchema(BaseSchema): - class Meta(BaseMeta): + class Meta(BaseSchema.Meta): model = models.Relay -# created_at = ma.auto_field(dump_only=True) -# updated_at = ma.auto_field(dump_only=True) -# comment = ma.auto_field() -# name = ma.auto_field(required=True) -# smtp = ma.auto_field(required=True) + schemas = { 'domains': DomainSchema, From 7413f9b7b42d8797f5de6203260761f2561859f0 Mon Sep 17 00:00:00 2001 From: Alexander Graf Date: Wed, 13 Jan 2021 00:05:43 +0100 Subject: [PATCH 053/596] config_dump now using marshmallow --- core/admin/mailu/manage.py | 46 +--- core/admin/mailu/models.py | 4 +- core/admin/mailu/schemas.py | 498 ++++++++++++++++++++++++------------ 3 files changed, 346 insertions(+), 202 deletions(-) diff --git a/core/admin/mailu/manage.py b/core/admin/mailu/manage.py index 569c161c..a67abdf7 100644 --- a/core/admin/mailu/manage.py +++ b/core/admin/mailu/manage.py @@ -1,5 +1,5 @@ from mailu import models -from .schemas import schemas +from .schemas import MailuConfig, MailuSchema from flask import current_app as app from flask.cli import FlaskGroup, with_appcontext @@ -310,41 +310,17 @@ def config_dump(full=False, secrets=False, dns=False, sections=None): SECTIONS can be: domains, relays, users, aliases """ - class spacedDumper(yaml.Dumper): + try: + config = MailuConfig(sections) + except ValueError as reason: + print(f'[ERROR] {reason}') + return 1 - def write_line_break(self, data=None): - super().write_line_break(data) - if len(self.indents) == 1: - super().write_line_break() - - def increase_indent(self, flow=False, indentless=False): - return super().increase_indent(flow, False) - - if sections: - for section in sections: - if section not in schemas: - print(f'[ERROR] Invalid section: {section}') - return 1 - else: - sections = sorted(schemas.keys()) - -# TODO: create World Schema and dump only this with Word.dumps ? - - for section in sections: - schema = schemas[section](many=True) - schema.context.update({ - 'full': full, - 'secrets': secrets, - 'dns': dns, - }) - yaml.dump( - {section: schema.dump(schema.Meta.model.query.all())}, - sys.stdout, - Dumper=spacedDumper, - default_flow_style=False, - allow_unicode=True - ) - sys.stdout.write('\n') + MailuSchema(context={ + 'full': full, + 'secrets': secrets, + 'dns': dns, + }).dumps(config, sys.stdout) @mailu.command() diff --git a/core/admin/mailu/models.py b/core/admin/mailu/models.py index 625d1fd6..8b77b011 100644 --- a/core/admin/mailu/models.py +++ b/core/admin/mailu/models.py @@ -110,7 +110,7 @@ class Base(db.Model): created_at = db.Column(db.Date, nullable=False, default=date.today) updated_at = db.Column(db.Date, nullable=True, onupdate=date.today) - comment = db.Column(db.String(255), nullable=True) + comment = db.Column(db.String(255), nullable=True, default='') @classmethod def _dict_pkey(cls): @@ -171,7 +171,7 @@ class Base(db.Model): if self.__mapper__.relationships[key].query_class is not None: if hasattr(items, 'all'): items = items.all() - if full or len(items): + if full or items: if key in secret: res[key] = '' else: diff --git a/core/admin/mailu/schemas.py b/core/admin/mailu/schemas.py index c2a3d03d..bd9d228a 100644 --- a/core/admin/mailu/schemas.py +++ b/core/admin/mailu/schemas.py @@ -1,63 +1,252 @@ -import marshmallow -import sqlalchemy -import flask_marshmallow +""" +Mailu marshmallow schema +""" -from . import models +from textwrap import wrap + +import re +import yaml + +from marshmallow import post_dump, fields, Schema +from flask_marshmallow import Marshmallow +from OpenSSL import crypto + +from . import models, dkim -ma = flask_marshmallow.Marshmallow() +ma = Marshmallow() +# TODO: +# how to mark keys as "required" while unserializing (in certain use cases/API)? +# - fields withoud default => required +# - fields which are the primary key => unchangeable when updating + + +### yaml render module ### + +class RenderYAML: + """ Marshmallow YAML Render Module + """ + + class SpacedDumper(yaml.Dumper): + """ YAML Dumper to add a newline between main sections + and double the indent used + """ + + def write_line_break(self, data=None): + super().write_line_break(data) + if len(self.indents) == 1: + super().write_line_break() + + def increase_indent(self, flow=False, indentless=False): + return super().increase_indent(flow, False) + + @staticmethod + def _update_dict(dict1, dict2): + """ sets missing keys in dict1 to values of dict2 + """ + for key, value in dict2.items(): + if key not in dict1: + dict1[key] = value + + _load_defaults = {} + @classmethod + def loads(cls, *args, **kwargs): + """ load yaml data from string + """ + cls._update_dict(kwargs, cls._load_defaults) + return yaml.load(*args, **kwargs) + + _dump_defaults = { + 'Dumper': SpacedDumper, + 'default_flow_style': False, + 'allow_unicode': True, + } + @classmethod + def dumps(cls, *args, **kwargs): + """ dump yaml data to string + """ + cls._update_dict(kwargs, cls._dump_defaults) + return yaml.dump(*args, **kwargs) + + +### field definitions ### + +class LazyString(fields.String): + """ Field that serializes a "false" value to the empty string + """ + + def _serialize(self, value, attr, obj, **kwargs): + """ serialize None to the empty string + """ + return value if value else '' + + +class CommaSeparatedList(fields.Raw): + """ Field that deserializes a string containing comma-separated values to + a list of strings + """ + # TODO: implement this + + +class DkimKey(fields.String): + """ Field that serializes a dkim key to a list of strings (lines) and + deserializes a string or list of strings. + """ + + _clean_re = re.compile( + r'(^-----BEGIN (RSA )?PRIVATE KEY-----|-----END (RSA )?PRIVATE KEY-----$|\s+)', + flags=re.UNICODE + ) + + def _serialize(self, value, attr, obj, **kwargs): + """ serialize dkim key to a list of strings (lines) + """ + + # map empty string and None to None + if not value: + return None + + # return list of key lines without header/footer + return value.decode('utf-8').strip().split('\n')[1:-1] + + def _deserialize(self, value, attr, data, **kwargs): + """ deserialize a string or list of strings to dkim key data + with verification + """ + + # convert list to str + if isinstance(value, list): + value = ''.join(value) + + # only strings are allowed + if not isinstance(value, str): + raise TypeError(f'invalid type: {type(value).__name__!r}') + + # clean value (remove whitespace and header/footer) + value = self._clean_re.sub('', value.strip()) + + # map empty string/list to None + if not value: + return None + + # handle special value 'generate' + elif value == 'generate': + return dkim.gen_key() + + # wrap value into valid pem layout and check validity + value = ( + '-----BEGIN PRIVATE KEY-----\n' + + '\n'.join(wrap(value, 64)) + + '\n-----END PRIVATE KEY-----\n' + ).encode('ascii') + try: + crypto.load_privatekey(crypto.FILETYPE_PEM, value) + except crypto.Error as exc: + raise ValueError('invalid dkim key') from exc + else: + return value + + +### schema definitions ### -import collections class BaseSchema(ma.SQLAlchemyAutoSchema): + """ Marshmallow base schema with custom exclude logic + and option to hide sqla defaults + """ class Meta: - base_hide_always = {'created_at', 'updated_at'} - base_hide_secrets = set() - base_hide_by_value = { -# 'comment': {'', None} - } + """ Schema config """ + model = None - @marshmallow.post_dump - def remove_skip_values(self, data, many, **kwargs): -# print(repr(data), self.context) + def __init__(self, *args, **kwargs): - # always hide - hide_by_key = self.Meta.base_hide_always | set(getattr(self.Meta, 'hide_always', ())) + # get and remove config from kwargs + context = kwargs.get('context', {}) - # hide secrets - if not self.context.get('secrets'): - hide_by_key |= self.Meta.base_hide_secrets - hide_by_key |= set(getattr(self.Meta, 'hide_secrets', ())) + # compile excludes + exclude = set(kwargs.get('exclude', [])) - # hide by value - hide_by_value = self.Meta.base_hide_by_value | getattr(self.Meta, 'hide_by_value', {}) + # always exclude + exclude.update({'created_at', 'updated_at'}) - # hide defaults - if not self.context.get('full'): + # add include_by_context + if context is not None: + for ctx, what in getattr(self.Meta, 'include_by_context', {}).items(): + if not context.get(ctx): + exclude |= set(what) + + # update excludes + kwargs['exclude'] = exclude + + # exclude_by_value + self._exclude_by_value = getattr(self.Meta, 'exclude_by_value', {}) + + # exclude default values + if not context.get('full'): for column in self.Meta.model.__table__.columns: -# print(column.name, column.default.arg if isinstance(column.default, sqlalchemy.sql.schema.ColumnDefault) else column.default) -# alias.destiantion has default [] - is this okay. how to check it? - if column.name not in hide_by_key: - hide_by_value.setdefault(column.name, set()).add(None if column.default is None else column.default.arg) + if column.name not in exclude: + self._exclude_by_value.setdefault(column.name, []).append( + None if column.default is None else column.default.arg + ) + # hide by context + self._hide_by_context = set() + if context is not None: + for ctx, what in getattr(self.Meta, 'hide_by_context', {}).items(): + if not context.get(ctx): + self._hide_by_context |= set(what) + + # init SQLAlchemyAutoSchema + super().__init__(*args, **kwargs) + + @post_dump + def _remove_skip_values(self, data, many, **kwargs): # pylint: disable=unused-argument + + if not self._exclude_by_value and not self._hide_by_context: + return data + + full = self.context.get('full') return { - key: value for key, value in data.items() - if - not isinstance(value, collections.Hashable) - or( - key not in hide_by_key - and - (key not in hide_by_value or value not in hide_by_value[key])) + key: '' if key in self._hide_by_context else value + for key, value in data.items() + if full or key not in self._exclude_by_value or value not in self._exclude_by_value[key] } + # TODO: remove LazyString and fix model definition (comment should not be nullable) + comment = LazyString() + class DomainSchema(BaseSchema): - class Meta(BaseSchema.Meta): + """ Marshmallow schema for Domain model """ + class Meta: + """ Schema config """ model = models.Domain + include_relationships = True + #include_fk = True + exclude = ['users', 'managers', 'aliases'] + + include_by_context = { + 'dns': {'dkim_publickey', 'dns_mx', 'dns_spf', 'dns_dkim', 'dns_dmarc'}, + } + hide_by_context = { + 'secrets': {'dkim_key'}, + } + exclude_by_value = { + 'alternatives': [[]], + 'dkim_key': [None], + 'dkim_publickey': [None], + 'dns_mx': [None], + 'dns_spf': [None], + 'dns_dkim': [None], + 'dns_dmarc': [None], + } + + dkim_key = DkimKey() + dkim_publickey = fields.String(dump_only=True) + dns_mx = fields.String(dump_only=True) + dns_spf = fields.String(dump_only=True) + dns_dkim = fields.String(dump_only=True) + dns_dmarc = fields.String(dump_only=True) - # _dict_hide = {'users', 'managers', 'aliases'} - # _dict_show = {'dkim_key'} - # _dict_extra = {'dns':{'dkim_publickey', 'dns_mx', 'dns_spf', 'dns_dkim', 'dns_dmarc'}} - # _dict_secret = {'dkim_key'} # _dict_types = { # 'dkim_key': (bytes, type(None)), # 'dkim_publickey': False, @@ -66,50 +255,62 @@ class DomainSchema(BaseSchema): # 'dns_dkim': False, # 'dns_dmarc': False, # } - # _dict_output = {'dkim_key': lambda key: key.decode('utf-8').strip().split('\n')[1:-1]} - # @staticmethod - # def _dict_input(data): - # if 'dkim_key' in data: - # key = data['dkim_key'] - # if key is not None: - # if type(key) is list: - # key = ''.join(key) - # if type(key) is str: - # key = ''.join(key.strip().split()) # removes all whitespace - # if key == 'generate': - # data['dkim_key'] = dkim.gen_key() - # elif key: - # m = re.match('^-----BEGIN (RSA )?PRIVATE KEY-----', key) - # if m is not None: - # key = key[m.end():] - # m = re.search('-----END (RSA )?PRIVATE KEY-----$', key) - # if m is not None: - # key = key[:m.start()] - # key = '\n'.join(wrap(key, 64)) - # key = f'-----BEGIN PRIVATE KEY-----\n{key}\n-----END PRIVATE KEY-----\n'.encode('ascii') - # try: - # dkim.strip_key(key) - # except: - # raise ValueError('invalid dkim key') - # else: - # data['dkim_key'] = key - # else: - # data['dkim_key'] = None - # name = db.Column(IdnaDomain, primary_key=True, nullable=False) - # managers = db.relationship('User', secondary=managers, - # backref=db.backref('manager_of'), lazy='dynamic') - # max_users = db.Column(db.Integer, nullable=False, default=-1) - # max_aliases = db.Column(db.Integer, nullable=False, default=-1) - # max_quota_bytes = db.Column(db.BigInteger(), nullable=False, default=0) - # signup_enabled = db.Column(db.Boolean(), nullable=False, default=False) + +class TokenSchema(BaseSchema): + """ Marshmallow schema for Token model """ + class Meta: + """ Schema config """ + model = models.Token + + # _dict_recurse = True + # _dict_hide = {'user', 'user_email'} + # _dict_mandatory = {'password'} + + # id = db.Column(db.Integer(), primary_key=True) + # user_email = db.Column(db.String(255), db.ForeignKey(User.email), + # nullable=False) + # user = db.relationship(User, + # backref=db.backref('tokens', cascade='all, delete-orphan')) + # password = db.Column(db.String(255), nullable=False) + # ip = db.Column(db.String(255)) + + +class FetchSchema(BaseSchema): + """ Marshmallow schema for Fetch model """ + class Meta: + """ Schema config """ + model = models.Fetch + include_by_context = { + 'full': {'last_check', 'error'}, + } + hide_by_context = { + 'secrets': {'password'}, + } + +# TODO: What about mandatory keys? + # _dict_mandatory = {'protocol', 'host', 'port', 'username', 'password'} class UserSchema(BaseSchema): - class Meta(BaseSchema.Meta): + """ Marshmallow schema for User model """ + class Meta: + """ Schema config """ model = models.User + include_relationships = True + exclude = ['localpart', 'domain', 'quota_bytes_used'] - # _dict_hide = {'domain_name', 'domain', 'localpart', 'quota_bytes_used'} + exclude_by_value = { + 'forward_destination': [[]], + 'tokens': [[]], + 'reply_enddate': ['2999-12-31'], + 'reply_startdate': ['1900-01-01'], + } + + tokens = fields.Nested(TokenSchema, many=True) + fetches = fields.Nested(FetchSchema, many=True) + +# TODO: deserialize password/password_hash! What about mandatory keys? # _dict_mandatory = {'localpart', 'domain', 'password'} # @classmethod # def _dict_input(cls, data): @@ -133,44 +334,19 @@ class UserSchema(BaseSchema): # del data['hash_scheme'] # del data['password_hash'] - # domain = db.relationship(Domain, - # backref=db.backref('users', cascade='all, delete-orphan')) - # password = db.Column(db.String(255), nullable=False) - # quota_bytes = db.Column(db.BigInteger(), nullable=False, default=10**9) - # quota_bytes_used = db.Column(db.BigInteger(), nullable=False, default=0) - # global_admin = db.Column(db.Boolean(), nullable=False, default=False) - # enabled = db.Column(db.Boolean(), nullable=False, default=True) - - # # Features - # enable_imap = db.Column(db.Boolean(), nullable=False, default=True) - # enable_pop = db.Column(db.Boolean(), nullable=False, default=True) - - # # Filters - # forward_enabled = db.Column(db.Boolean(), nullable=False, default=False) - # forward_destination = db.Column(CommaSeparatedList(), nullable=True, default=[]) - # forward_keep = db.Column(db.Boolean(), nullable=False, default=True) - # reply_enabled = db.Column(db.Boolean(), nullable=False, default=False) - # reply_subject = db.Column(db.String(255), nullable=True, default=None) - # reply_body = db.Column(db.Text(), nullable=True, default=None) - # reply_startdate = db.Column(db.Date, nullable=False, - # default=date(1900, 1, 1)) - # reply_enddate = db.Column(db.Date, nullable=False, - # default=date(2999, 12, 31)) - - # # Settings - # displayed_name = db.Column(db.String(160), nullable=False, default='') - # spam_enabled = db.Column(db.Boolean(), nullable=False, default=True) - # spam_threshold = db.Column(db.Integer(), nullable=False, default=80) class AliasSchema(BaseSchema): - class Meta(BaseSchema.Meta): + """ Marshmallow schema for Alias model """ + class Meta: + """ Schema config """ model = models.Alias - hide_always = {'localpart'} - hide_secrets = {'wildcard'} - hide_by_value = { - 'destination': set([]) # always hide empty lists?! + exclude = ['localpart'] + + exclude_by_value = { + 'destination': [[]], } +# TODO: deserialize destination! # @staticmethod # def _dict_input(data): # Email._dict_input(data) @@ -180,65 +356,57 @@ class AliasSchema(BaseSchema): # data['destination'] = list([adr.strip() for adr in dst.split(',')]) -class TokenSchema(BaseSchema): - class Meta(BaseSchema.Meta): - model = models.Token - - # _dict_recurse = True - # _dict_hide = {'user', 'user_email'} - # _dict_mandatory = {'password'} - - # id = db.Column(db.Integer(), primary_key=True) - # user_email = db.Column(db.String(255), db.ForeignKey(User.email), - # nullable=False) - # user = db.relationship(User, - # backref=db.backref('tokens', cascade='all, delete-orphan')) - # password = db.Column(db.String(255), nullable=False) - # ip = db.Column(db.String(255)) - - -class FetchSchema(BaseSchema): - class Meta(BaseSchema.Meta): - model = models.Fetch - - # _dict_recurse = True - # _dict_hide = {'user_email', 'user', 'last_check', 'error'} - # _dict_mandatory = {'protocol', 'host', 'port', 'username', 'password'} - # _dict_secret = {'password'} - - # id = db.Column(db.Integer(), primary_key=True) - # user_email = db.Column(db.String(255), db.ForeignKey(User.email), - # nullable=False) - # user = db.relationship(User, - # backref=db.backref('fetches', cascade='all, delete-orphan')) - # protocol = db.Column(db.Enum('imap', 'pop3'), nullable=False) - # host = db.Column(db.String(255), nullable=False) - # port = db.Column(db.Integer(), nullable=False) - # tls = db.Column(db.Boolean(), nullable=False, default=False) - # username = db.Column(db.String(255), nullable=False) - # password = db.Column(db.String(255), nullable=False) - # keep = db.Column(db.Boolean(), nullable=False, default=False) - # last_check = db.Column(db.DateTime, nullable=True) - # error = db.Column(db.String(1023), nullable=True) - - class ConfigSchema(BaseSchema): - class Meta(BaseSchema.Meta): + """ Marshmallow schema for Config model """ + class Meta: + """ Schema config """ model = models.Config -# TODO: how to mark keys as "required" while unserializing (in certain use cases/API) - name = ma.auto_field(required=True) - value = ma.auto_field(required=True) class RelaySchema(BaseSchema): - class Meta(BaseSchema.Meta): + """ Marshmallow schema for Relay model """ + class Meta: + """ Schema config """ model = models.Relay -schemas = { - 'domains': DomainSchema, - 'relays': RelaySchema, - 'users': UserSchema, - 'aliases': AliasSchema, -# 'config': ConfigSchema, -} +class MailuSchema(Schema): + """ Marshmallow schema for Mailu config """ + class Meta: + """ Schema config """ + render_module = RenderYAML + domains = fields.Nested(DomainSchema, many=True) + relays = fields.Nested(RelaySchema, many=True) + users = fields.Nested(UserSchema, many=True) + aliases = fields.Nested(AliasSchema, many=True) + config = fields.Nested(ConfigSchema, many=True) + + +### config class ### + +class MailuConfig: + """ Class which joins whole Mailu config for dumping + """ + + _models = { + 'domains': models.Domain, + 'relays': models.Relay, + 'users': models.User, + 'aliases': models.Alias, +# 'config': models.Config, + } + + def __init__(self, sections): + if sections: + for section in sections: + if section not in self._models: + raise ValueError(f'Unknown section: {section!r}') + self._sections = set(sections) + else: + self._sections = set(self._models.keys()) + + def __getattr__(self, section): + if section in self._sections: + return self._models[section].query.all() + else: + raise AttributeError From c24bff1c1be3980892fddb13a139b8142bb71f96 Mon Sep 17 00:00:00 2001 From: Alexander Graf Date: Thu, 14 Jan 2021 01:11:04 +0100 Subject: [PATCH 054/596] added config_import using marshmallow --- core/admin/mailu/api/__init__.py | 2 +- core/admin/mailu/manage.py | 314 ++++++++------- core/admin/mailu/models.py | 650 +++++++++++++++---------------- core/admin/mailu/schemas.py | 238 +++++------ 4 files changed, 611 insertions(+), 593 deletions(-) diff --git a/core/admin/mailu/api/__init__.py b/core/admin/mailu/api/__init__.py index a4d8689f..dee43036 100644 --- a/core/admin/mailu/api/__init__.py +++ b/core/admin/mailu/api/__init__.py @@ -26,7 +26,7 @@ def register(app): # add redirect to current api version @app.route(f'{ROOT}/') - def redir(): + def _redirect_to_active_api(): return redirect(url_for(f'{ACTIVE.blueprint.name}.root')) # swagger ui config diff --git a/core/admin/mailu/manage.py b/core/admin/mailu/manage.py index a67abdf7..6103f904 100644 --- a/core/admin/mailu/manage.py +++ b/core/admin/mailu/manage.py @@ -1,21 +1,25 @@ -from mailu import models -from .schemas import MailuConfig, MailuSchema - -from flask import current_app as app -from flask.cli import FlaskGroup, with_appcontext +""" Mailu command line interface +""" +import sys import os import socket import uuid + import click -import yaml -import sys + +from flask import current_app as app +from flask.cli import FlaskGroup, with_appcontext +from marshmallow.exceptions import ValidationError + +from . import models +from .schemas import MailuSchema db = models.db -@click.group(cls=FlaskGroup) +@click.group(cls=FlaskGroup, context_settings={'help_option_names': ['-?', '-h', '--help']}) def mailu(): """ Mailu command line """ @@ -26,17 +30,17 @@ def mailu(): def advertise(): """ Advertise this server against statistic services. """ - if os.path.isfile(app.config["INSTANCE_ID_PATH"]): - with open(app.config["INSTANCE_ID_PATH"], "r") as handle: + if os.path.isfile(app.config['INSTANCE_ID_PATH']): + with open(app.config['INSTANCE_ID_PATH'], 'r') as handle: instance_id = handle.read() else: instance_id = str(uuid.uuid4()) - with open(app.config["INSTANCE_ID_PATH"], "w") as handle: + with open(app.config['INSTANCE_ID_PATH'], 'w') as handle: handle.write(instance_id) - if not app.config["DISABLE_STATISTICS"]: + if not app.config['DISABLE_STATISTICS']: try: - socket.gethostbyname(app.config["STATS_ENDPOINT"].format(instance_id)) - except: + socket.gethostbyname(app.config['STATS_ENDPOINT'].format(instance_id)) + except OSError: pass @@ -171,156 +175,196 @@ def user_import(localpart, domain_name, password_hash, hash_scheme = None): db.session.commit() -yaml_sections = [ - ('domains', models.Domain), - ('relays', models.Relay), - ('users', models.User), - ('aliases', models.Alias), -# ('config', models.Config), -] +# @mailu.command() +# @click.option('-v', '--verbose', is_flag=True, help='Increase verbosity') +# @click.option('-d', '--delete-objects', is_flag=True, help='Remove objects not included in yaml') +# @click.option('-n', '--dry-run', is_flag=True, help='Perform a trial run with no changes made') +# @click.argument('source', metavar='[FILENAME|-]', type=click.File(mode='r'), default=sys.stdin) +# @with_appcontext +# def config_update(verbose=False, delete_objects=False, dry_run=False, source=None): +# """ Update configuration with data from YAML-formatted input +# """ + + # try: + # new_config = yaml.safe_load(source) + # except (yaml.scanner.ScannerError, yaml.parser.ParserError) as exc: + # print(f'[ERROR] Invalid yaml: {exc}') + # sys.exit(1) + # else: + # if isinstance(new_config, str): + # print(f'[ERROR] Invalid yaml: {new_config!r}') + # sys.exit(1) + # elif new_config is None or not new_config: + # print('[ERROR] Empty yaml: Please pipe yaml into stdin') + # sys.exit(1) + + # error = False + # tracked = {} + # for section, model in yaml_sections: + + # items = new_config.get(section) + # if items is None: + # if delete_objects: + # print(f'[ERROR] Invalid yaml: Section "{section}" is missing') + # error = True + # break + # else: + # continue + + # del new_config[section] + + # if not isinstance(items, list): + # print(f'[ERROR] Section "{section}" must be a list, not {items.__class__.__name__}') + # error = True + # break + # elif not items: + # continue + + # # create items + # for data in items: + + # if verbose: + # print(f'Handling {model.__table__} data: {data!r}') + + # try: + # changed = model.from_dict(data, delete_objects) + # except Exception as exc: + # print(f'[ERROR] {exc.args[0]} in data: {data}') + # error = True + # break + + # for item, created in changed: + + # if created is True: + # # flush newly created item + # db.session.add(item) + # db.session.flush() + # if verbose: + # print(f'Added {item!r}: {item.to_dict()}') + # else: + # print(f'Added {item!r}') + + # elif created: + # # modified instance + # if verbose: + # for key, old, new in created: + # print(f'Updated {key!r} of {item!r}: {old!r} -> {new!r}') + # else: + # print(f'Updated {item!r}: {", ".join(sorted([kon[0] for kon in created]))}') + + # # track primary key of all items + # tracked.setdefault(item.__class__, set()).update(set([item._dict_pval()])) + + # if error: + # break + + # # on error: stop early + # if error: + # print('[ERROR] An error occured. Not committing changes.') + # db.session.rollback() + # sys.exit(1) + + # # are there sections left in new_config? + # if new_config: + # print(f'[ERROR] Unknown section(s) in yaml: {", ".join(sorted(new_config.keys()))}') + # error = True + + # # test for conflicting domains + # domains = set() + # for model, items in tracked.items(): + # if model in (models.Domain, models.Alternative, models.Relay): + # if domains & items: + # for fqdn in domains & items: + # print(f'[ERROR] Duplicate domain name used: {fqdn}') + # error = True + # domains.update(items) + + # # delete items not tracked + # if delete_objects: + # for model, items in tracked.items(): + # for item in model.query.all(): + # if not item._dict_pval() in items: + # print(f'Deleted {item!r} {item}') + # db.session.delete(item) + + # # don't commit when running dry + # if dry_run: + # print('Dry run. Not commiting changes.') + # db.session.rollback() + # else: + # db.session.commit() + + +SECTIONS = {'domains', 'relays', 'users', 'aliases'} + @mailu.command() @click.option('-v', '--verbose', is_flag=True, help='Increase verbosity') -@click.option('-d', '--delete-objects', is_flag=True, help='Remove objects not included in yaml') @click.option('-n', '--dry-run', is_flag=True, help='Perform a trial run with no changes made') +@click.argument('source', metavar='[FILENAME|-]', type=click.File(mode='r'), default=sys.stdin) @with_appcontext -def config_update(verbose=False, delete_objects=False, dry_run=False, file=None): - """sync configuration with data from YAML-formatted stdin""" +def config_import(verbose=False, dry_run=False, source=None): + """ Import configuration YAML + """ - out = (lambda *args: print('(DRY RUN)', *args)) if dry_run else print + context = { + 'verbose': verbose, # TODO: use callback function to be verbose? + 'import': True, + } try: - new_config = yaml.safe_load(sys.stdin) - except (yaml.scanner.ScannerError, yaml.parser.ParserError) as reason: - out(f'[ERROR] Invalid yaml: {reason}') + config = MailuSchema(context=context).loads(source) + except ValidationError as exc: + print(f'[ERROR] {exc}') + # TODO: show nice errors + from pprint import pprint + pprint(exc.messages) sys.exit(1) else: - if type(new_config) is str: - out(f'[ERROR] Invalid yaml: {new_config!r}') - sys.exit(1) - elif new_config is None or not len(new_config): - out('[ERROR] Empty yaml: Please pipe yaml into stdin') - sys.exit(1) - - error = False - tracked = {} - for section, model in yaml_sections: - - items = new_config.get(section) - if items is None: - if delete_objects: - out(f'[ERROR] Invalid yaml: Section "{section}" is missing') - error = True - break - else: - continue - - del new_config[section] - - if type(items) is not list: - out(f'[ERROR] Section "{section}" must be a list, not {items.__class__.__name__}') - error = True - break - elif not items: - continue - - # create items - for data in items: - - if verbose: - out(f'Handling {model.__table__} data: {data!r}') - - try: - changed = model.from_dict(data, delete_objects) - except Exception as reason: - out(f'[ERROR] {reason.args[0]} in data: {data}') - error = True - break - - for item, created in changed: - - if created is True: - # flush newly created item - db.session.add(item) - db.session.flush() - if verbose: - out(f'Added {item!r}: {item.to_dict()}') - else: - out(f'Added {item!r}') - - elif len(created): - # modified instance - if verbose: - for key, old, new in created: - out(f'Updated {key!r} of {item!r}: {old!r} -> {new!r}') - else: - out(f'Updated {item!r}: {", ".join(sorted([kon[0] for kon in created]))}') - - # track primary key of all items - tracked.setdefault(item.__class__, set()).update(set([item._dict_pval()])) - - if error: - break - - # on error: stop early - if error: - out('An error occured. Not committing changes.') - db.session.rollback() - sys.exit(1) - - # are there sections left in new_config? - if new_config: - out(f'[ERROR] Unknown section(s) in yaml: {", ".join(sorted(new_config.keys()))}') - error = True - - # test for conflicting domains - domains = set() - for model, items in tracked.items(): - if model in (models.Domain, models.Alternative, models.Relay): - if domains & items: - for domain in domains & items: - out(f'[ERROR] Duplicate domain name used: {domain}') - error = True - domains.update(items) - - # delete items not tracked - if delete_objects: - for model, items in tracked.items(): - for item in model.query.all(): - if not item._dict_pval() in items: - out(f'Deleted {item!r} {item}') - db.session.delete(item) + print(config) + print(MailuSchema().dumps(config)) + # TODO: does not commit yet. + # TODO: delete other entries? # don't commit when running dry - if dry_run: + if True: #dry_run: + print('Dry run. Not commiting changes.') db.session.rollback() else: db.session.commit() @mailu.command() -@click.option('-f', '--full', is_flag=True, help='Include default attributes') -@click.option('-s', '--secrets', is_flag=True, help='Include secrets (dkim-key, plain-text / not hashed)') +@click.option('-f', '--full', is_flag=True, help='Include attributes with default value') +@click.option('-s', '--secrets', is_flag=True, + help='Include secret attributes (dkim-key, passwords)') @click.option('-d', '--dns', is_flag=True, help='Include dns records') +@click.option('-o', '--output-file', 'output', default=sys.stdout, type=click.File(mode='w'), + help='save yaml to file') @click.argument('sections', nargs=-1) @with_appcontext -def config_dump(full=False, secrets=False, dns=False, sections=None): - """dump configuration as YAML-formatted data to stdout +def config_dump(full=False, secrets=False, dns=False, output=None, sections=None): + """ Dump configuration as YAML to stdout or file SECTIONS can be: domains, relays, users, aliases """ - try: - config = MailuConfig(sections) - except ValueError as reason: - print(f'[ERROR] {reason}') - return 1 + if sections: + for section in sections: + if section not in SECTIONS: + print(f'[ERROR] Unknown section: {section!r}') + sys.exit(1) + sections = set(sections) + else: + sections = SECTIONS - MailuSchema(context={ + context={ 'full': full, 'secrets': secrets, 'dns': dns, - }).dumps(config, sys.stdout) + } + + MailuSchema(only=sections, context=context).dumps(models.MailuConfig(), output) @mailu.command() diff --git a/core/admin/mailu/models.py b/core/admin/mailu/models.py index 8b77b011..73d05801 100644 --- a/core/admin/mailu/models.py +++ b/core/admin/mailu/models.py @@ -1,23 +1,26 @@ -from mailu import dkim +""" Mailu config storage model +""" -from sqlalchemy.ext import declarative -from datetime import datetime, date +import re +import os +import smtplib +import json + +from datetime import date from email.mime import text -from flask import current_app as app -from textwrap import wrap import flask_sqlalchemy import sqlalchemy -import re -import time -import os import passlib -import glob -import smtplib import idna import dns -import json -import itertools + +from flask import current_app as app +from sqlalchemy.ext import declarative +from sqlalchemy.inspection import inspect +from werkzeug.utils import cached_property + +from . import dkim db = flask_sqlalchemy.SQLAlchemy() @@ -30,9 +33,11 @@ class IdnaDomain(db.TypeDecorator): impl = db.String(80) def process_bind_param(self, value, dialect): + """ encode unicode domain name to punycode """ return idna.encode(value).decode('ascii').lower() def process_result_value(self, value, dialect): + """ decode punycode domain name to unicode """ return idna.decode(value) python_type = str @@ -44,6 +49,7 @@ class IdnaEmail(db.TypeDecorator): impl = db.String(255) def process_bind_param(self, value, dialect): + """ encode unicode domain part of email address to punycode """ try: localpart, domain_name = value.split('@') return '{0}@{1}'.format( @@ -54,6 +60,7 @@ class IdnaEmail(db.TypeDecorator): pass def process_result_value(self, value, dialect): + """ decode punycode domain part of email to unicode """ localpart, domain_name = value.split('@') return '{0}@{1}'.format( localpart, @@ -69,14 +76,16 @@ class CommaSeparatedList(db.TypeDecorator): impl = db.String def process_bind_param(self, value, dialect): - if not isinstance(value, (list, set)): - raise TypeError('Must be a list') + """ join list of items to comma separated string """ + if not isinstance(value, (list, tuple, set)): + raise TypeError('Must be a list of strings') for item in value: if ',' in item: raise ValueError('Item must not contain a comma') return ','.join(sorted(value)) def process_result_value(self, value, dialect): + """ split comma separated string to list """ return list(filter(bool, value.split(','))) if value else [] python_type = list @@ -88,9 +97,11 @@ class JSONEncoded(db.TypeDecorator): impl = db.String def process_bind_param(self, value, dialect): + """ encode data as json """ return json.dumps(value) if value else None def process_result_value(self, value, dialect): + """ decode json to data """ return json.loads(value) if value else None python_type = str @@ -112,246 +123,172 @@ class Base(db.Model): updated_at = db.Column(db.Date, nullable=True, onupdate=date.today) comment = db.Column(db.String(255), nullable=True, default='') - @classmethod - def _dict_pkey(cls): - return cls.__mapper__.primary_key[0].name + # @classmethod + # def from_dict(cls, data, delete=False): - def _dict_pval(self): - return getattr(self, self._dict_pkey()) + # changed = [] - def to_dict(self, full=False, include_secrets=False, include_extra=None, recursed=False, hide=None): - """ Return a dictionary representation of this model. - """ + # pkey = cls._dict_pkey() - if recursed and not getattr(self, '_dict_recurse', False): - return str(self) + # # handle "primary key" only + # if not isinstance(data, dict): + # data = {pkey: data} - hide = set(hide or []) | {'created_at', 'updated_at'} - if hasattr(self, '_dict_hide'): - hide |= self._dict_hide + # # modify input data + # if hasattr(cls, '_dict_input'): + # try: + # cls._dict_input(data) + # except Exception as exc: + # raise ValueError(f'{exc}', cls, None, data) from exc - secret = set() - if not include_secrets and hasattr(self, '_dict_secret'): - secret |= self._dict_secret + # # check for primary key (if not recursed) + # if not getattr(cls, '_dict_recurse', False): + # if not pkey in data: + # raise KeyError(f'primary key {cls.__table__}.{pkey} is missing', cls, pkey, data) - convert = getattr(self, '_dict_output', {}) + # # check data keys and values + # for key in list(data.keys()): - extra_keys = getattr(self, '_dict_extra', {}) - if include_extra is None: - include_extra = [] + # # check key + # if not hasattr(cls, key) and not key in cls.__mapper__.relationships: + # raise KeyError(f'unknown key {cls.__table__}.{key}', cls, key, data) - res = {} + # # check value type + # value = data[key] + # col = cls.__mapper__.columns.get(key) + # if col is not None: + # if not ((value is None and col.nullable) or (isinstance(value, col.type.python_type))): + # raise TypeError(f'{cls.__table__}.{key} {value!r} has invalid type {type(value).__name__!r}', cls, key, data) + # else: + # rel = cls.__mapper__.relationships.get(key) + # if rel is None: + # itype = getattr(cls, '_dict_types', {}).get(key) + # if itype is not None: + # if itype is False: # ignore value. TODO: emit warning? + # del data[key] + # continue + # elif not isinstance(value, itype): + # raise TypeError(f'{cls.__table__}.{key} {value!r} has invalid type {type(value).__name__!r}', cls, key, data) + # else: + # raise NotImplementedError(f'type not defined for {cls.__table__}.{key}') - for key in itertools.chain( - self.__table__.columns.keys(), - getattr(self, '_dict_show', []), - *[extra_keys.get(extra, []) for extra in include_extra] - ): - if key in hide: - continue - if key in self.__table__.columns: - default = self.__table__.columns[key].default - if isinstance(default, sqlalchemy.sql.schema.ColumnDefault): - default = default.arg - else: - default = None - value = getattr(self, key) - if full or ((default or value) and value != default): - if key in secret: - value = '' - elif value is not None and key in convert: - value = convert[key](value) - res[key] = value + # # handle relationships + # if key in cls.__mapper__.relationships: + # rel_model = cls.__mapper__.relationships[key].argument + # if not isinstance(rel_model, sqlalchemy.orm.Mapper): + # add = rel_model.from_dict(value, delete) + # assert len(add) == 1 + # rel_item, updated = add[0] + # changed.append((rel_item, updated)) + # data[key] = rel_item - for key in self.__mapper__.relationships.keys(): - if key in hide: - continue - if self.__mapper__.relationships[key].uselist: - items = getattr(self, key) - if self.__mapper__.relationships[key].query_class is not None: - if hasattr(items, 'all'): - items = items.all() - if full or items: - if key in secret: - res[key] = '' - else: - res[key] = [item.to_dict(full, include_secrets, include_extra, True) for item in items] - else: - value = getattr(self, key) - if full or value is not None: - if key in secret: - res[key] = '' - else: - res[key] = value.to_dict(full, include_secrets, include_extra, True) + # # create item if necessary + # created = False + # item = cls.query.get(data[pkey]) if pkey in data else None + # if item is None: - return res + # # check for mandatory keys + # missing = getattr(cls, '_dict_mandatory', set()) - set(data.keys()) + # if missing: + # raise ValueError(f'mandatory key(s) {", ".join(sorted(missing))} for {cls.__table__} missing', cls, missing, data) - @classmethod - def from_dict(cls, data, delete=False): + # # remove mapped relationships from data + # mapped = {} + # for key in list(data.keys()): + # if key in cls.__mapper__.relationships: + # if isinstance(cls.__mapper__.relationships[key].argument, sqlalchemy.orm.Mapper): + # mapped[key] = data[key] + # del data[key] - changed = [] + # # create new item + # item = cls(**data) + # created = True - pkey = cls._dict_pkey() + # # and update mapped relationships (below) + # data = mapped - # handle "primary key" only - if isinstance(data, dict): - data = {pkey: data} + # # update item + # updated = [] + # for key, value in data.items(): - # modify input data - if hasattr(cls, '_dict_input'): - try: - cls._dict_input(data) - except Exception as reason: - raise ValueError(f'{reason}', cls, None, data) + # # skip primary key + # if key == pkey: + # continue - # check for primary key (if not recursed) - if not getattr(cls, '_dict_recurse', False): - if not pkey in data: - raise KeyError(f'primary key {cls.__table__}.{pkey} is missing', cls, pkey, data) + # if key in cls.__mapper__.relationships: + # # update relationship + # rel_model = cls.__mapper__.relationships[key].argument + # if isinstance(rel_model, sqlalchemy.orm.Mapper): + # rel_model = rel_model.class_ + # # add (and create) referenced items + # cur = getattr(item, key) + # old = sorted(cur, key=id) + # new = [] + # for rel_data in value: + # # get or create related item + # add = rel_model.from_dict(rel_data, delete) + # assert len(add) == 1 + # rel_item, rel_updated = add[0] + # changed.append((rel_item, rel_updated)) + # if rel_item not in cur: + # cur.append(rel_item) + # new.append(rel_item) - # check data keys and values - for key in list(data.keys()): + # # delete referenced items missing in yaml + # rel_pkey = rel_model._dict_pkey() + # new_data = list([i.to_dict(True, True, None, True, [rel_pkey]) for i in new]) + # for rel_item in old: + # if rel_item not in new: + # # check if item with same data exists to stabilze import without primary key + # rel_data = rel_item.to_dict(True, True, None, True, [rel_pkey]) + # try: + # same_idx = new_data.index(rel_data) + # except ValueError: + # same = None + # else: + # same = new[same_idx] - # check key - if not hasattr(cls, key) and not key in cls.__mapper__.relationships: - raise KeyError(f'unknown key {cls.__table__}.{key}', cls, key, data) + # if same is None: + # # delete items missing in new + # if delete: + # cur.remove(rel_item) + # else: + # new.append(rel_item) + # else: + # # swap found item with same data with newly created item + # new.append(rel_item) + # new_data.append(rel_data) + # new.remove(same) + # del new_data[same_idx] + # for i, (ch_item, _) in enumerate(changed): + # if ch_item is same: + # changed[i] = (rel_item, []) + # db.session.flush() + # db.session.delete(ch_item) + # break - # check value type - value = data[key] - col = cls.__mapper__.columns.get(key) - if col is not None: - if not ((value is None and col.nullable) or (isinstance(value, col.type.python_type))): - raise TypeError(f'{cls.__table__}.{key} {value!r} has invalid type {type(value).__name__!r}', cls, key, data) - else: - rel = cls.__mapper__.relationships.get(key) - if rel is None: - itype = getattr(cls, '_dict_types', {}).get(key) - if itype is not None: - if itype is False: # ignore value. TODO: emit warning? - del data[key] - continue - elif not isinstance(value, itype): - raise TypeError(f'{cls.__table__}.{key} {value!r} has invalid type {type(value).__name__!r}', cls, key, data) - else: - raise NotImplementedError(f'type not defined for {cls.__table__}.{key}') + # # remember changes + # new = sorted(new, key=id) + # if new != old: + # updated.append((key, old, new)) - # handle relationships - if key in cls.__mapper__.relationships: - rel_model = cls.__mapper__.relationships[key].argument - if not isinstance(rel_model, sqlalchemy.orm.Mapper): - add = rel_model.from_dict(value, delete) - assert len(add) == 1 - rel_item, updated = add[0] - changed.append((rel_item, updated)) - data[key] = rel_item + # else: + # # update key + # old = getattr(item, key) + # if isinstance(old, list): + # # deduplicate list value + # assert isinstance(value, list) + # value = set(value) + # old = set(old) + # if not delete: + # value = old | value + # if value != old: + # updated.append((key, old, value)) + # setattr(item, key, value) - # create item if necessary - created = False - item = cls.query.get(data[pkey]) if pkey in data else None - if item is None: + # changed.append((item, created if created else updated)) - # check for mandatory keys - missing = getattr(cls, '_dict_mandatory', set()) - set(data.keys()) - if missing: - raise ValueError(f'mandatory key(s) {", ".join(sorted(missing))} for {cls.__table__} missing', cls, missing, data) - - # remove mapped relationships from data - mapped = {} - for key in list(data.keys()): - if key in cls.__mapper__.relationships: - if isinstance(cls.__mapper__.relationships[key].argument, sqlalchemy.orm.Mapper): - mapped[key] = data[key] - del data[key] - - # create new item - item = cls(**data) - created = True - - # and update mapped relationships (below) - data = mapped - - # update item - updated = [] - for key, value in data.items(): - - # skip primary key - if key == pkey: - continue - - if key in cls.__mapper__.relationships: - # update relationship - rel_model = cls.__mapper__.relationships[key].argument - if isinstance(rel_model, sqlalchemy.orm.Mapper): - rel_model = rel_model.class_ - # add (and create) referenced items - cur = getattr(item, key) - old = sorted(cur, key=id) - new = [] - for rel_data in value: - # get or create related item - add = rel_model.from_dict(rel_data, delete) - assert len(add) == 1 - rel_item, rel_updated = add[0] - changed.append((rel_item, rel_updated)) - if rel_item not in cur: - cur.append(rel_item) - new.append(rel_item) - - # delete referenced items missing in yaml - rel_pkey = rel_model._dict_pkey() - new_data = list([i.to_dict(True, True, None, True, [rel_pkey]) for i in new]) - for rel_item in old: - if rel_item not in new: - # check if item with same data exists to stabilze import without primary key - rel_data = rel_item.to_dict(True, True, None, True, [rel_pkey]) - try: - same_idx = new_data.index(rel_data) - except ValueError: - same = None - else: - same = new[same_idx] - - if same is None: - # delete items missing in new - if delete: - cur.remove(rel_item) - else: - new.append(rel_item) - else: - # swap found item with same data with newly created item - new.append(rel_item) - new_data.append(rel_data) - new.remove(same) - del new_data[same_idx] - for i, (ch_item, _) in enumerate(changed): - if ch_item is same: - changed[i] = (rel_item, []) - db.session.flush() - db.session.delete(ch_item) - break - - # remember changes - new = sorted(new, key=id) - if new != old: - updated.append((key, old, new)) - - else: - # update key - old = getattr(item, key) - if isinstance(old, list): - # deduplicate list value - assert isinstance(value, list) - value = set(value) - old = set(old) - if not delete: - value = old | value - if value != old: - updated.append((key, old, value)) - setattr(item, key, value) - - changed.append((item, created if created else updated)) - - return changed + # return changed # Many-to-many association table for domain managers @@ -391,48 +328,6 @@ class Domain(Base): __tablename__ = 'domain' - _dict_hide = {'users', 'managers', 'aliases'} - _dict_show = {'dkim_key'} - _dict_extra = {'dns':{'dkim_publickey', 'dns_mx', 'dns_spf', 'dns_dkim', 'dns_dmarc'}} - _dict_secret = {'dkim_key'} - _dict_types = { - 'dkim_key': (bytes, type(None)), - 'dkim_publickey': False, - 'dns_mx': False, - 'dns_spf': False, - 'dns_dkim': False, - 'dns_dmarc': False, - } - _dict_output = {'dkim_key': lambda key: key.decode('utf-8').strip().split('\n')[1:-1]} - @staticmethod - def _dict_input(data): - if 'dkim_key' in data: - key = data['dkim_key'] - if key is not None: - if isinstance(key, list): - key = ''.join(key) - if isinstance(key, str): - key = ''.join(key.strip().split()) # removes all whitespace - if key == 'generate': - data['dkim_key'] = dkim.gen_key() - elif key: - match = re.match('^-----BEGIN (RSA )?PRIVATE KEY-----', key) - if match is not None: - key = key[match.end():] - match = re.search('-----END (RSA )?PRIVATE KEY-----$', key) - if match is not None: - key = key[:match.start()] - key = '\n'.join(wrap(key, 64)) - key = f'-----BEGIN PRIVATE KEY-----\n{key}\n-----END PRIVATE KEY-----\n'.encode('ascii') - try: - dkim.strip_key(key) - except: - raise ValueError('invalid dkim key') - else: - data['dkim_key'] = key - else: - data['dkim_key'] = None - name = db.Column(IdnaDomain, primary_key=True, nullable=False) managers = db.relationship('User', secondary=managers, backref=db.backref('manager_of'), lazy='dynamic') @@ -440,7 +335,7 @@ class Domain(Base): max_aliases = db.Column(db.Integer, nullable=False, default=-1) max_quota_bytes = db.Column(db.BigInteger, nullable=False, default=0) signup_enabled = db.Column(db.Boolean, nullable=False, default=False) - + _dkim_key = None _dkim_key_changed = False @@ -452,17 +347,20 @@ class Domain(Base): def dns_mx(self): hostname = app.config['HOSTNAMES'].split(',')[0] return f'{self.name}. 600 IN MX 10 {hostname}.' - + @property def dns_spf(self): hostname = app.config['HOSTNAMES'].split(',')[0] return f'{self.name}. 600 IN TXT "v=spf1 mx a:{hostname} ~all"' - + @property def dns_dkim(self): if os.path.exists(self._dkim_file()): selector = app.config['DKIM_SELECTOR'] - return f'{selector}._domainkey.{self.name}. 600 IN TXT "v=DKIM1; k=rsa; p={self.dkim_publickey}"' + return ( + f'{selector}._domainkey.{self.name}. 600 IN TXT' + f'"v=DKIM1; k=rsa; p={self.dkim_publickey}"' + ) @property def dns_dmarc(self): @@ -473,7 +371,7 @@ class Domain(Base): ruf = app.config['DMARC_RUF'] ruf = f' ruf=mailto:{ruf}@{domain};' if ruf else '' return f'_dmarc.{self.name}. 600 IN TXT "v=DMARC1; p=reject;{rua}{ruf} adkim=s; aspf=s"' - + @property def dkim_key(self): if self._dkim_key is None: @@ -525,7 +423,11 @@ class Domain(Base): try: return self.name == other.name except AttributeError: - return False + return NotImplemented + + def __hash__(self): + return hash(str(self.name)) + class Alternative(Base): @@ -551,8 +453,6 @@ class Relay(Base): __tablename__ = 'relay' - _dict_mandatory = {'smtp'} - name = db.Column(IdnaDomain, primary_key=True, nullable=False) smtp = db.Column(db.String(80), nullable=True) @@ -566,18 +466,8 @@ class Email(object): localpart = db.Column(db.String(80), nullable=False) - @staticmethod - def _dict_input(data): - if 'email' in data: - if 'localpart' in data or 'domain' in data: - raise ValueError('ambigous key email and localpart/domain') - elif isinstance(data['email'], str): - data['localpart'], data['domain'] = data['email'].rsplit('@', 1) - else: - data['email'] = f'{data["localpart"]}@{data["domain"]}' - @declarative.declared_attr - def domain_name(cls): + def domain_name(self): return db.Column(IdnaDomain, db.ForeignKey(Domain.name), nullable=False, default=IdnaDomain) @@ -585,7 +475,7 @@ class Email(object): # It is however very useful for quick lookups without joining tables, # especially when the mail server is reading the database. @declarative.declared_attr - def email(cls): + def email(self): updater = lambda context: '{0}@{1}'.format( context.current_parameters['localpart'], context.current_parameters['domain_name'], @@ -662,30 +552,6 @@ class User(Base, Email): __tablename__ = 'user' - _dict_hide = {'domain_name', 'domain', 'localpart', 'quota_bytes_used'} - _dict_mandatory = {'localpart', 'domain', 'password'} - @classmethod - def _dict_input(cls, data): - Email._dict_input(data) - # handle password - if 'password' in data: - if 'password_hash' in data or 'hash_scheme' in data: - raise ValueError('ambigous key password and password_hash/hash_scheme') - # check (hashed) password - password = data['password'] - if password.startswith('{') and '}' in password: - scheme = password[1:password.index('}')] - if scheme not in cls.scheme_dict: - raise ValueError(f'invalid password scheme {scheme!r}') - else: - raise ValueError(f'invalid hashed password {password!r}') - elif 'password_hash' in data and 'hash_scheme' in data: - if data['hash_scheme'] not in cls.scheme_dict: - raise ValueError(f'invalid password scheme {scheme!r}') - data['password'] = '{'+data['hash_scheme']+'}'+ data['password_hash'] - del data['hash_scheme'] - del data['password_hash'] - domain = db.relationship(Domain, backref=db.backref('users', cascade='all, delete-orphan')) password = db.Column(db.String(255), nullable=False) @@ -775,7 +641,8 @@ class User(Base, Email): if raw: self.password = '{'+hash_scheme+'}' + password else: - self.password = '{'+hash_scheme+'}' + self.get_password_context().encrypt(password, self.scheme_dict[hash_scheme]) + self.password = '{'+hash_scheme+'}' + \ + self.get_password_context().encrypt(password, self.scheme_dict[hash_scheme]) def get_managed_domains(self): if self.global_admin: @@ -812,15 +679,6 @@ class Alias(Base, Email): __tablename__ = 'alias' - _dict_hide = {'domain_name', 'domain', 'localpart'} - @staticmethod - def _dict_input(data): - Email._dict_input(data) - # handle comma delimited string for backwards compability - dst = data.get('destination') - if isinstance(dst, str): - data['destination'] = list([adr.strip() for adr in dst.split(',')]) - domain = db.relationship(Domain, backref=db.backref('aliases', cascade='all, delete-orphan')) wildcard = db.Column(db.Boolean, nullable=False, default=False) @@ -832,10 +690,10 @@ class Alias(Base, Email): sqlalchemy.and_(cls.domain_name == domain_name, sqlalchemy.or_( sqlalchemy.and_( - cls.wildcard == False, + cls.wildcard is False, cls.localpart == localpart ), sqlalchemy.and_( - cls.wildcard == True, + cls.wildcard is True, sqlalchemy.bindparam('l', localpart).like(cls.localpart) ) ) @@ -847,10 +705,10 @@ class Alias(Base, Email): sqlalchemy.and_(cls.domain_name == domain_name, sqlalchemy.or_( sqlalchemy.and_( - cls.wildcard == False, + cls.wildcard is False, sqlalchemy.func.lower(cls.localpart) == localpart_lower ), sqlalchemy.and_( - cls.wildcard == True, + cls.wildcard is True, sqlalchemy.bindparam('l', localpart_lower).like(sqlalchemy.func.lower(cls.localpart)) ) ) @@ -875,10 +733,6 @@ class Token(Base): __tablename__ = 'token' - _dict_recurse = True - _dict_hide = {'user', 'user_email'} - _dict_mandatory = {'password'} - id = db.Column(db.Integer, primary_key=True) user_email = db.Column(db.String(255), db.ForeignKey(User.email), nullable=False) @@ -904,11 +758,6 @@ class Fetch(Base): __tablename__ = 'fetch' - _dict_recurse = True - _dict_hide = {'user_email', 'user', 'last_check', 'error'} - _dict_mandatory = {'protocol', 'host', 'port', 'username', 'password'} - _dict_secret = {'password'} - id = db.Column(db.Integer, primary_key=True) user_email = db.Column(db.String(255), db.ForeignKey(User.email), nullable=False) @@ -926,3 +775,124 @@ class Fetch(Base): def __str__(self): return f'{self.protocol}{"s" if self.tls else ""}://{self.username}@{self.host}:{self.port}' + + +class MailuConfig: + """ Class which joins whole Mailu config for dumping + and loading + """ + + # TODO: add sqlalchemy session updating (.add & .del) + class MailuCollection: + """ Provides dict- and list-like access to all instances + of a sqlalchemy model + """ + + def __init__(self, model : db.Model): + self._model = model + + @cached_property + def _items(self): + return { + inspect(item).identity: item + for item in self._model.query.all() + } + + def __len__(self): + return len(self._items) + + def __iter__(self): + return iter(self._items.values()) + + def __getitem__(self, key): + return self._items[key] + + def __setitem__(self, key, item): + if not isinstance(item, self._model): + raise TypeError(f'expected {self._model.name}') + if key != inspect(item).identity: + raise ValueError(f'item identity != key {key!r}') + self._items[key] = item + + def __delitem__(self, key): + del self._items[key] + + def append(self, item): + """ list-like append """ + if not isinstance(item, self._model): + raise TypeError(f'expected {self._model.name}') + key = inspect(item).identity + if key in self._items: + raise ValueError(f'item {key!r} already present in collection') + self._items[key] = item + + def extend(self, items): + """ list-like extend """ + add = {} + for item in items: + if not isinstance(item, self._model): + raise TypeError(f'expected {self._model.name}') + key = inspect(item).identity + if key in self._items: + raise ValueError(f'item {key!r} already present in collection') + add[key] = item + self._items.update(add) + + def pop(self, *args): + """ list-like (no args) and dict-like (1 or 2 args) pop """ + if args: + if len(args) > 2: + raise TypeError(f'pop expected at most 2 arguments, got {len(args)}') + return self._items.pop(*args) + else: + return self._items.popitem()[1] + + def popitem(self): + """ dict-like popitem """ + return self._items.popitem() + + def remove(self, item): + """ list-like remove """ + if not isinstance(item, self._model): + raise TypeError(f'expected {self._model.name}') + key = inspect(item).identity + if not key in self._items: + raise ValueError(f'item {key!r} not found in collection') + del self._items[key] + + def clear(self): + """ dict-like clear """ + while True: + try: + self.pop() + except IndexError: + break + + def update(self, items): + """ dict-like update """ + for key, item in items: + if not isinstance(item, self._model): + raise TypeError(f'expected {self._model.name}') + if key != inspect(item).identity: + raise ValueError(f'item identity != key {key!r}') + if key in self._items: + raise ValueError(f'item {key!r} already present in collection') + + def setdefault(self, key, item=None): + """ dict-like setdefault """ + if key in self._items: + return self._items[key] + if item is None: + return None + if not isinstance(item, self._model): + raise TypeError(f'expected {self._model.name}') + if key != inspect(item).identity: + raise ValueError(f'item identity != key {key!r}') + self._items[key] = item + return item + + domains = MailuCollection(Domain) + relays = MailuCollection(Relay) + users = MailuCollection(User) + aliases = MailuCollection(Alias) + config = MailuCollection(Config) diff --git a/core/admin/mailu/schemas.py b/core/admin/mailu/schemas.py index bd9d228a..fc08b67c 100644 --- a/core/admin/mailu/schemas.py +++ b/core/admin/mailu/schemas.py @@ -1,13 +1,15 @@ +""" Mailu marshmallow fields and schema """ -Mailu marshmallow schema -""" + +import re from textwrap import wrap -import re import yaml -from marshmallow import post_dump, fields, Schema +from marshmallow import pre_load, post_dump, fields, Schema +from marshmallow.exceptions import ValidationError +from marshmallow_sqlalchemy import SQLAlchemyAutoSchemaOpts from flask_marshmallow import Marshmallow from OpenSSL import crypto @@ -15,9 +17,9 @@ from . import models, dkim ma = Marshmallow() -# TODO: -# how to mark keys as "required" while unserializing (in certain use cases/API)? -# - fields withoud default => required + +# TODO: how and where to mark keys as "required" while unserializing (on commandline, in api)? +# - fields without default => required # - fields which are the primary key => unchangeable when updating @@ -41,7 +43,7 @@ class RenderYAML: return super().increase_indent(flow, False) @staticmethod - def _update_dict(dict1, dict2): + def _update_items(dict1, dict2): """ sets missing keys in dict1 to values of dict2 """ for key, value in dict2.items(): @@ -53,8 +55,8 @@ class RenderYAML: def loads(cls, *args, **kwargs): """ load yaml data from string """ - cls._update_dict(kwargs, cls._load_defaults) - return yaml.load(*args, **kwargs) + cls._update_items(kwargs, cls._load_defaults) + return yaml.safe_load(*args, **kwargs) _dump_defaults = { 'Dumper': SpacedDumper, @@ -65,13 +67,33 @@ class RenderYAML: def dumps(cls, *args, **kwargs): """ dump yaml data to string """ - cls._update_dict(kwargs, cls._dump_defaults) + cls._update_items(kwargs, cls._dump_defaults) return yaml.dump(*args, **kwargs) +### functions ### + +def handle_email(data): + """ merge separate localpart and domain to email + """ + + localpart = 'localpart' in data + domain = 'domain' in data + + if 'email' in data: + if localpart or domain: + raise ValidationError('duplicate email and localpart/domain') + elif localpart and domain: + data['email'] = f'{data["localpart"]}@{data["domain"]}' + elif localpart or domain: + raise ValidationError('incomplete localpart/domain') + + return data + + ### field definitions ### -class LazyString(fields.String): +class LazyStringField(fields.String): """ Field that serializes a "false" value to the empty string """ @@ -81,14 +103,27 @@ class LazyString(fields.String): return value if value else '' -class CommaSeparatedList(fields.Raw): +class CommaSeparatedListField(fields.Raw): """ Field that deserializes a string containing comma-separated values to a list of strings """ - # TODO: implement this + + def _deserialize(self, value, attr, data, **kwargs): + """ deserialize comma separated string to list of strings + """ + + # empty + if not value: + return [] + + # split string + if isinstance(value, str): + return list([item.strip() for item in value.split(',') if item.strip()]) + else: + return value -class DkimKey(fields.String): +class DkimKeyField(fields.String): """ Field that serializes a dkim key to a list of strings (lines) and deserializes a string or list of strings. """ @@ -120,7 +155,7 @@ class DkimKey(fields.String): # only strings are allowed if not isinstance(value, str): - raise TypeError(f'invalid type: {type(value).__name__!r}') + raise ValidationError(f'invalid type {type(value).__name__!r}') # clean value (remove whitespace and header/footer) value = self._clean_re.sub('', value.strip()) @@ -133,6 +168,11 @@ class DkimKey(fields.String): elif value == 'generate': return dkim.gen_key() + # remember some keydata for error message + keydata = value + if len(keydata) > 40: + keydata = keydata[:25] + '...' + keydata[-10:] + # wrap value into valid pem layout and check validity value = ( '-----BEGIN PRIVATE KEY-----\n' + @@ -142,26 +182,37 @@ class DkimKey(fields.String): try: crypto.load_privatekey(crypto.FILETYPE_PEM, value) except crypto.Error as exc: - raise ValueError('invalid dkim key') from exc + raise ValidationError(f'invalid dkim key {keydata!r}') from exc else: return value -### schema definitions ### +### base definitions ### + +class BaseOpts(SQLAlchemyAutoSchemaOpts): + """ Option class with sqla session + """ + def __init__(self, meta, ordered=False): + if not hasattr(meta, 'sqla_session'): + meta.sqla_session = models.db.session + super(BaseOpts, self).__init__(meta, ordered=ordered) class BaseSchema(ma.SQLAlchemyAutoSchema): """ Marshmallow base schema with custom exclude logic and option to hide sqla defaults """ + OPTIONS_CLASS = BaseOpts + class Meta: """ Schema config """ model = None def __init__(self, *args, **kwargs): - # get and remove config from kwargs + # context? context = kwargs.get('context', {}) + flags = set([key for key, value in context.items() if value is True]) # compile excludes exclude = set(kwargs.get('exclude', [])) @@ -171,8 +222,8 @@ class BaseSchema(ma.SQLAlchemyAutoSchema): # add include_by_context if context is not None: - for ctx, what in getattr(self.Meta, 'include_by_context', {}).items(): - if not context.get(ctx): + for need, what in getattr(self.Meta, 'include_by_context', {}).items(): + if not flags & set(need): exclude |= set(what) # update excludes @@ -192,8 +243,8 @@ class BaseSchema(ma.SQLAlchemyAutoSchema): # hide by context self._hide_by_context = set() if context is not None: - for ctx, what in getattr(self.Meta, 'hide_by_context', {}).items(): - if not context.get(ctx): + for need, what in getattr(self.Meta, 'hide_by_context', {}).items(): + if not flags & set(need): self._hide_by_context |= set(what) # init SQLAlchemyAutoSchema @@ -212,23 +263,26 @@ class BaseSchema(ma.SQLAlchemyAutoSchema): if full or key not in self._exclude_by_value or value not in self._exclude_by_value[key] } - # TODO: remove LazyString and fix model definition (comment should not be nullable) - comment = LazyString() + # TODO: remove LazyString and change model (IMHO comment should not be nullable) + comment = LazyStringField() + + +### schema definitions ### class DomainSchema(BaseSchema): """ Marshmallow schema for Domain model """ class Meta: """ Schema config """ model = models.Domain + load_instance = True include_relationships = True - #include_fk = True exclude = ['users', 'managers', 'aliases'] include_by_context = { - 'dns': {'dkim_publickey', 'dns_mx', 'dns_spf', 'dns_dkim', 'dns_dmarc'}, + ('dns',): {'dkim_publickey', 'dns_mx', 'dns_spf', 'dns_dkim', 'dns_dmarc'}, } hide_by_context = { - 'secrets': {'dkim_key'}, + ('secrets',): {'dkim_key'}, } exclude_by_value = { 'alternatives': [[]], @@ -240,40 +294,20 @@ class DomainSchema(BaseSchema): 'dns_dmarc': [None], } - dkim_key = DkimKey() + dkim_key = DkimKeyField(allow_none=True) dkim_publickey = fields.String(dump_only=True) dns_mx = fields.String(dump_only=True) dns_spf = fields.String(dump_only=True) dns_dkim = fields.String(dump_only=True) dns_dmarc = fields.String(dump_only=True) - # _dict_types = { - # 'dkim_key': (bytes, type(None)), - # 'dkim_publickey': False, - # 'dns_mx': False, - # 'dns_spf': False, - # 'dns_dkim': False, - # 'dns_dmarc': False, - # } - class TokenSchema(BaseSchema): """ Marshmallow schema for Token model """ class Meta: """ Schema config """ model = models.Token - - # _dict_recurse = True - # _dict_hide = {'user', 'user_email'} - # _dict_mandatory = {'password'} - - # id = db.Column(db.Integer(), primary_key=True) - # user_email = db.Column(db.String(255), db.ForeignKey(User.email), - # nullable=False) - # user = db.relationship(User, - # backref=db.backref('tokens', cascade='all, delete-orphan')) - # password = db.Column(db.String(255), nullable=False) - # ip = db.Column(db.String(255)) + load_instance = True class FetchSchema(BaseSchema): @@ -281,58 +315,57 @@ class FetchSchema(BaseSchema): class Meta: """ Schema config """ model = models.Fetch + load_instance = True include_by_context = { - 'full': {'last_check', 'error'}, + ('full', 'import'): {'last_check', 'error'}, } hide_by_context = { - 'secrets': {'password'}, + ('secrets',): {'password'}, } -# TODO: What about mandatory keys? - # _dict_mandatory = {'protocol', 'host', 'port', 'username', 'password'} - class UserSchema(BaseSchema): """ Marshmallow schema for User model """ class Meta: """ Schema config """ model = models.User + load_instance = True include_relationships = True exclude = ['localpart', 'domain', 'quota_bytes_used'] exclude_by_value = { 'forward_destination': [[]], 'tokens': [[]], + 'manager_of': [[]], 'reply_enddate': ['2999-12-31'], 'reply_startdate': ['1900-01-01'], } + @pre_load + def _handle_password(self, data, many, **kwargs): # pylint: disable=unused-argument + data = handle_email(data) + if 'password' in data: + if 'password_hash' in data or 'hash_scheme' in data: + raise ValidationError('ambigous key password and password_hash/hash_scheme') + # check (hashed) password + password = data['password'] + if password.startswith('{') and '}' in password: + scheme = password[1:password.index('}')] + if scheme not in self.Meta.model.scheme_dict: + raise ValidationError(f'invalid password scheme {scheme!r}') + else: + raise ValidationError(f'invalid hashed password {password!r}') + elif 'password_hash' in data and 'hash_scheme' in data: + if data['hash_scheme'] not in self.Meta.model.scheme_dict: + raise ValidationError(f'invalid password scheme {scheme!r}') + data['password'] = '{'+data['hash_scheme']+'}'+ data['password_hash'] + del data['hash_scheme'] + del data['password_hash'] + return data + tokens = fields.Nested(TokenSchema, many=True) fetches = fields.Nested(FetchSchema, many=True) -# TODO: deserialize password/password_hash! What about mandatory keys? - # _dict_mandatory = {'localpart', 'domain', 'password'} - # @classmethod - # def _dict_input(cls, data): - # Email._dict_input(data) - # # handle password - # if 'password' in data: - # if 'password_hash' in data or 'hash_scheme' in data: - # raise ValueError('ambigous key password and password_hash/hash_scheme') - # # check (hashed) password - # password = data['password'] - # if password.startswith('{') and '}' in password: - # scheme = password[1:password.index('}')] - # if scheme not in cls.scheme_dict: - # raise ValueError(f'invalid password scheme {scheme!r}') - # else: - # raise ValueError(f'invalid hashed password {password!r}') - # elif 'password_hash' in data and 'hash_scheme' in data: - # if data['hash_scheme'] not in cls.scheme_dict: - # raise ValueError(f'invalid password scheme {scheme!r}') - # data['password'] = '{'+data['hash_scheme']+'}'+ data['password_hash'] - # del data['hash_scheme'] - # del data['password_hash'] class AliasSchema(BaseSchema): @@ -340,20 +373,18 @@ class AliasSchema(BaseSchema): class Meta: """ Schema config """ model = models.Alias + load_instance = True exclude = ['localpart'] exclude_by_value = { 'destination': [[]], } -# TODO: deserialize destination! - # @staticmethod - # def _dict_input(data): - # Email._dict_input(data) - # # handle comma delimited string for backwards compability - # dst = data.get('destination') - # if type(dst) is str: - # data['destination'] = list([adr.strip() for adr in dst.split(',')]) + @pre_load + def _handle_password(self, data, many, **kwargs): # pylint: disable=unused-argument + return handle_email(data) + + destination = CommaSeparatedListField() class ConfigSchema(BaseSchema): @@ -361,6 +392,7 @@ class ConfigSchema(BaseSchema): class Meta: """ Schema config """ model = models.Config + load_instance = True class RelaySchema(BaseSchema): @@ -368,45 +400,17 @@ class RelaySchema(BaseSchema): class Meta: """ Schema config """ model = models.Relay + load_instance = True class MailuSchema(Schema): - """ Marshmallow schema for Mailu config """ + """ Marshmallow schema for complete Mailu config """ class Meta: """ Schema config """ render_module = RenderYAML + domains = fields.Nested(DomainSchema, many=True) relays = fields.Nested(RelaySchema, many=True) users = fields.Nested(UserSchema, many=True) aliases = fields.Nested(AliasSchema, many=True) config = fields.Nested(ConfigSchema, many=True) - - -### config class ### - -class MailuConfig: - """ Class which joins whole Mailu config for dumping - """ - - _models = { - 'domains': models.Domain, - 'relays': models.Relay, - 'users': models.User, - 'aliases': models.Alias, -# 'config': models.Config, - } - - def __init__(self, sections): - if sections: - for section in sections: - if section not in self._models: - raise ValueError(f'Unknown section: {section!r}') - self._sections = set(sections) - else: - self._sections = set(self._models.keys()) - - def __getattr__(self, section): - if section in self._sections: - return self._models[section].query.all() - else: - raise AttributeError From 31a903f95959e4234f21b9c5e059a73d37d8ec46 Mon Sep 17 00:00:00 2001 From: Alexander Graf Date: Fri, 15 Jan 2021 13:45:36 +0100 Subject: [PATCH 055/596] revived & renamed config-fns. cosmetics. - revived original config-update function for backwards compability - renamed config-dump to config-export to be in line with config-import - converted '*'.format(*) to f-strings - converted string-concatenation to f-strings --- core/admin/mailu/manage.py | 310 +++++++++++++++++++++---------------- 1 file changed, 176 insertions(+), 134 deletions(-) diff --git a/core/admin/mailu/manage.py b/core/admin/mailu/manage.py index 6103f904..80a73230 100644 --- a/core/admin/mailu/manage.py +++ b/core/admin/mailu/manage.py @@ -7,6 +7,7 @@ import socket import uuid import click +import yaml from flask import current_app as app from flask.cli import FlaskGroup, with_appcontext @@ -64,7 +65,7 @@ def admin(localpart, domain_name, password, mode='create'): user = None if mode == 'ifmissing' or mode == 'update': - email = '{}@{}'.format(localpart, domain_name) + email = f'{localpart}@{domain_name}' user = models.User.query.get(email) if user and mode == 'ifmissing': @@ -122,14 +123,14 @@ def user(localpart, domain_name, password, hash_scheme=None): def password(localpart, domain_name, password, hash_scheme=None): """ Change the password of an user """ - email = '{0}@{1}'.format(localpart, domain_name) - user = models.User.query.get(email) + email = f'{localpart}@{domain_name}' + user = models.User.query.get(email) if hash_scheme is None: hash_scheme = app.config['PASSWORD_SCHEME'] if user: user.set_password(password, hash_scheme=hash_scheme) else: - print("User " + email + " not found.") + print(f'User {email} not found.') db.session.commit() @@ -157,7 +158,7 @@ def domain(domain_name, max_users=-1, max_aliases=-1, max_quota_bytes=0): @click.argument('hash_scheme') @with_appcontext def user_import(localpart, domain_name, password_hash, hash_scheme = None): - """ Import a user along with password hash. + """ Import a user along with password hash """ if hash_scheme is None: hash_scheme = app.config['PASSWORD_SCHEME'] @@ -175,124 +176,146 @@ def user_import(localpart, domain_name, password_hash, hash_scheme = None): db.session.commit() -# @mailu.command() -# @click.option('-v', '--verbose', is_flag=True, help='Increase verbosity') -# @click.option('-d', '--delete-objects', is_flag=True, help='Remove objects not included in yaml') -# @click.option('-n', '--dry-run', is_flag=True, help='Perform a trial run with no changes made') -# @click.argument('source', metavar='[FILENAME|-]', type=click.File(mode='r'), default=sys.stdin) -# @with_appcontext -# def config_update(verbose=False, delete_objects=False, dry_run=False, source=None): -# """ Update configuration with data from YAML-formatted input -# """ +# TODO: remove this deprecated function +@mailu.command() +@click.option('-v', '--verbose') +@click.option('-d', '--delete-objects') +@with_appcontext +def config_update(verbose=False, delete_objects=False): + """ Sync configuration with data from YAML (deprecated) + """ + new_config = yaml.safe_load(sys.stdin) + # print new_config + domains = new_config.get('domains', []) + tracked_domains = set() + for domain_config in domains: + if verbose: + print(str(domain_config)) + domain_name = domain_config['name'] + max_users = domain_config.get('max_users', -1) + max_aliases = domain_config.get('max_aliases', -1) + max_quota_bytes = domain_config.get('max_quota_bytes', 0) + tracked_domains.add(domain_name) + domain = models.Domain.query.get(domain_name) + if not domain: + domain = models.Domain(name=domain_name, + max_users=max_users, + max_aliases=max_aliases, + max_quota_bytes=max_quota_bytes) + db.session.add(domain) + print(f'Added {domain_config}') + else: + domain.max_users = max_users + domain.max_aliases = max_aliases + domain.max_quota_bytes = max_quota_bytes + db.session.add(domain) + print(f'Updated {domain_config}') - # try: - # new_config = yaml.safe_load(source) - # except (yaml.scanner.ScannerError, yaml.parser.ParserError) as exc: - # print(f'[ERROR] Invalid yaml: {exc}') - # sys.exit(1) - # else: - # if isinstance(new_config, str): - # print(f'[ERROR] Invalid yaml: {new_config!r}') - # sys.exit(1) - # elif new_config is None or not new_config: - # print('[ERROR] Empty yaml: Please pipe yaml into stdin') - # sys.exit(1) + users = new_config.get('users', []) + tracked_users = set() + user_optional_params = ('comment', 'quota_bytes', 'global_admin', + 'enable_imap', 'enable_pop', 'forward_enabled', + 'forward_destination', 'reply_enabled', + 'reply_subject', 'reply_body', 'displayed_name', + 'spam_enabled', 'email', 'spam_threshold') + for user_config in users: + if verbose: + print(str(user_config)) + localpart = user_config['localpart'] + domain_name = user_config['domain'] + password_hash = user_config.get('password_hash', None) + hash_scheme = user_config.get('hash_scheme', None) + domain = models.Domain.query.get(domain_name) + email = f'{localpart}@{domain_name}' + optional_params = {} + for k in user_optional_params: + if k in user_config: + optional_params[k] = user_config[k] + if not domain: + domain = models.Domain(name=domain_name) + db.session.add(domain) + user = models.User.query.get(email) + tracked_users.add(email) + tracked_domains.add(domain_name) + if not user: + user = models.User( + localpart=localpart, + domain=domain, + **optional_params + ) + else: + for k in optional_params: + setattr(user, k, optional_params[k]) + user.set_password(password_hash, hash_scheme=hash_scheme, raw=True) + db.session.add(user) - # error = False - # tracked = {} - # for section, model in yaml_sections: + aliases = new_config.get('aliases', []) + tracked_aliases = set() + for alias_config in aliases: + if verbose: + print(str(alias_config)) + localpart = alias_config['localpart'] + domain_name = alias_config['domain'] + if isinstance(alias_config['destination'], str): + destination = alias_config['destination'].split(',') + else: + destination = alias_config['destination'] + wildcard = alias_config.get('wildcard', False) + domain = models.Domain.query.get(domain_name) + email = f'{localpart}@{domain_name}' + if not domain: + domain = models.Domain(name=domain_name) + db.session.add(domain) + alias = models.Alias.query.get(email) + tracked_aliases.add(email) + tracked_domains.add(domain_name) + if not alias: + alias = models.Alias( + localpart=localpart, + domain=domain, + wildcard=wildcard, + destination=destination, + email=email + ) + else: + alias.destination = destination + alias.wildcard = wildcard + db.session.add(alias) - # items = new_config.get(section) - # if items is None: - # if delete_objects: - # print(f'[ERROR] Invalid yaml: Section "{section}" is missing') - # error = True - # break - # else: - # continue + db.session.commit() - # del new_config[section] + managers = new_config.get('managers', []) + # tracked_managers=set() + for manager_config in managers: + if verbose: + print(str(manager_config)) + domain_name = manager_config['domain'] + user_name = manager_config['user'] + domain = models.Domain.query.get(domain_name) + manageruser = models.User.query.get(f'{user_name}@{domain_name}') + if manageruser not in domain.managers: + domain.managers.append(manageruser) + db.session.add(domain) - # if not isinstance(items, list): - # print(f'[ERROR] Section "{section}" must be a list, not {items.__class__.__name__}') - # error = True - # break - # elif not items: - # continue + db.session.commit() - # # create items - # for data in items: - - # if verbose: - # print(f'Handling {model.__table__} data: {data!r}') - - # try: - # changed = model.from_dict(data, delete_objects) - # except Exception as exc: - # print(f'[ERROR] {exc.args[0]} in data: {data}') - # error = True - # break - - # for item, created in changed: - - # if created is True: - # # flush newly created item - # db.session.add(item) - # db.session.flush() - # if verbose: - # print(f'Added {item!r}: {item.to_dict()}') - # else: - # print(f'Added {item!r}') - - # elif created: - # # modified instance - # if verbose: - # for key, old, new in created: - # print(f'Updated {key!r} of {item!r}: {old!r} -> {new!r}') - # else: - # print(f'Updated {item!r}: {", ".join(sorted([kon[0] for kon in created]))}') - - # # track primary key of all items - # tracked.setdefault(item.__class__, set()).update(set([item._dict_pval()])) - - # if error: - # break - - # # on error: stop early - # if error: - # print('[ERROR] An error occured. Not committing changes.') - # db.session.rollback() - # sys.exit(1) - - # # are there sections left in new_config? - # if new_config: - # print(f'[ERROR] Unknown section(s) in yaml: {", ".join(sorted(new_config.keys()))}') - # error = True - - # # test for conflicting domains - # domains = set() - # for model, items in tracked.items(): - # if model in (models.Domain, models.Alternative, models.Relay): - # if domains & items: - # for fqdn in domains & items: - # print(f'[ERROR] Duplicate domain name used: {fqdn}') - # error = True - # domains.update(items) - - # # delete items not tracked - # if delete_objects: - # for model, items in tracked.items(): - # for item in model.query.all(): - # if not item._dict_pval() in items: - # print(f'Deleted {item!r} {item}') - # db.session.delete(item) - - # # don't commit when running dry - # if dry_run: - # print('Dry run. Not commiting changes.') - # db.session.rollback() - # else: - # db.session.commit() + if delete_objects: + for user in db.session.query(models.User).all(): + if not user.email in tracked_users: + if verbose: + print(f'Deleting user: {user.email}') + db.session.delete(user) + for alias in db.session.query(models.Alias).all(): + if not alias.email in tracked_aliases: + if verbose: + print(f'Deleting alias: {alias.email}') + db.session.delete(alias) + for domain in db.session.query(models.Domain).all(): + if not domain.name in tracked_domains: + if verbose: + print(f'Deleting domain: {domain.name}') + db.session.delete(domain) + db.session.commit() SECTIONS = {'domains', 'relays', 'users', 'aliases'} @@ -304,30 +327,51 @@ SECTIONS = {'domains', 'relays', 'users', 'aliases'} @click.argument('source', metavar='[FILENAME|-]', type=click.File(mode='r'), default=sys.stdin) @with_appcontext def config_import(verbose=False, dry_run=False, source=None): - """ Import configuration YAML + """ Import configuration from YAML """ + def log(**data): + caller = sys._getframe(1).f_code.co_name # pylint: disable=protected-access + if caller == '_track_import': + print(f'Handling {data["self"].opts.model.__table__} data: {data["data"]!r}') + + def format_errors(store, path=None): + if path is None: + path = [] + for key in sorted(store): + location = path + [str(key)] + value = store[key] + if isinstance(value, dict): + format_errors(value, location) + else: + for message in value: + print(f'[ERROR] {".".join(location)}: {message}') + context = { - 'verbose': verbose, # TODO: use callback function to be verbose? + 'callback': log if verbose else None, 'import': True, } + error = False try: config = MailuSchema(context=context).loads(source) except ValidationError as exc: - print(f'[ERROR] {exc}') - # TODO: show nice errors - from pprint import pprint - pprint(exc.messages) - sys.exit(1) + error = True + format_errors(exc.messages) else: print(config) print(MailuSchema().dumps(config)) - # TODO: does not commit yet. - # TODO: delete other entries? + # TODO: need to delete other entries - # don't commit when running dry - if True: #dry_run: + # TODO: enable commit + error = True + + # don't commit when running dry or validation errors occured + if error: + print('An error occured. Not committing changes.') + db.session.rollback() + sys.exit(2) + elif dry_run: print('Dry run. Not commiting changes.') db.session.rollback() else: @@ -343,10 +387,8 @@ def config_import(verbose=False, dry_run=False, source=None): help='save yaml to file') @click.argument('sections', nargs=-1) @with_appcontext -def config_dump(full=False, secrets=False, dns=False, output=None, sections=None): - """ Dump configuration as YAML to stdout or file - - SECTIONS can be: domains, relays, users, aliases +def config_export(full=False, secrets=False, dns=False, output=None, sections=None): + """ Export configuration as YAML to stdout or file """ if sections: @@ -407,7 +449,7 @@ def alias(localpart, domain_name, destination, wildcard=False): domain=domain, wildcard=wildcard, destination=destination.split(','), - email="%s@%s" % (localpart, domain_name) + email=f'{localpart}@{domain_name}' ) db.session.add(alias) db.session.commit() @@ -438,7 +480,7 @@ def setmanager(domain_name, user_name='manager'): """ Make a user manager of a domain """ domain = models.Domain.query.get(domain_name) - manageruser = models.User.query.get(user_name + '@' + domain_name) + manageruser = models.User.query.get(f'{user_name}@{domain_name}') domain.managers.append(manageruser) db.session.add(domain) db.session.commit() From 8213d044b25860046381c6766220c0855616b88c Mon Sep 17 00:00:00 2001 From: Alexander Graf Date: Fri, 15 Jan 2021 13:53:47 +0100 Subject: [PATCH 056/596] added docstrings, use f-strings, cleanup - idna.encode does not encode upper-case letters, so .lower() has to be called on value not on result - split email-address on '@' only once - converted '*'.format(*) to f-strings - added docstrings - removed from_dict method - code cleanup/style (list concat, exceptions, return&else, line-length) - added TODO comments on possible future changes --- core/admin/mailu/models.py | 376 ++++++++++++------------------------- 1 file changed, 122 insertions(+), 254 deletions(-) diff --git a/core/admin/mailu/models.py b/core/admin/mailu/models.py index 73d05801..3187f597 100644 --- a/core/admin/mailu/models.py +++ b/core/admin/mailu/models.py @@ -8,6 +8,7 @@ import json from datetime import date from email.mime import text +from itertools import chain import flask_sqlalchemy import sqlalchemy @@ -30,11 +31,12 @@ class IdnaDomain(db.TypeDecorator): """ Stores a Unicode string in it's IDNA representation (ASCII only) """ + # TODO: String(80) is too small? impl = db.String(80) def process_bind_param(self, value, dialect): """ encode unicode domain name to punycode """ - return idna.encode(value).decode('ascii').lower() + return idna.encode(value.lower()).decode('ascii') def process_result_value(self, value, dialect): """ decode punycode domain name to unicode """ @@ -46,26 +48,21 @@ class IdnaEmail(db.TypeDecorator): """ Stores a Unicode string in it's IDNA representation (ASCII only) """ + # TODO: String(255) is too small? impl = db.String(255) def process_bind_param(self, value, dialect): """ encode unicode domain part of email address to punycode """ - try: - localpart, domain_name = value.split('@') - return '{0}@{1}'.format( - localpart, - idna.encode(domain_name).decode('ascii'), - ).lower() - except ValueError: - pass + localpart, domain_name = value.rsplit('@', 1) + if '@' in localpart: + raise ValueError('email local part must not contain "@"') + domain_name = domain_name.lower() + return f'{localpart}@{idna.encode(domain_name).decode("ascii")}' def process_result_value(self, value, dialect): """ decode punycode domain part of email to unicode """ - localpart, domain_name = value.split('@') - return '{0}@{1}'.format( - localpart, - idna.decode(domain_name), - ) + localpart, domain_name = value.rsplit('@', 1) + return f'{localpart}@{idna.decode(domain_name)}' python_type = str @@ -81,7 +78,7 @@ class CommaSeparatedList(db.TypeDecorator): raise TypeError('Must be a list of strings') for item in value: if ',' in item: - raise ValueError('Item must not contain a comma') + raise ValueError('list item must not contain ","') return ','.join(sorted(value)) def process_result_value(self, value, dialect): @@ -123,173 +120,6 @@ class Base(db.Model): updated_at = db.Column(db.Date, nullable=True, onupdate=date.today) comment = db.Column(db.String(255), nullable=True, default='') - # @classmethod - # def from_dict(cls, data, delete=False): - - # changed = [] - - # pkey = cls._dict_pkey() - - # # handle "primary key" only - # if not isinstance(data, dict): - # data = {pkey: data} - - # # modify input data - # if hasattr(cls, '_dict_input'): - # try: - # cls._dict_input(data) - # except Exception as exc: - # raise ValueError(f'{exc}', cls, None, data) from exc - - # # check for primary key (if not recursed) - # if not getattr(cls, '_dict_recurse', False): - # if not pkey in data: - # raise KeyError(f'primary key {cls.__table__}.{pkey} is missing', cls, pkey, data) - - # # check data keys and values - # for key in list(data.keys()): - - # # check key - # if not hasattr(cls, key) and not key in cls.__mapper__.relationships: - # raise KeyError(f'unknown key {cls.__table__}.{key}', cls, key, data) - - # # check value type - # value = data[key] - # col = cls.__mapper__.columns.get(key) - # if col is not None: - # if not ((value is None and col.nullable) or (isinstance(value, col.type.python_type))): - # raise TypeError(f'{cls.__table__}.{key} {value!r} has invalid type {type(value).__name__!r}', cls, key, data) - # else: - # rel = cls.__mapper__.relationships.get(key) - # if rel is None: - # itype = getattr(cls, '_dict_types', {}).get(key) - # if itype is not None: - # if itype is False: # ignore value. TODO: emit warning? - # del data[key] - # continue - # elif not isinstance(value, itype): - # raise TypeError(f'{cls.__table__}.{key} {value!r} has invalid type {type(value).__name__!r}', cls, key, data) - # else: - # raise NotImplementedError(f'type not defined for {cls.__table__}.{key}') - - # # handle relationships - # if key in cls.__mapper__.relationships: - # rel_model = cls.__mapper__.relationships[key].argument - # if not isinstance(rel_model, sqlalchemy.orm.Mapper): - # add = rel_model.from_dict(value, delete) - # assert len(add) == 1 - # rel_item, updated = add[0] - # changed.append((rel_item, updated)) - # data[key] = rel_item - - # # create item if necessary - # created = False - # item = cls.query.get(data[pkey]) if pkey in data else None - # if item is None: - - # # check for mandatory keys - # missing = getattr(cls, '_dict_mandatory', set()) - set(data.keys()) - # if missing: - # raise ValueError(f'mandatory key(s) {", ".join(sorted(missing))} for {cls.__table__} missing', cls, missing, data) - - # # remove mapped relationships from data - # mapped = {} - # for key in list(data.keys()): - # if key in cls.__mapper__.relationships: - # if isinstance(cls.__mapper__.relationships[key].argument, sqlalchemy.orm.Mapper): - # mapped[key] = data[key] - # del data[key] - - # # create new item - # item = cls(**data) - # created = True - - # # and update mapped relationships (below) - # data = mapped - - # # update item - # updated = [] - # for key, value in data.items(): - - # # skip primary key - # if key == pkey: - # continue - - # if key in cls.__mapper__.relationships: - # # update relationship - # rel_model = cls.__mapper__.relationships[key].argument - # if isinstance(rel_model, sqlalchemy.orm.Mapper): - # rel_model = rel_model.class_ - # # add (and create) referenced items - # cur = getattr(item, key) - # old = sorted(cur, key=id) - # new = [] - # for rel_data in value: - # # get or create related item - # add = rel_model.from_dict(rel_data, delete) - # assert len(add) == 1 - # rel_item, rel_updated = add[0] - # changed.append((rel_item, rel_updated)) - # if rel_item not in cur: - # cur.append(rel_item) - # new.append(rel_item) - - # # delete referenced items missing in yaml - # rel_pkey = rel_model._dict_pkey() - # new_data = list([i.to_dict(True, True, None, True, [rel_pkey]) for i in new]) - # for rel_item in old: - # if rel_item not in new: - # # check if item with same data exists to stabilze import without primary key - # rel_data = rel_item.to_dict(True, True, None, True, [rel_pkey]) - # try: - # same_idx = new_data.index(rel_data) - # except ValueError: - # same = None - # else: - # same = new[same_idx] - - # if same is None: - # # delete items missing in new - # if delete: - # cur.remove(rel_item) - # else: - # new.append(rel_item) - # else: - # # swap found item with same data with newly created item - # new.append(rel_item) - # new_data.append(rel_data) - # new.remove(same) - # del new_data[same_idx] - # for i, (ch_item, _) in enumerate(changed): - # if ch_item is same: - # changed[i] = (rel_item, []) - # db.session.flush() - # db.session.delete(ch_item) - # break - - # # remember changes - # new = sorted(new, key=id) - # if new != old: - # updated.append((key, old, new)) - - # else: - # # update key - # old = getattr(item, key) - # if isinstance(old, list): - # # deduplicate list value - # assert isinstance(value, list) - # value = set(value) - # old = set(old) - # if not delete: - # value = old | value - # if value != old: - # updated.append((key, old, value)) - # setattr(item, key, value) - - # changed.append((item, created if created else updated)) - - # return changed - # Many-to-many association table for domain managers managers = db.Table('manager', Base.metadata, @@ -309,9 +139,7 @@ class Config(Base): # TODO: use sqlalchemy.event.listen() on a store method of object? @sqlalchemy.event.listens_for(db.session, 'after_commit') def store_dkim_key(session): - """ Store DKIM key on commit - """ - + """ Store DKIM key on commit """ for obj in session.identity_map.values(): if isinstance(obj, Domain): if obj._dkim_key_changed: @@ -340,21 +168,27 @@ class Domain(Base): _dkim_key_changed = False def _dkim_file(self): + """ return filename for active DKIM key """ return app.config['DKIM_PATH'].format( - domain=self.name, selector=app.config['DKIM_SELECTOR']) + domain=self.name, + selector=app.config['DKIM_SELECTOR'] + ) @property def dns_mx(self): - hostname = app.config['HOSTNAMES'].split(',')[0] + """ return MX record for domain """ + hostname = app.config['HOSTNAMES'].split(',', 1)[0] return f'{self.name}. 600 IN MX 10 {hostname}.' @property def dns_spf(self): - hostname = app.config['HOSTNAMES'].split(',')[0] + """ return SPF record for domain """ + hostname = app.config['HOSTNAMES'].split(',', 1)[0] return f'{self.name}. 600 IN TXT "v=spf1 mx a:{hostname} ~all"' @property def dns_dkim(self): + """ return DKIM record for domain """ if os.path.exists(self._dkim_file()): selector = app.config['DKIM_SELECTOR'] return ( @@ -364,6 +198,7 @@ class Domain(Base): @property def dns_dmarc(self): + """ return DMARC record for domain """ if os.path.exists(self._dkim_file()): domain = app.config['DOMAIN'] rua = app.config['DMARC_RUA'] @@ -374,6 +209,7 @@ class Domain(Base): @property def dkim_key(self): + """ return private DKIM key """ if self._dkim_key is None: file_path = self._dkim_file() if os.path.exists(file_path): @@ -385,6 +221,7 @@ class Domain(Base): @dkim_key.setter def dkim_key(self, value): + """ set private DKIM key """ old_key = self.dkim_key if value is None: value = b'' @@ -393,36 +230,40 @@ class Domain(Base): @property def dkim_publickey(self): + """ return public part of DKIM key """ dkim_key = self.dkim_key if dkim_key: return dkim.strip_key(dkim_key).decode('utf8') def generate_dkim_key(self): + """ generate and activate new DKIM key """ self.dkim_key = dkim.gen_key() def has_email(self, localpart): - for email in self.users + self.aliases: + """ checks if localpart is configured for domain """ + for email in chain(self.users, self.aliases): if email.localpart == localpart: return True return False def check_mx(self): + """ checks if MX record for domain points to mailu host """ try: - hostnames = app.config['HOSTNAMES'].split(',') + hostnames = set(app.config['HOSTNAMES'].split(',')) return any( - str(rset).split()[-1][:-1] in hostnames + rset.exchange.to_text().rstrip('.') in hostnames for rset in dns.resolver.query(self.name, 'MX') ) - except Exception: + except dns.exception.DNSException: return False def __str__(self): return str(self.name) def __eq__(self, other): - try: - return self.name == other.name - except AttributeError: + if isinstance(other, self.__class__): + return str(self.name) == str(other.name) + else: return NotImplemented def __hash__(self): @@ -432,7 +273,7 @@ class Domain(Base): class Alternative(Base): """ Alternative name for a served domain. - The name "domain alias" was avoided to prevent some confusion. + The name "domain alias" was avoided to prevent some confusion. """ __tablename__ = 'alternative' @@ -454,6 +295,7 @@ class Relay(Base): __tablename__ = 'relay' name = db.Column(IdnaDomain, primary_key=True, nullable=False) + # TODO: String(80) is too small? smtp = db.Column(db.String(80), nullable=True) def __str__(self): @@ -464,10 +306,14 @@ class Email(object): """ Abstraction for an email address (localpart and domain). """ + # TODO: validate max. total length of address (<=254) + + # TODO: String(80) is too large (>64)? localpart = db.Column(db.String(80), nullable=False) @declarative.declared_attr def domain_name(self): + """ the domain part of the email address """ return db.Column(IdnaDomain, db.ForeignKey(Domain.name), nullable=False, default=IdnaDomain) @@ -476,26 +322,18 @@ class Email(object): # especially when the mail server is reading the database. @declarative.declared_attr def email(self): - updater = lambda context: '{0}@{1}'.format( - context.current_parameters['localpart'], - context.current_parameters['domain_name'], - ) + """ the complete email address (localpart@domain) """ + updater = lambda ctx: '{localpart}@{domain_name}'.format(**ctx.current_parameters) return db.Column(IdnaEmail, primary_key=True, nullable=False, - default=updater) + default=updater + ) def sendmail(self, subject, body): - """ Send an email to the address. - """ - from_address = '{0}@{1}'.format( - app.config['POSTMASTER'], - idna.encode(app.config['DOMAIN']).decode('ascii'), - ) + """ send an email to the address """ + from_address = f'{app.config["POSTMASTER"]}@{idna.encode(app.config["DOMAIN"]).decode("ascii")}' with smtplib.SMTP(app.config['HOST_AUTHSMTP'], port=10025) as smtp: - to_address = '{0}@{1}'.format( - self.localpart, - idna.encode(self.domain_name).decode('ascii'), - ) + to_address = f'{self.localpart}@{idna.encode(self.domain_name).decode("ascii")}' msg = text.MIMEText(body) msg['Subject'] = subject msg['From'] = from_address @@ -504,7 +342,8 @@ class Email(object): @classmethod def resolve_domain(cls, email): - localpart, domain_name = email.split('@', 1) if '@' in email else (None, email) + """ resolves domain alternative to real domain """ + localpart, domain_name = email.rsplit('@', 1) if '@' in email else (None, email) alternative = Alternative.query.get(domain_name) if alternative: domain_name = alternative.domain_name @@ -512,17 +351,19 @@ class Email(object): @classmethod def resolve_destination(cls, localpart, domain_name, ignore_forward_keep=False): + """ return destination for email address localpart@domain_name """ + localpart_stripped = None stripped_alias = None if os.environ.get('RECIPIENT_DELIMITER') in localpart: localpart_stripped = localpart.rsplit(os.environ.get('RECIPIENT_DELIMITER'), 1)[0] - user = User.query.get('{}@{}'.format(localpart, domain_name)) + user = User.query.get(f'{localpart}@{domain_name}') if not user and localpart_stripped: - user = User.query.get('{}@{}'.format(localpart_stripped, domain_name)) + user = User.query.get(f'{localpart_stripped}@{domain_name}') if user: - email = '{}@{}'.format(localpart, domain_name) + email = f'{localpart}@{domain_name}' if user.forward_enabled: destination = user.forward_destination @@ -537,11 +378,15 @@ class Email(object): if pure_alias and not pure_alias.wildcard: return pure_alias.destination - elif stripped_alias: + + if stripped_alias: return stripped_alias.destination - elif pure_alias: + + if pure_alias: return pure_alias.destination + return None + def __str__(self): return str(self.email) @@ -586,11 +431,15 @@ class User(Base, Email): is_active = True is_anonymous = False + # TODO: remove unused user.get_id() def get_id(self): + """ return users email address """ return self.email + # TODO: remove unused user.destination @property def destination(self): + """ returns comma separated string of destinations """ if self.forward_enabled: result = list(self.forward_destination) if self.forward_keep: @@ -601,6 +450,7 @@ class User(Base, Email): @property def reply_active(self): + """ returns status of autoreply function """ now = date.today() return ( self.reply_enabled and @@ -608,49 +458,56 @@ class User(Base, Email): self.reply_enddate > now ) - scheme_dict = {'PBKDF2': 'pbkdf2_sha512', - 'BLF-CRYPT': 'bcrypt', - 'SHA512-CRYPT': 'sha512_crypt', - 'SHA256-CRYPT': 'sha256_crypt', - 'MD5-CRYPT': 'md5_crypt', - 'CRYPT': 'des_crypt'} + scheme_dict = { + 'PBKDF2': 'pbkdf2_sha512', + 'BLF-CRYPT': 'bcrypt', + 'SHA512-CRYPT': 'sha512_crypt', + 'SHA256-CRYPT': 'sha256_crypt', + 'MD5-CRYPT': 'md5_crypt', + 'CRYPT': 'des_crypt', + } - def get_password_context(self): + def _get_password_context(self): return passlib.context.CryptContext( schemes=self.scheme_dict.values(), default=self.scheme_dict[app.config['PASSWORD_SCHEME']], ) - def check_password(self, password): - context = self.get_password_context() - reference = re.match('({[^}]+})?(.*)', self.password).group(2) - result = context.verify(password, reference) - if result and context.identify(reference) != context.default_scheme(): - self.set_password(password) + def check_password(self, plain): + """ Check password against stored hash + Update hash when default scheme has changed + """ + context = self._get_password_context() + hashed = re.match('^({[^}]+})?(.*)$', self.password).group(2) + result = context.verify(plain, hashed) + if result and context.identify(hashed) != context.default_scheme(): + self.set_password(plain) db.session.add(self) db.session.commit() return result - def set_password(self, password, hash_scheme=None, raw=False): - """Set password for user with specified encryption scheme - @password: plain text password to encrypt (if raw == True the hash itself) + # TODO: remove kwarg hash_scheme - there is no point in setting a scheme, + # when the next check updates the password to the default scheme. + def set_password(self, new, hash_scheme=None, raw=False): + """ Set password for user with specified encryption scheme + @new: plain text password to encrypt (or, if raw is True: the hash itself) """ + # for the list of hash schemes see https://wiki2.dovecot.org/Authentication/PasswordSchemes if hash_scheme is None: hash_scheme = app.config['PASSWORD_SCHEME'] - # for the list of hash schemes see https://wiki2.dovecot.org/Authentication/PasswordSchemes - if raw: - self.password = '{'+hash_scheme+'}' + password - else: - self.password = '{'+hash_scheme+'}' + \ - self.get_password_context().encrypt(password, self.scheme_dict[hash_scheme]) + if not raw: + new = self._get_password_context().encrypt(new, self.scheme_dict[hash_scheme]) + self.password = f'{{{hash_scheme}}}{new}' def get_managed_domains(self): + """ return list of domains this user can manage """ if self.global_admin: return Domain.query.all() else: return self.manager_of def get_managed_emails(self, include_aliases=True): + """ returns list of email addresses this user can manage """ emails = [] for domain in self.get_managed_domains(): emails.extend(domain.users) @@ -659,16 +516,18 @@ class User(Base, Email): return emails def send_welcome(self): + """ send welcome email to user """ if app.config['WELCOME']: - self.sendmail(app.config['WELCOME_SUBJECT'], - app.config['WELCOME_BODY']) + self.sendmail(app.config['WELCOME_SUBJECT'], app.config['WELCOME_BODY']) @classmethod def get(cls, email): + """ find user object for email address """ return cls.query.get(email) @classmethod def login(cls, email, password): + """ login user when enabled and password is valid """ user = cls.query.get(email) return user if (user and user.enabled and user.check_password(password)) else None @@ -686,6 +545,8 @@ class Alias(Base, Email): @classmethod def resolve(cls, localpart, domain_name): + """ find aliases matching email address localpart@domain_name """ + alias_preserve_case = cls.query.filter( sqlalchemy.and_(cls.domain_name == domain_name, sqlalchemy.or_( @@ -709,24 +570,27 @@ class Alias(Base, Email): sqlalchemy.func.lower(cls.localpart) == localpart_lower ), sqlalchemy.and_( cls.wildcard is True, - sqlalchemy.bindparam('l', localpart_lower).like(sqlalchemy.func.lower(cls.localpart)) + sqlalchemy.bindparam('l', localpart_lower).like( + sqlalchemy.func.lower(cls.localpart)) ) ) ) - ).order_by(cls.wildcard, sqlalchemy.func.char_length(sqlalchemy.func.lower(cls.localpart)).desc()).first() + ).order_by(cls.wildcard, sqlalchemy.func.char_length( + sqlalchemy.func.lower(cls.localpart)).desc()).first() if alias_preserve_case and alias_lower_case: - if alias_preserve_case.wildcard: - return alias_lower_case - else: - return alias_preserve_case - elif alias_preserve_case and not alias_lower_case: - return alias_preserve_case - elif alias_lower_case and not alias_preserve_case: - return alias_lower_case - else: - return None + return alias_lower_case if alias_preserve_case.wildcard else alias_preserve_case + if alias_preserve_case and not alias_lower_case: + return alias_preserve_case + + if alias_lower_case and not alias_preserve_case: + return alias_lower_case + + return None + +# TODO: where are Tokens used / validated? +# TODO: what about API tokens? class Token(Base): """ A token is an application password for a given user. """ @@ -739,16 +603,20 @@ class Token(Base): user = db.relationship(User, backref=db.backref('tokens', cascade='all, delete-orphan')) password = db.Column(db.String(255), nullable=False) + # TODO: String(80) is too large? ip = db.Column(db.String(255)) def check_password(self, password): + """ verifies password against stored hash """ return passlib.hash.sha256_crypt.verify(password, self.password) + # TODO: use crypt context and default scheme from config? def set_password(self, password): + """ sets password using sha256_crypt(rounds=1000) """ self.password = passlib.hash.sha256_crypt.using(rounds=1000).hash(password) def __str__(self): - return self.comment or self.ip + return str(self.comment or self.ip) class Fetch(Base): From 65b1ad46d97c355ac93879b5b59dc75b77c579b9 Mon Sep 17 00:00:00 2001 From: Alexander Graf Date: Fri, 15 Jan 2021 13:57:20 +0100 Subject: [PATCH 057/596] order yaml data and allow callback on import - in yaml the primary key is now always first - calling a function on import allows import to be more verbose - skip "fetches" when empty --- core/admin/mailu/schemas.py | 88 ++++++++++++++++++++++++++++++------- 1 file changed, 72 insertions(+), 16 deletions(-) diff --git a/core/admin/mailu/schemas.py b/core/admin/mailu/schemas.py index fc08b67c..5dc10e17 100644 --- a/core/admin/mailu/schemas.py +++ b/core/admin/mailu/schemas.py @@ -3,11 +3,12 @@ import re +from collections import OrderedDict from textwrap import wrap import yaml -from marshmallow import pre_load, post_dump, fields, Schema +from marshmallow import pre_load, post_load, post_dump, fields, Schema from marshmallow.exceptions import ValidationError from marshmallow_sqlalchemy import SQLAlchemyAutoSchemaOpts from flask_marshmallow import Marshmallow @@ -25,6 +26,12 @@ ma = Marshmallow() ### yaml render module ### +# allow yaml module to dump OrderedDict +yaml.add_representer( + OrderedDict, + lambda cls, data: cls.represent_mapping('tag:yaml.org,2002:map', data.items()) +) + class RenderYAML: """ Marshmallow YAML Render Module """ @@ -62,6 +69,7 @@ class RenderYAML: 'Dumper': SpacedDumper, 'default_flow_style': False, 'allow_unicode': True, + 'sort_keys': False, } @classmethod def dumps(cls, *args, **kwargs): @@ -195,6 +203,8 @@ class BaseOpts(SQLAlchemyAutoSchemaOpts): def __init__(self, meta, ordered=False): if not hasattr(meta, 'sqla_session'): meta.sqla_session = models.db.session + if not hasattr(meta, 'ordered'): + meta.ordered = True super(BaseOpts, self).__init__(meta, ordered=ordered) class BaseSchema(ma.SQLAlchemyAutoSchema): @@ -206,13 +216,12 @@ class BaseSchema(ma.SQLAlchemyAutoSchema): class Meta: """ Schema config """ - model = None def __init__(self, *args, **kwargs): # context? context = kwargs.get('context', {}) - flags = set([key for key, value in context.items() if value is True]) + flags = {key for key, value in context.items() if value is True} # compile excludes exclude = set(kwargs.get('exclude', [])) @@ -234,7 +243,7 @@ class BaseSchema(ma.SQLAlchemyAutoSchema): # exclude default values if not context.get('full'): - for column in self.Meta.model.__table__.columns: + for column in getattr(self.Meta, 'model').__table__.columns: if column.name not in exclude: self._exclude_by_value.setdefault(column.name, []).append( None if column.default is None else column.default.arg @@ -250,20 +259,48 @@ class BaseSchema(ma.SQLAlchemyAutoSchema): # init SQLAlchemyAutoSchema super().__init__(*args, **kwargs) - @post_dump - def _remove_skip_values(self, data, many, **kwargs): # pylint: disable=unused-argument + # init order + if hasattr(self.Meta, 'order'): + # use user-defined order + self._order = list(reversed(getattr(self.Meta, 'order'))) + else: + # default order is: primary_key + other keys alphabetically + self._order = list(sorted(self.fields.keys())) + primary = self.opts.model.__table__.primary_key.columns.values()[0].name + self._order.remove(primary) + self._order.reverse() + self._order.append(primary) + @pre_load + def _track_import(self, data, many, **kwargs): # pylint: disable=unused-argument + call = self.context.get('callback') + if call is not None: + call(self=self, data=data, many=many, **kwargs) + return data + + @post_dump + def _hide_and_order(self, data, many, **kwargs): # pylint: disable=unused-argument + + # order output + for key in self._order: + try: + data.move_to_end(key, False) + except KeyError: + pass + + # stop early when not excluding/hiding if not self._exclude_by_value and not self._hide_by_context: return data + # exclude items or hide values full = self.context.get('full') - return { - key: '' if key in self._hide_by_context else value + return type(data)([ + (key, '' if key in self._hide_by_context else value) for key, value in data.items() if full or key not in self._exclude_by_value or value not in self._exclude_by_value[key] - } + ]) - # TODO: remove LazyString and change model (IMHO comment should not be nullable) + # TODO: remove LazyStringField and change model (IMHO comment should not be nullable) comment = LazyStringField() @@ -336,13 +373,14 @@ class UserSchema(BaseSchema): exclude_by_value = { 'forward_destination': [[]], 'tokens': [[]], + 'fetches': [[]], 'manager_of': [[]], 'reply_enddate': ['2999-12-31'], 'reply_startdate': ['1900-01-01'], } @pre_load - def _handle_password(self, data, many, **kwargs): # pylint: disable=unused-argument + def _handle_email_and_password(self, data, many, **kwargs): # pylint: disable=unused-argument data = handle_email(data) if 'password' in data: if 'password_hash' in data or 'hash_scheme' in data: @@ -358,16 +396,23 @@ class UserSchema(BaseSchema): elif 'password_hash' in data and 'hash_scheme' in data: if data['hash_scheme'] not in self.Meta.model.scheme_dict: raise ValidationError(f'invalid password scheme {scheme!r}') - data['password'] = '{'+data['hash_scheme']+'}'+ data['password_hash'] + data['password'] = f'{{{data["hash_scheme"]}}}{data["password_hash"]}' del data['hash_scheme'] del data['password_hash'] return data + # TODO: verify password (should this be done in model?) + # scheme, hashed = re.match('^(?:{([^}]+)})?(.*)$', self.password).groups() + # if not scheme... + # ctx = passlib.context.CryptContext(schemes=[scheme], default=scheme) + # try: + # ctx.verify('', hashed) + # =>? ValueError: hash could not be identified + tokens = fields.Nested(TokenSchema, many=True) fetches = fields.Nested(FetchSchema, many=True) - class AliasSchema(BaseSchema): """ Marshmallow schema for Alias model """ class Meta: @@ -381,7 +426,7 @@ class AliasSchema(BaseSchema): } @pre_load - def _handle_password(self, data, many, **kwargs): # pylint: disable=unused-argument + def _handle_email(self, data, many, **kwargs): # pylint: disable=unused-argument return handle_email(data) destination = CommaSeparatedListField() @@ -408,9 +453,20 @@ class MailuSchema(Schema): class Meta: """ Schema config """ render_module = RenderYAML + ordered = True + order = ['config', 'domains', 'users', 'aliases', 'relays'] + @post_dump(pass_many=True) + def _order(self, data : OrderedDict, many : bool, **kwargs): # pylint: disable=unused-argument + for key in reversed(self.Meta.order): + try: + data.move_to_end(key, False) + except KeyError: + pass + return data + + config = fields.Nested(ConfigSchema, many=True) domains = fields.Nested(DomainSchema, many=True) - relays = fields.Nested(RelaySchema, many=True) users = fields.Nested(UserSchema, many=True) aliases = fields.Nested(AliasSchema, many=True) - config = fields.Nested(ConfigSchema, many=True) + relays = fields.Nested(RelaySchema, many=True) From f56af3053aec9980d1b8b022307208b676c8bc06 Mon Sep 17 00:00:00 2001 From: Mordi Sacks Date: Sun, 17 Jan 2021 01:28:25 +0200 Subject: [PATCH 058/596] Removed email address --- core/admin/mailu/translations/he/LC_MESSAGES/messages.po | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/core/admin/mailu/translations/he/LC_MESSAGES/messages.po b/core/admin/mailu/translations/he/LC_MESSAGES/messages.po index 4fe58afc..e884b737 100644 --- a/core/admin/mailu/translations/he/LC_MESSAGES/messages.po +++ b/core/admin/mailu/translations/he/LC_MESSAGES/messages.po @@ -9,7 +9,7 @@ msgstr "" "Report-Msgid-Bugs-To: EMAIL@ADDRESS\n" "POT-Creation-Date: 2018-04-22 12:10+0200\n" "PO-Revision-Date: 2019-11-27 22:20+0000\n" -"Last-Translator: Mordi Sacks \n" +"Last-Translator: Mordi Sacks \n" "Language-Team: Hebrew \n" "Language: he\n" From 82b5920b160e8be37ab9b6b27311c19463739dee Mon Sep 17 00:00:00 2001 From: =?UTF-8?q?Rapha=C3=ABl=20P=2E=20Barazzutti?= Date: Thu, 21 Jan 2021 19:54:47 +0100 Subject: [PATCH 059/596] typos --- CHANGELOG.md | 4 ++-- docs/releases.rst | 4 ++-- 2 files changed, 4 insertions(+), 4 deletions(-) diff --git a/CHANGELOG.md b/CHANGELOG.md index a9c78f33..579f3e82 100644 --- a/CHANGELOG.md +++ b/CHANGELOG.md @@ -11,7 +11,7 @@ The Dovecot and Postfix overrides are moved in their own sub-directory. If there are local override files, they will need to be moved from overrides/ to overrides/dovecot and overrides/postfix/. See https://mailu.io/1.8/faq.html#how-can-i-override-settings for all the mappings. -Please not that the shipped image for PostgreSQL database is deprecated. +Please note that the shipped image for PostgreSQL database is deprecated. We advise to switch to an external database server. @@ -66,7 +66,7 @@ configuration and upgrade your mailu.env. If you run the PostgreSQL server, the database was upgrade, so you will need to dump the database before upgrading and load the dump after the upgrade is -complete. Please not that the shipped image for PostgreSQL database will be +complete. Please note that the shipped image for PostgreSQL database will be deprecated before 1.8.0, you can switch to an external database server by then. - Deprecation: using the internal postgres image will be deprecated by 1.8.0 diff --git a/docs/releases.rst b/docs/releases.rst index e5bd06c0..141b66fd 100644 --- a/docs/releases.rst +++ b/docs/releases.rst @@ -33,7 +33,7 @@ Upgrading Upgrade should run fine as long as you generate a new compose or stack configuration and upgrade your mailu.env. -Please not that the shipped image for PostgreSQL database is deprecated. +Please note that the shipped image for PostgreSQL database is deprecated. The shipped image for PostgreSQL is not maintained anymore from release 1.8. We recommend switching to an external database as soon as possible. @@ -95,7 +95,7 @@ configuration and upgrade your mailu.env. If you run the PostgreSQL server, the database was upgrade, so you will need to dump the database before upgrading and load the dump after the upgrade is -complete. Please not that the shipped image for PostgreSQL database will be +complete. Please note that the shipped image for PostgreSQL database will be deprecated before 1.8.0, you can switch to an external database server by then. From 444529b7df2fca33f7bd3cdc9a89c3bb51e4e0b9 Mon Sep 17 00:00:00 2001 From: =?UTF-8?q?Rapha=C3=ABl=20P=2E=20Barazzutti?= Date: Thu, 21 Jan 2021 19:55:53 +0100 Subject: [PATCH 060/596] rewording in doc --- docs/releases.rst | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/docs/releases.rst b/docs/releases.rst index 141b66fd..f8b1c731 100644 --- a/docs/releases.rst +++ b/docs/releases.rst @@ -35,7 +35,7 @@ configuration and upgrade your mailu.env. Please note that the shipped image for PostgreSQL database is deprecated. The shipped image for PostgreSQL is not maintained anymore from release 1.8. -We recommend switching to an external database as soon as possible. +We recommend switching to another database as soon as possible (SQLite or MySQL). Override location changes ^^^^^^^^^^^^^^^^^^^^^^^^^ From 9e8183ee7193cffd42cd3e3ba9678a67e53a621e Mon Sep 17 00:00:00 2001 From: =?UTF-8?q?Rapha=C3=ABl=20P=2E=20Barazzutti?= Date: Fri, 22 Jan 2021 05:29:54 +0100 Subject: [PATCH 061/596] rewording about the usage of PostgreSQL Co-authored-by: lub --- docs/releases.rst | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/docs/releases.rst b/docs/releases.rst index f8b1c731..7a15d1fa 100644 --- a/docs/releases.rst +++ b/docs/releases.rst @@ -35,7 +35,7 @@ configuration and upgrade your mailu.env. Please note that the shipped image for PostgreSQL database is deprecated. The shipped image for PostgreSQL is not maintained anymore from release 1.8. -We recommend switching to another database as soon as possible (SQLite or MySQL). +We recommend switching to an external PostgreSQL database as soon as possible. Override location changes ^^^^^^^^^^^^^^^^^^^^^^^^^ From 902b398127dc9abb6f3433ee7fccd51ddc0b3873 Mon Sep 17 00:00:00 2001 From: Alexander Graf Date: Sun, 24 Jan 2021 19:07:48 +0100 Subject: [PATCH 062/596] next step for import/export yaml & json --- core/admin/mailu/manage.py | 238 ++++++++++++++++++++++++++++++------ core/admin/mailu/models.py | 176 ++++++++++++++++++-------- core/admin/mailu/schemas.py | 157 +++++++++++++++++------- 3 files changed, 441 insertions(+), 130 deletions(-) diff --git a/core/admin/mailu/manage.py b/core/admin/mailu/manage.py index 80a73230..e02d9ad4 100644 --- a/core/admin/mailu/manage.py +++ b/core/admin/mailu/manage.py @@ -4,9 +4,15 @@ import sys import os import socket +import json +import logging import uuid +from collections import Counter +from itertools import chain + import click +import sqlalchemy import yaml from flask import current_app as app @@ -14,7 +20,7 @@ from flask.cli import FlaskGroup, with_appcontext from marshmallow.exceptions import ValidationError from . import models -from .schemas import MailuSchema +from .schemas import MailuSchema, get_schema db = models.db @@ -322,60 +328,211 @@ SECTIONS = {'domains', 'relays', 'users', 'aliases'} @mailu.command() -@click.option('-v', '--verbose', is_flag=True, help='Increase verbosity') +@click.option('-v', '--verbose', count=True, help='Increase verbosity') +@click.option('-q', '--quiet', is_flag=True, help='Quiet mode - only show errors') +@click.option('-u', '--update', is_flag=True, help='Update mode - merge input with existing config') @click.option('-n', '--dry-run', is_flag=True, help='Perform a trial run with no changes made') @click.argument('source', metavar='[FILENAME|-]', type=click.File(mode='r'), default=sys.stdin) @with_appcontext -def config_import(verbose=False, dry_run=False, source=None): - """ Import configuration from YAML +def config_import(verbose=0, quiet=False, update=False, dry_run=False, source=None): + """ Import configuration as YAML or JSON from stdin or file """ - def log(**data): - caller = sys._getframe(1).f_code.co_name # pylint: disable=protected-access - if caller == '_track_import': - print(f'Handling {data["self"].opts.model.__table__} data: {data["data"]!r}') + # verbose + # 0 : show number of changes + # 1 : also show changes + # 2 : also show secrets + # 3 : also show input data + # 4 : also show sql queries + + if quiet: + verbose = -1 + + counter = Counter() + dumper = {} def format_errors(store, path=None): + + res = [] if path is None: path = [] for key in sorted(store): location = path + [str(key)] value = store[key] if isinstance(value, dict): - format_errors(value, location) + res.extend(format_errors(value, location)) else: for message in value: - print(f'[ERROR] {".".join(location)}: {message}') + res.append((".".join(location), message)) - context = { - 'callback': log if verbose else None, + if path: + return res + + fmt = f' - {{:<{max([len(loc) for loc, msg in res])}}} : {{}}' + res = [fmt.format(loc, msg) for loc, msg in res] + num = f'error{["s",""][len(res)==1]}' + res.insert(0, f'[ValidationError] {len(res)} {num} occured during input validation') + + return '\n'.join(res) + + def format_changes(*message): + if counter: + changes = [] + last = None + for (action, what), count in sorted(counter.items()): + if action != last: + if last: + changes.append('/') + changes.append(f'{action}:') + last = action + changes.append(f'{what}({count})') + else: + changes = 'no changes.' + return chain(message, changes) + + def log(action, target, message=None): + if message is None: + message = json.dumps(dumper[target.__class__].dump(target), ensure_ascii=False) + print(f'{action} {target.__table__}: {message}') + + def listen_insert(mapper, connection, target): # pylint: disable=unused-argument + """ callback function to track import """ + counter.update([('Added', target.__table__.name)]) + if verbose >= 1: + log('Added', target) + + def listen_update(mapper, connection, target): # pylint: disable=unused-argument + """ callback function to track import """ + + changed = {} + inspection = sqlalchemy.inspect(target) + for attr in sqlalchemy.orm.class_mapper(target.__class__).column_attrs: + if getattr(inspection.attrs, attr.key).history.has_changes(): + if sqlalchemy.orm.attributes.get_history(target, attr.key)[2]: + before = sqlalchemy.orm.attributes.get_history(target, attr.key)[2].pop() + after = getattr(target, attr.key) + # only remember changed keys + if before != after and (before or after): + if verbose >= 1: + changed[str(attr.key)] = (before, after) + else: + break + + if verbose >= 1: + # use schema with dump_context to hide secrets and sort keys + primary = json.dumps(str(target), ensure_ascii=False) + dumped = get_schema(target)(only=changed.keys(), context=dump_context).dump(target) + for key, value in dumped.items(): + before, after = changed[key] + if value == '': + before = '' if before else before + after = '' if after else after + else: + # TODO: use schema to "convert" before value? + after = value + before = json.dumps(before, ensure_ascii=False) + after = json.dumps(after, ensure_ascii=False) + log('Modified', target, f'{primary} {key}: {before} -> {after}') + + if changed: + counter.update([('Modified', target.__table__.name)]) + + def listen_delete(mapper, connection, target): # pylint: disable=unused-argument + """ callback function to track import """ + counter.update([('Deleted', target.__table__.name)]) + if verbose >= 1: + log('Deleted', target) + + # this listener should not be necessary, when: + # dkim keys should be stored in database and it should be possible to store multiple + # keys per domain. the active key would be also stored on disk on commit. + def listen_dkim(session, flush_context): # pylint: disable=unused-argument + """ callback function to track import """ + for target in session.identity_map.values(): + if not isinstance(target, models.Domain): + continue + primary = json.dumps(str(target), ensure_ascii=False) + before = target._dkim_key_on_disk + after = target._dkim_key + if before != after and (before or after): + if verbose >= 2: + before = before.decode('ascii', 'ignore') + after = after.decode('ascii', 'ignore') + else: + before = '' if before else '' + after = '' if after else '' + before = json.dumps(before, ensure_ascii=False) + after = json.dumps(after, ensure_ascii=False) + log('Modified', target, f'{primary} dkim_key: {before} -> {after}') + counter.update([('Modified', target.__table__.name)]) + + def track_serialize(self, item): + """ callback function to track import """ + log('Handling', self.opts.model, item) + + # configure contexts + dump_context = { + 'secrets': verbose >= 2, + } + load_context = { + 'callback': track_serialize if verbose >= 3 else None, + 'clear': not update, 'import': True, } - error = False + # register listeners + for schema in get_schema(): + model = schema.Meta.model + dumper[model] = schema(context=dump_context) + sqlalchemy.event.listen(model, 'after_insert', listen_insert) + sqlalchemy.event.listen(model, 'after_update', listen_update) + sqlalchemy.event.listen(model, 'after_delete', listen_delete) + + # special listener for dkim_key changes + sqlalchemy.event.listen(db.session, 'after_flush', listen_dkim) + + if verbose >= 4: + logging.basicConfig() + logging.getLogger('sqlalchemy.engine').setLevel(logging.INFO) + try: - config = MailuSchema(context=context).loads(source) + with models.db.session.no_autoflush: + config = MailuSchema(only=SECTIONS, context=load_context).loads(source) except ValidationError as exc: - error = True - format_errors(exc.messages) - else: - print(config) + raise click.ClickException(format_errors(exc.messages)) from exc + except Exception as exc: + # (yaml.scanner.ScannerError, UnicodeDecodeError, ...) + raise click.ClickException(f'[{exc.__class__.__name__}] {" ".join(str(exc).split())}') from exc + + # flush session to show/count all changes + if dry_run or verbose >= 1: + db.session.flush() + + # check for duplicate domain names + dup = set() + for fqdn in chain(db.session.query(models.Domain.name), + db.session.query(models.Alternative.name), + db.session.query(models.Relay.name)): + if fqdn in dup: + raise click.ClickException(f'[ValidationError] Duplicate domain name: {fqdn}') + dup.add(fqdn) + + # TODO: implement special update "items" + # -pkey: which - remove item "which" + # -key: null or [] or {} - set key to default + # -pkey: null or [] or {} - remove all existing items in this list + + # don't commit when running dry + if dry_run: + db.session.rollback() + if not quiet: + print(*format_changes('Dry run. Not commiting changes.')) + # TODO: remove debug print(MailuSchema().dumps(config)) - # TODO: need to delete other entries - - # TODO: enable commit - error = True - - # don't commit when running dry or validation errors occured - if error: - print('An error occured. Not committing changes.') - db.session.rollback() - sys.exit(2) - elif dry_run: - print('Dry run. Not commiting changes.') - db.session.rollback() else: db.session.commit() + if not quiet: + print(*format_changes('Commited changes.')) @mailu.command() @@ -385,28 +542,35 @@ def config_import(verbose=False, dry_run=False, source=None): @click.option('-d', '--dns', is_flag=True, help='Include dns records') @click.option('-o', '--output-file', 'output', default=sys.stdout, type=click.File(mode='w'), help='save yaml to file') +@click.option('-j', '--json', 'as_json', is_flag=True, help='Dump in josn format') @click.argument('sections', nargs=-1) @with_appcontext -def config_export(full=False, secrets=False, dns=False, output=None, sections=None): - """ Export configuration as YAML to stdout or file +def config_export(full=False, secrets=False, dns=False, output=None, as_json=False, sections=None): + """ Export configuration as YAML or JSON to stdout or file """ if sections: for section in sections: if section not in SECTIONS: - print(f'[ERROR] Unknown section: {section!r}') - sys.exit(1) + print(f'[ERROR] Unknown section: {section}') + raise click.exceptions.Exit(1) sections = set(sections) else: sections = SECTIONS - context={ + context = { 'full': full, 'secrets': secrets, 'dns': dns, } - MailuSchema(only=sections, context=context).dumps(models.MailuConfig(), output) + if as_json: + schema = MailuSchema(only=sections, context=context) + schema.opts.render_module = json + print(schema.dumps(models.MailuConfig(), separators=(',',':')), file=output) + + else: + MailuSchema(only=sections, context=context).dumps(models.MailuConfig(), output) @mailu.command() diff --git a/core/admin/mailu/models.py b/core/admin/mailu/models.py index 3187f597..dac1dc70 100644 --- a/core/admin/mailu/models.py +++ b/core/admin/mailu/models.py @@ -12,7 +12,8 @@ from itertools import chain import flask_sqlalchemy import sqlalchemy -import passlib +import passlib.context +import passlib.hash import idna import dns @@ -79,11 +80,11 @@ class CommaSeparatedList(db.TypeDecorator): for item in value: if ',' in item: raise ValueError('list item must not contain ","') - return ','.join(sorted(value)) + return ','.join(sorted(set(value))) def process_result_value(self, value, dialect): """ split comma separated string to list """ - return list(filter(bool, value.split(','))) if value else [] + return list(filter(bool, [item.strip() for item in value.split(',')])) if value else [] python_type = list @@ -136,19 +137,11 @@ class Config(Base): value = db.Column(JSONEncoded) -# TODO: use sqlalchemy.event.listen() on a store method of object? -@sqlalchemy.event.listens_for(db.session, 'after_commit') -def store_dkim_key(session): - """ Store DKIM key on commit """ +def _save_dkim_keys(session): + """ store DKIM keys after commit """ for obj in session.identity_map.values(): if isinstance(obj, Domain): - if obj._dkim_key_changed: - file_path = obj._dkim_file() - if obj._dkim_key: - with open(file_path, 'wb') as handle: - handle.write(obj._dkim_key) - elif os.path.exists(file_path): - os.unlink(file_path) + obj.save_dkim_key() class Domain(Base): """ A DNS domain that has mail addresses associated to it. @@ -165,7 +158,7 @@ class Domain(Base): signup_enabled = db.Column(db.Boolean, nullable=False, default=False) _dkim_key = None - _dkim_key_changed = False + _dkim_key_on_disk = None def _dkim_file(self): """ return filename for active DKIM key """ @@ -174,6 +167,17 @@ class Domain(Base): selector=app.config['DKIM_SELECTOR'] ) + def save_dkim_key(self): + """ save changed DKIM key to disk """ + if self._dkim_key != self._dkim_key_on_disk: + file_path = self._dkim_file() + if self._dkim_key: + with open(file_path, 'wb') as handle: + handle.write(self._dkim_key) + elif os.path.exists(file_path): + os.unlink(file_path) + self._dkim_key_on_disk = self._dkim_key + @property def dns_mx(self): """ return MX record for domain """ @@ -189,7 +193,7 @@ class Domain(Base): @property def dns_dkim(self): """ return DKIM record for domain """ - if os.path.exists(self._dkim_file()): + if self.dkim_key: selector = app.config['DKIM_SELECTOR'] return ( f'{selector}._domainkey.{self.name}. 600 IN TXT' @@ -199,7 +203,7 @@ class Domain(Base): @property def dns_dmarc(self): """ return DMARC record for domain """ - if os.path.exists(self._dkim_file()): + if self.dkim_key: domain = app.config['DOMAIN'] rua = app.config['DMARC_RUA'] rua = f' rua=mailto:{rua}@{domain};' if rua else '' @@ -214,19 +218,19 @@ class Domain(Base): file_path = self._dkim_file() if os.path.exists(file_path): with open(file_path, 'rb') as handle: - self._dkim_key = handle.read() + self._dkim_key = self._dkim_key_on_disk = handle.read() else: - self._dkim_key = b'' + self._dkim_key = self._dkim_key_on_disk = b'' return self._dkim_key if self._dkim_key else None @dkim_key.setter def dkim_key(self, value): """ set private DKIM key """ old_key = self.dkim_key - if value is None: - value = b'' - self._dkim_key_changed = value != old_key - self._dkim_key = value + self._dkim_key = value if value is not None else b'' + if self._dkim_key != old_key: + if not sqlalchemy.event.contains(db.session, 'after_commit', _save_dkim_keys): + sqlalchemy.event.listen(db.session, 'after_commit', _save_dkim_keys) @property def dkim_publickey(self): @@ -331,14 +335,14 @@ class Email(object): def sendmail(self, subject, body): """ send an email to the address """ - from_address = f'{app.config["POSTMASTER"]}@{idna.encode(app.config["DOMAIN"]).decode("ascii")}' + f_addr = f'{app.config["POSTMASTER"]}@{idna.encode(app.config["DOMAIN"]).decode("ascii")}' with smtplib.SMTP(app.config['HOST_AUTHSMTP'], port=10025) as smtp: to_address = f'{self.localpart}@{idna.encode(self.domain_name).decode("ascii")}' msg = text.MIMEText(body) msg['Subject'] = subject - msg['From'] = from_address + msg['From'] = f_addr msg['To'] = to_address - smtp.sendmail(from_address, [to_address], msg.as_string()) + smtp.sendmail(f_addr, [to_address], msg.as_string()) @classmethod def resolve_domain(cls, email): @@ -589,7 +593,6 @@ class Alias(Base, Email): return None -# TODO: where are Tokens used / validated? # TODO: what about API tokens? class Token(Base): """ A token is an application password for a given user. @@ -650,20 +653,22 @@ class MailuConfig: and loading """ - # TODO: add sqlalchemy session updating (.add & .del) class MailuCollection: - """ Provides dict- and list-like access to all instances + """ Provides dict- and list-like access to instances of a sqlalchemy model """ def __init__(self, model : db.Model): - self._model = model + self.model = model + + def __str__(self): + return f'<{self.model.__name__}-Collection>' @cached_property def _items(self): return { inspect(item).identity: item - for item in self._model.query.all() + for item in self.model.query.all() } def __len__(self): @@ -676,8 +681,8 @@ class MailuConfig: return self._items[key] def __setitem__(self, key, item): - if not isinstance(item, self._model): - raise TypeError(f'expected {self._model.name}') + if not isinstance(item, self.model): + raise TypeError(f'expected {self.model.name}') if key != inspect(item).identity: raise ValueError(f'item identity != key {key!r}') self._items[key] = item @@ -685,23 +690,24 @@ class MailuConfig: def __delitem__(self, key): del self._items[key] - def append(self, item): + def append(self, item, update=False): """ list-like append """ - if not isinstance(item, self._model): - raise TypeError(f'expected {self._model.name}') + if not isinstance(item, self.model): + raise TypeError(f'expected {self.model.name}') key = inspect(item).identity if key in self._items: - raise ValueError(f'item {key!r} already present in collection') + if not update: + raise ValueError(f'item {key!r} already present in collection') self._items[key] = item - def extend(self, items): + def extend(self, items, update=False): """ list-like extend """ add = {} for item in items: - if not isinstance(item, self._model): - raise TypeError(f'expected {self._model.name}') + if not isinstance(item, self.model): + raise TypeError(f'expected {self.model.name}') key = inspect(item).identity - if key in self._items: + if not update and key in self._items: raise ValueError(f'item {key!r} already present in collection') add[key] = item self._items.update(add) @@ -721,8 +727,8 @@ class MailuConfig: def remove(self, item): """ list-like remove """ - if not isinstance(item, self._model): - raise TypeError(f'expected {self._model.name}') + if not isinstance(item, self.model): + raise TypeError(f'expected {self.model.name}') key = inspect(item).identity if not key in self._items: raise ValueError(f'item {key!r} not found in collection') @@ -739,12 +745,11 @@ class MailuConfig: def update(self, items): """ dict-like update """ for key, item in items: - if not isinstance(item, self._model): - raise TypeError(f'expected {self._model.name}') + if not isinstance(item, self.model): + raise TypeError(f'expected {self.model.name}') if key != inspect(item).identity: raise ValueError(f'item identity != key {key!r}') - if key in self._items: - raise ValueError(f'item {key!r} already present in collection') + self._items.update(items) def setdefault(self, key, item=None): """ dict-like setdefault """ @@ -752,13 +757,86 @@ class MailuConfig: return self._items[key] if item is None: return None - if not isinstance(item, self._model): - raise TypeError(f'expected {self._model.name}') + if not isinstance(item, self.model): + raise TypeError(f'expected {self.model.name}') if key != inspect(item).identity: raise ValueError(f'item identity != key {key!r}') self._items[key] = item return item + def __init__(self): + + # section-name -> attr + self._sections = { + name: getattr(self, name) + for name in dir(self) + if isinstance(getattr(self, name), self.MailuCollection) + } + + # known models + self._models = tuple(section.model for section in self._sections.values()) + + # model -> attr + self._sections.update({ + section.model: section for section in self._sections.values() + }) + + def _get_model(self, section): + if section is None: + return None + model = self._sections.get(section) + if model is None: + raise ValueError(f'Invalid section: {section!r}') + if isinstance(model, self.MailuCollection): + return model.model + return model + + def _add(self, items, section, update): + + model = self._get_model(section) + if isinstance(items, self._models): + items = [items] + elif not hasattr(items, '__iter__'): + raise ValueError(f'{items!r} is not iterable') + + for item in items: + if model is not None and not isinstance(item, model): + what = item.__class__.__name__.capitalize() + raise ValueError(f'{what} can not be added to section {section!r}') + self._sections[type(item)].append(item, update=update) + + def add(self, items, section=None): + """ add item to config """ + self._add(items, section, update=False) + + def update(self, items, section=None): + """ add or replace item in config """ + self._add(items, section, update=True) + + def remove(self, items, section=None): + """ remove item from config """ + model = self._get_model(section) + if isinstance(items, self._models): + items = [items] + elif not hasattr(items, '__iter__'): + raise ValueError(f'{items!r} is not iterable') + + for item in items: + if isinstance(item, str): + if section is None: + raise ValueError(f'Cannot remove key {item!r} without section') + del self._sections[model][item] + elif model is not None and not isinstance(item, model): + what = item.__class__.__name__.capitalize() + raise ValueError(f'{what} can not be removed from section {section!r}') + self._sections[type(item)].remove(item,) + + def clear(self, models=None): + """ remove complete configuration """ + for model in self._models: + if models is None or model in models: + db.session.query(model).delete() + domains = MailuCollection(Domain) relays = MailuCollection(Relay) users = MailuCollection(User) diff --git a/core/admin/mailu/schemas.py b/core/admin/mailu/schemas.py index 5dc10e17..04512f6d 100644 --- a/core/admin/mailu/schemas.py +++ b/core/admin/mailu/schemas.py @@ -24,6 +24,23 @@ ma = Marshmallow() # - fields which are the primary key => unchangeable when updating +### map model to schema ### + +_model2schema = {} + +def get_schema(model=None): + """ return schema class for model or instance of model """ + if model is None: + return _model2schema.values() + else: + return _model2schema.get(model) or _model2schema.get(model.__class__) + +def mapped(cls): + """ register schema in model2schema map """ + _model2schema[cls.Meta.model] = cls + return cls + + ### yaml render module ### # allow yaml module to dump OrderedDict @@ -79,26 +96,6 @@ class RenderYAML: return yaml.dump(*args, **kwargs) -### functions ### - -def handle_email(data): - """ merge separate localpart and domain to email - """ - - localpart = 'localpart' in data - domain = 'domain' in data - - if 'email' in data: - if localpart or domain: - raise ValidationError('duplicate email and localpart/domain') - elif localpart and domain: - data['email'] = f'{data["localpart"]}@{data["domain"]}' - elif localpart or domain: - raise ValidationError('incomplete localpart/domain') - - return data - - ### field definitions ### class LazyStringField(fields.String): @@ -177,9 +174,7 @@ class DkimKeyField(fields.String): return dkim.gen_key() # remember some keydata for error message - keydata = value - if len(keydata) > 40: - keydata = keydata[:25] + '...' + keydata[-10:] + keydata = f'{value[:25]}...{value[-10:]}' if len(value) > 40 else value # wrap value into valid pem layout and check validity value = ( @@ -197,6 +192,26 @@ class DkimKeyField(fields.String): ### base definitions ### +def handle_email(data): + """ merge separate localpart and domain to email + """ + + localpart = 'localpart' in data + domain = 'domain' in data + + if 'email' in data: + if localpart or domain: + raise ValidationError('duplicate email and localpart/domain') + data['localpart'], data['domain_name'] = data['email'].rsplit('@', 1) + elif localpart and domain: + data['domain_name'] = data['domain'] + del data['domain'] + data['email'] = f'{data["localpart"]}@{data["domain_name"]}' + elif localpart or domain: + raise ValidationError('incomplete localpart/domain') + + return data + class BaseOpts(SQLAlchemyAutoSchemaOpts): """ Option class with sqla session """ @@ -238,12 +253,15 @@ class BaseSchema(ma.SQLAlchemyAutoSchema): # update excludes kwargs['exclude'] = exclude + # init SQLAlchemyAutoSchema + super().__init__(*args, **kwargs) + # exclude_by_value self._exclude_by_value = getattr(self.Meta, 'exclude_by_value', {}) # exclude default values if not context.get('full'): - for column in getattr(self.Meta, 'model').__table__.columns: + for column in getattr(self.opts, 'model').__table__.columns: if column.name not in exclude: self._exclude_by_value.setdefault(column.name, []).append( None if column.default is None else column.default.arg @@ -256,10 +274,7 @@ class BaseSchema(ma.SQLAlchemyAutoSchema): if not flags & set(need): self._hide_by_context |= set(what) - # init SQLAlchemyAutoSchema - super().__init__(*args, **kwargs) - - # init order + # initialize attribute order if hasattr(self.Meta, 'order'): # use user-defined order self._order = list(reversed(getattr(self.Meta, 'order'))) @@ -267,17 +282,35 @@ class BaseSchema(ma.SQLAlchemyAutoSchema): # default order is: primary_key + other keys alphabetically self._order = list(sorted(self.fields.keys())) primary = self.opts.model.__table__.primary_key.columns.values()[0].name - self._order.remove(primary) - self._order.reverse() - self._order.append(primary) + if primary in self._order: + self._order.remove(primary) + self._order.reverse() + self._order.append(primary) + + # move pre_load hook "_track_import" to the front + hooks = self._hooks[('pre_load', False)] + if '_track_import' in hooks: + hooks.remove('_track_import') + hooks.insert(0, '_track_import') + # and post_load hook "_fooo" to the end + hooks = self._hooks[('post_load', False)] + if '_add_instance' in hooks: + hooks.remove('_add_instance') + hooks.append('_add_instance') @pre_load def _track_import(self, data, many, **kwargs): # pylint: disable=unused-argument - call = self.context.get('callback') - if call is not None: - call(self=self, data=data, many=many, **kwargs) +# TODO: also handle reset, prune and delete in pre_load / post_load hooks! +# print('!!!', repr(data)) + if callback := self.context.get('callback'): + callback(self, data) return data + @post_load + def _add_instance(self, item, many, **kwargs): # pylint: disable=unused-argument + self.opts.sqla_session.add(item) + return item + @post_dump def _hide_and_order(self, data, many, **kwargs): # pylint: disable=unused-argument @@ -306,6 +339,7 @@ class BaseSchema(ma.SQLAlchemyAutoSchema): ### schema definitions ### +@mapped class DomainSchema(BaseSchema): """ Marshmallow schema for Domain model """ class Meta: @@ -339,6 +373,7 @@ class DomainSchema(BaseSchema): dns_dmarc = fields.String(dump_only=True) +@mapped class TokenSchema(BaseSchema): """ Marshmallow schema for Token model """ class Meta: @@ -347,6 +382,7 @@ class TokenSchema(BaseSchema): load_instance = True +@mapped class FetchSchema(BaseSchema): """ Marshmallow schema for Fetch model """ class Meta: @@ -361,6 +397,7 @@ class FetchSchema(BaseSchema): } +@mapped class UserSchema(BaseSchema): """ Marshmallow schema for User model """ class Meta: @@ -368,7 +405,7 @@ class UserSchema(BaseSchema): model = models.User load_instance = True include_relationships = True - exclude = ['localpart', 'domain', 'quota_bytes_used'] + exclude = ['domain', 'quota_bytes_used'] exclude_by_value = { 'forward_destination': [[]], @@ -395,7 +432,7 @@ class UserSchema(BaseSchema): raise ValidationError(f'invalid hashed password {password!r}') elif 'password_hash' in data and 'hash_scheme' in data: if data['hash_scheme'] not in self.Meta.model.scheme_dict: - raise ValidationError(f'invalid password scheme {scheme!r}') + raise ValidationError(f'invalid password scheme {data["hash_scheme"]!r}') data['password'] = f'{{{data["hash_scheme"]}}}{data["password_hash"]}' del data['hash_scheme'] del data['password_hash'] @@ -409,17 +446,20 @@ class UserSchema(BaseSchema): # ctx.verify('', hashed) # =>? ValueError: hash could not be identified + localpart = fields.Str(load_only=True) + domain_name = fields.Str(load_only=True) tokens = fields.Nested(TokenSchema, many=True) fetches = fields.Nested(FetchSchema, many=True) +@mapped class AliasSchema(BaseSchema): """ Marshmallow schema for Alias model """ class Meta: """ Schema config """ model = models.Alias load_instance = True - exclude = ['localpart'] + exclude = ['domain'] exclude_by_value = { 'destination': [[]], @@ -429,9 +469,12 @@ class AliasSchema(BaseSchema): def _handle_email(self, data, many, **kwargs): # pylint: disable=unused-argument return handle_email(data) + localpart = fields.Str(load_only=True) + domain_name = fields.Str(load_only=True) destination = CommaSeparatedListField() +@mapped class ConfigSchema(BaseSchema): """ Marshmallow schema for Config model """ class Meta: @@ -440,6 +483,7 @@ class ConfigSchema(BaseSchema): load_instance = True +@mapped class RelaySchema(BaseSchema): """ Marshmallow schema for Relay model """ class Meta: @@ -453,18 +497,43 @@ class MailuSchema(Schema): class Meta: """ Schema config """ render_module = RenderYAML + ordered = True order = ['config', 'domains', 'users', 'aliases', 'relays'] - @post_dump(pass_many=True) - def _order(self, data : OrderedDict, many : bool, **kwargs): # pylint: disable=unused-argument - for key in reversed(self.Meta.order): - try: - data.move_to_end(key, False) - except KeyError: - pass + def __init__(self, *args, **kwargs): + super().__init__(*args, **kwargs) + # order fields + for field_list in self.load_fields, self.dump_fields, self.fields: + for section in reversed(self.Meta.order): + try: + field_list.move_to_end(section, False) + except KeyError: + pass + + @pre_load + def _clear_config(self, data, many, **kwargs): # pylint: disable=unused-argument + """ create config object in context if missing + and clear it if requested + """ + if 'config' not in self.context: + self.context['config'] = models.MailuConfig() + if self.context.get('clear'): + self.context['config'].clear( + models = {field.nested.opts.model for field in self.fields.values()} + ) return data + @post_load + def _make_config(self, data, many, **kwargs): # pylint: disable=unused-argument + """ update and return config object """ + config = self.context['config'] + for section in self.Meta.order: + if section in data: + config.update(data[section], section) + + return config + config = fields.Nested(ConfigSchema, many=True) domains = fields.Nested(DomainSchema, many=True) users = fields.Nested(UserSchema, many=True) From 1c9abf6e481fee92d87be7b8907d8b9fde06479b Mon Sep 17 00:00:00 2001 From: Alexander Graf Date: Sun, 24 Jan 2021 19:27:22 +0100 Subject: [PATCH 063/596] updated requirements for import/export api reqs (flask-restx, ...) are still missing --- core/admin/requirements-prod.txt | 3 +++ core/admin/requirements.txt | 3 +++ 2 files changed, 6 insertions(+) diff --git a/core/admin/requirements-prod.txt b/core/admin/requirements-prod.txt index a3c32855..8ad412cf 100644 --- a/core/admin/requirements-prod.txt +++ b/core/admin/requirements-prod.txt @@ -15,6 +15,7 @@ Flask-Bootstrap==3.3.7.1 Flask-DebugToolbar==0.10.1 Flask-Limiter==1.0.1 Flask-Login==0.4.1 +flask-marshmallow==0.14.0 Flask-Migrate==2.4.0 Flask-Script==2.0.6 Flask-SQLAlchemy==2.4.0 @@ -29,6 +30,8 @@ limits==1.3 Mako==1.0.9 MarkupSafe==1.1.1 mysqlclient==1.4.2.post1 +marshmallow==3.10.0 +marshmallow-sqlalchemy==0.24.1 passlib==1.7.1 psycopg2==2.8.2 pycparser==2.19 diff --git a/core/admin/requirements.txt b/core/admin/requirements.txt index 9739ed3f..59383a07 100644 --- a/core/admin/requirements.txt +++ b/core/admin/requirements.txt @@ -23,3 +23,6 @@ mysqlclient psycopg2 idna srslib +marshmallow +flask-marshmallow +marshmallow-sqlalchemy From 612632e4fc7b498d8fb1fc714af1a02d0e1a1859 Mon Sep 17 00:00:00 2001 From: ofthesun9 Date: Sun, 31 Jan 2021 10:58:41 +0100 Subject: [PATCH 064/596] Need to docker login before pulling images To avoid triggering the Download rate limite --- .travis.yml | 1 + 1 file changed, 1 insertion(+) diff --git a/.travis.yml b/.travis.yml index 467f6f5b..ae50eedb 100644 --- a/.travis.yml +++ b/.travis.yml @@ -34,6 +34,7 @@ install: before_script: - docker-compose -v + - docker login -u $DOCKER_UN -p $DOCKER_PW - docker-compose -f tests/build.yml build - sudo -- sh -c 'mkdir -p /mailu && cp -r tests/certs /mailu && chmod 600 /mailu/certs/*' From 788d069b5379b0ae86e5d955994e144cc4d246f8 Mon Sep 17 00:00:00 2001 From: ofthesun9 Date: Sun, 31 Jan 2021 15:39:32 +0100 Subject: [PATCH 065/596] Modify docker login cmd to use --password-stdin and avoid warning --- .travis.yml | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/.travis.yml b/.travis.yml index ae50eedb..f2a85630 100644 --- a/.travis.yml +++ b/.travis.yml @@ -34,7 +34,7 @@ install: before_script: - docker-compose -v - - docker login -u $DOCKER_UN -p $DOCKER_PW + - echo "$DOCKER_PW" | docker login --username $DOCKER_UN --password-stdin - docker-compose -f tests/build.yml build - sudo -- sh -c 'mkdir -p /mailu && cp -r tests/certs /mailu && chmod 600 /mailu/certs/*' From 906a051925766058feec60c0b768e274ccd7c862 Mon Sep 17 00:00:00 2001 From: Florent Daigniere Date: Sat, 6 Feb 2021 17:23:05 +0100 Subject: [PATCH 066/596] Make rainloop use internal auth --- core/admin/mailu/internal/nginx.py | 19 ++++++++++----- core/admin/mailu/internal/views/auth.py | 12 ++++++++++ core/admin/mailu/models.py | 10 ++++++++ core/admin/mailu/ui/views/base.py | 3 +++ core/nginx/conf/nginx.conf | 20 ++++++++++++++++ webmails/rainloop/Dockerfile | 1 + webmails/rainloop/application.ini | 2 ++ webmails/rainloop/sso.php | 31 +++++++++++++++++++++++++ webmails/rainloop/start.py | 1 + 9 files changed, 93 insertions(+), 6 deletions(-) create mode 100644 webmails/rainloop/sso.php diff --git a/core/admin/mailu/internal/nginx.py b/core/admin/mailu/internal/nginx.py index 1e0b16c2..a41543d3 100644 --- a/core/admin/mailu/internal/nginx.py +++ b/core/admin/mailu/internal/nginx.py @@ -7,7 +7,6 @@ import ipaddress import socket import tenacity - SUPPORTED_AUTH_METHODS = ["none", "plain"] @@ -49,11 +48,19 @@ def handle_authentication(headers): user = models.User.query.get(user_email) status = False if user: - for token in user.tokens: - if (token.check_password(password) and - (not token.ip or token.ip == ip)): - status = True - if user.check_password(password): + # webmails + if len(password) == 64 and ip == app.config['WEBMAIL_ADDRESS']: + if user.verify_temp_token(password): + status = True + + # All tokens are 32 characters hex lowercase + if len(password) == 32: + for token in user.tokens: + if (token.check_password(password) and + (not token.ip or token.ip == ip)): + status = True + break + if not status and user.check_password(password): status = True if status: if protocol == "imap" and not user.enable_imap: diff --git a/core/admin/mailu/internal/views/auth.py b/core/admin/mailu/internal/views/auth.py index 825dba56..338141d8 100644 --- a/core/admin/mailu/internal/views/auth.py +++ b/core/admin/mailu/internal/views/auth.py @@ -43,6 +43,18 @@ def admin_authentication(): return "" return flask.abort(403) +@internal.route("/auth/user") +def user_authentication(): + """ Fails if the user is not authenticated. + """ + if (not flask_login.current_user.is_anonymous + and flask_login.current_user.enabled): + response = flask.Response() + response.headers["X-User"] = flask_login.current_user.get_id() + response.headers["X-User-Token"] = models.User.get_temp_token(flask_login.current_user.get_id()) + return response + return flask.abort(403) + @internal.route("/auth/basic") def basic_authentication(): diff --git a/core/admin/mailu/models.py b/core/admin/mailu/models.py index bbc00f2d..6230e252 100644 --- a/core/admin/mailu/models.py +++ b/core/admin/mailu/models.py @@ -12,6 +12,7 @@ import re import time import os import glob +import hmac import smtplib import idna import dns @@ -425,6 +426,15 @@ class User(Base, Email): user = cls.query.get(email) return user if (user and user.enabled and user.check_password(password)) else None + @classmethod + def get_temp_token(cls, email): + user = cls.query.get(email) + return hmac.new(bytearray(app.secret_key,'utf-8'), bytearray("{}|{}".format(datetime.utcnow().strftime("%Y%m%d"), email), 'utf-8'), 'sha256').hexdigest() if (user and user.enabled) else None + + def verify_temp_token(self, token): + return hmac.compare_digest(b''.fromhex(self.get_temp_token(self.email)), b''.fromhex(token)) + + class Alias(Base, Email): """ An alias is an email address that redirects to some destination. diff --git a/core/admin/mailu/ui/views/base.py b/core/admin/mailu/ui/views/base.py index 7501a883..1aff7db1 100644 --- a/core/admin/mailu/ui/views/base.py +++ b/core/admin/mailu/ui/views/base.py @@ -47,6 +47,9 @@ def announcement(): flask.flash('Your announcement was sent', 'success') return flask.render_template('announcement.html', form=form) +@ui.route('/webmail', methods=['GET']) +def webmail(): + return flask.redirect(app.config['WEB_WEBMAIL']) @ui.route('/client', methods=['GET']) def client(): diff --git a/core/nginx/conf/nginx.conf b/core/nginx/conf/nginx.conf index df598c94..a7a9e134 100644 --- a/core/nginx/conf/nginx.conf +++ b/core/nginx/conf/nginx.conf @@ -133,7 +133,27 @@ http { {% endif %} include /etc/nginx/proxy.conf; client_max_body_size {{ MESSAGE_SIZE_LIMIT|int + 8388608 }}; + auth_request /internal/auth/user; proxy_pass http://$webmail; + error_page 403 @webmail_login; + } + location {{ WEB_WEBMAIL }}/sso.php { + {% if WEB_WEBMAIL != '/' %} + rewrite ^({{ WEB_WEBMAIL }})$ $1/ permanent; + rewrite ^{{ WEB_WEBMAIL }}/(.*) /$1 break; + {% endif %} + include /etc/nginx/proxy.conf; + client_max_body_size {{ MESSAGE_SIZE_LIMIT|int + 8388608 }}; + auth_request /internal/auth/user; + auth_request_set $user $upstream_http_x_user; + auth_request_set $token $upstream_http_x_user_token; + proxy_set_header X-Remote-User $user; + proxy_set_header X-Remote-User-Token $token; + proxy_pass http://$webmail; + error_page 403 @webmail_login; + } + location @webmail_login { + return 302 {{ WEB_ADMIN }}/ui/login?next=ui.webmail; } {% endif %} diff --git a/webmails/rainloop/Dockerfile b/webmails/rainloop/Dockerfile index c67c7496..640d7176 100644 --- a/webmails/rainloop/Dockerfile +++ b/webmails/rainloop/Dockerfile @@ -35,6 +35,7 @@ RUN apt-get update && apt-get install -y \ && rm -rf /var/lib/apt/lists COPY include.php /var/www/html/include.php +COPY sso.php /var/www/html/sso.php COPY php.ini /php.ini COPY application.ini /application.ini diff --git a/webmails/rainloop/application.ini b/webmails/rainloop/application.ini index 5bd9943d..bc953af1 100644 --- a/webmails/rainloop/application.ini +++ b/webmails/rainloop/application.ini @@ -7,6 +7,8 @@ attachment_size_limit = {{ MAX_FILESIZE }} allow_admin_panel = Off [labs] +custom_login_link='sso.php' +custom_logout_link='{{ WEB_ADMIN }}/ui/logout' allow_gravatar = Off [contacts] diff --git a/webmails/rainloop/sso.php b/webmails/rainloop/sso.php new file mode 100644 index 00000000..2415f45c --- /dev/null +++ b/webmails/rainloop/sso.php @@ -0,0 +1,31 @@ + Date: Sat, 6 Feb 2021 18:14:58 +0100 Subject: [PATCH 067/596] Make roundcube use internal auth --- webmails/roundcube/Dockerfile | 1 + webmails/roundcube/config.inc.php | 4 ++- webmails/roundcube/mailu.php | 59 +++++++++++++++++++++++++++++++ webmails/roundcube/start.py | 2 ++ 4 files changed, 65 insertions(+), 1 deletion(-) create mode 100644 webmails/roundcube/mailu.php diff --git a/webmails/roundcube/Dockerfile b/webmails/roundcube/Dockerfile index 79b911b0..da355cc3 100644 --- a/webmails/roundcube/Dockerfile +++ b/webmails/roundcube/Dockerfile @@ -46,6 +46,7 @@ RUN apt-get update && apt-get install -y \ COPY php.ini /php.ini COPY config.inc.php /var/www/html/config/ +COPY mailu.php /var/www/html/plugins/mailu/mailu.php COPY start.py /start.py EXPOSE 80/tcp diff --git a/webmails/roundcube/config.inc.php b/webmails/roundcube/config.inc.php index eb40047a..bb1a5e84 100644 --- a/webmails/roundcube/config.inc.php +++ b/webmails/roundcube/config.inc.php @@ -17,7 +17,8 @@ $config['plugins'] = array( 'markasjunk', 'managesieve', 'enigma', - 'carddav' + 'carddav', + 'mailu' ); $front = getenv('FRONT_ADDRESS') ? getenv('FRONT_ADDRESS') : 'front'; @@ -37,6 +38,7 @@ $config['managesieve_usetls'] = false; // Customization settings $config['support_url'] = getenv('WEB_ADMIN') ? '../..' . getenv('WEB_ADMIN') : ''; +$config['sso_logout_url'] = getenv('WEB_ADMIN').'/ui/logout'; $config['product_name'] = 'Mailu Webmail'; // We access the IMAP and SMTP servers locally with internal names, SSL diff --git a/webmails/roundcube/mailu.php b/webmails/roundcube/mailu.php new file mode 100644 index 00000000..bb4d65e9 --- /dev/null +++ b/webmails/roundcube/mailu.php @@ -0,0 +1,59 @@ +add_hook('startup', array($this, 'startup')); + $this->add_hook('authenticate', array($this, 'authenticate')); + $this->add_hook('login_after', array($this, 'login')); + $this->add_hook('login_failed', array($this, 'login_failed')); + $this->add_hook('logout_after', array($this, 'logout')); + } + + function startup($args) + { + if (empty($_SESSION['user_id'])) { + $args['action'] = 'login'; + } + + return $args; + } + + function authenticate($args) + { + if (!in_array('HTTP_X_REMOTE_USER', $_SERVER) || !in_array('HTTP_X_REMOTE_USER_TOKEN', $_SERVER)) { + header('HTTP/1.0 403 Forbidden'); + die(); + } + $args['user'] = $_SERVER['HTTP_X_REMOTE_USER']; + $args['pass'] = $_SERVER['HTTP_X_REMOTE_USER_TOKEN']; + + $args['cookiecheck'] = false; + $args['valid'] = true; + + return $args; + } + + function logout($args) { + // Redirect to global SSO logout path. + $this->load_config(); + + $sso_logout_url = rcmail::get_instance()->config->get('sso_logout_url'); + header("Location: " . $sso_logout_url, true); + exit; + } + + function login($args) + { + header('Location: index.php'); + exit(); + } + function login_failed($args) + { + header('Location: sso.php'); + exit(); + } + +} diff --git a/webmails/roundcube/start.py b/webmails/roundcube/start.py index 649f3324..9ce383c8 100755 --- a/webmails/roundcube/start.py +++ b/webmails/roundcube/start.py @@ -39,6 +39,8 @@ conf.jinja("/php.ini", os.environ, "/usr/local/etc/php/conf.d/roundcube.ini") os.system("mkdir -p /data/gpg /var/www/html/logs") os.system("touch /var/www/html/logs/errors.log") os.system("chown -R www-data:www-data /var/www/html/logs") +os.system("chmod -R a+rX /var/www/html/") +os.system("ln -s /var/www/html/index.php /var/www/html/sso.php") try: print("Initializing database") From 99c7420f923a8b2c44860d25c196fcf2c00280e4 Mon Sep 17 00:00:00 2001 From: Florent Daigniere Date: Sun, 7 Feb 2021 17:51:19 +0100 Subject: [PATCH 068/596] towncrier --- towncrier/newsfragments/783.feature | 1 + 1 file changed, 1 insertion(+) create mode 100644 towncrier/newsfragments/783.feature diff --git a/towncrier/newsfragments/783.feature b/towncrier/newsfragments/783.feature new file mode 100644 index 00000000..fcafceef --- /dev/null +++ b/towncrier/newsfragments/783.feature @@ -0,0 +1 @@ +Centralize the authentication of webmails behind the admin interface From ef637f51b7f620c88dff403cf1fdb4a74ccbe663 Mon Sep 17 00:00:00 2001 From: Florent Daigniere Date: Sun, 7 Feb 2021 17:58:19 +0100 Subject: [PATCH 069/596] derive the SSO keys from a KDF --- core/admin/mailu/__init__.py | 3 +++ core/admin/mailu/models.py | 4 ++-- 2 files changed, 5 insertions(+), 2 deletions(-) diff --git a/core/admin/mailu/__init__.py b/core/admin/mailu/__init__.py index 4de3e580..b37b4493 100644 --- a/core/admin/mailu/__init__.py +++ b/core/admin/mailu/__init__.py @@ -3,6 +3,7 @@ import flask_bootstrap from mailu import utils, debug, models, manage, configuration +import hmac def create_app_from_config(config): """ Create a new application based on the given configuration @@ -24,6 +25,8 @@ def create_app_from_config(config): utils.proxy.init_app(app) utils.migrate.init_app(app, models.db) + app.temp_token_key = hmac.new(bytearray(app.secret_key, 'utf-8'), bytearray('WEBMAIL_TEMP_TOKEN_KEY', 'utf-8'), 'sha256').digest() + # Initialize debugging tools if app.config.get("DEBUG"): debug.toolbar.init_app(app) diff --git a/core/admin/mailu/models.py b/core/admin/mailu/models.py index 6230e252..c3f79b92 100644 --- a/core/admin/mailu/models.py +++ b/core/admin/mailu/models.py @@ -429,10 +429,10 @@ class User(Base, Email): @classmethod def get_temp_token(cls, email): user = cls.query.get(email) - return hmac.new(bytearray(app.secret_key,'utf-8'), bytearray("{}|{}".format(datetime.utcnow().strftime("%Y%m%d"), email), 'utf-8'), 'sha256').hexdigest() if (user and user.enabled) else None + return hmac.new(app.temp_token_key, bytearray("{}|{}".format(datetime.utcnow().strftime("%Y%m%d"), email), 'utf-8'), 'sha256').hexdigest() if (user and user.enabled) else None def verify_temp_token(self, token): - return hmac.compare_digest(b''.fromhex(self.get_temp_token(self.email)), b''.fromhex(token)) + return hmac.compare_digest(self.get_temp_token(self.email), token) From b49554bec1156c69c5f24c5ca62cbb9a818ac8a7 Mon Sep 17 00:00:00 2001 From: Florent Daigniere Date: Sun, 7 Feb 2021 18:08:58 +0100 Subject: [PATCH 070/596] merge artifact --- core/admin/mailu/internal/nginx.py | 2 +- core/admin/mailu/ui/views/base.py | 1 + 2 files changed, 2 insertions(+), 1 deletion(-) diff --git a/core/admin/mailu/internal/nginx.py b/core/admin/mailu/internal/nginx.py index a41543d3..b2ee3fba 100644 --- a/core/admin/mailu/internal/nginx.py +++ b/core/admin/mailu/internal/nginx.py @@ -54,7 +54,7 @@ def handle_authentication(headers): status = True # All tokens are 32 characters hex lowercase - if len(password) == 32: + if not status and len(password) == 32: for token in user.tokens: if (token.check_password(password) and (not token.ip or token.ip == ip)): diff --git a/core/admin/mailu/ui/views/base.py b/core/admin/mailu/ui/views/base.py index 1aff7db1..457fd7fb 100644 --- a/core/admin/mailu/ui/views/base.py +++ b/core/admin/mailu/ui/views/base.py @@ -1,6 +1,7 @@ from mailu import models from mailu.ui import ui, forms, access +from flask import current_app as app import flask import flask_login From 2e749abe6118f9a2c414ac239efff099dc3fdb1a Mon Sep 17 00:00:00 2001 From: Florent Daigniere Date: Mon, 25 Jan 2021 13:19:21 +0100 Subject: [PATCH 071/596] DNS records for client autoconfiguration (RFC6186) --- core/admin/mailu/ui/templates/domain/details.html | 12 ++++++++++++ towncrier/newsfragments/224.feature | 1 + 2 files changed, 13 insertions(+) create mode 100644 towncrier/newsfragments/224.feature diff --git a/core/admin/mailu/ui/templates/domain/details.html b/core/admin/mailu/ui/templates/domain/details.html index 65c6ec1a..5822ffbc 100644 --- a/core/admin/mailu/ui/templates/domain/details.html +++ b/core/admin/mailu/ui/templates/domain/details.html @@ -50,5 +50,17 @@ 
_dmarc.{{ domain.name }}. 600 IN TXT "v=DMARC1; p=reject;{% if config["DMARC_RUA"] %} rua=mailto:{{ config["DMARC_RUA"] }}@{{ config["DOMAIN"] }};{% endif %}{% if config["DMARC_RUF"] %} ruf=mailto:{{ config["DMARC_RUF"] }}@{{ config["DOMAIN"] }};{% endif %} adkim=s; aspf=s"
{% endif %} + + {% trans %}DNS client auto-configuration (RFC6186) entries{% endtrans %} + + 
_submission._tcp.{{ domain.name }}. 600 IN SRV 1 1 587 {{ config["HOSTNAMES"].split(',')[0] }}.
+ 
_imap._tcp.{{ domain.name }}. 600 IN SRV 100 1 143 {{ config["HOSTNAMES"].split(',')[0] }}.
+ 
_pop3._tcp.{{ domain.name }}. 600 IN SRV 100 1 110 {{ config["HOSTNAMES"].split(',')[0] }}.
+{% if config["TLS_FLAVOR"] != "notls" %} + 
_submissions._tcp.{{ domain.name }}. 600 IN SRV 10 1 465 {{ config["HOSTNAMES"].split(',')[0] }}.
+ 
_imaps._tcp.{{ domain.name }}. 600 IN SRV 10 1 993 {{ config["HOSTNAMES"].split(',')[0] }}.
+ 
_pop3s._tcp.{{ domain.name }}. 600 IN SRV 10 1 995 {{ config["HOSTNAMES"].split(',')[0] }}.
+{% endif %} + {% endcall %} {% endblock %} diff --git a/towncrier/newsfragments/224.feature b/towncrier/newsfragments/224.feature new file mode 100644 index 00000000..9a2f479b --- /dev/null +++ b/towncrier/newsfragments/224.feature @@ -0,0 +1 @@ +Add instructions on how to create DNS records for email client auto-configuration (RFC6186 style) From 80f939cf1ac3fe9863825de1360ee7c5d67926f6 Mon Sep 17 00:00:00 2001 From: Florent Daigniere Date: Mon, 8 Feb 2021 10:16:03 +0100 Subject: [PATCH 072/596] Revert to the old behaviour when ADMIN=false --- core/nginx/conf/nginx.conf | 10 +++++++--- webmails/rainloop/application.ini | 4 +++- webmails/roundcube/config.inc.php | 10 ++++++---- 3 files changed, 16 insertions(+), 8 deletions(-) diff --git a/core/nginx/conf/nginx.conf b/core/nginx/conf/nginx.conf index a7a9e134..81f1ac0d 100644 --- a/core/nginx/conf/nginx.conf +++ b/core/nginx/conf/nginx.conf @@ -133,10 +133,12 @@ http { {% endif %} include /etc/nginx/proxy.conf; client_max_body_size {{ MESSAGE_SIZE_LIMIT|int + 8388608 }}; - auth_request /internal/auth/user; proxy_pass http://$webmail; + {% if ADMIN == 'true' %} + auth_request /internal/auth/user; error_page 403 @webmail_login; } + location {{ WEB_WEBMAIL }}/sso.php { {% if WEB_WEBMAIL != '/' %} rewrite ^({{ WEB_WEBMAIL }})$ $1/ permanent; @@ -152,11 +154,13 @@ http { proxy_pass http://$webmail; error_page 403 @webmail_login; } + location @webmail_login { return 302 {{ WEB_ADMIN }}/ui/login?next=ui.webmail; } - {% endif %} - + {% else %} + } + {% endif %}{% endif %} {% if ADMIN == 'true' %} location {{ WEB_ADMIN }} { return 301 {{ WEB_ADMIN }}/ui; diff --git a/webmails/rainloop/application.ini b/webmails/rainloop/application.ini index bc953af1..0504f174 100644 --- a/webmails/rainloop/application.ini +++ b/webmails/rainloop/application.ini @@ -7,9 +7,11 @@ attachment_size_limit = {{ MAX_FILESIZE }} allow_admin_panel = Off [labs] +allow_gravatar = Off +{% if ADMIN == "true" %} custom_login_link='sso.php' custom_logout_link='{{ WEB_ADMIN }}/ui/logout' -allow_gravatar = Off +{% endif %} [contacts] enable = On diff --git a/webmails/roundcube/config.inc.php b/webmails/roundcube/config.inc.php index bb1a5e84..d8028db3 100644 --- a/webmails/roundcube/config.inc.php +++ b/webmails/roundcube/config.inc.php @@ -17,8 +17,7 @@ $config['plugins'] = array( 'markasjunk', 'managesieve', 'enigma', - 'carddav', - 'mailu' + 'carddav' ); $front = getenv('FRONT_ADDRESS') ? getenv('FRONT_ADDRESS') : 'front'; @@ -37,8 +36,11 @@ $config['managesieve_host'] = $imap; $config['managesieve_usetls'] = false; // Customization settings -$config['support_url'] = getenv('WEB_ADMIN') ? '../..' . getenv('WEB_ADMIN') : ''; -$config['sso_logout_url'] = getenv('WEB_ADMIN').'/ui/logout'; +if (filter_var(getenv('ADMIN'), FILTER_VALIDATE_BOOLEAN, FILTER_NULL_ON_FAILURE)) { + array_push($config['plugins'], 'mailu'); + $config['support_url'] = getenv('WEB_ADMIN') ? '../..' . getenv('WEB_ADMIN') : ''; + $config['sso_logout_url'] = getenv('WEB_ADMIN').'/ui/logout'; +} $config['product_name'] = 'Mailu Webmail'; // We access the IMAP and SMTP servers locally with internal names, SSL From 0917a6817f845c01a2247d74f111abc2ef54e51a Mon Sep 17 00:00:00 2001 From: Florent Daigniere Date: Mon, 8 Feb 2021 10:17:43 +0100 Subject: [PATCH 073/596] Set ADMIN=false to ensure that the tests pass --- tests/compose/rainloop/mailu.env | 2 +- tests/compose/roundcube/mailu.env | 2 +- 2 files changed, 2 insertions(+), 2 deletions(-) diff --git a/tests/compose/rainloop/mailu.env b/tests/compose/rainloop/mailu.env index 9c31c8bb..47b0b934 100644 --- a/tests/compose/rainloop/mailu.env +++ b/tests/compose/rainloop/mailu.env @@ -51,7 +51,7 @@ DISABLE_STATISTICS=False ################################### # Expose the admin interface (value: true, false) -ADMIN=true +ADMIN=false # Choose which webmail to run if any (values: roundcube, rainloop, none) WEBMAIL=rainloop diff --git a/tests/compose/roundcube/mailu.env b/tests/compose/roundcube/mailu.env index dc503268..887cebbd 100644 --- a/tests/compose/roundcube/mailu.env +++ b/tests/compose/roundcube/mailu.env @@ -51,7 +51,7 @@ DISABLE_STATISTICS=False ################################### # Expose the admin interface (value: true, false) -ADMIN=true +ADMIN=false # Choose which webmail to run if any (values: roundcube, rainloop, none) WEBMAIL=roundcube From e8f70c12dcdce1064ea063cc5ddcae28a5221613 Mon Sep 17 00:00:00 2001 From: Florent Daigniere Date: Mon, 8 Feb 2021 10:22:25 +0100 Subject: [PATCH 074/596] avoid a warning --- webmails/roundcube/start.py | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/webmails/roundcube/start.py b/webmails/roundcube/start.py index 9ce383c8..45b2aa76 100755 --- a/webmails/roundcube/start.py +++ b/webmails/roundcube/start.py @@ -40,7 +40,7 @@ os.system("mkdir -p /data/gpg /var/www/html/logs") os.system("touch /var/www/html/logs/errors.log") os.system("chown -R www-data:www-data /var/www/html/logs") os.system("chmod -R a+rX /var/www/html/") -os.system("ln -s /var/www/html/index.php /var/www/html/sso.php") +os.system("ln -sf /var/www/html/index.php /var/www/html/sso.php") try: print("Initializing database") From 6c4fa5432f3f040cc16fca870f0382939ae2817b Mon Sep 17 00:00:00 2001 From: Nils Vogels Date: Thu, 11 Feb 2021 12:03:07 +0100 Subject: [PATCH 075/596] Provide fix in postgresql container for CVE-2021-23240, CVE-2021-3156, CVE-2021-23239 --- optional/postgresql/Dockerfile | 1 + 1 file changed, 1 insertion(+) diff --git a/optional/postgresql/Dockerfile b/optional/postgresql/Dockerfile index 95048147..ff25a66f 100644 --- a/optional/postgresql/Dockerfile +++ b/optional/postgresql/Dockerfile @@ -3,6 +3,7 @@ FROM $DISTRO # python3 shared with most images RUN apk add --no-cache \ python3 py3-pip bash py3-multidict \ + && apk add --upgrade sudo \ && pip3 install --upgrade pip # Shared layer between nginx, dovecot, postfix, postgresql, rspamd, unbound, rainloop, roundcube From 3b7ecb3a8b4c466ad2f37276107bb49efe978758 Mon Sep 17 00:00:00 2001 From: Nils Vogels Date: Thu, 11 Feb 2021 12:12:06 +0100 Subject: [PATCH 076/596] Add changelog --- towncrier/newsfragments/1760.bugfix | 2 ++ 1 file changed, 2 insertions(+) create mode 100644 towncrier/newsfragments/1760.bugfix diff --git a/towncrier/newsfragments/1760.bugfix b/towncrier/newsfragments/1760.bugfix new file mode 100644 index 00000000..9d6f38af --- /dev/null +++ b/towncrier/newsfragments/1760.bugfix @@ -0,0 +1,2 @@ +Fix CVE-2021-23240, CVE-2021-3156 and CVE-2021-23239 for postgresql +by force-upgrading sudo. From 1a365f469c85adfe125f04a4b29d6cdabe436709 Mon Sep 17 00:00:00 2001 From: lub Date: Fri, 12 Feb 2021 12:18:22 +0100 Subject: [PATCH 077/596] check for `ipv6_enabled` in the compose template Checking only `ipv6` isn't sufficient, because it has a default value. --- setup/flavors/compose/docker-compose.yml | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/setup/flavors/compose/docker-compose.yml b/setup/flavors/compose/docker-compose.yml index 3fbfb862..155e1180 100644 --- a/setup/flavors/compose/docker-compose.yml +++ b/setup/flavors/compose/docker-compose.yml @@ -26,7 +26,7 @@ services: {% if bind4 %} - "{{ bind4 }}:{{ port }}:{{ port }}" {% endif %} - {% if bind6 %} + {% if ipv6_enabled and bind6 %} - "{{ bind6 }}:{{ port }}:{{ port }}" {% endif %} {% endfor %} From 88f992de16ff6273b345136a0dca068c33cea9cc Mon Sep 17 00:00:00 2001 From: lub Date: Sat, 13 Feb 2021 13:34:44 +0100 Subject: [PATCH 078/596] show flash messages again This basically restores the behaviour, that got removed in ecdf0c25b3d9bbaa028bdc46cb721d2fb406dde2 during refactoring. --- core/admin/mailu/ui/templates/base.html | 2 ++ 1 file changed, 2 insertions(+) diff --git a/core/admin/mailu/ui/templates/base.html b/core/admin/mailu/ui/templates/base.html index 8a841e47..74d5653c 100644 --- a/core/admin/mailu/ui/templates/base.html +++ b/core/admin/mailu/ui/templates/base.html @@ -1,4 +1,5 @@ {% import "macros.html" as macros %} +{% import "bootstrap/utils.html" as utils %} @@ -37,6 +38,7 @@
+ {{ utils.flashed_messages(container=False) }} {% block content %}{% endblock %}
From 3ada506dbd08451deb24bfefa53efa7f4fc18b19 Mon Sep 17 00:00:00 2001 From: Stephan Holl <1610827+sholl@users.noreply.github.com> Date: Sat, 13 Feb 2021 17:35:33 +0100 Subject: [PATCH 079/596] Update docs/faq.rst as @lub suggests Co-authored-by: lub --- docs/faq.rst | 6 +++--- 1 file changed, 3 insertions(+), 3 deletions(-) diff --git a/docs/faq.rst b/docs/faq.rst index 0a171bc9..f200199d 100644 --- a/docs/faq.rst +++ b/docs/faq.rst @@ -258,9 +258,9 @@ Postfix, Dovecot, Nginx and Rspamd support overriding configuration files. Overr correct syntax. The following file names will be taken as override configuration: - `Postfix`_ : - - ``postfix.cf`` as ``/overrides/postfix.cf`` - - ``master.cf`` as ``/overrides/postfix.master`` - - All ``/overrides/*.map`` files + - ``main.cf`` as ``$ROOT/overrides/postfix/postfix.cf`` + - ``master.cf`` as ``$ROOT/overrides/postfix/postfix.master`` + - All ``$ROOT/overrides/postfix/*.map`` files - `Dovecot`_ - ``dovecot.conf`` in dovecot sub-directory; - `Nginx`_ - All ``*.conf`` files in the ``nginx`` sub-directory; - `Rspamd`_ - All files in the ``rspamd`` sub-directory. From 68caf501549d08c048a7c232ac72bb8a072dd9e6 Mon Sep 17 00:00:00 2001 From: Alexander Graf Date: Mon, 15 Feb 2021 00:46:59 +0100 Subject: [PATCH 080/596] new import/export using marshmallow --- core/admin/mailu/manage.py | 218 +++++---- core/admin/mailu/models.py | 132 +++--- core/admin/mailu/schemas.py | 608 +++++++++++++++++++++----- docs/cli.rst | 221 ++++++---- tests/compose/core/02_forward_test.sh | 4 +- tests/compose/core/03_alias_test.sh | 4 +- tests/compose/core/04_reply_test.sh | 4 +- towncrier/newsfragments/1604.feature | 2 +- 8 files changed, 851 insertions(+), 342 deletions(-) diff --git a/core/admin/mailu/manage.py b/core/admin/mailu/manage.py index e02d9ad4..a8d1d3cb 100644 --- a/core/admin/mailu/manage.py +++ b/core/admin/mailu/manage.py @@ -4,7 +4,6 @@ import sys import os import socket -import json import logging import uuid @@ -20,7 +19,7 @@ from flask.cli import FlaskGroup, with_appcontext from marshmallow.exceptions import ValidationError from . import models -from .schemas import MailuSchema, get_schema +from .schemas import MailuSchema, get_schema, get_fieldspec, colorize, RenderJSON, HIDDEN db = models.db @@ -182,7 +181,7 @@ def user_import(localpart, domain_name, password_hash, hash_scheme = None): db.session.commit() -# TODO: remove this deprecated function +# TODO: remove deprecated config_update function? @mailu.command() @click.option('-v', '--verbose') @click.option('-d', '--delete-objects') @@ -324,17 +323,16 @@ def config_update(verbose=False, delete_objects=False): db.session.commit() -SECTIONS = {'domains', 'relays', 'users', 'aliases'} - - @mailu.command() -@click.option('-v', '--verbose', count=True, help='Increase verbosity') -@click.option('-q', '--quiet', is_flag=True, help='Quiet mode - only show errors') -@click.option('-u', '--update', is_flag=True, help='Update mode - merge input with existing config') -@click.option('-n', '--dry-run', is_flag=True, help='Perform a trial run with no changes made') +@click.option('-v', '--verbose', count=True, help='Increase verbosity.') +@click.option('-s', '--secrets', is_flag=True, help='Show secret attributes in messages.') +@click.option('-q', '--quiet', is_flag=True, help='Quiet mode - only show errors.') +@click.option('-c', '--color', is_flag=True, help='Force colorized output.') +@click.option('-u', '--update', is_flag=True, help='Update mode - merge input with existing config.') +@click.option('-n', '--dry-run', is_flag=True, help='Perform a trial run with no changes made.') @click.argument('source', metavar='[FILENAME|-]', type=click.File(mode='r'), default=sys.stdin) @with_appcontext -def config_import(verbose=0, quiet=False, update=False, dry_run=False, source=None): +def config_import(verbose=0, secrets=False, quiet=False, color=False, update=False, dry_run=False, source=None): """ Import configuration as YAML or JSON from stdin or file """ @@ -344,12 +342,19 @@ def config_import(verbose=0, quiet=False, update=False, dry_run=False, source=No # 2 : also show secrets # 3 : also show input data # 4 : also show sql queries + # 5 : also show tracebacks if quiet: verbose = -1 + color_cfg = { + 'color': color or sys.stdout.isatty(), + 'lexer': 'python', + 'strip': True, + } + counter = Counter() - dumper = {} + logger = {} def format_errors(store, path=None): @@ -387,19 +392,26 @@ def config_import(verbose=0, quiet=False, update=False, dry_run=False, source=No last = action changes.append(f'{what}({count})') else: - changes = 'no changes.' + changes = ['No changes.'] return chain(message, changes) def log(action, target, message=None): if message is None: - message = json.dumps(dumper[target.__class__].dump(target), ensure_ascii=False) - print(f'{action} {target.__table__}: {message}') + # TODO: convert nested OrderedDict to dict + # see: flask mailu config-import -nvv yaml/dump4.yaml + try: + message = dict(logger[target.__class__].dump(target)) + except KeyError: + message = target + if not isinstance(message, str): + message = repr(message) + print(f'{action} {target.__table__}: {colorize(message, **color_cfg)}') def listen_insert(mapper, connection, target): # pylint: disable=unused-argument """ callback function to track import """ - counter.update([('Added', target.__table__.name)]) + counter.update([('Created', target.__table__.name)]) if verbose >= 1: - log('Added', target) + log('Created', target) def listen_update(mapper, connection, target): # pylint: disable=unused-argument """ callback function to track import """ @@ -407,32 +419,32 @@ def config_import(verbose=0, quiet=False, update=False, dry_run=False, source=No changed = {} inspection = sqlalchemy.inspect(target) for attr in sqlalchemy.orm.class_mapper(target.__class__).column_attrs: - if getattr(inspection.attrs, attr.key).history.has_changes(): - if sqlalchemy.orm.attributes.get_history(target, attr.key)[2]: - before = sqlalchemy.orm.attributes.get_history(target, attr.key)[2].pop() - after = getattr(target, attr.key) - # only remember changed keys - if before != after and (before or after): - if verbose >= 1: - changed[str(attr.key)] = (before, after) - else: - break + history = getattr(inspection.attrs, attr.key).history + if history.has_changes() and history.deleted: + before = history.deleted[-1] + after = getattr(target, attr.key) + # TODO: remove special handling of "comment" after modifying model + if attr.key == 'comment' and not before and not after: + pass + # only remember changed keys + elif before != after: + if verbose >= 1: + changed[str(attr.key)] = (before, after) + else: + break if verbose >= 1: # use schema with dump_context to hide secrets and sort keys - primary = json.dumps(str(target), ensure_ascii=False) - dumped = get_schema(target)(only=changed.keys(), context=dump_context).dump(target) + dumped = get_schema(target)(only=changed.keys(), context=diff_context).dump(target) for key, value in dumped.items(): before, after = changed[key] - if value == '': - before = '' if before else before - after = '' if after else after + if value == HIDDEN: + before = HIDDEN if before else before + after = HIDDEN if after else after else: - # TODO: use schema to "convert" before value? + # TODO: need to use schema to "convert" before value? after = value - before = json.dumps(before, ensure_ascii=False) - after = json.dumps(after, ensure_ascii=False) - log('Modified', target, f'{primary} {key}: {before} -> {after}') + log('Modified', target, f'{str(target)!r} {key}: {before!r} -> {after!r}') if changed: counter.update([('Modified', target.__table__.name)]) @@ -443,47 +455,60 @@ def config_import(verbose=0, quiet=False, update=False, dry_run=False, source=No if verbose >= 1: log('Deleted', target) - # this listener should not be necessary, when: - # dkim keys should be stored in database and it should be possible to store multiple - # keys per domain. the active key would be also stored on disk on commit. + # TODO: this listener will not be necessary, if dkim keys would be stored in database + _dedupe_dkim = set() def listen_dkim(session, flush_context): # pylint: disable=unused-argument """ callback function to track import """ for target in session.identity_map.values(): - if not isinstance(target, models.Domain): + # look at Domains originally loaded from db + if not isinstance(target, models.Domain) or not target._sa_instance_state.load_path: continue - primary = json.dumps(str(target), ensure_ascii=False) before = target._dkim_key_on_disk after = target._dkim_key - if before != after and (before or after): - if verbose >= 2: + if before != after: + if secrets: before = before.decode('ascii', 'ignore') after = after.decode('ascii', 'ignore') else: - before = '' if before else '' - after = '' if after else '' - before = json.dumps(before, ensure_ascii=False) - after = json.dumps(after, ensure_ascii=False) - log('Modified', target, f'{primary} dkim_key: {before} -> {after}') - counter.update([('Modified', target.__table__.name)]) + before = HIDDEN if before else '' + after = HIDDEN if after else '' + # "de-dupe" messages; this event is fired at every flush + if not (target, before, after) in _dedupe_dkim: + _dedupe_dkim.add((target, before, after)) + counter.update([('Modified', target.__table__.name)]) + if verbose >= 1: + log('Modified', target, f'{str(target)!r} dkim_key: {before!r} -> {after!r}') - def track_serialize(self, item): + def track_serialize(obj, item): """ callback function to track import """ - log('Handling', self.opts.model, item) + # hide secrets + data = logger[obj.opts.model].hide(item) + if 'hash_password' in data: + data['password'] = HIDDEN + if 'fetches' in data: + for fetch in data['fetches']: + fetch['password'] = HIDDEN + log('Handling', obj.opts.model, data) # configure contexts - dump_context = { - 'secrets': verbose >= 2, + diff_context = { + 'full': True, + 'secrets': secrets, + } + log_context = { + 'secrets': secrets, } load_context = { - 'callback': track_serialize if verbose >= 3 else None, - 'clear': not update, 'import': True, + 'update': update, + 'clear': not update, + 'callback': track_serialize if verbose >= 2 else None, } # register listeners for schema in get_schema(): model = schema.Meta.model - dumper[model] = schema(context=dump_context) + logger[model] = schema(context=log_context) sqlalchemy.event.listen(model, 'after_insert', listen_insert) sqlalchemy.event.listen(model, 'after_update', listen_update) sqlalchemy.event.listen(model, 'after_delete', listen_delete) @@ -491,18 +516,24 @@ def config_import(verbose=0, quiet=False, update=False, dry_run=False, source=No # special listener for dkim_key changes sqlalchemy.event.listen(db.session, 'after_flush', listen_dkim) - if verbose >= 4: + if verbose >= 3: logging.basicConfig() logging.getLogger('sqlalchemy.engine').setLevel(logging.INFO) try: with models.db.session.no_autoflush: - config = MailuSchema(only=SECTIONS, context=load_context).loads(source) + config = MailuSchema(only=MailuSchema.Meta.order, context=load_context).loads(source) except ValidationError as exc: raise click.ClickException(format_errors(exc.messages)) from exc except Exception as exc: - # (yaml.scanner.ScannerError, UnicodeDecodeError, ...) - raise click.ClickException(f'[{exc.__class__.__name__}] {" ".join(str(exc).split())}') from exc + if verbose >= 5: + raise + else: + # (yaml.scanner.ScannerError, UnicodeDecodeError, ...) + raise click.ClickException( + f'[{exc.__class__.__name__}] ' + f'{" ".join(str(exc).split())}' + ) from exc # flush session to show/count all changes if dry_run or verbose >= 1: @@ -510,53 +541,47 @@ def config_import(verbose=0, quiet=False, update=False, dry_run=False, source=No # check for duplicate domain names dup = set() - for fqdn in chain(db.session.query(models.Domain.name), - db.session.query(models.Alternative.name), - db.session.query(models.Relay.name)): + for fqdn in chain( + db.session.query(models.Domain.name), + db.session.query(models.Alternative.name), + db.session.query(models.Relay.name) + ): if fqdn in dup: raise click.ClickException(f'[ValidationError] Duplicate domain name: {fqdn}') dup.add(fqdn) - # TODO: implement special update "items" - # -pkey: which - remove item "which" - # -key: null or [] or {} - set key to default - # -pkey: null or [] or {} - remove all existing items in this list - # don't commit when running dry if dry_run: - db.session.rollback() if not quiet: print(*format_changes('Dry run. Not commiting changes.')) - # TODO: remove debug - print(MailuSchema().dumps(config)) + db.session.rollback() else: - db.session.commit() if not quiet: - print(*format_changes('Commited changes.')) + print(*format_changes('Committing changes.')) + db.session.commit() @mailu.command() -@click.option('-f', '--full', is_flag=True, help='Include attributes with default value') +@click.option('-f', '--full', is_flag=True, help='Include attributes with default value.') @click.option('-s', '--secrets', is_flag=True, - help='Include secret attributes (dkim-key, passwords)') -@click.option('-d', '--dns', is_flag=True, help='Include dns records') + help='Include secret attributes (dkim-key, passwords).') +@click.option('-c', '--color', is_flag=True, help='Force colorized output.') +@click.option('-d', '--dns', is_flag=True, help='Include dns records.') @click.option('-o', '--output-file', 'output', default=sys.stdout, type=click.File(mode='w'), - help='save yaml to file') -@click.option('-j', '--json', 'as_json', is_flag=True, help='Dump in josn format') -@click.argument('sections', nargs=-1) + help='Save configuration to file.') +@click.option('-j', '--json', 'as_json', is_flag=True, help='Export configuration in json format.') +@click.argument('only', metavar='[FILTER]...', nargs=-1) @with_appcontext -def config_export(full=False, secrets=False, dns=False, output=None, as_json=False, sections=None): +def config_export(full=False, secrets=False, color=False, dns=False, output=None, as_json=False, only=None): """ Export configuration as YAML or JSON to stdout or file """ - if sections: - for section in sections: - if section not in SECTIONS: - print(f'[ERROR] Unknown section: {section}') - raise click.exceptions.Exit(1) - sections = set(sections) + if only: + for spec in only: + if spec.split('.', 1)[0] not in MailuSchema.Meta.order: + raise click.ClickException(f'[ERROR] Unknown section: {spec}') else: - sections = SECTIONS + only = MailuSchema.Meta.order context = { 'full': full, @@ -564,13 +589,20 @@ def config_export(full=False, secrets=False, dns=False, output=None, as_json=Fal 'dns': dns, } - if as_json: - schema = MailuSchema(only=sections, context=context) - schema.opts.render_module = json - print(schema.dumps(models.MailuConfig(), separators=(',',':')), file=output) + schema = MailuSchema(only=only, context=context) + color_cfg = {'color': color or output.isatty()} - else: - MailuSchema(only=sections, context=context).dumps(models.MailuConfig(), output) + if as_json: + schema.opts.render_module = RenderJSON + color_cfg['lexer'] = 'json' + color_cfg['strip'] = True + + try: + print(colorize(schema.dumps(models.MailuConfig()), **color_cfg), file=output) + except ValueError as exc: + if spec := get_fieldspec(exc): + raise click.ClickException(f'[ERROR] Invalid filter: {spec}') from exc + raise @mailu.command() diff --git a/core/admin/mailu/models.py b/core/admin/mailu/models.py index dac1dc70..5799e282 100644 --- a/core/admin/mailu/models.py +++ b/core/admin/mailu/models.py @@ -19,6 +19,7 @@ import dns from flask import current_app as app from sqlalchemy.ext import declarative +from sqlalchemy.ext.hybrid import hybrid_property from sqlalchemy.inspection import inspect from werkzeug.utils import cached_property @@ -121,6 +122,36 @@ class Base(db.Model): updated_at = db.Column(db.Date, nullable=True, onupdate=date.today) comment = db.Column(db.String(255), nullable=True, default='') + def __str__(self): + pkey = self.__table__.primary_key.columns.values()[0].name + if pkey == 'email': + # ugly hack for email declared attr. _email is not always up2date + return str(f'{self.localpart}@{self.domain_name}') + elif pkey in {'name', 'email'}: + return str(getattr(self, pkey, None)) + else: + return self.__repr__() + return str(getattr(self, self.__table__.primary_key.columns.values()[0].name)) + + def __repr__(self): + return f'<{self.__class__.__name__} {str(self)!r}>' + + def __eq__(self, other): + if isinstance(other, self.__class__): + pkey = self.__table__.primary_key.columns.values()[0].name + this = getattr(self, pkey, None) + other = getattr(other, pkey, None) + return this is not None and other is not None and str(this) == str(other) + else: + return NotImplemented + + def __hash__(self): + primary = getattr(self, self.__table__.primary_key.columns.values()[0].name) + if primary is None: + return NotImplemented + else: + return hash(primary) + # Many-to-many association table for domain managers managers = db.Table('manager', Base.metadata, @@ -261,19 +292,6 @@ class Domain(Base): except dns.exception.DNSException: return False - def __str__(self): - return str(self.name) - - def __eq__(self, other): - if isinstance(other, self.__class__): - return str(self.name) == str(other.name) - else: - return NotImplemented - - def __hash__(self): - return hash(str(self.name)) - - class Alternative(Base): """ Alternative name for a served domain. @@ -287,9 +305,6 @@ class Alternative(Base): domain = db.relationship(Domain, backref=db.backref('alternatives', cascade='all, delete-orphan')) - def __str__(self): - return str(self.name) - class Relay(Base): """ Relayed mail domain. @@ -302,9 +317,6 @@ class Relay(Base): # TODO: String(80) is too small? smtp = db.Column(db.String(80), nullable=True) - def __str__(self): - return str(self.name) - class Email(object): """ Abstraction for an email address (localpart and domain). @@ -312,11 +324,11 @@ class Email(object): # TODO: validate max. total length of address (<=254) - # TODO: String(80) is too large (>64)? + # TODO: String(80) is too large (64)? localpart = db.Column(db.String(80), nullable=False) @declarative.declared_attr - def domain_name(self): + def domain_name(cls): """ the domain part of the email address """ return db.Column(IdnaDomain, db.ForeignKey(Domain.name), nullable=False, default=IdnaDomain) @@ -325,13 +337,33 @@ class Email(object): # It is however very useful for quick lookups without joining tables, # especially when the mail server is reading the database. @declarative.declared_attr - def email(self): + def _email(cls): """ the complete email address (localpart@domain) """ - updater = lambda ctx: '{localpart}@{domain_name}'.format(**ctx.current_parameters) - return db.Column(IdnaEmail, - primary_key=True, nullable=False, - default=updater - ) + + def updater(ctx): + key = f'{cls.__tablename__}_email' + if key in ctx.current_parameters: + return ctx.current_parameters[key] + return '{localpart}@{domain_name}'.format(**ctx.current_parameters) + + return db.Column('email', IdnaEmail, primary_key=True, nullable=False, onupdate=updater) + + # We need to keep email, localpart and domain_name in sync. + # But IMHO using email as primary key was not a good idea in the first place. + @hybrid_property + def email(self): + """ getter for email - gets _email """ + return self._email + + @email.setter + def email(self, value): + """ setter for email - sets _email, localpart and domain_name at once """ + self.localpart, self.domain_name = value.rsplit('@', 1) + self._email = value + + # hack for email declared attr - when _email is not updated yet + def __str__(self): + return str(f'{self.localpart}@{self.domain_name}') def sendmail(self, subject, body): """ send an email to the address """ @@ -391,9 +423,6 @@ class Email(object): return None - def __str__(self): - return str(self.email) - class User(Base, Email): """ A user is an email address that has a password to access a mailbox. @@ -435,12 +464,10 @@ class User(Base, Email): is_active = True is_anonymous = False - # TODO: remove unused user.get_id() def get_id(self): """ return users email address """ return self.email - # TODO: remove unused user.destination @property def destination(self): """ returns comma separated string of destinations """ @@ -471,17 +498,20 @@ class User(Base, Email): 'CRYPT': 'des_crypt', } - def _get_password_context(self): + @classmethod + def get_password_context(cls): + """ Create password context for hashing and verification + """ return passlib.context.CryptContext( - schemes=self.scheme_dict.values(), - default=self.scheme_dict[app.config['PASSWORD_SCHEME']], + schemes=cls.scheme_dict.values(), + default=cls.scheme_dict[app.config['PASSWORD_SCHEME']], ) def check_password(self, plain): """ Check password against stored hash Update hash when default scheme has changed """ - context = self._get_password_context() + context = self.get_password_context() hashed = re.match('^({[^}]+})?(.*)$', self.password).group(2) result = context.verify(plain, hashed) if result and context.identify(hashed) != context.default_scheme(): @@ -490,8 +520,6 @@ class User(Base, Email): db.session.commit() return result - # TODO: remove kwarg hash_scheme - there is no point in setting a scheme, - # when the next check updates the password to the default scheme. def set_password(self, new, hash_scheme=None, raw=False): """ Set password for user with specified encryption scheme @new: plain text password to encrypt (or, if raw is True: the hash itself) @@ -500,7 +528,7 @@ class User(Base, Email): if hash_scheme is None: hash_scheme = app.config['PASSWORD_SCHEME'] if not raw: - new = self._get_password_context().encrypt(new, self.scheme_dict[hash_scheme]) + new = self.get_password_context().encrypt(new, self.scheme_dict[hash_scheme]) self.password = f'{{{hash_scheme}}}{new}' def get_managed_domains(self): @@ -593,7 +621,7 @@ class Alias(Base, Email): return None -# TODO: what about API tokens? + class Token(Base): """ A token is an application password for a given user. """ @@ -606,20 +634,19 @@ class Token(Base): user = db.relationship(User, backref=db.backref('tokens', cascade='all, delete-orphan')) password = db.Column(db.String(255), nullable=False) - # TODO: String(80) is too large? + # TODO: String(255) is too large? (43 should be sufficient) ip = db.Column(db.String(255)) def check_password(self, password): """ verifies password against stored hash """ return passlib.hash.sha256_crypt.verify(password, self.password) - # TODO: use crypt context and default scheme from config? def set_password(self, password): """ sets password using sha256_crypt(rounds=1000) """ self.password = passlib.hash.sha256_crypt.using(rounds=1000).hash(password) - def __str__(self): - return str(self.comment or self.ip) + def __repr__(self): + return f'' class Fetch(Base): @@ -644,8 +671,11 @@ class Fetch(Base): last_check = db.Column(db.DateTime, nullable=True) error = db.Column(db.String(1023), nullable=True) - def __str__(self): - return f'{self.protocol}{"s" if self.tls else ""}://{self.username}@{self.host}:{self.port}' + def __repr__(self): + return ( + f'' + ) class MailuConfig: @@ -661,7 +691,7 @@ class MailuConfig: def __init__(self, model : db.Model): self.model = model - def __str__(self): + def __repr__(self): return f'<{self.model.__name__}-Collection>' @cached_property @@ -837,8 +867,8 @@ class MailuConfig: if models is None or model in models: db.session.query(model).delete() - domains = MailuCollection(Domain) - relays = MailuCollection(Relay) - users = MailuCollection(User) - aliases = MailuCollection(Alias) + domain = MailuCollection(Domain) + user = MailuCollection(User) + alias = MailuCollection(Alias) + relay = MailuCollection(Relay) config = MailuCollection(Config) diff --git a/core/admin/mailu/schemas.py b/core/admin/mailu/schemas.py index 04512f6d..54a2e928 100644 --- a/core/admin/mailu/schemas.py +++ b/core/admin/mailu/schemas.py @@ -1,27 +1,66 @@ """ Mailu marshmallow fields and schema """ -import re - +from copy import deepcopy from collections import OrderedDict from textwrap import wrap +import re +import json import yaml +import sqlalchemy + from marshmallow import pre_load, post_load, post_dump, fields, Schema +from marshmallow.utils import ensure_text_type from marshmallow.exceptions import ValidationError from marshmallow_sqlalchemy import SQLAlchemyAutoSchemaOpts + from flask_marshmallow import Marshmallow + from OpenSSL import crypto +try: + from pygments import highlight + from pygments.token import Token + from pygments.lexers import get_lexer_by_name + from pygments.lexers.data import YamlLexer + from pygments.formatters import get_formatter_by_name +except ModuleNotFoundError: + COLOR_SUPPORTED = False +else: + COLOR_SUPPORTED = True + from . import models, dkim ma = Marshmallow() -# TODO: how and where to mark keys as "required" while unserializing (on commandline, in api)? -# - fields without default => required -# - fields which are the primary key => unchangeable when updating +# TODO: how and where to mark keys as "required" while unserializing in api? +# - when modifying, nothing is required (only the primary key, but this key is in the uri) +# - the primary key from post data must not differ from the key in the uri +# - when creating all fields without default or auto-increment are required +# TODO: what about deleting list items and prung lists? +# - domain.alternatives, user.forward_destination, user.manager_of, aliases.destination +# TODO: validate everything! + + +### class for hidden values ### + +class _Hidden: + def __bool__(self): + return False + def __copy__(self): + return self + def __deepcopy__(self, _): + return self + def __eq__(self, other): + return str(other) == '' + def __repr__(self): + return '' + __str__ = __repr__ + +HIDDEN = _Hidden() ### map model to schema ### @@ -41,13 +80,90 @@ def mapped(cls): return cls -### yaml render module ### +### helper functions ### + +def get_fieldspec(exc): + """ walk traceback to extract spec of invalid field from marshmallow """ + path = [] + tbck = exc.__traceback__ + while tbck: + if tbck.tb_frame.f_code.co_name == '_serialize': + if 'attr' in tbck.tb_frame.f_locals: + path.append(tbck.tb_frame.f_locals['attr']) + elif tbck.tb_frame.f_code.co_name == '_init_fields': + path = '.'.join(path) + spec = ', '.join([f'{path}.{key}' for key in tbck.tb_frame.f_locals['invalid_fields']]) + return spec + tbck = tbck.tb_next + return None + +def colorize(data, lexer='yaml', formatter='terminal', color=None, strip=False): + """ add ANSI color to data """ + if color is None: + # autodetect colorize + color = COLOR_SUPPORTED + if not color: + # no color wanted + return data + if not COLOR_SUPPORTED: + # want color, but not supported + raise ValueError('Please install pygments to colorize output') + + scheme = { + Token: ('', ''), + Token.Name.Tag: ('cyan', 'brightcyan'), + Token.Literal.Scalar: ('green', 'green'), + Token.Literal.String: ('green', 'green'), + Token.Keyword.Constant: ('magenta', 'brightmagenta'), + Token.Literal.Number: ('magenta', 'brightmagenta'), + Token.Error: ('red', 'brightred'), + Token.Name: ('red', 'brightred'), + Token.Operator: ('red', 'brightred'), + } + + class MyYamlLexer(YamlLexer): + """ colorize yaml constants and integers """ + def get_tokens(self, text, unfiltered=False): + for typ, value in super().get_tokens(text, unfiltered): + if typ is Token.Literal.Scalar.Plain: + if value in {'true', 'false', 'null'}: + typ = Token.Keyword.Constant + elif value == HIDDEN: + typ = Token.Error + else: + try: + int(value, 10) + except ValueError: + try: + float(value) + except ValueError: + pass + else: + typ = Token.Literal.Number.Float + else: + typ = Token.Literal.Number.Integer + yield typ, value + + res = highlight( + data, + MyYamlLexer() if lexer == 'yaml' else get_lexer_by_name(lexer), + get_formatter_by_name(formatter, colorscheme=scheme) + ) + + return res.rstrip('\n') if strip else res + + +### render modules ### # allow yaml module to dump OrderedDict yaml.add_representer( OrderedDict, lambda cls, data: cls.represent_mapping('tag:yaml.org,2002:map', data.items()) ) +yaml.add_representer( + _Hidden, + lambda cls, data: cls.represent_data(str(data)) +) class RenderYAML: """ Marshmallow YAML Render Module @@ -67,19 +183,19 @@ class RenderYAML: return super().increase_indent(flow, False) @staticmethod - def _update_items(dict1, dict2): - """ sets missing keys in dict1 to values of dict2 + def _augment(kwargs, defaults): + """ add default kv's to kwargs if missing """ - for key, value in dict2.items(): - if key not in dict1: - dict1[key] = value + for key, value in defaults.items(): + if key not in kwargs: + kwargs[key] = value _load_defaults = {} @classmethod def loads(cls, *args, **kwargs): """ load yaml data from string """ - cls._update_items(kwargs, cls._load_defaults) + cls._augment(kwargs, cls._load_defaults) return yaml.safe_load(*args, **kwargs) _dump_defaults = { @@ -90,13 +206,52 @@ class RenderYAML: } @classmethod def dumps(cls, *args, **kwargs): - """ dump yaml data to string + """ dump data to yaml string """ - cls._update_items(kwargs, cls._dump_defaults) + cls._augment(kwargs, cls._dump_defaults) return yaml.dump(*args, **kwargs) +class JSONEncoder(json.JSONEncoder): + """ JSONEncoder supporting serialization of HIDDEN """ + def default(self, o): + """ serialize HIDDEN """ + if isinstance(o, _Hidden): + return str(o) + return json.JSONEncoder.default(self, o) -### field definitions ### +class RenderJSON: + """ Marshmallow JSON Render Module + """ + + @staticmethod + def _augment(kwargs, defaults): + """ add default kv's to kwargs if missing + """ + for key, value in defaults.items(): + if key not in kwargs: + kwargs[key] = value + + _load_defaults = {} + @classmethod + def loads(cls, *args, **kwargs): + """ load json data from string + """ + cls._augment(kwargs, cls._load_defaults) + return json.loads(*args, **kwargs) + + _dump_defaults = { + 'separators': (',',':'), + 'cls': JSONEncoder, + } + @classmethod + def dumps(cls, *args, **kwargs): + """ dump data to json string + """ + cls._augment(kwargs, cls._dump_defaults) + return json.dumps(*args, **kwargs) + + +### custom fields ### class LazyStringField(fields.String): """ Field that serializes a "false" value to the empty string @@ -107,9 +262,8 @@ class LazyStringField(fields.String): """ return value if value else '' - class CommaSeparatedListField(fields.Raw): - """ Field that deserializes a string containing comma-separated values to + """ Deserialize a string containing comma-separated values to a list of strings """ @@ -129,10 +283,15 @@ class CommaSeparatedListField(fields.Raw): class DkimKeyField(fields.String): - """ Field that serializes a dkim key to a list of strings (lines) and - deserializes a string or list of strings. + """ Serialize a dkim key to a list of strings (lines) and + Deserialize a string or list of strings to a valid dkim key """ + default_error_messages = { + "invalid": "Not a valid string or list.", + "invalid_utf8": "Not a valid utf-8 string or list.", + } + _clean_re = re.compile( r'(^-----BEGIN (RSA )?PRIVATE KEY-----|-----END (RSA )?PRIVATE KEY-----$|\s+)', flags=re.UNICODE @@ -156,11 +315,19 @@ class DkimKeyField(fields.String): # convert list to str if isinstance(value, list): - value = ''.join(value) + try: + value = ''.join([ensure_text_type(item) for item in value]) + except UnicodeDecodeError as exc: + raise self.make_error("invalid_utf8") from exc - # only strings are allowed - if not isinstance(value, str): - raise ValidationError(f'invalid type {type(value).__name__!r}') + # only text is allowed + else: + if not isinstance(value, (str, bytes)): + raise self.make_error("invalid") + try: + value = ensure_text_type(value) + except UnicodeDecodeError as exc: + raise self.make_error("invalid_utf8") from exc # clean value (remove whitespace and header/footer) value = self._clean_re.sub('', value.strip()) @@ -189,28 +356,53 @@ class DkimKeyField(fields.String): else: return value - -### base definitions ### - -def handle_email(data): - """ merge separate localpart and domain to email +class PasswordField(fields.Str): + """ Serialize a hashed password hash by stripping the obsolete {SCHEME} + Deserialize a plain password or hashed password into a hashed password """ - localpart = 'localpart' in data - domain = 'domain' in data + _hashes = {'PBKDF2', 'BLF-CRYPT', 'SHA512-CRYPT', 'SHA256-CRYPT', 'MD5-CRYPT', 'CRYPT'} - if 'email' in data: - if localpart or domain: - raise ValidationError('duplicate email and localpart/domain') - data['localpart'], data['domain_name'] = data['email'].rsplit('@', 1) - elif localpart and domain: - data['domain_name'] = data['domain'] - del data['domain'] - data['email'] = f'{data["localpart"]}@{data["domain_name"]}' - elif localpart or domain: - raise ValidationError('incomplete localpart/domain') + def _serialize(self, value, attr, obj, **kwargs): + """ strip obsolete {password-hash} when serializing """ + # strip scheme spec if in database - it's obsolete + if value.startswith('{') and (end := value.find('}', 1)) >= 0: + if value[1:end] in self._hashes: + return value[end+1:] + return value - return data + def _deserialize(self, value, attr, data, **kwargs): + """ hashes plain password or checks hashed password + also strips obsolete {password-hash} when deserializing + """ + + # when hashing is requested: use model instance to hash plain password + if data.get('hash_password'): + # hash password using model instance + inst = self.metadata['model']() + inst.set_password(value) + value = inst.password + del inst + + # strip scheme spec when specified - it's obsolete + if value.startswith('{') and (end := value.find('}', 1)) >= 0: + if value[1:end] in self._hashes: + value = value[end+1:] + + # check if algorithm is supported + inst = self.metadata['model'](password=value) + try: + # just check against empty string to see if hash is valid + inst.check_password('') + except ValueError as exc: + # ValueError: hash could not be identified + raise ValidationError(f'invalid password hash {value!r}') from exc + del inst + + return value + + +### base schema ### class BaseOpts(SQLAlchemyAutoSchemaOpts): """ Option class with sqla session @@ -220,6 +412,8 @@ class BaseOpts(SQLAlchemyAutoSchemaOpts): meta.sqla_session = models.db.session if not hasattr(meta, 'ordered'): meta.ordered = True + if not hasattr(meta, 'sibling'): + meta.sibling = False super(BaseOpts, self).__init__(meta, ordered=ordered) class BaseSchema(ma.SQLAlchemyAutoSchema): @@ -231,10 +425,15 @@ class BaseSchema(ma.SQLAlchemyAutoSchema): class Meta: """ Schema config """ + include_by_context = {} + exclude_by_value = {} + hide_by_context = {} + order = [] + sibling = False def __init__(self, *args, **kwargs): - # context? + # get context context = kwargs.get('context', {}) flags = {key for key, value in context.items() if value is True} @@ -261,7 +460,7 @@ class BaseSchema(ma.SQLAlchemyAutoSchema): # exclude default values if not context.get('full'): - for column in getattr(self.opts, 'model').__table__.columns: + for column in self.opts.model.__table__.columns: if column.name not in exclude: self._exclude_by_value.setdefault(column.name, []).append( None if column.default is None else column.default.arg @@ -274,45 +473,239 @@ class BaseSchema(ma.SQLAlchemyAutoSchema): if not flags & set(need): self._hide_by_context |= set(what) + # remember primary keys + self._primary = self.opts.model.__table__.primary_key.columns.values()[0].name + # initialize attribute order if hasattr(self.Meta, 'order'): # use user-defined order - self._order = list(reversed(getattr(self.Meta, 'order'))) + self._order = list(reversed(self.Meta.order)) else: # default order is: primary_key + other keys alphabetically self._order = list(sorted(self.fields.keys())) - primary = self.opts.model.__table__.primary_key.columns.values()[0].name - if primary in self._order: - self._order.remove(primary) + if self._primary in self._order: + self._order.remove(self._primary) self._order.reverse() - self._order.append(primary) + self._order.append(self._primary) # move pre_load hook "_track_import" to the front hooks = self._hooks[('pre_load', False)] - if '_track_import' in hooks: - hooks.remove('_track_import') - hooks.insert(0, '_track_import') - # and post_load hook "_fooo" to the end + hooks.remove('_track_import') + hooks.insert(0, '_track_import') + # move pre_load hook "_add_instance" to the end + hooks.remove('_add_required') + hooks.append('_add_required') + + # move post_load hook "_add_instance" to the end hooks = self._hooks[('post_load', False)] - if '_add_instance' in hooks: - hooks.remove('_add_instance') - hooks.append('_add_instance') + hooks.remove('_add_instance') + hooks.append('_add_instance') + + def hide(self, data): + """ helper method to hide input data for logging """ + # always returns a copy of data + return { + key: HIDDEN if key in self._hide_by_context else deepcopy(value) + for key, value in data.items() + } + + def _call_and_store(self, *args, **kwargs): + """ track curent parent field for pruning """ + self.context['parent_field'] = kwargs['field_name'] + return super()._call_and_store(*args, **kwargs) + + # this is only needed to work around the declared attr "email" primary key in model + def get_instance(self, data): + """ lookup item by defined primary key instead of key(s) from model """ + if self.transient: + return None + if keys := getattr(self.Meta, 'primary_keys', None): + filters = {key: data.get(key) for key in keys} + if None not in filters.values(): + return self.session.query(self.opts.model).filter_by(**filters).first() + return super().get_instance(data) + + @pre_load(pass_many=True) + def _patch_input(self, items, many, **kwargs): # pylint: disable=unused-argument + """ - flush sqla session before serializing a section when requested + (make sure all objects that could be referred to later are created) + - when in update mode: patch input data before deserialization + - handle "prune" and "delete" items + - replace values in keys starting with '-' with default + """ + + # flush sqla session + if not self.Meta.sibling: + self.opts.sqla_session.flush() + + # stop early when not updating + if not self.context.get('update'): + return items + + # patch "delete", "prune" and "default" + want_prune = [] + def patch(count, data, prune): + + # don't allow __delete__ coming from input + if '__delete__' in data: + raise ValidationError('Unknown field.', f'{count}.__delete__') + + # handle "prune list" and "delete item" (-pkey: none and -pkey: id) + for key in data: + if key.startswith('-'): + if key[1:] == self._primary: + # delete or prune + if data[key] is None: + # prune + prune.append(True) + return None + # mark item for deletion + return {key[1:]: data[key], '__delete__': True} + + # handle "set to default value" (-key: none) + def set_default(key, value): + if not key.startswith('-'): + return (key, value) + key = key[1:] + if not key in self.opts.model.__table__.columns: + return (key, None) + if value is not None: + raise ValidationError( + 'When resetting to default value must be null.', + f'{count}.{key}' + ) + value = self.opts.model.__table__.columns[key].default + if value is None: + raise ValidationError( + 'Field has no default value.', + f'{count}.{key}' + ) + return (key, value.arg) + + return dict([set_default(key, value) for key, value in data.items()]) + + # convert items to "delete" and filter "prune" item + items = [ + item for item in [ + patch(count, item, want_prune) for count, item in enumerate(items) + ] if item + ] + + # prune: determine if existing items in db need to be added or marked for deletion + add_items = False + del_items = False + if self.Meta.sibling: + # parent prunes automatically + if not want_prune: + # no prune requested => add old items + add_items = True + else: + # parent does not prune automatically + if want_prune: + # prune requested => mark old items for deletion + del_items = True + + if add_items or del_items: + existing = {item[self._primary] for item in items if self._primary in item} + for item in getattr(self.context['parent'], self.context['parent_field']): + key = getattr(item, self._primary) + if key not in existing: + if add_items: + items.append({self._primary: key}) + else: + items.append({self._primary: key, '__delete__': True}) + + return items @pre_load def _track_import(self, data, many, **kwargs): # pylint: disable=unused-argument -# TODO: also handle reset, prune and delete in pre_load / post_load hooks! -# print('!!!', repr(data)) + """ call callback function to track import + """ + # callback if callback := self.context.get('callback'): callback(self, data) + return data - @post_load - def _add_instance(self, item, many, **kwargs): # pylint: disable=unused-argument - self.opts.sqla_session.add(item) + @pre_load + def _add_required(self, data, many, **kwargs): # pylint: disable=unused-argument + """ when updating: + allow modification of existing items having required attributes + by loading existing value from db + """ + + if not self.opts.load_instance or not self.context.get('update'): + return data + + # stabilize import of auto-increment primary keys (not required), + # by matching import data to existing items and setting primary key + if not self._primary in data: + for item in getattr(self.context['parent'], self.context['parent_field']): + existing = self.dump(item, many=False) + this = existing.pop(self._primary) + if data == existing: + instance = item + data[self._primary] = this + break + + # try to load instance + instance = self.instance or self.get_instance(data) + if instance is None: + + if '__delete__' in data: + # deletion of non-existent item requested + raise ValidationError( + f'item not found: {data[self._primary]!r}', + field_name=f'?.{self._primary}', + ) + + else: + + if self.context.get('update'): + # remember instance as parent for pruning siblings + if not self.Meta.sibling: + self.context['parent'] = instance + # delete instance when marked + if '__delete__' in data: + self.opts.sqla_session.delete(instance) + + # add attributes required for validation from db + # TODO: this will cause validation errors if value from database does not validate + for attr_name, field_obj in self.load_fields.items(): + if field_obj.required and attr_name not in data: + data[attr_name] = getattr(instance, attr_name) + + return data + + @post_load(pass_original=True) + def _add_instance(self, item, original, many, **kwargs): # pylint: disable=unused-argument + """ add new instances to sqla session """ + + if item in self.opts.sqla_session: + # item was modified + if 'hash_password' in original: + # stabilize import of passwords to be hashed, + # by not re-hashing an unchanged password + if attr := getattr(sqlalchemy.inspect(item).attrs, 'password', None): + if attr.history.has_changes() and attr.history.deleted: + try: + # reset password hash, if password was not changed + inst = type(item)(password=attr.history.deleted[-1]) + if inst.check_password(original['password']): + item.password = inst.password + except ValueError: + # hash in db is invalid + pass + else: + del inst + else: + # new item + self.opts.sqla_session.add(item) return item @post_dump def _hide_and_order(self, data, many, **kwargs): # pylint: disable=unused-argument + """ hide secrets and order output """ # order output for key in self._order: @@ -325,15 +718,18 @@ class BaseSchema(ma.SQLAlchemyAutoSchema): if not self._exclude_by_value and not self._hide_by_context: return data - # exclude items or hide values + # exclude or hide values full = self.context.get('full') return type(data)([ - (key, '' if key in self._hide_by_context else value) + (key, HIDDEN if key in self._hide_by_context else value) for key, value in data.items() if full or key not in self._exclude_by_value or value not in self._exclude_by_value[key] ]) - # TODO: remove LazyStringField and change model (IMHO comment should not be nullable) + # this field is used to mark items for deletion + mark_delete = fields.Boolean(data_key='__delete__', load_only=True) + + # TODO: remove LazyStringField (when model was changed - IMHO comment should not be nullable) comment = LazyStringField() @@ -381,6 +777,11 @@ class TokenSchema(BaseSchema): model = models.Token load_instance = True + sibling = True + + password = PasswordField(required=True, metadata={'model': models.User}) + hash_password = fields.Boolean(load_only=True, missing=False) + @mapped class FetchSchema(BaseSchema): @@ -389,6 +790,8 @@ class FetchSchema(BaseSchema): """ Schema config """ model = models.Fetch load_instance = True + + sibling = True include_by_context = { ('full', 'import'): {'last_check', 'error'}, } @@ -405,52 +808,25 @@ class UserSchema(BaseSchema): model = models.User load_instance = True include_relationships = True - exclude = ['domain', 'quota_bytes_used'] + exclude = ['_email', 'domain', 'localpart', 'domain_name', 'quota_bytes_used'] + primary_keys = ['email'] exclude_by_value = { 'forward_destination': [[]], - 'tokens': [[]], - 'fetches': [[]], - 'manager_of': [[]], - 'reply_enddate': ['2999-12-31'], - 'reply_startdate': ['1900-01-01'], + 'tokens': [[]], + 'fetches': [[]], + 'manager_of': [[]], + 'reply_enddate': ['2999-12-31'], + 'reply_startdate': ['1900-01-01'], } - @pre_load - def _handle_email_and_password(self, data, many, **kwargs): # pylint: disable=unused-argument - data = handle_email(data) - if 'password' in data: - if 'password_hash' in data or 'hash_scheme' in data: - raise ValidationError('ambigous key password and password_hash/hash_scheme') - # check (hashed) password - password = data['password'] - if password.startswith('{') and '}' in password: - scheme = password[1:password.index('}')] - if scheme not in self.Meta.model.scheme_dict: - raise ValidationError(f'invalid password scheme {scheme!r}') - else: - raise ValidationError(f'invalid hashed password {password!r}') - elif 'password_hash' in data and 'hash_scheme' in data: - if data['hash_scheme'] not in self.Meta.model.scheme_dict: - raise ValidationError(f'invalid password scheme {data["hash_scheme"]!r}') - data['password'] = f'{{{data["hash_scheme"]}}}{data["password_hash"]}' - del data['hash_scheme'] - del data['password_hash'] - return data - - # TODO: verify password (should this be done in model?) - # scheme, hashed = re.match('^(?:{([^}]+)})?(.*)$', self.password).groups() - # if not scheme... - # ctx = passlib.context.CryptContext(schemes=[scheme], default=scheme) - # try: - # ctx.verify('', hashed) - # =>? ValueError: hash could not be identified - - localpart = fields.Str(load_only=True) - domain_name = fields.Str(load_only=True) + email = fields.String(required=True) tokens = fields.Nested(TokenSchema, many=True) fetches = fields.Nested(FetchSchema, many=True) + password = PasswordField(required=True, metadata={'model': models.User}) + hash_password = fields.Boolean(load_only=True, missing=False) + @mapped class AliasSchema(BaseSchema): @@ -459,18 +835,14 @@ class AliasSchema(BaseSchema): """ Schema config """ model = models.Alias load_instance = True - exclude = ['domain'] + exclude = ['_email', 'domain', 'localpart', 'domain_name'] + primary_keys = ['email'] exclude_by_value = { 'destination': [[]], } - @pre_load - def _handle_email(self, data, many, **kwargs): # pylint: disable=unused-argument - return handle_email(data) - - localpart = fields.Str(load_only=True) - domain_name = fields.Str(load_only=True) + email = fields.String(required=True) destination = CommaSeparatedListField() @@ -499,7 +871,7 @@ class MailuSchema(Schema): render_module = RenderYAML ordered = True - order = ['config', 'domains', 'users', 'aliases', 'relays'] + order = ['domain', 'user', 'alias', 'relay'] # 'config' def __init__(self, *args, **kwargs): super().__init__(*args, **kwargs) @@ -511,6 +883,14 @@ class MailuSchema(Schema): except KeyError: pass + def _call_and_store(self, *args, **kwargs): + """ track current parent and field for pruning """ + self.context.update({ + 'parent': self.context.get('config'), + 'parent_field': kwargs['field_name'], + }) + return super()._call_and_store(*args, **kwargs) + @pre_load def _clear_config(self, data, many, **kwargs): # pylint: disable=unused-argument """ create config object in context if missing @@ -534,8 +914,8 @@ class MailuSchema(Schema): return config - config = fields.Nested(ConfigSchema, many=True) - domains = fields.Nested(DomainSchema, many=True) - users = fields.Nested(UserSchema, many=True) - aliases = fields.Nested(AliasSchema, many=True) - relays = fields.Nested(RelaySchema, many=True) + domain = fields.Nested(DomainSchema, many=True) + user = fields.Nested(UserSchema, many=True) + alias = fields.Nested(AliasSchema, many=True) + relay = fields.Nested(RelaySchema, many=True) +# config = fields.Nested(ConfigSchema, many=True) diff --git a/docs/cli.rst b/docs/cli.rst index 1b2ed14f..497cdfc5 100644 --- a/docs/cli.rst +++ b/docs/cli.rst @@ -10,8 +10,9 @@ Managing users and aliases can be done from CLI using commands: * user * user-import * user-delete -* config-dump * config-update +* config-export +* config-import alias ----- @@ -69,104 +70,160 @@ user-delete docker-compose exec admin flask mailu user-delete foo@example.net -config-dump ------------ - -The purpose of this command is to dump domain-, relay-, alias- and user-configuration to a YAML template. - -.. code-block:: bash - - # docker-compose exec admin flask mailu config-dump --help - - Usage: flask mailu config-dump [OPTIONS] [SECTIONS]... - - dump configuration as YAML-formatted data to stdout - - SECTIONS can be: domains, relays, users, aliases - - Options: - -f, --full Include default attributes - -s, --secrets Include secrets (dkim-key, plain-text / not hashed) - -d, --dns Include dns records - --help Show this message and exit. - -If you want to export secrets (dkim-keys, plain-text / not hashed) you have to add the ``--secrets`` option. -Only non-default attributes are dumped. If you want to dump all attributes use ``--full``. -To include dns records (mx, spf, dkim and dmarc) add the ``--dns`` option. -Unless you specify some sections all sections are dumped by default. - -.. code-block:: bash - - docker-compose exec admin flask mailu config-dump > mail-config.yml - - docker-compose exec admin flask mailu config-dump --dns domains - config-update ------------- -The purpose of this command is for importing domain-, relay-, alias- and user-configuration in bulk and synchronizing DB entries with an external YAML template. +The sole purpose of this command is for importing users/aliases in bulk and synchronizing DB entries with external YAML template: .. code-block:: bash - # docker-compose exec admin flask mailu config-update --help + cat mail-config.yml | docker-compose exec -T admin flask mailu config-update --delete-objects - Usage: flask mailu config-update [OPTIONS] +where mail-config.yml looks like: - sync configuration with data from YAML-formatted stdin +.. code-block:: bash - Options: - -v, --verbose Increase verbosity - -d, --delete-objects Remove objects not included in yaml - -n, --dry-run Perform a trial run with no changes made - --help Show this message and exit. + users: + - localpart: foo + domain: example.com + password_hash: klkjhumnzxcjkajahsdqweqqwr + hash_scheme: MD5-CRYPT + aliases: + - localpart: alias1 + domain: example.com + destination: "user1@example.com,user2@example.com" + +without ``--delete-object`` option config-update will only add/update new values but will *not* remove any entries missing in provided YAML input. + +Users +----- + +following are additional parameters that could be defined for users: + +* comment +* quota_bytes +* global_admin +* enable_imap +* enable_pop +* forward_enabled +* forward_destination +* reply_enabled +* reply_subject +* reply_body +* displayed_name +* spam_enabled +* spam_threshold + +Alias +----- + +additional fields: + +* wildcard + +config-export +------------- + +The purpose of this command is to export domain-, relay-, alias- and user-configuration in YAML or JSON format. + +.. code-block:: bash + + # docker-compose exec admin flask mailu config-export --help + + Usage: flask mailu config-export [OPTIONS] [FILTER]... + + Export configuration as YAML or JSON to stdout or file + + Options: + -f, --full Include attributes with default value. + -s, --secrets Include secret attributes (dkim-key, passwords). + -c, --color Force colorized output. + -d, --dns Include dns records. + -o, --output-file FILENAME Save configuration to file. + -j, --json Export configuration in json format. + -?, -h, --help Show this message and exit. + +Only non-default attributes are exported. If you want to export all attributes use ``--full``. +If you want to export plain-text secrets (dkim-keys, passwords) you have to add the ``--secrets`` option. +To include dns records (mx, spf, dkim and dmarc) add the ``--dns`` option. +By default all configuration objects are exported (domain, user, alias, relay). You can specify +filters to export only some objects or attributes (try: ``user`` or ``domain.name``). + +.. code-block:: bash + + docker-compose exec admin flask mailu config-export -o mail-config.yml + + docker-compose exec admin flask mailu config-export --dns domain.dns_mx domain.dns_spf + +config-import +------------- + +The purpose of this command is for importing domain-, relay-, alias- and user-configuration in bulk and synchronizing DB entries with an external YAML/JOSN source. + +.. code-block:: bash + + # docker-compose exec admin flask mailu config-import --help + + Usage: flask mailu config-import [OPTIONS] [FILENAME|-] + + Import configuration as YAML or JSON from stdin or file + + Options: + -v, --verbose Increase verbosity. + -s, --secrets Show secret attributes in messages. + -q, --quiet Quiet mode - only show errors. + -c, --color Force colorized output. + -u, --update Update mode - merge input with existing config. + -n, --dry-run Perform a trial run with no changes made. + -?, -h, --help Show this message and exit. The current version of docker-compose exec does not pass stdin correctly, so you have to user docker exec instead: .. code-block:: bash - docker exec -i $(docker-compose ps -q admin) flask mailu config-update -nvd < mail-config.yml + docker exec -i $(docker-compose ps -q admin) flask mailu config-import -nv < mail-config.yml - -mail-config.yml looks like this: +mail-config.yml contains the configuration and looks like this: .. code-block:: yaml - - domains: + + domain: - name: example.com alternatives: - alternative.example.com - users: + user: - email: foo@example.com - password_hash: klkjhumnzxcjkajahsdqweqqwr + password_hash: '$2b$12$...' hash_scheme: MD5-CRYPT - aliases: + alias: - email: alias1@example.com - destination: "user1@example.com,user2@example.com" + destination: + - user1@example.com + - user2@example.com - relays: + relay: - name: relay.example.com comment: test smtp: mx.example.com -You can use ``--dry-run`` to test your YAML without comitting any changes to the database. -With ``--verbose`` config-update will show exactly what it changes in the database. -Without ``--delete-object`` option config-update will only add/update changed values but will *not* remove any entries missing in provided YAML input. +config-update shows the number of created/modified/deleted objects after import. +To suppress all messages except error messages use ``--quiet``. +By adding the ``--verbose`` switch (one or more times) the import gets more detailed and shows exactyl what attributes changed. +In all messages plain-text secrets (dkim-keys, passwords) are hidden by default. Use ``--secrets`` to show secrets. +If you want to test what would be done when importing use ``--dry-run``. +By default config-update replaces the whole configuration. You can use ``--update`` to change the existing configuration instead. +When updating you can add new and change existing objects. +To delete an object use ``-key: value`` (To delete the domain example.com ``-name: example.com`` for example). +To reset an attribute to default use ``-key: null`` (To reset enable_imap ``-enable_imap: null`` for example). -This is a complete YAML template with all additional parameters that could be defined: +This is a complete YAML template with all additional parameters that can be defined: .. code-block:: yaml - aliases: - - email: email@example.com - comment: '' - destination: - - address@example.com - wildcard: false - - domains: + domain: - name: example.com alternatives: - alternative.tld @@ -176,13 +233,8 @@ This is a complete YAML template with all additional parameters that could be de max_quota_bytes: 0 max_users: -1 signup_enabled: false - - relays: - - name: relay.example.com - comment: '' - smtp: mx.example.com - - users: + + user: - email: postmaster@example.com comment: '' displayed_name: 'Postmaster' @@ -192,13 +244,16 @@ This is a complete YAML template with all additional parameters that could be de fetches: - id: 1 comment: 'test fetch' - username: fetch-user + error: null host: other.example.com + keep: true + last_check: '2020-12-29T17:09:48.200179' password: 'secret' + hash_password: true port: 993 protocol: imap tls: true - keep: true + username: fetch-user forward_destination: - address@remote.example.com forward_enabled: true @@ -206,12 +261,13 @@ This is a complete YAML template with all additional parameters that could be de global_admin: true manager_of: - example.com - password: '{BLF-CRYPT}$2b$12$...' + password: '$2b$12$...' + hash_password: true quota_bytes: 1000000000 reply_body: '' reply_enabled: false - reply_enddate: 2999-12-31 - reply_startdate: 1900-01-01 + reply_enddate: '2999-12-31' + reply_startdate: '1900-01-01' reply_subject: '' spam_enabled: true spam_threshold: 80 @@ -219,5 +275,16 @@ This is a complete YAML template with all additional parameters that could be de - id: 1 comment: email-client ip: 192.168.1.1 - password: '$5$rounds=1000$...' + password: '$5$rounds=1$...' + aliases: + - email: email@example.com + comment: '' + destination: + - address@example.com + wildcard: false + + relay: + - name: relay.example.com + comment: '' + smtp: mx.example.com diff --git a/tests/compose/core/02_forward_test.sh b/tests/compose/core/02_forward_test.sh index 651e027c..595820cf 100755 --- a/tests/compose/core/02_forward_test.sh +++ b/tests/compose/core/02_forward_test.sh @@ -1,4 +1,4 @@ -cat << EOF | docker-compose -f tests/compose/core/docker-compose.yml exec -T admin flask mailu config-update --verbose +cat << EOF | docker-compose -f tests/compose/core/docker-compose.yml exec -T admin flask mailu config-update -v 1 users: - localpart: forwardinguser password_hash: "\$1\$F2OStvi1\$Q8hBIHkdJpJkJn/TrMIZ9/" @@ -10,7 +10,7 @@ EOF python3 tests/forward_test.py -cat << EOF | docker-compose -f tests/compose/core/docker-compose.yml exec -T admin flask mailu config-update --verbose +cat << EOF | docker-compose -f tests/compose/core/docker-compose.yml exec -T admin flask mailu config-update -v 1 users: - localpart: forwardinguser password_hash: "\$1\$F2OStvi1\$Q8hBIHkdJpJkJn/TrMIZ9/" diff --git a/tests/compose/core/03_alias_test.sh b/tests/compose/core/03_alias_test.sh index 2d40903a..dce1918a 100755 --- a/tests/compose/core/03_alias_test.sh +++ b/tests/compose/core/03_alias_test.sh @@ -1,4 +1,4 @@ -cat << EOF | docker-compose -f tests/compose/core/docker-compose.yml exec -T admin flask mailu config-update --verbose +cat << EOF | docker-compose -f tests/compose/core/docker-compose.yml exec -T admin flask mailu config-update -v 1 aliases: - localpart: alltheusers domain: mailu.io @@ -7,6 +7,6 @@ EOF python3 tests/alias_test.py -cat << EOF | docker-compose -f tests/compose/core/docker-compose.yml exec -T admin flask mailu config-update --verbose +cat << EOF | docker-compose -f tests/compose/core/docker-compose.yml exec -T admin flask mailu config-update -v 1 aliases: [] EOF diff --git a/tests/compose/core/04_reply_test.sh b/tests/compose/core/04_reply_test.sh index 7615a0f8..83c114f6 100755 --- a/tests/compose/core/04_reply_test.sh +++ b/tests/compose/core/04_reply_test.sh @@ -1,4 +1,4 @@ -cat << EOF | docker-compose -f tests/compose/core/docker-compose.yml exec -T admin flask mailu config-update --verbose +cat << EOF | docker-compose -f tests/compose/core/docker-compose.yml exec -T admin flask mailu config-update -v 1 users: - localpart: replyuser password_hash: "\$1\$F2OStvi1\$Q8hBIHkdJpJkJn/TrMIZ9/" @@ -11,7 +11,7 @@ EOF python3 tests/reply_test.py -cat << EOF | docker-compose -f tests/compose/core/docker-compose.yml exec -T admin flask mailu config-update --verbose +cat << EOF | docker-compose -f tests/compose/core/docker-compose.yml exec -T admin flask mailu config-update -v 1 users: - localpart: replyuser password_hash: "\$1\$F2OStvi1\$Q8hBIHkdJpJkJn/TrMIZ9/" diff --git a/towncrier/newsfragments/1604.feature b/towncrier/newsfragments/1604.feature index 06ee0beb..2b47791a 100644 --- a/towncrier/newsfragments/1604.feature +++ b/towncrier/newsfragments/1604.feature @@ -1 +1 @@ -Added cli command config-dump and enhanced config-update +Add cli commands config-import and config-export From 3937986e76f7079eb4488f12164f83f1966cb0ec Mon Sep 17 00:00:00 2001 From: Alexander Graf Date: Mon, 15 Feb 2021 10:01:35 +0100 Subject: [PATCH 081/596] Convert OrderedDict to dict for output --- core/admin/mailu/manage.py | 16 ++++++++++++---- 1 file changed, 12 insertions(+), 4 deletions(-) diff --git a/core/admin/mailu/manage.py b/core/admin/mailu/manage.py index a8d1d3cb..37d91f33 100644 --- a/core/admin/mailu/manage.py +++ b/core/admin/mailu/manage.py @@ -7,7 +7,7 @@ import socket import logging import uuid -from collections import Counter +from collections import Counter, OrderedDict from itertools import chain import click @@ -396,11 +396,19 @@ def config_import(verbose=0, secrets=False, quiet=False, color=False, update=Fal return chain(message, changes) def log(action, target, message=None): + + def od2d(val): + """ converts OrderedDicts to Dict for logging purposes """ + if isinstance(val, OrderedDict): + return {k: od2d(v) for k, v in val.items()} + elif isinstance(val, list): + return [od2d(v) for v in val] + else: + return val + if message is None: - # TODO: convert nested OrderedDict to dict - # see: flask mailu config-import -nvv yaml/dump4.yaml try: - message = dict(logger[target.__class__].dump(target)) + message = od2d(logger[target.__class__].dump(target)) except KeyError: message = target if not isinstance(message, str): From 8929912dea9c061632a3bde53726e0949ad455b9 Mon Sep 17 00:00:00 2001 From: Alexander Graf Date: Mon, 15 Feb 2021 21:56:58 +0100 Subject: [PATCH 082/596] remove OrderedDict - not necessary in python>=3.7 --- core/admin/mailu/manage.py | 24 ++++++------------ core/admin/mailu/schemas.py | 50 +++++++++++++------------------------ 2 files changed, 25 insertions(+), 49 deletions(-) diff --git a/core/admin/mailu/manage.py b/core/admin/mailu/manage.py index 37d91f33..a20c7d6d 100644 --- a/core/admin/mailu/manage.py +++ b/core/admin/mailu/manage.py @@ -7,7 +7,7 @@ import socket import logging import uuid -from collections import Counter, OrderedDict +from collections import Counter from itertools import chain import click @@ -397,18 +397,9 @@ def config_import(verbose=0, secrets=False, quiet=False, color=False, update=Fal def log(action, target, message=None): - def od2d(val): - """ converts OrderedDicts to Dict for logging purposes """ - if isinstance(val, OrderedDict): - return {k: od2d(v) for k, v in val.items()} - elif isinstance(val, list): - return [od2d(v) for v in val] - else: - return val - if message is None: try: - message = od2d(logger[target.__class__].dump(target)) + message = logger[target.__class__].dump(target) except KeyError: message = target if not isinstance(message, str): @@ -536,12 +527,11 @@ def config_import(verbose=0, secrets=False, quiet=False, color=False, update=Fal except Exception as exc: if verbose >= 5: raise - else: - # (yaml.scanner.ScannerError, UnicodeDecodeError, ...) - raise click.ClickException( - f'[{exc.__class__.__name__}] ' - f'{" ".join(str(exc).split())}' - ) from exc + # (yaml.scanner.ScannerError, UnicodeDecodeError, ...) + raise click.ClickException( + f'[{exc.__class__.__name__}] ' + f'{" ".join(str(exc).split())}' + ) from exc # flush session to show/count all changes if dry_run or verbose >= 1: diff --git a/core/admin/mailu/schemas.py b/core/admin/mailu/schemas.py index 54a2e928..8e91b4aa 100644 --- a/core/admin/mailu/schemas.py +++ b/core/admin/mailu/schemas.py @@ -2,7 +2,6 @@ """ from copy import deepcopy -from collections import OrderedDict from textwrap import wrap import re @@ -155,11 +154,7 @@ def colorize(data, lexer='yaml', formatter='terminal', color=None, strip=False): ### render modules ### -# allow yaml module to dump OrderedDict -yaml.add_representer( - OrderedDict, - lambda cls, data: cls.represent_mapping('tag:yaml.org,2002:map', data.items()) -) +# allow yaml to represent hidden attributes yaml.add_representer( _Hidden, lambda cls, data: cls.represent_data(str(data)) @@ -410,8 +405,6 @@ class BaseOpts(SQLAlchemyAutoSchemaOpts): def __init__(self, meta, ordered=False): if not hasattr(meta, 'sqla_session'): meta.sqla_session = models.db.session - if not hasattr(meta, 'ordered'): - meta.ordered = True if not hasattr(meta, 'sibling'): meta.sibling = False super(BaseOpts, self).__init__(meta, ordered=ordered) @@ -474,19 +467,23 @@ class BaseSchema(ma.SQLAlchemyAutoSchema): self._hide_by_context |= set(what) # remember primary keys - self._primary = self.opts.model.__table__.primary_key.columns.values()[0].name + self._primary = str(self.opts.model.__table__.primary_key.columns.values()[0].name) - # initialize attribute order + # determine attribute order if hasattr(self.Meta, 'order'): # use user-defined order - self._order = list(reversed(self.Meta.order)) + order = self.Meta.order else: # default order is: primary_key + other keys alphabetically - self._order = list(sorted(self.fields.keys())) - if self._primary in self._order: - self._order.remove(self._primary) - self._order.reverse() - self._order.append(self._primary) + order = list(sorted(self.fields.keys())) + if self._primary in order: + order.remove(self._primary) + order.insert(0, self._primary) + + # order dump_fields + for field in order: + if field in self.dump_fields: + self.dump_fields[field] = self.dump_fields.pop(field) # move pre_load hook "_track_import" to the front hooks = self._hooks[('pre_load', False)] @@ -704,16 +701,9 @@ class BaseSchema(ma.SQLAlchemyAutoSchema): return item @post_dump - def _hide_and_order(self, data, many, **kwargs): # pylint: disable=unused-argument + def _hide_values(self, data, many, **kwargs): # pylint: disable=unused-argument """ hide secrets and order output """ - # order output - for key in self._order: - try: - data.move_to_end(key, False) - except KeyError: - pass - # stop early when not excluding/hiding if not self._exclude_by_value and not self._hide_by_context: return data @@ -870,18 +860,14 @@ class MailuSchema(Schema): """ Schema config """ render_module = RenderYAML - ordered = True order = ['domain', 'user', 'alias', 'relay'] # 'config' def __init__(self, *args, **kwargs): super().__init__(*args, **kwargs) - # order fields - for field_list in self.load_fields, self.dump_fields, self.fields: - for section in reversed(self.Meta.order): - try: - field_list.move_to_end(section, False) - except KeyError: - pass + # order dump_fields + for field in self.Meta.order: + if field in self.dump_fields: + self.dump_fields[field] = self.dump_fields.pop(field) def _call_and_store(self, *args, **kwargs): """ track current parent and field for pruning """ From 70a1c79f81d4ceecba42e541947b14a2e9980bff Mon Sep 17 00:00:00 2001 From: Alexander Graf Date: Mon, 15 Feb 2021 22:57:37 +0100 Subject: [PATCH 083/596] handle prune and delete for lists and backrefs --- core/admin/mailu/manage.py | 13 ++++++++--- core/admin/mailu/schemas.py | 43 ++++++++++++++++++++++++++++++++++--- 2 files changed, 50 insertions(+), 6 deletions(-) diff --git a/core/admin/mailu/manage.py b/core/admin/mailu/manage.py index a20c7d6d..05eae010 100644 --- a/core/admin/mailu/manage.py +++ b/core/admin/mailu/manage.py @@ -478,9 +478,16 @@ def config_import(verbose=0, secrets=False, quiet=False, color=False, update=Fal if verbose >= 1: log('Modified', target, f'{str(target)!r} dkim_key: {before!r} -> {after!r}') - def track_serialize(obj, item): + def track_serialize(obj, item, backref=None): """ callback function to track import """ - # hide secrets + # called for backref modification? + if backref is not None: + log('Modified', item, '{target!r} {key}: {before!r} -> {after!r}'.format(**backref)) + return + # verbose? + if not verbose >= 2: + return + # hide secrets in data data = logger[obj.opts.model].hide(item) if 'hash_password' in data: data['password'] = HIDDEN @@ -501,7 +508,7 @@ def config_import(verbose=0, secrets=False, quiet=False, color=False, update=Fal 'import': True, 'update': update, 'clear': not update, - 'callback': track_serialize if verbose >= 2 else None, + 'callback': track_serialize, } # register listeners diff --git a/core/admin/mailu/schemas.py b/core/admin/mailu/schemas.py index 8e91b4aa..3e15ee26 100644 --- a/core/admin/mailu/schemas.py +++ b/core/admin/mailu/schemas.py @@ -14,6 +14,7 @@ from marshmallow import pre_load, post_load, post_dump, fields, Schema from marshmallow.utils import ensure_text_type from marshmallow.exceptions import ValidationError from marshmallow_sqlalchemy import SQLAlchemyAutoSchemaOpts +from marshmallow_sqlalchemy.fields import RelatedList from flask_marshmallow import Marshmallow @@ -39,8 +40,6 @@ ma = Marshmallow() # - when modifying, nothing is required (only the primary key, but this key is in the uri) # - the primary key from post data must not differ from the key in the uri # - when creating all fields without default or auto-increment are required -# TODO: what about deleting list items and prung lists? -# - domain.alternatives, user.forward_destination, user.manager_of, aliases.destination # TODO: validate everything! @@ -652,7 +651,7 @@ class BaseSchema(ma.SQLAlchemyAutoSchema): if '__delete__' in data: # deletion of non-existent item requested raise ValidationError( - f'item not found: {data[self._primary]!r}', + f'item to delete not found: {data[self._primary]!r}', field_name=f'?.{self._primary}', ) @@ -665,6 +664,44 @@ class BaseSchema(ma.SQLAlchemyAutoSchema): # delete instance when marked if '__delete__' in data: self.opts.sqla_session.delete(instance) + # delete item from lists or prune lists + # currently: domain.alternatives, user.forward_destination, + # user.manager_of, aliases.destination + for key, value in data.items(): + if isinstance(value, list): + new_value = set(value) + # handle list pruning + if '-prune-' in value: + value.remove('-prune-') + new_value.remove('-prune-') + else: + for old in getattr(instance, key): + # using str() is okay for now (see above) + new_value.add(str(old)) + # handle item deletion + for item in value: + if item.startswith('-'): + new_value.remove(item) + try: + new_value.remove(item[1:]) + except KeyError as exc: + raise ValidationError( + f'item to delete not found: {item[1:]!r}', + field_name=f'?.{key}', + ) from exc + # deduplicate and sort list + data[key] = sorted(new_value) + # log backref modification not catched by hook + if isinstance(self.fields[key], RelatedList): + if callback := self.context.get('callback'): + callback(self, instance, { + 'key': key, + 'target': str(instance), + 'before': [str(v) for v in getattr(instance, key)], + 'after': data[key], + }) + + # add attributes required for validation from db # TODO: this will cause validation errors if value from database does not validate From 1e2b5f26ab9fed0db8994ca54db7f57ce0792ce8 Mon Sep 17 00:00:00 2001 From: Alexander Graf Date: Tue, 16 Feb 2021 13:34:02 +0100 Subject: [PATCH 084/596] don't handle nested lists --- core/admin/mailu/schemas.py | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/core/admin/mailu/schemas.py b/core/admin/mailu/schemas.py index 3e15ee26..b9b8e393 100644 --- a/core/admin/mailu/schemas.py +++ b/core/admin/mailu/schemas.py @@ -668,7 +668,7 @@ class BaseSchema(ma.SQLAlchemyAutoSchema): # currently: domain.alternatives, user.forward_destination, # user.manager_of, aliases.destination for key, value in data.items(): - if isinstance(value, list): + if not isinstance(self.fields[key], fields.Nested) and isinstance(value, list): new_value = set(value) # handle list pruning if '-prune-' in value: From 10435114ec0206e3558734634d906cffbd49e783 Mon Sep 17 00:00:00 2001 From: Alexander Graf Date: Tue, 16 Feb 2021 15:36:01 +0100 Subject: [PATCH 085/596] updated remarks and docs --- core/admin/mailu/manage.py | 25 ++++++++++++--------- core/admin/mailu/models.py | 12 +++++----- core/admin/mailu/schemas.py | 3 ++- docs/cli.rst | 45 ++++++++++++++++++++++++------------- 4 files changed, 51 insertions(+), 34 deletions(-) diff --git a/core/admin/mailu/manage.py b/core/admin/mailu/manage.py index 05eae010..756400ad 100644 --- a/core/admin/mailu/manage.py +++ b/core/admin/mailu/manage.py @@ -337,16 +337,19 @@ def config_import(verbose=0, secrets=False, quiet=False, color=False, update=Fal """ # verbose - # 0 : show number of changes - # 1 : also show changes - # 2 : also show secrets - # 3 : also show input data - # 4 : also show sql queries - # 5 : also show tracebacks + # 0 : only show number of changes + # 1 : also show detailed changes + # 2 : also show input data + # 3 : also show sql queries (also needs -s, as sql may contain secrets) + # 4 : also show tracebacks (also needs -s, as tracebacks may contain secrets) if quiet: verbose = -1 + if verbose > 2 and not secrets: + print('[Warning] Verbosity level capped to 2. Specify --secrets to log sql and tracebacks.') + verbose = 2 + color_cfg = { 'color': color or sys.stdout.isatty(), 'lexer': 'python', @@ -376,7 +379,7 @@ def config_import(verbose=0, secrets=False, quiet=False, color=False, update=Fal fmt = f' - {{:<{max([len(loc) for loc, msg in res])}}} : {{}}' res = [fmt.format(loc, msg) for loc, msg in res] num = f'error{["s",""][len(res)==1]}' - res.insert(0, f'[ValidationError] {len(res)} {num} occured during input validation') + res.insert(0, f'[ValidationError] {len(res)} {num} occurred during input validation') return '\n'.join(res) @@ -484,7 +487,7 @@ def config_import(verbose=0, secrets=False, quiet=False, color=False, update=Fal if backref is not None: log('Modified', item, '{target!r} {key}: {before!r} -> {after!r}'.format(**backref)) return - # verbose? + # show input data? if not verbose >= 2: return # hide secrets in data @@ -532,7 +535,7 @@ def config_import(verbose=0, secrets=False, quiet=False, color=False, update=Fal except ValidationError as exc: raise click.ClickException(format_errors(exc.messages)) from exc except Exception as exc: - if verbose >= 5: + if verbose >= 3: raise # (yaml.scanner.ScannerError, UnicodeDecodeError, ...) raise click.ClickException( @@ -584,7 +587,7 @@ def config_export(full=False, secrets=False, color=False, dns=False, output=None if only: for spec in only: if spec.split('.', 1)[0] not in MailuSchema.Meta.order: - raise click.ClickException(f'[ERROR] Unknown section: {spec}') + raise click.ClickException(f'[ValidationError] Unknown section: {spec}') else: only = MailuSchema.Meta.order @@ -606,7 +609,7 @@ def config_export(full=False, secrets=False, color=False, dns=False, output=None print(colorize(schema.dumps(models.MailuConfig()), **color_cfg), file=output) except ValueError as exc: if spec := get_fieldspec(exc): - raise click.ClickException(f'[ERROR] Invalid filter: {spec}') from exc + raise click.ClickException(f'[ValidationError] Invalid filter: {spec}') from exc raise diff --git a/core/admin/mailu/models.py b/core/admin/mailu/models.py index 5799e282..4c119984 100644 --- a/core/admin/mailu/models.py +++ b/core/admin/mailu/models.py @@ -33,7 +33,7 @@ class IdnaDomain(db.TypeDecorator): """ Stores a Unicode string in it's IDNA representation (ASCII only) """ - # TODO: String(80) is too small? + # TODO: use db.String(255)? impl = db.String(80) def process_bind_param(self, value, dialect): @@ -50,7 +50,7 @@ class IdnaEmail(db.TypeDecorator): """ Stores a Unicode string in it's IDNA representation (ASCII only) """ - # TODO: String(255) is too small? + # TODO: use db.String(254)? impl = db.String(255) def process_bind_param(self, value, dialect): @@ -314,7 +314,7 @@ class Relay(Base): __tablename__ = 'relay' name = db.Column(IdnaDomain, primary_key=True, nullable=False) - # TODO: String(80) is too small? + # TODO: use db.String(266)? transport(8):(1)[nexthop(255)](2) smtp = db.Column(db.String(80), nullable=True) @@ -322,9 +322,7 @@ class Email(object): """ Abstraction for an email address (localpart and domain). """ - # TODO: validate max. total length of address (<=254) - - # TODO: String(80) is too large (64)? + # TODO: use db.String(64)? localpart = db.Column(db.String(80), nullable=False) @declarative.declared_attr @@ -634,7 +632,7 @@ class Token(Base): user = db.relationship(User, backref=db.backref('tokens', cascade='all, delete-orphan')) password = db.Column(db.String(255), nullable=False) - # TODO: String(255) is too large? (43 should be sufficient) + # TODO: use db.String(32)? ip = db.Column(db.String(255)) def check_password(self, password): diff --git a/core/admin/mailu/schemas.py b/core/admin/mailu/schemas.py index b9b8e393..7d0393f0 100644 --- a/core/admin/mailu/schemas.py +++ b/core/admin/mailu/schemas.py @@ -36,7 +36,7 @@ from . import models, dkim ma = Marshmallow() -# TODO: how and where to mark keys as "required" while unserializing in api? +# TODO: how and where to mark keys as "required" while deserializing in api? # - when modifying, nothing is required (only the primary key, but this key is in the uri) # - the primary key from post data must not differ from the key in the uri # - when creating all fields without default or auto-increment are required @@ -705,6 +705,7 @@ class BaseSchema(ma.SQLAlchemyAutoSchema): # add attributes required for validation from db # TODO: this will cause validation errors if value from database does not validate + # but there should not be an invalid value in the database for attr_name, field_obj in self.load_fields.items(): if field_obj.required and attr_name not in data: data[attr_name] = getattr(instance, attr_name) diff --git a/docs/cli.rst b/docs/cli.rst index 497cdfc5..6d48c576 100644 --- a/docs/cli.rst +++ b/docs/cli.rst @@ -97,7 +97,7 @@ where mail-config.yml looks like: without ``--delete-object`` option config-update will only add/update new values but will *not* remove any entries missing in provided YAML input. Users ------ +^^^^^ following are additional parameters that could be defined for users: @@ -116,7 +116,7 @@ following are additional parameters that could be defined for users: * spam_threshold Alias ------ +^^^^^ additional fields: @@ -125,11 +125,11 @@ additional fields: config-export ------------- -The purpose of this command is to export domain-, relay-, alias- and user-configuration in YAML or JSON format. +The purpose of this command is to export the complete configuration in YAML or JSON format. .. code-block:: bash - # docker-compose exec admin flask mailu config-export --help + $ docker-compose exec admin flask mailu config-export --help Usage: flask mailu config-export [OPTIONS] [FILTER]... @@ -152,18 +152,18 @@ filters to export only some objects or attributes (try: ``user`` or ``domain.nam .. code-block:: bash - docker-compose exec admin flask mailu config-export -o mail-config.yml + $ docker-compose exec admin flask mailu config-export -o mail-config.yml - docker-compose exec admin flask mailu config-export --dns domain.dns_mx domain.dns_spf + $ docker-compose exec admin flask mailu config-export --dns domain.dns_mx domain.dns_spf config-import ------------- -The purpose of this command is for importing domain-, relay-, alias- and user-configuration in bulk and synchronizing DB entries with an external YAML/JOSN source. +This command imports configuration data from an external YAML or JSON source. .. code-block:: bash - # docker-compose exec admin flask mailu config-import --help + $ docker-compose exec admin flask mailu config-import --help Usage: flask mailu config-import [OPTIONS] [FILENAME|-] @@ -211,13 +211,28 @@ mail-config.yml contains the configuration and looks like this: config-update shows the number of created/modified/deleted objects after import. To suppress all messages except error messages use ``--quiet``. -By adding the ``--verbose`` switch (one or more times) the import gets more detailed and shows exactyl what attributes changed. -In all messages plain-text secrets (dkim-keys, passwords) are hidden by default. Use ``--secrets`` to show secrets. -If you want to test what would be done when importing use ``--dry-run``. -By default config-update replaces the whole configuration. You can use ``--update`` to change the existing configuration instead. -When updating you can add new and change existing objects. -To delete an object use ``-key: value`` (To delete the domain example.com ``-name: example.com`` for example). -To reset an attribute to default use ``-key: null`` (To reset enable_imap ``-enable_imap: null`` for example). +By adding the ``--verbose`` switch (up to two times) the import gets more detailed and shows exactly what attributes changed. +In all log messages plain-text secrets (dkim-keys, passwords) are hidden by default. Use ``--secrets`` to log secrets. +If you want to test what would be done when importing without committing any changes, use ``--dry-run``. + +By default config-update replaces the whole configuration. ``--update`` allows to modify the existing configuration instead. +New elements will be added and existing elements will be modified. +It is possible to delete a single element or prune all elements from lists and associative arrays using a special notation: + ++-----------------------------+------------------+--------------------------+ +| Delete what? | notation | example | ++=============================+==================+==========================+ +| specific array object | ``- -key: id`` | ``- -name: example.com`` | ++-----------------------------+------------------+--------------------------+ +| specific list item | ``- -id`` | ``- -user1@example.com`` | ++-----------------------------+------------------+--------------------------+ +| all remaining array objects | ``- -key: null`` | ``- -email: null`` | ++-----------------------------+------------------+--------------------------+ +| all remaining list items | ``- -prune-`` | ``- -prune-`` | ++-----------------------------+------------------+--------------------------+ + +The ``-key: null`` notation can also be used to reset an attribute to its default. +To reset *spam_threshold* to it's default *80* use ``-spam_threshold: null``. This is a complete YAML template with all additional parameters that can be defined: From e4c83e162dfa37189b8eac506b07ec6f78a5b5d0 Mon Sep 17 00:00:00 2001 From: Alexander Graf Date: Tue, 16 Feb 2021 17:59:43 +0100 Subject: [PATCH 086/596] fixed colorize auto detection --- core/admin/mailu/manage.py | 6 +++--- core/admin/mailu/schemas.py | 8 ++++---- 2 files changed, 7 insertions(+), 7 deletions(-) diff --git a/core/admin/mailu/manage.py b/core/admin/mailu/manage.py index 756400ad..bef49faa 100644 --- a/core/admin/mailu/manage.py +++ b/core/admin/mailu/manage.py @@ -19,7 +19,7 @@ from flask.cli import FlaskGroup, with_appcontext from marshmallow.exceptions import ValidationError from . import models -from .schemas import MailuSchema, get_schema, get_fieldspec, colorize, RenderJSON, HIDDEN +from .schemas import MailuSchema, get_schema, get_fieldspec, colorize, canColorize, RenderJSON, HIDDEN db = models.db @@ -351,7 +351,7 @@ def config_import(verbose=0, secrets=False, quiet=False, color=False, update=Fal verbose = 2 color_cfg = { - 'color': color or sys.stdout.isatty(), + 'color': color or (canColorize and sys.stdout.isatty()), 'lexer': 'python', 'strip': True, } @@ -598,7 +598,7 @@ def config_export(full=False, secrets=False, color=False, dns=False, output=None } schema = MailuSchema(only=only, context=context) - color_cfg = {'color': color or output.isatty()} + color_cfg = {'color': color or (canColorize and output.isatty())} if as_json: schema.opts.render_module = RenderJSON diff --git a/core/admin/mailu/schemas.py b/core/admin/mailu/schemas.py index 7d0393f0..6a8303c5 100644 --- a/core/admin/mailu/schemas.py +++ b/core/admin/mailu/schemas.py @@ -27,9 +27,9 @@ try: from pygments.lexers.data import YamlLexer from pygments.formatters import get_formatter_by_name except ModuleNotFoundError: - COLOR_SUPPORTED = False + canColorize = False else: - COLOR_SUPPORTED = True + canColorize = True from . import models, dkim @@ -99,11 +99,11 @@ def colorize(data, lexer='yaml', formatter='terminal', color=None, strip=False): """ add ANSI color to data """ if color is None: # autodetect colorize - color = COLOR_SUPPORTED + color = canColorize if not color: # no color wanted return data - if not COLOR_SUPPORTED: + if not canColorize: # want color, but not supported raise ValueError('Please install pygments to colorize output') From aa8cb9890693b1563cb873df79d50a2fab6230e3 Mon Sep 17 00:00:00 2001 From: Florent Daigniere Date: Thu, 18 Feb 2021 12:31:45 +0100 Subject: [PATCH 087/596] Set sensible cookie options --- core/admin/mailu/configuration.py | 3 +++ 1 file changed, 3 insertions(+) diff --git a/core/admin/mailu/configuration.py b/core/admin/mailu/configuration.py index 2cf6a478..982a1eb0 100644 --- a/core/admin/mailu/configuration.py +++ b/core/admin/mailu/configuration.py @@ -123,6 +123,9 @@ class ConfigManager(dict): self.config['RATELIMIT_STORAGE_URL'] = 'redis://{0}/2'.format(self.config['REDIS_ADDRESS']) self.config['QUOTA_STORAGE_URL'] = 'redis://{0}/1'.format(self.config['REDIS_ADDRESS']) + self.config['SESSION_COOKIE_SAMESITE'] = 'Strict' + self.config['SESSION_COOKIE_HTTPONLY'] = True + self.config['SESSION_COOKIE_SECURE'] = self.config['TLS_FLAVOR'] != 'notls' # update the app config itself app.config = self From bde7a2b6c4a8a2351b461cee9be413d7683e95dd Mon Sep 17 00:00:00 2001 From: Alexander Graf Date: Fri, 19 Feb 2021 18:01:02 +0100 Subject: [PATCH 088/596] moved import logging to schema - yaml-import is now logged via schema.Logger - iremoved relative imports - not used in other mailu modules - removed develepment comments - added Mailconfig.check method to check for duplicate domain names - converted .format() to .format_map() where possible - switched to yaml multiline dump for dkim_key - converted dkim_key import from regex to string functions - automatically unhide/unexclude explicitly specified attributes on dump - use field order when loading to stabilize import - fail when using 'hash_password' without 'password' - fixed logging of dkim_key - fixed pruning and deleting of lists - modified error messages - added debug flag and two verbosity levels --- core/admin/mailu/manage.py | 280 ++--------- core/admin/mailu/models.py | 40 +- core/admin/mailu/schemas.py | 922 ++++++++++++++++++++++++------------ docs/cli.rst | 15 +- 4 files changed, 688 insertions(+), 569 deletions(-) diff --git a/core/admin/mailu/manage.py b/core/admin/mailu/manage.py index bef49faa..f9add0f4 100644 --- a/core/admin/mailu/manage.py +++ b/core/admin/mailu/manage.py @@ -4,22 +4,16 @@ import sys import os import socket -import logging import uuid -from collections import Counter -from itertools import chain - import click -import sqlalchemy import yaml from flask import current_app as app from flask.cli import FlaskGroup, with_appcontext -from marshmallow.exceptions import ValidationError -from . import models -from .schemas import MailuSchema, get_schema, get_fieldspec, colorize, canColorize, RenderJSON, HIDDEN +from mailu import models +from mailu.schemas import MailuSchema, Logger, RenderJSON db = models.db @@ -326,246 +320,53 @@ def config_update(verbose=False, delete_objects=False): @mailu.command() @click.option('-v', '--verbose', count=True, help='Increase verbosity.') @click.option('-s', '--secrets', is_flag=True, help='Show secret attributes in messages.') +@click.option('-d', '--debug', is_flag=True, help='Enable debug output.') @click.option('-q', '--quiet', is_flag=True, help='Quiet mode - only show errors.') @click.option('-c', '--color', is_flag=True, help='Force colorized output.') @click.option('-u', '--update', is_flag=True, help='Update mode - merge input with existing config.') @click.option('-n', '--dry-run', is_flag=True, help='Perform a trial run with no changes made.') @click.argument('source', metavar='[FILENAME|-]', type=click.File(mode='r'), default=sys.stdin) @with_appcontext -def config_import(verbose=0, secrets=False, quiet=False, color=False, update=False, dry_run=False, source=None): +def config_import(verbose=0, secrets=False, debug=False, quiet=False, color=False, + update=False, dry_run=False, source=None): """ Import configuration as YAML or JSON from stdin or file """ - # verbose - # 0 : only show number of changes - # 1 : also show detailed changes - # 2 : also show input data - # 3 : also show sql queries (also needs -s, as sql may contain secrets) - # 4 : also show tracebacks (also needs -s, as tracebacks may contain secrets) + log = Logger(want_color=color or None, can_color=sys.stdout.isatty(), secrets=secrets, debug=debug) + log.lexer = 'python' + log.strip = True + log.verbose = 0 if quiet else verbose + log.quiet = quiet - if quiet: - verbose = -1 - - if verbose > 2 and not secrets: - print('[Warning] Verbosity level capped to 2. Specify --secrets to log sql and tracebacks.') - verbose = 2 - - color_cfg = { - 'color': color or (canColorize and sys.stdout.isatty()), - 'lexer': 'python', - 'strip': True, - } - - counter = Counter() - logger = {} - - def format_errors(store, path=None): - - res = [] - if path is None: - path = [] - for key in sorted(store): - location = path + [str(key)] - value = store[key] - if isinstance(value, dict): - res.extend(format_errors(value, location)) - else: - for message in value: - res.append((".".join(location), message)) - - if path: - return res - - fmt = f' - {{:<{max([len(loc) for loc, msg in res])}}} : {{}}' - res = [fmt.format(loc, msg) for loc, msg in res] - num = f'error{["s",""][len(res)==1]}' - res.insert(0, f'[ValidationError] {len(res)} {num} occurred during input validation') - - return '\n'.join(res) - - def format_changes(*message): - if counter: - changes = [] - last = None - for (action, what), count in sorted(counter.items()): - if action != last: - if last: - changes.append('/') - changes.append(f'{action}:') - last = action - changes.append(f'{what}({count})') - else: - changes = ['No changes.'] - return chain(message, changes) - - def log(action, target, message=None): - - if message is None: - try: - message = logger[target.__class__].dump(target) - except KeyError: - message = target - if not isinstance(message, str): - message = repr(message) - print(f'{action} {target.__table__}: {colorize(message, **color_cfg)}') - - def listen_insert(mapper, connection, target): # pylint: disable=unused-argument - """ callback function to track import """ - counter.update([('Created', target.__table__.name)]) - if verbose >= 1: - log('Created', target) - - def listen_update(mapper, connection, target): # pylint: disable=unused-argument - """ callback function to track import """ - - changed = {} - inspection = sqlalchemy.inspect(target) - for attr in sqlalchemy.orm.class_mapper(target.__class__).column_attrs: - history = getattr(inspection.attrs, attr.key).history - if history.has_changes() and history.deleted: - before = history.deleted[-1] - after = getattr(target, attr.key) - # TODO: remove special handling of "comment" after modifying model - if attr.key == 'comment' and not before and not after: - pass - # only remember changed keys - elif before != after: - if verbose >= 1: - changed[str(attr.key)] = (before, after) - else: - break - - if verbose >= 1: - # use schema with dump_context to hide secrets and sort keys - dumped = get_schema(target)(only=changed.keys(), context=diff_context).dump(target) - for key, value in dumped.items(): - before, after = changed[key] - if value == HIDDEN: - before = HIDDEN if before else before - after = HIDDEN if after else after - else: - # TODO: need to use schema to "convert" before value? - after = value - log('Modified', target, f'{str(target)!r} {key}: {before!r} -> {after!r}') - - if changed: - counter.update([('Modified', target.__table__.name)]) - - def listen_delete(mapper, connection, target): # pylint: disable=unused-argument - """ callback function to track import """ - counter.update([('Deleted', target.__table__.name)]) - if verbose >= 1: - log('Deleted', target) - - # TODO: this listener will not be necessary, if dkim keys would be stored in database - _dedupe_dkim = set() - def listen_dkim(session, flush_context): # pylint: disable=unused-argument - """ callback function to track import """ - for target in session.identity_map.values(): - # look at Domains originally loaded from db - if not isinstance(target, models.Domain) or not target._sa_instance_state.load_path: - continue - before = target._dkim_key_on_disk - after = target._dkim_key - if before != after: - if secrets: - before = before.decode('ascii', 'ignore') - after = after.decode('ascii', 'ignore') - else: - before = HIDDEN if before else '' - after = HIDDEN if after else '' - # "de-dupe" messages; this event is fired at every flush - if not (target, before, after) in _dedupe_dkim: - _dedupe_dkim.add((target, before, after)) - counter.update([('Modified', target.__table__.name)]) - if verbose >= 1: - log('Modified', target, f'{str(target)!r} dkim_key: {before!r} -> {after!r}') - - def track_serialize(obj, item, backref=None): - """ callback function to track import """ - # called for backref modification? - if backref is not None: - log('Modified', item, '{target!r} {key}: {before!r} -> {after!r}'.format(**backref)) - return - # show input data? - if not verbose >= 2: - return - # hide secrets in data - data = logger[obj.opts.model].hide(item) - if 'hash_password' in data: - data['password'] = HIDDEN - if 'fetches' in data: - for fetch in data['fetches']: - fetch['password'] = HIDDEN - log('Handling', obj.opts.model, data) - - # configure contexts - diff_context = { - 'full': True, - 'secrets': secrets, - } - log_context = { - 'secrets': secrets, - } - load_context = { + context = { 'import': True, 'update': update, 'clear': not update, - 'callback': track_serialize, + 'callback': log.track_serialize, } - # register listeners - for schema in get_schema(): - model = schema.Meta.model - logger[model] = schema(context=log_context) - sqlalchemy.event.listen(model, 'after_insert', listen_insert) - sqlalchemy.event.listen(model, 'after_update', listen_update) - sqlalchemy.event.listen(model, 'after_delete', listen_delete) - - # special listener for dkim_key changes - sqlalchemy.event.listen(db.session, 'after_flush', listen_dkim) - - if verbose >= 3: - logging.basicConfig() - logging.getLogger('sqlalchemy.engine').setLevel(logging.INFO) + schema = MailuSchema(only=MailuSchema.Meta.order, context=context) try: + # import source with models.db.session.no_autoflush: - config = MailuSchema(only=MailuSchema.Meta.order, context=load_context).loads(source) - except ValidationError as exc: - raise click.ClickException(format_errors(exc.messages)) from exc + config = schema.loads(source) + # flush session to show/count all changes + if not quiet and (dry_run or verbose): + db.session.flush() + # check for duplicate domain names + config.check() except Exception as exc: - if verbose >= 3: - raise - # (yaml.scanner.ScannerError, UnicodeDecodeError, ...) - raise click.ClickException( - f'[{exc.__class__.__name__}] ' - f'{" ".join(str(exc).split())}' - ) from exc - - # flush session to show/count all changes - if dry_run or verbose >= 1: - db.session.flush() - - # check for duplicate domain names - dup = set() - for fqdn in chain( - db.session.query(models.Domain.name), - db.session.query(models.Alternative.name), - db.session.query(models.Relay.name) - ): - if fqdn in dup: - raise click.ClickException(f'[ValidationError] Duplicate domain name: {fqdn}') - dup.add(fqdn) + if msg := log.format_exception(exc): + raise click.ClickException(msg) from exc + raise # don't commit when running dry if dry_run: - if not quiet: - print(*format_changes('Dry run. Not commiting changes.')) + log.changes('Dry run. Not committing changes.') db.session.rollback() else: - if not quiet: - print(*format_changes('Committing changes.')) + log.changes('Committing changes.') db.session.commit() @@ -573,8 +374,8 @@ def config_import(verbose=0, secrets=False, quiet=False, color=False, update=Fal @click.option('-f', '--full', is_flag=True, help='Include attributes with default value.') @click.option('-s', '--secrets', is_flag=True, help='Include secret attributes (dkim-key, passwords).') -@click.option('-c', '--color', is_flag=True, help='Force colorized output.') @click.option('-d', '--dns', is_flag=True, help='Include dns records.') +@click.option('-c', '--color', is_flag=True, help='Force colorized output.') @click.option('-o', '--output-file', 'output', default=sys.stdout, type=click.File(mode='w'), help='Save configuration to file.') @click.option('-j', '--json', 'as_json', is_flag=True, help='Export configuration in json format.') @@ -584,32 +385,25 @@ def config_export(full=False, secrets=False, color=False, dns=False, output=None """ Export configuration as YAML or JSON to stdout or file """ - if only: - for spec in only: - if spec.split('.', 1)[0] not in MailuSchema.Meta.order: - raise click.ClickException(f'[ValidationError] Unknown section: {spec}') - else: - only = MailuSchema.Meta.order + only = only or MailuSchema.Meta.order context = { 'full': full, 'secrets': secrets, 'dns': dns, } - - schema = MailuSchema(only=only, context=context) - color_cfg = {'color': color or (canColorize and output.isatty())} - - if as_json: - schema.opts.render_module = RenderJSON - color_cfg['lexer'] = 'json' - color_cfg['strip'] = True + log = Logger(want_color=color or None, can_color=output.isatty()) try: - print(colorize(schema.dumps(models.MailuConfig()), **color_cfg), file=output) - except ValueError as exc: - if spec := get_fieldspec(exc): - raise click.ClickException(f'[ValidationError] Invalid filter: {spec}') from exc + schema = MailuSchema(only=only, context=context) + if as_json: + schema.opts.render_module = RenderJSON + log.lexer = 'json' + log.strip = True + print(log.colorize(schema.dumps(models.MailuConfig())), file=output) + except Exception as exc: + if msg := log.format_exception(exc): + raise click.ClickException(msg) from exc raise diff --git a/core/admin/mailu/models.py b/core/admin/mailu/models.py index 4c119984..1b3c787a 100644 --- a/core/admin/mailu/models.py +++ b/core/admin/mailu/models.py @@ -23,7 +23,7 @@ from sqlalchemy.ext.hybrid import hybrid_property from sqlalchemy.inspection import inspect from werkzeug.utils import cached_property -from . import dkim +from mailu import dkim db = flask_sqlalchemy.SQLAlchemy() @@ -33,7 +33,6 @@ class IdnaDomain(db.TypeDecorator): """ Stores a Unicode string in it's IDNA representation (ASCII only) """ - # TODO: use db.String(255)? impl = db.String(80) def process_bind_param(self, value, dialect): @@ -50,7 +49,6 @@ class IdnaEmail(db.TypeDecorator): """ Stores a Unicode string in it's IDNA representation (ASCII only) """ - # TODO: use db.String(254)? impl = db.String(255) def process_bind_param(self, value, dialect): @@ -127,11 +125,7 @@ class Base(db.Model): if pkey == 'email': # ugly hack for email declared attr. _email is not always up2date return str(f'{self.localpart}@{self.domain_name}') - elif pkey in {'name', 'email'}: - return str(getattr(self, pkey, None)) - else: - return self.__repr__() - return str(getattr(self, self.__table__.primary_key.columns.values()[0].name)) + return str(getattr(self, pkey)) def __repr__(self): return f'<{self.__class__.__name__} {str(self)!r}>' @@ -145,12 +139,15 @@ class Base(db.Model): else: return NotImplemented + # we need hashable instances here for sqlalchemy to update collections + # in collections.bulk_replace, but auto-incrementing don't always have + # a valid primary key, in this case we use the object's id + __hashed = None def __hash__(self): - primary = getattr(self, self.__table__.primary_key.columns.values()[0].name) - if primary is None: - return NotImplemented - else: - return hash(primary) + if self.__hashed is None: + primary = getattr(self, self.__table__.primary_key.columns.values()[0].name) + self.__hashed = id(self) if primary is None else hash(primary) + return self.__hashed # Many-to-many association table for domain managers @@ -314,7 +311,6 @@ class Relay(Base): __tablename__ = 'relay' name = db.Column(IdnaDomain, primary_key=True, nullable=False) - # TODO: use db.String(266)? transport(8):(1)[nexthop(255)](2) smtp = db.Column(db.String(80), nullable=True) @@ -322,7 +318,6 @@ class Email(object): """ Abstraction for an email address (localpart and domain). """ - # TODO: use db.String(64)? localpart = db.Column(db.String(80), nullable=False) @declarative.declared_attr @@ -342,7 +337,7 @@ class Email(object): key = f'{cls.__tablename__}_email' if key in ctx.current_parameters: return ctx.current_parameters[key] - return '{localpart}@{domain_name}'.format(**ctx.current_parameters) + return '{localpart}@{domain_name}'.format_map(ctx.current_parameters) return db.Column('email', IdnaEmail, primary_key=True, nullable=False, onupdate=updater) @@ -632,7 +627,6 @@ class Token(Base): user = db.relationship(User, backref=db.backref('tokens', cascade='all, delete-orphan')) password = db.Column(db.String(255), nullable=False) - # TODO: use db.String(32)? ip = db.Column(db.String(255)) def check_password(self, password): @@ -865,6 +859,18 @@ class MailuConfig: if models is None or model in models: db.session.query(model).delete() + def check(self): + """ check for duplicate domain names """ + dup = set() + for fqdn in chain( + db.session.query(Domain.name), + db.session.query(Alternative.name), + db.session.query(Relay.name) + ): + if fqdn in dup: + raise ValueError(f'Duplicate domain name: {fqdn}') + dup.add(fqdn) + domain = MailuCollection(Domain) user = MailuCollection(User) alias = MailuCollection(Alias) diff --git a/core/admin/mailu/schemas.py b/core/admin/mailu/schemas.py index 6a8303c5..4c5042ea 100644 --- a/core/admin/mailu/schemas.py +++ b/core/admin/mailu/schemas.py @@ -2,10 +2,10 @@ """ from copy import deepcopy -from textwrap import wrap +from collections import Counter -import re import json +import logging import yaml import sqlalchemy @@ -27,24 +27,309 @@ try: from pygments.lexers.data import YamlLexer from pygments.formatters import get_formatter_by_name except ModuleNotFoundError: - canColorize = False + COLOR_SUPPORTED = False else: - canColorize = True + COLOR_SUPPORTED = True -from . import models, dkim +from mailu import models, dkim ma = Marshmallow() -# TODO: how and where to mark keys as "required" while deserializing in api? -# - when modifying, nothing is required (only the primary key, but this key is in the uri) -# - the primary key from post data must not differ from the key in the uri -# - when creating all fields without default or auto-increment are required -# TODO: validate everything! + +### import logging and schema colorization ### + +_model2schema = {} + +def get_schema(cls=None): + """ return schema class for model """ + if cls is None: + return _model2schema.values() + return _model2schema.get(cls) + +def mapped(cls): + """ register schema in model2schema map """ + _model2schema[cls.Meta.model] = cls + return cls + +class MyYamlLexer(YamlLexer): + """ colorize yaml constants and integers """ + def get_tokens(self, text, unfiltered=False): + for typ, value in super().get_tokens(text, unfiltered): + if typ is Token.Literal.Scalar.Plain: + if value in {'true', 'false', 'null'}: + typ = Token.Keyword.Constant + elif value == HIDDEN: + typ = Token.Error + else: + try: + int(value, 10) + except ValueError: + try: + float(value) + except ValueError: + pass + else: + typ = Token.Literal.Number.Float + else: + typ = Token.Literal.Number.Integer + yield typ, value + +class Logger: + + def __init__(self, want_color=None, can_color=False, debug=False, secrets=False): + + self.lexer = 'yaml' + self.formatter = 'terminal' + self.strip = False + self.verbose = 0 + self.quiet = False + self.secrets = secrets + self.debug = debug + self.print = print + + if want_color and not COLOR_SUPPORTED: + raise ValueError('Please install pygments to colorize output') + + self.color = want_color or (can_color and COLOR_SUPPORTED) + + self._counter = Counter() + self._schemas = {} + + # log contexts + self._diff_context = { + 'full': True, + 'secrets': secrets, + } + log_context = { + 'secrets': secrets, + } + + # register listeners + for schema in get_schema(): + model = schema.Meta.model + self._schemas[model] = schema(context=log_context) + sqlalchemy.event.listen(model, 'after_insert', self._listen_insert) + sqlalchemy.event.listen(model, 'after_update', self._listen_update) + sqlalchemy.event.listen(model, 'after_delete', self._listen_delete) + + # special listener for dkim_key changes + # TODO: _listen_dkim can be removed when dkim keys are stored in database + self._dedupe_dkim = set() + sqlalchemy.event.listen(models.db.session, 'after_flush', self._listen_dkim) + + # register debug logger for sqlalchemy + if self.debug: + logging.basicConfig() + logging.getLogger('sqlalchemy.engine').setLevel(logging.INFO) + + def _log(self, action, target, message=None): + if message is None: + try: + message = self._schemas[target.__class__].dump(target) + except KeyError: + message = target + if not isinstance(message, str): + message = repr(message) + self.print(f'{action} {target.__table__}: {self.colorize(message)}') + + def _listen_insert(self, mapper, connection, target): # pylint: disable=unused-argument + """ callback method to track import """ + self._counter.update([('Created', target.__table__.name)]) + if self.verbose: + self._log('Created', target) + + def _listen_update(self, mapper, connection, target): # pylint: disable=unused-argument + """ callback method to track import """ + + changes = {} + inspection = sqlalchemy.inspect(target) + for attr in sqlalchemy.orm.class_mapper(target.__class__).column_attrs: + history = getattr(inspection.attrs, attr.key).history + if history.has_changes() and history.deleted: + before = history.deleted[-1] + after = getattr(target, attr.key) + # TODO: this can be removed when comment is not nullable in model + if attr.key == 'comment' and not before and not after: + pass + # only remember changed keys + elif before != after: + if self.verbose: + changes[str(attr.key)] = (before, after) + else: + break + + if self.verbose: + # use schema to log changed attributes + schema = get_schema(target.__class__) + only = set(changes.keys()) & set(schema().fields.keys()) + if only: + for key, value in schema( + only=only, + context=self._diff_context + ).dump(target).items(): + before, after = changes[key] + if value == HIDDEN: + before = HIDDEN if before else before + after = HIDDEN if after else after + else: + # also hide this + after = value + self._log('Modified', target, f'{str(target)!r} {key}: {before!r} -> {after!r}') + + if changes: + self._counter.update([('Modified', target.__table__.name)]) + + def _listen_delete(self, mapper, connection, target): # pylint: disable=unused-argument + """ callback method to track import """ + self._counter.update([('Deleted', target.__table__.name)]) + if self.verbose: + self._log('Deleted', target) + + # TODO: _listen_dkim can be removed when dkim keys are stored in database + def _listen_dkim(self, session, flush_context): # pylint: disable=unused-argument + """ callback method to track import """ + for target in session.identity_map.values(): + # look at Domains originally loaded from db + if not isinstance(target, models.Domain) or not target._sa_instance_state.load_path: + continue + before = target._dkim_key_on_disk + after = target._dkim_key + if before != after: + # "de-dupe" messages; this event is fired at every flush + if not (target, before, after) in self._dedupe_dkim: + self._dedupe_dkim.add((target, before, after)) + self._counter.update([('Modified', target.__table__.name)]) + if self.verbose: + if self.secrets: + before = before.decode('ascii', 'ignore') + after = after.decode('ascii', 'ignore') + else: + before = HIDDEN if before else '' + after = HIDDEN if after else '' + self._log('Modified', target, f'{str(target)!r} dkim_key: {before!r} -> {after!r}') + + def track_serialize(self, obj, item, backref=None): + """ callback method to track import """ + # called for backref modification? + if backref is not None: + self._log('Modified', item, '{target!r} {key}: {before!r} -> {after!r}'.format_map(backref)) + return + # show input data? + if self.verbose < 2: + return + # hide secrets in data + if not self.secrets: + item = self._schemas[obj.opts.model].hide(item) + if 'hash_password' in item: + item['password'] = HIDDEN + if 'fetches' in item: + for fetch in item['fetches']: + fetch['password'] = HIDDEN + self._log('Handling', obj.opts.model, item) + + def changes(self, *messages, **kwargs): + """ show changes gathered in counter """ + if self.quiet: + return + if self._counter: + changes = [] + last = None + for (action, what), count in sorted(self._counter.items()): + if action != last: + if last: + changes.append('/') + changes.append(f'{action}:') + last = action + changes.append(f'{what}({count})') + else: + changes = ['No changes.'] + self.print(*messages, *changes, **kwargs) + + def _format_errors(self, store, path=None): + + res = [] + if path is None: + path = [] + for key in sorted(store): + location = path + [str(key)] + value = store[key] + if isinstance(value, dict): + res.extend(self._format_errors(value, location)) + else: + for message in value: + res.append((".".join(location), message)) + + if path: + return res + + maxlen = max([len(loc) for loc, msg in res]) + res = [f' - {loc.ljust(maxlen)} : {msg}' for loc, msg in res] + errors = f'{len(res)} error{["s",""][len(res)==1]}' + res.insert(0, f'[ValidationError] {errors} occurred during input validation') + + return '\n'.join(res) + + def _is_validation_error(self, exc): + """ walk traceback to extract invalid field from marshmallow """ + path = [] + trace = exc.__traceback__ + while trace: + if trace.tb_frame.f_code.co_name == '_serialize': + if 'attr' in trace.tb_frame.f_locals: + path.append(trace.tb_frame.f_locals['attr']) + elif trace.tb_frame.f_code.co_name == '_init_fields': + spec = ', '.join(['.'.join(path + [key]) for key in trace.tb_frame.f_locals['invalid_fields']]) + return f'Invalid filter: {spec}' + trace = trace.tb_next + return None + + def format_exception(self, exc): + """ format ValidationErrors and other exceptions when not debugging """ + if isinstance(exc, ValidationError): + return self._format_errors(exc.messages) + if isinstance(exc, ValueError): + if msg := self._is_validation_error(exc): + return msg + if self.debug: + return None + msg = ' '.join(str(exc).split()) + return f'[{exc.__class__.__name__}] {msg}' + + colorscheme = { + Token: ('', ''), + Token.Name.Tag: ('cyan', 'cyan'), + Token.Literal.Scalar: ('green', 'green'), + Token.Literal.String: ('green', 'green'), + Token.Name.Constant: ('green', 'green'), # multiline strings + Token.Keyword.Constant: ('magenta', 'magenta'), + Token.Literal.Number: ('magenta', 'magenta'), + Token.Error: ('red', 'red'), + Token.Name: ('red', 'red'), + Token.Operator: ('red', 'red'), + } + + def colorize(self, data, lexer=None, formatter=None, color=None, strip=None): + """ add ANSI color to data """ + + if color is False or not self.color: + return data + + lexer = lexer or self.lexer + lexer = MyYamlLexer() if lexer == 'yaml' else get_lexer_by_name(lexer) + formatter = get_formatter_by_name(formatter or self.formatter, colorscheme=self.colorscheme) + if strip is None: + strip = self.strip + + res = highlight(data, lexer, formatter) + if strip: + return res.rstrip('\n') + return res -### class for hidden values ### +### marshmallow render modules ### +# hidden attributes class _Hidden: def __bool__(self): return False @@ -58,107 +343,24 @@ class _Hidden: return '' __str__ = __repr__ -HIDDEN = _Hidden() - - -### map model to schema ### - -_model2schema = {} - -def get_schema(model=None): - """ return schema class for model or instance of model """ - if model is None: - return _model2schema.values() - else: - return _model2schema.get(model) or _model2schema.get(model.__class__) - -def mapped(cls): - """ register schema in model2schema map """ - _model2schema[cls.Meta.model] = cls - return cls - - -### helper functions ### - -def get_fieldspec(exc): - """ walk traceback to extract spec of invalid field from marshmallow """ - path = [] - tbck = exc.__traceback__ - while tbck: - if tbck.tb_frame.f_code.co_name == '_serialize': - if 'attr' in tbck.tb_frame.f_locals: - path.append(tbck.tb_frame.f_locals['attr']) - elif tbck.tb_frame.f_code.co_name == '_init_fields': - path = '.'.join(path) - spec = ', '.join([f'{path}.{key}' for key in tbck.tb_frame.f_locals['invalid_fields']]) - return spec - tbck = tbck.tb_next - return None - -def colorize(data, lexer='yaml', formatter='terminal', color=None, strip=False): - """ add ANSI color to data """ - if color is None: - # autodetect colorize - color = canColorize - if not color: - # no color wanted - return data - if not canColorize: - # want color, but not supported - raise ValueError('Please install pygments to colorize output') - - scheme = { - Token: ('', ''), - Token.Name.Tag: ('cyan', 'brightcyan'), - Token.Literal.Scalar: ('green', 'green'), - Token.Literal.String: ('green', 'green'), - Token.Keyword.Constant: ('magenta', 'brightmagenta'), - Token.Literal.Number: ('magenta', 'brightmagenta'), - Token.Error: ('red', 'brightred'), - Token.Name: ('red', 'brightred'), - Token.Operator: ('red', 'brightred'), - } - - class MyYamlLexer(YamlLexer): - """ colorize yaml constants and integers """ - def get_tokens(self, text, unfiltered=False): - for typ, value in super().get_tokens(text, unfiltered): - if typ is Token.Literal.Scalar.Plain: - if value in {'true', 'false', 'null'}: - typ = Token.Keyword.Constant - elif value == HIDDEN: - typ = Token.Error - else: - try: - int(value, 10) - except ValueError: - try: - float(value) - except ValueError: - pass - else: - typ = Token.Literal.Number.Float - else: - typ = Token.Literal.Number.Integer - yield typ, value - - res = highlight( - data, - MyYamlLexer() if lexer == 'yaml' else get_lexer_by_name(lexer), - get_formatter_by_name(formatter, colorscheme=scheme) - ) - - return res.rstrip('\n') if strip else res - - -### render modules ### - -# allow yaml to represent hidden attributes yaml.add_representer( _Hidden, - lambda cls, data: cls.represent_data(str(data)) + lambda dumper, data: dumper.represent_data(str(data)) ) +HIDDEN = _Hidden() + +# multiline attributes +class _Multiline(str): + pass + +yaml.add_representer( + _Multiline, + lambda dumper, data: dumper.represent_scalar(u'tag:yaml.org,2002:str', data, style='|') + +) + +# yaml render module class RenderYAML: """ Marshmallow YAML Render Module """ @@ -178,7 +380,7 @@ class RenderYAML: @staticmethod def _augment(kwargs, defaults): - """ add default kv's to kwargs if missing + """ add defaults to kwargs if missing """ for key, value in defaults.items(): if key not in kwargs: @@ -205,6 +407,7 @@ class RenderYAML: cls._augment(kwargs, cls._dump_defaults) return yaml.dump(*args, **kwargs) +# json encoder class JSONEncoder(json.JSONEncoder): """ JSONEncoder supporting serialization of HIDDEN """ def default(self, o): @@ -213,13 +416,14 @@ class JSONEncoder(json.JSONEncoder): return str(o) return json.JSONEncoder.default(self, o) +# json render module class RenderJSON: """ Marshmallow JSON Render Module """ @staticmethod def _augment(kwargs, defaults): - """ add default kv's to kwargs if missing + """ add defaults to kwargs if missing """ for key, value in defaults.items(): if key not in kwargs: @@ -245,7 +449,7 @@ class RenderJSON: return json.dumps(*args, **kwargs) -### custom fields ### +### schema: custom fields ### class LazyStringField(fields.String): """ Field that serializes a "false" value to the empty string @@ -261,6 +465,11 @@ class CommaSeparatedListField(fields.Raw): a list of strings """ + default_error_messages = { + "invalid": "Not a valid string or list.", + "invalid_utf8": "Not a valid utf-8 string or list.", + } + def _deserialize(self, value, attr, data, **kwargs): """ deserialize comma separated string to list of strings """ @@ -269,16 +478,31 @@ class CommaSeparatedListField(fields.Raw): if not value: return [] - # split string - if isinstance(value, str): - return list([item.strip() for item in value.split(',') if item.strip()]) + # handle list + if isinstance(value, list): + try: + value = [ensure_text_type(item) for item in value] + except UnicodeDecodeError as exc: + raise self.make_error("invalid_utf8") from exc + + # handle text else: - return value + if not isinstance(value, (str, bytes)): + raise self.make_error("invalid") + try: + value = ensure_text_type(value) + except UnicodeDecodeError as exc: + raise self.make_error("invalid_utf8") from exc + else: + value = filter(None, [item.strip() for item in value.split(',')]) + + return list(value) class DkimKeyField(fields.String): - """ Serialize a dkim key to a list of strings (lines) and - Deserialize a string or list of strings to a valid dkim key + """ Serialize a dkim key to a multiline string and + deserialize a dkim key data as string or list of strings + to a valid dkim key """ default_error_messages = { @@ -286,21 +510,26 @@ class DkimKeyField(fields.String): "invalid_utf8": "Not a valid utf-8 string or list.", } - _clean_re = re.compile( - r'(^-----BEGIN (RSA )?PRIVATE KEY-----|-----END (RSA )?PRIVATE KEY-----$|\s+)', - flags=re.UNICODE - ) - def _serialize(self, value, attr, obj, **kwargs): - """ serialize dkim key to a list of strings (lines) + """ serialize dkim key as multiline string """ # map empty string and None to None if not value: - return None + return '' - # return list of key lines without header/footer - return value.decode('utf-8').strip().split('\n')[1:-1] + # return multiline string + return _Multiline(value.decode('utf-8')) + + def _wrap_key(self, begin, data, end): + """ generator to wrap key into RFC 7468 format """ + yield begin + pos = 0 + while pos < len(data): + yield data[pos:pos+64] + pos += 64 + yield end + yield '' def _deserialize(self, value, attr, data, **kwargs): """ deserialize a string or list of strings to dkim key data @@ -310,7 +539,7 @@ class DkimKeyField(fields.String): # convert list to str if isinstance(value, list): try: - value = ''.join([ensure_text_type(item) for item in value]) + value = ''.join([ensure_text_type(item) for item in value]).strip() except UnicodeDecodeError as exc: raise self.make_error("invalid_utf8") from exc @@ -319,34 +548,53 @@ class DkimKeyField(fields.String): if not isinstance(value, (str, bytes)): raise self.make_error("invalid") try: - value = ensure_text_type(value) + value = ensure_text_type(value).strip() except UnicodeDecodeError as exc: raise self.make_error("invalid_utf8") from exc - # clean value (remove whitespace and header/footer) - value = self._clean_re.sub('', value.strip()) + # generate new key? + if value.lower() == '-generate-': + return dkim.gen_key() - # map empty string/list to None + # no key? if not value: return None - # handle special value 'generate' - elif value == 'generate': - return dkim.gen_key() + # remember part of value for ValidationError + bad_key = value - # remember some keydata for error message - keydata = f'{value[:25]}...{value[-10:]}' if len(value) > 40 else value + # strip header and footer, clean whitespace and wrap to 64 characters + try: + if value.startswith('-----BEGIN '): + end = value.index('-----', 11) + 5 + header = value[:end] + value = value[end:] + else: + header = '-----BEGIN PRIVATE KEY-----' - # wrap value into valid pem layout and check validity - value = ( - '-----BEGIN PRIVATE KEY-----\n' + - '\n'.join(wrap(value, 64)) + - '\n-----END PRIVATE KEY-----\n' - ).encode('ascii') + if (pos := value.find('-----END ')) >= 0: + end = value.index('-----', pos+9) + 5 + footer = value[pos:end] + value = value[:pos] + else: + footer = '-----END PRIVATE KEY-----' + except ValueError: + raise ValidationError(f'invalid dkim key {bad_key!r}') from exc + + # remove whitespace from key data + value = ''.join(value.split()) + + # remember part of value for ValidationError + bad_key = f'{value[:25]}...{value[-10:]}' if len(value) > 40 else value + + # wrap key according to RFC 7468 + value = ('\n'.join(self._wrap_key(header, value, footer))).encode('ascii') + + # check key validity try: crypto.load_privatekey(crypto.FILETYPE_PEM, value) except crypto.Error as exc: - raise ValidationError(f'invalid dkim key {keydata!r}') from exc + raise ValidationError(f'invalid dkim key {bad_key!r}') from exc else: return value @@ -398,6 +646,27 @@ class PasswordField(fields.Str): ### base schema ### +class Storage: + """ Storage class to save information in context + """ + + context = {} + + def _bind(self, key, bind): + if bind is True: + return (self.__class__, key) + if isinstance(bind, str): + return (get_schema(self.recall(bind).__class__), key) + return (bind, key) + + def store(self, key, value, bind=None): + """ store value under key """ + self.context.setdefault('_track', {})[self._bind(key, bind)]= value + + def recall(self, key, bind=None): + """ recall value from key """ + return self.context['_track'][self._bind(key, bind)] + class BaseOpts(SQLAlchemyAutoSchemaOpts): """ Option class with sqla session """ @@ -408,7 +677,7 @@ class BaseOpts(SQLAlchemyAutoSchemaOpts): meta.sibling = False super(BaseOpts, self).__init__(meta, ordered=ordered) -class BaseSchema(ma.SQLAlchemyAutoSchema): +class BaseSchema(ma.SQLAlchemyAutoSchema, Storage): """ Marshmallow base schema with custom exclude logic and option to hide sqla defaults """ @@ -425,6 +694,9 @@ class BaseSchema(ma.SQLAlchemyAutoSchema): def __init__(self, *args, **kwargs): + # prepare only to auto-include explicitly specified attributes + only = set(kwargs.get('only') or []) + # get context context = kwargs.get('context', {}) flags = {key for key, value in context.items() if value is True} @@ -433,13 +705,13 @@ class BaseSchema(ma.SQLAlchemyAutoSchema): exclude = set(kwargs.get('exclude', [])) # always exclude - exclude.update({'created_at', 'updated_at'}) + exclude.update({'created_at', 'updated_at'} - only) # add include_by_context if context is not None: for need, what in getattr(self.Meta, 'include_by_context', {}).items(): if not flags & set(need): - exclude |= set(what) + exclude |= what - only # update excludes kwargs['exclude'] = exclude @@ -448,12 +720,15 @@ class BaseSchema(ma.SQLAlchemyAutoSchema): super().__init__(*args, **kwargs) # exclude_by_value - self._exclude_by_value = getattr(self.Meta, 'exclude_by_value', {}) + self._exclude_by_value = { + key: values for key, values in getattr(self.Meta, 'exclude_by_value', {}).items() + if key not in only + } # exclude default values if not context.get('full'): for column in self.opts.model.__table__.columns: - if column.name not in exclude: + if column.name not in exclude and column.name not in only: self._exclude_by_value.setdefault(column.name, []).append( None if column.default is None else column.default.arg ) @@ -463,7 +738,7 @@ class BaseSchema(ma.SQLAlchemyAutoSchema): if context is not None: for need, what in getattr(self.Meta, 'hide_by_context', {}).items(): if not flags & set(need): - self._hide_by_context |= set(what) + self._hide_by_context |= what - only # remember primary keys self._primary = str(self.opts.model.__table__.primary_key.columns.values()[0].name) @@ -479,20 +754,13 @@ class BaseSchema(ma.SQLAlchemyAutoSchema): order.remove(self._primary) order.insert(0, self._primary) - # order dump_fields - for field in order: - if field in self.dump_fields: - self.dump_fields[field] = self.dump_fields.pop(field) + # order fieldlists + for fieldlist in (self.fields, self.load_fields, self.dump_fields): + for field in order: + if field in fieldlist: + fieldlist[field] = fieldlist.pop(field) - # move pre_load hook "_track_import" to the front - hooks = self._hooks[('pre_load', False)] - hooks.remove('_track_import') - hooks.insert(0, '_track_import') - # move pre_load hook "_add_instance" to the end - hooks.remove('_add_required') - hooks.append('_add_required') - - # move post_load hook "_add_instance" to the end + # move post_load hook "_add_instance" to the end (after load_instance mixin) hooks = self._hooks[('post_load', False)] hooks.remove('_add_instance') hooks.append('_add_instance') @@ -506,8 +774,8 @@ class BaseSchema(ma.SQLAlchemyAutoSchema): } def _call_and_store(self, *args, **kwargs): - """ track curent parent field for pruning """ - self.context['parent_field'] = kwargs['field_name'] + """ track current parent field for pruning """ + self.store('field', kwargs['field_name'], True) return super()._call_and_store(*args, **kwargs) # this is only needed to work around the declared attr "email" primary key in model @@ -518,11 +786,13 @@ class BaseSchema(ma.SQLAlchemyAutoSchema): if keys := getattr(self.Meta, 'primary_keys', None): filters = {key: data.get(key) for key in keys} if None not in filters.values(): - return self.session.query(self.opts.model).filter_by(**filters).first() - return super().get_instance(data) + res= self.session.query(self.opts.model).filter_by(**filters).first() + return res + res= super().get_instance(data) + return res @pre_load(pass_many=True) - def _patch_input(self, items, many, **kwargs): # pylint: disable=unused-argument + def _patch_many(self, items, many, **kwargs): # pylint: disable=unused-argument """ - flush sqla session before serializing a section when requested (make sure all objects that could be referred to later are created) - when in update mode: patch input data before deserialization @@ -540,12 +810,19 @@ class BaseSchema(ma.SQLAlchemyAutoSchema): # patch "delete", "prune" and "default" want_prune = [] - def patch(count, data, prune): + def patch(count, data): # don't allow __delete__ coming from input if '__delete__' in data: raise ValidationError('Unknown field.', f'{count}.__delete__') + # fail when hash_password is specified without password + if 'hash_password' in data and not 'password' in data: + raise ValidationError( + 'Nothing to hash. Field "password" is missing.', + field_name = f'{count}.hash_password', + ) + # handle "prune list" and "delete item" (-pkey: none and -pkey: id) for key in data: if key.startswith('-'): @@ -553,10 +830,10 @@ class BaseSchema(ma.SQLAlchemyAutoSchema): # delete or prune if data[key] is None: # prune - prune.append(True) + want_prune.append(True) return None # mark item for deletion - return {key[1:]: data[key], '__delete__': True} + return {key[1:]: data[key], '__delete__': count} # handle "set to default value" (-key: none) def set_default(key, value): @@ -567,7 +844,7 @@ class BaseSchema(ma.SQLAlchemyAutoSchema): return (key, None) if value is not None: raise ValidationError( - 'When resetting to default value must be null.', + 'Value must be "null" when resetting to default.', f'{count}.{key}' ) value = self.opts.model.__table__.columns[key].default @@ -583,10 +860,128 @@ class BaseSchema(ma.SQLAlchemyAutoSchema): # convert items to "delete" and filter "prune" item items = [ item for item in [ - patch(count, item, want_prune) for count, item in enumerate(items) + patch(count, item) for count, item in enumerate(items) ] if item ] + # remember if prune was requested for _prune_items@post_load + self.store('prune', bool(want_prune), True) + + # remember original items to stabilize password-changes in _add_instance@post_load + self.store('original', items, True) + + return items + + @pre_load + def _patch_item(self, data, many, **kwargs): # pylint: disable=unused-argument + """ - call callback function to track import + - stabilize import of items with auto-increment primary key + - delete items + - delete/prune list attributes + - add missing required attributes + """ + + # callback + if callback := self.context.get('callback'): + callback(self, data) + + # stop early when not updating + if not self.opts.load_instance or not self.context.get('update'): + return data + + # stabilize import of auto-increment primary keys (not required), + # by matching import data to existing items and setting primary key + if not self._primary in data: + for item in getattr(self.recall('parent'), self.recall('field', 'parent')): + existing = self.dump(item, many=False) + this = existing.pop(self._primary) + if data == existing: + instance = item + data[self._primary] = this + break + + # try to load instance + instance = self.instance or self.get_instance(data) + if instance is None: + + if '__delete__' in data: + # deletion of non-existent item requested + raise ValidationError( + f'Item to delete not found: {data[self._primary]!r}.', + field_name = f'{data["__delete__"]}.{self._primary}', + ) + + else: + + if self.context.get('update'): + # remember instance as parent for pruning siblings + if not self.Meta.sibling: + self.store('parent', instance) + # delete instance from session when marked + if '__delete__' in data: + self.opts.sqla_session.delete(instance) + # delete item from lists or prune lists + # currently: domain.alternatives, user.forward_destination, + # user.manager_of, aliases.destination + for key, value in data.items(): + if not isinstance(self.fields.get(key), ( + RelatedList, CommaSeparatedListField, fields.Raw) + ) or not isinstance(value, list): + continue + # deduplicate new value + new_value = set(value) + # handle list pruning + if '-prune-' in value: + value.remove('-prune-') + new_value.remove('-prune-') + else: + for old in getattr(instance, key): + # using str() is okay for now (see above) + new_value.add(str(old)) + # handle item deletion + for item in value: + if item.startswith('-'): + new_value.remove(item) + try: + new_value.remove(item[1:]) + except KeyError as exc: + raise ValidationError( + f'Item to delete not found: {item[1:]!r}.', + field_name=f'?.{key}', + ) from exc + # sort list of new values + data[key] = sorted(new_value) + # log backref modification not catched by modify hook + if isinstance(self.fields[key], RelatedList): + if callback := self.context.get('callback'): + before = {str(v) for v in getattr(instance, key)} + after = set(data[key]) + if before != after: + callback(self, instance, { + 'key': key, + 'target': str(instance), + 'before': before, + 'after': after, + }) + + # add attributes required for validation from db + for attr_name, field_obj in self.load_fields.items(): + if field_obj.required and attr_name not in data: + data[attr_name] = getattr(instance, attr_name) + + return data + + @post_load(pass_many=True) + def _prune_items(self, items, many, **kwargs): # pylint: disable=unused-argument + """ handle list pruning """ + + # stop early when not updating + if not self.context.get('update'): + return items + + # get prune flag from _patch_many@pre_load + want_prune = self.recall('prune', True) + # prune: determine if existing items in db need to be added or marked for deletion add_items = False del_items = False @@ -603,144 +998,60 @@ class BaseSchema(ma.SQLAlchemyAutoSchema): if add_items or del_items: existing = {item[self._primary] for item in items if self._primary in item} - for item in getattr(self.context['parent'], self.context['parent_field']): + for item in getattr(self.recall('parent'), self.recall('field', 'parent')): key = getattr(item, self._primary) if key not in existing: if add_items: items.append({self._primary: key}) else: - items.append({self._primary: key, '__delete__': True}) + items.append({self._primary: key, '__delete__': '?'}) return items - @pre_load - def _track_import(self, data, many, **kwargs): # pylint: disable=unused-argument - """ call callback function to track import - """ - # callback - if callback := self.context.get('callback'): - callback(self, data) - - return data - - @pre_load - def _add_required(self, data, many, **kwargs): # pylint: disable=unused-argument - """ when updating: - allow modification of existing items having required attributes - by loading existing value from db + @post_load + def _add_instance(self, item, many, **kwargs): # pylint: disable=unused-argument + """ - undo password change in existing instances when plain password did not change + - add new instances to sqla session """ - if not self.opts.load_instance or not self.context.get('update'): - return data - - # stabilize import of auto-increment primary keys (not required), - # by matching import data to existing items and setting primary key - if not self._primary in data: - for item in getattr(self.context['parent'], self.context['parent_field']): - existing = self.dump(item, many=False) - this = existing.pop(self._primary) - if data == existing: - instance = item - data[self._primary] = this - break - - # try to load instance - instance = self.instance or self.get_instance(data) - if instance is None: - - if '__delete__' in data: - # deletion of non-existent item requested - raise ValidationError( - f'item to delete not found: {data[self._primary]!r}', - field_name=f'?.{self._primary}', - ) - - else: - - if self.context.get('update'): - # remember instance as parent for pruning siblings - if not self.Meta.sibling: - self.context['parent'] = instance - # delete instance when marked - if '__delete__' in data: - self.opts.sqla_session.delete(instance) - # delete item from lists or prune lists - # currently: domain.alternatives, user.forward_destination, - # user.manager_of, aliases.destination - for key, value in data.items(): - if not isinstance(self.fields[key], fields.Nested) and isinstance(value, list): - new_value = set(value) - # handle list pruning - if '-prune-' in value: - value.remove('-prune-') - new_value.remove('-prune-') - else: - for old in getattr(instance, key): - # using str() is okay for now (see above) - new_value.add(str(old)) - # handle item deletion - for item in value: - if item.startswith('-'): - new_value.remove(item) - try: - new_value.remove(item[1:]) - except KeyError as exc: - raise ValidationError( - f'item to delete not found: {item[1:]!r}', - field_name=f'?.{key}', - ) from exc - # deduplicate and sort list - data[key] = sorted(new_value) - # log backref modification not catched by hook - if isinstance(self.fields[key], RelatedList): - if callback := self.context.get('callback'): - callback(self, instance, { - 'key': key, - 'target': str(instance), - 'before': [str(v) for v in getattr(instance, key)], - 'after': data[key], - }) - - - - # add attributes required for validation from db - # TODO: this will cause validation errors if value from database does not validate - # but there should not be an invalid value in the database - for attr_name, field_obj in self.load_fields.items(): - if field_obj.required and attr_name not in data: - data[attr_name] = getattr(instance, attr_name) - - return data - - @post_load(pass_original=True) - def _add_instance(self, item, original, many, **kwargs): # pylint: disable=unused-argument - """ add new instances to sqla session """ - - if item in self.opts.sqla_session: - # item was modified - if 'hash_password' in original: - # stabilize import of passwords to be hashed, - # by not re-hashing an unchanged password - if attr := getattr(sqlalchemy.inspect(item).attrs, 'password', None): - if attr.history.has_changes() and attr.history.deleted: - try: - # reset password hash, if password was not changed - inst = type(item)(password=attr.history.deleted[-1]) - if inst.check_password(original['password']): - item.password = inst.password - except ValueError: - # hash in db is invalid - pass - else: - del inst - else: - # new item + if not item in self.opts.sqla_session: self.opts.sqla_session.add(item) + return item + + # stop early if item has no password attribute + if not hasattr(item, 'password'): + return item + + # did we hash a new plaintext password? + original = None + pkey = getattr(item, self._primary) + for data in self.recall('original', True): + if 'hash_password' in data and data.get(self._primary) == pkey: + original = data['password'] + break + if original is None: + # password was hashed by us + return item + + # reset hash if plain password matches hash from db + if attr := getattr(sqlalchemy.inspect(item).attrs, 'password', None): + if attr.history.has_changes() and attr.history.deleted: + try: + # reset password hash + inst = type(item)(password=attr.history.deleted[-1]) + if inst.check_password(original): + item.password = inst.password + except ValueError: + # hash in db is invalid + pass + else: + del inst + return item @post_dump def _hide_values(self, data, many, **kwargs): # pylint: disable=unused-argument - """ hide secrets and order output """ + """ hide secrets """ # stop early when not excluding/hiding if not self._exclude_by_value and not self._hide_by_context: @@ -757,7 +1068,7 @@ class BaseSchema(ma.SQLAlchemyAutoSchema): # this field is used to mark items for deletion mark_delete = fields.Boolean(data_key='__delete__', load_only=True) - # TODO: remove LazyStringField (when model was changed - IMHO comment should not be nullable) + # TODO: this can be removed when comment is not nullable in model comment = LazyStringField() @@ -892,27 +1203,28 @@ class RelaySchema(BaseSchema): load_instance = True -class MailuSchema(Schema): +@mapped +class MailuSchema(Schema, Storage): """ Marshmallow schema for complete Mailu config """ class Meta: """ Schema config """ + model = models.MailuConfig render_module = RenderYAML order = ['domain', 'user', 'alias', 'relay'] # 'config' def __init__(self, *args, **kwargs): super().__init__(*args, **kwargs) - # order dump_fields - for field in self.Meta.order: - if field in self.dump_fields: - self.dump_fields[field] = self.dump_fields.pop(field) + # order fieldlists + for fieldlist in (self.fields, self.load_fields, self.dump_fields): + for field in self.Meta.order: + if field in fieldlist: + fieldlist[field] = fieldlist.pop(field) def _call_and_store(self, *args, **kwargs): """ track current parent and field for pruning """ - self.context.update({ - 'parent': self.context.get('config'), - 'parent_field': kwargs['field_name'], - }) + self.store('field', kwargs['field_name'], True) + self.store('parent', self.context.get('config')) return super()._call_and_store(*args, **kwargs) @pre_load diff --git a/docs/cli.rst b/docs/cli.rst index 6d48c576..891db152 100644 --- a/docs/cli.rst +++ b/docs/cli.rst @@ -138,8 +138,8 @@ The purpose of this command is to export the complete configuration in YAML or J Options: -f, --full Include attributes with default value. -s, --secrets Include secret attributes (dkim-key, passwords). - -c, --color Force colorized output. -d, --dns Include dns records. + -c, --color Force colorized output. -o, --output-file FILENAME Save configuration to file. -j, --json Export configuration in json format. -?, -h, --help Show this message and exit. @@ -147,14 +147,18 @@ The purpose of this command is to export the complete configuration in YAML or J Only non-default attributes are exported. If you want to export all attributes use ``--full``. If you want to export plain-text secrets (dkim-keys, passwords) you have to add the ``--secrets`` option. To include dns records (mx, spf, dkim and dmarc) add the ``--dns`` option. + By default all configuration objects are exported (domain, user, alias, relay). You can specify filters to export only some objects or attributes (try: ``user`` or ``domain.name``). +Attributes explicitly specified in filters are automatically exported: there is no need to add ``--secrets`` or ``--full``. .. code-block:: bash - $ docker-compose exec admin flask mailu config-export -o mail-config.yml + $ docker-compose exec admin flask mailu config-export --output mail-config.yml - $ docker-compose exec admin flask mailu config-export --dns domain.dns_mx domain.dns_spf + $ docker-compose exec admin flask mailu config-export domain.dns_mx domain.dns_spf + + $ docker-compose exec admin flask mailu config-export user.spam_threshold config-import ------------- @@ -211,7 +215,7 @@ mail-config.yml contains the configuration and looks like this: config-update shows the number of created/modified/deleted objects after import. To suppress all messages except error messages use ``--quiet``. -By adding the ``--verbose`` switch (up to two times) the import gets more detailed and shows exactly what attributes changed. +By adding the ``--verbose`` switch the import gets more detailed and shows exactly what attributes changed. In all log messages plain-text secrets (dkim-keys, passwords) are hidden by default. Use ``--secrets`` to log secrets. If you want to test what would be done when importing without committing any changes, use ``--dry-run``. @@ -234,6 +238,9 @@ It is possible to delete a single element or prune all elements from lists and a The ``-key: null`` notation can also be used to reset an attribute to its default. To reset *spam_threshold* to it's default *80* use ``-spam_threshold: null``. +A new dkim key can be generated when adding or modifying a domain, by using the special value +``dkim_key: -generate-``. + This is a complete YAML template with all additional parameters that can be defined: .. code-block:: yaml From b6716f0d74681919384733414b8ca93c0dbf3dca Mon Sep 17 00:00:00 2001 From: Dario Ernst Date: Sat, 20 Feb 2021 13:03:08 +0100 Subject: [PATCH 089/596] Remove "CHUNKING" capability from nginx-smtp MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit With `CHUNKING`set as a capability, nginx advertises this capability to clients at a stage where the SMTP dialog does not seem to be forwarded to the proxy-target (postfix) yet. Nginx' SMTP parser itself does not support the `BDAT` command issued as part of a chunke-d dialog. This makes Nginx respond with a `250 2.0.0 OK` and close the connection, after the mail-data got sent by the client — without forwarding this to the proxy-target. With this, users mail can be lost. Furthermore, when a user uses a sieve filter to forward mail, dovecot sometimes chunks the forwarded mail when sending it through `front`. These forwards then fail. Removing `CHUNKING` from the capabilities fixes this behavior. --- core/nginx/conf/nginx.conf | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/core/nginx/conf/nginx.conf b/core/nginx/conf/nginx.conf index df598c94..f7d9f074 100644 --- a/core/nginx/conf/nginx.conf +++ b/core/nginx/conf/nginx.conf @@ -215,7 +215,7 @@ mail { {% endif %} # Advertise real capabilites of backends (postfix/dovecot) - smtp_capabilities PIPELINING SIZE {{ MESSAGE_SIZE_LIMIT }} ETRN ENHANCEDSTATUSCODES 8BITMIME DSN CHUNKING; + smtp_capabilities PIPELINING SIZE {{ MESSAGE_SIZE_LIMIT }} ETRN ENHANCEDSTATUSCODES 8BITMIME DSN; pop3_capabilities TOP UIDL RESP-CODES PIPELINING AUTH-RESP-CODE USER; imap_capabilities IMAP4 IMAP4rev1 UIDPLUS SASL-IR LOGIN-REFERRALS ID ENABLE IDLE LITERAL+; From 0a9f732faa4d1addcc848cda2678976fb244314c Mon Sep 17 00:00:00 2001 From: Alexander Graf Date: Mon, 22 Feb 2021 20:35:23 +0100 Subject: [PATCH 090/596] added docstring to Logger. use generators. --- core/admin/mailu/models.py | 2 +- core/admin/mailu/schemas.py | 48 +++++++++++++++++++++---------------- 2 files changed, 28 insertions(+), 22 deletions(-) diff --git a/core/admin/mailu/models.py b/core/admin/mailu/models.py index 1b3c787a..d134086f 100644 --- a/core/admin/mailu/models.py +++ b/core/admin/mailu/models.py @@ -83,7 +83,7 @@ class CommaSeparatedList(db.TypeDecorator): def process_result_value(self, value, dialect): """ split comma separated string to list """ - return list(filter(bool, [item.strip() for item in value.split(',')])) if value else [] + return list(filter(bool, (item.strip() for item in value.split(',')))) if value else [] python_type = list diff --git a/core/admin/mailu/schemas.py b/core/admin/mailu/schemas.py index 4c5042ea..68814170 100644 --- a/core/admin/mailu/schemas.py +++ b/core/admin/mailu/schemas.py @@ -76,6 +76,9 @@ class MyYamlLexer(YamlLexer): yield typ, value class Logger: + """ helps with counting and colorizing + imported and exported data + """ def __init__(self, want_color=None, can_color=False, debug=False, secrets=False): @@ -195,25 +198,26 @@ class Logger: continue before = target._dkim_key_on_disk after = target._dkim_key - if before != after: - # "de-dupe" messages; this event is fired at every flush - if not (target, before, after) in self._dedupe_dkim: - self._dedupe_dkim.add((target, before, after)) - self._counter.update([('Modified', target.__table__.name)]) - if self.verbose: - if self.secrets: - before = before.decode('ascii', 'ignore') - after = after.decode('ascii', 'ignore') - else: - before = HIDDEN if before else '' - after = HIDDEN if after else '' - self._log('Modified', target, f'{str(target)!r} dkim_key: {before!r} -> {after!r}') + # "de-dupe" messages; this event is fired at every flush + if before == after or (target, before, after) in self._dedupe_dkim: + continue + self._dedupe_dkim.add((target, before, after)) + self._counter.update([('Modified', target.__table__.name)]) + if self.verbose: + if self.secrets: + before = before.decode('ascii', 'ignore') + after = after.decode('ascii', 'ignore') + else: + before = HIDDEN if before else '' + after = HIDDEN if after else '' + self._log('Modified', target, f'{str(target)!r} dkim_key: {before!r} -> {after!r}') def track_serialize(self, obj, item, backref=None): """ callback method to track import """ # called for backref modification? if backref is not None: - self._log('Modified', item, '{target!r} {key}: {before!r} -> {after!r}'.format_map(backref)) + self._log( + 'Modified', item, '{target!r} {key}: {before!r} -> {after!r}'.format_map(backref)) return # show input data? if self.verbose < 2: @@ -263,7 +267,7 @@ class Logger: if path: return res - maxlen = max([len(loc) for loc, msg in res]) + maxlen = max(len(loc) for loc, msg in res) res = [f' - {loc.ljust(maxlen)} : {msg}' for loc, msg in res] errors = f'{len(res)} error{["s",""][len(res)==1]}' res.insert(0, f'[ValidationError] {errors} occurred during input validation') @@ -279,7 +283,9 @@ class Logger: if 'attr' in trace.tb_frame.f_locals: path.append(trace.tb_frame.f_locals['attr']) elif trace.tb_frame.f_code.co_name == '_init_fields': - spec = ', '.join(['.'.join(path + [key]) for key in trace.tb_frame.f_locals['invalid_fields']]) + spec = ', '.join( + '.'.join(path + [key]) + for key in trace.tb_frame.f_locals['invalid_fields']) return f'Invalid filter: {spec}' trace = trace.tb_next return None @@ -494,7 +500,7 @@ class CommaSeparatedListField(fields.Raw): except UnicodeDecodeError as exc: raise self.make_error("invalid_utf8") from exc else: - value = filter(None, [item.strip() for item in value.split(',')]) + value = filter(bool, (item.strip() for item in value.split(','))) return list(value) @@ -539,7 +545,7 @@ class DkimKeyField(fields.String): # convert list to str if isinstance(value, list): try: - value = ''.join([ensure_text_type(item) for item in value]).strip() + value = ''.join(ensure_text_type(item) for item in value).strip() except UnicodeDecodeError as exc: raise self.make_error("invalid_utf8") from exc @@ -855,7 +861,7 @@ class BaseSchema(ma.SQLAlchemyAutoSchema, Storage): ) return (key, value.arg) - return dict([set_default(key, value) for key, value in data.items()]) + return dict(set_default(key, value) for key, value in data.items()) # convert items to "delete" and filter "prune" item items = [ @@ -1059,11 +1065,11 @@ class BaseSchema(ma.SQLAlchemyAutoSchema, Storage): # exclude or hide values full = self.context.get('full') - return type(data)([ + return type(data)( (key, HIDDEN if key in self._hide_by_context else value) for key, value in data.items() if full or key not in self._exclude_by_value or value not in self._exclude_by_value[key] - ]) + ) # this field is used to mark items for deletion mark_delete = fields.Boolean(data_key='__delete__', load_only=True) From b55b53b781350addad83fe93ac68c5c6c7188847 Mon Sep 17 00:00:00 2001 From: Alexander Graf Date: Fri, 26 Feb 2021 20:51:58 +0100 Subject: [PATCH 091/596] optimize generation of transport nexthop --- core/admin/mailu/internal/views/postfix.py | 48 +++++++++++++++++++--- 1 file changed, 43 insertions(+), 5 deletions(-) diff --git a/core/admin/mailu/internal/views/postfix.py b/core/admin/mailu/internal/views/postfix.py index a5507830..7f8418cf 100644 --- a/core/admin/mailu/internal/views/postfix.py +++ b/core/admin/mailu/internal/views/postfix.py @@ -2,6 +2,7 @@ from mailu import models from mailu.internal import internal import flask +import idna import re import srslib @@ -37,11 +38,48 @@ def postfix_transport(email): return flask.abort(404) localpart, domain_name = models.Email.resolve_domain(email) relay = models.Relay.query.get(domain_name) or flask.abort(404) - ret = "smtp:[{0}]".format(relay.smtp) - if ":" in relay.smtp: - split = relay.smtp.split(':') - ret = "smtp:[{0}]:{1}".format(split[0], split[1]) - return flask.jsonify(ret) + target = relay.smtp.lower() + port = None + if use_mx := target.startswith('mx:'): + target = target[3:] + if target.startswith('['): + if use_mx or ']' not in target: + # invalid target (mx: and []) + flask.abort(400) + host, rest = target[1:].split(']', 1) + if rest.startswith(':'): + port = rest[1:] + elif rest: + # invalid target (rest should be :port) + flask.abort(400) + else: + if ':' in target: + host, port = target.rsplit(':', 1) + else: + host = target + if not host: + host = relay.name.lower() + use_mx = True + if ':' in host: + host = f'ipv6:{host}' + else: + try: + host = idna.encode(host).decode('ascii') + except idna.IDNAError: + # invalid target (fqdn not encodable) + flask.abort(400) + if port is not None: + try: + port = int(port, 10) + if port == 25: + port = None + except ValueError: + # invalid target (port should be numeric) + flask.abort(400) + if not use_mx: + host = f'[{host}]' + port = '' if port is None else f':{port}' + return flask.jsonify(f'smtp:{host}{port}') @internal.route("/postfix/recipient/map/") From 76e5614d13abf71fa39f9bbe0a4c450af897c665 Mon Sep 17 00:00:00 2001 From: Florent Daigniere Date: Sat, 27 Feb 2021 10:37:59 +0100 Subject: [PATCH 092/596] Add mergify to the list of trusted authors The idea is to prevent backports from being stuck pending for review for too long. --- .mergify.yml | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/.mergify.yml b/.mergify.yml index c1141a93..2af387ed 100644 --- a/.mergify.yml +++ b/.mergify.yml @@ -27,7 +27,7 @@ pull_request_rules: - name: Trusted author and 1 approved review; trigger bors r+ conditions: - - author~=^(kaiyou|muhlemmer|mildred|HorayNarea|adi90x|hoellen|ofthesun9|Nebukadneza|micw|lub|Diman0)$ + - author~=^(mergify|kaiyou|muhlemmer|mildred|HorayNarea|adi90x|hoellen|ofthesun9|Nebukadneza|micw|lub|Diman0)$ - -title~=(WIP|wip) - -label~=^(status/wip|status/blocked|review/need2)$ - "#approved-reviews-by>=1" From af251216b0fd9f16b778d7808a1c672eedb1f2b3 Mon Sep 17 00:00:00 2001 From: Jaume Barber Date: Wed, 3 Mar 2021 11:31:51 +0000 Subject: [PATCH 093/596] Translated using Weblate (English) Currently translated at 11.0% (18 of 163 strings) Translation: Mailu/admin Translate-URL: https://translate.tedomum.net/projects/mailu/admin/en/ --- core/admin/mailu/translations/en/LC_MESSAGES/messages.po | 8 ++++---- 1 file changed, 4 insertions(+), 4 deletions(-) diff --git a/core/admin/mailu/translations/en/LC_MESSAGES/messages.po b/core/admin/mailu/translations/en/LC_MESSAGES/messages.po index 2ada20b1..91fee01b 100644 --- a/core/admin/mailu/translations/en/LC_MESSAGES/messages.po +++ b/core/admin/mailu/translations/en/LC_MESSAGES/messages.po @@ -8,8 +8,8 @@ msgstr "" "Project-Id-Version: PROJECT VERSION\n" "Report-Msgid-Bugs-To: EMAIL@ADDRESS\n" "POT-Creation-Date: 2018-04-22 12:10+0200\n" -"PO-Revision-Date: 2020-03-11 23:03+0000\n" -"Last-Translator: Jae Beojkkoch \n" +"PO-Revision-Date: 2021-03-03 11:35+0000\n" +"Last-Translator: Jaume Barber \n" "Language-Team: English \n" "Language: en\n" @@ -17,7 +17,7 @@ msgstr "" "Content-Type: text/plain; charset=utf-8\n" "Content-Transfer-Encoding: 8bit\n" "Plural-Forms: nplurals=2; plural=n != 1;\n" -"X-Generator: Weblate 3.11.2\n" +"X-Generator: Weblate 4.0.1\n" "Generated-By: Babel 2.5.3\n" #: mailu/ui/forms.py:32 @@ -593,7 +593,7 @@ msgstr "" #: mailu/ui/templates/relay/create.html:4 msgid "New relay domain" -msgstr "" +msgstr "New relay domain" #: mailu/ui/templates/relay/edit.html:4 msgid "Edit relayd domain" From 3a9a133226cc67ad3a2c7260763b1fe3c646dea4 Mon Sep 17 00:00:00 2001 From: Anonymous Date: Wed, 3 Mar 2021 11:34:06 +0000 Subject: [PATCH 094/596] Translated using Weblate (English) Currently translated at 11.0% (18 of 163 strings) Translation: Mailu/admin Translate-URL: https://translate.tedomum.net/projects/mailu/admin/en/ --- core/admin/mailu/translations/en/LC_MESSAGES/messages.po | 6 +++--- 1 file changed, 3 insertions(+), 3 deletions(-) diff --git a/core/admin/mailu/translations/en/LC_MESSAGES/messages.po b/core/admin/mailu/translations/en/LC_MESSAGES/messages.po index 91fee01b..bdfe5716 100644 --- a/core/admin/mailu/translations/en/LC_MESSAGES/messages.po +++ b/core/admin/mailu/translations/en/LC_MESSAGES/messages.po @@ -9,7 +9,7 @@ msgstr "" "Report-Msgid-Bugs-To: EMAIL@ADDRESS\n" "POT-Creation-Date: 2018-04-22 12:10+0200\n" "PO-Revision-Date: 2021-03-03 11:35+0000\n" -"Last-Translator: Jaume Barber \n" +"Last-Translator: Anonymous \n" "Language-Team: English \n" "Language: en\n" @@ -105,7 +105,7 @@ msgstr "" #: mailu/ui/forms.py:80 mailu/ui/templates/user/list.html:22 #: mailu/ui/templates/user/signup_domain.html:16 msgid "Quota" -msgstr "" +msgstr "Quota" #: mailu/ui/forms.py:81 msgid "Allow IMAP access" @@ -229,7 +229,7 @@ msgstr "" #: mailu/ui/forms.py:159 mailu/ui/templates/client.html:20 #: mailu/ui/templates/client.html:47 msgid "TCP port" -msgstr "" +msgstr "TCP port" #: mailu/ui/forms.py:160 msgid "Enable TLS" From b9c2dc1a79f33bb4c38aca5b80501f4b74de6a2a Mon Sep 17 00:00:00 2001 From: Jaume Barber Date: Wed, 3 Mar 2021 11:24:19 +0000 Subject: [PATCH 095/596] Translated using Weblate (Catalan) Currently translated at 98.6% (149 of 151 strings) Translation: Mailu/admin Translate-URL: https://translate.tedomum.net/projects/mailu/admin/ca/ --- core/admin/mailu/translations/ca/LC_MESSAGES/messages.po | 6 +++--- 1 file changed, 3 insertions(+), 3 deletions(-) diff --git a/core/admin/mailu/translations/ca/LC_MESSAGES/messages.po b/core/admin/mailu/translations/ca/LC_MESSAGES/messages.po index f63b7083..76594a3c 100644 --- a/core/admin/mailu/translations/ca/LC_MESSAGES/messages.po +++ b/core/admin/mailu/translations/ca/LC_MESSAGES/messages.po @@ -8,7 +8,7 @@ msgstr "" "Project-Id-Version: PROJECT VERSION\n" "Report-Msgid-Bugs-To: EMAIL@ADDRESS\n" "POT-Creation-Date: 2018-04-22 12:10+0200\n" -"PO-Revision-Date: 2020-04-26 13:09+0000\n" +"PO-Revision-Date: 2021-03-03 11:35+0000\n" "Last-Translator: Jaume Barber \n" "Language-Team: Catalan \n" @@ -304,7 +304,7 @@ msgstr "Resposta automàtica" #: mailu/ui/templates/fetch/list.html:4 mailu/ui/templates/sidebar.html:26 #: mailu/ui/templates/user/list.html:36 msgid "Fetched accounts" -msgstr "Comptes trobats" +msgstr "Comptes vinculats" #: mailu/ui/templates/sidebar.html:31 mailu/ui/templates/token/list.html:4 msgid "Authentication tokens" @@ -324,7 +324,7 @@ msgstr "Administradors" #: mailu/ui/templates/sidebar.html:54 msgid "Relayed domains" -msgstr "Dominis tramesos" +msgstr "Dominis delegats" #: mailu/ui/templates/sidebar.html:59 mailu/ui/templates/user/settings.html:15 msgid "Antispam" From 725cdc270c0cce7f1e9ea5c3a14a4f7c5d22f39d Mon Sep 17 00:00:00 2001 From: Jaume Barber Date: Wed, 3 Mar 2021 11:37:34 +0000 Subject: [PATCH 096/596] Translated using Weblate (Spanish) Currently translated at 100.0% (163 of 163 strings) Translation: Mailu/admin Translate-URL: https://translate.tedomum.net/projects/mailu/admin/es/ --- core/admin/mailu/translations/es/LC_MESSAGES/messages.po | 8 ++++---- 1 file changed, 4 insertions(+), 4 deletions(-) diff --git a/core/admin/mailu/translations/es/LC_MESSAGES/messages.po b/core/admin/mailu/translations/es/LC_MESSAGES/messages.po index 94b39439..ff6b9f36 100644 --- a/core/admin/mailu/translations/es/LC_MESSAGES/messages.po +++ b/core/admin/mailu/translations/es/LC_MESSAGES/messages.po @@ -1,7 +1,7 @@ msgid "" msgstr "" "Project-Id-Version: Mailu\n" -"PO-Revision-Date: 2020-03-11 23:03+0000\n" +"PO-Revision-Date: 2021-03-03 12:37+0000\n" "Last-Translator: Jaume Barber \n" "Language-Team: Spanish \n" @@ -10,7 +10,7 @@ msgstr "" "Content-Type: text/plain; charset=UTF-8\n" "Content-Transfer-Encoding: 8bit\n" "Plural-Forms: nplurals=2; plural=n != 1;\n" -"X-Generator: Weblate 3.11.2\n" +"X-Generator: Weblate 4.0.1\n" #: mailu/ui/forms.py:32 msgid "Invalid email address." @@ -425,7 +425,7 @@ msgstr "Añadir una cuenta" #: mailu/ui/templates/fetch/list.html:19 msgid "Endpoint" -msgstr "Punto final" +msgstr "Endpoint" #: mailu/ui/templates/fetch/list.html:22 msgid "Last check" @@ -437,7 +437,7 @@ msgstr "Añadir un gestor" #: mailu/ui/templates/manager/list.html:4 msgid "Manager list" -msgstr "Gestor de lista" +msgstr "Lista de gestores" #: mailu/ui/templates/manager/list.html:12 msgid "Add manager" From 5e0aa65c8d29d6fc16d463fe4e34fa777b552a69 Mon Sep 17 00:00:00 2001 From: Jaume Barber Date: Wed, 3 Mar 2021 17:02:15 +0000 Subject: [PATCH 097/596] Translated using Weblate (Italian) Currently translated at 96.3% (157 of 163 strings) Translation: Mailu/admin Translate-URL: https://translate.tedomum.net/projects/mailu/admin/it/ --- core/admin/mailu/translations/it/LC_MESSAGES/messages.po | 5 +++-- 1 file changed, 3 insertions(+), 2 deletions(-) diff --git a/core/admin/mailu/translations/it/LC_MESSAGES/messages.po b/core/admin/mailu/translations/it/LC_MESSAGES/messages.po index 9ef5ac84..6ec219bf 100644 --- a/core/admin/mailu/translations/it/LC_MESSAGES/messages.po +++ b/core/admin/mailu/translations/it/LC_MESSAGES/messages.po @@ -1,7 +1,7 @@ msgid "" msgstr "" "Project-Id-Version: Mailu\n" -"PO-Revision-Date: 2020-03-11 23:03+0000\n" +"PO-Revision-Date: 2021-03-03 17:03+0000\n" "Last-Translator: Jaume Barber \n" "Language-Team: Italian \n" @@ -10,7 +10,7 @@ msgstr "" "Content-Type: text/plain; charset=UTF-8\n" "Content-Transfer-Encoding: 8bit\n" "Plural-Forms: nplurals=2; plural=n != 1;\n" -"X-Generator: Weblate 3.11.2\n" +"X-Generator: Weblate 4.0.1\n" #: mailu/ui/forms.py:32 msgid "Invalid email address." @@ -576,6 +576,7 @@ msgid "Relayed domain list" msgstr "Elenco di domini affidati" #: mailu/ui/templates/relay/list.html:9 +#, fuzzy msgid "New relayed domain" msgstr "Nuovo dominio affidato" From 43133d85154eae91c662b8b756dc3e89d26ac7cd Mon Sep 17 00:00:00 2001 From: Jaume Barber Date: Wed, 3 Mar 2021 17:05:23 +0000 Subject: [PATCH 098/596] Added translation using Weblate (Basque) --- .../translations/eu/LC_MESSAGES/messages.po | 669 ++++++++++++++++++ 1 file changed, 669 insertions(+) create mode 100644 core/admin/mailu/translations/eu/LC_MESSAGES/messages.po diff --git a/core/admin/mailu/translations/eu/LC_MESSAGES/messages.po b/core/admin/mailu/translations/eu/LC_MESSAGES/messages.po new file mode 100644 index 00000000..3a72c9af --- /dev/null +++ b/core/admin/mailu/translations/eu/LC_MESSAGES/messages.po @@ -0,0 +1,669 @@ +# Translations template for PROJECT. +# Copyright (C) 2018 ORGANIZATION +# This file is distributed under the same license as the PROJECT project. +# FIRST AUTHOR , 2018. +# +msgid "" +msgstr "" +"Project-Id-Version: PROJECT VERSION\n" +"Report-Msgid-Bugs-To: EMAIL@ADDRESS\n" +"POT-Creation-Date: 2018-04-22 12:10+0200\n" +"PO-Revision-Date: YEAR-MO-DA HO:MI+ZONE\n" +"Last-Translator: Automatically generated\n" +"Language-Team: none\n" +"Language: eu\n" +"MIME-Version: 1.0\n" +"Content-Type: text/plain; charset=utf-8\n" +"Content-Transfer-Encoding: 8bit\n" +"Generated-By: Babel 2.5.3\n" + +#: mailu/ui/forms.py:32 +msgid "Invalid email address." +msgstr "" + +#: mailu/ui/forms.py:36 +msgid "Confirm" +msgstr "" + +#: mailu/ui/forms.py:40 mailu/ui/forms.py:77 +msgid "E-mail" +msgstr "" + +#: mailu/ui/forms.py:41 mailu/ui/forms.py:78 mailu/ui/forms.py:90 +#: mailu/ui/forms.py:109 mailu/ui/forms.py:162 +#: mailu/ui/templates/client.html:32 mailu/ui/templates/client.html:59 +msgid "Password" +msgstr "" + +#: mailu/ui/forms.py:42 mailu/ui/templates/login.html:4 +#: mailu/ui/templates/sidebar.html:111 +msgid "Sign in" +msgstr "" + +#: mailu/ui/forms.py:46 mailu/ui/forms.py:56 +#: mailu/ui/templates/domain/details.html:27 +#: mailu/ui/templates/domain/list.html:18 mailu/ui/templates/relay/list.html:17 +msgid "Domain name" +msgstr "" + +#: mailu/ui/forms.py:47 +msgid "Maximum user count" +msgstr "" + +#: mailu/ui/forms.py:48 +msgid "Maximum alias count" +msgstr "" + +#: mailu/ui/forms.py:49 +msgid "Maximum user quota" +msgstr "" + +#: mailu/ui/forms.py:50 +msgid "Enable sign-up" +msgstr "" + +#: mailu/ui/forms.py:51 mailu/ui/forms.py:72 mailu/ui/forms.py:83 +#: mailu/ui/forms.py:128 mailu/ui/forms.py:140 +#: mailu/ui/templates/alias/list.html:21 mailu/ui/templates/domain/list.html:21 +#: mailu/ui/templates/relay/list.html:19 mailu/ui/templates/token/list.html:19 +#: mailu/ui/templates/user/list.html:23 +msgid "Comment" +msgstr "" + +#: mailu/ui/forms.py:52 mailu/ui/forms.py:61 mailu/ui/forms.py:66 +#: mailu/ui/forms.py:73 mailu/ui/forms.py:132 mailu/ui/forms.py:141 +msgid "Create" +msgstr "" + +#: mailu/ui/forms.py:57 +msgid "Initial admin" +msgstr "" + +#: mailu/ui/forms.py:58 +msgid "Admin password" +msgstr "" + +#: mailu/ui/forms.py:59 mailu/ui/forms.py:79 mailu/ui/forms.py:91 +msgid "Confirm password" +msgstr "" + +#: mailu/ui/forms.py:65 +msgid "Alternative name" +msgstr "" + +#: mailu/ui/forms.py:70 +msgid "Relayed domain name" +msgstr "" + +#: mailu/ui/forms.py:71 mailu/ui/templates/relay/list.html:18 +msgid "Remote host" +msgstr "" + +#: mailu/ui/forms.py:80 mailu/ui/templates/user/list.html:22 +#: mailu/ui/templates/user/signup_domain.html:16 +msgid "Quota" +msgstr "" + +#: mailu/ui/forms.py:81 +msgid "Allow IMAP access" +msgstr "" + +#: mailu/ui/forms.py:82 +msgid "Allow POP3 access" +msgstr "" + +#: mailu/ui/forms.py:84 +msgid "Enabled" +msgstr "" + +#: mailu/ui/forms.py:85 +msgid "Save" +msgstr "" + +#: mailu/ui/forms.py:89 +msgid "Email address" +msgstr "" + +#: mailu/ui/forms.py:93 mailu/ui/templates/sidebar.html:117 +#: mailu/ui/templates/user/signup.html:4 +#: mailu/ui/templates/user/signup_domain.html:4 +msgid "Sign up" +msgstr "" + +#: mailu/ui/forms.py:97 +msgid "Displayed name" +msgstr "" + +#: mailu/ui/forms.py:98 +msgid "Enable spam filter" +msgstr "" + +#: mailu/ui/forms.py:99 +msgid "Spam filter tolerance" +msgstr "" + +#: mailu/ui/forms.py:100 +msgid "Enable forwarding" +msgstr "" + +#: mailu/ui/forms.py:101 +msgid "Keep a copy of the emails" +msgstr "" + +#: mailu/ui/forms.py:103 mailu/ui/forms.py:139 +#: mailu/ui/templates/alias/list.html:20 +msgid "Destination" +msgstr "" + +#: mailu/ui/forms.py:105 +msgid "Save settings" +msgstr "" + +#: mailu/ui/forms.py:110 +msgid "Password check" +msgstr "" + +#: mailu/ui/forms.py:111 mailu/ui/templates/sidebar.html:16 +msgid "Update password" +msgstr "" + +#: mailu/ui/forms.py:115 +msgid "Enable automatic reply" +msgstr "" + +#: mailu/ui/forms.py:116 +msgid "Reply subject" +msgstr "" + +#: mailu/ui/forms.py:117 +msgid "Reply body" +msgstr "" + +#: mailu/ui/forms.py:119 +msgid "End of vacation" +msgstr "" + +#: mailu/ui/forms.py:120 +msgid "Update" +msgstr "" + +#: mailu/ui/forms.py:125 +msgid "Your token (write it down, as it will never be displayed again)" +msgstr "" + +#: mailu/ui/forms.py:130 mailu/ui/templates/token/list.html:20 +msgid "Authorized IP" +msgstr "" + +#: mailu/ui/forms.py:136 +msgid "Alias" +msgstr "" + +#: mailu/ui/forms.py:138 +msgid "Use SQL LIKE Syntax (e.g. for catch-all aliases)" +msgstr "" + +#: mailu/ui/forms.py:145 +msgid "Admin email" +msgstr "" + +#: mailu/ui/forms.py:146 mailu/ui/forms.py:151 mailu/ui/forms.py:164 +msgid "Submit" +msgstr "" + +#: mailu/ui/forms.py:150 +msgid "Manager email" +msgstr "" + +#: mailu/ui/forms.py:155 +msgid "Protocol" +msgstr "" + +#: mailu/ui/forms.py:158 +msgid "Hostname or IP" +msgstr "" + +#: mailu/ui/forms.py:159 mailu/ui/templates/client.html:20 +#: mailu/ui/templates/client.html:47 +msgid "TCP port" +msgstr "" + +#: mailu/ui/forms.py:160 +msgid "Enable TLS" +msgstr "" + +#: mailu/ui/forms.py:161 mailu/ui/templates/client.html:28 +#: mailu/ui/templates/client.html:55 mailu/ui/templates/fetch/list.html:20 +msgid "Username" +msgstr "" + +#: mailu/ui/forms.py:163 +msgid "Keep emails on the server" +msgstr "" + +#: mailu/ui/forms.py:168 +msgid "Announcement subject" +msgstr "" + +#: mailu/ui/forms.py:170 +msgid "Announcement body" +msgstr "" + +#: mailu/ui/forms.py:172 +msgid "Send" +msgstr "" + +#: mailu/ui/templates/announcement.html:4 +msgid "Public announcement" +msgstr "" + +#: mailu/ui/templates/client.html:4 mailu/ui/templates/sidebar.html:82 +msgid "Client setup" +msgstr "" + +#: mailu/ui/templates/client.html:16 mailu/ui/templates/client.html:43 +msgid "Mail protocol" +msgstr "" + +#: mailu/ui/templates/client.html:24 mailu/ui/templates/client.html:51 +msgid "Server name" +msgstr "" + +#: mailu/ui/templates/confirm.html:4 +msgid "Confirm action" +msgstr "" + +#: mailu/ui/templates/confirm.html:13 +#, python-format +msgid "You are about to %(action)s. Please confirm your action." +msgstr "" + +#: mailu/ui/templates/docker-error.html:4 +msgid "Docker error" +msgstr "" + +#: mailu/ui/templates/docker-error.html:12 +msgid "An error occurred while talking to the Docker server." +msgstr "" + +#: mailu/ui/templates/login.html:8 +msgid "to access the administration tools" +msgstr "" + +#: mailu/ui/templates/sidebar.html:11 mailu/ui/templates/user/list.html:34 +msgid "Settings" +msgstr "" + +#: mailu/ui/templates/sidebar.html:21 mailu/ui/templates/user/list.html:35 +msgid "Auto-reply" +msgstr "" + +#: mailu/ui/templates/fetch/list.html:4 mailu/ui/templates/sidebar.html:26 +#: mailu/ui/templates/user/list.html:36 +msgid "Fetched accounts" +msgstr "" + +#: mailu/ui/templates/sidebar.html:31 mailu/ui/templates/token/list.html:4 +msgid "Authentication tokens" +msgstr "" + +#: mailu/ui/templates/sidebar.html:35 +msgid "Administration" +msgstr "" + +#: mailu/ui/templates/sidebar.html:44 +msgid "Announcement" +msgstr "" + +#: mailu/ui/templates/sidebar.html:49 +msgid "Administrators" +msgstr "" + +#: mailu/ui/templates/sidebar.html:54 +msgid "Relayed domains" +msgstr "" + +#: mailu/ui/templates/sidebar.html:59 mailu/ui/templates/user/settings.html:15 +msgid "Antispam" +msgstr "" + +#: mailu/ui/templates/sidebar.html:66 +msgid "Mail domains" +msgstr "" + +#: mailu/ui/templates/sidebar.html:72 +msgid "Go to" +msgstr "" + +#: mailu/ui/templates/sidebar.html:76 +msgid "Webmail" +msgstr "" + +#: mailu/ui/templates/sidebar.html:87 +msgid "Website" +msgstr "" + +#: mailu/ui/templates/sidebar.html:92 +msgid "Help" +msgstr "" + +#: mailu/ui/templates/domain/signup.html:4 mailu/ui/templates/sidebar.html:98 +msgid "Register a domain" +msgstr "" + +#: mailu/ui/templates/sidebar.html:105 +msgid "Sign out" +msgstr "" + +#: mailu/ui/templates/working.html:4 +msgid "We are still working on this feature!" +msgstr "" + +#: mailu/ui/templates/admin/create.html:4 +msgid "Add a global administrator" +msgstr "" + +#: mailu/ui/templates/admin/list.html:4 +msgid "Global administrators" +msgstr "" + +#: mailu/ui/templates/admin/list.html:9 +msgid "Add administrator" +msgstr "" + +#: mailu/ui/templates/admin/list.html:16 mailu/ui/templates/alias/list.html:18 +#: mailu/ui/templates/alternative/list.html:18 +#: mailu/ui/templates/domain/list.html:16 mailu/ui/templates/fetch/list.html:18 +#: mailu/ui/templates/manager/list.html:18 +#: mailu/ui/templates/relay/list.html:16 mailu/ui/templates/token/list.html:18 +#: mailu/ui/templates/user/list.html:18 +msgid "Actions" +msgstr "" + +#: mailu/ui/templates/admin/list.html:17 mailu/ui/templates/alias/list.html:19 +#: mailu/ui/templates/manager/list.html:19 mailu/ui/templates/user/list.html:20 +msgid "Email" +msgstr "" + +#: mailu/ui/templates/admin/list.html:22 mailu/ui/templates/alias/list.html:29 +#: mailu/ui/templates/alternative/list.html:25 +#: mailu/ui/templates/domain/list.html:31 mailu/ui/templates/fetch/list.html:31 +#: mailu/ui/templates/manager/list.html:24 +#: mailu/ui/templates/relay/list.html:27 mailu/ui/templates/token/list.html:26 +#: mailu/ui/templates/user/list.html:31 +msgid "Delete" +msgstr "" + +#: mailu/ui/templates/alias/create.html:4 +msgid "Create alias" +msgstr "" + +#: mailu/ui/templates/alias/edit.html:4 +msgid "Edit alias" +msgstr "" + +#: mailu/ui/templates/alias/list.html:4 +msgid "Alias list" +msgstr "" + +#: mailu/ui/templates/alias/list.html:12 +msgid "Add alias" +msgstr "" + +#: mailu/ui/templates/alias/list.html:22 +#: mailu/ui/templates/alternative/list.html:20 +#: mailu/ui/templates/domain/list.html:22 mailu/ui/templates/fetch/list.html:24 +#: mailu/ui/templates/relay/list.html:20 mailu/ui/templates/token/list.html:21 +#: mailu/ui/templates/user/list.html:24 +msgid "Created" +msgstr "" + +#: mailu/ui/templates/alias/list.html:23 mailu/ui/templates/domain/list.html:23 +#: mailu/ui/templates/fetch/list.html:25 mailu/ui/templates/relay/list.html:21 +#: mailu/ui/templates/user/list.html:25 +msgid "Last edit" +msgstr "" + +#: mailu/ui/templates/alias/list.html:28 mailu/ui/templates/domain/list.html:30 +#: mailu/ui/templates/fetch/list.html:30 mailu/ui/templates/relay/list.html:26 +#: mailu/ui/templates/user/list.html:30 +msgid "Edit" +msgstr "" + +#: mailu/ui/templates/alternative/create.html:4 +msgid "Create alternative domain" +msgstr "" + +#: mailu/ui/templates/alternative/list.html:4 +msgid "Alternative domain list" +msgstr "" + +#: mailu/ui/templates/alternative/list.html:12 +msgid "Add alternative" +msgstr "" + +#: mailu/ui/templates/alternative/list.html:19 +msgid "Name" +msgstr "" + +#: mailu/ui/templates/domain/create.html:4 +#: mailu/ui/templates/domain/list.html:9 +msgid "New domain" +msgstr "" + +#: mailu/ui/templates/domain/details.html:4 +msgid "Domain details" +msgstr "" + +#: mailu/ui/templates/domain/details.html:15 +msgid "Regenerate keys" +msgstr "" + +#: mailu/ui/templates/domain/details.html:17 +msgid "Generate keys" +msgstr "" + +#: mailu/ui/templates/domain/details.html:31 +msgid "DNS MX entry" +msgstr "" + +#: mailu/ui/templates/domain/details.html:35 +msgid "DNS SPF entries" +msgstr "" + +#: mailu/ui/templates/domain/details.html:42 +msgid "DKIM public key" +msgstr "" + +#: mailu/ui/templates/domain/details.html:46 +msgid "DNS DKIM entry" +msgstr "" + +#: mailu/ui/templates/domain/details.html:50 +msgid "DNS DMARC entry" +msgstr "" + +#: mailu/ui/templates/domain/edit.html:4 +msgid "Edit domain" +msgstr "" + +#: mailu/ui/templates/domain/list.html:4 +msgid "Domain list" +msgstr "" + +#: mailu/ui/templates/domain/list.html:17 +msgid "Manage" +msgstr "" + +#: mailu/ui/templates/domain/list.html:19 +msgid "Mailbox count" +msgstr "" + +#: mailu/ui/templates/domain/list.html:20 +msgid "Alias count" +msgstr "" + +#: mailu/ui/templates/domain/list.html:28 +msgid "Details" +msgstr "" + +#: mailu/ui/templates/domain/list.html:35 +msgid "Users" +msgstr "" + +#: mailu/ui/templates/domain/list.html:36 +msgid "Aliases" +msgstr "" + +#: mailu/ui/templates/domain/list.html:37 +msgid "Managers" +msgstr "" + +#: mailu/ui/templates/domain/list.html:39 +msgid "Alternatives" +msgstr "" + +#: mailu/ui/templates/domain/signup.html:13 +msgid "" +"In order to register a new domain, you must first setup the\n" +" domain zone so that the domain MX points to this server" +msgstr "" + +#: mailu/ui/templates/domain/signup.html:18 +msgid "" +"If you do not know how to setup an MX record for your DNS " +"zone,\n" +" please contact your DNS provider or administrator. Also, please wait " +"a\n" +" couple minutes after the MX is set so the local server " +"cache\n" +" expires." +msgstr "" + +#: mailu/ui/templates/fetch/create.html:4 +msgid "Add a fetched account" +msgstr "" + +#: mailu/ui/templates/fetch/edit.html:4 +msgid "Update a fetched account" +msgstr "" + +#: mailu/ui/templates/fetch/list.html:12 +msgid "Add an account" +msgstr "" + +#: mailu/ui/templates/fetch/list.html:19 +msgid "Endpoint" +msgstr "" + +#: mailu/ui/templates/fetch/list.html:21 +msgid "Keep emails" +msgstr "" + +#: mailu/ui/templates/fetch/list.html:22 +msgid "Last check" +msgstr "" + +#: mailu/ui/templates/fetch/list.html:35 +msgid "yes" +msgstr "" + +#: mailu/ui/templates/fetch/list.html:35 +msgid "no" +msgstr "" + +#: mailu/ui/templates/manager/create.html:4 +msgid "Add a manager" +msgstr "" + +#: mailu/ui/templates/manager/list.html:4 +msgid "Manager list" +msgstr "" + +#: mailu/ui/templates/manager/list.html:12 +msgid "Add manager" +msgstr "" + +#: mailu/ui/templates/relay/create.html:4 +msgid "New relay domain" +msgstr "" + +#: mailu/ui/templates/relay/edit.html:4 +msgid "Edit relayd domain" +msgstr "" + +#: mailu/ui/templates/relay/list.html:4 +msgid "Relayed domain list" +msgstr "" + +#: mailu/ui/templates/relay/list.html:9 +msgid "New relayed domain" +msgstr "" + +#: mailu/ui/templates/token/create.html:4 +msgid "Create an authentication token" +msgstr "" + +#: mailu/ui/templates/token/list.html:12 +msgid "New token" +msgstr "" + +#: mailu/ui/templates/user/create.html:4 +msgid "New user" +msgstr "" + +#: mailu/ui/templates/user/create.html:15 +msgid "General" +msgstr "" + +#: mailu/ui/templates/user/create.html:22 +msgid "Features and quotas" +msgstr "" + +#: mailu/ui/templates/user/edit.html:4 +msgid "Edit user" +msgstr "" + +#: mailu/ui/templates/user/forward.html:4 +msgid "Forward emails" +msgstr "" + +#: mailu/ui/templates/user/list.html:4 +msgid "User list" +msgstr "" + +#: mailu/ui/templates/user/list.html:12 +msgid "Add user" +msgstr "" + +#: mailu/ui/templates/user/list.html:19 mailu/ui/templates/user/settings.html:4 +msgid "User settings" +msgstr "" + +#: mailu/ui/templates/user/list.html:21 +msgid "Features" +msgstr "" + +#: mailu/ui/templates/user/password.html:4 +msgid "Password update" +msgstr "" + +#: mailu/ui/templates/user/reply.html:4 +msgid "Automatic reply" +msgstr "" + +#: mailu/ui/templates/user/settings.html:22 +msgid "Auto-forward" +msgstr "" + +#: mailu/ui/templates/user/signup_domain.html:8 +msgid "pick a domain for the new account" +msgstr "" + +#: mailu/ui/templates/user/signup_domain.html:14 +msgid "Domain" +msgstr "" + +#: mailu/ui/templates/user/signup_domain.html:15 +msgid "Available slots" +msgstr "" From 7de94275a0c01f9b11bcf533b42a7e7148b6b4f2 Mon Sep 17 00:00:00 2001 From: Anonymous Date: Wed, 3 Mar 2021 11:35:58 +0000 Subject: [PATCH 099/596] Translated using Weblate (English) Currently translated at 17.7% (29 of 163 strings) Translation: Mailu/admin Translate-URL: https://translate.tedomum.net/projects/mailu/admin/en/ --- .../mailu/translations/en/LC_MESSAGES/messages.po | 10 +++++----- 1 file changed, 5 insertions(+), 5 deletions(-) diff --git a/core/admin/mailu/translations/en/LC_MESSAGES/messages.po b/core/admin/mailu/translations/en/LC_MESSAGES/messages.po index bdfe5716..0b050560 100644 --- a/core/admin/mailu/translations/en/LC_MESSAGES/messages.po +++ b/core/admin/mailu/translations/en/LC_MESSAGES/messages.po @@ -8,7 +8,7 @@ msgstr "" "Project-Id-Version: PROJECT VERSION\n" "Report-Msgid-Bugs-To: EMAIL@ADDRESS\n" "POT-Creation-Date: 2018-04-22 12:10+0200\n" -"PO-Revision-Date: 2021-03-03 11:35+0000\n" +"PO-Revision-Date: 2021-03-03 17:17+0000\n" "Last-Translator: Anonymous \n" "Language-Team: English \n" @@ -30,13 +30,13 @@ msgstr "Confirm" #: mailu/ui/forms.py:40 mailu/ui/forms.py:77 msgid "E-mail" -msgstr "" +msgstr "E-mail" #: mailu/ui/forms.py:41 mailu/ui/forms.py:78 mailu/ui/forms.py:90 #: mailu/ui/forms.py:109 mailu/ui/forms.py:162 #: mailu/ui/templates/client.html:32 mailu/ui/templates/client.html:59 msgid "Password" -msgstr "" +msgstr "Password" #: mailu/ui/forms.py:42 mailu/ui/templates/login.html:4 #: mailu/ui/templates/sidebar.html:111 @@ -283,7 +283,7 @@ msgstr "" #: mailu/ui/templates/docker-error.html:4 msgid "Docker error" -msgstr "" +msgstr "Docker error" #: mailu/ui/templates/docker-error.html:12 msgid "An error occurred while talking to the Docker server." @@ -669,7 +669,7 @@ msgstr "" #: mailu/ui/templates/user/signup_domain.html:14 msgid "Domain" -msgstr "" +msgstr "Domain" #: mailu/ui/templates/user/signup_domain.html:15 msgid "Available slots" From 7c0158c5f8fc64ca4dc913229e81e51367283ca7 Mon Sep 17 00:00:00 2001 From: Jaume Barber Date: Wed, 3 Mar 2021 11:36:15 +0000 Subject: [PATCH 100/596] Translated using Weblate (English) Currently translated at 17.7% (29 of 163 strings) Translation: Mailu/admin Translate-URL: https://translate.tedomum.net/projects/mailu/admin/en/ --- core/admin/mailu/translations/en/LC_MESSAGES/messages.po | 8 ++++---- 1 file changed, 4 insertions(+), 4 deletions(-) diff --git a/core/admin/mailu/translations/en/LC_MESSAGES/messages.po b/core/admin/mailu/translations/en/LC_MESSAGES/messages.po index 0b050560..a09ac1c7 100644 --- a/core/admin/mailu/translations/en/LC_MESSAGES/messages.po +++ b/core/admin/mailu/translations/en/LC_MESSAGES/messages.po @@ -9,7 +9,7 @@ msgstr "" "Report-Msgid-Bugs-To: EMAIL@ADDRESS\n" "POT-Creation-Date: 2018-04-22 12:10+0200\n" "PO-Revision-Date: 2021-03-03 17:17+0000\n" -"Last-Translator: Anonymous \n" +"Last-Translator: Jaume Barber \n" "Language-Team: English \n" "Language: en\n" @@ -328,7 +328,7 @@ msgstr "" #: mailu/ui/templates/sidebar.html:54 msgid "Relayed domains" -msgstr "" +msgstr "Relayed domains" #: mailu/ui/templates/sidebar.html:59 mailu/ui/templates/user/settings.html:15 msgid "Antispam" @@ -601,11 +601,11 @@ msgstr "" #: mailu/ui/templates/relay/list.html:4 msgid "Relayed domain list" -msgstr "" +msgstr "Relayed domain list" #: mailu/ui/templates/relay/list.html:9 msgid "New relayed domain" -msgstr "" +msgstr "New relayed domain" #: mailu/ui/templates/token/create.html:4 msgid "Create an authentication token" From a2933d00f3e9ab6834de095f9a6a5975be9c4bce Mon Sep 17 00:00:00 2001 From: Jaume Barber Date: Wed, 3 Mar 2021 17:17:19 +0000 Subject: [PATCH 101/596] Translated using Weblate (English) Currently translated at 29.4% (48 of 163 strings) Translation: Mailu/admin Translate-URL: https://translate.tedomum.net/projects/mailu/admin/en/ --- core/admin/mailu/translations/en/LC_MESSAGES/messages.po | 6 +++--- 1 file changed, 3 insertions(+), 3 deletions(-) diff --git a/core/admin/mailu/translations/en/LC_MESSAGES/messages.po b/core/admin/mailu/translations/en/LC_MESSAGES/messages.po index a09ac1c7..68f0177d 100644 --- a/core/admin/mailu/translations/en/LC_MESSAGES/messages.po +++ b/core/admin/mailu/translations/en/LC_MESSAGES/messages.po @@ -8,7 +8,7 @@ msgstr "" "Project-Id-Version: PROJECT VERSION\n" "Report-Msgid-Bugs-To: EMAIL@ADDRESS\n" "POT-Creation-Date: 2018-04-22 12:10+0200\n" -"PO-Revision-Date: 2021-03-03 17:17+0000\n" +"PO-Revision-Date: 2021-03-03 17:57+0000\n" "Last-Translator: Jaume Barber \n" "Language-Team: English \n" @@ -51,7 +51,7 @@ msgstr "" #: mailu/ui/forms.py:47 msgid "Maximum user count" -msgstr "" +msgstr "Maximum user count" #: mailu/ui/forms.py:48 msgid "Maximum alias count" @@ -332,7 +332,7 @@ msgstr "Relayed domains" #: mailu/ui/templates/sidebar.html:59 mailu/ui/templates/user/settings.html:15 msgid "Antispam" -msgstr "" +msgstr "Antispam" #: mailu/ui/templates/sidebar.html:66 msgid "Mail domains" From 3d17000ceb419014c9a7fb20fb90e4f9257cc33e Mon Sep 17 00:00:00 2001 From: Anonymous Date: Wed, 3 Mar 2021 17:20:37 +0000 Subject: [PATCH 102/596] Translated using Weblate (English) Currently translated at 29.4% (48 of 163 strings) Translation: Mailu/admin Translate-URL: https://translate.tedomum.net/projects/mailu/admin/en/ --- .../translations/en/LC_MESSAGES/messages.po | 20 +++++++++---------- 1 file changed, 10 insertions(+), 10 deletions(-) diff --git a/core/admin/mailu/translations/en/LC_MESSAGES/messages.po b/core/admin/mailu/translations/en/LC_MESSAGES/messages.po index 68f0177d..026f47c6 100644 --- a/core/admin/mailu/translations/en/LC_MESSAGES/messages.po +++ b/core/admin/mailu/translations/en/LC_MESSAGES/messages.po @@ -9,7 +9,7 @@ msgstr "" "Report-Msgid-Bugs-To: EMAIL@ADDRESS\n" "POT-Creation-Date: 2018-04-22 12:10+0200\n" "PO-Revision-Date: 2021-03-03 17:57+0000\n" -"Last-Translator: Jaume Barber \n" +"Last-Translator: Anonymous \n" "Language-Team: English \n" "Language: en\n" @@ -71,12 +71,12 @@ msgstr "" #: mailu/ui/templates/relay/list.html:19 mailu/ui/templates/token/list.html:19 #: mailu/ui/templates/user/list.html:23 msgid "Comment" -msgstr "" +msgstr "Comment" #: mailu/ui/forms.py:52 mailu/ui/forms.py:61 mailu/ui/forms.py:66 #: mailu/ui/forms.py:73 mailu/ui/forms.py:132 mailu/ui/forms.py:141 msgid "Create" -msgstr "" +msgstr "Create" #: mailu/ui/forms.py:57 msgid "Initial admin" @@ -88,11 +88,11 @@ msgstr "" #: mailu/ui/forms.py:59 mailu/ui/forms.py:79 mailu/ui/forms.py:91 msgid "Confirm password" -msgstr "" +msgstr "Confirm password" #: mailu/ui/forms.py:65 msgid "Alternative name" -msgstr "" +msgstr "Alternative name" #: mailu/ui/forms.py:70 msgid "Relayed domain name" @@ -109,19 +109,19 @@ msgstr "Quota" #: mailu/ui/forms.py:81 msgid "Allow IMAP access" -msgstr "" +msgstr "Allow IMAP access" #: mailu/ui/forms.py:82 msgid "Allow POP3 access" -msgstr "" +msgstr "Allow POP3 access" #: mailu/ui/forms.py:84 msgid "Enabled" -msgstr "" +msgstr "Enabled" #: mailu/ui/forms.py:85 msgid "Save" -msgstr "" +msgstr "Save" #: mailu/ui/forms.py:89 msgid "Email address" @@ -139,7 +139,7 @@ msgstr "" #: mailu/ui/forms.py:98 msgid "Enable spam filter" -msgstr "" +msgstr "Enable spam filter" #: mailu/ui/forms.py:99 msgid "Spam filter tolerance" From 0dc8817f326d03f7d0478d8e3356520b0e85f3d0 Mon Sep 17 00:00:00 2001 From: Jaume Barber Date: Wed, 3 Mar 2021 17:57:27 +0000 Subject: [PATCH 103/596] Translated using Weblate (English) Currently translated at 38.6% (63 of 163 strings) Translation: Mailu/admin Translate-URL: https://translate.tedomum.net/projects/mailu/admin/en/ --- .../translations/en/LC_MESSAGES/messages.po | 16 ++++++++-------- 1 file changed, 8 insertions(+), 8 deletions(-) diff --git a/core/admin/mailu/translations/en/LC_MESSAGES/messages.po b/core/admin/mailu/translations/en/LC_MESSAGES/messages.po index 026f47c6..6773c1f2 100644 --- a/core/admin/mailu/translations/en/LC_MESSAGES/messages.po +++ b/core/admin/mailu/translations/en/LC_MESSAGES/messages.po @@ -8,8 +8,8 @@ msgstr "" "Project-Id-Version: PROJECT VERSION\n" "Report-Msgid-Bugs-To: EMAIL@ADDRESS\n" "POT-Creation-Date: 2018-04-22 12:10+0200\n" -"PO-Revision-Date: 2021-03-03 17:57+0000\n" -"Last-Translator: Anonymous \n" +"PO-Revision-Date: 2021-03-03 18:18+0000\n" +"Last-Translator: Jaume Barber \n" "Language-Team: English \n" "Language: en\n" @@ -59,11 +59,11 @@ msgstr "" #: mailu/ui/forms.py:49 msgid "Maximum user quota" -msgstr "" +msgstr "Maximum user quota" #: mailu/ui/forms.py:50 msgid "Enable sign-up" -msgstr "" +msgstr "Enable sign-up" #: mailu/ui/forms.py:51 mailu/ui/forms.py:72 mailu/ui/forms.py:83 #: mailu/ui/forms.py:128 mailu/ui/forms.py:140 @@ -80,11 +80,11 @@ msgstr "Create" #: mailu/ui/forms.py:57 msgid "Initial admin" -msgstr "" +msgstr "Initial admin" #: mailu/ui/forms.py:58 msgid "Admin password" -msgstr "" +msgstr "Admin password" #: mailu/ui/forms.py:59 mailu/ui/forms.py:79 mailu/ui/forms.py:91 msgid "Confirm password" @@ -131,7 +131,7 @@ msgstr "" #: mailu/ui/templates/user/signup.html:4 #: mailu/ui/templates/user/signup_domain.html:4 msgid "Sign up" -msgstr "" +msgstr "Sign up" #: mailu/ui/forms.py:97 msgid "Displayed name" @@ -143,7 +143,7 @@ msgstr "Enable spam filter" #: mailu/ui/forms.py:99 msgid "Spam filter tolerance" -msgstr "" +msgstr "Spam filter tolerance" #: mailu/ui/forms.py:100 msgid "Enable forwarding" From 58c22fd2c6128e208c1fb7dc128ba7ac548261b0 Mon Sep 17 00:00:00 2001 From: Anonymous Date: Wed, 3 Mar 2021 18:09:32 +0000 Subject: [PATCH 104/596] Translated using Weblate (English) Currently translated at 38.6% (63 of 163 strings) Translation: Mailu/admin Translate-URL: https://translate.tedomum.net/projects/mailu/admin/en/ --- .../mailu/translations/en/LC_MESSAGES/messages.po | 12 ++++++------ 1 file changed, 6 insertions(+), 6 deletions(-) diff --git a/core/admin/mailu/translations/en/LC_MESSAGES/messages.po b/core/admin/mailu/translations/en/LC_MESSAGES/messages.po index 6773c1f2..f931b362 100644 --- a/core/admin/mailu/translations/en/LC_MESSAGES/messages.po +++ b/core/admin/mailu/translations/en/LC_MESSAGES/messages.po @@ -9,7 +9,7 @@ msgstr "" "Report-Msgid-Bugs-To: EMAIL@ADDRESS\n" "POT-Creation-Date: 2018-04-22 12:10+0200\n" "PO-Revision-Date: 2021-03-03 18:18+0000\n" -"Last-Translator: Jaume Barber \n" +"Last-Translator: Anonymous \n" "Language-Team: English \n" "Language: en\n" @@ -147,7 +147,7 @@ msgstr "Spam filter tolerance" #: mailu/ui/forms.py:100 msgid "Enable forwarding" -msgstr "" +msgstr "Enable forwarding" #: mailu/ui/forms.py:101 msgid "Keep a copy of the emails" @@ -160,7 +160,7 @@ msgstr "" #: mailu/ui/forms.py:105 msgid "Save settings" -msgstr "" +msgstr "Save settings" #: mailu/ui/forms.py:110 msgid "Password check" @@ -184,11 +184,11 @@ msgstr "" #: mailu/ui/forms.py:119 msgid "End of vacation" -msgstr "" +msgstr "End of vacation" #: mailu/ui/forms.py:120 msgid "Update" -msgstr "" +msgstr "Update" #: mailu/ui/forms.py:125 msgid "Your token (write it down, as it will never be displayed again)" @@ -200,7 +200,7 @@ msgstr "" #: mailu/ui/forms.py:136 msgid "Alias" -msgstr "" +msgstr "Alias" #: mailu/ui/forms.py:138 msgid "Use SQL LIKE Syntax (e.g. for catch-all aliases)" From 6da597887060fa66bb3d711dde96749718bfcf6b Mon Sep 17 00:00:00 2001 From: Anonymous Date: Wed, 3 Mar 2021 11:39:11 +0000 Subject: [PATCH 105/596] Translated using Weblate (German) Currently translated at 88.3% (144 of 163 strings) Translation: Mailu/admin Translate-URL: https://translate.tedomum.net/projects/mailu/admin/de/ --- .../mailu/translations/de/LC_MESSAGES/messages.po | 14 +++++++++----- 1 file changed, 9 insertions(+), 5 deletions(-) diff --git a/core/admin/mailu/translations/de/LC_MESSAGES/messages.po b/core/admin/mailu/translations/de/LC_MESSAGES/messages.po index 941c22ef..4ae71561 100644 --- a/core/admin/mailu/translations/de/LC_MESSAGES/messages.po +++ b/core/admin/mailu/translations/de/LC_MESSAGES/messages.po @@ -1,11 +1,16 @@ msgid "" msgstr "" +"Project-Id-Version: Mailu\n" +"PO-Revision-Date: 2021-03-04 18:46+0000\n" +"Last-Translator: Anonymous \n" +"Language-Team: German \n" +"Language: de\n" "MIME-Version: 1.0\n" "Content-Type: text/plain; charset=UTF-8\n" "Content-Transfer-Encoding: 8bit\n" -"X-Generator: POEditor.com\n" -"Project-Id-Version: Mailu\n" -"Language: de\n" +"Plural-Forms: nplurals=2; plural=n != 1;\n" +"X-Generator: Weblate 4.0.1\n" #: mailu/ui/forms.py:32 msgid "Invalid email address." @@ -64,7 +69,7 @@ msgstr "Passwort bestätigen" #: mailu/ui/forms.py:80 mailu/ui/templates/user/list.html:22 #: mailu/ui/templates/user/signup_domain.html:16 msgid "Quota" -msgstr "Quota" +msgstr "Kontingent" #: mailu/ui/forms.py:81 msgid "Allow IMAP access" @@ -699,4 +704,3 @@ msgstr "Domain" #: mailu/ui/templates/user/signup_domain.html:15 msgid "Available slots" msgstr "Verfügbare Plätze" - From 6143d66eb888cc02a58ba91bc5e912cdc1d31424 Mon Sep 17 00:00:00 2001 From: Jaume Barber Date: Wed, 3 Mar 2021 18:18:39 +0000 Subject: [PATCH 106/596] Translated using Weblate (English) Currently translated at 39.2% (64 of 163 strings) Translation: Mailu/admin Translate-URL: https://translate.tedomum.net/projects/mailu/admin/en/ --- core/admin/mailu/translations/en/LC_MESSAGES/messages.po | 6 +++--- 1 file changed, 3 insertions(+), 3 deletions(-) diff --git a/core/admin/mailu/translations/en/LC_MESSAGES/messages.po b/core/admin/mailu/translations/en/LC_MESSAGES/messages.po index f931b362..4db1dbf1 100644 --- a/core/admin/mailu/translations/en/LC_MESSAGES/messages.po +++ b/core/admin/mailu/translations/en/LC_MESSAGES/messages.po @@ -8,8 +8,8 @@ msgstr "" "Project-Id-Version: PROJECT VERSION\n" "Report-Msgid-Bugs-To: EMAIL@ADDRESS\n" "POT-Creation-Date: 2018-04-22 12:10+0200\n" -"PO-Revision-Date: 2021-03-03 18:18+0000\n" -"Last-Translator: Anonymous \n" +"PO-Revision-Date: 2021-03-04 18:46+0000\n" +"Last-Translator: Jaume Barber \n" "Language-Team: English \n" "Language: en\n" @@ -196,7 +196,7 @@ msgstr "" #: mailu/ui/forms.py:130 mailu/ui/templates/token/list.html:20 msgid "Authorized IP" -msgstr "" +msgstr "Authorized IP" #: mailu/ui/forms.py:136 msgid "Alias" From 5e96a4bfcfbe93a3a45922811b4e88ebad874fa8 Mon Sep 17 00:00:00 2001 From: Jaume Barber Date: Wed, 3 Mar 2021 13:11:18 +0000 Subject: [PATCH 107/596] Translated using Weblate (Spanish) Currently translated at 91.4% (149 of 163 strings) Translation: Mailu/admin Translate-URL: https://translate.tedomum.net/projects/mailu/admin/es/ --- core/admin/mailu/translations/es/LC_MESSAGES/messages.po | 4 ++-- 1 file changed, 2 insertions(+), 2 deletions(-) diff --git a/core/admin/mailu/translations/es/LC_MESSAGES/messages.po b/core/admin/mailu/translations/es/LC_MESSAGES/messages.po index ff6b9f36..c70ed6f5 100644 --- a/core/admin/mailu/translations/es/LC_MESSAGES/messages.po +++ b/core/admin/mailu/translations/es/LC_MESSAGES/messages.po @@ -1,7 +1,7 @@ msgid "" msgstr "" "Project-Id-Version: Mailu\n" -"PO-Revision-Date: 2021-03-03 12:37+0000\n" +"PO-Revision-Date: 2021-03-04 18:46+0000\n" "Last-Translator: Jaume Barber \n" "Language-Team: Spanish \n" @@ -578,7 +578,7 @@ msgstr "Lista de dominios externos (relayed)" #: mailu/ui/templates/relay/list.html:9 msgid "New relayed domain" -msgstr "Nuevo dominio externo (relayed)" +msgstr "Editar dominio externo (relay)" #: mailu/ui/forms.py:125 msgid "Your token (write it down, as it will never be displayed again)" From 480ec29d3d8dbde9bb1ad7b7ba9fb79b630d87a9 Mon Sep 17 00:00:00 2001 From: Jaume Barber Date: Wed, 3 Mar 2021 17:03:23 +0000 Subject: [PATCH 108/596] Translated using Weblate (Italian) Currently translated at 91.4% (149 of 163 strings) Translation: Mailu/admin Translate-URL: https://translate.tedomum.net/projects/mailu/admin/it/ --- core/admin/mailu/translations/it/LC_MESSAGES/messages.po | 3 +-- 1 file changed, 1 insertion(+), 2 deletions(-) diff --git a/core/admin/mailu/translations/it/LC_MESSAGES/messages.po b/core/admin/mailu/translations/it/LC_MESSAGES/messages.po index 6ec219bf..9ed5e132 100644 --- a/core/admin/mailu/translations/it/LC_MESSAGES/messages.po +++ b/core/admin/mailu/translations/it/LC_MESSAGES/messages.po @@ -1,7 +1,7 @@ msgid "" msgstr "" "Project-Id-Version: Mailu\n" -"PO-Revision-Date: 2021-03-03 17:03+0000\n" +"PO-Revision-Date: 2021-03-04 18:46+0000\n" "Last-Translator: Jaume Barber \n" "Language-Team: Italian \n" @@ -576,7 +576,6 @@ msgid "Relayed domain list" msgstr "Elenco di domini affidati" #: mailu/ui/templates/relay/list.html:9 -#, fuzzy msgid "New relayed domain" msgstr "Nuovo dominio affidato" From 7a01a6338968ceba61443416f697dc78662c26bd Mon Sep 17 00:00:00 2001 From: Jaume Barber Date: Wed, 3 Mar 2021 12:15:37 +0000 Subject: [PATCH 109/596] Translated using Weblate (Portuguese) Currently translated at 88.3% (144 of 163 strings) Translation: Mailu/admin Translate-URL: https://translate.tedomum.net/projects/mailu/admin/pt/ --- .../mailu/translations/pt/LC_MESSAGES/messages.po | 14 +++++++++----- 1 file changed, 9 insertions(+), 5 deletions(-) diff --git a/core/admin/mailu/translations/pt/LC_MESSAGES/messages.po b/core/admin/mailu/translations/pt/LC_MESSAGES/messages.po index 58338380..f9673767 100644 --- a/core/admin/mailu/translations/pt/LC_MESSAGES/messages.po +++ b/core/admin/mailu/translations/pt/LC_MESSAGES/messages.po @@ -1,11 +1,16 @@ msgid "" msgstr "" +"Project-Id-Version: Mailu\n" +"PO-Revision-Date: 2021-03-04 18:46+0000\n" +"Last-Translator: Jaume Barber \n" +"Language-Team: Portuguese \n" +"Language: pt\n" "MIME-Version: 1.0\n" "Content-Type: text/plain; charset=UTF-8\n" "Content-Transfer-Encoding: 8bit\n" -"X-Generator: POEditor.com\n" -"Project-Id-Version: Mailu\n" -"Language: pt\n" +"Plural-Forms: nplurals=2; plural=n > 1;\n" +"X-Generator: Weblate 4.0.1\n" #: mailu/ui/forms.py:32 msgid "Invalid email address." @@ -183,7 +188,7 @@ msgstr "Erro no docker" #: mailu/ui/templates/docker-error.html:12 msgid "An error occurred while talking to the Docker server." -msgstr "Um erro foi encontrado na conexão com o servidor Docker" +msgstr "Um erro foi encontrado na conexão com o servidor Docker." #: mailu/admin/templates/login.html:6 msgid "Your account" @@ -700,4 +705,3 @@ msgstr "Domínio" #: mailu/ui/templates/user/signup_domain.html:15 msgid "Available slots" msgstr "Slots disponíveis" - From afae5d1c24a33f8f6652e5d23105c63b18d5d546 Mon Sep 17 00:00:00 2001 From: Jaume Barber Date: Wed, 3 Mar 2021 12:16:00 +0000 Subject: [PATCH 110/596] Translated using Weblate (Russian) Currently translated at 88.3% (144 of 163 strings) Translation: Mailu/admin Translate-URL: https://translate.tedomum.net/projects/mailu/admin/ru/ --- core/admin/mailu/translations/ru/LC_MESSAGES/messages.po | 8 ++++---- 1 file changed, 4 insertions(+), 4 deletions(-) diff --git a/core/admin/mailu/translations/ru/LC_MESSAGES/messages.po b/core/admin/mailu/translations/ru/LC_MESSAGES/messages.po index 72e5f0cb..790119fc 100644 --- a/core/admin/mailu/translations/ru/LC_MESSAGES/messages.po +++ b/core/admin/mailu/translations/ru/LC_MESSAGES/messages.po @@ -1,8 +1,8 @@ msgid "" msgstr "" "Project-Id-Version: Mailu\n" -"PO-Revision-Date: 2019-07-22 06:23+0000\n" -"Last-Translator: kaiyou \n" +"PO-Revision-Date: 2021-03-04 18:46+0000\n" +"Last-Translator: Jaume Barber \n" "Language-Team: Russian \n" "Language: ru\n" @@ -11,7 +11,7 @@ msgstr "" "Content-Transfer-Encoding: 8bit\n" "Plural-Forms: nplurals=3; plural=n%10==1 && n%100!=11 ? 0 : n%10>=2 && n%10<=" "4 && (n%100<10 || n%100>=20) ? 1 : 2;\n" -"X-Generator: Weblate 3.3\n" +"X-Generator: Weblate 4.0.1\n" #: mailu/ui/forms.py:32 msgid "Invalid email address." @@ -189,7 +189,7 @@ msgstr "Ошибка Docker" #: mailu/ui/templates/docker-error.html:12 msgid "An error occurred while talking to the Docker server." -msgstr "Произошла ошибка при обращении к серверу Docker" +msgstr "Произошла ошибка при обращении к серверу Docker." #: mailu/admin/templates/login.html:6 msgid "Your account" From cd9992f79cbc399b30e713e9bea5ed98f4371227 Mon Sep 17 00:00:00 2001 From: Jaume Barber Date: Wed, 3 Mar 2021 12:16:15 +0000 Subject: [PATCH 111/596] Translated using Weblate (Swedish) Currently translated at 74.2% (121 of 163 strings) Translation: Mailu/admin Translate-URL: https://translate.tedomum.net/projects/mailu/admin/sv/ --- .../mailu/translations/sv/LC_MESSAGES/messages.po | 14 +++++++++----- 1 file changed, 9 insertions(+), 5 deletions(-) diff --git a/core/admin/mailu/translations/sv/LC_MESSAGES/messages.po b/core/admin/mailu/translations/sv/LC_MESSAGES/messages.po index 825888f1..071040f6 100644 --- a/core/admin/mailu/translations/sv/LC_MESSAGES/messages.po +++ b/core/admin/mailu/translations/sv/LC_MESSAGES/messages.po @@ -1,11 +1,16 @@ msgid "" msgstr "" +"Project-Id-Version: Mailu\n" +"PO-Revision-Date: 2021-03-04 18:46+0000\n" +"Last-Translator: Jaume Barber \n" +"Language-Team: Swedish \n" +"Language: sv\n" "MIME-Version: 1.0\n" "Content-Type: text/plain; charset=UTF-8\n" "Content-Transfer-Encoding: 8bit\n" -"X-Generator: POEditor.com\n" -"Project-Id-Version: Mailu\n" -"Language: sk\n" +"Plural-Forms: nplurals=2; plural=n != 1;\n" +"X-Generator: Weblate 4.0.1\n" #: mailu/ui/forms.py:32 msgid "Invalid email address." @@ -183,7 +188,7 @@ msgstr "Docker fel" #: mailu/ui/templates/docker-error.html:12 msgid "An error occurred while talking to the Docker server." -msgstr "Ett fel inträffade vid kommunikation med Docker" +msgstr "Ett fel inträffade vid kommunikation med Docker." #: mailu/admin/templates/login.html:6 msgid "Your account" @@ -699,4 +704,3 @@ msgstr "" #: mailu/ui/templates/user/signup_domain.html:15 msgid "Available slots" msgstr "" - From a49b9d79744282ea3b404845a262ed8324595504 Mon Sep 17 00:00:00 2001 From: Jaume Barber Date: Wed, 3 Mar 2021 11:43:04 +0000 Subject: [PATCH 112/596] Translated using Weblate (Catalan) Currently translated at 99.3% (150 of 151 strings) Translation: Mailu/admin Translate-URL: https://translate.tedomum.net/projects/mailu/admin/ca/ --- .../translations/ca/LC_MESSAGES/messages.po | 34 ++++++++++--------- 1 file changed, 18 insertions(+), 16 deletions(-) diff --git a/core/admin/mailu/translations/ca/LC_MESSAGES/messages.po b/core/admin/mailu/translations/ca/LC_MESSAGES/messages.po index 76594a3c..880709f1 100644 --- a/core/admin/mailu/translations/ca/LC_MESSAGES/messages.po +++ b/core/admin/mailu/translations/ca/LC_MESSAGES/messages.po @@ -8,7 +8,7 @@ msgstr "" "Project-Id-Version: PROJECT VERSION\n" "Report-Msgid-Bugs-To: EMAIL@ADDRESS\n" "POT-Creation-Date: 2018-04-22 12:10+0200\n" -"PO-Revision-Date: 2021-03-03 11:35+0000\n" +"PO-Revision-Date: 2021-03-04 18:46+0000\n" "Last-Translator: Jaume Barber \n" "Language-Team: Catalan \n" @@ -139,7 +139,7 @@ msgstr "Nom per mostrar" #: mailu/ui/forms.py:98 msgid "Enable spam filter" -msgstr "Activeu filtre d'spam" +msgstr "Activeu filtre spam" #: mailu/ui/forms.py:99 msgid "Spam filter tolerance" @@ -204,7 +204,8 @@ msgstr "Àlies" #: mailu/ui/forms.py:138 msgid "Use SQL LIKE Syntax (e.g. for catch-all aliases)" -msgstr "Feu servir sintaxi tipus SQL (ex. per a agafar tots els àlies)" +msgstr "" +"Feu servir sintaxi tipus SQL (ex. per seleccionar tots els àlies catch-all)" #: mailu/ui/forms.py:145 msgid "Admin email" @@ -246,11 +247,11 @@ msgstr "Mantén els correus al servidor" #: mailu/ui/forms.py:168 msgid "Announcement subject" -msgstr "Tema de l'avís" +msgstr "Tema de la notificació" #: mailu/ui/forms.py:170 msgid "Announcement body" -msgstr "Missatge de l'avís" +msgstr "Missatge de la notificació" #: mailu/ui/forms.py:172 msgid "Send" @@ -258,7 +259,7 @@ msgstr "Envia" #: mailu/ui/templates/announcement.html:4 msgid "Public announcement" -msgstr "Avís públic" +msgstr "Notificació pública" #: mailu/ui/templates/client.html:4 mailu/ui/templates/sidebar.html:82 msgid "Client setup" @@ -316,7 +317,7 @@ msgstr "Administració" #: mailu/ui/templates/sidebar.html:44 msgid "Announcement" -msgstr "Avís" +msgstr "Notificació" #: mailu/ui/templates/sidebar.html:49 msgid "Administrators" @@ -324,7 +325,7 @@ msgstr "Administradors" #: mailu/ui/templates/sidebar.html:54 msgid "Relayed domains" -msgstr "Dominis delegats" +msgstr "Dominis traspassats" #: mailu/ui/templates/sidebar.html:59 mailu/ui/templates/user/settings.html:15 msgid "Antispam" @@ -546,18 +547,19 @@ msgid "" " expires." msgstr "" "Si no sabeu configurar un registre MX a la zona DNS,\n" -"contacteu el vostre proveïdor o administrador de DNS. Per favor, espereu \n" +"contacteu amb el vostre proveïdor o administrador de DNS. Per favor, espereu " +"\n" "uns quants minuts despres d'ajustar el registre MX perquè la " "caixet \n" "del servidor local expire." #: mailu/ui/templates/fetch/create.html:4 msgid "Add a fetched account" -msgstr "Afegiu un compte (fetched)" +msgstr "Afegiu un compte extern" #: mailu/ui/templates/fetch/edit.html:4 msgid "Update a fetched account" -msgstr "Actualitzeu un compte (fetched)" +msgstr "Actualitzeu compte extern" #: mailu/ui/templates/fetch/list.html:12 msgid "Add an account" @@ -605,11 +607,11 @@ msgstr "Editeu domini llegat (relayed)" #: mailu/ui/templates/relay/list.html:4 msgid "Relayed domain list" -msgstr "Llista de dominis llegats (relayed)" +msgstr "Llista de dominis traspassats" #: mailu/ui/templates/relay/list.html:9 msgid "New relayed domain" -msgstr "Nou domini llegat (relayed)" +msgstr "Nou domini traspassat" #: mailu/ui/templates/token/create.html:4 msgid "Create an authentication token" @@ -653,7 +655,7 @@ msgstr "Ajustos d'usuari" #: mailu/ui/templates/user/list.html:21 msgid "Features" -msgstr "Funcions" +msgstr "Característiques" #: mailu/ui/templates/user/password.html:4 msgid "Password update" @@ -669,11 +671,11 @@ msgstr "Auto-reenviament" #: mailu/ui/templates/user/signup_domain.html:8 msgid "pick a domain for the new account" -msgstr "tria un domini per al compte nou" +msgstr "trieu un domini per al compte nou" #: mailu/ui/templates/user/signup_domain.html:14 msgid "Domain" -msgstr "Domini" +msgstr "Nom de domini" #: mailu/ui/templates/user/signup_domain.html:15 msgid "Available slots" From 5bb67dfcbb47fb0b0c43e631337676b17f9569b1 Mon Sep 17 00:00:00 2001 From: Jaume Barber Date: Wed, 3 Mar 2021 17:08:54 +0000 Subject: [PATCH 113/596] Translated using Weblate (Basque) Currently translated at 100.0% (151 of 151 strings) Translation: Mailu/admin Translate-URL: https://translate.tedomum.net/projects/mailu/admin/eu/ --- .../translations/eu/LC_MESSAGES/messages.po | 87 ++++++++++--------- 1 file changed, 45 insertions(+), 42 deletions(-) diff --git a/core/admin/mailu/translations/eu/LC_MESSAGES/messages.po b/core/admin/mailu/translations/eu/LC_MESSAGES/messages.po index 3a72c9af..6ca737a3 100644 --- a/core/admin/mailu/translations/eu/LC_MESSAGES/messages.po +++ b/core/admin/mailu/translations/eu/LC_MESSAGES/messages.po @@ -8,32 +8,35 @@ msgstr "" "Project-Id-Version: PROJECT VERSION\n" "Report-Msgid-Bugs-To: EMAIL@ADDRESS\n" "POT-Creation-Date: 2018-04-22 12:10+0200\n" -"PO-Revision-Date: YEAR-MO-DA HO:MI+ZONE\n" -"Last-Translator: Automatically generated\n" -"Language-Team: none\n" +"PO-Revision-Date: 2021-03-04 18:46+0000\n" +"Last-Translator: Jaume Barber \n" +"Language-Team: Basque \n" "Language: eu\n" "MIME-Version: 1.0\n" "Content-Type: text/plain; charset=utf-8\n" "Content-Transfer-Encoding: 8bit\n" +"Plural-Forms: nplurals=2; plural=n != 1;\n" +"X-Generator: Weblate 4.0.1\n" "Generated-By: Babel 2.5.3\n" #: mailu/ui/forms.py:32 msgid "Invalid email address." -msgstr "" +msgstr "baliogabeko helbide elektronikoa." #: mailu/ui/forms.py:36 msgid "Confirm" -msgstr "" +msgstr "Ados" #: mailu/ui/forms.py:40 mailu/ui/forms.py:77 msgid "E-mail" -msgstr "" +msgstr "E-mail" #: mailu/ui/forms.py:41 mailu/ui/forms.py:78 mailu/ui/forms.py:90 #: mailu/ui/forms.py:109 mailu/ui/forms.py:162 #: mailu/ui/templates/client.html:32 mailu/ui/templates/client.html:59 msgid "Password" -msgstr "" +msgstr "Pasahitza" #: mailu/ui/forms.py:42 mailu/ui/templates/login.html:4 #: mailu/ui/templates/sidebar.html:111 @@ -48,7 +51,7 @@ msgstr "" #: mailu/ui/forms.py:47 msgid "Maximum user count" -msgstr "" +msgstr "Erabiltzaileen gehieneko kopurua" #: mailu/ui/forms.py:48 msgid "Maximum alias count" @@ -56,11 +59,11 @@ msgstr "" #: mailu/ui/forms.py:49 msgid "Maximum user quota" -msgstr "" +msgstr "Erabiltzaile bakoitzeko gehieneko espazioa" #: mailu/ui/forms.py:50 msgid "Enable sign-up" -msgstr "" +msgstr "Gaitu erregistroa" #: mailu/ui/forms.py:51 mailu/ui/forms.py:72 mailu/ui/forms.py:83 #: mailu/ui/forms.py:128 mailu/ui/forms.py:140 @@ -68,57 +71,57 @@ msgstr "" #: mailu/ui/templates/relay/list.html:19 mailu/ui/templates/token/list.html:19 #: mailu/ui/templates/user/list.html:23 msgid "Comment" -msgstr "" +msgstr "Iruzkindua" #: mailu/ui/forms.py:52 mailu/ui/forms.py:61 mailu/ui/forms.py:66 #: mailu/ui/forms.py:73 mailu/ui/forms.py:132 mailu/ui/forms.py:141 msgid "Create" -msgstr "" +msgstr "Sortu" #: mailu/ui/forms.py:57 msgid "Initial admin" -msgstr "" +msgstr "Administratzailea" #: mailu/ui/forms.py:58 msgid "Admin password" -msgstr "" +msgstr "Administratzaileko pasahitza" #: mailu/ui/forms.py:59 mailu/ui/forms.py:79 mailu/ui/forms.py:91 msgid "Confirm password" -msgstr "" +msgstr "Berretsi pasahitza" #: mailu/ui/forms.py:65 msgid "Alternative name" -msgstr "" +msgstr "Izen alternatiboa" #: mailu/ui/forms.py:70 msgid "Relayed domain name" -msgstr "" +msgstr "Igorritako domeinu izena" #: mailu/ui/forms.py:71 mailu/ui/templates/relay/list.html:18 msgid "Remote host" -msgstr "" +msgstr "Urruneko ostalaria" #: mailu/ui/forms.py:80 mailu/ui/templates/user/list.html:22 #: mailu/ui/templates/user/signup_domain.html:16 msgid "Quota" -msgstr "" +msgstr "Espazioa" #: mailu/ui/forms.py:81 msgid "Allow IMAP access" -msgstr "" +msgstr "Baimendu IMAP sarbidea" #: mailu/ui/forms.py:82 msgid "Allow POP3 access" -msgstr "" +msgstr "Baimendu POP3 sarbidea" #: mailu/ui/forms.py:84 msgid "Enabled" -msgstr "" +msgstr "Gaituta" #: mailu/ui/forms.py:85 msgid "Save" -msgstr "" +msgstr "Gorde" #: mailu/ui/forms.py:89 msgid "Email address" @@ -128,7 +131,7 @@ msgstr "" #: mailu/ui/templates/user/signup.html:4 #: mailu/ui/templates/user/signup_domain.html:4 msgid "Sign up" -msgstr "" +msgstr "Erregistratu" #: mailu/ui/forms.py:97 msgid "Displayed name" @@ -136,15 +139,15 @@ msgstr "" #: mailu/ui/forms.py:98 msgid "Enable spam filter" -msgstr "" +msgstr "Gaitu spam iragazkia" #: mailu/ui/forms.py:99 msgid "Spam filter tolerance" -msgstr "" +msgstr "Spam iragazkiaren tolerantzia" #: mailu/ui/forms.py:100 msgid "Enable forwarding" -msgstr "" +msgstr "Gaitu birbidaltzea" #: mailu/ui/forms.py:101 msgid "Keep a copy of the emails" @@ -157,7 +160,7 @@ msgstr "" #: mailu/ui/forms.py:105 msgid "Save settings" -msgstr "" +msgstr "Gorde ezarpenak" #: mailu/ui/forms.py:110 msgid "Password check" @@ -181,11 +184,11 @@ msgstr "" #: mailu/ui/forms.py:119 msgid "End of vacation" -msgstr "" +msgstr "Oporren amaiera" #: mailu/ui/forms.py:120 msgid "Update" -msgstr "" +msgstr "Eguneratu" #: mailu/ui/forms.py:125 msgid "Your token (write it down, as it will never be displayed again)" @@ -193,11 +196,11 @@ msgstr "" #: mailu/ui/forms.py:130 mailu/ui/templates/token/list.html:20 msgid "Authorized IP" -msgstr "" +msgstr "Baimendutako IP" #: mailu/ui/forms.py:136 msgid "Alias" -msgstr "" +msgstr "Ezizenza" #: mailu/ui/forms.py:138 msgid "Use SQL LIKE Syntax (e.g. for catch-all aliases)" @@ -226,7 +229,7 @@ msgstr "" #: mailu/ui/forms.py:159 mailu/ui/templates/client.html:20 #: mailu/ui/templates/client.html:47 msgid "TCP port" -msgstr "" +msgstr "TCP ataka" #: mailu/ui/forms.py:160 msgid "Enable TLS" @@ -276,11 +279,11 @@ msgstr "" #: mailu/ui/templates/confirm.html:13 #, python-format msgid "You are about to %(action)s. Please confirm your action." -msgstr "" +msgstr "Zu zara %(action)s-etan. Mesedez ekintza honen berretsi." #: mailu/ui/templates/docker-error.html:4 msgid "Docker error" -msgstr "" +msgstr "Docker-en errorea" #: mailu/ui/templates/docker-error.html:12 msgid "An error occurred while talking to the Docker server." @@ -321,11 +324,11 @@ msgstr "" #: mailu/ui/templates/sidebar.html:54 msgid "Relayed domains" -msgstr "" +msgstr "Igorritako domeinuak" #: mailu/ui/templates/sidebar.html:59 mailu/ui/templates/user/settings.html:15 msgid "Antispam" -msgstr "" +msgstr "Antispam" #: mailu/ui/templates/sidebar.html:66 msgid "Mail domains" @@ -586,19 +589,19 @@ msgstr "" #: mailu/ui/templates/relay/create.html:4 msgid "New relay domain" -msgstr "" +msgstr "Igorritako domeinu berria" #: mailu/ui/templates/relay/edit.html:4 msgid "Edit relayd domain" -msgstr "" +msgstr "Editatu igorritako domeinua" #: mailu/ui/templates/relay/list.html:4 msgid "Relayed domain list" -msgstr "" +msgstr "Igorritako domeinuen zerrenda" #: mailu/ui/templates/relay/list.html:9 msgid "New relayed domain" -msgstr "" +msgstr "Igorritako domeinu berria" #: mailu/ui/templates/token/create.html:4 msgid "Create an authentication token" @@ -662,7 +665,7 @@ msgstr "" #: mailu/ui/templates/user/signup_domain.html:14 msgid "Domain" -msgstr "" +msgstr "Domeinu izena" #: mailu/ui/templates/user/signup_domain.html:15 msgid "Available slots" From 0dcc059cd6950422588a694c3727a443cf56feab Mon Sep 17 00:00:00 2001 From: Florent Daigniere Date: Fri, 5 Mar 2021 22:26:46 +0100 Subject: [PATCH 114/596] Add a new knob as discussed on matrix with lub --- core/admin/mailu/configuration.py | 2 +- docs/configuration.rst | 2 ++ 2 files changed, 3 insertions(+), 1 deletion(-) diff --git a/core/admin/mailu/configuration.py b/core/admin/mailu/configuration.py index 982a1eb0..dac913fa 100644 --- a/core/admin/mailu/configuration.py +++ b/core/admin/mailu/configuration.py @@ -54,6 +54,7 @@ DEFAULT_CONFIG = { # Advanced settings 'PASSWORD_SCHEME': 'PBKDF2', 'LOG_LEVEL': 'WARNING', + 'SESSION_COOKIE_SECURE': True, # Host settings 'HOST_IMAP': 'imap', 'HOST_LMTP': 'imap:2525', @@ -125,7 +126,6 @@ class ConfigManager(dict): self.config['QUOTA_STORAGE_URL'] = 'redis://{0}/1'.format(self.config['REDIS_ADDRESS']) self.config['SESSION_COOKIE_SAMESITE'] = 'Strict' self.config['SESSION_COOKIE_HTTPONLY'] = True - self.config['SESSION_COOKIE_SECURE'] = self.config['TLS_FLAVOR'] != 'notls' # update the app config itself app.config = self diff --git a/docs/configuration.rst b/docs/configuration.rst index 5ff3546a..9123054c 100644 --- a/docs/configuration.rst +++ b/docs/configuration.rst @@ -142,6 +142,8 @@ The ``PASSWORD_SCHEME`` is the password encryption scheme. You should use the default value, unless you are importing password from a separate system and want to keep using the old password encryption scheme. +The ``SESSION_COOKIE_SECURE`` (default: True) setting controls the secure flag on the cookies of the administrative interface. It should only be turned off if you intend to access it over plain HTTP. + The ``LOG_LEVEL`` setting is used by the python start-up scripts as a logging threshold. Log messages equal or higher than this priority will be printed. Can be one of: CRITICAL, ERROR, WARNING, INFO, DEBUG or NOTSET. From 58b2cdc4288854481e8914b3a1b4318708906e15 Mon Sep 17 00:00:00 2001 From: Florent Daigniere Date: Thu, 21 Jan 2021 20:01:57 +0100 Subject: [PATCH 115/596] Don't do more work than necessary --- core/admin/mailu/internal/nginx.py | 13 ++++++++----- 1 file changed, 8 insertions(+), 5 deletions(-) diff --git a/core/admin/mailu/internal/nginx.py b/core/admin/mailu/internal/nginx.py index 1e0b16c2..de4248fa 100644 --- a/core/admin/mailu/internal/nginx.py +++ b/core/admin/mailu/internal/nginx.py @@ -49,11 +49,14 @@ def handle_authentication(headers): user = models.User.query.get(user_email) status = False if user: - for token in user.tokens: - if (token.check_password(password) and - (not token.ip or token.ip == ip)): - status = True - if user.check_password(password): + # All tokens are 32 characters hex lowercase + if len(password) == 32: + for token in user.tokens: + if (token.check_password(password) and + (not token.ip or token.ip == ip)): + status = True + break + if not status and user.check_password(password): status = True if status: if protocol == "imap" and not user.enable_imap: From eb7895bd1cf5ae41ccfda384f06480767ed75172 Mon Sep 17 00:00:00 2001 From: Florent Daigniere Date: Sun, 7 Feb 2021 17:08:52 +0100 Subject: [PATCH 116/596] Don't do more work than necessary (/webdav) This is also fixing tokens on /webdav/ --- core/admin/mailu/internal/views/auth.py | 20 ++++++++++++++++---- 1 file changed, 16 insertions(+), 4 deletions(-) diff --git a/core/admin/mailu/internal/views/auth.py b/core/admin/mailu/internal/views/auth.py index 825dba56..26d57b3d 100644 --- a/core/admin/mailu/internal/views/auth.py +++ b/core/admin/mailu/internal/views/auth.py @@ -53,10 +53,22 @@ def basic_authentication(): encoded = authorization.replace("Basic ", "") user_email, password = base64.b64decode(encoded).split(b":") user = models.User.query.get(user_email.decode("utf8")) - if user and user.enabled and user.check_password(password.decode("utf8")): - response = flask.Response() - response.headers["X-User"] = user.email - return response + if user and user.enabled: + password = password.decode('utf-8') + status = False + # All tokens are 32 characters hex lowercase + if len(password) == 32: + for token in user.tokens: + if (token.check_password(password) and + (not token.ip or token.ip == ip)): + status = True + break + if not status and user.check_password(password): + status = True + if status: + response = flask.Response() + response.headers["X-User"] = user.email + return response response = flask.Response(status=401) response.headers["WWW-Authenticate"] = 'Basic realm="Login Required"' return response From 00b001f76b7818f2561b9212fbe22edad113a970 Mon Sep 17 00:00:00 2001 From: Florent Daigniere Date: Thu, 21 Jan 2021 20:37:25 +0100 Subject: [PATCH 117/596] Improve the token storage format shortcomings of the previous format included: - 1000x slower than it should be (no point in adding rounds since there is enough entropy: they are not bruteforceable) - vulnerable to DoS as explained in https://passlib.readthedocs.io/en/stable/lib/passlib.hash.sha256_crypt.html#security-issues --- core/admin/mailu/models.py | 12 ++++++++++-- 1 file changed, 10 insertions(+), 2 deletions(-) diff --git a/core/admin/mailu/models.py b/core/admin/mailu/models.py index bbc00f2d..164312ad 100644 --- a/core/admin/mailu/models.py +++ b/core/admin/mailu/models.py @@ -493,10 +493,18 @@ class Token(Base): ip = db.Column(db.String(255)) def check_password(self, password): - return hash.sha256_crypt.verify(password, self.password) + if self.password.startswith("$5$"): + if hash.sha256_crypt.verify(password, self.password): + self.set_password(password) + db.session.add(self) + db.session.commit() + return True + return False + return hash.pbkdf2_sha256.verify(password, self.password) def set_password(self, password): - self.password = hash.sha256_crypt.using(rounds=1000).hash(password) + # tokens have 128bits of entropy, they are not bruteforceable + self.password = hash.pbkdf2_sha256.using(rounds=1).hash(password) def __str__(self): return self.comment From 7137ba6ff18c87185e25b9913377d8e7ce3fa8b6 Mon Sep 17 00:00:00 2001 From: Florent Daigniere Date: Tue, 2 Feb 2021 20:10:18 +0100 Subject: [PATCH 118/596] Misc improvements to PASSWORD_SCHEME - remove PASSWORD_SCHEME altogether - introduce CREDENTIAL_ROUNDS - migrate all old hashes to the current format - auto-detect/enable all hash types that passlib supports - upgrade passlib to 1.7.4 (see #1706: ldap_salted_sha512 support) --- core/admin/mailu/configuration.py | 2 +- core/admin/mailu/manage.py | 24 +++++++-------------- core/admin/mailu/models.py | 35 ++++++++++++++++++++----------- core/admin/requirements-prod.txt | 2 +- docs/cli.rst | 1 - docs/compose/.env | 5 ++--- docs/configuration.rst | 4 +--- 7 files changed, 35 insertions(+), 38 deletions(-) diff --git a/core/admin/mailu/configuration.py b/core/admin/mailu/configuration.py index 3d4d8668..cdee1084 100644 --- a/core/admin/mailu/configuration.py +++ b/core/admin/mailu/configuration.py @@ -33,6 +33,7 @@ DEFAULT_CONFIG = { 'TLS_FLAVOR': 'cert', 'AUTH_RATELIMIT': '10/minute;1000/hour', 'AUTH_RATELIMIT_SUBNET': True, + 'CREDENTIAL_ROUNDS': 12, 'DISABLE_STATISTICS': False, # Mail settings 'DMARC_RUA': None, @@ -52,7 +53,6 @@ DEFAULT_CONFIG = { 'RECAPTCHA_PUBLIC_KEY': '', 'RECAPTCHA_PRIVATE_KEY': '', # Advanced settings - 'PASSWORD_SCHEME': 'PBKDF2', 'LOG_LEVEL': 'WARNING', 'SESSION_COOKIE_SECURE': True, # Host settings diff --git a/core/admin/mailu/manage.py b/core/admin/mailu/manage.py index 62f214d3..9c576404 100644 --- a/core/admin/mailu/manage.py +++ b/core/admin/mailu/manage.py @@ -86,13 +86,10 @@ def admin(localpart, domain_name, password, mode='create'): @click.argument('localpart') @click.argument('domain_name') @click.argument('password') -@click.argument('hash_scheme', required=False) @flask_cli.with_appcontext -def user(localpart, domain_name, password, hash_scheme=None): +def user(localpart, domain_name, password): """ Create a user """ - if hash_scheme is None: - hash_scheme = app.config['PASSWORD_SCHEME'] domain = models.Domain.query.get(domain_name) if not domain: domain = models.Domain(name=domain_name) @@ -102,7 +99,7 @@ def user(localpart, domain_name, password, hash_scheme=None): domain=domain, global_admin=False ) - user.set_password(password, hash_scheme=hash_scheme) + user.set_password(password) db.session.add(user) db.session.commit() @@ -111,17 +108,14 @@ def user(localpart, domain_name, password, hash_scheme=None): @click.argument('localpart') @click.argument('domain_name') @click.argument('password') -@click.argument('hash_scheme', required=False) @flask_cli.with_appcontext -def password(localpart, domain_name, password, hash_scheme=None): +def password(localpart, domain_name, password): """ Change the password of an user """ email = '{0}@{1}'.format(localpart, domain_name) user = models.User.query.get(email) - if hash_scheme is None: - hash_scheme = app.config['PASSWORD_SCHEME'] if user: - user.set_password(password, hash_scheme=hash_scheme) + user.set_password(password) else: print("User " + email + " not found.") db.session.commit() @@ -148,13 +142,10 @@ def domain(domain_name, max_users=-1, max_aliases=-1, max_quota_bytes=0): @click.argument('localpart') @click.argument('domain_name') @click.argument('password_hash') -@click.argument('hash_scheme') @flask_cli.with_appcontext -def user_import(localpart, domain_name, password_hash, hash_scheme = None): +def user_import(localpart, domain_name, password_hash): """ Import a user along with password hash. """ - if hash_scheme is None: - hash_scheme = app.config['PASSWORD_SCHEME'] domain = models.Domain.query.get(domain_name) if not domain: domain = models.Domain(name=domain_name) @@ -164,7 +155,7 @@ def user_import(localpart, domain_name, password_hash, hash_scheme = None): domain=domain, global_admin=False ) - user.set_password(password_hash, hash_scheme=hash_scheme, raw=True) + user.set_password(password_hash, raw=True) db.session.add(user) db.session.commit() @@ -217,7 +208,6 @@ def config_update(verbose=False, delete_objects=False): localpart = user_config['localpart'] domain_name = user_config['domain'] password_hash = user_config.get('password_hash', None) - hash_scheme = user_config.get('hash_scheme', None) domain = models.Domain.query.get(domain_name) email = '{0}@{1}'.format(localpart, domain_name) optional_params = {} @@ -239,7 +229,7 @@ def config_update(verbose=False, delete_objects=False): else: for k in optional_params: setattr(user, k, optional_params[k]) - user.set_password(password_hash, hash_scheme=hash_scheme, raw=True) + user.set_password(password_hash, raw=True) db.session.add(user) aliases = new_config.get('aliases', []) diff --git a/core/admin/mailu/models.py b/core/admin/mailu/models.py index 164312ad..905af4a2 100644 --- a/core/admin/mailu/models.py +++ b/core/admin/mailu/models.py @@ -1,7 +1,7 @@ from mailu import dkim from sqlalchemy.ext import declarative -from passlib import context, hash +from passlib import context, hash, registry from datetime import datetime, date from email.mime import text from flask import current_app as app @@ -370,17 +370,30 @@ class User(Base, Email): 'CRYPT': "des_crypt"} def get_password_context(self): + schemes = registry.list_crypt_handlers() + # scrypt throws a warning if the native wheels aren't found + schemes.remove('scrypt') + # we can't leave plaintext schemes as they will be misidentified + for scheme in schemes: + if scheme.endswith('plaintext'): + schemes.remove(scheme) return context.CryptContext( - schemes=self.scheme_dict.values(), - default=self.scheme_dict[app.config['PASSWORD_SCHEME']], + schemes=schemes, + default='bcrypt_sha256', + bcrypt_sha256__rounds=app.config['CREDENTIAL_ROUNDS'], + deprecated='auto' ) def check_password(self, password): context = self.get_password_context() - reference = re.match('({[^}]+})?(.*)', self.password).group(2) - result = context.verify(password, reference) - if result and context.identify(reference) != context.default_scheme(): - self.set_password(password) + # {scheme} will most likely be migrated on first use + reference = self.password + if self.password.startswith("{"): + reference = re.match('({[^}]+})?(.*)', reference).group(2) + + result, new_hash = context.verify_and_update(password, reference) + if new_hash: + self.password = new_hash db.session.add(self) db.session.commit() return result @@ -389,13 +402,11 @@ class User(Base, Email): """Set password for user with specified encryption scheme @password: plain text password to encrypt (if raw == True the hash itself) """ - if hash_scheme is None: - hash_scheme = app.config['PASSWORD_SCHEME'] - # for the list of hash schemes see https://wiki2.dovecot.org/Authentication/PasswordSchemes if raw: - self.password = '{'+hash_scheme+'}' + password + self.password = password else: - self.password = '{'+hash_scheme+'}' + self.get_password_context().encrypt(password, self.scheme_dict[hash_scheme]) + self.password = self.get_password_context().hash(password) + app.cache.delete(self.get_id()) def get_managed_domains(self): if self.global_admin: diff --git a/core/admin/requirements-prod.txt b/core/admin/requirements-prod.txt index a3c32855..f767f431 100644 --- a/core/admin/requirements-prod.txt +++ b/core/admin/requirements-prod.txt @@ -29,7 +29,7 @@ limits==1.3 Mako==1.0.9 MarkupSafe==1.1.1 mysqlclient==1.4.2.post1 -passlib==1.7.1 +passlib==1.7.4 psycopg2==2.8.2 pycparser==2.19 pyOpenSSL==19.0.0 diff --git a/docs/cli.rst b/docs/cli.rst index a9cff41c..8e94026b 100644 --- a/docs/cli.rst +++ b/docs/cli.rst @@ -85,7 +85,6 @@ where mail-config.yml looks like: - localpart: foo domain: example.com password_hash: klkjhumnzxcjkajahsdqweqqwr - hash_scheme: MD5-CRYPT aliases: - localpart: alias1 diff --git a/docs/compose/.env b/docs/compose/.env index 7f91c270..432b20b0 100644 --- a/docs/compose/.env +++ b/docs/compose/.env @@ -144,9 +144,8 @@ LOG_DRIVER=json-file # Docker-compose project name, this will prepended to containers names. COMPOSE_PROJECT_NAME=mailu -# Default password scheme used for newly created accounts and changed passwords -# (value: PBKDF2, BLF-CRYPT, SHA512-CRYPT, SHA256-CRYPT) -PASSWORD_SCHEME=PBKDF2 +# Number of rounds used by the password hashing scheme +CREDENTIAL_ROUNDS=12 # Header to take the real ip from REAL_IP_HEADER= diff --git a/docs/configuration.rst b/docs/configuration.rst index 9123054c..bc2027c6 100644 --- a/docs/configuration.rst +++ b/docs/configuration.rst @@ -138,9 +138,7 @@ Depending on your particular deployment you most probably will want to change th Advanced settings ----------------- -The ``PASSWORD_SCHEME`` is the password encryption scheme. You should use the -default value, unless you are importing password from a separate system and -want to keep using the old password encryption scheme. +The ``CREDENTIAL_ROUNDS`` (default: 12) is the number of rounds used by the password hashing scheme. You should use the default value. The ``SESSION_COOKIE_SECURE`` (default: True) setting controls the secure flag on the cookies of the administrative interface. It should only be turned off if you intend to access it over plain HTTP. From 57a6abaf50400802be5da48913c75cff00dce6f4 Mon Sep 17 00:00:00 2001 From: Florent Daigniere Date: Sun, 7 Feb 2021 09:31:07 +0100 Subject: [PATCH 119/596] Remove {scheme} from the DB if mailu has set it --- core/admin/mailu/models.py | 8 ++++++-- 1 file changed, 6 insertions(+), 2 deletions(-) diff --git a/core/admin/mailu/models.py b/core/admin/mailu/models.py index 905af4a2..9ab9088e 100644 --- a/core/admin/mailu/models.py +++ b/core/admin/mailu/models.py @@ -386,10 +386,14 @@ class User(Base, Email): def check_password(self, password): context = self.get_password_context() - # {scheme} will most likely be migrated on first use reference = self.password + # strip {scheme} if that's something mailu has added + # passlib will identify *crypt based hashes just fine + # on its own if self.password.startswith("{"): - reference = re.match('({[^}]+})?(.*)', reference).group(2) + scheme = self.password.split('}')[0][1:] + if scheme in scheme_dict: + reference = reference[len(scheme)+2:] result, new_hash = context.verify_and_update(password, reference) if new_hash: From 927bd2bd8ec8051e93609d3e8fd24706cc8dc8a2 Mon Sep 17 00:00:00 2001 From: Florent Daigniere Date: Sun, 7 Feb 2021 17:29:58 +0100 Subject: [PATCH 120/596] towncrier --- towncrier/newsfragments/1194.misc | 1 + towncrier/newsfragments/1662.feature | 1 + 2 files changed, 2 insertions(+) create mode 100644 towncrier/newsfragments/1194.misc create mode 100644 towncrier/newsfragments/1662.feature diff --git a/towncrier/newsfragments/1194.misc b/towncrier/newsfragments/1194.misc new file mode 100644 index 00000000..7cbf2b94 --- /dev/null +++ b/towncrier/newsfragments/1194.misc @@ -0,0 +1 @@ +Switch to bcrypt_sha256, remove PASSWORD_SCHEME and replace it with CREDENTIAL_ROUNDS diff --git a/towncrier/newsfragments/1662.feature b/towncrier/newsfragments/1662.feature new file mode 100644 index 00000000..4fc8b2fd --- /dev/null +++ b/towncrier/newsfragments/1662.feature @@ -0,0 +1 @@ +Enable support of all hash types passlib supports. Convert them to the default scheme on first use. From fda758e2b4ea19751369dd122797c5210e18dc15 Mon Sep 17 00:00:00 2001 From: Florent Daigniere Date: Sun, 7 Feb 2021 17:34:22 +0100 Subject: [PATCH 121/596] remove merge artifact --- core/admin/mailu/models.py | 3 +-- 1 file changed, 1 insertion(+), 2 deletions(-) diff --git a/core/admin/mailu/models.py b/core/admin/mailu/models.py index 9ab9088e..b7a4d501 100644 --- a/core/admin/mailu/models.py +++ b/core/admin/mailu/models.py @@ -392,7 +392,7 @@ class User(Base, Email): # on its own if self.password.startswith("{"): scheme = self.password.split('}')[0][1:] - if scheme in scheme_dict: + if scheme in self.scheme_dict: reference = reference[len(scheme)+2:] result, new_hash = context.verify_and_update(password, reference) @@ -410,7 +410,6 @@ class User(Base, Email): self.password = password else: self.password = self.get_password_context().hash(password) - app.cache.delete(self.get_id()) def get_managed_domains(self): if self.global_admin: From 89d88e0c19476692abef0b0b13c4765f2db7bacb Mon Sep 17 00:00:00 2001 From: Florent Daigniere Date: Mon, 8 Feb 2021 08:50:32 +0100 Subject: [PATCH 122/596] Fix the test --- tests/compose/core/00_create_users.sh | 4 ++-- 1 file changed, 2 insertions(+), 2 deletions(-) diff --git a/tests/compose/core/00_create_users.sh b/tests/compose/core/00_create_users.sh index 49d0511b..f5108302 100755 --- a/tests/compose/core/00_create_users.sh +++ b/tests/compose/core/00_create_users.sh @@ -6,6 +6,6 @@ echo "The above error was intended!" docker-compose -f tests/compose/core/docker-compose.yml exec -T admin flask mailu admin admin mailu.io 'FooBar' --mode=ifmissing || exit 1 # Should not fail and update the password; update mode docker-compose -f tests/compose/core/docker-compose.yml exec -T admin flask mailu admin admin mailu.io 'password' --mode=update || exit 1 -docker-compose -f tests/compose/core/docker-compose.yml exec -T admin flask mailu user user mailu.io 'password' 'SHA512-CRYPT' || exit 1 -docker-compose -f tests/compose/core/docker-compose.yml exec -T admin flask mailu user 'user/with/slash' mailu.io 'password' 'SHA512-CRYPT' || exit 1 +docker-compose -f tests/compose/core/docker-compose.yml exec -T admin flask mailu user user mailu.io 'password' || exit 1 +docker-compose -f tests/compose/core/docker-compose.yml exec -T admin flask mailu user 'user/with/slash' mailu.io 'password' || exit 1 echo "User testing succesfull!" From 29306d5abbd17ebd9b479d9438ccfa5cc00a3de6 Mon Sep 17 00:00:00 2001 From: Florent Daigniere Date: Mon, 8 Feb 2021 08:56:03 +0100 Subject: [PATCH 123/596] Fix the tests (again) --- tests/compose/core/02_forward_test.sh | 2 -- tests/compose/core/04_reply_test.sh | 2 -- 2 files changed, 4 deletions(-) diff --git a/tests/compose/core/02_forward_test.sh b/tests/compose/core/02_forward_test.sh index 595820cf..a53fa459 100755 --- a/tests/compose/core/02_forward_test.sh +++ b/tests/compose/core/02_forward_test.sh @@ -2,7 +2,6 @@ cat << EOF | docker-compose -f tests/compose/core/docker-compose.yml exec -T adm users: - localpart: forwardinguser password_hash: "\$1\$F2OStvi1\$Q8hBIHkdJpJkJn/TrMIZ9/" - hash_scheme: MD5-CRYPT domain: mailu.io forward_enabled: true forward_destination: ["user@mailu.io"] @@ -14,7 +13,6 @@ cat << EOF | docker-compose -f tests/compose/core/docker-compose.yml exec -T adm users: - localpart: forwardinguser password_hash: "\$1\$F2OStvi1\$Q8hBIHkdJpJkJn/TrMIZ9/" - hash_scheme: MD5-CRYPT domain: mailu.io forward_enabled: false forward_destination: [] diff --git a/tests/compose/core/04_reply_test.sh b/tests/compose/core/04_reply_test.sh index 83c114f6..e1479cf0 100755 --- a/tests/compose/core/04_reply_test.sh +++ b/tests/compose/core/04_reply_test.sh @@ -2,7 +2,6 @@ cat << EOF | docker-compose -f tests/compose/core/docker-compose.yml exec -T adm users: - localpart: replyuser password_hash: "\$1\$F2OStvi1\$Q8hBIHkdJpJkJn/TrMIZ9/" - hash_scheme: MD5-CRYPT domain: mailu.io reply_enabled: true reply_subject: This will not reach me @@ -15,7 +14,6 @@ cat << EOF | docker-compose -f tests/compose/core/docker-compose.yml exec -T adm users: - localpart: replyuser password_hash: "\$1\$F2OStvi1\$Q8hBIHkdJpJkJn/TrMIZ9/" - hash_scheme: MD5-CRYPT domain: mailu.io reply_enabled: false EOF From d0b34f8e240a1049ed5e1ccd3399af3ff18236e2 Mon Sep 17 00:00:00 2001 From: Florent Daigniere Date: Tue, 9 Feb 2021 08:56:06 +0100 Subject: [PATCH 124/596] Move CREDENTIAL_ROUNDS to advanced settings --- core/admin/mailu/configuration.py | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/core/admin/mailu/configuration.py b/core/admin/mailu/configuration.py index cdee1084..429e778c 100644 --- a/core/admin/mailu/configuration.py +++ b/core/admin/mailu/configuration.py @@ -33,7 +33,6 @@ DEFAULT_CONFIG = { 'TLS_FLAVOR': 'cert', 'AUTH_RATELIMIT': '10/minute;1000/hour', 'AUTH_RATELIMIT_SUBNET': True, - 'CREDENTIAL_ROUNDS': 12, 'DISABLE_STATISTICS': False, # Mail settings 'DMARC_RUA': None, @@ -55,6 +54,7 @@ DEFAULT_CONFIG = { # Advanced settings 'LOG_LEVEL': 'WARNING', 'SESSION_COOKIE_SECURE': True, + 'CREDENTIAL_ROUNDS': 12, # Host settings 'HOST_IMAP': 'imap', 'HOST_LMTP': 'imap:2525', From f9ed517b394e3d267fcae8f960e2299df9bb389f Mon Sep 17 00:00:00 2001 From: Florent Daigniere Date: Tue, 9 Feb 2021 09:02:09 +0100 Subject: [PATCH 125/596] Be specific token length --- core/admin/mailu/ui/views/tokens.py | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/core/admin/mailu/ui/views/tokens.py b/core/admin/mailu/ui/views/tokens.py index 069587e1..820dd405 100644 --- a/core/admin/mailu/ui/views/tokens.py +++ b/core/admin/mailu/ui/views/tokens.py @@ -26,7 +26,7 @@ def token_create(user_email): form = forms.TokenForm() wtforms_components.read_only(form.displayed_password) if not form.raw_password.data: - form.raw_password.data = pwd.genword(entropy=128, charset="hex") + form.raw_password.data = pwd.genword(entropy=128, length=32, charset="hex") form.displayed_password.data = form.raw_password.data if form.validate_on_submit(): token = models.Token(user=user) From df230cb482777e0b3c06e26174af203b5f3070b7 Mon Sep 17 00:00:00 2001 From: Florent Daigniere Date: Tue, 9 Feb 2021 09:20:02 +0100 Subject: [PATCH 126/596] Refactor auth under nginx.check_credentials() --- core/admin/mailu/internal/nginx.py | 32 ++++++++++++------------- core/admin/mailu/internal/views/auth.py | 20 ++++------------ 2 files changed, 19 insertions(+), 33 deletions(-) diff --git a/core/admin/mailu/internal/nginx.py b/core/admin/mailu/internal/nginx.py index de4248fa..f9ebbf13 100644 --- a/core/admin/mailu/internal/nginx.py +++ b/core/admin/mailu/internal/nginx.py @@ -19,6 +19,20 @@ STATUSES = { }), } +def check_credentials(user, password, ip, protocol=None): + if not user or not user.enabled or (protocol == "imap" and not user.enable_imap) or (protocol == "pop3" and not user.enable_pop): + return False + is_ok = False + # All tokens are 32 characters hex lowercase + if len(password) == 32: + for token in user.tokens: + if (token.check_password(password) and + (not token.ip or token.ip == ip)): + is_ok = True + break + if not is_ok and user.check_password(password): + is_ok = True + return is_ok def handle_authentication(headers): """ Handle an HTTP nginx authentication request @@ -47,23 +61,7 @@ def handle_authentication(headers): password = raw_password.encode("iso8859-1").decode("utf8") ip = urllib.parse.unquote(headers["Client-Ip"]) user = models.User.query.get(user_email) - status = False - if user: - # All tokens are 32 characters hex lowercase - if len(password) == 32: - for token in user.tokens: - if (token.check_password(password) and - (not token.ip or token.ip == ip)): - status = True - break - if not status and user.check_password(password): - status = True - if status: - if protocol == "imap" and not user.enable_imap: - status = False - elif protocol == "pop3" and not user.enable_pop: - status = False - if status and user.enabled: + if check_credentials(user, password, ip, protocol): return { "Auth-Status": "OK", "Auth-Server": server, diff --git a/core/admin/mailu/internal/views/auth.py b/core/admin/mailu/internal/views/auth.py index 26d57b3d..edd62e37 100644 --- a/core/admin/mailu/internal/views/auth.py +++ b/core/admin/mailu/internal/views/auth.py @@ -53,22 +53,10 @@ def basic_authentication(): encoded = authorization.replace("Basic ", "") user_email, password = base64.b64decode(encoded).split(b":") user = models.User.query.get(user_email.decode("utf8")) - if user and user.enabled: - password = password.decode('utf-8') - status = False - # All tokens are 32 characters hex lowercase - if len(password) == 32: - for token in user.tokens: - if (token.check_password(password) and - (not token.ip or token.ip == ip)): - status = True - break - if not status and user.check_password(password): - status = True - if status: - response = flask.Response() - response.headers["X-User"] = user.email - return response + if nginx.check_credentials(user, password.decode('utf-8'), flask.request.remote_addr, "web"): + response = flask.Response() + response.headers["X-User"] = user.email + return response response = flask.Response(status=401) response.headers["WWW-Authenticate"] = 'Basic realm="Login Required"' return response From 20d2b621aa42793eedb55219c01da3b9b8ee32f2 Mon Sep 17 00:00:00 2001 From: Florent Daigniere Date: Tue, 9 Feb 2021 09:33:37 +0100 Subject: [PATCH 127/596] Improve the description of CREDENTIAL_ROUNDS --- docs/configuration.rst | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/docs/configuration.rst b/docs/configuration.rst index bc2027c6..26bdb024 100644 --- a/docs/configuration.rst +++ b/docs/configuration.rst @@ -138,7 +138,7 @@ Depending on your particular deployment you most probably will want to change th Advanced settings ----------------- -The ``CREDENTIAL_ROUNDS`` (default: 12) is the number of rounds used by the password hashing scheme. You should use the default value. +The ``CREDENTIAL_ROUNDS`` (default: 12) setting is the number of rounds used by the password hashing scheme. The number of rounds can be reduced in case faster authentication is needed or increased when additional protection is desired. Keep in mind that this is a mitigation against offline attacks on password hashes, aiming to prevent credential stuffing (due to password re-use) on other systems. The ``SESSION_COOKIE_SECURE`` (default: True) setting controls the secure flag on the cookies of the administrative interface. It should only be turned off if you intend to access it over plain HTTP. From 45e5cb9bb37b49f294c973b98ed9cf8f0607498b Mon Sep 17 00:00:00 2001 From: Florent Daigniere Date: Tue, 9 Feb 2021 09:37:13 +0100 Subject: [PATCH 128/596] Improve the towncrier messages --- towncrier/newsfragments/1194.misc | 1 - towncrier/newsfragments/1662.feature | 2 +- towncrier/newsfragments/1753.feature | 1 + 3 files changed, 2 insertions(+), 2 deletions(-) delete mode 100644 towncrier/newsfragments/1194.misc create mode 100644 towncrier/newsfragments/1753.feature diff --git a/towncrier/newsfragments/1194.misc b/towncrier/newsfragments/1194.misc deleted file mode 100644 index 7cbf2b94..00000000 --- a/towncrier/newsfragments/1194.misc +++ /dev/null @@ -1 +0,0 @@ -Switch to bcrypt_sha256, remove PASSWORD_SCHEME and replace it with CREDENTIAL_ROUNDS diff --git a/towncrier/newsfragments/1662.feature b/towncrier/newsfragments/1662.feature index 4fc8b2fd..f8219757 100644 --- a/towncrier/newsfragments/1662.feature +++ b/towncrier/newsfragments/1662.feature @@ -1 +1 @@ -Enable support of all hash types passlib supports. Convert them to the default scheme on first use. +Enable support of all hash types passlib supports. diff --git a/towncrier/newsfragments/1753.feature b/towncrier/newsfragments/1753.feature new file mode 100644 index 00000000..09eb834a --- /dev/null +++ b/towncrier/newsfragments/1753.feature @@ -0,0 +1 @@ +Switch to bcrypt_sha256, replace PASSWORD_SCHEME with CREDENTIAL_ROUNDS and dynamically update existing hashes on first login From 1c5b58cba4da0b411aa4cd6bb911196e8a0fbf25 Mon Sep 17 00:00:00 2001 From: Florent Daigniere Date: Tue, 9 Feb 2021 11:19:28 +0100 Subject: [PATCH 129/596] Remove scheme_dict --- core/admin/mailu/models.py | 9 +-------- 1 file changed, 1 insertion(+), 8 deletions(-) diff --git a/core/admin/mailu/models.py b/core/admin/mailu/models.py index b7a4d501..fab2103a 100644 --- a/core/admin/mailu/models.py +++ b/core/admin/mailu/models.py @@ -362,13 +362,6 @@ class User(Base, Email): self.reply_enddate > now ) - scheme_dict = {'PBKDF2': "pbkdf2_sha512", - 'BLF-CRYPT': "bcrypt", - 'SHA512-CRYPT': "sha512_crypt", - 'SHA256-CRYPT': "sha256_crypt", - 'MD5-CRYPT': "md5_crypt", - 'CRYPT': "des_crypt"} - def get_password_context(self): schemes = registry.list_crypt_handlers() # scrypt throws a warning if the native wheels aren't found @@ -392,7 +385,7 @@ class User(Base, Email): # on its own if self.password.startswith("{"): scheme = self.password.split('}')[0][1:] - if scheme in self.scheme_dict: + if scheme in ['PBKDF2', 'BLF-CRYPT', 'SHA512-CRYPT', 'SHA256-CRYPT', 'MD5-CRYPT', 'CRYPT']: reference = reference[len(scheme)+2:] result, new_hash = context.verify_and_update(password, reference) From 5f05fee8b32209d4888de324cc3e3a578ff83715 Mon Sep 17 00:00:00 2001 From: Florent Daigniere Date: Tue, 9 Feb 2021 11:23:49 +0100 Subject: [PATCH 130/596] Don't need regexps anymore --- core/admin/mailu/models.py | 1 - 1 file changed, 1 deletion(-) diff --git a/core/admin/mailu/models.py b/core/admin/mailu/models.py index fab2103a..c8426fa2 100644 --- a/core/admin/mailu/models.py +++ b/core/admin/mailu/models.py @@ -8,7 +8,6 @@ from flask import current_app as app import flask_sqlalchemy import sqlalchemy -import re import time import os import glob From 96ae54d04db3ff26edf8413e99dc980d610562a8 Mon Sep 17 00:00:00 2001 From: Florent Daigniere Date: Thu, 11 Feb 2021 23:14:09 +0100 Subject: [PATCH 131/596] CryptContext should be a singleton --- core/admin/mailu/models.py | 14 +++++++++----- 1 file changed, 9 insertions(+), 5 deletions(-) diff --git a/core/admin/mailu/models.py b/core/admin/mailu/models.py index c8426fa2..a63c33a5 100644 --- a/core/admin/mailu/models.py +++ b/core/admin/mailu/models.py @@ -304,6 +304,7 @@ class User(Base, Email): """ A user is an email address that has a password to access a mailbox. """ __tablename__ = "user" + _ctx = None domain = db.relationship(Domain, backref=db.backref('users', cascade='all, delete-orphan')) @@ -361,7 +362,10 @@ class User(Base, Email): self.reply_enddate > now ) - def get_password_context(self): + def get_password_context(): + if User._ctx: + return User._ctx + schemes = registry.list_crypt_handlers() # scrypt throws a warning if the native wheels aren't found schemes.remove('scrypt') @@ -369,15 +373,15 @@ class User(Base, Email): for scheme in schemes: if scheme.endswith('plaintext'): schemes.remove(scheme) - return context.CryptContext( + User._ctx = context.CryptContext( schemes=schemes, default='bcrypt_sha256', bcrypt_sha256__rounds=app.config['CREDENTIAL_ROUNDS'], deprecated='auto' ) + return User._ctx def check_password(self, password): - context = self.get_password_context() reference = self.password # strip {scheme} if that's something mailu has added # passlib will identify *crypt based hashes just fine @@ -387,7 +391,7 @@ class User(Base, Email): if scheme in ['PBKDF2', 'BLF-CRYPT', 'SHA512-CRYPT', 'SHA256-CRYPT', 'MD5-CRYPT', 'CRYPT']: reference = reference[len(scheme)+2:] - result, new_hash = context.verify_and_update(password, reference) + result, new_hash = User.get_password_context().verify_and_update(password, reference) if new_hash: self.password = new_hash db.session.add(self) @@ -401,7 +405,7 @@ class User(Base, Email): if raw: self.password = password else: - self.password = self.get_password_context().hash(password) + self.password = User.get_password_context().hash(password) def get_managed_domains(self): if self.global_admin: From 22af5b8432b1f229d8a4a29e5ee46b1fe0c62bba Mon Sep 17 00:00:00 2001 From: Florent Daigniere Date: Sat, 20 Feb 2021 19:10:20 +0100 Subject: [PATCH 132/596] Switch to server-side sessions in redis --- core/admin/mailu/__init__.py | 4 ++++ core/admin/requirements-prod.txt | 1 + core/admin/requirements.txt | 1 + 3 files changed, 6 insertions(+) diff --git a/core/admin/mailu/__init__.py b/core/admin/mailu/__init__.py index 4de3e580..4d7efba1 100644 --- a/core/admin/mailu/__init__.py +++ b/core/admin/mailu/__init__.py @@ -1,5 +1,8 @@ import flask import flask_bootstrap +import redis +from flask_kvsession import KVSessionExtension +from simplekv.memory.redisstore import RedisStore from mailu import utils, debug, models, manage, configuration @@ -17,6 +20,7 @@ def create_app_from_config(config): # Initialize application extensions config.init_app(app) models.db.init_app(app) + KVSessionExtension(RedisStore(redis.StrictRedis().from_url('redis://{0}/3'.format(config['REDIS_ADDRESS']))), app) utils.limiter.init_app(app) utils.babel.init_app(app) utils.login.init_app(app) diff --git a/core/admin/requirements-prod.txt b/core/admin/requirements-prod.txt index f767f431..54cf9a14 100644 --- a/core/admin/requirements-prod.txt +++ b/core/admin/requirements-prod.txt @@ -13,6 +13,7 @@ Flask==1.0.2 Flask-Babel==0.12.2 Flask-Bootstrap==3.3.7.1 Flask-DebugToolbar==0.10.1 +Flask-KVSession==0.6.2 Flask-Limiter==1.0.1 Flask-Login==0.4.1 Flask-Migrate==2.4.0 diff --git a/core/admin/requirements.txt b/core/admin/requirements.txt index 9739ed3f..abb37234 100644 --- a/core/admin/requirements.txt +++ b/core/admin/requirements.txt @@ -3,6 +3,7 @@ Flask-Login Flask-SQLAlchemy Flask-bootstrap Flask-Babel +Flask-KVSession Flask-migrate Flask-script Flask-wtf From d459c374322f219ab12801d17817c12c628f1fdc Mon Sep 17 00:00:00 2001 From: Florent Daigniere Date: Mon, 22 Feb 2021 20:34:06 +0100 Subject: [PATCH 133/596] make session IDs 128bits --- core/admin/mailu/configuration.py | 1 + 1 file changed, 1 insertion(+) diff --git a/core/admin/mailu/configuration.py b/core/admin/mailu/configuration.py index 429e778c..a9ab937f 100644 --- a/core/admin/mailu/configuration.py +++ b/core/admin/mailu/configuration.py @@ -135,6 +135,7 @@ class ConfigManager(dict): self.config['QUOTA_STORAGE_URL'] = 'redis://{0}/1'.format(self.config['REDIS_ADDRESS']) self.config['SESSION_COOKIE_SAMESITE'] = 'Strict' self.config['SESSION_COOKIE_HTTPONLY'] = True + self.config['SESSION_KEY_BITS'] = 128 # update the app config itself app.config = self From a1d32568d6aceec8b0a2a0fd0514714585020edc Mon Sep 17 00:00:00 2001 From: Florent Daigniere Date: Mon, 22 Feb 2021 20:43:52 +0100 Subject: [PATCH 134/596] Regenerate session-ids to prevent session fixation --- core/admin/mailu/ui/views/base.py | 2 ++ core/admin/mailu/ui/views/users.py | 2 ++ 2 files changed, 4 insertions(+) diff --git a/core/admin/mailu/ui/views/base.py b/core/admin/mailu/ui/views/base.py index 7501a883..625c02e1 100644 --- a/core/admin/mailu/ui/views/base.py +++ b/core/admin/mailu/ui/views/base.py @@ -17,6 +17,7 @@ def login(): if form.validate_on_submit(): user = models.User.login(form.email.data, form.pw.data) if user: + flask.session.regenerate() flask_login.login_user(user) endpoint = flask.request.args.get('next', '.index') return flask.redirect(flask.url_for(endpoint) @@ -30,6 +31,7 @@ def login(): @access.authenticated def logout(): flask_login.logout_user() + flask.session.destroy() return flask.redirect(flask.url_for('.index')) diff --git a/core/admin/mailu/ui/views/users.py b/core/admin/mailu/ui/views/users.py index 8830ff5b..2784fe53 100644 --- a/core/admin/mailu/ui/views/users.py +++ b/core/admin/mailu/ui/views/users.py @@ -119,6 +119,7 @@ def user_password(user_email): if form.pw.data != form.pw2.data: flask.flash('Passwords do not match', 'error') else: + flask.session.regenerate() user.set_password(form.pw.data) models.db.session.commit() flask.flash('Password updated for %s' % user) @@ -186,6 +187,7 @@ def user_signup(domain_name=None): if domain.has_email(form.localpart.data): flask.flash('Email is already used', 'error') else: + flask.session.regenerate() user = models.User(domain=domain) form.populate_obj(user) user.set_password(form.pw.data) From b9becd86497fa685e80cca2ccbe20d54405e6d24 Mon Sep 17 00:00:00 2001 From: Florent Daigniere Date: Mon, 22 Feb 2021 21:15:25 +0100 Subject: [PATCH 135/596] make sessions expire --- core/admin/mailu/configuration.py | 3 +++ docs/configuration.rst | 2 ++ 2 files changed, 5 insertions(+) diff --git a/core/admin/mailu/configuration.py b/core/admin/mailu/configuration.py index a9ab937f..0f50bc95 100644 --- a/core/admin/mailu/configuration.py +++ b/core/admin/mailu/configuration.py @@ -1,5 +1,6 @@ import os +from datetime import timedelta from socrate import system DEFAULT_CONFIG = { @@ -53,6 +54,7 @@ DEFAULT_CONFIG = { 'RECAPTCHA_PRIVATE_KEY': '', # Advanced settings 'LOG_LEVEL': 'WARNING', + 'SESSION_LIFETIME': 24, 'SESSION_COOKIE_SECURE': True, 'CREDENTIAL_ROUNDS': 12, # Host settings @@ -136,6 +138,7 @@ class ConfigManager(dict): self.config['SESSION_COOKIE_SAMESITE'] = 'Strict' self.config['SESSION_COOKIE_HTTPONLY'] = True self.config['SESSION_KEY_BITS'] = 128 + self.config['PERMANENT_SESSION_LIFETIME'] = timedelta(hours=int(self.config['SESSION_LIFETIME'])) # update the app config itself app.config = self diff --git a/docs/configuration.rst b/docs/configuration.rst index 26bdb024..7cb53d13 100644 --- a/docs/configuration.rst +++ b/docs/configuration.rst @@ -142,6 +142,8 @@ The ``CREDENTIAL_ROUNDS`` (default: 12) setting is the number of rounds used by The ``SESSION_COOKIE_SECURE`` (default: True) setting controls the secure flag on the cookies of the administrative interface. It should only be turned off if you intend to access it over plain HTTP. +``SESSION_LIFETIME`` (default: 24) is the length in hours a session is valid for on the administrative interface. + The ``LOG_LEVEL`` setting is used by the python start-up scripts as a logging threshold. Log messages equal or higher than this priority will be printed. Can be one of: CRITICAL, ERROR, WARNING, INFO, DEBUG or NOTSET. From 481cb6739216d168e6c439852a7db2e441b13f68 Mon Sep 17 00:00:00 2001 From: Florent Daigniere Date: Mon, 22 Feb 2021 21:18:06 +0100 Subject: [PATCH 136/596] cleanup old sessions on startup --- core/admin/mailu/__init__.py | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/core/admin/mailu/__init__.py b/core/admin/mailu/__init__.py index 4d7efba1..f9ca2466 100644 --- a/core/admin/mailu/__init__.py +++ b/core/admin/mailu/__init__.py @@ -20,7 +20,7 @@ def create_app_from_config(config): # Initialize application extensions config.init_app(app) models.db.init_app(app) - KVSessionExtension(RedisStore(redis.StrictRedis().from_url('redis://{0}/3'.format(config['REDIS_ADDRESS']))), app) + KVSessionExtension(RedisStore(redis.StrictRedis().from_url('redis://{0}/3'.format(config['REDIS_ADDRESS']))), app).cleanup_sessions(app) utils.limiter.init_app(app) utils.babel.init_app(app) utils.login.init_app(app) From 64d757582d2f3531604503d3608dd2815a591c72 Mon Sep 17 00:00:00 2001 From: Florent Daigniere Date: Mon, 22 Feb 2021 21:59:15 +0100 Subject: [PATCH 137/596] Disable anti-csrf on the login form The rationale is that the attacker doesn't have the password... and that doing it this way we avoid creating useless sessions --- core/admin/mailu/ui/forms.py | 2 ++ 1 file changed, 2 insertions(+) diff --git a/core/admin/mailu/ui/forms.py b/core/admin/mailu/ui/forms.py index 356137e8..32bb31ab 100644 --- a/core/admin/mailu/ui/forms.py +++ b/core/admin/mailu/ui/forms.py @@ -46,6 +46,8 @@ class ConfirmationForm(flask_wtf.FlaskForm): class LoginForm(flask_wtf.FlaskForm): + class Meta: + csrf = False email = fields.StringField(_('E-mail'), [validators.Email()]) pw = fields.PasswordField(_('Password'), [validators.DataRequired()]) submit = fields.SubmitField(_('Sign in')) From 513d2a4c5ef6930fe4d6f6e7371039233227dfb5 Mon Sep 17 00:00:00 2001 From: Florent Daigniere Date: Tue, 9 Mar 2021 19:43:08 +0100 Subject: [PATCH 138/596] Fix bug #1660: nested headers shouldn't be touched --- core/postfix/conf/master.cf | 1 + 1 file changed, 1 insertion(+) diff --git a/core/postfix/conf/master.cf b/core/postfix/conf/master.cf index b43095ee..e45a8ccf 100644 --- a/core/postfix/conf/master.cf +++ b/core/postfix/conf/master.cf @@ -12,6 +12,7 @@ smtp inet n - n - - smtpd -o cleanup_service_name=outclean outclean unix n - n - 0 cleanup -o header_checks=pcre:/etc/postfix/outclean_header_filter.cf + -o nested_header_checks= # Internal postfix services pickup unix n - n 60 1 pickup From 97be7359fe1d2de4fcf9022325dcb66b387999c2 Mon Sep 17 00:00:00 2001 From: Florent Daigniere Date: Tue, 9 Mar 2021 19:47:34 +0100 Subject: [PATCH 139/596] towncrier --- towncrier/newsfragments/1660.bugfix | 1 + 1 file changed, 1 insertion(+) create mode 100644 towncrier/newsfragments/1660.bugfix diff --git a/towncrier/newsfragments/1660.bugfix b/towncrier/newsfragments/1660.bugfix new file mode 100644 index 00000000..a90fb099 --- /dev/null +++ b/towncrier/newsfragments/1660.bugfix @@ -0,0 +1 @@ +Don't replace nested headers (typically in attached emails) From b872b46097f507ae48fbb2a102207530678c36d7 Mon Sep 17 00:00:00 2001 From: Florent Daigniere Date: Tue, 9 Mar 2021 20:13:31 +0100 Subject: [PATCH 140/596] towncrier --- towncrier/newsfragments/1783.misc | 1 + 1 file changed, 1 insertion(+) create mode 100644 towncrier/newsfragments/1783.misc diff --git a/towncrier/newsfragments/1783.misc b/towncrier/newsfragments/1783.misc new file mode 100644 index 00000000..2ee4c97f --- /dev/null +++ b/towncrier/newsfragments/1783.misc @@ -0,0 +1 @@ +Switch from client side sessions (cookies) to server-side sessions (Redis). This simplies the security model a lot and allows for an easier recovery should a cookie ever land in the hands of an attacker. From e90d5548a651fe933b5977e15c52cd0be954e7dd Mon Sep 17 00:00:00 2001 From: Alexander Graf Date: Wed, 10 Mar 2021 18:30:28 +0100 Subject: [PATCH 141/596] use RFC3339 for last_check fixed to UTC for now --- core/admin/mailu/schemas.py | 1 + 1 file changed, 1 insertion(+) diff --git a/core/admin/mailu/schemas.py b/core/admin/mailu/schemas.py index 68814170..284c1551 100644 --- a/core/admin/mailu/schemas.py +++ b/core/admin/mailu/schemas.py @@ -1135,6 +1135,7 @@ class FetchSchema(BaseSchema): """ Schema config """ model = models.Fetch load_instance = True + datetimeformat = '%Y-%m-%dT%H:%M:%S.%fZ' # RFC3339, but fixed to UTC sibling = True include_by_context = { From c17bfae24050e48ae83cc21196346cfe7c6d90bc Mon Sep 17 00:00:00 2001 From: Alexander Graf Date: Wed, 10 Mar 2021 18:50:25 +0100 Subject: [PATCH 142/596] correct rfc3339 datetime serialization now using correct timezone --- core/admin/mailu/schemas.py | 17 +++++++++++++++-- 1 file changed, 15 insertions(+), 2 deletions(-) diff --git a/core/admin/mailu/schemas.py b/core/admin/mailu/schemas.py index 284c1551..277748f7 100644 --- a/core/admin/mailu/schemas.py +++ b/core/admin/mailu/schemas.py @@ -3,6 +3,7 @@ from copy import deepcopy from collections import Counter +from datetime import timezone import json import logging @@ -455,7 +456,20 @@ class RenderJSON: return json.dumps(*args, **kwargs) -### schema: custom fields ### +### marshmallow: custom fields ### + +def _rfc3339(datetime): + """ dump datetime according to rfc3339 """ + if datetime.tzinfo is None: + datetime = datetime.astimezone(timezone.utc) + res = datetime.isoformat() + if res.endswith('+00:00'): + return f'{res[:-6]}Z' + return res + +fields.DateTime.SERIALIZATION_FUNCS['rfc3339'] = _rfc3339 +fields.DateTime.DESERIALIZATION_FUNCS['rfc3339'] = fields.DateTime.DESERIALIZATION_FUNCS['iso'] +fields.DateTime.DEFAULT_FORMAT = 'rfc3339' class LazyStringField(fields.String): """ Field that serializes a "false" value to the empty string @@ -1135,7 +1149,6 @@ class FetchSchema(BaseSchema): """ Schema config """ model = models.Fetch load_instance = True - datetimeformat = '%Y-%m-%dT%H:%M:%S.%fZ' # RFC3339, but fixed to UTC sibling = True include_by_context = { From ce9a9ec572d8220e673b2817f1350ffb53b3215a Mon Sep 17 00:00:00 2001 From: Alexander Graf Date: Wed, 10 Mar 2021 18:50:52 +0100 Subject: [PATCH 143/596] always init Logger first --- core/admin/mailu/manage.py | 3 ++- 1 file changed, 2 insertions(+), 1 deletion(-) diff --git a/core/admin/mailu/manage.py b/core/admin/mailu/manage.py index f9c58d26..5708327e 100644 --- a/core/admin/mailu/manage.py +++ b/core/admin/mailu/manage.py @@ -375,6 +375,8 @@ def config_export(full=False, secrets=False, color=False, dns=False, output=None """ Export configuration as YAML or JSON to stdout or file """ + log = Logger(want_color=color or None, can_color=output.isatty()) + only = only or MailuSchema.Meta.order context = { @@ -382,7 +384,6 @@ def config_export(full=False, secrets=False, color=False, dns=False, output=None 'secrets': secrets, 'dns': dns, } - log = Logger(want_color=color or None, can_color=output.isatty()) try: schema = MailuSchema(only=only, context=context) From 9cb6962335d00fbb0505e7d571271232d4f54b97 Mon Sep 17 00:00:00 2001 From: Alexander Graf Date: Thu, 11 Mar 2021 18:12:50 +0100 Subject: [PATCH 144/596] Moved MyYamlLexer into logger now cmdline runs without pygments --- core/admin/mailu/schemas.py | 48 ++++++++++++++++++------------------- 1 file changed, 24 insertions(+), 24 deletions(-) diff --git a/core/admin/mailu/schemas.py b/core/admin/mailu/schemas.py index 277748f7..fdd766b3 100644 --- a/core/admin/mailu/schemas.py +++ b/core/admin/mailu/schemas.py @@ -53,34 +53,34 @@ def mapped(cls): _model2schema[cls.Meta.model] = cls return cls -class MyYamlLexer(YamlLexer): - """ colorize yaml constants and integers """ - def get_tokens(self, text, unfiltered=False): - for typ, value in super().get_tokens(text, unfiltered): - if typ is Token.Literal.Scalar.Plain: - if value in {'true', 'false', 'null'}: - typ = Token.Keyword.Constant - elif value == HIDDEN: - typ = Token.Error - else: - try: - int(value, 10) - except ValueError: - try: - float(value) - except ValueError: - pass - else: - typ = Token.Literal.Number.Float - else: - typ = Token.Literal.Number.Integer - yield typ, value - class Logger: """ helps with counting and colorizing imported and exported data """ + class MyYamlLexer(YamlLexer): + """ colorize yaml constants and integers """ + def get_tokens(self, text, unfiltered=False): + for typ, value in super().get_tokens(text, unfiltered): + if typ is Token.Literal.Scalar.Plain: + if value in {'true', 'false', 'null'}: + typ = Token.Keyword.Constant + elif value == HIDDEN: + typ = Token.Error + else: + try: + int(value, 10) + except ValueError: + try: + float(value) + except ValueError: + pass + else: + typ = Token.Literal.Number.Float + else: + typ = Token.Literal.Number.Integer + yield typ, value + def __init__(self, want_color=None, can_color=False, debug=False, secrets=False): self.lexer = 'yaml' @@ -323,7 +323,7 @@ class Logger: return data lexer = lexer or self.lexer - lexer = MyYamlLexer() if lexer == 'yaml' else get_lexer_by_name(lexer) + lexer = Logger.MyYamlLexer() if lexer == 'yaml' else get_lexer_by_name(lexer) formatter = get_formatter_by_name(formatter or self.formatter, colorscheme=self.colorscheme) if strip is None: strip = self.strip From 0c38128c4e459aa6fb783c86bea51ae2e19ac9b7 Mon Sep 17 00:00:00 2001 From: Alexander Graf Date: Thu, 11 Mar 2021 18:38:00 +0100 Subject: [PATCH 145/596] Add pygments to requirements --- core/admin/mailu/schemas.py | 20 ++++++-------------- core/admin/requirements-prod.txt | 1 + core/admin/requirements.txt | 1 + 3 files changed, 8 insertions(+), 14 deletions(-) diff --git a/core/admin/mailu/schemas.py b/core/admin/mailu/schemas.py index fdd766b3..28a5d6f4 100644 --- a/core/admin/mailu/schemas.py +++ b/core/admin/mailu/schemas.py @@ -21,16 +21,11 @@ from flask_marshmallow import Marshmallow from OpenSSL import crypto -try: - from pygments import highlight - from pygments.token import Token - from pygments.lexers import get_lexer_by_name - from pygments.lexers.data import YamlLexer - from pygments.formatters import get_formatter_by_name -except ModuleNotFoundError: - COLOR_SUPPORTED = False -else: - COLOR_SUPPORTED = True +from pygments import highlight +from pygments.token import Token +from pygments.lexers import get_lexer_by_name +from pygments.lexers.data import YamlLexer +from pygments.formatters import get_formatter_by_name from mailu import models, dkim @@ -92,10 +87,7 @@ class Logger: self.debug = debug self.print = print - if want_color and not COLOR_SUPPORTED: - raise ValueError('Please install pygments to colorize output') - - self.color = want_color or (can_color and COLOR_SUPPORTED) + self.color = want_color or can_color self._counter = Counter() self._schemas = {} diff --git a/core/admin/requirements-prod.txt b/core/admin/requirements-prod.txt index 7620fd95..5c291e24 100644 --- a/core/admin/requirements-prod.txt +++ b/core/admin/requirements-prod.txt @@ -36,6 +36,7 @@ marshmallow-sqlalchemy==0.24.1 passlib==1.7.4 psycopg2==2.8.2 pycparser==2.19 +Pygments==2.8.1 pyOpenSSL==19.0.0 python-dateutil==2.8.0 python-editor==1.0.4 diff --git a/core/admin/requirements.txt b/core/admin/requirements.txt index f9d175b3..46500295 100644 --- a/core/admin/requirements.txt +++ b/core/admin/requirements.txt @@ -17,6 +17,7 @@ gunicorn tabulate PyYAML PyOpenSSL +Pygments dnspython bcrypt tenacity From 8bc44455721471a7277196190bb31db9d48d4bce Mon Sep 17 00:00:00 2001 From: Alexander Graf Date: Fri, 12 Mar 2021 17:56:17 +0100 Subject: [PATCH 146/596] Sync update of localpart, domain_name and email --- core/admin/mailu/models.py | 18 +++++++++++++++--- core/admin/mailu/schemas.py | 4 ++-- 2 files changed, 17 insertions(+), 5 deletions(-) diff --git a/core/admin/mailu/models.py b/core/admin/mailu/models.py index b08c5bd2..93efd016 100644 --- a/core/admin/mailu/models.py +++ b/core/admin/mailu/models.py @@ -355,9 +355,21 @@ class Email(object): self.localpart, self.domain_name = value.rsplit('@', 1) self._email = value - # hack for email declared attr - when _email is not updated yet - def __str__(self): - return str(f'{self.localpart}@{self.domain_name}') + @staticmethod + def _update_localpart(target, value, *_): + if target.domain_name: + target._email = f'{value}@{target.domain_name}' + + @staticmethod + def _update_domain_name(target, value, *_): + if target.localpart: + target._email = f'{target.localpart}@{value}' + + @classmethod + def __declare_last__(cls): + # gets called after mappings are completed + sqlalchemy.event.listen(User.localpart, 'set', cls._update_localpart, propagate=True) + sqlalchemy.event.listen(User.domain_name, 'set', cls._update_domain_name, propagate=True) def sendmail(self, subject, body): """ send an email to the address """ diff --git a/core/admin/mailu/schemas.py b/core/admin/mailu/schemas.py index 28a5d6f4..2742edf1 100644 --- a/core/admin/mailu/schemas.py +++ b/core/admin/mailu/schemas.py @@ -1030,8 +1030,8 @@ class BaseSchema(ma.SQLAlchemyAutoSchema, Storage): self.opts.sqla_session.add(item) return item - # stop early if item has no password attribute - if not hasattr(item, 'password'): + # stop early when not updating or item has no password attribute + if not self.context.get('update') or not hasattr(item, 'password'): return item # did we hash a new plaintext password? From 83b1fbb9d67b6752a7d8bb4f8e41e3d35e56ed4d Mon Sep 17 00:00:00 2001 From: Alexander Graf Date: Sun, 14 Mar 2021 18:09:21 +0100 Subject: [PATCH 147/596] Lazy loading of KVSessionExtension - call cleanup_sessions on first kvstore access this allows to run cmdline actions without redis (and makes it faster) - Allow development using DictStore by setting REDIS_ADDRESS to the empty string in env - don't sign 64bit random session id as suggested by nextgens --- core/admin/mailu/__init__.py | 12 +++--- core/admin/mailu/utils.py | 65 +++++++++++++++++++++++++++++++- core/admin/requirements-prod.txt | 1 + 3 files changed, 70 insertions(+), 8 deletions(-) diff --git a/core/admin/mailu/__init__.py b/core/admin/mailu/__init__.py index f9ca2466..40cc9cff 100644 --- a/core/admin/mailu/__init__.py +++ b/core/admin/mailu/__init__.py @@ -1,8 +1,8 @@ +""" Mailu admin app +""" + import flask import flask_bootstrap -import redis -from flask_kvsession import KVSessionExtension -from simplekv.memory.redisstore import RedisStore from mailu import utils, debug, models, manage, configuration @@ -20,7 +20,8 @@ def create_app_from_config(config): # Initialize application extensions config.init_app(app) models.db.init_app(app) - KVSessionExtension(RedisStore(redis.StrictRedis().from_url('redis://{0}/3'.format(config['REDIS_ADDRESS']))), app).cleanup_sessions(app) + utils.kvsession.init_kvstore(config) + utils.kvsession.init_app(app) utils.limiter.init_app(app) utils.babel.init_app(app) utils.login.init_app(app) @@ -53,8 +54,7 @@ def create_app_from_config(config): def create_app(): - """ Create a new application based on the config module + """ Create a new application based on the config module """ config = configuration.ConfigManager() return create_app_from_config(config) - diff --git a/core/admin/mailu/utils.py b/core/admin/mailu/utils.py index ce12a09a..852fe8ad 100644 --- a/core/admin/mailu/utils.py +++ b/core/admin/mailu/utils.py @@ -1,11 +1,18 @@ -from mailu import models, limiter +""" Mailu admin app utilities +""" + +from mailu import limiter import flask import flask_login -import flask_script import flask_migrate import flask_babel +import flask_kvsession +import redis +from simplekv.memory import DictStore +from simplekv.memory.redisstore import RedisStore +from itsdangerous.encoding import want_bytes from werkzeug.contrib import fixers @@ -33,6 +40,10 @@ def get_locale(): # Proxy fixer class PrefixMiddleware(object): + """ fix proxy headers """ + def __init__(self): + self.app = None + def __call__(self, environ, start_response): prefix = environ.get('HTTP_X_FORWARDED_PREFIX', '') if prefix: @@ -48,3 +59,53 @@ proxy = PrefixMiddleware() # Data migrate migrate = flask_migrate.Migrate() + + +# session store +class NullSigner(object): + """NullSigner does not sign nor unsign""" + def __init__(self, *args, **kwargs): + pass + def sign(self, value): + """Signs the given string.""" + return want_bytes(value) + def unsign(self, signed_value): + """Unsigns the given string.""" + return want_bytes(signed_value) + +class KVSessionIntf(flask_kvsession.KVSessionInterface): + """ KVSession interface allowing to run int function on first access """ + def __init__(self, app, init_fn=None): + if init_fn: + app.kvsession_init = init_fn + else: + self._first_run(None) + def _first_run(self, app): + if app: + app.kvsession_init() + self.open_session = super().open_session + self.save_session = super().save_session + def open_session(self, app, request): + self._first_run(app) + return super().open_session(app, request) + def save_session(self, app, session, response): + self._first_run(app) + return super().save_session(app, session, response) + +class KVSessionExt(flask_kvsession.KVSessionExtension): + """ Activates Flask-KVSession for an application. """ + def init_kvstore(self, config): + """ Initialize kvstore - fallback to DictStore without REDIS_ADDRESS """ + if addr := config.get('REDIS_ADDRESS'): + self.default_kvstore = RedisStore(redis.StrictRedis().from_url(f'redis://{addr}/3')) + else: + self.default_kvstore = DictStore() + + def init_app(self, app, session_kvstore=None): + """ Initialize application and KVSession. """ + super().init_app(app, session_kvstore) + app.session_interface = KVSessionIntf(app, self.cleanup_sessions) + +kvsession = KVSessionExt() + +flask_kvsession.Signer = NullSigner diff --git a/core/admin/requirements-prod.txt b/core/admin/requirements-prod.txt index 54cf9a14..cd084684 100644 --- a/core/admin/requirements-prod.txt +++ b/core/admin/requirements-prod.txt @@ -39,6 +39,7 @@ python-editor==1.0.4 pytz==2019.1 PyYAML==5.1 redis==3.2.1 +simplekv==0.14.1 #alpine3:12 provides six==1.15.0 #six==1.12.0 socrate==0.1.1 From f0f79b23a328064a787d99b3755d43a946a87d9b Mon Sep 17 00:00:00 2001 From: Alexander Graf Date: Sun, 14 Mar 2021 21:38:16 +0100 Subject: [PATCH 148/596] Allow cleanup of sessions by key&value in data This can be used to delete all sessions belonging to a user/login. For no it just iterates over all sessions. This could be enhanced by using a prefix for and deleting by prefix. --- core/admin/mailu/utils.py | 32 +++++++++++++++++++++++++++++++- 1 file changed, 31 insertions(+), 1 deletion(-) diff --git a/core/admin/mailu/utils.py b/core/admin/mailu/utils.py index 852fe8ad..9394c38f 100644 --- a/core/admin/mailu/utils.py +++ b/core/admin/mailu/utils.py @@ -1,6 +1,8 @@ """ Mailu admin app utilities """ +from datetime import datetime + from mailu import limiter import flask @@ -22,6 +24,7 @@ login.login_view = "ui.login" @login.unauthorized_handler def handle_needs_login(): + """ redirect unauthorized requests to login page """ return flask.redirect( flask.url_for('ui.login', next=flask.request.endpoint) ) @@ -34,7 +37,8 @@ babel = flask_babel.Babel() @babel.localeselector def get_locale(): - translations = list(map(str, babel.list_translations())) + """ selects locale for translation """ + translations = [str(translation) for translation in babel.list_translations()] return flask.request.accept_languages.best_match(translations) @@ -101,6 +105,32 @@ class KVSessionExt(flask_kvsession.KVSessionExtension): else: self.default_kvstore = DictStore() + def cleanup_sessions(self, app=None, dkey=None, dvalue=None): + """ Remove sessions from the store. """ + if not app: + app = flask.current_app + if dkey is None and dvalue is None: + now = datetime.utcnow() + for key in app.kvsession_store.keys(): + try: + sid = flask_kvsession.SessionID.unserialize(key) + except ValueError: + pass + else: + if sid.has_expired( + app.config['PERMANENT_SESSION_LIFETIME'], + now + ): + app.kvsession_store.delete(key) + elif dkey is not None and dvalue is not None: + for key in app.kvsession_store.keys(): + if app.session_interface.serialization_method.loads( + app.kvsession_store.get(key) + ).get(dkey, None) == dvalue: + app.kvsession_store.delete(key) + else: + raise ValueError('Need dkey and dvalue.') + def init_app(self, app, session_kvstore=None): """ Initialize application and KVSession. """ super().init_app(app, session_kvstore) From 75baa1da993363f1fe6bef10758956502369befd Mon Sep 17 00:00:00 2001 From: ronivay Date: Thu, 18 Mar 2021 09:46:27 +0200 Subject: [PATCH 149/596] Update fail2ban documentation --- docs/faq.rst | 29 +++++++++++++++++++++++------ 1 file changed, 23 insertions(+), 6 deletions(-) diff --git a/docs/faq.rst b/docs/faq.rst index 59a841dc..14dcc4ed 100644 --- a/docs/faq.rst +++ b/docs/faq.rst @@ -528,25 +528,42 @@ The above will block flagged IPs for a week, you can of course change it to you actionstart = iptables -N f2b-bad-auth iptables -A f2b-bad-auth -j RETURN - iptables -I FORWARD -p tcp -m multiport --dports 1:1024 -j f2b-bad-auth + iptables -I DOCKER-USER -p tcp -m multiport --dports 1:1024 -j f2b-bad-auth - actionstop = iptables -D FORWARD -p tcp -m multiport --dports 1:1024 -j f2b-bad-auth + actionstop = iptables -D DOCKER-USER -p tcp -m multiport --dports 1:1024 -j f2b-bad-auth iptables -F f2b-bad-auth iptables -X f2b-bad-auth - actioncheck = iptables -n -L FORWARD | grep -q 'f2b-bad-auth[ \t]' + actioncheck = iptables -n -L DOCKER-USER | grep -q 'f2b-bad-auth[ \t]' actionban = iptables -I f2b-bad-auth 1 -s -j DROP actionunban = iptables -D f2b-bad-auth -s -j DROP -5. Restart Fail2Ban +Using DOCKER-USER chain ensures that blocked IPs are processed in correct order with Docker. See more in: https://docs.docker.com/network/iptables/ + +5. Configure and restart Fail2Ban service + +Make sure Fail2Ban is started after Docker service by adding partial override which appends this to existing configuration.. + +.. code-block:: bash + + sudo systemctl edit fail2ban + +Add override and save file. + +.. code-block:: bash + + [Unit] + After=docker.service + +Restart service. .. code-block:: bash sudo systemctl restart fail2ban -*Issue reference:* `85`_, `116`_, `171`_, `584`_, `592`_. +*Issue reference:* `85`_, `116`_, `171`_, `584`_, `592`_, `1727`_. Users can't change their password from webmail `````````````````````````````````````````````` @@ -670,7 +687,7 @@ iptables -t nat -A POSTROUTING -o eth0 -p tcp --dport 25 -j SNAT --to Date: Fri, 19 Mar 2021 09:59:16 +0100 Subject: [PATCH 150/596] Update faq.rst Some spelling improvements. --- docs/faq.rst | 10 +++++----- 1 file changed, 5 insertions(+), 5 deletions(-) diff --git a/docs/faq.rst b/docs/faq.rst index 14dcc4ed..9c4f1d75 100644 --- a/docs/faq.rst +++ b/docs/faq.rst @@ -540,24 +540,24 @@ The above will block flagged IPs for a week, you can of course change it to you actionunban = iptables -D f2b-bad-auth -s -j DROP -Using DOCKER-USER chain ensures that blocked IPs are processed in correct order with Docker. See more in: https://docs.docker.com/network/iptables/ +Using DOCKER-USER chain ensures that the blocked IPs are processed in the correct order with Docker. See more in: https://docs.docker.com/network/iptables/ -5. Configure and restart Fail2Ban service +5. Configure and restart the Fail2Ban service -Make sure Fail2Ban is started after Docker service by adding partial override which appends this to existing configuration.. +Make sure Fail2Ban is started after the Docker service by adding a partial override which appends this to the existing configuration. .. code-block:: bash sudo systemctl edit fail2ban -Add override and save file. +Add the override and save the file. .. code-block:: bash [Unit] After=docker.service -Restart service. +Restart the Fail2Ban service. .. code-block:: bash From c6d0ef229fe2d06fee304b704e1c2bb1857f8067 Mon Sep 17 00:00:00 2001 From: Vincent Kling Date: Fri, 19 Mar 2021 10:46:42 +0100 Subject: [PATCH 151/596] Update messages.po --- core/admin/mailu/translations/nl/LC_MESSAGES/messages.po | 4 ++-- 1 file changed, 2 insertions(+), 2 deletions(-) diff --git a/core/admin/mailu/translations/nl/LC_MESSAGES/messages.po b/core/admin/mailu/translations/nl/LC_MESSAGES/messages.po index 8d7b5054..c7f6db5e 100644 --- a/core/admin/mailu/translations/nl/LC_MESSAGES/messages.po +++ b/core/admin/mailu/translations/nl/LC_MESSAGES/messages.po @@ -528,7 +528,7 @@ msgstr "Alternatieve naam" #: mailu/ui/forms.py:70 msgid "Relayed domain name" -msgstr "Relayed domainnaam" +msgstr "Relayed domeinnaam" #: mailu/ui/forms.py:71 mailu/ui/templates/relay/list.html:18 msgid "Remote host" @@ -536,7 +536,7 @@ msgstr "Externe host" #: mailu/ui/templates/sidebar.html:54 msgid "Relayed domains" -msgstr "Relayed domainen" +msgstr "Relayed domeinen" #: mailu/ui/templates/alternative/create.html:4 msgid "Create alternative domain" From e46d9e1fc9f5b50a40f785628beb63850e0a6d8a Mon Sep 17 00:00:00 2001 From: DjVinnii Date: Thu, 1 Apr 2021 11:26:37 +0200 Subject: [PATCH 152/596] Update admin-lte version in package.json --- core/admin/package.json | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/core/admin/package.json b/core/admin/package.json index a1107c69..ac18f61e 100644 --- a/core/admin/package.json +++ b/core/admin/package.json @@ -11,7 +11,7 @@ "dependencies": { "@babel/core": "^7.4.4", "@babel/preset-env": "^7.4.4", - "admin-lte": "^2.4.10", + "admin-lte": "^3.1.0", "babel-loader": "^8.0.5", "bootstrap": "^3.4.1", "css-loader": "^2.1.1", From c97728289bbe0e466454832f060276e0fb29702e Mon Sep 17 00:00:00 2001 From: DjVinnii Date: Thu, 1 Apr 2021 11:34:03 +0200 Subject: [PATCH 153/596] Update node version for building the image (AdminLTE requires node 10 or higher) --- core/admin/Dockerfile | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/core/admin/Dockerfile b/core/admin/Dockerfile index f3b8643c..81ffd147 100644 --- a/core/admin/Dockerfile +++ b/core/admin/Dockerfile @@ -1,7 +1,7 @@ # First stage to build assets ARG DISTRO=alpine:3.12 ARG ARCH="" -FROM ${ARCH}node:8 as assets +FROM ${ARCH}node:15 as assets COPY --from=balenalib/rpi-alpine:3.10 /usr/bin/qemu-arm-static /usr/bin/qemu-arm-static COPY package.json ./ From 6b3170cb4c1f9bcda28c25b7c27849cc1de06f7f Mon Sep 17 00:00:00 2001 From: DjVinnii Date: Thu, 1 Apr 2021 14:42:15 +0200 Subject: [PATCH 154/596] Update side menu --- core/admin/assets/vendor.js | 18 +- core/admin/mailu/ui/templates/base.html | 48 ++-- core/admin/mailu/ui/templates/sidebar.html | 254 +++++++++++---------- 3 files changed, 176 insertions(+), 144 deletions(-) diff --git a/core/admin/assets/vendor.js b/core/admin/assets/vendor.js index f7ed03c8..a5ff7a3c 100644 --- a/core/admin/assets/vendor.js +++ b/core/admin/assets/vendor.js @@ -3,17 +3,21 @@ import jQuery from 'jquery'; import 'select2/dist/css/select2.css'; // bootstrap -import 'bootstrap/less/bootstrap.less'; -import 'bootstrap'; +// import 'bootstrap/less/bootstrap.less'; +// import 'bootstrap'; // FA -import 'font-awesome/scss/font-awesome.scss'; +// import 'font-awesome/scss/font-awesome.scss'; // FA is included with AdminLTE +import 'admin-lte/plugins/fontawesome-free/css/fontawesome.css'; +import 'admin-lte/plugins/fontawesome-free/css/regular.css'; +import 'admin-lte/plugins/fontawesome-free/css/solid.css'; // AdminLTE -import 'admin-lte/build/less/AdminLTE-without-plugins.less'; -import 'admin-lte/build/less/select2.less'; -import 'admin-lte/build/less/skins/skin-blue.less'; +import 'admin-lte/build/scss/adminlte.scss'; +// import 'admin-lte/build/less/AdminLTE-without-plugins.less'; // doesn't exist +// import 'admin-lte/build/less/select2.less'; // doesn't exist +// import 'admin-lte/build/less/skins/skin-blue.less'; // doesn't exist import 'admin-lte/build/js/Layout.js'; import 'admin-lte/build/js/ControlSidebar.js'; import 'admin-lte/build/js/PushMenu.js'; -import 'admin-lte/build/js/BoxRefresh.js'; +// import 'admin-lte/build/js/BoxRefresh.js'; // doesn't exist diff --git a/core/admin/mailu/ui/templates/base.html b/core/admin/mailu/ui/templates/base.html index 74d5653c..c5b09159 100644 --- a/core/admin/mailu/ui/templates/base.html +++ b/core/admin/mailu/ui/templates/base.html @@ -8,43 +8,47 @@ Mailu-Admin - {{ config["SITENAME"] }} - +
-
- -
-
diff --git a/core/admin/mailu/ui/templates/sidebar.html b/core/admin/mailu/ui/templates/sidebar.html index 78be75d5..7c86378d 100644 --- a/core/admin/mailu/ui/templates/sidebar.html +++ b/core/admin/mailu/ui/templates/sidebar.html @@ -1,120 +1,144 @@ -
+
{% if current_user.is_authenticated %} -

{{ current_user }}

+
+
+ {{ current_user }} +
+
{% endif %} -
+
  • {% trans %}Go to{% endtrans %}
    • + {% if config["WEBMAIL"] != "none" %} +
  • + + +

    {% trans %}Webmail{% endtrans %}

    +     +
    • + {% endif %} +
  • + + +

    {% trans %}Client setup{% endtrans %}

    +     +
    • +
  • + + +

    {% trans %}Website{% endtrans %}

    +     +
    • +
  • + + +

    {% trans %}Help{% endtrans %}

    +     +
    • + {% if config['DOMAIN_REGISTRATION'] %} +
  • + + +

    {% trans %}Register a domain{% endtrans %}

    +     +
    • + {% endif %} + {% if current_user.is_authenticated %} +
  • + + +

    {% trans %}Sign out{% endtrans %}

    +     +
    • + {% else %} +
  • + + +

    {% trans %}Sign in{% endtrans %}

    +     +
    • + {% if signup_domains %} +
  • + + +

    {% trans %}Sign up{% endtrans %}

    +     +
    • + {% endif %} + {% endif %} + + + From deca6e0c4a311d9156555ab65fb08f850cd4094e Mon Sep 17 00:00:00 2001 From: DjVinnii Date: Thu, 1 Apr 2021 14:45:12 +0200 Subject: [PATCH 155/596] update user/settings --- core/admin/mailu/ui/templates/macros.html | 10 +++++----- core/admin/mailu/ui/templates/user/settings.html | 6 +++--- 2 files changed, 8 insertions(+), 8 deletions(-) diff --git a/core/admin/mailu/ui/templates/macros.html b/core/admin/mailu/ui/templates/macros.html index ec4cf6e4..a1fc2919 100644 --- a/core/admin/mailu/ui/templates/macros.html +++ b/core/admin/mailu/ui/templates/macros.html @@ -64,18 +64,18 @@ {% endmacro %} -{% macro box(title=None, theme="primary", header=True) %} +{% macro card(title=None, theme="primary", header=True) %}
    -
    +
    {% if header %} -
    +
    {% if title %} -

    {{ title }}

    +

    {{ title }}

    {% endif %}
    {% endif %} -
    +
    {{ caller() }}
    diff --git a/core/admin/mailu/ui/templates/user/settings.html b/core/admin/mailu/ui/templates/user/settings.html index b6ade695..e58cc89a 100644 --- a/core/admin/mailu/ui/templates/user/settings.html +++ b/core/admin/mailu/ui/templates/user/settings.html @@ -12,18 +12,18 @@
    {{ form.hidden_tag() }} - {% call macros.box(title=_("Displayed name")) %} + {% call macros.card(title=_("Displayed name")) %} {{ macros.form_field(form.displayed_name) }} {% endcall %} - {% call macros.box(title=_("Antispam")) %} + {% call macros.card(title=_("Antispam")) %} {{ macros.form_field(form.spam_enabled) }} {{ macros.form_field(form.spam_threshold, step=1, max=100, prepend=''+form.spam_threshold.data.__str__()+' / 100', oninput='$("#threshold").text(this.value);') }} {% endcall %} - {% call macros.box(title=_("Auto-forward")) %} + {% call macros.card(title=_("Auto-forward")) %} {{ macros.form_field(form.forward_enabled, onchange="if(this.checked){$('#forward_destination,#forward_keep').removeAttr('disabled')} else{$('#forward_destination,#forward_keep').attr('disabled', '')}") }} From 0c5fda3fca6930d2ba1f6a34df7c223e097d8d12 Mon Sep 17 00:00:00 2001 From: DjVinnii Date: Thu, 1 Apr 2021 14:47:41 +0200 Subject: [PATCH 156/596] Change macros.box to macros.card --- core/admin/mailu/ui/templates/admin/create.html | 2 +- core/admin/mailu/ui/templates/alias/create.html | 2 +- core/admin/mailu/ui/templates/announcement.html | 2 +- core/admin/mailu/ui/templates/client.html | 4 ++-- core/admin/mailu/ui/templates/confirm.html | 2 +- core/admin/mailu/ui/templates/domain/create.html | 2 +- core/admin/mailu/ui/templates/domain/signup.html | 4 ++-- core/admin/mailu/ui/templates/fetch/create.html | 4 ++-- core/admin/mailu/ui/templates/form.html | 2 +- core/admin/mailu/ui/templates/manager/create.html | 2 +- core/admin/mailu/ui/templates/user/create.html | 4 ++-- core/admin/mailu/ui/templates/user/forward.html | 2 +- core/admin/mailu/ui/templates/user/reply.html | 2 +- core/admin/mailu/ui/templates/user/signup.html | 2 +- 14 files changed, 18 insertions(+), 18 deletions(-) diff --git a/core/admin/mailu/ui/templates/admin/create.html b/core/admin/mailu/ui/templates/admin/create.html index 8d3a7b58..6c2413bc 100644 --- a/core/admin/mailu/ui/templates/admin/create.html +++ b/core/admin/mailu/ui/templates/admin/create.html @@ -5,7 +5,7 @@ {% endblock %} {% block content %} -{% call macros.box() %} +{% call macros.card() %} {{ form.hidden_tag() }} {{ macros.form_field(form.admin, class_='mailselect') }} diff --git a/core/admin/mailu/ui/templates/alias/create.html b/core/admin/mailu/ui/templates/alias/create.html index 38d7e7e5..4b0ea3e2 100644 --- a/core/admin/mailu/ui/templates/alias/create.html +++ b/core/admin/mailu/ui/templates/alias/create.html @@ -9,7 +9,7 @@ {% endblock %} {% block content %} -{% call macros.box() %} +{% call macros.card() %} {{ form.hidden_tag() }} {{ macros.form_field(form.localpart, append='@'+domain.name+'') }} diff --git a/core/admin/mailu/ui/templates/announcement.html b/core/admin/mailu/ui/templates/announcement.html index 7dd34d3f..acdbde1a 100644 --- a/core/admin/mailu/ui/templates/announcement.html +++ b/core/admin/mailu/ui/templates/announcement.html @@ -5,7 +5,7 @@ {% endblock %} {% block content %} -{% call macros.box() %} +{% call macros.card() %} {{ form.hidden_tag() }} {{ macros.form_field(form.announcement_subject) }} diff --git a/core/admin/mailu/ui/templates/client.html b/core/admin/mailu/ui/templates/client.html index 81bee135..ee1b1c28 100644 --- a/core/admin/mailu/ui/templates/client.html +++ b/core/admin/mailu/ui/templates/client.html @@ -9,7 +9,7 @@ configure your email client {% endblock %} {% block content %} -{% call macros.box(title="Incoming mail") %} +{% call macros.card(title="Incoming mail") %} @@ -36,7 +36,7 @@ configure your email client
    {% endcall %} -{% call macros.box(title="Outgoing mail") %} +{% call macros.card(title="Outgoing mail") %} diff --git a/core/admin/mailu/ui/templates/confirm.html b/core/admin/mailu/ui/templates/confirm.html index 3b8a451f..d0f6acf3 100644 --- a/core/admin/mailu/ui/templates/confirm.html +++ b/core/admin/mailu/ui/templates/confirm.html @@ -9,7 +9,7 @@ {% endblock %} {% block content %} -{% call macros.box(theme="warning") %} +{% call macros.card(theme="warning") %}

    {% trans action %}You are about to {{ action }}. Please confirm your action.{% endtrans %}

    {{ macros.form(form) }} {% endcall %} diff --git a/core/admin/mailu/ui/templates/domain/create.html b/core/admin/mailu/ui/templates/domain/create.html index d67e3e89..d84159b1 100644 --- a/core/admin/mailu/ui/templates/domain/create.html +++ b/core/admin/mailu/ui/templates/domain/create.html @@ -5,7 +5,7 @@ {% endblock %} {% block content %} -{% call macros.box() %} +{% call macros.card() %} {{ form.hidden_tag() }} {{ macros.form_field(form.name) }} diff --git a/core/admin/mailu/ui/templates/domain/signup.html b/core/admin/mailu/ui/templates/domain/signup.html index c8a52f6c..77b7fec9 100644 --- a/core/admin/mailu/ui/templates/domain/signup.html +++ b/core/admin/mailu/ui/templates/domain/signup.html @@ -9,7 +9,7 @@ {{ form.hidden_tag() }} - {% call macros.box(title="Requirements") %} + {% call macros.card(title="Requirements") %}

    {% trans %}In order to register a new domain, you must first setup the domain zone so that the domain MX points to this server{% endtrans %} ({{ config["HOSTNAMES"].split(",")[0] }}). @@ -22,7 +22,7 @@

    {% endcall %} - {% call macros.box() %} + {% call macros.card() %} {% if form.localpart %} {{ macros.form_fields((form.localpart, form.name), append='@') }} {{ macros.form_fields((form.pw, form.pw2)) }} diff --git a/core/admin/mailu/ui/templates/fetch/create.html b/core/admin/mailu/ui/templates/fetch/create.html index bc69fc44..ed190961 100644 --- a/core/admin/mailu/ui/templates/fetch/create.html +++ b/core/admin/mailu/ui/templates/fetch/create.html @@ -11,7 +11,7 @@ {% block content %} {{ form.hidden_tag() }} - {% call macros.box(title="Remote server") %} + {% call macros.card(title="Remote server") %} {{ macros.form_field(form.protocol) }} {{ macros.form_fields((form.host, form.port)) }} {{ macros.form_field(form.tls) }} @@ -22,7 +22,7 @@ {{ macros.form_field(form.password) }} {% endcall %} - {% call macros.box(title="Settings") %} + {% call macros.card(title="Settings") %} {{ macros.form_field(form.keep) }} {% endcall %} diff --git a/core/admin/mailu/ui/templates/form.html b/core/admin/mailu/ui/templates/form.html index 25d7e83c..0e65ce33 100644 --- a/core/admin/mailu/ui/templates/form.html +++ b/core/admin/mailu/ui/templates/form.html @@ -1,7 +1,7 @@ {% extends "base.html" %} {% block content %} -{% call macros.box() %} +{% call macros.card() %} {{ macros.form(form) }} {% endcall %} {% endblock %} diff --git a/core/admin/mailu/ui/templates/manager/create.html b/core/admin/mailu/ui/templates/manager/create.html index e3911642..bc5e6ca9 100644 --- a/core/admin/mailu/ui/templates/manager/create.html +++ b/core/admin/mailu/ui/templates/manager/create.html @@ -9,7 +9,7 @@ {% endblock %} {% block content %} -{% call macros.box() %} +{% call macros.card() %} {{ form.hidden_tag() }} {{ macros.form_field(form.manager, class_='mailselect') }} diff --git a/core/admin/mailu/ui/templates/user/create.html b/core/admin/mailu/ui/templates/user/create.html index ed7b9884..a28c7048 100644 --- a/core/admin/mailu/ui/templates/user/create.html +++ b/core/admin/mailu/ui/templates/user/create.html @@ -12,7 +12,7 @@ {{ form.hidden_tag() }} - {% call macros.box(_("General")) %} + {% call macros.card(_("General")) %} {{ macros.form_field(form.localpart, append='@'+domain.name+'') }} {{ macros.form_fields((form.pw, form.pw2)) }} {{ macros.form_field(form.displayed_name) }} @@ -20,7 +20,7 @@ {{ macros.form_field(form.enabled) }} {% endcall %} - {% call macros.box(_("Features and quotas"), theme="success") %} + {% call macros.card(_("Features and quotas"), theme="success") %} {{ macros.form_field(form.quota_bytes, step=1000000000, max=(max_quota_bytes or domain.max_quota_bytes or 50000000000), prepend=''+((form.quota_bytes.data//1000000000).__str__() if form.quota_bytes.data else '∞')+' GiB', oninput='$("#quota").text(this.value == 0 ? "∞" : this.value/1000000000);') }} diff --git a/core/admin/mailu/ui/templates/user/forward.html b/core/admin/mailu/ui/templates/user/forward.html index 769d439d..6059f0ed 100644 --- a/core/admin/mailu/ui/templates/user/forward.html +++ b/core/admin/mailu/ui/templates/user/forward.html @@ -9,7 +9,7 @@ {% endblock %} {% block content %} -{% call macros.box() %} +{% call macros.card() %} {{ form.hidden_tag() }} {{ macros.form_field(form.forward_enabled, diff --git a/core/admin/mailu/ui/templates/user/reply.html b/core/admin/mailu/ui/templates/user/reply.html index 7225a178..7a6b7fa8 100644 --- a/core/admin/mailu/ui/templates/user/reply.html +++ b/core/admin/mailu/ui/templates/user/reply.html @@ -9,7 +9,7 @@ {% endblock %} {% block content %} -{% call macros.box() %} +{% call macros.card() %} {{ form.hidden_tag() }} {{ macros.form_field(form.reply_enabled, diff --git a/core/admin/mailu/ui/templates/user/signup.html b/core/admin/mailu/ui/templates/user/signup.html index f540d660..2b20bde9 100644 --- a/core/admin/mailu/ui/templates/user/signup.html +++ b/core/admin/mailu/ui/templates/user/signup.html @@ -11,7 +11,7 @@ {% block content %} {{ form.hidden_tag() }} - {% call macros.box() %} + {% call macros.card() %} {{ macros.form_field(form.localpart, append='@'+domain.name+'') }} {{ macros.form_fields((form.pw, form.pw2)) }} {% if form.captcha %} From cdfa94c2430ca43a43ba538ae7e538e8d71c33da Mon Sep 17 00:00:00 2001 From: DjVinnii Date: Thu, 1 Apr 2021 14:59:12 +0200 Subject: [PATCH 157/596] Make main action float right --- core/admin/mailu/ui/templates/admin/list.html | 2 +- core/admin/mailu/ui/templates/alias/list.html | 2 +- core/admin/mailu/ui/templates/alternative/list.html | 2 +- core/admin/mailu/ui/templates/domain/details.html | 2 +- core/admin/mailu/ui/templates/domain/list.html | 2 +- core/admin/mailu/ui/templates/fetch/list.html | 2 +- core/admin/mailu/ui/templates/manager/list.html | 2 +- core/admin/mailu/ui/templates/relay/list.html | 2 +- core/admin/mailu/ui/templates/token/list.html | 2 +- core/admin/mailu/ui/templates/user/list.html | 2 +- 10 files changed, 10 insertions(+), 10 deletions(-) diff --git a/core/admin/mailu/ui/templates/admin/list.html b/core/admin/mailu/ui/templates/admin/list.html index 72b5a1fa..856f6830 100644 --- a/core/admin/mailu/ui/templates/admin/list.html +++ b/core/admin/mailu/ui/templates/admin/list.html @@ -5,7 +5,7 @@ {% endblock %} {% block main_action %} - + {% trans %}Add administrator{% endtrans %} {% endblock %} diff --git a/core/admin/mailu/ui/templates/alias/list.html b/core/admin/mailu/ui/templates/alias/list.html index 29766b25..4bb69169 100644 --- a/core/admin/mailu/ui/templates/alias/list.html +++ b/core/admin/mailu/ui/templates/alias/list.html @@ -9,7 +9,7 @@ {% endblock %} {% block main_action %} -{% trans %}Add alias{% endtrans %} +{% trans %}Add alias{% endtrans %} {% endblock %} {% block content %} diff --git a/core/admin/mailu/ui/templates/alternative/list.html b/core/admin/mailu/ui/templates/alternative/list.html index 56e7565b..dc981ef6 100644 --- a/core/admin/mailu/ui/templates/alternative/list.html +++ b/core/admin/mailu/ui/templates/alternative/list.html @@ -9,7 +9,7 @@ {% endblock %} {% block main_action %} -{% trans %}Add alternative{% endtrans %} +{% trans %}Add alternative{% endtrans %} {% endblock %} {% block content %} diff --git a/core/admin/mailu/ui/templates/domain/details.html b/core/admin/mailu/ui/templates/domain/details.html index 65c6ec1a..8aa35f22 100644 --- a/core/admin/mailu/ui/templates/domain/details.html +++ b/core/admin/mailu/ui/templates/domain/details.html @@ -10,7 +10,7 @@ {% block main_action %} {% if current_user.global_admin %} - + {% if domain.dkim_publickey %} {% trans %}Regenerate keys{% endtrans %} {% else %} diff --git a/core/admin/mailu/ui/templates/domain/list.html b/core/admin/mailu/ui/templates/domain/list.html index 2431faa5..fcc3a72c 100644 --- a/core/admin/mailu/ui/templates/domain/list.html +++ b/core/admin/mailu/ui/templates/domain/list.html @@ -6,7 +6,7 @@ {% block main_action %} {% if current_user.global_admin %} -{% trans %}New domain{% endtrans %} +{% trans %}New domain{% endtrans %} {% endif %} {% endblock %} diff --git a/core/admin/mailu/ui/templates/fetch/list.html b/core/admin/mailu/ui/templates/fetch/list.html index 07ab5ad8..09253a21 100644 --- a/core/admin/mailu/ui/templates/fetch/list.html +++ b/core/admin/mailu/ui/templates/fetch/list.html @@ -9,7 +9,7 @@ {% endblock %} {% block main_action %} -{% trans %}Add an account{% endtrans %} +{% trans %}Add an account{% endtrans %} {% endblock %} {% block content %} diff --git a/core/admin/mailu/ui/templates/manager/list.html b/core/admin/mailu/ui/templates/manager/list.html index d818a876..b660bf6d 100644 --- a/core/admin/mailu/ui/templates/manager/list.html +++ b/core/admin/mailu/ui/templates/manager/list.html @@ -9,7 +9,7 @@ {% endblock %} {% block main_action %} -{% trans %}Add manager{% endtrans %} +{% trans %}Add manager{% endtrans %} {% endblock %} {% block content %} diff --git a/core/admin/mailu/ui/templates/relay/list.html b/core/admin/mailu/ui/templates/relay/list.html index bb00b1d9..aef189ad 100644 --- a/core/admin/mailu/ui/templates/relay/list.html +++ b/core/admin/mailu/ui/templates/relay/list.html @@ -6,7 +6,7 @@ {% block main_action %} {% if current_user.global_admin %} -{% trans %}New relayed domain{% endtrans %} +{% trans %}New relayed domain{% endtrans %} {% endif %} {% endblock %} diff --git a/core/admin/mailu/ui/templates/token/list.html b/core/admin/mailu/ui/templates/token/list.html index ad560e6a..99f67383 100644 --- a/core/admin/mailu/ui/templates/token/list.html +++ b/core/admin/mailu/ui/templates/token/list.html @@ -9,7 +9,7 @@ {% endblock %} {% block main_action %} -{% trans %}New token{% endtrans %} +{% trans %}New token{% endtrans %} {% endblock %} {% block content %} diff --git a/core/admin/mailu/ui/templates/user/list.html b/core/admin/mailu/ui/templates/user/list.html index 2aff662f..ab896ee6 100644 --- a/core/admin/mailu/ui/templates/user/list.html +++ b/core/admin/mailu/ui/templates/user/list.html @@ -9,7 +9,7 @@ {% endblock %} {% block main_action %} -{% trans %}Add user{% endtrans %} +{% trans %}Add user{% endtrans %} {% endblock %} {% block content %} From 7d3c9d412d430d7fbd578df1cb738126f7e1c53e Mon Sep 17 00:00:00 2001 From: DjVinnii Date: Thu, 1 Apr 2021 16:05:30 +0200 Subject: [PATCH 158/596] Change tables to datatables --- core/admin/assets/app.js | 11 ++- core/admin/assets/vendor.js | 2 + core/admin/mailu/ui/templates/admin/list.html | 28 ++++--- core/admin/mailu/ui/templates/alias/list.html | 46 ++++++------ .../mailu/ui/templates/alternative/list.html | 32 ++++---- .../admin/mailu/ui/templates/domain/list.html | 74 ++++++++++--------- core/admin/mailu/ui/templates/fetch/list.html | 54 +++++++------- core/admin/mailu/ui/templates/macros.html | 4 +- .../mailu/ui/templates/manager/list.html | 12 ++- core/admin/mailu/ui/templates/relay/list.html | 46 ++++++------ core/admin/mailu/ui/templates/token/list.html | 36 +++++---- core/admin/mailu/ui/templates/user/list.html | 68 +++++++++-------- 12 files changed, 228 insertions(+), 185 deletions(-) diff --git a/core/admin/assets/app.js b/core/admin/assets/app.js index dc981081..d87832a6 100644 --- a/core/admin/assets/app.js +++ b/core/admin/assets/app.js @@ -1,10 +1,17 @@ require('./app.css'); import 'select2'; +import 'admin-lte/plugins/datatables/jquery.dataTables.js'; +import 'admin-lte/plugins/datatables-bs4/js/dataTables.bootstrap4.js'; +import 'admin-lte/plugins/datatables-responsive/js/dataTables.responsive.js'; +import 'admin-lte/plugins/datatables-responsive/js/responsive.bootstrap4.js'; + jQuery("document").ready(function() { jQuery(".mailselect").select2({ tags: true, tokenSeparators: [',', ' '] - }) + }); + jQuery(".table").DataTable({ + "responsive": true, + }); }); - diff --git a/core/admin/assets/vendor.js b/core/admin/assets/vendor.js index a5ff7a3c..65bee27f 100644 --- a/core/admin/assets/vendor.js +++ b/core/admin/assets/vendor.js @@ -14,6 +14,8 @@ import 'admin-lte/plugins/fontawesome-free/css/solid.css'; // AdminLTE import 'admin-lte/build/scss/adminlte.scss'; +import 'admin-lte/plugins/datatables-bs4/css/dataTables.bootstrap4.css'; +import 'admin-lte/plugins/datatables-responsive/css/responsive.bootstrap4.css'; // import 'admin-lte/build/less/AdminLTE-without-plugins.less'; // doesn't exist // import 'admin-lte/build/less/select2.less'; // doesn't exist // import 'admin-lte/build/less/skins/skin-blue.less'; // doesn't exist diff --git a/core/admin/mailu/ui/templates/admin/list.html b/core/admin/mailu/ui/templates/admin/list.html index 856f6830..f2f5d229 100644 --- a/core/admin/mailu/ui/templates/admin/list.html +++ b/core/admin/mailu/ui/templates/admin/list.html @@ -12,17 +12,21 @@ {% block content %} {% call macros.table() %} - - - - -{% for admin in admins %} - - - - -{% endfor %} + + + + + + + + {% for admin in admins %} + + + + + {% endfor %} + {% endcall %} {% endblock %} diff --git a/core/admin/mailu/ui/templates/alias/list.html b/core/admin/mailu/ui/templates/alias/list.html index 4bb69169..e8ddc862 100644 --- a/core/admin/mailu/ui/templates/alias/list.html +++ b/core/admin/mailu/ui/templates/alias/list.html @@ -14,26 +14,30 @@ {% block content %} {% call macros.table() %} - - - - - - - - -{% for alias in domain.aliases %} - - - - - - - - -{% endfor %} + + + + + + + + + + + + {% for alias in domain.aliases %} + + + + + + + + + {% endfor %} + {% endcall %} {% endblock %} diff --git a/core/admin/mailu/ui/templates/alternative/list.html b/core/admin/mailu/ui/templates/alternative/list.html index dc981ef6..f123eb9f 100644 --- a/core/admin/mailu/ui/templates/alternative/list.html +++ b/core/admin/mailu/ui/templates/alternative/list.html @@ -14,19 +14,23 @@ {% block content %} {% call macros.table() %} - - - - - -{% for alternative in domain.alternatives %} - - - - - -{% endfor %} + + + + + + + + + {% for alternative in domain.alternatives %} + + + + + + {% endfor %} + {% endcall %} {% endblock %} diff --git a/core/admin/mailu/ui/templates/domain/list.html b/core/admin/mailu/ui/templates/domain/list.html index fcc3a72c..c82647df 100644 --- a/core/admin/mailu/ui/templates/domain/list.html +++ b/core/admin/mailu/ui/templates/domain/list.html @@ -12,40 +12,44 @@ {% block content %} {% call macros.table() %} - - - - - - - - - - -{% for domain in current_user.get_managed_domains() %} - - - - - - - - - - -{% endfor %} + + + + + + + + + + + + + + {% for domain in current_user.get_managed_domains() %} + + + + + + + + + + + {% endfor %} + {% endcall %} {% endblock %} diff --git a/core/admin/mailu/ui/templates/fetch/list.html b/core/admin/mailu/ui/templates/fetch/list.html index 09253a21..77f66bb1 100644 --- a/core/admin/mailu/ui/templates/fetch/list.html +++ b/core/admin/mailu/ui/templates/fetch/list.html @@ -14,30 +14,34 @@ {% block content %} {% call macros.table() %} - - - - - - - - - - -{% for fetch in user.fetches %} - - - - - - - - - - -{% endfor %} + + + + + + + + + + + + + + {% for fetch in user.fetches %} + + + + + + + + + + + {% endfor %} + {% endcall %} {% endblock %} diff --git a/core/admin/mailu/ui/templates/macros.html b/core/admin/mailu/ui/templates/macros.html index a1fc2919..6d821ef8 100644 --- a/core/admin/mailu/ui/templates/macros.html +++ b/core/admin/mailu/ui/templates/macros.html @@ -86,11 +86,9 @@ {% macro table(theme="primary") %}
    -
    +
    {% trans %}Actions{% endtrans %}{% trans %}Email{% endtrans %}
    - - {{ admin }}
    {% trans %}Actions{% endtrans %}{% trans %}Email{% endtrans %}
    + + {{ admin }}
    {% trans %}Actions{% endtrans %}{% trans %}Email{% endtrans %}{% trans %}Destination{% endtrans %}{% trans %}Comment{% endtrans %}{% trans %}Created{% endtrans %}{% trans %}Last edit{% endtrans %}
    -   - - {{ alias }}{{ alias.destination|join(', ') or '-' }}{{ alias.comment or '' }}{{ alias.created_at }}{{ alias.updated_at or '' }}
    {% trans %}Actions{% endtrans %}{% trans %}Email{% endtrans %}{% trans %}Destination{% endtrans %}{% trans %}Comment{% endtrans %}{% trans %}Created{% endtrans %}{% trans %}Last edit{% endtrans %}
    +   + + {{ alias }}{{ alias.destination|join(', ') or '-' }}{{ alias.comment or '' }}{{ alias.created_at }}{{ alias.updated_at or '' }}
    {% trans %}Actions{% endtrans %}{% trans %}Name{% endtrans %}{% trans %}Created{% endtrans %}
    - - {{ alternative }}{{ alternative.created_at }}
    {% trans %}Actions{% endtrans %}{% trans %}Name{% endtrans %}{% trans %}Created{% endtrans %}
    + + {{ alternative }}{{ alternative.created_at }}
    {% trans %}Actions{% endtrans %}{% trans %}Manage{% endtrans %}{% trans %}Domain name{% endtrans %}{% trans %}Mailbox count{% endtrans %}{% trans %}Alias count{% endtrans %}{% trans %}Comment{% endtrans %}{% trans %}Created{% endtrans %}{% trans %}Last edit{% endtrans %}
    -   - {% if current_user.global_admin %} -   -   - {% endif %} - -   -   -   - {% if current_user.global_admin %} -   - {% endif %} - {{ domain.name }}{{ domain.users | count }} / {{ '∞' if domain.max_users == -1 else domain.max_users }}{{ domain.aliases | count }} / {{ '∞' if domain.max_aliases == -1 else domain.max_aliases }}{{ domain.comment or '' }}{{ domain.created_at }}{{ domain.updated_at or '' }}
    {% trans %}Actions{% endtrans %}{% trans %}Manage{% endtrans %}{% trans %}Domain name{% endtrans %}{% trans %}Mailbox count{% endtrans %}{% trans %}Alias count{% endtrans %}{% trans %}Comment{% endtrans %}{% trans %}Created{% endtrans %}{% trans %}Last edit{% endtrans %}
    +   + {% if current_user.global_admin %} +   +   + {% endif %} + +   +   +   + {% if current_user.global_admin %} +   + {% endif %} + {{ domain.name }}{{ domain.users | count }} / {{ '∞' if domain.max_users == -1 else domain.max_users }}{{ domain.aliases | count }} / {{ '∞' if domain.max_aliases == -1 else domain.max_aliases }}{{ domain.comment or '' }}{{ domain.created_at }}{{ domain.updated_at or '' }}
    {% trans %}Actions{% endtrans %}{% trans %}Endpoint{% endtrans %}{% trans %}Username{% endtrans %}{% trans %}Keep emails{% endtrans %}{% trans %}Last check{% endtrans %}{% trans %}Status{% endtrans %}{% trans %}Created{% endtrans %}{% trans %}Last edit{% endtrans %}
    -   - - {{ fetch.protocol }}{{ 's' if fetch.tls else '' }}://{{ fetch.host }}:{{ fetch.port }}{{ fetch.username }}{% if fetch.keep %}{% trans %}yes{% endtrans %}{% else %}{% trans %}no{% endtrans %}{% endif %}{{ fetch.last_check or '-' }}{{ fetch.error or '-' }}{{ fetch.created_at }}{{ fetch.updated_at or '' }}
    {% trans %}Actions{% endtrans %}{% trans %}Endpoint{% endtrans %}{% trans %}Username{% endtrans %}{% trans %}Keep emails{% endtrans %}{% trans %}Last check{% endtrans %}{% trans %}Status{% endtrans %}{% trans %}Created{% endtrans %}{% trans %}Last edit{% endtrans %}
    +   + + {{ fetch.protocol }}{{ 's' if fetch.tls else '' }}://{{ fetch.host }}:{{ fetch.port }}{{ fetch.username }}{% if fetch.keep %}{% trans %}yes{% endtrans %}{% else %}{% trans %}no{% endtrans %}{% endif %}{{ fetch.last_check or '-' }}{{ fetch.error or '-' }}{{ fetch.created_at }}{{ fetch.updated_at or '' }}
    - {{ caller() }} -
    diff --git a/core/admin/mailu/ui/templates/manager/list.html b/core/admin/mailu/ui/templates/manager/list.html index b660bf6d..9a78e3ca 100644 --- a/core/admin/mailu/ui/templates/manager/list.html +++ b/core/admin/mailu/ui/templates/manager/list.html @@ -14,10 +14,13 @@ {% block content %} {% call macros.table() %} - - {% trans %}Actions{% endtrans %} - {% trans %}Email{% endtrans %} - + + + {% trans %}Actions{% endtrans %} + {% trans %}Email{% endtrans %} + + + {% for manager in domain.managers %} @@ -26,5 +29,6 @@ {{ manager }} {% endfor %} + {% endcall %} {% endblock %} diff --git a/core/admin/mailu/ui/templates/relay/list.html b/core/admin/mailu/ui/templates/relay/list.html index aef189ad..6c8c9196 100644 --- a/core/admin/mailu/ui/templates/relay/list.html +++ b/core/admin/mailu/ui/templates/relay/list.html @@ -12,26 +12,30 @@ {% block content %} {% call macros.table() %} - - {% trans %}Actions{% endtrans %} - {% trans %}Domain name{% endtrans %} - {% trans %}Remote host{% endtrans %} - {% trans %}Comment{% endtrans %} - {% trans %}Created{% endtrans %} - {% trans %}Last edit{% endtrans %} - -{% for relay in relays %} - - -  -   - - {{ relay.name }} - {{ relay.smtp or '-' }} - {{ relay.comment or '' }} - {{ relay.created_at }} - {{ relay.updated_at or '' }} - -{% endfor %} + + + {% trans %}Actions{% endtrans %} + {% trans %}Domain name{% endtrans %} + {% trans %}Remote host{% endtrans %} + {% trans %}Comment{% endtrans %} + {% trans %}Created{% endtrans %} + {% trans %}Last edit{% endtrans %} + + + + {% for relay in relays %} + + +   +   + + {{ relay.name }} + {{ relay.smtp or '-' }} + {{ relay.comment or '' }} + {{ relay.created_at }} + {{ relay.updated_at or '' }} + + {% endfor %} + {% endcall %} {% endblock %} diff --git a/core/admin/mailu/ui/templates/token/list.html b/core/admin/mailu/ui/templates/token/list.html index 99f67383..09d0fe76 100644 --- a/core/admin/mailu/ui/templates/token/list.html +++ b/core/admin/mailu/ui/templates/token/list.html @@ -14,21 +14,25 @@ {% block content %} {% call macros.table() %} - - {% trans %}Actions{% endtrans %} - {% trans %}Comment{% endtrans %} - {% trans %}Authorized IP{% endtrans %} - {% trans %}Created{% endtrans %} - -{% for token in user.tokens %} - - - - - {{ token.comment }} - {{ token.ip or "any" }} - {{ token.created_at }} - -{% endfor %} + + + {% trans %}Actions{% endtrans %} + {% trans %}Comment{% endtrans %} + {% trans %}Authorized IP{% endtrans %} + {% trans %}Created{% endtrans %} + + + + {% for token in user.tokens %} + + + + + {{ token.comment }} + {{ token.ip or "any" }} + {{ token.created_at }} + + {% endfor %} + {% endcall %} {% endblock %} diff --git a/core/admin/mailu/ui/templates/user/list.html b/core/admin/mailu/ui/templates/user/list.html index ab896ee6..3cc724c8 100644 --- a/core/admin/mailu/ui/templates/user/list.html +++ b/core/admin/mailu/ui/templates/user/list.html @@ -14,37 +14,41 @@ {% block content %} {% call macros.table() %} - - {% trans %}Actions{% endtrans %} - {% trans %}User settings{% endtrans %} - {% trans %}Email{% endtrans %} - {% trans %}Features{% endtrans %} - {% trans %}Quota{% endtrans %} - {% trans %}Comment{% endtrans %} - {% trans %}Created{% endtrans %} - {% trans %}Last edit{% endtrans %} - -{% for user in domain.users %} - - -   - - - -   -   -   - - {{ user }} - - {% if user.enable_imap %}imap{% endif %} - {% if user.enable_pop %}pop3{% endif %} - - {{ user.quota_bytes_used | filesizeformat }} / {{ (user.quota_bytes | filesizeformat) if user.quota_bytes else '∞' }} - {{ user.comment or '-' }} - {{ user.created_at }} - {{ user.updated_at or '' }} - -{% endfor %} + + + {% trans %}Actions{% endtrans %} + {% trans %}User settings{% endtrans %} + {% trans %}Email{% endtrans %} + {% trans %}Features{% endtrans %} + {% trans %}Quota{% endtrans %} + {% trans %}Comment{% endtrans %} + {% trans %}Created{% endtrans %} + {% trans %}Last edit{% endtrans %} + + + + {% for user in domain.users %} + + +   + + + +   +   +   + + {{ user }} + + {% if user.enable_imap %}imap{% endif %} + {% if user.enable_pop %}pop3{% endif %} + + {{ user.quota_bytes_used | filesizeformat }} / {{ (user.quota_bytes | filesizeformat) if user.quota_bytes else '∞' }} + {{ user.comment or '-' }} + {{ user.created_at }} + {{ user.updated_at or '' }} + + {% endfor %} + {% endcall %} {% endblock %} From 49d68fa6d10aef4b585ed30a5ea7e7ac08e7c869 Mon Sep 17 00:00:00 2001 From: DjVinnii Date: Thu, 1 Apr 2021 16:51:13 +0200 Subject: [PATCH 159/596] Fix horizontal scrollbar in sidebar --- core/admin/mailu/ui/templates/base.html | 2 +- core/admin/mailu/ui/templates/sidebar.html | 36 +++++++++++----------- 2 files changed, 19 insertions(+), 19 deletions(-) diff --git a/core/admin/mailu/ui/templates/base.html b/core/admin/mailu/ui/templates/base.html index c5b09159..568d0880 100644 --- a/core/admin/mailu/ui/templates/base.html +++ b/core/admin/mailu/ui/templates/base.html @@ -8,7 +8,7 @@ Mailu-Admin - {{ config["SITENAME"] }} - +
    From 84e59c0a6e4d84b1a91c8e84293a7abd12259f6d Mon Sep 17 00:00:00 2001 From: parisni Date: Sat, 19 Jun 2021 01:22:23 +0200 Subject: [PATCH 192/596] Add missing roundcube_db_flavor --- setup/flavors/compose/mailu.env | 1 + 1 file changed, 1 insertion(+) diff --git a/setup/flavors/compose/mailu.env b/setup/flavors/compose/mailu.env index 150c70a3..4dce34f1 100644 --- a/setup/flavors/compose/mailu.env +++ b/setup/flavors/compose/mailu.env @@ -176,6 +176,7 @@ DB_NAME={{ db_name }} {% endif %} {% if (postgresql == 'external' or db_flavor == 'mysql') and webmail_type == 'roundcube' %} +ROUNDCUBE_DB_FLAVOR={{ db_flavor }} ROUNDCUBE_DB_USER={{ roundcube_db_user }} ROUNDCUBE_DB_PW={{ roundcube_db_pw }} ROUNDCUBE_DB_HOST={{ roundcube_db_url }} From 14307c83c13877234d3daec11999f26d5bb0efff Mon Sep 17 00:00:00 2001 From: parisni Date: Sat, 19 Jun 2021 09:12:46 +0200 Subject: [PATCH 193/596] Document databases variable and deprecation --- docs/configuration.rst | 20 ++++++++++++++++++++ docs/database.rst | 3 ++- 2 files changed, 22 insertions(+), 1 deletion(-) diff --git a/docs/configuration.rst b/docs/configuration.rst index e08675a8..16ea23c3 100644 --- a/docs/configuration.rst +++ b/docs/configuration.rst @@ -195,4 +195,24 @@ resolved. This can be used to rely on DNS based service discovery with changing When using ``*_ADDRESS``, the hostnames must be full-qualified hostnames. Otherwise nginx will not be able to resolve the hostnames. +Database settings +----------------- + +The admin service stores configurations in a database. + +- ``DB_FLAVOR``: the database type for mailu admin service. (``sqlite``, ``postgresql``, ``mysql``) +- ``DB_HOST``: the database host for mailu admin service. (when not ``sqlite``) +- ``DB_PORT``: the database port for mailu admin service. (when not ``sqlite``) +- ``DB_PW``: the database password for mailu admin service. (when not ``sqlite``) +- ``DB_USER``: the database user for mailu admin service. (when not ``sqlite``) +- ``DB_NAME``: the database name for mailu admin service. (when not ``sqlite``) + +The roundcube service stores configurations in a database. + +- ``ROUNDCUBE_DB_FLAVOR``: the database type for roundcube service. (``sqlite``, ``postgresql``, ``mysql``) +- ``ROUNDCUBE_DB_HOST``: the database host for roundcube service. (when not ``sqlite``) +- ``ROUNDCUBE_DB_PORT``: the database port for roundcube service. (when not ``sqlite``) +- ``ROUNDCUBE_DB_PW``: the database password for roundcube service. (when not ``sqlite``) +- ``ROUNDCUBE_DB_USER``: the database user for roundcube service. (when not ``sqlite``) +- ``ROUNDCUBE_DB_NAME``: the database name for roundcube service. (when not ``sqlite``) diff --git a/docs/database.rst b/docs/database.rst index b2526d6f..c13ca0bf 100644 --- a/docs/database.rst +++ b/docs/database.rst @@ -8,7 +8,8 @@ This functionality should still be considered experimental! Mailu Postgresql ---------------- -Mailu optionally comes with a pre-configured Postgresql image. +Mailu optionally comes with a pre-configured Postgresql image, wich as of 1.8 is deprecated +will be removed in 1.9 This images has the following features: - Automatic creation of users, db, extensions and password; From a9548e4cbd31a9cd1b04414fc3ad4ca668f05bf6 Mon Sep 17 00:00:00 2001 From: parisni Date: Sat, 19 Jun 2021 09:20:23 +0200 Subject: [PATCH 194/596] Remove mailu/roundcube shared host --- webmails/roundcube/start.py | 4 ++-- 1 file changed, 2 insertions(+), 2 deletions(-) diff --git a/webmails/roundcube/start.py b/webmails/roundcube/start.py index 3e47ce69..f87e460f 100755 --- a/webmails/roundcube/start.py +++ b/webmails/roundcube/start.py @@ -17,14 +17,14 @@ elif db_flavor == "mysql": os.environ["DB_DSNW"] = "mysql://%s:%s@%s/%s" % ( os.environ.get("ROUNDCUBE_DB_USER", "roundcube"), os.environ.get("ROUNDCUBE_DB_PW"), - os.environ.get("ROUNDCUBE_DB_HOST", os.environ.get("DB_HOST", "database")), + os.environ.get("ROUNDCUBE_DB_HOST", "database"), os.environ.get("ROUNDCUBE_DB_NAME", "roundcube") ) elif db_flavor == "postgresql": os.environ["DB_DSNW"] = "pgsql://%s:%s@%s/%s" % ( os.environ.get("ROUNDCUBE_DB_USER", "roundcube"), os.environ.get("ROUNDCUBE_DB_PW"), - os.environ.get("ROUNDCUBE_DB_HOST", os.environ.get("DB_HOST", "database")), + os.environ.get("ROUNDCUBE_DB_HOST", "database"), os.environ.get("ROUNDCUBE_DB_NAME", "roundcube") ) else: From f4c76d49c1862894eba341eca318629a2137f274 Mon Sep 17 00:00:00 2001 From: parisni Date: Sat, 19 Jun 2021 09:30:32 +0200 Subject: [PATCH 195/596] Add changelog entry --- towncrier/newsfragments/1831.bugfix | 1 + 1 file changed, 1 insertion(+) create mode 100644 towncrier/newsfragments/1831.bugfix diff --git a/towncrier/newsfragments/1831.bugfix b/towncrier/newsfragments/1831.bugfix new file mode 100644 index 00000000..1094be34 --- /dev/null +++ b/towncrier/newsfragments/1831.bugfix @@ -0,0 +1 @@ +Fix roundcube environment configuration for databases \ No newline at end of file From 58235bcc44e9a7449229ea0d851e35846a5ce176 Mon Sep 17 00:00:00 2001 From: Dimitri Huisman Date: Sat, 26 Jun 2021 08:25:15 +0000 Subject: [PATCH 196/596] Switch to github actions for CI/CD --- .github/worfklows/CI.yml | 276 +++++++++++++++++++++++ .travis.yml | 56 ----- bors.toml | 3 +- tests/compose/filters/00_create_users.sh | 5 + 4 files changed, 283 insertions(+), 57 deletions(-) create mode 100644 .github/worfklows/CI.yml delete mode 100644 .travis.yml create mode 100755 tests/compose/filters/00_create_users.sh diff --git a/.github/worfklows/CI.yml b/.github/worfklows/CI.yml new file mode 100644 index 00000000..3dcaf096 --- /dev/null +++ b/.github/worfklows/CI.yml @@ -0,0 +1,276 @@ + + +name: CI +on: + push: + branches: + - staging + - testing + - '1.5' + - '1.6' + - '1.7' + - '1.8' + - master + # version tags, e.g. 1.7.1 + - '[1-9].[0-9].[0-9]' + # pre-releases, e.g. 1.8-pre1 + - 1.8-pre[0-9] + # test branches, e.g. test-debian + - test-* + +############################################### +# REQUIRED secrets +# DOCKER_UN: ${{ secrets.Docker_Login }} +# Username of docker login for pushing the images to repo $DOCKER_ORG +# DOCKER_PW: ${{ secrets.Docker_Password }} +# Password of docker login for pushing the images to repo $DOCKER_ORG +# DOCKER_ORG: ${{ secrets.DOCKER_ORG }} +# The docker repository where the images are pushed to. +# +# Add the above secrets to your github repo to determine where the images will be pushed. +################################################ + +jobs: + build: + name: build + runs-on: ubuntu-latest + + steps: + - uses: actions/checkout@v2 + - name: Extract branch name + shell: bash + run: | + echo "BRANCH=${GITHUB_REF#refs/heads/}" >> $GITHUB_ENV + - name: install python packages + run: python3 -m pip install -r tests/requirements.txt + - name: check docker-compose version + run: docker-compose -v + - name: login docker + env: + DOCKER_UN: ${{ secrets.Docker_Login }} + DOCKER_PW: ${{ secrets.Docker_Password }} + DOCKER_ORG: ${{ secrets.DOCKER_ORG }} + run: echo "$DOCKER_PW" | docker login --username $DOCKER_UN --password-stdin + # In this step, this action saves a list of existing images, + # the cache is created without them in the post run. + # It also restores the cache if it exists. + - uses: satackey/action-docker-layer-caching@v0.0.11 + # Ignore the failure of a step and avoid terminating the job. + continue-on-error: true + - name: build all docker images + env: + MAILU_VERSION: ${{ env.BRANCH }} + TRAVIS_BRANCH: ${{ env.BRANCH }} + DOCKER_ORG: ${{ secrets.DOCKER_ORG }} + run: docker-compose -f tests/build.yml build + +#NOTE: It appears the filter test depends on the core test. The filter test requires an email user +#that is created by the core test. + core-test: + name: core test + runs-on: ubuntu-latest + needs: + - build + steps: + - uses: actions/checkout@v2 + - name: Extract branch name + shell: bash + run: | + echo "BRANCH=${GITHUB_REF#refs/heads/}" >> $GITHUB_ENV + - name: install python packages + run: python3 -m pip install -r tests/requirements.txt + - uses: satackey/action-docker-layer-caching@v0.0.11 + # Ignore the failure of a step and avoid terminating the job. + continue-on-error: true + - name: copy all certs + run: sudo -- sh -c 'mkdir -p /mailu && cp -r tests/certs /mailu && chmod 600 /mailu/certs/*' + - name: test core suite + run: python tests/compose/test.py core 1 + env: + MAILU_VERSION: ${{ env.BRANCH }} + TRAVIS_BRANCH: ${{ env.BRANCH }} + DOCKER_ORG: ${{ secrets.DOCKER_ORG }} + + filter-test: + name: filter test + runs-on: ubuntu-latest + needs: + - build + steps: + - uses: actions/checkout@v2 + - name: Extract branch name + shell: bash + run: | + echo "BRANCH=${GITHUB_REF#refs/heads/}" >> $GITHUB_ENV + - name: install python packages + run: python3 -m pip install -r tests/requirements.txt + - uses: satackey/action-docker-layer-caching@v0.0.11 + # Ignore the failure of a step and avoid terminating the job. + continue-on-error: true + - name: copy all certs + run: sudo -- sh -c 'mkdir -p /mailu && cp -r tests/certs /mailu && chmod 600 /mailu/certs/*' + - name: 'test clamvav' + run: python tests/compose/test.py filters 2 + env: + MAILU_VERSION: ${{ env.BRANCH }} + TRAVIS_BRANCH: ${{ env.BRANCH }} + DOCKER_ORG: ${{ secrets.DOCKER_ORG }} + + fetch-test: + name: fetch test + runs-on: ubuntu-latest + needs: + - build + steps: + - uses: actions/checkout@v2 + - name: Extract branch name + shell: bash + run: | + echo "BRANCH=${GITHUB_REF#refs/heads/}" >> $GITHUB_ENV + - name: install python packages + run: python3 -m pip install -r tests/requirements.txt + - uses: satackey/action-docker-layer-caching@v0.0.11 + # Ignore the failure of a step and avoid terminating the job. + continue-on-error: true + - name: copy all certs + run: sudo -- sh -c 'mkdir -p /mailu && cp -r tests/certs /mailu && chmod 600 /mailu/certs/*' + - name: test fetch + run: python tests/compose/test.py fetchmail 1 + env: + MAILU_VERSION: ${{ env.BRANCH }} + TRAVIS_BRANCH: ${{ env.BRANCH }} + DOCKER_ORG: ${{ secrets.DOCKER_ORG }} + + rainloop-test: + name: rainloop test + runs-on: ubuntu-latest + needs: + - build + steps: + - uses: actions/checkout@v2 + - name: Extract branch name + shell: bash + run: | + echo "BRANCH=${GITHUB_REF#refs/heads/}" >> $GITHUB_ENV + - name: install python packages + run: python3 -m pip install -r tests/requirements.txt + - uses: satackey/action-docker-layer-caching@v0.0.11 + # Ignore the failure of a step and avoid terminating the job. + continue-on-error: true + - name: copy all certs + run: sudo -- sh -c 'mkdir -p /mailu && cp -r tests/certs /mailu && chmod 600 /mailu/certs/*' + - name: test rainloop + run: python tests/compose/test.py rainloop 1 + env: + MAILU_VERSION: ${{ env.BRANCH }} + TRAVIS_BRANCH: ${{ env.BRANCH }} + DOCKER_ORG: ${{ secrets.DOCKER_ORG }} + + roundcube-test: + name: roundcube test + runs-on: ubuntu-latest + needs: + - build + steps: + - uses: actions/checkout@v2 + - name: Extract branch name + shell: bash + run: | + echo "BRANCH=${GITHUB_REF#refs/heads/}" >> $GITHUB_ENV + - name: install python packages + run: python3 -m pip install -r tests/requirements.txt + - uses: satackey/action-docker-layer-caching@v0.0.11 + # Ignore the failure of a step and avoid terminating the job. + continue-on-error: true + - name: copy all certs + run: sudo -- sh -c 'mkdir -p /mailu && cp -r tests/certs /mailu && chmod 600 /mailu/certs/*' + - name: test roundcube + run: python tests/compose/test.py roundcube 1 + env: + MAILU_VERSION: ${{ env.BRANCH }} + TRAVIS_BRANCH: ${{ env.BRANCH }} + DOCKER_ORG: ${{ secrets.DOCKER_ORG }} + + webdav-test: + name: webdav test + runs-on: ubuntu-latest + needs: + - build + steps: + - uses: actions/checkout@v2 + - name: Extract branch name + shell: bash + run: | + echo "BRANCH=${GITHUB_REF#refs/heads/}" >> $GITHUB_ENV + - name: install python packages + run: python3 -m pip install -r tests/requirements.txt + - uses: satackey/action-docker-layer-caching@v0.0.11 + # Ignore the failure of a step and avoid terminating the job. + continue-on-error: true + - name: copy all certs + run: sudo -- sh -c 'mkdir -p /mailu && cp -r tests/certs /mailu && chmod 600 /mailu/certs/*' + - name: test webdav + run: python tests/compose/test.py webdav 1 + env: + MAILU_VERSION: ${{ env.BRANCH }} + TRAVIS_BRANCH: ${{ env.BRANCH }} + DOCKER_ORG: ${{ secrets.DOCKER_ORG }} + + deploy: + name: deploy step + runs-on: ubuntu-latest + needs: + - build + - core-test + - filter-test + - fetch-test + - rainloop-test + - roundcube-test + - webdav-test + steps: + - uses: actions/checkout@v2 + - name: Extract branch name + shell: bash + run: | + echo "BRANCH=${GITHUB_REF#refs/heads/}" >> $GITHUB_ENV + - name: install python packages + run: python3 -m pip install -r tests/requirements.txt + - uses: satackey/action-docker-layer-caching@v0.0.11 + # Ignore the failure of a step and avoid terminating the job. + continue-on-error: true + - name: copy all certs + run: sudo -- sh -c 'mkdir -p /mailu && cp -r tests/certs /mailu && chmod 600 /mailu/certs/*' + - name: login docker + env: + DOCKER_UN: ${{ secrets.Docker_Login }} + DOCKER_PW: ${{ secrets.Docker_Password }} + run: echo "$DOCKER_PW" | docker login --username $DOCKER_UN --password-stdin + - name: build all docker images + run: docker-compose -f tests/build.yml build + env: + MAILU_VERSION: ${{ env.BRANCH }} + TRAVIS_BRANCH: ${{ env.BRANCH }} + DOCKER_ORG: ${{ secrets.DOCKER_ORG }} + - name: deploy built docker images + env: + DOCKER_UN: ${{ secrets.Docker_Login }} + DOCKER_PW: ${{ secrets.Docker_Password }} + DOCKER_ORG: ${{ secrets.DOCKER_ORG }} + MAILU_VERSION: ${{ env.BRANCH }} + TRAVIS_BRANCH: ${{ env.BRANCH }} + TRAVIS_COMMIT_MESSAGE: ${{ github.event.head_commit.message }} + run: bash tests/deploy.sh + + # This job is watched by bors. It only complets if building,testing and deploy worked. + ci-success: + name: CI-Done + #Returns true when none of the **previous** steps have failed or been canceled. + if: ${{ success() }} + needs: + - deploy + runs-on: ubuntu-latest + steps: + - name: CI/CD succeeded. + run: exit 0 + + diff --git a/.travis.yml b/.travis.yml deleted file mode 100644 index f2a85630..00000000 --- a/.travis.yml +++ /dev/null @@ -1,56 +0,0 @@ -branches: - only: - - staging - - testing - - '1.5' - - '1.6' - - '1.7' - - '1.8' - - master - # version tags, e.g. 1.7.1 - - /^1\.[5678]\.\d+$/ - # pre-releases, e.g. 1.8-pre1 - - /^1\.8-pre\d+$/ - # test branches, e.g. test-debian - - /^test-[\w\-\.]+$/ - -sudo: required -services: docker -addons: - apt: - packages: - - docker-ce - -env: - - MAILU_VERSION=${TRAVIS_BRANCH////-} - -language: python -python: - - "3.6" -install: - - pip install -r tests/requirements.txt - - sudo curl -L https://github.com/docker/compose/releases/download/1.23.0-rc3/docker-compose-`uname -s`-`uname -m` -o /usr/local/bin/docker-compose - - sudo chmod +x /usr/local/bin/docker-compose - -before_script: - - docker-compose -v - - echo "$DOCKER_PW" | docker login --username $DOCKER_UN --password-stdin - - docker-compose -f tests/build.yml build - - sudo -- sh -c 'mkdir -p /mailu && cp -r tests/certs /mailu && chmod 600 /mailu/certs/*' - - -script: -# test.py, test name and timeout between start and tests. - - python tests/compose/test.py core 1 - - python tests/compose/test.py fetchmail 1 - - travis_wait python tests/compose/test.py filters 10 - - python tests/compose/test.py rainloop 1 - - python tests/compose/test.py roundcube 1 - - python tests/compose/test.py webdav 1 - -deploy: - provider: script - script: bash tests/deploy.sh - on: - all_branches: true - condition: -n $DOCKER_UN diff --git a/bors.toml b/bors.toml index 5279fe72..272a6047 100644 --- a/bors.toml +++ b/bors.toml @@ -1,3 +1,4 @@ status = [ - "continuous-integration/travis-ci/push" + "CI-Done" ] + diff --git a/tests/compose/filters/00_create_users.sh b/tests/compose/filters/00_create_users.sh new file mode 100755 index 00000000..3c581685 --- /dev/null +++ b/tests/compose/filters/00_create_users.sh @@ -0,0 +1,5 @@ +echo "Creating user required for next test ..." +# Should not fail and update the password; update mode +docker-compose -f tests/compose/filters/docker-compose.yml exec -T admin flask mailu admin admin mailu.io 'password' --mode=update || exit 1 +docker-compose -f tests/compose/filters/docker-compose.yml exec -T admin flask mailu user user mailu.io 'password' || exit 1 +echo "User created successfully" From 54dd4cf224b510f657f0b07696cebf9e8d89d7c2 Mon Sep 17 00:00:00 2001 From: Dimitri Huisman Date: Sat, 26 Jun 2021 19:16:56 +0000 Subject: [PATCH 197/596] Added new docker repo for test image. Adapted deploy script to use env var for test repo name. Modified travis references to github actions references in docs. Added changelog entry. --- .github/worfklows/CI.yml | 4 +++- docs/contributors/environment.rst | 14 +++++++------- docs/contributors/workflow.rst | 2 +- docs/faq.rst | 2 +- tests/deploy.sh | 2 +- towncrier/newsfragments/1828.misc | 1 + 6 files changed, 14 insertions(+), 11 deletions(-) create mode 100644 towncrier/newsfragments/1828.misc diff --git a/.github/worfklows/CI.yml b/.github/worfklows/CI.yml index 3dcaf096..421fec3c 100644 --- a/.github/worfklows/CI.yml +++ b/.github/worfklows/CI.yml @@ -26,7 +26,8 @@ on: # Password of docker login for pushing the images to repo $DOCKER_ORG # DOCKER_ORG: ${{ secrets.DOCKER_ORG }} # The docker repository where the images are pushed to. -# +# DOCKER_ORG_TESTS: ${{ secrets.DOCKER_ORG_TESTS }} +# The docker repository for test images. Only used for the branch TESTING (BORS try). # Add the above secrets to your github repo to determine where the images will be pushed. ################################################ @@ -256,6 +257,7 @@ jobs: DOCKER_UN: ${{ secrets.Docker_Login }} DOCKER_PW: ${{ secrets.Docker_Password }} DOCKER_ORG: ${{ secrets.DOCKER_ORG }} + DOCKER_ORG_TESTS: ${{ secrets.DOCKER_ORG_TESTS }} MAILU_VERSION: ${{ env.BRANCH }} TRAVIS_BRANCH: ${{ env.BRANCH }} TRAVIS_COMMIT_MESSAGE: ${{ github.event.head_commit.message }} diff --git a/docs/contributors/environment.rst b/docs/contributors/environment.rst index 26c04d0b..cef71c6c 100644 --- a/docs/contributors/environment.rst +++ b/docs/contributors/environment.rst @@ -178,9 +178,9 @@ In the case of a PR from a fellow team member, a single review is enough to initiate merging. In all other cases, two approving reviews are required. There is also a possibility to set the ``review/need2`` to require a second review. -After Travis successfully tests the PR and the required amount of reviews are acquired, +After the Github Action workflow successfully tests the PR and the required amount of reviews are acquired, Mergify will trigger with a ``bors r+`` command. Bors will batch any approved PR's, -merges them with master in a staging branch where Travis builds and tests the result. +merges them with master in a staging branch where the Github Action workflow builds and tests the result. After a successful test, the actual master gets fast-forwarded to that point. System requirements @@ -201,16 +201,16 @@ us on `Matrix`_. Test images ``````````` -All PR's automatically get build by Travis, controlled by `bors-ng`_. +All PR's automatically get build by a Github Action workflow, controlled by `bors-ng`_. Some primitive auto testing is done. The resulting images get uploaded to Docker hub, under the -tag name ``mailutest/:pr-``. +tag name ``mailuci/:pr-``. For example, to test PR #500 against master, reviewers can use: .. code-block:: bash - export DOCKER_ORG="mailutest" + export DOCKER_ORG="mailuci" export MAILU_VERSION="pr-500" docker-compose pull docker-compose up -d @@ -232,8 +232,8 @@ after Bors confirms a successful build. When bors try fails ``````````````````` -Sometimes Travis fails when another PR triggers a ``bors try`` command, -before Travis cloned the git repository. +Sometimes the Github Action workflow fails when another PR triggers a ``bors try`` command, +before the Github Action workflow cloned the git repository. Inspect the build log in the link provided by *bors-ng* to find out the cause. If you see something like the following error on top of the logs, feel free to write a comment with ``bors retry``. diff --git a/docs/contributors/workflow.rst b/docs/contributors/workflow.rst index 16dcef52..31ffd793 100644 --- a/docs/contributors/workflow.rst +++ b/docs/contributors/workflow.rst @@ -41,7 +41,7 @@ PR Workflow ----------- All pull requests have to be against the main ``master`` branch. -The PR gets build by Travis and some primitive auto-testing is done. +The PR gets build by a Github Action workflow and some primitive auto-testing is done. Test images get uploaded to a separate section in Docker hub. Reviewers will check the PR and test the resulting images. See the :ref:`testing` section for more info. diff --git a/docs/faq.rst b/docs/faq.rst index 9c4f1d75..c4cea444 100644 --- a/docs/faq.rst +++ b/docs/faq.rst @@ -61,7 +61,7 @@ have to prevent pushing out something quickly. We currently maintain a strict work flow: #. Someone writes a solution and sends a pull request; -#. We use Travis-CI for some very basic building and testing; +#. We use Github actions for some very basic building and testing; #. The pull request needs to be code-reviewed and tested by at least two members from the contributors team. diff --git a/tests/deploy.sh b/tests/deploy.sh index 21aec444..a836417b 100755 --- a/tests/deploy.sh +++ b/tests/deploy.sh @@ -5,7 +5,7 @@ # Retag in case of `bors try` if [ "$TRAVIS_BRANCH" = "testing" ]; then - export DOCKER_ORG="mailutest" + export DOCKER_ORG=$DOCKER_ORG_TESTS # Commit message is like "Try #99". # This sets the version tag to "pr-99" export MAILU_VERSION="pr-${TRAVIS_COMMIT_MESSAGE//[!0-9]/}" diff --git a/towncrier/newsfragments/1828.misc b/towncrier/newsfragments/1828.misc new file mode 100644 index 00000000..09da59ad --- /dev/null +++ b/towncrier/newsfragments/1828.misc @@ -0,0 +1 @@ +Switched from Travis to Github actions for CI/CD. Improved CI workflow to perform all tests in parallel. From c6a38bbbccafd8e2623645da6c5759c9a23b4382 Mon Sep 17 00:00:00 2001 From: Dimitri Huisman <52963853+Diman0@users.noreply.github.com> Date: Sat, 26 Jun 2021 21:50:55 +0200 Subject: [PATCH 198/596] Update CI.yml --- .github/worfklows/CI.yml | 2 -- 1 file changed, 2 deletions(-) diff --git a/.github/worfklows/CI.yml b/.github/worfklows/CI.yml index 421fec3c..e0462e1e 100644 --- a/.github/worfklows/CI.yml +++ b/.github/worfklows/CI.yml @@ -1,5 +1,3 @@ - - name: CI on: push: From fb30a62629b6372450b34b6c9a7238d40013ef58 Mon Sep 17 00:00:00 2001 From: Dimitri Huisman <52963853+Diman0@users.noreply.github.com> Date: Sat, 26 Jun 2021 21:52:09 +0200 Subject: [PATCH 199/596] Create CI.yml --- .github/workflows/CI.yml | 276 +++++++++++++++++++++++++++++++++++++++ 1 file changed, 276 insertions(+) create mode 100644 .github/workflows/CI.yml diff --git a/.github/workflows/CI.yml b/.github/workflows/CI.yml new file mode 100644 index 00000000..e0462e1e --- /dev/null +++ b/.github/workflows/CI.yml @@ -0,0 +1,276 @@ +name: CI +on: + push: + branches: + - staging + - testing + - '1.5' + - '1.6' + - '1.7' + - '1.8' + - master + # version tags, e.g. 1.7.1 + - '[1-9].[0-9].[0-9]' + # pre-releases, e.g. 1.8-pre1 + - 1.8-pre[0-9] + # test branches, e.g. test-debian + - test-* + +############################################### +# REQUIRED secrets +# DOCKER_UN: ${{ secrets.Docker_Login }} +# Username of docker login for pushing the images to repo $DOCKER_ORG +# DOCKER_PW: ${{ secrets.Docker_Password }} +# Password of docker login for pushing the images to repo $DOCKER_ORG +# DOCKER_ORG: ${{ secrets.DOCKER_ORG }} +# The docker repository where the images are pushed to. +# DOCKER_ORG_TESTS: ${{ secrets.DOCKER_ORG_TESTS }} +# The docker repository for test images. Only used for the branch TESTING (BORS try). +# Add the above secrets to your github repo to determine where the images will be pushed. +################################################ + +jobs: + build: + name: build + runs-on: ubuntu-latest + + steps: + - uses: actions/checkout@v2 + - name: Extract branch name + shell: bash + run: | + echo "BRANCH=${GITHUB_REF#refs/heads/}" >> $GITHUB_ENV + - name: install python packages + run: python3 -m pip install -r tests/requirements.txt + - name: check docker-compose version + run: docker-compose -v + - name: login docker + env: + DOCKER_UN: ${{ secrets.Docker_Login }} + DOCKER_PW: ${{ secrets.Docker_Password }} + DOCKER_ORG: ${{ secrets.DOCKER_ORG }} + run: echo "$DOCKER_PW" | docker login --username $DOCKER_UN --password-stdin + # In this step, this action saves a list of existing images, + # the cache is created without them in the post run. + # It also restores the cache if it exists. + - uses: satackey/action-docker-layer-caching@v0.0.11 + # Ignore the failure of a step and avoid terminating the job. + continue-on-error: true + - name: build all docker images + env: + MAILU_VERSION: ${{ env.BRANCH }} + TRAVIS_BRANCH: ${{ env.BRANCH }} + DOCKER_ORG: ${{ secrets.DOCKER_ORG }} + run: docker-compose -f tests/build.yml build + +#NOTE: It appears the filter test depends on the core test. The filter test requires an email user +#that is created by the core test. + core-test: + name: core test + runs-on: ubuntu-latest + needs: + - build + steps: + - uses: actions/checkout@v2 + - name: Extract branch name + shell: bash + run: | + echo "BRANCH=${GITHUB_REF#refs/heads/}" >> $GITHUB_ENV + - name: install python packages + run: python3 -m pip install -r tests/requirements.txt + - uses: satackey/action-docker-layer-caching@v0.0.11 + # Ignore the failure of a step and avoid terminating the job. + continue-on-error: true + - name: copy all certs + run: sudo -- sh -c 'mkdir -p /mailu && cp -r tests/certs /mailu && chmod 600 /mailu/certs/*' + - name: test core suite + run: python tests/compose/test.py core 1 + env: + MAILU_VERSION: ${{ env.BRANCH }} + TRAVIS_BRANCH: ${{ env.BRANCH }} + DOCKER_ORG: ${{ secrets.DOCKER_ORG }} + + filter-test: + name: filter test + runs-on: ubuntu-latest + needs: + - build + steps: + - uses: actions/checkout@v2 + - name: Extract branch name + shell: bash + run: | + echo "BRANCH=${GITHUB_REF#refs/heads/}" >> $GITHUB_ENV + - name: install python packages + run: python3 -m pip install -r tests/requirements.txt + - uses: satackey/action-docker-layer-caching@v0.0.11 + # Ignore the failure of a step and avoid terminating the job. + continue-on-error: true + - name: copy all certs + run: sudo -- sh -c 'mkdir -p /mailu && cp -r tests/certs /mailu && chmod 600 /mailu/certs/*' + - name: 'test clamvav' + run: python tests/compose/test.py filters 2 + env: + MAILU_VERSION: ${{ env.BRANCH }} + TRAVIS_BRANCH: ${{ env.BRANCH }} + DOCKER_ORG: ${{ secrets.DOCKER_ORG }} + + fetch-test: + name: fetch test + runs-on: ubuntu-latest + needs: + - build + steps: + - uses: actions/checkout@v2 + - name: Extract branch name + shell: bash + run: | + echo "BRANCH=${GITHUB_REF#refs/heads/}" >> $GITHUB_ENV + - name: install python packages + run: python3 -m pip install -r tests/requirements.txt + - uses: satackey/action-docker-layer-caching@v0.0.11 + # Ignore the failure of a step and avoid terminating the job. + continue-on-error: true + - name: copy all certs + run: sudo -- sh -c 'mkdir -p /mailu && cp -r tests/certs /mailu && chmod 600 /mailu/certs/*' + - name: test fetch + run: python tests/compose/test.py fetchmail 1 + env: + MAILU_VERSION: ${{ env.BRANCH }} + TRAVIS_BRANCH: ${{ env.BRANCH }} + DOCKER_ORG: ${{ secrets.DOCKER_ORG }} + + rainloop-test: + name: rainloop test + runs-on: ubuntu-latest + needs: + - build + steps: + - uses: actions/checkout@v2 + - name: Extract branch name + shell: bash + run: | + echo "BRANCH=${GITHUB_REF#refs/heads/}" >> $GITHUB_ENV + - name: install python packages + run: python3 -m pip install -r tests/requirements.txt + - uses: satackey/action-docker-layer-caching@v0.0.11 + # Ignore the failure of a step and avoid terminating the job. + continue-on-error: true + - name: copy all certs + run: sudo -- sh -c 'mkdir -p /mailu && cp -r tests/certs /mailu && chmod 600 /mailu/certs/*' + - name: test rainloop + run: python tests/compose/test.py rainloop 1 + env: + MAILU_VERSION: ${{ env.BRANCH }} + TRAVIS_BRANCH: ${{ env.BRANCH }} + DOCKER_ORG: ${{ secrets.DOCKER_ORG }} + + roundcube-test: + name: roundcube test + runs-on: ubuntu-latest + needs: + - build + steps: + - uses: actions/checkout@v2 + - name: Extract branch name + shell: bash + run: | + echo "BRANCH=${GITHUB_REF#refs/heads/}" >> $GITHUB_ENV + - name: install python packages + run: python3 -m pip install -r tests/requirements.txt + - uses: satackey/action-docker-layer-caching@v0.0.11 + # Ignore the failure of a step and avoid terminating the job. + continue-on-error: true + - name: copy all certs + run: sudo -- sh -c 'mkdir -p /mailu && cp -r tests/certs /mailu && chmod 600 /mailu/certs/*' + - name: test roundcube + run: python tests/compose/test.py roundcube 1 + env: + MAILU_VERSION: ${{ env.BRANCH }} + TRAVIS_BRANCH: ${{ env.BRANCH }} + DOCKER_ORG: ${{ secrets.DOCKER_ORG }} + + webdav-test: + name: webdav test + runs-on: ubuntu-latest + needs: + - build + steps: + - uses: actions/checkout@v2 + - name: Extract branch name + shell: bash + run: | + echo "BRANCH=${GITHUB_REF#refs/heads/}" >> $GITHUB_ENV + - name: install python packages + run: python3 -m pip install -r tests/requirements.txt + - uses: satackey/action-docker-layer-caching@v0.0.11 + # Ignore the failure of a step and avoid terminating the job. + continue-on-error: true + - name: copy all certs + run: sudo -- sh -c 'mkdir -p /mailu && cp -r tests/certs /mailu && chmod 600 /mailu/certs/*' + - name: test webdav + run: python tests/compose/test.py webdav 1 + env: + MAILU_VERSION: ${{ env.BRANCH }} + TRAVIS_BRANCH: ${{ env.BRANCH }} + DOCKER_ORG: ${{ secrets.DOCKER_ORG }} + + deploy: + name: deploy step + runs-on: ubuntu-latest + needs: + - build + - core-test + - filter-test + - fetch-test + - rainloop-test + - roundcube-test + - webdav-test + steps: + - uses: actions/checkout@v2 + - name: Extract branch name + shell: bash + run: | + echo "BRANCH=${GITHUB_REF#refs/heads/}" >> $GITHUB_ENV + - name: install python packages + run: python3 -m pip install -r tests/requirements.txt + - uses: satackey/action-docker-layer-caching@v0.0.11 + # Ignore the failure of a step and avoid terminating the job. + continue-on-error: true + - name: copy all certs + run: sudo -- sh -c 'mkdir -p /mailu && cp -r tests/certs /mailu && chmod 600 /mailu/certs/*' + - name: login docker + env: + DOCKER_UN: ${{ secrets.Docker_Login }} + DOCKER_PW: ${{ secrets.Docker_Password }} + run: echo "$DOCKER_PW" | docker login --username $DOCKER_UN --password-stdin + - name: build all docker images + run: docker-compose -f tests/build.yml build + env: + MAILU_VERSION: ${{ env.BRANCH }} + TRAVIS_BRANCH: ${{ env.BRANCH }} + DOCKER_ORG: ${{ secrets.DOCKER_ORG }} + - name: deploy built docker images + env: + DOCKER_UN: ${{ secrets.Docker_Login }} + DOCKER_PW: ${{ secrets.Docker_Password }} + DOCKER_ORG: ${{ secrets.DOCKER_ORG }} + DOCKER_ORG_TESTS: ${{ secrets.DOCKER_ORG_TESTS }} + MAILU_VERSION: ${{ env.BRANCH }} + TRAVIS_BRANCH: ${{ env.BRANCH }} + TRAVIS_COMMIT_MESSAGE: ${{ github.event.head_commit.message }} + run: bash tests/deploy.sh + + # This job is watched by bors. It only complets if building,testing and deploy worked. + ci-success: + name: CI-Done + #Returns true when none of the **previous** steps have failed or been canceled. + if: ${{ success() }} + needs: + - deploy + runs-on: ubuntu-latest + steps: + - name: CI/CD succeeded. + run: exit 0 + + From 006da4c5e409526ccf5fb3e62ac512d3d5e79593 Mon Sep 17 00:00:00 2001 From: Dimitri Huisman <52963853+Diman0@users.noreply.github.com> Date: Sat, 26 Jun 2021 21:52:51 +0200 Subject: [PATCH 200/596] My mistake. A typo --- .github/worfklows/CI.yml | 276 --------------------------------------- 1 file changed, 276 deletions(-) delete mode 100644 .github/worfklows/CI.yml diff --git a/.github/worfklows/CI.yml b/.github/worfklows/CI.yml deleted file mode 100644 index e0462e1e..00000000 --- a/.github/worfklows/CI.yml +++ /dev/null @@ -1,276 +0,0 @@ -name: CI -on: - push: - branches: - - staging - - testing - - '1.5' - - '1.6' - - '1.7' - - '1.8' - - master - # version tags, e.g. 1.7.1 - - '[1-9].[0-9].[0-9]' - # pre-releases, e.g. 1.8-pre1 - - 1.8-pre[0-9] - # test branches, e.g. test-debian - - test-* - -############################################### -# REQUIRED secrets -# DOCKER_UN: ${{ secrets.Docker_Login }} -# Username of docker login for pushing the images to repo $DOCKER_ORG -# DOCKER_PW: ${{ secrets.Docker_Password }} -# Password of docker login for pushing the images to repo $DOCKER_ORG -# DOCKER_ORG: ${{ secrets.DOCKER_ORG }} -# The docker repository where the images are pushed to. -# DOCKER_ORG_TESTS: ${{ secrets.DOCKER_ORG_TESTS }} -# The docker repository for test images. Only used for the branch TESTING (BORS try). -# Add the above secrets to your github repo to determine where the images will be pushed. -################################################ - -jobs: - build: - name: build - runs-on: ubuntu-latest - - steps: - - uses: actions/checkout@v2 - - name: Extract branch name - shell: bash - run: | - echo "BRANCH=${GITHUB_REF#refs/heads/}" >> $GITHUB_ENV - - name: install python packages - run: python3 -m pip install -r tests/requirements.txt - - name: check docker-compose version - run: docker-compose -v - - name: login docker - env: - DOCKER_UN: ${{ secrets.Docker_Login }} - DOCKER_PW: ${{ secrets.Docker_Password }} - DOCKER_ORG: ${{ secrets.DOCKER_ORG }} - run: echo "$DOCKER_PW" | docker login --username $DOCKER_UN --password-stdin - # In this step, this action saves a list of existing images, - # the cache is created without them in the post run. - # It also restores the cache if it exists. - - uses: satackey/action-docker-layer-caching@v0.0.11 - # Ignore the failure of a step and avoid terminating the job. - continue-on-error: true - - name: build all docker images - env: - MAILU_VERSION: ${{ env.BRANCH }} - TRAVIS_BRANCH: ${{ env.BRANCH }} - DOCKER_ORG: ${{ secrets.DOCKER_ORG }} - run: docker-compose -f tests/build.yml build - -#NOTE: It appears the filter test depends on the core test. The filter test requires an email user -#that is created by the core test. - core-test: - name: core test - runs-on: ubuntu-latest - needs: - - build - steps: - - uses: actions/checkout@v2 - - name: Extract branch name - shell: bash - run: | - echo "BRANCH=${GITHUB_REF#refs/heads/}" >> $GITHUB_ENV - - name: install python packages - run: python3 -m pip install -r tests/requirements.txt - - uses: satackey/action-docker-layer-caching@v0.0.11 - # Ignore the failure of a step and avoid terminating the job. - continue-on-error: true - - name: copy all certs - run: sudo -- sh -c 'mkdir -p /mailu && cp -r tests/certs /mailu && chmod 600 /mailu/certs/*' - - name: test core suite - run: python tests/compose/test.py core 1 - env: - MAILU_VERSION: ${{ env.BRANCH }} - TRAVIS_BRANCH: ${{ env.BRANCH }} - DOCKER_ORG: ${{ secrets.DOCKER_ORG }} - - filter-test: - name: filter test - runs-on: ubuntu-latest - needs: - - build - steps: - - uses: actions/checkout@v2 - - name: Extract branch name - shell: bash - run: | - echo "BRANCH=${GITHUB_REF#refs/heads/}" >> $GITHUB_ENV - - name: install python packages - run: python3 -m pip install -r tests/requirements.txt - - uses: satackey/action-docker-layer-caching@v0.0.11 - # Ignore the failure of a step and avoid terminating the job. - continue-on-error: true - - name: copy all certs - run: sudo -- sh -c 'mkdir -p /mailu && cp -r tests/certs /mailu && chmod 600 /mailu/certs/*' - - name: 'test clamvav' - run: python tests/compose/test.py filters 2 - env: - MAILU_VERSION: ${{ env.BRANCH }} - TRAVIS_BRANCH: ${{ env.BRANCH }} - DOCKER_ORG: ${{ secrets.DOCKER_ORG }} - - fetch-test: - name: fetch test - runs-on: ubuntu-latest - needs: - - build - steps: - - uses: actions/checkout@v2 - - name: Extract branch name - shell: bash - run: | - echo "BRANCH=${GITHUB_REF#refs/heads/}" >> $GITHUB_ENV - - name: install python packages - run: python3 -m pip install -r tests/requirements.txt - - uses: satackey/action-docker-layer-caching@v0.0.11 - # Ignore the failure of a step and avoid terminating the job. - continue-on-error: true - - name: copy all certs - run: sudo -- sh -c 'mkdir -p /mailu && cp -r tests/certs /mailu && chmod 600 /mailu/certs/*' - - name: test fetch - run: python tests/compose/test.py fetchmail 1 - env: - MAILU_VERSION: ${{ env.BRANCH }} - TRAVIS_BRANCH: ${{ env.BRANCH }} - DOCKER_ORG: ${{ secrets.DOCKER_ORG }} - - rainloop-test: - name: rainloop test - runs-on: ubuntu-latest - needs: - - build - steps: - - uses: actions/checkout@v2 - - name: Extract branch name - shell: bash - run: | - echo "BRANCH=${GITHUB_REF#refs/heads/}" >> $GITHUB_ENV - - name: install python packages - run: python3 -m pip install -r tests/requirements.txt - - uses: satackey/action-docker-layer-caching@v0.0.11 - # Ignore the failure of a step and avoid terminating the job. - continue-on-error: true - - name: copy all certs - run: sudo -- sh -c 'mkdir -p /mailu && cp -r tests/certs /mailu && chmod 600 /mailu/certs/*' - - name: test rainloop - run: python tests/compose/test.py rainloop 1 - env: - MAILU_VERSION: ${{ env.BRANCH }} - TRAVIS_BRANCH: ${{ env.BRANCH }} - DOCKER_ORG: ${{ secrets.DOCKER_ORG }} - - roundcube-test: - name: roundcube test - runs-on: ubuntu-latest - needs: - - build - steps: - - uses: actions/checkout@v2 - - name: Extract branch name - shell: bash - run: | - echo "BRANCH=${GITHUB_REF#refs/heads/}" >> $GITHUB_ENV - - name: install python packages - run: python3 -m pip install -r tests/requirements.txt - - uses: satackey/action-docker-layer-caching@v0.0.11 - # Ignore the failure of a step and avoid terminating the job. - continue-on-error: true - - name: copy all certs - run: sudo -- sh -c 'mkdir -p /mailu && cp -r tests/certs /mailu && chmod 600 /mailu/certs/*' - - name: test roundcube - run: python tests/compose/test.py roundcube 1 - env: - MAILU_VERSION: ${{ env.BRANCH }} - TRAVIS_BRANCH: ${{ env.BRANCH }} - DOCKER_ORG: ${{ secrets.DOCKER_ORG }} - - webdav-test: - name: webdav test - runs-on: ubuntu-latest - needs: - - build - steps: - - uses: actions/checkout@v2 - - name: Extract branch name - shell: bash - run: | - echo "BRANCH=${GITHUB_REF#refs/heads/}" >> $GITHUB_ENV - - name: install python packages - run: python3 -m pip install -r tests/requirements.txt - - uses: satackey/action-docker-layer-caching@v0.0.11 - # Ignore the failure of a step and avoid terminating the job. - continue-on-error: true - - name: copy all certs - run: sudo -- sh -c 'mkdir -p /mailu && cp -r tests/certs /mailu && chmod 600 /mailu/certs/*' - - name: test webdav - run: python tests/compose/test.py webdav 1 - env: - MAILU_VERSION: ${{ env.BRANCH }} - TRAVIS_BRANCH: ${{ env.BRANCH }} - DOCKER_ORG: ${{ secrets.DOCKER_ORG }} - - deploy: - name: deploy step - runs-on: ubuntu-latest - needs: - - build - - core-test - - filter-test - - fetch-test - - rainloop-test - - roundcube-test - - webdav-test - steps: - - uses: actions/checkout@v2 - - name: Extract branch name - shell: bash - run: | - echo "BRANCH=${GITHUB_REF#refs/heads/}" >> $GITHUB_ENV - - name: install python packages - run: python3 -m pip install -r tests/requirements.txt - - uses: satackey/action-docker-layer-caching@v0.0.11 - # Ignore the failure of a step and avoid terminating the job. - continue-on-error: true - - name: copy all certs - run: sudo -- sh -c 'mkdir -p /mailu && cp -r tests/certs /mailu && chmod 600 /mailu/certs/*' - - name: login docker - env: - DOCKER_UN: ${{ secrets.Docker_Login }} - DOCKER_PW: ${{ secrets.Docker_Password }} - run: echo "$DOCKER_PW" | docker login --username $DOCKER_UN --password-stdin - - name: build all docker images - run: docker-compose -f tests/build.yml build - env: - MAILU_VERSION: ${{ env.BRANCH }} - TRAVIS_BRANCH: ${{ env.BRANCH }} - DOCKER_ORG: ${{ secrets.DOCKER_ORG }} - - name: deploy built docker images - env: - DOCKER_UN: ${{ secrets.Docker_Login }} - DOCKER_PW: ${{ secrets.Docker_Password }} - DOCKER_ORG: ${{ secrets.DOCKER_ORG }} - DOCKER_ORG_TESTS: ${{ secrets.DOCKER_ORG_TESTS }} - MAILU_VERSION: ${{ env.BRANCH }} - TRAVIS_BRANCH: ${{ env.BRANCH }} - TRAVIS_COMMIT_MESSAGE: ${{ github.event.head_commit.message }} - run: bash tests/deploy.sh - - # This job is watched by bors. It only complets if building,testing and deploy worked. - ci-success: - name: CI-Done - #Returns true when none of the **previous** steps have failed or been canceled. - if: ${{ success() }} - needs: - - deploy - runs-on: ubuntu-latest - steps: - - name: CI/CD succeeded. - run: exit 0 - - From 606c039a6ffbcb63bb28bb1b422d0e622ebc0088 Mon Sep 17 00:00:00 2001 From: Dimitri Huisman Date: Sat, 26 Jun 2021 21:00:51 +0000 Subject: [PATCH 201/596] Switch back to sequential workflow --- .github/workflows/CI.yml | 126 ++--------------------- tests/compose/filters/00_create_users.sh | 5 - 2 files changed, 6 insertions(+), 125 deletions(-) delete mode 100755 tests/compose/filters/00_create_users.sh diff --git a/.github/workflows/CI.yml b/.github/workflows/CI.yml index e0462e1e..8cd9a8d3 100644 --- a/.github/workflows/CI.yml +++ b/.github/workflows/CI.yml @@ -30,8 +30,8 @@ on: ################################################ jobs: - build: - name: build + build-test: + name: build and test runs-on: ubuntu-latest steps: @@ -63,24 +63,6 @@ jobs: DOCKER_ORG: ${{ secrets.DOCKER_ORG }} run: docker-compose -f tests/build.yml build -#NOTE: It appears the filter test depends on the core test. The filter test requires an email user -#that is created by the core test. - core-test: - name: core test - runs-on: ubuntu-latest - needs: - - build - steps: - - uses: actions/checkout@v2 - - name: Extract branch name - shell: bash - run: | - echo "BRANCH=${GITHUB_REF#refs/heads/}" >> $GITHUB_ENV - - name: install python packages - run: python3 -m pip install -r tests/requirements.txt - - uses: satackey/action-docker-layer-caching@v0.0.11 - # Ignore the failure of a step and avoid terminating the job. - continue-on-error: true - name: copy all certs run: sudo -- sh -c 'mkdir -p /mailu && cp -r tests/certs /mailu && chmod 600 /mailu/certs/*' - name: test core suite @@ -90,74 +72,20 @@ jobs: TRAVIS_BRANCH: ${{ env.BRANCH }} DOCKER_ORG: ${{ secrets.DOCKER_ORG }} - filter-test: - name: filter test - runs-on: ubuntu-latest - needs: - - build - steps: - - uses: actions/checkout@v2 - - name: Extract branch name - shell: bash - run: | - echo "BRANCH=${GITHUB_REF#refs/heads/}" >> $GITHUB_ENV - - name: install python packages - run: python3 -m pip install -r tests/requirements.txt - - uses: satackey/action-docker-layer-caching@v0.0.11 - # Ignore the failure of a step and avoid terminating the job. - continue-on-error: true - - name: copy all certs - run: sudo -- sh -c 'mkdir -p /mailu && cp -r tests/certs /mailu && chmod 600 /mailu/certs/*' - name: 'test clamvav' run: python tests/compose/test.py filters 2 env: MAILU_VERSION: ${{ env.BRANCH }} TRAVIS_BRANCH: ${{ env.BRANCH }} DOCKER_ORG: ${{ secrets.DOCKER_ORG }} - - fetch-test: - name: fetch test - runs-on: ubuntu-latest - needs: - - build - steps: - - uses: actions/checkout@v2 - - name: Extract branch name - shell: bash - run: | - echo "BRANCH=${GITHUB_REF#refs/heads/}" >> $GITHUB_ENV - - name: install python packages - run: python3 -m pip install -r tests/requirements.txt - - uses: satackey/action-docker-layer-caching@v0.0.11 - # Ignore the failure of a step and avoid terminating the job. - continue-on-error: true - - name: copy all certs - run: sudo -- sh -c 'mkdir -p /mailu && cp -r tests/certs /mailu && chmod 600 /mailu/certs/*' + - name: test fetch run: python tests/compose/test.py fetchmail 1 env: MAILU_VERSION: ${{ env.BRANCH }} TRAVIS_BRANCH: ${{ env.BRANCH }} DOCKER_ORG: ${{ secrets.DOCKER_ORG }} - - rainloop-test: - name: rainloop test - runs-on: ubuntu-latest - needs: - - build - steps: - - uses: actions/checkout@v2 - - name: Extract branch name - shell: bash - run: | - echo "BRANCH=${GITHUB_REF#refs/heads/}" >> $GITHUB_ENV - - name: install python packages - run: python3 -m pip install -r tests/requirements.txt - - uses: satackey/action-docker-layer-caching@v0.0.11 - # Ignore the failure of a step and avoid terminating the job. - continue-on-error: true - - name: copy all certs - run: sudo -- sh -c 'mkdir -p /mailu && cp -r tests/certs /mailu && chmod 600 /mailu/certs/*' + - name: test rainloop run: python tests/compose/test.py rainloop 1 env: @@ -165,49 +93,13 @@ jobs: TRAVIS_BRANCH: ${{ env.BRANCH }} DOCKER_ORG: ${{ secrets.DOCKER_ORG }} - roundcube-test: - name: roundcube test - runs-on: ubuntu-latest - needs: - - build - steps: - - uses: actions/checkout@v2 - - name: Extract branch name - shell: bash - run: | - echo "BRANCH=${GITHUB_REF#refs/heads/}" >> $GITHUB_ENV - - name: install python packages - run: python3 -m pip install -r tests/requirements.txt - - uses: satackey/action-docker-layer-caching@v0.0.11 - # Ignore the failure of a step and avoid terminating the job. - continue-on-error: true - - name: copy all certs - run: sudo -- sh -c 'mkdir -p /mailu && cp -r tests/certs /mailu && chmod 600 /mailu/certs/*' - name: test roundcube run: python tests/compose/test.py roundcube 1 env: MAILU_VERSION: ${{ env.BRANCH }} TRAVIS_BRANCH: ${{ env.BRANCH }} DOCKER_ORG: ${{ secrets.DOCKER_ORG }} - - webdav-test: - name: webdav test - runs-on: ubuntu-latest - needs: - - build - steps: - - uses: actions/checkout@v2 - - name: Extract branch name - shell: bash - run: | - echo "BRANCH=${GITHUB_REF#refs/heads/}" >> $GITHUB_ENV - - name: install python packages - run: python3 -m pip install -r tests/requirements.txt - - uses: satackey/action-docker-layer-caching@v0.0.11 - # Ignore the failure of a step and avoid terminating the job. - continue-on-error: true - - name: copy all certs - run: sudo -- sh -c 'mkdir -p /mailu && cp -r tests/certs /mailu && chmod 600 /mailu/certs/*' + - name: test webdav run: python tests/compose/test.py webdav 1 env: @@ -219,13 +111,7 @@ jobs: name: deploy step runs-on: ubuntu-latest needs: - - build - - core-test - - filter-test - - fetch-test - - rainloop-test - - roundcube-test - - webdav-test + - build-test steps: - uses: actions/checkout@v2 - name: Extract branch name diff --git a/tests/compose/filters/00_create_users.sh b/tests/compose/filters/00_create_users.sh deleted file mode 100755 index 3c581685..00000000 --- a/tests/compose/filters/00_create_users.sh +++ /dev/null @@ -1,5 +0,0 @@ -echo "Creating user required for next test ..." -# Should not fail and update the password; update mode -docker-compose -f tests/compose/filters/docker-compose.yml exec -T admin flask mailu admin admin mailu.io 'password' --mode=update || exit 1 -docker-compose -f tests/compose/filters/docker-compose.yml exec -T admin flask mailu user user mailu.io 'password' || exit 1 -echo "User created successfully" From 24200ddb670cf500d83695241bb158a4324e94b7 Mon Sep 17 00:00:00 2001 From: Dimitri Huisman Date: Sat, 26 Jun 2021 21:49:37 +0000 Subject: [PATCH 202/596] Forgot to remove duplicate steps when switching back to sequential workflow --- .github/workflows/CI.yml | 35 +++-------------------------------- 1 file changed, 3 insertions(+), 32 deletions(-) diff --git a/.github/workflows/CI.yml b/.github/workflows/CI.yml index 8cd9a8d3..71897c52 100644 --- a/.github/workflows/CI.yml +++ b/.github/workflows/CI.yml @@ -30,7 +30,7 @@ on: ################################################ jobs: - build-test: + build-test-deploy: name: build and test runs-on: ubuntu-latest @@ -106,36 +106,7 @@ jobs: MAILU_VERSION: ${{ env.BRANCH }} TRAVIS_BRANCH: ${{ env.BRANCH }} DOCKER_ORG: ${{ secrets.DOCKER_ORG }} - - deploy: - name: deploy step - runs-on: ubuntu-latest - needs: - - build-test - steps: - - uses: actions/checkout@v2 - - name: Extract branch name - shell: bash - run: | - echo "BRANCH=${GITHUB_REF#refs/heads/}" >> $GITHUB_ENV - - name: install python packages - run: python3 -m pip install -r tests/requirements.txt - - uses: satackey/action-docker-layer-caching@v0.0.11 - # Ignore the failure of a step and avoid terminating the job. - continue-on-error: true - - name: copy all certs - run: sudo -- sh -c 'mkdir -p /mailu && cp -r tests/certs /mailu && chmod 600 /mailu/certs/*' - - name: login docker - env: - DOCKER_UN: ${{ secrets.Docker_Login }} - DOCKER_PW: ${{ secrets.Docker_Password }} - run: echo "$DOCKER_PW" | docker login --username $DOCKER_UN --password-stdin - - name: build all docker images - run: docker-compose -f tests/build.yml build - env: - MAILU_VERSION: ${{ env.BRANCH }} - TRAVIS_BRANCH: ${{ env.BRANCH }} - DOCKER_ORG: ${{ secrets.DOCKER_ORG }} + - name: deploy built docker images env: DOCKER_UN: ${{ secrets.Docker_Login }} @@ -153,7 +124,7 @@ jobs: #Returns true when none of the **previous** steps have failed or been canceled. if: ${{ success() }} needs: - - deploy + - build-test-deploy runs-on: ubuntu-latest steps: - name: CI/CD succeeded. From b560d1f36998c4c2f65875cbb88f8fbc319ffbe2 Mon Sep 17 00:00:00 2001 From: Nicolas Paris Date: Sun, 27 Jun 2021 10:38:32 +0200 Subject: [PATCH 203/596] Improve english Co-authored-by: decentral1se <1991377+decentral1se@users.noreply.github.com> --- docs/database.rst | 4 ++-- 1 file changed, 2 insertions(+), 2 deletions(-) diff --git a/docs/database.rst b/docs/database.rst index c13ca0bf..0f8318d5 100644 --- a/docs/database.rst +++ b/docs/database.rst @@ -8,8 +8,8 @@ This functionality should still be considered experimental! Mailu Postgresql ---------------- -Mailu optionally comes with a pre-configured Postgresql image, wich as of 1.8 is deprecated -will be removed in 1.9 +Mailu optionally comes with a pre-configured Postgresql image, which as of 1.8, is deprecated +and will be removed in 1.9. This images has the following features: - Automatic creation of users, db, extensions and password; From c0c8c4a55113237274ca4b4e32c9c46e5297998f Mon Sep 17 00:00:00 2001 From: Nicolas Paris Date: Sun, 27 Jun 2021 10:46:28 +0200 Subject: [PATCH 204/596] Fix typo Co-authored-by: decentral1se <1991377+decentral1se@users.noreply.github.com> --- setup/templates/steps/database.html | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/setup/templates/steps/database.html b/setup/templates/steps/database.html index d7184110..2727687f 100644 --- a/setup/templates/steps/database.html +++ b/setup/templates/steps/database.html @@ -46,7 +46,7 @@ - +
    From ab7264df0c9a115a7e9cf63cd4ff55e380c3e7f7 Mon Sep 17 00:00:00 2001 From: Nicolas Paris Date: Sun, 27 Jun 2021 10:46:41 +0200 Subject: [PATCH 205/596] Fix typo Co-authored-by: decentral1se <1991377+decentral1se@users.noreply.github.com> --- setup/templates/steps/database.html | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/setup/templates/steps/database.html b/setup/templates/steps/database.html index 2727687f..be32fe0b 100644 --- a/setup/templates/steps/database.html +++ b/setup/templates/steps/database.html @@ -44,7 +44,7 @@ - + From 7386257dedda678419e495c28e382251f7212bfa Mon Sep 17 00:00:00 2001 From: Nicolas Paris Date: Sun, 27 Jun 2021 10:47:28 +0200 Subject: [PATCH 206/596] Fix typo Co-authored-by: decentral1se <1991377+decentral1se@users.noreply.github.com> --- setup/templates/steps/database.html | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/setup/templates/steps/database.html b/setup/templates/steps/database.html index be32fe0b..5e942061 100644 --- a/setup/templates/steps/database.html +++ b/setup/templates/steps/database.html @@ -28,7 +28,7 @@
    - + diff --git a/setup/templates/steps/config.html b/setup/templates/steps/config.html index 29d8dddd..3769a210 100644 --- a/setup/templates/steps/config.html +++ b/setup/templates/steps/config.html @@ -83,7 +83,7 @@ manage your email domains, users, etc.

    - + diff --git a/setup/templates/steps/stack/02_services.html b/setup/templates/steps/stack/02_services.html index 3f5186b0..6fce0ae6 100644 --- a/setup/templates/steps/stack/02_services.html +++ b/setup/templates/steps/stack/02_services.html @@ -55,7 +55,7 @@ the security implications caused by such an increase of attack surface.

    Fetchmail allows users to retrieve mail from an external mail-server via IMAP/POP3 and puts it in their inbox. - + From b0fb9d822bed082ee6630baeaf8526ab75934ff4 Mon Sep 17 00:00:00 2001 From: Dimitri Huisman Date: Thu, 22 Jul 2021 14:44:30 +0000 Subject: [PATCH 257/596] Adapt requirements.txt to use pinned versions. --- setup/requirements.txt | 8 ++++---- 1 file changed, 4 insertions(+), 4 deletions(-) diff --git a/setup/requirements.txt b/setup/requirements.txt index b6bf2120..f2eb2157 100644 --- a/setup/requirements.txt +++ b/setup/requirements.txt @@ -1,4 +1,4 @@ -flask -flask-bootstrap -redis -gunicorn +Flask==1.0.2 +Flask-Bootstrap==3.3.7.1 +gunicorn==19.9.0 +redis==3.2.1 From 67e00bb1e77fa3350610e855c1396952ea76ea21 Mon Sep 17 00:00:00 2001 From: Dimitri Huisman Date: Thu, 22 Jul 2021 14:56:30 +0000 Subject: [PATCH 258/596] Add changelog --- towncrier/newsfragments/1880.feature | 1 + 1 file changed, 1 insertion(+) create mode 100644 towncrier/newsfragments/1880.feature diff --git a/towncrier/newsfragments/1880.feature b/towncrier/newsfragments/1880.feature new file mode 100644 index 00000000..212dc906 --- /dev/null +++ b/towncrier/newsfragments/1880.feature @@ -0,0 +1 @@ +Update jquery used in setup. Set pinned versions in requirements.txt for setup. This is a security update. From 9d2629a04e7aaa8c9fc199dac8a6a0631a857fac Mon Sep 17 00:00:00 2001 From: Florent Daigniere Date: Sat, 24 Jul 2021 12:40:38 +0200 Subject: [PATCH 259/596] fix 1884: always lookup a FQDN --- core/postfix/start.py | 6 +++++- 1 file changed, 5 insertions(+), 1 deletion(-) diff --git a/core/postfix/start.py b/core/postfix/start.py index 701efec3..7a2b57a1 100755 --- a/core/postfix/start.py +++ b/core/postfix/start.py @@ -38,7 +38,11 @@ os.environ["ANTISPAM_MILTER_ADDRESS"] = system.get_host_address_from_environment os.environ["LMTP_ADDRESS"] = system.get_host_address_from_environment("LMTP", "imap:2525") os.environ["OUTCLEAN"] = os.environ["HOSTNAMES"].split(",")[0] try: - os.environ["OUTCLEAN_ADDRESS"] = system.resolve_hostname(os.environ["OUTCLEAN"]) + _to_lookup = os.environ["OUTCLEAN"] + # Ensure we lookup a FQDN: @see #1884 + if not _to_lookup.endswith('.'): + _to_lookup += '.' + os.environ["OUTCLEAN_ADDRESS"] = system.resolve_hostname(_to_lookup) except: os.environ["OUTCLEAN_ADDRESS"] = "10.10.10.10" From fa915d78624114201074b0bbe919cdb708cca568 Mon Sep 17 00:00:00 2001 From: Florent Daigniere Date: Sat, 24 Jul 2021 14:39:40 +0200 Subject: [PATCH 260/596] Fix 1294 ensure podop's socket is owned by postfix --- core/postfix/start.py | 3 ++- towncrier/newsfragments/1294.bugfix | 1 + 2 files changed, 3 insertions(+), 1 deletion(-) create mode 100644 towncrier/newsfragments/1294.bugfix diff --git a/core/postfix/start.py b/core/postfix/start.py index 701efec3..125404f6 100755 --- a/core/postfix/start.py +++ b/core/postfix/start.py @@ -8,12 +8,13 @@ import logging as log import sys from podop import run_server +from pwd import getpwnam from socrate import system, conf log.basicConfig(stream=sys.stderr, level=os.environ.get("LOG_LEVEL", "WARNING")) def start_podop(): - os.setuid(100) + os.setuid(getpwnam('postfix').pw_uid) url = "http://" + os.environ["ADMIN_ADDRESS"] + "/internal/postfix/" # TODO: Remove verbosity setting from Podop? run_server(0, "postfix", "/tmp/podop.socket", [ diff --git a/towncrier/newsfragments/1294.bugfix b/towncrier/newsfragments/1294.bugfix new file mode 100644 index 00000000..68bb7a8a --- /dev/null +++ b/towncrier/newsfragments/1294.bugfix @@ -0,0 +1 @@ +Ensure that the podop socket is always owned by the postfix user (wasn't the case when build using non-standard base images... typically for arm64) From 8235085848e1e8586a7455bf214bd240f06d549e Mon Sep 17 00:00:00 2001 From: networkException Date: Sat, 24 Jul 2021 15:25:59 +0200 Subject: [PATCH 261/596] Docs: Limit fail2ban matches to front container Previously fail2ban matched against all journal entries. This patch adds a tag to the logdriver and fail2ban filter documentation that limits the matches to entries from the front container --- docs/faq.rst | 3 +++ 1 file changed, 3 insertions(+) diff --git a/docs/faq.rst b/docs/faq.rst index 5d975532..f38fdca2 100644 --- a/docs/faq.rst +++ b/docs/faq.rst @@ -497,6 +497,8 @@ follow these steps: logging: driver: journald + options: + tag: mailu-front 2. Add the /etc/fail2ban/filter.d/bad-auth.conf @@ -506,6 +508,7 @@ follow these steps: [Definition] failregex = .* client login failed: .+ client:\ ignoreregex = + journalmatch = CONTAINER_TAG=mailu-front 3. Add the /etc/fail2ban/jail.d/bad-auth.conf From 8d9f3214cc5663dc29f7dcf3a03bc373a51d010b Mon Sep 17 00:00:00 2001 From: Florent Daigniere Date: Sat, 24 Jul 2021 15:45:25 +0200 Subject: [PATCH 262/596] Use threads in gunicorn rather than processes This ensures that we share the auth-cache... will enable memory savings and may improve performances when a higher number of cores is available "smarter default" --- core/admin/start.py | 3 ++- 1 file changed, 2 insertions(+), 1 deletion(-) diff --git a/core/admin/start.py b/core/admin/start.py index 2c925e01..0eff3bbe 100755 --- a/core/admin/start.py +++ b/core/admin/start.py @@ -19,7 +19,8 @@ if account is not None and domain is not None and password is not None: os.system("flask mailu admin %s %s '%s' --mode %s" % (account, domain, password, mode)) start_command="".join([ - "gunicorn -w 4 -b :80 ", + "gunicorn --threads ", str(os.cpu_count()), + " -b :80 ", "--access-logfile - " if (log.root.level<=log.INFO) else "", "--error-logfile - ", "--preload ", From ad1b036f2068f5fe70641e549df34cb0b9cc0b21 Mon Sep 17 00:00:00 2001 From: Alexander Graf Date: Sat, 24 Jul 2021 20:21:38 +0200 Subject: [PATCH 263/596] fix Email class --- core/admin/mailu/models.py | 4 ++-- 1 file changed, 2 insertions(+), 2 deletions(-) diff --git a/core/admin/mailu/models.py b/core/admin/mailu/models.py index b5ba29c0..fc9fd623 100644 --- a/core/admin/mailu/models.py +++ b/core/admin/mailu/models.py @@ -371,8 +371,8 @@ class Email(object): @classmethod def __declare_last__(cls): # gets called after mappings are completed - sqlalchemy.event.listen(User.localpart, 'set', cls._update_localpart, propagate=True) - sqlalchemy.event.listen(User.domain_name, 'set', cls._update_domain_name, propagate=True) + sqlalchemy.event.listen(cls.localpart, 'set', cls._update_localpart, propagate=True) + sqlalchemy.event.listen(cls.domain_name, 'set', cls._update_domain_name, propagate=True) def sendmail(self, subject, body): """ send an email to the address """ From c2c3030a2f80bf0ae85d186d30a47ca97b9ce81b Mon Sep 17 00:00:00 2001 From: Alexander Graf Date: Sat, 24 Jul 2021 20:54:36 +0200 Subject: [PATCH 264/596] rephrased comments --- core/admin/mailu/models.py | 4 ++-- 1 file changed, 2 insertions(+), 2 deletions(-) diff --git a/core/admin/mailu/models.py b/core/admin/mailu/models.py index a86e005f..43f7196a 100644 --- a/core/admin/mailu/models.py +++ b/core/admin/mailu/models.py @@ -508,8 +508,8 @@ class User(Base, Email): return cls._ctx # compile schemes - # - scrypt throws a warning if the native wheels aren't found - # - we can't leave plaintext schemes as they will be misidentified + # - skip scrypt (throws a warning if the native wheels aren't found) + # - skip plaintext schemes (will be misidentified) schemes = [ scheme for scheme in passlib.registry.list_crypt_handlers() if not (scheme == 'scrypt' or scheme.endswith('plaintext')) From 54b46a13c6bc730d5d0faa27d5c540edae9b24d4 Mon Sep 17 00:00:00 2001 From: Alexander Graf Date: Sun, 25 Jul 2021 15:51:13 +0200 Subject: [PATCH 265/596] save dkim key after creation --- core/admin/mailu/models.py | 2 +- core/admin/mailu/ui/views/domains.py | 2 ++ 2 files changed, 3 insertions(+), 1 deletion(-) diff --git a/core/admin/mailu/models.py b/core/admin/mailu/models.py index b5ba29c0..7a6eb308 100644 --- a/core/admin/mailu/models.py +++ b/core/admin/mailu/models.py @@ -272,7 +272,7 @@ class Domain(Base): return dkim.strip_key(dkim_key).decode('utf8') def generate_dkim_key(self): - """ generate and activate new DKIM key """ + """ generate new DKIM key """ self.dkim_key = dkim.gen_key() def has_email(self, localpart): diff --git a/core/admin/mailu/ui/views/domains.py b/core/admin/mailu/ui/views/domains.py index 719d3844..f394ce7d 100644 --- a/core/admin/mailu/ui/views/domains.py +++ b/core/admin/mailu/ui/views/domains.py @@ -74,6 +74,8 @@ def domain_details(domain_name): def domain_genkeys(domain_name): domain = models.Domain.query.get(domain_name) or flask.abort(404) domain.generate_dkim_key() + models.db.session.add(domain) + models.db.session.commit() return flask.redirect( flask.url_for(".domain_details", domain_name=domain_name)) From 6856c2c80f1291771a2faed5b72883e62028fcc6 Mon Sep 17 00:00:00 2001 From: Alexander Graf Date: Fri, 30 Jul 2021 22:26:20 +0200 Subject: [PATCH 266/596] treat localpart case insensitive again by lowercasing it where necessary --- core/admin/mailu/models.py | 21 +++++++++++---------- 1 file changed, 11 insertions(+), 10 deletions(-) diff --git a/core/admin/mailu/models.py b/core/admin/mailu/models.py index b5ba29c0..4fdf5afb 100644 --- a/core/admin/mailu/models.py +++ b/core/admin/mailu/models.py @@ -57,10 +57,9 @@ class IdnaEmail(db.TypeDecorator): def process_bind_param(self, value, dialect): """ encode unicode domain part of email address to punycode """ - localpart, domain_name = value.rsplit('@', 1) + localpart, domain_name = value.lower().rsplit('@', 1) if '@' in localpart: raise ValueError('email local part must not contain "@"') - domain_name = domain_name.lower() return f'{localpart}@{idna.encode(domain_name).decode("ascii")}' def process_result_value(self, value, dialect): @@ -277,6 +276,7 @@ class Domain(Base): def has_email(self, localpart): """ checks if localpart is configured for domain """ + localpart = localpart.lower() for email in chain(self.users, self.aliases): if email.localpart == localpart: return True @@ -355,8 +355,8 @@ class Email(object): @email.setter def email(self, value): """ setter for email - sets _email, localpart and domain_name at once """ - self.localpart, self.domain_name = value.rsplit('@', 1) - self._email = value + self._email = value.lower() + self.localpart, self.domain_name = self._email.rsplit('@', 1) @staticmethod def _update_localpart(target, value, *_): @@ -389,8 +389,7 @@ class Email(object): def resolve_domain(cls, email): """ resolves domain alternative to real domain """ localpart, domain_name = email.rsplit('@', 1) if '@' in email else (None, email) - alternative = Alternative.query.get(domain_name) - if alternative: + if alternative := Alternative.query.get(domain_name): domain_name = alternative.domain_name return (localpart, domain_name) @@ -401,12 +400,14 @@ class Email(object): localpart_stripped = None stripped_alias = None - if os.environ.get('RECIPIENT_DELIMITER') in localpart: - localpart_stripped = localpart.rsplit(os.environ.get('RECIPIENT_DELIMITER'), 1)[0] + delim = os.environ.get('RECIPIENT_DELIMITER') + if delim in localpart: + localpart_stripped = localpart.rsplit(delim, 1)[0] user = User.query.get(f'{localpart}@{domain_name}') if not user and localpart_stripped: user = User.query.get(f'{localpart_stripped}@{domain_name}') + if user: email = f'{localpart}@{domain_name}' @@ -416,15 +417,15 @@ class Email(object): destination.append(email) else: destination = [email] + return destination pure_alias = Alias.resolve(localpart, domain_name) - stripped_alias = Alias.resolve(localpart_stripped, domain_name) if pure_alias and not pure_alias.wildcard: return pure_alias.destination - if stripped_alias: + if stripped_alias := Alias.resolve(localpart_stripped, domain_name): return stripped_alias.destination if pure_alias: From 3471ebb2146c210212dc0a7a1d7ed62eb7242ebb Mon Sep 17 00:00:00 2001 From: Florent Daigniere Date: Mon, 2 Aug 2021 19:18:42 +0200 Subject: [PATCH 267/596] Allow specific users to send email from any address --- core/admin/mailu/configuration.py | 1 + core/admin/mailu/internal/views/postfix.py | 5 ++++- docs/configuration.rst | 2 ++ towncrier/newsfragments/1096.feature | 1 + 4 files changed, 8 insertions(+), 1 deletion(-) create mode 100644 towncrier/newsfragments/1096.feature diff --git a/core/admin/mailu/configuration.py b/core/admin/mailu/configuration.py index 3d1b4fb5..3dd874f3 100644 --- a/core/admin/mailu/configuration.py +++ b/core/admin/mailu/configuration.py @@ -32,6 +32,7 @@ DEFAULT_CONFIG = { 'DOMAIN': 'mailu.io', 'HOSTNAMES': 'mail.mailu.io,alternative.mailu.io,yetanother.mailu.io', 'POSTMASTER': 'postmaster', + 'WILDCARD_SENDERS': '', 'TLS_FLAVOR': 'cert', 'INBOUND_TLS_ENFORCE': False, 'AUTH_RATELIMIT': '10/minute;1000/hour', diff --git a/core/admin/mailu/internal/views/postfix.py b/core/admin/mailu/internal/views/postfix.py index c358c37f..d1b53856 100644 --- a/core/admin/mailu/internal/views/postfix.py +++ b/core/admin/mailu/internal/views/postfix.py @@ -133,10 +133,13 @@ def postfix_sender_map(sender): @internal.route("/postfix/sender/login/") def postfix_sender_login(sender): + has_wildcard_senders = bool(flask.current_app.config["WILDCARD_SENDERS"]) + wildcard_senders = flask.current_app.config["WILDCARD_SENDERS"].lower().split(',') if has_wildcard_senders else [] localpart, domain_name = models.Email.resolve_domain(sender) if localpart is None: - return flask.abort(404) + return flask.jsonify(",".join(wildcard_senders)) if has_wildcard_senders else flask.abort(404) destination = models.Email.resolve_destination(localpart, domain_name, True) + destination = [*destination, *wildcard_senders] if destination else [*wildcard_senders] return flask.jsonify(",".join(destination)) if destination else flask.abort(404) diff --git a/docs/configuration.rst b/docs/configuration.rst index 16ea23c3..34b14868 100644 --- a/docs/configuration.rst +++ b/docs/configuration.rst @@ -37,6 +37,8 @@ The ``POSTMASTER`` is the local part of the postmaster email address. It is recommended to setup a generic value and later configure a mail alias for that address. +The ``WILDCARD_SENDERS`` setting is a comma delimited list of user email addresses that are allowed to send emails from any address (spoofing the sender). + The ``AUTH_RATELIMIT`` holds a security setting for fighting attackers that try to guess user passwords. The value is the limit of failed authentication attempts that a single IP address can perform against IMAP, POP and SMTP authentication endpoints. diff --git a/towncrier/newsfragments/1096.feature b/towncrier/newsfragments/1096.feature new file mode 100644 index 00000000..f3abd3dc --- /dev/null +++ b/towncrier/newsfragments/1096.feature @@ -0,0 +1 @@ +Allow specific users to send emails from any address using the WILDCARD_SENDERS setting From f8362d04e4d46c44ab07beffb77cdd041af193c0 Mon Sep 17 00:00:00 2001 From: Florent Daigniere Date: Tue, 3 Aug 2021 13:44:56 +0200 Subject: [PATCH 268/596] Switch to openssl to workaround alpine #12763 --- core/admin/Dockerfile | 4 ++-- 1 file changed, 2 insertions(+), 2 deletions(-) diff --git a/core/admin/Dockerfile b/core/admin/Dockerfile index 3153bd9e..a0d3d996 100644 --- a/core/admin/Dockerfile +++ b/core/admin/Dockerfile @@ -24,9 +24,9 @@ RUN mkdir -p /app WORKDIR /app COPY requirements-prod.txt requirements.txt -RUN apk add --no-cache libressl curl postgresql-libs mariadb-connector-c \ +RUN apk add --no-cache openssl curl postgresql-libs mariadb-connector-c \ && apk add --no-cache --virtual build-dep \ - libressl-dev libffi-dev python3-dev build-base postgresql-dev mariadb-connector-c-dev \ + openssl-dev libffi-dev python3-dev build-base postgresql-dev mariadb-connector-c-dev \ && pip3 install -r requirements.txt \ && apk del --no-cache build-dep From defea3258ddd3faa15563630b7a10625b5ffa4a6 Mon Sep 17 00:00:00 2001 From: Florent Daigniere Date: Tue, 3 Aug 2021 13:58:54 +0200 Subject: [PATCH 269/596] update arm builds too --- core/admin/Dockerfile | 2 +- webmails/rainloop/Dockerfile | 2 +- webmails/roundcube/Dockerfile | 2 +- 3 files changed, 3 insertions(+), 3 deletions(-) diff --git a/core/admin/Dockerfile b/core/admin/Dockerfile index 0a3b8468..97cf1736 100644 --- a/core/admin/Dockerfile +++ b/core/admin/Dockerfile @@ -2,7 +2,7 @@ ARG DISTRO=alpine:3.14 ARG ARCH="" FROM ${ARCH}node:8 as assets -COPY --from=balenalib/rpi-alpine:3.10 /usr/bin/qemu-arm-static /usr/bin/qemu-arm-static +COPY --from=balenalib/rpi-alpine:3.14 /usr/bin/qemu-arm-static /usr/bin/qemu-arm-static COPY package.json ./ RUN npm install diff --git a/webmails/rainloop/Dockerfile b/webmails/rainloop/Dockerfile index 9c65f277..9814413d 100644 --- a/webmails/rainloop/Dockerfile +++ b/webmails/rainloop/Dockerfile @@ -3,7 +3,7 @@ ARG QEMU=other # NOTE: only add file if building for arm FROM ${ARCH}php:7.4-apache as build_arm -ONBUILD COPY --from=balenalib/rpi-alpine:3.10 /usr/bin/qemu-arm-static /usr/bin/qemu-arm-static +ONBUILD COPY --from=balenalib/rpi-alpine:3.14 /usr/bin/qemu-arm-static /usr/bin/qemu-arm-static FROM ${ARCH}php:7.4-apache as build_other diff --git a/webmails/roundcube/Dockerfile b/webmails/roundcube/Dockerfile index fae02ce0..4d3e36df 100644 --- a/webmails/roundcube/Dockerfile +++ b/webmails/roundcube/Dockerfile @@ -2,7 +2,7 @@ ARG ARCH="" ARG QEMU=other FROM ${ARCH}php:7.4-apache as build_arm -ONBUILD COPY --from=balenalib/rpi-alpine:3.10 /usr/bin/qemu-arm-static /usr/bin/qemu-arm-static +ONBUILD COPY --from=balenalib/rpi-alpine:3.14 /usr/bin/qemu-arm-static /usr/bin/qemu-arm-static FROM ${ARCH}php:7.4-apache as build_other From ccb3631622e535c902d22c7ced649cd374d1ac97 Mon Sep 17 00:00:00 2001 From: Florent Daigniere Date: Tue, 3 Aug 2021 14:01:44 +0200 Subject: [PATCH 270/596] still need pip3 --- docs/Dockerfile | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/docs/Dockerfile b/docs/Dockerfile index 11f66b49..29234b19 100644 --- a/docs/Dockerfile +++ b/docs/Dockerfile @@ -6,7 +6,7 @@ COPY requirements.txt /requirements.txt ARG version=master ENV VERSION=$version -RUN apk add --no-cache nginx curl python3 \ +RUN apk add --no-cache nginx curl python3 py3-pip \ && pip3 install -r /requirements.txt \ && mkdir /run/nginx From 2b63280f59658ca05fd5fb3870126556d6c29004 Mon Sep 17 00:00:00 2001 From: Florent Daigniere Date: Tue, 3 Aug 2021 14:16:14 +0200 Subject: [PATCH 271/596] doh --- docs/Dockerfile | 7 +++---- 1 file changed, 3 insertions(+), 4 deletions(-) diff --git a/docs/Dockerfile b/docs/Dockerfile index 29234b19..253c8420 100644 --- a/docs/Dockerfile +++ b/docs/Dockerfile @@ -1,4 +1,4 @@ -ARG DISTRO=alpine:3.14 +ARG DISTRO=python:3.8-alpine3.14 FROM $DISTRO COPY requirements.txt /requirements.txt @@ -6,9 +6,8 @@ COPY requirements.txt /requirements.txt ARG version=master ENV VERSION=$version -RUN apk add --no-cache nginx curl python3 py3-pip \ - && pip3 install -r /requirements.txt \ - && mkdir /run/nginx +RUN apk add --no-cache nginx curl \ + && pip3 install -r /requirements.txt COPY ./nginx.conf /etc/nginx/conf.d/default.conf COPY . /docs From 609e0f9f7c5b60e5db498d3b3817b41e65c116c5 Mon Sep 17 00:00:00 2001 From: Dimitri Huisman <52963853+Diman0@users.noreply.github.com> Date: Thu, 5 Aug 2021 00:12:37 +0200 Subject: [PATCH 272/596] Env vars are not shared between jobs --- .github/workflows/CI.yml | 84 ++++++++++++++++++++++++++++++++++++++++ 1 file changed, 84 insertions(+) diff --git a/.github/workflows/CI.yml b/.github/workflows/CI.yml index 55595838..19a445b4 100644 --- a/.github/workflows/CI.yml +++ b/.github/workflows/CI.yml @@ -91,6 +91,20 @@ jobs: shell: bash run: | echo "BRANCH=${GITHUB_REF#refs/heads/}" >> $GITHUB_ENV + - name: Derive MAILU_VERSION for branch testing + if: ${{ env.BRANCH == 'testing' }} + shell: bash + env: + COMMIT_MESSAGE: ${{ github.event.head_commit.message }} + run: | + echo "MAILU_VERSION=pr-${COMMIT_MESSAGE//[!0-9]/}" >> $GITHUB_ENV + - name: Derive MAILU_VERSION for other branches than testing + if: ${{ env.BRANCH != 'testing' }} + shell: bash + env: + MAILU_BRANCH: ${{ env.BRANCH }} + run: | + echo "MAILU_VERSION=${{ env.MAILU_BRANCH }}" >> $GITHUB_ENV - name: Create folder for storing images run: | sudo mkdir -p /images @@ -124,6 +138,20 @@ jobs: shell: bash run: | echo "BRANCH=${GITHUB_REF#refs/heads/}" >> $GITHUB_ENV + - name: Derive MAILU_VERSION for branch testing + if: ${{ env.BRANCH == 'testing' }} + shell: bash + env: + COMMIT_MESSAGE: ${{ github.event.head_commit.message }} + run: | + echo "MAILU_VERSION=pr-${COMMIT_MESSAGE//[!0-9]/}" >> $GITHUB_ENV + - name: Derive MAILU_VERSION for other branches than testing + if: ${{ env.BRANCH != 'testing' }} + shell: bash + env: + MAILU_BRANCH: ${{ env.BRANCH }} + run: | + echo "MAILU_VERSION=${{ env.MAILU_BRANCH }}" >> $GITHUB_ENV - name: Create folder for storing images run: | sudo mkdir -p /images @@ -157,6 +185,20 @@ jobs: shell: bash run: | echo "BRANCH=${GITHUB_REF#refs/heads/}" >> $GITHUB_ENV + - name: Derive MAILU_VERSION for branch testing + if: ${{ env.BRANCH == 'testing' }} + shell: bash + env: + COMMIT_MESSAGE: ${{ github.event.head_commit.message }} + run: | + echo "MAILU_VERSION=pr-${COMMIT_MESSAGE//[!0-9]/}" >> $GITHUB_ENV + - name: Derive MAILU_VERSION for other branches than testing + if: ${{ env.BRANCH != 'testing' }} + shell: bash + env: + MAILU_BRANCH: ${{ env.BRANCH }} + run: | + echo "MAILU_VERSION=${{ env.MAILU_BRANCH }}" >> $GITHUB_ENV - name: Create folder for storing images run: | sudo mkdir -p /images @@ -190,6 +232,20 @@ jobs: shell: bash run: | echo "BRANCH=${GITHUB_REF#refs/heads/}" >> $GITHUB_ENV + - name: Derive MAILU_VERSION for branch testing + if: ${{ env.BRANCH == 'testing' }} + shell: bash + env: + COMMIT_MESSAGE: ${{ github.event.head_commit.message }} + run: | + echo "MAILU_VERSION=pr-${COMMIT_MESSAGE//[!0-9]/}" >> $GITHUB_ENV + - name: Derive MAILU_VERSION for other branches than testing + if: ${{ env.BRANCH != 'testing' }} + shell: bash + env: + MAILU_BRANCH: ${{ env.BRANCH }} + run: | + echo "MAILU_VERSION=${{ env.MAILU_BRANCH }}" >> $GITHUB_ENV - name: Create folder for storing images run: | sudo mkdir -p /images @@ -223,6 +279,20 @@ jobs: shell: bash run: | echo "BRANCH=${GITHUB_REF#refs/heads/}" >> $GITHUB_ENV + - name: Derive MAILU_VERSION for branch testing + if: ${{ env.BRANCH == 'testing' }} + shell: bash + env: + COMMIT_MESSAGE: ${{ github.event.head_commit.message }} + run: | + echo "MAILU_VERSION=pr-${COMMIT_MESSAGE//[!0-9]/}" >> $GITHUB_ENV + - name: Derive MAILU_VERSION for other branches than testing + if: ${{ env.BRANCH != 'testing' }} + shell: bash + env: + MAILU_BRANCH: ${{ env.BRANCH }} + run: | + echo "MAILU_VERSION=${{ env.MAILU_BRANCH }}" >> $GITHUB_ENV - name: Create folder for storing images run: | sudo mkdir -p /images @@ -256,6 +326,20 @@ jobs: shell: bash run: | echo "BRANCH=${GITHUB_REF#refs/heads/}" >> $GITHUB_ENV + - name: Derive MAILU_VERSION for branch testing + if: ${{ env.BRANCH == 'testing' }} + shell: bash + env: + COMMIT_MESSAGE: ${{ github.event.head_commit.message }} + run: | + echo "MAILU_VERSION=pr-${COMMIT_MESSAGE//[!0-9]/}" >> $GITHUB_ENV + - name: Derive MAILU_VERSION for other branches than testing + if: ${{ env.BRANCH != 'testing' }} + shell: bash + env: + MAILU_BRANCH: ${{ env.BRANCH }} + run: | + echo "MAILU_VERSION=${{ env.MAILU_BRANCH }}" >> $GITHUB_ENV - name: Create folder for storing images run: | sudo mkdir -p /images From 98933f9478aab9f8d04cb0afba5eeac2c4dee9ef Mon Sep 17 00:00:00 2001 From: Erriez Date: Thu, 5 Aug 2021 19:37:06 +0200 Subject: [PATCH 273/596] Optimize docs/Dockerfile - Convert .rst to .html in temporary python:3.8-alpine3.14 build image - Remove all unused packages - Use nginx:1.21-alpine deployment image --- docs/Dockerfile | 30 +++++++++++++++++++----------- 1 file changed, 19 insertions(+), 11 deletions(-) diff --git a/docs/Dockerfile b/docs/Dockerfile index 70c9c3c4..289697da 100644 --- a/docs/Dockerfile +++ b/docs/Dockerfile @@ -1,20 +1,28 @@ -ARG DISTRO=alpine:3.8 -FROM $DISTRO - -COPY requirements.txt /requirements.txt +# Convert .rst files to .html in temporary build container +FROM python:3.8-alpine3.14 AS build ARG version=master ENV VERSION=$version -RUN apk add --no-cache nginx curl python3 \ - && pip3 install -r /requirements.txt \ - && mkdir /run/nginx - -COPY ./nginx.conf /etc/nginx/conf.d/default.conf +COPY requirements.txt /requirements.txt COPY . /docs -RUN mkdir -p /build/$VERSION \ - && sphinx-build -W /docs /build/$VERSION +RUN apk add --no-cache --virtual .build-deps \ + gcc musl-dev \ + && pip3 install -r /requirements.txt \ + && mkdir -p /build/$VERSION \ + && sphinx-build -W /docs /build/$VERSION \ + && apk del .build-deps + + +# Build nginx deployment image including generated html +FROM nginx:1.21-alpine + +ARG version=master +ENV VERSION=$version + +COPY ./nginx.conf /etc/nginx/conf.d/default.conf +COPY --from=build /build/$VERSION /build/$VERSION EXPOSE 80/tcp From 588904078edd04ed08db309ceb7dd11e751c37b9 Mon Sep 17 00:00:00 2001 From: Diman0 Date: Fri, 6 Aug 2021 16:27:07 +0200 Subject: [PATCH 274/596] Set default of AUTH_RATELIMIT_SUBNET to False. Increase default AUTH_RATELIMIT value. --- core/admin/mailu/configuration.py | 4 ++-- docs/configuration.rst | 2 +- setup/templates/steps/config.html | 2 +- towncrier/newsfragments/1867.feature | 1 + 4 files changed, 5 insertions(+), 4 deletions(-) create mode 100644 towncrier/newsfragments/1867.feature diff --git a/core/admin/mailu/configuration.py b/core/admin/mailu/configuration.py index 3d1b4fb5..d2d34d88 100644 --- a/core/admin/mailu/configuration.py +++ b/core/admin/mailu/configuration.py @@ -34,8 +34,8 @@ DEFAULT_CONFIG = { 'POSTMASTER': 'postmaster', 'TLS_FLAVOR': 'cert', 'INBOUND_TLS_ENFORCE': False, - 'AUTH_RATELIMIT': '10/minute;1000/hour', - 'AUTH_RATELIMIT_SUBNET': True, + 'AUTH_RATELIMIT': '1000/minute;10000/hour', + 'AUTH_RATELIMIT_SUBNET': False, 'DISABLE_STATISTICS': False, # Mail settings 'DMARC_RUA': None, diff --git a/docs/configuration.rst b/docs/configuration.rst index 16ea23c3..d7ebfc11 100644 --- a/docs/configuration.rst +++ b/docs/configuration.rst @@ -41,7 +41,7 @@ The ``AUTH_RATELIMIT`` holds a security setting for fighting attackers that try to guess user passwords. The value is the limit of failed authentication attempts that a single IP address can perform against IMAP, POP and SMTP authentication endpoints. -If ``AUTH_RATELIMIT_SUBNET`` is ``True`` (which is the default), the ``AUTH_RATELIMIT`` +If ``AUTH_RATELIMIT_SUBNET`` is ``True`` (default: False), the ``AUTH_RATELIMIT`` rules does also apply to auth requests coming from ``SUBNET``, especially for the webmail. If you disable this, ensure that the rate limit on the webmail is enforced in a different way (e.g. roundcube plug-in), otherwise an attacker can simply bypass the limit using webmail. diff --git a/setup/templates/steps/config.html b/setup/templates/steps/config.html index 3769a210..fee66933 100644 --- a/setup/templates/steps/config.html +++ b/setup/templates/steps/config.html @@ -51,7 +51,7 @@ Or in plain english: if receivers start to classify your mail as spam, this post

    / minute + value="10000" required > / minute

    diff --git a/towncrier/newsfragments/1867.feature b/towncrier/newsfragments/1867.feature new file mode 100644 index 00000000..fbd3a7d7 --- /dev/null +++ b/towncrier/newsfragments/1867.feature @@ -0,0 +1 @@ +Changed default value of AUTH_RATELIMIT_SUBNET to false. Increased default value of the rate limit in setup utility (AUTH_RATELIMIT) to a higher value. From 4cfa2dbc2addc1bab34f5402f1b52df2d89ec536 Mon Sep 17 00:00:00 2001 From: Diman0 Date: Fri, 6 Aug 2021 16:44:18 +0200 Subject: [PATCH 275/596] Increase width of rate limiting text box. --- setup/templates/steps/config.html | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/setup/templates/steps/config.html b/setup/templates/steps/config.html index fee66933..72b83915 100644 --- a/setup/templates/steps/config.html +++ b/setup/templates/steps/config.html @@ -50,7 +50,7 @@ Or in plain english: if receivers start to classify your mail as spam, this post
    -

    / minute

    From a7d99bdedd3decc7dc02609fc070337451302eb2 Mon Sep 17 00:00:00 2001 From: Diman0 Date: Fri, 6 Aug 2021 22:35:37 +0200 Subject: [PATCH 276/596] Update CHANGELOG.md and process towncrier newsfragments. --- CHANGELOG.md | 32 +++++++++++++++++++++++++--- towncrier/newsfragments/1660.bugfix | 1 - towncrier/newsfragments/1686.bugfix | 1 - towncrier/newsfragments/1720.bugfix | 2 -- towncrier/newsfragments/1783.misc | 1 - towncrier/newsfragments/1837.bugfix | 1 - towncrier/newsfragments/1841.feature | 1 - towncrier/newsfragments/1845.feature | 1 - towncrier/newsfragments/1857.doc | 1 - towncrier/newsfragments/1861.bugfix | 1 - towncrier/newsfragments/1867.feature | 1 - towncrier/newsfragments/1874.bugfix | 1 - towncrier/newsfragments/1880.feature | 1 - towncrier/newsfragments/191.bugfix | 1 - 14 files changed, 29 insertions(+), 17 deletions(-) delete mode 100644 towncrier/newsfragments/1660.bugfix delete mode 100644 towncrier/newsfragments/1686.bugfix delete mode 100644 towncrier/newsfragments/1720.bugfix delete mode 100644 towncrier/newsfragments/1783.misc delete mode 100644 towncrier/newsfragments/1837.bugfix delete mode 100644 towncrier/newsfragments/1841.feature delete mode 100644 towncrier/newsfragments/1845.feature delete mode 100644 towncrier/newsfragments/1857.doc delete mode 100644 towncrier/newsfragments/1861.bugfix delete mode 100644 towncrier/newsfragments/1867.feature delete mode 100644 towncrier/newsfragments/1874.bugfix delete mode 100644 towncrier/newsfragments/1880.feature delete mode 100644 towncrier/newsfragments/191.bugfix diff --git a/CHANGELOG.md b/CHANGELOG.md index 579f3e82..09b9f68f 100644 --- a/CHANGELOG.md +++ b/CHANGELOG.md @@ -4,18 +4,44 @@ Changelog Upgrade should run fine as long as you generate a new compose or stack configuration and upgrade your mailu.env. -Please note that the current 1.8 is what we call a "soft release": It’s there for everyone to see and use, but to limit possible user-impact of this very big release, it’s not yet the default in the setup-utility for new users. When upgrading, please treat it with some care, and be sure to always have backups! - There are some changes to the configuration overrides. Override files are now mounted read-only into the containers. The Dovecot and Postfix overrides are moved in their own sub-directory. If there are local override files, they will need to be moved from overrides/ to overrides/dovecot and overrides/postfix/. See https://mailu.io/1.8/faq.html#how-can-i-override-settings for all the mappings. +<<<<<<< HEAD Please note that the shipped image for PostgreSQL database is deprecated. We advise to switch to an external database server. +======= +One major change for the docker compose file is that the antispam needs a fixed hostname [#1837](https://github.com/Mailu/Mailu/issues/1837). +This is handled when you regenerate the docker-compose file. A fixed hostname is required to retain rspamd history. + +Please not that the shipped image for PostgreSQL database is deprecated. +We advise to switch to an external PostgreSQL database server. +>>>>>>> afaacf5a... Update CHANGELOG.md and process towncrier newsfragments. -v1.8.0 - 2020-09-28 +1.8.0 - 2021-08-06 +-------------------- + +- Features: Update version of roundcube webmail and carddav plugin. This is a security update. ([#1841](https://github.com/Mailu/Mailu/issues/1841)) +- Features: Update version of rainloop webmail to 1.16.0. This is a security update. ([#1845](https://github.com/Mailu/Mailu/issues/1845)) +- Features: Changed default value of AUTH_RATELIMIT_SUBNET to false. Increased default value of the rate limit in setup utility (AUTH_RATELIMIT) to a higher value. ([#1867](https://github.com/Mailu/Mailu/issues/1867)) +- Features: Update jquery used in setup. Set pinned versions in requirements.txt for setup. This is a security update. ([#1880](https://github.com/Mailu/Mailu/issues/1880)) +- Bugfixes: Replace PUBLIC_HOSTNAME and PUBLIC_IP in "Received" headers to ensure that no undue spam points are attributed ([#191](https://github.com/Mailu/Mailu/issues/191)) +- Bugfixes: Don't replace nested headers (typically in attached emails) ([#1660](https://github.com/Mailu/Mailu/issues/1660)) +- Bugfixes: Fix letsencrypt access to certbot for the mail-letsencrypt flavour ([#1686](https://github.com/Mailu/Mailu/issues/1686)) +- Bugfixes: Fix CVE-2020-25275 and CVE-2020-24386 by using alpine 3.13 for + dovecot which contains a fixed dovecot version. ([#1720](https://github.com/Mailu/Mailu/issues/1720)) +- Bugfixes: Antispam service now uses a static hostname. Rspamd history is only retained when the service has a fixed hostname. ([#1837](https://github.com/Mailu/Mailu/issues/1837)) +- Bugfixes: Fix a bug preventing colons from being used in passwords when using radicale/webdav. ([#1861](https://github.com/Mailu/Mailu/issues/1861)) +- Bugfixes: Remove dot in blueprint name to prevent critical flask startup error in setup. ([#1874](https://github.com/Mailu/Mailu/issues/1874)) +- Bugfixes: fix punycode encoding of domain names ([#1891](https://github.com/Mailu/Mailu/issues/1891)) +- Improved Documentation: Update fail2ban documentation to use systemd backend instead of filepath for journald ([#1857](https://github.com/Mailu/Mailu/issues/1857)) +- Misc: ([#1783](https://github.com/Mailu/Mailu/issues/1783)) + + +v1.8.0rc - 2020-09-28 -------------------- - Features: Add support for backward-forwarding using SRS ([#328](https://github.com/Mailu/Mailu/issues/328)) diff --git a/towncrier/newsfragments/1660.bugfix b/towncrier/newsfragments/1660.bugfix deleted file mode 100644 index a90fb099..00000000 --- a/towncrier/newsfragments/1660.bugfix +++ /dev/null @@ -1 +0,0 @@ -Don't replace nested headers (typically in attached emails) diff --git a/towncrier/newsfragments/1686.bugfix b/towncrier/newsfragments/1686.bugfix deleted file mode 100644 index 932d7d7c..00000000 --- a/towncrier/newsfragments/1686.bugfix +++ /dev/null @@ -1 +0,0 @@ -Fix letsencrypt access to certbot for the mail-letsencrypt flavour diff --git a/towncrier/newsfragments/1720.bugfix b/towncrier/newsfragments/1720.bugfix deleted file mode 100644 index 0bf2b8e6..00000000 --- a/towncrier/newsfragments/1720.bugfix +++ /dev/null @@ -1,2 +0,0 @@ -Fix CVE-2020-25275 and CVE-2020-24386 by using alpine 3.13 for -dovecot which contains a fixed dovecot version. diff --git a/towncrier/newsfragments/1783.misc b/towncrier/newsfragments/1783.misc deleted file mode 100644 index 2ee4c97f..00000000 --- a/towncrier/newsfragments/1783.misc +++ /dev/null @@ -1 +0,0 @@ -Switch from client side sessions (cookies) to server-side sessions (Redis). This simplies the security model a lot and allows for an easier recovery should a cookie ever land in the hands of an attacker. diff --git a/towncrier/newsfragments/1837.bugfix b/towncrier/newsfragments/1837.bugfix deleted file mode 100644 index dcabcc6b..00000000 --- a/towncrier/newsfragments/1837.bugfix +++ /dev/null @@ -1 +0,0 @@ -Antispam service now uses a static hostname. Rspamd history is only retained when the service has a fixed hostname. diff --git a/towncrier/newsfragments/1841.feature b/towncrier/newsfragments/1841.feature deleted file mode 100644 index c91f805f..00000000 --- a/towncrier/newsfragments/1841.feature +++ /dev/null @@ -1 +0,0 @@ -Update version of roundcube webmail and carddav plugin. This is a security update. \ No newline at end of file diff --git a/towncrier/newsfragments/1845.feature b/towncrier/newsfragments/1845.feature deleted file mode 100644 index afde9313..00000000 --- a/towncrier/newsfragments/1845.feature +++ /dev/null @@ -1 +0,0 @@ -Update version of rainloop webmail to 1.16.0. This is a security update. diff --git a/towncrier/newsfragments/1857.doc b/towncrier/newsfragments/1857.doc deleted file mode 100644 index 06cb91ab..00000000 --- a/towncrier/newsfragments/1857.doc +++ /dev/null @@ -1 +0,0 @@ -Update fail2ban documentation to use systemd backend instead of filepath for journald \ No newline at end of file diff --git a/towncrier/newsfragments/1861.bugfix b/towncrier/newsfragments/1861.bugfix deleted file mode 100644 index 1e28d1b6..00000000 --- a/towncrier/newsfragments/1861.bugfix +++ /dev/null @@ -1 +0,0 @@ -Fix a bug preventing colons from being used in passwords when using radicale/webdav. diff --git a/towncrier/newsfragments/1867.feature b/towncrier/newsfragments/1867.feature deleted file mode 100644 index fbd3a7d7..00000000 --- a/towncrier/newsfragments/1867.feature +++ /dev/null @@ -1 +0,0 @@ -Changed default value of AUTH_RATELIMIT_SUBNET to false. Increased default value of the rate limit in setup utility (AUTH_RATELIMIT) to a higher value. diff --git a/towncrier/newsfragments/1874.bugfix b/towncrier/newsfragments/1874.bugfix deleted file mode 100644 index a301835e..00000000 --- a/towncrier/newsfragments/1874.bugfix +++ /dev/null @@ -1 +0,0 @@ -Remove dot in blueprint name to prevent critical flask startup error in setup. diff --git a/towncrier/newsfragments/1880.feature b/towncrier/newsfragments/1880.feature deleted file mode 100644 index 212dc906..00000000 --- a/towncrier/newsfragments/1880.feature +++ /dev/null @@ -1 +0,0 @@ -Update jquery used in setup. Set pinned versions in requirements.txt for setup. This is a security update. diff --git a/towncrier/newsfragments/191.bugfix b/towncrier/newsfragments/191.bugfix deleted file mode 100644 index 185d3074..00000000 --- a/towncrier/newsfragments/191.bugfix +++ /dev/null @@ -1 +0,0 @@ -Replace PUBLIC_HOSTNAME and PUBLIC_IP in "Received" headers to ensure that no undue spam points are attributed From 4b89143362d9ef8cfb985c030047ce74642f7952 Mon Sep 17 00:00:00 2001 From: Diman0 Date: Fri, 6 Aug 2021 23:00:27 +0200 Subject: [PATCH 277/596] Update documentation config and release notes page. --- docs/conf.py | 2 +- docs/releases.rst | 65 +++++++++++++++++++++++++++++++++++++++++++++-- 2 files changed, 64 insertions(+), 3 deletions(-) diff --git a/docs/conf.py b/docs/conf.py index 8f174b64..db7008b3 100644 --- a/docs/conf.py +++ b/docs/conf.py @@ -36,7 +36,7 @@ html_context = { 'github_user': 'mailu', 'github_repo': 'mailu', 'github_version': version, - 'stable_version': '1.7', + 'stable_version': '1.8', 'versions': [ ('1.5', '/1.5/'), ('1.6', '/1.6/'), diff --git a/docs/releases.rst b/docs/releases.rst index 7a15d1fa..7473b033 100644 --- a/docs/releases.rst +++ b/docs/releases.rst @@ -1,8 +1,69 @@ Release notes ============= -Mailu 1.8 - 2020-10-02 ----------------------- +Mailu 1.8 - 2021-08-7 +--------------------- + +The full 1.8 release is finally ready. There have been some changes in the contributors team. Many people from the contributors team have stepped back due to changed priorities in their life. +We are very grateful for all their contributions and hope we will see them back again in the future. +This is the main reason why it took so long for 1.8 to be fully released. + +Fortunately more people have decided to join the project. Some very nice contributions have been made which will become part of the next 1.9 release. +We hope that future Mailu releases will be released more quickly now we have more active contributors again. + +For a list of all changes refer to `CHANGELOG.md` in the root folder of the Mailu github project. Please read the 'Override location changes' section further on this page. It contains important information for the people who use the overrides folder. + +New Functionality & Improvements +```````````````````````````````` + +Here’s a short summary of new features: + +- Roundcube and Rainloop have been updated. +- All dependencies have been updated to the latest security update. +- Fail2ban documentation has been improved. +- Switch from client side (cookie) sessions to server side sessions. +- Full-text-search is back after having been disabled for a while due to nasty bugs. It can still be disabled via the mailu.env file. +- Tons of documentation improvements, especially geared towards new users. +- (Experimental) support for different architectures, such as ARM. +- Improvements around webmails, such as CardDAV, GPG and a new skin for an updated roundcube, and support for MySQL for it. Updated Rainloop, too. +- Improvements around relaying, such as AUTH LOGIN and non-standard port support. +- Update to alpine:3.14 as baseimage for most containers. +- Setup warns users about compose-IPv6 deployments which have caused open relays in the past. +- Improved handling of upper-vs-lowercase aliases and user-addresses. +- Improved rate-limiting system. +- Support for SRS. +- Japanese localisation is now available. + + +Upgrading +````````` + +Upgrade should run fine as long as you generate a new compose or stack +configuration and upgrade your mailu.env. + +Please not that the shipped image for PostgreSQL database is deprecated. +The shipped image for PostgreSQL is not maintained anymore from release 1.8. +We recommend switching to an external PostgreSQL image as soon as possible. + +Override location changes +^^^^^^^^^^^^^^^^^^^^^^^^^ + +If you have regenerated the Docker compose and environment files, there are some changes to the configuration overrides. +Override files are now mounted read-only into the containers. The Dovecot and Postfix overrides are moved in their own sub-directory. If there are local override files, they will need to be moved from ``overrides/`` to ``overrides/dovecot`` and ``overrides/postfix/``. + +Update your DNS SPF Records +^^^^^^^^^^^^^^^^^^^^^^^^^^^ + +It has become known that the SPF DNS records generated by the admin interface are not completely standard compliant anymore. Please check the DNS records for your domains and compare them to what the new admin-interface instructs you to use. In most cases, this should be a simple copy-paste operation for you …. + +Fixed hostname for antispam service +^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ + +For history to be retained in Rspamd, the antispam container requires a static hostname. When you re-generate your docker-compose.yml file (or helm-chart), this will be covered. + + +Mailu 1.8rc - 2020-10-02 +------------------------ Release 1.8 has come a long way again. Due to corona the project slowed down to a crawl. Fortunately new contributors have joined the team what enabled us to still release Mailu 1.8 this year. From 6581f8f087d62b841e9939ea3cf3814ea9e9a518 Mon Sep 17 00:00:00 2001 From: Dimitri Huisman <52963853+Diman0@users.noreply.github.com> Date: Fri, 6 Aug 2021 23:17:41 +0200 Subject: [PATCH 278/596] Resolve merge conflict --- CHANGELOG.md | 12 ++++-------- 1 file changed, 4 insertions(+), 8 deletions(-) diff --git a/CHANGELOG.md b/CHANGELOG.md index 09b9f68f..3ad0061b 100644 --- a/CHANGELOG.md +++ b/CHANGELOG.md @@ -9,18 +9,14 @@ The Dovecot and Postfix overrides are moved in their own sub-directory. If there are local override files, they will need to be moved from overrides/ to overrides/dovecot and overrides/postfix/. See https://mailu.io/1.8/faq.html#how-can-i-override-settings for all the mappings. -<<<<<<< HEAD -Please note that the shipped image for PostgreSQL database is deprecated. -We advise to switch to an external database server. -======= -One major change for the docker compose file is that the antispam needs a fixed hostname [#1837](https://github.com/Mailu/Mailu/issues/1837). -This is handled when you regenerate the docker-compose file. A fixed hostname is required to retain rspamd history. +One major change for the docker compose file is that the antispam container needs a fixed hostname [#1837](https://github.com/Mailu/Mailu/issues/1837). +This is handled when you regenerate the docker-compose file. A fixed hostname is required to retain rspamd history. +This is also handled in the helm-chart repo. Please not that the shipped image for PostgreSQL database is deprecated. We advise to switch to an external PostgreSQL database server. ->>>>>>> afaacf5a... Update CHANGELOG.md and process towncrier newsfragments. - + 1.8.0 - 2021-08-06 -------------------- From f0997ed0fd5b3be5e8a6964bfd3197ced1aa5ef9 Mon Sep 17 00:00:00 2001 From: Diman0 Date: Sat, 7 Aug 2021 09:12:43 +0200 Subject: [PATCH 279/596] Improved changelog entry --- CHANGELOG.md | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/CHANGELOG.md b/CHANGELOG.md index 3ad0061b..0a128163 100644 --- a/CHANGELOG.md +++ b/CHANGELOG.md @@ -27,7 +27,7 @@ We advise to switch to an external PostgreSQL database server. - Bugfixes: Replace PUBLIC_HOSTNAME and PUBLIC_IP in "Received" headers to ensure that no undue spam points are attributed ([#191](https://github.com/Mailu/Mailu/issues/191)) - Bugfixes: Don't replace nested headers (typically in attached emails) ([#1660](https://github.com/Mailu/Mailu/issues/1660)) - Bugfixes: Fix letsencrypt access to certbot for the mail-letsencrypt flavour ([#1686](https://github.com/Mailu/Mailu/issues/1686)) -- Bugfixes: Fix CVE-2020-25275 and CVE-2020-24386 by using alpine 3.13 for +- Bugfixes: Fix CVE-2020-25275 and CVE-2020-24386 by upgrading alpine for dovecot which contains a fixed dovecot version. ([#1720](https://github.com/Mailu/Mailu/issues/1720)) - Bugfixes: Antispam service now uses a static hostname. Rspamd history is only retained when the service has a fixed hostname. ([#1837](https://github.com/Mailu/Mailu/issues/1837)) - Bugfixes: Fix a bug preventing colons from being used in passwords when using radicale/webdav. ([#1861](https://github.com/Mailu/Mailu/issues/1861)) From 21e7a338e75437afd59ae4185c87f59d18f8f4e9 Mon Sep 17 00:00:00 2001 From: Diman0 Date: Sat, 7 Aug 2021 09:14:09 +0200 Subject: [PATCH 280/596] Fixed typing error. --- CHANGELOG.md | 2 +- docs/releases.rst | 2 +- 2 files changed, 2 insertions(+), 2 deletions(-) diff --git a/CHANGELOG.md b/CHANGELOG.md index 0a128163..82f04acc 100644 --- a/CHANGELOG.md +++ b/CHANGELOG.md @@ -13,7 +13,7 @@ One major change for the docker compose file is that the antispam container need This is handled when you regenerate the docker-compose file. A fixed hostname is required to retain rspamd history. This is also handled in the helm-chart repo. -Please not that the shipped image for PostgreSQL database is deprecated. +Please note that the shipped image for PostgreSQL database is deprecated. We advise to switch to an external PostgreSQL database server. diff --git a/docs/releases.rst b/docs/releases.rst index 7473b033..3ae25f48 100644 --- a/docs/releases.rst +++ b/docs/releases.rst @@ -41,7 +41,7 @@ Upgrading Upgrade should run fine as long as you generate a new compose or stack configuration and upgrade your mailu.env. -Please not that the shipped image for PostgreSQL database is deprecated. +Please note that the shipped image for PostgreSQL database is deprecated. The shipped image for PostgreSQL is not maintained anymore from release 1.8. We recommend switching to an external PostgreSQL image as soon as possible. From 14a18715111e4d21fd9cac0c7e18a293771f3fd0 Mon Sep 17 00:00:00 2001 From: Diman0 Date: Sat, 7 Aug 2021 09:25:40 +0200 Subject: [PATCH 281/596] enhanced security changelog entry and added recommendation to recreate secret_key --- CHANGELOG.md | 11 ++++++++++- docs/releases.rst | 14 +++++++++++++- 2 files changed, 23 insertions(+), 2 deletions(-) diff --git a/CHANGELOG.md b/CHANGELOG.md index 82f04acc..da945c72 100644 --- a/CHANGELOG.md +++ b/CHANGELOG.md @@ -13,6 +13,15 @@ One major change for the docker compose file is that the antispam container need This is handled when you regenerate the docker-compose file. A fixed hostname is required to retain rspamd history. This is also handled in the helm-chart repo. +Improvements have been made to protect again session-fixation attacks. +To be fully protected, it is required to change your SECRET_KEY in Mailu.env after upgrading. +A new SECRET_KEY is generated when you recreate your docker-compose.yml & mailu.env file via setup.mailu.io. + +The SECRET_KEY is an uppercase alphanumeric string of length 16. You can manually create such a string via +```cat /dev/urandom | tr -dc 'A-Z0-9' | fold -w ${1:-16} | head -n 1``` + +After changing mailu.env, it is required to recreate all containers for the changes to be propagated. + Please note that the shipped image for PostgreSQL database is deprecated. We advise to switch to an external PostgreSQL database server. @@ -34,7 +43,7 @@ We advise to switch to an external PostgreSQL database server. - Bugfixes: Remove dot in blueprint name to prevent critical flask startup error in setup. ([#1874](https://github.com/Mailu/Mailu/issues/1874)) - Bugfixes: fix punycode encoding of domain names ([#1891](https://github.com/Mailu/Mailu/issues/1891)) - Improved Documentation: Update fail2ban documentation to use systemd backend instead of filepath for journald ([#1857](https://github.com/Mailu/Mailu/issues/1857)) -- Misc: ([#1783](https://github.com/Mailu/Mailu/issues/1783)) +- Misc: Switch from client side (cookie) sessions to server side sessions and protect against session-fixation attacks. We recommend that you change your SECRET_KEY after upgrading. ([#1783](https://github.com/Mailu/Mailu/issues/1783)) v1.8.0rc - 2020-09-28 diff --git a/docs/releases.rst b/docs/releases.rst index 3ae25f48..6c672538 100644 --- a/docs/releases.rst +++ b/docs/releases.rst @@ -21,7 +21,7 @@ Here’s a short summary of new features: - Roundcube and Rainloop have been updated. - All dependencies have been updated to the latest security update. - Fail2ban documentation has been improved. -- Switch from client side (cookie) sessions to server side sessions. +- Switch from client side (cookie) sessions to server side sessions and protect against session-fixation attacks. We recommend that you change your SECRET_KEY after upgrading. - Full-text-search is back after having been disabled for a while due to nasty bugs. It can still be disabled via the mailu.env file. - Tons of documentation improvements, especially geared towards new users. - (Experimental) support for different architectures, such as ARM. @@ -51,6 +51,18 @@ Override location changes If you have regenerated the Docker compose and environment files, there are some changes to the configuration overrides. Override files are now mounted read-only into the containers. The Dovecot and Postfix overrides are moved in their own sub-directory. If there are local override files, they will need to be moved from ``overrides/`` to ``overrides/dovecot`` and ``overrides/postfix/``. +Recreate SECRET_KEY after upgrading +^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ + +Improvements have been made to protect again session-fixation attacks. +To be fully protected, it is required to change your SECRET_KEY in Mailu.env after upgrading. +A new SECRET_KEY is generated when you recreate your docker-compose.yml & mailu.env file via setup.mailu.io. + +The SECRET_KEY is an uppercase alphanumeric string of length 16. You can manually create such a string via +```cat /dev/urandom | tr -dc 'A-Z0-9' | fold -w ${1:-16} | head -n 1``` + +After changing mailu.env, it is required to recreate all containers for the changes to be propagated. + Update your DNS SPF Records ^^^^^^^^^^^^^^^^^^^^^^^^^^^ From 3157fc3623424f7af991f2de86981c2aeced4cca Mon Sep 17 00:00:00 2001 From: Diman0 Date: Sat, 7 Aug 2021 09:27:47 +0200 Subject: [PATCH 282/596] Give docker containers in each test one more minute for starting. --- .github/workflows/CI.yml | 12 ++++++------ 1 file changed, 6 insertions(+), 6 deletions(-) diff --git a/.github/workflows/CI.yml b/.github/workflows/CI.yml index 19a445b4..e2a535dd 100644 --- a/.github/workflows/CI.yml +++ b/.github/workflows/CI.yml @@ -121,7 +121,7 @@ jobs: - name: Copy all certs run: sudo -- sh -c 'mkdir -p /mailu && cp -r tests/certs /mailu && chmod 600 /mailu/certs/*' - name: Test core suite - run: python tests/compose/test.py core 1 + run: python tests/compose/test.py core 2 env: MAILU_VERSION: ${{ env.MAILU_VERSION }} TRAVIS_BRANCH: ${{ env.BRANCH }} @@ -168,7 +168,7 @@ jobs: - name: Copy all certs run: sudo -- sh -c 'mkdir -p /mailu && cp -r tests/certs /mailu && chmod 600 /mailu/certs/*' - name: Test fetch - run: python tests/compose/test.py fetchmail 1 + run: python tests/compose/test.py fetchmail 2 env: MAILU_VERSION: ${{ env.MAILU_VERSION }} TRAVIS_BRANCH: ${{ env.BRANCH }} @@ -215,7 +215,7 @@ jobs: - name: Copy all certs run: sudo -- sh -c 'mkdir -p /mailu && cp -r tests/certs /mailu && chmod 600 /mailu/certs/*' - name: Test clamvav - run: python tests/compose/test.py filters 2 + run: python tests/compose/test.py filters 3 env: MAILU_VERSION: ${{ env.MAILU_VERSION }} TRAVIS_BRANCH: ${{ env.BRANCH }} @@ -262,7 +262,7 @@ jobs: - name: Copy all certs run: sudo -- sh -c 'mkdir -p /mailu && cp -r tests/certs /mailu && chmod 600 /mailu/certs/*' - name: Test rainloop - run: python tests/compose/test.py rainloop 1 + run: python tests/compose/test.py rainloop 2 env: MAILU_VERSION: ${{ env.MAILU_VERSION }} TRAVIS_BRANCH: ${{ env.BRANCH }} @@ -309,7 +309,7 @@ jobs: - name: Copy all certs run: sudo -- sh -c 'mkdir -p /mailu && cp -r tests/certs /mailu && chmod 600 /mailu/certs/*' - name: Test roundcube - run: python tests/compose/test.py roundcube 1 + run: python tests/compose/test.py roundcube 2 env: MAILU_VERSION: ${{ env.MAILU_VERSION }} TRAVIS_BRANCH: ${{ env.BRANCH }} @@ -356,7 +356,7 @@ jobs: - name: Copy all certs run: sudo -- sh -c 'mkdir -p /mailu && cp -r tests/certs /mailu && chmod 600 /mailu/certs/*' - name: Test webdav - run: python tests/compose/test.py webdav 1 + run: python tests/compose/test.py webdav 2 env: MAILU_VERSION: ${{ env.MAILU_VERSION }} TRAVIS_BRANCH: ${{ env.BRANCH }} From 1438253a069da3b10831ef89dc119177f16f5216 Mon Sep 17 00:00:00 2001 From: Florent Daigniere Date: Sun, 8 Aug 2021 09:21:14 +0200 Subject: [PATCH 283/596] Ratelimit outgoing emails per user --- core/admin/mailu/configuration.py | 1 + core/admin/mailu/internal/views/postfix.py | 10 ++++++++-- core/admin/mailu/models.py | 8 +++++++- core/admin/mailu/ui/templates/user/list.html | 5 ++++- core/postfix/conf/main.cf | 1 + core/postfix/start.py | 3 ++- setup/flavors/compose/mailu.env | 5 +++++ setup/templates/steps/config.html | 7 +++++++ towncrier/newsfragments/1031.feature | 1 + 9 files changed, 36 insertions(+), 5 deletions(-) create mode 100644 towncrier/newsfragments/1031.feature diff --git a/core/admin/mailu/configuration.py b/core/admin/mailu/configuration.py index d2d34d88..50733d52 100644 --- a/core/admin/mailu/configuration.py +++ b/core/admin/mailu/configuration.py @@ -46,6 +46,7 @@ DEFAULT_CONFIG = { 'DKIM_SELECTOR': 'dkim', 'DKIM_PATH': '/dkim/{domain}.{selector}.key', 'DEFAULT_QUOTA': 1000000000, + 'MESSAGE_RATELIMIT': '100/hour', # Web settings 'SITENAME': 'Mailu', 'WEBSITE': 'https://mailu.io', diff --git a/core/admin/mailu/internal/views/postfix.py b/core/admin/mailu/internal/views/postfix.py index c358c37f..06918c61 100644 --- a/core/admin/mailu/internal/views/postfix.py +++ b/core/admin/mailu/internal/views/postfix.py @@ -1,5 +1,6 @@ -from mailu import models +from mailu import models, utils from mailu.internal import internal +from flask import current_app as app import flask import idna @@ -31,7 +32,6 @@ def postfix_alias_map(alias): destination = models.Email.resolve_destination(localpart, domain_name) return flask.jsonify(",".join(destination)) if destination else flask.abort(404) - @internal.route("/postfix/transport/") def postfix_transport(email): if email == '*' or re.match("(^|.*@)\[.*\]$", email): @@ -139,6 +139,12 @@ def postfix_sender_login(sender): destination = models.Email.resolve_destination(localpart, domain_name, True) return flask.jsonify(",".join(destination)) if destination else flask.abort(404) +@internal.route("/postfix/sender/rate/") +def postfix_sender_rate(sender): + """ Rate limit outbound emails per sender login + """ + user = models.User.get(sender) or flask.abort(404) + return flask.abort(404) if user.sender_limiter.hit() else flask.jsonify("REJECT") @internal.route("/postfix/sender/access/") def postfix_sender_access(sender): diff --git a/core/admin/mailu/models.py b/core/admin/mailu/models.py index 3a299786..5760c27f 100644 --- a/core/admin/mailu/models.py +++ b/core/admin/mailu/models.py @@ -27,7 +27,7 @@ from sqlalchemy.ext.hybrid import hybrid_property from sqlalchemy.inspection import inspect from werkzeug.utils import cached_property -from mailu import dkim +from mailu import dkim, utils db = flask_sqlalchemy.SQLAlchemy() @@ -501,6 +501,12 @@ class User(Base, Email): self.reply_enddate > now ) + @property + def sender_limiter(self): + return utils.limiter.get_limiter( + app.config["MESSAGE_RATELIMIT"], "sender", self.email + ) + @classmethod def get_password_context(cls): """ create password context for hashing and verification diff --git a/core/admin/mailu/ui/templates/user/list.html b/core/admin/mailu/ui/templates/user/list.html index 2aff662f..746afd45 100644 --- a/core/admin/mailu/ui/templates/user/list.html +++ b/core/admin/mailu/ui/templates/user/list.html @@ -19,7 +19,8 @@ {% trans %}User settings{% endtrans %} {% trans %}Email{% endtrans %} {% trans %}Features{% endtrans %} - {% trans %}Quota{% endtrans %} + {% trans %}Storage Quota{% endtrans %} + {% trans %}Sending Quota{% endtrans %} {% trans %}Comment{% endtrans %} {% trans %}Created{% endtrans %} {% trans %}Last edit{% endtrans %} @@ -41,6 +42,8 @@ {% if user.enable_pop %}pop3{% endif %} {{ user.quota_bytes_used | filesizeformat }} / {{ (user.quota_bytes | filesizeformat) if user.quota_bytes else '∞' }} + {% set limiter = user.sender_limiter %} + {{ limiter.get_window_stats()[1] }} / {{ limiter.limit }} {{ user.comment or '-' }} {{ user.created_at }} {{ user.updated_at or '' }} diff --git a/core/postfix/conf/main.cf b/core/postfix/conf/main.cf index 8f35f609..6f5a20b8 100644 --- a/core/postfix/conf/main.cf +++ b/core/postfix/conf/main.cf @@ -100,6 +100,7 @@ smtpd_sender_login_maps = ${podop}senderlogin smtpd_helo_required = yes smtpd_client_restrictions = + check_sasl_access ${podop}senderrate, permit_mynetworks, check_sender_access ${podop}senderaccess, reject_non_fqdn_sender, diff --git a/core/postfix/start.py b/core/postfix/start.py index e0c781b7..139616b2 100755 --- a/core/postfix/start.py +++ b/core/postfix/start.py @@ -25,7 +25,8 @@ def start_podop(): ("recipientmap", "url", url + "recipient/map/§"), ("sendermap", "url", url + "sender/map/§"), ("senderaccess", "url", url + "sender/access/§"), - ("senderlogin", "url", url + "sender/login/§") + ("senderlogin", "url", url + "sender/login/§"), + ("senderrate", "url", url + "sender/rate/§") ]) def is_valid_postconf_line(line): diff --git a/setup/flavors/compose/mailu.env b/setup/flavors/compose/mailu.env index d45f5517..52f4ee04 100644 --- a/setup/flavors/compose/mailu.env +++ b/setup/flavors/compose/mailu.env @@ -62,6 +62,11 @@ ANTIVIRUS={{ antivirus_enabled or 'none' }} # Max attachment size will be 33% smaller MESSAGE_SIZE_LIMIT={{ message_size_limit or '50000000' }} +# Message rate limit (per user) +{% if message_ratelimit_pd > '0' %} +MESSAGE_RATELIMIT={{ message_ratelimit_pd }}/day +{% endif %} + # Networks granted relay permissions # Use this with care, all hosts in this networks will be able to send mail without authentication! RELAYNETS= diff --git a/setup/templates/steps/config.html b/setup/templates/steps/config.html index 72b83915..87410fca 100644 --- a/setup/templates/steps/config.html +++ b/setup/templates/steps/config.html @@ -55,6 +55,13 @@ Or in plain english: if receivers start to classify your mail as spam, this post

    +
    + + +

    / day +

    +
    +
    +
    + +
    + +
    +
    +
    + {% block main_action %} + {% endblock %} +
    +

    + {% block title %}{% endblock %} + {% block subtitle %}{% endblock %} +

    +
    + +
    + {{ utils.flashed_messages(container=False) }} + {% block content %}{% endblock %} +
    +
    + +
    + + + + diff --git a/core/admin/mailu/sso/templates/form_sso.html b/core/admin/mailu/sso/templates/form_sso.html new file mode 100644 index 00000000..fcabad41 --- /dev/null +++ b/core/admin/mailu/sso/templates/form_sso.html @@ -0,0 +1,7 @@ +{% extends "base_sso.html" %} + +{% block content %} +{% call macros.box() %} +{{ macros.form(form) }} +{% endcall %} +{% endblock %} diff --git a/core/admin/mailu/sso/templates/login.html b/core/admin/mailu/sso/templates/login.html new file mode 100644 index 00000000..851e6643 --- /dev/null +++ b/core/admin/mailu/sso/templates/login.html @@ -0,0 +1,9 @@ +{% extends "form_sso.html" %} + +{% block title %} +{% trans %}Sign in{% endtrans %} +{% endblock %} + +{% block subtitle %} +{% trans %}to access IF statement for switch text for loggin in what the administration tools{% endtrans %} +{% endblock %} diff --git a/core/admin/mailu/sso/views/__init__.py b/core/admin/mailu/sso/views/__init__.py new file mode 100644 index 00000000..38efde4c --- /dev/null +++ b/core/admin/mailu/sso/views/__init__.py @@ -0,0 +1,3 @@ +__all__ = [ + 'base', 'hello' +] diff --git a/core/admin/mailu/sso/views/base.py b/core/admin/mailu/sso/views/base.py new file mode 100644 index 00000000..dd6f60c9 --- /dev/null +++ b/core/admin/mailu/sso/views/base.py @@ -0,0 +1,30 @@ +from mailu import models +from mailu.sso import sso, forms + +from flask import current_app as app +import flask +import flask_login + +@sso.route('/login', methods=['GET', 'POST']) +def login(): + form = forms.LoginForm() + if form.validate_on_submit(): + user = models.User.login(form.email.data, form.pw.data) + if user: + flask.session.regenerate() + flask_login.login_user(user) + endpoint = flask.request.args.get('next', 'ui.index') + return flask.redirect(flask.url_for(endpoint) + or flask.url_for('ui.index')) + else: + flask.flash('Wrong e-mail or password', 'error') + return flask.render_template('login.html', form=form) + +""" +@ui.route('/logout', methods=['GET']) +@access.authenticated +def logout(): + flask_login.logout_user() + flask.session.destroy() + return flask.redirect(flask.url_for('.index')) +""" \ No newline at end of file diff --git a/core/admin/mailu/sso/views/hello.py b/core/admin/mailu/sso/views/hello.py new file mode 100644 index 00000000..2e9b5e35 --- /dev/null +++ b/core/admin/mailu/sso/views/hello.py @@ -0,0 +1,6 @@ +from mailu.sso import sso +from flask import current_app as app + +@sso.route("/") +def hello_world(): + return "

    Hello, World!

    " diff --git a/core/admin/mailu/ui/forms.py b/core/admin/mailu/ui/forms.py index 32bb31ab..dff7008e 100644 --- a/core/admin/mailu/ui/forms.py +++ b/core/admin/mailu/ui/forms.py @@ -44,14 +44,14 @@ class MultipleEmailAddressesVerify(object): class ConfirmationForm(flask_wtf.FlaskForm): submit = fields.SubmitField(_('Confirm')) - +""" class LoginForm(flask_wtf.FlaskForm): class Meta: csrf = False email = fields.StringField(_('E-mail'), [validators.Email()]) pw = fields.PasswordField(_('Password'), [validators.DataRequired()]) submit = fields.SubmitField(_('Sign in')) - +""" class DomainForm(flask_wtf.FlaskForm): name = fields.StringField(_('Domain name'), [validators.DataRequired()]) diff --git a/core/admin/mailu/ui/templates/login.html b/core/admin/mailu/ui/templates/login.html deleted file mode 100644 index 26c47c08..00000000 --- a/core/admin/mailu/ui/templates/login.html +++ /dev/null @@ -1,9 +0,0 @@ -{% extends "form.html" %} - -{% block title %} -{% trans %}Sign in{% endtrans %} -{% endblock %} - -{% block subtitle %} -{% trans %}to access the administration tools{% endtrans %} -{% endblock %} diff --git a/core/admin/mailu/ui/views/base.py b/core/admin/mailu/ui/views/base.py index eb5490bc..fc9daba6 100644 --- a/core/admin/mailu/ui/views/base.py +++ b/core/admin/mailu/ui/views/base.py @@ -12,6 +12,7 @@ def index(): return flask.redirect(flask.url_for('.user_settings')) +""" @ui.route('/login', methods=['GET', 'POST']) def login(): form = forms.LoginForm() @@ -26,7 +27,7 @@ def login(): else: flask.flash('Wrong e-mail or password', 'error') return flask.render_template('login.html', form=form) - +""" @ui.route('/logout', methods=['GET']) @access.authenticated diff --git a/core/admin/mailu/utils.py b/core/admin/mailu/utils.py index 02150754..c30f259e 100644 --- a/core/admin/mailu/utils.py +++ b/core/admin/mailu/utils.py @@ -28,13 +28,13 @@ from werkzeug.contrib import fixers # Login configuration login = flask_login.LoginManager() -login.login_view = "ui.login" +login.login_view = "sso.login" @login.unauthorized_handler def handle_needs_login(): """ redirect unauthorized requests to login page """ return flask.redirect( - flask.url_for('ui.login', next=flask.request.endpoint) + flask.url_for('sso.login', next=flask.request.endpoint) ) # Rate limiter From 8868aec0dcd02101ea09396c5d064b046b75ea44 Mon Sep 17 00:00:00 2001 From: Diman0 Date: Thu, 2 Sep 2021 17:08:50 +0200 Subject: [PATCH 394/596] Merge master. Make sso login working for admin. --- core/admin/mailu/configuration.py | 1 + core/admin/mailu/sso/forms.py | 2 + core/admin/mailu/sso/templates/base_sso.html | 61 +++++++++++-------- core/admin/mailu/sso/templates/form_sso.html | 2 +- core/admin/mailu/sso/templates/login.html | 6 +- .../mailu/sso/templates/sidebar_sso.html | 60 ++++++++++++++++++ core/admin/mailu/sso/views/base.py | 15 ++--- core/admin/mailu/ui/templates/sidebar.html | 2 +- core/nginx/conf/nginx.conf | 8 ++- 9 files changed, 115 insertions(+), 42 deletions(-) create mode 100644 core/admin/mailu/sso/templates/sidebar_sso.html diff --git a/core/admin/mailu/configuration.py b/core/admin/mailu/configuration.py index 7cd3a56b..025a173c 100644 --- a/core/admin/mailu/configuration.py +++ b/core/admin/mailu/configuration.py @@ -51,6 +51,7 @@ DEFAULT_CONFIG = { # Web settings 'SITENAME': 'Mailu', 'WEBSITE': 'https://mailu.io', + 'ADMIN' : 'none', 'WEB_ADMIN': '/admin', 'WEB_WEBMAIL': '/webmail', 'WEBMAIL': 'none', diff --git a/core/admin/mailu/sso/forms.py b/core/admin/mailu/sso/forms.py index a81667a2..bc2b5363 100644 --- a/core/admin/mailu/sso/forms.py +++ b/core/admin/mailu/sso/forms.py @@ -6,6 +6,8 @@ import flask_login import flask_wtf import re +LOCALPART_REGEX = "^[a-zA-Z0-9!#$%&'*+/=?^_`{|}~-]+(?:\.[a-zA-Z0-9!#$%&'*+/=?^_`{|}~-]+)*$" + class LoginForm(flask_wtf.FlaskForm): class Meta: csrf = False diff --git a/core/admin/mailu/sso/templates/base_sso.html b/core/admin/mailu/sso/templates/base_sso.html index a95cb23b..0edd692c 100644 --- a/core/admin/mailu/sso/templates/base_sso.html +++ b/core/admin/mailu/sso/templates/base_sso.html @@ -6,46 +6,53 @@ - Mailu-login - {{ config["SITENAME"] }} + Mailu - {{ config["SITENAME"] }} - +
    -
    - -
    - + +
    -
    - {% block main_action %} - {% endblock %} +
    +
    +
    +

    {% block title %}{% endblock %}

    + {% block subtitle %}{% endblock %} +
    +
    + {% block main_action %} + {% endblock %} +
    +
    -

    - {% block title %}{% endblock %} - {% block subtitle %}{% endblock %} -

    -
    +
    {{ utils.flashed_messages(container=False) }} {% block content %}{% endblock %} -
    +
    - - + + diff --git a/core/admin/mailu/sso/templates/form_sso.html b/core/admin/mailu/sso/templates/form_sso.html index fcabad41..d28b82bf 100644 --- a/core/admin/mailu/sso/templates/form_sso.html +++ b/core/admin/mailu/sso/templates/form_sso.html @@ -1,7 +1,7 @@ {% extends "base_sso.html" %} {% block content %} -{% call macros.box() %} +{% call macros.card() %} {{ macros.form(form) }} {% endcall %} {% endblock %} diff --git a/core/admin/mailu/sso/templates/login.html b/core/admin/mailu/sso/templates/login.html index 851e6643..81d83a46 100644 --- a/core/admin/mailu/sso/templates/login.html +++ b/core/admin/mailu/sso/templates/login.html @@ -5,5 +5,9 @@ {% endblock %} {% block subtitle %} -{% trans %}to access IF statement for switch text for loggin in what the administration tools{% endtrans %} +{% if endpoint == 'ui.index' %} +{% trans %}to access the configuration page{% endtrans %} +{% elif endpoint == 'ui.webmail' %} +{% trans %}to access the webmail page{% endtrans %} +{% endif %} {% endblock %} diff --git a/core/admin/mailu/sso/templates/sidebar_sso.html b/core/admin/mailu/sso/templates/sidebar_sso.html new file mode 100644 index 00000000..6e919731 --- /dev/null +++ b/core/admin/mailu/sso/templates/sidebar_sso.html @@ -0,0 +1,60 @@ +
    + + + +
    diff --git a/core/admin/mailu/sso/views/base.py b/core/admin/mailu/sso/views/base.py index dd6f60c9..702b1582 100644 --- a/core/admin/mailu/sso/views/base.py +++ b/core/admin/mailu/sso/views/base.py @@ -1,5 +1,6 @@ from mailu import models from mailu.sso import sso, forms +from mailu.ui import access from flask import current_app as app import flask @@ -8,23 +9,15 @@ import flask_login @sso.route('/login', methods=['GET', 'POST']) def login(): form = forms.LoginForm() + endpoint = flask.request.args.get('next', 'ui.index') if form.validate_on_submit(): user = models.User.login(form.email.data, form.pw.data) if user: flask.session.regenerate() flask_login.login_user(user) - endpoint = flask.request.args.get('next', 'ui.index') return flask.redirect(flask.url_for(endpoint) or flask.url_for('ui.index')) else: flask.flash('Wrong e-mail or password', 'error') - return flask.render_template('login.html', form=form) - -""" -@ui.route('/logout', methods=['GET']) -@access.authenticated -def logout(): - flask_login.logout_user() - flask.session.destroy() - return flask.redirect(flask.url_for('.index')) -""" \ No newline at end of file + return flask.render_template('login.html', form=form, endpoint=endpoint) + \ No newline at end of file diff --git a/core/admin/mailu/ui/templates/sidebar.html b/core/admin/mailu/ui/templates/sidebar.html index 0fdae9db..0938f8ac 100644 --- a/core/admin/mailu/ui/templates/sidebar.html +++ b/core/admin/mailu/ui/templates/sidebar.html @@ -125,7 +125,7 @@ {% else %}
  • - +

    {% trans %}Sign in{% endtrans %}

     diff --git a/core/nginx/conf/nginx.conf b/core/nginx/conf/nginx.conf index 9ce12980..ac5aabe2 100644 --- a/core/nginx/conf/nginx.conf +++ b/core/nginx/conf/nginx.conf @@ -117,6 +117,12 @@ http { include /overrides/*.conf; # Actual logic + location /sso { + include /etc/nginx/proxy.conf; + proxy_set_header Host $host; + proxy_pass http://$admin/sso; + } + {% if WEB_WEBMAIL != '/' and WEBROOT_REDIRECT != 'none' %} location / { {% if WEBROOT_REDIRECT %} @@ -158,7 +164,7 @@ http { } location @webmail_login { - return 302 {{ WEB_ADMIN }}/ui/login?next=ui.webmail; + return 302 {{ WEB_ADMIN }}/sso/login?next=ui.webmail; } {% else %} } From 960033525d4075b5f6c91da50b4b2e2b0a0e2303 Mon Sep 17 00:00:00 2001 From: Diman0 Date: Thu, 2 Sep 2021 18:02:20 +0200 Subject: [PATCH 395/596] configure sso in nginx --- core/admin/mailu/sso/__init__.py | 1 - core/admin/mailu/sso/views/__init__.py | 2 +- core/admin/mailu/sso/views/hello.py | 6 ------ core/admin/mailu/ui/forms.py | 9 --------- core/admin/mailu/ui/views/base.py | 18 ------------------ core/nginx/conf/nginx.conf | 10 ++++++++-- 6 files changed, 9 insertions(+), 37 deletions(-) delete mode 100644 core/admin/mailu/sso/views/hello.py diff --git a/core/admin/mailu/sso/__init__.py b/core/admin/mailu/sso/__init__.py index 2d3c6d84..fcdc9cd0 100644 --- a/core/admin/mailu/sso/__init__.py +++ b/core/admin/mailu/sso/__init__.py @@ -1,6 +1,5 @@ from flask import Blueprint - sso = Blueprint('sso', __name__, static_folder='static', template_folder='templates') from mailu.sso.views import * diff --git a/core/admin/mailu/sso/views/__init__.py b/core/admin/mailu/sso/views/__init__.py index 38efde4c..990c7919 100644 --- a/core/admin/mailu/sso/views/__init__.py +++ b/core/admin/mailu/sso/views/__init__.py @@ -1,3 +1,3 @@ __all__ = [ - 'base', 'hello' + 'base' ] diff --git a/core/admin/mailu/sso/views/hello.py b/core/admin/mailu/sso/views/hello.py deleted file mode 100644 index 2e9b5e35..00000000 --- a/core/admin/mailu/sso/views/hello.py +++ /dev/null @@ -1,6 +0,0 @@ -from mailu.sso import sso -from flask import current_app as app - -@sso.route("/") -def hello_world(): - return "

    Hello, World!

    " diff --git a/core/admin/mailu/ui/forms.py b/core/admin/mailu/ui/forms.py index dff7008e..640408e3 100644 --- a/core/admin/mailu/ui/forms.py +++ b/core/admin/mailu/ui/forms.py @@ -44,15 +44,6 @@ class MultipleEmailAddressesVerify(object): class ConfirmationForm(flask_wtf.FlaskForm): submit = fields.SubmitField(_('Confirm')) -""" -class LoginForm(flask_wtf.FlaskForm): - class Meta: - csrf = False - email = fields.StringField(_('E-mail'), [validators.Email()]) - pw = fields.PasswordField(_('Password'), [validators.DataRequired()]) - submit = fields.SubmitField(_('Sign in')) -""" - class DomainForm(flask_wtf.FlaskForm): name = fields.StringField(_('Domain name'), [validators.DataRequired()]) max_users = fields_.IntegerField(_('Maximum user count'), [validators.NumberRange(min=-1)], default=10) diff --git a/core/admin/mailu/ui/views/base.py b/core/admin/mailu/ui/views/base.py index fc9daba6..278176fa 100644 --- a/core/admin/mailu/ui/views/base.py +++ b/core/admin/mailu/ui/views/base.py @@ -11,24 +11,6 @@ import flask_login def index(): return flask.redirect(flask.url_for('.user_settings')) - -""" -@ui.route('/login', methods=['GET', 'POST']) -def login(): - form = forms.LoginForm() - if form.validate_on_submit(): - user = models.User.login(form.email.data, form.pw.data) - if user: - flask.session.regenerate() - flask_login.login_user(user) - endpoint = flask.request.args.get('next', '.index') - return flask.redirect(flask.url_for(endpoint) - or flask.url_for('.index')) - else: - flask.flash('Wrong e-mail or password', 'error') - return flask.render_template('login.html', form=form) -""" - @ui.route('/logout', methods=['GET']) @access.authenticated def logout(): diff --git a/core/nginx/conf/nginx.conf b/core/nginx/conf/nginx.conf index ac5aabe2..d428b0d0 100644 --- a/core/nginx/conf/nginx.conf +++ b/core/nginx/conf/nginx.conf @@ -117,10 +117,16 @@ http { include /overrides/*.conf; # Actual logic - location /sso { + location ^~ /sso { include /etc/nginx/proxy.conf; proxy_set_header Host $host; - proxy_pass http://$admin/sso; + proxy_pass http://$admin; + } + location ^~ {{ WEB_ADMIN }}/sso { + include /etc/nginx/proxy.conf; + rewrite ^{{ WEB_ADMIN }}/(.*)$ /$1 break; + proxy_set_header Host $host; + proxy_pass http://$admin; } {% if WEB_WEBMAIL != '/' and WEBROOT_REDIRECT != 'none' %} From 39d7a5c504e93f1f6940f7fb2e42256681970ee6 Mon Sep 17 00:00:00 2001 From: Alexander Graf Date: Thu, 2 Sep 2021 20:46:08 +0200 Subject: [PATCH 396/596] pngcrushed images --- core/nginx/static/android-chrome-192x192.png | Bin 6274 -> 4993 bytes core/nginx/static/android-chrome-512x512.png | Bin 18169 -> 15628 bytes core/nginx/static/apple-touch-icon.png | Bin 5802 -> 4662 bytes core/nginx/static/favicon-16x16.png | Bin 978 -> 924 bytes core/nginx/static/favicon-32x32.png | Bin 1523 -> 1483 bytes core/nginx/static/mstile-150x150.png | Bin 4788 -> 3897 bytes 6 files changed, 0 insertions(+), 0 deletions(-) diff --git a/core/nginx/static/android-chrome-192x192.png b/core/nginx/static/android-chrome-192x192.png index 86231db9042b8cf7191d469d5c1c39c164ba9514..75665d8c2e5be997b14c00a099033bee75f9a94b 100644 GIT binary patch delta 4723 zcmV-(5{&JFF@YzLgntquNklQ52N|#K>xy*M#fp z3c6UkYd{HGt1H%;%Uv<+mH~A^L`4}SN>D)%RFoM6QD8uV7#PyL$$4JyIj_%5&-2Zj zmpa_;{%_TXRw?iGyZ!&&=iYnnxtfNSd7GKfG$#%wx)3K2XMYn{5F?4P#KXjs#B0R= z;=upy3F0B*ZeoOd#!#XUakzX(^T>Cip*bvarkRMahlqTnh}d>c-`**~%#D6fN9mWj)ecJaV?jn}Sz)=TL z@VH?Kafi$*;jQ2U;qwg6K;mf$acUs+q-K_KWj$hm^da=*G2tIXoJUL{c0<%@vx|70 zI9K`<`cgc;ZzaN?M&v`lmXR-gdY<$##t90BI=uMv7KCrR5qQOWQ2insnybR+g`IxH zi;^w?k$*Zvp3EHe!Qz^G;SMK;-6LrL5T!c{Y&=*VFDvH=iyRU^$DmgdD35pj0X(y^O?DC(Kc? z5msw;g?Jqhe{(5if{q6&ZK8QZv-P;9=<s$wleE1Qkj{fFT-=Dh1xI?n9>mpD`|R z7-akpoI|k`9R_^HqQoo1h5-f<`(3@*ET9DbG60A4r^h#dLgCMVC58HJYfAOM0oNF@ zx@HjEx@#nOhzgxqtoipYUCoj^=L5E3egiE-LWbn_zbQEt7@;PLdHb3;+Z8 z(EvM$-mwOVE_PJ|1EBfT09C~Gv0;#z`!mV)i-ZAS0DE~Mo&h>Zg)JBW&Cdq-oaoS) z0S3B0{RsoW0M1N@p3;~BZq-yc(H#bW0qlj(8T*rI05ksouM!ss1Ao8(_Vc7U4lw?O zmx!0a0B8X*z+$cbOaa3HJvHMrZWsV92nN`!)t6H@3@}V;slWheK`=lOF|?rp{;C-@ z)?om&AQ*rrdqy@iz(i#(4hDb$oZ+!L1MpU{8884^zzi^rXu$?Jl=vD3Kns`wmJ{tY z$&T9^Gy~Lzb-lDE9Dkuq{;4tzuQ3b*Siz>qOc4EXHnBHo24EpQvawtr@@bKN<}zk8{iRT z)&N`@1cyji9 zt_X?&-qa*B4u1xaRz)UY;dP$xk_qTLk6uuqzxPvRWU3bH0Jbuh4e$jF5W7ae0z7Jg z!vdi1I(~71z96p(3t`T%lnsy#1H?9d_%0+N_Hs%D^d0>_DbzpPSB;fm=lD{S+-Mje zPWxaTtFhCNGg*li6YJZCu=rb=|QLJw)s!{4U2gCs1 z!vIOP4J6PDNT^l{;sAnIqIZVt~ysKsu=t7W~fUALQ#JS38&500RWb05y{7 zX-8whvT&Pc@EZ@Dm9LLpTdL>Vl-j^X{kE`19RsLarajk}2H0O2j(o-sCH6dB_tEFG zfLnf$`hRb0GdmQywql!BxQtM(NPl@-g%Sqf9IGQgEQm}grvtBvw70KJdk!05y&nzG z@1sI}*k?r%LD zn-_|RvdX5Ae|dxHS~kEdiWq=DBQL{o&SRU(?FxlfYRVMEvcmvb<^t2Qz4k0HRnonE zd25BC_!DUWe)^I^eYH(v z4^G>Ds)@917$7Oak=j7?$304vMR&pg@_+d(oTo`ZcCA729DHuEJ>ut$I}dzcW^CNa zioe1P10!r+33is%>VLym zmFUedl7%lH3nGGrx^G=L5TK0Qv5-NpLJYx7*5n)%t)X z>Fq>=0g~dyhUWwP69zE9pM|&5#ecoIi}I@UK8w@eh6w{C#n^@hxJH?&P`3>r*T4e& zXt%S~8F&~z^Oq|9*oB!14+cnT3iQ&31~^p?5QYI7Tn7u01?fb69?mD4yrWV-Vt(d1 z_%J|1ekJ-fG{9lRS{NYqTDr^+MTBYRdKqVwKDVVJQVP+E8w~>_gb#%2(0|YXEs6JF zfX1(<0}1n`t@h*(l_$HWzzqX%C^jRrvhMsD3L0Pv(Y&DnnEPOW_}9ewLlbkYEmX_9 z$|gj1m08<|Rv3V{eR2T+PU;)Dv?wk3@A7BxXDMcY@#abKTD=J_AhJ7fUKF`=ME=mt*(G{j8u3tFZJ}m1_8|<=Br{~p8)E?Gei$G*Sisk$ zSkjUlayNmVW|bXsq~619WkH zJ-|Z-sJmW1!gNg&m7Tf5G*_l@s>~n^(B%Aor=|?RiE%H%0I6T|DJ1X}zbU;^SF)s+ z!!rK%?*!{}u5KV4C1q%{WFL z21utAAPN4Rot65e?SGAzEP(-BGr&68{H4Wjh5?$nxi8GQHziEIr#2hyaaQC{>NFgXr8h=Zbyz+1)<>2_m!8QQz zG1!x4AF2fZhI`kSWmW;&$a72C0L)E9rPDXC^Zby0@+XCn10T|Xi_^B66$+&I>)`c= z{u%V4GSF}6!rMgC>j$35xh}QvCpI-A+pV#Hq6e#agDfABem{H>KlLH`x4^xD7ZdOz zT=9HNdQbmo6Mytp?j%V760<=zd1>i4GePcgbSUr`|H|y?9~}mrNNhu=0*}DCY`roQ zf9zY}-{@H25gy3==^twXehX|x=Yrro{bSz(52J&D!i1&1m1eC0OcV4V)}y0=qVS;? zM_Vuc=mGI)q8gnI6oWGs{$kJ5KiULbNOuW394N*Dvwz^NW~%|1%ZU&=9Vo(n;zE0$ z{?R5lka!s#4;10~*lKU~o1ibT5uFcQhBv$SbWZ%yq0p_a?HvZdsPuf3Goer8Cg3Ah z-^GmpF7Z0i+IjIuo1h;;A>abv5`Ekge{?7`LUX&QE(j8Z#2?%VeHu3b@40+ImZEWk zfZgd1Pk%Z)>yzADpc}CqHwf6w$HWn8h(Fc@XAwJalYmWZAqJ`?{#X;-g7E;b0Y2Q~ z1~uaT#H+-QBVi6$oapwTI&ptu6Lch|;l=@rm`t=&U;NRb&@spw1s1SSCRjA@G}8p9 zXl|c52E5E_*@~-q%K*_PI9F5Q0B+#Ly}d)cDS!T06Y$wgKjWr?B-!Cye!nv&;0^VB z>dY?OT#$gDh^zb|{#X-?B=+JagC@j(wU*RbUfwhTAHa7ru?IIB1V{X_CK!$}0cgNZ z5x?eN@yCWkS8Av?3XIC*7y0|iAAfe}d=%OWG~Y-9r4}6V$C}_YVgQFDwP9UL-?78M7$AtE>Q(xd1CwGOn+hkLLO^FBAz)IPy7gxDgD$RUB7)1;OHcahKYy5JXe)Uu<3$j^__VZpwN&EZoT{<%n_ENlXbsJ9 zkQ8I(0$#5XYlx6XL#0}|_8dt`=7S9T%5Q||(Gr@02Y{VrZQ%~toA8xP3#yovS}AWY z9_UY!H1feR?!jOdEn`FQY*Gi}B;xnPy|UK8XCUNCOkHl{+afHVT_)=rn^2^(!fP_a zb-BE~I>@!+-k~`%#d?VBcp4%FfXB$@{&&S3IZ_^;%Rv?;7;J~-|Jg0`@SN=Tv7ELu zk+@sFV~~7jJBU5A_J7Q!bkAkbyO@))7l8r@nUgdaJ`h%*GN}Ln002ovPDHLkV1ga5 ByypM_ delta 6014 zcmV-^7lG)3CxS7Mgnt(!NklaGkc6xdmOxhcL>>4j zh$D;x3Qvb6-VbcZb6>2!5h zbydCj<5cBUI$hnhyjO3j^Zk74ldej4)qStd@7{CIJ@;G#rGI>u?*ym;8i6P<4mbc9 z3p8h*Hv^H(Yc+Y^DFy5V`ZE9S0=hCkZ_hlBW&S_K(s6EIx$iODZh&tPBL@f}hCC6N z0?Y(v1Jg3kCjm`}5E_B{%-82=EQ7BFCVwDCJ|CEyc}&f`9zhJ;4N{2FZ$m`!4`5a1u^t&Ul7I>#4=R9M#OOo7 zcwi3j4qypz6k_mE+%xtjiHPCP0>)E_h<1VaFpe_s#06Zj0`?limIw1BV+cpSI~nKZ|N zK+F^TF96HcSpu97d;m3kr`QEN2HXj>07<`#z`p_z!#98;4&lBS?|hc4iL;1*=1*%zaM_XS{uH9rhoiDY7Wzunb| z39JOZjjSWYy)FXp8j{uP&p-_SEOhnOFYE^H18zk2EqHb`@TLH)9airHz6l)T*|sYA zFn@uUfUf~h;nB1TZyA-<>yJS~n^&NrO~2EDOpxzJ@{K$-8hA$lmhsO9z5#rM>dn#& z8gXRZ;2I>!&NCwLh5#(%KMwI5m$=;|#db_&L(Ufw*T@qhaDRKBwIl0HWI;uZKbROu z2loIH(W}Yn_*`w2d&muF1TF!tLUVYkg?}ALaN|d~oMGyo0L1vm0$)J{(BN8I6gP+g zw<2o>J#LA>9RY~(Pt08WC((3d_d&q-k;)8;G8|_hi=hWp;=`O3fb{<3kZ8{tsL!vA zjDc)#y$%@T^kSKF0+5DBfv*E+qsFfkP6#*$36C~9DFSB%U{z|k9NDw0p-!b(!pH*Y zML^go5!i2-R<(w+fG;6?con4-8h?>YfsX<>Ap(2d603plLss~vICyiE>fsIGQ@}HT zt&t%oT$96r@1Vx7*vAaudZZ|Vt=K03>HWte5uZhsxi3l;u>`mlXtL84vPS?`PXFb= zsTIF9O0}>QxELGv5|l3ht05oS+c-sWhA>hW>J-41NpX1suo`d#67SayEq{vBj7Rp2 z+cF=p!vz|JELa@n;!RPii$%bfv0;&{bOBf&d=`@CpeU8(OyJ`{S*}pIBftXS3+Q%6 zrSdc)3$0~&9;FFD6gLVPhHQ_`DoTMIh7?aL*%c~Y0M-Y67|rch+{d!R5=2m{3p5M( z9P0Wg?h{4a@M)#|ad85$ihtH#2+VW)$`vKUV&q^%qu5cPZ~<6v{7xjlUs1e*0W3p= zP?RfFtP2!Dabp?@&404yb)3>@$F6)MUIoB*6u=r0#K0yG2XqG3u3PcEvA55SxvT^o#igU@7O_z`NYOB1I{Zca!y6 zh8YD$8UY%RMXwQ+q$nO^4Dbo0K2=^u62L;TaaBd}A147v5BCd03&1LFwG=(4SMe_g zA?Hx!b%TZ*0j8m=zJH4UIgMdzRP8SSODXRK4t2Zric%(rBNO5wM}d4JKqHb@p(Rcf zzf*^l>=`nQpHBdDk+f$;2?EC<+b8#9Ujgh}+*T*QN>Dfmaq0JSgN7ObMk4}Hlpyhb zhFH%U>VWDaf%&dB)pYgKl`O$J(`OMfA#TZi(Gq|)G@OW>>wo1M3!7_L(i~PDxJWEz z^4Qh^b`J#RTbO{P^S7{k=iW0029E#{vLPe4FyWL*wfy(oMpv6r?R561d0|JA-3c>D zBf)!sJAwY}3xnqXv(Rb}N^n^K9Gv@t2*7fGjsYgR-AYAqh{!h*We@Q5l2e(Fb~Oi-Yb9u$?=O6CE&~?lkAxlz;rbJuT&2Q zl35VT3g8ejLxQ4I0a0K!>??raWPB*f@PvRl*&pf=pMOYEs)-puID`xbIssOyje~(` z7&&@=yxXm*G-(t1(~|3M;Qc)~xovE$G1OlzpGiPd7?}}|al18@rax_R)4DiMbPQnl z#F`A?!&B=9~GF8IcfoEsWA!7vuho36dUFK|>8(Fsq(xj%cK*)+fW2_h|qo zgb`n;YWtK4^#(T`9ib^4;*R!y;%T>Al!FPQ!UkWM)4-SJ)Dx*uUZI)kf^XyBq*4S(wP*iQ(U5TXhrKtmA?8+`e&1`$Ho za5pSu!o&uH>yK{a(%EYK4l*u85l2b}o%#@5GP{8r=QT2+!6gxxaBw8VZSx~6n_f@2 zs>f!>X;uLYqBaD}rq^@({0LJcp~{NDggK)_+_fmeN2k^iQsZ};Q6Z#yu98E@fR9YA zj;iO5mwu-<2PM=cC-HRd|K3a|61%GN)00nZy=o;=`6yfwKwYDsp)EIE~v^wrs z5Mg>mzXrDmCConP z@Y2pCw=alrgw6i-_&S5@<~Gt4Hn@3RoW67cUycc5Y7M?}cmtQsuBX9)yq=b>6c@L} z_}k7TOQ(1sx^2SbNQj>u8)ZUW2!FGL?!3k@_(6L=x2|`@OcPgbzrK)u1Y<~DNf^m<2TUe9b#a!Ff^Roy8J z!#l}cwFVp-4Kb+P>81UkbB+FLx^7Qtk?Qh>$V}Ee@Aq`xA zR3qc-^Q2CgFu5_rt@9&%c82}NZvu~Z46v-FmsQ;5C-x_1Xo^&vc z-&yvAfL(64!f^~(*OTV_)_)kk*p#3Um%SKj3^?=PI__K;VP-V6H+nI*sfHgfigM~f z_JmGzNt$ra8*wgei?J!D#$O^k!@%2ax1u5#uql@2(zY182TabHVQ;3T0i1eJEzxk4 zOWR^J*1(+$qa0)N+yfwC*4Nt5f>_H)y^IK4?V{xV1dTf@LsAVoC@0KkxS zf~#Nar#or##Y63n$2%2&?;0?lpAv#{i#ciP6^W zz6hS*kz`p*FVAmJ>SWg$HUhB_(gC)+-5O6|z_Z&khPT81)QLj*L+1eJw!~=N70}cM zZ?FNZ(C1oU!0i@!0s~rir8u`G#*^DBdrM5_l9ky#m+-Y;n6)p2C2Qy=g9L z?dASAoee)G(s28RIGv)ZPD#J0gPql*ujY z`?>nHK6WKMmfBG6Rz~NyG?vP1WY)g2TOtR5HsGUfw|~rY7|@k4x$4zE_9RU%JH)

    NIAd)ssrbR-8y;tIHC6fL_n5E-j`A!f4 z^r6S4?oTFv=$cIS<(F;P&$}~W^7Yky#qTSw5aa3$9y~T$_WUPg$i^WL(F?Q!uyh=` zo8ng^^M8QZZnw~rWb%h@T-(p?q{%mrY-DVShgEd+r@8XgK7RIQ0{2%O@?LIG&m-7O zCIkxmW`J%*d?RHT;wJpCJ1764&_;Z6wujH02S=ptj4Ibb*U`Or?g9Ts$eyuv!fwsUQrp^D?a1v~>B=5~vHhXHTK(p=CQBMt1)k|kc^ z8D#tKKJ4oPi89i_?+J3rL?-PxQAY%#xRCIdtqdr23FvksjNc7Em(st>7lBiO9YYyE4DSMgRE_&3ZU~^-4Y*wN{-uHc zVT5^X{>^=sVje+O(GKj64E}2u$Dsw1%kSebyiX7_s+<;*LdHjkrRJ z;#ExGJ__vpp-=%>qriPgVv(YFkv6iObOF6T6n`*YJ8&0rz?hWX*RNr^UDoB%9k{Sg@n6vYcX2>c&VRPPU^ zi~#Zh3E)-+)mTv!7gz~=9~das_@TVh;a>-CME4FW?$Ha}KpDl{N)v#kv|j*^x_uXv zs^wu|IZ(9glcl_*S=lo%7k^2XJKXKNp;Q&CfR6xcN;Q5c&jrfz58yhqsIF2ub_3U9 zL*-$4@vPRCh#NR*FedxPZpa_yCM#J7kJji8>3VQe+F)%#Hi0;?AbVH zZK6L8iOEiP=iVp*XFYHx@Uk7o4|a_JR%4z6u0iJmiv4s0UjtsY^M6h4a)EM1@DOkt znlPu>M*>N8kgd5k_5NU|0IY^3ki`kzL8jQnLx=!u@%wYJR{$2ffd2xXa`wI`Rlx6o zuL8U6HhyqI09GU41U?VE;@rJa0?yxoFCb_8I)XDJfF*)gfd51f-cyQa4R9IRv#TBG z{lPgGD7V2+1E0e|6@M#%V2Ib~37de+fhPe+jUSvA zfOP>TvUGhVvZG8vGvqYQDDms3=>hOnee zgn%=ETYxDQ{d~o%yoCfyY)Nn{hRPSw77-Z8$ur*rW>&lnihq|_hfH|?NCkWTEbdg1 zx02^hK=yMU(I@OLPn}FE|;fb-P*0uq2S=$7_)|zkjD(p22uq0FdhvH3R1$OXkzu zZk946?a1o>{YddD&$&E<@xB1ESfjxrB%$s!G+gQx_8@sYHzKipPxtz{_(uR)thvGI zNZ$BD++5JBZc<2@lAD3w0WrTAeir`;0CL@;slc+#LyK`$4ih;z{T|?_z*fNX?#@v7 zQvg}4^ndEPz=gn4V4~a23OF4|$(y^8qjS@KH2f_77C;seLM^ZuNu@Xy@b9KlpcGlG{!5IUT@n5W`=A1I4M_Bsc=d#TpS7AjMQa zgoI9YvRoJ`#K(UOS;$&Nzzjc2Fa?lBgir&_Mt?+b1~3mzt;k0VS-koc@LMFV?`t1_ zB!Vq~EY=#rc%+2;8NhpxJ;~KRDx{GO=}#ehGG0K=-SuTur&y{(09mYgLLD+HychTY zvbZ*(itdMrROWaYcmntn(%TOJRblv9s^tscA4?2TU=EUkeIl~3HW3KB-9(#70-eZ2 z_J1j)V&CgPA5c|BpG)-$AQwvv4ZtiU`R;fmtU41Iga2HmG?I_gj`Z};Xa0SI?Agv$ zYV^6N00v`;A%x5sW+N`rVx)S-w9M;X%SufkRrlUPPET*mJk}#VeV`|^rJ@2D2}=+) zz-T1>Vis@&a40fDOa;awBSjs~j233*5nD%k_AX!xQh9qFQboBv^I8v(QiC5E6+o$2 zA=U_zsyzib2sjuSB_<6O}(*^XR s#$Gc254p}h($07*qoM6N<$f+aoj&j0`b diff --git a/core/nginx/static/android-chrome-512x512.png b/core/nginx/static/android-chrome-512x512.png index f029c9fd484a7603a43cb63159b8d95fca06f95a..32a69b21675b748d731d9cd624c8624a412f35f3 100644 GIT binary patch literal 15628 zcmZ{LXH=6-(CDTBK@d<7R8WJBUPOv0kkAxFdIxC&0#XgVCMY%p0clbM6lo$wkX{lS zRf>T0rZlAsQUW2lPkjBpd+v{WJjduhv(t8Fc4v0>k)F<#{risY0|4yT)KD`301o{V z4p5BH&!2$49q0$?q^zwBz}pxWiX8*=f3&@Zfi?g^X8|BY06+&J`~<*P5`cMI04_cQ zfX6GnR$l@7z<5LJiW;EN|711f#X}=ZKAPIbl?cC7DZu3xgW4HwJ1My6Po_%FvRY+ZhK{ycHz3q?!xvc8ez z{LslfY;DSNkKsN-h=(>^t7RAITak(ncb#<@3JT-PEfHA9#=Zuyd-%o)KU^U5lW(@vM-AXR)% z^o>tB$Hx^NeJ*wN+14?|Gh3AcJ_DmwamO60L$o?$Z7fu2b@igXGhzj?5d|)}CiDIJ znRfd(F#~&ZeZ}ienl!TTVi)CpR0Xk}*2chnn5~Ss&gRE@e~_1vCv0C}JQl*tQ1LY{ z^Tz{=cwT;{wnw-W#T<;weq6XrhcEAPZ9TzdX{DR8@~o7&%j+jf9T>k^Eyuh&lJ?}- z;^EEms#8crSIdCk)dBJ(y+;CvA==~n%{vL zpBY~rvz>T>KkMHrQWkZYSI{Hm{mB$(1P84-thd3PWXB&)cqu4ct3A=sGckPmRh=oB znDk&t6UU$}?PD^S&{Q|WvlcoukSoOPrL4!Les5QZ7^BEg`*nMu6Q87RS6hv~amS&p!|G3VyaZMXFwz z9pZ=IORGW(T$Bsl>hmq_*a&O0?k#?Cj-P!L*JLwfy`Oz3E&O@P4Gn#+#G0m++=cz| zEHmfG8YaDj$#Ufgan^cwEgl=&h5eEZv5;C3y1QvZQYFjxcQVFo$+_&r!U}KS`_WhA z?<=xEvdfMx+D8X}%Hy7`X&gFxkvVlA{GxSMPi}gnU7BiW(Ga<_^FCzCjlXz?J%V@! z6&CLq+?X(PIgM%$2tPQ|;`}CE9I-Sifkx?Z9m;81@d#;fd^LJ9QzQLzAV-`oQhn2W z;cT!-cg5{hv>oAPK8I}B)~nvJersBzSoc)FQ^}3_<0p4avU-|W6!@zuiyj29Xf4^*#I*Qj1VrxB?e%AD zF7%?IU z!40z4!W^@2ll$38p7w~F?SW+syZiKvZl)50i~M30L=TNRgY9R$Q6Y~PwNP34oei7* zB^2GnBW~jeG`0|>6P@FCZ0X}1h`fbg_kX-|U8Fs5h8;~PMAbUBl*GEDU|ioSn^vNp zURm-zD#rP5Yjs-%gb zdJB)E;KQcoq<4*htN>Eas8-QNQm@8?>eQkjt^!)p0{tY;BK725sCzTbYw|oN!1VJv zo=i&HUO1ffi^yeTM?W>@%rv?BI^P`l>WH%gc*B6y2HD69DesIM&V#>pRPf zU$59RALjvgx|VuzUgg0Gdrxjvkp1IMIUeBI4olEDHQCx8};Li{ACIP$V#4{oRGZ~)p+FP?8L6CMe6on*b>c2iQVdDKGf&n=6 z;2n9!k=x~DPci@VKiNSngUs5MBUsd`>H5m41|(-=*+?l8?svRbc^~%1%UcJsAo#{e6%ji z{OHC?u2r8`84QFpb!gNPbYUs%V7a1AFnFV@3J(Yq(t~+~xYi7H$5-S!A-w7gL&4N( z1lTqzQV>D}vc#ra+~C;eRHCJk9d`*824GgaliEY! z9Nmqx6CK>f8eRus0N%FWnsy166$+b_HviCy0H|s*j+H;ly#p_h(nhDJ(a#u=Q~?{V z9H(WzN{D9!90y&e=MV29-QJP$1B}$$9{~8^kh-V>oE5>Jgex}||1bemrf|*kDo}*k z%RHi0wB7s&O5d{OaSHrdj$lwHV{RINh|!Nd4?y>yKlq$a^5>ucP-WCPBZi#^i$W6K zP||nuMu=#S2r@Sn|6)|oH=P3z9(q|a+HiMN0qqhhQE0_W3Ib|sAdg>q4I{a4?E34# z0Tx;F&7&l__$xC^TbFjF^*Vh_twHR;H^Q6&urL#yV4JWc>1G0B(JzYa6yi2Ir&Oaz+`GR~SH+wuk3+9&{`}dV^`@ zpykHh3X8`RA*`c2a`%sXP30k*FC8|(qo z%6aTO`>5`6Y*3g60K+*t#qs0snIQ$z{GZ;VJOsdJTa2{%9x}U~yPHpG0x}c=G^U}_ z8NWZ^rjIRH1=S1<~Is{G$TNk;i&YEy}L!j3W9_V{KhD` zIt2sWHSY;iuSzBm1k25>pNG>R_lOlLbh8M3k(o&y^SMWp+GYoYsDdQ?X(q&uG^oCZ z-;AH3m;vAr#jA(0sa&6PBJ_cKm}axB6Hw^z&g_@ioPdTFXkrfBhBN?L+7d`EQY0Hs zm_Z2*FRV_1WOHzroP0}V1_gPv<7@SOYX}&Kt$0s}Ienrwlnby$zFo!uFeGcbx!MdZ z3#hhgBilikgtrRzOv&t^U>2h{{{0FJpal%!Hg9!T0@;Dnc{=@c^V3(RC4lr2Wy>ro zkxu#ER*;d1Y8AbB59bUcWG z10Zke>I$FUh8E<+wtGrP3;8FYn1|bqn zKOlj(KVOqS9SqC*wL$Ye=SOWXb^?T8{$V$F(ZWlRYte45Ni$u#R2wC}#`xrMuxdt3 za2n#Fg1_$M_(RzsbZ{oj05<7Geu#K}Ox5?j&8}w7*5u1uD~2!&Fy1M_?d_>%Srp~y=t;=^N@MqmE$0D# zDQX}~FcR8ScGG-1+)x?}gu4z$nVtctegFfFf9hGgE`x2&s3JTw;8H&TCQa9;Bfc=u zMdh#Z^7=REI`CPlQr&Bw=*)AZJ^T!Sa%w;*Q502`Is%@=OI#>Lg2nw0|Vb z@K0gKm_aa9Q2IAZW_)wa#UEUVTKcCc4zqPfqY(u5HJPVH_d>vb8bG}{YWbxkdI}iO z9^akeW;+gpGDtXpFAek*D#C1?Qb^&a{%Q}KrRZ>XXQTqwt-{9l56J!gV$Cnm{SI`; z88rVdTw<%-@KI3PeveQ3KYk-Vm>X38=7`+~RHwz+4*xe<PI^(q0OT0F#F!rb6XMTX#l;dihHiJbQP=;%9UA%= z_84>+Jjz2mGU+ zbm0e1M%4bS$M^-MY&E40JABNBPEo=GQ?y&`d(kl4hxJzh8f+I`_2sD#4y%xJyYbeP z6nId0?9gIoXw_(%X;U@^2IN@*ZQva@wpW%^E`I?DFeZsRAJ1!Xjz5VYbf+(jUEvAD z7_Zxy>8wNiG{Yuk{_-=TDa7;uOb)!(&ph1lrfDRcgJXL=q-8q*$4ctCnhN0!lx#k} z!;2uHG`(z5AdJYbV{W_-znM}x-N-HGWVCs5AUVpz!hc&p#zV~+474-duEx90ZwX)U59y2^*(RQ0o+WdF$xPs7@fY$6Rn(X7FdW-`OF6K? zKU@VM5PNy;KTmp=Bwux=LQ&>%@F30cFQ}=<8E9GWSgap{hy7N*H{j`(zTR@B15Y!( zTA!lsOXrW0(|LSnw;HmsrZRQ_L$_4Oew`eX&Wq%jM2t1PN~P8PuAR<18VSGz1Hm$$ zomP3Wn+viF^v@Z_XJc6$l3v%-b>V?Mh8?u+UykF!e{t5IAjM0ekAZR9BerntVc}v9 zujcys>DS@urZwMGJ7}Ne+Up$?H;y-C0${^LC}lvEC|AcT@V@7ea;urbD_D+f4o*#JUPVdyI9GMcPF3&=j8EC zrxsf}gMFUx%Mw}vOgcipYY)Q9aGh)h>%<-Vq|Gw`ZQI|X2ki(@Dg{1Vk2mQHcyLW@ zqZXIp_$OqHdhf=Hf2K*0>p{9K@LUD*Y6+hF#0ZB)0Xufs6lxx#Wpjgh zjDmA2=zw?lfKHq+8w!SA+|a-M+anw$m37nG(D}Q>2T4+b}n8@E^L8XuUFLCR3_7m&beM zoTG7(sqPA-lAo`ideSx|1P%fe&@X(criuf+&@p;$cHkSu<^^Ok9J#=&j0F(mZqeDg zI-gts0F#6cMNwm5M3;Q zc%SCaDt^DamCWoZ;Qul7?C{(V!tL8DJu$-dXA(u1v;g)Yf}~0J3CBb|V(@KsZQQTP z{QCr|{nYr(Eo!0pSPURYYPD?8K`J`6w-SwITFJ}$mzbDhVuJChmY;hXksu;-fn0R} zlE3fb%;h>_Wo`d%ziUpMJng=c&EgnQ@;x0$O#qD2!T6{qIoiqrai{IS#5kBIqpn66f{jV zCdQ}D2Q1ee8gIJ=2eu-xaV$jA=XK+`ioQ!8LFTyc>8TGlczA*`%u`o4Lu)=|@{>|& zXKqp3PYzQ`(v83G?>pz!{DZQE0^|!!h%*+!uyk!Y260etG5zUWNpiG47`{f9CaEMiNHnm2z7~LUEetMgEUAqQ8TlJR0{P z{9W+p`#{$#lLT*m+n&gy5|M?iPxMh2Zds%Zbg<7og%A00MmFF>^q=!(AE?-Q_e#9R z<71Z3Iz9=+vU>zC%@JS!K3lv>!(=P?mga{`0frr!f2X}ERJmLYyrObosgf{s zIgIq@1kV<<0jc8%CVeV|5w)M5>#ytIPfeM-P(7?@>_5I*j+LmyP-Dce23*n+^;qBd zkNKmC?MdQ*wMrTb>Cxc~xS-hlDL%V=F5$ALPEgA79gokS98fLYH%>J)aRufzSS5}2 z3EVRNl}h^&A@@aw<`jv7d=HKlfRmCoJ)J)w32zv{7O_`V_f4Mq1!VQFxQ_yd{4XM_ z#U6FxOGbC4?yyj*efMY=@}z27^y{9Sg2FUi3mjmL3tWpiw_M?yvNb>db1CRqzOLBe zcLQ2yjo(ShI%Zc)WEZ>B=({5U#P*{zTcAvfH8D}z4vrOv6M7we;{03s zdZv5c>x=lShIj;O3nSmH^ZmleE<+oxa7|MvR~nI5FXj4h)!hM}zr>wcK}qY)x6?!g zOX&`(>d|vB^a2d6sJ%K_aAsWVz zrlX#AUbmqKP#C(vI#-q*7%pL69pD7GgW#za6Qu4ZKm4+5va5Yo>(qV6&-J#+9!X$~ zjuA-S6f24ze|fcCcp+Xy#4jkG`>#yh3Lq*m9Fpm2GNf6;btZzeT-L(x&yS|%*7Cmv z|29Gi@{I+p`=1)KG1HGv-{j80;m722e8+$Hwo8?(^8OV`qp;kHu-o*M_H;+iSNaAV zs|3ehL{eOprR7S=DsmYoxW>AI1Z1OtFc>a8=g*j z!(d+B_}_ek==8H=H(K$5l|PMk*bY7H*v)aD{czoE4u*ub$n0oSv5D|eNwPc%rt_m#`cN(k9~$^;1*8)&k~U|EtZy&%)vG5Q96c8C&Z{1orc3TL z749_5Uqzewi%c%{z|c2o_m9DtLE}E~i(_&&oLEySi!1q@-A(a)X83D=>*U$LyD-c) zp1ZBT13}o8TZd!fhe5Y#)gm_}@vZ{=X&^z@I8wEaZ)=?m$B#X2C-= zMogz$8)3BE+i~Ty@A%M4fKyFuz&WCl075-*RwU~dr0Hu+T z;hqkr6c@~978`-X;)kT|;aFQZ;aFAp)Q|BqCrnOXNj9VkRz3P!uH{;PApYuFa2N*^ zc#(uzrFgp{2Ex(|XDa0UdN9%xtCe%=tz&<<55hyfQsT9Cp{!eBD)S!^u7=@vD7|)2 znsY7d2q%E^fW_?^?TAtBac!CGwXI=VlYHX%CgovM2dVwQ`3M-b09#+9 zm_axz@M8^*eiVE~TTATNwV6ipgk)V0?^?*H%|Cm5WpyUhf(0~8pQN)?3PU&GNk3ms z=dT}$Af$gAR=expVDrxgH7?l%RdR#cmbY|)yAjxR4oP3*qGn|HV$w)ov)Jl++jQOk zf#n^#uw_EubgY}iA{*5j2-(^!% z0B>T)EM;j){I)#g^!wKUN)5 zyZG)cC@oUZRXqzLgupA$%j}|WHX~gF4#hs2#b|ZV>PnK&PLv^Mc1}X^|KxFS+ zdIOZ!jH=S7h=B`xO-=E;ObHzDi{BEI1a*oQY&c5)uEN6rP~GUlL+VEmc#mykq#d9X zh%R~Q#jm8K4@+-SgSyh*mG$%HOw^+Ze*-FTC;2kqCSHGLz%Mj%LTL)l_bkplVbvLQrB9@q5<{)nB325G2u|JgDdtaV_T@CwR#P zdiZT6?h(GWAB=OQ7WVN;kV!JM0Kt;K*6|WH*{aPBuH)57BQ9T%gnR}lOl_yCkjk~P z55PxrmAtPm)R*5WEnxraK+Yu-TEtBxsFGQ|Cw&u+-P zWe08Wm*aa&D5s%Ef_J_QB(q>R!TVK0cRD07D{vX9*IV3kx2`Zf@_}PAmA3o);>u4K zN9*rDS8Z${kM!7MsM z`#km4S5*kXOZbsFC*!nD{``(P&~?dc?fW~o;wEJWANK1 ze0ilw=Y`i=M)DZ^_59vf+u5XVftZjyg^KJGe;Kg=&rS#X)|2HJ2=3G8JWkSO%}r(t z`_Ud@nlZ<4cQ&a%Fz<_C%&aX2DhL8EsyIUG07$p-=KPx>xUX(kL)o{Q^(K3Mfv(un zLMn#H0LkYDG+G#*yHdqeNkWyml5z(jFSg~BHAMwD>z}M?@F;GJ#&~I!9=}*2UBP{c zh=8Wt0^4eAT@Jv@n>o+x6dn8U&F<=IXv{Q>Wh6TAE~_4@ z8fW5&GYXSUrR}j4HXQ4*Y{Jzu1EW3!=_>j5Y%1Omx)D~Iys1KW6>rWL=hVl|R}zBD zrK)|r0@r7|enVU!2?+&dp0aS#tQ-3^2+n7*%6I*@ZifDBsL?2yLodO3wK&sClg==b z;5&9ou?*7Nf?Fg^j($APYPiHKjO06C^d+tA^%U`~#)ZC) zacnezOOZ&dC__H=`)pCZ%fM|vch}qM4gQI#cs#wF1d-Ti5SeJZ2?i}P zrtZ`E_K`&<1aUUL{ykiVE8Vp*&!5FNjm*4R(52fq?^*)umz&UXidKvo(F5iak%Y0c z$pJBe8C3|b z#w$$=^(X4hXMODrPteU|u$0|B*IG03j=CYOE^Ynp{zY7UqvmW~ z2@MK8y^b(BdQyP)CK7ok=aSf{BlW_bnaNu80v5zIclq?lkSA3eW{b*%(M9~SP{<)^ zdwq1_gDakn%Q$5b3a@GdmMdDHC;OV>=GB}L%MUjp{i-X7u*FOUOq!O#*)ah|iq zbU#sE+b72Z5e9TJvEKXF+-3);p1V#2-kbW157~c=l$EVb zJSeLn!m+|gpE$gxj9mJYu`Ivm*8nK<4YkS#rI7&Zei|&UtN=ZJ{IyMs3FGq4z77>YM?{9u3^x4;l+YxiM< z{P_m4Yj5JxkZb5+a6?w&cOY**NjvX(ZH&;`eQnWAzqIQI4>x2yULWD-V2pYk{81ng zIlrT_y!dK%!aK^dYvL$5=Fi&10xc;MHVW?HJT4r=8y^v*+??D|G27l66-)e>BRM;A zHI#9%M^?!8DjI<2h5{1pSenwFTqvVgFsDtsEf#8`?kFnToie+%t~kL2wzzGX$dM=5 zpk@#4yrJG%r-ln}Ig!Wh^)KvADK?q#9 zJFOXf?>#$oPd!o-{s*da1}}ZOV^{ztbCF@qCK5lIAp}b%Lur5Rpc#SF!v==Qzaz(E zat8L;jezq$m`D4E^C<^W0Gn3x+xx*#K&{#8>tAA|N~x3+T!0`OXjF-s?WOH_!a9G5 z0qC|p!Fsa_T}|J|4xi;D+rhwK{(~C@XEyBLmly#rt;1BwB#WL{`bGW5p?3ZEVH*=c zs6nBmVc};djzD8$5|;P!=q6vhcB)|d3M0t6SAlb@&Y(i>_g(7AiKBqPz05KJQ7lAc zaN0w+yP#+EHC`ZM{TD=XP(5Rci3y+=b!iJLTmcygzq1kGjoy}!U-SJKnfRMy(r;f#EGGaQ`c;|}Kc#6&o-myRi3`jC z99VzEccdr}I&){mjJ7_48i-;oYQE++HNP~~_JobrHkV)EhoH_`u|?iox4|GsPc!T6 z1K5_DGkA%Z*+N>sD#=hX}Ql7$3LAq5)cMl&u<{L-vFYT>|Ij?VzrX->IzG zrOwyAjN zf>e(DIN>(ENhMdUIEU=7fOHc%|0;vt+rl9uWi2ZJd=!veEl1w=4Y#Hn0Rr0W0_v+qZvo@Y1yO2p}giDad=fl^MEpdmu55Zrq)lGbDMM@j@*GOdg`cU57U^i5!#G zb(A#D10Z&4g}T+&U%%7kiMpURi&KFE-1PpT2eT_N1>u;alk|?LB(cOP^72Y%6-aB% zyiwr$OkNoVhGJtE+*pt5(v%?INhCc=&jJL|&@KFRzgOhl^GLo^nYfp5Ahmexi2Br) zC(3JA!sZk7jDo;wanfmZdWFUzso{}yRmTEg1!~-Xju{ZgT&V9+Fpq*sr^8$*@H0ZV zMz`_>nMnl6CwyK$90rc@{gKn$%po&*>rJMV5jc;4Zk=-XM$E(>qFy6*M8Z}^;5^8R zPHO!evc(>TI_x&#MMpHqRrPxDgd^y{@IguB5eb>}Q9#)5#U9Dts+WqdRw1>R-9j+k zgx-Sm&jfvRecSiC>`w;=+ddx=hR!)PCh%tJy!lw1Nm(22gu! zIlC2Sc8hj+=f)=$tg>i@6UnDw<^~|Nc(I4Yv?jG$W7XHhurC&cJg+M9Fn|c_BE{|K zQYPgcyNEnSzc~Fe$)sUQI13VdD0rVr%h&`h`FIs9E@<-W+rB2{2mnw5{xK6Vv$_@B z@F($Su0t1Z7d;MC+uIjt|8S=E1@UKOwhsiX6?$d^%11AHA3G^n|c;@ApH>@fU0XbBl$l>#66%^UJV)k7@9EI<+{#vW-h@N4Dqw^0+eVg6ddPO!g8o5JKt~_v=LGA$58Q4X$9*SAn zQiNWw zD?{7+)8JQ8P?6z;a3ZgpF%enwcMA>o)zWRE8zOrZT4)HeF~6lH8X+op|MVZ$(i z*u)spl{;Blz(Kgyxa`mJR8ip6(u=#Zy822iu+DzjLryOmUQaPHW@X=Uiy&Gyl3kocu^B9EL#Wj!Cw`;@p=y*Tk>Tt(jqeTQUQV{~)P9AJ5C2KIVq6n<$<<>lKx z2yd7PGyj5C2E2h|7*mgAE0-OvLr|^AxHF-(Qj5os5%wx?vna`FMx)NuYvNTt2V=OM ztmCR!TK~uW91n}H26iYN+RXcHXtaG78?HT9(_Fda5R2w$CaSaR_?5X|Bdq>Y}z}B3}+eZw>mgJli_f&d8aN29%O9!!qsflp(ZT>l$do_{VzkQ_@9K}EnN7!HtmF(N!ahY7Vrx@05#f_e3YRA&Q)a-*lXvH|l zYq21p23!9}(OvR{CS83ar90RcZfAE}GaZkl?T49)We4N?vIbv1M>`7<`_C zWqo#~?&BTY(k0(-m|bZr42yjEfdjd3>CmiFCBrYbw974)Fg4?7_wMyZinZUnTpNw4 zJVD_o|5j{;+tXl4B;?M`h>?Il6&5mN0#i8i?vS&e z@X{RZc6VpAX0MV`qCK_H|HA_6j*IjX9JCfMeG6aaR6b|Vz_IGN>*{XWkvE+u6>(En zpR3|KdNL!5?dihInKYeeSufCN;l=dlZkE4~YxL#`LJuzvd$p$@7xHg4GXEj~-~Mxm zxLb_Tq~4kn)%s(=rS6DEaSh)P^soxQwA4XL7FCQ=km%UWdH#4gT|u4k{IeM7EnBc% z*5gH6yGg&`d5gBlyDeW;6rs0be#?6;ou9>}L!ozIqxWHQoem%ewV{K63_7Eda@3uAL3 zab{@P$$ovzCvHAma9+%uAxu{4p=U;eGH$L-*X?W@`lyn+zESB%g5~{yh=ya;YRgHc z?zg%7k3O7@3=vu4^E!X6Gx?NTot58MHuN0M1ua<0U_RQ2Y;NfiU{@#H5WNL0^73k&kxaNOJ;Gm7;p%_`YMpo`%vS|CU0#Z8I$mFQSNwK zRS5kBUBxRMddvg-^pS)1`qeWeNpN2+XI&$E#k#;Nb%-+``l1p`BB4>t7fR)t!*FG64#lPr6-@; z*RmRnRw!u7FIPdP5kbbMKspkDH+S{Ak zPu$N~depTWr_A5%D9<%FEOO;WmkqTn$ac55E_NkRy-l8dll|5yHb5bCyd{33c4jXg zc5}2BADxvCeRah?J8y_CCjWkvmD}!=wQQ#?;azl7P}$OkCJKl$+ZmhwvzHXrXWmjV z-SNkhVlVn^Ls{1uA9XVy2Rk3fi}pB2=m$tio|6}oJSQe8YjjTfqLk!Csq-R|=PpW0 zX0+Aw{67=iJsn(b2LAsitRy1~p$VozMrJ++cK&EDoaap!H%GKjpqC@s#mmPYI?JU@ zt+AqRn+OWF>9&8-oG}21Q!L{4ES#q}wc@!DoM<(I^hXE8sd6K9`(R(or!SxQJ~6cu dG_tOWgL$3zvk}UB|F~gl&h%y7rq+!^{zRB@at8O z03P8XwwsE)cx;W)88e~uW5-DhEvH{i90D$itohjijV`vstFnZmiy7M$iE0umH4X8h zo-%D3&Ces^gE{wyI33jfF8M&6P-Z)tqfg~8Bk%%k*npoD2F8ECf1bx4@kdmsJJMO5 zhmE(uZ=sj-$#xyW>+Err6Gn@wfKj|{*lj=5nP|r|7idF=a+q?AO&2zzB3I89L}z#_ z%~52C7ihG;>gjvLYXUK>v_5>mLNzC&1v#qmcUsX|q>ufj$Sxx+D)J@wvjSqfiP#4l z4C?;(i%dYdqZePqX_dZc@i+F?gmv*8cXGtjjF~dd*1z0*`{yBVXO&km zp@lffWeN8lIf`SUm%j`TuPxuZ0=2)>q6OaKiJv+EaS@B<&PXAAvt(ZMwbOZh(sp&BDc>m{7w0 znCq2%4?5{D8iTFvQSI8`tU-yh%q!Q0jxqd4xKEFtqm_>Uee_wn7IeA&Q}YHG7z-45 zFEH10KcjqoCiatSfB4Ap!`7VWE^QRJ?zvcntcwrb05K4z{RSD(lv+F^ zcF5?De53HtNB-L1cWzj9alq{*^480c^ME@a-+mLs>%uwF@s4H`!^d0^(~7Q$xx^B} zOQL{-h*=T>Us+!Nged2)r_!0P|8hb}+IZLZ2VeW#`>>cU$m=-wNg3ojw-4ZzRJf(2 z^uFR0E5nKs*}g)PYM+@P_5ONmum-jzqDRNYe`?<|gAjL;zaf^GF#*9e6 zD8JM@OJUOPxBtg8NU|}w<19$MRCxUxcpZP)4Y;`3UOZu@GmAUihzrnJ^`>BHEw-*$ zgY@?{?NRS15V+%V7|~PKbKCJ&zqpelnN@z_j)k(UrQi}Y%aqJY<(Na{Xj0S&ms8IS zIZ|1f0QCmVst{?a2PGl)&i+75?XK0SqgXkVaO@=vgE?GeXmahG`Y9BcWudd&ixg$j zi63gh=`1#vR=e3soZc?ns4e5FMpGO zOeudvT17H2qmh02s)tXUX9CRH9?0w?RXTAzjX9dNxVXk^)ft-?OU^t;?!_KyZ$RZ^ z=+TWI#1Xs?zHRgcHTvRu>(zTThPi-12tL+SFS#4>fl^oW_V+fC2I%XxK~HUS(!7ui>-pJ3_{ze3d$36-n+8eX+* z5?#T2-C5X@XPe7ezJ>_RA_j92_65uG*Me;$K-{?1;*`pH2VC~{0 zuFAlIf8BIAdyL85JOfkOK)+~K@$@hkEk^U)wyra~GZ#YS6Ht(KrF%S?)WMs(Cwl(P zYHEtY)!69=6PPy4A62vm76o1iUd?9H$+Dj&Kh_+VksZ-MO-NTs(j_bb0T+c0H2F8XzmClw5VawYHe&nidZx0D%gv1dbf{gf}bW|k<9PlO=f%mGq>-G z&sv`Q!?i8-&4!xopdwEQyO^aDIo%tVK~q9HveC4Z^EH3V$WAcDjxdo?En@br>f&M? zK;_H&W(Jya9vUD8*4(Ks{*X!)14xA)pzl8~K3w|%{Z{N9`r@`qdQgy! zvp|3Yb>G1Koin$GmN`>pvBsJ}40{+|bo(mBw58Clf58SEG*G8Z#Fn0m0*|-?9{hER zkz?(b7^#qc%-DWpj#Ru2{{kwKZB+TME58v3vhrQ5#FCfAzE68HJ&wd+?vePqW72QH zf!yLw1uc-mvEt6>#iu!d%eG;^^DpGg6Gi+b()OeV?arY|4AeM=_OLOc&kG+V2)tgP zl+hnSIPNi8DW${OY8nffQ$DoQLC(-we>bhmH_VpHiD#%r6tAflj)W{z273$twnD%| zxc8@{z-z2DLxbWyWy+8Te|d75Sow}|wxl;+72jk!id6joBx0DIe(nJjA(t;sNm=i# zC}YrSY%?GnXHkCS!4az&QnP37szEi z!u!~AK@o8cmdGQGgWZp^vJnNB&;KIIS^g#`u^=4BCCh)=#FU<0mDFGp_17xU5(Kg= zP3i108f{99V!e-IgPLTd;jvl5bxPnG*@?nzEh%`ZD|4qQiq_Sdy5B$k( zw-sEbdy?Sw%8_CS#X|{*ElhDckQkLgYmO-uwBr-l9ZDfsIa-4wvDp{~-w9!)bUwoQ z^~V{aD%FJ|1<87H5t6K4BevZs$xuWT*m)DkFjd)6LW~=w(A`UQu>|8^iiLrcW-uOp z2y4`v;R|FW)tSOX;c4duv9g<(a8xL>zc9DLLrxUuO~9jgcxHlACo6QFeE90FasRX3 zm`;Wbq!e59APQts1=Ngv*7K1%JfU3va=x3%#YN@F3h)-}@1-E_K5-`S!gDivZOkXV zYLr_sys$11J!Je4EP?dk#{d&}FRqh}|MQoqMfGZVZFHpI4Ojis^*`g=N} zAkQnAv1pJS*Im<;fQ`3{^$-i^dxg=*UlRm=OGz`1`}0_?laDPNh%_Z2!l1UGf)u8V z{vMAg5cRU4rz6Freb)oa$k;(c995l?4_<{IK0-Jbx~#9FSD3@W#-WZ*r=;JvxVIIk zub+vH9o0N5O302Go?#eF1#zef5_@UVVTan5B!Zw{J}vNOw3?R4PN3ymEdCRE#;o|F=W zY2Tuh0V0w@uRK#;0*}v!-~S@sMfXD=DS>yWtEm}Wim4*mB0oWZ@)ebxUSyt$gKfYC zcu*`NfCyzPqaVj2oYi2_u#@bNfdjC%Es95hDyqP-gk&=I<|7Gbj!|^zNtK-~pjTY+ zU%+~L@m$@}P=Q zb_9iEarlp`jpz)poTo%W(J>a>1goVJymqW)_6#n{BG|$V4$DB7Awv=~cQ=q!{G*?V zrV{XaiK8HKi#u@w;1=6PuEkP=r#dI#9lc_w{Y{K=fr1tPnF`XFK;`IdgHX1$SP6oc zPcAeIZIJi|L`}2S1p%-9eJ6a-?viPkQBpBsfU^a$9p-H8SOM&?19fz|65xpY8%VHf z`^)V!o(}}gtDtA(fp+Fbkx<@%+KKOPqG6jlPimwrw?ajolcf+14GuGJT7cOgd=@5Q zW$%}VOvDn*mJBG5 zpjyzSA{zX`3Dj|oUW(yb$T^ss*S#{{4pctNo1iiXel%$T646JBbz$MUokTjKbWSXj z)U0kXfkxi*e6n9h_OS$4OF)#Zg+sH#rUIxG0H#oPSn0)$vu8>c!nrs`C=#ef790@Vsw}_ z5gxI#pf{`m%+7K(7dBq@kj;{c-M*jAVGBL3V5RH#sPnba%P%d+jHjExX%O$gCr=ECJi{bakvQVgT%k z0==2QzVoIuC6h4mq~g)0tf8RsyiNQk`7nfdk}aY!aT_o-JJ6OiA6hKakT<^tTEFIXH* z9R`MY=ar=hnLki1srG4#Zslx=@xaBf=&u+@=A5Son=AA>rZG})7v+vc-;e|zDpD5P zZ{jWsHDJr@_(|QdMWoEjY@fRfuAoKnL3`bQp0X3Hy0gyZur+2p?y3BO`&`n@)*cXd z8Yr6{CS%KY4#%v343?hH?^W>|%z%fa?@rjv!3KI=Scaxa?n$Jb7fmR<&ORJCAZiAz zU@>cmgtET!_c(Ox$`@gvFkDrnktrj(gh;U35O!p1WLxl_D*@+V)!KnR_)q6!Wwqae zGJIxBS+;?&TA6OiTsisGC`>a68~^QOp$UcNR*6(|0%Ag}nZH?IXHBC^5q@2QeQ}H2 zGU?bY@XJCqX!!J56k8Hj#D$HDr38Gj$9Pl$+^Yq(u_7sDWxB@*Nzx0ZQ`nInH+IAq zKEP`=W1jCC8?W6Uuwm<%U3?Uwb z>P09T!P|=+{DlIEKvpSRVkE2&l61Dlm$IMWEsVIs*YHb^)#E?zzLUwzI+V-M{Wr|V z304ns*mW=&S(|wAW#FO--a_&o-6ZLI7yW+5hE+9EO&>!+7M@f+wM;Tf`tDrDX9Mnd za$#HH-7aIHaen)2m{hFO_;_e5&5K_=7oOn*u|qm(9HZGM30Ap1Kpu-j&YXC^12O~Q zfY3SdB?QD0WVbI-ZWriQ%0V-*{tNFaPc1qn6}QL&3CyC%;@5{0Z84M^6s6b^I(&2# zF(WLbOU$Ur!hjd+yeg&Kv*le9Ry`IOa9lj?y-X8Z;sp#PsPIpK6&s@-OZaj_f_l{X z#|_4ZWC6gdbMCCcj#Co$JU@lk*{T9azJ}1uwO`HnTP#ViO1^?uYtPorWGu{}!7GJL zxTis|l)emr>?A@3($ozec50Ntn!N32oRorb=lw#Gu4L=m2Dk~O_#mD2gE>ovoe6NI z6k8*tE0D?|3!K?{1V$c~a5?bIHxYlU7eG90tL#L@<6R8+JX^?PJ&6=@#CSLW&pByW z$|@+!8DRo1v8|~+3er^g2?K})DNti-k+IIi&`!eE{pXIH=iZwaDpf%gq}jXi-eTy+ zhfBg9P(p@>b3^E@U)5875czGC(itC)g1=V5Pz2jQ-sQZ?rTu-XdmS|(x5 zMmY*gBCl-?4xTbQ3`#+guV_jHtJWZ70<(Gf`~~P96L|dF8|S&=wyW?$eK10OSIhLO zPw;knQ&{ck>Dcn|Qr?gy(`q~Z0!Kn2K1Jp$R78Yr;VO?khfYggwZF^p*dU}u$-C16 zgrpSLQ!yo*Z*@F=Rv`Ux6yzV848f`>h{q7wThXD4Z$-l4v+!Y98K!egN4b=*^RWX@ z@ov0^jM7q?`BQB9;1@KVcOlDzzd)29l;O^1z)fBG!9cKEdiTFiVzAy#t8h%%og^d) zE#D}Vfgbv@DAfbodTrh-j7$&jDit0?OZo*xK5}W6Wb zqZ)>8hw110G&B*mLru-vaBArhZC&mTD-Y`+K{5O#5yH)7cmY(~~F;|Bo*% z_e*M+z&%vlK%~jl!JMzi-t38w?>*0x7fNy}q^X}~rRukKzTk^qXLxV~ZFn%=+oPaI z?dlI-G@#vrk8D>AoC&4Iy6(R+@={Ml`(ClpfFXYxH~+3BGx*I1`S>40&;WQvn-I4l zf0ZIk2det>^yv)`0IpgX|CPQtSb6l~z@4J~53a1hJdYF;A?bT~8Ze)*$t8vZ3}m0h zonVChtA-tkr-X`K!tehm-)w=e?PIV3t2du-mpa>_J`B9dU>((hcUSu3qsO(9;kz3HU#b06N%#G8fp56`z#l%?)!u zw#p?EhrlNl|G5j)M6W)7at!d2dN2_41r1{}?XQxeQwhKA`9Te5gulzM-y_%*lLttb zz@inDrY)d}Y1CY9z{5vZ_g#q9aPNyN2GnY#!?tF7%~WkeyAb~TE0hku?B$t!*JIN- z9iaWnS<;a1!as5n_U3@jlNv6H2S0e3V?q{|!0YD^8b4Rh8d;&vsPAT3MEnree7}CAQ&)v|hjbH*bq}lK5zmo>X!_L5A;s(g zbN7J4YMcImoU=VUC`lC5^2Sgq{<|%>E*?&)Qr+W3D3n*)Q&h@=sl}JHPwZ|>s9!;7 z#!FfQ<|bh``i%n5u(#Z^6bO9FHrn{2=hL{{C-BF&%MabZF3k)}>7LA)D3MlC|#|?z#&`Jo_;-Z@)&ukGH?LS&)9Tb#l~K5mz}Njk5|o!OgYr zv#ie;P^tMTQ{wvW`|;|l{buewdbKtA5&W+BQy%irzPZPAQM&o;6yGo`jyaW4j^{q1 ze3auaffGev^RCSJ%y0SsoANDf*z?S(f8t_F7i(r%E9NfFD5*a?RB#cifi$-WQ|mYA z`F)c5I`D1d?SRv#wAo63r348~*2siV*Y+%CllX=`Hc%4k*~pqd6B;e?Z&C0n8~M(# zwM^wGy44GWPW)C~x>w=JC2RNgKkCq>Gu!DVrT$(C*Mv)?HPWBKSmh7$`4N@cBB)y+ zcr7Zh`LgoI;f))aKO8Bnf!f;zumlA^lmX`N*}y9Xh|qDGA}^)X=BBOKbJc}$=#!DN zn>)!p_g80~$y=?ukv=K)rpa*}M!Wp^O#)7zt z^r10}t{CLE{f1VN1@ka9E6KxPx5d&8udAff)h9Nxukp)ya=oy@`p=~t@71bEw)-_~ z*OlcPB?oMjUt`189UrvTl?hOI6g}!;p4G+rA=~lM<;EpfSy6R-6x@I5CR8pJ{hUS= zB|{(F8UjCHbaifl(@;qOq-H`qBN9MKGc3fNm(1yl??v??7Txoq zzXfw0h-bOx&HYT^ct;2_1G`#RTu5cP<7clTs;)Y^EYrQJ1U{NCoS-tk{+{VNKbG^& z^+eJY>nXbibWj8T_R@g%PM_4J(7u=TqoPl8;Ty=i?swLs0uOBbhYeb>9x2Ybc=COI z?&w)Vo+?M%sfI1iQr%*LdUGvw>&TsrLd>lOm$le6=hJ0@C_ykO9C*qmL0wZ&Q{4%0Uue|-}@a1rCOPTYe99OxKb z8Bubqvsdzo8q-IlL(nTI>5(QnY}INF4@7%+2u&_P;*-f1 zL7!3ovs^MkhqwQHqNjO^IGlv0Jbl9d^vmkpu+`4}fh-wa>vlZz_GzMy;la1w z99lU6H4V6~6BoQ?h_cl$0iz=2vI+>+b;Nf2uHGgGHq(K3;RIA=ry?F?0{d)U^j`^h#c88|t*+hq-kBgUC5v(v8qc%2)gj175kC@BP{U(Gu zNloHbR_2|4Zq+v94D(<1*YK;Zi>;EvJv+sQ!YhT5kANWu0 zP8khd8ge-F>*d^og`dp&@x7?B*(btAuf8hbw1QG?xphfm*^H#fHRPYufXBHu{}9|y z1*NsxTz;n8){Y{F5UY&^(Q)P;x4v>KtL-qxF6TNl7iTwVHUIIr>Xe#oo7xn4KGSFR zg`HrTMCXu6q&3OL7x}&YFy(4C^U#&5^(X9Mq2CE2K{GSLKCtp>wkEqE#@`n9EQc|9 zX3f20?a8*9S==~tcSq}5PFigz(WI%Xc~x%1Z)xwBZesQ7pD%^AEc2#5T^`{X$Udh> zVa5+qHa5gBd3D6sWIadHBDch=ui8#63qbjKqi6Gkx~@fDk-<@;(it(X4iw=pJprm?re7(Kw>n~D@I?v zZqGvP5P)Uq5N*8=Ed_mPW^S5$!mFfUbN6fJwV;`WXB$cuOEZ7UeOhW*CVnNkuozK# zV5@gX>Fp$bY&$#trm*Ly#@40sscEzSO5eP5HShfT0UAcx=w;Marw2QW)QF0k8SyvH zww;B|u`~(Vm=C!*7e_2)gC3n44jtLiJq-H!i8%Mm_p&;u*2c>Vgls%)5^2h!G4nOv z>=j>a3G`Us>LY)W@UwWojCUG-aB<0qt%kl^pL&z&1-0jtww8(vpJffV1SN|7UIwo2 zbM%i*v++-#(TaJ$K3(lyom>mlR<#VSO!q?KepyZ`$Q#qb-KFe0tV&`}xR~4KNK;~kMNeVL7C{1)|-pK)Is(yMPv7=QOL zJ$K;a-GSPGlg|(Noe6A?bEtcsJ@A&@f&9AX>zR;eaX;FYTQ=T{DrROTY%RTlJCtNI zg>L-E3!jVT^!_0%ly}@;08TsijguZ)yj-1sJA2c1rMowB02vV5$37RbmbP&(W9;(t z5xXlEJc_v12ib~QNzI*PwG3*i8Ooudu>I{ZkzYx_D@BG95O_{oQPb^DbCGP&foMk; zceErmD|O$l-D!A-tCXv~H_@JFU-pJSbhqWuc#tF4#_QW+XLJ8lU|WcHGU4-N-Z!6C z!rJ*WgsY?sh+w7tOpA03xTS;}w&r98I4Gl+~9l%m4kenPeVe9&#dtcJGYQK6Z+Da(VU# z9hr&vCe^{81X*Z~d;<_66bzrF^D`nw4Kbm~Jv#0hP@x)!ZBN2}y>v{LSS z$@w>@qzt1E?CBOm+-}TqDs1b}%wm*A`8VFe-%hf;7K{cL3|_XW*?M+Z+c?}yUo0%} zN7pj_)W|jjpNw-Y=5M(&DZN5jG{&FeCk|%xL9Alhy-hzZYOW8A98lKajqSVLN-8D4 zSFBR)CTyqlZ|x1)%tvW`S;;9s@7wY!+C7Iy0zSJw4*VDH;j!LpVuLP7g8V}4LOT^& zVZT-ov0t)l0`48Xf_#o_g%R&>5!2A_F4e=79rrsbWfh6p*UDkdhmW|^kC5JfL}qdY zEs#aXX^5HIo*YVGGB14I5;;*EZEeX*tkQ1Y-kWEAS_Q$HO z>+Z9uX4%~~ra1zc!Q2SD8ciUZ|4CZEE1KBU%Lip1=0Q?T;@fokme91EQH~ zUYMa@>D2M&u9lT2yi13+!+)LJt)Nyapd4CXwpWMT8)^KGwLMk1vHCge({5}(m?DB1 z8q24~-ibKyzxVX%MX2mAwF8r1R1&&n4vwAeZ184vO#k%J_P1*XiS@N5k z6V4&V{e4ewl+7$08k6p~dn9a^y8ZLt{^;0MDL7x4)p_NKC4zdQ;B`4Ldv<&DQ>0^f zpb~lL=C8Z2VF#BRlC}+HwPnx-L=%hkD{r)0F+Q{5SsS(Bkbc@0v{Ijh&AQLYEVKj0 zkvDAj+MTMO+_EFS$5qR2Jgb9s%;X?U3+ijdMcV>8tOY&rUDmV+zhj7Dte8(VMfTmN z;a~aG|E`AJo$!JIeCJqK3>)+rP&Rsl3KmFJtUWozjZO;y`fVS19eN*Z$BBS z4V$ln0&&dnB2G5)58>7Y4silETlnw_9$!du)oqpS?yD>A)e_F5a;&3o-4lO~+D=7} z$ZKwyWCom$R@>7Kiu)URT*^qn9fp=!H>ChLE|#g)X5V}+_<$rw^EZ6d@bbi3TrU65 zy%O)stbd=-9_h;i5$!f4E0bY6LgT&5v0K%OwM{dZ|9$&i{XV3D(|>Tel@=Zi$B-)g zg#&&!+w;gg2=T~(?HABqzi8$66MOQkt1Y!XW)-_bH&`^QB2ph#`>f2vRx^4pA4d5c zZ{zfTFea&N0Ho)9cep53$Wxq@@KPag`h{xu)*&I}6PI56`?AxUf&X6f!a=6adI$&M zi!VQXUlj(zbic~)9uchJH_XJU$wlp7niXz|dBN>vH2$t_PMV~JucyEM9fP14MObZ3 zTjFQi0qSxkD&+<1>70oVD2q3Dl%e`IIkC$l9UVXa!ONOax;d)qy!wnBTFi!53UEzK zVMnpj7RK4^l;qdOd{BJEM}iG?4ztVh-L!$+n;Kd>CYSAyKLS59;WgEO?~#FfeQ_!5 z`>Ztout)NKN^5mrb~q6uT{h#9?wJYIt3?hU;fSD#~O=x?G=E!J;I9at8*YPC#FbM2>0x$NjWOY=<;1>jG61u0dZ)xQGFrcI#XnPTR=gWNz{X47 zIz$~FF&V81QLg36sH_5L-@Ny%JkI%|>P41MC;ZF(imtpB$CWCS4g-jLN4cyrOd%Rsi4WRB0FHXT``ACO*1P1Wt)um#)3( zHvcMIinIlLpS(dBC78by*FcC4u8Cf+ch(nQ6pt8u2*ZBL@0(I-tGDRfuiV8aqf({d z>+;QAS6SQ&MH4{@)i$lC3_|_a;bHy#pgZ%yfegM8Prvsfz3_&PhZ@HAj*j)_A<80U z4bx7Aqp2sJ*ddOJD>4m77eC;L2IuKj>G`WKFM1a4s3U^^$~ZK%w3GdU*3)mhp^6#@ zw-yiu^;{I+QD$+GZ_d1&59pI~EQncYt&3zY2CP(d=?T?Ku1=?<_z209<}NFGzhPC6 z!~n+$Lr3h-{5M%eQfg~TXN;;rq?EA5i6+0qpA(jI8zEcClYDVWpZqW%5R{`A_yCRT zIAAd&N*|KFb!c`6_k#E<)qf+=!*%zFl3hZIDbrLT^C)=GBk61}0|-Mg9}` z_sq6iYi)x!>(!Off2y(J5hgeo;!c%~5+1y&a4>tHm3gSk5a(Y5} z|EV_othqxaJ?msdrQZA@_WaC_$^Fn>u{(8^gym!HDNjMEiq#6CPa*{8iiMCzW$*B_hheG&pR_?FmH-%MD^R`609wrsXLMohxpTyYVDu8d!t ztu|FHt?m70`R=P zGFV@xemJ((-2o9w|DFZ~ErAn{|0%`G0N&VL#I^zfzFF`-X>)g9L~11zlMw-qNkKO{F??}9YX~NcG7#R7%2u5$1?b{q2Z}p$>z7e`l zeEewyAEX^HT9C4NRk1E?ac&x5_Y!d$dti&BdiK*SKEoGmlk`iaFY1RU%pCW$M-$P#c3|!sU{QX$6dk# zCxk+^W7ueo`Sj~k`~WBhTpft?gR@3_A*hMW1^G;SAZDiKzK6cO`JOIJRfrq0c)7ot z`*{HOy$lIZ`JL^$Ru98Dm9bz-+`rKC|JVTB2lo4LPNta6f3L6mp_vX~!0Ljd-f?#N zZ}lkL@RuXS1kW(&W?Uq*;m@H#pT$-uxcOz)nw?eh(m%E(KHzw)H#)K%{g!w3+Oxz> zmSXsplEynwqqZ||7^@jfbkfbEU#I;3qhA{g4sL8J4|fm*&(-Y(4~rAr;H!T&e_R;o5d0CN3J zdmDJ&keCXCpn-uOGg?6pIa>!y6Ta|5)xkC}v^Q?fuRe31?)84oYGB1x42a|y#r*58 z{oh-wEOKZ=RGMDlnv+zI2oDqpeM?DQj!*mmWnH;)1I|F4+R&#(WdZ zKL7zMvRTy5x0Za+YIEmb$sa*%6>VSPj-Hp(D(e}EDpoqu@->{Dn?4QDwEYXls~dSN z7WSd}SaCRi^+^Vy!IS#$EzbJqX`ubv!-pO=+fs!=aEAF)6_O_Zp>lI6h~{Ts^kcyU zDx=;%=y6Nn+ZI_O(6b1M=eM^|Q}CatggIH4FnY{?fZz1LT=6VG-PskXC#W=`zhB|> zy3I0LPk*#-fID8N^fb#Tf@X(eaKfASjmX!^5Vk$^F2LWRAtb)WEXc4NQ+dtwEn7i$9$lAgGW)ohYQJcb5vYHyL; zs8#03L$p`V7-?9-uPuIO!9=y&>(QM{gXByjwM|SScJcEj_dLD9{1fg6O#NVW17IHY z*NZTfXQdnR!RwnaK1grWx5dMm;s3REZU(E7SEZN4a&iucBIm3T^pMx%nuH`(7g_&P z%CZ&=Tzje9==N&KjFt>GL_4Ys9fT=}la6I{nO`OpGsv1E?NG~snNt2Pl zt0)hy(O2|OaOSh!mZELCK7ca~24ZeplDdZs4|_dfPe{t@IUVb$mAw0xYtb4;`be70 zE7=kvoW*@JM)*UUNc)7-+kvIvABi6ghVzJ`5pk2vx21O$#ri7%{GWR?rWzSIp{&#N zAe6=22j_4GZX47V9frUXcB~}Q-W6&4vRN<^eKA0x7}NlcToQ$Oc*(Or+vmvd-bfRT zP66M?7Z}n{Fq>r>N8r48hhm|`c=dp&(OZ7-3d=U~Lx(tQW!K2kqW3Pz?{k@iu{sZ_ zPKmt!e^QaSFxdMIn>{fAXT#Bd@4iGDC0!X151{SG;eT5&`Zy2;Zt_&wEUX}$fHka$ z=_Gq@Hme%~&!wI*kdX#_tq-t-tO8Vf6~iO)CmQZ>5&=X_Nbpmf4q*HfEaA?&wu|qr ztWL2kSpcGH&$+qaKw+d5;c^4#_Q_YzGNp+g^}xjqK}x#8c~}Z3e@njIk5GiUTZ9b^ zXk;`(+Cqfh7a|ggl?pLg#O0Tq+C*f|!Bj6UGo&9f7EK3V`h~Ad7 zyZ~p9lwsvEWZlMo`=vn$Q|{;=g>lqjYOjHSR5}u-6L@iN<0wUV;wjv|el0Tq%DYKH z7bw>nB`zj?&*7aykC;I}o;6jJs}lhh{z5aV-K*0eOp>#XiC@A=$|1~@(Z?`S zs@l%m|FE)lyp%AgjC6bWfncRK8niboM}{D)WfW}_VKTZm^EjtBQ<9t0o>&@8;yVS< zxI=Nauv#pd#7Gj!(vl^N>%U-bq(NQ6iCUWK|5%5s!jQnx@#4IvWkm4YFzRS+Ni#4^ z!lCpqJ^$kvI$S~Nptd;$<@UL{%`p^b{=&}pVF#1+TwiAPMf=J;NXFez0gkmCrbv!5%RWt&IA|FG@|tm-YnJYk5o^aE0z zg6y;URd2Dx#WLdODHTW(CL;m~l8T&XaQ_>z%MY|e;Cgr4aVJOk!RvKgpSWVWkJca1 z(5<=~qdorZZMufWfB(jTD=N&}C1BX96#on|+(jE@|BbN`tTKW;z(o-&9 z+^!9y;3r>=;nkT8qdZf7z!Zz(`jiE(S8!}p(3eY?3yCZ%235-V3n0USh#3AEtO zK0SK)1mR5HUd-Jhsh|6XUSk08LBd(apN|@GoZBcV2rEUoWw1!Jkhh@s{f2EwAG!uz z1)U~40Yd*IQ>+_q6;WTeJHvIC1yCnr#?42?PZ0WzZh;lxO;4^HuV*BI?@6Ouq$P0q zVAH;?|C{1zss^Rozy_a`qI9#+u15xpxSLITW>bX{O@OJO&~dmj z-~nm#fLxn(c;6s_Q4u4;APl37_33heI_oQ;P>F!QMOmBhS;1*1Vg6_g@Rppr@6Z(p zfEj=7kakGn&NZ3)*jYT1mK92Ej>{qT_O9{*Q2(_LxFs|_rSYGq-d!Tu$m-S=W(UM* zm;d1D=v#~r>(>y^BPe?$F+hn~qXaGmuo|nxI9^cO?c9fD#0B_!uxzUY_<;?~##T%A zoGr+0(GMZ-XTsN+O~#nxq)3Z!WyLL0^8c9W=ju^Ph_*uQlB-4_t#U4OxlsbqDjFe%-!bg|MLzs|cX&~IS0oK2XmLdq+AtE_jj@rjs z(C@ec&&ya4A`tHs?=xo3eKOYb&n`HDR5~sP*vK$;YfP^ZQP6D)#-RL#(lUZz#gqyv zA&L6o7L}#C1-7o3g7HE&V69D;KUAA){l99n;>SZKqwjY$rdr@L`DZ?&z|!k11xqo8 z>FzxTZ#VOmp&pu&+Z+Rvi4TqfLgh!0=0KjsALra`xKON|l@lo(lPBkdxrdNpz~w5Y zQNyex8D0}4``SYg%UJ=Cv2D0dcsb>K>uNh0`~2|so)AjN*h3^ ztPt4dp_)@dzRiL=CUWtM*s9ur>;fjScn6G>YE=%eljpS|*xC0(X;FkhLlMi3YSEo zNz5pim(G)MP)XqVc#~I(k$^mns@gg&h{D(Y$NdV}zU4xosL?_RUXeD=>QqNcHl%A# z`_jo;k~x@$7DYI)>u_9UG4XQbFRYko2J;b8GK{&Kz~}oV&Hq6=2Ga%ZsQ;Bhh`5NY zeP?J<6bpXOOpL-+XmC*)@|y~E4O#+g*QRA;BUt^l1D^J7fqP(kCwWJUbmM%b6aKWn zXh0ox!g&i%STu@q)MIUdEAUPOjlYy6;W7Bm7O>>7=_D_RhpI9m2+3t^FPVLh5ATe} zS?p)J0-pZC}MUQi&3D1L_J zo)zAz)du7=Xkx^*+qCKZbI(ACFC7{ehU+qfs3%&ijjST9NvbvPR_{@^II0=iJMLMC z0{-vk0D}(&0z4sXIfsIZb#(}t>cC5JMd}VR;90^f{m6B|LyDQ16wT9tQIgDJZ6aLk z_uxcolNb_THyLy;^7PDvDqKYP{0Q0&)hl$bi6tF_6>(DXXQ%o>>EJ%dM`Tw8-n?Z& z380i@%-M3DWTT}-d;xp*aN%v1S6u{hoglC-eh#l4P)peWqq)10ku)hOco8z~Um|dT zoH@O+msQ;2I7`m?Q5hOXOZ4at)*?<&bNP*@L7jBlo7d#;dt!>5=)n8=XL+ zJ9GY#ZDC4)0iQ&*aN3n#MM_!wSGpgnxhB)1N)S*xh8~8CvXkMuh4}j#JV~#sk+h@s z#woO)s0ZdWRp7bweY`+`Hla`S)r;MQx8*MnHPBRuEEkaD zClX-U`;#OBIYRRH)FmPVZ6m&?F_rS}A@i785;fKP9))lkxmrq8&F_nRgWP3R*3xtS z+9)vrgLVomK|I|k;-QupP2-{XF-(c;BJ(b|$}_)BWceewduL?xMHL6&^?0}+uWiW> zS0iyj`mxQ-Z>bv=_fKr3$2#s$p!L{Cnf~&o0w()jEM!DQjn~-%Bh%eDk~=Cv#{7`! zhZk^pJuqZ=Y<5qGDoF7?&TWm5iBwNP;FF7vprdf}bG$xLj3|5249wqRUDNV+ycED+ z>09|~9ITCPI9gkBn{+JPxcUf_cpiO~ONTua?JvZ`1be4aYeW9RcCzTUUQHX3*%HHI zPRqIhlLew#Fz(0%CQ*~obMS_=^z~l4X~{pt=`E!=vvY{5;M|xxPlClVx zA(ofYKQ-MVi3tBQ0F_0%oWH3yOh8@KuE3=dY=J3pM4q=_fATGR|J4WlmLaigg|8!j{ z>qR%Ns~0brFT5{mBJdyr{X@Vaa87~XMw+iEJ+V^Cg_rEu^&17M2 zDAC!vkJba*NT}c}yt`3KTT$(^wfmkmDyM!{0~am)?kwBm_v4bG z1jCYzT>%$3ON0+5a2d?Ch%I0%xlz*`G%D!`Mb$fqqE#L5j!`{DR`7d6V{J-~w z)8Cs53jg`w&3XQ@ck(lH#g%ed)tpwH0T(R47)_0EvzUJ)x_~Xv(qg7XuYt`?QMWw> zC(dtPXZ7XriO$+5xA)GQ@x+`5xQK00&wgPGhm!lR)zyC0#q6H`YEkIEgC1wqofwZ? zj}*T+b3^TmiXjarylyh zIvH43UvcTZ>aOQs1^wUO?l@Z!-D&!(*Tw%Ymw~|-qd%W3Wc|JcIRibr<;pj4k{2*RS%D(bD-Zopm^ZuO~ zdc@U&`N*w5a!LX`Jbx`JPI!x6f4TI^pZb)Y5{=e{22+3sx2Tr5MwFx^mZVxG7o`Fz z1|tJQOI-s)T?4ZaLlY|_11lqAZ39Cq1A}>|_K2fs$jwj5OsmAL;r(=$)j$p5RUr{2 zL5bxG1x5L3nK`KnC6xuK3Yi5Z$qWn?a~^-<;V2B#&^YCP`i$q(AO>b-ZoOn~VP#?O z$s)|c3N8&Mhf|o9H-{*kzH#NmkuyhRj5!vplEw&_(gntm%NkloSz)UWaiENy*qdA_ucOX0yq*srWG=6h>pZz#D59Ixx^6SPGTf6j(ClD zhj=drZ_EGlJn=i?4&rKJ5OKWx?hbN3+;<$Nh|6RWJ&6Itb;M)DBw`McLu@4qh*F}K zXdqM)GzhiwJN6P=|K+^37^84w*uq5^j&`bm%b2Vy$0hWLr7Nm5obsgW-Z zU&~bDS>hVvIDdIsKyXMLlHdU1Na9KvDtu1tAgUddH&OT+w-NJX=yCCKI_tr3(a(M+Nb{jL&Z+`pOp;?e0N|+Y(0*!{num$G&wQkhrxW%4Ye(-XuMD zTgdJHG4X-KV3|=^=kPe#$)U+w;uYdNq7%m4?weNYEq~K!bBQ7!N*s$IF`F1Jvo{c1 zbCLP2e>Abur@e79Ly{bsH}5O=4jySFG6y5Z5F3bk2%9kV@=`oT?jgi>c#M3kG{(jh zf*PP@fQRql)5Osjir6nQN6a3UX)1_pZja5un+J$PA-44*^Bek5Vu^%{A)$3+?5`x` zj+_#U#(&9Y5L1XU2xl9dU-B>FRE)39>y^6`k4i8Ek~u>j@epwkdh7I}d%q;IFa@Vg zfQB;w2cox5zE|!ldBQ(JB<cWYxxM z2;&-7Na`(8R`-*+3-P!l(nAz?u~%N&JNsK|V}FRjsf}-8h~fGn$V8%-KZhWO%>8AR z2%>%7h%)k8%b)v8YC~quQ(grTJV%bKg!h5eM%3?G;s-RoXW>MrtGqp5ZOmlwTE1Oq zbkD;vgd4rN^vhUb{cBl{j|TS#1+sF;LsExj&XDL*5RL3Rcq;ZTZ%A$AWQ@StK1l6u zsecWb??k`^$nayBJ6{xysks}7d^9Eyc1j5|7ftCIGG8K@S!hVGe{S{p?vUD$d4Lo* zK?8zVNlpFrmpXib$H;m#G#qG+&m@mWYni*rx<51+NQ8e%rlq!X&Cg-E6i9Me*Asf> zQyb&vVVG+MDX>q2z>}K6O97WWNK5H1c$Tdp7nU~ zaA04xrqRws5{;q8rLv|B4Fb$ymn{6Sc|09HA+H}>ghl`+^Mw@Sv_0u)EY<=7i`FJS zZO_od7;0QV?7__k7Or*~WS7wVCOR889azjXYomxp?|d(6XaJjPecWl?3!a~_TVa%hzET8tY8G>5kX zJTMWVc}uP!ZWz#(S_r%Dram{F)keScS>-rQ29 zCj9`ci zE)GIN=oCZYBO--1>N|f3p&_)(m>h_xFC7;82qG6kLui+=n&=r#=!=Om2o0fKMlo?- zG@(a3d0`M5LZ`ulhS0{k!G8}RG=z33?-=V1!$Na@=_&{fpg3m`Ov zb~#z0L?!yiQBFn(AvAq6HyD7C4qUac*zQ|n4JhJUVaJ^fi(*8riD zUuW2#g&snb`a$TwZLU&B&Mi{iX9d+E8p79g*zBNsVr_*gt%cA@!u!l$#zNm6&|oK6 zJBA!p^-8_6u~HpG=hCh*>3TA!7pjrpRj8tx1`Iiphj*eK-nitd!7pQI?)Z4c2KR*8 zG$>t1=TCy_`Sq2mtbeWnW9Za#qaS0TC+KiNcD@wWlQSRfs16N39B))Q|G_hZ>UA0p z@e>3ug=z5$3;mJbgl;^t@@rJT`3_1==jA8GJ6o$&ZG){J-K#=R4oH0i<=1Ju#RawM zv@eS7Z(KUxQS^kE`eSwIiGgW5QaX5(NQK7S0=uHqhd~7YRSJ(fq)Ve|} zB+$8Em8exkb%+#NLkQEnhVXj4;3ay**HPyr45y;Yl@-r_A?6A!>h|x(Bx2d4R!Pj_SUMsX?#gc5qfHx znD5ah&j`&y5-XB=Se}X0_?$eHa3e@4`)TUo%?@Q((|`4FRPn3@CF+N=dPjxsJFh4- z-FZ%u@Xzo}PsR+>JzB{^uknJ=(|6W{Dwy84vOF{fHuGR7-``%X4xf`iRFVF^8-13S zH&&V+$}Xo18J}CJI?W7 zLw^FZ!y8tle)1RO7Yj>TeffmY5%4m?dsyh3i7HPE-T2Zr$C{TmGmnL_UOFLkJh-;` zbu9E?+4IfYLQ5kYD${B;nn}%rjno`bQ=) z&2K_8oF2-_Lxt7ONL@*T`4>}3euB`^_#m_!mXy@p)S145fYkQmfl=%by^i*{V-W{X%?1g(*JI|+E58H;cs^!{);=);=} z-6@o*f3_>8|whKX_%vfCVO(8Z$v?rw&e@yRj=6=mg$Ai&3U%T-xf#sKVC&tK=H-AEi_R_uS*?&A*Es`?o;oMph9$ zTNAo1@fL)Rlm9^s&n{EDrGHcc_t;G(25hP?4WVTXVR$FxK=dOl_02LB0-@vNe{kvD zoM2t3xJ`zQ=Z4TBDv4`j%N%GC2G51iiB%QlX}t`a&kLbLtd${hT%mc`@g{^$qF~`P z2F}IoKAEEhWlsD|mO5uJv z4PW!bysFR+x+#4AZNy25%WMdp5!#v4#)jgG_bhfQWX|x3%+00^JIn~Ce7~HgJzaZ> z9nc%0#jr^UZAkqq8GnE)05qVOxF~si$wK9XA|U z2`@4^(K?x1z92s0#4;`58>)%>%zY7zKaH1#8xJfDad)#`=9azm`9wZ$J}?sN+LtX?>heiMwrRD;9d(c=Q4!emf^^`ese)<6!=%e94G)iP5p^d&Wx0IR(_~Vi9 z0b=B7jFGx80e2k6f`ExcJsJ@R994eZnuL9Oq&}2*4}a5nKpc46xk#XSQuCC|R5YmX zWqx13>ABQrK90hAM98f7j7IWOA1%oz7*j)L&q&=%J|Lv_#|z-Uy(e>|)IDUOZ7mwz z(^GSQZ+ZdroXnA8^YWS(k)h|=%(=0|f!-E5QtA%Gqr^T);2HK3_v@88XRp*;iTy?* z4>GtOPJc=pBB>7ne;&rTdJyq70x+(f$yg|vmI43D9DS~uBS|RR>mK+WXcBP*WR5O1 z$M_#6c0v|clpEk;Fr6TB{Pfcch((yM))H=$1;n}TNdMHQx9%-ddxell8-60jc{f;M zz1SJDs$mIwYsaI0yo1A~)>Z@cS8x4Xz{Z_zL4OlFh$o4|&?}oiXU#RwJ_>}lDS&L& zQYyLZXZm|?*`Aq@8)e=a`5@*~Bg2d9n`ST|vu#OEycl;S#@D8ela~{BOU*)v?2yb=kzFD-=i^>1iRO7eACB;6-xvt*DmA#Ni~vM-?R4L9^0G5aa(4ZY)yJQa zw+!L^N=6WUG5zMfUb~|d)gCEBl6;TNkkm`fxViEfPm|A#-q;6n^Bn7`#683mV!cc2 z@$ogUBi@yIas4rW48b`+p^uEYe@}cYl79p>>%r?KS&OGAKa}apizPJxITko#_TCv% za)Cqbw-YY}!kegPOKwB4#Zk&ynT_Cc@~07*qoM6N<$f^~^q8vpj-rCh zfDA5!GC*|36@?f?AOdbMhz_ebj0QwR77+ym+#MJc9R)u`V1OhfWFa95Az_k`&Ytet zx~iUkoT|J^r;iV?9@~M=f}bS-nA>jp$OD6AdH}ve#03#@1TY?$1WW^_ z0*3)(5RqFE!6QHerJYkd_iqM~Hx6_IJAf^~CSX0V4tO2djJyGQ02RFtRA?v^xD}E2 zD4+wF2^;~81Am4BjfO_p1MC5|0;_>nfF-~RMDjSGf>X~zi^O3>)|tSGz&n5=ku;*I z?C17|6e9R4;054WMD(pdre5W)FQG-^CM11$7jPOP@gyW=2r_YC1MocXXGHK#K)Rmf zu6?0J;t+5sa5C^g;3QxIlJnLgJ-|lbZ@}+>XONscTYu|v*QU@;PCNv78*mnII+72Y z%7aaj0$v0D0Q?qM1jK-v%(-hzXs4_;7I+`uDhxxmMOW2nDqTr54nGT>q0*GTSd*Miu=7TOVcD)4dO6G*K%WO`@;A&aCV59J=~ z2)Nk66o1;uk*5Hk%sr+B_IS)=18af%fd_y$fIy}q!4kj`c{Gw%d=9A_2kSWeOBPrO z+yy*>OvMLE>|luCh#Up}8Mquc0ht(4+@}YrV}BR;Jpt+|@xRbco;(})5>kR|H9f34 zp$qsGun?Jr^nHHqU!k3-+zxyO_yRJcr+CkLB!61pkCgs=8?F5*v@^T+PT=dv6t$M% z%#uQC*Vh9t;?rpDFQJ{NJQSFRq!{B(4@)_Kjlj2o`;e5xSJB!(LOU`a30wo5jsH8| z>Xjt$3nW^v0emC1KZF*U8;}ad4Zv}xholt665wj2PsA53tLB9kncIMifiEM=n3Ukq z34eSGsa8Qh#c1gmzX~e;W7>veH_qZN>xN2hPPqsjDxv z97rSZc|_)so*bFt2Sy^Zil4?KsjGWY==5ZqpF1#J+gG3L0KNkJgetE*ub$A3%;zF9 z>x4;tvmNP}e1s}CHdRMx=c8pVl1GiH%IhdWV>6O^J_=N&r02fSj?DiA+zU*s(tpWO zg2{T|Jm6`-b*bGI+L5^fX$C*0$|pw&CQFbD#a94VrLIPmV=^+Gtuib9aXeD-a=q8x zJ)tE!4@25N|IwopqXd`Jf$M-aH_LhM2<_CGFGsfDQj}u&Byb7P;Fi>t3+)W#!@yR&TywdRavXv*c275WQj}nG4C2u6 zN~3i7LOb1ammq5a6{Ql+1TLV$mecZtc3$v3$XdW!-3edujz;7fOe$n$N?l3HF&0_z zJIeH#P-=#8h*P7>eS*@3cAAbZM($anC~k5p6{H@e3GKYV$@^baQQRd2Tz`mc16S4` zm&$OYx3%n#1w|Qz@ko8TZ&(6f;(2o%LDEJWy{gXrKwJ6jN3K#AdB zQCwgOk{J}WYP{G8JqOvZS$|PX@L`JFWouxeo#=cXy3kcI#{`Pp+oPBW-GS^yrzj?w zOA#C4_b;?F#r+91IxFUxfZYCiz$x*;BJ>RMs|ggvJm(;{!0wm6g?8Qn=OC-N6~zxs z0saN(w=O+UgdRtJ>QPbrfrV5&`rYtzAfZo1_tsGS$1DceCgWhCC4ZDRBk6|rY$$#s zg3O6D_LX+@7om>^-fep1iqa3KAO(wqI8bQE_z$3^I>o<8X-Bx%FAYV6 z_MAuvE}0r(dRvIBZGY$#)d>w2Jk^=zp-m~%L#dSa0N(?;^WW$#^mGi>o*P3JXCB(f z$%lk1e==oIGB!N4F%|T1eH<_qSeE}r?;QF>^yWTFa2O4oQ1}fI+S%XvB-0~QloGIz zJ!1CnFI^a+$DsG#R)WWD3VJ&Vg`R=*oGMC?m`uS|&V@qH0e@Oek4{laWGE7)0cVX? z7}*0sQG!SZIhXShdN^>D=}{?4nH){-b|`tFCs2iZ#wm(NOau-EN9bu7obFbX3Ta1{ zXYChyItR}hDoRjjB3J1MA;swBK=XT2a)g3h79&@j^A zG2HZMs!iI44S%sL$&8KV=MFO)LKaMphKMxyd7!@I5MX2&iOzK~L-@qj9&TQfpgUvx zvTM=?!WPV%+{{%oBBqBEG=>1PlF2tQD96Mxq+q|_x$!f%^~>0vq77LGya?zA#n(N|ovZ{;>0V7P0!W8cx)5s4yTA4Mx z!F#D~n12`zamVZ^=T2-wQ)X8fk)K$pTTUJs=0|f{dDkIf@1(Y2dRqhcbVNC4d=nv+ z*)7^a=!T4R3~%7RIZ@s_rm^}`+c2x(_0(lo8B+HVI?s%@2JY^Na?bcFN@l}JBg6dU zZLOTB+ci`JD@2`??22JhG{hapM>%&=6Ae{({(tw4ZRAIDqRbjrjb~TL^>Qc$jA;(B zaAuT?rZ&^$Qm76AbH_JvPe+vLLsaG}(xgK7L3@+MjmNa`l^GFQDy$2KE%@}LChnLW z<**h#-PNXM&Tv=SCZ2KcimokearKM{*B%vNXxJ+2VzD^{7f)?w;c-z$H@laa_hjM8 z&VMwWNyD>3c7+f3$=1x3$um3BJiR^5RWsb{gNQU(TsAF2TcgF*E8^@(7CBX(o65fY zh-R*w;qL5TDhqeMp5%cyQvCcKt&DQ|frrZzKwax*<88r=Y_*q7m~=epSD?x##93?vMm7FS7S z;jZ;b_V(D^JhO#y5qE5Au;8PIHPO~+ap}?+>%01q+J;fhl7F8!x!Kj(zdb!R-+x@4 z;I_315*e_3P{EEiSY>}24h>samW9i?M>0gg-nxOuf(-D}O(`y1665u5_dd@o;Dciu zxwoU0V}|uHRc^!hXo%ZpN15lc%-d5oS1ymU@U;X9cR!+Qjjj-|*Yqgd#s(hU+{0%V zce83=#?|LOWmK5^=0rK6yxEl z1ne+9imG72liSjK=A{@f?Q&P@w~r91PZ=#z&m7jky>nVQV{D_Vud}{8%Y{o~JiIA| z2IivJ5dyZE9!1r#;Dw!O&Rf*Y)7#UoKK0Q<8@Q*Vl`lK`8N0JVZrh}8O~qS&Es3$m1A0Kh?|dX;e-+H%KXBvH0Ldf z@zl0-9WA}6m@Kd*1Z)KYfA^RHv0z6QhGfaL(`sV_k8P0)*%f;-Ds$;{1Div@W?-M`(RhLd z+mkk5ekGo(=la9yj13RvN{#*2xRt@!2fP&mb^)EHN8=e5>`mKzePx21Rww8(Tnp>5 z;qKRyT)H$i_$;I{5L-f*AiU0yo5dx zr9a*T_J)xQ!u6&{V}g_ox380_@|%urVT^~%uDjAUH?2-^>)Ir-jLPf+>&W$T0IW7W z8dGF!`0*Pl_NC!FGh3J#t-jQ|Qa1mwD$ZT&lNkDNs5NZhRdD*DU!z_w4Es47zkD;r z1xsSAc6qA2H*Y1gTz|GK#t&3xH`#+k=>0-(pq?%a`@(`hZ0%v*;uuSIS4Zkiu`Hio z661l5DecQ}m$!g7z!AC=SZjJz#0wc)fooOz4 zImY5${+j*k7~Vj~@S0eG-UBSoe^Gczznpr{qYJU%^<4YrLz_~RGpYEeZ9Sa7$bT}| z7+Zmr`7a8E-hW6Pm+$(-f-P~U!M@l9oi^~B%{|Ot9OKo!Dsz=t0c^^DQ7H62;Ca)d z^BorKNZMSnJkG+ki9xqY$k_0}#uOJViLs$ur?NfZc?uSKi_ml1^Ity$#7&RTcUZ75 zZFB9)1UI~zU|&Bqw^SB>us+GZ2oIaf~!{Kex@>e!CK&z!f*5zdJFKZ=~4Qblnr;TPjcC^Togv4_cf~$ z-0*5#qPNQI70=RZ385pjgzPNvC!`dj^g{-CaAS&#UyiY8cZMsM$GJtf81R||@JGNd zNSXVrIe$Ngbl=S|Jz_s&1Hawe!*e?`Y)@u=w*saPSO>h=`0Qy*kk55QNIy^f+VNr0S?_ zW&zD1!+S(mK-Pv-)-I=|`flJNAXB{1keB)cz|VmprcZ#PAUoxr4LsSG%rN*pR-Pe0 zSAVW3=J^YQG|&wu^giHzAZhwYl)5JlJOIS{i_S2R(9Rz{fwq+@=6M?U-@$&Vm`x~l z0{5a(S}{i)DNyVg>=y?X+KJMSqfuHhhZCI#NH?JP8*lCe?n0--74L~5rMf|*GZZ8A zfl>NN(?_S&9Zvv%00xcDP;7*jC)x{q4}aKg`UsV}VLNa;&|QqoQ2M>Tp8|ei`Y4sU z;bGvfrTjthLd!cOg+%FfrjJyqc~%2=0O?|*Gn5{oK^7(6jxN(usz(aAjocQwrBIsC z&O7CP;LoOyTB%u{0DcM-bs<ZiIH$9^Qd0C07(TcpPc7C`)E2Kjp}?1Gom=ZAqz|wZL`2u5x9D zN`!V^^abFX=(M;}KHb0#6zua{E|o=SN9qTGALFN+6evN#MtU_K0m@D_^0>1iO_pmN zij?zCGk9{8;PE@)eBiB0WroUg=zl^s16Lq7a8Z8uC&nf&WB`K#$upLpAr#l*-3QBvOB@s(-Jg1OXd~ z%3q?w$&-Pou4E$h7^LT8ZdG4P@iR73XTBVl%5(kUe#&t$ZvkIK=INFC<3O3KOm&5J z-iaH5i;-Uk` z+oM*hMJI4AvJYL-Ynj1hZCIpkK+1VH0aJ{PMR9{QNN>htK-PrJV0yAcqy~5wa5Hd{ z@i8f-Vj~-oe+6ky^DJNP3vVt8`@=c#=YUTjGfBE_Z*Yhs&H3lz1Am#pFVWD6)UCj0 zk&muJO%F>MkWIk1kRTK@Epi@z2%Sg74gn`43nt%<=}!@*H(4ab_y%%=R@Xb`D#2eu zLt(TY2V4fs2S%74mIBBs|2vWW4Lz+d7mNRd&g0xG@pPoL_f9lgn<9g3EOaBX18qqg zD0+#1g@)eI`g6bqcz<+fTV+3Q0zW`%%d)i2r!s@zh0f#Dtly4w?wtWdO%JQa=|Yy% zEd-Weu(n(Z!4MjHS3=G}rp@0;U-v6lDxn9t*t-qc1vCcuIw}_u9HH|#Vvhp;4VVWU zhvwInk^zO;_eekeKIfyI1H(C2Z?SZ@-9Yh^8P$>AIU(~kKLmF zgzk+99!4%Ej|AS9dmN3lunxsP5j=}r)NMw#qFt1GtbarK zshmte8m8KT;fU-lNR_3(s!9gwm52d*f!)Z=+!mx6XjAU-CK7q~Aa&!qOCd_-@8sP3 k^au2iW|L(RfdY7(ljspX5b{Dbl>h($07*qoM6N<$g2cdk2mk;8 diff --git a/core/nginx/static/favicon-16x16.png b/core/nginx/static/favicon-16x16.png index ff35d4b0a7618b3b4df63d27c6d0793ced5735d1..6affc1f31df4cb428670cc8f6411d827946f7333 100644 GIT binary patch delta 212 zcmV;_04x8}2b>4690Gr)NklXFs!|adOsc*@i;xX6Q%u6(r0Rw>o2$_?F13nN|pfafd O0000xLS delta 266 zcmV+l0rmcz2hs z!O;mQAjTjL5^#2Lb#wP{GV=72010^e`1<(=1O^3%1cpj7NEv2YhJ{B&Mn%WO#>K_P zOEbtACL|^$r=%LCrDtShX2~+h$!ELf0BttR&CAa(P*7x0QZ_V#I11zx6;%dCHFbnT zHPnE9VN}!9f^fBz)fmD4)z;C~)7RJ2HP8lf0bMOF-=5jEFq6vyfdY7(lTHIZ5b{Db Ql>h($07*qoM6N<$f-bOhmjD0& diff --git a/core/nginx/static/favicon-32x32.png b/core/nginx/static/favicon-32x32.png index 2b6b749b92a5d0f19c470f28ae0cc8eb70efec23..3b9362452f4349cd82f468dfb034fdca9ccd6560 100644 GIT binary patch delta 514 zcmV+d0{#8-3(E_z9|M2VNkl+W-RJ5UNjlUU1igt&amq`y(V;VI9?p%>#>npwU5xI~-fY#yA|=s6`!x6+&IxdmIL5p%q0X1{uLEH% zpoR8nxUt|oQhp8aB~sE3M}9bQD3F9wI~+Oht#YUuMbY2QF#JP99LIoMflQaQ-lM#< z)|FEQ;$kh(YEpl8IWnCR-(68F1JX9BiWNHGvJ;s~Sj=@Vl?kK8Yf@~07*qoM6N<$ Eg1{5@=>Px# delta 555 zcmV+`0@VG>3-b%G9|M2-NklF2fFfxiu#A5*x;t&;#jFM9ESb+o= zSOufBOgs)C%m-A!C@UwAOM$$CEF%M>qEb9gAfU|1AgdCO2dIC_GN`G87zv4}APK5K zLz6*E8%QUoq^2Q*)btF90&N`zT@am_nuQ9op$g*l81z9@4oE?6UOosE6e25-VlaR! zC@L-~EiEgrs61c3s0oRfV8J`nOkHI)DW002ovPDHLkV1l>b>3;wK diff --git a/core/nginx/static/mstile-150x150.png b/core/nginx/static/mstile-150x150.png index 04c609f70489bd0c6579c3fb4723a217eb24168a..71c4702d19e1aff3ba8befca74cbfcff6431648e 100644 GIT binary patch delta 3618 zcmb_fX*d+@*C#zvi6>cWP>g+-7)wmHF-FRkJ^SvVVUVq%$TH1f#u`F{gk;}$#$?|r zLTC)5$1)ki$Wm|h<^SRR@V~D2dhbu?zV6?B{eI^@=iKKy^ISN{zg~+1gj={=EU0S<)k$Zlz!_-xuv~1aFSK(Z2`Wz+7G^oivRjxGP zn2Skm=q0}RZGJl*SSOYXL~9TA1kxK+=kV!iSeRHsB=rK=f9T~CvJAFj3zB5D@7>)G z+{0#IYp^o@Nr+{>Gvrr_`x#L%4qV8V_z@uVJ4Y5Kc{U4-{eTfiU204!(gG7+qdy!2TmcrXucvTi9G6Ol@&vU%`Mgv$h;set+YG z`MH;)l{HRFp)SjXsW81RC4ql*#x2-$O6+KgOYMbsVJoKxE3hgasUcJQ_~9LPAh)>> zy~&SPGS+!A-i@MMc$ZD$*51&ElpK1IM+3QyT^`t=LYMxCtN`ZbY!-Rp9c76ToWLrT zr_e^FLDAuR_9N+MxOR8+8AG#Uz78TaoYMH?E zgfTCsUcwZ}mZpsZ1=^C{WH1Ay(c0M8CX4M5zFEPMn7H#LMMvIRW5s9#x+(`UV{^1y ze*xvzT#Up(ujH=35TCiOc#$#%O6gE>ZF@zuWxv~Bqe3^H`D!vX>zQ24$HD`bzThYj z!D-z(@FRJy&Bitv!~;RL`KFz{OkLkn9Czs!;V7B3Fz+o!h!Ykazr|?r5%(|)kJ6$- z73PcB9msVm^s#SLC4u~5*FhI{XL1F|Q1NWOBa%N5< z&f<#u1eOzDsvI#}LF)`ic*{thPTSOR@ITSFJpFy12-KEom@?d4 ztPyed0bSJz^+mkI5R{+EwW9=Xqi3pz0s~-3cFCj}&42w#1}V`3?)_tRYmGI|5|Ch& zX$DAAz?5JY*&oJP0VptODNH{f1ldCNXVvPJ#(YU*w@eldAh8J~$U(jL0^CjF zzx-*7Q&$@E6tQig;NuH*hp%s4s`6YG^@i&=HF+Bml-%rb-+w%o4&Fg7NbhciPf|p} z^LqlVb2+7-<7@z}8V4M|X^0kZ1y#&opqP<`arX$o4Fb{uVNzV>PFLaBNV z1n`{5f52u35Q8glln7sWkq({R$aw$ilGaPc(Fdv3wGVP=s5Y@kV7wbLL9$ubg_qT* z6VG3{JmP{I?o;4IhIisht+HDRRK!lmWu6=?!naUvEi!~6`g9+86up)$buBYNYuxs_ z%qZ3}_*ztjYx{h}DQ1-08yKNxI&bUtn!dU&n}ZPT;5_BtbDoURcv?v1&i2Ha)3O*j zdENv&SqPigP}<`qnzzDapr5hdRD3-~&tmAen%RGS;m9)1RU?<>yv<=eo;(*;KAL%m zD5ffu5^CaOQ~J3RmI8J55?v;arKxNpI7Y+FfpYN5)Mbm0v!o3G>e0;wE)Tf-;fI=^ zOQZK#wYe++b&YB{Ddi`E%>Grd+@7G*lQn3@`-bL`h3`vt7OTOw>yHI*d%pkmoY|OC z5Pf3}i7!ph6?O8{bPRDsEzOHD{cuZ>p{H4a?mCv?4&$oKNN|op(m0aeII@U;XibU z7SD==y_?oH))U^%)7qcNM~&S#)rZTv|IR)Wlp}nWC?tpc0#iM*|M7-$6**A(TJ6g>7r^)6f4b3fCd1dHTxo%3V~tbQaY*g?*MfnVBK2PV{bFg7 zG;zdIFbamu65ugpIm$2K_KzU)jVU!WU$H&iY|bOp^7~rhH_K-$sQLfe>}B`OP`U4rT^-4jy^kHzcHG9w*15 z%TOyeZFeQplS&y6JJz=TRmC@{PULD6TUGoIJMRTl4J~^mD_aklr8IH+&QrrOnz}g? zn?Z9h!-NRTwOk~(IyTEKb!2hl3A-=3Tx%@dKiHEC_c3tK#&SP|RVBy@5D6Vn-pL!k zwAgqS%rU&8H7H@C?N#?dGw3w!!TTNVOB0{8#wJ;5Rzgdu{;pgaSr@Pt0^RSd;fO?f zrwp)sffB9BO0LmD;0hQy&&J%7iWOM+;F2jtSPgNHO(fo)<$hm`fkQGrmmDtcxrx`z z4)f?1FUbQkB|cp6Br_Bk|{QArk2ZN4Xz1($=}$6G6Kx?b&VAWc}6sKIMh6kgLp z_gq7ek^s3B`Ni!|1|Ano*76LfJCRLc7e$sIgs+pT6M(l({(U#?{8rF4jgQMrs8#cj ze@K)z#lOmPBej}#a9!A5`)z-h>GH8P+`zv z2m_5BW0&RD4M(IqT(cb}Gx9%#zA-t&%6;w)5Ku#6a1fDd#K|PZ7p*>2lF*6uK1bGV8~vGix{^3C?}f==FIJ3Mh>BA; zpq+{IQqKAqvmWUP6mSWY>gf~!W6tY$^=@tblV)wXUwp#*QoGdpt$Y)zsZ-)BL4h+7 zl7y(%tn^FL60vnwI0d5=NlxuIP+&sHMh|#)7@8x-MmULHFYNVn_O0rx`SX24dW(0B z3S>N}uCJOz4p3yfqq&=(7?&Qj86gx>@nS5%8K_V-qb=W0hveVr^j(!&1fX|W&Eawm z)7%=PH&6CkSZ68QhGze5sQLexfBqZC|2nTa(y3Kl8RXgE`3tvlG4>Q&4$hNH`|o>& R%CCQ8p$4YMh$uz2QXxWRG#R@wC@~DmzE-1RXfpQQw-JLx z2$OXzMcK)&dgu2(@89oV?|Ywn|2XH|=bn4+`J8*sz4ek+@+q-6euoR^&)El$E>DD{ zU3b9K*H;%KE<6>&$LiV%u(hY~p~lfEC+N!yh~dP}7iuvx%+8r;;=;MLg%>kVPD?=A zljQOOtUP-?$2$y;RUEy*MG%md=J(KZGkr#xZ2XZtW`IyDd0o0cT>3Y%w*<3+YB(5i z7;0@;Z8=y0UwC#B#~S}vy*>RS?1hiLB{HfxDgqDwH>bs$#ofc^18(9M5KaqZ+9 zq?~`ZtIIegwXQNTjl50Av> z;)eh!mJSaSb22YMZt680MGL$ko2)dmI(k4AP-E;VP3POLIhH_&_FYO2jnd%@&Gyv&SZCd z-giG4Zh>#_KUN@i=Q`w2SbM&_=n5%TS6cKLuJ2s}+(HBu{-iCcmPb>)-Dao=cjY2Y_7<)t$-4x&qK871Fc30_ z@$-nwj#ncdKl;;;OEA*f1JzZR4j9Z~YCpnG4?0N?scJ zsX#@%v93$lf-pY;hl~K1o+C^%ptXL|BQ*G)5!}=#&lF)u)%mpwV)_9{MO2{gDFUgV zCcc*sh;q4ZmuPLvsa-XZ!KP#bGk2S6`C*rSbihu7mpH@YP4_bx=UnpJ8Z1)J`fDqf7wQlWy**zQM28Ld=FJwr zHQo1Rx{ON*6X7VN7*1c5hgxU?kg>=9tUiZFaWV3ooC zJ{~UXr$+P@8qlE4yreMf?DdU+G3BDkg~p>Is6ZSHOiY5C1_nv=Iy{ zGvGrQW1p%~5h@OrVe2_**U~}ng9snGL#{j4w*&owe~9u3>=ew_z|+e%yGjwN{{a5I zF*)fYk-c0Zn2kwU5H3fk>>L$C|2({1IcgAbJ7{T2J%@Uw#5uvk7=26xWo~F+a$e@X zA}+52vr{}gbVLGKpPSpgc7fc}61(}w_dW`&ailb=-7dQw_&_x)o)(#TOb}-6rxJp0 ztRL3rM4V>ruN=`0ZFFz^82$sHf{pqQnQK#*a7DXqT4>-eLnbG`-EOiaHvMtV37A6D zk5Mpz)MSQ!6QPhz2%1&>#BuApIu4W*x^U*S)*q8fNR{H3{!mrdeW;Yke@~qsoU1jd z+l$v~X?gH;=J?6MwKzk&=#UAh6tXN%ZpSzI&*F%y=yaQD*sXxHk>72@j&q~+7+221 zy=A^e3+tpCQ!dee!USz^4AA*2T`4-!hGCR%%55`B{%%X4@4ssl!-E5Gy!DjW%~f&F zMRZIz2}1CyU9Wx}(G=izm6!I$H*yK|xwor-gB4Miy(=5kRx7$-m5SmVqL6OmjNR-1Y#_K5wE)p=7$Of|Gu$x1q z{j8B26Rt&&W5mo~#`*_|Qdhc2*z7;$SsL2zZ%Q4;s{#|jVPj5TZ%#;UMz`ouoQCO3 zz2_)5OlZ53fBpT%!_fh}QYK|_&bn&c&2sQrM^`!`$c{IkXV)obEd5lY_DlP7TKB{> zv=DArV|3baZcje;0_3RP=XJTcYuw)e=Z0 zQqVXgnAcls)aFvy*6a0FS9K>odS6myXz?$YyFTd+bdPrQ=W()oBcByTZ|%<|rxVes@P#L%uUm zF?_qdxyk!>(9lR>T9mLj${@@p6%}NQ{<=9Tt~WgOR(1Nt;8;lIzYHd<|+?;TNraO}Kt# zonL{R2VS}tqb-81oJp??-aPE?tHUkn8QRUL^6~^sHpdwD7yEh4`1Fo6#Uto;lTxe& zY@zp#=P}C~2}=^{0WRrrV)^&WahJ9V;b0GWjY#xUO%udNCgj~UF_5@BODz%69d>Lx z_=q+IQUlA}?QD(--^Sj{Zv=Rf!!L7e`uDTb-R=x0`GvcQ)WhyJxo_B@SMLp~nPu)yI0t zVO{esn9*9IA>1`vH`ec3g9m4c6}!n8qT8fQf_XD@8|P3hx2LI>{BU?8rfkAs^2-3? z!IW*wzk$CAl>5pyE9ZyL#>9@eumGICr@n@wP4raC<8x~g1G;a3ueVesdprz_?6L-u z^%Ebiyg7-dA5qw)51CV(O=v5D?lH6SEo)4^ z`6>O*?bH*lVq;zVacw#eiKK@+Y`h>%0^Ad4CBQ)Fd@%4!Sdvg(?~Q5LJ&ZBC-Vj#5 z-=oG8bpXXUlcVH_R^KtoYo8cezhv?E<>Y%?{ZnekYZglevc5%LFs?PiJ}oOpkTjD?EUh~7QMoJ>jXE;d=IrEeyz+X|kR$g`$HVBUR+p{vQg>fyot^jvt zGQ0XbBl7(?xHz=3z{%=0+PC)ERHbQTsSb-q#oLVac zzHRM#En;oTA2;QiSyeI&lhxqpOGFs^sb zp#tF>#5pZ!{q2!aIV(+)^wXB_(!dEB<-c{B{4UC=ryL&_E`?Ml4bX zn|5O59CMY$mvQlyBiCX#4_sw)ZyDa^leUhh1GD{RWAe0uB1mOC(g4!BC z(>h@UIRZ_lhk&nT^XDLfLfz@)FMMc|=_eEVpw=yc-6blbLM@M$Z=J;%qP-?Y6WCs1 z^cQ$#mBVHgZgHSh_r`}AFYS>8l7rBI9uq`!Ntunx6ttaR`TYAmUje9wUyr*5UYClF zSxK71%u)+gZ1UUt6KFU~Pfg|?U+ZL`>cUAumbiOl@>Z5PMhmy$9nb@}CH4Y{9$1{qocds6 z{dhT4$wlaY|6joU@8bU_>6MH>Qls+dSSvsF<+|s0VR-i7yyuHBW4;Yj-E Date: Thu, 2 Sep 2021 20:46:56 +0200 Subject: [PATCH 397/596] pre-compress assets (*.ico for now) --- core/nginx/Dockerfile | 2 ++ 1 file changed, 2 insertions(+) diff --git a/core/nginx/Dockerfile b/core/nginx/Dockerfile index 1906ed31..8b5be4f8 100644 --- a/core/nginx/Dockerfile +++ b/core/nginx/Dockerfile @@ -16,6 +16,8 @@ COPY conf /conf COPY static /static COPY *.py / +RUN gzip -k9 /static/*.ico + EXPOSE 80/tcp 443/tcp 110/tcp 143/tcp 465/tcp 587/tcp 993/tcp 995/tcp 25/tcp 10025/tcp 10143/tcp VOLUME ["/certs"] VOLUME ["/overrides"] From f4e7ce099052486d64eafdfb112f4a4f4a68b358 Mon Sep 17 00:00:00 2001 From: Alexander Graf Date: Thu, 2 Sep 2021 20:48:44 +0200 Subject: [PATCH 398/596] enabled caching, gzip and robots.txt --- core/nginx/conf/nginx.conf | 25 +++++++++++++++++++++++++ 1 file changed, 25 insertions(+) diff --git a/core/nginx/conf/nginx.conf b/core/nginx/conf/nginx.conf index 9ce12980..616ddcc1 100644 --- a/core/nginx/conf/nginx.conf +++ b/core/nginx/conf/nginx.conf @@ -33,6 +33,17 @@ http { default $http_x_forwarded_proto; '' $scheme; } + map $uri $expires { + default off; + ~*\.(ico|css|js|gif|jpeg|jpg|png|woff|ttf|otf|svg|woff2|eot)$ 97d; + } + + # compression + gzip on; + gzip_static on; + gzip_types text/plain text/css application/xml application/javascript + gzip_min_length 1024; + # TODO: figure out how to server pre-compressed assets from admin container {% if KUBERNETES_INGRESS != 'true' and TLS_FLAVOR in [ 'letsencrypt', 'cert' ] %} # Enable the proxy for certbot if the flavor is letsencrypt and not on kubernetes @@ -46,10 +57,16 @@ http { proxy_pass http://127.0.0.1:8008; } {% endif %} + # robots.txt + location = /robots.txt { + add_header Content-Type text/plain; + return 200 "User-agent: *\nDisallow: /\n"; + } # redirect to https location / { return 301 https://$host$request_uri; } + } {% endif %} @@ -95,6 +112,8 @@ http { proxy_hide_header X-XSS-Protection; proxy_hide_header X-Powered-By; + expires $expires; + add_header X-Frame-Options 'SAMEORIGIN'; add_header X-Content-Type-Options 'nosniff'; add_header X-Permitted-Cross-Domain-Policies 'none'; @@ -114,6 +133,12 @@ http { } {% else %} + # robots.txt + location = /robots.txt { + add_header Content-Type text/plain; + return 200 "User-agent: *\nDisallow: /\n"; + } + include /overrides/*.conf; # Actual logic From 34df8b316832b43a2962c74ff7fd53c5ae725745 Mon Sep 17 00:00:00 2001 From: Alexander Graf Date: Thu, 2 Sep 2021 22:49:36 +0200 Subject: [PATCH 399/596] AdminLTE3 optimizations & compression and caching - fixed copy of qemu-arm-static for alpine - added 'set -eu' safeguard - silenced npm update notification - added color to webpack call - changed Admin-LTE default blue (core/admin/Dockerfile) - AdminLTE 3 style tweaks (core/admin/assets/app.css) (core/admin/mailu/ui/templates/base.html) (core/admin/mailu/ui/templates/sidebar.html) - localized datatables (core/admin/Dockerfile) (core/admin/assets/app.js) (core/admin/package.json) - moved external javascript code to vendor.js (core/admin/assets/app.js) (core/admin/assets/vendor.js) (core/admin/webpack.config.js) - added mailu logo (core/admin/assets/app.js) (core/admin/assets/app.css) (core/admin/assets/mailu.png) - moved all inline javascript to app.js (core/admin/assets/app.js) (core/admin/mailu/ui/templates/domain/create.html) (core/admin/mailu/ui/templates/user/create.html) - added iframe display of rspamd page (core/admin/assets/app.js) (core/admin/mailu/ui/views/base.py) (core/admin/mailu/ui/templates/sidebar.html) (core/admin/mailu/ui/templates/antispam.html) - updated language-selector to display full language names and use post (core/admin/assets/app.js) (core/admin/mailu/__init__.py) (core/admin/mailu/utils.py) (core/admin/mailu/ui/views/languages.py) - added fieldset to group and en/disable input fields (core/admin/assets/app.js) (core/admin/mailu/ui/templates/macros.html) (core/admin/mailu/ui/templates/user/settings.html) (core/admin/mailu/ui/templates/user/reply.html) - added clipboard copy buttons (core/admin/assets/app.js) (core/admin/assets/vendor.js) (core/admin/mailu/ui/templates/macros.html) (core/admin/mailu/ui/templates/domain/details.html) - cleaned external javascript imports (core/admin/assets/vendor.js) - pre-split first hostname for further use (core/admin/mailu/__init__.py) (core/admin/mailu/models.py) (core/admin/mailu/ui/templates/client.html) (core/admin/mailu/ui/templates/domain/signup.html) - cache dns_* properties of domain object (immutable during runtime) (core/admin/mailu/models.py) (core/admin/mailu/ui/templates/domain/details.html) - fixed and splitted dns_dkim property of domain object (space missing) - added autoconfig and tlsa properties to domain object (core/admin/mailu/models.py) - suppressed extra vertical spacing in jinja2 templates - improved accessibility for screen reader (core/admin/mailu/ui/templates/**.html) - deleted unused/broken /user/forward route (core/admin/mailu/ui/templates/user/forward.html) (core/admin/mailu/ui/views/users.py) - updated gunicorn to 20.1.0 to get rid of buffering error at startup (core/admin/requirements-prod.txt) - switched webpack to production mode (core/admin/webpack.config.js) - added css and javascript minimization - added pre-compression of assets (gzip) (core/admin/webpack.config.js) (core/admin/package.json) - removed obsolte dependencies - switched from node-sass to dart-sass (core/admin/package.json) - changed startup cleaning message from error to info (core/admin/mailu/utils.py) - move client config to "my account" section when logged in (core/admin/mailu/ui/templates/sidebar.html) --- core/admin/Dockerfile | 35 +-- core/admin/assets/app.css | 54 +++-- core/admin/assets/app.js | 73 +++++- core/admin/assets/mailu.png | Bin 0 -> 4935 bytes core/admin/assets/vendor.js | 40 ++-- core/admin/mailu/__init__.py | 16 +- core/admin/mailu/configuration.py | 1 + core/admin/mailu/models.py | 47 +++- core/admin/mailu/ui/forms.py | 2 +- .../mailu/ui/templates/admin/create.html | 14 +- core/admin/mailu/ui/templates/admin/list.html | 22 +- .../mailu/ui/templates/alias/create.html | 18 +- core/admin/mailu/ui/templates/alias/edit.html | 10 +- core/admin/mailu/ui/templates/alias/list.html | 26 +-- .../ui/templates/alternative/create.html | 10 +- .../mailu/ui/templates/alternative/list.html | 26 +-- .../mailu/ui/templates/announcement.html | 14 +- core/admin/mailu/ui/templates/antispam.html | 15 ++ core/admin/mailu/ui/templates/base.html | 70 +++--- core/admin/mailu/ui/templates/client.html | 38 ++-- core/admin/mailu/ui/templates/confirm.html | 18 +- .../mailu/ui/templates/docker-error.html | 16 +- .../mailu/ui/templates/domain/create.html | 19 +- .../mailu/ui/templates/domain/details.html | 71 +++--- .../admin/mailu/ui/templates/domain/edit.html | 10 +- .../admin/mailu/ui/templates/domain/list.html | 34 +-- .../mailu/ui/templates/domain/signup.html | 26 +-- .../mailu/ui/templates/fetch/create.html | 26 +-- core/admin/mailu/ui/templates/fetch/edit.html | 10 +- core/admin/mailu/ui/templates/fetch/list.html | 26 +-- core/admin/mailu/ui/templates/form.html | 10 +- core/admin/mailu/ui/templates/login.html | 10 +- core/admin/mailu/ui/templates/macros.html | 141 +++++++----- .../mailu/ui/templates/manager/create.html | 18 +- .../mailu/ui/templates/manager/list.html | 26 +-- .../mailu/ui/templates/relay/create.html | 6 +- core/admin/mailu/ui/templates/relay/edit.html | 10 +- core/admin/mailu/ui/templates/relay/list.html | 26 +-- core/admin/mailu/ui/templates/sidebar.html | 210 +++++++++--------- .../mailu/ui/templates/token/create.html | 10 +- core/admin/mailu/ui/templates/token/list.html | 26 +-- .../admin/mailu/ui/templates/user/create.html | 27 ++- core/admin/mailu/ui/templates/user/edit.html | 10 +- .../mailu/ui/templates/user/forward.html | 25 --- core/admin/mailu/ui/templates/user/list.html | 28 +-- .../mailu/ui/templates/user/password.html | 10 +- core/admin/mailu/ui/templates/user/reply.html | 35 ++- .../mailu/ui/templates/user/settings.html | 46 ++-- .../admin/mailu/ui/templates/user/signup.html | 22 +- .../ui/templates/user/signup_domain.html | 22 +- core/admin/mailu/ui/templates/working.html | 6 +- core/admin/mailu/ui/views/base.py | 5 + core/admin/mailu/ui/views/languages.py | 6 +- core/admin/mailu/ui/views/users.py | 17 -- core/admin/mailu/utils.py | 13 +- core/admin/package.json | 17 +- core/admin/requirements-prod.txt | 2 +- core/admin/webpack.config.js | 101 +++++---- 58 files changed, 912 insertions(+), 760 deletions(-) create mode 100644 core/admin/assets/mailu.png create mode 100644 core/admin/mailu/ui/templates/antispam.html delete mode 100644 core/admin/mailu/ui/templates/user/forward.html diff --git a/core/admin/Dockerfile b/core/admin/Dockerfile index fa75e8dc..edb8c1fd 100644 --- a/core/admin/Dockerfile +++ b/core/admin/Dockerfile @@ -3,33 +3,40 @@ ARG DISTRO=alpine:3.14 ARG ARCH="" FROM ${ARCH}node:16 as assets -COPY --from=balenalib/rpi-alpine:3.14 /usr/bin/qemu-arm-static /usr/bin/qemu-arm-static COPY package.json ./ -RUN npm install +RUN set -eu \ + && npm config set update-notifier false \ + && npm install --no-fund -COPY ./webpack.config.js ./ -COPY ./assets ./assets -RUN mkdir static \ - && ./node_modules/.bin/webpack-cli +COPY webpack.config.js ./ +COPY assets ./assets +RUN set -eu \ + && sed -i 's/#007bff/#367fa9/' node_modules/admin-lte/build/scss/_bootstrap-variables.scss \ + && for l in ca da de:de_de en:en-gb es:es_es eu fr:fr_fr he hu is it:it_it ja nb_NO:no_nb nl:nl_nl pl pt:pt_pt ru sv:sv_se zh_CN:zh; do \ + cp node_modules/datatables.net-plugins/i18n/${l#*:}.json assets/${l%:*}.json; \ + done \ + && node_modules/.bin/webpack-cli --color # Actual application FROM $DISTRO +COPY --from=balenalib/rpi-alpine:3.14 /usr/bin/qemu-arm-static /usr/bin/qemu-arm-static + # python3 shared with most images -RUN apk add --no-cache \ - python3 py3-pip git bash \ - && pip3 install --upgrade pip +RUN set -eu \ + && apk add --no-cache python3 py3-pip git bash \ + && pip3 install --upgrade pip RUN mkdir -p /app WORKDIR /app COPY requirements-prod.txt requirements.txt -RUN apk add --no-cache openssl curl postgresql-libs mariadb-connector-c \ - && apk add --no-cache --virtual build-dep \ - openssl-dev libffi-dev python3-dev build-base postgresql-dev mariadb-connector-c-dev cargo \ - && pip3 install -r requirements.txt \ - && apk del --no-cache build-dep +RUN set -eu \ + && apk add --no-cache openssl curl postgresql-libs mariadb-connector-c \ + && apk add --no-cache --virtual build-dep openssl-dev libffi-dev python3-dev build-base postgresql-dev mariadb-connector-c-dev cargo \ + && pip3 install -r requirements.txt \ + && apk del --no-cache build-dep COPY --from=assets static ./mailu/ui/static COPY mailu ./mailu diff --git a/core/admin/assets/app.css b/core/admin/assets/app.css index 8351eed8..12df605c 100644 --- a/core/admin/assets/app.css +++ b/core/admin/assets/app.css @@ -1,23 +1,51 @@ -.select2-search--inline .select2-search__field:focus { - border: none; +/* mailu logo */ +.mailu-logo { + opacity: .8; } -.sidebar h4 { - padding-left: 5px; - padding-right: 5px; - overflow: hidden; - text-overflow: ellipsis; +/* user image */ +.div-circle { + position: relative; + width: 2.1rem; + height: 2.1rem; + opacity: .8; + background-color: white; + border-radius: 50%; +} +.div-circle > i { + display: block; + position: absolute; + top: 50%; + left: 50%; + transform: translate(-50%, -50%) } -.sidebar-collapse .sidebar h4 { - display: none !important; +/* nice round preformatted configuration display */ +.pre-config { + padding: 9px; + margin: 0; + white-space: pre-wrap; + word-wrap: anywhere; + border-radius: 4px; } -.logo a { - color: #fff; +/* fieldset */ +legend { + font-size: inherit; +} +fieldset:disabled :not(legend) label { + opacity: .5; +} +fieldset:disabled .form-control:disabled { + color: gray; } -.sidebar-toggle { - padding: unset !important; +/* fix animation for icons in menu text */ +.sidebar .nav-link p i { + transition: margin-left .3s linear,opacity .3s ease,visibility .3s ease; } +/* fix select2 text color */ +.select2-container--default .select2-selection--multiple .select2-selection__choice { + color: black; +} diff --git a/core/admin/assets/app.js b/core/admin/assets/app.js index 364f8429..dc3414f2 100644 --- a/core/admin/assets/app.js +++ b/core/admin/assets/app.js @@ -1,17 +1,70 @@ require('./app.css'); -import 'admin-lte/plugins/select2/js/select2.js'; -import 'admin-lte/plugins/datatables/jquery.dataTables.js'; -import 'admin-lte/plugins/datatables-bs4/js/dataTables.bootstrap4.js'; -import 'admin-lte/plugins/datatables-responsive/js/dataTables.responsive.js'; -import 'admin-lte/plugins/datatables-responsive/js/responsive.bootstrap4.js'; +import logo from './mailu.png'; +import modules from "./*.json"; -jQuery("document").ready(function() { - jQuery(".mailselect").select2({ +// TODO: conditionally (or lazy) load select2 and dataTable +$('document').ready(function() { + + // intercept anchors with data-clicked attribute and open alternate location instead + $('[data-clicked]').click(function(e) { + e.preventDefault(); + window.location.href = $(this).data('clicked'); + }); + + // use post for language selection + $('#mailu-languages > a').click(function(e) { + e.preventDefault(); + $.post({ + url: $(this).attr('href'), + success: function() { + location.reload(); + }, + }); + }); + + // allow en-/disabling of inputs in fieldset with checkbox in legend + $('fieldset legend input[type=checkbox]').change(function() { + var fieldset = $(this).parents('fieldset'); + if (this.checked) { + fieldset.removeAttr('disabled'); + fieldset.find('input').not(this).removeAttr('disabled'); + } else { + fieldset.attr('disabled', ''); + fieldset.find('input').not(this).attr('disabled', ''); + } + }); + + // display of range input value + $('input[type=range]').each(function() { + var value_element = $('#'+this.id+'_value'); + if (value_element.length) { + value_element = $(value_element[0]); + var infinity = $(this).data('infinity'); + var step = $(this).attr('step'); + $(this).on('input', function() { + value_element.text((infinity && this.value == 0) ? '∞' : this.value/step); + }).trigger('input'); + } + }); + + // init select2 + $('.mailselect').select2({ tags: true, - tokenSeparators: [',', ' '] + tokenSeparators: [',', ' '], }); - jQuery(".dataTable").DataTable({ - "responsive": true, + + // init dataTable + var d = $(document.documentElement); + $('.dataTable').DataTable({ + 'responsive': true, + language: { + url: d.data('static') + d.attr('lang') + '.json', + }, }); + + // init clipboard.js + new ClipboardJS('.btn-clip'); + }); + diff --git a/core/admin/assets/mailu.png b/core/admin/assets/mailu.png new file mode 100644 index 0000000000000000000000000000000000000000..e4f5021f4665bf509f6bbbb5deedc399a7e65b1c GIT binary patch literal 4935 zcmV-N6S(Y&P)z>JG1pdd*w<^g$p_K$R4aFcZR?e2TJ@9*3H zP+ht7z32Nm=lss^oIXb+vSJX~WHX!}Q%WhNJi&+9(T6-?ed5OqC!1^ri7F(f_aTQ7 ze4mx9;v*_3r<_LIs*!Taso*14v6AmIf*ks2DVu0~CZlCacblY2ieX-E@6PW z5=@=%$K`BbKaFwp?cRgX$bL3(IsHgcm;A9lm3~a+We#Gp4)QWn=|`$M=51_0gQq!Q zP@V6(JvhMA%+T?I7uNSChuiodp*_7jo#6v+BS*W0c+4+m7he))eaRyf(_7tiYJLuX z$9vQg1=aE%C7iFW+Bg4EJ||kL<#Qe-N8Rn-{HOVXsOk%zRtP#be>g92gsAHXFECu) zKHL0Ztl}8SqGPOLn7ZC#em2i0k@;;9JWIBk--GG=g=E!Vm`;!6HUB)WVmI|9v+CK+ zn50Y=F`Q*S&D)2sGgy{nVIjSl&By9|JAKUTBxwgoA&=b=$?*Z*8iQUos?0_K#t!y6R9{ z850%zucKW3bfI#t3%~gT*`Q5-99=DQ)V@F> z0S2>8%i2(F3t)ag7knA4dKUPf`P1g#>`1`R%pgP9sby@qcKY~UG0$oFhCt8xW&Z*y zv~)wD3SS}tGI(FhHY9qV4DSJ0uH_mcE%!e9|3*tSMEb_l(Eyq3)iMp4cF@y103Oit z44rDY)hqjlNB9hnDC-MP&Nn8F_p~fSsvml=?;O zp8lcMY5qFC?B8K6!;ouV99cnD(4-|8ay7BSefsA!Ex!opGnd0bp3?dskv8BdclBTU z|ETD&i#C8#t^W~f14^CRpT!|9w+QKw!}kYSuJu1ss$sel`+MnzZ;{h_2OG^YlWHxo zh^d;H_Ur!!Ew2b_gUvR8Yqb7HQpXu)4}eFstRkwVIsi;={n^I;qTH(G6yZSdyWQEJ zuIGG3T4khn*8v>SQi`ySbaOPdE&z|b=6Brz?9&p8xOUQ`tF0gFw1gtAqYT#K< zVF5RC6F0Je`4m$`5yi}JcejLMiYcPRy4(Dh;s5*#oG}G}Z0Pt|q127Ju*dzcNDL5AZpALOh0Ga_P;aii2g6qyb`R`)|AZc0vc>inw z#?#;z02%{~Gc5*y3i3JvaI=2^JfuOOX`=wBr?4XcZ~FD5<`!T?(gQHpw1t+hwzof( zcN74>0Wj=*&3D=*gc(Vt0+3_?R5G#+fXV)C1f>8p12D%l0BXr^1K?%OLEtD-5roI7#m3qknj!0J&0>&Zh%XAA|=Oa(&)*+Qwm&+sY{ zG>QHQ!jplVVAH@xMk4fJe01{waEALs13()BkNb{5Bb&+f%zk52!2qQp0$?&1TL?VH z*FJvZll&ti5z;-f-`G@e_yA~Y1!ns;fq9a@K@idz7xh$dPyl#47chzbq#(@U&%W6I zw+O=d9@=kk5D2BhmLLG|G8J6F2vQK{alj}0hsGWEha?pYrQ-J!{FnlPM&R6a?WHaUsyiI}{+KdEWPjAO;8u0Dl&Q6f%~+2*SU*Mxc@3 zQ;3kp*f=x(p&SJS0f4`20w-DsJm+dkqmk`a?H}jcH#9mA5PkqAGl@P3!ZVH$XySb? z@!fu-Q^BSIFo}s=>>7b*DR=PAPEpE4tMPZ9ix|rJT*xE^z)Juol7`TiiLMct&*!%7 ze;7gN!5A0zUtld2$t%OG%qRjmD7A$JGdIn z2VjIF0GbhS8k1kbFa+UdKDAlywpT+?&fyb=bxID)Q6fzt^_+FO?oZ?3a!UZnuA4@ud@F2%o;s^jI ziJ%(hI9jy2B#vyTaR6M+aej>;q>$?Zfe{G8EqofYP@4GfIP4!oI>Km{QbD~#0OmRZ zU@k&7BW+w4Mh}20YI&EraUw8_G5r*#{jtA;?1lDArlb3 zV*vmMnT?Rb2p0&9p*Mo?t5yV>*lpQ=k*oi&+5WQDqHf?`-$nt|6gdLmUb6vs*bx9% zw*mm4G7BMvp)Qg`6Rd^gFH#RQ@c~m2&T}>F8$v3=XtuU1=mzfhEe}vlF=8`2nTK$H zlmM8)sdk?}#I*>*02c^Mv=I0?RqSDkRr{UWe<6Y}lD9e(vr_;{905?G0JI^{yiTWo z90=S@FNE_L?b`mue12BC#W?`>Q^DURGXU+ofR>oJ)IwmAYXpQI~V{>3rvcRmc9=Lmp#!3+Z{$ujEo@RmeS>_Yh$1fdtB0@>B4CxUQSr|%%u z`<@C8AQ4narcoP2bOk^g0;!Dj-Tqz(!}wt*wciZ@MrHz!kzv#pFp6?50p7kqTaA*fw#UjRHDqAK&o_fkLbA z&*qJ8>y*<}u%AJo>s0WIhHVsSrv(5wz?BH8WVlEUb!C4BsR$Wtu+gMC83y{D2XHwI zY~3i*n5|-CWddXCAXi#Tx-Rz2{-M@7f!A#Hs1Zm7w{8`i7d`-BKT8mVfu11HV*f%8 z*$k*6kP2=Ez?kjgo4bHhw!i-hj}Ryz13|czuWbLW2&95rwu?9CzzBOmV0#1}^PnRr zAj`4+kw^uvHSWk5yFp-^s;r43_Xre_ZQ1{&qu&+!Q9y?yV+=Vos=Gu`Yy_UPIs!L4 z`i{4DAzjEZhrcWIso)NWM&*YW0I1?|YYkw&WBZ#IjxFMdnd({ zw}^SnL;MGz{lSr@93AUC258Y4{*x;Z(rnKaHrwBabuO1fI7tPMxAr=Aa;4hbrO(ww$g1|z?G1lQV&Ss+9We|AI;n7AMA0Hn8 z@TXP;3c7iLta+i>zs0c_%E>U$&&%a7uz~R%0qE~n6yRSXD3<=nWe6!5)>}hK2f#WxTYNXJ*USxR z0icO5S=I>xh1Lt?3pg70?+P^ljTD>%?rbTzjEeHg=JCRiKD+{9-d z{VpeoAin`{oCul~8-Pna*>*PpnZV9kfo0Y^LAjRwcX>GP9by3LximHaJ^3U`0Kh5U zWMXS>?G+FI=THOiN$Wa*a}Zb_oi3o|>ARR@jr!|6{ap=40pKVGST3E3D?BcxKvTh; zJ%11PdTxJ+gTOkj=p2A__C;e9(D55I_>kfV?J#g3>C(A)Anb{tpcL9f&;!zynCA7l z20^ES6@V^vgsiRsNT*C&fCvLn)@27EBEqdvivj!sAe1q{td*kN#PKpOzb@N(fJelRxW>z^?VH$W zEr2Ydh+^imfCa4!Ae*m}c6XojJ{94M8u@L^?_W$2MeP=Pwk-g)_xs&^{C;#6dESW7MTg*9%2ef#?f{i#PIqvA8M#Js@6* zY>(clRt2ZMuG2p;5m>0*K&Vv14dSKf#oJn{A=2B9mVeknAeS$+TtgH7<%;)Wk!A!! zGWuU6eu%!jp`{xFy+L2`M~rf~slFoypCMO{ds zUh#b1-!TFMSff59NNX4%i5CNSO}$8fUNgde>(s$b>PNuZbk50t2}PhEThx;PwS|6S zmgvi7^(7!}_HE5ioE^b>^(FwVH>&YJD;!*>{`jnQ#)N*IY_2< zwAy(4MMM~+2>76HSYGRUn(HKdVx~U_(kGqv zJJ{&mFa*YXeFj+)sw^};ejB|MHVag zR(Q&sC&LvWhHzmc)a@WBknVfsPS`Ae&z#VP|lRXUP_ohzLEIMp+U; zaGEkMR^QJa8BFJG>LPD`9d9!|sUm(y5X@k6gg2(IU^6q+{O%x_#j_DSpyG2LB~Q(_ zi$M;{BeOkyEr(c6j=Jg;f)U)!u8=2}b*jnR8D)|F9 zke`H;Itc+GpUe0i`#F; zFqF|uWesaNN|lc&)KSGz*0P4Fj85W7T!T@_B8L)IvXZ@2P(e8jaarC#ITci}mzAuf zgdCl|4GRhb$s(I {{ form.hidden_tag() }} {{ macros.form_field(form.admin, class_='mailselect') }} {{ macros.form_field(form.submit) }} -{% endcall %} -{% endblock %} +{%- endcall %} +{%- endblock %} diff --git a/core/admin/mailu/ui/templates/admin/list.html b/core/admin/mailu/ui/templates/admin/list.html index f2f5d229..84d954a0 100644 --- a/core/admin/mailu/ui/templates/admin/list.html +++ b/core/admin/mailu/ui/templates/admin/list.html @@ -1,17 +1,17 @@ -{% extends "base.html" %} +{%- extends "base.html" %} -{% block title %} +{%- block title %} {% trans %}Global administrators{% endtrans %} -{% endblock %} +{%- endblock %} -{% block main_action %} +{%- block main_action %} {% trans %}Add administrator{% endtrans %} -{% endblock %} +{%- endblock %} -{% block content %} -{% call macros.table() %} +{%- block content %} +{%- call macros.table() %} {% trans %}Actions{% endtrans %} @@ -19,14 +19,14 @@ - {% for admin in admins %} + {%- for admin in admins %} {{ admin }} - {% endfor %} + {%- endfor %} -{% endcall %} -{% endblock %} +{%- endcall %} +{%- endblock %} diff --git a/core/admin/mailu/ui/templates/alias/create.html b/core/admin/mailu/ui/templates/alias/create.html index 2079d191..ce9f8167 100644 --- a/core/admin/mailu/ui/templates/alias/create.html +++ b/core/admin/mailu/ui/templates/alias/create.html @@ -1,15 +1,15 @@ -{% extends "base.html" %} +{%- extends "base.html" %} -{% block title %} +{%- block title %} {% trans %}Create alias{% endtrans %} -{% endblock %} +{%- endblock %} -{% block subtitle %} +{%- block subtitle %} {{ domain }} -{% endblock %} +{%- endblock %} -{% block content %} -{% call macros.card() %} +{%- block content %} +{%- call macros.card() %}

    {{ form.hidden_tag() }} {{ macros.form_field(form.localpart, append='@'+domain.name+'') }} @@ -18,5 +18,5 @@ {{ macros.form_field(form.comment) }} {{ macros.form_field(form.submit) }} -{% endcall %} -{% endblock %} +{%- endcall %} +{%- endblock %} diff --git a/core/admin/mailu/ui/templates/alias/edit.html b/core/admin/mailu/ui/templates/alias/edit.html index b28ea170..4dc13cce 100644 --- a/core/admin/mailu/ui/templates/alias/edit.html +++ b/core/admin/mailu/ui/templates/alias/edit.html @@ -1,9 +1,9 @@ -{% extends "alias/create.html" %} +{%- extends "alias/create.html" %} -{% block title %} +{%- block title %} {% trans %}Edit alias{% endtrans %} -{% endblock %} +{%- endblock %} -{% block subtitle %} +{%- block subtitle %} {{ alias }} -{% endblock %} +{%- endblock %} diff --git a/core/admin/mailu/ui/templates/alias/list.html b/core/admin/mailu/ui/templates/alias/list.html index e8ddc862..0b784d52 100644 --- a/core/admin/mailu/ui/templates/alias/list.html +++ b/core/admin/mailu/ui/templates/alias/list.html @@ -1,19 +1,19 @@ -{% extends "base.html" %} +{%- extends "base.html" %} -{% block title %} +{%- block title %} {% trans %}Alias list{% endtrans %} -{% endblock %} +{%- endblock %} -{% block subtitle %} +{%- block subtitle %} {{ domain.name }} -{% endblock %} +{%- endblock %} -{% block main_action %} +{%- block main_action %} {% trans %}Add alias{% endtrans %} -{% endblock %} +{%- endblock %} -{% block content %} -{% call macros.table() %} +{%- block content %} +{%- call macros.table() %} {% trans %}Actions{% endtrans %} @@ -25,7 +25,7 @@ - {% for alias in domain.aliases %} + {%- for alias in domain.aliases %}   @@ -37,7 +37,7 @@ {{ alias.created_at }} {{ alias.updated_at or '' }} - {% endfor %} + {%- endfor %} -{% endcall %} -{% endblock %} +{%- endcall %} +{%- endblock %} diff --git a/core/admin/mailu/ui/templates/alternative/create.html b/core/admin/mailu/ui/templates/alternative/create.html index 75461c67..f10cb718 100644 --- a/core/admin/mailu/ui/templates/alternative/create.html +++ b/core/admin/mailu/ui/templates/alternative/create.html @@ -1,9 +1,9 @@ -{% extends "form.html" %} +{%- extends "form.html" %} -{% block title %} +{%- block title %} {% trans %}Create alternative domain{% endtrans %} -{% endblock %} +{%- endblock %} -{% block subtitle %} +{%- block subtitle %} {{ domain }} -{% endblock %} +{%- endblock %} diff --git a/core/admin/mailu/ui/templates/alternative/list.html b/core/admin/mailu/ui/templates/alternative/list.html index f123eb9f..b56cd751 100644 --- a/core/admin/mailu/ui/templates/alternative/list.html +++ b/core/admin/mailu/ui/templates/alternative/list.html @@ -1,19 +1,19 @@ -{% extends "base.html" %} +{%- extends "base.html" %} -{% block title %} +{%- block title %} {% trans %}Alternative domain list{% endtrans %} -{% endblock %} +{%- endblock %} -{% block subtitle %} +{%- block subtitle %} {{ domain.name }} -{% endblock %} +{%- endblock %} -{% block main_action %} +{%- block main_action %} {% trans %}Add alternative{% endtrans %} -{% endblock %} +{%- endblock %} -{% block content %} -{% call macros.table() %} +{%- block content %} +{%- call macros.table() %} {% trans %}Actions{% endtrans %} @@ -22,7 +22,7 @@ - {% for alternative in domain.alternatives %} + {%- for alternative in domain.alternatives %} @@ -30,7 +30,7 @@ {{ alternative }} {{ alternative.created_at }} - {% endfor %} + {%- endfor %} -{% endcall %} -{% endblock %} +{%- endcall %} +{%- endblock %} diff --git a/core/admin/mailu/ui/templates/announcement.html b/core/admin/mailu/ui/templates/announcement.html index acdbde1a..ed7fe772 100644 --- a/core/admin/mailu/ui/templates/announcement.html +++ b/core/admin/mailu/ui/templates/announcement.html @@ -1,16 +1,16 @@ -{% extends "base.html" %} +{%- extends "base.html" %} -{% block title %} +{%- block title %} {% trans %}Public announcement{% endtrans %} -{% endblock %} +{%- endblock %} -{% block content %} -{% call macros.card() %} +{%- block content %} +{%- call macros.card() %}
    {{ form.hidden_tag() }} {{ macros.form_field(form.announcement_subject) }} {{ macros.form_field(form.announcement_body, rows=10) }} {{ macros.form_field(form.submit) }}
    -{% endcall %} -{% endblock %} +{%- endcall %} +{%- endblock %} diff --git a/core/admin/mailu/ui/templates/antispam.html b/core/admin/mailu/ui/templates/antispam.html new file mode 100644 index 00000000..0b2713b9 --- /dev/null +++ b/core/admin/mailu/ui/templates/antispam.html @@ -0,0 +1,15 @@ +{%- extends "base.html" %} + +{%- block title %} +{% trans %}Antispam{% endtrans %} +{%- endblock %} + +{%- block subtitle %} +{% trans %}RSPAMD status page{% endtrans %} +{%- endblock %} + +{%- block content %} +
    + +
    +{%- endblock %} diff --git a/core/admin/mailu/ui/templates/base.html b/core/admin/mailu/ui/templates/base.html index 89695e50..acef4b86 100644 --- a/core/admin/mailu/ui/templates/base.html +++ b/core/admin/mailu/ui/templates/base.html @@ -1,65 +1,83 @@ -{% import "macros.html" as macros %} -{% import "bootstrap/utils.html" as utils %} +{%- import "macros.html" as macros %} +{%- import "bootstrap/utils.html" as utils %} - + - + + + + + Mailu-Admin | {{ config["SITENAME"] }} - Mailu-Admin - {{ config["SITENAME"] }}
    -
  • - {%- for page, url in path %} + {%- for page, url in path %} {%- if loop.index > 1 %} {%- endif %} @@ -28,25 +28,25 @@ {%- else %} {{ page }} {%- endif %} - {%- endfor %} + {%- endfor %}