lub
/
mailu
1
0
Fork 0
Code Issues Pull Requests Packages Releases Wiki Activity

Merge #1604

Browse Source 
1604: Added CLI command to export and import the configuration r=mergify[bot] a=ghostwheel42

## What type of PR?

enhancement

## What does this PR do?

This PR adds a `config-dump` cli command and updates the `config-update` cli command to
handle all possible items and parameters.
This was done by adding generic **to_dict** and **from_dict** methods to the Base model, so it should be quite future-proof.
The changes to `config-update` are backwards-compatible to the old command.
I've only removed the undocumented yaml-section _managers_ - managers can now be defined in the _users_ section.

The YAML now looks like this:

```
  aliases:
    - email: email@example.com
      destination:
        - address@example.com
  
  domains:
    - name: example.com
      alternatives:
        - alternative.tld
  
  relays:
    - name: relay.example.com
      smtp: mx.example.com
  
  users:
    - email: postmaster@example.com
      displayed_name: 'Postmaster'
      enable_imap: true
      enable_pop: false
      enabled: true
      forward_destination:
        - address@remote.example.com
      forward_enabled: true
      forward_keep: true
      global_admin: true
      manager_of:
        - example.com
      password: '{BLF-CRYPT}$2b$12$...'
      spam_enabled: true
      spam_threshold: 80

```

### Related issue(s)
- Closes issue #1377

## Prerequistes
Before we can consider review and merge, please make sure the following list is done and checked.
If an entry in not applicable, you can check it or remove it from the list.

- [X] In case of feature or enhancement: documentation updated accordingly
- [X] Unless it's docs or a minor change: add [changelog](https://mailu.io/master/contributors/guide.html#changelog) entry file.


Co-authored-by: Alexander Graf <ghostwheel42@users.noreply.github.com>
master
bors[bot] 3 years ago committed by GitHub
parent fc1a663da2 fbd945390d
commit 7481a6d272
No known key found for this signature in database
GPG Key ID: 4AEE18F83AFDEB23
7 changed files with 2148 additions and 221 deletions
Show Stats Download Patch File Download Diff File

193
core/admin/mailu/manage.py
Unescape Escape View File

@ -1,40 +1,46 @@
from mailu import models
""" Mailu command line interface
"""
from flask import current_app as app
from flask import cli as flask_cli
import flask
import sys
import os
import socket
import uuid
import click
import yaml
from flask import current_app as app
from flask.cli import FlaskGroup, with_appcontext
from mailu import models
from mailu.schemas import MailuSchema, Logger, RenderJSON
db = models.db
@click.group()
def mailu(cls=flask_cli.FlaskGroup):
@click.group(cls=FlaskGroup, context_settings={'help_option_names': ['-?', '-h', '--help']})
def mailu():
""" Mailu command line
"""
@mailu.command()
@flask_cli.with_appcontext
@with_appcontext
def advertise():
""" Advertise this server against statistic services.
"""
if os.path.isfile(app.config["INSTANCE_ID_PATH"]):
with open(app.config["INSTANCE_ID_PATH"], "r") as handle:
if os.path.isfile(app.config['INSTANCE_ID_PATH']):
with open(app.config['INSTANCE_ID_PATH'], 'r') as handle:
instance_id = handle.read()
else:
instance_id = str(uuid.uuid4())
with open(app.config["INSTANCE_ID_PATH"], "w") as handle:
with open(app.config['INSTANCE_ID_PATH'], 'w') as handle:
handle.write(instance_id)
if not app.config["DISABLE_STATISTICS"]:
if not app.config['DISABLE_STATISTICS']:
try:
socket.gethostbyname(app.config["STATS_ENDPOINT"].format(instance_id))
except:
socket.gethostbyname(app.config['STATS_ENDPOINT'].format(instance_id))
except OSError:
pass
@ -43,7 +49,7 @@ def advertise():
@click.argument('domain_name')
@click.argument('password')
@click.option('-m', '--mode')
@flask_cli.with_appcontext
@with_appcontext
def admin(localpart, domain_name, password, mode='create'):
""" Create an admin user
'mode' can be:
@ -58,7 +64,7 @@ def admin(localpart, domain_name, password, mode='create'):
user = None
if mode == 'ifmissing' or mode == 'update':
email = '{}@{}'.format(localpart, domain_name)
email = f'{localpart}@{domain_name}'
user = models.User.query.get(email)
if user and mode == 'ifmissing':
@ -86,7 +92,7 @@ def admin(localpart, domain_name, password, mode='create'):
@click.argument('localpart')
@click.argument('domain_name')
@click.argument('password')
@flask_cli.with_appcontext
@with_appcontext
def user(localpart, domain_name, password):
""" Create a user
"""
@ -108,16 +114,16 @@ def user(localpart, domain_name, password):
@click.argument('localpart')
@click.argument('domain_name')
@click.argument('password')
@flask_cli.with_appcontext
@with_appcontext
def password(localpart, domain_name, password):
""" Change the password of an user
"""
email = '{0}@{1}'.format(localpart, domain_name)
email = f'{localpart}@{domain_name}'
user = models.User.query.get(email)
if user:
user.set_password(password)
else:
print("User " + email + " not found.")
print(f'User {email} not found.')
db.session.commit()
@ -126,7 +132,7 @@ def password(localpart, domain_name, password):
@click.option('-u', '--max-users')
@click.option('-a', '--max-aliases')
@click.option('-q', '--max-quota-bytes')
@flask_cli.with_appcontext
@with_appcontext
def domain(domain_name, max_users=-1, max_aliases=-1, max_quota_bytes=0):
""" Create a domain
"""
@ -142,9 +148,9 @@ def domain(domain_name, max_users=-1, max_aliases=-1, max_quota_bytes=0):
@click.argument('localpart')
@click.argument('domain_name')
@click.argument('password_hash')
@flask_cli.with_appcontext
@with_appcontext
def user_import(localpart, domain_name, password_hash):
""" Import a user along with password hash.
""" Import a user along with password hash
"""
domain = models.Domain.query.get(domain_name)
if not domain:
@ -160,14 +166,14 @@ def user_import(localpart, domain_name, password_hash):
db.session.commit()
# TODO: remove deprecated config_update function?
@mailu.command()
@click.option('-v', '--verbose')
@click.option('-d', '--delete-objects')
@flask_cli.with_appcontext
@with_appcontext
def config_update(verbose=False, delete_objects=False):
"""sync configuration with data from YAML-formatted stdin"""
import yaml
import sys
""" Sync configuration with data from YAML (deprecated)
"""
new_config = yaml.safe_load(sys.stdin)
# print new_config
domains = new_config.get('domains', [])
@ -187,13 +193,13 @@ def config_update(verbose=False, delete_objects=False):
max_aliases=max_aliases,
max_quota_bytes=max_quota_bytes)
db.session.add(domain)
print("Added " + str(domain_config))
print(f'Added {domain_config}')
else:
domain.max_users = max_users
domain.max_aliases = max_aliases
domain.max_quota_bytes = max_quota_bytes
db.session.add(domain)
print("Updated " + str(domain_config))
print(f'Updated {domain_config}')
users = new_config.get('users', [])
tracked_users = set()
@ -209,7 +215,7 @@ def config_update(verbose=False, delete_objects=False):
domain_name = user_config['domain']
password_hash = user_config.get('password_hash', None)
domain = models.Domain.query.get(domain_name)
email = '{0}@{1}'.format(localpart, domain_name)
email = f'{localpart}@{domain_name}'
optional_params = {}
for k in user_optional_params:
if k in user_config:
@ -239,13 +245,13 @@ def config_update(verbose=False, delete_objects=False):
print(str(alias_config))
localpart = alias_config['localpart']
domain_name = alias_config['domain']
if type(alias_config['destination']) is str:
if isinstance(alias_config['destination'], str):
destination = alias_config['destination'].split(',')
else:
destination = alias_config['destination']
wildcard = alias_config.get('wildcard', False)
domain = models.Domain.query.get(domain_name)
email = '{0}@{1}'.format(localpart, domain_name)
email = f'{localpart}@{domain_name}'
if not domain:
domain = models.Domain(name=domain_name)
db.session.add(domain)
@ -275,7 +281,7 @@ def config_update(verbose=False, delete_objects=False):
domain_name = manager_config['domain']
user_name = manager_config['user']
domain = models.Domain.query.get(domain_name)
manageruser = models.User.query.get(user_name + '@' + domain_name)
manageruser = models.User.query.get(f'{user_name}@{domain_name}')
if manageruser not in domain.managers:
domain.managers.append(manageruser)
db.session.add(domain)
@ -284,26 +290,117 @@ def config_update(verbose=False, delete_objects=False):
if delete_objects:
for user in db.session.query(models.User).all():
if not (user.email in tracked_users):
if not user.email in tracked_users:
if verbose:
print("Deleting user: " + str(user.email))
print(f'Deleting user: {user.email}')
db.session.delete(user)
for alias in db.session.query(models.Alias).all():
if not (alias.email in tracked_aliases):
if not alias.email in tracked_aliases:
if verbose:
print("Deleting alias: " + str(alias.email))
print(f'Deleting alias: {alias.email}')
db.session.delete(alias)
for domain in db.session.query(models.Domain).all():
if not (domain.name in tracked_domains):
if not domain.name in tracked_domains:
if verbose:
print("Deleting domain: " + str(domain.name))
print(f'Deleting domain: {domain.name}')
db.session.delete(domain)
db.session.commit()
@mailu.command()
@click.option('-v', '--verbose', count=True, help='Increase verbosity.')
@click.option('-s', '--secrets', is_flag=True, help='Show secret attributes in messages.')
@click.option('-d', '--debug', is_flag=True, help='Enable debug output.')
@click.option('-q', '--quiet', is_flag=True, help='Quiet mode - only show errors.')
@click.option('-c', '--color', is_flag=True, help='Force colorized output.')
@click.option('-u', '--update', is_flag=True, help='Update mode - merge input with existing config.')
@click.option('-n', '--dry-run', is_flag=True, help='Perform a trial run with no changes made.')
@click.argument('source', metavar='[FILENAME|-]', type=click.File(mode='r'), default=sys.stdin)
@with_appcontext
def config_import(verbose=0, secrets=False, debug=False, quiet=False, color=False,
update=False, dry_run=False, source=None):
""" Import configuration as YAML or JSON from stdin or file
"""
log = Logger(want_color=color or None, can_color=sys.stdout.isatty(), secrets=secrets, debug=debug)
log.lexer = 'python'
log.strip = True
log.verbose = 0 if quiet else verbose
log.quiet = quiet
context = {
'import': True,
'update': update,
'clear': not update,
'callback': log.track_serialize,
}
schema = MailuSchema(only=MailuSchema.Meta.order, context=context)
try:
# import source
with models.db.session.no_autoflush:
config = schema.loads(source)
# flush session to show/count all changes
if not quiet and (dry_run or verbose):
db.session.flush()
# check for duplicate domain names
config.check()
except Exception as exc:
if msg := log.format_exception(exc):
raise click.ClickException(msg) from exc
raise
# don't commit when running dry
if dry_run:
log.changes('Dry run. Not committing changes.')
db.session.rollback()
else:
log.changes('Committing changes.')
db.session.commit()
@mailu.command()
@click.option('-f', '--full', is_flag=True, help='Include attributes with default value.')
@click.option('-s', '--secrets', is_flag=True,
help='Include secret attributes (dkim-key, passwords).')
@click.option('-d', '--dns', is_flag=True, help='Include dns records.')
@click.option('-c', '--color', is_flag=True, help='Force colorized output.')
@click.option('-o', '--output-file', 'output', default=sys.stdout, type=click.File(mode='w'),
help='Save configuration to file.')
@click.option('-j', '--json', 'as_json', is_flag=True, help='Export configuration in json format.')
@click.argument('only', metavar='[FILTER]...', nargs=-1)
@with_appcontext
def config_export(full=False, secrets=False, color=False, dns=False, output=None, as_json=False, only=None):
""" Export configuration as YAML or JSON to stdout or file
"""
log = Logger(want_color=color or None, can_color=output.isatty())
only = only or MailuSchema.Meta.order
context = {
'full': full,
'secrets': secrets,
'dns': dns,
}
try:
schema = MailuSchema(only=only, context=context)
if as_json:
schema.opts.render_module = RenderJSON
log.lexer = 'json'
log.strip = True
print(log.colorize(schema.dumps(models.MailuConfig())), file=output)
except Exception as exc:
if msg := log.format_exception(exc):
raise click.ClickException(msg) from exc
raise
@mailu.command()
@click.argument('email')
@flask_cli.with_appcontext
@with_appcontext
def user_delete(email):
"""delete user"""
user = models.User.query.get(email)
@ -314,7 +411,7 @@ def user_delete(email):
@mailu.command()
@click.argument('email')
@flask_cli.with_appcontext
@with_appcontext
def alias_delete(email):
"""delete alias"""
alias = models.Alias.query.get(email)
@ -328,7 +425,7 @@ def alias_delete(email):
@click.argument('domain_name')
@click.argument('destination')
@click.option('-w', '--wildcard', is_flag=True)
@flask_cli.with_appcontext
@with_appcontext
def alias(localpart, domain_name, destination, wildcard=False):
""" Create an alias
"""
@ -341,7 +438,7 @@ def alias(localpart, domain_name, destination, wildcard=False):
domain=domain,
wildcard=wildcard,
destination=destination.split(','),
email="%s@%s" % (localpart, domain_name)
email=f'{localpart}@{domain_name}'
)
db.session.add(alias)
db.session.commit()
@ -352,7 +449,7 @@ def alias(localpart, domain_name, destination, wildcard=False):
@click.argument('max_users')
@click.argument('max_aliases')
@click.argument('max_quota_bytes')
@flask_cli.with_appcontext
@with_appcontext
def setlimits(domain_name, max_users, max_aliases, max_quota_bytes):
""" Set domain limits
"""
@ -367,16 +464,12 @@ def setlimits(domain_name, max_users, max_aliases, max_quota_bytes):
@mailu.command()
@click.argument('domain_name')
@click.argument('user_name')
@flask_cli.with_appcontext
@with_appcontext
def setmanager(domain_name, user_name='manager'):
""" Make a user manager of a domain
"""
domain = models.Domain.query.get(domain_name)
manageruser = models.User.query.get(user_name + '@' + domain_name)
manageruser = models.User.query.get(f'{user_name}@{domain_name}')
domain.managers.append(manageruser)
db.session.add(domain)
db.session.commit()
if __name__ == '__main__':
cli()

693
core/admin/mailu/models.py
View File

File diff suppressed because it is too large Load Diff

1269
core/admin/mailu/schemas.py
View File

File diff suppressed because it is too large Load Diff

4
core/admin/requirements-prod.txt
Unescape Escape View File

@ -16,6 +16,7 @@ Flask-DebugToolbar==0.10.1
Flask-KVSession==0.6.2
Flask-Limiter==1.0.1
Flask-Login==0.4.1
flask-marshmallow==0.14.0
Flask-Migrate==2.4.0
Flask-Script==2.0.6
Flask-SQLAlchemy==2.4.0
@ -30,9 +31,12 @@ limits==1.3
Mako==1.0.9
MarkupSafe==1.1.1
mysqlclient==1.4.2.post1
marshmallow==3.10.0
marshmallow-sqlalchemy==0.24.1
passlib==1.7.4
psycopg2==2.8.2
pycparser==2.19
Pygments==2.8.1
pyOpenSSL==19.0.0
python-dateutil==2.8.0
python-editor==1.0.4

4
core/admin/requirements.txt
Unescape Escape View File

@ -17,6 +17,7 @@ gunicorn
tabulate
PyYAML
PyOpenSSL
Pygments
dnspython
bcrypt
tenacity
@ -24,3 +25,6 @@ mysqlclient
psycopg2
idna
srslib
marshmallow
flask-marshmallow
marshmallow-sqlalchemy

197
docs/cli.rst
Unescape Escape View File

@ -11,6 +11,8 @@ Managing users and aliases can be done from CLI using commands:
* user-import
* user-delete
* config-update
* config-export
* config-import
alias
-----
@ -62,7 +64,7 @@ primary difference with simple `user` command is that password is being imported
docker-compose run --rm admin flask mailu user-import myuser example.net '$6$51ebe0cb9f1dab48effa2a0ad8660cb489b445936b9ffd812a0b8f46bca66dd549fea530ce' 'SHA512-CRYPT'
user-delete
------------
-----------
.. code-block:: bash
@ -94,7 +96,7 @@ where mail-config.yml looks like:
without ``--delete-object`` option config-update will only add/update new values but will *not* remove any entries missing in provided YAML input.
Users
-----
^^^^^
following are additional parameters that could be defined for users:
@ -113,8 +115,197 @@ following are additional parameters that could be defined for users:
* spam_threshold
Alias
-----
^^^^^
additional fields:
* wildcard
config-export
-------------
The purpose of this command is to export the complete configuration in YAML or JSON format.
.. code-block:: bash
$ docker-compose exec admin flask mailu config-export --help
Usage: flask mailu config-export [OPTIONS] [FILTER]...
Export configuration as YAML or JSON to stdout or file
Options:
-f, --full Include attributes with default value.
-s, --secrets Include secret attributes (dkim-key, passwords).
-d, --dns Include dns records.
-c, --color Force colorized output.
-o, --output-file FILENAME Save configuration to file.
-j, --json Export configuration in json format.
-?, -h, --help Show this message and exit.
Only non-default attributes are exported. If you want to export all attributes use ``--full``.
If you want to export plain-text secrets (dkim-keys, passwords) you have to add the ``--secrets`` option.
To include dns records (mx, spf, dkim and dmarc) add the ``--dns`` option.
By default all configuration objects are exported (domain, user, alias, relay). You can specify
filters to export only some objects or attributes (try: ``user`` or ``domain.name``).
Attributes explicitly specified in filters are automatically exported: there is no need to add ``--secrets`` or ``--full``.
.. code-block:: bash
$ docker-compose exec admin flask mailu config-export --output mail-config.yml
$ docker-compose exec admin flask mailu config-export domain.dns_mx domain.dns_spf
$ docker-compose exec admin flask mailu config-export user.spam_threshold
config-import
-------------
This command imports configuration data from an external YAML or JSON source.
.. code-block:: bash
$ docker-compose exec admin flask mailu config-import --help
Usage: flask mailu config-import [OPTIONS] [FILENAME|-]
Import configuration as YAML or JSON from stdin or file
Options:
-v, --verbose Increase verbosity.
-s, --secrets Show secret attributes in messages.
-q, --quiet Quiet mode - only show errors.
-c, --color Force colorized output.
-u, --update Update mode - merge input with existing config.
-n, --dry-run Perform a trial run with no changes made.
-?, -h, --help Show this message and exit.
The current version of docker-compose exec does not pass stdin correctly, so you have to user docker exec instead:
.. code-block:: bash
docker exec -i $(docker-compose ps -q admin) flask mailu config-import -nv < mail-config.yml
mail-config.yml contains the configuration and looks like this:
.. code-block:: yaml
domain:
- name: example.com
alternatives:
- alternative.example.com
user:
- email: foo@example.com
password_hash: '$2b$12$...'
hash_scheme: MD5-CRYPT
alias:
- email: alias1@example.com
destination:
- user1@example.com
- user2@example.com
relay:
- name: relay.example.com
comment: test
smtp: mx.example.com
config-import shows the number of created/modified/deleted objects after import.
To suppress all messages except error messages use ``--quiet``.
By adding the ``--verbose`` switch the import gets more detailed and shows exactly what attributes changed.
In all log messages plain-text secrets (dkim-keys, passwords) are hidden by default. Use ``--secrets`` to log secrets.
If you want to test what would be done when importing without committing any changes, use ``--dry-run``.
By default config-import replaces the whole configuration. ``--update`` allows to modify the existing configuration instead.
New elements will be added and existing elements will be modified.
It is possible to delete a single element or prune all elements from lists and associative arrays using a special notation:
+-----------------------------+------------------+--------------------------+
| Delete what? | notation | example |
+=============================+==================+==========================+
| specific array object | ``- -key: id`` | ``- -name: example.com`` |
+-----------------------------+------------------+--------------------------+
| specific list item | ``- -id`` | ``- -user1@example.com`` |
+-----------------------------+------------------+--------------------------+
| all remaining array objects | ``- -key: null`` | ``- -email: null`` |
+-----------------------------+------------------+--------------------------+
| all remaining list items | ``- -prune-`` | ``- -prune-`` |
+-----------------------------+------------------+--------------------------+
The ``-key: null`` notation can also be used to reset an attribute to its default.
To reset *spam_threshold* to it's default *80* use ``-spam_threshold: null``.
A new dkim key can be generated when adding or modifying a domain, by using the special value
``dkim_key: -generate-``.
This is a complete YAML template with all additional parameters that can be defined:
.. code-block:: yaml
domain:
- name: example.com
alternatives:
- alternative.tld
comment: ''
dkim_key: ''
max_aliases: -1
max_quota_bytes: 0
max_users: -1
signup_enabled: false
user:
- email: postmaster@example.com
comment: ''
displayed_name: 'Postmaster'
enable_imap: true
enable_pop: false
enabled: true
fetches:
- id: 1
comment: 'test fetch'
error: null
host: other.example.com
keep: true
last_check: '2020-12-29T17:09:48.200179'
password: 'secret'
hash_password: true
port: 993
protocol: imap
tls: true
username: fetch-user
forward_destination:
- address@remote.example.com
forward_enabled: true
forward_keep: true
global_admin: true
manager_of:
- example.com
password: '$2b$12$...'
hash_password: true
quota_bytes: 1000000000
reply_body: ''
reply_enabled: false
reply_enddate: '2999-12-31'
reply_startdate: '1900-01-01'
reply_subject: ''
spam_enabled: true
spam_threshold: 80
tokens:
- id: 1
comment: email-client
ip: 192.168.1.1
password: '$5$rounds=1$...'
aliases:
- email: email@example.com
comment: ''
destination:
- address@example.com
wildcard: false
relay:
- name: relay.example.com
comment: ''
smtp: mx.example.com

1
towncrier/newsfragments/1604.feature
Unescape Escape View File

@ -0,0 +1 @@
Add cli commands config-import and config-export
Write Preview
Loading…
Cancel
Save