2719: Introduce connection string (database url) for roundcube & simplify setup r=mergify[bot] a=Diman0

## What type of PR?

enhancement

## What does this PR do?
As discussed in earlier Mailu meetings (#1582), we want to simplify configuring Mailu and make it more user-friendly. Especially the last part is an important mission statement of the Mailu project. 

This PR will remove the choice of what DB to use from setup. New users are guided now to make the correct choice of using SQLite.

For simplifying the configuration, all the database environment variables have been removed and replaced with a single connection string environment variable. 

For backwards compatibility, the old *DB_* setting can still be used. This is to make sure that master does not immediately break for all users. After X months after the next Mailu release, we can remove the old settings from the software. This provides a transition period. 

### Related issue(s)
- #2533

## Prerequisites
Before we can consider review and merge, please make sure the following list is done and checked.
If an entry in not applicable, you can check it or remove it from the list.

- [x] In case of feature or enhancement: documentation updated accordingly
- [x] Unless it's docs or a minor change: add [changelog](https://mailu.io/master/contributors/workflow.html#changelog) entry file.


Co-authored-by: Dimitri Huisman <diman@huisman.xyz>
Co-authored-by: Florent Daigniere <nextgens@users.noreply.github.com>
main
bors[bot] 2 years ago committed by GitHub
commit b243ea084d
No known key found for this signature in database
GPG Key ID: 4AEE18F83AFDEB23

@ -154,6 +154,7 @@ jobs:
shell: bash shell: bash
command: | command: |
set -euxo pipefail \ set -euxo pipefail \
; /usr/bin/docker info \
; echo "${{ github.token }}" | docker login --username "${{ github.repository_owner }}" --password-stdin ghcr.io \ ; echo "${{ github.token }}" | docker login --username "${{ github.repository_owner }}" --password-stdin ghcr.io \
; echo "$DOCKER_PASSW" | docker login --username "$DOCKER_LOGIN" --password-stdin \ ; echo "$DOCKER_PASSW" | docker login --username "$DOCKER_LOGIN" --password-stdin \
; /usr/bin/docker buildx rm builder-${{ env.BUILDER }} \ ; /usr/bin/docker buildx rm builder-${{ env.BUILDER }} \
@ -220,6 +221,7 @@ jobs:
shell: bash shell: bash
command: | command: |
set -euxo pipefail \ set -euxo pipefail \
; /usr/bin/docker info \
; echo "${{ github.token }}" | docker login --username "${{ github.repository_owner }}" --password-stdin ghcr.io \ ; echo "${{ github.token }}" | docker login --username "${{ github.repository_owner }}" --password-stdin ghcr.io \
; echo "$DOCKER_PASSW2" | docker login --username "$DOCKER_LOGIN2" --password-stdin \ ; echo "$DOCKER_PASSW2" | docker login --username "$DOCKER_LOGIN2" --password-stdin \
; /usr/bin/docker buildx rm builder-${{ env.BUILDER }} \ ; /usr/bin/docker buildx rm builder-${{ env.BUILDER }} \
@ -295,6 +297,7 @@ jobs:
shell: bash shell: bash
command: | command: |
set -euxo pipefail \ set -euxo pipefail \
; /usr/bin/docker info \
; echo "${{ github.token }}" | docker login --username "${{ github.repository_owner }}" --password-stdin ghcr.io \ ; echo "${{ github.token }}" | docker login --username "${{ github.repository_owner }}" --password-stdin ghcr.io \
; echo "$DOCKER_PASSW" | docker login --username "$DOCKER_LOGIN" --password-stdin \ ; echo "$DOCKER_PASSW" | docker login --username "$DOCKER_LOGIN" --password-stdin \
; /usr/bin/docker buildx rm builder-${{ env.BUILDER }} \ ; /usr/bin/docker buildx rm builder-${{ env.BUILDER }} \
@ -368,6 +371,7 @@ jobs:
shell: bash shell: bash
command: | command: |
set -euxo pipefail \ set -euxo pipefail \
; /usr/bin/docker info \
; echo "${{ github.token }}" | docker login --username "${{ github.repository_owner }}" --password-stdin ghcr.io \ ; echo "${{ github.token }}" | docker login --username "${{ github.repository_owner }}" --password-stdin ghcr.io \
; echo "$DOCKER_PASSW2" | docker login --username "$DOCKER_LOGIN2" --password-stdin \ ; echo "$DOCKER_PASSW2" | docker login --username "$DOCKER_LOGIN2" --password-stdin \
; /usr/bin/docker buildx rm builder-${{ env.BUILDER }} \ ; /usr/bin/docker buildx rm builder-${{ env.BUILDER }} \

@ -27,6 +27,7 @@ DEFAULT_CONFIG = {
'DB_NAME': 'mailu', 'DB_NAME': 'mailu',
'SQLITE_DATABASE_FILE': 'data/main.db', 'SQLITE_DATABASE_FILE': 'data/main.db',
'SQLALCHEMY_DATABASE_URI': 'sqlite:////data/main.db', 'SQLALCHEMY_DATABASE_URI': 'sqlite:////data/main.db',
'SQLALCHEMY_DATABASE_URI_ROUNDCUBE': 'sqlite:////data/roundcube.db',
'SQLALCHEMY_TRACK_MODIFICATIONS': False, 'SQLALCHEMY_TRACK_MODIFICATIONS': False,
# Statistics management # Statistics management
'INSTANCE_ID_PATH': '/data/instance', 'INSTANCE_ID_PATH': '/data/instance',

@ -24,7 +24,7 @@ itsdangerous
limits limits
marshmallow marshmallow
marshmallow-sqlalchemy marshmallow-sqlalchemy
mysql-connector-python==8.0.29 mysql-connector-python
passlib passlib
psycopg2-binary psycopg2-binary
Pygments Pygments

@ -45,7 +45,7 @@ marshmallow==3.18.0
marshmallow-sqlalchemy==0.28.1 marshmallow-sqlalchemy==0.28.1
msoffcrypto-tool==5.0.0 msoffcrypto-tool==5.0.0
multidict==6.0.2 multidict==6.0.2
mysql-connector-python==8.0.29 mysql-connector-python==8.0.32
olefile==0.46 olefile==0.46
oletools==0.60.1 oletools==0.60.1
packaging==21.3 packaging==21.3

@ -285,26 +285,22 @@ These are used for DNS based service discovery with possibly changing services I
Database settings Database settings
----------------- -----------------
Both the admin and roundcube services store their configurations in a SQLite database.
Alternatives hosted options like PostgreSQL and MariaDB/MySQL can be configured using `DB URL`_
but the development team recommends against it. Indeed, there is currently very little data
to be stored and SQLite is deemed both sufficient, simpler and more reliable overall.
The admin service stores configurations in a database. - ``SQLALCHEMY_DATABASE_URI`` (default: sqlite:////data/main.db): the SQLAlchemy database URL for accessing the database
- ``SQLALCHEMY_DATABASE_URI_ROUNDCUBE`` (default: sqlite:////data/roundcube.db): the Roundcube database URL for accessing the Roundcube database
- ``DB_FLAVOR``: the database type for mailu admin service. (``sqlite``, ``postgresql``, ``mysql``) For PostgreSQL use driver postgresql (``SQLALCHEMY_DATABASE_URI=postgresql://mailu:mailu_secret_password@database/mailu``).
- ``DB_HOST``: the database host for mailu admin service. For non-default ports use the notation `host:port`. (when not ``sqlite``)
- ``DB_PW``: the database password for mailu admin service. (when not ``sqlite``)
- ``DB_USER``: the database user for mailu admin service. (when not ``sqlite``)
- ``DB_NAME``: the database name for mailu admin service. (when not ``sqlite``)
Alternatively, if you need more control, you can use a `DB URL`_ : do not set any of the ``DB_`` settings and set ``SQLALCHEMY_DATABASE_URI`` instead. For MariaDB/MySQL use driver mysql+mysqlconnector (``SQLALCHEMY_DATABASE_URI= mysql+mysqlconnector://mailu:mailu_secret_password@database/mailu```).
For Roundcube, refer to the `roundcube documentation`_ for the URL specification.
.. _`DB URL`: https://docs.sqlalchemy.org/en/latest/core/engines.html#database-urls .. _`DB URL`: https://docs.sqlalchemy.org/en/latest/core/engines.html#database-urls
.. _`roundcube documentation`: https://github.com/roundcube/roundcubemail/blob/master/config/defaults.inc.php#L28
The roundcube service stores configurations in a database.
- ``ROUNDCUBE_DB_FLAVOR``: the database type for roundcube service. (``sqlite``, ``postgresql``, ``mysql``)
- ``ROUNDCUBE_DB_HOST``: the database host for roundcube service. For non-default ports use the notation `host:port`. (when not ``sqlite``)
- ``ROUNDCUBE_DB_PW``: the database password for roundcube service. (when not ``sqlite``)
- ``ROUNDCUBE_DB_USER``: the database user for roundcube service. (when not ``sqlite``)
- ``ROUNDCUBE_DB_NAME``: the database name for roundcube service. (when not ``sqlite``)
Webmail settings Webmail settings
---------------- ----------------

@ -1,8 +1,10 @@
Changing the database back-end Changing the database back-end
============================== ==============================
By default Mailu uses a SQLite database. We have changed the internals of Mailu By default Mailu uses a SQLite database. It is possible to use an alternative database solutions such as PostgreSQL and MySQL/MariaDB.
to enable the support of alternative database solutions such as PostgreSQL and MySQL/MariaDB.
The Mailu database contains static data. SQLite is sufficient for any deployment scenario of Mailu. There is no need to use a different database system.
The Mailu development team recommends to use SQLite.
Migrating to a different database back-end Migrating to a different database back-end
@ -17,7 +19,7 @@ To switch to a different database back-end:
1. Run config-export to export the configuration. E.g. `docker compose exec admin flask mailu config-export --secrets --output mail-config.yml` 1. Run config-export to export the configuration. E.g. `docker compose exec admin flask mailu config-export --secrets --output mail-config.yml`
2. Set up your new database server. Refer to the subsequent sections for tips for creating the database. 2. Set up your new database server. Refer to the subsequent sections for tips for creating the database.
3. Modify the database settings (DB_*) in mailu.env. Refer to the :ref:`configuration guide (link) <db_settings>` for the exact settings. 3. Modify the database settings (SQLAlchemy database URL) in mailu.env. Refer to the :ref:`configuration guide (link) <db_settings>` for the exact settings.
4. Start your Mailu deployment. 4. Start your Mailu deployment.
5. Run config-import to import the configuration. E.g. `docker exec -i $(docker compose ps -q admin) flask mailu config-import -v < mail-config.yml` 5. Run config-import to import the configuration. E.g. `docker exec -i $(docker compose ps -q admin) flask mailu config-import -v < mail-config.yml`
@ -80,7 +82,7 @@ In ``pg_hba.conf`` there should be a line like this:
host mailu mailu <mailu_host>/32 md5 host mailu mailu <mailu_host>/32 md5
Note that this example is the bare-minimum to get Mailu working. Additional work needs to be Note that this example is the bare-minimum to get Mailu working. Additional work needs to be
done by the database admin to setup their own means of backups and TLS encrypted connections. done by the database admin to setup their own means of backups and TLS encrypted connections.
Nowadays it is recommended to use the official PostgreSQL image from the PostgreSQL community. The repository is located `here <https://hub.docker.com/_/postgres>`_. Nowadays it is recommended to use the official PostgreSQL image from the PostgreSQL community. The repository is located `here <https://hub.docker.com/_/postgres>`_.
@ -91,10 +93,10 @@ Mailu PostgreSQL
---------------- ----------------
Mailu optionally came with a pre-configured PostgreSQL image which was deprecated in Mailu 1.8. Mailu optionally came with a pre-configured PostgreSQL image which was deprecated in Mailu 1.8.
Since Mailu 1.9 it is removed from Mailu. The following section describes how to move to a different Since Mailu 1.9 it is removed from Mailu. The following section describes how to move to a different
PostgreSQL image for novice administrators. The official PostgreSQL image (Postgres) will be used. PostgreSQL image for novice administrators. The official PostgreSQL image (Postgres) will be used.
A Mailu deployment with the Mailu PostgreSQL image, will only use PostgreSQL for the Admin container A Mailu deployment with the Mailu PostgreSQL image, will only use PostgreSQL for the Admin container
(Web administration interface). Roundcube uses SQLite as database back-end. (Web administration interface). Roundcube uses SQLite as database back-end.
Mailu uses the following configuration for connecting to the database: Mailu uses the following configuration for connecting to the database:
@ -192,10 +194,7 @@ to
.. code-block:: docker .. code-block:: docker
DB_HOST=database SQLALCHEMY_DATABASE_URI=postgresql://mailu:mailu@database/mailu
DB_PORT=5432
DB_USER=mailu
DB_NAME=mailu
Mailu is now configured to use the official PostgreSQL docker image. Bring your new deployment online Mailu is now configured to use the official PostgreSQL docker image. Bring your new deployment online

@ -192,21 +192,3 @@ DEFAULT_SPAM_THRESHOLD=80
# This is a mandatory setting for using the RESTful API. # This is a mandatory setting for using the RESTful API.
API_TOKEN={{ api_token }} API_TOKEN={{ api_token }}
###################################
# Database settings
###################################
DB_FLAVOR={{ db_flavor }}
{% if db_flavor == 'postgresql' or db_flavor == 'mysql' %}
DB_USER={{ db_user }}
DB_PW={{ db_pw }}
DB_HOST={{ db_url }}
DB_NAME={{ db_name }}
{% endif %}
{% if ((db_flavor_rc == 'postgresql' or db_flavor_rc == 'mysql')) and webmail_type == 'roundcube' %}
ROUNDCUBE_DB_FLAVOR={{ db_flavor_rc }}
ROUNDCUBE_DB_USER={{ roundcube_db_user }}
ROUNDCUBE_DB_PW={{ roundcube_db_pw }}
ROUNDCUBE_DB_HOST={{ roundcube_db_url }}
ROUNDCUBE_DB_NAME={{ roundcube_db_name }}
{% endif %}

@ -1,45 +0,0 @@
{% call macros.panel("info", "Database preferences") %}
<div class="form-group">
<label>Which database back end would you like to use?</label>
<br/>
<select class="btn btn-primary dropdown-toggle" name="db_flavor" id="database">
{% for dbflavor in ["sqlite", "postgresql", "mysql"] %}
<option value="{{ dbflavor }}" >{{ dbflavor }}</option>
{% endfor %}
</select>
<div class="form-group" id="external_db" style="display: none">
<p>Set external database parameters for <b>Admin UI</b></p>
<label>DB User</label>
<input class="form-control" type="text" name="db_user" placeholder="Username" id="db_user">
<label>Db Password</label>
<input class="form-control" type="password" name="db_pw" placeholder="Password" id="db_pw">
<label>Db URL (for using a different port use the notation host:port )</label>
<input class="form-control" type="text" name="db_url" placeholder="URL" id="db_url">
<label>Db Name</label>
<input class="form-control" type="text" name="db_name" placeholder="Database Name" id="db_name">
<br/>
<div class="form-group" id="db_flavor_rc_sel" style="display: none">
<label>Which database back end would you like to use for roundcube?</label>
<br/>
<select class="btn btn-primary dropdown-toggle" name=db_flavor_rc id="database_rc">
{% for dbflavor_rc in ["sqlite", "postgresql", "mysql"] %}
<option value="{{ dbflavor_rc }}" >{{ dbflavor_rc }}</option>
{% endfor %}
</select>
<div class="form-group" id="roundcube_external_db" style="display: none">
<p>Set external database parameters for <b>Roundcube</b></p>
<label>DB User</label>
<input class="form-control" type="text" name="roundcube_db_user" placeholder="Username" id="roundcube_db_user">
<label>DB Password</label>
<input class="form-control" type="password" name="roundcube_db_pw" placeholder="Password" id="roundcube_db_pw">
<label>DB URL (for using a different port use the notation host:port )</label>
<input class="form-control" type="text" name="roundcube_db_url" placeholder="URL" id="roundcube_db_url">
<label>DB Name</label>
<input class="form-control" type="text" name="roundcube_db_name" placeholder="Database Name" id="roundcube_db_name">
</div>
</div>
</div>
</div>
{% endcall %}

@ -14,7 +14,6 @@
{%for file in steps %} {%for file in steps %}
{% include "steps/" + flavor + "/" + file %} {% include "steps/" + flavor + "/" + file %}
{% endfor %} {% endfor %}
{% include "steps/database.html" %}
<input class="btn btn-primary" type="submit" value="Setup Mailu"> <input class="btn btn-primary" type="submit" value="Setup Mailu">
</form> </form>
{% endblock %} {% endblock %}

@ -0,0 +1,17 @@
Introduce SQLAlchemy database uris for configuring the admin and roundcube database.
Remove the database configuration option from the setup utility. Using a different
database system than SQLite is not necessary for Mailu. The Mailu database generally
contains static data.
The usage of the *DB_* environment variables is deprecated now.
They can still be used in the release after Mailu 1.9, but will be removed
after that version. This means it will be removed from master after the upcoming
Mailu release.
To start using the new environment variables, all *DB_* environment variables must be changed to:
SQLALCHEMY_DATABASE_URI=<SQLAlchemy database URL>
SQLALCHEMY_DATABASE_URI_ROUNDCUBE=<Roundcube database URL>
If no URI is specified, SQLite is used with these settings:
SQLALCHEMY_DATABASE_URI=sqlite:////data/main.db
SQLALCHEMY_DATABASE_URI_ROUNDCUBE=sqlite:////data/roundcube.db

@ -26,7 +26,8 @@ with open("/etc/resolv.conf") as handle:
resolver = content[content.index("nameserver") + 1] resolver = content[content.index("nameserver") + 1]
context["RESOLVER"] = f"[{resolver}]" if ":" in resolver else resolver context["RESOLVER"] = f"[{resolver}]" if ":" in resolver else resolver
db_flavor = env.get("ROUNDCUBE_DB_FLAVOR", "sqlite") db_uri = env.get("SQLALCHEMY_DATABASE_URI_ROUNDCUBE", "sqlite:////data/roundcube.db")
db_flavor = env.get("ROUNDCUBE_DB_FLAVOR")
if db_flavor == "sqlite": if db_flavor == "sqlite":
context["DB_DSNW"] = "sqlite:////data/roundcube.db" context["DB_DSNW"] = "sqlite:////data/roundcube.db"
elif db_flavor == "mysql": elif db_flavor == "mysql":
@ -43,6 +44,8 @@ elif db_flavor == "postgresql":
env.get("ROUNDCUBE_DB_HOST", "database"), env.get("ROUNDCUBE_DB_HOST", "database"),
env.get("ROUNDCUBE_DB_NAME", "roundcube") env.get("ROUNDCUBE_DB_NAME", "roundcube")
) )
elif db_uri:
context["DB_DSNW"] = db_uri
else: else:
print(f"Unknown ROUNDCUBE_DB_FLAVOR: {db_flavor}", file=sys.stderr) print(f"Unknown ROUNDCUBE_DB_FLAVOR: {db_flavor}", file=sys.stderr)
exit(1) exit(1)

Loading…
Cancel
Save