Introduce connection string (database url) for roundcube.

Remove database choice from setup.
Remove the old *DB_* database env variables from the documentation.
The env vars are deprecated now. They will be removed after the upcoming
Mailu release.

.github/workflows/build_test_deploy.yml

@@ -154,6 +154,7 @@ jobs:
           shell: bash
           command: |
             set -euxo pipefail \
+              ; /usr/bin/docker info \
               ; echo "${{ github.token }}" |        docker login --username "${{ github.repository_owner }}" --password-stdin ghcr.io \
               ; echo "$DOCKER_PASSW" |    docker login --username "$DOCKER_LOGIN" --password-stdin \
               ; /usr/bin/docker buildx rm builder-${{ env.BUILDER }} \
@@ -220,6 +221,7 @@ jobs:
           shell: bash
           command: |
             set -euxo pipefail \
+              ; /usr/bin/docker info \
               ; echo "${{ github.token }}" |        docker login --username "${{ github.repository_owner }}" --password-stdin ghcr.io \
               ; echo "$DOCKER_PASSW2" |    docker login --username "$DOCKER_LOGIN2" --password-stdin \
               ; /usr/bin/docker buildx rm builder-${{ env.BUILDER }} \
@@ -295,6 +297,7 @@ jobs:
           shell: bash
           command: |
             set -euxo pipefail \
+              ; /usr/bin/docker info \
               ; echo "${{ github.token }}" |        docker login --username "${{ github.repository_owner }}" --password-stdin ghcr.io \
               ; echo "$DOCKER_PASSW" |   docker login --username "$DOCKER_LOGIN" --password-stdin \
               ; /usr/bin/docker buildx rm builder-${{ env.BUILDER }} \
@@ -368,6 +371,7 @@ jobs:
           shell: bash
           command: |
             set -euxo pipefail \
+              ; /usr/bin/docker info \
               ; echo "${{ github.token }}" |        docker login --username "${{ github.repository_owner }}" --password-stdin ghcr.io \
               ; echo "$DOCKER_PASSW2" |    docker login --username "$DOCKER_LOGIN2" --password-stdin \
               ; /usr/bin/docker buildx rm builder-${{ env.BUILDER }} \

core/admin/mailu/configuration.py

@@ -27,6 +27,7 @@ DEFAULT_CONFIG = {
     'DB_NAME': 'mailu',
     'SQLITE_DATABASE_FILE': 'data/main.db',
     'SQLALCHEMY_DATABASE_URI': 'sqlite:////data/main.db',
+    'SQLALCHEMY_DATABASE_URI_ROUNDCUBE': 'sqlite:////data/roundcube.db',
     'SQLALCHEMY_TRACK_MODIFICATIONS': False,
     # Statistics management
     'INSTANCE_ID_PATH': '/data/instance',

core/base/requirements-prod.txt

@@ -45,7 +45,7 @@ marshmallow==3.18.0
 marshmallow-sqlalchemy==0.28.1
 msoffcrypto-tool==5.0.0
 multidict==6.0.2
-mysql-connector-python==8.0.29
+mysql-connector-python==8.0.32
 olefile==0.46
 oletools==0.60.1
 packaging==21.3

docs/configuration.rst

@@ -285,26 +285,22 @@ These are used for DNS based service discovery with possibly changing services I
 Database settings
 -----------------
 
+The admin service stores configurations in a database. The roundcube service also stores configurations in a database.
+By default SQLite is used. SQLite is sufficient for practically any deployment.
+The database contains static configuration. It is recommended to use SQLite. However, a different database back-end can be used via a
+`DB URL`_. For Admin, only supported are SQLite, PostgreSQL and MariaDB/MySQL.
 
-The admin service stores configurations in a database.
+- ``SQLALCHEMY_DATABASE_URI`` (default: sqlite:////data/main.db): the SQLAlchemy database URL for accessing the database
+- ``SQLALCHEMY_DATABASE_URI_ROUNDCUBE`` (default: sqlite:////data/roundcube.db): the Roundcube database URL for accessing the Roundcube database
 
-- ``DB_FLAVOR``: the database type for mailu admin service. (``sqlite``, ``postgresql``, ``mysql``)
+For PostgreSQL use driver postgresql (``SQLALCHEMY_DATABASE_URI=postgresql://mailu:mailu_secret_password@database/mailu``).
-- ``DB_HOST``: the database host for mailu admin service. For non-default ports use the notation `host:port`. (when not ``sqlite``)
-- ``DB_PW``: the database password for mailu admin service. (when not ``sqlite``)
-- ``DB_USER``: the database user for mailu admin service. (when not ``sqlite``)
-- ``DB_NAME``: the database name for mailu admin service. (when not ``sqlite``)
 
-Alternatively, if you need more control, you can use a `DB URL`_ : do not set any of the ``DB_`` settings and set ``SQLALCHEMY_DATABASE_URI`` instead.
+For MariaDB/MySQL use driver mysql+mysqlconnector (``SQLALCHEMY_DATABASE_URI= mysql+mysqlconnector://mailu:mailu_secret_password@database/mailu```).
+
+For Roundcube, refer to the `roundcube documentation`_ for the URL specification.
 
 .. _`DB URL`: https://docs.sqlalchemy.org/en/latest/core/engines.html#database-urls
-
+.. _`roundcube documentation`: https://github.com/roundcube/roundcubemail/blob/master/config/defaults.inc.php#L28
-The roundcube service stores configurations in a database.
-
-- ``ROUNDCUBE_DB_FLAVOR``: the database type for roundcube service. (``sqlite``, ``postgresql``, ``mysql``)
-- ``ROUNDCUBE_DB_HOST``: the database host for roundcube service. For non-default ports use the notation `host:port`. (when not ``sqlite``)
-- ``ROUNDCUBE_DB_PW``: the database password for roundcube service. (when not ``sqlite``)
-- ``ROUNDCUBE_DB_USER``: the database user for roundcube service. (when not ``sqlite``)
-- ``ROUNDCUBE_DB_NAME``: the database name for roundcube service. (when not ``sqlite``)
 
 Webmail settings
 ----------------

docs/database.rst

@@ -1,8 +1,10 @@
 Changing the database back-end
 ==============================
 
-By default Mailu uses a SQLite database. We have changed the internals of Mailu
+By default Mailu uses a SQLite database. It is possible to use an alternative database solutions such as PostgreSQL and MySQL/MariaDB.
-to enable the support of alternative database solutions such as PostgreSQL and MySQL/MariaDB.
+
+The Mailu database contains static data. SQLite is sufficient for any deployment scenario of Mailu. There is no need to use a different database system.
+The Mailu development team recommends to use SQLite.
 
 
 Migrating to a different database back-end
@@ -17,7 +19,7 @@ To switch to a different database back-end:
 
 1. Run config-export to export the configuration. E.g.  `docker compose exec admin flask mailu config-export --secrets --output mail-config.yml`
 2. Set up your new database server. Refer to the subsequent sections for tips for creating the database.
-3. Modify the database settings (DB_*) in mailu.env. Refer to the :ref:`configuration guide (link) <db_settings>` for the exact settings.
+3. Modify the database settings (SQLAlchemy database URL) in mailu.env. Refer to the :ref:`configuration guide (link) <db_settings>` for the exact settings.
 4. Start your Mailu deployment.
 5. Run config-import to import the configuration. E.g. `docker exec -i $(docker compose ps -q admin) flask mailu config-import -v < mail-config.yml`
 
@@ -80,7 +82,7 @@ In ``pg_hba.conf`` there should be a line like this:
 
   host    mailu           mailu           <mailu_host>/32            md5
 
-Note that this example is the bare-minimum to get Mailu working. Additional work needs to be 
+Note that this example is the bare-minimum to get Mailu working. Additional work needs to be
 done by the database admin to setup their own means of backups and TLS encrypted connections.
 
 Nowadays it is recommended to use the official PostgreSQL image from the PostgreSQL community. The repository is located `here <https://hub.docker.com/_/postgres>`_.
@@ -91,10 +93,10 @@ Mailu PostgreSQL
 ----------------
 
 Mailu optionally came with a pre-configured PostgreSQL image which was deprecated in Mailu 1.8.
-Since Mailu 1.9 it is removed from Mailu. The following section describes how to move to a different 
+Since Mailu 1.9 it is removed from Mailu. The following section describes how to move to a different
 PostgreSQL image for novice administrators. The official PostgreSQL image (Postgres) will be used.
 
-A Mailu deployment with the Mailu PostgreSQL image, will only use PostgreSQL for the Admin container 
+A Mailu deployment with the Mailu PostgreSQL image, will only use PostgreSQL for the Admin container
 (Web administration interface). Roundcube uses SQLite as database back-end.
 Mailu uses the following configuration for connecting to the database:
 
@@ -192,10 +194,7 @@ to
 
 .. code-block:: docker
 
-   DB_HOST=database
+   SQLALCHEMY_DATABASE_URI=postgresql://mailu:mailu@database/mailu
-   DB_PORT=5432
-   DB_USER=mailu
-   DB_NAME=mailu
 
 Mailu is now configured to use the official PostgreSQL docker image. Bring your new deployment online
 

setup/flavors/compose/mailu.env

@@ -192,21 +192,3 @@ DEFAULT_SPAM_THRESHOLD=80
 # This is a mandatory setting for using the RESTful API.
 API_TOKEN={{ api_token }}
 
-###################################
-# Database settings
-###################################
-DB_FLAVOR={{ db_flavor }}
-{% if db_flavor == 'postgresql' or db_flavor == 'mysql' %}
-DB_USER={{ db_user }}
-DB_PW={{ db_pw }}
-DB_HOST={{ db_url }}
-DB_NAME={{ db_name }}
-{% endif %}
-
-{% if ((db_flavor_rc == 'postgresql' or db_flavor_rc == 'mysql')) and webmail_type == 'roundcube' %}
-ROUNDCUBE_DB_FLAVOR={{ db_flavor_rc }}
-ROUNDCUBE_DB_USER={{ roundcube_db_user }}
-ROUNDCUBE_DB_PW={{ roundcube_db_pw }}
-ROUNDCUBE_DB_HOST={{ roundcube_db_url }}
-ROUNDCUBE_DB_NAME={{ roundcube_db_name }}
-{% endif %}

setup/templates/steps/database.html

@@ -1,45 +0,0 @@
-{% call macros.panel("info", "Database preferences") %}
-
-<div class="form-group">
-	<label>Which database back end would you like to use?</label>
-	<br/>
-	 <select class="btn btn-primary dropdown-toggle" name="db_flavor" id="database">
-		{% for dbflavor in ["sqlite", "postgresql", "mysql"] %}
-			<option value="{{ dbflavor }}" >{{ dbflavor }}</option>
-		{% endfor %}
-	</select>
-	<div class="form-group" id="external_db" style="display: none">
-		<p>Set external database parameters for <b>Admin UI</b></p>
-		<label>DB User</label>
-		<input class="form-control" type="text" name="db_user" placeholder="Username" id="db_user">
-		<label>Db Password</label>
-		<input class="form-control" type="password" name="db_pw" placeholder="Password" id="db_pw">
-		<label>Db URL (for using a different port use the notation host:port )</label>
-		<input class="form-control" type="text" name="db_url" placeholder="URL" id="db_url">
-		<label>Db Name</label>
-		<input class="form-control" type="text" name="db_name" placeholder="Database Name" id="db_name">
-		<br/>
-		<div class="form-group" id="db_flavor_rc_sel" style="display: none">
-			<label>Which database back end would you like to use for roundcube?</label>
-			<br/>
-			<select class="btn btn-primary dropdown-toggle" name=db_flavor_rc id="database_rc">
-				{% for dbflavor_rc in ["sqlite", "postgresql", "mysql"] %}
-					<option value="{{ dbflavor_rc }}" >{{ dbflavor_rc }}</option>
-				{% endfor %}
-			</select>
-			<div class="form-group" id="roundcube_external_db" style="display: none">
-				<p>Set external database parameters for <b>Roundcube</b></p>
-				<label>DB User</label>
-				<input class="form-control" type="text" name="roundcube_db_user" placeholder="Username" id="roundcube_db_user">
-				<label>DB Password</label>
-				<input class="form-control" type="password" name="roundcube_db_pw" placeholder="Password" id="roundcube_db_pw">
-				<label>DB URL (for using a different port use the notation host:port )</label>
-				<input class="form-control" type="text" name="roundcube_db_url" placeholder="URL" id="roundcube_db_url">
-				<label>DB Name</label>
-				<input class="form-control" type="text" name="roundcube_db_name" placeholder="Database Name" id="roundcube_db_name">
-			</div>
-		</div>
-	</div>
-</div>
-
-{% endcall %}

setup/templates/wizard.html

@@ -14,7 +14,6 @@
   {%for file in steps %}
   {% include "steps/" + flavor + "/" + file %}
   {% endfor %}
-  {% include "steps/database.html" %}
   <input class="btn btn-primary" type="submit" value="Setup Mailu">
   </form>
 {% endblock %}

towncrier/newsfragments/2533.misc

@@ -0,0 +1,17 @@
+Introduce SQLAlchemy database uris for configuring the admin and roundcube database.
+Remove the database configuration option from the setup utility. Using a different
+database system than SQLite is not necessary for Mailu. The Mailu database generally
+contains static data.
+
+The usage of the *DB_* environment variables is deprecated now.
+They can still be used in the release after Mailu 1.9, but will be removed
+after that version. This means it will be removed from master after the upcoming
+Mailu release.
+
+To start using the new environment variables, all *DB_* environment variables must be changed to:
+SQLALCHEMY_DATABASE_URI=<SQLAlchemy database URL>
+SQLALCHEMY_DATABASE_URI_ROUNDCUBE=<Roundcube database URL>
+
+If no URI is specified, SQLite is used with these settings:
+SQLALCHEMY_DATABASE_URI=sqlite:////data/main.db
+SQLALCHEMY_DATABASE_URI_ROUNDCUBE=sqlite:////data/roundcube.db

webmails/start.py

@@ -26,7 +26,8 @@ with open("/etc/resolv.conf") as handle:
     resolver = content[content.index("nameserver") + 1]
     context["RESOLVER"] = f"[{resolver}]" if ":" in resolver else resolver
 
-db_flavor = env.get("ROUNDCUBE_DB_FLAVOR", "sqlite")
+db_uri = env.get("SQLALCHEMY_DATABASE_URI_ROUNDCUBE", "sqlite:////data/roundcube.db")
+db_flavor = env.get("ROUNDCUBE_DB_FLAVOR")
 if db_flavor == "sqlite":
     context["DB_DSNW"] = "sqlite:////data/roundcube.db"
 elif db_flavor == "mysql":
@@ -43,6 +44,8 @@ elif db_flavor == "postgresql":
         env.get("ROUNDCUBE_DB_HOST", "database"),
         env.get("ROUNDCUBE_DB_NAME", "roundcube")
     )
+elif db_uri:
+    context["DB_DSNW"] = db_uri
 else:
     print(f"Unknown ROUNDCUBE_DB_FLAVOR: {db_flavor}", file=sys.stderr)
     exit(1)