lub
/
mailu
1
0
Fork 0
Code Issues Pull Requests Packages Releases Wiki Activity

Merge #2427

Browse Source 
2427: Switch ci/cd workflow to use local build cache for buildx r=mergify[bot] a=Diman0

## What type of PR?
enhancement

## What does this PR do?
Switch to local build cache, cached via actions/cache@v3
The previous method of using gha cache via buildx proved to be unreliable. Using a local cache via actions/cache@v3 is much more reliable. The build job will re-use cache from previous workflow runs.
The total workflow time is still similar ~12 minutes.

If the cache action does intermittently seem to have issues with slow download, we can configure a lower timeout. It is now set on the default 60 minutes.

Some important tidbits:
Cache fragment in build step:
```
     - name: Configure actions/cache@v3 action for storing build cache in the /tmp/cache folder
        uses: actions/cache@v3
        with:
          path: /tmp/cache/${{ matrix.target }}
          key: ${{ github.ref }}-${{ inputs.mailu_version }}-${{ matrix.target }}-${{ github.run_id }}
          restore-keys: |
            ${{ github.ref }}-${{ inputs.mailu_version }}-${{ matrix.target }}
```
- actions/cache will never update a cache. So on cache-hit (key is found), the cache will not be updated. 
  - To workaround this, it is possible to use a key that will not have a cache hit. And use restore-keys to lookup and load an existing cache. `${{ github.ref }}-${{ inputs.mailu_version }}-${{ matrix.target }}` matches with `${{ github.ref }}-${{ inputs.mailu_version }}-${{ matrix.target }}-${{ github.run_id }}`.
  - So this will result the cache being loaded from a previous workflow. For more info see https://docs.github.com/en/actions/using-workflows/caching-dependencies-to-speed-up-workflows#example-using-multiple-restore-keys
- Two jobs cannot write to the same cache simultaneously. If two jobs (in the same workflow or between workflows) access the same cache (key), then only one of the two workflows can update the cache. For this reason the cache key used in the build step must be unique.
- ${{ inputs.mailu_version }} is used to make sure x64 and arm do not access each others build cache.

 Unfortunately it is not possible to use a for loop to loop through steps. For this reason it is not possible possible to shorten the amount of action/Cache@v3 steps. The only possibility is to create our own [composite action](https://docs.github.com/en/actions/creating-actions/creating-a-composite-action). But this makes it maybe to complex. Then the action.yml of the composite action must be maintained as well.

## Prerequisites
Before we can consider review and merge, please make sure the following list is done and checked.
If an entry in not applicable, you can check it or remove it from the list.

- [n/a] In case of feature or enhancement: documentation updated accordingly
- [n/a] Unless it's docs or a minor change: add [changelog](https://mailu.io/master/contributors/workflow.html#changelog) entry file.


Co-authored-by: Dimitri Huisman <diman@huisman.xyz>
master
bors[bot] 3 years ago committed by GitHub
parent 25413eaf2a 5732b2316f
commit 6bed48e2ac
No known key found for this signature in database
GPG Key ID: 4AEE18F83AFDEB23
1 changed files with 261 additions and 43 deletions
Show Stats Download Patch File Download Diff File

304
.github/workflows/build_test_deploy.yml vendored
Unescape Escape View File

@ -112,6 +112,13 @@ jobs:
echo "MAILU_VERSION=${{ inputs.mailu_version }}" >> $GITHUB_ENV
echo "PINNED_MAILU_VERSION=${{ inputs.pinned_mailu_version }}" >> $GITHUB_ENV
echo "DOCKER_ORG=${{ inputs.docker_org }}" >> $GITHUB_ENV
- name: Configure actions/cache@v3 action for storing build cache in the /tmp/cache folder
uses: actions/cache@v3
with:
path: /tmp/cache/${{ matrix.target }}
key: ${{ github.ref }}-${{ inputs.mailu_version }}-${{ matrix.target }}-${{ github.run_id }}
restore-keys: |
${{ github.ref }}-${{ inputs.mailu_version }}-${{ matrix.target }}
- name: Set up QEMU
uses: docker/setup-qemu-action@v2
- uses: crazy-max/ghaction-github-runtime@v2
@ -134,7 +141,8 @@ jobs:
load: false
push: false
set: |
*.cache-to=type=gha,scope=${{ github.run_id }}-${{ github.ref }}-${{ matrix.target }}-build,mode=min
*.cache-from=type=local,src=/tmp/cache/${{ matrix.target }}
*.cache-to=type=local,dest=/tmp/cache/${{ matrix.target }},mode=max
*.platform=${{ inputs.architecture }}
# This job runs all the tests.
@ -168,6 +176,76 @@ jobs:
echo "MAILU_VERSION=${{ inputs.mailu_version }}" >> $GITHUB_ENV
echo "PINNED_MAILU_VERSION=${{ inputs.pinned_mailu_version }}" >> $GITHUB_ENV
echo "DOCKER_ORG=${{ inputs.docker_org }}" >> $GITHUB_ENV
- name: Configure /cache for image docs
uses: actions/cache@v3
with:
path: /tmp/cache/docs
key: ${{ github.ref }}-${{ inputs.mailu_version }}-docs-${{ github.run_id }}
- name: Configure /cache for image setup
uses: actions/cache@v3
with:
path: /tmp/cache/setup
key: ${{ github.ref }}-${{ inputs.mailu_version }}-setup-${{ github.run_id }}
- name: Configure /cache for image admin
uses: actions/cache@v3
with:
path: /tmp/cache/admin
key: ${{ github.ref }}-${{ inputs.mailu_version }}-admin-${{ github.run_id }}
- name: Configure /cache for image antispam
uses: actions/cache@v3
with:
path: /tmp/cache/antispam
key: ${{ github.ref }}-${{ inputs.mailu_version }}-antispam-${{ github.run_id }}
- name: Configure /cache for image front
uses: actions/cache@v3
with:
path: /tmp/cache/front
key: ${{ github.ref }}-${{ inputs.mailu_version }}-front-${{ github.run_id }}
- name: Configure /cache for image imap
uses: actions/cache@v3
with:
path: /tmp/cache/imap
key: ${{ github.ref }}-${{ inputs.mailu_version }}-imap-${{ github.run_id }}
- name: Configure /cache for image smtp
uses: actions/cache@v3
with:
path: /tmp/cache/smtp
key: ${{ github.ref }}-${{ inputs.mailu_version }}-smtp-${{ github.run_id }}
- name: Configure /cache for image snappymail
uses: actions/cache@v3
with:
path: /tmp/cache/snappymail
key: ${{ github.ref }}-${{ inputs.mailu_version }}-snappymail-${{ github.run_id }}
- name: Configure /cache for image roundcube
uses: actions/cache@v3
with:
path: /tmp/cache/roundcube
key: ${{ github.ref }}-${{ inputs.mailu_version }}-roundcube-${{ github.run_id }}
- name: Configure /cache for image antivirus
uses: actions/cache@v3
with:
path: /tmp/cache/antivirus
key: ${{ github.ref }}-${{ inputs.mailu_version }}-antivirus-${{ github.run_id }}
- name: Configure /cache for image fetchmail
uses: actions/cache@v3
with:
path: /tmp/cache/fetchmail
key: ${{ github.ref }}-${{ inputs.mailu_version }}-fetchmail-${{ github.run_id }}
- name: Configure /cache for image resolver
uses: actions/cache@v3
with:
path: /tmp/cache/resolver
key: ${{ github.ref }}-${{ inputs.mailu_version }}-resolver-${{ github.run_id }}
- name: Configure /cache for image traefik-certdumper
uses: actions/cache@v3
with:
path: /tmp/cache/traefik-certdumper
key: ${{ github.ref }}-${{ inputs.mailu_version }}-traefik-certdumper-${{ github.run_id }}
- name: Configure /cache for image webdav
uses: actions/cache@v3
with:
path: /tmp/cache/webdav
key: ${{ github.ref }}-${{ inputs.mailu_version }}-webdav-${{ github.run_id }}
- name: Set up QEMU
uses: docker/setup-qemu-action@v2
- uses: crazy-max/ghaction-github-runtime@v2
@ -189,20 +267,20 @@ jobs:
load: true
push: false
set: |
*.cache-from=type=gha,scope=${{ github.run_id }}-${{ github.ref }}-docs-build
*.cache-from=type=gha,scope=${{ github.run_id }}-${{ github.ref }}-setup-build
*.cache-from=type=gha,scope=${{ github.run_id }}-${{ github.ref }}-admin-build
*.cache-from=type=gha,scope=${{ github.run_id }}-${{ github.ref }}-antispam-build
*.cache-from=type=gha,scope=${{ github.run_id }}-${{ github.ref }}-front-build
*.cache-from=type=gha,scope=${{ github.run_id }}-${{ github.ref }}-imap-build
*.cache-from=type=gha,scope=${{ github.run_id }}-${{ github.ref }}-smtp-build
*.cache-from=type=gha,scope=${{ github.run_id }}-${{ github.ref }}-snappymail-build
*.cache-from=type=gha,scope=${{ github.run_id }}-${{ github.ref }}-roundcube-build
*.cache-from=type=gha,scope=${{ github.run_id }}-${{ github.ref }}-antivirus-build
*.cache-from=type=gha,scope=${{ github.run_id }}-${{ github.ref }}-fetchmail-build
*.cache-from=type=gha,scope=${{ github.run_id }}-${{ github.ref }}-resolver-build
*.cache-from=type=gha,scope=${{ github.run_id }}-${{ github.ref }}-traefik-certdumper-build
*.cache-from=type=gha,scope=${{ github.run_id }}-${{ github.ref }}-webdav-build
*.cache-from=type=local,src=/tmp/cache/docs
*.cache-from=type=local,src=/tmp/cache/setup
*.cache-from=type=local,src=/tmp/cache/admin
*.cache-from=type=local,src=/tmp/cache/antispam
*.cache-from=type=local,src=/tmp/cache/front
*.cache-from=type=local,src=/tmp/cache/imap
*.cache-from=type=local,src=/tmp/cache/smtp
*.cache-from=type=local,src=/tmp/cache/snappymail
*.cache-from=type=local,src=/tmp/cache/roundcube
*.cache-from=type=local,src=/tmp/cache/antivirus
*.cache-from=type=local,src=/tmp/cache/fetchmail
*.cache-from=type=local,src=/tmp/cache/resolver
*.cache-from=type=local,src=/tmp/cache/traefik-certdumper
*.cache-from=type=local,src=/tmp/cache/webdav
*.platform=${{ inputs.architecture }}
- name: Install python packages
run: python3 -m pip install -r tests/requirements.txt
@ -234,6 +312,76 @@ jobs:
echo "MAILU_VERSION=${{ inputs.mailu_version }}" >> $GITHUB_ENV
echo "PINNED_MAILU_VERSION=${{ inputs.pinned_mailu_version }}" >> $GITHUB_ENV
echo "DOCKER_ORG=${{ inputs.docker_org }}" >> $GITHUB_ENV
- name: Configure /cache for image docs
uses: actions/cache@v3
with:
path: /tmp/cache/docs
key: ${{ github.ref }}-${{ inputs.mailu_version }}-docs-${{ github.run_id }}
- name: Configure /cache for image setup
uses: actions/cache@v3
with:
path: /tmp/cache/setup
key: ${{ github.ref }}-${{ inputs.mailu_version }}-setup-${{ github.run_id }}
- name: Configure /cache for image admin
uses: actions/cache@v3
with:
path: /tmp/cache/admin
key: ${{ github.ref }}-${{ inputs.mailu_version }}-admin-${{ github.run_id }}
- name: Configure /cache for image antispam
uses: actions/cache@v3
with:
path: /tmp/cache/antispam
key: ${{ github.ref }}-${{ inputs.mailu_version }}-antispam-${{ github.run_id }}
- name: Configure /cache for image front
uses: actions/cache@v3
with:
path: /tmp/cache/front
key: ${{ github.ref }}-${{ inputs.mailu_version }}-front-${{ github.run_id }}
- name: Configure /cache for image imap
uses: actions/cache@v3
with:
path: /tmp/cache/imap
key: ${{ github.ref }}-${{ inputs.mailu_version }}-imap-${{ github.run_id }}
- name: Configure /cache for image smtp
uses: actions/cache@v3
with:
path: /tmp/cache/smtp
key: ${{ github.ref }}-${{ inputs.mailu_version }}-smtp-${{ github.run_id }}
- name: Configure /cache for image snappymail
uses: actions/cache@v3
with:
path: /tmp/cache/snappymail
key: ${{ github.ref }}-${{ inputs.mailu_version }}-snappymail-${{ github.run_id }}
- name: Configure /cache for image roundcube
uses: actions/cache@v3
with:
path: /tmp/cache/roundcube
key: ${{ github.ref }}-${{ inputs.mailu_version }}-roundcube-${{ github.run_id }}
- name: Configure /cache for image antivirus
uses: actions/cache@v3
with:
path: /tmp/cache/antivirus
key: ${{ github.ref }}-${{ inputs.mailu_version }}-antivirus-${{ github.run_id }}
- name: Configure /cache for image fetchmail
uses: actions/cache@v3
with:
path: /tmp/cache/fetchmail
key: ${{ github.ref }}-${{ inputs.mailu_version }}-fetchmail-${{ github.run_id }}
- name: Configure /cache for image resolver
uses: actions/cache@v3
with:
path: /tmp/cache/resolver
key: ${{ github.ref }}-${{ inputs.mailu_version }}-resolver-${{ github.run_id }}
- name: Configure /cache for image traefik-certdumper
uses: actions/cache@v3
with:
path: /tmp/cache/traefik-certdumper
key: ${{ github.ref }}-${{ inputs.mailu_version }}-traefik-certdumper-${{ github.run_id }}
- name: Configure /cache for image webdav
uses: actions/cache@v3
with:
path: /tmp/cache/webdav
key: ${{ github.ref }}-${{ inputs.mailu_version }}-webdav-${{ github.run_id }}
- name: Set up QEMU
uses: docker/setup-qemu-action@v2
- uses: crazy-max/ghaction-github-runtime@v2
@ -254,20 +402,20 @@ jobs:
files: ${{env.HCL_FILE}}
push: true
set: |
*.cache-from=type=gha,scope=${{ github.run_id }}-${{ github.ref }}-docs-build
*.cache-from=type=gha,scope=${{ github.run_id }}-${{ github.ref }}-setup-build
*.cache-from=type=gha,scope=${{ github.run_id }}-${{ github.ref }}-admin-build
*.cache-from=type=gha,scope=${{ github.run_id }}-${{ github.ref }}-antispam-build
*.cache-from=type=gha,scope=${{ github.run_id }}-${{ github.ref }}-front-build
*.cache-from=type=gha,scope=${{ github.run_id }}-${{ github.ref }}-imap-build
*.cache-from=type=gha,scope=${{ github.run_id }}-${{ github.ref }}-smtp-build
*.cache-from=type=gha,scope=${{ github.run_id }}-${{ github.ref }}-snappymail-build
*.cache-from=type=gha,scope=${{ github.run_id }}-${{ github.ref }}-roundcube-build
*.cache-from=type=gha,scope=${{ github.run_id }}-${{ github.ref }}-antivirus-build
*.cache-from=type=gha,scope=${{ github.run_id }}-${{ github.ref }}-fetchmail-build
*.cache-from=type=gha,scope=${{ github.run_id }}-${{ github.ref }}-resolver-build
*.cache-from=type=gha,scope=${{ github.run_id }}-${{ github.ref }}-traefik-certdumper-build
*.cache-from=type=gha,scope=${{ github.run_id }}-${{ github.ref }}-webdav-build
*.cache-from=type=local,src=/tmp/cache/docs
*.cache-from=type=local,src=/tmp/cache/setup
*.cache-from=type=local,src=/tmp/cache/admin
*.cache-from=type=local,src=/tmp/cache/antispam
*.cache-from=type=local,src=/tmp/cache/front
*.cache-from=type=local,src=/tmp/cache/imap
*.cache-from=type=local,src=/tmp/cache/smtp
*.cache-from=type=local,src=/tmp/cache/snappymail
*.cache-from=type=local,src=/tmp/cache/roundcube
*.cache-from=type=local,src=/tmp/cache/antivirus
*.cache-from=type=local,src=/tmp/cache/fetchmail
*.cache-from=type=local,src=/tmp/cache/resolver
*.cache-from=type=local,src=/tmp/cache/traefik-certdumper
*.cache-from=type=local,src=/tmp/cache/webdav
*.platform=${{ inputs.architecture }}
deploy-arm:
@ -286,6 +434,76 @@ jobs:
echo "MAILU_VERSION=${{ inputs.mailu_version }}" >> $GITHUB_ENV
echo "PINNED_MAILU_VERSION=${{ inputs.pinned_mailu_version }}" >> $GITHUB_ENV
echo "DOCKER_ORG=${{ inputs.docker_org }}" >> $GITHUB_ENV
- name: Configure /cache for image docs
uses: actions/cache@v3
with:
path: /tmp/cache/docs
key: ${{ github.ref }}-${{ inputs.mailu_version }}-docs-${{ github.run_id }}
- name: Configure /cache for image setup
uses: actions/cache@v3
with:
path: /tmp/cache/setup
key: ${{ github.ref }}-${{ inputs.mailu_version }}-setup-${{ github.run_id }}
- name: Configure /cache for image admin
uses: actions/cache@v3
with:
path: /tmp/cache/admin
key: ${{ github.ref }}-${{ inputs.mailu_version }}-admin-${{ github.run_id }}
- name: Configure /cache for image antispam
uses: actions/cache@v3
with:
path: /tmp/cache/antispam
key: ${{ github.ref }}-${{ inputs.mailu_version }}-antispam-${{ github.run_id }}
- name: Configure /cache for image front
uses: actions/cache@v3
with:
path: /tmp/cache/front
key: ${{ github.ref }}-${{ inputs.mailu_version }}-front-${{ github.run_id }}
- name: Configure /cache for image imap
uses: actions/cache@v3
with:
path: /tmp/cache/imap
key: ${{ github.ref }}-${{ inputs.mailu_version }}-imap-${{ github.run_id }}
- name: Configure /cache for image smtp
uses: actions/cache@v3
with:
path: /tmp/cache/smtp
key: ${{ github.ref }}-${{ inputs.mailu_version }}-smtp-${{ github.run_id }}
- name: Configure /cache for image snappymail
uses: actions/cache@v3
with:
path: /tmp/cache/snappymail
key: ${{ github.ref }}-${{ inputs.mailu_version }}-snappymail-${{ github.run_id }}
- name: Configure /cache for image roundcube
uses: actions/cache@v3
with:
path: /tmp/cache/roundcube
key: ${{ github.ref }}-${{ inputs.mailu_version }}-roundcube-${{ github.run_id }}
- name: Configure /cache for image antivirus
uses: actions/cache@v3
with:
path: /tmp/cache/antivirus
key: ${{ github.ref }}-${{ inputs.mailu_version }}-antivirus-${{ github.run_id }}
- name: Configure /cache for image fetchmail
uses: actions/cache@v3
with:
path: /tmp/cache/fetchmail
key: ${{ github.ref }}-${{ inputs.mailu_version }}-fetchmail-${{ github.run_id }}
- name: Configure /cache for image resolver
uses: actions/cache@v3
with:
path: /tmp/cache/resolver
key: ${{ github.ref }}-${{ inputs.mailu_version }}-resolver-${{ github.run_id }}
- name: Configure /cache for image traefik-certdumper
uses: actions/cache@v3
with:
path: /tmp/cache/traefik-certdumper
key: ${{ github.ref }}-${{ inputs.mailu_version }}-traefik-certdumper-${{ github.run_id }}
- name: Configure /cache for image webdav
uses: actions/cache@v3
with:
path: /tmp/cache/webdav
key: ${{ github.ref }}-${{ inputs.mailu_version }}-webdav-${{ github.run_id }}
- name: Set up QEMU
uses: docker/setup-qemu-action@v2
- uses: crazy-max/ghaction-github-runtime@v2
@ -306,20 +524,20 @@ jobs:
files: ${{env.HCL_FILE}}
push: true
set: |
*.cache-from=type=gha,scope=${{ github.run_id }}-${{ github.ref }}-docs-build
*.cache-from=type=gha,scope=${{ github.run_id }}-${{ github.ref }}-setup-build
*.cache-from=type=gha,scope=${{ github.run_id }}-${{ github.ref }}-admin-build
*.cache-from=type=gha,scope=${{ github.run_id }}-${{ github.ref }}-antispam-build
*.cache-from=type=gha,scope=${{ github.run_id }}-${{ github.ref }}-front-build
*.cache-from=type=gha,scope=${{ github.run_id }}-${{ github.ref }}-imap-build
*.cache-from=type=gha,scope=${{ github.run_id }}-${{ github.ref }}-smtp-build
*.cache-from=type=gha,scope=${{ github.run_id }}-${{ github.ref }}-snappymail-build
*.cache-from=type=gha,scope=${{ github.run_id }}-${{ github.ref }}-roundcube-build
*.cache-from=type=gha,scope=${{ github.run_id }}-${{ github.ref }}-antivirus-build
*.cache-from=type=gha,scope=${{ github.run_id }}-${{ github.ref }}-fetchmail-build
*.cache-from=type=gha,scope=${{ github.run_id }}-${{ github.ref }}-resolver-build
*.cache-from=type=gha,scope=${{ github.run_id }}-${{ github.ref }}-traefik-certdumper-build
*.cache-from=type=gha,scope=${{ github.run_id }}-${{ github.ref }}-webdav-build
*.cache-from=type=local,src=/tmp/cache/docs
*.cache-from=type=local,src=/tmp/cache/setup
*.cache-from=type=local,src=/tmp/cache/admin
*.cache-from=type=local,src=/tmp/cache/antispam
*.cache-from=type=local,src=/tmp/cache/front
*.cache-from=type=local,src=/tmp/cache/imap
*.cache-from=type=local,src=/tmp/cache/smtp
*.cache-from=type=local,src=/tmp/cache/snappymail
*.cache-from=type=local,src=/tmp/cache/roundcube
*.cache-from=type=local,src=/tmp/cache/antivirus
*.cache-from=type=local,src=/tmp/cache/fetchmail
*.cache-from=type=local,src=/tmp/cache/resolver
*.cache-from=type=local,src=/tmp/cache/traefik-certdumper
*.cache-from=type=local,src=/tmp/cache/webdav
*.platform=${{ inputs.architecture }}
#This job creates a tagged release. A tag is created for the pinned version x.y.z. The GH release refers to this tag.

Write Preview
Loading…
Cancel
Save