2507: Fix the CI for ARM builds r=mergify[bot] a=nextgens
We should install the dependencies everywhere where we may have to rebuild the wheels.
If other people use other arch and want their builds to go faster we can whitelist them too after they have confirmed it works.
Co-authored-by: Florent Daigniere <nextgens@freenetproject.org>