2422: Build wheels only if we have to. r=mergify[bot] a=nextgens
## What type of PR?
enhancement
## What does this PR do?
This massively speeds the CI up; We don't need to install a compiler and rebuild when we have wheels available... with this admin builds 4 times faster.
### Related issue(s)
- #1830
Co-authored-by: Florent Daigniere <nextgens@freenetproject.org>
Co-authored-by: Dimitri Huisman <diman@huisman.xyz>