It feels like the CI should be a completely separate system, and then you can't get surprises like cache poisoning or accessing secrets, or pushing code: if you never gave the CI any credentials, it is a RO client like everything else.
If its cache gets poisoned, won't affect the repo.