Plus, in theory you'd also need reproducible builds for everything because who knows what your compiler did to the source ;-)

Reality is, as you already implied: in practice you cannot "be careful" except avoiding obvious malware.

At SOME point you have to trust SOMEONE, unless you use TempleOS in which case you can trust whatever god you have.