How? Barring a database leak I don't see a way for someone to simply scan all the links. Putting something like Cloudflare in front of the shortener with a rate limit would prevent brute force scanning. I assume google semi-competently made the shortener (using a random number generator) which would make it pretty hard to find links in the first place.

Removing inactive links also doesn't solve this problem. You can still have active links to secret docs.

Back when it was made, shorteners were competing to see who could make the shortest URL, so I bet a brute force scan would find everything.

If they're have a (passwordless) URL they're not secret.