It’s all based on energy and economy. The amount of energy required (even for a relatively slow chemical rocket!) is so astronomical, it’s apparently not worthwhile to do at scale.

Or there are no actual aliens, except us. For whatever reason.