Sure we should, but that’s the case for all transport modes. Remember that you can just pave the smaller amount of space needed for cycling down a street and not a full-width road. A cycle track uses less pavement than a roadway thanks to being narrower, and probably lasts longer too due to less damage from heavy vehicles.
It is but it's a lot less comfortable to ride on, and that makes a difference for longer distances. (They also have the unpleasant tendency to become mud when it rains.) But ultimately sure you can usually get around on a dirt path.