It seems to me that maybe the most efficient course is to have the person who is sitting in the car drive it, thus removing roughly ten billion dependencies, security vulnerabilities, and failure modes.
Sometimes I get impatient at an old person behind the wheel, and my wife says, "He can't see that fast."
There's not a hard line. There's a degradation in response time, and so old people drive slower (at least, they drive slower if they're wise). And then there's a point where even that isn't safe, and they need to stop.
But yes, not everyone can drive. Some are too old. Some are too young. Some can't afford a car for the amount of time they need to use one. Some are dealing with a temporary thing, like a broken leg or a bad sprain, and they can't drive just for a few days, but they still shouldn't drive right now. Some are coming back from a bar, or from a medical procedure involving anesthesia. (I forget the procedure, but I was told "don't sign any legal documents in the next 24 hours" along with "don't drive".) Lots of people can't drive for lots of reasons.
But, you know, we used to have things called "cabs" that helped with all of that. "Not everyone can drive" is orthogonal to "should a car be self-driving, or should a human drive it".
Cabs cost significant money and will continue to cost relatively more and more due to Baumol's cost disease preventing productivity increases (1 driver can only drive 1 cab at a time).
Self driving cars offer the chance for inexpensive point to point transportation. Just because humans were once needed to drive cabs doesn’t mean they always will, any more than we need elevator operators anymore.
We still have cabs, urber... They are all too expensive to be your normal way of getting around. Useful for rare trips or emergencies, but not for the normal day to day trips. A personal car is quickly cheaper, while a cab becomes unaffordable. (Cars are not cheap, but most of the costs are fixed)
