That's my point. Aircraft can fly in IMC (Instrument Meteorological Conditions) because they have redundant equipment to do so. Anything, really anything but vision, which is in fact the last resource you should be using when flying as it can be highly misleading.
You name it (I simplify on purpose, for all other pilots reading): radio assistance beacons, GPS, can land with zero visibility down to the runway with precision approach equipment guiding the aircraft on predetermined glideslopes, weather radar for avoiding convective clouds and thunderstorms (and mapping terrain), have procedures for terrain separation in zero visibility operation and are obviously controlled by radar operators for separation by other traffic, plus anti-collision capabilities via transponder (TCAS).
However, I'm talking about marginal conditions, obviously in zero visibility you wouldn't be able to drive manual, so robo taxies privy of radars should indeed stop. Relying only on vision makes me uncomfortable when marginal, that's my point.
That's key. You live where these cars have been designed, even trained. Your roads are another thing, where cities are built around roads, not the other way 'round. Try driving in Europe, around ancient cities, coast lines, mountains, etc.