In the entire history of electromechanical switching in the Bell System, no central office was ever out of service for more than 30 minutes for any reason other than a natural disaster, or, on one single occasion, a major fire in NYC.
The AT&T Long Lines system in the 1960s and 1970s had ten regional centers, all independent and heavily interconnected. There was a control center, in Bedminster, NJ, but it just monitored and sent out routing updates every 15 minutes or so. All switches could revert to default routing if needed, which meant that some calls would not get through under heavy load. Most calls would still work.
There was the Mother's Day outage of 1990. That was caused by someone swapping a break statement for a continue statement in some C code that handled the routing, and there was a cascading effect.
Then again. That only affected long-distance service.
I've been thinking about how pretty much no USA infrastructure of today is as reliable as of the last half of the 20th century. Just my imagination, or true? And what does it mean?
Airliners crash far less, road fatalities kept going down till 2010, so taking the broad view of were infrastructure is I’d say at least those got better.
Anecdotally, I remember more electricity interruptions and plumbing issues when I was a kid, but that could be location dependent and I couldn’t quickly find good numbers going back that far.
Edit:
While the phone network didn’t necessarily go down, I frequently got “all circuits busy” when I was a kid. I don’t remember the last time that happened.
I wish we had metrics for utility companies. In my midwestern experience, things have gotten worse. I don’t remember any outages as a kid in the 90s that were over 24 hours aside from the major blackout in the early 2000s. As an adult I’ve experienced several outages greater than 24 hours in both summer and winter months. It’d be nice to be able to measure this.
In USA for example, road design AND vehicle design is directly linked and beholden to NHSTA regulations and policies.
Infrastructure (IE State route roads) are ever trending towards wider lanes and more gentle shoulder. This is precisely due to vehicle industry requirements requiring more vehicle safety features (and thus width and length), height, and shoulder level clearance @ windows.
All these infrastructure and endpoint changes are driven "organically" by the USA trend towards SUVs, but mainly driven by insurance requirements. Insurance and gov't "make out" on safer/roads/vehicles due to (perceivabally) less accidents and road maintenance.
I can't speak to airplanes, but I imagine the fact that far, far, far more people are able to fly today than even 25 years ago should show that the infrastructure has drastically improved.
And tangentially related, it was much easier for anyone to eavesdrop on your conversations.
When it rained, I could pick up my phone and hear conversations from my neighbor on my landline and talk to them without calling.
Not to mention if you were in the same house, you could surreptitiously here conversations by just picking up the phone or getting a device from radio shack that didn’t have a microphone, that you could plug in to another phone outlet.
With analog cellular, you could also buy a receiver from Radio Shack and hack it to pick up the unencrypted signals from cell phones.
Redundancy was needed because individual nodes/machines were more prone to failure. As machines got more and more reliable, having highly redundant infrastructure was seen as an extra cost.
Yes. Electromechanical switching systems were substantially more reliable than their components. How this was done should be understood by anybody designing high-reliability systems today.
"A History of Science and Engineering in the Bell System - Switching Technology 1925-1975" is a readable reference. The Internet Archive has it.[1]
More hardcore: "No. 5 Crossbar"[2]
The Connections Museum in Seattle still has a #5 Crossbar working.[3] Long distance used toll switches, "#4 Crossbar", and there were 202 of them.
#4 and #5 Crossbar machines are collections of stateless microservices, implemented from electromechanical components. The terminology is used in the old books is completely different, but that's what they are. Each service always has at least two servers. The parts that do have state are distributed. The crossbar switches that make actual connections have state, but are dumb - they are told what to do by "markers", which are stateless but can read the state of the crossbars and of other components. Failure of a single crossbar unit can take down less than a hundred lines at most. Other than the crossbars to external lines, everything had alternate routes. Everything has fault detection, with lights and alarm bells.
Error rates were fairly high. In the previous "step by step" system, a good central office misdirected about 1% of calls. With bad maintenance (and those things were high maintenance) that could get much worse. Crossbar was better, maybe 0.1% misdirected calls.
Routing tables in crossbar were mostly static ROMs of one kind or another. Routing consisted of trying a predetermined set of routes, in order. Clunky, but reliable.
Highway construction standards are much higher than they were 30-50 years ago, but it's a mixed bag. Administrative costs are significantly higher. Survey has dramatically improved with GPS. Highway engineering has not improved since about the 90s. Automated machine guidance has significantly improved the potential accuracy of grading operations in th last decade.
It depends on you view this. If you measure by total bit / data / time per customer served divided by the total outage time I would think we are still very reliable in terms of telecoms. It is worth remember we are likely serving 1 to 100 million times of what we did on mobile network 30 years ago.
Just purely in terms of total downtime itself we are less realible. Purely because of the complexity involves.
I worked for Southwestern Bell in the 90s pre-SBC (aka just before remote terminals and dslams became common). COs handled mid-tens of thousands of lines in big cities, for smaller more rural areas they generally covered a single town or less often there were a few in a county where the full county was under 50k people.
In towns, we generally tried to keep loop lengths under 30k feet, but in rural areas that simply wasn’t possible. You’d often find remnants of party line systems in those areas and definitely load coils out the wazzoo. It was “fun” unwinding all that crap to install ISDN circuits and later DSL.
I remember the old hats at the time laughing about VDSL saying “leave it to the nerds to dream up some unrealistic shit where the loop length can be at most 2k feet, where does that exist!?” not realizing a few years later RTs and DSLAMs would mean a significant portion of city and suburban customers would be.
I used to work at BellSouth in outside plant engineering in the early 2000s. That's exactly what it was. Of course, by then any expansion was done via remote terminals and COs were becoming very antiquated.
Didn't take out the entire Chicago area but was probably the worst case scenario for a suburban switch. Hinsdale handled the airports, FAA ATC offices, and the emerging mobile/cellular network. Long distance and 411 was down for whole counties.
This is a pretty good USENET archive/digest of the event, if you can get past the Web0.1 formatting:
Hopefully this get wider coverage and report. One of the top discussion point of 6G was to simplify it both for technical reason and cost reduction reason. 5G brought us enough capacity headroom that most MNOs today still dont find enough incentive to deploy.
May be it doesn't even need to be 6G. With 5.5G and above and OpenRAN there is another opportunity to radically reduce complexity I could only hope there is enough of push towards this.
And centralized. Data is cheap (though they won’t admit that) while big iron cellular core stuff is expensive.
Funny when they billed extra for long distance calls even though all calls were routed through one place for a huge geographic area. Calling your neighbour could be a hundreds of miles round trip over mobile.
Yes. Too much of routing is centralized. Since phone numbers are no longer locative (the area code and exchange number don't map to physical equipment) all calls require a lookup. It's not that big a table by modern standards. Tens of gigabytes. All switches should have a database slave of each telco's phone number routing list, to allow most local calls if external database connectivity is lost. It may be behind, and some roaming phones won't work. But most would get through.
Agreed. Not a justification, but an explanation/excuse as to why systems are less reliable. You're right on the mark -- when reliability doesn't scale with complexity, you get this.
In the entire history of electromechanical switching in the Bell System, no central office was ever out of service for more than 30 minutes for any reason other than a natural disaster, or, on one single occasion, a major fire in NYC.
The AT&T Long Lines system in the 1960s and 1970s had ten regional centers, all independent and heavily interconnected. There was a control center, in Bedminster, NJ, but it just monitored and sent out routing updates every 15 minutes or so. All switches could revert to default routing if needed, which meant that some calls would not get through under heavy load. Most calls would still work.