It's a huge mess. It's been going on for hours and there is still no status report on how long it will take to fix this. Fire, Ambulance and police are all affected, and half the regular phone network is also out so there are no alternate paths available. Big fuckup, people will die because of this and the post-mortem will be a very interesting read.
For many business customers they will be contractually obliged to provide a root cause analysis (pretty standard for telcos).
So if they don't publicly out the report, then the odd's of it staying contained out of public eye are pretty darn high.
The only area in which they could curtail it's release would be if some terrorist type factor or related activity that they could class in that catch-all definition. But even then, details will emerge and it would only delay its publication to fulfill their contractual obligations (think mobile telcos and large business that contract network capacity and links from them with lovely contract SLA's).
> What makes you think a telco will ever do a publicly accessible post-mortem after an outage?
The fact that they have a lot of egg on their face after a similar outage in 2012. Heads will roll over this one and politicians will certainly want answers.
In an actual democracy, they would be required to release it to the public, since they work for the public (and provide public services/infrastructure) and thus are accountable to the public.
Unfortunately, I don't know of any such actual democracy.
if it's released to government it's in the public domain as the government serves the public... oh wait :D ... democracy never happend :D i remember now
Is this based on history or just an attitudinal expression? If the latter, it's not helpful to discourage people when they are in the midst of dealing with a problem.
In America there tend to be an alternative number ending in 911 (ex: 202-222-0911 or something similar) for each area - it's the # the software forwards a call too. Is there similar in NL so people who were proactive can reach EMS?
Yes, every township/locality will have a switchboard. In larger cities it could be zone based. But "911" always has a second # that can directly connect you, ending in 911.
Usually for each NPA (the 3 digit # after the area code) anything ending in 911 redirects to that locality's 911.
This was used by griefers on the early net. You say "you wanna call me up huh go ahead my phone number is XXX-XXX-X911" and they get connected with the police.
Do you know of any place these numbers are commonly available? I can't find anything similar on my local municipalitiy's site, but that doesn't mean anything (the site is terrible). I'm curious though, as the published non-emergency numbers don't follow this pattern.
I'm not going to out myself by listing my local PD, but I can see a 9 digit number that, when I've called in the past, routed me to dispatch.
If 911 is down, you should call a specific police station. They will be broken down into zones or districts in larger cities, and I can see phone numbers for zones in two cities I've lived in on their sites.
>Yes, every township/locality will have a switchboard. In larger cities it could be zone based.
That much is obvious. A single switchboard can also cover multiple towns in some circumstances.
>In larger cities it could be zone based. But "911" always has a second # that can directly connect you, ending in 911.
I'll take your word for it since I'm obviously not going to check... I just checked my local PD and I can't find any mention of this whatsoever on their website.
There are a number of numbers (there is a pun in there somewhere), but most people never use them or bother to look them up. At the same time, a good chunk of the phone network in general was down, so knowing the alternate numbers doesn't do you much good if you can't call at all.
One of the two fiber links had been down for three weeks, which went unnoticed because of lack of proper monitoring. When Eurofiber started their maintenance on the second link, they caused an outage that lasted all night.
KPN does al government telco stuff because it used to be semi-public. As such, they have a lot of institutional knowledge behind this kind of thing.
I would not trust any other telco to do this better. It seems to me the issue is the commercialization/privatization of KPN. Instead, we should go back to a more semi-public KPN that is assigned to cover this form of critical infrastructure.
Or alternatively, we should not have any telco be a single point of failure.
This might be your first instinct, but a new contractor would have to gain the knowledge KPN did on preventing these outages, there is no guarantee any other company would have better uptime.
Reminds me of the quote or story that someone rather hired an ops engineer that accidentally dropped a database than someone who didn't. As the first one would go at more length preventing to do it again.
I was playing SimCity, extremely immersed in the game, and I'd just bulldozed a road that had a power line going across it, then the power in the actual house I was in suddenly went out! I was stunned, and regretted what I'd just done, afraid it was my fault.
In the first emergency alert a completely wrong alternative number being broadcasted, plus a spelling error in the police station. After an hour the first mistake was corrected. I assume the authorities are collectively hanging in the curtains under the influence of a wildly spread panic attack.
Ugh. That is really sloppy, it also conditions people to ignore such errors in the future which will make it much easier to get them to believe all kinds of crap.
I just got an "NL Alert" about this, some kind of faux SMS sent straight to my phone. With a WhatsApp number, an alternative phone number, and a Twitter account. They seemed pretty well prepared. Emergency service via Twitter! Call me impressed if they can pull that off.
They messed up. The WhatsApp number they included in the first NL-Alert is the phone number of the local newspaper De Telegraaf. It’s not owned by the emergency services.
They just sent a new NL-Alert with the right WhatsApp number.
The fact that they sent this specific number - the anonymous tip line of the largest Dutch newspaper - probably means that someone in the emergency taskforce team was actively leaking information to the press...
The third alert included the Whatsapp number and twitter address for the police.
There have been three alerts in the past hour. I'm hearing other people's loud phone alarms go off outside, several minutes after mine, so they weren't all sent out at the same time.
It is the same kind of format as the Amber Alert things I used to get in the states. How do they push that out? Based on location, cell tower usage, sim card?
All of these kinds of alerts (amber, tornado, flood, fire, and presidential test) that I've received on the west coast of the USA have been based on very coarse regions e.g. entire cities, counties, or districts. Whether that is cell tower or not I can't tell, but I've always assumed it was.
Yes, amber alerts are one class of alert that uses WEA. It's based on the mobile station (phone's) location. The rules used to be written allowing per-tower granularity, but new FCC rules require 0.1 mile accuracy to whatever geo-bounding the author of the alert sets. The bounding can be specified as a circle, polgyon, or by geocode. Geocodes specify a state (or all), county (or all). Geocodes for EAS (the alert system on FM radio) allowed for county subdivisions drawn by the NWS. I don't actually know if those made it into WEA.
Yes, access to the base stations in the network still goes through the provider and apparently that is the layer that has problems with interfacing to the outside world. Intra KPN traffic works just fine, including SMS.
You are of course correct. But getting into the details behind MVNOs and roaming is a much more complicated topic than I'd want to type into a small comment box, and not really relevant to the issue at hand. :)
My point is just that CMAS-style cell broadcasts need to be repeated by each cell site, and cell sites aren't universally shared by all carriers.
It appears the problem here was at a higher level, the signalling fabric that transports the data for call setup, teardown, roaming and SMS, and to complicate matters even further apparently the 4G network is where the problem originated but it had effects far outside that. That's also why the whole thing appeared so patchy from outside.
It is and it isn't. The provider is still in the loop for the signalling portion and that happens to be the part that does not work. So likely all these alerts will appear after the problem has been solved.
This has happened several times over the past couple of years in Belgium as well. Bugs and mishaps happen, and I'm sure the systems were designed to be fault tolerant. But perhaps it's time to stop relying on POTS alone for emergency numbers? Is there any particular reason we can't reach emergency services over the internet? Even something as banal as Skype or Facetime would make for a decent backup.
It's the other way round: POTS is still the most reliable thing we have. The "intelligent" services in call and dispatching centers are what fail.
The more the telephony services are mingled with Internet, the more they are exposed to both Internet security threats and bungle-ups created when applying the various patches and updates that are made necessarily by the constant stream of security updates. Plain POTS has less of that.
It's true that if you put enough quality and maintenance into a system, it can stay at ~100% uptime. But very few organizations have those kinds of resources, and government systems tend to not be funded more than they need to be, and generally aren't staffed with state of the art tech teams.
But it's not unreasonable to imagine the possibility that this can change in the future, if the cost and risk for maintaining such a system were to decrease below a certain threshold.
Redundancy cost money, and most companies do not want to pay for it.
I work for a company with about 40 employees. Our offices are connected to two different ISPs: one by optical fiber and one by VDSL. We have two different telephone operators for our outgoing calls from the offices, and I have two mobile phones with different operators.
During the last five years there have been multiple outages that lasted between minutes and days, but most of the times the redundancy have saved us. But when we lost power (the whole city went dark for hours) we went down when the batteries in the UPS was drained. Having our own generators is too expensive.
We can, we should and in some countries we do, but the increasing reliance of POTS services on internet technology make them vulnerable to the same weaknesses that exist in the internet.
Not in this case, IMS Core was offline for the whole country which is the backing system behind pretty much all fixed infrastructure for POTS, ISDN and VoIP ingress/egress as well.
Well, IMS stands for "IP Multimedia Subsystem" and having IMS in the telephone network is, in my view, basically a part of the process of moving POTS services to IP and Internet technologies... of course, that is necessary for many of the centralized services that phone users are used to, but it is a possible point of failure of epic proportions.
The main challenge is reliably localizing callers to the right 911 dispatch... without allowing anyone to spoof their origin, and without inadvertently allowing location data to leak out to non-emergency calls (or texts, or tweets, or whatever emergency medium this scheme would support).
I doubt that. That's a very critical feature for a small subset of emergency calls, where the victim is the one making the call and is unable to communicate their location. But in the majority of cases, something as simple as a text message like "armed robbery at $address" or "fell down stairs at $location" should suffice.
Maybe that would increase the number of prank calls, but maybe that's worth the price.
You overestimate human rationality in emergency situations. Last time I was calling for an ambulance, I had a hard time giving my own damn home address, and I've been living here for decades.
I guess it varies from place to place. A lot of small town police departments are dispatched from a larger center. In those set ups, one dispatcher will be in charge of dispatching a channel, which represents a geographical area with a certain number of units working at any particular time. Their job is to read reports coming in, ask any questions they have to the call taker, and coordinate all units on their channel. It’s a very hectic job, and an emergency call can easily take more than 30 minutes to complete. They have to be coordinating units constantly, if you’re getting more than one call concurrently, there’s no way a dispatcher could do both.
What I was trying to say was that users are only able contact emergency services through POTS (and, by extension, GSM, obviously). The fact that operators and emergency services use Astrid was sort of beside the point. Although I do believe the last major outage was because of Astrid.
Emergency services are apparantly too important to allow something as 'unreliable' as skype/facetime.
I don't understand why regulators can't drop the reliability requirements and understand that if skype, facetime, whatsapp, viber, twitter, etc. all have 99.9% reliability, then the chances of them all being down at once is far lower than your POTS line being down.
Just have a #911Emergency official hashtag and have a bunch of people reply to it. That's the way the people want it.
2. > the chances of them all being down at once is far lower than your POTS line being down.
That depends crucially on the correlation. If there are common fault modes (such as: internet down), the probability might be way higher. (Remember, it used to be common wisdom that it was possible that one or two home owners won't be able to pay their mortgages, but the probability that many will be unable to pay their mortgages is so low that all these MBS and CDO's can be rated AAA...)
3. EDIT to add: Of course, adding communication options can only increase the probability that you can reach your emergency provider, but there's a reason that Skype etc. used to come with these big disclaimers that they're no replacement for POTS, namely the excellent reliability of POTS.
[1] "Although POTS provides limited features, low bandwidth, and no mobile capabilities, it provides greater reliability than other telephony systems (mobile phone, VoIP, etc.). Many telephone service providers attempt to achieve dial-tone availability more than 99.999% of the time the telephone is taken off-hook. This is an often cited benchmark in marketing and systems-engineering comparisons, called the "five nines" reliability standard."
This is exactly the reason why I find all the gleeful tearing out of old POTS copper lines very distressing. This was a proven system with incredible reliability, which worked in power outages for residential users due to centralized backup systems, and still has plenty of usefulness.
Until we come up with some universal multi-layered standards which provide at least the same level of uptime and utility, it seems more than a little premature to pull all of it up as a matter of course. Obviously in many places there are space constraints, but that is not usually an issue in arguably the most important use case, at home.
I would argue that in the real-world, for emergency use, POTS only has ~60% reliability.
When an emergency happens, what are the chances a POTS phone is nearby? If there isn't a landline within 100 yards of the incident, the POTS network has failed to help.
Now given that, does it really matter if the actual phones have are "3 nines" or "5 nines"?
Internet outages happens too often for any internet based services to be reliable in an emergency. Any larger ISP have several local outages per week. Last week one of the largest mobile carries in Sweden had major problems three days in a row, affecting both telephone and internet.
That's what they did when 112 went down, they just started accepting twitter/instagram/facebook/whatsapp. Problem is that those services can be much harder to manage making them a problem in itself (i.e. spoofing, overloading).
It's not supposed to depend on the ordinary electrical system. In practice that independence isn't always tested as often as it should be, and mistakes happen.
True, but at least it is supposed to. What about "the internet"? Which parts of that will still work when there's a power outage? Not your home Wifi, presumably. Mobile data? For how long?
> What about "the internet"? Which parts of that will still work when there's a power outage? Not your home Wifi, presumably. Mobile data? For how long?
The true-but-unhelpful answer is "depends on your ISP and what contract you have with them". A typical home user's "last mile" is almost certainly less reliable than POTS, agreed. But if you make it onto the backbone the internet is better at routing around disruption than the circuit-oriented phone system. So I wouldn't want to count on the phone system always being more reliable.
Tens of minutes on average. Most BTSes are battery-backed, without generators.
(Home wifi could work, but the usefulness of an island system is limited. Outer link also worked last time, but what good is that when the other end is not even on UPS? Next hop was optical, but who knows how that is powered and whence.)
Generally, no, but if you have a mains-powered wireless phone as your only phone (as my grandparents have for over a decade, it essentially is. This isn't necessarily something you can rely on (unless you specifically plan ahead for this).
It is, we just use batteries. Anyway I'm no expert of that, I just see batteries when I enter a BTS, but never checked if there was a separate power line or anything. I'm talking about one ISP in Spain though.
We are talking about a countries 911 service going down and killing people and you suggest a lower (3 nines) services as a better replacement oh and a back up of a *f&^ing Hashtag!
Having the Dutch government nationalise the 911 service with no compensation to the share holders would be a start.
I believe they meant the new communication as an additional option, not a replacement. It's not "now I have to install Skype", it's "the normal phone line is dead, I've got an emergency, now I've got an option to try Skype instead of possibly just dying".
If that's the case, they shouldn't be using the possibility of using skype as an excuse to reduce the availability requirements for getting ahold of 911
Local police departments very often have social media presence
After that you have people you know will be available to come to where you are, email or im
After that you have people you know will answer like your manager, the front desk at work, even filing a support ticket at work, the whole goal being getting the attention of someone who knows you
My girlfriend works as a GP. One of her colleagues was on a visit and had to call an ambulance. She was able to do so by reaching out to the police via Facebook.. (not sure how the address was communicated and what the privacy impact is of that)
Internet outages are much more common than phone outages. Having two independent systems available would be ideal, but governments would never go for that. It's better to rely on the phone network.
The alternate path is whatsapp and twitter as mentioned in the alerts. Or ask the person with t-mobile/vodafone connections to call the alternative number. Though I do agree about the other parts.
My empirical study for this is as follows:
I have 7 phones on and around my desk, of which are 3 iOS devices (4s and newer), and 4 android (4.4 and up all different manufacturers, sony, samsung, hmd, one plus brand) with different or no operator sim installed and all but the 4s got it.
I guess it is cell related in that case.
edit: the 4s was subscribed to the kpn network. Others were not or I can no longer check. Strange thing is I'm getting them first through my personal phone's second sim (vodafone, primary is t-mobile).
Are those on KPN? If so then that is interesting, if not then that is pretty much in line with the article. SMS and voice traffic within the KPN network is just fine, going outside or dialing in is not working and SMS traffic from the outside in also does not work (so far).
No SIM is only interesting in sofar as those phones have attached themselves to the KPN network, they could choose any network for emergency services.
My wife and I both have iPhones on KPN. No alert. We have an old Moto X 2013 that we use when we have to check our old German T-Mobile numbers for missed calls or SMSes. That phone did get an alert. Thrice.
It is worrying that these alerts do not seem to work correctly, since it is supposed to replace the 'the air-raid alarm' (in 2021).
It's also bad that these messages are pretty annoying _and_ spammed to the point where people are turning the NL alert messages off. Last week too, there was a big fire in the region I'm from, and we received ~5 messages through the NL Alert system warning us about the fire and telling us to keep windows closed in the space of 20 minutes. Seems excessive to me and dangerous should the boy at one point cry about an actual wolf.
Is it anything like the American system, which overrides your phone’s volume settings? I was driving in the US last year in a pretty amazing thunderstorm and a hideous alarm went off in the car. I thought I’d crashed the thing.
Pulling over and investigating, we discovered my phone ‘alerting’ me to the heavy rain. Err ... cheers. If it did that more than about twice a year I’d be turning it off for sure.
Ofcourse, I am not advocating for turning the system off. But I would tread more cautiously with something like this if a) it is very easily turned off if perceived as of net negative value in a spur of the moment decision, but difficult to opt back in, b) the security of the system as a whole depends on at least a few people in any given group at any time of day receiving the messages, especially if these technical issues persist with random arrivals of message delivery and c) the scope of the system is expected to expand in the near future.
You have a very valid point. Systems like this should work flawlessly and should only be activated when it is really necessary otherwise by the time we need them they'll be switched off by people that get tired of being woken up.
Case in point: smoke alarms that wake up half the neighborhood when their batteries need replacement.
My phone is 3G, and it does not work, only internal. Old Nokia, but I have access to lots of other phones and so far all KPN network phones have issues of one form or another.
But interesting that there are 3G/4G differences. Have you tried to switch back?
KPN now reports that the problem is 'only on the 4G network', but I'm pretty sure that that is not the case for all subscribers (I'm exclusively on 3G and also have problems), also, their advice to reconfigure phones to connect to 3G only could easily backfire because the 3G network is not capable of dealing with all the traffic if everybody switches.
I used to work on reliability of the emergency number in South Australia which could stand:
- failure of reserved circuits from outlying exchanges
- failure of trunk exchange
- failure of operators exchange
- evacuation of operators
- failure of backup operator exchange
- evacuation of backup operators
- failure of special circuits to emergency services
- evacuation of emergency services operations rooms
- failure of backup lines to emergency services
- emergency services advertised alternate numbers (spread across multiple exchanges in some cases)
etc etc
Can't work out why just the emergency service number failed not other things as well. Bit surprised.
I find it interesting that they published alternative emergency numbers upon social media and more importantly, push notification via mobile phone networks.
Why couldn't those mobile phone networks of just rerouted the emergency number to the alternative number?
I was referring to the mobile networks able to do the broadcast and questioning why they couldn't of done a number routing for their network to rout all emergency calls to the alternative number. Which for their customer permutations of use, would be far more efficient and indeed safer if you factor in the delay in the user having to read an sms in the middle of a crisis that dictated calling the emergency service in the first place.
It wasn't just the emergency number that was borken. The entire KPN telephony network was having issues. The alternative number could not be reached from phones using the KPN network, so having emergency calls on those phones redirect to the alternative wouldn't help much. Presumably the other networks didn't put routing into place since that'd be a rather large configuration change on short notice to a safety critical system.
Again - the point I was making was.
IF mobile networks were sending out a cell broadcast of an alternative number to use upon `their` networks.
THEN why didn't they just do a routing change upon `their` networks.
`their` networks were working - hence they was able to send out a network broadcast. They equally sent a new emergency number that was not upon KPN and as such worked.
As for large network configuration changes - pish, it really isn't that hard and would literally be a simple update upon their network only that they could just as easily change back. Not talking a global change update, just a per network (mobile) that was working (or the ones that were and able to push out a cell notification).
Searched for more technical details but couldn't find anything. Anyone knows who is the main vendor of KPN? Huawei, Ericsson, NokiaSiemens, ZTE?
I bet the supplier is going to feel some 'heat' from this outage.
Huawei runs most of KPN. Bet you we'll never know the real cause just management bllsht. The real reason probably is not spending money on the right places (aka people, redundancy, protocols, upgrades, etc) but on CEO lunches or stuff like that.
Yep, I just received it on my phone, which wasn't fun because I was listening to music, my phone blasted the emergency tone at full volume through my earpods. It really hurt :(
Anyway, good to see that the emergency system works. I just received a second message with a whatsapp number and twitter handle which you can use in case of emergency.
Oh shit -- it was a really loud jarring sound, it must have been painful. They have recently started sending test phone alerts at the same time they perform the regular first Monday of the month noon emergency alert siren tests in Amsterdam. Maybe you should set an alarm just before then to make sure you're not listening to music!
Oh, I'm using this since early 2000s. And I've subscribed to several channels like location, emergency, tourism, football, weather. Most of them don't work. I get location messages regularly but they make SMS like sound. And my LG G5 doesn't have special sound settings in Cell Broatcasting app. So it looks like wheter you have the alert sound depends a bit on what phone/software you are using. I though that there is a new protocol. But anyways, I like cell broatcasting.
I feel like that is a pretty significant issue with the alert system, and could cause legitimate hearing damage. If you have headphones in, it should be assumed that you are paying attention and don't need to be woken up, and play at low volume.
Strange thing is that it didn't send it over my Bluetooth headset as I wasn't listening at the time.
>> it should be assumed that you are paying attention and don't need to be woken up
That assumption is false. Last week we got one for likely dangerous smoke and we had to get out of the smoke and close all windows and stop ventilation in the house. You might want to wake up for that as some people use the headphones and fall asleep.
That's true, it is possible that someone could be sleeping with headphones in or something, but I don't think that is common enough to justify potentially giving people hearing damage. Maybe as a compromise it could play at the current volume (or lowest volume if volume is off) through headphones, rather than going up to full volume.
Aside: Shout out to the telemarketer who pressured me to get a land line years ago because they're the only guaranteed access to emergency numbers and "never go down."
"Eschew flamebait. Don't introduce flamewar topics unless you have something genuinely new to say. Avoid unrelated controversies and generic tangents."
- GRU has been stepping up its attacks on the West (anti-doping, chemical weapons, MH17 investigation)
- US, Britain, and Israel attacked Iran with Stuxnet
- Iran retaliated with cyber attacks on banks and electrical grids
- Iran downed a drone
- Number stations go crazy
- US almost goes for military strike, but Trump calls it off and orders US Cyber Command to cyber-attack Iran, with focus on missile systems and spy networks.
- BGP route leak (Allegheny Technologies Inc. caused the leak, and was a target for Titan Rain around 2014).
- Dutch payment - and emergency response network goes down.
Edit: According to a KPN spokesperson: "We have an indication of the cause, but a lot remains unclear. Anyway, this was not the result of a hack."
I'm not sure you can call this a big fuckup, we will have to wait for news about the cause.
Yes it is bad, very bad. But I'm sure they got all the redundancy in place and thought about so much more than we can imagine. But even then there might be some extreme event that will throw 'soot in the food'.
I think it is just great how this event is handled. Almost everybody I know knows the alternative alarm number. Police is in the streets so you can walk to them. And as far as we know there are no looters taking their chances.
So yeah, bad, but we have to wait before we can mark it as big fuckup.
This event should not have happened in the first place and it was demonstrably mis-handled in several critical ways.
The fact that the police has responded in the proper way has nothing to do with how KPN has handled this so far and it immediately puts in question the retiring of the siren system used for emergencies, which for cost-cutting reasons has been replaced by the national mobile phone alert system, which apparently does not work when you really need it.
> demonstrably mis-handled in several critical ways.
- No fall-back system in place.
- Alternative alert system did not work.
- Alternative alert went out with the wrong information.
That seems plenty to me.
As far as the siren is concerned: the general alert system is meant for serious threats to public safety, it does not require 'a new phone number in morse', the output is a single bit: siren on: stay indoors, close doors and windows, wait for all clear sign (three short bursts on the siren).
Agreed, but 112 not being reachable definitely is. KPN made a big stink about it at the time how other 'crappy' providers could not be trusted to deliver a thing of such national importance and so they got to do it exclusively rather than to have multiple redundant systems by different providers. I'm sure there will be some second guessing of that decision now.
Agreed that sounding the national alarm over this would be an overreaction, as is the fact that they used the national alert, lots of people are going to disable it as a result of this and that will make the system that much less valuable.
No, I just think that if the national alert system is activated that it should work as advertised, deliver each alert once and that everybody should receive it.
As it was some people got no alerts, others got nine and the information dispensed through it was faulty. That's pretty fucked up.
Those are pretty far down on the likelihood scale, though we have had instances of that, especially near river dikes in the last couple of years. We do not have a lot of dams here.
The more likely cause for such an alarm is chemical pollution from either an accidental release or a fire. Their old use was for air raids but those are even more unlikely (at the moment, of course this can change).
The parent is questioning the reliability of the mobile phone alert system as a replacement for the sirens: it has failed for many people (and has so in the past), and as such one can assert that it cannot replace the siren.
