On 2 June, between 16h45 and midnight, Orange’s voice services as well as access to certain emergency services were severely disrupted on a national level in France.
>
During this period, an 11% deterioration in the rate of call routing (or approximately 11,800 calls) to emergency services was observed. The investigation confirms that this was due to a software malfunction.
This malfunction impacted the interconnection between mobile voice and Voice over IP (VoIP) services on the one hand and those hosted on the PSTN switched network (including most emergency numbers) on the other hand. The incident occurred following a network modernization operation that was initiated in early May to increase capacity in response to growing traffic.
The interconnection architecture is based on a platform of “Call Serversâ€. A bug in the Call Server software provoked a malfunction of this platform resulting in the disruption to the service. This bug was activated following the application of standard reconnection commands, disrupting the overall operation of the Call Servers, despite their redundancy over six different sites.
This software failure has now been identified by the Group’s partner and supplier of the equipment concerned, and a fix has been issued.
Orange’s technical teams identified the software malfunction immediately thanks to the internal alert systems. They initiated the first technical operations but were unfortunately unable to restore the service.
The complexity of the malfunction, the variety of technologies involved and the specific network architectures of the emergency services delayed the diagnosis. Once the diagnostic had been established, the problem was resolved within a few hours thanks to the mobilization of around one hundred experts.
Despite the mobilization of the technical teams, the delay in activating the managerial crisis committee resulted in late communication to all stakeholders.
On 3 June, in agreement with the Inter-ministerial Crisis Committee, Orange set-up a dedicated structure aimed at dealing with residual local situations reported by the Prefectures (French regional authorities) on a case-by-case basis. This structure was kept on standby until Monday, 7 June.
The investigation recognizes the immediate involvement and strong mobilization of the Group’s technical teams from the onset of the disruption, as well as the collaboration with authorities in order to restore normal service.
It nevertheless underlined the need for faster dissemination of information to the various stakeholders concerned: public authorities, emergency services and the media. ■