Below will follow a postmortem report from our hosting partner (Intility) about the disturbance on the 30th of March 2024.

‌

User impact and scope of impact
The issue affected access to Internet-dependent services, i.e. services and service deliveries via Internet, such as connections to Azure and VPN-connections to certain customer locations.

Incident start date and time
Saturday, March 30, 2024, at 02:45.

Incident end date and time
Saturday, March 30, 2024, at 04:00.

Total incident time
1 hour, and 15 minutes.

Cause of events
Intility delivers redundant Internet connection via an Internet Service Provider (ISP) and the link was up during the incident. The incident was caused by defect hardware in the ISP-s core network. The defect core network component caused loss of communication for Internet traffic. The issue was temporarily resolved by Intility re-routing affected internet traffic until the ISP had resolved the central issue by replacing the defect component.

Measures
Intility is in dialogue with the relevant Internet service provider to ensure notification of similar issues to reduce impact at any similar incidents in the future.

Sequence of events
Saturday, March 30, 2024:
• 01:53: Intility register that one of two redundant connections for central internet traffic goes down. No services are affected by the issue.
• 02:15: The affected connection is reestablished.
• 02:45: Intility’s monitoring system alerts about loss of contact with Internet-dependent services. This includes connections to Azure and to certain customer locations. The Intility response team is alerted via live chat. Troubleshooting is initiated and the issue is categorized with highest criticality.
• From 02:48: Intility receives a few inquiries from affected customers. Users are followed up in each issued case throughout the incident.
• 03:15: Intility publish an announcement about the ongoing issue on the portal My Intility.
• 04:00: Intility performs a manual shutdown of the affected connection. This resolves the problem and verification of normal functionality of affected services is initiated.
• 04:25: Normal functionality of affected services is verified.
• 04:26: Primary Contacts are notified by SMS about the experienced issue: Intility experienced central issues that affected services and service deliveries over the internet, such as connections to Azure and certain customer locations. The problem has now been resolved, and our internet provider has been contacted for root cause identification. An incident report will follow.
• 04:50: Intility opens a ticket to the relevant ISP for identification of the root cause of the issue.

Monday, April 1, 2024:
• 10:54: The previously shutdown connection for central internet traffic is re-activated.