A Postmortem report will follow below for the short outage on March 1st 2024:

M2-2024 – Incident Report: Central network traffic flow issue, 2024-03-01

All times below refer to 24-hour clock and Central European Time, CET

Intility experienced an issue affecting network traffic for customers on Friday, March 1, 2024.

User impact and scope of impact
The issue potentially affected all network traffic for users at customers on Intility, hence affecting access to Internet, central services and business web services.

Incident start date and time
Friday, March 1, 2024, at 13:14.

Incident end date and time
Friday, March 1, 2024, at 13:28.

Total incident time
14 minutes.

Cause of events
A software error in the firewall operating system caused issues with the network traffic flow. This led to a failover to the passive unit in the redundant firewall pair. The software error caused issues with processing network traffic that passed through the affected component. As a result of the problems, an automatic failover was performed, routing all traffic to the secondary device in the redundant firewall pair. During this failover, all network traffic potentially experienced a network outage. The issue was resolved when network traffic was re-established after failover to the passive unit in the redundant firewall pair. As a part of this procedure, an automatically restart of affected unit was performed.

Measures
After the incident, Intility opened a ticket to the relevant vendor, who identified the root cause to a known software error. This software error is remediated in newer versions of the firewall operating system, and Intility performed a version upgrade in notified maintenance work on March 8, 2024, between 00:00 and 01:00.

Sequence of events
Friday, March 1, 2024:
• 13:14: Intility’s monitoring service detects and alerts about loss of communication towards certain services. Interdisciplinary troubleshooting is initiated. The issue is categorized with highest criticality.
• 13:28: An automatically failover to passive unit in the redundant firewall pair is performed. Network traffic is re-established on the now active unit.
• 13:40: Company contacts are notified about the experienced issue: Intility experienced a central network issue which affected access to Internet and central services. The issue is now resolved and Intility is monitoring the situation. Incident report will follow. • 14:30: Logg files are retrieved and a support case with the relevant vendor is opened to identify the root cause of the experienced issue.