Cosmic Global Status - Incident history https://status.as30456.net Cosmic Global Fri, 3 Apr 2026 05:32:47 +0000 Network Outage Type: Incident Duration: 37 minutes Affected Components: London, United Kingdom, Dallas, Texas, Los Angeles, California, Amsterdam, Netherlands, Ashburn, Virginia, , Network Points of Presence → Apr 3, 05:32:47 GMT+0 - Investigating - We are currently investigating this incident. Apr 3, 05:41:05 GMT+0 - Monitoring - We implemented a fix and are currently monitoring the result. Apr 3, 05:50:54 GMT+0 - Monitoring - All PoPs have recovered except for LAX. We are working on restoring traffic to LAX. Apr 3, 06:09:40 GMT+0 - Resolved - This incident has been resolved. Apr 3, 06:32:20 GMT+0 - Postmortem - At approximately 1:25AM EST during a routine filter re-deployment (as a hotfix) to address concerns visualized through our monitoring, the deployment process of our Luascript code went haywire. This was not due to any code logic or bugs introduced into the code, as we redeployed the hotfix moments after without issue, but what we believe to be an extremely unlucky software bug that cascaded through all PoPs due to the way the syncing system works. This was not a full outage, but it is what we would classify as a major outage with more than 80% of traffic being dropped at the time. The issue was 95% resolved within the first 10 mins by bringing back up Ashburn, Dallas, London and Amsterdam without issue. Los Angeles took about 12 minutes extra due to needing to reboot the filtering appliances there, and gradually shifting traffic back online. Frankfurt was not affected during this time and its traffic was still flowing through the network. We are currently investigating why this specific filter deployment went haywire, as we have deployed code over 30 times in the month of March without a single issue, so the situation is obviously one of concern to our customers, and we acknowledge that and are investigation due to it's how peculiar this situation was. We do also acknowledge how the 3 outages within the last 30 days are a significant cause for concern, and we do not want to make excuses, but we do want to iterate that all 3 outages have been caused by things out of our control at the time, but we are currently implementing ways to control them. We do understand that to you, our customers, these are in our control as you trust us to maintain the stability of the network you use, so we take full responsibility for the outages even if they weren't directly caused by us, and are making it our goal to implement increased redundancy and resiliency. We are currently in the process of shipping out upgraded filtering hardware & software, upgraded routers and upgraded router components to critical locations to not only introduce further redundancy, but as previously mentioned to improve resliency overall. We will have more to share on this in the coming weeks, but rest assured we are working on the issues, and we completely understand your frustrations as a customer. Do not hesitate to voice any concerns to us, and we will be glad to respond. Please also take a look at the RFO for the outage in Dallas last week and our plans for the future: <https://status.as30456.net/cmn9oqp8405n49b3k562hrh6i> Type: Incident

Duration: 37 minutes

Affected Components: , , , , ,

<p><small>Apr <var data-var='date'> 3</var>, <var data-var='time'>05:32:47</var> GMT+0</small><br><strong>Investigating</strong> - We are currently investigating this incident. .</p> <p><small>Apr <var data-var='date'> 3</var>, <var data-var='time'>05:41:05</var> GMT+0</small><br><strong>Monitoring</strong> - We implemented a fix and are currently monitoring the result..</p> <p><small>Apr <var data-var='date'> 3</var>, <var data-var='time'>05:50:54</var> GMT+0</small><br><strong>Monitoring</strong> - All PoPs have recovered except for LAX. We are working on restoring traffic to LAX..</p> <p><small>Apr <var data-var='date'> 3</var>, <var data-var='time'>06:09:40</var> GMT+0</small><br><strong>Resolved</strong> - This incident has been resolved..</p> <p><small>Apr <var data-var='date'> 3</var>, <var data-var='time'>06:32:20</var> GMT+0</small><br><strong>Postmortem</strong> - At approximately 1:25AM EST during a routine filter re-deployment (as a hotfix) to address concerns visualized through our monitoring, the deployment process of our Luascript code went haywire. This was not due to any code logic or bugs introduced into the code, as we redeployed the hotfix moments after without issue, but what we believe to be an extremely unlucky software bug that cascaded through all PoPs due to the way the syncing system works. This was not a full outage, but it is what we would classify as a major outage with more than 80% of traffic being dropped at the time. The issue was 95% resolved within the first 10 mins by bringing back up Ashburn, Dallas, London and Amsterdam without issue. Los Angeles took about 12 minutes extra due to needing to reboot the filtering appliances there, and gradually shifting traffic back online. Frankfurt was not affected during this time and its traffic was still flowing through the network. We are currently investigating why this specific filter deployment went haywire, as we have deployed code over 30 times in the month of March without a single issue, so the situation is obviously one of concern to our customers, and we acknowledge that and are investigation due to it's how peculiar this situation was. We do also acknowledge how the 3 outages within the last 30 days are a significant cause for concern, and we do not want to make excuses, but we do want to iterate that all 3 outages have been caused by things out of our control at the time, but we are currently implementing ways to control them. We do understand that to you, our customers, these are in our control as you trust us to maintain the stability of the network you use, so we take full responsibility for the outages even if they weren't directly caused by us, and are making it our goal to implement increased redundancy and resiliency. We are currently in the process of shipping out upgraded filtering hardware & software, upgraded routers and upgraded router components to critical locations to not only introduce further redundancy, but as previously mentioned to improve resliency overall. We will have more to share on this in the coming weeks, but rest assured we are working on the issues, and we completely understand your frustrations as a customer. Do not hesitate to voice any concerns to us, and we will be glad to respond. Please also take a look at the RFO for the outage in Dallas last week and our plans for the future: <https://status.as30456.net/cmn9oqp8405n49b3k562hrh6i>.</p> ]]> Fri, 3 Apr 2026 05:32:47 +0000 https://status.as30456.net/incident/cmnigw4ki0ejswkl8ij6nn44f https://status.as30456.net/incident/cmnigw4ki0ejswkl8ij6nn44f DFW Outage Type: Incident Duration: 12 hours and 8 minutes Affected Components: Dallas, Texas Mar 28, 04:43:41 GMT+0 - Monitoring - The incident was resolved shortly after onset, and we have been monitoring since. A full RFO will be posted when this status is closed. In short, the root cause was a cascading failure triggered by a bug in the routing software itself — not by any action taken by our team. The issue was entirely outside our control, and we responded quickly to restore normal operation. Mar 28, 14:03:21 GMT+0 - Resolved - This incident has been resolved. Mar 28, 01:55:00 GMT+0 - Investigating - We are currently investigating this incident. Mar 28, 02:08:45 GMT+0 - Identified - We identified the root cause, and have been working on resolution. Recovery efforts are showing progress as traffic is beginning to restore in Dallas. Apr 3, 06:31:01 GMT+0 - Postmortem - ### Our explanation: Over the past few weeks, some of you may have experienced brief service interruptions across parts of our network. We want to be upfront about what happened, what we've learned, and most importantly what we've done about it. Our edge routers have always been built with internal redundancy in mind - redundant supervisors, redundant power supplies, multiple line cards, and redundant fabric modules. That level of hardware resilience handles the vast majority of failure scenarios well. However, the recent outages exposed a gap: when an issue affects the chassis itself such as a software defect, a firmware upgrade that requires a full reload, or cases like where a software process on a router crashes (as has happened recently in London -> Twice) - there was no second device to immediately absorb the traffic. The router was redundant in every way except the one that mattered in these incidents. ### What we're doing: We're rolling out a dual router design across all six of our points of presence — Dallas, Ashburn, Los Angeles, London, Amsterdam, and Frankfurt. Once complete, every PoP will operate with two independent edge routers in an active/active configuration, with full BGP session redundancy to all upstream and peering partners. If an entire chassis needs to be taken offline for maintenance, a software upgrade, or an unexpected failure then traffic will automatically reconverge on the second device with no customer-facing impact. Each router in the pair will run on independent power feeds with independent management and control planes. We're also using this as an opportunity to standardize failover testing procedures across all PoPs, so this architecture is validated continuously, not just at deployment. This also provides protection against cases where a configuration change (with possibly human error involved) leads to a change which ends up knocking out a bunch of traffic. The investments for these changes were made during the last couple of weeks, so were already in the works and unrelated to incidents in March, but with summer right around the corner we wanted to let you know you'll be in good hands. These changes will also allow for things like DDoS mitigation changes to be performed in a more controlled rollout (e.g. to parts of traffic only), zero-downtime maintenance windows for core networking equipment and a stronger foundation for the capacity expansions we have planned for the rest of 2026. Type: Incident

Duration: 12 hours and 8 minutes

Affected Components:

<p><small>Mar <var data-var='date'> 28</var>, <var data-var='time'>04:43:41</var> GMT+0</small><br><strong>Monitoring</strong> - The incident was resolved shortly after onset, and we have been monitoring since. A full RFO will be posted when this status is closed. In short, the root cause was a cascading failure triggered by a bug in the routing software itself — not by any action taken by our team. The issue was entirely outside our control, and we responded quickly to restore normal operation..</p> <p><small>Mar <var data-var='date'> 28</var>, <var data-var='time'>14:03:21</var> GMT+0</small><br><strong>Resolved</strong> - This incident has been resolved..</p> <p><small>Mar <var data-var='date'> 28</var>, <var data-var='time'>01:55:00</var> GMT+0</small><br><strong>Investigating</strong> - We are currently investigating this incident. .</p> <p><small>Mar <var data-var='date'> 28</var>, <var data-var='time'>02:08:45</var> GMT+0</small><br><strong>Identified</strong> - We identified the root cause, and have been working on resolution. Recovery efforts are showing progress as traffic is beginning to restore in Dallas. .</p> <p><small>Apr <var data-var='date'> 3</var>, <var data-var='time'>06:31:01</var> GMT+0</small><br><strong>Postmortem</strong> - ### Our explanation: Over the past few weeks, some of you may have experienced brief service interruptions across parts of our network. We want to be upfront about what happened, what we've learned, and most importantly what we've done about it. Our edge routers have always been built with internal redundancy in mind - redundant supervisors, redundant power supplies, multiple line cards, and redundant fabric modules. That level of hardware resilience handles the vast majority of failure scenarios well. However, the recent outages exposed a gap: when an issue affects the chassis itself such as a software defect, a firmware upgrade that requires a full reload, or cases like where a software process on a router crashes (as has happened recently in London -> Twice) - there was no second device to immediately absorb the traffic. The router was redundant in every way except the one that mattered in these incidents. ### What we're doing: We're rolling out a dual router design across all six of our points of presence — Dallas, Ashburn, Los Angeles, London, Amsterdam, and Frankfurt. Once complete, every PoP will operate with two independent edge routers in an active/active configuration, with full BGP session redundancy to all upstream and peering partners. If an entire chassis needs to be taken offline for maintenance, a software upgrade, or an unexpected failure then traffic will automatically reconverge on the second device with no customer-facing impact. Each router in the pair will run on independent power feeds with independent management and control planes. We're also using this as an opportunity to standardize failover testing procedures across all PoPs, so this architecture is validated continuously, not just at deployment. This also provides protection against cases where a configuration change (with possibly human error involved) leads to a change which ends up knocking out a bunch of traffic. The investments for these changes were made during the last couple of weeks, so were already in the works and unrelated to incidents in March, but with summer right around the corner we wanted to let you know you'll be in good hands. These changes will also allow for things like DDoS mitigation changes to be performed in a more controlled rollout (e.g. to parts of traffic only), zero-downtime maintenance windows for core networking equipment and a stronger foundation for the capacity expansions we have planned for the rest of 2026..</p> ]]> Sat, 28 Mar 2026 01:55:00 +0000 https://status.as30456.net/incident/cmn9oqp8405n49b3k562hrh6i https://status.as30456.net/incident/cmn9oqp8405n49b3k562hrh6i Brief Blip of Users Routing Through AMS Type: Incident Duration: 2 hours and 35 minutes Affected Components: Amsterdam, Netherlands Mar 26, 18:37:00 GMT+0 - Investigating - We are currently investigating a brief, but sizable blip of traffic that occurred for users routing through Amsterdam. Mar 26, 19:31:19 GMT+0 - Monitoring - We located the problem to be a private transport link between AMS and LON had caused the drop due to issues outside of our control. The resulting drop was because of the traffic shifting over. As a result of the private transport link failure, we have switched the mode of traffic for this transport, and do not expect it to occur again, however we are monitoring the situation in the event it does. We have also linked this private link to a few other recent small drops in the EU region (unrelated to the last status post which was a hardware issue), and apologize for any impact from that. In regards to network stability, our London PoP will be undergoing an upgrade to improve a routing appliance constraint, and furthermore we will be implementing increased redundancy across all PoPs in the near future to increase resiliency during both planned and unplanned outages. Mar 26, 21:11:36 GMT+0 - Resolved - This incident has been resolved. Type: Incident

Duration: 2 hours and 35 minutes

Affected Components:

<p><small>Mar <var data-var='date'> 26</var>, <var data-var='time'>18:37:00</var> GMT+0</small><br><strong>Investigating</strong> - We are currently investigating a brief, but sizable blip of traffic that occurred for users routing through Amsterdam..</p> <p><small>Mar <var data-var='date'> 26</var>, <var data-var='time'>19:31:19</var> GMT+0</small><br><strong>Monitoring</strong> - We located the problem to be a private transport link between AMS and LON had caused the drop due to issues outside of our control. The resulting drop was because of the traffic shifting over. As a result of the private transport link failure, we have switched the mode of traffic for this transport, and do not expect it to occur again, however we are monitoring the situation in the event it does. We have also linked this private link to a few other recent small drops in the EU region (unrelated to the last status post which was a hardware issue), and apologize for any impact from that. In regards to network stability, our London PoP will be undergoing an upgrade to improve a routing appliance constraint, and furthermore we will be implementing increased redundancy across all PoPs in the near future to increase resiliency during both planned and unplanned outages..</p> <p><small>Mar <var data-var='date'> 26</var>, <var data-var='time'>21:11:36</var> GMT+0</small><br><strong>Resolved</strong> - This incident has been resolved..</p> ]]> Thu, 26 Mar 2026 18:37:00 +0000 https://status.as30456.net/incident/cmn7ui65m02iqzuwrzrj7mvg8 https://status.as30456.net/incident/cmn7ui65m02iqzuwrzrj7mvg8 London Connectivity Issues Type: Incident Duration: 7 hours and 46 minutes Affected Components: London, United Kingdom, , Network Points of Presence → Mar 23, 16:46:14 GMT+0 - Resolved - Following emergency maintenance yesterday that required a reboot of a core router in our London facility, an Arista runtime software bug caused the router's ARP entries to gradually decay from active memory. Although the router's configuration remained correct throughout, the hardware chip (ASIC) responsible for directing network traffic failed to correctly reload the address mappings after the reboot. These mappings are what tell the router how to reach a set of internal endpoints used for multicast traffic forwarding. With them missing from the hardware's active memory, traffic that should have been flowing through those paths was silently dropped. Because the configuration itself was never corrupted, the root cause was not immediately obvious. A number of other potential causes were investigated before the true issue was identified — a desync between the router's stored configuration and what the hardware had actually loaded into memory. We sincerely apologize for the impact this had on your services and for the time it took to identify the root cause. We understand how frustrating extended investigations can be, and we appreciate your patience while our engineers worked methodically through the contributing factors to reach a definitive resolution. Mar 23, 09:00:04 GMT+0 - Identified - We have identified an issue with our filtering software in our London PoP and are working on a resolution as quickly as possible. Mar 23, 10:40:23 GMT+0 - Monitoring - We have rolled out a batch of fixes and are seeing connectivity recover. We are continuing to monitor the situation closely. Mar 23, 11:47:07 GMT+0 - Identified - We are continuing to monitor reports of elevated issues and are still working towards a permanent resolution. Mar 23, 14:10:22 GMT+0 - Identified - We are continuing to investigate TCP issues in the London PoP. We apologize for the continued problems today and are making progress toward a full resolution for this location. Mar 23, 15:39:38 GMT+0 - Monitoring - We have implemented another round of fixes and connectivity is recovering. Please reach out to us if you are still having issue while we continue to monitor. Thank you again for your patience in this matter. We will provide a full report when we confirm all is well. Type: Incident

Duration: 7 hours and 46 minutes

Affected Components: ,

<p><small>Mar <var data-var='date'> 23</var>, <var data-var='time'>16:46:14</var> GMT+0</small><br><strong>Resolved</strong> - Following emergency maintenance yesterday that required a reboot of a core router in our London facility, an Arista runtime software bug caused the router's ARP entries to gradually decay from active memory. Although the router's configuration remained correct throughout, the hardware chip (ASIC) responsible for directing network traffic failed to correctly reload the address mappings after the reboot. These mappings are what tell the router how to reach a set of internal endpoints used for multicast traffic forwarding. With them missing from the hardware's active memory, traffic that should have been flowing through those paths was silently dropped. Because the configuration itself was never corrupted, the root cause was not immediately obvious. A number of other potential causes were investigated before the true issue was identified — a desync between the router's stored configuration and what the hardware had actually loaded into memory. We sincerely apologize for the impact this had on your services and for the time it took to identify the root cause. We understand how frustrating extended investigations can be, and we appreciate your patience while our engineers worked methodically through the contributing factors to reach a definitive resolution..</p> <p><small>Mar <var data-var='date'> 23</var>, <var data-var='time'>09:00:04</var> GMT+0</small><br><strong>Identified</strong> - We have identified an issue with our filtering software in our London PoP and are working on a resolution as quickly as possible..</p> <p><small>Mar <var data-var='date'> 23</var>, <var data-var='time'>10:40:23</var> GMT+0</small><br><strong>Monitoring</strong> - We have rolled out a batch of fixes and are seeing connectivity recover. We are continuing to monitor the situation closely..</p> <p><small>Mar <var data-var='date'> 23</var>, <var data-var='time'>11:47:07</var> GMT+0</small><br><strong>Identified</strong> - We are continuing to monitor reports of elevated issues and are still working towards a permanent resolution..</p> <p><small>Mar <var data-var='date'> 23</var>, <var data-var='time'>14:10:22</var> GMT+0</small><br><strong>Identified</strong> - We are continuing to investigate TCP issues in the London PoP. We apologize for the continued problems today and are making progress toward a full resolution for this location..</p> <p><small>Mar <var data-var='date'> 23</var>, <var data-var='time'>15:39:38</var> GMT+0</small><br><strong>Monitoring</strong> - We have implemented another round of fixes and connectivity is recovering. Please reach out to us if you are still having issue while we continue to monitor. Thank you again for your patience in this matter. We will provide a full report when we confirm all is well..</p> ]]> Mon, 23 Mar 2026 09:00:04 +0000 https://status.as30456.net/incident/cmn31qz0511dmt3c6v3eht9fm https://status.as30456.net/incident/cmn31qz0511dmt3c6v3eht9fm Quick maintenance to resolve packet loss problems in London Type: Maintenance Duration: 1 hour Affected Components: London, United Kingdom Mar 22, 16:00:00 GMT+0 - Identified - Quick maintenance to resolve packet loss problems in London Mar 22, 15:15:19 GMT+0 - Identified - Maintenance is now in progress. Mar 22, 16:15:00 GMT+0 - Completed - Maintenance has completed successfully Type: Maintenance

Duration: 1 hour

Affected Components:

<p><small>Mar <var data-var='date'> 22</var>, <var data-var='time'>16:00:00</var> GMT+0</small><br><strong>Identified</strong> - Quick maintenance to resolve packet loss problems in London.</p> <p><small>Mar <var data-var='date'> 22</var>, <var data-var='time'>15:15:19</var> GMT+0</small><br><strong>Identified</strong> - Maintenance is now in progress..</p> <p><small>Mar <var data-var='date'> 22</var>, <var data-var='time'>16:15:00</var> GMT+0</small><br><strong>Completed</strong> - Maintenance has completed successfully.</p> ]]> Sun, 22 Mar 2026 16:00:00 +0000 https://status.as30456.net/maintenance/cmn1wddx20p8x995p2hmz35sn https://status.as30456.net/maintenance/cmn1wddx20p8x995p2hmz35sn Upstream Issues Type: Incident Duration: 5 hours and 58 minutes Affected Components: London, United Kingdom, Dallas, Texas, Los Angeles, California, Amsterdam, Netherlands, Ashburn, Virginia, , Network Points of Presence → Feb 20, 17:52:00 GMT+0 - Investigating - We are currently investigating this incident. We are seeing issues with one of our upstreams losing announcement of prefixes temporarily. We are working with them in an emergency fashion. You may see blips as traffic fails over and back at times. Feb 20, 23:50:00 GMT+0 - Resolved - Cloudflare resolved their incident. <https://www.cloudflarestatus.com/incidents/kwy3dt82bwbt> Type: Incident

Duration: 5 hours and 58 minutes

Affected Components: , , , , ,

<p><small>Feb <var data-var='date'> 20</var>, <var data-var='time'>17:52:00</var> GMT+0</small><br><strong>Investigating</strong> - We are currently investigating this incident. We are seeing issues with one of our upstreams losing announcement of prefixes temporarily. We are working with them in an emergency fashion. You may see blips as traffic fails over and back at times..</p> <p><small>Feb <var data-var='date'> 20</var>, <var data-var='time'>23:50:00</var> GMT+0</small><br><strong>Resolved</strong> - Cloudflare resolved their incident. <https://www.cloudflarestatus.com/incidents/kwy3dt82bwbt>.</p> ]]> Fri, 20 Feb 2026 17:52:00 +0000 https://status.as30456.net/incident/cmlv8a2wn0009x6oklppemogx https://status.as30456.net/incident/cmlv8a2wn0009x6oklppemogx Upstream Provider Outage Type: Incident Duration: 52 minutes Affected Components: London, United Kingdom, Dallas, Texas, Los Angeles, California, Amsterdam, Netherlands, Ashburn, Virginia, , Network Points of Presence → Feb 18, 15:38:00 GMT+0 - Investigating - One of our upstream providers appears to be having a brief outage related to the services we use with them. We are in contact with them getting it restored, but traffic has already failed over. Feb 18, 16:30:07 GMT+0 - Resolved - This incident has been resolved. Type: Incident

Duration: 52 minutes

Affected Components: , , , , ,

<p><small>Feb <var data-var='date'> 18</var>, <var data-var='time'>15:38:00</var> GMT+0</small><br><strong>Investigating</strong> - One of our upstream providers appears to be having a brief outage related to the services we use with them. We are in contact with them getting it restored, but traffic has already failed over..</p> <p><small>Feb <var data-var='date'> 18</var>, <var data-var='time'>16:30:07</var> GMT+0</small><br><strong>Resolved</strong> - This incident has been resolved..</p> ]]> Wed, 18 Feb 2026 15:38:00 +0000 https://status.as30456.net/incident/cmls7mtae0b0peookj4gf4or0 https://status.as30456.net/incident/cmls7mtae0b0peookj4gf4or0 Connection Drop in EU & US Type: Incident Duration: 18 hours and 3 minutes Affected Components: London, United Kingdom, Dallas, Texas, Los Angeles, California, Amsterdam, Netherlands, Ashburn, Virginia Jan 10, 18:57:07 GMT+0 - Investigating - We are currently investigating this incident. We are aware of a mass disconnection in the EU and US regions primarily effecting the ASH, LAX and LON regions. Jan 11, 01:01:11 GMT+0 - Identified - We identified the issue as a software bug within our filtering stack caused by a specific set of triggers. The issue was quickly under control, and no further impact was seen as a result of the event this morning. We are working on a permanent fix, and will provide an update once the permanent fix has been applied, and monitored. Jan 11, 13:00:00 GMT+0 - Resolved - This incident has been resolved. Type: Incident

Duration: 18 hours and 3 minutes

Affected Components: , , , ,

<p><small>Jan <var data-var='date'> 10</var>, <var data-var='time'>18:57:07</var> GMT+0</small><br><strong>Investigating</strong> - We are currently investigating this incident. We are aware of a mass disconnection in the EU and US regions primarily effecting the ASH, LAX and LON regions..</p> <p><small>Jan <var data-var='date'> 11</var>, <var data-var='time'>01:01:11</var> GMT+0</small><br><strong>Identified</strong> - We identified the issue as a software bug within our filtering stack caused by a specific set of triggers. The issue was quickly under control, and no further impact was seen as a result of the event this morning. We are working on a permanent fix, and will provide an update once the permanent fix has been applied, and monitored..</p> <p><small>Jan <var data-var='date'> 11</var>, <var data-var='time'>13:00:00</var> GMT+0</small><br><strong>Resolved</strong> - This incident has been resolved..</p> ]]> Sat, 10 Jan 2026 18:57:07 +0000 https://status.as30456.net/incident/cmk8o2ssi1eva9wwudrvoei9h https://status.as30456.net/incident/cmk8o2ssi1eva9wwudrvoei9h Widespread Connectivity Issues Type: Incident Affected Components: Transit, London, United Kingdom, Dallas, Texas, Customer Portal, Los Angeles, California, Amsterdam, Netherlands, Ashburn, Virginia Nov 28, 19:18:57 GMT+0 - Postmortem - At approximately 1:43 PM EST, we detected a service anomaly affecting our filter infrastructure. Upon investigation, we determined this incident was a bug related to the earlier issue reported this morning: <https://status.as30456.net/cmiivbxm6004wt518aafiq5dd> This caused a brief drop in connections. Within about 60 seconds, we pushed a hotfix and connections stabilized. Nov 28, 10:00:00 GMT+0 - Resolved - ## Incident Report: DDoS Mitigation Infrastructure Outage **Date:** November 28, 2025 **Duration:** Approximately 2.5 hours **Impact:** Full network unavailability **Summary** On November 28, 2025, our DDoS mitigation infrastructure experienced a service interruption lasting approximately one hour. As all inbound and outbound traffic flows through our mitigation systems, this caused the network to appear entirely offline from the outside. --- ### What Happened Several days ago, we deployed an enhancement to our rate limiting module to improve DDoS mitigation coordination across our globally distributed filtering nodes. After running without issue for several days, a latent memory corruption bug manifested under specific production conditions, causing the filtering nodes to crash. Due to the synchronized nature of our filtering infrastructure—where nodes coordinate with each other to provide consistent global protection—once the issue manifested at one location, it propagated to all locations worldwide within seconds. --- ### Root Cause The enhancement was written in C++, as it extended an existing C++ module. The code contained a memory corruption bug that only surfaced after several days of continuous runtime under production load. This class of bug is notoriously difficult to detect in testing, as it may only manifest under specific memory states that develop over extended periods. --- ### Resolution Upon detecting the issue, our engineering team immediately began a full rewrite of the affected module in Rust. Within one hour, we deployed the rewritten module, restoring full service. Rust is a memory-safe language that eliminates entire categories of bugs at compile time—before code ever runs in production. Unlike C++, where programmers must manually manage memory and the compiler does not verify correctness, Rust's compiler enforces strict ownership and borrowing rules that make memory corruption, buffer overflows, and use-after-free vulnerabilities impossible in safe code. These guarantees are achieved without sacrificing performance; Rust runs as fast as C++ while providing the safety guarantees typically found only in higher-level languages. For some time now, all new features for our DDoS mitigation systems have been developed in Rust. However, because this particular feature extended an existing C++ module, it was originally written in C++ for consistency. --- ### What we're doing to ensure this doesn't happen again. Going forward, we will cease using C++ for any new development in our codebase. All future features—including extensions to existing modules—will be written in Rust to prevent this class of issue from occurring again. We will also be replacing any existing C++ code over the next couple of weeks. --- ### Apology We sincerely apologize for the disruption this caused to your services. We truly appreciate the confidence you place in us to keep your services online and secure. We are committed to learning from this incident and improving our systems to prevent similar issues in the future. Thank you for your understanding and continued support. Nov 28, 12:58:37 GMT+0 - Postmortem - ## Incident Report: DDoS Mitigation Infrastructure Outage **Date:** November 28, 2025 **Duration:** Approximately 2.5 hours **Impact:** Full network unavailability **Summary** On November 28, 2025, our DDoS mitigation infrastructure experienced a service interruption lasting approximately one hour. As all inbound and outbound traffic flows through our mitigation systems, this caused the network to appear entirely offline from the outside. --- ### What Happened Several days ago, we deployed an enhancement to our rate limiting module to improve DDoS mitigation coordination across our globally distributed filtering nodes. After running without issue for several days, a latent memory corruption bug manifested under specific production conditions, causing the filtering nodes to crash. Due to the synchronized nature of our filtering infrastructure—where nodes coordinate with each other to provide consistent global protection—once the issue manifested at one location, it propagated to all locations worldwide within seconds. --- ### Root Cause The enhancement was written in C++, as it extended an existing C++ module. The code contained a memory corruption bug that only surfaced after several days of continuous runtime under production load. This class of bug is notoriously difficult to detect in testing, as it may only manifest under specific memory states that develop over extended periods. --- ### Resolution Upon detecting the issue, our engineering team immediately began a full rewrite of the affected module in Rust. Within one hour, we deployed the rewritten module, restoring full service. Rust is a memory-safe language that eliminates entire categories of bugs at compile time—before code ever runs in production. Unlike C++, where programmers must manually manage memory and the compiler does not verify correctness, Rust's compiler enforces strict ownership and borrowing rules that make memory corruption, buffer overflows, and use-after-free vulnerabilities impossible in safe code. These guarantees are achieved without sacrificing performance; Rust runs as fast as C++ while providing the safety guarantees typically found only in higher-level languages. For some time now, all new features for our DDoS mitigation systems have been developed in Rust. However, because this particular feature extended an existing C++ module, it was originally written in C++ for consistency. --- ### What we're doing to ensure this doesn't happen again. Going forward, we will cease using C++ for any new development in our codebase. All future features—including extensions to existing modules—will be written in Rust to prevent this class of issue from occurring again. We will also be replacing any existing C++ code over the next couple of weeks. --- ### Apology We sincerely apologize for the disruption this caused to your services. We truly appreciate the confidence you place in us to keep your services online and secure. We are committed to learning from this incident and improving our systems to prevent similar issues in the future. Thank you for your understanding and continued support. Type: Incident

Affected Components: , , , , , ,

<p><small>Nov <var data-var='date'> 28</var>, <var data-var='time'>19:18:57</var> GMT+0</small><br><strong>Postmortem</strong> - At approximately 1:43 PM EST, we detected a service anomaly affecting our filter infrastructure. Upon investigation, we determined this incident was a bug related to the earlier issue reported this morning: <https://status.as30456.net/cmiivbxm6004wt518aafiq5dd> This caused a brief drop in connections. Within about 60 seconds, we pushed a hotfix and connections stabilized..</p> <p><small>Nov <var data-var='date'> 28</var>, <var data-var='time'>10:00:00</var> GMT+0</small><br><strong>Resolved</strong> - ## Incident Report: DDoS Mitigation Infrastructure Outage **Date:** November 28, 2025 **Duration:** Approximately 2.5 hours **Impact:** Full network unavailability **Summary** On November 28, 2025, our DDoS mitigation infrastructure experienced a service interruption lasting approximately one hour. As all inbound and outbound traffic flows through our mitigation systems, this caused the network to appear entirely offline from the outside. --- ### What Happened Several days ago, we deployed an enhancement to our rate limiting module to improve DDoS mitigation coordination across our globally distributed filtering nodes. After running without issue for several days, a latent memory corruption bug manifested under specific production conditions, causing the filtering nodes to crash. Due to the synchronized nature of our filtering infrastructure—where nodes coordinate with each other to provide consistent global protection—once the issue manifested at one location, it propagated to all locations worldwide within seconds. --- ### Root Cause The enhancement was written in C++, as it extended an existing C++ module. The code contained a memory corruption bug that only surfaced after several days of continuous runtime under production load. This class of bug is notoriously difficult to detect in testing, as it may only manifest under specific memory states that develop over extended periods. --- ### Resolution Upon detecting the issue, our engineering team immediately began a full rewrite of the affected module in Rust. Within one hour, we deployed the rewritten module, restoring full service. Rust is a memory-safe language that eliminates entire categories of bugs at compile time—before code ever runs in production. Unlike C++, where programmers must manually manage memory and the compiler does not verify correctness, Rust's compiler enforces strict ownership and borrowing rules that make memory corruption, buffer overflows, and use-after-free vulnerabilities impossible in safe code. These guarantees are achieved without sacrificing performance; Rust runs as fast as C++ while providing the safety guarantees typically found only in higher-level languages. For some time now, all new features for our DDoS mitigation systems have been developed in Rust. However, because this particular feature extended an existing C++ module, it was originally written in C++ for consistency. --- ### What we're doing to ensure this doesn't happen again. Going forward, we will cease using C++ for any new development in our codebase. All future features—including extensions to existing modules—will be written in Rust to prevent this class of issue from occurring again. We will also be replacing any existing C++ code over the next couple of weeks. --- ### Apology We sincerely apologize for the disruption this caused to your services. We truly appreciate the confidence you place in us to keep your services online and secure. We are committed to learning from this incident and improving our systems to prevent similar issues in the future. Thank you for your understanding and continued support..</p> <p><small>Nov <var data-var='date'> 28</var>, <var data-var='time'>12:58:37</var> GMT+0</small><br><strong>Postmortem</strong> - ## Incident Report: DDoS Mitigation Infrastructure Outage **Date:** November 28, 2025 **Duration:** Approximately 2.5 hours **Impact:** Full network unavailability **Summary** On November 28, 2025, our DDoS mitigation infrastructure experienced a service interruption lasting approximately one hour. As all inbound and outbound traffic flows through our mitigation systems, this caused the network to appear entirely offline from the outside. --- ### What Happened Several days ago, we deployed an enhancement to our rate limiting module to improve DDoS mitigation coordination across our globally distributed filtering nodes. After running without issue for several days, a latent memory corruption bug manifested under specific production conditions, causing the filtering nodes to crash. Due to the synchronized nature of our filtering infrastructure—where nodes coordinate with each other to provide consistent global protection—once the issue manifested at one location, it propagated to all locations worldwide within seconds. --- ### Root Cause The enhancement was written in C++, as it extended an existing C++ module. The code contained a memory corruption bug that only surfaced after several days of continuous runtime under production load. This class of bug is notoriously difficult to detect in testing, as it may only manifest under specific memory states that develop over extended periods. --- ### Resolution Upon detecting the issue, our engineering team immediately began a full rewrite of the affected module in Rust. Within one hour, we deployed the rewritten module, restoring full service. Rust is a memory-safe language that eliminates entire categories of bugs at compile time—before code ever runs in production. Unlike C++, where programmers must manually manage memory and the compiler does not verify correctness, Rust's compiler enforces strict ownership and borrowing rules that make memory corruption, buffer overflows, and use-after-free vulnerabilities impossible in safe code. These guarantees are achieved without sacrificing performance; Rust runs as fast as C++ while providing the safety guarantees typically found only in higher-level languages. For some time now, all new features for our DDoS mitigation systems have been developed in Rust. However, because this particular feature extended an existing C++ module, it was originally written in C++ for consistency. --- ### What we're doing to ensure this doesn't happen again. Going forward, we will cease using C++ for any new development in our codebase. All future features—including extensions to existing modules—will be written in Rust to prevent this class of issue from occurring again. We will also be replacing any existing C++ code over the next couple of weeks. --- ### Apology We sincerely apologize for the disruption this caused to your services. We truly appreciate the confidence you place in us to keep your services online and secure. We are committed to learning from this incident and improving our systems to prevent similar issues in the future. Thank you for your understanding and continued support..</p> ]]> Fri, 28 Nov 2025 10:00:00 +0000 https://status.as30456.net/incident/cmiivbxm6004wt518aafiq5dd https://status.as30456.net/incident/cmiivbxm6004wt518aafiq5dd Emergency Maintenance to Address TCP Issues in LAX Type: Maintenance Duration: 5 hours and 34 minutes Affected Components: Dallas, Texas, Los Angeles, California, Ashburn, Virginia Nov 27, 18:00:00 GMT+0 - Identified - Maintenance is in progress and has been extended to ASH and Dallas due to BGP reconvergence. Nov 27, 16:00:01 GMT+0 - Identified - Maintenance is now in progress Nov 27, 21:34:18 GMT+0 - Completed - Maintenance has completed successfully. The maintenance was done to fix a software bug in one of our core routers that put it into a bad state. The maintenance was extended due to a side effect from the reboot of the router. Immediately after the router came back up we saw a complete service restoration in LAX, but we began seeing BGP errors between our PoPs. Eventually, this led to multiple pops having a quick, but pretty full drop. We immediately began restarting our BGP daemons on all of the core routers so that inter-pop communications could be restored. We are monitoring the network to ensure no further issues do occur, but based on the results of the reboot, and our subsequent fixes we think we are in good order. Thanks for your patience, and Happy Thanksgiving to those in the U.S.! Type: Maintenance

Duration: 5 hours and 34 minutes

Affected Components: , ,

<p><small>Nov <var data-var='date'> 27</var>, <var data-var='time'>18:00:00</var> GMT+0</small><br><strong>Identified</strong> - Maintenance is in progress and has been extended to ASH and Dallas due to BGP reconvergence..</p> <p><small>Nov <var data-var='date'> 27</var>, <var data-var='time'>16:00:01</var> GMT+0</small><br><strong>Identified</strong> - Maintenance is now in progress.</p> <p><small>Nov <var data-var='date'> 27</var>, <var data-var='time'>21:34:18</var> GMT+0</small><br><strong>Completed</strong> - Maintenance has completed successfully. The maintenance was done to fix a software bug in one of our core routers that put it into a bad state. The maintenance was extended due to a side effect from the reboot of the router. Immediately after the router came back up we saw a complete service restoration in LAX, but we began seeing BGP errors between our PoPs. Eventually, this led to multiple pops having a quick, but pretty full drop. We immediately began restarting our BGP daemons on all of the core routers so that inter-pop communications could be restored. We are monitoring the network to ensure no further issues do occur, but based on the results of the reboot, and our subsequent fixes we think we are in good order. Thanks for your patience, and Happy Thanksgiving to those in the U.S.!.</p> ]]> Thu, 27 Nov 2025 16:00:00 +0000 https://status.as30456.net/maintenance/cmih0ifxr0055129ncshhhdrp https://status.as30456.net/maintenance/cmih0ifxr0055129ncshhhdrp Line card failure Type: Incident Duration: 1 hour and 15 minutes Affected Components: Los Angeles, California Nov 15, 11:33:43 GMT+0 - Postmortem - Line card replacement was carried out shortly after, stabilizing services. Nov 15, 00:50:00 GMT+0 - Investigating - A line card has failed in our LAX router. We are currently investigating this incident. Nov 15, 01:43:38 GMT+0 - Monitoring - We implemented a fix and are currently monitoring the result. Nov 15, 02:05:00 GMT+0 - Resolved - This incident has been resolved. Type: Incident

Duration: 1 hour and 15 minutes

Affected Components:

<p><small>Nov <var data-var='date'> 15</var>, <var data-var='time'>11:33:43</var> GMT+0</small><br><strong>Postmortem</strong> - Line card replacement was carried out shortly after, stabilizing services. .</p> <p><small>Nov <var data-var='date'> 15</var>, <var data-var='time'>00:50:00</var> GMT+0</small><br><strong>Investigating</strong> - A line card has failed in our LAX router. We are currently investigating this incident. .</p> <p><small>Nov <var data-var='date'> 15</var>, <var data-var='time'>01:43:38</var> GMT+0</small><br><strong>Monitoring</strong> - We implemented a fix and are currently monitoring the result..</p> <p><small>Nov <var data-var='date'> 15</var>, <var data-var='time'>02:05:00</var> GMT+0</small><br><strong>Resolved</strong> - This incident has been resolved..</p> ]]> Sat, 15 Nov 2025 00:50:00 +0000 https://status.as30456.net/incident/cmhzm31gi025kzlw0h32h8qtd https://status.as30456.net/incident/cmhzm31gi025kzlw0h32h8qtd Emergency maintenance Type: Maintenance Duration: 15 minutes Affected Components: Dallas, Texas Oct 31, 21:15:00 GMT+0 - Identified - Hi everyone, In 5 minutes we'll need to perform an emergency procedure to our primary dallas router, traffic will likely drop for a few minutes. We apologize for the inconvenience this might bring to you. Oct 31, 21:15:01 GMT+0 - Identified - Maintenance is now in progress Oct 31, 21:30:00 GMT+0 - Completed - Maintenance has completed successfully Type: Maintenance

Duration: 15 minutes

Affected Components:

<p><small>Oct <var data-var='date'> 31</var>, <var data-var='time'>21:15:00</var> GMT+0</small><br><strong>Identified</strong> - Hi everyone, In 5 minutes we'll need to perform an emergency procedure to our primary dallas router, traffic will likely drop for a few minutes. We apologize for the inconvenience this might bring to you. .</p> <p><small>Oct <var data-var='date'> 31</var>, <var data-var='time'>21:15:01</var> GMT+0</small><br><strong>Identified</strong> - Maintenance is now in progress.</p> <p><small>Oct <var data-var='date'> 31</var>, <var data-var='time'>21:30:00</var> GMT+0</small><br><strong>Completed</strong> - Maintenance has completed successfully.</p> ]]> Fri, 31 Oct 2025 21:15:00 +0000 https://status.as30456.net/maintenance/cmhfcidb80192u5ib7zr51wy8 https://status.as30456.net/maintenance/cmhfcidb80192u5ib7zr51wy8 Cloudflare Maintenance Type: Maintenance Duration: 3 days Affected Components: London, United Kingdom, Dallas, Texas, Los Angeles, California, Amsterdam, Netherlands, Ashburn, Virginia Oct 29, 05:00:00 GMT+0 - Identified - Hi all, Cloudflare has a very heavy maintenance schedule going on currently: <https://www.cloudflarestatus.com/> Which may affect connectivity to our network briefly at times. We're publishing this status post as a warning. Oct 29, 05:00:01 GMT+0 - Identified - Maintenance is now in progress Nov 1, 05:00:00 GMT+0 - Completed - Maintenance has completed successfully Type: Maintenance

Duration: 3 days

Affected Components: , , , ,

<p><small>Oct <var data-var='date'> 29</var>, <var data-var='time'>05:00:00</var> GMT+0</small><br><strong>Identified</strong> - Hi all, Cloudflare has a very heavy maintenance schedule going on currently: <https://www.cloudflarestatus.com/> Which may affect connectivity to our network briefly at times. We're publishing this status post as a warning..</p> <p><small>Oct <var data-var='date'> 29</var>, <var data-var='time'>05:00:01</var> GMT+0</small><br><strong>Identified</strong> - Maintenance is now in progress.</p> <p><small>Nov <var data-var='date'> 1</var>, <var data-var='time'>05:00:00</var> GMT+0</small><br><strong>Completed</strong> - Maintenance has completed successfully.</p> ]]> Wed, 29 Oct 2025 05:00:00 +0000 https://status.as30456.net/maintenance/cmhbsahxj01i811bzuhxnewsj https://status.as30456.net/maintenance/cmhbsahxj01i811bzuhxnewsj Partial Server Outage Type: Incident Duration: 7 hours and 21 minutes Affected Components: London, United Kingdom Mar 30, 18:50:00 GMT+0 - Investigating - We are currently investigating this incident. We are aware that some client machines are facing a network outage in London. We have already dispatched our technicians to be onsite, and are working diligently to resolve this issue. Mar 30, 19:06:06 GMT+0 - Identified - A redundant line card has failed in our LON Docklands Core Router, primarily impacting 10GE devices. A replacement is currently underway. Mar 30, 19:17:08 GMT+0 - Monitoring - The affected line card has been replaced using a spare from our on-site inventory. Impacted customer services will begin restoring as the new card completes its boot process. Mar 30, 20:18:31 GMT+0 - Identified - We are aware the outage has been extended, and our techs are still onsite diligently working towards the full fix. We will continue to update. Mar 30, 20:36:46 GMT+0 - Monitoring - The LON Docklands Core Router has fully recovered, with all line cards now online. A supervisor switchover was performed, which caused temporary impact. We are actively monitoring to ensure all services remain stable. Mar 31, 02:11:13 GMT+0 - Resolved - We are closing this issue as it has been 6 hours since the resolution and we have confirmed everything has remained stable since. Type: Incident

Duration: 7 hours and 21 minutes

Affected Components:

<p><small>Mar <var data-var='date'> 30</var>, <var data-var='time'>18:50:00</var> GMT+0</small><br><strong>Investigating</strong> - We are currently investigating this incident. We are aware that some client machines are facing a network outage in London. We have already dispatched our technicians to be onsite, and are working diligently to resolve this issue..</p> <p><small>Mar <var data-var='date'> 30</var>, <var data-var='time'>19:06:06</var> GMT+0</small><br><strong>Identified</strong> - A redundant line card has failed in our LON Docklands Core Router, primarily impacting 10GE devices. A replacement is currently underway..</p> <p><small>Mar <var data-var='date'> 30</var>, <var data-var='time'>19:17:08</var> GMT+0</small><br><strong>Monitoring</strong> - The affected line card has been replaced using a spare from our on-site inventory. Impacted customer services will begin restoring as the new card completes its boot process..</p> <p><small>Mar <var data-var='date'> 30</var>, <var data-var='time'>20:18:31</var> GMT+0</small><br><strong>Identified</strong> - We are aware the outage has been extended, and our techs are still onsite diligently working towards the full fix. We will continue to update..</p> <p><small>Mar <var data-var='date'> 30</var>, <var data-var='time'>20:36:46</var> GMT+0</small><br><strong>Monitoring</strong> - The LON Docklands Core Router has fully recovered, with all line cards now online. A supervisor switchover was performed, which caused temporary impact. We are actively monitoring to ensure all services remain stable..</p> <p><small>Mar <var data-var='date'> 31</var>, <var data-var='time'>02:11:13</var> GMT+0</small><br><strong>Resolved</strong> - We are closing this issue as it has been 6 hours since the resolution and we have confirmed everything has remained stable since..</p> ]]> Sun, 30 Mar 2025 18:50:00 +0000 https://status.as30456.net/incident/cm8w06l3d003m411zgdqxsrxh https://status.as30456.net/incident/cm8w06l3d003m411zgdqxsrxh Latency Spikes Caused by Upstream Carriers Type: Incident Affected Components: London, United Kingdom, Amsterdam, Netherlands Mar 21, 16:00:00 GMT+0 - Resolved - We are aware of the latency spikes that have been occurring since Friday, and have been diligently routing around our upstream carriers in preparation of/during these events. It appears based on our metrics, and the way these events are occurring that our upstream carrier(s) are feeling congestion/latency caused by the large botnet attacks that have been plaguing the internet. We are continuing to monitor the situation, route around where possible, and deploy all possibilities. We will update when needed. Type: Incident

Affected Components: ,

<p><small>Mar <var data-var='date'> 21</var>, <var data-var='time'>16:00:00</var> GMT+0</small><br><strong>Resolved</strong> - We are aware of the latency spikes that have been occurring since Friday, and have been diligently routing around our upstream carriers in preparation of/during these events. It appears based on our metrics, and the way these events are occurring that our upstream carrier(s) are feeling congestion/latency caused by the large botnet attacks that have been plaguing the internet. We are continuing to monitor the situation, route around where possible, and deploy all possibilities. We will update when needed..</p> ]]> Fri, 21 Mar 2025 16:00:00 +0000 https://status.as30456.net/incident/cm8opn9jf000z963jczv8cokf https://status.as30456.net/incident/cm8opn9jf000z963jczv8cokf London/Amsterdam Transport Outage Type: Incident Duration: 9 hours Affected Components: London, United Kingdom, Amsterdam, Netherlands Mar 13, 15:27:00 GMT+0 - Investigating - Mar 13, 15:36:00 GMT+0 - Resolved - We’re experiencing an outage with one of our transport providers between London and Amsterdam, which temporarily affected connectivity to our London PoP for users routed through Amsterdam. We’ve implemented adjustments to reroute traffic around the issue, and no further impact is expected. We sincerely apologize for the disruption and appreciate your patience and understanding. Impact was between 10:27 AM CT and 10:36 AM CT (9 minutes). Type: Incident

Duration: 9 hours

Affected Components: ,

<p><small>Mar <var data-var='date'> 13</var>, <var data-var='time'>15:27:00</var> GMT+0</small><br><strong>Investigating</strong> - .</p> <p><small>Mar <var data-var='date'> 13</var>, <var data-var='time'>15:36:00</var> GMT+0</small><br><strong>Resolved</strong> - We’re experiencing an outage with one of our transport providers between London and Amsterdam, which temporarily affected connectivity to our London PoP for users routed through Amsterdam. We’ve implemented adjustments to reroute traffic around the issue, and no further impact is expected. We sincerely apologize for the disruption and appreciate your patience and understanding. Impact was between 10:27 AM CT and 10:36 AM CT (9 minutes)..</p> ]]> Thu, 13 Mar 2025 15:27:00 +0000 https://status.as30456.net/incident/cm87j9fjc002g13ry0rvhagsy https://status.as30456.net/incident/cm87j9fjc002g13ry0rvhagsy Multiple PoP Degradation of Service Type: Incident Duration: 37 minutes Affected Components: London, United Kingdom, Ashburn, Virginia, Los Angeles, California, Amsterdam, Netherlands, Dallas, Texas Feb 15, 16:40:00 GMT+0 - Investigating - We are investigating a multi-pop degradation of service. Feb 15, 17:03:17 GMT+0 - Identified - Ashburn and Amsterdam should be recovered. We are still working diligently. Feb 15, 17:08:45 GMT+0 - Monitoring - All degraded PoPs should have recovered/are recovering. Feb 15, 17:17:21 GMT+0 - Resolved - Hi everyone, During an emergency update to our DDoS mitigation systems, an error caused the automated rollback functionality to fail. As a result, the systems entered a continuous boot loop and, due to limited connectivity, were unable to restore the latest version automatically. What would typically be a five-second operation took over 15 minutes as we manually restored the systems. Although we normally do not perform updates on weekends, a critical bug required an immediate fix to prevent potential issues later today. We appreciate your understanding. The situation has now been resolved, and no further impact is expected over the remainder of the weekend. Thank you. Type: Incident

Duration: 37 minutes

Affected Components: , , , ,

<p><small>Feb <var data-var='date'> 15</var>, <var data-var='time'>16:40:00</var> GMT+0</small><br><strong>Investigating</strong> - We are investigating a multi-pop degradation of service..</p> <p><small>Feb <var data-var='date'> 15</var>, <var data-var='time'>17:03:17</var> GMT+0</small><br><strong>Identified</strong> - Ashburn and Amsterdam should be recovered. We are still working diligently..</p> <p><small>Feb <var data-var='date'> 15</var>, <var data-var='time'>17:08:45</var> GMT+0</small><br><strong>Monitoring</strong> - All degraded PoPs should have recovered/are recovering..</p> <p><small>Feb <var data-var='date'> 15</var>, <var data-var='time'>17:17:21</var> GMT+0</small><br><strong>Resolved</strong> - Hi everyone, During an emergency update to our DDoS mitigation systems, an error caused the automated rollback functionality to fail. As a result, the systems entered a continuous boot loop and, due to limited connectivity, were unable to restore the latest version automatically. What would typically be a five-second operation took over 15 minutes as we manually restored the systems. Although we normally do not perform updates on weekends, a critical bug required an immediate fix to prevent potential issues later today. We appreciate your understanding. The situation has now been resolved, and no further impact is expected over the remainder of the weekend. Thank you..</p> ]]> Sat, 15 Feb 2025 16:40:00 +0000 https://status.as30456.net/incident/cm76fpl170016eqct6zqk4le7 https://status.as30456.net/incident/cm76fpl170016eqct6zqk4le7 London Partial Outage Type: Incident Duration: 16 minutes Affected Components: London, United Kingdom Feb 15, 00:50:00 GMT+0 - Monitoring - Our network PoP in London had a brief blip in traffic. We were aware of this blip immediately, and a fix has been applied. We will continue monitoring, and provide an update later. Feb 15, 00:58:00 GMT+0 - Resolved - We experienced an outage in London due to an unexpected ASIC restart on our Arista core router. We're doing a root cause analysis to determine the cause. We believe it is related to new capacity we had provisioned pending setup next week. Everything is back online. Please feel free to reach out to our team if you have any questions. Type: Incident

Duration: 16 minutes

Affected Components:

<p><small>Feb <var data-var='date'> 15</var>, <var data-var='time'>00:50:00</var> GMT+0</small><br><strong>Monitoring</strong> - Our network PoP in London had a brief blip in traffic. We were aware of this blip immediately, and a fix has been applied. We will continue monitoring, and provide an update later..</p> <p><small>Feb <var data-var='date'> 15</var>, <var data-var='time'>00:58:00</var> GMT+0</small><br><strong>Resolved</strong> - We experienced an outage in London due to an unexpected ASIC restart on our Arista core router. We're doing a root cause analysis to determine the cause. We believe it is related to new capacity we had provisioned pending setup next week. Everything is back online. Please feel free to reach out to our team if you have any questions..</p> ]]> Sat, 15 Feb 2025 00:50:00 +0000 https://status.as30456.net/incident/cm75horij002gfgi00njh9muz https://status.as30456.net/incident/cm75horij002gfgi00njh9muz GTT Maintenance Event (Ashburn, VA) Type: Maintenance Duration: 4 hours Affected Components: Ashburn, Virginia Feb 12, 11:00:00 GMT+0 - Completed - Maintenance has completed successfully Feb 12, 07:00:01 GMT+0 - Identified - Maintenance is now in progress Feb 12, 07:00:00 GMT+0 - Identified - GTT has scheduled an emergency maintenance in Ashburn, VA (US East). Service impact is not expected, as traffic will automatically reroute to alternative carriers. However, customers utilizing Anycast proxies may experience a brief disconnect during the traffic transition. Start: 2025-02-12 07:00:00 GMT End: 2025-02-12 11:00:00 GMT Type: Maintenance

Duration: 4 hours

Affected Components:

<p><small>Feb <var data-var='date'> 12</var>, <var data-var='time'>11:00:00</var> GMT+0</small><br><strong>Completed</strong> - Maintenance has completed successfully.</p> <p><small>Feb <var data-var='date'> 12</var>, <var data-var='time'>07:00:01</var> GMT+0</small><br><strong>Identified</strong> - Maintenance is now in progress.</p> <p><small>Feb <var data-var='date'> 12</var>, <var data-var='time'>07:00:00</var> GMT+0</small><br><strong>Identified</strong> - GTT has scheduled an emergency maintenance in Ashburn, VA (US East). Service impact is not expected, as traffic will automatically reroute to alternative carriers. However, customers utilizing Anycast proxies may experience a brief disconnect during the traffic transition. Start: 2025-02-12 07:00:00 GMT End: 2025-02-12 11:00:00 GMT.</p> ]]> Wed, 12 Feb 2025 07:00:00 +0000 https://status.as30456.net/maintenance/cm71jsoeb0001x0d9uzdqpzyr https://status.as30456.net/maintenance/cm71jsoeb0001x0d9uzdqpzyr LAX Outage Type: Incident Duration: 40 minutes Affected Components: Los Angeles, California Nov 30, 14:14:41 GMT+0 - Resolved - This incident has been resolved. Nov 30, 13:34:39 GMT+0 - Investigating - We are currently investigating this incident. Type: Incident

Duration: 40 minutes

Affected Components:

<p><small>Nov <var data-var='date'> 30</var>, <var data-var='time'>14:14:41</var> GMT+0</small><br><strong>Resolved</strong> - This incident has been resolved..</p> <p><small>Nov <var data-var='date'> 30</var>, <var data-var='time'>13:34:39</var> GMT+0</small><br><strong>Investigating</strong> - We are currently investigating this incident..</p> ]]> Sat, 30 Nov 2024 13:34:39 +0000 https://status.as30456.net/incident/cm447s9r7000imv9k9xw3vhke https://status.as30456.net/incident/cm447s9r7000imv9k9xw3vhke Network Blip in Dallas Type: Incident Duration: 54 minutes Affected Components: Dallas, Texas Oct 25, 18:15:00 GMT+0 - Identified - We are currently investigating this incident. We are aware of what caused the issue, and are investigating why. We will update this status page when we have more information. Oct 25, 19:08:43 GMT+0 - Resolved - This incident has been resolved. No further issues or updates are expected. Thank you. Type: Incident

Duration: 54 minutes

Affected Components:

<p><small>Oct <var data-var='date'> 25</var>, <var data-var='time'>18:15:00</var> GMT+0</small><br><strong>Identified</strong> - We are currently investigating this incident. We are aware of what caused the issue, and are investigating why. We will update this status page when we have more information..</p> <p><small>Oct <var data-var='date'> 25</var>, <var data-var='time'>19:08:43</var> GMT+0</small><br><strong>Resolved</strong> - This incident has been resolved. No further issues or updates are expected. Thank you..</p> ]]> Fri, 25 Oct 2024 18:15:00 +0000 https://status.as30456.net/incident/cm2p2qiwq0076miqcgbpcscuc https://status.as30456.net/incident/cm2p2qiwq0076miqcgbpcscuc ASH Scheduled Maintenance Type: Maintenance Duration: 6 minutes Affected Components: Ashburn, Virginia Aug 21, 05:00:01 GMT+0 - Identified - Maintenance is now in progress Aug 15, 03:31:22 GMT+0 - Identified - Revised scheduled time. Aug 21, 05:00:00 GMT+0 - Identified - We are planning for a maintenance at 1AM ET in our ASH region. We’re applying software updates to our routers. Although we have completed DFW last week and previously AMS without interruption, we still advise caution and to expect brief interruption while BGP sessions renegotiate. Aug 21, 05:06:08 GMT+0 - Completed - Maintenance is complete. We observed no impact to our network. Type: Maintenance

Duration: 6 minutes

Affected Components:

<p><small>Aug <var data-var='date'> 21</var>, <var data-var='time'>05:00:01</var> GMT+0</small><br><strong>Identified</strong> - Maintenance is now in progress.</p> <p><small>Aug <var data-var='date'> 15</var>, <var data-var='time'>03:31:22</var> GMT+0</small><br><strong>Identified</strong> - Revised scheduled time..</p> <p><small>Aug <var data-var='date'> 21</var>, <var data-var='time'>05:00:00</var> GMT+0</small><br><strong>Identified</strong> - We are planning for a maintenance at 1AM ET in our ASH region. We’re applying software updates to our routers. Although we have completed DFW last week and previously AMS without interruption, we still advise caution and to expect brief interruption while BGP sessions renegotiate..</p> <p><small>Aug <var data-var='date'> 21</var>, <var data-var='time'>05:06:08</var> GMT+0</small><br><strong>Completed</strong> - Maintenance is complete. We observed no impact to our network..</p> ]]> Wed, 21 Aug 2024 05:00:00 +0000 https://status.as30456.net/maintenance/clzuo8juv30889hhn5pohzane5 https://status.as30456.net/maintenance/clzuo8juv30889hhn5pohzane5 LAX Scheduled Maintenance Type: Maintenance Duration: 18 minutes Affected Components: Los Angeles, California Aug 19, 08:00:01 GMT+0 - Identified - Maintenance is now in progress Aug 19, 08:17:37 GMT+0 - Completed - Maintenance is complete. We observed minimal impact to traffic routing via LAX. Aug 15, 03:31:56 GMT+0 - Identified - Revised scheduled time. Aug 19, 08:00:00 GMT+0 - Identified - We are planning for a maintenance at 1AM PT in our LAX region. We’re applying software updates to our routers. Although we have completed DFW last week and previously AMS without interruption, we still advise caution and to expect brief interruption while BGP sessions renegotiate. Type: Maintenance

Duration: 18 minutes

Affected Components:

<p><small>Aug <var data-var='date'> 19</var>, <var data-var='time'>08:00:01</var> GMT+0</small><br><strong>Identified</strong> - Maintenance is now in progress.</p> <p><small>Aug <var data-var='date'> 19</var>, <var data-var='time'>08:17:37</var> GMT+0</small><br><strong>Completed</strong> - Maintenance is complete. We observed minimal impact to traffic routing via LAX..</p> <p><small>Aug <var data-var='date'> 15</var>, <var data-var='time'>03:31:56</var> GMT+0</small><br><strong>Identified</strong> - Revised scheduled time..</p> <p><small>Aug <var data-var='date'> 19</var>, <var data-var='time'>08:00:00</var> GMT+0</small><br><strong>Identified</strong> - We are planning for a maintenance at 1AM PT in our LAX region. We’re applying software updates to our routers. Although we have completed DFW last week and previously AMS without interruption, we still advise caution and to expect brief interruption while BGP sessions renegotiate..</p> ]]> Mon, 19 Aug 2024 08:00:00 +0000 https://status.as30456.net/maintenance/clzuo59tg26663han5lio9ylb0 https://status.as30456.net/maintenance/clzuo59tg26663han5lio9ylb0 LON Scheduled Maintenance Type: Maintenance Duration: 16 minutes Affected Components: London, United Kingdom Aug 16, 01:00:00 GMT+0 - Identified - We are planning for a maintenance at 2AM BST in our London region. We’re applying software updates to our routers. Although we have completed DFW last week and previously AMS without interruption, we still advise caution and to expect brief interruption while BGP sessions renegotiate. Aug 15, 03:32:17 GMT+0 - Identified - Revised scheduled time. Aug 16, 01:00:01 GMT+0 - Identified - Maintenance is now in progress Aug 16, 01:16:06 GMT+0 - Completed - Maintenance is complete. We observed no impact to our network. Type: Maintenance

Duration: 16 minutes

Affected Components:

<p><small>Aug <var data-var='date'> 16</var>, <var data-var='time'>01:00:00</var> GMT+0</small><br><strong>Identified</strong> - We are planning for a maintenance at 2AM BST in our London region. We’re applying software updates to our routers. Although we have completed DFW last week and previously AMS without interruption, we still advise caution and to expect brief interruption while BGP sessions renegotiate..</p> <p><small>Aug <var data-var='date'> 15</var>, <var data-var='time'>03:32:17</var> GMT+0</small><br><strong>Identified</strong> - Revised scheduled time..</p> <p><small>Aug <var data-var='date'> 16</var>, <var data-var='time'>01:00:01</var> GMT+0</small><br><strong>Identified</strong> - Maintenance is now in progress.</p> <p><small>Aug <var data-var='date'> 16</var>, <var data-var='time'>01:16:06</var> GMT+0</small><br><strong>Completed</strong> - Maintenance is complete. We observed no impact to our network..</p> ]]> Fri, 16 Aug 2024 01:00:00 +0000 https://status.as30456.net/maintenance/clzuo1t1h28668hhn5nn32f34l https://status.as30456.net/maintenance/clzuo1t1h28668hhn5nn32f34l DFW Scheduled Maintenance Type: Maintenance Duration: 41 minutes Affected Components: Dallas, Texas Aug 13, 06:00:00 GMT+0 - Identified - We are planning for a maintenance event early tomorrow morning at 1AM Central Time in our Dallas Fort-Worth region. We expect some turbulence for around 30 minutes to 1 hour while BGP sessions renegotiate. Aug 13, 06:40:40 GMT+0 - Completed - Maintenance is complete. We observed no impact to our network. Aug 13, 06:00:01 GMT+0 - Identified - Maintenance is now in progress Type: Maintenance

Duration: 41 minutes

Affected Components:

<p><small>Aug <var data-var='date'> 13</var>, <var data-var='time'>06:00:00</var> GMT+0</small><br><strong>Identified</strong> - We are planning for a maintenance event early tomorrow morning at 1AM Central Time in our Dallas Fort-Worth region. We expect some turbulence for around 30 minutes to 1 hour while BGP sessions renegotiate. .</p> <p><small>Aug <var data-var='date'> 13</var>, <var data-var='time'>06:40:40</var> GMT+0</small><br><strong>Completed</strong> - Maintenance is complete. We observed no impact to our network..</p> <p><small>Aug <var data-var='date'> 13</var>, <var data-var='time'>06:00:01</var> GMT+0</small><br><strong>Identified</strong> - Maintenance is now in progress.</p> ]]> Tue, 13 Aug 2024 06:00:00 +0000 https://status.as30456.net/maintenance/clzrclike30721hnodejvro46x https://status.as30456.net/maintenance/clzrclike30721hnodejvro46x GTT Outage - London Type: Incident Duration: 9 hours and 53 minutes Affected Components: London, United Kingdom Jun 27, 20:26:12 GMT+0 - Investigating - We are seeing flaps on our transit BGP session with GTT in London, UK. We’re monitoring and will temporarily drop the session if interruptions continue to occur. Jun 27, 20:29:02 GMT+0 - Monitoring - We have temporarily dropped the BGP session for GTT in London, UK. Traffic routed via GTT will re-route to Amsterdam, NL. We are still peering via BGP with NTT and Liberty Global in London, UK. Jun 28, 06:19:12 GMT+0 - Resolved - We have resumed BGP advertisements over GTT in London, UK. Type: Incident

Duration: 9 hours and 53 minutes

Affected Components:

<p><small>Jun <var data-var='date'> 27</var>, <var data-var='time'>20:26:12</var> GMT+0</small><br><strong>Investigating</strong> - We are seeing flaps on our transit BGP session with GTT in London, UK. We’re monitoring and will temporarily drop the session if interruptions continue to occur..</p> <p><small>Jun <var data-var='date'> 27</var>, <var data-var='time'>20:29:02</var> GMT+0</small><br><strong>Monitoring</strong> - We have temporarily dropped the BGP session for GTT in London, UK. Traffic routed via GTT will re-route to Amsterdam, NL. We are still peering via BGP with NTT and Liberty Global in London, UK..</p> <p><small>Jun <var data-var='date'> 28</var>, <var data-var='time'>06:19:12</var> GMT+0</small><br><strong>Resolved</strong> - We have resumed BGP advertisements over GTT in London, UK..</p> ]]> Thu, 27 Jun 2024 20:26:12 +0000 https://status.as30456.net/incident/clxxpsmg782774xdn1fe348rao https://status.as30456.net/incident/clxxpsmg782774xdn1fe348rao NTT Scheduled Maintenance Event (Dallas, TX) Type: Maintenance Duration: 3 hours Affected Components: London, United Kingdom May 27, 00:00:00 GMT+0 - Identified - NTT is planning a scheduled maintenance for this time. There is the possibility of downtime during this event. May 27, 03:00:00 GMT+0 - Completed - Maintenance has completed successfully May 27, 00:00:01 GMT+0 - Identified - Maintenance is now in progress Type: Maintenance

Duration: 3 hours

Affected Components:

<p><small>May <var data-var='date'> 27</var>, <var data-var='time'>00:00:00</var> GMT+0</small><br><strong>Identified</strong> - NTT is planning a scheduled maintenance for this time. There is the possibility of downtime during this event..</p> <p><small>May <var data-var='date'> 27</var>, <var data-var='time'>03:00:00</var> GMT+0</small><br><strong>Completed</strong> - Maintenance has completed successfully.</p> <p><small>May <var data-var='date'> 27</var>, <var data-var='time'>00:00:01</var> GMT+0</small><br><strong>Identified</strong> - Maintenance is now in progress.</p> ]]> Mon, 27 May 2024 00:00:00 +0000 https://status.as30456.net/maintenance/clw6qad393556b2n6zcro2r5n https://status.as30456.net/maintenance/clw6qad393556b2n6zcro2r5n Power outage at DFW Evocative Type: Incident Duration: 17 hours and 23 minutes Affected Components: API, Dallas, Texas May 26, 12:52:59 GMT+0 - Investigating - DFW Evocative’s datacenter staff are investigating an issue with their UPS equipment. ETA provided by the datacenter has been missed and we have not yet been provided an additional ETA. We will update as soon as possible. May 26, 11:51:54 GMT+0 - Investigating - There is a power outage in DFW Evocative related to a storm. Our DFW Equinix location is still online. We have been provided a 30-60 minute ETA by the facility. May 26, 12:28:26 GMT+0 - Investigating - DFW Evocative is investigating an issue with their UPS equipment. They are communicating with the vendor. May 26, 14:03:32 GMT+0 - Monitoring - Services are being restored and we are currently monitoring the result. May 26, 14:20:11 GMT+0 - Monitoring - As of 15 minutes ago, DFW Evocative’s power has recovered and services have started to come online. We will report when all clear. May 26, 15:34:55 GMT+0 - Monitoring - We have restored 95% of customer services in DFW Evocative. We will update when all clear. May 26, 20:09:03 GMT+0 - Monitoring - We are still working on restoring the remaining 5% of customers affected in our Dallas data center. It is isolated to a few racks at this point, and seems to have been caused by the power outage itself. We will continue updating as soon as we can. May 26, 23:06:27 GMT+0 - Monitoring - We are nearing the end of the issues caused by the power outage today. We do apologize that this has taken longer than expected, and are hoping to restore the rest of our affected clients in the next few hours. We will continue to update here when possible. Thank you for your patience. May 27, 05:14:47 GMT+0 - Resolved - We have restored 99% of services under our control, and thus we are closing this issue. Any remaining issues are still being looked into as isolated issues out of our control. We do not expect any further issues. The datacenter is in the process of performing a root cause analysis to figure out what went wrong during today's power outage, and they have already had the UPS vendors come out and check everything over and get it all operational again. Thank you for your patience today. Type: Incident

Duration: 17 hours and 23 minutes

Affected Components: ,

<p><small>May <var data-var='date'> 26</var>, <var data-var='time'>12:52:59</var> GMT+0</small><br><strong>Investigating</strong> - DFW Evocative’s datacenter staff are investigating an issue with their UPS equipment. ETA provided by the datacenter has been missed and we have not yet been provided an additional ETA. We will update as soon as possible..</p> <p><small>May <var data-var='date'> 26</var>, <var data-var='time'>11:51:54</var> GMT+0</small><br><strong>Investigating</strong> - There is a power outage in DFW Evocative related to a storm. Our DFW Equinix location is still online. We have been provided a 30-60 minute ETA by the facility..</p> <p><small>May <var data-var='date'> 26</var>, <var data-var='time'>12:28:26</var> GMT+0</small><br><strong>Investigating</strong> - DFW Evocative is investigating an issue with their UPS equipment. They are communicating with the vendor..</p> <p><small>May <var data-var='date'> 26</var>, <var data-var='time'>14:03:32</var> GMT+0</small><br><strong>Monitoring</strong> - Services are being restored and we are currently monitoring the result..</p> <p><small>May <var data-var='date'> 26</var>, <var data-var='time'>14:20:11</var> GMT+0</small><br><strong>Monitoring</strong> - As of 15 minutes ago, DFW Evocative’s power has recovered and services have started to come online. We will report when all clear..</p> <p><small>May <var data-var='date'> 26</var>, <var data-var='time'>15:34:55</var> GMT+0</small><br><strong>Monitoring</strong> - We have restored 95% of customer services in DFW Evocative. We will update when all clear..</p> <p><small>May <var data-var='date'> 26</var>, <var data-var='time'>20:09:03</var> GMT+0</small><br><strong>Monitoring</strong> - We are still working on restoring the remaining 5% of customers affected in our Dallas data center. It is isolated to a few racks at this point, and seems to have been caused by the power outage itself. We will continue updating as soon as we can..</p> <p><small>May <var data-var='date'> 26</var>, <var data-var='time'>23:06:27</var> GMT+0</small><br><strong>Monitoring</strong> - We are nearing the end of the issues caused by the power outage today. We do apologize that this has taken longer than expected, and are hoping to restore the rest of our affected clients in the next few hours. We will continue to update here when possible. Thank you for your patience..</p> <p><small>May <var data-var='date'> 27</var>, <var data-var='time'>05:14:47</var> GMT+0</small><br><strong>Resolved</strong> - We have restored 99% of services under our control, and thus we are closing this issue. Any remaining issues are still being looked into as isolated issues out of our control. We do not expect any further issues. The datacenter is in the process of performing a root cause analysis to figure out what went wrong during today's power outage, and they have already had the UPS vendors come out and check everything over and get it all operational again. Thank you for your patience today..</p> ]]> Sun, 26 May 2024 11:51:54 +0000 https://status.as30456.net/incident/clwnhbyuj188982axoaqzvrcjis https://status.as30456.net/incident/clwnhbyuj188982axoaqzvrcjis NTT Scheduled Maintenance Event (London, UK) Type: Maintenance Duration: 3 hours Affected Components: Dallas, Texas May 23, 06:00:00 GMT+0 - Identified - NTT is planning a scheduled maintenance for this time. There is the possibility of downtime during this event. May 23, 06:00:01 GMT+0 - Identified - Maintenance is now in progress May 23, 09:00:00 GMT+0 - Completed - Maintenance has completed successfully Type: Maintenance

Duration: 3 hours

Affected Components:

<p><small>May <var data-var='date'> 23</var>, <var data-var='time'>06:00:00</var> GMT+0</small><br><strong>Identified</strong> - NTT is planning a scheduled maintenance for this time. There is the possibility of downtime during this event..</p> <p><small>May <var data-var='date'> 23</var>, <var data-var='time'>06:00:01</var> GMT+0</small><br><strong>Identified</strong> - Maintenance is now in progress.</p> <p><small>May <var data-var='date'> 23</var>, <var data-var='time'>09:00:00</var> GMT+0</small><br><strong>Completed</strong> - Maintenance has completed successfully.</p> ]]> Thu, 23 May 2024 06:00:00 +0000 https://status.as30456.net/maintenance/clw6qi6vb7950b2n6gpv9ondo https://status.as30456.net/maintenance/clw6qi6vb7950b2n6gpv9ondo Scheduled Maintenance (Amsterdam, NL) Type: Maintenance Duration: 1 hour Affected Components: Amsterdam, Netherlands May 16, 03:00:01 GMT+0 - Identified - Maintenance is now in progress May 16, 04:00:00 GMT+0 - Completed - Maintenance has completed successfully May 16, 03:00:00 GMT+0 - Identified - We will be performing software upgrades on our routing infrastructure in Amsterdam which will require a hard reboot. Expected downtime is 15 minutes. This may be felt in neighboring PoPs as we re-route traffic between PoPs. Type: Maintenance

Duration: 1 hour

Affected Components:

<p><small>May <var data-var='date'> 16</var>, <var data-var='time'>03:00:01</var> GMT+0</small><br><strong>Identified</strong> - Maintenance is now in progress.</p> <p><small>May <var data-var='date'> 16</var>, <var data-var='time'>04:00:00</var> GMT+0</small><br><strong>Completed</strong> - Maintenance has completed successfully.</p> <p><small>May <var data-var='date'> 16</var>, <var data-var='time'>03:00:00</var> GMT+0</small><br><strong>Identified</strong> - We will be performing software upgrades on our routing infrastructure in Amsterdam which will require a hard reboot. Expected downtime is 15 minutes. This may be felt in neighboring PoPs as we re-route traffic between PoPs..</p> ]]> Thu, 16 May 2024 03:00:00 +0000 https://status.as30456.net/maintenance/clw79jpa2130547bdogtzycpbyj https://status.as30456.net/maintenance/clw79jpa2130547bdogtzycpbyj Scheduled maintenance Type: Maintenance Duration: 57 minutes Affected Components: London, United Kingdom, Amsterdam, Netherlands, Ashburn, Virginia, Los Angeles, California, Dallas, Texas Apr 26, 05:56:40 GMT+0 - Completed - All set. Apr 26, 05:00:01 GMT+0 - Identified - Maintenance is now in progress Apr 26, 05:00:00 GMT+0 - Identified - We'll be performing upgrades to our DDoS mitigation platform. We're not expecting this to impact user traffic. But we do recommend making your players aware of this window. Type: Maintenance

Duration: 57 minutes

Affected Components: , , , ,

<p><small>Apr <var data-var='date'> 26</var>, <var data-var='time'>05:56:40</var> GMT+0</small><br><strong>Completed</strong> - All set..</p> <p><small>Apr <var data-var='date'> 26</var>, <var data-var='time'>05:00:01</var> GMT+0</small><br><strong>Identified</strong> - Maintenance is now in progress.</p> <p><small>Apr <var data-var='date'> 26</var>, <var data-var='time'>05:00:00</var> GMT+0</small><br><strong>Identified</strong> - We'll be performing upgrades to our DDoS mitigation platform. We're not expecting this to impact user traffic. But we do recommend making your players aware of this window..</p> ]]> Fri, 26 Apr 2024 05:00:00 +0000 https://status.as30456.net/maintenance/clvg1hvoy49810bsn6p420fwnc https://status.as30456.net/maintenance/clvg1hvoy49810bsn6p420fwnc GTT Planned Work - Reston, VA Type: Maintenance Duration: 4 hours Affected Components: Ashburn, Virginia Apr 4, 23:00:00 GMT+0 - Identified - GTT is planning a scheduled maintenance during the below timeframe. Service interruption is possible. Maximum service interruption expected is 15 minutes. **Start: 2024-04-05 00:00:00 GMT** **End: 2024-04-05 04:00:00 GMT** **Backup Start: 2024-04-09 00:00:00 GMT** **Backup End: 2024-04-09 04:00:00 GMT** Apr 5, 03:00:00 GMT+0 - Completed - Maintenance has completed successfully Apr 4, 23:00:01 GMT+0 - Identified - Maintenance is now in progress Type: Maintenance

Duration: 4 hours

Affected Components:

<p><small>Apr <var data-var='date'> 4</var>, <var data-var='time'>23:00:00</var> GMT+0</small><br><strong>Identified</strong> - GTT is planning a scheduled maintenance during the below timeframe. Service interruption is possible. Maximum service interruption expected is 15 minutes. **Start: 2024-04-05 00:00:00 GMT** **End: 2024-04-05 04:00:00 GMT** **Backup Start: 2024-04-09 00:00:00 GMT** **Backup End: 2024-04-09 04:00:00 GMT**.</p> <p><small>Apr <var data-var='date'> 5</var>, <var data-var='time'>03:00:00</var> GMT+0</small><br><strong>Completed</strong> - Maintenance has completed successfully.</p> <p><small>Apr <var data-var='date'> 4</var>, <var data-var='time'>23:00:01</var> GMT+0</small><br><strong>Identified</strong> - Maintenance is now in progress.</p> ]]> Thu, 4 Apr 2024 23:00:00 +0000 https://status.as30456.net/maintenance/clu8v53e0113985b8myz3ljlkxl https://status.as30456.net/maintenance/clu8v53e0113985b8myz3ljlkxl Mass Disconnections in EU and US Type: Incident Duration: 1 day, 6 hours and 7 minutes Affected Components: London, United Kingdom, Ashburn, Virginia, Dallas, Texas Mar 12, 05:02:57 GMT+0 - Investigating - We are currently investigating this incident. We are aware of some disconnects affecting a large number of users in the last 30 minutes. We will update ASAP. Mar 13, 00:04:42 GMT+0 - Resolved - This incident is confirmed to be resolved. A post-mortem will follow shortly. We appreciate your patience during this time.This incident has been resolved. Mar 13, 02:44:27 GMT+0 - Investigating - We're currently investigating a similar disconnection event as last night. We're already working on a fix, and will update everyone soon. Thanks! Mar 13, 05:04:01 GMT+0 - Identified - The issue has been narrowed down, and we are diligently working on a permanent and full fix. We hope to have a more positive update to share soon. Mar 13, 08:26:35 GMT+0 - Monitoring - We have several engineers working on pushing out a full fix ASAP. Updates will follow as we have them for you. Mar 13, 11:10:16 GMT+0 - Resolved - We have pushed fixes to all locations and are monitoring proactively to ensure all services fully stabilize and remain stable. Type: Incident

Duration: 1 day, 6 hours and 7 minutes

Affected Components: , ,

<p><small>Mar <var data-var='date'> 12</var>, <var data-var='time'>05:02:57</var> GMT+0</small><br><strong>Investigating</strong> - We are currently investigating this incident. We are aware of some disconnects affecting a large number of users in the last 30 minutes. We will update ASAP..</p> <p><small>Mar <var data-var='date'> 13</var>, <var data-var='time'>00:04:42</var> GMT+0</small><br><strong>Resolved</strong> - This incident is confirmed to be resolved. A post-mortem will follow shortly. We appreciate your patience during this time.This incident has been resolved..</p> <p><small>Mar <var data-var='date'> 13</var>, <var data-var='time'>02:44:27</var> GMT+0</small><br><strong>Investigating</strong> - We're currently investigating a similar disconnection event as last night. We're already working on a fix, and will update everyone soon. Thanks!.</p> <p><small>Mar <var data-var='date'> 13</var>, <var data-var='time'>05:04:01</var> GMT+0</small><br><strong>Identified</strong> - The issue has been narrowed down, and we are diligently working on a permanent and full fix. We hope to have a more positive update to share soon. .</p> <p><small>Mar <var data-var='date'> 13</var>, <var data-var='time'>08:26:35</var> GMT+0</small><br><strong>Monitoring</strong> - We have several engineers working on pushing out a full fix ASAP. Updates will follow as we have them for you..</p> <p><small>Mar <var data-var='date'> 13</var>, <var data-var='time'>11:10:16</var> GMT+0</small><br><strong>Resolved</strong> - We have pushed fixes to all locations and are monitoring proactively to ensure all services fully stabilize and remain stable. .</p> ]]> Tue, 12 Mar 2024 05:02:57 +0000 https://status.as30456.net/incident/cltnwp5vx36991buoajbrl5r76 https://status.as30456.net/incident/cltnwp5vx36991buoajbrl5r76 Packet Loss / Disconnects Type: Incident Duration: 8 hours and 9 minutes Affected Components: London, United Kingdom, Ashburn, Virginia, Dallas, Texas Mar 4, 02:43:15 GMT+0 - Resolved - This incident has been resolved. Mar 3, 18:33:51 GMT+0 - Investigating - Good Morning/Afternoon/Evening, We're aware of the packet loss events, and disconnects that are currently happening, and we've been looking into them. We're in the process of implementing some things, and we will have things calmed down ASAP. Thanks for your patience. Type: Incident

Duration: 8 hours and 9 minutes

Affected Components: , ,

<p><small>Mar <var data-var='date'> 4</var>, <var data-var='time'>02:43:15</var> GMT+0</small><br><strong>Resolved</strong> - This incident has been resolved..</p> <p><small>Mar <var data-var='date'> 3</var>, <var data-var='time'>18:33:51</var> GMT+0</small><br><strong>Investigating</strong> - Good Morning/Afternoon/Evening, We're aware of the packet loss events, and disconnects that are currently happening, and we've been looking into them. We're in the process of implementing some things, and we will have things calmed down ASAP. Thanks for your patience..</p> ]]> Sun, 3 Mar 2024 18:33:51 +0000 https://status.as30456.net/incident/cltbupbdf28412biogddn28jlw https://status.as30456.net/incident/cltbupbdf28412biogddn28jlw Outage in Ashburn Evoque Type: Incident Duration: 3 hours and 44 minutes Affected Components: Ashburn, Virginia Feb 19, 17:44:59 GMT+0 - Investigating - We're currently investigating an outage in Evoque. We should have an update shortly. Feb 19, 18:08:51 GMT+0 - Monitoring - The services in Ashburn were restored shortly after posting this incident. We are monitoring currently. Feb 19, 21:28:36 GMT+0 - Resolved - This issue has been resolved as we have not experienced any issues since the 10 minute downtime. No further issues expected. Type: Incident

Duration: 3 hours and 44 minutes

Affected Components:

<p><small>Feb <var data-var='date'> 19</var>, <var data-var='time'>17:44:59</var> GMT+0</small><br><strong>Investigating</strong> - We're currently investigating an outage in Evoque. We should have an update shortly..</p> <p><small>Feb <var data-var='date'> 19</var>, <var data-var='time'>18:08:51</var> GMT+0</small><br><strong>Monitoring</strong> - The services in Ashburn were restored shortly after posting this incident. We are monitoring currently..</p> <p><small>Feb <var data-var='date'> 19</var>, <var data-var='time'>21:28:36</var> GMT+0</small><br><strong>Resolved</strong> - This issue has been resolved as we have not experienced any issues since the 10 minute downtime. No further issues expected..</p> ]]> Mon, 19 Feb 2024 17:44:59 +0000 https://status.as30456.net/incident/clst88eij85293ccoo6pkyzh1y https://status.as30456.net/incident/clst88eij85293ccoo6pkyzh1y Moving AMS -> LON traffic to private transport capacity Type: Maintenance Duration: 7 minutes Affected Components: London, United Kingdom, Amsterdam, Netherlands Feb 17, 06:00:01 GMT+0 - Identified - Maintenance is now in progress Feb 17, 06:00:00 GMT+0 - Identified - Hi everyone, We'll be moving our traffic between london and amsterdam (in both directions) over to private transport capacity rather than the tunneled system we've historically relied on. This should help keep things smooth during the busiest times of the day. No impact is expected while we route traffic over. Feb 17, 06:06:39 GMT+0 - Completed - Complete. Type: Maintenance

Duration: 7 minutes

Affected Components: ,

<p><small>Feb <var data-var='date'> 17</var>, <var data-var='time'>06:00:01</var> GMT+0</small><br><strong>Identified</strong> - Maintenance is now in progress.</p> <p><small>Feb <var data-var='date'> 17</var>, <var data-var='time'>06:00:00</var> GMT+0</small><br><strong>Identified</strong> - Hi everyone, We'll be moving our traffic between london and amsterdam (in both directions) over to private transport capacity rather than the tunneled system we've historically relied on. This should help keep things smooth during the busiest times of the day. No impact is expected while we route traffic over..</p> <p><small>Feb <var data-var='date'> 17</var>, <var data-var='time'>06:06:39</var> GMT+0</small><br><strong>Completed</strong> - Complete..</p> ]]> Sat, 17 Feb 2024 06:00:00 +0000 https://status.as30456.net/maintenance/clspmzr6g57118balh3fc7ez8d https://status.as30456.net/maintenance/clspmzr6g57118balh3fc7ez8d Outage in VA/DFW Type: Incident Duration: 17 hours and 38 minutes Affected Components: Ashburn, Virginia, Dallas, Texas Feb 7, 04:03:59 GMT+0 - Investigating - We are currently investigating this incident. Feb 7, 04:40:49 GMT+0 - Identified - No updates to share. Still working/investigating the issue. Feb 7, 05:20:48 GMT+0 - Identified - We have located the issue, and implemented a temporary fix for the issue. As of now, traffic previously routing through Ashburn has been diverted to the next closest pop for your user. We are working on a full scale solution, and will have an update ASAP. For now, things should be stable with slightly increased latency being possible. We will also be sharing a full scale RFO in the next few days. Feb 7, 21:41:41 GMT+0 - Resolved - This incident has been resolved. Type: Incident

Duration: 17 hours and 38 minutes

Affected Components: ,

<p><small>Feb <var data-var='date'> 7</var>, <var data-var='time'>04:03:59</var> GMT+0</small><br><strong>Investigating</strong> - We are currently investigating this incident..</p> <p><small>Feb <var data-var='date'> 7</var>, <var data-var='time'>04:40:49</var> GMT+0</small><br><strong>Identified</strong> - No updates to share. Still working/investigating the issue..</p> <p><small>Feb <var data-var='date'> 7</var>, <var data-var='time'>05:20:48</var> GMT+0</small><br><strong>Identified</strong> - We have located the issue, and implemented a temporary fix for the issue. As of now, traffic previously routing through Ashburn has been diverted to the next closest pop for your user. We are working on a full scale solution, and will have an update ASAP. For now, things should be stable with slightly increased latency being possible. We will also be sharing a full scale RFO in the next few days..</p> <p><small>Feb <var data-var='date'> 7</var>, <var data-var='time'>21:41:41</var> GMT+0</small><br><strong>Resolved</strong> - This incident has been resolved..</p> ]]> Wed, 7 Feb 2024 04:03:59 +0000 https://status.as30456.net/incident/clsb9md3m16193agn67p64xhem https://status.as30456.net/incident/clsb9md3m16193agn67p64xhem Portal + API Degradation Type: Incident Duration: 19 days, 2 hours and 6 minutes Affected Components: Customer Portal Feb 14, 04:56:05 GMT+0 - Resolved - This incident has been resolved. Jan 26, 02:50:00 GMT+0 - Investigating - Portal + API are currently degraded due to maintenance on backend cluster hardware and changes. Type: Incident

Duration: 19 days, 2 hours and 6 minutes

Affected Components:

<p><small>Feb <var data-var='date'> 14</var>, <var data-var='time'>04:56:05</var> GMT+0</small><br><strong>Resolved</strong> - This incident has been resolved..</p> <p><small>Jan <var data-var='date'> 26</var>, <var data-var='time'>02:50:00</var> GMT+0</small><br><strong>Investigating</strong> - Portal + API are currently degraded due to maintenance on backend cluster hardware and changes. .</p> ]]> Fri, 26 Jan 2024 02:50:00 +0000 https://status.as30456.net/incident/clrim6xoa87312b8ogxgxppglr https://status.as30456.net/incident/clrim6xoa87312b8ogxgxppglr GTT Major Outage in London Type: Incident Duration: 2 days, 19 hours and 50 minutes Affected Components: London, United Kingdom Jan 25, 17:30:00 GMT+0 - Resolved - GTT is reporting a Major Outage in London due to a faulty device. Reported start time is: 2024-01-25 14:00:00 GMT We will provide updates as they are received. Type: Incident

Duration: 2 days, 19 hours and 50 minutes

Affected Components:

<p><small>Jan <var data-var='date'> 25</var>, <var data-var='time'>17:30:00</var> GMT+0</small><br><strong>Resolved</strong> - GTT is reporting a Major Outage in London due to a faulty device. Reported start time is: 2024-01-25 14:00:00 GMT We will provide updates as they are received..</p> ]]> Thu, 25 Jan 2024 17:30:00 +0000 https://status.as30456.net/incident/clrtf9lgu94492binc615ssgku https://status.as30456.net/incident/clrtf9lgu94492binc615ssgku Small Disconnect in DFW Region Type: Incident Affected Components: Dallas, Texas Jan 7, 04:26:48 GMT+0 - Resolved - Dear customers, In an attempt to resolve the packet loss issues we've been seeing for the last few days in Dallas, we made some routing changes to address the issues based on the data we have, but unfortunately it caused some heavy BGP turbulence causing a small Disconnect on certain connections in Dallas. We do not expect any further disconnects, and we should hopefully have a fix on our end for most of the packet loss issues you've been seeing. Thanks for your patience. Type: Incident

Affected Components:

<p><small>Jan <var data-var='date'> 7</var>, <var data-var='time'>04:26:48</var> GMT+0</small><br><strong>Resolved</strong> - Dear customers, In an attempt to resolve the packet loss issues we've been seeing for the last few days in Dallas, we made some routing changes to address the issues based on the data we have, but unfortunately it caused some heavy BGP turbulence causing a small Disconnect on certain connections in Dallas. We do not expect any further disconnects, and we should hopefully have a fix on our end for most of the packet loss issues you've been seeing. Thanks for your patience..</p> ]]> Sun, 7 Jan 2024 04:26:48 +0000 https://status.as30456.net/incident/clr2zsas944100b6na8a6rx21x https://status.as30456.net/incident/clr2zsas944100b6na8a6rx21x Configuration Change in London Type: Maintenance Duration: 30 minutes Affected Components: London, United Kingdom Nov 14, 01:00:01 GMT+0 - Identified - Maintenance is now in progress Nov 14, 01:00:00 GMT+0 - Identified - We are planning for a scheduled maintenance at 1AM GMT on Tuesday, November 14th. This maintenance will not last more than 30 minutes. It could cause a flap and disconnection of traffic. Nov 14, 01:30:00 GMT+0 - Completed - Maintenance has completed successfully Type: Maintenance

Duration: 30 minutes

Affected Components:

<p><small>Nov <var data-var='date'> 14</var>, <var data-var='time'>01:00:01</var> GMT+0</small><br><strong>Identified</strong> - Maintenance is now in progress.</p> <p><small>Nov <var data-var='date'> 14</var>, <var data-var='time'>01:00:00</var> GMT+0</small><br><strong>Identified</strong> - We are planning for a scheduled maintenance at 1AM GMT on Tuesday, November 14th. This maintenance will not last more than 30 minutes. It could cause a flap and disconnection of traffic..</p> <p><small>Nov <var data-var='date'> 14</var>, <var data-var='time'>01:30:00</var> GMT+0</small><br><strong>Completed</strong> - Maintenance has completed successfully.</p> ]]> Tue, 14 Nov 2023 01:00:00 +0000 https://status.as30456.net/maintenance/clow1t9c414812baoe42y6k93k https://status.as30456.net/maintenance/clow1t9c414812baoe42y6k93k Activating Peering Capacity to Liberty Global Type: Maintenance Duration: 1 hour Affected Components: London, United Kingdom Jul 11, 04:00:00 GMT+0 - Completed - Maintenance has completed successfully Jul 11, 03:00:00 GMT+0 - Identified - Hi there, We'll be provisioning capacity to Liberty Global B.V. in london to improve residential connectivity in the E.U. Region. Upon activation a subset of users may temporarily disconnect. Total maintenance will be less than 5 minutes. Jul 11, 03:00:01 GMT+0 - Identified - Maintenance is now in progress Type: Maintenance

Duration: 1 hour

Affected Components:

<p><small>Jul <var data-var='date'> 11</var>, <var data-var='time'>04:00:00</var> GMT+0</small><br><strong>Completed</strong> - Maintenance has completed successfully.</p> <p><small>Jul <var data-var='date'> 11</var>, <var data-var='time'>03:00:00</var> GMT+0</small><br><strong>Identified</strong> - Hi there, We'll be provisioning capacity to Liberty Global B.V. in london to improve residential connectivity in the E.U. Region. Upon activation a subset of users may temporarily disconnect. Total maintenance will be less than 5 minutes..</p> <p><small>Jul <var data-var='date'> 11</var>, <var data-var='time'>03:00:01</var> GMT+0</small><br><strong>Identified</strong> - Maintenance is now in progress.</p> ]]> Tue, 11 Jul 2023 03:00:00 +0000 https://status.as30456.net/maintenance/cljxn0t6o111803xmojcjm7jrc3 https://status.as30456.net/maintenance/cljxn0t6o111803xmojcjm7jrc3