Cosmic Global - Notice history

We have implemented a new upstream link to restore service for the time being. GTT is still investigating the issue on their end; we suspect either a dark fiber cut or a core system failure. So far they have acknowledged the issue and will provide updates over the next hour.

Once GTT service is restored, we will be able to cut back over without any interruption to traffic, and the new link will remain in place as a backup to increase resilience. We do not anticipate any further issues at this time.

We are working to spin up an emergency link to restore service until the GTT connection at LAX Equinix is repaired.

For context: GTT is used to backhaul traffic between our CoreSite and Equinix facilities. While we do maintain redundant links for this purpose, both are currently down, which has effectively left us without connectivity until the issue is resolved. To address this, we are bringing an additional link online through a separate upstream provider — both to restore service now and to provide added resilience against a similar failure in the future.

We have confirmed that the issue is not caused by a broken uplink on the upstream device as initially suspected. Instead, the GTT port we receive at this location is experiencing problems. We have shifted our focus accordingly and are in contact with GTT for further updates.

As of this morning, the data center has confirmed that the rooftop condensers are fully back in working order, along with the AC units that tripped under the increased pressure when the condensers failed. We are now seeing ambient temperatures restored to normal levels within the facility and do not anticipate any further issues.

Ambient temperatures in the London data center have returned to near-normal levels. An HVAC technician is on site this morning to complete repairs on the remaining air conditioning unit and to verify the integrity of the other cooling systems in the facility.

We will continue to monitor the situation closely and will provide a final update once all systems have been fully restored and verified.

Technicians are on site actively working to repair the rooftop condenser and restore the facility to normal operating temperatures. We will continue to provide updates as the situation develops.

We have confirmed the source of the issue: a failed condenser unit on the roof of the data center, compounded by a separate fault with an air conditioning unit inside the facility. Combined with the ongoing heat wave, these failures have caused ambient temperatures to rise to levels that are triggering thermal throttling and, in some cases, thermal-related crashes on affected hardware.

We suffered from a dark fiber cut in London. Due to a policy issue in our routing configuration, failover to our secondary dark fiber route was not instantaneous. This has been resolved. Traffic is now flowing over our 2nd path and traffic has been restored. In the meantime we'll be repairing the primary path.

We have identified an issue with our primary dark fiber connection and provider. We are working on resolving the issue with that, and in the processing of forcing a failover of traffic.

Our explanation:

Over the past few weeks, some of you may have experienced brief service interruptions across parts of our network. We want to be upfront about what happened, what we've learned, and most importantly what we've done about it.

Our edge routers have always been built with internal redundancy in mind - redundant supervisors, redundant power supplies, multiple line cards, and redundant fabric modules. That level of hardware resilience handles the vast majority of failure scenarios well.

However, the recent outages exposed a gap: when an issue affects the chassis itself such as a software defect, a firmware upgrade that requires a full reload, or cases like where a software process on a router crashes (as has happened recently in London -> Twice) - there was no second device to immediately absorb the traffic. The router was redundant in every way except the one that mattered in these incidents.

What we're doing:

We're rolling out a dual router design across all six of our points of presence — Dallas, Ashburn, Los Angeles, London, Amsterdam, and Frankfurt. Once complete, every PoP will operate with two independent edge routers in an active/active configuration, with full BGP session redundancy to all upstream and peering partners. If an entire chassis needs to be taken offline for maintenance, a software upgrade, or an unexpected failure then traffic will automatically reconverge on the second device with no customer-facing impact.

Each router in the pair will run on independent power feeds with independent management and control planes. We're also using this as an opportunity to standardize failover testing procedures across all PoPs, so this architecture is validated continuously, not just at deployment. This also provides protection against cases where a configuration change (with possibly human error involved) leads to a change which ends up knocking out a bunch of traffic. The investments for these changes were made during the last couple of weeks, so were already in the works and unrelated to incidents in March, but with summer right around the corner we wanted to let you know you'll be in good hands.

These changes will also allow for things like DDoS mitigation changes to be performed in a more controlled rollout (e.g. to parts of traffic only), zero-downtime maintenance windows for core networking equipment and a stronger foundation for the capacity expansions we have planned for the rest of 2026.

The incident was resolved shortly after onset, and we have been monitoring since. A full RFO will be posted when this status is closed. In short, the root cause was a cascading failure triggered by a bug in the routing software itself — not by any action taken by our team. The issue was entirely outside our control, and we responded quickly to restore normal operation.

We identified the root cause, and have been working on resolution. Recovery efforts are showing progress as traffic is beginning to restore in Dallas.

We located the problem to be a private transport link between AMS and LON had caused the drop due to issues outside of our control. The resulting drop was because of the traffic shifting over. As a result of the private transport link failure, we have switched the mode of traffic for this transport, and do not expect it to occur again, however we are monitoring the situation in the event it does. We have also linked this private link to a few other recent small drops in the EU region (unrelated to the last status post which was a hardware issue), and apologize for any impact from that.

In regards to network stability, our London PoP will be undergoing an upgrade to improve a routing appliance constraint, and furthermore we will be implementing increased redundancy across all PoPs in the near future to increase resiliency during both planned and unplanned outages.

Following emergency maintenance yesterday that required a reboot of a core router in our London facility, an Arista runtime software bug caused the router's ARP entries to gradually decay from active memory.

Although the router's configuration remained correct throughout, the hardware chip (ASIC) responsible for directing network traffic failed to correctly reload the address mappings after the reboot. These mappings are what tell the router how to reach a set of internal endpoints used for multicast traffic forwarding. With them missing from the hardware's active memory, traffic that should have been flowing through those paths was silently dropped.

Because the configuration itself was never corrupted, the root cause was not immediately obvious. A number of other potential causes were investigated before the true issue was identified — a desync between the router's stored configuration and what the hardware had actually loaded into memory.

We sincerely apologize for the impact this had on your services and for the time it took to identify the root cause. We understand how frustrating extended investigations can be, and we appreciate your patience while our engineers worked methodically through the contributing factors to reach a definitive resolution.

We have implemented another round of fixes and connectivity is recovering. Please reach out to us if you are still having issue while we continue to monitor.

Thank you again for your patience in this matter. We will provide a full report when we confirm all is well.

We are continuing to investigate TCP issues in the London PoP. We apologize for the continued problems today and are making progress toward a full resolution for this location.

We are continuing to monitor reports of elevated issues and are still working towards a permanent resolution.

We have rolled out a batch of fixes and are seeing connectivity recover. We are continuing to monitor the situation closely.