Cosmic Global - Notice history

We are marking this as resolved as everything has cleared up after both maintenance events.

The GTT instability is still being investigated with them.

We have since disabled Telia routing in Dallas until the optics are replaced, and everything has been stable since for over an hour.

We have implemented a new upstream link to restore service for the time being. GTT is still investigating the issue on their end; we suspect either a dark fiber cut or a core system failure. So far they have acknowledged the issue and will provide updates over the next hour.

Once GTT service is restored, we will be able to cut back over without any interruption to traffic, and the new link will remain in place as a backup to increase resilience. We do not anticipate any further issues at this time.

We are working to spin up an emergency link to restore service until the GTT connection at LAX Equinix is repaired.

For context: GTT is used to backhaul traffic between our CoreSite and Equinix facilities. While we do maintain redundant links for this purpose, both are currently down, which has effectively left us without connectivity until the issue is resolved. To address this, we are bringing an additional link online through a separate upstream provider — both to restore service now and to provide added resilience against a similar failure in the future.

We have confirmed that the issue is not caused by a broken uplink on the upstream device as initially suspected. Instead, the GTT port we receive at this location is experiencing problems. We have shifted our focus accordingly and are in contact with GTT for further updates.

As of this morning, the data center has confirmed that the rooftop condensers are fully back in working order, along with the AC units that tripped under the increased pressure when the condensers failed. We are now seeing ambient temperatures restored to normal levels within the facility and do not anticipate any further issues.

Ambient temperatures in the London data center have returned to near-normal levels. An HVAC technician is on site this morning to complete repairs on the remaining air conditioning unit and to verify the integrity of the other cooling systems in the facility.

We will continue to monitor the situation closely and will provide a final update once all systems have been fully restored and verified.

Technicians are on site actively working to repair the rooftop condenser and restore the facility to normal operating temperatures. We will continue to provide updates as the situation develops.

We have confirmed the source of the issue: a failed condenser unit on the roof of the data center, compounded by a separate fault with an air conditioning unit inside the facility. Combined with the ongoing heat wave, these failures have caused ambient temperatures to rise to levels that are triggering thermal throttling and, in some cases, thermal-related crashes on affected hardware.

We suffered from a dark fiber cut in London. Due to a policy issue in our routing configuration, failover to our secondary dark fiber route was not instantaneous. This has been resolved. Traffic is now flowing over our 2nd path and traffic has been restored. In the meantime we'll be repairing the primary path.

We have identified an issue with our primary dark fiber connection and provider. We are working on resolving the issue with that, and in the processing of forcing a failover of traffic.