Cosmic Global - Multiple PoP Degradation of Service – Incident details

All systems operational

Multiple PoP Degradation of Service

Resolved
Operational
Started over 1 year agoLasted 37 minutes

Affected

Network Points of Presence

Degraded performance from 4:40 PM to 5:03 PM, Operational from 4:40 PM to 5:03 PM, Degraded performance from 5:03 PM to 5:08 PM, Operational from 5:03 PM to 5:17 PM

Los Angeles, California

Operational from 4:40 PM to 5:17 PM

Dallas, Texas

Degraded performance from 4:40 PM to 5:08 PM, Operational from 5:08 PM to 5:17 PM

Ashburn, Virginia

Degraded performance from 4:40 PM to 5:08 PM, Operational from 5:08 PM to 5:17 PM

London, United Kingdom

Degraded performance from 4:40 PM to 5:08 PM, Operational from 5:08 PM to 5:17 PM

Amsterdam, Netherlands

Degraded performance from 4:40 PM to 5:08 PM, Operational from 5:08 PM to 5:17 PM

Updates
  • Resolved
    UTC
    Resolved

    Hi everyone, During an emergency update to our DDoS mitigation systems, an error caused the automated rollback functionality to fail. As a result, the systems entered a continuous boot loop and, due to limited connectivity, were unable to restore the latest version automatically. What would typically be a five-second operation took over 15 minutes as we manually restored the systems.

    Although we normally do not perform updates on weekends, a critical bug required an immediate fix to prevent potential issues later today. We appreciate your understanding. The situation has now been resolved, and no further impact is expected over the remainder of the weekend.

    Thank you.

  • Monitoring
    UTC
    Monitoring

    All degraded PoPs should have recovered/are recovering.

  • Identified
    UTC
    Identified

    Ashburn and Amsterdam should be recovered. We are still working diligently.

  • Investigating
    UTC
    Investigating

    We are investigating a multi-pop degradation of service.