Cosmic Global - GTT Outages – Incident details

All systems operational

GTT Outages

Resolved
Operational
Started about 3 years agoLasted 5 days

Affected

Network Points of Presence

Operational from 1:56 AM to 3:52 AM

Los Angeles, California

Operational from 1:56 AM to 3:52 AM

Dallas, Texas

Operational from 1:56 AM to 3:52 AM

Ashburn, Virginia

Operational from 1:56 AM to 3:52 AM

London, United Kingdom

Operational from 1:56 AM to 3:52 AM

Amsterdam, Netherlands

Operational from 1:56 AM to 3:52 AM

Updates
  • Resolved
    UTC
    Resolved

    This incident has been resolved.

  • Monitoring
    UTC
    Monitoring

    Hello! The GTT maintenance completed a few days ago, and since then we've been in the monitoring phase. We now believe it is safe to turn back on our AMS circuit through GTT. We plan to do this in around 1 hour at 12:30PM EST. We do expect this to cause a slight disturbance for a few minutes as traffic in those regions reroute for players who will ingress through AMS. We do not expect any other issues to follow, however we will be closely monitoring the situation.

  • Identified
    UTC
    Identified

    GTT Tier 3 Support has identified the issue with their circuits following their maintenance this weekend. They plan to rectify the issue around 22:00 UTC, and their expected impact window is to last 360 minutes. Final completion of any maintenance is set for the 18th at 04:00 UTC. We will be monitoring their maintenance closely, and will reactivate AMS routing once we see it is safe to do so.

  • Monitoring
    UTC
    Monitoring

    Hello,

    Early this morning between 12AM-6AM Eastern Time, we identified and quickly reported numerous drops within the AMS region which caused other regions to potentially have some traffic issues as well. These drops were well past the scheduled maintenance period with our upstream GTT. We identified these issues to be within GTT circuits and have escalated them to their highest tier of support and are working with them to understand why this is happening and how they are quickly rectifying it.

    For the time being, we are routing traffic around Europe as best as we can, and we have disabled GTT routing in the AMS region until we can confirm this has been fixed.

    We are also signing with and bringing on a third upstream in the next few days which will help to optimize routing, increase capacity, and provide some support in an issue like this so we can alleviate the load from a carrier which is having issues.

    We will provide more updates when we have them.