Cosmic Global - Carrier Instability in Dallas – Incident details

Dallas, Texas experiencing degraded performance

Carrier Instability in Dallas

Monitoring
Degraded performance
Started 2 days ago

Affected

Network Points of Presence

Degraded performance from 5:33 PM to 8:07 PM, Operational from 8:07 PM to 3:08 PM, Degraded performance from 3:08 PM to 12:00 AM

Dallas, Texas

Degraded performance from 5:33 PM to 8:07 PM, Operational from 8:07 PM to 3:08 PM, Degraded performance from 3:08 PM to 12:00 AM

Updates
  • Monitoring
    Monitoring

    Telia Link - The Telia link was restored last night at approximately 7:00 PM CST and has remained stable since. While we observed normal light levels and no errors on the port on our end, we proactively replaced the optic, which appears to have resolved the issue.

    GTT Link - The GTT port showed similar symptoms—normal light levels with no errors. We replaced that optic yesterday as well, but the link continued to experience problems. GTT was scheduled to perform maintenance on their end at 1:00 AM CST but missed that window. We've temporarily removed this link from service and are awaiting their maintenance, which we've asked them to complete as soon as possible.

  • Identified
    Identified

    The GTT instability is still being investigated with them.

    We have since disabled Telia routing in Dallas until the optics are replaced, and everything has been stable since for over an hour.

  • Investigating
    Investigating

    We are currently investigating two separate issues in the Dallas region. We have an open ticket with one upstream regarding potential congestion we are seeing on their end. Separately, we have identified a faulty optic in a separate transit partner and we have dispatched someone to replace the faulty optics.