Cosmic Global - Power outage at DFW Evocative – Incident details

Power outage at DFW Evocative

Resolved
Operational
Started 6 months agoLasted about 17 hours

Affected

Network Points of Presence

Major outage from 11:51 AM to 3:34 PM, Degraded performance from 3:34 PM to 5:14 AM

Dallas, TX

Major outage from 11:51 AM to 3:34 PM, Degraded performance from 3:34 PM to 5:14 AM

Services

Major outage from 11:51 AM to 3:34 PM, Degraded performance from 3:34 PM to 5:14 AM

API

Major outage from 11:51 AM to 3:34 PM, Degraded performance from 3:34 PM to 5:14 AM

Updates
  • Resolved
    Resolved

    We have restored 99% of services under our control, and thus we are closing this issue. Any remaining issues are still being looked into as isolated issues out of our control. We do not expect any further issues. The datacenter is in the process of performing a root cause analysis to figure out what went wrong during today's power outage, and they have already had the UPS vendors come out and check everything over and get it all operational again. Thank you for your patience today.

  • Update
    Update

    We are nearing the end of the issues caused by the power outage today. We do apologize that this has taken longer than expected, and are hoping to restore the rest of our affected clients in the next few hours. We will continue to update here when possible. Thank you for your patience.

  • Update
    Update

    We are still working on restoring the remaining 5% of customers affected in our Dallas data center. It is isolated to a few racks at this point, and seems to have been caused by the power outage itself. We will continue updating as soon as we can.

  • Update
    Update

    We have restored 95% of customer services in DFW Evocative. We will update when all clear.

  • Update
    Update

    As of 15 minutes ago, DFW Evocative’s power has recovered and services have started to come online. We will report when all clear.

  • Monitoring
    Monitoring

    Services are being restored and we are currently monitoring the result.

  • Update
    Update

    DFW Evocative’s datacenter staff are investigating an issue with their UPS equipment. ETA provided by the datacenter has been missed and we have not yet been provided an additional ETA. We will update as soon as possible.

  • Update
    Update

    DFW Evocative is investigating an issue with their UPS equipment. They are communicating with the vendor.

  • Investigating
    Investigating

    There is a power outage in DFW Evocative related to a storm. Our DFW Equinix location is still online. We have been provided a 30-60 minute ETA by the facility.