Cosmic Global - Widespread Connectivity Issues – Incident details

At approximately 1:43 PM EST, we detected a service anomaly affecting our filter infrastructure. Upon investigation, we determined this incident was a bug related to the earlier issue reported this morning: https://status.as30456.net/cmiivbxm6004wt518aafiq5dd

This caused a brief drop in connections. Within about 60 seconds, we pushed a hotfix and connections stabilized.

Incident Report: DDoS Mitigation Infrastructure Outage

Date: November 28, 2025
Duration: Approximately 2.5 hours
Impact: Full network unavailability

Summary

On November 28, 2025, our DDoS mitigation infrastructure experienced a service interruption lasting approximately one hour. As all inbound and outbound traffic flows through our mitigation systems, this caused the network to appear entirely offline from the outside.

What Happened

Several days ago, we deployed an enhancement to our rate limiting module to improve DDoS mitigation coordination across our globally distributed filtering nodes. After running without issue for several days, a latent memory corruption bug manifested under specific production conditions, causing the filtering nodes to crash. Due to the synchronized nature of our filtering infrastructure—where nodes coordinate with each other to provide consistent global protection—once the issue manifested at one location, it propagated to all locations worldwide within seconds.

Root Cause

The enhancement was written in C++, as it extended an existing C++ module. The code contained a memory corruption bug that only surfaced after several days of continuous runtime under production load. This class of bug is notoriously difficult to detect in testing, as it may only manifest under specific memory states that develop over extended periods.

Resolution

Upon detecting the issue, our engineering team immediately began a full rewrite of the affected module in Rust. Within one hour, we deployed the rewritten module, restoring full service.

Rust is a memory-safe language that eliminates entire categories of bugs at compile time—before code ever runs in production. Unlike C++, where programmers must manually manage memory and the compiler does not verify correctness, Rust's compiler enforces strict ownership and borrowing rules that make memory corruption, buffer overflows, and use-after-free vulnerabilities impossible in safe code. These guarantees are achieved without sacrificing performance; Rust runs as fast as C++ while providing the safety guarantees typically found only in higher-level languages.

For some time now, all new features for our DDoS mitigation systems have been developed in Rust. However, because this particular feature extended an existing C++ module, it was originally written in C++ for consistency.

What we're doing to ensure this doesn't happen again.

Going forward, we will cease using C++ for any new development in our codebase. All future features—including extensions to existing modules—will be written in Rust to prevent this class of issue from occurring again. We will also be replacing any existing C++ code over the next couple of weeks.

Apology

We sincerely apologize for the disruption this caused to your services. We truly appreciate the confidence you place in us to keep your services online and secure. We are committed to learning from this incident and improving our systems to prevent similar issues in the future. Thank you for your understanding and continued support.