Cloudflare has confirmed that the underlying issue—the internal service degradation affecting their Global Network—is fixed and they have exited their monitoring phase for the primary network services.
Menuella Status: Our internal monitoring confirms that error rates and latency for Menuella have returned to normal pre-incident baseline levels. The services are stable and fully operational.
Root Cause: The outage was confirmed by Cloudflare to be caused by a latent bug in a service underpinning their bot mitigation capability that cascaded across the global network after a routine configuration change.
Post-Mortem Note: While the immediate crisis is over, we will review the incident in our post-mortem to determine if any compensating controls or architectural changes are necessary to further insulate Menuella from future widespread third-party outages.
Resolved
Cloudflare has confirmed that the underlying issue—the internal service degradation affecting their Global Network—is fixed and they have exited their monitoring phase for the primary network services.
Menuella Status: Our internal monitoring confirms that error rates and latency for Menuella have returned to normal pre-incident baseline levels. The services are stable and fully operational.
Root Cause: The outage was confirmed by Cloudflare to be caused by a latent bug in a service underpinning their bot mitigation capability that cascaded across the global network after a routine configuration change.
Post-Mortem Note: While the immediate crisis is over, we will review the incident in our post-mortem to determine if any compensating controls or architectural changes are necessary to further insulate Menuella from future widespread third-party outages.
Monitoring
The underlying Cloudflare Global Network degradation has been resolved by Cloudflare, who is now in a monitoring phase. As Cloudflare acts as a critical intermediary for much of the internet's traffic (including security, caching, and load balancing), any impact on Menuella that was related to increased external 5xx errors or network latency should now be subsiding or fully resolved.
Investigating
Cloudflare is currently experiencing an internal service degradation impacting its Global Network, leading to a widespread increase in 5xx errors and intermittent impact on customer services globally. While Cloudflare has reported seeing services recover and is continuing remediation efforts, customers may still observe higher-than-normal error rates.