Summary
- A conversion bug expanded the scope of planned network maintenance and removed routes from additional Azure devices.
- West US connectivity was disrupted for almost five hours, affecting security, networking, application, data, and AI services.
- Microsoft’s designation under the UK critical-third-party regime places cloud change controls and concentration risk within direct regulatory oversight.
A defect in Microsoft’s automated maintenance process removed network routes from more devices than intended, disrupting access to a major Azure region and allowing one incorrect infrastructure change to affect a broad cloud-service estate.
Microsoft said customers experienced connectivity failures, increased latency, or difficulty reaching services in its West US region between 14:44 and 19:41 UTC on 23 July. Traffic that remained entirely within the region was unaffected, but connections entering or leaving it were disrupted.
The affected services included Azure App Service, API Management, Azure Kubernetes Service, Azure Firewall, Azure Monitor, Microsoft Sentinel, Microsoft Graph, ExpressRoute, VPN Gateway, virtual desktops, databases, analytics platforms, and several AI services.
Routine maintenance had been scheduled to isolate specified network paths. Microsoft’s process converts a maintenance request into instructions for the network estate and checks that redundant paths remain healthy before changes begin.
A defect in the conversion system incorrectly marked additional devices as part of the work. The resulting instructions removed IP routes from more equipment than planned, affecting communication between the West US datacentre and Microsoft’s wider network.
Engineers initially observed large-scale route churn and spent several hours narrowing the problem to the regional datacentre and recent maintenance activity. Rollback began at 17:45 UTC, network connectivity was restored by 18:26, and the affected services had recovered by 19:41.
Microsoft’s preliminary incident review says a fuller investigation will examine the automated maintenance-request process and its safety checks. The initial record does not explain why the unintended devices passed pre-change validation or whether a separate control could have limited the number of routes removed.
Although the failed region is located in the United States, organisations in the UK and Europe can still depend on workloads, suppliers, control planes, identity services, and software components hosted there. A customer’s principal data region does not necessarily describe every infrastructure dependency involved in delivering its service.
Microsoft Ireland Operations Ltd also became one of the first technology providers designated as a critical third party to the UK financial sector in July. The Bank of England, Prudential Regulation Authority, and Financial Conduct Authority can now oversee services whose disruption could affect several institutions or wider market stability.
The designation does not transfer responsibility away from banks, insurers, and market-infrastructure operators. Regulated organisations remain accountable for outsourcing decisions, continuity arrangements, recovery objectives, and their ability to operate within tolerable disruption.
Multi-region architecture can reduce dependence on one location, but effective failover requires more than replicated application data. Identity, DNS, monitoring, administrative access, deployment pipelines, security controls, and communications must remain available when the primary region or its external connectivity fails.
A secondary environment that relies on the same unavailable management service or network path may provide little practical protection. Recovery plans therefore need to be exercised under conditions that remove control-plane functions and operator access, rather than assuming that only an application workload will fail.
The incident also demonstrates the scale at which infrastructure automation can propagate an error. Automated maintenance allows cloud providers to operate large estates consistently, yet a mistake in request translation can apply a broader change than manual intervention would normally reach.
Change controls need to constrain the potential blast radius of authorised automation. Limits on the number of devices altered at once, staged execution, independent validation, route-change thresholds, live health checks, and rapid rollback can interrupt an incorrect plan before it affects an entire regional boundary.
Microsoft restored the region without reporting data loss or a security compromise. Its final review will need to account for how the maintenance request expanded beyond its intended scope and why the safety mechanisms did not stop the additional route removals before customer services were disrupted.
