FullHost - Toronto Data Center – Incident details

All systems operational

Toronto Data Center

Resolved
Major outage
Started 27 days agoLasted 36 minutes

Affected

FullHost.Com

Major outage from 8:45 PM to 9:20 PM

Toronto

Major outage from 8:45 PM to 9:20 PM

Elastic Hosting

Major outage from 8:45 PM to 9:20 PM

WordPress Hosting

Major outage from 8:45 PM to 9:20 PM

Virtual Machines

Major outage from 8:45 PM to 9:20 PM

Updates
  • Postmortem
    Postmortem

    We experienced a temporary network reachability issue affecting the IP range 162.223.224.0/21. A configuration update caused some external networks to temporarily lose access to this range.

    What Happened


    At 3:32 PM ET, we enabled an RPKI security setting intended to improve routing security. The change contained an error that caused some networks around the world to reject traffic to our IP range.

    We identified the issue quickly and rolled back the change at 3:43 PM ET. However, some providers had already cached the incorrect routing information, which delayed full recovery.

    Resolution

    Recovery began immediately after the rollback. Most networks had restored connectivity by 4:10 PM ET, with full global recovery shortly after.

    What We’re Doing to Prevent This

    We are reviewing our deployment and validation processes to ensure similar issues do not occur again. Additional safeguards will be implemented before making future routing-related changes.

    We sincerely apologize for the disruption and appreciate your patience while we resolved this issue.

  • Resolved
    Resolved

    This incident has been resolved.

  • Monitoring
    Monitoring

    Network connectivity is begin to recover across most networks now, and a full recovery should be expected soon.

  • Investigating
    Investigating

    We are experiencing intermittent connection issues from some networks. We are working to resolve this as a high priority. More information to come.