FullHost - Notice history

Post-Incident Follow-Up: Vancouver Data Center Outage - December 11, 2025

We want to provide you with additional information regarding this morning's network outage affecting our Vancouver Data Center.

What Happened

Our upstream provider experienced an unexpected failure affecting multiple fiber paths between Cologix VAN3 and Harbour Center. These paths are designed to be fully redundant and geographically diverse, and should not fail simultaneously. The outage was triggered by emergency maintenance activity involving the relocation of a segment of dark fiber by the fiber provider, which unexpectedly impacted both redundant circuits.

Our Commitment Moving Forward

We sincerely apologize for this disruption to your services. We understand the trust you place in us to keep your operations running smoothly, and we take this responsibility seriously.

Both FullHost and our upstream provider will be conducting comprehensive post-incident reviews to fully understand how both supposedly diverse fiber paths were affected simultaneously. Based on these findings, we will be implementing additional safeguards, which will likely include:

• Adding additional diverse fiber paths between Cologix VAN3 and Harbour Center

• Enhanced monitoring and alerting for upstream connectivity

• Improved communication protocols with our fiber providers regarding maintenance activities

Next Steps

We are committed to transparency throughout this process. Once our joint review is complete, we will share a detailed root cause analysis and the specific improvements we're implementing to prevent similar incidents.
Thank you for your patience and continued trust in FullHost. If you have any questions or concerns, please don't hesitate to reach out to our support team

Partial connectivity has been restored. We will be continuing to monitor as well as getting more information from the upstream provider and share information with you accordingly when it's available.

Our upstream provider currently has a technician on site and is working to migrate services to an alternate fibre path. This should allow for partial connectivity to be restored while full repairs continue.

We will continue to keep you posted on this incident and will share information as it becomes available to us.

Thank you for your patience as we continue to work toward resolving this incident.

We are continuing to work closely with our fibre provider to restore connectivity. Please rest assured that this remains our highest priority and all teams are actively engaged.

We will provide additional updates as soon as more information becomes available.

This is from our upstream provider on the incident:

Approximately 30 minutes ago we were notified by our fibre provider of an emergency maintenance activity involving the relocation of a segment of their dark fibre.

At this time, we are experiencing outages on multiple fibre paths that are designed to be fully redundant and geographically diverse. These paths should not be impacted simultaneously, and we are actively investigating with the provider to understand why both circuits are currently down.

We are pushing for immediate restoration of service and will provide further updates as soon as more information becomes available.

We are continuing to investigate with our upstream providers for an update.

We experienced a temporary network reachability issue affecting the IP range 162.223.224.0/21. A configuration update caused some external networks to temporarily lose access to this range.

What Happened

At 3:32 PM ET, we enabled an RPKI security setting intended to improve routing security. The change contained an error that caused some networks around the world to reject traffic to our IP range.

We identified the issue quickly and rolled back the change at 3:43 PM ET. However, some providers had already cached the incorrect routing information, which delayed full recovery.

Resolution

Recovery began immediately after the rollback. Most networks had restored connectivity by 4:10 PM ET, with full global recovery shortly after.

What We’re Doing to Prevent This

We are reviewing our deployment and validation processes to ensure similar issues do not occur again. Additional safeguards will be implemented before making future routing-related changes.

We sincerely apologize for the disruption and appreciate your patience while we resolved this issue.

This incident has been resolved.

Network connectivity is begin to recover across most networks now, and a full recovery should be expected soon.

We have been advised that today's issue was caused in a failure in some of the network equipment, owned by an upstream provider.

They have rerouted the traffic to stabilize traffic at this time, while the equipment is replaced.

We are seeking clarity about the redundancy aspects that did not kick in on this event.

Please note that this event was unrelated to other incident(s) this month, and we've been given assurance that the root cause of those former incidents were located and resolved.

There was an network issue with an upstream provider and currently is resolved. We are waiting further information on the incident itself. We are additionally aware of recent compounding issues with our Calgary location and our leadership team is reviewing long term plans.

If you have any specific concerns you wish to address, please do not hesitate to reach out to our Quality Assurance department to discuss, including discussing migration options to other locations.

The datacenter network is unstable; investigating the issue.

Network recovered. Servers back up. The data center experienced BGP flaps on one of its primary routers.

This incident has been resolved.

We are currently experiencing network connectivity issues in Calgary Datacenter, and are investigating