FullHost - Investigating: Connectivity issues on hypervisor tor-kvm104 – Incident details

All systems operational

Investigating: Connectivity issues on hypervisor tor-kvm104

Resolved
Major outage
Started about 1 month agoLasted 1 hour 10 minutes

Affected

FullHost.Com

Major outage from 3:24 PM to 4:34 PM

Toronto

Major outage from 3:24 PM to 4:34 PM

WordPress Hosting

Major outage from 3:24 PM to 4:34 PM

Virtual Machines

Major outage from 3:24 PM to 4:34 PM

Updates
  • Resolved
    UTC
    Resolved

    Moving this to resolved, but we are continuing to monitor this incident closely.

  • Monitoring
    UTC
    Monitoring

    Our on-site engineers have corrected the issue and the performance degradation affecting services in this pod has been resolved. All affected servers are operating normally.

    We are continuing to closely monitor the situation.

  • Identified
    UTC
    Identified

    One of the redundant power feeds serving a pod at our 151 Front St. facility is currently offline. The secondary feed remains active and servers in the affected pod are powered and online. However, a firmware bug in the affected Supermicro hardware is causing CPUs to throttle when running on a single power feed, resulting in significantly degraded performance for services hosted in this pod. Our engineers are actively working on a resolution and will post an update as soon as more information is available.

  • Update
    UTC
    Update

    Incident is affecting multiple hypervisors in Toronto

  • Investigating
    UTC
    Investigating

    We are aware of connectivity issues affecting hypervisor tor-kvm104 in our Toronto facility. A small subset of client VMs hosted on this node may be experiencing intermittent or degraded network connectivity. Our engineers are actively investigating and will provide an update as more information becomes available.