NERC - Most of the NERC VMs are in "Shutoff" mode – Incident details

GETTING HELP
Email: help@nerc.mghpcc.org or, using the NERC's Support Ticketing System
NERC Documentation: https://nerc-project.github.io/nerc-docs/
Status page for the New England Research Cloud (NERC) and other resources.
Please scroll down to see details on any Incidents or maintenance notices.

Most of the NERC VMs are in "Shutoff" mode

ResolvedPartial outage19 hours 50 minutes

Affected

NERC OPENSTACK

Partial outage from 11:59 PM to 7:50 PM

NERC OPENSTACK COMPUTE SERVICE (NOVA)

Partial outage from 11:59 PM to 7:50 PM

  • ResolvedUTC

    We just resolved the issue!

  • MonitoringUTC

    We implemented a fix and currently monitoring the result.

  • IdentifiedUTC

    The MGHPCC main meter detected an ITIC Level 2 power sag event this afternoon.
    The event affected a single power phase.

    Details Time: 3:10:32 PM Percent of Nominal: 65% Duration: 7 cycles (116 milliseconds)

    The sag coincided with severe thunderstorms that passed through the area between 3 and 4PM.

  • InvestigatingUTC
    • [x] We noticed most of user projects' VMs are in "Shutoff" status and "Shut Down" power state.
    • [x] User can manually start their instances and that will start your VM and put in "Running" power state and "Running" status.
    • [x] We are currently investigating this incident.