NERC - Most of the NERC VMs are in "Shutoff" mode – Incident details

All systems operational

GETTING HELP
Email: help@nerc.mghpcc.org or, using the NERC's Support Ticketing System
NERC Documentation: https://nerc-project.github.io/nerc-docs/
Status page for the New England Research Cloud (NERC) and other resources.
Please scroll down to see details on any Incidents or maintenance notices.

Most of the NERC VMs are in "Shutoff" mode

Resolved
Partial outage
Started about 3 years agoLasted about 20 hours

Affected

NERC OPENSTACK

Partial outage from 11:59 PM to 7:50 PM

NERC OPENSTACK COMPUTE SERVICE (NOVA)

Partial outage from 11:59 PM to 7:50 PM

Updates
  • Resolved
    Resolved

    We just resolved the issue!

  • Monitoring
    Monitoring

    We implemented a fix and currently monitoring the result.

  • Identified
    Identified

    The MGHPCC main meter detected an ITIC Level 2 power sag event this afternoon.
    The event affected a single power phase.

    Details Time: 3:10:32 PM Percent of Nominal: 65% Duration: 7 cycles (116 milliseconds)

    The sag coincided with severe thunderstorms that passed through the area between 3 and 4PM.

  • Investigating
    Investigating
    • [x] We noticed most of user projects' VMs are in "Shutoff" status and "Shut Down" power state.
    • [x] User can manually start their instances and that will start your VM and put in "Running" power state and "Running" status.
    • [x] We are currently investigating this incident.
For issues not shown here, please contact NERC via email help@nerc.mghpcc.org or, using the NERC's Support Ticketing System.