NERC - Notice history

All systems operational

GETTING HELP
Email: help@nerc.mghpcc.org or, using the NERC's Support Ticketing System
NERC Documentation: https://nerc-project.github.io/nerc-docs/
Status page for the New England Research Cloud (NERC) and other resources.
Please scroll down to see details on any Incidents or maintenance notices.

MGHPCC SHARED SERVICES (MGHPCC-SS) ACCOUNT PORTAL - Operational

100% - uptime
Jun 2023 · 99.92%Jul · 100.0%Aug · 100.0%

Previous page

Next page

Jun 2023
Jul 2023
Aug 2023

NERC COLDFRONT - Operational

100% - uptime
Jun 2023 · 100.0%Jul · 100.0%Aug · 100.0%

Previous page

Next page

Jun 2023
Jul 2023
Aug 2023

NERC ACADEMIC (EDU) OPENSHIFT CLUSTER - Operational

100% - uptime
Jun 2023 · 100.0%Jul · 100.0%Aug · 100.0%

Previous page

Next page

Jun 2023
Jul 2023
Aug 2023

NETWORKING - Operational

100% - uptime
Jun 2023 · 100.0%Jul · 100.0%Aug · 100.0%

Previous page

Next page

Jun 2023
Jul 2023
Aug 2023

STORAGE - Operational

100% - uptime
Jun 2023 · 100.0%Jul · 100.0%Aug · 100.0%

Previous page

Next page

Jun 2023
Jul 2023
Aug 2023
100% - uptime

NERC WEBSITE - Operational

100% - uptime
Jun 2023 · 100.0%Jul · 100.0%Aug · 100.0%

Previous page

Next page

Jun 2023
Jul 2023
Aug 2023

NERC DOCUMENTATION WEBSITE - Operational

100% - uptime
Jun 2023 · 100.0%Jul · 100.0%Aug · 100.0%

Previous page

Next page

Jun 2023
Jul 2023
Aug 2023

NERC TICKETING SYSTEM - Operational

100% - uptime
Jun 2023 · 100.0%Jul · 99.91%Aug · 100.0%

Previous page

Next page

Jun 2023
Jul 2023
Aug 2023

Notice history

Aug 2023

NERC OpenShift Container Platform (OCP) Maintenance [August 21, 2023 9:00 AM - 5:00 PM]
  • Completed
    August 21, 2023 at 4:53 PMUTC
    Completed
    August 21, 2023 at 4:53 PMUTC

    Maintenance has completed successfully.

  • In progress
    August 21, 2023 at 1:00 PMUTC
    In progress
    August 21, 2023 at 1:00 PMUTC

    Maintenance is now in progress

  • Planned
    August 21, 2023 at 1:00 PMUTC
    Planned
    August 21, 2023 at 1:00 PMUTC

    NERC’s planned OpenShift container platform (OCP) maintenance will occur on Monday August 21, 2023 from 9:00 AM – 5:00 PM.

    GENERAL MAINTENANCE

    • We will be moving the OpenShift container platform worker nodes to a new location within the datacenter. The core OpenShift services will be interrupted during this time. Any critical workloads that are deployed in the cluster need to be stopped until the maintenance is complete.

    NOTICES

    • We will be powering off all OpenShift cluster hosts prior to the move.

    • The NERC OpenShift cluster will be unavailable during this time. Please let us know If you encounter any issues after the maintenance has completed.

    • The estimated time to complete this update is 8 hours. It can take more or less time, so we urge you to keep an eye on https://nerc.instatus.com/ to get the progress during this time.

    • Please do subscribe to the NERC’s status for any future updates: https://nerc.instatus.com/subscribe/email

    Our priority is to help make science happen, so If you or your research team have any questions or need to escalate an issue, please contact us via email (help@nerc.mghpcc.org).

    Thanks,

    New England Research Cloud (NERC)

    https://nerc.mghpcc.org/

    https://nerc-project.github.io/nerc-docs/

Jul 2023

Unexpected power shut down of all NERC OpenStack VMs
  • Resolved
    UTC
    Resolved

    We now know that this is due to the MGHPCC power sag event and now has been resolved.

    The MGHPCC main meter detected an ITIC Level 2 power sag event at 3:59PM on 
Thursday July 27, during the wave of thunderstorms that passed through the area.   
The event affected a single power phase.  

Details
    Date:  Thursday July 27, 2023
    Time: 15:59:25 PM
    Percent of Nominal: 67%
    Duration: 6 cycles (100 milliseconds)
  • Monitoring
    UTC
    Monitoring

    We implemented a fix and are currently monitoring the result.

  • Investigating
    UTC
    Investigating

    It looks like most of the NERC's OpenStack VMs are in shut down state. You might need to manually restart them from OpenStack Dashboard: https://stack.nerc.mghpcc.org/dashboard/project/instances/

    We are currently investigating this incident.

OTHER WEBSITES & TOOLS - NERC TICKETING SYSTEM isn't accessible at the moment (automated)
  • Resolved
    UTC
    Resolved

    OTHER WEBSITES & TOOLS - NERC TICKETING SYSTEM is now operational! This update was created by an automated monitoring service.

  • Investigating
    UTC
    Investigating

    OTHER WEBSITES & TOOLS - NERC TICKETING SYSTEM cannot be accessed at the moment. This incident was created by an automated monitoring service.

Upcoming NERC system maintenance and update Monday July 24, 2023 9:00 AM – 01:00 PM
  • Completed
    July 24, 2023 at 4:30 PMUTC
    Completed
    July 24, 2023 at 4:30 PMUTC

    Maintenance has completed successfully.

  • In progress
    July 24, 2023 at 1:00 PMUTC
    In progress
    July 24, 2023 at 1:00 PMUTC

    Maintenance is now in progress

  • Planned
    July 24, 2023 at 1:00 PMUTC
    Planned
    July 24, 2023 at 1:00 PMUTC

    NERC’s planned system maintenance and update will occur on Monday July 10, 2023 from 9:00 AM – 01:00 PM.

    GENERAL MAINTENANCE

    • We are conducting follow-up maintenance to the July 10th update. This requires updating the OpenStack nova configuration in order to support the new NVIDIA V100 GPUs. This will enable us to offer additional GPU flavors on the NERC OpenStack cluster.

    NOTICES

    • Any running user VMs/containers/workloads should not be interrupted.

    • This work will involve a brief outage to the NERC OpenStack’s nova scheduler.

    • Users will be able to access their VMs and storage via configured SSH and API settings.

    • The estimated time to complete this update is 4 hours. It can take more or less time, so we urge you to keep an eye on https://nerc.instatus.com/ to get the progress during this time.

    • Please do subscribe to the NERC’s status for any future updates: https://nerc.instatus.com/subscribe/email

    Our priority is to help make science happen, so If you or your research team have any questions or need to escalate an issue, please contact us via email (help@nerc.mghpcc.org).

    Thanks,

    New England Research Cloud (NERC)

    https://nerc.mghpcc.org/

    https://nerc-project.github.io/nerc-docs/

NERC's ColdFront version update to address the latest security fix.
  • Completed
    July 17, 2023 at 1:34 PMUTC
    Completed
    July 17, 2023 at 1:34 PMUTC

    Maintenance has completed successfully.

  • Planned
    July 17, 2023 at 1:15 PMUTC
    Planned
    July 17, 2023 at 1:15 PMUTC

    We are planning for a scheduled maintenance during that time.
    Details:
    We will update ColdFront to the latest released version, which addresses a security vulnerability in the code.

Upcoming NERC system maintenance and update Monday July 10, 2023 9:00 AM – 01:00 PM
  • Completed
    July 10, 2023 at 7:02 PMUTC
    Completed
    July 10, 2023 at 7:02 PMUTC

    Maintenance has completed successfully.

  • Update
    July 10, 2023 at 5:53 PMUTC
    Update
    July 10, 2023 at 5:53 PMUTC

    Maintenance is now in progress. We are extending this update for another hour.

  • Update
    July 10, 2023 at 5:11 PMUTC
    Update
    July 10, 2023 at 5:11 PMUTC

    Maintenance is now in progress. We are extending this update for another hour.

  • In progress
    July 10, 2023 at 2:37 PMUTC
    In progress
    July 10, 2023 at 2:37 PMUTC

    Maintenance is now in progress.

  • Planned
    July 10, 2023 at 1:00 PMUTC
    Planned
    July 10, 2023 at 1:00 PMUTC

    NERC’s planned system maintenance and update will occur on Monday July 10, 2023 from 9:00 AM – 01:00 PM.

    GENERAL MAINTENANCE

    • Our plan includes enhancing the NERC’s OpenStack cluster by adding additional NVIDIA V100 GPUs. Additionally, we will be making modifications to the MGHPCC Shared Services Account Portal (aka RegApp) and Keycloak instance in order to allow logins from additional domains. Consequently, during this time, both new user registrations and login to NERC web services may be temporarily impacted.

    NOTICES

    • We will be adding additional NVIDIA V100 GPUs to the NERC’s OpenStack cluster configuration which will enable us to distribute more GPU based flavors.

    • Any running user VMs/containers/workloads should not be interrupted.

    • Users will be able to access their VMs and storage via configured SSH and API settings.

    • The estimated time to complete this update is 4 hours. It can take more or less time, so we urge you to keep an eye on https://nerc.instatus.com/ to get the progress during this time.

    • Please do subscribe to the NERC’s status for any future updates: https://nerc.instatus.com/subscribe/email

    Our priority is to help make science happen, so If you or your research team have any questions or need to escalate an issue, please contact us via email (help@nerc.mghpcc.org).

    Thanks,

    New England Research Cloud (NERC)

    https://nerc.mghpcc.org/

    https://nerc-project.github.io/nerc-docs/

Jun 2023

We are facing issues with approved resource allocation in both NERC OpenStack and OpenShift based resources.
  • Resolved
    UTC
    Resolved

    This incident has been resolved.

    **NOTE: ** If you are experiencing ongoing difficulties with your previously approved resource allocations, please reach out to us via email at help@nerc.mghpcc.org so that we can promptly assist you in resolving the issues.

  • Monitoring
    UTC
    Monitoring

    We implemented a fix and are currently monitoring the result.

  • Investigating
    UTC
    Investigating

    We are currently investigating this incident.

MGHPCC SHARED SERVICES (MGHPCC-SS) ACCOUNT PORTAL isn't accessible at the moment (automated)
  • Resolved
    UTC
    Resolved

    MGHPCC SHARED SERVICES (MGHPCC-SS) ACCOUNT PORTAL is now operational! This update was created by an automated monitoring service.

  • Investigating
    UTC
    Investigating

    MGHPCC SHARED SERVICES (MGHPCC-SS) ACCOUNT PORTAL cannot be accessed at the moment. This incident was created by an automated monitoring service.

MGHPCC SHARED SERVICES (MGHPCC-SS) ACCOUNT PORTAL isn't accessible at the moment (automated)
  • Resolved
    UTC
    Resolved

    MGHPCC SHARED SERVICES (MGHPCC-SS) ACCOUNT PORTAL is now operational! This update was created by an automated monitoring service.

  • Investigating
    UTC
    Investigating

    MGHPCC SHARED SERVICES (MGHPCC-SS) ACCOUNT PORTAL cannot be accessed at the moment. This incident was created by an automated monitoring service.

MGHPCC SHARED SERVICES (MGHPCC-SS) ACCOUNT PORTAL isn't accessible at the moment (automated)
  • Resolved
    UTC
    Resolved

    MGHPCC SHARED SERVICES (MGHPCC-SS) ACCOUNT PORTAL is now operational! This update was created by an automated monitoring service.

  • Investigating
    UTC
    Investigating

    MGHPCC SHARED SERVICES (MGHPCC-SS) ACCOUNT PORTAL cannot be accessed at the moment. This incident was created by an automated monitoring service.

NERC OPENSHIFT CONTAINER PLATFORM - NERC OPENSHIFT WEB CONSOLE isn't accessible at the moment (automated)
  • Resolved
    UTC
    Resolved

    NERC OPENSHIFT CONTAINER PLATFORM - NERC OPENSHIFT WEB CONSOLE is now operational! This update was created by an automated monitoring service.

  • Investigating
    UTC
    Investigating

    NERC OPENSHIFT CONTAINER PLATFORM - NERC OPENSHIFT WEB CONSOLE cannot be accessed at the moment. This incident was created by an automated monitoring service.

Jun 2023 to Aug 2023

Next

For issues not shown here, please contact NERC via email help@nerc.mghpcc.org or, using the NERC's Support Ticketing System.