NERC - Notice history

All systems operational

GETTING HELP
Email: help@nerc.mghpcc.org or, using the NERC's Support Ticketing System
NERC Documentation: https://nerc-project.github.io/nerc-docs/
Status page for the New England Research Cloud (NERC) and other resources.
Please scroll down to see details on any Incidents or maintenance notices.

MGHPCC SHARED SERVICES (MGHPCC-SS) ACCOUNT PORTAL - Operational

100% - uptime
May 2023 · 100.0%Jun · 99.92%Jul · 100.0%

Previous page

Next page

May 2023
Jun 2023
Jul 2023

NERC COLDFRONT - Operational

100% - uptime
May 2023 · 100.0%Jun · 100.0%Jul · 100.0%

Previous page

Next page

May 2023
Jun 2023
Jul 2023

NETWORKING - Operational

100% - uptime
May 2023 · 100.0%Jun · 100.0%Jul · 100.0%

Previous page

Next page

May 2023
Jun 2023
Jul 2023

STORAGE - Operational

100% - uptime
May 2023 · 100.0%Jun · 100.0%Jul · 100.0%

Previous page

Next page

May 2023
Jun 2023
Jul 2023
100% - uptime

NERC WEBSITE - Operational

100% - uptime
May 2023 · 100.0%Jun · 100.0%Jul · 100.0%

Previous page

Next page

May 2023
Jun 2023
Jul 2023

NERC DOCUMENTATION WEBSITE - Operational

100% - uptime
May 2023 · 100.0%Jun · 100.0%Jul · 100.0%

Previous page

Next page

May 2023
Jun 2023
Jul 2023

NERC TICKETING SYSTEM - Operational

100% - uptime
May 2023 · 100.0%Jun · 100.0%Jul · 99.91%

Previous page

Next page

May 2023
Jun 2023
Jul 2023

Notice history

Jul 2023

Unexpected power shut down of all NERC OpenStack VMs
  • Resolved
    Resolved

    We now know that this is due to the MGHPCC power sag event and now has been resolved.

    The MGHPCC main meter detected an ITIC Level 2 power sag event at 3:59PM on 
Thursday July 27, during the wave of thunderstorms that passed through the area.   
The event affected a single power phase.  

Details
    Date:  Thursday July 27, 2023
    Time: 15:59:25 PM
    Percent of Nominal: 67%
    Duration: 6 cycles (100 milliseconds)
  • Monitoring
    Monitoring

    We implemented a fix and are currently monitoring the result.

  • Investigating
    Investigating

    It looks like most of the NERC's OpenStack VMs are in shut down state. You might need to manually restart them from OpenStack Dashboard: https://stack.nerc.mghpcc.org/dashboard/project/instances/

    We are currently investigating this incident.

OTHER WEBSITES & TOOLS - NERC TICKETING SYSTEM isn't accessible at the moment (automated)
  • Resolved
    Resolved

    OTHER WEBSITES & TOOLS - NERC TICKETING SYSTEM is now operational! This update was created by an automated monitoring service.

  • Investigating
    Investigating

    OTHER WEBSITES & TOOLS - NERC TICKETING SYSTEM cannot be accessed at the moment. This incident was created by an automated monitoring service.

Upcoming NERC system maintenance and update Monday July 24, 2023 9:00 AM – 01:00 PM
  • Completed
    July 24, 2023 at 4:30 PM
    Completed
    July 24, 2023 at 4:30 PM

    Maintenance has completed successfully.

  • In progress
    July 24, 2023 at 1:00 PM
    In progress
    July 24, 2023 at 1:00 PM

    Maintenance is now in progress

  • Planned
    July 24, 2023 at 1:00 PM
    Planned
    July 24, 2023 at 1:00 PM

    NERC’s planned system maintenance and update will occur on Monday July 10, 2023 from 9:00 AM – 01:00 PM.

    GENERAL MAINTENANCE

    • We are conducting follow-up maintenance to the July 10th update. This requires updating the OpenStack nova configuration in order to support the new NVIDIA V100 GPUs. This will enable us to offer additional GPU flavors on the NERC OpenStack cluster.

    NOTICES

    • Any running user VMs/containers/workloads should not be interrupted.

    • This work will involve a brief outage to the NERC OpenStack’s nova scheduler.

    • Users will be able to access their VMs and storage via configured SSH and API settings.

    • The estimated time to complete this update is 4 hours. It can take more or less time, so we urge you to keep an eye on https://nerc.instatus.com/ to get the progress during this time.

    • Please do subscribe to the NERC’s status for any future updates: https://nerc.instatus.com/subscribe/email

    Our priority is to help make science happen, so If you or your research team have any questions or need to escalate an issue, please contact us via email (help@nerc.mghpcc.org).

    Thanks,

    New England Research Cloud (NERC)

    https://nerc.mghpcc.org/

    https://nerc-project.github.io/nerc-docs/

NERC's ColdFront version update to address the latest security fix.
  • Completed
    July 17, 2023 at 1:34 PM
    Completed
    July 17, 2023 at 1:34 PM

    Maintenance has completed successfully.

  • Planned
    July 17, 2023 at 1:15 PM
    Planned
    July 17, 2023 at 1:15 PM

    We are planning for a scheduled maintenance during that time.
    Details:
    We will update ColdFront to the latest released version, which addresses a security vulnerability in the code.

Upcoming NERC system maintenance and update Monday July 10, 2023 9:00 AM – 01:00 PM
  • Completed
    July 10, 2023 at 7:02 PM
    Completed
    July 10, 2023 at 7:02 PM

    Maintenance has completed successfully.

  • Update
    July 10, 2023 at 5:53 PM
    In progress
    July 10, 2023 at 5:53 PM

    Maintenance is now in progress. We are extending this update for another hour.

  • Update
    July 10, 2023 at 5:11 PM
    In progress
    July 10, 2023 at 5:11 PM

    Maintenance is now in progress. We are extending this update for another hour.

  • In progress
    July 10, 2023 at 2:37 PM
    In progress
    July 10, 2023 at 2:37 PM

    Maintenance is now in progress.

  • Planned
    July 10, 2023 at 1:00 PM
    Planned
    July 10, 2023 at 1:00 PM

    NERC’s planned system maintenance and update will occur on Monday July 10, 2023 from 9:00 AM – 01:00 PM.

    GENERAL MAINTENANCE

    • Our plan includes enhancing the NERC’s OpenStack cluster by adding additional NVIDIA V100 GPUs. Additionally, we will be making modifications to the MGHPCC Shared Services Account Portal (aka RegApp) and Keycloak instance in order to allow logins from additional domains. Consequently, during this time, both new user registrations and login to NERC web services may be temporarily impacted.

    NOTICES

    • We will be adding additional NVIDIA V100 GPUs to the NERC’s OpenStack cluster configuration which will enable us to distribute more GPU based flavors.

    • Any running user VMs/containers/workloads should not be interrupted.

    • Users will be able to access their VMs and storage via configured SSH and API settings.

    • The estimated time to complete this update is 4 hours. It can take more or less time, so we urge you to keep an eye on https://nerc.instatus.com/ to get the progress during this time.

    • Please do subscribe to the NERC’s status for any future updates: https://nerc.instatus.com/subscribe/email

    Our priority is to help make science happen, so If you or your research team have any questions or need to escalate an issue, please contact us via email (help@nerc.mghpcc.org).

    Thanks,

    New England Research Cloud (NERC)

    https://nerc.mghpcc.org/

    https://nerc-project.github.io/nerc-docs/

Jun 2023

We are facing issues with approved resource allocation in both NERC OpenStack and OpenShift based resources.
  • Resolved
    Resolved

    This incident has been resolved.

    **NOTE: ** If you are experiencing ongoing difficulties with your previously approved resource allocations, please reach out to us via email at help@nerc.mghpcc.org so that we can promptly assist you in resolving the issues.

  • Monitoring
    Monitoring

    We implemented a fix and are currently monitoring the result.

  • Investigating
    Investigating

    We are currently investigating this incident.

MGHPCC SHARED SERVICES (MGHPCC-SS) ACCOUNT PORTAL isn't accessible at the moment (automated)
  • Resolved
    Resolved

    MGHPCC SHARED SERVICES (MGHPCC-SS) ACCOUNT PORTAL is now operational! This update was created by an automated monitoring service.

  • Investigating
    Investigating

    MGHPCC SHARED SERVICES (MGHPCC-SS) ACCOUNT PORTAL cannot be accessed at the moment. This incident was created by an automated monitoring service.

MGHPCC SHARED SERVICES (MGHPCC-SS) ACCOUNT PORTAL isn't accessible at the moment (automated)
  • Resolved
    Resolved

    MGHPCC SHARED SERVICES (MGHPCC-SS) ACCOUNT PORTAL is now operational! This update was created by an automated monitoring service.

  • Investigating
    Investigating

    MGHPCC SHARED SERVICES (MGHPCC-SS) ACCOUNT PORTAL cannot be accessed at the moment. This incident was created by an automated monitoring service.

MGHPCC SHARED SERVICES (MGHPCC-SS) ACCOUNT PORTAL isn't accessible at the moment (automated)
  • Resolved
    Resolved

    MGHPCC SHARED SERVICES (MGHPCC-SS) ACCOUNT PORTAL is now operational! This update was created by an automated monitoring service.

  • Investigating
    Investigating

    MGHPCC SHARED SERVICES (MGHPCC-SS) ACCOUNT PORTAL cannot be accessed at the moment. This incident was created by an automated monitoring service.

NERC OPENSHIFT CONTAINER PLATFORM - NERC OPENSHIFT WEB CONSOLE isn't accessible at the moment (automated)
  • Resolved
    Resolved

    NERC OPENSHIFT CONTAINER PLATFORM - NERC OPENSHIFT WEB CONSOLE is now operational! This update was created by an automated monitoring service.

  • Investigating
    Investigating

    NERC OPENSHIFT CONTAINER PLATFORM - NERC OPENSHIFT WEB CONSOLE cannot be accessed at the moment. This incident was created by an automated monitoring service.

May 2023

NERC’s ColdFront and OpenShift container platform (OCP) maintenance [May 30, 2023 9:00 AM - 1:15 PM]
  • Completed
    May 30, 2023 at 5:11 PM
    Completed
    May 30, 2023 at 5:11 PM

    Maintenance has completed successfully.

  • Update
    May 30, 2023 at 5:00 PM
    In progress
    May 30, 2023 at 5:00 PM

    Apologies for the inconvenience caused. We need to address a minor issue, so we will be extending the maintenance window by an additional 15 minutes.

  • Update
    May 30, 2023 at 4:31 PM
    In progress
    May 30, 2023 at 4:31 PM

    Apologies for the inconvenience caused. We need to address a minor issue, so we will be extending the maintenance window by an additional 30 minutes.

  • Update
    May 30, 2023 at 4:00 PM
    In progress
    May 30, 2023 at 4:00 PM

    Apologies for the inconvenience caused. We need to address a minor issue, so we will be extending the maintenance window by an additional 30 minutes.

  • Update
    May 30, 2023 at 2:59 PM
    In progress
    May 30, 2023 at 2:59 PM

    Apologies for the inconvenience caused. We need to address a minor issue, so we will be extending the maintenance window by an additional 1 hour.

  • In progress
    May 30, 2023 at 1:00 PM
    In progress
    May 30, 2023 at 1:00 PM

    Maintenance is now in progress

  • Update
    May 30, 2023 at 1:00 PM
    Planned
    May 30, 2023 at 1:00 PM
    • NERC’s ColdFront service will be temporarily down for a minor update.
  • Planned
    May 30, 2023 at 1:00 PM
    Planned
    May 30, 2023 at 1:00 PM

    NERC's planned OpenShift container platform (OCP) maintenance and update will occur on Tuesday May 30, 2023 from 9:00 AM - 11:00 AM.

    GENERAL MAINTENANCE

    • We will be updating the NERC’s OpenShift container platform cluster configuration. The core OpenShift services should not be interrupted during this time, however, all hosts will be rebooted as a consequence. Any user workloads that are not deployed in a highly-available fashion (e.g. via a deployment spanning multiple pods/hosts) will be temporarily interrupted.

    NOTICES

    • We will be making a change to the NERC's OpenShift container platform cluster configuration which will involve rolling reboots of all cluster hosts.

    • Any user workloads that are not deployed in a highly-available fashion will be temporarily interrupted.

    • We do not expect any core OpenShift services to go down during this time but please keep an eye on the status page in the case of any unintended side-effects.

    • The estimated time to complete this update is 2 hours. It can take more or less time, so we urge you to keep an eye on https://nerc.instatus.com/ to get the progress during this time.

    • Please do subscribe to the NERC’s status for any future updates: https://nerc.instatus.com/subscribe/email

    Our priority is to help make science happen, so If you or your research team have any questions or need to escalate an issue, please contact us via email (help@nerc.mghpcc.org).

    Thanks,

    New England Research Cloud (NERC)

    https://nerc.mghpcc.org/

    https://nerc-project.github.io/nerc-docs/

NERC system maintenance Monday May 15, 2023 9:00 AM - 11:30 AM
  • Completed
    May 15, 2023 at 3:30 PM
    Completed
    May 15, 2023 at 3:30 PM

    Maintenance has completed successfully

  • Update
    May 15, 2023 at 2:58 PM
    In progress
    May 15, 2023 at 2:58 PM

    Apologies for the inconvenience caused. We need to address a minor issue, so we will be extending the maintenance window by an additional 30 minutes.

  • In progress
    May 15, 2023 at 1:00 PM
    In progress
    May 15, 2023 at 1:00 PM

    Maintenance is now in progress

  • Planned
    May 15, 2023 at 1:00 PM
    Planned
    May 15, 2023 at 1:00 PM

    NERC planned system maintenance will occur on Monday May 15, 2023 from 9:00 AM - 11:00 AM.

    GENERAL MAINTENANCE

    • We will be updating the NERC’s ColdFront services. Access to the NERC’s API services will be briefly disrupted during this time.

    NOTICES

    • We are upgrading and updating NERC's ColdFront deployment.

    • We will resolve the problem with the automatic approval of users' Rados Gateway Object Storage/Swift quota.

    • After the upgrade, PIs and Project Manager(s) will be able to request OpenShift resources in addition to the existing OpenStack resources.

    • Another issue we will address is the maximum length of project titles on NERC's ColdFront, which previously caused unsuccessful allocations on both NERC’s OpenStack and OpenShift platforms.

    • The estimated time to complete this upgrade is 2 hours. It can take less, so we urge you to keep an eye on https://nerc.instatus.com/ to get the progress during this time.

    • Please do subscribe to the NERC’s status for any future updates: https://nerc.instatus.com/subscribe/email

    Our priority is to help make science happen, so If you or your research team have any questions or need to escalate an issue, please contact us via email (help@nese.mghpcc.org).

    Thanks,

    New England Research Cloud (NERC)

    https://nerc.mghpcc.org/

    https://nerc-project.github.io/nerc-docs/

May 2023 to Jul 2023

Next

For issues not shown here, please contact NERC via email help@nerc.mghpcc.org or, using the NERC's Support Ticketing System.