NERC - Notice history

All systems operational

GETTING HELP
Email: help@nerc.mghpcc.org or, using the NERC's Support Ticketing System
NERC Documentation: https://nerc-project.github.io/nerc-docs/
Status page for the New England Research Cloud (NERC) and other resources.
Please scroll down to see details on any Incidents or maintenance notices.

MGHPCC SHARED SERVICES (MGHPCC-SS) ACCOUNT PORTAL - Operational

100% - uptime
Jul 2023 · 100.0%Aug · 100.0%Sep · 99.84%

Previous page

Next page

Jul 2023
Aug 2023
Sep 2023

NERC COLDFRONT - Operational

100% - uptime
Jul 2023 · 100.0%Aug · 100.0%Sep · 99.83%

Previous page

Next page

Jul 2023
Aug 2023
Sep 2023

NETWORKING - Operational

100% - uptime
Jul 2023 · 100.0%Aug · 100.0%Sep · 100.0%

Previous page

Next page

Jul 2023
Aug 2023
Sep 2023

STORAGE - Operational

100% - uptime
Jul 2023 · 100.0%Aug · 100.0%Sep · 100.0%

Previous page

Next page

Jul 2023
Aug 2023
Sep 2023
100% - uptime

NERC WEBSITE - Operational

100% - uptime
Jul 2023 · 100.0%Aug · 100.0%Sep · 100.0%

Previous page

Next page

Jul 2023
Aug 2023
Sep 2023

NERC DOCUMENTATION WEBSITE - Operational

100% - uptime
Jul 2023 · 100.0%Aug · 100.0%Sep · 100.0%

Previous page

Next page

Jul 2023
Aug 2023
Sep 2023

NERC TICKETING SYSTEM - Operational

100% - uptime
Jul 2023 · 99.91%Aug · 100.0%Sep · 100.0%

Previous page

Next page

Jul 2023
Aug 2023
Sep 2023

Notice history

Sep 2023

Upcoming Red Hat's OpenShift container platform (OCP) Version Upgrade on NERC will occur on Wednesday September 27, 2023 from 8:00 AM – 5:00 PM.
  • Completed
    September 28, 2023 at 1:02 AM
    Completed
    September 28, 2023 at 1:02 AM

    Maintenance has completed successfully.

  • Update
    September 27, 2023 at 8:17 PM
    Update
    September 27, 2023 at 8:17 PM

    Maintenance requires more incremental upgrade of OpenShift Versions so we are extending this till midnight.

  • In progress
    September 27, 2023 at 12:00 PM
    In progress
    September 27, 2023 at 12:00 PM

    Maintenance is now in progress

  • Planned
    September 27, 2023 at 12:00 PM
    Planned
    September 27, 2023 at 12:00 PM

    Upcoming Red Hat’s OpenShift container platform (OCP) upgrade on NERC from version 4.10 to 4.13 will occur on Wednesday September 27, 2023 from 8:00 AM – 5:00 PM.

    GENERAL MAINTENANCE

    • We are writing to inform you that we are planning to upgrade our OpenShift cluster from version 4.10 to 4.13 on Wednesday, September 27, 2023. As our current version of OpenShift has already ended Red Hat’s maintenance support on September 10, 2023, this upgrade is crucial and will bring several new features and improvements to our cluster. Also, we will enable GPU resources to our OpenShift and Red Hat OpenShift Data Science Platform (RHODS) that will provide a fully supported sandbox environment for data scientists/researchers to develop, train and test AI/ML models and deploy them for use in intelligent applications.

    NOTICES

    • Please keep a backup for your critical data or application running on your project on NERC’s OpenShift cluster.

    • The new upgrade to the latest OpenShift version will provide improved performance, enhanced security, additional operators and enhanced user interface along with updated Kubernetes version.

    • The estimated time to complete this update is 4 hours. It can take more or less time, so we urge you to keep an eye on https://nerc.instatus.com/ to get the progress during this time.

    • During the upgrade, there will be a period of downtime to ensure a seamless transition. We anticipate this downtime to last approximately 1 day, and we apologize for any inconvenience this may cause. It can take more or less time, so we urge you to keep an eye on https://nerc.instatus.com/ and also subscribe to https://nerc.instatus.com/subscribe/email to get the progress updates during this time.

    • If you have not used or tested, our state-of-the-art OpenShift and RHODS platform that enables you and your team to deploy containerized applications in a cloud-native environment, providing a reliable, isolated, and scalable solution for your complex research computing and teaching needs. Please get started using these platforms by requesting a new resource allocation to your project using NERC’s ColdFront web console or you can get in touch with us to have a quick demo.

    More information about NERC is available on NERC’s website (https://nerc.mghpcc.org/). If you have any questions, please don’t hesitate to reach out to us via email (help@nerc.mghpcc.org) or, by submitting a new ticket at the NERC's Support Ticketing System (osTicket).

    Thanks,

    New England Research Cloud (NERC)

    https://nerc.mghpcc.org/

    https://nerc-project.github.io/nerc-docs/

Upcoming NERC OpenStack Maintenance - Additional GPU hosts and RadosGW improvements Monday September 18, 2023 9:00 AM -1:00 PM
  • Completed
    September 19, 2023 at 3:15 PM
    Completed
    September 19, 2023 at 3:15 PM

    Maintenance has completed successfully.

  • Update
    September 18, 2023 at 5:02 PM
    Update
    September 18, 2023 at 5:02 PM

    Maintenance is still in progress. We are extending this maintenace for another couple hours and will update as this progress …

  • In progress
    September 18, 2023 at 1:00 PM
    In progress
    September 18, 2023 at 1:00 PM

    Maintenance is now in progress

  • Planned
    September 18, 2023 at 1:00 PM
    Planned
    September 18, 2023 at 1:00 PM

    NERC’s planned OpenStack Maintenance - Additional GPU hosts and RadosGW improvements will occur on Monday September 18, 2023 9:00 AM -1:00 PM.

    GENERAL MAINTENANCE

    • The NERC will be adding additional GPU nodes (K80) to the OpenStack deployment. In addition, we will also be performing maintenance on the RadosGW (Swift/S3) object storage service.

    NOTICES

    • During this time the RadosGW object storage service will be unavailable.

    • Please be aware that during this maintenance, project and project change requests will not be accepted and approved.

    • The estimated time to complete this update is 4 hours. It can take more or less time, so we urge you to keep an eye on https://nerc.instatus.com/ to get the progress during this time.

    • Please do subscribe to the NERC’s status for any future updates: https://nerc.instatus.com/subscribe/email

    Our priority is to help make science happen, so If you or your research team have any questions or need to escalate an issue, please don't hesitate to reach out to us via email (help@nerc.mghpcc.org) or, by submitting a new ticket at the NERC's Support Ticketing System (osTicket).

    Thanks,

    New England Research Cloud (NERC)

    https://nerc.mghpcc.org/

    https://nerc-project.github.io/nerc-docs/

MGHPCC SHARED SERVICES (MGHPCC-SS) ACCOUNT PORTAL isn't accessible at the moment (automated)
  • Resolved
    Resolved

    MGHPCC SHARED SERVICES (MGHPCC-SS) ACCOUNT PORTAL cannot be accessed at the moment. This incident was created by an automated monitoring service.

NERC COLDFRONT isn't accessible at the moment (automated)
  • Resolved
    Resolved

    NERC COLDFRONT cannot be accessed at the moment. This incident was created by an automated monitoring service.

MGHPCC SHARED SERVICES (MGHPCC-SS) ACCOUNT PORTAL isn't accessible at the moment (automated)
  • Resolved
    Resolved

    MGHPCC SHARED SERVICES (MGHPCC-SS) ACCOUNT PORTAL is now operational! This update was created by an automated monitoring service.

  • Investigating
    Investigating

    MGHPCC SHARED SERVICES (MGHPCC-SS) ACCOUNT PORTAL cannot be accessed at the moment. This incident was created by an automated monitoring service.

Aug 2023

NERC OpenShift Container Platform (OCP) Maintenance [August 21, 2023 9:00 AM - 5:00 PM]
  • Completed
    August 21, 2023 at 4:53 PM
    Completed
    August 21, 2023 at 4:53 PM

    Maintenance has completed successfully.

  • In progress
    August 21, 2023 at 1:00 PM
    In progress
    August 21, 2023 at 1:00 PM

    Maintenance is now in progress

  • Planned
    August 21, 2023 at 1:00 PM
    Planned
    August 21, 2023 at 1:00 PM

    NERC’s planned OpenShift container platform (OCP) maintenance will occur on Monday August 21, 2023 from 9:00 AM – 5:00 PM.

    GENERAL MAINTENANCE

    • We will be moving the OpenShift container platform worker nodes to a new location within the datacenter. The core OpenShift services will be interrupted during this time. Any critical workloads that are deployed in the cluster need to be stopped until the maintenance is complete.

    NOTICES

    • We will be powering off all OpenShift cluster hosts prior to the move.

    • The NERC OpenShift cluster will be unavailable during this time. Please let us know If you encounter any issues after the maintenance has completed.

    • The estimated time to complete this update is 8 hours. It can take more or less time, so we urge you to keep an eye on https://nerc.instatus.com/ to get the progress during this time.

    • Please do subscribe to the NERC’s status for any future updates: https://nerc.instatus.com/subscribe/email

    Our priority is to help make science happen, so If you or your research team have any questions or need to escalate an issue, please contact us via email (help@nerc.mghpcc.org).

    Thanks,

    New England Research Cloud (NERC)

    https://nerc.mghpcc.org/

    https://nerc-project.github.io/nerc-docs/

Jul 2023

Unexpected power shut down of all NERC OpenStack VMs
  • Resolved
    Resolved

    We now know that this is due to the MGHPCC power sag event and now has been resolved.

    The MGHPCC main meter detected an ITIC Level 2 power sag event at 3:59PM on 
Thursday July 27, during the wave of thunderstorms that passed through the area.   
The event affected a single power phase.  

Details
    Date:  Thursday July 27, 2023
    Time: 15:59:25 PM
    Percent of Nominal: 67%
    Duration: 6 cycles (100 milliseconds)
  • Monitoring
    Monitoring

    We implemented a fix and are currently monitoring the result.

  • Investigating
    Investigating

    It looks like most of the NERC's OpenStack VMs are in shut down state. You might need to manually restart them from OpenStack Dashboard: https://stack.nerc.mghpcc.org/dashboard/project/instances/

    We are currently investigating this incident.

OTHER WEBSITES & TOOLS - NERC TICKETING SYSTEM isn't accessible at the moment (automated)
  • Resolved
    Resolved

    OTHER WEBSITES & TOOLS - NERC TICKETING SYSTEM is now operational! This update was created by an automated monitoring service.

  • Investigating
    Investigating

    OTHER WEBSITES & TOOLS - NERC TICKETING SYSTEM cannot be accessed at the moment. This incident was created by an automated monitoring service.

Upcoming NERC system maintenance and update Monday July 24, 2023 9:00 AM – 01:00 PM
  • Completed
    July 24, 2023 at 4:30 PM
    Completed
    July 24, 2023 at 4:30 PM

    Maintenance has completed successfully.

  • In progress
    July 24, 2023 at 1:00 PM
    In progress
    July 24, 2023 at 1:00 PM

    Maintenance is now in progress

  • Planned
    July 24, 2023 at 1:00 PM
    Planned
    July 24, 2023 at 1:00 PM

    NERC’s planned system maintenance and update will occur on Monday July 10, 2023 from 9:00 AM – 01:00 PM.

    GENERAL MAINTENANCE

    • We are conducting follow-up maintenance to the July 10th update. This requires updating the OpenStack nova configuration in order to support the new NVIDIA V100 GPUs. This will enable us to offer additional GPU flavors on the NERC OpenStack cluster.

    NOTICES

    • Any running user VMs/containers/workloads should not be interrupted.

    • This work will involve a brief outage to the NERC OpenStack’s nova scheduler.

    • Users will be able to access their VMs and storage via configured SSH and API settings.

    • The estimated time to complete this update is 4 hours. It can take more or less time, so we urge you to keep an eye on https://nerc.instatus.com/ to get the progress during this time.

    • Please do subscribe to the NERC’s status for any future updates: https://nerc.instatus.com/subscribe/email

    Our priority is to help make science happen, so If you or your research team have any questions or need to escalate an issue, please contact us via email (help@nerc.mghpcc.org).

    Thanks,

    New England Research Cloud (NERC)

    https://nerc.mghpcc.org/

    https://nerc-project.github.io/nerc-docs/

NERC's ColdFront version update to address the latest security fix.
  • Completed
    July 17, 2023 at 1:34 PM
    Completed
    July 17, 2023 at 1:34 PM

    Maintenance has completed successfully.

  • Planned
    July 17, 2023 at 1:15 PM
    Planned
    July 17, 2023 at 1:15 PM

    We are planning for a scheduled maintenance during that time.
    Details:
    We will update ColdFront to the latest released version, which addresses a security vulnerability in the code.

Upcoming NERC system maintenance and update Monday July 10, 2023 9:00 AM – 01:00 PM
  • Completed
    July 10, 2023 at 7:02 PM
    Completed
    July 10, 2023 at 7:02 PM

    Maintenance has completed successfully.

  • Update
    July 10, 2023 at 5:53 PM
    Update
    July 10, 2023 at 5:53 PM

    Maintenance is now in progress. We are extending this update for another hour.

  • Update
    July 10, 2023 at 5:11 PM
    Update
    July 10, 2023 at 5:11 PM

    Maintenance is now in progress. We are extending this update for another hour.

  • In progress
    July 10, 2023 at 2:37 PM
    In progress
    July 10, 2023 at 2:37 PM

    Maintenance is now in progress.

  • Planned
    July 10, 2023 at 1:00 PM
    Planned
    July 10, 2023 at 1:00 PM

    NERC’s planned system maintenance and update will occur on Monday July 10, 2023 from 9:00 AM – 01:00 PM.

    GENERAL MAINTENANCE

    • Our plan includes enhancing the NERC’s OpenStack cluster by adding additional NVIDIA V100 GPUs. Additionally, we will be making modifications to the MGHPCC Shared Services Account Portal (aka RegApp) and Keycloak instance in order to allow logins from additional domains. Consequently, during this time, both new user registrations and login to NERC web services may be temporarily impacted.

    NOTICES

    • We will be adding additional NVIDIA V100 GPUs to the NERC’s OpenStack cluster configuration which will enable us to distribute more GPU based flavors.

    • Any running user VMs/containers/workloads should not be interrupted.

    • Users will be able to access their VMs and storage via configured SSH and API settings.

    • The estimated time to complete this update is 4 hours. It can take more or less time, so we urge you to keep an eye on https://nerc.instatus.com/ to get the progress during this time.

    • Please do subscribe to the NERC’s status for any future updates: https://nerc.instatus.com/subscribe/email

    Our priority is to help make science happen, so If you or your research team have any questions or need to escalate an issue, please contact us via email (help@nerc.mghpcc.org).

    Thanks,

    New England Research Cloud (NERC)

    https://nerc.mghpcc.org/

    https://nerc-project.github.io/nerc-docs/

Jul 2023 to Sep 2023

Next

For issues not shown here, please contact NERC via email help@nerc.mghpcc.org or, using the NERC's Support Ticketing System.