Duration: 6 days, 4 hours and 35 minutes

Affected Components: OPENSHIFT WORKERS NODES, NERC RED HAT OPENSHIFT AI (RHOAI)

Jun 29, 21:27:50 GMT+0
Resolved - This incident has been resolved. H100's are available now..

Jun 23, 16:53:16 GMT+0
Investigating - The **H100 GPUs** are currently unavailable for use while a required firmware update is being completed. We will provide an update once the firmware process has finished and the GPUs have been validated for production use. Thank you for your patience..

Duration: 7 days and 8 hours

Affected Components: OPENSTACK IMAGE SERVICE (GLANCE), NERC ACADEMIC (EDU) OPENSHIFT CLUSTER, STORAGE, NERC OPENSHIFT WEB CONSOLE, NERC COLDFRONT, OPENSTACK DASHBOARD (HORIZON), OPENSTACK OBJECT STORE (SWIFT), OPENSHIFT CONTROLLER, OPENSHIFT WORKERS NODES, MGHPCC SHARED SERVICES (MGHPCC-SS) ACCOUNT PORTAL, OPENSTACK NETWORKING SERVICE (NEUTRON), NERC OPENSTACK COMPUTE SERVICE (NOVA) , NERC RED HAT OPENSHIFT AI (RHOAI), NETWORKING, OPENSTACK BLOCK STORAGE (CINDER), OPENSTACK EC2 API PROXY (EC2API)

Jun 22, 21:00:00 GMT+0
Completed - Maintenance has completed successfully.

Jun 15, 13:00:01 GMT+0
Identified - Maintenance is now in progress.

Jun 15, 13:00:00 GMT+0
Identified - Due to the annual MGHPCC (Holyoke data center) power maintenance as well as other maintenance activities, the New England Research Cloud (NERC) services will be unavailable from **Monday, June 15th, 9 AM until Monday, June 22nd, 5 PM.** NERC services that will be impacted include * NERC's OpenStack * NERC's OpenShift, which includes the Red Hat OpenShift AI (RHOAI) platform on the Production and Academic (EDU) Clusters * NERC supporting shared services ColdFront, RegApp, ESI, and Keycloak We likely will not require the full planned outage window. However, since we are performing NERC hardware and network maintenance and upgrades, we are planning for the worst-case scenario while working to restore services as quickly as possible. Our goal is to resume full services by **the end of Monday, June 22nd, at 5 PM** or **earlier**. Please refer to the timeline below for details:  **NOTICES** * This outage affects all services and resources in the MGHPCC/Holyoke data center, including NERC's core and shared services, as well as the underlying compute, storage, and networking. * Please keep a backup of your critical data or application running on your project on NERC’s OpenStack and OpenShift cluster. * The estimated time for power-up and return to full service by **the end of Monday, June 22nd, at 5 PM** or **earlier**. We encourage you to monitor the progress at <https://nerc.instatus.com/> during this time. * Please subscribe to the NERC's status for any future updates as they happen: <https://nerc.instatus.com/subscribe/email> Our priority is helping make research happen. If you or your research team have any questions or need to escalate an issue, please do not hesitate to contact us via email ([help@nerc.mghpcc.org](mailto:help@nerc.mghpcc.org)) or by submitting a service request through [NERC's Support Ticketing System](https://mghpcc.supportsystem.com/open.php). Thank you, New England Research Cloud (NERC) [https://nerc.mghpcc.org/ ](https://nerc.mghpcc.org/)<https://nerc.instatus.com/>.

Duration: 10 hours

Affected Components: NERC COLDFRONT, NERC RED HAT OPENSHIFT AI (RHOAI), OPENSHIFT CONTROLLER, NERC OPENSHIFT WEB CONSOLE, OPENSHIFT WORKERS NODES

May 12, 22:00:00 GMT+0
Completed - Maintenance has completed successfully.

May 12, 20:22:44 GMT+0
Identified - Maintenance is now in progress. We need to extend the timeline for one more hour as we fixing issues with worker nodes..

May 12, 12:00:01 GMT+0
Identified - Maintenance is now in progress.

May 12, 12:00:00 GMT+0
Identified - **Upcoming NERC system maintenance and upgrade will occur on Tuesday, May 12, 2026, 8:00 AM - 5:00 PM.** **GENERAL MAINTENANCE** \- We will be upgrading [**Red Hat OpenShift AI (RHOAI)**](https://nerc-project.github.io/nerc-docs/openshift-ai/) on the NERC OpenShift Production Cluster to version 3.3\. This upgrade will provide [new features](https://docs.redhat.com/en/documentation/red%5Fhat%5Fopenshift%5Fai%5Fself-managed/3.3/html/release%5Fnotes/new-features-and-enhancements%5Frelnotes?ref=myopenshiftblog.com#new%5Ffeatures), bug fixes, and improved support for AI/ML workflows. \- We will be continue our last issue to fix the support for **Remote Direct Memory Access (RDMA)** to our production cluster. \- We are upgrading our [ColdFront](https://coldfront.mss.mghpcc.org/) deployment. **NOTICES** \- The Production Cluster’s OpenShift, OpenShift AI and ColdFront services will be unavailable during this maintenance. \- Please keep a backup of your critical data or application running on your project on NERC’s OpenShift cluster. \- Project allocation requests and change requests for both OpenStack and OpenShift Resources will not be accepted or approved during this time. \- The estimated time to complete this maintenance is **9 hours**. It can take more or less time, so we urge you to keep an eye on <https://nerc.instatus.com/> for real-time updates on progress. \- Please subscribe to <https://nerc.instatus.com/subscribe/email> in order to get the progress updates during this time. Our priority is to help make science happen, so If you or your research team have any questions or need to escalate an issue, please contact us via email ([help@nerc.mghpcc.org](mailto:help@nerc.mghpcc.org)) or, by submitting a new ticket at [the NERC's Support Ticketing System](https://mghpcc.supportsystem.com/open.php)..

May 12, 12:00:00 GMT+0
Identified - We will not be able to perform the [**Red Hat OpenShift AI (RHOAI)**](https://nerc-project.github.io/nerc-docs/openshift-ai/) upgrade on the NERC OpenShift Production Cluster at this time. The upgrade will be scheduled and performed at a later date. However, the other planned upgrades will continue as scheduled, which should make the maintenance window shorter than anticipated..

Duration: 10 days, 5 hours and 29 minutes

Affected Components: MGHPCC SHARED SERVICES (MGHPCC-SS) ACCOUNT PORTAL, OPENSTACK NETWORKING SERVICE (NEUTRON), OPENSTACK DASHBOARD (HORIZON), NERC COLDFRONT, OPENSTACK BLOCK STORAGE (CINDER), OPENSTACK IMAGE SERVICE (GLANCE), NERC ACADEMIC (EDU) OPENSHIFT CLUSTER, OPENSTACK OBJECT STORE (SWIFT), NERC OPENSTACK COMPUTE SERVICE (NOVA) , OPENSTACK EC2 API PROXY (EC2API)

Apr 29, 18:28:45 GMT+0
Resolved - This incident has been resolved..

Apr 21, 16:09:56 GMT+0
Monitoring - We implemented a fix and are currently monitoring the result..

Apr 20, 16:56:17 GMT+0
Investigating - Front-end shared services - most critically **Keycloak, ColdFront,** and **RegApp** \- are currently unreachable. We are continuing to work on resolving this incident. You can continue to use the **NERC OCP (OpenShift) Production** and **NERC EDU (Academic)** clusters as normal to deploy your workloads..

Apr 20, 00:20:09 GMT+0
Identified - We are continuing to work on a fix for this incident..

Apr 19, 13:00:00 GMT+0
Investigating - We are currently investigating this incident..

Duration: 3 hours

Affected Components: NERC RED HAT OPENSHIFT AI (RHOAI), OPENSHIFT CONTROLLER, NERC OPENSHIFT WEB CONSOLE, OPENSHIFT WORKERS NODES

Apr 15, 16:00:00 GMT+0
Completed - Maintenance has completed successfully.

Apr 15, 13:00:01 GMT+0
Identified - Maintenance is now in progress.

Apr 15, 13:00:00 GMT+0
Identified - We are planning for a scheduled maintenance during that time to add remaining H100s with RDMA support..

Duration: 9 hours

Affected Components: NERC COLDFRONT, NERC RED HAT OPENSHIFT AI (RHOAI), OPENSHIFT CONTROLLER, NERC OPENSHIFT WEB CONSOLE, OPENSHIFT WORKERS NODES

Apr 14, 21:01:18 GMT+0
Identified - Maintenance is now in progress. We are adding 1 more hour to complete and test the new upgrade..

Apr 14, 21:00:00 GMT+0
Completed - Maintenance has completed successfully.

Apr 14, 21:00:00 GMT+0
Completed - Maintenance has completed successfully.

Apr 14, 12:00:01 GMT+0
Identified - Maintenance is now in progress.

Apr 14, 12:00:00 GMT+0
Identified - **Upcoming NERC system maintenance and upgrade will occur on Tuesday, April 14, 2026, 8:00 AM - 5:00 PM.** **GENERAL MAINTENANCE** \- We will be upgrading the [**Openshift Container Platform (OCP)**](https://nerc-project.github.io/nerc-docs/openshift/) on the **NERC OpenShift Production Cluster** to version **4.21**. \- We will be adding support for **Remote Direct Memory Access (RDMA)** to our production cluster. \- We are upgrading our [ColdFront](https://coldfront.mss.mghpcc.org/) deployment. **NOTICES** \- The Production Cluster’s OpenShift, OpenShift AI and ColdFront services will be unavailable during this maintenance. \- Please keep a backup of your critical data or application running on your project on NERC’s OpenShift cluster. \- Project allocation requests and change requests for both OpenStack and OpenShift Resources will not be accepted or approved during this time. \- The estimated time to complete this maintenance is **9 hours**. It can take more or less time, so we urge you to keep an eye on <https://nerc.instatus.com/> for real-time updates on progress. \- Please subscribe to <https://nerc.instatus.com/subscribe/email> in order to get the progress updates during this time. Our priority is to help make science happen, so If you or your research team have any questions or need to escalate an issue, please contact us via email ([help@nerc.mghpcc.org](mailto:help@nerc.mghpcc.org)) or, by submitting a new ticket at [the NERC's Support Ticketing System](https://mghpcc.supportsystem.com/open.php)..

Duration: 1 day and 17 hours

Mar 20, 21:00:28 GMT+0
Resolved - This incident has been resolved..

Mar 20, 18:27:22 GMT+0
Identified - We are continuing to work on a fix for this incident..

Mar 19, 04:00:00 GMT+0
Investigating - We have identified an issue with the "**Cumulative Daily Charges for Month**" value shown in daily cost notifications for OpenStack allocations. Since the March ColdFront maintenance on **03/19**, some OpenStack storage charges have not been fully included in this field, causing reported cumulative charges to be lower than actual costs. As a result, some users may have already exceeded their cost alert thresholds without being notified. Until this is fixed, please do not rely solely on the “**Cumulative Daily Charges for Month**” value and instead **review** your approved **storage allocation quota** on the NERC's ColdFront..

Duration: 10 hours

Affected Components: NERC RED HAT OPENSHIFT AI (RHOAI), OPENSHIFT CONTROLLER, NERC COLDFRONT, OPENSHIFT WORKERS NODES, NERC OPENSHIFT WEB CONSOLE, , NERC OPENSHIFT CONTAINER PLATFORM - PRODUCTION →

Mar 10, 22:00:00 GMT+0
Completed - Maintenance has completed successfully.

Mar 10, 20:52:07 GMT+0
Identified - Maintenance is now in progress. We will extend this for another one hour..

Mar 10, 12:00:01 GMT+0
Identified - Maintenance is now in progress.

Mar 10, 12:00:00 GMT+0
Identified - **Upcoming NERC system maintenance and upgrade will occur on Tuesday, March 10, 2026, 8:00 AM - 5:00 PM.** **GENERAL MAINTENANCE** \- We will be upgrading the [Openshift Container Platform (OCP)](https://nerc-project.github.io/nerc-docs/openshift/) on the **NERC OpenShift Production Cluster** to version **4.20**. \- We are upgrading our [ColdFront](https://coldfront.mss.mghpcc.org/) deployment to support quota management across different storage types. \- We will be adding additional [**H100 to NERC OpenShift Production Cluster**](https://nerc.mghpcc.org/nvidia-h100-gpus-on-nerc/) for general use. **NOTICES** \- The Production Cluster’s OpenShift, OpenShift AI and ColdFront services will be unavailable during this maintenance. \- Please keep a backup of your critical data or application running on your project on NERC’s OpenShift cluster. \- Project allocation requests and change requests for both OpenStack and OpenShift Resources will not be accepted or approved during this time. \- The estimated time to complete this maintenance is **9 hours**. It can take more or less time, so we urge you to keep an eye on <https://nerc.instatus.com/> for real-time updates on progress. \- Please subscribe to <https://nerc.instatus.com/subscribe/email> in order to get the progress updates during this time. Our priority is to help make science happen, so If you or your research team have any questions or need to escalate an issue, please contact us via email ([help@nerc.mghpcc.org](mailto:help@nerc.mghpcc.org)) or, by submitting a new ticket at [the NERC's Support Ticketing System](https://mghpcc.supportsystem.com/open.php)..

Duration: 2 hours and 35 minutes

Affected Components: OPENSTACK EC2 API PROXY (EC2API), , OPENSTACK DASHBOARD (HORIZON), NERC COLDFRONT, NERC OPENSHIFT WEB CONSOLE, , NERC OPENSTACK → NERC OPENSHIFT CONTAINER PLATFORM - PRODUCTION →

Mar 5, 17:36:47 GMT+0
Resolved - This incident has been resolved..

Mar 5, 17:18:07 GMT+0
Monitoring - We implemented a fix and are currently monitoring the result..

Mar 5, 16:47:15 GMT+0
Investigating - We are currently investigating this incident. This is an isolated issue that is affecting Harvard users only!.

Mar 5, 15:02:09 GMT+0
Investigating - We are currently investigating this incident..

Duration: 9 hours

Affected Components: OPENSHIFT WORKERS NODES, NERC RED HAT OPENSHIFT AI (RHOAI), OPENSHIFT CONTROLLER, NERC COLDFRONT, NERC OPENSHIFT WEB CONSOLE, , NERC OPENSHIFT CONTAINER PLATFORM - PRODUCTION →

Feb 10, 22:00:00 GMT+0
Completed - Maintenance has completed successfully.

Feb 10, 15:01:00 GMT+0
Identified - We are performing **only** the upgrade of [**Red Hat OpenShift AI (RHOAI)**](https://nerc-project.github.io/nerc-docs/openshift-ai/) on the NERC OpenShift Production (OCP-Prod) Cluster today. The upgrades for the [**Openshift Container Platform (OCP)**](https://nerc-project.github.io/nerc-docs/openshift/) and [**ColdFront**](https://coldfront.mss.mghpcc.org/) have been postponed to the next maintenance window..

Feb 10, 14:46:59 GMT+0
Identified - Maintenance is now in progress. Please note that the planned upgrade to our ColdFront deployment will not be performed today and will be rescheduled for a later date..

Feb 10, 13:00:01 GMT+0
Identified - Maintenance is now in progress..

Feb 10, 13:00:00 GMT+0
Identified - **Upcoming NERC system maintenance and upgrade will occur on Tuesday, February 10, 2026 8:00 AM - 5:00 PM.** **GENERAL MAINTENANCE** \- We will be upgrading the [Openshift Container Platform (OCP)](https://nerc-project.github.io/nerc-docs/openshift/) on the NERC OpenShift Production Cluster to version **4.20**. \- We will be upgrading [Red Hat OpenShift AI (RHOAI)](https://nerc-project.github.io/nerc-docs/openshift-ai/) on the NERC OpenShift Production Cluster to version **2.25**. This upgrade will provide new features, bug fixes, and improved support for AI/ML workflows. \- We are upgrading our [ColdFront](https://coldfront.mss.mghpcc.org/) deployment to support quota management across different storage types. **NOTICES** \- The Production Cluster’s OpenShift, OpenShift AI and ColdFront services will be unavailable during this maintenance. \- Please keep a backup of your critical data or application running on your project on NERC’s OpenShift cluster. \- Project allocation requests and change requests for both OpenStack and OpenShift Resources will not be accepted or approved during this time. \- The estimated time to complete this maintenance is 9 hours. It can take more or less time, so we urge you to keep an eye on <https://nerc.instatus.com/> for real-time updates on progress. \- Please subscribe to <https://nerc.instatus.com/subscribe/email> in order to get the progress updates during this time. Our priority is to help make science happen, so If you or your research team have any questions or need to escalate an issue, please contact us via email ([help@nerc.mghpcc.org](mailto:help@nerc.mghpcc.org)) or, by submitting a new ticket at [the NERC's Support Ticketing System](https://mghpcc.supportsystem.com/open.php)..

Duration: 4 hours

Affected Components: NERC ACADEMIC (EDU) OPENSHIFT CLUSTER

Feb 2, 18:00:00 GMT+0
Completed - Maintenance has completed successfully.

Feb 2, 14:00:01 GMT+0
Identified - Maintenance is now in progress.

Feb 2, 14:00:00 GMT+0
Identified - **Upcoming Maintenance and Upgrade for NERC**[ **Academic (EDU) OpenShift cluster**](https://console.apps.edu.nerc.mghpcc.org/) **Only - Monday, February 2, 2026 9:00 AM - 1:00 PM.** **GENERAL MAINTENANCE** \- We will be upgrading our[ Red Hat OpenShift AI (RHOAI)](https://nerc-project.github.io/nerc-docs/openshift-ai/) version of **NERC Academic (EDU) OpenShift Cluster** to version **2.22.** This release brings important enhancements to improve performance, security, and the overall user experience. **NOTICES** \- The NERC Academic (EDU) OpenShift Cluster’s OpenShift and OpenShift AI services will be unavailable during this maintenance. \- Please inform all users and students to save any critical data or applications (**Jupyter Notebooks**) running on their workbenches and projects on NERC’s Academic (EDU) OpenShift Cluster. \- Project allocation requests and change requests for NERC Academic (EDU) OpenShift Resources will not be accepted or approved during this time. \- The estimated time to complete this maintenance is **4 hours**. It can take more or less time, so we urge you to keep an eye on[ https://nerc.instatus.com/](https://nerc.instatus.com/) for real-time updates on progress. \- Please subscribe to[ https://nerc.instatus.com/subscribe/email](https://nerc.instatus.com/subscribe/email) in order to get the progress updates during this time. Our priority is to help make science happen, so If you or your research team have any questions or need to escalate an issue, please contact us via email ([help@nerc.mghpcc.org](mailto:help@nerc.mghpcc.org)) or, by submitting a new ticket at[the NERC's Support Ticketing System](https://mghpcc.supportsystem.com/open.php)..

Duration: 3 hours and 2 minutes

Affected Components: OPENSTACK BLOCK STORAGE (CINDER), STORAGE, OPENSTACK OBJECT STORE (SWIFT), NERC COLDFRONT

Jan 13, 16:02:23 GMT+0
Completed - Maintenance has completed successfully..

Jan 13, 13:00:01 GMT+0
Identified - Maintenance is now in progress.

Jan 13, 13:00:00 GMT+0
Identified - **Upcoming NERC system maintenance and upgrade will occur on Tuesday, January 13, 2026 8:00 AM - 12:00 PM.** **GENERAL MAINTENANCE** \- We are upgrading NERC’s [ColdFront](https://coldfront.mss.mghpcc.org/) Plugin to the latest version. \- We will be adding **8 H100 GPUs** on the [NERC OpenShift](https://nerc-project.github.io/nerc-docs/openshift/) Production cluster. \- We will be adding more **NESE Storage Pools** to increase the current storage quota. **NOTICES** \- This upgrade will cause some outages to the [NERC’s ColdFront service](https://nerc-project.github.io/nerc-docs/get-started/allocation/coldfront/). \- The estimated time to complete this maintenance is **4 hours**. It can take more or less time, so we urge you to keep an eye on <https://nerc.instatus.com/> for real-time updates on progress. \- Please subscribe to <https://nerc.instatus.com/subscribe/email> in order to get the progress updates during this time. Our priority is to help make science happen, so If you or your research team have any questions or need to escalate an issue, please contact us via email ([help@nerc.mghpcc.org](mailto:help@nerc.mghpcc.org)) or, by submitting a new ticket at [the NERC's Support Ticketing System](https://mghpcc.supportsystem.com/open.php)..

Duration: 3 days, 19 hours and 25 minutes

Affected Components: OPENSTACK BLOCK STORAGE (CINDER), STORAGE, OPENSTACK OBJECT STORE (SWIFT)

Jan 13, 16:01:52 GMT+0
Resolved - This incident has been resolved..

Jan 12, 14:17:21 GMT+0
Identified - We are continuing to work on a fix for this incident. We have informed users who are hosting large amount of fragmented objects and begin cleaning up some..

Jan 9, 20:36:55 GMT+0
Investigating - We are currently investigating this incident..

Duration: 5 minutes

Affected Components: NERC WEBSITE

Dec 11, 15:25:57 GMT+0
Resolved - OTHER WEBSITES & TOOLS - NERC WEBSITE is now operational! This update was created by an automated monitoring service..

Dec 11, 15:20:31 GMT+0
Investigating - OTHER WEBSITES & TOOLS - NERC WEBSITE cannot be accessed at the moment. This incident was created by an automated monitoring service..

Duration: 3 hours and 56 minutes

Affected Components: OPENSTACK BLOCK STORAGE (CINDER), OPENSTACK DASHBOARD (HORIZON), NERC RED HAT OPENSHIFT AI (RHOAI), OPENSHIFT CONTROLLER, OPENSTACK OBJECT STORE (SWIFT), OPENSHIFT WORKERS NODES, NERC COLDFRONT, , OPENSTACK EC2 API PROXY (EC2API), OPENSTACK IMAGE SERVICE (GLANCE), OPENSTACK NETWORKING SERVICE (NEUTRON), NERC OPENSHIFT WEB CONSOLE, , NERC OPENSTACK COMPUTE SERVICE (NOVA) , NERC OPENSTACK → NERC OPENSHIFT CONTAINER PLATFORM - PRODUCTION →

Dec 9, 16:55:42 GMT+0
Completed - Maintenance has completed successfully..

Dec 9, 16:43:45 GMT+0
Identified - The openstack SSL certificates are updated Successfully..

Dec 9, 14:47:34 GMT+0
Identified - The openstack SSL certificates being updated. This may cause partial outage of OSP services for brief moments during this maintenance period..

Dec 9, 13:00:01 GMT+0
Identified - Maintenance is now in progress.

Dec 9, 13:00:00 GMT+0
Identified - **Upcoming NERC system maintenance and upgrade will occur on Tuesday, December 9, 2025 8:00 AM - 12:00 PM.** **GENERAL MAINTENANCE** \- We are upgrading NERC’s [ColdFront deployment](https://coldfront.mss.mghpcc.org/) to the latest version, which will allow users to view a Daily Invoice display in ColdFront and receive threshold budget notification emails. \- We will also validate all current project allocations to ensure each namespace meets the minimum LimitRange requirements. **NOTICES** \- This upgrade will cause some outages to the [NERC’s ColdFront service](https://nerc-project.github.io/nerc-docs/get-started/allocation/coldfront/). \- The estimated time to complete this maintenance is **4 hours**. It can take more or less time, so we urge you to keep an eye on <https://nerc.instatus.com/> for real-time updates on progress. \- Please subscribe to <https://nerc.instatus.com/subscribe/email> in order to get the progress updates during this time. Our priority is to help make science happen, so If you or your research team have any questions or need to escalate an issue, please contact us via email ([help@nerc.mghpcc.org](mailto:help@nerc.mghpcc.org)) or, by submitting a new ticket at [the NERC's Support Ticketing System](https://mghpcc.supportsystem.com/open.php)..

Duration: 5 minutes

Affected Components: NERC WEBSITE

Dec 5, 21:06:42 GMT+0
Resolved - OTHER WEBSITES & TOOLS - NERC WEBSITE is now operational! This update was created by an automated monitoring service..

Dec 5, 21:01:28 GMT+0
Investigating - OTHER WEBSITES & TOOLS - NERC WEBSITE cannot be accessed at the moment. This incident was created by an automated monitoring service..

Duration: 6 minutes

Affected Components: NERC WEBSITE

Dec 1, 22:02:49 GMT+0
Resolved - OTHER WEBSITES & TOOLS - NERC WEBSITE is now operational! This update was created by an automated monitoring service..

Dec 1, 21:57:15 GMT+0
Investigating - OTHER WEBSITES & TOOLS - NERC WEBSITE cannot be accessed at the moment. This incident was created by an automated monitoring service..

Duration: 5 minutes

Affected Components: NERC WEBSITE

Dec 1, 16:53:29 GMT+0
Resolved - OTHER WEBSITES & TOOLS - NERC WEBSITE is now operational! This update was created by an automated monitoring service..

Dec 1, 16:48:23 GMT+0
Investigating - OTHER WEBSITES & TOOLS - NERC WEBSITE cannot be accessed at the moment. This incident was created by an automated monitoring service..

Duration: 22 minutes

Affected Components: NERC WEBSITE

Nov 26, 23:15:41 GMT+0
Resolved - OTHER WEBSITES & TOOLS - NERC WEBSITE is now operational! This update was created by an automated monitoring service..

Nov 26, 22:53:49 GMT+0
Investigating - OTHER WEBSITES & TOOLS - NERC WEBSITE cannot be accessed at the moment. This incident was created by an automated monitoring service..

Duration: 15 minutes

Affected Components: NERC WEBSITE

Nov 19, 06:38:42 GMT+0
Resolved - OTHER WEBSITES & TOOLS - NERC WEBSITE is now operational! This update was created by an automated monitoring service..

Nov 19, 06:23:15 GMT+0
Investigating - OTHER WEBSITES & TOOLS - NERC WEBSITE cannot be accessed at the moment. This incident was created by an automated monitoring service..

Duration: 26 minutes

Affected Components: NERC WEBSITE

Nov 19, 06:12:09 GMT+0
Resolved - OTHER WEBSITES & TOOLS - NERC WEBSITE is now operational! This update was created by an automated monitoring service..

Nov 19, 05:46:12 GMT+0
Investigating - OTHER WEBSITES & TOOLS - NERC WEBSITE cannot be accessed at the moment. This incident was created by an automated monitoring service..

Duration: 12 minutes

Affected Components: NERC WEBSITE

Nov 19, 05:39:51 GMT+0
Resolved - OTHER WEBSITES & TOOLS - NERC WEBSITE is now operational! This update was created by an automated monitoring service..

Nov 19, 05:27:47 GMT+0
Investigating - OTHER WEBSITES & TOOLS - NERC WEBSITE cannot be accessed at the moment. This incident was created by an automated monitoring service..

Duration: 5 minutes

Affected Components: NERC WEBSITE

Nov 16, 15:01:06 GMT+0
Resolved - OTHER WEBSITES & TOOLS - NERC WEBSITE is now operational! This update was created by an automated monitoring service..

Nov 16, 14:55:58 GMT+0
Investigating - OTHER WEBSITES & TOOLS - NERC WEBSITE cannot be accessed at the moment. This incident was created by an automated monitoring service..

Duration: 4 hours

Affected Components: NERC COLDFRONT

Nov 11, 15:34:45 GMT+0
Completed - Maintenance has completed successfully..

Nov 11, 14:59:25 GMT+0
Identified - Maintenance is still in progress. We are extending the maintenance for another 2 hours..

Nov 11, 13:00:01 GMT+0
Identified - Maintenance is now in progress.

Nov 11, 13:00:00 GMT+0
Identified - **NERC’s** [**ColdFront**](https://coldfront.mss.mghpcc.org/) **upgrade and maintenance will occur on Tuesday Nov 11, 2025 from 8:00 AM - 12:00 PM.** **GENERAL MAINTENANCE** \- We are upgrading NERC’s ColdFront deployment to the latest version of ColdFront. **NOTICES** \- This upgrade will cause some outages to the [NERC’s ColdFront service](https://nerc-project.github.io/nerc-docs/get-started/allocation/coldfront/). \- The estimated time to complete this maintenance is **4 hours**. It can take more or less time, so we urge you to keep an eye on <https://nerc.instatus.com/> for real-time updates on progress. \- Please subscribe to <https://nerc.instatus.com/subscribe/email> in order to get the progress updates during this time. Our priority is to help make science happen, so If you or your research team have any questions or need to escalate an issue, please contact us via email ([help@nerc.mghpcc.org](mailto:help@nerc.mghpcc.org)) or, by submitting a new ticket at [the NERC's Support Ticketing System](https://mghpcc.supportsystem.com/open.php). Thank you, New England Research Cloud (NERC) [https://nerc.mghpcc.org/ ](https://nerc.mghpcc.org/)<https://nerc-project.github.io/nerc-docs/> .

Duration: 5 minutes

Affected Components: NERC WEBSITE

Nov 5, 09:38:27 GMT+0
Resolved - OTHER WEBSITES & TOOLS - NERC WEBSITE is now operational! This update was created by an automated monitoring service..

Nov 5, 09:33:19 GMT+0
Investigating - OTHER WEBSITES & TOOLS - NERC WEBSITE cannot be accessed at the moment. This incident was created by an automated monitoring service..

Duration: 28 minutes

Affected Components: NERC WEBSITE

Oct 31, 08:35:43 GMT+0
Resolved - OTHER WEBSITES & TOOLS - NERC WEBSITE is now operational! This update was created by an automated monitoring service..

Oct 31, 08:07:55 GMT+0
Investigating - OTHER WEBSITES & TOOLS - NERC WEBSITE cannot be accessed at the moment. This incident was created by an automated monitoring service..

Duration: 22 minutes

Affected Components: NERC WEBSITE

Oct 24, 06:19:10 GMT+0
Resolved - OTHER WEBSITES & TOOLS - NERC WEBSITE is now operational! This update was created by an automated monitoring service..

Oct 24, 05:57:11 GMT+0
Investigating - OTHER WEBSITES & TOOLS - NERC WEBSITE cannot be accessed at the moment. This incident was created by an automated monitoring service..

Duration: 9 hours

Affected Components: NERC RED HAT OPENSHIFT AI (RHOAI), NERC COLDFRONT

Oct 14, 21:00:00 GMT+0
Completed - Maintenance has completed successfully.

Oct 14, 12:00:01 GMT+0
Identified - Maintenance is now in progress.

Oct 14, 12:00:00 GMT+0
Identified - Upcoming NERC system maintenance and upgrade will occur on Tuesday, October 14, 2025 8:00 AM - 5:00 PM. **GENERAL MAINTENANCE** \- We will be upgrading our [Red Hat OpenShift AI (RHOAI)](https://nerc-project.github.io/nerc-docs/openshift-ai/) version of NERC OpenShift Production Cluster to version **2.22.** This release brings important enhancements to improve performance, security, and the overall user experience. \- We are upgrading our [ColdFront](https://coldfront.mss.mghpcc.org/) deployment. **NOTICES** \- The **OpenShift AI and ColdFront** services will be unavailable during this maintenance. \- Please keep a backup of your critical data or application running on your project on NERC’s OpenShift cluster. \- Project allocation requests and change requests for both OpenStack and OpenShift Resources **will not be accepted or approved** during this time. \- The estimated time to complete this maintenance is **9 hours**. It can take more or less time, so we urge you to keep an eye on <https://nerc.instatus.com/> for real-time updates on progress. \- Please subscribe to <https://nerc.instatus.com/subscribe/email> in order to get the progress updates during this time. Our priority is to help make science happen, so If you or your research team have any questions or need to escalate an issue, please contact us via email ([help@nerc.mghpcc.org](mailto:help@nerc.mghpcc.org)) or, by submitting a new ticket at [the NERC's Support Ticketing System](https://mghpcc.supportsystem.com/open.php). Thank you, New England Research Cloud (NERC) [https://nerc.mghpcc.org/ ](https://nerc.mghpcc.org/)<https://nerc-project.github.io/nerc-docs/> .

Duration: 28 minutes

Affected Components: NERC WEBSITE

Oct 5, 21:05:18 GMT+0
Resolved - OTHER WEBSITES & TOOLS - NERC WEBSITE is now operational! This update was created by an automated monitoring service..

Oct 5, 20:37:32 GMT+0
Investigating - OTHER WEBSITES & TOOLS - NERC WEBSITE cannot be accessed at the moment. This incident was created by an automated monitoring service..

Duration: 7 minutes

Affected Components: NERC OPENSHIFT WEB CONSOLE, , NERC RED HAT OPENSHIFT AI (RHOAI), OPENSHIFT WORKERS NODES, OPENSHIFT CONTROLLER, NERC OPENSHIFT CONTAINER PLATFORM - PRODUCTION →

Oct 3, 15:37:46 GMT+0
Resolved - This unplanned maintenance has been completed..

Oct 3, 15:30:32 GMT+0
Monitoring - We are performing an unplanned maintenance to upgrade the RedHat OpenShift AI operator to 2.19.3 in order to fix CVE-2025-10725.

Duration: 9 hours

Affected Components: , NERC COLDFRONT, NERC OPENSHIFT WEB CONSOLE, NERC RED HAT OPENSHIFT AI (RHOAI), OPENSHIFT WORKERS NODES, OPENSHIFT CONTROLLER, NERC OPENSHIFT CONTAINER PLATFORM - PRODUCTION →

Sep 9, 21:00:00 GMT+0
Completed - Maintenance has completed successfully.

Sep 9, 12:15:09 GMT+0
Identified - Maintenance is now in progress. We will also conduct NERC ColdFront upgrade and maintenance during this timeline..

Sep 9, 12:00:01 GMT+0
Identified - Maintenance is now in progress.

Sep 9, 12:00:00 GMT+0
Identified - **Upcoming NERC system maintenance and upgrade will occur on Tuesday, September 9, 2025 8:00 AM - 5:00 PM.** **GENERAL MAINTENANCE** \- We will be upgrading our Red Hat [OpenShift container platform (OCP)](https://nerc-project.github.io/nerc-docs/openshift/) version on NERC to version **4.19.** This release brings important enhancements to improve performance, security, and the overall user experience. \- **Unified Console Experience**: The Admin and Developer views are now combined into one interface, reducing context switching and enabling seamless end-to-end workflows. \- We have also set up and now offer an isolated [Academic (EDU) OpenShift cluster](https://console.apps.edu.nerc.mghpcc.org/) designed to fully support academic research and coursework, providing a stable and dedicated environment for instructors, principal investigators, students, and other users. If you are **planning to run courses on NERC** in **Fall 2025** or **Spring 2026**, please contact us so we can coordinate and provide the necessary support. **NOTICES** \- The OpenShift services will be unavailable during this maintenance. \- Please keep a backup of your critical data or application running on your project on NERC’s OpenShift cluster. \- Project allocation requests and change requests for both OpenStack and OpenShift Resources **will not be accepted or approved** during this time. \- The estimated time to complete this maintenance is **9 hours**. It can take more or less time, so we urge you to keep an eye on <https://nerc.instatus.com/> for real-time updates on progress. \- Please subscribe to <https://nerc.instatus.com/subscribe/email> in order to get the progress updates during this time. Our priority is to help make science happen, so If you or your research team have any questions or need to escalate an issue, please contact us via email ([help@nerc.mghpcc.org](mailto:help@nerc.mghpcc.org)) or, by submitting a new ticket at [the NERC's Support Ticketing System](https://mghpcc.supportsystem.com/open.php). Thank you, New England Research Cloud (NERC) [https://nerc.mghpcc.org/ ](https://nerc.mghpcc.org/)<https://nerc-project.github.io/nerc-docs/>.

Duration: 11 minutes

Affected Components: NERC WEBSITE

Sep 2, 12:25:59 GMT+0
Resolved - OTHER WEBSITES & TOOLS - NERC WEBSITE is now operational! This update was created by an automated monitoring service..

Sep 2, 12:15:17 GMT+0
Investigating - OTHER WEBSITES & TOOLS - NERC WEBSITE cannot be accessed at the moment. This incident was created by an automated monitoring service..

Duration: 1 hour and 57 minutes

Affected Components: NERC WEBSITE

Aug 28, 13:23:32 GMT+0
Resolved - OTHER WEBSITES & TOOLS - NERC WEBSITE is now operational! This update was created by an automated monitoring service..

Aug 28, 11:26:57 GMT+0
Investigating - OTHER WEBSITES & TOOLS - NERC WEBSITE cannot be accessed at the moment. This incident was created by an automated monitoring service..

Duration: 9 hours

Affected Components: , NERC COLDFRONT, NERC OPENSHIFT WEB CONSOLE, NERC RED HAT OPENSHIFT AI (RHOAI), OPENSHIFT WORKERS NODES, OPENSHIFT CONTROLLER, NERC OPENSHIFT CONTAINER PLATFORM - PRODUCTION →

Aug 12, 21:00:00 GMT+0
Completed - Maintenance has completed successfully.

Aug 12, 12:00:01 GMT+0
Identified - Maintenance is now in progress.

Aug 12, 12:00:00 GMT+0
Identified - **Upcoming NERC system maintenance and upgrade will occur on Tuesday August 12, 2025 8:00 AM - 5:00 PM.** **GENERAL MAINTENANCE** \- We will be upgrading our Red Hat [OpenShift container platform (OCP)](https://nerc-project.github.io/nerc-docs/openshift/) version on NERC to **4.18**. \- As part of the upcoming maintenance, we’ll be rolling out several key enhancements to [Red Hat OpenShift AI (RHOAI)](https://nerc-project.github.io/nerc-docs/openshift-ai/). These include a redesigned dashboard layout, support for both single and multi-model serving, and distributed training capabilities using CodeFlare and KubeRay. The platform will also support high-performance inference runtimes beyond the currently offered OpenVINO, such as [vLLM](https://www.redhat.com/en/topics/ai/what-is-vllm). Additional improvements include a [GPU pruner](https://ai-on-openshift.io/odh-rhoai/gpu-pruner/) for more efficient GPU utilization, enhanced resource scheduling with [Kueue](https://kueue.sigs.k8s.io/), and an upgraded version of Kubeflow Pipelines. This update will further strengthen MLOps capabilities with an integrated model registry and catalog to manage the full model lifecycle. \- We will be upgrading our [ColdFront](https://coldfront.mss.mghpcc.org/) deployment. **NOTICES** \- The OpenShift and ColdFront services will be unavailable during this maintenance. \- Please keep a backup of your critical data or application running on your project on NERC’s OpenShift cluster. \- Project allocation requests and change requests for both OpenStack and OpenShift Resources **will not be accepted or approved** during this time. \- The estimated time to complete this maintenance is **9 hours**. It can take more or less time, so we urge you to keep an eye on <https://nerc.instatus.com/> for real-time updates on progress. \- Please subscribe to <https://nerc.instatus.com/subscribe/email> in order to get the progress updates during this time. Our priority is to help make science happen, so If you or your research team have any questions or need to escalate an issue, please contact us via email ([help@nerc.mghpcc.org](mailto:help@nerc.mghpcc.org)) or, by submitting a new ticket at [the NERC's Support Ticketing System](https://mghpcc.supportsystem.com/open.php). Thank you, New England Research Cloud (NERC) [https://nerc.mghpcc.org/ ](https://nerc.mghpcc.org/)<https://nerc-project.github.io/nerc-docs/>.

Duration: 5 hours and 9 minutes

Affected Components: NERC WEBSITE

Jun 20, 13:19:25 GMT+0
Resolved - OTHER WEBSITES & TOOLS - NERC WEBSITE is now operational! This update was created by an automated monitoring service..

Jun 20, 13:14:16 GMT+0
Investigating - OTHER WEBSITES & TOOLS - NERC WEBSITE cannot be accessed at the moment. This incident was created by an automated monitoring service..

Duration: 10 hours and 18 minutes

Affected Components: NERC WEBSITE

Jun 20, 12:40:30 GMT+0
Resolved - OTHER WEBSITES & TOOLS - NERC WEBSITE is now operational! This update was created by an automated monitoring service..

Jun 20, 12:30:13 GMT+0
Investigating - OTHER WEBSITES & TOOLS - NERC WEBSITE cannot be accessed at the moment. This incident was created by an automated monitoring service..

Duration: 2 hours

Affected Components: , OPENSHIFT WORKERS NODES, OPENSHIFT CONTROLLER, NERC OPENSHIFT WEB CONSOLE, NERC RED HAT OPENSHIFT AI (RHOAI), NERC OPENSHIFT CONTAINER PLATFORM - PRODUCTION →

Jun 18, 15:00:21 GMT+0
Completed - Maintenance has completed successfully..

Jun 18, 13:00:01 GMT+0
Identified - Maintenance is now in progress.

Jun 18, 13:00:00 GMT+0
Identified - We need to perform rolling reboots of all nodes on our [OpenShift cluster](https://nerc-project.github.io/nerc-docs/openshift/) to adjust the cluster MTU setting (from 8900 to 1400) and rectify downstream issues. This was inadvertently updated during last week’s cluster version upgrade to 4.17. This maintenance will cause running workloads to be interrupted and relaunched..

Duration: 1 day, 3 hours and 27 minutes

Affected Components: , OPENSHIFT WORKERS NODES, NERC OPENSHIFT WEB CONSOLE, OPENSHIFT CONTROLLER, NERC RED HAT OPENSHIFT AI (RHOAI), NERC OPENSHIFT CONTAINER PLATFORM - PRODUCTION →

Jun 12, 21:12:08 GMT+0
Resolved - This incident has been resolved. OpenShift is fully usable with one exception. The H100s are not yet ready for use..

Jun 11, 17:44:44 GMT+0
Investigating - We are currently investigating this incident..

Duration: 13 hours

Affected Components: , OPENSHIFT WORKERS NODES, NERC OPENSHIFT WEB CONSOLE, OPENSHIFT CONTROLLER, NERC RED HAT OPENSHIFT AI (RHOAI), NERC OPENSHIFT CONTAINER PLATFORM - PRODUCTION →

Jun 11, 01:00:00 GMT+0
Completed - Maintenance has completed successfully.

Jun 10, 20:51:06 GMT+0
Identified - Maintenance is still in progress. We are extending the maintenance for another 4 hours..

Jun 10, 12:00:01 GMT+0
Identified - Maintenance is now in progress.

Jun 10, 12:00:00 GMT+0
Identified - **Upcoming Red Hat's** [**OpenShift container platform (OCP)**](http://%28https//nerc-project.github.io/nerc-docs/openshift/) **upgrade on NERC from version 4.15 to 4.17 will occur on Tuesday, June 10, 2025, from 8:00 AM – 5:00 PM.** **GENERAL MAINTENANCE** We are writing to inform you that we plan to upgrade our [OpenShift cluster](https://nerc-project.github.io/nerc-docs/openshift/) from **version 4.15 to 4.17** on **Tuesday, June 10, 2025**. As our current version of OpenShift will reach its End of Life (EOL) in less than three months, this upgrade is essential and will introduce several new features and improvements to our cluster. Additionally, we will upgrade and maintain the [Red Hat OpenShift AI Platform (RHOAI)](https://nerc-project.github.io/nerc-docs/openshift-ai/) to **version 2.19**, which provides a fully supported, collaborative sandbox environment for data scientists, AI engineers, and researchers to develop, train, and test AI/ML models, as well as deploy them for use in intelligent applications. **NOTICES** \- Please keep a backup of your critical data or application running on your project on NERC's OpenShift cluster. \- The new upgrade to the [latest OpenShift version](https://www.redhat.com/en/whats-new-red-hat-openshift) will provide improved performance, enhanced security, additional operators and enhanced user interface along with updated Kubernetes version. \- During the upgrade, there will be a period of downtime to ensure a seamless transition. We anticipate this downtime to last **approximately 9 hours**, and we apologize for any inconvenience this may cause. It can take more or less time, so we urge you to keep an eye on <https://nerc.instatus.com/> and also subscribe to <https://nerc.instatus.com/subscribe/email> to get the progress updates during this time. \- Our state-of-the-art [OpenShift](https://nerc-project.github.io/nerc-docs/openshift/gpus/intro-to-gpus-on-nerc-ocp/) and [RHOAI](https://nerc-project.github.io/nerc-docs/openshift-ai/other-projects/predictive-and-generative-AI/) platforms enable you and your team to deploy containerized applications in a cloud-native environment, providing a reliable, isolated, and scalable solution for your complex research computing and teaching needs. If you haven't used or tested these platforms, please get started by requesting a new resource allocation to your project using [NERC's ColdFront web console](https://nerc-project.github.io/nerc-docs/get-started/get-an-allocation/#request-a-new-openshift-resource-allocation-for-openshift-project) or you can get in touch with us to have a quick demo. More information about NERC is available on [NERC's website](https://nerc.mghpcc.org/). If you have any questions, please don't hesitate to reach out to us via email ([help@nerc.mghpcc.org](mailto:help@nerc.mghpcc.org)) or by submitting a new ticket at the [NERC's Support Ticketing System](https://mghpcc.supportsystem.com/open.php). Thanks, New England Research Cloud (NERC) <https://nerc.mghpcc.org/> <https://nerc-project.github.io/nerc-docs/>.

Duration: 2 days and 2 hours

Affected Components: STORAGE, NERC OPENSTACK COMPUTE SERVICE (NOVA) , MGHPCC SHARED SERVICES (MGHPCC-SS) ACCOUNT PORTAL, OPENSTACK NETWORKING SERVICE (NEUTRON), OPENSTACK OBJECT STORE (SWIFT), NERC OPENSHIFT WEB CONSOLE, OPENSHIFT WORKERS NODES, NERC COLDFRONT, OPENSTACK EC2 API PROXY (EC2API), OPENSTACK BLOCK STORAGE (CINDER), OPENSHIFT CONTROLLER, NETWORKING, OPENSTACK DASHBOARD (HORIZON), OPENSTACK IMAGE SERVICE (GLANCE)

Jun 4, 21:00:00 GMT+0
Completed - Maintenance has completed successfully.

Jun 2, 19:00:01 GMT+0
Identified - Maintenance is now in progress.

Jun 2, 19:00:00 GMT+0
Identified - Upcoming MGHPCC (Holyoke) Annual Power Shutdown \[June 2, 2025 3:00 PM ET - June 4, 2025 5:00 PM ET\] (multi-day) Due to the annual MGHPCC (Holyoke data center) power maintenance as well as other maintenance activities, the New England Research Cloud (NERC) services will be unavailable from **Monday, June 2nd, 3 PM until Wednesday, June 4th, 5 PM**. NERC services that will be impacted include: * NERC's OpenStack * NERC's OpenShift, which includes the Red Hat OpenShift AI (RHOAI) platform * NERC supporting shared services ColdFront, RegApp, ESI, and Keycloak We have worked to reduce the total outage time this year. Our goal is to resume full services by **the end of Wednesday, June 4th, at 5 PM** or **earlier**. Please refer to the timeline below for details: !\[Maintenance Overview\](<https://nerc.mghpcc.org/wp-content/uploads/2025/05/mghpcc-power-outage-2025.png>) **NOTICES** **\---------** \- This outage affects all services and resources in the MGHPCC/Holyoke data center, including NERC’s core and shared services, as well as the underlying compute, storage, and networking. \- The estimated time for power-up and return to full service is **5 PM on Wednesday, June 4th**. We encourage you to monitor the progress at <https://nerc.instatus.com/> during this time. \- Please subscribe to the NERC’s status for any future updates as they happen: <https://nerc.instatus.com/subscribe/email> Our priority is helping make research happen. If you or your research team have any questions or need to escalate an issue, please do not hesitate to contact us via email ([help@nerc.mghpcc.org](mailto:help@nerc.mghpcc.org)) or by submitting a service request through [the NERC's Support Ticketing System](https://mghpcc.supportsystem.com/open.php). Thanks, New England Research Cloud (NERC) <https://nerc.mghpcc.org/> <https://nerc-project.github.io/nerc-docs/>.

Duration: 9 hours

Affected Components: STORAGE, OPENSTACK DASHBOARD (HORIZON), OPENSTACK NETWORKING SERVICE (NEUTRON), OPENSTACK OBJECT STORE (SWIFT), NERC OPENSHIFT WEB CONSOLE, OPENSHIFT WORKERS NODES, NERC COLDFRONT, NERC OPENSTACK COMPUTE SERVICE (NOVA) , OPENSTACK IMAGE SERVICE (GLANCE), OPENSTACK EC2 API PROXY (EC2API), OPENSTACK BLOCK STORAGE (CINDER), OPENSHIFT CONTROLLER

May 13, 21:00:00 GMT+0
Completed - Maintenance has completed successfully.

May 13, 12:00:01 GMT+0
Identified - Maintenance is now in progress.

May 13, 12:00:00 GMT+0
Identified - Upcoming NERC system maintenance and upgrade - Tuesday, May 13, 2025 8:00 AM - 5:00 PM. **GENERAL MAINTENANCE** **\------------------------------** \- We will resolve issues related to Shelving VMs to save on resource costs when they are not in use. \- We will conduct maintenance on the [NERC OpenStack RadosGW (Swift/S3) object storage](https://nerc-project.github.io/nerc-docs/openstack/persistent-storage/object-storage/) service. \- Additionally, we will deploy the [H100 GPU](https://nerc.mghpcc.org/nvidia-h100-gpus-on-nerc/) configuration on [NERC OpenShift](https://nerc-project.github.io/nerc-docs/openshift/). \- Users will be able to add external hostname (DNS) to their ingress with TLS/SSL certificates from Let's Encrypt as [described here](https://nerc-project.github.io/nerc-docs/openshift/domain-name-system/domain-name-for-your-application/). **NOTICES** **\----------** \- The OpenStack and OpenShift services will be unavailable during this maintenance. \- Project allocation requests and change requests for both OpenStack and OpenShift Resources **will not be accepted or approved** during this time. \- The estimated time to complete this maintenance is **9 hours**. It can take more or less time, so we urge you to keep an eye on <https://nerc.instatus.com/> for real-time updates on progress. \- Please do subscribe to <https://nerc.instatus.com/subscribe/email> in order to get the progress updates during this time. Our priority is to help make science happen, so If you or your research team have any questions or need to escalate an issue, please contact us via email ([help@nerc.mghpcc.org](mailto:help@nerc.mghpcc.org)) or, by submitting a new ticket at [the NERC's Support Ticketing System](https://mghpcc.supportsystem.com/open.php). Thank you, New England Research Cloud (NERC) [https://nerc.mghpcc.org/ https://nerc-project.github.io/nerc-docs/](https://nerc.mghpcc.org/https://nerc-project.github.io/nerc-docs/).

Duration: 3 hours and 7 minutes

Affected Components: OPENSTACK DASHBOARD (HORIZON), OPENSTACK NETWORKING SERVICE (NEUTRON), OPENSTACK OBJECT STORE (SWIFT), NERC OPENSTACK COMPUTE SERVICE (NOVA) , NERC OPENSHIFT WEB CONSOLE, OPENSTACK IMAGE SERVICE (GLANCE), OPENSTACK EC2 API PROXY (EC2API), OPENSTACK BLOCK STORAGE (CINDER), OPENSHIFT CONTROLLER, OPENSHIFT WORKERS NODES

May 12, 16:30:50 GMT+0
Resolved - This incident has been resolved..

May 12, 13:23:47 GMT+0
Investigating - We are currently investigating this incident..

Duration: 1 hour and 34 minutes

Affected Components: NERC COLDFRONT, OPENSTACK OBJECT STORE (SWIFT)

Feb 11, 15:34:13 GMT+0
Completed - We have successfully completed the maintenance and upgrade!.

Feb 11, 14:00:01 GMT+0
Identified - Maintenance is now in progress.

Feb 11, 14:00:00 GMT+0
Identified - Upcoming NERC system maintenance and upgrade will occur on Tuesday February 11, 2025 9:00 AM - 11:00 AM. **GENERAL MAINTENANCE** \- We will be upgrading our [ColdFront](https://coldfront.mss.mghpcc.org/) deployment. Additionally, we will be performing maintenance on the NERC [OpenStack RadosGW (Swift/S3) object storage service](https://nerc-project.github.io/nerc-docs/openstack/persistent-storage/object-storage/). \- Other than OpenStack projects which use Swift/S3 this maintenance should not affect any other projects using the NERC. **NOTICES** \- The OpenStack RadosGW (Swift/S3) object storage service will be unavailable during this maintenance. \- Project allocation requests and change requests **will not be accepted or approved** during this time. \- The maintenance and upgrade are expected to be brief. We encourage you to monitor <https://nerc.instatus.com/> for real-time updates on progress. \- Please subscribe to <https://nerc.instatus.com/subscribe/email> in order to get the progress updates during this time. Our priority is to help make science happen, so If you or your research team have any questions or need to escalate an issue, please contact us via email ([help@nerc.mghpcc.org](mailto:help@nerc.mghpcc.org)) or, by submitting a new ticket at [the NERC's Support Ticketing System](https://mghpcc.supportsystem.com/open.php). Thank you, New England Research Cloud (NERC) [https://nerc.mghpcc.org/ https://nerc-project.github.io/nerc-docs/](https://nerc.mghpcc.org/https://nerc-project.github.io/nerc-docs/).

Duration: 1 day and 3 hours

Affected Components: NERC OPENSTACK COMPUTE SERVICE (NOVA) , OPENSHIFT CONTROLLER, NETWORKING, OPENSTACK IMAGE SERVICE (GLANCE), OPENSTACK EC2 API PROXY (EC2API), STORAGE, OPENSTACK OBJECT STORE (SWIFT), OPENSHIFT WORKERS NODES, NERC OPENSHIFT WEB CONSOLE, OPENSTACK BLOCK STORAGE (CINDER), OPENSTACK DASHBOARD (HORIZON), OPENSTACK NETWORKING SERVICE (NEUTRON)

Jan 15, 17:00:00 GMT+0
Completed - Maintenance has completed successfully.

Jan 15, 14:53:54 GMT+0
Identified - We need to extend the maintenance for 3 more hours. We will update you as it progress..

Jan 14, 14:00:01 GMT+0
Identified - Maintenance is now in progress.

Jan 14, 14:00:00 GMT+0
Identified - We want to inform you about an update to the previously announced maintenance schedule. The maintenance activity has been pushed back by one week. Please find the revised details here: <https://nerc.mghpcc.org/event/network-equipment-switch-maintenance-20250114/>.

Jan 14, 14:00:00 GMT+0
Identified - Upcoming NERC Network Equipment and Switch Maintenance is scheduled from Tuesday, January 14, 2025, at 9:00 AM to Wednesday, January 15, 2025, at 9:00 AM. **GENERAL MAINTENANCE** \- We plan to perform changes to the management and configuration of our network infrastructure. This will take the majority of the maintenance period. These updates are part of our ongoing efforts to improve network performance, reliability, and scalability for all users. \- During the maintenance window, we will also update the NERC OpenShift deployment to include patches for the NVIDIA A100 and V100 GPU Accelerator profiles. **NOTICES** \- [NERC OpenStack’s Horizon dashboard](https://stack.nerc.mghpcc.org/) and other [OpenStack API](https://nerc-project.github.io/nerc-docs/openstack/openstack-cli/openstack-CLI/) services will be inaccessible. \- During this process, Access to the [NERC OpenShift cluster](https://nerc-project.github.io/nerc-docs/openshift/) and [NERC Red Hat OpenShift AI (RHOAI)](https://nerc-project.github.io/nerc-docs/openshift-ai/) via the web console and other OpenShift CLI tools will be unavailable. \- Although we do not anticipate that ESI-provided systems will be affected, except for those connected to the NERC network, there is a slight risk due to the tasks we will be performing in their general vicinity. \- All services will be available once the OpenShift Deployment portion of this maintenance is completed, and we will post updates to the status page. \- Please be aware that during this maintenance, project change requests for both OpenStack and OpenShift Resources **will not be accepted or approved**. \- The estimated time to complete this maintenance is **24 hours**. It can take more or less time, so we urge you to keep an eye on <https://nerc.instatus.com/> to receive updated progress. \- Please subscribe to <https://nerc.instatus.com/subscribe/email> to receive progress updates. \- Please let us know if you encounter any issues after the maintenance is completed. Our priority is to help make science happen, if you or your research team have any questions or need to escalate an issue, please contact us via email ([help@nerc.mghpcc.org](mailto:help@nerc.mghpcc.org)) or, by submitting a new ticket at [the NERC's Support Ticketing System](https://mghpcc.supportsystem.com/open.php). Thank you, New England Research Cloud (NERC) [https://nerc.mghpcc.org/ https://nerc-project.github.io/nerc-docs/](https://nerc.mghpcc.org/https://nerc-project.github.io/nerc-docs/).

Duration: 2 days and 12 hours

Affected Components: NERC OPENSTACK COMPUTE SERVICE (NOVA) , OPENSTACK IMAGE SERVICE (GLANCE), OPENSTACK EC2 API PROXY (EC2API), OPENSTACK OBJECT STORE (SWIFT), OPENSTACK BLOCK STORAGE (CINDER), OPENSTACK DASHBOARD (HORIZON), OPENSTACK NETWORKING SERVICE (NEUTRON)

Dec 15, 01:00:00 GMT+0
Completed - Maintenance has completed successfully.

Dec 12, 13:00:01 GMT+0
Identified - Maintenance is now in progress.

Dec 12, 13:00:00 GMT+0
Identified - Upcoming [NERC OpenStack Platform](https://nerc-project.github.io/nerc-docs/openstack/) upgrade to version 17.1 will occur from Thursday December 12th, 2024 from 8:00 AM until Saturday, December 14th, 8:00 PM. **GENERAL MAINTENANCE** \- We are planning to upgrade our NERC OpenStack Platform to version 17.1\. This upgrade will provide enhanced features, performance improvements, and additional security & compliance measures to support your work and ensure a reliable cloud experience. This major upgrade has been long-pending, so we are reserving a 3-day window for this process to ensure a smooth transition. \- The [NERC OpenShift Container Platform (OCP)](https://console.apps.shift.nerc.mghpcc.org/) will remain available during this time. **NOTICES** \- We recommend saving your work and preparing for the downtime in advance.\- NERC [OpenStack’s Horizon dashboard](https://stack.nerc.mghpcc.org/) and other [OpenStack API](https://nerc-project.github.io/nerc-docs/openstack/openstack-cli/openstack-CLI/) services may be inaccessible at times during this process. \- This upgrade process should not affect the state of your VMs, persistent storage, or data stored in volumes and object storage. \- Please be aware that during this maintenance, project, and project change requests for NERC OpenStack Resources **will not be accepted and approved**. \- We are reserving **the three days**, as this is a major and long pending upgrade to our cluster. It can take more or less time, so we urge you to keep an eye on <https://nerc.instatus.com/> to get the progress during this time. \- Please do subscribe to <https://nerc.instatus.com/subscribe/email> in order to get the progress updates during this time. Our priority is to help make science happen, so If you or your research team have any questions or need to escalate an issue, please contact us via email ([help@nerc.mghpcc.org](mailto:help@nerc.mghpcc.org)) or, by submitting a new ticket at [the NERC's Support Ticketing System](https://mghpcc.supportsystem.com/open.php). Thank you, New England Research Cloud (NERC) [https://nerc.mghpcc.org/ https://nerc-project.github.io/nerc-docs/](https://nerc.mghpcc.org/https://nerc-project.github.io/nerc-docs/).

Duration: 8 hours

Affected Components: NERC COLDFRONT, NERC OPENSTACK COMPUTE SERVICE (NOVA) , OPENSTACK IMAGE SERVICE (GLANCE), OPENSTACK EC2 API PROXY (EC2API), OPENSTACK OBJECT STORE (SWIFT), OPENSTACK BLOCK STORAGE (CINDER), OPENSTACK DASHBOARD (HORIZON), OPENSTACK NETWORKING SERVICE (NEUTRON)

Sep 3, 21:00:00 GMT+0
Completed - Maintenance has completed successfully.

Sep 3, 13:00:01 GMT+0
Identified - Maintenance is now in progress.

Sep 3, 13:00:00 GMT+0
Identified - Upcoming NERC[ OpenStack Platform](https://stack.nerc.mghpcc.org/),[ ColdFront service](https://coldfront.mss.mghpcc.org/), and OpenShift Account Manager maintenance will occur on Tuesday September 3, 2024 from 9:00 AM – 5:00 PM. **GENERAL MAINTENANCE** We are planning to migrate a set of GPU hypervisors (V100) from the NERC OpenStack cluster to the NERC OpenShift cluster. We will also be upgrading both the ColdFront allocation system and the OpenShift account management software. This maintenance is scheduled for Tuesday September 3, 2024 from 9:00 AM – 5:00 PM. **NOTICES** \- We will be moving some of the GPU OpenStack hypervisors (V100) to our [OpenShift Container Platform (OCP)](https://console.apps.shift.nerc.mghpcc.org/) cluster. \- Some OpenStack APIs may briefly experience outages, and any active object storage transfers will be interrupted. \- Please do not launch any new VMs or initiate data transfers to/from existing VMs during this time. \- We will be upgrading the ColdFront allocation system and also OpenShift account management software. All OpenShift APIs and workloads will be available and functional during this portion of the maintenance. \- Please be aware that during this maintenance, project and project change requests (Related to the OpenStack Resource Only) **will not be accepted and approved**. \- We are reserving **the whole day**, from 9am to 5pm. It can take more or less time, so we urge you to keep an eye on <https://nerc.instatus.com/> to get the progress during this time. We will post updates to our status page when the ColdFront and OpenShift Account Manager portion of this maintenance has been completed. \- Please do subscribe to <https://nerc.instatus.com/subscribe/email> in order to get the progress updates during this time. Our priority is to help make science happen, so If you or your research team have any questions or need to escalate an issue, please contact us via email ([help@nerc.mghpcc.org](mailto:help@nerc.mghpcc.org)) or, by submitting a new ticket at [the NERC's Support Ticketing System](https://mghpcc.supportsystem.com/open.php). Thank you, New England Research Cloud (NERC) [https://nerc.mghpcc.org/ https://nerc-project.github.io/nerc-docs/](https://nerc.mghpcc.org/https://nerc-project.github.io/nerc-docs/).

Duration: 10 hours

Affected Components: NERC COLDFRONT, OPENSHIFT CONTROLLER, OPENSHIFT WORKERS NODES, NERC OPENSHIFT WEB CONSOLE

Aug 5, 23:00:00 GMT+0
Completed - Maintenance has completed successfully.

Aug 5, 20:46:55 GMT+0
Identified - We are still in progress to upgrade our NERC OCP cluster version, so extending this maintenance for another 2 hours until 7 :00 PM..

Aug 5, 13:44:39 GMT+0
Identified - We have successfully completed the NERC ColdFront service upgrade. Still the OpenShift upgrade will take whole day. Please be aware that any long-running pods or jobs on the NERC OpenShift platforms will be terminated and restarted. If they are not using persistent volumes or data, they might not be accessible even after the maintenance is complete. Please review them quickly and ensure you have a backup..

Aug 5, 13:00:01 GMT+0
Identified - Maintenance is now in progress.

Aug 5, 13:00:00 GMT+0
Identified - Upcoming NERC [OpenShift Container Platform (OCP)](https://console.apps.shift.nerc.mghpcc.org/) and [ColdFront service](https://coldfront.mss.mghpcc.org/) maintenance and upgrade will occur on Monday August 5, 2024 9 AM - 5 PM. **GENERAL MAINTENANCE** \- We plan to perform maintenance and a version upgrade on the NERC OpenShift deployment. This maintenance will involve a rolling reboot of all hosts in the cluster. \- We are also upgrading and updating NERC's ColdFront deployment. **NOTICES** \- We plan to upgrade the NERC OpenShift deployment to a newer **Red Hat OpenShift (OCP)** version 4.15 and **Red Hat OpenShift AI (RHOAI)** to v2.8.2, incorporating new features and functionalities. \- [This version](https://docs.openshift.com/container-platform/4.15/nodes/containers/nodes-containers-dev-fuse.html) will allow accessing faster builds with **_/dev/fuse_** on users' unprivileged pods, enabling users to leverage the **Filesystem in Userspace (FUSE)** interface to create and manage custom filesystems without requiring elevated privileges. \- During this process, Access to the [NERC OpenShift cluster](https://nerc-project.github.io/nerc-docs/openshift/) and [NERC Red Hat OpenShift AI (RHOAI)](https://nerc-project.github.io/nerc-docs/openshift-ai/) via the web console and other OpenShift CLI tools will be unavailable. \- This upgrade will cause some outages to the NERC’s ColdFront service. \- Please be aware that during this maintenance project, project change requests for OpenShift Resource Type **will not be accepted or approved**. \- The estimated time to complete ColdFront service update is **2 hours** and for OCP update is **8 hours**. It can take more or less time, so we urge you to keep an eye on <https://nerc.instatus.com/> to get the progress during this time. We will post updates to the status page when the ColdFront portion of this maintenance has been completed. \- Please do subscribe to <https://nerc.instatus.com/subscribe/email> to get the progress updates during this time. \- Please let us know if you encounter any issues after the maintenance is completed. Our priority is to help make science happen, so If you or your research team have any questions or need to escalate an issue, please don't hesitate to reach out to us via email ([help@nerc.mghpcc.org](mailto:help@nerc.mghpcc.org)) or, by submitting a new ticket at [the NERC's Support Ticketing System](https://mghpcc.supportsystem.com/open.php). Thank you, New England Research Cloud (NERC) <https://nerc.mghpcc.org/> <https://nerc-project.github.io/nerc-docs/> .

Duration: 1 hour and 34 minutes

Affected Components: NERC COLDFRONT, MGHPCC SHARED SERVICES (MGHPCC-SS) ACCOUNT PORTAL

Jun 18, 14:33:37 GMT+0
Completed - All upgrade and maintenance is complete now!.

Jun 18, 13:00:01 GMT+0
Identified - Maintenance is now in progress.

Jun 18, 13:00:00 GMT+0
Identified - **NERC’s** [**RegApp**](https://regapp.mss.mghpcc.org/) **&** [**ColdFront**](https://coldfront.mss.mghpcc.org/) **upgrade and maintenance will occur on Tuesday June 18, 2024 from 9:00 AM - 11:00 AM.** **GENERAL MAINTENANCE** \- We are upgrading and updating NERC's RegApp deployment. \- We are upgrading and updating NERC's ColdFront deployment. **NOTICES** \- This upgrade will cause some outages to the NERC’s RegApp & ColdFront service. \- The estimated time to complete this update is **2 hours**. It can take more or less time, so we urge you to keep an eye on <https://nerc.instatus.com/> to get the progress during this time. \- Please do subscribe to <https://nerc.instatus.com/subscribe/email> in order to get the progress updates during this time. Our priority is to help make science happen, so If you or your research team have any questions or need to escalate an issue, please don't hesitate to reach out to us via email ([help@nerc.mghpcc.org](mailto:help@nerc.mghpcc.org)) or, by submitting a new ticket at [the NERC's Support Ticketing System](https://mghpcc.supportsystem.com/open.php). Thank you, New England Research Cloud (NERC) [https://nerc.mghpcc.org/ https://nerc-project.github.io/nerc-docs/](https://nerc.mghpcc.org/https://nerc-project.github.io/nerc-docs/).

Duration: 6 days and 12 hours

Affected Components: NERC COLDFRONT, NERC OPENSTACK COMPUTE SERVICE (NOVA) , OPENSHIFT CONTROLLER, NETWORKING, OPENSTACK IMAGE SERVICE (GLANCE), OPENSTACK EC2 API PROXY (EC2API), STORAGE, OPENSTACK OBJECT STORE (SWIFT), OPENSHIFT WORKERS NODES, NERC OPENSHIFT WEB CONSOLE, OPENSTACK BLOCK STORAGE (CINDER), OPENSTACK DASHBOARD (HORIZON), MGHPCC SHARED SERVICES (MGHPCC-SS) ACCOUNT PORTAL, OPENSTACK NETWORKING SERVICE (NEUTRON)

May 29, 00:00:00 GMT+0
Completed - Maintenance has completed successfully.

May 22, 12:00:01 GMT+0
Identified - Maintenance is now in progress.

May 22, 12:00:00 GMT+0
Identified - Due to the annual MGHPCC (Holyoke data center) and other maintenance activities, New England Research Cloud (NERC) services will be unavailable from **Wednesday, May 22nd, 8 AM until Tuesday, May 28th, 8 PM**. While the timing is not ideal due to the long Memorial Day weekend, it is necessary due to upgrades on related core services such as networking and the Northeast Storage Exchange (NESE). NERC services that will be impacted include: * NERC's RedHat OpenStack * NERC's RedHat OpenShift, which includes the Red Hat OpenShift AI (RHOAI) platform. * NERC supporting services ColdFront and RegApp Due to the long holiday weekend, our goal is to resume full services by **the end of Tuesday, May 28th, at 8 PM** or **earlier**. Please refer to the timeline below for details:  **NOTICES** \- This outage affects all services and resources in the MGHPCC/Holyoke data center which include NERC’s core services and underlying compute, storage and networking. \- The estimated time for power-up and return to full service is **8 PM on Tuesday, May 28th**. We encourage you to monitor the progress at <https://nerc.instatus.com/> during this time. \- Please subscribe to the NERC’s status for any future updates as they happen: <https://nerc.instatus.com/subscribe/email> Our priority is to help make science happen, so If you or your research team have any questions or need to escalate an issue, please contact us via email ([help@nerc.mghpcc.org](mailto:help@nerc.mghpcc.org)) or, by submitting a new ticket at [the NERC's Support Ticketing System](https://mghpcc.supportsystem.com/open.php). Thank you, New England Research Cloud (NERC) [https://nerc.mghpcc.org/ https://nerc.instatus.com/](https://nerc.mghpcc.org/https://nerc.instatus.com/).

Duration: 3 hours

Affected Components: NERC COLDFRONT

May 14, 15:00:00 GMT+0
Completed - Maintenance has completed successfully.

May 14, 14:03:53 GMT+0
Identified - We are extending this maintenance for 1 more hour..

May 14, 12:00:01 GMT+0
Identified - Maintenance is now in progress.

May 14, 12:00:00 GMT+0
Identified - **Upcoming NERC's** [**ColdFront**](https://coldfront.mss.mghpcc.org/) **upgrade and maintenance will occur on Tuesday May 14, 2024 from 8:00 AM – 10:00 AM.** **ENERAL MAINTENANCE** \- We are upgrading and updating NERC's ColdFront deployment. \- This will allow users to request minimal Swift Object Storage during the [decommission process](https://nerc-project.github.io/nerc-docs/openstack/decommission/decommission-openstack-resources/#review-your-object-storageswift-quota). **NOTICES** \- This upgrade will cause some outages to the NERC’s ColdFront service. \- The estimated time to complete this update is **2 hours**. It can take more or less time, so we urge you to keep an eye on <https://nerc.instatus.com/> to get the progress during this time. \- Please do subscribe to <https://nerc.instatus.com/subscribe/email> in order to get the progress updates during this time. Our priority is to help make science happen, so If you or your research team have any questions or need to escalate an issue, please contact us via email ([help@nerc.mghpcc.org](mailto:help@nerc.mghpcc.org)) or, by submitting a new ticket at [the NERC’s Support Ticketing System](https://mghpcc.supportsystem.com/open.php). Thanks, New England Research Cloud (NERC) <https://nerc.mghpcc.org/> <https://nerc-project.github.io/nerc-docs/>.