NERC Status - Incident history

Upcoming NERC system maintenance and upgrade - Tuesday, March 10, 2026 8:00 AM - 5:00 PM

Tue, 10 Mar 2026 12:00:00 +0000

Type: Maintenance

Duration: 10 hours

Affected Components: , , , , ,

Mar 10, 12:00:00 GMT+0
Identified - **Upcoming NERC system maintenance and upgrade will occur on Tuesday, March 10, 2026, 8:00 AM - 5:00 PM.** **GENERAL MAINTENANCE** \- We will be upgrading the [Openshift Container Platform (OCP)](https://nerc-project.github.io/nerc-docs/openshift/) on the **NERC OpenShift Production Cluster** to version **4.20**. \- We are upgrading our [ColdFront](https://coldfront.mss.mghpcc.org/) deployment to support quota management across different storage types. \- We will be adding additional [**H100 to NERC OpenShift Production Cluster**](https://nerc.mghpcc.org/nvidia-h100-gpus-on-nerc/) for general use. **NOTICES** \- The Production Cluster’s OpenShift, OpenShift AI and ColdFront services will be unavailable during this maintenance. \- Please keep a backup of your critical data or application running on your project on NERC’s OpenShift cluster. \- Project allocation requests and change requests for both OpenStack and OpenShift Resources will not be accepted or approved during this time. \- The estimated time to complete this maintenance is **9 hours**. It can take more or less time, so we urge you to keep an eye on for real-time updates on progress. \- Please subscribe to in order to get the progress updates during this time. Our priority is to help make science happen, so If you or your research team have any questions or need to escalate an issue, please contact us via email ([help@nerc.mghpcc.org](mailto:help@nerc.mghpcc.org)) or, by submitting a new ticket at [the NERC's Support Ticketing System](https://mghpcc.supportsystem.com/open.php)..

Mar 10, 12:00:01 GMT+0
Identified - Maintenance is now in progress.

Mar 10, 20:52:07 GMT+0
Identified - Maintenance is now in progress. We will extend this for another one hour..

Mar 10, 22:00:00 GMT+0
Completed - Maintenance has completed successfully.

Unable to get Access to the NERC Shared Services i.e. RegApp, ColdFront, OpenStack and OpenShift Web Consoles

Thu, 5 Mar 2026 15:02:09 +0000

Type: Incident

Duration: 2 hours and 35 minutes

Affected Components: , , , , ,

Mar 5, 17:36:47 GMT+0
Resolved - This incident has been resolved..

Mar 5, 15:02:09 GMT+0
Investigating - We are currently investigating this incident..

Mar 5, 16:47:15 GMT+0
Investigating - We are currently investigating this incident. This is an isolated issue that is affecting Harvard users only!.

Mar 5, 17:18:07 GMT+0
Monitoring - We implemented a fix and are currently monitoring the result..

Upcoming NERC system maintenance and upgrade - Tuesday, February 10, 2026 8:00 AM - 5:00 PM

Tue, 10 Feb 2026 13:00:00 +0000

Type: Maintenance

Duration: 9 hours

Affected Components: , , , , ,

Feb 10, 13:00:00 GMT+0
Identified - **Upcoming NERC system maintenance and upgrade will occur on Tuesday, February 10, 2026 8:00 AM - 5:00 PM.** **GENERAL MAINTENANCE** \- We will be upgrading the [Openshift Container Platform (OCP)](https://nerc-project.github.io/nerc-docs/openshift/) on the NERC OpenShift Production Cluster to version **4.20**. \- We will be upgrading [Red Hat OpenShift AI (RHOAI)](https://nerc-project.github.io/nerc-docs/openshift-ai/) on the NERC OpenShift Production Cluster to version **2.25**. This upgrade will provide new features, bug fixes, and improved support for AI/ML workflows. \- We are upgrading our [ColdFront](https://coldfront.mss.mghpcc.org/) deployment to support quota management across different storage types. **NOTICES** \- The Production Cluster’s OpenShift, OpenShift AI and ColdFront services will be unavailable during this maintenance. \- Please keep a backup of your critical data or application running on your project on NERC’s OpenShift cluster. \- Project allocation requests and change requests for both OpenStack and OpenShift Resources will not be accepted or approved during this time. \- The estimated time to complete this maintenance is 9 hours. It can take more or less time, so we urge you to keep an eye on for real-time updates on progress. \- Please subscribe to in order to get the progress updates during this time. Our priority is to help make science happen, so If you or your research team have any questions or need to escalate an issue, please contact us via email ([help@nerc.mghpcc.org](mailto:help@nerc.mghpcc.org)) or, by submitting a new ticket at [the NERC's Support Ticketing System](https://mghpcc.supportsystem.com/open.php)..

Feb 10, 13:00:01 GMT+0
Identified - Maintenance is now in progress..

Feb 10, 14:46:59 GMT+0
Identified - Maintenance is now in progress. Please note that the planned upgrade to our ColdFront deployment will not be performed today and will be rescheduled for a later date..

Feb 10, 15:01:00 GMT+0
Identified - We are performing **only** the upgrade of [**Red Hat OpenShift AI (RHOAI)**](https://nerc-project.github.io/nerc-docs/openshift-ai/) on the NERC OpenShift Production (OCP-Prod) Cluster today. The upgrades for the [**Openshift Container Platform (OCP)**](https://nerc-project.github.io/nerc-docs/openshift/) and [**ColdFront**](https://coldfront.mss.mghpcc.org/) have been postponed to the next maintenance window..

Feb 10, 22:00:00 GMT+0
Completed - Maintenance has completed successfully.

Upcoming Maintenance and Upgrade for NERC Academic (EDU) OpenShift Cluster Only - Monday, Feb 2, 2026 9:00 AM - 1:00 PM

Mon, 2 Feb 2026 14:00:00 +0000

Type: Maintenance

Duration: 4 hours

Affected Components:

Feb 2, 14:00:00 GMT+0
Identified - **Upcoming Maintenance and Upgrade for NERC**[ **Academic (EDU) OpenShift cluster**](https://console.apps.edu.nerc.mghpcc.org/) **Only - Monday, February 2, 2026 9:00 AM - 1:00 PM.** **GENERAL MAINTENANCE** \- We will be upgrading our[ Red Hat OpenShift AI (RHOAI)](https://nerc-project.github.io/nerc-docs/openshift-ai/) version of **NERC Academic (EDU) OpenShift Cluster** to version **2.22.** This release brings important enhancements to improve performance, security, and the overall user experience. **NOTICES** \- The NERC Academic (EDU) OpenShift Cluster’s OpenShift and OpenShift AI services will be unavailable during this maintenance. \- Please inform all users and students to save any critical data or applications (**Jupyter Notebooks**) running on their workbenches and projects on NERC’s Academic (EDU) OpenShift Cluster. \- Project allocation requests and change requests for NERC Academic (EDU) OpenShift Resources will not be accepted or approved during this time. \- The estimated time to complete this maintenance is **4 hours**. It can take more or less time, so we urge you to keep an eye on[ https://nerc.instatus.com/](https://nerc.instatus.com/) for real-time updates on progress. \- Please subscribe to[ https://nerc.instatus.com/subscribe/email](https://nerc.instatus.com/subscribe/email) in order to get the progress updates during this time. Our priority is to help make science happen, so If you or your research team have any questions or need to escalate an issue, please contact us via email ([help@nerc.mghpcc.org](mailto:help@nerc.mghpcc.org)) or, by submitting a new ticket at[the NERC's Support Ticketing System](https://mghpcc.supportsystem.com/open.php)..

Feb 2, 14:00:01 GMT+0
Identified - Maintenance is now in progress.

Feb 2, 18:00:00 GMT+0
Completed - Maintenance has completed successfully.

Upcoming NERC system maintenance and upgrade - Tuesday, Jan 13, 2026 8:00 AM - 12:00 PM

Tue, 13 Jan 2026 13:00:00 +0000

Type: Maintenance

Duration: 3 hours and 2 minutes

Affected Components: , , ,

Jan 13, 13:00:00 GMT+0
Identified - **Upcoming NERC system maintenance and upgrade will occur on Tuesday, January 13, 2026 8:00 AM - 12:00 PM.** **GENERAL MAINTENANCE** \- We are upgrading NERC’s [ColdFront](https://coldfront.mss.mghpcc.org/) Plugin to the latest version. \- We will be adding **8 H100 GPUs** on the [NERC OpenShift](https://nerc-project.github.io/nerc-docs/openshift/) Production cluster. \- We will be adding more **NESE Storage Pools** to increase the current storage quota. **NOTICES** \- This upgrade will cause some outages to the [NERC’s ColdFront service](https://nerc-project.github.io/nerc-docs/get-started/allocation/coldfront/). \- The estimated time to complete this maintenance is **4 hours**. It can take more or less time, so we urge you to keep an eye on for real-time updates on progress. \- Please subscribe to in order to get the progress updates during this time. Our priority is to help make science happen, so If you or your research team have any questions or need to escalate an issue, please contact us via email ([help@nerc.mghpcc.org](mailto:help@nerc.mghpcc.org)) or, by submitting a new ticket at [the NERC's Support Ticketing System](https://mghpcc.supportsystem.com/open.php)..

Jan 13, 13:00:01 GMT+0
Identified - Maintenance is now in progress.

Jan 13, 16:02:23 GMT+0
Completed - Maintenance has completed successfully..

Unable to Access NERC OpenStack Object Storage (Console/API) or Modify Storage Quota via ColdFront

Fri, 9 Jan 2026 20:36:55 +0000

Type: Incident

Duration: 3 days, 19 hours and 25 minutes

Affected Components: , ,

Jan 9, 20:36:55 GMT+0
Investigating - We are currently investigating this incident..

Jan 12, 14:17:21 GMT+0
Identified - We are continuing to work on a fix for this incident. We have informed users who are hosting large amount of fragmented objects and begin cleaning up some..

Jan 13, 16:01:52 GMT+0
Resolved - This incident has been resolved..

OTHER WEBSITES & TOOLS - NERC WEBSITE isn't accessible at the moment (automated)

Thu, 11 Dec 2025 15:20:31 +0000

Type: Incident

Duration: 5 minutes

Affected Components:

Dec 11, 15:20:31 GMT+0
Investigating - OTHER WEBSITES & TOOLS - NERC WEBSITE cannot be accessed at the moment. This incident was created by an automated monitoring service..

Dec 11, 15:25:57 GMT+0
Resolved - OTHER WEBSITES & TOOLS - NERC WEBSITE is now operational! This update was created by an automated monitoring service..

Upcoming NERC system maintenance and upgrade - Tuesday, Dec 9, 2025 8:00 AM - 12:00 PM

Tue, 9 Dec 2025 13:00:00 +0000

Type: Maintenance

Duration: 3 hours and 56 minutes

Nov 11, 13:00:01 GMT+0
Identified - Maintenance is now in progress.

OTHER WEBSITES & TOOLS - NERC WEBSITE isn't accessible at the moment (automated)

Wed, 5 Nov 2025 09:33:19 +0000

Type: Incident

Duration: 5 minutes

Affected Components:

Nov 5, 09:33:19 GMT+0
Investigating - OTHER WEBSITES & TOOLS - NERC WEBSITE cannot be accessed at the moment. This incident was created by an automated monitoring service..

Nov 5, 09:38:27 GMT+0
Resolved - OTHER WEBSITES & TOOLS - NERC WEBSITE is now operational! This update was created by an automated monitoring service..

OTHER WEBSITES & TOOLS - NERC WEBSITE isn't accessible at the moment (automated)

Fri, 31 Oct 2025 08:07:55 +0000

Type: Incident

Duration: 28 minutes

Affected Components:

Oct 31, 08:07:55 GMT+0
Investigating - OTHER WEBSITES & TOOLS - NERC WEBSITE cannot be accessed at the moment. This incident was created by an automated monitoring service..

Oct 31, 08:35:43 GMT+0
Resolved - OTHER WEBSITES & TOOLS - NERC WEBSITE is now operational! This update was created by an automated monitoring service..

OTHER WEBSITES & TOOLS - NERC WEBSITE isn't accessible at the moment (automated)

Fri, 24 Oct 2025 05:57:11 +0000

Type: Incident

Duration: 22 minutes

Affected Components:

Oct 24, 05:57:11 GMT+0
Investigating - OTHER WEBSITES & TOOLS - NERC WEBSITE cannot be accessed at the moment. This incident was created by an automated monitoring service..

Oct 24, 06:19:10 GMT+0
Resolved - OTHER WEBSITES & TOOLS - NERC WEBSITE is now operational! This update was created by an automated monitoring service..

Upcoming NERC system maintenance and upgrade - Tuesday, October 14, 2025 8:00 AM - 5:00 PM.

Tue, 14 Oct 2025 12:00:00 +0000

Type: Maintenance

Duration: 9 hours

Affected Components: ,

Oct 14, 21:00:00 GMT+0
Completed - Maintenance has completed successfully.

Oct 14, 12:00:01 GMT+0
Identified - Maintenance is now in progress.

Oct 14, 12:00:00 GMT+0
Identified - Upcoming NERC system maintenance and upgrade will occur on Tuesday, October 14, 2025 8:00 AM - 5:00 PM. **GENERAL MAINTENANCE** \- We will be upgrading our [Red Hat OpenShift AI (RHOAI)](https://nerc-project.github.io/nerc-docs/openshift-ai/) version of NERC OpenShift Production Cluster to version **2.22.** This release brings important enhancements to improve performance, security, and the overall user experience. \- We are upgrading our [ColdFront](https://coldfront.mss.mghpcc.org/) deployment. **NOTICES** \- The **OpenShift AI and ColdFront** services will be unavailable during this maintenance. \- Please keep a backup of your critical data or application running on your project on NERC’s OpenShift cluster. \- Project allocation requests and change requests for both OpenStack and OpenShift Resources **will not be accepted or approved** during this time. \- The estimated time to complete this maintenance is **9 hours**. It can take more or less time, so we urge you to keep an eye on for real-time updates on progress. \- Please subscribe to in order to get the progress updates during this time. Our priority is to help make science happen, so If you or your research team have any questions or need to escalate an issue, please contact us via email ([help@nerc.mghpcc.org](mailto:help@nerc.mghpcc.org)) or, by submitting a new ticket at [the NERC's Support Ticketing System](https://mghpcc.supportsystem.com/open.php). Thank you, New England Research Cloud (NERC) [https://nerc.mghpcc.org/ ](https://nerc.mghpcc.org/) .

OTHER WEBSITES & TOOLS - NERC WEBSITE isn't accessible at the moment (automated)

Sun, 5 Oct 2025 20:37:32 +0000

Type: Incident

Duration: 28 minutes

Affected Components:

Oct 5, 20:37:32 GMT+0
Investigating - OTHER WEBSITES & TOOLS - NERC WEBSITE cannot be accessed at the moment. This incident was created by an automated monitoring service..

Oct 5, 21:05:18 GMT+0
Resolved - OTHER WEBSITES & TOOLS - NERC WEBSITE is now operational! This update was created by an automated monitoring service..

Unplanned Maintenance to upgrade the RedHat OpenShift AI operator to 2.19.3 in order to fix CVE-2025-10725

Fri, 3 Oct 2025 15:30:32 +0000

Type: Incident

Duration: 7 minutes

Affected Components: , , , ,

Oct 3, 15:30:32 GMT+0
Monitoring - We are performing an unplanned maintenance to upgrade the RedHat OpenShift AI operator to 2.19.3 in order to fix CVE-2025-10725.

Oct 3, 15:37:46 GMT+0
Resolved - This unplanned maintenance has been completed..

Upcoming NERC system maintenance and upgrade - Tuesday, September 9, 2025 8:00 AM - 5:00 PM.

Tue, 9 Sep 2025 12:00:00 +0000

Type: Maintenance

Duration: 9 hours

Affected Components: , , , , ,

Sep 9, 12:00:00 GMT+0
Identified - **Upcoming NERC system maintenance and upgrade will occur on Tuesday, September 9, 2025 8:00 AM - 5:00 PM.** **GENERAL MAINTENANCE** \- We will be upgrading our Red Hat [OpenShift container platform (OCP)](https://nerc-project.github.io/nerc-docs/openshift/) version on NERC to version **4.19.** This release brings important enhancements to improve performance, security, and the overall user experience. \- **Unified Console Experience**: The Admin and Developer views are now combined into one interface, reducing context switching and enabling seamless end-to-end workflows. \- We have also set up and now offer an isolated [Academic (EDU) OpenShift cluster](https://console.apps.edu.nerc.mghpcc.org/) designed to fully support academic research and coursework, providing a stable and dedicated environment for instructors, principal investigators, students, and other users. If you are **planning to run courses on NERC** in **Fall 2025** or **Spring 2026**, please contact us so we can coordinate and provide the necessary support. **NOTICES** \- The OpenShift services will be unavailable during this maintenance. \- Please keep a backup of your critical data or application running on your project on NERC’s OpenShift cluster. \- Project allocation requests and change requests for both OpenStack and OpenShift Resources **will not be accepted or approved** during this time. \- The estimated time to complete this maintenance is **9 hours**. It can take more or less time, so we urge you to keep an eye on for real-time updates on progress. \- Please subscribe to in order to get the progress updates during this time. Our priority is to help make science happen, so If you or your research team have any questions or need to escalate an issue, please contact us via email ([help@nerc.mghpcc.org](mailto:help@nerc.mghpcc.org)) or, by submitting a new ticket at [the NERC's Support Ticketing System](https://mghpcc.supportsystem.com/open.php). Thank you, New England Research Cloud (NERC) [https://nerc.mghpcc.org/ ](https://nerc.mghpcc.org/).

Sep 9, 12:00:01 GMT+0
Identified - Maintenance is now in progress.

Sep 9, 21:00:00 GMT+0
Completed - Maintenance has completed successfully.

Sep 9, 12:15:09 GMT+0
Identified - Maintenance is now in progress. We will also conduct NERC ColdFront upgrade and maintenance during this timeline..

OTHER WEBSITES & TOOLS - NERC WEBSITE isn't accessible at the moment (automated)

Tue, 2 Sep 2025 12:15:17 +0000

Type: Incident

Duration: 11 minutes

Affected Components:

Sep 2, 12:15:17 GMT+0
Investigating - OTHER WEBSITES & TOOLS - NERC WEBSITE cannot be accessed at the moment. This incident was created by an automated monitoring service..

Sep 2, 12:25:59 GMT+0
Resolved - OTHER WEBSITES & TOOLS - NERC WEBSITE is now operational! This update was created by an automated monitoring service..

OTHER WEBSITES & TOOLS - NERC WEBSITE isn't accessible at the moment (automated)

Thu, 28 Aug 2025 11:26:57 +0000

Type: Incident

Duration: 1 hour and 57 minutes

Affected Components:

Aug 28, 11:26:57 GMT+0
Investigating - OTHER WEBSITES & TOOLS - NERC WEBSITE cannot be accessed at the moment. This incident was created by an automated monitoring service..

Aug 28, 13:23:32 GMT+0
Resolved - OTHER WEBSITES & TOOLS - NERC WEBSITE is now operational! This update was created by an automated monitoring service..

Upcoming NERC system maintenance and upgrade will occur on Tuesday August 12, 2025 8:00 AM - 5:00 PM.

Tue, 12 Aug 2025 12:00:00 +0000

Type: Maintenance

Duration: 9 hours

Affected Components: , , , , ,

Aug 12, 21:00:00 GMT+0
Completed - Maintenance has completed successfully.

Aug 12, 12:00:00 GMT+0
Identified - **Upcoming NERC system maintenance and upgrade will occur on Tuesday August 12, 2025 8:00 AM - 5:00 PM.** **GENERAL MAINTENANCE** \- We will be upgrading our Red Hat [OpenShift container platform (OCP)](https://nerc-project.github.io/nerc-docs/openshift/) version on NERC to **4.18**. \- As part of the upcoming maintenance, we’ll be rolling out several key enhancements to [Red Hat OpenShift AI (RHOAI)](https://nerc-project.github.io/nerc-docs/openshift-ai/). These include a redesigned dashboard layout, support for both single and multi-model serving, and distributed training capabilities using CodeFlare and KubeRay. The platform will also support high-performance inference runtimes beyond the currently offered OpenVINO, such as [vLLM](https://www.redhat.com/en/topics/ai/what-is-vllm). Additional improvements include a [GPU pruner](https://ai-on-openshift.io/odh-rhoai/gpu-pruner/) for more efficient GPU utilization, enhanced resource scheduling with [Kueue](https://kueue.sigs.k8s.io/), and an upgraded version of Kubeflow Pipelines. This update will further strengthen MLOps capabilities with an integrated model registry and catalog to manage the full model lifecycle. \- We will be upgrading our [ColdFront](https://coldfront.mss.mghpcc.org/) deployment. **NOTICES** \- The OpenShift and ColdFront services will be unavailable during this maintenance. \- Please keep a backup of your critical data or application running on your project on NERC’s OpenShift cluster. \- Project allocation requests and change requests for both OpenStack and OpenShift Resources **will not be accepted or approved** during this time. \- The estimated time to complete this maintenance is **9 hours**. It can take more or less time, so we urge you to keep an eye on for real-time updates on progress. \- Please subscribe to in order to get the progress updates during this time. Our priority is to help make science happen, so If you or your research team have any questions or need to escalate an issue, please contact us via email ([help@nerc.mghpcc.org](mailto:help@nerc.mghpcc.org)) or, by submitting a new ticket at [the NERC's Support Ticketing System](https://mghpcc.supportsystem.com/open.php). Thank you, New England Research Cloud (NERC) [https://nerc.mghpcc.org/ ](https://nerc.mghpcc.org/).

Aug 12, 12:00:01 GMT+0
Identified - Maintenance is now in progress.

OTHER WEBSITES & TOOLS - NERC WEBSITE isn't accessible at the moment (automated)

Fri, 20 Jun 2025 13:14:16 +0000

Type: Incident

Duration: 5 hours and 9 minutes

Affected Components:

Jun 20, 13:14:16 GMT+0
Investigating - OTHER WEBSITES & TOOLS - NERC WEBSITE cannot be accessed at the moment. This incident was created by an automated monitoring service..

Jun 20, 13:19:25 GMT+0
Resolved - OTHER WEBSITES & TOOLS - NERC WEBSITE is now operational! This update was created by an automated monitoring service..

OTHER WEBSITES & TOOLS - NERC WEBSITE isn't accessible at the moment (automated)

Fri, 20 Jun 2025 12:30:13 +0000

Type: Incident

Duration: 10 hours and 18 minutes

Affected Components:

Jun 20, 12:30:13 GMT+0
Investigating - OTHER WEBSITES & TOOLS - NERC WEBSITE cannot be accessed at the moment. This incident was created by an automated monitoring service..

Jun 20, 12:40:30 GMT+0
Resolved - OTHER WEBSITES & TOOLS - NERC WEBSITE is now operational! This update was created by an automated monitoring service..

Unplanned Red Hat's Production OpenShift Cluster Platform (OCP) Maintenance on NERC Wednesday, June 18, 2025, from 9 AM to 12 PM

Wed, 18 Jun 2025 13:00:00 +0000

Type: Maintenance

Duration: 2 hours

Affected Components: , , , ,

Jun 18, 13:00:01 GMT+0
Identified - Maintenance is now in progress.

Jun 18, 13:00:00 GMT+0
Identified - We need to perform rolling reboots of all nodes on our [OpenShift cluster](https://nerc-project.github.io/nerc-docs/openshift/) to adjust the cluster MTU setting (from 8900 to 1400) and rectify downstream issues. This was inadvertently updated during last week’s cluster version upgrade to 4.17. This maintenance will cause running workloads to be interrupted and relaunched..

Jun 18, 15:00:21 GMT+0
Completed - Maintenance has completed successfully..

NERC OpenShift (OCP) Cluster is not fully upgraded and functional at this moment.

Wed, 11 Jun 2025 17:44:44 +0000

Type: Incident

Duration: 1 day, 3 hours and 27 minutes

Affected Components: , , , ,

Jun 11, 17:44:44 GMT+0
Investigating - We are currently investigating this incident..

Jun 12, 21:12:08 GMT+0
Resolved - This incident has been resolved. OpenShift is fully usable with one exception. The H100s are not yet ready for use..

Upcoming Red Hat's OpenShift container platform (OCP) upgrade on NERC from version 4.15 to 4.17 will occur on Tuesday, June 10, 2025, from 8:00 AM – 5:00 PM.

Tue, 10 Jun 2025 12:00:00 +0000

Type: Maintenance

Duration: 13 hours

Affected Components: , , , ,

Jun 10, 12:00:01 GMT+0
Identified - Maintenance is now in progress.

Jun 10, 20:51:06 GMT+0
Identified - Maintenance is still in progress. We are extending the maintenance for another 4 hours..

Jun 11, 01:00:00 GMT+0
Completed - Maintenance has completed successfully.

Jun 10, 12:00:00 GMT+0
Identified - **Upcoming Red Hat's** [**OpenShift container platform (OCP)**](http://%28https//nerc-project.github.io/nerc-docs/openshift/) **upgrade on NERC from version 4.15 to 4.17 will occur on Tuesday, June 10, 2025, from 8:00 AM – 5:00 PM.** **GENERAL MAINTENANCE** We are writing to inform you that we plan to upgrade our [OpenShift cluster](https://nerc-project.github.io/nerc-docs/openshift/) from **version 4.15 to 4.17** on **Tuesday, June 10, 2025**. As our current version of OpenShift will reach its End of Life (EOL) in less than three months, this upgrade is essential and will introduce several new features and improvements to our cluster. Additionally, we will upgrade and maintain the [Red Hat OpenShift AI Platform (RHOAI)](https://nerc-project.github.io/nerc-docs/openshift-ai/) to **version 2.19**, which provides a fully supported, collaborative sandbox environment for data scientists, AI engineers, and researchers to develop, train, and test AI/ML models, as well as deploy them for use in intelligent applications. **NOTICES** \- Please keep a backup of your critical data or application running on your project on NERC's OpenShift cluster. \- The new upgrade to the [latest OpenShift version](https://www.redhat.com/en/whats-new-red-hat-openshift) will provide improved performance, enhanced security, additional operators and enhanced user interface along with updated Kubernetes version. \- During the upgrade, there will be a period of downtime to ensure a seamless transition. We anticipate this downtime to last **approximately 9 hours**, and we apologize for any inconvenience this may cause. It can take more or less time, so we urge you to keep an eye on and also subscribe to to get the progress updates during this time. \- Our state-of-the-art [OpenShift](https://nerc-project.github.io/nerc-docs/openshift/gpus/intro-to-gpus-on-nerc-ocp/) and [RHOAI](https://nerc-project.github.io/nerc-docs/openshift-ai/other-projects/predictive-and-generative-AI/) platforms enable you and your team to deploy containerized applications in a cloud-native environment, providing a reliable, isolated, and scalable solution for your complex research computing and teaching needs. If you haven't used or tested these platforms, please get started by requesting a new resource allocation to your project using [NERC's ColdFront web console](https://nerc-project.github.io/nerc-docs/get-started/get-an-allocation/#request-a-new-openshift-resource-allocation-for-openshift-project) or you can get in touch with us to have a quick demo. More information about NERC is available on [NERC's website](https://nerc.mghpcc.org/). If you have any questions, please don't hesitate to reach out to us via email ([help@nerc.mghpcc.org](mailto:help@nerc.mghpcc.org)) or by submitting a new ticket at the [NERC's Support Ticketing System](https://mghpcc.supportsystem.com/open.php). Thanks, New England Research Cloud (NERC) .

Upcoming MGHPCC (Holyoke) Annual Power Shutdown [June 2, 2025 3:00 PM ET - June 4, 2025 5:00 PM ET]

Mon, 2 Jun 2025 19:00:00 +0000

Type: Maintenance

Duration: 2 days and 2 hours

Affected Components: , , , , , , , , , , , , ,

Jun 2, 19:00:01 GMT+0
Identified - Maintenance is now in progress.

Jun 2, 19:00:00 GMT+0
Identified - Upcoming MGHPCC (Holyoke) Annual Power Shutdown \[June 2, 2025 3:00 PM ET - June 4, 2025 5:00 PM ET\] (multi-day) Due to the annual MGHPCC (Holyoke data center) power maintenance as well as other maintenance activities, the New England Research Cloud (NERC) services will be unavailable from **Monday, June 2nd, 3 PM until Wednesday, June 4th, 5 PM**. NERC services that will be impacted include: * NERC's OpenStack * NERC's OpenShift, which includes the Red Hat OpenShift AI (RHOAI) platform * NERC supporting shared services ColdFront, RegApp, ESI, and Keycloak We have worked to reduce the total outage time this year. Our goal is to resume full services by **the end of Wednesday, June 4th, at 5 PM** or **earlier**. Please refer to the timeline below for details: !\[Maintenance Overview\]() **NOTICES** **\---------** \- This outage affects all services and resources in the MGHPCC/Holyoke data center, including NERC’s core and shared services, as well as the underlying compute, storage, and networking. \- The estimated time for power-up and return to full service is **5 PM on Wednesday, June 4th**. We encourage you to monitor the progress at during this time. \- Please subscribe to the NERC’s status for any future updates as they happen: Our priority is helping make research happen. If you or your research team have any questions or need to escalate an issue, please do not hesitate to contact us via email ([help@nerc.mghpcc.org](mailto:help@nerc.mghpcc.org)) or by submitting a service request through [the NERC's Support Ticketing System](https://mghpcc.supportsystem.com/open.php). Thanks, New England Research Cloud (NERC) .

Jun 4, 21:00:00 GMT+0
Completed - Maintenance has completed successfully.

Upcoming NERC system maintenance and upgrade - [Tuesday, May 13, 2025 8:00 AM - 5:00 PM]

Tue, 13 May 2025 12:00:00 +0000

Type: Maintenance

Duration: 9 hours

Affected Components: , , , , , , , , , , ,

May 13, 12:00:01 GMT+0
Identified - Maintenance is now in progress.

May 13, 21:00:00 GMT+0
Completed - Maintenance has completed successfully.

May 13, 12:00:00 GMT+0
Identified - Upcoming NERC system maintenance and upgrade - Tuesday, May 13, 2025 8:00 AM - 5:00 PM. **GENERAL MAINTENANCE** **\------------------------------** \- We will resolve issues related to Shelving VMs to save on resource costs when they are not in use. \- We will conduct maintenance on the [NERC OpenStack RadosGW (Swift/S3) object storage](https://nerc-project.github.io/nerc-docs/openstack/persistent-storage/object-storage/) service. \- Additionally, we will deploy the [H100 GPU](https://nerc.mghpcc.org/nvidia-h100-gpus-on-nerc/) configuration on [NERC OpenShift](https://nerc-project.github.io/nerc-docs/openshift/). \- Users will be able to add external hostname (DNS) to their ingress with TLS/SSL certificates from Let's Encrypt as [described here](https://nerc-project.github.io/nerc-docs/openshift/domain-name-system/domain-name-for-your-application/). **NOTICES** **\----------** \- The OpenStack and OpenShift services will be unavailable during this maintenance. \- Project allocation requests and change requests for both OpenStack and OpenShift Resources **will not be accepted or approved** during this time. \- The estimated time to complete this maintenance is **9 hours**. It can take more or less time, so we urge you to keep an eye on for real-time updates on progress. \- Please do subscribe to in order to get the progress updates during this time. Our priority is to help make science happen, so If you or your research team have any questions or need to escalate an issue, please contact us via email ([help@nerc.mghpcc.org](mailto:help@nerc.mghpcc.org)) or, by submitting a new ticket at [the NERC's Support Ticketing System](https://mghpcc.supportsystem.com/open.php). Thank you, New England Research Cloud (NERC) [https://nerc.mghpcc.org/ https://nerc-project.github.io/nerc-docs/](https://nerc.mghpcc.org/https://nerc-project.github.io/nerc-docs/).

Expired KeyCloak SSL Certificate - Unable to access the cluster both NERC OpenStack and OpenShift

Mon, 12 May 2025 13:23:47 +0000

Type: Incident

Duration: 3 hours and 7 minutes

Affected Components: , , , , , , , , ,

May 12, 16:30:50 GMT+0
Resolved - This incident has been resolved..

May 12, 13:23:47 GMT+0
Investigating - We are currently investigating this incident..

NERC COLDFRONT isn't accessible at the moment (automated)

Mon, 28 Apr 2025 13:43:14 +0000

Type: Incident

Duration: 16 hours and 12 minutes

Affected Components:

Apr 28, 13:43:14 GMT+0
Investigating - NERC COLDFRONT cannot be accessed at the moment. This incident was created by an automated monitoring service..

Apr 28, 13:59:26 GMT+0
Resolved - NERC COLDFRONT is now operational! This update was created by an automated monitoring service..

MGHPCC SHARED SERVICES (MGHPCC-SS) ACCOUNT PORTAL isn't accessible at the moment (automated)

Mon, 28 Apr 2025 13:42:06 +0000

Type: Incident

Duration: 16 hours and 14 minutes

Affected Components:

Apr 28, 13:42:06 GMT+0
Investigating - MGHPCC SHARED SERVICES (MGHPCC-SS) ACCOUNT PORTAL cannot be accessed at the moment. This incident was created by an automated monitoring service..

Apr 28, 13:58:20 GMT+0
Resolved - MGHPCC SHARED SERVICES (MGHPCC-SS) ACCOUNT PORTAL is now operational! This update was created by an automated monitoring service..

Sunday 04/27/2025- Unexpected Overnight Compute Node Outage at MGHPCC Due to Chiller/Power Issue

Sun, 27 Apr 2025 05:05:00 +0000

Type: Incident Affected Components: NERC OPENSTACK COMPUTE SERVICE (NOVA) , OPENSHIFT CONTROLLER, NETWORKING, OPENSTACK IMAGE SERVICE (GLANCE), OPENSTACK EC2 API PROXY (EC2API), OPENSTACK OBJECT STORE (SWIFT), OPENSHIFT WORKERS NODES, NERC OPENSHIFT WEB CONSOLE, OPENSTACK BLOCK STORAGE (CINDER), OPENSTACK DASHBOARD (HORIZON), OPENSTACK NETWORKING SERVICE (NEUTRON) Apr 27, 05:05:00 GMT+0 - Resolved - There was a chiller issue at the MGHPCC overnight that caused all non-UPS power to be shut off (at 1:05AM on Sunday 04/27/2025). The chiller has been fixed and non-UPS power was restored at 5:43am. We have manually powered on all affected nodes. So if you still facing any issues please let us know via email help@nerc.mghpcc.org. This incident has been resolved.

Upcoming NERC system maintenance and upgrade - OpenStack and ColdFront - [Tuesday February 11, 2025 9:00 AM - 11:00 AM]

Tue, 11 Feb 2025 14:00:00 +0000

Type: Maintenance

Duration: 1 hour and 34 minutes

Affected Components: ,

Feb 11, 14:00:00 GMT+0
Identified - Upcoming NERC system maintenance and upgrade will occur on Tuesday February 11, 2025 9:00 AM - 11:00 AM. **GENERAL MAINTENANCE** \- We will be upgrading our [ColdFront](https://coldfront.mss.mghpcc.org/) deployment. Additionally, we will be performing maintenance on the NERC [OpenStack RadosGW (Swift/S3) object storage service](https://nerc-project.github.io/nerc-docs/openstack/persistent-storage/object-storage/). \- Other than OpenStack projects which use Swift/S3 this maintenance should not affect any other projects using the NERC. **NOTICES** \- The OpenStack RadosGW (Swift/S3) object storage service will be unavailable during this maintenance. \- Project allocation requests and change requests **will not be accepted or approved** during this time. \- The maintenance and upgrade are expected to be brief. We encourage you to monitor for real-time updates on progress. \- Please subscribe to in order to get the progress updates during this time. Our priority is to help make science happen, so If you or your research team have any questions or need to escalate an issue, please contact us via email ([help@nerc.mghpcc.org](mailto:help@nerc.mghpcc.org)) or, by submitting a new ticket at [the NERC's Support Ticketing System](https://mghpcc.supportsystem.com/open.php). Thank you, New England Research Cloud (NERC) [https://nerc.mghpcc.org/ https://nerc-project.github.io/nerc-docs/](https://nerc.mghpcc.org/https://nerc-project.github.io/nerc-docs/).

Feb 11, 15:34:13 GMT+0
Completed - We have successfully completed the maintenance and upgrade!.

Feb 11, 14:00:01 GMT+0
Identified - Maintenance is now in progress.

Upcoming NERC Network Equipment and Switch Maintenance Scheduled for [Tuesday January 14, 2025 9 AM - Wednesday January 15, 2025 12 PM]

Tue, 14 Jan 2025 14:00:00 +0000

Type: Maintenance

Duration: 1 day and 3 hours

Affected Components: , , , , , , , , , , ,

Jan 14, 14:00:00 GMT+0
Identified - We want to inform you about an update to the previously announced maintenance schedule. The maintenance activity has been pushed back by one week. Please find the revised details here: .

Jan 15, 14:53:54 GMT+0
Identified - We need to extend the maintenance for 3 more hours. We will update you as it progress..

Jan 14, 14:00:00 GMT+0
Identified - Upcoming NERC Network Equipment and Switch Maintenance is scheduled from Tuesday, January 14, 2025, at 9:00 AM to Wednesday, January 15, 2025, at 9:00 AM. **GENERAL MAINTENANCE** \- We plan to perform changes to the management and configuration of our network infrastructure. This will take the majority of the maintenance period. These updates are part of our ongoing efforts to improve network performance, reliability, and scalability for all users. \- During the maintenance window, we will also update the NERC OpenShift deployment to include patches for the NVIDIA A100 and V100 GPU Accelerator profiles. **NOTICES** \- [NERC OpenStack’s Horizon dashboard](https://stack.nerc.mghpcc.org/) and other [OpenStack API](https://nerc-project.github.io/nerc-docs/openstack/openstack-cli/openstack-CLI/) services will be inaccessible. \- During this process, Access to the [NERC OpenShift cluster](https://nerc-project.github.io/nerc-docs/openshift/) and [NERC Red Hat OpenShift AI (RHOAI)](https://nerc-project.github.io/nerc-docs/openshift-ai/) via the web console and other OpenShift CLI tools will be unavailable. \- Although we do not anticipate that ESI-provided systems will be affected, except for those connected to the NERC network, there is a slight risk due to the tasks we will be performing in their general vicinity. \- All services will be available once the OpenShift Deployment portion of this maintenance is completed, and we will post updates to the status page. \- Please be aware that during this maintenance, project change requests for both OpenStack and OpenShift Resources **will not be accepted or approved**. \- The estimated time to complete this maintenance is **24 hours**. It can take more or less time, so we urge you to keep an eye on to receive updated progress. \- Please subscribe to to receive progress updates. \- Please let us know if you encounter any issues after the maintenance is completed. Our priority is to help make science happen, if you or your research team have any questions or need to escalate an issue, please contact us via email ([help@nerc.mghpcc.org](mailto:help@nerc.mghpcc.org)) or, by submitting a new ticket at [the NERC's Support Ticketing System](https://mghpcc.supportsystem.com/open.php). Thank you, New England Research Cloud (NERC) [https://nerc.mghpcc.org/ https://nerc-project.github.io/nerc-docs/](https://nerc.mghpcc.org/https://nerc-project.github.io/nerc-docs/).

Jan 15, 17:00:00 GMT+0
Completed - Maintenance has completed successfully.

Jan 14, 14:00:01 GMT+0
Identified - Maintenance is now in progress.

Upgrade NERC OpenStack Platform to version 17.1 [Dec 12, 2024 8:00 AM ET - Dec 14, 2024 8:00 PM ET] (multi-day)

Thu, 12 Dec 2024 13:00:00 +0000

Type: Maintenance

Duration: 2 days and 12 hours

Affected Components: , , , , , ,

Dec 12, 13:00:00 GMT+0
Identified - Upcoming [NERC OpenStack Platform](https://nerc-project.github.io/nerc-docs/openstack/) upgrade to version 17.1 will occur from Thursday December 12th, 2024 from 8:00 AM until Saturday, December 14th, 8:00 PM. **GENERAL MAINTENANCE** \- We are planning to upgrade our NERC OpenStack Platform to version 17.1\. This upgrade will provide enhanced features, performance improvements, and additional security & compliance measures to support your work and ensure a reliable cloud experience. This major upgrade has been long-pending, so we are reserving a 3-day window for this process to ensure a smooth transition. \- The [NERC OpenShift Container Platform (OCP)](https://console.apps.shift.nerc.mghpcc.org/) will remain available during this time. **NOTICES** \- We recommend saving your work and preparing for the downtime in advance.\- NERC [OpenStack’s Horizon dashboard](https://stack.nerc.mghpcc.org/) and other [OpenStack API](https://nerc-project.github.io/nerc-docs/openstack/openstack-cli/openstack-CLI/) services may be inaccessible at times during this process. \- This upgrade process should not affect the state of your VMs, persistent storage, or data stored in volumes and object storage. \- Please be aware that during this maintenance, project, and project change requests for NERC OpenStack Resources **will not be accepted and approved**. \- We are reserving **the three days**, as this is a major and long pending upgrade to our cluster. It can take more or less time, so we urge you to keep an eye on to get the progress during this time. \- Please do subscribe to in order to get the progress updates during this time. Our priority is to help make science happen, so If you or your research team have any questions or need to escalate an issue, please contact us via email ([help@nerc.mghpcc.org](mailto:help@nerc.mghpcc.org)) or, by submitting a new ticket at [the NERC's Support Ticketing System](https://mghpcc.supportsystem.com/open.php). Thank you, New England Research Cloud (NERC) [https://nerc.mghpcc.org/ https://nerc-project.github.io/nerc-docs/](https://nerc.mghpcc.org/https://nerc-project.github.io/nerc-docs/).

Dec 15, 01:00:00 GMT+0
Completed - Maintenance has completed successfully.

Dec 12, 13:00:01 GMT+0
Identified - Maintenance is now in progress.

NERC OpenStack, ColdFront service, and OpenShift Account Manager Maintenance Scheduled for [September 3, 2024 9AM - 5PM]

Tue, 3 Sep 2024 13:00:00 +0000

Type: Maintenance

Duration: 8 hours

Affected Components: , , , , , , ,

Sep 3, 13:00:01 GMT+0
Identified - Maintenance is now in progress.

Sep 3, 13:00:00 GMT+0
Identified - Upcoming NERC[ OpenStack Platform](https://stack.nerc.mghpcc.org/),[ ColdFront service](https://coldfront.mss.mghpcc.org/), and OpenShift Account Manager maintenance will occur on Tuesday September 3, 2024 from 9:00 AM – 5:00 PM. **GENERAL MAINTENANCE** We are planning to migrate a set of GPU hypervisors (V100) from the NERC OpenStack cluster to the NERC OpenShift cluster. We will also be upgrading both the ColdFront allocation system and the OpenShift account management software. This maintenance is scheduled for Tuesday September 3, 2024 from 9:00 AM – 5:00 PM. **NOTICES** \- We will be moving some of the GPU OpenStack hypervisors (V100) to our [OpenShift Container Platform (OCP)](https://console.apps.shift.nerc.mghpcc.org/) cluster. \- Some OpenStack APIs may briefly experience outages, and any active object storage transfers will be interrupted. \- Please do not launch any new VMs or initiate data transfers to/from existing VMs during this time. \- We will be upgrading the ColdFront allocation system and also OpenShift account management software. All OpenShift APIs and workloads will be available and functional during this portion of the maintenance. \- Please be aware that during this maintenance, project and project change requests (Related to the OpenStack Resource Only) **will not be accepted and approved**. \- We are reserving **the whole day**, from 9am to 5pm. It can take more or less time, so we urge you to keep an eye on to get the progress during this time. We will post updates to our status page when the ColdFront and OpenShift Account Manager portion of this maintenance has been completed. \- Please do subscribe to in order to get the progress updates during this time. Our priority is to help make science happen, so If you or your research team have any questions or need to escalate an issue, please contact us via email ([help@nerc.mghpcc.org](mailto:help@nerc.mghpcc.org)) or, by submitting a new ticket at [the NERC's Support Ticketing System](https://mghpcc.supportsystem.com/open.php). Thank you, New England Research Cloud (NERC) [https://nerc.mghpcc.org/ https://nerc-project.github.io/nerc-docs/](https://nerc.mghpcc.org/https://nerc-project.github.io/nerc-docs/).

Sep 3, 21:00:00 GMT+0
Completed - Maintenance has completed successfully.

NERC OpenShift Container Platform (OCP) and ColdFront service maintenance and upgrade [Monday August 5, 2024 9 AM - 5 PM]

Mon, 5 Aug 2024 13:00:00 +0000

Type: Maintenance

Duration: 10 hours

Affected Components: , , ,

Aug 5, 20:46:55 GMT+0
Identified - We are still in progress to upgrade our NERC OCP cluster version, so extending this maintenance for another 2 hours until 7 :00 PM..

Aug 5, 13:44:39 GMT+0
Identified - We have successfully completed the NERC ColdFront service upgrade. Still the OpenShift upgrade will take whole day. Please be aware that any long-running pods or jobs on the NERC OpenShift platforms will be terminated and restarted. If they are not using persistent volumes or data, they might not be accessible even after the maintenance is complete. Please review them quickly and ensure you have a backup..

Aug 5, 23:00:00 GMT+0
Completed - Maintenance has completed successfully.

Aug 5, 13:00:01 GMT+0
Identified - Maintenance is now in progress.

Aug 5, 13:00:00 GMT+0
Identified - Upcoming NERC [OpenShift Container Platform (OCP)](https://console.apps.shift.nerc.mghpcc.org/) and [ColdFront service](https://coldfront.mss.mghpcc.org/) maintenance and upgrade will occur on Monday August 5, 2024 9 AM - 5 PM. **GENERAL MAINTENANCE** \- We plan to perform maintenance and a version upgrade on the NERC OpenShift deployment. This maintenance will involve a rolling reboot of all hosts in the cluster. \- We are also upgrading and updating NERC's ColdFront deployment. **NOTICES** \- We plan to upgrade the NERC OpenShift deployment to a newer **Red Hat OpenShift (OCP)** version 4.15 and **Red Hat OpenShift AI (RHOAI)** to v2.8.2, incorporating new features and functionalities. \- [This version](https://docs.openshift.com/container-platform/4.15/nodes/containers/nodes-containers-dev-fuse.html) will allow accessing faster builds with **_/dev/fuse_** on users' unprivileged pods, enabling users to leverage the **Filesystem in Userspace (FUSE)** interface to create and manage custom filesystems without requiring elevated privileges. \- During this process, Access to the [NERC OpenShift cluster](https://nerc-project.github.io/nerc-docs/openshift/) and [NERC Red Hat OpenShift AI (RHOAI)](https://nerc-project.github.io/nerc-docs/openshift-ai/) via the web console and other OpenShift CLI tools will be unavailable. \- This upgrade will cause some outages to the NERC’s ColdFront service. \- Please be aware that during this maintenance project, project change requests for OpenShift Resource Type **will not be accepted or approved**. \- The estimated time to complete ColdFront service update is **2 hours** and for OCP update is **8 hours**. It can take more or less time, so we urge you to keep an eye on to get the progress during this time. We will post updates to the status page when the ColdFront portion of this maintenance has been completed. \- Please do subscribe to to get the progress updates during this time. \- Please let us know if you encounter any issues after the maintenance is completed. Our priority is to help make science happen, so If you or your research team have any questions or need to escalate an issue, please don't hesitate to reach out to us via email ([help@nerc.mghpcc.org](mailto:help@nerc.mghpcc.org)) or, by submitting a new ticket at [the NERC's Support Ticketing System](https://mghpcc.supportsystem.com/open.php). Thank you, New England Research Cloud (NERC) .

Upcoming NERC’s RegApp & ColdFront service upgrade and maintenance June 18, 2024 9:00 AM - 11:00 AM

Tue, 18 Jun 2024 13:00:00 +0000

Type: Maintenance

Duration: 1 hour and 34 minutes

Affected Components: ,

Jun 18, 14:33:37 GMT+0
Completed - All upgrade and maintenance is complete now!.

Jun 18, 13:00:01 GMT+0
Identified - Maintenance is now in progress.

Jun 18, 13:00:00 GMT+0
Identified - **NERC’s** [**RegApp**](https://regapp.mss.mghpcc.org/) **&** [**ColdFront**](https://coldfront.mss.mghpcc.org/) **upgrade and maintenance will occur on Tuesday June 18, 2024 from 9:00 AM - 11:00 AM.** **GENERAL MAINTENANCE** \- We are upgrading and updating NERC's RegApp deployment. \- We are upgrading and updating NERC's ColdFront deployment. **NOTICES** \- This upgrade will cause some outages to the NERC’s RegApp & ColdFront service. \- The estimated time to complete this update is **2 hours**. It can take more or less time, so we urge you to keep an eye on to get the progress during this time. \- Please do subscribe to in order to get the progress updates during this time. Our priority is to help make science happen, so If you or your research team have any questions or need to escalate an issue, please don't hesitate to reach out to us via email ([help@nerc.mghpcc.org](mailto:help@nerc.mghpcc.org)) or, by submitting a new ticket at [the NERC's Support Ticketing System](https://mghpcc.supportsystem.com/open.php). Thank you, New England Research Cloud (NERC) [https://nerc.mghpcc.org/ https://nerc-project.github.io/nerc-docs/](https://nerc.mghpcc.org/https://nerc-project.github.io/nerc-docs/).

Upcoming MGHPCC (Holyoke) Annual Power Shutdown [May 22, 2023 8:00 AM ET - May 28, 2023 8:00 PM ET] (multi-day)

Wed, 22 May 2024 12:00:00 +0000

Type: Maintenance

Duration: 6 days and 12 hours

Affected Components: , , , , , , , , , , , , ,

May 22, 12:00:00 GMT+0
Identified - Due to the annual MGHPCC (Holyoke data center) and other maintenance activities, New England Research Cloud (NERC) services will be unavailable from **Wednesday, May 22nd, 8 AM until Tuesday, May 28th, 8 PM**. While the timing is not ideal due to the long Memorial Day weekend, it is necessary due to upgrades on related core services such as networking and the Northeast Storage Exchange (NESE). NERC services that will be impacted include: * NERC's RedHat OpenStack * NERC's RedHat OpenShift, which includes the Red Hat OpenShift AI (RHOAI) platform. * NERC supporting services ColdFront and RegApp Due to the long holiday weekend, our goal is to resume full services by **the end of Tuesday, May 28th, at 8 PM** or **earlier**. Please refer to the timeline below for details: ![MGHPCC_annual_power_shutdown_2024.png](https://res.cloudinary.com/sup/image/upload/v1714151495/gghntjaqmk1lb1upzkje.png) **NOTICES** \- This outage affects all services and resources in the MGHPCC/Holyoke data center which include NERC’s core services and underlying compute, storage and networking. \- The estimated time for power-up and return to full service is **8 PM on Tuesday, May 28th**. We encourage you to monitor the progress at during this time. \- Please subscribe to the NERC’s status for any future updates as they happen: Our priority is to help make science happen, so If you or your research team have any questions or need to escalate an issue, please contact us via email ([help@nerc.mghpcc.org](mailto:help@nerc.mghpcc.org)) or, by submitting a new ticket at [the NERC's Support Ticketing System](https://mghpcc.supportsystem.com/open.php). Thank you, New England Research Cloud (NERC) [https://nerc.mghpcc.org/ https://nerc.instatus.com/](https://nerc.mghpcc.org/https://nerc.instatus.com/).

May 22, 12:00:01 GMT+0
Identified - Maintenance is now in progress.

May 29, 00:00:00 GMT+0
Completed - Maintenance has completed successfully.

Upcoming NERC's ColdFront service upgrade and maintenance Tuesday May 14, 2024 8:00 AM - 10:00 AM

Tue, 14 May 2024 12:00:00 +0000

Type: Maintenance

Duration: 3 hours

Affected Components:

May 14, 15:00:00 GMT+0
Completed - Maintenance has completed successfully.

May 14, 14:03:53 GMT+0
Identified - We are extending this maintenance for 1 more hour..

May 14, 12:00:01 GMT+0
Identified - Maintenance is now in progress.

May 14, 12:00:00 GMT+0
Identified - **Upcoming NERC's** [**ColdFront**](https://coldfront.mss.mghpcc.org/) **upgrade and maintenance will occur on Tuesday May 14, 2024 from 8:00 AM – 10:00 AM.** **ENERAL MAINTENANCE** \- We are upgrading and updating NERC's ColdFront deployment. \- This will allow users to request minimal Swift Object Storage during the [decommission process](https://nerc-project.github.io/nerc-docs/openstack/decommission/decommission-openstack-resources/#review-your-object-storageswift-quota). **NOTICES** \- This upgrade will cause some outages to the NERC’s ColdFront service. \- The estimated time to complete this update is **2 hours**. It can take more or less time, so we urge you to keep an eye on to get the progress during this time. \- Please do subscribe to in order to get the progress updates during this time. Our priority is to help make science happen, so If you or your research team have any questions or need to escalate an issue, please contact us via email ([help@nerc.mghpcc.org](mailto:help@nerc.mghpcc.org)) or, by submitting a new ticket at [the NERC’s Support Ticketing System](https://mghpcc.supportsystem.com/open.php). Thanks, New England Research Cloud (NERC) .

Upcoming NERC system maintenance and upgrade - OpenStack and ColdFront - Tuesday April 16, 2024 9AM - 5PM

Tue, 16 Apr 2024 13:00:00 +0000

Type: Maintenance

Duration: 8 hours

Affected Components: , , , , , , ,

Apr 16, 13:00:00 GMT+0
Identified - **Upcoming NERC system maintenance and upgrade will occur on Tuesday April 16, 2024 from 9:00 AM – 5:00 PM.** **GENERAL MAINTENANCE** \- We are writing to inform you that we are planning to perform maintenance on the [NERC OpenStack](https://stack.nerc.mghpcc.org/) deployment. This will take the majority of the maintenance period. \- We will also be upgrading our [ColdFront](https://coldfront.mss.mghpcc.org/) deployment to support minimal Swift Object Storage during the [decommission process](https://nerc-project.github.io/nerc-docs/openstack/decommission/decommission-openstack-resources/#review-your-object-storageswift-quota). **NOTICES** \- We plan to perform maintenance on the NERC OpenStack deployment during this time. \- NERC OpenStack’s Horizon dashboard and other OpenStack API services may be inaccessible at times during this process. \- The OpenStack maintenance should not affect any of your running VMs and used storage including volumes and object storage. \- We will post updates to the status page when the ColdFront portion of this maintenance has been completed. \- We are reserving **the whole day**, from 9am to 5pm. It can take more or less time, so we urge you to keep an eye on to get the progress during this time. \- Please do subscribe to in order to get the progress updates during this time. Our priority is to help make science happen, so If you or your research team have any questions or need to escalate an issue, please contact us via email ([help@nerc.mghpcc.org](mailto:help@nerc.mghpcc.org)) or, by submitting a new ticket at [the NERC's Support Ticketing System](https://mghpcc.supportsystem.com/open.php)..

Apr 16, 21:00:00 GMT+0
Completed - Maintenance has completed successfully.

Apr 16, 13:00:01 GMT+0
Identified - Maintenance is now in progress.

Upcoming NERC system maintenance and upgrade - OpenStack, ColdFront and RegApp - Tuesday Mar 26, 2024 9AM - 5PM

Tue, 26 Mar 2024 13:00:00 +0000

Type: Maintenance

Duration: 13 hours and 59 minutes

Affected Components: , , , , , , , ,

Mar 26, 13:01:29 GMT+0
Identified - Maintenance is now in progress.

Mar 26, 22:41:37 GMT+0
Identified - We are still working on this upgrade so will add 2 more hours to complete..

Mar 27, 00:53:30 GMT+0
Identified - We are still working on this upgrade so will add 2 more hours to complete..

Mar 26, 13:00:00 GMT+0
Identified - **Upcoming NERC system maintenance and upgrade will occur on Tuesday March 26, 2024 from 9:00 AM – 5:00 PM.** **GENERAL MAINTENANCE** \- We are writing to inform you that we are planning to perform maintenance on the NERC OpenStack deployment. This will take the majority of the maintenance period. \- We will also be upgrading our RegApp deployment to support registration from all CILogon supported IdPs. **NOTICES** \- We plan to perform maintenance on the NERC OpenStack deployment during this time. \- NERC OpenStack’s Horizon dashboard and other OpenStack API services may be inaccessible at times during this process. \- The OpenStack maintenance should not affect any of your running VMs and used storage including volumes and object storage. \- We will post updates to the status page when the ColdFront and RegApp portion of this maintenance has been completed. \- We are reserving **the whole day**, from 9am to 5pm. It can take more or less time, so we urge you to keep an eye on to get the progress during this time. \- Please do subscribe to in order to get the progress updates during this time. Our priority is to help make science happen, so If you or your research team have any questions or need to escalate an issue, please contact us via email ([help@nerc.mghpcc.org](mailto:help@nerc.mghpcc.org)) or, by submitting a new ticket at [the NERC's Support Ticketing System](https://mghpcc.supportsystem.com/open.php)..

Mar 26, 20:56:29 GMT+0
Identified - We are still working on this upgrade so will add 2 more hours to complete..

Mar 27, 03:00:00 GMT+0
Completed - Maintenance has completed successfully.

Mar 26, 14:09:43 GMT+0
Identified - MGHPCC SHARED SERVICES (MGHPCC-SS) ACCOUNT PORTAL a.k.a **RegApp** is fully upgraded and functional now..

Upcoming NERC OpenShift Container Platform (OCP) maintenance and upgrade – [Tuesday March 12, 2024 9:00 AM - 12:00 PM]

Tue, 12 Mar 2024 13:00:00 +0000

Type: Maintenance

Duration: 4 hours

Affected Components: , ,

Mar 12, 15:57:48 GMT+0
Identified - We are still in progress to add the GPU nodes, so extending this maintenance for another 1 hour..

Mar 12, 13:00:00 GMT+0
Identified - **Upcoming NERC OpenShift Container Platform (OCP) maintenance and upgrade will occur on Tuesday March 12, 2024 from 9 AM - 12 PM.** **GENERAL MAINTENANCE** \- We are writing to inform you that we are planning to perform maintenance on the NERC OpenShift deployment which will involve a rolling reboot of all hosts in the cluster. **NOTICES** \- We plan to perform maintenance on the NERC OpenShift deployment during this time. \- NERC OpenShift web console and other OpenShift CLI may be inaccessible at times during this process. \- We will be expanding the NERC OpenShift cluster with new hosts that will provide 40 NVIDIA A100 SMX4 GPUs. This will involve a rolling reboot of all hosts in the OpenShift cluster. **Please be aware this may cause issues for any workloads on the cluster that are not configured to automatically failover to other nodes.** \- Please be aware that during this maintenance, project and project change requests for OpenShift Resource Type **will not be accepted and approved**. \- We are reserving **three hours**, from 9am to 12pm. It can take more or less time, so we urge you to keep an eye on to get the progress during this time. \- Please do subscribe to in order to get the progress updates during this time. Our priority is to help make science happen, so If you or your research team have any questions or need to escalate an issue, please contact us via email ([help@nerc.mghpcc.org](mailto:help@nerc.mghpcc.org)) or, by submitting a new ticket at [the NERC's Support Ticketing System](https://mghpcc.supportsystem.com/open.php). Thanks, New England Research Cloud (NERC) .

Mar 12, 13:00:01 GMT+0
Identified - Maintenance is now in progress.

Mar 12, 17:00:00 GMT+0
Completed - Maintenance has completed successfully.

Upcoming NERC OpenShift Container Platform (OCP) maintenance and upgrade – [Thursday February 22, 2024 from 2:00 PM – 5:00 PM]

Thu, 22 Feb 2024 19:00:00 +0000

Type: Maintenance

Duration: 3 hours

Affected Components: , ,

Feb 22, 22:00:00 GMT+0
Completed - Maintenance has completed successfully.

Feb 22, 19:00:00 GMT+0
Identified - **Upcoming NERC OpenShift Container Platform (OCP) maintenance and upgrade will occur on Thursday February 22, 2024 from 2 PM - 5 PM.** **GENERAL MAINTENANCE** – We are writing to inform you that we are planning to perform maintenance on the NERC OpenShift deployment. **NOTICES** \- We will be expanding the NERC OpenShift cluster with new compute hosts that will provide **40 NVIDIA A100 SXM4 GPUs**. This will involve a rolling reboot of all hosts in the OpenShift cluster. Please be aware this may cause issues for any workloads on the cluster that are not configured to automatically failover to other nodes. \- NERC OpenShift web console and other OpenShift CLI may be inaccessible at times during this process. \- Please ensure that all your Jupyter notebooks and RHODS workloads have been saved and exited before 2 PM. \- Please be aware that during this maintenance, project and project change requests for OpenShift Resource Type **will not be accepted and approved**. \- We are reserving **three hours**, from 2pm to 5pm. It can take more or less time, so we urge you to keep an eye on to get the progress during this time. \- Please do subscribe to in order to get the progress updates during this time. Our priority is to help make science happen, so If you or your research team have any questions or need to escalate an issue, please contact us via email ([help@nerc.mghpcc.org](mailto:help@nerc.mghpcc.org)) or, by submitting a new ticket at [the NERC’s Support Ticketing System](https://mghpcc.supportsystem.com/open.php). Thanks, New England Research Cloud (NERC) .

Feb 22, 19:00:01 GMT+0
Identified - Maintenance is now in progress.