NERC - Notice history

All systems operational

GETTING HELP
Email: help@nerc.mghpcc.org or, using the NERC's Support Ticketing System
NERC Documentation: https://nerc-project.github.io/nerc-docs/
Status page for the New England Research Cloud (NERC) and other resources.
Please scroll down to see details on any Incidents or maintenance notices.

MGHPCC SHARED SERVICES (MGHPCC-SS) ACCOUNT PORTAL - Operational

99.99% uptime
Mar 2022 · 99.98%Apr · 100.0%May · 100.0%

Previous page

Next page

Mar 2022
Apr 2022
May 2022

NERC COLDFRONT - Operational

99.99% uptime
Mar 2022 · 99.98%Apr · 100.0%May · 100.0%

Previous page

Next page

Mar 2022
Apr 2022
May 2022

NERC ACADEMIC (EDU) OPENSHIFT CLUSTER - Operational

100.0% uptime
Mar 2022 · 100.0%Apr · 100.0%May · 100.0%

Previous page

Next page

Mar 2022
Apr 2022
May 2022

NETWORKING - Operational

99.99% uptime
Mar 2022 · 99.98%Apr · 100.0%May · 100.0%

Previous page

Next page

Mar 2022
Apr 2022
May 2022

STORAGE - Operational

99.99% uptime
Mar 2022 · 99.98%Apr · 100.0%May · 100.0%

Previous page

Next page

Mar 2022
Apr 2022
May 2022
99.75% uptime

NERC WEBSITE - Operational

100.0% uptime
Mar 2022 · 100.0%Apr · 100.0%May · 100.0%

Previous page

Next page

Mar 2022
Apr 2022
May 2022

NERC DOCUMENTATION WEBSITE - Operational

100.0% uptime
Mar 2022 · 100.0%Apr · 100.0%May · 100.0%

Previous page

Next page

Mar 2022
Apr 2022
May 2022

NERC TICKETING SYSTEM - Operational

99.24% uptime
Mar 2022 · 97.77%Apr · 100.0%May · 100.0%

Previous page

Next page

Mar 2022
Apr 2022
May 2022

Notice history

View current status

May 2022

MAY23
MGHPCC (Holyoke) Annual Power Shutdown [May 23, 2022 6:00 PM – May 26, 2022 12:00 PM] (multi-day)
CompletedMaintenance66 hours
  • CompletedMay 26, 2022 at 4:00 PMUTC

    Maintenance has completed successfully

  • In progressMay 23, 2022 at 10:00 PMUTC

    Maintenance is now in progress

  • PlannedMay 23, 2022 at 10:00 PMUTC

    The annual MGHPCC data center power shutdown will occur May 24th. Even though we are going to begin the NERC shutdown process from May 23rd at 6 PM along with Research Computing at Harvard FAS (FASRC)and Northeast Storage Exchange (NESE) team.

    GENERAL MAINTENANCE

    – Power-down of the cluster will begin from 6 PM on May 23rd.

    – Power will be out that night and through the following day, May 24th.

    – Maintenance to NERC, FASRC, NESE infrastructure will occur on May 25th.

    – Power-up ETA and return to service is noon on May 26th.

    NOTICES

    – While this outage impacts all services and resources in the MGHPCC/Holyoke data center, please be aware NERC services along with storage will not be available during this timeline.

    – Power-up ETA and return to service is noon on May 26th; so we urge you to keep an eye on https://nerc.instatus.com/ to get the progress during this time.

    – Please do subscribe to the NERC’s status for any future updates: https://nerc.instatus.com/subscribe/email

    Our priority is to help make science happen, so If you or your research team have any questions or need to escalate an issue, please contact us via email (help@nese.mghpcc.org).

    Thanks,

    New England Research Cloud (NERC)

    https://nerc.mghpcc.org/

    https://nerc-project.github.io/nerc-docs/

Apr 2022

APR28
NERC system upgrade and maintenance April 28, 2022 from 10:00 AM - 12:00 PM
CompletedMaintenance2 hours
  • CompletedApril 28, 2022 at 4:00 PMUTC

    Maintenance has completed successfully

  • In progressApril 28, 2022 at 2:00 PMUTC

    Maintenance is now in progress

  • PlannedApril 28, 2022 at 2:00 PMUTC

    NERC planned system upgrade and maintenance will occur on Thursday April 28, 2022 from 10:00 AM - 12:00 PM.

    GENERAL MAINTENANCE

    • We will be reconfiguring SSL certificates for keycloak, regapp, and coldfront services with LetsEncrypt. This will cause some outages to login to these services as well as OpenStack.

    NOTICES

    • Automatic SSL certificate renewal will help keep all our web services more secure.

    • We don’t know the ETA for this upgrade but the tentative estimated time is around 2 hours - so we urge you to keep an eye on https://nerc.instatus.com/ to get the progress during this time.

    • Please do subscribe to the NERC’s status for any future updates: https://nerc.instatus.com/subscribe/email

    Our priority is to help make science happen, so If you or your research team have any questions or need to escalate an issue, please contact us via email (help@nese.mghpcc.org).

    Thanks,

    New England Research Cloud (NERC)

    https://nerc.mghpcc.org/

    https://nerc-project.github.io/nerc-docs/

An incident at the MGHPCC Datacenter has effected a number of systems.
ResolvedDegraded performance2 hours 16 minutes
  • ResolvedUTC

    We just successfully resolved the issue! But still our storage may be slow for a bit while things recover back to its normal state but should be functional for general purpose.

  • IdentifiedUTC

    We are continuing to work on a fix for this incident.

  • InvestigatingUTC

    The extent of the issue is currently unknown, it is under investigation. We are currently investigating this incident. We'll update here as the situation progresses.

APR6
NERC system upgrade and maintenance April 6, 2022 3:00PM - 6:00PM
CompletedMaintenance3 hours
  • UpdateApril 06, 2022 at 10:00 PMUTC

    Maintenance has completed successfully

  • UpdateApril 06, 2022 at 10:00 PMUTC

    Maintenance has completed successfully

  • CompletedApril 06, 2022 at 10:00 PMUTC

    [NERC Users List]

    NERC planned system upgrade and maintenance on Wednesday April 6, 2022 from 3:00PM - 6:00PM has completed successfully.

    Please make sure to clearly refer and follow the following important information:

    • All your existing VMs that had a floating IP address assigned previously now have a new floating IP address. Please lookup the new address using either the Horizon web interface or the OpenStack CLI.

    • All users will need to disconnect and reconnect to VPN in order to connect to the new floating IP(s).

    Our priority is to help make science happen, so If you or your research team have any questions or need to escalate an issue, please contact us via email (help@nese.mghpcc.org).

    Thank you, New England Research Cloud (NERC)

    https://nerc.mghpcc.org/

    https://nerc.instatus.com/

  • In progressApril 06, 2022 at 7:00 PMUTC

    Maintenance is now in progress

  • PlannedApril 06, 2022 at 7:00 PMUTC

    NERC planned system upgrade and maintenance will occur on Wednesday, April 6, 2022 from 3:00PM - 6:00PM.

    GENERAL MAINTENANCE

    • Remove all floating IPs and external gateways on all virtual routers

    • Expand the floating IP pool

    • Create new floating IPs for any instances that had them removed

    • Restore external gateways for all virtual routers that had them.

    NOTICES

    • Internet connectivity both to and from the instances will be unavailable during this work.

    • We will double the amount of Floating IPs to use under NERC resources.

    • Users will need to change any reference they are putting for existing Floating IPs, i.e. on SSH configuration or any other services they are using them previously.

    • We don’t know the ETA for this upgrade but the tentative estimated time is around 3 hours - so we urge you to keep an eye on https://nerc.instatus.com/ to get the progress during this time.

    • Please do subscribe to the NERC’s status for any future updates: https://nerc.instatus.com/subscribe/email

    Our priority is to help make science happen, so If you or your research team have any questions or need to escalate an issue, please contact us via email (help@nese.mghpcc.org).

    Thanks,

    New England Research Cloud (NERC)

    https://nerc.mghpcc.org/

    https://nerc-project.github.io/nerc-docs/

Mar 2022

MAR30
NERC system upgrade and maintenance March 30, 2022 9:00AM - 5:00PM
CompletedMaintenance8 hours
  • CompletedMarch 30, 2022 at 9:00 PMUTC

    Maintenance has completed successfully. But still we have not conducted floating IP work at this time - we’ll arrange a new date/time for this task and communicate the announcement soon.

  • UpdateMarch 30, 2022 at 4:02 PMUTC

    Maintenance is still in progress, we are extending the ETA for till 5PM.

  • In progressMarch 30, 2022 at 1:00 PMUTC

    Maintenance is now in progress

  • PlannedMarch 30, 2022 at 1:00 PMUTC

    NERC planned system upgrade and maintenance will occur on Wednesday, March 30, 2022 from 9:00AM - 12:00PM.

    GENERAL MAINTENANCE

    • Remove all floating IPs and external gateways on all virtual routers

    • Expand the floating IP pool

    • Create new floating IPs for any instances that had them removed

    • Restore external gateways for all virtual routers that had them.

    • Fix for Maximum transmission unit (MTU) issue on tenant/overlay network

    NOTICES

    • Internet connectivity both to and from the instances will be unavailable during this work.

    • We will double the amount of Floating IPs to use under NERC resources.

    • Users will need to change any reference they are putting for existing Floating IPs, i.e. on SSH configuration or any other services they are using them previously.

    • We don’t know the ETA for this upgrade but the tentative estimated time is around 3 hours - so we urge you to keep an eye on https://nerc.instatus.com/ to get the progress during this time.

    • Please do subscribe to the NERC’s status for any future updates: https://nerc.instatus.com/subscribe/email

    Our priority is to help make science happen, so If you or your research team have any questions or need to escalate an issue, please contact us via email (help@nese.mghpcc.org).

    Thanks,

    New England Research Cloud (NERC)

    https://nerc.mghpcc.org/

    https://nerc-project.github.io/nerc-docs/

OTHER WEBSITES & TOOLS - NERC TICKETING SYSTEM isn't accessible at the moment (automated)
ResolvedMajor outage16 hours 33 minutes
  • ResolvedUTC

    OTHER WEBSITES & TOOLS - NERC TICKETING SYSTEM is now operational! This update was created by an automated monitoring service.

  • InvestigatingUTC

    OTHER WEBSITES & TOOLS - NERC TICKETING SYSTEM cannot be accessed at the moment. This incident was created by an automated monitoring service.

Long path fibre to MGHPCC Holyoke is also down
ResolvedMinor outage35 minutes
  • ResolvedUTC

    We just resolved the issue!

  • MonitoringUTC

    The long path has returned to operation and initial checks indicate no substantial job or cluster issues as a result.

    The short path remains down, so latency between Boston and Holyoke will still be higher than normal. See short path incident for updates as that progresses (no ETA at this time)

    This may cause some unstable performance issues on the NERC services.

  • IdentifiedUTC

    The short path fibre connection to Holyoke/MGHPCC suffered a failure late yesterday (3/15/22). The short path is still down as of this morning.

    However, we have just seen the long path also drop connection and are awaiting more information on this incident. It is our hope this is a minor disconnect related to their repair work.

The short path fibre to MGHPCC Holyoke is down - long path remains up
ResolvedDegraded performance20 hours 15 minutes
  • ResolvedUTC

    We just resolved the issue!

  • IdentifiedUTC

    We are continuing to work on a fix for this incident. No ETA on short path repair.

  • InvestigatingUTC

    We've been informed that the 'short path' fibre connection to Holyoke is down. No ETA.

    The 'long path' remains in operation, but this means slightly more latency when performing operations between Boston and Holyoke.

    We will relay updates when we receive them. We are currently investigating this incident.

MAR4
System Upgrade/ Maintenance - 03/04/2022
CompletedMaintenance2 hours 13 minutes
  • CompletedMarch 04, 2022 at 5:43 PMUTC

    Maintenance has completed successfully.

  • In progressMarch 04, 2022 at 3:33 PMUTC

    Maintenance is now in progress.

  • PlannedMarch 04, 2022 at 3:30 PMUTC

    This is a minor maintenance on the NERC infrastructure.

    NOTICES

    • - Users will not be able to login to the NERC setup

    • - Users can not login to the NERC's OpenStack environment to use their allocated resources i.e. VM

    GENERAL MAINTENANCE

    • - KeyCloak database will be down during this maintenance window
    • - We are adding fully automated public VIP setup on the nerc-shift-1 cluster

    Thanks, New England Research Cloud (NERC) https://nerc.mghpcc.org/ https://nerc-project.github.io/nerc-docs/

Previous

Mar 2022 to May 2022

Next

For issues not shown here, please contact NERC via email help@nerc.mghpcc.org or, using the NERC's Support Ticketing System.