NERC - Notice history

System Under Maintenance

GETTING HELP
Email: help@nerc.mghpcc.org or, using the NERC's Support Ticketing System
NERC Documentation: https://nerc-project.github.io/nerc-docs/
Status page for the New England Research Cloud (NERC) and other resources.
Please scroll down to see details on any Incidents or maintenance notices.

MGHPCC SHARED SERVICES (MGHPCC-SS) ACCOUNT PORTAL - Operational

100% - uptime
Feb 2022 · 100.0%Mar · 99.98%Apr · 100.0%

Previous page

Next page

Feb 2022
Mar 2022
Apr 2022

NERC COLDFRONT - Operational

100% - uptime
Feb 2022 · 100.0%Mar · 99.98%Apr · 100.0%

Previous page

Next page

Feb 2022
Mar 2022
Apr 2022

NETWORKING - Under maintenance

100% - uptime
Feb 2022 · 100.0%Mar · 99.98%Apr · 100.0%

Previous page

Next page

Feb 2022
Mar 2022
Apr 2022

STORAGE - Under maintenance

100% - uptime
Feb 2022 · 100.0%Mar · 99.98%Apr · 100.0%

Previous page

Next page

Feb 2022
Mar 2022
Apr 2022
100% - uptime

NERC WEBSITE - Operational

100% - uptime
Feb 2022 · 100.0%Mar · 100.0%Apr · 100.0%

Previous page

Next page

Feb 2022
Mar 2022
Apr 2022

NERC DOCUMENTATION WEBSITE - Operational

100% - uptime
Feb 2022 · 100.0%Mar · 100.0%Apr · 100.0%

Previous page

Next page

Feb 2022
Mar 2022
Apr 2022

NERC TICKETING SYSTEM - Operational

99% - uptime
Feb 2022 · 100.0%Mar · 97.77%Apr · 100.0%

Previous page

Next page

Feb 2022
Mar 2022
Apr 2022

Notice history

Apr 2022

NERC system upgrade and maintenance April 28, 2022 from 10:00 AM - 12:00 PM
  • Completed
    April 28, 2022 at 4:00 PM
    Completed
    April 28, 2022 at 4:00 PM

    Maintenance has completed successfully

  • In progress
    April 28, 2022 at 2:00 PM
    In progress
    April 28, 2022 at 2:00 PM

    Maintenance is now in progress

  • Planned
    April 28, 2022 at 2:00 PM
    Planned
    April 28, 2022 at 2:00 PM

    NERC planned system upgrade and maintenance will occur on Thursday April 28, 2022 from 10:00 AM - 12:00 PM.

    GENERAL MAINTENANCE

    • We will be reconfiguring SSL certificates for keycloak, regapp, and coldfront services with LetsEncrypt. This will cause some outages to login to these services as well as OpenStack.

    NOTICES

    • Automatic SSL certificate renewal will help keep all our web services more secure.

    • We don’t know the ETA for this upgrade but the tentative estimated time is around 2 hours - so we urge you to keep an eye on https://nerc.instatus.com/ to get the progress during this time.

    • Please do subscribe to the NERC’s status for any future updates: https://nerc.instatus.com/subscribe/email

    Our priority is to help make science happen, so If you or your research team have any questions or need to escalate an issue, please contact us via email (help@nese.mghpcc.org).

    Thanks,

    New England Research Cloud (NERC)

    https://nerc.mghpcc.org/

    https://nerc-project.github.io/nerc-docs/

An incident at the MGHPCC Datacenter has effected a number of systems.
  • Resolved
    Resolved

    We just successfully resolved the issue! But still our storage may be slow for a bit while things recover back to its normal state but should be functional for general purpose.

  • Identified
    Identified

    We are continuing to work on a fix for this incident.

  • Investigating
    Investigating

    The extent of the issue is currently unknown, it is under investigation. We are currently investigating this incident. We'll update here as the situation progresses.

NERC system upgrade and maintenance April 6, 2022 3:00PM - 6:00PM
  • Update
    April 06, 2022 at 10:00 PM
    Completed
    April 06, 2022 at 10:00 PM

    Maintenance has completed successfully

  • Update
    April 06, 2022 at 10:00 PM
    Completed
    April 06, 2022 at 10:00 PM

    Maintenance has completed successfully

  • Completed
    April 06, 2022 at 10:00 PM
    Completed
    April 06, 2022 at 10:00 PM

    [NERC Users List]

    NERC planned system upgrade and maintenance on Wednesday April 6, 2022 from 3:00PM - 6:00PM has completed successfully.

    Please make sure to clearly refer and follow the following important information:

    • All your existing VMs that had a floating IP address assigned previously now have a new floating IP address. Please lookup the new address using either the Horizon web interface or the OpenStack CLI.

    • All users will need to disconnect and reconnect to VPN in order to connect to the new floating IP(s).

    Our priority is to help make science happen, so If you or your research team have any questions or need to escalate an issue, please contact us via email (help@nese.mghpcc.org).

    Thank you, New England Research Cloud (NERC)

    https://nerc.mghpcc.org/

    https://nerc.instatus.com/

  • In progress
    April 06, 2022 at 7:00 PM
    In progress
    April 06, 2022 at 7:00 PM

    Maintenance is now in progress

  • Planned
    April 06, 2022 at 7:00 PM
    Planned
    April 06, 2022 at 7:00 PM

    NERC planned system upgrade and maintenance will occur on Wednesday, April 6, 2022 from 3:00PM - 6:00PM.

    GENERAL MAINTENANCE

    • Remove all floating IPs and external gateways on all virtual routers

    • Expand the floating IP pool

    • Create new floating IPs for any instances that had them removed

    • Restore external gateways for all virtual routers that had them.

    NOTICES

    • Internet connectivity both to and from the instances will be unavailable during this work.

    • We will double the amount of Floating IPs to use under NERC resources.

    • Users will need to change any reference they are putting for existing Floating IPs, i.e. on SSH configuration or any other services they are using them previously.

    • We don’t know the ETA for this upgrade but the tentative estimated time is around 3 hours - so we urge you to keep an eye on https://nerc.instatus.com/ to get the progress during this time.

    • Please do subscribe to the NERC’s status for any future updates: https://nerc.instatus.com/subscribe/email

    Our priority is to help make science happen, so If you or your research team have any questions or need to escalate an issue, please contact us via email (help@nese.mghpcc.org).

    Thanks,

    New England Research Cloud (NERC)

    https://nerc.mghpcc.org/

    https://nerc-project.github.io/nerc-docs/

Mar 2022

NERC system upgrade and maintenance March 30, 2022 9:00AM - 5:00PM
  • Completed
    March 30, 2022 at 9:00 PM
    Completed
    March 30, 2022 at 9:00 PM

    Maintenance has completed successfully. But still we have not conducted floating IP work at this time - we’ll arrange a new date/time for this task and communicate the announcement soon.

  • Update
    March 30, 2022 at 4:02 PM
    In progress
    March 30, 2022 at 4:02 PM

    Maintenance is still in progress, we are extending the ETA for till 5PM.

  • In progress
    March 30, 2022 at 1:00 PM
    In progress
    March 30, 2022 at 1:00 PM

    Maintenance is now in progress

  • Planned
    March 30, 2022 at 1:00 PM
    Planned
    March 30, 2022 at 1:00 PM

    NERC planned system upgrade and maintenance will occur on Wednesday, March 30, 2022 from 9:00AM - 12:00PM.

    GENERAL MAINTENANCE

    • Remove all floating IPs and external gateways on all virtual routers

    • Expand the floating IP pool

    • Create new floating IPs for any instances that had them removed

    • Restore external gateways for all virtual routers that had them.

    • Fix for Maximum transmission unit (MTU) issue on tenant/overlay network

    NOTICES

    • Internet connectivity both to and from the instances will be unavailable during this work.

    • We will double the amount of Floating IPs to use under NERC resources.

    • Users will need to change any reference they are putting for existing Floating IPs, i.e. on SSH configuration or any other services they are using them previously.

    • We don’t know the ETA for this upgrade but the tentative estimated time is around 3 hours - so we urge you to keep an eye on https://nerc.instatus.com/ to get the progress during this time.

    • Please do subscribe to the NERC’s status for any future updates: https://nerc.instatus.com/subscribe/email

    Our priority is to help make science happen, so If you or your research team have any questions or need to escalate an issue, please contact us via email (help@nese.mghpcc.org).

    Thanks,

    New England Research Cloud (NERC)

    https://nerc.mghpcc.org/

    https://nerc-project.github.io/nerc-docs/

OTHER WEBSITES & TOOLS - NERC TICKETING SYSTEM isn't accessible at the moment (automated)
  • Resolved
    Resolved

    OTHER WEBSITES & TOOLS - NERC TICKETING SYSTEM is now operational! This update was created by an automated monitoring service.

  • Investigating
    Investigating

    OTHER WEBSITES & TOOLS - NERC TICKETING SYSTEM cannot be accessed at the moment. This incident was created by an automated monitoring service.

Long path fibre to MGHPCC Holyoke is also down
  • Resolved
    Resolved

    We just resolved the issue!

  • Monitoring
    Monitoring

    The long path has returned to operation and initial checks indicate no substantial job or cluster issues as a result.

    The short path remains down, so latency between Boston and Holyoke will still be higher than normal. See short path incident for updates as that progresses (no ETA at this time)

    This may cause some unstable performance issues on the NERC services.

  • Identified
    Identified

    The short path fibre connection to Holyoke/MGHPCC suffered a failure late yesterday (3/15/22). The short path is still down as of this morning.

    However, we have just seen the long path also drop connection and are awaiting more information on this incident. It is our hope this is a minor disconnect related to their repair work.

The short path fibre to MGHPCC Holyoke is down - long path remains up
  • Resolved
    Resolved

    We just resolved the issue!

  • Identified
    Identified

    We are continuing to work on a fix for this incident. No ETA on short path repair.

  • Investigating
    Investigating

    We've been informed that the 'short path' fibre connection to Holyoke is down. No ETA.

    The 'long path' remains in operation, but this means slightly more latency when performing operations between Boston and Holyoke.

    We will relay updates when we receive them. We are currently investigating this incident.

System Upgrade/ Maintenance - 03/04/2022
  • Completed
    March 04, 2022 at 5:43 PM
    Completed
    March 04, 2022 at 5:43 PM

    Maintenance has completed successfully.

  • In progress
    March 04, 2022 at 3:33 PM
    In progress
    March 04, 2022 at 3:33 PM

    Maintenance is now in progress.

  • Planned
    March 04, 2022 at 3:30 PM
    Planned
    March 04, 2022 at 3:30 PM

    This is a minor maintenance on the NERC infrastructure.

    NOTICES

    • - Users will not be able to login to the NERC setup

    • - Users can not login to the NERC's OpenStack environment to use their allocated resources i.e. VM

    GENERAL MAINTENANCE

    • - KeyCloak database will be down during this maintenance window
    • - We are adding fully automated public VIP setup on the nerc-shift-1 cluster

    Thanks, New England Research Cloud (NERC) https://nerc.mghpcc.org/ https://nerc-project.github.io/nerc-docs/

Feb 2022

NERC OPENSTACK isn't accessible at the moment (automated)
  • Resolved
    Resolved

    NERC OPENSTACK is now operational! This update was created by an automated monitoring service.

  • Investigating
    Investigating

    NERC OPENSTACK cannot be accessed at the moment. This incident was created by an automated monitoring service.

OTHER WEBSITES & TOOLS - NERC WEBSITE isn't accessible at the moment (automated)
  • Resolved
    Resolved

    OTHER WEBSITES & TOOLS - NERC WEBSITE is now operational! This update was created by an automated monitoring service.

  • Investigating
    Investigating

    OTHER WEBSITES & TOOLS - NERC WEBSITE cannot be accessed at the moment. This incident was created by an automated monitoring service.

MGHPCC SHARED SERVICES (MGHPCC-SS) ACCOUNT PORTAL isn't accessible at the moment (automated)
  • Resolved
    Resolved

    MGHPCC SHARED SERVICES (MGHPCC-SS) ACCOUNT PORTAL is now operational! This update was created by an automated monitoring service.

  • Investigating
    Investigating

    MGHPCC SHARED SERVICES (MGHPCC-SS) ACCOUNT PORTAL cannot be accessed at the moment. This incident was created by an automated monitoring service.

Feb 2022 to Apr 2022

Next

For issues not shown here, please contact NERC via email help@nerc.mghpcc.org or, using the NERC's Support Ticketing System.