NERC - Notice history

All systems operational

GETTING HELP
Email: help@nerc.mghpcc.org or, using the NERC's Support Ticketing System
NERC Documentation: https://nerc-project.github.io/nerc-docs/
Status page for the New England Research Cloud (NERC) and other resources.
Please scroll down to see details on any Incidents or maintenance notices.

MGHPCC SHARED SERVICES (MGHPCC-SS) ACCOUNT PORTAL - Operational

98% - uptime
Jun 2022 · 100.0%Jul · 100.0%Aug · 93.23%

Previous page

Next page

Jun 2022
Jul 2022
Aug 2022

NERC COLDFRONT - Operational

100% - uptime
Jun 2022 · 100.0%Jul · 100.0%Aug · 100.0%

Previous page

Next page

Jun 2022
Jul 2022
Aug 2022

NETWORKING - Operational

100% - uptime
Jun 2022 · 100.0%Jul · 99.79%Aug · 100.0%

Previous page

Next page

Jun 2022
Jul 2022
Aug 2022

STORAGE - Operational

100% - uptime
Jun 2022 · 100.0%Jul · 99.79%Aug · 100.0%

Previous page

Next page

Jun 2022
Jul 2022
Aug 2022
100% - uptime

NERC WEBSITE - Operational

100% - uptime
Jun 2022 · 100.0%Jul · 100.0%Aug · 100.0%

Previous page

Next page

Jun 2022
Jul 2022
Aug 2022

NERC DOCUMENTATION WEBSITE - Operational

100% - uptime
Jun 2022 · 100.0%Jul · 100.0%Aug · 100.0%

Previous page

Next page

Jun 2022
Jul 2022
Aug 2022

NERC TICKETING SYSTEM - Operational

100% - uptime
Jun 2022 · 100.0%Jul · 100.0%Aug · 100.0%

Previous page

Next page

Jun 2022
Jul 2022
Aug 2022

Notice history

Aug 2022

Most of the NERC VMs are in "Shutoff" mode
  • Resolved
    Resolved

    We just resolved the issue!

  • Monitoring
    Monitoring

    We implemented a fix and currently monitoring the result.

  • Identified
    Identified

    The MGHPCC main meter detected an ITIC Level 2 power sag event this afternoon.
    The event affected a single power phase.

    Details Time: 3:10:32 PM Percent of Nominal: 65% Duration: 7 cycles (116 milliseconds)

    The sag coincided with severe thunderstorms that passed through the area between 3 and 4PM.

  • Investigating
    Investigating
    • [x] We noticed most of user projects' VMs are in "Shutoff" status and "Shut Down" power state.
    • [x] User can manually start their instances and that will start your VM and put in "Running" power state and "Running" status.
    • [x] We are currently investigating this incident.
NERC system upgrade and maintenance August 16, 2022 10:00 AM – 11:00 PM
  • Completed
    August 16, 2022 at 3:38 PM
    Completed
    August 16, 2022 at 3:38 PM

    Maintenance has completed successfully.

  • Update
    August 16, 2022 at 3:06 PM
    In progress
    August 16, 2022 at 3:06 PM

    Maintenance is still in progress.

  • In progress
    August 16, 2022 at 2:00 PM
    In progress
    August 16, 2022 at 2:00 PM

    Maintenance is now in progress

  • Planned
    August 16, 2022 at 2:00 PM
    Planned
    August 16, 2022 at 2:00 PM

    NERC system upgrade and maintenance will occur on Tuesday August 16, 2022 from 10:00 AM – 11:00 AM.

    GENERAL MAINTENANCE

    – We have a new ticketing support system to submit tickets for support and help.

    – Update and upgrade new version of NERC’s RegApp.

    – Update and upgrade new version of NERC’s ColdFront.

    NOTICES

    – We will roll out our new ticketing support system for NERC, so now users can create a ticket via sending email to help@nerc.mghpcc.org or via our new osTicket website.

    – We update RegApp to reference our new support email address and inform about available Microsoft based identity providers to login and register user accounts.

    – We update ColdFront to restrict project name length when requesting resource allocations.

    – We update ColdFront to reference our support email address i.e. help@nerc.mghpcc.org.

    – We don’t know the ETA for this upgrade but the tentative estimated time is around 1 hours – Please keep an eye on https://nerc.instatus.com/ to get the progress during this time.

    – Please do subscribe to the NERC’s status for any future updates: https://nerc.instatus.com/subscribe/email.

    Our priority is to help make science happen, so If you or your research team have any questions or need to escalate an issue, please contact us via email (help@nerc.mghpcc.org) or, by submitting a new ticket at the NERC’s Support Ticketing System (osTicket).

    Thanks,

    New England Research Cloud (NERC)

    https://nerc.mghpcc.org/

    https://nerc-project.github.io/nerc-docs/

NERC Object Storage/ Swift System Upgrade And Maintenance
  • Resolved
    Resolved

    We just resolved the issue!

  • Monitoring
    Monitoring

    Unfortunately, we’ve discovered a need to substantially reconfigure our object/blob storage/Swift setup. This will make it more optimal and highly available for your project’s needs. Currently there is no ETA for when we’ll deploy the new setup but we anticipate being ready in few weeks. We will be in touch in a future communication about ETA and any migration steps, if necessary, to move data from the old swift setup to a new setup.

    NOTICES

    – Although we will keep the current Swift setup up for now, we highly suggest users do not upload any more data to their object storage/Swift until further notice as there might be migration of data involved. Users may still able to download existing data from their object storage/Swift.

    – We will temporarily not accept/approve change requests on Object Storage(GB) quota from NERC’s ColdFront until further notice.

    – If your project still needs to be uploaded to object storage during this time, please contact us for an ultimate solution.

    – For the Mass Open Cloud (MOC) teams who are trying to migrate their projects to NERC, please contact us to facilitate the migration steps.

    – We don't know the ETA for the cutover to the new setup but we will update status and also communicate any progress on this. We urge you to keep an eye on your email and also on https://nerc.instatus.com/ for future updates regarding object storage/Swift.

    – Please do subscribe to the NERC’s status for any future updates: https://nerc.instatus.com/subscribe/email

    Our priority is to help make science happen, so If you or your research team have any questions or need to escalate an issue, please contact us via email (help@nerc.mghpcc.org).

    Thank you, New England Research Cloud (NERC)

    https://nerc.mghpcc.org/

    https://nerc-project.github.io/nerc-docs/

NERC RegApp not functional and users are not able to register.
  • Resolved
    Resolved

    We just resolved the issue!

  • Investigating
    Investigating

    NERC RegApp is currently experiencing instability and user are not able to register any new account.

    Also, existing users are not allowed to navigate to their user profile.

    Still, any existing users are allowed to login to other web services i.e. NERC's ColdFront and NERC's OpenStack to use NERC resources.

    We are looking into the issue and will update this incident as the situation progresses.

Jul 2022

NERC's ColdFront service upgrade and maintenance July 26, 2022 from 11:00 AM – 12:00 PM
  • Completed
    July 26, 2022 at 3:21 PM
    Completed
    July 26, 2022 at 3:21 PM

    Maintenance has completed successfully.

  • In progress
    July 26, 2022 at 3:00 PM
    In progress
    July 26, 2022 at 3:00 PM

    Maintenance is now in progress

  • Planned
    July 26, 2022 at 3:00 PM
    Planned
    July 26, 2022 at 3:00 PM

    NERC's ColdFront service upgrade and maintenance will occur on Tuesday July 26, 2022 from 11:00 AM – 12:00 PM.

    GENERAL MAINTENANCE

    • We are upgrading and updating new changes on NERC's ColdFront.

    • We can now able to audit and report on any GPU resources usage on your projects.

    NOTICES

    • This will affect other services of NERC other than NERC’s ColdFront web interface.

    • Please do not request or manage NERC resource allocations during this time - please wait till this upgrade is completed.

    • We don’t know the ETA for this upgrade but the tentative estimated time is around 1 hour - so we urge you to keep an eye on https://nerc.instatus.com/ to get the progress during this time.

    • Please do subscribe to the NERC’s status for any future updates: https://nerc.instatus.com/subscribe/email

    Our priority is to help make science happen, so If you or your research team have any questions or need to escalate an issue, please contact us via email (help@nese.mghpcc.org).

    Thanks,

    New England Research Cloud (NERC)

    https://nerc.mghpcc.org/

    https://nerc-project.github.io/nerc-docs/

NERC's ColdFront service upgrade and maintenance July 25, 2022 from 10:00 AM – 11:00 AM
  • Completed
    July 25, 2022 at 3:00 PM
    Completed
    July 25, 2022 at 3:00 PM

    Maintenance has completed successfully

  • In progress
    July 25, 2022 at 2:00 PM
    In progress
    July 25, 2022 at 2:00 PM

    Maintenance is now in progress

  • Planned
    July 25, 2022 at 2:00 PM
    Planned
    July 25, 2022 at 2:00 PM

    NERC's ColdFront service upgrade and maintenance will occur on Monday July 25, 2022 from 10:00 AM – 11:00 AM.

    GENERAL MAINTENANCE

    • We are upgrading and updating new changes on NERC's ColdFront.

    • We can now able to audit and report on any GPU resources usage on your projects.

    NOTICES

    • This will affect other services of NERC other than NERC’s ColdFront web interface.

    • Please do not request or manage NERC resource allocations during this time - please wait till this upgrade is completed.

    • We don’t know the ETA for this upgrade but the tentative estimated time is around 1 hour - so we urge you to keep an eye on https://nerc.instatus.com/ to get the progress during this time.

    • Please do subscribe to the NERC’s status for any future updates: https://nerc.instatus.com/subscribe/email

    Our priority is to help make science happen, so If you or your research team have any questions or need to escalate an issue, please contact us via email (help@nese.mghpcc.org).

    Thanks,

    New England Research Cloud (NERC)

    https://nerc.mghpcc.org/

    https://nerc-project.github.io/nerc-docs/

System upgrade and maintenance July 15, 2022 from 10:00 AM – 11:00 AM
  • Completed
    July 15, 2022 at 2:25 PM
    Completed
    July 15, 2022 at 2:25 PM

    Maintenance and upgrade has completed successfully.

  • In progress
    July 15, 2022 at 2:00 PM
    In progress
    July 15, 2022 at 2:00 PM

    Maintenance is now in progress

  • Planned
    July 15, 2022 at 2:00 PM
    Planned
    July 15, 2022 at 2:00 PM

    NERC upgrade and maintenance will occur on Friday July 15, 2022 from 10:00 AM – 11:00 AM.

    GENERAL MAINTENANCE

    • We are support a new Identity Provider (IdP) i.e. Microsoft as NERC login option.
    • Microsoft will be listed as a new CILogon dropdown option during login while using NERC's web portals i.e. RegApp, ColdFront and OpenStack Dashboard.

    NOTICES

    • This will affect on-boarding new users to NERC as they will not be able to access the NERC's user account registration service (RegApp) web portal.

    • If you are already logged into the system - this will not affect your workflow.

    • If you are not logged in to the system during this time - please wait till this upgrade is completed.

    • We don’t know the ETA for this upgrade but the tentative estimated time is around 1 hour - so we urge you to keep an eye on https://nerc.instatus.com/ to get the progress during this time.

    • Please do subscribe to the NERC’s status for any future updates: https://nerc.instatus.com/subscribe/email

    Our priority is to help make science happen, so If you or your research team have any questions or need to escalate an issue, please contact us via email (help@nese.mghpcc.org).

    Thanks,

    New England Research Cloud (NERC)

    https://nerc.mghpcc.org/

    https://nerc-project.github.io/nerc-docs/

The MGHPCC main meter detected an ITIC Level 2 power sag event.
  • Resolved
    Resolved

    We just resolved the issue!

  • Monitoring
    Monitoring

    We implemented a fix and currently monitoring the result.

  • Investigating
    Investigating

    The MGHPCC main meter detected an ITIC Level 2 power sag event yesterday afternoon. The event affected a single power phase. Details Time: 5:52:23 PM Percent of Nominal: 66% Duration: 6 cycles (100 milliseconds)

    The sag coincided with severe thunderstorms that passed through the area between 5 and 7PM.

    Notices:

    • Some hypervisors are shutoff and some of them were rebooted.

    We are currently investigating this incident.

Jun 2022

MGHPCC Shared Services Account Portal a.k.a. RegApp upgrade and maintenance June 15, 2022 from 10:00 AM – 12:00 PM
  • Completed
    June 15, 2022 at 2:25 PM
    Completed
    June 15, 2022 at 2:25 PM

    Maintenance has completed successfully.

  • In progress
    June 15, 2022 at 2:00 PM
    In progress
    June 15, 2022 at 2:00 PM

    Maintenance is now in progress

  • Planned
    June 15, 2022 at 2:00 PM
    Planned
    June 15, 2022 at 2:00 PM

    NERC’s MGHPCC Shared Services Account Portal a.k.a. RegApp upgrade and maintenance will occur on Wednesday June 15, 2022 from 10:00 AM – 12:00 PM.

    GENERAL MAINTENANCE

    • We will be rebuilding regapp that includes fixes for user logout issues. This will cause some outages to user registration service.

    NOTICES

    • This will affect on-boarding new users to NERC as they will not be able to access the NERC's user account registration service (RegApp) web portal.

    • All other NERC services will be functional during this maintenance period.

    • We don’t know the ETA for this upgrade but the tentative estimated time is around 2 hours - so we urge you to keep an eye on https://nerc.instatus.com/ to get the progress during this time.

    • Please do subscribe to the NERC’s status for any future updates: https://nerc.instatus.com/subscribe/email

    Our priority is to help make science happen, so If you or your research team have any questions or need to escalate an issue, please contact us via email (help@nese.mghpcc.org).

    Thanks,

    New England Research Cloud (NERC)

    https://nerc.mghpcc.org/

    https://nerc-project.github.io/nerc-docs/

The MGHPCC main meter detected an ITIC Level 2 power sag
  • Resolved
    Resolved

    We just resolved the issue!

  • Monitoring
    Monitoring

    We implemented a fix and currently monitoring the result.

  • Identified
    Identified

    The MGHPCC main meter detected an ITIC Level 2 power sag events this morning. The event affected a single power phase.

    Details: Time: 04:45:31AM Percent of Nominal: 68% Duration: 3 cycles (50 milliseconds)

    Notices:

    • NERCs OpenStack instances were shut down.
    • Some hypervisors are rebooted.
    • NERC Openshift cluster is not accessible.

    We are currently investigating this incident.

Jun 2022 to Aug 2022

Next

For issues not shown here, please contact NERC via email help@nerc.mghpcc.org or, using the NERC's Support Ticketing System.