NERC - Notice history

All systems operational

GETTING HELP
Email: help@nerc.mghpcc.org or, using the NERC's Support Ticketing System
NERC Documentation: https://nerc-project.github.io/nerc-docs/
Status page for the New England Research Cloud (NERC) and other resources.
Please scroll down to see details on any Incidents or maintenance notices.

MGHPCC SHARED SERVICES (MGHPCC-SS) ACCOUNT PORTAL - Operational

98% - uptime
Jul 2022 · 100.0%Aug · 93.23%Sep · 100.0%

Previous page

Next page

Jul 2022
Aug 2022
Sep 2022

NERC COLDFRONT - Operational

100% - uptime
Jul 2022 · 100.0%Aug · 100.0%Sep · 100.0%

Previous page

Next page

Jul 2022
Aug 2022
Sep 2022

NETWORKING - Operational

100% - uptime
Jul 2022 · 99.79%Aug · 100.0%Sep · 100.0%

Previous page

Next page

Jul 2022
Aug 2022
Sep 2022

STORAGE - Operational

100% - uptime
Jul 2022 · 99.79%Aug · 100.0%Sep · 100.0%

Previous page

Next page

Jul 2022
Aug 2022
Sep 2022
100% - uptime

NERC WEBSITE - Operational

100% - uptime
Jul 2022 · 100.0%Aug · 100.0%Sep · 100.0%

Previous page

Next page

Jul 2022
Aug 2022
Sep 2022

NERC DOCUMENTATION WEBSITE - Operational

100% - uptime
Jul 2022 · 100.0%Aug · 100.0%Sep · 100.0%

Previous page

Next page

Jul 2022
Aug 2022
Sep 2022

NERC TICKETING SYSTEM - Operational

100% - uptime
Jul 2022 · 100.0%Aug · 100.0%Sep · 100.0%

Previous page

Next page

Jul 2022
Aug 2022
Sep 2022

Notice history

Sep 2022

NERC Horizon: https://stack.nerc.mghpcc.org/ is not accessible
  • Resolved
    Resolved

    We just resolved the issue!

NERC Swift API upgrade and maintenance September 28, 2022 9:00 AM – 5:00 PM
  • Completed
    September 28, 2022 at 8:57 PM
    Completed
    September 28, 2022 at 8:57 PM

    Maintenance has completed successfully.

  • Update
    September 28, 2022 at 8:00 PM
    Update
    September 28, 2022 at 8:00 PM

    We are testing out this upgraded NERC Swift API.

  • In progress
    September 28, 2022 at 1:00 PM
    In progress
    September 28, 2022 at 1:00 PM

    Maintenance is now in progress

  • Planned
    September 28, 2022 at 1:00 PM
    Planned
    September 28, 2022 at 1:00 PM

    NERC Swift API upgrade and maintenance will occur on Wednesday September 28, 2022 9:00 AM - 5:00 PM.

    GENERAL MAINTENANCE

    • NERC Swift API support for CEPH RADOS Gateway backed by NESE storage.

    NOTICES

    • We will stop the existing object/blob storage/Swift setup and will switch to a new CEPH RADOS gateway backend.

    • All existing data from all of the object storage/Swift of projects will be transferred to the new storage service.

    • We will soon accept/approve change requests on Object Storage(GB) quota from NERC’s ColdFront.

    • Please be aware that during this maintenance, project and project change requests will not be accepted and approved.

    • We are reserving the whole day as the ETA for the cutover to the new setup but we will update status and also communicate any progress on this. We urge you to keep an eye on your email and also on https://nerc.instatus.com/ for future updates regarding object storage/Swift.

    • Please do subscribe to the NERC’s status for any future updates: https://nerc.instatus.com/subscribe/email

    Our priority is to help make science happen, so If you or your research team have any questions or need to escalate an issue, please contact the NERC Help Desk for assistance or email us.

    Thanks,

    New England Research Cloud (NERC)

    https://nerc.mghpcc.org/

    https://nerc-project.github.io/nerc-docs/

    https://mghpcc.supportsystem.com/

NERC Swift API upgrade and maintenance September 21, 2022 9:00 AM – 5:00 PM
  • Completed
    September 21, 2022 at 9:00 PM
    Completed
    September 21, 2022 at 9:00 PM

    Maintenance has completed successfully

  • In progress
    September 21, 2022 at 1:00 PM
    In progress
    September 21, 2022 at 1:00 PM

    Maintenance is now in progress

  • Planned
    September 21, 2022 at 1:00 PM
    Planned
    September 21, 2022 at 1:00 PM

    NERC Swift API upgrade and maintenance will occur on Wednesday September 21, 2022 9:00 AM - 5:00 PM.

    GENERAL MAINTENANCE

    • NERC Swift API support for CEPH RADOS Gateway backed by NESE storage.

    NOTICES

    • We will stop the existing object/blob storage/Swift setup and will switch to a new CEPH RADOS gateway backend.

    • All existing data from all of the object storage/Swift of projects will be transferred to the new storage service.

    • We will soon accept/approve change requests on Object Storage(GB) quota from NERC’s ColdFront.

    • Please be aware that during this maintenance, project and project change requests will not be accepted and approved.

    • We are reserving the whole day as the ETA for the cutover to the new setup but we will update status and also communicate any progress on this. We urge you to keep an eye on your email and also on https://nerc.instatus.com/ for future updates regarding object storage/Swift.

    • Please do subscribe to the NERC’s status for any future updates: https://nerc.instatus.com/subscribe/email

    Our priority is to help make science happen, so If you or your research team have any questions or need to escalate an issue, please contact the NERC Help Desk for assistance or email us.

    Thanks,

    New England Research Cloud (NERC)

    https://nerc.mghpcc.org/

    https://nerc-project.github.io/nerc-docs/

    https://mghpcc.supportsystem.com/

NERC's MGHPCC Shared Services Account Portal a.k.a. RegApp service upgrade and maintenance Sep 13, 2022 from 10:00 AM – 11:00 AM
  • Completed
    September 13, 2022 at 3:00 PM
    Completed
    September 13, 2022 at 3:00 PM

    Maintenance has completed successfully

  • In progress
    September 13, 2022 at 2:00 PM
    In progress
    September 13, 2022 at 2:00 PM

    Maintenance is now in progress

  • Planned
    September 13, 2022 at 2:00 PM
    Planned
    September 13, 2022 at 2:00 PM

    NERC's MGHPCC Shared Services Account Portal a.k.a. RegApp service upgrade and maintenance will occur on Tuesday September 13, 2022 10:00 AM - 11:00 AM.

    GENERAL MAINTENANCE

    • We are upgrading and updating NERC's RegApp to apply new changes.

    • We will be rebuilding RegApp to add new identity providers (IdPs). This will expand the ability to register and log in to NERC from the broader community.

    NOTICES

    • This will allow user registration from other MGHPCC partner institutions i.e. Northeastern University and UMass.

    • This will allow user registration and login from Harvard Business School.

    • This can cause outages to user registration and login. Please avoid login during this maintenance.

    • All other NERC services that depends on NERC's RegApp login system i.e. ColdFront, OpenStack Dashboard (Horizon) services will also be affected during this maintenance period.

    • The estimated time to complete for this upgrade is 1 hour. It can take longer; so we urge you to keep an eye on https://nerc.instatus.com/ to get the progress during this time.

    • Please do subscribe to the NERC’s status for any future updates: https://nerc.instatus.com/subscribe/email

    Our priority is to help make science happen, so If you or your research team have any questions or need to escalate an issue, please contact the NERC Help Desk for assistance or email us.

    Thanks,

    New England Research Cloud (NERC)

    https://nerc.mghpcc.org/

    https://nerc-project.github.io/nerc-docs/

    https://mghpcc.supportsystem.com/

Network issues affecting access to/from MGHPCC.
  • Resolved
    Resolved

    We just resolved the issue!

  • Monitoring
    Monitoring

    We implemented a fix and currently monitoring the result.

  • Identified
    Identified

    We are continuing to work on a fix for this incident.

  • Resolved
    Resolved

    • We noticed users are unable to access NERC's OpenStack Dashboard (Horizon) after they login.

    • It may affect users' OpenStack running VMs.

    • We are currently investigating this incident. We will gather more details as we inspect more details about this issue.

Aug 2022

Most of the NERC VMs are in "Shutoff" mode
  • Resolved
    Resolved

    We just resolved the issue!

  • Monitoring
    Monitoring

    We implemented a fix and currently monitoring the result.

  • Identified
    Identified

    The MGHPCC main meter detected an ITIC Level 2 power sag event this afternoon.
    The event affected a single power phase.

    Details Time: 3:10:32 PM Percent of Nominal: 65% Duration: 7 cycles (116 milliseconds)

    The sag coincided with severe thunderstorms that passed through the area between 3 and 4PM.

  • Investigating
    Investigating
    • [x] We noticed most of user projects' VMs are in "Shutoff" status and "Shut Down" power state.
    • [x] User can manually start their instances and that will start your VM and put in "Running" power state and "Running" status.
    • [x] We are currently investigating this incident.
NERC system upgrade and maintenance August 16, 2022 10:00 AM – 11:00 PM
  • Completed
    August 16, 2022 at 3:38 PM
    Completed
    August 16, 2022 at 3:38 PM

    Maintenance has completed successfully.

  • Update
    August 16, 2022 at 3:06 PM
    Update
    August 16, 2022 at 3:06 PM

    Maintenance is still in progress.

  • In progress
    August 16, 2022 at 2:00 PM
    In progress
    August 16, 2022 at 2:00 PM

    Maintenance is now in progress

  • Planned
    August 16, 2022 at 2:00 PM
    Planned
    August 16, 2022 at 2:00 PM

    NERC system upgrade and maintenance will occur on Tuesday August 16, 2022 from 10:00 AM – 11:00 AM.

    GENERAL MAINTENANCE

    – We have a new ticketing support system to submit tickets for support and help.

    – Update and upgrade new version of NERC’s RegApp.

    – Update and upgrade new version of NERC’s ColdFront.

    NOTICES

    – We will roll out our new ticketing support system for NERC, so now users can create a ticket via sending email to help@nerc.mghpcc.org or via our new osTicket website.

    – We update RegApp to reference our new support email address and inform about available Microsoft based identity providers to login and register user accounts.

    – We update ColdFront to restrict project name length when requesting resource allocations.

    – We update ColdFront to reference our support email address i.e. help@nerc.mghpcc.org.

    – We don’t know the ETA for this upgrade but the tentative estimated time is around 1 hours – Please keep an eye on https://nerc.instatus.com/ to get the progress during this time.

    – Please do subscribe to the NERC’s status for any future updates: https://nerc.instatus.com/subscribe/email.

    Our priority is to help make science happen, so If you or your research team have any questions or need to escalate an issue, please contact us via email (help@nerc.mghpcc.org) or, by submitting a new ticket at the NERC’s Support Ticketing System (osTicket).

    Thanks,

    New England Research Cloud (NERC)

    https://nerc.mghpcc.org/

    https://nerc-project.github.io/nerc-docs/

NERC Object Storage/ Swift System Upgrade And Maintenance
  • Resolved
    Resolved

    We just resolved the issue!

  • Monitoring
    Monitoring

    Unfortunately, we’ve discovered a need to substantially reconfigure our object/blob storage/Swift setup. This will make it more optimal and highly available for your project’s needs. Currently there is no ETA for when we’ll deploy the new setup but we anticipate being ready in few weeks. We will be in touch in a future communication about ETA and any migration steps, if necessary, to move data from the old swift setup to a new setup.

    NOTICES

    – Although we will keep the current Swift setup up for now, we highly suggest users do not upload any more data to their object storage/Swift until further notice as there might be migration of data involved. Users may still able to download existing data from their object storage/Swift.

    – We will temporarily not accept/approve change requests on Object Storage(GB) quota from NERC’s ColdFront until further notice.

    – If your project still needs to be uploaded to object storage during this time, please contact us for an ultimate solution.

    – For the Mass Open Cloud (MOC) teams who are trying to migrate their projects to NERC, please contact us to facilitate the migration steps.

    – We don't know the ETA for the cutover to the new setup but we will update status and also communicate any progress on this. We urge you to keep an eye on your email and also on https://nerc.instatus.com/ for future updates regarding object storage/Swift.

    – Please do subscribe to the NERC’s status for any future updates: https://nerc.instatus.com/subscribe/email

    Our priority is to help make science happen, so If you or your research team have any questions or need to escalate an issue, please contact us via email (help@nerc.mghpcc.org).

    Thank you, New England Research Cloud (NERC)

    https://nerc.mghpcc.org/

    https://nerc-project.github.io/nerc-docs/

NERC RegApp not functional and users are not able to register.
  • Resolved
    Resolved

    We just resolved the issue!

  • Investigating
    Investigating

    NERC RegApp is currently experiencing instability and user are not able to register any new account.

    Also, existing users are not allowed to navigate to their user profile.

    Still, any existing users are allowed to login to other web services i.e. NERC's ColdFront and NERC's OpenStack to use NERC resources.

    We are looking into the issue and will update this incident as the situation progresses.

Jul 2022

NERC's ColdFront service upgrade and maintenance July 26, 2022 from 11:00 AM – 12:00 PM
  • Completed
    July 26, 2022 at 3:21 PM
    Completed
    July 26, 2022 at 3:21 PM

    Maintenance has completed successfully.

  • In progress
    July 26, 2022 at 3:00 PM
    In progress
    July 26, 2022 at 3:00 PM

    Maintenance is now in progress

  • Planned
    July 26, 2022 at 3:00 PM
    Planned
    July 26, 2022 at 3:00 PM

    NERC's ColdFront service upgrade and maintenance will occur on Tuesday July 26, 2022 from 11:00 AM – 12:00 PM.

    GENERAL MAINTENANCE

    • We are upgrading and updating new changes on NERC's ColdFront.

    • We can now able to audit and report on any GPU resources usage on your projects.

    NOTICES

    • This will affect other services of NERC other than NERC’s ColdFront web interface.

    • Please do not request or manage NERC resource allocations during this time - please wait till this upgrade is completed.

    • We don’t know the ETA for this upgrade but the tentative estimated time is around 1 hour - so we urge you to keep an eye on https://nerc.instatus.com/ to get the progress during this time.

    • Please do subscribe to the NERC’s status for any future updates: https://nerc.instatus.com/subscribe/email

    Our priority is to help make science happen, so If you or your research team have any questions or need to escalate an issue, please contact us via email (help@nese.mghpcc.org).

    Thanks,

    New England Research Cloud (NERC)

    https://nerc.mghpcc.org/

    https://nerc-project.github.io/nerc-docs/

NERC's ColdFront service upgrade and maintenance July 25, 2022 from 10:00 AM – 11:00 AM
  • Completed
    July 25, 2022 at 3:00 PM
    Completed
    July 25, 2022 at 3:00 PM

    Maintenance has completed successfully

  • In progress
    July 25, 2022 at 2:00 PM
    In progress
    July 25, 2022 at 2:00 PM

    Maintenance is now in progress

  • Planned
    July 25, 2022 at 2:00 PM
    Planned
    July 25, 2022 at 2:00 PM

    NERC's ColdFront service upgrade and maintenance will occur on Monday July 25, 2022 from 10:00 AM – 11:00 AM.

    GENERAL MAINTENANCE

    • We are upgrading and updating new changes on NERC's ColdFront.

    • We can now able to audit and report on any GPU resources usage on your projects.

    NOTICES

    • This will affect other services of NERC other than NERC’s ColdFront web interface.

    • Please do not request or manage NERC resource allocations during this time - please wait till this upgrade is completed.

    • We don’t know the ETA for this upgrade but the tentative estimated time is around 1 hour - so we urge you to keep an eye on https://nerc.instatus.com/ to get the progress during this time.

    • Please do subscribe to the NERC’s status for any future updates: https://nerc.instatus.com/subscribe/email

    Our priority is to help make science happen, so If you or your research team have any questions or need to escalate an issue, please contact us via email (help@nese.mghpcc.org).

    Thanks,

    New England Research Cloud (NERC)

    https://nerc.mghpcc.org/

    https://nerc-project.github.io/nerc-docs/

System upgrade and maintenance July 15, 2022 from 10:00 AM – 11:00 AM
  • Completed
    July 15, 2022 at 2:25 PM
    Completed
    July 15, 2022 at 2:25 PM

    Maintenance and upgrade has completed successfully.

  • In progress
    July 15, 2022 at 2:00 PM
    In progress
    July 15, 2022 at 2:00 PM

    Maintenance is now in progress

  • Planned
    July 15, 2022 at 2:00 PM
    Planned
    July 15, 2022 at 2:00 PM

    NERC upgrade and maintenance will occur on Friday July 15, 2022 from 10:00 AM – 11:00 AM.

    GENERAL MAINTENANCE

    • We are support a new Identity Provider (IdP) i.e. Microsoft as NERC login option.
    • Microsoft will be listed as a new CILogon dropdown option during login while using NERC's web portals i.e. RegApp, ColdFront and OpenStack Dashboard.

    NOTICES

    • This will affect on-boarding new users to NERC as they will not be able to access the NERC's user account registration service (RegApp) web portal.

    • If you are already logged into the system - this will not affect your workflow.

    • If you are not logged in to the system during this time - please wait till this upgrade is completed.

    • We don’t know the ETA for this upgrade but the tentative estimated time is around 1 hour - so we urge you to keep an eye on https://nerc.instatus.com/ to get the progress during this time.

    • Please do subscribe to the NERC’s status for any future updates: https://nerc.instatus.com/subscribe/email

    Our priority is to help make science happen, so If you or your research team have any questions or need to escalate an issue, please contact us via email (help@nese.mghpcc.org).

    Thanks,

    New England Research Cloud (NERC)

    https://nerc.mghpcc.org/

    https://nerc-project.github.io/nerc-docs/

The MGHPCC main meter detected an ITIC Level 2 power sag event.
  • Resolved
    Resolved

    We just resolved the issue!

  • Monitoring
    Monitoring

    We implemented a fix and currently monitoring the result.

  • Investigating
    Investigating

    The MGHPCC main meter detected an ITIC Level 2 power sag event yesterday afternoon. The event affected a single power phase. Details Time: 5:52:23 PM Percent of Nominal: 66% Duration: 6 cycles (100 milliseconds)

    The sag coincided with severe thunderstorms that passed through the area between 5 and 7PM.

    Notices:

    • Some hypervisors are shutoff and some of them were rebooted.

    We are currently investigating this incident.

Jul 2022 to Sep 2022

Next

For issues not shown here, please contact NERC via email help@nerc.mghpcc.org or, using the NERC's Support Ticketing System.