NERC - Notice history

All systems operational

GETTING HELP
Email: help@nerc.mghpcc.org or, using the NERC's Support Ticketing System
NERC Documentation: https://nerc-project.github.io/nerc-docs/
Status page for the New England Research Cloud (NERC) and other resources.
Please scroll down to see details on any Incidents or maintenance notices.

MGHPCC SHARED SERVICES (MGHPCC-SS) ACCOUNT PORTAL - Operational

98% - uptime
Aug 2022 · 93.23%Sep · 100.0%Oct · 100.0%

Previous page

Next page

Aug 2022
Sep 2022
Oct 2022

NERC COLDFRONT - Operational

100% - uptime
Aug 2022 · 100.0%Sep · 100.0%Oct · 100.0%

Previous page

Next page

Aug 2022
Sep 2022
Oct 2022

NETWORKING - Operational

100% - uptime
Aug 2022 · 100.0%Sep · 100.0%Oct · 100.0%

Previous page

Next page

Aug 2022
Sep 2022
Oct 2022

STORAGE - Operational

100% - uptime
Aug 2022 · 100.0%Sep · 100.0%Oct · 100.0%

Previous page

Next page

Aug 2022
Sep 2022
Oct 2022
100% - uptime

NERC WEBSITE - Operational

100% - uptime
Aug 2022 · 100.0%Sep · 100.0%Oct · 100.0%

Previous page

Next page

Aug 2022
Sep 2022
Oct 2022

NERC DOCUMENTATION WEBSITE - Operational

100% - uptime
Aug 2022 · 100.0%Sep · 100.0%Oct · 100.0%

Previous page

Next page

Aug 2022
Sep 2022
Oct 2022

NERC TICKETING SYSTEM - Operational

100% - uptime
Aug 2022 · 100.0%Sep · 100.0%Oct · 99.89%

Previous page

Next page

Aug 2022
Sep 2022
Oct 2022

Notice history

Oct 2022

OTHER WEBSITES & TOOLS - NERC TICKETING SYSTEM isn't accessible at the moment (automated)
  • Resolved
    Resolved

    OTHER WEBSITES & TOOLS - NERC TICKETING SYSTEM is now operational! This update was created by an automated monitoring service.

  • Investigating
    Investigating

    OTHER WEBSITES & TOOLS - NERC TICKETING SYSTEM cannot be accessed at the moment. This incident was created by an automated monitoring service.

Inconsistency between approved allocation Volume quota on NERC's ColdFront web interface vs OpenStack dashboard.
  • Resolved
    Resolved
    • Eventhough, your volume quota on ColdFront resource allocations are approved, those values are not correctly reflected on your NERC"s OpenStack Horizon Dashboard. Eventhough you will be able to use the shown volume space on your NERC Dashboard.
    • We will not be approving Volume Quota change until this issue is fixed!
      We are currently investigating this incident.
Upgrade NERC OpenStack Platform to version 16.2.3 October 13, 2022 9AM – 5PM
  • Completed
    October 13, 2022 at 9:02 PM
    Completed
    October 13, 2022 at 9:02 PM

    Maintenance has completed successfully.

  • In progress
    October 13, 2022 at 1:02 PM
    In progress
    October 13, 2022 at 1:02 PM

    Maintenance is now in progress.

  • Planned
    October 13, 2022 at 1:00 PM
    Planned
    October 13, 2022 at 1:00 PM

    Upgrade NERC OpenStack Platform to version 16.2.3 will occur on Thursday October 13, 2022 9AM – 5PM.

    GENERAL MAINTENANCE

    • We will be upgrading our RedHat OpenStack Platform deployment to the latest minor release 16.2.3.

    NOTICES

    • NERC OpenStack’s Horizon dashboard may be inaccessible at times during this process.

    • Please be aware that during this maintenance, project and project change requests will not be accepted and approved.

    • We are reserving the whole day, from 9am to 5pm, as the ETA for the cutover to the new setup but we will update status and also communicate any progress on this. We urge you to keep an eye on your email and also on https://nerc.instatus.com/ for future updates regarding object storage/Swift.

    • Please do subscribe to the NERC’s status for any future updates: https://nerc.instatus.com/subscribe/email

    Our priority is to help make science happen, so If you or your research team have any questions or need to escalate an issue, please contact the NERC Help Desk for assistance or email us.

    Thanks,

    New England Research Cloud (NERC)

    https://nerc.mghpcc.org/

    https://nerc-project.github.io/nerc-docs/

    https://mghpcc.supportsystem.com/

NERC Ticketing System is not working properly!
  • Resolved
    Resolved

    We just resolved the issue!

  • Monitoring
    Monitoring

    We implemented a fix and currently monitoring the result.

  • Investigating
    Investigating
    • We recently noticed that any emails sent to help@nerc.mghpcc.org are not creating any new tickets.
    • Please beware, even you send out an help request or issues to help@nerc.mghpcc.org. We may not have received those correspondance so unable to review and reply those emails.
    • Any reply from NERC's osTicketing platform are not sent out to the users.
OTHER WEBSITES & TOOLS - NERC TICKETING SYSTEM isn't accessible at the moment (automated)
  • Resolved
    Resolved

    OTHER WEBSITES & TOOLS - NERC TICKETING SYSTEM is now operational! This update was created by an automated monitoring service.

  • Investigating
    Investigating

    OTHER WEBSITES & TOOLS - NERC TICKETING SYSTEM cannot be accessed at the moment. This incident was created by an automated monitoring service.

Sep 2022

NERC Horizon: https://stack.nerc.mghpcc.org/ is not accessible
  • Resolved
    Resolved

    We just resolved the issue!

NERC Swift API upgrade and maintenance September 28, 2022 9:00 AM – 5:00 PM
  • Completed
    September 28, 2022 at 8:57 PM
    Completed
    September 28, 2022 at 8:57 PM

    Maintenance has completed successfully.

  • Update
    September 28, 2022 at 8:00 PM
    In progress
    September 28, 2022 at 8:00 PM

    We are testing out this upgraded NERC Swift API.

  • In progress
    September 28, 2022 at 1:00 PM
    In progress
    September 28, 2022 at 1:00 PM

    Maintenance is now in progress

  • Planned
    September 28, 2022 at 1:00 PM
    Planned
    September 28, 2022 at 1:00 PM

    NERC Swift API upgrade and maintenance will occur on Wednesday September 28, 2022 9:00 AM - 5:00 PM.

    GENERAL MAINTENANCE

    • NERC Swift API support for CEPH RADOS Gateway backed by NESE storage.

    NOTICES

    • We will stop the existing object/blob storage/Swift setup and will switch to a new CEPH RADOS gateway backend.

    • All existing data from all of the object storage/Swift of projects will be transferred to the new storage service.

    • We will soon accept/approve change requests on Object Storage(GB) quota from NERC’s ColdFront.

    • Please be aware that during this maintenance, project and project change requests will not be accepted and approved.

    • We are reserving the whole day as the ETA for the cutover to the new setup but we will update status and also communicate any progress on this. We urge you to keep an eye on your email and also on https://nerc.instatus.com/ for future updates regarding object storage/Swift.

    • Please do subscribe to the NERC’s status for any future updates: https://nerc.instatus.com/subscribe/email

    Our priority is to help make science happen, so If you or your research team have any questions or need to escalate an issue, please contact the NERC Help Desk for assistance or email us.

    Thanks,

    New England Research Cloud (NERC)

    https://nerc.mghpcc.org/

    https://nerc-project.github.io/nerc-docs/

    https://mghpcc.supportsystem.com/

NERC Swift API upgrade and maintenance September 21, 2022 9:00 AM – 5:00 PM
  • Completed
    September 21, 2022 at 9:00 PM
    Completed
    September 21, 2022 at 9:00 PM

    Maintenance has completed successfully

  • In progress
    September 21, 2022 at 1:00 PM
    In progress
    September 21, 2022 at 1:00 PM

    Maintenance is now in progress

  • Planned
    September 21, 2022 at 1:00 PM
    Planned
    September 21, 2022 at 1:00 PM

    NERC Swift API upgrade and maintenance will occur on Wednesday September 21, 2022 9:00 AM - 5:00 PM.

    GENERAL MAINTENANCE

    • NERC Swift API support for CEPH RADOS Gateway backed by NESE storage.

    NOTICES

    • We will stop the existing object/blob storage/Swift setup and will switch to a new CEPH RADOS gateway backend.

    • All existing data from all of the object storage/Swift of projects will be transferred to the new storage service.

    • We will soon accept/approve change requests on Object Storage(GB) quota from NERC’s ColdFront.

    • Please be aware that during this maintenance, project and project change requests will not be accepted and approved.

    • We are reserving the whole day as the ETA for the cutover to the new setup but we will update status and also communicate any progress on this. We urge you to keep an eye on your email and also on https://nerc.instatus.com/ for future updates regarding object storage/Swift.

    • Please do subscribe to the NERC’s status for any future updates: https://nerc.instatus.com/subscribe/email

    Our priority is to help make science happen, so If you or your research team have any questions or need to escalate an issue, please contact the NERC Help Desk for assistance or email us.

    Thanks,

    New England Research Cloud (NERC)

    https://nerc.mghpcc.org/

    https://nerc-project.github.io/nerc-docs/

    https://mghpcc.supportsystem.com/

NERC's MGHPCC Shared Services Account Portal a.k.a. RegApp service upgrade and maintenance Sep 13, 2022 from 10:00 AM – 11:00 AM
  • Completed
    September 13, 2022 at 3:00 PM
    Completed
    September 13, 2022 at 3:00 PM

    Maintenance has completed successfully

  • In progress
    September 13, 2022 at 2:00 PM
    In progress
    September 13, 2022 at 2:00 PM

    Maintenance is now in progress

  • Planned
    September 13, 2022 at 2:00 PM
    Planned
    September 13, 2022 at 2:00 PM

    NERC's MGHPCC Shared Services Account Portal a.k.a. RegApp service upgrade and maintenance will occur on Tuesday September 13, 2022 10:00 AM - 11:00 AM.

    GENERAL MAINTENANCE

    • We are upgrading and updating NERC's RegApp to apply new changes.

    • We will be rebuilding RegApp to add new identity providers (IdPs). This will expand the ability to register and log in to NERC from the broader community.

    NOTICES

    • This will allow user registration from other MGHPCC partner institutions i.e. Northeastern University and UMass.

    • This will allow user registration and login from Harvard Business School.

    • This can cause outages to user registration and login. Please avoid login during this maintenance.

    • All other NERC services that depends on NERC's RegApp login system i.e. ColdFront, OpenStack Dashboard (Horizon) services will also be affected during this maintenance period.

    • The estimated time to complete for this upgrade is 1 hour. It can take longer; so we urge you to keep an eye on https://nerc.instatus.com/ to get the progress during this time.

    • Please do subscribe to the NERC’s status for any future updates: https://nerc.instatus.com/subscribe/email

    Our priority is to help make science happen, so If you or your research team have any questions or need to escalate an issue, please contact the NERC Help Desk for assistance or email us.

    Thanks,

    New England Research Cloud (NERC)

    https://nerc.mghpcc.org/

    https://nerc-project.github.io/nerc-docs/

    https://mghpcc.supportsystem.com/

Network issues affecting access to/from MGHPCC.
  • Resolved
    Resolved

    We just resolved the issue!

  • Monitoring
    Monitoring

    We implemented a fix and currently monitoring the result.

  • Identified
    Identified

    We are continuing to work on a fix for this incident.

  • Resolved
    Resolved

    • We noticed users are unable to access NERC's OpenStack Dashboard (Horizon) after they login.

    • It may affect users' OpenStack running VMs.

    • We are currently investigating this incident. We will gather more details as we inspect more details about this issue.

Aug 2022

Most of the NERC VMs are in "Shutoff" mode
  • Resolved
    Resolved

    We just resolved the issue!

  • Monitoring
    Monitoring

    We implemented a fix and currently monitoring the result.

  • Identified
    Identified

    The MGHPCC main meter detected an ITIC Level 2 power sag event this afternoon.
    The event affected a single power phase.

    Details Time: 3:10:32 PM Percent of Nominal: 65% Duration: 7 cycles (116 milliseconds)

    The sag coincided with severe thunderstorms that passed through the area between 3 and 4PM.

  • Investigating
    Investigating
    • [x] We noticed most of user projects' VMs are in "Shutoff" status and "Shut Down" power state.
    • [x] User can manually start their instances and that will start your VM and put in "Running" power state and "Running" status.
    • [x] We are currently investigating this incident.
NERC system upgrade and maintenance August 16, 2022 10:00 AM – 11:00 PM
  • Completed
    August 16, 2022 at 3:38 PM
    Completed
    August 16, 2022 at 3:38 PM

    Maintenance has completed successfully.

  • Update
    August 16, 2022 at 3:06 PM
    In progress
    August 16, 2022 at 3:06 PM

    Maintenance is still in progress.

  • In progress
    August 16, 2022 at 2:00 PM
    In progress
    August 16, 2022 at 2:00 PM

    Maintenance is now in progress

  • Planned
    August 16, 2022 at 2:00 PM
    Planned
    August 16, 2022 at 2:00 PM

    NERC system upgrade and maintenance will occur on Tuesday August 16, 2022 from 10:00 AM – 11:00 AM.

    GENERAL MAINTENANCE

    – We have a new ticketing support system to submit tickets for support and help.

    – Update and upgrade new version of NERC’s RegApp.

    – Update and upgrade new version of NERC’s ColdFront.

    NOTICES

    – We will roll out our new ticketing support system for NERC, so now users can create a ticket via sending email to help@nerc.mghpcc.org or via our new osTicket website.

    – We update RegApp to reference our new support email address and inform about available Microsoft based identity providers to login and register user accounts.

    – We update ColdFront to restrict project name length when requesting resource allocations.

    – We update ColdFront to reference our support email address i.e. help@nerc.mghpcc.org.

    – We don’t know the ETA for this upgrade but the tentative estimated time is around 1 hours – Please keep an eye on https://nerc.instatus.com/ to get the progress during this time.

    – Please do subscribe to the NERC’s status for any future updates: https://nerc.instatus.com/subscribe/email.

    Our priority is to help make science happen, so If you or your research team have any questions or need to escalate an issue, please contact us via email (help@nerc.mghpcc.org) or, by submitting a new ticket at the NERC’s Support Ticketing System (osTicket).

    Thanks,

    New England Research Cloud (NERC)

    https://nerc.mghpcc.org/

    https://nerc-project.github.io/nerc-docs/

NERC Object Storage/ Swift System Upgrade And Maintenance
  • Resolved
    Resolved

    We just resolved the issue!

  • Monitoring
    Monitoring

    Unfortunately, we’ve discovered a need to substantially reconfigure our object/blob storage/Swift setup. This will make it more optimal and highly available for your project’s needs. Currently there is no ETA for when we’ll deploy the new setup but we anticipate being ready in few weeks. We will be in touch in a future communication about ETA and any migration steps, if necessary, to move data from the old swift setup to a new setup.

    NOTICES

    – Although we will keep the current Swift setup up for now, we highly suggest users do not upload any more data to their object storage/Swift until further notice as there might be migration of data involved. Users may still able to download existing data from their object storage/Swift.

    – We will temporarily not accept/approve change requests on Object Storage(GB) quota from NERC’s ColdFront until further notice.

    – If your project still needs to be uploaded to object storage during this time, please contact us for an ultimate solution.

    – For the Mass Open Cloud (MOC) teams who are trying to migrate their projects to NERC, please contact us to facilitate the migration steps.

    – We don't know the ETA for the cutover to the new setup but we will update status and also communicate any progress on this. We urge you to keep an eye on your email and also on https://nerc.instatus.com/ for future updates regarding object storage/Swift.

    – Please do subscribe to the NERC’s status for any future updates: https://nerc.instatus.com/subscribe/email

    Our priority is to help make science happen, so If you or your research team have any questions or need to escalate an issue, please contact us via email (help@nerc.mghpcc.org).

    Thank you, New England Research Cloud (NERC)

    https://nerc.mghpcc.org/

    https://nerc-project.github.io/nerc-docs/

NERC RegApp not functional and users are not able to register.
  • Resolved
    Resolved

    We just resolved the issue!

  • Investigating
    Investigating

    NERC RegApp is currently experiencing instability and user are not able to register any new account.

    Also, existing users are not allowed to navigate to their user profile.

    Still, any existing users are allowed to login to other web services i.e. NERC's ColdFront and NERC's OpenStack to use NERC resources.

    We are looking into the issue and will update this incident as the situation progresses.

Aug 2022 to Oct 2022

Next

For issues not shown here, please contact NERC via email help@nerc.mghpcc.org or, using the NERC's Support Ticketing System.