Skip to main content
Sign Out

Welcome to Dell

My Account
  • Place orders quickly and easily
  • View orders and track your shipping status
  • Enjoy members-only rewards and discounts
  • Create and access a list of your products
  • Manage your Dell EMC sites, products, and product-level contacts using Company Administration.
Sign In Create an Account Dell Financial Services Premier Sign In Partner Program Sign In
Contact Us

Your Dell.com Carts

Some article numbers may have changed. If this isn't what you're looking for, try searching all articles. Search articles

Article Number: 000193645

PowerStore Alerts: System Full Capacity Alerts

Summary: This article discusses alerts for relating to full capacity Write Protect Mode (WPM) and the resulting alerts.

This article may have been automatically translated. If you have any feedback regarding its quality, please let us know using the form at the bottom of this page.

Article Content

Symptoms

When a PowerStore encounters an issue, an error alert is generated to help identify the issue. This article explains the different possible causes and the relevant remediation for these error alerts. If your issue is not resolved despite the remediation that is provided, check the technical support contact options.
 

SLN317187_en_US__1icon Note: Cannot find the information that you need? Let us know using the feedback form at the bottom of this article.

Cause

When an appliance reaches its full capacity, it enters a condition that is known as Write Protect Mode (WPM). In WPM, all volumes transition to read-only, and a repair of the system is required.

For an explanation of how Physical Capacity is calculated, see KB article 188491: PowerStore: How PowerStore Physical Capacity is calculated.

Resolution

Alert ID DATA_PATH_WRITE_PROTECT_MODE_ENTER
DATA_PATH_APP_EXIT_WPM_WITH_ENOUGH_MD_NOTIFY
DATA_PATH_APP_WPM_SHORT_OF_MD_NOTIFY
Alert Text The <user or system > data capacity has reached its full capacity.
Appliance has exited the write protection state with 16 GB of free capacity.
Appliance cannot exit the Write Protection mode due to insufficient system data capacity.
Error Code (0x00200701)
(0x00201401)
(0x00201501)
Resolution What happens when in Write Protect Mode (WPM)
When an appliance reaches its full capacity, it enters a condition that is known as Write Protect Mode (WPM), where all volumes transition to read-only.
  • The alert 0x00B00702 (alert text: Service level is degraded, Management functions are limited, and I/O is impacted) is raised.
  • Any Write I/O fails
  • Operations such as Snapshots and Replication may be disabled.
  • All metrics (capacity and performance) are unavailable. In PowerStore 3.0, and later, the metric table is updated while in WPM.
Recommendation: the system should not cross 85% threshold.
 

For an explanation of how Physical Capacity is calculated, see KB article 188491: PowerStore: How PowerStore Physical Capacity is calculated.


Cause

The PowerStore Total Physical Capacity is comprised of both User Data and System Data.

On PowerStoreOS 3.0 and above, the following is displayed on PowerStore Manager. The user is presented with a more precise breakdown of both user data and system data:
image.png
On PowerStoreOS below 3.0, the following is displayed on PowerStore Manager. The user is only presented with a breakdown of Physical Used Capacity:

:
image.png

 

When the system reaches 70% capacity, alerts are raised to warn of lack of space. See KB article 000123351: PowerStore Alerts: Capacity Utilization.

If the user takes no action, and the system reaches 100%, the system enters Write Protect Mode (WPM), which indicates that all volumes enter read-only mode, and the alert 0x00200701 is raised.

Alert 0x00200701 indicates space issue with either User Data or System Data.

Note: If a node is down for a prolong period, it may lead to physical capacity increase.

How do we know the root cause for the WPM:

The user can determine the WPM condition root cause based on the alert text:

  • User Data - (0x00200701) The user data has reached its full capacity.
  • System Data - (0x00200701) The system data has reached its full capacity.

According to the type of WPM condition (User Data or System Data) the workaround is determined.



Solution

Recovering from OOS condition depends on the reason for the OOS. The cause of the WPM condition is contained in the alert text:

  • User Data - (0x00200701) The user data has reached its full capacity.
  • System Data - (0x00200701) The system data has reached its full capacity.

Depending on the type of WPM condition (User Data or System Data), the workaround is determined.


 

1. User Data

Use one of the following options:

  • Add Drives - (this is the recommended option).
    • Consult with your account team to analyze your capacity requirements and add more drives.
  • Delete Volumes and or Snapshots.
    • If the user requires information about which volumes or snapshots to delete, escalate to Global Services for guidance.

Additional information

  1. Before you carry out any actions, you should do the following:
    1. Stop all I/O to the array (disable FC/iSCSI ports on the switch) to avoid running into WPM again.
    2. Pause all snapshot schedules, Replication, and any external automation related to volume management.
  2. To get out of WPM condition, the appliance must reclaim at minimum 16 GB of space. To do so, the user must delete a volume with at minimum 16 GB of unique data.
    If the user manages to reclaim such space, and there is sufficient space for System Data, alert 0x00200701 (Alert text: The user data has reached its full capacity) clears, and alert 0x00201401(Alert text: Appliance has exited the write protection state with 16 GB of free capacity) appears.
  3. Once in this state, volumes transition to read/write access, and user should continue reclaiming space until the appliance reaches 85% Used Physical Capacity.
    1. It is highly advised deleting volumes or snapshots one at a time.
    2. See the specific Operating System manual for instructions how to reclaim space from host side.
  4. Once at 85% Used Physical Capacity, the user must manually clear the alert 0x00201401.
  5. Note: Reclaiming space by deleting snapshots or volumes initially requires additional space for System Data, and hence, may lead to alert 0x00201501(Alert text: Appliance cannot exit the Write Protection mode due to insufficient system data capacity).
  6. PowerStoreOS 1.0.3 introduced a new service script (svc_volume_space_metrics) that allows the user to view additional information regarding volume space statistics when appliance is in WPM.
    For more information, see the section "How to use svc_volume_space_metrics" later in this KBA.
 

2. System Data

Use one of the following options:

  • Add Drives - (this is the recommended option).
    • Consult with you account team to analyze your capacity requirements and add more drives.
  • If the user cannot add drives, escalate to technical support for assistance.
This information can be viewed from Volume Capacity Dashboard.
In this example, deleting the below volume reclaims 59.8 GB of space that can be used for either User Data or System Data.

Note that System Data requires a minimum of around 100 GB of unique space to be able to expand. Sometimes it may require 200 GB of space to properly expand.
image.png

Additional information

  1. Usually, when an appliance enters WPM due to System Data, User Data also runs out of space. 
    This is because there is no capacity that is left for expanding (either System Data or User Data).
  2. To get out of WPM, typically the user is required to add additional drives and scale up the appliance.
  3. When the system raises the alert 0x00200701 (Alert text: The System capacity has reached its full capacity.) , it indicates that System Data has run out of space and cannot be expanded.
  4. Avoid deleting volumes in this condition, and escalate to Global Services for support. 



How to use svc_volume_space_metrics

  1. PowerStoreOS 1.0.3 introduced a service script that allows users when in WPM to see additional information about each volume tree.
  2. To run the script, SSH must be enabled on the specified appliance, and the user must connect using SSH to cluster IP using service account.

  3. When trying to run this service script when not in WPM, the service script fails.

  4. When appliance runs out of space, the user can run the script and see additional information, such as:
    Family Unique Physical Used - This represents the unique data (compressed) this volume, and all its family (snapshots and clones) release if this volume is deleted.
    Example: 

    [SVC:service@68R1BW2-A user]$ svc_volume_space_metrics
2020-10-29 12:28:33,997 - INFO - event_client init() address: localhost node: A
2020-10-29 12:28:34,248 - INFO -  PyCycloneDp.init localhost
2020-10-29 12:28:34,318 - PyCycloneDp - INFO - logging in PyCycloneDp.__init__
  
Namespace state is ro-oos
Retrieve family space stats from DP namespace. It will take some time...
2020-10-29 12:28:35,972 - INFO - event_client init() address: localhost node: A
2020-10-29 12:28:36,175 - INFO -  PyCycloneDp.init localhost
2020-10-29 12:28:36,220 - PyCycloneDp - INFO - logging in PyCycloneDp.__init__
  
--------------------------------------+------------------------------------------------------+-------------+----------------+----------------+-----------+----------
              Family ID               |             Primary Volume/FS/vVol Name              |     Type    | Family Unique  | Family Shared  |   No. of  |  No. of
                                      |                                                      |             | Physical Used  |  Logical Used  |   Clones  | Snapshots
--------------------------------------+------------------------------------------------------+-------------+----------------+----------------+-----------+----------
 d301959c-6eb9-4e23-967a-e0b3cdf06bea |                     labtea-vol-1                     |Block        |        97.91 GB|       486.36 GB|     0     |     1
 6c97fdd8-5b01-44ef-adaf-047449f83c27 |                     labtea-vol-2                     |Block        |        99.47 GB|       492.30 GB|     0     |     1

     

 
Alert ID DATA_PATH_WRITE_PROTECT_MODE_EXTREMELY_HIGH 
Alert Text The space utilization of %(tier_name) is extremely high.
Error Code 0x00200703
Impact Several internal services (such as NAS and UI) may not operate optimally.
Resolution  Cause:
Some of the appliance capacity is reserved for internal services (such as NAS core services, Data Collects, internal DB and more).
When the appliance reaches Write Protect Mode (0x00200701), the user can no longer write any data, however internal services can still access critical information (only from PowerStoreOS 3.0), once the internal reserved capacity also runs out, this alert is raised, CP and SDNAS (and other services) may not operate optimally.

Solution:
Contact Technical Support for immediate assistance.

Additional Information

Known Issues:

Article Properties

Affected Product

PowerStore

Last Published Date

24 Nov 2022

Version

5

Article Type

Solution

Back to Top