What is the storage efficiency of a three mirror in a Microsoft failover cluster?

This topic discusses common questions about Storage Spaces on Windows Server 2012 R2, Windows Server 2012, Windows 8.1, and Windows 8. For basic info about Storage Spaces, see Storage Spaces Overview. For design info, see Software-Defined Storage Design Considerations Guide. For performance information, see Storage Spaces - Designing for Performance.

Table of Contents

• How can I manage Storage Spaces?
• What are the recommended configuration limits?
• What are the best uses of simple, mirror, and parity spaces?
• What types of drives can I use with Storage Spaces?
• What types of storage arrays can I use with Storage Spaces?
  • Enclosure Awareness Support - Tolerating an Entire Enclosure Failing
  • What are the resiliency levels provided by Enclosure Awareness?
• What types of storage spaces can I use with a failover cluster?
• What properties can I change after creating a storage space?
• How does Windows let me know of a disk failure?
• How does Storage Spaces respond to errors on physical disks?
• What information will Storage Spaces put in the Event log?
• How do I replace a physical disk?
• What are columns and how does Storage Spaces decide how many to use?
  • Example 1: A Two-Column Simple Space
  • Example 2: A Three-Column Parity Space
  • Example 3: A Two-Column Two-Way Mirror Space
  • Controlling the Number of Columns
• Why do I have a low capacity warning even though I still have unused pool capacity?
  • A two column, two-way mirror space that uses thin provisioning in a four disk pool
• How do I increase pool capacity?
• What happens to Storage Spaces when moving physical disks between servers?
• How do I check Storage Spaces resiliency health in a failover cluster?
• How do I know repairing a storage space starts and successfully completes?
• What happens if I run out of physical capacity on a thinly provisioned storage space?
• See Also

There are three ways to manage Storage Spaces and associated storage pools and drives:

• The Storage Spaces item in Control Panel Available in Windows 10, Windows 8.1 and Windows 8, you can use the Storage Spaces item in Control Panel to easily create and maintain storage spaces and their associated storage pools. The Storage Spaces item in Control Panel provides simple but powerful control over tasks a user might want to perform.
• The File and Storage Services section of Server Manager Available in Windows Server, you can use Server Manager to create and manage storage spaces (also referred to as virtual disks) and their associated storage pools on the Storage Spaces subsystem as well as on non-Microsoft storage subsystems. Server Manager provides support for performing all of the tasks administrators commonly perform with Storage Spaces.
• The Storage module in Windows PowerShell Available in Windows and Windows Server, you can use the cmdlets included in the Storage module to create and manage storage spaces (also referred to as virtual disks) and their associated storage pools on the Storage Spaces subsystem as well as on non-Microsoft storage subsystems. Windows PowerShell provides comprehensive support for managing Storage Spaces.

The following are the recommended configuration limits for Storage Spaces:

For clustered storage pools:

• Up to 80 physical disks in a clustered storage pool (to allow time for the pool to fail over to other nodes)
• Up to four storage pools per cluster

For non-clustered storage pools:

• Up to 240 physical disks in a non-clustered Windows Server 2012 R2 storage pool (4 x 60 disk JBOD); you can, however, have multiple pools of 240 disks
• In Windows Server 2012, you can have up to 160 physical disks in a storage pool; you can, however, have multiple pools of 160 disks.

For both clustered and non-clustered pools:

• Up to 480 TB of capacity in a single storage pool
• Up to 64 storage spaces in a single storage pool
• Up to 10 TB per virtual disk when using persistent VSS snapshots and the Volsnap VSS system provider

What are the best uses of simple, mirror, and parity spaces?

Storage Spaces offers simple spaces, mirror spaces, and parity spaces, each with unique characteristics.

• Simple spaces are designed for increased performance when resiliency isn't very important. They are best suited for temporary data, such as video rendering files, image editor scratch files, and intermediary compiler object files. Simple spaces require a minimum of one physical disk.
• Mirror spaces are designed for increased performance and increased resiliency. Two-way mirror spaces can tolerate one disk failure and three-way mirror spaces can tolerate two disk failures. They are well suited to storing a broad range of data, from a general-purpose file share to a VHD library. When a mirror space is formatted with the Resilient File System (ReFS), Windows offers automatic data integrity maintenance. This is a layer of resiliency is above and beyond the resiliency achieved from maintaining multiple data copies to tolerate drive failure. 

  Mirror spaces require at least two disks (per tier if you use storage tiers) to protect you from a single disk failure. To protect a virtual disk from two simultaneous disk failures, you need at least three disks (per tier), with a minimum of five disks total in the pool to maintain pool metadata. 

• Parity spaces are designed for capacity efficiency and increased resiliency. Parity spaces are best suited for archival data and streaming media, such as music and videos. This storage layout requires at least three disks to protect you from a single disk failure and at least seven disks to protect you from two disk failures.

For details on how to design mirror and parity spaces for maximum reliability and performance, including how to use journal disks to improve parity write performance, see the Software-Defined Storage Design Considerations Guide.

You can use commodity drives attached via Serial-Attached SCSI (SAS), Serial ATA (SATA), PCI-Express, M.2, U.2, or USB. Storage layers that abstract the physical disks are not compatible with Storage Spaces. This includes VHDs and pass-through disks in a virtual machine, and storage subsystems that layer a RAID implementation on top of the physical disks. iSCSI and Fibre Channel controllers are not supported by Storage Spaces.

RAID adapters, if used, must be in non-RAID mode with all RAID functionality disabled. Such adapters must not abstract the physical disks, cache data, or obscure any attached devices including enclosure services provided by attached just-a-bunch-of-disks (JBOD) devices. Storage Spaces is compatible only with RAID adapters that support completely disabling all RAID functionality.

What types of storage arrays can I use with Storage Spaces? 

Storage arrays that provide direct connectivity to the physical disks they house and do not layer RAID implementations or abstract the disks in any way are compatible with Storage Spaces. Such arrays are also known as Just a Bunch of Disks (JBOD).

For Storage Spaces to identify disks by slot and leverage the array’s failure and identify/locate lights, the array must support SCSI Enclosure Services (SES) version 3. For a list of compatible arrays, see the Windows Server Hardware Catalog.

Enclosure Awareness Support - Tolerating an Entire Enclosure Failing

To support deployments that require an added level of fault tolerance, Storage Spaces supports associating each copy of data with a particular JBOD enclosure. This capability is known as enclosure awareness. With enclosure awareness, if one enclosure fails or goes offline, the data remains available in one or more alternate enclosures. 

To use enclosure awareness with Storage Spaces, your environment must meet the following requirements:

• JBOD storage enclosures must support SCSI Enclosure Services (SES).
• Storage spaces must use the mirror resiliency type if you're using Windows Server 2012:
  • To tolerate one failed enclosure with two-way mirrors, you need three compatible storage enclosures.
  • To tolerate two failed enclosures with three-way mirrors, you need five compatible storage enclosures.

What are the resiliency levels provided by Enclosure Awareness?

The following table lists the maximum simultaneous fault tolerance of solutions that have enclosure awareness enabled:

Simple

Not supported

Not supported

Two-way Mirror

1 enclosure; or 1 disk*

1 enclosure; or 1 disk*

Three-way Mirror

1 enclosure + 1 disk*; or 2 disks*

1 enclosure + 1 disk*;

or 2 disks*

Single Parity

Not supported

Not supported

Dual Parity

Not supported

1 enclosure + 1 disk*;

or 2 disks*

* per pool

Storage Spaces on Windows Server 2012 R2 supports creating a clustered storage pool when using mirror spaces, parity spaces, and simple spaces. Windows Server 2012 doesn't support parity spaces on clustered storage pools. To cluster Storage Spaces, your environment must meet the following requirements:

• All storage spaces in the storage pool must use fixed provisioning.
• Two-way mirror spaces must use three or more physical disks.
• Three-way mirror spaces must use five or more physical disks.
• Parity spaces are supported on Windows Server 2012 R2, but not Windows Server 2012.
• All physical disks in a clustered pool must be connected via SAS.
• All physical disks must support persistent reservations and pass the failover cluster validation tests.

Note: The SAS JBOD must be physically connected to all cluster nodes which will use the storage pool.  Direct attached storage that is not connected to all cluster nodes is not usable for clustered pools with Storage Spaces.

Upon creating a storage space, certain fundamental properties are locked in place. These fundamental properties include the provisioning type (thin or fixed), the resiliency type (simple, mirror, or parity), the number of columns the storage space striping requires, and the striping interleave.

After creating a storage space, you can change properties that do not affect fundamental characteristics, such as the size of the storage space and its name.

When a disk failure occurs, Windows displays information on the disk failure in the following locations:

• The taskbar
• The Action Center
• The Storage Spaces Item in Control Panel (on Windows 8.1 and Windows 8)
• The Storage Pools section of the File and Storage Services role in Server Manager (on Windows Server)
• The Windows PowerShell cmdlet Get-PhysicalDisk

Physical disks often experience errors of varying severity, from errors that the disk can transparently recover from, without interruption or data loss; to errors that are catastrophic and can cause data loss. For more information, see How Storage Spaces responds to errors on physical disks.

What information will Storage Spaces put in the Event log?

To help with administration, Storage Spaces will log certain events in the Event log. For more information on the events logged by Storage Spaces, see the following:

• Storage Spaces Event 100
• Storage Spaces Event 102
• Storage Spaces Event 103
• Storage Spaces Event 104
• Storage Spaces Event 200
• Storage Spaces Event 201
• Storage Spaces Event 202
• Storage Spaces Event 203
• Storage Spaces Event 300
• Storage Spaces Event 301
• Storage Spaces Event 302
• Storage Spaces Event 303
• Storage Spaces Event 304
• Storage Spaces Event 306
• Storage Spaces Event 307
• Storage Spaces Event 308

If you are using the Storage Spaces item in Control Panel, replacing a physical disk is as simple as clicking the command to remove the old disk. After that, you can immediately disconnect the disk. In the background, Storage Spaces leverages resiliency to reconstruct your data. Disk removal has three prerequisites:

• All storage spaces that depend on the disk are resilient and healthy (removing the disk will not succeed if a simple space depends on the disk).   You can discover which storage spaces depend on a physical disk by piping the Get-PhysicalDisk cmdlet to Get-VirtualDisk. For example:

  Get-PhysicalDisk -FriendlyName PhysicalDisk1 | Get-VirtualDisk

  You can check the resiliency and health of your storage spaces by using the Get-VirtualDisk cmdlet. A storage space is resilient when ResiliencySettingName is either Mirror or Parity and healthy when HealthStatus is Healthy.   

• The pool has at least as much unused capacity on other disks as the amount of data the old disk hosts. This ensures that the storage spaces can be reconstructed with a replacement disk. If the usage type of the disk is manual, then the storage spaces that depend on the disk must have sufficient unused capacity on other disks in their list of disks they can use. In both cases, the unused capacity must be on disks that do not already host an extent of the same stripe as the old disk. This can be satisfied by first adding the new disk to the pool, and, if applicable, adding it to the storage spaces’ lists of disks that they can use.
  
  TipYou can check for unused capacity on disks the affected storage spaces are not using by subtracting AllocatedSize from Size, as reported by the Get-PhysicalDisk command.

• If the pool is clustered, removing the disk would not cause the pool to have fewer than three disks, if the pool does not contain any three-way mirror spaces, or fewer than five disks, if the pool contains a three-way mirror space.

If you are using Windows PowerShell, you must perform following procedure to remove the old disk, assuming that you meet the prerequisites for removing a physical disk.

Removing a physical disk by using Windows PowerShell

1. Open a Windows PowerShell session as an administrator.

2. Type the following command, replacing <diskname> with the friendly name of the disk:

   Set-PhysicalDisk -FriendlyName <diskname> -Usage Retired

   If the pool is clustered, you will need to run this command on the cluster node which has the cluster resource for the pool online.

3. Type the following command for each storage space that depended on the old disk, replacing <virtualdiskname> with the friendly name of the storage space:

   Repair-VirtualDisk -FriendlyName <virtualdiskname>

   If the pool is clustered, run this command on the cluster nodes that have the individual storage spaces’ cluster resource online and then run it again on the cluster node that has the pool’s cluster resource online.

4. Prior to performing step 5, you must wait for all active repair jobs to complete. To monitor the status of repair jobs, type the following command:

   Get-StorageJob

5. Type the following command:

   Remove-PhysicalDisk -FriendlyName <diskname> 
   The above command did not work in my Windows Server 2012 R2 Essentials. 
   You have to do this in 2 steps. First give the disk to be removed a name and then remove; 
   example;
   PS C:\> $PDToRemove = Get-PhysicalDisk -Friendlyname "PhysicalDisk25"
   PS C:\> Remove-PhysicalDisk -PhysicalDisks $PDToRemove -StoragePoolFriendlyName "DemoPool"

6. Physically disconnect the old disk.

Besides offering resiliency to drive failures, Storage Spaces also offers increased performance by striping data across multiple disks. Storage Spaces describes a stripe via two parameters, NumberOfColumns and Interleave.

• A stripe represents one pass of data written to a storage space, with data written in multiple stripes (passes).
• Columns correlate to underlying physical disks across which one stripe of data for a storage space is written.
• Interleave represents the amount of data written to a single column per stripe.

The NumberOfColumns and Interleave parameters, which are accessible via Windows PowerShell or WMI, determine the width of the stripe (stripe_width = NumberOfColumns * Interleave). The stripe width determines how much data and parity (in the case of parity spaces) Storage Spaces writes across multiple disks to increase performance available to apps.

Example 1: A Two-Column Simple Space

A simple example is a two-column simple space, which offers striping with no resiliency.

For the first stripe of data in this example, Storage Spaces writes 256 KB (the default Interleave value) to the first disk (column) in the storage pool, then 256 KB of data to the second disk in the pool. This yields a stripe width of 512 KB (2 columns * 256 KB interleave).

Example 2: A Three-Column Parity Space

Another example is a three-column parity space (with a 256 KB stripe interleave), the simplest form of a parity space.

For the first stripe of data in this example, Storage Spaces writes 256 KB of data to the first disk (column), 256 KB of data to the second disk, and 256 KB of parity to the third disk. This yields a stripe size of 768 KB (3 columns * 256 KB of interleave). As more data is written to the parity space, it rotates the column for the parity information among all three disks.

Example 3: A Two-Column Two-Way Mirror Space

Another example is a two-column two-way mirror space. Mirror spaces add a layer of data copies below the stripe, which means that a two-way mirror space duplicates each individual column's data onto two disks.

For the first stripe of data in this example, Storage Spaces writes 256 KB of data to the first column, which is written in duplicate to the first two disks. For the second column of data, Storage Spaces writes 256 KB of data to the second column, which is written in duplicate to the next two disks. The column-to-disk correlation of a two-way mirror is 1:2; for a three-way mirror, the correlation is 1:3.

Controlling the Number of Columns

You can control the number of columns and the stripe interleave when creating a new storage space by using the Windows PowerShell cmdlet New-VirtualDisk with the NumberOfColumns and Interleave parameters.

Each type of storage space has a minimum number of stripe columns which translates to a minimum number of physical disks, given their column-to-disk correlation (below).

Resiliency type

Minimum number of columns

Column-to-disk correlation

Minimum number of disks

Simple (no resiliency)

1

1:1

1

Two-way mirror

1

1:2

2

Three-way mirror

1

1:3

5

Single Parity

3

1:1

3

Due to striping, a storage space simultaneously allocates capacity from as many disks as its stripe requires. Therefore, when increasing pool capacity, you can usually achieve optimal pool capacity utilization when you add disks in multiples of the number of disks the storage space needs. For example, adding disks in multiples of four might provide optimal capacity utilization for a pool comprised of two-column, two-way mirror spaces (2 columns + 2 data copies = 4 disks per stripe).

Note Storage Spaces in Windows Server 2012 R2 and earlier by default uses the largest column count possible given the number of disks you have and the resiliency type you select. This is useful for environments with a small number of disks where you want to maximize performance and don't have to replace disks very often. However, in environments with a large number of disks, use a column count number one less than the maximum. This allows Storage Spaces to automatically repair virtual disks following a disk failure or after you retire a disk.

Storage Spaces provides advance notification of thinly provisioned storage spaces when the storage pool does not have enough capacity spread among a sufficient number of disks to continue to write new data. The default warning point is 70% capacity utilization. To learn when Storage Spaces will generate a warning, consider the following example.

A two column, two-way mirror space that uses thin provisioning in a four disk pool

Two of the disks have 1TB capacity and two have 2TB capacity. Because a two column, two-way mirror space needs four disks (number_of_disks = NumberOfColumns * NumberOfDataCopies), it will evenly consume all four disks as it writes new data. When capacity utilization of the two 1TB disks reaches 70%, Storage Spaces will warn of a low capacity condition. Even though the entire pool has 3.2TB free capacity, the thinly provisioned space will soon not be able to write any more data because the 1TB disks are nearly fully consumed.

You can easily keep individual storage spaces’ low capacity warning synchronized with each other and with the pool by following the guidance in the next section, “How do I increase pool capacity?” from the moment of creating the pool and through all subsequent expansions of the pool.

How do I increase pool capacity?

Storage Spaces can usually use the additional capacity from even a single additional drive. However, for optimal capacity utilization, consider the number of columns your storage spaces have and add disks in multiples of that number.

For example, consider a pool which has a four-column simple space, a one-column, two-way mirror space, and an eight-column parity space. The four-column simple space suggests you expand pool capacity in sets of four disks. The one-column, two-way mirror space suggests you expand pool capacity in sets of two disks (for mirror spaces, you have to multiple the number of columns by the number of copies). The eight-column parity space suggests you expand pool capacity in sets of eight disks. In this example, you would want to expand pool capacity in sets of eight disks as eight is a common multiple of the number of columns of each of the storage spaces in the pool.

What happens to Storage Spaces when moving physical disks between servers?

Storage Spaces records information about pools and storage spaces on the physical disks that compose the storage pool. Therefore, your pool and storage spaces are preserved when you move an entire storage pool and its physical disks from one computer to another.

Windows Server 2012 starts storage that could potentially be shared with a cluster in a safe state. For Storage Spaces, that means the first time Windows connects to a storage pool, the pool starts as read-only and the storage spaces will start in a detached state. To access your data, you must set the storage pool to read-write and then attach the storage spaces.

These steps do not apply to Windows 8 – storage pools start as read-write and storage spaces start as attached.

How do I check Storage Spaces resiliency health in a failover cluster?

The health of each storage space is available only from the cluster node that has the cluster resource for the storage space in the online state. Use Failover Cluster Management to discover which node has the cluster resource for a storage space in the online state. Then use the Storage Pools section of the File and Storage Services role in Server Manager to view the health of the appropriate storage space.

You can also use the Get-VirtualDisk cmdlet on the node that has the cluster resource online to retrieve the storage space’s properties. The resiliency health of the storage space is noted in the storage space’s properties. To view the health status of a storage pool, use the Get-StoragePool cmdlet on the cluster node that has the cluster resource for the pool in the online state.

For the repair operation to complete successfully, the pool must have at least as much unused capacity as data to be repaired. If the storage space has a list of disks to use, then the unused capacity must reside on those disks. Unused capacity must reside on disks which do not already host an extent of the stripe to be repaired. Because repair happens over time, this requirement must be continuously fulfilled for a successful repair completion.

For a quick check on whether repair is successfully progressing, use the Get-VirtualDisk cmdlet to look at the storage space’s OperationalStatus. While repair is successfully progressing, OperationalStatus will be In Service. When repair successfully completes, OperationalStatus is OK. If repair is unable to continue, OperationalStatus returns to Degraded.

Since the storage space is thinly provisioned, allocation of new capacity occurs as needed.  Once capacity is exhausted, there is no room to properly manage files on the storage space and there is a risk of data loss due to file corruption.  To protect against that, Windows takes the storage space offline – removing the storage space from the view of anything that was writing and thereby keeping your files safe.  Once the pool has more drives, you can bring the storage space back online and continue running.

When additional drives are not immediately available, you can try to just bring the storage space back online and get to your files until additional drives are available.  However, sometimes applications, or the file system itself, may continue writing once the storage space is online.  If that happens, then Windows’ safety measures will again protect your files by taking the storage space offline.

If additional drives are not immediately available and the storage space cannot remain online, then can still get to your files by using PowerShell to temporarily make your storage space read-only.

Get-VirtualDisk –Friendlyname name_of_your_space | Get-Disk | Set-Disk –IsReadOnly $true

Get-VirtualDisk –Friendlyname name_of_your_space | Get-Disk | Set-Disk –IsOffline $false

If you do that, then you will want to switch your storage space back to read-write once your storage pool has additional drives.  You can use PowerShell to do that as well.

Get-VirtualDisk –Friendlyname name_of_your_space | Get-Disk | Set-Disk –IsOffline $true

Get-VirtualDisk –Friendlyname name_of_your_space | Get-Disk | Set-Disk –IsReadOnly $false

Get-VirtualDisk –Friendlyname name_of_your_space | Get-Disk | Set-Disk –IsOffline $false

Finally, if you do not add more drives to the storage pool (the control panel will tell you the minimum number of drives to add), then Windows will protect the files which are already on the storage space by taking the storage space offline.  Adding new drives to the storage pool provides the additional capacity the storage space needs.  You will be able to bring your storage space back online and continue to copy new files into it.

• File and Storage Services
• Storage Spaces Overview
• Storage Management Overview
• Modifying a Storage Pool that has a Read-Only Configuration
• Provide cost-effective storage for Hyper-V workloads by using Windows Server (2012 R2)
• Provide cost-effective storage for Hyper-V workloads by using Windows Server: planning and design guide (2012 R2)

@Jason Johnson

Parity in storage spaces cannot match the redundancy of a RAID 6 solution.  Using a smaller number of columns than physical disks will still only guarantee one drive failure.  The reason is simple - only one parity value is generated for a stripe (column set), thus a stripe can only tolerate one failure.  It's more like a RAID 50 if the number of drives is evenly divisible by the number of columns.  In theory, this special case should allow two drive failures if they are from different column sets, but I haven't seen any official word if more than a single drive failure is allowed in anything other than a 3-way mirror -- even with a 2-column mirror space that could theoretically support n/2 failures (I might have to test this).  I suspect in an effort to get away from the rigid structure required by most RAID solutions that storage spaces is flexible with column placement -- so 3 columns of a given parity stripe must be on 3 different drives but it's not necessarily the same 3 drives for the next stripe.  Imagine a 3-column parity across 8 drives - not ideal, but to balance space utilization you'll have overlap of column sets across physical drives.

The general case of ANY 2 drive failures is not possible with parity storage spaces, and if 2 drive failures were supported in certain configurations then MS would be listing that in feature bullet points and configuration guides.  I'd really like to see some in-depth technical documentation on the architecture of Storage Spaces but for now we can only guess by experimentation.

How many disks must you add to a storage pool to create a three

How many disk failures can a two

What minimum amount of disks are required for a storage pool with parity?

What is Microsoft storage spaces?