Step-by-step for Storage Spaces Tiering in Windows Server 2012 R2

August 28, 2013, 10:00 am

≫ Next: What’s new in SMB PowerShell in Windows Server 2012 R2

≪ Previous: SNIA’s Storage Developer Conference 2013 is just a few weeks away

1. Overview

In this document, I am sharing all the steps I used to create a demo or test environment of a storage space with storage tiers on Windows Server 2012 R2 Preview so that you can experiment with some of the new technologies yourself. You need only a single computer with one SSD and one HDD for this demo (the specs are provided below).

If you're not familiar with Storage Spaces or storage tiers (also known as Storage Spaces Tiering), I would strong encourage you to review the Storage Spaces Overview and TechEd 2013 talks on “Storage Spaces”, which you can find below:

2. Simulated environment with limited hardware

In a typical storage tier configuration (non-clustered), you have a JBOD with a few hard disks, some being SSDs and some being HDD. For instance, you could have 4 SSDs and 8 HDDs. You would then combine all 12 disks into a pool, create the two tiers (SSD and HDD) and then create spaces (virtual disks) on top of those. Here’s what it looks like:

However, if you’re just experimenting with this feature or trying to learn how to configure it, the investment in the hardware (1 JBOD, 4 SSDs, 8 HDDs, SAS HBA, cables) might be holding you back. So, in this blog post, we’ll show how to configure storage tiers using just one SSD and one HDD, with the help of Hyper-V. Here’s what the simulated environment looks like:

As you can see in the diagram, the basic goal here is to simulate a tiered storage space using 4 (simulated) SSDs and 8 (simulated) HDDs. We’ll show the Windows PowerShell cmdlets to create the VHDs and the VM required to simulate the environment. Then, inside the guest, we’ll create the pool, storage tiers and storage spaces (both simple and mirrored). To verify it’s all working, we’ll create some test files and run a simulated SQL workload. We’ll also look into the storage tier maintenance tasks and extending an existing storage space.

3. Required Hardware and Software, plus Disclaimers

You will need the following hardware to perform the steps described here:

One computer capable of running the Windows Server 2012 R2 Preview and Hyper-V.
On that computer, one SSD with at least 50GB of free space available.
On that computer, one HDD with at least 250GB of free space available.
As required by Hyper-V, enable the virtualization settings in the BIOS. For details, check this blog post: http://blogs.technet.com/b/iftekhar/archive/2010/08/09/enable-hardware-settings-in-bios-to-run-hyper-v.aspx and http://technet.microsoft.com/en-us/library/cc731898.aspx
For this setup, the computer should have at least 4GB of RAM.
An 8GB USB stick, if you’re installing Windows Server from USB and copying the downloaded software around

You will need the following software to perform the steps described here:

Windows Server 2012 R2 Preview from http://technet.microsoft.com/en-US/evalcenter/dn205286.aspx
SQLIO, as described at http://blogs.technet.com/b/josebda/archive/2013/03/25/sqlio-powershell-and-storage-performance-measuring-iops-throughput-and-latency-for-both-local-disks-and-smb-file-shares.aspx

Notes and disclaimers:

A certain familiarity with Windows administration and configuration is assumed. If you're new to Windows, this document is not for you. Sorry...
If you are asked a question or required to perform an action that you do not see described in these steps, go with the default option.
There are usually several ways to perform a specific configuration or administration task. What I describe here is one of those many ways. It's not necessarily the best way, just the one I personally like best at the moment.
For the most part, I use PowerShell to configure the systems. You can also use a graphical interface instead, but I did not describe those steps here.
The specific Storage Spaces configuration shown in this blog post is not supported. Microsoft Support will only answer questions and assist in troubleshooting configurations where Storage Spaces uses a physical machine (not a VM) and uses one of the certified JBOD hardware solutions (see http://www.windowsservercatalog.com/results.aspx?bCatID=1573&cpID=0&avc=10&OR=1)
Because of the item above, the configuration described here should only be used for demos, testing or learning environments.

4. Configure the physical host

# Preparation steps: Install Window Server 2012 R2 Preview

# Install required roles and features, restart at the end

Install-WindowsFeature Hyper-V -IncludeManagementTools –Restart

# Create 4 VHDX files on the SSD with 10GB each

1..4 | % { New-VHD -Path D:\VMS\VMA_SSD_$_.VHDX -Fixed –Size 10GB}

# Create 8 VHDX files on the HDD with 30GB each

1..8 | % { New-VHD -Path E:\VMS\VMA_HDD_$_.VHDX -Fixed –Size 30GB}

# Create a new VM. Assumes you have an Windows Server 2012 R2 OS VHDX in place

New-VM -Name VMA -Path D:\VMS –VHDPath D:\VMS\VMA_OS.VHDX -Memory 2GB

# Add all data disks to the VM

1..4 | % { Add-VMHardDiskDrive -VMName VMA -ControllerType SCSI -Path D:\VMS\VMA_SSD_$_.VHDX }

1..8 | % { Add-VMHardDiskDrive -VMName VMA -ControllerType SCSI -Path E:\VMS\VMA_HDD_$_.VHDX }

# Start the VM

Start-VM VMA

5. Check VM configuration (from the Host, with output)

PS C:\>Get-VM VMA

Name State   CPUUsage(%) MemoryAssigned(M) Uptime   Status
---- -----   ----------- ----------------- ------   ------
VMA Running 0           2048              00:01:52 Operating normally

PS C:\>Get-VM VMA | Get-VMHardDiskDrive

VMName ControllerType ControllerNumber ControllerLocation DiskNumber Path
------ -------------- ---------------- ------------------ ---------- ----
VMA    IDE            0                0                             D:\VMS\VMA_OS.VHDX
VMA    SCSI           0                0                             D:\VMS\VMA_SSD_1.VHDX
VMA    SCSI           0                1                             D:\VMS\VMA_SSD_2.VHDX
VMA    SCSI           0                2                             D:\VMS\VMA_SSD_3.VHDX
VMA    SCSI           0                3                             D:\VMS\VMA_SSD_4.VHDX
VMA    SCSI           0                4                             E:\VMS\VMA_HDD_1.VHDX
VMA    SCSI           0                5                             E:\VMS\VMA_HDD_2.VHDX
VMA    SCSI           0                6                             E:\VMS\VMA_HDD_3.VHDX
VMA    SCSI           0                7                             E:\VMS\VMA_HDD_4.VHDX
VMA    SCSI           0                8                           E:\VMS\VMA_HDD_5.VHDX
VMA    SCSI           0                9                             E:\VMS\VMA_HDD_6.VHDX
VMA    SCSI           0                10                            E:\VMS\VMA_HDD_7.VHDX
VMA    SCSI           0                11                            E:\VMS\VMA_HDD_8.VHDX

6. Check VM configuration (from the Guest, with output)

PS C:\> Get-PhysicalDisk | Sort Size | FT DeviceId, FriendlyName, CanPool, Size, MediaType -AutoSize

DeviceId FriendlyName   CanPool        Size MediaType
-------- ------------   -------        ---- ---------
2        PhysicalDisk2     True 10737418240 UnSpecified
4        PhysicalDisk4     True 10737418240 UnSpecified
3        PhysicalDisk3     True 10737418240 UnSpecified
1        PhysicalDisk1     True 10737418240 UnSpecified
12       PhysicalDisk12    True 32212254720 UnSpecified
11       PhysicalDisk11    True 32212254720 Unspecified
7        PhysicalDisk7     True 32212254720 Unspecified
6        PhysicalDisk6     True 32212254720 Unspecified
5        PhysicalDisk5     True 32212254720 Unspecified
10       PhysicalDisk10    True 32212254720 UnSpecified
9        PhysicalDisk9     True 32212254720 Unspecified
8        PhysicalDisk8     True 32212254720 Unspecified
0        PhysicalDisk0    False 42949672960 Unspecified

PS C:\> Get-PhysicalDisk -CanPool $true | ? Size -lt 20GB | Sort Size | FT -AutoSize

FriendlyName CanPool OperationalStatus HealthStatus Usage        Size
------------ ------- ----------------- ------------ -----        ----
PhysicalDisk4 True    OK                Healthy      Auto-Select 10 GB
PhysicalDisk2 True    OK                Healthy      Auto-Select 10 GB
PhysicalDisk1 True    OK                Healthy      Auto-Select 10 GB
PhysicalDisk3 True    OK                Healthy      Auto-Select 10 GB

PS C:\> Get-PhysicalDisk -CanPool $true | ? Size -gt 20GB | Sort Size | FT -AutoSize

FriendlyName   CanPool OperationalStatus HealthStatus Usage        Size
------------   ------- ----------------- ------------ -----        ----
PhysicalDisk6 True    OK                Healthy      Auto-Select 30 GB
PhysicalDisk11 True    OK                Healthy      Auto-Select 30 GB
PhysicalDisk12 True    OK                Healthy    Auto-Select 30 GB
PhysicalDisk7 True    OK                Healthy      Auto-Select 30 GB
PhysicalDisk5 True    OK                Healthy      Auto-Select 30 GB
PhysicalDisk10 True    OK                Healthy      Auto-Select 30 GB
PhysicalDisk8 True    OK                Healthy      Auto-Select 30 GB
PhysicalDisk9 True    OK                Healthy      Auto-Select 30 GB

7. Configure media type for virtual SAS disks

# Create Storage Pool

$s = Get-StorageSubSystem

New-StoragePool -StorageSubSystemId $s.UniqueId -FriendlyName Pool1 -PhysicalDisks (Get-PhysicalDisk -CanPool $true)

# Configure media type for virtual SAS disks

Get-StoragePool Pool1 | Get-PhysicalDisk | ? Size -lt 20GB | Set-PhysicalDisk –MediaType SSD

Get-StoragePool Pool1 | Get-PhysicalDisk | ? Size -gt 20GB | Set-PhysicalDisk –MediaType HDD

8. Check media type configuration (with output)

PS C:\> Get-StoragePool Pool1

FriendlyName OperationalStatus HealthStatus IsPrimordial IsReadOnly

------------ ----------------- ------------ ------------ ----------

Pool1 OK Healthy False False

PS C:\> Get-StoragePool Pool1 | Get-PhysicalDisk | Sort Size | FT –AutoSize

FriendlyName CanPool OperationalStatus HealthStatus Usage Size

------------ ------- ----------------- ------------ ----- ----

PhysicalDisk3 False OK Healthy Auto-Select 9.25 GB

PhysicalDisk2 False OK Healthy Auto-Select 9.25 GB

PhysicalDisk1 False OK Healthy Auto-Select 9.25 GB

PhysicalDisk4 False OK Healthy Auto-Select 9.25 GB

PhysicalDisk11 False OK Healthy Auto-Select 29.25 GB

PhysicalDisk12 False OK Healthy Auto-Select 29.25 GB

PhysicalDisk7 False OK Healthy Auto-Select 29.25 GB

PhysicalDisk6 False OK Healthy Auto-Select 29.25 GB

PhysicalDisk9 False OK Healthy Auto-Select 29.25 GB

PhysicalDisk5 False OK Healthy Auto-Select 29.25 GB

PhysicalDisk8 False OK Healthy Auto-Select 29.25 GB

PhysicalDisk10 False OK Healthy Auto-Select 29.25 GB

PS C:\> Get-StoragePool Pool1 | Get-PhysicalDisk | Sort Size | FT FriendlyName, Size, MediaType, HealthStatus, OperationalStatus -AutoSize

FriendlyName Size MediaType HealthStatus OperationalStatus

------------ ---- --------- ------------ -----------------

PhysicalDisk3 9932111872 SSD Healthy OK

PhysicalDisk2 9932111872 SSD Healthy OK

PhysicalDisk1 9932111872 SSD Healthy OK

PhysicalDisk4 9932111872 SSD Healthy OK

PhysicalDisk11 31406948352 HDD Healthy OK

PhysicalDisk12 31406948352 HDD Healthy OK

PhysicalDisk7 31406948352 HDD Healthy OK

PhysicalDisk6 31406948352 HDD Healthy OK

PhysicalDisk9 31406948352 HDD Healthy OK

PhysicalDisk5 31406948352 HDD Healthy OK

PhysicalDisk8 31406948352 HDD Healthy OK

PhysicalDisk10 31406948352 HDD Healthy OK

PS C:\> Get-StoragePool Pool1 | Get-PhysicalDisk | Group MediaType, Size | Sort Name | FT -AutoSize

Count Name Group

----- ---- -----

8 HDD, 31406948352 {MSFT_PhysicalDisk (ObjectId = "{1}\\WIN-T5PORQGQECN\root/Microsoft/Win...), MSFT_PhysicalDis...

4 SSD, 9932111872 {MSFT_PhysicalDisk (ObjectId = "{1}\\WIN-T5PORQGQECN\root/Microsoft/Win...), MSFT_PhysicalDis...

PS C:\> Get-StoragePool Pool1 | FL Size, AllocatedSize

Size : 290984034304
AllocatedSize : 3221225472

9. Configure Tiers

# Configure two tiers

Get-StoragePool Pool1 | New-StorageTier –FriendlyName SSDTier –MediaType SSD

Get-StoragePool Pool1 | New-StorageTier –FriendlyName HDDTier –MediaType HDD

10. Check Tiers configuration (with output)

PS C:\> Get-StorageTier | FT FriendlyName, MediaType, Size -AutoSize

FriendlyName MediaType Size

------------ --------- ----

SSDTier SSD 0

HDDTier HDD 0

PS C:\> Get-StoragePool Pool1 | FL Size, AllocatedSize

Size : 290984034304

AllocatedSize : 3221225472

Note: 3GB used out of 271GB total. Storage Spaces reserves 256MB on each disk in the pool for internal VD metadata. 256MB * 12 drives = 3GB.

PS C:\> Get-StorageTierSupportedSize SSDTier -ResiliencySettingName Simple | FT -AutoSize

SupportedSizes TierSizeMin TierSizeMax TierSizeDivisor

-------------- ----------- ----------- ---------------

{} 4294967296 34359738368 4294967296

Note: 32GB on the SSD Tier (8 GB * 4). Minimum size is 4GB.

PS C:\> Get-StorageTierSupportedSize SSDTier -ResiliencySettingName Mirror | FT -AutoSize

SupportedSizes TierSizeMin TierSizeMax TierSizeDivisor

-------------- ----------- ----------- ---------------

{} 2147483648 17179869184 2147483648

Note: Mirrored offers 16GB on the SSD Tier (8 GB * 4 / 2). Minimum size is 2GB.

PS C:\> Get-StorageTierSupportedSize HDDTier -ResiliencySettingName Simple | FT -AutoSize

SupportedSizes TierSizeMin TierSizeMax TierSizeDivisor

-------------- ----------- ----------- ---------------

{} 8589934592 249108103168 8589934592

Note: 232GB on the HDD Tier (29 GB * 8). Minimum size is 8GB.

PS C:\> Get-StorageTierSupportedSize HDDTier -ResiliencySettingName Mirror | FT -AutoSize

SupportedSizes TierSizeMin TierSizeMax TierSizeDivisor

-------------- ----------- ----------- ---------------

{} 4294967296 124554051584 4294967296

Note: Mirrored offers 116GB on the HDD Tier (29 GB * 8 / 2). Minimum size is 4GB.

11. Resiliency Settings and Spaces

# Configure resiliency settings

Get-StoragePool Pool1 | Set-ResiliencySetting -Name Simple -NumberOfColumnsDefault 4

Get-StoragePool Pool1 | Set-ResiliencySetting -Name Mirror -NumberOfColumnsDefault 2

# Create simple and mirrored spaces with tiering

$SSD = Get-StorageTier -FriendlyName SSDTier

$HDD = Get-StorageTier -FriendlyName HDDTier

Get-StoragePool Pool1 | New-VirtualDisk -FriendlyName Space1 -ResiliencySettingName Simple –StorageTiers $SSD, $HDD -StorageTierSizes 8GB, 32GB -WriteCacheSize 1GB

Get-StoragePool Pool1 | New-VirtualDisk -FriendlyName Space2 -ResiliencySettingName Mirror -StorageTiers $SSD, $HDD -StorageTierSizes 8GB, 32GB –WriteCacheSize 1GB

12. Check Storage Spaces configuration (with output)

PS C:\> Get-StoragePool Pool1 | Get-ResiliencySetting | FT -AutoSize

Name NumberOfDataCopies PhysicalDiskRedundancy NumberOfColumns Interleave

---- ------------------ ---------------------- --------------- ----------

Simple 1 0 4 262144

Mirror 2 1 2 262144

Parity 1 1 Auto 262144

PS C:\> Get-VirtualDisk | FT -AutoSize

FriendlyName ResiliencySettingName OperationalStatus HealthStatus IsManualAttach Size

------------ --------------------- ----------------- ------------ -------------- ----

Space1 Simple OK Healthy False 40 GB

Space2 Mirror OK Healthy False 40 GB

PS C:\> Get-StorageTier | FT FriendlyName, MediaType, Size -AutoSize

FriendlyName MediaType Size

------------ --------- ----

SSDTier SSD 0

HDDTier HDD 0

Space1_SSDTier SSD 8589934592

Space1_HDDTier HDD 34359738368

Space2_SSDTier SSD 8589934592

Space2_HDDTier HDD 34359738368

PS C:\> Get-StoragePool Pool1 | FL Size, AllocatedSize

Size : 290984034304

AllocatedSize : 136365211648

Note: 127GB allocated after creating the two spaces. 144GB left out of 271GB total.

PS C:\> Get-StorageTierSupportedSize SSDTier -ResiliencySettingName Simple | FT -AutoSize

SupportedSizes TierSizeMin TierSizeMax TierSizeDivisor

-------------- ----------- ----------- ---------------

{} 4294967296 4294967296 4294967296

Note: 4GB left on the SSD Tier.

PS C:\> Get-StorageTierSupportedSize SSDTier -ResiliencySettingName Mirror | FT -AutoSize

SupportedSizes TierSizeMin TierSizeMax TierSizeDivisor

-------------- ----------- ----------- ---------------

{} 2147483648 2147483648 2147483648

Note: 4GB left on the SSD Tier. 2GB available if mirroring.

PS C:\> Get-StorageTierSupportedSize HDDTier -ResiliencySettingName Simple | FT -AutoSize

SupportedSizes TierSizeMin TierSizeMax TierSizeDivisor

-------------- ----------- ----------- ---------------

{} 4294967296 146028888064 4294967296

Note: 136GB left on the SSD Tier.

PS C:\> Get-StorageTierSupportedSize HDDTier -ResiliencySettingName Mirror | FT -AutoSize

SupportedSizes TierSizeMin TierSizeMax TierSizeDivisor

-------------- ----------- ----------- ---------------

{} 2147483648 73014444032 2147483648

Note: 136GB left on the SSD Tier. 68GB available if mirroring.

13. Create Partitions and Volumes

# Configure volume “F” on Space1

Get-VirtualDisk Space1 | Get-Disk | Set-Disk -IsReadOnly 0

Get-VirtualDisk Space1 | Get-Disk | Set-Disk -IsOffline 0

Get-VirtualDisk Space1 | Get-Disk | Initialize-Disk -PartitionStyle GPT

Get-VirtualDisk Space1 | Get-Disk | New-Partition -DriveLetter “F” -UseMaximumSize

Initialize-Volume -DriveLetter “F” -FileSystem NTFS -Confirm:$false

# Configure volume “G” on Space2

Get-VirtualDisk Space2 | Get-Disk | Set-Disk -IsReadOnly 0

Get-VirtualDisk Space2 | Get-Disk | Set-Disk -IsOffline 0

Get-VirtualDisk Space2 | Get-Disk | Initialize-Disk -PartitionStyle GPT

Get-VirtualDisk Space2 | Get-Disk | New-Partition -DriveLetter “G” -UseMaximumSize

Initialize-Volume -DriveLetter “G” -FileSystem NTFS -Confirm:$false

14. Check Partitions and Volumes (with output)

PS C:\> Get-Partition | ? DriveLetter -ge "F" | FT -AutoSize

Disk Number: 13

PartitionNumber DriveLetter Offset Size Type

--------------- ----------- ------ ---- ----

2 F 135266304 39.87 GB Basic

Disk Number: 14

PartitionNumber DriveLetter Offset Size Type

--------------- ----------- ------ ---- ----

2 G 135266304 39.87 GB Basic

PS C:\> Get-Volume | ? DriveLetter -ge "F" | FT -AutoSize

DriveLetter FileSystemLabel FileSystem DriveType HealthStatus SizeRemaining Size

----------- --------------- ---------- --------- ------------ ------------- ----

F NTFS Fixed Healthy 39.75 GB 39.87 GB

G NTFS Fixed Healthy 39.75 GB 39.87 GB

15. Create Test Files

# Create 3 files on volume “F”, place them on different tiers

1..3 | % {

fsutil file createnew f:\file$_.dat (4GB)

fsutil file setvaliddata f:\file$_.dat (4GB)

}

Set-FileStorageTier -FilePath f:\file1.dat -DesiredStorageTierFriendlyName Space1_SSDTier

Set-FileStorageTier -FilePath f:\file2.dat -DesiredStorageTierFriendlyName Space1_HDDTier

Get-FileStorageTier -VolumeDriveLetter F

# Create 3 files on volume “G”, place them on different tiers

1..3 | % {

fsutil file createnew g:\file$_.dat (4GB)

fsutil file setvaliddata g:\file$_.dat (4GB)

}

Set-FileStorageTier -FilePath g:\file1.dat -DesiredStorageTierFriendlyName Space2_SSDTier

Set-FileStorageTier -FilePath g:\file2.dat -DesiredStorageTierFriendlyName Space2_HDDTier

Get-FileStorageTier -VolumeDriveLetter G

16. Check Test Files (with output)

PS C:\> Dir F:

Directory: f:\

Mode LastWriteTime Length Name

---- ------------- ------ ----

-a--- 8/21/2013 10:09 AM 4294967296 file1.dat

-a--- 8/21/2013 10:09 AM 4294967296 file2.dat

-a--- 8/21/2013 10:09 AM 4294967296 file3.dat

PS C:\> Dir G:

Directory: g:\

Mode LastWriteTime Length Name

---- ------------- ------ ----

-a--- 8/21/2013 10:08 AM 4294967296 file1.dat

-a--- 8/21/2013 10:08 AM 4294967296 file2.dat

-a--- 8/21/2013 10:08 AM 4294967296 file3.dat

PS C:\sqlio> Get-Volume | ? DriveLetter -ge "F" | FT -AutoSize

DriveLetter FileSystemLabel FileSystem DriveType HealthStatus SizeRemaining Size

----------- --------------- ---------- --------- ------------ ------------- ----

F NTFS Fixed Healthy 27.75 GB 39.87 GB

G NTFS Fixed Healthy 27.75 GB 39.87 GB

PS C:\> Get-FileStorageTier -VolumeDriveLetter F | FT -AutoSize

FilePath DesiredStorageTierName PlacementStatus State

-------- ---------------------- --------------- -----

F:\file1.dat Space1_SSDTier Completely on tier OK

F:\file2.dat Space1_HDDTier Partially on tier Pending

PS C:\> Get-FileStorageTier -VolumeDriveLetter G | FT -AutoSize

FilePath DesiredStorageTierName PlacementStatus State

-------- ---------------------- --------------- -----

G:\file1.dat Space2_SSDTier Completely on tier OK

G:\file2.dat Space2_HDDTier Partially on tier Pending

17. Tasks for Storage Tiering

# Check tasks used by Storage Tiering

Get-ScheduledTask -TaskName *Tier* | FT –AutoSize

Get-ScheduledTask -TaskName *Tier* | Get-ScheduledTaskInfo

# Manually running the “Storage Tiers Optimization” task

Get-ScheduledTask -TaskName "Storage Tiers Optimization" | Start-ScheduledTask

18. Tasks for Storage Tiering (with Output)

PS C:\> Get-ScheduledTask -TaskName *Tier* | FT –AutoSize

TaskPath TaskName State

-------- -------- -----

\Microsoft\Windows\Storage Tiers Management\ Storage Tiers Management Initialization Ready

\Microsoft\Windows\Storage Tiers Management\ Storage Tiers Optimization Ready

PS C:\> Get-ScheduledTask -TaskName *Tier* | Get-ScheduledTaskInfo

LastRunTime :

LastTaskResult : 1

NextRunTime : 8/22/2013 1:00:00 AM

NumberOfMissedRuns : 0

TaskName : Storage Tiers Optimization

TaskPath : \Microsoft\Windows\Storage Tiers Management\

PSComputerName :

LastRunTime : 8/21/2013 11:18:18 AM

LastTaskResult : 0

NextRunTime :

NumberOfMissedRuns : 0

TaskName : Storage Tiers Management Initialization

TaskPath : \Microsoft\Windows\Storage Tiers Management\

PSComputerName :

PS C:\> Get-ScheduledTask -TaskName "Storage Tiers Optimization" | Start-ScheduledTask

PS C:\> Get-ScheduledTask -TaskName "Storage Tiers Optimization" | Get-ScheduledTaskInfo

LastRunTime : 8/21/2013 12:11:11 PM

LastTaskResult : 267009

NextRunTime : 8/22/2013 1:00:00 AM

NumberOfMissedRuns : 0

TaskName : Storage Tiers Optimization

TaskPath : \Microsoft\Windows\Storage Tiers Management\

PSComputerName :

19. Running SQLIO

# These commands assume that the SQLIO2.EXE file was copied to the C:\SQLIO folder

# SQLIO workload 1 : 30 seconds, random, read, 8KB, 4 thread, 16 outstanding IOs, no buffering

# SQLIO workload 2 : 30 seconds, sequential, read, 512KB, 4 thread, 4 outstanding IOs, no buffering

# Check file location on tiers for volume F:

Get-FileStorageTier -VolumeDriveLetter F | FT -AutoSize

# Running SQLIO on F:, using File1 (HDD tier), File2 (HDD tier) and File 3 (unspecified tier)

c:\sqlio\sqlio2.exe -s30 -frandom -kR -b8 -t4 -o16 -BN f:\file1.dat

c:\sqlio\sqlio2.exe -s30 -frandom -kR -b8 -t4 -o16 -BN f:\file2.dat

c:\sqlio\sqlio2.exe -s30 -frandom -kR -b8 -t4 -o16 -BN f:\file3.dat

c:\sqlio\sqlio2.exe -s30 -fsequential -kR -b512 -t4 -o4 -BN f:\file1.dat

c:\sqlio\sqlio2.exe -s30 -fsequential -kR -b512 -t4 -o4 -BN f:\file2.dat

c:\sqlio\sqlio2.exe -s30 -fsequential -kR -b512 -t4 -o4 -BN f:\file3.dat

# Check file location on tiers for volume G:

Get-FileStorageTier -VolumeDriveLetter G | FT -AutoSize

# Running SQLIO on G:, using File1 (HDD tier), File2 (HDD tier) and File 3 (unspecified tier)

c:\sqlio\sqlio2.exe -s30 -frandom -kR -b8 -t4 -o16 -BN g:\file1.dat

c:\sqlio\sqlio2.exe -s30 -frandom -kR -b8 -t4 -o16 -BN g:\file2.dat

c:\sqlio\sqlio2.exe -s30 -frandom -kR -b8 -t4 -o16 -BN g:\file3.dat

c:\sqlio\sqlio2.exe -s30 -fsequential -kR -b512 -t4 -o4 -BN g:\file1.dat

c:\sqlio\sqlio2.exe -s30 -fsequential -kR -b512 -t4 -o4 -BN g:\file2.dat

c:\sqlio\sqlio2.exe -s30 -fsequential -kR -b512 -t4 -o4 -BN g:\file3.dat

20. Running SQLIO (with Output)

PS C:\> Get-FileStorageTier -VolumeDriveLetter F | FT -AutoSize

FilePath DesiredStorageTierName PlacementStatus State

-------- ---------------------- --------------- -----

F:\file1.dat Space1_SSDTier Completely on tier OK

F:\file2.dat Space1_HDDTier Completely on tier OK

PS C:\> c:\sqlio\sqlio2.exe -s30 -frandom -kR -b8 -t4 -o16 -BN f:\file1.dat

sqlio v2.15. 64bit_SG

4 threads reading for 30 secs to file f:\file1.dat

using 8KB random IOs

enabling multiple I/Os per thread with 16 outstanding

buffering set to not use file nor disk caches (as is SQL Server)

using current size: 4096 MB for file: f:\file1.dat

initialization done

CUMULATIVE DATA:

throughput metrics:

IOs/sec: 35745.63

MBs/sec: 279.26

PS C:\> c:\sqlio\sqlio2.exe -s30 -frandom -kR -b8 -t4 -o16 -BN f:\file2.dat

sqlio v2.15. 64bit_SG

4 threads reading for 30 secs to file f:\file2.dat

using 8KB random IOs

enabling multiple I/Os per thread with 16 outstanding

buffering set to not use file nor disk caches (as is SQL Server)

using current size: 4096 MB for file: f:\file2.dat

initialization done

CUMULATIVE DATA:

throughput metrics:

IOs/sec: 141.57

MBs/sec: 1.10

PS C:\> c:\sqlio\sqlio2.exe -s30 -frandom -kR -b8 -t4 -o16 -BN f:\file3.dat

sqlio v2.15. 64bit_SG

4 threads reading for 30 secs to file f:\file3.dat

using 8KB random IOs

enabling multiple I/Os per thread with 16 outstanding

buffering set to not use file nor disk caches (as is SQL Server)

using current size: 4096 MB for file: f:\file3.dat

initialization done

CUMULATIVE DATA:

throughput metrics:

IOs/sec: 384.86

MBs/sec: 3.00

PS C:\> c:\sqlio\sqlio2.exe -s30 -fsequential -kR -b512 -t4 -o4 -BN f:\file1.dat

sqlio v2.15. 64bit_SG

4 threads reading for 30 secs to file f:\file1.dat

using 512KB sequential IOs

enabling multiple I/Os per thread with 4 outstanding

buffering set to not use file nor disk caches (as is SQL Server)

using current size: 4096 MB for file: f:\file1.dat

initialization done

CUMULATIVE DATA:

throughput metrics:

IOs/sec: 998.25

MBs/sec: 499.12

PS C:\> c:\sqlio\sqlio2.exe -s30 -fsequential -kR -b512 -t4 -o4 -BN f:\file2.dat

sqlio v2.15. 64bit_SG

4 threads reading for 30 secs to file f:\file2.dat

using 512KB sequential IOs

enabling multiple I/Os per thread with 4 outstanding

buffering set to not use file nor disk caches (as is SQL Server)

using current size: 4096 MB for file: f:\file2.dat

initialization done

CUMULATIVE DATA:

throughput metrics:

IOs/sec: 81.30

MBs/sec: 40.65

PS C:\> c:\sqlio\sqlio2.exe -s30 -fsequential -kR -b512 -t4 -o4 -BN f:\file3.dat

sqlio v2.15. 64bit_SG

4 threads reading for 30 secs to file f:\file3.dat

using 512KB sequential IOs

enabling multiple I/Os per thread with 4 outstanding

buffering set to not use file nor disk caches (as is SQL Server)

using current size: 4096 MB for file: f:\file3.dat

initialization done

CUMULATIVE DATA:

throughput metrics:

IOs/sec: 148.51

MBs/sec: 74.25

PS C:\> Get-FileStorageTier -VolumeDriveLetter G | FT -AutoSize

FilePath DesiredStorageTierName PlacementStatus State

-------- ---------------------- --------------- -----

G:\file1.dat Space2_SSDTier Completely on tier OK

G:\file2.dat Space2_HDDTier Completely on tier OK

PS C:\> c:\sqlio\sqlio2.exe -s30 -frandom -kR -b8 -t4 -o16 -BN g:\file1.dat

sqlio v2.15. 64bit_SG

4 threads reading for 30 secs to file g:\file1.dat

using 8KB random IOs

enabling multiple I/Os per thread with 16 outstanding

buffering set to not use file nor disk caches (as is SQL Server)

using current size: 4096 MB for file: g:\file1.dat

initialization done

CUMULATIVE DATA:

throughput metrics:

IOs/sec: 35065.17

MBs/sec: 273.94

PS C:\> c:\sqlio\sqlio2.exe -s30 -frandom -kR -b8 -t4 -o16 -BN g:\file2.dat

sqlio v2.15. 64bit_SG

4 threads reading for 30 secs to file g:\file2.dat

using 8KB random IOs

enabling multiple I/Os per thread with 16 outstanding

buffering set to not use file nor disk caches (as is SQL Server)

using current size: 4096 MB for file: g:\file2.dat

initialization done

CUMULATIVE DATA:

throughput metrics:

IOs/sec: 138.98

MBs/sec: 1.08

PS C:\> c:\sqlio\sqlio2.exe -s30 -frandom -kR -b8 -t4 -o16 -BN g:\file3.dat

sqlio v2.15. 64bit_SG

4 threads reading for 30 secs to file g:\file3.dat

using 8KB random IOs

enabling multiple I/Os per thread with 16 outstanding

buffering set to not use file nor disk caches (as is SQL Server)

using current size: 4096 MB for file: g:\file3.dat

initialization done

CUMULATIVE DATA:

throughput metrics:

IOs/sec: 360.33

MBs/sec: 2.81

PS C:\> c:\sqlio\sqlio2.exe -s30 -fsequential -kR -b512 -t4 -o4 -BN g:\file1.dat

sqlio v2.15. 64bit_SG

4 threads reading for 30 secs to file g:\file1.dat

using 512KB sequential IOs

enabling multiple I/Os per thread with 4 outstanding

buffering set to not use file nor disk caches (as is SQL Server)

using current size: 4096 MB for file: g:\file1.dat

initialization done

CUMULATIVE DATA:

throughput metrics:

IOs/sec: 997.90

MBs/sec: 498.95

PS C:\> c:\sqlio\sqlio2.exe -s30 -fsequential -kR -b512 -t4 -o4 -BN g:\file2.dat

sqlio v2.15. 64bit_SG

4 threads reading for 30 secs to file g:\file2.dat

using 512KB sequential IOs

enabling multiple I/Os per thread with 4 outstanding

buffering set to not use file nor disk caches (as is SQL Server)

using current size: 4096 MB for file: g:\file2.dat

initialization done

CUMULATIVE DATA:

throughput metrics:

IOs/sec: 118.34

MBs/sec: 59.17

PS C:\> c:\sqlio\sqlio2.exe -s30 -fsequential -kR -b512 -t4 -o4 -BN g:\file3.dat

sqlio v2.15. 64bit_SG

4 threads reading for 30 secs to file g:\file3.dat

using 512KB sequential IOs

enabling multiple I/Os per thread with 4 outstanding

buffering set to not use file nor disk caches (as is SQL Server)

using current size: 4096 MB for file: g:\file3.dat

initialization done

CUMULATIVE DATA:

throughput metrics:

IOs/sec: 197.65

MBs/sec: 98.82

21. Summary of SQLIO Results

Here’s the summary of the SQLIO runs:

Resiliency	Workload	Tier	IOPs	MB/s
Simple	8KB Read	File1 (SSD)	35,746	279
		File2 (HDD)	142	1
		File3 (Mixed)	385	3
	512KB Sequential	File1 (SSD)	998	499
		File2 (HDD)	81	41
		File3 (Mixed)	149	74
Mirror	8KB Read	File1 (SSD)	35,065	274
		File2 (HDD)	39	1
		File3 (Mixed)	360	3
	512KB Sequential	File1 (SSD)	998	499
		File2 (HDD)	118	59
		File3 (Mixed)	198	99

Note 1: In general, this is working as expected. File1 (SSD Tier) shows SSD performance characteristics while File 2 (HDD Tier) behaves like a spinning disk. File3 is somewhere between the two.

Note 2: These results are not meant to showcase the true performance of a production tiered storage solution. Keep in mind that there is only one physical SSD and one physical HDD behind all the virtual layers. Both simple and mirrored are spaces are basically limited by the hardware limitations of the two disks.

Note 3: This shows the results of a single run of each workload, so some variance is expected. If you are running this in your own setup, you might want to run each workload multiple times and average it out. There’s some guidance to that regard at http://blogs.technet.com/b/josebda/archive/2013/03/25/sqlio-powershell-and-storage-performance-measuring-iops-throughput-and-latency-for-both-local-disks-and-smb-file-shares.aspx

22. Extending existing Spaces and Volumes

# Check state before change

Get-VirtualDisk Space1 | FT -AutoSize

Get-StorageTier Space1* | FT FriendlyName, Size –AutoSize

Get-StorageTierSupportedSize SSDTier -ResiliencySettingName Simple | FT -AutoSize

Get-VirtualDisk Space1 | Get-Disk | FT -AutoSize

Get-VirtualDisk Space1 | Get-Disk | Get-Partition | FT –AutoSize

Get-VirtualDisk Space1 | Get-Disk | Get-Partition | Get-Volume | FT –AutoSize

# Add 4GB on the SSD Tier

Resize-StorageTier Space1_SSDTier -Size 12GB

Get-VirtualDisk Space1 | Get-Disk | Update-Disk

# Check after Virtual Disk change

Get-VirtualDisk Space1 | FT -AutoSize

Get-StorageTier Space1* | FT FriendlyName, Size –AutoSize

Get-StorageTierSupportedSize SSDTier -ResiliencySettingName Simple | FT -AutoSize

Get-VirtualDisk Space1 | Get-Disk | FT -AutoSize

Get-VirtualDisk Space1 | Get-Disk | Get-Partition | FT –AutoSize

Get-VirtualDisk Space1 | Get-Disk | Get-Partition | Get-Volume | FT –AutoSize

# Extend partition (also extends the volume)

Resize-Partition -DriveLetter F -Size 43.87GB

# Check after Partition/Volume change

Get-VirtualDisk Space1 | Get-Disk | Get-Partition | FT –AutoSize

Get-VirtualDisk Space1 | Get-Disk | Get-Partition | Get-Volume | FT –AutoSize

23. Extending existing Spaces and Volumes(with output)

PS C:\> Get-VirtualDisk Space1 | FT -AutoSize

FriendlyName ResiliencySettingName OperationalStatus HealthStatus IsManualAttach Size

------------ --------------------- ----------------- ------------ -------------- ----

Space1 Simple OK Healthy False 40 GB

PS C:\> Get-StorageTier Space1* | FT FriendlyName, Size –AutoSize

FriendlyName Size

------------ ----

Space1_SSDTier 8589934592

Space1_HDDTier 34359738368

PS C:\> Get-StorageTierSupportedSize SSDTier -ResiliencySettingName Simple | FT -AutoSize

SupportedSizes TierSizeMin TierSizeMax TierSizeDivisor

-------------- ----------- ----------- ---------------

{} 4294967296 4294967296 4294967296

PS C:\> Get-VirtualDisk Space1 | Get-Disk | FT -AutoSize

Number Friendly Name OperationalStatus Total Size Partition Style

------ ------------- ----------------- ---------- ---------------

13 Microsoft Storage Space Device Online 40 GB GPT

PS C:\> Get-VirtualDisk Space1 | Get-Disk | Get-Partition | FT -AutoSize

Disk Number: 13

PartitionNumber DriveLetter Offset Size Type

--------------- ----------- ------ ---- ----

1 24576 128 MB Reserved

2 F 135266304 39.87 GB Basic

PS C:\> Get-VirtualDisk Space1 | Get-Disk | Get-Partition | Get-Volume | FT -AutoSize

DriveLetter FileSystemLabel FileSystem DriveType HealthStatus SizeRemaining Size

----------- --------------- ---------- --------- ------------ ------------- ----

F NTFS Fixed Healthy 27.75 GB 39.87 GB

PS C:\> Resize-StorageTier Space1_SSDTier -Size 12GB

PS C:\> Get-VirtualDisk Space1 | Get-Disk | Update-Disk

PS C:\> Get-VirtualDisk Space1 | FT -AutoSize

FriendlyName ResiliencySettingName OperationalStatus HealthStatus IsManualAttach Size

------------ --------------------- ----------------- ------------ -------------- ----

Space1 Simple OK Healthy False 44 GB

PS C:\> Get-StorageTier Space1* | FT FriendlyName, Size –AutoSize

FriendlyName Size

------------ ----

Space1_SSDTier 12884901888

Space1_HDDTier 34359738368

PS C:\> Get-StorageTierSupportedSize SSDTier -ResiliencySettingName Simple | FT -AutoSize

SupportedSizes TierSizeMin TierSizeMax TierSizeDivisor

-------------- ----------- ----------- ---------------

{} 0 0 0

PS C:\> Get-VirtualDisk Space1 | Get-Disk | FT -AutoSize

Number Friendly Name OperationalStatus Total Size Partition Style

------ ------------- ----------------- ---------- ---------------

13 Microsoft Storage Space Device Online 44 GB GPT

PS C:\> Get-VirtualDisk Space1 | Get-Disk | Get-Partition | FT –AutoSize

Disk Number: 13

PartitionNumber DriveLetter Offset Size Type

--------------- ----------- ------ ---- ----

1 24576 128 MB Reserved

2 F 135266304 39.87 GB Basic

PS C:\> Get-VirtualDisk Space1 | Get-Disk | Get-Partition | Get-Volume | FT –AutoSize

DriveLetter FileSystemLabel FileSystem DriveType HealthStatus SizeRemaining Size

----------- --------------- ---------- --------- ------------ ------------- ----

F NTFS Fixed Healthy 27.75 GB 39.87 GB

PS C:\> Resize-Partition -DriveLetter F -Size 43.87GB

PS C:\> Get-VirtualDisk Space1 | Get-Disk | Get-Partition | FT –AutoSize

Disk Number: 13

PartitionNumber DriveLetter Offset Size Type

--------------- ----------- ------ ---- ----

1 24576 128 MB Reserved

2 F 135266304 43.87 GB Basic

PS C:\> Get-VirtualDisk Space1 | Get-Disk | Get-Partition | Get-Volume | FT –AutoSize

DriveLetter FileSystemLabel FileSystem DriveType HealthStatus SizeRemaining Size

----------- --------------- ---------- --------- ------------ ------------- ----

F NTFS Fixed Healthy 31.75 GB 43.87 GB

24. Final Notes

I hope you enjoyed these step-by-step instructions. I strongly encourage you to try them out and perform the entire installation yourself. It’s a good learning experience.
Let me know how these steps worked for you using the comment section. If you run into any issues or found anything particularly interesting, don’t forget to mention the number of the step.

↧

What’s new in SMB PowerShell in Windows Server 2012 R2

September 3, 2013, 1:20 pm

≫ Next: Raw notes from the Storage Developers Conference (SDC 2013)

≪ Previous: Step-by-step for Storage Spaces Tiering in Windows Server 2012 R2

Windows Server 2012 R2 introduced a new version of SMB. Technically it’s SMB 3.02, but we continue to call it just SMB3. The main changes are described at http://technet.microsoft.com/en-us/library/hh831474.aspx. With this new release, we made a few changes in SMB PowerShell to support the new scenarios and features. This includes a few new cmdlets and some changes to existing cmdlets, with extra care not break any of your existing scripts. This blog post outlines 7 set of changes related to SMB PowerShell in Windows Server 2012 R2.

1) Simpler setting of ACL for the folder behind a share

In Windows Server 2012, the SMB share has a property that facilitates applying the share ACL to the file system folder used by the share. Here’s the syntax for a share named Share1:

(Get-SmbShare –Name Share1 ).PresetPathACL | Set-Acl

In Windows Server 2012 R2, we have improved this scenario by providing a proper cmdlet to apply the share ACL to the file system used by the share. Here’s the new syntax for the same share:

Set-SmbPathAcl –ShareName Share1

Note1 : The Windows Server 2012 syntax continues to work with Windows Server 2012 R2, but the new syntax is much simpler and therefore recommended.

Note 2: There is known issue with Windows Server 2012 R2 Preview that causes this new cmdlet to fail when using non-Unicode languages. As a workaround, you can use the old syntax.

2) Scale-Out Rebalancing (per-share redirection)

Per-share redirection is the new default behavior for the new SMB Clients (Windows 8.1 or Windows Server 2012 R2), when connecting to a Scale-Out clusters that use a storage system that does not support Direct I/O from all nodes, if running Windows Server 2012 R2. The common scenario here is a Scale-Out File Server backed by Mirrored Storage Spaces.

Here are the changes in SMB PowerShell to support SMB Scale-Out per-share redirection:

New “Redirected” Boolean property to Get-SmbConnection to indicate that per-share redirection is being used.
Get-SmbWitnessClient now includes a “ShareName” property, since witness can track connections per share, not only per server.
Get-SmbWitnessClient now includes a “Flags” property, which will show “Share” when doing per-share redirection

3) Other SMB Witness Changes

While the overall functionality of the SMB Witness is largely unchanged in Windows Server 2012 R2 (aside from per-share redirection), we have put some effort in improving the SMB PowerShell cmdlets associated with it. Here are the changes:

The Move-SmbWitnessClient cmdlet can be referred to now simply as Move-SmbClient, This is simply a new alias that better describes what the cmdlet actually does.
The default view for Get-SmbWitnessClient was improved. Here’s the new list of items shown:

Client Computer   Witness Node File Server   Network Name Share Name  Client State
Name              Name             Node Name
----------------- ------------- ------------- ------------ ----------  ------------
JOSE-V            JOSE-A1       JOSE-A2       JOSE-F        VMS4        RequestedNoti...
JOSE-V            JOSE-A1       JOSE-A2       JOSE-F        VMS3        RequestedNoti...
JOSE-V            JOSE-A1       JOSE-A3       JOSE-F        VMS1        RequestedNoti...
JOSE-V            JOSE-A1       JOSE-A3       JOSE-F        VMS2        RequestedNoti...
JOSE-V            JOSE-A2       JOSE-A1       JOSE-F        VMS6        RequestedNoti...
JOSE-V            JOSE-A2       JOSE-A1       JOSE-F        VMS5        RequestedNoti...

There is a new “NetworkName” parameter in Move-SmbClient. If a NetworkName is specified, then only those will be moved.

Windows Server 2012 syntax for Move-SmbWitnessClient:

Move-SmbWitnessClient -ClientName X -DestinationNode Y

Windows Server 2012 R2 syntax:

Move-SmbClient -ClientName X -DestinationNode Y [ -NetworkName Z ]

Note 1: If the –NetworkName is omitted in the Move-SmbClient cmdlet, all client connections will be moved to the destination.

Note 2: When using per-share redirection, the SMB client will always move to the file server node that owns the volume behind the file share. Using Move-SmbClient (or Move-SmbWitnessClient) in that situation has no effect.

4) SMB Bandwidth Limits

Starting with Windows Server 2012 R2, administrators can set a bandwidth limit for one or more categories by using simple PowerShell cmdlets. They can also query the limit for each category.

There are 3 categories of traffic that can be limited:

VirtualMachine: Hyper-V over SMB traffic. This limit can only be applied on Hyper-V hosts.
LiveMigration: Hyper-V Live Migration over SMB traffic. This limit can only be applied on Hyper-V hosts.
Default: All other types of SMB traffic. This limit can be applied on any SMB client.

A category limit is an absolute value expressed in bytes per second. For example, a category limit of 500MB means the total throughput for this category should not exceed 500 megabytes per second.

The PowerShell cmdlets used to manage SMB Bandwidth Limits are:

Get-SmbBandwidthLimit [ –Category {Default/VirtualMachine/LiveMigration} ]
Set-SmbBandwidthLimit –Category {Default/VirtualMachine/LiveMigration} –BytesPerSecond x
Remove-SmbBandwidthLimit –Category {Default/ VirtualMachine/LiveMigration}

Note 1: Before you can use Bandwidth Limits, the feature needs to be enabled. For that, you should use:

Add-WindowsFeature FS-SMBBW

Note 2: A new SMB Performance counter set with an instance per category becomes available after you install the feature. The performance counters in for this set will use the same counters used today for the SMB Client Shares counter.

Note 3: PowerShell allows us to specify units like KB, MB, GB after the number of bytes when specifying the BytesPerSecond parameter.

Note 4: The Set-SmbBandwidthLimit cmdlet won’t accept a BytesPerSecond parameter smaller than 1MB (1048576).

5) SMB Multi-instance

SMB Multi-instance is a new feature in Windows Server 2012 R2 that separates regular SMB traffic from CSV-related inter-node SMB traffic in distinct SMB instances. This is designed to improve isolation between the two types of traffic in improve the reliability of the SMB servers. Information related to this new CSV-only instance in Windows Server 2012 R2 is hidden in all PowerShell cmdlets.

However, there are the changes in SMB PowerShell so an Administrator can view information related to the hidden CSV instance:

The –IncludeHidden option in Get-SmbConnection and Get-SmbMultichannelConnection will show the connections associated with the hidden CSV instance.
There is now an InstanceName property in the full output of Get-SmbConnection, Get-SmbMultichannelConnection, Get-SmbSession and Get-SmbOpenFile. It shows either “Default” or “CSV” (only shows if using the –IncludeHidden option).

Note: There is really little use in inspecting the information on the hidden CSV instance, except if you’re troubleshooting CSV inter-node communications.

6) SMB Delegation

When you configure Hyper-V over SMB and you manage your Hyper-V hosts remotely using Hyper-V Manager, you will might run into access denied messages. This is because you’re using your credentials from the remote machine running Hyper-V Manager in the Hyper-V host to access a third machine (the file server). This is what we call a “double-hop”, and it’s not allowed by default for security reasons.The main problem with the scenario is that an intruder that compromises one computer in your environment could then connect to other systems in your environments without the need to provide a username and password. One way to work around this issue is to connect directly to the Hyper-V host and providing your credentials at that time, avoiding the double-hop.

You can also address this by configuring Constrained Delegation for SMB shares, which is a process that involves changing properties in Active Directory. The security risk is reduced here because a potential intruder double-hop would be limited to that specific use case (using SMB shares on the specified servers). The constrained delegation process was greatly simplified in Windows Server 2012 when the the Active Directory team introduced resource-based Kerberos constrained delegation, as explained at http://technet.microsoft.com/library/hh831747.aspx. However, even with this new resource-based constrained delegation, there are still quite a few steps to enable it.

For Hyper-V over SMB in Windows Server 2012, we provided TechNet and blog-based guidance on how to automate constrained delegation. In Windows Server 2012 R2, SMB has a new set of cmdlets to simplify the configuration of resource-based constrained Delegation in SMB scenarios. Here are the new cmdlets introduced:

Get-SmbDelegation –SmbServer X
Enable-SmbDelegation –SmbServer X –SmbClient Y
Disable-SmbDelegation –SmbServer X [–SmbClient Y] [-Force]

Note 1: For the Disable-SmbDelegation cmdlet, if no client is specified, delegation will be removed for all clients.

Note 2: These 3 new SMB cmdlets do rely on Active Directory PowerShell to perform their actions. For this reason, you need to install the Active Directory cmdlets before using the SMB delegation cmdlets. To install the Active Directory cmdlets, use:

Install-WindowsFeature RSAT-AD-PowerShell

Note 3: Because these cmdlets only work with the new resource-based delegation, the Active Directory forest must be in “Windows Server 2012” functional level. To check the Active Directory Forest Functional level, use:

Get-ADForest

Note 4: Virtual Machine Manager uses a different method to remote into the Hyper-V host. For VMM, constrained delegation is not required for management of Hyper-V of SMB.

7) Disable SMB1

SMB1 now can be completely disabled, so that the components are not even loaded. For scenarios where SMB1 is not required, this means less resource utilization, less need for patching and improved security.

To disable SMB1 completely, use the following PowerShell cmdlet:

Remove-WindowsFeature FS-SMB1

You can re-enable it by using:

Add-WindowsFeature FS-SMB1

Note 1: A reboot is required after this feature is enabled or disabled.

Note 2: For information worker scenarios, if you have Windows XP clients, you absolutely still need SMB1, since that is the only SMB version supported by Windows XP. Windows Vista and Windows 7 do not need SMB1 since they support SMB2. Windows 8 and Windows 8.1 do not need SMB1, since they support both SMB2 and SMB3.

Note 3: For classic server scenarios, if you have Windows Server 2003 or Windows Server 2003 R2 servers, you absolutely still need SMB1, since that is the only SMB version supported by them. Windows Server 2008 and Windows Server 2008 R2 do not need SMB1 since they support SMB2. Windows Server 2012 and Windows Server 2012 R2 do not need SMB1, since they support both SMB2 and SMB3.

↧

Raw notes from the Storage Developers Conference (SDC 2013)

September 19, 2013, 5:25 pm

≫ Next: Windows Server 2012 R2: Which version of the SMB protocol (SMB 1.0, SMB 2.0, SMB 2.1, SMB 3.0 or SMB 3.02) are you using?

≪ Previous: What’s new in SMB PowerShell in Windows Server 2012 R2

This blog post is a compilation of my raw notes from SNIA’s SDC 2013 (Storage Developers Conference).

Notes and disclaimers:

These notes were typed during the talks and they may include typos and my own misinterpretations.
Text in the bullets under each talk are quotes from the speaker or text from the speaker slides, not my personal opinion.
If you feel that I misquoted you or badly represented the content of a talk, please add a comment to the post.
I spent limited time fixing typos or correcting the text after the event. Just so many hours in a day...
I have not attended all sessions (since there are 4 or 5 at a time, that would actually not be possible :-)…
SNIA usually posts the actual PDF decks a few weeks after the event. Attendees have access immediately.
You can find the event agenda at http://www.snia.org/events/storage-developer2013/agenda2013

SMB3 Meets Linux: The Linux Kernel Client
Steven French, Senior Engineer SMB3 Architecture, IBM

Title showing is (with the strikethrough text): CIFSSMB2SMB2.1SMB3 SMB3.02 and Linux, a Status Update.
How do you use it? What works? What is coming?
Who is Steven French: maintains the Linux kernel client, at SMB3 Architect for IBM Storage
Excited about SMB3
Why SMB3 is important: cluster friendly, large IO sizes, more scalable.
Goals: local/remote transparency, near POSIX semantics to Samba, fast/efficient/full function/secure method, as reliable as possible over bad networks
Focused on SMB 2.1, 3, 3.02 (SMB 2.02 works, but lower priority)
SMB3 faster than CIFS. SMB3 remote file access near local file access speed (with RDMA)
Last year SMB 2.1, this year SMB 3.0 and minimal SMB 3.02 support
308 kernel changes this year, a very active year. More than 20 developers contributed
A year ago 3.6-rc5 – now at 3.11 going to 3.12
Working on today copy offload, full linux xattr support, SMB3 UNIX extension prototyping, recover pending locks, starting work on Multichannel
Outline of changes in the latest releases (from kernel version 3.4 to 3.12), version by version
Planned for kernel 3.13: copy chunk, quota support, per-share encryption, multichannel, considering RDMA (since Samba is doing RDMA)
Improvements for performance: large IO sizes, credit based flow control, improved caching model. Still need to add compounding,
Status: can negotiate multiple dialects (SMB 2.1, 3, 3.02)
Working well: basic file/dir operations, passes most functional tests, can follow symlinks, can leverage durable and persistent handles, file leases
Need to work on: cluster enablement, persistent handles, witness, directory leases, per-share encryption, multichannel, RDMA
Plans: SMB 2.1 no longer experimental in 3.12, SMB 2.1 and 3 passing similar set of functional tests to CIFS
Configuration hints: adjusting rsize, wsize, max_pending, cache, smb3 signing, UNIX extension, nosharelock
UNIX extensions: POSIX pathnames, case sensitive path name, POSIX delete/rename/create/mkdir, minor extensions to stat/statfs, brl, xattr, symlinks, POSIX ACLs
Optional POSIX SMB3 features outlined: list of flags used for each capability
Question: Encryption: Considering support for multiple algorithms, since AES support just went in the last kernel.
Development is active! Would like to think more seriously about NAS appliances. This can be extended…
This is a nice, elegant protocol. SMB3 fits well with Linux workloads like HPC, databases. Unbelievable performance with RDMA.
Question: Cluster enablement? Durable handle support is in. Pieces missing for persistent handle and witness are small. Discussing option to implement and test witness.
Need to look into the failover timing for workloads other than Hyper-V.
Do we need something like p-NFS? Probably not, with these very fast RDMA interfaces…

Mapping SMB onto Distributed Storage
Christopher R. Hertel, Senior Principal Software Engineer, Red Hat
José Rivera, Software Engineer, Red Hat

Trying to get SMB running on top of a distributed file system, Gluster
Chris and Jose: Both work for RedHat, both part of the Samba team, authors, etc…
Metadata: data about data, pathnames, inode numbers, timestamps, permissions, access controls, file size, allocation, quota.
Metadata applies to volumes, devices, file systems, directories, shares, files, pipes, etc…
Semantics are interpreted in different contexts
Behavior: predictable outcomes. Make them the same throughout the environments, even if they are not exactly the same
Windows vs. POSIX: different metadata + different semantics = different behavior
That’s why we have a plugfest downstairs
Long list of things to consider: ADS, BRL, deleteonclose, directory change notify, NTFS attributes, offline ops, quota, etc…
Samba is a Semantic Translator. Clients expect Windows semantics from the server, Samba expects POSIX semantics from the underlying file system
UNIX extensions for SMB allows POSIX clients to bypass some of this translation
If Samba does not properly handle the SMB protocol, we call it a bug. If cannot handle the POSIX translation, it’s also a bug.
General Samba approach: Emulate the Windows behavior, translate the semantics to POSIX (ensure other local processes play by similar rules)
The Samba VFS layers SMB Protocol Initial Request Handling  VFS Layer  Default VFS Layer  actual file system
Gluster: Distributed File System, not a cluster file system. Brick  Directory in the underlying file system. Bricks bound together as a volume. Access via SMB, NFS, REST.
Gluster can be FUSE mounted. Just another access method. FUSE hides the fact that it’s Gluster underneath.
Explaining translations: Samba/Gluster/FUSE. Gluster is adaptable. Translator stack like Samba VFS modules…
Can add support for: Windows ACLs, oplocks, leases, Windows timestamps.
Vfs_glusterfs: Relatively new code, similar to other Samba VFS modules. Took less than a week to write.
Can bypass the lower VFS layers by using libgfapi. All VFS calls must be implemented to avoid errors.
CTDB offers three basics services: distributed metadata database (for SMB state), node failure detection/recovery, IP address service failover.
CTDB forms a Samba cluster. Separate from the underlying Gluster cluster. May duplicate some activity. Flexible configuration.
SMB testing, compared to other access methods: has different usage patterns, has tougher requirements, pushes corner cases.
Red Hat using stable versions, kernel 2.x or something. So using SMB1 still…
Fixed: Byte range locking. Fixed a bug in F_GETLK to get POSIX byte range locking to work.
Fixed: SMB has strict locking and data consistency requirements. Stock Gluster config failed ping_pong test. Fixed cache bugs  ping_pong passes
Fixed: Slow directory lookups. Samba must do extra work to detect and avoid name collisions. Windows is case-INsensitive, POSIX is case-sensitive. Fixed by using vfs_glusterfs.
Still working on: CTDB node banning. Under heavy load (FSCT), CTDB permanently bans a running node. Goal: reach peak capacity without node banning. New CTDB versions improved capacity.
Still working on: CTDB recovery lock file loss. Gluster is a distributed FS, not a Cluster FS. In replicated mode, there are two copies of each file. If Recovery Lock File is partitioned, CTDB cannot recover.
Conclusion: If implementing SMB in a cluster or distributed environment, you should know enough about SMB to know where to look for trouble… Make sure metadata is correct and consistent.
Question: Gluster and Ceph have VFS. Is Samba suitable for that? Yes. Richard wrote a guide on how to write a VFS. Discussing a few issues around passing user context.
Question: How to change SMB3 to be more distributed? Client could talk to multiple nodes. Gluster working on RDMA between nodes. Protocol itself could offer more about how the cluster is setup.

Pike - Making SMB Testing Less Torturous
Brian Koropoff, Consulting Software Engineer, EMC Isilon

Pike – written in Python – starting with a demo
Support for a modest subset of SMB2/3. Currently more depth than breadth.
Emphasis on fiddly cases like failover, complex creates
Mature solutions largely in C (not convenient for prototyping)
Why python: ubiquitous, expressive, flexible, huge ecosystem.
Flexibility and ease of use over performance. Convenient abstractions. Extensible, re-usable.
Layers: core primitives (abstract data model), SMB2/3 packet definitions, SMB2/3 client model (connection, state, request, response), test harness
Core primitives: Cursor (buffer+offset indicating read/write location), frame (packet model), enums, anti-boilerplate magic. Examples.
SMB2/SMB3 protocol (pike.smb2) header, request/response, create {request/response} context, concrete frame. Examples.
SMB2/SMB3 model: SMB3 object model + glue. Future, client, connection (submit, trasceive, error handling), session, channel (treeconnect, create, read), tree, open, lease, oplocks.
Examples: Connect, tree connect, create, write, close. Oplocks. Leases.
Advanced uses. Manually construct and submit exotic requests. Override _encode. Example of a manual request.
Test harness (pike,test): quickly establish connection, session and tree connect to server. Host, credentials, share parameters taken from environment.
Odds and ends: NT time class, signing, key derivation helpers.
Future work: increase breadth of SMB2/3 support. Security descriptors, improvement to mode, NTLM story, API documentation, more tests!
http://github.com/emc-isilon/pike - open source, patches are welcome. Has to figure out how to accept contributions with lawyers…
Question: Microsoft has a test suite. It’s in C#, doesn’t work in our environment. Could bring it to the plugfest.
Question: I would like to work on implementing it for SMB1. What do you think? Not a priority for me. Open to it, but should use a different model to avoid confusion.
Example: Multichannel. Create a session, bind another channel to the same session, pretend failover occurred. Write fencing of stable write.

Exploiting the High Availability features in SMB 3.0 to support Speed and Scale
James Cain, Principal Software Architect, Quantel Ltd

Working with TV/Video production. We only care about speed.
RESTful recap. RESTful filesystems talk from SDC 2010. Allows for massive scale by storing application state in the URLs instead of in the servers.
Demo (skipped due to technical issues): RESTful SMB3.
Filling pipes: Speed (throughput) vs. Bandwidth vs. Latency. Keeping packets back to back on the wire.
TCP Window size used to limit it. Mitigate by using multiple wires, multiple connections.
Filling the pipes: SMB1 – XP era. Filling the pipes required application participation. 1 session could do about 60MBps. Getting Final Cut Pro 7 to lay over SMB1 was hard. No choice to reduce latency.
Filling the pipes: SMB 2.0 – Vista era. Added credits, SMB2 server can control overlapped requests using credits. Client application could make normal requests and fill the pipe.
Filling the pipes: SMB 2.1 – 7 era. Large MTU helps.
Filling the pipes: SMB 3 – 8 era. Multi-path support. Enables: RSS, Multiple NICs, Multiple machines, RDMA.
SMB3 added lots of other features for high availability and fault tolerance. SignKey derivation.
Filesystem has DirectX GUI :-) - We use GPUs to render, so our SMB3 server has Cuda compute built in too. Realtime visualization tool for optimization.
SMB3 Multi-machine with assumed shared state. Single SMB3 client talking to two SMB3 servers. Distributed non-homogeneous storage behind the SMB servers.
Second NIC (channel) initiation has no additional CREATE. No distinction on the protocol between single server or multiple server. Assume homogeneous storage.
Asking Microsoft to consider “NUMA for disks”. Currently, shared nothing is not possible. Session, trees, handles are shared state.
“SMB2++” is getting massive traction. Simple use cases are well supported by the protocol. SMB3 has a high cost of entry, but lower than writing n IFS in kernel mode.
There are limits to how far SMB3 can scale due to its model.
I know this is not what the protocol is designed to do. But want to see how far I can go.
It could be help by changing the protocol to have duplicate handle semantics associated with the additional channels.
The protocol is really, really flexible. But I’m having a hard time doing what I was trying to do.
Question: You’re basic trying to do Multichannel to multiple machines. Do you have a use case? I’m experimenting with it. Trying to discover new things.
Question: You could use CTDB to solve the problem. How much would it slow down? It could be a solution, not an awful lot of state.

SMB3 Update
David Kruse, Development Lead, Microsoft

SMB 3.02 - Don’t panic! If you’re on the road to SMB3, there are no radical changes.
Considered not revving the dialect and doing just capability bits, but thought it would be better to rev the dialect.
Dialects vs. Capabilities: Assymetric Shares, FILE_ATTRIBUTE_INTEGRITY_STREAMS.
SMB 2.0 client attempting MC or CA? Consistency/documentation question.
A server that receives a request from a client with a flag/option/capability that is not valid for the dialect should ignore it.
Showing code on how to mask the capabilities that don’t make sense for a specific dialect
Read/Write changes: request specific flag for unbuffered IO. RDMA flag for invalidation.
Comparing “Traditional” File Server Cluster vs. “Scale-Out” File Server cluster
Outlining the asymmetric scale-out file server cluster. Server-side redirection. Can we get the client to the optimal node?
Asymmetric shares. New capability in the TREE_CONNECT response. Witness used to notify client to move.
Different connections for different shares in the same scale-out file server cluster. Share scope is the unit of resource location.
Client processes share-level “move” in the same fashion as a server-level “move” (disconnect, reconnects to IP, rebinds handle).
If the cost accessing the data is the same for all nodes, there is no need to move the client to another node.
Move-SmbWitnessClient will not work with asymmetric shares.
In Windows, asymmetric shares are typically associated with Mirrored Storage Spaces, not iSCSI/FC uniform deployment. Registry key to override.
Witness changes: Additional fields: Sharename, Flags, KeepAliveTimeOutInSeconds.
Witness changes: Multichannel notification request. Insight into arrival/loss of network interfaces.
Witness changes: Keepalive. Timeout for async IO are very coarse. Guarantees client and server discover lost peer in minutes instead of hours.
Demos in Jose’s blog. Thanks for the plug!
Diagnosability events. New always-on events. Example: failed to reconnect a persistent handle includes previous reconnect error and reason. New events on server and client.
If Asymmetric is not important to you, you don’t need to implement it.
SMB for IPC (Inter-process communications) – What happened to named pipes?
Named pipes over SMB has been declined in popularity. Performance concerns with serialized IO. But this is a property of named pipes, not SMB.
SMB provides: discovery, negotiation, authentication, authorization, message semantics, multichannel, RDMA, etc…
If you can abstract your application as a file system interface, you could extend it to removte via SMB.
First example: Remote Shared Virtual Disk Protocol
Second example: Hyper-V Live Migration over SMB. VID issues writes over SMB to target for memory pages. Leverages SMB Multichannel, SMB Direct.
Future thoughts on SMB for IPC. Not a protocol change or Microsoft new feature. Just ideas shared as a thought experiment.

MessageFs – User mode-client and user-mode server. Named Pipes vs. MessageFs. Each offset marks a distinct transaction, enables parallel actions.
MemFs – Kernel mode component on the server side. Server registers a memory region and clients can access that memory region.
MemFs+ - What if we combine the two? Fast exchange for small messages plus high bandwidth, zero copy access for large transfers. Model maps directly to RDMA: send/receive messages, read/write memory access.

One last thing… On Windows 8.1, you can actually disable SMB 1.0 completely.

Architecting Block and Object Geo-replication Solutions with Ceph
Sage Weil, Founder & CTO, Inktank

Impossible to take notes, speaker goes too fast :-)

1 S(a) 2 M 3 B(a) 4
Michael Adam, SerNet GmbH - Delivered by Volker

What is Samba? The open source SMB server (Samba3). The upcoming open source AD controller (Samba4). Two different projects.
Who is Samba? List of team members. Some 35 or so people… www.samba.org/samba/team
Development focus: Not a single concentrated development effort. Various companies (RedHat, SuSE, IBM, SerNet, …) Different interests, changing interests.
Development quality: Established. Autobuild selftest mechanism. New voluntary review system (since October 2012).
What about Samba 4.0 after all?

First (!?) open source Active Directory domain controller
The direct continuation of the Samba 3.6 SMB file server
A big success in reuniting two de-facto separated projects!
Also a big and important file server release (SMB 2.0 with durable handles, SMB 2.1 (no leases), SMB 3.0 (basic support)

History. Long slide with history from 2003-06-07 (Samba 3.0.0 beta 1) to 2012-12-11 (Samba 4.0.0). Samba4 switched to using SMB2 by default.
What will 4.1 bring? Current 4.1.0rc3 – final planned for 2013-09-27.
Samba 4.1 details: mostly stabilization (AD, file server). SMB2/3 support in smbclient, including SMB3 encryption. Server side copy. Removed SWAT.
Included in Samba 4.0: SMB 2.0 (durable handles). SMB 2.1 (multi-credit, large MTU, dynamic reauth), SMB 3.0 (signing, encryption, secure negotiate, durable handles v2)
Missing in Samba 4.0: SMB 2.1 (leasing*, resilient file handles), SMB 3.0 (persistent file handles, multichannel*, SMB direct*, witness*, cluster features, storage features*, …) *=designed, started or in progress
Leases: Oplocks done right. Remove 1:1 relationship between open and oplock, add lease/oplock key. http://wiki.samba.org/index.php/Samba3/SMB2#Leases
Witness: Explored protocol with Samba rpcclient implementation. Working on pre-req async RPC. http://wiki.samba.org/index.php/Samba3/SMB2#Witness_Notification_Protocol
SMB Direct: Currently approaching from the Linux kernel side. See related SDC talk. http://wiki.samba.org/index.php/Samba3/SMB2#SMB_Direct
Multichannel and persistent handles: just experimentation and discussion for now. No code yet.

Keynote: The Impact of the NVM Programming Model
Andy Rudoff, Intel

Title is Impact of NVM Programming Model (… and Persistent Memory!)
What do we need to do to prepare, to leverage persistent memory
Why now? Programming model is decades old!
What changes? Incremental changes vs. major disruptions
What does this means to developers? This is SDC…
Why now?
One movements here: Block mode innovation (atomics, access hints, new types of trim, NVM-oriented operations). Incremental.
The other movement: Emerging NVM technologies (Performance, performance, perf… okay, Cost)
Started talking to companies in the industry  SNIA NVM Programming TWG - http://snia.org/forums/sssi/nvmp
NVM TWG: Develop specifications for new software “programming models”as NVM becomes a standard feature of platforms
If you don’t build it and show that it works…
NVM TWG: Programming Model is not an API. Cannot define those in a committee and push on OSVs. Cannot define one API for multiple OS platforms
Next best thing is to agree on an overall model.
What changes?
Focus on major disruptions.
Next generation scalable NVM: Talking about resistive RAM NVM options. 1000x speed up over NND, closer do DRAM.
Phase Change Memory, Magnetic Tunnel Junction (MT), Electrochemical Cells (ECM), Binary Oxide Filament Cells, Interfacial Switching
Timing. Chart showing NAND SATA3 (ONFI2, ONFI3), NAND PCIe Gen3 x4 ONFI3 and future NVM PCIE Gen3 x4.
Cost of software stack is not changing, for the last one (NVM PCIe) read latency, software is 60% of it?!
Describing Persistent Memory…
Byte-addressable (as far as programming model goes), load/store access (not demand-paged), memory-like performance (would stall a CPU load waiting for PM), probably DMA-able (including RDMA)
For modeling, think battery-backed RAM. These are clunky and expensive, but it’s a good model.
It is not tablet-like memory for the entire system. It is not NAND Flash (at least not directly, perhaps with caching). It is not block-oriented.
PM does not surprise the program with unexpected latencies (no major page faults). Does not kick other things out of memory. Does not use page cache unexpectedly.
PM stores are not durable until data is flushed. Looks like a bug, but it’s always been like this. Same behavior that’s been around for decades. It’s how physics works.
PM may not always stay in the same address (physically, virtually). Different location each time your program runs. Don’t store pointers and expect them to work. You have to use relative pointers. Welcome to the world of file systems…
Types of Persistent Memory: Battery-backed RAM. DRAM saved on power failure. NVM with significant caching. Next generation NVM (still quite a bit unknown/emerging here).
Existing use cases: From volatile use cases (typical) to persistent memory use case (emerging). NVDIMM, Copy to Flash, NVM used as memory.
Value: Data sets with no DRAM footprint. RDMA directly to persistence (no buffer copy required!). The “warm cache” effect. Byte-addressable. Direct user-mode access.
Challenges: New programming models, API. It’s not storage, it’s not memory. Programming challenges. File system engineers and database engineers always did this. Now other apps need to learn.
Comparing to the change that happened when we switched to parallel programming. Some things can be parallelized, some cannot.
Two persistent memory programming models (there are four models, more on the talk this afternoon).
First: NVM PM Volume mode. PM-aware kernel module. A list of physical ranges of NVMs (GET_RANGESET).
For example, used by file systems, memory management, storage stack components like RAID, caches.
Second: NVM PM File. Uses a persistent-memory-aware file system. Open a file and memory map it. But when you do load and store you go directly to persistent memory.
Native file APIs and management. Did a prototype on Linux.
Application memory allocation. Ptr=malloc(len). Simple, familiar interface. But it’s persistent and you need to have a way to get back to it, give it a name. Like a file…
Who uses NVM.PM.FILE. Applications, must reconnect with blobs of persistence (name, permissions)
What does it means to developers?
Mmap() on UNIX, MapViewOfFile() on Windows. Have been around for decades. Present in all modern operating systems. Shared or Copy-on-write.
NVM.PM.FILE – surfaces PM to application. Still somewhat raw at this point. Two ways: 1-Build on it with additional libraries. 2-Eventually turn to language extensions…
All these things are coming. Libraries, language extensions. But how does it work?
Creating resilient data structures. Resilient to a power failure. It will be in state you left it before the power failure. Full example: resilient malloc.
In summary: models are evolving. Many companies in the TWG. Apps can make a big splash by leveraging this… Looking forward to libraries and language extensions.

Keynote: Windows Azure Storage – Scaling Cloud Storage
Andrew Edwards, Microsoft

Turning block devices into very, very large block devices. Overview, architecture, key points.
Overview
Cloud storage: Blobs, disks, tables and queues. Highly durable, available and massively scalable.
10+ trillion objects. 1M+ requests per seconds average. Exposed via easy and open REST APIs
Blobs: Simple interface to retrieve files in the cloud. Data sharing, big data, backups.
Disks: Built on top on blobs. Mounted disks as VHDs stored on blobs.
Tables: Massively scalable key-value pairs. You can do queries, scan. Metadata for your systems.
Queues: Reliable messaging system. Deals with failure cases.
Azure is spread all over the world.
Storage Concepts: Accounts  ContainerBlobs/TableEntities/QueuesMessages. URLs to identify.
Used by Microsoft (XBOX, SkyDrive, etc…) and many external companies
Architecture
Design Goals: Highly available with strong consistency. Durability, scalability (to zettabytes). Additional information in the SOSP paper.
Storage stamps: Access to blog via the URL. LB  Front-end  Partition layer  DFS Layer. Inter-stamp partition replication.
Architecture layer: Distributed file system layer. JBODs, append-only file system, each extent is replicated 3 times.
Architecture layer: Partition layer. Understands our data abstractions (blobs, queues, etc). Massively scalable index. Log Structure Merge Tree. Linked list of extents
Architecture layer: Front-end layer. REST front end. Authentication/authorization. Metrics/logging.
Key Design Points
Availability with consistency for writing. All writes we do are to a log. Append to the last extent of the log.
Ordered the same across all 3 replicas. Success only if 3 replicas are commited. Extents get sealed (no more appends) when they get to a certain size.
If you lose a node, seal the old two copies, create 3 new instances to append to. Also make a 3rd copy for the old one.
Availability with consistency for reading. Can read from any replica. Send out parallel read requests if first read is taking higher than 95% latency.
Partition Layer: spread index/transaction processing across servers. If there is a hot node, split that part of the index off. Dynamically load balance. Just the index, this does not move the data.
DFS Layer: load balancing there as well. No disk or node should be hot. Applies to both reads and writes. Lazily move replicas around to load balancing.
Append only system. Benefits: simple replication, easier diagnostics, erasure coding, keep snapshots with no extra cost, works well with future dirve technology. Tradeoff: GC overhead.
Our approach to the CAP theorem. Tradeoff in Availability vs. Consistency. Extra flexibility to achieve C and A at the same time.
Lessons learned: Automatic load balancing. Adapt to conditions. Tunable and extensible to tune load balancing rules. Tune based on any dimension (CPU, network, memory, tpc, GC load, etc.)
Lessons learned: Achieve consistently low append latencies. Ended up using SSD journaling.
Lessons learned: Efficient upgrade support. We update frequently, almost consistently. Handle them almost as failures.
Lessons learned: Pressure point testing. Make sure we’re resilient despite errors.
Erasure coding. Implemented at the DFS Layer. See last year’s SDC presentation.
Azure VM persistent disks: VHDs for persistent disks are directly stored in Windows Azure Storage blobs. You can access your VHDs via REST.
Easy to upload/download your own VHD and mount them. REST writes are blocked when mounted to a VM. Snapshots and Geo replication as well.
Separating compute from storage. Allows them to be scaled separately. Provide flat network storage. Using a Quantum 10 network architecture.
Summary: Durability (3 copies), Consistency (commit across 3 copies). Availability (can read from any of the 3 relicas). Performance/Scale.
Windows Azure developer website: http://www.windowsazure.com/en-us/develop/net
Windows Azure storage blog: http://blogs.msdn.com/b/windowsazurestorage
SOSP paper/talk: http://blogs.msdn.com/b/windowsazure/archive/2011/11/21/windows-azure-storage-a-highly-available-cloud-storage-service-with-strong-consistency.aspx

SMB Direct update
Greg Kramer, Microsoft
Tom Talpey, Microsoft

Two parts: 1 - Tom shares Ecosystem status and updates, 2 - Greg shares SMB Direct details
Protocols and updates: SMB 3.02 is a minor update. Documented in MS-SMB2 and MS-SMBD. See Dave's talk yesterday.
SMB Direct specifies the SMB3 RDMA transport, works with both SMB 3.0 and SMB 3.02
Windows Server 2012 R2 – GA in October, download from MSDN
Applications using SMB3 and SMB Direct: Hyper-V VHD, SQL Server
New in R2: Hyper-V Live Migration over SMB, Shared VHDX (remote shared virtual disk, MS-RSVD protocol)
RDMA Transports: iWARP (IETF RDMA over TCP), InfiniBand, RoCE. Ethernet: iWARP and RoCE – 10 or 40GbE, InfiniBand: 32Gbps (QDR) or 54Gbps (FDR)
RDMA evolution: iWARP (IETF standard, extensions currently active in IETF). RoCE (routable RoCE to improve scale, DCB deployment still problematic). InfiniBand (Roadmap to 100Gbps, keeping up as the bandwidth/latency leader).
iWARP: Ethernet, routable, no special fabric required, Up to 40GbE with good latency and full throughput
RoCE: Ethernet, not routable, requires PFC/DCB, Up to 40GbE with good latency and full throughput
InfinBand: Specialized interconnect, not routable, dedicated fabric and switching, Up to 56Gbps with excellent latency and throughput
SMB3 Services: Connection management, authentication, multichannel, networking resilience/recovery, RDMA, File IO Semantics, control and extension semantics, remote file system access, RPC
The ISO 7-layer model: SMB presents new value as a Session layer (RDMA, multichannel, replay/recover). Move the value of SMB up the stack.
SMB3 as a session layer: Applications can get network transparency, performance, recovery, protection (signing, encryption, AD integration). Not something you see with other file systems or file protocols.
Other: Great use by clustering (inter-node communication), quality of service, cloud deployment
In summary. Look to SMB for even broader application (like Hyper-V Live Migration did). Broader use of SMB Direct. Look to see greater application “fidelity” (sophisticated applications transparently server by SMB3)
Protocol enhancements and performance results
Where can we reduce IO costs? We were extremely happy about performance, there was nothing extremely easy to do next, no low-hanging fruit.
Diagram showing the App/SMB client/Client RNIC/Server RNIC. How requests flow in SMB Direct.
Interesting: client has to wait for the invalidation completion. Invalidation popped up as an area of improvement. Consumes cycles, bus. Adds IO, latency. But it’s required.
Why pend IO until invalidation is completed? This is storage, we need to be strictly correct. Invalidation guarantees: data is in a consistent state after DMA, peers no longer has access.
Registration caches cannot provides these guarantees, leading to danger of corruption.
Back to the diagram. There is a way to decorate a request with the invalidation  Send and Invalidate. Provides all the guarantees that we need!
Reduces RNIC work requests per IO by one third for high IOPs workload. That’s huge! Already supported by iWARP/RoCE/InfiniBand
No changes required at the SMB Direct protocol. Minor protocol change in SMB 3.02 to support invalidation. New channel value in the SMB READ and SMB WRITE.
Using Send and Invalidate (Server). Only one invalidate per request, have to be associated with the request in question. You can leverage SMB compounding.
Only the first memory descriptor in the SMB3 read/write array may be remotely invalidated. Keeping it simple.
Using Send and Invalidate (Client). Not a mandate, you can still invalidate “manually” if not using remote invalidate. Must validate that the response matches.
Performance Results (drumroll…)
Benchmark configuration: Client and Server config: Xeon E5-2660. 2 x ConnectX-3 56Gbps InfiniBand. Shunt filter in the IO path. Comparing WS2012 vs. WS2012 R2 on same hardware.
1KB random IO. Uses RDMA send/receive path. Unbuffered, 64 queue depth.

Reads: 881K IOPs. 2012 R2 is +12.5% over 2012. Both client and server CPU/IO reduced (-17.3%, -36.7%)
Writes: 808K IOPs. 2012 R2 is +13.5% over 2012. Both client and server CPU/IO reduced (-16%, -32.7%)

8KB random IO. Uses RDMA read/writes. Unbuffered, 64 queue depth.
- Reads: 835K IOPs. 2012 R2 is +43.3% over 2012. Both client and server CPU/IO reduced (-37.1%, -33.2%)
- Writes: 712K IOPs. 2012 R2 is +30.2% over 2012. Both client and server CPU/IO reduced (-26%, -14.9%)
512KB sequential IO. Unbuffered, 12 queue depth. Already maxing out before. Remains awesome. Minor CPU utilization decrease.

Reads: 11,366 MBytes/sec. 2012 R2 is +6.2% over 2012. Both client and server CPU/IO reduced (-9.3%, -14.3%)
Writes: 11,412 MBytes/sec: 2012 R2 is +6% over 2012. Both client and server CPU/IO reduced (-12.2%, -10.3%)

Recap: Increased IOPS (up to 43%) and high bandwidth. Decrease CPU per IO (up to 36%).
Client has more CPU for applications. Server scales to more clients.
This includes other optimization in both the client in the server. NUMA is very important.
No new hardware required. No increase number of connections, MRs, etc.
Results reflect the untuned, out-of-the-box customer experience.
One more thing… You might be skeptical, especially about the use of shunt filter.
We never get to see this in our dev environment, we don’t have the high end gear. But...
Describing the 3U Violin memory array running Windows Server in a clustered configuration. All flash storage. Let’s see what happens…
Performance on real IO going to real, continuously available storage:

100% Reads – 4KiB: >1Million IOPS
100% Reads – 8KiB: >500K IOPS
100% Writes – 4KiB: >600K IOPS
100% Writes – 8KiB: >300K IOPS

Questions?

A Status Report on SMB Direct (RDMA) for Samba
Richard Sharpe, Samba Team Member, Panzura

I work at Panzura but has been done on my weekends
Looking at options to implement SMB Direct
2011 – Microsoft introduced SMB direct at SDC 2011. I played around with RDMA
May 2012 – Tutorial on SMB 3.0 at Samba XP
Mellanox supplied some IB cards to Samba team members
May 2013 – More Microsoft presentations with Microsoft at Samba XP
June 2013 – Conversations with Mellanox to discuss options
August 2013 – Started circulating a design document
Another month or two before it’s hooked up with Samba.
Relevant protocol details: Client connections via TCP first (port 445). Session setup, connects to a share. Queries network interfaces. Place an RDMA Connection to server on port 5445, brings up SMB Direct Protocol engine
Client sends negotiate request, Dialect 0x300, capabilities field. Server Responds.
Diagram with SMB2 spec section 4.8 has an example
SMB Direct: Small protocol - Negotiate exchange phase, PDU transfer phase.
Structure of Samba. Why did it take us two years? Samba uses a fork model. Master smbd forks a child. Easy with TCP. Master does not handle SMB PDUs.
Separate process per connection. No easy way to transfer connection between them.
Diagram with Samba structure. Problem: who should listen on port 5445? Wanted RDMA connection to go to the child process.
3 options:
1 - Convert Samba to a threaded model, everything in one address space. Would simplify TCP as well. A lot of work… Presents other problems.
2 - Separate process to handle RDMA. Master dmbd, RDMA handler, multiple child smbd, shared memory. Layering violation! Context switches per send/receive/read/write. Big perf hit.
3 - Kernel driver to handle RDMA. Smbdirect support / RDMA support (rdmacm, etc) / device drivers. Use IOCTLs. All RDMA work in kernel, including RDMA negotiate on port 5445. Still a layering violation. Will require both kernel and Samba knowledge.
I decided I will follow this kernel option.
Character mode device. Should be agnostic of the card used. Communicate via IOCTLs (setup, memory params, send/receive, read/write).
Mmap for RDMA READ and RDMA WRITE. Can copy memory for other requests. Event/callback driven. Memory registration.
The fact that is looks like a device driver is a convenience.
IOCTLs: set parameters, set session ID, get mem params, get event (includes receive, send complete), send pdu, rdma read and write, disconnect.
Considering option 2. Doing the implementation of option 3 will give us experience and might change later.
Amortizing the mode switch. Get, send, etc, multiple buffers per IOCTL. Passing an array of objects at a time.
Samba changes needed….
Goals at this stage: Get something working. Allow others to complete. It will be up on github. Longer term: improve performance with help from others.
Some of this work could be used by the SMB client
Status: A start has been made. Driver loads and unloads, listens to connections. Working through the details of registering memory. Understand the Samba changes needed.
Weekend project! http://github.com/RichardSharpe/smbdirect-driver
Acknowledgments: Microsoft. Tom Talpey. Mellanox. Or Gerlitz. Samba team members.

CDMI and Scale Out File System for Hadoop
Philippe Nicolas, Scality

Short summary of who is Scality. Founded 2009. HQ in SF. ~60 employees, ~25 engineers in Paris. 24x7 support team. 3 US patents. $35Min 3 rounds.
Scality RING. Topology and name of the product. Currently in the 4.2 release. Commodity servers and storage. Support 4 LINUX distributions. Configure Scality layer. Create a large pool of storage.
Ring Topology. End-to-end Paralelism. Object Storage. NewSQL DB. Replication. Erasure coding. Geo Redundancy. Tiering. Multiple access methods (HTTP/REST, CDMI, NFS, CIFS, SOFS). GUI/CLI management.
Usage: e-mail, file storage, StaaS, Digital Media, Big Data, HPC
Access methods: APIs: RS2 (S3 compatible API), Sproxyd, RS2 light, SNIA CDMI. File interface: Scality Scale Out File System (SOFS), NFS, CIFS, AFP, FTP. Hadoop HDFS. OpenStack Cinder (since April 2013).
Parallel network file system. Limits are huge – 2^32 volumes (FS), 2^24 namespaces, 2^64 files. Sparse files. Aggregated throughput, auto-scaling with storage or access node addition.
CDMI (path and ID based access). Versions 1.0.1., 1.0.2. On github. CDMI client java library (CaDMIum), set of open source filesystem tools. On github.
Apache Hadoop. Transform commodity hard in data storage service. Largely supported by industry and end user community. Industry adoption: big names adopting Hadoop.
Scality RING for Hadoop. Replace HDFS with the Scality FS. We validate Hortonworks and Cloudera. Example with 12 Hadoop nodes for 12 storage nodes. Hadoop task trackers on RING storage nodes.
Data compute and storage platform in ONE cluster. Scality Scale Out File System (SOFS) instead of HDFS. Advanced data protection (data replication up to 6 copies, erasure coding). Integration with Hortonworks HDP 1.0 & Cloudera CDH3/CDH4. Not another Hadoop distribution!
Summary: This is Open Cloud Access: access local or remotely via file and block interface. Full CDMI server and client. Hadoop integration (convergence approach). Comprehensive data storage platform.

Introduction to HP Moonshot
Tracy Shintaku, HP

Today’s demands – pervasive computing estimates. Growing internet of things (IoT).
Using SoC technologies used in other scenarios for the datacenter.
HP Moonshot System. 4U. World’s first low-energy software-defined server. HP Moonshot 1500 Chassis.
45 individually serviceable hot-plug artrdges. 2 network switches, private fabric. Passive base plane.
Introducing HP Proliant Moonshot Server (passing around the room). 2000 of these servers in a rack. Intel Atom S1260 2GHz, 8GB DDR ECC 1333MHz. 500GB or 1TBHDD or SSD.
Single server = 45 servers per chassis. Quad-server = 180 servers per chassis. Compute, storage or combination. Storage cartridges with 2 HDD shared by 8 servers.
Rear view of the chassis. Dual 4QSFP network uplinks each with 4 x 40GB), 5 hot-plug fans, Power supplies, management module.
Ethernet – traffic isolation and stacking for resiliency with dual low-latency switches. 45 servers  dual switches  dual uplink modules.
Storage fabric. Different module form factors allow for different options: Local storage. Low cost boot and logging. Distributed storage and RAID. Drive slices reduce cost of a boot drive 87%.
Inter-cartridge private 2D Taurus Ring – available in future cartridges. High speed communication lanes between servers. Ring fabric, where efficient localized traffic is benefitial.
Cartridge roadmap. Today to near future to future. CPU: Atom  Atom, GPU, DSP, x86, ARM. Increasing variety of workloads static web servers now to hosting, financial servers in the future.
Enablement: customer and partner programs. Partner program. Logo wall for technology partners. Solution building program. Lab, services, consulting, financing.
Partners include Redhat, Suse, Ubuntu, Hortonworks, MapR, Cloudera, Couchbase, Citrix, Intel, AMD, Calxeda, Applied Micro, TI, Marvell, others. There’s a lot of commonality with OpenStack.
Web site: http://h17007.www1.hp.com/us/en/enterprise/servers/products/moonshot/index.aspx

NFS on Steroids: Building Worldwide Distributed File System
Gregory Touretsky, Intel

Intel is a big organization, 6500 Its @ 59 sites, 95,200 employes, 142000 devices.
Every employee doing design has a Windows machines, but also interact with NFS backend
Remote desktop in interactive pool. Talks to NFS file servers, glued together with name spaces.
Large batch pools that do testing. Models stored in NFS.
Various application servers, running various systems. Also NIS, Cron servers, event monitors, configuration management.
Uses Samba to provide access to NFS file servers using SMB.
Many sites, many projects. Diagram with map of the world and multiple projects spanning geographies.
Latency between 10s t 100s ms. Bandwidth: 10s Mbps to 10s of Gbpsp.
Challenge: how to get to our customers and provide the ability to collaborate across the globe in a secure way
Cross-site data access in 2012. Multiple file servers, each with multiple exports. Clients access servers on same site. Rsync++ for replication between sites (time consuming).
Global user and group accounts. Users belong to different groups in different sites.
Goals: Access every file in the world from anywhere, same path, fault tolerant, WAN friendly, every user account and group on every site, Local IO performance not compromised.
Options: OpenAFS (moved out many years ago, decided not to got back). Cloud storage (concern with performance). NFS client-side caching (does not work well, many issues). WAN optimization (have some in production, help with some protocols, but not suitable for NFS). NFS site-level caching (proprietary and open source NFS Ganesha). In house (decided not to go there).
Direct NFS mount over WAN optimized tunnel. NFS ops terminated at the remote site, multiple potential routes, cache miss. Not the right solution for us.
Select: Site-level NFS caching and Kerberos. Each site has NFS servers and Cache servers. Provides instant visibility and minimizes amount of data transfer across sites.
Cache is also writable. Solutions with write-through and write-back caching.
Kerberos authentication with NFS v3. There are some problems there.
Cache implementations: half a dozen vendors, many not suitable for WAN. Evaluating alternatives.
Many are unable to provide a disconnected mode of operation. That eliminated a number of vendors.
Consistency vs. performance. Attribute cache timeout. Nobody integrates this at the directory level. Max writeback delay.
Optimizations. Read cache vs. Read/Write cache, delegations, proactive attribute validation for hot files, cache pre-population.
Where is it problematic? Application is very NFS unfriendly, does not work well with caching. Some cases it cannot do over cache, must use replication.
Problems: Read once over high latency link. First read, large file, interactive work. Large % of non-cacheable ops (write-through). Seldom access, beyond cache timeout.
Caching is not a business continuity solution. Only a partial copy of the data.
Cache management implementation. Doing it at scale is hard. Nobody provides a solution that fits our needs.
Goal: self-service management for data caching. Today administrators are involved in the process.
Use cases: cache my disk at site X, modify cache parameters, remove cache, migrate source/cache, get cache statistics, shared capacity management, etc.
Abstract the differences between the different vendors with this management system.
Management system example: report with project, path (mount point), size, usage, cached cells. Create cell in specific site for specific site.
Cache capacity planning. Goal: every file to be accessible on-demand everywhere.
Track cache usage by org/project. Shared cache capacity, multi-tenant. Initial rule of thumb: 7-10% of the source capacity, seeding capacity at key locations
Usage models: Remote validation (write once, read many). Get results back from remote sites (write once, read once). Drop box (generate in one site, get anywhere). Single home directory (avoid home directory in every site for every user, cache remote home directories). Quick remote environment setup, data access from branch location.
NFS (RPC) Authentication. Comparing AUTH_SYS and RPCSEC_GSS (KRB5). Second one uses an external KDC, gets past the AUTH_SYS limitation of 16 group IDs.
Bringing Kerberos? Needs to make sure this works as well as Windows with Active Directory. Need to touch everything (Linux client, NFS file servers, SSH, batch scheduler, remote desktop/interactive servers, name space/automounter (trusted hosts vs. regular hosts), Samba (used as an SMB gateway to NFS), setuid/sudo, cron jobs and service accounts (keytab management system),
Supporting the transition from legacy mounts to Kerberos mount. Must support a mixed environment. Introducing second NIS domain.
Welcome on board GDA airlines (actual connections between different sites). Good initial feedback from users (works like magic!)
Summary: NFS can be accessed over WAN – using NFS caching proxy. NFSv3 environment can be kerberized (major effort is required, transition is challenging, it would be as challenging for NFSv5/KRB)

Forget IOPS: A Proper Way to Characterize & Test Storage Performance
Peter Murray, SwiftTest

About what we learned in the last few years
Evolution: Vendor IOPs claims, test in production and pray, validate with freeware tools (iometer, IOZone), validate with workload models
What is storage validation? Characterize the various applications, workloads. Diagram: validation appliance, workload emulations, storage under test.
Why should you care? Because customers do care! Product evaluations, vendor bakeoffs, new feature and technology evaluations, etc…
IOPS: definition from the SNIA dictionary. Not really well defined. One size does not fit all. Looking at different sizes.
Real IO does not use a fixed size. Read/write may be a small portion of it in certain workloads. RDMA read/write may erode the usefulness of isolated read/write.
Metadata: data about data. Often in excess of 50%, sometimes more than 90%. GoDaddy mentioned that 94% of workloads are not read/write.
Reducing metadata impact: caching with ram, flash, SSD helps but it’s expensive.
Workloads: IOPS, metadata and your access pattern. Write/read, random/sequential, IO/metadata, block/chunk size, etc.
The important of workloads: Understand overload and failure conditions. Understand server, cluster, deduplication, compression, network configuration and conditions
Creating and understanding workloads. Access patterns (I/O mix: read/write%, metadata%) , file system (depth, files/folder, file size distribution), IO parameters (block size, chunk size, direction), load properties (number of users, actions/second, load variability/time)
Step 1 - Creating a production model. It’s an art, working to make it a science. Production stats + packet captures + pre-built test suites = accurate, realistic work model.
Looking at various workload analysis
Workload re-creation challenges: difficult. Many working on these. Big data, VDI, general VM, infinite permutations.
Complex workloads emulation is difficult and time consuming. You need smart people, you need to spend the time.
Go Daddy shared some of the work on simulation of their workload. Looking at diagram with characteristics of a workload.
Looking at table with NFSv3/SMB2 vs. file action distribution.
Step 2: Run workload model against the target.
Step 3: Analyze the results for better decisions. Analytics leads to Insight. Blocks vs. file. Boot storm handing, limits testing, failure modes, effects of flash/dedup/tiering/scale-out.
I think we’ll see dramatic changes with the use of Flash. Things are going to change in the next few years.
Results analysis: Performance. You want to understand performance, spikes during the day, what causes them. Response times, throughput.
Results analysis: Command mix. Verify that the execution reflects the expected mix. Attempts, successes, errors, aborts.
Summary: IOPs alone cannot characterize real app storage performance. Inclusion of metadata is essential, workload modeling and purpose-build load generation appliances are the way to emulate applications. The more complete the emulation, the deeper the understanding.
If we can reduce storage cost from 40% to 20% of the solution by better understanding the workload, you can save a lot of money.

pNFS, NFSv4.1, FedFS and Future NFS Developments
Tom Haynes, NetApp

Tom covering for Alex McDonald, who is sick. His slides.
We want to talk about how the protocol get defined, how it interfact with different application vendors and customers.
Looking at what is happening on the Linux client these days.
NFS: Ubiquitous and everywhere. NFSv3 is very successful, we can’t dislodge it. We though everyone would go for NFSv4 and it’s now 10 years later…
NFSv2 in 1983, NFSv3 in 1995, NFSv4 in 2003, NFSv4.1 in 2010. NFSv4.2 to be agreed at the IETF – still kinks in the protocol that need to be ironed out. 2000=DAS, 2010=NAS, 2020=Scale-Out
Evolving requirements. Adoption is slow. Lack of clients was a problem with NFSv4. NFSv3 was just “good enough”. (It actually is more than good enough!)
Industry is changing, as are requirements. Economic trends (cheap and fast cluster, cheap and fast network, etc…)
Performance: NFSv3 single threaded bottlenecks in applications (you can work around it).
Business requirements. Reliability (sessions) is a big requirement
NFSv4 and beyond.
Areas for NFSv4, NFSv4.1 and pNFS: Security, uniform namespaces, statefulness/sessions, compound operations, caching (directory and file delegations) parallelization (layout and pFNS)
Future NFSv4.2 and FedFS (Global namespace; IESG has approved Dec 2012)
NFSv4.1 failed to talk to the applications and customers and ask what they needed. We did that for NFSv4.2
Selecting the application for NFSv4.1, planning, server and client availability. High level overview
Selecting the parts: 1 – NFSv4.1 compliant server (Files, blocks or objects?), 2-compliant client. The rise of the embedded client (Oracle, VMware). 3 – Auxiliary tools (Kerberos, DNS, NTP, LDAP). 4 – If you can, use NFS v4.1 over NFSv4.
If you’re implementing something today, skip NFS v4 and go straight to NFS v4.1
First task: select an application: Home directories, HPC applications.
Don’t select: Oracle (use dNFS built in), VMware and other virtualization tools (NFSv3). Oddball apps that expect to be able to internally manage NFSv3 “maps”. Any application that required UDP, since v4.1 doesn’t support anything but TCP.
NSFv4 stateful clients. Gives client independence (client has state). Allows delegation and caching. No automounter required, simplified locking
Why? Compute nodes work best with local data, NFSv4 eliminates the need for local storage, exposes more of the backed storage functionality (hints), removes stale locks (major source of NFSv3 irritation)
NFSv4.1 Delegations. Server delegates certain responsabilities to the client (directory and file, caching). Read and write delegation. Allows clients to locally service operations (open, close, lock, etc.)
NFSv4.1 Sessions. In v3, server never knows if client got the reply message. In v4.1, sessions introduced.
Sessions: Major protocol infrastructure change. Exactly once semantics (EOS), bounded size of reply cache. Unlimited parallelism. Maintains server’s state relative to the connections belonging to a client.
Use delegation and caching transparently; client and server provide transparency. Session lock clean up automatically.
NFSv4 Compound operations – NFSv3 protocol can be “chatty”, unsuitable for WANs with poor latency. Typical NFSv3: open, read & close a file. Compounds many operations into one to reduce wire time and simple error recovery.
GETATTR is the bad boy. We spent 10 years with the Linux client to get rid of many of the GETATTR (26% of SPECsfs2008).
NFSv4 Namespace. Uniform and “infinite” namespace. Moving from user/home directories to datacenter and corporate use. Meets demand for “large scale” protocol. Unicode support for UTF-8 codepoints. No automounter required (simplifies administration). Pseudo-file system constructed by the server.
Looking at NFSv4 Namespace example. Consider the flexibility of pseudo-filesystems to permit easier migration.
NFSv4 I18N Directory and File Names. Uses UTF-8, check filenames for compatibility, review filenames for compatibility. Review existing NFSv3 names to ensure they are 7-bit ASCII clean.
NFSv4 Security. Strong security framework. ACLs for security and Windows compatibility. Security with Kerberos. NFSv4 can be implemented without Kerberos security, but not advisable.
Implementing without Kerberos (no security is a last resort!). NFSv4 represents users/groups as strings (NFSv3 used 32-bit integers, UID/GID). Requires UID/GID to be converted to all numeric strings.
Implementing with Kerberos. Find a security expert. Consider using Windows AD Server.
NFSv4 Security. Firewalls. NFSv4 ha no auxiliary protocols. Uses port 2049 with TCP only. Just open that port.
NFSv4 Layouts. Files, objects and block layouts. Flexibility for storage that underpins it. Location transparent. Layouts available from various vendors.
pNFS. Can aggregate bandwidth. Modern approach, relieves issues associated with point-to-point connections.
pNFS Filesystem implications.
pNFS terminology. Important callback mechanism to provide information about the resource.
pNFS: Commercial server implementations. NetApp has it. Panasas is the room as well. Can’t talk about other vendors…
Going very fast through a number of slides on pNFS: NFS client mount, client to MDS, MDS Layout to NFS client, pNFS client to DEVICEINFO from MDS,
In summary: Go adopt NFS 4.1, it’s the greatest thing since sliced bread, skip NFS 4.0
List of papers and references. RFCs: 1813 (NFSv3), 3530 (NFSv4), 5661 (NFSv4.1), 5663 (NFSv4.1 block layout), 5664 (NFSv4.1 object layout)

pNFS Directions / NFSv4 Agility
Adam Emerson, CohortFS, LLC

Stimulate discussion about agility as a guiding vision for future protocol evaluation
NFSv4: A standard file acces/storage protocol, that is agile
Incremental advances shouldn’t require a new access protocol. Capture more value from the engineering already done. Retain broad applicability, yet adapt quickly to new challenges/opportunities
NFSv4 has delivered (over 10+ years of effort) on a set of features designers had long aspired to: atomicity, consistency, integration, referrals, single namespaces
NFSv4 has sometimes been faulted for delivering slowly and imperfect on some key promises: flexible and easy wire security , capable and interoperable ACLs, RDMA acceleration
NFSv4 has a set of Interesting optional features not widely implemented: named attributes, write delegations, directory delegations, security state verifier, retention policy
Related discussion in the NFSv4 Community (IETF): The minor version/extension debate: de-serializing independent, potentially parallel extension efforts, fixing defect in prior protocol revisions, rationalizing past and future extension mechanisms
Related discussion in the NFSv4 Community (IETF): Extensions drafts leave my options open, but prescribes: process to support development of new features proposals in parallel, capability negotiation, experimentation
Embracing agility: Noveck formulation is subtle: rooted in NFS and WG, future depends on participants, can encompass but perhaps does not call out for an agile future.
Capability negotiation and experimental codepoint ranges strongly support agility. What we really want is a model that encourages movement of features from private experimentation to shared experimentation to standardization.
Efforts promoting agility: user-mode (and open source) NFSv4 servers (Ganesha, others?) and clients (CITI Windows NFSv4.1 client, library client implementations)
Some of the people in the original CITI team now working with us and are continuing to work on it
library client implementations: Allow novel semantics and features like pre-seeding of files, HPC workloads, etc.
NFSv4 Protocol Concepts promoting agility: Not just new RPCs and union types.
Compound: Grouping operations with context operations. Context evolves with operations and inflects the operations. It could be pushed further…
Named Attributes: Support elaboration of conventions and even features above the protocol, with minimal effort and coordination. Subfiles, proplists. Namespace issues: System/user/other, non-atomicity, not inlined.
Layout: Powerful structuring concept carrying simplified transaction pattern. Typed, Operations carry opaque data nearly everywhere, application to data striping compelling.
Futures/experimental work – some of them are ridicuolous and I apologize in advance
pNFS striping flexibility/flexible files (Halevy). Per-file striping and specific parity applications to file layout. OSDv2 layout, presented at IETF 87.
pNFS metastripe (Eisler, further WG drafts). Scale-out metadata and parallel operations for NFSv4. Generalizing parallel access concept of NFSv4 for metadata. Built on layout and attribute hints. CohortFS prototyping metastripe on a parallel version of the Ceph file system. NFSv4 missing a per-file redirect, so this has file redirection hints.
End-to-end Data Integrity (Lever/IBM). Add end-to-end data integrity primitives (NFSv4.2). Build on new READ_PLUS and WRITE ops. Potentially high value for many applications.
pNFS Placement Layouts (CohortFS). Design for algorithmic placement in pNFS layout extension. OSD selection and placement computed by a function returned at GETDEVICEINFO. Client execution of placement codes, complex parity, volumes, etc.
Replication Layouts (CohortFS). Client based replication with integrity. Synchronous wide-area replication. Built on Layout.
Client Encryption (CohortFS). Relying on named attribute extension only, could use atomicity. Hopefully combined with end-to-end integrity being worked on
Cache consistency. POSIX/non-CTO recently proposed (eg, Eshel/IBM). Potentially, more generality. Eg, flexible client cache consistency models in NFSv4. Add value to existing client caching like CacheFS.
New participants. You? The future is in the participants…

Closing Keynote: Worlds Colliding: Why Big Data Changes How to Think about Enterprise Storage
Addison Snell, CEO, Intersect360 Research

Analyst, researching big data. There are hazard of forecasting big data. Big Data goes beyond Hadoop.
What is the real opportunity? Hype vs. Reality. Perception: HPC is not something I want in my Enterprise computing environment
Technical computing vs. Enterprise computing. Both defined as mission-critical.
Vendors do not know how much money is going on each, we need to do a lot of user research.
Technical computing is about the top-line mission of the business. Enterprise computing is about keeping the business running.
TC is driven by price/performance, faster adoption of new tech. EC is driven by RAS (reliability, availability, serviceability), slow adoption of new tech.
Survey results, 278 respondents, April to August 2013. Built on earlier surveys. 178 technical, 100 enterprise, 165 commercial, 67 academic, 46 government
Insight 1: big data is a big opportunity. Money is being spent. But people that do not do big data are not likely to answer the survey…
Some to spend 25% of total IT budget in 2013. Use caution when describing the big data market. Not sure about what’s being counted.
What are big data applications? Vendors talk “Hadoop” and “graph”. Users talk about “analyze” and “algorithm”. ISV software for big data is thinly scattered.
Insight #2: it’s not just Hadoop: Out of 574 apps mentioned in the survey, about half is internally developed and this grew from last year. Only 75 mentioned Hadoop in 2012 and it’s going down.
It’s like HPC, top 20 applications cover only 40% of the market. It’s like measuring the tail of a snake…
Defining the opportunity looking at IT budget growth. Expect two year change in IT budget. Look at “satisfaction gaps”.
Insight #3: Performance counts. Top three satisfaction gaps are IO performance, storage capacity and RAS. For both TC and EC.
HPC-like mentality is starting to creep into the enterprise. Big data will be a driver for expanded usage of HPC, IF they can still meet enterprise requirements.
Faulty logic: I like sushi, I like ice cream, therefore I like sushi-flavored ice cream… big data and public cloud not necessarily go together.
Maybe private cloud is better. Lead with private cloud and burst some of it to the public cloud.
Technologies in the discussion. Storage beyond spinning disks (flash for max IO, tape for capacity), parallel file systems, high speed fabrics (InfiniBand), MPI, large shared memory spaces, accelerators.

Tunneling SCSI over SMB: Shared VHDX files for Guest Clustering in Windows Server 2012 R2
Jose Barreto, Microsoft
Matt Kurjanowicz, Microsoft

Could not take notes while presenting :-)
It was really good, though…

An SMB3 Engineer’s View of Windows Server 2012 Hyper-V Workloads
Gerald Carter, EMC

FAS paper on Virtual Machine workloads
Does the current workload characterization done for NFS work for SMB3?
What do the IO and jump distance patterns look like?
What SMB3 protocol features does Hyper-V use?
Original scope abstract was reduced due to time constraint and hardware availability (no SMB Direct, no SMB Multichannel, no failover)
Building of tools that look into this using just pcap files.
Hardware: couple of quad-core Intel Q9650 @ 3GHz, 16GB RAM, SATA II disks, networking is 1Gbps, Windows Server 2012 (not R2)
OS on the VMs: Ubuntu 12.04 x64, Windows XP SP3 x86, Windows Server 2012 Standard
Deployment to SMB file share. Sharing PowerShell scripts.
Scenarios. 1: Boot host idle for 10 minutes. Scenario 2: Build Linux Kernel. 3: Random file operations (different size file copies)
Cold boot, start packet capture, cold boot hyper-v host, launch vm, cleanly shutdown scenario, stop packet capture.
Wanted to understand distribution. Read/write mix, how much of each command was used. Distribution of read size/write size/jumps.
Sharktools: Python extension library for data analysis. Example code: import pyshark.
Dictionary composition: request/response by exchanges by file handles.
Scenario 1 - Single client boot
Boot, leave it running for 10 minutes, then shutdown.
Huge table with lots of results. You expected lots of reads and writes. Large number of query info – hypervisor queries configuration files over and over. Lots of reads (second place), then writes.
Another huge table with command occurrences. More IOs in Windows Server 2012 than on Windows XP and Linux.
VHDX handle (one slide for Linux, Windows XP, Windows Server 2012). One handle has most of the operations.
Did not look into the create options.
IO Size distribution (one slide for Linux, Windows XP, Windows Server 2012). Theory: read and write sizes relate to the latency of the network. Most frequent; 4KB, 32KB, 128KB.
Jump distance. Tables wit Time, Jump distance, offset, length. Scatter plot for the three operating systems.
Trying to measure the degree of randomness over time. Linux is more sequential, maybe due to the nature of the file system. Not authoritative, just an observation.
Comment from audience: different styles of booting Linux, some more sequential read, some more parallel.
Boot – single host summary. All read/writes are multiple of 512. 4KB, 32KB, 128KB are favorites. Size and jump distance distribution changes with guest OS (file system).
Multiple persistent handles (4 or more) opened per VM instance.
Scenario 2 – Linux kernel compile
Command distribution. Consistent – 24% query info, 33% read, 33% write.
Command occurrence.
IO size: Lots of 4KB and 128KB reads. Big spikes in writes (lots of 128KB).
Jump distances: Read and write charts. Random IO, no real surprise.
Scenario 3 – Random file operations
Guest using SCSI virtual disk.
Command distribution. 62% read, 32% write, IOCTL: 3.1% (request for resilient handles, trim), queryinfo 1.3%.
Random tree copy. IO Size: Lots of 4KB, some 1MB writes.
Jump – random tree copy. Nice and sequential. Looks like a spaceship, or the Eiffel tower sideways. Two threads contending for the same file? Metadata vs. data?
Closing thoughts. Heavy query info. Not as many large size IOs as expected. Multiple long lived file handles per guest.
Future work: Include SMB Direct, SMB Multichannel, Failover. Include more workloads. More generalized SMB3 workloads.

That's it for this year. I'm posting this last update from the San Jose airport, on my way back to Redmond...

↧

Windows Server 2012 R2: Which version of the SMB protocol (SMB 1.0, SMB 2.0, SMB 2.1, SMB 3.0 or SMB 3.02) are you using?

October 2, 2013, 3:50 pm

≫ Next: Networking configurations for Hyper-V over SMB in Windows Server 2012 and Windows Server 2012 R2

≪ Previous: Raw notes from the Storage Developers Conference (SDC 2013)

Note: This blog post is a Windows Server 2012 R2 update on a previous version focused on Windows Server 2012.

1. Introduction

With the release of Windows 8.1 and Windows Server 2012 R2, I am frequently asked about how older versions of Windows will behave when connecting to or from these new versions. Upgrading to a new version of SMB is something that happened a few times over the years and we established a process in the protocol itself by which clients and servers negotiate the highest version that both support.

2. Versions

There are several different versions of SMB used by Windows operating systems:

CIFS – The ancient version of SMB that was part of Microsoft Windows NT 4.0 in 1996. SMB1 supersedes this version.
SMB 1.0 (or SMB1) – The version used in Windows 2000, Windows XP, Windows Server 2003 and Windows Server 2003 R2
SMB 2.0 (or SMB2) – The version used in Windows Vista (SP1 or later) and Windows Server 2008
SMB 2.1 (or SMB2.1) – The version used in Windows 7 and Windows Server 2008 R2
SMB 3.0 (or SMB3) – The version used in Windows 8 and Windows Server 2012
SMB 3.02 (or SMB3) – The version used in Windows 8.1 and Windows Server 2012 R2

Windows NT is no longer supported, so CIFS is definitely out. Windows Server 2003 R2 with a current service pack is under Extended Support, so SMB1 is still around for a little while. SMB 2.x in Windows Server 2008 and Windows Server 2008 R2 are under Mainstream Support until 2015. You can find the most current information on the support lifecycle page for Windows Server. The information is subject to the Microsoft Policy Disclaimer and Change Notice. You can use the support pages to also find support policy information for Windows XP, Windows Vista, Windows 7 and Windows 8.

In Windows 8.1 and Windows Server 2012 R2, we introduced the option to completely disable CIFS/SMB1 support, including the actual removal of the related binaries. While this is not the default configuration, we recommend disabling this older version of the protocol in scenarios where it’s not useful, like Hyper-V over SMB. You can find details about this new option in item 7 of this blog post: What’s new in SMB PowerShell in Windows Server 2012 R2.

3. Negotiated Versions

Here’s a table to help you understand what version you will end up using, depending on what Windows version is running as the SMB client and what version of Windows is running as the SMB server:

OS	Windows 8.1 WS 2012 R2	Windows 8 WS 2012	Windows 7 WS 2008 R2	Windows Vista WS 2008	Previous versions
Windows 8.1 WS 2012 R2	SMB 3.02	SMB 3.0	SMB 2.1	SMB 2.0	SMB 1.0
Windows 8 WS 2012	SMB 3.0	SMB 3.0	SMB 2.1	SMB 2.0	SMB 1.0
Windows 7 WS 2008 R2	SMB 2.1	SMB 2.1	SMB 2.1	SMB 2.0	SMB 1.0
Windows Vista WS 2008	SMB 2.0	SMB 2.0	SMB 2.0	SMB 2.0	SMB 1.0
Previous versions	SMB 1.0	SMB 1.0	SMB 1.0	SMB 1.0	SMB 1.0

* WS = Windows Server

4. Using PowerShell to check the SMB version

In Windows 8 or Windows Server 2012, there is a new PowerShell cmdlet that can easily tell you what version of SMB the client has negotiated with the File Server. You simply access a remote file server (or create a new mapping to it) and use Get-SmbConnection. Here’s an example:

PS C:\> Get-SmbConnection

ServerName   ShareName UserName            Credential          Dialect   NumOpens
----------   --------- --------            ----------          -------   --------
FileServer1 IPC$       DomainName\UserN... DomainName.Testi... 3.00      0
FileServer1 FileShare DomainName\UserN... DomainName.Testi... 3.00      14
FileServ2    FS2        DomainName\UserN... DomainName.Testi... 3.02      3
VNX3         Share1     DomainName\UserN... DomainName.Testi... 3.00      6
Filer2       Library    DomainName\UserN... DomainName.Testi... 3.00      8
DomainCtrl1 netlogon   DomainName\Compu... DomainName.Testi... 2.10      1

In the example above, a server called “FileServer1” was able to negotiate up to version 3.0. FileServ2 can use version 3.02. That means that both the client and the server support the latest version of the SMB protocol. You can also see that another server called “DomainCtrl1” was only able to negotiate up to version 2.1. You can probably guess that it’s a domain controller running Windows Server 2008 R2. Some of the servers on the list are not running Windows, showing the dialect that these non-Windows SMB implementations negotiated with this specific Windows client.

If you just want to find the version of SMB running on your own computer, you can use a loopback share combined with the Get-SmbConnection cmdlet. Here’s an example:

PS C:\> dir \\localhost\c$

Directory: \\localhost\c$

Mode                LastWriteTime     Length Name
----                -------------     ------ ----
d----         5/19/2012   1:54 AM            PerfLogs
d-r--          6/1/2012 11:58 PM            Program Files
d-r--          6/1/2012 11:58 PM            Program Files (x86)
d-r--         5/24/2012   3:56 PM            Users
d----          6/5/2012   3:00 PM            Windows

PS C:\> Get-SmbConnection -ServerName localhost

ServerName ShareName UserName            Credential          Dialect NumOpens
---------- --------- --------            ----------          ------- --------
localhost   c$         DomainName\UserN... DomainName.Testi... 3.02     0

You have about 10 seconds after you issue the “dir” command to run the “Get-SmbConnection” cmdlet. The SMB client will tear down the connections if there is no activity between the client and the server. It might help to know that you can use the alias “gsmbc” instead of the full cmdlet name.

5. Features and Capabilities

Here’s a very short summary of what changed with each version of SMB:

From SMB 1.0 to SMB 2.0 - The first major redesign of SMB
- Increased file sharing scalability
- Improved performance
  - Request compounding
  - Asynchronous operations
  - Larger reads/writes
- More secure and robust
  - Small command set
  - Signing now uses HMAC SHA-256 instead of MD5
  - SMB2 durability
From SMB 2.0 to SMB 2.1
- File leasing improvements
- Large MTU support
- BranchCache
From SMB 2.1 to SMB 3.0
- Availability
  - SMB Transparent Failover
  - SMB Witness
  - SMB Multichannel
- Performance
  - SMB Scale-Out
  - SMB Direct (SMB 3.0 over RDMA)
  - SMB Multichannel
  - Directory Leasing
  - BranchCache V2
- Backup
  - VSS for Remote File Shares
- Security
  - SMB Encryption using AES-CCM (Optional)
  - Signing now uses AES-CMAC
- Management
  - SMB PowerShell
  - Improved Performance Counters
  - Improved Eventing
From SMB 3.0 to SMB 3.02
- Automatic rebalancing of Scale-Out File Server clients
- Improved performance of SMB Direct (SMB over RDMA)
- Support for multiple SMB instances on a Scale-Out File Server

You can get additional details on the SMB 2.0 improvements listed above at
http://blogs.technet.com/b/josebda/archive/2008/12/09/smb2-a-complete-redesign-of-the-main-remote-file-protocol-for-windows.aspx

You can get additional details on the SMB 3.0 improvements listed above at
http://blogs.technet.com/b/josebda/archive/2012/05/03/updated-links-on-windows-server-2012-file-server-and-smb-3-0.aspx

You can get additional details on the SMB 3.02 improvements in Windows Server 2012 R2 at
http://technet.microsoft.com/en-us/library/hh831474.aspx

6. Recommendation

We strongly encourage you to update to the latest version of SMB, which will give you the most scalability, the best performance, the highest availability and the most secure SMB implementation.

Keep in mind that Windows Server 2012 Hyper-V and Windows Server 2012 R2 Hyper-V only support SMB 3.0 for remote file storage. This is due mainly to the availability features (SMB Transparent Failover, SMB Witness and SMB Multichannel), which did not exist in previous versions of SMB. The additional scalability and performance is also very welcome in this virtualization scenario. The Hyper-V Best Practices Analyzer (BPA) will warn you if an older version is detected.

7. Conclusion

We’re excited about SMB3, but we are also always concerned about keeping as much backwards compatibility as possible. Both SMB 3.0 and SMB 3.02 bring several key new capabilities and we encourage you to learn more about them. We hope you will be convinced to start planning your upgrades as early as possible.

Note 1: Protocol Documentation

If you consider yourself an SMB geek and you actually want to understand the SMB NEGOTIATE command in greater detail, you can read the [MS-SMB2-Preview] protocol documentation (which covers SMB 2.0, 2.1, 3.0 and 3.02), currently available from http://msdn.microsoft.com/en-us/library/ee941641.aspx. In regards to protocol version negotiation, you should pay attention to the following sections of the document:

1.7: Versioning and Capability Negotiation
2.2.3: SMB2 Negotiate Request
2.2.4: SMB2 Negotiate Response

Section 1.7 includes this nice state diagram describing the inner workings of protocol negotiation:

Note 2: Third-party implementations

There are several implementations of the SMB protocol from someone other than Microsoft. If you use one of those implementations of SMB, you should ask whoever is providing the implementation which version of SMB they implement for each version of their product. Here are a few of these implementations of SMB:

Apple – Up to SMB2 implemented in OS X 10 Mavericks - http://images.apple.com/osx/preview/docs/OSX_Mavericks_Core_Technology_Overview.pdf
EMC– Up to SMB3 implemented in VNX - http://www.emc.com/collateral/white-papers/h11427-vnx-introduction-smb-30-support-wp.pdf
Linux (Client) – Up to SMB3 implemented in the Linux kernel 3.11 – http://www.snia.org/sites/default/files2/SDC2012/presentations/Revisions/SteveFrench_Linux_CIFS-SMB2-year-in-review-revision.pdf (waiting for the latest SDC talk to become available, which has updated info on Linux support for SMB3)
NetApp– Up to SMB3 implemented in Data ONTAP 8.2 - https://communities.netapp.com/community/netapp-blogs/cloud/blog/2013/06/11/clustered-ontap-82-with-windows-server-2012-r2-and-system-center-2012-r2-innovation-in-storage-and-the-cloud
Samba (Server) – Up to SMB3 implemented in Samba 4.1 - http://www.samba.org/samba/history/samba-4.1.0.html

Please note that is not a complete list of implementations and the list is bound to become obsolete the minute I post it. Please refer to the specific implementers for up-to-date information on their specific implementations and which version and optional portions of the protocol they offer.

↧

Networking configurations for Hyper-V over SMB in Windows Server 2012 and Windows Server 2012 R2

October 9, 2013, 3:00 pm

≫ Next: Automatic SMB Scale-Out Rebalancing in Windows Server 2012 R2

≪ Previous: Windows Server 2012 R2: Which version of the SMB protocol (SMB 1.0, SMB 2.0, SMB 2.1, SMB 3.0 or SMB 3.02) are you using?

One of the questions regarding Hyper-V over SMB that I get the most relates to how the network should be configured. Networking is key to several aspects of the scenario, including performance, availability and scalability.

The main challenge is to provide a fault-tolerant and high-performance network for the two clusters typically involved: the Hyper-V cluster (also referred to as the Compute Cluster) and the Scale-out File Server Cluster (also referred to as the Storage Cluster).

Not too long ago, the typical configuration for virtualization deployments would call for up to 6 distinct networks for these two clusters:

Client (traffic between the outside and VMs running in the Compute Cluster)
Storage (main communications between the Compute and Storage clusters)
Cluster (communication between nodes in both clusters, including heartbeat)
Migration (used for moving VMs between nodes in the Compute Cluster)
Replication (used by Hyper-V replica to send changes to another site)
Management (used to configuring and monitoring the systems, typically also including DC and DNS traffic)

These days, it’s common to consolidate these different types of traffic, with the proper fault tolerance and Quality of Service (QoS) guarantees.

There are certainly many different ways to configure the network for your Hyper-V over SMB, but this blog post will focus on two of them:

A basic fault-tolerant solution using just two physical network ports per node
A high-end solution using RDMA networking for the highest throughput, highest density, lowest latency and low CPU utilization.

Both configurations presented here work with Windows Server 2012 and Windows Server 2012 R2, the two versions of Windows Server that support the Hyper-V over SMB scenario.

Configuration 1 – Basic fault-tolerant Hyper-V over SMB configuration with two non-RDMA port

The solution below using two network ports for each node of both the Compute Cluster and the Storage Cluster. NIC teaming is the main technology used for fault tolerance and load balancing.

Configuration 1: click on diagram to see a larger picture

Notes:

A single dual-port network adapter per host can be used. Network failures are usually related to cables and switches, not the NIC itself. It the NIC does fail, failover clustering on the Hyper-V or Storage side would kick in. Two network adapters each with one port is also an option.
The 2 VNICs on the Hyper-V host are used to provide additional throughput for the SMB client via SMB Multichannel, since the VNIC does not support RSS (Receive Side Scaling, which helps spread the CPU load of networking activity across multiple cores). Depending on configuration, increasing it up to 4 VNICs per Hyper-V host might be beneficial to increase throughput.
You can use additional VNICs that are dedicated for other kinds of traffic like migration, replication, cluster and management. In that case, you can optionally configure SMB Multichannel constraints to limit the SMB client to a specific subset of the VNICs. More details can be found in item 7 of the following article: The basics of SMB Multichannel, a feature of Windows Server 2012 and SMB 3.0
If RDMA NICs are used in this configuration, their RDMA capability will not be leveraged, since the physical port capabilities are hidden behind NIC teaming and the virtual switch.
Network QoS should be used to tame each individual type of traffic on the Hyper-V host. In this configuration, it’s recommended to implement the network QoS at the virtual switch level. See http://technet.microsoft.com/en-us/library/jj735302.aspx for details (the above configuration matches the second one described in the linked article).

Configuration 2 - High-performance fault-tolerant Hyper-V over SMB configuration with two RDMA ports and two non-RDMA ports

The solution below requires four network ports for each node of both the Compute Cluster and the Storage Cluster, two of them being RDMA-capable. NIC teaming is the main technology used for fault tolerance and load balancing on the two non-RDMA ports, but SMB Multichannel covers those capabilities for the two RDMA ports.

Configuration 2: click on diagram to see a larger picture

Notes:

Two dual-port network adapter per host can be used, one RDMA and one non-RDMA.
In this configuration, Storage, Migration and Clustering traffic should leverage the RDMA path. The client, replication and management traffic should use the teamed NIC path.
In this configuration, if using Windows Server 2012 R2, Hyper-V should be configured to use SMB for Live Migration. This is not the default setting.
The SMB client will naturally prefer the RDMA paths, so there is no need to specifically configure that preference via SMB Multichannel constraints.
There are three different types of RDMA NICs that can be used: iWARP, RoCE and InifiniBand. Below are links to step-by-step configuration instructions for each one:
Network QoS should be used to tame traffic flowing through the virtual switch on the Hyper-V host. If your NIC and switch support Data Center Bridging (DCB) and Priority Flow Control (PFC), there are additional options available as well. See http://technet.microsoft.com/en-us/library/jj735302.aspx for details (the above configuration matches the fourth one described in the linked article).
In most environments, RDMA provides enough bandwidth without the need of any traffic shaping. If using Windows Server 2012 R2, SMB Bandwidth Limits can optionally be used to shape the Storage and Live Migration traffic. More details can be found in item 4 of the following article: What’s new in SMB PowerShell in Windows Server 2012 R2. SMB Bandwidth Limits can also be used for configuration 1, but it's more common here.

I hope this blog posts helps with the network planning for your Private Cloud deployment. Feel free to ask questions via the comments below.

↧

Automatic SMB Scale-Out Rebalancing in Windows Server 2012 R2

October 30, 2013, 4:30 pm

≫ Next: Updated Links on Windows Server 2012 R2 File Server and SMB 3.02

≪ Previous: Networking configurations for Hyper-V over SMB in Windows Server 2012 and Windows Server 2012 R2

Introduction

This blog post focus on the new SMB Scale-Out Rebalancing introduced in Windows Server 2012 R2. If you haven’t seen it yet, it delivers a new way of balancing file clients accessing a Scale-Out File Server.

In Windows Server 2012, each client would be randomly directed via DNS Round Robin to a node of the cluster and stick with that one for all shares, all traffic going to that Scale-Out File Server. If necessary, some server-side redirection of individual IO requests could happen in order to fulfill the client request.

In Windows Server 2012 R2, a single client might be directed to a different node for each file share. The idea here is that the client will connect to the best node for each individual file share in the Scale-Out File Server Cluster, avoiding any kind of server-side redirection.

Now there are some details about when redirection can happen and when the new behavior will apply. Let’s look into the 3 types of scenarios you might encounter.

Hyper-V over SMB with Windows Server 2012 and a SAN back-end (symmetric)

When we first introduced the SMB Scale-Out File Server in Windows Server 2012, as mentioned in the introduction, the client would be randomly directed to one and only one node for all shares in that cluster.

If the storage is equally accessible from every node (what we call symmetric cluster storage), then you can do reads and writes from every file server cluster node, even if it’s not the owner node for that Cluster Shared Volume (CSV). We refer to this as Direct IO.

However, metadata operations (like creating a new file, renaming a file or locking byte range on a file) must be done orchestrated cross the cluster and will be executed on a single node called the coordinator node or the owner node. Any other node will simply redirect these metadata operations to the coordinator node.

The diagram below illustrates these behaviors:

Figure 1: Windows Server 2012 Scale-Out File Server on symmetric storage

The most common example of symmetric storage is when the Scale-Out File Server is put in front of a SAN. The common setup is to have every file server node connected to the SAN.

Another common example is when the Scale-Out File Server is using a clustered Storage Spaces solution with a shared SAS JBOD using Simple Spaces (no resiliency).

Hyper-V over SMB with Windows Server 2012 and Mirrored Storage Spaces (asymmetric)

When using a Mirrored Storage Spaces, the CSV operates in a block level redirected IO mode. This means that every read and write to the volume must be performed through the coordinator node of that CSV.

This configuration, where not every node has the ability to read/write to the storage, is generically called asymmetric storage. In those cases, every data and metadata request must be redirected to the coordinator node.

In Windows Server 2012, the SMB client chooses one of the nodes of the Scale-Out File Server cluster using DNS Round Robin and that may not necessarily be the coordinator node that owns the CSV that contains the file share it wants to access.

In fact, if using multiple file shares in a well-balanced cluster, it’s likely that the node will own some but not all of the CSVs required.

That means some SMB requests (for data or metadata) are handled by the node and some are redirected via the cluster back-end network to the right owner node. This redirection, commonly referred to as “double-hop”, is a very common occurrence in Windows Server 2012 when using the Scale-Out File Server combined with Mirrored Storage Spaces.

It’s important to mention that this cluster-side redirection is something that is implemented by CSV and it can be very efficient, especially if your cluster network uses RDMA-capable interfaces.

The diagram below illustrates these behaviors:

Figure 2: Windows Server 2012 Scale-Out File Server on asymmetric storage

The most common example of asymmetric storage is when the Scale-Out File Server is using a Clustered Storage Spaces solution with a Shared SAS JBOD using Mirrored Spaces.

Another common example is when only a subset of the file server nodes is directly connected to a portion backend storage, be it Storage Spaces or a SAN.

A possible asymmetric setup would be a 4-node cluster where 2 nodes are connected to one SAN and the other 2 nodes are connected to a different SAN.

Hyper-V over SMB with Windows Server 2012 R2 and Mirrored Storage Spaces (asymmetric)

If you’re following my train of thought here, you probably noticed that the previous configuration has a potential for further optimization and that’s exactly what we did in Windows Server 2012 R2.

In this new release, the SMB client gained the flexibility to connect to different Scale-Out File Server cluster nodes for each independent share that it needs to access.

The SMB server also gained the ability to tell its clients (using the existing Witness protocol) what is the ideal node to access the storage, in case it happens to be asymmetric.

With the combination of these two behavior changes, a Windows Server 2012 R2 SMB client and server are capable to optimize the traffic, so that no redirection is required even for asymmetric configurations.

The diagram below illustrates these behaviors:

Figure 3: Windows Server 2012 R2 Scale-Out File Server on asymmetric storage

Note that the SMB client now always talks to the Scale-Out File Server node that is the coordinator of the CSV where the share is.

Note also that the CSV ownership is shared across nodes in the example. That is not a coincidence. CSV now includes the ability to spread its CSVs across the nodes uniformly.

If you add or remove nodes or CSVs in the Scale-Out File Server cluster, the CSVs will be rebalanced. The SMB clients will then also be rebalanced to follow the CSV owner nodes for their shares.

Key configuration requirements for asymmetric storage in Windows Server 2012 R2

Because of this new automatic rebalancing, there are key new considerations when designing asymmetric (Mirrored or Parity Storage Spaces) storage when using Windows Server 2012 R2.

First of all, you should have at least as many CSVs as you have file server cluster nodes. For instance, for a 3-node Scale-Out File Server, you should have at least 3 CSVs. Having 6 CSVs is also a valid configuration, which will help with rebalancing when one of the nodes is down for maintenance.

To be clear, if you have a single CSV in such asymmetric configuration in Windows Server 2012 R2 Scale-Out File Server cluster, only one node will be actively accessed by SMB clients.

You should also try, as much as possible, to have your file shares and workloads evenly spread across the multiple CSVs. This way you won’t have some nodes working much harder than others.

Forcing per-share redirection for symmetric storage in Windows Server 2012 R2

The new per-share redirection does not happen by default in the Scale-Out File Server if the back-end storage is found to be symmetric.

For instance, if every node of your file server is connected to a SAN back-end, you will continue to have the behavior described on Figure 1 (Direct IO from every node plus metadata redirection).

The CSVs will automatically be balanced across file server cluster nodes even in symmetric storage configurations. You can turn that behavior off using the cmdlet below, although I'm hard pressed to find any good reason to do it.

(Get-Cluster). CSVBalancer = 0

However, when using symmetric storage, the SMB clients will continue to each connect a single file server cluster node for all shares. We opted for this behavior by default because Direct IO tends to be efficient in these configurations and the amount of metadata redirection should be fairly small.

You can override this setting and make the symmetric cluster use the same rebalancing behavior as an asymmetric cluster by using the following PowerShell cmdlet:

Set-ItemProperty HKLM:\System\CurrentControlSet\Services\LanmanServer\Parameters -Name AsymmetryMode -Type DWord -Value 2 -Force

You must apply the setting above to every file server cluster node. The new behavior won’t apply to existing client sessions.

If you switch to this configuration, you must apply the same planning rules outlined previously (at least one CSV per file server node, ideally two).

Conclusion

I hope this clarifies the behavior changes introduced with SMB Scale-Out Automatic Rebalancing in Windows Server 2012 R2.

While most of it is designed to just work, I do get some questions about it from those interested in understanding what happens behind the scenes.

Let me know if you find those useful or if you have any additional questions.

↧

Updated Links on Windows Server 2012 R2 File Server and SMB 3.02

November 4, 2013, 9:00 am

≫ Next: Storage Developer Conference - SDC 2013 slides now publicly available. Here are the links to Microsoft slides...

≪ Previous: Automatic SMB Scale-Out Rebalancing in Windows Server 2012 R2

In this post, I'm providing a reference to the most relevant content related to Windows Server 2012 R2 that is related to the File Server, the SMB 3.02 features and its associated scenarios like Hyper-V over SMB and SQL Server over SMB. It's obviously not a complete reference (there are always new blogs and articles being posted), but hopefully this is a useful collection of links for Windows Server 2012 R2 users.

This post covers only articles that are specific to Windows Server 2012 R2. However, note that there’s also a Windows Server 2012 version of this post. Most concepts, step-by-steps and tools listed there also apply to Windows Server 2012 R2.

Overview articles on Windows Server 2012 R2 File Server and SMB 3.02 (and related topics)

TechNet - What's New for SMB in Windows Server 2012 R2
TechNet - Virtual Hard Disk Sharing Overview
TechNet - What's New in Failover Clustering in Windows Server 2012 R2
TechNet - What’s New in Hyper-V in Windows Server 2012 R2
TechNet - Performance Tuning Guidelines for Windows Server 2012 R2 - Server Roles - File Server
Jose's Blog - Windows Server 2012 R2: Which version of the SMB protocol (SMB 1.0, SMB 2.0, SMB 2.1, SMB 3.0 or SMB 3.02) are you using?
Jose's Blog - What’s new in SMB PowerShell in Windows Server 2012 R2
Jose's Blog - Raw notes from the Storage Developers Conference (SDC 2013) (includes developer-related updates on SMB 3.02 and SMB Direct v2)
Jose's Blog - Networking configurations for Hyper-V over SMB in Windows Server 2012 and Windows Server 2012 R2
Jose's Blog - Selecting the number of nodes for your Scale-Out File Server Cluster
Windows Server Blog - Storage Transformation for your Datacenter
In The Cloud Blog - What’s New in 2012 R2: IaaS Innovations (many additional storage links here)

Step-by-step instructions for Windows Server 2012 R2 File Server and SMB 3.0 (and related topics)

Jose's Blog - Windows Server 2012 R2 Storage: Step-by-step with Storage Spaces, SMB Scale-Out and Shared VHDX (Virtual)
Jose's Blog - Windows Server 2012 R2 Storage: Step-by-step with Storage Spaces, SMB Scale-Out and Shared VHDX (Physical)
Jose's Blog - Step-by-step for Storage Spaces Tiering in Windows Server 2012 R2
Jose's Blog - Automatic SMB Scale-Out Rebalancing in Windows Server 2012 R2
TechNet - Deploy a Guest Cluster Using a Shared Virtual Hard Disk
TechNet - How to Create a Guest Cluster by Using a Service Template in VMM

TechEd 2013 presentations (with video recording) on Windows Server 2012 R2 File Server and SMB 3.0 (and related topics)

MDC-B333 - Storage and Availability Improvements in Windows Server 2012 R2 (includes overview of new SMB, Storage Spaces and Clustering features in Windows Server 2012 R2)
MDC-B335 - Understanding the Hyper-V over SMB Scenario, Configurations, and End-to-End Performance (includes new SMB features in Windows Server 2012 R2)
MDC-B311 - Application Availability Strategies for the Private Cloud (includes details on the Shared VHDX feature in Windows Server 2012 R2)
MDC-B331 - Upgrading Your Private Cloud with Windows Server 2012 R2 (includes details on upgrading a File Server clustering to Windows Server 2012 R2)
MDC-B218 - Storage Spaces: What’s New in Windows Server 2012 R2 (includes details on the new Tiering and Write-back Cache features in Storage Spaces)
MDC-B342 - Reduce Storage Costs with Data Deduplication (includes details on how to use Deduplication with live VHD files in a VDI scenario)
MDC-B336 - Cluster in a Box in 2013: How Real Customers Are Making Their Businesses Highly-Available with… (updated view of the CiB solutions)
MDC-B330 - Hyper-V - What's New in Windows Server 2012 R2 (include overview and/or demos of Storage QoS and Live Migration over SMB and Shared VHDX)
MDC-B357 - What’s New in Microsoft System Center 2012 R2 - Virtual Machine Manager (includes overview of VMM management of Storage Spaces, Scale-out File Servers and Shared VHDX)

Demos, Interviews and other video recordings

Demo: SMB Scale-Out Rebalancing (only the demo from the MDC-B335 presentation above)
Demo: SMB Direct Performance (only the demo from the MDC-B335 presentation above)
Demo: Hyper-V Live Migration over SMB (only the demo from the MDC-B335 presentation above)
Demo: Guest Clustering with Shared VHDX (only the demo from the MDC-311 presentation above)
Channel 9 Live with Rick Claus - Storage Changes in Windows Server 2012 R2
Carsten Rachfahl (MVP) - Interview with Jose Barreto about the SMB Enhancements in WS2012R2

Windows Server 2012 R2 download links

TechNet Evaluation Center - Download Windows Server 2012 R2 Preview (public download)
MSDN Subscriber Downloads - Windows Server 2012 R2 Datacenter Preview (x64) - DVD (English) (subscriber login required)
Drivers - Chelsio - Latest RDMA drivers for Chelsio for Windows Server 2012 R2 Preview (public download, select latest Unified Wire driver)
Drivers - Mellanox - Latest RDMA drivers for Mellanox for Windows Server 2012 R2 Preview (public download)

Blog posts by Microsoft MVPs on Windows Server 2012 R2 File Server and SMB 3.0 (and related topics)

Aidan Finn - Windows Server 2012 R2 Hyper-V – Guest Clustering With Shared VHDX
Aidan Finn - Windows Server 2012 R2 Hyper-V – Live Migration Improvements (Compression and SMB 3.0)
Aidan Finn - Creating & Adding Lots Of Shared VHDX Files To VMs Using PowerShell
Aidan Finn - Adding a Disk to a SOFS Clustered Storage Pool
Aidan Finn - Creating a Virtual WS2012R2 SOFS with Shared VHDX & Storage Spaces
Aidan Finn - A Converged Networks Design For SMB 3.0 Storage & SMB Direct Live Migration
Didier Van Hoye - Verifying SMB 3.0 Multichannel/RDMA Is Working In Windows Server 2012 (R2)
Didier Van Hoye - Shared Virtual Disks in Windows Server 2012 R2 Hyper-V Maximizes TCO/ROI
Didier Van Hoye - Preliminary Results With Live Migration Over RDMA Speed & Useful Number Of NICs
Didier Van Hoye - An Early Look At Live Migration Over TCP/IP & Multichannel In Windows Server 2012 R2 Preview
Ray Lucchesi - Windows Server 2012 R2 storage changes announced at TechEd

Windows Server 2012 R2 File Server Tips and Q&A

Coming later

Protocol Documentation

[MS-SMB2]: Server Message Block (SMB) Protocol Versions 2 and 3 (SMB protocol versions 2 and 3, including changes in this latest version)
[MS-SMBD]: SMB2 Remote Direct Memory Access (RDMA) Transport Protocol (SMB Direct protocol, including changes in this latest version)
[MS-SWN]: Service Witness Protocol (Witness protocol, including changes in this latest version)
[MS-RSVD]: Remote Shared Virtual Disk Protocol (the new Remote Shared Virtual Disk protocol)

Other relevant links related to Windows Server 2012 R2 SMB features

Windows Server Catalog – JBODs certified for Storage Spaces
Wiki - Storage Spaces Frequently Asked Questions (FAQ)

Knowledge Base - KB Articles

KB 2883200 - Windows Server 2012 R2 General Availability Update Rollup.

-------

Change tracking:

06/26/2013: Original post
07/11/2013: Added two demos, added Mellanox driver download, updated links
07/18/2013: Added more MVP blog links
07/31/2013: Additional blog links (including main In The Cloud blog, step-by-step guides)
09/23/2013: Additional blog links (SMB PowerShell, Storage Spaces) and links to final protocol documents
10/17/2013: Additional blog links (Networking) and GA Update Rollup link.
11/04/2013: Additional blog link (Rebalancing), Storage Spaces FAQ wiki link
11/10/2013: Additional blog link (Number of nodes), Performance guide link

↧

Storage Developer Conference - SDC 2013 slides now publicly available. Here are the links to Microsoft slides...

December 12, 2013, 12:07 pm

≫ Next: Troubleshooting File Server Networking Issues in Windows Server 2012 R2

≪ Previous: Updated Links on Windows Server 2012 R2 File Server and SMB 3.02

The Storage Networking Industry Association (SNIA) hosted the 10th Storage Developer Conference (SDC) in the Hyatt Regency in beautiful Santa Clara, CA (Silicon Valley) on the week of September 16th 2013.

This week, the presentation slides were made publicly available. You can find them all at http://snia.org/events/storage-developer2013/presentations13

For those focused on Microsoft technologies, here are some direct links to slides for the talks delivered by Microsoft this year:

Title	Presenters
Advancements in Windows File Systems	Neal Christiansen, Principal Development Lead, Microsoft
LRC Erasure Coding in Windows Storage Spaces	Cheng Huang, Researcher, Microsoft Research
SMB3 Update	David Kruse, Development Lead, Microsoft
Cluster Shared Volumes	Vladimir Petter, Principal Software Design Engineer, Microsoft
Tunneling SCSI over SMB: Shared VHDX files for Guest Clustering in Windows Server 2012 R2	Jose Barreto, Principal Program Manager, Microsoft Matt Kurjanowicz, Software Development Engineer, Microsoft
Windows Azure Storage - Speed and Scale in the Cloud	Joe Giardino, Senior Development Lead, Microsoft
SMB Direct update	Greg Kramer, Sr. Software Engineer, Microsoft
Scaled RDMA Performance & Storage Design with Windows Server SMB 3.0	Dan Lovinger, Principal Software Design Engineer, Microsoft
Data Deduplication as a Platform for Virtualization and High Scale Storage	Adi Oltean, Principal Software Design Engineer, Microsoft Sudipta Sengupta, Sr. Researcher, Microsoft

↧

Troubleshooting File Server Networking Issues in Windows Server 2012 R2

January 25, 2014, 9:25 am

≫ Next: SMB Direct and RDMA performance demo from TechEd (includes summary, PowerShell scripts and links)

≪ Previous: Storage Developer Conference - SDC 2013 slides now publicly available. Here are the links to Microsoft slides...

In this blog post, I’m sharing a few troubleshooting tips related to File Server Networking in Windows Server 2012 R2.

For each one, I first describe the issue that is commonly reported, followed by a quick explanation of the root cause of the issue and finally a way to solve it.

Let me know if those are helpful and feel free to share your own issues related to File Server Networking using the comments.

1. Make sure your network interfaces are RSS-capable

Issue:

Certain 10GbE NICs won’t perform as well as others
Multichannel might not aggregate multiple 10GbE in certain configurations

Cause

Some 10GbE NICs show as non-RSS capable. Without RSS, SMB uses 1 TCP connection.
If mixing RSS and non-RSS NICs, Multichannel will use only the RSS-capable NICs

Solution (both are complete solutions, choose one)

Server-class NICs should show as RSS-capable. Update the driver. Check RSS settings.
You might need to disable RSS on both to aggregate throughput.
Blog: Make sure your network interfaces are RSS-capable

2. Use Multiple VNICs on the host

Issue

When using a VNIC on the host, I cannot achieve maximum performance.
This is noticeable when using 10GbE NICs RSS-capable NICs and/or NIC Teaming.

Cause

The VNIC on the host is not RSS capable. Without RSS, SMB uses only 1 TCP connection.

Solution

Use multiple VNICs to make sure you have multiple connections.
Blog: Make sure your network interfaces are RSS-capable

3. SMB prefers slow NIC to faster VNICs

Issue

When using a VNIC on the host, SMB prefers a slower physical NIC to the VNIC.
This is important when using 10GbE NICs RSS-capable NICs and/or NIC Teaming.

Cause

The VNIC on the host is not RSS capable and the physical NIC is RSS capable.
SMB will always prefer RSS NICs to non-RSS NICs, even if at slower speeds.

Solution (both are complete solutions, choose one)

Disable the RSS-capability of the physical NIC
Use SMB Multichannel Constraints to prefer the VNICs

4. More TCP connections for SMB inside a VM

Issue

In Windows Server 2012, I got 1 TCP connection per VMNIC. In R2 I now get 4.
Why has this changed? Is there a problem?

Cause

With the new Virtual RSS feature, VMNICs now report themselves as RSS-capable.

Solution

This is by design. Enjoy the increased performance...

5. More connections to a Scale-Out File Server

Issue

Windows Server 2012 R2 creates more connections to Scale-out File Servers.
Why has this changed? Is there a problem?

Cause

Windows Server 2012 used one set of connections per Scale-Out File Server.
Windows Server 2012 R2 uses one set of connections per Share on a SOFS,
when this helps avoid server-side redirection (typical case: Mirrored Storage Spaces)

Solution

This is by design. Enjoy the increased performance...

6. Use multiple subnets when deploying SMB Multichannel in a cluster

Issue

When using multiple NICs on the same subnet in a cluster, only one is used

Cause

Cluster networking won’t use more than one NIC per subnet
You can confirm by using the cmdlet Get-SmbServerNetworkInterface

Solution

Configure each NIC to use a different subnet. Make sure cluster shows multiple networks.
Blog: Use multiple subnets when deploying SMB Multichannel in a cluster

7. Update firmware and driver for your NICs

Issue

I’m using an RDMA NIC, but SMB reports the NIC as not RDMA capable.

Cause

NICs with older firmware or driver might not report themselves correctly.

Solution

Update the firmware and driver with the latest from the manufacturer’s website.
Link: Latest RDMA drivers for Chelsio for Windows Server 2012 R2 (Select latest Unified Wire driver)
Link: Latest RDMA drivers for Mellanox for Windows Server 2012 R2

8. How much traffic needs to pass between the SMB Client and Server before Multichannel actually starts?

Issue

SMB3 always starts with single TCP/IP then moves to multiple TCP/IP or RDMA.
Concern with timing of initial handshake and speed of transition to faster behavior.

Cause

SMB Multichannel is used to discover RSS and RDMA capabilities.
For server SKUs, Multichannel starts on the first read or write operation.
For client SKUs, Multichannel won’t start unless you’re doing some amount of work.

Solution (both are complete solutions, choose one)

On server SKUs, don’t worry. This happens fast and is done only once per session.
On client SKUs, there is a registry configuration to use the server behavior, if necessary.
Blog: How much traffic needs to pass between the SMB Client and Server before Multichannel actually starts?

9. Can I use SMB3 storage without RDMA?

Issue

Concern around performance of SMB3 without RDMA NICs

Cause

We have talked so much about RDMA...

Solution (both items contribute to the solution)

Using non-RDMA NICs is fine. You can get good performance with non-RDMA 10GbE.
Watch your CPU utilization. make sure you’re using RSS NICs.
Blog: Can I use SMB3 storage without RDMA?

10. Is it possible to run SMB Direct from within a VM?

Issue

Running file server or SQL Server in a VM
Desire to use RDMA networking from the guest

Cause

Windows Server 2012 and Windows Server 2012 R2 cannot do RDMA to the guest

Solution (both are complete solutions, choose one)

Run the workload on the bare metal
Consider solutions that use the Hyper-V storage path (Hyper-V over SMB, Shared VHDX)
Blog: Is it possible to run SMB Direct from within a VM?

11. Use Client Access network for CSV Traffic

Issue

Limited performance when accessing a scale-out file server in Windows Server 2012
Happens when client hitting non-owner node and redirection is required

Cause

The client access network is high speed RDMA but the cluster network is not
Redirection is happening over cluster network only (usually a 1GbE NIC)

Solution (both are complete solutions, choose one)

Enable option to use client network for CSV traffic: (Get-Cluster).UseClientAccessNetworksForSharedVolumes=1
Upgrade to Windows Server 2012 R2 (automatic rebalancing avoids redirection)
Blog - Automatic SMB Scale-Out Rebalancing in Windows Server 2012 R2

12. Single file copy performance

Issue

Limited performance when copying a single large file to a scale-out file server
Using a 10GbE connection, can only achieve less than 150MB/sec throughput

Cause

File extension done in 1MB increments, serialized in write-trough mode
This leads to loss of the asynchronous nature of SMB2/3

Solution (both items contribute to the solution)

Copy multiple files in parallel (ROBOCOPY /MT)
Apply Windows Server 2012 R2 GA patch (changes to 8MB increments)
KB article and patch download - http ://support.microsoft.com/kb/2883200

↧

SMB Direct and RDMA performance demo from TechEd (includes summary, PowerShell scripts and links)

March 10, 2014, 10:00 am

≫ Next: TechEd 2014: File Server Networking for a Private Cloud Storage Infrastructure

≪ Previous: Troubleshooting File Server Networking Issues in Windows Server 2012 R2

Overview

My last TechEd demo showed some interesting performance data for SMB3 over RDMA (SMB Direct), including the latest small IO improvements in Windows Server 2012 R2.

Since I keep getting questions about the setup I used, here are some of the details about the hardware, software, results and script used.

Hardware

For that demo, I used a single storage server and a single compute server.

I used an EchoStreams FlacheSAN2working as File Server, with 2 Intel CPUs at 2.40 Ghz and 64GB of RAM. It includes 6 LSI SAS adapters and 48 Intel SSDs attached directly to the server. This is an impressively packed 2U unit.

The Hyper-V Server was a Dell PowerEdge R720 with 2 Intel CPUs at 2.70 GHz and 64GB of RAM.

Both the file server and the Hyper-V host had 3 RDMA-capable 54 Gbps NICs (Mellanox ConnectX-3 using FDR InfiniBand) were used simultaneously via SMB Multichannel.

Software and Configuration

For that demo, I ran Windows Server 2012 R2 on both the storage and the compute servers.

I used a standalone SMB3 file server backed by Storage Spaces, with 1 single share backed by a single mirrored space carved from a single storage pool with all 48 SSDs.

In previous demos I used multiple pools and spaces, but I switched to using a single pool and a single space, since it was simpler and provided me with similar performance.

From a networking perspective, the 3 NICs were configured each in a separate subnet. I also had a fourth NIC (1GbE) for DNS, AD and management traffic.

Speaking of that, this setup also communicated with a third Windows Server 2012 R2 server used solely as DNS Server and Active Directory domain controller.

Script to prepare the demo

Here’s the PowerShell script that I used to configure the environment for this specific demo:

# Create pool and mirrored space

$s = Get-StorageSubSystem -FriendlyName *Spaces*
$d = Get-PhysicalDisk –CanPool $true
New-StoragePool -FriendlyName Pool1 -StorageSubSystemFriendlyName $s.FriendlyName -PhysicalDisks $d

Set-ResiliencySetting -Name Mirror -NumberofColumnsDefault 24 -StoragePool ( Get-StoragePool -FriendlyName Pool1 )
New-VirtualDisk -FriendlyName Space1 -StoragePoolFriendlyName Pool1 -ResiliencySettingName Mirror –UseMaximumSize

# Initialize disk, partition and volume

$c = Get-VirtualDisk -FriendlyName Space1 | Get-Disk
Set-Disk -Number $c.Number -IsReadOnly 0
Set-Disk -Number $c.Number -IsOffline 0
Initialize-Disk -Number $c.Number -PartitionStyle GPT
New-Partition -DiskNumber $c.Number -DriveLetter X -UseMaximumSize
Initialize-Volume -DriveLetter X -FileSystem NTFS -Confirm:$false

# Create data files for SQLIO

1..16 | % {
$f=”X:\test”+$_+”.dat”
fsutil file createnew $f (128GB)
fsutil file setvaliddata $f (128GB)
$f=”X:\small”+$_+”.dat”
fsutil file createnew $f (8MB)
fsutil file setvaliddata $f (8MB)
}

# Create the SMB Share

New-SmbShare -Name Share1 -Path X:\ -FullAccess Domain\Administrator, Domain\HV1$, Domain\HV2$
Set-SmbPathAcl –ShareName Share1

Results

Here’s the summary of the results from the 3 phases of the demo:

Demo 1: Small IOs (8KB) from real storage

The first demo used 16 instances of SQLIO generating 8KB IOs against 16 distinct files on the SMB server.

As shown in the screenshot below, we hit over 600,000 IOs per second (IOPS). At that point, the data rate was about 5 gigabytes per seconds and the client was using a little over 60% of the CPU.

In this demo, every one of the 600,000 IOs are flowing through the entire Microsoft storage stack, from physical disks to storage spaces to NTFS to the SMB server over the network to the SMB client and finally to the SQLIO app.

There was a fair amount of queuing to keep all 48 SSDs and the entire stack busy (over 235 queue depth), but the overall latency was still below 1 millisecond (performance monitor shows 0).

Demo 2: Small IOs (8KB) from the SMB server cache

The second demo focus on the raw network performance of SMB Direct and SMB Multichannel by using cached IOs.

We’re still travelling from SQLIO to the SMB client over the network to the SMB server, but we’re satisfying the 8KB IOs from the RAM-based cache on the server side.

To accomplish this I basically used a similar workload as before, but employed smaller files and the SQLIO option to allow caching of the IOs.

As shown on the screenshot below, we have 1.1 million IOPS of 8KB each. At this rate, we are CPU bound at around 98% of the SMB client CPU.

Again you can see a deep queue (nearly 300 queued IOs), but the latency is still under 1 millisecond (performance monitor showing 0 again).

Note that, even with small IOs, we are hitting over 9 gigabytes per second in terms of bandwidth.

Demo 3: Larger IOs (32KB) from real storage

The last of the 3 demos used larger IOs (32KB) in order to reach higher bandwidth utilization.

Before the Windows Server 2012 R2 optimizations, reaching high bandwidth in this configuration would require larger IOs, like 128KB, 256KB or 512KB.

In fact, this was the first time I was able to nearly saturate this 3 * 54 Gbps network setup using 32KB IOs, which is not really a very large size.

You can see we’re hitting the incredible rate of 16.4 gigabytes per second, which is nearly saturating our 162 Gbps bandwidth.

To put it into perspective, that’s about 14 times the through of a regular 10GbE NIC (which typically delivers 1.1 gigabytes per second each way) or over 20 times the rate of a regular 8GB Fibre Channel HBA (which delivers about 800 megabytes per second each way).

Note also that we’re using about 64% of the CPU and the latency is under 2ms (performance monitor shows 1 millisecond).

Script to run the demo

Finally, as requested, here is the PowerShell script I used to generate the workload during the demo.

It effectively runs 16 instances of SQLIO to give me 16 independent processes, each running against one of the 16 cores in the machine.

Each instance uses a separate file on the share, which is mapped to the X: drive. I used either X:\Test<n>.dat (demos 1 and 3) or X:\Small<x>.dat (demo 2).

Note also somewhat unusual SQLIO options -BYRT to buffer IOs (demo 2) and -a to affinitize the instance to a specific set of CPU cores (used in all 3 demos).

During the demo, I focus on looking at Performance Monitor itself (not the output of SQLIO), so role of the script is really to drive the workload.

However, to make it look pretty, I used a few tricks. For instance, I clear the screen between demos and repaint the history of the results.

I also use the trick on Write-Host to stay on the same line and overwrite the contents. This is useful, for instance, when counting from job 1 to job 16.

1..16 | % {
    $file = $_
    $cpum = 1 -shl ($file - 1)
    Write-Host "`rStarting job ", $file -NoNewLine
    $ScriptBlock = {
        param($f, $m)
        $pa = "-a" + $m
        $pf = "X:\Test" + $f + ".dat"
        c:\sqlio\sqlio2.exe -s1000 -T100 -t1 -o16 -b8 $pa -BN -LS -frandom $pf
    }

$job = Start-Job -ScriptBlock $ScriptBlock -ArgumentList $file, $cpum
}
Write-Host "`rAll jobs have been started"
" "
Read-Host "Press [ENTER] to continue"

# Stop All jobs

$item = 0
get-job | % {
   $item++
   Write-Host "`rStopping job ", $item -NoNewLine
   Stop-Job $_
   Remove-Job $_
}

1..16 | % {
    $file = $_
    $cpum = 1 -shl ($file - 1)
    Write-Host "`rStarting job ", $file -NoNewLine
    $ScriptBlock = {
        param($f, $m)
        $pa = "-a" + $m
        $pf = "X:\Small" + $f + ".dat"
        c:\sqlio\sqlio2.exe -s1000 -T100 -t1 -o32 -b8 $pa -BYRT -LS -frandom $pf
    }
    $job = Start-Job -ScriptBlock $ScriptBlock -ArgumentList $file, $cpum
}
Write-Host "`rAll jobs have been started"
" "
Read-Host "Press [ENTER] to continue"

# Stop All jobs

$item = 0
get-job | % {
   $item++
   Write-Host "`rStopping job ", $item -NoNewLine
   Stop-Job $_
   Remove-Job $_
}
Write-Host "`rAll jobs have been stopped"

Cls
"   ___ _ ______            _ _     _    "
" ( _ )| |/ / __ )        __| (_)___| | __"
" / _ \| ' /| _ \ _____ / _' | / __| |/ /"
" | (_) | . \| |_) |_____| (_| | \__ \   < "
" \___/|_|\_\____/       \__,_|_|___/_|\_\"
""
"Workload: 8KB random read IOs coming from remote disk"
"Results: around 600,000 8KB IOPs"
" "
"   ___ _ ______                       _          "
" ( _ )| |/ / __ )        ___ __ _ ___| |__   ___ "
" / _ \| ' /| _ \ _____ / __/ _' |/ __| '_ \ / _ \"
" | (_) | . \| |_) |_____| (_| (_| | (__| | | | __/"
" \___/|_|\_\____/       \___\__,_|\___|_|_|_|\___|"
" "
"Workload: 8KB random read IOs coming remote cached (RAM)"
"Results: around 1,000,000 8KB IOPs"
" "
" _____ ___ _ __ ___            _ _     _        "
" |___ /___ \| |/ / __ )        __| (_)___| | __    "
"   |_ \ __) | ' /| _ \ _____ / _' | / __| |/ /    "
" ___) / __/| . \| |_) |_____| (_| | \__ \   <     "
" |____/_____|_|\_\____/       \__,_|_|___/_|\_\    "
" "
"Workload: 32KB random read IOs coming from remote disk"
" "
1..16 | % {
    $file = $_
    $cpum = 1 -shl ($file - 1)
    Write-Host "`rStarting job ", $file -NoNewLine
    $ScriptBlock = {
        param($f, $m)
        $pa = "-a" + $m
        $pf = "X:\Test" + $f + ".dat"
        c:\sqlio\sqlio2.exe -s1000 -T100 -t1 -o32 -b32 $pa -BN -LS -frandom $pf
    }

$job = Start-Job -ScriptBlock $ScriptBlock -ArgumentList $file, $cpum
}
Write-Host "`rAll jobs have been started"
" "
Read-Host "Press [ENTER] to continue"

# Stop All jobs

$item = 0
get-job | % {
   $item++
   Write-Host "`rStopping job ", $item -NoNewLine
   Stop-Job $_
   Remove-Job $_
}
Write-Host "`rAll jobs have been stopped"

Cls
"   ___ _ ______            _ _     _    "
" ( _ )| |/ / __ )        __| (_)___| | __"
" / _ \| ' /| _ \ _____ / _' | / __| |/ /"
" | (_) | . \| |_) |_____| (_| | \__ \   < "
" \___/|_|\_\____/       \__,_|_|___/_|\_\"
""
"Workload: 8KB random read IOs coming from remote disk"
"Results: around 600,000 8KB IOPs"
" "
"   ___ _ ______                       _          "
" ( _ )| |/ / __ )        ___ __ _ ___| |__   ___ "
" / _ \| ' /| _ \ _____ / __/ _' |/ __| '_ \ / _ \"
" | (_) | . \| |_) |_____| (_| (_| | (__| | | | __/"
" \___/|_|\_\____/       \___\__,_|\___|_|_|_|\___|"
" "
"Workload: 8KB random read IOs coming remote cached (RAM)"
"Results: around 1,000,000 8KB IOPs"
" "
" _____ ___ _ __ ___            _ _     _        "
" |___ /___ \| |/ / __ )        __| (_)___| | __    "
"   |_ \ __) | ' /| _ \ _____ / _' | / __| |/ /    "
" ___) / __/| . \| |_) |_____| (_| | \__ \   <     "
" |____/_____|_|\_\____/       \__,_|_|___/_|\_\    "
" "
"Workload: 32KB random read IOs coming from remote disk"
"Results: around 500,000 IOPs, around 16.5 GBytes/sec throughput"
" "
" "

TechEd 2014: File Server Networking for a Private Cloud Storage Infrastructure

March 13, 2014, 11:14 pm

≫ Next: Deploying a Windows Server 2012 R2 Scale-Out File Server Cluster using Azure VMs

≪ Previous: SMB Direct and RDMA performance demo from TechEd (includes summary, PowerShell scripts and links)

Just a reminder that I will be delivering a session at TechEd 2014 in Houston, Texas. Here are the details:

Code		DCIM-B337
Title		File Server Networking for a Private Cloud Storage Infrastructure in Windows Server 2012 R2
Date		Tuesday, May 13 3:15 PM - 4:30 PM
Track		Datacenter and Infrastructure Management
Topic		Windows Server
Abstract		Microsoft has shared the Private Cloud Vision that leverages the storage innovations in Windows Server 2012 and Windows Server 2012 R2. That vision is implemented as a Software-Defined Storage Architecture that leverages JBODs, Storage Spaces, Cluster Shared Volumes, Scale-Out File Servers and SMB3 file shares. In this session, we introduce those concepts and focus on a key aspect of it: the network that connects the compute nodes (Hyper-V hosts) and the storage nodes (File Servers). After stating the overall goals of network fault tolerance and performance, we describe specific features and capabilities in Windows Server that deliver on them, including NIC Teaming, Failover Cluster Networking, SMB Multichannel and SMB Direct. We close by sharing, as end-to-end examples, the two most common networking configurations for Hyper-V over SMB.

The talk is really about about Networking for the Private Cloud that uses SMB3. This goes from the overall architecture to the individual features (SMB Multichannel, SMB Direct, NIC Teaming) to a few recommended end-to-end configurations. Obviously we’ll also talk about performance and there’s a new focus on troubleshooting of some common networking issues.

If you want to get a taste of a few topics we’ll cover, here are few links to get you warmed up. You should consider this “recommended reading” for those attending the session.

I look forward to seeing you there and make sure to bring your File Server networking questions.

To see the other sessions at TechEd 2014, make sure to visit http://northamerica.msteched.com/Catalog

↧

Deploying a Windows Server 2012 R2 Scale-Out File Server Cluster using Azure VMs

March 29, 2014, 10:36 am

≫ Next: Step-by-Step for Mirrored Storage Spaces Resiliency using PowerShell

≪ Previous: TechEd 2014: File Server Networking for a Private Cloud Storage Infrastructure

0) Overview

In this document, I am sharing the steps I used to create a Windows Server 2012 R2 Scale-Out File Server Cluster environment using only Azure VMs, for demo, test or development purposes.

This includes a few important tips and tricks to install Active Directory and Windows Server Failover Clustering in an Azure VM environment which are not well known.

Following these steps will probably requires a reasonable amount of work, but it is a great way to experiment with a large set of Microsoft technologies in or related to Windows Server 2012 R2, including:

Azure VMs
Azure Virtual Networks
Domain Name Services (DNS)
Active Directory Domain Services (AD-DS)
Failover Clustering
Scale-Out File Servers
PowerShell

You will need the following to perform the steps described here:

Any computer with access to the internet that can connect to Azure
An active Azure subscription (you can get a free trial from http://azure.com)
No server hardware is required. No download of Windows Server 2012 R2 is required.

Notes and disclaimers:

A certain familiarity with Windows administration and configuration is assumed.
Very little knowledge of Azure is assumed. If you’re already familiar with it, sorry for boring you with all the details.
If you are asked a question or required to perform an action that you do not see described in these steps, go with the default option.
There are usually several ways to perform a specific configuration or administration task. What I describe here is one of those many ways. It's not necessarily the best way, just the one I personally like best at the moment.
Throughout these instructions, I’ll prefix all items with “JOSE-324”. For your specific setup, choose your own unique prefix to avoid naming conflicts with other users in the Azure cloud.
The iSCSI Target Server configuration shown here is not highly available. For this reason, this setup should only be used for demo, test or development purposes.
IMPORTANT NOTE: Due to the item above this should *not* be used for Production environments.

1) Get to the portal

Browse to http://azure.com, then click on “portal” in the top bar.

2) Create an Azure Virtual Network

Click on the “Virtual Networks” icon (shown above).

Click on the “Create a Virtual Network” link.

Enter the name for the virtual network, the region closest to you and an affinity group name for the new virtual network. Click on the right arrow on the bottom right to continue.

Leave everything blank on the “DNS Servers and VPN Connectivity” page. Click on the right arrow to continue.

Accept the default 10.0.0.0/8 address space and configure a subnet for 10.1.1.0/24. Click on the Check icon to finish creating the virtual network. Then wait until the virtual network is created (this will take a moment).

Review the configuration of the virtual network.

3) Create an Azure Storage Account

Click on the “Storage” icon (shown above).

Click on the “Create a Storage Account” link.

Enter the name for the storage account (URL), the affinity group (same you used on the virtual network) and click on “Create Storage Account”. Wait for the storage account to be created (this will take a moment).

Review the configuration of the storage account.

4) Create an Azure Virtual Machine for the domain controller

Click on the “Virtual Machines” icon (shown above).

Click on the “Create a Virtual Machine” link.

Instead of using the “Quick Create” option, click on the “From Gallery” option to bring in the full wizard.

On the “Choose an Image” page, click on “Windows Server 2012 R2 Datacenter”. Click on the right arrow on the bottom right to continue.

Use the latest release date, enter a virtual machine name for your domain controller, select the small size, enter a user name and type in a password twice. Click on the right arrow on the bottom right to continue.

On this page, keep the default to “create a new cloud service”, keep the cloud service DNS name the same as the virtual machine name, select the virtual name we created previously, select the subnet that we created previously, select the storage account we created previously and keep the availability set to “(None)”. Click on the right arrow on the bottom right to continue.

Keep all the defaults for the virtual machine configuration. Click on the Check icon to finish creating the virtual machine. Then wait until the virtual machine is created (this will take a moment).

Review the configuration of the Virtual Machine.

5) Review the items created so far

Click on the “Virtual Networks” icon (shown above).

Click on the network we created previously, then click on the option to show the “Dashboard”.

Verify that the newly created virtual machine is using the 10.1.1.4 IP address on the subnet specified.

Click on the “Storage” icon (shown above).

Click on the Storage Account we created previously, then click on the option to show the “Containers”.

Click on the container called “vhds”.

Verify that you have the blobs for the virtual machine VHD file and its associated status.

Click on the “Cloud Services” icon (shown above).

Verify you have a cloud service associated with the newly created virtual machine, using the correct availability group.

Click on the “All Items” icon (shown above).

Review all the items we created so far.

6) Connect to the Azure Virtual Machine

Click on the “Virtual Machines” icon (shown above).

On the bar at the bottom of the screen, click on the “Connect” icon to connect to your virtual machine.

You will be offered a link to a Remote Desktop session file. Click on the “Open” button to start the remote desktop session.

On the Remote Desktop Connection dialog (shown above), click “Connect”.

On the Window Security dialog, enter a dot (.), followed by a backslash (\) and the name of the user you specified when creating the virtual machine. The dot means "this computer". Enter the password you specified previously and Click “OK” to continue.

In the final “Remote Desktop Connection” dialog, click “Yes” to accept the certificate. You will be connected to the virtual machine.

NOTE: To avoid this warning (and, more importantly, to be more secure), you can create certificates to use with these RDP connections by checking this page: http://msdn.microsoft.com/en-us/library/windowsazure/gg432987.aspx .

7) Install Active Directory in the Azure VM

Inside the virtual machine, bring up a PowerShell command line and use the following two cmdlets to load the Active Directory Domain Services role and promote this machine to a domain controller. This will take a while.

Install-WindowsFeature AD-Domain-Services -IncludeManagementTools
Install-ADDSForest -DomainName "JOSE.TEST"

You will be prompted for a Safe Mode Administrator password twice. You will also be asked to confirm the operation. After this, the configuration will start. This will take a while and you will see a few warnings.

After the script finishes, the virtual machine will reboot and your Remote Desktop session will be disconnected.

8) Review DNS and Active Directory configuration

Reconnect to the Azure VM using the instructions on step 6, this time using the new domain name instead of the dot (.) used in the first logon. You should use the same username, though. Note that the first logon on the freshly installed domain controller will take a little longer than usual.

Use the DNS Manager tool to verify that the DNS was properly configured:

Use the Active Directory Users and Computers tool to verify that the domain was properly configured:

9) Configure the DNS for your Azure Virtual Network

Click on the “Virtual Networks” icon (shown above).

Click on the network we created previously, then click on the option to “Configure”.

Add a DNS Server to the empty list, specifying the name and IP address of the virtual machine you configured (10.1.1.4).

Click on the “Save” button on the bottom bar (shown above) to save the changes.

Confirm the changes to the virtual network. Wait for the configuration to complete (this will take a moment).

10) Create two additional virtual machines

Click on the “Virtual Machines” icon (shown above).

Use the “+ New” link on the bottom left of the screen to add two virtual machines that we’ll use as file server cluster nodes.

Use steps similar to what you did in item 4, being careful to use the same Image, Storage Account, Virtual Network and Subnet as the first one. Use an FS1 and FS2 suffix for their names.

Here’s what the configuration should look like after you create them:

Make sure the VMs all use the same location.

Click on the “Virtual Networks” icon (shown above).

Click on the network we created previously, then click on the option to show the “Dashboard”.

Make sure the VMs all use the same subnet. Note the IP addresses used by each one.

11) Configure the two file server nodes

Using steps similar to item 6, connect to each of the two new virtual machines. Note that, on the first boot, you must use the computer name when logging in.

The steps are described here just once, but you must perform them for both VMs.

After the first login, verify that the DNS configuration received via DHCP is correct by running this cmdlet from PowerShell inside the virtual machine:

Get-DnsClientServerAddress -InterfaceAlias Ethernet* | FL

Next, configure the roles and features to be used (File Server and Clustering) using these PowerShell cmdlets:

Install-WindowsFeature File-Services, FS-FileServer, Failover-Clustering –IncludeManagementTools

Next, use the following cmdlet to join the domain:

Add-Computer -DomainName JOSE.TEST –Restart

Enter the domain credentials when prompted. The virtual machine will restart after that and you will be disconnected.

Reconnect to the Azure VM using the instructions on step 6, this time using the new domain name instead of the dot (.) used in the first logon. You should use the same username, though.

Note that the first logon after you joined the domain will take a little longer than usual.

Don’t forget to repeat these steps on the other VM.

11) Confirm that the two file server nodes have joined the domain

Reconnect to the Azure VM running the DNS/domain controller using the instructions on step 8.

Use the DNS Manager tool to verify that the DNS is showing all three virtual machines:

Use the Active Directory Users and Computers tool to verify that the two file servers are now part of the domain:

12) Configure the iSCSI Target in the DNS/Domain Controller

Connect to the Azure VM running the DNS/domain controller using the instructions on step 8.

Enable the iSCSI Target Server and configure it using the following PowerShell cmdlets:

Install-WindowsFeature FS-iSCSITarget-Server
MD C:\LUN
New-IscsiServerTarget -TargetName FileCluster -InitiatorID IPAddress:10.1.1.5, IPAddress:10.1.1.6
New-IscsiVirtualDisk -DevicePath C:\LUN\LUN0.VHDX -Size 1GB
1..2 | % {New-IscsiVirtualDisk -DevicePath C:\LUN\LUN$_.VHDX -Size 20GB}
0..2 | % {Add-iSCSIVirtualDiskTargetMapping -TargetName FileCluster -DevicePath C:\LUN\LUN$_.VHDX}

Verify using Server Manager that the iSCSI Target Server was properly created and that the LUNs are properly associated with it. See sample below:

13) Configure the iSCSI Initiator on the File Server virtual machine

Reconnect to the two File Server virtual machines.

Configure them to use the newly created iSCSI Target using the following PowerShell cmdlets:

Set-Service MSiSCSI -StartupType automatic
Start-Service MSiSCSI
New-iSCSITargetPortal -TargetPortalAddress 10.1.1.4
Get-iSCSITarget | Connect-iSCSITarget
Get-iSCSISession | Register-iSCSISession

IMPORTANT: The next step should be performed only on first of the File Server VM, not both.

On the first File Server VM, initialize the iSCSI LUNs (disks 2, 3 and 4) using the following PowerShell cmdlets:

2..4 | % {
$d = “--WXY”[$_]
Set-Disk -Number $_ -IsReadOnly 0
Set-Disk -Number $_ -IsOffline 0
Initialize-Disk -Number $_ -PartitionStyle MBR
New-Partition -DiskNumber $_ -DriveLetter $d –UseMaximumSize
Start-Sleep –Seconds 10
Initialize-Volume -DriveLetter $d -FileSystem NTFS -Confirm:$false
}

Verify using the Disk Management tool that you see the

14) Run Cluster Validation and create the cluster

From one of the two File Server cluster nodes, run cluster validation using the following PowerShell cmdlet:

Test-Cluster JOSE-324-FS1, JOSE-324-FS2

Validation will take a while to complete. Here’s what you should expect as the output of that cmdlet:

Note that you have a warning. If you look into the report, you’ll see that it relates to the fact that you have a single network for the cluster. Since this is a test configuration only, that’s fine.

Now go ahead and create the cluster using the following PowerShell cmdlet:

New-Cluster –Name JOSE-324-CL -Node JOSE-324-FS1, JOSE-324-FS2

15) Fixing the issue with the Cluster IP address

There is one issue you will see in this environment, which is the fact that the DHCP address assigned to the Cluster Name resource is invalid.

The Azure Network will give you the same IP that is already in use by one of the existing nodes, creating an IP Address conflict. I assume this issue is caused by the fact that this Cluster Name resource is not known to Azure.

You can confirm the issue using the Failover Cluster Manager tool to view the Cluster Core Resources:

If you dig deeper in the event log, you will find the message about the duplicate IP address:

There is a workaround, though. You need to:

Remove the existing DHCP-based Cluster IP Address Resource
Add a new Cluster IP Address Resource with a Static IP
Make sure all the dependencies are properly set with the new resource

You can do this via the GUI or you can run the following PowerShell script:

Stop-ClusterResource "Cluster Name"
Get-ClusterResource | ? { (($_.ResourceType -like "*Address*") -and ($_.OwnerGroup -eq "Cluster Group")) } | Remove-ClusterResource –Force
Add-ClusterResource -Name "Cluster IP Address" -Group "Cluster Group" -ResourceType "IP Address"
Get-ClusterResource –Name "Cluster IP Address" | Set-ClusterParameter -Multiple @{ “Network”=”Cluster Network 1”; "Address"="10.1.1.100"; ”SubnetMask”=”255.255.255.0”; "EnableDhcp"=0 }
Get-ClusterResource “Cluster Name” | Add-ClusterResourceDependency –Resource "Cluster IP Address"
Start-ClusterResource "Cluster Name"

After that, you will have the cluster fully functional.

16) Create the File Server Scale-Out Cluster

With the cluster created, you can go ahead and configure it for a Scale-Out File Server and create a few continuously available file shares.

You can do it from either of the two File Server cluster nodes, using the following PowerShell cmdlets:

Get-ClusterResource | ? OwnerGroup -like Available* | Add-ClusterSharedVolume
Add-ClusterScaleOutFileServerRole -Name JOSE-324-FS
MD C:\ClusterStorage\Volume1\DATA1
New-SmbShare -Name Share1 -Path C:\ClusterStorage\Volume1\DATA1 -FullAccess JOSE.TEST\Jose
Set-SmbPathAcl –ShareName Share1
MD C:\ClusterStorage\Volume2\DATA2
New-SmbShare -Name Share2 -Path C:\ClusterStorage\Volume2\DATA2 -FullAccess JOSE.TEST\Jose
Set-SmbPathAcl –ShareName Share2

You can check the configuration using the Failover Cluster Manager:

You can also use PowerShell to verify that configuration:

Get-ClusterResource | FT -AutoSize
Get-ClusterSharedVolume | FT -AutoSize
Get-SmbShare | FT -AutoSize

17) Final Notes

After you’re done with the demo/test/dev environment, don’t forget shutdown the VMs to save on your Azure resources.
I hope you enjoyed these step-by-step instructions. I strongly encourage you to try them out and perform the entire installation yourself. It’s a good learning experience.
Let me know how these steps worked for you using the comment section. If you run into any issues or found anything particularly interesting, don’t forget to mention the number of the step.

↧

Step-by-Step for Mirrored Storage Spaces Resiliency using PowerShell

April 1, 2014, 3:18 pm

≫ Next: Building your schedule for TechEd 2014? Here are some Storage-related sessions!

≪ Previous: Deploying a Windows Server 2012 R2 Scale-Out File Server Cluster using Azure VMs

0 - Overview

This blog post is a walkthrough of the resiliency offered by Storage Spaces when using mirrored spaces.
For this exercise we have a single virtual machine with multiple data disks used to create the spaces.
The idea is to simulate a number of different failure and recovery scenarios, reviewing the behavior.
It includes the procedure to replace a disk and repair a degraded Space.

1 - Two-way mirror with 3 disks

Part 1 of this post focuses on testing the resiliency of a two-way mirrored virtual disk (storage space) on a storage pool with 3 physical disks.
For each section you have the steps to perform (useful if you want to cut/paste), the actual output of the cmdlets on my test system and finally some notes explaining what happened.

1.1 - Attach 3 disks to a VM

Steps to perform on the host:

1..3 | % { New-VHD –Path D:\demo\disk$_.VHDX –Dynamic –SizeBytes 20GB } | Out-Null

1..3 | % { Add-VMHardDiskDrive –VMName VM2 –Path D:\demo\disk$_.VHDX -ControllerType SCSI}

Get-VMHardDiskDrive -VMName VM2

Output:

PS C:\> 1..3 | % { New-VHD –Path D:\demo\disk$_.VHDX –Dynamic –SizeBytes 20GB } | Out-Null

PS C:\> 1..3 | % { Add-VMHardDiskDrive –VMName VM2 –Path D:\demo\disk$_.VHDX -ControllerType SCSI}

PS C:\> Get-VMHardDiskDrive -VMName VM2

VMName ControllerType ControllerNumber ControllerLocation DiskNumber Path

------ -------------- ---------------- ------------------ ---------- ----

VM2 IDE 0 0 c:\demo\v2.vhdx

VM2 SCSI 0 0 D:\demo\disk1.VHDX

VM2 SCSI 0 1 D:\demo\disk2.VHDX

VM2 SCSI 0 2 D:\demo\disk3.VHDX

Notes:

Nothing much. Just your regular VM with 1 IDE boot disk and 3 SCSI data disks.

1.2 - Create pool and mirrored space

Steps to perform on the guest:

$s = Get-StorageSubSystem -FriendlyName *Spaces*

$d = Get-PhysicalDisk -CanPool $true
New-StoragePool -FriendlyName Pool1 -StorageSubSystemFriendlyName $s.FriendlyName -PhysicalDisks $d
$p = Get-StoragePool Pool1
Set-ResiliencySetting -StoragePool $p -Name Mirror -NumberofColumnsDefault 1 -NumberOfDataCopiesDefault 2
$p | New-VirtualDisk -FriendlyName Space1 -ResiliencySettingName Mirror –UseMaximumSize

$L ="X”
$N = (Get-VirtualDisk -FriendlyName Space1 | Get-Disk).Number
Set-Disk -Number $N -IsReadOnly 0
Set-Disk -Number $N -IsOffline 0
Initialize-Disk -Number $N -PartitionStyle MBR
New-Partition -DiskNumber $N -DriveLetter $L -UseMaximumSize
Start-Sleep –Seconds 10
Initialize-Volume -DriveLetter $L -FileSystem NTFS -Confirm:$false

Output:

PS C:\> $s = Get-StorageSubSystem -FriendlyName *Spaces*

PS C:\> $d = Get-PhysicalDisk -CanPool $true

PS C:\> New-StoragePool -FriendlyName Pool1 -StorageSubSystemFriendlyName $s.FriendlyName -PhysicalDisks $d

FriendlyName OperationalStatus HealthStatus IsPrimordial IsReadOnly

------------ ----------------- ------------ ------------ ----------

Pool1 OK Healthy False False

PS C:\> $p = Get-StoragePool Pool1

PS C:\> Set-ResiliencySetting -StoragePool $p -Name Mirror -NumberofColumnsDefault 1 -NumberOfDataCopiesDefault 2

PS C:\> $p | New-VirtualDisk -FriendlyName Space1 -ResiliencySettingName Mirror –UseMaximumSize

FriendlyName ResiliencySettingNa OperationalStatus HealthStatus IsManualAttach Size

------------ ------------------- ----------------- ------------ -------------- ----

Space1 Mirror OK Healthy False 27 GB

PS C:\> $L ="X”

PS C:\> $N = (Get-VirtualDisk -FriendlyName Space1 | Get-Disk).Number

PS C:\> Set-Disk -Number $N -IsReadOnly 0

PS C:\> Set-Disk -Number $N -IsOffline 0

PS C:\> Initialize-Disk -Number $N -PartitionStyle MBR

PS C:\> New-Partition -DiskNumber $N -DriveLetter $L -UseMaximumSize

Disk Number: 4

PartitionNumber DriveLetter Offset Size Type

--------------- ----------- ------ ---- ----

1 X 1048576 27 GB Logical

PS C:\> Start-Sleep –Seconds 10

PS C:\> Initialize-Volume -DriveLetter $L -FileSystem NTFS -Confirm:$false

DriveLetter FileSystemLabel FileSystem DriveType HealthStatus SizeRemaining Size

----------- --------------- ---------- --------- ------------ ------------- ----

X NTFS Fixed Healthy 26.9 GB 27 GB

Notes:

Created a Pool and Space. Resulting disk is initialized and mounted as volume X:.

1.3 - Verify health of the pool and space

Steps to perform on the guest:

Get-Volume X | FT -AutoSize

Get-Volume X | Get-Partition | FT -AutoSize

Get-Volume X | Get-Partition | Get-Disk | FT -AutoSize

Get-Volume X | Get-Partition | Get-Disk | Get-VirtualDisk | FT -AutoSize

Output:

PS C:\> Get-Volume X | FT -AutoSize

DriveLetter FileSystemLabel FileSystem DriveType HealthStatus SizeRemaining Size

----------- --------------- ---------- --------- ------------ ------------- ----

X NTFS Fixed Healthy 26.9 GB 27 GB

PS C:\> Get-Volume X | Get-Partition | FT -AutoSize

Disk Number: 4

PartitionNumber DriveLetter Offset Size Type

--------------- ----------- ------ ---- ----

1 X 1048576 27 GB IFS

PS C:\> Get-Volume X | Get-Partition | Get-Disk | FT -AutoSize

Number Friendly Name OperationalStatus Total Size Partition Style

------ ------------- ----------------- ---------- ---------------

4 Microsoft Storage Space Device Online 27 GB MBR

PS C:\> Get-Volume X | Get-Partition | Get-Disk | Get-VirtualDisk | FT -AutoSize

FriendlyName ResiliencySettingName OperationalStatus HealthStatus IsManualAttach Size

------------ --------------------- ----------------- ------------ -------------- ----

Space1 Mirror OK Healthy False 27 GB

FriendlyName OperationalStatus HealthStatus IsPrimordial IsReadOnly

------------ ----------------- ------------ ------------ ----------

Pool1 OK Healthy False False

FriendlyName CanPool OperationalStatus HealthStatus Usage Size

------------ ------- ----------------- ------------ ----- ----

PhysicalDisk2 False OK Healthy Auto-Select 19.25 GB

PhysicalDisk1 False OK Healthy Auto-Select 19.25 GB

PhysicalDisk3 False OK Healthy Auto-Select 19.25 GB

Notes:

All is healthy: Volume, Space (Virtual Disk), Pool and Physical Disks.

1.4 - Remove 1 disk

Steps to perform on the host:

Remove-VMHardDiskDrive -VMName VM2 -ControllerType SCSI -ControllerNumber 0 -ControllerLocation 0

Get-VMHardDiskDrive -VMName VM2

Output:

PS C:\> Remove-VMHardDiskDrive -VMName VM2 -ControllerType SCSI -ControllerNumber 0 -ControllerLocation 0

PS C:\> Get-VMHardDiskDrive -VMName VM2

VMName ControllerType ControllerNumber ControllerLocation DiskNumber Path

------ -------------- ---------------- ------------------ ---------- ----

VM2 IDE 0 0 c:\demo\v2.vhdx

VM2 SCSI 0 1 D:\demo\disk2.VHDX

VM2 SCSI 0 2 D:\demo\disk3.VHDX

Notes:

Removed one of the 3 SCSI disks for the VM.

That is the equivalent, in a physical machine, to pulling a drive out while the system is running (assuming you have a drive that supports hot removal).

1.5 - Verify effect on the pool and space

Steps to perform on the guest:

Get-Volume X | FT -AutoSize

Get-Volume X | Get-Partition | Get-Disk | Get-VirtualDisk | FT -AutoSize

Get-EventLog -LogName System -Source Disk | ? EntryType -ne "Information" | FL

Get-WinEvent -LogName Microsoft-Windows-Stor* | ? Level -lt 4 | FL

Output:

PS C:\> Get-Volume X | FT -AutoSize

DriveLetter FileSystemLabel FileSystem DriveType HealthStatus SizeRemaining Size

----------- --------------- ---------- --------- ------------ ------------- ----

X NTFS Fixed Healthy 26.9 GB 27 GB

PS C:\> Get-Volume X | Get-Partition | Get-Disk | Get-VirtualDisk | FT -AutoSize

FriendlyName ResiliencySettingName OperationalStatus HealthStatus IsManualAttach Size

------------ --------------------- ----------------- ------------ -------------- ----

Space1 Mirror Incomplete Warning False 27 GB

FriendlyName OperationalStatus HealthStatus IsPrimordial IsReadOnly

------------ ----------------- ------------ ------------ ----------

Pool1 Degraded Warning False False

FriendlyName CanPool OperationalStatus HealthStatus Usage Size

------------ ------- ----------------- ------------ ----- ----

PhysicalDisk2 False OK Healthy Auto-Select 19.25 GB

PhysicalDisk-1 False Lost Communication Warning Auto-Select 19.25 GB

PhysicalDisk3 False OK Healthy Auto-Select 19.25 GB

PS C:\> Get-EventLog -LogName System -Source Disk | ? EntryType -ne "Information" | FL

Index : 14802

EntryType : Warning

InstanceId : 2147745949

Message : Disk 1 has been surprise removed.

Category : (0)

CategoryNumber : 0

ReplacementStrings : {\DR11, 1}

Source : disk

TimeGenerated : 3/29/2014 5:04:16 PM

TimeWritten : 3/29/2014 5:04:16 PM

UserName :

PS C:\> Get-WinEvent -LogName Microsoft-Windows-Stor* | ? Level -lt 4 | FL

TimeCreated : 3/29/2014 5:04:16 PM

ProviderName : Microsoft-Windows-StorageSpaces-Driver

Id : 304

Message : One or more drives hosting data for storage space {1874345F-B794-11E3-80B7-00155D010205} have failed or

are missing. As a result, at least one copy of data is not available. However, at least one copy of

data is still available. Return Code: STATUS_SUCCESS

TimeCreated : 3/29/2014 4:54:00 PM

ProviderName : Microsoft-Windows-StorageSpaces-Driver

Id : 103

Message : The capacity consumption of the storage pool {1874344D-B794-11E3-80B7-00155D010205} has exceeded the

threshold limit set on the pool. Return Code: STATUS_SUCCESS

Notes:

As expected, the mirror is now incomplete and pool is degraded, but the volume is still healthy.

Applications won’t see the event, but alarm bells are ringing in the event logs.

1.6 - Add back the removed disk

Steps to perform on the host:

Add-VMHardDiskDrive -VMName VM2 -ControllerType SCSI -ControllerNumber 0 -ControllerLocation 0 –Path D:\demo\disk1.VHDX

Get-VMHardDiskDrive -VMName VM2

Output:

PS C:\> Add-VMHardDiskDrive -VMName VM2 -ControllerType SCSI -ControllerNumber 0 -ControllerLocation 0 –Path D:\demo\disk1.VHDX

PS C:\> Get-VMHardDiskDrive -VMName VM2

VMName ControllerType ControllerNumber ControllerLocation DiskNumber Path

------ -------------- ---------------- ------------------ ---------- ----

VM2 IDE 0 0 c:\demo\v2.vhdx

VM2 SCSI 0 0 D:\demo\disk1.VHDX

VM2 SCSI 0 1 D:\demo\disk2.VHDX

VM2 SCSI 0 2 D:\demo\disk3.VHDX

Notes:

Added the VHDX file back.

That is the equivalent, in a physical machine, plugging the drive back in (assuming you have a drive that supports hot plugging).

1.7 - Verify that the pool and space recover

Steps to perform on the guest:

Get-Volume X | FT -AutoSize

Get-Volume X | Get-Partition | Get-Disk | Get-VirtualDisk | FT -AutoSize

Get-WinEvent -LogName Microsoft-Windows-Stor* | ? Message -like "*repair*" | FL

Output:

PS C:\> Get-Volume X | FT -AutoSize

DriveLetter FileSystemLabel FileSystem DriveType HealthStatus SizeRemaining Size

----------- --------------- ---------- --------- ------------ ------------- ----

X NTFS Fixed Healthy 26.9 GB 27 GB

PS C:\> Get-Volume X | Get-Partition | Get-Disk | Get-VirtualDisk | FT -AutoSize

FriendlyName ResiliencySettingName OperationalStatus HealthStatus IsManualAttach Size

------------ --------------------- ----------------- ------------ -------------- ----

Space1 Mirror OK Healthy False 27 GB

FriendlyName OperationalStatus HealthStatus IsPrimordial IsReadOnly

------------ ----------------- ------------ ------------ ----------

Pool1 OK Healthy False False

FriendlyName CanPool OperationalStatus HealthStatus Usage Size

------------ ------- ----------------- ------------ ----- ----

PhysicalDisk2 False OK Healthy Auto-Select 19.25 GB

PhysicalDisk1 False OK Healthy Auto-Select 19.25 GB

PhysicalDisk3 False OK Healthy Auto-Select 19.25 GB

PS C:\> Get-WinEvent -LogName Microsoft-Windows-Stor* | ? Message -like "*repair*" | FL

TimeCreated : 3/29/2014 5:11:39 PM

ProviderName : Microsoft-Windows-StorageSpaces-Driver

Id : 308

Message : A repair attempt for storage space {1874345F-B794-11E3-80B7-00155D010205} was initiated by the driver.

Return Code: STATUS_SUCCESS

TimeCreated : 3/29/2014 4:54:00 PM

ProviderName : Microsoft-Windows-StorageSpaces-Driver

Id : 308

Message : A repair attempt for storage space {1874345F-B794-11E3-80B7-00155D010205} was initiated by the driver.

Return Code: STATUS_SUCCESS

Notes:

Storage Spaces, seeing the drive coming back, attempts a repair and succeeds. Not sure why the log shows two repairs.

After that, all is well again: virtual disks, storage pools and physical disks all report as healthy.

Again, this is completely transparent to the applications.

1.8 - Remove 2 disks

Steps to perform on the host:

Remove-VMHardDiskDrive -VMName VM2 -ControllerType SCSI -ControllerNumber 0 -ControllerLocation 0

Remove-VMHardDiskDrive -VMName VM2 -ControllerType SCSI -ControllerNumber 0 -ControllerLocation 1

Get-VMHardDiskDrive -VMName VM2

Output:

PS C:\> Remove-VMHardDiskDrive -VMName VM2 -ControllerType SCSI -ControllerNumber 0 -ControllerLocation 0

PS C:\> Remove-VMHardDiskDrive -VMName VM2 -ControllerType SCSI -ControllerNumber 0 -ControllerLocation 1

PS C:\> Get-VMHardDiskDrive -VMName VM2

VMName ControllerType ControllerNumber ControllerLocation DiskNumber Path

------ -------------- ---------------- ------------------ ---------- ----

VM2 IDE 0 0 c:\demo\v2.vhdx

VM2 SCSI 0 2 D:\demo\disk3.VHDX

Notes:

Removing 2 of the 3 disks now. There is now only one surviving disk. This will be trouble...

1.9 - Verify effect on the pool and space

Steps to perform on the guest:

Get-Volume X | FT –AutoSize

Get-VirtualDisk Space1 | FT -AutoSize

Get-VirtualDisk Space1 | Get-StoragePool | FT -AutoSize

Get-VirtualDisk Space1 | Get-PhysicalDisk | FT -AutoSize

Get-EventLog -LogName System -Source Disk -After "3/29/2014 5:10 PM” | ? EntryType -ne "Information" | FL

Get-WinEvent -LogName Microsoft-Windows-Stor* | ? { ($_.Level -lt 4) -and ($_.TimeCreated -gt "3/29/2014 5:10 PM") } | FL

Output:

PS C:\> Get-Volume X | FT –AutoSize

Get-Volume : No MSFT_Volume objects found with property 'DriveLetter' equal to 'X'. Verify the value of the property and retry.

At line:1 char:1

+ Get-Volume X | FT –AutoSize

+ ~~~~~~~~~~~~

+ CategoryInfo : ObjectNotFound: (X:Char) [Get-Volume], CimJobException

+ FullyQualifiedErrorId : CmdletizationQuery_NotFound_DriveLetter,Get-Volume

PS C:\> Get-VirtualDisk Space1 | FT -AutoSize

FriendlyName ResiliencySettingName OperationalStatus HealthStatus IsManualAttach Size

------------ --------------------- ----------------- ------------ -------------- ----

Space1 Mirror Detached Unhealthy False 27 GB

PS C:\> Get-VirtualDisk Space1 | Get-StoragePool | FT -AutoSize

FriendlyName OperationalStatus HealthStatus IsPrimordial IsReadOnly

------------ ----------------- ------------ ------------ ----------

Pool1 Read-only Unhealthy False False

PS C:\> Get-VirtualDisk Space1 | Get-PhysicalDisk | FT -AutoSize

FriendlyName CanPool OperationalStatus HealthStatus Usage Size

------------ ------- ----------------- ------------ ----- ----

PhysicalDisk-1 False Lost Communication Warning Auto-Select 19.25 GB

PhysicalDisk3 False OK Healthy Auto-Select 19.25 GB

PS C:\> Get-EventLog -LogName System -Source Disk -After "3/29/2014 5:10 PM” | ? EntryType -ne "Information" | FL

Index : 14807

EntryType : Warning

InstanceId : 2147745949

Message : Disk 4 has been surprise removed.

Category : (0)

CategoryNumber : 0

ReplacementStrings : {\DR14, 4}

Source : disk

TimeGenerated : 3/29/2014 5:17:32 PM

TimeWritten : 3/29/2014 5:17:32 PM

UserName :

Index : 14806

EntryType : Warning

InstanceId : 2147745949

Message : Disk 2 has been surprise removed.

Category : (0)

CategoryNumber : 0

ReplacementStrings : {\...\DR12, 2}

Source : disk

TimeGenerated : 3/29/2014 5:17:32 PM

TimeWritten : 3/29/2014 5:17:32 PM

UserName :

Index : 14805

EntryType : Warning

InstanceId : 2147745949

Message : Disk 1 has been surprise removed.

Category : (0)

CategoryNumber : 0

ReplacementStrings : {\DR15, 1}

Source : disk

TimeGenerated : 3/29/2014 5:17:31 PM

TimeWritten : 3/29/2014 5:17:31 PM

UserName :

PS C:\> Get-WinEvent -LogName Microsoft-Windows-Stor* | ? { ($_.Level -lt 4) -and ($_.TimeCreated -gt "3/29/2014 5:10 PM") } | FL

TimeCreated : 3/29/2014 5:17:32 PM

ProviderName : Microsoft-Windows-StorageSpaces-Driver

Id : 304

Message : One or more drives hosting data for storage space {1874345F-B794-11E3-80B7-00155D010205} have failed or

are missing. As a result, at least one copy of data is not available. However, at least one copy of

data is still available. Return Code: STATUS_SUCCESS

TimeCreated : 3/29/2014 5:17:32 PM

ProviderName : Microsoft-Windows-StorageSpaces-Driver

Id : 203

Message : Physical drive {38133EAF-D68C-7229-E30B-64D68782E509} failed an IO operation. Return Code: This device

does not exist.. Additional related events may be found in the System event log for Disk 2.

This drive may need to be replaced. To view its reliability counters, run this command in PowerShell:

Get-PhysicalDisk | ?{ $_.ObjectId -Match "{38133EAF-D68C-7229-E30B-64D68782E509}" } |

Get-StorageReliabilityCounter

This drive may be located using the following information:

Drive Manufacturer: Msft

Drive Model Number: Virtual Disk

Drive Serial Number:

More information can be obtained using this PowerShell command:

Get-PhysicalDisk | ?{ $_.ObjectId -Match "{38133EAF-D68C-7229-E30B-64D68782E509}" }

If this drive is in an enclosure, it may be located using the following information:

Enclosure Manufacturer:

Enclosure Model Number:

Enclosure Serial Number:

Enclosure Slot: -1

It may also be located by running this command in PowerShell:

Get-PhysicalDisk | ?{ $_.ObjectId -Match "{38133EAF-D68C-7229-E30B-64D68782E509}" } |

Enable-PhysicalDiskIndication

TimeCreated : 3/29/2014 5:17:32 PM

ProviderName : Microsoft-Windows-StorageSpaces-Driver

Id : 303

Message : Drives hosting data for storage space {1874345F-B794-11E3-80B7-00155D010205} have failed or are

missing. As a result, no copy of data is available. Return Code: This device does not exist.

TimeCreated : 3/29/2014 5:17:32 PM

ProviderName : Microsoft-Windows-StorageSpaces-Driver

Id : 303

Message : Drives hosting data for storage space {18743460-B794-11E3-80B7-00155D010205} have failed or are

missing. As a result, no copy of data is available. Return Code: This device does not exist.

TimeCreated : 3/29/2014 5:17:31 PM

ProviderName : Microsoft-Windows-StorageSpaces-Driver

Id : 304

Message : One or more drives hosting data for storage space {1874345F-B794-11E3-80B7-00155D010205} have failed or

are missing. As a result, at least one copy of data is not available. However, at least one copy of

data is still available. Return Code: STATUS_SUCCESS

TimeCreated : 3/29/2014 5:11:38 PM

ProviderName : Microsoft-Windows-StorageSpaces-Driver

Id : 304

Message : One or more drives hosting data for storage space {1874345F-B794-11E3-80B7-00155D010205} have failed or

are missing. As a result, at least one copy of data is not available. However, at least one copy of

data is still available. Return Code: STATUS_SUCCESS

Notes:

As expected, things get broken. The virtual disk is detached and the volume simply disappears completely.

The storage pool is showing as unhealthy and read-only. Two physical disks are correctly reported missing.

If there is an application using this specific volume, there will be failures and down time associated with this event.

1.10 - Add back the removed disks

Steps to perform on the host:

Add-VMHardDiskDrive -VMName VM2 -ControllerType SCSI -ControllerNumber 0 -ControllerLocation 0 –Path D:\demo\disk1.VHDX

Add-VMHardDiskDrive -VMName VM2 -ControllerType SCSI -ControllerNumber 0 -ControllerLocation 1 –Path D:\demo\disk2.VHDX

Get-VMHardDiskDrive -VMName VM2

Output:

PS C:\> Add-VMHardDiskDrive -VMName VM2 -ControllerType SCSI -ControllerNumber 0 -ControllerLocation 0 –Path D:\demo\disk1.VHDX

PS C:\> Add-VMHardDiskDrive -VMName VM2 -ControllerType SCSI -ControllerNumber 0 -ControllerLocation 1 –Path D:\demo\disk2.VHDX

PS C:\> Get-VMHardDiskDrive -VMName VM2

VMName ControllerType ControllerNumber ControllerLocation DiskNumber Path

------ -------------- ---------------- ------------------ ---------- ----

VM2 IDE 0 0 c:\demo\v2.vhdx

VM2 SCSI 0 0 D:\demo\disk1.VHDX

VM2 SCSI 0 1 D:\demo\disk2.VHDX

VM2 SCSI 0 2 D:\demo\disk3.VHDX

Notes:

Putting the two drives back in. This should fix things.

1.11 - Verify that the pool and space recover

Steps to perform on the guest:

Get-Volume X | FT -AutoSize

Get-Volume X | Get-Partition | Get-Disk | Get-VirtualDisk | FT -AutoSize

Get-WinEvent -LogName Microsoft-Windows-Stor* | ? { ($_.Message -like "*repair*") -and ($_.TimeCreated -gt "3/29/2014 5:20 PM") } | FL

Output:

PS C:\> Get-Volume X | FT -AutoSize

DriveLetter FileSystemLabel FileSystem DriveType HealthStatus SizeRemaining Size

----------- --------------- ---------- --------- ------------ ------------- ----

X NTFS Fixed Healthy 26.9 GB 27 GB

PS C:\> Get-Volume X | Get-Partition | Get-Disk | Get-VirtualDisk | FT -AutoSize

FriendlyName ResiliencySettingName OperationalStatus HealthStatus IsManualAttach Size

------------ --------------------- ----------------- ------------ -------------- ----

Space1 Mirror OK Healthy False 27 GB

FriendlyName OperationalStatus HealthStatus IsPrimordial IsReadOnly

------------ ----------------- ------------ ------------ ----------

Pool1 OK Healthy False False

FriendlyName CanPool OperationalStatus HealthStatus Usage Size

------------ ------- ----------------- ------------ ----- ----

PhysicalDisk4 False OK Healthy Auto-Select 19.25 GB

PhysicalDisk1 False OK Healthy Auto-Select 19.25 GB

PhysicalDisk3 False OK Healthy Auto-Select 19.25 GB

PS C:\> Get-WinEvent -LogName Microsoft-Windows-Stor* | ? { ($_.Message -like "*repair*") -and ($_.TimeCreated -gt "3/29/2014 5:20 PM") } | FL

TimeCreated : 3/29/2014 6:12:50 PM

ProviderName : Microsoft-Windows-StorageSpaces-Driver

Id : 308

Message : A repair attempt for storage space {1874345F-B794-11E3-80B7-00155D010205} was initiated by the driver.

Return Code: STATUS_SUCCESS

TimeCreated : 3/29/2014 6:12:47 PM

ProviderName : Microsoft-Windows-StorageSpaces-Driver

Id : 308

Message : A repair attempt for storage space {1874345F-B794-11E3-80B7-00155D010205} was initiated by the driver.

Return Code: STATUS_SUCCESS

Notes:

Without any need for repair commands, the physical disks are detected and go back to a healthy state.

The storage pool goes back to healthy and is no longer read-only. The virtual disk recovers and the volume shows up again.

You will have some work to do restart and recover your applications, which suffered the equivalent of a hard crash.

1.12- Remove 1 disk, add replacement disk

Steps to perform on the host:

Remove-VMHardDiskDrive -VMName VM2 -ControllerType SCSI -ControllerNumber 0 -ControllerLocation 0

New-VHD –Path D:\demo\disk1r.VHDX –Dynamic –SizeBytes 20GB

Add-VMHardDiskDrive –VMName VM2 -ControllerType SCSI -ControllerNumber 0 -ControllerLocation 0 –Path D:\demo\disk1r.VHDX

Get-VMHardDiskDrive -VMName VM2

Output:

PS C:\> Remove-VMHardDiskDrive -VMName VM2 -ControllerType SCSI -ControllerNumber 0 -ControllerLocation 0

PS C:\> New-VHD –Path D:\demo\disk1r.VHDX –Dynamic –SizeBytes 20GB | Out-Null

PS C:\> Add-VMHardDiskDrive –VMName VM2 -ControllerType SCSI -ControllerNumber 0 -ControllerLocation 0 –Path D:\demo\disk1r.VHDX

PS C:\> Get-VMHardDiskDrive -VMName VM2

VMName ControllerType ControllerNumber ControllerLocation DiskNumber Path

------ -------------- ---------------- ------------------ ---------- ----

VM2 IDE 0 0 c:\demo\v2.vhdx

VM2 SCSI 0 0 D:\demo\disk1r.VHDX

VM2 SCSI 0 1 D:\demo\disk2.VHDX

VM2 SCSI 0 2 D:\demo\disk3.VHDX

Notes:

Now we’re simulating that you lost a disk and it’s really broken and never coming back.

You add a brand new disk that will be the replacement to the old disk.

1.13 - Verify effect on the pool and space

Steps to perform on the guest:

Get-Volume X | FT -AutoSize

Get-Volume X | Get-Partition | Get-Disk | Get-VirtualDisk | FT -AutoSize

Get-PhysicalDisk –CanPool $true | FT -AutoSize

Get-EventLog -LogName System -Source Disk -After "3/29/2014 6:20 PM” | ? EntryType -ne "Information" | FL

Get-WinEvent -LogName Microsoft-Windows-Stor* | ? { ($_.Level -lt 4) -and ($_.TimeCreated -gt "3/29/2014 6:20 PM") } | FL

Output:

PS C:\> Get-Volume X | FT –AutoSize

DriveLetter FileSystemLabel FileSystem DriveType HealthStatus SizeRemaining Size

----------- --------------- ---------- --------- ------------ ------------- ----

X NTFS Fixed Healthy 26.9 GB 27 GB

PS C:\> Get-Volume X | Get-Partition | Get-Disk | Get-VirtualDisk | FT -AutoSize

FriendlyName ResiliencySettingName OperationalStatus HealthStatus IsManualAttach Size

------------ --------------------- ----------------- ------------ -------------- ----

Space1 Mirror Incomplete Warning False 27 GB

FriendlyName OperationalStatus HealthStatus IsPrimordial IsReadOnly

------------ ----------------- ------------ ------------ ----------

Pool1 Degraded Warning False False

FriendlyName CanPool OperationalStatus HealthStatus Usage Size

------------ ------- ----------------- ------------ ----- ----

PhysicalDisk4 False OK Healthy Auto-Select 19.25 GB

PhysicalDisk-1 False Lost Communication Warning Auto-Select 19.25 GB

PhysicalDisk3 False OK Healthy Auto-Select 19.25 GB

PS C:\> Get-PhysicalDisk –CanPool $true | FT -AutoSize

FriendlyName CanPool OperationalStatus HealthStatus Usage Size

------------ ------- ----------------- ------------ ----- ----

PhysicalDisk1 True OK Healthy Auto-Select 20 GB

PS C:\> Get-EventLog -LogName System -Source Disk -After "3/29/2014 6:20 PM” | ? EntryType -ne "Information" | FL

Index : 14813

EntryType : Warning

InstanceId : 2147745949

Message : Disk 1 has been surprise removed.

Category : (0)

CategoryNumber : 0

ReplacementStrings : {\DR16, 1}

Source : disk

TimeGenerated : 3/29/2014 6:24:13 PM

TimeWritten : 3/29/2014 6:24:13 PM

UserName :

PS C:\> Get-WinEvent -LogName Microsoft-Windows-Stor* | ? { ($_.Level -lt 4) -and ($_.TimeCreated -gt "3/29/2014 6:20 PM") } | FL

TimeCreated : 3/29/2014 6:24:13 PM

ProviderName : Microsoft-Windows-StorageSpaces-Driver

Id : 304

Message : One or more drives hosting data for storage space {1874345F-B794-11E3-80B7-00155D010205} have failed or

are missing. As a result, at least one copy of data is not available. However, at least one copy of

data is still available. Return Code: STATUS_SUCCESS

Notes:

As we saw previously, we survive the failure of a single disk just fine. Volume is healthy, but the virtual disk and pool are degraded.

We note that there is a brand new disk available that can be pooled, but Storage Spaces will not automatically attempt to use it.

1.14 - Retire old disk, add new disk, repair and verify

Steps to perform on the guest:

Get-PhysicalDisk | ? OperationalStatus –eq “Lost Communication” | Set-PhysicalDisk –Usage Retired

Add-PhysicalDisk –StoragePoolFriendlyName Pool1 -PhysicalDisks (Get-PhysicalDisk –CanPool $True)

Repair-VirtualDisk Space1

Get-Volume X | FT -AutoSize

Get-Volume X | Get-Partition | Get-Disk | Get-VirtualDisk | FT -AutoSize

Get-StoragePool Pool1 | Get-PhysicalDisk | FT –AutoSize

Remove-PhysicalDisk -StoragePoolFriendlyName Pool1 -PhysicalDisks (Get-PhysicalDisk | ? OperationalStatus -eq "Lost Communication" )

Get-StoragePool Pool1 | Get-PhysicalDisk | FT –AutoSize

Output:

PS C:\> Get-PhysicalDisk | ? OperationalStatus –eq “Lost Communication” | Set-PhysicalDisk –Usage Retired

PS C:\> Add-PhysicalDisk –StoragePoolFriendlyName Pool1 -PhysicalDisks (Get-PhysicalDisk –CanPool $True)

PS C:\> Repair-VirtualDisk Space1

PS C:\> Get-Volume X | FT -AutoSize

DriveLetter FileSystemLabel FileSystem DriveType HealthStatus SizeRemaining Size

----------- --------------- ---------- --------- ------------ ------------- ----

X NTFS Fixed Healthy 26.9 GB 27 GB

PS C:\> Get-Volume X | Get-Partition | Get-Disk | Get-VirtualDisk | FT -AutoSize

FriendlyName ResiliencySettingName OperationalStatus HealthStatus IsManualAttach Size

------------ --------------------- ----------------- ------------ -------------- ----

Space1 Mirror OK Healthy False 27 GB

FriendlyName OperationalStatus HealthStatus IsPrimordial IsReadOnly

------------ ----------------- ------------ ------------ ----------

Pool1 Degraded Warning False False

FriendlyName CanPool OperationalStatus HealthStatus Usage Size

------------ ------- ----------------- ------------ ----- ----

PhysicalDisk4 False OK Healthy Auto-Select 19.25 GB

PhysicalDisk1 False OK Healthy Auto-Select 19.25 GB

PhysicalDisk3 False OK Healthy Auto-Select 19.25 GB

PS C:\> Get-StoragePool Pool1 | Get-PhysicalDisk | FT -AutoSize

FriendlyName CanPool OperationalStatus HealthStatus Usage Size

------------ ------- ----------------- ------------ ----- ----

PhysicalDisk4 False OK Healthy Auto-Select 19.25 GB

PhysicalDisk-1 False Lost Communication Warning Retired 19.25 GB

PhysicalDisk1 False OK Healthy Auto-Select 19.25 GB

PhysicalDisk3 False OK Healthy Auto-Select 19.25 GB

PS C:\> Remove-PhysicalDisk -StoragePoolFriendlyName Pool1 -PhysicalDisks (Get-PhysicalDisk | ? OperationalStatus -eq "Lost Communication" )

Confirm

Are you sure you want to perform this action?

Removing a physical disk will cause problems with the fault tolerance capabilities of the following storage pool:"Pool1".

[Y] Yes [A] Yes to All [N] No [L] No to All [S] Suspend [?] Help (default is "Y"):

PS C:\> Get-StoragePool Pool1 | Get-PhysicalDisk | FT -AutoSize

FriendlyName CanPool OperationalStatus HealthStatus Usage Size

------------ ------- ----------------- ------------ ----- ----

PhysicalDisk4 False OK Healthy Auto-Select 19.25 GB

PhysicalDisk1 False OK Healthy Auto-Select 19.25 GB

PhysicalDisk3 False OK Healthy Auto-Select 19.25 GB

FriendlyName OperationalStatus HealthStatus IsPrimordial IsReadOnly

------------ ----------------- ------------ ------------ ----------

Pool1 OK Healthy False False

Notes:

After we add the new disk to the pool and command a repair, things go back to normal.

The old disk can either be retired (remains showing in the pool) or it can be completely removed from the pool.

1.15 - Add the old disk back

Steps to perform on the host:

Add-VMHardDiskDrive –VMName VM2 -ControllerType SCSI -ControllerNumber 0 -ControllerLocation 3 –Path D:\demo\disk1.VHDX

Get-VMHardDiskDrive -VMName VM2

Output:

PS C:\> Add-VMHardDiskDrive –VMName VM2 -ControllerType SCSI -ControllerNumber 0 -ControllerLocation 3 –Path D:\demo\disk1.VHDX

PS C:\> Get-VMHardDiskDrive -VMName VM2

VMName ControllerType ControllerNumber ControllerLocation DiskNumber Path

------ -------------- ---------------- ------------------ ---------- ----

VM2 IDE 0 0 c:\demo\v2.vhdx

VM2 SCSI 0 0 D:\demo\disk1r.VHDX

VM2 SCSI 0 1 D:\demo\disk2.VHDX

VM2 SCSI 0 2 D:\demo\disk3.VHDX

VM2 SCSI 0 3 D:\demo\disk1.VHDX

Notes:

Simulating an odd but plausible situation. You have cleaned up the pool and repaired everything after a disk went bad, but someone put that old disk back in.

Since you’ve already removed it from the pool, what is the result?

1.16 - Verify effect on the pool, reset disk

Steps to perform on the guest:

Get-PhysicalDisk | FT –AutoSize

Get-StoragePool Pool1 | Get-PhysicalDisk

Get-PhysicalDisk | ? OperationalStatus -eq "Unrecognized Metadata" | Reset-PhysicalDisk

Get-PhysicalDisk | FT -AutoSize

Get-EventLog -LogName System -Source Disk -After "3/29/2014 6:20 PM” | ? EntryType -ne "Information" | FL

Get-WinEvent -LogName Microsoft-Windows-Stor* | ? { ($_.Level -lt 4) -and ($_.TimeCreated -gt "3/29/2014 6:20 PM") } | FL

Output:

PS C:\> Get-PhysicalDisk | FT -AutoSize

FriendlyName CanPool OperationalStatus HealthStatus Usage Size

------------ ------- ----------------- ------------ ----- ----

PhysicalDisk0 False OK Healthy Auto-Select 32 GB

PhysicalDisk4 False OK Healthy Auto-Select 19.25 GB

PhysicalDisk5 False Unrecognized MetadataUnhealthy Unknown 20 GB

PhysicalDisk1 False OK Healthy Auto-Select 19.25 GB

PhysicalDisk3 False OK Healthy Auto-Select 19.25 GB

PS C:\> Get-StoragePool Pool1 | Get-PhysicalDisk

FriendlyName CanPool OperationalStatus HealthStatus Usage Size

------------ ------- ----------------- ------------ ----- ----

PhysicalDisk4 False OK Healthy Auto-Select 19.25 GB

PhysicalDisk1 False OK Healthy Auto-Select 19.25 GB

PhysicalDisk3 False OK Healthy Auto-Select 19.25 GB

PS C:\> Get-PhysicalDisk | ? OperationalStatus -eq "Unrecognized Metadata" | Reset-PhysicalDisk

PS C:\> Get-PhysicalDisk | FT -AutoSize

FriendlyName CanPool OperationalStatus HealthStatus Usage Size

------------ ------- ----------------- ------------ ----- ----

PhysicalDisk0 False OK Healthy Auto-Select 32 GB

PhysicalDisk4 False OK Healthy Auto-Select 19.25 GB

PhysicalDisk5 True OK Healthy Auto-Select 20 GB

PhysicalDisk1 False OK Healthy Auto-Select 19.25 GB

PhysicalDisk3 False OK Healthy Auto-Select 19.25 GB

PS C:\> Get-WinEvent -LogName Microsoft-Windows-Stor* | ? { ($_.Level -lt 4) -and ($_.TimeCreated -gt "3/29/2014 7:00PM") } | FL

TimeCreated : 3/29/2014 7:10:53 PM

ProviderName : Microsoft-Windows-StorageSpaces-Driver

Id : 201

Message : Physical drive {5A464324-FBA5-A176-5DE8-DE8FE63E3951} has invalid meta-data. Resetting the health

status, using command line or GUI, might bring the physical drive to the primordial pool. Return Code:

STATUS_SUCCESS

Notes:

So you can see that the old disk comes back as unhealthy, with a status of “unrecognized metadata”.

It has already been removed from the pool and contains metadata that is inconsistent with the current state of the pools and virtual disks in the system.

There is an interesting event log in the Storage Spaces event highlighting the situation. The recommendation is to reset the health status.

We go ahead and reset it as suggested, making it healthy again and available to add back to the pool if necessary (that last portion is not shown).

2 - Three-way mirror with 5 disks

Part 2 of this post focuses on testing the resiliency of a 3-way mirrored virtual disk on a storage pool with 5 physical disks. It follows the same format as Part 1.

We are going to start fresh with a new set of disks. At this point, if you’re following along on your system, you should:

Inside the VM, remove all virtual disks and pools from Part 1.
On the host, remove the data VHDX files from the VM, then delete them.
Inside the VM, clear the System log and the Storage Spaces.

2.1 - Attach 5 disks to a VM

Steps to perform on the host:

1..5 | % { New-VHD –Path D:\demo\disk$_.VHDX –Dynamic –SizeBytes 20GB }

1..5 | % { Add-VMHardDiskDrive –VMName VM2 –Path D:\demo\disk$_.VHDX -ControllerType SCSI}

Get-VMHardDiskDrive -VMName VM2

Output:

PS C:\> 1..5 | % { New-VHD –Path D:\demo\disk$_.VHDX –Dynamic –SizeBytes 20GB } | Out-Null

PS C:\> 1..5 | % { Add-VMHardDiskDrive –VMName VM2 –Path D:\demo\disk$_.VHDX -ControllerType SCSI}

PS C:\> Get-VMHardDiskDrive -VMName VM2

VMName ControllerType ControllerNumber ControllerLocation DiskNumber Path

------ -------------- ---------------- ------------------ ---------- ----

VM2 IDE 0 0 c:\demo\v2.vhdx

VM2 SCSI 0 0 D:\demo\disk1.VHDX

VM2 SCSI 0 1 D:\demo\disk2.VHDX

VM2 SCSI 0 2 D:\demo\disk3.VHDX

VM2 SCSI 0 3 D:\demo\disk4.VHDX

VM2 SCSI 0 4 D:\demo\disk5.VHDX

Notes:

Nothing much. Just your regular VM with 1 IDE boot disk and 5 SCSI data disks.

2.2 - Create pool and 3-way mirrored space

Steps to perform on the guest:

$s = Get-StorageSubSystem -FriendlyName *Spaces*

$d = Get-PhysicalDisk -CanPool $true
New-StoragePool -FriendlyName Pool1 -StorageSubSystemFriendlyName $s.FriendlyName -PhysicalDisks $d
$p = Get-StoragePool Pool1
Set-ResiliencySetting -StoragePool $p -Name Mirror -NumberofColumnsDefault 1 -NumberOfDataCopiesDefault 3
$p | New-VirtualDisk -FriendlyName Space1 -ResiliencySettingName Mirror –UseMaximumSize

Output:

PS C:\> $s = Get-StorageSubSystem -FriendlyName *Spaces*

PS C:\> $d = Get-PhysicalDisk -CanPool $true

PS C:\> New-StoragePool -FriendlyName Pool1 -StorageSubSystemFriendlyName $s.FriendlyName -PhysicalDisks $d

FriendlyName OperationalStatus HealthStatus IsPrimordial IsReadOnly

------------ ----------------- ------------ ------------ ----------

Pool1 OK Healthy False False

PS C:\> $p = Get-StoragePool Pool1

PS C:\> Set-ResiliencySetting -StoragePool $p -Name Mirror -NumberofColumnsDefault 1 -NumberOfDataCopiesDefault 3

PS C:\> $p | New-VirtualDisk -FriendlyName Space1 -ResiliencySettingName Mirror –UseMaximumSize

FriendlyName ResiliencySettingNa OperationalStatus HealthStatus IsManualAttach Size

------------ ------------------- ----------------- ------------ -------------- ----

Space1 Mirror OK Healthy False 30 GB

PS C:\> $L ="X”

PS C:\> $N = (Get-VirtualDisk -FriendlyName Space1 | Get-Disk).Number

PS C:\> Set-Disk -Number $N -IsReadOnly 0

PS C:\> Set-Disk -Number $N -IsOffline 0

PS C:\> Initialize-Disk -Number $N -PartitionStyle MBR

PS C:\> New-Partition -DiskNumber $N -DriveLetter $L -UseMaximumSize

Disk Number: 6

PartitionNumber DriveLetter Offset Size Type

--------------- ----------- ------ ---- ----

1 X 1048576 30 GB Logical

PS C:\> Start-Sleep –Seconds 10

PS C:\> Initialize-Volume -DriveLetter $L -FileSystem NTFS -Confirm:$false

DriveLetter FileSystemLabel FileSystem DriveType HealthStatus SizeRemaining Size

----------- --------------- ---------- --------- ------------ ------------- ----

X NTFS Fixed Healthy 29.9 GB 30 GB

Notes:

Created a storage pool and storage space. Resulting disk is initialized and mounted as volume X:.

2.3 - Verify health of the pool and space

Steps to perform on the guest:

Get-Volume X | FT -AutoSize

Get-Volume X | Get-Partition | Get-Disk | Get-VirtualDisk | FT -AutoSize

Output:

PS C:\> Get-Volume X | FT -AutoSize

DriveLetter FileSystemLabel FileSystem DriveType HealthStatus SizeRemaining Size

----------- --------------- ---------- --------- ------------ ------------- ----

X NTFS Fixed Healthy 29.9 GB 30 GB

PS C:\> Get-Volume X | Get-Partition | Get-Disk | Get-VirtualDisk | FT -AutoSize

FriendlyName ResiliencySettingName OperationalStatus HealthStatus IsManualAttach Size

------------ --------------------- ----------------- ------------ -------------- ----

Space1 Mirror OK Healthy False 30 GB

FriendlyName OperationalStatus HealthStatus IsPrimordial IsReadOnly

------------ ----------------- ------------ ------------ ----------

Pool1 OK Healthy False False

FriendlyName CanPool OperationalStatus HealthStatus Usage Size

------------ ------- ----------------- ------------ ----- ----

PhysicalDisk3 False OK Healthy Auto-Select 19.25 GB

PhysicalDisk1 False OK Healthy Auto-Select 19.25 GB

PhysicalDisk4 False OK Healthy Auto-Select 19.25 GB

PhysicalDisk2 False OK Healthy Auto-Select 19.25 GB

PhysicalDisk5 False OK Healthy Auto-Select 19.25 GB

Notes:

All is healthy: volume, storage space (virtual disk), storage pool and physical disks.

2.4 - Remove 2 disks

Steps to perform on the host:

Remove-VMHardDiskDrive -VMName VM2 -ControllerType SCSI -ControllerNumber 0 -ControllerLocation 0

Remove-VMHardDiskDrive -VMName VM2 -ControllerType SCSI -ControllerNumber 0 -ControllerLocation 1

Get-VMHardDiskDrive -VMName VM2

Output:

PS C:\> Remove-VMHardDiskDrive -VMName VM2 -ControllerType SCSI -ControllerNumber 0 -ControllerLocation 0

PS C:\> Remove-VMHardDiskDrive -VMName VM2 -ControllerType SCSI -ControllerNumber 0 -ControllerLocation 1

PS C:\> Get-VMHardDiskDrive -VMName VM2

VMName ControllerType ControllerNumber ControllerLocation DiskNumber Path

------ -------------- ---------------- ------------------ ---------- ----

VM2 IDE 0 0 c:\demo\v2.vhdx

VM2 SCSI 0 2 D:\demo\disk3.VHDX

VM2 SCSI 0 3 D:\demo\disk4.VHDX

VM2 SCSI 0 4 D:\demo\disk5.VHDX

Notes:

Removed 2 of the 5 SCSI disks for the VM.

Since this is a 3-way mirror, we’re going to hit it right away with a double disk failure. But it should be able to survive that.

2.5 - Verify effect on the pool and space

Steps to perform on the guest:

Get-Volume X | FT -AutoSize

Get-Volume X | Get-Partition | Get-Disk | Get-VirtualDisk | FT -AutoSize

Get-EventLog -LogName System -Source Disk | ? EntryType -ne "Information" | FL

Get-WinEvent -LogName Microsoft-Windows-Stor* | ? Level -lt 4 | FL

Output:

PS C:\> Get-Volume X | FT -AutoSize

DriveLetter FileSystemLabel FileSystem DriveType HealthStatus SizeRemaining Size

----------- --------------- ---------- --------- ------------ ------------- ----

X NTFS Fixed Healthy 29.9 GB 30 GB

PS C:\> Get-Volume X | Get-Partition | Get-Disk | Get-VirtualDisk | FT -AutoSize

FriendlyName ResiliencySettingName OperationalStatus HealthStatus IsManualAttach Size

------------ --------------------- ----------------- ------------ -------------- ----

Space1 Mirror Incomplete Warning False 30 GB

FriendlyName OperationalStatus HealthStatus IsPrimordial IsReadOnly

------------ ----------------- ------------ ------------ ----------

Pool1 Degraded Warning False False

FriendlyName CanPool OperationalStatus HealthStatus Usage Size

------------ ------- ----------------- ------------ ----- ----

PhysicalDisk3 False OK Healthy Auto-Select 19.25 GB

PhysicalDisk-1 False Lost Communication Warning Auto-Select 19.25 GB

PhysicalDisk4 False OK Healthy Auto-Select 19.25 GB

PhysicalDisk-1 False Lost Communication Warning Auto-Select 19.25 GB

PhysicalDisk5 False OK Healthy Auto-Select 19.25 GB

PS C:\> Get-EventLog -LogName System -Source Disk | ? EntryType -ne "Information" | FL

Index : 14841

EntryType : Warning

InstanceId : 2147745949

Message : Disk 2 has been surprise removed.

Category : (0)

CategoryNumber : 0

ReplacementStrings : {\DR22, 2}

Source : disk

TimeGenerated : 3/29/2014 8:38:11 PM

TimeWritten : 3/29/2014 8:38:11 PM

UserName :

Index : 14840

EntryType : Warning

InstanceId : 2147745949

Message : Disk 1 has been surprise removed.

Category : (0)

CategoryNumber : 0

ReplacementStrings : {\DR21, 1}

Source : disk

TimeGenerated : 3/29/2014 8:38:10 PM

TimeWritten : 3/29/2014 8:38:10 PM

UserName :

PS C:\> Get-WinEvent -LogName Microsoft-Windows-Stor* | ? Level -lt 4 | FL

TimeCreated : 3/29/2014 8:38:11 PM

ProviderName : Microsoft-Windows-StorageSpaces-Driver

Id : 304

Message : One or more drives hosting data for storage space {1874353C-B794-11E3-80B7-00155D010205} have failed or

are missing. As a result, at least one copy of data is not available. However, at least one copy of

data is still available. Return Code: STATUS_SUCCESS

TimeCreated : 3/29/2014 8:38:10 PM

ProviderName : Microsoft-Windows-StorageSpaces-Driver

Id : 304

Message : One or more drives hosting data for storage space {1874353C-B794-11E3-80B7-00155D010205} have failed or

are missing. As a result, at least one copy of data is not available. However, at least one copy of

data is still available. Return Code: STATUS_SUCCESS

TimeCreated : 3/29/2014 8:31:56 PM

ProviderName : Microsoft-Windows-StorageSpaces-Driver

Id : 103

Message : The capacity consumption of the storage pool {18743521-B794-11E3-80B7-00155D010205} has exceeded the

threshold limit set on the pool. Return Code: STATUS_SUCCESS

Notes:

As expected, the 3-way mirrored space was able to survive the double disk failure. The storage space is now incomplete and the storage pool is degraded, but the volume is still healthy.

Applications won’t see the event, but again there are many alarm bells ringing in the event logs. You better act fast because the next failure will be fatal…

2.6 - Add back the removed disks

Steps to perform on the host:

Add-VMHardDiskDrive -VMName VM2 -ControllerType SCSI -ControllerNumber 0 -ControllerLocation 0 –Path D:\demo\disk1.VHDX

Add-VMHardDiskDrive -VMName VM2 -ControllerType SCSI -ControllerNumber 0 -ControllerLocation 1 –Path D:\demo\disk2.VHDX

Get-VMHardDiskDrive -VMName VM2

Output:

PS C:\> Add-VMHardDiskDrive -VMName VM2 -ControllerType SCSI -ControllerNumber 0 -ControllerLocation 0 –Path D:\demo\disk1.VHDX

PS C:\> Add-VMHardDiskDrive -VMName VM2 -ControllerType SCSI -ControllerNumber 0 -ControllerLocation 1 –Path D:\demo\disk2.VHDX

PS C:\> Get-VMHardDiskDrive -VMName VM2

VMName ControllerType ControllerNumber ControllerLocation DiskNumber Path

------ -------------- ---------------- ------------------ ---------- ----

VM2 IDE 0 0 c:\demo\v2.vhdx

VM2 SCSI 0 0 D:\demo\disk1.VHDX

VM2 SCSI 0 1 D:\demo\disk2.VHDX

VM2 SCSI 0 2 D:\demo\disk3.VHDX

VM2 SCSI 0 3 D:\demo\disk4.VHDX

VM2 SCSI 0 4 D:\demo\disk5.VHDX

Notes:

Added the two VHDX files back to restore the VM to the original state.

2.7 - Verify that the pool and space recover

Steps to perform on the guest:

Get-Volume X | FT -AutoSize

Get-Volume X | Get-Partition | Get-Disk | Get-VirtualDisk | FT -AutoSize

Get-WinEvent -LogName Microsoft-Windows-Stor* | ? Message -like "*repair*" | FL

Output:

PS C:\> Get-Volume X | FT -AutoSize

DriveLetter FileSystemLabel FileSystem DriveType HealthStatus SizeRemaining Size

----------- --------------- ---------- --------- ------------ ------------- ----

X NTFS Fixed Healthy 29.9 GB 30 GB

PS C:\> Get-Volume X | Get-Partition | Get-Disk | Get-VirtualDisk | FT -AutoSize

FriendlyName ResiliencySettingName OperationalStatus HealthStatus IsManualAttach Size

------------ --------------------- ----------------- ------------ -------------- ----

Space1 Mirror OK Healthy False 30 GB

FriendlyName OperationalStatus HealthStatus IsPrimordial IsReadOnly

------------ ----------------- ------------ ------------ ----------

Pool1 OK Healthy False False

FriendlyName CanPool OperationalStatus HealthStatus Usage Size

------------ ------- ----------------- ------------ ----- ----

PhysicalDisk3 False OK Healthy Auto-Select 19.25 GB

PhysicalDisk1 False OK Healthy Auto-Select 19.25 GB

PhysicalDisk4 False OK Healthy Auto-Select 19.25 GB

PhysicalDisk2 False OK Healthy Auto-Select 19.25 GB

PhysicalDisk5 False OK Healthy Auto-Select 19.25 GB

PS C:\> Get-WinEvent -LogName Microsoft-Windows-Stor* | ? Message -like "*repair*" | FL

TimeCreated : 3/29/2014 8:45:51 PM

ProviderName : Microsoft-Windows-StorageSpaces-Driver

Id : 308

Message : A repair attempt for storage space {1874353C-B794-11E3-80B7-00155D010205} was initiated by the driver.

Return Code: STATUS_SUCCESS

TimeCreated : 3/29/2014 8:45:47 PM

ProviderName : Microsoft-Windows-StorageSpaces-Driver

Id : 308

Message : A repair attempt for storage space {1874353C-B794-11E3-80B7-00155D010205} was initiated by the driver.

Return Code: STATUS_SUCCESS

TimeCreated : 3/29/2014 8:31:56 PM

ProviderName : Microsoft-Windows-StorageSpaces-Driver

Id : 308

Message : A repair attempt for storage space {1874353C-B794-11E3-80B7-00155D010205} was initiated by the driver.

Return Code: STATUS_SUCCESS

Notes:

Storage Spaces, seeing the drives coming back, attempts a repair and succeeds.

After that, all is well again: virtual disks, storage pools and physical disks all report as healthy.

This shows that we can survive two disk failures in a way that is completely transparent to the applications.

2.8 - Remove 3 disks

Steps to perform on the host:

Remove-VMHardDiskDrive -VMName VM2 -ControllerType SCSI -ControllerNumber 0 -ControllerLocation 0

Remove-VMHardDiskDrive -VMName VM2 -ControllerType SCSI -ControllerNumber 0 -ControllerLocation 1

Remove-VMHardDiskDrive -VMName VM2 -ControllerType SCSI -ControllerNumber 0 -ControllerLocation 2

Get-VMHardDiskDrive -VMName VM2

Output:

PS C:\> Remove-VMHardDiskDrive -VMName VM2 -ControllerType SCSI -ControllerNumber 0 -ControllerLocation 0

PS C:\> Remove-VMHardDiskDrive -VMName VM2 -ControllerType SCSI -ControllerNumber 0 -ControllerLocation 1

PS C:\> Remove-VMHardDiskDrive -VMName VM2 -ControllerType SCSI -ControllerNumber 0 -ControllerLocation 2

PS C:\> Get-VMHardDiskDrive -VMName VM2

VMName ControllerType ControllerNumber ControllerLocation DiskNumber Path

------ -------------- ---------------- ------------------ ---------- ----

VM2 IDE 0 0 c:\demo\v2.vhdx

VM2 SCSI 0 3 D:\demo\disk4.VHDX

VM2 SCSI 0 4 D:\demo\disk5.VHDX

Notes:

Removing 3 of the 5 disks now. There are now only two surviving disks. That should be too much to bear.

2.9 - Verify effect on the pool and space

Steps to perform on the guest:

Get-Volume X | FT –AutoSize

Get-VirtualDisk Space1 | FT -AutoSize

Get-VirtualDisk Space1 | Get-StoragePool | FT -AutoSize

Get-VirtualDisk Space1 | Get-PhysicalDisk | FT -AutoSize

Get-EventLog -LogName System -Source Disk -After "3/29/2014 8:40 PM” | ? EntryType -ne "Information" | FT TimeGenerated, Source, Index, EventID, EntryType, Message -AutoSize

Get-WinEvent -LogName Microsoft-Windows-Stor* | ? { ($_.Level -lt 4) -and ($_.TimeCreated -gt "3/29/2014 8:40 PM") } | FL

Output:

PS C:\> Get-Volume X | FT –AutoSize

Get-Volume : No MSFT_Volume objects found with property 'DriveLetter' equal to 'X'. Verify the value of the property and retry.

At line:1 char:1

+ Get-Volume X | FT –AutoSize

+ ~~~~~~~~~~~~

+ CategoryInfo : ObjectNotFound: (X:Char) [Get-Volume], CimJobException

+ FullyQualifiedErrorId : CmdletizationQuery_NotFound_DriveLetter,Get-Volume

PS C:\> Get-VirtualDisk Space1 | FT -AutoSize

FriendlyName ResiliencySettingName OperationalStatus HealthStatus IsManualAttach Size

------------ --------------------- ----------------- ------------ -------------- ----

Space1 Mirror Detached Unhealthy False 30 GB

PS C:\> Get-VirtualDisk Space1 | Get-StoragePool | FT -AutoSize

FriendlyName OperationalStatus HealthStatus IsPrimordial IsReadOnly

------------ ----------------- ------------ ------------ ----------

Pool1 Read-only Unhealthy False False

PS C:\> Get-VirtualDisk Space1 | Get-PhysicalDisk | FT -AutoSize

FriendlyName CanPool OperationalStatus HealthStatus Usage Size

------------ ------- ----------------- ------------ ----- ----

PhysicalDisk-1 False Lost Communication Warning Auto-Select 19.25 GB

PhysicalDisk4 False OK Healthy Auto-Select 19.25 GB

PhysicalDisk-1 False Lost Communication Warning Auto-Select 19.25 GB

PhysicalDisk5 False OK Healthy Auto-Select 19.25 GB

PS C:\> Get-EventLog -LogName System -Source Disk -After "3/29/2014 8:40 PM” | ? EntryType -ne "Information" | FT TimeGenerated, Source, Index, EventID, EntryType, Message -AutoSize

TimeGenerated Source Index EventID EntryType Message

------------- ------ ----- ------- --------- -------

3/29/2014 8:50:21 PM disk 14847 157 Warning Disk 6 has been surprise removed.

3/29/2014 8:50:21 PM disk 14846 157 Warning Disk 3 has been surprise removed.

3/29/2014 8:50:20 PM disk 14845 157 Warning Disk 2 has been surprise removed.

3/29/2014 8:50:19 PM disk 14844 157 Warning Disk 1 has been surprise removed.

PS C:\> Get-WinEvent -LogName Microsoft-Windows-Stor* | ? { ($_.Level -lt 4) -and ($_.TimeCreated -gt "3/29/2014 8:40 PM") } | FL

TimeCreated : 3/29/2014 8:50:21 PM

ProviderName : Microsoft-Windows-StorageSpaces-Driver

Id : 304

Message : One or more drives hosting data for storage space {1874353C-B794-11E3-80B7-00155D010205} have failed or

are missing. As a result, at least one copy of data is not available. However, at least one copy of

data is still available. Return Code: STATUS_SUCCESS

TimeCreated : 3/29/2014 8:50:21 PM

ProviderName : Microsoft-Windows-StorageSpaces-Driver

Id : 203

Message : Physical drive {06724EEA-84C4-283C-1EF9-F26D195E0D7A} failed an IO operation. Return Code: This device

does not exist.. Additional related events may be found in the System event log for Disk 3.

This drive may need to be replaced. To view its reliability counters, run this command in PowerShell:

Get-PhysicalDisk | ?{ $_.ObjectId -Match "{06724EEA-84C4-283C-1EF9-F26D195E0D7A}" } |

Get-StorageReliabilityCounter

This drive may be located using the following information:

Drive Manufacturer: Msft

Drive Model Number: Virtual Disk

Drive Serial Number:

More information can be obtained using this PowerShell command:

Get-PhysicalDisk | ?{ $_.ObjectId -Match "{06724EEA-84C4-283C-1EF9-F26D195E0D7A}" }

If this drive is in an enclosure, it may be located using the following information:

Enclosure Manufacturer:

Enclosure Model Number:

Enclosure Serial Number:

Enclosure Slot: -1

It may also be located by running this command in PowerShell:

Get-PhysicalDisk | ?{ $_.ObjectId -Match "{06724EEA-84C4-283C-1EF9-F26D195E0D7A}" } |

Enable-PhysicalDiskIndication

TimeCreated : 3/29/2014 8:50:21 PM

ProviderName : Microsoft-Windows-StorageSpaces-Driver

Id : 303

Message : Drives hosting data for storage space {1874353C-B794-11E3-80B7-00155D010205} have failed or are

missing. As a result, no copy of data is available. Return Code: This device does not exist.

TimeCreated : 3/29/2014 8:50:20 PM

ProviderName : Microsoft-Windows-StorageSpaces-Driver

Id : 304

Message : One or more drives hosting data for storage space {1874353C-B794-11E3-80B7-00155D010205} have failed or

are missing. As a result, at least one copy of data is not available. However, at least one copy of

data is still available. Return Code: STATUS_SUCCESS

TimeCreated : 3/29/2014 8:50:19 PM

ProviderName : Microsoft-Windows-StorageSpaces-Driver

Id : 304

Message : One or more drives hosting data for storage space {1874353C-B794-11E3-80B7-00155D010205} have failed or

are missing. As a result, at least one copy of data is not available. However, at least one copy of

data is still available. Return Code: STATUS_SUCCESS

TimeCreated : 3/29/2014 8:45:49 PM

ProviderName : Microsoft-Windows-StorageSpaces-Driver

Id : 304

Message : One or more drives hosting data for storage space {1874353C-B794-11E3-80B7-00155D010205} have failed or

are missing. As a result, at least one copy of data is not available. However, at least one copy of

data is still available. Return Code: STATUS_SUCCESS

TimeCreated : 3/29/2014 8:45:46 PM

ProviderName : Microsoft-Windows-StorageSpaces-Driver

Id : 304

Message : One or more drives hosting data for storage space {1874353C-B794-11E3-80B7-00155D010205} have failed or

are missing. As a result, at least one copy of data is not available. However, at least one copy of

data is still available. Return Code: STATUS_SUCCESS

Notes:

As expected, the volume is gone and any applications running here would have stopped.

The virtual disk is detached, the volume is dismounted, the pool is unhealthy and 3 physical disks are missing.

Again we get more disk failures and Storage Spaces failures as I would expect. Not sure why that happens.

2.10 - Add back the 3 removed disks

Steps to perform on the host:

Add-VMHardDiskDrive -VMName VM2 -ControllerType SCSI -ControllerNumber 0 -ControllerLocation 5 –Path D:\demo\disk1.VHDX

Add-VMHardDiskDrive -VMName VM2 -ControllerType SCSI -ControllerNumber 0 -ControllerLocation 6 –Path D:\demo\disk2.VHDX

Add-VMHardDiskDrive -VMName VM2 -ControllerType SCSI -ControllerNumber 0 -ControllerLocation 7 –Path D:\demo\disk3.VHDX

Get-VMHardDiskDrive -VMName VM2

Output:

PS C:\> Add-VMHardDiskDrive -VMName VM2 -ControllerType SCSI -ControllerNumber 0 -ControllerLocation 5 –Path D:\demo\disk1.VHDX

PS C:\> Add-VMHardDiskDrive -VMName VM2 -ControllerType SCSI -ControllerNumber 0 -ControllerLocation 6 –Path D:\demo\disk2.VHDX

PS C:\> Add-VMHardDiskDrive -VMName VM2 -ControllerType SCSI -ControllerNumber 0 -ControllerLocation 7 –Path D:\demo\disk3.VHDX

PS C:\> Get-VMHardDiskDrive -VMName VM2

VMName ControllerType ControllerNumber ControllerLocation DiskNumber Path

------ -------------- ---------------- ------------------ ---------- ----

VM2 IDE 0 0 c:\demo\v2.vhdx

VM2 SCSI 0 3 D:\demo\disk4.VHDX

VM2 SCSI 0 4 D:\demo\disk5.VHDX

VM2 SCSI 0 5 D:\demo\disk1.VHDX

VM2 SCSI 0 6 D:\demo\disk2.VHDX

VM2 SCSI 0 7 D:\demo\disk3.VHDX

Notes:

Simulating putting the 3 drives back in. This should fix things.

Just for fun, adding them in different controller locations, the equivalent of putting it back in a drive bay different than where it was originally.

Storage Spaces relies on the metadata stored in the disk itself, so this should not make any difference.

2.11 - Verify that the pool and space recover

Steps to perform on the guest:

Get-Volume X | FT -AutoSize

Get-Volume X | Get-Partition | Get-Disk | Get-VirtualDisk | FT -AutoSize

Get-WinEvent -LogName Microsoft-Windows-Stor* | ? { ($_.Message -like "*repair*") -and ($_.TimeCreated -gt "3/29/2014 9:00 PM") } | FL

Output:

PS C:\> Get-Volume X | FT -AutoSize

DriveLetter FileSystemLabel FileSystem DriveType HealthStatus SizeRemaining Size

----------- --------------- ---------- --------- ------------ ------------- ----

X NTFS Fixed Healthy 29.9 GB 30 GB

PS C:\> Get-Volume X | Get-Partition | Get-Disk | Get-VirtualDisk | FT -AutoSize

FriendlyName ResiliencySettingName OperationalStatus HealthStatus IsManualAttach Size

------------ --------------------- ----------------- ------------ -------------- ----

Space1 Mirror OK Healthy False 30 GB

FriendlyName OperationalStatus HealthStatus IsPrimordial IsReadOnly

------------ ----------------- ------------ ------------ ----------

Pool1 OK Healthy False False

FriendlyName CanPool OperationalStatus HealthStatus Usage Size

------------ ------- ----------------- ------------ ----- ----

PhysicalDisk6 False OK Healthy Auto-Select 19.25 GB

PhysicalDisk1 False OK Healthy Auto-Select 19.25 GB

PhysicalDisk4 False OK Healthy Auto-Select 19.25 GB

PhysicalDisk3 False OK Healthy Auto-Select 19.25 GB

PhysicalDisk5 False OK Healthy Auto-Select 19.25 GB

PS C:\> Get-WinEvent -LogName Microsoft-Windows-Stor* | ? { ($_.Message -like "*repair*") -and ($_.TimeCreated -gt "3/29/2014 9:00 PM") } | FL

TimeCreated : 3/29/2014 9:13:14 PM

ProviderName : Microsoft-Windows-StorageSpaces-Driver

Id : 308

Message : A repair attempt for storage space {1874353C-B794-11E3-80B7-00155D010205} was initiated by the driver.

Return Code: STATUS_SUCCESS

TimeCreated : 3/29/2014 9:13:08 PM

ProviderName : Microsoft-Windows-StorageSpaces-Driver

Id : 308

Message : A repair attempt for storage space {1874353C-B794-11E3-80B7-00155D010205} was initiated by the driver.

Return Code: STATUS_SUCCESS

TimeCreated : 3/29/2014 9:13:05 PM

ProviderName : Microsoft-Windows-StorageSpaces-Driver

Id : 308

Message : A repair attempt for storage space {1874353C-B794-11E3-80B7-00155D010205} was initiated by the driver.

Return Code: STATUS_SUCCESS

Notes:

Again, without any need for repair commands, the physical disks are detected and everything goes back to a healthy state.

Keep in mind that your application probably will need to be restarted and you should follow the required recovery steps, since there was down time.

3 - Conclusion

This is it. I hope this tour of mirrored storage spaces resiliency was helpful.

The idea was to save you the trouble of finding this out for yourself, but I’m sure you’ll still be interested in experimenting with it further.

The goal was also to prepare you for an eventual disk failure. You will likely be a bit more stressed at that time, so knowing what to expect and where to look might be helpful.

I have lots of additional ideas to try like trying more columns, adding storage tiers to the mix, using parity spaces or actually using an actual bad disk to go beyond just surprise removals.

But I guess that will have to wait for another opportunity. This one is already too long as it is… :-)

For additional content related to Storage Spaces, check these:

Blog - Step-by-step for Storage Spaces Tiering in Windows Server 2012 R2
Blog - How to Configure a Clustered Storage Space in Windows Server 2012
Blog - Windows Server 2012 R2 Storage: Step-by-step with Storage Spaces, SMB Scale-Out and Shared VHDX (Virtual)
Blog - Windows Server 2012 R2 Storage: Step-by-step with Storage Spaces, SMB Scale-Out and Shared VHDX (Physical)
Wiki - Storage Spaces Frequently Asked Questions (FAQ)
Script Center - Storage Spaces Physical Disk Validation Script
Windows Server Catalog – JBODs certified for Storage Spaces
TechEd - Storage Spaces: What’s New in Windows Server 2012 R2 (includes details on the new Tiering and Write-back Cache features in Storage Spaces)

↧

Building your schedule for TechEd 2014? Here are some Storage-related sessions!

April 27, 2014, 2:01 pm

≫ Next: Demo details for my TechEd 2014 presentation on File Server Networking

≪ Previous: Step-by-Step for Mirrored Storage Spaces Resiliency using PowerShell

TechEd NorthAmerica 2014 is just a few weeks away and it’s a sold-out event now.

If you’re attending and you haven’t yet started building your schedule, you can consider yourself officially late. :-)

If you’re focused on Private Cloud Storage and High Availability, let me suggest a few Breakout Sessions, Instructor-Led Labs (ILL) and Hands-On Labs (HOL) that you should consider:

Type	Title	Presenter(s)	Date/Time
Foundational	FDN06 Transform the Datacenter: Making the Promise of Connected Clouds a Reality	Brian Hillger, Elden Christensen, Jeff Woolsey, Jeffrey Snover, Matt McSpirit	Mon 11:00 AM
Breakout	DCIM-B349 Software-Defined Storage in Windows Server 2012 R2 and Microsoft System Center 2012 R2	Elden Christensen, Hector Linares, Jose Barreto	Wed 8:30 AM
Breakout	DCIM-B354 Failover Clustering: What's New in Windows Server 2012 R2	Elden Christensen, John Marlin	Tue 1:30 PM
Breakout	DCIM-B337 File Server Networking for a Private Cloud Storage Infrastructure in Windows Server 2012 R2	Jose Barreto	Tue 3:15 PM
Breakout	DCIM-B333 Distributed File System Replication (DFSR) Scalability Enhancements in Windows Server 2012 R2	Ned Pyle	Wed 5:00 PM
Breakout	DCIM-B335 Microsoft Storage Solutions in Production Environments	Prabu Rambadran	Tue 1:30 PM
Breakout	DCIM-B364 Step-by-step to Deploying Microsoft SQL Server 2014 with Cluster Shared Volumes	Subhasish Bhattacharya	Thu 8:30 AM
Breakout	DCIM-B310 The StorSimple Approach to Solving Issues Related to Growing Data Trends	Badri Venkatachari, Marc Farley	Mon 3:00 PM
Breakout	DCIM-B357 StorSimple: Enabling Microsoft Azure Cloud Storage for Enterprise Workloads	Jai Desai , Meghan Liese	Wed 1:30 PM
Breakout	DCIM-B311 Delivering Exceptional IOPS/$ with Windows Server 2012 R2	Bryan Matthew, Tobias Klima	Mon 1:15 PM
Breakout	DCIM-B346 Best Practices for Deploying Tiered Storage Spaces in Windows Server 2012 R2	Bryan Matthew, Tobias Klima	Wed 8:30 AM
Breakout	DCIM-B338 Software-Defined Storage Solutions: Storage Control Plane Functionality in Windows Server 2012 R2 and System Center 2012 R2	Hector Linares	Mon 1:15 PM
Breakout	DCIM-B327 Cluster-in-a-Box Using Windows Server 2012 R2: Solutions and Applications	John Loveall	Tue 8:30 AM
Breakout	PCIT-B214 Using Dynamic Access Control and Rights Management for Information Protection	Nir Ben Zvi, Stan Symms	Thu 2:45 PM
Breakout	DCIM-B345 Networking Solutions in Action: How Customers Implement Software-Defined Networking Using Windows Server 2012 R2 and System Center 2012 R2	Prabu Rambadran	Tue 10:15 AM
Breakout	DCIM-B211 Service Provider Datacenter Architecture	Philip Moss	Mon 3:00 PM
Breakout	DCIM-B212 The Enterprise: Be Your Own Cloud Provider	Philip Moss	Tue 8:30 AM
Breakout	DCIM-B378 Converged Networking for Windows Server 2012 R2 Hyper-V	Don Stanwyck, Taylor Brown	Mon 12:45 PM
Breakout	DBI-B314 CAT: Microsoft SQL Server High Availability and Disaster Recovery in Windows Azure	Luis Carlos Vargas Herring	Thu 2:45 PM
Breakout	DBI-B332 AlwaysOn in Microsoft SQL Server 2014	Luis Carlos Vargas Herring	Mon 1:15 PM
Breakout	DCIM-B313 Best Practices for Virtualizing and Managing Microsoft SharePoint 2013 with Microsoft System Center 2012 R2 and Windows Server 2012 R2	Marcos Nogueira	Mon 3:00 PM
Breakout	OFC-B318 Microsoft Exchange Server 2013 SP1 High Availability and Site Resilience	Scott Schnoll	Tue 3:00 PM
Breakout	OFC-B415 Microsoft Lync Server 2013 High Availability and Disaster Recovery	Brandon Bernier, Bryan Nyce	Tue 10:15 AM
ILL	DCIM-IL200 Build Your Storage Infrastructure with Windows Server 2012 R2	Prabu Rambadran	Wed 8:30 AM
ILL	DCIM-IL200-R Build Your Storage Infrastructure with Windows Server 2012 R2	Prabu Rambadran	Wed 5:00 PM
ILL	DCIM-IL308 Windows Server 2012 R2: Introduction to Failover Clustering with Hyper-V	Symon Perriman	Mon 3:00 PM
ILL	DCIM-IL308-R Windows Server 2012 R2: Introduction to Failover Clustering with Hyper-V	Symon Perriman	Tue 1:30 PM
ILL	DCIM-IL305 Managing Your Storage with Microsoft System Center 2012 Virtual Machine Manager	Prabu Rambadran	Wed 3:15 AM
HOL	DCIM-H205 Build Your Storage Infrastructure with Windows Server 2012 R2
HOL	DCIM-H321 Windows Server 2012 R2: Introduction to Failover Clustering with Hyper-V
HOL	DCIM-H202 Introduction to Microsoft Virtualization, Part 2: Virtual Machine Management
HOL	DBI-H304 Implementing HA/DR with Microsoft SQL Server 2014 AlwaysOn Availability Groups

Get that schedule going. Looking forward to meeting you in Houston…

Disclaimers:

While I tried to get all sessions related to Windows Server and Storage, I might have missed some. Post a comment if you find more.
Sessions could still change date and time. The best thing to do is add them to your schedule builder, which will keep track of those last-minute changes.
This list includes Storage sessions related to Private Cloud and Hybrid Cloud (StorSimple), but not Public Cloud (Azure Storage).

↧

Demo details for my TechEd 2014 presentation on File Server Networking

May 15, 2014, 9:53 pm

≫ Next: How the Test-StorageHealth.ps1 script checks for Storage Cluster health

≪ Previous: Building your schedule for TechEd 2014? Here are some Storage-related sessions!

During my TechEd 2014 presentation on File Server Networking for a Private Cloud Storage Infrastructure in Windows Server 2012 R2, I did a demo showing a Scale-Out File Server Cluster. The demo consisted of showing the networking details of this cluster, looking both from the cluster perspective and the client side. I was asked after the presentation to share the details of the PowerShell script that I used. So, here it goes.

First, here is the output of the script. It’s broken into a few parts:

It starts by showing the cluster configuration (cluster nodes, cluster networks, cluster resources and cluster shared volumes)
Note that the name of the cluster itself is JOSE-S and the name of the Scale-Out File Server is JOSE-SO.
Then it shows the IP addresses on each node, plus what’s registered on DNS for the scale-out file server name.
Next, it outputs the SMB server network information, with focus on the scopes defined on the server.
Then we switch to the client view of connections, after just showing the directory from the 3 shares created. This shows that multichannel won’t engage until you have a read or a write.
Finally, after running a quick workload on the shares, it shows the full set of 24 connections (3 shares * 2 NICs * 4 connections per NIC).
The 10.1.x.100 IP addresses are associated with the cluster name used for cluster management. They don't relate directly to the Scale-Out File Server.
The scope name starting with "FE80" is the internal cluster scope used for inter-node communication.

Cluster Information

PS C:\> Get-ClusterNode -Cluster JOSE-S

Name ID State
---- -- -----
JOSE-S1 2 Up
JOSE-S2 1 Up
JOSE-S3 3 Up

PS C:\> Get-ClusterNetwork -Cluster JOSE-S

Cluster Name              State Address Role
------- ----              ----- ------- ----
JOSE-S Cluster Network 1    Up 10.1.2.0    3
JOSE-S Cluster Network 2    Up 10.1.1.0    3

PS C:\> Get-ClusterResource -Cluster JOSE-S

Name                              State OwnerGroup    ResourceType
----                              ----- ----------    ------------
Cluster Disk 1                    Online Cluster Group Physical Disk
Cluster IP Address                Online Cluster Group IP Address
Cluster IP Address 10.1.1.100     Online Cluster Group IP Address
Cluster Name                      Online Cluster Group Network Name
JOSE-SO                           Online JOSE-SO       Distributed Network Name
Scale-Out File Server (\\JOSE-SO) Online JOSE-SO       Scale Out File Server

PS C:\> Get-ClusterSharedVolume -Cluster JOSE-S

Name State Node
---- ----- ----
Cluster Disk 2 Online JOSE-S2
Cluster Disk 3 Online JOSE-S1
Cluster Disk 4 Online JOSE-S3

IP Addresses for each node, DNS for Scale-Out Name

PS C:\> Get-NetIPAddress -CimSession <node>

PSComputerName ifIndex IPAddress PrefixLength
-------------- ------- --------- ------------
JOSE-S1             12 10.1.1.100           24
JOSE-S1             12 10.1.1.2             24
JOSE-S1             14 10.1.2.100           24
JOSE-S1             14 10.1.2.2             24

PSComputerName ifIndex IPAddress PrefixLength
-------------- ------- --------- ------------
JOSE-S2 12 10.1.1.3 24
JOSE-S2 14 10.1.2.3 24

PSComputerName ifIndex IPAddress PrefixLength
-------------- ------- --------- ------------
JOSE-S3 12 10.1.1.4 24
JOSE-S3 14 10.1.2.4 24

PS C:\> Resolve-DNSName JOSE-SO

Name              Type TTL Section IPAddress
----              ---- --- ------- ---------
JOSE-SO.JOSE.TEST A    1200 Answer 10.1.1.3
JOSE-SO.JOSE.TEST A    1200 Answer 10.1.1.2
JOSE-SO.JOSE.TEST A    1200 Answer 10.1.1.4
JOSE-SO.JOSE.TEST A    1200 Answer 10.1.2.3
JOSE-SO.JOSE.TEST A    1200 Answer 10.1.2.4
JOSE-SO.JOSE.TEST A    1200 Answer 10.1.2.2

IP Addresses on each node per Scope name

PS C:\> Get-SmbServerNetworkInterface -CimSession JOSE-SO

PSComputerName ScopeName                 IPAddress
-------------- ---------                 ---------
JOSE-S1        FE80::9CBF:78F5:DFA8:803B 10.1.2.2
JOSE-S1        FE80::9CBF:78F5:DFA8:803B 10.1.1.2
JOSE-S1        JOSE-S                    10.1.2.100
JOSE-S1        JOSE-S                    10.1.1.100
JOSE-S1        JOSE-SO                   10.1.2.2
JOSE-S1        JOSE-SO                   10.1.1.2

PSComputerName ScopeName                 IPAddress
-------------- ---------                 ---------
JOSE-S2        FE80::69AF:5813:D729:CEFB 10.1.1.3
JOSE-S2        FE80::69AF:5813:D729:CEFB 10.1.2.3
JOSE-S2        JOSE-SO                   10.1.2.3
JOSE-S2        JOSE-SO                   10.1.1.3

PSComputerName ScopeName                 IPAddress
-------------- ---------                 ---------
JOSE-S3        FE80::F4BE:F77B:7EAC:6CED 10.1.1.4
JOSE-S3        FE80::F4BE:F77B:7EAC:6CED 10.1.2.4
JOSE-S3        JOSE-SO                   10.1.2.4
JOSE-S3        JOSE-SO                   10.1.1.4

Simple connection just to check directory

PS C:\> Dir \\JOSE-SO\<share>

Directory: \\JOSE-SO\Share1

Mode                LastWriteTime     Length Name
----                -------------     ------ ----
-a---          5/4/2014   5:40 PM 1717986918 testfile.dat
                                           4

Directory: \\JOSE-SO\Share2

Directory: \\JOSE-SO\Share3

SMB Connection

PS C:\> Get-SmbConnection

ServerName ShareName Dialect Redirected Credential
---------- --------- ------- ---------- ----------
JOSE-SO    Share1    3.02          True JOSE.TEST\Administrator
JOSE-SO    Share2    3.02          True JOSE.TEST\Administrator
JOSE-SO    Share3    3.02          True JOSE.TEST\Administrator

TCP connections used by the client

PS C:\> Get-NetTCPConnection -RemotePort 445

LocalAddress RemoteAddress LocalPort RemotePort
------------ ------------- --------- ----------
10.1.1.5     10.1.1.2          49453        445
10.1.1.5     10.1.1.3          49470        445
10.1.1.5     10.1.1.4          49461        445

Putting some load on the shares

PS C:\> C:\SQLIO\SQLIO.EXE -s10 -kR -frandom -b8 -t8 -o16 -LS -BN <file>
sqlio v1.5.SG
using system counter for latency timings, 10000000 counts per second
8 threads reading for 10 secs from files \\JOSE-SO\Share1\testfile.dat, \\JOSE-SO\Share2\testfile.dat and \\JOSE-SO\Share3\testfile.dat
        using 8KB random IOs
        enabling multiple I/Os per thread with 16 outstanding
        buffering set to not use file nor disk caches (as is SQL Server)
using current size: 16384 MB for file: \\JOSE-SO\Share1\testfile.dat
using current size: 16384 MB for file: \\JOSE-SO\Share2\testfile.dat
using current size: 16384 MB for file: \\JOSE-SO\Share3\testfile.dat
initialization done
CUMULATIVE DATA:
throughput metrics:
IOs/sec: 26822.20
MBs/sec:   209.54
latency metrics:
Min_Latency(ms): 0
Avg_Latency(ms): 13
Max_Latency(ms): 171
histogram:
ms: 0 1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19 20 21 22 23 24+
%: 0 0 0 0 1 2 4 6 8 8 9 8 8 7 6 5 5 4 3 3 2 2 1 1 7

SMB Connection

PS C:\> Get-SmbConnection

SMB Multichannel Connection

PS C:\> Get-SmbMultichannelConnection

ClientIPAddress ServerIPAddress CurrentChannels
--------------- --------------- ---------------
10.1.1.5        10.1.1.2                      4
10.1.1.5        10.1.1.3                      4
10.1.1.5        10.1.1.4                      4
10.1.2.5        10.1.2.2                      4
10.1.2.5        10.1.2.3                      4
10.1.2.5        10.1.2.4                      4

TCP connections used by the client

PS C:\> Get-NetTCPConnection -RemotePort 445

LocalAddress RemoteAddress LocalPort RemotePort
------------ ------------- --------- ----------
10.1.1.5     10.1.1.2          49511        445
10.1.1.5     10.1.1.2          49513        445
10.1.1.5     10.1.1.2          49516        445
10.1.1.5     10.1.1.2          49480        445
10.1.1.5     10.1.1.3          49518        445
10.1.1.5     10.1.1.3          49497        445
10.1.1.5     10.1.1.3          49512        445
10.1.1.5     10.1.1.3          49515        445
10.1.1.5     10.1.1.4          49514        445
10.1.1.5     10.1.1.4          49517        445
10.1.1.5     10.1.1.4          49492        445
10.1.1.5     10.1.1.4          49509        445
10.1.2.5     10.1.2.2          49498        445
10.1.2.5     10.1.2.2          49501        445
10.1.2.5     10.1.2.2          49508        445
10.1.2.5     10.1.2.2          49505        445
10.1.2.5     10.1.2.3          49500        445
10.1.2.5     10.1.2.3          49510        445
10.1.2.5     10.1.2.3          49506        445
10.1.2.5     10.1.2.3          49503        445
10.1.2.5     10.1.2.4          49507        445
10.1.2.5     10.1.2.4          49502        445
10.1.2.5     10.1.2.4          49499        445
10.1.2.5     10.1.2.4          49504        445

Next, here’s the PowerShell script behind what was shown above. Here are a few notes on the script:

It shows some text with a simplified view of the cmdlet about to be run, but the actual cmdlet is usually a bit more complex since it usually includes additional parameters and some formatting.
I tried to use some color and extra blank lines so things would be easier to read.
The “read-host” cmdlets are used to pause the script until I press enter to continue to the next phase.
I pipe the results to “Out-Host” to make sure everything shows in proper order on the screen (without it, some of the longer running cmdlets might show out of order).
The script is run from a client, so cmdlets that target the cluster use the –Cluster parameter and cmdlets that target a specific server use the –CimSession parameter.
SQLIO is used as a workload generator. For more details, see this other blog post on using SQLIO.
I make a point of using the PowerShell cmdlets for network information (Resolve-DnsName instead of NSLOOKUP.EXE, Get-NetTcpConnection instead of NETSTAT.EXE).

Cls

Write-Host -ForegroundColor Green "Cluster Information"
Write-Host " "

Write-Host -ForegroundColor Yellow "PS C:\> Get-ClusterNode -Cluster JOSE-S"
Get-ClusterNode -Cluster JOSE-S | FT -AutoSize | Out-Host

Write-Host -ForegroundColor Yellow "PS C:\> Get-ClusterNetwork -Cluster JOSE-S"
Get-ClusterNetwork -Cluster JOSE-S | FT Cluster, Name, State, Address, Role -AutoSize | Out-Host

Write-Host -ForegroundColor Yellow "PS C:\> Get-ClusterResource -Cluster JOSE-S"
Get-ClusterResource -Cluster JOSE-S | FT -AutoSize | Out-Host

Write-Host -ForegroundColor Yellow "PS C:\> Get-ClusterSharedVolume -Cluster JOSE-S"
Get-ClusterSharedVolume -Cluster JOSE-S | FT -AutoSize | Out-Host

Read-Host

Write-Host -ForegroundColor Green "IP Addresses for each node, DNS for Scale-Out Name"
Write-Host " "

Write-Host -ForegroundColor Yellow "PS C:\> Get-NetIPAddress -CimSession <node>"
"JOSE-S1", "JOSE-S2", "JOSE-S3" | % { Get-NetIPAddress -CimSession $_ -AddressFamily IPv4 -IPAddress 10.1.* | Sort IPAddress | FT PSComputerName, ifIndex, IPAddress, PrefixLength -AutoSize } | Out-Host

Write-Host -ForegroundColor Yellow "PS C:\> Resolve-DNSName JOSE-SO"
Resolve-DnsName JOSE-SO | FT -AutoSize | Out-Host

Read-Host

Write-Host -ForegroundColor Green "IP Addresses on each node per Scope name", ""
Write-Host " "

Write-Host -ForegroundColor Yellow "PS C:\> Get-SmbServerNetworkInterface -CimSession JOSE-SO"
1..3 | % { Get-SmbServerNetworkInterface -CimSession JOSE-S$_ | ? ScopeName -ne "*" | Sort ScopeName | FT PSComputerName, ScopeName, IPAddress -AutoSize } | Out-Host

Read-Host

Write-Host -ForegroundColor Green "Simple connection just to check directory", ""
Write-Host " "

Write-Host -ForegroundColor Yellow "PS C:\> Dir \\JOSE-SO\<share>"
Dir \\JOSE-SO\Share1\testfile.dat, \\JOSE-SO\Share2\testfile.dat, \\JOSE-SO\Share3\testfile.dat | Out-Host

Write-Host -ForegroundColor Green "SMB Connection", ""
Write-Host " "
Write-Host -ForegroundColor Yellow "PS C:\> Get-SmbConnection"
Get-SmbConnection | FT ServerName, ShareName, Dialect, Redirected, Credential -AutoSize | Out-Host

Write-Host -ForegroundColor Green "TCP connections used by the client"
Write-Host " "

Write-Host -ForegroundColor Yellow "PS C:\> Get-NetTCPConnection -RemotePort 445"
Get-NetTCPConnection -RemotePort 445 | Sort LocalAddress, RemoteAddress | FT LocalAddress, RemoteAddress, LocalPort, RemotePort -AutoSize | Out-Host
Read-Host

Write-Host -ForegroundColor Green "Putting some load on the shares", ""
Write-Host " "
Write-Host -ForegroundColor Yellow "PS C:\> C:\SQLIO\SQLIO.EXE -s10 -kR -frandom -b8 -t8 -o16 -LS -BN <file>"
C:\SQLIO\SQLIO.EXE -s10 -kR -frandom -b8 -t8 -o16 -LS -BN \\JOSE-SO\Share1\testfile.dat \\JOSE-SO\Share2\testfile.dat \\JOSE-SO\Share3\testfile.dat

Write-Host -ForegroundColor Green "SMB Connection"
Write-Host " "

Write-Host -ForegroundColor Yellow "PS C:\> Get-SmbConnection"
Get-SmbConnection | FT ServerName, ShareName, Dialect, Redirected, Credential -AutoSize | Out-Host

Write-Host -ForegroundColor Green "SMB Multichannel Connection"
Write-Host " "

Write-Host -ForegroundColor Yellow "PS C:\> Get-SmbMultichannelConnection"
Get-SmbMultichannelConnection | Sort ClientIPAddress, ServerIPAddress | FT ClientIPAddress, ServerIPAddress, CurrentChannels | Out-Host

Write-Host -ForegroundColor Green "TCP connections used by the client"
Write-Host " "

You can find the slides and the recording for the presentation at http://channel9.msdn.com/Events/TechEd/NorthAmerica/2014/DCIM-B337

↧

How the Test-StorageHealth.ps1 script checks for Storage Cluster health

July 26, 2014, 11:16 pm

≫ Next: PowerShell script for Storage Cluster Health Test published to the TechNet Script Center

≪ Previous: Demo details for my TechEd 2014 presentation on File Server Networking

I recently published the Test-StorageHealth.ps1 script to the TechNet Script Center, which checks for the health and capacity of a Storage Cluster based on Windows Server 2012 R2 Scale-Out File Servers. This blog post explains the portion of the script that checks for Storage Cluster health, which is labeled “Phase 1”.

To assess health, the script basically looks at the few objects using PowerShell, making sure they are being reported as healthy. That includes a few cluster objects (nodes, networks, resources, CSV volumes) and some storage objects (volumes, virtual disks, storage pools, physical disks, enclosures). The script also checks related items like deduplicated volumes, continuously available SMB file shares, SMB open files and SMB witness connections. The script also checks a minimum number of items (4 enclosures, 240 physical disks, 3 pools, etc.) which you will likely have to adjust for your specific configuration.

Here’s a table of the objects, the properties that indicate health and the number of expected items included by default in the current version of the script (version 1.5):

Object	PowerShell cmdlet	Health Check	Expected
Cluster Nodes	Get-ClusterNode	State –eq “Up”	4
Cluster Networks	Get-ClusterNetwork	State –eq “Up”	2
Cluster Resources	Get-ClusterResource	State –eq “Online”
Cluster Shared Volumes	Get-ClusterSharedVolume	State –eq “Online”
Volumes	Get-Volume	HealthStatus –eq “Healthy”	33
Deduplicated Volumes	Get-DedupStatus	LastOptimizationResult –eq 0	16
Virtual Disks	Get-VirtualDisk	HealthStatus –eq “Healthy”
Storage Pools	Get-StoragePool	HealthStatus –eq “Healthy”	3
Physical Disks	Get-PhysicalDisk	HealthStatus –eq “Healthy”	240
Storage Enclosures	Get-StorageEnclosure	HealthStatus –eq “Healthy”	4
SMB Shares	Get-SmbShare	Test-Path(SharePath)
SMB Open Files	Get-SmbOpenFile		>0
SMB Witness	Get-SmbWitnessClient		>0

It’s also important to note that, in general, the script looks at cluster objects and objects related to the scale-out file server, skipping local items. Here are a few examples:

When checking storage objects, it filters for objects in the Cluster Storage Subsystem
When checking volumes, it filters for volumes with the CSVFS File System
When check SMB shares, it filter for those marked as continuously available

I hope you find the script and this blog post useful. You can use it directly, adjust a few items for your configuration or just use it as a starting point for creating your own custom script. You can download the script from the TechNet Script Center at http://gallery.technet.microsoft.com/scriptcenter/Test-StorageHealthps1-66d84fd4

↧

PowerShell script for Storage Cluster Health Test published to the TechNet Script Center

July 27, 2014, 10:11 am

≫ Next: Sample C# code for using the latest WMI classes to manage Windows Storage

≪ Previous: How the Test-StorageHealth.ps1 script checks for Storage Cluster health

I just published a PowerShell script to check the health and capacity of a Storage Cluster based on Windows Server 2012 R2 Scale-Out File Servers.

It assumes a specific configuration for Private Clouds using the solution described at http://technet.microsoft.com/en-us/library/dn554251.aspx.

It performs specific health checks for

Failover Clustering (Cluster, Resources, Networks, Nodes)
Storage Spaces (Physical Disks, Enclosures, Virtual Disks)
Cluster Shared Volumes
SMB File Shares
Deduplication

It includes reporting of capacity by Pool, Volume and Deduplicated volume. It also includes collection of cluster logs and error events from all nodes in the cluster.

You can review and download the script from http://gallery.technet.microsoft.com/scriptcenter/Test-StorageHealthps1-66d84fd4

Lastly, if you're interested in how the script was written, you might also want to read these other blog posts:

PowerShell tips for building objects with custom properties and special formatting
http://blogs.technet.com/b/josebda/archive/2014/04/19/powershell-tips-for-building-objects-with-custom-properties-and-special-formatting.aspx
How the Test-StorageHealth.ps1 script checks for Storage Cluster health
http://blogs.technet.com/b/josebda/archive/2014/07/26/how-the-test-storagehealth-ps1-script-checks-for-storage-cluster-health.aspx

↧

Sample C# code for using the latest WMI classes to manage Windows Storage

August 2, 2014, 7:37 pm

≫ Next: New poster for Cloud Ecosystem: Microsoft Azure, Windows Server 2012 R2 and System Center 2012 R2

≪ Previous: PowerShell script for Storage Cluster Health Test published to the TechNet Script Center

This blog post shows a bit of C# code to use the Windows Storage Management API (SM-API) classes that were introduced in Windows Server 2012 and Windows 8.

You can find a list of these classes at class described at http://msdn.microsoft.com/en-us/library/hh830612.aspx, including MSFT_PhysicalDisk, MSFT_StoragePool or MSFT_VirtualDisk.

I found a number of examples with the old interface using the old classes like Win32_Volume, but few good ones with the new classes like MSFT_Volume.

This is some simple C# code using console output. The main details to highlight here are the use of System.Management and how to specify the scope, which allows you to manage a remote computer.

Please note that you might need to enable WMI on the computer, which can be easily done with the command line “winrm quickconfig”.

Here is the source code, for the console application, which lists information about volumes and physical disks on the local machine.

using System;
using System.Text;
using System.Threading;
using System.Management;

namespace SMAPIQuery
{
    class Program
    {
        static void Main(string[] args)
        {
            // Use the Storage management scope
            ManagementScope scope = new ManagementScope("\\\\localhost\\ROOT\\Microsoft\\Windows\\Storage");
            // Define the query for volumes
            ObjectQuery query = new ObjectQuery("SELECT * FROM MSFT_Volume");

            // create the search for volumes
            ManagementObjectSearcher searcher = new ManagementObjectSearcher(scope, query);
            // Get the volumes
            ManagementObjectCollection allVolumes = searcher.Get();
            // Loop through all volumes
            foreach (ManagementObject oneVolume in allVolumes)
            {
                // Show volume information
                if (oneVolume["DriveLetter"].ToString()[0] > ' ' )
                {
                    Console.WriteLine("Volume '{0}' has {1} bytes total, {2} bytes available", oneVolume["DriveLetter"], oneVolume["Size"], oneVolume["SizeRemaining"]);
                }
            }

// Define the query for physical disks
query = new ObjectQuery("SELECT * FROM MSFT_PhysicalDisk");

// create the search for physical disks
searcher = new ManagementObjectSearcher(scope, query);

// Get the physical disks
ManagementObjectCollection allPDisks = searcher.Get();

            // Loop through all physical disks
            foreach (ManagementObject onePDisk in allPDisks)
            {
                // Show physical disk information
                Console.WriteLine("Disk {0} is model {1}, serial number {2}", onePDisk["DeviceId"], onePDisk["Model"], onePDisk["SerialNumber"]);
            }

            Console.ReadLine();
         }
        }
    }

Here is some sample output from this application:

Volume 'D' has 500104687616 bytes total, 430712184832 bytes available
Volume 'E' has 132018860032 bytes total, 110077665280 bytes available
Volume 'F' has 500105216000 bytes total, 356260683776 bytes available
Volume 'C' has 255690010624 bytes total, 71789502464 bytes available

Disk 2 is model SD              , serial number
Disk 0 is model MTFDDAK256MAM-1K12, serial number         131109303905
Disk 3 is model 5AS             , serial number 00000000e45ca01b30c1
Disk 1 is model ST9500325AS, serial number             6VEK9B89

↧

New poster for Cloud Ecosystem: Microsoft Azure, Windows Server 2012 R2 and System Center 2012 R2

August 7, 2014, 9:28 am

≫ Next: SMB3 PowerShell changes in Windows Server 2012 R2: Simpler setting of ACL for the folder behind a share

≪ Previous: Sample C# code for using the latest WMI classes to manage Windows Storage

For those who enjoy the Server Posterpedia posters, there is a new one that just became available for download. It’s called the "Cloud Ecosystem: Microsoft Azure, Windows Server 2012 R2 and System Center 2012 R2" poster and it depicts both public and on-premises cloud technologies.

Here’s a little thumbnail to give you an idea of what it looks like:

As you probably can’t read the small font above :-), here are some details on what this poster includes:

Microsoft Azure Services including Service Categories, Compute Services, Data Services and App Services
System Center 2012 R2 including App Controller, Virtual Machine Manager, Operations Manager, Configuration Manager, Service Manager, Orchestrator, Data Protection Manager and Azure Pack
Windows Intune
Windows Server 2012 R2 including Storage Spaces, Data Deduplication, Resilient File System, SMB Transparent Failover, Storage Quality of Service, Generation 2 Virtual Machines, Online VHDX Resize, Enhanced Session Mode, Live Migration, Failover Clustering, Cluster Shared Volumes, Scale-Out File Server, Shared Virtual Hard Disks, Hyper-V Extensible Switch, Remote Desktop Services, SMB Direct, SMB Multi-channel and NIC Teaming.

You can get the new poster from the download center at http://www.microsoft.com/en-us/download/details.aspx?id=43718.

You can also get this and other posters using the free copy Posterpedia app from the Windows Store at http://aka.ms/sposterpedia.

↧

SMB3 PowerShell changes in Windows Server 2012 R2: Simpler setting of ACL for the folder behind a share

August 11, 2014, 10:01 am

≫ Next: SMB3 PowerShell changes in Windows Server 2012 R2: Scale-Out Rebalancing (per-share redirection)

≪ Previous: New poster for Cloud Ecosystem: Microsoft Azure, Windows Server 2012 R2 and System Center 2012 R2

Introduction

Windows Server 2012 R2 introduced a new version of SMB. Technically it’s SMB version 3.02, but we continue to call it just SMB3. The main changes are described at http://technet.microsoft.com/en-us/library/hh831474.aspx.

With this new release, we made a few changes in SMB PowerShell to support the new scenarios and features.

This includes a few new cmdlets and some changes to existing cmdlets, with extra care not break any of your existing scripts.This blog post outlines one of the 7 set of changes related to SMB PowerShell in Windows Server 2012 R2.

Simpler setting of ACL for the folder behind a share

In Windows Server 2012, the SMB share has a property that facilitates applying the share ACL to the file system folder used by the share.

Here’s the syntax for a share named Share1:

(Get-SmbShare –Name Share1 ).PresetPathACL | Set-Acl

In Windows Server 2012 R2, we have improved this scenario by providing a proper cmdlet to apply the share ACL to the file system used by the share.

Here’s the new syntax for the same share:

Set-SmbPathAcl –ShareName Share1

Notes

1) The Windows Server 2012 syntax continues to work with Windows Server 2012 R2, but the new syntax is much simpler and therefore recommended.

2) There is known issue with Windows Server 2012 R2 Preview that causes this new cmdlet to fail when using non-Unicode languages. As a workaround, you can use the old syntax.

3) This blog post is an updated version of the September 2013 post at http://blogs.technet.com/b/josebda/archive/2013/09/03/what-s-new-in-smb-powershell-in-windows-server-2012-r2.aspx focused on a single topic.

↧