Note: This blog post is a Windows Server 2012 R2 update on a previous version focused on Windows Server 2012.

1. Introduction

With the release of Windows 8.1 and Windows Server 2012 R2, I am frequently asked about how older versions of Windows will behave when connecting to or from these new versions. Upgrading to a new version of SMB is something that happened a few times over the years and we established a process in the protocol itself by which clients and servers negotiate the highest version that both support.

2. Versions

There are several different versions of SMB used by Windows operating systems:

• CIFS – The ancient version of SMB that was part of Microsoft Windows NT 4.0 in 1996. SMB1 supersedes this version.
• SMB 1.0 (or SMB1) – The version used in Windows 2000, Windows XP, Windows Server 2003 and Windows Server 2003 R2
• SMB 2.0 (or SMB2) – The version used in Windows Vista (SP1 or later) and Windows Server 2008
• SMB 2.1 (or SMB2.1) – The version used in Windows 7 and Windows Server 2008 R2
• SMB 3.0 (or SMB3) – The version used in Windows 8 and Windows Server 2012
• SMB 3.02 (or SMB3) – The version used in Windows 8.1 and Windows Server 2012 R2

Windows NT is no longer supported, so CIFS is definitely out. Windows Server 2003 R2 with a current service pack is under Extended Support, so SMB1 is still around for a little while. SMB 2.x in Windows Server 2008 and Windows Server 2008 R2 are under Mainstream Support until 2015. You can find the most current information on the support lifecycle page for Windows Server. The information is subject to the Microsoft Policy Disclaimer and Change Notice.  You can use the support pages to also find support policy information for Windows XP, Windows Vista, Windows 7 and Windows 8.

In Windows 8.1 and Windows Server 2012 R2, we introduced the option to completely disable CIFS/SMB1 support, including the actual removal of the related binaries. While this is not the default configuration, we recommend disabling this older version of the protocol in scenarios where it’s not useful, like Hyper-V over SMB. You can find details about this new option in item 7 of this blog post: What’s new in SMB PowerShell in Windows Server 2012 R2.

3. Negotiated Versions

Here’s a table to help you understand what version you will end up using, depending on what Windows version is running as the SMB client and what version of Windows is running as the SMB server:

* WS = Windows Server

4. Using PowerShell to check the SMB version

In Windows 8 or Windows Server 2012, there is a new PowerShell cmdlet that can easily tell you what version of SMB the client has negotiated with the File Server. You simply access a remote file server (or create a new mapping to it) and use Get-SmbConnection. Here’s an example:

PS C:\> Get-SmbConnection
 
ServerName   ShareName  UserName            Credential          Dialect   NumOpens
----------   ---------  --------            ----------          -------   --------
FileServer1  IPC$       DomainName\UserN... DomainName.Testi... 3.00      0
FileServer1  FileShare  DomainName\UserN... DomainName.Testi... 3.00      14
FileServ2    FS2        DomainName\UserN... DomainName.Testi... 3.02      3 
VNX3         Share1     DomainName\UserN... DomainName.Testi... 3.00      6
Filer2       Library    DomainName\UserN... DomainName.Testi... 3.00      8
DomainCtrl1  netlogon   DomainName\Compu... DomainName.Testi... 2.10      1

In the example above, a server called “FileServer1” was able to negotiate up to version 3.0. FileServ2 can use version 3.02. That means that both the client and the server support the latest version of the SMB protocol. You can also see that another server called “DomainCtrl1” was only able to negotiate up to version 2.1. You can probably guess that it’s a domain controller running Windows Server 2008 R2. Some of the servers on the list are not running Windows, showing the dialect that these non-Windows SMB implementations negotiated with this specific Windows client.

If you just want to find the version of SMB running on your own computer, you can use a loopback share combined with the Get-SmbConnection cmdlet. Here’s an example:

PS C:\> dir \\localhost\c$
 
Directory: \\localhost\c$
 
Mode                LastWriteTime     Length Name
----                -------------     ------ ----
d----         5/19/2012   1:54 AM            PerfLogs
d-r--          6/1/2012  11:58 PM            Program Files
d-r--          6/1/2012  11:58 PM            Program Files (x86)
d-r--         5/24/2012   3:56 PM            Users
d----          6/5/2012   3:00 PM            Windows
 
PS C:\> Get-SmbConnection -ServerName localhost
 
ServerName  ShareName  UserName            Credential          Dialect  NumOpens
----------  ---------  --------            ----------          -------  --------
localhost   c$         DomainName\UserN... DomainName.Testi... 3.02     0

You have about 10 seconds after you issue the “dir” command to run the “Get-SmbConnection” cmdlet. The SMB client will tear down the connections if there is no activity between the client and the server. It might help to know that you can use the alias “gsmbc” instead of the full cmdlet name.

5. Features and Capabilities

Here’s a very short summary of what changed with each version of SMB:

• From SMB 1.0 to SMB 2.0 - The first major redesign of SMB
  • Increased file sharing scalability
  • Improved performance
    • Request compounding
    • Asynchronous operations
    • Larger reads/writes
  • More secure and robust
    • Small command set
    • Signing now uses HMAC SHA-256 instead of MD5
    • SMB2 durability
• From SMB 2.0 to SMB 2.1
  • File leasing improvements
  • Large MTU support
  • BranchCache
• From SMB 2.1 to SMB 3.0
  • Availability
    • SMB Transparent Failover
    • SMB Witness
    • SMB Multichannel
  • Performance
    • SMB Scale-Out
    • SMB Direct (SMB 3.0 over RDMA)
    • SMB Multichannel
    • Directory Leasing
    • BranchCache V2
  • Backup
    • VSS for Remote File Shares
  • Security
    • SMB Encryption using AES-CCM (Optional)
    • Signing now uses AES-CMAC
  • Management
    • SMB PowerShell
    • Improved Performance Counters
    • Improved Eventing
• From SMB 3.0 to SMB 3.02
  • Automatic rebalancing of Scale-Out File Server clients
  • Improved performance of SMB Direct (SMB over RDMA)
  • Support for multiple SMB instances on a Scale-Out File Server

You can get additional details on the SMB 2.0 improvements listed above at
http://blogs.technet.com/b/josebda/archive/2008/12/09/smb2-a-complete-redesign-of-the-main-remote-file-protocol-for-windows.aspx

You can get additional details on the SMB 3.0 improvements listed above at
http://blogs.technet.com/b/josebda/archive/2012/05/03/updated-links-on-windows-server-2012-file-server-and-smb-3-0.aspx

You can get additional details on the SMB 3.02 improvements in Windows Server 2012 R2 at
http://technet.microsoft.com/en-us/library/hh831474.aspx

6. Recommendation

We strongly encourage you to update to the latest version of SMB, which will give you the most scalability, the best performance, the highest availability and the most secure SMB implementation.

Keep in mind that Windows Server 2012 Hyper-V and Windows Server 2012 R2 Hyper-V only support SMB 3.0 for remote file storage. This is due mainly to the availability features (SMB Transparent Failover, SMB Witness and SMB Multichannel), which did not exist in previous versions of SMB. The additional scalability and performance is also very welcome in this virtualization scenario. The Hyper-V Best Practices Analyzer (BPA) will warn you if an older version is detected.

7. Conclusion

We’re excited about SMB3, but we are also always concerned about keeping as much backwards compatibility as possible. Both SMB 3.0 and SMB 3.02 bring several key new capabilities and we encourage you to learn more about them. We hope you will be convinced to start planning your upgrades as early as possible.

Note 1: Protocol Documentation

If you consider yourself an SMB geek and you actually want to understand the SMB NEGOTIATE command in greater detail, you can read the [MS-SMB2-Preview] protocol documentation (which covers SMB 2.0, 2.1, 3.0 and 3.02), currently available from http://msdn.microsoft.com/en-us/library/ee941641.aspx. In regards to protocol version negotiation, you should pay attention to the following sections of the document:

• 1.7: Versioning and Capability Negotiation
• 2.2.3: SMB2 Negotiate Request
• 2.2.4: SMB2 Negotiate Response

Section 1.7 includes this nice state diagram describing the inner workings of protocol negotiation:

Note 2: Third-party implementations

There are several implementations of the SMB protocol from someone other than Microsoft. If you use one of those implementations of SMB, you should ask whoever is providing the implementation which version of SMB they implement for each version of their product. Here are a few of these implementations of SMB:

• Apple – Up to SMB2 implemented in OS X 10 Mavericks - http://images.apple.com/osx/preview/docs/OSX_Mavericks_Core_Technology_Overview.pdf
• EMC– Up to SMB3 implemented in VNX - http://www.emc.com/collateral/white-papers/h11427-vnx-introduction-smb-30-support-wp.pdf
• Linux (Client) – Up to SMB3 implemented in the Linux kernel 3.11 – http://www.snia.org/sites/default/files2/SDC2012/presentations/Revisions/SteveFrench_Linux_CIFS-SMB2-year-in-review-revision.pdf (waiting for the latest SDC talk to become available, which has updated info on Linux support for SMB3)
• NetApp– Up to SMB3 implemented in Data ONTAP 8.2 - https://communities.netapp.com/community/netapp-blogs/cloud/blog/2013/06/11/clustered-ontap-82-with-windows-server-2012-r2-and-system-center-2012-r2-innovation-in-storage-and-the-cloud
• Samba (Server) – Up to SMB3 implemented in Samba 4.1 - http://www.samba.org/samba/history/samba-4.1.0.html

Please note that is not a complete list of implementations and the list is bound to become obsolete the minute I post it. Please refer to the specific implementers for up-to-date information on their specific implementations and which version and optional portions of the protocol they offer.

One of the questions regarding Hyper-V over SMB that I get the most relates to how the network should be configured. Networking is key to several aspects of the scenario, including performance, availability and scalability.

The main challenge is to provide a fault-tolerant and high-performance network for the two clusters typically involved: the Hyper-V cluster (also referred to as the Compute Cluster) and the Scale-out File Server Cluster (also referred to as the Storage Cluster).

Not too long ago, the typical configuration for virtualization deployments would call for up to 6 distinct networks for these two clusters:

• Client (traffic between the outside and VMs running in the Compute Cluster)
• Storage (main communications between the Compute and Storage clusters)
• Cluster (communication between nodes in both clusters, including heartbeat)
• Migration (used for moving VMs between nodes in the Compute Cluster)
• Replication (used by Hyper-V replica to send changes to another site)
• Management (used to configuring and monitoring the systems, typically also including DC and DNS traffic)

These days, it’s common to consolidate these different types of traffic, with the proper fault tolerance and Quality of Service (QoS) guarantees.

There are certainly many different ways to configure the network for your Hyper-V over SMB, but this blog post will focus on two of them:

• A basic fault-tolerant solution using just two physical network ports per node
• A high-end solution using RDMA networking for the highest throughput, highest density, lowest latency and low CPU utilization.

Both configurations presented here work with Windows Server 2012 and Windows Server 2012 R2, the two versions of Windows Server that support the Hyper-V over SMB scenario.

Configuration 1 – Basic fault-tolerant Hyper-V over SMB configuration with two non-RDMA port

The solution below using two network ports for each node of both the Compute Cluster and the Storage Cluster. NIC teaming is the main technology used for fault tolerance and load balancing.

Configuration 1: click on diagram to see a larger picture

Notes:

• A single dual-port network adapter per host can be used. Network failures are usually related to cables and switches, not the NIC itself. It the NIC does fail, failover clustering on the Hyper-V or Storage side would kick in. Two network adapters each with one port is also an option.
• The 2 VNICs on the Hyper-V host are used to provide additional throughput for the SMB client via SMB Multichannel, since the VNIC does not support RSS (Receive Side Scaling, which helps spread the CPU load of networking activity across multiple cores). Depending on configuration, increasing it up to 4 VNICs per Hyper-V host might be beneficial to increase throughput.
• You can use additional VNICs that are dedicated for other kinds of traffic like migration, replication, cluster and management. In that case, you can optionally configure SMB Multichannel constraints to limit the SMB client to a specific subset of the VNICs. More details can be found in item 7 of the following article: The basics of SMB Multichannel, a feature of Windows Server 2012 and SMB 3.0
• If RDMA NICs are used in this configuration, their RDMA capability will not be leveraged, since the physical port capabilities are hidden behind NIC teaming and the virtual switch.
• Network QoS should be used to tame each individual type of traffic on the Hyper-V host. In this configuration, it’s recommended to implement the network QoS at the virtual switch level. See http://technet.microsoft.com/en-us/library/jj735302.aspx for details (the above configuration matches the second one described in the linked article).

Configuration 2 - High-performance fault-tolerant Hyper-V over SMB configuration with two RDMA ports and two non-RDMA ports

The solution below requires four network ports for each node of both the Compute Cluster and the Storage Cluster, two of them being RDMA-capable. NIC teaming is the main technology used for fault tolerance and load balancing on the two non-RDMA ports, but SMB Multichannel covers those capabilities for the two RDMA ports.

Configuration 2: click on diagram to see a larger picture

Notes:

• Two dual-port network adapter per host can be used, one RDMA and one non-RDMA.
• In this configuration, Storage, Migration and Clustering traffic should leverage the RDMA path. The client, replication and management traffic should use the teamed NIC path.
• In this configuration, if using Windows Server 2012 R2, Hyper-V should be configured to use SMB for Live Migration. This is not the default setting.
• The SMB client will naturally prefer the RDMA paths, so there is no need to specifically configure that preference via SMB Multichannel constraints.
• There are three different types of RDMA NICs that can be used: iWARP, RoCE and InifiniBand. Below are links to step-by-step configuration instructions for each one:
  • iWARP – Deploying SMB Direct using the Chelsio T4 cards – Step by Step
  • RoCE - Deploying SMB Direct using the Mellanox ConnectX-3 – Step by Step
  • InfiniBand - Deploying SMB Direct using the Mellanox ConnectX-2/ConnectX-3– Step by Step
• Network QoS should be used to tame traffic flowing through the virtual switch on the Hyper-V host. If your NIC and switch support Data Center Bridging (DCB) and Priority Flow Control (PFC), there are additional options available as well. See http://technet.microsoft.com/en-us/library/jj735302.aspx for details (the above configuration matches the fourth one described in the linked article).
• In most environments, RDMA provides enough bandwidth without the need of any traffic shaping. If using Windows Server 2012 R2, SMB Bandwidth Limits can optionally be used to shape the Storage and Live Migration traffic. More details can be found in item 4 of the following article: What’s new in SMB PowerShell in Windows Server 2012 R2. SMB Bandwidth Limits can also be used for configuration 1, but it's more common here.

I hope this blog posts helps with the network planning for your Private Cloud deployment. Feel free to ask questions via the comments below.

Introduction

This blog post focus on the new SMB Scale-Out Rebalancing introduced in Windows Server 2012 R2. If you haven’t seen it yet, it delivers a new way of balancing file clients accessing a Scale-Out File Server.

In Windows Server 2012, each client would be randomly directed via DNS Round Robin to a node of the cluster and stick with that one for all shares, all traffic going to that Scale-Out File Server. If necessary, some server-side redirection of individual IO requests could happen in order to fulfill the client request.

In Windows Server 2012 R2, a single client might be directed to a different node for each file share. The idea here is that the client will connect to the best node for each individual file share in the Scale-Out File Server Cluster, avoiding any kind of server-side redirection.

Now there are some details about when redirection can happen and when the new behavior will apply. Let’s look into the 3 types of scenarios you might encounter.

Hyper-V over SMB with Windows Server 2012 and a SAN back-end (symmetric)

When we first introduced the SMB Scale-Out File Server in Windows Server 2012, as mentioned in the introduction, the client would be randomly directed to one and only one node for all shares in that cluster.

If the storage is equally accessible from every node (what we call symmetric cluster storage), then you can do reads and writes from every file server cluster node, even if it’s not the owner node for that Cluster Shared Volume (CSV). We refer to this as Direct IO.

However, metadata operations (like creating a new file, renaming a file or locking byte range on a file) must be done orchestrated cross the cluster and will be executed on a single node called the coordinator node or the owner node. Any other node will simply redirect these metadata operations to the coordinator node.

The diagram below illustrates these behaviors:

Figure 1: Windows Server 2012 Scale-Out File Server on symmetric storage

The most common example of symmetric storage is when the Scale-Out File Server is put in front of a SAN. The common setup is to have every file server node connected to the SAN.

Another common example is when the Scale-Out File Server is using a clustered Storage Spaces solution with a shared SAS JBOD using Simple Spaces (no resiliency).

Hyper-V over SMB with Windows Server 2012 and Mirrored Storage Spaces (asymmetric)

When using a Mirrored Storage Spaces, the CSV operates in a block level redirected IO mode. This means that every read and write to the volume must be performed through the coordinator node of that CSV.

This configuration, where not every node has the ability to read/write to the storage, is generically called asymmetric storage. In those cases, every data and metadata request must be redirected to the coordinator node.

In Windows Server 2012, the SMB client chooses one of the nodes of the Scale-Out File Server cluster using DNS Round Robin and that may not necessarily be the coordinator node that owns the CSV that contains the file share it wants to access.

In fact, if using multiple file shares in a well-balanced cluster, it’s likely that the node will own some but not all of the CSVs required.

That means some SMB requests (for data or metadata) are handled by the node and some are redirected via the cluster back-end network to the right owner node. This redirection, commonly referred to as “double-hop”, is a very common occurrence in Windows Server 2012 when using the Scale-Out File Server combined with Mirrored Storage Spaces.

It’s important to mention that this cluster-side redirection is something that is implemented by CSV and it can be very efficient, especially if your cluster network uses RDMA-capable interfaces.

The diagram below illustrates these behaviors:

Figure 2: Windows Server 2012 Scale-Out File Server on asymmetric storage

The most common example of asymmetric storage is when the Scale-Out File Server is using a Clustered Storage Spaces solution with a Shared SAS JBOD using Mirrored Spaces.

Another common example is when only a subset of the file server nodes is directly connected to a portion backend storage, be it Storage Spaces or a SAN.

A possible asymmetric setup would be a 4-node cluster where 2 nodes are connected to one SAN and the other 2 nodes are connected to a different SAN.

Hyper-V over SMB with Windows Server 2012 R2 and Mirrored Storage Spaces (asymmetric)

If you’re following my train of thought here, you probably noticed that the previous configuration has a potential for further optimization and that’s exactly what we did in Windows Server 2012 R2.

In this new release, the SMB client gained the flexibility to connect to different Scale-Out File Server cluster nodes for each independent share that it needs to access.

The SMB server also gained the ability to tell its clients (using the existing Witness protocol) what is the ideal node to access the storage, in case it happens to be asymmetric.

With the combination of these two behavior changes, a Windows Server 2012 R2 SMB client and server are capable to optimize the traffic, so that no redirection is required even for asymmetric configurations.

The diagram below illustrates these behaviors:

Figure 3: Windows Server 2012 R2 Scale-Out File Server on asymmetric storage

Note that the SMB client now always talks to the Scale-Out File Server node that is the coordinator of the CSV where the share is.

Note also that the CSV ownership is shared across nodes in the example. That is not a coincidence. CSV now includes the ability to spread its CSVs across the nodes uniformly.

If you add or remove nodes or CSVs in the Scale-Out File Server cluster, the CSVs will be rebalanced. The SMB clients will then also be rebalanced to follow the CSV owner nodes for their shares.

Key configuration requirements for asymmetric storage in Windows Server 2012 R2

Because of this new automatic rebalancing, there are key new considerations when designing asymmetric (Mirrored or Parity Storage Spaces) storage when using Windows Server 2012 R2.

First of all, you should have at least as many CSVs as you have file server cluster nodes. For instance, for a 3-node Scale-Out File Server, you should have at least 3 CSVs. Having 6 CSVs is also a valid configuration, which will help with rebalancing when one of the nodes is down for maintenance.

To be clear, if you have a single CSV in such asymmetric configuration in Windows Server 2012 R2 Scale-Out File Server cluster, only one node will be actively accessed by SMB clients.

You should also try, as much as possible, to have your file shares and workloads evenly spread across the multiple CSVs. This way you won’t have some nodes working much harder than others.

Forcing per-share redirection for symmetric storage in Windows Server 2012 R2

The new per-share redirection does not happen by default in the Scale-Out File Server if the back-end storage is found to be symmetric.

For instance, if every node of your file server is connected to a SAN back-end, you will continue to have the behavior described on Figure 1 (Direct IO from every node plus metadata redirection).

The CSVs will automatically be balanced across file server cluster nodes even in symmetric storage configurations. You can turn that behavior off using the cmdlet below, although I'm hard pressed to find any good reason to do it.

(Get-Cluster). CSVBalancer = 0

However, when using symmetric storage, the SMB clients will continue to each connect a single file server cluster node for all shares. We opted for this behavior by default because Direct IO tends to be efficient in these configurations and the amount of metadata redirection should be fairly small.

You can override this setting and make the symmetric cluster use the same rebalancing behavior as an asymmetric cluster by using the following PowerShell cmdlet:

Set-ItemProperty HKLM:\System\CurrentControlSet\Services\LanmanServer\Parameters -Name AsymmetryMode -Type DWord -Value 2 -Force

You must apply the setting above to every file server cluster node. The new behavior won’t apply to existing client sessions.

If you switch to this configuration, you must apply the same planning rules outlined previously (at least one CSV per file server node, ideally two).

Conclusion

I hope this clarifies the behavior changes introduced with SMB Scale-Out Automatic Rebalancing in Windows Server 2012 R2.

While most of it is designed to just work, I do get some questions about it from those interested in understanding what happens behind the scenes.

Let me know if you find those useful or if you have any additional questions.

I’m always amazed by the vast amount of knowledge being posted online for free download. If you are an IT Professional who deals with Storage (or if you manage one of them), check out the high-quality tutorials just posted online by the Storage Network Industry Association (SNIA).

These peer-reviewed, vendor-neutral tutorials, offered by the SNIA Education Committee, cover a variety of very current Storage-related topics. They were delivered live during the recent Storage Networking World conference in Santa Clara, CA (SNW Fall 2012). I delivered one of them and I was also able to attend a few of them in person.

Here’s a snapshot of what they covered, organized by track:

Track: Big Data

• Introduction to Analytics and Big Data – Hadoop (Rob Peglar)
• Protecting Data in the "Big Data" World (Thomas Rivera)
• How to Cost Effectively Retain Reference Data for Analytics and Big Data (Molly Rector)
• Big Data Storage Options for Hadoop (Dr. Sam Fineberg)
• Massively Scalable File Storage (Philippe Nicolas)
• Can You Manage at Petabyte Scale? (John Webster)

Track: Cloud and Emerging Technologies

• Interoperable Cloud Storage with the CDMI Standard (Mark Carlson)
• The Business Case for the Cloud (Alex McDonald)
• Archiving and Preservation in the Cloud: Business Case, Challenges and Best Practices (Chad Thibodeau, Sebastian Zangaro)
• Enterprise Architecture and Storage Clouds (Marty Stogsdill)

Track: Data Protection and Management

• Introduction to Data Protection: Backup to Tape, Disk and Beyond (Jason Iehl)
• Trends in Data Protection and Restoration Technologies (Michael Fishman)
• Advanced Data Reduction Concepts (Gene Nagle, Thomas Rivera)

Track: File Systems and File Management

• The File Systems Evolution (Thomas Rivera)
• SMB Remote File Protocol, including SMB 3.0 (Jose Barreto)

Track: Green Storage

• Green Storage and Energy Efficiency (Carlos Pratt)
• Green Storage - the Big Picture ("Green is More Than kWh!") (SW Worth)

Track: Networking

• Technical Overview of Data Center Networks (Dr. Joseph White)
• Single and Multi Switch Designs with FCoE (Chad Hintz)
• The Unintended Consequences of Converged Data Center Deployment Models (Simon Gordon)
• FCoE Direct End-Node to End-Node (a/k/a FCoE VN2VN) (John Hufferd)
• PCI Express IO Virtualization Overview (Ron Emerick)
• How Infrastructure as a Service (laaS) and Software Defined Networks (SDN) will Change the Data Center (Samir Sharma)
• IP Storage Protocols: iSCSI (John Hufferd)
• Fibre Channel Over Ethernet (FCoE) (John Hufferd)

Track: Professional Development

• Social Media and the IT Professional - Are You a Match? (Marty Foltyn)
• Reaction Management and Self-facilitation Techniques (David Deming)

Track: Security

• Practical Storage Security with Key Management (Russ Fellows)
• Unmasking Virtualization Security (Eric Hibbard)
• Data Breaches and the Encryption Safe Harbor (Eric Hibbard)
• Storage Security - the ISO/IEC Standard (Eric Hibbard)
• A Hype-free Stroll Through Cloud Security (Eric Hibbard)
• Got Lawyers? They've Got Storage and ESI in the Cross-hairs! (Eric Hibbard)

Track: Solid State Storage

• The Benefits of Solid State in Enterprise Storage Systems (David Dale)
• An In-depth Look at SNIA's Enterprise Solid State Storage Test Specification (PTS v1.1) (Easen Ho)
• Realities of Solid State Storage (Luanne Dauber)
• What Happens When Flash Moves to Triple Level Cell (TLC) (Luanne Dauber)
• NVMe the nextGen Interface for Solid State Storage (Anil Vasudeva)
• SCSI Express - Fast & Reliable Flash Storage for the Enterprise (Marty Czekalski)

Track: Storage and Storage Management

• What's Old is New Again - Storage Tiering (Kevin Dudak)
• Simplified Integration and Management in Multi-Vendor SAN Environments (Chauncey Schwartz)
• SAS: The Emerging Storage Fabric (Marty Czekalski)

Track: Virtualization/Applications

• Virtualization Practices: Providing a Complete Virtual Solution in a Box (Jyh-shing)
• VDI Optimization - Real World Learnings (Russ Fellows) 

I would encourage all my industry colleagues to check these out!

To view the slides for each one of the tutorials listed above in PDF format, visit http://www.snia.org/education/tutorials/2012/fall. Be sure to check the Legal Notices on the documents about how you can use them.

For more information about SNIA, check http://www.snia.org/. It's also never too early to plan to check the next wave, to be delivered during the next SNW in the Spring.

When you install a fresh copy of Windows Server 2012 and configure it with the File Server role, the default Power Setting balances power efficiency and performance.

For this reason, even if you have a few high speed network interfaces and the fastest SSD storage out there, you might not be getting the very best IOPS and throughput possible.

To get the absolute best performance from your file server, you can set the server to the "High Performance" power profile.

To configure this using a GUI, go to the Start Menu, search for "Choose a Power Plan" under Settings, then select the "High Performance" option.

To configure this from a command line, use "POWERCFG.EXE /S SCHEME_MIN".

NOTE: This is not recommended for every file server. The Balanced (default) profile will be enough for most cases, with high and constant load being the exception.

A while back, I showed how to use PowerShell V2 and our old SMB WMIv1 object to explain how to find the free space behind a file share (essentially the free space for the volume that contains the file share). That post is available at http://blogs.technet.com/b/josebda/archive/2010/04/08/using-powershell-v2-to-gather-info-on-free-space-on-the-volumes-of-your-remote-file-server.aspx. While that post was a good example of how to construct a more elaborate solution using PowerShell, it was a little complicated :-).

Now, with Windows Server 2012, PowerShell V3 and SMB PowerShell, things got much simpler. I can essentially do the same thing with a simple one-liner.

For instance, to see the free space on the volume behind a specific share named TEST, you can use

Get-Volume -Id (Get-SmbShare TEST).Volume

To list the volumes for all shares on a specific server, you can use:

Get-SmbShare | ? Volume -ne $null | % { $_ | FL ; Get-Volume -Id $_.Volume | FL }

Note that you can also execute those remotely, pointing to the server where the shares are located:

Get-Volume  -CimSession Server1 -Id (Get-SmbShare TEST -CimSession Server1).Volume

Get-SmbShare -CimSession Server1 | ? Volume  -ne $null | % { $_ | FL ; Get-Volume -CimSession Server1 -Id $_.Volume | FL  }

Here is a complete example, with output. First a simple query to find the information for a volume behind a share

PS C:\> Get-Volume -Id (Get-SmbShare VMS1).Volume

DriveLetter       FileSystemLabel  FileSystem       DriveType        HealthStatus        SizeRemaining             Size
-----------       ---------------  ----------       ---------        ------------        -------------             ----
I                                  NTFS             Fixed            Healthy                  78.85 GB           100 GB

PS C:\> Get-Volume -Id (Get-SmbShare Projects).Volume | Select *

HealthStatus          : Healthy
DriveType             : Fixed
DriveLetter           :
FileSystem            : CSVFS
FileSystemLabel       :
ObjectId              : \\?\Volume{20795fea-b7da-43dd-81d7-4d346c337a73}\
Path                  : \\?\Volume{20795fea-b7da-43dd-81d7-4d346c337a73}\
Size                  : 107372081152
SizeRemaining         : 85759995904
PSComputerName        :
CimClass              : ROOT/Microsoft/Windows/Storage:MSFT_Volume
CimInstanceProperties : {DriveLetter, DriveType, FileSystem, FileSystemLabel...}
CimSystemProperties   : Microsoft.Management.Infrastructure.CimSystemProperties

Now a more complete query, showing all shares starting with VMS and information on the volume behind them:

PS C:\> Get-SmbShare VMS* | ? Volume -ne $null | % { $_ | FL ; Get-Volume -Id $_.Volume | FL }

Name        : VMS1
ScopeName   : FST2-FS
Path        : I:\VMS
Description :

DriveLetter     : I
DriveType       : Fixed
FileSystem      : NTFS
FileSystemLabel :
HealthStatus    : Healthy
ObjectId        : \\?\Volume{b02c4ba7-e6f1-11e1-93eb-0008a1c0ef0d}\
Path            : \\?\Volume{b02c4ba7-e6f1-11e1-93eb-0008a1c0ef0d}\
Size            : 107372081152
SizeRemaining   : 84665225216
PSComputerName  :

Name        : VMS2
ScopeName   : FST2-FS
Path        : J:\VMS
Description :

DriveLetter     : J
DriveType       : Fixed
FileSystem      : NTFS
FileSystemLabel :
HealthStatus    : Healthy
ObjectId        : \\?\Volume{b02c4bb1-e6f1-11e1-93eb-0008a1c0ef0d}\
Path            : \\?\Volume{b02c4bb1-e6f1-11e1-93eb-0008a1c0ef0d}\
Size            : 107372081152
SizeRemaining   : 84665225216
PSComputerName  :

Name        : VMS3
ScopeName   : FST2-SO
Path        : C:\ClusterStorage\Volume1\VMS
Description :

DriveLetter     :
DriveType       : Fixed
FileSystem      : CSVFS
FileSystemLabel :
HealthStatus    : Healthy
ObjectId        : \\?\Volume{20795fea-b7da-43dd-81d7-4d346c337a73}\
Path            : \\?\Volume{20795fea-b7da-43dd-81d7-4d346c337a73}\
Size            : 107372081152
SizeRemaining   : 85759995904
PSComputerName  :

Name        : VMS4
ScopeName   : FST2-SO
Path        : C:\ClusterStorage\Volume2\VMS
Description :

DriveLetter     :
DriveType       : Fixed
FileSystem      : CSVFS
FileSystemLabel :
HealthStatus    : Healthy
ObjectId        : \\?\Volume{fb69e20a-5d6a-4dc6-a0e9-750291644165}\
Path            : \\?\Volume{fb69e20a-5d6a-4dc6-a0e9-750291644165}\
Size            : 107372081152
SizeRemaining   : 84665225216
PSComputerName  :

Name        : VMS5
ScopeName   : *
Path        : D:\VMS
Description :

DriveLetter     : D
DriveType       : Fixed
FileSystem      : NTFS
FileSystemLabel : LocalFS1
HealthStatus    : Healthy
ObjectId        : \\?\Volume{58a38e4e-e2fd-11e1-93e8-806e6f6e6963}\
Path            : \\?\Volume{58a38e4e-e2fd-11e1-93e8-806e6f6e6963}\
Size            : 181336535040
SizeRemaining   : 136311140352
PSComputerName  :

Finally, not for the faint of heart, a more complex query from a remote file server which creates a custom result combining share information and volume information:

PS C:\> Get-SmbShare -CimSession FST2-FS2 | ? Volume  -ne $null | % { $R = "" | Select Share, Path, Size, Free; $R.Share=$_.Name; $R.Path=$_.Path; Get-Volume -CimSession FST2-FS2 -Id $_.Volume | % { $R.Size=$_.Size; $R.Free=$_.SizeRemaining; $R | FL }}

Share : ADMIN$
Path  : C:\Windows
Size  : 68352471040
Free  : 44692242432

Share : C$
Path  : C:\
Size  : 68352471040
Free  : 44692242432

Share : ClusterStorage$
Path  : C:\ClusterStorage
Size  : 68352471040
Free  : 44692242432

Share : D$
Path  : D:\
Size  : 181336535040
Free  : 177907777536

Share : Projects
Path  : C:\ClusterStorage\Volume1\SHARES\PROJECTS
Size  : 107372081152
Free  : 85759995904

Share : VMFiles
Path  : C:\ClusterStorage\Volume1\VMFiles
Size  : 107372081152
Free  : 85759995904

Share : VMS3
Path  : C:\ClusterStorage\Volume1\VMS
Size  : 107372081152
Free  : 85759995904

Share : VMS4
Path  : C:\ClusterStorage\Volume2\VMS
Size  : 107372081152
Free  : 84665225216

Share : Witness
Path  : C:\ClusterStorage\Volume1\Witness
Size  : 107372081152
Free  : 85759995904

Windows Server 2012 and Windows 8 include a new set of performance counters that can greatly help understand the performance of the SMB file protocol. These include new counters on both the server side and the client side.

In this post, I wanted to call your attention to the new client-side counters that show the traffic going to a specific file share. In the example below, we see the performance counters on a computer called FST2-HV1, which is accessing some files on the VMS1 share on a file server cluster called FST2-FS. This is actually a Hyper-V over SMB scenario, where a VM is storing its VHDX file on a remote file share. I started a light copy inside the VM to generate some activity.

This simple set of counters offers a lot of information. If you know how to interpret it, you can get information on IOPS (data requests per second), throughput (data bytes per second), latency (average seconds per request), IO size (average bytes per request) and outstanding IOs (data queue length) . You can also get all this information for read and write operations.  There’s a lot you can tell from these data points. You can see we’re doing about 910 IOPS and we’re doing more writes than reads (648 write IOPS vs. 262 read IOPS). However, we’re doing larger reads than writes (in average, 104KB reads vs. 64KB writes). Reads are faster than writes (5 milliseconds vs. 8 milliseconds). In case you’re wondering, the workload running here was a simple copy of lots of fairly large files from one folder to another inside the VM.

If you’re familiar with the regular disk counters in Windows, you might notice a certain resemblance. That’s not by accident. The SMB client shares performance counters were designed to exactly match the disk counters. This way you can easily reuse any guidance on application disk performance tuning you currently have . Here’s a little table comparing the two sets:

Type	Disk Object Counter	SMB Client Shares Counter
IOPS	Disk transfers / sec	Data Requests/sec
	Disk reads / sec	Read Requests/sec
	Disk writes / sec	Write Requests/sec
Latency	Avg disk sec / transfer	Avg. sec/Data Request
	Avg disk sec / read	Avg. sec/Read
	Avg disk sec / write	Avg. sec/Write
IO Size	Avg disk bytes / transfer	Avg. Bytes/Data Request
	Avg disk bytes / read	Avg. Bytes/Read
	Avg disk bytes / write	Avg. Bytes/Write
Throughput	Disk bytes / sec	Data Bytes/sec
	Disk read bytes / sec	Read Bytes/sec
	Disk write bytes / sec	Write Bytes/sec
Queue Length	Avg. Disk Read Queue Length	Avg. Read Queue Length
	Avg. Disk Write Queue Length	Avg. Write Queue Length
	Avg Disk Queue Length	Avg. Data Queue Length

I would encourage you to understand your application storage performance and capture/save some samples after you tune your applications. This way you can build a library of baseline performance counters that you can later use for troubleshooting performance issues.

There is a new ESG report out that shows the Storage and Networking performance of Windows Server 2012.

It highlights the findings around a few key new features, including:

• Storage Spaces
• SMB 3.0 File Servers
• Deduplication
• CHKDSK Online Scanning
• Offloaded Data Transfers (ODX)

The numbers speak for themselves and the report provides plenty of tables, charts and configuration details.

Check it out at http://download.microsoft.com/download/8/0/F/80FCCBEF-BC4D-4B84-950B-07FBE31022B4/ESG-Lab-Validation-Windows-Server-Storage.pdf

1. Introduction

If you follow this blog, you probably already had a chance to review the “Hyper-V over SMB” overview talk that I delivered at TechEd 2012 and other conferences. I am now working on a new version of that talk that still covers the basics, but adds brand new segments focused on end-to-end performance and detailed sample configurations. This post looks at this new end-to-end performance portion.

2. Typical Hyper-V over SMB configuration

End-to-end performance starts by drawing an end-to-end configuration. The diagram below shows a typical Hyper-V over SMB configuration including:

• Clients that access virtual machines
• Nodes in a Hyper-V Cluster
• Nodes in a File Server Cluster
• SAS JBODs acting as shared storage for the File Server Cluster

The main highlights of the diagram above include the redundancy in all layers and the different types of network connecting the layers.

3. Performance considerations

With the above configuration in mind, you can then start to consider the many different options at each layer that can affect the end-to-end performance of the solution. The diagram below highlights a few of the items, in the different layers, that would have a significant impact.

These items include, in each layer:

• Clients
  • Number of clients
  • Speed of the client NICs
• Virtual Machines
  • VMs per host
  • Virtual processors and RAM per VM
• Hyper-V Hosts
  • Number of Hyper-V hosts
  • Cores and RAM per Hyper-V host
  • NICs per Hyper-V host (connecting to clients) and the speed of those NICs
  • RDMA NICs (R-NICs) per Hyper-V host (connecting to file servers) and the speed of those NICs
• File Servers
  • Number of File Servers (typically 2)
  • RAM per File Server, plus how much is used for CSV caching
  • Storage Spaces configuration, including number of spaces, resiliency settings and number of columns per space
  • RDMA NICs (R-NICs) per File Server (connecting to Hyper-V hosts) and the speed of those NICs
  • SAS HBAs per File Server (connecting to the JBODs) and speed of those HBAs
• JBODs
  • SAS ports per module and the speed of those ports
  • Disks per JBOD, plus the speed of the disks and of their SAS connections

It’s also important to note that the goal is not to achieve the highest performance possible, but to find a balanced configuration that delivers the performance required by the workload at the best possible cost.

4. Sample configuration

To make things a bit more concrete, you can look at a sample VDI workload. Suppose you need to create a solution to host 500 VDI VMs. Here is an outline of the thought process you would have to go through:

• Workload, disks, JBODs, hosts
  • Start with the stated workload requirements: 500 VDI VMs, 2GB RAM, 1 virtual processor, ~50GB per VM, ~30 IOPS per VM, ~64KB per IO
  • Next, select the type of disk to use: 900 GB HDD at 10,000 rpm, around 140 IOPS
  • Also, the type of JBOD to use: SAS JBOD with dual SAS modules, two 4-lane 6Gbps port per module, up to 60 disks per JBOD
  • Finally, this is the agreed upon spec for the Hyper-V host: 16 cores, 128GB RAM
• Storage
  • Number of disks required based on IOPS: 30 * 500 /140 = ~107 disks
  • Number of disks required based on capacity: 50GB * 2 * 500 / 900 = ~56 disks.
  • Some additional capacity is required for snapshots and backups.
  • It seems like we need 107 disks for IOPS to fulfill both the IOPS and capacity requirements
  • We can then conclude we need 2 JBODs with 60 disks each (that would give us 120 disks, including some spares)
• Hyper-V hosts
  • 2 GB VM / 128GB = ~ 50 VM/host – leaving some RAM for host
  • 50 VMs * 1 virtual procs / 16 cores = ~ 3:1 ratio between virtual and physical processors.
  • 500 VMs / 50 = ~ 10 hosts – We could use 11 hosts, filling all the requirements plus one as spare
• Networking
  • 500 VMs*30 IOPS*64KB = 937 MBps required – This works well with a single 10GbE which can deliver 1100 MBps . 2 for fault tolerance.
  • Single 4-lane SAS at 6Gbps delivers 2200 MBps. 2 for fault tolerance. You could actually use 3Gbps SAS HBAs here if you wanted.
• File Server
  • 500 * 25 IOPS = 12,500 IOPS. Single file server can deliver that without any problem. 2 for fault tolerance.
  • RAM = 64GB, good size that allows for some CSV caching (up to 20% of RAM)

Please note that this is simply as an example, since your specific workload requirements may vary. There’s no general industry agreement on exactly what a VDI workload looks like, which kind of disks should be used with it or how much RAM would work best for the Hyper-V hosts in this scenario. So, take this example with a grain of salt :-)

Obviously you could have decided to go with a different type of disk, JBOD or host. In general, higher-end equipment will handle more load, but will be more expensive. For disks, deciding factors will include price, performance, capacity and endurance. Comparing SSDs and HDDs, for instance, is an interesting exercise and that equation changes constantly as new models become available and prices fluctuate. You might need to repeat the above exercise a few times with different options to find the ideal solution for your specific workload. You might want to calculate your cost per VM for each specific iteration.

Assuming you did all that and liked the results, let’s draw it out:

Now it’s up to you to work out the specific details of your own workload and hardware options.

5. Configuration Variations

It’s also important to notice that there are several potential configuration variations for the Hyper-V over SMB scenario, including:

• Using a regular Ethernet NICs instead of RDMA NICs between the Hyper-V hosts and the File Servers
• Using a third-party SMB 3.0 NAS instead of a Windows File Server
• Using Fibre Channel or iSCSI instead of SAS, along with a traditional SAN instead of JBODs and Storage Spaces

6. Speeds and feeds

In order to make some of the calculations, you might need to understand the maximum theoretical throughput of the interfaces involved. For instance, it helps to know that a 10GbE NIC cannot deliver up more than 1.1 GBytes per second or that a single SAS HBA sitting on an 8-lane PCIe Gen2 slot cannot deliver more than 3.4 GBytes per second. Here are some tables to help out with that portion:

Also, here is some fine print on those tables:

• Only a few common configurations listed.
• All numbers are rough approximations.
• Actual throughput in real life will be lower than these theoretical maximums.
• Numbers provided are for one way traffic only (you should double for full duplex).
• Numbers are for one interface and one port only.
• Numbers use base 10 (1 GB/sec = 1,000,000,000 bytes per second)

7. Conclusion

I’m still working out the details of this new Hyper-V over SMB presentation, but this posts summarizes the portion related to end-to-end performance.

I plan to deliver this talk to an internal Microsoft audience this week and also during the MVP Summit later this month. I am also considering submissions for MMS 2013 and TechEd 2013.

P.S.: You can get a preview of this portion of the talk by watching this recent TechNet Radio show I recorded with Bob Hunt: Hyper-V over SMB 3.0 Performance Considerations.

P.P.S: This content was delivered as part of my TechEd 2013 North America talk: Understanding the Hyper-V over SMB Scenario, Configurations, and End-to-End Performance

Highly available Windows Server 2012 systems using shared, directly-attached storage can be built using either Storage Spaces or a validated clustered RAID controller.

Option 1 – Storage Spaces

You can build a highly available shared SAS system today using Storage Spaces.

Storage Spaces works well in a standalone PC, but it is also capable of working in a Windows Server Failover Clustering environment. 

For implementing Clustered Storage Spaces, you will need the following Windows Server 2012 certified hardware:

• Any SAS Host Bus Adapter or HBA (as long as it’s SAS and not a RAID controller, you should be fine)
• SAS JBODs or disk enclosures (listed under the “Storage Spaces” category on the Server catalog)
• SAS disks (there’s a wide variety of those, including capacity HDDs, performance HDDs and SSDs)

You can find instructions on how to configure a Clustered Storage Space in Windows Server 2012 at http://blogs.msdn.com/b/clustering/archive/2012/06/02/10314262.aspx.

A good overview of Storage Spaces and its capabilities can be found at http://social.technet.microsoft.com/wiki/contents/articles/15198.storage-spaces-overview.aspx

There's also an excellent presentation from TechEd that covers Storage Spaces at http://channel9.msdn.com/Events/TechEd/NorthAmerica/2012/WSV315

 

Option 2 – Clustered RAID Controllers

The second option is to build a highly available shared storage system using RAID Controllers that are designed to work in a Windows Server Failover Cluster configuration.

The main distinction between these RAID controllers and the ones we used before is that they work in sets (typically a pair) and coordinate their actions against the shared disks.

Here are some examples:

• The HP StoreEasy 5000 cluster-in-a-box uses Clustered RAID controllers that HP sources and certifies. You can find details at the HP StoreEasy product page.
• LSI is working on a Clustered RAID controller with Windows Server 2012 support. This new line of SAS RAID Controllers is scheduled for later this year. You can get details on availability dates from LSI.

Both options work great for all kinds of Windows Server 2012 Clusters, including Hyper-V Clusters, SQL Server Clusters, Classic File Server Clusters and Scale-Out File Servers.

You can learn more about these solutions in this TechEd presentation: http://channel9.msdn.com/Events/TechEd/Europe/2012/WSV310

Introduction

Increasing availability is a key concern with computer systems. With all the consolidation and virtualization efforts under way, you need to make sure your services are always up and running, even when some components fail. However, it’s usually hard to understand the details of what it takes to make systems highly available (or continuously available). And there are so many options…

In this blog post, I will describe four principles that cover the different requirements for Availability: Redundancy, Entanglement, Awareness and Persistence. They apply to different types of services and I’ll provide some examples related to the most common server roles, including DHCP, DNS, Active Directory, Hyper-V, IIS, Remote Desktop Services, SQL Server, Exchange Server, and obviously File Services (I am in the “File Server and Clustering” team, after all). Every service employs different strategies to implement these “REAP Principles” but they all must implement them in some fashion to increase availability.

Note: A certain familiarity with common Windows Server roles and services is assumed here. If you are not familiar with the meaning of DHCP, DNS or Active Directory, this post is not intended for you. If that’s the case, you might want to do some reading on those topics before moving forward here.

Redundancy – There is more than one of everything

Availability starts with redundancy. In order to provide the ability to survive failures, you must have multiple instance of everything that can possibly fail in that system. That means multiple servers, multiple networks, multiple power supplies, multiple storage devices. You should be seeing everything (at least) doubled in your configuration. Whatever is not redundant is commonly labeled a “Single Point of Failure”.

Redundancy is not cheap, though. By definition, it will increase the cost of your infrastructure. So it’s an investment that can only be justified when there is understanding of the risks and needs associated with service disruption, which should be balanced with the cost of higher availability. Sadly, that understanding sometimes only comes after a catastrophic event (such as data loss or an extended outage).

Ideally, you would have a redundant instance that is as capable as your primary one. That would make your system work as well after the failure as it did before. It might be acceptable, though, to have a redundant component that is less capable. In that case, you’ll be in a degraded (although functional) state after a failure, while the original part is being replaced. Also keep in mind that, these days, redundancy in the cloud might be a viable option.

For this principle, there’s really not much variance per type of Windows Server role. You basically need to make sure that you have multiple servers providing the service, and make sure the other principles are applied.

Entanglement – Achieving shared state via spooky action at a distance

Having redundant equipment is required but certainly not sufficient to provide increased availability. Once any meaningful computer system is up and running, it is constantly gathering information and keeping track of it. If you have multiple instances running, they must be “entangled” somehow. That means that the current state of the system should be shared across the multiple instances so it can survive the loss of any individual component without losing that state. It will typically include some complex “spooky action at a distance”, as Einstein famously said of Quantum Mechanics.

A common way to do it is using a database (like SQL Server) to store your state. Every transaction performed by a set of web servers, for instance, could be stored in a common database and any web server can be quickly reprovisioned and connected to the database again. In a similar fashion, you can use Active Directory as a data store, as it’s done by services like DFS Namespaces and Exchange Server (for user mailbox information). Even a file server could serve a similar purpose, providing a location to store files that can be changed at any time and accessed by a set of web servers. If you lose a web server, you can quickly reprovision it and point it to the shared file server.

If using SQL Server to store the shared state, you must also abide by the Redundancy principle by using multiple SQL Servers, which must be entangled as well. One common way to do it is using shared storage. You can wire these servers to a Fibre Channel SAN or an iSCSI SAN or even a file server to store the data. Failover clustering in Windows Server (used by certain deployments of Hyper-V, File Servers and SQL Server, just to name a few) levarages shared storage as a common mechanism for entanglement.

Peeling the onion further, you will need multiple heads of those storage systems and they must also be entangled. Redundancy at the storage layer is commonly achieved by sharing physical disks and writing the data to multiple places. Most SANs have the option of using dual controllers that are connected to a shared set of disks. Every piece of data is stored synchronously to at least two disks (sometimes more). These SANs can tolerate the failure of individual controllers or disks, preserving their shared state without any disruption. In Windows Server 2012, Clustered Storage Spaces provides a simple solution for shared storage for a set of Windows Servers using only Shared SAS disks, without the need for a SAN.

There are other strategies for Entanglement that do not require shared storage, depending on how much and how frequently the state changes. If you have a web site with only static files, you could maintain shared state by simply provisioning multiple IIS servers with the exact same files. Whenever you lose one, simply replace it. For instance, Windows Azure and Virtual Machine Manager provide mechanisms to quickly add/remove instances of web servers in this fashion through the use of a service template.

If the shared state changes, which is often the case for most web sites, you could go up a notch by regularly copying updated files to the servers. You could have a central location with the current version of the shared state (a remote file server, for instance) plus a process to regularly send full updates to any of the nodes every day (either pushed from the central store or pulled by the servers). This is not very efficient for large amounts of data updated frequently, but could be enough if the total amount of data is small or it changes very infrequently. Examples of this strategy include SQL Server Snapshot Replication, DNS full zone transfers or a simple script using ROBOCOPY to copy files on a daily schedule.

In most cases, however, it’s best to employ a mechanism that can cope with more frequently changing state. Going up the scale you could have a system that sends data to its peers every hour or every few minutes, being careful to send only the data that has changed instead of the full set. That is the case for DNS incremental zone transfers, Active Directory Replication, many types of SQL Server Replication, SQL Server Log Shipping, Asynchronous SQL Server Mirroring (High-Performance Mode), SQL Server AlwaysOn Availability Groups (asynchronous-commit mode), DFS Replication and Hyper-V Replica. These models provide systems that are loosely converging, but do not achieve up-to-the-second coherent shared state. However, that is good enough for some scenarios.

At the high end of replication and right before actual shared storage, you have synchronous replication. This provides the ability to update the information on every entangled system before considering the shared state actually changed. This might slow down the overall performance of the system, especially when the connectivity between the peers suffers from latency. However, there’s something to be said of just having a set of nodes with local storage that achieve a coherent shared state using only software. Common examples here include a few types of SAN replication, Exchange Server (Database Availability Groups), Synchronous SQL Mirroring (High Safety Mode) and SQL Server AlwaysOn Availability Groups (synchronous-commit mode).

As you can see, the Entanglement principle can be addressed in a number of different ways depending on the service. Many services, like File Server and SQL Server, provide multiple mechanisms to deal with it, with varying degrees of cost, complexity, performance and coherence.

Awareness – Telling if Schrödinger's servers are alive or not

Your work is not done after you have a redundant entangled system. In order to provide clients with seamless access to your service, you must implement some method to find one of the many sources for the service. The awareness principle refers to how your clients will discover the location of the access points for your service, ideally with a mechanism to do it quickly while avoiding any failed instances. There a few different ways to achieve it, including manual configuration, broadcast, DNS, load balancers, or a service-specific method.

One simple method is to statically configure each client with the name or IP Address of two or more instances of the service. This method is effective if the configuration of the service is not expected to change. If it ever does change, you would need to reconfigure each client. A common example here is how static DNS is configured: you simply specify the IP address of your preferred DNS server and also the IP address if an alternate DNS server in case the preferred one fails.

Another common mechanism is to broadcast a request for the service and wait for a response. This mechanism works only if there’s someone in your local network capable of providing an answer. There’s also a concern about the legitimacy of the response, since a rogue system on the network might be used to provide a malicious version of the service. Common examples here include DHCP service requests and Wireless Access Point discovery. It is fairly common to use one service to provide awareness for others. For instance, once you access your Wireless Access Point, you get DHCP service. Once you get DHCP service, you get your DNS configuration from it.

As you know, the most common use for a DNS server is to map a network name to an IP address (using an A, AAAA or CNAME DNS record). That in itself implements a certain level of this awareness principle. DNS can also associate multiple IP addresses with a single name, effectively providing a mechanism to give you a list of servers that provide a specific service. That list is provided by the DNS server in a round robin fashion, so it even includes a certain level of load balancing as part of it. Clients looking for Web Servers and File Servers commonly use this mechanism alone for finding the many devices providing a service.

DNS also provides a different type of record specifically designed for providing service awareness. This is implemented as SRV (Service) records, which not only offer the name and IP address of a host providing a service, but can decorate it with information about priority, weight and port number where the service is provided. This is a simple but remarkably effective way to provide service awareness through DNS, which is effectively a mandatory infrastructure service these days. Active Directory is the best example of using SRV records, using DNS to allow clients to learn information about the location of Domain Controllers and services provided by them, including details about Active Directory site topology.

Windows Server failover clustering includes the ability to perform dynamic DNS registrations when creating clustered services. Each cluster role (formerly known as a cluster group) can include a Network Name resource which is registered with DNS when the service is started. Multiple IP addresses can be registered for a given cluster Network  Name if the server has multiple interfaces. In Windows Server 2012, a single cluster role can be active on multiple nodes (that’s the case of a Scale-Out File Server) and the new Distributed Network Name implements this as a DNS name with multiple IP addresses (at least one from each node).

DNS does have a few limitations. The main one is the fact that the clients will cache the name/IP information for some time, as specified in the TTL (time to live) for the record. If the service is reconfigure and new addresses or service records are published, DNS clients might take some time to become aware of the change. You can reduce the TTL, but that has a performance impact, causing DNS clients to query the server more frequently. There is no mechanism in DNS to have a server proactively tell a client that a published record has changed. Another issue with DNS is that it provides no method to tell if the service is actually being provided at the moment or even if the server ever functioned properly. It is up to the client to attempt communication and handle failures. Last but not least, DNS cannot help with intelligently balancing clients based on the current load of a server.

Load balancers are the next step in providing awareness. These are network devices that function as an intelligent router of traffic based on a set of rules. If you point your clients to the IP address of the load balancer, that device can intelligently forward the requests to a set for servers. As the name implies, load balancers typically distribute the clients across the servers and can even detect if a certain server is unresponsive, dynamically taking it out of the list. Another concern here is affinity, which is an optimization that consistently forwards a given client to the same server. Since these devices can become a single point of failure, the redundancy principle must be applied here. The most common solution is to have two load balancers in combination with two records in DNS.

SQL Server again uses multiple mechanisms for implementing this principle. DNS name resolution is common, both statically or dynamically using failover clustering Network Name resources. That name is then used as part of the client configuration known as a “Connection String”. Typically, this string will provide the name of a single server providing the SQL Service, along with the database name and authentication details. For instance, a typical connection string would be: "Server=SQLSERV1A; Database=DB301; Integrated Security=True;". For SQL Mirroring, there is a mechanism to provide a second server name in the connection string itself. Here’s an example: "Server=SQLSERV1A; Failover_Partner=SQLSRV1B; Database=DB301; Integrated Security=True;".

Other services provide a specific layers of Awareness, implementing a broker service or client access layer. This is the case of DFS (Distributed File System), which simplifies access to multiple file servers using a unified namespace mechanism. In a similar way, SharePoint web front end servers will abstract the fact that multiple content databases live behind a specific SharePoint farm or site collection. SharePoint Server 2013 goes one step further by implementing a Request Manager service that can even be configure as a Web Server farm placed in front of the main SharePoint web front end farm, with the purpose of routing and throttling incoming requests to improve both performance and availability.

Exchange Server Client Access Servers will query Active Directory to find which Mailbox Server or Database Access Group contains the mailbox for an incoming client. Remote Desktop Connection Broker (formerly known as Terminal Services Session Broker), is used to provide users with access to Remote Desktop services across a set of servers. All these brokers services can typically handle a fair amount of load balancing and be aware of the state of the services behind it. Since these can become single point of failures, they are typically placed behind DNS round robin and/or load balancers.

Persistence – The one that is the most adaptable to change will survive

Now that you have redundant entangled services and clients are aware of them, here comes the greatest challenge in availability. Persisting the service in the event of a failure. There are three basic steps to make it happen: server failure detection, failing over to a surviving server (if required) and client reconnection (if required).

Detecting the failure is the first step. It requires a mechanism for aliveness checks, which can be performed by the servers themselves, by a witness service, by the clients accessing the services or a combination of these. For instance, Windows Server failover clustering makes cluster nodes check each other (through network checks), in an effort to determine when a node becomes unresponsive.

Once a failure is detected, for services that work in an active/passive fashion (only one server provides the service and the other remains on standby), a failover is required. This can only be safely achieved automatically if the entanglement is done via Shared Storage or Synchronous Replication, which means that the data from the server that is lost is properly persisted. If using other entanglement methods (like backups or asynchronous replication), an IT Administrator typically has to manually intervene to make sure the proper state is restored before failing over the service. For all active/active solutions, with multiple servers providing the same service all the time, a failover is not required.

Finally, the client might need to reconnect to the service. If the server being used by the client has failed, many services will lose their connections and require intervention. In an ideal scenario, the client will automatically detect (or be notified of) the server failure. Then, because it is aware of other instances of the service, it will automatically connect to a surviving instance, restoring the exact same client state before the failure. This is how Windows Server 2012 implements failover of file servers though a process called SMB 3.0 Continuous Availability, available for both Classic and Scale-Out file server clusters. The file server cluster goes one step further, providing a Witness Service that will proactively notify SMB 3.0 clients of a server failure and point them to an alternate server, even before current pending requests to the failed server time out.

File servers might also leverage a combination of DFS Namespaces and DFS Replication that will automatically recover from a failed server situation, with some potential side effects. While the file client will find an alternative file server via DFS Namespaces, the connection state will be lost and need to be reestablished. Another persistence mechanism in the file server is the Offline Files option (also known as Client Side Caching) commonly used with the Folder Redirection feature. This allows you to keep working on local storage while your file server is unavailable, synchronizing again when the server comes back.

For other services, like SQL Server, the client will surface an error to the application indicating that a failover has occurred and the connection has been lost. If the application is properly coded to handle that situation, the end user will be shielded from error message because the application will simply reconnect to the SQL Server using either the same name (in the case of another server taking over that name) or a Failover Partner name (in case of SQL Server Mirroring) or another instance of SQL Server (in case of more complex log shipping or replication scenarios).

Clients of Web Servers and other load balanced workloads without any persistent state might be able to simply retry an operation in case of a failure. This might happen automatically or require the end-user to retry the operation manually. This might also be the case of a web front end layer that communicates with a web services layer. Again a savvy programmer could code that front end server to automatically retry web services requests, if they are idempotent.

Another interesting example of client persistence is provided by an Outlook client connecting to an Exchange Server. As we mentioned, Exchange Servers implement a sophisticated method of synchronous replication of mailbox databases between servers, plus a Client Access layer that brokers connections to the right set of mailbox servers. On top of that, the Outlook client will simply continue to work from its cache (using only local storage) if for any reason the server becomes unavailable. Whenever the server comes back online, the client will transparent reconnect and synchronize. The entire process is automated, without any action required during or after the failure from either end users and IT Administrators.

Samples of how services implement the REAP principles

Now that you have the principles down, let’s look at how the main services we mentioned implement them.

Conclusion

I hope this post helped you understand the principles behind increasing server availability.

As a final note, please take into consideration that not all services require the highest possible level of availability. This might be an easier decision for certain services like DHCP, DNS and Active Directory, where the additional cost is relatively small and the benefits are sizable. You might want to think twice when increasing the availability of a large backup server, where some hours of down time might be acceptable and the cost of duplicating the infrastructure is significantly higher.

Depending on how much availability you service level agreement states, you might need different types of solutions. We generally measure availability in “nines”, as described in the table below:

You should consider your overall requirements and the related infrastructure investments that would give you the most “nines” per dollar.

1. Introduction

I have been doing storage-related demos and publishing blogs with some storage performance numbers for a while, and I commonly get questions such as “How do you run these tests?” or “What tools do you use to generate IOs for your demos?”. While it’s always best to use a real workload to test storage, sometimes that is not convenient. So, I very frequently use a free tool from Microsoft to simulate IOs called SQLIO. It’s a small and simple tool that simulate several types of workloads, including common SQL Server ones. And you can apply it to several configurations, from a physical host or virtual machine, using a local disk, a LUN on a SAN, a Storage Space or an SMB file share.

2. Download the tool

To get started, you need to download and install the SQLIO tool. You can get it from http://www.microsoft.com/en-us/download/details.aspx?id=20163. The setup will install the tool in a folder of your choice. In the end, you really only need one file: SQLIO.EXE. You can copy it to any folder and it runs in pretty much every Windows version, client or server. In this blog post, I assume that you installed SQLIO on the C:\SQLIO folder.

3. Prepare a test file

Next, you need to create a file in the disk or file share that you will be using for your demo or test.

Ideally, you should create a file as big as possible, so that you can exercise the entire disk. For hard disks, creating a small file causes the head movement to be restricted to a portion of the disk. Unless you’re willing to use only a fraction of the hard disk capacity, these numbers show unrealistically high random IO performance. Storage professionals call this technique “short stroking”. For SANs, small files might end up being entirely cached in the controller RAM, again giving you great numbers that won’t hold true for real deployments. You can actually use SQLIO to measure the difference between using a large file and a small file for your specific configuration.

To create a large file for your test, the easiest way is using the FSUTIL.EXE tool, which is included with all versions of Windows.

For instance, to create a 1TB file on the X: drive, using the following command from a PowerShell prompt:

FSUTIL.EXE file createnew X:\testfile.dat (1TB)
FSUTIL.EXE file setvaliddata X:\testfile.dat (1TB)

Note 1: You must do this from PowerShell, in order to use the convenient (1TB) notation. If you run this from an old command prompt, you need to calculate 1 terabyte in bytes, which is 1099511627776 (2^40). Before you Storage professionals rush to correct me, I know this is technically incorrect. One terabyte is 10^12 (1000000000000) and 2^40 is actually one Tebibyte (1TiB). However, since both PowerShell and SQLIO use the TB/GB/MB/KB notation when referring to powers of 2, I will ask you to give me a pass here.

Note 2: The “set valid data” command lets you move the “end of file” marker, avoiding a lengthy initialization of the file. This is much faster than writing over the entire file. However, there are security implications for “set valid data” (it might expose leftover data on the disk if you don’t properly initialize the file) and you must be an administrator on the machine to use it.

Here’s another example, with output, using a smaller file size:

PS C:\> FSUTIL.EXE File CreateNew X:\TestFile.DAT (40GB)
File X:\TestFile.DAT is created
PS C:\> FSUTIL.EXE File SetValidData X:\TestFile.DAT (40GB)
Valid data length is changed

4. Run the tool

With the tool installed and the test file created, you can start running SQLIO.

You also want to make sure there’s nothing else running on the computer, so that other running process don’t interfere with your results by putting additional load on the CPU, network or storage. If the disk you are using is shared in any way (like a LUN on a SAN), you want to make sure that nothing else is competing with your testing. If you’re using any form of IP storage (iSCSI LUN, SMB file share), you want to make sure that you’re not running on a network congested with other kinds of traffic.

WARNING: You could be generating a whole lot of disk IO, network traffic and/or CPU load when you run SQLIO. If you’re in a shared environment, you might want to talk to your administrator and ask permission. This could generate a whole lot of load and disturb anyone else using other VMs in the same host, other LUNs on the same SAN or other traffic on the same network.

From an old command prompt or a PowerShell prompt, issue a single command line to start getting some performance results. Here is your first example, with output, generating random 8KB reads on that file we just created:

PS C:\> C:\SQLIO\SQLIO.EXE -s10 -kR -frandom -b8 -t8 -o16 -LS -BN X:\TestFile.DAT
sqlio v1.5.SG
using system counter for latency timings, 2337894 counts per second
8 threads reading for 10 secs from file X:\TestFile.DAT
        using 8KB random IOs
        enabling multiple I/Os per thread with 16 outstanding
        buffering set to not use file nor disk caches (as is SQL Server)
using current size: 40960 MB for file: X:\TestFile.DAT
initialization done
CUMULATIVE DATA:
throughput metrics:
IOs/sec: 36096.60
MBs/sec:   282.00
latency metrics:
Min_Latency(ms): 0
Avg_Latency(ms): 3
Max_Latency(ms): 55
histogram:
ms: 0  1  2  3  4  5  6  7  8  9 10 11 12 13 14 15 16 17 18 19 20 21 22 23 24+
%: 30 19 12  8  6  5  4  3  3  2  2  2  1  1  1  1  0  0  0  0  0  0  0  0  0

So, for this specific disk (a simple Storage Space created from a pool of 3 SSDs), I am getting over 36,000 IOPs of 8KB each with an average of 3 milliseconds of  latency (time it takes for the operation to complete, from start to finish). Not bad in terms of IOPS, but the latency for 8KB IOs seems a little high for SSD-based storage. We’ll investigate that later.

Let’s try now another command using sequential 512KB reads on that same file:

PS C:\> C:\SQLIO\SQLIO.EXE -s10 -kR -fsequential -b512 -t2 -o16 -LS -BN X:\TestFile.DAT
sqlio v1.5.SG
using system counter for latency timings, 2337894 counts per second
2 threads reading for 10 secs from file X:\TestFile.DAT
        using 512KB sequential IOs
        enabling multiple I/Os per thread with 16 outstanding
        buffering set to not use file nor disk caches (as is SQL Server)
using current size: 40960 MB for file: X:\TestFile.DAT
initialization done
CUMULATIVE DATA:
throughput metrics:
IOs/sec:  1376.09
MBs/sec:   688.04
latency metrics:
Min_Latency(ms): 6
Avg_Latency(ms): 22
Max_Latency(ms): 23
histogram:
ms: 0  1  2  3  4  5  6  7  8  9 10 11 12 13 14 15 16 17 18 19 20 21 22 23 24+
%:  0  0  0  0  0  0  0  0  0  0  0  0  0  0  0  0  0  0  0  0  0  0 33 67  0

I got about 688 MB/sec with an average latency of 22 milliseconds per IO. Again, good throughput, but the latency looks high for SSDs. We’ll dig deeper in a moment.

5. Understand the parameters used

Now let’s inspect the parameters on those SQLIO command line. I know it’s a bit overwhelming at first, so we’ll go slow. And keep in mind that, for SQLIO parameters, lowercase and uppercase mean different things, so be careful.

Here is the explanation for the parameters used above:

For OLTP workloads, I commonly start with 8KB random IOs, 8 threads, 16 outstanding. 8KB is the size of the page used by SQL Server for its data files. In parameter form, that would be: -frandom -b8 -t8 -o16. For reporting or OLAP workloads with large IO, I commonly start with 512KB IOs, 2 threads and 16 outstanding. 512KB is a common IO size when SQL Server loads a batch of 64 data pages when using the read ahead technique for a table scan. In parameter form, that would be: -fsequential -b512 -t2 -o16. These numbers will need to be adjusted if you machine has many cores and/or if you volume is backed up by a large number of physical disks.

If you’re curious, here are more details about parameters for SQLIO, coming from the tool’s help itself:

Usage: D:\sqlio\sqlio.exe [options] [<filename>...]
        [options] may include any of the following:
        -k<R|W>                 kind of IO (R=reads, W=writes)
        -t<threads>             number of threads
        -s<secs>                number of seconds to run
        -d<drv_A><drv_B>..      use same filename on each drive letter given
        -R<drv_A/0>,<drv_B/1>.. raw drive letters/number for I/O
        -f<stripe factor>       stripe size in blocks, random, or sequential
        -p[I]<cpu affinity>     cpu number for affinity (0 based)(I=ideal)
        -a[R[I]]<cpu mask>      cpu mask for (R=roundrobin (I=ideal)) affinity
        -o<#outstanding>        depth to use for completion routines
        -b<io size(KB)>         IO block size in KB
        -i<#IOs/run>            number of IOs per IO run
        -m<[C|S]><#sub-blks>    do multi blk IO (C=copy, S=scatter/gather)
        -L<[S|P][i|]>           latencies from (S=system, P=processor) timer
        -B<[N|Y|H|S]>           set buffering (N=none, Y=all, H=hdwr, S=sfwr)
        -S<#blocks>             start I/Os #blocks into file
        -v1.1.1                 I/Os runs use same blocks, as in version 1.1.1
        -F<paramfile>           read parameters from <paramfile>
Defaults:
        -kR -t1 -s30 -f64 -b2 -i64 -BN testfile.dat
Maximums:
        -t (threads):                   256
        no. of files, includes -d & -R: 256
        filename length:                256

6. Tune the parameters for large IO

Now the you have the basics down, we can spend some time looking at how you can refine your number of threads and queue depth for your specific configuration. This might help us figure out why we had those higher than expected latency numbers in the initial runs. You basically need to experiment with the -t and the -o parameters until you find the one that give you the best results.

Let’s start with queue depth. You first want to find out the latency for a given system with a small queue depth, like 1 or 2. For 512KB IOs, here’s what I get from my test disk with a queue depth of 1 and a thread count of 1:

PS C:\> C:\SQLIO\SQLIO.EXE -s10 -kR -fsequential -b512 -t1 -o1 -LS -BN X:\TestFile.DAT
sqlio v1.5.SG
using system counter for latency timings, 2337894 counts per second
1 thread reading for 10 secs from file X:\TestFile.DAT
        using 512KB sequential IOs
        enabling multiple I/Os per thread with 1 outstanding
        buffering set to not use file nor disk caches (as is SQL Server)
using current size: 40960 MB for file: X:\TestFile.DAT
initialization done
CUMULATIVE DATA:
throughput metrics:
IOs/sec:   871.00
MBs/sec:   435.50
latency metrics:
Min_Latency(ms): 1
Avg_Latency(ms): 1
Max_Latency(ms): 1
histogram:
ms: 0  1  2  3  4  5  6  7  8  9 10 11 12 13 14 15 16 17 18 19 20 21 22 23 24+
%:  0 100  0  0  0  0  0  0  0  0  0  0  0  0  0  0  0  0  0  0  0  0  0  0  0

For large IOs, we typically look at the throughput (in MB/sec). With 1 outstanding IO, we are at 435 MB/sec with just 1 millisecond of latency per IO. However, if you don’t queue up some IO, we’re not extracting the full throughput of the disk, since we’ll be processing the data while the disk is idle waiting for more work. Let’s see what happens if we queue up more IOs:

PS C:\> C:\SQLIO\SQLIO.EXE -s10 -kR -fsequential -b512 -t1 -o2 -LS -BN X:\TestFile.DAT
sqlio v1.5.SG
using system counter for latency timings, 2337894 counts per second
1 thread reading for 10 secs from file X:\TestFile.DAT
        using 512KB sequential IOs
        enabling multiple I/Os per thread with 2 outstanding
        buffering set to not use file nor disk caches (as is SQL Server)
using current size: 40960 MB for file: X:\TestFile.DAT
initialization done
CUMULATIVE DATA:
throughput metrics:
IOs/sec:  1377.70
MBs/sec:   688.85
latency metrics:
Min_Latency(ms): 1
Avg_Latency(ms): 1
Max_Latency(ms): 2
histogram:
ms: 0  1  2  3  4  5  6  7  8  9 10 11 12 13 14 15 16 17 18 19 20 21 22 23 24+
%:  0 100  0  0  0  0  0  0  0  0  0  0  0  0  0  0  0  0  0  0  0  0  0  0  0

OK. We are now up to 688 MB/sec with 2 outstanding IOs, and our average latency is still at the same 1 milliseconds per IO. You can also see that we now have a max latency of 2 milliseconds to complete, although in the histogram shows that most are still taking 1ms. Let’s double it up again to see what happens:

PS C:\> C:\SQLIO\SQLIO.EXE -s10 -kR -fsequential -b512 -t1 -o4 -LS -BN X:\TestFile.DAT
sqlio v1.5.SG
using system counter for latency timings, 2337894 counts per second
1 thread reading for 10 secs from file X:\TestFile.DAT
        using 512KB sequential IOs
        enabling multiple I/Os per thread with 4 outstanding
        buffering set to not use file nor disk caches (as is SQL Server)
using current size: 40960 MB for file: X:\TestFile.DAT
initialization done
CUMULATIVE DATA:
throughput metrics:
IOs/sec:  1376.70
MBs/sec:   688.35
latency metrics:
Min_Latency(ms): 1
Avg_Latency(ms): 2
Max_Latency(ms): 3
histogram:
ms: 0  1  2  3  4  5  6  7  8  9 10 11 12 13 14 15 16 17 18 19 20 21 22 23 24+
%:  0  0 67 33  0  0  0  0  0  0  0  0  0  0  0  0  0  0  0  0  0  0  0  0  0

Well, at a queue depth of 4, we gained nothing (we are still at 688 MB/sec), but our latency is now solid at 2 milliseconds, with 33% of the IOs taking 3 milliseconds. Let’s give it one more bump to see what happens. Trying now 8 outstanding IOs:

PS C:\> C:\SQLIO\SQLIO.EXE -s10 -kR -fsequential -b512 -t1 -o8 -LS -BN X:\TestFile.DAT
sqlio v1.5.SG
using system counter for latency timings, 2337894 counts per second
1 thread reading for 10 secs from file X:\TestFile.DAT
        using 512KB sequential IOs
        enabling multiple I/Os per thread with 8 outstanding
        buffering set to not use file nor disk caches (as is SQL Server)
using current size: 40960 MB for file: X:\TestFile.DAT
initialization done
CUMULATIVE DATA:
throughput metrics:
IOs/sec:  1376.50
MBs/sec:   688.25
latency metrics:
Min_Latency(ms): 2
Avg_Latency(ms): 5
Max_Latency(ms): 6
histogram:
ms: 0  1  2  3  4  5  6  7  8  9 10 11 12 13 14 15 16 17 18 19 20 21 22 23 24+
%:  0  0  0  0  0 68 32  0  0  0  0  0  0  0  0  0  0  0  0  0  0  0  0  0  0

As you can see, increasing the –o parameter is not helping. After we doubled the queue depth from 4 to 8, there was no improvement in throughput. All we did was more than double our latency to an average of 5 milliseconds, with many IOs taking 6 milliseconds. That’s when you know you’re queueing up too much IO.

So, it seems like 2 outstanding IOs is a reasonable number for this disk. Now we can see if we can gain by spreading this across multiple threads. What we want to avoide here is bottlenecking on a single CPU core, which is very common we doing lots and lots of IO. A simple experiment is to double the number of threads while halfing the queue depth.  Let’s now try 2 threads with 1 outstanding IOs each. This will give us the same 2 outstanding IOs total:

PS C:\> C:\SQLIO\SQLIO.EXE -s10 -kR -fsequential -b512 -t2 -o1 -LS -BN X:\TestFile.DAT
sqlio v1.5.SG
using system counter for latency timings, 2337894 counts per second
2 threads reading for 10 secs from file X:\TestFile.DAT
        using 512KB sequential IOs
        enabling multiple I/Os per thread with 1 outstanding
        buffering set to not use file nor disk caches (as is SQL Server)
using current size: 40960 MB for file: X:\TestFile.DAT
initialization done
CUMULATIVE DATA:
throughput metrics:
IOs/sec:  1377.90
MBs/sec:   688.95
latency metrics:
Min_Latency(ms): 1
Avg_Latency(ms): 1
Max_Latency(ms): 2
histogram:
ms: 0  1  2  3  4  5  6  7  8  9 10 11 12 13 14 15 16 17 18 19 20 21 22 23 24+
%:  0 100  0  0  0  0  0  0  0  0  0  0  0  0  0  0  0  0  0  0  0  0  0  0  0

Well, it seems like using two threads here did not buy us anything. We’re still at about the same throughput and latency. That pretty much proves that 1 thread was enough for this kind of configuration and workload. This is not surprising for large IO. However, for smaller IO size, the CPU is more taxed and we might hit a single core bottleneck. Just in case, I looked at the CPU via Task Manager and confirmed we were only using 7% of the CPU and obviously none of the 4 cores were too busy.

7. Tune queue depth for for small IO

Performing the same tuning exercise for small IO is typically more interesting. For this one, we’ll automate things a bit using a little PowerShell scripting to run SQLIO in a loop and parse its output. This way we can try a lot of different options and see which one works best. This might take a while to run, though… Here’s a script that you can run from a PowerShell prompt, trying out many different queue depths:

1..64 | % { 
   $o = "-o $_"; 
   $r = C:\SQLIO\SQLIO.EXE -s10 -kR -frandom -b8 $o -t1 -LS -BN X:\testfile.dat
   $i = $r.Split("`n")[10].Split(":")[1].Trim()
   $m = $r.Split("`n")[11].Split(":")[1].Trim()
   $l = $r.Split("`n")[14].Split(":")[1].Trim()
   $o + ", " + $i + " iops, " + $m + " MB/sec, " + $l + " ms"
}

The  script basically runs SQLIO 64 times, each time using a different queue depth, from 1 to 64. The results from SQLIO are stored in the $r variable and parsed to show IOPs, throughput and latency on a single line. There is some fun string parsing there, leveraging the Split() function to break the output by line and then again to break each line in half to get the actual numbers. Here’s the sample output from my system:

-o 1, 9446.79 iops, 73.80 MB/sec, 0 ms
-o 2, 15901.80 iops, 124.23 MB/sec, 0 ms
-o 3, 20758.20 iops, 162.17 MB/sec, 0 ms
-o 4, 24021.20 iops, 187.66 MB/sec, 0 ms
-o 5, 26047.90 iops, 203.49 MB/sec, 0 ms
-o 6, 27559.10 iops, 215.30 MB/sec, 0 ms
-o 7, 28666.40 iops, 223.95 MB/sec, 0 ms
-o 8, 29320.90 iops, 229.06 MB/sec, 0 ms
-o 9, 29733.70 iops, 232.29 MB/sec, 0 ms
-o 10, 30337.00 iops, 237.00 MB/sec, 0 ms
-o 11, 30407.50 iops, 237.55 MB/sec, 0 ms
-o 12, 30609.78 iops, 239.13 MB/sec, 0 ms
-o 13, 30843.40 iops, 240.96 MB/sec, 0 ms
-o 14, 31548.50 iops, 246.47 MB/sec, 0 ms
-o 15, 30692.10 iops, 239.78 MB/sec, 0 ms
-o 16, 30810.40 iops, 240.70 MB/sec, 0 ms
-o 17, 31815.00 iops, 248.55 MB/sec, 0 ms
-o 18, 33115.19 iops, 258.71 MB/sec, 0 ms
-o 19, 31290.40 iops, 244.45 MB/sec, 0 ms
-o 20, 32430.40 iops, 253.36 MB/sec, 0 ms
-o 21, 33345.60 iops, 260.51 MB/sec, 0 ms
-o 22, 31634.80 iops, 247.14 MB/sec, 0 ms
-o 23, 31330.50 iops, 244.76 MB/sec, 0 ms
-o 24, 32769.40 iops, 256.01 MB/sec, 0 ms
-o 25, 34264.30 iops, 267.68 MB/sec, 0 ms
-o 26, 31679.00 iops, 247.49 MB/sec, 0 ms
-o 27, 31501.60 iops, 246.10 MB/sec, 0 ms
-o 28, 33259.40 iops, 259.83 MB/sec, 0 ms
-o 29, 33882.30 iops, 264.70 MB/sec, 0 ms
-o 30, 32009.40 iops, 250.07 MB/sec, 0 ms
-o 31, 31518.10 iops, 246.23 MB/sec, 0 ms
-o 32, 33548.30 iops, 262.09 MB/sec, 0 ms
-o 33, 33912.19 iops, 264.93 MB/sec, 0 ms
-o 34, 32640.00 iops, 255.00 MB/sec, 0 ms
-o 35, 31529.30 iops, 246.32 MB/sec, 0 ms
-o 36, 33973.50 iops, 265.41 MB/sec, 0 ms
-o 37, 34174.62 iops, 266.98 MB/sec, 0 ms
-o 38, 32556.50 iops, 254.34 MB/sec, 0 ms
-o 39, 31521.00 iops, 246.25 MB/sec, 0 ms
-o 40, 34337.60 iops, 268.26 MB/sec, 0 ms
-o 41, 34455.00 iops, 269.17 MB/sec, 0 ms
-o 42, 32265.00 iops, 252.07 MB/sec, 0 ms
-o 43, 31681.80 iops, 247.51 MB/sec, 0 ms
-o 44, 34017.69 iops, 265.76 MB/sec, 0 ms
-o 45, 34433.80 iops, 269.01 MB/sec, 0 ms
-o 46, 33213.19 iops, 259.47 MB/sec, 0 ms
-o 47, 31475.20 iops, 245.90 MB/sec, 0 ms
-o 48, 34467.50 iops, 269.27 MB/sec, 0 ms
-o 49, 34529.69 iops, 269.76 MB/sec, 0 ms
-o 50, 33086.19 iops, 258.48 MB/sec, 0 ms
-o 51, 31157.90 iops, 243.42 MB/sec, 1 ms
-o 52, 34075.30 iops, 266.21 MB/sec, 1 ms
-o 53, 34475.90 iops, 269.34 MB/sec, 1 ms
-o 54, 33333.10 iops, 260.41 MB/sec, 1 ms
-o 55, 31437.60 iops, 245.60 MB/sec, 1 ms
-o 56, 34072.69 iops, 266.19 MB/sec, 1 ms
-o 57, 34352.80 iops, 268.38 MB/sec, 1 ms
-o 58, 33524.21 iops, 261.90 MB/sec, 1 ms
-o 59, 31426.10 iops, 245.51 MB/sec, 1 ms
-o 60, 34763.19 iops, 271.58 MB/sec, 1 ms
-o 61, 34418.10 iops, 268.89 MB/sec, 1 ms
-o 62, 33223.19 iops, 259.55 MB/sec, 1 ms
-o 63, 31959.30 iops, 249.68 MB/sec, 1 ms
-o 64, 34760.90 iops, 271.56 MB/sec, 1 ms

As you can see, for small IOs, we got consistently better performance as we increased the queue depth for the first few runs. After 14 outstanding IOs, adding more started giving us very little improvement until things flatten out completely. As we keept adding more queue depth, all he had was more latency with no additional benefit in IOPS or throughput. Here’s that same data on a chart:

So, in this setup, we seem to start losing steam at around 10 outstanding IOs. However, I noticed in Task Manager that one core was really busy and our overall CPU utilization was at 40%.

In this quad-core system, any overall utilization above 25% could mean there was a core bottleneck when using a single thread. Maybe we can do better with multiple threads. Let’s try increasing the number of threads with a matching reduction of queue depth so we end up with the same number of total outstanding IOs.

$o = 32
$t = 1
While ($o -ge 1) { 
   $pt = "-t $t"; 
   $po = "-o $o"; 
   $r = C:\SQLIO\SQLIO.EXE -s10 -kR -frandom -b8 $po $pt -LS -BN X:\testfile.dat
   $i = $r.Split("`n")[10].Split(":")[1].Trim()
   $m = $r.Split("`n")[11].Split(":")[1].Trim()
   $l = $r.Split("`n")[14].Split(":")[1].Trim()
   $pt + “ “ + $po + ", " + $i + " iops, " + $m + " MB/sec, " + $l + " ms"
   $o = $o / 2
   $t = $t * 2
}

Here’s the output:

-t 1 -o 32, 32859.30 iops, 256.71 MB/sec, 0 ms
-t 2 -o 16, 35946.30 iops, 280.83 MB/sec, 0 ms
-t 4 -o 8, 35734.80 iops, 279.17 MB/sec, 0 ms
-t 8 -o 4, 35470.69 iops, 277.11 MB/sec, 0 ms
-t 16 -o 2, 35418.60 iops, 276.70 MB/sec, 0 ms
-t 32 -o 1, 35273.60 iops, 275.57 MB/sec, 0 ms

As you can see, in my system, adding a second thread improved things by about 10%, reaching nearly 36,000 IOPS. It seems like we were a bit limited by the performance of a single core. We call that being “core bound”. See below the more even per-core CPU utilization when using 2 threads.

However, 4 threads did not help and the overall CPU utilization was below 50% the whole time. Here’s the full SQLIO.EXE output with my final selected parameters for 8KB random IO in this configuration:

PS C:\> C:\SQLIO\SQLIO.EXE -s10 -kR -frandom -b8 -t2 -o16 -LS -BN X:\TestFile.DAT
sqlio v1.5.SG
using system counter for latency timings, 2337894 counts per second
2 threads reading for 10 secs from file X:\TestFile.DAT
        using 8KB random IOs
        enabling multiple I/Os per thread with 16 outstanding
        buffering set to not use file nor disk caches (as is SQL Server)
using current size: 40960 MB for file: X:\TestFile.DAT
initialization done
CUMULATIVE DATA:
throughput metrics:
IOs/sec: 35917.90
MBs/sec:   280.60
latency metrics:
Min_Latency(ms): 0
Avg_Latency(ms): 0
Max_Latency(ms): 4
histogram:
ms: 0  1  2  3  4  5  6  7  8  9 10 11 12 13 14 15 16 17 18 19 20 21 22 23 24+
%: 66 26  7  1  0  0  0  0  0  0  0  0  0  0  0  0  0  0  0  0  0  0  0  0  0

For systems with more capable storage, it’s easier to get “core bound” and adding more threads can make a much more significant difference. As I mentioned, it’s important to to monitor the per-core CPU utilization via Task Manager or Performance monitor to look out for these bottlenecks.

8. Multiple runs are better than one

One thing you might have notice with SQLIO (or any other tools like it) is that the results are not always the same given the same parameters. For instance, one of our “-b8 -t2 -o16” runs yielded 35,946 IOPs while another gave us 35,917 IOPs. How can you tell which one is right? Ideally, once you settle on a specific set of parameters, you should run SQLIO a few times and average out the results. Here’s a sample PowerShell script to do it, using the last set of parameters we used for the 8KB IOs:

$ti=0
$tm=0
$tl=0
$tr=10
1..$tr | % {
   $r = C:\SQLIO\SQLIO.EXE -s10 -kR -frandom -b8 -t2 -o16 -LS -BN X:\TestFile.DAT
   $i = $r.Split("`n")[10].Split(":")[1].Trim()
   $m = $r.Split("`n")[11].Split(":")[1].Trim()
   $l = $r.Split("`n")[14].Split(":")[1].Trim()
   "Run " + $_ + " = " + $i + " IOPs, " + $m + " MB/sec, " + $l + " ms"
   $ti = $ti + $i
   $tm = $tm + $m
   $tl = $tl + $l
}
$ai = $ti / $tr
$am = $tm / $tr
$al = $tl / $tr
"Average = " + $ai + " IOPs, " + $am + " MB/sec, " + $al + " ms"

The script essentially runs SQLIO that number of times, totalling the numbers for IOPs, throughput and latency, so it can show an average at the end. The $tr variable represents the total number of runs desired. Variables starting with $t hold the totals. Variables starting with $a hold averages. Here’s a sample output:

Run 1 = 36027.40 IOPs, 281.46 MB/sec, 0 ms
Run 2 = 35929.80 IOPs, 280.70 MB/sec, 0 ms
Run 3 = 35955.90 IOPs, 280.90 MB/sec, 0 ms
Run 4 = 35963.30 IOPs, 280.96 MB/sec, 0 ms
Run 5 = 35944.19 IOPs, 280.81 MB/sec, 0 ms
Run 6 = 35903.60 IOPs, 280.49 MB/sec, 0 ms
Run 7 = 35922.60 IOPs, 280.64 MB/sec, 0 ms
Run 8 = 35949.19 IOPs, 280.85 MB/sec, 0 ms
Run 9 = 35979.30 IOPs, 281.08 MB/sec, 0 ms
Run 10 = 35921.60 IOPs, 280.63 MB/sec, 0 ms
Average = 35949.688 IOPs, 280.852 MB/sec, 0 ms

As you can see, there’s a bit of variance there and it’s always a good idea to capture multiple runs. You might want to run each iteration for a longer time, like 60 seconds each.

9. Performance Monitor

Performance Monitor is a tool built into Windows (client and server) that shows specific performance information for several components of the system. For local storage, you can look into details about the performance of physical disks, logical disks and Hyper-V virtual disks. For remote storage you can inspect networking, SMB file shares and much more. In any case, you want to keep an eye on your processors, as a whole and per core.

Here are a few counters we can inspect, for instance, while running that random 8KB IO workload we just finished investigating:

Performance Monitor defaults to a line graph view, but I personally prefer to use the report view (you can get to it from the line chart view by pressing CTRL-G twice). Here’s an example of what I see for my test system while running “C:\SQLIO\SQLIO.EXE -s10 -kR -frandom -b8 -t2 -o16 -LS -BN X:\TestFile.DAT”.

Note 1: Disk counters here are in bytes, base 10. That means that what SQLIO defines as 8KB shows here as 8,192 and the 282.49 MB/sec shows as 296,207,602 bytes/sec. So, for those concerned with the difference between a megabyte (MB) and a mebibyte (MiB), there’s some more food  for thought and debate.

Note 2: Performance Monitor, by default, updates the information once every second and you will sometimes see numbers that are slightly higher or slightly lower than the SQLIO full run average.

10. SQLIO and SMB file shares

You can use SQLIO to get the same type of performance information for SMB file shares. It is as simple as mapping the file share to a drive letter using the old “NET USE” command or the new PowerShell cmdlet “New-SmbMapping”. You can also use a UNC path directly instead of using drive letters. Here are a couple of examples:

PS C:\> C:\SQLIO\SQLIO.EXE -s10 -kR -fsequential -b512 -t1 -o3 -LS -BN \\FSC5-D\X$\TestFile.DAT
sqlio v1.5.SG
using system counter for latency timings, 2337892 counts per second
1 thread reading for 10 secs from file \\FSC5-D\X$\TestFile.DAT
        using 512KB sequential IOs
        enabling multiple I/Os per thread with 3 outstanding
        buffering set to not use file nor disk caches (as is SQL Server)
using current size: 40960 MB for file: \\FSC5-D\X$\TestFile.DAT
initialization done
CUMULATIVE DATA:
throughput metrics:
IOs/sec:  1376.40
MBs/sec:   688.20
latency metrics:
Min_Latency(ms): 2
Avg_Latency(ms): 2
Max_Latency(ms): 3
histogram:
ms: 0  1  2  3  4  5  6  7  8  9 10 11 12 13 14 15 16 17 18 19 20 21 22 23 24+
%:  0  0 100  0  0  0  0  0  0  0  0  0  0  0  0  0  0  0  0  0  0  0  0  0  0

Notice I bumped up the queue depth a bit to get the same throughput as we were getting on the local disk. We’re at 2 milliseconds of latency here. As you can probably tell, this SMB configuration is using an RDMA network interface.

PS C:\> C:\SQLIO\SQLIO.EXE -s10 -kR -frandom -b8 -t2 -o24 -LS -BN \\FSC5-D\X$\TestFile.DAT
sqlio v1.5.SG
using system counter for latency timings, 2337892 counts per second
2 threads reading for 10 secs from file \\FSC5-D\X$\TestFile.DAT
        using 8KB random IOs
        enabling multiple I/Os per thread with 24 outstanding
        buffering set to not use file nor disk caches (as is SQL Server)
using current size: 40960 MB for file: \\FSC5-D\X$\TestFile.DAT
initialization done
CUMULATIVE DATA:
throughput metrics:
IOs/sec: 34020.69
MBs/sec:   265.78
latency metrics:
Min_Latency(ms): 0
Avg_Latency(ms): 0
Max_Latency(ms): 6
histogram:
ms: 0  1  2  3  4  5  6  7  8  9 10 11 12 13 14 15 16 17 18 19 20 21 22 23 24+
%: 44 33 15  6  2  0  0  0  0  0  0  0  0  0  0  0  0  0  0  0  0  0  0  0  0

Again I increased the queue depth a bit to get the best IOPs in this configuration. This is also close to the local performance and average latency is still under 1 millisecond.

11. Performance Monitor and SMB shares

When using Performance Monitor to look at SMB Shares, you should use the “SMB Client Shares” set of performance counters. Here are the main counters to watch:

Also, here is a view of performance monitor while running the random 8KB workload shown above:

12. Conclusion

I hope you have learned how to use SQLIO to perform some storage testing of your own. I encourage you to use it to look at the performance of the new features in Windows Server 2012, like Storage Spaces and SMB 3.0. Let me know if you were able to try it out and feel free to share some of your experiments via blog comments.

Question via e-mail:

I am using blade servers for my Hyper-V cluster and I can only have two NICs per blade in this configuration.

I am considering two options on how to configure the NICs:

1)      Use one NIC for internal network and one NIC for external network, connected to the virtual switch
2)      Team the two NICs together and use the same path for all kinds of traffic

What would you recommend?

Answer

If you're using clusters, I assume you're concerned with high availability and network fault tolerance. In this case using one NIC for each kind of traffic creates two single points of failure. You should avoid that.

I would recommend that you team the two NICs, connect the team to the virtual switch and add a few virtual NICs to the parent partition for you storage, migration, cluster and management traffic. You can then use QoS policies manage your quality of service.

If you're using SMB for storage, be sure to have multiple vNICs (one for each physical NIC behind the team), so you can properly leverage SMB Multichannel in combination with NIC teaming. By the way, SMB Direct (RDMA) won't work with this scenario.

The first thing you want to do is create a team out of the two NICs and connect the team to a Hyper-V virtual switch. For instance:

New-NetLbfoTeam Team1 –TeamMembers NIC1, NIC2 –TeamNicName TeamNIC1
New-VMSwitch TeamSwitch –NetAdapterName TeamNIC1 –MinimumBandwidthMode Weight –AllowManagementOS $false

Next, you want to create multiple vNICs on the parent partition, one for each kind of traffic (two for SMB). Here's an example:

Add-VMNetworkAdapter –ManagementOS –Name SMB1 –SwitchName TeamSwitch
Add-VMNetworkAdapter –ManagementOS –Name SMB2 –SwitchName TeamSwitch
Add-VMNetworkAdapter –ManagementOS –Name Migration –SwitchName TeamSwitch
Add-VMNetworkAdapter –ManagementOS –Name Cluster –SwitchName TeamSwitch
Add-VMNetworkAdapter –ManagementOS –Name Management –SwitchName TeamSwitch

After this, you want to configure the NICs properly. This will include setting IP addresses, creating separate subnets for each kind of traffic. You can optionally put them each on a different VLAN.

Since you have lots of NICs now and you're already in manual configuration territory anyway, you might want to help the SMB Multichannel by pointing it to the NICs that should be used by SMB. You can do this by configuring SMB Multichannel constraints instead of letting SMB try all different paths. For instance, assuming that your Scale-Out File Server name is SOFS, you could use:

New-SmbMultichannelConstraint -ServerName SOFS -InterfaceAlias SMB1, SMB2

Last but not least you might also want set QoS for each kind of traffic, using the facilities provided by the Hyper-V virtual switch. One way to do it is:

Set-VMNetworkAdapter –ManagementOS –Name SMB1 –MinimumBandwidthWeight 20
Set-VMNetworkAdapter –ManagementOS –Name SMB2 –MinimumBandwidthWeight 20
Set-VMNetworkAdapter –ManagementOS –Name Migration –MinimumBandwidthWeight 20
Set-VMNetworkAdapter –ManagementOS –Name Cluster –MinimumBandwidthWeight 5
Set-VMNetworkAdapter –ManagementOS –Name Management –MinimumBandwidthWeight 5
Set-VMNetworkAdapter –VMName * -MinimumBandwidthWeight 1

There is a great TechNet page with details on this and other network configurations at http://technet.microsoft.com/en-us/library/jj735302.aspx

I delivered an instructor-led lab on Windows Server 2012 Storage at MMS 2013 yesterday (4/8), with a repeat scheduled for tomorrow (4/10) at 2:45 PM. You can find the details on this lab at http://www.2013mms.com/topic/details/WS-IL303

I used a few slides to comment on the contents of the lab, with an overview on the capabilities covered at the lab, including:

• Storage Spaces
• Data Deduplication
• SMB 3.0 and application storage support
• SMB Multichannel
• SMB Transparent Failover
• Cluster-Aware Updating (CAU)

A few people in the lab asked for the slides, so I am attaching it to this blog post in PDF format. If you're at MMS, you can run this also as a self-paced lab. Look under the "Windows Server Labs" and find the lab called "Building a Windows Server 2012 Storage Infrastructure".

Also, if you're here at MMS, don't miss my talk tomorrow (4/10) at 8:30 AM on "File Storage Strategies for Private Cloud" at Ballroom F. The session code is WS-B309 and you can see the details at http://www.2013mms.com/topic/details/WS-B309

Overview

I delivered a demo of Hyper-V over SMB this week at MMS 2013 that’s an evolution of a demo I did back in the Windows Server 2012 launch and also via a TechNet Radio session.

Back then I showed a two physical servers running a SQLIO simulation. One played the role of the File Server and the other worked as a SQL Server.

This time around used using 12 VMs accessing a File Server at the same time. So this is a SQL in a VM running Hyper-V over SMB demo instead of showing SQL Server directly over SMB.

Hardware

The diagram below shows the details of the configuration.

You have an EchoStreams FlacheSAN2 working as File Server, with 2 Intel CPUs at 2.40 Ghz and 64GB of RAM. It includes 6 LSI SAS adapters and 48 Intel SSDs attached directly to the server. This is an impressively packed 2U unit.

The Hyper-V Server is a Dell PowerEdge R720 with 2 Intel CPUs at 2.70 GHz and 128GB of RAM. There are 12 VMs configured in the Hyper-V host, each with 4 virtual processors and 8GB of RAM.

Both the File Server and the Hyper-V host use three 54 Gbps Mellanox ConnectX-3 network interfaces sitting on PCIe Gen3 x8 slots.

Results

The demo showcases two workloads are shown: SQLIO with 512KB IOs and SQLIO with 32KB IOs. For each one, the results are shown for a physical host (single instance of SQLIO running over SMB, but without Hyper-V) and with virtualization (12 Hyper-V VMs running simultaneously over SMB). See the details below.

The first workload (using 512KB IOs) shows very high throughput from the VMs (around 16.8 GBytes/sec combined from all 12 VMs). That’s roughly the equivalent of fifteen 10Gbps Ethernet ports combined or around twenty 8Gbps Fibre Channel ports. And look at that low CPU utilization...

The second workload shows high IOPS (over 300,000 IOPs of 32KB each). That IO size is definitely larger than most high IOPs demos you’ve seen before. This also delivers throughput of over 10 GBytes/sec. It’s important to note that this demo accomplishes this on 2-socket/16-core servers, even though this specific workload is fairly CPU-intensive.

Notes:

• The screenshots above show an instant snapshot of a running workload using Performance Monitor. I also ran each workload for only 20 seconds. Ideally you would run the workload multiple times with a longer duration and average things out.
• Some of the 6 SAS HBAs on the File Server are sitting on a x4 PCIe slot, since not every one of the 9 slots on the server are x8. For this reason some of the HBAs perform better than others.
• Using 4 virtual processors for each of the 12 VMs appears to be less than ideal. I'm planning to experiment with using more virtual processors per VM to potentially improve the performance a bit.

Conclusion

This is yet another example of how SMB Direct and SMB Multichannel can be combined to produce a high performance File Server for Hyper-V Storage.

This specific configuration pushes the limits of this box with 9 PCIe Gen3 slots in use (six for SAS HBAs and three for RDMA NICs).

I am planning to showcase this setup in a presentation planned for the MMS 2013 conference. If you’re planning to attend, I look forward to seeing you there.

P.S.: Some of the steps used for setting up a configuration similar to this one using PowerShell are available at http://blogs.technet.com/b/josebda/archive/2013/01/26/sample-scripts-for-storage-spaces-standalone-hyper-v-over-smb-and-sqlio-testing.aspx

P.P.S.: For further details on how to run SQLIO, you might want to read this post: http://blogs.technet.com/b/josebda/archive/2013/03/28/sqlio-powershell-and-storage-performance-measuring-iops-throughput-and-latency-for-both-local-disks-and-smb-file-shares.aspx

I have been working with Bob Hunt at the TechNet Radio team to provide a series of webcasts with information about SMB 3.0 and the File Server role in Windows Server 2012.

These are fairly informal conversations, but Bob is really good at posing interesting questions, clarifying the different scenarios and teasing out relevant details on SMB 3.0.

By the way, don’t be fooled by the “Radio” in the name. These are available as both video and audio, typically including at least one demo for each episode.

Here is a list of the TechNet Radio episodes Bob and I recorded (including one with Claus Joergensen, another PM in the SMB team), in the order they were published:

1) Windows Server 2012 Hyper-V over SMB (August 31^st)

Summary: Bob Hunt and Jose Barreto join us for today’s show as they discuss Windows Server 2012 Hyper-V support for remote file storage using SMB 3.0. Tune in as they discuss the basic requirements for Hyper-V over SMB, as well as its latest enhancements and why this solution is an easy and affordable file storage alternative.

Link: http://channel9.msdn.com/Shows/TechNet+Radio/TechNet-Radio-Windows-Server-2012-Hyper-V-over-SMB

2) Windows Server 2012 - How to Scale-Out a File Server and use it for Hyper-V (September 5^th)

Summary: Bob Hunt and Jose Barreto are back for today’s episode where they show us how to scale-out a file server in Windows Server 2012 and how to use it with Hyper-V. Tune in as they discuss the advancements made to file servers in terms of scale, storage, virtual processors, support for VMs per host and per cluster as well as demoing a classic vs. scaled out file server.

Link: http://channel9.msdn.com/Shows/TechNet+Radio/TechNet-Radio-Windows-Server-2012-How-to-Scale-Out-a-File-Server-using-Hyper-V

3) Hyper-V over SMB: Step-by-Step Installation using PowerShell (September 16^th)

Summary: Bob Hunt and Jose Barreto continue their Hyper-V over SMB for Windows Server 2012 series, and in today’s episode they discuss how you can configure this installation using PowerShell. Tune in as they take a deep dive into how you can leverage all the features of SMB 3.0 as they go through this extensive step-by-step walkthrough.

Link: http://channel9.msdn.com/Shows/TechNet+Radio/TechNet-Radio-Hyper-V-over-SMB-Step-by-Step-Installation-using-PowerShell

4) SMB Multi-channel Basics for Windows Server 2012 and SMB 3.0 (October 8^th)

Summary: Bob Hunt and Jose Barreto continue their SMB 3.0 for Windows Server 2012 series, and in today’s episode they discuss the recent improvements made around networking capabilities found within SMB Multichannel which can help increase network performance and availability for File Servers.

Link: http://channel9.msdn.com/Shows/TechNet+Radio/TechNet-Radio-SMB-Multi-channel-Basics-for-Windows-Server-2012-and-SMB-30

5) SQL Server over SMB 3.0 Overview (October 23^rd)

Summary: Principal Program Manager from the Windows File Server team, Claus Joergensen joins Bob Hunt and Jose Barreto as they discuss how and why you would want to implement SQL Server 2012 over SMB 3.0. Tune in as they chat about the benefits, how to set it up as well as address any potential concerns you may have such as performance issues.

Link: http://channel9.msdn.com/Shows/TechNet+Radio/TechNet-Radio-SQL-Server-over-SMB-30-Overview

6) SMB 3.0 Encryption Overview (November 26th)

Summary: Bob Hunt and Jose Barreto continue their SMB series for Windows Server 2012 and in today’s episode they chat about SMB Encryption. Tune in as they discuss this security component from what it is and why its important as well as how this is implemented and configured in your environment with a quick demo on how to do this via the GUI and in PowerShell.

Link: http://channel9.msdn.com/Shows/TechNet+Radio/TechNet-Radio-SMB-30-Encryption-Overview

7) SMB 3.0 Deployment Scenarios (December 6th)

Summary: Bob Hunt and Jose Barreto continue their SMB series for Windows Server 2012 and in today’s episode they chat about deployment scenarios and ways in which you can implement all of the new features in SMB 3.0. Tune in as they dive deep into various deployment strategies for SMB.

Link: http://channel9.msdn.com/Shows/TechNet+Radio/TechNet-Radio-SMB-30-Deployment-Scenarios

8) Hyper-V over SMB 3.0 Performance Considerations(January 14th)

Summary: Bob Hunt and Jose Barreto continue their Hyper-V over SMB series and in today’s episode they discuss performance considerations for a sample Enterprise configuration and what you may want to think about during your deployment.

Link: http://channel9.msdn.com/Shows/TechNet+Radio/TechNet-Radio-Hyper-V-over-SMB-30-Performance-Considerations

9) Hyper-V Local vs. Hyper-V SMB Performance(February 7th)

Summary: Bob Hunt and Jose Barreto continue their Hyper-V over SMB series and in today’s episode they discuss a recently released independent study that compares the performance between Hyper-V local and Hyper-V over SMB. Tune in as they chat about the results as they take a deep dive to see how Windows Server 2012 performs in terms of storage, networking performance, efficiency and scalability.

Link: http://channel9.msdn.com/Shows/TechNet+Radio/TechNet-Radio-Hyper-V-Local-vs-Hyper-V-SMB-Performance

10) Windows Server 2012 File Server Tips and Tricks (March 5th)

Summary: Bob Hunt and Jose Barreto continue their Windows Server 2012 File Server and SMB 3.0 series and in today’s episode they lay out their top Tips and Tricks. Tune in as they disclose a number of useful bits of information such as how to use multiple subnets when deploying SMB multichannel in a cluster from how to avoid loopback configurations for Hyper-V over SMB.

Link: http://channel9.msdn.com/Shows/TechNet+Radio/TechNet-Radio-Windows-Server-2012-File-Server-Tips-and-Tricks

 11) Windows Server 2012 File Server and SMB 3.0 – Simpler By Design (April 11h)

Summary: Just how easy is it to use SMB 3.0? That’s the topic for today’s show as Bob Hunt and Jose Barreto discuss the ins and outs of Windows Server 2012 File Storage for Virtualization. Tune in as they chat about SMB Transparent Failover, Scale-Out, Multichannel, SMB Direct and Encryption and much, much more!

Link: http://channel9.msdn.com/Shows/TechNet+Radio/TechNet-Radio-Windows-Server-2012-File-Server-and-SMB-30--Simpler-By-Design

More episodes are coming. Check back soon…

I was visiting the expo area at MMS 2013 earlier today and saw that Iron Networks was showing a private cloud pod there, complete with a set of well-matched layers of compute, storage and networking.

They were demonstrating several of the latest capabilities in Windows Server 2012 and System Center 2012 SP1, including:

• Shared SAS Storage with Windows Server 2012 Storage Spaces, with multiple JBODs and different tiers of storage (SSDs, performance HDDs, capacity HDDs)
• Windows Server 2012 Scale-Out File Server Clusters using SMB 3.0, including SMB Multichannel and SMB Direct.
• Dual 10GbE RDMA networks for storage, dual 10GbE networks for tenants. Plus dual 40GbE aggregate switches and a management switch.
• Cluster-in-a-box solutions implementing several roles, including a Network Virtualization Gateway, System Center 2012 SP1 and File Server Clusters.
• Dense set of compute nodes running Windows Server 2012 Hyper-V hosts with lots of cores and memory

Here is some additional information from their booth signage:

They had a live system in the booth that was actually used in several demos at MMS 2013, including some of the keynote demos.
I took a picture and put some labels on it. Compare the physical systems below with their diagram above.

Loved this hardware! Makes our software solution really come to life...

This post is a part of the nine-part “What’s New in Windows Server & System Center 2012 R2” series that is featured on Brad Anderson’s In the Cloud blog.  Today’s blog post covers Windows Server 2012 R2 Storage and how it applies to the larger topic of “Transform the Datacenter.”  To read that post and see the other technologies discussed, read today’s post: “What’s New in 2012 R2: IaaS Innovations.”

1) Overview

In this document, I am sharing all the steps I used to create a Windows Server 2012 R2 File Server environment, so you can try the new features like SMB Scale-Out Rebalancing and Shared VHDX for Guest Clustering. This configuration uses seven physical computers and a JBOD for shared storage. If you're not familiar with these technologies, I would strong encourage you to review the TechEd 2013 talks on Hyper-V over SMB (Understanding the Hyper-V over SMB Scenario, Configurations, and End-to-End Performance) and Shared VHDX (second half of Application Availability Strategies for the Private Cloud).

The demo setup includes the following:

• 1 DNS / Domain Controller
• 1 JBOD with SAS disks (suggested at least 4 HDDs with 250GB each)
• 3 Scale-Out File Server Cluster nodes
• 2 Hyper-V cluster nodes
• 1 VMM Server

Here’s a diagram of the setup:

Here are the details about the names, roles and IP addresses for each of the computers involved:

Following these steps will probably require a few hours of work end-to-end, but it is a great way to experiment with a large set of Microsoft technologies in or related to Windows Server 2012 R2, including:

• Hyper-V
• Networking
• Domain Name Services (DNS)
• Active Directory Domain Services (AD-DS)
• Shared VHDX
• Storage Spaces
• Failover Clustering
• File Servers
• PowerShell
• SQL Server
• Virtual Machine Manager

2) Required Hardware and Software, plus Disclaimers

You will need the following hardware to perform the steps described here:

• Seven computers capable of running the Windows Server 2012 R2 Preview (three of them capable of running Hyper-V).
• Virtualization settings enabled in the BIOS for the three computers that will be running Hyper-V. For details, check this blog post: http://blogs.technet.com/b/iftekhar/archive/2010/08/09/enable-hardware-settings-in-bios-to-run-hyper-v.aspx and http://technet.microsoft.com/en-us/library/cc731898.aspx
• For this setup, each computer should have at least 4GB of RAM (8GB of RAM is recommended, especially for the computer running SQL and VMM).
• An 8GB USB stick, if you’re installing Windows Server from USB and copying the downloaded software around.

You will need the following software to perform the steps described here:

• Windows Server 2012 R2 Preview from http://technet.microsoft.com/en-US/evalcenter/dn205286.aspx
• System Center VMM 2012 Preview from http://technet.microsoft.com/en-US/evalcenter/dn205295
• SQL Server 2012 SP1 Evaluation from http://www.microsoft.com/sqlserver/en/us/get-sql-server/try-it.aspx
• SQLIO, as described at http://blogs.technet.com/b/JOSE/archive/2013/03/25/sqlio-powershell-and-storage-performance-measuring-iops-throughput-and-latency-for-both-local-disks-and-smb-file-shares.aspx

Notes and disclaimers:

• A certain familiarity with Windows administration and configuration is assumed. If you're new to Windows, this document is not for you. Sorry...
• If you are asked a question or required to perform an action that you do not see described in these steps, go with the default option.
• There are usually several ways to perform a specific configuration or administration task. What I describe here is one of those many ways. It's not necessarily the best way, just the one I personally like best at the moment.
• For the most part, I use PowerShell to configure the systems. You can also use a graphical interface instead, but I did not describe those steps here.
• The JBOD used must be certified for use with Storage Spaces. For a list of certified JBODs, check the Windows Server Catalog at http://www.windowsservercatalog.com/results.aspx?bCatID=1573&cpID=0&avc=10&OR=1
• The Storage Spaces configuration shown here uses only HDDs. You could enhance it to use both SSDs and HDDs, in order to leverage new Storage Spaces features like Tiering or Write-back Cache.
• The configuration uses three file servers to demonstrate the new SMB Scale-Out Balancing capabilities when adding a node to a two-node file server cluster. It uses only two Hyper-V nodes because that’s the minimum required for a fault-tolerant Shared VHDX deployment. Typical setups will use a two-node file server cluster and a larger number of Hyper-V cluster nodes.

3) Summarized instructions for experts

If you are already familiar with Failover Clustering, Scale-Out File Servers and Hyper-V, here’s a short introduction to what it takes to configure a Shared VHDX environment with a Scale-Out File Server. A detailed step-by-step is available in the following sections of this document.

• Configure a three-node Scale-Out File Server Cluster using Windows Server 2012 R2 and Clustered Storage Spaces with Shared SAS disks.
• Configure a Hyper-V Failover cluster using Windows Server 2012 R2 with SMB shared storage as you normally would.
• Create an OS VHD or VHDX file as you would regularly, on an SMB file share. Create your VMs as you would regularly would, using the OS VHD or VHDX file. Both Generation 1 and Generation 2 VMs are supported.
• Create your VHDX data files to be shared as fixed-size or dynamically expanding, on a Clustered Shared Volume. Old VHD files or differencing disks are not supported with Shared VHDX.
• Add the shared VHDX data files to multiple VMs, using the Add-VMHardDiskDrive and the “-ShareVirtualDisk” option. If using the GUI, check the box for “Share virtual disk” when adding the VHDX data files to the VM.
• Inside the VM, install Windows Server 2012 R2 or Windows Server 2012. Make sure to install the Hyper-V integration components.
• Validate and configure the cluster inside the VMs as you normally would. The guest cluster can use classic cluster disks or Cluster Shared Volumes. Any cluster role should work as it would in a physical machine with shared SAS storage.

The rest of this document contains detailed step-by-step instructions for each of the items outlined above.

4) Configure JOSE-D (DNS, DC)

# Preparation steps: Install WS2012R2, rename computer
Rename-Computer -NewName JOSE-D -Restart

# Install required roles and features, restarts at the end
Install-WindowsFeature AD-Domain-Services, RSAT-ADDS, RSAT-ADDS-Tools

# Rename DHCP network adapter to Net1, private to Net2
Get-NetIPAddress -PrefixOrigin DHCP | Get-NetAdapter | Rename-NetAdapter -NewName Net1

# Configure Net2 with a static IP address for DNS / DC
Get-NetAdapter Eth* | ? Status -eq Up | ? InterfaceDescription -notmatch "Mellanox*" | Rename-NetAdapter -NewName Net2
Set-NetIPInterface -InterfaceAlias Net2 -DHCP Disabled
Remove-NetIPAddress -InterfaceAlias Net2 -Confirm:$false
New-NetIPAddress -InterfaceAlias Net2 -IPAddress 192.168.100.1 -PrefixLength 24
Set-DnsClientServerAddress -InterfaceAlias Net2 -ServerAddresses 192.168.100.1

# Configure NetR1 with a static IP address for the RDMA network
Get-NetAdapter -InterfaceDescription "*IPoIB*" | ? Status -eq Up | Rename-NetAdapter -NewName NetR1
Set-NetIPInterface -InterfaceAlias NetR1 -DHCP Disabled
Remove-NetIPAddress -InterfaceAlias NetR1 -Confirm:$false
New-NetIPAddress -InterfaceAlias NetR1 -IPAddress 192.168.101.1 -PrefixLength 24
Set-DnsClientServerAddress -InterfaceAlias NetR1 -ServerAddresses 192.168.100.1

# Disable all disconnected adapters
Get-NetAdapter -InterfaceDescription "*IPoIB*" | ? Status -ne Up | Rename-NetAdapter -NewName NetRX
Get-NetAdapter | ? Status -ne Up | Disable-NetAdapter -Confirm:$false

# Create AD forest, reboots at the end
Install-ADDSForest -CreateDNSDelegation:$false -DatabasePath "C:\Windows\NTDS" -DomainMode "Win2012" -DomainName "JOSE.TEST" -DomainNetBIOSName "JOSE" -ForestMode "Win2012" -InstallDNS:$true -LogPath "C:\Windows\NTDS" -SYSVOLPath "C:\Windows\SYSVOL"

5) Configure JOSE-A1 (File Server Cluster A, Node 1)

# Preparation steps: Install WS2012R2, rename computer
Rename-Computer -NewName JOSE-A1 -Restart

# Install required roles and features, restarts at the end
Install-WindowsFeature File-Services, FS-FileServer, Failover-Clustering
Install-WindowsFeature RSAT-Clustering -IncludeAllSubFeature

# Rename DHCP network adapter to Net1
Get-NetIPAddress -PrefixOrigin DHCP | Get-NetAdapter | Rename-NetAdapter -NewName Net1

# Configure Net2 with a static IP address for DNS / DC
Get-NetAdapter Eth* | ? Status -eq Up | ? InterfaceDescription -notmatch "Mellanox*" | Rename-NetAdapter -NewName Net2
Set-NetIPInterface -InterfaceAlias Net2 -DHCP Disabled
Remove-NetIPAddress -InterfaceAlias Net2 -Confirm:$false
New-NetIPAddress -InterfaceAlias Net2 -IPAddress 192.168.100.11 -PrefixLength 24
Set-DnsClientServerAddress -InterfaceAlias Net2 -ServerAddresses 192.168.100.1

# Configure NetR1 with a static IP address for the RDMA network
Get-NetAdapter -InterfaceDescription "*IPoIB*" | ? Status -eq Up | Rename-NetAdapter -NewName NetR1
Set-NetIPInterface -InterfaceAlias NetR1 -DHCP Disabled
Remove-NetIPAddress -InterfaceAlias NetR1 -Confirm:$false
New-NetIPAddress -InterfaceAlias NetR1 -IPAddress 192.168.101.11 -PrefixLength 24
Set-DnsClientServerAddress -InterfaceAlias NetR1 -ServerAddresses 192.168.100.1

# Disable all disconnected adapters
Get-NetAdapter -InterfaceDescription "*IPoIB*" | ? Status -ne Up | Rename-NetAdapter -NewName NetRX
Get-NetAdapter | ? Status -ne Up | Disable-NetAdapter -Confirm:$false

# Configure Storage Spaces. Assumes JBOD with 4 x 250GB HDDs = 1TB total raw. 2 mirror columns. 80GB spaces.
# Adjust the size of the volumes and the number of mirror columns based on the actual number of HDDs you have.

$s = Get-StorageSubSystem -FriendlyName *Spaces*
New-StoragePool -FriendlyName Pool1 -StorageSubSystemFriendlyName $s.FriendlyName -PhysicalDisks (Get-PhysicalDisk -CanPool $true)
Set-ResiliencySetting -Name Mirror -NumberofColumnsDefault 2 -StoragePool (Get-StoragePool -FriendlyName Pool1)

New-VirtualDisk -FriendlyName Space1 -StoragePoolFriendlyName Pool1 -ResiliencySettingName Mirror –Size 1GB
2..7 | % { New-VirtualDisk -FriendlyName Space$_ -StoragePoolFriendlyName Pool1 -ResiliencySettingName Mirror –Size 80GB }

1..7 | % {
   $Letter ="PQRSTUV”[($_-1)]
$Number = (Get-VirtualDisk -FriendlyName Space$_ | Get-Disk).Number
Set-Disk -Number $Number -IsReadOnly 0
   Set-Disk -Number $Number -IsOffline 0
Initialize-Disk -Number $Number -PartitionStyle MBR
   New-Partition -DiskNumber $Number -DriveLetter $Letter -UseMaximumSize 
   Initialize-Volume -DriveLetter $Letter -FileSystem NTFS -Confirm:$false
}

# Join domain, restart the machine
Add-Computer -DomainName JOSE.TEST -Credential (Get-Credential) -Restart

6) Configure JOSE-A2 (File Server Cluster A, Node 2)

# Preparation steps: Install WS2012R2, rename computer
Rename-Computer -NewName JOSE-A2 -Restart

# Install required roles and features, restarts at the end
Install-WindowsFeature File-Services, FS-FileServer, Failover-Clustering
Install-WindowsFeature RSAT-Clustering -IncludeAllSubFeature

# Rename DHCP network adapter to Net1
Get-NetIPAddress -PrefixOrigin DHCP | Get-NetAdapter | Rename-NetAdapter -NewName Net1

# Configure Net2 with a static IP address for DNS / DC
Get-NetAdapter Eth* | ? Status -eq Up | ? InterfaceDescription -notmatch "Mellanox*" | Rename-NetAdapter -NewName Net2
Set-NetIPInterface -InterfaceAlias Net2 -DHCP Disabled
Remove-NetIPAddress -InterfaceAlias Net2 -Confirm:$false
New-NetIPAddress -InterfaceAlias Net2 -IPAddress 192.168.100.12 -PrefixLength 24
Set-DnsClientServerAddress -InterfaceAlias Net2 -ServerAddresses 192.168.100.1

# Configure NetR1 with a static IP address for the RDMA network
Get-NetAdapter -InterfaceDescription "*IPoIB*" | ? Status -eq Up | Rename-NetAdapter -NewName NetR1
Set-NetIPInterface -InterfaceAlias NetR1 -DHCP Disabled
Remove-NetIPAddress -InterfaceAlias NetR1 -Confirm:$false
New-NetIPAddress -InterfaceAlias NetR1 -IPAddress 192.168.101.12 -PrefixLength 24
Set-DnsClientServerAddress -InterfaceAlias NetR1 -ServerAddresses 192.168.100.1

# Disable all disconnected adapters
Get-NetAdapter -InterfaceDescription "*IPoIB*" | ? Status -ne Up | Rename-NetAdapter -NewName NetRX
Get-NetAdapter | ? Status -ne Up | Disable-NetAdapter -Confirm:$false

# Join domain, restart the machine
Add-Computer -DomainName JOSE.TEST -Credential (Get-Credential) –Restart

7) Configure JOSE-A3 (File Server Cluster A, Node 3)

# Preparation steps: Install WS2012R2, rename computer
Rename-Computer -NewName JOSE-A3 -Restart

# Install required roles and features, restarts at the end
Install-WindowsFeature File-Services, FS-FileServer, Failover-Clustering
Install-WindowsFeature RSAT-Clustering -IncludeAllSubFeature

# Rename DHCP network adapter to Net1
Get-NetIPAddress -PrefixOrigin DHCP | Get-NetAdapter | Rename-NetAdapter -NewName Net1

# Configure Net2 with a static IP address for DNS / DC
Get-NetAdapter Eth* | ? Status -eq Up | ? InterfaceDescription -notmatch "Mellanox*" | Rename-NetAdapter -NewName Net2
Set-NetIPInterface -InterfaceAlias Net2 -DHCP Disabled
Remove-NetIPAddress -InterfaceAlias Net2 -Confirm:$false
New-NetIPAddress -InterfaceAlias Net2 -IPAddress 192.168.100.13 -PrefixLength 24
Set-DnsClientServerAddress -InterfaceAlias Net2 -ServerAddresses 192.168.100.1

# Configure NetR1 with a static IP address for the RDMA network
Get-NetAdapter -InterfaceDescription "*IPoIB*" | ? Status -eq Up | Rename-NetAdapter -NewName NetR1
Set-NetIPInterface -InterfaceAlias NetR1 -DHCP Disabled
Remove-NetIPAddress -InterfaceAlias NetR1 -Confirm:$false
New-NetIPAddress -InterfaceAlias NetR1 -IPAddress 192.168.101.13 -PrefixLength 24
Set-DnsClientServerAddress -InterfaceAlias NetR1 -ServerAddresses 192.168.100.1

# Disable all disconnected adapters
Get-NetAdapter -InterfaceDescription "*IPoIB*" | ? Status -ne Up | Rename-NetAdapter -NewName NetRX
Get-NetAdapter | ? Status -ne Up | Disable-NetAdapter -Confirm:$false

# Join domain, restart the machine
Add-Computer -DomainName JOSE.TEST -Credential (Get-Credential) -Restart

8) Configure File Server Cluster JOSE-A

# Validate cluster
Test-Cluster -Node JOSE-A1, JOSE-A2, JOSE-A3

# Create cluster
New-Cluster –Name JOSE-A -Node JOSE-A1, JOSE-A2, JOSE-A3

# Rename and configure networks
(Get-ClusterNetwork | ? {$_.Address -notlike "192.*" }).Name = "Net1"
(Get-ClusterNetwork | ? {$_.Address -like "192.168.100.*" }).Name = "Net2"
(Get-ClusterNetwork | ? {$_.Address -like "192.168.101.*" }).Name = "NetR1"
(Get-ClusterNetwork Net1).Role = 1
(Get-ClusterNetwork Net2).Role = 3
(Get-ClusterNetwork NetR1).Role = 3
(Get-Cluster).UseClientAccessNetworksForSharedVolumes=1

# Remove default DHCP-based IP address for JOSE-A and add 2 IP addresses on 100/101 networks
Stop-ClusterResource "Cluster Name"
Get-ClusterResource | ? { (($_.ResourceType -like "*Address*") -and ($_.OwnerGroup -eq "Cluster Group")) } | Remove-ClusterResource –Force

Add-ClusterResource -Name "Cluster IP Address 100" -Group "Cluster Group" -ResourceType "IP Address"
Get-ClusterResource –Name "Cluster IP Address 100" | Set-ClusterParameter -Multiple @{ “Network”=”Net2”; "Address"="192.168.100.19"; ”SubnetMask”=”255.255.255.0”; "EnableDhcp"=0 }
Get-ClusterResource “Cluster Name” | Add-ClusterResourceDependency –Resource "Cluster IP Address 100"

Add-ClusterResource -Name "Cluster IP Address 101" -Group "Cluster Group" -ResourceType "IP Address"
Get-ClusterResource –Name "Cluster IP Address 101" | Set-ClusterParameter -Multiple @{ “Network”=”NetR1”; "Address"="192.168.101.19"; ”SubnetMask”=”255.255.255.0”; "EnableDhcp"=0 }
Get-ClusterResource “Cluster Name” | Add-ClusterResourceDependency –Resource "Cluster IP Address 101"

Set-ClusterResourceDependency -Resource "Cluster Name" -Dependency "[Cluster IP Address 100] OR [Cluster IP Address 101] "
Start-ClusterResource "Cluster Name"

# Rename Witness Disk
$w = Get-ClusterResource | ? { $_.OwnerGroup -eq "Cluster Group" -and $_.ResourceType -eq "Physical Disk"}
$w.Name = "WitnessDisk"

# Add remaining disks to Cluster Shared Volumes
Get-ClusterResource | ? OwnerGroup -eq "Available Storage" | Add-ClusterSharedVolume

# Create Scale-Out File Server
Add-ClusterScaleOutFileServerRole -Name JOSE-F

9) Configure JOSE-B1 (Hyper-V Host Cluster B, Node 1)

# Preparation steps: Install WS2012R2, rename computer
Rename-Computer -NewName JOSE-B1 -Restart

# Install required roles and features, restarts at the end
Install-WindowsFeature Failover-Clustering
Install-WindowsFeature RSAT-Clustering -IncludeAllSubFeature
Install-WindowsFeature Hyper-V, Hyper-V-PowerShell, Hyper-V-Tools -Restart

# Rename DHCP network adapter to Net1
Get-NetIPAddress -PrefixOrigin DHCP | Get-NetAdapter | Rename-NetAdapter -NewName Net1
New-VMSwitch -NetAdapterName Net1 -Name Net1
Rename-NetAdapter -Name "vEthernet (Net1)" -NewName VNet1

# Configure Net2 with a static IP address for DNS / DC
Get-NetAdapter Eth* | ? Status -eq Up | ? InterfaceDescription -notmatch "Mellanox*" | Rename-NetAdapter -NewName Net2
New-VMSwitch -NetAdapterName Net2 -Name Net2
Rename-NetAdapter -Name "vEthernet (Net2)" -NewName VNet2
Set-NetIPInterface -InterfaceAlias VNet2 -DHCP Disabled
Remove-NetIPAddress -InterfaceAlias VNet2 -Confirm:$false
New-NetIPAddress -InterfaceAlias VNet2 -IPAddress 192.168.100.21 -PrefixLength 24
Set-DnsClientServerAddress -InterfaceAlias VNet2 -ServerAddresses 192.168.100.1

# Configure NetR1 with a static IP address for the RDMA network
Get-NetAdapter -InterfaceDescription "*IPoIB*" | ? Status -eq Up | Rename-NetAdapter -NewName NetR1
Set-NetIPInterface -InterfaceAlias NetR1 -DHCP Disabled
Remove-NetIPAddress -InterfaceAlias NetR1 -Confirm:$false
New-NetIPAddress -InterfaceAlias NetR1 -IPAddress 192.168.101.21 -PrefixLength 24
Set-DnsClientServerAddress -InterfaceAlias NetR1 -ServerAddresses 192.168.100.1

# Disable all disconnected adapters
Get-NetAdapter -InterfaceDescription "*IPoIB*" | ? Status -ne Up | Rename-NetAdapter -NewName NetRX
Get-NetAdapter | ? Status -ne Up | Disable-NetAdapter -Confirm:$false

# Join domain, restart the machine
Add-Computer -DomainName JOSE.TEST -Credential (Get-Credential) -Restart

10) Configure JOSE-B2 (Hyper-V Host Cluster B, Node 2)

# Preparation steps: Install WS2012R2, rename computer
Rename-Computer -NewName JOSE-B2 -Restart

# Install required roles and features, restarts at the end
Install-WindowsFeature Failover-Clustering
Install-WindowsFeature RSAT-Clustering -IncludeAllSubFeature
Install-WindowsFeature Hyper-V, Hyper-V-PowerShell, Hyper-V-Tools -Restart

# Rename DHCP network adapter to Net1
Get-NetIPAddress -PrefixOrigin DHCP | Get-NetAdapter | Rename-NetAdapter -NewName Net1
New-VMSwitch -NetAdapterName Net1 -Name Net1
Rename-NetAdapter -Name "vEthernet (Net1)" -NewName VNet1

# Configure Net2 with a static IP address for DNS / DC
Get-NetAdapter Eth* | ? Status -eq Up | ? InterfaceDescription -notmatch "Mellanox*" | Rename-NetAdapter -NewName Net2
New-VMSwitch -NetAdapterName Net2 -Name Net2
Rename-NetAdapter -Name "vEthernet (Net2)" -NewName VNet2
Set-NetIPInterface -InterfaceAlias VNet2 -DHCP Disabled
Remove-NetIPAddress -InterfaceAlias VNet2 -Confirm:$false
New-NetIPAddress -InterfaceAlias VNet2 -IPAddress 192.168.100.22 -PrefixLength 24
Set-DnsClientServerAddress -InterfaceAlias VNet2 -ServerAddresses 192.168.100.1

# Configure NetR1 with a static IP address for the RDMA network
Get-NetAdapter -InterfaceDescription "*IPoIB*" | ? Status -eq Up | Rename-NetAdapter -NewName NetR1
Set-NetIPInterface -InterfaceAlias NetR1 -DHCP Disabled
Remove-NetIPAddress -InterfaceAlias NetR1 -Confirm:$false
New-NetIPAddress -InterfaceAlias NetR1 -IPAddress 192.168.101.22 -PrefixLength 24
Set-DnsClientServerAddress -InterfaceAlias NetR1 -ServerAddresses 192.168.100.1

# Disable all disconnected adapters
Get-NetAdapter -InterfaceDescription "*IPoIB*" | ? Status -ne Up | Rename-NetAdapter -NewName NetRX
Get-NetAdapter | ? Status -ne Up | Disable-NetAdapter -Confirm:$false

# Join domain, restart the machine
Add-Computer -DomainName JOSE.TEST -Credential (Get-Credential) -Restart

11) Configure Hyper-V Host Cluster JOSE-B

# Validate cluster
Test-Cluster -Node JOSE-B1, JOSE-B2

# Create cluster
New-Cluster –Name JOSE-B -Node JOSE-B1, JOSE-B2
# Rename and configure networks
(Get-ClusterNetwork | ? {$_.Address -notlike "192.*" }).Name = "Net1"
(Get-ClusterNetwork | ? {$_.Address -like "192.168.100.*" }).Name = "Net2"
(Get-ClusterNetwork | ? {$_.Address -like "192.168.101.*" }).Name = "NetR1"
(Get-ClusterNetwork Net1).Role = 1
(Get-ClusterNetwork Net2).Role = 3
(Get-ClusterNetwork NetR1).Role = 3
(Get-Cluster).UseClientAccessNetworksForSharedVolumes=1

# Remove default DHCP-based IP address for JOSE-A and add 2 IP addresses on 100/101 networks
Stop-ClusterResource "Cluster Name"
Get-ClusterResource | ? { (($_.ResourceType -like "*Address*") -and ($_.OwnerGroup -eq "Cluster Group")) } | Remove-ClusterResource –Force

Add-ClusterResource -Name "Cluster IP Address 100" -Group "Cluster Group" -ResourceType "IP Address"
Get-ClusterResource –Name "Cluster IP Address 100" | Set-ClusterParameter -Multiple @{ “Network”=”Net2”; "Address"="192.168.100.29"; ”SubnetMask”=”255.255.255.0”; "EnableDhcp"=0 }
Get-ClusterResource “Cluster Name” | Add-ClusterResourceDependency –Resource "Cluster IP Address 100"

Add-ClusterResource -Name "Cluster IP Address 101" -Group "Cluster Group" -ResourceType "IP Address"
Get-ClusterResource –Name "Cluster IP Address 101" | Set-ClusterParameter -Multiple @{ “Network”=”NetR1”; "Address"="192.168.101.29"; ”SubnetMask”=”255.255.255.0”; "EnableDhcp"=0 }
Get-ClusterResource “Cluster Name” | Add-ClusterResourceDependency –Resource "Cluster IP Address 101"

Set-ClusterResourceDependency -Resource "Cluster Name" -Dependency "[Cluster IP Address 100] OR [Cluster IP Address 101] "
Start-ClusterResource "Cluster Name"

# Create Share for VMs (run from JOSE-A1)
1..6 | % {
MD C:\ClusterStorage\Volume$_\Share
New-SmbShare -Name SHARE$_ -Path C:\ClusterStorage\Volume$_\Share -FullAccess JOSE.Test\Administrator, JOSE.Test\JOSE-B1$, JOSE.Test\JOSE-B2$, JOSE.Test\JOSE-B$
Set-SmbPathAcl -ShareName SHARE$_
}

# Create Share for File Share Witness (run from JOSE-A1)
MD C:\ClusterStorage\Volume6\Witness
New-SmbShare -Name Witness -Path C:\ClusterStorage\Volume6\Witness -FullAccess JOSE.Test\Administrator, JOSE.Test\JOSE-B$
Set-SmbPathAcl -ShareName Witness

# Configure JOSE-B Cluster with a File Share Witness (run from JOSE-B1)
Set-ClusterQuorum -NodeAndFileShareMajority \\JOSE-F.JOSE.TEST\Witness

12) Configure VMs on the Hyper-V Host Cluster JOSE-B

# Create VHD files for VMs (run on JOSE-B1)
New-VHD -Path \\JOSE-F.JOSE.TEST\Share1\VM1OS.VHDX -Fixed -SizeBytes 40GB
New-VHD -Path \\JOSE-F.JOSE.TEST\Share2\VM2OS.VHDX -Fixed -SizeBytes 40GB
New-VHD -Path \\JOSE-F.JOSE.TEST\Share3\VM12Witness.VHDX -Fixed -SizeBytes 1GB
New-VHD -Path \\JOSE-F.JOSE.TEST\Share3\VM12Data.VHDX -Fixed -SizeBytes 10GB

# Create VM1 (run on JOSE-B1)
New-VM -Path \\JOSE-F.JOSE.TEST\Share1 -Name VM1 -VHDPath \\JOSE-F.JOSE.TEST\Share1\VM1OS.VHDX -SwitchName Net1 -Memory 2GB
Get-VMProcessor * | Set-VMProcessor -CompatibilityForMigrationEnabled 1
Add-VMNetworkAdapter -VMName VM1 -SwitchName Net2
Add-VMHardDiskDrive -VMName VM1 -Path \\JOSE-F.JOSE.TEST\Share3\VM12Witness.VHDX-ShareVirtualDisk
Add-VMHardDiskDrive -VMName VM1 -Path \\JOSE-F.JOSE.TEST\Share3\VM12Data.VHDX-ShareVirtualDisk
Set-VMDvdDrive -VMName VM1 -Path D:\WindowsServer2012.ISO
Start-VM VM1
Add-VMToCluster VM1

# Create VM2 (run on JOSE-B2)
New-VM -Path \\JOSE-F.JOSE.TEST\Share2 -Name VM2 -VHDPath \\JOSE-F.JOSE.TEST\Share2\VM2OS.VHDX -SwitchName Net1 -Memory 2GB
Get-VMProcessor * | Set-VMProcessor -CompatibilityForMigrationEnabled 1
Add-VMNetworkAdapter -VMName VM2 -SwitchName Net2
Add-VMHardDiskDrive -VMName VM2 -Path \\JOSE-F.JOSE.TEST\Share3\VM12Witness.VHDX-ShareVirtualDisk
Add-VMHardDiskDrive -VMName VM2 -Path \\JOSE-F.JOSE.TEST\Share3\VM12Data.VHDX-ShareVirtualDisk
Set-VMDvdDrive -VMName VM2 -Path D:\WindowsServer2012.ISO
Start-VM VM2
Add-VMToCluster VM2

13) Configure JOSE-X1 (SQL Server Guest Cluster X, Node 1)

# Preparation steps: Install WS2012R2, rename computer, install Hyper-V IC
Rename-Computer -NewName JOSE-X1 -Restart

# Install required roles and features, restarts at the end
Install-WindowsFeature File-Services, FS-FileServer, Failover-Clustering
Install-WindowsFeature RSAT-Clustering -IncludeAllSubFeature

# Rename the two virtual ports as Net1 and Net2
Get-NetIPAddress -PrefixOrigin DHCP | Get-NetAdapter | Rename-NetAdapter -NewName Net1
Get-NetAdapter Ethernet* | Rename-NetAdapter -NewName Net2

# Configure Net2 with a static IP address for DNS / DC
Set-NetIPInterface -InterfaceAlias Net2 -DHCP Disabled
Remove-NetIPAddress -InterfaceAlias Net2 -Confirm:$false
New-NetIPAddress -InterfaceAlias Net2 -IPAddress 192.168.100.41 -PrefixLength 24
Set-DnsClientServerAddress -InterfaceAlias Net2 -ServerAddresses 192.168.100.1

# Configure 2 shared disks
1..2 | % { 
    $Letter ="JK"[($_-1)]
    Set-Disk -Number $_ -IsReadOnly 0
    Set-Disk -Number $_ -IsOffline 0
    Initialize-Disk -Number $_ -PartitionStyle MBR
    New-Partition -DiskNumber $_ -DriveLetter $Letter -UseMaximumSize 
    Initialize-Volume -DriveLetter $Letter -FileSystem NTFS -Confirm:$false
}

# Join domain, restart the machine
Add-Computer -DomainName JOSE.TEST -Credential (Get-Credential) –Restart

14) Configure JOSE-X2 (SQL Server Guest Cluster X, Node 2)

# Preparation steps: Install WS2012R2, rename computer, install Hyper-V IC
Rename-Computer -NewName JOSE-X2 -Restart

# Install required roles and features, restarts at the end
Install-WindowsFeature File-Services, FS-FileServer, Failover-Clustering
Install-WindowsFeature RSAT-Clustering -IncludeAllSubFeature

# Rename the two virtual ports as Net1 and Net2
Get-NetIPAddress -PrefixOrigin DHCP | Get-NetAdapter | Rename-NetAdapter -NewName Net1
Get-NetAdapter Ethernet* | Rename-NetAdapter -NewName Net2

# Configure Net2 with a static IP address for DNS / DC
Set-NetIPInterface -InterfaceAlias Net2 -DHCP Disabled
Remove-NetIPAddress -InterfaceAlias Net2 -Confirm:$false
New-NetIPAddress -InterfaceAlias Net2 -IPAddress 192.168.100.42 -PrefixLength 24
Set-DnsClientServerAddress -InterfaceAlias Net2 -ServerAddresses 192.168.100.1

# Shared Disks already configured on the first node

# Join domain, restart the machine
Add-Computer -DomainName JOSE.TEST -Credential (Get-Credential) –Restart

15) Configure SQL Server Guest Cluster JOSE-X

# Validate cluster
Test-Cluster -Node JOSE-X1, JOSE-X2

# Create cluster
New-Cluster –Name JOSE-X -Node JOSE-X1, JOSE-X2
# Rename and configure networks
(Get-ClusterNetwork | ? {$_.Address -notlike "192.*" }).Name = "Net1"
(Get-ClusterNetwork | ? {$_.Address -like "192.168.100.*" }).Name = "Net2"
(Get-ClusterNetwork Net1).Role = 1
(Get-ClusterNetwork Net2).Role = 3

# Remove default DHCP-based IP address for JOSE-V and add IP address on 100 network
Stop-ClusterResource "Cluster Name"
Get-ClusterResource | ? { (($_.ResourceType -like "*Address*") -and ($_.OwnerGroup -eq "Cluster Group")) } | Remove-ClusterResource –Force

Add-ClusterResource -Name "Cluster IP Address" -Group "Cluster Group" -ResourceType "IP Address"
Get-ClusterResource –Name "Cluster IP Address" | Set-ClusterParameter -Multiple @{ “Network”=”Net2”; "Address"="192.168.100.49"; ”SubnetMask”=”255.255.255.0”; "EnableDhcp"=0 }
Get-ClusterResource “Cluster Name” | Add-ClusterResourceDependency –Resource "Cluster IP Address"

Start-ClusterResource "Cluster Name"

# Rename Witness Disk
$w = Get-ClusterResource | ? { $_.OwnerGroup -eq "Cluster Group" -and $_.ResourceType -eq "Physical Disk"}
$w.Name = "WitnessDisk"

# Install SQL Server on nodes X1 and X2
# Use IP address 192.168.100.48 for the SQL Server group

16) Configure JOSE-V (VMM Server)

# Preparation steps: Install WS2012R2, rename computer
Rename-Computer -NewName JOSE-V -Restart

# Install required roles and features, restarts at the end
Install-WindowsFeature File-Services, FS-FileServer, Failover-Clustering
Install-WindowsFeature RSAT-Clustering -IncludeAllSubFeature

# Rename DHCP network adapter to Net1
Get-NetIPAddress -PrefixOrigin DHCP | Get-NetAdapter | Rename-NetAdapter -NewName Net1

# Configure Net2 with a static IP address for DNS / DC
Get-NetAdapter Eth* | ? Status -eq Up | ? InterfaceDescription -notmatch "Mellanox*" | Rename-NetAdapter -NewName Net2
Set-NetIPInterface -InterfaceAlias Net2 -DHCP Disabled
Remove-NetIPAddress -InterfaceAlias Net2 -Confirm:$false
New-NetIPAddress -InterfaceAlias Net2 -IPAddress 192.168.100.31 -PrefixLength 24
Set-DnsClientServerAddress -InterfaceAlias Net2 -ServerAddresses 192.168.100.1

# Configure NetR1 with a static IP address for the RDMA network
Get-NetAdapter -InterfaceDescription "*IPoIB*" | ? Status -eq Up | Rename-NetAdapter -NewName NetR1
Set-NetIPInterface -InterfaceAlias NetR1 -DHCP Disabled
Remove-NetIPAddress -InterfaceAlias NetR1 -Confirm:$false
New-NetIPAddress -InterfaceAlias NetR1 -IPAddress 192.168.101.31 -PrefixLength 24
Set-DnsClientServerAddress -InterfaceAlias NetR1 -ServerAddresses 192.168.100.1

# Disable all disconnected adapters
Get-NetAdapter -InterfaceDescription "*IPoIB*" | ? Status -ne Up | Rename-NetAdapter -NewName NetRX
Get-NetAdapter | ? Status -ne Up | Disable-NetAdapter -Confirm:$false

# Join domain, restart the machine
Add-Computer -DomainName JOSE.TEST -Credential (Get-Credential) –Restart

# Install .NET Framework
Install-WindowsFeature NET-Framework-Core

# Install Windows Server 2012 R2 ADK
D:\adk\adksetup.exe
# Select only the “Deployment Tools” and “Windows PE” options

# Install SQL Server 2012 SP1
# New SQL Server standalone installation, Feature Selection: Database Engine Services

# Install VMM 2012 R2
# Features selected to be added: VMM management server, VMM console
# Database: VirtualManagerDB database will be created on JOSE-V
# Service Account: Local System account

17) Running the Scale-Out demo (from JOSE-V)

# Prepare - Map the SMB shares
1..6 | % { 
    $d ="PQRSTU"[($_-1)] + “:”
New-SmbMapping -LocalPath $d -RemotePath \\JOSE-f\Share$_ -Persistent $true
}

# Prepare - Create the test files
1..6 | % { 
   $f ="PQRSTU"[($_-1)] + “:\testfile.dat”
fsutil file createnew $f (5GB)
   fsutil file setvaliddata $f (5GB)
}

# Remove the third cluster node (demo starts with 2 nodes)
Remove-ClusterNode -Cluster JOSE-A -Name JOSE-A3

Start Performance Monitor

Start a performance monitor session

Switch to “Histogram Bar” view to show the performance side-by-side

Add a counter for SMB Server Shares, Data bytes/sec, _total instance for JOSE-A1, JOSE-A2 and JOSE-A3.

Query the cluster shared volume ownership on Cluster A, with 2 nodes

Get-ClusterSharedVolume -Cluster JOSE-A | Sort OwnerNode | FT OwnerNode, Name, State -AutoSize

OwnerNode Name           State
--------- ----           -----
JOSE-A1   Cluster Disk 6 Online
JOSE-A1   Cluster Disk 3 Online
JOSE-A1   Cluster Disk 4 Online
JOSE-A2   Cluster Disk 7 Online
JOSE-A2   Cluster Disk 2 Online
JOSE-A2   Cluster Disk 5 Online

Run SQLIO to issue 8KB IOs

C:\sqlio\sqlio.exe -s9999 -T100 -t1 -o16 -b8 -BN -LS -c2000 -frandom -dPQRSTU testfile.dat

Add a 3^rd node and wait for it to take 1/3 of the load

Add-ClusterNode -Cluster JOSE-A -Name JOSE-A3

# Wait 2 and a half minutes to transition to the following state.

Re- query the cluster shared volume ownership on Cluster A, now with 3 nodes

Get-ClusterSharedVolume -Cluster JOSE-A | Sort OwnerNode | FT OwnerNode, Name, State -AutoSize

OwnerNode Name           State
--------- ----           -----
JOSE-A1   Cluster Disk 3 Online
JOSE-A1   Cluster Disk 6 Online
JOSE-A2   Cluster Disk 5 Online
JOSE-A2   Cluster Disk 7 Online
JOSE-A3   Cluster Disk 2 Online
JOSE-A3   Cluster Disk 4 Online

18) Configuring VMM on JOSE-V

Bring the File Server Cluster under VMM management

Select Fabric and use the option to add Storage Devices

Add a Windows-based file server

Specify the full path to the file server cluster:

Verify the File Server Cluster was properly discovered by VMM

Check the provider

Check the Storage Spaces discovery

Check the Scale-Out File Server and file share discovery

Remove the File Server Cluster node (demo starts with 2 nodes)

Under the properties of the File Server Cluster, remove node 3

Check progress under running Jobs

While running a workload, add a File Server Cluster node

Under the properties of the File Server Cluster, add node 3 specifying the full path of to the server

Check progress under running Jobs

19) Verifying systems’ configuration

# Commands to verify the configuration of all systems

"B1", "B2" | % { Get-VM -ComputerName JOSE-$_ }

"B1", "B2" | % { $_; Get-VM -ComputerName JOSE-$_ | Get-VMHardDiskDrive | FT VMName, ControllerType, ControllerLocation, Path -AutoSize}

Get-SmbOpenFile -CimSession JOSE-A1 | Sort ClientUserName, ShareRelativePath | FT ClientUserName, ShareRelativePath –AutoSize

"X1", "X2" | % { $_; Get-Disk -CimSession JOSE-$_ } | FT -AutoSize

"A", "B", "X" | % { $_; Get-ClusterNode -Cluster JOSE-$_ | FT Cluster, NodeName, State, Id -AutoSize }

"A", "B", "X" | % { $_; Get-ClusterResource -Cluster JOSE-$_ | FT -AutoSize}

20) Issues and FAQs (Frequently asked questions)

Failover Cluster creation in the guest fails.

• Make sure you’re logged on as a domain user, not a local user
• Make sure the Windows Server 2012 R2 integration components are installed in the guest.
• Check for individual warnings and errors in the Failover Clustering validation report.

I can’t add a shared VHDX to a VM. I get a message saying that “the storage where the virtual hard disk is located does not support virtual hard disk sharing.”

• Make sure your using a CSV disk or an SMB Scale-Out file share to store your VHDX files
• Make sure the SMB Scale-Out file server cluster is running Windows Server 2012 R2

21) Final Notes

• Keep in mind that there are dependencies between the services running on each computer.
• To shut them down, start with VMM server, then the Hyper-V servers, then the File Server and finally the DNS/DC, waiting for each one to go down completely before moving to the next one.
• To bring the servers up again, go from the DNS/DC to the File Server Cluster, then the Hyper-V Cluster and finally the VMM Server, waiting for the previous one to be fully up before starting the next one.
• I hope you enjoyed these step-by-step instructions. I strongly encourage you to try them out and perform the entire installation yourself. It’s a good learning experience.
• Let me know how these steps worked for you using the comment section. If you run into any issues or found anything particularly interesting, don’t forget to mention the number of the step.

To see all of the posts in this series, check out the What’s New in Windows Server & System Center 2012 R2 archive.

This post is a part of the nine-part “What’s New in Windows Server & System Center 2012 R2” series that is featured on Brad Anderson’s In the Cloud blog.  Today’s blog post covers Windows Server 2012 R2 Storage and how it applies to the larger topic of “Transform the Datacenter.”  To read that post and see the other technologies discussed, read today’s post: “What’s New in 2012 R2: IaaS Innovations.”

1) Overview

In this document, I am sharing all the steps I used to create a Windows Server 2012 R2 File Server demo or test environment, so you can experiment with some of the new technologies yourself. You only need a single computer (the specs are provided below) and the ISO file with the Windows Server 2012 R2 Preview available as a free download.

The demo setup includes 5 virtual machines: 1 domain controller, 3 file servers and 1 file client/VMM Server. It uses the new Shared VHDX feature to provide shared storage for the guest failover cluster and it showcases both Storage Spaces and Scale-Out File Servers.

If you're not familiar with these technologies, I would strong encourage you to review the TechEd 2013 talks on Hyper-V over SMB (Understanding the Hyper-V over SMB Scenario, Configurations, and End-to-End Performance) and Shared VHDX (second half of Application Availability Strategies for the Private Cloud).

Here’s a diagram of the setup:

Here are the details about the names, roles and IP addresses for each of the computers involved:

Following these steps will probably require a few hours of work end-to-end, but it is a great way to experiment with a large set of Microsoft technologies in or related to Windows Server 2012 R2, including:

• Hyper-V
• Networking
• Domain Name Services (DNS)
• Active Directory Domain Services (AD-DS)
• Shared VHDX
• Storage Spaces
• Failover Clustering
• File Servers
• PowerShell
• SQL Server
• Virtual Machine Manager

2) Required Hardware and Software, plus Disclaimers

You will need the following hardware to perform the steps described here:

• One computer capable of running the Windows Server 2012 R2 Preview and Hyper-V.
• As required by Hyper-V, enable the virtualization settings in the BIOS. For details, check this blog post: http://blogs.technet.com/b/iftekhar/archive/2010/08/09/enable-hardware-settings-in-bios-to-run-hyper-v.aspx and http://technet.microsoft.com/en-us/library/cc731898.aspx
• For this setup, the computer should have at least 16GB of RAM. You can use a system with 8GB of total RAM (use half the memory specified for each VM), but it will run slower – specially the VM running SQL and VMM.
• An 8GB USB stick, if you’re installing Windows Server from USB and copying the downloaded software around

You will need the following software to perform the steps described here:

• Windows Server 2012 R2 Preview from http://technet.microsoft.com/en-US/evalcenter/dn205286.aspx
• System Center VMM 2012 Preview from http://technet.microsoft.com/en-US/evalcenter/dn205295
• SQL Server 2012 SP1 Evaluation from http://www.microsoft.com/sqlserver/en/us/get-sql-server/try-it.aspx
• SQLIO, as described at http://blogs.technet.com/b/josebda/archive/2013/03/25/sqlio-powershell-and-storage-performance-measuring-iops-throughput-and-latency-for-both-local-disks-and-smb-file-shares.aspx

Notes and disclaimers:

• A certain familiarity with Windows administration and configuration is assumed. If you're new to Windows, this document is not for you. Sorry...
• If you are asked a question or required to perform an action that you do not see described in these steps, go with the default option.
• There are usually several ways to perform a specific configuration or administration task. What I describe here is one of those many ways. It's not necessarily the best way, just the one I personally like best at the moment.
• For the most part, I use PowerShell to configure the systems. You can also use a graphical interface instead, but I did not describe those steps here.
• Obviously, a single-computer solution can never be tolerant to the failure of that computer. So, the configuration described here is not really continuously available. It’s just a simulation.
• The specific Shared VHDX configuration shown in this blog post is not supported. Microsoft Support will only answer questions and assist in troubleshooting configurations where the Shared VHDX files are stored either on a Cluster Shared Volume (CSV) directly or on a file share on a Scale-Out File Server Cluster.
• The specific Storage Spaces configuration shown in this blog post is not supported. Microsoft Support will only answer questions and assist in troubleshooting configurations where Storage Spaces uses a physical machine (not a VM) and uses one of the certified JBOD hardware solutions (see http://www.windowsservercatalog.com/results.aspx?bCatID=1573&cpID=0&avc=10&OR=1)
• Because of the two items above, the configuration described here should only be used for demos, testing or learning environments.
• The Storage Spaces configuration shown here is not capable of showcasing some of the new features like Tiering or Write-back Cache, since it uses virtual disks without a proper media type. To use those features, you’ll need a storage pool with physical SSDs and HDDs.

3) Summarized instructions for experts

If you are already familiar with Failover Clustering, Scale-Out File Servers and Hyper-V, here’s a short introduction to what it takes to configure this environment on a single computer. A detailed step-by-step is available in the following sections of this document.

• Configure a Hyper-V capable computer with at least 8GB of RAM with Windows Server 2012 R2. Make sure to load the Failover Clustering feature. No need to actually create a failover cluster on the physical machine.
• Manually attach the Shared VHDX filter to the disk where the VHDX files will be located:
  fltmc.exe attach svhdxflt D:\
• Create OS VHD or VHDX files as you would regularly, on an SMB file share. Create your VMs as you would regularly would, using the OS VHD or VHDX file. Both Generation 1 and Generation 2 VMs are fine.
• Create your VHDX data files to be shared as fixed-size or dynamically expanding, on the disk where you manually attached the Shared VHDX filter. Old VHD files are not allowed. Differencing disks are not allowed.
• Add the shared VHDX data files to multiple VMs, using the Add-VMHardDiskDrive and the “-ShareVirtualDisk” option. If using the GUI, check the box for “Share virtual disk” when adding the VHDX data files to the VM.
• Inside the VM, install Windows Server 2012 R2 or Windows Server 2012. Make sure to install the Hyper-V integration components.
• Configure Storage Spaces using the Virtual SAS disks exposed to the VMs.
• Validate and configure the cluster inside the VMs as you normally would. Configure a Scale-Out file server and create the file shares for testing.
• Install VMM 2012 R2 Preview and use it to manage the Scale-Out File Server.

The rest of this document contains detailed step-by-step instructions for each of the items outlined above.

4) Configure the physical host

# Preparation steps: Install WS2012R2, rename computer, enable remote desktop

# Install required roles and features, restart at the end
Install-WindowsFeature Failover-Clustering -IncludeManagementTools
Install-WindowsFeature Hyper-V -IncludeManagementTools -Restart

# Create a new Internal Switch
New-VMSwitch -Name Net2 -SwitchType Internal
Get-NetAdapter *Net2* | Rename-NetAdapter -NewName vNet2

# Configure vNet2 (parent port for internal switch) with a static IP address for DNS / DC
Set-NetIPInterface -InterfaceAlias vNet2 -DHCP Disabled
Remove-NetIPAddress -InterfaceAlias vNet2 -Confirm:$false
New-NetIPAddress -InterfaceAlias vNet2 -IPAddress 192.168.100.1 -PrefixLength 24
Set-DnsClientServerAddress -InterfaceAlias vNet2 -ServerAddresses 192.168.100.1

# Create 5 VMs, assumes there is a base VM ready at D:\VMS\BaseOS.VHDX
1..5 | % { New-VHD -Path D:\VMS\VM$_.VHDX –ParentPath D:\VMS\BaseOS.VHDX }
1..5 | % { New-VM -Name VM$_ -Path D:\VMS –VHDPath D:\VMS\VM$_.VHDX -Memory 2GB -SwitchName Net2 }
# Give the VMM VM a little more RAM
Set-VM -VMName VM5 -MemoryStartupBytes 4GB

# Create 3 data VHDX files
1..3 | % { New-VHD -Path D:\VMS\Data$_.VHDX -Fixed -SizeBytes 20GB }

# Manually attach volume D:\
fltMC.exe attach svhdxflt D:\
# NOTE 1: Non-CSV deployments of Shared VHDX are not supported. See the disclaimer section of this document.
# NOTE 2: Manual attached is not saved across reboots. You will have to re-issue the command after each reboot.

# Add the 3 data VHDX files to each of the 3 VMs, with Sharing option
1..3 | % { $p = ”D:\VMS\Data” + $_ + ”.VHDX” ; 2..4 | % { $v = ”VM” + $_; Write-Host $v, $p; Add-VMHardDiskDrive -VMName $v -Path $p -ShareVirtualDisk } }

# Start all the VMs
Get-VM | Start-VM

5) Configure JOSE-D (DNS, DC)

# Preparation steps: Install WS2012R2, rename computer, enable remote desktop

# Install required roles and features, restarts at the end
Install-WindowsFeature AD-Domain-Services, RSAT-ADDS, RSAT-ADDS-Tools

# Configure Internal NIC with a static IP address for DNS / DC
Get-NetAdapter | Rename-NetAdapter -NewName Internal
Set-NetIPInterface -InterfaceAlias Internal -DHCP Disabled
Remove-NetIPAddress -InterfaceAlias Internal -Confirm:$false
New-NetIPAddress -InterfaceAlias Internal -IPAddress 192.168.100.100 -PrefixLength 24
Set-DnsClientServerAddress -InterfaceAlias Internal -ServerAddresses 192.168.100.100

# create AD forest, reboots at the end
Install-ADDSForest -CreateDNSDelegation:$false -DatabasePath "C:\Windows\NTDS" -DomainMode "Win2012" -DomainName "JOSE.TEST" -DomainNetBIOSName "JOSE" -ForestMode "Win2012" -InstallDNS:$true -LogPath "C:\Windows\NTDS" -SYSVOLPath "C:\Windows\SYSVOL" 

6) Configure JOSE-A1 (Cluster A)

# Preparation steps: Install WS2012R2, rename computer, enable remote desktop

# Install required roles and features, restarts at the end
Install-WindowsFeature File-Services, FS-FileServer, Failover-Clustering
Install-WindowsFeature RSAT-Clustering -IncludeAllSubFeature

# Configure Internal NIC with a static IP address for DNS / DC
Get-NetAdapter | Rename-NetAdapter -NewName Internal
Set-NetIPInterface -InterfaceAlias Internal -DHCP Disabled
Remove-NetIPAddress -InterfaceAlias Internal -Confirm:$false
New-NetIPAddress -InterfaceAlias Internal -IPAddress 192.168.100.101 -PrefixLength 24
Set-DnsClientServerAddress -InterfaceAlias Internal -ServerAddresses 192.168.100.100

# Create 1 Pool, 7 Spaces and Initialize them
# NOTE 3: Running Storage Spaces in a guest is not supported. See the disclaimer section of this document.
# NOTE 4: This unsupported configuration cannot simulate Tiering or Write-back cache, since there are no SSDs.

$s = Get-StorageSubSystem -FriendlyName *Spaces*
New-StoragePool -FriendlyName Pool1 -StorageSubSystemFriendlyName $s.FriendlyName -PhysicalDisks (Get-PhysicalDisk -CanPool $true)
Set-ResiliencySetting -Name Mirror -NumberofColumnsDefault 1 -StoragePool (Get-StoragePool -FriendlyName Pool1)

New-VirtualDisk -FriendlyName Space1 -StoragePoolFriendlyName Pool1 -ResiliencySettingName Mirror –Size 1GB
2..7 | % { New-VirtualDisk -FriendlyName Space$_ -StoragePoolFriendlyName Pool1 -ResiliencySettingName Mirror –Size 3GB }

1..7 | % {
   $Letter ="PQRSTUV”[($_-1)]
$Number = (Get-VirtualDisk -FriendlyName Space$_ | Get-Disk).Number
Set-Disk -Number $Number -IsReadOnly 0
   Set-Disk -Number $Number -IsOffline 0
Initialize-Disk -Number $Number -PartitionStyle MBR
   New-Partition -DiskNumber $Number -DriveLetter $Letter -UseMaximumSize 
   Initialize-Volume -DriveLetter $Letter -FileSystem NTFS -Confirm:$false
}

# Join domain, restart the machine
Add-Computer -DomainName JOSE.TEST -Credential (Get-Credential) –Restart

7) Configure JOSE-A2 (Cluster A)

# Preparation steps: Install WS2012R2, rename computer, enable remote desktop

# Install required roles and features, restarts at the end
Install-WindowsFeature File-Services, FS-FileServer, Failover-Clustering
Install-WindowsFeature RSAT-Clustering -IncludeAllSubFeature

# Configure Internal NIC with a static IP address for DNS / DC
Get-NetAdapter | Rename-NetAdapter -NewName Internal
Set-NetIPInterface -InterfaceAlias Internal -DHCP Disabled
Remove-NetIPAddress -InterfaceAlias Internal -Confirm:$false
New-NetIPAddress -InterfaceAlias Internal -IPAddress 192.168.100.102 -PrefixLength 24
Set-DnsClientServerAddress -InterfaceAlias Internal -ServerAddresses 192.168.100.100

# Join domain, restart the machine
Add-Computer -DomainName JOSE.TEST -Credential (Get-Credential) –Restart

8) Configure JOSE-A3 (Cluster A)

# Preparation steps: Install WS2012R2, rename computer, enable remote desktop

# Install required roles and features, restarts at the end
Install-WindowsFeature File-Services, FS-FileServer, Failover-Clustering
Install-WindowsFeature RSAT-Clustering -IncludeAllSubFeature

# Configure Internal NIC with a static IP address for DNS / DC
Get-NetAdapter | Rename-NetAdapter -NewName Internal
Set-NetIPInterface -InterfaceAlias Internal -DHCP Disabled
Remove-NetIPAddress -InterfaceAlias Internal -Confirm:$false
New-NetIPAddress -InterfaceAlias Internal -IPAddress 192.168.100.103 -PrefixLength 24
Set-DnsClientServerAddress -InterfaceAlias Internal -ServerAddresses 192.168.100.100

# Join domain, restart the machine
Add-Computer -DomainName JOSE.TEST -Credential (Get-Credential) -Restart

9) Configure Cluster JOSE-A

# Validate cluster
Test-Cluster -Node JOSE-A1, JOSE-A2, JOSE-A3

# Create cluster
New-Cluster –Name JOSE-A -Node JOSE-A1, JOSE-A2, JOSE-A3 -StaticAddress 192.168.100.110

# Rename and configure networks
(Get-ClusterNetwork).Name = “Internal”
(Get-ClusterNetwork).Role = 3
(Get-Cluster).UseClientAccessNetworksForSharedVolumes=1

# Rename Witness Disk
$w = Get-ClusterResource | ? { $_.OwnerGroup -eq "Cluster Group" -and $_.ResourceType -eq "Physical Disk"}
$w.Name = "WitnessDisk"

# Add remaining disks to Cluster Shared Volumes
Get-ClusterResource | ? OwnerGroup -eq "Available Storage" | Add-ClusterSharedVolume

# Create Scale-Out File Server
Add-ClusterScaleOutFileServerRole JOSE-F

# Create SMB shares
1..6 | % {
MD C:\ClusterStorage\Volume$_\Share
New-SmbShare -Name Share$_ -Path C:\ClusterStorage\Volume$_\Share -FullAccess JOSE.Test\Administrator
Set-SmbPathAcl -ShareName Share$_
}

10) Configure JOSE-V

# Preparation steps: Install WS2012R2, rename computer, enable remote desktop

# Install required roles and features, restarts at the end
Install-WindowsFeature RSAT-Clustering -IncludeAllSubFeature

# Configure Internal NIC with a static IP address for DNS / DC
Get-NetAdapter | Rename-NetAdapter -NewName Internal
Set-NetIPInterface -InterfaceAlias Internal -DHCP Disabled
Remove-NetIPAddress -InterfaceAlias Internal -Confirm:$false
New-NetIPAddress -InterfaceAlias Internal -IPAddress 192.168.100.104 -PrefixLength 24
Set-DnsClientServerAddress -InterfaceAlias Internal -ServerAddresses 192.168.100.100

# Join domain, restart the machine
Add-Computer -DomainName JOSE.TEST -Credential (Get-Credential) -Restart

# Map the SMB shares
1..6 | % { 
    $d ="PQRSTU"[($_-1)] + “:”
New-SmbMapping -LocalPath $d -RemotePath \\JOSE-F\Share$_ -Persistent $true
}

# Create the test files
1..6 | % { 
   $f ="PQRSTU"[($_-1)] + “:\testfile.dat”
fsutil file createnew $f (256MB)
   fsutil file setvaliddata $f (256MB)
}

# Configure SMB Bandwidth Limits to avoid overloading the virtual environment

Add-WindowsFeature FS-SMBBW
Set-SmbBandwidthLimit -Category Default -BytesPerSecond 16MB

# Run SQLIO (assumes SQLIO.EXE was copied to C:\SLQIO)

c:\sqlio\sqlio.exe -s9999 -kR -t1 -o16 -b8 -BN -LS -frandom -dPQRSTU testfile.dat

# To remove a node while the workload is running (wait a few minutes for rebalancing)
Remove-ClusterNode -Cluster JOSE-A –Name JOSE-A3

# To add a node while the workload is running (wait a few minutes for rebalancing)
Add-ClusterNode -Cluster JOSE-A –Name JOSE-A3

11) Running the tests without VMM (from JOSE-V)

Start Performance Monitor

Start a performance monitor session

Switch to “Histogram Bar” view to show the performance side-by-side

Add a counter for SMB Server Shares, Data bytes/sec, _total instance for JOSE-A1, JOSE-A2 and JOSE-A3.

Query the cluster shared volume ownership on Cluster A, with 2 nodes

Get-ClusterSharedVolume -Cluster JOSE-A | Sort OwnerNode | FT OwnerNode, Name, State -AutoSize

OwnerNode Name           State
--------- ----           -----
JOSE-A1   Cluster Disk 6 Online
JOSE-A1   Cluster Disk 3 Online
JOSE-A1   Cluster Disk 4 Online
JOSE-A2   Cluster Disk 7 Online
JOSE-A2   Cluster Disk 2 Online
JOSE-A2   Cluster Disk 5 Online

Run SQLIO to issue 8KB IOs

C:\sqlio\sqlio.exe -s9999 -kR -t1 -o16 -b8 -BN -LS -frandom -dPQRSTU testfile.dat

Add a 3^rd node and wait for it to take 1/3 of the load

Add-ClusterNode -Cluster JOSE-A -Name JOSE-A3

Wait 2 and a half minutes to transition to the following state.

Re- query the cluster shared volume ownership on Cluster A, now with 3 nodes

Get-ClusterSharedVolume -Cluster JOSE-A | Sort OwnerNode | FT OwnerNode, Name, State -AutoSize

OwnerNode Name           State
--------- ----           -----
JOSE-A1   Cluster Disk 3 Online
JOSE-A1   Cluster Disk 6 Online
JOSE-A2   Cluster Disk 5 Online
JOSE-A2   Cluster Disk 7 Online
JOSE-A3   Cluster Disk 2 Online
JOSE-A3   Cluster Disk 4 Online

12) Installing VMM on JOSE-V

# Install .NET Framework
Install-WindowsFeature NET-Framework-Core

# Install Windows Server 2012 R2 ADK
C:\ADK\adksetup.exe
# Select only the “Deployment Tools” and “Windows PE” options

# Install SQL Server 2012 SP1
# New SQL Server standalone installation, Feature Selection: Database Engine Services

# Install VMM 2012 R2
# Features selected to be added: VMM management server, VMM console
# Database: VirtualManagerDB database will be created on JOSEBDA-V
# Service Account: Local System account

13) Configuring VMM on JOSE-V

Bring the File Server Cluster under VMM management

Select Fabric and use the option to add Storage Devices

Add a Windows-based file server

Specify the full path to the file server cluster:

Verify the File Server Cluster was properly discovered by VMM

Check the provider

Check the Storage Spaces discovery

Check the Scale-Out File Server and file share discovery

Remove the File Server Cluster node (demo starts with 2 nodes)

Under the properties of the File Server Cluster, remove node 3

Check progress under running Jobs

While running a workload, add a File Server Cluster node

Under the properties of the File Server Cluster, add node 3 specifying the full path of to the server

Check progress under running Jobs

14) Verifying systems’ configuration

 PS C:\> Get-PhysicalDisk -CimSession JOSE-A1, JOSE-A2, JOSE-A3 | Sort PSComputerName, Size

PS C:\> Get-Disk -CimSession JOSE-A1, JOSE-A2, JOSE-A3 | Sort PSComputerName, Size

Number Friendly Name                  OperationalStatus Total Size Partition Style PSComputerName
------ -------------                  ----------------- ---------- --------------- --------------
4      Microsoft Storage Space Device Online                  1 GB MBR             JOSE-A1
10     Microsoft Storage Space Device Online                  3 GB MBR             JOSE-A1
8      Microsoft Storage Space Device Online                  3 GB MBR             JOSE-A1
0      Virtual HD ATA Device          Online                 40 GB MBR             JOSE-A1
4      Microsoft Storage Space Device Online                  3 GB MBR             JOSE-A2
5      Microsoft Storage Space Device Online                  3 GB MBR             JOSE-A2
0      Virtual HD ATA Device          Online                 40 GB MBR             JOSE-A2
4      Microsoft Storage Space Device Online                  3 GB MBR             JOSE-A3
5      Microsoft Storage Space Device Online                  3 GB MBR             JOSE-A3
0      Virtual HD ATA Device          Online                 40 GB MBR             JOSE-A3

PS C:\> Get-ClusterNode -Cluster JOSE-A | FT Cluster, NodeName, State, Id -AutoSize

Cluster NodeName State Id
------- -------- ----- --
JOSE-A  JOSE-A1     Up 2
JOSE-A  JOSE-A2     Up 1
JOSE-A  JOSE-A3     Up 3

PS C:\> Get-ClusterResource -Cluster JOSE-A | FT –AutoSize

Name                             State  OwnerGroup    ResourceType
----                             -----  ----------    ------------
Cluster IP Address               Online Cluster Group IP Address
Cluster Name                     Online Cluster Group Network Name
Cluster Pool 1                   Online Cluster Group Storage Pool
JOSE-F                           Online JOSE-F        Distributed Network Name
Scale-Out File Server (\\JOSE-F) Online JOSE-F        Scale Out File Server
WitnessDisk                      Online Cluster Group Physical Disk

PS C:\> Get-ClusterSharedVolume -Cluster JOSE-A | FT -AutoSize

Name           State  Node
----           -----  ----
Cluster Disk 2 Online JOSE-A3
Cluster Disk 3 Online JOSE-A2
Cluster Disk 4 Online JOSE-A3
Cluster Disk 5 Online JOSE-A1
Cluster Disk 6 Online JOSE-A2
Cluster Disk 7 Online JOSE-A1

PS C:\> Get-SmbShare Share* -CimSession JOSE-F | FT -AutoSize

Name   ScopeName Path                            Description PSComputerName
----   --------- ----                            ----------- --------------
Share1 JOSE-F    C:\ClusterStorage\Volume1\Share             JOSE-F
Share2 JOSE-F    C:\ClusterStorage\Volume2\SHARE             JOSE-F
Share3 JOSE-F    C:\ClusterStorage\Volume3\Share             JOSE-F
Share4 JOSE-F    C:\ClusterStorage\Volume4\Share             JOSE-F
Share5 JOSE-F    C:\ClusterStorage\Volume5\SHARE             JOSE-F
Share6 JOSE-F    C:\ClusterStorage\Volume6\Share             JOSE-F

15) Final Notes

• Keep in mind that there are dependencies between the services running on each VM.
• To shut them down, start with VM5 and end with VM1, waiting for each one to go down completely before moving to the next one.
• To bring the VMs up again, go from VM1 to VM5, waiting for the previous one to be fully up (with low to no CPU usage) before starting the next one.
• I hope you enjoyed these step-by-step instructions. I strongly encourage you to try them out and perform the entire installation yourself. It’s a good learning experience.
• Let me know how these steps worked for you using the comment section. If you run into any issues or found anything particularly interesting, don’t forget to mention the number of the step.

To see all of the posts in this series, check out the What’s New in Windows Server & System Center 2012 R2 archive.

The Storage Networking Industry Association (SNIA) is hosting the 10th Storage Developer Conference (SDC) in the Hyatt Regency in beautiful Santa Clara, CA (Silicon Valley) on the week of September 16th. As usual, Microsoft is one of the underwriters of the SNIA SMB2/SMB3 PlugFest, which is co-located with the SDC event.

For developers working with storage-related technologies, this event gathers a unique crowd and includes a rich agenda that you can find at http://www.storagedeveloper.org. Many of the key industry players are represented and this year’s agenda lists presentations from EMC, Fujitsu, Google, Hortonworks, HP, Go Daddy, Huawei, IBM, Intel, Microsoft, NEC, NetApp, Netflix, Oracle, Red Hat, Samba Team, Samsung, Spectra Logic, SwitfTest, Tata and many others.

It’s always worth reminding you that the SDC presentations are usually delivered to developers by the actual product development teams and frequently the actual developer of the technology is either delivering the presentation or is in the room to take questions. That kind of deep insight is not common in every conference out there.

Presentations by Microsoft this year include:

For a taste of what SDC presentations look like, make sure to visit the site for last year’s event, where you can find the downloadable PDF files for most and video recordings for some. You can find them at http://www.snia.org/events/storage-developer2012/presentations12.

Registration for SDC 2013 is open at http://www.storagedeveloper.org and you should definitely plan to attend. If you are registered, leave a comment and let’s plan to meet when we get there!