Quantcast
Channel: Jose Barreto's Blog
Viewing all 155 articles
Browse latest View live

Windows Server 2012: Which version of the SMB protocol (SMB 1.0, SMB 2.0, SMB 2.1 or SMB 3.0) you are using on your File Server?

$
0
0

1. Introduction

With the upcoming release of SMB 3.0 as part of Windows 8 and Windows Server 2012, I am frequently asked about how older versions of Windows will behave when connecting to or from these new versions. Upgrading to a new version of SMB is something that happened a few times over the years and we established a process in the protocol itself by which clients and servers negotiate the highest version that both support.

 

2. Versions

There are several different versions of SMB used by Windows operating systems:

  • CIFS – The ancient version of SMB that was part of Microsoft Windows NT 4.0 in 1996. SMB1 supersedes this version.
  • SMB 1.0 (or SMB1) – The version used in Windows 2000, Windows XP, Windows Server 2003 and Windows Server 2003 R2
  • SMB 2.0 (or SMB2) – The version used in Windows Vista (SP1 or later) and Windows Server 2008
  • SMB 2.1 (or SMB2.1) – The version used in Windows 7 and Windows Server 2008 R2
  • SMB 3.0 (or SMB3) – The version used in Windows 8 and Windows Server 2012

Windows NT is no longer supported, so CIFS is definitely out. Windows Server 2003 R2 with a current service pack is under Extended Support until 2015, so SMB1 is still around for a little while. SMB 2.x in Windows Server 2008 and Windows Server 2008 R2 will be around at least until 2018, when Extended Support for those products is scheduled to expire. You can find the most current information on the support lifecycle page for Windows Server. The information is subject to the Microsoft Policy Disclaimer and Change Notice.  You can use the support pages to also find support policy information for Windows XP, Windows Vista and Windows 7.

 

3. Negotiated Versions

Here’s a table to help you understand what version you will end up using, depending on what Windows version is running as the SMB client and what version of Windows is running as the SMB server:

Client / Server OSWindows 8
Windows Server 2012
Windows 7
Windows Server 2008 R2
Windows Vista
Windows Server 2008
Previous versions
of Windows
Windows 8
Windows Server 2012
SMB 3.0SMB 2.1SMB 2.0SMB 1.0
Windows 7
Windows Server 2008 R2
SMB 2.1SMB 2.1SMB 2.0SMB 1.0
Windows Vista
Windows Server 2008
SMB 2.0SMB 2.0SMB 2.0SMB 1.0
Previous versions
of Windows
SMB 1.0SMB 1.0SMB 1.0SMB 1.0

 

 

4. Using PowerShell to check the SMB version

In Windows 8 or Windows Server 2012, there is a new PowerShell cmdlet that can easily tell you what version of SMB the client has negotiated with the File Server. You simply access a remote file server (or create a new mapping to it) and use Get-SmbConnection. Here’s an example:
 

PS C:\> Get-SmbConnection
ServerName   ShareName  UserName            Credential          Dialect   NumOpens
----------   ---------  --------            ----------          -------   --------
FileServer1  IPC$       DomainName\UserN... DomainName.Testi... 3.00      0
FileServer1  FileShare  DomainName\UserN... DomainName.Testi... 3.00      14
DomainCtrl1  netlogon   DomainName\Compu... DomainName.Testi... 2.10      1

 
In the example above, a server called “FileServer1” was able to negotiate up to version 3.0. That means that both the client and the server support the latest version of the SMB protocol. You can also see that another server called “DomainCtrl1” was only able to negotiate up to version 2.1. You can probably guess that it’s a domain controller running Windows Server 2008 R2.

If you just want to find the version of SMB running on your own computer, you can use a loopback share combined with the Get-SmbConnection cmdlet. Here’s an example:
 

PS C:\> dir \\localhost\c$
 
Directory:
\\localhost\c$
 
Mode                LastWriteTime     Length Name
----                -------------     ------ ----
d----         5/19/2012   1:54 AM            PerfLogs
d-r--          6/1/2012  11:58 PM            Program Files
d-r--          6/1/2012  11:58 PM            Program Files (x86)
d-r--         5/24/2012   3:56 PM            Users
d----          6/5/2012   3:00 PM            Windows

PS C:\> Get-SmbConnection -ServerName localhost

ServerName  ShareName  UserName            Credential          Dialect  NumOpens
----------  ---------  --------            ----------          -------  --------
localhost   c$         DomainName\UserN... DomainName.Testi... 3.00     0

 
Please note that you have about 10 seconds after you issue the “dir” command to run the “Get-SmbConnection” cmdlet. The SMB client will tear down the connections if there is no activity between the client and the server. It might help to know that you can use the alias “gsmbc” instead of the full cmdlet name.

 

5. Features and Capabilities

Here’s a very short summary of what changed with each version of SMB:

  • From SMB 1.0 to SMB 2.0 - The first major redesign of SMB
    • Increased file sharing scalability
    • Improved performance
      • Request compounding
      • Asynchronous operations
      • Larger reads/writes
    • More secure and robust
      • Small command set
      • Signing now uses HMAC SHA-256 instead of MD5
      • SMB2 durability
  • From SMB 2.0 to SMB 2.1
    • File leasing improvements
    • Large MTU support
    • BranchCache
  • From SMB 2.1 to SMB 3.0
    • Availability
      • SMB Transparent Failover
      • SMB Witness
      • SMB Multichannel
    • Performance
      • SMB Scale-Out
      • SMB Direct (SMB 3.0 over RDMA)
      • SMB Multichannel
      • Directory Leasing
      • BrachCache V2
    • Backup
      • VSS for Remote File Shares
    • Security
      • SMB Encryption using AES-CCM (Optional)
      • Signing now uses AES-CMAC
    • o Management
      • SMB PowerShell
      • Improved Performance Counters
      • Improved Eventing

You can get additional details on the SMB 2.0 improvements listed above at
http://blogs.technet.com/b/josebda/archive/2008/12/09/smb2-a-complete-redesign-of-the-main-remote-file-protocol-for-windows.aspx

You can get additional details on the SMB 3.0 improvements listed above at
http://blogs.technet.com/b/josebda/archive/2012/05/03/updated-links-on-windows-server-2012-file-server-and-smb-3-0.aspx

   

6. Recommendation

We strongly encourage you to update to the latest version of SMB, which will give you the most scalability, the best performance, the highest availability and the most secure SMB implementation. The motto we used at SDC 2011 for the new version was “Bigger. Faster. Scalier.”

Please note that Windows Server 2012 Hyper-V only supports SMB 3.0 for remote file storage. This is due mainly to the availability features (SMB Transparent Failover, SMB Witness and SMB Multichannel), which did not exist in previous versions of SMB. The additional scalability and performance is also very welcome in this scenario. The Hyper-V Best Practices Analyzer (BPA) will warn you if an older version is detected.

   

7. Conclusion

We’re excited about SMB 3.0, but always concerned about keeping as much backwards compatibility as possible. This release brings several must-have new capabilities and we encourage you to learn more about them. We hope you will be convinced to start planning your upgrades as early as possible.

 


 

Note 1: SMB 2.2

You might have heard of an SMB version 2.2. We actually used that version number in some pre-release versions of Windows 8 and Windows Server 2012 and several events that happened before April 2012. However, we decided to rename SMB 2.2 to SMB 3.0 to better convey the amount of change included in this new release. You can get details about that version number change at http://blogs.technet.com/b/windowsserver/archive/2012/04/19/smb-2-2-is-now-smb-3-0.aspx.

 

Note 2: Protocol Documentation

If you consider yourself an SMB geek and you actually want to understand the SMB NEGOTIATE command in greater detail, you can read the [MS-SMB2-Preview] protocol documentation (which covers SMB 2.0, 2.1 and 3.0), currently available from http://msdn.microsoft.com/en-us/library/ee941641.aspx. In regards to protocol version negotiation, you should pay attention to the following sections of the document:

  • 1.7: Versioning and Capability Negotiation
  • 2.2.3: SMB2 Negotiate Request
  • 2.2.4: SMB2 Negotiate Response

Section 1.7 includes this nice state diagram describing the inner workings of protocol negotiation:

SMB 3.0 Protocol Negotiation

 

Note 3: Third-party implementations

There are several implementations of the SMB protocol from someone other than Microsoft. If you use one of those implementations of SMB, you should ask whoever is providing the implementation which version of SMB they implement for each version of their product.

During the keynote at the SNIA’s Storage Developers Conference in September 2011, two major storage vendors (NetApp and EMC) announced their commitment to implementing SMB 3.0 (back then called SMB 2.2) by the time Windows Server 2012 is generally available. You can view the recording of that keynote at http://www.snia.org/news_events/multimedia/video/events/featured#pfenning_pinkerton_file_protocols (watch the entire keynote and or fast forward to minute 50 to see the partner comments).

Also, a member of the Samba team posted a blog in March 2012 about their work on a Samba implementation of SMB 3.0 (back then called SMB 2.2) at http://blog.obnox.de/samba-team-visits-microsoft-for-smb2-2-interop-event/. Samba runs on several non-Windows operating systems and it’s the default SMB implementation for Linux.

We are hoping for many more implementations and we’ll all be meeting at the next SMB Plugfest in Santa Clara, CA in September 2012.


Microsoft Partners showcase solutions related to Windows Server 2012 at TechEd 2012 North America

$
0
0

The Microsoft TechEd 2012 North America conference is happening this week in Orlando, FL. As part of that event, a number of partners are showing their hardware and/or software at the exhibit area called the TechExpo.

I have spent some time there and took a few pictures of a few partners that have solutions related to the File Server or SMB 3.0 (storage, networking, servers). I tried to capture the signage on each booth, but you need to click on each picture to see a larger version of it.

 

Chelsio

www.chelsio.com

Chelsio

DataOn

www.dataonstorage.com

dataon

EMC

www.emc.com

emc

Fujitsu

www.fujitsu.com

fujitsu

Hitachi

www.hds.com

hitachi

HP

www.hp.com

hp

IBM

www.ibm.com

ibm

Intel

www.intel.com

intel

LSI

www.lsi.com

lsi

Mellanox

www.mellanox.com

mellanox

NetApp

www.netapp.com

netapp

Quanta

www.quantaqct.com

quanta

RAID Inc.

www.raidinc.com

raidinc

Violin Memory

www.violin-memory.com

violin

X-IO

www.x-io.com

xio

 

A big THANKS to each and every one of them for their hard work with Windows Server 2012. There are more partners at the expo, but I did not have time to visit them all. Sorry if I missed your booth.

If you're attending the event, a reminder that the TechExpo will be open tomorrow from 10:30 AM to 2:00 PM.

File Server team sessions at TechEd 2012 North America available for streaming/download

$
0
0

We had a great time showing Windows Server 2012 at TechEd North America and I wanted to share that the recording for our team’s sessions are now publicly available.

That includes our sessions on SMB 3.0 and a number of demos, many of them using one of the Cluster-in-box configurations we were showing at the Technical Learning Center.

Find the links in the table below, sorted by session code:

 

CodeTitleSpeaker(s)
SIA207Windows Server 2012 Dynamic Access Control OverviewGunjan Jain, Nir Ben Zvi
SIA316Windows Server 2012 Dynamic Access Control Best Practices and Case Study Deployments in Microsoft ITBrian Puhl, Matthias Wollnik
VIR306Hyper-V over SMB: Remote File Storage Support in Windows Server 2012 Hyper-VJose Barreto
WSV303Windows Server 2012 High-Performance, Highly-Available Storage Using SMBClaus Joergensen, Gene Chellis
WSV308Standards Support and Interoperability in Windows Server 2012: Storage, Networking, and ManagementGene Chellis, Jeffrey Snover,
See-Mong Tan, Wojtek Kozaczynski
WSV310Windows Server 2012: Cluster-in-a-Box, RDMA, and MoreJohn Loveall, Spencer Shepler
WSV314Windows Server 2012 NIC Teaming and SMB Multichannel SolutionsDon Stanwyck, Jose Barreto
WSV322Update Management in Windows Server 2012: Revealing Cluster-Aware Updating and the New Generation of WSUSErin Chapple, Mallikarjun Chadalapaka
WSV328The Path to Continuous Availability with Windows Server 2012Gene Chellis, Jim Pinkerton
WSV330How to Increase SQL Availability and Performance Using Window Server 2012 SMB 3.0 SolutionsClaus Joergensen, Gunter Zink
WSV334Windows Server 2012 File and Storage Services ManagementFabian Uhse, Mathew Dickson
WSV410Continuously Available File Server: Under the HoodClaus Joergensen

 

Also check this blog post showing the partner booths in the TechExpo area.

The basics of SMB Multichannel, a feature of Windows Server 2012 and SMB 3.0

$
0
0

1. Introduction

Windows Server 2012 includes a new feature called SMB Multichannel, part of the SMB 3.0 protocol, which increases the network performance and availability for File Servers.

 

1.1. Benefits

SMB Multichannel allows file servers to use multiple network connections simultaneously and provides the following capabilities:

  • Increased throughput. The file server can simultaneously transmit more data using multiple connections for high speed network adapters or multiple network adapters.
  • Network Fault Tolerance. When using multiple network connections at the same time, the clients can continue to work uninterrupted despite the loss of a network connection.
  • Automatic Configuration: SMB Multichannel automatically discovers the existence of multiple available network paths and dynamically adds connections as required.

 

1.2. Requirements

SMB Multichannel requires the following:

  • At least two computers running Windows Server 2012 or Windows 8.

At least one of the configurations below:

  • Multiple network adapters
  • One or more network adapters that support RSS (Receive Side Scaling)
  • One of more network adapters configured with NIC Teaming
  • One or more network adapters that support RDMA (Remote Direct Memory Access)

 

2. Configuration

 

2.1. Installing

SMB Multichannel is enabled by default. There is no need to install components, roles, role services or features.
The SMB client will automatically detect and use multiple network connections if a proper configuration is identified.

 

2.2. Disabling

SMB Multichannel is enabled by default and there is typically no need to disable it.
However, if you want to disable SMB Multichannel (for testing purposes, for instance), you can use the following PowerShell cmdlets:

On the SMB server side:

Set-SmbServerConfiguration -EnableMultiChannel $false

On the SMB client side:

Set-SmbClientConfiguration -EnableMultiChannel $false

Note: Disabling the feature on either the client or the server prevent the systems from using it.

 

2.3. Re-enabling

You can re-enable SMB Multichannel after you disabled it by using:

On the SMB server side:

Set-SmbServerConfiguration -EnableMultiChannel $true

On the SMB client side:

Set-SmbClientConfiguration -EnableMultiChannel $true

Note: You need to enable the feature on both the client or the server to start using it again.

 

3. Sample Configurations

This section provides details on how SMB Multichannel works with a few different configurations using a variety of Network Interface Cards (NIC). Please note that these are only samples and many other configurations not detailed here are possible.

 

3.1. Single RSS-capable NIC

This typical configuration involves an SMB client and SMB Server configured with a single 10GbE NIC. Without SMB multichannel, if there is only one SMB session established, SMB uses a single TCP/IP connection, which naturally gets affinitized with a single CPU core. If lots of small IOs are performed, it’s possible for that core to become a performance bottleneck.

Most NICs today offer a capability called Receive Side Scaling (RSS), which allows multiple connections to be spread across multiple CPU cores automatically. However, when using a single connection, RSS cannot help.

With SMB Multichannel, if the NIC is RSS-capable, SMB will create multiple TCP/IP connections for that single session, avoiding a potential bottleneck on a single CPU core when lots of small IOs are required.

image

 

3.2. Multiple NICs

When using multiple NICs without SMB multichannel, if there is only one SMB session established, SMB creates a single TCP/IP connection using only one of the many NICs available. In this case, not only it’s not possible to aggregate the bandwidth of the multiple NICs (achieve 2Gbps when using two 1GbE NICs, for instance), but there is a potential for failure if the specific NIC chosen is somehow disconnected or disabled.

With Multichannel, SMB will create multiple TCP/IP connections for that single session (at least one per interface or more if they are RSS-capable). This allows SMB to use the combined NIC bandwidth available and makes it possible for the SMB client to continue to work uninterrupted if a NIC fails.

image

 

3.3. Teamed NICs

Windows Server 2012 supports the ability to combine multiple NICs into one using a new feature commonly referred to as NIC teaming. Although a team always provides fault tolerance, SMB without Multichannel will create only one TCP/IP connection per team, leading to limitations in both the number of CPU cores engaged and the use of the full team bandwidth.

SMB Multichannel will create multiple TCP/IP connections, allowing for better balancing across CPU cores with a single SMB session and better use of the available bandwidth. NIC Teaming will continue to offer the failover capability, which will work faster than using SMB Multichannel by itself. NIC Teaming is also recommended because it offers failover capabilities to other workloads that do not rely on SMB and therefore cannot benefit from the failover capabilities of SMB Multichannel.

image

Note: A team of RDMA-capable NICs is always reported as non-RDMA capable. If you intend to use the RDMA capabilities of the NIC, do not team them.

 

3.4. Single or Multiple RDMA NICs

SMB Multichannel is the feature responsible for detecting the RDMA capabilities of NICs to enable the SMB Direct feature (SMB over RDMA). Without SMB Multichannel, SMB will use regular TCP/IP with these RDMA-capable NICs (they all provide a TCP/IP stack side-by-side with the new RDMA stack).

With SMB Multichannel, SMB will detect the RDMA capability and create multiple RDMA connections for that single session (two per interface). This allows SMB to use the high throughput, low latency and low CPU utilization offered by these RDMA NICs. It will also offer fault tolerance if you’re using multiple RDMA interfaces.

image

Note 1: A team of RDMA-capable teams is reported as non-RDMA capable. If you intend to use the RDMA capability of the NIC, do not team them.

Note 2: After at least one RDMA connection is created, the TCP/IP connection used for the original protocol negotiation is no longer used. However, that connection is kept around in case the RDMA connections fail.

 

3.5. Multichannel, RDMA and NIC Teaming compatibility

Here’s a table summarizing the different capabilities available when combining SMB Multichannel, RDMA (SMB Direct) and NIC Teaming:

image

For non-RDMA NICs, your best bet is combining NIC Teaming with SMB Multichannel. This will give you the best throughput, plus fault tolerance for applications using SMB and other protocols.

When using RDMA NICs, LBFO is not a good option, since it disables the RDMA capability of the NIC.

 

3.6. Sample Configurations that do not use SMB Multichannel

The following are sample network configurations that do not use SMB Multichannel:

  • Single non-RSS-capable network adapters. This configuration would not benefit from multiple network connections, so SMB Multichannel is not used.
  • Network adapters of different speeds. SMB Multichannel will choose to use the faster network adapter. Only network interfaces of same type (RDMA, RSS or none) and speed will be used simultaneously by SMB Multichannel, so the slower adapter will be idle.

 

4. Operations

Here are a few common scenarios for testing SMB Multichannel:

 

4.1. Compare a file copy with and without SMB Multichannel

To measure the increased throughput provided by SMB Multichannel, follow these steps:

  • Setup SMB Multichannel in one the configurations described earlier
  • Measure the time to perform a long-running file copy using SMB Multichannel
  • Disable SMB Multichannel (see instructions in previous topic).
  • Measure the time it takes to perform the same file copy without SMB Multichannel
  • Re-enable SMB Multichannel (see instructions in previous topic).
  • Compare the two results.

Note: to avoid the effects of caching, you should:

  • Copy a large amount of data (more data than would fit on memory).
  • Perform the copy twice, using first copy as a warm-up and timing only the second copy.
  • Restart both the server and the client before each test to make sure they operate under similar conditions.

 

4.2. Fail one of multiple NICs during a file copy with SMB Multichannel

To confirm the failover capability of SMB Multichannel, follow these steps:

  • Make sure SMB Multichannel is operating in a multi-NIC configuration.
  • Perform a long-running file copy.
  • While the file copy is running, simulate a failure one of the network paths by disconnecting one of the cables (or by disabling one of the NICs)
  • Confirm that the file copy continues using the surviving NIC, without any file copy errors.

Note: Make sure there are no other workloads using the disconnected/disabled path, to avoid failures in workloads that do not leverage SMB Multichannel.

 

5. Troubleshooting

Here are troubleshooting tips for SMB Multichannel.

 

5.1. Verifying if you’re using SMB Multichannel

You can use the following steps to verify you are using SMB Multichannel.

Step 1: Verify network adapter configuration

Use the following PowerShell cmdlets to verify you have multiple NICs and/or to verify the RSS and RDMA capabilities of the NICs. Run on both the SMB server and the SMB client.

Get-NetAdapter

Get-NetAdapterRSS

Get-NetAdapterRDMA

Get-NetAdapterHardwareInfo

 

Step 2: Verify SMB configuration

Use the following PowerShell cmdlets to make sure SMB Multichannel is enabled, confirm the NICs are being properly recognized by SMB and that their RSS and RDMA capabilities are being properly identified.

On the SMB client, run the following PowerShell cmdlets:

Get-SmbClientConfiguration | Select EnableMultichannel

Get-SmbClientNetworkInterface

On the SMB server, run the following PowerShell cmdlets:

Get-SmbServerConfiguration | Select EnableMultichannel

Get-SmbServerNetworkInterface

 

Step 3: Verify the SMB connection

On the SMB client, start a long-running file copy to create a lasting session with the SMB Server. While the copy is ongoing, open a PowerShell window and run the following cmdlets to verify the connection is using the right version of SMB and that SMB Multichannel is working:

Get-SmbConnection

Get-SmbMultichannelConnection

Get-SmbMultichannelConnection -IncludeNotSelected

 

5.2. View SMB Multichannel Events (Optional)

SMB 3.0 offers a “Object State Diagnostic” event log that can be used to troubleshoot Multichannel (and therefore RDMA) connections.
Keep  in mind that this is a debug log, so it’s very verbose and requires a special procedure for gathering the events. You can follow the steps below:

Step 1: Capture the events

First, enable the log in Event Viewer:

  • Open Server Manager
  • In Server Manager, click on “Tools”, then “Event Viewer”
  • On the menu, select “View” then “Show Analytic and Debug Logs”
  • Expand the tree on the left: Applications and Services Log, Microsoft, Windows, SMB Client, ObjectStateDiagnostic
  • On the “Actions” pane on the right, select “Enable Log”
  • Click OK to confirm the action.

After the log is enabled, perform the operation that requires an RDMA connection. For instance, copy a file or run a specific operation.
If  you’re using mapped drives, be sure to map them after you enable the log, or else the connection events won’t be properly captured.

Next, disable the log in Event Viewer:

  • In Event Viewer, make sure you select Applications and Services Log, Microsoft, Windows, SMB Client, ObjectStateDiagnostic
  • On the “Actions” page on the right, “Disable Log”

Step 2: Option 1: View the events in Event Viewer

You can review the events on the log in Event Viewer. You can filter the log to include only the SMB events that confirm that you have an SMB Multichannel or only error events.

The “Smb_MultiChannel” keyword will filter for connection, disconnection and error events related to SMB. You can also filter by event numbers 30700 to 30706.

  • Click on the “ObjectStateDiagnostic” item on the tree on the left.
  • On the “Actions” pane on the right, select “Filter Current Log…”
  • Click on “Filter Current Log…” on the Actions pane on the right and enter “30700-30705” on the filter for Event IDs.

To view the SMB Multichannel events using Event Viewer, use the following steps on the SMB Client:

  • In Event Viewer, expand the tree on the left to show “Applications and Service Logs”, “Microsoft”, “Windows”, “SMB Client”, “Operational”
  • Click on “Filter Current Log…” on the Actions pane on the right and enter “30700-30705” on the filter for Event IDs.

To view only errors:

  • Click on “Filter Current Log…” on the Actions pane on the right and click on the checkbox labeled “Errors”.

Step 2: Option 2: View the events using PowerShell

You can also use a PowerShell window and run the following cmdlets to view the events.

To see any RDMA-related connection events, you can use the following:

Get-WinEvent -LogName Microsoft-Windows-SMBClient/ObjectStateDiagnostic -Oldest |
? Message -match "RDMA"

To view all types of SMB Multichannel events , use the following cmdlet on the SMB Client:

Get-WinEvent -LogName Microsoft-Windows-SMBClient/ObjectStateDiagnostic –Oldest |
? { $_.Id -ge 30700 –and $_.Id –le 30705 }

To list only errors, use the following cmdlet:

Get-WinEvent -LogName Microsoft-Windows-SMBClient/ObjectStateDiagnostic -Oldest |
? { $_.Id -ge 30700 –and $_.Id –le 30705 –and $_.Level –eq 2 }

 

5.3. View SMB Performance Counters

There are several Performance counters related to SMB in Windows Server 2012. To view SMB-related performance information, follow these steps:

  • Open Server Manager
  • In the Tools menu, open Performance Monitor
  • In Performance Monitoring, click on “Performance Monitor” on the tree on the left.
  • Switch to the Report View by pressing CTRL-G twice (or using the icon on the toolbar)
  • Add performance counters to the view by pressing CTRL-N (or using the icon on the toolbar)

The following Performance Counters are useful when looking at activities related to SMB, SMB Direct and Network Direct:

Counter NameShows information for
SMB Server SharesShares on the SMB Server
SMB Server SessionsSessions on the SMB Server
SMB Client SharesShares on the SMB Client

 

6. Number of SMB Connections per Interface

SMB Multichannel will use a different number of connections depending on the type of interface:

  • For RSS-capable interfaces, 4 TCP/IP connections per interface are used
  • For RDMA-capable interfaces, 2 RDMA connections per interface are used
  • For all other interfaces, 1 TCP/IP connection per interface is used

There is also a limit of 8 connections total per client/server pair which will limit the number connections per interface.

For instance, if you have 3 RSS-capable interfaces, you will end up with 3 connections on the first, 3 connections on the second and 2 connections on the third interface.

We recommend that you keep the default settings for SMB Multichannel. However, those parameters can be adjusted.

 

6.1. Total Connections per client/server pair

You can configure the maximum total number of connections per client/server pair using:

Set-SmbClientConfiguration –MaximumConnectionCountPerServer <n>

 

6.2. Connections per RSS-capable NIC

You can configure the number SMB Multichannel connections per RSS-capable network interface using the PowerShell cmdlet:

Set-SmbClientConfiguration -ConnectionCountPerRssNetworkInterface <n>

 

6.3. Connections per RDMA-capable NIC

It is even less likely that you’ll need to adjust the number of connections per RDMA-capable interface.

That can be configured via a registry key using the following PowerShell command:

Set-ItemProperty -Path ` "HKLM:\SYSTEM\CurrentControlSet\Services\LanmanWorkstation\Parameters" `
ConnectionCountPerRdmaNetworkInterface -Type DWORD -Value <n> –Force

 

6.4. Connections for other types of NIC

For NICs that are not RSS-capable or RDMA-capable, there is likely no benefit of using multiple connections. In fact, this will likely reduce your performance.

However, for troubleshooting purposes, there is also a registry key to change the default settings of 1 connection per NIC.

That can be configured via a registry key using the following PowerShell command:

Set-ItemProperty -Path ` "HKLM:\SYSTEM\CurrentControlSet\Services\LanmanWorkstation\Parameters" `
ConnectionCountPerNetworkInterface -Type DWORD -Value <n> –Force

 

7. SMB Multichannel Constraints

Because of feedback during the Beta, we added in the Release Candidate a new set of cmdlets using the SmbMultichannelConstraint noun to allow you to control the SMB Multichannel behavior in more complex network configurations.

For instance, if you have 4 NICs in your system and you want the SMB client to use only two specific NICs when you access server FS1, you can accomplish that with the following PowerShell cmdlets:

New-SmbMultichannelConstraint -ServerName FS1 -InterfaceIndex 12, 19

Or

New-SmbMultichannelConstraint -ServerName FS1 -InterfaceAlias RDMA1, RDMA2

 

7.1. Notes on SMB Multichannel Constraints

Here are a few notes and comments on how to use SMB Multichannel Constraints:

  • This is configured on the SMB 3.0 client and it’s persisted in the registry.
  • Constraints are configured per server because you can configure different set of interfaces for each server if required.
  • You list the interfaces you want to use, not the interfaces you want to avoid.
  • You get a list of interfaces in your system using Get-NetAdapter.
  • For servers not specified, the default SMB Multichannel behavior for that client continues to apply.
  • For a reference of other cmdlets related to SMB Multichannel Constraints, check the blog post on the basics of SMB PowerShell

 

7.2. Recommendation: keep the default behavior, avoid constraints

This is an advanced setting, available only via PowerShell (or WMI). Please be careful when configuring these SMB Multichannel constraints, because you have the potential to apply a less-than-ideal configuration and end up with something that does not provide you the best level of network fault tolerance (also known as “shooting yourself in the foot”).

This is specially true if you add/remove network adapters from your system and forget to adjust your constraints accordingly. We highly recommend that you stick to the default, let-SMB-Multichannel-figure-it-out configuration if at all possible, since this will let SMB 3.0 dynamically and automatically adjust to any changes in network configuration shortly after it happens.

 

8. Conclusion

I hope this blog post has helped you understand the basics of SMB Multichannel. I would encourage you to try it out in any of the configurations shown.

For additional information on the Windows Server 2012 File Server, SMB 3.0 and SMB Direct, see the links at Updated Links on Windows Server 2012 File Server and SMB 3.0.

 


Update on 5/14: Minor content update

Update on 6/28: Added section on SMB Multichannel Constraints

Windows Server 2012 Scale-Out File Server for SQL Server 2012 - Step-by-step Installation

$
0
0

Outline

1. Introduction
1.1. Overview
1.2. Hardware
1.3. Software
1.4. Notes and disclaimers

2. Install Windows Server 2012
2.1. Preparations
2.2. Install the OS
2.3. Rename the computer
2.4. Enable Remote Desktop

3. Configure the Hyper-V Host
3.1. Install the Hyper-V role to the server
3.2. Create the VM switches
3.3. Rename the network adapters
3.4. Assign static IP addresses for the Hyper-V host

4. Create the Base VM
4.1. Preparations
4.2. Create a Base VM
4.3. Install Windows Server 2012 on the VM
4.4. Sysprep the VM
4.5. Remove the base VM

5. Configure the 4 VMs
5.1. Create 4 new differencing VHDs using the Base VHD
5.2. Create 4 similarly configured VMs
5.3. Start the 4 VMs
5.4. Complete the mini-setup for the 4 VMs
5.5. Change the computer name for each VM
5.6. For each VM, configure the networks
5.7. Review VM name and network configuration

6. Configure DNS and Active Directory
6.1. Install DNS and Active Directory Domain Services
6.2. Configure Active Directory
6.3. Join the other VMs to the domain
6.4. Create the SQL Service account

7. Configure iSCSI
7.1. Add the iSCSI Software Target
7.2. Create the LUNs and Target
7.3. Configure the iSCSI Initiators
7.4. Configure the disks

8. Configure the File Server
8.1 Install the required roles and features
8.2. Validate the Failover Cluster Configuration
8.3. Create a Failover Cluster
8.4. Configure the Cluster Networks
8.6. Create the Scale-Out File Server
8.7. Create the folders and shares

9. Configure the SQL Server
9.1. Mount the SQL Server ISO file
9.2. Run SQL Server Setup
9.3. Create a database using the clustered file share

10. Verify SMB features
10.1. Verify that SMB Multichannel is working
10.2. Query the SMB Sessions and Open Files
10.3. Planned move of a file server node
10.4. Unplanned failure of a file server node
10.5. Surviving the loss of a client NIC

11. Shut down, startup and install final notes

12. Conclusion

 


1. Introduction

1.1. Overview

In this document, I am sharing all the steps I used to create a Windows Server 2012 File Server demo or test environment, so you can experiment with some of the new technologies yourself. You only need a single computer (the specs are provided below) and the ISO file with the Windows Server 2012 evaluation version available as a free download. For the SQL part, you will need the SQL Server 2012 evaluation version, which is also available as a free download.

The demo setup includes 4 virtual machines: one domain controller and iSCSI target, two file servers and a SQL server. You need the iSCSI target and two file servers because we’re using Failover Clustering to showcase SMB Transparent Failover and SMB Scale-Out. We’ll also use multiple Hyper-V virtual networks, so we can showcase SMB Multichannel.

image

This will probably require a few hours of work end-to-end, but it is a great way to experiment with a large set of Microsoft technologies in Windows Server 2012, including:

  • Hyper-V
  • Networking
  • Domain Name Services (DNS)
  • Active Directory Domain Services (AD-DS)
  • iSCSI Target
  • iSCSI Initiator
  • Failover Clustering
  • File Servers
  • PowerShell

Follow the steps and let me know how it goes in the comment section. If you run into any issues or found anything particularly interesting, don’t forget to mention the number of the step.

1.2. Hardware

You will need the following hardware to perform the steps described here:

  • Computer capable of running Windows Server 2012 and Hyper-V (64-bit, virtualization technology) with at least 8GB of RAM
  • An 8GB USB stick, if you’re installing Windows Server from USB and copying the downloaded software around (you can also burn the software to a DVD)
  • Internet connection for downloading software and updates (DHCP enabled)

1.3. Software

You will need the following software to perform the steps described here:

1.4. Notes and disclaimers

  • The text for each step also focuses on the specific actions that deviate from the default or where a clear default is not provided. If you are asked a question or required to perform an action that you do not see described in these steps, go with the default option.
  • Obviously, a single-computer solution can never be tolerant to the failure of that computer. So, the configuration described here is not really continuously available. It’s just a simulation.
  • The configuration described here is for demo, testing or learning. You would definitely need a different configuration for a production deployment.
  • A certain familiarity with Windows administration and configuration is assumed. If you're new to Windows, this document is not for you. Sorry...
  • There are usually several ways to perform a specific configuration or administration task. What I describe here is one of those many ways. It's not necessarily the best way, just the one I personally like best at the moment.
  • In theory, you could run this demo on a host computer with only 4GB of RAM, but you would need to configure each VM to run with only 512MB of RAM, which is the bare minimum required. It works, but it will run slower.
  • This blog post is an update on a previous blog post that provided similar steps for Windows Server 2012 Beta. This post supersedes the old one.

2. Install Windows Server 2012

2.1. Preparations

  • Format a USB disk using Windows 8 or Windows Server 2012.
  • Copy the contents of the Windows Server 2012 ISO file to the USB disk.
  • To read the files from within the ISO file using Windows 8 or Windows Server 2012, just double click the ISO file to mount it.
  • Make sure the computer BIOS is configured for Virtualization. Each computer BIOS is different, so you need to find the right settings.

2.2. Install the OS

  • Use your computer’s BIOS option to boot from the USB disk.
  • After “Windows Setup” starts from the USB disk, enter the required information to install the OS:
  • Select language, time and currency format and keyboard. Then click “Next”.
  • Click “Install Now”.
  • Select the “Windows Server Datacenter - Server with a GUI” option and click “Next”.  
  • Accept the license agreement and click “Next”.
  • Select the “Custom: Install Windows only” option.
  • Select the install location for Windows Server and click “Next”.
  • Wait for the installation to complete. This will take a few minutes.
  • After the installation is completed, the OS will boot.
  • Type the administrator password twice, then click on “Finish”.

2.3. Rename the computer

  • Login to the computer using the Administrator password and rename the computer.

2.3.PS. Using PowerShell

Rename-Computer DEMO-HV0 -Restart

2.3.GUI. Using Server Manager

  • In Server Manager, click on “Local Server” on the list on left.
  • Click on the default name next to “Computer Name”.
  • Click on “Change”.
  • Enter “DEMO-HV0” as the new Computer Name and click “OK”.
  • Accept the option to restart the computer.

2.4. Enable Remote Desktop

  • Log in using the Administrator account and enable Remote Desktop.
  • After completing this step, you will be able work from a Remote Desktop connection.

2.4.PS. Using SCONFIG.EXE

  • Use a command prompt to start SCONFIG.EXE
  • Use option 7 in SCONFIG to enable remote desktop

2.4.GUI. Using Server Manager

  • In Server Manager, click on “Local Server” on the list on left.
  • Click on the “Disabled” item next to “Remote Desktop”.
  • Select the option to “Allow connections from computers running any version…”
  • Click “OK” multiple times until you’re back to the “Local Server” screen.

3. Configure the Hyper-V Host

3.1. Install the Hyper-V role to the server

  • Install the Hyper-V role and the required management tools. The computer will restart.

3.1.PS. Using PowerShell

Install-WindowsFeature Hyper-V, Hyper-V-PowerShell, Hyper-V-Tools -Restart

3.2.OUT. Sample output

PS C:\> Get-WindowsFeature *Hyper*

Display Name                                            Name                 Install State
------------                                            ----                 -------------
[X] Hyper-V                                             Hyper-V              Installed
        [X] Hyper-V Management Tools                    RSAT-Hyper-V-Tools   Installed
            [X] Hyper-V GUI Management Tools            Hyper-V-Tools        Installed
            [X] Hyper-V Module for Windows PowerShell   Hyper-V-PowerShell   Installed

3.1.GUI. Using Server Manager

  • In Server Manager, click on “Dashboard” on the list on left.
  • Click on the “Add Roles and Features”, which is option 2 under “Configure this local server”.
  • On the “Before You Begin” page, just click “Next”.
  • On the “Installation Type” page, click “Role-base or feature-based installation” and click “Next”.
  • On the “Server Selection” page, select your server and click “Next”.
  • On the “Server Role” page, select “Hyper-V”.
  • On the “Add features that are required for Hyper-V” dialog, click “Add features”, then click “Next”.  
  • On the “Features” page, just click “Next”.
  • On the “Hyper-V” page, just click “Next”.
  • On the “Create Virtual Switches” page, just click “Next”.
  • On the “Virtual Machine Migration” page, just click “Next”.
  • On the “Default Stores” page, just click “Next”.
  • On the “Confirmation” page, click “Restart target machine automatically if needed”, click “Yes” to confirm and then click “Install”.
  • The role will be installed and the computer will restart in the end.

3.2. Create the VM switches

  • Create one external virtual network (VM switch that is connected to the external network interface).
  • Create three internal virtual networks (VM switches used just to communicate between the VMs).

3.2.PS. Using PowerShell

Get-NetAdapter
Rename-NetAdapter –InterfaceDescription *Gigabit* -NewName External
New-VMSwitch –Name External -NetAdapterName External
1..3 | % { New-VMSwitch -Name Internal$_ -SwitchType Internal }
Get-VMSwitch
Get-NetAdapter

  • Note: If you’re connected via “Remote Desktop” to the server, you might temporarily lose the connection when you create the External VMSwitch.
    If using DHCP on that interface, you will be able to reconnect. If you’re using static IP addresses, you should run this step locally, not via “Remote Desktop”.

3.2.GUI. Using Hyper-V Manager

  • In Server Manager, click on “Tools” in the upper left and select “Hyper-V Manager”
  • In Hyper-V Manager, click on the server name (DEMO-HV0) on the pane on the left
  • On the task pane on the right, click on the “Virtual Switch Manager”  
  • Use the “New virtual switch” option to add 1 network named “External”, select “External” for type and select your physical NIC. Click “Apply” to confirm.
  • Use the “New virtual switch” option to add 3 networks named “Internal1”, “Internal2” and “Internal3”, each of type “Internal”. 
  • After creating the four new NICs, you should see the four new NICs in Server Manager, under Local Server
     

3.3. Rename the network adapters

  • You should now configure the 4 virtual network interfaces on the parent.
  • This includes renaming them to the names of the switches and configuring static IP addresses for the 3 internal NICs (the external NIC should be DHCP enabled, so it does not need IP address configuration).

3.3.PS. Using PowerShell

1..3 | % {Rename-NetAdapter *Internal$_* -NewName ParentInternal$_}
Rename-NetAdapter "vEthernet (External)" -NewName ParentExternal
Get-NetAdapter

3.3.OUT. Sample Output

PS C:\> Get-NetAdapter

Name               InterfaceDescription                    ifIndex Status  MacAddress          LinkSpeed
----               --------------------                    ------- ------  ----------          ---------
ParentInternal3    Hyper-V Virtual Ethernet Adapter #5          45 Up      00-15-5D-B5-AE-07     10 Gbps
ParentInternal2    Hyper-V Virtual Ethernet Adapter #4          36 Up      00-15-5D-B5-AE-06     10 Gbps
ParentInternal1    Hyper-V Virtual Ethernet Adapter #3          20 Up      00-15-5D-B5-AE-05     10 Gbps
ParentExternal     Hyper-V Virtual Ethernet Adapter #2          16 Up      00-21-9B-31-BA-15     10 Gbps
External           Intel(R) 82566DM-2 Gigabit Network C...      12 Up      00-21-9B-31-BA-15      1 Gbps

3.3.GUI. Using Server Manager

  • In Server Manager, click on “Local Server” on the list on left.
  • In the properties pane on the right, scroll to see the list of “Wired Internet Connections” (there will be 4 of them, as we showed in the previous Server Manager screenshot).
  • Click on the “Ipv4 address…” link next to one of the interfaces, the “Network Connections” window will show.
  • Select the interface that shows as “Enabled” and click on “Rename this connection” to rename it to “External”.
  • Rename the 3 interfaces on an “Unidentified network” to “Internal1”, “Internal2” and “Internal3”  
  • Close the “Network Connections” and refresh the “Local Server” view. 
     

3.4. Assign static IP addresses for the Hyper-V host

  • In this step, you will assign a static IP address to the 3 internal virtual NICs on the parent partition.
  • These NICs initially use the default setting (DHCP), but there is no DHCP server for the internal network.
  • The table below shows the desired configuration for each interface.

Machine

Parent
External

Parent
Internal1

Parent
Internal2

Parent
Internal3

Parent

DHCP

192.168.101.100

192.168.102.100

192.168.103.100

  • Note 1: The ParentExternal network does not need any further configuration, since the default is already to use DHCP.
  • Note 2: The preferred DNS IP address for all 3 internal interfaces should be set to 192.168.101.1 (that will be the IP address of the DNS server we will configure later).

3.4.PS. Using PowerShell

1..3 | % {
Set-NetIPInterface –InterfaceAlias ParentInternal$_ -DHCP Disabled
Remove-NetIPAddress –InterfaceAlias ParentInternal$_ -AddressFamily IPv4 -Confirm:$false
New-NetIPAddress –InterfaceAlias ParentInternal$_  -IPAddress "192.168.10$_.100" -PrefixLength 24 -Type Unicast
Set-DnsClientServerAddress –InterfaceAlias ParentInternal$_ -ServerAddresses 192.168.101.1
}

Get-NetIPAddress –AddressFamily Ipv4 | FT

3.4.OUT. Sample Output

PS C:\> Get-NetIPAddress –AddressFamily Ipv4 | Format-Table

ifIndex IPAddress           PrefixLength PrefixOrigin SuffixOrigin AddressState PolicyStore
------- ---------           ------------ ------------ ------------ ------------ -----------
45      192.168.103.100               24 Manual       Manual       Preferred    ActiveStore
36      192.168.102.100               24 Manual       Manual       Preferred    ActiveStore
20      192.168.101.100               24 Manual       Manual       Preferred    ActiveStore
16      10.123.181.174                23 Dhcp         Dhcp         Preferred    ActiveStore
1       127.0.0.1                      8 WellKnown    WellKnown    Preferred    ActiveStore
   

3.4.GUI. Using Server Manager

  • In Server Manager, click on “Local Server” on the list on left.
  • In the properties pane on the right, scroll to see the list of Ethernet interfaces (there will be 4 of them)
  • Click on the “Ipv4 address…” link next to one of the interfaces, the “Network Connections” window will show
  • On the list of network connections, right click the Internal1 interface and click “Properties”
  • On the “ParentInternal1 Properties” window, select “Internet Protocol Version 4 (TCP/IPv4)” and click “Properties”.
  • On TCP/IPv4 Properties window, select the option to “Use the following IP address”.
  • Enter the IP address 192.168.101.100 and the subnet mask 255.255.255.0 (as shown below).
  • Select the option “Use the following DNS server address”, enter 192.168.101.1 and click “OK”
  • Repeat this for the Internal2 and Internal3 networks, making sure to use the correct IP address (see table shown in item 3.4) and use the same Preferred DNS server IP address.
  • Close the “Network Connections” and refresh the “Local Server” view.

4. Create the Base VM

4.1. Preparations

  • Create a folder for your ISO files at C:\ISO and a folder for your VMs at C:\VMS
  • Copy the Windows Server 2012 ISO file to C:\ISO (I renamed the file to WindowsServer2012.ISO)

4.2. Create a Base VM

  • Create a new VM that will be used as the base image for our 4 VMs.
  • Store it in the C:\VMS folder and attach the Windows Server 2012 ISO file to it.

4.2.PS. Using PowerShell

MD C:\VMS
New-VHD -Path C:\VMS\BASE.VHDX -Dynamic -SizeBytes 127GB
New-VM -Name Base -VHDPath C:\VMS\BASE.VHDX -SwitchName External -Memory 1GB
Set-VMDvdDrive –VMName Base -Path C:\ISO\WindowsServer2012.ISO
Start-VM Base

4.2.OUT. Sample Output

PS C:\> Get-VM

Name State   CPUUsage(%) MemoryAssigned(M) Uptime   Status
---- -----   ----------- ----------------- ------   ------
Base Running 2           1024              00:01:49 Operating normally

4.2.GUI. Using Hyper-V Manager

  • In Windows Explorer, create a new C:\VMS folder.
  • In Server Manager, click on “Tools” in the upper left and select “Hyper-V Manager”.
  • In Hyper-V Manager, click on the server name on the pane on the left.
  • On the task pane on the right, click on “New”, then click on “Virtual Machine…”.
  • On the “Before you begin” page, just click “Next”.
  • On the “Specify Name and Location” page, use “Base” for the name, and “C:\VMS” for location. Click “Next”.  
  • On the “Assign Memory” page, use “1024” MB and click “Next”.
  • On the “Configure Networking” page, use “External”.
  • On the “Connect Virtual Disk” page, select the option to “Create a virtual hard disk”.
  • Use “Base.vhdx” for name, “C:\VMS” for location and “127” GB for size. Click “Next”. 
  • On the “Installation Options” page, select the option to install from DVD, select the option to use an ISO file and enter the path to the Windows Server 2012 ISO file on your C:\ISO folder. Click “Finish”.  
  • In Hyper-V Manager, right-click the VM and select “Start”

4.3. Install Windows Server 2012 on the VM

  • In Server Manager, click on “Tools” in the upper left and select “Hyper-V Manager”.
  • In Hyper-V Manager, click on the server name on the pane on the left.
  • On the list of VMs, right-click the VM called “Base” and click on “Connect…”
  • Follow the instructions on the screen, as you did in item 2.2.
  • Set a password, but don’t install any roles.
  • Don’t bother renaming the computer, since we’re sysprep’ing the VM anyway.

4.4. Sysprep the VM

  • After you have the OS installed on the VM, sign in and run C:\Windows\System32\Sysprep\Sysprep.exe
  • Select the options to run the OOBE, generalize and shutdown 
  • After Sysprep completes, the VM will be shut down.

4.5. Remove the base VM

  • Remove the BASE VM, leaving just the BASE.VHDX.
  • You should have a new base VHD ready to use at C:\VMS\BASE.VHD. Its size should be around 9GB.

4.5.PS. Using PowerShell

Remove-VM Base

4.5.GUI. Using Hyper-V Manager

  • In Hyper-V Manager, click on the server name on the pane on the left
  • On the list of VMs, right-click the VM called “Base” and click on “Delete”

5. Configure the 4 VMs

5.1. Create 4 new differencing VHDs using the Base VHD

5.1.PS. Using PowerShell

1..4 | % { New-VHD -ParentPath C:\VMS\BASE.VHDX –Path C:\VMS\VM$_.VHDX }

5.1.OUT. Sample Output

PS C:\> Dir C:\VMS

    Directory: C:\VMS

Mode                LastWriteTime     Length Name
----                -------------     ------ ----
-a---         8/17/2012  10:00 AM 9634316288 BASE.VHDX
-a---         8/17/2012  10:01 AM    4194304 VM1.VHDX
-a---         8/17/2012  10:01 AM    4194304 VM2.VHDX
-a---         8/17/2012  10:01 AM    4194304 VM3.VHDX
-a---         8/17/2012  10:01 AM    4194304 VM4.VHDX

5.1.GUI. Using Hyper-V Manager

  • In Hyper-V Manager, click on the server name on the pane on the left.
  • On the task pane on the right, click on “New”, then click on “Hard disk…”
  • On the “Before you begin” page, just click “Next”.
  • On the “Choose disk format” page, select “VHDX” and click “Next”.
  • On the “Choose disk type” page, select “Differencing”.  
  • On the “Specify Name and Location” page, use “VM1.VHDX” for name and “C:\VMS” for location. Click “Next”.
  • On the “Configure disk” page, use “C:\VMS\BASE.VHDX” for the location of the parent VHD.
  • After this, you will have a new differencing VHD at VM1.VHD that’s 4MB in size.
  • Since we’re creating 4 VMS, copy that file into VM2.VHD, VM3.VHD and VM4.VHD.  

5.2. Create 4 similarly configured VMs

  • You should create five VMs as follows:

VM

Role

Computer Name

External 

Internal 1

Internal 2

Internal 3

VM1

DNS, DC, iSCSI Target

DEMO-DC.DEMO.TEST

DHCP

192.168.101.1

N/A

N/A

VM2

File Server 1

DEMO-F1.DEMO.TEST

DHCP

192.168.101.3

192.168.102.3

192.168.103.3

VM3

File Server 2

DEMO-F2.DEMO.TEST

DHCP

192.168.101.4

192.168.102.4

192.168.103.4

VM4

SQL Server

DEMO-DB.DEMO.TEST

DHCP

192.168.101.5

192.168.102.5

192.168.103.5

  • Note 1: Each VM will use one of the VHD files we created in the previous step.
  • Note 2: Each VM will use 1GB of memory.

5.2.PS. Using PowerShell

1..4 | % { New-VM -Name VM$_ -VHDPath C:\VMS\VM$_.VHDX -Memory 1GB -SwitchName External}
1..4 | % { Add-VMNetworkAdapter VM$_ –SwitchName Internal1 }
2..4 | % { Add-VMNetworkAdapter VM$_ –SwitchName Internal2 }
2..4 | % { Add-VMNetworkAdapter VM$_ –SwitchName Internal3 }

5.2.OUT. Sample Output

PS C:\> Get-VM | % { $_ ; $_ | Get-VMNetworkAdapter | FT }

Name State CPUUsage(%) MemoryAssigned(M) Uptime   Status
---- ----- ----------- ----------------- ------   ------
VM1  Off   0           0                 00:00:00 Operating normally

Name            IsManagementOs VMName SwitchName MacAddress   Status IPAddresses
----            -------------- ------ ---------- ----------   ------ -----------
Network Adapter False          VM1    External   000000000000        {}
Network Adapter False          VM1    Internal1  000000000000        {}

Name State CPUUsage(%) MemoryAssigned(M) Uptime   Status
---- ----- ----------- ----------------- ------   ------
VM2  Off   0           0                 00:00:00 Operating normally

Name            IsManagementOs VMName SwitchName MacAddress   Status IPAddresses
----            -------------- ------ ---------- ----------   ------ -----------
Network Adapter False          VM2    External   000000000000        {}
Network Adapter False          VM2    Internal1  000000000000        {}
Network Adapter False          VM2    Internal2  000000000000        {}
Network Adapter False          VM2    Internal3  000000000000        {}

Name State CPUUsage(%) MemoryAssigned(M) Uptime   Status
---- ----- ----------- ----------------- ------   ------
VM3  Off   0           0                 00:00:00 Operating normally

Name            IsManagementOs VMName SwitchName MacAddress   Status IPAddresses
----            -------------- ------ ---------- ----------   ------ -----------
Network Adapter False          VM3    External   000000000000        {}
Network Adapter False          VM3    Internal1  000000000000        {}
Network Adapter False          VM3    Internal2  000000000000        {}
Network Adapter False          VM3    Internal3  000000000000        {}

Name State CPUUsage(%) MemoryAssigned(M) Uptime   Status
---- ----- ----------- ----------------- ------   ------
VM4  Off   0           0                 00:00:00 Operating normally

Name            IsManagementOs VMName SwitchName MacAddress   Status IPAddresses
----            -------------- ------ ---------- ----------   ------ -----------
Network Adapter False          VM4    External   000000000000        {}
Network Adapter False          VM4    Internal1  000000000000        {}
Network Adapter False          VM4    Internal2  000000000000        {}
Network Adapter False          VM4    Internal3  000000000000        {}

5.2.GUI. Using Hyper-V Manager

  • In Hyper-V Manager, click on the server name on the pane on the left.
  • On the task pane on the right, click on “New”, then click on “Virtual Machine…”
  • On the “Before you begin” page, just click “Next”.
  • On the “Specify Name and Location” page, use “VM1” for the name, and “C:\VMS” for location. Click “Next”.
  • On the “Assign Memory” page, use “1024” MB and click “Next”.
  • On the “Configure Networking” page, use “External”.
  • On the “Connect Virtual Disk” page, select the option to “Use an existing virtual hard disk”.
  • Use “C:\VMS\VM1.vhdx” for name. Click on “Finish”.
  • In Hyper-V Manager, click on the server name on the pane on the left.
  • On the list of VMs, right-click the VM you just created (VM1) and click on “Settings…”
  • On the “Settings for VM1” window, select “Add Hardware”, then “Network Adapter”.
  • Select the “Internal1” interface and click OK.
  • Repeat the process for VMs 2 to 4.
  • For VMs 2 to 4, make sure to add networks Internal2 and Internal3 as well Internal1.

5.3. Start the 4 VMs

5.3.PS. Using PowerShell

Start-VM VM*

5.3.GUI. Using Hyper-V Manager

  • In Hyper-V Manager, click on the server name on the pane on the left
  • In Hyper-V Manager, multi-select VMs 1 to 5, right click them and click on “Start”

5.4. Complete the mini-setup for the 4 VMs

  • Using Hyper-V manager, multi-select VMs 1 to 4, right click them and click on “Connect…”
  • Let the mini-setup complete, and configure each of the five VMs.
  • You will be prompted for the usual items, like license agreement, clock/language/region settings and a password.

5.5. Change the computer name for each VM

  • Change the computer name for each VM, using the names defined in item 5.2
  • The examples below are for VM1 (the DNS and Domain Controller).
  • You should repeat this for each of the 4 VMs.
  • At this point, you can also use this opportunity to enable Remote Desktop for each VM.
  • This would be done for each VM as you did for the host in item 2.4, if you want to access the VMs remotely.

5.5.PS. Using PowerShell (for VM1, for instance)

Rename-Computer DEMO-DC -Restart

5.5.GUI. Using Server Manager (for VM1, for instance)

  • In Server Manager, click on “Local Server” on the list on left.
  • Click on the default name next to “Computer Name”.
  • Click on “Change”.
  • Enter the new computer name as “DEMO-DC”.
  • Click “OK” accept the changes.
  • Click “OK” to acknowledge that you need to restart to apply changes.
  • Click “Restart Now”.

5.6. For each VM, configure the networks

  • In this step you will configure the network for each VM as shown on the table in item 5.2
  • We first rename the Network Connections in each guest for easy identification.
  • The External network is identified as being the only one with a DHCP address.
  • The remaining networks are renamed to Internal1, Internal2 and Internal3.
  • For internal networks static IPs are configured, with mask 255.255.255.0 and DNS set to 192.168.101.1.
  • The Internal 1 network will be used for DNS, Active Directory and the iSCSI Target.
  • The External network is useful for downloading from the Internet and remotely connecting to the 4 VMs.
  • You could configure a DHCP server for the internal interfaces.
  • However, due to the risk of accidentally creating a rogue DHCP server in your network, fixed IPs are used.

5.6.PS. Using PowerShell (for VM1, for instance)

## External NIC is the only one with a DHCP server
 
Get-NetIPAddress -PrefixOrigin DHCP | % { 
     Rename-NetAdapter -InterfaceAlias $_.InterfaceAlias –NewName External
}
 
## $IC – Internal Count – Number of Internal networks
 
$IC=0
Get-NetAdapter Ethernet* | Sort MacAddress | % {
   $IC++
   Rename-NetAdapter -InterfaceAlias $_.InterfaceAlias –NewName Internal$IC
}
 
## $VM is the VM Number, between 1 and 4. Used as the last portion of the IP address.
 
$VM=1
1..$IC | % {
   Set-NetIPInterface –InterfaceAlias Internal$_ -DHCP Disabled
   Remove-NetIPAddress –InterfaceAlias Internal$_ -AddressFamily IPv4 –Confirm:$false
   New-NetIPAddress –InterfaceAlias Internal$_ -IPAddress "192.168.10$_.$VM" -PrefixLength 24 -Type Unicast 
   Set-DnsClientServerAddress –InterfaceAlias Internal$_ -ServerAddresses 192.168.101.1
}

5.6.GUI. Using Server Manager (For VM1, for instance)

  • This step is similar to step 3.4, but this time performed on the 4 VM.
  • Inside the VM, in Server Manager, click on “Local Server” on the list on left.
  • In the properties pane on the right, Click on the “Ipv4 address…” link next to one of the interfaces.
  • The “Network Connections” window will show.
  • As you did with the Hyper-V host, rename the NIC with the DHCP connection to “External”.
    This NIC needs no further configuration.
  • Rename the remaining NIC to “Internal1”. (Other VMs will also have Internal2 and Internal3).
  • For each internal NIC, right click the Internal1 interface and click “Properties”.
  • On the “Internal1 Properties” window, select “Internet Protocol Version 4 (TCP/IPv4)” and click “Properties”.
  • On TCP/IPv4 Properties window, select the option to “Use the following IP address”.
  • Enter the corresponding IP address (see table on item 5.2) and the subnet mask 255.255.255.0.
  • Select the option “Use the following DNS server address”, enter 192.168.101.1 and click “OK”.
  • Repeat this for the Internal2 and Internal3 networks using the corresponding IP address and the DNS above.
  • Close the “Network Connections” and refresh the “Local Server” view.

 

  • Note: If you can’t tell which Internal network is which inside the VMs with multiple Internal networks, you can temporarily set one of the adapters to “Not Connected” in the VM Settings and verify which one shows as “Network cable unplugged”.

5.7. Review VM name and network configuration

  • After renaming the computer, renaming the network and configuring IP addresses, review the configuration on each VM to make sure you did not miss any step. Examples are shown below for VM1 and VM2.

5.7.PS. Using PowerShell

Get-WmiObject Win32_ComputerSystem
Get-NetAdapter
Get-NetIPAddress -AddressFamily IPv4 | Sort IfIndex | FT

5.7.OUT. Sample Output (for VM2, a.k.a. DEMO-F1)

PS C:\> Get-WmiObject Win32_ComputerSystem

Domain              : WORKGROUP
Manufacturer        : Microsoft Corporation
Model               : Virtual Machine
Name                : DEMO-F1
PrimaryOwnerName    : Windows User
TotalPhysicalMemory : 1072799744

PS C:\> Get-NetAdapter

Name         InterfaceDescription                    ifIndex Status  MacAddress          LinkSpeed
----         --------------------                    ------- ------  ----------          ---------
External     Microsoft Hyper-V Network Adapter #4         15 Up      00-15-5D-B5-AE-12     10 Gbps
Internal3    Microsoft Hyper-V Network Adapter #3         14 Up      00-15-5D-B5-AE-15     10 Gbps
Internal1    Microsoft Hyper-V Network Adapter            12 Up      00-15-5D-B5-AE-13     10 Gbps
Internal2    Microsoft Hyper-V Network Adapter #2         13 Up      00-15-5D-B5-AE-14     10 Gbps

PS C:\> Get-NetIPAddress -AddressFamily IPv4 | Sort IfIndex | FT

ifIndex IPAddress         PrefixLength PrefixOrigin SuffixOrigin AddressState PolicyStore
------- ---------         ------------ ------------ ------------ ------------ -----------
1       127.0.0.1                    8 WellKnown    WellKnown    Preferred    ActiveStore
12      192.168.101.2               24 Manual       Manual       Preferred    ActiveStore
13      192.168.102.2               24 Manual       Manual       Preferred    ActiveStore
14      192.168.103.2               24 Manual       Manual       Preferred    ActiveStore
15      10.123.181.211              23 Dhcp         Dhcp         Preferred    ActiveStore

5.7.GUI. Using Server Manager

  • In Server Manager, click on “Local Server” on the list on left.
  • Verify the network configuration.

6. Configure DNS and Active Directory

6.1. Install DNS and Active Directory Domain Services

  • Install the required DNS and Active Directory Domain Services roles to VM1 (DEMO-DC)

6.1.PS. Using PowerShell

Install-WindowsFeature DNS, AD-Domain-Services, RSAT-AD-PowerShell, RSAT-ADDS-Tools

6.1.GUI. Using Server Manager

  • In Server Manager, click on “Dashboard” on the list on left.
  • Click on the “Add Roles and Features”, which is option 2 under “Configure this local server”.
  • On the “Before You Begin” page, just click “Next”.
  • On the “Installation Type” page, click “Role-base or feature-based installation” and click “Next”.
  • On the “Server Selection” page, select your server and click “Next”.
  • On the “Server Role” page, select “Active Directory Domain Services”.
  • On the dialog about adding required services, click “Add Features”.
  • On the “Server Role” page, select “DNS Server” and click “Next”.
  • On the dialog about adding required services, click “Add Features”.   
  • On the “Feature” page, just click “Next”.
  • On the “Active Directory Domain Services” page, just click “Next”.
  • On the “DNS Server” page, just click “Next”.
  • On the “Confirmation” page, click “Install”.
  • The roles will be installed.

6.2. Configure Active Directory

  • Create a new domain and forest for the DEMO.TEST domain.

6.2.PS. Using PowerShell

Import-Module ADDSDeployment
Install-ADDSForest `
-CreateDNSDelegation:$false `
-DatabasePath "C:\Windows\NTDS" `
-DomainMode "Win2008R2" `
-DomainName "DEMO.TEST" `
-DomainNetBIOSName "DEMO" `
-ForestMode "Win2008R2" `
-InstallDNS:$true `
-LogPath "C:\Windows\NTDS" `
-SafeModeAdministratorPassword (Read-Host -AsSecureString -Prompt "Enter Password") `
-SYSVOLPath "C:\Windows\SYSVOL"

6.2.GUI. Using Server Manager

  • Open Server Manager and click on the “AD DS” option on the right.
  • On the yellow band showing “Configuration Required for Active Directory...” click “More…”
  • On the “All Server Task Details”, click on the action to “Promote this server…”
  • The “Active Directory Domain Services Configuration Wizard” will start.
  • On the “Deployment Configuration” page, select “Add a new forest”.
  • Enter “DEMO.TEST” as the root domain name and click “Next”.
  • On the “Domain Controller Option”, enter the password twice and click “Next”.
  • On the “DNS Options” page, click “Next”.
  • On the “Additional Options” page, click “Next”. (NETBIOS name check takes a while)
  • On the “Paths” page, click “Next”.
  • On the “Review Options” page, click “Next”.
  • On the “Pre-requisites” page, click “Next”. (Pre-requisite checks takes a while)
  • Click “Install”.

6.3. Join the other VMs to the domain

  • After the Domain Controller reboots, for every one of the other 3 VMs, join the domain
  • You will need to provide the domain name (DEMO.TEST) and the Administrator credentials
  • From now on, always log on to any of the VMs using the domain credentials: DEMO.TEST\Administrator

6.3.PS. Using PowerShell (for VM2 to VM4)

Add-Computer -DomainName DEMO.TEST -Restart

6.4. Create the SQL Service account

  • In the Domain Controller, use Active Directory Users and Computers to create a new user account for SQL.
  • The account should be called SQLService and should not require change in the next logon.
  • Set a password for the SQLService account.

6.4.PS. Using PowerShell

New-ADUser -Name SQLService –Enabled $True -UserPrincipalName SQLService@DEMO.TEST `
-DisplayName SQLService -ChangePasswordAtLogon $False -PasswordNeverExpires $True `
-AccountPassword (Read-Host -AsSecureString "Enter password")

6.4.OUT. Sample Output

PS C:\> Get-ADUser -Filter {Name -like "SQL*"}

DistinguishedName : CN=SQLService,CN=Users,DC=DEMO,DC=TEST
Enabled           : True
GivenName         :
Name              : SQLService
ObjectClass       : user
ObjectGUID        : 7a02941d-10c7-45f8-b986-1b67a08ddd06
SamAccountName    : SQLService
SID               : S-1-5-21-3876617879-1076079722-1647216889-1107
Surname           :
UserPrincipalName :
SQLService@DEMO.TEST

6.4.GUI. Using Server Manager

  • Open Server Manager
  • In the Tools menu on the upper right, select “Active Directory Users and Computers”
  • Right click the “Users” container on the tree on the left, then select “New”, then “User”
  • Enter “SQLService” as Full Name and User Logon Name, then click “Next”
  • Enter the password twice as required
  • Uncheck “user must change password at next logon” and check “Password never expires”
  • Click “Next”, then click “Finish”

7. Configure iSCSI

  • We’ll create a single Target with 3 Devices (LUNs or VHD files) and used by 2 initiators (DEMO-F1 and DEMO-F2).
  • The devices will include a 1GB VHD for the Cluster Witness volume and two 20GB VHDs for the data volumes.
  • We’ll then configure the initiators and volumes from the File Server side.

7.1. Add the iSCSI Software Target

  • Add the iSCSI Software Target role service to VM2 (DEMO-IT.DEMO.TEST)

7.1.PS. Using PowerShell

Install-WindowsFeature FS-iSCSITarget-Server

7.1.GUI. Using Server Manager

  • In Server Manager, click on “Dashboard” on the list on left
  • Click on the “Add Roles and Features”, which is option 2 under “Configure this local server”
  • On the “Before You Begin” page, just click “Next”
  • On the “Installation Type” page, click “Role-base or feature-based installation” and click “Next”
  • On the “Server Selection” page, expand “File and Storage Services”, then “File Services”
  • Select the “iSCSI Target Server”
  • On the dialog about adding required services, click “Add Features”   
  • Click “Next”
  • On the “Feature” page, just click “Next”
  • On the “Confirmation” page, click “Install”
  • The role will be installed

7.2. Create the LUNs and Target

  • Create the 1st LUN with the file at C:\LUN0.VHD, 1GB in size, description “LUN0”.
  • Create the 2nd and 3rd LUNs at C:\LUN1.VHD and C:\LUN2.VHD, both with 20GB.
  • Add those to a single target, exposed to two initiators by IP address (192.168.101.3 and 192.168.101.4)

7.2.PS. Using PowerShell

New-IscsiServerTarget -TargetName FileCluster -InitiatorID IPAddress:192.168.101.2, IPAddress:192.168.101.3
New-IscsiVirtualDisk -DevicePath C:\LUN0.VHD -Size 1GB
1..2 | % {New-IscsiVirtualDisk -DevicePath C:\LUN$_.VHD -Size 20GB}
0..2 | % {Add-iSCSIVirtualDiskTargetMapping -TargetName FileCluster -DevicePath C:\LUN$_.VHD}

7.2.OUT. Sample output

PS C:\> Get-IscsiServerTarget 

ChapUserName                :
ClusterGroupName            :
ComputerName                : DEMO-DC.DEMO.TEST
Description                 :
EnableChap                  : False
EnableReverseChap           : False
EnforceIdleTimeoutDetection : True
FirstBurstLength            : 65536
IdleDuration                : 00:00:21
InitiatorIds                : {IPAddress:192.168.101.2, IPAddress:192.168.101.3}
LastLogin                   :
LunMappings                 : {TargetName:FileCluster;VHD:"C:\LUN0.VHD";LUN:0,
                              TargetName:FileCluster;VHD:"C:\LUN1.VHD";LUN:1,
                              TargetName:FileCluster;VHD:"C:\LUN2.VHD";LUN:2}
MaxBurstLength              : 262144
MaxReceiveDataSegmentLength : 65536
ReceiveBufferCount          : 10
ReverseChapUserName         :
Sessions                    : {}
Status                      : NotConnected
TargetIqn                   : iqn.1991-05.com.microsoft:demo-dc-filecluster-target
TargetName                  : FileCluster 

PS C:\> Get-IscsiVirtualDisk 

ClusterGroupName   :
ComputerName       : DEMO-DC.DEMO.TEST
Description        :
DiskType           : Fixed
HostVolumeId       : {C4A5E065-E88F-11E1-93EB-806E6F6E6963}
LocalMountDeviceId :
OriginalPath       :
ParentPath         :
Path               : C:\LUN0.VHD
SerialNumber       : 3FDD6603-2F45-4E95-8C0F-0B6A574DA84A
Size               : 1073741824
SnapshotIds        :
Status             : NotConnected
VirtualDiskIndex   : 119718233

ClusterGroupName   :
ComputerName       : DEMO-DC.DEMO.TEST
Description        :
DiskType           : Fixed
HostVolumeId       : {C4A5E065-E88F-11E1-93EB-806E6F6E6963}
LocalMountDeviceId :
OriginalPath       :
ParentPath         :
Path               : C:\LUN2.VHD
SerialNumber       : 981545EA-32FF-4BA4-856D-C6F464FEC82F
Size               : 21474836480
SnapshotIds        :
Status             : NotConnected
VirtualDiskIndex   : 1469988013

ClusterGroupName   :
ComputerName       : DEMO-DC.DEMO.TEST
Description        :
DiskType           : Fixed
HostVolumeId       : {C4A5E065-E88F-11E1-93EB-806E6F6E6963}
LocalMountDeviceId :
OriginalPath       :
ParentPath         :
Path               : C:\LUN1.VHD
SerialNumber       : BBCB273F-74EF-4E50-AA07-EDCD2E955A3B
Size               : 21474836480
SnapshotIds        :
Status             : NotConnected
VirtualDiskIndex   : 1581769191

7.2.GUI. Using Server Manager

  • In Server Manager, click on “File and Storage Services” on the list on left
  • Click on “iSCSI Virtual Disks”
  • On the “Tasks” menu on the right, select “New Virtual Disk…”
  • The “New iSCSI Virtual Disk Wizard” will start
  • On the “Virtual Disk Location” page, with the DEMO-IT server and “C:” volume selected, click “Next”
  • On the “Virtual Disk Name” page, enter “LUN0” as the Name, then click “Next”
  • On the “Virtual Disk Size” page, enter 1GB as the size, then click “Next”
  • On the “iSCSI Target” page, with the “New iSCSI target” option selected, click “Next”
  • On the “iSCSI Target Name” page, enter “FileCluster” as the name, then click “Next”
  • On the “Access Servers” page, click on “Add…”
  • Select “Enter a value...”, select “IP Address”, enter “192.168.101.3”, then click “OK”
  • On the “Access Servers” page, click on “Add…” again
  • Select “Enter a value...”, select “IP Address”, enter “192.168.101.4”, then click “OK”
  • With the two iSCSI Initiators specified, click “Next”
  • On the “Enable Authentication” page, click “Next”
  • On the “Confirmation” page, click “Create”
  • When the wizard is done, click “Close”.
  • On the “Tasks” menu on the right, select “New Virtual Disk…”
  • The “New iSCSI Virtual Disk Wizard” will start
  • On the “Virtual Disk Location” page, with the DEMO-IT server and “C:” volume selected, click “Next”
  • On the “Virtual Disk Name” page, enter “LUN1” as the Name, then click “Next”
  • On the “Virtual Disk Size” page, enter 20GB as the size, then click “Next”
  • On the “iSCSI Target” page, with the “Select Existing iSCSI target” option selected, click “Next”
  • On the “Confirmation” page, click “Create”
  • When the wizard is done, click “Close”.
  • Repeat the steps above to create a LUN2 with 20GB and add to the same target.

7.3. Configure the iSCSI Initiators

  • Now we shift to the two File Servers, which will run the iSCSI Initiator.
  • We’ll do this on VM2 and VM3 (or DEMO-F1 and DEMO-F2).
  • Make sure to log on using the domain administrator (DEMO\Administrator), not the local Administrator.
  • You will then start the iSCSI Initiator, configuring the service to start automatically.
  • You will then connect the initiator to the iSCSI Target we just configured on DEMO-IT

7.3.PS. Using PowerShell

Set-Service MSiSCSI -StartupType automatic
Start-Service MSiSCSI
New-iSCSITargetPortal -TargetPortalAddress 192.168.101.1
Get-iSCSITarget | Connect-iSCSITarget
Get-iSCSISession | Register-iSCSISession

7.3.OUT. Sample output

PS C:\> Get-IscsiTargetPortal 

InitiatorInstanceName  :
InitiatorPortalAddress :
IsDataDigest           : False
IsHeaderDigest         : False
TargetPortalAddress    : 192.168.101.1
TargetPortalPortNumber : 3260
PSComputerName         : 

PS C:\> Get-IscsiTarget | Format-List 

IsConnected    : True
NodeAddress    : iqn.1991-05.com.microsoft:demo-dc-filecluster-target
PSComputerName : 

PS C:\> Get-IscsiConnection 

ConnectionIdentifier : fffffa8002d12020-3
InitiatorAddress     : 0.0.0.0
InitiatorPortNumber  : 37119
TargetAddress        : 192.168.101.1
TargetPortNumber     : 3260
PSComputerName       :

PS C:\> Get-Disk

Number Friendly Name                            OperationalStatus      Total Size Partition Style
------ -------------                            -----------------      ---------- ---------------
0      Virtual HD ATA Device                    Online                     127 GB MBR
1      MSFT Virtual HD SCSI Disk Device         Offline                      1 GB RAW
2      MSFT Virtual HD SCSI Disk Device         Offline                     20 GB RAW
3      MSFT Virtual HD SCSI Disk Device         Offline                     20 GB RAW

7.3.GUI. Using Server Manager

  • Open Server Manager
  • In the Tools menu on the upper right, select “iSCSI Initator”
  • Click on “Yes” on the prompt about automatically starting the iSCSI Initiator.
  • Enter “192.168.101.2” on the Target field and click the “Quick Connect…” button next to it.
  • Verify the status shows as “Connect” and click on “Done”
  • Click on the “Volume and Devices” tab and click on the “Auto Configure” button.
  • Verify that three volumes show up on the Volume List.
  • Click “OK” to close the iSCSI Initiator.

7.4. Configure the disks

  • Execute this task only on the first of the two file server (VM2, a.k.a. DEMO-F1).
  • This will configure the three disks exposed by the iSCSI Target (the iSCSI LUNs).
  • They first need to be onlined, initialized and partitioned (we’re using MBR partitions, since the disks are small).
  • Then you will format them and assign each one a driver letter (W:, X: and Y:).
  • Drive W: will be the used as witness disks, while X: and Y: will be data disks for the file server cluster.

7.4.PS. Using PowerShell

1..3 | % {
$d = “-WXY”[$_]
Set-Disk -Number $_ -IsReadOnly 0
Set-Disk -Number $_ -IsOffline 0
Initialize-Disk -Number $_ -PartitionStyle MBR
New-Partition -DiskNumber $_ -DriveLetter $d –UseMaximumSize
Initialize-Volume -DriveLetter $d -FileSystem NTFS -Confirm:$false
}

7.4.OUT. Sample output

PS C:\> Get-Disk

Number Friendly Name                     OperationalStatus  Total Size Partition Style
------ -------------                     -----------------  ---------- ---------------
0      Virtual HD ATA Device             Online                 127 GB MBR
1      MSFT Virtual HD SCSI Disk Device  Online                   1 GB MBR
2      MSFT Virtual HD SCSI Disk Device  Online                  20 GB MBR
3      MSFT Virtual HD SCSI Disk Device  Online                  20 GB MBR


PS C:\> Get-Volume | Sort DriveLetter

DriveLetter   FileSystemLabel  FileSystem   DriveType   HealthStatus   SizeRemaining        Size
-----------   ---------------  ----------   ---------   ------------   -------------        ---- 
              System Reserved  NTFS         Fixed       Healthy             108.7 MB      350 MB
A                                           Removable   Healthy                  0 B         0 B
C                              NTFS         Fixed       Healthy            118.29 GB   126.66 GB
D                                           CD-ROM      Healthy                  0 B         0 B
W                              NTFS         Fixed       Healthy            981.06 MB  1022.93 MB
X                              NTFS         Fixed       Healthy              19.9 GB       20 GB
Y                              NTFS         Fixed       Healthy              19.9 GB       20 GB

7.4.GUI. Using Server Manager

  • Open the Disk Management tool
  • Online all three offline disks (the iSCSI LUNs)
  • Initialize them (you can use MBR partitions, since they are small)
  • Create a new Simple Volume on each one using all the disk space on the LUN
  • Quick-format them with NTFS as the file system
  • Assign each one a drive letter (W:, X: and Y:)

8. Configure the File Server

8.1 Install the required roles and features

  • Now we need to configure VM2 and VM3 as file servers and cluster nodes

8.1.PS. Using PowerShell (from both VM2 and VM3)

Install-WindowsFeature File-Services, FS-FileServer, Failover-Clustering
Install-WindowsFeature RSAT-Clustering -IncludeAllSubFeature
Install-WindowsFeature RSAT-File-Services -IncludeAllSubFeature

8.1.OUT. Sample output

PS C:\> Get-WindowsFeature *File*, *Cluster*

Display Name                                            Name                       Install State
------------                                            ----                       -------------
[X] File And Storage Services                           FileAndStorage-Services        Installed
    [X] File and iSCSI Services                         File-Services                  Installed
        [X] File Server                                 FS-FileServer                  Installed
[X] Failover Clustering                                 Failover-Clustering            Installed
        [X] Failover Clustering Tools                   RSAT-Clustering                Installed
            [X] Failover Cluster Management Tools       RSAT-Clustering-Mgmt           Installed
            [X] Failover Cluster Module for Windows ... RSAT-Clustering-Powe...        Installed
            [X] Failover Cluster Automation Server      RSAT-Clustering-Auto...        Installed
            [X] Failover Cluster Command Interface      RSAT-Clustering-CmdI...        Installed
        [X] File Services Tools                         RSAT-File-Services             Installed
            [X] Share and Storage Management Tool       RSAT-CoreFile-Mgmt             Installed

8.1.GUI. Using Server Manager

  • For both DEMO-F1 and DEMO-F2, from Server Manager, select Add Role and check File and Storage Services.
  • Next, select Add Feature and check Failover Clustering

8.2. Validate the Failover Cluster Configuration

8.2.PS. Using PowerShell (From VM2, DEMO-F1)

Test-Cluster -Node DEMO-F1, DEMO-F2

8.2.OUT. Sample output

image

8.2.GUI. Using Server Manager

  • On VM2 (DEMO-F1), open Server Manager.
  • On the Tools menu on the upper right, select “Failover Cluster Manager”
  • In Failover Cluster Manager and click on the option to “Validate a Configuration…”
  • The “Validate a Configuration Wizard” will start. Click “Next”
  • Add the two file servers: DEMO-F1 and DEMO-F2. Then click “Next”
  • Select the option to “Run all tests”. Click “Next”. Click “Next” again to confirm.
  • Let the validation process run. It will take a few minutes to complete.
  • Validation should not return any errors.
  • If it does, review the previous steps and make sure to address any issues listed in the validation report.

8.3. Create a Failover Cluster

8.3.PS. Using PowerShell (From VM2, DEMO-F1)

New-Cluster –Name DEMO-FC -Node DEMO-F1, DEMO-F2

8.3.GUI. Using Server Manager

  • On VM2 (DEMO-F1), open Server Manager.
  • On the Tools menu on the upper right, select “Failover Cluster Manager”
  • In Failover Cluster Manager and click on the option to “Create a Cluster…”
  • The “Create a Cluster Wizard” will start. Click “Next”
  • Add the two file servers: DEMO-F1 and DEMO-F2. Then click “Next”
  • Enter the Cluster Name: DEMO-FC. Then click “Next”
  • Click “Next” again to confirm.
  • Click “Finish” after the cluster is created.

image

8.4. Configure the Cluster Networks

  • For consistency, you should rename the Cluster networks to match the names used previously.
  • You should also configure the Internal networks to be used by cluster, but not the External one.

8.4.PS. Using PowerShell (From VM2, DEMO-F1)

(Get-ClusterNetwork | ? Address -like 192.168.101.* ).Name = "Internal1"
(Get-ClusterNetwork | ? Address -like 192.168.102.* ).Name = "Internal2”
(Get-ClusterNetwork | ? Address -like 192.168.103.* ).Name = "Internal3”
(Get-ClusterNetwork | ? Name -notlike Internal* ).Name = "External"
(Get-ClusterNetwork Internal1).Role = 3
(Get-ClusterNetwork Internal2).Role = 3
(Get-ClusterNetwork Internal3).Role = 3
(Get-ClusterNetwork External).Role = 1

8.4.OUT. Sample Output

PS C:\> Get-ClusterNetwork | Select *


Cluster           : DEMO-FC
State             : Up
Name              : External
Ipv6Addresses     : {2001:4898:2a:3::, 2001:4898:0:fff:0:5efe:10.123.180.0}
Ipv6PrefixLengths : {64, 119}
Ipv4Addresses     : {10.123.180.0}
Ipv4PrefixLengths : {23}
Address           : 10.123.180.0
AddressMask       : 255.255.254.0
Description       :
Role              : 1
AutoMetric        : True
Metric            : 30240
Id                : 14cab1e8-c16c-46fa-bf01-afc808d29368

Cluster           : DEMO-FC
State             : Up
Name              : Internal1
Ipv6Addresses     : {}
Ipv6PrefixLengths : {}
Ipv4Addresses     : {192.168.101.0}
Ipv4PrefixLengths : {24}
Address           : 192.168.101.0
AddressMask       : 255.255.255.0
Description       :
Role              : 3
AutoMetric        : True
Metric            : 70242
Id                : 16603183-5639-44a0-8e5e-3934280866cd

Cluster           : DEMO-FC
State             : Up
Name              : Internal2
Ipv6Addresses     : {}
Ipv6PrefixLengths : {}
Ipv4Addresses     : {192.168.102.0}
Ipv4PrefixLengths : {24}
Address           : 192.168.102.0
AddressMask       : 255.255.255.0
Description       :
Role              : 3
AutoMetric        : True
Metric            : 70241
Id                : 528c89bc-8704-4d1a-aa80-65bd5c25e3e5

Cluster           : DEMO-FC
State             : Up
Name              : Internal3
Ipv6Addresses     : {}
Ipv6PrefixLengths : {}
Ipv4Addresses     : {192.168.103.0}
Ipv4PrefixLengths : {24}
Address           : 192.168.103.0
AddressMask       : 255.255.255.0
Description       :
Role              : 3
AutoMetric        : True
Metric            : 70243
Id                : 0f59076d-5536-4d69-af43-271cc4f76723

8.4.GUI. Using Server Manager

  • In Failover Cluster Manager, expand the nodes until you find the “Networks” node.
  • For each network, right-click the name and click “Properties”.
  • Enter the name Internal1, Internal2, Internal3 or External, according to their IP addresses.
  • For the External network, make sure “Allow cluster…” is selected and “Allow clients…” is *not* checked.
  • For all Internal networks, select “Allow cluster…” and check the “Allow clients…” checkbox.

image

8.5. Add data disks to Cluster Shared Volumes (CSV)

  • Add the disks to the list of Cluster Shared Volumes.

8.5.PS. Using PowerShell (From VM2, DEMO-F1)

Get-ClusterResource | ? OwnerGroup -like Available* | Add-ClusterSharedVolume

8.5.GUI. Sample Output

PS C:\> Get-ClusterResource

Name                          State    OwnerGroup      ResourceType
----                          -----    ----------      ------------
Cluster Disk 3                Online   Cluster Group   Physical Disk
Cluster IP Address            Online   Cluster Group   IP Address
Cluster IP Address 2001:48... Online   Cluster Group   IPv6 Address
Cluster Name                  Online   Cluster Group   Network Name


PS C:\> Get-ClusterSharedVolume

Name              State      Node
----              -----      ----
Cluster Disk 1    Online     DEMO-F1
Cluster Disk 2    Online     DEMO-F2

8.5.GUI. Using Server Manager

  • In Failover Cluster Manager, expand the nodes until you find the “Storage” node.
  • Select the two disks currently assigned to “Available Storage”.
  • Right click the two selected disks and click on “Add to Cluster Shared Volumes”

image

8.6. Create the Scale-Out File Server

  • Create a Scale-Out File Server.

8.6.PS. Using PowerShell (From VM2, DEMO-F1)

Add-ClusterScaleOutFileServerRole -Name DEMO-FS

8.6.OUT. Sample Output

PS C:\> Get-ClusterGroup DEMO-FS

Name        OwnerNode    State
----        ---------    -----
DEMO-FS     DEMO-F1      Online


PS C:\> Get-ClusterGroup DEMO-FS | Get-ClusterResource

Name                          State     OwnerGroup     ResourceType
----                          -----     ----------     ------------
DEMO-FS                       Online    DEMO-FS        Distributed Network Name
Scale-Out File Server (\\D... Online    DEMO-FS        Scale Out File Server

8.6.GUI. Using Server Manager

  • On the Failover Cluster Manager, select the main node on the tree (with the cluster name)
  • On the actions menu on the right, select “Configure Role…”
  • The “High Availability Wizard” will start. Click “Next”
  • On the “Select Role” page, select “File Server” and click “Next”
  • On the “File Server Type” page, select “File Server for scale-out application data” and click “Next”
  • On the “Client Access Point” page, specify the name of the service as DEMO-FS
  • On the “Confirmation” page, click “Next”.
  • Click “Finish” after the configuration is completed.

image

8.7. Create the folders and shares

  • In this step, you will create two shares: one for database files and one for log files

8.7.PS. Using PowerShell (From VM2, DEMO-F1)

MD C:\ClusterStorage\Volume1\DATA
New-SmbShare -Name DATA -Path C:\ClusterStorage\Volume1\DATA -FullAccess DEMO.TEST\Administrator, DEMO.TEST\SQLService
(Get-SmbShare DATA).PresetPathAcl | Set-Acl

MD C:\ClusterStorage\Volume2\LOG
New-SmbShare -Name LOG -Path C:\ClusterStorage\Volume2\LOG -FullAccess DEMO.TEST\Administrator, DEMO.TEST\SQLService
(Get-SmbShare LOG).PresetPathAcl | Set-Acl

8.7.OUT. Sample Output

PS C:\> Get-SmbShare Data, Log

Name    ScopeName    Path                          Description
----    ---------    ----                          -----------
DATA    DEMO-FS      C:\ClusterStorage\Volume1\...
LOG     DEMO-FS      C:\ClusterStorage\Volume2\LOG

PS C:\> Get-SmbShare Data, Log | Select *

PresetPathAcl         : System.Security.AccessControl.DirectorySecurity
ShareState            : Online
AvailabilityType      : ScaleOut
ShareType             : FileSystemDirectory
FolderEnumerationMode : Unrestricted
CachingMode           : Manual
CATimeout             : 0
ConcurrentUserLimit   : 0
ContinuouslyAvailable : True
CurrentUsers          : 0
Description           :
EncryptData           : False
Name                  : DATA
Path                  : C:\ClusterStorage\Volume1\DATA
Scoped                : True
ScopeName             : DEMO-FS
SecurityDescriptor    : O:BAG:DUD:(A;;FA;;;S-1-5-21-3876617879-1076079722-1647216889-500)(A;;FA;;;S-1-5-21-3876617879-1
                        076079722-1647216889-1107)
ShadowCopy            : False
Special               : False
Temporary             : False
Volume                : \\?\Volume{4789973e-1f33-4d27-9bf1-2e9ec6da13a0}\
PSComputerName        :
CimClass              : ROOT/Microsoft/Windows/SMB:MSFT_SmbShare
CimInstanceProperties : {AvailabilityType, CachingMode, CATimeout, ConcurrentUserLimit...}
CimSystemProperties   : Microsoft.Management.Infrastructure.CimSystemProperties

PresetPathAcl         : System.Security.AccessControl.DirectorySecurity
ShareState            : Online
AvailabilityType      : ScaleOut
ShareType             : FileSystemDirectory
FolderEnumerationMode : Unrestricted
CachingMode           : Manual
CATimeout             : 0
ConcurrentUserLimit   : 0
ContinuouslyAvailable : True
CurrentUsers          : 0
Description           :
EncryptData           : False
Name                  : LOG
Path                  : C:\ClusterStorage\Volume2\LOG
Scoped                : True
ScopeName             : DEMO-FS
SecurityDescriptor    : O:BAG:DUD:(A;;FA;;;S-1-5-21-3876617879-1076079722-1647216889-500)(A;;FA;;;S-1-5-21-3876617879-1
                        076079722-1647216889-1107)
ShadowCopy            : False
Special               : False
Temporary             : False
Volume                : \\?\Volume{888f5e8c-c91c-4bcf-b4b2-cc4e427ee54c}\
PSComputerName        :
CimClass              : ROOT/Microsoft/Windows/SMB:MSFT_SmbShare
CimInstanceProperties : {AvailabilityType, CachingMode, CATimeout, ConcurrentUserLimit...}
CimSystemProperties   : Microsoft.Management.Infrastructure.CimSystemProperties

8.7.GUI. Using Server Manager

  • On the Failover Cluster Manager, select the Roles node on the tree on the left.
  • Click on the DEMO-FS role and then click on “Add Shared Folder” on the actions menu on the right.
  • The “New Share Wizard” will start.
  • On the “Select Profile” page, select “SMB Share – Server Application” and click “Next”
  • On the “Share Location” page, select “C:\ClusterStorage\Volume1” as the location. Click “Next”.
  • On the “Share Name” page, enter “Data” as the share name, click “Next”.
  • On the “Other Settings” page, just click “Next”
  • On the “Permissions” page, click on “Customize permissions…”
  • Click on “Add”, then click on “Select a Principal”.
  • Enter “DEMO\Administrator”, click on “Check Names” and then click “OK”.
  • Click “Full Control” and click “OK”.
  • Click on “Add”, then click on “Select a Principal”.
  • Enter “DEMO\SQLService”, click on “Check Names” and then click “OK”.
  • Click “Full Control” and click “OK”.
  • Click “OK”, then click on “Next”, then click on “Create”
  • Click on “Close” after the wizard finishes creating the share.
  • Repeat the process for a share called “LOG” on Volume2.

image


9. Configure the SQL Server

9.1. Mount the SQL Server ISO file

  • Copy the SQL Server 2012 ISO file to the C:\ISO folder.
  • Mount that in the DVD for VM4, DEMO-DB.

9.1.PS. Using PowerShell

Set-VMDvdDrive –VMName VM4 -Path C:\ISO\SQLFULL_ENU.iso

9.1.GUI. Using Hyper-V Manager

  • In Server Manager, click on “Tools” in the upper left and select “Hyper-V Manager”
  • In Hyper-V Manager, click on the server name on the pane on the left
  • Right-click VM4 and click on “Connect…”
  • In the “Media” menu, select “DVD Drive” and then “Insert Disk…”
  • Point to the SQL Server 2012 ISO file under the C:\ISO folder.

9.2. Run SQL Server Setup

  • From VM4 (DEMO-DB), run SQL Server 2012 setup from the DVD.
  • In the “SQL Server Installation Center”, click on “Installation”, then select “New SQL Server stand-alone…”
  • Let it verify the SQL Server Setup Support Rules pass and click “OK”
  • Select “Evaluation” under “Specify a free edition” and click “Next”
  • Review the licensing terms and click “Next”
  • “SQL Server 2012 Setup” will start. Let it verify Setup Support Rules pass and click “Next”.
  • In the “Setup Role” page, select “SQL Server Feature Installation” and click “Next”.
  • In the “Feature selection” page, select only the “Database Engine Services” and the “Management Tools”.
  • Use the default directories. Click “Next”.
  • In the “Installation Rules” page, click “Next”.
  • In the “Instance Configuration” page, click “Next”
  • In the “Disk Space Requirements” page, click “Next”.
  • In the “Server Configuration” page, enter “DEMO.TEST\SQLService” as the account name for the SQL Server Database Engine and the SQL Server Agent, set them both to start automatically. Click “Next”
     
    clip_image024
     
  • In the “Database Engine Configuration” page, click on “Add Current User”.
  • Click on the “Data Directories” tab. Enter “\\DEMO-FS\DATA” as the “Data Root Directory”.
  • Fix the two path for Log directories to use “\\DEMO-FS\LOG” instead of the data folder.
     
    clip_image025
     
  • You will be prompted to confirm the right permissions are assigned on the share. Click “Yes”.
     
    clip_image026
     
  • On the “Error reporting” page, click “Next”
  • On the “Installation Configuration Rules” page, click “Next”
  • On the “Ready to Install” page, click “Install”
  • The installation will complete.

9.3. Create a database using the clustered file share

  • On the SQL Server VM, open SQL Server Management Studio.
  • On the “Connect to Server” window, accept the default server name and authentication. Click “Connect”.
  • Right click the main node, select Properties and click on the “Database Settings”.
  • Verify that that the “Database default locations” point to the file shares entered earlier.
     
    clip_image027
     
  • Click “OK” to close the “Server Properties”.
  • Expand the tree on the left to find the Databases node.
  • Right-click “Databases” and select “New Database…”
  • Enter “Orders” as the database name and note the path pointing to the clustered file share.
  • Scroll the bar to the right to see the Path column:
     
    clip_image028
     
  • Click “OK” to create the database.

10. Verify SMB features

10.1. Verify that SMB Multichannel is working

  • Use PowerShell to verify that SMB is indeed using multiple interfaces.

10.1.PS. Using PowerShell (from VM4, DEMO-DB)

Get-SmbConnection
Get-SmbMultichannelConnection

10.1.OUT. Sample output

PS C:\> Get-SmbConnection

ServerName   ShareName    UserName            Credential          Dialect     NumOpens
----------   ---------    --------            ----------          -------     --------
DEMO-FS      Data         DEMO\SQLService     DEMO.TEST\SQLSer... 3.00        11
DEMO-FS      Log          DEMO\SQLService     DEMO.TEST\SQLSer... 3.00        2

PS C:\> Get-SmbMultichannelConnection

Server Name  Selected   Client IP      Server IP      Client         Server         Client RSS     Client RDMA
                                                      Interface      Interface      Capable        Capable
                                                      Index          Index
-----------  --------   ---------      ---------      -------------- -------------- -------------- --------------
DEMO-FS      True       192.168.103.4  192.168.103.3  23             22             True           False
DEMO-FS      True       192.168.101.4  192.168.101.3  19             18             True           False
DEMO-FS      True       192.168.102.4  192.168.102.3  21             16             True           False

10.2. Query the SMB Sessions and Open Files

  • Use PowerShell to verify sessions and open files.

10.2.PS. Using PowerShell (from VM2, DEMO-F1 or VM3, DEMO-F2)

Get-SmbSession
Get-SmbOpenFile | Sort Path
Get-SmbOpenFile | Sort Path | FT Path

10.2.OUT. Sample output

PS C:\> Get-SmbSession

SessionId       ClientComputerName            ClientUserName                NumOpens
---------       ------------------            --------------                --------
154618822685    [fe80::a08a:1e3d:8e27:3288]   DEMO\DEMO-F2$                 0
154618822681    [fe80::407e:dd35:3c1c:bec5]   DEMO\DEMO-F1$                 0
8813272891477   192.168.101.4                 DEMO\SQLService               13

PS C:\> Get-SmbOpenFile | Sort Path

FileId          SessionId        Path                ShareRelativePath   ClientComputerName  ClientUserName
------          ---------        ----                -----------------   ------------------  --------------
8813272893453   8813272891477    C:\ClusterStorag... MSSQL11.MSSQLSER... 192.168.101.4       DEMO\SQLService
8813272893465   8813272891477    C:\ClusterStorag... MSSQL11.MSSQLSER... 192.168.101.4       DEMO\SQLService
8813272893577   8813272891477    C:\ClusterStorag... MSSQL11.MSSQLSER... 192.168.101.4       DEMO\SQLService
8813272893589   8813272891477    C:\ClusterStorag... MSSQL11.MSSQLSER... 192.168.101.4       DEMO\SQLService
8813272893545   8813272891477    C:\ClusterStorag... MSSQL11.MSSQLSER... 192.168.101.4       DEMO\SQLService
8813272893557   8813272891477    C:\ClusterStorag... MSSQL11.MSSQLSER... 192.168.101.4       DEMO\SQLService
8813272893993   8813272891477    C:\ClusterStorag... MSSQL11.MSSQLSER... 192.168.101.4       DEMO\SQLService
8813272893665   8813272891477    C:\ClusterStorag... MSSQL11.MSSQLSER... 192.168.101.4       DEMO\SQLService
8813272893417   8813272891477    C:\ClusterStorag... MSSQL11.MSSQLSER... 192.168.101.4       DEMO\SQLService
8813272893505   8813272891477    C:\ClusterStorag... MSSQL11.MSSQLSER... 192.168.101.4       DEMO\SQLService
8813272893497   8813272891477    C:\ClusterStorag... MSSQL11.MSSQLSER... 192.168.101.4       DEMO\SQLService
8813272894041   8813272891477    C:\ClusterStorag... MSSQL11.MSSQLSER... 192.168.101.4       DEMO\SQLService
8813272893673   8813272891477    C:\ClusterStorag... MSSQL11.MSSQLSER... 192.168.101.4       DEMO\SQLService

PS C:\> Get-SmbOpenFile | Sort Path | FT Path

Path
----
C:\ClusterStorage\Volume1\DATA\MSSQL11.MSSQLSERVER\MSSQL\DATA\master.mdf
C:\ClusterStorage\Volume1\DATA\MSSQL11.MSSQLSERVER\MSSQL\DATA\mastlog.ldf
C:\ClusterStorage\Volume1\DATA\MSSQL11.MSSQLSERVER\MSSQL\DATA\model.mdf
C:\ClusterStorage\Volume1\DATA\MSSQL11.MSSQLSERVER\MSSQL\DATA\modellog.ldf
C:\ClusterStorage\Volume1\DATA\MSSQL11.MSSQLSERVER\MSSQL\DATA\MSDBData.mdf
C:\ClusterStorage\Volume1\DATA\MSSQL11.MSSQLSERVER\MSSQL\DATA\MSDBLog.ldf
C:\ClusterStorage\Volume1\DATA\MSSQL11.MSSQLSERVER\MSSQL\DATA\Orders.mdf
C:\ClusterStorage\Volume1\DATA\MSSQL11.MSSQLSERVER\MSSQL\DATA\tempdb.mdf
C:\ClusterStorage\Volume1\DATA\MSSQL11.MSSQLSERVER\MSSQL\Log\ERRORLOG
C:\ClusterStorage\Volume1\DATA\MSSQL11.MSSQLSERVER\MSSQL\Log\log_1.trc
C:\ClusterStorage\Volume1\DATA\MSSQL11.MSSQLSERVER\MSSQL\Log\system_health_0_129897856369790000.xel
C:\ClusterStorage\Volume2\LOG\MSSQL11.MSSQLSERVER\MSSQL\Data\Orders_log.ldf
C:\ClusterStorage\Volume2\LOG\MSSQL11.MSSQLSERVER\MSSQL\Data\templog.ldf

10.3. Planned move of a file server node (with SMB Transparent Failover of SQL Client)

  • Use SMB Transparent Failover and Witness to move the SMB Client (SQL Server) between the two File Server cluster nodes.
  • Note: Moves can sometimes take a few seconds to complete, so you might need to repeat the command to see the witness.

10.3.PS. Using PowerShell (from VM2, DEMO-F1 or VM3, DEMO-F2)

Get-SmbWitnessClient | FT ClientName, NetworkName, FileServerNodeName
Move-SmbWitnessClient -ClientName DEMO-DB -DestinationNode DEMO-F1
1..10 | % {Start-Sleep 5; Get-SmbWitnessClient | FT ClientName, NetworkName, FileServerNodeName }

10.3.OUT. Sample output

PS C:\> Get-SmbWitnessClient | FT ClientName, NetworkName, FileServerNodeName

ClientName     NetworkName     FileServerNodeName
----------     -----------     ------------------
DEMO-DB        DEMO-FS         DEMO-F1

PS C:\> Move-SmbWitnessClient -ClientName DEMO-DB -DestinationNode DEMO-F2

Confirm
Are you sure you want to perform this action?
Performing operation 'Move' on Target 'DEMO-DB'.
[Y] Yes  [A] Yes to All  [N] No  [L] No to All  [S] Suspend  [?] Help (default is "Y"):

PS C:\> Get-SmbWitnessClient | FT ClientName, NetworkName, FileServerNodeName

ClientName     NetworkName     FileServerNodeName
----------     -----------     ------------------
DEMO-DB        DEMO-FS         DEMO-F2


PS C:\> Move-SmbWitnessClient -ClientName DEMO-DB -DestinationNode DEMO-F1

Confirm
Are you sure you want to perform this action?
Performing operation 'Move' on Target 'DEMO-DB'.
[Y] Yes  [A] Yes to All  [N] No  [L] No to All  [S] Suspend  [?] Help (default is "Y"):

PS C:\> Get-SmbWitnessClient | FT ClientName, NetworkName, FileServerNodeName

ClientName     NetworkName     FileServerNodeName
----------     -----------     ------------------
DEMO-DB        DEMO-FS         DEMO-F2


PS C:\> Get-SmbWitnessClient | FT ClientName, NetworkName, FileServerNodeName

ClientName     NetworkName     FileServerNodeName
----------     -----------     ------------------
DEMO-DB        DEMO-FS         DEMO-F1

10.4. Unplanned failure of a file server node (with SMB Transparent Failover of SQL Client)

  • Restart one of the servers to force an SMB Transparent Failover to move the SMB Client (SQL Server) between the two File Server cluster nodes.
  • Note: After the failover, the client moves to the surviving nodes transparently (in just a few seconds). However, there will be no witness servers available then. That’s fine, but during that time we’re running with a single surviving file server node, with no witness. After the file server restarts and becomes available again (which could take a minute or so) the client will reconnect to it as a witness and we are back in a two-file-server configuration.

10.4.PS. Using PowerShell (from VM2, DEMO-F1)

Get-SmbWitnessClient | FT ClientName, NetworkName, FileServerNodeName
Restart-Computer –ComputerName DEMO-F2 –Force
1..100 | % { Start-Sleep 5; Get-SmbWitnessClient | FT ClientName, NetworkName, FileServerNodeName }

10.4.OUT. Sample output

PS C:\> Get-SmbWitnessClient | FT ClientName, NetworkName, FileServerNodeName

ClientName     NetworkName     FileServerNodeName
----------     -----------     ------------------
DEMO-DB        DEMO-FS         DEMO-F2

PS C:\> Restart-Computer -ComputerName DEMO-F2 -Force

PS C:\> Get-SmbWitnessClient | FT ClientName, NetworkName, FileServerNodeName

PS C:\> Get-SmbWitnessClient | FT ClientName, NetworkName, FileServerNodeName

ClientName     NetworkName     FileServerNodeName
----------     -----------     ------------------
DEMO-DB        DEMO-FS         DEMO-F1

10.5. Surviving the loss of a client NIC

  • Observe that SMB Multichannel will protect the SQL Server from the failure of a NIC.

10.5.PS. Using PowerShell (from VM4, DEMO-DB)

Get-SmbMultichannelConnection
Disable-NetAdapter -InterfaceAlias Internal3 –Confirm:$false ; Start-Sleep 20

Get-SmbMultichannelConnection

Enable-NetAdapter -InterfaceAlias Internal3 ; Start-Sleep 20
Get-SmbMultichannelConnection

10.5.OUT. Sample output

PS C:\> Get-SmbMultichannelConnection

Server Name  Selected   Client IP      Server IP      Client         Server         Client RSS     Client RDMA
                                                      Interface      Interface      Capable        Capable
                                                      Index          Index
-----------  --------   ---------      ---------      -------------- -------------- -------------- --------------
DEMO-FS      True       192.168.103.4  192.168.103.3  23             22             True           False
DEMO-FS      True       192.168.101.4  192.168.101.3  19             18             True           False
DEMO-FS      True       192.168.102.4  192.168.102.3  21             16             True           False


PS C:\> Disable-NetAdapter -InterfaceAlias Internal3 –Confirm:$false ; Start-Sleep 20
PS C:\> Get-SmbMultichannelConnection

Server Name  Selected   Client IP      Server IP      Client         Server         Client RSS     Client RDMA
                                                      Interface      Interface      Capable        Capable
                                                      Index          Index
-----------  --------   ---------      ---------      -------------- -------------- -------------- --------------
DEMO-FS      True       192.168.101.4  192.168.101.3  19             18             True           False
DEMO-FS      True       192.168.102.4  192.168.102.3  21             16             True           False


PS C:\> Enable-NetAdapter -InterfaceAlias Internal3 ; Start-Sleep 20
PS C:\> Get-SmbMultichannelConnection

Server Name  Selected   Client IP      Server IP      Client         Server         Client RSS     Client RDMA
                                                      Interface      Interface      Capable        Capable
                                                      Index          Index
-----------  --------   ---------      ---------      -------------- -------------- -------------- --------------
DEMO-FS      True       192.168.103.4  192.168.103.3  23             22             True           False
DEMO-FS      True       192.168.101.4  192.168.101.3  19             18             True           False
DEMO-FS      True       192.168.102.4  192.168.102.3  21             16             True           False


11. Shut down, startup and install final notes

  • Keep in mind that there are dependencies between the services running on each VM.
  • To shut them down, start with VM4 and end with VM1, waiting for each one to go down completely before moving to the next one.
  • To bring the VMs up, go from VM1 to VM4, waiting for the previous one to be fully up (with low to no CPU usage) before starting the next one.
  • You might want to also take a snapshot of the VMs after you shut them down, just in case you want to bring them back to the original state after experimenting with them for a while.
  • If you do, you should always snapshot all of them, again due to dependencies between them. Just right-click the VM and select the “Snapshot” option.
  • As a last note, the total size of the VHDX files (base plus 4 diffs), after all the steps were performed, was around 19 GB. 

image


12. Conclusion

I hope you enjoyed these step-by-step instructions. I strongly encourage you to try them out and perform the entire installation yourself. It’s a good learning experience.

 

Note: If you want to print this post, I have attached a printer-friendly PDF version below. Thanks for the suggestion, Keith!

Test Hyper-V over SMB configuration with Windows Server 2012 - Step-by-step Installation using PowerShell

$
0
0

1. Overview

In this post, I am sharing all the steps I used to create a Windows Server 2012 File Server test environment that I used for some of my Hyper-V over SMB demonstrations. My goal with this post is to share some of configuration details and the exact PowerShell scripts I used to configure the environment (if you look carefully, you might be able to spot a few PowerShell tricks and tips). For me, this is a convenient reference post that I will likely use myself to cut/paste from when configuring my demo systems in the future.

This uses 5 physical machines, since the scenario involves deploying Hyper-V hosts and you can’t virtualize Hyper-V itself (I have another post that cover SQL Server over SMB in a fully virtualized environment). I am also using RDMA interfaces on the setup with SMB Direct, and those also can’t be virtualized. The demo setup includes one domain controller (which also doubles as an iSCSI target), two file servers and two Hyper-V hosts.

This is probably the most basic fault-tolerant Hyper-V over SMB setup you can create that covers the entire spectrum of new SMB 3.0 capabilities (including SMB Transparent Failover, SMB Scale-Out, SMB Direct and SMB Multichannel). If you build a similar configuration, please share your experience in the comments below the post.

Please keep in mind that this is not a production-ready configuration. I built it entirely using 5-year-old desktop class machines. To improved disk performance, I did add 3 SSDs to one of the machines to used as storage for my cluster, which I configured using Storage Spaces and the Microsoft iSCSI Software target included in Windows Server 2012. However, since I only had three small SSDs, I used a simple space, which cannot tolerate disk failures. In production, you should use mirrored spaces. Also keep in mind that the FST2-DC1 machine itself is a single point of failure, so you’re really only tolerant to the failure of one the two Hyper-V hosts or one of the File Server nodes. In summary, this is a test-only configuration.

 

2. Environment details

The environment is deployed as 5 physical machines, all using the FST2.TEST domain. Here’s a diagram of the setup so you can better understand it:

image

 

Here are the details about the names, roles and IP addresses for each of the computers involved, including the cluster objects and VMs:

ComputerRoleExternalInternalRDMA 1RDMA 2
fst2-dc1DNS, Domain Controller, iSCSI TargetDHCP192.168.100.10/24192.168.101.10/24N/A
fst2-fs1File Server 1DHCP192.168.100.11/24192.168.101.11/24192.168.102.11/24
fst2-fs2File Server 2DHCP192.168.100.12/24192.168.101.12/24192.168.102.12/24
fst2-hv1Hyper-V Server 1DHCP192.168.100.13/24192.168.101.13/24192.168.102.13/24
fst2-hv2Hyper-V Server 2DHCP192.168.100.14/24N/A192.168.102.14/24
fst2-fscFile Server Cluster Name ObjectDHCPN/AN/AN/A
fst2-fsClassic File Server ClusterN/A192.168.100.22/24192.168.101.22/24192.168.102.22/24
fst2-soScale-Out​ File Server ClusterN/AN/AN/AN/A
fst2-hvcHyper-V Cluster Name ObjectDHCPN/AN/AN/A
fst2-vm*Virtual MachineDHCPN/AN/AN/A

 

Last but not least, here’s a picture of the setup, so you can get a sense of what it looks like:

image

 

3. Steps to configure FST2-DC1 (DNS, Domain Controller, iSCSI Target)

# Note 1: This post assumes you already installed Windows Server 2012 and configured the computer name. For details on how to do this, see this previous blog post.
# Note 2: This setup uses InfiniBand RDMA interfaces. For specific instructions on that part of the configuration (driver download, OpenSM subnet manager), see this previous blog post.
#

#
# Set power profile
#
POWERCFG.EXE /S SCHEME_MIN

#
# Configure all 4 interfaces (1 DHCP, 3 static)

#
# Rename External, no further action required, since this is DHCP
#
Get-NetAdapter -InterfaceDescription "*Intel*"   | Rename-NetAdapter -NewName "External"

#
# Rename Internal, set to manual IP address, configure IP Address, DNS
#
Get-NetAdapter -InterfaceDescription "*Realtek*" | Rename-NetAdapter -NewName "Internal"
Set-NetIPInterface -InterfaceAlias Internal -DHCP Disabled
Remove-NetIPAddress -InterfaceAlias Internal -Confirm:$false
New-NetIPAddress -InterfaceAlias Internal -IPAddress 192.168.100.10 -PrefixLength 24
Set-DnsClientServerAddress -InterfaceAlias Internal -ServerAddresses 192.168.100.10

#
# Rename RDMA1, set to manual IP address, configure IP Address, DNS
#
Get-NetAdapter -InterfaceDescription "*IPoIB*" | Select -Last 1 | Rename-NetAdapter -NewName RDMA1
Set-NetIPInterface -InterfaceAlias RDMA1 -DHCP Disabled
Remove-NetIPAddress -InterfaceAlias RDMA1 -Confirm:$false
New-NetIPAddress -InterfaceAlias RDMA1 -IPAddress 192.168.101.10 -PrefixLength 24
Set-DnsClientServerAddress -InterfaceAlias RDMA1 -ServerAddresses 192.168.100.10
Get-NetAdapter -InterfaceDescription "*IPoIB*" | ? {$_.Name -ne "RDMA1"} | Rename-NetAdapter -NewName RDMA2

#
# Disable RDMA2, since this system only uses one RDMA interface
#
Disable-NetAdapter -InterfaceAlias RDMA2 -Confirm:$false

#
# Configure Storage Spaces, create pool with 3 disks, single simple space
#
$s = Get-StorageSubSystem -FriendlyName *Spaces*
New-StoragePool -FriendlyName Pool1 -StorageSubSystemFriendlyName $s.FriendlyName -PhysicalDisks (Get-PhysicalDisk -CanPool $true)
Set-ResiliencySetting -Name Simple -NumberofColumnsDefault 3 -StoragePool (Get-StoragePool -FriendlyName Pool1)

#
# Create Space (virtual disk)
#
New-VirtualDisk -FriendlyName Space1 -StoragePoolFriendlyName Pool1 -ResiliencySettingName Simple -UseMaximumSize

#
# Initialize Space, partition, create volume, format as X:
#
$c = Get-VirtualDisk -FriendlyName Space1 | Get-Disk
Set-Disk -Number $c.Number -IsReadOnly 0
Set-Disk -Number $c.Number -IsOffline 0
Initialize-Disk -Number $c.Number -PartitionStyle GPT
New-Partition -DiskNumber $c.Number -DriveLetter X -UseMaximumSize
Initialize-Volume -DriveLetter X -FileSystem NTFS -Confirm:$false

#
# Install iSCSI Software Target
#
Install-WindowsFeature FS-iSCSITarget-Server

#
# Create iSCSI target for two initiators (configured by IP address) with 5 LUNs (1GB for witness disks, four 100GB for data disks)
#
New-IscsiServerTarget -TargetName FSTarget -InitiatorID IPAddress:192.168.101.11, IPAddress:192.168.101.12
New-IscsiVirtualDisk -DevicePath X:\LUN0.VHD -size 1GB
1..4 | % {New-IscsiVirtualDisk -DevicePath X:\LUN$_.VHD -size 100GB}
Add-iSCSIVirtualDiskTargetMapping -TargetName FSTarget -DevicePath X:\LUN0.VHD
1..4 | % {Add-iSCSIVirtualDiskTargetMapping -TargetName FSTarget -DevicePath X:\LUN$_.VHD}

#
# Install Active Directory
#
Install-WindowsFeature AD-Domain-Services

#
# Create AD forest, reboots at the end
#
Install-ADDSForest `
-CreateDNSDelegation:$false `
-DatabasePath "C:\Windows\NTDS" `
-DomainMode "Win2008R2" `
-DomainName "FST2.TEST" `
-DomainNetBIOSName "FST2" `
-ForestMode "Win2008R2" `
-InstallDNS:$true `
-LogPath "C:\Windows\NTDS" `
-SafeModeAdministratorPassword (Read-Host -AsSecureString -Prompt "Enter Password") `
-SYSVOLPath "C:\Windows\SYSVOL"

 

4. Steps to configure FST2-FS1 (File Server 1)

#
# Set service power profile
#
POWERCFG.EXE /S SCHEME_MIN  
 
#
# Configure all 4 interfaces (1 DHCP, 3 static)
#

#
# Rename External, no further action required, since this is DHCP
#
Get-NetAdapter -InterfaceDescription "*Intel*" | Rename-NetAdapter -NewName "External"

#
# Rename Internal, set to manual IP address, configure IP Address, DNS
#
Get-NetAdapter -InterfaceDescription "*Realtek*" | Rename-NetAdapter -NewName "Internal"
Set-NetIPInterface -InterfaceAlias Internal -DHCP Disabled
Remove-NetIPAddress -InterfaceAlias Internal -Confirm:$false
New-NetIPAddress -InterfaceAlias Internal -IPAddress 192.168.100.11 -PrefixLength 24
Set-DnsClientServerAddress -InterfaceAlias Internal -ServerAddresses 192.168.100.10

#
# Rename RDMA1, set to manual IP address, configure IP Address, DNS
#
Get-NetAdapter -InterfaceDescription "*IPoIB*" | Select -Last 1 | Rename-NetAdapter -NewName RDMA1
Set-NetIPInterface -InterfaceAlias RDMA1 -DHCP Disabled
Remove-NetIPAddress -InterfaceAlias RDMA1 -Confirm:$false
New-NetIPAddress -InterfaceAlias RDMA1 -IPAddress 192.168.101.11 -PrefixLength 24
Set-DnsClientServerAddress -InterfaceAlias RDMA1 -ServerAddresses 192.168.100.10

#
# Rename RDMA2, set to manual IP address, configure IP Address, DNS
#
Get-NetAdapter -InterfaceDescription "*IPoIB*" | ? {$_.Name -ne "RDMA1"} | Rename-NetAdapter -NewName RDMA2
Set-NetIPInterface -InterfaceAlias RDMA2 -DHCP Disabled
Remove-NetIPAddress -InterfaceAlias RDMA2 -Confirm:$false
New-NetIPAddress -InterfaceAlias RDMA2 -IPAddress 192.168.102.11 -PrefixLength 24
Set-DnsClientServerAddress -InterfaceAlias RDMA2 -ServerAddresses 192.168.100.10

#
# Join Domain, restart the machine
#
Add-Computer -DomainName FST2.TEST -Credential (Get-Credential) -Restart

#
# Install File Server
#
Install-WindowsFeature File-Services, FS-FileServer, Failover-Clustering
Install-WindowsFeature RSAT-Clustering -IncludeAllSubFeature

#
# Start iSCSI Software Initiator
#
Set-Service MSiSCSI -StartupType automatic
Start-Service MSiSCSI

#
# Configure iSCSI Software Initiator
#

New-iSCSITargetPortal -TargetPortalAddress 192.168.101.10
Get-iSCSITarget | Connect-iSCSITarget
Get-iSCSISession | Register-iSCSISession

#
# Configure the five iSCSI LUNs (initialize, create partition, volume, format as drives J: to N:
#
1..5 | % { 
    $Letter ="JKLMN"[($_-1)]
    Set-Disk -Number $_ -IsReadOnly 0
    Set-Disk -Number $_ -IsOffline 0
    Initialize-Disk -Number $_ -PartitionStyle MBR
    New-Partition -DiskNumber $_ -DriveLetter $Letter -UseMaximumSize 
    Initialize-Volume -DriveLetter $Letter -FileSystem NTFS -Confirm:$false
}

 

5. Steps to configure FST2-FS2 (File Server 2)

#
# Set service power profile
#
POWERCFG.EXE /S SCHEME_MIN 

#
# Configure all 4 interfaces (1 DHCP, 3 static)
#

#
# Rename External, no further action required, since this is DHCP
#
Get-NetAdapter -InterfaceDescription "*Intel*" | Rename-NetAdapter -NewName "External"

#
# Rename Internal, set to manual IP address, configure IP Address, DNS
#
Get-NetAdapter -InterfaceDescription "*Realtek*" | Rename-NetAdapter -NewName "Internal"
Set-NetIPInterface -InterfaceAlias Internal -DHCP Disabled
Remove-NetIPAddress -InterfaceAlias Internal -Confirm:$false
New-NetIPAddress -InterfaceAlias Internal -IPAddress 192.168.100.12 -PrefixLength 24
Set-DnsClientServerAddress -InterfaceAlias Internal -ServerAddresses 192.168.100.10

#
# Rename RDMA1, set to manual IP address, configure IP Address, DNS
#
Get-NetAdapter -InterfaceDescription "*IPoIB*" | Select -Last 1 | Rename-NetAdapter -NewName RDMA1
Set-NetIPInterface -InterfaceAlias RDMA1 -DHCP Disabled
Remove-NetIPAddress -InterfaceAlias RDMA1 -Confirm:$false
New-NetIPAddress -InterfaceAlias RDMA1 -IPAddress 192.168.101.12 -PrefixLength 24
Set-DnsClientServerAddress -InterfaceAlias RDMA1 -ServerAddresses 192.168.100.10

#
# Rename RDMA2, set to manual IP address, configure IP Address, DNS
#
Get-NetAdapter -InterfaceDescription "*IPoIB*" | ? {$_.Name -ne "RDMA1"} | Rename-NetAdapter -NewName RDMA2
Set-NetIPInterface -InterfaceAlias RDMA2 -DHCP Disabled
Remove-NetIPAddress -InterfaceAlias RDMA2 -Confirm:$false
New-NetIPAddress -InterfaceAlias RDMA2 -IPAddress 192.168.102.12 -PrefixLength 24
Set-DnsClientServerAddress -InterfaceAlias RDMA2 -ServerAddresses 192.168.100.10

#
# Join Domain
#
Add-Computer -DomainName FST2.TEST -Credential (Get-Credential) -Restart

#
# Install File Server
#
Install-WindowsFeature File-Services, FS-FileServer, Failover-Clustering
Install-WindowsFeature RSAT-Clustering -IncludeAllSubFeature

#
# Start iSCSI Software Initiator
#
Set-Service MSiSCSI -StartupType automatic
Start-Service MSiSCSI

#
# Configure iSCSI Software Initiator
#
New-iSCSITargetPortal -TargetPortalAddress 192.168.101.10
Get-iSCSITarget | Connect-iSCSITarget
Get-iSCSISession | Register-iSCSISession

#
# No need to configure LUNs here. In a cluster, this is done only from one of the nodes. We did it in FS1.
#

 

6. Steps to configure FST2-HV1 (Hyper-V host 1)

#
# Set service power profile
#
POWERCFG.EXE /S SCHEME_MIN 

#
# Configure all 4 interfaces (1 DHCP, 3 static)
#

#
# Rename External, no further action required, since this is DHCP
#
Get-NetAdapter -InterfaceDescription "*82566DM*" | Rename-NetAdapter -NewName "External"

#
# Rename Internal, set to manual IP address, configure IP Address, DNS
#
Get-NetAdapter -InterfaceDescription "*PRO/100*" | Rename-NetAdapter -NewName "Internal"
Set-NetIPInterface -InterfaceAlias Internal -DHCP Disabled
Remove-NetIPAddress -InterfaceAlias Internal -Confirm:$false
New-NetIPAddress -InterfaceAlias Internal -IPAddress 192.168.100.13 -PrefixLength 24
Set-DnsClientServerAddress -InterfaceAlias Internal -ServerAddresses 192.168.100.10

#
# Rename RDMA1, set to manual IP address, configure IP Address, DNS
#
Get-NetAdapter -InterfaceDescription "*IPoIB*" | Select -Last 1 | Rename-NetAdapter -NewName RDMA1
Set-NetIPInterface -InterfaceAlias RDMA1 -DHCP Disabled
Remove-NetIPAddress -InterfaceAlias RDMA1 -Confirm:$false
New-NetIPAddress -InterfaceAlias RDMA1 -IPAddress 192.168.101.13 -PrefixLength 24
Set-DnsClientServerAddress -InterfaceAlias RDMA1 -ServerAddresses 192.168.100.10

#
# Rename RDMA2, set to manual IP address, configure IP Address, DNS
#
Get-NetAdapter -InterfaceDescription "*IPoIB*" | ? {$_.Name -ne "RDMA1"} | Rename-NetAdapter -NewName RDMA2
Set-NetIPInterface -InterfaceAlias RDMA2 -DHCP Disabled
Remove-NetIPAddress -InterfaceAlias RDMA2 -Confirm:$false
New-NetIPAddress -InterfaceAlias RDMA2 -IPAddress 192.168.102.13 -PrefixLength 24
Set-DnsClientServerAddress -InterfaceAlias RDMA2 -ServerAddresses 192.168.100.10

#
# Install Hyper-V
#
Install-WindowsFeature Hyper-V, Hyper-V-PowerShell, Hyper-V-Tools, Failover-Clustering
Install-WindowsFeature RSAT-Clustering -IncludeAllSubFeature

#
# Join Domain, restart
#
Add-Computer -DomainName FST2.Test -Credential (Get-Credential) –Restart

 

7. Steps to configure FST2-HV2 (Hyper-V host 2)

#
# Set service power profile
#
POWERCFG.EXE /S SCHEME_MIN 

#
# Configure all 4 interfaces (1 DHCP, 3 static)
#

#
# Rename External, no further action required, since this is DHCP
#
Get-NetAdapter -InterfaceDescription "*82566DM*" | Rename-NetAdapter -NewName "External"

#
# Rename Internal, set to manual IP address, configure IP Address, DNS
#
Get-NetAdapter -InterfaceDescription "*PRO/100*" | Rename-NetAdapter -NewName "Internal"
Set-NetIPInterface -InterfaceAlias Internal -DHCP Disabled
Remove-NetIPAddress -InterfaceAlias Internal -Confirm:$false
New-NetIPAddress -InterfaceAlias Internal -IPAddress 192.168.100.14 -PrefixLength 24
Set-DnsClientServerAddress -InterfaceAlias Internal -ServerAddresses 192.168.100.10

#
# Rename RDMA1, set to manual IP address, configure IP Address, DNS
#
Get-NetAdapter -InterfaceDescription "*IPoIB*" | Select -Last 1 | Rename-NetAdapter -NewName RDMA1
Set-NetIPInterface -InterfaceAlias RDMA1 -DHCP Disabled
Remove-NetIPAddress -InterfaceAlias RDMA1 -Confirm:$false
New-NetIPAddress -InterfaceAlias RDMA1 -IPAddress 192.168.102.14 -PrefixLength 24
Set-DnsClientServerAddress -InterfaceAlias RDMA1 -ServerAddresses 192.168.100.10

#
# Disable RDMA2, since this system only uses one RDMA interface
#
Get-NetAdapter -InterfaceDescription "*IPoIB*" | ? {$_.Name -ne "RDMA1"} | Rename-NetAdapter -NewName RDMA2
Disable-NetAdapter -InterfaceAlias RDMA2 -Confirm:$false

#
# Install Hyper-V
#
Install-WindowsFeature Hyper-V, Hyper-V-PowerShell, Hyper-V-Tools, Failover-Clustering
Install-WindowsFeature RSAT-Clustering -IncludeAllSubFeature

#
# Join Domain, restart
#
Add-Computer -DomainName FST2.Test -Credential (Get-Credential) -Restart

 

8. Steps to configure the Cluster FST2-FSC (run from FST2-FS1)

#
# Run Failover Cluster Validation
#
Test-Cluster -Node FST2-FS1, FST2-FS2

#
# Create cluster
#
New-Cluster –Name FST2-FSC -Node FST2-FS1, FST2-FS2

#
# Rename Networks
#
(Get-ClusterNetwork | ? {$_.Address -like "192.168.100.*" }).Name = "Internal"
(Get-ClusterNetwork | ? {$_.Address -like "192.168.101.*" }).Name = "RDMA1"
(Get-ClusterNetwork | ? {$_.Address -like "192.168.102.*" }).Name = "RDMA2"
(Get-ClusterNetwork | ? {$_.Address -like "172.*" }).Name = "External"

#
# Configure Cluster Network Roles (0=Not used, 1=Cluster only, 3=Cluster+Clients)
#
(Get-ClusterNetwork Internal).Role = 3
(Get-ClusterNetwork RDMA1).Role = 3
(Get-ClusterNetwork RDMA2).Role = 3
(Get-ClusterNetwork External).Role = 1

#
# Rename Witness Disk
#
$w = Get-ClusterResource | ? { $_.OwnerGroup -eq "Cluster Group" -and $_.ResourceType -eq "Physical Disk"}
$w.Name = "WitnessDisk"

 

9. Steps to configure the Classic File Server Cluster FST2-FS (run from FST2-FS1):

#
# Move all disks to node one, rename Cluster Disks
#
Get-ClusterGroup | Move-ClusterGroup -Node FST2-FS1
(Get-Volume -DriveLetter I | Get-Partition | Get-Disk | Get-ClusterResource).Name = "FSDisk1"
(Get-Volume -DriveLetter J | Get-Partition | Get-Disk | Get-ClusterResource).Name = "FSDisk2"

#
# Create a classic file server resource group
#
Add-ClusterFileServerRole -Name FST2-FS -Storage FSDisk1, FSDisk2 –StaticAddress 192.168.100.22/24, 192.168.101.22/24, 192.168.102.22/24

#
# Create Folders
#
Move-ClusterGroup -Name FST2-FS -Node FST2-FS1
md I:\VMS
md J:\VMS

#
# Create File Shares
#
New-SmbShare -Name VMS1 -Path I:\VMS -FullAccess FST2.Test\Administrator, FST2.Test\FST2-HV1$, FST2.Test\FST2-HV2$
New-SmbShare -Name VMS2 -Path J:\VMS -FullAccess FST2.Test\Administrator, FST2.Test\FST2-HV1$, FST2.Test\FST2-HV2$

#
# Set NTFS permissions
#
(Get-SmbShare VMS1).PresetPathAcl | Set-Acl
(Get-SmbShare VMS2).PresetPathAcl | Set-Acl

 

10. Steps to configure the Scale-Out File Server Cluster FST2-SO (run from FST2-FS1):

#
# Add two remaining disks to Cluster Shared Volumes
#
Get-ClusterResource | ? OwnerGroup -eq "Available Storage" | Add-ClusterSharedVolume

#
# Create a scale out file server resource group
#
Add-ClusterScaleOutFileServerRole -Name FST2-SO

#
# Create Folders
#
MD C:\ClusterStorage\Volume1\VMS
MD C:\ClusterStorage\Volume2\VMS

#
# Create File Shares
#
New-SmbShare -Name VMS3 -Path C:\ClusterStorage\Volume1\VMS -FullAccess FST2.Test\Administrator, FST2.Test\FST2-HV1$, FST2.Test\FST2-HV2$
New-SmbShare -Name VMS4 -Path C:\ClusterStorage\Volume2\VMS -FullAccess FST2.Test\Administrator, FST2.Test\FST2-HV1$, FST2.Test\FST2-HV2$

#
# Set NTFS permissions
#
(Get-SmbShare VMS3).PresetPathAcl | Set-Acl
(Get-SmbShare VMS4).PresetPathAcl | Set-Acl

 

11. Steps to configure the VMs in FST2-HV1

#
# Create VM Switch (if doing this remotely, you will need to reconnect)
#
New-VMSwitch -NetAdapterName External -Name External
Get-NetAdapter -InterfaceDescription Hyper* | Rename-NetAdapter -NewName ExternalVirtual

#
# Create VHD files for two VMs
#
New-VHD -Path \\FST2-FS\VMS1\VM1.VHDX -Fixed -SizeBytes 20GB
New-VHD -Path \\FST2-SO\VMS3\VM3.VHDX -Fixed -SizeBytes 20GB

#
# Create two VMs
#
New-VM -Path \\FST2-FS\VMS1 -Name VM1 -VHDPath \\FST2-FS\VMS1\VM1.VHDX -SwitchName External -Memory 1GB
New-VM -Path \\FST2-SO\VMS3 -Name VM3 -VHDPath \\FST2-SO\VMS3\VM3.VHDX -SwitchName External -Memory 1GB
Set-VMDvdDrive -VMName VM1 -Path D:\WindowsServer2012.iso
Set-VMDvdDrive -VMName VM3 -Path D:\WindowsServer2012.iso
Start-VM VM1, VM3

 

12. Steps to configure the VMs in FST2-HV2:

#
# Create VM Switch (if doing this remotely, you will need to reconnect)
#
New-VMSwitch -NetAdapterName External -Name External
Get-NetAdapter -InterfaceDescription Hyper* | Rename-NetAdapter -NewName ExternalVirtual

#
# Create VHD files for two VMs
#
New-VHD -Path \\FST2-FS\VMS2\VM2.VHDX -Fixed -SizeBytes 20GB
New-VHD -Path \\FST2-SO\VMS4\VM4.VHDX -Fixed -SizeBytes 20GB

#
# Create and start two VMs
#
New-VM -Path \\FST2-FS\VMS2 -Name VM2 -VHDPath \\FST2-FS\VMS2\VM2.VHDX -SwitchName External -Memory 1GB
New-VM -Path \\FST2-SO\VMS4 -Name VM4 -VHDPath \\FST2-SO\VMS4\VM4.VHDX -SwitchName External -Memory 1GB
Set-VMDvdDrive -VMName VM2 -Path D:\WindowsServer2012.iso
Set-VMDvdDrive -VMName VM4 -Path D:\WindowsServer2012.iso
Start-VM VM2, VM4

 

13. Steps to create a Hyper-V Cluster using file share storage

#
# on FST2-HV1
#

#
# Create Hyper-V Cluster called FST2-HVC
#
New-Cluster –Name FST2-HVC -Node FST2-HV1, FST2-HV2

#
# on FST2-FS1
#

#
# Create Folder and File Share for File Share Witness
#
MD C:\ClusterStorage\Volume1\Witness
New-SmbShare -Name Witness -Path C:\ClusterStorage\Volume1\Witness -FullAccess FST2.Test\Administrator, FST2.Test\FST2-HVC$
(Get-SmbShare Witness).PresetPathAcl | Set-Acl

#
# on FST2-HV1
#

#
# Configure FST2-HVC Cluster with a File Share Witness
#
Set-ClusterQuorum -NodeAndFileShareMajority \\FST2-SO\Witness

#
# Make VMs in FST2-HV1 Highly available
#
Add-VMToCluster VM1
Add-VMToCluster VM3

#
# on FST2-HV2
#

#
# Make VMs in FST2-HV2 Highly available
#
Add-VMToCluster VM2
Add-VMToCluster VM4

 

14. Optional: Steps to create a nonclustered file share on FST2-FS1

#
# on FST2-FS1
#
MD D:\VMS
New-SmbShare -Name VMS5 -Path D:\VMS -FullAccess FST2.Test\Administrator, FST2.Test\FST2-HV1$, FST2.Test\FST2-HV2$
(Get-SmbShare VMS5).PresetPathAcl | Set-Acl

#
# on FST2-HV1
#
New-VHD -Path \\FST2-FS1\VMS5\VM5.VHDX -Fixed -SizeBytes 20GB
New-VM -Path \\FST2-FS1\VMS5 -Name VM5 -VHDPath \\FST2-SO\VMS3\VM3.VHDX -SwitchName External -Memory 1GB
Set-VMDvdDrive -VMName VM5 -Path D:\WindowsServer2012.iso
Start-VM VM5

 

15. Conclusion

Sorry for the somewhat terse post, written mostly in PowerShell instead of English :-).
I hope you enjoy the PowerShell scripting and try at least some of it in your configurations.
For additional test scenarios you could try on a Hyper-V over SMB setup, see this previous blog post.

SNIA’s Storage Developers Conference 2012 is just a few weeks away

$
0
0

The Storage Networking Industry Association (SNIA) is hosting the 9th Storage Developer Conference (SDC) in the Hyatt Regency in beautiful Santa Clara, CA (Silicon Valley) on the week of September 17th. As usual, Microsoft is underwriting the SMB/SMB2/SMB3 Plugfest, which is co-located with the SDC event.

For developers working with storage-related technologies, this event gathers a unique crowd and includes a rich agenda that you can find at http://www.snia.org/events/storage-developer2012/agenda2012.  All the key industry players are represented. It lists presentations from Arista, Cleversafe, Dell, EMC, Fusion-io, HP, IBM, Intel, Mellanox, Micron, Microsoft, NEC, NetApp, Oracle, Pure Storage, Red Hat, Samba Team, Seagate, Spectra Logic, SwiftTest, Tata, Wipro and many others.

It’s always worth reminding you that the SDC presentations are usually delivered to developers by the actual product development teams and frequently the actual developer of the technology is either delivering the presentation or is in the room to take questions. That kind of deep insight is not common in every conference out there.

Presentations by Microsoft this year include:

DateTimeSessionPresenter(s)
Mon10:35SMB 3.0 ( Because 3 > 2 )David Kruse, Principal Software Development Lead
Mon11:35Understanding Hyper-V over SMB 3.0 Through Specific Test CasesJose Barreto, Principal Program Manager
Mon1:30Continuously Available SMB – Observations and Lessons LearnedDavid Kruse, Principal Software Development Lead Mathew George, Principal Software Developer
Mon2:30“Storage Spaces” - Next Generation Virtualized Storage for WindowsKaran Mehra, Principal Software Development Engineer
Tue10:40Windows File and Storage DirectionsSurendra Verma, Partner Development Manager
Tue1:00Hyper-V Storage Performance and ScalingJoe Dai, Principal Software Design Engineer
Liang Yang, Senior Performance Engineer
Tue2:00NFSv4.1 Architecture and Tradeoffs in Windows Server 2012Roopesh Battepati, Principal Development Lead
Tue2:00The Virtual Desktop Infrastructure Storage Behaviors and RequirementsSpencer Shepler, Performance Architect
Tue3:05NoSQL in the Clouds with Windows Azure TableJai Haridas, Principal Development Manager
Tue3:05NFSv4.1 Server Protocol Compliance, Security, Performance and Scalability Testing: Implement the RFC, Going Beyond POSIX Interop!Raymond Wang, Senior Software Design Engineer in Test
Tue3:05SQL Server: Understanding the Application/Data Workload, and Designing Storage Products to Match Desired Characteristics for Better PerformanceGunter Zink, Principal Program Manager
Claus Joergensen, Principal Program Manager
Wed1:15NAS Management using Microsoft Corporation System Center 2012 Virtual Machine Manager and SMI-SAlex Naparu, Software Design Engineer
Madhu Jujare, Senior Software Design Engineer
Wed3:20Erasure Coding in Windows Azure StorageCheng Huang, Researcher
Wed4:20ReFS - Next Generation File System for WindowsJ.R. Tipton, Principal Software Development Engineer
Malcolm Smith, Senior Software Design Engineer
Thu9:30Primary Data Deduplication in Windows Server 8Sudipta Sengupta, Senior Research Scientist
Jim Benton, Principal Software Design Engineer
Thu10:30High Performance File Serving with SMB3 and RDMA via the SMBDirect ProtocolTom Talpey, Software Architect
Greg Kramer, Sr. Software Development Engineer
Thu11:25SMB 3.0 Application End-to-End PerformanceDan Lovinger, Principal Software Architect

Registration is open at http://www.snia.org/events/storage-developer2012/registration and you should definitely plan to attend. If you are registered, leave a comment and let’s plan to meet when we get there!

Compilation of my live tweets from SNIA’s SDC 2012 (Storage Developer Conference)

$
0
0

Here is a compilation of my live tweets from SNIA’s SDC 2012 (Storage Developers Conference).
You can also read those directly from twitter at http://twitter.com/josebarreto (in reverse order)

Notes and disclaimers

  • These tweets were typed during the talks and they include typos and my own misinterpretations.
  • Text under each talk are quotes from the speaker or text from the speaker slides, not my personal opinion.
  • If you feel that I misquoted you or badly represented the content of a talk, please add a comment to the post.
  • I spent just limited time fixing typos or correcting the text after the event. Just so many hours in a day...
  • I have not attended all sessions (since there are 4 or 5 at a time, that would actually not be possible :-)…
  • SNIA usually posts the actual PDF decks a few weeks after the event. Attendees have access immediately.

Linux CIFS/SMB2 Kernel Clients - A Year In Review by Steven French, IBM

  • SMB3 will be important for Linux, not just Windows #sdc2012
  • Linux kernel supports SMB. Kernel 3.7 (Q4-2012) includes 71 changes related to SMB (including SMB 2.1), 3.6 has 61, 3.5 has 42
  • SMB 2.1 kernel code in Linux enabled as experimental in 3.7. SMB 2.1 will replace CIFS as the default client when stable.
  • SMB3 client (CONFIG_EXPERIMENTAL) expected by Linux kernel 3.8.
  • While implementing Linux client for SMB3, focusing on strengths: clustering, RDMA. Take advantage of great protocol docs

Multiuser CIFS Mounts, Jeff Layton, Red Hat

  • I attended this session, but tweeted just the session title.

How Many IOPS is Enough by Thomas Coughlin, Coughlin Associates

  • 79% of surveyed people said they need between 1K and 1M IOPs. Capacity: from 1GB to 50TB with sweet spot on 500GB.
  • 78% of surveyed people said hardware delivers between 1K and 1M IOPs, with a sweet spot at 100K IOPs. Matches requirements  
  • Minimum latency system hardware (before other bottleneck) ranges between >1sec to <10ns. 35% at 10ms latency.
  • $/GB for SDD and HDD both declining in parallel paths. $/GB roughly follows IOPs.
  • Survey results will be available in October...

SMB 3.0 ( Because 3 > 2 ) - David Kruse, Microsoft

  • Fully packed room to hear David's SMB3 talk. Plus a few standing in the back... pic.twitter.com/TT5mRXiT
  • Time to ponder: When should we recommend disabling SMB1/CIFS by default?

Understanding Hyper-V over SMB 3.0 Through Specific Test Cases with Jose Barreto

  • No tweets during this session. Hard to talk and tweet at the same time :-)

Continuously Available SMB – Observations and Lessons Learned - David Kruse and Mathew George.

  • I attended this session, but tweeted just the session title.

Status of SMB2/SMB3 Development in Samba, Michael Adam, Samba Team

  • SMB 2.0 officially supported in Samba 3.6 (about a year ago, August 2011)
  • SMB 2.1 work done in Samba for Large MTU, multi-credit, dynamic re-authentication
  • Samba 4.0 will be the release to incorporate SMB 3.0 (encryption and secure negotiate already done)

The Solid State Storage (R-)Evolution, Michael Krause, Hewlett-Packard

  • Storage (especially SSD) performance constrained by SAS interconnects
  • Looking at serviceability from DIMM to PCIe to SATA to SAS. Easy to replace x perfor
  • No need to re-invent SCSI. All OS, hypervisors, file systems, PCIe storage support SCSI.
  • Talking SCSI Express. Potential to take advantage of PCIe capabilities.
  • PCIe has benefits but some challenges: Non optimal DMA "caching", non optimal MMIO performance
  • everything in the world of storage is about to radically change in a few years: SATA, SAS, PCIe, Memory
  • Downstream Port Containment. OS informed of async communications lost.
  • OCuLink: new PCIe cable technology
  • Hardware revolution: stacked media, MCM / On-die, DIMM. Main memory in 1 to 10 TB. Everything in memory?
  • Express Bay (SFF 8639 connector), PCIe CEM (MMIO based semantics), yet to be developed modules
  • Media is going to change. $/bit, power, durability, performance vs. persistence. NAND future bleak.
  • Will every memory become persistent memory? Not sic-fi, this could happen in a few years...
  • Revolutionary changes coming in media. New protocols, new hardware, new software. This is only the beginning

Block Storage and Fabric Management Using System Center 2012 Virtual Machine Manager and SMI-S, Madhu Jujare, Microsoft

  • Windows Server 2012 Storage Management APIs are used by VMM 5012. An abstraction of SMI-S APIs.
  • SMAPI Operations: Discovery, Provisioning, Replication, Monitoring, Pass-thru layer
  • Demo of storage discovery and mapping with Virtual Machine Manager 2012.SP1. Using Microsoft iSCSI Target!

Linux Filesystems: Details on Recent Developments in Linux Filesystems and Storage by Chris Mason, Fusion-io

  • Many journaled file systems introduced in Linux 2.4.x in the early 2000s.
  • Linux 2.6.x. Source control at last. Kernel development moved more rapidly. Specially after Git.
  • Backporting to Enterprise. Enterprise kernels are 2-3 years behind mainline. Some distros more than others.
  • Why are there so many filesystems? Why not pick one? Because it's easy and people need specific things.
  • Where Linux is now. Ext4, XFS (great for large files). Btrfs (snapshots, online maintenance). Device Mapper.
  • Where Linux is now. CF (Compact Flash). Block. SCSI (4K, unmap, trim, t10 pi, multipath, Cgroups).
  • NFS. Still THE filesystem for Linux. Revisions introduce new features and complexity. Interoperable.
  • Futures. Atomic writes. Copy offload (block range cloning or new token based standard). Shingled drives (hybrid)
  • Futures. Hinting (tiers, connect blocks, IO priorities). Flash (seems appropriate to end here :-)

Non-volatile Memory in the Storage Hierarchy: Opportunities and Challenges by Dhruva Chakrabarti, HP

  • Will cover a few technologies coming the near future. From disks to flash and beyond...
  • Flash is a huge leap, but NVRAM presents even bigger opportunities.
  • Comparing density/retention/endurance/latency/cost for hdd/sdd (nand flash)/dram/nvram
  • Talking SCM (Storage Class Memory). Access choices: block interface or byte-addressable model.
  • Architectural model for NVRAM. Coexist with DRAM. Buffers/caches still there. Updates may linger...
  • Failure models. Fail-stop. Byzantine. Arbitrary state corruption. Memory protection.
  • Store to memory must be failure-atomic.
  • NVRAM challenges. Keep persistent data consistent. Programming complexity. Models require flexibility.
  • Visibility ordering requirements. Crash can lead to pointers to uninitialized memory, wild pointers.
  • Potential inconsistencies like persistent memory leaks. There are analogs in multi-threading.
  • Insert a cache line flush to ensure visibility in NVRAM. Reminiscent of a disk cache flush.
  • Many flavors of cache flushes. Intended semantics must be honored red. CPU instruction or API?
  • Fence-based programming has not been well accepted. Higher level abstractions? Wrap in transactions?
  • Conclusion. What is the right API for persistent memory? How much effort? What's the implementation cost?

Building Next Generation Cloud Networks for Big Data Applications by Jayshree Ullal, Arista Networks

  • Agenda: Big Data Trends, Data Analytics, Hadoop.
  • 64-bit CPUs trends, Data storage trends. Moore's law is alive and well.
  • Memory hierarchy is not changing. Hard drives not keeping up, but Flash...
  • Moore's law for Big Data, Digital data doubling every 2 years. DAS/NAS/SAN not keeping up.
  • Variety of data. Raw, unstructured. Not enough minds around to deal with all the issues here.
  • Hadoop means the return of DAS. Racks of servers, DAS, flash cache, non-blocking fabric.
  • Hadoop. 3 copies of the data, one in another rack. Protect you main node, single point of failure.
  • Hadoop. Minimum 10Gb. Shift from north-south communications to east-west. Servers talking to each other.
  • From mainframe, to client/server, to Hadoop clusters.
  • Hadoop pitfalls. Not a layer 2 thing. Highly redundant, many paths, routing. Rack locality. Data integrity
  • Hadoop. File transfers in chunks and blocks. Pipelines replication east-west. Map and Reduce.
  • showing sample 2-rack solution. East-west interconnect is very important. Non-blocking. Buffering.
  • Sample conf. 4000 nodes. 48 servers per cabinet. High speed network backbone. Fault tolerant main node
  • Automating cluster provisioning. Script using DHCP for zero touch provisioning.
  • Buffer challenges. Dynamic allocations, survive micro bursts.
  • Advanced diagnostics and management. Visibility to the queue depth and buffering. Graph historical latency.
  • my power is running out. I gotta speak fast. :-)

Windows File and Storage Directions by Surendra Verma, Microsoft

  • Landscape: pooled resources, self-service, elasticity, virtualization, usage-based, highly available
  • Industry-standard parts to build very high scale, performing systems. Greater number of less reliable parts.
  • Services influencing hardware. New technologies to address specific needs. Example: Hadoop.
  • OS storage built to address specific needs. Changing that requires significant effort.
  • You have to assume that disks and other parts will fail. Need to address that in software.
  • If you have 1000 disks in a system, some are always failing, you're always reconstructing.
  • ReFS: new file system in Windows 8, assumes that everything is unreliable underneath.
  • Other relevant features in Windows Server 2012: Storage Spaces, Clustered Shared Volumes, SMB Direct.
  • Storage Spaces provides resiliency to media failures. Mirror (2 or 3 way), parity, hot spares.
  • Shared Storage Spaces. Resiliency to node and path failures using shared SAS disks.
  • Storage Spaces is aware of enclosures, can tolerate failure of an entire enclosure.
  • ReFS provides resiliency to media failures. Never write metadata in place. Integrity streams checksum.
  • integrity Streams. User data checksum, validated on every read. Uses Storage Spaces to find a good copy.
  • You own application can use an API to talk to Storage Spaces, find all copies of the data, correct things.
  • Resiliency to latent media errors. Proactively detect and correct, keeping redundancy levels intact.
  • ReFS can detect/correct corrupted data even for data not frequently read. Do it on a regular basis.
  • What if all copies are lost? ReFS will keep the volume online, you can still read what's not corrupted.
  • example configuration with 4 Windows Server 2012 nodes connected to multiple JBODs.

Hyper-V Storage Performance and Scaling with Joe Dai & Liang Yang, Microsoft

Joe Dai:

  • New option in Windows Server 2012: Virtual Fibre Channel. FC to the guest. Uses NPIV. Live migration just works.
  • New in WS2012: SMB 3.0 support in Hyper-V. Enables Shared Nothing Live Migration, Cross-cluster Live Migration.
  • New in WS 2012: Storage Spaces. Pools, Spaces. Thin provisioning. Resiliency.
  • Clustered PCI RAID. Host hardware RAID in a cluster setup.
  • Improved VHD format used by Hyper-V. VHDX. Format specification at http://www.microsoft.com/en-us/download/details.aspx?id=29681 Currently v0.95. 1.0 soon
  • VHDX: Up to 64TB. Internal log for resiliency. MB aligned. Larger blocks for better perf. Custom metadata support.
  • Comparing performance. Pass thru, fixed, dynamic, differencing. VHDX dynamic ~= VHD fixed ~= physical disk.
  • Offloaded Data Transfers (ODX). Reduces times to merge, mirror and create VHD/VHDX. Also works for IO inside the VM.
  • Hyper-V support for UNMAP. Supported on VHDX, Pass-thru. Supported on VHDX Virtual SCSI, Virtual FC, Virtual IDE.
  • UNMAP in Windows Server 2012 can flow from virtual IDE in VM to VHDX to SMB share to block storage behind share.

Laing Yang:

  • My job is to find storage bottlenecks in Hyper-V storage and hand over to Joe to fix them. :-)
  • Finding scale limits in Hyper-V synthetic SCSI IO path in WS2008R2. 1 VSP thread, 1 VMBus channel per VM, 256 queue depth per
  • WS2012: From 4 VPs per VM to 64 VP per VM. Multi-threaded IO model. 1 channel per 16 VPs. Breaks 1 million IOPs.
  • Huge performance jump in WS2012 Hyper-V. Really close to physical even with high performance storage.
  • Hyper-V Multichannel (not to be confused with SMB Multichannel) enables the jump on performance.
  • Built 1 million IOPs setup for about $10K (excluding server) using SSDs. Demo using IOmeter. Over 1.22M IOPs...

The Virtual Desktop Infrastructure Storage Behaviors and Requirements with Spencer Shepler, Microsoft

  • Storage for Hyper-V in Windows Server 2012: VHDX, NTFS, CSV, SMB 3.0.
  • Review of SMB 3.0 advantages for Hyper-V: active recovery, Multichannel, RDMA.
  • Showing results for SMB Multichannel with four traditional 10GbE. Line rate with 64KB IOs. CPU bound with 8KB.
  • Files used by Hyper-V. XML, BIN, CSV, VHD, VHDX, AVHDX. Gold, diff and snapshot disk relationships.
  • improvements in VHDX. Up to 64TB size. 4KB logical sector size, 1MB alignment for allocations. UNMAP. TRIM.
  • VDI: Personal desktops vs. Pooled desktops. Pros and cons.
  • Test environment. WS2012 servers. Win7 desktops. Login VSI http://www.loginvsi.com - 48 10K rpm HDD.
  • Workload. Copy, word, print pdf, find/replace, zip, outlook e-mail, ppt, browsing, freemind. Realistic!
  • Login VSI fairly complex to setup. Login frequency 30 seconds. Workload started "randomly" after login.
  • Example output from Login VSI. Showing VSI Max.
  • Reading of BIN file during VM restore is sequential. IO size varies.
  • Gold VHDX activity. 77GB over 1 hour. Only reads, 512 bytes to 1MB size IOs. 25KB average. 88% are <=32KB
  • Distribution for all IO. Reads are 90% 64KB or less. Writes mostly 20KB or less.
  • AVHD activity 1/10 read to write ratio. Flush/write is 1/10. Range 512 bytes to 1MB. 90% are 64KB or less.
  • At the end of test run for 1 hour with 85 desktops. 2000 IOPs from all 85 VMs, 2:1 read/write ratio.

SQL Server: Understanding the Data Workload by Gunter Zink, Microsoft (original title did not fit a tweet)

  • Looking at OLTP and data warehousing workloads. What's new in SQL Server 2012.
  • Understanding SQL Server. Store and retrieve structured data, Relation, ACID, using schema.
  • Data organized in tables. tables have columns. Tables stored in 8KB pages. Page size fixed, not configurable.
  • SQL Server Datafile. Header, GAM page (bitmap for 4GB of pages), 4GB of pages, GAM page, 4GB of pages, etc...
  • SQL Server file space allocated in extents. An extent is 8 pages or 64KB. Parameter for larger extent size.
  • SQL Server log file: Hreader, log records (512 bytes to 60KB). Checkpoint markers. truncated after backup.
  • If your storage reports 4KB sector size, minimum log write for SQL Server is 4KB. Records are padded.
  • 2/3 of SQL Servers run OLTP workloads. Many active users, lightweight transactions.
  • Going over what happens when you run OLTP. Read cache or read disk, write log to disk and mark page as dirty
  • Log buffer. Circular buffer, no fixed size. One buffer written to disk, another being filled with changes.
  • If storage is not fast enough, writing log takes longer and buffer changes grows larger.
  • Lazy writer. Writes dirty pages to disk (memory pressure). Checkpoint: Writes pages, marks log file (time limit)
  • Checkpoint modes: Automatic, Indirect, Manual. Write rate reduced if latency reaches 20ms (can be configured)
  • Automatic SQL Checkpoint. Write intensity controlled by recovery interval. Default is 0 = every two minutes.
  • New in SQL Server 2012. Target_Recovery_Time. Makes checkpoint less spikey by constantly writing dirty pages.
  • SQL Server log file. Change records in sequence. Mostly just writes. Except in recovery or transaction rollback.
  • Data file IO. 8KB random reads, buffered (based on number of user queries). Can be done in 64KB at SQL start up.
  • Log file IO: unbuffered small sequential writes (depends on how many inserts/updates/deletes).
  • About 80% of SQL Server performance problems are storage performance problems. Not enough spindles or memory.
  • SQL Server problems. 20ms threshold too high for SSDs. Use -k parameter to limit (specified in MB/sec)
  • Issues. Checkpoint floods array cache (20ms). Cache de-staging causes log drive write performance.
  • Log writes must go to disk, no buffering. Data writes can be buffered, since it can recover from the log.
  • SQL Server and Tiered Storage. We probably won't read what we've just written.
  • Data warehouse. Read large amounts of data, mostly no index, table scans. Hourly or daily updates (from OLTP).
  • Understanding a data warehouse query. Lots of large reads. Table scans and range scans. Reads: 64KB up to 512KB.
  • DW. Uses TempDB to handle intermediate results, sort. Mostly 64KB writes, 8KB reads. SSDs are good for this.
  • DW common problems: Not enough IO bandwidth. 2P server can ingest 10Gbytes/sec. Careful with TP, pooled LUNs.
  • DW common problems. Arrays don't read from multiple mirror copies.
  • SMB file server and SQL Server. Limited support in SQL Server 2008 R2. Fully supported with SQL Server 2012.
  • I got my fastest data warehouse performance using SMB 3.0 with RDMA. Also simpler to manage.
  • Comparing steps to update SQL Server with Fibre Channel and SMB 3.0 (many more steps using FC).
  • SQL Server - FC vs. SMB 3.0 connectivity cost comparison. Comparing $/MB/sec with 1GbE, 10GbE, QDR IB, 8G FC.

The Future of Protocol and SMB2/3 Analysis with Paul Long, Microsoft

  • We'll talk about Message Analyzer. David is helping.
  • Protocol Engineering Framework
  • Like Network Monitor. Modern message analysis tool built on the Protocol Engineering Framework
  • Source for Message Analyzer can be network packets, ETW events, text logs, other sources. Can validate messages.
  • Browse for message sources, Select a subset of messages, View using a viewer like a grid..
  • New way of viewing starting from the top down, instead of the bottom up in NetMon.
  • Unlike NetMon, you can group by any field or message property. Also payload rendering (like JPG)
  • Switching to demo mode...
  • Guidance shipped online. Starting with a the "Capture/Trace" option.
  • Trace scenarios: NDIS, Firewall, Web Proxy, LAN , WLAN, Wifi. Trace filter as well.
  • Doing a link layer capture (just like old NetMon). Start capture. Generate some web traffic.
  • Stop the trace. Group by module. Look at all protocols. Like HTTP. Drill in to see operations.
  • Looking at operations. HTTP GET. Look at the details. High level stack view.
  • Now grouping on both protocol and content type. Easily spots pictures over HTTP. Image preview.
  • Easier to see time elapsed per operation when you group messages. You dig to individual messages
  • Now looking at SMB trace. Trace of a file copy. Group on the file name (search for the property)
  • Now grouped on SMB.Filename. You can see all SB operations to copy a specific file.
  • Now looking at a trace of SMB file copy to an encrypted file share.
  • Built in traces to capture from the client side or server side. Can do full PDU or header.only
  • This can also be used to capture SMB Direct data, using the SMB client trace.
  • Showing the trace now with both network traffic and SMB client trace data (unencrypted).
  • Want to associate the wire capture with the SMB client ETW trace? Use the message ID
  • Showing mix of firewall trace and SMB client ETW trace. You see it both encrypted and not.
  • SMB team at Microsoft is the first to add native protocol unit tracing. Very useful...
  • Most providers have ETW debug logging but not the actual messages.
  • You can also get the trace with just NetSh or LogMan and load the trace in the tool later.
  • We also can deploy the tool and use PowerShell to start/stop capture.
  • If the event provider offers them, you can specify level and keywords during the capture.
  • Add some files (log file and wireshark trace). Narrow down the time. Add selection filter.
  • Mixing wireshark trace with a Samba text log file (pattern matching text log).
  • Audience: As a Samba hacker, Message Analyzer is one of the most interesting tools I have seen!
  • Jaws are dropping as Paul demos analyzing a trace from WireShark + Samba taken on Linux.
  • Next demo: visualizations. Two separate file copies. Showing summary view for SMB reads/writes
  • Looking at a graph of bytes/second for SMB reads and writes. Zooming into a specific time.
  • From any viewer you should be any to do any kind of selection and then launch another viewer.
  • If you're a developer, you can create a very sophisticated viewer.
  • Next demo: showing the protocol dashboard viewer. Charts with protocol bars. Drills into HTTP.

Storage Systems for Shingled Disks, with Garth Gibson, Panasas

  • Talking about disk technology. Reaction of HDD to what's going with SSDs.
  • Kryder's law for magnetic disks. Expectation is that disks will cost next to nothing.
  • High capacity disk. As bits get smaller, the bit might not hold it's orientation 10 years later.
  • Heat assisted to make it possible to write, then keep it longer when cold. Need to aim that laser precisely..
  • New technology. Shingled writing. Write head is wider than read head. Density defined by read head, not write head.
  • As you write, you overwrite a portion of what you wrote before, but you can still read it.
  • Shingled can be done with today's heads with minor changes, no need to wait for heat assisted technology.
  • Shingled disks. Large sequential writes. Disks becomes tape!!
  • Hard to see just the one bit. Safe plan is to see the bit from slightly different angles and use signal processing.
  • if aiming at 3x the density: cross talk. Signal processing using 2 dimensions TMDR. 3-5 revs to to read a track.
  • Shingled disks. Initial multiplier will be a factor of 2. Seek 10nm instead of 30 nm. Wider band with sharp edges.
  • Write head edge needs to be sharp on one side, where the tracks will overlap. Looking at different widths.
  • Aerial density favors large bands that overlap. Looking at some math that proves this.
  • You could have a special place in the disk with no shingles for good random write performance, mixed with shingled.
  • Lots of question on shingled disks. How to handle performance, errors, etc.
  • Shingled disks. Same problem for Flash. Shingled disks - same algorithms as Flash.
  • modify software to avoid or minimize read, modify, write. Log structured file systems are 20 years old.
  • Key idea is that disk attribute says "sequential writing". T13 and t10 standards.
  • Shingled disks. Hadoop as initial target. Project with mix of shingled and unshingled disks. Could also be SSD+HDD.
  • Prototype banded disk API. Write forward or move back to 0. Showing test results with new file system.
  • future work. Move beyond hadoop to general workloads, hurts with lots of small files. Large files ok.
  • future work. Pack metadata. All of the metadata into tables, backed on disk by large blob of changes.
  • Summary of status. Appropriate for Big Data. One file = one band. Hadoop is write once. Next steps: pack metadata.

The Big Deal of Big Data to Big Storage with Benjamin Woo, Neuralytix

  • Can't project to both screens because laptop does not have VGA. Ah, technology... Will use just right screen.
  • Even Batman is into big data. ?!
  • What's the big picture for big data. Eye chart with lots of companies, grouped into areas...
  • We have a problem with storage/data processing today. Way too many hops. (comparing to airline routes ?!)
  • Sample path: Oracle to Informatica to Microstategy and Hadoop. Bring them together. Single copy of "the truth".
  • Eliminate the process of ETL. Eliminate the need for exports. Help customers to find stuff in the single copy.
  • You are developers. You need to find a solution for this problem. Do you buy into this?
  • Multiple copies OK for redundancy or performance, but shouldn't it all be same source of truth?
  • Single copy of the truth better for discovery. Don't sample, don't summarize. You will find more than you expect.
  • We're always thinking about the infrastructure. Remove yourself from the hardware and think about the data!
  • The challenge is how to think about the data. Storage developers can map that to the hardware.
  • Send complaints to /dev/null. Tweet at @BenWooNY
  • Should we drop RDBMS altogether? Should we add more metadata to them? Maybe.
  • Our abstractions are already far removed from the hardware. Think virtual disks in VM to file system to SAN array.
  • Software Defined Storage is something we've been doing for years in silicon.
  • Remember what we're here for. It's about the data. Otherwise there is no point in doing storage.
  • Is there more complexity in having a single copy of the truth? Yes, but that is part of what we do! We thrive there!
  • Think about Hadoop. They take on all the complexity and use dumb hardware. That's how they create value!

Unified Storage for the Private Cloud with Dennis Chapman, NetApp

  • 10th anniversary of SMI-S. Also 10th anniversary of pirate day. Arghhh...
  • application silos to virtualization to private clouds (plus public and hybrid clouds)
  • Focusing on the network. Fundamentally clients talking to storage in some way...
  • storage choices for physical servers. Local (DAS) and remote (FC, iSCSI, SMB). Local for OS, remote for data.
  • Linux pretty much the same as Windows. Difference is NFS instead of SMB. Talking storage affinities.
  • Windows OS. Limited booting from iSCSI and FC. Mostly local.
  • Windows. Data mostly on FC and iSCSI, SMB still limited (NFS more well established on Linux).
  • shifting to virtualized workloads on Windows. Opts for local and remote. More choices, storage to the guest.
  • Virtualized workloads are the #1 configuration we provide storage for.
  • Looking at options for Windows and Linux guests, hosted on both VMware and Hyper-V hosts. Table shows options
  • FC to the guest. Primary on Linux, secondary on Windows. Jose: FC to the guest new in WS2012.
  • File storage (NFS) primary on Linux, but secondary on Windows (SMB). Jose: again, SMB support new in WS2012.
  • iSCSI secondary for Linux guest, but primary for Windows guests.
  • SMB still limited right now, expect it to grow. Interested on how it will play, maybe as high as NFS on Linux
  • Distributed workload state. Workload domain, hypervisors domain, storage domain.
  • Guest point in time consistency. Crash consistency or application consistency. OS easier, applications harder
  • Hibernation consistency. Put the guest to sleep and snapshot. Works well for Windows VMs. Costs time.
  • Application consistency. Specific APIs. VSS for Windows. I love this! Including remote VSS for SMB shares.
  • Application consistency for Linux. Missing VSS. We have to do specific things to make it work. Not easy.
  • hypervisors PIT consistency. VMware, cluster file system VMFS. Can store files on NFS as well.
  • Hypervisors PIT for Hyper-V. Similar choices with VHD on CSV. Also now option for SMB in WS2012.
  • Affinities and consistency. Workload domain, Hypervisors domain and Storage domain backups. Choices.
  • VSS is the major difference between Windows and Linux in terms of backup and consistency.
  • Moving to the Storage domain. Data ONTAP 8 Clustering. Showing 6-node filer cluster diagram.
  • NetApp Vservers owns a set of Flexvols, with contain close objects (either LUN or file).
  • Sample workflow with NetApp with remote SMB storage. Using remote VSS to create a backup using clones.
  • Sample workflow. App consistent backup from a guest using an iSCSI LUN.
  • Showing eye charts with integration with VMware and Microsoft.
  • Talking up the use of PowerShell, SMB when integrating with Microsoft.
  • Talk multiple protocols, rich services, deep management integration, highly available and reliable.

SNIA SSSI PCIe SSD Round Table. Moderator + four members.

  • Introductions, overview of SSSI PCIe task force and committee.
  • 62 companies in the last conference. Presentations available for download. http://www.snia.org/forums/sssi/pcie
  • Covering standards, sites and tools available from the group. See link posted
  • difference between PCIE SSDs look just other drives, but there are differences. Bandwidth is one of them.
  • Looking at random 4KB write IOPs and response time for different types of disks: HDD, MLC, SLC, PCIe.
  • Different SSD tech offer similar response rates. Some high latencies due to garbage collection.
  • comparing now DRAM, PCIe, SAS and SATA. Lower latencies in first two.
  • Comparing CPU utilization. From less than 10% to over 50%. What CPU utilization to achieve IOPs...
  • Other system factors. Looking at CPU affinity effect on random 4KB writes... Wide variation.
  • Performance measurement. Response time is key when testing PCIe SSDs. Power mgmt? Heat mgmt? Protocol effect on perf?
  • Extending the SCSI platform for performance. SCSI is everywhere in storage.
  • Looking at server attached SSDs and how much is SATA, SAS, PCIe, boot drive. Power envelope is a consideration.
  • SCSI is everywhere. SCSI Express protocol for standard path to PCIe. SoP (SCSI over PCIe). Hardware and software.
  • SCSI Express: Controllers, Drive/Device, Drivers. Express bay connector. 25 watts of power.
  • Future: 12Gbps SAS in volume at the end of 2013. Extended copy feature. 25W devices. Atomic writes. Hinting. SCSI Express.
  • SAS controllers > 1 million IOPs and increased power for SAS. Reduces PCIe SSD differentiation. New form factors?
  • Flash drives: block storage or memory.
  • Block versus Memory access. Storage SSDs, PCIe SSDs, memory class SCM compared in a block diagram. Looking at app performance
  • optimization required for apps to realize the memory class benefits. Looking at ways to address this.
  • Open industry directions. Make all storage look like SCSI or offer apps other access models for storage?
  • Mapping NVMExpress capability to SCSI commands. User-level abstractions. Enabling SCM by making it easy.
  • Panel done with introductions. Moving to questions.
  • How is Linux support for this? NVMExpress driver is all that exists now.
  • How much of the latency is owned by the host and the PCIe device? Difficult to answer. Hardware, transport, driver.
  • Comparing to DRAM was excellent. That was very helpful.
  • How are form factors moving forward? 2.5" HDD format will be around for a long time. Serviceability.
  • Memory like access semantics - advantages over SSDs. Lower overhead, lots in the hardware.
  • Difference between NVMe and SOP/PQI? Capabilities voted down due to complexity.
  • What are the abstractions like? Something like a file? NVMe has a namespace. Atomic write is a good example. How to overlay?
  • It's easy to use just a malloc, but it's a cut the block, run with memory. However, how do you transition?

NAS Management using System Center 2012 Virtual Machine Manager and SMI-S with Alex Naparu and Madhu Jujare

  • VMM for Management of Virtualized Infrastructure: VMM 2012 SP1 covers block storage and SMB3 shares
  • Lots of SMB 3.0 sessions here at SDC...
  • VMM offers to manage your infrastructure. We'll be focusing on storage. Lots enabled by Windows Server 2012.
  • There's an entire layer in Windows Server 2012 dedicated to manage storage. Includes translation of WMI to SMI-S
  • All of this can be leveraged using PowerShell.
  • VMM NAS Management: Discovery (Servers, Systems, Shares), Creation/Removal (Systems, Shares), Share Permissions
  • How did we get there? With a lot of help from our partners. Kick-off with EMC and NetApp. More soon. Plugfests.
  • Pre-release providers. If you have any questions on the availability of providers, please ask EMC and NetApp.
  • Moving now into demo mode. Select provider type. Specify discovery scope. Provide credentials. Discovering...
  • Discovered some block storage and file storage. Some providers expose one of them, some expose both.
  • Looking at all the pools and all the shares. Shallow discovery at first. After selection, we do deep discovery.
  • Each pool is given a tag, called classification. Tagged some as Gold, some as Platinum. Finishing discovery.
  • Deep discovery completed. Looking at the Storage tree in VMM, with arrays, pools, LUNs, file shares.
  • Now using VMM to create a file share. Provide a name, description, file server, storage pool and size.
  • Creates a logical disk in the pool, format with a file system, then create a file share. All automated.
  • Now going to a Hyper-V host, add a file share to the host using VMM. Sets appropriate permissions for the share.
  • VMM also checks the file access is good from that host.
  • Now let's see how that works for Windows. Add a provider, abstracted. Using WMI, not SMI-S. Need credentials.
  • Again, shows all shares, select for deep discovery. Full management available after that.
  • Now we can assign Windows file share to the host, ACLs are set. Create a share. All very much the same as NAS.
  • VMM also verifies the right permissions are set. VMM can also repair permission to the share if necessary.
  • Now using VMM to create a new VM on the Windows SMB 3.0 file share. Same as NAS device with SMB 3.0.
  • SMI-S support. Basic operations supported on SMI-S 1.4 and later. ACL management. requires SMI-S 1.6.
  • SMI-S 1.4 profiles: File server, file share, file system discovery, file share creation, file share removal.
  • Listing profiles that as required for SMI-S support with VMM. Partial list: NAS Head, File System, File Export
  • SMI-S defines a number of namespaces. "Interop" namespace required. Associations are critical.
  • Details on Discovery. namespaces, protocol support. Filter to get only SMB 3.0 shares.
  • Discovery of File Systems. Reside on logical disks. That's the tie from file storage to block storage.
  • Different vendors have different way to handle File Systems. Creating a new one is not trivial. Another profile.
  • VMM creates the file system and file share in one step. Root of FS is the share. Keeping things simple.
  • Permissions management. Integrated with Active Directory. Shares "registered" with Hyper-V host. VMM adds ACLs.
  • Demo of VMM specific PowerShell walking the hierarchy from the array to the share and back.
  • For VMM, NAS device and SMI-S must be integrated with Active Directory. Simple Identity Management Subprofile.
  • CIM Passthrough API. WMI provider can be leveraged via code or PowerShell.

SMB 3, Hyper-V and ONTAP, Garrett Mueller, NetApp

  • Senior Engineer at NetApp focused on CIFS/SMB.
  • What we've done with over 30 developers: features, content for Windows Server 2012. SMB3, Witness, others.
  • Data ONTAP cluster-mode architecture. HA pairs with high speed interconnect. disk "blade" in each node.
  • Single SMB server spread across multiple nodes in the cluster. Each an SMB server with same configuration
  • Each instance of the SMB server in a node has access to the volumes.
  • Non-disruptive operations. Volume move (SMB1+). Logical Interface move (SMB2+). Move node/aggregate (SMB3).
  • We did not have a method to preserve the locks between nodes. That was disruptive before SMB3.
  • SMB 3 and Persistent Handles. Showing two nodes and how you can move a persistent SMB 3 handle.
  • Witness can be used in lots of different ways. Completely separate protocol. NetApp scoped it to an HA pair.
  • Diagram explaining how NetApp uses Witness protocol with SMB3 to discover, monitor, report failure.
  • Remote VSS. VSS is Microsoft's solution for app consistent snapshot. You need to back up your shares!
  • NetApp implemented a provider for Remote VSS for SMB shares using the documented protocol. Showing workflow.
  • All VMs within a share are SIS cloned. SnapManager does backup. After done, temp SIS clones are removed.
  • Can a fault occur during a backup. If there is a failure, the backup will fail. Not protected in that way.
  • Offloaded Data Transfer (ODX). Intra-volume: SIS clones. Inter-volume/inter-node: back-end copy engine.
  • ODX: The real benefit is in the fact that it's used by default in Windows Server 2012. It just works!
  • ODX implications for Hyper-V over SMB: Rapid provisioning, rapid storage migrations, even disk within a VM.
  • Hyper-V over SMB. Putting it all together. Non-disruptive operations, Witness, Remote VSS, ODX.
  • No NetApp support for SMB Multichannel or SMB Direct (RDMA) with SMB 3.

Design and Implementation of SMB Locking in a Clustered File System with Aravind Velamur Srinivasan, EMC - Isilon

  • Part of SMB team at EMC/Isilon. Talk agenda covers OneFS and its distributed locking mechanism.
  • Overview of OneFS. NAS file server, scalable, 8x mirror, +4 parity. 3 to 144 nodes, using commodity hardware.
  • Locking: avoid multiple writers to the same file. Potentially in different file server nodes.
  • DLM challenges: Performance, multiple protocols ands requirements. Expose appropriate APIs.
  • Diagram explaining the goals and mechanism of the Distributed Locking Manager (DLM) Isilon's OneFS
  • Going over requirements of the DLM. Long list...

Scaling Storage to the Cloud and Beyond with Ceph with Sage Weil, Inktank

  • Trying to catch up with ongoing talk on ceph. Sage Weil talks really fast and uses dense slides...
  • Covering RADOS block device being used by virtualization, shared storage. http://ceph.com/category/rados/
  • Covering ceph-fs. Metadata and data paths. Metadata server components. Combined with the object store for data.
  • Legacy metadata storage: bad. Ceph-fs metadata does not use block lists or inode tables. Inode in directory.
  • Dynamic subtree partitioning very scalable. Hundreds of metadata servers. Adaptive. Preserves locality.
  • Challenge dealing metadata Io. Use metadata server as cache, prefect dir-inode. Large journal or log.
  • What is journaled? Lots of state. Sessions, metadata changes. Lazy flush.
  • Client protocol highly stateful. Metadata servers, direct access to IDS.
  • explaining the ceph-fs workflow using ceph-mon, ceph-mds, ceph-osd.
  • Snapshots. Volume and subvolume unusable at petabyte scale. Snapshot arbitrary directory
  • client implementations. Linux kernel client. Use Samba to reexport as CIFS. Also NFS and Hadoop.
  • Current status of the project: most components: status=awesome. Ceph-fs nearly awesome :-)
  • Why do it? Limited options for scalable open source storage. Proprietary solutions expensive.
  • What to do with hard links? They are rare. Using auxiliary table, a little more expensive, but works.
  • How do you deal with running out of space? You don't. Make sure utilization on nodes balanced. Add nodes.

Introduction to the last day

  • Big Data is like crude oil, it needs a lot of refining and filtering...
  • Growing from 2.75 Zettabytes in 2012 to 8 ZB in 2015. Nice infographic showing projected growth...

The Evolving Apache Hadoop Eco System - What It Means for Big Data and Storage Developers, Sanjay Radia, Hortonworks

  • One of the surprising things about Hadoop is that is does not RAID on the disks. It does surprise people.
  • Data is growing. Lots of companies developing custom solutions since nothing commercial could handle the volume.
  • web logs with terabytes of data. Video data is huge, sensors. Big Data = transactions + interactions + observations.
  • Hadoop is commodity servers, jbod disk, horizontal scaling. Scale from small to clusters of thousands of servers..
  • Large table with use cases for Hadoop. Retail, intelligence, finance, ...
  • Going over classic processes with ETL, BI, Analytics. A single system cannot process huge amounts of data.
  • Big change is introducing a "big data refinery". But you need a platform that scales. That's why we need Hadoop.
  • Hadoop can use a SQL engine, or you can do key-value store, NoSQL. Big diagram with Enterprise data architecture.
  • Hadoop offers a lot tools. Flexible metadata services across tools. Helps with the integration, format changes.
  • Moving to Hadoop and Storage. Looking at diagram showing racks, servers, 6k nodes, 120PB. Fault tolerant, disk or node
  • manageability. One operator managing 3000 nodes! Same boxes do both storage and computation.
  • Hadoop uses very high bandwidth. Ethernet or InfiniBand. Commonly uses 40GbE.
  • Namespace layer and Block storage layer. Block pool Isis a set of blocks, like a LUN. Did/file abstraction on namesp.
  • Data is normally accessed locally, but can pull from any other servers. Deals with failures automatically.
  • looking at HDFS. Goes back to 1978 paper on separating data from function in a DFS. Luster, Google, pNFS.
  • I attribute the use of commodity hardware and replication to the GoogleFS. Circa 2003. Non-posix semantics.
  • Computation close to data is an old model. Map Reduce.
  • Significance of not using disk RAID. Replication factor of Hadoop is 3. Node can be fixed when convenient.
  • HDFS recovers at a rate of 12GB in minutes, done in parallel. Even faster for larger clusters. Recovers automatically.
  • Clearly there is an overhead. It's 3x instead of much less for RAID. Used only for some of the data.
  • Generic storage service opportunities for innovation. Federation, partitioned namespace, independent block pools.
  • Archival data. Where should it sit? Hadoop encourages keeping old data for future analysis. Hot/ cold? Tiers? Tape?
  • Two versions of Hadoop. Hadoop 1 (GA) and Hadoop 2 (alpha). One is stable. Full stack HA work in progress.
  • Hadoop full stack HA architecture diagram. Slave nodes layer + HA Cluster layer. Improving performance, DR, upgrades.
  • upcoming features include snapshots, heterogeneous storage (flash drives), block grouping, other protocols (NFS).
  • Which Apache Hadoop distro should you use? Little marketing of Hortonworks. Most stable version of components.
  • It's a new product. At yahoo we needed to make sure we did not lose any data. Needs it to be stable.
  • Hadoop changes the game. Cost, storage and compute. Scales to very very large. Open, growing ecosystem, no lock in.
  • Question from the audience. What is Big Data? What is Hadoop? You don't' need to know what it is, just buy it :-)
  • Sizing? The CPU performance and disk performance/capacity varies a lot. 90% of disk performance for sequential IO.
  • Question: Security? Uses Kerberos authentication, you can conned to Active Directory. There is a paper on this.
  • 1 name node to thousands of nodes, 200M files. Hadoop moving to more name nodes to match the capacity of working set.

Primary Data Deduplication in Windows Server 2012 with Sudipta Sengupta, Jim Benton

Sudipta Sengupta:

  • Growing file storage market. Dedup is the #1 feature customers asking for. Lots of acquisitions in dedup space.
  • What is deduplication, how to do it. Content based chucking using a sliding window, computing hashes. Rabin method.
  • Dedup for data at rst, data on the wire. Savings in your primary storage more valuable, more expensive disks...
  • Dimensions of the problem: Primary storage, locality, service data to components, commodity hardware.
  • Extending the envelope from backup scenarios only to primary deduplication.
  • Key design decisions: post-processing, granularity and chucking, scale slowly to data size, crash consistent
  • Large scale study of primary datasets. Table with different workloads, chunking.
  • Looking at whole-file vs. sub-file. Decided early on to do chunking. Looking at chunk size. Compress the chunks!
  • Compression is more efficient on larger chunk sizes. Decided to use larger chunk size, pays off in metadata size.
  • You don't want to compress unless there's a bang for the buck. 50% of chunks = 80% for compression savings.
  • Basic version of the Rabin fingerprinting based chunking. Large chunks, but more uniform chunk size distribution
  • In Windows average chunk size is 64KB. Jose: Really noticing this guy is in research :-) Math, diagrams, statistics
  • Chunk indexing problem. Metadata too big to fit in RAM. Solution via unique chunk index architecture. Locality.
  • Index very frugal on both memory usage and IOPs. 6 bytes of RAM per chunk. Data partitioning and reconciliation.

Jim Benton:

  • Windows approach to data consistency and integrity. Mandatory block diagram with deduplication components.
  • Looking at deduplication on-disk structures. Identify duplicate data (chunks), optimize target files (stream map)
  • Chunk store file layout. Data container files: chunks and stream maps. Chunk ID has enough data to locate chunk
  • Look at the rehydration process. How to get the file back from the steam map and chunks.
  • Deduplicated file write path partial recall. Recall bitmap allows serving IO from file stream or chunk store.
  • Crash consistency state diagram. One example with partial recall. Generated a lot of these diagrams for confidence.
  • Used state diagrams to allow test team to induce failures and verify deduplication is indeed crash consistent.
  • Data scrubbing. Induce redundancy back in, but strategically. Popular chunks get more copies. Checksum verified.
  • Data scrubbing approach: Detection, containment, resiliency, scrubbing, repair, reporting. Lots of defensive code!
  • Deduplication uses Storage Spaces redundancy. Can use that level to recover the data from another copy if possible.
  • Performance for deduplication. Looking at a table with impact of dedup. Looking at options using less/more memory.
  • Looking at resource utilization for dedup. Focus on converging them.
  • Dedup performance varies depending on data access pattern. Time to open office file, almost no difference.
  • Dedup. Time to copy large VHD file. Lots of common chunks. Actually reduces copy time for those VHD files. Caching.
  • dedup write performance. Crash consistency hurts performance, so there is a hit. In a scenario, around 30% slower.
  • Deduplication around the top features in Windows Server 2012. Mentions at The Register, Ars Technica, Windows IT Pro
  • Lots of great questions being asked. Could not capture it all.

High Performance File Serving with SMB3 and RDMA via the SMBDirect Protocol with Tom Talpey and Greg Kramer

Tom Talpey:

  • Where we are with SMB Direct, where we are going, some pretty cool performance results.
  • Last year here at SDC we had our coming out party for SMB Direct. Review of what's SMB Direct.
  • Nice palindromic port for SMB direct 5455. Protocol documented at MS-SMBD. http://msdn.microsoft.com/en-us/library/hh536346(v=PROT.13).aspx
  • Covering the basic of SMB Direct. Only 3 message types. 2 way full duplex. Discovered via SMB Multichannel.
  • Relationship with the NDKPI in Windows. Provider interface implemented by adapter vendors.
  • Send/receive model. Possibly sent as train. Implements crediting. Direct placement (read/write). Scatter/gather list
  • Going over the details on SMB Direct send transfers. Reads and writes, how they map to SMB3. Looking at read transfer
  • looking at exactly how the RDMA reads and writes work. Actual offloaded transfers via RDMA. Also covering credits.
  • Just noticed we have a relatively packed room for such a technical talk...And it's the larger room here...
  • interesting corner cases for crediting. Last credit case. Async, cancels and errors. No reply, many/large replies
  • SMB Direct efficiency. Two pipes, one per direction, independent. Truly bidirectional. Server pull model. Options.
  • SMB Direct options for RDMA efficiency. FRMR, silent completions, coalescing, etc.
  • Server pull model allows for added efficiency, in addition to improved security. Server controls all RDMA operations.

Greg Kramer:

  • On the main event. That's why you're here, right? Performance...
  • SDC 2011 results. 160k iops, 3.2 GBytes/sec.
  • New SDC 2012 results. Dual CX3 InfiniBand, Storage Spaces, two SAS HBAs, SSDs. SQLIO tool.
  • Examining the results. 7.3 Gbytes / sec with 512KB IOs at 8.6% CPU. 453K 8KB IOs at 60% CPU.
  • Taking it 11. Three InfiniBand links. Six SAS HBAs. 48 SSDs. 16.253 GBytes/sec!!! Still low CPU utilization...
  • NUMA effects on performance. Looking at NUMA disabled versus enabled. 16% percent in CPU utilization.
  • That's great! Now what? Looking at potential techniques to reduce the cost of IOs, increase IOPs further.
  • Looking at improving how invalidation consumes CPU cycles, RNIC bus cycles. But you do need to invalidate agressively
  • Make invalidate cheaper. Using "send with invalidate". Invalidate done as early as possible, fewer round trips.
  • Send with invalidate: supported in InfiniBand, iwarp and roce. No changes to SMB direct protocol. Not committed plan
  • Shout out to http://smb3.info  Thanks, Greg!
  • Question: RDMA and encryption? Yes, you can combine them. SMB Direct will use RDMA send recives in that case.
  • Question: How do you monitor at packet level? Use Message Analyzer. But careful drinking from the fire hose :-)
  • Question: Performance monitor? There are counters for RDMA, look out for stalls, hints on how to optimize.

SMB 3.0 Application End-to-End Performance with Dan Lovinger

  • Product is released now, unlike last year. We're now showing final results...
  • Scenarios with OLTP database, cluster motion, Multichannel. How we found issues during development.
  • Summary statistics. You can drown on river with an average depth of six inches.
  • Starting point: Metric versus time. Averages are not enough, completely miss what's going on.
  • You should think about distribution. Looking at histogram. The classic Bell Curve. 34% to each side.
  • Standard deviation and median. Mid point of all data points. What makes sense for latency, bandwidth?
  • Looking at percentiles. Cumulative distributions. Remember that from College?
  • OLTP workload. Transaction rate, cumulative distribution. How we found and solved an issue that makes SMB ~= DAS
  • OLTP. Log file is small to midsize sequential IO, database file is small random IO.
  • Found 18-year-old perfor bug that affects only SMB and only in an OLTP workload. Leftover from FAT implementation.
  • Found this "write bubble" performance bug look at average queue length. Once fixed, SMB =~ DAS.
  • back to OLTP hardware configuration. IOPs limited workload does not need fast interconnect.
  • Comparing SMB v. DAS transaction rate at ingest. 1GbE over SMB compared to 4GbFC. Obviously limited by bandwidth.
  • As soon as the ingest phase is done, then 1GbE is nearly identical to 4GbFC. IOPs limited on disks. SMB=~DAS.
  • This is just a sample of why workload matters, why we need these performance analysis to find what we can improve.
  • IOmeter and SQLIO are not enough. You need to look at a real workload to find these performance issues.
  • Fix for this issue in Windows Server 2012 and also back ported to Windows Server 2008 R2.
  • Another case: Looking at what happens when you move a cluster resource group from one node to another.
  • 3 file server cluster groups, 40 disk on each. How resource control manager handles the move. Needed visualization.
  • Looking at a neat visualization of how cluster disks are moved from one node to another. Long pole operations.
  • Found that every time we offline a disk, there as a long running operation that was not needed. We fixed that.
  • We also found a situation that took multiple TCP timeouts, leading to long delay in the overall move. Fixed!
  • Final result, dramatic reduction of cluster move time. Entire move time from 55 seconds to under 10 seconds.
  • Now we can do large cluster resource group moves with 120 disks in under 10 seconds. Not bad...
  • Last case study. SMB Multichannel performance. Looking at test hardware configuration. 24 SSDs, 2 SAS HBAs, IOmeter
  • Looking at local throughput at different IO sizes, as a baseline.
  • SMB Multichannel. We can achieve line rate saturation at about 16KB with four 40GBE interfaces.
  • Curve for small IOs matches between DAS and SMB at line rate..

Closing tweet

  • #SDConference is finished. Thanks for a great event! Meet you at SNW Fall 2012 in a month, right back here. On my way back to Redmond now...

Windows Server 2012, File Servers and SMB 3.0 – Simpler and Easier by Design

$
0
0

1. Introduction

I have been presenting our solution in Windows Server 2012 for File Storage for Virtualization (Hyper-V over SMB) and Databases (SQL Server over SMB) for a while now. I always start the conversation talking about how simple and easy it is for an IT Administrator or an Application Developer to use SMB 3.0.

However, I am frequently challenged to detail exactly what that means. After all, both “simple” and “easy” are fairly subjective concepts. In this blog post, I will enumerate some design decisions regarding the SMB 3.0 protocol and its implementation in Windows to make this more concrete.

Please note that, while the topic here is simplicity, I will essentially be going into details behind the implementation, so this could get a little intricate. Not to scare anyone, but if you’re just getting started with SMB, you should probably start with a lighter topic. Or proceed at your own risk :-).

Here’s a summary of the items on this post:

    • 1. Introduction
    • 2. SMB Transparent Failover
      • 2.1. SMB Transparent Failover – No application failures, no application changes
      • 2.2. SMB Transparent Failover – Continuous Availability turned on by default
      • 2.3. SMB Transparent Failover – Witness Service
    • 3. SMB Scale-Out
      • 3.1. SMB Scale-Out – Volumes, not drive letters
      • 3.2. SMB Scale-Out – Single name, dynamic
      • 3.3. SMB Scale-Out – Using node IP addresses
    • 4. SMB Multichannel
      • 4.1. SMB Multichannel – Auto-discovery
      • 4.2. SMB Multichannel – Transparent failover
      • 4.3. SMB Multichannel – Interface arrival
      • 4.4. SMB Multichannel – Link-local
    • 5. SMB Direct
      • 5.1. SMB Direct – Discovery over TCP/IP
      • 5.2. SMB Direct – Fail back to TCP/IP
    • 6. VSS for SMB File Share
      • 6.1. VSS for SMB File Share – Same model as block VSS
      • 6.2. VSS for SMB File Shares – Stream snapshots from file server
    • 7. SMB Encryption
      • 7.1. SMB Encryption – No PKI or certificates required
      • 7.2. SMB Encryption – Hardware acceleration
    • 8. Server Manager
      • 8.1. Server Manager – Simple Wizards
      • 8.2. Server Manager – Fewer knobs
    • 9. SMB PowerShell
      • 9.1. SMB PowerShell – Permissions for a New Share
      • 9.2. SMB PowerShell – Permissions on the Folder
      • 9.3. SMB PowerShell – Cluster type based on disk type
      • 9.4. SMB PowerShell – Shares, Sessions and Open Files across Scale-Out nodes
      • 9.5. SMB PowerShell – Connection
    • 10. Conclusion

2. SMB Transparent Failover

SMB Transparent Failover is a feature that allows an SMB client to continue to work uninterrupted when there’s a failure in the SMB file server cluster node that it’s using. This includes preserving information on the server side plus allowing the client to automatically reconnect to the same share and files on a surviving file server cluster node.

More information: http://blogs.technet.com/b/clausjor/archive/2012/06/07/smb-transparent-failover-making-file-shares-continuously-available.aspx

2.1. SMB Transparent Failover – No application failures, no application changes

Over the years, we have improved SMB to make the protocol more fault tolerant. We’ve been taking some concrete steps that direction, like introducing SMB Durability in SMB 2.0 (if you lose a connection, SMB will attempt to reconnect, but without guarantees) and SMB Resiliency in SMB 2.1 (guaranteed re-connection, but application need to be changed to take advantage of it).

With SMB 3.0, we can now finally guarantee a transparent failover without any application changes through the use of SMB Persistence (also known as Continuously Available file shares). This final step gives us the ability to simply create a share that supports persistency (Continuously Available share) with the application accessing that share automatically enjoying the benefit, without any changes to the way files are opened and no perceived errors that the application needs to handle.

2.2. SMB Transparent Failover – Continuous Availability turned on by default

Once we had the option to create Continuously Available file shares, we considered the option to make it a default setting or leaving it up to the administrator to set it. In the end, we made Continuous Availability the default for any share in a file server cluster. In order to feel comfortable with that decision, we worked through all the different scenarios to make sure we would not break any existing functionality.

clip_image001

Expert Mode: There is a PowerShell setting to turn off Continuous Availability. You can also use PowerShell to tweak the default time that SMB 3.0 waits for a reconnection, which defaults to 60 seconds. These parameters are available for both New-SmbShare and Set-SmbShare. See http://blogs.technet.com/b/josebda/archive/2012/06/27/the-basics-of-smb-powershell-a-feature-of-windows-server-2012-and-smb-3-0.aspx

2.3. SMB Transparent Failover – Witness Service

Another problem we had with the Clustered File Server failover was TCP/IP timeouts. If the file server cluster node you were talking to had a hardware issue (say, someone pulled the plug on it), the SMB client would need to wait for the TCP/IP timeout. This was really bad if we had a requirement to make the failover process automatic and fast. To make that work better, we created a new (optional) protocol and service to simply detect these situations and act faster.

In Windows Server 2012, the client connect to one cluster node for file access and to another cluster node for witness service. The witness service is automatically loaded on the server when necessary and the client is always ready for it, with no additional configuration required on either client or server. Another key aspect was making the client find a witness when connecting to a cluster, if available, without any manual steps. Also, there’s logic in place to find a new witness if the one chosen by the client for some reason is no longer there. Witness proved out to be a key component, which is also used for coordinating moves to rebalance a Scale-Out cluster.

Regarding simplicity, no additional IT Administrator action is required on the file server cluster to enable the Witness service. No client configuration is required either.

3. SMB Scale-Out

SMB Scale-out is the new ability in SMB 3.0 in cluster configuration to show a share in all nodes of a cluster. This active/active configuration makes it possible to scale file server clusters further, without a complex configuration with multiple volumes, shares and cluster resources.

clip_image002

Expert mode: There are specific capabilities that cannot be combined with the Scale-Out file servers, like DFS Replication, File Server quotas and a few others, as outlined in the screenshot above. If you have workloads that require those abilities, you will need to deploy Classic File Servers (called File Server for general use in the screenshot above) to handle them. The Hyper-V over SMB and SQL Server over SMB scenarios are a good fit for Scale-Out File Servers.

More information: http://technet.microsoft.com/en-us/library/hh831349.aspx

3.1. SMB Scale-Out – Volumes, not drive letters

This new Scale-Out capability is enabled by using a couple of feature from Failover Clustering: Cluster Shared Volumes (CSV) and the Dynamic Network Name (DNN).

CSV is a special volume that shows on every cluster node simultaneously, so you don’t have to be worried about which cluster node can access the data (they all do). These CSV volumes show under the C:\ClusterStorage path, which means you don’t need a drive letter for every cluster disk. Definitely simpler.

3.2. SMB Scale-Out – Single name, dynamic

The DNN is also much easier to use, since you can have a single name that allows SMB clients to connect to the cluster nodes. A client will be automatically directed to one of the nodes in the cluster in a round-robin fashion. No extra configuration required. Also, there is really no good reason anymore to have multiple names in a single cluster for File Servers. One name to rule them all.

3.3. SMB Scale-Out – Using node IP addresses

In the past, File Server Cluster groups required specific IP addresses in addition to the regular IP addresses for each node (or a set of IP addresses per cluster group, if you have multiple networks). Combined with the fact that we used to need multiple cluster names to leverage multiple nodes, that made for a whole lot of IP addresses to assign. Even if you were using DHCP, that would still be complex.

With SMB Scale-Out, as we explained, you only need one name. In addition to that, we no longer need additional IP addresses besides the regular cluster node IP addresses. The DNN simply points to the existing node IP addresses of every node in the cluster. This means fewer IP addresses to configure and fewer Cluster resources to manage.

4. SMB Multichannel

We introduced the ability to use multiple network interfaces for aggregated throughput and fault tolerance in SMB 3.0, which we call SMB Multichannel.

More information: http://blogs.technet.com/b/josebda/archive/2012/06/28/the-basics-of-smb-multichannel-a-feature-of-windows-server-2012-and-smb-3-0.aspx

4.1. SMB Multichannel – Auto-discovery

We struggled with the best way to configure SMB Multichannel, since there are many possible configurations. In other solutions in the same space, you are sometimes required to specify each client the multiple IP addresses on each server, requiring updates whenever the cluster or networking layout changes. We thought that was too complex.

In probably the best example of the commitment to keeps things simple and easy, we decided to change the SMB protocol to include the ability for the client to query the server network configuration. This allows the client to know the type, speed and IP address of every NIC on the server. With that information at hand, the client can then automatically select the best combination of paths to use. It was not an easy thing to implement, but we thought it was worth doing. The end result was better than what we expected. SMB 3.0 does all the hard work and we got extensive feedback that this is one of the most impressive new abilities in this version.

Expert mode: There are PowerShell cmdlets to define SMB Multichannel Constraints, in case the configuration automatically calculated by SMB Multichannel is less than ideal. We strongly recommend against using this, since it will make SMB unable to adapt to changing configurations. It’s an option, though. See link above for details.

4.2. SMB Multichannel – Transparent failover

With multiple network interfaces used by SMB Multichannel, we now have the ability to survive the failure of a network path, as long as there is at least one surviving path. To keep things simple and easy, this happens automatically and the user and applications do not see any interruption of their work.

For instance, if you have two 10GbE NICs in use on both server and client, SMB Multichannel will use them both to achieve up to 20Gbps throughput. If someone pulls off one of the cables in the client, SMB will automatically detect that situation instantly and move all the items queued on the failed NIC to the surviving one. Since SMB is already fully asynchronous, there is no problem with packet ordering.  You obviously will work at only 10Gbps after this failure, but this will otherwise be completely transparent.

If an interface fails on the server side, the behavior is a bit different. It will take a TCP/IP timeout to figure out that server interface was lost and this means a few packets might be delayed a few seconds. Once the situation is detected, the behavior described for the client-side interface loss will take place. Again, this is all transparent to users and applications, but it will take a few more seconds to happen.

4.3. SMB Multichannel – Interface arrival

Beyond surviving the failure of an interface, SMB Multichannel can also re-establish the connections when a NIC “comes back”. Again, this happens automatically and typically within seconds, due to the fact that the SMB client will listen to the networking activity of the machine, being notified whenever a new network interface arrives on the system.

Using the example on the item above, if the cable for the second 10GbE interface is plugged back in, the SMB client will get a notification and re-evaluate its policy. This will lead to the second set of channel being brought back and the throughput going up to 20Gbps again, automatically.

If a new interface arrives on the server side, the behavior is slightly different. If you lost one of the server NICs and it comes back, the server will immediately accept connections on the new interface. However, clients with existing connections might take up to 10 minutes the readjust their policies. This is because they will poll the server for configuration adjustments in 10-minute intervals. After 10 minutes, all clients will be back to full throughput automatically.

4.4. SMB Multichannel – Link-local

There is a class of IP addresses referred to as link-local addresses. These are the IPv4 addresses starting with 169.254 and IPv6 addresses starting with FE80. These special IP addresses are automatically assigned when no manual addresses are configured and no DHCP server is found on the network.  They are not routable (in these configurations, there is no default gateway) and we were tempted to drop support for them in SMB Multichannel, since it would complicate our automatic policy logic.

However, we heard clear feedback that these type of addresses can be useful and they are the simplest and easiest form of addressing. For certain types of back-end network like a cluster internal network, they could be an interesting choice. Because of that, we put in the work to transparently support both IPv4 and IPv6 link-local addresses.

Expert mode: You should never mix static and link-local addresses on a single interface. If that happens, SMB will ignore link-local addresses in favor of the other address.

5. SMB Direct

SMB Direct introduced the ability to use RDMA network interfaces for high throughput with low latency and low CPU utilization.

More information: http://technet.microsoft.com/en-us/library/jj134210.aspx

5.1. SMB Direct – Discovery over TCP/IP

When using these RDMA network interfaces, you need specific connections to leverage their enhanced data transfer mode. However, since not every server hardware out there will have this RDMA capability, you have to somehow discover when it’s OK to use RDMA connections.

In order to keep things simple for the administrator, all RDMA network interfaces that support SMB Direct, regardless of technology, are required to behave as a regular NIC, with an IP address and a TCP/IP stack.  For the IT administrator, these RDMA-capable interfaces look exactly like regular network interfaces. Their configuration is also done in the same familiar fashion.

The initial SMB negotiation will always happens over traditional TCP/IP connections and SMB Multichannel will be used to find if a network interface has this RDMA capability. The shift from TCP/IP to RDMA connections happens automatically, shortly after SMB discovers that the client and server are capable of using the 3.0 dialect, they both support the SMB Multichannel capability and that there are RDMA NICs on both ends.

Expert mode: This process is so simple and transparent that IT Administrators sometimes have difficulty telling that RDMA is actually being used, besides the fact that you enjoy better throughput and lower CPU usage. You can specifically find out by using PowerShell cmdlets (like Get-SmbMultichannelConnection) and performance counters (there are counters for the RDMA traffic and the SMB Direct connections).

5.2. SMB Direct – Fail back to TCP/IP

Whenever SMB detects an RDMA-capable network, it will automatically try to use their RDMA capability. However, if for any reason the SMB client fails to connect using the RDMA path, it will simply continue to use TCP/IP connections instead. As we mentioned before, all RDMA interfaces compatible with SMB Direct are required to also implement a TCP/IP stack, and SMB Multichannel is aware of that.

Expert mode: If SMB is using the fallback TCP/IP connection instead of RDMA connections, you are essentially using a less efficient mode of your RDMA-capable network interface. As explained before, you can detect this condition by using specific PowerShell cmdlets and performance counters. There are also specific events logged when that occurs.

6. VSS for SMB File Share

Backup and Restore are important capabilities in Storage. In Windows Server, Volume Shadowcopy Services (VSS) is a key piece of infrastructure for creating application-consistent snapshots of data volumes.

More information: http://blogs.technet.com/b/clausjor/archive/2012/06/14/vss-for-smb-file-shares.aspx

6.1. VSS for SMB File Share – Same model as block VSS

VSS has a well-established ecosystem for block solutions and we decided that the best path for file shares was to follow the same model. By offering VSS for SMB File Shares, the backup applications and the protected applications can leverage their investments in VSS. For an IT Administrators already using VSS, the solution looks familiar and easy to understand.

6.2. VSS for SMB File Shares – Stream snapshots from file server

In most backup scenarios, after a snapshot is created, you need to stream the data from the application server to a backup server. That’s, for instance, the model used by Data Protection Manager (DPM). While there are technologies to move snapshots around in some cases, some types of snapshot can only be surfaced on the same host where the volume was originally snapped.

By using VSS for SMB File Shares the resulting snapshot is represented as a UNC path pointing to the file server. That means it’s simple to tell that you can rely on the file server to stream the backup data and not have to bother the application server for anything else after the snapshot is taken.

7. SMB Encryption

Encryption solutions are an important capability, but they are typically associated with Public Key Infrastructure (PKI) and the management of certificates. This leads to everyone concluding this is complex and hard to deploy. While SMB can be used in conjunction with IPSec encryption, we thought we needed something much simpler to deploy.

More information: http://blogs.technet.com/b/josebda/archive/2012/06/26/episode-20-of-quot-from-end-to-edge-and-beyond-quot-covers-smb-encryption-in-windows-server-2012-and-smb-3-0.aspx

7.1. SMB Encryption – No PKI or certificates required

With SMB Encryption, the encryption key is derived from the existing session key, so you don’t need PKI or certificates. These keys are not shared on the wire and you can simply enjoy yet another benefit of having already deployed your Active Directory infrastructure. From an IT administrator perspective, enabling encryption simply means checking a box (or adding a single parameter to the PowerShell cmdlet that creates the share). Clients don’t need to do anything, other than support SMB 3.0.

clip_image003

Expert mode: The default configuration is to configure encryption per share, but there is an option to enable encryption for the entire server, configured via PowerShell. You simply need to use Set-SmbServerConfiguration –EncryptData $true.

7.2. SMB Encryption – Accelerated Encryption

Probably the second highest concern with encryption solution is that it would make everything run slower. For SMB encryption, we decided to use a modern algorithm (128-bit AES-CCM) which can be accelerated significantly on CPUs supporting the AES-NI instruction set. That means that, for most modern CPUs like the Intel Core i5 and Intel Core i7, the cost of encryption becomes extremely low. That removes another potential blocker for adoption of encryption, which is the complexity associated in planning additional processing capacity to handle it. It’s also nice to note that AES-CCM provides integrity, in addition to privacy.

There’s no need to configure any settings on the client or the server to benefit from this accelerated encryption.

8. Server Manager

For IT Administrator handling individual shares using a GUI, we are providing in Windows Server 2012 a new Server Manager tool. This tool allows you to create your SMB file shares using a simple set of wizards.

More information: http://technet.microsoft.com/en-us/library/hh831456.aspx

8.1. Server Manager – Simple Wizards

In Server Manager, you get an “SMB Share – Quick” that creates a simple share, an “SMB Share – Advanced” wizard to create shares with additional options (like quotas and screening) and an “SMB Share – Applications” wizard specifically for creating shares intended for server-to-server workloads (like Hyper-V over SMB or SQL Server over SMB).

clip_image005

These wizards are exactly what you would expect from Windows: a set of simple questions that are easy to follow, with plenty of explanatory text around them. It’s also important to mention that you can run Server Manager against remote servers. In fact, you can even run it on the IT Administrator’s Windows 8 PC or tablet.

8.2. Server Manager – Fewer knobs

When creating the new wizards, we tried as much as possible to simplify the workflow. The simple share wizard asks only a few questions: what disk the share will sit on, what’s the name of the share, what are the options you want to enable and who has access. For each page, we provided a simple selection to make the process as straightforward as possible.

For instance, in the selection of the location of the share, we ask you to select a server, than a volume (based what’s available to the server you targeted). You are shown information on the type of server and on capacity/free disk on each volume to help with your selections. You don’t even have to create a folder structure. The wizard will automatically create a folder with the specified share name under the “Shares” folder on the selected volume. You can still specify a full path if you want.

clip_image007

When we started thinking about the options page, we had a huge list of knobs and switches. In the end, the options page was trimmed down to show only the four most commonly used options, graying them out if not relevant for that type of wizard. We also spent lots of time thinking of the right default selections for these options. For instance, in the “SMB Share – Applications” options page, only encryption is offered. In that case, Continuous Availability is always selected, while Access-based enumeration and BranchCache are always deselected.

9. SMB PowerShell

While most people looking for the most simplicity and ease of use will likely be using the Server Manager GUI, most IT Administrators managing a significant number of shares will be using PowerShell. PowerShell is also a tool that will let you go deeper into understanding the inner workings of SMB 3.0. However, we wanted to make SMB PowerShell simple enough that any IT Administrator could understand it.

More information: http://blogs.technet.com/b/josebda/archive/2012/06/27/the-basics-of-smb-powershell-a-feature-of-windows-server-2012-and-smb-3-0.aspx

9.1. SMB PowerShell – Permissions for a New Share

The New-SmbShare cmdlet, used to create new SMB file shares, is very straightforward. You provide a share name, a path and optionally some share settings like access-based enumeration or encryption. Then there’s permissions. At first, we considered requiring the administrator to use a second cmdlet (Grant-SmbShareAccess) to add the required permissions to the share. However, since some set of permissions are always required for a share, we thought of a way to add permissions in the same cmdlet you use to create the share. We did this by using the –FullAccess, -ChangeAccess, –ReadAccess and –NoAccess parameters. This makes the entire share creation a single line of PowerShell that is easy to write and understand.

For instance, to create a share for Hyper-V over SMB using a couple of Hyper-V hosts, you can use:
New-SmbShare –Name VMS –Path C:\ClusterStorage\Volume1\VMS –FullAccess Domain\HVAdmin, Domain\HV1$, Domain\HV2$

9.2. SMB PowerShell – Permissions on the Folder

While we believe we found an simple and easy way to declare the share permissions, anyone experienced with SMB knows that we also need to grant permissions at the file system level. Essentially, we need to grant the NTFS permissions on the folder behind the SMB share. In most cases, we want the NTFS folder permissions to match the SMB share permissions. However, we initially overlooked the fact that we were requiring the IT Administrator to declare those permissions twice. That was additional, repetitive work that we could potentially avoid. And granting permissions at the NTFS level is a little more complex, because there are options for inheritance and a more granular permission model than the SMB permissions.

To facilitate that, we introduced an option to address this. The basic idea is to provide a preconfigured set of NTFS permissions based on the SMB share permissions. The goal was to create the Access Control List (ACL) automatically (we called it the “Preset Path ACL”), providing the IT Administrator with a simple way to apply the SMB share ACL to the NTFS folder. It is an optional step, but we believe it is useful.

Here’s what that command line looks like, assuming your share is called “VMS”:
(Get-SmbShare –Name VMS).PresetPathAcl | Set-Acl

9.3. SMB PowerShell – Cluster type based on disk type

Another item that was high on our list was automatically detecting if the share was sitting on a Nonclustered File Server, on a Classic File Server cluster or on a Scale-Out File Server. This is not a trivial thing, since the same Windows Server is actually capable of hosting all three types of file shares at the same time. At first, we were considering requiring the IT Administrator to explicitly state what type of file share should be used and exactly what cluster name (scope name) should be used.

In the end, we decided that we could infer that information on almost all cases. A Scale-Out file share always uses a folder in a Cluster Shared Volumes (CSV), while a Classic clustered file share will always use a traditional cluster disk and a nonclustered file share will use a disk that is not part of the cluster. With that in mind, the “New-SmbShare” cmdlet (or more precisely the infrastructure behind it) is able to find the scope name for a given share without requiring the IT Administrator to specify the scope name.

Type of DiskType of shareScope Name
Local disk (non-clustered)Nonclustered file shareN/A
Traditional cluster diskClassic clustered file shareName resource on Cluster Group
Cluster Shared VolumeScale-Out file shareDistributed Network Name (DNN)

Expert mode: The only situation where you must specify the Scope Name while creating a new SMB file share is when there are two DNNs in a clusters. We recommend that you use only one DNN per cluster. In fact, we don’t see any good reason to use more than one DNN per cluster.

9.4. SMB PowerShell – Shares, Sessions and Open Files across Scale-Out nodes

SMB Scale-Out introduces the concept of multiple file server cluster nodes hosting the same shares and spreading client connections across nodes. However, for an IT Administrator managing this kind of cluster, it might seem more complicated. To avoid that, we made sure that any node of a Scale-Out File Server answers consistently for the entire cluster. For the list of shares that was easy, since every node (by definition) has the full list of shares. For sessions and open files that was a bit more complex and we actually had make the node you’re talking to gather the information from the other nodes.

In the end, the IT Administrator can rest assured that, when using Get-SmbSession or a Get-SmbOpenFile in a Scale-Out File Server, what is provided is a combined view of the cluster and there’s no need to query every node in the cluster and collate that information manually.

9.5. SMB PowerShell – Connection

With all these new capabilities in SMB 3.0 and the potential in the protocol to negotiate down to previous versions of SMB, it’s important to understand what dialect was negotiated. There is a simple “Get-SmbConnection” cmdlet, which you should run on an SMB client, that shows information on which SMB connections are currently established.

clip_image009

This can tell you which shares you are connected to (for a given server in the example above, but you can also omit the -ServerName parameter to see all connections) and which version of SMB (dialect) you are using (SMB 3.0 in the example above). You can also see how many files you have opened and what credentials you are offering to the server. If you look at all the data returned (by piping the output to “Select *” as shown below), you get the details on whether the share is continuously available or if the connection is encrypted.

clip_image011

You might argue that is not necessarily the simplest command output you have seen. However, this is much simpler than what we used before. To find out this type of information in previous releases, you would need to break out a network tracing tool like Network Monitor or WireShark and starting capturing network packets, analyzing the traffic to find the negotiation messages.

10. Conclusion

I hope this post clarifies our claims for simplicity and ease of use in SMB 3.0 and the Windows Server 2012 File Server. It is always harder to code a solution that “just works” with minimal administration intervention, but we do believe this is something worth doing it.

I would love to hear back from you on the progress we made in Windows Server 2012. Were these the right decisions? Did we miss anything important? Are there areas where things are still complex or hard to use? Let me know through the blog post comments…

SNIA Tutorials cover a wealth of Storage knowledge (SNW Fall 2012 Update)

$
0
0

I’m always amazed by the vast amount of knowledge being posted online for free download. If you are an IT Professional who deals with Storage (or if you manage one of them), check out the high-quality tutorials just posted online by the Storage Network Industry Association (SNIA).

These peer-reviewed, vendor-neutral tutorials, offered by the SNIA Education Committee, cover a variety of very current Storage-related topics. They were delivered live during the recent Storage Networking World conference in Santa Clara, CA (SNW Fall 2012). I delivered one of them and I was also able to attend a few of them in person.

 

Here’s a snapshot of what they covered, organized by track:

Track: Big Data

  • Introduction to Analytics and Big Data – Hadoop (Rob Peglar)
  • Protecting Data in the "Big Data" World (Thomas Rivera)
  • How to Cost Effectively Retain Reference Data for Analytics and Big Data (Molly Rector)
  • Big Data Storage Options for Hadoop (Dr. Sam Fineberg)
  • Massively Scalable File Storage (Philippe Nicolas)
  • Can You Manage at Petabyte Scale? (John Webster)

Track: Cloud and Emerging Technologies

  • Interoperable Cloud Storage with the CDMI Standard (Mark Carlson)
  • The Business Case for the Cloud (Alex McDonald)
  • Archiving and Preservation in the Cloud: Business Case, Challenges and Best Practices (Chad Thibodeau, Sebastian Zangaro)
  • Enterprise Architecture and Storage Clouds (Marty Stogsdill)

Track: Data Protection and Management

  • Introduction to Data Protection: Backup to Tape, Disk and Beyond (Jason Iehl)
  • Trends in Data Protection and Restoration Technologies (Michael Fishman)
  • Advanced Data Reduction Concepts (Gene Nagle, Thomas Rivera)

Track: File Systems and File Management

  • The File Systems Evolution (Thomas Rivera)
  • SMB Remote File Protocol, including SMB 3.0 (Jose Barreto)

Track: Green Storage

  • Green Storage and Energy Efficiency (Carlos Pratt)
  • Green Storage - the Big Picture ("Green is More Than kWh!") (SW Worth)

Track: Networking

  • Technical Overview of Data Center Networks (Dr. Joseph White)
  • Single and Multi Switch Designs with FCoE (Chad Hintz)
  • The Unintended Consequences of Converged Data Center Deployment Models (Simon Gordon)
  • FCoE Direct End-Node to End-Node (a/k/a FCoE VN2VN) (John Hufferd)
  • PCI Express IO Virtualization Overview (Ron Emerick)
  • How Infrastructure as a Service (laaS) and Software Defined Networks (SDN) will Change the Data Center (Samir Sharma)
  • IP Storage Protocols: iSCSI (John Hufferd)
  • Fibre Channel Over Ethernet (FCoE) (John Hufferd)

Track: Professional Development

  • Social Media and the IT Professional - Are You a Match? (Marty Foltyn)
  • Reaction Management and Self-facilitation Techniques (David Deming)

Track: Security

  • Practical Storage Security with Key Management (Russ Fellows)
  • Unmasking Virtualization Security (Eric Hibbard)
  • Data Breaches and the Encryption Safe Harbor (Eric Hibbard)
  • Storage Security - the ISO/IEC Standard (Eric Hibbard)
  • A Hype-free Stroll Through Cloud Security (Eric Hibbard)
  • Got Lawyers? They've Got Storage and ESI in the Cross-hairs! (Eric Hibbard)

Track: Solid State Storage

  • The Benefits of Solid State in Enterprise Storage Systems (David Dale)
  • An In-depth Look at SNIA's Enterprise Solid State Storage Test Specification (PTS v1.1) (Easen Ho)
  • Realities of Solid State Storage (Luanne Dauber)
  • What Happens When Flash Moves to Triple Level Cell (TLC) (Luanne Dauber)
  • NVMe the nextGen Interface for Solid State Storage (Anil Vasudeva)
  • SCSI Express - Fast & Reliable Flash Storage for the Enterprise (Marty Czekalski)

Track: Storage and Storage Management

  • What's Old is New Again - Storage Tiering (Kevin Dudak)
  • Simplified Integration and Management in Multi-Vendor SAN Environments (Chauncey Schwartz)
  • SAS: The Emerging Storage Fabric (Marty Czekalski)

Track: Virtualization/Applications

  • Virtualization Practices: Providing a Complete Virtual Solution in a Box (Jyh-shing)
  • VDI Optimization - Real World Learnings (Russ Fellows) 

 

I would encourage all my industry colleagues to check these out!

To view the slides for each one of the tutorials listed above in PDF format, visit http://www.snia.org/education/tutorials/2012/fall. Be sure to check the Legal Notices on the documents about how you can use them.

For more information about SNIA, check http://www.snia.org/. It's also never too early to plan to check the next wave, to be delivered during the next SNW in the Spring.

Windows Server 2012 File Server Tip: Switch to the High Performance power profile

$
0
0

When you install a fresh copy of Windows Server 2012 and configure it with the File Server role, the default Power Setting balances power efficiency and performance.

For this reason, even if you have a few high speed network interfaces and the fastest SSD storage out there, you might not be getting the very best IOPS and throughput possible.

To get the absolute best performance from your file server, you can set the server to the "High Performance" power profile.

To configure this using a GUI, go to the Start Menu, search for "Choose a Power Plan" under Settings, then select the "High Performance" option.

image

To configure this from a command line, use "POWERCFG.EXE /S SCHEME_MIN".

 

NOTE: This is not recommended for every file server. The Balanced (default) profile will be enough for most cases, with high and constant load being the exception.

Windows Server 2012 File Server Tip: Use PowerShell to find the free space on the volume behind an SMB file share

$
0
0

A while back, I showed how to use PowerShell V2 and our old SMB WMIv1 object to explain how to find the free space behind a file share (essentially the free space for the volume that contains the file share). That post is available at http://blogs.technet.com/b/josebda/archive/2010/04/08/using-powershell-v2-to-gather-info-on-free-space-on-the-volumes-of-your-remote-file-server.aspx. While that post was a good example of how to construct a more elaborate solution using PowerShell, it was a little complicated :-).

Now, with Windows Server 2012, PowerShell V3 and SMB PowerShell, things got much simpler. I can essentially do the same thing with a simple one-liner.

For instance, to see the free space on the volume behind a specific share named TEST, you can use

Get-Volume -Id (Get-SmbShare TEST).Volume

To list the volumes for all shares on a specific server, you can use:

Get-SmbShare | ? Volume -ne $null | % { $_ | FL ; Get-Volume -Id $_.Volume | FL }

Note that you can also execute those remotely, pointing to the server where the shares are located:

Get-Volume  -CimSession Server1 -Id (Get-SmbShare TEST -CimSession Server1).Volume

Get-SmbShare -CimSession Server1 | ? Volume  -ne $null | % { $_ | FL ; Get-Volume -CimSession Server1 -Id $_.Volume | FL  }

Here is a complete example, with output. First a simple query to find the information for a volume behind a share

PS C:\> Get-Volume -Id (Get-SmbShare VMS1).Volume

DriveLetter       FileSystemLabel  FileSystem       DriveType        HealthStatus        SizeRemaining             Size
-----------       ---------------  ----------       ---------        ------------        -------------             ----
I                                  NTFS             Fixed            Healthy                  78.85 GB           100 GB

PS C:\> Get-Volume -Id (Get-SmbShare Projects).Volume | Select *

HealthStatus          : Healthy
DriveType             : Fixed
DriveLetter           :
FileSystem            : CSVFS
FileSystemLabel       :
ObjectId              : \\?\Volume{20795fea-b7da-43dd-81d7-4d346c337a73}\
Path                  : \\?\Volume{20795fea-b7da-43dd-81d7-4d346c337a73}\
Size                  : 107372081152
SizeRemaining         : 85759995904
PSComputerName        :
CimClass              : ROOT/Microsoft/Windows/Storage:MSFT_Volume
CimInstanceProperties : {DriveLetter, DriveType, FileSystem, FileSystemLabel...}
CimSystemProperties   : Microsoft.Management.Infrastructure.CimSystemProperties

Now a more complete query, showing all shares starting with VMS and information on the volume behind them:

PS C:\> Get-SmbShare VMS* | ? Volume -ne $null | % { $_ | FL ; Get-Volume -Id $_.Volume | FL }

Name        : VMS1
ScopeName   : FST2-FS
Path        : I:\VMS
Description :

DriveLetter     : I
DriveType       : Fixed
FileSystem      : NTFS
FileSystemLabel :
HealthStatus    : Healthy
ObjectId        : \\?\Volume{b02c4ba7-e6f1-11e1-93eb-0008a1c0ef0d}\
Path            : \\?\Volume{b02c4ba7-e6f1-11e1-93eb-0008a1c0ef0d}\
Size            : 107372081152
SizeRemaining   : 84665225216
PSComputerName  :

Name        : VMS2
ScopeName   : FST2-FS
Path        : J:\VMS
Description :

DriveLetter     : J
DriveType       : Fixed
FileSystem      : NTFS
FileSystemLabel :
HealthStatus    : Healthy
ObjectId        : \\?\Volume{b02c4bb1-e6f1-11e1-93eb-0008a1c0ef0d}\
Path            : \\?\Volume{b02c4bb1-e6f1-11e1-93eb-0008a1c0ef0d}\
Size            : 107372081152
SizeRemaining   : 84665225216
PSComputerName  :

Name        : VMS3
ScopeName   : FST2-SO
Path        : C:\ClusterStorage\Volume1\VMS
Description :

DriveLetter     :
DriveType       : Fixed
FileSystem      : CSVFS
FileSystemLabel :
HealthStatus    : Healthy
ObjectId        : \\?\Volume{20795fea-b7da-43dd-81d7-4d346c337a73}\
Path            : \\?\Volume{20795fea-b7da-43dd-81d7-4d346c337a73}\
Size            : 107372081152
SizeRemaining   : 85759995904
PSComputerName  :

Name        : VMS4
ScopeName   : FST2-SO
Path        : C:\ClusterStorage\Volume2\VMS
Description :

DriveLetter     :
DriveType       : Fixed
FileSystem      : CSVFS
FileSystemLabel :
HealthStatus    : Healthy
ObjectId        : \\?\Volume{fb69e20a-5d6a-4dc6-a0e9-750291644165}\
Path            : \\?\Volume{fb69e20a-5d6a-4dc6-a0e9-750291644165}\
Size            : 107372081152
SizeRemaining   : 84665225216
PSComputerName  :

Name        : VMS5
ScopeName   : *
Path        : D:\VMS
Description :

DriveLetter     : D
DriveType       : Fixed
FileSystem      : NTFS
FileSystemLabel : LocalFS1
HealthStatus    : Healthy
ObjectId        : \\?\Volume{58a38e4e-e2fd-11e1-93e8-806e6f6e6963}\
Path            : \\?\Volume{58a38e4e-e2fd-11e1-93e8-806e6f6e6963}\
Size            : 181336535040
SizeRemaining   : 136311140352
PSComputerName  :

Finally, not for the faint of heart, a more complex query from a remote file server which creates a custom result combining share information and volume information:

PS C:\> Get-SmbShare -CimSession FST2-FS2 | ? Volume  -ne $null | % { $R = "" | Select Share, Path, Size, Free; $R.Share=$_.Name; $R.Path=$_.Path; Get-Volume -CimSession FST2-FS2 -Id $_.Volume | % { $R.Size=$_.Size; $R.Free=$_.SizeRemaining; $R | FL }}


Share : ADMIN$
Path  : C:\Windows
Size  : 68352471040
Free  : 44692242432

Share : C$
Path  : C:\
Size  : 68352471040
Free  : 44692242432

Share : ClusterStorage$
Path  : C:\ClusterStorage
Size  : 68352471040
Free  : 44692242432

Share : D$
Path  : D:\
Size  : 181336535040
Free  : 177907777536

Share : Projects
Path  : C:\ClusterStorage\Volume1\SHARES\PROJECTS
Size  : 107372081152
Free  : 85759995904

Share : VMFiles
Path  : C:\ClusterStorage\Volume1\VMFiles
Size  : 107372081152
Free  : 85759995904

Share : VMS3
Path  : C:\ClusterStorage\Volume1\VMS
Size  : 107372081152
Free  : 85759995904

Share : VMS4
Path  : C:\ClusterStorage\Volume2\VMS
Size  : 107372081152
Free  : 84665225216

Share : Witness
Path  : C:\ClusterStorage\Volume1\Witness
Size  : 107372081152
Free  : 85759995904

Windows Server 2012 File Server Tip: New per-share SMB client performance counters provide great insight

$
0
0

Windows Server 2012 and Windows 8 include a new set of performance counters that can greatly help understand the performance of the SMB file protocol. These include new counters on both the server side and the client side.

In this post, I wanted to call your attention to the new client-side counters that show the traffic going to a specific file share. In the example below, we see the performance counters on a computer called FST2-HV1, which is accessing some files on the VMS1 share on a file server cluster called FST2-FS. This is actually a Hyper-V over SMB scenario, where a VM is storing its VHDX file on a remote file share. I started a light copy inside the VM to generate some activity.

 

image

 

This simple set of counters offers a lot of information. If you know how to interpret it, you can get information on IOPS (data requests per second), throughput (data bytes per second), latency (average seconds per request), IO size (average bytes per request) and outstanding IOs (data queue length) . You can also get all this information for read and write operations.  There’s a lot you can tell from these data points. You can see we’re doing about 910 IOPS and we’re doing more writes than reads (648 write IOPS vs. 262 read IOPS). However, we’re doing larger reads than writes (in average, 104KB reads vs. 64KB writes). Reads are faster than writes (5 milliseconds vs. 8 milliseconds). In case you’re wondering, the workload running here was a simple copy of lots of fairly large files from one folder to another inside the VM.

If you’re familiar with the regular disk counters in Windows, you might notice a certain resemblance. That’s not by accident. The SMB client shares performance counters were designed to exactly match the disk counters. This way you can easily reuse any guidance on application disk performance tuning you currently have . Here’s a little table comparing the two sets:

TypeDisk Object Counter SMB Client Shares Counter
IOPS Disk transfers / sec Data Requests/sec
Disk reads / sec Read Requests/sec
Disk writes / sec Write Requests/sec
Latency Avg disk sec / transfer Avg. sec/Data Request
Avg disk sec / read Avg. sec/Read
Avg disk sec / write Avg. sec/Write
IO Size Avg disk bytes / transfer Avg. Bytes/Data Request
Avg disk bytes / read Avg. Bytes/Read
Avg disk bytes / write Avg. Bytes/Write
Throughput Disk bytes / sec Data Bytes/sec
Disk read bytes / sec Read Bytes/sec
Disk write bytes / sec Write Bytes/sec
Queue Length Avg. Disk Read Queue Length Avg. Read Queue Length
Avg. Disk Write Queue Length Avg. Write Queue Length
Avg Disk Queue Length Avg. Data Queue Length

I would encourage you to understand your application storage performance and capture/save some samples after you tune your applications. This way you can build a library of baseline performance counters that you can later use for troubleshooting performance issues.

Updated Links on Windows Server 2012 R2 File Server and SMB 3.0

$
0
0

In this post, I'm providing a reference to the most relevant content related to Windows Server 2012 R2 that is related to the File Server, the SMB 3.0 features and its associated scenarios like Hyper-V over SMB and SQL Server over SMB. It's obviously not a complete reference (there are always new blogs and articles being posted), but hopefully this is a useful collection of links for Windows Server 2012 R2 users.

This post covers only articles that are specific to Windows Server 2012 R2. However, note that there’s also a Windows Server 2012 version of this post. Most concepts, step-by-steps and tools listed there also apply to Windows Server 2012 R2.

 

Overview articles on Windows Server 2012 R2 File Server and SMB 3.0 (and related topics)

 

Step-by-step instructions for Windows Server 2012 R2 File Server and SMB 3.0 (and related topics)

 

TechEd 2013 presentations (with video recording) on Windows Server 2012 R2 File Server and SMB 3.0 (and related topics)

 

Demos, Interviews and other video recordings

 

Windows Server 2012 R2 download links

 

Blog posts by Microsoft MVPs on Windows Server 2012 R2 File Server and SMB 3.0 (and related topics)

 

Windows Server 2012 R2 File Server Tips and Q&A

 

Protocol Documentation Preview

 

Other relevant links related to Windows Server 2012 R2 SMB features

 

-------

 

Change tracking:

Windows Server 2012 R2 Storage: Step-by-step with Storage Spaces, SMB Scale-Out and Shared VHDX (Physical)

$
0
0

This post is a part of the nine-part “What’s New in Windows Server & System Center 2012 R2” series that is featured on Brad Anderson’s In the Cloud blog.  Today’s blog post covers Windows Server 2012 R2 Storage and how it applies to the larger topic of “Transform the Datacenter.”  To read that post and see the other technologies discussed, read today’s post: “What’s New in 2012 R2: IaaS Innovations.”

 

1) Overview

 

In this document, I am sharing all the steps I used to create a Windows Server 2012 R2 File Server environment, so you can try the new features like SMB Scale-Out Rebalancing and Shared VHDX for Guest Clustering. This configuration uses seven physical computers and a JBOD for shared storage. If you're not familiar with these technologies, I would strong encourage you to review the TechEd 2013 talks on Hyper-V over SMB (Understanding the Hyper-V over SMB Scenario, Configurations, and End-to-End Performance) and Shared VHDX (second half of Application Availability Strategies for the Private Cloud).

 

The demo setup includes the following:

  • 1 DNS / Domain Controller
  • 1 JBOD with SAS disks (suggested at least 4 HDDs with 250GB each)
  • 3 Scale-Out File Server Cluster nodes
  • 2 Hyper-V cluster nodes
  • 1 VMM Server

 

Here’s a diagram of the setup:

clip_image002

 

Here are the details about the names, roles and IP addresses for each of the computers involved:

 

ComputerRoleNet1 (Corp)Net2 (DC/DNS)NetR1 (RDMA)
JOSE-DDNS, DCDHCP192.168.100.1/24192.168.101.1/24
JOSE-A1Cluster A Node 1DHCP192.168.100.11/24192.168.101.11/24
JOSE-A2Cluster A Node 2DHCP192.168.100.12/24192.168.101.12/24
JOSE-A3Cluster A Node 3DHCP192.168.100.13/24192.168.101.13/24
JOSE-ACluster A CNON/A192.168.100.19/24192.168.101.19/24
JOSE-FScale-Out​ File Server DNNN/AN/AN/A
JOSE-B1Cluster B Node 1DHCP192.168.100.21/24192.168.101.21/24
JOSE-B2Cluster B Node 2DHCP192.168.100.22/24192.168.101.22/24
JOSE-BCluster B CNON/A192.168.100.29/24192.168.101.29/24
JOSE-VVMM ServerDHCP192.168.100.31/24192.168.101.31/24
JOSE-X1Cluster X, VM Node 1DHCP192.168.100.41/24 
JOSE-X2Cluster X, VM Node 2DHCP192.168.100.42/24 
JOSE-XGuest Cluster X CNODHCP192.168.100.49/24 
JOSE-XSSQL Server, Cluster XDHCP192.168.100.48/24 

 

Following these steps will probably require a few hours of work end-to-end, but it is a great way to experiment with a large set of Microsoft technologies in or related to Windows Server 2012 R2, including:

  • Hyper-V
  • Networking
  • Domain Name Services (DNS)
  • Active Directory Domain Services (AD-DS)
  • Shared VHDX
  • Storage Spaces
  • Failover Clustering
  • File Servers
  • PowerShell
  • SQL Server
  • Virtual Machine Manager

 

2) Required Hardware and Software, plus Disclaimers

 

You will need the following hardware to perform the steps described here:

 

You will need the following software to perform the steps described here:

 

Notes and disclaimers:

  • A certain familiarity with Windows administration and configuration is assumed. If you're new to Windows, this document is not for you. Sorry...
  • If you are asked a question or required to perform an action that you do not see described in these steps, go with the default option.
  • There are usually several ways to perform a specific configuration or administration task. What I describe here is one of those many ways. It's not necessarily the best way, just the one I personally like best at the moment.
  • For the most part, I use PowerShell to configure the systems. You can also use a graphical interface instead, but I did not describe those steps here.
  • The JBOD used must be certified for use with Storage Spaces. For a list of certified JBODs, check the Windows Server Catalog at http://www.windowsservercatalog.com/results.aspx?bCatID=1573&cpID=0&avc=10&OR=1
  • The Storage Spaces configuration shown here uses only HDDs. You could enhance it to use both SSDs and HDDs, in order to leverage new Storage Spaces features like Tiering or Write-back Cache.
  • The configuration uses three file servers to demonstrate the new SMB Scale-Out Balancing capabilities when adding a node to a two-node file server cluster. It uses only two Hyper-V nodes because that’s the minimum required for a fault-tolerant Shared VHDX deployment. Typical setups will use a two-node file server cluster and a larger number of Hyper-V cluster nodes.

 

3) Summarized instructions for experts

 

If you are already familiar with Failover Clustering, Scale-Out File Servers and Hyper-V, here’s a short introduction to what it takes to configure a Shared VHDX environment with a Scale-Out File Server. A detailed step-by-step is available in the following sections of this document.

  • Configure a three-node Scale-Out File Server Cluster using Windows Server 2012 R2 and Clustered Storage Spaces with Shared SAS disks.
  • Configure a Hyper-V Failover cluster using Windows Server 2012 R2 with SMB shared storage as you normally would.
  • Create an OS VHD or VHDX file as you would regularly, on an SMB file share. Create your VMs as you would regularly would, using the OS VHD or VHDX file. Both Generation 1 and Generation 2 VMs are supported.
  • Create your VHDX data files to be shared as fixed-size or dynamically expanding, on a Clustered Shared Volume. Old VHD files or differencing disks are not supported with Shared VHDX.
  • Add the shared VHDX data files to multiple VMs, using the Add-VMHardDiskDrive and the “-ShareVirtualDisk” option. If using the GUI, check the box for “Share virtual disk” when adding the VHDX data files to the VM.
  • Inside the VM, install Windows Server 2012 R2 or Windows Server 2012. Make sure to install the Hyper-V integration components.
  • Validate and configure the cluster inside the VMs as you normally would. The guest cluster can use classic cluster disks or Cluster Shared Volumes. Any cluster role should work as it would in a physical machine with shared SAS storage.

 

The rest of this document contains detailed step-by-step instructions for each of the items outlined above.

 

4) Configure JOSE-D (DNS, DC)

 

# Preparation steps: Install WS2012R2, rename computer
Rename-Computer -NewName JOSE-D -Restart

# Install required roles and features, restarts at the end
Install-WindowsFeature AD-Domain-Services, RSAT-ADDS, RSAT-ADDS-Tools

# Rename DHCP network adapter to Net1, private to Net2
Get-NetIPAddress -PrefixOrigin DHCP | Get-NetAdapter | Rename-NetAdapter -NewName Net1

# Configure Net2 with a static IP address for DNS / DC
Get-NetAdapter Eth* | ? Status -eq Up | ? InterfaceDescription -notmatch "Mellanox*" | Rename-NetAdapter -NewName Net2
Set-NetIPInterface -InterfaceAlias Net2 -DHCP Disabled
Remove-NetIPAddress -InterfaceAlias Net2 -Confirm:$false
New-NetIPAddress -InterfaceAlias Net2 -IPAddress 192.168.100.1 -PrefixLength 24
Set-DnsClientServerAddress -InterfaceAlias Net2 -ServerAddresses 192.168.100.1

# Configure NetR1 with a static IP address for the RDMA network
Get-NetAdapter -InterfaceDescription "*IPoIB*" | ? Status -eq Up | Rename-NetAdapter -NewName NetR1
Set-NetIPInterface -InterfaceAlias NetR1 -DHCP Disabled
Remove-NetIPAddress -InterfaceAlias NetR1 -Confirm:$false
New-NetIPAddress -InterfaceAlias NetR1 -IPAddress 192.168.101.1 -PrefixLength 24
Set-DnsClientServerAddress -InterfaceAlias NetR1 -ServerAddresses 192.168.100.1

# Disable all disconnected adapters
Get-NetAdapter -InterfaceDescription "*IPoIB*" | ? Status -ne Up | Rename-NetAdapter -NewName NetRX
Get-NetAdapter | ? Status -ne Up | Disable-NetAdapter -Confirm:$false

# Create AD forest, reboots at the end
Install-ADDSForest -CreateDNSDelegation:$false -DatabasePath "C:\Windows\NTDS" -DomainMode "Win2012" -DomainName "JOSE.TEST" -DomainNetBIOSName "JOSE" -ForestMode "Win2012" -InstallDNS:$true -LogPath "C:\Windows\NTDS" -SYSVOLPath "C:\Windows\SYSVOL"

 

 

5) Configure JOSE-A1 (File Server Cluster A, Node 1)

 

# Preparation steps: Install WS2012R2, rename computer
Rename-Computer -NewName JOSE-A1 -Restart

# Install required roles and features, restarts at the end
Install-WindowsFeature File-Services, FS-FileServer, Failover-Clustering
Install-WindowsFeature RSAT-Clustering -IncludeAllSubFeature

# Rename DHCP network adapter to Net1
Get-NetIPAddress -PrefixOrigin DHCP | Get-NetAdapter | Rename-NetAdapter -NewName Net1

# Configure Net2 with a static IP address for DNS / DC
Get-NetAdapter Eth* | ? Status -eq Up | ? InterfaceDescription -notmatch "Mellanox*" | Rename-NetAdapter -NewName Net2
Set-NetIPInterface -InterfaceAlias Net2 -DHCP Disabled
Remove-NetIPAddress -InterfaceAlias Net2 -Confirm:$false
New-NetIPAddress -InterfaceAlias Net2 -IPAddress 192.168.100.11 -PrefixLength 24
Set-DnsClientServerAddress -InterfaceAlias Net2 -ServerAddresses 192.168.100.1

# Configure NetR1 with a static IP address for the RDMA network
Get-NetAdapter -InterfaceDescription "*IPoIB*" | ? Status -eq Up | Rename-NetAdapter -NewName NetR1
Set-NetIPInterface -InterfaceAlias NetR1 -DHCP Disabled
Remove-NetIPAddress -InterfaceAlias NetR1 -Confirm:$false
New-NetIPAddress -InterfaceAlias NetR1 -IPAddress 192.168.101.11 -PrefixLength 24
Set-DnsClientServerAddress -InterfaceAlias NetR1 -ServerAddresses 192.168.100.1

# Disable all disconnected adapters
Get-NetAdapter -InterfaceDescription "*IPoIB*" | ? Status -ne Up | Rename-NetAdapter -NewName NetRX
Get-NetAdapter | ? Status -ne Up | Disable-NetAdapter -Confirm:$false

# Configure Storage Spaces. Assumes JBOD with 4 x 250GB HDDs = 1TB total raw. 2 mirror columns. 80GB spaces.
# Adjust the size of the volumes and the number of mirror columns based on the actual number of HDDs you have.

$s = Get-StorageSubSystem -FriendlyName *Spaces*
New-StoragePool -FriendlyName Pool1 -StorageSubSystemFriendlyName $s.FriendlyName -PhysicalDisks (Get-PhysicalDisk -CanPool $true)
Set-ResiliencySetting -Name Mirror -NumberofColumnsDefault 2 -StoragePool (Get-StoragePool -FriendlyName Pool1)

New-VirtualDisk -FriendlyName Space1 -StoragePoolFriendlyName Pool1 -ResiliencySettingName Mirror –Size 1GB
2..7 | % { New-VirtualDisk -FriendlyName Space$_ -StoragePoolFriendlyName Pool1 -ResiliencySettingName Mirror –Size 80GB }

1..7 | % {
   $Letter ="PQRSTUV”[($_-1)]
$Number = (Get-VirtualDisk -FriendlyName Space$_ | Get-Disk).Number
Set-Disk -Number $Number -IsReadOnly 0
   Set-Disk -Number $Number -IsOffline 0
Initialize-Disk -Number $Number -PartitionStyle MBR
   New-Partition -DiskNumber $Number -DriveLetter $Letter -UseMaximumSize 
   Initialize-Volume -DriveLetter $Letter -FileSystem NTFS -Confirm:$false
}

# Join domain, restart the machine
Add-Computer -DomainName JOSE.TEST -Credential (Get-Credential) -Restart

 

 

6) Configure JOSE-A2 (File Server Cluster A, Node 2)

 

# Preparation steps: Install WS2012R2, rename computer
Rename-Computer -NewName JOSE-A2 -Restart

# Install required roles and features, restarts at the end
Install-WindowsFeature File-Services, FS-FileServer, Failover-Clustering
Install-WindowsFeature RSAT-Clustering -IncludeAllSubFeature

# Rename DHCP network adapter to Net1
Get-NetIPAddress -PrefixOrigin DHCP | Get-NetAdapter | Rename-NetAdapter -NewName Net1

# Configure Net2 with a static IP address for DNS / DC
Get-NetAdapter Eth* | ? Status -eq Up | ? InterfaceDescription -notmatch "Mellanox*" | Rename-NetAdapter -NewName Net2
Set-NetIPInterface -InterfaceAlias Net2 -DHCP Disabled
Remove-NetIPAddress -InterfaceAlias Net2 -Confirm:$false
New-NetIPAddress -InterfaceAlias Net2 -IPAddress 192.168.100.12 -PrefixLength 24
Set-DnsClientServerAddress -InterfaceAlias Net2 -ServerAddresses 192.168.100.1

# Configure NetR1 with a static IP address for the RDMA network
Get-NetAdapter -InterfaceDescription "*IPoIB*" | ? Status -eq Up | Rename-NetAdapter -NewName NetR1
Set-NetIPInterface -InterfaceAlias NetR1 -DHCP Disabled
Remove-NetIPAddress -InterfaceAlias NetR1 -Confirm:$false
New-NetIPAddress -InterfaceAlias NetR1 -IPAddress 192.168.101.12 -PrefixLength 24
Set-DnsClientServerAddress -InterfaceAlias NetR1 -ServerAddresses 192.168.100.1

# Disable all disconnected adapters
Get-NetAdapter -InterfaceDescription "*IPoIB*" | ? Status -ne Up | Rename-NetAdapter -NewName NetRX
Get-NetAdapter | ? Status -ne Up | Disable-NetAdapter -Confirm:$false

# Join domain, restart the machine
Add-Computer -DomainName JOSE.TEST -Credential (Get-Credential) –Restart

 

 

7) Configure JOSE-A3 (File Server Cluster A, Node 3)

 

# Preparation steps: Install WS2012R2, rename computer
Rename-Computer -NewName JOSE-A3 -Restart

# Install required roles and features, restarts at the end
Install-WindowsFeature File-Services, FS-FileServer, Failover-Clustering
Install-WindowsFeature RSAT-Clustering -IncludeAllSubFeature

# Rename DHCP network adapter to Net1
Get-NetIPAddress -PrefixOrigin DHCP | Get-NetAdapter | Rename-NetAdapter -NewName Net1

# Configure Net2 with a static IP address for DNS / DC
Get-NetAdapter Eth* | ? Status -eq Up | ? InterfaceDescription -notmatch "Mellanox*" | Rename-NetAdapter -NewName Net2
Set-NetIPInterface -InterfaceAlias Net2 -DHCP Disabled
Remove-NetIPAddress -InterfaceAlias Net2 -Confirm:$false
New-NetIPAddress -InterfaceAlias Net2 -IPAddress 192.168.100.13 -PrefixLength 24
Set-DnsClientServerAddress -InterfaceAlias Net2 -ServerAddresses 192.168.100.1

# Configure NetR1 with a static IP address for the RDMA network
Get-NetAdapter -InterfaceDescription "*IPoIB*" | ? Status -eq Up | Rename-NetAdapter -NewName NetR1
Set-NetIPInterface -InterfaceAlias NetR1 -DHCP Disabled
Remove-NetIPAddress -InterfaceAlias NetR1 -Confirm:$false
New-NetIPAddress -InterfaceAlias NetR1 -IPAddress 192.168.101.13 -PrefixLength 24
Set-DnsClientServerAddress -InterfaceAlias NetR1 -ServerAddresses 192.168.100.1

# Disable all disconnected adapters
Get-NetAdapter -InterfaceDescription "*IPoIB*" | ? Status -ne Up | Rename-NetAdapter -NewName NetRX
Get-NetAdapter | ? Status -ne Up | Disable-NetAdapter -Confirm:$false

# Join domain, restart the machine
Add-Computer -DomainName JOSE.TEST -Credential (Get-Credential) -Restart

 

8) Configure File Server Cluster JOSE-A

 

# Validate cluster
Test-Cluster -Node JOSE-A1, JOSE-A2, JOSE-A3

# Create cluster
New-Cluster –Name JOSE-A -Node JOSE-A1, JOSE-A2, JOSE-A3

# Rename and configure networks
(Get-ClusterNetwork | ? {$_.Address -notlike "192.*" }).Name = "Net1"
(Get-ClusterNetwork | ? {$_.Address -like "192.168.100.*" }).Name = "Net2"
(Get-ClusterNetwork | ? {$_.Address -like "192.168.101.*" }).Name = "NetR1"
(Get-ClusterNetwork Net1).Role = 1
(Get-ClusterNetwork Net2).Role = 3
(Get-ClusterNetwork NetR1).Role = 3
(Get-Cluster).UseClientAccessNetworksForSharedVolumes=1

# Remove default DHCP-based IP address for JOSE-A and add 2 IP addresses on 100/101 networks
Stop-ClusterResource "Cluster Name"
Get-ClusterResource | ? { (($_.ResourceType -like "*Address*") -and ($_.OwnerGroup -eq "Cluster Group")) } | Remove-ClusterResource –Force

Add-ClusterResource -Name "Cluster IP Address 100" -Group "Cluster Group" -ResourceType "IP Address"
Get-ClusterResource –Name "Cluster IP Address 100" | Set-ClusterParameter -Multiple @{ “Network”=”Net2”; "Address"="192.168.100.19"; ”SubnetMask”=”255.255.255.0”; "EnableDhcp"=0 }
Get-ClusterResource “Cluster Name” | Add-ClusterResourceDependency –Resource "Cluster IP Address 100"

Add-ClusterResource -Name "Cluster IP Address 101" -Group "Cluster Group" -ResourceType "IP Address"
Get-ClusterResource –Name "Cluster IP Address 101" | Set-ClusterParameter -Multiple @{ “Network”=”NetR1”; "Address"="192.168.101.19"; ”SubnetMask”=”255.255.255.0”; "EnableDhcp"=0 }
Get-ClusterResource “Cluster Name” | Add-ClusterResourceDependency –Resource "Cluster IP Address 101"

Set-ClusterResourceDependency -Resource "Cluster Name" -Dependency "[Cluster IP Address 100] OR [Cluster IP Address 101] "
Start-ClusterResource "Cluster Name"

# Rename Witness Disk
$w = Get-ClusterResource | ? { $_.OwnerGroup -eq "Cluster Group" -and $_.ResourceType -eq "Physical Disk"}
$w.Name = "WitnessDisk"

# Add remaining disks to Cluster Shared Volumes
Get-ClusterResource | ? OwnerGroup -eq "Available Storage" | Add-ClusterSharedVolume

# Create Scale-Out File Server
Add-ClusterScaleOutFileServerRole -Name JOSE-F

 

9) Configure JOSE-B1 (Hyper-V Host Cluster B, Node 1)

 

# Preparation steps: Install WS2012R2, rename computer
Rename-Computer -NewName JOSE-B1 -Restart

# Install required roles and features, restarts at the end
Install-WindowsFeature Failover-Clustering
Install-WindowsFeature RSAT-Clustering -IncludeAllSubFeature
Install-WindowsFeature Hyper-V, Hyper-V-PowerShell, Hyper-V-Tools -Restart

# Rename DHCP network adapter to Net1
Get-NetIPAddress -PrefixOrigin DHCP | Get-NetAdapter | Rename-NetAdapter -NewName Net1
New-VMSwitch -NetAdapterName Net1 -Name Net1
Rename-NetAdapter -Name "vEthernet (Net1)" -NewName VNet1

# Configure Net2 with a static IP address for DNS / DC
Get-NetAdapter Eth* | ? Status -eq Up | ? InterfaceDescription -notmatch "Mellanox*" | Rename-NetAdapter -NewName Net2
New-VMSwitch -NetAdapterName Net2 -Name Net2
Rename-NetAdapter -Name "vEthernet (Net2)" -NewName VNet2
Set-NetIPInterface -InterfaceAlias VNet2 -DHCP Disabled
Remove-NetIPAddress -InterfaceAlias VNet2 -Confirm:$false
New-NetIPAddress -InterfaceAlias VNet2 -IPAddress 192.168.100.21 -PrefixLength 24
Set-DnsClientServerAddress -InterfaceAlias VNet2 -ServerAddresses 192.168.100.1

# Configure NetR1 with a static IP address for the RDMA network
Get-NetAdapter -InterfaceDescription "*IPoIB*" | ? Status -eq Up | Rename-NetAdapter -NewName NetR1
Set-NetIPInterface -InterfaceAlias NetR1 -DHCP Disabled
Remove-NetIPAddress -InterfaceAlias NetR1 -Confirm:$false
New-NetIPAddress -InterfaceAlias NetR1 -IPAddress 192.168.101.21 -PrefixLength 24
Set-DnsClientServerAddress -InterfaceAlias NetR1 -ServerAddresses 192.168.100.1

# Disable all disconnected adapters
Get-NetAdapter -InterfaceDescription "*IPoIB*" | ? Status -ne Up | Rename-NetAdapter -NewName NetRX
Get-NetAdapter | ? Status -ne Up | Disable-NetAdapter -Confirm:$false

# Join domain, restart the machine
Add-Computer -DomainName JOSE.TEST -Credential (Get-Credential) -Restart

 

 

10) Configure JOSE-B2 (Hyper-V Host Cluster B, Node 2)

 

# Preparation steps: Install WS2012R2, rename computer
Rename-Computer -NewName JOSE-B2 -Restart

# Install required roles and features, restarts at the end
Install-WindowsFeature Failover-Clustering
Install-WindowsFeature RSAT-Clustering -IncludeAllSubFeature
Install-WindowsFeature Hyper-V, Hyper-V-PowerShell, Hyper-V-Tools -Restart

# Rename DHCP network adapter to Net1
Get-NetIPAddress -PrefixOrigin DHCP | Get-NetAdapter | Rename-NetAdapter -NewName Net1
New-VMSwitch -NetAdapterName Net1 -Name Net1
Rename-NetAdapter -Name "vEthernet (Net1)" -NewName VNet1

# Configure Net2 with a static IP address for DNS / DC
Get-NetAdapter Eth* | ? Status -eq Up | ? InterfaceDescription -notmatch "Mellanox*" | Rename-NetAdapter -NewName Net2
New-VMSwitch -NetAdapterName Net2 -Name Net2
Rename-NetAdapter -Name "vEthernet (Net2)" -NewName VNet2
Set-NetIPInterface -InterfaceAlias VNet2 -DHCP Disabled
Remove-NetIPAddress -InterfaceAlias VNet2 -Confirm:$false
New-NetIPAddress -InterfaceAlias VNet2 -IPAddress 192.168.100.22 -PrefixLength 24
Set-DnsClientServerAddress -InterfaceAlias VNet2 -ServerAddresses 192.168.100.1

# Configure NetR1 with a static IP address for the RDMA network
Get-NetAdapter -InterfaceDescription "*IPoIB*" | ? Status -eq Up | Rename-NetAdapter -NewName NetR1
Set-NetIPInterface -InterfaceAlias NetR1 -DHCP Disabled
Remove-NetIPAddress -InterfaceAlias NetR1 -Confirm:$false
New-NetIPAddress -InterfaceAlias NetR1 -IPAddress 192.168.101.22 -PrefixLength 24
Set-DnsClientServerAddress -InterfaceAlias NetR1 -ServerAddresses 192.168.100.1

# Disable all disconnected adapters
Get-NetAdapter -InterfaceDescription "*IPoIB*" | ? Status -ne Up | Rename-NetAdapter -NewName NetRX
Get-NetAdapter | ? Status -ne Up | Disable-NetAdapter -Confirm:$false

# Join domain, restart the machine
Add-Computer -DomainName JOSE.TEST -Credential (Get-Credential) -Restart

 

 

11) Configure Hyper-V Host Cluster JOSE-B

 

# Validate cluster
Test-Cluster -Node JOSE-B1, JOSE-B2

# Create cluster
New-Cluster –Name JOSE-B -Node JOSE-B1, JOSE-B2
# Rename and configure networks
(Get-ClusterNetwork | ? {$_.Address -notlike "192.*" }).Name = "Net1"
(Get-ClusterNetwork | ? {$_.Address -like "192.168.100.*" }).Name = "Net2"
(Get-ClusterNetwork | ? {$_.Address -like "192.168.101.*" }).Name = "NetR1"
(Get-ClusterNetwork Net1).Role = 1
(Get-ClusterNetwork Net2).Role = 3
(Get-ClusterNetwork NetR1).Role = 3
(Get-Cluster).UseClientAccessNetworksForSharedVolumes=1

# Remove default DHCP-based IP address for JOSE-A and add 2 IP addresses on 100/101 networks
Stop-ClusterResource "Cluster Name"
Get-ClusterResource | ? { (($_.ResourceType -like "*Address*") -and ($_.OwnerGroup -eq "Cluster Group")) } | Remove-ClusterResource –Force

Add-ClusterResource -Name "Cluster IP Address 100" -Group "Cluster Group" -ResourceType "IP Address"
Get-ClusterResource –Name "Cluster IP Address 100" | Set-ClusterParameter -Multiple @{ “Network”=”Net2”; "Address"="192.168.100.29"; ”SubnetMask”=”255.255.255.0”; "EnableDhcp"=0 }
Get-ClusterResource “Cluster Name” | Add-ClusterResourceDependency –Resource "Cluster IP Address 100"

Add-ClusterResource -Name "Cluster IP Address 101" -Group "Cluster Group" -ResourceType "IP Address"
Get-ClusterResource –Name "Cluster IP Address 101" | Set-ClusterParameter -Multiple @{ “Network”=”NetR1”; "Address"="192.168.101.29"; ”SubnetMask”=”255.255.255.0”; "EnableDhcp"=0 }
Get-ClusterResource “Cluster Name” | Add-ClusterResourceDependency –Resource "Cluster IP Address 101"

Set-ClusterResourceDependency -Resource "Cluster Name" -Dependency "[Cluster IP Address 100] OR [Cluster IP Address 101] "
Start-ClusterResource "Cluster Name"

# Create Share for VMs (run from JOSE-A1)
1..6 | % {
MD C:\ClusterStorage\Volume$_\Share
New-SmbShare -Name SHARE$_ -Path C:\ClusterStorage\Volume$_\Share -FullAccess JOSE.Test\Administrator, JOSE.Test\JOSE-B1$, JOSE.Test\JOSE-B2$, JOSE.Test\JOSE-B$
Set-SmbPathAcl -ShareName SHARE$_
}

# Create Share for File Share Witness (run from JOSE-A1)
MD C:\ClusterStorage\Volume6\Witness
New-SmbShare -Name Witness -Path C:\ClusterStorage\Volume6\Witness -FullAccess JOSE.Test\Administrator, JOSE.Test\JOSE-B$
Set-SmbPathAcl -ShareName Witness

# Configure JOSE-B Cluster with a File Share Witness (run from JOSE-B1)
Set-ClusterQuorum -NodeAndFileShareMajority \\JOSE-F.JOSE.TEST\Witness

 

12) Configure VMs on the Hyper-V Host Cluster JOSE-B

 

# Create VHD files for VMs (run on JOSE-B1)
New-VHD -Path \\JOSE-F.JOSE.TEST\Share1\VM1OS.VHDX -Fixed -SizeBytes 40GB
New-VHD -Path \\JOSE-F.JOSE.TEST\Share2\VM2OS.VHDX -Fixed -SizeBytes 40GB
New-VHD -Path \\JOSE-F.JOSE.TEST\Share3\VM12Witness.VHDX -Fixed -SizeBytes 1GB
New-VHD -Path \\JOSE-F.JOSE.TEST\Share3\VM12Data.VHDX -Fixed -SizeBytes 10GB

# Create VM1 (run on JOSE-B1)
New-VM -Path \\JOSE-F.JOSE.TEST\Share1 -Name VM1 -VHDPath \\JOSE-F.JOSE.TEST\Share1\VM1OS.VHDX -SwitchName Net1 -Memory 2GB
Get-VMProcessor * | Set-VMProcessor -CompatibilityForMigrationEnabled 1
Add-VMNetworkAdapter -VMName VM1 -SwitchName Net2
Add-VMHardDiskDrive -VMName VM1 -Path \\JOSE-F.JOSE.TEST\Share3\VM12Witness.VHDX-ShareVirtualDisk
Add-VMHardDiskDrive -VMName VM1 -Path \\JOSE-F.JOSE.TEST\Share3\VM12Data.VHDX-ShareVirtualDisk
Set-VMDvdDrive -VMName VM1 -Path D:\WindowsServer2012.ISO
Start-VM VM1
Add-VMToCluster VM1

# Create VM2 (run on JOSE-B2)
New-VM -Path \\JOSE-F.JOSE.TEST\Share2 -Name VM2 -VHDPath \\JOSE-F.JOSE.TEST\Share2\VM2OS.VHDX -SwitchName Net1 -Memory 2GB
Get-VMProcessor * | Set-VMProcessor -CompatibilityForMigrationEnabled 1
Add-VMNetworkAdapter -VMName VM2 -SwitchName Net2
Add-VMHardDiskDrive -VMName VM2 -Path \\JOSE-F.JOSE.TEST\Share3\VM12Witness.VHDX-ShareVirtualDisk
Add-VMHardDiskDrive -VMName VM2 -Path \\JOSE-F.JOSE.TEST\Share3\VM12Data.VHDX-ShareVirtualDisk
Set-VMDvdDrive -VMName VM2 -Path D:\WindowsServer2012.ISO
Start-VM VM2
Add-VMToCluster VM2

 

13) Configure JOSE-X1 (SQL Server Guest Cluster X, Node 1)

 

# Preparation steps: Install WS2012R2, rename computer, install Hyper-V IC
Rename-Computer -NewName JOSE-X1 -Restart

# Install required roles and features, restarts at the end
Install-WindowsFeature File-Services, FS-FileServer, Failover-Clustering
Install-WindowsFeature RSAT-Clustering -IncludeAllSubFeature

# Rename the two virtual ports as Net1 and Net2
Get-NetIPAddress -PrefixOrigin DHCP | Get-NetAdapter | Rename-NetAdapter -NewName Net1
Get-NetAdapter Ethernet* | Rename-NetAdapter -NewName Net2

# Configure Net2 with a static IP address for DNS / DC
Set-NetIPInterface -InterfaceAlias Net2 -DHCP Disabled
Remove-NetIPAddress -InterfaceAlias Net2 -Confirm:$false
New-NetIPAddress -InterfaceAlias Net2 -IPAddress 192.168.100.41 -PrefixLength 24
Set-DnsClientServerAddress -InterfaceAlias Net2 -ServerAddresses 192.168.100.1

# Configure 2 shared disks
1..2 | % { 
    $Letter ="JK"[($_-1)]
    Set-Disk -Number $_ -IsReadOnly 0
    Set-Disk -Number $_ -IsOffline 0
    Initialize-Disk -Number $_ -PartitionStyle MBR
    New-Partition -DiskNumber $_ -DriveLetter $Letter -UseMaximumSize 
    Initialize-Volume -DriveLetter $Letter -FileSystem NTFS -Confirm:$false
}

# Join domain, restart the machine
Add-Computer -DomainName JOSE.TEST -Credential (Get-Credential) –Restart

 

14) Configure JOSE-X2 (SQL Server Guest Cluster X, Node 2)

 

# Preparation steps: Install WS2012R2, rename computer, install Hyper-V IC
Rename-Computer -NewName JOSE-X2 -Restart

# Install required roles and features, restarts at the end
Install-WindowsFeature File-Services, FS-FileServer, Failover-Clustering
Install-WindowsFeature RSAT-Clustering -IncludeAllSubFeature

# Rename the two virtual ports as Net1 and Net2
Get-NetIPAddress -PrefixOrigin DHCP | Get-NetAdapter | Rename-NetAdapter -NewName Net1
Get-NetAdapter Ethernet* | Rename-NetAdapter -NewName Net2

# Configure Net2 with a static IP address for DNS / DC
Set-NetIPInterface -InterfaceAlias Net2 -DHCP Disabled
Remove-NetIPAddress -InterfaceAlias Net2 -Confirm:$false
New-NetIPAddress -InterfaceAlias Net2 -IPAddress 192.168.100.42 -PrefixLength 24
Set-DnsClientServerAddress -InterfaceAlias Net2 -ServerAddresses 192.168.100.1

# Shared Disks already configured on the first node

# Join domain, restart the machine
Add-Computer -DomainName JOSE.TEST -Credential (Get-Credential) –Restart

 

15) Configure SQL Server Guest Cluster JOSE-X

 

# Validate cluster
Test-Cluster -Node JOSE-X1, JOSE-X2

# Create cluster
New-Cluster –Name JOSE-X -Node JOSE-X1, JOSE-X2
# Rename and configure networks
(Get-ClusterNetwork | ? {$_.Address -notlike "192.*" }).Name = "Net1"
(Get-ClusterNetwork | ? {$_.Address -like "192.168.100.*" }).Name = "Net2"
(Get-ClusterNetwork Net1).Role = 1
(Get-ClusterNetwork Net2).Role = 3

# Remove default DHCP-based IP address for JOSE-V and add IP address on 100 network
Stop-ClusterResource "Cluster Name"
Get-ClusterResource | ? { (($_.ResourceType -like "*Address*") -and ($_.OwnerGroup -eq "Cluster Group")) } | Remove-ClusterResource –Force

Add-ClusterResource -Name "Cluster IP Address" -Group "Cluster Group" -ResourceType "IP Address"
Get-ClusterResource –Name "Cluster IP Address" | Set-ClusterParameter -Multiple @{ “Network”=”Net2”; "Address"="192.168.100.49"; ”SubnetMask”=”255.255.255.0”; "EnableDhcp"=0 }
Get-ClusterResource “Cluster Name” | Add-ClusterResourceDependency –Resource "Cluster IP Address"

Start-ClusterResource "Cluster Name"

# Rename Witness Disk
$w = Get-ClusterResource | ? { $_.OwnerGroup -eq "Cluster Group" -and $_.ResourceType -eq "Physical Disk"}
$w.Name = "WitnessDisk"

# Install SQL Server on nodes X1 and X2
# Use IP address 192.168.100.48 for the SQL Server group

 

 

 

16) Configure JOSE-V (VMM Server)

 

# Preparation steps: Install WS2012R2, rename computer
Rename-Computer -NewName JOSE-V -Restart

# Install required roles and features, restarts at the end
Install-WindowsFeature File-Services, FS-FileServer, Failover-Clustering
Install-WindowsFeature RSAT-Clustering -IncludeAllSubFeature

# Rename DHCP network adapter to Net1
Get-NetIPAddress -PrefixOrigin DHCP | Get-NetAdapter | Rename-NetAdapter -NewName Net1

# Configure Net2 with a static IP address for DNS / DC
Get-NetAdapter Eth* | ? Status -eq Up | ? InterfaceDescription -notmatch "Mellanox*" | Rename-NetAdapter -NewName Net2
Set-NetIPInterface -InterfaceAlias Net2 -DHCP Disabled
Remove-NetIPAddress -InterfaceAlias Net2 -Confirm:$false
New-NetIPAddress -InterfaceAlias Net2 -IPAddress 192.168.100.31 -PrefixLength 24
Set-DnsClientServerAddress -InterfaceAlias Net2 -ServerAddresses 192.168.100.1

# Configure NetR1 with a static IP address for the RDMA network
Get-NetAdapter -InterfaceDescription "*IPoIB*" | ? Status -eq Up | Rename-NetAdapter -NewName NetR1
Set-NetIPInterface -InterfaceAlias NetR1 -DHCP Disabled
Remove-NetIPAddress -InterfaceAlias NetR1 -Confirm:$false
New-NetIPAddress -InterfaceAlias NetR1 -IPAddress 192.168.101.31 -PrefixLength 24
Set-DnsClientServerAddress -InterfaceAlias NetR1 -ServerAddresses 192.168.100.1

# Disable all disconnected adapters
Get-NetAdapter -InterfaceDescription "*IPoIB*" | ? Status -ne Up | Rename-NetAdapter -NewName NetRX
Get-NetAdapter | ? Status -ne Up | Disable-NetAdapter -Confirm:$false

# Join domain, restart the machine
Add-Computer -DomainName JOSE.TEST -Credential (Get-Credential) –Restart

# Install .NET Framework
Install-WindowsFeature NET-Framework-Core

# Install Windows Server 2012 R2 ADK
D:\adk\adksetup.exe
# Select only the “Deployment Tools” and “Windows PE” options

# Install SQL Server 2012 SP1
# New SQL Server standalone installation, Feature Selection: Database Engine Services

# Install VMM 2012 R2
# Features selected to be added: VMM management server, VMM console
# Database: VirtualManagerDB database will be created on JOSE-V
# Service Account: Local System account

 

17) Running the Scale-Out demo (from JOSE-V)

 

# Prepare - Map the SMB shares
1..6 | % { 
    $d ="PQRSTU"[($_-1)] + “:”
New-SmbMapping -LocalPath $d -RemotePath \\JOSE-f\Share$_ -Persistent $true
}

# Prepare - Create the test files
1..6 | % { 
   $f ="PQRSTU"[($_-1)] + “:\testfile.dat”
fsutil file createnew $f (5GB)
   fsutil file setvaliddata $f (5GB)
}

# Remove the third cluster node (demo starts with 2 nodes)
Remove-ClusterNode -Cluster JOSE-A -Name JOSE-A3

 

Start Performance Monitor

Start a performance monitor session

Switch to “Histogram Bar” view to show the performance side-by-side

Add a counter for SMB Server Shares, Data bytes/sec, _total instance for JOSE-A1, JOSE-A2 and JOSE-A3.

clip_image003

 

Query the cluster shared volume ownership on Cluster A, with 2 nodes

Get-ClusterSharedVolume -Cluster JOSE-A | Sort OwnerNode | FT OwnerNode, Name, State -AutoSize

OwnerNode Name           State
--------- ----           -----
JOSE-A1   Cluster Disk 6 Online
JOSE-A1   Cluster Disk 3 Online
JOSE-A1   Cluster Disk 4 Online
JOSE-A2   Cluster Disk 7 Online
JOSE-A2   Cluster Disk 2 Online
JOSE-A2   Cluster Disk 5 Online

 

Run SQLIO to issue 8KB IOs

C:\sqlio\sqlio.exe -s9999 -T100 -t1 -o16 -b8 -BN -LS -c2000 -frandom -dPQRSTU testfile.dat

clip_image001[7]

 

Add a 3rd node and wait for it to take 1/3 of the load

Add-ClusterNode -Cluster JOSE-A -Name JOSE-A3

# Wait 2 and a half minutes to transition to the following state.

clip_image002[9]

 

 

Re- query the cluster shared volume ownership on Cluster A, now with 3 nodes

Get-ClusterSharedVolume -Cluster JOSE-A | Sort OwnerNode | FT OwnerNode, Name, State -AutoSize

OwnerNode Name           State
--------- ----           -----
JOSE-A1   Cluster Disk 3 Online
JOSE-A1   Cluster Disk 6 Online
JOSE-A2   Cluster Disk 5 Online
JOSE-A2   Cluster Disk 7 Online
JOSE-A3   Cluster Disk 2 Online
JOSE-A3   Cluster Disk 4 Online

 

18) Configuring VMM on JOSE-V

 

Bring the File Server Cluster under VMM management

Select Fabric and use the option to add Storage Devices

clip_image001

Add a Windows-based file server

clip_image002[7]

Specify the full path to the file server cluster:

clip_image003[11]

 

Verify the File Server Cluster was properly discovered by VMM

Check the provider

clip_image004

Check the Storage Spaces discovery

clip_image005

Check the Scale-Out File Server and file share discovery

clip_image006

 

 

Remove the File Server Cluster node (demo starts with 2 nodes)

Under the properties of the File Server Cluster, remove node 3

clip_image007

Check progress under running Jobs

clip_image008[6]

 

 

While running a workload, add a File Server Cluster node

Under the properties of the File Server Cluster, add node 3 specifying the full path of to the server

clip_image009

Check progress under running Jobs

clip_image010

 

 

19) Verifying systems’ configuration

 

# Commands to verify the configuration of all systems

"B1", "B2" | % { Get-VM -ComputerName JOSE-$_ }

"B1", "B2" | % { $_; Get-VM -ComputerName JOSE-$_ | Get-VMHardDiskDrive | FT VMName, ControllerType, ControllerLocation, Path -AutoSize}

Get-SmbOpenFile -CimSession JOSE-A1 | Sort ClientUserName, ShareRelativePath | FT ClientUserName, ShareRelativePath –AutoSize

"X1", "X2" | % { $_; Get-Disk -CimSession JOSE-$_ } | FT -AutoSize

"A", "B", "X" | % { $_; Get-ClusterNode -Cluster JOSE-$_ | FT Cluster, NodeName, State, Id -AutoSize }

"A", "B", "X" | % { $_; Get-ClusterResource -Cluster JOSE-$_ | FT -AutoSize}

 

20) Issues and FAQs (Frequently asked questions)

 

Failover Cluster creation in the guest fails.

  • Make sure you’re logged on as a domain user, not a local user
  • Make sure the Windows Server 2012 R2 integration components are installed in the guest.
  • Check for individual warnings and errors in the Failover Clustering validation report.

 

I can’t add a shared VHDX to a VM. I get a message saying that “the storage where the virtual hard disk is located does not support virtual hard disk sharing.”

  • Make sure your using a CSV disk or an SMB Scale-Out file share to store your VHDX files
  • Make sure the SMB Scale-Out file server cluster is running Windows Server 2012 R2

 

21) Final Notes

 

  • Keep in mind that there are dependencies between the services running on each computer.
  • To shut them down, start with VMM server, then the Hyper-V servers, then the File Server and finally the DNS/DC, waiting for each one to go down completely before moving to the next one.
  • To bring the servers up again, go from the DNS/DC to the File Server Cluster, then the Hyper-V Cluster and finally the VMM Server, waiting for the previous one to be fully up before starting the next one.
  • I hope you enjoyed these step-by-step instructions. I strongly encourage you to try them out and perform the entire installation yourself. It’s a good learning experience.
  • Let me know how these steps worked for you using the comment section. If you run into any issues or found anything particularly interesting, don’t forget to mention the number of the step.

 

To see all of the posts in this series, check out the What’s New in Windows Server & System Center 2012 R2 archive.


Windows Server 2012 R2 Storage: Step-by-step with Storage Spaces, SMB Scale-Out and Shared VHDX (Virtual)

$
0
0

This post is a part of the nine-part “What’s New in Windows Server & System Center 2012 R2” series that is featured on Brad Anderson’s In the Cloud blog.  Today’s blog post covers Windows Server 2012 R2 Storage and how it applies to the larger topic of “Transform the Datacenter.”  To read that post and see the other technologies discussed, read today’s post: “What’s New in 2012 R2: IaaS Innovations.”

  

1) Overview

 

In this document, I am sharing all the steps I used to create a Windows Server 2012 R2 File Server demo or test environment, so you can experiment with some of the new technologies yourself. You only need a single computer (the specs are provided below) and the ISO file with the Windows Server 2012 R2 Preview available as a free download.

 

The demo setup includes 5 virtual machines: 1 domain controller, 3 file servers and 1 file client/VMM Server. It uses the new Shared VHDX feature to provide shared storage for the guest failover cluster and it showcases both Storage Spaces and Scale-Out File Servers.

 

If you're not familiar with these technologies, I would strong encourage you to review the TechEd 2013 talks on Hyper-V over SMB (Understanding the Hyper-V over SMB Scenario, Configurations, and End-to-End Performance) and Shared VHDX (second half of Application Availability Strategies for the Private Cloud).

 

Here’s a diagram of the setup:

clip_image002

 

Here are the details about the names, roles and IP addresses for each of the computers involved:

VMComputerRoleNet2 (DC/DNS)
HostJOSE-EWHyper-V Host192.168.100.99/24
VM1JOSE-DDNS, DC192.168.101.100/24
VM2JOSE-A1Cluster Node 1192.168.102.101/24
VM3JOSE-A2Cluster Node 2192.168.100.102/24
VM4JOSE-A3Cluster Node 3192.168.100.103/24
VM5JOSE-VVMM / Client192.168.100.104/24
 JOSE-ACluster192.168.100.110/24
 JOSE-FScale-Out​ File ServerN/A

 

Following these steps will probably require a few hours of work end-to-end, but it is a great way to experiment with a large set of Microsoft technologies in or related to Windows Server 2012 R2, including:

  • Hyper-V
  • Networking
  • Domain Name Services (DNS)
  • Active Directory Domain Services (AD-DS)
  • Shared VHDX
  • Storage Spaces
  • Failover Clustering
  • File Servers
  • PowerShell
  • SQL Server
  • Virtual Machine Manager

 

2) Required Hardware and Software, plus Disclaimers

 

You will need the following hardware to perform the steps described here:

 

You will need the following software to perform the steps described here:

 

Notes and disclaimers:

  • A certain familiarity with Windows administration and configuration is assumed. If you're new to Windows, this document is not for you. Sorry...
  • If you are asked a question or required to perform an action that you do not see described in these steps, go with the default option.
  • There are usually several ways to perform a specific configuration or administration task. What I describe here is one of those many ways. It's not necessarily the best way, just the one I personally like best at the moment.
  • For the most part, I use PowerShell to configure the systems. You can also use a graphical interface instead, but I did not describe those steps here.
  • Obviously, a single-computer solution can never be tolerant to the failure of that computer. So, the configuration described here is not really continuously available. It’s just a simulation.
  • The specific Shared VHDX configuration shown in this blog post is not supported. Microsoft Support will only answer questions and assist in troubleshooting configurations where the Shared VHDX files are stored either on a Cluster Shared Volume (CSV) directly or on a file share on a Scale-Out File Server Cluster.
  • The specific Storage Spaces configuration shown in this blog post is not supported. Microsoft Support will only answer questions and assist in troubleshooting configurations where Storage Spaces uses a physical machine (not a VM) and uses one of the certified JBOD hardware solutions (see http://www.windowsservercatalog.com/results.aspx?bCatID=1573&cpID=0&avc=10&OR=1)
  • Because of the two items above, the configuration described here should only be used for demos, testing or learning environments.
  • The Storage Spaces configuration shown here is not capable of showcasing some of the new features like Tiering or Write-back Cache, since it uses virtual disks without a proper media type. To use those features, you’ll need a storage pool with physical SSDs and HDDs.

 

3) Summarized instructions for experts

 

If you are already familiar with Failover Clustering, Scale-Out File Servers and Hyper-V, here’s a short introduction to what it takes to configure this environment on a single computer. A detailed step-by-step is available in the following sections of this document.

  • Configure a Hyper-V capable computer with at least 8GB of RAM with Windows Server 2012 R2. Make sure to load the Failover Clustering feature. No need to actually create a failover cluster on the physical machine.
  • Manually attach the Shared VHDX filter to the disk where the VHDX files will be located:
    fltmc.exe attach svhdxflt D:\
  • Create OS VHD or VHDX files as you would regularly, on an SMB file share. Create your VMs as you would regularly would, using the OS VHD or VHDX file. Both Generation 1 and Generation 2 VMs are fine.
  • Create your VHDX data files to be shared as fixed-size or dynamically expanding, on the disk where you manually attached the Shared VHDX filter. Old VHD files are not allowed. Differencing disks are not allowed.
  • Add the shared VHDX data files to multiple VMs, using the Add-VMHardDiskDrive and the “-ShareVirtualDisk” option. If using the GUI, check the box for “Share virtual disk” when adding the VHDX data files to the VM.
  • Inside the VM, install Windows Server 2012 R2 or Windows Server 2012. Make sure to install the Hyper-V integration components.
  • Configure Storage Spaces using the Virtual SAS disks exposed to the VMs.
  • Validate and configure the cluster inside the VMs as you normally would. Configure a Scale-Out file server and create the file shares for testing.
  • Install VMM 2012 R2 Preview and use it to manage the Scale-Out File Server.

 

The rest of this document contains detailed step-by-step instructions for each of the items outlined above.

 

4) Configure the physical host

 

# Preparation steps: Install WS2012R2, rename computer, enable remote desktop

# Install required roles and features, restart at the end
Install-WindowsFeature Failover-Clustering -IncludeManagementTools
Install-WindowsFeature Hyper-V -IncludeManagementTools -Restart

# Create a new Internal Switch
New-VMSwitch -Name Net2 -SwitchType Internal
Get-NetAdapter *Net2* | Rename-NetAdapter -NewName vNet2

# Configure vNet2 (parent port for internal switch) with a static IP address for DNS / DC
Set-NetIPInterface -InterfaceAlias vNet2 -DHCP Disabled
Remove-NetIPAddress -InterfaceAlias vNet2 -Confirm:$false
New-NetIPAddress -InterfaceAlias vNet2 -IPAddress 192.168.100.1 -PrefixLength 24
Set-DnsClientServerAddress -InterfaceAlias vNet2 -ServerAddresses 192.168.100.1

# Create 5 VMs, assumes there is a base VM ready at D:\VMS\BaseOS.VHDX
1..5 | % { New-VHD -Path D:\VMS\VM$_.VHDX –ParentPath D:\VMS\BaseOS.VHDX }
1..5 | % { New-VM -Name VM$_ -Path D:\VMS –VHDPath D:\VMS\VM$_.VHDX -Memory 2GB -SwitchName Net2 }
# Give the VMM VM a little more RAM
Set-VM -VMName VM5 -MemoryStartupBytes 4GB

# Create 3 data VHDX files
1..3 | % { New-VHD -Path D:\VMS\Data$_.VHDX -Fixed -SizeBytes 20GB }

# Manually attach volume D:\
fltMC.exe attach svhdxflt D:\
# NOTE 1: Non-CSV deployments of Shared VHDX are not supported. See the disclaimer section of this document.
# NOTE 2: Manual attached is not saved across reboots. You will have to re-issue the command after each reboot.

# Add the 3 data VHDX files to each of the 3 VMs, with Sharing option
1..3 | % { $p = ”D:\VMS\Data” + $_ + ”.VHDX” ; 2..4 | % { $v = ”VM” + $_; Write-Host $v, $p; Add-VMHardDiskDrive -VMName $v -Path $p -ShareVirtualDisk } }

# Start all the VMs
Get-VM | Start-VM

 

5) Configure JOSE-D (DNS, DC)

 

# Preparation steps: Install WS2012R2, rename computer, enable remote desktop

# Install required roles and features, restarts at the end
Install-WindowsFeature AD-Domain-Services, RSAT-ADDS, RSAT-ADDS-Tools

# Configure Internal NIC with a static IP address for DNS / DC
Get-NetAdapter | Rename-NetAdapter -NewName Internal
Set-NetIPInterface -InterfaceAlias Internal -DHCP Disabled
Remove-NetIPAddress -InterfaceAlias Internal -Confirm:$false
New-NetIPAddress -InterfaceAlias Internal -IPAddress 192.168.100.100 -PrefixLength 24
Set-DnsClientServerAddress -InterfaceAlias Internal -ServerAddresses 192.168.100.100

# create AD forest, reboots at the end
Install-ADDSForest -CreateDNSDelegation:$false -DatabasePath "C:\Windows\NTDS" -DomainMode "Win2012" -DomainName "JOSE.TEST" -DomainNetBIOSName "JOSE" -ForestMode "Win2012" -InstallDNS:$true -LogPath "C:\Windows\NTDS" -SYSVOLPath "C:\Windows\SYSVOL" 

 

6) Configure JOSE-A1 (Cluster A)

 

# Preparation steps: Install WS2012R2, rename computer, enable remote desktop

# Install required roles and features, restarts at the end
Install-WindowsFeature File-Services, FS-FileServer, Failover-Clustering
Install-WindowsFeature RSAT-Clustering -IncludeAllSubFeature

# Configure Internal NIC with a static IP address for DNS / DC
Get-NetAdapter | Rename-NetAdapter -NewName Internal
Set-NetIPInterface -InterfaceAlias Internal -DHCP Disabled
Remove-NetIPAddress -InterfaceAlias Internal -Confirm:$false
New-NetIPAddress -InterfaceAlias Internal -IPAddress 192.168.100.101 -PrefixLength 24
Set-DnsClientServerAddress -InterfaceAlias Internal -ServerAddresses 192.168.100.100

# Create 1 Pool, 7 Spaces and Initialize them
# NOTE 3: Running Storage Spaces in a guest is not supported. See the disclaimer section of this document.
# NOTE 4: This unsupported configuration cannot simulate Tiering or Write-back cache, since there are no SSDs.

$s = Get-StorageSubSystem -FriendlyName *Spaces*
New-StoragePool -FriendlyName Pool1 -StorageSubSystemFriendlyName $s.FriendlyName -PhysicalDisks (Get-PhysicalDisk -CanPool $true)
Set-ResiliencySetting -Name Mirror -NumberofColumnsDefault 1 -StoragePool (Get-StoragePool -FriendlyName Pool1)

New-VirtualDisk -FriendlyName Space1 -StoragePoolFriendlyName Pool1 -ResiliencySettingName Mirror –Size 1GB
2..7 | % { New-VirtualDisk -FriendlyName Space$_ -StoragePoolFriendlyName Pool1 -ResiliencySettingName Mirror –Size 3GB }

1..7 | % {
   $Letter ="PQRSTUV”[($_-1)]
$Number = (Get-VirtualDisk -FriendlyName Space$_ | Get-Disk).Number
Set-Disk -Number $Number -IsReadOnly 0
   Set-Disk -Number $Number -IsOffline 0
Initialize-Disk -Number $Number -PartitionStyle MBR
   New-Partition -DiskNumber $Number -DriveLetter $Letter -UseMaximumSize 
   Initialize-Volume -DriveLetter $Letter -FileSystem NTFS -Confirm:$false
}

# Join domain, restart the machine
Add-Computer -DomainName JOSE.TEST -Credential (Get-Credential) –Restart

 

7) Configure JOSE-A2 (Cluster A)

 

# Preparation steps: Install WS2012R2, rename computer, enable remote desktop

# Install required roles and features, restarts at the end
Install-WindowsFeature File-Services, FS-FileServer, Failover-Clustering
Install-WindowsFeature RSAT-Clustering -IncludeAllSubFeature

# Configure Internal NIC with a static IP address for DNS / DC
Get-NetAdapter | Rename-NetAdapter -NewName Internal
Set-NetIPInterface -InterfaceAlias Internal -DHCP Disabled
Remove-NetIPAddress -InterfaceAlias Internal -Confirm:$false
New-NetIPAddress -InterfaceAlias Internal -IPAddress 192.168.100.102 -PrefixLength 24
Set-DnsClientServerAddress -InterfaceAlias Internal -ServerAddresses 192.168.100.100

# Join domain, restart the machine
Add-Computer -DomainName JOSE.TEST -Credential (Get-Credential) –Restart

 

8) Configure JOSE-A3 (Cluster A)

 

# Preparation steps: Install WS2012R2, rename computer, enable remote desktop

# Install required roles and features, restarts at the end
Install-WindowsFeature File-Services, FS-FileServer, Failover-Clustering
Install-WindowsFeature RSAT-Clustering -IncludeAllSubFeature

# Configure Internal NIC with a static IP address for DNS / DC
Get-NetAdapter | Rename-NetAdapter -NewName Internal
Set-NetIPInterface -InterfaceAlias Internal -DHCP Disabled
Remove-NetIPAddress -InterfaceAlias Internal -Confirm:$false
New-NetIPAddress -InterfaceAlias Internal -IPAddress 192.168.100.103 -PrefixLength 24
Set-DnsClientServerAddress -InterfaceAlias Internal -ServerAddresses 192.168.100.100

# Join domain, restart the machine
Add-Computer -DomainName JOSE.TEST -Credential (Get-Credential) -Restart

 

9) Configure Cluster JOSE-A

 

# Validate cluster
Test-Cluster -Node JOSE-A1, JOSE-A2, JOSE-A3

# Create cluster
New-Cluster –Name JOSE-A -Node JOSE-A1, JOSE-A2, JOSE-A3 -StaticAddress 192.168.100.110

# Rename and configure networks
(Get-ClusterNetwork).Name = “Internal”
(Get-ClusterNetwork).Role = 3
(Get-Cluster).UseClientAccessNetworksForSharedVolumes=1

# Rename Witness Disk
$w = Get-ClusterResource | ? { $_.OwnerGroup -eq "Cluster Group" -and $_.ResourceType -eq "Physical Disk"}
$w.Name = "WitnessDisk"

# Add remaining disks to Cluster Shared Volumes
Get-ClusterResource | ? OwnerGroup -eq "Available Storage" | Add-ClusterSharedVolume

# Create Scale-Out File Server
Add-ClusterScaleOutFileServerRole JOSE-F

# Create SMB shares
1..6 | % {
MD C:\ClusterStorage\Volume$_\Share
New-SmbShare -Name Share$_ -Path C:\ClusterStorage\Volume$_\Share -FullAccess JOSE.Test\Administrator
Set-SmbPathAcl -ShareName Share$_
}

 

10) Configure JOSE-V

 

# Preparation steps: Install WS2012R2, rename computer, enable remote desktop

# Install required roles and features, restarts at the end
Install-WindowsFeature RSAT-Clustering -IncludeAllSubFeature

# Configure Internal NIC with a static IP address for DNS / DC
Get-NetAdapter | Rename-NetAdapter -NewName Internal
Set-NetIPInterface -InterfaceAlias Internal -DHCP Disabled
Remove-NetIPAddress -InterfaceAlias Internal -Confirm:$false
New-NetIPAddress -InterfaceAlias Internal -IPAddress 192.168.100.104 -PrefixLength 24
Set-DnsClientServerAddress -InterfaceAlias Internal -ServerAddresses 192.168.100.100

# Join domain, restart the machine
Add-Computer -DomainName JOSE.TEST -Credential (Get-Credential) -Restart

# Map the SMB shares
1..6 | % { 
    $d ="PQRSTU"[($_-1)] + “:”
New-SmbMapping -LocalPath $d -RemotePath \\JOSE-F\Share$_ -Persistent $true
}

# Create the test files
1..6 | % { 
   $f ="PQRSTU"[($_-1)] + “:\testfile.dat”
fsutil file createnew $f (256MB)
   fsutil file setvaliddata $f (256MB)
}

# Run SQLIO (assumes SQLIO.EXE was copied to C:\SLQIO)
c:\sqlio\sqlio.exe -s9999 -T100 -t1 -o16 -b8 -BN -LS -c2000 -frandom -dPQRSTU testfile.dat

# To remove a node while the workload is running (wait a few minutes for rebalancing)
Remove-ClusterNode -Cluster JOSE-A –Name JOSE-A3

# To add a node while the workload is running (wait a few minutes for rebalancing)
Add-ClusterNode -Cluster JOSE-A –Name JOSE-A3

 

 

11) Running the tests without VMM (from JOSE-V)

 

Start Performance Monitor

Start a performance monitor session

Switch to “Histogram Bar” view to show the performance side-by-side

Add a counter for SMB Server Shares, Data bytes/sec, _total instance for JOSE-A1, JOSE-A2 and JOSE-A3.

clip_image003

 

Query the cluster shared volume ownership on Cluster A, with 2 nodes

Get-ClusterSharedVolume -Cluster JOSE-A | Sort OwnerNode | FT OwnerNode, Name, State -AutoSize

OwnerNode Name           State
--------- ----           -----
JOSE-A1   Cluster Disk 6 Online
JOSE-A1   Cluster Disk 3 Online
JOSE-A1   Cluster Disk 4 Online
JOSE-A2   Cluster Disk 7 Online
JOSE-A2   Cluster Disk 2 Online
JOSE-A2   Cluster Disk 5 Online

 

Run SQLIO to issue 8KB IOs

C:\sqlio\sqlio.exe -s9999 -T100 -t1 -o16 -b8 -BN -LS -c2000 -frandom -dPQRSTU testfile.dat

clip_image004

  

Add a 3rd node and wait for it to take 1/3 of the load

Add-ClusterNode -Cluster JOSE-A -Name JOSE-A3

Wait 2 and a half minutes to transition to the following state.

clip_image005

 

Re- query the cluster shared volume ownership on Cluster A, now with 3 nodes

Get-ClusterSharedVolume -Cluster JOSE-A | Sort OwnerNode | FT OwnerNode, Name, State -AutoSize

OwnerNode Name           State
--------- ----           -----
JOSE-A1   Cluster Disk 3 Online
JOSE-A1   Cluster Disk 6 Online
JOSE-A2   Cluster Disk 5 Online
JOSE-A2   Cluster Disk 7 Online
JOSE-A3   Cluster Disk 2 Online
JOSE-A3   Cluster Disk 4 Online

 

12) Installing VMM on JOSE-V

 

# Install .NET Framework
Install-WindowsFeature NET-Framework-Core

# Install Windows Server 2012 R2 ADK
C:\ADK\adksetup.exe
# Select only the “Deployment Tools” and “Windows PE” options

# Install SQL Server 2012 SP1
# New SQL Server standalone installation, Feature Selection: Database Engine Services

# Install VMM 2012 R2
# Features selected to be added: VMM management server, VMM console
# Database: VirtualManagerDB database will be created on JOSEBDA-V
# Service Account: Local System account

 

13) Configuring VMM on JOSE-V

 

Bring the File Server Cluster under VMM management

Select Fabric and use the option to add Storage Devices

clip_image006

Add a Windows-based file server

clip_image007

Specify the full path to the file server cluster:

clip_image008

 

Verify the File Server Cluster was properly discovered by VMM

Check the provider

clip_image009

Check the Storage Spaces discovery

clip_image010

Check the Scale-Out File Server and file share discovery

clip_image011

 

 

Remove the File Server Cluster node (demo starts with 2 nodes)

Under the properties of the File Server Cluster, remove node 3

clip_image012

Check progress under running Jobs

clip_image013


 

While running a workload, add a File Server Cluster node

Under the properties of the File Server Cluster, add node 3 specifying the full path of to the server

clip_image014

Check progress under running Jobs

clip_image015

 

14) Verifying systems’ configuration

 PS C:\> Get-PhysicalDisk -CimSession JOSE-A1, JOSE-A2, JOSE-A3 | Sort PSComputerName, Size

 

PS C:\> Get-Disk -CimSession JOSE-A1, JOSE-A2, JOSE-A3 | Sort PSComputerName, Size

Number Friendly Name                  OperationalStatus Total Size Partition Style PSComputerName
------ -------------                  ----------------- ---------- --------------- --------------
4      Microsoft Storage Space Device Online                  1 GB MBR             JOSE-A1
10     Microsoft Storage Space Device Online                  3 GB MBR             JOSE-A1
8      Microsoft Storage Space Device Online                  3 GB MBR             JOSE-A1
0      Virtual HD ATA Device          Online                 40 GB MBR             JOSE-A1
4      Microsoft Storage Space Device Online                  3 GB MBR             JOSE-A2
5      Microsoft Storage Space Device Online                  3 GB MBR             JOSE-A2
0      Virtual HD ATA Device          Online                 40 GB MBR             JOSE-A2
4      Microsoft Storage Space Device Online                  3 GB MBR             JOSE-A3
5      Microsoft Storage Space Device Online                  3 GB MBR             JOSE-A3
0      Virtual HD ATA Device          Online                 40 GB MBR             JOSE-A3

PS C:\> Get-ClusterNode -Cluster JOSE-A | FT Cluster, NodeName, State, Id -AutoSize

Cluster NodeName State Id
------- -------- ----- --
JOSE-A  JOSE-A1     Up 2
JOSE-A  JOSE-A2     Up 1
JOSE-A  JOSE-A3     Up 3

PS C:\> Get-ClusterResource -Cluster JOSE-A | FT –AutoSize

Name                             State  OwnerGroup    ResourceType
----                             -----  ----------    ------------
Cluster IP Address               Online Cluster Group IP Address
Cluster Name                     Online Cluster Group Network Name
Cluster Pool 1                   Online Cluster Group Storage Pool
JOSE-F                           Online JOSE-F        Distributed Network Name
Scale-Out File Server (\\JOSE-F) Online JOSE-F        Scale Out File Server
WitnessDisk                      Online Cluster Group Physical Disk

PS C:\> Get-ClusterSharedVolume -Cluster JOSE-A | FT -AutoSize

Name           State  Node
----           -----  ----
Cluster Disk 2 Online JOSE-A3
Cluster Disk 3 Online JOSE-A2
Cluster Disk 4 Online JOSE-A3
Cluster Disk 5 Online JOSE-A1
Cluster Disk 6 Online JOSE-A2
Cluster Disk 7 Online JOSE-A1

PS C:\> Get-SmbShare Share* -CimSession JOSE-F | FT -AutoSize

Name   ScopeName Path                            Description PSComputerName
----   --------- ----                            ----------- --------------
Share1 JOSE-F    C:\ClusterStorage\Volume1\Share             JOSE-F
Share2 JOSE-F    C:\ClusterStorage\Volume2\SHARE             JOSE-F
Share3 JOSE-F    C:\ClusterStorage\Volume3\Share             JOSE-F
Share4 JOSE-F    C:\ClusterStorage\Volume4\Share             JOSE-F
Share5 JOSE-F    C:\ClusterStorage\Volume5\SHARE             JOSE-F
Share6 JOSE-F    C:\ClusterStorage\Volume6\Share             JOSE-F

  

15) Final Notes

 

  • Keep in mind that there are dependencies between the services running on each VM.
  • To shut them down, start with VM5 and end with VM1, waiting for each one to go down completely before moving to the next one.
  • To bring the VMs up again, go from VM1 to VM5, waiting for the previous one to be fully up (with low to no CPU usage) before starting the next one.
  • I hope you enjoyed these step-by-step instructions. I strongly encourage you to try them out and perform the entire installation yourself. It’s a good learning experience.
  • Let me know how these steps worked for you using the comment section. If you run into any issues or found anything particularly interesting, don’t forget to mention the number of the step.

 

To see all of the posts in this series, check out the What’s New in Windows Server & System Center 2012 R2 archive.

SNIA’s Storage Developer Conference 2013 is just a few weeks away

$
0
0

The Storage Networking Industry Association (SNIA) is hosting the 10th Storage Developer Conference (SDC) in the Hyatt Regency in beautiful Santa Clara, CA (Silicon Valley) on the week of September 16th. As usual, Microsoft is one of the underwriters of the SNIA SMB2/SMB3 PlugFest, which is co-located with the SDC event.

For developers working with storage-related technologies, this event gathers a unique crowd and includes a rich agenda that you can find at http://www.storagedeveloper.org. Many of the key industry players are represented and this year’s agenda lists presentations from EMC, Fujitsu, Google, Hortonworks, HP, Go Daddy, Huawei, IBM, Intel, Microsoft, NEC, NetApp, Netflix, Oracle, Red Hat, Samba Team, Samsung, Spectra Logic, SwitfTest, Tata and many others.

It’s always worth reminding you that the SDC presentations are usually delivered to developers by the actual product development teams and frequently the actual developer of the technology is either delivering the presentation or is in the room to take questions. That kind of deep insight is not common in every conference out there.

Presentations by Microsoft this year include:

TitlePresenters
Advancements in Windows File SystemsNeal Christiansen, Principal Development Lead, Microsoft
LRC Erasure Coding in Windows Storage SpacesCheng Huang, Researcher, Microsoft Research
SMB3 UpdateDavid Kruse, Development Lead, Microsoft
Cluster Shared VolumesVladimir Petter, Principal Software Design Engineer, Microsoft
Tunneling SCSI over SMB: Shared VHDX files for Guest Clustering in Windows Server 2012 R2Jose Barreto, Principal Program Manager, Microsoft
Matt Kurjanowicz, Software Development Engineer, Microsoft
Windows Azure Storage - Speed and Scale in the CloudJoe Giardino, Senior Development Lead, Microsoft
SMB Direct updateGreg Kramer, Sr. Software Engineer, Microsoft
Scaled RDMA Performance & Storage Design with Windows Server SMB 3.0Dan Lovinger, Principal Software Design Engineer, Microsoft
SPEC SFS Benchmark - The Next GenerationSpencer Shepler, Architect, Microsoft
Data Deduplication as a Platform for Virtualization and High Scale StorageAdi Oltean, Principal Software Design Engineer, Microsoft
Sudipta Sengupta, Sr. Researcher, Microsoft

For a taste of what SDC presentations look like, make sure to visit the site for last year’s event, where you can find the downloadable PDF files for most and video recordings for some. You can find them at http://www.snia.org/events/storage-developer2012/presentations12.

Registration for SDC 2013 is open at http://www.storagedeveloper.org and you should definitely plan to attend. If you are registered, leave a comment and let’s plan to meet when we get there!

Step-by-step for Storage Spaces Tiering in Windows Server 2012 R2

$
0
0

1. Overview

 

In this document, I am sharing all the steps I used to create a demo or test environment of a storage space with storage tiers on Windows Server 2012 R2 Preview so that you can experiment with some of the new technologies yourself. You need only a single computer with one SSD and one HDD for this demo (the specs are provided below).

If you're not familiar with Storage Spaces or storage tiers (also known as Storage Spaces Tiering), I would strong encourage you to review the Storage Spaces Overview and TechEd 2013 talks on “Storage Spaces”, which you can find below:

- Storage Spaces Overview

· Storage Spaces: What’s New in Windows Server 2012 R2

· Deploying Windows Server 2012 R2 File Services for Exceptional $/IOPS

 

2. Simulated environment with limited hardware

 

In a typical storage tier configuration (non-clustered), you have a JBOD with a few hard disks, some being SSDs and some being HDD. For instance, you could have 4 SSDs and 8 HDDs. You would then combine all 12 disks into a pool, create the two tiers (SSD and HDD) and then create spaces (virtual disks) on top of those. Here’s what it looks like:

image

However, if you’re just experimenting with this feature or trying to learn how to configure it, the investment in the hardware (1 JBOD, 4 SSDs, 8 HDDs, SAS HBA, cables) might be holding you back. So, in this blog post, we’ll show how to configure storage tiers using just one SSD and one HDD, with the help of Hyper-V. Here’s what the simulated environment looks like:

image

As you can see in the diagram, the basic goal here is to simulate a tiered storage space using 4 (simulated) SSDs and 8 (simulated) HDDs. We’ll show the Windows PowerShell cmdlets to create the VHDs and the VM required to simulate the environment. Then, inside the guest, we’ll create the pool, storage tiers and storage spaces (both simple and mirrored). To verify it’s all working, we’ll create some test files and run a simulated SQL workload. We’ll also look into the storage tier maintenance tasks and extending an existing storage space.

 

3. Required Hardware and Software, plus Disclaimers

 

You will need the following hardware to perform the steps described here:

You will need the following software to perform the steps described here:

Notes and disclaimers:

  • A certain familiarity with Windows administration and configuration is assumed. If you're new to Windows, this document is not for you. Sorry...
  • If you are asked a question or required to perform an action that you do not see described in these steps, go with the default option.
  • There are usually several ways to perform a specific configuration or administration task. What I describe here is one of those many ways. It's not necessarily the best way, just the one I personally like best at the moment.
  • For the most part, I use PowerShell to configure the systems. You can also use a graphical interface instead, but I did not describe those steps here.
  • The specific Storage Spaces configuration shown in this blog post is not supported. Microsoft Support will only answer questions and assist in troubleshooting configurations where Storage Spaces uses a physical machine (not a VM) and uses one of the certified JBOD hardware solutions (see http://www.windowsservercatalog.com/results.aspx?bCatID=1573&cpID=0&avc=10&OR=1)
  • Because of the item above, the configuration described here should only be used for demos, testing or learning environments.

 

4. Configure the physical host

 

# Preparation steps: Install Window Server 2012 R2 Preview

# Install required roles and features, restart at the end

Install-WindowsFeature Hyper-V -IncludeManagementTools –Restart

 

# Create 4 VHDX files on the SSD with 10GB each

1..4 | % { New-VHD -Path D:\VMS\VMA_SSD_$_.VHDX -Fixed –Size 10GB}

 

# Create 8 VHDX files on the HDD with 30GB each

1..8 | % { New-VHD -Path E:\VMS\VMA_HDD_$_.VHDX -Fixed –Size 30GB}

 

# Create a new VM. Assumes you have an Windows Server 2012 R2 OS VHDX in place

New-VM -Name VMA -Path D:\VMS –VHDPath D:\VMS\VMA_OS.VHDX -Memory 2GB

 

# Add all data disks to the VM

1..4 | % { Add-VMHardDiskDrive -VMName VMA -ControllerType SCSI -Path D:\VMS\VMA_SSD_$_.VHDX }

1..8 | % { Add-VMHardDiskDrive -VMName VMA -ControllerType SCSI -Path E:\VMS\VMA_HDD_$_.VHDX }

 

# Start the VM

Start-VM VMA

   

5. Check VM configuration (from the Host, with output)

 

 

PS C:\>Get-VM VMA
 

Name State   CPUUsage(%) MemoryAssigned(M) Uptime   Status
---- -----   ----------- ----------------- ------   ------
VMA  Running 0           2048              00:01:52 Operating normally
 

PS C:\>Get-VM VMA | Get-VMHardDiskDrive
 
VMName ControllerType ControllerNumber ControllerLocation DiskNumber Path
------ -------------- ---------------- ------------------ ---------- ----
VMA    IDE            0                0                             D:\VMS\VMA_OS.VHDX
VMA    SCSI           0                0                             D:\VMS\VMA_SSD_1.VHDX
VMA    SCSI           0                1                             D:\VMS\VMA_SSD_2.VHDX
VMA    SCSI           0                2                             D:\VMS\VMA_SSD_3.VHDX
VMA    SCSI           0                3                             D:\VMS\VMA_SSD_4.VHDX
VMA    SCSI           0                4                             E:\VMS\VMA_HDD_1.VHDX
VMA    SCSI           0                5                             E:\VMS\VMA_HDD_2.VHDX
VMA    SCSI           0                6                             E:\VMS\VMA_HDD_3.VHDX
VMA    SCSI           0                7                             E:\VMS\VMA_HDD_4.VHDX
VMA    SCSI           0                8                             E:\VMS\VMA_HDD_5.VHDX
VMA    SCSI           0                9                             E:\VMS\VMA_HDD_6.VHDX
VMA    SCSI           0                10                            E:\VMS\VMA_HDD_7.VHDX
VMA    SCSI           0                11                            E:\VMS\VMA_HDD_8.VHDX
 

 

6. Check VM configuration (from the Guest, with output)

 

PS C:\> Get-PhysicalDisk | Sort Size | FT DeviceId, FriendlyName, CanPool, Size, MediaType -AutoSize
 
DeviceId FriendlyName   CanPool        Size MediaType
-------- ------------   -------        ---- ---------
2        PhysicalDisk2     True 10737418240 UnSpecified
4        PhysicalDisk4     True 10737418240 UnSpecified
3        PhysicalDisk3     True 10737418240 UnSpecified
1        PhysicalDisk1     True 10737418240 UnSpecified
12       PhysicalDisk12    True 32212254720 UnSpecified
11       PhysicalDisk11    True 32212254720 Unspecified
7        PhysicalDisk7     True 32212254720 Unspecified
6        PhysicalDisk6     True 32212254720 Unspecified
5        PhysicalDisk5     True 32212254720 Unspecified
10       PhysicalDisk10    True 32212254720 UnSpecified
9        PhysicalDisk9     True 32212254720 Unspecified
8        PhysicalDisk8     True 32212254720 Unspecified
0        PhysicalDisk0    False 42949672960 Unspecified
 

PS C:\> Get-PhysicalDisk -CanPool $true | ? Size -lt 20GB | Sort Size | FT -AutoSize
 
FriendlyName  CanPool OperationalStatus HealthStatus Usage        Size
------------  ------- ----------------- ------------ -----        ----
PhysicalDisk4 True    OK                Healthy      Auto-Select 10 GB
PhysicalDisk2 True    OK                Healthy      Auto-Select 10 GB
PhysicalDisk1 True    OK                Healthy      Auto-Select 10 GB
PhysicalDisk3 True    OK                Healthy      Auto-Select 10 GB
 

PS C:\> Get-PhysicalDisk -CanPool $true | ? Size -gt 20GB | Sort Size | FT -AutoSize
 

FriendlyName   CanPool OperationalStatus HealthStatus Usage        Size
------------   ------- ----------------- ------------ -----        ----
PhysicalDisk6  True    OK                Healthy      Auto-Select 30 GB
PhysicalDisk11 True    OK                Healthy      Auto-Select 30 GB
PhysicalDisk12 True    OK                Healthy      Auto-Select 30 GB
PhysicalDisk7  True    OK                Healthy      Auto-Select 30 GB
PhysicalDisk5  True    OK                Healthy      Auto-Select 30 GB
PhysicalDisk10 True    OK                Healthy      Auto-Select 30 GB
PhysicalDisk8  True    OK                Healthy      Auto-Select 30 GB
PhysicalDisk9  True    OK                Healthy      Auto-Select 30 GB
 

7. Configure media type for virtual SAS disks

 

# Create Storage Pool

$s = Get-StorageSubSystem

New-StoragePool -StorageSubSystemId $s.UniqueId -FriendlyName Pool1 -PhysicalDisks (Get-PhysicalDisk -CanPool $true)

 

# Configure media type for virtual SAS disks

Get-StoragePool Pool1 | Get-PhysicalDisk | ? Size -lt 20GB | Set-PhysicalDisk –MediaType SSD

Get-StoragePool Pool1 | Get-PhysicalDisk | ? Size -gt 20GB | Set-PhysicalDisk –MediaType HDD

 

 

8. Check media type configuration (with output)

 

PS C:\> Get-StoragePool Pool1

 

FriendlyName            OperationalStatus       HealthStatus            IsPrimordial            IsReadOnly

------------            -----------------       ------------            ------------            ----------

Pool1                   OK                      Healthy                 False                   False

 

 

PS C:\> Get-StoragePool Pool1 | Get-PhysicalDisk | Sort Size | FT –AutoSize

 

FriendlyName   CanPool OperationalStatus HealthStatus Usage           Size

------------   ------- ----------------- ------------ -----           ----

PhysicalDisk3  False   OK                Healthy      Auto-Select  9.25 GB

PhysicalDisk2  False   OK                Healthy      Auto-Select  9.25 GB

PhysicalDisk1  False   OK                Healthy      Auto-Select  9.25 GB

PhysicalDisk4  False   OK                Healthy      Auto-Select  9.25 GB

PhysicalDisk11 False   OK                Healthy      Auto-Select 29.25 GB

PhysicalDisk12 False   OK                Healthy      Auto-Select 29.25 GB

PhysicalDisk7  False   OK                Healthy      Auto-Select 29.25 GB

PhysicalDisk6  False   OK                Healthy      Auto-Select 29.25 GB

PhysicalDisk9  False   OK                Healthy      Auto-Select 29.25 GB

PhysicalDisk5  False   OK                Healthy      Auto-Select 29.25 GB

PhysicalDisk8  False   OK                Healthy      Auto-Select 29.25 GB

PhysicalDisk10 False   OK                Healthy      Auto-Select 29.25 GB

 

 

PS C:\> Get-StoragePool Pool1 | Get-PhysicalDisk | Sort Size | FT FriendlyName, Size, MediaType, HealthStatus, OperationalStatus -AutoSize

 

FriendlyName          Size MediaType HealthStatus OperationalStatus

------------          ---- --------- ------------ -----------------

PhysicalDisk3   9932111872 SSD       Healthy      OK

PhysicalDisk2   9932111872 SSD       Healthy      OK

PhysicalDisk1   9932111872 SSD       Healthy      OK

PhysicalDisk4   9932111872 SSD       Healthy      OK

PhysicalDisk11 31406948352 HDD       Healthy      OK

PhysicalDisk12 31406948352 HDD       Healthy      OK

PhysicalDisk7  31406948352 HDD       Healthy      OK

PhysicalDisk6  31406948352 HDD       Healthy      OK

PhysicalDisk9  31406948352 HDD       Healthy      OK

PhysicalDisk5  31406948352 HDD       Healthy      OK

PhysicalDisk8  31406948352 HDD       Healthy      OK

PhysicalDisk10 31406948352 HDD       Healthy      OK

 

PS C:\> Get-StoragePool Pool1 | Get-PhysicalDisk | Group MediaType, Size | Sort Name | FT -AutoSize

 

Count Name             Group

----- ----             -----

    8 HDD, 31406948352 {MSFT_PhysicalDisk (ObjectId = "{1}\\WIN-T5PORQGQECN\root/Microsoft/Win...), MSFT_PhysicalDis...

    4 SSD, 9932111872  {MSFT_PhysicalDisk (ObjectId = "{1}\\WIN-T5PORQGQECN\root/Microsoft/Win...), MSFT_PhysicalDis...

 

PS C:\> Get-StoragePool Pool1 | FL Size, AllocatedSize

 

Size          : 290984034304
AllocatedSize : 3221225472 

 

9. Configure Tiers

 

# Configure two tiers

Get-StoragePool Pool1 | New-StorageTier –FriendlyName SSDTier –MediaType SSD

Get-StoragePool Pool1 | New-StorageTier –FriendlyName HDDTier –MediaType HDD

 

 

10. Check Tiers configuration (with output)

 

PS C:\> Get-StorageTier | FT FriendlyName, MediaType, Size -AutoSize

 

FriendlyName   MediaType        Size

------------   ---------        ----

SSDTier        SSD                 0

HDDTier        HDD                 0

 

PS C:\> Get-StoragePool Pool1 | FL Size, AllocatedSize

 

Size          : 290984034304

AllocatedSize : 3221225472

 

Note: 3GB used out of 271GB total. Storage Spaces reserves 256MB on each disk in the pool for internal VD metadata. 256MB * 12 drives = 3GB.

 

 

PS C:\> Get-StorageTierSupportedSize SSDTier -ResiliencySettingName Simple | FT -AutoSize

 

 

SupportedSizes TierSizeMin TierSizeMax TierSizeDivisor

-------------- ----------- ----------- ---------------

{}              4294967296 34359738368      4294967296

 

Note: 32GB on the SSD Tier (8 GB * 4). Minimum size is 4GB.

 

PS C:\> Get-StorageTierSupportedSize SSDTier -ResiliencySettingName Mirror | FT -AutoSize

 

SupportedSizes TierSizeMin TierSizeMax TierSizeDivisor

-------------- ----------- ----------- ---------------

{}              2147483648 17179869184      2147483648

 

Note: Mirrored offers 16GB on the SSD Tier (8 GB * 4 / 2). Minimum size is 2GB.

 

 

PS C:\> Get-StorageTierSupportedSize HDDTier -ResiliencySettingName Simple | FT -AutoSize

 

SupportedSizes TierSizeMin  TierSizeMax TierSizeDivisor

-------------- -----------  ----------- ---------------

{}              8589934592 249108103168      8589934592

 

Note: 232GB on the HDD Tier (29 GB * 8). Minimum size is 8GB.

 

PS C:\> Get-StorageTierSupportedSize HDDTier -ResiliencySettingName Mirror | FT -AutoSize

 

SupportedSizes TierSizeMin  TierSizeMax TierSizeDivisor

-------------- -----------  ----------- ---------------

{}              4294967296 124554051584      4294967296

 

Note: Mirrored offers 116GB on the HDD Tier (29 GB * 8 / 2). Minimum size is 4GB.

 

11. Resiliency Settings and Spaces

 

 

# Configure resiliency settings

Get-StoragePool Pool1 | Set-ResiliencySetting -Name Simple -NumberOfColumnsDefault 4

Get-StoragePool Pool1 | Set-ResiliencySetting -Name Mirror -NumberOfColumnsDefault 2

 

# Create simple and mirrored spaces with tiering

$SSD = Get-StorageTier -FriendlyName SSDTier

$HDD = Get-StorageTier -FriendlyName HDDTier

Get-StoragePool Pool1 | New-VirtualDisk -FriendlyName Space1 -ResiliencySettingName Simple –StorageTiers $SSD, $HDD -StorageTierSizes 8GB, 32GB -WriteCacheSize 1GB

Get-StoragePool Pool1 | New-VirtualDisk -FriendlyName Space2 -ResiliencySettingName Mirror -StorageTiers $SSD, $HDD -StorageTierSizes 8GB, 32GB –WriteCacheSize 1GB

 

12. Check Storage Spaces configuration (with output)

 

PS C:\> Get-StoragePool Pool1 | Get-ResiliencySetting | FT -AutoSize

 

Name   NumberOfDataCopies PhysicalDiskRedundancy NumberOfColumns Interleave

----   ------------------ ---------------------- --------------- ----------

Simple 1                  0                      4               262144

Mirror 2                  1                      2               262144

Parity 1                  1                      Auto            262144

 

PS C:\> Get-VirtualDisk | FT -AutoSize

 

FriendlyName ResiliencySettingName OperationalStatus HealthStatus IsManualAttach  Size

------------ --------------------- ----------------- ------------ --------------  ----

Space1       Simple                OK                Healthy      False          40 GB

Space2       Mirror                OK                Healthy      False          40 GB

 

PS C:\> Get-StorageTier | FT FriendlyName, MediaType, Size -AutoSize

 

FriendlyName   MediaType        Size

------------   ---------        ----

SSDTier        SSD                 0

HDDTier        HDD                 0

Space1_SSDTier SSD        8589934592

Space1_HDDTier HDD       34359738368

Space2_SSDTier SSD        8589934592

Space2_HDDTier HDD       34359738368

 

PS C:\> Get-StoragePool Pool1 | FL Size, AllocatedSize

 

Size          : 290984034304

AllocatedSize : 136365211648

 

Note: 127GB allocated after creating the two spaces. 144GB left out of 271GB total.

 

PS C:\> Get-StorageTierSupportedSize SSDTier -ResiliencySettingName Simple | FT -AutoSize

 

SupportedSizes TierSizeMin TierSizeMax TierSizeDivisor

-------------- ----------- ----------- ---------------

{}              4294967296  4294967296      4294967296

 

Note: 4GB left on the SSD Tier.

 

PS C:\> Get-StorageTierSupportedSize SSDTier -ResiliencySettingName Mirror | FT -AutoSize

 

SupportedSizes TierSizeMin TierSizeMax TierSizeDivisor

-------------- ----------- ----------- ---------------

{}              2147483648  2147483648      2147483648

 

Note: 4GB left on the SSD Tier. 2GB available if mirroring.

 

PS C:\> Get-StorageTierSupportedSize HDDTier -ResiliencySettingName Simple | FT -AutoSize

 

SupportedSizes TierSizeMin  TierSizeMax TierSizeDivisor

-------------- -----------  ----------- ---------------

{}              4294967296 146028888064      4294967296

 

Note: 136GB left on the SSD Tier.

 

PS C:\> Get-StorageTierSupportedSize HDDTier -ResiliencySettingName Mirror | FT -AutoSize

 

SupportedSizes TierSizeMin TierSizeMax TierSizeDivisor

-------------- ----------- ----------- ---------------

{}              2147483648 73014444032      2147483648

 

Note: 136GB left on the SSD Tier. 68GB available if mirroring.

 

13. Create Partitions and Volumes

 

 

# Configure volume “F” on Space1

Get-VirtualDisk Space1 | Get-Disk | Set-Disk -IsReadOnly 0

Get-VirtualDisk Space1 | Get-Disk | Set-Disk -IsOffline 0

Get-VirtualDisk Space1 | Get-Disk | Initialize-Disk -PartitionStyle GPT

Get-VirtualDisk Space1 | Get-Disk | New-Partition -DriveLetter “F” -UseMaximumSize

Initialize-Volume -DriveLetter “F” -FileSystem NTFS -Confirm:$false

 

# Configure volume “G” on Space2

Get-VirtualDisk Space2 | Get-Disk | Set-Disk -IsReadOnly 0

Get-VirtualDisk Space2 | Get-Disk | Set-Disk -IsOffline 0

Get-VirtualDisk Space2 | Get-Disk | Initialize-Disk -PartitionStyle GPT

Get-VirtualDisk Space2 | Get-Disk | New-Partition -DriveLetter “G” -UseMaximumSize

Initialize-Volume -DriveLetter “G” -FileSystem NTFS -Confirm:$false

 

14. Check Partitions and Volumes (with output)

 

PS C:\> Get-Partition | ? DriveLetter -ge "F" | FT -AutoSize

 

   Disk Number: 13

 

PartitionNumber DriveLetter Offset        Size Type

--------------- ----------- ------        ---- ----

2               F           135266304 39.87 GB Basic

 

   Disk Number: 14

 

PartitionNumber DriveLetter Offset        Size Type

--------------- ----------- ------        ---- ----

2               G           135266304 39.87 GB Basic

 

PS C:\> Get-Volume | ? DriveLetter -ge "F" | FT -AutoSize

 

DriveLetter FileSystemLabel FileSystem DriveType HealthStatus SizeRemaining     Size

----------- --------------- ---------- --------- ------------ -------------     ----

F                           NTFS       Fixed     Healthy           39.75 GB 39.87 GB

G                           NTFS       Fixed     Healthy           39.75 GB 39.87 GB

 

15. Create Test Files

 

 

# Create 3 files on volume “F”, place them on different tiers

1..3 | % {

   fsutil file createnew f:\file$_.dat (4GB)

   fsutil file setvaliddata f:\file$_.dat (4GB)

}

Set-FileStorageTier -FilePath f:\file1.dat -DesiredStorageTierFriendlyName Space1_SSDTier

Set-FileStorageTier -FilePath f:\file2.dat -DesiredStorageTierFriendlyName Space1_HDDTier

Get-FileStorageTier -VolumeDriveLetter F

 

# Create 3 files on volume “G”, place them on different tiers

1..3 | % {

   fsutil file createnew g:\file$_.dat (4GB)

   fsutil file setvaliddata g:\file$_.dat (4GB)

}

Set-FileStorageTier -FilePath g:\file1.dat -DesiredStorageTierFriendlyName Space2_SSDTier

Set-FileStorageTier -FilePath g:\file2.dat -DesiredStorageTierFriendlyName Space2_HDDTier

Get-FileStorageTier -VolumeDriveLetter G

 

16. Check Test Files (with output)

 

PS C:\> Dir F:

 

    Directory: f:\

 

Mode                LastWriteTime     Length Name

----                -------------     ------ ----

-a---         8/21/2013  10:09 AM 4294967296 file1.dat

-a---         8/21/2013  10:09 AM 4294967296 file2.dat

-a---         8/21/2013  10:09 AM 4294967296 file3.dat

 

PS C:\> Dir G:

 

    Directory: g:\

 

Mode                LastWriteTime     Length Name

----                -------------     ------ ----

-a---         8/21/2013  10:08 AM 4294967296 file1.dat

-a---         8/21/2013  10:08 AM 4294967296 file2.dat

-a---         8/21/2013  10:08 AM 4294967296 file3.dat

 

PS C:\sqlio> Get-Volume | ? DriveLetter -ge "F" | FT -AutoSize

 

DriveLetter FileSystemLabel FileSystem DriveType HealthStatus SizeRemaining     Size

----------- --------------- ---------- --------- ------------ -------------     ----

F                           NTFS       Fixed     Healthy           27.75 GB 39.87 GB

G                           NTFS       Fixed     Healthy           27.75 GB 39.87 GB

 

PS C:\> Get-FileStorageTier -VolumeDriveLetter F | FT -AutoSize

 

FilePath     DesiredStorageTierName PlacementStatus    State

--------     ---------------------- ---------------    -----

F:\file1.dat Space1_SSDTier         Completely on tier OK

F:\file2.dat Space1_HDDTier         Partially on tier  Pending

 

PS C:\> Get-FileStorageTier -VolumeDriveLetter G | FT -AutoSize

 

FilePath     DesiredStorageTierName PlacementStatus    State

--------     ---------------------- ---------------    -----

G:\file1.dat Space2_SSDTier         Completely on tier OK

G:\file2.dat Space2_HDDTier         Partially on tier  Pending

 

 

17. Tasks for Storage Tiering

 

# Check tasks used by Storage Tiering

Get-ScheduledTask -TaskName *Tier* | FT –AutoSize

Get-ScheduledTask -TaskName *Tier* | Get-ScheduledTaskInfo

 

# Manually running the “Storage Tiers Optimization” task

Get-ScheduledTask -TaskName "Storage Tiers Optimization" | Start-ScheduledTask

 

18. Tasks for Storage Tiering (with Output)

 

PS C:\> Get-ScheduledTask -TaskName *Tier*  | FT –AutoSize

 

TaskPath                                     TaskName                                State

--------                                     --------                                -----

\Microsoft\Windows\Storage Tiers Management\ Storage Tiers Management Initialization Ready

\Microsoft\Windows\Storage Tiers Management\ Storage Tiers Optimization              Ready

 

PS C:\> Get-ScheduledTask -TaskName *Tier*  | Get-ScheduledTaskInfo

 

LastRunTime        :

LastTaskResult     : 1

NextRunTime        : 8/22/2013 1:00:00 AM

NumberOfMissedRuns : 0

TaskName           : Storage Tiers Optimization

TaskPath           : \Microsoft\Windows\Storage Tiers Management\

PSComputerName     :

 

LastRunTime        : 8/21/2013 11:18:18 AM

LastTaskResult     : 0

NextRunTime        :

NumberOfMissedRuns : 0

TaskName           : Storage Tiers Management Initialization

TaskPath           : \Microsoft\Windows\Storage Tiers Management\

PSComputerName     :

 

PS C:\> Get-ScheduledTask -TaskName "Storage Tiers Optimization" | Start-ScheduledTask

PS C:\> Get-ScheduledTask -TaskName "Storage Tiers Optimization" | Get-ScheduledTaskInfo

 

LastRunTime        : 8/21/2013 12:11:11 PM

LastTaskResult     : 267009

NextRunTime        : 8/22/2013 1:00:00 AM

NumberOfMissedRuns : 0

TaskName           : Storage Tiers Optimization

TaskPath           : \Microsoft\Windows\Storage Tiers Management\

PSComputerName     :

 

 

19. Running SQLIO

 

# These commands assume that the SQLIO2.EXE file was copied to the C:\SQLIO folder

# SQLIO workload 1 : 30 seconds, random, read, 8KB, 4 thread, 16 outstanding IOs, no buffering

# SQLIO workload 2 : 30 seconds, sequential, read, 512KB, 4 thread, 4 outstanding IOs, no buffering

 

# Check file location on tiers for volume F:

Get-FileStorageTier -VolumeDriveLetter F | FT -AutoSize

 

# Running SQLIO on F:, using File1 (HDD tier), File2 (HDD tier) and File 3 (unspecified tier)

c:\sqlio\sqlio2.exe -s30 -frandom -kR -b8 -t4 -o16 -BN f:\file1.dat

c:\sqlio\sqlio2.exe -s30 -frandom -kR -b8 -t4 -o16 -BN f:\file2.dat

c:\sqlio\sqlio2.exe -s30 -frandom -kR -b8 -t4 -o16 -BN f:\file3.dat

c:\sqlio\sqlio2.exe -s30 -fsequential -kR -b512 -t4 -o4 -BN f:\file1.dat

c:\sqlio\sqlio2.exe -s30 -fsequential -kR -b512 -t4 -o4 -BN f:\file2.dat

c:\sqlio\sqlio2.exe -s30 -fsequential -kR -b512 -t4 -o4 -BN f:\file3.dat

 

# Check file location on tiers for volume G:

Get-FileStorageTier -VolumeDriveLetter G | FT -AutoSize

 

# Running SQLIO on G:, using File1 (HDD tier), File2 (HDD tier) and File 3 (unspecified tier)

c:\sqlio\sqlio2.exe -s30 -frandom -kR -b8 -t4 -o16 -BN g:\file1.dat

c:\sqlio\sqlio2.exe -s30 -frandom -kR -b8 -t4 -o16 -BN g:\file2.dat

c:\sqlio\sqlio2.exe -s30 -frandom -kR -b8 -t4 -o16 -BN g:\file3.dat

c:\sqlio\sqlio2.exe -s30 -fsequential -kR -b512 -t4 -o4 -BN g:\file1.dat

c:\sqlio\sqlio2.exe -s30 -fsequential -kR -b512 -t4 -o4 -BN g:\file2.dat

c:\sqlio\sqlio2.exe -s30 -fsequential -kR -b512 -t4 -o4 -BN g:\file3.dat

 

20. Running SQLIO (with Output)

 

PS C:\> Get-FileStorageTier -VolumeDriveLetter F | FT -AutoSize

 

FilePath     DesiredStorageTierName PlacementStatus    State

--------     ---------------------- ---------------    -----

F:\file1.dat Space1_SSDTier         Completely on tier OK

F:\file2.dat Space1_HDDTier         Completely on tier OK

 

PS C:\> c:\sqlio\sqlio2.exe -s30 -frandom -kR -b8 -t4 -o16 -BN f:\file1.dat

 

sqlio v2.15. 64bit_SG

4 threads reading for 30 secs to file f:\file1.dat

        using 8KB random IOs

        enabling multiple I/Os per thread with 16 outstanding

        buffering set to not use file nor disk caches (as is SQL Server)

using current size: 4096 MB for file: f:\file1.dat

initialization done

CUMULATIVE DATA:

throughput metrics:

IOs/sec: 35745.63

MBs/sec:   279.26

 

PS C:\> c:\sqlio\sqlio2.exe -s30 -frandom -kR -b8 -t4 -o16 -BN f:\file2.dat

 

sqlio v2.15. 64bit_SG

4 threads reading for 30 secs to file f:\file2.dat

        using 8KB random IOs

        enabling multiple I/Os per thread with 16 outstanding

        buffering set to not use file nor disk caches (as is SQL Server)

using current size: 4096 MB for file: f:\file2.dat

initialization done

CUMULATIVE DATA:

throughput metrics:

IOs/sec:   141.57

MBs/sec:     1.10

 

PS C:\> c:\sqlio\sqlio2.exe -s30 -frandom -kR -b8 -t4 -o16 -BN f:\file3.dat

 

sqlio v2.15. 64bit_SG

4 threads reading for 30 secs to file f:\file3.dat

        using 8KB random IOs

        enabling multiple I/Os per thread with 16 outstanding

        buffering set to not use file nor disk caches (as is SQL Server)

using current size: 4096 MB for file: f:\file3.dat

initialization done

CUMULATIVE DATA:

throughput metrics:

IOs/sec:   384.86

MBs/sec:     3.00

 

PS C:\> c:\sqlio\sqlio2.exe -s30 -fsequential -kR -b512 -t4 -o4 -BN f:\file1.dat

 

sqlio v2.15. 64bit_SG

4 threads reading for 30 secs to file f:\file1.dat

        using 512KB sequential IOs

        enabling multiple I/Os per thread with 4 outstanding

        buffering set to not use file nor disk caches (as is SQL Server)

using current size: 4096 MB for file: f:\file1.dat

initialization done

CUMULATIVE DATA:

throughput metrics:

IOs/sec:   998.25

MBs/sec:   499.12

 

PS C:\> c:\sqlio\sqlio2.exe -s30 -fsequential -kR -b512 -t4 -o4 -BN f:\file2.dat

 

sqlio v2.15. 64bit_SG

4 threads reading for 30 secs to file f:\file2.dat

        using 512KB sequential IOs

        enabling multiple I/Os per thread with 4 outstanding

        buffering set to not use file nor disk caches (as is SQL Server)

using current size: 4096 MB for file: f:\file2.dat

initialization done

CUMULATIVE DATA:

throughput metrics:

IOs/sec:    81.30

MBs/sec:    40.65

 

PS C:\> c:\sqlio\sqlio2.exe -s30 -fsequential -kR -b512 -t4 -o4 -BN f:\file3.dat

 

sqlio v2.15. 64bit_SG

4 threads reading for 30 secs to file f:\file3.dat

        using 512KB sequential IOs

        enabling multiple I/Os per thread with 4 outstanding

        buffering set to not use file nor disk caches (as is SQL Server)

using current size: 4096 MB for file: f:\file3.dat

initialization done

CUMULATIVE DATA:

throughput metrics:

IOs/sec:   148.51

MBs/sec:    74.25

 

PS C:\> Get-FileStorageTier -VolumeDriveLetter G | FT -AutoSize

 

FilePath     DesiredStorageTierName PlacementStatus    State

--------     ---------------------- ---------------    -----

G:\file1.dat Space2_SSDTier         Completely on tier OK

G:\file2.dat Space2_HDDTier         Completely on tier OK

 

PS C:\> c:\sqlio\sqlio2.exe -s30 -frandom -kR -b8 -t4 -o16 -BN g:\file1.dat

 

sqlio v2.15. 64bit_SG

4 threads reading for 30 secs to file g:\file1.dat

        using 8KB random IOs

        enabling multiple I/Os per thread with 16 outstanding

        buffering set to not use file nor disk caches (as is SQL Server)

using current size: 4096 MB for file: g:\file1.dat

initialization done

CUMULATIVE DATA:

throughput metrics:

IOs/sec: 35065.17

MBs/sec:   273.94

 

PS C:\> c:\sqlio\sqlio2.exe -s30 -frandom -kR -b8 -t4 -o16 -BN g:\file2.dat

 

sqlio v2.15. 64bit_SG

4 threads reading for 30 secs to file g:\file2.dat

        using 8KB random IOs

        enabling multiple I/Os per thread with 16 outstanding

        buffering set to not use file nor disk caches (as is SQL Server)

using current size: 4096 MB for file: g:\file2.dat

initialization done

CUMULATIVE DATA:

throughput metrics:

IOs/sec:   138.98

MBs/sec:     1.08

 

PS C:\> c:\sqlio\sqlio2.exe -s30 -frandom -kR -b8 -t4 -o16 -BN g:\file3.dat

 

sqlio v2.15. 64bit_SG

4 threads reading for 30 secs to file g:\file3.dat

        using 8KB random IOs

        enabling multiple I/Os per thread with 16 outstanding

        buffering set to not use file nor disk caches (as is SQL Server)

using current size: 4096 MB for file: g:\file3.dat

initialization done

CUMULATIVE DATA:

throughput metrics:

IOs/sec:   360.33

MBs/sec:     2.81

 

PS C:\> c:\sqlio\sqlio2.exe -s30 -fsequential -kR -b512 -t4 -o4 -BN g:\file1.dat

 

sqlio v2.15. 64bit_SG

4 threads reading for 30 secs to file g:\file1.dat

        using 512KB sequential IOs

        enabling multiple I/Os per thread with 4 outstanding

        buffering set to not use file nor disk caches (as is SQL Server)

using current size: 4096 MB for file: g:\file1.dat

initialization done

CUMULATIVE DATA:

throughput metrics:

IOs/sec:   997.90

MBs/sec:   498.95

 

PS C:\> c:\sqlio\sqlio2.exe -s30 -fsequential -kR -b512 -t4 -o4 -BN g:\file2.dat

 

sqlio v2.15. 64bit_SG

4 threads reading for 30 secs to file g:\file2.dat

        using 512KB sequential IOs

        enabling multiple I/Os per thread with 4 outstanding

        buffering set to not use file nor disk caches (as is SQL Server)

using current size: 4096 MB for file: g:\file2.dat

initialization done

CUMULATIVE DATA:

throughput metrics:

IOs/sec:   118.34

MBs/sec:    59.17

 

PS C:\> c:\sqlio\sqlio2.exe -s30 -fsequential -kR -b512 -t4 -o4 -BN g:\file3.dat

 

sqlio v2.15. 64bit_SG

4 threads reading for 30 secs to file g:\file3.dat

        using 512KB sequential IOs

        enabling multiple I/Os per thread with 4 outstanding

        buffering set to not use file nor disk caches (as is SQL Server)

using current size: 4096 MB for file: g:\file3.dat

initialization done

CUMULATIVE DATA:

throughput metrics:

IOs/sec:   197.65

MBs/sec:    98.82

 

21. Summary of SQLIO Results

 

Here’s the summary of the SQLIO runs:

Resiliency

Workload

Tier

IOPs

MB/s

Simple

8KB
Read

File1 (SSD)

35,746

279

File2 (HDD)

142

1

File3 (Mixed)

385

3

512KB
Sequential

File1 (SSD)

998

499

File2 (HDD)

81

41

File3 (Mixed)

149

74

Mirror

8KB
Read

File1 (SSD)

35,065

274

File2 (HDD)

39

1

File3 (Mixed)

360

3

512KB
Sequential

File1 (SSD)

998

499

File2 (HDD)

118

59

File3 (Mixed)

198

99

 

Note 1: In general, this is working as expected. File1 (SSD Tier) shows SSD performance characteristics while File 2 (HDD Tier) behaves like a spinning disk. File3 is somewhere between the two.

Note 2: There is only one physical SSD and one physical HDD behind all the virtual layers. Both simple and mirrored are spaces are basically limited by the hardware limitations of the two disks.

Note 3: This shows the results of a single run of each workload, so some variance is expected. If you are running this in your own setup, you might want to run each workload multiple times and average it out. There’s some guidance to that regard at http://blogs.technet.com/b/josebda/archive/2013/03/25/sqlio-powershell-and-storage-performance-measuring-iops-throughput-and-latency-for-both-local-disks-and-smb-file-shares.aspx

 

 

22. Extending existing Spaces and Volumes

 

# Check state before change

Get-VirtualDisk Space1 | FT -AutoSize

Get-StorageTier Space1* | FT FriendlyName, Size –AutoSize

Get-StorageTierSupportedSize SSDTier -ResiliencySettingName Simple | FT -AutoSize

Get-VirtualDisk Space1 | Get-Disk | FT -AutoSize

Get-VirtualDisk Space1 | Get-Disk | Get-Partition | FT –AutoSize

Get-VirtualDisk Space1 | Get-Disk | Get-Partition | Get-Volume | FT –AutoSize

 

# Add 4GB on the SSD Tier

Resize-StorageTier Space1_SSDTier -Size 12GB

Get-VirtualDisk Space1 | Get-Disk | Update-Disk

 

# Check after Virtual Disk change

Get-VirtualDisk Space1 | FT -AutoSize

Get-StorageTier Space1* | FT FriendlyName, Size –AutoSize

Get-StorageTierSupportedSize SSDTier -ResiliencySettingName Simple | FT -AutoSize

Get-VirtualDisk Space1 | Get-Disk | FT -AutoSize

Get-VirtualDisk Space1 | Get-Disk | Get-Partition | FT –AutoSize

Get-VirtualDisk Space1 | Get-Disk | Get-Partition | Get-Volume | FT –AutoSize

 

# Extend partition (also extends the volume)

Resize-Partition -DriveLetter F -Size 43.87GB

 

# Check after Partition/Volume change

Get-VirtualDisk Space1 | Get-Disk | Get-Partition | FT –AutoSize

Get-VirtualDisk Space1 | Get-Disk | Get-Partition | Get-Volume | FT –AutoSize

 

23. Extending existing Spaces and Volumes(with output)

 

PS C:\> Get-VirtualDisk Space1 | FT -AutoSize

 

FriendlyName ResiliencySettingName OperationalStatus HealthStatus IsManualAttach  Size

------------ --------------------- ----------------- ------------ --------------  ----

Space1       Simple                OK                Healthy      False          40 GB

 

PS C:\> Get-StorageTier Space1* | FT FriendlyName, Size –AutoSize

 

FriendlyName          Size

------------          ----

Space1_SSDTier  8589934592

Space1_HDDTier 34359738368

 

PS C:\> Get-StorageTierSupportedSize SSDTier -ResiliencySettingName Simple | FT -AutoSize

 

SupportedSizes TierSizeMin TierSizeMax TierSizeDivisor

-------------- ----------- ----------- ---------------

{}              4294967296  4294967296      4294967296

 

PS C:\> Get-VirtualDisk Space1 | Get-Disk | FT -AutoSize

 

Number Friendly Name                  OperationalStatus Total Size Partition Style

------ -------------                  ----------------- ---------- ---------------

13     Microsoft Storage Space Device Online                 40 GB GPT

 

PS C:\> Get-VirtualDisk Space1 | Get-Disk | Get-Partition | FT -AutoSize

 

   Disk Number: 13

 

PartitionNumber DriveLetter Offset        Size Type

--------------- ----------- ------        ---- ----

1                           24576       128 MB Reserved

2               F           135266304 39.87 GB Basic

 

PS C:\> Get-VirtualDisk Space1 | Get-Disk | Get-Partition | Get-Volume | FT -AutoSize

 

DriveLetter FileSystemLabel FileSystem DriveType HealthStatus SizeRemaining     Size

----------- --------------- ---------- --------- ------------ -------------     ----

F                           NTFS       Fixed     Healthy           27.75 GB 39.87 GB

 

PS C:\> Resize-StorageTier Space1_SSDTier -Size 12GB

 

PS C:\> Get-VirtualDisk Space1 | Get-Disk | Update-Disk

 

PS C:\> Get-VirtualDisk Space1 | FT -AutoSize

 

FriendlyName ResiliencySettingName OperationalStatus HealthStatus IsManualAttach  Size

------------ --------------------- ----------------- ------------ --------------  ----

Space1       Simple                OK                Healthy      False          44 GB

 

PS C:\> Get-StorageTier Space1* | FT FriendlyName, Size –AutoSize

 

FriendlyName          Size

------------          ----

Space1_SSDTier 12884901888

Space1_HDDTier 34359738368

 

PS C:\> Get-StorageTierSupportedSize SSDTier -ResiliencySettingName Simple | FT -AutoSize

 

SupportedSizes TierSizeMin TierSizeMax TierSizeDivisor

-------------- ----------- ----------- ---------------

{}                       0           0               0

 

PS C:\> Get-VirtualDisk Space1 | Get-Disk | FT -AutoSize

 

Number Friendly Name                  OperationalStatus Total Size Partition Style

------ -------------                  ----------------- ---------- ---------------

13     Microsoft Storage Space Device Online                 44 GB GPT

 

PS C:\> Get-VirtualDisk Space1 | Get-Disk | Get-Partition | FT –AutoSize

 

   Disk Number: 13

 

PartitionNumber DriveLetter Offset        Size Type

--------------- ----------- ------        ---- ----

1                           24576       128 MB Reserved

2               F           135266304 39.87 GB Basic

 

PS C:\> Get-VirtualDisk Space1 | Get-Disk | Get-Partition | Get-Volume | FT –AutoSize

 

DriveLetter FileSystemLabel FileSystem DriveType HealthStatus SizeRemaining     Size

----------- --------------- ---------- --------- ------------ -------------     ----

F                           NTFS       Fixed     Healthy           27.75 GB 39.87 GB

 

PS C:\> Resize-Partition -DriveLetter F -Size 43.87GB

 

PS C:\> Get-VirtualDisk Space1 | Get-Disk | Get-Partition | FT –AutoSize

 

   Disk Number: 13

 

PartitionNumber DriveLetter Offset        Size Type

--------------- ----------- ------        ---- ----

1                           24576       128 MB Reserved

2               F           135266304 43.87 GB Basic

 

PS C:\> Get-VirtualDisk Space1 | Get-Disk | Get-Partition | Get-Volume | FT –AutoSize

 

DriveLetter FileSystemLabel FileSystem DriveType HealthStatus SizeRemaining     Size

----------- --------------- ---------- --------- ------------ -------------     ----

F                           NTFS       Fixed     Healthy           31.75 GB 43.87 GB

 

 

24. Final Notes

 

  • I hope you enjoyed these step-by-step instructions. I strongly encourage you to try them out and perform the entire installation yourself. It’s a good learning experience.
  • Let me know how these steps worked for you using the comment section. If you run into any issues or found anything particularly interesting, don’t forget to mention the number of the step.

What’s new in SMB PowerShell in Windows Server 2012 R2

$
0
0

Windows Server 2012 R2 introduced a new version of SMB. Technically it’s SMB 3.02, but we continue to call it just SMB3. The main changes are described at http://technet.microsoft.com/en-us/library/hh831474.aspx. With this new release, we made a few changes in SMB PowerShell to support the new scenarios and features. This includes a few new cmdlets and some changes to existing cmdlets, with extra care not break any of your existing scripts. This blog post outlines 7 set of changes related to SMB PowerShell in Windows Server 2012 R2.

 

1) Simpler setting of ACL for the folder behind a share

 

In Windows Server 2012, the SMB share has a property that facilitates applying the share ACL to the file system folder used by the share. Here’s the syntax:

  • (Get-SmbShare –Name <ShareName> ).PresetPathACL | Set-Acl

 

In Windows Server 2012 R2, we have improved this scenario by providing a proper cmdlet to apply the share ACL to the file system used by the share. Here’s the new syntax:

  • Set-SmbPathAcl –Name ShareName

 

Note1 : The Windows Server 2012 syntax continues to work with Windows Server 2012 R2, but the new syntax is much simpler and therefore recommended.

 

Note 2: There is known issue with Windows Server 2012 R2 Preview that causes this new cmdlet to fail when using non-Unicode languages. As a workaround, you can use the old syntax.

 

2) Scale-Out Rebalancing (per-share redirection)

 

Per-share redirection is the new default behavior for the new SMB Clients (Windows 8.1 or Windows Server 2012 R2), when connecting to a Scale-Out clusters that use a storage system that does not support Direct I/O from all nodes, if running Windows Server 2012 R2. The common scenario here is a Scale-Out File Server backed by Mirrored Storage Spaces.

 

Here are the changes in SMB PowerShell to support SMB Scale-Out per-share redirection:

  • New “Redirected” Boolean property to Get-SmbConnection to indicate that per-share redirection is being used.
  • Get-SmbWitnessClient now includes a “ShareName” property, since witness can track connections per share, not only per server. 
  • Get-SmbWitnessClient now includes a “Flags” property, which will show “Share” when doing per-share redirection

 

3) Other SMB Witness Changes

 

While the overall functionality of the SMB Witness is largely unchanged in Windows Server 2012 R2 (aside from per-share redirection), we have put some effort in improving the SMB PowerShell cmdlets associated with it. Here are the changes:

  • The Move-SmbWitnessClient cmdlet can be referred to now simply as Move-SmbClient, This is simply a new alias that better describes what the cmdlet actually does.
  • The default view for Get-SmbWitnessClient was improved. Here’s the new list of items shown: 


Client Computer   Witness Node  File Server   Network Name  Share Name  Client State    
Name              Name             Node Name
----------------- ------------- ------------- ------------  ----------  ------------    
JOSE-V            JOSE-A1       JOSE-A2       JOSE-F        VMS4        RequestedNoti...
JOSE-V            JOSE-A1       JOSE-A2       JOSE-F        VMS3        RequestedNoti...
JOSE-V            JOSE-A1       JOSE-A3       JOSE-F        VMS1        RequestedNoti...
JOSE-V            JOSE-A1       JOSE-A3       JOSE-F        VMS2        RequestedNoti...
JOSE-V            JOSE-A2       JOSE-A1       JOSE-F        VMS6        RequestedNoti...
JOSE-V            JOSE-A2       JOSE-A1       JOSE-F        VMS5        RequestedNoti...

 

  • There is a new “NetworkName” parameter in Move-SmbClient.  If a NetworkName is specified, then only those will be moved. 

Windows Server 2012 syntax for Move-SmbWitnessClient:

    • Move-SmbWitnessClient -ClientName X -DestinationNode Y

Windows Server 2012 R2 syntax:

    • Move-SmbClient -ClientName X -DestinationNode Y [ -NetworkName Z ]

 

Note 1: If the –NetworkName is omitted in the Move-SmbClient cmdlet, all client connections will be moved to the destination.

 

Note 2: When using per-share redirection, the SMB client will always move to the file server node that owns the volume behind the file share. Using Move-SmbClient (or Move-SmbWitnessClient) in that situation has no effect.

 

4) SMB Bandwidth Limits

 

Starting with Windows Server 2012 R2, administrators can set a bandwidth limit for one or more categories by using simple PowerShell cmdlets. They can also query the limit for each category.  

 

There are 3 categories of traffic that can be limited:

  • VirtualMachine: Hyper-V over SMB traffic. This limit can only be applied on Hyper-V hosts. 
  • LiveMigration: Hyper-V Live Migration over SMB traffic. This limit can only be applied on Hyper-V hosts. 
  • Default: All other types of SMB traffic. This limit can be applied on any SMB client.

 

A category limit is an absolute value expressed in bytes per second.  For example, a category limit of 500MB means the total throughput for this category should not exceed 500 megabytes per second.

 

The PowerShell cmdlets used to manage SMB Bandwidth Limits are:

  • Get-SmbBandwidthLimit [ –Category {Default/VirtualMachine/LiveMigration} ]
  • Set-SmbBandwidthLimit –Category {Default/VirtualMachine/LiveMigration} –BytesPerSecond x 
  • Remove-SmbBandwidthLimit –Category {Default/ VirtualMachine/LiveMigration}

 

Note 1: Before you can use Bandwidth Limits, the feature needs to be enabled. For that, you should use:

  • Add-WindowsFeature FS-SMBBW

 

Note 2: A new SMB Performance counter set with an instance per category becomes available after you install the feature.  The performance counters in for this set will use the same counters used today for the SMB Client Shares counter.

 

Note 3: PowerShell allows us to specify units like KB, MB, GB after the number of bytes when specifying the BytesPerSecond parameter.

 

Note 4: The Set-SmbBandwidthLimit cmdlet won’t accept a BytesPerSecond parameter smaller than 1MB (1048576).

 

5) SMB Multi-instance

 

SMB Multi-instance is a new feature in Windows Server 2012 R2 that separates regular SMB traffic from CSV-related inter-node SMB traffic in distinct  SMB instances. This is designed to improve isolation between the two types of traffic in improve the reliability of the SMB servers. Information related to this new CSV-only instance in Windows Server 2012 R2 is hidden in all PowerShell cmdlets.

 

However, there are the changes in SMB PowerShell so an Administrator can view information related to the hidden CSV instance:

  • The –IncludeHidden option in Get-SmbConnection and Get-SmbMultichannelConnection will show the connections associated with the hidden CSV instance. 
  • There is now an InstanceName property in the full output of Get-SmbConnection, Get-SmbMultichannelConnection, Get-SmbSession and Get-SmbOpenFile. It shows either “Default” or “CSV” (only shows if using the –IncludeHidden option).

 

Note: There is really little use in inspecting the information on the hidden CSV instance, except if you’re troubleshooting CSV inter-node communications.

 

6) SMB Delegation

 

When you configure Hyper-V over SMB and you manage your Hyper-V hosts remotely using Hyper-V Manager, you will might run into access denied messages. This is because you’re using your credentials from the remote machine running Hyper-V Manager in the Hyper-V host to access a third machine (the file server). This is what we call a “double-hop”, and it’s not allowed by default for security reasons.The main problem with the scenario is that an intruder that compromises one computer in your environment could then connect to other systems in your environments without the need to provide a username and password. One way to work around this issue is to connect directly to the Hyper-V host and providing your credentials at that time, avoiding the double-hop.

 

You can also address this by configuring Constrained Delegation for SMB shares, which is a process that involves changing properties in Active Directory. The security risk is reduced here because a potential intruder double-hop would be limited to that specific use case (using SMB shares on the specified servers). The constrained delegation process was greatly simplified in Windows Server 2012 when the the Active Directory team introduced resource-based Kerberos constrained delegation, as explained at http://technet.microsoft.com/library/hh831747.aspx. However, even with  this new resource-based constrained delegation, there are still quite a few steps to enable it.

 

For Hyper-V over SMB in Windows Server 2012, we provided TechNet and blog-based guidance on how to automate constrained delegation. In Windows Server 2012 R2, SMB has a new set of cmdlets to simplify the configuration of resource-based constrained Delegation in SMB scenarios. Here are the new cmdlets introduced:

  • Get-SmbDelegation –SmbServer X
  • Enable-SmbDelegation –SmbServer X –SmbClient Y 
  • Disable-SmbDelegation –SmbServer X [–SmbClient Y] [-Force]

 

Note 1: For the Disable-SmbDelegation cmdlet, if no client is specified, delegation will be removed for all clients.

 

Note 2: These 3 new SMB cmdlets do rely on Active Directory PowerShell to perform their actions. For this reason, you need to install the Active Directory cmdlets before using the SMB delegation cmdlets. To install the Active Directory cmdlets, use:

  • Install-WindowsFeature RSAT-AD-PowerShell

 

Note 3: Because these cmdlets only work with the new resource-based delegation, the Active Directory forest must be in “Windows Server 2012” functional level. To check the Active Directory Forest Functional level, use:

  • Get-ADForest

 

Note 4: Virtual Machine Manager uses a different method to remote into the Hyper-V host. For VMM, constrained delegation is not required for management of Hyper-V of SMB.

 

7) Disable SMB1

 

SMB1 now can be completely disabled, so that the components are not even loaded. For scenarios where SMB1 is not required, this means less resource utilization, less need for patching and improved security.

 

To disable SMB1 completely, use the following PowerShell cmdlet:

  • Remove-WindowsFeature FS-SMB1

 

You can re-enable it by using:

  • Add-WindowsFeature FS-SMB1

 

Note 1: A reboot is required after this feature is enabled or disabled.

 

Note 2: For information worker scenarios, if you have Windows XP clients, you absolutely still need SMB1, since that is the only SMB version supported by Windows XP. Windows Vista and Windows 7 do not need SMB1 since they support SMB2. Windows 8 and Windows 8.1 do not need SMB1, since they support both SMB2 and SMB3.

 

Note 3: For classic server scenarios, if you have Windows Server 2003 or Windows Server 2003 R2 servers, you absolutely still need SMB1, since that is the only SMB version supported by them. Windows Server 2008 and Windows Server 2008 R2 do not need SMB1 since they support SMB2. Windows Server 2012 and Windows Server 2012 R2 do not need SMB1, since they support both SMB2 and SMB3.

 

Raw notes from the Storage Developers Conference (SDC 2013)

$
0
0

This blog post is a compilation of my raw notes from SNIA’s SDC 2013 (Storage Developers Conference).

Notes and disclaimers:

  • These notes were typed during the talks and they may include typos and my own misinterpretations.
  • Text in the bullets under each talk are quotes from the speaker or text from the speaker slides, not my personal opinion.
  • If you feel that I misquoted you or badly represented the content of a talk, please add a comment to the post.
  • I spent limited time fixing typos or correcting the text after the event. Just so many hours in a day...
  • I have not attended all sessions (since there are 4 or 5 at a time, that would actually not be possible :-)…
  • SNIA usually posts the actual PDF decks a few weeks after the event. Attendees have access immediately.
  • You can find the event agenda at http://www.snia.org/events/storage-developer2013/agenda2013

SMB3 Meets Linux: The Linux Kernel Client
Steven French, Senior Engineer SMB3 Architecture, IBM

  • Title showing is (with the strikethrough text): CIFSSMB2SMB2.1SMB3 SMB3.02 and Linux, a Status Update.
  • How do you use it? What works? What is coming?
  • Who is Steven French: maintains the Linux kernel client, at SMB3 Architect for IBM Storage
  • Excited about SMB3
  • Why SMB3 is important: cluster friendly, large IO sizes, more scalable.
  • Goals: local/remote transparency, near POSIX semantics to Samba, fast/efficient/full function/secure method, as reliable as possible over bad networks
  • Focused on SMB 2.1, 3, 3.02 (SMB 2.02 works, but lower priority)
  • SMB3 faster than CIFS. SMB3 remote file access near local file access speed (with RDMA)
  • Last year SMB 2.1, this year SMB 3.0 and minimal SMB 3.02 support
  • 308 kernel changes this year, a very active year. More than 20 developers contributed
  • A year ago 3.6-rc5 – now at 3.11 going to 3.12
  • Working on today copy offload, full linux xattr support, SMB3 UNIX extension prototyping, recover pending locks, starting work on Multichannel
  • Outline of changes in the latest releases (from kernel version 3.4 to 3.12), version by version
  • Planned for kernel 3.13: copy chunk, quota support, per-share encryption, multichannel, considering RDMA (since Samba is doing RDMA)
  • Improvements for performance: large IO sizes, credit based flow control, improved caching model. Still need to add compounding,
  • Status: can negotiate multiple dialects (SMB 2.1, 3, 3.02)
  • Working well: basic file/dir operations, passes most functional tests, can follow symlinks, can leverage durable and persistent handles, file leases
  • Need to work on: cluster enablement, persistent handles, witness, directory leases, per-share encryption, multichannel, RDMA
  • Plans: SMB 2.1 no longer experimental in 3.12, SMB 2.1 and 3 passing similar set of functional tests to CIFS
  • Configuration hints: adjusting rsize, wsize, max_pending, cache, smb3 signing, UNIX extension, nosharelock
  • UNIX extensions: POSIX pathnames, case sensitive path name, POSIX delete/rename/create/mkdir, minor extensions to stat/statfs, brl, xattr, symlinks, POSIX ACLs
  • Optional POSIX SMB3 features outlined: list of flags used for each capability
  • Question: Encryption: Considering support for multiple algorithms, since AES support just went in the last kernel.
  • Development is active! Would like to think more seriously about NAS appliances. This can be extended…
  • This is a nice, elegant protocol. SMB3 fits well with Linux workloads like HPC, databases. Unbelievable performance with RDMA.
  • Question: Cluster enablement? Durable handle support is in. Pieces missing for persistent handle and witness are small. Discussing option to implement and test witness.
  • Need to look into the failover timing for workloads other than Hyper-V.
  • Do we need something like p-NFS? Probably not, with these very fast RDMA interfaces…

Mapping SMB onto Distributed Storage
Christopher R. Hertel, Senior Principal Software Engineer, Red Hat
José Rivera, Software Engineer, Red Hat

  • Trying to get SMB running on top of a distributed file system, Gluster
  • Chris and Jose: Both work for RedHat, both part of the Samba team, authors, etc…
  • Metadata: data about data, pathnames, inode numbers, timestamps, permissions, access controls, file size, allocation, quota.
  • Metadata applies to volumes, devices, file systems, directories, shares, files, pipes, etc…
  • Semantics are interpreted in different contexts
  • Behavior: predictable outcomes. Make them the same throughout the environments, even if they are not exactly the same
  • Windows vs. POSIX: different metadata + different semantics = different behavior
  • That’s why we have a plugfest downstairs
  • Long list of things to consider: ADS, BRL, deleteonclose, directory change notify, NTFS attributes, offline ops, quota, etc…
  • Samba is a Semantic Translator. Clients expect Windows semantics from the server, Samba expects POSIX semantics from the underlying file system
  • UNIX extensions for SMB allows POSIX clients to bypass some of this translation
  • If Samba does not properly handle the SMB protocol, we call it a bug. If cannot handle the POSIX translation, it’s also a bug.
  • General Samba approach: Emulate the Windows behavior, translate the semantics to POSIX (ensure other local processes play by similar rules)
  • The Samba VFS layers SMB Protocol Initial Request Handling  VFS Layer  Default VFS Layer  actual file system
  • Gluster: Distributed File System, not a cluster file system. Brick  Directory in the underlying file system. Bricks bound together as a volume. Access via SMB, NFS, REST.
  • Gluster can be FUSE mounted. Just another access method. FUSE hides the fact that it’s Gluster underneath.
  • Explaining translations: Samba/Gluster/FUSE. Gluster is adaptable. Translator stack like Samba VFS modules…
  • Can add support for: Windows ACLs, oplocks, leases, Windows timestamps.
  • Vfs_glusterfs: Relatively new code, similar to other Samba VFS modules. Took less than a week to write.
  • Can bypass the lower VFS layers by using libgfapi. All VFS calls must be implemented to avoid errors.
  • CTDB offers three basics services: distributed metadata database (for SMB state), node failure detection/recovery, IP address service failover.
  • CTDB forms a Samba cluster. Separate from the underlying Gluster cluster. May duplicate some activity. Flexible configuration.
  • SMB testing, compared to other access methods: has different usage patterns, has tougher requirements, pushes corner cases.
  • Red Hat using stable versions, kernel 2.x or something. So using SMB1 still…
  • Fixed: Byte range locking. Fixed a bug in F_GETLK to get POSIX byte range locking to work.
  • Fixed:  SMB has strict locking and data consistency requirements. Stock Gluster config failed ping_pong test. Fixed cache bugs  ping_pong passes
  • Fixed: Slow directory lookups. Samba must do extra work to detect and avoid name collisions. Windows is case-INsensitive, POSIX is case-sensitive. Fixed by using vfs_glusterfs.
  • Still working on: CTDB node banning. Under heavy load (FSCT), CTDB permanently bans a running node. Goal: reach peak capacity without node banning. New CTDB versions improved capacity.
  • Still working on: CTDB recovery lock file loss. Gluster is a distributed FS, not a Cluster FS. In replicated mode, there are two copies of each file. If Recovery Lock File is partitioned, CTDB cannot recover.
  • Conclusion: If implementing SMB in a cluster or distributed environment, you should know enough about SMB to know where to look for trouble… Make sure metadata is correct and consistent.
  • Question: Gluster and Ceph have VFS. Is Samba suitable for that? Yes. Richard wrote a guide on how to write a VFS. Discussing a few issues around passing user context.
  • Question: How to change SMB3 to be more distributed? Client could talk to multiple nodes. Gluster working on RDMA between nodes. Protocol itself could offer more about how the cluster is setup.

Pike - Making SMB Testing Less Torturous
Brian Koropoff, Consulting Software Engineer, EMC Isilon

  • Pike – written in Python – starting with a demo
  • Support for a modest subset of SMB2/3. Currently more depth than breadth.
  • Emphasis on fiddly cases like failover, complex creates
  • Mature solutions largely in C (not convenient for prototyping)
  • Why python: ubiquitous, expressive, flexible, huge ecosystem.
  • Flexibility and ease of use over performance. Convenient abstractions. Extensible, re-usable.
  • Layers: core primitives (abstract data model), SMB2/3 packet definitions, SMB2/3 client model (connection, state, request, response), test harness
  • Core primitives: Cursor (buffer+offset indicating read/write location), frame (packet model), enums, anti-boilerplate magic. Examples.
  • SMB2/SMB3 protocol (pike.smb2) header, request/response, create {request/response} context, concrete frame. Examples.
  • SMB2/SMB3 model: SMB3 object model + glue. Future, client, connection (submit, trasceive, error handling), session, channel (treeconnect, create, read), tree, open, lease, oplocks.
  • Examples: Connect, tree connect, create, write, close. Oplocks. Leases.
  • Advanced uses. Manually construct and submit exotic requests. Override _encode. Example of a manual request.
  • Test harness (pike,test): quickly establish connection, session and tree connect to server. Host, credentials, share parameters taken from environment.
  • Odds and ends: NT time class, signing, key derivation helpers.
  • Future work: increase breadth of SMB2/3 support. Security descriptors, improvement to mode, NTLM story, API documentation, more tests!
  • http://github.com/emc-isilon/pike - open source, patches are welcome. Has to figure out how to accept contributions with lawyers…
  • Question: Microsoft has a test suite. It’s in C#, doesn’t work in our environment. Could bring it to the plugfest.
  • Question: I would like to work on implementing it for SMB1. What do you think? Not a priority for me. Open to it, but should use a different model to avoid confusion.
  • Example: Multichannel. Create a session, bind another channel to the same session, pretend failover occurred. Write fencing of stable write.

 Exploiting the High Availability features in SMB 3.0 to support Speed and Scale
James Cain, Principal Software Architect, Quantel Ltd

  • Working with TV/Video production. We only care about speed.
  • RESTful recap. RESTful filesystems talk from SDC 2010. Allows for massive scale by storing application state in the URLs instead of in the servers.
  • Demo (skipped due to technical issues): RESTful SMB3.
  • Filling pipes: Speed (throughput) vs. Bandwidth vs. Latency. Keeping packets back to back on the wire.
  • TCP Window size used to limit it. Mitigate by using multiple wires, multiple connections.
  • Filling the pipes: SMB1 – XP era. Filling the pipes required application participation. 1 session could do about 60MBps. Getting Final Cut Pro 7 to lay over SMB1 was hard. No choice to reduce latency.
  • Filling the pipes: SMB 2.0 – Vista era. Added credits, SMB2 server can control overlapped requests using credits. Client application could make normal requests and fill the pipe.
  • Filling the pipes: SMB 2.1 – 7 era. Large MTU helps.
  • Filling the pipes: SMB 3 – 8 era. Multi-path support. Enables: RSS, Multiple NICs, Multiple machines, RDMA.
  • SMB3 added lots of other features for high availability and fault tolerance. SignKey derivation.
  • Filesystem has DirectX GUI :-) - We use GPUs to render, so our SMB3 server has Cuda compute built in too. Realtime visualization tool for optimization.
  • SMB3 Multi-machine with assumed shared state. Single SMB3 client talking to two SMB3 servers. Distributed non-homogeneous storage behind the SMB servers.
  • Second NIC (channel) initiation has no additional CREATE. No distinction on the protocol between single server or multiple server. Assume homogeneous storage.
  • Asking Microsoft to consider “NUMA for disks”. Currently, shared nothing is not possible. Session, trees, handles are shared state.
  • “SMB2++” is getting massive traction. Simple use cases are well supported by the protocol. SMB3 has a high cost of entry, but lower than writing n IFS in kernel mode.
  • There are limits to how far SMB3 can scale due to its model.
  • I know this is not what the protocol is designed to do. But want to see how far I can go.
  • It could be help by changing the protocol to have duplicate handle semantics associated with the additional channels.
  • The protocol is really, really flexible. But I’m having a hard time doing what I was trying to do.
  • Question: You’re basic trying to do Multichannel  to multiple machines. Do you have a use case? I’m experimenting with it. Trying to discover new things.
  • Question: You could use CTDB to solve the problem. How much would it slow down? It could be a solution, not an awful lot of state.             

SMB3 Update
David Kruse, Development Lead, Microsoft

  • SMB 3.02 - Don’t panic! If you’re on the road to SMB3, there are no radical changes.
  • Considered not revving the dialect and doing just capability bits, but thought it would be better to rev the dialect.
  • Dialects vs. Capabilities: Assymetric Shares, FILE_ATTRIBUTE_INTEGRITY_STREAMS.
  • SMB 2.0 client attempting MC or CA? Consistency/documentation question.
  • A server that receives a request from a client with a flag/option/capability that is not valid for the dialect should ignore it.
  • Showing code on how to mask the capabilities that don’t make sense for a specific dialect
  • Read/Write changes: request specific flag for unbuffered IO. RDMA flag for invalidation.
  • Comparing “Traditional” File Server Cluster vs. “Scale-Out” File Server cluster
  • Outlining the asymmetric scale-out file server cluster. Server-side redirection. Can we get the client to the optimal node?
  • Asymmetric shares. New capability in the TREE_CONNECT response. Witness used to notify client to move.
  • Different connections for different shares in the same scale-out file server cluster. Share scope is the unit of resource location.
  • Client processes share-level “move” in the same fashion as a server-level “move” (disconnect, reconnects to IP, rebinds handle).
  • If the cost accessing the data is the same for all nodes, there is no need to move the client to another node.
  • Move-SmbWitnessClient will not work with asymmetric shares.
  • In Windows, asymmetric shares are typically associated with Mirrored Storage Spaces, not iSCSI/FC uniform deployment. Registry key to override.
  • Witness changes: Additional fields: Sharename, Flags, KeepAliveTimeOutInSeconds.
  • Witness changes: Multichannel notification request. Insight into arrival/loss of network interfaces.
  • Witness changes: Keepalive. Timeout for async IO are very coarse. Guarantees client and server discover lost peer in minutes instead of hours.
  • Demos in Jose’s blog. Thanks for the plug!
  • Diagnosability events. New always-on events. Example: failed to reconnect a persistent handle includes previous reconnect error and reason. New events on server and client.
  • If Asymmetric is not important to you, you don’t need to implement it.
  • SMB for IPC (Inter-process communications) – What happened to named pipes?
  • Named pipes over SMB has been declined in popularity. Performance concerns with serialized IO. But this is a property of named pipes, not SMB.
  • SMB provides: discovery, negotiation, authentication, authorization, message semantics, multichannel, RDMA, etc…
  • If you can abstract your application as a file system interface, you could extend it to removte via SMB.
  • First example: Remote Shared Virtual Disk Protocol
  • Second example: Hyper-V Live Migration over SMB. VID issues writes over SMB to target for memory pages. Leverages SMB Multichannel, SMB Direct.
  • Future thoughts on SMB for IPC. Not a protocol change or Microsoft new feature. Just ideas shared as a thought experiment.
    • MessageFs – User mode-client and user-mode server. Named Pipes vs. MessageFs. Each offset marks a distinct transaction, enables parallel actions.
    • MemFs – Kernel mode component on the server side. Server registers a memory region and clients can access that memory region.
    • MemFs+ - What if we combine the two? Fast exchange for small messages plus high bandwidth, zero copy access for large transfers. Model maps directly to RDMA: send/receive messages, read/write memory access.
  • One last thing… On Windows 8.1, you can actually disable SMB 1.0 completely.

Architecting Block and Object Geo-replication Solutions with Ceph
Sage Weil, Founder & CTO, Inktank

  • Impossible to take notes, speaker goes too fast :-)

1 S(a) 2 M 3 B(a) 4
Michael Adam, SerNet GmbH - Delivered by Volker

  • What is Samba? The open source SMB server (Samba3). The upcoming open source AD controller (Samba4). Two different projects.
  • Who is Samba? List of team members. Some 35 or so people… www.samba.org/samba/team
  • Development focus: Not a single concentrated development effort. Various companies (RedHat, SuSE, IBM, SerNet, …) Different interests, changing interests.
  • Development quality: Established. Autobuild selftest mechanism. New voluntary review system (since October 2012).
  • What about Samba 4.0 after all?
    • First (!?) open source Active Directory domain controller
    • The direct continuation of the Samba 3.6 SMB file server
    • A big success in reuniting two de-facto separated projects!
    • Also a big and important file server release (SMB 2.0 with durable handles, SMB 2.1 (no leases), SMB 3.0 (basic support)
  • History. Long slide with history from 2003-06-07 (Samba 3.0.0 beta 1) to 2012-12-11 (Samba 4.0.0). Samba4 switched to using SMB2 by default.
  • What will 4.1 bring? Current 4.1.0rc3 – final planned for 2013-09-27.
  • Samba 4.1 details: mostly stabilization (AD, file server). SMB2/3 support in smbclient, including SMB3 encryption. Server side copy. Removed SWAT.
  • Included in Samba 4.0: SMB 2.0 (durable handles). SMB 2.1 (multi-credit, large MTU, dynamic reauth), SMB 3.0 (signing, encryption, secure negotiate, durable handles v2)
  • Missing in Samba 4.0: SMB 2.1 (leasing*, resilient file handles), SMB 3.0 (persistent file handles, multichannel*, SMB direct*, witness*, cluster features, storage features*, …) *=designed, started or in progress
  • Leases: Oplocks done right. Remove 1:1 relationship between open and oplock, add lease/oplock key. http://wiki.samba.org/index.php/Samba3/SMB2#Leases
  • Witness: Explored protocol with Samba rpcclient implementation. Working on pre-req async RPC. http://wiki.samba.org/index.php/Samba3/SMB2#Witness_Notification_Protocol
  • SMB Direct:  Currently approaching from the Linux kernel side. See related SDC talk. http://wiki.samba.org/index.php/Samba3/SMB2#SMB_Direct
  • Multichannel and persistent handles: just experimentation and discussion for now. No code yet.

Keynote: The Impact of the NVM Programming Model
Andy Rudoff, Intel

  • Title is Impact of NVM Programming Model (… and Persistent Memory!)
  • What do we need to do to prepare, to leverage persistent memory
  • Why now? Programming model is decades old!
  • What changes? Incremental changes vs. major disruptions
  • What does this means to developers? This is SDC…
  • Why now?
  • One movements here: Block mode innovation (atomics, access hints, new types of trim, NVM-oriented operations). Incremental.
  • The other movement: Emerging NVM technologies (Performance, performance, perf… okay, Cost)
  • Started talking to companies in the industry  SNIA NVM Programming TWG - http://snia.org/forums/sssi/nvmp
  • NVM TWG: Develop specifications for new software “programming models”as NVM becomes a standard feature of platforms
  • If you don’t build it and show that it works…
  • NVM TWG: Programming Model is not an API. Cannot define those in a committee and push on OSVs. Cannot define one API for multiple OS platforms
  • Next best thing is to agree on an overall model.
  • What changes?
  • Focus on major disruptions.
  • Next generation scalable NVM: Talking about resistive RAM NVM options. 1000x speed up over NND, closer do DRAM.
  • Phase Change Memory, Magnetic Tunnel Junction (MT), Electrochemical Cells (ECM), Binary Oxide Filament Cells, Interfacial Switching
  • Timing. Chart showing NAND SATA3 (ONFI2, ONFI3), NAND PCIe Gen3 x4 ONFI3 and future NVM PCIE Gen3 x4.
  • Cost of software stack is not changing, for the last one (NVM PCIe) read latency, software is 60% of it?!
  • Describing Persistent Memory…
  • Byte-addressable (as far as programming model goes), load/store access (not demand-paged), memory-like performance (would stall a CPU load waiting for PM), probably DMA-able (including RDMA)
  • For modeling, think battery-backed RAM. These are clunky and expensive, but it’s a good model.
  • It is not tablet-like memory for the entire system. It is not NAND Flash (at least not directly, perhaps with caching). It is not block-oriented.
  • PM does not surprise the program with unexpected latencies (no major page faults). Does not kick other things out of memory. Does not use page cache unexpectedly.
  • PM stores are not durable until data is flushed. Looks like a bug, but it’s always been like this. Same behavior that’s been around for decades. It’s how physics works.
  • PM may not always stay in the same address (physically, virtually). Different location each time your program runs. Don’t store pointers and expect them to work. You have to use relative pointers. Welcome to the world of file systems…
  • Types of Persistent Memory: Battery-backed RAM. DRAM saved on power failure. NVM with significant caching. Next generation NVM (still quite a bit unknown/emerging here).
  • Existing use cases: From volatile use cases (typical) to persistent memory use case (emerging). NVDIMM, Copy to Flash, NVM used as memory.
  • Value: Data sets with no DRAM footprint. RDMA directly to persistence (no buffer copy required!). The “warm cache” effect. Byte-addressable. Direct user-mode access.
  • Challenges: New programming models, API. It’s not storage, it’s not memory. Programming challenges. File system engineers and database engineers always did this. Now other apps need to learn.
  • Comparing to the change that happened when we switched to parallel programming. Some things can be parallelized, some cannot.
  • Two persistent memory programming models (there are four models, more on the talk this afternoon).
  • First: NVM PM Volume mode. PM-aware kernel module. A list of physical ranges of NVMs (GET_RANGESET).
  • For example, used by file systems, memory management, storage stack components like RAID, caches.
  • Second: NVM PM File. Uses a persistent-memory-aware file system. Open a file and memory map it. But when you do load and store you go directly to persistent memory.
  • Native file APIs and management. Did a prototype on Linux.
  • Application memory allocation. Ptr=malloc(len). Simple, familiar interface. But it’s persistent and you need to have a way to get back to it, give it a name. Like a file…
  • Who uses NVM.PM.FILE. Applications, must reconnect with blobs of persistence (name, permissions)
  • What does it means to developers?
  • Mmap() on UNIX, MapViewOfFile() on Windows. Have been around for decades. Present in all modern operating systems. Shared or Copy-on-write.
  • NVM.PM.FILE – surfaces PM to application. Still somewhat raw at this point. Two ways: 1-Build on it with additional libraries. 2-Eventually turn to language extensions…
  • All these things are coming. Libraries, language extensions. But how does it work?
  • Creating resilient data structures. Resilient to a power failure. It will be in state you left it before the power failure. Full example: resilient malloc.
  • In summary: models are evolving. Many companies in the TWG. Apps can make a big splash by leveraging this… Looking forward to libraries and language extensions.

Keynote: Windows Azure Storage – Scaling Cloud Storage
Andrew Edwards, Microsoft

  • Turning block devices into very, very large block devices. Overview, architecture, key points.
  • Overview
  • Cloud storage: Blobs, disks, tables and queues. Highly durable, available and massively scalable.
  • 10+ trillion objects. 1M+ requests per seconds average. Exposed via easy and open REST APIs
  • Blobs: Simple interface to retrieve files in the cloud. Data sharing, big data, backups.
  • Disks: Built on top on blobs. Mounted disks as VHDs stored on blobs.
  • Tables: Massively scalable key-value pairs. You can do queries, scan. Metadata for your systems.
  • Queues: Reliable messaging system. Deals with failure cases.
  • Azure is spread all over the world.
  • Storage Concepts: Accounts  ContainerBlobs/TableEntities/QueuesMessages. URLs to identify.
  • Used by Microsoft (XBOX, SkyDrive, etc…) and many external companies
  • Architecture
  • Design Goals: Highly available with strong consistency. Durability, scalability (to zettabytes). Additional information in the SOSP paper.
  • Storage stamps: Access to blog via the URL. LB  Front-end  Partition layer  DFS Layer. Inter-stamp partition replication.
  • Architecture layer: Distributed file system layer. JBODs, append-only file system, each extent is replicated 3 times.
  • Architecture layer: Partition layer. Understands our data abstractions (blobs, queues, etc). Massively scalable index. Log Structure Merge Tree. Linked list of extents
  • Architecture layer: Front-end layer. REST front end. Authentication/authorization. Metrics/logging.
  • Key Design Points
  • Availability with consistency for writing. All writes we do are to a log. Append to the last extent of the log.
  • Ordered the same across all 3 replicas. Success only if 3 replicas are commited. Extents get sealed (no more appends) when they get to a certain size.
  • If you lose a node, seal the old two copies, create 3 new instances to append to. Also make a 3rd copy for the old one.
  • Availability with consistency for reading. Can read from any replica. Send out parallel read requests if first read is taking higher than 95% latency.
  • Partition Layer: spread index/transaction processing across servers. If there is a hot node, split that part of the index off. Dynamically load balance. Just the index, this does not move the data.
  • DFS Layer: load balancing there as well. No disk or node should be hot. Applies to both reads and writes. Lazily move replicas around to load balancing.
  • Append only system. Benefits: simple replication, easier diagnostics, erasure coding, keep snapshots with no extra cost, works well with future dirve technology. Tradeoff: GC overhead.
  • Our approach to the CAP theorem. Tradeoff in Availability vs. Consistency. Extra flexibility to achieve C and A at the same time.
  • Lessons learned: Automatic load balancing. Adapt to conditions. Tunable and extensible to tune load balancing rules. Tune based on any dimension (CPU, network, memory, tpc, GC load, etc.)
  • Lessons learned: Achieve consistently low append latencies. Ended up using SSD journaling.
  • Lessons learned: Efficient upgrade support. We update frequently, almost consistently. Handle them almost as failures.
  • Lessons learned: Pressure point testing. Make sure we’re resilient despite errors.
  • Erasure coding. Implemented at the DFS Layer. See last year’s SDC presentation.
  • Azure VM persistent disks: VHDs for persistent disks are directly stored in Windows Azure Storage blobs. You can access your VHDs via REST.
  • Easy to upload/download your own VHD and mount them. REST writes are blocked when mounted to a VM. Snapshots and Geo replication as well.
  • Separating compute from storage. Allows them to be scaled separately. Provide flat network storage. Using a Quantum 10 network architecture.
  • Summary: Durability (3 copies), Consistency (commit across 3 copies). Availability (can read from any of the 3 relicas). Performance/Scale.
  • Windows Azure developer website: http://www.windowsazure.com/en-us/develop/net
  • Windows Azure storage blog: http://blogs.msdn.com/b/windowsazurestorage
  • SOSP paper/talk: http://blogs.msdn.com/b/windowsazure/archive/2011/11/21/windows-azure-storage-a-highly-available-cloud-storage-service-with-strong-consistency.aspx

SMB Direct update
Greg Kramer, Microsoft
Tom Talpey, Microsoft

  • Two parts: 1 - Tom shares Ecosystem status and updates, 2 - Greg shares SMB Direct details
  • Protocols and updates: SMB 3.02 is a minor update. Documented in MS-SMB2 and MS-SMBD. See Dave's talk yesterday.
  • SMB Direct specifies the SMB3 RDMA transport, works with both SMB 3.0 and SMB 3.02
  • Windows Server 2012 R2 – GA in October, download from MSDN
  • Applications using SMB3 and SMB Direct: Hyper-V VHD, SQL Server
  • New in R2: Hyper-V Live Migration over SMB, Shared VHDX (remote shared virtual disk, MS-RSVD protocol)
  • RDMA Transports: iWARP (IETF RDMA over TCP), InfiniBand, RoCE. Ethernet: iWARP and RoCE – 10 or 40GbE, InfiniBand: 32Gbps (QDR) or 54Gbps (FDR)
  • RDMA evolution: iWARP (IETF standard, extensions currently active in IETF). RoCE (routable RoCE to improve scale, DCB deployment still problematic). InfiniBand (Roadmap to 100Gbps, keeping up as the bandwidth/latency leader).
  • iWARP: Ethernet, routable, no special fabric required, Up to 40GbE with good latency and full throughput
  • RoCE: Ethernet, not routable, requires PFC/DCB, Up to 40GbE with good latency and full throughput
  • InfinBand: Specialized interconnect, not routable, dedicated fabric and switching, Up to 56Gbps with excellent latency and throughput
  • SMB3 Services: Connection management, authentication, multichannel, networking resilience/recovery, RDMA, File IO Semantics, control and extension semantics, remote file system access, RPC
  • The ISO 7-layer model: SMB presents new value as a Session layer (RDMA, multichannel, replay/recover). Move the value of SMB up the stack.
  • SMB3 as a session layer: Applications can get network transparency, performance, recovery, protection (signing, encryption, AD integration). Not something you see with other file systems or file protocols.
  • Other: Great use by clustering (inter-node communication), quality of service, cloud deployment
  • In summary. Look to SMB for even broader application (like Hyper-V Live Migration did). Broader use of SMB Direct. Look to see greater application “fidelity” (sophisticated applications transparently server by SMB3)
  • Protocol enhancements and performance results
  • Where can we reduce IO costs? We were extremely happy about performance, there was nothing extremely easy to do next, no low-hanging fruit.
  • Diagram showing the App/SMB client/Client RNIC/Server RNIC. How requests flow in SMB Direct.
  • Interesting: client has to wait for the invalidation completion. Invalidation popped up as an area of improvement. Consumes cycles, bus. Adds IO, latency. But it’s required.
  • Why pend IO until invalidation is completed? This is storage, we need to be strictly correct. Invalidation guarantees: data is in a consistent state after DMA, peers no longer has access.
  • Registration caches cannot provides these guarantees, leading to danger of corruption.
  • Back to the diagram. There is a way to decorate a request with the invalidation  Send and Invalidate. Provides all the guarantees that we need!
  • Reduces RNIC work requests per IO by one third for high IOPs workload. That’s huge! Already supported by iWARP/RoCE/InfiniBand
  • No changes required at the SMB Direct protocol. Minor protocol change in SMB 3.02 to support invalidation. New channel value in the SMB READ and SMB WRITE.
  • Using Send and Invalidate (Server). Only one invalidate per request, have to be associated with the request in question. You can leverage SMB compounding.
  • Only the first memory descriptor in the SMB3 read/write array may be remotely invalidated. Keeping it simple.
  • Using Send and Invalidate (Client). Not a mandate, you can still invalidate “manually” if not using remote invalidate. Must validate that the response matches.
  • Performance Results (drumroll…)
  • Benchmark configuration: Client and Server config: Xeon E5-2660. 2 x ConnectX-3 56Gbps InfiniBand. Shunt filter in the IO path. Comparing WS2012 vs. WS2012 R2 on same hardware.
  • 1KB random IO. Uses RDMA send/receive path. Unbuffered, 64 queue depth.
    • Reads: 881K IOPs. 2012 R2 is +12.5% over 2012. Both client and server CPU/IO reduced (-17.3%, -36.7%)
    • Writes: 808K IOPs. 2012 R2 is +13.5% over 2012. Both client and server CPU/IO reduced (-16%, -32.7%)
  • 8KB random IO. Uses RDMA read/writes. Unbuffered, 64 queue depth.
    • Reads: 835K IOPs. 2012 R2 is +43.3% over 2012. Both client and server CPU/IO reduced (-37.1%, -33.2%)
    • Writes: 712K IOPs. 2012 R2 is +30.2% over 2012. Both client and server CPU/IO reduced (-26%, -14.9%)
  • 512KB sequential IO. Unbuffered, 12 queue depth. Already maxing out before. Remains awesome. Minor CPU utilization decrease.
    • Reads: 11,366 MBytes/sec. 2012 R2 is +6.2% over 2012. Both client and server CPU/IO reduced (-9.3%, -14.3%)
    • Writes: 11,412 MBytes/sec: 2012 R2 is +6% over 2012. Both client and server CPU/IO reduced (-12.2%, -10.3%)
  • Recap: Increased IOPS (up to 43%) and high bandwidth. Decrease CPU per IO (up to 36%).
  • Client has more CPU for applications. Server scales to more clients.
  • This includes other optimization in both the client in the server. NUMA is very important.
  • No new hardware required. No increase number of connections, MRs, etc.
  • Results reflect the untuned, out-of-the-box customer experience.
  • One more thing… You might be skeptical, especially about the use of shunt filter.
  • We never get to see this in our dev environment, we don’t have the high end gear. But...
  • Describing the 3U Violin memory array running Windows Server in a clustered configuration. All flash storage. Let’s see what happens…
  • Performance on real IO going to real, continuously available storage:
    • 100% Reads – 4KiB: >1Million IOPS
    • 100% Reads – 8KiB: >500K IOPS
    • 100% Writes – 4KiB: >600K IOPS
    • 100% Writes – 8KiB: >300K IOPS
  • Questions?

A Status Report on SMB Direct (RDMA) for Samba
Richard Sharpe, Samba Team Member, Panzura

  • I work at Panzura but has been done on my weekends
  • Looking at options to implement SMB Direct
  • 2011 – Microsoft introduced SMB direct at SDC 2011. I played around with RDMA
  • May 2012 – Tutorial on SMB 3.0 at Samba XP
  • Mellanox supplied some IB cards to Samba team members
  • May 2013 – More Microsoft presentations with Microsoft at Samba XP
  • June 2013 – Conversations with Mellanox to discuss options
  • August 2013 – Started circulating a design document
  • Another month or two before it’s hooked up with Samba.
  • Relevant protocol details: Client connections via TCP first (port 445). Session setup, connects to a share. Queries network interfaces. Place an RDMA Connection to server on port 5445, brings up SMB Direct Protocol engine
  • Client sends negotiate request, Dialect 0x300, capabilities field. Server Responds.
  • Diagram with SMB2 spec section 4.8 has an example
  • SMB Direct: Small protocol - Negotiate exchange phase, PDU transfer phase.
  • Structure of Samba. Why did it take us two years? Samba uses a fork model. Master smbd forks a child. Easy with TCP. Master does not handle SMB PDUs.
  • Separate process per connection. No easy way to transfer connection between them.
  • Diagram with Samba structure. Problem: who should listen on port 5445? Wanted RDMA connection to go to the child process.
  • 3 options:
  • 1 - Convert Samba to a threaded model, everything in one address space. Would simplify TCP as well. A lot of work… Presents other problems.
  • 2 - Separate process to handle RDMA. Master dmbd, RDMA handler, multiple child smbd, shared memory. Layering violation! Context switches per send/receive/read/write. Big perf hit.
  • 3 - Kernel driver to handle RDMA. Smbdirect support / RDMA support (rdmacm, etc) / device drivers. Use IOCTLs. All RDMA work in kernel, including RDMA negotiate on port 5445. Still a layering violation. Will require both kernel and Samba knowledge.
  • I decided I will follow this kernel option.
  • Character mode device. Should be agnostic of the card used. Communicate via IOCTLs (setup, memory params, send/receive, read/write).
  • Mmap for RDMA READ and RDMA WRITE. Can copy memory for other requests. Event/callback driven. Memory registration.
  • The fact that is looks like a device driver is a convenience.
  • IOCTLs: set parameters, set session ID, get mem params, get event (includes receive, send complete), send pdu, rdma read and write, disconnect.
  • Considering option 2. Doing the implementation of option 3 will give us experience and might change later.
  • Amortizing the mode switch. Get, send, etc, multiple buffers per IOCTL. Passing an array of objects at a time.
  • Samba changes needed….
  • Goals at this stage: Get something working. Allow others to complete. It will be up on github. Longer term: improve performance with help from others.
  • Some of this work could be used by the SMB client
  • Status: A start has been made. Driver loads and unloads, listens to connections. Working through the details of registering memory. Understand the Samba changes needed.
  • Weekend project! http://github.com/RichardSharpe/smbdirect-driver
  • Acknowledgments: Microsoft. Tom Talpey. Mellanox. Or Gerlitz. Samba team members.

CDMI and Scale Out File System for Hadoop
Philippe Nicolas, Scality

  • Short summary of who is Scality. Founded 2009. HQ in SF. ~60 employees, ~25 engineers in Paris. 24x7 support team. 3 US patents. $35Min 3 rounds.
  • Scality RING. Topology and name of the product. Currently in the 4.2 release. Commodity servers and storage. Support 4 LINUX distributions. Configure Scality layer. Create a large pool of storage.
  • Ring Topology. End-to-end Paralelism. Object Storage. NewSQL DB. Replication. Erasure coding. Geo Redundancy. Tiering. Multiple access methods (HTTP/REST, CDMI, NFS, CIFS, SOFS). GUI/CLI management.
  • Usage: e-mail, file storage, StaaS, Digital Media, Big Data, HPC
  • Access methods: APIs: RS2 (S3 compatible API), Sproxyd, RS2 light, SNIA CDMI. File interface: Scality Scale Out File System (SOFS), NFS, CIFS, AFP, FTP. Hadoop HDFS. OpenStack Cinder (since April 2013).
  • Parallel network file system. Limits are huge – 2^32 volumes (FS), 2^24 namespaces, 2^64 files. Sparse files. Aggregated throughput, auto-scaling with storage or access node addition.
  • CDMI (path and ID based access). Versions 1.0.1., 1.0.2. On github. CDMI client java library (CaDMIum), set of open source filesystem tools. On github.
  • Apache Hadoop. Transform commodity hard in data storage service. Largely supported by industry and end user community. Industry adoption: big names adopting Hadoop.
  • Scality RING for Hadoop. Replace HDFS with the Scality FS. We validate Hortonworks and Cloudera. Example with 12 Hadoop nodes for 12 storage nodes. Hadoop task trackers on RING storage nodes.
  • Data compute and storage platform in ONE cluster. Scality Scale Out File System (SOFS)  instead of HDFS. Advanced data protection (data replication up to 6 copies, erasure coding). Integration with Hortonworks HDP 1.0 & Cloudera CDH3/CDH4. Not another Hadoop distribution!
  • Summary: This is Open Cloud Access: access local or remotely via file and block interface. Full CDMI server and client. Hadoop integration (convergence approach). Comprehensive data storage platform.

Introduction to HP Moonshot
Tracy Shintaku, HP

  • Today’s demands – pervasive computing estimates. Growing internet of things (IoT).
  • Using SoC technologies used in other scenarios for the datacenter.
  • HP Moonshot System. 4U. World’s first low-energy software-defined server. HP Moonshot 1500 Chassis.
  • 45 individually serviceable hot-plug artrdges. 2 network switches, private fabric. Passive base plane.
  • Introducing HP Proliant Moonshot Server (passing around the room). 2000 of these servers in a rack. Intel Atom S1260 2GHz, 8GB DDR ECC 1333MHz. 500GB or 1TBHDD or SSD.
  • Single server = 45 servers per chassis. Quad-server = 180 servers per chassis. Compute, storage or combination. Storage cartridges with 2 HDD shared by 8 servers.
  • Rear view of the chassis. Dual 4QSFP network uplinks each with 4 x 40GB), 5 hot-plug fans, Power supplies, management module.
  • Ethernet – traffic isolation and stacking for resiliency with dual low-latency switches. 45 servers  dual switches  dual uplink modules.
  • Storage fabric. Different module form factors allow for different options: Local storage. Low cost boot and logging. Distributed storage and RAID. Drive slices reduce cost of a boot drive 87%.
  • Inter-cartridge private 2D Taurus Ring – available in future cartridges. High speed communication lanes between servers. Ring fabric, where efficient localized traffic is benefitial.
  • Cartridge roadmap. Today to near future to future. CPU: Atom  Atom, GPU, DSP, x86, ARM. Increasing variety of workloads static web servers now to hosting, financial servers in the future.
  • Enablement: customer and partner programs. Partner program. Logo wall for technology partners. Solution building program. Lab, services, consulting, financing.
  • Partners include Redhat, Suse, Ubuntu, Hortonworks, MapR, Cloudera, Couchbase, Citrix, Intel, AMD, Calxeda, Applied Micro, TI, Marvell, others. There’s a lot of commonality with OpenStack.
  • Web site: http://h17007.www1.hp.com/us/en/enterprise/servers/products/moonshot/index.aspx 

NFS on Steroids: Building Worldwide Distributed File System
Gregory Touretsky, Intel

  • Intel is a big organization, 6500 Its @ 59 sites, 95,200 employes, 142000 devices.
  • Every employee doing design has a Windows machines, but also interact with NFS backend
  • Remote desktop in interactive pool. Talks to NFS file servers, glued together with name spaces.
  • Large batch pools that do testing. Models stored in NFS.
  • Various application servers, running various systems. Also NIS, Cron servers, event monitors, configuration management.
  • Uses Samba to provide access to NFS file servers using SMB.
  • Many sites, many projects. Diagram with map of the world and multiple projects spanning geographies.
  • Latency between 10s t 100s ms. Bandwidth: 10s Mbps to 10s of Gbpsp.
  • Challenge: how to get to our customers and provide the ability to collaborate across the globe in a secure way
  • Cross-site data access in 2012. Multiple file servers, each with multiple exports. Clients access servers on same site. Rsync++ for replication between sites (time consuming).
  • Global user and group accounts. Users belong to different groups in different sites.
  • Goals: Access every file in the world from anywhere, same path, fault tolerant, WAN friendly, every user account and group on every site, Local IO performance not compromised.
  • Options: OpenAFS (moved out many years ago, decided not to got back). Cloud storage (concern with performance). NFS client-side caching (does not work well, many issues). WAN optimization (have some in production, help with some protocols, but not suitable for NFS). NFS site-level caching (proprietary and open source NFS Ganesha). In house (decided not to go there).
  • Direct NFS mount over WAN optimized tunnel. NFS ops terminated at the remote site, multiple potential routes, cache miss. Not the right solution for us.
  • Select: Site-level NFS caching and Kerberos. Each site has NFS servers and Cache servers. Provides instant visibility and minimizes amount of data transfer across sites.
  • Cache is also writable. Solutions with write-through and write-back caching.
  • Kerberos authentication with NFS v3. There are some problems there.
  • Cache implementations: half a dozen vendors, many not suitable for WAN. Evaluating alternatives.
  • Many are unable to provide a disconnected mode of operation. That eliminated a number of vendors.
  • Consistency vs. performance. Attribute cache timeout. Nobody integrates this at the directory level. Max writeback delay.
  • Optimizations. Read cache vs. Read/Write cache, delegations, proactive attribute validation for hot files, cache pre-population.
  • Where is it problematic? Application is very NFS unfriendly, does not work well with caching. Some cases it cannot do over cache, must use replication.
  • Problems: Read once over high latency link. First read, large file, interactive work. Large % of non-cacheable ops (write-through). Seldom access, beyond cache timeout.
  • Caching is not a business continuity solution. Only a partial copy of the data.
  • Cache management implementation. Doing it at scale is hard. Nobody provides a solution that fits our needs.
  • Goal: self-service management for data caching. Today administrators are involved in the process.
  • Use cases: cache my disk at site X, modify cache parameters, remove cache, migrate source/cache, get cache statistics, shared capacity management, etc.
  • Abstract the differences between the different vendors with this management system.
  • Management system example: report with project, path (mount point), size, usage, cached cells. Create cell in specific site for specific site.
  • Cache capacity planning. Goal: every file to be accessible on-demand everywhere.
  • Track cache usage by org/project. Shared cache capacity, multi-tenant. Initial rule of thumb: 7-10% of the source capacity, seeding capacity at key locations
  • Usage models: Remote validation (write once, read many). Get results back from remote sites (write once, read once). Drop box (generate in one site, get anywhere). Single home directory (avoid home directory in every site for every user, cache remote home directories). Quick remote environment setup, data access from branch location.
  • NFS (RPC) Authentication. Comparing AUTH_SYS and RPCSEC_GSS (KRB5). Second one uses an external KDC, gets past the AUTH_SYS limitation of 16 group IDs.
  • Bringing Kerberos? Needs to make sure this works as well as Windows with Active Directory. Need to touch everything (Linux client, NFS file servers, SSH, batch scheduler, remote desktop/interactive servers, name space/automounter (trusted hosts vs. regular hosts), Samba (used as an SMB gateway to NFS), setuid/sudo, cron jobs and service accounts (keytab management system),
  • Supporting the transition from legacy mounts to Kerberos mount. Must support a mixed environment. Introducing second NIS domain.
  • Welcome on board GDA airlines (actual connections between different sites). Good initial feedback from users (works like magic!)
  • Summary: NFS can be accessed over WAN – using NFS caching proxy. NFSv3 environment can be kerberized (major effort is required, transition is challenging, it would be as challenging for NFSv5/KRB)

Forget IOPS: A Proper Way to Characterize & Test Storage Performance
Peter Murray, SwiftTest

  • About what we learned in the last few years
  • Evolution: Vendor IOPs claims, test in production and pray, validate with freeware tools (iometer, IOZone), validate with workload models
  • What is storage validation? Characterize the various applications, workloads. Diagram: validation appliance, workload emulations, storage under test.
  • Why should you care? Because customers do care! Product evaluations, vendor bakeoffs, new feature and technology evaluations, etc…
  • IOPS: definition from the SNIA dictionary. Not really well defined. One size does not fit all. Looking at different sizes.
  • Real IO does not use a fixed size. Read/write may be a small portion of it in certain workloads. RDMA read/write may erode the usefulness of isolated read/write.
  • Metadata: data about data. Often in excess of 50%, sometimes more than 90%. GoDaddy mentioned that 94% of workloads are not read/write.
  • Reducing metadata impact: caching with ram, flash, SSD helps but it’s expensive.
  • Workloads: IOPS, metadata and your access pattern. Write/read, random/sequential, IO/metadata, block/chunk size, etc.
  • The important of workloads: Understand overload and failure conditions. Understand server, cluster, deduplication, compression, network configuration and conditions
  • Creating and understanding workloads. Access patterns (I/O mix: read/write%, metadata%) , file system (depth, files/folder, file size distribution), IO parameters (block size, chunk size, direction), load properties (number of users, actions/second, load variability/time)
  • Step 1 - Creating a production model. It’s an art, working to make it a science. Production stats + packet captures + pre-built test suites = accurate, realistic work model.
  • Looking at various workload analysis
  • Workload re-creation challenges: difficult. Many working on these. Big data, VDI, general VM, infinite permutations.
  • Complex workloads emulation is difficult and time consuming. You need smart people, you need to spend the time.
  • Go Daddy shared some of the work on simulation of their workload. Looking at diagram with characteristics of a workload.
  • Looking at table with NFSv3/SMB2 vs. file action distribution.
  • Step 2: Run workload model against the target.
  • Step 3: Analyze the results for better decisions. Analytics leads to Insight. Blocks vs. file. Boot storm handing, limits testing, failure modes, effects of flash/dedup/tiering/scale-out.
  • I think we’ll see dramatic changes with the use of Flash. Things are going to change in the next few years.
  • Results analysis: Performance. You want to understand performance, spikes during the day, what causes them. Response times, throughput.
  • Results analysis: Command mix. Verify that the execution reflects the expected mix. Attempts, successes, errors, aborts.
  • Summary: IOPs alone cannot characterize real app storage performance. Inclusion of metadata is essential, workload modeling and purpose-build load generation appliances are the way to emulate applications. The more complete the emulation, the deeper the understanding.
  • If we can reduce storage cost from 40% to 20% of the solution by better understanding the workload, you can save a lot of money.

pNFS, NFSv4.1, FedFS and Future NFS Developments
Tom Haynes, NetApp

  • Tom covering for Alex McDonald, who is sick. His slides.
  • We want to talk about how the protocol get defined, how it interfact with different application vendors and customers.
  • Looking at what is happening on the Linux client these days.
  • NFS: Ubiquitous and everywhere. NFSv3 is very successful, we can’t dislodge it. We though everyone would go for NFSv4 and it’s now 10 years later…
  • NFSv2 in 1983, NFSv3 in 1995, NFSv4 in 2003, NFSv4.1 in 2010. NFSv4.2 to be agreed at the IETF – still kinks in the protocol that need to be ironed out. 2000=DAS, 2010=NAS, 2020=Scale-Out
  • Evolving requirements. Adoption is slow. Lack of clients was a problem with NFSv4. NFSv3 was just “good enough”. (It actually is more than good enough!)
  • Industry is changing, as are requirements. Economic trends (cheap and fast cluster, cheap and fast network, etc…)
  • Performance: NFSv3 single threaded bottlenecks in applications (you can work around it).
  • Business requirements. Reliability (sessions) is a big requirement
  • NFSv4 and beyond.
  • Areas for NFSv4, NFSv4.1 and pNFS: Security, uniform namespaces, statefulness/sessions, compound operations, caching (directory and file delegations) parallelization (layout and pFNS)
  • Future NFSv4.2 and FedFS (Global namespace; IESG has approved Dec 2012)
  • NFSv4.1 failed to talk to the applications and customers and ask what they needed. We did that for NFSv4.2
  • Selecting the application for NFSv4.1, planning, server and client availability. High level overview
  • Selecting the parts: 1 – NFSv4.1 compliant server (Files, blocks or objects?), 2-compliant client. The rise of the embedded client (Oracle, VMware). 3 – Auxiliary tools (Kerberos, DNS, NTP, LDAP). 4 – If you can, use NFS v4.1 over NFSv4.
  • If you’re implementing something today, skip NFS v4 and go straight to NFS v4.1
  • First task: select an application: Home directories, HPC applications.
  • Don’t select: Oracle (use dNFS built in), VMware and other virtualization tools (NFSv3). Oddball apps that expect to be able to internally manage NFSv3 “maps”. Any application that required UDP, since v4.1 doesn’t support anything but TCP.
  • NSFv4 stateful clients. Gives client independence (client has state). Allows delegation and caching. No automounter required, simplified locking
  • Why? Compute nodes work best with local data, NFSv4 eliminates the need for local storage, exposes more of the backed storage functionality (hints), removes stale locks (major source of NFSv3 irritation)
  • NFSv4.1 Delegations. Server delegates certain responsabilities to the client (directory and file, caching). Read and write delegation. Allows clients to locally service operations (open, close, lock, etc.)
  • NFSv4.1 Sessions. In v3, server never knows if client got the reply message. In v4.1, sessions introduced.
  • Sessions: Major protocol infrastructure change. Exactly once semantics (EOS), bounded size of reply cache. Unlimited parallelism. Maintains server’s state relative to the connections belonging to a client.
  • Use delegation and caching transparently; client and server provide transparency. Session lock clean up automatically.
  • NFSv4 Compound operations – NFSv3 protocol can be “chatty”, unsuitable for WANs with poor latency. Typical NFSv3: open, read & close a file. Compounds many operations into one to reduce wire time and simple error recovery.
  • GETATTR is the bad boy. We spent 10 years with the Linux client to get rid of many of the GETATTR (26% of SPECsfs2008).
  • NFSv4 Namespace. Uniform and “infinite” namespace. Moving from user/home directories to datacenter and corporate use. Meets demand for “large scale” protocol. Unicode support for UTF-8 codepoints. No automounter required (simplifies administration). Pseudo-file system constructed by the server.
  • Looking at NFSv4 Namespace example. Consider the flexibility of pseudo-filesystems to permit easier migration.
  • NFSv4 I18N Directory and File Names. Uses UTF-8, check filenames for compatibility, review filenames for compatibility. Review existing NFSv3 names to ensure they are 7-bit ASCII clean.
  • NFSv4 Security. Strong security framework. ACLs for security and Windows compatibility. Security with Kerberos. NFSv4 can be implemented without Kerberos security, but not advisable.
  • Implementing without Kerberos (no security is a last resort!). NFSv4 represents users/groups as strings (NFSv3 used 32-bit integers, UID/GID). Requires UID/GID to be converted to all numeric strings.
  • Implementing with Kerberos. Find a security expert. Consider using Windows AD Server.
  • NFSv4 Security. Firewalls. NFSv4 ha no auxiliary protocols. Uses port 2049 with TCP only. Just open that port.
  • NFSv4 Layouts. Files, objects and block layouts. Flexibility for storage that underpins it. Location transparent. Layouts available from various vendors.
  • pNFS. Can aggregate bandwidth. Modern approach, relieves issues associated with point-to-point connections.
  • pNFS Filesystem implications.
  • pNFS terminology. Important callback mechanism to provide information about the resource.
  • pNFS: Commercial server implementations. NetApp has it. Panasas is the room as well. Can’t talk about other vendors…
  • Going very fast through a number of slides on pNFS: NFS client mount, client to MDS, MDS Layout to NFS client, pNFS client to DEVICEINFO from MDS,
  • In summary: Go adopt NFS 4.1, it’s the greatest thing since sliced bread, skip NFS 4.0
  • List of papers and references. RFCs: 1813 (NFSv3), 3530 (NFSv4), 5661 (NFSv4.1), 5663 (NFSv4.1 block layout), 5664 (NFSv4.1 object layout)

pNFS Directions / NFSv4 Agility
Adam Emerson, CohortFS, LLC

  • Stimulate discussion about agility as a guiding vision for future protocol evaluation
  • NFSv4: A standard file acces/storage protocol, that is agile
  • Incremental advances shouldn’t require a new access protocol. Capture more value from the engineering already done. Retain broad applicability, yet adapt quickly to new challenges/opportunities
  • NFSv4 has delivered (over 10+ years of effort) on a set of features designers had long aspired to: atomicity, consistency, integration, referrals, single namespaces
  • NFSv4 has sometimes been faulted for delivering slowly and imperfect on some key promises: flexible and easy wire security , capable and interoperable ACLs, RDMA acceleration
  • NFSv4 has a set of Interesting optional features not widely implemented: named attributes, write delegations, directory delegations, security state verifier, retention policy
  • Related discussion in the NFSv4 Community (IETF): The minor version/extension debate: de-serializing independent, potentially parallel extension efforts, fixing defect in prior protocol revisions, rationalizing past and future extension mechanisms
  • Related discussion in the NFSv4 Community (IETF): Extensions drafts leave my options open, but prescribes: process to support development of new features proposals in parallel, capability negotiation, experimentation
  • Embracing agility: Noveck formulation is subtle: rooted in NFS and WG, future depends on participants, can encompass but perhaps does not call out for an agile future.
  • Capability negotiation and experimental codepoint ranges strongly support agility. What we really want is a model that encourages movement of features from private experimentation to shared experimentation to standardization.
  • Efforts promoting agility: user-mode (and open source) NFSv4 servers (Ganesha, others?) and clients (CITI Windows NFSv4.1 client, library client implementations)
  • Some of the people in the original CITI team now working with us and are continuing to work on it
  • library client implementations: Allow novel semantics and features like pre-seeding of files, HPC workloads, etc.
  • NFSv4 Protocol Concepts promoting agility: Not just new RPCs and union types.
  • Compound: Grouping operations with context operations. Context evolves with operations and inflects the operations. It could be pushed further…
  • Named Attributes: Support elaboration of conventions and even features above the protocol, with minimal effort and coordination. Subfiles, proplists. Namespace issues: System/user/other, non-atomicity, not inlined.
  • Layout: Powerful structuring concept carrying simplified transaction pattern. Typed, Operations carry opaque data nearly everywhere, application to data striping compelling.
  • Futures/experimental work – some of them are ridicuolous and I apologize in advance
  • pNFS striping flexibility/flexible files (Halevy). Per-file striping and specific parity applications to file layout. OSDv2 layout, presented at IETF 87.
  • pNFS metastripe (Eisler, further WG drafts). Scale-out metadata and parallel operations for NFSv4. Generalizing parallel access concept of NFSv4 for metadata. Built on layout and attribute hints. CohortFS prototyping metastripe on a parallel version of the Ceph file system. NFSv4 missing a per-file redirect, so this has file redirection hints.
  • End-to-end Data Integrity (Lever/IBM). Add end-to-end data integrity primitives (NFSv4.2). Build on new READ_PLUS and WRITE ops. Potentially high value for many applications.
  • pNFS Placement Layouts (CohortFS). Design for algorithmic placement in pNFS layout extension. OSD selection and placement computed by a function returned at GETDEVICEINFO. Client execution of placement codes, complex parity, volumes, etc.
  • Replication Layouts (CohortFS). Client based replication with integrity. Synchronous wide-area replication. Built on Layout.
  • Client Encryption (CohortFS). Relying on named attribute extension only, could use atomicity. Hopefully combined with end-to-end integrity being worked on
  • Cache consistency. POSIX/non-CTO recently proposed (eg, Eshel/IBM). Potentially, more generality. Eg, flexible client cache consistency models in NFSv4. Add value to existing client caching like CacheFS.
  • New participants. You? The future is in the participants…
Viewing all 155 articles
Browse latest View live