Hello! Welcome to my first post here at Storage Monkeys. Some of you may know me from this other little site that I also maintain, and I may refer you there occasionally for more in-depth information. I do hope that I am able to share some useful information here with you. I’ll try my best not to focus on virtualization-related topics here, since that’s not the focus of this community, but I hope that you’ll indulge me every now and then.
For my debut, I’m going to tackle what will probably be a sensitive topic for some readers: VMware over NFS. All across the Internet, I run into article after article after article that sings the praises of NFS for VMware. Consider some of the following examples:
That first link looks to be mostly a reprint of this blog post by Nick Triantos. Now, Nick is a solid storage engineer; there is no question in my mind that he knows Fibre Channel, iSCSI, and NFS inside out. Nick is certainly someone who is more than qualified to speak to the validity of using NFS for VMware storage. But…
I am going to have to disagree with some of the statements that are being propagated about NFS for VMware storage. Is NFS for VMware environments a valid choice? Yes, absolutely. However, there are some myths about NFS for VMware storage that need to be addressed.
- Myth #1: All VMDKs are thin provisioned by default with NFS, and that saves significant amounts of storage space. That’s true—to a certain point. What I pointed out back in March of this year, though, was that these VMDKs are only thin provisioned at the beginning. What does that mean? Perform a Storage VMotion operation to move those VMDKs from one NFS datastore to a different NFS datastore, and the VMDK will inflate to become a thick provisioned file. Clone another VM from the VM with the thin provisioned disks, and you’ll find that the cloned VM has thick VMDKs. That’s right—the only way to get those thin provisioned VMDKs is to create all your VMs from scratch. Is that what you really want to do?
- Myth #2: NFS uses Ethernet as the transport, so I can just add more network connections to scale the bandwidth. Well, not exactly. Yes, it is possible to add Ethernet links and get more bandwidth. However, you’ll have to deal with a whole list of issues: link aggregation/802.3ad, physical switch redundancy (which is further complicated when you want to use link aggregation/802.3ad), multiple IP addresses on the NFS server(s), multiple VMkernel ports on the VMware ESX servers, and multiple IP subnets. Let’s just say that scaling NFS bandwidth with VMware ESX isn’t as straightforward as it may seem. This article I wrote back in July may help shed some light on the particulars that are involved when it comes to ESX and NIC utilization.
- Myth #3: Performance over NFS is better than Fibre Channel or iSCSI. Based on this technical report by NetApp—no doubt one of the biggest proponents of NFS for VMware storage—NFS performance trails Fibre Channel, although by less than 10%. So, performance is comparable in almost all cases, and the difference is small enough not to be noticeable. The numbers do not, however, indicate that NFS is better than Fibre Channel. You can read my views on this storage protocol comparison at my site. By the way, also check the comments; you’ll see that the results in the technical report were independently verified by VMware as well. Based on this information, someone could certainly say that NFS performance is perfectly reasonable, but one could not say that NFS performance is better than Fibre Channel.
Now, one might look at this article and say, “Scott, you hate NFS!” No, actually, I like using NFS for VMware Infrastructure implementations, and here’s why:
- Provisioning is a breeze. It’s dead simple to add NFS datastores.
- You can easily (depending upon the storage platform) increase or decrease the size of NFS datastores. Try decreasing the size of a VMFS datastore and see what happens!
- You don’t need to deal with the complexity of a Fibre Channel fabric, switches, WWNs, zones, ISLs, and all that. Now, there is some complexity involved (see Myth #2 above), but it’s generally easier than Fibre Channel. Unless you’re a Fibre Channel expert, of course…
So there are some tangible benefits to using NFS for VMware Infrastructure. But let’s be real about this, and not try to gloss over technical details. While NFS has some real advantages, it also has some real disadvantages, and organizations choosing a storage protocol need to understand both sides of the coin.
8 responses so far ↓
1 Brian Knutsson // Aug 19, 2008 at 12:50 pm
Hi,
You are so right.
I have had some long discussions with NetAPP technicians about them not telling customers about the disadvantages.
One complaint that I have is that a task like deploying a template, using VC and not NetAPP’s tools, takes longer than on VMFS, because of the lower (Single Process) bandwidth 1Gbit Ethernet vs. 4 Gbit fiber, that we often see.
And the bandwidth that customers often focus on is the point-to-point disk transfer test.
/Brian
2 Nick Triantos // Aug 22, 2008 at 5:57 pm
Scott,
I had no idea you’re a blog double dipper
Just run into this blog so know I have to bookmark it.
A few comments on the article:
Technically, I see no advantage of thin provisioning VMDKs on NFS given that the same thing can be done with VMFS using vmkfstools. Thus I would not use this argument to convince someone to consider NFS. .
It is true that NFS requires some design thought and considerations prior to deployment. In fact, I believe that NFS requires more upfront planning than FC, however, the operational benefits NFS offers outweigh the ones offered by FC by a mile as far as ESX goes.
FC vs NFS is a kinda like comparing Price vs Value.
Do you buy an inexpensive car that requires more money to run and maintain or do you buy a more expensive car that requires less operational costs and maintenance?
The fact of the matter is that while NFS requires more upfront planning than FC, from an operational standpoint and day to day tasks such as (growing/shirking datastores, backup/recovery) NFS requires fewer administrative steps and intervention than FC.
What’s i;ve found out over the years is that decisions made about protocol deployments are based more on political issues and stale viewpoints rather than anything else.
I have a rhetorical question…Since vmdks are files, why are we using block protocols to manage them? Wasn’t NFS created for this reason?
3 slowe // Aug 24, 2008 at 3:00 pm
Nick,
Thanks for finding me over here! You never know where I may pop up next….
I’m glad to hear you say that you wouldn’t consider thin provisioning a benefit of NFS. Unfortunately, that messaging hasn’t made it out to all the NetApp folks yet.
Clearly, customers need to evaluate the long-term benefits and costs of their storage protocol choices, and you are correct that in some cases NFS may prove better than FC in that regard. My primary purpose in this article was not to bash NFS for VMware, but rather to bring some balanced views to the discussion. It sounds like we both agree that while NFS does offer some operational benefits, there are cases where Fibre Channel–and perhaps even iSCSI, the red-headed stepchild of this discussion–may prove to be a better choice for some organizations, depending upon their needs.
Keep up the good work, Nick!
4 Nick Triantos // Aug 25, 2008 at 1:02 pm
Incidently, Gartner released a 4pg Report on July 16 titled “Choosing Networked Attached Storage for VMware: What You should know” by Pushan Rinnen.
While I can not make the entire report available due to Gartner’s restrictions, we can quote statements of the report.
Vaughn published a blog entry about this on Friday with the highlights of the report:
http://blogs.netapp.com/virtualstorageguy/
5 Steve Brant // Sep 3, 2008 at 6:44 pm
Ok guys what I’m going to say may not fit here but I would like to get some help if I could. I would like to ask how does one properly test these storage solutions? Of course the first test would be testing disk performance and than after that testing the network performance. Right now we use EMC fiber channel we also use a open source soultion called openfiler for iscsi. So I ask how do you storage guys test disk and network performance what tools do you use for this. As I stated we use openfiler for iscsi I want to be able to say ok we have 6 15k sas drives in the box these drives should give x amount of throughput than I can say now lets test the nework and see where the bottle neck is
6 slowe // Sep 4, 2008 at 7:09 pm
Steve,
Others may be interested or able to provide more detailed information, so I’ll just touch on this at a high level.
First, testing storage protocol performance is tricky; there are many details to consider. If you ask 10 different storage vendors how to test their arrays, you’ll probably get 10 different answers. So keep that in mind as you proceed.
With that in mind, the general tool of choice for testing storage performance is IOmeter. This will test the storage protocol (FC, NFS, iSCSI) as well as the back-end disks in the array.
If you are interested in just testing network performance (i.e., you’re using NFS or iSCSI), then I’d suggest having a look at the open source PCATTCP tool. There are ports for various operating systems, including Windows. Google knows where to find it.
I hope this helps!
7 Glenn Dekhayser // Sep 23, 2008 at 9:47 am
Scott:
Thanks so much for pointing out the thin-provisioning myth on NFS. You are correct that Netapp has done an insufficient job at being clear about this; and I was surprised (and dismayed) when setting up NFS-based VMDKs that they were coming up thick. The problem is that the netapp SE’s trumpet this feature way too loudly, and never explain the details.
8 slowe // Sep 23, 2008 at 5:46 pm
Glenn, thanks for your comment. I’m hopeful that thought leaders within NetApp will bring some sensibility to the hysteria over NFS. Like any storage technology, NFS has both advantages and disadvantages. It’s important to look closer and make sure that it’s the right fit for your needs.
Thanks again for reading and commenting!
Leave a Comment