The Debate – Why NFS vs Block Access for OS/Applications?
I’m going to see if I can use my blog to generate a discussion. First and foremost, I don’t want this to be a competitive “ours is better than yours”. So let’s keep it civil!!
Specifically I’m trying to wrap my head around why someone would choose to go NFS instead of doing block based connectivity for things like VSPhere? For instance, NetApp supports NFS and block based FC connectivity. EMC typically uses CX behind their NS gateway devices. Most Storage companies (Xiotech included) offer a NAS solution – either native or gateway.
So, why would I choose to run for instance VMware Vsphere (as an example) on NFS when I could just as easily run it without ? Is it that the file system used for that particular companies NAS solution offers something that their block based solution can’t? ie) thin provisioning, Native DeDupe, Replication etc. Is it used more as some sort of caching mechanism and gives better performance? Is it more fundamental and it’s more of a connectivity choice (IP vs FC vs NFS Mount)?
Please feel free to post your responses.
Thanks,
StorageTexan's Blog 
It’s probably a cost issue. Everyone has IP. FC is expensive.
Oh, and ours IS better than yours. 🙂
but i’ve i’ve already made the investment in FC (the gateway device uses it to connect to the array is my assumption) then the cost is an HBA right? is the cost savings of an HBA enough to justify?
Performance benchmarks show that NFS vs iSCSI are quite similar.
It’s a matter of ease of development to “V-Mote” something between systems. If you use Block IO you need to create concurrent access from different systems, in other words: clustering. With NFS you just share a file.
if i’m using VSphere – doesn’t that already do that for me via HA?
Fundementally – i think it just has to do with the array missing some sort of feature that the gateway or NAS device can do?
One thing for me is that NFS eliminates the lun locking that takes place anytime a metadata update takes place (snapsot, power on/off, thin provision disk growth, etc). Since NFS is just a bunch of files only the file is locked, not the entire lun with multiple vmdk disks. Another for NetApp is dedupe savings that are seen immediately to the VI Admin instead of only being reclaimed at the array level for block based storage. Also I can run many more vms on a single NFS export than on a single lun. The one real detriment to NFS versus block is the tools for measuring disk performance. You’ll have to use vscsistats to gather perf data whereas with block level storage esxtop has a wealth of different counters to measure performance and latency.
Ironically I don’t think NFS vs VMFS (FC, FCoE, iSCSI) is an all or nothing discussion. Instead I believe a unified storage platform which offers all protocols is the best fit for VMware as he hypervisor is also natively multiprotocol.
At the end of the day VMs are files and the fiels must be accessed by multiple hosts. So how can one achieve this access…
To date VMFS is probably the most successful clustered file system released. It allows LUNs, which natively cannot provide concurrent access by multiple hosts, to be shared. It provides this functionality on any array and any type of drive. It is a true storage abstraction layer.
NFS is a network file system. Storage on the network. Simple and easy.
VMFS requires file locking and SCSI reservations to be handled by the host. It also runs into LUN related issues like queue depths (found in traditional arrays).
NFS has file locking but it is handled by the NFS server/array. LUN queues also do not occur between the host and storage.
At this point it should seem like VMFS and NFS are very similar, and that is correct.
I’d like to suggest that VMs can typically be classified into one of two groups which are infrastructure VMs and high performance VMs. The two groups have an 80/20 in terms of number of VMs.
Infrastructure VMs are addressed in consolidation efforts and use shared datastores.
High performance, business critical VMs can be more demanding and are stored in isolated datastores (one which only stores the data for a single VM).
With shared datastores VMFS can hit artificial performance scaling limits related to shallow LUN queues and can hit max cluster limits due to file locks and SCSI reservations. The later will be addressed in the next update of vSphere.
With shared datastores NFS really shines as the array manages the queues, locks, etc… So performance and cluster scaling is exceptional.
With high performance VMs VMFS and NFS are on par as long as the link speed is not the bottleneck (ala 1GbE with NFS). However, some applications require a LUN in order to function or receive support.
If one wants to leverage storage virtualization one needs to understand that with VMFS the virtualization happens at the LUN level and is not visible within vCenter. With NFS, storage virtualization is at the VM level and is observable within vCenter. As an example when one deduplicates a VMFS and a NFS datastore, the savings are identical; however the VMFS datastore is unchanged while the NFS datastore returns capacity to use for additional VMs.
I guarantee you if you have a large number of VMs to address (say 500 or more) leveraging both protocols is the best.
VMDKs on VMFS have more options… thin, thick, eager-zero thick. So the question is when to use, why use, and what do I have to do to change the mode? With NFS VMDK types are controlled by the NFS server/array. With NetApp all VMDKs are thin and provide support for cluster services like FT (w/o being eager zero thick).
Its late, and I’m not sure I’ve been as clear as I’d like in this discussion; however, my view is a multiprotocol storage array is the best fit for a multiprotocol hypervisor.
If you only have VMFS, everything will work great and you may have to manage a few more datastores for the consolidated VMs. If you only have NFS, then you will run into some grey areas around LUN only solutions.
(note this last point isn’t really an issue as NFS is Ethernet, and Ethernet also simultaneously offers iSCSI and FCoE – so there is not limit).
Tommy, great question and Vaughn very good response. My opinion is the answer often comes down to what the storage manager of the moment is the most comfortable with. If there is a strong fibre pedigree they go fibre, if there is more of an IP based pedigree they go either iSCSI or NFS.
In smaller installations in the SMB world it is a matter of just get the job done. iSCSI often wins here and systems like Data Robotics and Starwind are hitting that price point squarely. See: http://bit.ly/VM-SMB
In larger environments NAS is taking hold but I have concerns about its ability to scale in that plus 500 VM category. Not because of NFS but because you start to really crush the NAS head at this point. See: http://bit.ly/NAS-Bottleneck
One of the key things about fibre is despite is complexity (which I think is blown out of proportion) is its flexibility. The capability of NPIV and some the basic QoS capabilities that some of the switch manufacturers are providing are critical in enabling truly large deployments. See: http://bit.ly/NPIV-ServerVirt
In short, flexibility is the key, and a mixture of protocols is likely. But understanding what a virtual environment with its highly random I/O nature is going to do to the entire storage I/O chain is critical as well.
What if you could have it all?
What if you could have an array that NAS doesn’t bottleneck, and that also does great with FC and iSCSI?
I’m probably biased since I’m with NetApp but that’s the message in a nutshell.
You’re not forced to choose, and there are some truly gigantic VMware workloads running on NetApp gear regardless of protocol.
The NFS inconveniences first:
Can’t do RDMs obviously, and there are some key pieces that require raw LUN access. So, you may end up with a percentage of VMs that need RDMs.
You obviously also need a good network infrastructure (holds true for iSCSI as well). 10Gig preferable.
NFS conveniences:
* Brain-numbingly-simple administration…
* Many more VMs per store
* The ability to do cool stuff like VM cloning more easily than with VMFS
* More granular control of array-level snapshots
* No need to worry about zoning, initiators, multipathing, ALUA etc etc.
* Space savings with deduplication immediately available to the datastore
* Easily grow AND shrink datastores! No need for extents!
* Great performance in most cases
But, ultimately, we just prefer to not force people one way or another, so we focus on providing a device that can accommodate all the protocols.
Choice is a good thing.
D
If you’re of any size less than “have a fulltime storage/backups admin on staff” fibre channel is a nightmarishly complex world to step into. Then once you get it working you’re frozen in place as any minor change in hardware, operating system, firmware, switches, etc triggers an exponential amount of work cross-verifying various vendors “support matrixes” (aka the “if we make this complicated enough we can always find a reason not to support you” matrix). Netapp/nfs will never tell you to check the ios version on your switch before they work with you. They’ll never send you on a two day adventure to try and get a dos floppy image and schedule a downtime window to do an hba firmware update. The engineer in me wants to think the FC solution has fewer layers and is therefore simpler/better. The sysadmin-with-a-day-job in me knows I add days if not weeks to any project that goes near the FC network.
Now, the new paradigm from IBM with EX5 servers: why not internal disks?
New servers from IBM can hold 16 disks and you can use Flashpacks or SSDs to get more IOPS than High End arrays with standard disks, and this is a more native or with less layers and complexity than standard FC solutions
Tommy, first of all, thanks for writing the blog. Second, thanks to all of you for responding with great comments (the last comment by Servadac excepted; complete pitch, not nearly as worthy as the other comments).
True to the diversity of the comments, there are some inherent assumptions floating around which need some clarification.
1) “SAN is hard”. Sorry, let’s be precise; SOME SANs are hard. Others are not, to the point of being no harder to administer than servers. The effort Xiotech has put forth in ICON Manager for ISE and Virtual View is testament to that point – SANs can actually be easy to use, and I’m sorry others have suffered through complex SANs. Those folks should have, truth be told, done a bit more homework before investing in SAN technology. But that’s water over the dam.
2) “NFS can do more than LUNs”. Sorry, again not accurate. There have been SAN filesystems for over a decade, starting back in 1998 with the original SAN filesystem, helped by one of the first TWGs I sat in – the old Filesystems Working Group. So, yes indeed LUNs can be shared, without SCSI reservations. There is next to nothing you can accomplish with NFS that cannot be accomplished with LUNs, small-grained sharing included, down to the extent level.
3) “VMFS uses SCSI reservations for locking”. True enough, but let’s not tar the entire virtual world with the same brush. There are other virtual server methods that do not use SCSI reservations while updating metadata. All the world is not VMFS.
4) “You can do more cool stuff with NFS”. Again, you need to do your homework and look at intelligent approaches such as those put forth by ICON Manager for ISE and Virtual View. Growing VMFS dynamically and datastores is a piece of cake, once you use Web Services and RESTful techniques to do so.
None of this is intended as a slam against NFS. I believe both file-level and block-level protocols are useful, each have their best use cases and each have their drawbacks relative to the other. The nice thing here is that the users themselves can choose which protocol(s) to use, and implement them as they see fit. There is no Hobson’s choice here.
But remember this – at the end of the day, the data has to go somewhere, and that somewhere is blocks on disk. Those who provide the most intelligence, the most performance, the most scalability w.r.t. blocks on disk are going to have optimal solutions, whether or not file-level protocols are layered on top, at the user’s control.
This is where great file-level systems and techniques on top of great block-level storage blades come in. Users get the best of both worlds. Imagine the world’s best filer head on top of the world’s best block-level device. Now, that’s worth writing home about 🙂
Thanks again Tommy for blogging!
@George – I concur with your sentiments on iSCSI. We often see iSCSI with customers deploying unto a few hundred VMs. It’s easy and very robust.
I don’t know what NAS heads you are referring to when you state that you’ve seen them become the bottleneck. Every array has a performance range, and I’d suggest if a NAS array was getting ‘crushed’ that any protocol on the same hardware would suffer the same fate.
In regards to FC, I believe it be covered well here. The biggest drawback to FC is array implementations which have per LUN IO queues. I challenge you assessment of NPIV, not wether it works, but wether it is worth the investment. Again, we don’t see large adoption of NPIV.
We agree in summary, a native multi-protocol array is the best fir for VMWare.
@Hector – In order to enable the high availability features in VMware (HA, DRS, FT, etc…) one must deploy on shared storage. The storage in the EX5 cannot provide this capability.
@Rob – I disagree with your comments around LUNs. NFS provides storage virtualization at the VM level. With LUNs, and storage capabilities are limited to the datstores. Please check out our demos on YouTube (http://www.youtube.com/user/NetAppTube) for more details around technologies like instant VM cloning, deduce differences between SAN and NAS usage, etc…
As this discussion is specific to VMware I don’t understand your comment of “All the world is not VMFS.” With VMware we can only leverage VMFS & NFS for datastores.
Everyone has IP sure (and most IP networks are pretty busy as they are)
Now you want to introduce another protocol on top of that.
And people wonder why they run into performance issues.
NFS is good for certain applications and environments just see what your currents workloads are now , before you introduce something else.
@Vaughn, appreciate your comments. I fully understand how NFS fits into a VM environment, thanks. Nothing against that at all. However, you need to study our approach to using LUNs and integrating directly into VMFS and the VMs themselves via Web Services/REST, and look in turn at our Youtube videos on VMs. We provide virtualization services as well without NFS. Again, nothing against NFS, you can use it if you like, but the implicit statement that LUNs are not useful is just not the case, _if_ you use LUNs intelligently and understand filesystems via WS/REST.
Pingback: The journey to the Unified Storage Platform « StorageTexan's Blog
I think the real issue is when you deploy VDI. In server deployments you have 10-15-20 VMs per LUN. In a VDI deployment you have linked clones and you may have 250 VMs accessing that linked clone. VMFS is quite frankly terrible for VDI because of SCSI reservations. The question isn’t NFS vs. Block and all the points above can be argued. What cannot be argued is that in a production VDI environment where you have more than 75 desktop VMs, you will have performance issues utilizing VMFS. NFS offloads and handles this, eliminating the SCSI reservation issue.
In a typical server environment, pick whatever you are comfortable with…NFS over VMFS or VMFS with iSCSI, FC or FCoE. The issue really comes to light in VDI environments.
All the manufacturers have their marketing spin and solution, but the issue isn’t really block vs. NFS and they agree…it’s NFS vs. VMFS.
I’d be interested in hearing from people who have deployed production View environments and not just a 10 pack.