A few weeks ago I examined FCoE in the context of it’s description as an “I/O virtualization” technology in my discussion of FCoE versus MR-IOV. (Despite protestations otherwise, I’ll continue to maintain that FCoE is not an I/O virtualization technology.)

Since that time, I read a few more posts about FCoE in various spots on the Internet:

Is FCoE a viable option for SMB/Commercial?
Is the FCoE Starting Pistol Aimed at iSCSI?
Reality Check: The FCoE Forecast

Tonight, after reading a blog post by Dave Graham regarding FCoE vs. InfiniBand, I started thinking about FCoE again, and I came up with a question I want to ask. I’m not a storage expert, and I don’t have decades of experience in the storage arena like many others that write about storage. The question I’m about to ask, then, may just be the uneducated ranting of a fool. If so, you’re welcome to enlighten me in the comments.

Here’s the question: how is FCoE any better than iSCSI?

Now, before your head explodes with unbelief at the horror that anyone could ask that question, let me frame that question with more questions. Note that these are mostly rhetorical questions, but if the underlying concepts behind these questions are incorrect you are, again, welcome to enlighten me in the comments. Here are the framing questions that support my primary question above:

1. FCoE is always mentioned hand-in-hand with 10 Gigabit Ethernet. Can’t iSCSI take advantage of 10 Gigabit Ethernet too?
2. FCoE is almost always mentioned in the same breath as “low latency” and “lossless operation”. Truth be told, it’s not FCoE that’s providing that functionality, it’s CEE (Converged Enhanced Ethernet). Does that mean that FCoE without CEE would suffer from the same “problems” as iSCSI?
3. If iSCSI was running on a CEE network, wouldn’t it exhibit predictable latencies and lossless operation like FCoE?

These questions—and the thoughts behind them—are not necessarily mine alone. In October Stephen Foskett wrote:

And iSCSI isn’t done evolving. Folks like Mellor, Chuck Hollis, and Storagebod are lauding FCoE at 10 gigabit speeds, but seem to forget that iSCSI can run at that speed, too. It can also run on the same CNAs and enterprise switches.

If those Converged Network Adapters (CNAs) and enterprise switches are creating the lossless CEE fabric, then iSCSI benefits as much as FCoE. Dante Malagrino agrees on the Data Center Networks blog:

I certainly agree that Data Center Ethernet (if properly implemented) is the real key differentiator and enabler of Unified Fabric, whether we like to build it with iSCSI or FCoE.

Seems to me that all the things that FCoE has going for it—10 Gigabit speeds, lossless operation, low latency operation—are equally applicable to iSCSI as they are functions of CEE and not FCoE itself. So, with that in mind, I bring myself again to the main question: how is FCoE any better than iSCSI?

You might read this and say, “Oh, he’s an FCoE hater and an iSCSI lover.” No, not really; it just doesn’t make any sense to me how FCoE is touted as so great and iSCSI is treated like the red-headed stepchild. I have nothing against FCoE—just don’t say that it’s an enabler of the Unified Fabric. (It’s not. CEE is what enables the Unified Fabric.) Don’t say that it’s an I/O virtualization technology. (It’s not. It’s just a new transport option for Fibre Channel Protocol.) Don’t say that it will solve world hunger or bring about world peace. (It won’t, although I wish it would!)

Of course, despite all these facts, it’s looking more and more like FCoE is VHS and iSCSI is Betamax. Sometimes the “best” technology doesn’t always win…

FalconStor today announced a new file-interface deduplication system and enhancements to its Virtual Tape Library (VTL) software. FalconStor’s VTL product is resold by a number of other vendors, including EMC, Pillar, and SUN.

The new functionality being announced by FalconStor today is actually two-fold:

1. First, FalconStor will now offer NFS and CIFS access to the deduplication repository through its file interface deduplication system (FIDS), enabling NAS-based disk-to-disk (D2D) backup solutions to integrate seamlessly with the FalconStor data repository. D2D backup solutions that are designed to write to NFS or CIFS can now dump backup data to a FalconStor appliance and take advantage of the deduplication functionality.
2. Second, version 5.1 of the FalconStor VTL software adds support for 8Gb Fibre Channel and 10Gb Ethernet, enabling faster backups via these high-speed technologies. For backup solutions that utilize a “traditional” VTL, FalconStor can now offer higher throughput in those environments.

Overall, FalconStor’s system continues to retain or enhance features like many-to-one data replication, global deduplication across multiple appliances, and availability as a virtual appliance.

More information should be available on FalconStor’s web site, although at the time of publication of this article their site had not yet been updated.

(Author’s note: This was also cross-published to my own site as well.)

Last week I provided a list of virtualization-related items that had made their way into my Inbox in some form or another; today I’ll share storage-related items with you in Storage Short Take #4!

• Stephen Foskett has a nice round-up of some of the storage-related changes available to users in VMWare ESX 3.5 Update 3. Of particular note to many users is the VMDK Recovery Tool. Oh, and be sure to have a look at Stephen’s list of top 10 innovative enterprise storage hardware products. He invited me to participate in creating the list, but I just didn’t feel like I would have been able to contribute anything genuinely useful. Storage is an area I enjoy, but I don’t think I’ve risen to the ranks of “storage guru” just yet.
• And in the area of top 10 storage lists, Marc Farley shares his list of top 10 network storage innovations as well. I’ll have to be honest—I recognize more of these products than I did ones on Stephen’s list.
• Robin Harris of StorageMojo provides some great insight into the details behind EMC’s Atmos cloud storage product. I won’t even begin to try to summarize some of that information here as it’s way past my level, but it’s fascinating reading. What’s also interesting to me is that EMC chose to require users to use an API to really interact with the Atmos (more detailed reasons why provided here by Chad Sakac), while child company VMware is seeking to prevent users from having to modify their applications to take advantage of “the cloud.” I don’t necessarily see a conflict between these two approaches as they are seeking to address two different issues. Actually, I see similarities between EMC’s Atmos approach and Microsoft’s Azure approach, both which require retooling applications to take advantage of the new technology.
• Speaking of Chad, here’s a recent post on how to add storage to the Celerra Virtual Appliance.
• Andy Leonard took up a concern about NetApp deduplication and volume size limits a while back. The basic gist of the concern is that in its current incarnation, NetApp deduplication limits the size of the volume that can be deduplicated. If the size of the volume ever exceeds that limit, it can’t be deduplicated—even if the volume is subsequently resized back within the limit. With that in mind, users must actively track deduplication space savings so that, in the event they need to undo the deduplication, they don’t inadvertently lose the ability to deduplicate because they exceeded the size limit. Although Larry Freeman aka “Dr Dedupe” responded in the comments to Andy’s post, I don’t think that he actually addressed the problem Andy was trying to state. Although the logical data size can grow to 16TB within a deduplicated volume, you’ll still need to watch deduplication space savings if you think you might need to undo the deduplication for whatever reason. Otherwise, you could exceed the volume size limitations and lose the ability to deduplicate that volume.
• And while we are on the subject of NetApp, a blog post by Beth Pariseau from earlier in the year recently caught my attention; it was in regards to NetApp Snapshots in LUN environments. I’ve discussed a little bit of this before in my post about managing space requirements with LUNs. The basic question: how much additional space is recommended—or required—when using Snapshots and LUNs? Before the advent of Snapshot auto-delete and volume autogrow, the mantra from NetApp was “2x + delta”—two times the size of the LUN plus changes. With the addition of these features, deduplication, and additional thin provisioning functionality, NetApp has now moved their focus to “1x + Delta”—the size of the LUN plus space needed for changes. It’s not surprising to me that there is confusion in this area, as NetApp themselves has worked so hard to preach “2x + Delta” and now has to go back and change their message. Bottom line: You’re going to need additional space for storing Snapshots of your LUNs, and the real amount is determined by your change rate, how many Snapshots you will keep, and for how long you will keep them. 20% might be enough, or you might need 120%. It all depends upon your applications and your business needs.
• If you’re into Solaris ZFS, be sure to have a look at this NFS performance white paper by Sun. It provides some good details on recent changes to how NFS exports are implemented in conjunction with ZFS.

That’s all for now, but feel free to add your own thoughts on recent storage news items in the comments below. Thanks for reading!

One of my greatest frustrations as an end-user and as a researcher of storage technologies is the lack of quality commentary on storage technologies. This is an industry full of people who don’t actually work with storage technology as part of their normal job routines yet feel uniquely qualified to give opinions and lay claim to being a consultant or analyst. I don’t pretend to be an expert and I don’t feel passionate enough to tell others what type of technologies they should adopt. At best, I understand the environment I work in and I understand what has worked and not worked… in our environment… nothing more, nothing less. I am, however, wise enough to know when someone is simply pretending to understand storage technology.

The latest culprit is Henry Newman, a regular contributor to Enterprise Storage Forum and specifically his recent article, “Storage Virtualization: Good News for Consultants“. Newman may be a genius, but his lack of depth in understanding the benefits of storage virtualization makes him wholly unqualified to write about it. Newman doesn’t split hairs - from the first paragraph he makes it known that he does not care for storage virtualization. Although he is honest about his bias up front, he doesn’t demonstrate through the article that he understands storage virtualization at all.

For starters…

Before I start with the problems I see with storage virtualization, let me state up front that there are many environments where storage virtualization can help while reducing costs. These environments generally have low performance requirements and can often support applications with high latencies. For example, if you are virtualizing a Web application and you increase latency to storage by 20 milliseconds and reduce bandwidth by 20 percent, it will often make little difference when running over the internet, given the internet latency and the bandwidth between the storage and the user of the system over the internet. On the other hand, if you are running storage virtualization over a local high-speed, low-latency network, the increase in latency and reduction in performance could be catastrophic to the user community.

And…

The whole point of virtualization is to allow the administrator the ability to provision storage as needed. That often means allowing file systems to grow and shrink based on the availability of storage space and to allow the administrator to provision storage from a pool of LUNs not knowing where they are or sometimes even how big the LUNs are.

Newman reads like he only trusts monolithic storage frames. True that storage virtualization provides an extra layer (effecting both administration and performance), but this is mitigated by the extra services that storage virtualization provides. When administrating a virtual storage environment you need to be able to differentiate between the physical and virtual layers… if you can’t, you should probably find a different job. Any virtual storage product worth its salt will allow you to easily identify which physical LUN(s) a virtual LUN resides on.

With performance, it is true that a virtual storage appliance will impose some latency while addressing the storage devices being virtualized. This is due to having to recieve/analyze/pass all storage I/O through the appliances. Most storage virtualization solutions provide methods to overcome this by implementing caching solutions. Cache can be memory, SSD, or high speed disk. We have enabled virtualization in our fairly high transaction environment and have not seen performance degrade at all. Newman goes on…

Any type of storage migration today is complex work. There are vendors that claim ease of migration from one storage environment to another, and for many simpler environments migration is fairly easy, but even when it is easy it is not cheap. If you have a virtualized environment, you have another level of indirection between the file system and the storage and where the blocks of storage really are. The migration hardware and software must work together with the new virtualization environment to migrate your old environment to the new environment.

When dealing with migration, most virtual storage solutions require some third party method to transfer the virtual storage to tradition storage. Some vendors solve this by offering a method to proxy a LUN without destroying the LUN format. This allows for easy migration to or from virtual/traditional storage solutions. The speed of migration is usually dictated by how fast the source system can read and the target system can write. In this instance, Newman couldn’t be further from reality - without virtualization, data migration would be a nightmare and in our case, we would be locked into a single, proprietary vendor which would ramp up our storage costs dramatically. Which brings me to some key points about the benefits of storage virtualization:

1) Easy administration and lower storage costs: In a virtualized environment, storage is managed at the virtualization level and not by the storage itself. This allows for centralized administration and, depending on the storage virtualization vendor, opens the SAN to multiple storage hardware vendors. This has allowed us to retain and leverage our legacy storage hardware while adding new storage that had multiple vendors competing for the business. A conservative estimate is that our storage administration and costs have decreased by 40 percent.

2) Capacity planning: Storage Virtualization technologies give storage administrators an easy method for managing storage growth while efficiently allocating storage on the fly. Storage allocation requests in our environment have gone from days to fulfill to just minutes. We also have the ability to tier storage by application quickly which has also given us greater flexibility to manage our storage resources.

3) Disaster Recovery: Replication and recovery tasks can be managed at the storage virtualization level under a single pane of glass for the multiple types of storage hardware within the SAN.

Storage virtualization has clearly been a been a benefit in our demanding environment in both cost and time savings. Newman’s comments doesn’t reflect our experience at all and I have some serious doubts if he really understands -or has even used- the technology.

(Author’s note: I’ve cross-posted this article to my own site as well.)

I’ve had this link sitting in my “Articles To Read” list for quite some time, but—to be perfectly honest—I’ve been just too busy to really do anything about it. Now that a hectic few weeks has wrapped up and I have a small breather before the next hectic few weeks, I wanted to comment briefly on Doug Gourlay’s discussion of FCoE versus MR-IOV.

First, some background: For those that aren’t familiar, FCoE is Fibre Channel over Ethernet, a T11 standard for running Fibre Channel Protocol over Ethernet, specifically 10 Gigabit Ethernet. More information on FCoE is found here. MR-IOV is Multi-Root I/O Virtualization, a PCI SIG specification for using PCI Express (PCIe) to connect and share multiple devices. More information on MR-IOV can be found here. MR-IOV is a multi-server extension to Single-Root I/O Virtualization, or SR-IOV.

Like Doug, I’ll put in a disclaimer that I haven’t read the report to which he’s referring in his article, either. However, as an individual who has done some research on the topic of I/O virtualization, I will say that anyone who compares FCoE to MR-IOV is comparing apples to oranges. These two technologies, in my mind, are designed to address two different problems.

FCoE provides the ability to use a single physical transport—10 Gigabit Ethernet, in this case—for Fibre Channel Protocol (FCP) as well as TCP/IP, iSCSI, and other Ethernet-borne protocols. This allows for the creation of a unified fabric, a single physical transport that carries all the various kinds of traffic that Ethernet-based Local Area Networks (LANs) and Storage Area Networks (SANs) carry separately today. Via the IETF Converged Enhanced Ethernet (CEE) standard—adopted by Cisco as Data Center Ethernet^TM—FCoE will ultimately have the same low, predictable latency and error-free operation that FCP enjoys today. FCoE is not, however, designed or architected to do anything other than allow FCP to run over Ethernet. It’s not intended to be a server interconnect technology. (Unless I’m missing something?)

MR-IOV, on the other hand, is intended to play in the server interconnect field. Its purpose is not to allow FCP to run over Ethernet, or to allow FCoE, iSCSI, and other TCP/IP protocols share the same physical connections. MR-IOV’s purpose is to allow multiple servers to share PCIe-based devices, like a FC Host Bus Adapter (HBA), or an iSCSI HBA, a 10 Gigabit Ethernet network interface card (NIC), or a video capture card. MR-IOV is intended to provide I/O virtualization, regardless of what type of I/O that might be. As long as the I/O runs across a PCI Express bus, MR-IOV comes into play.

I’ve heard multiple people refer to FCoE as an I/O virtualization technology, but I just don’t agree. FCoE only applies to FCP over Ethernet. It doesn’t apply to iSCSI. It doesn’t apply to video traffic, or audio traffic, or HTTP traffic. It only applies to FCP over Ethernet. While I might allow that FCoE does allow for a form of virtualization, by virtualizing the physical transport beneath FCP, I would not call it I/O virtualization. Further, FCoE and MR-IOV are complementary. You could use MR-IOV to share a single Converged Network Adapter (CNA), which provides FCoE and 10 Gigabit Ethernet functionality, among multiple servers. In this situation, what’s providing the I/O virtualization: MR-IOV, which is allowing multiple servers to use a single I/O card, or the CNA, which is putting the traffic onto the converged fabric?

I’m probably missing something huge here, some vital piece of information that would make sense why FCoE and MR-IOV would be considered competitive standards/specifications. Without that information, though, it just doesn’t make any sense to me to compare these two different yet complementary technologies. Someone want to enlighten me?

UPDATE: I’ve corrected my use of “Data Center Ethernet” to Converged Enhanced Ethernet (CEE) when referring to the IETF standard. As correctly pointed out in the comments, Data Center Ethernet^TM is a Cisco trademarked term referring to their implementation of CEE.

You may have noticed that posts have slowed down in the last couple of weeks. The reason is that we are in the process of making a pretty dramatic update to the main site. We have engaged a team of professional developers and have settled on a new design for the site which will provide a number of enhancements including:

· Daily news commentary – brief snippets of linked news items with commentary
· Daily email of news and commentary updates every morning (don’t worry, you will not be spammed, only people who opt-in for the email will receive it)
· An online knowledgebase of vendor and analyst information
· Social networking enhancements - Further Twitter and LinkedIn integration. We are also looking at integration with other social networking sites

The downside of the upgrade is that we will have to remove the “Groups” section due to some technical and browser compatibility issues that we are in the process of resolving.

The new site is not only visually appealing, we believe it is far more functional and useful than the current site. Our original target date for launching the new site was November 1st… but this was delayed due to further changes we needed to make. Or our target now for the relaunch is now the first week in December.

(Author’s note: This was cross-published to my site as well.)

I’m sure that others have already picked this up, but I thought it was worth mentioning that NetApp has added deduplication functionality to their virtual tape library (VTL) product line. The official press release was published on October 28, 2008:

NetApp (NASDAQ: NTAP) today announced the availability of deduplication on NetApp® Virtual Tape Library (VTL) systems, enabling customers to lower the disk capacity required to backup any storage system, including EMC and HP, up to 95%¹. NetApp delivers unparalleled cost savings for traditional backup environments by reducing tape media usage and tape drive infrastructure and, now, through highly efficient usage of VTL disk storage through deduplication. With the addition of deduplication on VTL, NetApp offers the most complete deduplication portfolio, spanning backup, archive, and primary storage applications.

Per the press release, it also appears that VTL deduplication will be available to both new and existing customers at no additional charge.

Check out the official press release for all the details.

The economic downturn is starting to really hurt vendors. While Overland Storage is beginning to circle the bowl, Pillar Data Systems CEO, Mike Workman, announced on his blog yesterday that they too will be facing some dramatic changes:

Pillar is anticipating a tough economic climate will persist throughout 2009 and into 2010, so we have decided to batten down the hatches and lay off some of our employees.  These decisions are always among the toughest decisions a CEO has to make. But our jobs require we do what’s necessary to fulfill our duties to the company and its shareholders.

In my opinion, Pillar is an odd company. They have some really interesting technology in the way they provide an ILM solution by creating tiers of storage based on where the data is physically located on the disk. Unless you are required by management to tier storage out this way, it seems like a “nice to have” rather than a “need to have”.

The challenges keep mounting for tape library and (now) NAS vendor Overland Storage. Just a quarter after purchasing Snap Appliance from Adaptec, Overland Storage has been warned that it could be delisted from the NASDAQ. Overland Storage also announced yesterday that it had posted a first-quarter net loss of $6.9 million for it’s fiscal year first quarter (which ended September 30th). At 11:30am EST today, Overland Storage stock was down to .17 a share, down over 40% from yesterday. Overland Storage also noted that it is seeking $10 million “to continue to execute its strategy“.

Besides manufacturing a series of tape libraries and the Snap Server NAS, Overland Storage also produces a line of Virtual Tape Libraries, Tape Autoloaders and SAN Fibre Channel storage devices.

(Author’s note: This was cross-published on my own site as well.)

We ran into an interesting set of problems at work this week, all of them related to running virtual machines on VMware ESX over NetApp NFS. While we haven’t yet determined the root cause for all of the problem, we did uncover the root cause for one of the issues, and additional testing seems to indicate that one of the other problems may also be suffering from the same condition.

The problem seems to lie within some (now outdated) recommendations by NetApp for using NFS with VMware ESX. As late as June of this year, NetApp was recommending—via TR-3428, “NetApp and VMware Virtual Infrastructure 3 Storage Best Practices”—that users disable NFS locking on the storage system by setting the NFS.Lock.Disable to 1. Version 4.0 of TR-3428 was the version published in June of this year, and it still contained this recommendation.

As I understand it, this recommendation stemmed from a bug that existed in VMware ESX that caused long delays when removing VMware snapshots. As a workaround, NetApp recommended setting the NFS.Lock.Disable setting. Supposedly, VMware has fixed that bug with this patch, and the current version of TR-3428 (version 4.2, published in September 2008 and available from NetApp’s web site) no longer contains the recommendation to set NFS.Lock.Disable.

Unfortunately, customers who followed (then current) best practices may have set this value, and in so doing may have exposed themselves to a “split brain” scenario in which a VM is simultaneously registered and running on multiple VMware ESX hosts. Users may be particularly at risk if they also run VMware HA and have not protected themselves against isolation response (these articles are tagged ‘VMwareHA’ and will provide more information on isolation response). We’ve had one large customer be affected by this problem, and another large customer who is having problems that we believe are related to this same issue.

There are a couple of important things to note:

• This is not a NetApp problem, but rather the result of a recommendation from NetApp. At that time, it was believed that this setting was necessary.
• In most cases VMs affected by this split-brain scenario will get corrupted and must be rebuilt or restored from backup.

We’re not the only ones who have seen this behavior, either:

VMWare and NFS on NetApp Filers
NFS Datastores and what was their BIG issue…

The key takeaway from this is the following:

• If you haven’t yet applied ESX350-200808401-BG, you should apply it. Don’t wait. Now.
• If you did follow NetApp’s recommendations and set NFS.Lock.Disable, then review the instructions below (credit to Rick Scherer and several others, thank you!) to remedy the situation.

The instructions you should follow to restore NFS locking are here (these instructions include applying the patch):

1. Download patch ESX350-200808401-BG
2. Identify an ESX server against which you will apply the patch
3. Use VMotion and migrate the running VMs to other VMware ESX nodes
4. Install this patch on the selected VMware ESX server
5. In the Advanced Configuration settings ensure that NFS file locking is enabled with NFS.Lock.Disable=0 (you can also use the esxcfg-advcfg command to set this value)
6. Edit the /etc/vmware/config file and add this line:
   prefvmx.ConsolidateDeleteNFSLocks = "TRUE"
7. Save the changes to the file and reboot the VMware ESX host
8. Use VMotion to move VMs back to patched/NFS lock enabled/prefvmx enabled host
9. Repeat the steps above on each host in the cluster

So, again, if you haven’t applied ESX350-200808401-BG, please do so as quickly as possible, and then ensure that NFS locking is enabled. If you’ve previously disabled NFS locking, then follow the steps above to restore NFS locking and protect yourself against this possible split-brain scenario.