(Author’s note: I’ve cross-posted this article to my own site as well.)

I’ve had this link sitting in my “Articles To Read” list for quite some time, but—to be perfectly honest—I’ve been just too busy to really do anything about it. Now that a hectic few weeks has wrapped up and I have a small breather before the next hectic few weeks, I wanted to comment briefly on Doug Gourlay’s discussion of FCoE versus MR-IOV.

First, some background: For those that aren’t familiar, FCoE is Fibre Channel over Ethernet, a T11 standard for running Fibre Channel Protocol over Ethernet, specifically 10 Gigabit Ethernet. More information on FCoE is found here. MR-IOV is Multi-Root I/O Virtualization, a PCI SIG specification for using PCI Express (PCIe) to connect and share multiple devices. More information on MR-IOV can be found here. MR-IOV is a multi-server extension to Single-Root I/O Virtualization, or SR-IOV.

Like Doug, I’ll put in a disclaimer that I haven’t read the report to which he’s referring in his article, either. However, as an individual who has done some research on the topic of I/O virtualization, I will say that anyone who compares FCoE to MR-IOV is comparing apples to oranges. These two technologies, in my mind, are designed to address two different problems.

FCoE provides the ability to use a single physical transport—10 Gigabit Ethernet, in this case—for Fibre Channel Protocol (FCP) as well as TCP/IP, iSCSI, and other Ethernet-borne protocols. This allows for the creation of a unified fabric, a single physical transport that carries all the various kinds of traffic that Ethernet-based Local Area Networks (LANs) and Storage Area Networks (SANs) carry separately today. Via the IETF Converged Enhanced Ethernet (CEE) standard—adopted by Cisco as Data Center Ethernet^TM—FCoE will ultimately have the same low, predictable latency and error-free operation that FCP enjoys today. FCoE is not, however, designed or architected to do anything other than allow FCP to run over Ethernet. It’s not intended to be a server interconnect technology. (Unless I’m missing something?)

MR-IOV, on the other hand, is intended to play in the server interconnect field. Its purpose is not to allow FCP to run over Ethernet, or to allow FCoE, iSCSI, and other TCP/IP protocols share the same physical connections. MR-IOV’s purpose is to allow multiple servers to share PCIe-based devices, like a FC Host Bus Adapter (HBA), or an iSCSI HBA, a 10 Gigabit Ethernet network interface card (NIC), or a video capture card. MR-IOV is intended to provide I/O virtualization, regardless of what type of I/O that might be. As long as the I/O runs across a PCI Express bus, MR-IOV comes into play.

I’ve heard multiple people refer to FCoE as an I/O virtualization technology, but I just don’t agree. FCoE only applies to FCP over Ethernet. It doesn’t apply to iSCSI. It doesn’t apply to video traffic, or audio traffic, or HTTP traffic. It only applies to FCP over Ethernet. While I might allow that FCoE does allow for a form of virtualization, by virtualizing the physical transport beneath FCP, I would not call it I/O virtualization. Further, FCoE and MR-IOV are complementary. You could use MR-IOV to share a single Converged Network Adapter (CNA), which provides FCoE and 10 Gigabit Ethernet functionality, among multiple servers. In this situation, what’s providing the I/O virtualization: MR-IOV, which is allowing multiple servers to use a single I/O card, or the CNA, which is putting the traffic onto the converged fabric?

I’m probably missing something huge here, some vital piece of information that would make sense why FCoE and MR-IOV would be considered competitive standards/specifications. Without that information, though, it just doesn’t make any sense to me to compare these two different yet complementary technologies. Someone want to enlighten me?

UPDATE: I’ve corrected my use of “Data Center Ethernet” to Converged Enhanced Ethernet (CEE) when referring to the IETF standard. As correctly pointed out in the comments, Data Center Ethernet^TM is a Cisco trademarked term referring to their implementation of CEE.

You may have noticed that posts have slowed down in the last couple of weeks. The reason is that we are in the process of making a pretty dramatic update to the main site. We have engaged a team of professional developers and have settled on a new design for the site which will provide a number of enhancements including:

· Daily news commentary – brief snippets of linked news items with commentary
· Daily email of news and commentary updates every morning (don’t worry, you will not be spammed, only people who opt-in for the email will receive it)
· An online knowledgebase of vendor and analyst information
· Social networking enhancements - Further Twitter and LinkedIn integration. We are also looking at integration with other social networking sites

The downside of the upgrade is that we will have to remove the “Groups” section due to some technical and browser compatibility issues that we are in the process of resolving.

The new site is not only visually appealing, we believe it is far more functional and useful than the current site. Our original target date for launching the new site was November 1st… but this was delayed due to further changes we needed to make. Or our target now for the relaunch is now the first week in December.

(Author’s note: This was cross-published to my site as well.)

I’m sure that others have already picked this up, but I thought it was worth mentioning that NetApp has added deduplication functionality to their virtual tape library (VTL) product line. The official press release was published on October 28, 2008:

NetApp (NASDAQ: NTAP) today announced the availability of deduplication on NetApp® Virtual Tape Library (VTL) systems, enabling customers to lower the disk capacity required to backup any storage system, including EMC and HP, up to 95%¹. NetApp delivers unparalleled cost savings for traditional backup environments by reducing tape media usage and tape drive infrastructure and, now, through highly efficient usage of VTL disk storage through deduplication. With the addition of deduplication on VTL, NetApp offers the most complete deduplication portfolio, spanning backup, archive, and primary storage applications.

Per the press release, it also appears that VTL deduplication will be available to both new and existing customers at no additional charge.

Check out the official press release for all the details.

The economic downturn is starting to really hurt vendors. While Overland Storage is beginning to circle the bowl, Pillar Data Systems CEO, Mike Workman, announced on his blog yesterday that they too will be facing some dramatic changes:

Pillar is anticipating a tough economic climate will persist throughout 2009 and into 2010, so we have decided to batten down the hatches and lay off some of our employees.  These decisions are always among the toughest decisions a CEO has to make. But our jobs require we do what’s necessary to fulfill our duties to the company and its shareholders.

In my opinion, Pillar is an odd company. They have some really interesting technology in the way they provide an ILM solution by creating tiers of storage based on where the data is physically located on the disk. Unless you are required by management to tier storage out this way, it seems like a “nice to have” rather than a “need to have”.

The challenges keep mounting for tape library and (now) NAS vendor Overland Storage. Just a quarter after purchasing Snap Appliance from Adaptec, Overland Storage has been warned that it could be delisted from the NASDAQ. Overland Storage also announced yesterday that it had posted a first-quarter net loss of $6.9 million for it’s fiscal year first quarter (which ended September 30th). At 11:30am EST today, Overland Storage stock was down to .17 a share, down over 40% from yesterday. Overland Storage also noted that it is seeking $10 million “to continue to execute its strategy“.

Besides manufacturing a series of tape libraries and the Snap Server NAS, Overland Storage also produces a line of Virtual Tape Libraries, Tape Autoloaders and SAN Fibre Channel storage devices.

(Author’s note: This was cross-published on my own site as well.)

We ran into an interesting set of problems at work this week, all of them related to running virtual machines on VMware ESX over NetApp NFS. While we haven’t yet determined the root cause for all of the problem, we did uncover the root cause for one of the issues, and additional testing seems to indicate that one of the other problems may also be suffering from the same condition.

The problem seems to lie within some (now outdated) recommendations by NetApp for using NFS with VMware ESX. As late as June of this year, NetApp was recommending—via TR-3428, “NetApp and VMware Virtual Infrastructure 3 Storage Best Practices”—that users disable NFS locking on the storage system by setting the NFS.Lock.Disable to 1. Version 4.0 of TR-3428 was the version published in June of this year, and it still contained this recommendation.

As I understand it, this recommendation stemmed from a bug that existed in VMware ESX that caused long delays when removing VMware snapshots. As a workaround, NetApp recommended setting the NFS.Lock.Disable setting. Supposedly, VMware has fixed that bug with this patch, and the current version of TR-3428 (version 4.2, published in September 2008 and available from NetApp’s web site) no longer contains the recommendation to set NFS.Lock.Disable.

Unfortunately, customers who followed (then current) best practices may have set this value, and in so doing may have exposed themselves to a “split brain” scenario in which a VM is simultaneously registered and running on multiple VMware ESX hosts. Users may be particularly at risk if they also run VMware HA and have not protected themselves against isolation response (these articles are tagged ‘VMwareHA’ and will provide more information on isolation response). We’ve had one large customer be affected by this problem, and another large customer who is having problems that we believe are related to this same issue.

There are a couple of important things to note:

• This is not a NetApp problem, but rather the result of a recommendation from NetApp. At that time, it was believed that this setting was necessary.
• In most cases VMs affected by this split-brain scenario will get corrupted and must be rebuilt or restored from backup.

We’re not the only ones who have seen this behavior, either:

VMWare and NFS on NetApp Filers
NFS Datastores and what was their BIG issue…

The key takeaway from this is the following:

• If you haven’t yet applied ESX350-200808401-BG, you should apply it. Don’t wait. Now.
• If you did follow NetApp’s recommendations and set NFS.Lock.Disable, then review the instructions below (credit to Rick Scherer and several others, thank you!) to remedy the situation.

The instructions you should follow to restore NFS locking are here (these instructions include applying the patch):

1. Download patch ESX350-200808401-BG
2. Identify an ESX server against which you will apply the patch
3. Use VMotion and migrate the running VMs to other VMware ESX nodes
4. Install this patch on the selected VMware ESX server
5. In the Advanced Configuration settings ensure that NFS file locking is enabled with NFS.Lock.Disable=0 (you can also use the esxcfg-advcfg command to set this value)
6. Edit the /etc/vmware/config file and add this line:
   prefvmx.ConsolidateDeleteNFSLocks = "TRUE"
7. Save the changes to the file and reboot the VMware ESX host
8. Use VMotion to move VMs back to patched/NFS lock enabled/prefvmx enabled host
9. Repeat the steps above on each host in the cluster

So, again, if you haven’t applied ESX350-200808401-BG, please do so as quickly as possible, and then ensure that NFS locking is enabled. If you’ve previously disabled NFS locking, then follow the steps above to restore NFS locking and protect yourself against this possible split-brain scenario.

NetApp has emerged as the first major vendor to announce native Fibre Channel over Ethernet (FCoE) SAN storage solutions and new questions are being raised about the wisdom of Dell’s $1.4 billion acquisition of Equallogic.

From NetApp’s announcement:

“Enterprise companies today are looking to transition to a converged data center, and FCoE is going to play a big part in that move,” said Patrick Rogers, vice president of Solutions Marketing, NetApp. “With companies keeping a closer eye than ever on their IT budgets, data center convergence will enable them to reduce their costs and power requirements. As a leader in FCoE, NetApp will be at the forefront in helping customers realize the benefits of a more streamlined and efficient data center.”

Meanwhile, Chris Mellor at The Register raises concerns about what FCoE means for iSCSI and Equallogic/Dell:

Xiotech’s chief technology officer, Steve Sicola, not only thinks this is the way things will play out but that it could substantially reduce the value of Dell’s EqualLogic acquisition. Dell spent $1.4bn buying EqualLogic’s iSCSI market technology late last year and now the whole iSCSI market looks set to be killed by FCoE. Bad move, guys.

FCoE levels the block-level access playing field. There’s no longer the marketing distinction between a Fibre Channel array - fast, costs more - and an iSCSI array - slower, less reliable access, cheaper. It’s all Fibre Channel. The distinction will move to the media inside the array and the array software. So all iSCSI array vendors will need to rethink their marketing propositions and retool their product strategies.

There is no way out for iSCSI. It’s at war and FCoE is the killing field.

iSCSI isn’t going to fade away any time soon, but it will probably be relegated to small business SAN’s. Enterprises with Fibre Channel will almost certainly go with FCoE once it’s available instead of iSCSI where it’s of little technical value.

How bad is this for Equallogic/Dell?

This is a tough call because there are a number of scenarios that could play out but this ultimately depends on what their roadmap for Equallogic looks like. Do they even want a piece of the fibre channel storage market leveraging Equallogic or do they exploit the rest of the Dell storage line which is better suited to address enterprise users?

I’m not sure it makes sense for Dell to leverage Equallogic for the Fibre Channel enterprise user. Equallogic uses a proprietary card for iSCSI so it would not be a simple matter for them to make it FCoE… it would need to be developed. They could always use another vendor’s FCoE technology, but then there would be little differention in the market for Equallogic.

As FCoE rolls out, I think the protocols will be split by end user size and requirements. My guess is that Dell will keep Equallogic as an iSCSI only SAN and will use it for smaller businesses requiring a feature rich, dynamic storage. FCoE will start in the enterprise and push it’s way down to the mid-sized companies requiring performance and security.

Chris Mellor has another excellent scoop from Compellent that they will also be adding solid state drives to their systems. Stephen Foskett likes it too:

Yes! Compellent has just announced at Storage Networking World that they’ll be adding enterprise solid state drives (SSDs) to their excellent fully-virtualized storage arrays. Why is this worth shouting about? Simply because their automated block-based tiered storage architecture ought to be able to really take full advantage of the performance offered by SSDs. If you’ll pardon the pun, SSD in a Compellent array is positively compelling!

Compellent’s “Data Progression” technology is true automated Information Lifecycle Management that no other vendor (that I’m aware of) is offering. “Data Progression” automatically moves data to the appropriate tier of storage. With this announcement, Compellent is introducing SSD’s into the enterprise storage market with intelligence as opposed to be just another tier of storage.

This will be the first Storage Networking World that I will not be attending in the last three years. Between existing projects and the company’s decision to limit travel, SNW is not a priority right now for us.

I’ve always had a mixed opinion about the value of SNW. I think the education is good and the interaction with vendors is helpful if we have a project coming up… but the last few conferences seem to be less about end-users and more about analysts and vendors spinning the hype with new announcements. To be fair, any event of this size is going to get the marketing folks working overtime to maximize their exposure - there is no avoiding it. I simply think this needs to be balanced if this is an end-user event and not an industry insider event… which is what it feels like. I think VMworld has done a much better job of balancing the value between end-users and vendors and I hope SNW gets to that point.

Then there’s the SNW hangover which usually lasts for a month… the period of time following SNW when every vendor you talked to or who scanned your badge will call and email you non-stop. As a consequence, I learned some creative ways to make up my name and contact info when registering. It’s not that I don’t respect the job they do, it’s that I also have a job to do and I can’t be on the phone 4 hours a day with telemarketers following up from the show. As the economy tightens up, I suspect this will only get worse.

The deduplication wars are starting.

Chris Mellor at The Register points out Sepaton’s new guarantee of VTL data deduplication ratios on the heels of NetApp’s recent guarantee to reduce storage in virtual environments by 50%. The Sepaton guarantee is that the S2100-ES2 virtual tape library will reduce Microsoft Exchange and Oracle databases by a 40:1.

This is a bold move for Sepaton which has generally been viewed as a “me too” VTL until HP decided to start selling it. The big difference between the two is that one is for backup data, the other is for virtual environments… but both guarantee target data that is predictably redundant.

What’s next… Data Domain promising 100:1 deduplication ratios for all backed up word documents on a server?

While guarantees like these are intended to grab some attention, both miss an important point. When end-users are evaluating deduplication solutions, we want to see reasonable ratios for all deduped data that these solutions might provide… not just data being served up to virtualized servers and not just two applications that are backed up every night. I haven’t seen any first hand, but I would have to imagine that some vendor has an analytic tool that could look at all of the data that would be deduped and come up with an expected dedupe ratio. This kind of information would be far more valuable when discerning deduplication options than a narrow guarantee.