NVMe as a connection method for flash storage seems to be achieving near ubiquity. Perhaps that’s not quite true in all storage use cases, but for all-flash and hybrid flash arrays that service mainstream datacentre workloads, it is certainly true.
But NVMe is merely the drive-to-backplane connection. To connect over longer distances and fabrics/networks – spanning the rack and beyond – and to retain the advantages of NVMe connectivity, NVMe-over-fabrics is required.
NVMe-over-fabrics extends that super-fast flash access beyond the backplane.
In this article, we look at five key things you need to know about NVMe-over-fabrics.
What is NVMe-over-fabrics?
NVMe-over-fabrics arose because NVMe traffic – which superseded SCSI as the core storage transport – could only travel within host machines, between flash storage and the CPU, etc.
NVMe-over-fabrics implemented a messaging layer that enabled traffic between one device and another, and therefore across an extended network or fabric. There is a cost in terms of added latency, but this should be minimal when considered against the gains brought by NVMe-enabled flash storage.
NVMe-over-fabrics: RDMA, Fibre Channel and TCP
The application of NVMe-over-fabrics dates back to about 2016. It was decided to make NVMe-over-fabrics modular so that vendors could write driver that allowed it to run over a variety of transports.
Flavours that built on RDMA – such as over Converged Ethernet, iWARP and Infiniband – were early starters, with an approach that built NVMe connectivity as if it was an extension of memory between storage, hosts, etc. NVMe-over-RDMA formats often form part of shorter-distance connectivity, such as between controller and shelf.
Soon after, NVMe began to be implemented via recognised storage networking protocols. These included Fibre Channel, which despite its secular decline, is still a core technology for datacentre storage deployments.
The most recent destination for NVMe traffic over the fabric has been via Ethernet networks and using TCP, which offers the lure of little in the way of extra hardware.
Do we need extra hardware for NVMe-over-fabrics?
Some NVMe-over-fabrics approaches do not require extra hardware, but some do.
NVMe via RDMA-based approaches require RDMA-enabled network interface cards, and this is something that exists in almost all Gigabit Ethernet NICs. RDMA NVMe deployments are best carried out with configurations that eliminate lost packets (PFC and ECN).
Organisations that use Fibre Channel have been able to take advantage of NVMe-over-fabrics since Gen 5 without adding or replacing hardware. Also, Fibre Channel and NVMe-over-Fibre Channel can exist on the same fabric.
NVMe-over-TCP is the latest iteration of the over-fabrics approaches. It runs on standard Ethernet networking equipment, is fully routable and is not limited by distance. Having said that, NVMe-over-TCP brings much higher latency than the shorter-range methods, so you would need to be clear about what that meant for your applications.
NetApp recently added NVMe-over-TCP to its Ontap OS stack.
What operating systems support NVMe-over-fabrics?
Most notably, Microsoft Windows Server does not support NVMe-over-fabrics, instead sticking with SMB Direct, which is based on RDMA. Windows Server customers may be able to leverage NVMe-over-fabrics by using an initiator from a third party.
Red Hat Enterprise Linux and Suse Linux support RoCE, Fibre Channel and TCP deployments of NVMe-over-fabrics.
VMware supports RoCE and Fibre Channel NVMe-over-fabrics, but not over TCP yet.
NVMe-over-fabrics use cases
Clearly, the key benefit of NVMe-over-fabrics is low latency and storage performance, so any use case with rapid I/O requirements fits the bill, including transactional workloads, analytics, and possibly high-performance computing (HPC).
Where NVMe-over-fabrics benefits extend depends on the deployment.
Approaches that rely on RDMA etc are widely in use as transport within the storage subsystem – Pure Storage’s FlashArray//X, for example – and within the rack.
Longer-distance deployments will require, for example, Fibre Channel across the datacentre or even between sites. NVMe-over-TCP, meanwhile, is only really limited in distance terms by how much performance degradation you are prepared to suffer.
