What is NVMe? Solution Overview

Following a recent discussion with some customers, the conversation drifted towards NVMe. I was surprised to hear that people considered NVMe to simply be a new type of SSD. Read on as I address this misconception and give an overview of NVMe and why its so much more than SSD.

NvMe is a whole new transport layer protocol that removes the constraints of backward compatibility which was imposed on previous standards such as SAS, SATA and SCSI. Those were developed in an era of rotational storage media and it was that reliance on rotational media that created the largest amount of inherent latency in the I/O path. However, with the advent of Flash media those latencies were stripped away as there is no need to wait for the head to move to the correct area of the platter and the disk to rotate to the correct sector. As a result the bottleneck has moved from being the media access, to the actual protocol itself. SAS as a protocol has a single queue and the queue has a maximum of 254 entries, with SATA being lower again at 32 entries. If the queue becomes full then there is a pause on new I/O whilst the current entries in the queue are serviced to make space. In the days of rotational media those queues could be considered more than sufficient, however with the adoption of Flash based arrays the more legacy protocols of SAS and SATS have lead to the protocols becoming the key factor in maintaining the higher latencies.

Enter NVMe, if we compare the protocols as if they were a road network then SAS could be represented as a single track road with some traffic lights to slow the traffic. Whereas NVMe appears as an incredibly wide super highway, with thousands of lanes each representing a queue with no traffic lights at all, so all traffic flows incredibly fast with traffic jams.

The actual figures for NVMe are that it has 64000 queues and each queue has up to 64000 possible entries. Whilst it is conceivable that these could become filled at some time in the future as storage media is further developed the reality is that with current technologies that prospect is quite some time away. The performance differential is enormous, using near equivalent arrays running with SAS vs NVMe the latency figures are seen to reduce by at least 25% which is significant.

Will NVMe replace SAS, certainly over the next 12 months increasing numbers of storage systems will continue that transition as NVMe finds its way into more of the mid range storage products.  But when purchasing the systems it will become a trade off between justifying the additional cost vs the relative performance boost. Do all customers currently need that more extreme level of performance possibly not but it is an important discussion point.

Another relatively new media beginning to find its way into Enterprise class storage systems is SCM. So what is SCM, well in reality it is the next evolution in persistent storage media a next generation of Flash if you like.  SCM stands for Storage Class Memory and it is aiming to bridge the gap between current flash storage speeds and DRAM. SCM is something we will see increasingly as an extension of Memory but will also see a return to a tiered storage architecture where a relatively small amount of SCM mixed with NVMe Flash will create a more cost effective way of delivering extreme performance without the need for full array SCM. This technology is already available in the DellEMC Powermax and it uses machine intelligent learning to position the most frequently accessed data on SCM.

If latency has been removed from the access media the next challenge is to ensure that significant latency is not added in the other areas of the I/O path so NVMeOF (NVMe Over Fabric) was developed. NVMeOF can be delivered via differing media such as Fibre Channel, Infinniband and Ethernet, there are a number of considerations in that HBA driver and firmware changes may be required to support NVMeOF from the host perspective. From an array point of view ensuring that NVMeOF is supported on the I/O cards included will be crucial along with the transport medium being of sufficient speed. For example using Broacade Gen 6 FC switches using NVMe over FC vs FCP with SCSI they observed 58% higher IOPS and 34% lower latency using the same hardware. Since FC is the dominant connection medium in the Datacentre and that on current supported hardware both NVMe over FC and traditional FC SCSI can coexist then it would be expected that FC will retain its dominance for the moment.

Stephen Keohane | Ergo Solutions Architect