Efficient Networking for Distributed Cloud Analytics

When networking becomes a bottleneck in applications, developers often turn to the Data Plane Development Kit (DPDK) to increase networking speed. This is effective, since DPDK enables networking in user space, which evades costly system calls and hardware interrupts. Launched by Intel [source] and managed by the Linux Foundation, DPDK is widely adopted.

But how much of the DPDK is really necessary for a distributed database system? In many cases, more code correlates with more overhead, slowing computation down. One central part of the DPDK is the Environment Abstraction Layer (EAL), which abstracts from differences in hardware and even operating systems.[source] Is it possible to strip away everything on top of the EAL and build a small layer on top that provides especially the services needed by distributed database systems in order to communicate? How performant would that be in comparison to the original DPDK?

Losing data during the shuffle phase of a distributed join due to faulty network connections is no option for database systems. The TCP/IP stack provides the TCP protocol for reliable communication over an unreliable network. But in distributed database systems today, the network connection is often not as defective, since servers are often located in the same data center, often even on the same rack. Package loss is to be expected to be much lower than through the whole internet.

Content