Intel DPDK in virtualization environment

[cianfa72]

Hi,

I'm aware of Intel DPDK allows code in usermode running inside a process to interact directly with HW NIC bypassing the (host) kernel/OS. From my understanding DPDK is basically a library designed to be linked against the usermode code.

From an architectural point of view, I believe the model is that the physical NIC is polled by the usermode DPDK PMD (Poll Mode Driver) so there is actually no DMA performed by the NIC itself (i.e. dedicate host Cores poll the NIC and copy the NIC provided packets into userspace buffers provided by the DPDK library/user code).

If the above is correct, I've not however a clear understanding how DPDK model can be employed to exchange packets to/from Virtual Machines (VMs) that runs on the same physical box -- just to fix ideas assume QEM/KVM as virtualization environment running as hypervisor on the same box.

How does the DPDK library running inside its own process copy buffers to/from emulated NICs inside each QEMU hypervisor process ?

Thank you

[Octocontrabass]

From an architectural point of view, I believe the model is that the physical NIC is polled by the usermode DPDK PMD (Poll Mode Driver) so there is actually no DMA performed by the NIC itself (i.e. dedicate host Cores poll the NIC and copy the NIC provided packets into userspace buffers provided by the DPDK library/user code).

Polling means the driver doesn't use IRQs for transmit or receive. The driver can still use DMA.

Routing packets between VMs typically requires the hypervisor to provide a virtual switch and emulate the NICs, but some NICs include a hardware switch.

[cianfa72]

Polling means the driver doesn't use IRQs for transmit or receive.
The driver can still use DMA.

ok, so that actually means the driver code may program the DMA-capable NIC to perform DMA to/from buffers provided by PMD driver/DPDK library in userspace.

In that case which is the role of DPDK dedicated CPU Cores since the NIC itself perform the DMA in userspace?

[Octocontrabass]

DPDK is designed to be as fast as possible. Dedicating a core to each task means those tasks will never have to wait for another task to run, or wait while the OS migrates them to a different core.

This kind of extreme optimization is necessary when handling tens of gigabits of network traffic.

[cianfa72]

DPDK is designed to be as fast as possible. Dedicating a core to each task means those tasks will never have to wait for another task to run, or wait while the OS migrates them to a different core.

Yes that's definitely makes sense. The point unclear to me is what kind of job the DPDK dedicated Cores actually do.

As said before if the copy of data packets to/from target userspace buffers is done directly by NIC using DMA, then DPDK dedicated Cores's job is just manage and setp the the descriptor'structures, registers ecc.. shared between the PMD driver and DMA-capable NIC ? Thank you.

[Octocontrabass]

then DPDK dedicated Cores's job is just manage and setp the the descriptor'structures, registers ecc.. shared between the PMD driver and DMA-capable NIC ?

The dedicated cores are in charge of receiving packets from one or more Rx queues, performing work on those packets, and transmitting them to one or more Tx queues. The things you've listed are part of the process for receiving and transmitting packets.

For example, a VPN device built around DPDK might use two dedicated cores. One core would receive packets from the local network, decide whether they need encryption, encrypt them as needed, and send them to the internet. Another core would receive packets from the internet, decide which ones need decryption, decrypt them as needed, and send them to the local network.

[cianfa72]

The dedicated cores are in charge of receiving packets from one or more Rx queues, performing work on those packets, and transmitting them to one or more Tx queues. The things you've listed are part of the process for receiving and transmitting packets.

ok, so basically it is in charge of DPDK dedicated cores to "pullout" packets from Rx queue (Rx ring), process them according some logic, and possibly send/queue them into Tx queue (Tx ring).

Does it make sense ? Thank you.

[Octocontrabass]

Yes, that's correct.