I have few questions regarding peer-to-peer communication of two end point devices in PCIe hierarchy on new Intel Ivy Bridge platform.
In the end I would like to program two PCIe end point devices to establish peer-to-peer communication
and I want to find a method in software to determine if such communication is possible on given platform.
Just to make sure I properly understand Root Complex (RC) and Root Port roles in PCIe, as described by PCIe spec.
1. RC is the root of an PCIe hierarchy that connects the CPU/memory subsystem to the PCIe.
2. RC consists of Root Ports -- each Root Port connects to RC its PCIe hierarchy that accessible through this Root Port.
3. RC is logical aggregation of root ports into one PCIe hierarchy.
4. Routing of peer-to-peer communication between Root Ports in same RC is optional.
Am I right?
Now, for example, I would like to utilize many GPU based accelerators that can communicate peer-to-peer without CPU involvement -- PCIe P2P communication.
I am looking on the Intel's Ivy Bridge based dual socket platform. For example Server Board S2600WP.
http://download.intel.com/support/mothe ... s_r1_3.pdf
According to the schematic on page 23, Fig 12, PCIe lines connected directly to the CPU's I/O subsystem controller
and some PCIe line connected to the PCH (Platform Controller Hub), page 9, Figure 5.
Assume that I have two PCIe devices e.g. GPU PCIe x16. One connected to CPU0 and another connected to CPU1
and I can not move devices to another slots (all slots are full).
Code: Select all
GPU0 ----- I/O CPU0 --- QPI ---- I/O CPU1 ----- GPU1
|
|
PCH --- more PCIe lines
q1. Is there any way for software to check if GPU0 can establish peer-to-peer communication with GPU1?
I mean, does peer-to-peer routing implemented in Ivy Bridge? I could not find any statement in data-sheets.
q2. Do we have two Root Complex on Ivy Bridge? Is it possible at all to have multiple RCs on same system, any examples of such systems? If yes, what does it mean from the OS programming perspective (two PCIe hierarchies, enumeration, addressing, interrupt routing)?
q3. What system components included in Root Complex on this board?
q4. Are Root Ports using QPI for interconnection on Ivy Bridge?
Thank you.