Commit | Line | Data |
---|---|---|
8d497515 CY |
1 | An Intel MIC X100 device is a PCIe form factor add-in coprocessor |
2 | card based on the Intel Many Integrated Core (MIC) architecture | |
3 | that runs a Linux OS. It is a PCIe endpoint in a platform and therefore | |
4 | implements the three required standard address spaces i.e. configuration, | |
5 | memory and I/O. The host OS loads a device driver as is typical for | |
6 | PCIe devices. The card itself runs a bootstrap after reset that | |
af190494 DC |
7 | transfers control to the card OS downloaded from the host driver. The |
8 | host driver supports OSPM suspend and resume operations. It shuts down | |
9 | the card during suspend and reboots the card OS during resume. | |
8d497515 CY |
10 | The card OS as shipped by Intel is a Linux kernel with modifications |
11 | for the X100 devices. | |
12 | ||
13 | Since it is a PCIe card, it does not have the ability to host hardware | |
14 | devices for networking, storage and console. We provide these devices | |
c74c9318 SD |
15 | on X100 coprocessors thus enabling a self-bootable equivalent |
16 | environment for applications. A key benefit of our solution is that it | |
17 | leverages the standard virtio framework for network, disk and console | |
18 | devices, though in our case the virtio framework is used across a PCIe | |
19 | bus. A Virtio Over PCIe (VOP) driver allows creating user space | |
20 | backends or devices on the host which are used to probe virtio drivers | |
21 | for these devices on the MIC card. The existing VRINGH infrastructure | |
22 | in the kernel is used to access virtio rings from the host. The card | |
23 | VOP driver allows card virtio drivers to communicate with their user | |
24 | space backends on the host via a device page. Ring 3 apps on the host | |
25 | can add, remove and configure virtio devices. A thin MIC specific | |
26 | virtio_config_ops is implemented which is borrowed heavily from | |
27 | previous similar implementations in lguest and s390. | |
8d497515 | 28 | |
a8035843 SY |
29 | MIC PCIe card has a dma controller with 8 channels. These channels are |
30 | shared between the host s/w and the card s/w. 0 to 3 are used by host | |
31 | and 4 to 7 by card. As the dma device doesn't show up as PCIe device, | |
32 | a virtual bus called mic bus is created and virtual dma devices are | |
33 | created on it by the host/card drivers. On host the channels are private | |
34 | and used only by the host driver to transfer data for the virtio devices. | |
35 | ||
7df20f2d SD |
36 | The Symmetric Communication Interface (SCIF (pronounced as skiff)) is a |
37 | low level communications API across PCIe currently implemented for MIC. | |
38 | More details are available at scif_overview.txt. | |
39 | ||
f4a66c20 AD |
40 | The Coprocessor State Management (COSM) driver on the host allows for |
41 | boot, shutdown and reset of Intel MIC devices. It communicates with a COSM | |
42 | "client" driver on the MIC cards over SCIF to perform these functions. | |
43 | ||
8d497515 CY |
44 | Here is a block diagram of the various components described above. The |
45 | virtio backends are situated on the host rather than the card given better | |
46 | single threaded performance for the host compared to MIC, the ability of | |
47 | the host to initiate DMA's to/from the card using the MIC DMA engine and | |
48 | the fact that the virtio block storage backend can only be on the host. | |
49 | ||
a8035843 SY |
50 | +----------+ | +----------+ |
51 | | Card OS | | | Host OS | | |
52 | +----------+ | +----------+ | |
53 | | | |
54 | +-------+ +--------+ +------+ | +---------+ +--------+ +--------+ | |
55 | | Virtio| |Virtio | |Virtio| | |Virtio | |Virtio | |Virtio | | |
56 | | Net | |Console | |Block | | |Net | |Console | |Block | | |
57 | | Driver| |Driver | |Driver| | |backend | |backend | |backend | | |
c74c9318 | 58 | +---+---+ +---+----+ +--+---+ | +---------+ +----+---+ +--------+ |
a8035843 SY |
59 | | | | | | | | |
60 | | | | |User | | | | |
c74c9318 SD |
61 | | | | |------|------------|--+------|------- |
62 | +---------+---------+ |Kernel | | |
63 | | | | | |
64 | +---------+ +---+----+ +------+ | +------+ +------+ +--+---+ +-------+ | |
65 | |MIC DMA | | VOP | | SCIF | | | SCIF | | COSM | | VOP | |MIC DMA| | |
66 | +---+-----+ +---+----+ +--+---+ | +--+---+ +--+---+ +------+ +----+--+ | |
67 | | | | | | | | | |
68 | +---+-----+ +---+----+ +--+---+ | +--+---+ +--+---+ +------+ +----+--+ | |
69 | |MIC | | VOP | |SCIF | | |SCIF | | COSM | | VOP | | MIC | | |
70 | |HW Bus | | HW Bus| |HW Bus| | |HW Bus| | Bus | |HW Bus| |HW Bus | | |
71 | +---------+ +--------+ +--+---+ | +--+---+ +------+ +------+ +-------+ | |
72 | | | | | | | | | |
73 | | +-----------+--+ | | | +---------------+ | | |
74 | | |Intel MIC | | | | |Intel MIC | | | |
75 | | |Card Driver | | | | |Host Driver | | | |
76 | +---+--------------+------+ | +----+---------------+-----+ | |
f4a66c20 | 77 | | | | |
a8035843 SY |
78 | +-------------------------------------------------------------+ |
79 | | | | |
80 | | PCIe Bus | | |
81 | +-------------------------------------------------------------+ |