Commit | Line | Data |
---|---|---|
617b586b HR |
1 | INFINIBAND MIDLAYER LOCKING |
2 | ||
3 | This guide is an attempt to make explicit the locking assumptions | |
4 | made by the InfiniBand midlayer. It describes the requirements on | |
5 | both low-level drivers that sit below the midlayer and upper level | |
6 | protocols that use the midlayer. | |
7 | ||
8 | Sleeping and interrupt context | |
9 | ||
10 | With the following exceptions, a low-level driver implementation of | |
11 | all of the methods in struct ib_device may sleep. The exceptions | |
12 | are any methods from the list: | |
13 | ||
14 | create_ah | |
15 | modify_ah | |
16 | query_ah | |
17 | destroy_ah | |
617b586b HR |
18 | post_send |
19 | post_recv | |
20 | poll_cq | |
21 | req_notify_cq | |
22 | map_phys_fmr | |
23 | ||
24 | which may not sleep and must be callable from any context. | |
25 | ||
26 | The corresponding functions exported to upper level protocol | |
27 | consumers: | |
28 | ||
29 | ib_create_ah | |
30 | ib_modify_ah | |
31 | ib_query_ah | |
32 | ib_destroy_ah | |
617b586b HR |
33 | ib_post_send |
34 | ib_post_recv | |
35 | ib_req_notify_cq | |
36 | ib_map_phys_fmr | |
37 | ||
38 | are therefore safe to call from any context. | |
39 | ||
40 | In addition, the function | |
41 | ||
42 | ib_dispatch_event | |
43 | ||
44 | used by low-level drivers to dispatch asynchronous events through | |
45 | the midlayer is also safe to call from any context. | |
46 | ||
47 | Reentrancy | |
48 | ||
49 | All of the methods in struct ib_device exported by a low-level | |
50 | driver must be fully reentrant. The low-level driver is required to | |
51 | perform all synchronization necessary to maintain consistency, even | |
52 | if multiple function calls using the same object are run | |
53 | simultaneously. | |
54 | ||
55 | The IB midlayer does not perform any serialization of function calls. | |
56 | ||
57 | Because low-level drivers are reentrant, upper level protocol | |
58 | consumers are not required to perform any serialization. However, | |
59 | some serialization may be required to get sensible results. For | |
60 | example, a consumer may safely call ib_poll_cq() on multiple CPUs | |
61 | simultaneously. However, the ordering of the work completion | |
62 | information between different calls of ib_poll_cq() is not defined. | |
63 | ||
64 | Callbacks | |
65 | ||
66 | A low-level driver must not perform a callback directly from the | |
67 | same callchain as an ib_device method call. For example, it is not | |
68 | allowed for a low-level driver to call a consumer's completion event | |
69 | handler directly from its post_send method. Instead, the low-level | |
70 | driver should defer this callback by, for example, scheduling a | |
71 | tasklet to perform the callback. | |
72 | ||
73 | The low-level driver is responsible for ensuring that multiple | |
74 | completion event handlers for the same CQ are not called | |
75 | simultaneously. The driver must guarantee that only one CQ event | |
76 | handler for a given CQ is running at a time. In other words, the | |
77 | following situation is not allowed: | |
78 | ||
79 | CPU1 CPU2 | |
80 | ||
81 | low-level driver -> | |
82 | consumer CQ event callback: | |
83 | /* ... */ | |
84 | ib_req_notify_cq(cq, ...); | |
85 | low-level driver -> | |
86 | /* ... */ consumer CQ event callback: | |
87 | /* ... */ | |
88 | return from CQ event handler | |
89 | ||
90 | The context in which completion event and asynchronous event | |
91 | callbacks run is not defined. Depending on the low-level driver, it | |
92 | may be process context, softirq context, or interrupt context. | |
93 | Upper level protocol consumers may not sleep in a callback. | |
94 | ||
95 | Hot-plug | |
96 | ||
97 | A low-level driver announces that a device is ready for use by | |
98 | consumers when it calls ib_register_device(), all initialization | |
99 | must be complete before this call. The device must remain usable | |
100 | until the driver's call to ib_unregister_device() has returned. | |
101 | ||
102 | A low-level driver must call ib_register_device() and | |
103 | ib_unregister_device() from process context. It must not hold any | |
104 | semaphores that could cause deadlock if a consumer calls back into | |
105 | the driver across these calls. | |
106 | ||
107 | An upper level protocol consumer may begin using an IB device as | |
108 | soon as the add method of its struct ib_client is called for that | |
109 | device. A consumer must finish all cleanup and free all resources | |
110 | relating to a device before returning from the remove method. | |
111 | ||
112 | A consumer is permitted to sleep in its add and remove methods. |