065c6359 |
1 | |
2 | PCI Error Recovery |
3 | ------------------ |
4 | May 31, 2005 |
5 | |
6 | Current document maintainer: |
7 | Linas Vepstas <linas@austin.ibm.com> |
8 | |
9 | |
10 | Some PCI bus controllers are able to detect certain "hard" PCI errors |
11 | on the bus, such as parity errors on the data and address busses, as |
12 | well as SERR and PERR errors. These chipsets are then able to disable |
13 | I/O to/from the affected device, so that, for example, a bad DMA |
14 | address doesn't end up corrupting system memory. These same chipsets |
15 | are also able to reset the affected PCI device, and return it to |
16 | working condition. This document describes a generic API form |
17 | performing error recovery. |
18 | |
19 | The core idea is that after a PCI error has been detected, there must |
20 | be a way for the kernel to coordinate with all affected device drivers |
21 | so that the pci card can be made operational again, possibly after |
22 | performing a full electrical #RST of the PCI card. The API below |
23 | provides a generic API for device drivers to be notified of PCI |
24 | errors, and to be notified of, and respond to, a reset sequence. |
25 | |
26 | Preliminary sketch of API, cut-n-pasted-n-modified email from |
27 | Ben Herrenschmidt, circa 5 april 2005 |
28 | |
29 | The error recovery API support is exposed to the driver in the form of |
30 | a structure of function pointers pointed to by a new field in struct |
31 | pci_driver. The absence of this pointer in pci_driver denotes an |
32 | "non-aware" driver, behaviour on these is platform dependant. |
33 | Platforms like ppc64 can try to simulate pci hotplug remove/add. |
34 | |
35 | The definition of "pci_error_token" is not covered here. It is based on |
36 | Seto's work on the synchronous error detection. We still need to define |
37 | functions for extracting infos out of an opaque error token. This is |
38 | separate from this API. |
39 | |
40 | This structure has the form: |
41 | |
42 | struct pci_error_handlers |
43 | { |
44 | int (*error_detected)(struct pci_dev *dev, pci_error_token error); |
45 | int (*mmio_enabled)(struct pci_dev *dev); |
46 | int (*resume)(struct pci_dev *dev); |
47 | int (*link_reset)(struct pci_dev *dev); |
48 | int (*slot_reset)(struct pci_dev *dev); |
49 | }; |
50 | |
51 | A driver doesn't have to implement all of these callbacks. The |
52 | only mandatory one is error_detected(). If a callback is not |
53 | implemented, the corresponding feature is considered unsupported. |
54 | For example, if mmio_enabled() and resume() aren't there, then the |
55 | driver is assumed as not doing any direct recovery and requires |
56 | a reset. If link_reset() is not implemented, the card is assumed as |
57 | not caring about link resets, in which case, if recover is supported, |
58 | the core can try recover (but not slot_reset() unless it really did |
59 | reset the slot). If slot_reset() is not supported, link_reset() can |
60 | be called instead on a slot reset. |
61 | |
62 | At first, the call will always be : |
63 | |
64 | 1) error_detected() |
65 | |
66 | Error detected. This is sent once after an error has been detected. At |
67 | this point, the device might not be accessible anymore depending on the |
68 | platform (the slot will be isolated on ppc64). The driver may already |
69 | have "noticed" the error because of a failing IO, but this is the proper |
70 | "synchronisation point", that is, it gives a chance to the driver to |
71 | cleanup, waiting for pending stuff (timers, whatever, etc...) to |
72 | complete; it can take semaphores, schedule, etc... everything but touch |
73 | the device. Within this function and after it returns, the driver |
74 | shouldn't do any new IOs. Called in task context. This is sort of a |
75 | "quiesce" point. See note about interrupts at the end of this doc. |
76 | |
77 | Result codes: |
78 | - PCIERR_RESULT_CAN_RECOVER: |
79 | Driever returns this if it thinks it might be able to recover |
80 | the HW by just banging IOs or if it wants to be given |
81 | a chance to extract some diagnostic informations (see |
82 | below). |
83 | - PCIERR_RESULT_NEED_RESET: |
84 | Driver returns this if it thinks it can't recover unless the |
85 | slot is reset. |
86 | - PCIERR_RESULT_DISCONNECT: |
87 | Return this if driver thinks it won't recover at all, |
88 | (this will detach the driver ? or just leave it |
89 | dangling ? to be decided) |
90 | |
91 | So at this point, we have called error_detected() for all drivers |
92 | on the segment that had the error. On ppc64, the slot is isolated. What |
93 | happens now typically depends on the result from the drivers. If all |
94 | drivers on the segment/slot return PCIERR_RESULT_CAN_RECOVER, we would |
95 | re-enable IOs on the slot (or do nothing special if the platform doesn't |
96 | isolate slots) and call 2). If not and we can reset slots, we go to 4), |
97 | if neither, we have a dead slot. If it's an hotplug slot, we might |
98 | "simulate" reset by triggering HW unplug/replug though. |
99 | |
100 | >>> Current ppc64 implementation assumes that a device driver will |
101 | >>> *not* schedule or semaphore in this routine; the current ppc64 |
102 | >>> implementation uses one kernel thread to notify all devices; |
103 | >>> thus, of one device sleeps/schedules, all devices are affected. |
104 | >>> Doing better requires complex multi-threaded logic in the error |
105 | >>> recovery implementation (e.g. waiting for all notification threads |
106 | >>> to "join" before proceeding with recovery.) This seems excessively |
107 | >>> complex and not worth implementing. |
108 | |
109 | >>> The current ppc64 implementation doesn't much care if the device |
110 | >>> attempts i/o at this point, or not. I/O's will fail, returning |
111 | >>> a value of 0xff on read, and writes will be dropped. If the device |
112 | >>> driver attempts more than 10K I/O's to a frozen adapter, it will |
113 | >>> assume that the device driver has gone into an infinite loop, and |
114 | >>> it will panic the the kernel. |
115 | |
116 | 2) mmio_enabled() |
117 | |
118 | This is the "early recovery" call. IOs are allowed again, but DMA is |
119 | not (hrm... to be discussed, I prefer not), with some restrictions. This |
120 | is NOT a callback for the driver to start operations again, only to |
121 | peek/poke at the device, extract diagnostic information, if any, and |
122 | eventually do things like trigger a device local reset or some such, |
123 | but not restart operations. This is sent if all drivers on a segment |
124 | agree that they can try to recover and no automatic link reset was |
125 | performed by the HW. If the platform can't just re-enable IOs without |
126 | a slot reset or a link reset, it doesn't call this callback and goes |
127 | directly to 3) or 4). All IOs should be done _synchronously_ from |
128 | within this callback, errors triggered by them will be returned via |
129 | the normal pci_check_whatever() api, no new error_detected() callback |
130 | will be issued due to an error happening here. However, such an error |
131 | might cause IOs to be re-blocked for the whole segment, and thus |
132 | invalidate the recovery that other devices on the same segment might |
133 | have done, forcing the whole segment into one of the next states, |
134 | that is link reset or slot reset. |
135 | |
136 | Result codes: |
137 | - PCIERR_RESULT_RECOVERED |
138 | Driver returns this if it thinks the device is fully |
139 | functionnal and thinks it is ready to start |
140 | normal driver operations again. There is no |
141 | guarantee that the driver will actually be |
142 | allowed to proceed, as another driver on the |
143 | same segment might have failed and thus triggered a |
144 | slot reset on platforms that support it. |
145 | |
146 | - PCIERR_RESULT_NEED_RESET |
147 | Driver returns this if it thinks the device is not |
148 | recoverable in it's current state and it needs a slot |
149 | reset to proceed. |
150 | |
151 | - PCIERR_RESULT_DISCONNECT |
152 | Same as above. Total failure, no recovery even after |
153 | reset driver dead. (To be defined more precisely) |
154 | |
155 | >>> The current ppc64 implementation does not implement this callback. |
156 | |
157 | 3) link_reset() |
158 | |
159 | This is called after the link has been reset. This is typically |
160 | a PCI Express specific state at this point and is done whenever a |
161 | non-fatal error has been detected that can be "solved" by resetting |
162 | the link. This call informs the driver of the reset and the driver |
163 | should check if the device appears to be in working condition. |
164 | This function acts a bit like 2) mmio_enabled(), in that the driver |
165 | is not supposed to restart normal driver I/O operations right away. |
166 | Instead, it should just "probe" the device to check it's recoverability |
167 | status. If all is right, then the core will call resume() once all |
168 | drivers have ack'd link_reset(). |
169 | |
170 | Result codes: |
171 | (identical to mmio_enabled) |
172 | |
173 | >>> The current ppc64 implementation does not implement this callback. |
174 | |
175 | 4) slot_reset() |
176 | |
177 | This is called after the slot has been soft or hard reset by the |
178 | platform. A soft reset consists of asserting the adapter #RST line |
179 | and then restoring the PCI BARs and PCI configuration header. If the |
180 | platform supports PCI hotplug, then it might instead perform a hard |
181 | reset by toggling power on the slot off/on. This call gives drivers |
182 | the chance to re-initialize the hardware (re-download firmware, etc.), |
183 | but drivers shouldn't restart normal I/O processing operations at |
184 | this point. (See note about interrupts; interrupts aren't guaranteed |
185 | to be delivered until the resume() callback has been called). If all |
186 | device drivers report success on this callback, the patform will call |
187 | resume() to complete the error handling and let the driver restart |
188 | normal I/O processing. |
189 | |
190 | A driver can still return a critical failure for this function if |
191 | it can't get the device operational after reset. If the platform |
192 | previously tried a soft reset, it migh now try a hard reset (power |
193 | cycle) and then call slot_reset() again. It the device still can't |
194 | be recovered, there is nothing more that can be done; the platform |
195 | will typically report a "permanent failure" in such a case. The |
196 | device will be considered "dead" in this case. |
197 | |
198 | Result codes: |
199 | - PCIERR_RESULT_DISCONNECT |
200 | Same as above. |
201 | |
202 | >>> The current ppc64 implementation does not try a power-cycle reset |
203 | >>> if the driver returned PCIERR_RESULT_DISCONNECT. However, it should. |
204 | |
205 | 5) resume() |
206 | |
207 | This is called if all drivers on the segment have returned |
208 | PCIERR_RESULT_RECOVERED from one of the 3 prevous callbacks. |
209 | That basically tells the driver to restart activity, tht everything |
210 | is back and running. No result code is taken into account here. If |
211 | a new error happens, it will restart a new error handling process. |
212 | |
213 | That's it. I think this covers all the possibilities. The way those |
214 | callbacks are called is platform policy. A platform with no slot reset |
215 | capability for example may want to just "ignore" drivers that can't |
216 | recover (disconnect them) and try to let other cards on the same segment |
217 | recover. Keep in mind that in most real life cases, though, there will |
218 | be only one driver per segment. |
219 | |
220 | Now, there is a note about interrupts. If you get an interrupt and your |
221 | device is dead or has been isolated, there is a problem :) |
222 | |
223 | After much thinking, I decided to leave that to the platform. That is, |
224 | the recovery API only precies that: |
225 | |
226 | - There is no guarantee that interrupt delivery can proceed from any |
227 | device on the segment starting from the error detection and until the |
228 | restart callback is sent, at which point interrupts are expected to be |
229 | fully operational. |
230 | |
231 | - There is no guarantee that interrupt delivery is stopped, that is, ad |
232 | river that gets an interrupts after detecting an error, or that detects |
233 | and error within the interrupt handler such that it prevents proper |
234 | ack'ing of the interrupt (and thus removal of the source) should just |
235 | return IRQ_NOTHANDLED. It's up to the platform to deal with taht |
236 | condition, typically by masking the irq source during the duration of |
237 | the error handling. It is expected that the platform "knows" which |
238 | interrupts are routed to error-management capable slots and can deal |
239 | with temporarily disabling that irq number during error processing (this |
240 | isn't terribly complex). That means some IRQ latency for other devices |
241 | sharing the interrupt, but there is simply no other way. High end |
242 | platforms aren't supposed to share interrupts between many devices |
243 | anyway :) |
244 | |
245 | |
246 | Revised: 31 May 2005 Linas Vepstas <linas@austin.ibm.com> |