Commit | Line | Data |
---|---|---|
f212ec4b BK |
1 | |
2 | Using physical DMA provided by OHCI-1394 FireWire controllers for debugging | |
3 | --------------------------------------------------------------------------- | |
4 | ||
5 | Introduction | |
6 | ------------ | |
7 | ||
8 | Basically all FireWire controllers which are in use today are compliant | |
9 | to the OHCI-1394 specification which defines the controller to be a PCI | |
10 | bus master which uses DMA to offload data transfers from the CPU and has | |
11 | a "Physical Response Unit" which executes specific requests by employing | |
12 | PCI-Bus master DMA after applying filters defined by the OHCI-1394 driver. | |
13 | ||
14 | Once properly configured, remote machines can send these requests to | |
15 | ask the OHCI-1394 controller to perform read and write requests on | |
16 | physical system memory and, for read requests, send the result of | |
17 | the physical memory read back to the requester. | |
18 | ||
19 | With that, it is possible to debug issues by reading interesting memory | |
20 | locations such as buffers like the printk buffer or the process table. | |
21 | ||
22 | Retrieving a full system memory dump is also possible over the FireWire, | |
23 | using data transfer rates in the order of 10MB/s or more. | |
24 | ||
25 | Memory access is currently limited to the low 4G of physical address | |
26 | space which can be a problem on IA64 machines where memory is located | |
27 | mostly above that limit, but it is rarely a problem on more common | |
28 | hardware such as hardware based on x86, x86-64 and PowerPC. | |
29 | ||
30 | Together with a early initialization of the OHCI-1394 controller for debugging, | |
31 | this facility proved most useful for examining long debugs logs in the printk | |
32 | buffer on to debug early boot problems in areas like ACPI where the system | |
33 | fails to boot and other means for debugging (serial port) are either not | |
34 | available (notebooks) or too slow for extensive debug information (like ACPI). | |
35 | ||
36 | Drivers | |
37 | ------- | |
38 | ||
09d7328e SR |
39 | The ohci1394 driver in drivers/ieee1394 initializes the OHCI-1394 controllers |
40 | to a working state and enables physical DMA by default for all remote nodes. | |
41 | This can be turned off by ohci1394's module parameter phys_dma=0. | |
42 | ||
43 | The alternative firewire-ohci driver in drivers/firewire uses filtered physical | |
44 | DMA, hence is not yet suitable for remote debugging. | |
45 | ||
46 | Because ohci1394 depends on the PCI enumeration to be completed, an | |
47 | initialization routine which runs pretty early (long before console_init() | |
f212ec4b BK |
48 | which makes the printk buffer appear on the console can be called) was written. |
49 | ||
50 | To activate it, enable CONFIG_PROVIDE_OHCI1394_DMA_INIT (Kernel hacking menu: | |
51 | Provide code for enabling DMA over FireWire early on boot) and pass the | |
52 | parameter "ohci1394_dma=early" to the recompiled kernel on boot. | |
53 | ||
54 | Tools | |
55 | ----- | |
56 | ||
57 | firescope - Originally developed by Benjamin Herrenschmidt, Andi Kleen ported | |
58 | it from PowerPC to x86 and x86_64 and added functionality, firescope can now | |
59 | be used to view the printk buffer of a remote machine, even with live update. | |
60 | ||
61 | Bernhard Kaindl enhanced firescope to support accessing 64-bit machines | |
62 | from 32-bit firescope and vice versa: | |
63 | - ftp://ftp.suse.de/private/bk/firewire/tools/firescope-0.2.2.tar.bz2 | |
64 | ||
65 | and he implemented fast system dump (alpha version - read README.txt): | |
66 | - ftp://ftp.suse.de/private/bk/firewire/tools/firedump-0.1.tar.bz2 | |
67 | ||
68 | There is also a gdb proxy for firewire which allows to use gdb to access | |
69 | data which can be referenced from symbols found by gdb in vmlinux: | |
70 | - ftp://ftp.suse.de/private/bk/firewire/tools/fireproxy-0.33.tar.bz2 | |
71 | ||
72 | The latest version of this gdb proxy (fireproxy-0.34) can communicate (not | |
73 | yet stable) with kgdb over an memory-based communication module (kgdbom). | |
74 | ||
75 | Getting Started | |
76 | --------------- | |
77 | ||
78 | The OHCI-1394 specification regulates that the OHCI-1394 controller must | |
79 | disable all physical DMA on each bus reset. | |
80 | ||
81 | This means that if you want to debug an issue in a system state where | |
82 | interrupts are disabled and where no polling of the OHCI-1394 controller | |
83 | for bus resets takes place, you have to establish any FireWire cable | |
84 | connections and fully initialize all FireWire hardware __before__ the | |
85 | system enters such state. | |
86 | ||
87 | Step-by-step instructions for using firescope with early OHCI initialization: | |
88 | ||
89 | 1) Verify that your hardware is supported: | |
90 | ||
91 | Load the ohci1394 or the fw-ohci module and check your kernel logs. | |
92 | You should see a line similar to | |
93 | ||
94 | ohci1394: fw-host0: OHCI-1394 1.1 (PCI): IRQ=[18] MMIO=[fe9ff800-fe9fffff] | |
95 | ... Max Packet=[2048] IR/IT contexts=[4/8] | |
96 | ||
97 | when loading the driver. If you have no supported controller, many PCI, | |
98 | CardBus and even some Express cards which are fully compliant to OHCI-1394 | |
99 | specification are available. If it requires no driver for Windows operating | |
100 | systems, it most likely is. Only specialized shops have cards which are not | |
101 | compliant, they are based on TI PCILynx chips and require drivers for Win- | |
102 | dows operating systems. | |
103 | ||
104 | 2) Establish a working FireWire cable connection: | |
105 | ||
106 | Any FireWire cable, as long at it provides electrically and mechanically | |
107 | stable connection and has matching connectors (there are small 4-pin and | |
108 | large 6-pin FireWire ports) will do. | |
109 | ||
110 | If an driver is running on both machines you should see a line like | |
111 | ||
112 | ieee1394: Node added: ID:BUS[0-01:1023] GUID[0090270001b84bba] | |
113 | ||
114 | on both machines in the kernel log when the cable is plugged in | |
115 | and connects the two machines. | |
116 | ||
117 | 3) Test physical DMA using firescope: | |
118 | ||
119 | On the debug host, | |
120 | - load the raw1394 module, | |
121 | - make sure that /dev/raw1394 is accessible, | |
122 | then start firescope: | |
123 | ||
124 | $ firescope | |
125 | Port 0 (ohci1394) opened, 2 nodes detected | |
126 | ||
127 | FireScope | |
128 | --------- | |
129 | Target : <unspecified> | |
130 | Gen : 1 | |
131 | [Ctrl-T] choose target | |
132 | [Ctrl-H] this menu | |
133 | [Ctrl-Q] quit | |
134 | ||
135 | ------> Press Ctrl-T now, the output should be similar to: | |
136 | ||
137 | 2 nodes available, local node is: 0 | |
138 | 0: ffc0, uuid: 00000000 00000000 [LOCAL] | |
139 | 1: ffc1, uuid: 00279000 ba4bb801 | |
140 | ||
141 | Besides the [LOCAL] node, it must show another node without error message. | |
142 | ||
143 | 4) Prepare for debugging with early OHCI-1394 initialization: | |
144 | ||
145 | 4.1) Kernel compilation and installation on debug target | |
146 | ||
147 | Compile the kernel to be debugged with CONFIG_PROVIDE_OHCI1394_DMA_INIT | |
148 | (Kernel hacking: Provide code for enabling DMA over FireWire early on boot) | |
149 | enabled and install it on the machine to be debugged (debug target). | |
150 | ||
151 | 4.2) Transfer the System.map of the debugged kernel to the debug host | |
152 | ||
153 | Copy the System.map of the kernel be debugged to the debug host (the host | |
154 | which is connected to the debugged machine over the FireWire cable). | |
155 | ||
156 | 5) Retrieving the printk buffer contents: | |
157 | ||
158 | With the FireWire cable connected, the OHCI-1394 driver on the debugging | |
159 | host loaded, reboot the debugged machine, booting the kernel which has | |
160 | CONFIG_PROVIDE_OHCI1394_DMA_INIT enabled, with the option ohci1394_dma=early. | |
161 | ||
162 | Then, on the debugging host, run firescope, for example by using -A: | |
163 | ||
164 | firescope -A System.map-of-debug-target-kernel | |
165 | ||
166 | Note: -A automatically attaches to the first non-local node. It only works | |
167 | reliably if only connected two machines are connected using FireWire. | |
168 | ||
169 | After having attached to the debug target, press Ctrl-D to view the | |
170 | complete printk buffer or Ctrl-U to enter auto update mode and get an | |
171 | updated live view of recent kernel messages logged on the debug target. | |
172 | ||
173 | Call "firescope -h" to get more information on firescope's options. | |
174 | ||
175 | Notes | |
176 | ----- | |
177 | Documentation and specifications: ftp://ftp.suse.de/private/bk/firewire/docs | |
178 | ||
179 | FireWire is a trademark of Apple Inc. - for more information please refer to: | |
180 | http://en.wikipedia.org/wiki/FireWire |