Commit | Line | Data |
---|---|---|
b089f4a6 VG |
1 | Documentation for kdump - the kexec-based crash dumping solution |
2 | ================================================================ | |
3 | ||
4 | DESIGN | |
5 | ====== | |
6 | ||
7 | Kdump uses kexec to reboot to a second kernel whenever a dump needs to be taken. | |
8 | This second kernel is booted with very little memory. The first kernel reserves | |
9 | the section of memory that the second kernel uses. This ensures that on-going | |
10 | DMA from the first kernel does not corrupt the second kernel. | |
11 | ||
12 | All the necessary information about Core image is encoded in ELF format and | |
13 | stored in reserved area of memory before crash. Physical address of start of | |
14 | ELF header is passed to new kernel through command line parameter elfcorehdr=. | |
15 | ||
16 | On i386, the first 640 KB of physical memory is needed to boot, irrespective | |
17 | of where the kernel loads. Hence, this region is backed up by kexec just before | |
18 | rebooting into the new kernel. | |
19 | ||
20 | In the second kernel, "old memory" can be accessed in two ways. | |
21 | ||
22 | - The first one is through a /dev/oldmem device interface. A capture utility | |
23 | can read the device file and write out the memory in raw format. This is raw | |
24 | dump of memory and analysis/capture tool should be intelligent enough to | |
25 | determine where to look for the right information. ELF headers (elfcorehdr=) | |
26 | can become handy here. | |
27 | ||
28 | - The second interface is through /proc/vmcore. This exports the dump as an ELF | |
29 | format file which can be written out using any file copy command | |
30 | (cp, scp, etc). Further, gdb can be used to perform limited debugging on | |
31 | the dump file. This method ensures methods ensure that there is correct | |
32 | ordering of the dump pages (corresponding to the first 640 KB that has been | |
33 | relocated). | |
34 | ||
35 | SETUP | |
36 | ===== | |
37 | ||
38 | 1) Download http://www.xmission.com/~ebiederm/files/kexec/kexec-tools-1.101.tar.gz | |
39 | and apply http://lse.sourceforge.net/kdump/patches/kexec-tools-1.101-kdump.patch | |
40 | and after that build the source. | |
41 | ||
42 | 2) Download and build the appropriate (latest) kexec/kdump (-mm) kernel | |
43 | patchset and apply it to the vanilla kernel tree. | |
44 | ||
45 | Two kernels need to be built in order to get this feature working. | |
46 | ||
47 | A) First kernel: | |
48 | a) Enable "kexec system call" feature (in Processor type and features). | |
49 | CONFIG_KEXEC=y | |
50 | b) This kernel's physical load address should be the default value of | |
51 | 0x100000 (0x100000, 1 MB) (in Processor type and features). | |
52 | CONFIG_PHYSICAL_START=0x100000 | |
53 | c) Enable "sysfs file system support" (in Pseudo filesystems). | |
54 | CONFIG_SYSFS=y | |
55 | d) Boot into first kernel with the command line parameter "crashkernel=Y@X". | |
56 | Use appropriate values for X and Y. Y denotes how much memory to reserve | |
57 | for the second kernel, and X denotes at what physical address the reserved | |
58 | memory section starts. For example: "crashkernel=64M@16M". | |
59 | ||
60 | B) Second kernel: | |
61 | a) Enable "kernel crash dumps" feature (in Processor type and features). | |
62 | CONFIG_CRASH_DUMP=y | |
63 | b) Specify a suitable value for "Physical address where the kernel is | |
64 | loaded" (in Processor type and features). Typically this value | |
65 | should be same as X (See option d) above, e.g., 16 MB or 0x1000000. | |
66 | CONFIG_PHYSICAL_START=0x1000000 | |
67 | c) Enable "/proc/vmcore support" (Optional, in Pseudo filesystems). | |
68 | CONFIG_PROC_VMCORE=y | |
69 | ||
70 | Note: Options a) and b) depend upon "Configure standard kernel features | |
71 | (for small systems)" (under General setup). | |
72 | Option a) also depends on CONFIG_HIGHMEM (under Processor | |
73 | type and features). | |
74 | Both option a) and b) are under "Processor type and features". | |
75 | ||
76 | 3) Boot into the first kernel. You are now ready to try out kexec-based crash | |
77 | dumps. | |
78 | ||
79 | 4) Load the second kernel to be booted using: | |
80 | ||
81 | kexec -p <second-kernel> --crash-dump --args-linux --append="root=<root-dev> | |
82 | maxcpus=1 init 1" | |
83 | ||
84 | Note: i) <second-kernel> has to be a vmlinux image. bzImage will not work, | |
85 | as of now. | |
86 | ii) By default ELF headers are stored in ELF32 format (for i386). This | |
87 | is sufficient to represent the physical memory up to 4GB. To store | |
88 | headers in ELF64 format, specifiy "--elf64-core-headers" on the | |
89 | kexec command line additionally. | |
90 | iii) For now (or until it is fixed), it's best to build the | |
91 | second-kernel without multi-processor support, i.e., make it | |
92 | a uniprocessor kernel. | |
93 | ||
94 | 5) System reboots into the second kernel when a panic occurs. A module can be | |
95 | written to force the panic, for testing purposes. | |
96 | ||
97 | 6) Write out the dump file using | |
98 | ||
99 | cp /proc/vmcore <dump-file> | |
100 | ||
101 | Dump memory can also be accessed as a /dev/oldmem device for a linear/raw | |
102 | view. To create the device, type: | |
103 | ||
104 | mknod /dev/oldmem c 1 12 | |
105 | ||
106 | Use "dd" with suitable options for count, bs and skip to access specific | |
107 | portions of the dump. | |
108 | ||
109 | Entire memory: dd if=/dev/oldmem of=oldmem.001 | |
110 | ||
111 | ANALYSIS | |
112 | ======== | |
113 | ||
114 | Limited analysis can be done using gdb on the dump file copied out of | |
115 | /proc/vmcore. Use vmlinux built with -g and run | |
116 | ||
117 | gdb vmlinux <dump-file> | |
118 | ||
119 | Stack trace for the task on processor 0, register display, memory display | |
120 | work fine. | |
121 | ||
122 | Note: gdb cannot analyse core files generated in ELF64 format for i386. | |
123 | ||
124 | TODO | |
125 | ==== | |
126 | ||
127 | 1) Provide a kernel pages filtering mechanism so that core file size is not | |
128 | insane on systems having huge memory banks. | |
129 | 2) Modify "crash" tool to make it recognize this dump. | |
130 | ||
131 | CONTACT | |
132 | ======= | |
133 | ||
134 | Hariprasad Nellitheertha - hari at in dot ibm dot com | |
135 | Vivek Goyal (vgoyal@in.ibm.com) |