Commit | Line | Data |
---|---|---|
b089f4a6 VG |
1 | Documentation for kdump - the kexec-based crash dumping solution |
2 | ================================================================ | |
3 | ||
4 | DESIGN | |
5 | ====== | |
6 | ||
a7e670d8 MS |
7 | Kdump uses kexec to reboot to a second kernel whenever a dump needs to be |
8 | taken. This second kernel is booted with very little memory. The first kernel | |
9 | reserves the section of memory that the second kernel uses. This ensures that | |
10 | on-going DMA from the first kernel does not corrupt the second kernel. | |
b089f4a6 VG |
11 | |
12 | All the necessary information about Core image is encoded in ELF format and | |
13 | stored in reserved area of memory before crash. Physical address of start of | |
14 | ELF header is passed to new kernel through command line parameter elfcorehdr=. | |
15 | ||
16 | On i386, the first 640 KB of physical memory is needed to boot, irrespective | |
17 | of where the kernel loads. Hence, this region is backed up by kexec just before | |
18 | rebooting into the new kernel. | |
19 | ||
20 | In the second kernel, "old memory" can be accessed in two ways. | |
21 | ||
22 | - The first one is through a /dev/oldmem device interface. A capture utility | |
23 | can read the device file and write out the memory in raw format. This is raw | |
24 | dump of memory and analysis/capture tool should be intelligent enough to | |
25 | determine where to look for the right information. ELF headers (elfcorehdr=) | |
26 | can become handy here. | |
27 | ||
28 | - The second interface is through /proc/vmcore. This exports the dump as an ELF | |
29 | format file which can be written out using any file copy command | |
30 | (cp, scp, etc). Further, gdb can be used to perform limited debugging on | |
31 | the dump file. This method ensures methods ensure that there is correct | |
32 | ordering of the dump pages (corresponding to the first 640 KB that has been | |
33 | relocated). | |
34 | ||
35 | SETUP | |
36 | ===== | |
37 | ||
a7e670d8 MS |
38 | 1) Download the upstream kexec-tools userspace package from |
39 | http://www.xmission.com/~ebiederm/files/kexec/kexec-tools-1.101.tar.gz. | |
b089f4a6 | 40 | |
a7e670d8 MS |
41 | Apply the latest consolidated kdump patch on top of kexec-tools-1.101 |
42 | from http://lse.sourceforge.net/kdump/. This arrangment has been made | |
43 | till all the userspace patches supporting kdump are integrated with | |
44 | upstream kexec-tools userspace. | |
b089f4a6 | 45 | |
a7e670d8 | 46 | 2) Download and build the appropriate (2.6.13-rc1 onwards) vanilla kernels. |
b089f4a6 | 47 | Two kernels need to be built in order to get this feature working. |
a7e670d8 MS |
48 | Following are the steps to properly configure the two kernels specific |
49 | to kexec and kdump features: | |
b089f4a6 | 50 | |
a7e670d8 MS |
51 | A) First kernel or regular kernel: |
52 | ---------------------------------- | |
b089f4a6 | 53 | a) Enable "kexec system call" feature (in Processor type and features). |
a7e670d8 MS |
54 | CONFIG_KEXEC=y |
55 | b) Enable "sysfs file system support" (in Pseudo filesystems). | |
56 | CONFIG_SYSFS=y | |
57 | c) make | |
b089f4a6 VG |
58 | d) Boot into first kernel with the command line parameter "crashkernel=Y@X". |
59 | Use appropriate values for X and Y. Y denotes how much memory to reserve | |
a7e670d8 MS |
60 | for the second kernel, and X denotes at what physical address the |
61 | reserved memory section starts. For example: "crashkernel=64M@16M". | |
62 | ||
63 | ||
64 | B) Second kernel or dump capture kernel: | |
65 | --------------------------------------- | |
66 | a) For i386 architecture enable Highmem support | |
67 | CONFIG_HIGHMEM=y | |
68 | b) Enable "kernel crash dumps" feature (under "Processor type and features") | |
69 | CONFIG_CRASH_DUMP=y | |
70 | c) Make sure a suitable value for "Physical address where the kernel is | |
71 | loaded" (under "Processor type and features"). By default this value | |
72 | is 0x1000000 (16MB) and it should be same as X (See option d above), | |
73 | e.g., 16 MB or 0x1000000. | |
74 | CONFIG_PHYSICAL_START=0x1000000 | |
75 | d) Enable "/proc/vmcore support" (Optional, under "Pseudo filesystems"). | |
76 | CONFIG_PROC_VMCORE=y | |
77 | ||
78 | 3) After booting to regular kernel or first kernel, load the second kernel | |
79 | using the following command: | |
b089f4a6 | 80 | |
952b6492 | 81 | kexec -p <second-kernel> --args-linux --elf32-core-headers |
a7e670d8 MS |
82 | --append="root=<root-dev> init 1 irqpoll maxcpus=1" |
83 | ||
84 | Notes: | |
85 | ====== | |
86 | i) <second-kernel> has to be a vmlinux image ie uncompressed elf image. | |
87 | bzImage will not work, as of now. | |
88 | ii) --args-linux has to be speicfied as if kexec it loading an elf image, | |
89 | it needs to know that the arguments supplied are of linux type. | |
90 | iii) By default ELF headers are stored in ELF64 format to support systems | |
91 | with more than 4GB memory. Option --elf32-core-headers forces generation | |
92 | of ELF32 headers. The reason for this option being, as of now gdb can | |
93 | not open vmcore file with ELF64 headers on a 32 bit systems. So ELF32 | |
94 | headers can be used if one has non-PAE systems and hence memory less | |
95 | than 4GB. | |
96 | iv) Specify "irqpoll" as command line parameter. This reduces driver | |
97 | initialization failures in second kernel due to shared interrupts. | |
98 | v) <root-dev> needs to be specified in a format corresponding to the root | |
99 | device name in the output of mount command. | |
100 | vi) If you have built the drivers required to mount root file system as | |
101 | modules in <second-kernel>, then, specify | |
102 | --initrd=<initrd-for-second-kernel>. | |
103 | vii) Specify maxcpus=1 as, if during first kernel run, if panic happens on | |
104 | non-boot cpus, second kernel doesn't seem to be boot up all the cpus. | |
105 | The other option is to always built the second kernel without SMP | |
106 | support ie CONFIG_SMP=n | |
107 | ||
108 | 4) After successfully loading the second kernel as above, if a panic occurs | |
109 | system reboots into the second kernel. A module can be written to force | |
110 | the panic or "ALT-SysRq-c" can be used initiate a crash dump for testing | |
111 | purposes. | |
112 | ||
113 | 5) Once the second kernel has booted, write out the dump file using | |
b089f4a6 VG |
114 | |
115 | cp /proc/vmcore <dump-file> | |
116 | ||
117 | Dump memory can also be accessed as a /dev/oldmem device for a linear/raw | |
118 | view. To create the device, type: | |
119 | ||
120 | mknod /dev/oldmem c 1 12 | |
121 | ||
122 | Use "dd" with suitable options for count, bs and skip to access specific | |
123 | portions of the dump. | |
124 | ||
125 | Entire memory: dd if=/dev/oldmem of=oldmem.001 | |
126 | ||
a7e670d8 | 127 | |
b089f4a6 VG |
128 | ANALYSIS |
129 | ======== | |
b089f4a6 VG |
130 | Limited analysis can be done using gdb on the dump file copied out of |
131 | /proc/vmcore. Use vmlinux built with -g and run | |
132 | ||
133 | gdb vmlinux <dump-file> | |
134 | ||
135 | Stack trace for the task on processor 0, register display, memory display | |
136 | work fine. | |
137 | ||
138 | Note: gdb cannot analyse core files generated in ELF64 format for i386. | |
139 | ||
a7e670d8 MS |
140 | Latest "crash" (crash-4.0-2.18) as available on Dave Anderson's site |
141 | http://people.redhat.com/~anderson/ works well with kdump format. | |
142 | ||
143 | ||
b089f4a6 VG |
144 | TODO |
145 | ==== | |
b089f4a6 VG |
146 | 1) Provide a kernel pages filtering mechanism so that core file size is not |
147 | insane on systems having huge memory banks. | |
a7e670d8 MS |
148 | 2) Relocatable kernel can help in maintaining multiple kernels for crashdump |
149 | and same kernel as the first kernel can be used to capture the dump. | |
150 | ||
b089f4a6 VG |
151 | |
152 | CONTACT | |
153 | ======= | |
b089f4a6 | 154 | Vivek Goyal (vgoyal@in.ibm.com) |
d58831e4 | 155 | Maneesh Soni (maneesh@in.ibm.com) |