Commit | Line | Data |
---|---|---|
35243421 CL |
1 | Short users guide for SLUB |
2 | -------------------------- | |
3 | ||
35243421 CL |
4 | The basic philosophy of SLUB is very different from SLAB. SLAB |
5 | requires rebuilding the kernel to activate debug options for all | |
c1aee215 | 6 | slab caches. SLUB always includes full debugging but it is off by default. |
35243421 CL |
7 | SLUB can enable debugging only for selected slabs in order to avoid |
8 | an impact on overall system performance which may make a bug more | |
9 | difficult to find. | |
10 | ||
11 | In order to switch debugging on one can add a option "slub_debug" | |
12 | to the kernel command line. That will enable full debugging for | |
13 | all slabs. | |
14 | ||
15 | Typically one would then use the "slabinfo" command to get statistical | |
16 | data and perform operation on the slabs. By default slabinfo only lists | |
17 | slabs that have data in them. See "slabinfo -h" for more options when | |
18 | running the command. slabinfo can be compiled with | |
19 | ||
9fe49611 | 20 | gcc -o slabinfo tools/vm/slabinfo.c |
35243421 CL |
21 | |
22 | Some of the modes of operation of slabinfo require that slub debugging | |
23 | be enabled on the command line. F.e. no tracking information will be | |
24 | available without debugging on and validation can only partially | |
25 | be performed if debugging was not switched on. | |
26 | ||
27 | Some more sophisticated uses of slub_debug: | |
28 | ------------------------------------------- | |
29 | ||
30 | Parameters may be given to slub_debug. If none is specified then full | |
31 | debugging is enabled. Format: | |
32 | ||
33 | slub_debug=<Debug-Options> Enable options for all slabs | |
34 | slub_debug=<Debug-Options>,<slab name> | |
35 | Enable options only for select slabs | |
36 | ||
37 | Possible debug options are | |
38 | F Sanity checks on (enables SLAB_DEBUG_FREE. Sorry | |
39 | SLAB legacy issues) | |
40 | Z Red zoning | |
41 | P Poisoning (object and padding) | |
42 | U User tracking (free and alloc) | |
43 | T Trace (please only use on single slabs) | |
4c13dd3b | 44 | A Toggle failslab filter mark for the cache |
fa5ec8a1 DR |
45 | O Switch debugging off for caches that would have |
46 | caused higher minimum slab orders | |
f0630fff CL |
47 | - Switch all debugging off (useful if the kernel is |
48 | configured with CONFIG_SLUB_DEBUG_ON) | |
35243421 CL |
49 | |
50 | F.e. in order to boot just with sanity checks and red zoning one would specify: | |
51 | ||
52 | slub_debug=FZ | |
53 | ||
54 | Trying to find an issue in the dentry cache? Try | |
55 | ||
989a7241 | 56 | slub_debug=,dentry |
35243421 CL |
57 | |
58 | to only enable debugging on the dentry cache. | |
59 | ||
60 | Red zoning and tracking may realign the slab. We can just apply sanity checks | |
61 | to the dentry cache with | |
62 | ||
989a7241 | 63 | slub_debug=F,dentry |
35243421 | 64 | |
fa5ec8a1 DR |
65 | Debugging options may require the minimum possible slab order to increase as |
66 | a result of storing the metadata (for example, caches with PAGE_SIZE object | |
67 | sizes). This has a higher liklihood of resulting in slab allocation errors | |
68 | in low memory situations or if there's high fragmentation of memory. To | |
69 | switch off debugging for such caches by default, use | |
70 | ||
71 | slub_debug=O | |
72 | ||
35243421 CL |
73 | In case you forgot to enable debugging on the kernel command line: It is |
74 | possible to enable debugging manually when the kernel is up. Look at the | |
75 | contents of: | |
76 | ||
081248de | 77 | /sys/kernel/slab/<slab name>/ |
35243421 CL |
78 | |
79 | Look at the writable files. Writing 1 to them will enable the | |
80 | corresponding debug option. All options can be set on a slab that does | |
81 | not contain objects. If the slab already contains objects then sanity checks | |
82 | and tracing may only be enabled. The other options may cause the realignment | |
83 | of objects. | |
84 | ||
85 | Careful with tracing: It may spew out lots of information and never stop if | |
86 | used on the wrong slab. | |
87 | ||
c1aee215 | 88 | Slab merging |
35243421 CL |
89 | ------------ |
90 | ||
c1aee215 | 91 | If no debug options are specified then SLUB may merge similar slabs together |
35243421 CL |
92 | in order to reduce overhead and increase cache hotness of objects. |
93 | slabinfo -a displays which slabs were merged together. | |
94 | ||
c1aee215 CL |
95 | Slab validation |
96 | --------------- | |
97 | ||
98 | SLUB can validate all object if the kernel was booted with slub_debug. In | |
99 | order to do so you must have the slabinfo tool. Then you can do | |
100 | ||
101 | slabinfo -v | |
102 | ||
103 | which will test all objects. Output will be generated to the syslog. | |
104 | ||
105 | This also works in a more limited way if boot was without slab debug. | |
106 | In that case slabinfo -v simply tests all reachable objects. Usually | |
107 | these are in the cpu slabs and the partial slabs. Full slabs are not | |
108 | tracked by SLUB in a non debug situation. | |
109 | ||
35243421 CL |
110 | Getting more performance |
111 | ------------------------ | |
112 | ||
113 | To some degree SLUB's performance is limited by the need to take the | |
114 | list_lock once in a while to deal with partial slabs. That overhead is | |
115 | governed by the order of the allocation for each slab. The allocations | |
116 | can be influenced by kernel parameters: | |
117 | ||
c1aee215 | 118 | slub_min_objects=x (default 4) |
35243421 | 119 | slub_min_order=x (default 0) |
25f4379b | 120 | slub_max_order=x (default 3 (PAGE_ALLOC_COSTLY_ORDER)) |
35243421 CL |
121 | |
122 | slub_min_objects allows to specify how many objects must at least fit | |
123 | into one slab in order for the allocation order to be acceptable. | |
124 | In general slub will be able to perform this number of allocations | |
125 | on a slab without consulting centralized resources (list_lock) where | |
126 | contention may occur. | |
127 | ||
128 | slub_min_order specifies a minim order of slabs. A similar effect like | |
129 | slub_min_objects. | |
130 | ||
131 | slub_max_order specified the order at which slub_min_objects should no | |
132 | longer be checked. This is useful to avoid SLUB trying to generate | |
133 | super large order pages to fit slub_min_objects of a slab cache with | |
888a214d SG |
134 | large object sizes into one high order page. Setting command line |
135 | parameter debug_guardpage_minorder=N (N > 0), forces setting | |
136 | slub_max_order to 0, what cause minimum possible order of slabs | |
137 | allocation. | |
35243421 | 138 | |
c1aee215 CL |
139 | SLUB Debug output |
140 | ----------------- | |
141 | ||
142 | Here is a sample of slub debug output: | |
143 | ||
24922684 CL |
144 | ==================================================================== |
145 | BUG kmalloc-8: Redzone overwritten | |
146 | -------------------------------------------------------------------- | |
147 | ||
148 | INFO: 0xc90f6d28-0xc90f6d2b. First byte 0x00 instead of 0xcc | |
149 | INFO: Slab 0xc528c530 flags=0x400000c3 inuse=61 fp=0xc90f6d58 | |
150 | INFO: Object 0xc90f6d20 @offset=3360 fp=0xc90f6d58 | |
151 | INFO: Allocated in get_modalias+0x61/0xf5 age=53 cpu=1 pid=554 | |
152 | ||
153 | Bytes b4 0xc90f6d10: 00 00 00 00 00 00 00 00 5a 5a 5a 5a 5a 5a 5a 5a ........ZZZZZZZZ | |
154 | Object 0xc90f6d20: 31 30 31 39 2e 30 30 35 1019.005 | |
155 | Redzone 0xc90f6d28: 00 cc cc cc . | |
156 | Padding 0xc90f6d50: 5a 5a 5a 5a 5a 5a 5a 5a ZZZZZZZZ | |
157 | ||
c1aee215 CL |
158 | [<c010523d>] dump_trace+0x63/0x1eb |
159 | [<c01053df>] show_trace_log_lvl+0x1a/0x2f | |
160 | [<c010601d>] show_trace+0x12/0x14 | |
161 | [<c0106035>] dump_stack+0x16/0x18 | |
162 | [<c017e0fa>] object_err+0x143/0x14b | |
163 | [<c017e2cc>] check_object+0x66/0x234 | |
164 | [<c017eb43>] __slab_free+0x239/0x384 | |
165 | [<c017f446>] kfree+0xa6/0xc6 | |
166 | [<c02e2335>] get_modalias+0xb9/0xf5 | |
167 | [<c02e23b7>] dmi_dev_uevent+0x27/0x3c | |
168 | [<c027866a>] dev_uevent+0x1ad/0x1da | |
169 | [<c0205024>] kobject_uevent_env+0x20a/0x45b | |
170 | [<c020527f>] kobject_uevent+0xa/0xf | |
171 | [<c02779f1>] store_uevent+0x4f/0x58 | |
172 | [<c027758e>] dev_attr_store+0x29/0x2f | |
173 | [<c01bec4f>] sysfs_write_file+0x16e/0x19c | |
174 | [<c0183ba7>] vfs_write+0xd1/0x15a | |
175 | [<c01841d7>] sys_write+0x3d/0x72 | |
176 | [<c0104112>] sysenter_past_esp+0x5f/0x99 | |
177 | [<b7f7b410>] 0xb7f7b410 | |
178 | ======================= | |
c1aee215 | 179 | |
24922684 | 180 | FIX kmalloc-8: Restoring Redzone 0xc90f6d28-0xc90f6d2b=0xcc |
c1aee215 | 181 | |
24922684 CL |
182 | If SLUB encounters a corrupted object (full detection requires the kernel |
183 | to be booted with slub_debug) then the following output will be dumped | |
184 | into the syslog: | |
c1aee215 | 185 | |
24922684 | 186 | 1. Description of the problem encountered |
c1aee215 CL |
187 | |
188 | This will be a message in the system log starting with | |
189 | ||
24922684 CL |
190 | =============================================== |
191 | BUG <slab cache affected>: <What went wrong> | |
192 | ----------------------------------------------- | |
c1aee215 | 193 | |
24922684 CL |
194 | INFO: <corruption start>-<corruption_end> <more info> |
195 | INFO: Slab <address> <slab information> | |
196 | INFO: Object <address> <object information> | |
197 | INFO: Allocated in <kernel function> age=<jiffies since alloc> cpu=<allocated by | |
198 | cpu> pid=<pid of the process> | |
199 | INFO: Freed in <kernel function> age=<jiffies since free> cpu=<freed by cpu> | |
200 | pid=<pid of the process> | |
c1aee215 | 201 | |
24922684 CL |
202 | (Object allocation / free information is only available if SLAB_STORE_USER is |
203 | set for the slab. slub_debug sets that option) | |
c1aee215 | 204 | |
24922684 | 205 | 2. The object contents if an object was involved. |
c1aee215 | 206 | |
24922684 | 207 | Various types of lines can follow the BUG SLUB line: |
c1aee215 CL |
208 | |
209 | Bytes b4 <address> : <bytes> | |
24922684 | 210 | Shows a few bytes before the object where the problem was detected. |
c1aee215 CL |
211 | Can be useful if the corruption does not stop with the start of the |
212 | object. | |
213 | ||
214 | Object <address> : <bytes> | |
215 | The bytes of the object. If the object is inactive then the bytes | |
24922684 | 216 | typically contain poison values. Any non-poison value shows a |
c1aee215 CL |
217 | corruption by a write after free. |
218 | ||
219 | Redzone <address> : <bytes> | |
24922684 | 220 | The Redzone following the object. The Redzone is used to detect |
c1aee215 CL |
221 | writes after the object. All bytes should always have the same |
222 | value. If there is any deviation then it is due to a write after | |
223 | the object boundary. | |
224 | ||
24922684 CL |
225 | (Redzone information is only available if SLAB_RED_ZONE is set. |
226 | slub_debug sets that option) | |
c1aee215 | 227 | |
24922684 | 228 | Padding <address> : <bytes> |
c1aee215 CL |
229 | Unused data to fill up the space in order to get the next object |
230 | properly aligned. In the debug case we make sure that there are | |
24922684 | 231 | at least 4 bytes of padding. This allows the detection of writes |
c1aee215 CL |
232 | before the object. |
233 | ||
24922684 CL |
234 | 3. A stackdump |
235 | ||
236 | The stackdump describes the location where the error was detected. The cause | |
237 | of the corruption is may be more likely found by looking at the function that | |
238 | allocated or freed the object. | |
239 | ||
240 | 4. Report on how the problem was dealt with in order to ensure the continued | |
241 | operation of the system. | |
242 | ||
243 | These are messages in the system log beginning with | |
244 | ||
245 | FIX <slab cache affected>: <corrective action taken> | |
246 | ||
247 | In the above sample SLUB found that the Redzone of an active object has | |
248 | been overwritten. Here a string of 8 characters was written into a slab that | |
249 | has the length of 8 characters. However, a 8 character string needs a | |
250 | terminating 0. That zero has overwritten the first byte of the Redzone field. | |
251 | After reporting the details of the issue encountered the FIX SLUB message | |
e02f0e86 | 252 | tells us that SLUB has restored the Redzone to its proper value and then |
24922684 CL |
253 | system operations continue. |
254 | ||
255 | Emergency operations: | |
256 | --------------------- | |
257 | ||
258 | Minimal debugging (sanity checks alone) can be enabled by booting with | |
259 | ||
260 | slub_debug=F | |
261 | ||
262 | This will be generally be enough to enable the resiliency features of slub | |
263 | which will keep the system running even if a bad kernel component will | |
264 | keep corrupting objects. This may be important for production systems. | |
265 | Performance will be impacted by the sanity checks and there will be a | |
266 | continual stream of error messages to the syslog but no additional memory | |
267 | will be used (unlike full debugging). | |
268 | ||
269 | No guarantees. The kernel component still needs to be fixed. Performance | |
270 | may be optimized further by locating the slab that experiences corruption | |
271 | and enabling debugging only for that cache | |
272 | ||
273 | I.e. | |
274 | ||
275 | slub_debug=F,dentry | |
276 | ||
277 | If the corruption occurs by writing after the end of the object then it | |
278 | may be advisable to enable a Redzone to avoid corrupting the beginning | |
279 | of other objects. | |
280 | ||
281 | slub_debug=FZ,dentry | |
c1aee215 | 282 | |
cde53535 | 283 | Christoph Lameter, May 30, 2007 |