Commit | Line | Data |
---|---|---|
1da177e4 LT |
1 | |
2 | The intent of this file is to give a brief summary of hugetlbpage support in | |
3 | the Linux kernel. This support is built on top of multiple page size support | |
4 | that is provided by most modern architectures. For example, i386 | |
5 | architecture supports 4K and 4M (2M in PAE mode) page sizes, ia64 | |
6 | architecture supports multiple page sizes 4K, 8K, 64K, 256K, 1M, 4M, 16M, | |
7 | 256M and ppc64 supports 4K and 16M. A TLB is a cache of virtual-to-physical | |
8 | translations. Typically this is a very scarce resource on processor. | |
9 | Operating systems try to make best use of limited number of TLB resources. | |
10 | This optimization is more critical now as bigger and bigger physical memories | |
11 | (several GBs) are more readily available. | |
12 | ||
13 | Users can use the huge page support in Linux kernel by either using the mmap | |
14 | system call or standard SYSv shared memory system calls (shmget, shmat). | |
15 | ||
5c7ad510 MBY |
16 | First the Linux kernel needs to be built with the CONFIG_HUGETLBFS |
17 | (present under "File systems") and CONFIG_HUGETLB_PAGE (selected | |
18 | automatically when CONFIG_HUGETLBFS is selected) configuration | |
19 | options. | |
1da177e4 LT |
20 | |
21 | The kernel built with hugepage support should show the number of configured | |
5c7ad510 | 22 | hugepages in the system by running the "cat /proc/meminfo" command. |
1da177e4 LT |
23 | |
24 | /proc/meminfo also provides information about the total number of hugetlb | |
25 | pages configured in the kernel. It also displays information about the | |
26 | number of free hugetlb pages at any time. It also displays information about | |
27 | the configured hugepage size - this is needed for generating the proper | |
28 | alignment and size of the arguments to the above system calls. | |
29 | ||
30 | The output of "cat /proc/meminfo" will have output like: | |
31 | ||
32 | ..... | |
33 | HugePages_Total: xxx | |
34 | HugePages_Free: yyy | |
35 | Hugepagesize: zzz KB | |
36 | ||
37 | /proc/filesystems should also show a filesystem of type "hugetlbfs" configured | |
38 | in the kernel. | |
39 | ||
40 | /proc/sys/vm/nr_hugepages indicates the current number of configured hugetlb | |
41 | pages in the kernel. Super user can dynamically request more (or free some | |
5c7ad510 MBY |
42 | pre-configured) hugepages. |
43 | The allocation (or deallocation) of hugetlb pages is possible only if there are | |
1da177e4 | 44 | enough physically contiguous free pages in system (freeing of hugepages is |
5c7ad510 | 45 | possible only if there are enough hugetlb pages free that can be transfered |
1da177e4 LT |
46 | back to regular memory pool). |
47 | ||
48 | Pages that are used as hugetlb pages are reserved inside the kernel and can | |
5c7ad510 | 49 | not be used for other purposes. |
1da177e4 LT |
50 | |
51 | Once the kernel with Hugetlb page support is built and running, a user can | |
52 | use either the mmap system call or shared memory system calls to start using | |
53 | the huge pages. It is required that the system administrator preallocate | |
5c7ad510 | 54 | enough memory for huge page purposes. |
1da177e4 LT |
55 | |
56 | Use the following command to dynamically allocate/deallocate hugepages: | |
57 | ||
58 | echo 20 > /proc/sys/vm/nr_hugepages | |
59 | ||
60 | This command will try to configure 20 hugepages in the system. The success | |
61 | or failure of allocation depends on the amount of physically contiguous | |
62 | memory that is preset in system at this time. System administrators may want | |
63 | to put this command in one of the local rc init file. This will enable the | |
64 | kernel to request huge pages early in the boot process (when the possibility | |
65 | of getting physical contiguous pages is still very high). | |
66 | ||
67 | If the user applications are going to request hugepages using mmap system | |
68 | call, then it is required that system administrator mount a file system of | |
69 | type hugetlbfs: | |
70 | ||
71 | mount none /mnt/huge -t hugetlbfs <uid=value> <gid=value> <mode=value> | |
72 | <size=value> <nr_inodes=value> | |
73 | ||
74 | This command mounts a (pseudo) filesystem of type hugetlbfs on the directory | |
75 | /mnt/huge. Any files created on /mnt/huge uses hugepages. The uid and gid | |
76 | options sets the owner and group of the root of the file system. By default | |
77 | the uid and gid of the current process are taken. The mode option sets the | |
78 | mode of root of file system to value & 0777. This value is given in octal. | |
79 | By default the value 0755 is picked. The size option sets the maximum value of | |
80 | memory (huge pages) allowed for that filesystem (/mnt/huge). The size is | |
81 | rounded down to HPAGE_SIZE. The option nr_inode sets the maximum number of | |
82 | inodes that /mnt/huge can use. If the size or nr_inode options are not | |
83 | provided on command line then no limits are set. For size and nr_inodes | |
5c7ad510 MBY |
84 | options, you can use [G|g]/[M|m]/[K|k] to represent giga/mega/kilo. For |
85 | example, size=2K has the same meaning as size=2048. An example is given at | |
86 | the end of this document. | |
1da177e4 LT |
87 | |
88 | read and write system calls are not supported on files that reside on hugetlb | |
89 | file systems. | |
90 | ||
91 | A regular chown, chgrp and chmod commands (with right permissions) could be | |
92 | used to change the file attributes on hugetlbfs. | |
93 | ||
94 | Also, it is important to note that no such mount command is required if the | |
95 | applications are going to use only shmat/shmget system calls. Users who | |
96 | wish to use hugetlb page via shared memory segment should be a member of | |
97 | a supplementary group and system admin needs to configure that gid into | |
98 | /proc/sys/vm/hugetlb_shm_group. It is possible for same or different | |
99 | applications to use any combination of mmaps and shm* calls. Though the | |
100 | mount of filesystem will be required for using mmaps. | |
101 | ||
102 | ******************************************************************* | |
103 | ||
104 | /* | |
105 | * Example of using hugepage memory in a user application using Sys V shared | |
106 | * memory system calls. In this example the app is requesting 256MB of | |
107 | * memory that is backed by huge pages. The application uses the flag | |
108 | * SHM_HUGETLB in the shmget system call to inform the kernel that it is | |
109 | * requesting hugepages. | |
110 | * | |
111 | * For the ia64 architecture, the Linux kernel reserves Region number 4 for | |
112 | * hugepages. That means the addresses starting with 0x800000... will need | |
113 | * to be specified. Specifying a fixed address is not required on ppc64, | |
114 | * i386 or x86_64. | |
115 | * | |
116 | * Note: The default shared memory limit is quite low on many kernels, | |
117 | * you may need to increase it via: | |
118 | * | |
119 | * echo 268435456 > /proc/sys/kernel/shmmax | |
120 | * | |
121 | * This will increase the maximum size per shared memory segment to 256MB. | |
122 | * The other limit that you will hit eventually is shmall which is the | |
123 | * total amount of shared memory in pages. To set it to 16GB on a system | |
124 | * with a 4kB pagesize do: | |
125 | * | |
126 | * echo 4194304 > /proc/sys/kernel/shmall | |
127 | */ | |
128 | #include <stdlib.h> | |
129 | #include <stdio.h> | |
130 | #include <sys/types.h> | |
131 | #include <sys/ipc.h> | |
132 | #include <sys/shm.h> | |
133 | #include <sys/mman.h> | |
134 | ||
135 | #ifndef SHM_HUGETLB | |
136 | #define SHM_HUGETLB 04000 | |
137 | #endif | |
138 | ||
139 | #define LENGTH (256UL*1024*1024) | |
140 | ||
141 | #define dprintf(x) printf(x) | |
142 | ||
143 | /* Only ia64 requires this */ | |
144 | #ifdef __ia64__ | |
145 | #define ADDR (void *)(0x8000000000000000UL) | |
146 | #define SHMAT_FLAGS (SHM_RND) | |
147 | #else | |
148 | #define ADDR (void *)(0x0UL) | |
149 | #define SHMAT_FLAGS (0) | |
150 | #endif | |
151 | ||
152 | int main(void) | |
153 | { | |
154 | int shmid; | |
155 | unsigned long i; | |
156 | char *shmaddr; | |
157 | ||
158 | if ((shmid = shmget(2, LENGTH, | |
159 | SHM_HUGETLB | IPC_CREAT | SHM_R | SHM_W)) < 0) { | |
160 | perror("shmget"); | |
161 | exit(1); | |
162 | } | |
163 | printf("shmid: 0x%x\n", shmid); | |
164 | ||
165 | shmaddr = shmat(shmid, ADDR, SHMAT_FLAGS); | |
166 | if (shmaddr == (char *)-1) { | |
167 | perror("Shared memory attach failure"); | |
168 | shmctl(shmid, IPC_RMID, NULL); | |
169 | exit(2); | |
170 | } | |
171 | printf("shmaddr: %p\n", shmaddr); | |
172 | ||
173 | dprintf("Starting the writes:\n"); | |
174 | for (i = 0; i < LENGTH; i++) { | |
175 | shmaddr[i] = (char)(i); | |
176 | if (!(i % (1024 * 1024))) | |
177 | dprintf("."); | |
178 | } | |
179 | dprintf("\n"); | |
180 | ||
181 | dprintf("Starting the Check..."); | |
182 | for (i = 0; i < LENGTH; i++) | |
183 | if (shmaddr[i] != (char)i) | |
184 | printf("\nIndex %lu mismatched\n", i); | |
185 | dprintf("Done.\n"); | |
186 | ||
187 | if (shmdt((const void *)shmaddr) != 0) { | |
188 | perror("Detach failure"); | |
189 | shmctl(shmid, IPC_RMID, NULL); | |
190 | exit(3); | |
191 | } | |
192 | ||
193 | shmctl(shmid, IPC_RMID, NULL); | |
194 | ||
195 | return 0; | |
196 | } | |
197 | ||
198 | ******************************************************************* | |
199 | ||
200 | /* | |
201 | * Example of using hugepage memory in a user application using the mmap | |
202 | * system call. Before running this application, make sure that the | |
203 | * administrator has mounted the hugetlbfs filesystem (on some directory | |
204 | * like /mnt) using the command mount -t hugetlbfs nodev /mnt. In this | |
205 | * example, the app is requesting memory of size 256MB that is backed by | |
206 | * huge pages. | |
207 | * | |
208 | * For ia64 architecture, Linux kernel reserves Region number 4 for hugepages. | |
209 | * That means the addresses starting with 0x800000... will need to be | |
210 | * specified. Specifying a fixed address is not required on ppc64, i386 | |
211 | * or x86_64. | |
212 | */ | |
213 | #include <stdlib.h> | |
214 | #include <stdio.h> | |
215 | #include <unistd.h> | |
216 | #include <sys/mman.h> | |
217 | #include <fcntl.h> | |
218 | ||
219 | #define FILE_NAME "/mnt/hugepagefile" | |
220 | #define LENGTH (256UL*1024*1024) | |
221 | #define PROTECTION (PROT_READ | PROT_WRITE) | |
222 | ||
223 | /* Only ia64 requires this */ | |
224 | #ifdef __ia64__ | |
225 | #define ADDR (void *)(0x8000000000000000UL) | |
226 | #define FLAGS (MAP_SHARED | MAP_FIXED) | |
227 | #else | |
228 | #define ADDR (void *)(0x0UL) | |
229 | #define FLAGS (MAP_SHARED) | |
230 | #endif | |
231 | ||
232 | void check_bytes(char *addr) | |
233 | { | |
234 | printf("First hex is %x\n", *((unsigned int *)addr)); | |
235 | } | |
236 | ||
237 | void write_bytes(char *addr) | |
238 | { | |
239 | unsigned long i; | |
240 | ||
241 | for (i = 0; i < LENGTH; i++) | |
242 | *(addr + i) = (char)i; | |
243 | } | |
244 | ||
245 | void read_bytes(char *addr) | |
246 | { | |
247 | unsigned long i; | |
248 | ||
249 | check_bytes(addr); | |
250 | for (i = 0; i < LENGTH; i++) | |
251 | if (*(addr + i) != (char)i) { | |
252 | printf("Mismatch at %lu\n", i); | |
253 | break; | |
254 | } | |
255 | } | |
256 | ||
257 | int main(void) | |
258 | { | |
259 | void *addr; | |
260 | int fd; | |
261 | ||
262 | fd = open(FILE_NAME, O_CREAT | O_RDWR, 0755); | |
263 | if (fd < 0) { | |
264 | perror("Open failed"); | |
265 | exit(1); | |
266 | } | |
267 | ||
268 | addr = mmap(ADDR, LENGTH, PROTECTION, FLAGS, fd, 0); | |
269 | if (addr == MAP_FAILED) { | |
270 | perror("mmap"); | |
271 | unlink(FILE_NAME); | |
272 | exit(1); | |
273 | } | |
274 | ||
275 | printf("Returned address is %p\n", addr); | |
276 | check_bytes(addr); | |
277 | write_bytes(addr); | |
278 | read_bytes(addr); | |
279 | ||
280 | munmap(addr, LENGTH); | |
281 | close(fd); | |
282 | unlink(FILE_NAME); | |
283 | ||
284 | return 0; | |
285 | } |