Mathieu Desnoyers [Wed, 9 Oct 2024 01:16:35 +0000 (21:16 -0400)]
Cleanup: Replace "unsigned" by "unsigned int"
Signed-off-by: Mathieu Desnoyers <mathieu.desnoyers@efficios.com>
Change-Id: Id77e0140ec3594d70bd91d5cecdfa9a1d738aff1
Mathieu Desnoyers [Sat, 28 Sep 2024 10:57:42 +0000 (06:57 -0400)]
Fix: racy init_done initialization
If init_done is set while another thread is loading its value, the other
thread can erroneously expect initialization to be complete while the
initialization is still in progress.
Set init_done after initialization with a store-release, and load it
with load-acquire outside of the mutex critical section.
Signed-off-by: Mathieu Desnoyers <mathieu.desnoyers@efficios.com>
Change-Id: Idb1345c6dcf3da7d711937bd5072527c0bf1d33a
Mathieu Desnoyers [Thu, 29 Aug 2024 15:05:11 +0000 (11:05 -0400)]
Add basic_percpu_benchmark for intermittent workload benchmarking
Signed-off-by: Mathieu Desnoyers <mathieu.desnoyers@efficios.com>
Michael Jeanson [Thu, 29 Aug 2024 18:53:02 +0000 (14:53 -0400)]
Add dummy TLS variable to no_syscall test
Ensure the main executable has at least one TLS variable which will be
allocated before the rseq area, making sure the rseq_offset is not 0. This
allows testing that the rseq_offset variable is properly initialized by
checking it is not 0.
Most toolchains will add at least one main exec TLS variable but it's
currently not the case on RISC-V.
Change-Id: Ie29bcaa1e008e99953a4f19e19f05c8b85dac62d
Signed-off-by: Michael Jeanson <mjeanson@efficios.com>
Signed-off-by: Mathieu Desnoyers <mathieu.desnoyers@efficios.com>
Mathieu Desnoyers [Thu, 29 Aug 2024 19:22:32 +0000 (15:22 -0400)]
mempool: Remove deprecated rseq_mempool_init_numa from API
Signed-off-by: Mathieu Desnoyers <mathieu.desnoyers@efficios.com>
Change-Id: I59d0852676442e9bda9fe1ddc9f66182caa14cb6
Mathieu Desnoyers [Mon, 26 Aug 2024 13:24:55 +0000 (09:24 -0400)]
fix: mempool: rseq_mempool_free maps to librseq_mempool_percpu_free
Signed-off-by: Mathieu Desnoyers <mathieu.desnoyers@efficios.com>
Change-Id: I260cbbfbe71b5b544f2ca07835d5d8bd8a7e195e
Michael Jeanson [Wed, 31 Jul 2024 15:45:54 +0000 (11:45 -0400)]
Add riscv thread_pointer support with GCC < 10.3
Change-Id: I17831a217c31bc0a6f298d4a0904b89c2190a3a7
Signed-off-by: Michael Jeanson <mjeanson@efficios.com>
Signed-off-by: Mathieu Desnoyers <mathieu.desnoyers@efficios.com>
Michael Jeanson [Wed, 31 Jul 2024 18:30:07 +0000 (14:30 -0400)]
Add dummy TLS variable to rseq_offset tests
Ensure the main executable has at least one TLS variable which will be
allocated before the rseq area, making sure the rseq_offset is not 0. This
allows testing that the rseq_offset variable is properly initialized by
checking it is not 0.
Most toolchains will add at least one main exec TLS variable but it's
currently not the case on RISC-V.
Change-Id: I1c50640c4f32ea5ef65b0b823e7be75fd92b303d
Signed-off-by: Michael Jeanson <mjeanson@efficios.com>
Signed-off-by: Mathieu Desnoyers <mathieu.desnoyers@efficios.com>
Michael Jeanson [Mon, 29 Jul 2024 18:10:46 +0000 (18:10 +0000)]
Add a test to check the registration after fork
Change-Id: I5a8321945edbc1bb00122a5809cecd4be6c7de20
Signed-off-by: Michael Jeanson <mjeanson@efficios.com>
Signed-off-by: Mathieu Desnoyers <mathieu.desnoyers@efficios.com>
Michael Jeanson [Fri, 26 Jul 2024 15:15:44 +0000 (15:15 +0000)]
Add a test to validate the syscall errors
Change-Id: Ia589597d23df3fe317a009d409272f4664f0a7e0
Signed-off-by: Michael Jeanson <mjeanson@efficios.com>
Signed-off-by: Mathieu Desnoyers <mathieu.desnoyers@efficios.com>
Michael Jeanson [Fri, 26 Jul 2024 19:50:54 +0000 (19:50 +0000)]
Only set 'rseq_size' on first thread registration
To mimic the libc behavior, don't set the feature size in 'rseq_size'
until at least one thread has successfully registered. This allows using
'rseq_size' in fast-paths to test for both registration status and
available features. The caveat is that on libc either all threads are
registered or none are, while with bare librseq it is the responsability
of the user to register all threads using rseq.
Change-Id: Idc491153de55f9ac38bd1fcad425aee9549fc910
Signed-off-by: Michael Jeanson <mjeanson@efficios.com>
Signed-off-by: Mathieu Desnoyers <mathieu.desnoyers@efficios.com>
Michael Jeanson [Fri, 26 Jul 2024 18:03:23 +0000 (14:03 -0400)]
doc: clarifiy EINVAL/EBUSY errors with an active registration
Clarify that EBUSY will only be returned when the rseq syscall is called
with the same arguments as the active registration. If any argument
differs, EINVAL wil be returned instead.
Change-Id: I3b6a8d971489e3fc64daf85484ae8e1b8143b1d3
Signed-off-by: Michael Jeanson <mjeanson@efficios.com>
Signed-off-by: Mathieu Desnoyers <mathieu.desnoyers@efficios.com>
Michael Jeanson [Tue, 23 Jul 2024 15:40:33 +0000 (11:40 -0400)]
fix: Add LICENSES/LGPL-2.1-or-later.txt to distribution
Change-Id: I0356db30975cf0232ea0dfa165cf37431d9dfa3f
Signed-off-by: Michael Jeanson <mjeanson@efficios.com>
Signed-off-by: Mathieu Desnoyers <mathieu.desnoyers@efficios.com>
Mathieu Desnoyers [Tue, 23 Jul 2024 16:29:58 +0000 (12:29 -0400)]
Fix: adapt no syscall test number of tests
Signed-off-by: Mathieu Desnoyers <mathieu.desnoyers@efficios.com>
Change-Id: I5baaa5da7b87495a71f103fbc555ef58b1469e6d
Mathieu Desnoyers [Tue, 9 Jul 2024 15:23:09 +0000 (11:23 -0400)]
Adapt to glibc __rseq_size feature detection
Adapt the rseq.c code to follow GNU C library changes introduced by:
commit
2e456ccf0c34 ("Linux: Make __rseq_size useful for feature detection (bug 31965)")
Signed-off-by: Mathieu Desnoyers <mathieu.desnoyers@efficios.com>
Change-Id: I83990c9c3b1e9277af48edbfc3176d3dc5160d3c
Michael Jeanson [Tue, 23 Jul 2024 15:10:04 +0000 (11:10 -0400)]
Add missing LGPL-2.1-or-later license for 'src/list.h'
Change-Id: Ia7b0eb9724c309fc7193e281de3007a4051bb8f2
Signed-off-by: Michael Jeanson <mjeanson@efficios.com>
Signed-off-by: Mathieu Desnoyers <mathieu.desnoyers@efficios.com>
Mathieu Desnoyers [Thu, 2 May 2024 14:43:16 +0000 (10:43 -0400)]
fix: handle EINTR correctly in get_cpu_mask_from_sysfs
If the read() in get_cpu_mask_from_sysfs() fails with EINTR, the code is
supposed to retry, but the while loop condition has (bytes_read > 0),
which is false when read() fails with EINTR. The result is that the code
exits the loop, having only read part of the string.
Use (bytes_read != 0) in the while loop condition instead, since the
(bytes_read < 0) case is already handled in the loop.
Original fix in liburcu from Benjamin Marzinski <bmarzins@redhat.com>:
commit
9922f33e2986 ("fix: handle EINTR correctly in get_cpu_mask_from_sysfs")
Signed-off-by: Mathieu Desnoyers <mathieu.desnoyers@efficios.com>
Change-Id: I134662fe83613ff914433d3b30a63a0134ffa124
Mathieu Desnoyers [Fri, 12 Apr 2024 19:43:26 +0000 (15:43 -0400)]
Revert "mempool: Track allocated items per range"
This reverts commit
ffea0dea60059c855cb0b4e1f784352012ae0b37.
Counting allocated items per range has no inherent use and adds a
counter to the allocation/free fast path: we could not easily unmap
ranges when the allocated items count reaches 0 anyway because the pool
free list spawns across items from all ranges.
Signed-off-by: Mathieu Desnoyers <mathieu.desnoyers@efficios.com>
Mathieu Desnoyers [Fri, 12 Apr 2024 19:29:05 +0000 (15:29 -0400)]
mempool: malloc: handle empty range list
Although it is not currently possible to have an empty range list
because there is a range created on pool creation, handle empty range
list in __rseq_percpu_malloc for future-proofing.
Signed-off-by: Mathieu Desnoyers <mathieu.desnoyers@efficios.com>
Change-Id: Ie646b16114eb1812916dad2b08f9d87884925748
Mathieu Desnoyers [Fri, 12 Apr 2024 19:20:35 +0000 (15:20 -0400)]
mempool: use list.h API
Signed-off-by: Mathieu Desnoyers <mathieu.desnoyers@efficios.com>
Change-Id: I2cabb4b8b6ff70cd111961335c0306b0ec76e827
Mathieu Desnoyers [Fri, 12 Apr 2024 18:59:39 +0000 (14:59 -0400)]
mempool: Track allocated items per range
Signed-off-by: Mathieu Desnoyers <mathieu.desnoyers@efficios.com>
Change-Id: I48c05318c39b3f3bcf9c14a5aa3181e29754db97
Mathieu Desnoyers [Fri, 22 Mar 2024 19:32:22 +0000 (15:32 -0400)]
mempool: default to per-cpu mempool type
The primary use-case for mempool is per-cpu data. Default to this rather
than global.
Signed-off-by: Mathieu Desnoyers <mathieu.desnoyers@efficios.com>
Change-Id: I63d994de80e8ed16e0f5666a846d7e872a58cdca
Mathieu Desnoyers [Wed, 20 Mar 2024 15:26:33 +0000 (11:26 -0400)]
mempool: Rename pool policy
Rename pool policy from:
RSEQ_MEMPOOL_POPULATE_PRIVATE_NONE -> RSEQ_MEMPOOL_POPULATE_COW_INIT
RSEQ_MEMPOOL_POPULATE_PRIVATE_ALL -> RSEQ_MEMPOOL_POPULATE_COW_ZERO
to make it clear to the user that the COW_ZERO pool policy benefits from
leaving the memory zero-allocated.
Also, now that zmalloc checks the prior content for zero to prevent
useless COW of the per-cpu ranges, the term "populate all" does not fit
well anymore. Copy-on-write from the zero page is clearer.
Signed-off-by: Mathieu Desnoyers <mathieu.desnoyers@efficios.com>
Change-Id: If97984225d971a2f48ba241b2a9570e81079bd03
Mathieu Desnoyers [Wed, 20 Mar 2024 00:22:33 +0000 (20:22 -0400)]
Add Olivier Dion to list of mempool authors
Signed-off-by: Mathieu Desnoyers <mathieu.desnoyers@efficios.com>
Change-Id: I472019d46812e574bc6bef3271635c786263c4a3
Mathieu Desnoyers [Tue, 19 Mar 2024 22:18:21 +0000 (18:18 -0400)]
mempool: Introduce COW_ZERO poison value
Signed-off-by: Mathieu Desnoyers <mathieu.desnoyers@efficios.com>
Change-Id: Id7fdfa0ceb8631e1b4bfe594d9e68c4d8663c6ef
Mathieu Desnoyers [Tue, 19 Mar 2024 21:21:29 +0000 (17:21 -0400)]
mempool: Document destroy after fork for populate none
Signed-off-by: Mathieu Desnoyers <mathieu.desnoyers@efficios.com>
Change-Id: I826990c9ee58dbda80f3271e361328b474edbf08
Mathieu Desnoyers [Tue, 19 Mar 2024 21:19:57 +0000 (17:19 -0400)]
mempool: Tests mempool destroy in child process
Signed-off-by: Mathieu Desnoyers <mathieu.desnoyers@efficios.com>
Change-Id: I8615436deaf089ba0f560145d604251307b307b9
Mathieu Desnoyers [Tue, 19 Mar 2024 21:18:51 +0000 (17:18 -0400)]
mempool: Allow mempool destroy in children processes
Allow destroy of mempool objects from a child process by validating
whether the mappings are accessible or not.
Signed-off-by: Mathieu Desnoyers <mathieu.desnoyers@efficios.com>
Change-Id: Ice9dd8cd1fad1e656dab0b457fc5469ecfae1130
Olivier Dion [Tue, 19 Mar 2024 15:39:18 +0000 (11:39 -0400)]
tests/mempool: Allocate pool in child for robust tests
Depending on the populate policy, it is not possible to use a pool
across a fork. Since robust tests rely on SIGABRT to be triggered in a
child process, the pool must be allocated in the child instead of the
parent for the RSEQ_MEMPOOL_POPULATE_PRIVATE_NONE policy.
Change-Id: I78bc19d12b4f8bc0c7e2a2f05fab870ed7524e13
Signed-off-by: Olivier Dion <odion@efficios.com>
Signed-off-by: Mathieu Desnoyers <mathieu.desnoyers@efficios.com>
Mathieu Desnoyers [Tue, 19 Mar 2024 15:21:02 +0000 (11:21 -0400)]
mempool: Fix error pool->range
Signed-off-by: Mathieu Desnoyers <mathieu.desnoyers@efficios.com>
Change-Id: I563b33d55e565a8e1af1558f7ade56e977fd98bd
Mathieu Desnoyers [Tue, 19 Mar 2024 15:20:06 +0000 (11:20 -0400)]
mempool: Protect whole range of populate none across fork
Signed-off-by: Mathieu Desnoyers <mathieu.desnoyers@efficios.com>
Change-Id: I6928128ceffed57fd4ecfdaee50a3085e585387a
Mathieu Desnoyers [Tue, 19 Mar 2024 14:50:29 +0000 (10:50 -0400)]
mempool: Fix memfd leaks on error
Signed-off-by: Mathieu Desnoyers <mathieu.desnoyers@efficios.com>
Change-Id: Id22b56ffc10f57021a8cb1828ef5b032436d63ea
Mathieu Desnoyers [Tue, 19 Mar 2024 14:36:53 +0000 (10:36 -0400)]
mempool: Use MADV_DONTFORK on init values
The init values shared mapping should not be shared across fork. Use
madvise MADV_DONTFORK on the memory range to ensure those pages are not
shared with children processes across fork.
Signed-off-by: Mathieu Desnoyers <mathieu.desnoyers@efficios.com>
Change-Id: I4b3541d21401227ca568ef6e8105d088746341cb
Olivier Dion [Tue, 19 Mar 2024 14:09:50 +0000 (10:09 -0400)]
mempool: Add custom name to memfd_create
Change-Id: I12459129e2c4aa7e35ab557fbc6060bd63de816b
Signed-off-by: Olivier Dion <odion@efficios.com>
Signed-off-by: Mathieu Desnoyers <mathieu.desnoyers@efficios.com>
Mathieu Desnoyers [Tue, 19 Mar 2024 13:53:56 +0000 (09:53 -0400)]
mempool: Create memfd only for range creation
Don't share memfd across ranges, we can close its file descriptor
immediately after mapping it.
Signed-off-by: Mathieu Desnoyers <mathieu.desnoyers@efficios.com>
Change-Id: Ia0c63f74e17b16039dff0b9367a19c411396605d
Mathieu Desnoyers [Mon, 18 Mar 2024 18:56:13 +0000 (14:56 -0400)]
mempool: Rename DEFAULT_POISON_VALUE to DEFAULT_PRIVATE_POISON_VALUE
I expect the eventual default SHARED poison value to be 0x0 to eliminate
useless allocation over zero pages for unused cpus.
Signed-off-by: Mathieu Desnoyers <mathieu.desnoyers@efficios.com>
Change-Id: I6faad228bce381485ed60cd92b3f155187a68b4e
Mathieu Desnoyers [Mon, 18 Mar 2024 18:55:05 +0000 (14:55 -0400)]
mempool: Introduce "private" populate policy
Signed-off-by: Mathieu Desnoyers <mathieu.desnoyers@efficios.com>
Change-Id: I3a4907ee65a58e7dbb3d03bf5096175620892915
Mathieu Desnoyers [Mon, 18 Mar 2024 18:49:41 +0000 (14:49 -0400)]
mempool: Remove mmap/munmap callbacks
Exposes too much internals of mempool to the application.
Signed-off-by: Mathieu Desnoyers <mathieu.desnoyers@efficios.com>
Change-Id: I49bd3613b473882eb97e71bce2cf31fcb052dc5a
Mathieu Desnoyers [Mon, 18 Mar 2024 15:25:34 +0000 (11:25 -0400)]
mempool: replace memcmpbyte by rseq_cmp_item
Signed-off-by: Mathieu Desnoyers <mathieu.desnoyers@efficios.com>
Change-Id: Ic45d9a6a64e86772c45815127d3d0a601e344278
Mathieu Desnoyers [Mon, 18 Mar 2024 15:19:33 +0000 (11:19 -0400)]
mempool: use bzero
Signed-off-by: Mathieu Desnoyers <mathieu.desnoyers@efficios.com>
Change-Id: I1a1d13e856dd73576cc0c2fea46fb4dcd3eef963
Mathieu Desnoyers [Mon, 18 Mar 2024 15:04:56 +0000 (11:04 -0400)]
mempool: do not overwrite same per-cpu values
If the per-cpu items already have the correct values, do not write to
the pages. This eliminates useless COW in populate-all pools over the
zero page.
Signed-off-by: Mathieu Desnoyers <mathieu.desnoyers@efficios.com>
Change-Id: I4e47db82ccf15c3cca30a8df48d367cbc05f3887
Mathieu Desnoyers [Mon, 18 Mar 2024 11:28:08 +0000 (07:28 -0400)]
mempool cow test: use default stride
Fix ppc64 which has a 64kB page size.
Signed-off-by: Mathieu Desnoyers <mathieu.desnoyers@efficios.com>
Change-Id: I2debd1c50f1ecc0486a557b69af3fda70bae2dcd
Mathieu Desnoyers [Sun, 17 Mar 2024 20:31:11 +0000 (16:31 -0400)]
mempool: Test COW vs malloc_init race
Test that the entire malloc init value is visible in CPU mappings. If
the COW page copy race vs init happens while init is in the middle of
storing to the newly allocated area, iteration on all CPUs comparing the
visible content to the init value is responsible for detecting and
mitigating uninitialized or partially initialized init value from the
point of view of a CPU. Validate that this scheme has the intended
effect wrt a concurrent COW caused by storing to a nearby per-cpu area
on the same page.
Signed-off-by: Mathieu Desnoyers <mathieu.desnoyers@efficios.com>
Change-Id: Iebde10eb61e80cc0d8eb0ebc89925f4a2ad9de9c
Mathieu Desnoyers [Sun, 17 Mar 2024 02:43:48 +0000 (22:43 -0400)]
mempool tests: Adapt to free list new location
Signed-off-by: Mathieu Desnoyers <mathieu.desnoyers@efficios.com>
Change-Id: If582864dcc7aceea99f866ad3ca0f6391d6ca5c5
Mathieu Desnoyers [Sun, 17 Mar 2024 02:42:48 +0000 (22:42 -0400)]
Move robust pool free list to own memory area
Consumes more memory in the robust pool, but removes blind spots for
validation of poison due to aliasing with free list.
Signed-off-by: Mathieu Desnoyers <mathieu.desnoyers@efficios.com>
Change-Id: Ia8187cf3049aad8e54daaced4005fbe2e841ae7e
Mathieu Desnoyers [Sat, 16 Mar 2024 21:16:57 +0000 (17:16 -0400)]
mempool: Fix off_t cast warning on 32-bit
Signed-off-by: Mathieu Desnoyers <mathieu.desnoyers@efficios.com>
Change-Id: I93978e7f0f583ae4a9bf62d3f707258c385bb0a8
Mathieu Desnoyers [Sat, 16 Mar 2024 20:23:45 +0000 (16:23 -0400)]
mempool test: test populate none policy
Signed-off-by: Mathieu Desnoyers <mathieu.desnoyers@efficios.com>
Change-Id: I2eb34565ba84ae9479eb235743fbba07fe277165
Mathieu Desnoyers [Thu, 14 Mar 2024 01:50:56 +0000 (21:50 -0400)]
mempool: Implement populate none policy
Implement lazy-populate policy (populate none) relying on kernel COW of
init values pages.
Signed-off-by: Mathieu Desnoyers <mathieu.desnoyers@efficios.com>
Change-Id: Ie4467f1806208271a5462605445abc2f0c1c1d38
Mathieu Desnoyers [Fri, 15 Mar 2024 19:37:56 +0000 (15:37 -0400)]
mempool: Fix leak on pool destroy
Signed-off-by: Mathieu Desnoyers <mathieu.desnoyers@efficios.com>
Change-Id: I3534084f1055b0db73ac9ec5005a6c88d8056d78
Mathieu Desnoyers [Thu, 14 Mar 2024 00:22:08 +0000 (20:22 -0400)]
mempool: Introduce mmap_addr and mmap_len range fields
Signed-off-by: Mathieu Desnoyers <mathieu.desnoyers@efficios.com>
Change-Id: Ia91571ba9f3535402f456074973875503e688690
Mathieu Desnoyers [Sun, 10 Mar 2024 19:14:08 +0000 (15:14 -0400)]
mempool_test: test rseq_mempool_percpu_malloc_init
Signed-off-by: Mathieu Desnoyers <mathieu.desnoyers@efficios.com>
Change-Id: Ia266d69a4007340ffe1d98950619ce90582c2e40
Mathieu Desnoyers [Sun, 10 Mar 2024 19:06:16 +0000 (15:06 -0400)]
mempool: implement malloc_init
Add a *_malloc_init to the API to allow allocating memory initialized to
a given initial value.
This replaces application patterns where a loop on all possible CPUs is
needed after allocation to initialize memory to a given nonzero value,
and will eventually allow doing the initialization lazily (the first
time the cpu touches memory within the pool).
Signed-off-by: Mathieu Desnoyers <mathieu.desnoyers@efficios.com>
Change-Id: I3acb1505d7b4f46ceb6a8128d2ebda346b8e359c
Mathieu Desnoyers [Sun, 10 Mar 2024 03:03:49 +0000 (22:03 -0500)]
mempool_test: test poison corruption detection
Signed-off-by: Mathieu Desnoyers <mathieu.desnoyers@efficios.com>
Change-Id: Ia284caa599f2c9c7c142efe6508779d36d828e17
Mathieu Desnoyers [Sat, 9 Mar 2024 22:02:40 +0000 (17:02 -0500)]
mempool: Use default poison value when robust is set
Signed-off-by: Mathieu Desnoyers <mathieu.desnoyers@efficios.com>
Change-Id: I84a75bd14f6798261fd630aaf076cc3981bad0a1
Mathieu Desnoyers [Sat, 9 Mar 2024 21:55:24 +0000 (16:55 -0500)]
mempool: check poison on destroy
Signed-off-by: Mathieu Desnoyers <mathieu.desnoyers@efficios.com>
Change-Id: I40fe7d4179e42a392f3528cc9697552fe7c4ad09
Mathieu Desnoyers [Sat, 9 Mar 2024 21:44:17 +0000 (16:44 -0500)]
mempool: Detect poison corruption on alloc
Signed-off-by: Mathieu Desnoyers <mathieu.desnoyers@efficios.com>
Change-Id: I1d4181f089aa08d1c14466751661bba8a3727cf5
Mathieu Desnoyers [Sat, 9 Mar 2024 21:43:40 +0000 (16:43 -0500)]
mempool test: add one extra object after free
Signed-off-by: Mathieu Desnoyers <mathieu.desnoyers@efficios.com>
Change-Id: Ie4b5f47ac4e2108cecb079c63767259ee0493a48
Mathieu Desnoyers [Sat, 9 Mar 2024 21:14:54 +0000 (16:14 -0500)]
mempool: introduce poison attribute
Signed-off-by: Mathieu Desnoyers <mathieu.desnoyers@efficios.com>
Change-Id: I056c0d8c4f0d5b1d2c401592a08b22ff6f239d11
Mathieu Desnoyers [Sat, 9 Mar 2024 20:40:22 +0000 (15:40 -0500)]
mempool test: Only use a single range for large stride
Signed-off-by: Mathieu Desnoyers <mathieu.desnoyers@efficios.com>
Change-Id: I62baaa751d46abf0341e4e1c3dffecd1b2402986
Mathieu Desnoyers [Sat, 9 Mar 2024 20:38:54 +0000 (15:38 -0500)]
mempool test: test max_nr_ranges > 1
Signed-off-by: Mathieu Desnoyers <mathieu.desnoyers@efficios.com>
Change-Id: I7d3b48690c5e61789f0bc5f50db27898499c335a
Mathieu Desnoyers [Sat, 9 Mar 2024 20:26:12 +0000 (15:26 -0500)]
mempool: Introduce rseq_mempool_attr_set_max_nr_ranges
Signed-off-by: Mathieu Desnoyers <mathieu.desnoyers@efficios.com>
Change-Id: I5417673d16ce7cfa5e8f2b9c120978f8633664bc
Mathieu Desnoyers [Sat, 9 Mar 2024 19:59:34 +0000 (14:59 -0500)]
mempool: Introduce support for ranges linked list
Each pool can allocate additional ranges when the initial range is full.
There is currently a limit of 1 range (to be made configurable in the
future).
Signed-off-by: Mathieu Desnoyers <mathieu.desnoyers@efficios.com>
Change-Id: Ibc59f140a8dc1ac9152b0a4ae82d32db76a2c42e
Mathieu Desnoyers [Sat, 9 Mar 2024 20:09:00 +0000 (15:09 -0500)]
mempool: remove unused FIRST_POOL
Signed-off-by: Mathieu Desnoyers <mathieu.desnoyers@efficios.com>
Change-Id: I545c7f2ff7f4b760c92bf0689b5f4edbe1894329
Mathieu Desnoyers [Sat, 9 Mar 2024 18:39:31 +0000 (13:39 -0500)]
mempool: cpu=-1 for init_func argument (global pool)
Signed-off-by: Mathieu Desnoyers <mathieu.desnoyers@efficios.com>
Change-Id: I956006f59186252d3dcc767fa21e5928f68d5cb8
Mathieu Desnoyers [Sat, 9 Mar 2024 16:40:05 +0000 (11:40 -0500)]
mempool: init_func can return an error
Signed-off-by: Mathieu Desnoyers <mathieu.desnoyers@efficios.com>
Change-Id: I212fd5a6ec6ae7580a28068de4ee88d7955da9d0
Mathieu Desnoyers [Sat, 9 Mar 2024 15:46:15 +0000 (10:46 -0500)]
mempool: aligned_mmap_anonymous: support non-power-of-2 nr_cpus
Signed-off-by: Mathieu Desnoyers <mathieu.desnoyers@efficios.com>
Change-Id: I7231ee9ac6e822b9d43dae596763d84b0abfcbe1
Mathieu Desnoyers [Sat, 9 Mar 2024 14:40:27 +0000 (09:40 -0500)]
rseq: Introduce rseq_get_max_nr_cpus
Signed-off-by: Mathieu Desnoyers <mathieu.desnoyers@efficios.com>
Change-Id: Ie7d24470994b3a6ec805bfec7111ff3492b46997
Mathieu Desnoyers [Sat, 9 Mar 2024 14:31:29 +0000 (09:31 -0500)]
param_test: Use mempool max_nr_cpus rather than CPU_SETSIZE
Signed-off-by: Mathieu Desnoyers <mathieu.desnoyers@efficios.com>
Change-Id: I6401e5584ee97ea8958898e377e1ec52476814c6
Mathieu Desnoyers [Sat, 9 Mar 2024 14:14:39 +0000 (09:14 -0500)]
mempool: Introduce rseq_mempool_get_max_nr_cpus
Signed-off-by: Mathieu Desnoyers <mathieu.desnoyers@efficios.com>
Change-Id: Ibd5a7abf40e56d8e57c7b51a5dd9495c51bd9624
Mathieu Desnoyers [Sat, 9 Mar 2024 13:59:57 +0000 (08:59 -0500)]
mempool: Introduce rseq_mempool_range_init_numa helper
Signed-off-by: Mathieu Desnoyers <mathieu.desnoyers@efficios.com>
Change-Id: Icc0ac582fbd21a71f01212d23b86a9ecd5679465
Mathieu Desnoyers [Sat, 9 Mar 2024 13:35:44 +0000 (08:35 -0500)]
mempool: Introduce init_func attribute
Signed-off-by: Mathieu Desnoyers <mathieu.desnoyers@efficios.com>
Change-Id: I1e36bae465b0f459b4b6a4f8182cf524e0e6ade5
Mathieu Desnoyers [Sat, 9 Mar 2024 13:24:49 +0000 (08:24 -0500)]
mempool: reenable rseq_mempool_range_init_numa
Signed-off-by: Mathieu Desnoyers <mathieu.desnoyers@efficios.com>
Change-Id: I8c7d3ccdf0279cca44e8ad33f8904e34b38f8a79
Mathieu Desnoyers [Sat, 9 Mar 2024 03:48:44 +0000 (22:48 -0500)]
smp.c: fix includes and macro namespace
Signed-off-by: Mathieu Desnoyers <mathieu.desnoyers@efficios.com>
Change-Id: I5ef3c45240120f7f415c3bb38a280e3a7ceb410a
Mathieu Desnoyers [Sat, 9 Mar 2024 03:16:21 +0000 (22:16 -0500)]
Mempool: default as global
Signed-off-by: Mathieu Desnoyers <mathieu.desnoyers@efficios.com>
Change-Id: I1a2c44188404e1703a9996a83495086d2b0e3ca3
Mathieu Desnoyers [Sat, 9 Mar 2024 02:47:22 +0000 (21:47 -0500)]
mempool: Receive mempool type (percpu/global) as attribute
Also introduce a smp.c/h helper to get the number of possible cpus,
copied from libside (MIT).
Signed-off-by: Mathieu Desnoyers <mathieu.desnoyers@efficios.com>
Change-Id: I4dfc8fa56ae29e5586eb029377abad62c93de6a9
Mathieu Desnoyers [Sat, 9 Mar 2024 02:16:33 +0000 (21:16 -0500)]
mempool test: iterate from page size
Fix test on ppc64, which has a 64kB page size.
Signed-off-by: Mathieu Desnoyers <mathieu.desnoyers@efficios.com>
Change-Id: I6e309855c190b056d886aa3c187a9dc1929d7391
Mathieu Desnoyers [Sat, 9 Mar 2024 01:55:08 +0000 (20:55 -0500)]
mempool: update comments in header file
Signed-off-by: Mathieu Desnoyers <mathieu.desnoyers@efficios.com>
Change-Id: I35a017716d5f49e2eb827cd7ed4f56855d0f8e1c
Mathieu Desnoyers [Sat, 9 Mar 2024 01:33:25 +0000 (20:33 -0500)]
mempool: reduce default RSEQ_PERCPU_STRIDE to 64kB on 64-bit
aarch64 has issues with mmap() of 16MB * 1000 cpus (ENOMEM).
Signed-off-by: Mathieu Desnoyers <mathieu.desnoyers@efficios.com>
Change-Id: If22aab4f9afacaf9bdf0318c8876486cdda2ec25
Mathieu Desnoyers [Sat, 9 Mar 2024 00:24:13 +0000 (19:24 -0500)]
membarrier rseq fence test: adapt to RSEQ_PERCPU_STRIDE
Signed-off-by: Mathieu Desnoyers <mathieu.desnoyers@efficios.com>
Change-Id: Ifaa59012cf19695eabe9977cae73a281e34afaa7
Mathieu Desnoyers [Sat, 9 Mar 2024 00:20:14 +0000 (19:20 -0500)]
Add missing rseq_after_asm_goto in rseq_load_add_load_load_add_store__ptr
Change-Id: If2a966e39302d5163203769855ed0d5f5e6012a7
Signed-off-by: Mathieu Desnoyers <mathieu.desnoyers@efficios.com>
Mathieu Desnoyers [Sat, 9 Mar 2024 00:18:39 +0000 (19:18 -0500)]
Revert "x86-64: Implement/use load_cbne_load_add_load_add_store"
Now that the percpu mempool allocator has a stride known in advance, it
is possible to use the offset approach in the critical section to index
by cpu number.
This reverts commit
d87b0cd222a1c11751be7f8b655fc0dc45b7dc12.
Signed-off-by: Mathieu Desnoyers <mathieu.desnoyers@efficios.com>
Change-Id: I8b394650f1dbe409eb9ed3db88afcc43cd230133
Mathieu Desnoyers [Fri, 8 Mar 2024 22:22:07 +0000 (17:22 -0500)]
mempool: namespacing, add global alloc/free
Signed-off-by: Mathieu Desnoyers <mathieu.desnoyers@efficios.com>
Change-Id: I6b3a6f9948f8400ae691738d7caa15164e634b74
Mathieu Desnoyers [Fri, 8 Mar 2024 21:32:22 +0000 (16:32 -0500)]
mempool: Introduce optional stride parameter
Signed-off-by: Mathieu Desnoyers <mathieu.desnoyers@efficios.com>
Change-Id: Ic7119bb33f86cf72c73c2d50dbe55e4769ce0b22
Mathieu Desnoyers [Fri, 8 Mar 2024 21:13:09 +0000 (16:13 -0500)]
mempool: mass renaming of API
- rseq_percpu_pool -> rseq_mempool
- rseq_pool_attr -> rseq_mempool_attr
- rseq_percpu_pool_set -> rseq_mempool_set
Signed-off-by: Mathieu Desnoyers <mathieu.desnoyers@efficios.com>
Change-Id: Ic34a4163854e91838b2f0727d26469d55c69a0e0
Olivier Dion [Fri, 8 Mar 2024 15:59:57 +0000 (10:59 -0500)]
tests/mempool_test: Add robust testing
Change-Id: I0e7c9dd7ad03070f0524edb4252cb11988418108
Signed-off-by: Olivier Dion <odion@efficios.com>
Signed-off-by: Mathieu Desnoyers <mathieu.desnoyers@efficios.com>
Mathieu Desnoyers [Fri, 8 Mar 2024 20:53:27 +0000 (15:53 -0500)]
mempool: check for NULL pool on destroy
Signed-off-by: Mathieu Desnoyers <mathieu.desnoyers@efficios.com>
Change-Id: I1a095ac04c4545095ed53699402a30d2d1ace8a7
Mathieu Desnoyers [Fri, 8 Mar 2024 20:51:24 +0000 (15:51 -0500)]
mempool: update header comments
Signed-off-by: Mathieu Desnoyers <mathieu.desnoyers@efficios.com>
Change-Id: I7648cd668ddb4a18dd6d04d021198833bfc078b5
Mathieu Desnoyers [Fri, 8 Mar 2024 20:47:25 +0000 (15:47 -0500)]
mempool: Remove unused defines
Signed-off-by: Mathieu Desnoyers <mathieu.desnoyers@efficios.com>
Change-Id: I35f3ce87da25c47097bb76ab9c3aeadee46f5e32
Mathieu Desnoyers [Fri, 8 Mar 2024 20:44:37 +0000 (15:44 -0500)]
mempool: Remove pool array and lock
Those are not needed anymore with the new mask-based scheme.
Signed-off-by: Mathieu Desnoyers <mathieu.desnoyers@efficios.com>
Change-Id: Id0d401d21b4ac3b8b7c7e50fecf6bd92513bd162
Mathieu Desnoyers [Fri, 8 Mar 2024 20:34:01 +0000 (15:34 -0500)]
mempool: Use masks to get pool range from pointer and stride for percpu offset
Signed-off-by: Mathieu Desnoyers <mathieu.desnoyers@efficios.com>
Change-Id: I45bb460be66bc4843ead316d7b870958cd1217c2
Mathieu Desnoyers [Fri, 8 Mar 2024 19:52:31 +0000 (14:52 -0500)]
Memory map ranges aligned on the stride
This will allow finding the range header with a mask applied on the
pointer in free().
Signed-off-by: Mathieu Desnoyers <mathieu.desnoyers@efficios.com>
Change-Id: I25821ebb4a7333fe53658932f58c6d9cad1bd64a
Mathieu Desnoyers [Fri, 8 Mar 2024 19:07:11 +0000 (14:07 -0500)]
percpu alloc: Introduce stride parameter
Signed-off-by: Mathieu Desnoyers <mathieu.desnoyers@efficios.com>
Change-Id: I6469725da2610560872d2fe678128455d7ff22f1
Olivier Dion [Fri, 8 Mar 2024 17:29:36 +0000 (12:29 -0500)]
rseq-mempool: Call set_alloc_slot() on all paths
set_alloc_slot() was not called when getting free slot from free-list.
Change-Id: I21d633ced6592039fe39e0ae121128958048f80e
Signed-off-by: Olivier Dion <odion@efficios.com>
Signed-off-by: Mathieu Desnoyers <mathieu.desnoyers@efficios.com>
Mathieu Desnoyers [Fri, 8 Mar 2024 16:37:05 +0000 (11:37 -0500)]
mempool: implement one range
Signed-off-by: Mathieu Desnoyers <mathieu.desnoyers@efficios.com>
Change-Id: I189ee936cd11229b5770ed95c8ac32b67fede20d
Mathieu Desnoyers [Fri, 8 Mar 2024 14:51:41 +0000 (09:51 -0500)]
Update mempool test
Signed-off-by: Mathieu Desnoyers <mathieu.desnoyers@efficios.com>
Change-Id: I57fe799bcdb4df1d6d6fb6e605e17cf581f13197
Mathieu Desnoyers [Fri, 8 Mar 2024 14:48:30 +0000 (09:48 -0500)]
Add mempool test to make check
Signed-off-by: Mathieu Desnoyers <mathieu.desnoyers@efficios.com>
Change-Id: Ia998f1157083df566e8e8e9c331983bab256880f
Mathieu Desnoyers [Fri, 8 Mar 2024 14:25:16 +0000 (09:25 -0500)]
mempool test: safe iteration for free
Signed-off-by: Mathieu Desnoyers <mathieu.desnoyers@efficios.com>
Change-Id: Id5fe8f73dff0c914646a76af45738a2e80a8e2b2
Mathieu Desnoyers [Fri, 8 Mar 2024 13:51:33 +0000 (08:51 -0500)]
Introduce mempool test
Signed-off-by: Mathieu Desnoyers <mathieu.desnoyers@efficios.com>
Mathieu Desnoyers [Fri, 8 Mar 2024 12:04:19 +0000 (07:04 -0500)]
Rename percpu-alloc files to mempool
Signed-off-by: Mathieu Desnoyers <mathieu.desnoyers@efficios.com>
Change-Id: I0a1ba042649cf3b2224a90a05e095733e9a597fc
Mathieu Desnoyers [Fri, 8 Mar 2024 12:06:31 +0000 (07:06 -0500)]
Update gitignore
Signed-off-by: Mathieu Desnoyers <mathieu.desnoyers@efficios.com>
Change-Id: I55d5e3571e7115cec5d09fac52ce51fa3601c12d
This page took 0.040211 seconds and 4 git commands to generate.