Mathieu Desnoyers [Thu, 7 Mar 2024 18:47:07 +0000 (13:47 -0500)] 
Fix typos in fprintf
Signed-off-by: Mathieu Desnoyers <mathieu.desnoyers@efficios.com>
Change-Id: Ic82493c649a2389391808f079d9ed58c81e61018
Olivier Dion [Thu, 7 Mar 2024 18:16:03 +0000 (13:16 -0500)] 
percpu-alloc: Add free-list check on pool destroy
Change-Id: I68bab4673aabde92dc0d8321de2d5561c5e7a11c
Signed-off-by: Olivier Dion <odion@efficios.com>
Signed-off-by: Mathieu Desnoyers <mathieu.desnoyers@efficios.com>
Mathieu Desnoyers [Thu, 7 Mar 2024 17:39:34 +0000 (12:39 -0500)]
rseq percpu: print caller ip on double-free/alloc corruption and leak
Signed-off-by: Mathieu Desnoyers <mathieu.desnoyers@efficios.com>
Change-Id: I5b1bc7a2a5711e6c9de4bc82df92a49df5654f8b
Mathieu Desnoyers [Thu, 7 Mar 2024 17:33:24 +0000 (12:33 -0500)]
Fix: add missing \n in fprintf
Signed-off-by: Mathieu Desnoyers <mathieu.desnoyers@efficios.com>
Change-Id: Ibbb61e3b7f66f8d9b7ed125e4123581b2c1b93d9
Mathieu Desnoyers [Thu, 7 Mar 2024 17:32:36 +0000 (12:32 -0500)]
Implement rseq_hweight_ulong with __builtin_popcountl
Signed-off-by: Mathieu Desnoyers <mathieu.desnoyers@efficios.com>
Change-Id: I483d53c7c5402e034b8d468e0ed2ab048184f069
Mathieu Desnoyers [Thu, 7 Mar 2024 17:29:50 +0000 (12:29 -0500)]
Fix destroy_alloc_bitmap size
Signed-off-by: Mathieu Desnoyers <mathieu.desnoyers@efficios.com>
Change-Id: Ic53da70a6fbcbbdd87c885d01d80a5ba65befb5a
Mathieu Desnoyers [Thu, 7 Mar 2024 17:24:43 +0000 (12:24 -0500)]
percpu alloc: Improve robust pools
Signed-off-by: Mathieu Desnoyers <mathieu.desnoyers@efficios.com>
Change-Id: I4966965cca5143640347a4b21fef3e2c353c8366
Olivier Dion [Thu, 7 Mar 2024 15:39:29 +0000 (10:39 -0500)]
percu-allocator: Implement robust pool validation
When set, the RSEQ_POOL_ROBUST flag will add a bitmap to the pool which
is used to track double-free and allocation leaks in the pool.
Signed-off-by: Olivier Dion <odion@efficios.com>
Signed-off-by: Mathieu Desnoyers <mathieu.desnoyers@efficios.com>
Change-Id: I531782ee57d33c9704b9cca44af1a9464f60cb90
Mathieu Desnoyers [Tue, 5 Mar 2024 16:13:55 +0000 (11:13 -0500)]
percpu: remove rseq_percpu_pool_ptr_offset
The rseq_percpu_pool_ptr_offset API is now unused, and it leaks
implementation details about per-cpu pools which may prevent
implementing auto-extensible pools efficiently in the future.
Remove it.
Signed-off-by: Mathieu Desnoyers <mathieu.desnoyers@efficios.com>
Change-Id: Ib62c4af34c04f9c4e5227dab2f487356478b32fa
Mathieu Desnoyers [Tue, 5 Mar 2024 15:06:51 +0000 (10:06 -0500)]
x86-64: Implement/use load_cbne_load_add_load_add_store
Change the current load_add_load_load_add_store for a
load_cbne_load_add_load_add_store, which allows validating that the
percpu pointer did not change since it was loaded from C.
This allows precomputing the address of the percpu memory area in C and
provide it as a second pointer argument.
The comparison approach is prefered to the offset-from-pointer approach
because it does not leak implementation details of the percpu allocator.
Add missing rseq_after_asm_goto() in the static inline function.
Signed-off-by: Mathieu Desnoyers <mathieu.desnoyers@efficios.com>
Change-Id: I3fe9d57d13f7507d5af95ef37391ad36fe2221fe
Mathieu Desnoyers [Tue, 5 Mar 2024 15:10:53 +0000 (10:10 -0500)]
Remove riscv has_load_add_load_load_add_store implementation
The implementation is currently buggy: RSEQ_ASM_OP_R_DEREF_ADDV()
should increment the address _target_ as a final store, not just the
address within the temporary register.
Remove it for now.
Signed-off-by: Mathieu Desnoyers <mathieu.desnoyers@efficios.com>
Change-Id: I0f2a854cec4ab5e58ea6cbb74d40ca29b256d27e
Mathieu Desnoyers [Tue, 5 Mar 2024 15:31:19 +0000 (10:31 -0500)]
param test: membarrier: validate total number of increments
While reviewing the riscv implementation for this test, I noticed that
the final store is only incrementing the content of its register,
without ever storing it to memory.
This passes testing because the increment is effectively a no-op, and
the test never validates that any increment happen in the first place.
Introduce a validation of the increment total to eliminate those
false-positives.
Signed-off-by: Mathieu Desnoyers <mathieu.desnoyers@efficios.com>
Change-Id: Ie9ee1c7902509da75dd85babd07bc180ad0b9ae9
Mathieu Desnoyers [Tue, 5 Mar 2024 13:42:54 +0000 (08:42 -0500)]
percpu allocator: Add flags argument for future extensions
Signed-off-by: Mathieu Desnoyers <mathieu.desnoyers@efficios.com>
Change-Id: I4c507a99351c450ef2c6396af5391f019f8e3d91
Mathieu Desnoyers [Mon, 4 Mar 2024 21:26:50 +0000 (16:26 -0500)]
percpu alloc: introduce rseq mmap attributes
Introduce rseq mmap attribute argument to pool creation, allowing
users to override the memory allocator used to allocate/free the pool
memory.
Move NUMA page move to a separate rseq_percpu_pool_init_numa() API.
Signed-off-by: Mathieu Desnoyers <mathieu.desnoyers@efficios.com>
Change-Id: Ie08c7250fce5f567c7a4aee469e010c5dcf39bc6
Mathieu Desnoyers [Mon, 4 Mar 2024 18:47:34 +0000 (13:47 -0500)]
Cleanup: Remove whiteline
Signed-off-by: Mathieu Desnoyers <mathieu.desnoyers@efficios.com>
Change-Id: I537b82707f1f9a95005da672b98196ab44875e2a
Mathieu Desnoyers [Mon, 4 Mar 2024 16:50:11 +0000 (11:50 -0500)]
Rename RSEQ_ARCH_HAS_OFFSET_DEREF_ADDV to rseq_arch_has_load_add_load_load_add_store
Fix param test to use rseq_arch_has_load_add_load_load_add_store
rather than rseq_arch_has_load_cbne_load_add_store.
Signed-off-by: Mathieu Desnoyers <mathieu.desnoyers@efficios.com>
Change-Id: I373c91dc9178dd69ed159d568090e8c732cf4485
Mathieu Desnoyers [Mon, 4 Mar 2024 16:43:18 +0000 (11:43 -0500)]
Revert "Introduce rseq_load_cbne_load_add_store pseudocode"
This reverts commit
3eff770448484493fa15ea90db643f86c124db6c.
Signed-off-by: Mathieu Desnoyers <mathieu.desnoyers@efficios.com>
Mathieu Desnoyers [Mon, 4 Mar 2024 16:43:08 +0000 (11:43 -0500)]
Revert "x86: Introduce rseq_load_cbne_load_add_store"
This reverts commit
18f21b27f01b9147cd95c538ece07f95c37150a2.
Signed-off-by: Mathieu Desnoyers <mathieu.desnoyers@efficios.com>
Change-Id: Ieffcd50965fe6dcf2c7d067f5fddb4e9f62e82f3
Mathieu Desnoyers [Mon, 4 Mar 2024 16:42:17 +0000 (11:42 -0500)]
Revert "rseq.h: Introduce rseq_load_cbne_load_add_store__ptr"
This is not needed anymore.
This reverts commit
040147907ff31cdc279568ffcb74197650430e4b.
Signed-off-by: Mathieu Desnoyers <mathieu.desnoyers@efficios.com>
Mathieu Desnoyers [Mon, 4 Mar 2024 16:36:03 +0000 (11:36 -0500)]
param tests: membarrier: pre-decode pointer offset
Loading "cpulist->head" outside of the rseq critical section is
unintended: loading cpulist->head loads memory from the per-cpu list
outside of the rseq critical section, which happens to work because the
memory is not actually freed in this specific test case, but would break
if memory would be reclaimed after the rseq fence.
So go back to the originally used rseq_load_add_load_load_add_store__ptr(),
and use the new rseq_percpu_pool_ptr_offset() to pre-calculate the
offset.
Signed-off-by: Mathieu Desnoyers <mathieu.desnoyers@efficios.com>
Change-Id: I5e621c724e143e38a6dfabd26bbf682a6fcc5182
Mathieu Desnoyers [Mon, 4 Mar 2024 16:34:41 +0000 (11:34 -0500)]
percpu alloc: introduce rseq_percpu_pool_ptr_offset
Introduce rseq_percpu_pool_ptr_offset to pre-decode the offset from
all __rseq_percpu pointers for a given pool. This is useful to prepare
offsets that would need to be calculated within rseq critical sections.
Signed-off-by: Mathieu Desnoyers <mathieu.desnoyers@efficios.com>
Change-Id: I42d2669ed5ed81d74aae551b721c9cb04b25fe45
Mathieu Desnoyers [Mon, 4 Mar 2024 15:08:14 +0000 (10:08 -0500)]
Document percpu alloc
Signed-off-by: Mathieu Desnoyers <mathieu.desnoyers@efficios.com>
Change-Id: I8fa4b68e8a68c57da8ab726d0f68ee12aad2f15e
Mathieu Desnoyers [Mon, 4 Mar 2024 14:47:32 +0000 (09:47 -0500)]
rseq percpu: Remove __rseq_percpu annotation from returned pointer
Signed-off-by: Mathieu Desnoyers <mathieu.desnoyers@efficios.com>
Change-Id: Iea04ce173cc529612b1024393794a304b0e7e219
Mathieu Desnoyers [Sun, 3 Mar 2024 17:08:15 +0000 (12:08 -0500)]
x86-32: ip-relative: return to post-call address
This simplifies the assembler returns to the expected (speculated)
address.
Signed-off-by: Mathieu Desnoyers <mathieu.desnoyers@efficios.com>
Change-Id: I68c59f72ac32011d37cc9399414ff328f7b8fdd0
Mathieu Desnoyers [Sat, 2 Mar 2024 21:57:58 +0000 (16:57 -0500)]
param test: x86-32: use ip-relative addressing for RSEQ_INJECT_ASM
Signed-off-by: Mathieu Desnoyers <mathieu.desnoyers@efficios.com>
Change-Id: Id8349df846f5b0491b348b603f0423b726c55b8c
Mathieu Desnoyers [Sat, 2 Mar 2024 21:57:32 +0000 (16:57 -0500)]
x86-32: Save ip reference for ip-relative addressing
Signed-off-by: Mathieu Desnoyers <mathieu.desnoyers@efficios.com>
Change-Id: Ie95e1bf409f0ca0c271f529b694839905d4d0d91
Mathieu Desnoyers [Sat, 2 Mar 2024 17:57:28 +0000 (12:57 -0500)]
x86-32: Use ip-relative addressing to store rseq_cs
Signed-off-by: Mathieu Desnoyers <mathieu.desnoyers@efficios.com>
Change-Id: I294a4844bf8022be480059a028b9a17a5a2dcc4b
Mathieu Desnoyers [Sat, 2 Mar 2024 16:54:28 +0000 (11:54 -0500)]
x86-32: work-around register pressure limitations
Use explicit register clobbers and load input values from "m" input
operands to reduce register pressure.
Note that old gcc does not support output operands for asm goto, so we
cannot simply re-use an input register as output.
Signed-off-by: Mathieu Desnoyers <mathieu.desnoyers@efficios.com>
Change-Id: I745d27f416ef303b1049e8c12b122e5b4a271f63
Mathieu Desnoyers [Sat, 2 Mar 2024 16:19:51 +0000 (11:19 -0500)]
x86-32: Document linker warning
Signed-off-by: Mathieu Desnoyers <mathieu.desnoyers@efficios.com>
Change-Id: I35e1db5eda78ea4b6349039ed34d739b33b81d6d
Mathieu Desnoyers [Sat, 2 Mar 2024 16:08:25 +0000 (11:08 -0500)]
arm: work-around register pressure limitations
The recent move to percpu allocator breaks the 32-bit ARM build due to
register pressure. Fix this by using explicit register clobbers and
loading input values from "m" input operands.
Note that old gcc does not support output operands for asm goto, so we
cannot simply re-use an input register as output.
Signed-off-by: Mathieu Desnoyers <mathieu.desnoyers@efficios.com>
Change-Id: I1d546e427c7f6b62035433815806c8ec275a55f8
Mathieu Desnoyers [Sat, 2 Mar 2024 01:13:52 +0000 (20:13 -0500)]
param tests: percpu memcpu buffer: use percpu alloc
Signed-off-by: Mathieu Desnoyers <mathieu.desnoyers@efficios.com>
Change-Id: I86c975aa8578469f685dcdb4951f4af50f06b3fe
Mathieu Desnoyers [Sat, 2 Mar 2024 00:58:22 +0000 (19:58 -0500)]
param tests: percpu buffer: use percpu alloc
Signed-off-by: Mathieu Desnoyers <mathieu.desnoyers@efficios.com>
Change-Id: I49f29134679931eed4e06140a72cc56592cc12b8
Mathieu Desnoyers [Fri, 1 Mar 2024 19:55:24 +0000 (14:55 -0500)]
param tests: percpu list: use percpu alloc
Signed-off-by: Mathieu Desnoyers <mathieu.desnoyers@efficios.com>
Change-Id: Ie1415e33187902e76dd3d492cd8ed2fe9b980565
Mathieu Desnoyers [Sat, 2 Mar 2024 00:44:59 +0000 (19:44 -0500)]
rseq.h: Introduce rseq_load_cbne_load_add_store__ptr
Signed-off-by: Mathieu Desnoyers <mathieu.desnoyers@efficios.com>
Change-Id: I5011d4115d5dc0064b0d3fc359c037dcda02508c
Mathieu Desnoyers [Sat, 2 Mar 2024 00:38:07 +0000 (19:38 -0500)]
x86: Introduce rseq_load_cbne_load_add_store
Signed-off-by: Mathieu Desnoyers <mathieu.desnoyers@efficios.com>
Change-Id: Id5d8ebfe056ef60ba38a7ab0bf1e368ca324ab25
Mathieu Desnoyers [Sat, 2 Mar 2024 00:36:12 +0000 (19:36 -0500)]
Introduce rseq_load_cbne_load_add_store pseudocode
Signed-off-by: Mathieu Desnoyers <mathieu.desnoyers@efficios.com>
Change-Id: I2c5099af7958591521f6e934da128b54ca3535c3
Mathieu Desnoyers [Sat, 2 Mar 2024 00:32:51 +0000 (19:32 -0500)]
Rename rseq_load_add_load_add_store to rseq_load_add_load_load_add_store
Match the pseudo-code.
Signed-off-by: Mathieu Desnoyers <mathieu.desnoyers@efficios.com>
Change-Id: I48d60a9688cf76936fa1ba7a3b1d6b7412a77e69
Mathieu Desnoyers [Fri, 1 Mar 2024 18:41:20 +0000 (13:41 -0500)]
param test: counter increment: use percpu alloc
Signed-off-by: Mathieu Desnoyers <mathieu.desnoyers@efficios.com>
Change-Id: Ia98154794a80ce917a742961c95d3aef28f7aa1e
Mathieu Desnoyers [Fri, 1 Mar 2024 18:36:46 +0000 (13:36 -0500)]
param test: use __rseq_percpu pointer tagging
Signed-off-by: Mathieu Desnoyers <mathieu.desnoyers@efficios.com>
Change-Id: I2735cd5d08e5f7ed085411934cc5e74938a7ca46
Mathieu Desnoyers [Fri, 1 Mar 2024 18:36:26 +0000 (13:36 -0500)]
Add __rseq_percpu pointer tagging
Signed-off-by: Mathieu Desnoyers <mathieu.desnoyers@efficios.com>
Change-Id: I7a10ac1ad1e97aaaad947e29f377a103e380286d
Mathieu Desnoyers [Fri, 1 Mar 2024 15:56:13 +0000 (10:56 -0500)]
Adapt spinlock param test to percpu alloc
Signed-off-by: Mathieu Desnoyers <mathieu.desnoyers@efficios.com>
Change-Id: I7602983d2aaf9f9a59209c9b3b1cce6fed4ca74d
Mathieu Desnoyers [Fri, 1 Mar 2024 16:07:23 +0000 (11:07 -0500)]
Skip first pool
Signed-off-by: Mathieu Desnoyers <mathieu.desnoyers@efficios.com>
Change-Id: If07b9bbe91fa24b805d7943d4c0cb448cc89c210
Mathieu Desnoyers [Fri, 1 Mar 2024 15:55:45 +0000 (10:55 -0500)]
percpu alloc: add missing extern C
Signed-off-by: Mathieu Desnoyers <mathieu.desnoyers@efficios.com>
Change-Id: I3039487f8e9a130b2fbcb8e6771df449d6021087
Mathieu Desnoyers [Fri, 1 Mar 2024 15:51:13 +0000 (10:51 -0500)]
Add missing include in percpu-alloc.h
Signed-off-by: Mathieu Desnoyers <mathieu.desnoyers@efficios.com>
Change-Id: Ic3d124544c8a651d61f69a872ed03bdfcb638232
Mathieu Desnoyers [Fri, 1 Mar 2024 15:46:02 +0000 (10:46 -0500)]
Set errno on malloc error
Signed-off-by: Mathieu Desnoyers <mathieu.desnoyers@efficios.com>
Change-Id: Ia03a24e63e10907525042f59b1c062ca4f8fe8fc
Mathieu Desnoyers [Fri, 1 Mar 2024 15:17:56 +0000 (10:17 -0500)]
percpu pool set malloc: start search at relevant alloc order
Start search for a pool which can accomodate the @len allocation at an
allocation order where it can fit.
Signed-off-by: Mathieu Desnoyers <mathieu.desnoyers@efficios.com>
Change-Id: Ib887759416120cad78da6a3758d5a8e5781380ad
Mathieu Desnoyers [Fri, 1 Mar 2024 15:14:16 +0000 (10:14 -0500)]
rseq-percpu: Improve documentation
Signed-off-by: Mathieu Desnoyers <mathieu.desnoyers@efficios.com>
Change-Id: Ib9e850d38c7c8bac2a7b94df767275f8f659cb9c
Mathieu Desnoyers [Fri, 1 Mar 2024 15:07:23 +0000 (10:07 -0500)]
Introduce rseq-alloc-utils.h
Signed-off-by: Mathieu Desnoyers <mathieu.desnoyers@efficios.com>
Change-Id: I33970b558ad0d59629737d1a6c0d8531fbcf93e9
Mathieu Desnoyers [Fri, 1 Mar 2024 14:54:47 +0000 (09:54 -0500)]
rseq percpu: Use high bits for pool index
This leaves the low bits available for pointer tagging.
Signed-off-by: Mathieu Desnoyers <mathieu.desnoyers@efficios.com>
Change-Id: I7cebf88a1db4d6cdfd5d287626c67888e80ffbd2
Mathieu Desnoyers [Thu, 29 Feb 2024 22:24:33 +0000 (17:24 -0500)]
rseq percpu alloc: Implement numa support
Signed-off-by: Mathieu Desnoyers <mathieu.desnoyers@efficios.com>
Change-Id: I732b632f476ffef362a1ab486bcf425e4ded6644
Mathieu Desnoyers [Thu, 29 Feb 2024 19:34:08 +0000 (14:34 -0500)]
Implement per-cpu memory allocator
Signed-off-by: Mathieu Desnoyers <mathieu.desnoyers@efficios.com>
Change-Id: I54f43ccec743b7c5311221fac22a0b690a863377
Mathieu Desnoyers [Thu, 29 Feb 2024 15:01:11 +0000 (10:01 -0500)]
Introduce __RSEQ_ASM_DEFINE_CS_FIELDS
The new macro __RSEQ_ASM_DEFINE_CS_FIELDS can be used by
__RSEQ_ASM_DEFINE_TABLE as well as __RSEQ_ASM_DEFINE_ABORT on
arm and mips, thus removing redundant code.
Signed-off-by: Mathieu Desnoyers <mathieu.desnoyers@efficios.com>
Change-Id: Ie03f5d51d20027a3eb3b9eba636b0f5cc890c333
Mathieu Desnoyers [Thu, 29 Feb 2024 14:55:05 +0000 (09:55 -0500)]
Introduce common generic header file
Move:
- RSEQ_ASM_DEFINE_TABLE,
- __RSEQ_ASM_DEFINE_TABLE,
- RSEQ_ASM_DEFINE_EXIT_POINT.
to a generic common header file now that those are exact copy across all
architectures.
Signed-off-by: Mathieu Desnoyers <mathieu.desnoyers@efficios.com>
Change-Id: I1f27e8889b13d263467a1bab92b272cc80aa0743
Mathieu Desnoyers [Thu, 29 Feb 2024 14:45:10 +0000 (09:45 -0500)]
x86: Introduce RSEQ_ASM_U32
This will allow moving __RSEQ_ASM_DEFINE_TABLE and
RSEQ_ASM_DEFINE_EXIT_POINT to a generic common header file.
Signed-off-by: Mathieu Desnoyers <mathieu.desnoyers@efficios.com>
Change-Id: I966b67f00bf39bd0b26f8efd6c4f40cba4caa1cf
Mathieu Desnoyers [Thu, 29 Feb 2024 14:45:04 +0000 (09:45 -0500)]
riscv: Introduce RSEQ_ASM_U64_PTR and RSEQ_ASM_U32
These will allow moving __RSEQ_ASM_DEFINE_TABLE and
RSEQ_ASM_DEFINE_EXIT_POINT to a generic common header file.
Signed-off-by: Mathieu Desnoyers <mathieu.desnoyers@efficios.com>
Change-Id: I9cac7076b9ef9bd7dd4a818b9ad7aae1df45ff5a
Mathieu Desnoyers [Thu, 29 Feb 2024 14:44:55 +0000 (09:44 -0500)]
s390: Introduce RSEQ_ASM_U32
This will allow moving __RSEQ_ASM_DEFINE_TABLE and
RSEQ_ASM_DEFINE_EXIT_POINT to a generic common header file.
Signed-off-by: Mathieu Desnoyers <mathieu.desnoyers@efficios.com>
Change-Id: I0af09a3fa3970efbe523db04154f1969241369a1
Mathieu Desnoyers [Thu, 29 Feb 2024 14:44:47 +0000 (09:44 -0500)]
ppc: Introduce RSEQ_ASM_U32
This will allow moving __RSEQ_ASM_DEFINE_TABLE and
RSEQ_ASM_DEFINE_EXIT_POINT to a generic common header file.
Signed-off-by: Mathieu Desnoyers <mathieu.desnoyers@efficios.com>
Change-Id: Ic476d6dfca6f714cf892f84560884157abbc0360
Mathieu Desnoyers [Thu, 29 Feb 2024 14:44:37 +0000 (09:44 -0500)]
mips: Introduce RSEQ_ASM_U32
This will allow moving __RSEQ_ASM_DEFINE_TABLE and
RSEQ_ASM_DEFINE_EXIT_POINT to a generic common header file.
Signed-off-by: Mathieu Desnoyers <mathieu.desnoyers@efficios.com>
Change-Id: I5a68ce2782bc36d5fc8c030ca7716e41e925cd54
Mathieu Desnoyers [Thu, 29 Feb 2024 14:44:28 +0000 (09:44 -0500)]
arm: Introduce RSEQ_ASM_U32
This will allow moving __RSEQ_ASM_DEFINE_TABLE and
RSEQ_ASM_DEFINE_EXIT_POINT to a generic common header file.
Signed-off-by: Mathieu Desnoyers <mathieu.desnoyers@efficios.com>
Change-Id: I32a0eff75998afb1f8257c3312187ab2938d59a3
Mathieu Desnoyers [Thu, 29 Feb 2024 14:43:58 +0000 (09:43 -0500)]
aarch64: Introduce RSEQ_ASM_U32
This will allow moving __RSEQ_ASM_DEFINE_TABLE and
RSEQ_ASM_DEFINE_EXIT_POINT to a generic common header file.
Signed-off-by: Mathieu Desnoyers <mathieu.desnoyers@efficios.com>
Change-Id: I4496c41d37094983b724fc8bb101688f75fefbbd
Mathieu Desnoyers [Thu, 29 Feb 2024 14:23:50 +0000 (09:23 -0500)]
thread-pointer: suffix include guard with _H
Signed-off-by: Mathieu Desnoyers <mathieu.desnoyers@efficios.com>
Change-Id: Id35f1cb103e1d68bde10785e44c5c16104f000d4
Mathieu Desnoyers [Thu, 29 Feb 2024 14:22:28 +0000 (09:22 -0500)]
arm: Fix big endian, introduce RSEQ_ASM_U64_PTR
Fix support for Big endian ARM by introducing RSEQ_ASM_U64_PTR.
This will allow moving __RSEQ_ASM_DEFINE_TABLE and
RSEQ_ASM_DEFINE_EXIT_POINT to a generic common header file.
Signed-off-by: Mathieu Desnoyers <mathieu.desnoyers@efficios.com>
Change-Id: I6271f217b0e1463968bd7d520ddd0fb61678bb84
Mathieu Desnoyers [Thu, 29 Feb 2024 14:21:07 +0000 (09:21 -0500)]
arm: Cleanup header codying style
Signed-off-by: Mathieu Desnoyers <mathieu.desnoyers@efficios.com>
Change-Id: Iab1aee9103eef2179fcdc8c2171afda3d311379b
Mathieu Desnoyers [Thu, 29 Feb 2024 14:20:07 +0000 (09:20 -0500)]
aarch64: Introduce RSEQ_ASM_U64_PTR
This will allow moving __RSEQ_ASM_DEFINE_TABLE and
RSEQ_ASM_DEFINE_EXIT_POINT to a generic common header file.
Signed-off-by: Mathieu Desnoyers <mathieu.desnoyers@efficios.com>
Change-Id: Ic5470bb851be1b79bbba387a4db2ce0911bbe23d
Mathieu Desnoyers [Thu, 29 Feb 2024 14:19:35 +0000 (09:19 -0500)]
aarch64: Cleanup header coding style
Signed-off-by: Mathieu Desnoyers <mathieu.desnoyers@efficios.com>
Change-Id: I7986cf871be81fc8bb94d0d44d9f61ee41ba0fc1
Mathieu Desnoyers [Wed, 28 Feb 2024 21:49:43 +0000 (16:49 -0500)]
s390: Introduce code deduplication macros
Introduce RSEQ_ASM_U64_PTR() which expresses the type and instance
of the defined variable, allowing deduplication of the following macros:
- __RSEQ_ASM_DEFINE_TABLE,
- RSEQ_ASM_DEFINE_EXIT_POINT.
Separate load/store/op helper macros, RSEQ_ASM_STORE_RSEQ_CS, and
RSEQ_ASM_U64_PTR into their respective ifdef regions, as the resulting
documentation is clearer (and not duplicated).
Signed-off-by: Mathieu Desnoyers <mathieu.desnoyers@efficios.com>
Change-Id: I37c481c9eb726b49dfdf00539c42d68bb500b2c7
Mathieu Desnoyers [Wed, 28 Feb 2024 21:33:02 +0000 (16:33 -0500)]
ppc: Introduce code deduplication macros
Introduce RSEQ_ASM_U64_PTR() which expresses the type and instance
of the defined variable, allowing deduplication of the following macros:
- __RSEQ_ASM_DEFINE_TABLE,
- RSEQ_ASM_DEFINE_EXIT_POINT.
Separate load/store/op helper macros, RSEQ_ASM_STORE_RSEQ_CS, and
RSEQ_ASM_U64_PTR into their respective ifdef regions, as the resulting
documentation is clearer (and not duplicated).
Signed-off-by: Mathieu Desnoyers <mathieu.desnoyers@efficios.com>
Change-Id: I7accb4925e1e294f2d65267641fd4c8dccb4837f
Mathieu Desnoyers [Wed, 28 Feb 2024 21:15:10 +0000 (16:15 -0500)]
x86: Introduce code deduplication macros
Introduce RSEQ_ASM_U64_PTR() which expresses the type and instance
of the defined variable, allowing deduplication of the following macros:
- __RSEQ_ASM_DEFINE_TABLE,
- RSEQ_ASM_DEFINE_EXIT_POINT.
Separate RSEQ_ASM_TP_SEGMENT, RSEQ_ASM_STORE_RSEQ_CS, and
RSEQ_ASM_U64_PTR into their respective ifdef regions, as the resulting
documentation is clearer (and not duplicated).
Signed-off-by: Mathieu Desnoyers <mathieu.desnoyers@efficios.com>
Change-Id: I13e90120f9f50bb898f99f545c3eb205fe9d8437
Mathieu Desnoyers [Wed, 28 Feb 2024 21:12:19 +0000 (16:12 -0500)]
mips: Reorganise code deduplication macros
Introduce RSEQ_ASM_U64_PTR() which expresses the type and instance
of the defined variable.
Separate the RSEQ_ASM_LONG* macros from the 64-bit pointer types, as
those two are unrelated. Having them in the same vincinity is
misleading.
Test on RSEQ_BITS_PER_LONG rather than _MIPS_SZLONG to make it
consistent across architectures.
Use both __BYTE_ORDER and __BIG_ENDIAN to detect byte order.
Signed-off-by: Mathieu Desnoyers <mathieu.desnoyers@efficios.com>
Change-Id: I1a65210e9ab0c00a5625c9feba7971081eb34c35
Michael Jeanson [Fri, 23 Feb 2024 21:20:25 +0000 (16:20 -0500)] 
Re-organise public headers
The structure is inspired by the liburcu project.
- Remove the double namespacing 'rseq/rseq-'
- Add a structured architecture support layout
- Add missing guards on arch headers
- Split the main rseq.h header into utils.h and inject.h
- Add the always_inline attribute to all static inline functions
Change-Id: I6a244e5f364d28d5d8c3d426d6039c4d8aa15a36
Signed-off-by: Michael Jeanson <mjeanson@efficios.com>
Signed-off-by: Mathieu Desnoyers <mathieu.desnoyers@efficios.com>
Michael Jeanson [Fri, 23 Feb 2024 20:22:09 +0000 (15:22 -0500)]
Fix supported autotools versions in README.md
Change-Id: I07257e51544546a8c39325dd05a0a0b99b47a5e2
Signed-off-by: Michael Jeanson <mjeanson@efficios.com>
Signed-off-by: Mathieu Desnoyers <mathieu.desnoyers@efficios.com>
Mathieu Desnoyers [Tue, 27 Feb 2024 22:26:32 +0000 (17:26 -0500)]
RSEQ_ASM_DEFINE_TABLE: use parentheses around parameters on all archs
Signed-off-by: Mathieu Desnoyers <mathieu.desnoyers@efficios.com>
Change-Id: I09ad49a2eb132768d9fca4547d602240bc3d84a4
Mathieu Desnoyers [Tue, 27 Feb 2024 22:12:34 +0000 (17:12 -0500)]
Comment s390 macros implementation
Signed-off-by: Mathieu Desnoyers <mathieu.desnoyers@efficios.com>
Change-Id: Ifaeeed3e7721b9c0f758abef63c286a4d22e5268
Mathieu Desnoyers [Tue, 27 Feb 2024 22:07:49 +0000 (17:07 -0500)]
s390: namespace public header macros under RSEQ_ASM_
Signed-off-by: Mathieu Desnoyers <mathieu.desnoyers@efficios.com>
Change-Id: I777a7b3feb9361c9dbc5327f643fc2d09aaa238e
Mathieu Desnoyers [Tue, 27 Feb 2024 22:04:44 +0000 (17:04 -0500)]
Comment riscv macros implementation
Signed-off-by: Mathieu Desnoyers <mathieu.desnoyers@efficios.com>
Change-Id: I4299061a38443bff9ff0701b73752153d2c3df54
Mathieu Desnoyers [Tue, 27 Feb 2024 22:04:01 +0000 (17:04 -0500)]
riscv: Add missing "inc" parameter to RSEQ_ASM_OP_R_DEREF_ADDV
Signed-off-by: Mathieu Desnoyers <mathieu.desnoyers@efficios.com>
Change-Id: Iff9fb8f46acef741a22065a6c0b976794c4606dd
Mathieu Desnoyers [Tue, 27 Feb 2024 21:48:10 +0000 (16:48 -0500)]
riscv: namespace public header macros under RSEQ_ASM_
Signed-off-by: Mathieu Desnoyers <mathieu.desnoyers@efficios.com>
Change-Id: If8750ceaee9ca4d30309c974e675336ea4290d19
Mathieu Desnoyers [Tue, 27 Feb 2024 21:41:47 +0000 (16:41 -0500)]
mips: Move RSEQ_ASM_DEFINE_ABORT table_label parameter to 4th argument
Makes the first 3 arguments same as other architectures.
Signed-off-by: Mathieu Desnoyers <mathieu.desnoyers@efficios.com>
Change-Id: Icf41ded4ce5860cf8c12a23dff1fe434210844b0
Mathieu Desnoyers [Tue, 27 Feb 2024 21:39:02 +0000 (16:39 -0500)]
arm: Move RSEQ_ASM_DEFINE_ABORT table_label parameter to 4th argument
Makes the first 3 arguments same as other architectures.
Signed-off-by: Mathieu Desnoyers <mathieu.desnoyers@efficios.com>
Change-Id: I855d645ec5e315ef19836adebfd2f9350a3763c5
Mathieu Desnoyers [Tue, 27 Feb 2024 21:34:12 +0000 (16:34 -0500)]
Comment mips macros implementation
Signed-off-by: Mathieu Desnoyers <mathieu.desnoyers@efficios.com>
Change-Id: I6a1646a8cc18cf249b7323afc9ceb0600f001649
Mathieu Desnoyers [Tue, 27 Feb 2024 21:07:16 +0000 (16:07 -0500)]
Comment ppc macros implementation
Signed-off-by: Mathieu Desnoyers <mathieu.desnoyers@efficios.com>
Change-Id: I0b51030c416d875de296ba7f4490bd59ba810f72
Mathieu Desnoyers [Tue, 27 Feb 2024 20:46:58 +0000 (15:46 -0500)]
Comment arm64 macros implementation
Signed-off-by: Mathieu Desnoyers <mathieu.desnoyers@efficios.com>
Change-Id: Id918fa702bfbee014766f949329d1068a0404340
Mathieu Desnoyers [Tue, 27 Feb 2024 20:23:17 +0000 (15:23 -0500)]
Comment arm macros implementation
Signed-off-by: Mathieu Desnoyers <mathieu.desnoyers@efficios.com>
Change-Id: Ibb74479b2219cf2da92a17034785f94cb7dba932
Mathieu Desnoyers [Tue, 27 Feb 2024 19:08:46 +0000 (14:08 -0500)]
Comment x86 asm macros implementation
Combine common code between x86-32/x86-64. Comment asm macro
implementation for both architectures.
Signed-off-by: Mathieu Desnoyers <mathieu.desnoyers@efficios.com>
Change-Id: If6bb763edd4821173babfbc3ac1557f4ae576a86
Mathieu Desnoyers [Tue, 27 Feb 2024 21:21:14 +0000 (16:21 -0500)]
mips: Namespace type helpers under RSEQ_ASM_*
Signed-off-by: Mathieu Desnoyers <mathieu.desnoyers@efficios.com>
Change-Id: If509780615849c0bf69b36a4363a6ff146b22c50
Mathieu Desnoyers [Tue, 27 Feb 2024 20:57:36 +0000 (15:57 -0500)]
ppc: Rename RSEQ_ASM_OP_R_MEMCPY to RSEQ_ASM_OP_R_BYTEWISE_MEMCPY
Signed-off-by: Mathieu Desnoyers <mathieu.desnoyers@efficios.com>
Change-Id: I5ef55c2d23a0cd5b62c0276698a9ccf8e839fc26
Mathieu Desnoyers [Tue, 27 Feb 2024 20:45:32 +0000 (15:45 -0500)]
riscv: Rename RSEQ_ASM_OP_R_BAD_MEMCPY to RSEQ_ASM_OP_R_BYTEWISE_MEMCPY
This is more descriptive.
Signed-off-by: Mathieu Desnoyers <mathieu.desnoyers@efficios.com>
Change-Id: I3f397cc63f60c2d806b1665de9b6bae4106f1db1
Mathieu Desnoyers [Tue, 27 Feb 2024 20:44:30 +0000 (15:44 -0500)]
arm64: Rename RSEQ_ASM_OP_R_BAD_MEMCPY to RSEQ_ASM_OP_R_BYTEWISE_MEMCPY
This is more descriptive.
Signed-off-by: Mathieu Desnoyers <mathieu.desnoyers@efficios.com>
Change-Id: Ic5b33693ae39729859ea734721ffd323a7151c68
Mathieu Desnoyers [Tue, 27 Feb 2024 20:55:22 +0000 (15:55 -0500)]
ppc: Rename RSEQ_ helpers to RSEQ_ASM_
Signed-off-by: Mathieu Desnoyers <mathieu.desnoyers@efficios.com>
Change-Id: I0baec337248866688b2534164a29ab525c18265f
Mathieu Desnoyers [Tue, 27 Feb 2024 19:38:40 +0000 (14:38 -0500)]
Rename RSEQ_TEMPLATE macros
For added clarity, rename:
RSEQ_TEMPLATE_CPU_ID -> RSEQ_TEMPLATE_INDEX_CPU_ID
RSEQ_TEMPLATE_MM_CID -> RSEQ_TEMPLATE_INDEX_MM_CID
RSEQ_TEMPLATE_CPU_ID_NONE -> RSEQ_TEMPLATE_INDEX_NONE
Signed-off-by: Mathieu Desnoyers <mathieu.desnoyers@efficios.com>
Change-Id: Ie3c80d15755b295543ae41b86e8879a61d7a8863
Mathieu Desnoyers [Tue, 27 Feb 2024 19:31:43 +0000 (14:31 -0500)]
x86: Use RSEQ_ASM_ prefix for RSEQ_CPU_ID_OFFSET, RSEQ_CS_OFFSET, RSEQ_MM_CID_OFFSET
This makes this clearly internal and not part of the public API.
Signed-off-by: Mathieu Desnoyers <mathieu.desnoyers@efficios.com>
Change-Id: I695af5866ac8bb8f1b62c705df85f96edaa40fa0
Mathieu Desnoyers [Tue, 27 Feb 2024 19:09:58 +0000 (14:09 -0500)]
x86-32: Remove support for old IDT WinChip and PPRO
The rseq system call first appeared in Linux v4.18, which did not
support the old IDT WinChip Intel clones and PentiumPro CPUs anymore.
Therefore, remove CPU barrier instructions from the smp_rmb() and
smp_wmb() macros given that librseq will realistically never be used on
such hardware.
Expect all x86-32 hardware to be TSO.
Signed-off-by: Mathieu Desnoyers <mathieu.desnoyers@efficios.com>
Change-Id: I07a03006e2eb8ec0928c63f16df94ad42d440b10
Michael Jeanson [Tue, 27 Feb 2024 21:43:19 +0000 (16:43 -0500)]
Fix: seccomp tests require shared libs
Disable the seccomp dependent tests when the project is built without
shared library support.
Change-Id: If17caf056c612a22b649281f25775f55546074f1
Signed-off-by: Michael Jeanson <mjeanson@efficios.com>
Signed-off-by: Mathieu Desnoyers <mathieu.desnoyers@efficios.com>
Mathieu Desnoyers [Sun, 25 Feb 2024 21:25:46 +0000 (16:25 -0500)]
x86-64: change jnz/jz for jne/je after cmp/test
Those are the exact same Jcc instructions (they both depend on the ZF),
but it's clearer to use jne/je after cmp/test rather than jnz/jz.
Note that the jnz instruction is kept when it is used to check the
result of a dec instruction.
Signed-off-by: Mathieu Desnoyers <mathieu.desnoyers@efficios.com>
Change-Id: Ibc2e5902b3047cb291d11d82b7e9dfa530a46891
Mathieu Desnoyers [Fri, 23 Feb 2024 21:09:02 +0000 (16:09 -0500)]
Rename RSEQ_ASM_DEFINE_CMPFAIL to RSEQ_ASM_DEFINE_TEARDOWN
Signed-off-by: Mathieu Desnoyers <mathieu.desnoyers@efficios.com>
Change-Id: Ic386511c72e99b0818c6ff29fd2996d110296e59
Mathieu Desnoyers [Fri, 23 Feb 2024 21:03:08 +0000 (16:03 -0500)]
Rename branch labels from cmpfail to ne/eq
This is done to match the pseudo-code.
Signed-off-by: Mathieu Desnoyers <mathieu.desnoyers@efficios.com>
Change-Id: I56e2bd0c838e16ef2de459da7d15493dd191df4a
Mathieu Desnoyers [Fri, 23 Feb 2024 19:49:09 +0000 (14:49 -0500)]
Update rseq critical section identifiers to match pseudo-code
Apply the following mapping:
rseq_cmpeqv_storev -> rseq_load_cbne_store__ptr
rseq_addv -> rseq_load_add_store__ptr
rseq_cmpnev_storeoffp_load -> rseq_load_cbeq_store_add_load_store__ptr
rseq_offset_deref_addv -> rseq_load_add_load_add_store__ptr
rseq_cmpeqv_cmpeqv_storev -> rseq_load_cbne_load_cbne_store__ptr
rseq_cmpeqv_trystorev_storev -> rseq_load_cbne_store_store__ptr
rseq_cmpeqv_trymemcpy_storev -> rseq_load_cbne_memcpy_store__ptr
Summary of the identifier convention:
- Begin with an "rseq_" prefix,
- Followed by the pseudo-code,
- Followed by __ and the type (or eventually types) on which the API
applies (similar to the approach taken for C++ mangling).
Signed-off-by: Mathieu Desnoyers <mathieu.desnoyers@efficios.com>
Change-Id: I59f251bcf177779ee5f8eb18c629dcd8bf1d633a
Mathieu Desnoyers [Fri, 23 Feb 2024 19:48:28 +0000 (14:48 -0500)]
Add rseq critical section pseudocode documentation
Signed-off-by: Mathieu Desnoyers <mathieu.desnoyers@efficios.com>
Change-Id: Ic7bd7df3566401562a41ffc8112fe95d740c0016
Michael Jeanson [Fri, 23 Feb 2024 15:38:46 +0000 (10:38 -0500)]
tests: add unregistered and no syscall tests
Add tests to validate the state of the public symbols when no
registration has occured and when the rseq syscall is unavailable.
The no syscall test uses seccomp to deny access to rseq, libseccomp is
added as an optionnal build dependency.
Change-Id: I0bf6249fa13bd39af80c21a9d2892cd132644e69
Signed-off-by: Michael Jeanson <mjeanson@efficios.com>
Signed-off-by: Mathieu Desnoyers <mathieu.desnoyers@efficios.com>
Michael Jeanson [Thu, 22 Feb 2024 23:16:13 +0000 (18:16 -0500)]
fix: always set the rseq offset and flags
Even when the rseq syscall is unavailable, set the __rseq_offset to the
correct value as application or library code might try to check the
registration status by reading the cpu_id.
Change-Id: I8f579af236e17a45dfb71c3310bc41a9c17652fd
Signed-off-by: Michael Jeanson <mjeanson@efficios.com>
Signed-off-by: Mathieu Desnoyers <mathieu.desnoyers@efficios.com>
This page took 0.044155 seconds and 4 git commands to generate.
projects / librseq.git / log
