Lai Jiangshan [Mon, 28 Nov 2011 13:11:38 +0000 (08:11 -0500)]
rename min_alloc_size/min_alloc_order
[ Mathieu Desnoyers:
This clarifies that the amount of buckets allocated can differ from
the number of buckets chained together, especially for small hash
tables, where we don't free under min_nr_alloc_buckets. ]
Suggested-by: Mathieu Desnoyers <mathieu.desnoyers@efficios.com>
Signed-off-by: Lai Jiangshan <laijs@cn.fujitsu.com>
Signed-off-by: Mathieu Desnoyers <mathieu.desnoyers@efficios.com>
Lai Jiangshan [Mon, 28 Nov 2011 13:07:32 +0000 (08:07 -0500)]
remove struct rcu_level
Signed-off-by: Lai Jiangshan <laijs@cn.fujitsu.com>
Signed-off-by: Mathieu Desnoyers <mathieu.desnoyers@efficios.com>
Lai Jiangshan [Mon, 28 Nov 2011 13:05:09 +0000 (08:05 -0500)]
it is not required that ht->t.size >= ht->min_table_size anymore
Original code always ensure ht->t.size >= ht->min_table_size.
This can be improved: ht->min_table_size should be used for allocation,
not for the bucket size in use.
Why does the original code need to always ensure this ?
Original code don't do special alloc/free when
ht->t.size < ht->min_table_size
in init_table()/fini_table(), so we have to force
ht->t.size >= ht->min_table_size.
Why does new code become flexible ?
New code use the wrappers, they handle the special cases.
Signed-off-by: Lai Jiangshan <laijs@cn.fujitsu.com>
Signed-off-by: Mathieu Desnoyers <mathieu.desnoyers@efficios.com>
Lai Jiangshan [Mon, 28 Nov 2011 13:01:54 +0000 (08:01 -0500)]
proper wrapper for bucket table alloc and free
Signed-off-by: Lai Jiangshan <laijs@cn.fujitsu.com>
Signed-off-by: Mathieu Desnoyers <mathieu.desnoyers@efficios.com>
Lai Jiangshan [Mon, 28 Nov 2011 13:00:23 +0000 (08:00 -0500)]
introduce bucket_at() and improve readability
Fast path is not changed.
It will slow down very little for slow path.
Signed-off-by: Lai Jiangshan <laijs@cn.fujitsu.com>
Signed-off-by: Mathieu Desnoyers <mathieu.desnoyers@efficios.com>
Lai Jiangshan [Wed, 23 Nov 2011 05:45:57 +0000 (06:45 +0100)]
Add Lai Jiangshan's copyright to rculfhash
Signed-off-by: Lai Jiangshan <laijs@cn.fujitsu.com>
Signed-off-by: Mathieu Desnoyers <mathieu.desnoyers@efficios.com>
Lai Jiangshan [Wed, 23 Nov 2011 05:44:35 +0000 (06:44 +0100)]
Update comments of cds_lfht_new()
Signed-off-by: Lai Jiangshan <laijs@cn.fujitsu.com>
Signed-off-by: Mathieu Desnoyers <mathieu.desnoyers@efficios.com>
Mathieu Desnoyers [Wed, 16 Nov 2011 12:23:19 +0000 (07:23 -0500)]
rculfhash: constify all key arguments passed to API
The hash table never needs to modify the key, it is only ever used for
"match", so it should always be received as a const argument.
Reported-by: Stephen Hemminger <shemminger@vyatta.com>
Signed-off-by: Mathieu Desnoyers <mathieu.desnoyers@efficios.com>
Lai Jiangshan [Mon, 14 Nov 2011 12:56:10 +0000 (07:56 -0500)]
Fix arguments order
use: hash, match, key
Signed-off-by: Lai Jiangshan <laijs@cn.fujitsu.com>
Signed-off-by: Mathieu Desnoyers <mathieu.desnoyers@efficios.com>
Mathieu Desnoyers [Thu, 10 Nov 2011 18:55:07 +0000 (13:55 -0500)]
rculfhash: Add parenthesis around macro arg use in iterator macro
Signed-off-by: Mathieu Desnoyers <mathieu.desnoyers@efficios.com>
Mathieu Desnoyers [Wed, 9 Nov 2011 21:18:42 +0000 (16:18 -0500)]
Add hash table iterator macros
Signed-off-by: Mathieu Desnoyers <mathieu.desnoyers@efficios.com>
Mathieu Desnoyers [Wed, 9 Nov 2011 19:50:34 +0000 (14:50 -0500)]
Remove whiteline
Signed-off-by: Mathieu Desnoyers <mathieu.desnoyers@efficios.com>
Lai Jiangshan [Sat, 5 Nov 2011 13:52:41 +0000 (09:52 -0400)]
rculfhash: rename dummy to bucket
[ Edit by Mathieu Desnoyers: rebased. ]
Signed-off-by: Lai Jiangshan <laijs@cn.fujitsu.com>
Signed-off-by: Mathieu Desnoyers <mathieu.desnoyers@efficios.com>
Lai Jiangshan [Sat, 5 Nov 2011 13:48:18 +0000 (09:48 -0400)]
rculfhash: Move key out of struct lfht_test_node, move key_len to caller
Make struct lfht_test_node only contains basic field.
Let user take the responsibility to handle the <key>(hash_set)
or <key,value>(hash_map) management and calculation.
[ Edit by Mathieu Desnoyers: rebased on top of commit
0422d92c2d658f6093b8209f75808efd2109a110 ]
Signed-off-by: Lai Jiangshan <laijs@cn.fujitsu.com>
Signed-off-by: Mathieu Desnoyers <mathieu.desnoyers@efficios.com>
Mathieu Desnoyers [Sat, 5 Nov 2011 13:10:12 +0000 (09:10 -0400)]
rculfhash: Remove leftover hash_seed field
Signed-off-by: Mathieu Desnoyers <mathieu.desnoyers@efficios.com>
Mathieu Desnoyers [Sat, 5 Nov 2011 13:01:09 +0000 (09:01 -0400)]
rculfhash: extract compare_fct and hash_fct from the hash table
Leave the compare_fct and hash_fct to the caller, which can therefore
become the only one which needs to know the layout of struct
cds_lfht_node (except for the _cds_lfht_node part).
This will enable us to remove the cds_lfht_node structure from the hash
table (moving it entirely to the caller), replacing it by
_cds_lfht_node, which is really the only part the hash table needs to
know about.
Use-cases that can benefit from this extra flexibility:
>> Sometimes the caller don't have the key or the keys which are big
>> are saved separated. Example: file server
>> Or key comparation is too slow.
>> Or caller just need approximate lookup.
>> Or caller want to compare keys with different strategies for
>> performance.
Suggested-by: Lai Jiangshan <laijs@cn.fujitsu.com>
Signed-off-by: Mathieu Desnoyers <mathieu.desnoyers@efficios.com>
Mathieu Desnoyers [Thu, 3 Nov 2011 14:58:54 +0000 (10:58 -0400)]
Merge branch 'master' into urcu/ht-shrink
Mathieu Desnoyers [Wed, 14 Sep 2011 17:36:12 +0000 (13:36 -0400)]
qsbr vs call_rcu : remove exit assertion
Signed-off-by: Mathieu Desnoyers <mathieu.desnoyers@efficios.com>
Lai Jiangshan [Wed, 2 Nov 2011 17:22:30 +0000 (13:22 -0400)]
rculfhash: Move "struct rcu_head head" out of "struct cds_lfht_node"
It is user's responsibility to free it by call_rcu(),
synchronize_rcu() or defer_rcu().
Signed-off-by: Lai Jiangshan <laijs@cn.fujitsu.com>
Signed-off-by: Mathieu Desnoyers <mathieu.desnoyers@efficios.com>
Lai Jiangshan [Wed, 2 Nov 2011 17:20:57 +0000 (13:20 -0400)]
rculfhash, test: wrap struct cds_lfht_node
Use struct lfht_test_node for test instead of struct cds_lfht_node.
We can add test related things in struct lfht_test_node in future.
Signed-off-by: Lai Jiangshan <laijs@cn.fujitsu.com>
Signed-off-by: Mathieu Desnoyers <mathieu.desnoyers@efficios.com>
Lai Jiangshan [Wed, 2 Nov 2011 15:31:14 +0000 (11:31 -0400)]
rculfhash: Change lazy shrink strategy
We can aggressively grow a ht,
but we should conservatively shrink a ht.
When we try to shrink the ht after a deletion,
but if a growing is required after it, we should
stop this shrink.
But it is more complicated to implement it, so we use more
conservative strategy to shrink a ht to gain simple code:
We stop to (lazy) shrink when we found growing is
in progress.
[ Edit by Mathieu Desnoyers: Add documentation, cleanup. ]
Signed-off-by: Lai Jiangshan <laijs@cn.fujitsu.com>
Signed-off-by: Mathieu Desnoyers <mathieu.desnoyers@efficios.com>
Lai Jiangshan [Wed, 2 Nov 2011 15:21:35 +0000 (11:21 -0400)]
rculfhash: merge duplicated code of cds_lfht_resize_lazy_*()
Signed-off-by: Lai Jiangshan <laijs@cn.fujitsu.com>
Signed-off-by: Mathieu Desnoyers <mathieu.desnoyers@efficios.com>
Lai Jiangshan [Wed, 2 Nov 2011 15:20:11 +0000 (11:20 -0400)]
rculfhash: Fix ht lazy grow logic.
"size < target_size" in cds_lfht_resize_lazy() which is always true
confuses me.
I think we should grow the ht only when we success to increase
the resize_target.
Signed-off-by: Lai Jiangshan <laijs@cn.fujitsu.com>
Signed-off-by: Mathieu Desnoyers <mathieu.desnoyers@efficios.com>
Mathieu Desnoyers [Wed, 2 Nov 2011 00:17:16 +0000 (20:17 -0400)]
rculfhash: rename likely/unlikely (add caa_ prefix)
Signed-off-by: Mathieu Desnoyers <mathieu.desnoyers@efficios.com>
Mathieu Desnoyers [Wed, 2 Nov 2011 00:15:55 +0000 (20:15 -0400)]
Merge branch 'master' into urcu/ht-shrink
Fixed conflicts:
tests/test_urcu_qsbr.c
tests/test_urcu_qsbr_gc.c
Signed-off-by: Mathieu Desnoyers <mathieu.desnoyers@efficios.com>
Mathieu Desnoyers [Tue, 1 Nov 2011 23:58:52 +0000 (19:58 -0400)]
Rename likely/unlikely to caa_likely/caa_unlikely
This fixes namespace conflicts.
Signed-off-by: Mathieu Desnoyers <mathieu.desnoyers@efficios.com>
Mathieu Desnoyers [Tue, 1 Nov 2011 19:45:01 +0000 (15:45 -0400)]
Add structure descriptions, remove redundant clear_flag()
Reported-by: Paul E. McKenney <paulmck@linux.vnet.ibm.com>
Signed-off-by: Mathieu Desnoyers <mathieu.desnoyers@efficios.com>
Paul E. McKenney [Tue, 1 Nov 2011 18:48:31 +0000 (14:48 -0400)]
Fix CAA_BITS_PER_lONG typo
Should instead be CAA_BITS_PER_LONG.
Signed-off-by: Paul E. McKenney <paul.mckenney@linaro.org>
Signed-off-by: Mathieu Desnoyers <mathieu.desnoyers@efficios.com>
Lai Jiangshan [Tue, 1 Nov 2011 17:09:13 +0000 (13:09 -0400)]
rculfhash: add min_alloc_size test parameter
Signed-off-by: Lai Jiangshan <laijs@cn.fujitsu.com>
Signed-off-by: Mathieu Desnoyers <mathieu.desnoyers@efficios.com>
Lai Jiangshan [Tue, 1 Nov 2011 17:08:13 +0000 (13:08 -0400)]
rculfhash: Fix min_alloc_size bug
When I change MIN_ALLOC_ORDER macro to parameter,
I forgot set min_alloc_order before used it which causes bug.
Signed-off-by: Lai Jiangshan <laijs@cn.fujitsu.com>
Signed-off-by: Mathieu Desnoyers <mathieu.desnoyers@efficios.com>
Lai Jiangshan [Tue, 1 Nov 2011 16:54:04 +0000 (12:54 -0400)]
rculfhash: Fix ht allocation bug
Fix a bug introduced by Lai Jiangshan <laijs@cn.fujitsu.com>:
alloc_split_items_count() use a wrong flags.
ht->flags should be inited earlier.
Signed-off-by: Lai Jiangshan <laijs@cn.fujitsu.com>
Signed-off-by: Mathieu Desnoyers <mathieu.desnoyers@efficios.com>
Lai Jiangshan [Fri, 28 Oct 2011 08:16:34 +0000 (10:16 +0200)]
rculfhash: add CDS_LFHT_ACCOUNTING flag
[ Edit by Mathieu Desnoyers: fix null pointer in poison_free. ]
Signed-off-by: Lai Jiangshan <laijs@cn.fujitsu.com>
Signed-off-by: Mathieu Desnoyers <mathieu.desnoyers@efficios.com>
Lai Jiangshan [Fri, 28 Oct 2011 07:31:51 +0000 (09:31 +0200)]
rculfhash: use DEFAULT_SPLIT_COUNT_MASK for !HAVE_SYSCONF
Now, item accounting is always available.
Signed-off-by: Lai Jiangshan <laijs@cn.fujitsu.com>
Signed-off-by: Mathieu Desnoyers <mathieu.desnoyers@efficios.com>
Lai Jiangshan [Fri, 28 Oct 2011 07:29:47 +0000 (09:29 +0200)]
rculfhash: use hash for index if !HAVE_SCHED_GETCPU
Signed-off-by: Lai Jiangshan <laijs@cn.fujitsu.com>
Signed-off-by: Mathieu Desnoyers <mathieu.desnoyers@efficios.com>
Lai Jiangshan [Fri, 28 Oct 2011 07:28:14 +0000 (09:28 +0200)]
rculfhash: rename percpu_count to split_count
Signed-off-by: Lai Jiangshan <laijs@cn.fujitsu.com>
Signed-off-by: Mathieu Desnoyers <mathieu.desnoyers@efficios.com>
Mathieu Desnoyers [Fri, 28 Oct 2011 06:18:30 +0000 (08:18 +0200)]
rculfhash: Cast to struct rcu_level * instead of void * (cleanup)
Signed-off-by: Mathieu Desnoyers <mathieu.desnoyers@efficios.com>
Lai Jiangshan [Fri, 28 Oct 2011 06:14:07 +0000 (08:14 +0200)]
rculfhash: directly lookup bucket in the first order table
[ Edit by Mathieu Desnoyers: add a dbg_printf so the start-of-table
lookup prints the same dbg messages as normal lookups. ]
Signed-off-by: Lai Jiangshan <laijs@cn.fujitsu.com>
Signed-off-by: Mathieu Desnoyers <mathieu.desnoyers@efficios.com>
Lai Jiangshan [Fri, 28 Oct 2011 06:09:40 +0000 (08:09 +0200)]
Avoid alloc small memory pieces
Add min_alloc_size (minimum allocation size) for allocation.
Now we can gain better cache locality by specifying larger minimum
allocation size.
Signed-off-by: Lai Jiangshan <laijs@cn.fujitsu.com>
Signed-off-by: Mathieu Desnoyers <mathieu.desnoyers@efficios.com>
Lai Jiangshan [Fri, 28 Oct 2011 06:02:12 +0000 (08:02 +0200)]
rculfhash: Remove unneed branches
By the effect of previous patch, some unneed branches can be removed.
One of them is in the fast path.
These branches are mostly special-cases for the node "0", which is
always populated after hash table new.
Signed-off-by: Lai Jiangshan <laijs@cn.fujitsu.com>
Signed-off-by: Mathieu Desnoyers <mathieu.desnoyers@efficios.com>
Lai Jiangshan [Fri, 28 Oct 2011 05:57:44 +0000 (07:57 +0200)]
rculfhash: create dummy nodes directly when create lfht
make cds_lfht_new() can be called earlier(before rcu is initialized
..etc) If caller want to *parallelly* init the dummy nodes with large
init_size, he can use cds_lfht_new()+cds_lfht_resize() combination.
Signed-off-by: Lai Jiangshan <laijs@cn.fujitsu.com>
Signed-off-by: Mathieu Desnoyers <mathieu.desnoyers@efficios.com>
Lai Jiangshan [Fri, 28 Oct 2011 04:23:40 +0000 (06:23 +0200)]
rculfhash: fix uniquely add vs cds_lfht_next observation semantic
Is it a bug?
It depends.
Current code(a combination of add_replace and add_unique)
does never generate duplicated keys, but it only
guarantees snapshot semantic: If we take a snapshot of the
hash table, we can observe that there are no duplicated keys
in the snapshot.
But this definition don't work in practice, we don't take snapshot
before observe, we actually observe them one by one.
Example:
cds_lfht_lookup(&iter);
while (should_end(&iter)) {
do_something_with(&iter);
cds_lfht_next(&iter);
}
In the old code, we may do_something_with/observe 2 duplicate nodes
in this practice!
Here is an identical-hash-value node chain with no duplicated keys
-->[p1]-->[p2]-->[p3]-->[p4]-->
Now thread 1 is the observing thread, it is travelling and do
something with the nodes. thread 2 deletes [p1], thread 3
add_unique [p1'] with the same key as [p1]
thread 1 thread 2 thread 3
---------------------------------------------------
do something with [p1]
do something with [p2]
delete [p1]
uniquely add [p1']
(-->[p2]-->[p3]-->[p4]-->[p1']-->)
do something with [p3]
do something with [p4]
do something with [p1']
---------------------------------------------------
Now, thread 1 unexpectly handles the same key twice.
It is BUG!
If we just provide snapshot semantic, it is not useful.
So we need to provide more strict add_replace/add_unique semantic.
1) no duplicated keys should ever be observable in the table(snapshot semantic)
2) no duplicated keys should ever be observed by forward iteration in the table.
The fix:
To provide forward-iteration-observation semantic, I ensure the new inserted
node(add_unique/add_replace) is inserted as the first node of
the identical-hash-value node chain.
A little another behavior is changed in this patch, we will no do gc in
cds_lfht_next_duplicate(), because it does need.
[ Note by Mathieu Desnoyers:
I agree that your semantic fix is needed for ensuring that cds_lfht_next
is semantically correct with add_unique semantic.
Just a note: the most important lookup/get next API semantic in my view
is the lookup + cds_lfht_next_duplicate, which is the typical use case
for looking up and node and getting all the duplicate keys.
Ideally, we want both get_next and get_next_duplicate to provide the
correct semantic.
I think the reason your fix here seems to work and not break the
behavior of cds_lfht_next_duplicate is because you only put nodes added
with add_unique at the beginning of the same-hash-value-chain. It's a
good thing that you don't try doing this for other add mode (allowing
duplicates), because cds_lfht_next_duplicate requires duplicate nodes to
be next one to another. And you patch keeps this behavior.
]
Signed-off-by: Lai Jiangshan <laijs@cn.fujitsu.com>
Signed-off-by: Mathieu Desnoyers <mathieu.desnoyers@efficios.com>
Lai Jiangshan [Fri, 28 Oct 2011 03:15:46 +0000 (05:15 +0200)]
Cleanup order semantic
[ Edit by Mathieu Desnoyers: renamed "end_order" to "last_order", to
match the "first_order" inclusive semantic. Normally, "end" excludes
the range limit value, but "last" includes it. We use an inclusive
range end value here. ]
Signed-off-by: Lai Jiangshan <laijs@cn.fujitsu.com>
Signed-off-by: Mathieu Desnoyers <mathieu.desnoyers@efficios.com>
Lai Jiangshan [Thu, 27 Oct 2011 07:01:08 +0000 (09:01 +0200)]
Comment cleanup: fix ascii art
Fix an incorrect arrow in ascii art description of the table.
Signed-off-by: Lai Jiangshan <laijs@cn.fujitsu.com>
Signed-off-by: Mathieu Desnoyers <mathieu.desnoyers@efficios.com>
Lai Jiangshan [Thu, 27 Oct 2011 05:14:50 +0000 (07:14 +0200)]
Cleanup _cds_lfht_del()
Signed-off-by: Lai Jiangshan <laijs@cn.fujitsu.com>
Signed-off-by: Mathieu Desnoyers <mathieu.desnoyers@efficios.com>
Lai Jiangshan [Thu, 27 Oct 2011 05:11:47 +0000 (07:11 +0200)]
Cleanup _cds_lfht_replace()
[ Edit by Mathieu Desnoyers: re-inserted a comment. Edit title. ]
Signed-off-by: Lai Jiangshan <laijs@cn.fujitsu.com>
Signed-off-by: Mathieu Desnoyers <mathieu.desnoyers@efficios.com>
Lai Jiangshan [Thu, 27 Oct 2011 05:07:56 +0000 (07:07 +0200)]
Remove dead code in _cds_lfht_gc_bucket()
Signed-off-by: Lai Jiangshan <laijs@cn.fujitsu.com>
Signed-off-by: Mathieu Desnoyers <mathieu.desnoyers@efficios.com>
Mathieu Desnoyers [Thu, 27 Oct 2011 04:30:49 +0000 (06:30 +0200)]
Move replace code out from _cds_lfht_add()
Original patch from Lai Jiangshan, reworked significantly to keep the
iterator semantic, where a "next" node is never re-read after being
looked up within a RCU read-side critical section after it has been
validated.
Although strictly speaking the "replace" code re-checks that the next
node is not logically removed with a cmpxchg, I prefer to ensure that
all lookup paths behave similarly. It will make it easier to track
behavior later on if they all follow the same rules.
Reported-by: Lai Jiangshan <laijs@cn.fujitsu.com>
Signed-off-by: Mathieu Desnoyers <mathieu.desnoyers@efficios.com>
Lai Jiangshan [Mon, 17 Oct 2011 14:40:06 +0000 (10:40 -0400)]
Rename the parameter name of _cds_lfht_replace()
Although _cds_lfht_replace() is not a plublic API,
but a proper name will help for reading or maintainance.
Signed-off-by: Lai Jiangshan <laijs@cn.fujitsu.com>
Signed-off-by: Mathieu Desnoyers <mathieu.desnoyers@efficios.com>
Lai Jiangshan [Mon, 17 Oct 2011 14:34:36 +0000 (10:34 -0400)]
Fix dummy node insertion bug
dummy node is the first node of the identical-hash-value chain
Signed-off-by: Lai Jiangshan <laijs@cn.fujitsu.com>
Signed-off-by: Mathieu Desnoyers <mathieu.desnoyers@efficios.com>
Lai Jiangshan [Mon, 17 Oct 2011 14:26:58 +0000 (10:26 -0400)]
compare hash value before call compare_fct()
[ Edit by Mathieu Desnoyers:
This is an optimisation that checks if the reverse hash value is
equal before calling the comparison function. Given comparing a
reverse hash is much faster than comparison, this accelarates the
lookups and add. ]
Signed-off-by: Lai Jiangshan <laijs@cn.fujitsu.com>
Signed-off-by: Mathieu Desnoyers <mathieu.desnoyers@efficios.com>
Lai Jiangshan [Mon, 17 Oct 2011 14:19:10 +0000 (10:19 -0400)]
rculfhash: Simplify lookup_bucket()
They are the same, but I don't think the compiler can optimize it.
And it also helps for understanding the following code.
[ Edit by Mathieu Desnoyers:
- Add a comment that describes the equivalence between get_count_order
and fls for this lookup of index + 1. ]
Signed-off-by: Lai Jiangshan <laijs@cn.fujitsu.com>
Signed-off-by: Mathieu Desnoyers <mathieu.desnoyers@efficios.com>
Lai Jiangshan [Mon, 17 Oct 2011 14:10:27 +0000 (10:10 -0400)]
rculfhash: avoid unneed garbage collect
[ Edit by Mathieu Desnoyers:
If we have a concurrent removal executing while the add is performed,
I just want to ensure we agree that this scenario will be correct:
1) removal succeeds. We have flagged
a node as "logically removed", but
it is still in the linked list.
2) garbage collection is initiated.
3) add succeeds. We insert a
node with reverse hash value
higher than the logically
removed node. This means the
add takes care of garbage
collecting the logically
removed node prior to adding
the new node.
4) garbage collection of logically
removed node fails
(we ignore the return value of
(void) uatomic_cmpxchg(&iter_prev->p.next, iter, new_next);)
retry and iterate until we find the reverse
hash value higher than the logically
remove node.
So I think this is OK: given that the add path takes care of garbage
collecting any logically removed node located prior to the insert
position, we don't need to perform gargage collection after the
insertion. ]
Signed-off-by: Lai Jiangshan <laijs@cn.fujitsu.com>
Signed-off-by: Mathieu Desnoyers <mathieu.desnoyers@efficios.com>
Lai Jiangshan [Mon, 17 Oct 2011 13:46:10 +0000 (09:46 -0400)]
rculfhash: merge duplicated code for bucket lookup
[ Edit by Mathieu Desnoyers: also remove unnecessary lookups that end up
returning an already looked up node. Same behavior is guaranteed
because the hash and size stay invariant across retry. ]
Signed-off-by: Lai Jiangshan <laijs@cn.fujitsu.com>
Signed-off-by: Mathieu Desnoyers <mathieu.desnoyers@efficios.com>
Lai Jiangshan [Mon, 17 Oct 2011 13:32:00 +0000 (09:32 -0400)]
rculfhash: make struct rcu_level size power of 2
By adding a small slowpath overhead (added synchronize_rcu call in the
last iteration of the resize), we can reduce the amount of wasted memory
for memory allocators that allocate power of two memory areas. This is
achieved by removing the call_rcu head from struct rcu_level.
[ Edit by Mathieu Desnoyers:
- add comment about need to manually update the allocation size of
fields are added to struct rcu_level.
- create a more explanatory title and changelog. ]
Signed-off-by: Lai Jiangshan <laijs@cn.fujitsu.com>
Signed-off-by: Mathieu Desnoyers <mathieu.desnoyers@efficios.com>
Mathieu Desnoyers [Sat, 15 Oct 2011 14:08:46 +0000 (09:08 -0500)]
Reinsert missing test_urcu_*.c files (missing in rename)
Signed-off-by: Mathieu Desnoyers <mathieu.desnoyers@efficios.com>
Mathieu Desnoyers [Sat, 15 Oct 2011 14:08:46 +0000 (09:08 -0500)]
Reinsert missing test_urcu_*.c files (missing in rename)
Signed-off-by: Mathieu Desnoyers <mathieu.desnoyers@efficios.com>
Lai Jiangshan [Fri, 14 Oct 2011 14:59:42 +0000 (09:59 -0500)]
rculfhash: simplify get_count_order()
[ Edit by Mathieu Desnoyers:
- Updated comments.
- Quick benchmark on Intel(R) Atom(TM) CPU Z520 @ 1.33GHz:
get_count_order_ulong, from 0 to
100000000:
previous: 3.840s
new: 3.187s
- Test comparing bit-exactness for ranges near 0:
/*
* fls: returns the position of the most significant bit.
* Returns 0 if no bit is set, else returns the position of the most
* significant bit (from 1 to 32 on 32-bit, from 1 to 64 on 64-bit).
*/
static inline
unsigned int fls_u32(uint32_t x)
{
int r;
asm("bsrl %1,%0\n\t"
"jnz 1f\n\t"
"movl $-1,%0\n\t"
"1:\n\t"
: "=r" (r) : "rm" (x));
return r + 1;
}
static inline
unsigned int fls_u64(uint64_t x)
{
long r;
asm("bsrq %1,%0\n\t"
"jnz 1f\n\t"
"movq $-1,%0\n\t"
"1:\n\t"
: "=r" (r) : "rm" (x));
return r + 1;
}
static __attribute__((unused))
unsigned int fls_u64(uint64_t x)
{
unsigned int r = 64;
if (!x)
return 0;
if (!(x & 0xFFFFFFFF00000000ULL)) {
x <<= 32;
r -= 32;
}
if (!(x & 0xFFFF000000000000ULL)) {
x <<= 16;
r -= 16;
}
if (!(x & 0xFF00000000000000ULL)) {
x <<= 8;
r -= 8;
}
if (!(x & 0xF000000000000000ULL)) {
x <<= 4;
r -= 4;
}
if (!(x & 0xC000000000000000ULL)) {
x <<= 2;
r -= 2;
}
if (!(x & 0x8000000000000000ULL)) {
x <<= 1;
r -= 1;
}
return r;
}
static __attribute__((unused))
unsigned int fls_u32(uint32_t x)
{
unsigned int r = 32;
if (!x)
return 0;
if (!(x & 0xFFFF0000U)) {
x <<= 16;
r -= 16;
}
if (!(x & 0xFF000000U)) {
x <<= 8;
r -= 8;
}
if (!(x & 0xF0000000U)) {
x <<= 4;
r -= 4;
}
if (!(x & 0xC0000000U)) {
x <<= 2;
r -= 2;
}
if (!(x & 0x80000000U)) {
x <<= 1;
r -= 1;
}
return r;
}
unsigned int fls_ulong(unsigned long x)
{
return fls_u32(x);
return fls_u64(x);
}
int get_count_order_u32(uint32_t x)
{
int order;
order = fls_u32(x) - 1;
if (x & (x - 1))
order++;
return order;
}
int get_count_order_ulong(unsigned long x)
{
int order;
order = fls_ulong(x) - 1;
if (x & (x - 1))
order++;
return order;
}
int test_get_count_order_u32(uint32_t x)
{
if (!x)
return -1;
return fls_u32(x - 1);
}
/* find the minimum order that x <= (1UL << order) */
int test_get_count_order_ulong(unsigned long x)
{
if (!x)
return -1;
return fls_ulong(x - 1);
}
int main(int argc, char **argv)
{
unsigned long i;
for (i = 0UL; i < 10000; i++) {
if (get_count_order_ulong(i) != test_get_count_order_ulong(i))
printf("Error for %lu\n", i);
}
for (i = 4294967293UL; i != 0; i++) {
if (get_count_order_ulong(i) != test_get_count_order_ulong(i))
printf("Error for %lu\n", i);
}
return 0;
}
]
Signed-off-by: Lai Jiangshan <laijs@cn.fujitsu.com>
Signed-off-by: Mathieu Desnoyers <mathieu.desnoyers@efficios.com>
Lai Jiangshan [Fri, 14 Oct 2011 14:14:04 +0000 (09:14 -0500)]
rculfhash: use dbg_printf() for grow/shrink printout
Signed-off-by: Lai Jiangshan <laijs@cn.fujitsu.com>
Signed-off-by: Mathieu Desnoyers <mathieu.desnoyers@efficios.com>
Lai Jiangshan [Fri, 14 Oct 2011 13:56:54 +0000 (08:56 -0500)]
rculfhash: merge thread_id into struct partition_resize_work
Signed-off-by: Lai Jiangshan <laijs@cn.fujitsu.com>
Signed-off-by: Mathieu Desnoyers <mathieu.desnoyers@efficios.com>
Lai Jiangshan [Fri, 14 Oct 2011 13:55:16 +0000 (08:55 -0500)]
rculfhash: remove unused rcu_head in partition_resize_work
Signed-off-by: Lai Jiangshan <laijs@cn.fujitsu.com>
Signed-off-by: Mathieu Desnoyers <mathieu.desnoyers@efficios.com>
Lai Jiangshan [Fri, 14 Oct 2011 13:46:30 +0000 (08:46 -0500)]
rename test_qsbr to test_urcu_qsbr
[ edit by Mathieu Desnoyers: update runtests*.sh scripts accordingly ]
Signed-off-by: Lai Jiangshan <laijs@cn.fujitsu.com>
Signed-off-by: Mathieu Desnoyers <mathieu.desnoyers@efficios.com>
Mathieu Desnoyers [Wed, 5 Oct 2011 02:42:29 +0000 (22:42 -0400)]
rculfhash test: handle write return value
Signed-off-by: Mathieu Desnoyers <mathieu.desnoyers@efficios.com>
Mathieu Desnoyers [Wed, 5 Oct 2011 02:34:10 +0000 (22:34 -0400)]
rcuhash test: fix 32-bit type warning
Signed-off-by: Mathieu Desnoyers <mathieu.desnoyers@efficios.com>
Mathieu Desnoyers [Wed, 5 Oct 2011 02:07:32 +0000 (22:07 -0400)]
Merge branch 'master' into urcu/ht-shrink
Mathieu Desnoyers [Wed, 5 Oct 2011 02:01:50 +0000 (22:01 -0400)]
urcu-pointer: fix rcu_set_pointer unset return value
The problem only affected non-_LGPL_SOURCE configs.
Reported-by: Stephen Hemminger <shemminger@vyatta.com>
Fix-proposed-by: "Paul E. McKenney" <paulmck@linux.vnet.ibm.com>
Signed-off-by: Mathieu Desnoyers <mathieu.desnoyers@efficios.com>
Mathieu Desnoyers [Sun, 2 Oct 2011 16:56:36 +0000 (12:56 -0400)]
Merge branch 'master' into urcu/ht-shrink
Conflicts:
urcu-call-rcu.h
Signed-off-by: Mathieu Desnoyers <mathieu.desnoyers@efficios.com>
Mathieu Desnoyers [Sun, 2 Oct 2011 16:53:44 +0000 (12:53 -0400)]
Enhance API.txt documentation, add to Makefile as EXTRA_DIST
Signed-off-by: Mathieu Desnoyers <mathieu.desnoyers@efficios.com>
Mathieu Desnoyers [Fri, 30 Sep 2011 17:37:07 +0000 (13:37 -0400)]
rculfhash: do not sample in_progress_destroy in the middle of a level
Caused destroy assert/segfaults.
Signed-off-by: Mathieu Desnoyers <mathieu.desnoyers@efficios.com>
Mathieu Desnoyers [Fri, 30 Sep 2011 16:59:33 +0000 (12:59 -0400)]
rculfhash: break resize loop on destroy
Signed-off-by: Mathieu Desnoyers <mathieu.desnoyers@efficios.com>
Mathieu Desnoyers [Fri, 30 Sep 2011 16:50:11 +0000 (12:50 -0400)]
rculfhash: decrement resize cnt on destroy
Signed-off-by: Mathieu Desnoyers <mathieu.desnoyers@efficios.com>
Mathieu Desnoyers [Fri, 30 Sep 2011 16:45:23 +0000 (12:45 -0400)]
Check for in progress destroy before calling call_rcu thread
Signed-off-by: Mathieu Desnoyers <mathieu.desnoyers@efficios.com>
Mathieu Desnoyers [Fri, 30 Sep 2011 16:23:05 +0000 (12:23 -0400)]
rculfhash: Test for initiated destroy before performing resize
Signed-off-by: Mathieu Desnoyers <mathieu.desnoyers@efficios.com>
Mathieu Desnoyers [Thu, 29 Sep 2011 22:15:42 +0000 (18:15 -0400)]
Version 0.6.5
Signed-off-by: Mathieu Desnoyers <mathieu.desnoyers@efficios.com>
Mathieu Desnoyers [Thu, 29 Sep 2011 21:40:01 +0000 (17:40 -0400)]
call_rcu: Document call_rcu requirements
Signed-off-by: Mathieu Desnoyers <mathieu.desnoyers@efficios.com>
Mathieu Desnoyers [Thu, 29 Sep 2011 21:40:01 +0000 (17:40 -0400)]
call_rcu: Document call_rcu requirements
Signed-off-by: Mathieu Desnoyers <mathieu.desnoyers@efficios.com>
Mathieu Desnoyers [Thu, 29 Sep 2011 21:17:41 +0000 (17:17 -0400)]
Merge branch 'master' into urcu/ht-shrink
Mathieu Desnoyers [Thu, 29 Sep 2011 21:17:05 +0000 (17:17 -0400)]
call_rcu: fix error handling of malloc error
Signed-off-by: Mathieu Desnoyers <mathieu.desnoyers@efficios.com>
Mathieu Desnoyers [Thu, 29 Sep 2011 21:14:54 +0000 (17:14 -0400)]
Merge branch 'master' into urcu/ht-shrink
Mathieu Desnoyers [Thu, 29 Sep 2011 21:13:48 +0000 (17:13 -0400)]
urcu call_rcu: Use RCU read-side protection for per-cpu call_rcu data
A concurrent get_cpu_call_rcu_data(), called by get_call_rcu_data(),
could dereference this pointer without holding any mutex. So this
situation would happen if we have a concurrent call_rcu() executing
while we do the create_all_cpu_call_rcu_data().
I think we would need to put a rcu_dereference() around
per_cpu_call_rcu_data read within get_cpu_call_rcu_data() too.
per_cpu_call_rcu_data should be done with rcu_set_pointer.
Also, a rcu read-side critical section would be required around any
usage of per_cpu_call_rcu_data, and the action of tearing down the
per-cpu data would require to wait for a quiescent state. So we would
basically require that the call_rcu users need to be registered as
RCU reader threads.
Signed-off-by: Mathieu Desnoyers <mathieu.desnoyers@efficios.com>
Lai Jiangshan [Thu, 29 Sep 2011 19:56:43 +0000 (15:56 -0400)]
urcu,call_rcu: Cleanup call_rcu_data pointers before use in child
[ Edit by Mathieu Desnoyers: create maxcpus_reset to handle cases where
maxcpus is 0 and -1, depending on the configuration. ]
Signed-off-by: Lai Jiangshan <laijs@cn.fujitsu.com>
Signed-off-by: Mathieu Desnoyers <mathieu.desnoyers@efficios.com>
Lai Jiangshan [Thu, 29 Sep 2011 17:55:11 +0000 (13:55 -0400)]
urcu,call_rcu: avoid create call_rcu_data for child when unneed
Signed-off-by: Lai Jiangshan <laijs@cn.fujitsu.com>
Reviewed-by: Paul E. McKenney <paulmck@linux.vnet.ibm.com>
Signed-off-by: Mathieu Desnoyers <mathieu.desnoyers@efficios.com>
Lai Jiangshan [Thu, 29 Sep 2011 17:47:13 +0000 (13:47 -0400)]
urcu,defer_rcu: Make defer_rcu encoding more compact for marker
When the function changes (and the function is aligned), and only the
data is the marker, we can get away with using only 2 pointers rather
than 3.
[ Edit by Mathieu Desnoyers: patch cleanup, changelog updates ]
Signed-off-by: Lai Jiangshan <laijs@cn.fujitsu.com>
Signed-off-by: Mathieu Desnoyers <mathieu.desnoyers@efficios.com>
Mathieu Desnoyers [Thu, 29 Sep 2011 17:28:04 +0000 (13:28 -0400)]
urcu_defer: Use cancellation flag instead of pthread_cancel()
- Provides better control over cancellation point location.
- Set the futex to 0 before exiting the defer thread.
This patch combines and enhances patches from Lai Jiangshan:
urcu,defer_rcu: fix missing respond to a cancellation request
urcu,defer_rcu: Avoid thread exit unexpected
Reported-by: Lai Jiangshan <laijs@cn.fujitsu.com>
Signed-off-by: Mathieu Desnoyers <mathieu.desnoyers@efficios.com>
Lai Jiangshan [Thu, 29 Sep 2011 17:04:12 +0000 (13:04 -0400)]
urcu,call_rcu: protects call_rcu_data_list when remove node
Signed-off-by: Lai Jiangshan <laijs@cn.fujitsu.com>
Signed-off-by: Mathieu Desnoyers <mathieu.desnoyers@efficios.com>
Mathieu Desnoyers [Wed, 28 Sep 2011 20:03:40 +0000 (16:03 -0400)]
Merge branch 'master' into urcu/ht-shrink
Mathieu Desnoyers [Wed, 28 Sep 2011 20:03:13 +0000 (16:03 -0400)]
Create default call rcu data upon per-cpu call-rcu teardown
Signed-off-by: Mathieu Desnoyers <mathieu.desnoyers@efficios.com>
Mathieu Desnoyers [Wed, 28 Sep 2011 17:46:23 +0000 (13:46 -0400)]
rculfhash test: fix 32-bit hash
Signed-off-by: Mathieu Desnoyers <mathieu.desnoyers@efficios.com>
Mathieu Desnoyers [Wed, 28 Sep 2011 03:27:17 +0000 (23:27 -0400)]
rculfhash test: Use get first/get next to delete all entries
Signed-off-by: Mathieu Desnoyers <mathieu.desnoyers@efficios.com>
Mathieu Desnoyers [Wed, 28 Sep 2011 03:27:02 +0000 (23:27 -0400)]
rculfhash: fix get first / get next iterator
Signed-off-by: Mathieu Desnoyers <mathieu.desnoyers@efficios.com>
Mathieu Desnoyers [Wed, 28 Sep 2011 00:01:51 +0000 (20:01 -0400)]
Fix handling of systems without sysconf nr possible cpu support
Signed-off-by: Mathieu Desnoyers <mathieu.desnoyers@efficios.com>
Mathieu Desnoyers [Tue, 27 Sep 2011 23:54:08 +0000 (19:54 -0400)]
rculfhash needs local config.h
Signed-off-by: Mathieu Desnoyers <mathieu.desnoyers@efficios.com>
Mathieu Desnoyers [Tue, 27 Sep 2011 21:47:35 +0000 (17:47 -0400)]
rculfhash: update header documentation
Signed-off-by: Mathieu Desnoyers <mathieu.desnoyers@efficios.com>
Mathieu Desnoyers [Tue, 27 Sep 2011 21:39:30 +0000 (17:39 -0400)]
rculfhash test: move init node outside of rcu read-side c.s. (unneeded protection)
Signed-off-by: Mathieu Desnoyers <mathieu.desnoyers@efficios.com>
Mathieu Desnoyers [Tue, 27 Sep 2011 21:24:55 +0000 (17:24 -0400)]
Add cds_lfht_first/cds_lfht_next for hash table iteration
Signed-off-by: Mathieu Desnoyers <mathieu.desnoyers@efficios.com>
Mathieu Desnoyers [Tue, 27 Sep 2011 18:45:41 +0000 (14:45 -0400)]
rculfhash: rename _next into _next_duplicate
Signed-off-by: Mathieu Desnoyers <mathieu.desnoyers@efficios.com>
Mathieu Desnoyers [Tue, 27 Sep 2011 15:19:21 +0000 (11:19 -0400)]
rculfhash: document use of caa_container_of()
Signed-off-by: Mathieu Desnoyers <mathieu.desnoyers@efficios.com>
Mathieu Desnoyers [Mon, 26 Sep 2011 18:48:30 +0000 (14:48 -0400)]
rculfhash: cleanup includes
Signed-off-by: Mathieu Desnoyers <mathieu.desnoyers@efficios.com>
Mathieu Desnoyers [Thu, 22 Sep 2011 15:01:54 +0000 (11:01 -0400)]
Merge branch 'master' into urcu/ht-shrink
Mathieu Desnoyers [Thu, 22 Sep 2011 15:00:14 +0000 (11:00 -0400)]
powerpc: use __NO_LWSYNC__ check to use appropriate lwsync/sync opcode
We already used it in uatomic code, move it to arch ppc.
Signed-off-by: Mathieu Desnoyers <mathieu.desnoyers@efficios.com>
Mathieu Desnoyers [Thu, 22 Sep 2011 14:50:04 +0000 (10:50 -0400)]
Merge branch 'master' into urcu/ht-shrink
This page took 0.041743 seconds and 4 git commands to generate.