| Message ID | 20250203-slub-tiny-kfree_rcu-v1-0-d4428bf9a8a1@suse.cz (mailing list archive) |
|---|---|
| Headers | show |
| Series | slab, rcu: move and consolidate TINY_RCU kvfree_rcu() to SLAB | expand |
On Mon, Feb 03, 2025 at 10:28:46AM +0100, Vlastimil Babka wrote: > Following the move of the TREE_RCU batching kvfree_rcu() implementation > to slab, we still have the simple non-batching implementation in tiny > RCU, and RCU implementation specific ifdefs in slab code. > > Finish the move and integration into slab. Allow using the simple > call_rcu() based implementation also with tree RCU when SLUB_TINY is > enabled, as its goal is also to limit memory footprint with less concern > for top performance. > > In order to avoid RCU having to recognize the fake callback function > pointers (__is_kvfree_rcu_offset()) when handling call_rcu(), implement > a callback that can calculate the object's address from the embedded > rcu_head pointer without knowing the specific offset (previously SLOB > would not have made it possible, but it's gone now). > > After this series, AFAIK only the following kvfree_rcu specific code > remains in RCU: > > - a call to kfree_rcu_scheduler_running() from rcu_set_runtime_mode() > > - probably necessary and a generic registration interface would be > unnecessary bloat? I am OK with this calling directly. Alternatively, you could create a core_initcall() that did nothing but invoke kfree_rcu_scheduler_running(), and then drop that call from rcu_set_runtime_mode(). Up to you guys! > - declarations of kfree_rcu() API in include/linux/rcupdate.h > > - could be moved to slab.h after checking for/fixing up potential > missing includes It would be good to move them, but no real hurry from this end. > git tree: > https://git.kernel.org/pub/scm/linux/kernel/git/vbabka/linux.git/log/?h=b4/slub-tiny-kfree_rcu > > Changes since RFC [1]: > - Rebased to v6.14-rc1 > - Fixed build errors reported by bots. > - Changed where kvfree_call_rcu() is moved to in patch 1 to avoid another > move in patch 4. > - R-b's by Ulad and Joel > - Fix a memory leak in kvfree_rcu_list() thanks to Ulad > - Various comments' improvements and fixes (Joel, Ulad) > - Rename config to CONFIG_KFREE_RCU_BATCHED (Ulad) > > Due to changes, didn't keep Paul's Tested-by (but thanks!) I re-ran the tests, so feel free to add it back in. ;-) Thanx, Paul > Will include in slab/for-next if no objection. > > [1] https://lore.kernel.org/all/20250123-slub-tiny-kfree_rcu-v1-0-0e386ef1541a@suse.cz/ > > To: Christoph Lameter <cl@linux.com> > To: David Rientjes <rientjes@google.com> > Cc: Andrew Morton <akpm@linux-foundation.org> > Cc: Roman Gushchin <roman.gushchin@linux.dev> > Cc: Hyeonggon Yoo <42.hyeyoo@gmail.com> > Cc: linux-mm@kvack.org > To: "Paul E. McKenney" <paulmck@kernel.org> > To: Joel Fernandes <joel@joelfernandes.org> > To: Josh Triplett <josh@joshtriplett.org> > To: Boqun Feng <boqun.feng@gmail.com> > To: Uladzislau Rezki <urezki@gmail.com> > Cc: Steven Rostedt <rostedt@goodmis.org> > Cc: Mathieu Desnoyers <mathieu.desnoyers@efficios.com> > Cc: Lai Jiangshan <jiangshanlai@gmail.com> > Cc: Zqiang <qiang.zhang1211@gmail.com> > Cc: rcu@vger.kernel.org > > Signed-off-by: Vlastimil Babka <vbabka@suse.cz> > --- > Vlastimil Babka (4): > slab, rcu: move TINY_RCU variant of kvfree_rcu() to SLAB > rcu: remove trace_rcu_kvfree_callback > rcu, slab: use a regular callback function for kvfree_rcu > slab: don't batch kvfree_rcu() with SLUB_TINY > > include/linux/rcupdate.h | 33 ++++++++++++++++++--------------- > include/linux/rcutiny.h | 36 ------------------------------------ > include/linux/rcutree.h | 3 --- > include/linux/slab.h | 14 ++++++++++++++ > include/trace/events/rcu.h | 34 ---------------------------------- > kernel/rcu/tiny.c | 25 ------------------------- > kernel/rcu/tree.c | 9 ++------- > mm/Kconfig | 4 ++++ > mm/slab.h | 2 ++ > mm/slab_common.c | 33 +++++++++++++++++++++++++++------ > mm/slub.c | 46 ++++++++++++++++++++++++++++++++++++++++++++++ > 11 files changed, 113 insertions(+), 126 deletions(-) > --- > base-commit: 2014c95afecee3e76ca4a56956a936e23283f05b > change-id: 20250123-slub-tiny-kfree_rcu-bd65bfe222f2 > > Best regards, > -- > Vlastimil Babka <vbabka@suse.cz> >
On 2/3/25 20:07, Paul E. McKenney wrote: > On Mon, Feb 03, 2025 at 10:28:46AM +0100, Vlastimil Babka wrote: >> Following the move of the TREE_RCU batching kvfree_rcu() implementation >> to slab, we still have the simple non-batching implementation in tiny >> RCU, and RCU implementation specific ifdefs in slab code. >> >> Finish the move and integration into slab. Allow using the simple >> call_rcu() based implementation also with tree RCU when SLUB_TINY is >> enabled, as its goal is also to limit memory footprint with less concern >> for top performance. >> >> In order to avoid RCU having to recognize the fake callback function >> pointers (__is_kvfree_rcu_offset()) when handling call_rcu(), implement >> a callback that can calculate the object's address from the embedded >> rcu_head pointer without knowing the specific offset (previously SLOB >> would not have made it possible, but it's gone now). >> >> After this series, AFAIK only the following kvfree_rcu specific code >> remains in RCU: >> >> - a call to kfree_rcu_scheduler_running() from rcu_set_runtime_mode() >> >> - probably necessary and a generic registration interface would be >> unnecessary bloat? > > I am OK with this calling directly. > > Alternatively, you could create a core_initcall() that did nothing > but invoke kfree_rcu_scheduler_running(), and then drop that call from > rcu_set_runtime_mode(). Up to you guys! I was afraid there's some ordering necessary with the rest of what rcu_set_runtime_mode() does, but perhaps there's actually not. >> - declarations of kfree_rcu() API in include/linux/rcupdate.h >> >> - could be moved to slab.h after checking for/fixing up potential >> missing includes > > It would be good to move them, but no real hurry from this end. > >> git tree: >> https://git.kernel.org/pub/scm/linux/kernel/git/vbabka/linux.git/log/?h=b4/slub-tiny-kfree_rcu >> >> Changes since RFC [1]: >> - Rebased to v6.14-rc1 >> - Fixed build errors reported by bots. >> - Changed where kvfree_call_rcu() is moved to in patch 1 to avoid another >> move in patch 4. >> - R-b's by Ulad and Joel >> - Fix a memory leak in kvfree_rcu_list() thanks to Ulad >> - Various comments' improvements and fixes (Joel, Ulad) >> - Rename config to CONFIG_KFREE_RCU_BATCHED (Ulad) >> >> Due to changes, didn't keep Paul's Tested-by (but thanks!) > > I re-ran the tests, so feel free to add it back in. ;-) Thanks, will do :)
On Tue, Feb 04, 2025 at 11:02:47AM +0100, Vlastimil Babka wrote: > On 2/3/25 20:07, Paul E. McKenney wrote: > > On Mon, Feb 03, 2025 at 10:28:46AM +0100, Vlastimil Babka wrote: > >> Following the move of the TREE_RCU batching kvfree_rcu() implementation > >> to slab, we still have the simple non-batching implementation in tiny > >> RCU, and RCU implementation specific ifdefs in slab code. > >> > >> Finish the move and integration into slab. Allow using the simple > >> call_rcu() based implementation also with tree RCU when SLUB_TINY is > >> enabled, as its goal is also to limit memory footprint with less concern > >> for top performance. > >> > >> In order to avoid RCU having to recognize the fake callback function > >> pointers (__is_kvfree_rcu_offset()) when handling call_rcu(), implement > >> a callback that can calculate the object's address from the embedded > >> rcu_head pointer without knowing the specific offset (previously SLOB > >> would not have made it possible, but it's gone now). > >> > >> After this series, AFAIK only the following kvfree_rcu specific code > >> remains in RCU: > >> > >> - a call to kfree_rcu_scheduler_running() from rcu_set_runtime_mode() > >> > >> - probably necessary and a generic registration interface would be > >> unnecessary bloat? > > > > I am OK with this calling directly. > > > > Alternatively, you could create a core_initcall() that did nothing > > but invoke kfree_rcu_scheduler_running(), and then drop that call from > > rcu_set_runtime_mode(). Up to you guys! > > I was afraid there's some ordering necessary with the rest of what > rcu_set_runtime_mode() does, but perhaps there's actually not. I *think* that they are independent, but I agree that it would be good to carefully double-check. The basis for my belief is that it is OK to invoke call_rcu() long before rcu_init() is invoked, let alone the scheduler being ready. Thanx, Paul > >> - declarations of kfree_rcu() API in include/linux/rcupdate.h > >> > >> - could be moved to slab.h after checking for/fixing up potential > >> missing includes > > > > It would be good to move them, but no real hurry from this end. > > > >> git tree: > >> https://git.kernel.org/pub/scm/linux/kernel/git/vbabka/linux.git/log/?h=b4/slub-tiny-kfree_rcu > >> > >> Changes since RFC [1]: > >> - Rebased to v6.14-rc1 > >> - Fixed build errors reported by bots. > >> - Changed where kvfree_call_rcu() is moved to in patch 1 to avoid another > >> move in patch 4. > >> - R-b's by Ulad and Joel > >> - Fix a memory leak in kvfree_rcu_list() thanks to Ulad > >> - Various comments' improvements and fixes (Joel, Ulad) > >> - Rename config to CONFIG_KFREE_RCU_BATCHED (Ulad) > >> > >> Due to changes, didn't keep Paul's Tested-by (but thanks!) > > > > I re-ran the tests, so feel free to add it back in. ;-) > > Thanks, will do :) > >
On 2/3/25 10:28, Vlastimil Babka wrote: > Following the move of the TREE_RCU batching kvfree_rcu() implementation > to slab, we still have the simple non-batching implementation in tiny > RCU, and RCU implementation specific ifdefs in slab code. > > Finish the move and integration into slab. Allow using the simple > call_rcu() based implementation also with tree RCU when SLUB_TINY is > enabled, as its goal is also to limit memory footprint with less concern > for top performance. > > In order to avoid RCU having to recognize the fake callback function > pointers (__is_kvfree_rcu_offset()) when handling call_rcu(), implement > a callback that can calculate the object's address from the embedded > rcu_head pointer without knowing the specific offset (previously SLOB > would not have made it possible, but it's gone now). > > After this series, AFAIK only the following kvfree_rcu specific code > remains in RCU: > > - a call to kfree_rcu_scheduler_running() from rcu_set_runtime_mode() > > - probably necessary and a generic registration interface would be > unnecessary bloat? > > - declarations of kfree_rcu() API in include/linux/rcupdate.h > > - could be moved to slab.h after checking for/fixing up potential > missing includes > > git tree: > https://git.kernel.org/pub/scm/linux/kernel/git/vbabka/linux.git/log/?h=b4/slub-tiny-kfree_rcu > > Changes since RFC [1]: > - Rebased to v6.14-rc1 > - Fixed build errors reported by bots. > - Changed where kvfree_call_rcu() is moved to in patch 1 to avoid another > move in patch 4. > - R-b's by Ulad and Joel > - Fix a memory leak in kvfree_rcu_list() thanks to Ulad > - Various comments' improvements and fixes (Joel, Ulad) > - Rename config to CONFIG_KFREE_RCU_BATCHED (Ulad) > > Due to changes, didn't keep Paul's Tested-by (but thanks!) > > Will include in slab/for-next if no objection. Now done, thanks to Paul for testing and Harry for reviews!
Following the move of the TREE_RCU batching kvfree_rcu() implementation to slab, we still have the simple non-batching implementation in tiny RCU, and RCU implementation specific ifdefs in slab code. Finish the move and integration into slab. Allow using the simple call_rcu() based implementation also with tree RCU when SLUB_TINY is enabled, as its goal is also to limit memory footprint with less concern for top performance. In order to avoid RCU having to recognize the fake callback function pointers (__is_kvfree_rcu_offset()) when handling call_rcu(), implement a callback that can calculate the object's address from the embedded rcu_head pointer without knowing the specific offset (previously SLOB would not have made it possible, but it's gone now). After this series, AFAIK only the following kvfree_rcu specific code remains in RCU: - a call to kfree_rcu_scheduler_running() from rcu_set_runtime_mode() - probably necessary and a generic registration interface would be unnecessary bloat? - declarations of kfree_rcu() API in include/linux/rcupdate.h - could be moved to slab.h after checking for/fixing up potential missing includes git tree: https://git.kernel.org/pub/scm/linux/kernel/git/vbabka/linux.git/log/?h=b4/slub-tiny-kfree_rcu Changes since RFC [1]: - Rebased to v6.14-rc1 - Fixed build errors reported by bots. - Changed where kvfree_call_rcu() is moved to in patch 1 to avoid another move in patch 4. - R-b's by Ulad and Joel - Fix a memory leak in kvfree_rcu_list() thanks to Ulad - Various comments' improvements and fixes (Joel, Ulad) - Rename config to CONFIG_KFREE_RCU_BATCHED (Ulad) Due to changes, didn't keep Paul's Tested-by (but thanks!) Will include in slab/for-next if no objection. [1] https://lore.kernel.org/all/20250123-slub-tiny-kfree_rcu-v1-0-0e386ef1541a@suse.cz/ To: Christoph Lameter <cl@linux.com> To: David Rientjes <rientjes@google.com> Cc: Andrew Morton <akpm@linux-foundation.org> Cc: Roman Gushchin <roman.gushchin@linux.dev> Cc: Hyeonggon Yoo <42.hyeyoo@gmail.com> Cc: linux-mm@kvack.org To: "Paul E. McKenney" <paulmck@kernel.org> To: Joel Fernandes <joel@joelfernandes.org> To: Josh Triplett <josh@joshtriplett.org> To: Boqun Feng <boqun.feng@gmail.com> To: Uladzislau Rezki <urezki@gmail.com> Cc: Steven Rostedt <rostedt@goodmis.org> Cc: Mathieu Desnoyers <mathieu.desnoyers@efficios.com> Cc: Lai Jiangshan <jiangshanlai@gmail.com> Cc: Zqiang <qiang.zhang1211@gmail.com> Cc: rcu@vger.kernel.org Signed-off-by: Vlastimil Babka <vbabka@suse.cz> --- Vlastimil Babka (4): slab, rcu: move TINY_RCU variant of kvfree_rcu() to SLAB rcu: remove trace_rcu_kvfree_callback rcu, slab: use a regular callback function for kvfree_rcu slab: don't batch kvfree_rcu() with SLUB_TINY include/linux/rcupdate.h | 33 ++++++++++++++++++--------------- include/linux/rcutiny.h | 36 ------------------------------------ include/linux/rcutree.h | 3 --- include/linux/slab.h | 14 ++++++++++++++ include/trace/events/rcu.h | 34 ---------------------------------- kernel/rcu/tiny.c | 25 ------------------------- kernel/rcu/tree.c | 9 ++------- mm/Kconfig | 4 ++++ mm/slab.h | 2 ++ mm/slab_common.c | 33 +++++++++++++++++++++++++++------ mm/slub.c | 46 ++++++++++++++++++++++++++++++++++++++++++++++ 11 files changed, 113 insertions(+), 126 deletions(-) --- base-commit: 2014c95afecee3e76ca4a56956a936e23283f05b change-id: 20250123-slub-tiny-kfree_rcu-bd65bfe222f2 Best regards,