[11/62] mm/slub: Convert kfree() to use a struct slab

Message ID	20211004134650.4031813-12-willy@infradead.org (mailing list archive)
State	New
Headers	show Return-Path: <SRS0=Fowb=OY=kvack.org=owner-linux-mm@kernel.org> DMARC-Filter: OpenDMARC Filter v1.4.1 mail.kernel.org ABAC961131 From: "Matthew Wilcox (Oracle)" <willy@infradead.org> To: linux-mm@kvack.org Cc: "Matthew Wilcox (Oracle)" <willy@infradead.org> Subject: [PATCH 11/62] mm/slub: Convert kfree() to use a struct slab Date: Mon, 4 Oct 2021 14:45:59 +0100 Message-Id: <20211004134650.4031813-12-willy@infradead.org> In-Reply-To: <20211004134650.4031813-1-willy@infradead.org> References: <20211004134650.4031813-1-willy@infradead.org> MIME-Version: 1.0 Content-Transfer-Encoding: quoted-printable Sender: owner-linux-mm@kvack.org Precedence: bulk
Series	Separate struct slab from struct page \| expand [00/62] Separate struct slab from struct page [01/62] mm: Convert page_to_section() to pgflags_section() [02/62] mm: Add pgflags_nid() [03/62] mm: Split slab into its own type [04/62] mm: Add account_slab() and unaccount_slab() [05/62] mm: Convert virt_to_cache() to use struct slab [06/62] mm: Convert __ksize() to struct slab [07/62] mm: Use struct slab in kmem_obj_info() [08/62] mm: Convert check_heap_object() to use struct slab [09/62] mm/slub: Convert process_slab() to take a struct slab [10/62] mm/slub: Convert detached_freelist to use a struct slab [11/62] mm/slub: Convert kfree() to use a struct slab [12/62] mm/slub: Convert __slab_free() to take a struct slab [13/62] mm/slub: Convert new_slab() to return a struct slab [14/62] mm/slub: Convert early_kmem_cache_node_alloc() to use struct slab [15/62] mm/slub: Convert kmem_cache_cpu to struct slab [16/62] mm/slub: Convert show_slab_objects() to struct slab [17/62] mm/slub: Convert validate_slab() to take a struct slab [18/62] mm/slub: Convert count_partial() to struct slab [19/62] mm/slub: Convert bootstrap() to struct slab [20/62] mm/slub: Convert __kmem_cache_do_shrink() to struct slab [21/62] mm/slub: Convert free_partial() to use struct slab [22/62] mm/slub: Convert list_slab_objects() to take a struct slab [23/62] mm/slub: Convert slab_alloc_node() to use a struct slab [24/62] mm/slub: Convert get_freelist() to take a struct slab [25/62] mm/slub: Convert node_match() to take a struct slab [26/62] mm/slub: Convert slab flushing to struct slab [27/62] mm/slub: Convert __unfreeze_partials to take a struct slab [28/62] mm/slub: Convert deactivate_slab() to take a struct slab [29/62] mm/slub: Convert acquire_slab() to take a struct page [30/62] mm/slub: Convert partial slab management to struct slab [31/62] mm/slub: Convert slab freeing to struct slab [32/62] mm/slub: Convert shuffle_freelist to struct slab [33/62] mm/slub: Remove struct page argument to next_freelist_entry() [34/62] mm/slub: Remove struct page argument from setup_object() [35/62] mm/slub: Convert freelist_corrupted() to struct slab [36/62] mm/slub: Convert full slab management to struct slab [37/62] mm/slub: Convert free_consistency_checks() to take a struct slab [38/62] mm/slub: Convert alloc_debug_processing() to struct slab [39/62] mm/slub: Convert check_object() to struct slab [40/62] mm/slub: Convert on_freelist() to struct slab [41/62] mm/slub: Convert check_slab() to struct slab [42/62] mm/slub: Convert check_valid_pointer() to struct slab [43/62] mm/slub: Convert object_err() to take a struct slab [44/62] mm/slub: Convert print_trailer() to struct slab [45/62] mm/slub: Convert slab_err() to take a struct slab [46/62] mm/slub: Convert print_page_info() to print_slab_info() [47/62] mm/slub: Convert trace() to take a struct slab [48/62] mm/slub: Convert cmpxchg_double_slab to struct slab [49/62] mm/slub: Convert get_map() and __fill_map() to struct slab [50/62] mm/slub: Convert slab_lock() and slab_unlock() to struct slab [51/62] mm/slub: Convert setup_page_debug() to setup_slab_debug() [52/62] mm/slub: Convert pfmemalloc_match() to take a struct slab [53/62] mm/slub: Remove pfmemalloc_match_unsafe() [54/62] mm: Convert slab to use struct slab [55/62] mm: Convert slob to use struct slab [56/62] mm: Convert slub to use struct slab [57/62] memcg: Convert object cgroups from struct page to struct slab [58/62] mm/kasan: Convert to struct slab [59/62] zsmalloc: Stop using slab fields in struct page [60/62] bootmem: Use page->index instead of page->freelist [61/62] iommu: Use put_pages_list [62/62] mm: Remove slab from struct page

Message ID

20211004134650.4031813-12-willy@infradead.org (mailing list archive)

State

New

Headers

DMARC-Filter: OpenDMARC Filter v1.4.1 mail.kernel.org ABAC961131
From: "Matthew Wilcox (Oracle)" <willy@infradead.org>
To: linux-mm@kvack.org
Cc: "Matthew Wilcox (Oracle)" <willy@infradead.org>
Subject: [PATCH 11/62] mm/slub: Convert kfree() to use a struct slab
Date: Mon,  4 Oct 2021 14:45:59 +0100
Message-Id: <20211004134650.4031813-12-willy@infradead.org>
In-Reply-To: <20211004134650.4031813-1-willy@infradead.org>
References: <20211004134650.4031813-1-willy@infradead.org>
MIME-Version: 1.0
Content-Transfer-Encoding: quoted-printable
Sender: owner-linux-mm@kvack.org
Precedence: bulk

Series

Separate struct slab from struct page | expand

Commit Message

Matthew Wilcox Oct. 4, 2021, 1:45 p.m. UTC

With kfree() using a struct slab, we can also convert slab_free()
and do_slab_free() to use a slab instead of a page.

Signed-off-by: Matthew Wilcox (Oracle) <willy@infradead.org>
---
 mm/slub.c | 30 +++++++++++++++---------------
 1 file changed, 15 insertions(+), 15 deletions(-)

diff --git a/mm/slub.c b/mm/slub.c
index 050a0610b3ef..15996ea165ac 100644
--- a/mm/slub.c
+++ b/mm/slub.c
@@ -3402,11 +3402,11 @@  static void __slab_free(struct kmem_cache *s, struct page *page,
  * with all sorts of special processing.
  *
  * Bulk free of a freelist with several objects (all pointing to the
- * same page) possible by specifying head and tail ptr, plus objects
+ * same slab) possible by specifying head and tail ptr, plus objects
  * count (cnt). Bulk free indicated by tail pointer being set.
  */
 static __always_inline void do_slab_free(struct kmem_cache *s,
-				struct page *page, void *head, void *tail,
+				struct slab *slab, void *head, void *tail,
 				int cnt, unsigned long addr)
 {
 	void *tail_obj = tail ? : head;
@@ -3427,7 +3427,7 @@  static __always_inline void do_slab_free(struct kmem_cache *s,
 	/* Same with comment on barrier() in slab_alloc_node() */
 	barrier();
 
-	if (likely(page == c->page)) {
+	if (likely(slab_page(slab) == c->page)) {
 #ifndef CONFIG_PREEMPT_RT
 		void **freelist = READ_ONCE(c->freelist);
 
@@ -3453,7 +3453,7 @@  static __always_inline void do_slab_free(struct kmem_cache *s,
 
 		local_lock(&s->cpu_slab->lock);
 		c = this_cpu_ptr(s->cpu_slab);
-		if (unlikely(page != c->page)) {
+		if (unlikely(slab_page(slab) != c->page)) {
 			local_unlock(&s->cpu_slab->lock);
 			goto redo;
 		}
@@ -3468,11 +3468,11 @@  static __always_inline void do_slab_free(struct kmem_cache *s,
 #endif
 		stat(s, FREE_FASTPATH);
 	} else
-		__slab_free(s, page, head, tail_obj, cnt, addr);
+		__slab_free(s, slab_page(slab), head, tail_obj, cnt, addr);
 
 }
 
-static __always_inline void slab_free(struct kmem_cache *s, struct page *page,
+static __always_inline void slab_free(struct kmem_cache *s, struct slab *slab,
 				      void *head, void *tail, int cnt,
 				      unsigned long addr)
 {
@@ -3481,13 +3481,13 @@  static __always_inline void slab_free(struct kmem_cache *s, struct page *page,
 	 * to remove objects, whose reuse must be delayed.
 	 */
 	if (slab_free_freelist_hook(s, &head, &tail))
-		do_slab_free(s, page, head, tail, cnt, addr);
+		do_slab_free(s, slab, head, tail, cnt, addr);
 }
 
 #ifdef CONFIG_KASAN_GENERIC
 void ___cache_free(struct kmem_cache *cache, void *x, unsigned long addr)
 {
-	do_slab_free(cache, virt_to_head_page(x), x, NULL, 1, addr);
+	do_slab_free(cache, virt_to_slab(x), x, NULL, 1, addr);
 }
 #endif
 
@@ -3496,7 +3496,7 @@  void kmem_cache_free(struct kmem_cache *s, void *x)
 	s = cache_from_obj(s, x);
 	if (!s)
 		return;
-	slab_free(s, virt_to_head_page(x), x, NULL, 1, _RET_IP_);
+	slab_free(s, virt_to_slab(x), x, NULL, 1, _RET_IP_);
 	trace_kmem_cache_free(_RET_IP_, x, s->name);
 }
 EXPORT_SYMBOL(kmem_cache_free);
@@ -3621,7 +3621,7 @@  void kmem_cache_free_bulk(struct kmem_cache *s, size_t size, void **p)
 		if (!df.slab)
 			continue;
 
-		slab_free(df.s, slab_page(df.slab), df.freelist, df.tail, df.cnt, _RET_IP_);
+		slab_free(df.s, df.slab, df.freelist, df.tail, df.cnt, _RET_IP_);
 	} while (likely(size));
 }
 EXPORT_SYMBOL(kmem_cache_free_bulk);
@@ -4527,7 +4527,7 @@  EXPORT_SYMBOL(__ksize);
 
 void kfree(const void *x)
 {
-	struct page *page;
+	struct slab *slab;
 	void *object = (void *)x;
 
 	trace_kfree(_RET_IP_, x);
@@ -4535,12 +4535,12 @@  void kfree(const void *x)
 	if (unlikely(ZERO_OR_NULL_PTR(x)))
 		return;
 
-	page = virt_to_head_page(x);
-	if (unlikely(!PageSlab(page))) {
-		free_nonslab_page(page, object);
+	slab = virt_to_slab(x);
+	if (unlikely(!SlabAllocation(slab))) {
+		free_nonslab_page(slab_page(slab), object);
 		return;
 	}
-	slab_free(page->slab_cache, page, object, NULL, 1, _RET_IP_);
+	slab_free(slab->slab_cache, slab, object, NULL, 1, _RET_IP_);
 }
 EXPORT_SYMBOL(kfree);

[11/62] mm/slub: Convert kfree() to use a struct slab

Commit Message

Patch