From patchwork Mon Oct 4 13:46:46 2021 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Matthew Wilcox X-Patchwork-Id: 12534257 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from mail.kernel.org (mail.kernel.org [198.145.29.99]) by smtp.lore.kernel.org (Postfix) with ESMTP id 744D4C433EF for ; Mon, 4 Oct 2021 14:59:29 +0000 (UTC) Received: from kanga.kvack.org (kanga.kvack.org [205.233.56.17]) by mail.kernel.org (Postfix) with ESMTP id 099A9611CA for ; Mon, 4 Oct 2021 14:59:29 +0000 (UTC) DMARC-Filter: OpenDMARC Filter v1.4.1 mail.kernel.org 099A9611CA Authentication-Results: mail.kernel.org; dmarc=none (p=none dis=none) header.from=infradead.org Authentication-Results: mail.kernel.org; spf=pass smtp.mailfrom=kvack.org Received: by kanga.kvack.org (Postfix) id 67C74940058; Mon, 4 Oct 2021 10:59:28 -0400 (EDT) Received: by kanga.kvack.org (Postfix, from userid 40) id 6047394000B; Mon, 4 Oct 2021 10:59:28 -0400 (EDT) X-Delivered-To: int-list-linux-mm@kvack.org Received: by kanga.kvack.org (Postfix, from userid 63042) id 4A5DF940058; Mon, 4 Oct 2021 10:59:28 -0400 (EDT) X-Delivered-To: linux-mm@kvack.org Received: from forelay.hostedemail.com (smtprelay0225.hostedemail.com [216.40.44.225]) by kanga.kvack.org (Postfix) with ESMTP id 3743994000B for ; Mon, 4 Oct 2021 10:59:28 -0400 (EDT) Received: from smtpin24.hostedemail.com (10.5.19.251.rfc1918.com [10.5.19.251]) by forelay04.hostedemail.com (Postfix) with ESMTP id E894631E7E for ; Mon, 4 Oct 2021 14:59:27 +0000 (UTC) X-FDA: 78659063574.24.3722434 Received: from casper.infradead.org (casper.infradead.org [90.155.50.34]) by imf21.hostedemail.com (Postfix) with ESMTP id 8DD9DD03885E for ; Mon, 4 Oct 2021 14:59:27 +0000 (UTC) DKIM-Signature: v=1; a=rsa-sha256; q=dns/txt; c=relaxed/relaxed; d=infradead.org; s=casper.20170209; h=Content-Transfer-Encoding:MIME-Version: References:In-Reply-To:Message-Id:Date:Subject:Cc:To:From:Sender:Reply-To: Content-Type:Content-ID:Content-Description; bh=s1TXas+N6RfOF2JRawS5DBEjkpGLTm/HmH2iNZr6p0g=; b=ud6Mhpm1E/+cjRwPX9ALraobgm x/KV4EdElHZ8WhXsQgK6pd2PuRhqkBv/j3EGi3tbStKqE8Jyf5piWEzmMHAOHIYt25EeurH9LZoCX ov9SlQFe/eLc3N6T/6IUDCWMNXLgjPwburHuprTOkl0vfwsyghibxpPhkxZQqESUaDRyD5P3laGxP wEWgQ6P6ngCezUpHRXOUYXidQ4dRoqZmn3UKRYtY2Es4islRoBE1RLmN8O31sSa/jArznqsIAxyUZ Ce2IbyTxupvSM3p9Sao/jneN5REB3AK2kb/B8UO9KMeyli9kXS4Ra4GmnKyZ6lrrXeRq/u0jZ7aGV lJnVhgEQ==; Received: from willy by casper.infradead.org with local (Exim 4.94.2 #2 (Red Hat Linux)) id 1mXPP3-00H259-4L; Mon, 04 Oct 2021 14:57:53 +0000 From: "Matthew Wilcox (Oracle)" To: linux-mm@kvack.org Cc: "Matthew Wilcox (Oracle)" Subject: [PATCH 58/62] mm/kasan: Convert to struct slab Date: Mon, 4 Oct 2021 14:46:46 +0100 Message-Id: <20211004134650.4031813-59-willy@infradead.org> X-Mailer: git-send-email 2.31.1 In-Reply-To: <20211004134650.4031813-1-willy@infradead.org> References: <20211004134650.4031813-1-willy@infradead.org> MIME-Version: 1.0 X-Rspamd-Server: rspam05 X-Rspamd-Queue-Id: 8DD9DD03885E X-Stat-Signature: j66ah93jkxyy51jzibi7tjw7h38rjdtg Authentication-Results: imf21.hostedemail.com; dkim=pass header.d=infradead.org header.s=casper.20170209 header.b=ud6Mhpm1; dmarc=none; spf=none (imf21.hostedemail.com: domain of willy@infradead.org has no SPF policy when checking 90.155.50.34) smtp.mailfrom=willy@infradead.org X-HE-Tag: 1633359567-117993 X-Bogosity: Ham, tests=bogofilter, spamicity=0.000000, version=1.2.4 Sender: owner-linux-mm@kvack.org Precedence: bulk X-Loop: owner-majordomo@kvack.org List-ID: This should all be split up and done better. Signed-off-by: Matthew Wilcox (Oracle) --- include/linux/kasan.h | 8 ++++---- include/linux/slab_def.h | 6 +++--- include/linux/slub_def.h | 8 ++++---- mm/kasan/common.c | 23 ++++++++++++----------- mm/kasan/generic.c | 8 ++++---- mm/kasan/kasan.h | 2 +- mm/kasan/quarantine.c | 2 +- mm/kasan/report.c | 16 ++++++++-------- mm/kasan/report_tags.c | 10 +++++----- mm/slab.c | 2 +- mm/slub.c | 2 +- 11 files changed, 44 insertions(+), 43 deletions(-) diff --git a/include/linux/kasan.h b/include/linux/kasan.h index dd874a1ee862..59c860295618 100644 --- a/include/linux/kasan.h +++ b/include/linux/kasan.h @@ -188,11 +188,11 @@ static __always_inline size_t kasan_metadata_size(struct kmem_cache *cache) return 0; } -void __kasan_poison_slab(struct page *page); -static __always_inline void kasan_poison_slab(struct page *page) +void __kasan_poison_slab(struct slab *slab); +static __always_inline void kasan_poison_slab(struct slab *slab) { if (kasan_enabled()) - __kasan_poison_slab(page); + __kasan_poison_slab(slab); } void __kasan_unpoison_object_data(struct kmem_cache *cache, void *object); @@ -317,7 +317,7 @@ static inline void kasan_cache_create(struct kmem_cache *cache, slab_flags_t *flags) {} static inline void kasan_cache_create_kmalloc(struct kmem_cache *cache) {} static inline size_t kasan_metadata_size(struct kmem_cache *cache) { return 0; } -static inline void kasan_poison_slab(struct page *page) {} +static inline void kasan_poison_slab(struct slab *slab) {} static inline void kasan_unpoison_object_data(struct kmem_cache *cache, void *object) {} static inline void kasan_poison_object_data(struct kmem_cache *cache, diff --git a/include/linux/slab_def.h b/include/linux/slab_def.h index f81a41f9d5d1..f1bfcb10f5e0 100644 --- a/include/linux/slab_def.h +++ b/include/linux/slab_def.h @@ -87,11 +87,11 @@ struct kmem_cache { struct kmem_cache_node *node[MAX_NUMNODES]; }; -static inline void *nearest_obj(struct kmem_cache *cache, struct page *page, +static inline void *nearest_obj(struct kmem_cache *cache, struct slab *slab, void *x) { - void *object = x - (x - page->s_mem) % cache->size; - void *last_object = page->s_mem + (cache->num - 1) * cache->size; + void *object = x - (x - slab->s_mem) % cache->size; + void *last_object = slab->s_mem + (cache->num - 1) * cache->size; if (unlikely(object > last_object)) return last_object; diff --git a/include/linux/slub_def.h b/include/linux/slub_def.h index 994a60da2f2e..4db01470a9e3 100644 --- a/include/linux/slub_def.h +++ b/include/linux/slub_def.h @@ -167,11 +167,11 @@ static inline void sysfs_slab_release(struct kmem_cache *s) void *fixup_red_left(struct kmem_cache *s, void *p); -static inline void *nearest_obj(struct kmem_cache *cache, struct page *page, +static inline void *nearest_obj(struct kmem_cache *cache, struct slab *slab, void *x) { - void *object = x - (x - page_address(page)) % cache->size; - void *last_object = page_address(page) + - (page->objects - 1) * cache->size; + void *object = x - (x - slab_address(slab)) % cache->size; + void *last_object = slab_address(slab) + + (slab->objects - 1) * cache->size; void *result = (unlikely(object > last_object)) ? last_object : object; result = fixup_red_left(cache, result); diff --git a/mm/kasan/common.c b/mm/kasan/common.c index f3972af7fa1b..85774174a437 100644 --- a/mm/kasan/common.c +++ b/mm/kasan/common.c @@ -247,8 +247,9 @@ struct kasan_free_meta *kasan_get_free_meta(struct kmem_cache *cache, } #endif -void __kasan_poison_slab(struct page *page) +void __kasan_poison_slab(struct slab *slab) { + struct page *page = slab_page(slab); unsigned long i; for (i = 0; i < compound_nr(page); i++) @@ -341,7 +342,7 @@ static inline bool ____kasan_slab_free(struct kmem_cache *cache, void *object, if (is_kfence_address(object)) return false; - if (unlikely(nearest_obj(cache, virt_to_head_page(object), object) != + if (unlikely(nearest_obj(cache, virt_to_slab(object), object) != object)) { kasan_report_invalid_free(tagged_object, ip); return true; @@ -401,9 +402,9 @@ void __kasan_kfree_large(void *ptr, unsigned long ip) void __kasan_slab_free_mempool(void *ptr, unsigned long ip) { - struct page *page; + struct slab *slab; - page = virt_to_head_page(ptr); + slab = virt_to_slab(ptr); /* * Even though this function is only called for kmem_cache_alloc and @@ -411,12 +412,12 @@ void __kasan_slab_free_mempool(void *ptr, unsigned long ip) * !PageSlab() when the size provided to kmalloc is larger than * KMALLOC_MAX_SIZE, and kmalloc falls back onto page_alloc. */ - if (unlikely(!PageSlab(page))) { + if (unlikely(!slab_test_cache(slab))) { if (____kasan_kfree_large(ptr, ip)) return; - kasan_poison(ptr, page_size(page), KASAN_FREE_PAGE, false); + kasan_poison(ptr, slab_size(slab), KASAN_FREE_PAGE, false); } else { - ____kasan_slab_free(page->slab_cache, ptr, ip, false, false); + ____kasan_slab_free(slab->slab_cache, ptr, ip, false, false); } } @@ -560,7 +561,7 @@ void * __must_check __kasan_kmalloc_large(const void *ptr, size_t size, void * __must_check __kasan_krealloc(const void *object, size_t size, gfp_t flags) { - struct page *page; + struct slab *slab; if (unlikely(object == ZERO_SIZE_PTR)) return (void *)object; @@ -572,13 +573,13 @@ void * __must_check __kasan_krealloc(const void *object, size_t size, gfp_t flag */ kasan_unpoison(object, size, false); - page = virt_to_head_page(object); + slab = virt_to_slab(object); /* Piggy-back on kmalloc() instrumentation to poison the redzone. */ - if (unlikely(!PageSlab(page))) + if (unlikely(!slab_test_cache(slab))) return __kasan_kmalloc_large(object, size, flags); else - return ____kasan_kmalloc(page->slab_cache, object, size, flags); + return ____kasan_kmalloc(slab->slab_cache, object, size, flags); } bool __kasan_check_byte(const void *address, unsigned long ip) diff --git a/mm/kasan/generic.c b/mm/kasan/generic.c index c3f5ba7a294a..6153f85b90cb 100644 --- a/mm/kasan/generic.c +++ b/mm/kasan/generic.c @@ -330,16 +330,16 @@ DEFINE_ASAN_SET_SHADOW(f8); void kasan_record_aux_stack(void *addr) { - struct page *page = kasan_addr_to_page(addr); + struct slab *slab = kasan_addr_to_slab(addr); struct kmem_cache *cache; struct kasan_alloc_meta *alloc_meta; void *object; - if (is_kfence_address(addr) || !(page && PageSlab(page))) + if (is_kfence_address(addr) || !(slab && slab_test_cache(slab))) return; - cache = page->slab_cache; - object = nearest_obj(cache, page, addr); + cache = slab->slab_cache; + object = nearest_obj(cache, slab, addr); alloc_meta = kasan_get_alloc_meta(cache, object); if (!alloc_meta) return; diff --git a/mm/kasan/kasan.h b/mm/kasan/kasan.h index 8bf568a80eb8..8f9aca95db72 100644 --- a/mm/kasan/kasan.h +++ b/mm/kasan/kasan.h @@ -249,7 +249,7 @@ bool kasan_report(unsigned long addr, size_t size, bool is_write, unsigned long ip); void kasan_report_invalid_free(void *object, unsigned long ip); -struct page *kasan_addr_to_page(const void *addr); +struct slab *kasan_addr_to_slab(const void *addr); depot_stack_handle_t kasan_save_stack(gfp_t flags); void kasan_set_track(struct kasan_track *track, gfp_t flags); diff --git a/mm/kasan/quarantine.c b/mm/kasan/quarantine.c index d8ccff4c1275..587da8995f2d 100644 --- a/mm/kasan/quarantine.c +++ b/mm/kasan/quarantine.c @@ -117,7 +117,7 @@ static unsigned long quarantine_batch_size; static struct kmem_cache *qlink_to_cache(struct qlist_node *qlink) { - return virt_to_head_page(qlink)->slab_cache; + return virt_to_slab(qlink)->slab_cache; } static void *qlink_to_object(struct qlist_node *qlink, struct kmem_cache *cache) diff --git a/mm/kasan/report.c b/mm/kasan/report.c index 884a950c7026..49b58221755a 100644 --- a/mm/kasan/report.c +++ b/mm/kasan/report.c @@ -151,11 +151,11 @@ static void print_track(struct kasan_track *track, const char *prefix) } } -struct page *kasan_addr_to_page(const void *addr) +struct slab *kasan_addr_to_slab(const void *addr) { if ((addr >= (void *)PAGE_OFFSET) && (addr < high_memory)) - return virt_to_head_page(addr); + return virt_to_slab(addr); return NULL; } @@ -251,14 +251,14 @@ static inline bool init_task_stack_addr(const void *addr) static void print_address_description(void *addr, u8 tag) { - struct page *page = kasan_addr_to_page(addr); + struct slab *slab = kasan_addr_to_slab(addr); dump_stack_lvl(KERN_ERR); pr_err("\n"); - if (page && PageSlab(page)) { - struct kmem_cache *cache = page->slab_cache; - void *object = nearest_obj(cache, page, addr); + if (slab && slab_test_cache(slab)) { + struct kmem_cache *cache = slab->slab_cache; + void *object = nearest_obj(cache, slab, addr); describe_object(cache, object, addr, tag); } @@ -268,9 +268,9 @@ static void print_address_description(void *addr, u8 tag) pr_err(" %pS\n", addr); } - if (page) { + if (slab) { pr_err("The buggy address belongs to the page:\n"); - dump_page(page, "kasan: bad access detected"); + dump_page(slab_page(slab), "kasan: bad access detected"); } kasan_print_address_stack_frame(addr); diff --git a/mm/kasan/report_tags.c b/mm/kasan/report_tags.c index 8a319fc16dab..16a3c55ce698 100644 --- a/mm/kasan/report_tags.c +++ b/mm/kasan/report_tags.c @@ -12,7 +12,7 @@ const char *kasan_get_bug_type(struct kasan_access_info *info) #ifdef CONFIG_KASAN_TAGS_IDENTIFY struct kasan_alloc_meta *alloc_meta; struct kmem_cache *cache; - struct page *page; + struct slab *slab; const void *addr; void *object; u8 tag; @@ -20,10 +20,10 @@ const char *kasan_get_bug_type(struct kasan_access_info *info) tag = get_tag(info->access_addr); addr = kasan_reset_tag(info->access_addr); - page = kasan_addr_to_page(addr); - if (page && PageSlab(page)) { - cache = page->slab_cache; - object = nearest_obj(cache, page, (void *)addr); + slab = kasan_addr_to_slab(addr); + if (slab && SlabAllocation(slab)) { + cache = slab->slab_cache; + object = nearest_obj(cache, slab, (void *)addr); alloc_meta = kasan_get_alloc_meta(cache, object); if (alloc_meta) { diff --git a/mm/slab.c b/mm/slab.c index 3e9cd3ecc9ab..8cbb6e91922e 100644 --- a/mm/slab.c +++ b/mm/slab.c @@ -2612,7 +2612,7 @@ static struct slab *cache_grow_begin(struct kmem_cache *cachep, * slab_address() in the latter returns a non-tagged pointer, * as it should be for slab pages. */ - kasan_poison_slab(slab_page(slab)); + kasan_poison_slab(slab); /* Get slab management. */ freelist = alloc_slabmgmt(cachep, slab, offset, diff --git a/mm/slub.c b/mm/slub.c index 659b30afbb58..998c1eefd205 100644 --- a/mm/slub.c +++ b/mm/slub.c @@ -1918,7 +1918,7 @@ static struct slab *allocate_slab(struct kmem_cache *s, gfp_t flags, int node) account_slab(slab, oo_order(oo), s, flags); slab->slab_cache = s; - kasan_poison_slab(slab_page(slab)); + kasan_poison_slab(slab); start = slab_address(slab);