From patchwork Fri Aug 25 21:14:26 2023 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Jann Horn X-Patchwork-Id: 13366330 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from kanga.kvack.org (kanga.kvack.org [205.233.56.17]) by smtp.lore.kernel.org (Postfix) with ESMTP id 37765EE49B5 for ; Fri, 25 Aug 2023 21:15:07 +0000 (UTC) Received: by kanga.kvack.org (Postfix) id 45D692800E1; Fri, 25 Aug 2023 17:15:06 -0400 (EDT) Received: by kanga.kvack.org (Postfix, from userid 40) id 40DF82800DD; Fri, 25 Aug 2023 17:15:06 -0400 (EDT) X-Delivered-To: int-list-linux-mm@kvack.org Received: by kanga.kvack.org (Postfix, from userid 63042) id 2AEB52800E1; Fri, 25 Aug 2023 17:15:06 -0400 (EDT) X-Delivered-To: linux-mm@kvack.org Received: from relay.hostedemail.com (smtprelay0017.hostedemail.com [216.40.44.17]) by kanga.kvack.org (Postfix) with ESMTP id 1997C2800DD for ; Fri, 25 Aug 2023 17:15:06 -0400 (EDT) Received: from smtpin08.hostedemail.com (a10.router.float.18 [10.200.18.1]) by unirelay04.hostedemail.com (Postfix) with ESMTP id D7CB51A06BA for ; Fri, 25 Aug 2023 21:15:05 +0000 (UTC) X-FDA: 81163882170.08.D9271FF Received: from mail-ed1-f49.google.com (mail-ed1-f49.google.com [209.85.208.49]) by imf04.hostedemail.com (Postfix) with ESMTP id 4C7864001D for ; Fri, 25 Aug 2023 21:15:03 +0000 (UTC) Authentication-Results: imf04.hostedemail.com; dkim=pass header.d=google.com header.s=20221208 header.b=jxy91XGR; spf=pass (imf04.hostedemail.com: domain of jannh@google.com designates 209.85.208.49 as permitted sender) smtp.mailfrom=jannh@google.com; dmarc=pass (policy=reject) header.from=google.com ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=hostedemail.com; s=arc-20220608; t=1692998104; h=from:from:sender:reply-to:subject:subject:date:date: message-id:message-id:to:to:cc:cc:mime-version:mime-version: content-type:content-transfer-encoding:content-transfer-encoding: in-reply-to:references:dkim-signature; bh=aPiiwNs7Vv+AACjowQ9IHJzrj9oQRfhXdiPEG1HBSek=; b=VUECWCsqhGBQLPulK2fqgkWzBGFoRRwp0WgKsJqdLpFAXbckHj6zyHFcYl++9NZplfIkiR 7iNtO2QkVsTAslXcPVxEpMv5behWNzDom/suFby6lbhbZW1YoiPU7GWuqRBZJJOMMopx7I 4dqq3PW+FRLgVHJTlsLsl00UM7yiIx4= ARC-Seal: i=1; s=arc-20220608; d=hostedemail.com; t=1692998104; a=rsa-sha256; cv=none; b=TWMmf4862JXJAysbvedbS0t3e7oUGD4baB2KhRK9nt0XVmRj7XEpPCWCfD4D/HD74Rp5UQ dnAS6yOTesEY76QzBYbMHI0kHWPyn8dR+SZgRRxbqSpr65A5FRlY7HF67AAFw7L51Y8HFX pLGtaciRhrBnpF7j3/UYl5/yxa6wN5M= ARC-Authentication-Results: i=1; imf04.hostedemail.com; dkim=pass header.d=google.com header.s=20221208 header.b=jxy91XGR; spf=pass (imf04.hostedemail.com: domain of jannh@google.com designates 209.85.208.49 as permitted sender) smtp.mailfrom=jannh@google.com; dmarc=pass (policy=reject) header.from=google.com Received: by mail-ed1-f49.google.com with SMTP id 4fb4d7f45d1cf-5232ce75e26so1097a12.1 for ; Fri, 25 Aug 2023 14:15:03 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=google.com; s=20221208; t=1692998102; x=1693602902; darn=kvack.org; h=content-transfer-encoding:mime-version:message-id:date:subject:cc :to:from:from:to:cc:subject:date:message-id:reply-to; bh=aPiiwNs7Vv+AACjowQ9IHJzrj9oQRfhXdiPEG1HBSek=; b=jxy91XGRnAKBTpBqSYVjac8UVd5pQcw846JfD9lZo4rOaXgE33WeJMFLYXb5kqNxhT Ut9fcmP5UPqvXTgMwhZyRcNACeUWL6agPUWDlCmE0DJOs0xDaV88CmoTxFhljrKON7pn m77NbYJYdTcGZluu3O90XWVoBDFepD4d5IVEIRBUT3bvY1fMnGliIOzrIxC88KLIkVZ+ xRhT46POKs5tBZMh7tYYCqzn3VZo11ACwo7OwJrZlwtKqMcpaPDtSyp8s4X5s9+A1lnH sv+SJCb1+gQrdeoLkC3K1xStkYPkmjOrfB5d+c99OV5L/ZRQa2YaoQHmk8uw7LpdNLt1 0lRg== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20221208; t=1692998102; x=1693602902; h=content-transfer-encoding:mime-version:message-id:date:subject:cc :to:from:x-gm-message-state:from:to:cc:subject:date:message-id :reply-to; bh=aPiiwNs7Vv+AACjowQ9IHJzrj9oQRfhXdiPEG1HBSek=; b=hs3+2ZiVkw3/Ajt72F5gFkmgSPwlaES8yNJ8sNJl/I3tPlmqWjP4sdlriXESjbxYWy XvRoyKCMLYYQqC2ze9uLx9LcnILhkQgo3NHT+AYpEtTrgHiM9ZLu54GIXiEyzKIsh0SL 3DTvz2vA/UmhT9noLo+wmY0b+r3fVakV3dxuP1Wj7SKUiA9PZTglklB1nIo/Zhdh3fxU OSQT0FjPr8nTqCkY4a6F2ZqWXGUopxSNN+LkYRgBVsb14ZlFyTcOGV6WODKhMJdWYHn1 ZwGTjkOcPO/QDuuJvYILUnBIQQaeB48GVftaOZqsMyaLDCQpom2fqyf9QMZEE2A0tnZa 8AHA== X-Gm-Message-State: AOJu0YyoTC8Y3sWSu9AdfSkf0H/J2u+Hl5hraaWhlEKh89xT4cA4Aw0w Llyw27Bj3BkGusJnIItHyWQLrg== X-Google-Smtp-Source: AGHT+IFOC7fOBw85xfs7OafKE6fcFNFqYDODY0kmvrFIRNiwhJgQsfC+F07WbB8Fp3sjyDsjlChDzw== X-Received: by 2002:a50:d55c:0:b0:523:d5bc:8424 with SMTP id f28-20020a50d55c000000b00523d5bc8424mr27062edj.5.1692998101863; Fri, 25 Aug 2023 14:15:01 -0700 (PDT) Received: from localhost ([2a00:79e0:9d:4:7803:f309:e9d9:478]) by smtp.gmail.com with ESMTPSA id d7-20020adffd87000000b00317a04131c5sm3177620wrr.57.2023.08.25.14.15.00 (version=TLS1_3 cipher=TLS_AES_256_GCM_SHA384 bits=256/256); Fri, 25 Aug 2023 14:15:00 -0700 (PDT) From: Jann Horn To: Andrey Ryabinin , Christoph Lameter , Pekka Enberg , David Rientjes , Joonsoo Kim , Vlastimil Babka Cc: Alexander Potapenko , Andrey Konovalov , Dmitry Vyukov , Vincenzo Frascino , Andrew Morton , Roman Gushchin , Hyeonggon Yoo <42.hyeyoo@gmail.com>, kasan-dev@googlegroups.com, linux-kernel@vger.kernel.org, linux-mm@kvack.org, linux-hardening@vger.kernel.org, kernel-hardening@lists.openwall.com Subject: [PATCH] slub: Introduce CONFIG_SLUB_RCU_DEBUG Date: Fri, 25 Aug 2023 23:14:26 +0200 Message-ID: <20230825211426.3798691-1-jannh@google.com> X-Mailer: git-send-email 2.42.0.rc1.204.g551eb34607-goog MIME-Version: 1.0 X-Rspamd-Queue-Id: 4C7864001D X-Rspam-User: X-Stat-Signature: sqf5xhwung6mayozqxkcw3unohtuua9d X-Rspamd-Server: rspam03 X-HE-Tag: 1692998103-485049 X-HE-Meta: U2FsdGVkX19txaDuF8xKXA2RulCUw/FaUYKv+tsV7MmYzCgqfZyCZ4/gG2CuoUDBv+URBFd7HdQ/nHIYDl4Eiqcp74SWZJTBAGhl/XCkCLH7u11KAHAx4odiDYL0szknz4wJvBS+tgfo2PXHscK1800S73qgt5oD3CytLpsTf4L8cVBB1bDnFjtgf7fS8kXN7iXYR+hjtPIVpXfM1T9Q/CQF+lT9TMqhykEL6CFzCCMn46GnbOCD/OiufozOACu0tWo9LwZJw5sPwdvrNCKWTwNz1fgSVPQAOdQo6Fqyjy23pTbMoTe76dtlNcUZ3yqQd3ih2PsqCMXol8td4A/0WHWzkx4dIcSvMurql4fp5bT2fEn4ru1pQvn2uP5YQSeJRZDPJO0PqKjdiH9Y9Mh3uC0I8xg83aPuUa4wPZj8FgAlJJwQrq5NRmxQLK0n4yl0kIhWLcSMNGZSfb4bwRyzS8jUAC6iqSUwcwkSd077+ICM4Tn5t97uwi+JRreCuYxF4CnZQ4cACvbu6a64NgySMnIlZU19isCx9dsO20Cf1JftIclbw4T3FII7hhxmFQqOqGForLnJ8xmhbeFYJUB8Bu31WFCHsI/KzQJ7aJ8RduQfvXcaqUpxmkca7YG+apqpBbL7meqS+NNGo2lily2fuPay2WjBxBj7rzhR9ZPU5C3s9idboTZs5xY/IDrWTzWfmZWDbpR4zyvFGgxNDNpM2zIeSBn710Iu7yjee68zituEzsDLaur/inTNoeX3gU774pxDcnggszGGBDsq4suawoOCYCA3SnMGWY5UqGOI1Kgb8a9ROmHNSITW9kpyvcWqe18sfnwpIHxLTQ/fk2eAQb5BHB70s2bvwBURP7xfkJJ87K548nWuhw8p59DSg3urXZbulFkDqShRQuICiEnInmW4pXsEE8UY3z8dOJS72Q3fXUPgu0bUU/EPzEs+6fhnWYZJ/zFgUuSq9LLTW9z H4OEhzZN n1pdr8e4W/+24vlSZzZBJYsBPd2SFJYRjzUpXx2v8SRSB8N7aldy9vmPPBCb2aSbVtAO13IJmqbiPdxfzrFTkBFAP9ku/R/KvKjWGu8tikzoGuRTEGU5/5ZVNArt04RkEXjn8QsEw/C67u60RjfLdMeS1VEXQqn1Z7FdSCZLVKtNAXhs/QIQDINpxwz9R6i46kJtgiw0Q3rzPtvU92hLD8RTRb9tJD/NlwvziRazcmjbwyScbWCMWYItyIs8gAFsQpej3Ykh3xF0ewOM3y44fiuZEmboD/CESiuejAWS4oMLV/6K6r9SqUZ8cA6UilF8YvSZ+qOR07w3o9gPjfr80IjuatjN+kSoOW3Hr2EZ5OlrNPD3fTfKHqqrc84/0fPMcID48hI0ELZD8auvNrnLI0hXOy/vtLWXUS18uBIUACN4X9irtKFbxt2fQa9FBdpwShQrGL2FUkTUz48nYFdpNt+j6q5QOu6o+OLDDZyDTgeAykClro/+Op3AAsw== X-Bogosity: Ham, tests=bogofilter, spamicity=0.000000, version=1.2.4 Sender: owner-linux-mm@kvack.org Precedence: bulk X-Loop: owner-majordomo@kvack.org List-ID: Currently, KASAN is unable to catch use-after-free in SLAB_TYPESAFE_BY_RCU slabs because use-after-free is allowed within the RCU grace period by design. Add a SLUB debugging feature which RCU-delays every individual kmem_cache_free() before either actually freeing the object or handing it off to KASAN, and change KASAN to poison freed objects as normal when this option is enabled. Note that this creates a 16-byte unpoisoned area in the middle of the slab metadata area, which kinda sucks but seems to be necessary in order to be able to store an rcu_head in there without triggering an ASAN splat during RCU callback processing. For now I've configured Kconfig.kasan to always enable this feature in the GENERIC and SW_TAGS modes; I'm not forcibly enabling it in HW_TAGS mode because I'm not sure if it might have unwanted performance degradation effects there. Signed-off-by: Jann Horn --- can I get a review from the KASAN folks of this? I have been running it on my laptop for a bit and it seems to be working fine. Notes: With this patch, a UAF on a TYPESAFE_BY_RCU will splat with an error like this (tested by reverting a security bugfix). Note that, in the ASAN memory state dump, we can see the little unpoisoned 16-byte areas storing the rcu_head. BUG: KASAN: slab-use-after-free in folio_lock_anon_vma_read+0x129/0x4c0 Read of size 8 at addr ffff888004e85b00 by task forkforkfork/592 CPU: 0 PID: 592 Comm: forkforkfork Not tainted 6.5.0-rc7-00105-gae70c1e1f6f5-dirty #334 Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.16.2-debian-1.16.2-1 04/01/2014 Call Trace: dump_stack_lvl+0x4a/0x80 print_report+0xcf/0x660 kasan_report+0xd4/0x110 folio_lock_anon_vma_read+0x129/0x4c0 rmap_walk_anon+0x1cc/0x290 folio_referenced+0x277/0x2a0 shrink_folio_list+0xb8c/0x1680 reclaim_folio_list+0xdc/0x1f0 reclaim_pages+0x211/0x280 madvise_cold_or_pageout_pte_range+0x812/0xb70 walk_pgd_range+0x70b/0xce0 __walk_page_range+0x343/0x360 walk_page_range+0x227/0x280 madvise_pageout+0x1cd/0x2d0 do_madvise+0x552/0x15a0 __x64_sys_madvise+0x62/0x70 do_syscall_64+0x3b/0x90 entry_SYSCALL_64_after_hwframe+0x6e/0xd8 [...] Allocated by task 574: kasan_save_stack+0x33/0x60 kasan_set_track+0x25/0x30 __kasan_slab_alloc+0x6e/0x70 kmem_cache_alloc+0xfd/0x2b0 anon_vma_fork+0x88/0x270 dup_mmap+0x87c/0xc10 copy_process+0x3399/0x3590 kernel_clone+0x10e/0x480 __do_sys_clone+0xa1/0xe0 do_syscall_64+0x3b/0x90 entry_SYSCALL_64_after_hwframe+0x6e/0xd8 Freed by task 0: kasan_save_stack+0x33/0x60 kasan_set_track+0x25/0x30 kasan_save_free_info+0x2b/0x50 __kasan_slab_free+0xfe/0x180 slab_free_after_rcu_debug+0xad/0x200 rcu_core+0x638/0x1620 __do_softirq+0x14c/0x581 Last potentially related work creation: kasan_save_stack+0x33/0x60 __kasan_record_aux_stack+0x94/0xa0 __call_rcu_common.constprop.0+0x47/0x730 __put_anon_vma+0x6e/0x150 unlink_anon_vmas+0x277/0x2e0 vma_complete+0x341/0x580 vma_merge+0x613/0xff0 mprotect_fixup+0x1c0/0x510 do_mprotect_pkey+0x5a7/0x710 __x64_sys_mprotect+0x47/0x60 do_syscall_64+0x3b/0x90 entry_SYSCALL_64_after_hwframe+0x6e/0xd8 Second to last potentially related work creation: [...] The buggy address belongs to the object at ffff888004e85b00 which belongs to the cache anon_vma of size 192 The buggy address is located 0 bytes inside of freed 192-byte region [ffff888004e85b00, ffff888004e85bc0) The buggy address belongs to the physical page: [...] Memory state around the buggy address: ffff888004e85a00: 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 ffff888004e85a80: 00 00 00 00 00 00 00 00 fc 00 00 fc fc fc fc fc >ffff888004e85b00: fa fb fb fb fb fb fb fb fb fb fb fb fb fb fb fb ^ ffff888004e85b80: fb fb fb fb fb fb fb fb fc 00 00 fc fc fc fc fc ffff888004e85c00: fa fb fb fb fb fb fb fb fb fb fb fb fb fb fb fb include/linux/kasan.h | 6 ++++ include/linux/slub_def.h | 3 ++ lib/Kconfig.kasan | 2 ++ mm/Kconfig.debug | 21 +++++++++++++ mm/kasan/common.c | 15 ++++++++- mm/slub.c | 66 +++++++++++++++++++++++++++++++++++++--- 6 files changed, 107 insertions(+), 6 deletions(-) base-commit: 4f9e7fabf8643003afefc172e62dd276686f016e diff --git a/include/linux/kasan.h b/include/linux/kasan.h index 819b6bc8ac08..45e07caf4704 100644 --- a/include/linux/kasan.h +++ b/include/linux/kasan.h @@ -229,6 +229,8 @@ static __always_inline bool kasan_check_byte(const void *addr) return true; } +size_t kasan_align(size_t size); + #else /* CONFIG_KASAN */ static inline void kasan_unpoison_range(const void *address, size_t size) {} @@ -278,6 +280,10 @@ static inline bool kasan_check_byte(const void *address) { return true; } +static inline size_t kasan_align(size_t size) +{ + return size; +} #endif /* CONFIG_KASAN */ diff --git a/include/linux/slub_def.h b/include/linux/slub_def.h index deb90cf4bffb..b87be8fce64a 100644 --- a/include/linux/slub_def.h +++ b/include/linux/slub_def.h @@ -120,6 +120,9 @@ struct kmem_cache { int refcount; /* Refcount for slab cache destroy */ void (*ctor)(void *); unsigned int inuse; /* Offset to metadata */ +#ifdef CONFIG_SLUB_RCU_DEBUG + unsigned int debug_rcu_head_offset; +#endif unsigned int align; /* Alignment */ unsigned int red_left_pad; /* Left redzone padding size */ const char *name; /* Name (only for display!) */ diff --git a/lib/Kconfig.kasan b/lib/Kconfig.kasan index fdca89c05745..7ff7de96c6e4 100644 --- a/lib/Kconfig.kasan +++ b/lib/Kconfig.kasan @@ -79,6 +79,7 @@ config KASAN_GENERIC depends on HAVE_ARCH_KASAN && CC_HAS_KASAN_GENERIC depends on CC_HAS_WORKING_NOSANITIZE_ADDRESS select SLUB_DEBUG if SLUB + select SLUB_RCU_DEBUG if SLUB_DEBUG select CONSTRUCTORS help Enables Generic KASAN. @@ -96,6 +97,7 @@ config KASAN_SW_TAGS depends on HAVE_ARCH_KASAN_SW_TAGS && CC_HAS_KASAN_SW_TAGS depends on CC_HAS_WORKING_NOSANITIZE_ADDRESS select SLUB_DEBUG if SLUB + select SLUB_RCU_DEBUG if SLUB_DEBUG select CONSTRUCTORS help Enables Software Tag-Based KASAN. diff --git a/mm/Kconfig.debug b/mm/Kconfig.debug index 018a5bd2f576..99cce7f0fbef 100644 --- a/mm/Kconfig.debug +++ b/mm/Kconfig.debug @@ -78,6 +78,27 @@ config SLUB_DEBUG_ON off in a kernel built with CONFIG_SLUB_DEBUG_ON by specifying "slub_debug=-". +config SLUB_RCU_DEBUG + bool "Make use-after-free detection possible in TYPESAFE_BY_RCU caches" + depends on SLUB && SLUB_DEBUG + default n + help + Make SLAB_TYPESAFE_BY_RCU caches behave approximately as if the cache + was not marked as SLAB_TYPESAFE_BY_RCU and every caller used + kfree_rcu() instead. + + This is intended for use in combination with KASAN, to enable KASAN to + detect use-after-free accesses in such caches. + (KFENCE is able to do that independent of this flag.) + + This might degrade performance. + + If you're using this for testing bugs / fuzzing and care about + catching all the bugs WAY more than performance, you might want to + also turn on CONFIG_RCU_STRICT_GRACE_PERIOD. + + If unsure, say N. + config PAGE_OWNER bool "Track page owner" depends on DEBUG_KERNEL && STACKTRACE_SUPPORT diff --git a/mm/kasan/common.c b/mm/kasan/common.c index 256930da578a..b4a3504f9f5e 100644 --- a/mm/kasan/common.c +++ b/mm/kasan/common.c @@ -191,6 +191,13 @@ void * __must_check __kasan_init_slab_obj(struct kmem_cache *cache, if (kasan_requires_meta()) kasan_init_object_meta(cache, object); +#ifdef CONFIG_SLUB_RCU_DEBUG + if (cache->flags & SLAB_TYPESAFE_BY_RCU) { + kasan_unpoison(object + cache->debug_rcu_head_offset, + sizeof(struct rcu_head), false); + } +#endif /* CONFIG_SLUB_RCU_DEBUG */ + /* Tag is ignored in set_tag() without CONFIG_KASAN_SW/HW_TAGS */ object = set_tag(object, assign_tag(cache, object, true)); @@ -218,7 +225,8 @@ static inline bool ____kasan_slab_free(struct kmem_cache *cache, void *object, } /* RCU slabs could be legally used after free within the RCU period */ - if (unlikely(cache->flags & SLAB_TYPESAFE_BY_RCU)) + if (unlikely(cache->flags & SLAB_TYPESAFE_BY_RCU) && + !IS_ENABLED(CONFIG_SLUB_RCU_DEBUG)) return false; if (!kasan_byte_accessible(tagged_object)) { @@ -450,3 +458,8 @@ bool __kasan_check_byte(const void *address, unsigned long ip) } return true; } + +size_t kasan_align(size_t size) +{ + return round_up(size, KASAN_GRANULE_SIZE); +} diff --git a/mm/slub.c b/mm/slub.c index e3b5d5c0eb3a..bae6c2bc1e5f 100644 --- a/mm/slub.c +++ b/mm/slub.c @@ -1108,7 +1108,8 @@ static int check_bytes_and_report(struct kmem_cache *s, struct slab *slab, * A. Free pointer (if we cannot overwrite object on free) * B. Tracking data for SLAB_STORE_USER * C. Original request size for kmalloc object (SLAB_STORE_USER enabled) - * D. Padding to reach required alignment boundary or at minimum + * D. RCU head for CONFIG_SLUB_RCU_DEBUG (with padding around it) + * E. Padding to reach required alignment boundary or at minimum * one word if debugging is on to be able to detect writes * before the word boundary. * @@ -1134,6 +1135,11 @@ static int check_pad_bytes(struct kmem_cache *s, struct slab *slab, u8 *p) off += sizeof(unsigned int); } +#ifdef CONFIG_SLUB_RCU_DEBUG + if (s->flags & SLAB_TYPESAFE_BY_RCU) + off = kasan_align(s->debug_rcu_head_offset + sizeof(struct rcu_head)); +#endif /* CONFIG_SLUB_RCU_DEBUG */ + off += kasan_metadata_size(s, false); if (size_from_object(s) == off) @@ -1751,12 +1757,17 @@ static bool freelist_corrupted(struct kmem_cache *s, struct slab *slab, #endif #endif /* CONFIG_SLUB_DEBUG */ +#ifdef CONFIG_SLUB_RCU_DEBUG +static void slab_free_after_rcu_debug(struct rcu_head *rcu_head); +#endif + /* * Hooks for other subsystems that check memory allocations. In a typical * production configuration these hooks all should produce no code at all. */ static __always_inline bool slab_free_hook(struct kmem_cache *s, - void *x, bool init) + void *x, bool init, + bool after_rcu_delay) { kmemleak_free_recursive(x, s->flags); kmsan_slab_free(s, x); @@ -1766,8 +1777,18 @@ static __always_inline bool slab_free_hook(struct kmem_cache *s, if (!(s->flags & SLAB_DEBUG_OBJECTS)) debug_check_no_obj_freed(x, s->object_size); +#ifdef CONFIG_SLUB_RCU_DEBUG + /* kfence does its own RCU delay */ + if ((s->flags & SLAB_TYPESAFE_BY_RCU) && !after_rcu_delay && + !is_kfence_address(x)) { + call_rcu(kasan_reset_tag(x) + s->debug_rcu_head_offset, + slab_free_after_rcu_debug); + return true; + } +#endif /* CONFIG_SLUB_RCU_DEBUG */ + /* Use KCSAN to help debug racy use-after-free. */ - if (!(s->flags & SLAB_TYPESAFE_BY_RCU)) + if (!(s->flags & SLAB_TYPESAFE_BY_RCU) || after_rcu_delay) __kcsan_check_access(x, s->object_size, KCSAN_ACCESS_WRITE | KCSAN_ACCESS_ASSERT); @@ -1802,7 +1823,7 @@ static inline bool slab_free_freelist_hook(struct kmem_cache *s, void *old_tail = *tail ? *tail : *head; if (is_kfence_address(next)) { - slab_free_hook(s, next, false); + slab_free_hook(s, next, false, false); return true; } @@ -1815,7 +1836,7 @@ static inline bool slab_free_freelist_hook(struct kmem_cache *s, next = get_freepointer(s, object); /* If object's reuse doesn't have to be delayed */ - if (!slab_free_hook(s, object, slab_want_init_on_free(s))) { + if (!slab_free_hook(s, object, slab_want_init_on_free(s), false)) { /* Move object to the new freelist */ set_freepointer(s, object, *head); *head = object; @@ -3802,6 +3823,31 @@ static __fastpath_inline void slab_free(struct kmem_cache *s, struct slab *slab, do_slab_free(s, slab, head, tail, cnt, addr); } +#ifdef CONFIG_SLUB_RCU_DEBUG +static void slab_free_after_rcu_debug(struct rcu_head *rcu_head) +{ + struct slab *slab = virt_to_slab(rcu_head); + struct kmem_cache *s; + void *object; + + if (WARN_ON(is_kfence_address(rcu_head))) + return; + + /* find the object and the cache again */ + if (WARN_ON(!slab)) + return; + s = slab->slab_cache; + if (WARN_ON(!(s->flags & SLAB_TYPESAFE_BY_RCU))) + return; + object = (void *)rcu_head - s->debug_rcu_head_offset; + + /* resume freeing */ + if (slab_free_hook(s, object, slab_want_init_on_free(s), true)) + return; + do_slab_free(s, slab, object, NULL, 1, _THIS_IP_); +} +#endif /* CONFIG_SLUB_RCU_DEBUG */ + #ifdef CONFIG_KASAN_GENERIC void ___cache_free(struct kmem_cache *cache, void *x, unsigned long addr) { @@ -4443,6 +4489,16 @@ static int calculate_sizes(struct kmem_cache *s) if (flags & SLAB_KMALLOC) size += sizeof(unsigned int); } + +#ifdef CONFIG_SLUB_RCU_DEBUG + if (flags & SLAB_TYPESAFE_BY_RCU) { + size = kasan_align(size); + size = ALIGN(size, __alignof__(struct rcu_head)); + s->debug_rcu_head_offset = size; + size += sizeof(struct rcu_head); + size = kasan_align(size); + } +#endif /* CONFIG_SLUB_RCU_DEBUG */ #endif kasan_cache_create(s, &size, &s->flags);