From patchwork Sat Dec 5 00:40:52 2020 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: "Paul E. McKenney" X-Patchwork-Id: 11952641 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org X-Spam-Level: X-Spam-Status: No, score=-17.0 required=3.0 tests=BAYES_00,INCLUDES_CR_TRAILER, INCLUDES_PATCH,MAILING_LIST_MULTI,SPF_HELO_NONE,SPF_PASS,URIBL_BLOCKED, USER_AGENT_GIT autolearn=ham autolearn_force=no version=3.4.0 Received: from mail.kernel.org (mail.kernel.org [198.145.29.99]) by smtp.lore.kernel.org (Postfix) with ESMTP id 88DD1C433FE for ; Sat, 5 Dec 2020 00:41:03 +0000 (UTC) Received: from kanga.kvack.org (kanga.kvack.org [205.233.56.17]) by mail.kernel.org (Postfix) with ESMTP id 0D30422DA9 for ; Sat, 5 Dec 2020 00:41:02 +0000 (UTC) DMARC-Filter: OpenDMARC Filter v1.3.2 mail.kernel.org 0D30422DA9 Authentication-Results: mail.kernel.org; dmarc=fail (p=none dis=none) header.from=kernel.org Authentication-Results: mail.kernel.org; spf=pass smtp.mailfrom=owner-linux-mm@kvack.org Received: by kanga.kvack.org (Postfix) id 24F1A6B0036; Fri, 4 Dec 2020 19:41:02 -0500 (EST) Received: by kanga.kvack.org (Postfix, from userid 40) id 1FF3E6B005D; Fri, 4 Dec 2020 19:41:02 -0500 (EST) X-Delivered-To: int-list-linux-mm@kvack.org Received: by kanga.kvack.org (Postfix, from userid 63042) id 115D36B0068; Fri, 4 Dec 2020 19:41:02 -0500 (EST) X-Delivered-To: linux-mm@kvack.org Received: from forelay.hostedemail.com (smtprelay0046.hostedemail.com [216.40.44.46]) by kanga.kvack.org (Postfix) with ESMTP id F012A6B0036 for ; Fri, 4 Dec 2020 19:41:01 -0500 (EST) Received: from smtpin14.hostedemail.com (10.5.19.251.rfc1918.com [10.5.19.251]) by forelay03.hostedemail.com (Postfix) with ESMTP id A95708249980 for ; Sat, 5 Dec 2020 00:41:01 +0000 (UTC) X-FDA: 77557373922.14.wall14_5b047c3273c8 Received: from filter.hostedemail.com (10.5.16.251.rfc1918.com [10.5.16.251]) by smtpin14.hostedemail.com (Postfix) with ESMTP id 8613918229837 for ; Sat, 5 Dec 2020 00:41:01 +0000 (UTC) X-HE-Tag: wall14_5b047c3273c8 X-Filterd-Recvd-Size: 7204 Received: from mail.kernel.org (mail.kernel.org [198.145.29.99]) by imf50.hostedemail.com (Postfix) with ESMTP for ; Sat, 5 Dec 2020 00:41:00 +0000 (UTC) From: paulmck@kernel.org Authentication-Results: mail.kernel.org; dkim=permerror (bad message/signature format) To: rcu@vger.kernel.org Cc: linux-kernel@vger.kernel.org, kernel-team@fb.com, mingo@kernel.org, jiangshanlai@gmail.com, akpm@linux-foundation.org, mathieu.desnoyers@efficios.com, josh@joshtriplett.org, tglx@linutronix.de, peterz@infradead.org, rostedt@goodmis.org, dhowells@redhat.com, edumazet@google.com, fweisbec@gmail.com, oleg@redhat.com, joel@joelfernandes.org, "Paul E. McKenney" , Christoph Lameter , Pekka Enberg , David Rientjes , Joonsoo Kim , linux-mm@kvack.org Subject: [PATCH sl-b 1/6] mm: Add kmem_last_alloc() to return last allocation for memory block Date: Fri, 4 Dec 2020 16:40:52 -0800 Message-Id: <20201205004057.32199-1-paulmck@kernel.org> X-Mailer: git-send-email 2.9.5 In-Reply-To: <20201205004022.GA31166@paulmck-ThinkPad-P72> References: <20201205004022.GA31166@paulmck-ThinkPad-P72> X-Bogosity: Ham, tests=bogofilter, spamicity=0.000000, version=1.2.4 Sender: owner-linux-mm@kvack.org Precedence: bulk X-Loop: owner-majordomo@kvack.org List-ID: From: "Paul E. McKenney" There are kernel facilities such as per-CPU reference counts that give error messages in generic handlers or callbacks, whose messages are unenlightening. In the case of per-CPU reference-count underflow, this is not a problem when creating a new use of this facility because in that case the bug is almost certainly in the code implementing that new use. However, trouble arises when deploying across many systems, which might exercise corner cases that were not seen during development and testing. Here, it would be really nice to get some kind of hint as to which of several uses the underflow was caused by. This commit therefore exposes a new kmem_last_alloc() function that takes a pointer to dynamically allocated memory and returns the return address of the call that allocated it. This pointer can reference the middle of the block as well as the beginning of the block, as needed by things like RCU callback functions and timer handlers that might not know where the beginning of the memory block is. These functions and handlers can use the return value from kmem_last_alloc() to give the kernel hacker a better hint as to where the problem might lie. This kmem_last_alloc() function returns NULL for slob and when the necessary debug has not been enabled for slab and slub. For slub, build with CONFIG_SLUB_DEBUG=y and boot with slub_debug=U, or pass SLAB_STORE_USER to kmem_cache_create() if more focused use is desired. Cc: Christoph Lameter Cc: Pekka Enberg Cc: David Rientjes Cc: Joonsoo Kim Cc: Andrew Morton Cc: Reported-by: Andrii Nakryiko Signed-off-by: Paul E. McKenney --- include/linux/slab.h | 2 ++ mm/slab.c | 19 +++++++++++++++++++ mm/slab_common.c | 20 ++++++++++++++++++++ mm/slob.c | 5 +++++ mm/slub.c | 26 ++++++++++++++++++++++++++ 5 files changed, 72 insertions(+) diff --git a/include/linux/slab.h b/include/linux/slab.h index dd6897f..06dd56b 100644 --- a/include/linux/slab.h +++ b/include/linux/slab.h @@ -186,6 +186,8 @@ void kfree(const void *); void kfree_sensitive(const void *); size_t __ksize(const void *); size_t ksize(const void *); +void *kmem_cache_last_alloc(struct kmem_cache *s, void *object); +void *kmem_last_alloc(void *object); #ifdef CONFIG_HAVE_HARDENED_USERCOPY_ALLOCATOR void __check_heap_object(const void *ptr, unsigned long n, struct page *page, diff --git a/mm/slab.c b/mm/slab.c index b111356..2ab93b8 100644 --- a/mm/slab.c +++ b/mm/slab.c @@ -3602,6 +3602,25 @@ void *kmem_cache_alloc_node_trace(struct kmem_cache *cachep, EXPORT_SYMBOL(kmem_cache_alloc_node_trace); #endif +void *kmem_cache_last_alloc(struct kmem_cache *cachep, void *object) +{ +#ifdef DEBUG + unsigned int objnr; + void *objp; + struct page *page; + + if (!(cachep->flags & SLAB_STORE_USER)) + return NULL; + objp = object - obj_offset(cachep); + page = virt_to_head_page(objp); + objnr = obj_to_index(cachep, page, objp); + objp = index_to_obj(cachep, page, objnr); + return *dbg_userword(cachep, objp); +#else + return NULL; +#endif +} + static __always_inline void * __do_kmalloc_node(size_t size, gfp_t flags, int node, unsigned long caller) { diff --git a/mm/slab_common.c b/mm/slab_common.c index f9ccd5d..3f647982 100644 --- a/mm/slab_common.c +++ b/mm/slab_common.c @@ -536,6 +536,26 @@ bool slab_is_available(void) return slab_state >= UP; } +/* + * If the pointer references a slab-allocated object and if sufficient + * debugging is enabled, return the returrn address for the corresponding + * allocation. Otherwise, return NULL. Note that passing random pointers + * to this function (including addresses of on-stack variables) is likely + * to result in panics. + */ +void *kmem_last_alloc(void *object) +{ + struct page *page; + + if (!virt_addr_valid(object)) + return NULL; + page = virt_to_head_page(object); + if (!PageSlab(page)) + return NULL; + return kmem_cache_last_alloc(page->slab_cache, object); +} +EXPORT_SYMBOL_GPL(kmem_last_alloc); + #ifndef CONFIG_SLOB /* Create a cache during boot when no slab services are available yet */ void __init create_boot_cache(struct kmem_cache *s, const char *name, diff --git a/mm/slob.c b/mm/slob.c index 7cc9805..c1f8ed7 100644 --- a/mm/slob.c +++ b/mm/slob.c @@ -461,6 +461,11 @@ static void slob_free(void *block, int size) spin_unlock_irqrestore(&slob_lock, flags); } +void *kmem_cache_last_alloc(struct kmem_cache *s, void *object) +{ + return NULL; +} + /* * End of slob allocator proper. Begin kmem_cache_alloc and kmalloc frontend. */ diff --git a/mm/slub.c b/mm/slub.c index b30be23..8ed3ba2 100644 --- a/mm/slub.c +++ b/mm/slub.c @@ -3918,6 +3918,32 @@ int __kmem_cache_shutdown(struct kmem_cache *s) return 0; } +void *kmem_cache_last_alloc(struct kmem_cache *s, void *object) +{ +#ifdef CONFIG_SLUB_DEBUG + void *base; + unsigned int objnr; + void *objp; + struct page *page; + struct track *trackp; + + if (!(s->flags & SLAB_STORE_USER)) + return NULL; + page = virt_to_head_page(object); + base = page_address(page); + objp = kasan_reset_tag(object); + objp = restore_red_left(s, objp); + objnr = obj_to_index(s, page, objp); + objp = base + s->size * objnr; + if (objp < base || objp >= base + page->objects * s->size || (objp - base) % s->size) + return NULL; + trackp = get_track(s, objp, TRACK_ALLOC); + return (void *)trackp->addr; +#else + return NULL; +#endif +} + /******************************************************************** * Kmalloc subsystem *******************************************************************/ From patchwork Sat Dec 5 00:40:53 2020 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: "Paul E. McKenney" X-Patchwork-Id: 11952643 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org X-Spam-Level: X-Spam-Status: No, score=-17.0 required=3.0 tests=BAYES_00,INCLUDES_CR_TRAILER, INCLUDES_PATCH,MAILING_LIST_MULTI,SPF_HELO_NONE,SPF_PASS,URIBL_BLOCKED, USER_AGENT_GIT autolearn=ham autolearn_force=no version=3.4.0 Received: from mail.kernel.org (mail.kernel.org [198.145.29.99]) by smtp.lore.kernel.org (Postfix) with ESMTP id 6C146C4361B for ; Sat, 5 Dec 2020 00:41:04 +0000 (UTC) Received: from kanga.kvack.org (kanga.kvack.org [205.233.56.17]) by mail.kernel.org (Postfix) with ESMTP id DBB6F22DD3 for ; Sat, 5 Dec 2020 00:41:03 +0000 (UTC) DMARC-Filter: OpenDMARC Filter v1.3.2 mail.kernel.org DBB6F22DD3 Authentication-Results: mail.kernel.org; dmarc=fail (p=none dis=none) header.from=kernel.org Authentication-Results: mail.kernel.org; spf=pass smtp.mailfrom=owner-linux-mm@kvack.org Received: by kanga.kvack.org (Postfix) id 70B7B6B005D; Fri, 4 Dec 2020 19:41:02 -0500 (EST) Received: by kanga.kvack.org (Postfix, from userid 40) id 6E90A6B006C; Fri, 4 Dec 2020 19:41:02 -0500 (EST) X-Delivered-To: int-list-linux-mm@kvack.org Received: by kanga.kvack.org (Postfix, from userid 63042) id 5AB406B006E; Fri, 4 Dec 2020 19:41:02 -0500 (EST) X-Delivered-To: linux-mm@kvack.org Received: from forelay.hostedemail.com (smtprelay0190.hostedemail.com [216.40.44.190]) by kanga.kvack.org (Postfix) with ESMTP id 3D0616B0068 for ; Fri, 4 Dec 2020 19:41:02 -0500 (EST) Received: from smtpin28.hostedemail.com (10.5.19.251.rfc1918.com [10.5.19.251]) by forelay03.hostedemail.com (Postfix) with ESMTP id F3FF7824999B for ; Sat, 5 Dec 2020 00:41:01 +0000 (UTC) X-FDA: 77557373964.28.grain03_0a17842273c8 Received: from filter.hostedemail.com (10.5.16.251.rfc1918.com [10.5.16.251]) by smtpin28.hostedemail.com (Postfix) with ESMTP id CC0306D76 for ; Sat, 5 Dec 2020 00:41:01 +0000 (UTC) X-HE-Tag: grain03_0a17842273c8 X-Filterd-Recvd-Size: 6568 Received: from mail.kernel.org (mail.kernel.org [198.145.29.99]) by imf31.hostedemail.com (Postfix) with ESMTP for ; Sat, 5 Dec 2020 00:41:01 +0000 (UTC) From: paulmck@kernel.org Authentication-Results: mail.kernel.org; dkim=permerror (bad message/signature format) To: rcu@vger.kernel.org Cc: linux-kernel@vger.kernel.org, kernel-team@fb.com, mingo@kernel.org, jiangshanlai@gmail.com, akpm@linux-foundation.org, mathieu.desnoyers@efficios.com, josh@joshtriplett.org, tglx@linutronix.de, peterz@infradead.org, rostedt@goodmis.org, dhowells@redhat.com, edumazet@google.com, fweisbec@gmail.com, oleg@redhat.com, joel@joelfernandes.org, "Paul E. McKenney" , Christoph Lameter , Pekka Enberg , David Rientjes , Joonsoo Kim , linux-mm@kvack.org Subject: [PATCH sl-b 2/6] mm: Add kmem_last_alloc_errstring() to provide more kmem_last_alloc() info Date: Fri, 4 Dec 2020 16:40:53 -0800 Message-Id: <20201205004057.32199-2-paulmck@kernel.org> X-Mailer: git-send-email 2.9.5 In-Reply-To: <20201205004022.GA31166@paulmck-ThinkPad-P72> References: <20201205004022.GA31166@paulmck-ThinkPad-P72> X-Bogosity: Ham, tests=bogofilter, spamicity=0.000000, version=1.2.4 Sender: owner-linux-mm@kvack.org Precedence: bulk X-Loop: owner-majordomo@kvack.org List-ID: From: "Paul E. McKenney" NULL pointers can be useful, but the NULL pointers from kmem_last_alloc() might be caused by any number of things: A not-to-a-slab pointer, failure to enable all the needed debugging, and bogus slob block-address computations. This commit therefore introduces error codes to the kmem_last_alloc() function using the ERR_PTR() facility, and also introduces kmem_last_alloc_errstring(), which translates the error codes into strings. Cc: Christoph Lameter Cc: Pekka Enberg Cc: David Rientjes Cc: Joonsoo Kim Cc: Andrew Morton Cc: Reported-by: Andrii Nakryiko Signed-off-by: Paul E. McKenney --- include/linux/slab.h | 10 ++++++++++ mm/slab.c | 2 +- mm/slab_common.c | 28 ++++++++++++++++++++++++++-- mm/slob.c | 2 +- mm/slub.c | 4 ++-- 5 files changed, 40 insertions(+), 6 deletions(-) diff --git a/include/linux/slab.h b/include/linux/slab.h index 06dd56b..031e630 100644 --- a/include/linux/slab.h +++ b/include/linux/slab.h @@ -133,6 +133,15 @@ #define ZERO_OR_NULL_PTR(x) ((unsigned long)(x) <= \ (unsigned long)ZERO_SIZE_PTR) +/* + * kmem_last_alloc error codes. + */ +#define KMEM_LA_NO_PAGE 1 /* No page structure for pointer. */ +#define KMEM_LA_NO_SLAB 2 /* Pointer not from slab allocator. */ +#define KMEM_LA_SLOB 3 /* No debugging info for slob. */ +#define KMEM_LA_NO_DEBUG 4 /* Debugging not enabled for slab/slub. */ +#define KMEM_LA_INCONSISTENT 5 /* Bogus block within slub page. */ + #include struct mem_cgroup; @@ -188,6 +197,7 @@ size_t __ksize(const void *); size_t ksize(const void *); void *kmem_cache_last_alloc(struct kmem_cache *s, void *object); void *kmem_last_alloc(void *object); +const char *kmem_last_alloc_errstring(void *lastalloc); #ifdef CONFIG_HAVE_HARDENED_USERCOPY_ALLOCATOR void __check_heap_object(const void *ptr, unsigned long n, struct page *page, diff --git a/mm/slab.c b/mm/slab.c index 2ab93b8..1f3b263 100644 --- a/mm/slab.c +++ b/mm/slab.c @@ -3610,7 +3610,7 @@ void *kmem_cache_last_alloc(struct kmem_cache *cachep, void *object) struct page *page; if (!(cachep->flags & SLAB_STORE_USER)) - return NULL; + return ERR_PTR(-KMEM_LA_NO_DEBUG); objp = object - obj_offset(cachep); page = virt_to_head_page(objp); objnr = obj_to_index(cachep, page, objp); diff --git a/mm/slab_common.c b/mm/slab_common.c index 3f647982..8430a14 100644 --- a/mm/slab_common.c +++ b/mm/slab_common.c @@ -537,6 +537,30 @@ bool slab_is_available(void) } /* + * If the pointer corresponds to a kmem_last_alloc() error, return + * a pointer to the corresponding string, otherwise NULL. + */ +const char *kmem_last_alloc_errstring(void *lastalloc) +{ + long klaerrno; + static const char * const es[] = { + "local memory", /* KMEM_LA_NO_PAGE - 1 */ + "non-slab memory", /* KMEM_LA_NO_SLAB - 1 */ + "slob doesn't do debug", /* KMEM_LA_SLOB - 1 */ + "debugging disabled", /* KMEM_LA_NO_DEBUG - 1 */ + "bogus slub block", /* KMEM_LA_INCONSISTENT - 1 */ + }; + + if (!IS_ERR(lastalloc)) + return NULL; + klaerrno = -PTR_ERR(lastalloc) - 1; + if (WARN_ON_ONCE(klaerrno >= ARRAY_SIZE(es))) + return "kmem_last_alloc error out of range"; + return es[klaerrno]; +} +EXPORT_SYMBOL_GPL(kmem_last_alloc_errstring); + +/* * If the pointer references a slab-allocated object and if sufficient * debugging is enabled, return the returrn address for the corresponding * allocation. Otherwise, return NULL. Note that passing random pointers @@ -548,10 +572,10 @@ void *kmem_last_alloc(void *object) struct page *page; if (!virt_addr_valid(object)) - return NULL; + return ERR_PTR(-KMEM_LA_NO_PAGE); page = virt_to_head_page(object); if (!PageSlab(page)) - return NULL; + return ERR_PTR(-KMEM_LA_NO_SLAB); return kmem_cache_last_alloc(page->slab_cache, object); } EXPORT_SYMBOL_GPL(kmem_last_alloc); diff --git a/mm/slob.c b/mm/slob.c index c1f8ed7..e7d6b90 100644 --- a/mm/slob.c +++ b/mm/slob.c @@ -463,7 +463,7 @@ static void slob_free(void *block, int size) void *kmem_cache_last_alloc(struct kmem_cache *s, void *object) { - return NULL; + return ERR_PTR(-KMEM_LA_SLOB); } /* diff --git a/mm/slub.c b/mm/slub.c index 8ed3ba2..3ddf16a 100644 --- a/mm/slub.c +++ b/mm/slub.c @@ -3928,7 +3928,7 @@ void *kmem_cache_last_alloc(struct kmem_cache *s, void *object) struct track *trackp; if (!(s->flags & SLAB_STORE_USER)) - return NULL; + return ERR_PTR(-KMEM_LA_NO_DEBUG); page = virt_to_head_page(object); base = page_address(page); objp = kasan_reset_tag(object); @@ -3936,7 +3936,7 @@ void *kmem_cache_last_alloc(struct kmem_cache *s, void *object) objnr = obj_to_index(s, page, objp); objp = base + s->size * objnr; if (objp < base || objp >= base + page->objects * s->size || (objp - base) % s->size) - return NULL; + return ERR_PTR(-KMEM_LA_INCONSISTENT); trackp = get_track(s, objp, TRACK_ALLOC); return (void *)trackp->addr; #else From patchwork Sat Dec 5 00:40:54 2020 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: "Paul E. McKenney" X-Patchwork-Id: 11952645 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org X-Spam-Level: X-Spam-Status: No, score=-17.0 required=3.0 tests=BAYES_00,INCLUDES_CR_TRAILER, INCLUDES_PATCH,MAILING_LIST_MULTI,SPF_HELO_NONE,SPF_PASS,URIBL_BLOCKED, USER_AGENT_GIT autolearn=ham autolearn_force=no version=3.4.0 Received: from mail.kernel.org (mail.kernel.org [198.145.29.99]) by smtp.lore.kernel.org (Postfix) with ESMTP id 2CD60C433FE for ; Sat, 5 Dec 2020 00:41:06 +0000 (UTC) Received: from kanga.kvack.org (kanga.kvack.org [205.233.56.17]) by mail.kernel.org (Postfix) with ESMTP id C302E22D75 for ; Sat, 5 Dec 2020 00:41:05 +0000 (UTC) DMARC-Filter: OpenDMARC Filter v1.3.2 mail.kernel.org C302E22D75 Authentication-Results: mail.kernel.org; dmarc=fail (p=none dis=none) header.from=kernel.org Authentication-Results: mail.kernel.org; spf=pass smtp.mailfrom=owner-linux-mm@kvack.org Received: by kanga.kvack.org (Postfix) id AB6456B006C; Fri, 4 Dec 2020 19:41:02 -0500 (EST) Received: by kanga.kvack.org (Postfix, from userid 40) id 871A66B0068; Fri, 4 Dec 2020 19:41:02 -0500 (EST) X-Delivered-To: int-list-linux-mm@kvack.org Received: by kanga.kvack.org (Postfix, from userid 63042) id 61DD66B0068; Fri, 4 Dec 2020 19:41:02 -0500 (EST) X-Delivered-To: linux-mm@kvack.org Received: from forelay.hostedemail.com (smtprelay0094.hostedemail.com [216.40.44.94]) by kanga.kvack.org (Postfix) with ESMTP id 4992D6B006C for ; Fri, 4 Dec 2020 19:41:02 -0500 (EST) Received: from smtpin03.hostedemail.com (10.5.19.251.rfc1918.com [10.5.19.251]) by forelay02.hostedemail.com (Postfix) with ESMTP id 03E2D362A for ; Sat, 5 Dec 2020 00:41:02 +0000 (UTC) X-FDA: 77557373964.03.limit14_530ea1e273c8 Received: from filter.hostedemail.com (10.5.16.251.rfc1918.com [10.5.16.251]) by smtpin03.hostedemail.com (Postfix) with ESMTP id DA3A028A4E9 for ; Sat, 5 Dec 2020 00:41:01 +0000 (UTC) X-HE-Tag: limit14_530ea1e273c8 X-Filterd-Recvd-Size: 3523 Received: from mail.kernel.org (mail.kernel.org [198.145.29.99]) by imf33.hostedemail.com (Postfix) with ESMTP for ; Sat, 5 Dec 2020 00:41:01 +0000 (UTC) From: paulmck@kernel.org Authentication-Results: mail.kernel.org; dkim=permerror (bad message/signature format) To: rcu@vger.kernel.org Cc: linux-kernel@vger.kernel.org, kernel-team@fb.com, mingo@kernel.org, jiangshanlai@gmail.com, akpm@linux-foundation.org, mathieu.desnoyers@efficios.com, josh@joshtriplett.org, tglx@linutronix.de, peterz@infradead.org, rostedt@goodmis.org, dhowells@redhat.com, edumazet@google.com, fweisbec@gmail.com, oleg@redhat.com, joel@joelfernandes.org, "Paul E. McKenney" , Christoph Lameter , Pekka Enberg , David Rientjes , Joonsoo Kim , linux-mm@kvack.org, Andrii Nakryiko Subject: [PATCH sl-b 3/6] rcu: Make call_rcu() print allocation address of double-freed callback Date: Fri, 4 Dec 2020 16:40:54 -0800 Message-Id: <20201205004057.32199-3-paulmck@kernel.org> X-Mailer: git-send-email 2.9.5 In-Reply-To: <20201205004022.GA31166@paulmck-ThinkPad-P72> References: <20201205004022.GA31166@paulmck-ThinkPad-P72> X-Bogosity: Ham, tests=bogofilter, spamicity=0.000000, version=1.2.4 Sender: owner-linux-mm@kvack.org Precedence: bulk X-Loop: owner-majordomo@kvack.org List-ID: From: "Paul E. McKenney" The debug-object double-free checks in __call_rcu() print out the RCU callback function, which is usually sufficient to track down the double free. However, all uses of things like queue_rcu_work() will have the same RCU callback function (rcu_work_rcufn() in this case), so a diagnostic message for a double queue_rcu_work() needs more than just the callback function. This commit therefore prints the last allocation address of the double-freed callback when the callback is slab-allocated and sufficient debugging is enabled. It uses the shiny new kmem_last_alloc() and kmem_last_alloc_errstring() functions for this purpose. Cc: Christoph Lameter Cc: Pekka Enberg Cc: David Rientjes Cc: Joonsoo Kim Cc: Andrew Morton Cc: Cc: Andrii Nakryiko Signed-off-by: Paul E. McKenney --- kernel/rcu/tree.c | 12 ++++++++++-- 1 file changed, 10 insertions(+), 2 deletions(-) diff --git a/kernel/rcu/tree.c b/kernel/rcu/tree.c index b6c9c49..788a072 100644 --- a/kernel/rcu/tree.c +++ b/kernel/rcu/tree.c @@ -2957,6 +2957,8 @@ static void check_cb_ovld(struct rcu_data *rdp) static void __call_rcu(struct rcu_head *head, rcu_callback_t func) { + void *allocaddr; + const char *allocerr; unsigned long flags; struct rcu_data *rdp; bool was_alldone; @@ -2970,8 +2972,14 @@ __call_rcu(struct rcu_head *head, rcu_callback_t func) * Use rcu:rcu_callback trace event to find the previous * time callback was passed to __call_rcu(). */ - WARN_ONCE(1, "__call_rcu(): Double-freed CB %p->%pS()!!!\n", - head, head->func); + allocaddr = kmem_last_alloc(head); + allocerr = kmem_last_alloc_errstring(allocaddr); + if (allocerr) + WARN_ONCE(1, "__call_rcu(): Double-freed CB %p->%pS()!!! (%s)\n", + head, head->func, allocerr); + else + WARN_ONCE(1, "__call_rcu(): Double-freed CB %p->%pS()!!! (Allocated at %pS)\n", + head, head->func, allocaddr); WRITE_ONCE(head->func, rcu_leak_callback); return; } From patchwork Sat Dec 5 00:40:55 2020 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: "Paul E. McKenney" X-Patchwork-Id: 11952647 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org X-Spam-Level: X-Spam-Status: No, score=-17.0 required=3.0 tests=BAYES_00,INCLUDES_CR_TRAILER, INCLUDES_PATCH,MAILING_LIST_MULTI,SPF_HELO_NONE,SPF_PASS,URIBL_BLOCKED, USER_AGENT_GIT autolearn=ham autolearn_force=no version=3.4.0 Received: from mail.kernel.org (mail.kernel.org [198.145.29.99]) by smtp.lore.kernel.org (Postfix) with ESMTP id 2DE07C4361A for ; Sat, 5 Dec 2020 00:41:08 +0000 (UTC) Received: from kanga.kvack.org (kanga.kvack.org [205.233.56.17]) by mail.kernel.org (Postfix) with ESMTP id BC5E922D75 for ; Sat, 5 Dec 2020 00:41:07 +0000 (UTC) DMARC-Filter: OpenDMARC Filter v1.3.2 mail.kernel.org BC5E922D75 Authentication-Results: mail.kernel.org; dmarc=fail (p=none dis=none) header.from=kernel.org Authentication-Results: mail.kernel.org; spf=pass smtp.mailfrom=owner-linux-mm@kvack.org Received: by kanga.kvack.org (Postfix) id E0E816B0068; Fri, 4 Dec 2020 19:41:02 -0500 (EST) Received: by kanga.kvack.org (Postfix, from userid 40) id C11C76B0070; Fri, 4 Dec 2020 19:41:02 -0500 (EST) X-Delivered-To: int-list-linux-mm@kvack.org Received: by kanga.kvack.org (Postfix, from userid 63042) id 9F8FA6B0071; Fri, 4 Dec 2020 19:41:02 -0500 (EST) X-Delivered-To: linux-mm@kvack.org Received: from forelay.hostedemail.com (smtprelay0202.hostedemail.com [216.40.44.202]) by kanga.kvack.org (Postfix) with ESMTP id 7E6836B0070 for ; Fri, 4 Dec 2020 19:41:02 -0500 (EST) Received: from smtpin21.hostedemail.com (10.5.19.251.rfc1918.com [10.5.19.251]) by forelay05.hostedemail.com (Postfix) with ESMTP id 48E11181AEF2A for ; Sat, 5 Dec 2020 00:41:02 +0000 (UTC) X-FDA: 77557373964.21.seed81_1704679273c8 Received: from filter.hostedemail.com (10.5.16.251.rfc1918.com [10.5.16.251]) by smtpin21.hostedemail.com (Postfix) with ESMTP id 27C7C180442C0 for ; Sat, 5 Dec 2020 00:41:02 +0000 (UTC) X-HE-Tag: seed81_1704679273c8 X-Filterd-Recvd-Size: 9192 Received: from mail.kernel.org (mail.kernel.org [198.145.29.99]) by imf18.hostedemail.com (Postfix) with ESMTP for ; Sat, 5 Dec 2020 00:41:01 +0000 (UTC) From: paulmck@kernel.org Authentication-Results: mail.kernel.org; dkim=permerror (bad message/signature format) To: rcu@vger.kernel.org Cc: linux-kernel@vger.kernel.org, kernel-team@fb.com, mingo@kernel.org, jiangshanlai@gmail.com, akpm@linux-foundation.org, mathieu.desnoyers@efficios.com, josh@joshtriplett.org, tglx@linutronix.de, peterz@infradead.org, rostedt@goodmis.org, dhowells@redhat.com, edumazet@google.com, fweisbec@gmail.com, oleg@redhat.com, joel@joelfernandes.org, "Paul E. McKenney" , Christoph Lameter , Pekka Enberg , David Rientjes , Joonsoo Kim , linux-mm@kvack.org Subject: [PATCH sl-b 4/6] mm: Create kmem_last_alloc_stack() to provide stack trace in slub Date: Fri, 4 Dec 2020 16:40:55 -0800 Message-Id: <20201205004057.32199-4-paulmck@kernel.org> X-Mailer: git-send-email 2.9.5 In-Reply-To: <20201205004022.GA31166@paulmck-ThinkPad-P72> References: <20201205004022.GA31166@paulmck-ThinkPad-P72> X-Bogosity: Ham, tests=bogofilter, spamicity=0.000000, version=1.2.4 Sender: owner-linux-mm@kvack.org Precedence: bulk X-Loop: owner-majordomo@kvack.org List-ID: From: "Paul E. McKenney" In some cases, the allocator return address is in a common function, so that more information is desired. For example, a percpu_ref reference-count underflow only has access to a data structure that is allocated in percpu_ref_init(). In this case, the return address from the allocator provides no additional information. This commit therefore creates a kmem_cache_last_alloc() function that can be passed stackp and nstackp parameters, allowing CONFIG_STACKTRACE=y slub stack traces to be provided to the caller. Please note that stack traces cannot be provided unless they are collected. Collecting stack traces requires that the kernel: (1) Use the slub allocator, (2) Be built with CONFIG_STACKTRACE=y (which is the case when ftrace is configured), and (3) Have slub debugging enabled one way or another, for example, by booting with the "slub_debug=U" kernel boot parameter. Cc: Christoph Lameter Cc: Pekka Enberg Cc: David Rientjes Cc: Joonsoo Kim Cc: Andrew Morton Cc: Reported-by: Andrii Nakryiko [ paulmck: Move slab definition per Stephen Rothwell and kbuild test robot. ] Signed-off-by: Paul E. McKenney --- include/linux/slab.h | 3 ++- mm/slab.c | 40 +++++++++++++++++++++------------------- mm/slab_common.c | 39 ++++++++++++++++++++++++++++++++------- mm/slob.c | 4 +++- mm/slub.c | 14 +++++++++++++- 5 files changed, 71 insertions(+), 29 deletions(-) diff --git a/include/linux/slab.h b/include/linux/slab.h index 031e630..bdedefd 100644 --- a/include/linux/slab.h +++ b/include/linux/slab.h @@ -195,8 +195,9 @@ void kfree(const void *); void kfree_sensitive(const void *); size_t __ksize(const void *); size_t ksize(const void *); -void *kmem_cache_last_alloc(struct kmem_cache *s, void *object); +void *kmem_cache_last_alloc(struct kmem_cache *s, void *object, void **stackp, int nstackp); void *kmem_last_alloc(void *object); +void *kmem_last_alloc_stack(void *object, void **stackp, int nstackp); const char *kmem_last_alloc_errstring(void *lastalloc); #ifdef CONFIG_HAVE_HARDENED_USERCOPY_ALLOCATOR diff --git a/mm/slab.c b/mm/slab.c index 1f3b263..ae1a74c 100644 --- a/mm/slab.c +++ b/mm/slab.c @@ -3602,25 +3602,6 @@ void *kmem_cache_alloc_node_trace(struct kmem_cache *cachep, EXPORT_SYMBOL(kmem_cache_alloc_node_trace); #endif -void *kmem_cache_last_alloc(struct kmem_cache *cachep, void *object) -{ -#ifdef DEBUG - unsigned int objnr; - void *objp; - struct page *page; - - if (!(cachep->flags & SLAB_STORE_USER)) - return ERR_PTR(-KMEM_LA_NO_DEBUG); - objp = object - obj_offset(cachep); - page = virt_to_head_page(objp); - objnr = obj_to_index(cachep, page, objp); - objp = index_to_obj(cachep, page, objnr); - return *dbg_userword(cachep, objp); -#else - return NULL; -#endif -} - static __always_inline void * __do_kmalloc_node(size_t size, gfp_t flags, int node, unsigned long caller) { @@ -3652,6 +3633,27 @@ void *__kmalloc_node_track_caller(size_t size, gfp_t flags, EXPORT_SYMBOL(__kmalloc_node_track_caller); #endif /* CONFIG_NUMA */ +void *kmem_cache_last_alloc(struct kmem_cache *cachep, void *object, void **stackp, int nstackp) +{ +#ifdef DEBUG + unsigned int objnr; + void *objp; + struct page *page; + + if (!(cachep->flags & SLAB_STORE_USER)) + return ERR_PTR(-KMEM_LA_NO_DEBUG); + objp = object - obj_offset(cachep); + page = virt_to_head_page(objp); + objnr = obj_to_index(cachep, page, objp); + objp = index_to_obj(cachep, page, objnr); + if (stackp && nstackp) + stackp[0] = NULL; + return *dbg_userword(cachep, objp); +#else + return NULL; +#endif +} + /** * __do_kmalloc - allocate memory * @size: how many bytes of memory are required. diff --git a/mm/slab_common.c b/mm/slab_common.c index 8430a14..b70f357 100644 --- a/mm/slab_common.c +++ b/mm/slab_common.c @@ -560,14 +560,22 @@ const char *kmem_last_alloc_errstring(void *lastalloc) } EXPORT_SYMBOL_GPL(kmem_last_alloc_errstring); -/* +/** + * kmem_last_alloc_stack - Get return address and stack for last allocation + * @object: object for which to find last-allocation return address. + * @stackp: %NULL or pointer to location to place return-address stack. + * @nstackp: maximum number of return addresses that may be stored. + * * If the pointer references a slab-allocated object and if sufficient - * debugging is enabled, return the returrn address for the corresponding - * allocation. Otherwise, return NULL. Note that passing random pointers - * to this function (including addresses of on-stack variables) is likely - * to result in panics. + * debugging is enabled, return the return address for the corresponding + * allocation. If stackp is non-%NULL in %CONFIG_STACKTRACE kernels running + * the slub allocator, also copy the return-address stack into @stackp, + * limited by @nstackp. Otherwise, return %NULL or an appropriate error + * code using %ERR_PTR(). + * + * Return: return address from last allocation, %NULL or negative error code. */ -void *kmem_last_alloc(void *object) +void *kmem_last_alloc_stack(void *object, void **stackp, int nstackp) { struct page *page; @@ -576,7 +584,24 @@ void *kmem_last_alloc(void *object) page = virt_to_head_page(object); if (!PageSlab(page)) return ERR_PTR(-KMEM_LA_NO_SLAB); - return kmem_cache_last_alloc(page->slab_cache, object); + return kmem_cache_last_alloc(page->slab_cache, object, stackp, nstackp); +} +EXPORT_SYMBOL_GPL(kmem_last_alloc_stack); + +/** + * kmem_last_alloc - Get return address for last allocation + * @object: object for which to find last-allocation return address. + * + * If the pointer references a slab-allocated object and if sufficient + * debugging is enabled, return the return address for the corresponding + * allocation. Otherwise, return %NULL or an appropriate error code using + * %ERR_PTR(). + * + * Return: return address from last allocation, %NULL or negative error code. + */ +void *kmem_last_alloc(void *object) +{ + return kmem_last_alloc_stack(object, NULL, 0); } EXPORT_SYMBOL_GPL(kmem_last_alloc); diff --git a/mm/slob.c b/mm/slob.c index e7d6b90..dab7f3b 100644 --- a/mm/slob.c +++ b/mm/slob.c @@ -461,8 +461,10 @@ static void slob_free(void *block, int size) spin_unlock_irqrestore(&slob_lock, flags); } -void *kmem_cache_last_alloc(struct kmem_cache *s, void *object) +void *kmem_cache_last_alloc(struct kmem_cache *s, void *object, void **stackp, int nstackp) { + if (stackp && nstackp) + stackp[0] = NULL; return ERR_PTR(-KMEM_LA_SLOB); } diff --git a/mm/slub.c b/mm/slub.c index 3ddf16a..a918b1d 100644 --- a/mm/slub.c +++ b/mm/slub.c @@ -3918,10 +3918,11 @@ int __kmem_cache_shutdown(struct kmem_cache *s) return 0; } -void *kmem_cache_last_alloc(struct kmem_cache *s, void *object) +void *kmem_cache_last_alloc(struct kmem_cache *s, void *object, void **stackp, int nstackp) { #ifdef CONFIG_SLUB_DEBUG void *base; + int i = 0; unsigned int objnr; void *objp; struct page *page; @@ -3938,6 +3939,17 @@ void *kmem_cache_last_alloc(struct kmem_cache *s, void *object) if (objp < base || objp >= base + page->objects * s->size || (objp - base) % s->size) return ERR_PTR(-KMEM_LA_INCONSISTENT); trackp = get_track(s, objp, TRACK_ALLOC); +#ifdef CONFIG_STACKTRACE + if (stackp) { + for (; i < nstackp && i < TRACK_ADDRS_COUNT; i++) { + stackp[i] = (void *)trackp->addrs[i]; + if (!stackp[i]) + break; + } + } +#endif + if (stackp && i < nstackp) + stackp[i] = NULL; return (void *)trackp->addr; #else return NULL;