From patchwork Fri Jan 28 13:09:53 2022 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Michel Lespinasse X-Patchwork-Id: 12728522 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from kanga.kvack.org (kanga.kvack.org [205.233.56.17]) by smtp.lore.kernel.org (Postfix) with ESMTP id 716D9C433F5 for ; Fri, 28 Jan 2022 13:10:53 +0000 (UTC) Received: by kanga.kvack.org (Postfix) id CB2056B009D; Fri, 28 Jan 2022 08:10:14 -0500 (EST) Received: by kanga.kvack.org (Postfix, from userid 40) id 92B4F6B0082; Fri, 28 Jan 2022 08:10:14 -0500 (EST) X-Delivered-To: int-list-linux-mm@kvack.org Received: by kanga.kvack.org (Postfix, from userid 63042) id 026F16B008A; Fri, 28 Jan 2022 08:10:13 -0500 (EST) X-Delivered-To: linux-mm@kvack.org Received: from forelay.hostedemail.com (smtprelay0239.hostedemail.com [216.40.44.239]) by kanga.kvack.org (Postfix) with ESMTP id 1061F6B008A for ; Fri, 28 Jan 2022 08:10:13 -0500 (EST) Received: from smtpin05.hostedemail.com (10.5.19.251.rfc1918.com [10.5.19.251]) by forelay02.hostedemail.com (Postfix) with ESMTP id C2C3792E03 for ; Fri, 28 Jan 2022 13:10:12 +0000 (UTC) X-FDA: 79079729064.05.37B6A91 Received: from server.lespinasse.org (server.lespinasse.org [63.205.204.226]) by imf07.hostedemail.com (Postfix) with ESMTP id D479D4000F for ; Fri, 28 Jan 2022 13:10:11 +0000 (UTC) DKIM-Signature: v=1; a=ed25519-sha256; c=relaxed/relaxed; d=lespinasse.org; i=@lespinasse.org; q=dns/txt; s=srv-52-ed; t=1643375407; h=from : to : cc : subject : date : message-id : in-reply-to : references : mime-version : content-transfer-encoding : from; bh=OOok3oAXLMDgG2pHbvztom5rL8XJmkTzwxFo3hW65dY=; b=apOsMK7sYOiHe+UmWef7PYiCm3i1aOipU8SikDOdqQIVQ92th1dViE6N0eom1nHudhL5n 4/szWqoaviTOG4LCg== DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=lespinasse.org; i=@lespinasse.org; q=dns/txt; s=srv-52-rsa; t=1643375407; h=from : to : cc : subject : date : message-id : in-reply-to : references : mime-version : content-transfer-encoding : from; bh=OOok3oAXLMDgG2pHbvztom5rL8XJmkTzwxFo3hW65dY=; b=7jHZVpmUHaV2hihoS/xDYZ7lIHZFiNosHjwrjMvHdF49E3nYRWUd6B0f1WZbzTOTJ3DbR KdoA+Tk76uCvQdhq/sopUjEvWhuCJjuvJQc8/G1yZ/OyLY6M4cq0fNS3bDS0MU2I06fz25j YiRmitgizvCq5ZyOFmdCiJVrVCui+9lSU2DRXUyV0dSwntrbLtIIY9VqGev1BV8uIRUsKsJ PXqDwUkAU6QwETRspdM3sJvxgXFDgnHHdQOdh4r2QQb2pZGwRMwbdT0yCFjPQ6WywdVwJoL kwdzpN8FW1tj5a9YAAtuAb61eSHPIKy+MDAGv2lLbceDf265pbntCbCh2Hrw== Received: from zeus.lespinasse.org (zeus.lespinasse.org [10.0.0.150]) by server.lespinasse.org (Postfix) with ESMTPS id 25423160AAA; Fri, 28 Jan 2022 05:10:07 -0800 (PST) Received: by zeus.lespinasse.org (Postfix, from userid 1000) id 093D420472; Fri, 28 Jan 2022 05:10:07 -0800 (PST) From: Michel Lespinasse To: Linux-MM , linux-kernel@vger.kernel.org, Andrew Morton Cc: kernel-team@fb.com, Laurent Dufour , Jerome Glisse , Peter Zijlstra , Michal Hocko , Vlastimil Babka , Davidlohr Bueso , Matthew Wilcox , Liam Howlett , Rik van Riel , Paul McKenney , Song Liu , Suren Baghdasaryan , Minchan Kim , Joel Fernandes , David Rientjes , Axel Rasmussen , Andy Lutomirski , Michel Lespinasse Subject: [PATCH v2 22/35] percpu-rwsem: enable percpu_sem destruction in atomic context Date: Fri, 28 Jan 2022 05:09:53 -0800 Message-Id: <20220128131006.67712-23-michel@lespinasse.org> X-Mailer: git-send-email 2.20.1 In-Reply-To: <20220128131006.67712-1-michel@lespinasse.org> References: <20220128131006.67712-1-michel@lespinasse.org> MIME-Version: 1.0 X-Rspam-User: nil X-Rspamd-Server: rspam10 X-Rspamd-Queue-Id: D479D4000F X-Stat-Signature: y6mfpfaymsug6tocms8tfohw9h797764 Authentication-Results: imf07.hostedemail.com; dkim=pass header.d=lespinasse.org header.s=srv-52-ed header.b=apOsMK7s; dkim=pass header.d=lespinasse.org header.s=srv-52-rsa header.b=7jHZVpmU; spf=pass (imf07.hostedemail.com: domain of walken@lespinasse.org designates 63.205.204.226 as permitted sender) smtp.mailfrom=walken@lespinasse.org; dmarc=pass (policy=none) header.from=lespinasse.org X-HE-Tag: 1643375411-581502 X-Bogosity: Ham, tests=bogofilter, spamicity=0.000000, version=1.2.4 Sender: owner-linux-mm@kvack.org Precedence: bulk X-Loop: owner-majordomo@kvack.org List-ID: From: Suren Baghdasaryan Calling percpu_free_rwsem in atomic context results in "scheduling while atomic" bug being triggered: BUG: scheduling while atomic: klogd/158/0x00000002 ... __schedule_bug+0x191/0x290 schedule_debug+0x97/0x180 __schedule+0xdc/0xba0 schedule+0xda/0x250 schedule_timeout+0x92/0x2d0 __wait_for_common+0x25b/0x430 wait_for_completion+0x1f/0x30 rcu_barrier+0x440/0x4f0 rcu_sync_dtor+0xaa/0x190 percpu_free_rwsem+0x41/0x80 Introduce percpu_rwsem_destroy function to perform semaphore destruction in a worker thread. Signed-off-by: Suren Baghdasaryan Signed-off-by: Michel Lespinasse --- include/linux/percpu-rwsem.h | 13 ++++++++++++- kernel/locking/percpu-rwsem.c | 32 ++++++++++++++++++++++++++++++++ 2 files changed, 44 insertions(+), 1 deletion(-) diff --git a/include/linux/percpu-rwsem.h b/include/linux/percpu-rwsem.h index 5fda40f97fe9..bf1668fc9c5e 100644 --- a/include/linux/percpu-rwsem.h +++ b/include/linux/percpu-rwsem.h @@ -13,7 +13,14 @@ struct percpu_rw_semaphore { struct rcu_sync rss; unsigned int __percpu *read_count; struct rcuwait writer; - wait_queue_head_t waiters; + /* + * destroy_list_entry is used during object destruction when waiters + * can't be used, therefore reusing the same space. + */ + union { + wait_queue_head_t waiters; + struct list_head destroy_list_entry; + }; atomic_t block; #ifdef CONFIG_DEBUG_LOCK_ALLOC struct lockdep_map dep_map; @@ -127,8 +134,12 @@ extern void percpu_up_write(struct percpu_rw_semaphore *); extern int __percpu_init_rwsem(struct percpu_rw_semaphore *, const char *, struct lock_class_key *); +/* Can't be called in atomic context. */ extern void percpu_free_rwsem(struct percpu_rw_semaphore *); +/* Invokes percpu_free_rwsem and frees the semaphore from a worker thread. */ +extern void percpu_rwsem_async_destroy(struct percpu_rw_semaphore *sem); + #define percpu_init_rwsem(sem) \ ({ \ static struct lock_class_key rwsem_key; \ diff --git a/kernel/locking/percpu-rwsem.c b/kernel/locking/percpu-rwsem.c index 70a32a576f3f..a3d37bf83c60 100644 --- a/kernel/locking/percpu-rwsem.c +++ b/kernel/locking/percpu-rwsem.c @@ -7,6 +7,7 @@ #include #include #include +#include #include int __percpu_init_rwsem(struct percpu_rw_semaphore *sem, @@ -268,3 +269,34 @@ void percpu_up_write(struct percpu_rw_semaphore *sem) rcu_sync_exit(&sem->rss); } EXPORT_SYMBOL_GPL(percpu_up_write); + +static LIST_HEAD(destroy_list); +static DEFINE_SPINLOCK(destroy_list_lock); + +static void destroy_list_workfn(struct work_struct *work) +{ + struct percpu_rw_semaphore *sem, *sem2; + LIST_HEAD(to_destroy); + + spin_lock(&destroy_list_lock); + list_splice_init(&destroy_list, &to_destroy); + spin_unlock(&destroy_list_lock); + + if (list_empty(&to_destroy)) + return; + + list_for_each_entry_safe(sem, sem2, &to_destroy, destroy_list_entry) { + percpu_free_rwsem(sem); + kfree(sem); + } +} + +static DECLARE_WORK(destroy_list_work, destroy_list_workfn); + +void percpu_rwsem_async_destroy(struct percpu_rw_semaphore *sem) +{ + spin_lock(&destroy_list_lock); + list_add_tail(&sem->destroy_list_entry, &destroy_list); + spin_unlock(&destroy_list_lock); + schedule_work(&destroy_list_work); +}