From patchwork Fri Feb 3 07:18:33 2023 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Nicholas Piggin X-Patchwork-Id: 13127089 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from kanga.kvack.org (kanga.kvack.org [205.233.56.17]) by smtp.lore.kernel.org (Postfix) with ESMTP id 26E22C636CC for ; Fri, 3 Feb 2023 07:19:04 +0000 (UTC) Received: by kanga.kvack.org (Postfix) id 540E56B0073; Fri, 3 Feb 2023 02:19:03 -0500 (EST) Received: by kanga.kvack.org (Postfix, from userid 40) id 4F0DC6B0074; Fri, 3 Feb 2023 02:19:03 -0500 (EST) X-Delivered-To: int-list-linux-mm@kvack.org Received: by kanga.kvack.org (Postfix, from userid 63042) id 392416B0075; Fri, 3 Feb 2023 02:19:03 -0500 (EST) X-Delivered-To: linux-mm@kvack.org Received: from relay.hostedemail.com (smtprelay0010.hostedemail.com [216.40.44.10]) by kanga.kvack.org (Postfix) with ESMTP id 2B69B6B0073 for ; Fri, 3 Feb 2023 02:19:03 -0500 (EST) Received: from smtpin23.hostedemail.com (a10.router.float.18 [10.200.18.1]) by unirelay10.hostedemail.com (Postfix) with ESMTP id CFE3CC02D0 for ; Fri, 3 Feb 2023 07:19:02 +0000 (UTC) X-FDA: 80425128924.23.DC2C3AC Received: from mail-pl1-f177.google.com (mail-pl1-f177.google.com [209.85.214.177]) by imf21.hostedemail.com (Postfix) with ESMTP id 132A51C000A for ; Fri, 3 Feb 2023 07:19:00 +0000 (UTC) Authentication-Results: imf21.hostedemail.com; dkim=pass header.d=gmail.com header.s=20210112 header.b=AHiRLfcf; spf=pass (imf21.hostedemail.com: domain of npiggin@gmail.com designates 209.85.214.177 as permitted sender) smtp.mailfrom=npiggin@gmail.com; dmarc=pass (policy=none) header.from=gmail.com ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=hostedemail.com; s=arc-20220608; t=1675408741; h=from:from:sender:reply-to:subject:subject:date:date: message-id:message-id:to:to:cc:cc:mime-version:mime-version: content-type:content-transfer-encoding:content-transfer-encoding: in-reply-to:in-reply-to:references:references:dkim-signature; bh=XnlBvkqnb+l7IeUx8vg7gSpE5ieSkYXmCCbewvNAOcI=; b=WVwTP2m7a8A+NAI0Vx+qpOxBAhSejpdDFpxymtCuRk4f+Koli+0dv5wYvjozEIain2SCiT gMQYZDsYG0q1QyUeff7wu+haTFAx/+d/SMyJQJN0dI5Yb6K9lFyJRhbvMPCUUra3Kk0+5y vOijuCcdh/CU6CSIRh3j5SZmWAcePs4= ARC-Authentication-Results: i=1; imf21.hostedemail.com; dkim=pass header.d=gmail.com header.s=20210112 header.b=AHiRLfcf; spf=pass (imf21.hostedemail.com: domain of npiggin@gmail.com designates 209.85.214.177 as permitted sender) smtp.mailfrom=npiggin@gmail.com; dmarc=pass (policy=none) header.from=gmail.com ARC-Seal: i=1; s=arc-20220608; d=hostedemail.com; t=1675408741; a=rsa-sha256; cv=none; b=pU5Gl2iw5k0SfJB80kCZjpUeXQFROACCNh0LMVVb8sx5RcP4yjDAjy9lxOXM0fjdtAjbZl VdH4O8XxzDODiwsUNF5c/olWwiprXQbbIEFd4Y4AOuxCu2JG+JAVJe9h57GQzwm+AcHMtg EVvBEpTIRLvG4U+tUHQsOQRs5JaIh7M= Received: by mail-pl1-f177.google.com with SMTP id d3so4393258plr.10 for ; Thu, 02 Feb 2023 23:19:00 -0800 (PST) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=20210112; h=content-transfer-encoding:mime-version:references:in-reply-to :message-id:date:subject:cc:to:from:from:to:cc:subject:date :message-id:reply-to; bh=XnlBvkqnb+l7IeUx8vg7gSpE5ieSkYXmCCbewvNAOcI=; b=AHiRLfcf8eUNQBnlAsruaF06DgmSEYeMrWMiRHEZ9X5oCS8EFKY0D1+uSzk001rKHG wM6Q2U10Pn14JQyW+l0O/c8z+fkwxIm/NPitBdfs40sfGlyXsENjatpI1RZhPk232lvA zDdUzhnba9x94swlHDb7UkVrJFnMqWpYzmGo4TPTsGtXHfXv9sfrWlf3syDxwLR5ipEX dFRmiJAgq8XvaTNIFp5Qu+WClm6+oFC9LekqRQUzG9QfdZgJPNJtJkIdPe1uHKJYpQRj YyJEeIrYiO6SEa4xtr2WXCweUe8cAWZ/KQ4ykObFGtMcv3jpBg0TZs/m6RR8iSh/nfe2 i5Jg== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20210112; h=content-transfer-encoding:mime-version:references:in-reply-to :message-id:date:subject:cc:to:from:x-gm-message-state:from:to:cc :subject:date:message-id:reply-to; bh=XnlBvkqnb+l7IeUx8vg7gSpE5ieSkYXmCCbewvNAOcI=; b=XTE7KXIRr67TKBGxetO82CmZJVMU4xxzDORE7hMigaIbysNZnjpJ5QJXs7lHlMlbbR L23FTrFlsS+sqL8UtwHpDSfrAZUsmfKBuBmAOVyoP6KynGjata4qYHyxWHHy/DNL+rgk ey9wWvuaB5asqLExmCDPf9kStzK6MX8HKogFnSu++M9XiyBCQr3vLPUCSGJMf5/kqxX5 /CCsiO3bVl1B2QkLwjVf8/Ttu29jL2cafFPK92F6BQeY+hE5B8yTVInJvlsPNm23j3IE Mcz1GzF8fIqNhr/PiOrsF8m2Q7CqUsolF8U6vRdMTj65scYxkt2lc1UjoabD6dARPIc5 x8gw== X-Gm-Message-State: AO0yUKVIEyNvrBJe1G/5G8G0vQsBED/NjsYNEUtjFyXnsw2aAeI55lVJ BFG5ecfObq2pgD+yo4jU7e8= X-Google-Smtp-Source: AK7set9/19Ishv+YbUs//IpJgvi84ti72zlNw3k1E7pi36wh6xfDzDjNIuz8qP6WMzuLeNCLPjc4Ag== X-Received: by 2002:a05:6a20:4413:b0:be:cd2f:1951 with SMTP id ce19-20020a056a20441300b000becd2f1951mr12089686pzb.41.1675408740099; Thu, 02 Feb 2023 23:19:00 -0800 (PST) Received: from bobo.ibm.com (193-116-117-77.tpgi.com.au. [193.116.117.77]) by smtp.gmail.com with ESMTPSA id f20-20020a637554000000b004df4ba1ebfesm877558pgn.66.2023.02.02.23.18.54 (version=TLS1_3 cipher=TLS_AES_256_GCM_SHA384 bits=256/256); Thu, 02 Feb 2023 23:18:59 -0800 (PST) From: Nicholas Piggin To: Andrew Morton Cc: Nicholas Piggin , Linus Torvalds , Nadav Amit , Dave Hansen , Andy Lutomirski , Peter Zijlstra , Catalin Marinas , Will Deacon , Rik van Riel , linux-arch@vger.kernel.org, linux-mm@kvack.org, linuxppc-dev@lists.ozlabs.org Subject: [PATCH v7 1/5] kthread: simplify kthread_use_mm refcounting Date: Fri, 3 Feb 2023 17:18:33 +1000 Message-Id: <20230203071837.1136453-2-npiggin@gmail.com> X-Mailer: git-send-email 2.37.2 In-Reply-To: <20230203071837.1136453-1-npiggin@gmail.com> References: <20230203071837.1136453-1-npiggin@gmail.com> MIME-Version: 1.0 X-Rspam-User: X-Rspamd-Server: rspam03 X-Stat-Signature: ti8kno6ywput44a9prazmyki1f55e4e7 X-Rspamd-Queue-Id: 132A51C000A X-HE-Tag: 1675408740-446276 X-HE-Meta: U2FsdGVkX19Ij2NNjTAd10HtTSn/w5hsuhIXfjUD4xZZWk5CIQ8+NujxBFNLBaeeUs1+nOI9ToimRUmdVl9JP4lf6959ukHJj+sr2HN0NdP8j5boJG52kLZ39fUddnWpXryyD56unrQsGYGvS44bjRB1SpTLXHhFvKlj+tO4mWZ76/X166wwlr8LFS3Itg0srAFLTA7R35ng/PedGi4W1Ua+BUt0po52M15bfJd5fSihveNEIAqY/vVltia/cL+VF0g+ghU89qWSwGe2by5SEqTwtGwBgzcyrlhuniwrYYbYq0TTL9BE2E9PvOlFDYY5qLRvt6X0qRYtKTgZwl94vBrQ/wzzFqSEBg+dKJ7THEGaraXQpcgHNOnt1YoWqhPOX9CMFzYMCLQ15riix27AFN6+RO4evB4ejqHUxV8E/p9bmE8NyMxvt+bB4/9pxKo5HfJ6pkpLPfGv5baytJcwYCMjSWysbrm7I4g6VnUu0MglZdHSyiYzK7sJDSYelPWyrbuwKhTeOY7cX7oBJ3adD0q+EuMZhSHE5yijv3aQmlBTksY9Yc0aqgJjit2bWOQenCbb9NakLmvgsNGeD6A6YAAZ3m9MBGlmYqYMaH03Wj77CcLKnOF7OHIYcVNMiSAebqFP4W1LXq0kgan+BXzBfaqLRcyvyIvzUyUC3kTB//DfxAnyE/gOQ3GDKabfc07foThZROCo2E0X6BUyLHJmQQgDKs4gWAwhpzfMGxNwfNpUfQ54q+2TfkOgnySYPvi0ZCDj2QFgLoAGZUdmu/5oLm1RnxqFofFan9bHRZPEpSZeU5qEzjIH33gn6fKCPvXGgWP+O8u7lEf7RfGshxJRYhOkaWfqF+EaOAfcTyPALGYiCvMQOK08qFdNCAyLKMTXKo5wx0wbVXfnfbjAyjl+SmGyC2oOTVSWziU9xjgNIDFUC6M9W4PTdbIRqKE4+JucfhKgzAcg+7af3oe8y1O 6VJfyh4F vtYTSdRUVXELWE2YldT9Y74SXI2wvEGhhulPJvWalfGhGlFFMw66Snucu/VqMzC0MbTXXltQLez6POShYWbk3Q/lQhfqMLPIFeDa96xJju8QXHh0jYDYqmvSeY/+fYkG3KI8lPX+/7hPipSGIYcpGnWMiEWpBK8qB67xAtMWKg1cT6WogcGm9KtKr4JIlr4q4JcW+JKZOrqi8td3erod4ugWqcOp/P+SD5vGJsHRo5nwrA7U8P3fgTSHJmTWEt4+Qunr9kHZb33r+Fvb0Ip4hB9qP9rfrLRWG0nnuRKh2TphvSTU704+G/E6l9iR2s9lsRZjyNoh0RKaBAyibENtVUNwrUMR5JumhM9PEkCdb7IYNKwh//7HIst5igWWlqHxJ91kJbUf/snGzI51g7jy7OURGJAvQ8tmSB/Hf/Np6wuAVkg45bICEemS+elMvjGskuaMZqW+V86T7wAZcP2omWUHMpqiwllrX0tk+WXLmFpTJvreeK2YiyyYbI7nFnumifwN5snvtGW/LgeGiaIhn27Kx9CKzBWrHX05OmHC0rUfJ9hAGNm9E27x4pwkfmdPg88rq5QAFs3KzZ87jzdLdjnh/iYRjCvPQbBdakaq1YvmxY65rltOTJ1Vr0VmR+4f9o8sUZmefS2zkcrs= X-Bogosity: Ham, tests=bogofilter, spamicity=0.000000, version=1.2.4 Sender: owner-linux-mm@kvack.org Precedence: bulk X-Loop: owner-majordomo@kvack.org List-ID: Remove the special case avoiding refcounting when the mm to be used is the same as the kernel thread's active (lazy tlb) mm. kthread_use_mm() should not be such a performance critical path that this matters much. This simplifies a later change to lazy tlb mm refcounting. Acked-by: Linus Torvalds Signed-off-by: Nicholas Piggin --- kernel/kthread.c | 14 +++++--------- 1 file changed, 5 insertions(+), 9 deletions(-) diff --git a/kernel/kthread.c b/kernel/kthread.c index f97fd01a2932..7424a1839e9a 100644 --- a/kernel/kthread.c +++ b/kernel/kthread.c @@ -1410,14 +1410,13 @@ void kthread_use_mm(struct mm_struct *mm) WARN_ON_ONCE(!(tsk->flags & PF_KTHREAD)); WARN_ON_ONCE(tsk->mm); + mmgrab(mm); + task_lock(tsk); /* Hold off tlb flush IPIs while switching mm's */ local_irq_disable(); active_mm = tsk->active_mm; - if (active_mm != mm) { - mmgrab(mm); - tsk->active_mm = mm; - } + tsk->active_mm = mm; tsk->mm = mm; membarrier_update_current_mm(mm); switch_mm_irqs_off(active_mm, mm, tsk); @@ -1434,12 +1433,9 @@ void kthread_use_mm(struct mm_struct *mm) * memory barrier after storing to tsk->mm, before accessing * user-space memory. A full memory barrier for membarrier * {PRIVATE,GLOBAL}_EXPEDITED is implicitly provided by - * mmdrop(), or explicitly with smp_mb(). + * mmdrop(). */ - if (active_mm != mm) - mmdrop(active_mm); - else - smp_mb(); + mmdrop(active_mm); } EXPORT_SYMBOL_GPL(kthread_use_mm); From patchwork Fri Feb 3 07:18:34 2023 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Nicholas Piggin X-Patchwork-Id: 13127090 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from kanga.kvack.org (kanga.kvack.org [205.233.56.17]) by smtp.lore.kernel.org (Postfix) with ESMTP id 508CAC636CC for ; Fri, 3 Feb 2023 07:19:09 +0000 (UTC) Received: by kanga.kvack.org (Postfix) id C6EB96B0074; Fri, 3 Feb 2023 02:19:08 -0500 (EST) Received: by kanga.kvack.org (Postfix, from userid 40) id C1D676B0075; Fri, 3 Feb 2023 02:19:08 -0500 (EST) X-Delivered-To: int-list-linux-mm@kvack.org Received: by kanga.kvack.org (Postfix, from userid 63042) id AE4E76B0078; Fri, 3 Feb 2023 02:19:08 -0500 (EST) X-Delivered-To: linux-mm@kvack.org Received: from relay.hostedemail.com (smtprelay0015.hostedemail.com [216.40.44.15]) by kanga.kvack.org (Postfix) with ESMTP id 9FF5D6B0074 for ; Fri, 3 Feb 2023 02:19:08 -0500 (EST) Received: from smtpin19.hostedemail.com (a10.router.float.18 [10.200.18.1]) by unirelay04.hostedemail.com (Postfix) with ESMTP id 7FC621A0372 for ; Fri, 3 Feb 2023 07:19:08 +0000 (UTC) X-FDA: 80425129176.19.926BFE1 Received: from mail-pj1-f51.google.com (mail-pj1-f51.google.com [209.85.216.51]) by imf30.hostedemail.com (Postfix) with ESMTP id A64D580008 for ; Fri, 3 Feb 2023 07:19:06 +0000 (UTC) Authentication-Results: imf30.hostedemail.com; dkim=pass header.d=gmail.com header.s=20210112 header.b=RjMq26cA; spf=pass (imf30.hostedemail.com: domain of npiggin@gmail.com designates 209.85.216.51 as permitted sender) smtp.mailfrom=npiggin@gmail.com; dmarc=pass (policy=none) header.from=gmail.com ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=hostedemail.com; s=arc-20220608; t=1675408746; h=from:from:sender:reply-to:subject:subject:date:date: message-id:message-id:to:to:cc:cc:mime-version:mime-version: content-type:content-transfer-encoding:content-transfer-encoding: in-reply-to:in-reply-to:references:references:dkim-signature; bh=5zyO0Na48QxjwG/96vzDf++z0O8XkPhrocrCmdAQIr4=; b=K6L6VRcDcu1l7zDkrhPmQdQ/Xm1tclL/C8Ms788IQkjk8kH48Bxlx2Om3hWdmqFrkncb0A fFlOZsyDprdN1GESzGX7klZSRSD+Da9BMUl0Ll0affXbs7OWFpBpEhPh+HJZWqBh5dCWmF 0koVl6vu3Qtog8CLmVHfL8YvGebSVUs= ARC-Authentication-Results: i=1; imf30.hostedemail.com; dkim=pass header.d=gmail.com header.s=20210112 header.b=RjMq26cA; spf=pass (imf30.hostedemail.com: domain of npiggin@gmail.com designates 209.85.216.51 as permitted sender) smtp.mailfrom=npiggin@gmail.com; dmarc=pass (policy=none) header.from=gmail.com ARC-Seal: i=1; s=arc-20220608; d=hostedemail.com; t=1675408746; a=rsa-sha256; cv=none; b=ngsBbwZ8G0Thl2Ty1c7gG7ZDhxu2hoUxW8L3mllP/0LWiWEJzMCHs1+h/PUOYy3or3+bco rkxejb7rWrz/0OQaiFP3k2SmuS4v3INYXhbq+7WeBhm9mIQGhLnFUbO20EXDWI648W3LAe Bi2/v0Bj0Ij6DCWJ6c1G7UVZvNrBDyA= Received: by mail-pj1-f51.google.com with SMTP id bx22so1283502pjb.3 for ; Thu, 02 Feb 2023 23:19:06 -0800 (PST) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=20210112; h=content-transfer-encoding:mime-version:references:in-reply-to :message-id:date:subject:cc:to:from:from:to:cc:subject:date :message-id:reply-to; bh=5zyO0Na48QxjwG/96vzDf++z0O8XkPhrocrCmdAQIr4=; b=RjMq26cALu6UQb/DQZXQLPnJ8q+5dxPkkAeNEEvpF3YmP/iIxIovFGviX1JlRz0jeg M7qLKRmW/kcc5x1c9SHmOSYnJe9+31+D24kVK96Ksr1S7Zt/aXoepztE4675W5+TKCmI l7ZXGa+Hkrlr29qD2BElEYcG56srSLiqGLTXdDhjWEbhTeezj+o3ucEl/DnGScKqEdos gjnHqOBFBvOGIB8B0ihoLS1aLm2EEzjTn2NTiSQLlM02bDjGVFKn0ywDHx4b4Xqh8fdJ /YT8fJBKv4a5hmE07ZK5OFW3fBPnCb8a4hDphqWN5QDzyLQrPez4UtETRdkzThwbDqBE bt7g== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20210112; h=content-transfer-encoding:mime-version:references:in-reply-to :message-id:date:subject:cc:to:from:x-gm-message-state:from:to:cc :subject:date:message-id:reply-to; bh=5zyO0Na48QxjwG/96vzDf++z0O8XkPhrocrCmdAQIr4=; b=a6LLG0swK9FDbxTAw+QUYILpIsf8DY6i/3iuVQQtaJjlCfArcCuIzLF9qRr7ayPCmo wHc+CmdE64I6TqYp1lTvEVVjHn/DvljPmE8J7p9EF2ye+1WNcMfNLT4c2BGLm8q2izFw 6Z/B9bPvnOce5fCvfx1S1VmTRUH2OMP/q9TrpDdnWn3oPXYbziPK8WDjNtpm/r2HyVfn hV+4lEPs57Z1qmG9p9g8T6+Ef235Yug2t363+OZu6K1f5DuouEjiHoPGOWPoWr0krRi5 /bZ9F95BfKqxuQbcP8SQRIvgrQnjzmhlR9QkZetcj2DO0PNSNl1T7Pq6eDOWq2SIX+Cb UhRA== X-Gm-Message-State: AO0yUKW0+UZcDDppQGDQwKbc9XH5WfGLpTt/WxTCCnuGIWdHGaE29AML Q8KCEzjpNjVjacrXAiqACuY= X-Google-Smtp-Source: AK7set+0jg4e6z5KWbhW7fRuU0ZT2jMMbq5RDBzo5fVKlfYThFxG8WOLHSndBW6UWk3y/w+dt7g5fQ== X-Received: by 2002:a05:6a20:4428:b0:bd:278:f68f with SMTP id ce40-20020a056a20442800b000bd0278f68fmr11365775pzb.52.1675408745589; Thu, 02 Feb 2023 23:19:05 -0800 (PST) Received: from bobo.ibm.com (193-116-117-77.tpgi.com.au. [193.116.117.77]) by smtp.gmail.com with ESMTPSA id f20-20020a637554000000b004df4ba1ebfesm877558pgn.66.2023.02.02.23.19.00 (version=TLS1_3 cipher=TLS_AES_256_GCM_SHA384 bits=256/256); Thu, 02 Feb 2023 23:19:05 -0800 (PST) From: Nicholas Piggin To: Andrew Morton Cc: Nicholas Piggin , Linus Torvalds , Nadav Amit , Dave Hansen , Andy Lutomirski , Peter Zijlstra , Catalin Marinas , Will Deacon , Rik van Riel , linux-arch@vger.kernel.org, linux-mm@kvack.org, linuxppc-dev@lists.ozlabs.org Subject: [PATCH v7 2/5] lazy tlb: introduce lazy tlb mm refcount helper functions Date: Fri, 3 Feb 2023 17:18:34 +1000 Message-Id: <20230203071837.1136453-3-npiggin@gmail.com> X-Mailer: git-send-email 2.37.2 In-Reply-To: <20230203071837.1136453-1-npiggin@gmail.com> References: <20230203071837.1136453-1-npiggin@gmail.com> MIME-Version: 1.0 X-Rspamd-Server: rspam05 X-Rspamd-Queue-Id: A64D580008 X-Stat-Signature: iugc79yg6sstpdsfdomdpc5tkn9xu5xd X-Rspam-User: X-HE-Tag: 1675408746-382211 X-HE-Meta: U2FsdGVkX1/7TWXvv9WD5UmDiZiutLMjJVt8/6prg4mi0CGZoAIkD+mSjCtm/mcJ6yE94IB8u85K6Ucxp3xRMEY7VzDsa/Gw47Eipx/wWasd/V8E+Qpo01PLDW9iQe2TxXDTmXvU4g0wMuvG7zStkZbtHZ8jYnEPi1z/1laaPWhjNFxzmN162UPLOMWZ4tqt7qZaA1NoFXE/Gax0ng833Q8gMT/U9tHmT+6YMuH8ucqh8FWmHvzGVFGbWBCyhfU0JGeBUvXy8QNkPDKcMKg8P8WQuuPyGjFs6Jv3BeWXUcObsOKdS64IzD0T99pi3lqmdPpQoL9egjeSPvqsusOcVw7uENjq3JkiQIgVU2+E3pw3HLyNX4y65UycNXbfrS9wCs5ETa8LQkPcTAX3WZw264Ae2nTMNVHII1wTG3rDi7Q4BkvZfB2F/ipAOmcQ08svjREx4llaQZ3vOrSSXwZGnha8EcxAMobs+4wqh6Oi2g7im8KiX30f+fvvlCWEixPg0DO+vSbDrNVRT0heOl36xhW+XkTXbFwf5qSyIVGN9pPEV5ZhqdXcuev7iOv0A0+zduG+ltd6ub3zF//3R6KTmxHhyXoY1XXfsXSatJMzwiUnRlZ3HordFRCYjMCCCoH6Eu7vQnKB7aCMFnWw5YjMG69YVvm9UQddJkzqsCK9njo+nLv001B4pLFeNzIG6y2+6ZpdN/+digdHU8HqUuer8RM0acShyw1e3a64LkDcECC1AkpY6vpSl8TFwf3MYqdbMoeIIGHjVc4GrIvvCyBwJzj6s0aHL7UT7KNlLnU3h4bV4Pm/Df0giW7tc+xTLA8nnF8h4a7riv+khXtq1ax7+BtrreFvH/7f/QmsZAaslgZkhnTPIN6Ijl6iz1leoz1MZk5CeJkTxgdDMRCqYQFc0p6eULhn3qRM4W1tV2bMxMORDsC8+YBXK/RdY6IgePA+mx/ucNafymOe+yF2tDx 8fIOaeRC EH12OtCHrBzkPpJ/RC5F+SzaXn5JgPgEkHXRWXZFzdpIrWvqJItXBappoT93W9C2JJd99uoDugGR5NFZ77fDomcAdAJZeWJYBN1iJmujwwCLwz014rl8ELDY/L+hHKkWEGQiU1oPKfoHOCZwXIL81UV908Uf9mXw0688YVtvB/4TriH1LFqTtQqQ1X/YGbieUmMbdTe6+xTswnMgw0VxU52BDzdp44pxlqojj7cMT+LPrn1gsmHjCsNw4eyURdRC2jHk/EgLtoogkBv2tQl8x6SHSnxZIb/WdQTM9fDBqaHv2Ixo/24lVObjEK8LXGmr8UnSyWnWor2pkTOhOX1ybcksyqXf9S1SP6gQBvbfiVoargPv6ovjKgwwmqT+wJSUrOdxg8e0vLUmPVpeUBTogJk8CDPiWED2FEUagHdR4PFj8l87gfAZYhwksdogYwUAuGtppdJ+a7MMDKpxij9VktasFWbgvIf33jj95qTOQVtrZt2buEPSp+w+HNbyJeR1TuyRhWJHVZdyY0ZrAbteO3v87MosJI05KVSGJedovCWJVqjwLYRBOjQf2wHq8suGQDh4kVulXiijq1NQfDFcDREppDw== X-Bogosity: Ham, tests=bogofilter, spamicity=0.000000, version=1.2.4 Sender: owner-linux-mm@kvack.org Precedence: bulk X-Loop: owner-majordomo@kvack.org List-ID: Add explicit _lazy_tlb annotated functions for lazy tlb mm refcounting. This makes the lazy tlb mm references more obvious, and allows the refcounting scheme to be modified in later changes. There is no functional change with this patch. Acked-by: Linus Torvalds Signed-off-by: Nicholas Piggin --- arch/arm/mach-rpc/ecard.c | 2 +- arch/powerpc/kernel/smp.c | 2 +- arch/powerpc/mm/book3s64/radix_tlb.c | 4 ++-- fs/exec.c | 2 +- include/linux/sched/mm.h | 16 ++++++++++++++++ kernel/cpu.c | 2 +- kernel/exit.c | 2 +- kernel/kthread.c | 12 ++++++++++-- kernel/sched/core.c | 15 ++++++++------- 9 files changed, 41 insertions(+), 16 deletions(-) diff --git a/arch/arm/mach-rpc/ecard.c b/arch/arm/mach-rpc/ecard.c index 53813f9464a2..c30df1097c52 100644 --- a/arch/arm/mach-rpc/ecard.c +++ b/arch/arm/mach-rpc/ecard.c @@ -253,7 +253,7 @@ static int ecard_init_mm(void) current->mm = mm; current->active_mm = mm; activate_mm(active_mm, mm); - mmdrop(active_mm); + mmdrop_lazy_tlb(active_mm); ecard_init_pgtables(mm); return 0; } diff --git a/arch/powerpc/kernel/smp.c b/arch/powerpc/kernel/smp.c index 6b90f10a6c81..7db6b3faea65 100644 --- a/arch/powerpc/kernel/smp.c +++ b/arch/powerpc/kernel/smp.c @@ -1611,7 +1611,7 @@ void start_secondary(void *unused) if (IS_ENABLED(CONFIG_PPC32)) setup_kup(); - mmgrab(&init_mm); + mmgrab_lazy_tlb(&init_mm); current->active_mm = &init_mm; smp_store_cpu_info(cpu); diff --git a/arch/powerpc/mm/book3s64/radix_tlb.c b/arch/powerpc/mm/book3s64/radix_tlb.c index 4e29b619578c..282359ab525b 100644 --- a/arch/powerpc/mm/book3s64/radix_tlb.c +++ b/arch/powerpc/mm/book3s64/radix_tlb.c @@ -794,10 +794,10 @@ void exit_lazy_flush_tlb(struct mm_struct *mm, bool always_flush) if (current->active_mm == mm) { WARN_ON_ONCE(current->mm != NULL); /* Is a kernel thread and is using mm as the lazy tlb */ - mmgrab(&init_mm); + mmgrab_lazy_tlb(&init_mm); current->active_mm = &init_mm; switch_mm_irqs_off(mm, &init_mm, current); - mmdrop(mm); + mmdrop_lazy_tlb(mm); } /* diff --git a/fs/exec.c b/fs/exec.c index ab913243a367..1a32a88db173 100644 --- a/fs/exec.c +++ b/fs/exec.c @@ -1033,7 +1033,7 @@ static int exec_mmap(struct mm_struct *mm) mmput(old_mm); return 0; } - mmdrop(active_mm); + mmdrop_lazy_tlb(active_mm); return 0; } diff --git a/include/linux/sched/mm.h b/include/linux/sched/mm.h index 2a243616f222..5376caf6fcf3 100644 --- a/include/linux/sched/mm.h +++ b/include/linux/sched/mm.h @@ -79,6 +79,22 @@ static inline void mmdrop_sched(struct mm_struct *mm) } #endif +/* Helpers for lazy TLB mm refcounting */ +static inline void mmgrab_lazy_tlb(struct mm_struct *mm) +{ + mmgrab(mm); +} + +static inline void mmdrop_lazy_tlb(struct mm_struct *mm) +{ + mmdrop(mm); +} + +static inline void mmdrop_lazy_tlb_sched(struct mm_struct *mm) +{ + mmdrop_sched(mm); +} + /** * mmget() - Pin the address space associated with a &struct mm_struct. * @mm: The address space to pin. diff --git a/kernel/cpu.c b/kernel/cpu.c index 6c0a92ca6bb5..189895288d9d 100644 --- a/kernel/cpu.c +++ b/kernel/cpu.c @@ -623,7 +623,7 @@ static int finish_cpu(unsigned int cpu) */ if (mm != &init_mm) idle->active_mm = &init_mm; - mmdrop(mm); + mmdrop_lazy_tlb(mm); return 0; } diff --git a/kernel/exit.c b/kernel/exit.c index 15dc2ec80c46..1a4608d765e4 100644 --- a/kernel/exit.c +++ b/kernel/exit.c @@ -537,7 +537,7 @@ static void exit_mm(void) return; sync_mm_rss(mm); mmap_read_lock(mm); - mmgrab(mm); + mmgrab_lazy_tlb(mm); BUG_ON(mm != current->active_mm); /* more a memory barrier than a real lock */ task_lock(current); diff --git a/kernel/kthread.c b/kernel/kthread.c index 7424a1839e9a..e4bc32a88866 100644 --- a/kernel/kthread.c +++ b/kernel/kthread.c @@ -1410,6 +1410,11 @@ void kthread_use_mm(struct mm_struct *mm) WARN_ON_ONCE(!(tsk->flags & PF_KTHREAD)); WARN_ON_ONCE(tsk->mm); + /* + * It is possible for mm to be the same as tsk->active_mm, but + * we must still mmgrab(mm) and mmdrop_lazy_tlb(active_mm), + * because these references are not equivalent. + */ mmgrab(mm); task_lock(tsk); @@ -1433,9 +1438,9 @@ void kthread_use_mm(struct mm_struct *mm) * memory barrier after storing to tsk->mm, before accessing * user-space memory. A full memory barrier for membarrier * {PRIVATE,GLOBAL}_EXPEDITED is implicitly provided by - * mmdrop(). + * mmdrop_lazy_tlb(). */ - mmdrop(active_mm); + mmdrop_lazy_tlb(active_mm); } EXPORT_SYMBOL_GPL(kthread_use_mm); @@ -1463,10 +1468,13 @@ void kthread_unuse_mm(struct mm_struct *mm) local_irq_disable(); tsk->mm = NULL; membarrier_update_current_mm(NULL); + mmgrab_lazy_tlb(mm); /* active_mm is still 'mm' */ enter_lazy_tlb(mm, tsk); local_irq_enable(); task_unlock(tsk); + + mmdrop(mm); } EXPORT_SYMBOL_GPL(kthread_unuse_mm); diff --git a/kernel/sched/core.c b/kernel/sched/core.c index e838feb6adc5..495f9a021de9 100644 --- a/kernel/sched/core.c +++ b/kernel/sched/core.c @@ -5189,13 +5189,14 @@ static struct rq *finish_task_switch(struct task_struct *prev) * rq->curr, before returning to userspace, so provide them here: * * - a full memory barrier for {PRIVATE,GLOBAL}_EXPEDITED, implicitly - * provided by mmdrop(), + * provided by mmdrop_lazy_tlb(), * - a sync_core for SYNC_CORE. */ if (mm) { membarrier_mm_sync_core_before_usermode(mm); - mmdrop_sched(mm); + mmdrop_lazy_tlb_sched(mm); } + if (unlikely(prev_state == TASK_DEAD)) { if (prev->sched_class->task_dead) prev->sched_class->task_dead(prev); @@ -5252,9 +5253,9 @@ context_switch(struct rq *rq, struct task_struct *prev, /* * kernel -> kernel lazy + transfer active - * user -> kernel lazy + mmgrab() active + * user -> kernel lazy + mmgrab_lazy_tlb() active * - * kernel -> user switch + mmdrop() active + * kernel -> user switch + mmdrop_lazy_tlb() active * user -> user switch */ if (!next->mm) { // to kernel @@ -5262,7 +5263,7 @@ context_switch(struct rq *rq, struct task_struct *prev, next->active_mm = prev->active_mm; if (prev->mm) // from user - mmgrab(prev->active_mm); + mmgrab_lazy_tlb(prev->active_mm); else prev->active_mm = NULL; } else { // to user @@ -5279,7 +5280,7 @@ context_switch(struct rq *rq, struct task_struct *prev, lru_gen_use_mm(next->mm); if (!prev->mm) { // from kernel - /* will mmdrop() in finish_task_switch(). */ + /* will mmdrop_lazy_tlb() in finish_task_switch(). */ rq->prev_mm = prev->active_mm; prev->active_mm = NULL; } @@ -9916,7 +9917,7 @@ void __init sched_init(void) /* * The boot idle thread does lazy MMU switching as well: */ - mmgrab(&init_mm); + mmgrab_lazy_tlb(&init_mm); enter_lazy_tlb(&init_mm, current); /* From patchwork Fri Feb 3 07:18:35 2023 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Nicholas Piggin X-Patchwork-Id: 13127091 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from kanga.kvack.org (kanga.kvack.org [205.233.56.17]) by smtp.lore.kernel.org (Postfix) with ESMTP id DAADEC636CC for ; Fri, 3 Feb 2023 07:19:14 +0000 (UTC) Received: by kanga.kvack.org (Postfix) id 67A966B0075; Fri, 3 Feb 2023 02:19:14 -0500 (EST) Received: by kanga.kvack.org (Postfix, from userid 40) id 6294B6B0078; Fri, 3 Feb 2023 02:19:14 -0500 (EST) X-Delivered-To: int-list-linux-mm@kvack.org Received: by kanga.kvack.org (Postfix, from userid 63042) id 4F15B6B007B; Fri, 3 Feb 2023 02:19:14 -0500 (EST) X-Delivered-To: linux-mm@kvack.org Received: from relay.hostedemail.com (smtprelay0010.hostedemail.com [216.40.44.10]) by kanga.kvack.org (Postfix) with ESMTP id 411006B0075 for ; Fri, 3 Feb 2023 02:19:14 -0500 (EST) Received: from smtpin21.hostedemail.com (a10.router.float.18 [10.200.18.1]) by unirelay04.hostedemail.com (Postfix) with ESMTP id 1F4701A0372 for ; Fri, 3 Feb 2023 07:19:14 +0000 (UTC) X-FDA: 80425129428.21.526A0F9 Received: from mail-pj1-f53.google.com (mail-pj1-f53.google.com [209.85.216.53]) by imf06.hostedemail.com (Postfix) with ESMTP id 392E3180016 for ; Fri, 3 Feb 2023 07:19:12 +0000 (UTC) Authentication-Results: imf06.hostedemail.com; dkim=pass header.d=gmail.com header.s=20210112 header.b=pLnnqDRI; spf=pass (imf06.hostedemail.com: domain of npiggin@gmail.com designates 209.85.216.53 as permitted sender) smtp.mailfrom=npiggin@gmail.com; dmarc=pass (policy=none) header.from=gmail.com ARC-Seal: i=1; s=arc-20220608; d=hostedemail.com; t=1675408752; a=rsa-sha256; cv=none; b=Ybe7hBsDY51GZZrTTmMU4ELW6I7Rq2ogwtcinzon9aNUW+ioy5MO+u267YYTHpFv6AR6h5 DfZtrjcsaV0Kr/nLjdSYYMvzyMqh5bSSay3EPhFuPSSLdVzl5uQfCF/ZBFW8czeUaaUGoJ dX7aS0HwNeunTUgaQgkyAZcYTiMQTjQ= ARC-Authentication-Results: i=1; imf06.hostedemail.com; dkim=pass header.d=gmail.com header.s=20210112 header.b=pLnnqDRI; spf=pass (imf06.hostedemail.com: domain of npiggin@gmail.com designates 209.85.216.53 as permitted sender) smtp.mailfrom=npiggin@gmail.com; dmarc=pass (policy=none) header.from=gmail.com ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=hostedemail.com; s=arc-20220608; t=1675408752; h=from:from:sender:reply-to:subject:subject:date:date: message-id:message-id:to:to:cc:cc:mime-version:mime-version: content-type:content-transfer-encoding:content-transfer-encoding: in-reply-to:in-reply-to:references:references:dkim-signature; bh=XUGCesZ+ECKfxuO6+UdvSldGg8VBbbvoIh7jdtjrlkU=; b=GxLKLk3pAE3wpo/riCVx/7iRhKmSmx+tYP/wQaj+6Cmy8HikTdpIBbD6lU2wtnd9f431df xfz9yut6FSL87jYKUsQ23XndpXI+C6YMgy++R7VY8Pkiu0QTTbhxb+2OXRlu7HggabiX40 ap/dbLh14evKmdKB5Io7zHSllv3fc0c= Received: by mail-pj1-f53.google.com with SMTP id c10-20020a17090a1d0a00b0022e63a94799so7995625pjd.2 for ; Thu, 02 Feb 2023 23:19:11 -0800 (PST) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=20210112; h=content-transfer-encoding:mime-version:references:in-reply-to :message-id:date:subject:cc:to:from:from:to:cc:subject:date :message-id:reply-to; bh=XUGCesZ+ECKfxuO6+UdvSldGg8VBbbvoIh7jdtjrlkU=; b=pLnnqDRIDYF0X+7odeq2MLz8sT29lO0h9nXJRzhqSWb2cKDmw+AjnJVXwKX4LO0+YN sy6TWCVWrrhdAqdi0Athmqz7hnxbv9sXhy7zHgw5JMbTMALz+lrRMnA7mhKMW0UOaN3X QCHhXYJFOAJf6mMzTg6U9SYCB0rngQkrXqgCC9sJi9COvb9A1wQ5Pk8YAxIHPyyrhAK8 45tFQTFNKq2y3gE/ag6BjXudpVKSfK+TUTK3G6sll8axgwNiKP1UqpP4lrWqQnLzlWn0 p2jjdVOgV/IsArGVMlzfnf1uVgzpoRPyeV1+V87NiHGnhmhY+aWACx4xhLz/DXOY+iS+ dKVw== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20210112; h=content-transfer-encoding:mime-version:references:in-reply-to :message-id:date:subject:cc:to:from:x-gm-message-state:from:to:cc :subject:date:message-id:reply-to; bh=XUGCesZ+ECKfxuO6+UdvSldGg8VBbbvoIh7jdtjrlkU=; b=NdUtk6+LFCBZhYSuQYE4t5qQPG1Zmb7bUz1Wft1I1BSOgOEpPXpYx9SyFNRZZBsOR4 EPC5jHwwUWhsQOJtYCzAX/eLrgCzvsBI6i9s6duarBv5OrDHqgZfICtNzQ8xPBxrlU1M EpfiQjw53tP+uYp3Z9Lb5Yj7Br0hxKdBgNDINp22Nls3ftZ7sbO765D8DNrsbN+0O68N e0e2xltky1Wgh18+/06C15J0ZK9fg8FsrBbLXNu3Y25pQB8wEezO0O7CXX4kiC9kWGpf hThAgfYYpFEcvKn4aTj0JiGcfd7jJClCoQd6GEbvB9lT+wkVYOy7tqyOMpWDDUWvAqsh 85LA== X-Gm-Message-State: AO0yUKXzahn1R4fmaziPi8osR4iZ87omtMS4eM598gH0WY/H5v/bat2A ELuWBQysLI+ShTzcsEgj6ss= X-Google-Smtp-Source: AK7set9Pa1+UA0tNLvIosl6qOQXe2T4uQwVT+pZWT/dLmS0GvpN8PpOxJcaipNBn1b+VrjOFZQErtw== X-Received: by 2002:a05:6a21:78a9:b0:be:a944:b07f with SMTP id bf41-20020a056a2178a900b000bea944b07fmr12686834pzc.61.1675408751037; Thu, 02 Feb 2023 23:19:11 -0800 (PST) Received: from bobo.ibm.com (193-116-117-77.tpgi.com.au. [193.116.117.77]) by smtp.gmail.com with ESMTPSA id f20-20020a637554000000b004df4ba1ebfesm877558pgn.66.2023.02.02.23.19.05 (version=TLS1_3 cipher=TLS_AES_256_GCM_SHA384 bits=256/256); Thu, 02 Feb 2023 23:19:10 -0800 (PST) From: Nicholas Piggin To: Andrew Morton Cc: Nicholas Piggin , Linus Torvalds , Nadav Amit , Dave Hansen , Andy Lutomirski , Peter Zijlstra , Catalin Marinas , Will Deacon , Rik van Riel , linux-arch@vger.kernel.org, linux-mm@kvack.org, linuxppc-dev@lists.ozlabs.org Subject: [PATCH v7 3/5] lazy tlb: allow lazy tlb mm refcounting to be configurable Date: Fri, 3 Feb 2023 17:18:35 +1000 Message-Id: <20230203071837.1136453-4-npiggin@gmail.com> X-Mailer: git-send-email 2.37.2 In-Reply-To: <20230203071837.1136453-1-npiggin@gmail.com> References: <20230203071837.1136453-1-npiggin@gmail.com> MIME-Version: 1.0 X-Rspam-User: X-Rspamd-Queue-Id: 392E3180016 X-Rspamd-Server: rspam01 X-Stat-Signature: dnph1hz8yj3rrukokqkutajrafiq6wif X-HE-Tag: 1675408752-790495 X-HE-Meta: U2FsdGVkX18BRGFUKe+QuvlF6Nq7Yyu352SiLvK8AIr3oeEEoqD4M36z0mdk+hwuDzs3rC+kbmsrynK4PKf3s/+jWSRKSzH867Or8kVMudDipZn7BTdx2omiqwlyMGVqUJ7R2hMk38Bb//S4LPGTMb9erBWcgKszv1saLVTuWmxbmd+GLms4dUTyNPZIQJ9nSiJEaPeNIUZdpxmNWyeDEnmOCX6AC2HcvUtcEBOpn819nUHsag3yoL02QFVr3PkS0cqfkEKwWd5q0nD5PJLJcPy7edyMXz/RtJB/4InYZrvSE7mUae6R+cakdWbKjQZGflUmTAAbeJhM2YeI0jMs6BRsCdvRB0Z/vxWphAP7CVzi8Llkk3QTtAqTbROg6B97+AHoGtJVppX/UinZpNbYrqNuVVa26jFKkfKqEPuYSnHElwCTGoKX7+YJaVmsc9n1fo0jBX0MR5YD8KFZPsOin+A91mn0od+6ohbdmS4r2zbTv9A8LQUPaf5Jf70vZ3ksdA1aCzUokmDT0rnKjPLC5GEcDC5K/bxAHfb6sbEoWBprr8a/GzHBOC/NJpi5IBv2J0q02nsYAkwiPonfAvY4DoSDmkLiVw+krdcVILbsvFTMCfYCX9xvdf1hnrF9Dme+sh8rSqJeEzOopR8EI0tqGMc9hkR1k5PWHF/0uH3Td4FQ3KH52L15R8ijTZTppazbITy9nQd3ZyLtwD1QCLHFAqB/BwPPuPiSSd0046oFbyYvnUJ4UZW6r01AIPE6a/AMW594NMhALGAokQMH0niha00wbF52CcYUFuYmVPNd/3EZ/WwBnnjDfIOvzZChOt9xC3Lqpb/EGsN4xxViPhLqoapSMFvxIG7fTYxBGZXT4mAkJ+hsaRC0zWDVlDVWTEQFrSyfO3aKUz8bo11FORxadAvFIW5KCZVWKqU+S6xSSR5OTpY+0u5wtH6fEcJH1l6/qc38CAZNBgIWmq0iYWr 9Y/jUhIg VB6vMf7XSkBrItvhR0vn/JoorFQ3mrBi3Qx3iewQ9ERDBmPnaav7pC/Ern96NFuHNxLxZ/LFzCXQWXmZbICi2OL+abAsLIyy4Xr/+Tpu4tg2rHnviOqGuC05d/wiYsYfAiGFFa56e2pua+i/IZR4XwOaN2uA8aNT8NmbkUGaogCw0rO4y9ZLPTHOXbOXtvzLXIFqrZSWur+xWhTu/64EkkbogiS+J+dXRzhKnroN/q0CVXokUqDtyb7iD0JDy6ORNB/EPQ/tebokvsYwaD0q1zqJoUMkeQ0XPmfdjV0JAukffPYsmoFLgxoCzAPULpH7V/44C4+aZvHuf4Ue6Jue0GbRbG18jNpi1YeBjIVN7bOt+DqdCKJDKjSzb4z/PEC3RBE3ff5P3tSQncKMYYTQLviFOICbicmoSg+9Idc5U+jFHBgOMBVQVK91iFK221BH+xjkdOpigc5vNoyCBLymAlB8V0ZZeNTn8CBS+GmZjkScmrUprK5XJNmh3tc0DVvCgTA/fmO1FLkoSYFD5/LCoQbKr7J3YTJdC6znlbPNB/cjC4+3IdUP9JDKQjQKMbz3PrqJqDlXOwW+rbD5D6miya8a2Xg== X-Bogosity: Ham, tests=bogofilter, spamicity=0.000000, version=1.2.4 Sender: owner-linux-mm@kvack.org Precedence: bulk X-Loop: owner-majordomo@kvack.org List-ID: Add CONFIG_MMU_TLB_REFCOUNT which enables refcounting of the lazy tlb mm when it is context switched. This can be disabled by architectures that don't require this refcounting if they clean up lazy tlb mms when the last refcount is dropped. Currently this is always enabled, so the patch introduces no functional change. Acked-by: Linus Torvalds Signed-off-by: Nicholas Piggin --- Documentation/mm/active_mm.rst | 6 ++++++ arch/Kconfig | 17 +++++++++++++++++ include/linux/sched/mm.h | 18 +++++++++++++++--- 3 files changed, 38 insertions(+), 3 deletions(-) diff --git a/Documentation/mm/active_mm.rst b/Documentation/mm/active_mm.rst index 6f8269c284ed..0114d80d406a 100644 --- a/Documentation/mm/active_mm.rst +++ b/Documentation/mm/active_mm.rst @@ -4,6 +4,12 @@ Active MM ========= +Note, the mm_count refcount may no longer include the "lazy" users +(running tasks with ->active_mm == mm && ->mm == NULL) on kernels +with CONFIG_MMU_LAZY_TLB_REFCOUNT=n. Taking and releasing these lazy +references must be done with mmgrab_lazy_tlb() and mmdrop_lazy_tlb() +helpers, which abstract this config option. + :: List: linux-kernel diff --git a/arch/Kconfig b/arch/Kconfig index 12e3ddabac9d..11e8915c0652 100644 --- a/arch/Kconfig +++ b/arch/Kconfig @@ -465,6 +465,23 @@ config ARCH_WANT_IRQS_OFF_ACTIVATE_MM irqs disabled over activate_mm. Architectures that do IPI based TLB shootdowns should enable this. +# Use normal mm refcounting for MMU_LAZY_TLB kernel thread references. +# MMU_LAZY_TLB_REFCOUNT=n can improve the scalability of context switching +# to/from kernel threads when the same mm is running on a lot of CPUs (a large +# multi-threaded application), by reducing contention on the mm refcount. +# +# This can be disabled if the architecture ensures no CPUs are using an mm as a +# "lazy tlb" beyond its final refcount (i.e., by the time __mmdrop frees the mm +# or its kernel page tables). This could be arranged by arch_exit_mmap(), or +# final exit(2) TLB flush, for example. +# +# To implement this, an arch *must*: +# Ensure the _lazy_tlb variants of mmgrab/mmdrop are used when manipulating +# the lazy tlb reference of a kthread's ->active_mm (non-arch code has been +# converted already). +config MMU_LAZY_TLB_REFCOUNT + def_bool y + config ARCH_HAVE_NMI_SAFE_CMPXCHG bool diff --git a/include/linux/sched/mm.h b/include/linux/sched/mm.h index 5376caf6fcf3..689dbe812563 100644 --- a/include/linux/sched/mm.h +++ b/include/linux/sched/mm.h @@ -82,17 +82,29 @@ static inline void mmdrop_sched(struct mm_struct *mm) /* Helpers for lazy TLB mm refcounting */ static inline void mmgrab_lazy_tlb(struct mm_struct *mm) { - mmgrab(mm); + if (IS_ENABLED(CONFIG_MMU_LAZY_TLB_REFCOUNT)) + mmgrab(mm); } static inline void mmdrop_lazy_tlb(struct mm_struct *mm) { - mmdrop(mm); + if (IS_ENABLED(CONFIG_MMU_LAZY_TLB_REFCOUNT)) { + mmdrop(mm); + } else { + /* + * mmdrop_lazy_tlb must provide a full memory barrier, see the + * membarrier comment finish_task_switch which relies on this. + */ + smp_mb(); + } } static inline void mmdrop_lazy_tlb_sched(struct mm_struct *mm) { - mmdrop_sched(mm); + if (IS_ENABLED(CONFIG_MMU_LAZY_TLB_REFCOUNT)) + mmdrop_sched(mm); + else + smp_mb(); /* see mmdrop_lazy_tlb() above */ } /** From patchwork Fri Feb 3 07:18:36 2023 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Nicholas Piggin X-Patchwork-Id: 13127092 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from kanga.kvack.org (kanga.kvack.org [205.233.56.17]) by smtp.lore.kernel.org (Postfix) with ESMTP id 4AE86C61DA4 for ; Fri, 3 Feb 2023 07:19:20 +0000 (UTC) Received: by kanga.kvack.org (Postfix) id DA5A66B0078; Fri, 3 Feb 2023 02:19:19 -0500 (EST) Received: by kanga.kvack.org (Postfix, from userid 40) id D55806B007B; Fri, 3 Feb 2023 02:19:19 -0500 (EST) X-Delivered-To: int-list-linux-mm@kvack.org Received: by kanga.kvack.org (Postfix, from userid 63042) id BF6DA6B007D; Fri, 3 Feb 2023 02:19:19 -0500 (EST) X-Delivered-To: linux-mm@kvack.org Received: from relay.hostedemail.com (smtprelay0012.hostedemail.com [216.40.44.12]) by kanga.kvack.org (Postfix) with ESMTP id B0FA16B0078 for ; Fri, 3 Feb 2023 02:19:19 -0500 (EST) Received: from smtpin28.hostedemail.com (a10.router.float.18 [10.200.18.1]) by unirelay04.hostedemail.com (Postfix) with ESMTP id 7824D1A10A1 for ; Fri, 3 Feb 2023 07:19:19 +0000 (UTC) X-FDA: 80425129638.28.27FD3BD Received: from mail-pl1-f172.google.com (mail-pl1-f172.google.com [209.85.214.172]) by imf21.hostedemail.com (Postfix) with ESMTP id ADBF71C0002 for ; Fri, 3 Feb 2023 07:19:17 +0000 (UTC) Authentication-Results: imf21.hostedemail.com; dkim=pass header.d=gmail.com header.s=20210112 header.b=GCqfd0gi; dmarc=pass (policy=none) header.from=gmail.com; spf=pass (imf21.hostedemail.com: domain of npiggin@gmail.com designates 209.85.214.172 as permitted sender) smtp.mailfrom=npiggin@gmail.com ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=hostedemail.com; s=arc-20220608; t=1675408757; h=from:from:sender:reply-to:subject:subject:date:date: message-id:message-id:to:to:cc:cc:mime-version:mime-version: content-type:content-transfer-encoding:content-transfer-encoding: in-reply-to:in-reply-to:references:references:dkim-signature; bh=ISWO9h6SOLQP6WCeahRDXPdJymqgjLrbpJ6FMzUJXpQ=; b=h4UcBBM08AsIhgXDm10IBdhltrxrjPrDBN0dn9VuKbsXRE/USXkI8ArNLnFvQ3p5m5nH0r V1pY5EqZ1ALXx6BYFPJVq5OA2zaqqLfkuA/LeZptCvlqspvUqlmhx3fiXHc4BtrRVXLDKC NN5w6X3lYaLguFzEuCHC1jQl3kFvLY0= ARC-Authentication-Results: i=1; imf21.hostedemail.com; dkim=pass header.d=gmail.com header.s=20210112 header.b=GCqfd0gi; dmarc=pass (policy=none) header.from=gmail.com; spf=pass (imf21.hostedemail.com: domain of npiggin@gmail.com designates 209.85.214.172 as permitted sender) smtp.mailfrom=npiggin@gmail.com ARC-Seal: i=1; s=arc-20220608; d=hostedemail.com; t=1675408757; a=rsa-sha256; cv=none; b=MELzVPVGQlrVwfi1Aanf9TcFBPNcrromr94KFMMNjCpifawLbduJ5wLJ3u8gXCdCN5bAn6 Q4DyhdDBC1/P16YnanWG8wT4pZf3DWWkLGqI/SPIuUj1b3hfmvun0A/f6bSrzxY7YFonrI izRXsn53UN00jhkLeWu6OR1xnSPrAtU= Received: by mail-pl1-f172.google.com with SMTP id u9so148988plf.3 for ; Thu, 02 Feb 2023 23:19:17 -0800 (PST) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=20210112; h=content-transfer-encoding:mime-version:references:in-reply-to :message-id:date:subject:cc:to:from:from:to:cc:subject:date :message-id:reply-to; bh=ISWO9h6SOLQP6WCeahRDXPdJymqgjLrbpJ6FMzUJXpQ=; b=GCqfd0gisgkgKlxYZuLHUdcwKWxDsZidEXcZ1aNaZZ2ElupC8rz5g3nIkJ3bobpmqi DprWpFD65mlWDpzUu/unAkIGKL37dle+6bCw2lSx+pZmmhDM758k0n8nOhE+ohbJrWlG nnQ2rVqpFtFDnzZ9uApSYB6/eFyeFbXLHLC1IdotvCqzph/BPkikJi0WfNZA9G0CsEND hYHG9zGzwoF31aJ8l6S4dqlVqQzy8grON+LE2Meicn20ZhPs4l0HXU0uHvm0k+HFrdBm AlnDrtDXEIMMr2ZDtrtaE7KDDPGpuU2KJd9TLaiqM1heX5H+w6emkvg9SMXCTv+a9luZ bBzQ== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20210112; h=content-transfer-encoding:mime-version:references:in-reply-to :message-id:date:subject:cc:to:from:x-gm-message-state:from:to:cc :subject:date:message-id:reply-to; bh=ISWO9h6SOLQP6WCeahRDXPdJymqgjLrbpJ6FMzUJXpQ=; b=oM8RI8MGD9cRsOWVsDez8IugLh8ZwA8x2Ran/Y+7BC2yFVa+4bKyWsOnrgKwRkMGKI 9ZBS5UkMkav+0pmbXNBwGrQYMGJ2p6iaoJ4X0psyis84pI9h58MWhBGn4KNaaNC2vogZ RNgbwGl05TZf6pdc8a8ULsjEpn+c5/MYRNPJR4+CnajwYWbEBLcnPUJKMP4NkOxgncdt EFNNAnEsDp7JxXz2+jOvhaBw5NpN45TqSKmm5AHVHBUjSTaMfz3+b6jahJql91KXG2LN cLD+zXSPOAfuivszPz+wf+udavH69paddJ1DMIjY2GMoCSOF+rHWI753aocLxiEYRG/x 9ckA== X-Gm-Message-State: AO0yUKWI793es6f9plDc2nkgHRLBujpd/lfA5IjRVoJD54GtVa2yRi2m HZ8Id8jWQHUJGC1YP/ZWLSg= X-Google-Smtp-Source: AK7set+asp6Wk4F7oZN8F9+ScVFgiV4xOvEyfkj0CJUO9gvbWyyWzAGztHv22zCvJ0UaYCbOJNjkVQ== X-Received: by 2002:a05:6a20:6909:b0:b6:b6a6:9753 with SMTP id q9-20020a056a20690900b000b6b6a69753mr11785825pzj.8.1675408756632; Thu, 02 Feb 2023 23:19:16 -0800 (PST) Received: from bobo.ibm.com (193-116-117-77.tpgi.com.au. [193.116.117.77]) by smtp.gmail.com with ESMTPSA id f20-20020a637554000000b004df4ba1ebfesm877558pgn.66.2023.02.02.23.19.11 (version=TLS1_3 cipher=TLS_AES_256_GCM_SHA384 bits=256/256); Thu, 02 Feb 2023 23:19:16 -0800 (PST) From: Nicholas Piggin To: Andrew Morton Cc: Nicholas Piggin , Linus Torvalds , Nadav Amit , Dave Hansen , Andy Lutomirski , Peter Zijlstra , Catalin Marinas , Will Deacon , Rik van Riel , linux-arch@vger.kernel.org, linux-mm@kvack.org, linuxppc-dev@lists.ozlabs.org Subject: [PATCH v7 4/5] lazy tlb: shoot lazies, non-refcounting lazy tlb mm reference handling scheme Date: Fri, 3 Feb 2023 17:18:36 +1000 Message-Id: <20230203071837.1136453-5-npiggin@gmail.com> X-Mailer: git-send-email 2.37.2 In-Reply-To: <20230203071837.1136453-1-npiggin@gmail.com> References: <20230203071837.1136453-1-npiggin@gmail.com> MIME-Version: 1.0 X-Rspamd-Queue-Id: ADBF71C0002 X-Rspamd-Server: rspam09 X-Rspam-User: X-Stat-Signature: s8u56c6b6xahtkdxw6j8phzksb3je4yp X-HE-Tag: 1675408757-419578 X-HE-Meta: U2FsdGVkX1+IFPU6jZeajRJ+TigUZwuk5FqHBx8UesydjentBE4iV5KAqEvwcQSR97dQ1dpBPyunokiczEGzks/Roj6fVSnFuUjF3xQh46o3vb0BpA12d00UouZ/IZSK2NLusqUK2rEAirBaWaKAOxQAZbz/VrMLWSk4hAVT5xAawEHGekfPTlF+cRIXfQqWMo8sR3nCfXusGhR2IJKSkA3Vq9Qhuuh8TlXMl1460BY+ttzZH4cq+dF7RPQ/mlEi6DiEyAYvKrEyPU4cTB5SWyKJIVBPDtfKKJJ4NquZDmNspAwkyZgkNGqagzW48LQcUIY2hCfoiZXP/BXWaF1S5xclGtW8iIjZM1NosciDmMYOlpcjJokdVhMY8T/cNDhvjimu8me2Od2socuk6cCbFEST6Q6d8lIjuXsNbSfD4IIDD06OJ8kqlSsZ0mH9+O00BpgwyXBfOLGsxef/9OKlM04Q5uR3ZTrqgWbdHePwnRy/YxrX0x0Ct4jwm0FRTvetkW9zryRYHYygWz31rZmdN79hBmPKRsSxlj92laJ7/HVDAgdF7yacXcpwQO9V/m6dixcS02KxG1zlpLYZ4KOoALFOdzr0epou/+63JStpVuYUg2T1BG/6Hq1v7JC8/QPpY884eRgi09p2FHGpZV7aG6KuHXwFmC2hO6dnUyX2PzdYbpzFgDOsiVROPEimw6BRMDRS+u7lXwT06gyVfLhnr1ETEBT1DsmvhIFHrGeOnhZs5zCajQnthqDzP4FgDvowuw0zJcAPuzHNGtvgZasH/os2vF/rTeN9Qu/ovab4b6EDiaRaYnI2Ks3L6rtgK4WL2cJPEGAB1qek4/WwkSBupGrNIjNOplbqpTYkllelSu4/leuJqOEA/Dxl5By82XOZrqeiWRy4qELTQU/zXwUakwrK076Nnt1QxdmrCZFkh8z76Mskuy6UkGkwHjtIbydp9c7mplf53591yqIS+6y tZIPtp7+ G4I5q8o1ILxPxl24QeHbBYhhgrmq8zUAlNqGNezQVAlX3dsxVa8l/CNykkP3/TSIp2nJ+hqo/R5pgBoY3TEK6eEpzkRBU3/xSkJQI3trjXuPBzx4k2LEb7sCLtiZQcJwzzZUUmDMPvR+4jlL54KvQYMt7YHBHEYetdtkcUjFVCyC+TEp24lk5ZP/6oIPtPyILsIG6fVVu6dPZAgc4ca/PdhQVRYDOqwXcmZl4wQC6zwqrSYUMkWd9hrStrpcuJkkZuZ6Bt+YftLqKGlZbpWix/bmf4jeA/Kjb9SntCg35QzJ7J7BM/Y2/HSm/SRtxEw+lUnL8GMWZHqN/Q97S5BhmEyv5pTC6pW7d7bvfKl1MlUlX+Db6ZfT+OQJH2whhg6zUY06/hKD8+8poWL4uLbHXCIqbAv+nOu27dTKNBUy5f4NleosJuxJllswG333AcXKjSSjHKVIqkUSDcUHm8sIPLvfRlvITbZBqbqkC9DTFjraZfIEDjSUuA7YNLb2gKxpp8qLQ2FCNav41gQmc7HxVhWcuv+ehrfWSzgMPVIeDBWk5RoqhpvhxfT8SRY1hoGa3b6BCNKs1PXFToHnI9rc8aDEDxw8DYq3fp/3P5RqXZmUers9kPdApC9/uacyjSkeEOI1W57sby9S8DjjQzWlKJW0qLKrte7J9qly0XmvHDbfDrxM= X-Bogosity: Ham, tests=bogofilter, spamicity=0.000000, version=1.2.4 Sender: owner-linux-mm@kvack.org Precedence: bulk X-Loop: owner-majordomo@kvack.org List-ID: On big systems, the mm refcount can become highly contented when doing a lot of context switching with threaded applications. user<->idle switch is one of the important cases. Abandoning lazy tlb entirely slows this switching down quite a bit in the common uncontended case, so that is not viable. Implement a scheme where lazy tlb mm references do not contribute to the refcount, instead they get explicitly removed when the refcount reaches zero. The final mmdrop() sends IPIs to all CPUs in the mm_cpumask and they switch away from this mm to init_mm if it was being used as the lazy tlb mm. Enabling the shoot lazies option therefore requires that the arch ensures that mm_cpumask contains all CPUs that could possibly be using mm. A DEBUG_VM option IPIs every CPU in the system after this to ensure there are no references remaining before the mm is freed. Shootdown IPIs cost could be an issue, but they have not been observed to be a serious problem with this scheme, because short-lived processes tend not to migrate CPUs much, therefore they don't get much chance to leave lazy tlb mm references on remote CPUs. There are a lot of options to reduce them if necessary, described in comments. The near-worst-case can be benchmarked with will-it-scale: context_switch1_threads -t $(($(nproc) / 2)) This will create nproc threads (nproc / 2 switching pairs) all sharing the same mm that spread over all CPUs so each CPU does thread->idle->thread switching. [ Rik came up with basically the same idea a few years ago, so credit to him for that. ] Link: https://lore.kernel.org/linux-mm/20230118080011.2258375-1-npiggin@gmail.com/ Link: https://lore.kernel.org/all/20180728215357.3249-11-riel@surriel.com/ Acked-by: Linus Torvalds Signed-off-by: Nicholas Piggin --- arch/Kconfig | 15 +++++++++++ kernel/fork.c | 65 +++++++++++++++++++++++++++++++++++++++++++++++ lib/Kconfig.debug | 10 ++++++++ 3 files changed, 90 insertions(+) diff --git a/arch/Kconfig b/arch/Kconfig index 11e8915c0652..0d2021aed57e 100644 --- a/arch/Kconfig +++ b/arch/Kconfig @@ -481,6 +481,21 @@ config ARCH_WANT_IRQS_OFF_ACTIVATE_MM # converted already). config MMU_LAZY_TLB_REFCOUNT def_bool y + depends on !MMU_LAZY_TLB_SHOOTDOWN + +# This option allows MMU_LAZY_TLB_REFCOUNT=n. It ensures no CPUs are using an +# mm as a lazy tlb beyond its last reference count, by shooting down these +# users before the mm is deallocated. __mmdrop() first IPIs all CPUs that may +# be using the mm as a lazy tlb, so that they may switch themselves to using +# init_mm for their active mm. mm_cpumask(mm) is used to determine which CPUs +# may be using mm as a lazy tlb mm. +# +# To implement this, an arch *must*: +# - At the time of the final mmdrop of the mm, ensure mm_cpumask(mm) contains +# at least all possible CPUs in which the mm is lazy. +# - It must meet the requirements for MMU_LAZY_TLB_REFCOUNT=n (see above). +config MMU_LAZY_TLB_SHOOTDOWN + bool config ARCH_HAVE_NMI_SAFE_CMPXCHG bool diff --git a/kernel/fork.c b/kernel/fork.c index 9f7fe3541897..e7d81db7e885 100644 --- a/kernel/fork.c +++ b/kernel/fork.c @@ -780,6 +780,67 @@ static void check_mm(struct mm_struct *mm) #define allocate_mm() (kmem_cache_alloc(mm_cachep, GFP_KERNEL)) #define free_mm(mm) (kmem_cache_free(mm_cachep, (mm))) +static void do_check_lazy_tlb(void *arg) +{ + struct mm_struct *mm = arg; + + WARN_ON_ONCE(current->active_mm == mm); +} + +static void do_shoot_lazy_tlb(void *arg) +{ + struct mm_struct *mm = arg; + + if (current->active_mm == mm) { + WARN_ON_ONCE(current->mm); + current->active_mm = &init_mm; + switch_mm(mm, &init_mm, current); + } +} + +static void cleanup_lazy_tlbs(struct mm_struct *mm) +{ + if (!IS_ENABLED(CONFIG_MMU_LAZY_TLB_SHOOTDOWN)) { + /* + * In this case, lazy tlb mms are refounted and would not reach + * __mmdrop until all CPUs have switched away and mmdrop()ed. + */ + return; + } + + /* + * Lazy mm shootdown does not refcount "lazy tlb mm" usage, rather it + * requires lazy mm users to switch to another mm when the refcount + * drops to zero, before the mm is freed. This requires IPIs here to + * switch kernel threads to init_mm. + * + * archs that use IPIs to flush TLBs can piggy-back that lazy tlb mm + * switch with the final userspace teardown TLB flush which leaves the + * mm lazy on this CPU but no others, reducing the need for additional + * IPIs here. There are cases where a final IPI is still required here, + * such as the final mmdrop being performed on a different CPU than the + * one exiting, or kernel threads using the mm when userspace exits. + * + * IPI overheads have not found to be expensive, but they could be + * reduced in a number of possible ways, for example (roughly + * increasing order of complexity): + * - The last lazy reference created by exit_mm() could instead switch + * to init_mm, however it's probable this will run on the same CPU + * immediately afterwards, so this may not reduce IPIs much. + * - A batch of mms requiring IPIs could be gathered and freed at once. + * - CPUs store active_mm where it can be remotely checked without a + * lock, to filter out false-positives in the cpumask. + * - After mm_users or mm_count reaches zero, switching away from the + * mm could clear mm_cpumask to reduce some IPIs, perhaps together + * with some batching or delaying of the final IPIs. + * - A delayed freeing and RCU-like quiescing sequence based on mm + * switching to avoid IPIs completely. + */ + on_each_cpu_mask(mm_cpumask(mm), do_shoot_lazy_tlb, (void *)mm, 1); + if (IS_ENABLED(CONFIG_DEBUG_VM_SHOOT_LAZIES)) + on_each_cpu(do_check_lazy_tlb, (void *)mm, 1); +} + /* * Called when the last reference to the mm * is dropped: either by a lazy thread or by @@ -791,6 +852,10 @@ void __mmdrop(struct mm_struct *mm) BUG_ON(mm == &init_mm); WARN_ON_ONCE(mm == current->mm); + + /* Ensure no CPUs are using this as their lazy tlb mm */ + cleanup_lazy_tlbs(mm); + WARN_ON_ONCE(mm == current->active_mm); mm_free_pgd(mm); destroy_context(mm); diff --git a/lib/Kconfig.debug b/lib/Kconfig.debug index 61a9425a311f..1a5849f9f414 100644 --- a/lib/Kconfig.debug +++ b/lib/Kconfig.debug @@ -852,6 +852,16 @@ config DEBUG_VM If unsure, say N. +config DEBUG_VM_SHOOT_LAZIES + bool "Debug MMU_LAZY_TLB_SHOOTDOWN implementation" + depends on DEBUG_VM + depends on MMU_LAZY_TLB_SHOOTDOWN + help + Enable additional IPIs that ensure lazy tlb mm references are removed + before the mm is freed. + + If unsure, say N. + config DEBUG_VM_MAPLE_TREE bool "Debug VM maple trees" depends on DEBUG_VM From patchwork Fri Feb 3 07:18:37 2023 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Nicholas Piggin X-Patchwork-Id: 13127093 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from kanga.kvack.org (kanga.kvack.org [205.233.56.17]) by smtp.lore.kernel.org (Postfix) with ESMTP id 697DEC61DA4 for ; Fri, 3 Feb 2023 07:19:26 +0000 (UTC) Received: by kanga.kvack.org (Postfix) id EB8436B007B; Fri, 3 Feb 2023 02:19:25 -0500 (EST) Received: by kanga.kvack.org (Postfix, from userid 40) id E66B26B007D; Fri, 3 Feb 2023 02:19:25 -0500 (EST) X-Delivered-To: int-list-linux-mm@kvack.org Received: by kanga.kvack.org (Postfix, from userid 63042) id D60096B007E; Fri, 3 Feb 2023 02:19:25 -0500 (EST) X-Delivered-To: linux-mm@kvack.org Received: from relay.hostedemail.com (smtprelay0017.hostedemail.com [216.40.44.17]) by kanga.kvack.org (Postfix) with ESMTP id C50F46B007B for ; Fri, 3 Feb 2023 02:19:25 -0500 (EST) Received: from smtpin30.hostedemail.com (a10.router.float.18 [10.200.18.1]) by unirelay08.hostedemail.com (Postfix) with ESMTP id E425F1402AA for ; Fri, 3 Feb 2023 07:19:24 +0000 (UTC) X-FDA: 80425129848.30.24655EA Received: from mail-pj1-f44.google.com (mail-pj1-f44.google.com [209.85.216.44]) by imf13.hostedemail.com (Postfix) with ESMTP id 25F1D2000C for ; Fri, 3 Feb 2023 07:19:22 +0000 (UTC) Authentication-Results: imf13.hostedemail.com; dkim=pass header.d=gmail.com header.s=20210112 header.b=h7nAnXYf; spf=pass (imf13.hostedemail.com: domain of npiggin@gmail.com designates 209.85.216.44 as permitted sender) smtp.mailfrom=npiggin@gmail.com; dmarc=pass (policy=none) header.from=gmail.com ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=hostedemail.com; s=arc-20220608; t=1675408763; h=from:from:sender:reply-to:subject:subject:date:date: message-id:message-id:to:to:cc:cc:mime-version:mime-version: content-type:content-transfer-encoding:content-transfer-encoding: in-reply-to:in-reply-to:references:references:dkim-signature; bh=x14kRrZZe2kdSbgJ1PJqUMQQfgWflSMZWL5LFkwvByE=; b=NMDMLkMsAB8XnHvoj2nH/iMXZPvVn7StzqxTjV3QglHtLlJJZIf+TrySuSF6Rx4F/n/IAi w8OIvuoHXmskPRrgBiLswQVT5YWvFZMTi0pIxyz027ANUc/HqOLzHmOPF3pAeZfzVYUkjQ vFVBNlv3WE0VxBO+rz0y2NCSEU4NmN0= ARC-Authentication-Results: i=1; imf13.hostedemail.com; dkim=pass header.d=gmail.com header.s=20210112 header.b=h7nAnXYf; spf=pass (imf13.hostedemail.com: domain of npiggin@gmail.com designates 209.85.216.44 as permitted sender) smtp.mailfrom=npiggin@gmail.com; dmarc=pass (policy=none) header.from=gmail.com ARC-Seal: i=1; s=arc-20220608; d=hostedemail.com; t=1675408763; a=rsa-sha256; cv=none; b=aUOTYvyTMR7zMKhcOIZ+b5Jg81abe37PrLtS01rZJmRAtSTMHoQHihdu8kv7vVEdlkxrOH 99KR2tQFhxR/38DkBqg6H3PUgzehqpX9Ll9u38aCF7j56NMQ3s1cdvxs6DhuiZRdvUnCKG QvB+VPpojv6R70sQl3d21afYZ4OXtas= Received: by mail-pj1-f44.google.com with SMTP id m11so4239233pji.0 for ; Thu, 02 Feb 2023 23:19:22 -0800 (PST) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=20210112; h=content-transfer-encoding:mime-version:references:in-reply-to :message-id:date:subject:cc:to:from:from:to:cc:subject:date :message-id:reply-to; bh=x14kRrZZe2kdSbgJ1PJqUMQQfgWflSMZWL5LFkwvByE=; b=h7nAnXYffJkXAeKtJ4+Pb47ZmdkwzoIxpzRgCBuRqFSEuDy+TP7MF7IO/iVxeE9h3P aG7Esd2gb18pAjYT42bsVp7oBg2AItWYJWILKrT1Rv1wljyPlzKVe3HEajToWCRcZVtL zy4qIqHGZ3aoBW3DgCSA/Vp7o+H66URuFiIddPmq0wydnig1tvUkwDog28Na9VCQmXCY VaSnJ2jOszZ6JD6S3Lt/kqQ/A4667tSOoYJbLAZ3Dg/8Mt+fnJ7poqcB8fdgDCdZgr9h DFZ/UDXahTH+Bf05/C7Y+Y3gm5kj6MbI54iJLvql0E7GMYErZm9nVGNUKj5tyaG4S5jq 3YEA== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20210112; h=content-transfer-encoding:mime-version:references:in-reply-to :message-id:date:subject:cc:to:from:x-gm-message-state:from:to:cc :subject:date:message-id:reply-to; bh=x14kRrZZe2kdSbgJ1PJqUMQQfgWflSMZWL5LFkwvByE=; b=dIUqEOnJIK+WXSMIG6h+h+vZ78VliBly5D6UaZ7p5D7clMXKmC59Arj2Fib11w6Jey fsA0ormqQV1t4AseS+8HQ9k8wuweGCyuNCfq26wyEN7PcAxArmEMVeDuHLMZ4PvO5yD/ NjTMb1YYgVOLvxtxhSfU91Njx8xFYxLU4AikOG0G1zvYhmS/ErHW8LWNJYy33yWZpJZ2 E6v84gKCRUEHS5Hh1iiPNsMhNqYV5XMO693dAzmhNX3qVrADjdGkwZtB3lyJXFuiJJRm fKTIT4NnpIw1FRbvkFlllA0iQClnEnkqASHWeyaaeTfTsSRNl1M/ZKUPPRNQ3zAqZJFF oFbg== X-Gm-Message-State: AO0yUKUlkR5p38KLjcCRiKn8/rP+L2Iq37lfG85E/fvpOYpi5jbqQ+ax MewvdJDxReJLPHO9TA3yS4J70CNzgGE= X-Google-Smtp-Source: AK7set/pN1qkE5vW6sEUJPLmtT3NU/AerHL7q01jFwtmVHRhEkUBx1PKvOIigiSi96c+JcvQpUaLgA== X-Received: by 2002:a17:90a:11:b0:22c:8dfe:d6a6 with SMTP id 17-20020a17090a001100b0022c8dfed6a6mr9294792pja.4.1675408762194; Thu, 02 Feb 2023 23:19:22 -0800 (PST) Received: from bobo.ibm.com (193-116-117-77.tpgi.com.au. [193.116.117.77]) by smtp.gmail.com with ESMTPSA id f20-20020a637554000000b004df4ba1ebfesm877558pgn.66.2023.02.02.23.19.16 (version=TLS1_3 cipher=TLS_AES_256_GCM_SHA384 bits=256/256); Thu, 02 Feb 2023 23:19:21 -0800 (PST) From: Nicholas Piggin To: Andrew Morton Cc: Nicholas Piggin , Linus Torvalds , Nadav Amit , Dave Hansen , Andy Lutomirski , Peter Zijlstra , Catalin Marinas , Will Deacon , Rik van Riel , linux-arch@vger.kernel.org, linux-mm@kvack.org, linuxppc-dev@lists.ozlabs.org Subject: [PATCH v7 5/5] powerpc/64s: enable MMU_LAZY_TLB_SHOOTDOWN Date: Fri, 3 Feb 2023 17:18:37 +1000 Message-Id: <20230203071837.1136453-6-npiggin@gmail.com> X-Mailer: git-send-email 2.37.2 In-Reply-To: <20230203071837.1136453-1-npiggin@gmail.com> References: <20230203071837.1136453-1-npiggin@gmail.com> MIME-Version: 1.0 X-Rspam-User: X-Rspamd-Server: rspam03 X-Stat-Signature: hhk3j18rua548arwxytk1dyag41b7btp X-Rspamd-Queue-Id: 25F1D2000C X-HE-Tag: 1675408762-727050 X-HE-Meta: U2FsdGVkX19LGLx7uK0GfimiFdM2OeptKrviT9ryXa16SA4ntBlwXnvHhtltCCfKVi5GKJ8+okUhnxJ5JpkVXviSswHuRXSBjJd+XSdQ9SNzmakDk3qBsOrK4FK0nA1tV59g9PAClhKTpeOd1HfkQRz3sTJOVpG/9aGule7bfudCdYVLVsSuhpaMBLbiXM+FdbNcD3ldVgBSW5hQNIMD0NV0CE41AxsTmXCyjhzjFYxB/lXL8cvMQVDnqEFDnCheKXjzZkbHhQsSMoVX/MjgWetDnq8kpqq6HlD/RM3zavsAXPEo0uohHDtT2sC6dgRQ+ciFsRxEuUzpLOPi3fCvksZ1+BraxCf15miBPS3MFMLMur1PZ3gfE/BgV2G81WJvDONAh7P1zWtKBQY3v7s61twblHGtkZAoE7KGwWY1T7X4UlZBeCGoCzmbaohr6oI6aRI8jY+XJJhBa+qm5nh6F1kvXRVdoRxN/coA7bUImi5CqvOpDphDYQ6/KZLP/tPNvwUUToRxuOHmbPFSrK4A6ILRq4M++UxjwuXmNr4RWdeln1Il7ZhYSybF1XE+pF1PnIoccPyyWq7zmH1TIdnyuir+4rqYpfHvz1fjSYLPrnJyrQ0OcwtLw1kZvT61ES1X3qv7YLrpYIJNVa0w/i24bhRxc/Z22iE/OLWSfVAvm1fgGAjyMaebuXJGSlOLhr17JbG+WtdYBtTYBdCUS6XFEZwBHbL3Fj2xu0ARLsRnMaVMm8fw7ks/JfJtDP5VykyBQL9N8Qz9i4ycy8dYyDPrX+NQJUq246Wr7rJ3HdEDhZayl3504dAqbgqEI/diu4LUNW3FvW+13AiovY5v1qiay36AKUzXObNw5cyotM9I0sqGjYu3kzK4JIH78iBJCts5ZctgHh29Ze0hdkE8DtJoDBwGR/9MUKzgSP6JZs7hSfhU83xogtC3g0K8nwCenweNMhD8Pa12WIGTzJTOR9a /WaO+a5K 14iwq5gav8jamU2pRMkaQGG3bITc82jSPOZH2/WjdbUDnkjYLeays0BLeweC3jib+59SrG3MW1RmBn0nUf8rZIQrYY3Tza1GISsZjXaLqwp4sYlgb8Vmr2YF0nrmCbD+UlXsDUpQGkI4r5KCGDk+3unSh8+qwVKtjBqarEYbz3PN1nDyZYQJdfOcKNXj/1Cz6ONKB/8cOCuJpgz7X0gfGjdGClHkQwLZFQkfzHrwdqQxZP/BS6kNHtmvsdx6WqQG6vpJZ4yj2p6B6ti3LA3uag9GJYMovxGt29tGB8W+Myave3Uqlk4WR6vXvMajvVaXc9mobaaLs/euTWJErpWVlHxKsQF6BoK2Z5LRc/uOZ/HCqZJ+dQgPMDO0UPaozUQ5rcpQaEf34H6lJPCxL86c9Cq3R16dRu2cts5aQHO129QMrn/SYIAZvo0+9mYz8SbBC/zdHVP1C3MJAvcK1/XHj6+cCSNPodv78HzGaAwgTjO9CtkKhXnKiEut3sdvWiYfIHe4x7jnzyq9P7VDQXiaE9/87hOrAPhBRCNP2Kx1a6Hrgayfk79Y1RoFwbHQZJ/YFjjYKnqEQzn++J+7ykyDRQ1cUXg+da68+vPtJ X-Bogosity: Ham, tests=bogofilter, spamicity=0.000000, version=1.2.4 Sender: owner-linux-mm@kvack.org Precedence: bulk X-Loop: owner-majordomo@kvack.org List-ID: On a 16-socket 192-core POWER8 system, the context_switch1_threads benchmark from will-it-scale (see earlier changelog), upstream can achieve a rate of about 1 million context switches per second, due to contention on the mm refcount. 64s meets the prerequisites for CONFIG_MMU_LAZY_TLB_SHOOTDOWN, so enable the option. This increases the above benchmark to 118 million context switches per second. This generates 314 additional IPI interrupts on a 144 CPU system doing a kernel compile, which is in the noise in terms of kernel cycles. Acked-by: Linus Torvalds Signed-off-by: Nicholas Piggin --- arch/powerpc/Kconfig | 1 + 1 file changed, 1 insertion(+) diff --git a/arch/powerpc/Kconfig b/arch/powerpc/Kconfig index b8c4ac56bddc..600ace5a7f1a 100644 --- a/arch/powerpc/Kconfig +++ b/arch/powerpc/Kconfig @@ -265,6 +265,7 @@ config PPC select MMU_GATHER_PAGE_SIZE select MMU_GATHER_RCU_TABLE_FREE select MMU_GATHER_MERGE_VMAS + select MMU_LAZY_TLB_SHOOTDOWN if PPC_BOOK3S_64 select MODULES_USE_ELF_RELA select NEED_DMA_MAP_STATE if PPC64 || NOT_COHERENT_CACHE select NEED_PER_CPU_EMBED_FIRST_CHUNK if PPC64