From patchwork Fri Feb 3 07:18:34 2023 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Nicholas Piggin X-Patchwork-Id: 13127090 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from kanga.kvack.org (kanga.kvack.org [205.233.56.17]) by smtp.lore.kernel.org (Postfix) with ESMTP id 508CAC636CC for ; Fri, 3 Feb 2023 07:19:09 +0000 (UTC) Received: by kanga.kvack.org (Postfix) id C6EB96B0074; Fri, 3 Feb 2023 02:19:08 -0500 (EST) Received: by kanga.kvack.org (Postfix, from userid 40) id C1D676B0075; Fri, 3 Feb 2023 02:19:08 -0500 (EST) X-Delivered-To: int-list-linux-mm@kvack.org Received: by kanga.kvack.org (Postfix, from userid 63042) id AE4E76B0078; Fri, 3 Feb 2023 02:19:08 -0500 (EST) X-Delivered-To: linux-mm@kvack.org Received: from relay.hostedemail.com (smtprelay0015.hostedemail.com [216.40.44.15]) by kanga.kvack.org (Postfix) with ESMTP id 9FF5D6B0074 for ; Fri, 3 Feb 2023 02:19:08 -0500 (EST) Received: from smtpin19.hostedemail.com (a10.router.float.18 [10.200.18.1]) by unirelay04.hostedemail.com (Postfix) with ESMTP id 7FC621A0372 for ; Fri, 3 Feb 2023 07:19:08 +0000 (UTC) X-FDA: 80425129176.19.926BFE1 Received: from mail-pj1-f51.google.com (mail-pj1-f51.google.com [209.85.216.51]) by imf30.hostedemail.com (Postfix) with ESMTP id A64D580008 for ; Fri, 3 Feb 2023 07:19:06 +0000 (UTC) Authentication-Results: imf30.hostedemail.com; dkim=pass header.d=gmail.com header.s=20210112 header.b=RjMq26cA; spf=pass (imf30.hostedemail.com: domain of npiggin@gmail.com designates 209.85.216.51 as permitted sender) smtp.mailfrom=npiggin@gmail.com; dmarc=pass (policy=none) header.from=gmail.com ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=hostedemail.com; s=arc-20220608; t=1675408746; h=from:from:sender:reply-to:subject:subject:date:date: message-id:message-id:to:to:cc:cc:mime-version:mime-version: content-type:content-transfer-encoding:content-transfer-encoding: in-reply-to:in-reply-to:references:references:dkim-signature; bh=5zyO0Na48QxjwG/96vzDf++z0O8XkPhrocrCmdAQIr4=; b=K6L6VRcDcu1l7zDkrhPmQdQ/Xm1tclL/C8Ms788IQkjk8kH48Bxlx2Om3hWdmqFrkncb0A fFlOZsyDprdN1GESzGX7klZSRSD+Da9BMUl0Ll0affXbs7OWFpBpEhPh+HJZWqBh5dCWmF 0koVl6vu3Qtog8CLmVHfL8YvGebSVUs= ARC-Authentication-Results: i=1; imf30.hostedemail.com; dkim=pass header.d=gmail.com header.s=20210112 header.b=RjMq26cA; spf=pass (imf30.hostedemail.com: domain of npiggin@gmail.com designates 209.85.216.51 as permitted sender) smtp.mailfrom=npiggin@gmail.com; dmarc=pass (policy=none) header.from=gmail.com ARC-Seal: i=1; s=arc-20220608; d=hostedemail.com; t=1675408746; a=rsa-sha256; cv=none; b=ngsBbwZ8G0Thl2Ty1c7gG7ZDhxu2hoUxW8L3mllP/0LWiWEJzMCHs1+h/PUOYy3or3+bco rkxejb7rWrz/0OQaiFP3k2SmuS4v3INYXhbq+7WeBhm9mIQGhLnFUbO20EXDWI648W3LAe Bi2/v0Bj0Ij6DCWJ6c1G7UVZvNrBDyA= Received: by mail-pj1-f51.google.com with SMTP id bx22so1283502pjb.3 for ; Thu, 02 Feb 2023 23:19:06 -0800 (PST) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=20210112; h=content-transfer-encoding:mime-version:references:in-reply-to :message-id:date:subject:cc:to:from:from:to:cc:subject:date :message-id:reply-to; bh=5zyO0Na48QxjwG/96vzDf++z0O8XkPhrocrCmdAQIr4=; b=RjMq26cALu6UQb/DQZXQLPnJ8q+5dxPkkAeNEEvpF3YmP/iIxIovFGviX1JlRz0jeg M7qLKRmW/kcc5x1c9SHmOSYnJe9+31+D24kVK96Ksr1S7Zt/aXoepztE4675W5+TKCmI l7ZXGa+Hkrlr29qD2BElEYcG56srSLiqGLTXdDhjWEbhTeezj+o3ucEl/DnGScKqEdos gjnHqOBFBvOGIB8B0ihoLS1aLm2EEzjTn2NTiSQLlM02bDjGVFKn0ywDHx4b4Xqh8fdJ /YT8fJBKv4a5hmE07ZK5OFW3fBPnCb8a4hDphqWN5QDzyLQrPez4UtETRdkzThwbDqBE bt7g== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20210112; h=content-transfer-encoding:mime-version:references:in-reply-to :message-id:date:subject:cc:to:from:x-gm-message-state:from:to:cc :subject:date:message-id:reply-to; bh=5zyO0Na48QxjwG/96vzDf++z0O8XkPhrocrCmdAQIr4=; b=a6LLG0swK9FDbxTAw+QUYILpIsf8DY6i/3iuVQQtaJjlCfArcCuIzLF9qRr7ayPCmo wHc+CmdE64I6TqYp1lTvEVVjHn/DvljPmE8J7p9EF2ye+1WNcMfNLT4c2BGLm8q2izFw 6Z/B9bPvnOce5fCvfx1S1VmTRUH2OMP/q9TrpDdnWn3oPXYbziPK8WDjNtpm/r2HyVfn hV+4lEPs57Z1qmG9p9g8T6+Ef235Yug2t363+OZu6K1f5DuouEjiHoPGOWPoWr0krRi5 /bZ9F95BfKqxuQbcP8SQRIvgrQnjzmhlR9QkZetcj2DO0PNSNl1T7Pq6eDOWq2SIX+Cb UhRA== X-Gm-Message-State: AO0yUKW0+UZcDDppQGDQwKbc9XH5WfGLpTt/WxTCCnuGIWdHGaE29AML Q8KCEzjpNjVjacrXAiqACuY= X-Google-Smtp-Source: AK7set+0jg4e6z5KWbhW7fRuU0ZT2jMMbq5RDBzo5fVKlfYThFxG8WOLHSndBW6UWk3y/w+dt7g5fQ== X-Received: by 2002:a05:6a20:4428:b0:bd:278:f68f with SMTP id ce40-20020a056a20442800b000bd0278f68fmr11365775pzb.52.1675408745589; Thu, 02 Feb 2023 23:19:05 -0800 (PST) Received: from bobo.ibm.com (193-116-117-77.tpgi.com.au. [193.116.117.77]) by smtp.gmail.com with ESMTPSA id f20-20020a637554000000b004df4ba1ebfesm877558pgn.66.2023.02.02.23.19.00 (version=TLS1_3 cipher=TLS_AES_256_GCM_SHA384 bits=256/256); Thu, 02 Feb 2023 23:19:05 -0800 (PST) From: Nicholas Piggin To: Andrew Morton Cc: Nicholas Piggin , Linus Torvalds , Nadav Amit , Dave Hansen , Andy Lutomirski , Peter Zijlstra , Catalin Marinas , Will Deacon , Rik van Riel , linux-arch@vger.kernel.org, linux-mm@kvack.org, linuxppc-dev@lists.ozlabs.org Subject: [PATCH v7 2/5] lazy tlb: introduce lazy tlb mm refcount helper functions Date: Fri, 3 Feb 2023 17:18:34 +1000 Message-Id: <20230203071837.1136453-3-npiggin@gmail.com> X-Mailer: git-send-email 2.37.2 In-Reply-To: <20230203071837.1136453-1-npiggin@gmail.com> References: <20230203071837.1136453-1-npiggin@gmail.com> MIME-Version: 1.0 X-Rspamd-Server: rspam05 X-Rspamd-Queue-Id: A64D580008 X-Stat-Signature: iugc79yg6sstpdsfdomdpc5tkn9xu5xd X-Rspam-User: X-HE-Tag: 1675408746-382211 X-HE-Meta: U2FsdGVkX1/7TWXvv9WD5UmDiZiutLMjJVt8/6prg4mi0CGZoAIkD+mSjCtm/mcJ6yE94IB8u85K6Ucxp3xRMEY7VzDsa/Gw47Eipx/wWasd/V8E+Qpo01PLDW9iQe2TxXDTmXvU4g0wMuvG7zStkZbtHZ8jYnEPi1z/1laaPWhjNFxzmN162UPLOMWZ4tqt7qZaA1NoFXE/Gax0ng833Q8gMT/U9tHmT+6YMuH8ucqh8FWmHvzGVFGbWBCyhfU0JGeBUvXy8QNkPDKcMKg8P8WQuuPyGjFs6Jv3BeWXUcObsOKdS64IzD0T99pi3lqmdPpQoL9egjeSPvqsusOcVw7uENjq3JkiQIgVU2+E3pw3HLyNX4y65UycNXbfrS9wCs5ETa8LQkPcTAX3WZw264Ae2nTMNVHII1wTG3rDi7Q4BkvZfB2F/ipAOmcQ08svjREx4llaQZ3vOrSSXwZGnha8EcxAMobs+4wqh6Oi2g7im8KiX30f+fvvlCWEixPg0DO+vSbDrNVRT0heOl36xhW+XkTXbFwf5qSyIVGN9pPEV5ZhqdXcuev7iOv0A0+zduG+ltd6ub3zF//3R6KTmxHhyXoY1XXfsXSatJMzwiUnRlZ3HordFRCYjMCCCoH6Eu7vQnKB7aCMFnWw5YjMG69YVvm9UQddJkzqsCK9njo+nLv001B4pLFeNzIG6y2+6ZpdN/+digdHU8HqUuer8RM0acShyw1e3a64LkDcECC1AkpY6vpSl8TFwf3MYqdbMoeIIGHjVc4GrIvvCyBwJzj6s0aHL7UT7KNlLnU3h4bV4Pm/Df0giW7tc+xTLA8nnF8h4a7riv+khXtq1ax7+BtrreFvH/7f/QmsZAaslgZkhnTPIN6Ijl6iz1leoz1MZk5CeJkTxgdDMRCqYQFc0p6eULhn3qRM4W1tV2bMxMORDsC8+YBXK/RdY6IgePA+mx/ucNafymOe+yF2tDx 8fIOaeRC EH12OtCHrBzkPpJ/RC5F+SzaXn5JgPgEkHXRWXZFzdpIrWvqJItXBappoT93W9C2JJd99uoDugGR5NFZ77fDomcAdAJZeWJYBN1iJmujwwCLwz014rl8ELDY/L+hHKkWEGQiU1oPKfoHOCZwXIL81UV908Uf9mXw0688YVtvB/4TriH1LFqTtQqQ1X/YGbieUmMbdTe6+xTswnMgw0VxU52BDzdp44pxlqojj7cMT+LPrn1gsmHjCsNw4eyURdRC2jHk/EgLtoogkBv2tQl8x6SHSnxZIb/WdQTM9fDBqaHv2Ixo/24lVObjEK8LXGmr8UnSyWnWor2pkTOhOX1ybcksyqXf9S1SP6gQBvbfiVoargPv6ovjKgwwmqT+wJSUrOdxg8e0vLUmPVpeUBTogJk8CDPiWED2FEUagHdR4PFj8l87gfAZYhwksdogYwUAuGtppdJ+a7MMDKpxij9VktasFWbgvIf33jj95qTOQVtrZt2buEPSp+w+HNbyJeR1TuyRhWJHVZdyY0ZrAbteO3v87MosJI05KVSGJedovCWJVqjwLYRBOjQf2wHq8suGQDh4kVulXiijq1NQfDFcDREppDw== X-Bogosity: Ham, tests=bogofilter, spamicity=0.000000, version=1.2.4 Sender: owner-linux-mm@kvack.org Precedence: bulk X-Loop: owner-majordomo@kvack.org List-ID: Add explicit _lazy_tlb annotated functions for lazy tlb mm refcounting. This makes the lazy tlb mm references more obvious, and allows the refcounting scheme to be modified in later changes. There is no functional change with this patch. Acked-by: Linus Torvalds Signed-off-by: Nicholas Piggin --- arch/arm/mach-rpc/ecard.c | 2 +- arch/powerpc/kernel/smp.c | 2 +- arch/powerpc/mm/book3s64/radix_tlb.c | 4 ++-- fs/exec.c | 2 +- include/linux/sched/mm.h | 16 ++++++++++++++++ kernel/cpu.c | 2 +- kernel/exit.c | 2 +- kernel/kthread.c | 12 ++++++++++-- kernel/sched/core.c | 15 ++++++++------- 9 files changed, 41 insertions(+), 16 deletions(-) diff --git a/arch/arm/mach-rpc/ecard.c b/arch/arm/mach-rpc/ecard.c index 53813f9464a2..c30df1097c52 100644 --- a/arch/arm/mach-rpc/ecard.c +++ b/arch/arm/mach-rpc/ecard.c @@ -253,7 +253,7 @@ static int ecard_init_mm(void) current->mm = mm; current->active_mm = mm; activate_mm(active_mm, mm); - mmdrop(active_mm); + mmdrop_lazy_tlb(active_mm); ecard_init_pgtables(mm); return 0; } diff --git a/arch/powerpc/kernel/smp.c b/arch/powerpc/kernel/smp.c index 6b90f10a6c81..7db6b3faea65 100644 --- a/arch/powerpc/kernel/smp.c +++ b/arch/powerpc/kernel/smp.c @@ -1611,7 +1611,7 @@ void start_secondary(void *unused) if (IS_ENABLED(CONFIG_PPC32)) setup_kup(); - mmgrab(&init_mm); + mmgrab_lazy_tlb(&init_mm); current->active_mm = &init_mm; smp_store_cpu_info(cpu); diff --git a/arch/powerpc/mm/book3s64/radix_tlb.c b/arch/powerpc/mm/book3s64/radix_tlb.c index 4e29b619578c..282359ab525b 100644 --- a/arch/powerpc/mm/book3s64/radix_tlb.c +++ b/arch/powerpc/mm/book3s64/radix_tlb.c @@ -794,10 +794,10 @@ void exit_lazy_flush_tlb(struct mm_struct *mm, bool always_flush) if (current->active_mm == mm) { WARN_ON_ONCE(current->mm != NULL); /* Is a kernel thread and is using mm as the lazy tlb */ - mmgrab(&init_mm); + mmgrab_lazy_tlb(&init_mm); current->active_mm = &init_mm; switch_mm_irqs_off(mm, &init_mm, current); - mmdrop(mm); + mmdrop_lazy_tlb(mm); } /* diff --git a/fs/exec.c b/fs/exec.c index ab913243a367..1a32a88db173 100644 --- a/fs/exec.c +++ b/fs/exec.c @@ -1033,7 +1033,7 @@ static int exec_mmap(struct mm_struct *mm) mmput(old_mm); return 0; } - mmdrop(active_mm); + mmdrop_lazy_tlb(active_mm); return 0; } diff --git a/include/linux/sched/mm.h b/include/linux/sched/mm.h index 2a243616f222..5376caf6fcf3 100644 --- a/include/linux/sched/mm.h +++ b/include/linux/sched/mm.h @@ -79,6 +79,22 @@ static inline void mmdrop_sched(struct mm_struct *mm) } #endif +/* Helpers for lazy TLB mm refcounting */ +static inline void mmgrab_lazy_tlb(struct mm_struct *mm) +{ + mmgrab(mm); +} + +static inline void mmdrop_lazy_tlb(struct mm_struct *mm) +{ + mmdrop(mm); +} + +static inline void mmdrop_lazy_tlb_sched(struct mm_struct *mm) +{ + mmdrop_sched(mm); +} + /** * mmget() - Pin the address space associated with a &struct mm_struct. * @mm: The address space to pin. diff --git a/kernel/cpu.c b/kernel/cpu.c index 6c0a92ca6bb5..189895288d9d 100644 --- a/kernel/cpu.c +++ b/kernel/cpu.c @@ -623,7 +623,7 @@ static int finish_cpu(unsigned int cpu) */ if (mm != &init_mm) idle->active_mm = &init_mm; - mmdrop(mm); + mmdrop_lazy_tlb(mm); return 0; } diff --git a/kernel/exit.c b/kernel/exit.c index 15dc2ec80c46..1a4608d765e4 100644 --- a/kernel/exit.c +++ b/kernel/exit.c @@ -537,7 +537,7 @@ static void exit_mm(void) return; sync_mm_rss(mm); mmap_read_lock(mm); - mmgrab(mm); + mmgrab_lazy_tlb(mm); BUG_ON(mm != current->active_mm); /* more a memory barrier than a real lock */ task_lock(current); diff --git a/kernel/kthread.c b/kernel/kthread.c index 7424a1839e9a..e4bc32a88866 100644 --- a/kernel/kthread.c +++ b/kernel/kthread.c @@ -1410,6 +1410,11 @@ void kthread_use_mm(struct mm_struct *mm) WARN_ON_ONCE(!(tsk->flags & PF_KTHREAD)); WARN_ON_ONCE(tsk->mm); + /* + * It is possible for mm to be the same as tsk->active_mm, but + * we must still mmgrab(mm) and mmdrop_lazy_tlb(active_mm), + * because these references are not equivalent. + */ mmgrab(mm); task_lock(tsk); @@ -1433,9 +1438,9 @@ void kthread_use_mm(struct mm_struct *mm) * memory barrier after storing to tsk->mm, before accessing * user-space memory. A full memory barrier for membarrier * {PRIVATE,GLOBAL}_EXPEDITED is implicitly provided by - * mmdrop(). + * mmdrop_lazy_tlb(). */ - mmdrop(active_mm); + mmdrop_lazy_tlb(active_mm); } EXPORT_SYMBOL_GPL(kthread_use_mm); @@ -1463,10 +1468,13 @@ void kthread_unuse_mm(struct mm_struct *mm) local_irq_disable(); tsk->mm = NULL; membarrier_update_current_mm(NULL); + mmgrab_lazy_tlb(mm); /* active_mm is still 'mm' */ enter_lazy_tlb(mm, tsk); local_irq_enable(); task_unlock(tsk); + + mmdrop(mm); } EXPORT_SYMBOL_GPL(kthread_unuse_mm); diff --git a/kernel/sched/core.c b/kernel/sched/core.c index e838feb6adc5..495f9a021de9 100644 --- a/kernel/sched/core.c +++ b/kernel/sched/core.c @@ -5189,13 +5189,14 @@ static struct rq *finish_task_switch(struct task_struct *prev) * rq->curr, before returning to userspace, so provide them here: * * - a full memory barrier for {PRIVATE,GLOBAL}_EXPEDITED, implicitly - * provided by mmdrop(), + * provided by mmdrop_lazy_tlb(), * - a sync_core for SYNC_CORE. */ if (mm) { membarrier_mm_sync_core_before_usermode(mm); - mmdrop_sched(mm); + mmdrop_lazy_tlb_sched(mm); } + if (unlikely(prev_state == TASK_DEAD)) { if (prev->sched_class->task_dead) prev->sched_class->task_dead(prev); @@ -5252,9 +5253,9 @@ context_switch(struct rq *rq, struct task_struct *prev, /* * kernel -> kernel lazy + transfer active - * user -> kernel lazy + mmgrab() active + * user -> kernel lazy + mmgrab_lazy_tlb() active * - * kernel -> user switch + mmdrop() active + * kernel -> user switch + mmdrop_lazy_tlb() active * user -> user switch */ if (!next->mm) { // to kernel @@ -5262,7 +5263,7 @@ context_switch(struct rq *rq, struct task_struct *prev, next->active_mm = prev->active_mm; if (prev->mm) // from user - mmgrab(prev->active_mm); + mmgrab_lazy_tlb(prev->active_mm); else prev->active_mm = NULL; } else { // to user @@ -5279,7 +5280,7 @@ context_switch(struct rq *rq, struct task_struct *prev, lru_gen_use_mm(next->mm); if (!prev->mm) { // from kernel - /* will mmdrop() in finish_task_switch(). */ + /* will mmdrop_lazy_tlb() in finish_task_switch(). */ rq->prev_mm = prev->active_mm; prev->active_mm = NULL; } @@ -9916,7 +9917,7 @@ void __init sched_init(void) /* * The boot idle thread does lazy MMU switching as well: */ - mmgrab(&init_mm); + mmgrab_lazy_tlb(&init_mm); enter_lazy_tlb(&init_mm, current); /*