From patchwork Thu Feb 29 23:21:43 2024 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Samuel Holland X-Patchwork-Id: 13577707 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from kanga.kvack.org (kanga.kvack.org [205.233.56.17]) by smtp.lore.kernel.org (Postfix) with ESMTP id E264BC54E49 for ; Thu, 29 Feb 2024 23:22:19 +0000 (UTC) Received: by kanga.kvack.org (Postfix) id A38406B0093; Thu, 29 Feb 2024 18:22:18 -0500 (EST) Received: by kanga.kvack.org (Postfix, from userid 40) id 96DB06B0096; Thu, 29 Feb 2024 18:22:18 -0500 (EST) X-Delivered-To: int-list-linux-mm@kvack.org Received: by kanga.kvack.org (Postfix, from userid 63042) id 7258C6B0098; Thu, 29 Feb 2024 18:22:18 -0500 (EST) X-Delivered-To: linux-mm@kvack.org Received: from relay.hostedemail.com (smtprelay0010.hostedemail.com [216.40.44.10]) by kanga.kvack.org (Postfix) with ESMTP id 5243C6B0093 for ; Thu, 29 Feb 2024 18:22:18 -0500 (EST) Received: from smtpin04.hostedemail.com (a10.router.float.18 [10.200.18.1]) by unirelay01.hostedemail.com (Postfix) with ESMTP id 1F1B91C1034 for ; Thu, 29 Feb 2024 23:22:18 +0000 (UTC) X-FDA: 81846417156.04.D9DB16C Received: from mail-pf1-f177.google.com (mail-pf1-f177.google.com [209.85.210.177]) by imf11.hostedemail.com (Postfix) with ESMTP id 533194001D for ; Thu, 29 Feb 2024 23:22:16 +0000 (UTC) Authentication-Results: imf11.hostedemail.com; dkim=pass header.d=sifive.com header.s=google header.b=GjHBX7vh; dmarc=pass (policy=reject) header.from=sifive.com; spf=pass (imf11.hostedemail.com: domain of samuel.holland@sifive.com designates 209.85.210.177 as permitted sender) smtp.mailfrom=samuel.holland@sifive.com ARC-Seal: i=1; s=arc-20220608; d=hostedemail.com; t=1709248936; a=rsa-sha256; cv=none; b=L7MKOFJa7Cpbu2WUOhFZh3F/ti5ZNzB7VoD6MgaKOVO+e/kDt+eMpKEbptjfvslqp0Rybv OXI7+xhBTyUfPFMaKh9aqYUV3cXwpyERqPLIvHDATAc/cOvgza83yESFZT8UfTAjW/PAw9 SUssHlQ9fFpV1vwRt3gRm7IxKEm6KzM= ARC-Authentication-Results: i=1; imf11.hostedemail.com; dkim=pass header.d=sifive.com header.s=google header.b=GjHBX7vh; dmarc=pass (policy=reject) header.from=sifive.com; spf=pass (imf11.hostedemail.com: domain of samuel.holland@sifive.com designates 209.85.210.177 as permitted sender) smtp.mailfrom=samuel.holland@sifive.com ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=hostedemail.com; s=arc-20220608; t=1709248936; h=from:from:sender:reply-to:subject:subject:date:date: message-id:message-id:to:to:cc:cc:mime-version:mime-version: content-type:content-transfer-encoding:content-transfer-encoding: in-reply-to:in-reply-to:references:references:dkim-signature; bh=XEa0bZw6X2PF+fvXkyrnyWuZOURSNgi5dGIyeM9k7ic=; b=R0EJWyIAHxFPNWFjZ3gJd4ATDIeXVSJEiynvES0HsfC0vK6ird6G77boCe0pmnWWUnYEJ9 +QQlQMaioQlgWAyNzfMBbX6JOZ168M8E3RAxtp58SfzOr1ZrlFBRxh8Nl7bGHdhMZAeMZc 8PSUjatYtnrE3M+iUjuGtX4H6XK55Mg= Received: by mail-pf1-f177.google.com with SMTP id d2e1a72fcca58-6e5a232fe80so701422b3a.0 for ; Thu, 29 Feb 2024 15:22:15 -0800 (PST) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=sifive.com; s=google; t=1709248935; x=1709853735; darn=kvack.org; h=content-transfer-encoding:mime-version:references:in-reply-to :message-id:date:subject:cc:to:from:from:to:cc:subject:date :message-id:reply-to; bh=XEa0bZw6X2PF+fvXkyrnyWuZOURSNgi5dGIyeM9k7ic=; b=GjHBX7vhDQmk3uieuGL7DUacpJPTVEn2KmhcEq1I8CRE7TWwRqIkAJHwFpFDp6771t YxmJeG/obOn4oH+I7O7bh1ESlFZYXDqzN85XXeJR0rSTVs973bsAvHvQzpVqacd8LpK+ 9a8ZhSltc9AhEFWjHW0N7cvdlx+6wa9yaglYATKFtSghVRyRi6mX8DqBtn+UBri9YQ8O 7UjfQvSoTO1RKIFxYkffV54g4xOkH87XkRqBr+Rg0GRhcEuySW8YVXdcmra01MeLw40/ lDwkjtDBJ92QRQS+Tbjg7DeaQy+rep0wrIJYHdbpc7UuJVpNZ8IHTV5G1g1R98ckVLcg VKQg== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20230601; t=1709248935; x=1709853735; h=content-transfer-encoding:mime-version:references:in-reply-to :message-id:date:subject:cc:to:from:x-gm-message-state:from:to:cc :subject:date:message-id:reply-to; bh=XEa0bZw6X2PF+fvXkyrnyWuZOURSNgi5dGIyeM9k7ic=; b=QmTqIMk83ngvLhQFWttNbzblGjNClkrh5Um+p7yqwQYCQ27llZqPtNgmqvESyMM7i4 UjQl5cx3B2RpnQE4k9KMkDFyFODKxAWbTgFLRIPinBGUZ0Z+4U8ovxIlUbIBE4rnP/jb Y4bScmPuFvugrYjWZqcxMCsQi8j/KtLrC8C38ctnQ9b3A8RU27nBJ7qenzhbJKcGzhls +oWkUQWWy5NVZCUAjyfV0BiEz7zTvHpYBTAhuHGzstzVAd02kiSN7aO4nRV0+FoZqFCm axgehkpbDI6dq+brH29DM95LXqjUE5XkRO+okGHt2NfhUyinPjLlGSNHjLHanRwp4N+0 BIvQ== X-Forwarded-Encrypted: i=1; AJvYcCWLETTs9UXYgqGqwogv0j3q/fCp33jQYhlPfCQJ67FYGhNu0AfD4Dvm78UfOdvTa3hBAfCImjkHikavBJkLikDMCWE= X-Gm-Message-State: AOJu0YwfvabG3qDz+FBB+LwMN033craegafnbHzg4y/ZOEfGJuXRgWoI MQfp1RQnHZpf20EMCOUyf0kbwC44WqGCy+Rm5QknVHzlmC1PzUUNer1ZW6LFZOw= X-Google-Smtp-Source: AGHT+IFdgcTr06rQ4rVKFKQwHiCzNjuzvF9/zHzBzMpbTCozBAylfcba6wC7aAVCdn9otyCllPR9tg== X-Received: by 2002:a05:6a00:2293:b0:6e5:599a:5252 with SMTP id f19-20020a056a00229300b006e5599a5252mr606918pfe.15.1709248935218; Thu, 29 Feb 2024 15:22:15 -0800 (PST) Received: from sw06.internal.sifive.com ([4.53.31.132]) by smtp.gmail.com with ESMTPSA id c6-20020aa78806000000b006e55aa75d6csm1779719pfo.122.2024.02.29.15.22.14 (version=TLS1_3 cipher=TLS_AES_256_GCM_SHA384 bits=256/256); Thu, 29 Feb 2024 15:22:14 -0800 (PST) From: Samuel Holland To: Palmer Dabbelt , linux-riscv@lists.infradead.org Cc: linux-kernel@vger.kernel.org, linux-mm@kvack.org, Alexandre Ghiti , Jisheng Zhang , Yunhui Cui , Samuel Holland Subject: [PATCH v5 02/13] riscv: Factor out page table TLB synchronization Date: Thu, 29 Feb 2024 15:21:43 -0800 Message-ID: <20240229232211.161961-3-samuel.holland@sifive.com> X-Mailer: git-send-email 2.43.1 In-Reply-To: <20240229232211.161961-1-samuel.holland@sifive.com> References: <20240229232211.161961-1-samuel.holland@sifive.com> MIME-Version: 1.0 X-Rspam-User: X-Rspamd-Server: rspam06 X-Rspamd-Queue-Id: 533194001D X-Stat-Signature: ttdy1d8hg4cnsus8o3rwkyrnwg4npfc3 X-HE-Tag: 1709248936-781377 X-HE-Meta: U2FsdGVkX1/irKsFpJnQoEYvzShGqH+fNhFIJKCbXSKU7IAuxnM3k2XIQCUoyEe8ARKav7ooZkPB4GvpuoAIrefr84Fro7wBs7j7pERX4Ni+Baqo+M46HGys9URkBFPOnjvrQr9FgV80pEmocU3X9NtNohcEURm7gokGjnO0wjZa5KHX0xlER7nJi/NFAvIYJuQHjoUPVewGMnHB/a+BnsxaTKeFaJmYAKkcDchbP6T2UgEtw0RVR07KPtJCJ2j76kpqwPTIZLEnuNsz/I/tvtSWtycIzHv1W561fCwcQzsy1EWb+Bqjg88EHId1EnwD1mYuEhPfj3E1OnQFy2LEplxCnJamnhNGOLFOm+fktI3Mo79TEPvc1Ara0w4u6c9YHdcXW7Px1pyHd67ulQBb6LWGxvl9ZqK6GqJlDkEyg2urVXbOu8vXsl8K/k35+t8gjBx0qW2IQ+9P4BmNGUTzL3z5zsq+FoJn88tukDg6u5IQAXPWpyScHEZrFtf7qVj/eNmNkIOuu80ZFJEmxqTAg10gWKSaAsEp3JClzrRfhEcYCpEHdqmIJcsdP2dhodBnc8+b5QOWgwsTG1PBiKkVi/M7+skwFGGyLvvQAVe0NqOrnCWvfw+520FvESRED8lIXnFcAOB0I+JOiLGxEZguAIntX6qkb/YuPEN2O5qSCbwTvRAJKgypTArK5OywZqmKCGBlMKHK6wWVN1M0iShJ8Ka8KDJM1nLzxqM6c4kzBRwpPMsH5EePt+vlAbrr0VxKn3/3WvDBJN6ElaQi/OCV85gpdVJPBRVQ/tZsjvsa4wRQTUJxi2FuYYrWFTmneBItf9WeBc4owV7Khpahmsysoz7uW4UkzPgWop341iPAnvnbnZaBEw9PDuFz2WUyl7IamHpwvuEmMDY6D4p52DNJrHVp5WyxfV43h/MF2Vx7/VdxX7zzVOJFzV51M73N/NyvWwpgbDrQ2iVPdPRqm8G FGocI7Pp hiyqqi7eUX2S+kWG3kIcd5n6naywaTYbglj6/miH/ADPnC+OyRpqLYWSWsLVhYxZuR0tqT88c0GPc7QtOZu6ADggCjtOi4aWALUOIr+munG6PD6JALkybOYtVWw== X-Bogosity: Ham, tests=bogofilter, spamicity=0.000230, version=1.2.4 Sender: owner-linux-mm@kvack.org Precedence: bulk X-Loop: owner-majordomo@kvack.org List-ID: List-Subscribe: List-Unsubscribe: The logic is the same for all page table levels. See commit 69be3fb111e7 ("riscv: enable MMU_GATHER_RCU_TABLE_FREE for SMP && MMU"). Signed-off-by: Samuel Holland --- Changes in v5: - New patch for v5 arch/riscv/include/asm/pgalloc.h | 31 +++++++++++++------------------ 1 file changed, 13 insertions(+), 18 deletions(-) diff --git a/arch/riscv/include/asm/pgalloc.h b/arch/riscv/include/asm/pgalloc.h index deaf971253a2..87468f67951a 100644 --- a/arch/riscv/include/asm/pgalloc.h +++ b/arch/riscv/include/asm/pgalloc.h @@ -88,6 +88,14 @@ static inline pud_t *pud_alloc_one(struct mm_struct *mm, unsigned long addr) return NULL; } +static inline void riscv_tlb_remove_ptdesc(struct mmu_gather *tlb, void *pt) +{ + if (riscv_use_ipi_for_rfence()) + tlb_remove_page_ptdesc(tlb, pt); + else + tlb_remove_ptdesc(tlb, pt); +} + #define pud_free pud_free static inline void pud_free(struct mm_struct *mm, pud_t *pud) { @@ -102,10 +110,7 @@ static inline void __pud_free_tlb(struct mmu_gather *tlb, pud_t *pud, struct ptdesc *ptdesc = virt_to_ptdesc(pud); pagetable_pud_dtor(ptdesc); - if (riscv_use_ipi_for_rfence()) - tlb_remove_page_ptdesc(tlb, ptdesc); - else - tlb_remove_ptdesc(tlb, ptdesc); + riscv_tlb_remove_ptdesc(tlb, ptdesc); } } @@ -139,12 +144,8 @@ static inline void p4d_free(struct mm_struct *mm, p4d_t *p4d) static inline void __p4d_free_tlb(struct mmu_gather *tlb, p4d_t *p4d, unsigned long addr) { - if (pgtable_l5_enabled) { - if (riscv_use_ipi_for_rfence()) - tlb_remove_page_ptdesc(tlb, virt_to_ptdesc(p4d)); - else - tlb_remove_ptdesc(tlb, virt_to_ptdesc(p4d)); - } + if (pgtable_l5_enabled) + riscv_tlb_remove_ptdesc(tlb, virt_to_ptdesc(p4d)); } #endif /* __PAGETABLE_PMD_FOLDED */ @@ -176,10 +177,7 @@ static inline void __pmd_free_tlb(struct mmu_gather *tlb, pmd_t *pmd, struct ptdesc *ptdesc = virt_to_ptdesc(pmd); pagetable_pmd_dtor(ptdesc); - if (riscv_use_ipi_for_rfence()) - tlb_remove_page_ptdesc(tlb, ptdesc); - else - tlb_remove_ptdesc(tlb, ptdesc); + riscv_tlb_remove_ptdesc(tlb, ptdesc); } #endif /* __PAGETABLE_PMD_FOLDED */ @@ -190,10 +188,7 @@ static inline void __pte_free_tlb(struct mmu_gather *tlb, pgtable_t pte, struct ptdesc *ptdesc = page_ptdesc(pte); pagetable_pte_dtor(ptdesc); - if (riscv_use_ipi_for_rfence()) - tlb_remove_page_ptdesc(tlb, ptdesc); - else - tlb_remove_ptdesc(tlb, ptdesc); + riscv_tlb_remove_ptdesc(tlb, ptdesc); } #endif /* CONFIG_MMU */