From patchwork Mon Dec 23 09:40:56 2024 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Qi Zheng X-Patchwork-Id: 13918675 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from kanga.kvack.org (kanga.kvack.org [205.233.56.17]) by smtp.lore.kernel.org (Postfix) with ESMTP id 397DFE7718B for ; Mon, 23 Dec 2024 09:45:15 +0000 (UTC) Received: by kanga.kvack.org (Postfix) id C88DC6B0092; Mon, 23 Dec 2024 04:45:14 -0500 (EST) Received: by kanga.kvack.org (Postfix, from userid 40) id C386C6B0095; Mon, 23 Dec 2024 04:45:14 -0500 (EST) X-Delivered-To: int-list-linux-mm@kvack.org Received: by kanga.kvack.org (Postfix, from userid 63042) id AB3446B00A3; Mon, 23 Dec 2024 04:45:14 -0500 (EST) X-Delivered-To: linux-mm@kvack.org Received: from relay.hostedemail.com (smtprelay0016.hostedemail.com [216.40.44.16]) by kanga.kvack.org (Postfix) with ESMTP id 8D0FF6B0092 for ; Mon, 23 Dec 2024 04:45:14 -0500 (EST) Received: from smtpin28.hostedemail.com (a10.router.float.18 [10.200.18.1]) by unirelay04.hostedemail.com (Postfix) with ESMTP id 500801A111B for ; Mon, 23 Dec 2024 09:45:14 +0000 (UTC) X-FDA: 82925740128.28.29770AF Received: from mail-pf1-f181.google.com (mail-pf1-f181.google.com [209.85.210.181]) by imf08.hostedemail.com (Postfix) with ESMTP id 3594E160013 for ; Mon, 23 Dec 2024 09:44:46 +0000 (UTC) Authentication-Results: imf08.hostedemail.com; dkim=pass header.d=bytedance.com header.s=google header.b=l4gSxPRh; dmarc=pass (policy=quarantine) header.from=bytedance.com; spf=pass (imf08.hostedemail.com: domain of zhengqi.arch@bytedance.com designates 209.85.210.181 as permitted sender) smtp.mailfrom=zhengqi.arch@bytedance.com ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=hostedemail.com; s=arc-20220608; t=1734947095; h=from:from:sender:reply-to:subject:subject:date:date: message-id:message-id:to:to:cc:cc:mime-version:mime-version: content-type:content-transfer-encoding:content-transfer-encoding: in-reply-to:in-reply-to:references:references:dkim-signature; bh=76YtbImY0jPwdXnQQVs9oYKl35yZZ81uc4A2YpilpOk=; b=qq4l+5Qn432t+J3AZZfr3yOOT8rh76isOc6aLGOpKeBE11XiViO2MTVP0i/3Ej1Ncw6TL5 YJNDqPmSffWV6DLLX5iWN7BIvza/OCSLU07Uy282f7yIbBRqEZ2fUX7fgHjbjmPjtAhilH HKuzPZp5RA7hxG+WIOSPrlu4sXU6ksI= ARC-Seal: i=1; s=arc-20220608; d=hostedemail.com; t=1734947095; a=rsa-sha256; cv=none; b=r8wto/U6uFCF9I+2L0uyHnXlldK4LwCbrh4xsQQKGCaDV/iHPg3wqJi4zy/pYjKvKd9R2X q3CKbZxBrbC2paP6cqHRIm4+Q0CheGrYDkV29fP+m6Xt/rGSuH2A5Bi6Lxc1ML4yCDPRHu 79MjxLVoAr17uN0OuIAF2xnHrrnrsCE= ARC-Authentication-Results: i=1; imf08.hostedemail.com; dkim=pass header.d=bytedance.com header.s=google header.b=l4gSxPRh; dmarc=pass (policy=quarantine) header.from=bytedance.com; spf=pass (imf08.hostedemail.com: domain of zhengqi.arch@bytedance.com designates 209.85.210.181 as permitted sender) smtp.mailfrom=zhengqi.arch@bytedance.com Received: by mail-pf1-f181.google.com with SMTP id d2e1a72fcca58-725dc290c00so3781373b3a.0 for ; Mon, 23 Dec 2024 01:45:12 -0800 (PST) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=bytedance.com; s=google; t=1734947111; x=1735551911; darn=kvack.org; h=content-transfer-encoding:mime-version:references:in-reply-to :message-id:date:subject:cc:to:from:from:to:cc:subject:date :message-id:reply-to; bh=76YtbImY0jPwdXnQQVs9oYKl35yZZ81uc4A2YpilpOk=; b=l4gSxPRhZKC8nb3zwN5DsmqLMtJUCtiifkAUBdhfApgInNdbRtx6nMDE+afeUChcMN uuk8AtAiWvyM6xBr5Zlfa/8pg+frfCobNGa8mL4pWF44Rj1sX+Begs4BHGNH5YRwchqG b84+pRN1RTaV/za5UeXwnMe+bLHYg+M/XtMQRq/MtDFgZQHzd7+VveqzrXQcJYRYKQ5q LG4FgEJZfZtu5+NJiIHJlYIrIsiLzvtMmtUg9921ElBskbDPaXwhHswuG78RGgzglFr5 oPMU8swoCC0MrsK1wZaQDve1IOnR2EkUCkdSSDOxP+Epwf9VWvHsVQyBwYO8AFQZyjLC Ns6A== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20230601; t=1734947111; x=1735551911; h=content-transfer-encoding:mime-version:references:in-reply-to :message-id:date:subject:cc:to:from:x-gm-message-state:from:to:cc :subject:date:message-id:reply-to; bh=76YtbImY0jPwdXnQQVs9oYKl35yZZ81uc4A2YpilpOk=; b=KCfvq7u+ml+tZtJfXPMPV2BuzgXaMbw2fkk/9BaNX9AnKEX2/pUokvpB9NTrKd9v1n tCcCGQO3i3A/tXAbaPg8f5BTqonZ6X5siP7iJZmbQnGje0Lk9BoNj5jsBmCxBDyq9fS2 7sMbGsXwPAS/Of/amWRFYnz16cBHRcEXseNjRWXhIJQBSc9+ENRC+BuiDlLSpzCsS5/t EWM5Ys3z/FvhJmhl7jb2B0YRnxTmXGrDcXZPwxp4McUmh/7K8cTDym21jVV5Xt+9ah6w 99py6rPL9e9nSyfyrZbKJL1oaVoezUX392uqpd4Z2wBsqq0SBsOzcMpfR4r0y/7b4Mbi a0sA== X-Gm-Message-State: AOJu0YwpB90Y1ffPuPz0SkzRKPJxK92jAbPIR5e6+141/1KVFlqxi04N +D6AyX5lDxHv84ZU9Jy1WqTbBQJqaKtBSe+L4456VwH953RI9Pbqgd7jMPptCj4= X-Gm-Gg: ASbGnctPj4HWOz8tcDjndMShM+h4uix7haspTN3oTjPzdzyYRJaI6OCfurHXjPgyorY FJWJ7JgEnU/qpNdg8Zn6puuGpGmibBBJSwp+pJfEF5lUw+qaYRom99sAethsKJpyVVJaNxIsj/p IVYqRchX8IR8FDAFKZg5JkXZE0iP4JyCtzkQ6WEO5cK1QKq29AJY9Ti1ReEF/rtuSO1Kw2b1x5n mNFyvWXo0VB9+EpvRMf3pYxPcd7pdbh99/GYu7e9smE4gHyE0rwZqDAG1iMi9ia6gBMpHyTAkbI I0OReLH/P7m91fyJU+mW/A== X-Google-Smtp-Source: AGHT+IFO8x3oURIM+5JbbGjULQ8gSrXSXvX9ZJiJMgilE8gLJBNqFvKuKmavbyH4ASOcC8EYHQP7xQ== X-Received: by 2002:a05:6a20:1593:b0:1e1:ffec:b1a9 with SMTP id adf61e73a8af0-1e5c6ec6f11mr24851906637.3.1734947111156; Mon, 23 Dec 2024 01:45:11 -0800 (PST) Received: from C02DW0BEMD6R.bytedance.net ([203.208.167.150]) by smtp.gmail.com with ESMTPSA id d2e1a72fcca58-72aad8486c6sm7468309b3a.85.2024.12.23.01.44.58 (version=TLS1_3 cipher=TLS_AES_256_GCM_SHA384 bits=256/256); Mon, 23 Dec 2024 01:45:10 -0800 (PST) From: Qi Zheng To: peterz@infradead.org, agordeev@linux.ibm.com, kevin.brodsky@arm.com, tglx@linutronix.de, david@redhat.com, jannh@google.com, hughd@google.com, yuzhao@google.com, willy@infradead.org, muchun.song@linux.dev, vbabka@kernel.org, lorenzo.stoakes@oracle.com, akpm@linux-foundation.org, rientjes@google.com, vishal.moola@gmail.com, arnd@arndb.de, will@kernel.org, aneesh.kumar@kernel.org, npiggin@gmail.com, dave.hansen@linux.intel.com, rppt@kernel.org, ryan.roberts@arm.com Cc: linux-mm@kvack.org, linux-arm-kernel@lists.infradead.org, linuxppc-dev@lists.ozlabs.org, linux-riscv@lists.infradead.org, linux-s390@vger.kernel.org, sparclinux@vger.kernel.org, linux-kernel@vger.kernel.org, x86@kernel.org, linux-arch@vger.kernel.org, linux-csky@vger.kernel.org, linux-hexagon@vger.kernel.org, loongarch@lists.linux.dev, linux-m68k@lists.linux-m68k.org, linux-mips@vger.kernel.org, linux-openrisc@vger.kernel.org, linux-sh@vger.kernel.org, linux-um@lists.infradead.org, Qi Zheng Subject: [PATCH v3 10/17] riscv: pgtable: move pagetable_dtor() to __tlb_remove_table() Date: Mon, 23 Dec 2024 17:40:56 +0800 Message-Id: <0e8f0b3835c15e99145e0006ac1020ae45a2b166.1734945104.git.zhengqi.arch@bytedance.com> X-Mailer: git-send-email 2.24.3 (Apple Git-128) In-Reply-To: References: MIME-Version: 1.0 X-Stat-Signature: gupgethoicbsx7hbamzkuaxxpdgrcgdo X-Rspamd-Queue-Id: 3594E160013 X-Rspam-User: X-Rspamd-Server: rspam01 X-HE-Tag: 1734947086-228854 X-HE-Meta: U2FsdGVkX18MzK12J9c4wT8A9afcgngn1JukSuhboZR3Tm3WiYjPrarL6jI4XM1nnNFWUEEe98vv3Er8K5+ye0HUV483mNfojyHDpxs6oU8Uaws6vkatMN5aWn4UAhLUr8PlEm61stgFJlYsHGH1DmcBFBlsbE9o8vO4ZJq0WKmyxOVMDw5fCcWi6eadhDnb6TRNRskYXEhHcLWMU2QU7tkJfSDs/aWVEUXLBYKhBJANU6wOgHidUpaFIR3Ots767b3k9Y3Vtu67qqBgjI509/9jsAdlc7X1r+QXVPdUDNA23KXLMINw+OfXm4G6MCGEfMto3pW44uzaH/cUpWYJ/K0e+2+2kHqkZdBsqrC1R4NRYwTZKa/peAQbskId/0d5AM/0EUtHiTWmK3rmyQpMkMFcL/kSY32/azRHv9ZonQGgv0FNd6OCxWmbZsHb7y6ciq5oCYQo3S+7AyTw6rR+xDHMQaadN6gMKsakBqSZgO2BlUXU4EUeI+Ui3ASz/IA01qFCe/5F6ODDqwUZ4NFmApdoCV79wCehu0DJIoFsuyRdeQ461Sgk6iOJOG+/7apJ81Me2svzuAFagR1X30jRxyvVfoQ/y7ZQH4UB+KruqkB2ScguLU65XLg6g7jBoU+cXPCimA9gFA9FnMtzNtL8jltzqybNwHkNU0uL2ha2olZk5ohvqp+sM5oBl7bE5C4pTfOx1STF3k5d66y+9WXXHCHZ3HeESqrEu7htOotzzD1AwDEsttXuJijQt0gFtNaTaENeu52zWuaTsFeUnQP7U9DxL1vF/keo8NovLO+F0MoZU1hgfi3ClYUrQwKmVXPGhQT173G5IPwQhYL1TcoUElJXqwks5GPIkdaDfOLKsRuzjJGYxSTMcdqSUt4NHABCYpNYXWyT88Q7mKFnIIOzCaptpo0WT0/C8jZNjBAfWXeOhnMZdsLjMYud+lYB5jP/iSFohoaIzdnPt3tdLxv asJDbl76 jKd8gWdgTGj0lGEEHO+kTgVpide0PEmTRVBlA787BJQ3UWASLGR94lrziF/b3e4rWeJ2B4L3AmDxxQK5/d+Ull6w8z5OLxY6inwlnSeARSl2I1CxwL86hFW16zwtEOaUegnYd9SMqAeOi5QJdESmlJViGqLZ8kHZ98YoWnXuJVdeVat1lNkA6WvN/XT6zsXiIGiL/7UW7bYSIr9nB0SMT4rilCbEdJF1+of6wgj79tr5XmL5H3lqdi/c2qO6nVCYYA9DEgjGw2+Nm7yHJLbQ43PfIEI+w1NgT/F+S53iDUMNdBm7Lw1Va/9BHTP5vqadODdCAl1g7oPh36Ceade+o45YnWHH7qW4NqCGkVDWSzi0+Gl7GjMe1zA+GuZXqR5WkBanjlI5l2+LhkQg5JEx8vHs1xEas+T7nczEG1boX23zdIFrNqlrhlpWvRU1drBqI4Zh3cFgkdcMxyBUHLcVlrOROM+XjFK7ARpo0FwVSSyITZpk= X-Bogosity: Ham, tests=bogofilter, spamicity=0.000001, version=1.2.4 Sender: owner-linux-mm@kvack.org Precedence: bulk X-Loop: owner-majordomo@kvack.org List-ID: List-Subscribe: List-Unsubscribe: Move pagetable_dtor() to __tlb_remove_table(), so that ptlock and page table pages can be freed together (regardless of whether RCU is used). This prevents the use-after-free problem where the ptlock is freed immediately but the page table pages is freed later via RCU. Page tables shouldn't have swap cache, so use pagetable_free() instead of free_page_and_swap_cache() to free page table pages. By the way, move the comment above __tlb_remove_table() to riscv_tlb_remove_ptdesc(), it will be more appropriate. Signed-off-by: Qi Zheng Suggested-by: Peter Zijlstra (Intel) Cc: linux-riscv@lists.infradead.org --- arch/riscv/include/asm/pgalloc.h | 38 ++++++++++++++------------------ arch/riscv/include/asm/tlb.h | 14 ++++-------- 2 files changed, 21 insertions(+), 31 deletions(-) diff --git a/arch/riscv/include/asm/pgalloc.h b/arch/riscv/include/asm/pgalloc.h index b6793c5c99296..c8907b8317115 100644 --- a/arch/riscv/include/asm/pgalloc.h +++ b/arch/riscv/include/asm/pgalloc.h @@ -15,12 +15,22 @@ #define __HAVE_ARCH_PUD_FREE #include +/* + * While riscv platforms with riscv_ipi_for_rfence as true require an IPI to + * perform TLB shootdown, some platforms with riscv_ipi_for_rfence as false use + * SBI to perform TLB shootdown. To keep software pagetable walkers safe in this + * case we switch to RCU based table free (MMU_GATHER_RCU_TABLE_FREE). See the + * comment below 'ifdef CONFIG_MMU_GATHER_RCU_TABLE_FREE' in include/asm-generic/tlb.h + * for more details. + */ static inline void riscv_tlb_remove_ptdesc(struct mmu_gather *tlb, void *pt) { - if (riscv_use_sbi_for_rfence()) + if (riscv_use_sbi_for_rfence()) { tlb_remove_ptdesc(tlb, pt); - else + } else { + pagetable_dtor(pt); tlb_remove_page_ptdesc(tlb, pt); + } } static inline void pmd_populate_kernel(struct mm_struct *mm, @@ -97,23 +107,15 @@ static inline void pud_free(struct mm_struct *mm, pud_t *pud) static inline void __pud_free_tlb(struct mmu_gather *tlb, pud_t *pud, unsigned long addr) { - if (pgtable_l4_enabled) { - struct ptdesc *ptdesc = virt_to_ptdesc(pud); - - pagetable_dtor(ptdesc); - riscv_tlb_remove_ptdesc(tlb, ptdesc); - } + if (pgtable_l4_enabled) + riscv_tlb_remove_ptdesc(tlb, virt_to_ptdesc(pud)); } static inline void __p4d_free_tlb(struct mmu_gather *tlb, p4d_t *p4d, unsigned long addr) { - if (pgtable_l5_enabled) { - struct ptdesc *ptdesc = virt_to_ptdesc(p4d); - - pagetable_dtor(ptdesc); + if (pgtable_l5_enabled) riscv_tlb_remove_ptdesc(tlb, virt_to_ptdesc(p4d)); - } } #endif /* __PAGETABLE_PMD_FOLDED */ @@ -142,10 +144,7 @@ static inline pgd_t *pgd_alloc(struct mm_struct *mm) static inline void __pmd_free_tlb(struct mmu_gather *tlb, pmd_t *pmd, unsigned long addr) { - struct ptdesc *ptdesc = virt_to_ptdesc(pmd); - - pagetable_dtor(ptdesc); - riscv_tlb_remove_ptdesc(tlb, ptdesc); + riscv_tlb_remove_ptdesc(tlb, virt_to_ptdesc(pmd)); } #endif /* __PAGETABLE_PMD_FOLDED */ @@ -153,10 +152,7 @@ static inline void __pmd_free_tlb(struct mmu_gather *tlb, pmd_t *pmd, static inline void __pte_free_tlb(struct mmu_gather *tlb, pgtable_t pte, unsigned long addr) { - struct ptdesc *ptdesc = page_ptdesc(pte); - - pagetable_dtor(ptdesc); - riscv_tlb_remove_ptdesc(tlb, ptdesc); + riscv_tlb_remove_ptdesc(tlb, page_ptdesc(pte)); } #endif /* CONFIG_MMU */ diff --git a/arch/riscv/include/asm/tlb.h b/arch/riscv/include/asm/tlb.h index 1f6c38420d8e0..ded8724b3c4f7 100644 --- a/arch/riscv/include/asm/tlb.h +++ b/arch/riscv/include/asm/tlb.h @@ -11,19 +11,13 @@ struct mmu_gather; static void tlb_flush(struct mmu_gather *tlb); #ifdef CONFIG_MMU -#include -/* - * While riscv platforms with riscv_ipi_for_rfence as true require an IPI to - * perform TLB shootdown, some platforms with riscv_ipi_for_rfence as false use - * SBI to perform TLB shootdown. To keep software pagetable walkers safe in this - * case we switch to RCU based table free (MMU_GATHER_RCU_TABLE_FREE). See the - * comment below 'ifdef CONFIG_MMU_GATHER_RCU_TABLE_FREE' in include/asm-generic/tlb.h - * for more details. - */ static inline void __tlb_remove_table(void *table) { - free_page_and_swap_cache(table); + struct ptdesc *ptdesc = (struct ptdesc *)table; + + pagetable_dtor(ptdesc); + pagetable_free(ptdesc); } #endif /* CONFIG_MMU */