From patchwork Mon Dec 23 09:40:47 2024 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Qi Zheng X-Patchwork-Id: 13918700 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from bombadil.infradead.org (bombadil.infradead.org [198.137.202.133]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (No client certificate requested) by smtp.lore.kernel.org (Postfix) with ESMTPS id CD890E7718B for ; Mon, 23 Dec 2024 09:44:28 +0000 (UTC) DKIM-Signature: v=1; a=rsa-sha256; q=dns/txt; c=relaxed/relaxed; d=lists.infradead.org; s=bombadil.20210309; h=Sender: Content-Transfer-Encoding:Content-Type:List-Subscribe:List-Help:List-Post: List-Archive:List-Unsubscribe:List-Id:MIME-Version:References:In-Reply-To: Message-Id:Date:Subject:Cc:To:From:Reply-To:Content-ID:Content-Description: Resent-Date:Resent-From:Resent-Sender:Resent-To:Resent-Cc:Resent-Message-ID: List-Owner; bh=AaQA0vW9SeL7IfwbPZcCnUno84K3+V8aNEGaGgb7enw=; b=zEp0Iyy1faVT3A 41ml7hgbWtDt5/8IFCf07FtaOP7uFHekTlqDdISw+LLQ/YOc3cBMg9k7I5TMdQKwJxzx53CGUvcY1 fRRj4RmvtOGl2bPInpY5PYdAS4VZI3uaLplNHev6XLE7lfWPWH90zvbdjXSGYGHp0/B0pU267ueNm 5FnAJtGoFPd0X9rGQXYDBPD2P0XwGHKy/k8o0WVmoywhpnRm2zfzbsa5e0lLk3LdjiaUGNJIADfVW KBsJjyJRR4vI4IXtGDnR+4tfeVpgrWNwXD09U1P9jKb2I1aJvgWYgfeFArPLYkI5dmYcYU9Vv9yI0 w9vd4PdhCLb7vGs/qMnQ==; Received: from localhost ([::1] helo=bombadil.infradead.org) by bombadil.infradead.org with esmtp (Exim 4.98 #2 (Red Hat Linux)) id 1tPez9-00000009gZG-0Z7n; Mon, 23 Dec 2024 09:44:23 +0000 Received: from mail-pg1-x534.google.com ([2607:f8b0:4864:20::534]) by bombadil.infradead.org with esmtps (Exim 4.98 #2 (Red Hat Linux)) id 1tPey6-00000009g1r-29Um for linux-riscv@lists.infradead.org; Mon, 23 Dec 2024 09:43:20 +0000 Received: by mail-pg1-x534.google.com with SMTP id 41be03b00d2f7-8019338c2b2so2381096a12.3 for ; Mon, 23 Dec 2024 01:43:18 -0800 (PST) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=bytedance.com; s=google; t=1734946997; x=1735551797; darn=lists.infradead.org; h=content-transfer-encoding:mime-version:references:in-reply-to :message-id:date:subject:cc:to:from:from:to:cc:subject:date :message-id:reply-to; bh=SyzxI6H4ZAISLyWZ1lhQ49SW3wSD1SDR6GIh5AHA3wQ=; b=WUUl13y18QDSw12DqRLV7i2gVw7rLPArPIqqp2sNn603znj6op/lBKerqaxfkCfNBv vRTN7uBU2Pv59i1vtATFzQMuR6qKhQVz4SaPS5y5A0gKgoudoPHiHRIuhAXa8MMByoLr bS/fMpiZ9uxKK0oj9SG5bGVBQOtpgsaJffm6EaHBuW98XA0rb7PLZ8ZJtKtKn56h/iK7 CpH5A7E5ZqB2HG3Qtgp2YPuQ/l9ENrtJnyMDU0TMuqxk0W8/QERSjn06WoGnNz8sSvnh 8WOc+sShRmNusE7378FxayJsQoBWholP+aRP/U5rk1qUvujd3QjxxRoavMQD8KCQR/QP oK/g== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20230601; t=1734946997; x=1735551797; h=content-transfer-encoding:mime-version:references:in-reply-to :message-id:date:subject:cc:to:from:x-gm-message-state:from:to:cc :subject:date:message-id:reply-to; bh=SyzxI6H4ZAISLyWZ1lhQ49SW3wSD1SDR6GIh5AHA3wQ=; b=LfxTsnHv4uFUl2JKbCkruK23IPtd/QCEfnvynxbNlZEEdeXglAAZVsr8ZwTv5pCbSp qLN2u6LNhjRC75IdUxtD0ccu5pUiAe6EEb7a8lHLfaxidX8h4vKLA3qIOxNvtpjicZ5K v0eAjtIJzy8f0p33DS1DMNzPk549pbrvO59LKklGyjS42ydk+vBfHVZMCiUuoX31F8t5 2PQghzr26QpbDWQHbU8dW4Gri43kpwt7hNMPppEuf+VGe+DEBO99+X7epHqHlc0Lppfy qw8HNbt0HL/deKM6KbIz6lfPPK/Mwb2i5cH4QMYBjtENt27NFlMb4Z+qh4GUTLxle2Lh QkEg== X-Forwarded-Encrypted: i=1; AJvYcCVgODA/r2wg1Rb7V+7rMzAG5q9/LcaLcN427NZcV+bBu/YrUVFVI4OYhaFIxTjV9R1EqnF2cOTs+rYjSA==@lists.infradead.org X-Gm-Message-State: AOJu0Yx385xxh30eJoVF+rR0h3tbGb9/ORhID0CH50J3VZ8ya2hB4aK+ oCUKbWpieMq3218lpB8tiRxA79AwvSoMhQ4i8WZXUbYU45iWe8gZQTYWxTryeJA= X-Gm-Gg: ASbGncuoqu3ezt8GRxgxY3BKxotErta3eNvYBFMc2Ft+me9B0xovU+Fp02C5AEPZHL9 5KzioNdERsL6/q5xFRz8UDVdCVGme/zZH+NJIJNN0/J51yY/ppyFm2ynNJZYRY467FsUGgzfegc PKdqt0xRKgnhMZ9mGEt9APPareNhhT8V/bbE617XDeplJT5eyTO2YJRAxJtDVCTYhHp8SnwKvF1 49VXB/bkfMC41qgUC4C4mIlXZiPMWov4aNqQlYJiMo+cZG1KRa+sd41NeCHfkfLA5+i2trD5eSm tkRuLLnjAxLMfU9IHORWpQ== X-Google-Smtp-Source: AGHT+IETS5CqVMOZwuklYBKyw2d6q7hBKdfBjuJ9ZQ+tZcvMnTE5pZk49SgpvcM01OaAXbZV7hq4vg== X-Received: by 2002:a05:6a21:6da9:b0:1db:e338:ab0a with SMTP id adf61e73a8af0-1e5e0447f5bmr22664720637.8.1734946996090; Mon, 23 Dec 2024 01:43:16 -0800 (PST) Received: from C02DW0BEMD6R.bytedance.net ([203.208.167.150]) by smtp.gmail.com with ESMTPSA id d2e1a72fcca58-72aad8486c6sm7468309b3a.85.2024.12.23.01.43.03 (version=TLS1_3 cipher=TLS_AES_256_GCM_SHA384 bits=256/256); Mon, 23 Dec 2024 01:43:15 -0800 (PST) From: Qi Zheng To: peterz@infradead.org, agordeev@linux.ibm.com, kevin.brodsky@arm.com, tglx@linutronix.de, david@redhat.com, jannh@google.com, hughd@google.com, yuzhao@google.com, willy@infradead.org, muchun.song@linux.dev, vbabka@kernel.org, lorenzo.stoakes@oracle.com, akpm@linux-foundation.org, rientjes@google.com, vishal.moola@gmail.com, arnd@arndb.de, will@kernel.org, aneesh.kumar@kernel.org, npiggin@gmail.com, dave.hansen@linux.intel.com, rppt@kernel.org, ryan.roberts@arm.com Cc: linux-mm@kvack.org, linux-arm-kernel@lists.infradead.org, linuxppc-dev@lists.ozlabs.org, linux-riscv@lists.infradead.org, linux-s390@vger.kernel.org, sparclinux@vger.kernel.org, linux-kernel@vger.kernel.org, x86@kernel.org, linux-arch@vger.kernel.org, linux-csky@vger.kernel.org, linux-hexagon@vger.kernel.org, loongarch@lists.linux.dev, linux-m68k@lists.linux-m68k.org, linux-mips@vger.kernel.org, linux-openrisc@vger.kernel.org, linux-sh@vger.kernel.org, linux-um@lists.infradead.org, Qi Zheng Subject: [PATCH v3 01/17] Revert "mm: pgtable: make ptlock be freed by RCU" Date: Mon, 23 Dec 2024 17:40:47 +0800 Message-Id: X-Mailer: git-send-email 2.24.3 (Apple Git-128) In-Reply-To: References: MIME-Version: 1.0 X-CRM114-Version: 20100106-BlameMichelson ( TRE 0.8.0 (BSD) ) MR-646709E3 X-CRM114-CacheID: sfid-20241223_014318_577428_8419E1AF X-CRM114-Status: GOOD ( 11.05 ) X-BeenThere: linux-riscv@lists.infradead.org X-Mailman-Version: 2.1.34 Precedence: list List-Id: List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Sender: "linux-riscv" Errors-To: linux-riscv-bounces+linux-riscv=archiver.kernel.org@lists.infradead.org This reverts commit 2f3443770437e49abc39af26962d293851cbab6d. Signed-off-by: Qi Zheng --- include/linux/mm.h | 2 +- include/linux/mm_types.h | 9 +-------- mm/memory.c | 22 ++++++---------------- 3 files changed, 8 insertions(+), 25 deletions(-) diff --git a/include/linux/mm.h b/include/linux/mm.h index d61b9c7a3a7b0..c49bc7b764535 100644 --- a/include/linux/mm.h +++ b/include/linux/mm.h @@ -2925,7 +2925,7 @@ void ptlock_free(struct ptdesc *ptdesc); static inline spinlock_t *ptlock_ptr(struct ptdesc *ptdesc) { - return &(ptdesc->ptl->ptl); + return ptdesc->ptl; } #else /* ALLOC_SPLIT_PTLOCKS */ static inline void ptlock_cache_init(void) diff --git a/include/linux/mm_types.h b/include/linux/mm_types.h index 90ab8293d714a..6b27db7f94963 100644 --- a/include/linux/mm_types.h +++ b/include/linux/mm_types.h @@ -434,13 +434,6 @@ FOLIO_MATCH(flags, _flags_2a); FOLIO_MATCH(compound_head, _head_2a); #undef FOLIO_MATCH -#if ALLOC_SPLIT_PTLOCKS -struct pt_lock { - spinlock_t ptl; - struct rcu_head rcu; -}; -#endif - /** * struct ptdesc - Memory descriptor for page tables. * @__page_flags: Same as page flags. Powerpc only. @@ -489,7 +482,7 @@ struct ptdesc { union { unsigned long _pt_pad_2; #if ALLOC_SPLIT_PTLOCKS - struct pt_lock *ptl; + spinlock_t *ptl; #else spinlock_t ptl; #endif diff --git a/mm/memory.c b/mm/memory.c index b9b05c3f93f11..9423967b24180 100644 --- a/mm/memory.c +++ b/mm/memory.c @@ -7034,34 +7034,24 @@ static struct kmem_cache *page_ptl_cachep; void __init ptlock_cache_init(void) { - page_ptl_cachep = kmem_cache_create("page->ptl", sizeof(struct pt_lock), 0, + page_ptl_cachep = kmem_cache_create("page->ptl", sizeof(spinlock_t), 0, SLAB_PANIC, NULL); } bool ptlock_alloc(struct ptdesc *ptdesc) { - struct pt_lock *pt_lock; + spinlock_t *ptl; - pt_lock = kmem_cache_alloc(page_ptl_cachep, GFP_KERNEL); - if (!pt_lock) + ptl = kmem_cache_alloc(page_ptl_cachep, GFP_KERNEL); + if (!ptl) return false; - ptdesc->ptl = pt_lock; + ptdesc->ptl = ptl; return true; } -static void ptlock_free_rcu(struct rcu_head *head) -{ - struct pt_lock *pt_lock; - - pt_lock = container_of(head, struct pt_lock, rcu); - kmem_cache_free(page_ptl_cachep, pt_lock); -} - void ptlock_free(struct ptdesc *ptdesc) { - struct pt_lock *pt_lock = ptdesc->ptl; - - call_rcu(&pt_lock->rcu, ptlock_free_rcu); + kmem_cache_free(page_ptl_cachep, ptdesc->ptl); } #endif From patchwork Mon Dec 23 09:40:48 2024 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Qi Zheng X-Patchwork-Id: 13918701 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from bombadil.infradead.org (bombadil.infradead.org [198.137.202.133]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (No client certificate requested) by smtp.lore.kernel.org (Postfix) with ESMTPS id 2DE88E7718B for ; Mon, 23 Dec 2024 09:45:45 +0000 (UTC) DKIM-Signature: v=1; a=rsa-sha256; q=dns/txt; c=relaxed/relaxed; d=lists.infradead.org; s=bombadil.20210309; h=Sender: Content-Transfer-Encoding:Content-Type:List-Subscribe:List-Help:List-Post: List-Archive:List-Unsubscribe:List-Id:MIME-Version:References:In-Reply-To: Message-Id:Date:Subject:Cc:To:From:Reply-To:Content-ID:Content-Description: Resent-Date:Resent-From:Resent-Sender:Resent-To:Resent-Cc:Resent-Message-ID: List-Owner; bh=HLtCsCM+J7vEXuSG+zOKc2i81pNQkcn/iYTgoK8rWu4=; b=wRGUdEidbNkwvl SOAR4iQVZxFuhO80eh7vjhzQBYRhW9+TicgcUPOrI4H5fphkgCxII8F3dNd1tfVO/v2gstzSj51vr YyonZiv6XnpRNxGVyxtfBV6q1ETILrdCL/gWLvkrwqdUBc0eldtsmOqa5FbLm1+QbqQ4EYWMujl8P 9vbRVBbrWMlX3YYvkzaWnJ7LMF9HGjMisS9hQLDlnJHgX5X8RVxVcaItHkWoEWivmQbmEUriMiJPO 5+/RgBVnyi515V+D3dn6r1pbdpzPeCds/EGoQJ7bNKBwesYnjirN4UN2tNsW63jQZMJUgR7vPjrmy rZZBqD0q15Xabr2dSMPQ==; Received: from localhost ([::1] helo=bombadil.infradead.org) by bombadil.infradead.org with esmtp (Exim 4.98 #2 (Red Hat Linux)) id 1tPf0O-00000009hJ1-2nbx; Mon, 23 Dec 2024 09:45:40 +0000 Received: from mail-pf1-x435.google.com ([2607:f8b0:4864:20::435]) by bombadil.infradead.org with esmtps (Exim 4.98 #2 (Red Hat Linux)) id 1tPeyH-00000009g9m-2wLI for linux-riscv@lists.infradead.org; Mon, 23 Dec 2024 09:43:31 +0000 Received: by mail-pf1-x435.google.com with SMTP id d2e1a72fcca58-728f28744c5so3822100b3a.1 for ; Mon, 23 Dec 2024 01:43:29 -0800 (PST) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=bytedance.com; s=google; t=1734947009; x=1735551809; darn=lists.infradead.org; h=content-transfer-encoding:mime-version:references:in-reply-to :message-id:date:subject:cc:to:from:from:to:cc:subject:date :message-id:reply-to; bh=B/tJ15+vmer/NbqXoUznjybhQp/MgC6GOdT/GuQUXpM=; b=VY3XOzHwLvx6Xf7W1z+BGyRQXEuw2kIxzA6nwdtUvWhxV1/4eJWcv9c200FLpgIiRr yX587j8rvDds4waM/KQr8jLfs6Ywk1kwf6TVdec6zB+7hnK3DKDeDYSL0IQs5zJQXPBc ywWxVCi+uX+xpoLhp7kVlsqvJBeSkDRjMmfFm9QD52hKRiyQsj7buF8QCwBg5d7Q19pL 15vGfuIVB9S63saaxU2AKg5KLZMyhNi/rWOdYvZRpMAlSZ7fQn5RKQKgjaikC3Yvssd5 Y9CoePAOVXr78Wp3JDD7PtgdvigEU6i6f/fJij/S/66ogzdeNWg0KEsSFdZQHIjwA7wm 8cJQ== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20230601; t=1734947009; x=1735551809; h=content-transfer-encoding:mime-version:references:in-reply-to :message-id:date:subject:cc:to:from:x-gm-message-state:from:to:cc :subject:date:message-id:reply-to; bh=B/tJ15+vmer/NbqXoUznjybhQp/MgC6GOdT/GuQUXpM=; b=NpAL7XvzSWzOwjO24+4WYVpfNiTxgNvlzTzTJI6alvVU5bQVIi0EDKjSxqE3KlxD6L QA0DaS90bBMiIvFdPBuMTHT4Ba8RlUU/bbFzjAPZMiToZ/Ej5m+sNL0ZJ66kxfWojlbh 8tmUVDeGg3gJ4Alp8GAaQtmEjiC4olB0VHUQZ/1PE+PTbyxBUZUo178q6vOmtsauR6rc tvX3QJOuNm2IMQRDuYjqRr0ZdGx0Rc6AedhK2IaSRG64xNGatl781iRrZ2yYLyh8z26P 0q+L22XZnT1VPhBNRUvRgWKAbQBWHkgn47SK51fh65N9TJ6bFZTfS0MVSdT4EJnUaC81 H+Nw== X-Forwarded-Encrypted: i=1; AJvYcCXHOLr2k4UPZf3pkYCHOVYjnjuzsvwAI2k3V9vy9yTpIjZklt5M7Jr9lQxsdEKpwOTSpqBQeKCIV0rryA==@lists.infradead.org X-Gm-Message-State: AOJu0YxXAhpVNQqno8cOxMerKzQAzVTyE94WsT3RpDblI/8tNaSyAd45 1E1xs8XGBrYZ/blvDI/8+0IW35ufyrm2TMUT2OFKtFg+Zb3Xf82A7GfPs+bsOxY= X-Gm-Gg: ASbGncsFwsNDsRzPTDzuAu82TgV2YsPAjANno2Ap8FBHIlEfSZzgEG6SXEe357WDnIy l5Ep/yY1T3enC/Leerd/xSHvq+Vi/9d39kQ7da5sjvvejwQTPNkMXUNByMwoU+sNO947DA2K1+r ntL39js8eYu9AL4tYb/2DKZEv5Gxp+TBcIrlIBK3WQSH3AvU3s93VERBl2s87lovdcPm1modRiz d8j9EsbgNbelMCAEFQi1gPpdD/6/MSnrh5yv3bhculoYlYBYYTwWx1V/tKHs/rv2l0XN4+mZemE Jfy1IiYQrlKSkN067RUfYg== X-Google-Smtp-Source: AGHT+IGn0WgNlH4n6jFU+xV15Lu8l6dnMV2FFWuW4zvXTMVJTCCaTT+ZxewqFKpQcJgKvmYU0RJrVw== X-Received: by 2002:a05:6a00:35ca:b0:724:f86e:e3d9 with SMTP id d2e1a72fcca58-72abdecbdb4mr16058641b3a.14.1734947008718; Mon, 23 Dec 2024 01:43:28 -0800 (PST) Received: from C02DW0BEMD6R.bytedance.net ([203.208.167.150]) by smtp.gmail.com with ESMTPSA id d2e1a72fcca58-72aad8486c6sm7468309b3a.85.2024.12.23.01.43.16 (version=TLS1_3 cipher=TLS_AES_256_GCM_SHA384 bits=256/256); Mon, 23 Dec 2024 01:43:28 -0800 (PST) From: Qi Zheng To: peterz@infradead.org, agordeev@linux.ibm.com, kevin.brodsky@arm.com, tglx@linutronix.de, david@redhat.com, jannh@google.com, hughd@google.com, yuzhao@google.com, willy@infradead.org, muchun.song@linux.dev, vbabka@kernel.org, lorenzo.stoakes@oracle.com, akpm@linux-foundation.org, rientjes@google.com, vishal.moola@gmail.com, arnd@arndb.de, will@kernel.org, aneesh.kumar@kernel.org, npiggin@gmail.com, dave.hansen@linux.intel.com, rppt@kernel.org, ryan.roberts@arm.com Cc: linux-mm@kvack.org, linux-arm-kernel@lists.infradead.org, linuxppc-dev@lists.ozlabs.org, linux-riscv@lists.infradead.org, linux-s390@vger.kernel.org, sparclinux@vger.kernel.org, linux-kernel@vger.kernel.org, x86@kernel.org, linux-arch@vger.kernel.org, linux-csky@vger.kernel.org, linux-hexagon@vger.kernel.org, loongarch@lists.linux.dev, linux-m68k@lists.linux-m68k.org, linux-mips@vger.kernel.org, linux-openrisc@vger.kernel.org, linux-sh@vger.kernel.org, linux-um@lists.infradead.org, Qi Zheng Subject: [PATCH v3 02/17] riscv: mm: Skip pgtable level check in {pud,p4d}_alloc_one Date: Mon, 23 Dec 2024 17:40:48 +0800 Message-Id: <84ddf857508b98a195a790bc6ff6ab8849b44633.1734945104.git.zhengqi.arch@bytedance.com> X-Mailer: git-send-email 2.24.3 (Apple Git-128) In-Reply-To: References: MIME-Version: 1.0 X-CRM114-Version: 20100106-BlameMichelson ( TRE 0.8.0 (BSD) ) MR-646709E3 X-CRM114-CacheID: sfid-20241223_014329_751795_09A77CBC X-CRM114-Status: GOOD ( 10.67 ) X-BeenThere: linux-riscv@lists.infradead.org X-Mailman-Version: 2.1.34 Precedence: list List-Id: List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Sender: "linux-riscv" Errors-To: linux-riscv-bounces+linux-riscv=archiver.kernel.org@lists.infradead.org From: Kevin Brodsky {pmd,pud,p4d}_alloc_one() is never called if the corresponding page table level is folded, as {pmd,pud,p4d}_alloc() already does the required check. We can therefore remove the runtime page table level checks in {pud,p4d}_alloc_one. The PUD helper becomes equivalent to the generic version, so we remove it altogether. This is consistent with the way arm64 and x86 handle this situation (runtime check in p4d_free() only). Signed-off-by: Kevin Brodsky Acked-by: Dave Hansen Signed-off-by: Qi Zheng --- arch/riscv/include/asm/pgalloc.h | 22 ++++------------------ 1 file changed, 4 insertions(+), 18 deletions(-) diff --git a/arch/riscv/include/asm/pgalloc.h b/arch/riscv/include/asm/pgalloc.h index f52264304f772..8ad0bbe838a24 100644 --- a/arch/riscv/include/asm/pgalloc.h +++ b/arch/riscv/include/asm/pgalloc.h @@ -12,7 +12,6 @@ #include #ifdef CONFIG_MMU -#define __HAVE_ARCH_PUD_ALLOC_ONE #define __HAVE_ARCH_PUD_FREE #include @@ -88,15 +87,6 @@ static inline void pgd_populate_safe(struct mm_struct *mm, pgd_t *pgd, } } -#define pud_alloc_one pud_alloc_one -static inline pud_t *pud_alloc_one(struct mm_struct *mm, unsigned long addr) -{ - if (pgtable_l4_enabled) - return __pud_alloc_one(mm, addr); - - return NULL; -} - #define pud_free pud_free static inline void pud_free(struct mm_struct *mm, pud_t *pud) { @@ -118,15 +108,11 @@ static inline void __pud_free_tlb(struct mmu_gather *tlb, pud_t *pud, #define p4d_alloc_one p4d_alloc_one static inline p4d_t *p4d_alloc_one(struct mm_struct *mm, unsigned long addr) { - if (pgtable_l5_enabled) { - gfp_t gfp = GFP_PGTABLE_USER; - - if (mm == &init_mm) - gfp = GFP_PGTABLE_KERNEL; - return (p4d_t *)get_zeroed_page(gfp); - } + gfp_t gfp = GFP_PGTABLE_USER; - return NULL; + if (mm == &init_mm) + gfp = GFP_PGTABLE_KERNEL; + return (p4d_t *)get_zeroed_page(gfp); } static inline void __p4d_free(struct mm_struct *mm, p4d_t *p4d) From patchwork Mon Dec 23 09:40:49 2024 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Qi Zheng X-Patchwork-Id: 13918702 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from bombadil.infradead.org (bombadil.infradead.org [198.137.202.133]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (No client certificate requested) by smtp.lore.kernel.org (Postfix) with ESMTPS id 19C1FE7718B for ; Mon, 23 Dec 2024 09:46:58 +0000 (UTC) DKIM-Signature: v=1; a=rsa-sha256; q=dns/txt; c=relaxed/relaxed; d=lists.infradead.org; s=bombadil.20210309; h=Sender: Content-Transfer-Encoding:Content-Type:List-Subscribe:List-Help:List-Post: List-Archive:List-Unsubscribe:List-Id:MIME-Version:References:In-Reply-To: Message-Id:Date:Subject:Cc:To:From:Reply-To:Content-ID:Content-Description: Resent-Date:Resent-From:Resent-Sender:Resent-To:Resent-Cc:Resent-Message-ID: List-Owner; bh=RJXl4DGjC5f4ctxeltcE+YwLQooy718Qojj2NIDv6go=; b=LP+ff/1sYdsWZb 9abzZpYPBVCX+/23Jh7nmO1sIW90EDWBe+9a5ErjYUEdCp3S/X0YjR/LskFYX5ELrQzE9rOc62b9u oYFpgbI+bV+DZBp9Xe6W8+HSc8rimy5cRj01R6obYz0Gd+/vh2PpepIuEQMXrNXuUubgCr6KGMteF XHKNzM2j32dsfEeP0+VTUyw0Qa4Yjy84qWLVkR6wrD3wcfvaaNwOekeXrxHB+Z3N8dZzpR0dzoWAb tpuBN9E8n+2k5A9mRqnDSPFGaulwZ4LeZv3kpELz5fUFk7W1J0V+L/Pxr/jjDvVO6cbrPLer1zPEX c495u6QzLhQoUBE/zfVA==; Received: from localhost ([::1] helo=bombadil.infradead.org) by bombadil.infradead.org with esmtp (Exim 4.98 #2 (Red Hat Linux)) id 1tPf1Z-00000009hvv-2qWt; Mon, 23 Dec 2024 09:46:53 +0000 Received: from mail-pf1-x42a.google.com ([2607:f8b0:4864:20::42a]) by bombadil.infradead.org with esmtps (Exim 4.98 #2 (Red Hat Linux)) id 1tPeyU-00000009gFh-00TZ for linux-riscv@lists.infradead.org; Mon, 23 Dec 2024 09:43:43 +0000 Received: by mail-pf1-x42a.google.com with SMTP id d2e1a72fcca58-725c86bbae7so3476932b3a.3 for ; Mon, 23 Dec 2024 01:43:41 -0800 (PST) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=bytedance.com; s=google; t=1734947021; x=1735551821; darn=lists.infradead.org; h=content-transfer-encoding:mime-version:references:in-reply-to :message-id:date:subject:cc:to:from:from:to:cc:subject:date :message-id:reply-to; bh=Hi4+N7L/R07Fq6g8WRpMxop4cD3t01a1BHl7awbkQjI=; b=Xn6BxD2tNy7iPiUKE57yvu7O+G6khuCm5ERyB2wOZDOLAwFHlmE8/+8lfblcv1aHXc iIZgLtuxxgnHt1b4apM9Aaw6jxzqWQc3OVgcBa7sxWQnH+zrBl94g7RmVwci0HjpVANN jaA4dBcvsUea0c6GvTw+4yLs2+a21QvEGY3gw4gNwAkuWg7csUXvHhXvxcb/goOTin0n o+w7bTrb39jWk2C03VgHv6ozH2T3OctYiIHIu9IuzO079Pbi6J8Xlu0+GKIhjpFNtG3C SxGuMEVO+UrYeF2KNaTMMeKk7wQcDyoaSM+NRt/OqIeJNVG2bR3UxfJvRxUnPfo03ZPq dWuA== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20230601; t=1734947021; x=1735551821; h=content-transfer-encoding:mime-version:references:in-reply-to :message-id:date:subject:cc:to:from:x-gm-message-state:from:to:cc :subject:date:message-id:reply-to; bh=Hi4+N7L/R07Fq6g8WRpMxop4cD3t01a1BHl7awbkQjI=; b=CoJeW94+RTXlQa86+CBgUlgiSHYkTsBvuHx9WQLj9t8bmkSWmXCce25F5RwPBJgfzx XJOuQmzEsRHNaavmhguUqhsBjDgeaVugcjzDvPyp4/o7xD6wg6fcjy9EjkI5X1DCejDQ nqkvMkXdm/xW66o2clj4CZzgbydGnNKyWeKPVqBxtmIt+7ZFO8goRO/Ny3F2oY1Pl4eM E5djsaUepZEUX6uOoZoN65Q9e0pyYcWR+BUnOWqwk825mkIoV0n+sBsPRNE3wMhXEk8/ SlXG607pJMGCkdSBCPrGOsb5xzYnqWn/LNoBPAmTKgSYFmB/Yk/8f7jClC1u4s+u16WI 5kcw== X-Forwarded-Encrypted: i=1; AJvYcCXel6ZQRjDiZqWQR9Jl5VC3RQHI1ZWvfALKgItn+RHSgAoESQqGc+8La9aaMkhL4VxBruIwlbt3D9lWdA==@lists.infradead.org X-Gm-Message-State: AOJu0YwnYkR2XlFMAXUW1MkHfuqjVOgUtvsAdysDodas5zfv4wjU8QBm fri1YW1pYZjTm4gUAY0K1mwtFUvU1sSbS02BZfCpw4rMwrU2g0DtTNjyO9hujog= X-Gm-Gg: ASbGncshClyh4aYy3s592T162GCMxU6fvFhRBR/FL82sCKFSbNSSc0ErvHfG0dF7vK6 8W2ETlW63/3DL1VwbZtgNn2n3WUK93KSeoiXlooQogOiJyIqesNb0z9294huhJVog/bpL3zQ1z6 csnnHFykbQzcXOPIBBHRYiSNMuklKNCVwZh2O0SDwmQfI98sRrw3RViQNTUa30QT+/w76UgvRN1 6c3v7PBF7BFMDm1JhCDMSZoKqp0ZAT8atvx9u3Gt2lfFuByO4zj2OqOqP6mLnZpi0yFcNjMnPD+ m95dh4CukRg6zILuAz8Dlg== X-Google-Smtp-Source: AGHT+IHP/sWNqLjbiRRokya3RRzP3A7lT6GiLvDE2E3HVCuxopdI/+yKYyLcU7/i5H4uYS+gGEKb/A== X-Received: by 2002:a05:6a21:9017:b0:1e5:ddac:1eff with SMTP id adf61e73a8af0-1e5e04a0c7cmr19747203637.20.1734947021430; Mon, 23 Dec 2024 01:43:41 -0800 (PST) Received: from C02DW0BEMD6R.bytedance.net ([203.208.167.150]) by smtp.gmail.com with ESMTPSA id d2e1a72fcca58-72aad8486c6sm7468309b3a.85.2024.12.23.01.43.29 (version=TLS1_3 cipher=TLS_AES_256_GCM_SHA384 bits=256/256); Mon, 23 Dec 2024 01:43:40 -0800 (PST) From: Qi Zheng To: peterz@infradead.org, agordeev@linux.ibm.com, kevin.brodsky@arm.com, tglx@linutronix.de, david@redhat.com, jannh@google.com, hughd@google.com, yuzhao@google.com, willy@infradead.org, muchun.song@linux.dev, vbabka@kernel.org, lorenzo.stoakes@oracle.com, akpm@linux-foundation.org, rientjes@google.com, vishal.moola@gmail.com, arnd@arndb.de, will@kernel.org, aneesh.kumar@kernel.org, npiggin@gmail.com, dave.hansen@linux.intel.com, rppt@kernel.org, ryan.roberts@arm.com Cc: linux-mm@kvack.org, linux-arm-kernel@lists.infradead.org, linuxppc-dev@lists.ozlabs.org, linux-riscv@lists.infradead.org, linux-s390@vger.kernel.org, sparclinux@vger.kernel.org, linux-kernel@vger.kernel.org, x86@kernel.org, linux-arch@vger.kernel.org, linux-csky@vger.kernel.org, linux-hexagon@vger.kernel.org, loongarch@lists.linux.dev, linux-m68k@lists.linux-m68k.org, linux-mips@vger.kernel.org, linux-openrisc@vger.kernel.org, linux-sh@vger.kernel.org, linux-um@lists.infradead.org, Qi Zheng Subject: [PATCH v3 03/17] asm-generic: pgalloc: Provide generic p4d_{alloc_one,free} Date: Mon, 23 Dec 2024 17:40:49 +0800 Message-Id: <4c4bcc1aa565c6252183553aecd5e5cbd1a0f6ea.1734945104.git.zhengqi.arch@bytedance.com> X-Mailer: git-send-email 2.24.3 (Apple Git-128) In-Reply-To: References: MIME-Version: 1.0 X-CRM114-Version: 20100106-BlameMichelson ( TRE 0.8.0 (BSD) ) MR-646709E3 X-CRM114-CacheID: sfid-20241223_014342_054122_F5CDEA64 X-CRM114-Status: GOOD ( 13.24 ) X-BeenThere: linux-riscv@lists.infradead.org X-Mailman-Version: 2.1.34 Precedence: list List-Id: List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Sender: "linux-riscv" Errors-To: linux-riscv-bounces+linux-riscv=archiver.kernel.org@lists.infradead.org From: Kevin Brodsky Four architectures currently implement 5-level pgtables: arm64, riscv, x86 and s390. The first three have essentially the same implementation for p4d_alloc_one() and p4d_free(), so we've got an opportunity to reduce duplication like at the lower levels. Provide a generic version of p4d_alloc_one() and p4d_free(), and make use of it on those architectures. Their implementation is the same as at PUD level, except that p4d_free() performs a runtime check by calling mm_p4d_folded(). 5-level pgtables depend on a runtime-detected hardware feature on all supported architectures, so we might as well include this check in the generic implementation. No runtime check is required in p4d_alloc_one() as the top-level p4d_alloc() already does the required check. Signed-off-by: Kevin Brodsky Acked-by: Dave Hansen Signed-off-by: Qi Zheng --- arch/arm64/include/asm/pgalloc.h | 17 ------------ arch/riscv/include/asm/pgalloc.h | 23 ---------------- arch/x86/include/asm/pgalloc.h | 18 ------------- include/asm-generic/pgalloc.h | 45 ++++++++++++++++++++++++++++++++ 4 files changed, 45 insertions(+), 58 deletions(-) diff --git a/arch/arm64/include/asm/pgalloc.h b/arch/arm64/include/asm/pgalloc.h index e75422864d1bd..2965f5a7e39e3 100644 --- a/arch/arm64/include/asm/pgalloc.h +++ b/arch/arm64/include/asm/pgalloc.h @@ -85,23 +85,6 @@ static inline void pgd_populate(struct mm_struct *mm, pgd_t *pgdp, p4d_t *p4dp) __pgd_populate(pgdp, __pa(p4dp), pgdval); } -static inline p4d_t *p4d_alloc_one(struct mm_struct *mm, unsigned long addr) -{ - gfp_t gfp = GFP_PGTABLE_USER; - - if (mm == &init_mm) - gfp = GFP_PGTABLE_KERNEL; - return (p4d_t *)get_zeroed_page(gfp); -} - -static inline void p4d_free(struct mm_struct *mm, p4d_t *p4d) -{ - if (!pgtable_l5_enabled()) - return; - BUG_ON((unsigned long)p4d & (PAGE_SIZE-1)); - free_page((unsigned long)p4d); -} - #define __p4d_free_tlb(tlb, p4d, addr) p4d_free((tlb)->mm, p4d) #else static inline void __pgd_populate(pgd_t *pgdp, phys_addr_t p4dp, pgdval_t prot) diff --git a/arch/riscv/include/asm/pgalloc.h b/arch/riscv/include/asm/pgalloc.h index 8ad0bbe838a24..551d614d3369c 100644 --- a/arch/riscv/include/asm/pgalloc.h +++ b/arch/riscv/include/asm/pgalloc.h @@ -105,29 +105,6 @@ static inline void __pud_free_tlb(struct mmu_gather *tlb, pud_t *pud, } } -#define p4d_alloc_one p4d_alloc_one -static inline p4d_t *p4d_alloc_one(struct mm_struct *mm, unsigned long addr) -{ - gfp_t gfp = GFP_PGTABLE_USER; - - if (mm == &init_mm) - gfp = GFP_PGTABLE_KERNEL; - return (p4d_t *)get_zeroed_page(gfp); -} - -static inline void __p4d_free(struct mm_struct *mm, p4d_t *p4d) -{ - BUG_ON((unsigned long)p4d & (PAGE_SIZE-1)); - free_page((unsigned long)p4d); -} - -#define p4d_free p4d_free -static inline void p4d_free(struct mm_struct *mm, p4d_t *p4d) -{ - if (pgtable_l5_enabled) - __p4d_free(mm, p4d); -} - static inline void __p4d_free_tlb(struct mmu_gather *tlb, p4d_t *p4d, unsigned long addr) { diff --git a/arch/x86/include/asm/pgalloc.h b/arch/x86/include/asm/pgalloc.h index dcd836b59bebd..dd4841231bb9f 100644 --- a/arch/x86/include/asm/pgalloc.h +++ b/arch/x86/include/asm/pgalloc.h @@ -147,24 +147,6 @@ static inline void pgd_populate_safe(struct mm_struct *mm, pgd_t *pgd, p4d_t *p4 set_pgd_safe(pgd, __pgd(_PAGE_TABLE | __pa(p4d))); } -static inline p4d_t *p4d_alloc_one(struct mm_struct *mm, unsigned long addr) -{ - gfp_t gfp = GFP_KERNEL_ACCOUNT; - - if (mm == &init_mm) - gfp &= ~__GFP_ACCOUNT; - return (p4d_t *)get_zeroed_page(gfp); -} - -static inline void p4d_free(struct mm_struct *mm, p4d_t *p4d) -{ - if (!pgtable_l5_enabled()) - return; - - BUG_ON((unsigned long)p4d & (PAGE_SIZE-1)); - free_page((unsigned long)p4d); -} - extern void ___p4d_free_tlb(struct mmu_gather *tlb, p4d_t *p4d); static inline void __p4d_free_tlb(struct mmu_gather *tlb, p4d_t *p4d, diff --git a/include/asm-generic/pgalloc.h b/include/asm-generic/pgalloc.h index 7c48f5fbf8aa7..59131629ac9cc 100644 --- a/include/asm-generic/pgalloc.h +++ b/include/asm-generic/pgalloc.h @@ -215,6 +215,51 @@ static inline void pud_free(struct mm_struct *mm, pud_t *pud) #endif /* CONFIG_PGTABLE_LEVELS > 3 */ +#if CONFIG_PGTABLE_LEVELS > 4 + +static inline p4d_t *__p4d_alloc_one_noprof(struct mm_struct *mm, unsigned long addr) +{ + gfp_t gfp = GFP_PGTABLE_USER; + struct ptdesc *ptdesc; + + if (mm == &init_mm) + gfp = GFP_PGTABLE_KERNEL; + gfp &= ~__GFP_HIGHMEM; + + ptdesc = pagetable_alloc_noprof(gfp, 0); + if (!ptdesc) + return NULL; + + return ptdesc_address(ptdesc); +} +#define __p4d_alloc_one(...) alloc_hooks(__p4d_alloc_one_noprof(__VA_ARGS__)) + +#ifndef __HAVE_ARCH_P4D_ALLOC_ONE +static inline p4d_t *p4d_alloc_one_noprof(struct mm_struct *mm, unsigned long addr) +{ + return __p4d_alloc_one_noprof(mm, addr); +} +#define p4d_alloc_one(...) alloc_hooks(p4d_alloc_one_noprof(__VA_ARGS__)) +#endif + +static inline void __p4d_free(struct mm_struct *mm, p4d_t *p4d) +{ + struct ptdesc *ptdesc = virt_to_ptdesc(p4d); + + BUG_ON((unsigned long)p4d & (PAGE_SIZE-1)); + pagetable_free(ptdesc); +} + +#ifndef __HAVE_ARCH_P4D_FREE +static inline void p4d_free(struct mm_struct *mm, p4d_t *p4d) +{ + if (!mm_p4d_folded(mm)) + __p4d_free(mm, p4d); +} +#endif + +#endif /* CONFIG_PGTABLE_LEVELS > 4 */ + #ifndef __HAVE_ARCH_PGD_FREE static inline void pgd_free(struct mm_struct *mm, pgd_t *pgd) { From patchwork Mon Dec 23 09:40:50 2024 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Qi Zheng X-Patchwork-Id: 13918703 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from bombadil.infradead.org (bombadil.infradead.org [198.137.202.133]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (No client certificate requested) by smtp.lore.kernel.org (Postfix) with ESMTPS id 2267FE7718D for ; Mon, 23 Dec 2024 09:49:21 +0000 (UTC) DKIM-Signature: v=1; a=rsa-sha256; q=dns/txt; c=relaxed/relaxed; d=lists.infradead.org; s=bombadil.20210309; h=Sender: Content-Transfer-Encoding:Content-Type:List-Subscribe:List-Help:List-Post: List-Archive:List-Unsubscribe:List-Id:MIME-Version:References:In-Reply-To: Message-Id:Date:Subject:Cc:To:From:Reply-To:Content-ID:Content-Description: Resent-Date:Resent-From:Resent-Sender:Resent-To:Resent-Cc:Resent-Message-ID: List-Owner; bh=cB8faopMQRkkXYKIF0sWGnuLJx0A08eE48gfMyt4miA=; b=T3cxhaoTLeXRag ku5kAw4DYeJ6iAXWXisc33owGynMiPJ8/hso74jit0cboGuSaf9TDwSz72qpoQlI6cM7DeAUYsMEQ Kbr3SIWNAt3x06JntcnVWoY9UMxbJ+965T77zPjfPnUOpWKuDGzIyNYTy4y3abWXVH5ZWV89y1xN7 tKUwg6d7OP86/NFcTm+ybopSJiR/M9Ejeb7JYcA4RzPLKzdHQdSUdsOJtd82FHajzJVziGkZQSw8j PclVxNjUJLmY2WRI6DD/q5gzt5y34M/dV/FT5AzkNYcqqWLQOeZ3hYNyXaO03ACg9OuQWAcBa65ZN cjUFSWiJ7oV3Sg8zWLxQ==; Received: from localhost ([::1] helo=bombadil.infradead.org) by bombadil.infradead.org with esmtp (Exim 4.98 #2 (Red Hat Linux)) id 1tPf3s-00000009iTA-1xQg; Mon, 23 Dec 2024 09:49:16 +0000 Received: from mail-pf1-x430.google.com ([2607:f8b0:4864:20::430]) by bombadil.infradead.org with esmtps (Exim 4.98 #2 (Red Hat Linux)) id 1tPeyh-00000009gKW-070A for linux-riscv@lists.infradead.org; Mon, 23 Dec 2024 09:43:57 +0000 Received: by mail-pf1-x430.google.com with SMTP id d2e1a72fcca58-728ec840a8aso4529812b3a.0 for ; Mon, 23 Dec 2024 01:43:54 -0800 (PST) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=bytedance.com; s=google; t=1734947034; x=1735551834; darn=lists.infradead.org; h=content-transfer-encoding:mime-version:references:in-reply-to :message-id:date:subject:cc:to:from:from:to:cc:subject:date :message-id:reply-to; bh=t6vBY23hDAeU19z0pXyQ9vuAeZavJyfGUN8npoqwlxU=; b=FRuwHpDTAKgAjS7NJ778RZgmtB7x56W2wOl+DX5SC8EvaSzgeqPusu9GErCFFMc1JE oxGS7UfPZp7MhqO/IELTzKMAOepmFdkK6Im0IjT1tvjTVxfMsiKqD7gIrPJlimlshiPB sADVnFpNQZrGRQUV/z2JE3WwcMVkEWV9T+CE7Q+i//OfstTWaz+2lL8XXCSOd1xEbQvL 3vg8gFJa3QgDlvug+whSDaoDLvw0S/AVQ2Txyaj0hEosbWAv3okWZOA7qTiciItYxIQL E61XMEDcMaxfDOmKR/yisPhaiyvGlR16WEQ1A6eBMmmXBSwcVw4rtpR1REe66h+Nj8dF WSfg== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20230601; t=1734947034; x=1735551834; h=content-transfer-encoding:mime-version:references:in-reply-to :message-id:date:subject:cc:to:from:x-gm-message-state:from:to:cc :subject:date:message-id:reply-to; bh=t6vBY23hDAeU19z0pXyQ9vuAeZavJyfGUN8npoqwlxU=; b=sghx7I1ygRfF1d19xcbtKuJOXwqfclC8UuSGXoQ+/08GpstXhDS44EJq7gqykOrYpr nSgAxusd839BvyZcS12Z5g2J4YUS8Hw+FyGfESY3u+zrxAJg+k1XYYQlifrbES6C+f/c PJBQD34eXWIf9Dn8XgOECTj3b/aMoEZDOeX0CwINnGIn6ZunSuaL1kLKf0RHjq4cWKW6 NTgTUvp5/kPHHHoY3FnTXuf/q0ux6mrZ0geygcidNAQ8cmxJApHVaGxtXmSc2/LrH4b8 AS1J+gBhHHxQV/yEpZw5fEd1YaPYX4HxXDDdq+LlBwhdtVuhRwgVl/wOorCKzpFHKiwW 55rA== X-Forwarded-Encrypted: i=1; AJvYcCUPpIuqkPw0MfpJksccwzmTVhoiWh6TmrxcLk45XkCUsaYeLo0ua/J042Pgf51aqq5pNYBPZY5cBACtcA==@lists.infradead.org X-Gm-Message-State: AOJu0YwM77AdxrB4R+tOwV0bDk/XPrBprPDkgsvHfd5hhwOvi/dSp4Xa 3mJBwPuNdTEIfVO/ZFPns6mXo++iISSS9OZ2Mru9V9IBrP+RjcGOkwkq83IpKNo= X-Gm-Gg: ASbGncs3370uhr0DJDhMAKBbjpPueaZQbVyoJNYdvgBogT//LWWeFKimwttqQFIoZZQ 5UECE2UtJCV7YLEmZ+A2a8umd3seFKQVev1wZfYaYQCoeYm6X9/JFfeoXph3SfzjxlMI0pBG0zK EFqGZQqx9TESE5oD18dIxO0nOPTOAgifWnsD5cSELFVm/EcEdZANo+eA4gARfjukb+wCV3C5X2l MnPtcFvlFIlcc6GX9AGlQroLph0HBhtLCCUczC02CRHbG4fPSJfda8CN3AAEjVDlZ+D9zs/esEG yXCtMH2419hMSAegDBbvLg== X-Google-Smtp-Source: AGHT+IGO0+mWINscMdGqF/L7XDe6Dv/dfGXrIYILRYMXJn04aY3ZCG1rf5V2YGuLlSic6wG5VWjqsA== X-Received: by 2002:a05:6a00:4090:b0:72a:a7a4:b4cd with SMTP id d2e1a72fcca58-72abdeb85b1mr18289604b3a.21.1734947034102; Mon, 23 Dec 2024 01:43:54 -0800 (PST) Received: from C02DW0BEMD6R.bytedance.net ([203.208.167.150]) by smtp.gmail.com with ESMTPSA id d2e1a72fcca58-72aad8486c6sm7468309b3a.85.2024.12.23.01.43.41 (version=TLS1_3 cipher=TLS_AES_256_GCM_SHA384 bits=256/256); Mon, 23 Dec 2024 01:43:53 -0800 (PST) From: Qi Zheng To: peterz@infradead.org, agordeev@linux.ibm.com, kevin.brodsky@arm.com, tglx@linutronix.de, david@redhat.com, jannh@google.com, hughd@google.com, yuzhao@google.com, willy@infradead.org, muchun.song@linux.dev, vbabka@kernel.org, lorenzo.stoakes@oracle.com, akpm@linux-foundation.org, rientjes@google.com, vishal.moola@gmail.com, arnd@arndb.de, will@kernel.org, aneesh.kumar@kernel.org, npiggin@gmail.com, dave.hansen@linux.intel.com, rppt@kernel.org, ryan.roberts@arm.com Cc: linux-mm@kvack.org, linux-arm-kernel@lists.infradead.org, linuxppc-dev@lists.ozlabs.org, linux-riscv@lists.infradead.org, linux-s390@vger.kernel.org, sparclinux@vger.kernel.org, linux-kernel@vger.kernel.org, x86@kernel.org, linux-arch@vger.kernel.org, linux-csky@vger.kernel.org, linux-hexagon@vger.kernel.org, loongarch@lists.linux.dev, linux-m68k@lists.linux-m68k.org, linux-mips@vger.kernel.org, linux-openrisc@vger.kernel.org, linux-sh@vger.kernel.org, linux-um@lists.infradead.org, Qi Zheng Subject: [PATCH v3 04/17] mm: pgtable: add statistics for P4D level page table Date: Mon, 23 Dec 2024 17:40:50 +0800 Message-Id: <2fa644e37ab917292f5c342e40fa805aa91afbbd.1734945104.git.zhengqi.arch@bytedance.com> X-Mailer: git-send-email 2.24.3 (Apple Git-128) In-Reply-To: References: MIME-Version: 1.0 X-CRM114-Version: 20100106-BlameMichelson ( TRE 0.8.0 (BSD) ) MR-646709E3 X-CRM114-CacheID: sfid-20241223_014355_103078_F6B122B6 X-CRM114-Status: GOOD ( 11.70 ) X-BeenThere: linux-riscv@lists.infradead.org X-Mailman-Version: 2.1.34 Precedence: list List-Id: List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Sender: "linux-riscv" Errors-To: linux-riscv-bounces+linux-riscv=archiver.kernel.org@lists.infradead.org Like other levels of page tables, add statistics for P4D level page table. Signed-off-by: Qi Zheng Originally-by: Peter Zijlstra (Intel) --- arch/riscv/include/asm/pgalloc.h | 6 +++++- arch/x86/mm/pgtable.c | 3 +++ include/asm-generic/pgalloc.h | 2 ++ include/linux/mm.h | 16 ++++++++++++++++ 4 files changed, 26 insertions(+), 1 deletion(-) diff --git a/arch/riscv/include/asm/pgalloc.h b/arch/riscv/include/asm/pgalloc.h index 551d614d3369c..3466fbe2e508d 100644 --- a/arch/riscv/include/asm/pgalloc.h +++ b/arch/riscv/include/asm/pgalloc.h @@ -108,8 +108,12 @@ static inline void __pud_free_tlb(struct mmu_gather *tlb, pud_t *pud, static inline void __p4d_free_tlb(struct mmu_gather *tlb, p4d_t *p4d, unsigned long addr) { - if (pgtable_l5_enabled) + if (pgtable_l5_enabled) { + struct ptdesc *ptdesc = virt_to_ptdesc(p4d); + + pagetable_p4d_dtor(ptdesc); riscv_tlb_remove_ptdesc(tlb, virt_to_ptdesc(p4d)); + } } #endif /* __PAGETABLE_PMD_FOLDED */ diff --git a/arch/x86/mm/pgtable.c b/arch/x86/mm/pgtable.c index 69a357b15974a..3d6e84da45b24 100644 --- a/arch/x86/mm/pgtable.c +++ b/arch/x86/mm/pgtable.c @@ -94,6 +94,9 @@ void ___pud_free_tlb(struct mmu_gather *tlb, pud_t *pud) #if CONFIG_PGTABLE_LEVELS > 4 void ___p4d_free_tlb(struct mmu_gather *tlb, p4d_t *p4d) { + struct ptdesc *ptdesc = virt_to_ptdesc(p4d); + + pagetable_p4d_dtor(ptdesc); paravirt_release_p4d(__pa(p4d) >> PAGE_SHIFT); paravirt_tlb_remove_table(tlb, virt_to_page(p4d)); } diff --git a/include/asm-generic/pgalloc.h b/include/asm-generic/pgalloc.h index 59131629ac9cc..bb482eeca0c3e 100644 --- a/include/asm-generic/pgalloc.h +++ b/include/asm-generic/pgalloc.h @@ -230,6 +230,7 @@ static inline p4d_t *__p4d_alloc_one_noprof(struct mm_struct *mm, unsigned long if (!ptdesc) return NULL; + pagetable_p4d_ctor(ptdesc); return ptdesc_address(ptdesc); } #define __p4d_alloc_one(...) alloc_hooks(__p4d_alloc_one_noprof(__VA_ARGS__)) @@ -247,6 +248,7 @@ static inline void __p4d_free(struct mm_struct *mm, p4d_t *p4d) struct ptdesc *ptdesc = virt_to_ptdesc(p4d); BUG_ON((unsigned long)p4d & (PAGE_SIZE-1)); + pagetable_p4d_dtor(ptdesc); pagetable_free(ptdesc); } diff --git a/include/linux/mm.h b/include/linux/mm.h index c49bc7b764535..5d82f42ddd5cc 100644 --- a/include/linux/mm.h +++ b/include/linux/mm.h @@ -3175,6 +3175,22 @@ static inline void pagetable_pud_dtor(struct ptdesc *ptdesc) lruvec_stat_sub_folio(folio, NR_PAGETABLE); } +static inline void pagetable_p4d_ctor(struct ptdesc *ptdesc) +{ + struct folio *folio = ptdesc_folio(ptdesc); + + __folio_set_pgtable(folio); + lruvec_stat_add_folio(folio, NR_PAGETABLE); +} + +static inline void pagetable_p4d_dtor(struct ptdesc *ptdesc) +{ + struct folio *folio = ptdesc_folio(ptdesc); + + __folio_clear_pgtable(folio); + lruvec_stat_sub_folio(folio, NR_PAGETABLE); +} + extern void __init pagecache_init(void); extern void free_initmem(void); From patchwork Mon Dec 23 09:40:51 2024 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Qi Zheng X-Patchwork-Id: 13918704 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from bombadil.infradead.org (bombadil.infradead.org [198.137.202.133]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (No client certificate requested) by smtp.lore.kernel.org (Postfix) with ESMTPS id B2B02E7718B for ; Mon, 23 Dec 2024 09:49:21 +0000 (UTC) DKIM-Signature: v=1; a=rsa-sha256; q=dns/txt; c=relaxed/relaxed; d=lists.infradead.org; s=bombadil.20210309; h=Sender: Content-Transfer-Encoding:Content-Type:List-Subscribe:List-Help:List-Post: List-Archive:List-Unsubscribe:List-Id:MIME-Version:References:In-Reply-To: Message-Id:Date:Subject:Cc:To:From:Reply-To:Content-ID:Content-Description: Resent-Date:Resent-From:Resent-Sender:Resent-To:Resent-Cc:Resent-Message-ID: List-Owner; bh=u+WohJfce4Pw1L7JLtbP3Gpmp6Yy15IeLJMkiY0Pgq8=; b=qyK7FeZ1I/igUz rCg3f0UjAytStX+U87q7wT8fn/rTEQgBNMPk9zOD/hlFf1brIIWPxtg+MDjLthsp9wz62bJo5m5wm GuYqw6DssBviCyE6omrJfEGjM/EXT+d7pBMKzZGNot10V0kH7mfNcK806qJ2nlywqR/NLSW06ZIY9 F/KxbB5lO/NgCJkHLCEa/4JBOPRWI0ii5RXm83G5+g11skbEOSFDHdqf04no6EebpxKXGDgpm93h/ owdgIkFNX/8aa7XLdXXC9DgyxeIFNTpIwc1Tfd+6wQray3Z3MTznp4Gkq4fSeg6q+qWFLRizJkMok pLQ3wyCg1Dul5mP7bGLg==; Received: from localhost ([::1] helo=bombadil.infradead.org) by bombadil.infradead.org with esmtp (Exim 4.98 #2 (Red Hat Linux)) id 1tPf3t-00000009iTX-0iVI; Mon, 23 Dec 2024 09:49:17 +0000 Received: from mail-pg1-x52d.google.com ([2607:f8b0:4864:20::52d]) by bombadil.infradead.org with esmtps (Exim 4.98 #2 (Red Hat Linux)) id 1tPeyt-00000009gRw-0xt6 for linux-riscv@lists.infradead.org; Mon, 23 Dec 2024 09:44:08 +0000 Received: by mail-pg1-x52d.google.com with SMTP id 41be03b00d2f7-7fc99fc2b16so2431288a12.3 for ; Mon, 23 Dec 2024 01:44:07 -0800 (PST) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=bytedance.com; s=google; t=1734947047; x=1735551847; darn=lists.infradead.org; h=content-transfer-encoding:mime-version:references:in-reply-to :message-id:date:subject:cc:to:from:from:to:cc:subject:date :message-id:reply-to; bh=eYmLHVK0AtU6ilI7ucnXt5jLNeYE7S1uMrvnVL/VK/o=; b=IIm/baMhWvQfvQ5iFy2R44CbxdR8F3uP791p9uNvvS/Nb3tLall8WATSIFTw/WZZpT zGzqAJTn/xEIDZzmFRH4237i+LtStaPSrdduWCWqzUj/TJldo/sz0Od8UPmrp8LoJmVd Dka3r2QWKlVrBX3JqwJSW7ou3OaXPLwPlrHTFB2E0ilgNZn8v0Gmx7en+JeNSjLvVhbl JlbN+AhhZXionScNsbU64PJR+RrfV1KOIz+t5vebjVinLj+JxKgVhhaQdj0ZCLbkqQCv PXs/kJw8sQLYHKDR5jBpRts2Hr2a7MEq33vttBJPxlyrsHlR4T7qtZnzZuSqKqNHZMkX JnJA== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20230601; t=1734947047; x=1735551847; h=content-transfer-encoding:mime-version:references:in-reply-to :message-id:date:subject:cc:to:from:x-gm-message-state:from:to:cc :subject:date:message-id:reply-to; bh=eYmLHVK0AtU6ilI7ucnXt5jLNeYE7S1uMrvnVL/VK/o=; b=IwPLBbi87W2xtNxjATtfooNfP5UU2LpKL4nDTPMVuQi/pzQ3ovLukv/RmufLevsO10 5x1yMZRN6ULT8aw8hl8FEl1ftPwZO223nqfGfx/JxPTQBy7bAd62wGDQIJLKO9YqS7oB Cij6HyNx/BFa4JqSXpYZ5wL2PhvO6wHDuHinbhpIKlwr2DcBsrFZ69xcAgvgKADilWTX /uJbxqaqAQaoiNHXkrswQavMCwCAcYKy68fPbygXLeZv9GGsFu8YEK5YrfMW0UKvL5Po O3HoqV7DM0Q7gfJCGn2yTwKVmo2dWzPPqLmFk80RkD8hhlZ7WsfGcuKTeHs/dra9SQwO j+3A== X-Forwarded-Encrypted: i=1; AJvYcCVZA6gHjzbBH2xi2A8XtcVBMd2pok+cNWME248iiu0HmrAv4pFZDZeiLaGG70JukceA95P6a6HSpMG0Gw==@lists.infradead.org X-Gm-Message-State: AOJu0Yxjz67YRCBJnUPdPQq4S/gquIhGojs3l4ZYe5o19+SbqwxkHeBA rbdk7lHaxHapsVRb5bRyKAVqiuHsbuhUHrucPCBgUWNni9YLzpSyexVQFsPQA/A= X-Gm-Gg: ASbGncuo39RaHgD1C7MEf3U21DXxYgLyqaQueHMmzC+iTOE8eLkcB0LC1XT6571RWwh 5obxtqAH+FcPp2bRMtCnh1dfovk8hAV6yUa3bd+vcMEqvlxhzF8iUUDfUzpYN57m/kQTpq6t6E1 0kcscudfmRltk8SguxHCy70NXhMNVPUVOUq+mswiSt3b1j3bbIZQGatECK1JwJdw+xcku2z+qF0 3ZxiZnksJ77yKS3Sm3pCedybnD7+tTdJGiO0c74FsAa6X7N6esUM3j4iAJGI2phm6juZX4UMUbh SUbWRYCGPSDFDr2OJkXqVA== X-Google-Smtp-Source: AGHT+IEE3DAo82DBKWUM6FD2Qxwd8GTGkBziQ/dfcNtaRbY+efEiB7FXaTM2aY8g2l19VVq60zkLag== X-Received: by 2002:a05:6a20:7f82:b0:1e0:d575:8d4f with SMTP id adf61e73a8af0-1e5e081c5acmr19810935637.37.1734947046731; Mon, 23 Dec 2024 01:44:06 -0800 (PST) Received: from C02DW0BEMD6R.bytedance.net ([203.208.167.150]) by smtp.gmail.com with ESMTPSA id d2e1a72fcca58-72aad8486c6sm7468309b3a.85.2024.12.23.01.43.54 (version=TLS1_3 cipher=TLS_AES_256_GCM_SHA384 bits=256/256); Mon, 23 Dec 2024 01:44:06 -0800 (PST) From: Qi Zheng To: peterz@infradead.org, agordeev@linux.ibm.com, kevin.brodsky@arm.com, tglx@linutronix.de, david@redhat.com, jannh@google.com, hughd@google.com, yuzhao@google.com, willy@infradead.org, muchun.song@linux.dev, vbabka@kernel.org, lorenzo.stoakes@oracle.com, akpm@linux-foundation.org, rientjes@google.com, vishal.moola@gmail.com, arnd@arndb.de, will@kernel.org, aneesh.kumar@kernel.org, npiggin@gmail.com, dave.hansen@linux.intel.com, rppt@kernel.org, ryan.roberts@arm.com Cc: linux-mm@kvack.org, linux-arm-kernel@lists.infradead.org, linuxppc-dev@lists.ozlabs.org, linux-riscv@lists.infradead.org, linux-s390@vger.kernel.org, sparclinux@vger.kernel.org, linux-kernel@vger.kernel.org, x86@kernel.org, linux-arch@vger.kernel.org, linux-csky@vger.kernel.org, linux-hexagon@vger.kernel.org, loongarch@lists.linux.dev, linux-m68k@lists.linux-m68k.org, linux-mips@vger.kernel.org, linux-openrisc@vger.kernel.org, linux-sh@vger.kernel.org, linux-um@lists.infradead.org, Qi Zheng Subject: [PATCH v3 05/17] arm64: pgtable: use mmu gather to free p4d level page table Date: Mon, 23 Dec 2024 17:40:51 +0800 Message-Id: <7c12112047ac230809aacd0379259414b9b0d3a3.1734945104.git.zhengqi.arch@bytedance.com> X-Mailer: git-send-email 2.24.3 (Apple Git-128) In-Reply-To: References: MIME-Version: 1.0 X-CRM114-Version: 20100106-BlameMichelson ( TRE 0.8.0 (BSD) ) MR-646709E3 X-CRM114-CacheID: sfid-20241223_014407_284542_690BC7FB X-CRM114-Status: UNSURE ( 8.50 ) X-CRM114-Notice: Please train this message. X-BeenThere: linux-riscv@lists.infradead.org X-Mailman-Version: 2.1.34 Precedence: list List-Id: List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Sender: "linux-riscv" Errors-To: linux-riscv-bounces+linux-riscv=archiver.kernel.org@lists.infradead.org Like other levels of page tables, also use mmu gather mechanism to free p4d level page table. Signed-off-by: Qi Zheng Originally-by: Peter Zijlstra (Intel) Cc: linux-arm-kernel@lists.infradead.org --- arch/arm64/include/asm/pgalloc.h | 1 - arch/arm64/include/asm/tlb.h | 14 ++++++++++++++ 2 files changed, 14 insertions(+), 1 deletion(-) diff --git a/arch/arm64/include/asm/pgalloc.h b/arch/arm64/include/asm/pgalloc.h index 2965f5a7e39e3..1b4509d3382c6 100644 --- a/arch/arm64/include/asm/pgalloc.h +++ b/arch/arm64/include/asm/pgalloc.h @@ -85,7 +85,6 @@ static inline void pgd_populate(struct mm_struct *mm, pgd_t *pgdp, p4d_t *p4dp) __pgd_populate(pgdp, __pa(p4dp), pgdval); } -#define __p4d_free_tlb(tlb, p4d, addr) p4d_free((tlb)->mm, p4d) #else static inline void __pgd_populate(pgd_t *pgdp, phys_addr_t p4dp, pgdval_t prot) { diff --git a/arch/arm64/include/asm/tlb.h b/arch/arm64/include/asm/tlb.h index a947c6e784ed2..445282cde9afb 100644 --- a/arch/arm64/include/asm/tlb.h +++ b/arch/arm64/include/asm/tlb.h @@ -111,4 +111,18 @@ static inline void __pud_free_tlb(struct mmu_gather *tlb, pud_t *pudp, } #endif +#if CONFIG_PGTABLE_LEVELS > 4 +static inline void __p4d_free_tlb(struct mmu_gather *tlb, p4d_t *p4dp, + unsigned long addr) +{ + struct ptdesc *ptdesc = virt_to_ptdesc(p4dp); + + if (!pgtable_l5_enabled()) + return; + + pagetable_p4d_dtor(ptdesc); + tlb_remove_ptdesc(tlb, ptdesc); +} +#endif + #endif From patchwork Mon Dec 23 09:40:52 2024 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Qi Zheng X-Patchwork-Id: 13918706 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from bombadil.infradead.org (bombadil.infradead.org [198.137.202.133]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (No client certificate requested) by smtp.lore.kernel.org (Postfix) with ESMTPS id C1D5BE7718B for ; Mon, 23 Dec 2024 09:50:32 +0000 (UTC) DKIM-Signature: v=1; a=rsa-sha256; q=dns/txt; c=relaxed/relaxed; d=lists.infradead.org; s=bombadil.20210309; h=Sender: Content-Transfer-Encoding:Content-Type:List-Subscribe:List-Help:List-Post: List-Archive:List-Unsubscribe:List-Id:MIME-Version:References:In-Reply-To: Message-Id:Date:Subject:Cc:To:From:Reply-To:Content-ID:Content-Description: Resent-Date:Resent-From:Resent-Sender:Resent-To:Resent-Cc:Resent-Message-ID: List-Owner; bh=yvwMHI/S+/LK5dOZNPu8JgLIeteGJaRgCMSQfsKjOsk=; b=LIxmsF2/W/Ga/4 qJ6osLfizhWoUEmsM/X/ZJbMq8o+bE7Jcx0o2dULFBDOKhHlftxD2/hiV6pKrrBKRoCaa4HWoWITj uJu7Yvff4CVPmVhbJIuoExtJEymzYlkLOlflQFrfKgCnhJ+C8HxaYFODKXRuwAzjrLlf5ab2+exgR z0zqU7a/xlPbcmoQ4T9xyjUVmvJDt0+IrUIOTxfuP6pftS/zDnqLauL9hPvFeVBF2R/GwETwx3QAu gTsjTe5NpA+06XLTvjN9QiBhLKJmz2N/8up2VMPrlMma2t/2QqWYrTkB99DqhEgAIxCwWxh2MDyV/ BeVMxJFu5X7f2FGzFV2w==; Received: from localhost ([::1] helo=bombadil.infradead.org) by bombadil.infradead.org with esmtp (Exim 4.98 #2 (Red Hat Linux)) id 1tPf52-00000009itP-23EN; Mon, 23 Dec 2024 09:50:28 +0000 Received: from mail-pf1-x432.google.com ([2607:f8b0:4864:20::432]) by bombadil.infradead.org with esmtps (Exim 4.98 #2 (Red Hat Linux)) id 1tPez6-00000009gWl-2BTA for linux-riscv@lists.infradead.org; Mon, 23 Dec 2024 09:44:22 +0000 Received: by mail-pf1-x432.google.com with SMTP id d2e1a72fcca58-72764c995e5so2420483b3a.2 for ; Mon, 23 Dec 2024 01:44:20 -0800 (PST) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=bytedance.com; s=google; t=1734947060; x=1735551860; darn=lists.infradead.org; h=content-transfer-encoding:mime-version:references:in-reply-to :message-id:date:subject:cc:to:from:from:to:cc:subject:date :message-id:reply-to; bh=xY41yoAggnrpKZDOvMIy0+FhpPRF9QFn+bDK7HKPzWw=; b=i+WTVhdo3Kw+Qe62tI/pXWrKmBDLKaRAxA5NP1PtFq6LfV/HS7I5a2bXsiezt3/jlA zOivCtcqC/ItjW4Ejba5S/Fzi32ztC4Fox1L9kRL+OoQWHaODfUgZSkp6N8h21c6DF7O jGX2CQHAvMYFaQ2TIcPI9uAP8RJDJHClnVB1K56V8ld38+l5hgre/+ZHsyzizjYB69Qo NEyE/8XbKPJunC6kVwJAibJAmHiDforg4xDSBSla1xqqHStnQ361Yvidt39riHIPN3Yd dzPD5k+mP2U8AaulCs0Tm3Olp6lknwpwcBLRWBbEOjHKFW9znvN6qYcGNkFiDq7l3nU2 3GSg== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20230601; t=1734947060; x=1735551860; h=content-transfer-encoding:mime-version:references:in-reply-to :message-id:date:subject:cc:to:from:x-gm-message-state:from:to:cc :subject:date:message-id:reply-to; bh=xY41yoAggnrpKZDOvMIy0+FhpPRF9QFn+bDK7HKPzWw=; b=EY10hcdh7Lp8cyw9jxNqsF8FDvDxTUA9UdNhJti6SfVWbaT0KKUetyv3mtBfLSgOTj 5dwpQFHbHgc1H6cSdtYKYFJsqgrrpIVH/RMyhOSAnOwnAu9m2EyhJzRCr3z1j84lCTg/ P0hPdpJtxkypInn65ncgSyu4eDa4unYuyHeu92NbJlo5V+VsSPd4y1FEy4/61eITyVqN soJXjw8bJHJKQ7qPE/6DLffrRp7FiWmEOfBWYyUuMr2tA9k1nGKeosksJMzFsZOi6Wnk 3D1Pb//KYRvddfu8u/0Lgzgg+pT+8j3KborNokdfg/9HQ320O5mAuW2LA6NHQT0BwTq6 Ik4w== X-Forwarded-Encrypted: i=1; AJvYcCX+xphq9AsxFjcNuCfJzGc3l5NyWc548/nzZSTt8c//z8lYn7rLlDZcv9jqsMiGcHzSn+Aaa6b1dwcZnA==@lists.infradead.org X-Gm-Message-State: AOJu0YxAmslHXzClUyYQjFwqJJz9DeS/sWXTqcZFJ1m6g3LIGclfSOjY T1bU7hSuCen56GcG3gzy904i1Sjn1hcmL1frNYmy02nNEoft0aWpuIG1YIYbZyM= X-Gm-Gg: ASbGncsOiF6cp+RvBDY5cB7czJeZ09TkE1qy5bN6bMJgsbHfPhum+XI5WKg2m4R4uda zwTLVtF3Ap/60egfY2OH0kUrLbhOTOtYBYIHGNGeZ9LdFEttqsTSy/UFZ9YztFYCV1+sngg4yRH VCMFt+rW6THrF2yZQq50QzO9tZ79h0bq19SWgprMw9rAChPcTUAgBUNtTjY+cwfR0K6WPfqkLK7 b98cxbuGI2dEWgC8dabHrFGi7jqKs3TuMRx4a1reThMBpbZn30Dyqw2Jj6LFs4Jk34o7UaTezxD ZDWv5265qlchKEiDLls5yg== X-Google-Smtp-Source: AGHT+IF7O/ZMCvOOtywTANPL1zwoTq06yXXDMhnBrPgrfov3E/r1DzhOhOYD8wImAojzeP7TdceoFw== X-Received: by 2002:a05:6a21:3285:b0:1e1:aad7:d50d with SMTP id adf61e73a8af0-1e5e084b681mr22018514637.46.1734947059944; Mon, 23 Dec 2024 01:44:19 -0800 (PST) Received: from C02DW0BEMD6R.bytedance.net ([203.208.167.150]) by smtp.gmail.com with ESMTPSA id d2e1a72fcca58-72aad8486c6sm7468309b3a.85.2024.12.23.01.44.07 (version=TLS1_3 cipher=TLS_AES_256_GCM_SHA384 bits=256/256); Mon, 23 Dec 2024 01:44:19 -0800 (PST) From: Qi Zheng To: peterz@infradead.org, agordeev@linux.ibm.com, kevin.brodsky@arm.com, tglx@linutronix.de, david@redhat.com, jannh@google.com, hughd@google.com, yuzhao@google.com, willy@infradead.org, muchun.song@linux.dev, vbabka@kernel.org, lorenzo.stoakes@oracle.com, akpm@linux-foundation.org, rientjes@google.com, vishal.moola@gmail.com, arnd@arndb.de, will@kernel.org, aneesh.kumar@kernel.org, npiggin@gmail.com, dave.hansen@linux.intel.com, rppt@kernel.org, ryan.roberts@arm.com Cc: linux-mm@kvack.org, linux-arm-kernel@lists.infradead.org, linuxppc-dev@lists.ozlabs.org, linux-riscv@lists.infradead.org, linux-s390@vger.kernel.org, sparclinux@vger.kernel.org, linux-kernel@vger.kernel.org, x86@kernel.org, linux-arch@vger.kernel.org, linux-csky@vger.kernel.org, linux-hexagon@vger.kernel.org, loongarch@lists.linux.dev, linux-m68k@lists.linux-m68k.org, linux-mips@vger.kernel.org, linux-openrisc@vger.kernel.org, linux-sh@vger.kernel.org, linux-um@lists.infradead.org, Qi Zheng Subject: [PATCH v3 06/17] s390: pgtable: add statistics for PUD and P4D level page table Date: Mon, 23 Dec 2024 17:40:52 +0800 Message-Id: <35be22a2b1666df729a9fc108c2da5cce266e4be.1734945104.git.zhengqi.arch@bytedance.com> X-Mailer: git-send-email 2.24.3 (Apple Git-128) In-Reply-To: References: MIME-Version: 1.0 X-CRM114-Version: 20100106-BlameMichelson ( TRE 0.8.0 (BSD) ) MR-646709E3 X-CRM114-CacheID: sfid-20241223_014420_568280_D31F45A3 X-CRM114-Status: GOOD ( 11.08 ) X-BeenThere: linux-riscv@lists.infradead.org X-Mailman-Version: 2.1.34 Precedence: list List-Id: List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Sender: "linux-riscv" Errors-To: linux-riscv-bounces+linux-riscv=archiver.kernel.org@lists.infradead.org Like PMD and PTE level page table, also add statistics for PUD and P4D page table. Signed-off-by: Qi Zheng Suggested-by: Peter Zijlstra (Intel) Cc: linux-s390@vger.kernel.org --- arch/s390/include/asm/pgalloc.h | 29 +++++++++++++++++++------- arch/s390/include/asm/tlb.h | 37 +++++++++++++++++---------------- 2 files changed, 40 insertions(+), 26 deletions(-) diff --git a/arch/s390/include/asm/pgalloc.h b/arch/s390/include/asm/pgalloc.h index 7b84ef6dc4b6d..a0c1ca5d8423c 100644 --- a/arch/s390/include/asm/pgalloc.h +++ b/arch/s390/include/asm/pgalloc.h @@ -53,29 +53,42 @@ static inline p4d_t *p4d_alloc_one(struct mm_struct *mm, unsigned long address) { unsigned long *table = crst_table_alloc(mm); - if (table) - crst_table_init(table, _REGION2_ENTRY_EMPTY); + if (!table) + return NULL; + crst_table_init(table, _REGION2_ENTRY_EMPTY); + pagetable_p4d_ctor(virt_to_ptdesc(table)); + return (p4d_t *) table; } static inline void p4d_free(struct mm_struct *mm, p4d_t *p4d) { - if (!mm_p4d_folded(mm)) - crst_table_free(mm, (unsigned long *) p4d); + if (mm_p4d_folded(mm)) + return; + + pagetable_p4d_dtor(virt_to_ptdesc(p4d)); + crst_table_free(mm, (unsigned long *) p4d); } static inline pud_t *pud_alloc_one(struct mm_struct *mm, unsigned long address) { unsigned long *table = crst_table_alloc(mm); - if (table) - crst_table_init(table, _REGION3_ENTRY_EMPTY); + + if (!table) + return NULL; + crst_table_init(table, _REGION3_ENTRY_EMPTY); + pagetable_pud_ctor(virt_to_ptdesc(table)); + return (pud_t *) table; } static inline void pud_free(struct mm_struct *mm, pud_t *pud) { - if (!mm_pud_folded(mm)) - crst_table_free(mm, (unsigned long *) pud); + if (mm_pud_folded(mm)) + return; + + pagetable_pud_dtor(virt_to_ptdesc(pud)); + crst_table_free(mm, (unsigned long *) pud); } static inline pmd_t *pmd_alloc_one(struct mm_struct *mm, unsigned long vmaddr) diff --git a/arch/s390/include/asm/tlb.h b/arch/s390/include/asm/tlb.h index e95b2c8081eb8..b946964afce8e 100644 --- a/arch/s390/include/asm/tlb.h +++ b/arch/s390/include/asm/tlb.h @@ -110,24 +110,6 @@ static inline void pmd_free_tlb(struct mmu_gather *tlb, pmd_t *pmd, tlb_remove_ptdesc(tlb, pmd); } -/* - * p4d_free_tlb frees a pud table and clears the CRSTE for the - * region second table entry from the tlb. - * If the mm uses a four level page table the single p4d is freed - * as the pgd. p4d_free_tlb checks the asce_limit against 8PB - * to avoid the double free of the p4d in this case. - */ -static inline void p4d_free_tlb(struct mmu_gather *tlb, p4d_t *p4d, - unsigned long address) -{ - if (mm_p4d_folded(tlb->mm)) - return; - __tlb_adjust_range(tlb, address, PAGE_SIZE); - tlb->mm->context.flush_mm = 1; - tlb->freed_tables = 1; - tlb_remove_ptdesc(tlb, p4d); -} - /* * pud_free_tlb frees a pud table and clears the CRSTE for the * region third table entry from the tlb. @@ -140,11 +122,30 @@ static inline void pud_free_tlb(struct mmu_gather *tlb, pud_t *pud, { if (mm_pud_folded(tlb->mm)) return; + pagetable_pud_dtor(virt_to_ptdesc(pud)); tlb->mm->context.flush_mm = 1; tlb->freed_tables = 1; tlb->cleared_p4ds = 1; tlb_remove_ptdesc(tlb, pud); } +/* + * p4d_free_tlb frees a p4d table and clears the CRSTE for the + * region second table entry from the tlb. + * If the mm uses a four level page table the single p4d is freed + * as the pgd. p4d_free_tlb checks the asce_limit against 8PB + * to avoid the double free of the p4d in this case. + */ +static inline void p4d_free_tlb(struct mmu_gather *tlb, p4d_t *p4d, + unsigned long address) +{ + if (mm_p4d_folded(tlb->mm)) + return; + pagetable_p4d_dtor(virt_to_ptdesc(p4d)); + __tlb_adjust_range(tlb, address, PAGE_SIZE); + tlb->mm->context.flush_mm = 1; + tlb->freed_tables = 1; + tlb_remove_ptdesc(tlb, p4d); +} #endif /* _S390_TLB_H */ From patchwork Mon Dec 23 09:40:53 2024 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Qi Zheng X-Patchwork-Id: 13918763 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from bombadil.infradead.org (bombadil.infradead.org [198.137.202.133]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (No client certificate requested) by smtp.lore.kernel.org (Postfix) with ESMTPS id 2B1EDE7718F for ; Mon, 23 Dec 2024 11:06:16 +0000 (UTC) DKIM-Signature: v=1; a=rsa-sha256; q=dns/txt; c=relaxed/relaxed; d=lists.infradead.org; s=bombadil.20210309; h=Sender: Content-Transfer-Encoding:Content-Type:List-Subscribe:List-Help:List-Post: List-Archive:List-Unsubscribe:List-Id:MIME-Version:References:In-Reply-To: Message-Id:Date:Subject:Cc:To:From:Reply-To:Content-ID:Content-Description: Resent-Date:Resent-From:Resent-Sender:Resent-To:Resent-Cc:Resent-Message-ID: List-Owner; bh=to2lPRtdrrT0NP3b6dLX7YnxYq6lDwGjp84Rw0DfZ20=; b=IUm8YLIIxjcxyu VUHu0CQAXeFz+QxFtiOY+slQHZOgSiB3TiraP9HNXIh474t7WoUidRGLX9UzeqcdrBDr6rnb5JFpB IPxVT5a5Ii3yDyS/lvft+uIm8yP1gYYGPL+dMcPb8/JUisYOwJNMApW2tOguD5Hvl8ehZoX2tEyVT jEEpj7GDB68fHJ6inirbUtVptGA1uY/TMNPa3ab0j/2QNKthUlBQsFuOODU/t58zT+RUKQcnZhwYw ZC09vcVizNH5WdesvG/Ch3jiJGgniFogP2Lf4OrNQts7KXDiH98tmihu9oO8rT4cV7oyS9B9v4oQ3 M4G+RDZuIgwQjEEgwVGw==; Received: from localhost ([::1] helo=bombadil.infradead.org) by bombadil.infradead.org with esmtp (Exim 4.98 #2 (Red Hat Linux)) id 1tPgGG-00000009qHd-0MIA; Mon, 23 Dec 2024 11:06:08 +0000 Received: from mail-pf1-x429.google.com ([2607:f8b0:4864:20::429]) by bombadil.infradead.org with esmtps (Exim 4.98 #2 (Red Hat Linux)) id 1tPezJ-00000009ggR-1rYg for linux-riscv@lists.infradead.org; Mon, 23 Dec 2024 09:44:35 +0000 Received: by mail-pf1-x429.google.com with SMTP id d2e1a72fcca58-728eccf836bso3361571b3a.1 for ; Mon, 23 Dec 2024 01:44:33 -0800 (PST) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=bytedance.com; s=google; t=1734947073; x=1735551873; darn=lists.infradead.org; h=content-transfer-encoding:mime-version:references:in-reply-to :message-id:date:subject:cc:to:from:from:to:cc:subject:date :message-id:reply-to; bh=if0a5qgfKyLZEs2UFcbOJ2pRLoezhNdJsDQpJdHhbkw=; b=hgZopQ/5E/36VWfHKDyvHYpt6vBY+bc+vL4y3gg3dX5nwxRILsgo1jDIGviyC/xnFz PapVJh/nGkI/XpHQWe+39Zb5a4/NqWktmIPCMTLgEjOe9+bUgwfyAXCHp4ZMY0iuFWp1 PXF0++SRU6D4ui4h4MJuy5VrafxlL+uxrWLRyqniZSIl9rqm4rJnXGwmuiZYTEc7nt0F dQRRoXQa43x44SgWGC3nQVhn4oCaTo+2DscAM1UmFM+YM77CafG+ERoooYhYQtEdbYDg 2aCf4DVMvnm+F9BRLqtj2daI6rc7+T2GuLTYwVIBpGgybD1n03IZ05AMLLVy9Ugdv3PF gzrQ== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20230601; t=1734947073; x=1735551873; h=content-transfer-encoding:mime-version:references:in-reply-to :message-id:date:subject:cc:to:from:x-gm-message-state:from:to:cc :subject:date:message-id:reply-to; bh=if0a5qgfKyLZEs2UFcbOJ2pRLoezhNdJsDQpJdHhbkw=; b=ityEcBonYK9OwhgaCW6qxaSbCnLtH/Jtxl/+lVUqQ7OVWH/2WZD0P+V2a0ZahScvmc g55QSwyeHxZyEh10BUdORKtsMSIP8fbPP7ebpfw3Uf59Fc9sWa+caNe0GlAM/C9RPgTm jlRDIqnRtpW6sdoESpY2Ms9y95pReN7GlBXOGxwTkILtOCDFYvsyqNNJ/tDvGCBwQm2j S2n+rCidLWYDFttAQZQxsLPSfm417TY13hUVs2URP07J86niTHRcvYGF3hlhWOlssuG1 OSYxgzARZrRXvD8bykm47NUuCACNSCCk86ouK/XsfqQEKkBuZ7Z3wUdG4hjHDhnq1iwN ojfw== X-Forwarded-Encrypted: i=1; AJvYcCXhHVZlWtTJtSgZrMbLrzJ7qvUCgzd7vKYLg+BF/7PNDZIYnpiSPKyx4PWROHwfQcRjV2jdtNiixKmZqQ==@lists.infradead.org X-Gm-Message-State: AOJu0Yyt/baSQu8PktVyHQAJdln9yo96uU82X4sKU4o5oxA9XJDwWztJ SyKGEHHq6zbJl7G5tVjuYVGuooHiTm5sQkQ1r+mCCq/OY0EkddtLjXt2Or3OUG4= X-Gm-Gg: ASbGncuhegdJ+OSGyOqiC18h7GNce6SZAan5+U6xYV+TtUdyN9ZIkXy3S9Upzo7KxPJ In8/v1IfiWc7joaR1H8MxfCa3Lg5w8u2b9RPa2uURoO2o0xIek0k2e2XqMq6kysrcUxl2qcYANg tNl9n9Q+DGpep9rK6R2vS9yM5fYntDHAYthgr41RwmL+33v3lQ0yR9CU0jb0t8u6uUa8b8EWjuH 9MOqfpLyxcH0myL2b6U0znS/FSplZtvcP16PLZuRkUXUI/je3mQHiAShANZxN6pNrZJmdncIkwm 3Sz3oTUFX/JGpMWbVNeDoA== X-Google-Smtp-Source: AGHT+IHNcG21n8t2Tfuvp8LnDtpoSzOkwaxGY/BugzvHG6e31xXuyXvNhN+O2xx2uo+0V3sMDp7pew== X-Received: by 2002:a05:6a21:6f02:b0:1e5:b082:e38f with SMTP id adf61e73a8af0-1e5e081d16cmr18629210637.45.1734947072789; Mon, 23 Dec 2024 01:44:32 -0800 (PST) Received: from C02DW0BEMD6R.bytedance.net ([203.208.167.150]) by smtp.gmail.com with ESMTPSA id d2e1a72fcca58-72aad8486c6sm7468309b3a.85.2024.12.23.01.44.20 (version=TLS1_3 cipher=TLS_AES_256_GCM_SHA384 bits=256/256); Mon, 23 Dec 2024 01:44:32 -0800 (PST) From: Qi Zheng To: peterz@infradead.org, agordeev@linux.ibm.com, kevin.brodsky@arm.com, tglx@linutronix.de, david@redhat.com, jannh@google.com, hughd@google.com, yuzhao@google.com, willy@infradead.org, muchun.song@linux.dev, vbabka@kernel.org, lorenzo.stoakes@oracle.com, akpm@linux-foundation.org, rientjes@google.com, vishal.moola@gmail.com, arnd@arndb.de, will@kernel.org, aneesh.kumar@kernel.org, npiggin@gmail.com, dave.hansen@linux.intel.com, rppt@kernel.org, ryan.roberts@arm.com Cc: linux-mm@kvack.org, linux-arm-kernel@lists.infradead.org, linuxppc-dev@lists.ozlabs.org, linux-riscv@lists.infradead.org, linux-s390@vger.kernel.org, sparclinux@vger.kernel.org, linux-kernel@vger.kernel.org, x86@kernel.org, linux-arch@vger.kernel.org, linux-csky@vger.kernel.org, linux-hexagon@vger.kernel.org, loongarch@lists.linux.dev, linux-m68k@lists.linux-m68k.org, linux-mips@vger.kernel.org, linux-openrisc@vger.kernel.org, linux-sh@vger.kernel.org, linux-um@lists.infradead.org, Qi Zheng Subject: [PATCH v3 07/17] mm: pgtable: introduce pagetable_dtor() Date: Mon, 23 Dec 2024 17:40:53 +0800 Message-Id: <8ada95453180c71b7fca92b9a9f11fa0f92d45a6.1734945104.git.zhengqi.arch@bytedance.com> X-Mailer: git-send-email 2.24.3 (Apple Git-128) In-Reply-To: References: MIME-Version: 1.0 X-CRM114-Version: 20100106-BlameMichelson ( TRE 0.8.0 (BSD) ) MR-646709E3 X-CRM114-CacheID: sfid-20241223_014433_522856_E5608463 X-CRM114-Status: GOOD ( 14.27 ) X-BeenThere: linux-riscv@lists.infradead.org X-Mailman-Version: 2.1.34 Precedence: list List-Id: List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Sender: "linux-riscv" Errors-To: linux-riscv-bounces+linux-riscv=archiver.kernel.org@lists.infradead.org The pagetable_p*_dtor() are exactly the same except for the handling of ptlock. If we make ptlock_free() handle the case where ptdesc->ptl is NULL and remove VM_BUG_ON_PAGE() from pmd_ptlock_free(), we can unify pagetable_p*_dtor() into one function. Let's introduce pagetable_dtor() to do this. Later, pagetable_dtor() will be moved to tlb_remove_ptdesc(), so that ptlock and page table pages can be freed together (regardless of whether RCU is used). This prevents the use-after-free problem where the ptlock is freed immediately but the page table pages is freed later via RCU. Signed-off-by: Qi Zheng Originally-by: Peter Zijlstra (Intel) --- Documentation/mm/split_page_table_lock.rst | 4 +- arch/arm/include/asm/tlb.h | 4 +- arch/arm64/include/asm/tlb.h | 8 ++-- arch/csky/include/asm/pgalloc.h | 2 +- arch/hexagon/include/asm/pgalloc.h | 2 +- arch/loongarch/include/asm/pgalloc.h | 2 +- arch/m68k/include/asm/mcf_pgalloc.h | 4 +- arch/m68k/include/asm/sun3_pgalloc.h | 2 +- arch/m68k/mm/motorola.c | 2 +- arch/mips/include/asm/pgalloc.h | 2 +- arch/nios2/include/asm/pgalloc.h | 2 +- arch/openrisc/include/asm/pgalloc.h | 2 +- arch/powerpc/mm/book3s64/mmu_context.c | 2 +- arch/powerpc/mm/book3s64/pgtable.c | 2 +- arch/powerpc/mm/pgtable-frag.c | 4 +- arch/riscv/include/asm/pgalloc.h | 8 ++-- arch/riscv/mm/init.c | 4 +- arch/s390/include/asm/pgalloc.h | 6 +-- arch/s390/include/asm/tlb.h | 6 +-- arch/s390/mm/pgalloc.c | 2 +- arch/sh/include/asm/pgalloc.h | 2 +- arch/sparc/mm/init_64.c | 2 +- arch/sparc/mm/srmmu.c | 2 +- arch/um/include/asm/pgalloc.h | 6 +-- arch/x86/mm/pgtable.c | 12 ++--- include/asm-generic/pgalloc.h | 8 ++-- include/linux/mm.h | 52 ++++------------------ mm/memory.c | 3 +- 28 files changed, 62 insertions(+), 95 deletions(-) diff --git a/Documentation/mm/split_page_table_lock.rst b/Documentation/mm/split_page_table_lock.rst index 581446d4a4eba..8e1ceb0a6619a 100644 --- a/Documentation/mm/split_page_table_lock.rst +++ b/Documentation/mm/split_page_table_lock.rst @@ -62,7 +62,7 @@ Support of split page table lock by an architecture =================================================== There's no need in special enabling of PTE split page table lock: everything -required is done by pagetable_pte_ctor() and pagetable_pte_dtor(), which +required is done by pagetable_pte_ctor() and pagetable_dtor(), which must be called on PTE table allocation / freeing. Make sure the architecture doesn't use slab allocator for page table @@ -73,7 +73,7 @@ PMD split lock only makes sense if you have more than two page table levels. PMD split lock enabling requires pagetable_pmd_ctor() call on PMD table -allocation and pagetable_pmd_dtor() on freeing. +allocation and pagetable_dtor() on freeing. Allocation usually happens in pmd_alloc_one(), freeing in pmd_free() and pmd_free_tlb(), but make sure you cover all PMD table allocation / freeing diff --git a/arch/arm/include/asm/tlb.h b/arch/arm/include/asm/tlb.h index f40d06ad5d2a3..ef79bf1e8563f 100644 --- a/arch/arm/include/asm/tlb.h +++ b/arch/arm/include/asm/tlb.h @@ -41,7 +41,7 @@ __pte_free_tlb(struct mmu_gather *tlb, pgtable_t pte, unsigned long addr) { struct ptdesc *ptdesc = page_ptdesc(pte); - pagetable_pte_dtor(ptdesc); + pagetable_dtor(ptdesc); #ifndef CONFIG_ARM_LPAE /* @@ -61,7 +61,7 @@ __pmd_free_tlb(struct mmu_gather *tlb, pmd_t *pmdp, unsigned long addr) #ifdef CONFIG_ARM_LPAE struct ptdesc *ptdesc = virt_to_ptdesc(pmdp); - pagetable_pmd_dtor(ptdesc); + pagetable_dtor(ptdesc); tlb_remove_ptdesc(tlb, ptdesc); #endif } diff --git a/arch/arm64/include/asm/tlb.h b/arch/arm64/include/asm/tlb.h index 445282cde9afb..408d0f36a8a8f 100644 --- a/arch/arm64/include/asm/tlb.h +++ b/arch/arm64/include/asm/tlb.h @@ -82,7 +82,7 @@ static inline void __pte_free_tlb(struct mmu_gather *tlb, pgtable_t pte, { struct ptdesc *ptdesc = page_ptdesc(pte); - pagetable_pte_dtor(ptdesc); + pagetable_dtor(ptdesc); tlb_remove_ptdesc(tlb, ptdesc); } @@ -92,7 +92,7 @@ static inline void __pmd_free_tlb(struct mmu_gather *tlb, pmd_t *pmdp, { struct ptdesc *ptdesc = virt_to_ptdesc(pmdp); - pagetable_pmd_dtor(ptdesc); + pagetable_dtor(ptdesc); tlb_remove_ptdesc(tlb, ptdesc); } #endif @@ -106,7 +106,7 @@ static inline void __pud_free_tlb(struct mmu_gather *tlb, pud_t *pudp, if (!pgtable_l4_enabled()) return; - pagetable_pud_dtor(ptdesc); + pagetable_dtor(ptdesc); tlb_remove_ptdesc(tlb, ptdesc); } #endif @@ -120,7 +120,7 @@ static inline void __p4d_free_tlb(struct mmu_gather *tlb, p4d_t *p4dp, if (!pgtable_l5_enabled()) return; - pagetable_p4d_dtor(ptdesc); + pagetable_dtor(ptdesc); tlb_remove_ptdesc(tlb, ptdesc); } #endif diff --git a/arch/csky/include/asm/pgalloc.h b/arch/csky/include/asm/pgalloc.h index 9c84c9012e534..f1ce5b7b28f22 100644 --- a/arch/csky/include/asm/pgalloc.h +++ b/arch/csky/include/asm/pgalloc.h @@ -63,7 +63,7 @@ static inline pgd_t *pgd_alloc(struct mm_struct *mm) #define __pte_free_tlb(tlb, pte, address) \ do { \ - pagetable_pte_dtor(page_ptdesc(pte)); \ + pagetable_dtor(page_ptdesc(pte)); \ tlb_remove_page_ptdesc(tlb, page_ptdesc(pte)); \ } while (0) diff --git a/arch/hexagon/include/asm/pgalloc.h b/arch/hexagon/include/asm/pgalloc.h index 55988625e6fbc..40e42a0e71673 100644 --- a/arch/hexagon/include/asm/pgalloc.h +++ b/arch/hexagon/include/asm/pgalloc.h @@ -89,7 +89,7 @@ static inline void pmd_populate_kernel(struct mm_struct *mm, pmd_t *pmd, #define __pte_free_tlb(tlb, pte, addr) \ do { \ - pagetable_pte_dtor((page_ptdesc(pte))); \ + pagetable_dtor((page_ptdesc(pte))); \ tlb_remove_page_ptdesc((tlb), (page_ptdesc(pte))); \ } while (0) diff --git a/arch/loongarch/include/asm/pgalloc.h b/arch/loongarch/include/asm/pgalloc.h index a7b9c9e73593d..7211dff8c969e 100644 --- a/arch/loongarch/include/asm/pgalloc.h +++ b/arch/loongarch/include/asm/pgalloc.h @@ -57,7 +57,7 @@ static inline pte_t *pte_alloc_one_kernel(struct mm_struct *mm) #define __pte_free_tlb(tlb, pte, address) \ do { \ - pagetable_pte_dtor(page_ptdesc(pte)); \ + pagetable_dtor(page_ptdesc(pte)); \ tlb_remove_page_ptdesc((tlb), page_ptdesc(pte)); \ } while (0) diff --git a/arch/m68k/include/asm/mcf_pgalloc.h b/arch/m68k/include/asm/mcf_pgalloc.h index 302c5bf67179e..22d6c1fcabfb4 100644 --- a/arch/m68k/include/asm/mcf_pgalloc.h +++ b/arch/m68k/include/asm/mcf_pgalloc.h @@ -37,7 +37,7 @@ static inline void __pte_free_tlb(struct mmu_gather *tlb, pgtable_t pgtable, { struct ptdesc *ptdesc = virt_to_ptdesc(pgtable); - pagetable_pte_dtor(ptdesc); + pagetable_dtor(ptdesc); pagetable_free(ptdesc); } @@ -61,7 +61,7 @@ static inline void pte_free(struct mm_struct *mm, pgtable_t pgtable) { struct ptdesc *ptdesc = virt_to_ptdesc(pgtable); - pagetable_pte_dtor(ptdesc); + pagetable_dtor(ptdesc); pagetable_free(ptdesc); } diff --git a/arch/m68k/include/asm/sun3_pgalloc.h b/arch/m68k/include/asm/sun3_pgalloc.h index 4a137eecb6fe4..2b626cb3ad0ae 100644 --- a/arch/m68k/include/asm/sun3_pgalloc.h +++ b/arch/m68k/include/asm/sun3_pgalloc.h @@ -19,7 +19,7 @@ extern const char bad_pmd_string[]; #define __pte_free_tlb(tlb, pte, addr) \ do { \ - pagetable_pte_dtor(page_ptdesc(pte)); \ + pagetable_dtor(page_ptdesc(pte)); \ tlb_remove_page_ptdesc((tlb), page_ptdesc(pte)); \ } while (0) diff --git a/arch/m68k/mm/motorola.c b/arch/m68k/mm/motorola.c index c1761d309fc61..81715cece70c6 100644 --- a/arch/m68k/mm/motorola.c +++ b/arch/m68k/mm/motorola.c @@ -201,7 +201,7 @@ int free_pointer_table(void *table, int type) list_del(dp); mmu_page_dtor((void *)page); if (type == TABLE_PTE) - pagetable_pte_dtor(virt_to_ptdesc((void *)page)); + pagetable_dtor(virt_to_ptdesc((void *)page)); free_page (page); return 1; } else if (ptable_list[type].next != dp) { diff --git a/arch/mips/include/asm/pgalloc.h b/arch/mips/include/asm/pgalloc.h index f4440edcd8fe2..36d9805033c4b 100644 --- a/arch/mips/include/asm/pgalloc.h +++ b/arch/mips/include/asm/pgalloc.h @@ -56,7 +56,7 @@ static inline void pgd_free(struct mm_struct *mm, pgd_t *pgd) #define __pte_free_tlb(tlb, pte, address) \ do { \ - pagetable_pte_dtor(page_ptdesc(pte)); \ + pagetable_dtor(page_ptdesc(pte)); \ tlb_remove_page_ptdesc((tlb), page_ptdesc(pte)); \ } while (0) diff --git a/arch/nios2/include/asm/pgalloc.h b/arch/nios2/include/asm/pgalloc.h index ce6bb8e74271f..12a536b7bfbd4 100644 --- a/arch/nios2/include/asm/pgalloc.h +++ b/arch/nios2/include/asm/pgalloc.h @@ -30,7 +30,7 @@ extern pgd_t *pgd_alloc(struct mm_struct *mm); #define __pte_free_tlb(tlb, pte, addr) \ do { \ - pagetable_pte_dtor(page_ptdesc(pte)); \ + pagetable_dtor(page_ptdesc(pte)); \ tlb_remove_page_ptdesc((tlb), (page_ptdesc(pte))); \ } while (0) diff --git a/arch/openrisc/include/asm/pgalloc.h b/arch/openrisc/include/asm/pgalloc.h index c6a73772a5466..596e2355824e3 100644 --- a/arch/openrisc/include/asm/pgalloc.h +++ b/arch/openrisc/include/asm/pgalloc.h @@ -68,7 +68,7 @@ extern pte_t *pte_alloc_one_kernel(struct mm_struct *mm); #define __pte_free_tlb(tlb, pte, addr) \ do { \ - pagetable_pte_dtor(page_ptdesc(pte)); \ + pagetable_dtor(page_ptdesc(pte)); \ tlb_remove_page_ptdesc((tlb), (page_ptdesc(pte))); \ } while (0) diff --git a/arch/powerpc/mm/book3s64/mmu_context.c b/arch/powerpc/mm/book3s64/mmu_context.c index 1715b07c630c9..4e1e45420bd49 100644 --- a/arch/powerpc/mm/book3s64/mmu_context.c +++ b/arch/powerpc/mm/book3s64/mmu_context.c @@ -253,7 +253,7 @@ static void pmd_frag_destroy(void *pmd_frag) count = ((unsigned long)pmd_frag & ~PAGE_MASK) >> PMD_FRAG_SIZE_SHIFT; /* We allow PTE_FRAG_NR fragments from a PTE page */ if (atomic_sub_and_test(PMD_FRAG_NR - count, &ptdesc->pt_frag_refcount)) { - pagetable_pmd_dtor(ptdesc); + pagetable_dtor(ptdesc); pagetable_free(ptdesc); } } diff --git a/arch/powerpc/mm/book3s64/pgtable.c b/arch/powerpc/mm/book3s64/pgtable.c index 3745425280808..3f28e4acd920b 100644 --- a/arch/powerpc/mm/book3s64/pgtable.c +++ b/arch/powerpc/mm/book3s64/pgtable.c @@ -477,7 +477,7 @@ void pmd_fragment_free(unsigned long *pmd) BUG_ON(atomic_read(&ptdesc->pt_frag_refcount) <= 0); if (atomic_dec_and_test(&ptdesc->pt_frag_refcount)) { - pagetable_pmd_dtor(ptdesc); + pagetable_dtor(ptdesc); pagetable_free(ptdesc); } } diff --git a/arch/powerpc/mm/pgtable-frag.c b/arch/powerpc/mm/pgtable-frag.c index e89f64a0f24ae..713268ccb1a0e 100644 --- a/arch/powerpc/mm/pgtable-frag.c +++ b/arch/powerpc/mm/pgtable-frag.c @@ -25,7 +25,7 @@ void pte_frag_destroy(void *pte_frag) count = ((unsigned long)pte_frag & ~PAGE_MASK) >> PTE_FRAG_SIZE_SHIFT; /* We allow PTE_FRAG_NR fragments from a PTE page */ if (atomic_sub_and_test(PTE_FRAG_NR - count, &ptdesc->pt_frag_refcount)) { - pagetable_pte_dtor(ptdesc); + pagetable_dtor(ptdesc); pagetable_free(ptdesc); } } @@ -111,7 +111,7 @@ static void pte_free_now(struct rcu_head *head) struct ptdesc *ptdesc; ptdesc = container_of(head, struct ptdesc, pt_rcu_head); - pagetable_pte_dtor(ptdesc); + pagetable_dtor(ptdesc); pagetable_free(ptdesc); } diff --git a/arch/riscv/include/asm/pgalloc.h b/arch/riscv/include/asm/pgalloc.h index 3466fbe2e508d..b6793c5c99296 100644 --- a/arch/riscv/include/asm/pgalloc.h +++ b/arch/riscv/include/asm/pgalloc.h @@ -100,7 +100,7 @@ static inline void __pud_free_tlb(struct mmu_gather *tlb, pud_t *pud, if (pgtable_l4_enabled) { struct ptdesc *ptdesc = virt_to_ptdesc(pud); - pagetable_pud_dtor(ptdesc); + pagetable_dtor(ptdesc); riscv_tlb_remove_ptdesc(tlb, ptdesc); } } @@ -111,7 +111,7 @@ static inline void __p4d_free_tlb(struct mmu_gather *tlb, p4d_t *p4d, if (pgtable_l5_enabled) { struct ptdesc *ptdesc = virt_to_ptdesc(p4d); - pagetable_p4d_dtor(ptdesc); + pagetable_dtor(ptdesc); riscv_tlb_remove_ptdesc(tlb, virt_to_ptdesc(p4d)); } } @@ -144,7 +144,7 @@ static inline void __pmd_free_tlb(struct mmu_gather *tlb, pmd_t *pmd, { struct ptdesc *ptdesc = virt_to_ptdesc(pmd); - pagetable_pmd_dtor(ptdesc); + pagetable_dtor(ptdesc); riscv_tlb_remove_ptdesc(tlb, ptdesc); } @@ -155,7 +155,7 @@ static inline void __pte_free_tlb(struct mmu_gather *tlb, pgtable_t pte, { struct ptdesc *ptdesc = page_ptdesc(pte); - pagetable_pte_dtor(ptdesc); + pagetable_dtor(ptdesc); riscv_tlb_remove_ptdesc(tlb, ptdesc); } #endif /* CONFIG_MMU */ diff --git a/arch/riscv/mm/init.c b/arch/riscv/mm/init.c index fc53ce748c804..8d703fb51b1dc 100644 --- a/arch/riscv/mm/init.c +++ b/arch/riscv/mm/init.c @@ -1558,7 +1558,7 @@ static void __meminit free_pte_table(pte_t *pte_start, pmd_t *pmd) return; } - pagetable_pte_dtor(ptdesc); + pagetable_dtor(ptdesc); if (PageReserved(page)) free_reserved_page(page); else @@ -1580,7 +1580,7 @@ static void __meminit free_pmd_table(pmd_t *pmd_start, pud_t *pud, bool is_vmemm } if (!is_vmemmap) - pagetable_pmd_dtor(ptdesc); + pagetable_dtor(ptdesc); if (PageReserved(page)) free_reserved_page(page); else diff --git a/arch/s390/include/asm/pgalloc.h b/arch/s390/include/asm/pgalloc.h index a0c1ca5d8423c..5fced6d3c36b0 100644 --- a/arch/s390/include/asm/pgalloc.h +++ b/arch/s390/include/asm/pgalloc.h @@ -66,7 +66,7 @@ static inline void p4d_free(struct mm_struct *mm, p4d_t *p4d) if (mm_p4d_folded(mm)) return; - pagetable_p4d_dtor(virt_to_ptdesc(p4d)); + pagetable_dtor(virt_to_ptdesc(p4d)); crst_table_free(mm, (unsigned long *) p4d); } @@ -87,7 +87,7 @@ static inline void pud_free(struct mm_struct *mm, pud_t *pud) if (mm_pud_folded(mm)) return; - pagetable_pud_dtor(virt_to_ptdesc(pud)); + pagetable_dtor(virt_to_ptdesc(pud)); crst_table_free(mm, (unsigned long *) pud); } @@ -109,7 +109,7 @@ static inline void pmd_free(struct mm_struct *mm, pmd_t *pmd) { if (mm_pmd_folded(mm)) return; - pagetable_pmd_dtor(virt_to_ptdesc(pmd)); + pagetable_dtor(virt_to_ptdesc(pmd)); crst_table_free(mm, (unsigned long *) pmd); } diff --git a/arch/s390/include/asm/tlb.h b/arch/s390/include/asm/tlb.h index b946964afce8e..74b6fba4c2ee3 100644 --- a/arch/s390/include/asm/tlb.h +++ b/arch/s390/include/asm/tlb.h @@ -102,7 +102,7 @@ static inline void pmd_free_tlb(struct mmu_gather *tlb, pmd_t *pmd, { if (mm_pmd_folded(tlb->mm)) return; - pagetable_pmd_dtor(virt_to_ptdesc(pmd)); + pagetable_dtor(virt_to_ptdesc(pmd)); __tlb_adjust_range(tlb, address, PAGE_SIZE); tlb->mm->context.flush_mm = 1; tlb->freed_tables = 1; @@ -122,7 +122,7 @@ static inline void pud_free_tlb(struct mmu_gather *tlb, pud_t *pud, { if (mm_pud_folded(tlb->mm)) return; - pagetable_pud_dtor(virt_to_ptdesc(pud)); + pagetable_dtor(virt_to_ptdesc(pud)); tlb->mm->context.flush_mm = 1; tlb->freed_tables = 1; tlb->cleared_p4ds = 1; @@ -141,7 +141,7 @@ static inline void p4d_free_tlb(struct mmu_gather *tlb, p4d_t *p4d, { if (mm_p4d_folded(tlb->mm)) return; - pagetable_p4d_dtor(virt_to_ptdesc(p4d)); + pagetable_dtor(virt_to_ptdesc(p4d)); __tlb_adjust_range(tlb, address, PAGE_SIZE); tlb->mm->context.flush_mm = 1; tlb->freed_tables = 1; diff --git a/arch/s390/mm/pgalloc.c b/arch/s390/mm/pgalloc.c index 58696a0c4e4ac..569de24d33761 100644 --- a/arch/s390/mm/pgalloc.c +++ b/arch/s390/mm/pgalloc.c @@ -182,7 +182,7 @@ unsigned long *page_table_alloc(struct mm_struct *mm) static void pagetable_pte_dtor_free(struct ptdesc *ptdesc) { - pagetable_pte_dtor(ptdesc); + pagetable_dtor(ptdesc); pagetable_free(ptdesc); } diff --git a/arch/sh/include/asm/pgalloc.h b/arch/sh/include/asm/pgalloc.h index 5d8577ab15911..96d938fdf2244 100644 --- a/arch/sh/include/asm/pgalloc.h +++ b/arch/sh/include/asm/pgalloc.h @@ -34,7 +34,7 @@ static inline void pmd_populate(struct mm_struct *mm, pmd_t *pmd, #define __pte_free_tlb(tlb, pte, addr) \ do { \ - pagetable_pte_dtor(page_ptdesc(pte)); \ + pagetable_dtor(page_ptdesc(pte)); \ tlb_remove_page_ptdesc((tlb), (page_ptdesc(pte))); \ } while (0) diff --git a/arch/sparc/mm/init_64.c b/arch/sparc/mm/init_64.c index 21f8cbbd0581c..05882bca5b732 100644 --- a/arch/sparc/mm/init_64.c +++ b/arch/sparc/mm/init_64.c @@ -2915,7 +2915,7 @@ static void __pte_free(pgtable_t pte) { struct ptdesc *ptdesc = virt_to_ptdesc(pte); - pagetable_pte_dtor(ptdesc); + pagetable_dtor(ptdesc); pagetable_free(ptdesc); } diff --git a/arch/sparc/mm/srmmu.c b/arch/sparc/mm/srmmu.c index 9df51a62333d6..e3a72c884b867 100644 --- a/arch/sparc/mm/srmmu.c +++ b/arch/sparc/mm/srmmu.c @@ -372,7 +372,7 @@ void pte_free(struct mm_struct *mm, pgtable_t ptep) page = pfn_to_page(__nocache_pa((unsigned long)ptep) >> PAGE_SHIFT); spin_lock(&mm->page_table_lock); if (page_ref_dec_return(page) == 1) - pagetable_pte_dtor(page_ptdesc(page)); + pagetable_dtor(page_ptdesc(page)); spin_unlock(&mm->page_table_lock); srmmu_free_nocache(ptep, SRMMU_PTE_TABLE_SIZE); diff --git a/arch/um/include/asm/pgalloc.h b/arch/um/include/asm/pgalloc.h index 04fb4e6969a46..f0af23c3aeb2b 100644 --- a/arch/um/include/asm/pgalloc.h +++ b/arch/um/include/asm/pgalloc.h @@ -27,7 +27,7 @@ extern pgd_t *pgd_alloc(struct mm_struct *); #define __pte_free_tlb(tlb, pte, address) \ do { \ - pagetable_pte_dtor(page_ptdesc(pte)); \ + pagetable_dtor(page_ptdesc(pte)); \ tlb_remove_page_ptdesc((tlb), (page_ptdesc(pte))); \ } while (0) @@ -35,7 +35,7 @@ do { \ #define __pmd_free_tlb(tlb, pmd, address) \ do { \ - pagetable_pmd_dtor(virt_to_ptdesc(pmd)); \ + pagetable_dtor(virt_to_ptdesc(pmd)); \ tlb_remove_page_ptdesc((tlb), virt_to_ptdesc(pmd)); \ } while (0) @@ -43,7 +43,7 @@ do { \ #define __pud_free_tlb(tlb, pud, address) \ do { \ - pagetable_pud_dtor(virt_to_ptdesc(pud)); \ + pagetable_dtor(virt_to_ptdesc(pud)); \ tlb_remove_page_ptdesc((tlb), virt_to_ptdesc(pud)); \ } while (0) diff --git a/arch/x86/mm/pgtable.c b/arch/x86/mm/pgtable.c index 3d6e84da45b24..a6cd9660e29ec 100644 --- a/arch/x86/mm/pgtable.c +++ b/arch/x86/mm/pgtable.c @@ -60,7 +60,7 @@ early_param("userpte", setup_userpte); void ___pte_free_tlb(struct mmu_gather *tlb, struct page *pte) { - pagetable_pte_dtor(page_ptdesc(pte)); + pagetable_dtor(page_ptdesc(pte)); paravirt_release_pte(page_to_pfn(pte)); paravirt_tlb_remove_table(tlb, pte); } @@ -77,7 +77,7 @@ void ___pmd_free_tlb(struct mmu_gather *tlb, pmd_t *pmd) #ifdef CONFIG_X86_PAE tlb->need_flush_all = 1; #endif - pagetable_pmd_dtor(ptdesc); + pagetable_dtor(ptdesc); paravirt_tlb_remove_table(tlb, ptdesc_page(ptdesc)); } @@ -86,7 +86,7 @@ void ___pud_free_tlb(struct mmu_gather *tlb, pud_t *pud) { struct ptdesc *ptdesc = virt_to_ptdesc(pud); - pagetable_pud_dtor(ptdesc); + pagetable_dtor(ptdesc); paravirt_release_pud(__pa(pud) >> PAGE_SHIFT); paravirt_tlb_remove_table(tlb, virt_to_page(pud)); } @@ -96,7 +96,7 @@ void ___p4d_free_tlb(struct mmu_gather *tlb, p4d_t *p4d) { struct ptdesc *ptdesc = virt_to_ptdesc(p4d); - pagetable_p4d_dtor(ptdesc); + pagetable_dtor(ptdesc); paravirt_release_p4d(__pa(p4d) >> PAGE_SHIFT); paravirt_tlb_remove_table(tlb, virt_to_page(p4d)); } @@ -233,7 +233,7 @@ static void free_pmds(struct mm_struct *mm, pmd_t *pmds[], int count) if (pmds[i]) { ptdesc = virt_to_ptdesc(pmds[i]); - pagetable_pmd_dtor(ptdesc); + pagetable_dtor(ptdesc); pagetable_free(ptdesc); mm_dec_nr_pmds(mm); } @@ -867,7 +867,7 @@ int pud_free_pmd_page(pud_t *pud, unsigned long addr) free_page((unsigned long)pmd_sv); - pagetable_pmd_dtor(virt_to_ptdesc(pmd)); + pagetable_dtor(virt_to_ptdesc(pmd)); free_page((unsigned long)pmd); return 1; diff --git a/include/asm-generic/pgalloc.h b/include/asm-generic/pgalloc.h index bb482eeca0c3e..4afb346eae255 100644 --- a/include/asm-generic/pgalloc.h +++ b/include/asm-generic/pgalloc.h @@ -109,7 +109,7 @@ static inline void pte_free(struct mm_struct *mm, struct page *pte_page) { struct ptdesc *ptdesc = page_ptdesc(pte_page); - pagetable_pte_dtor(ptdesc); + pagetable_dtor(ptdesc); pagetable_free(ptdesc); } @@ -153,7 +153,7 @@ static inline void pmd_free(struct mm_struct *mm, pmd_t *pmd) struct ptdesc *ptdesc = virt_to_ptdesc(pmd); BUG_ON((unsigned long)pmd & (PAGE_SIZE-1)); - pagetable_pmd_dtor(ptdesc); + pagetable_dtor(ptdesc); pagetable_free(ptdesc); } #endif @@ -202,7 +202,7 @@ static inline void __pud_free(struct mm_struct *mm, pud_t *pud) struct ptdesc *ptdesc = virt_to_ptdesc(pud); BUG_ON((unsigned long)pud & (PAGE_SIZE-1)); - pagetable_pud_dtor(ptdesc); + pagetable_dtor(ptdesc); pagetable_free(ptdesc); } @@ -248,7 +248,7 @@ static inline void __p4d_free(struct mm_struct *mm, p4d_t *p4d) struct ptdesc *ptdesc = virt_to_ptdesc(p4d); BUG_ON((unsigned long)p4d & (PAGE_SIZE-1)); - pagetable_p4d_dtor(ptdesc); + pagetable_dtor(ptdesc); pagetable_free(ptdesc); } diff --git a/include/linux/mm.h b/include/linux/mm.h index 5d82f42ddd5cc..cad11fa10c192 100644 --- a/include/linux/mm.h +++ b/include/linux/mm.h @@ -2992,6 +2992,15 @@ static inline bool ptlock_init(struct ptdesc *ptdesc) { return true; } static inline void ptlock_free(struct ptdesc *ptdesc) {} #endif /* defined(CONFIG_SPLIT_PTE_PTLOCKS) */ +static inline void pagetable_dtor(struct ptdesc *ptdesc) +{ + struct folio *folio = ptdesc_folio(ptdesc); + + ptlock_free(ptdesc); + __folio_clear_pgtable(folio); + lruvec_stat_sub_folio(folio, NR_PAGETABLE); +} + static inline bool pagetable_pte_ctor(struct ptdesc *ptdesc) { struct folio *folio = ptdesc_folio(ptdesc); @@ -3003,15 +3012,6 @@ static inline bool pagetable_pte_ctor(struct ptdesc *ptdesc) return true; } -static inline void pagetable_pte_dtor(struct ptdesc *ptdesc) -{ - struct folio *folio = ptdesc_folio(ptdesc); - - ptlock_free(ptdesc); - __folio_clear_pgtable(folio); - lruvec_stat_sub_folio(folio, NR_PAGETABLE); -} - pte_t *___pte_offset_map(pmd_t *pmd, unsigned long addr, pmd_t *pmdvalp); static inline pte_t *__pte_offset_map(pmd_t *pmd, unsigned long addr, pmd_t *pmdvalp) @@ -3088,14 +3088,6 @@ static inline bool pmd_ptlock_init(struct ptdesc *ptdesc) return ptlock_init(ptdesc); } -static inline void pmd_ptlock_free(struct ptdesc *ptdesc) -{ -#ifdef CONFIG_TRANSPARENT_HUGEPAGE - VM_BUG_ON_PAGE(ptdesc->pmd_huge_pte, ptdesc_page(ptdesc)); -#endif - ptlock_free(ptdesc); -} - #define pmd_huge_pte(mm, pmd) (pmd_ptdesc(pmd)->pmd_huge_pte) #else @@ -3106,7 +3098,6 @@ static inline spinlock_t *pmd_lockptr(struct mm_struct *mm, pmd_t *pmd) } static inline bool pmd_ptlock_init(struct ptdesc *ptdesc) { return true; } -static inline void pmd_ptlock_free(struct ptdesc *ptdesc) {} #define pmd_huge_pte(mm, pmd) ((mm)->pmd_huge_pte) @@ -3131,15 +3122,6 @@ static inline bool pagetable_pmd_ctor(struct ptdesc *ptdesc) return true; } -static inline void pagetable_pmd_dtor(struct ptdesc *ptdesc) -{ - struct folio *folio = ptdesc_folio(ptdesc); - - pmd_ptlock_free(ptdesc); - __folio_clear_pgtable(folio); - lruvec_stat_sub_folio(folio, NR_PAGETABLE); -} - /* * No scalability reason to split PUD locks yet, but follow the same pattern * as the PMD locks to make it easier if we decide to. The VM should not be @@ -3167,14 +3149,6 @@ static inline void pagetable_pud_ctor(struct ptdesc *ptdesc) lruvec_stat_add_folio(folio, NR_PAGETABLE); } -static inline void pagetable_pud_dtor(struct ptdesc *ptdesc) -{ - struct folio *folio = ptdesc_folio(ptdesc); - - __folio_clear_pgtable(folio); - lruvec_stat_sub_folio(folio, NR_PAGETABLE); -} - static inline void pagetable_p4d_ctor(struct ptdesc *ptdesc) { struct folio *folio = ptdesc_folio(ptdesc); @@ -3183,14 +3157,6 @@ static inline void pagetable_p4d_ctor(struct ptdesc *ptdesc) lruvec_stat_add_folio(folio, NR_PAGETABLE); } -static inline void pagetable_p4d_dtor(struct ptdesc *ptdesc) -{ - struct folio *folio = ptdesc_folio(ptdesc); - - __folio_clear_pgtable(folio); - lruvec_stat_sub_folio(folio, NR_PAGETABLE); -} - extern void __init pagecache_init(void); extern void free_initmem(void); diff --git a/mm/memory.c b/mm/memory.c index 9423967b24180..ad871e564568b 100644 --- a/mm/memory.c +++ b/mm/memory.c @@ -7051,7 +7051,8 @@ bool ptlock_alloc(struct ptdesc *ptdesc) void ptlock_free(struct ptdesc *ptdesc) { - kmem_cache_free(page_ptl_cachep, ptdesc->ptl); + if (ptdesc->ptl) + kmem_cache_free(page_ptl_cachep, ptdesc->ptl); } #endif From patchwork Mon Dec 23 09:40:54 2024 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Qi Zheng X-Patchwork-Id: 13918762 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from bombadil.infradead.org (bombadil.infradead.org [198.137.202.133]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (No client certificate requested) by smtp.lore.kernel.org (Postfix) with ESMTPS id D1B77E7718B for ; Mon, 23 Dec 2024 11:06:15 +0000 (UTC) DKIM-Signature: v=1; a=rsa-sha256; q=dns/txt; c=relaxed/relaxed; d=lists.infradead.org; s=bombadil.20210309; h=Sender: Content-Transfer-Encoding:Content-Type:List-Subscribe:List-Help:List-Post: List-Archive:List-Unsubscribe:List-Id:MIME-Version:References:In-Reply-To: Message-Id:Date:Subject:Cc:To:From:Reply-To:Content-ID:Content-Description: Resent-Date:Resent-From:Resent-Sender:Resent-To:Resent-Cc:Resent-Message-ID: List-Owner; bh=I8mCXOEhoAkkTwvTMZVSvC6sgQ/jZRVJcLn/S1+2b+0=; b=WBQ9BeCjtkWbfH v+T6s31qzE8TZHyh0N+6F0GpdGxe6SH7FFtrj+XYt/qA59tK2lPDqpAydkTReSQFyVIfiqcDTxENE KecRdS0aNCJOBn/zJjNOVIC8LF4rGL0JXPGZAJij6K5EuzqtEnhGTRyK/O2oapJkKMoALjPUdwaH6 EtIoxSJeqBgjgftxHdbU6D3ilsH4yfXdA8bP5EiDr5eOhG8DSkfLMO65LLuvyjfguYQsBVQAPlKco 3sGdqm8PTqHrCEVTTDsbJrEDdlHjhgr0cXFhsUUYkUwnvm7oS4C6bR6Zs54fxrb74vXPSkG1uQ5yy krho3LopV4ult3XDmPQA==; Received: from localhost ([::1] helo=bombadil.infradead.org) by bombadil.infradead.org with esmtp (Exim 4.98 #2 (Red Hat Linux)) id 1tPgGI-00000009qJI-0z7y; Mon, 23 Dec 2024 11:06:10 +0000 Received: from mail-pf1-x42d.google.com ([2607:f8b0:4864:20::42d]) by bombadil.infradead.org with esmtps (Exim 4.98 #2 (Red Hat Linux)) id 1tPezW-00000009gq7-0HwA for linux-riscv@lists.infradead.org; Mon, 23 Dec 2024 09:44:47 +0000 Received: by mail-pf1-x42d.google.com with SMTP id d2e1a72fcca58-725abf74334so3354387b3a.3 for ; Mon, 23 Dec 2024 01:44:45 -0800 (PST) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=bytedance.com; s=google; t=1734947085; x=1735551885; darn=lists.infradead.org; h=content-transfer-encoding:mime-version:references:in-reply-to :message-id:date:subject:cc:to:from:from:to:cc:subject:date :message-id:reply-to; bh=qA0mhWnXo6upWemI3POPEc6VJyAkyxFJggkOB6WNq2k=; b=FE452Oqusb0rgBr6YngBczaIZtEjQK09AFdFZIxckz6Xr+PLf+KBXxu4GjmlQdbcD5 MnwMxLVk8A2O13eLCuT1rD6LzHnfa3icb5t2bHmQ7jaU6XeqImTR+arkQ6Gma9oZBIpZ PyBLtSj0xITvtIpAszE8FkLmnqBtcaLkygNZ6f3KU9X3tXXbdKxMfCmUvVAsBZ7JrcX9 jSRBaoz8CWnotjj2fOpzzV9r+RTVSguntC/dC44O+E2ceeTEoex2Ds9n8bruAdbfo9hX ItgWksMMWSENZmlPmTGXxggqUFfLoOF895Sj+FSe1cHC8wR9Mv/6RhCTVy686hB/8mqg NV0Q== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20230601; t=1734947085; x=1735551885; h=content-transfer-encoding:mime-version:references:in-reply-to :message-id:date:subject:cc:to:from:x-gm-message-state:from:to:cc :subject:date:message-id:reply-to; bh=qA0mhWnXo6upWemI3POPEc6VJyAkyxFJggkOB6WNq2k=; b=S0ZPDmgvUP0WuBs6te8HrbMe4tlK9Bl9eXY9GlCj4nubBQipaf7GFN5tsRwdtjAKe5 ykNaAs4Sb4jR9LfNoh3Bqcb3p3Q2WxoiG7EU1MaJoh0cSWhaw4613p12oqhExGIr5h39 kx8JvpexZHQ7HuExggYbBo0tH0lqDuiCaZofgD91oL59lahnAeXFPV1r3cOM+A2RWcEj /m7NS3f0jmYM/+djpOFWENKi0ceHL6xzjcKmnMm50cYA02mEEldAkN/aLbGxmtDFYnlS GHRqTWilLgk+AGTt8+SWxMnPda0ZnaTkaMts3mPPvL1p6rU7oU1nJZefl7ullHya7H3U o4Ng== X-Forwarded-Encrypted: i=1; AJvYcCXMOFr+Fedor517gmsyEfmRZ/ym+P7DPstlzlFb/BSwtrksZYKIHU2CaTcu8hkQcjxUdfPsCAh1LG3d2Q==@lists.infradead.org X-Gm-Message-State: AOJu0YwS+uIwyCsL+CtpPRnEL2i9wc0VuUtTp0YpGffnV8sgpJnMJvd+ BQ4iL+a1o+UPwQFD/f7QHOv8LZduz/Gqd8CGidf0KqHFD5iyb3j2QlFzTZzITOU= X-Gm-Gg: ASbGncs0cvYEwn8cSe/+0qBjjClSFMYoFFlFcneGL/rr9qP8QNi9XKeiTvKLC0is4Tu NaUfqvJwlUGIo3/Sy6jk08Dyg4L3VXebXborfMSCp8O5AwJ6eLGJDnln51tGe/0oSVK+pd6cZY1 OWszgrodoZIRvHquarQOrWgYDFAaPfleDgSxIoIcogr6x4tqCvAWbBl0rgtuc/gWaoXHBoWigs9 Gy+LemmGAS9fmvwbpK5vJ2wzsoCOo7psYF3Sm+vPU7VXrA1+qlLP5B9PnXDvlpK+dPx/dzDkTuN Rs9h42YdCztjNvib4RpxlA== X-Google-Smtp-Source: AGHT+IEwnxNRxjpY566VNLJW6lRIrSoOOApqv/9DKS3yUbVNE+/391+nOnqyFIwO+T3gR5XgQach6A== X-Received: by 2002:a05:6a00:330b:b0:728:e382:5f14 with SMTP id d2e1a72fcca58-72abdd7bae3mr14443469b3a.9.1734947085429; Mon, 23 Dec 2024 01:44:45 -0800 (PST) Received: from C02DW0BEMD6R.bytedance.net ([203.208.167.150]) by smtp.gmail.com with ESMTPSA id d2e1a72fcca58-72aad8486c6sm7468309b3a.85.2024.12.23.01.44.33 (version=TLS1_3 cipher=TLS_AES_256_GCM_SHA384 bits=256/256); Mon, 23 Dec 2024 01:44:45 -0800 (PST) From: Qi Zheng To: peterz@infradead.org, agordeev@linux.ibm.com, kevin.brodsky@arm.com, tglx@linutronix.de, david@redhat.com, jannh@google.com, hughd@google.com, yuzhao@google.com, willy@infradead.org, muchun.song@linux.dev, vbabka@kernel.org, lorenzo.stoakes@oracle.com, akpm@linux-foundation.org, rientjes@google.com, vishal.moola@gmail.com, arnd@arndb.de, will@kernel.org, aneesh.kumar@kernel.org, npiggin@gmail.com, dave.hansen@linux.intel.com, rppt@kernel.org, ryan.roberts@arm.com Cc: linux-mm@kvack.org, linux-arm-kernel@lists.infradead.org, linuxppc-dev@lists.ozlabs.org, linux-riscv@lists.infradead.org, linux-s390@vger.kernel.org, sparclinux@vger.kernel.org, linux-kernel@vger.kernel.org, x86@kernel.org, linux-arch@vger.kernel.org, linux-csky@vger.kernel.org, linux-hexagon@vger.kernel.org, loongarch@lists.linux.dev, linux-m68k@lists.linux-m68k.org, linux-mips@vger.kernel.org, linux-openrisc@vger.kernel.org, linux-sh@vger.kernel.org, linux-um@lists.infradead.org, Qi Zheng Subject: [PATCH v3 08/17] arm: pgtable: move pagetable_dtor() to __tlb_remove_table() Date: Mon, 23 Dec 2024 17:40:54 +0800 Message-Id: <955162bfbbcd9fbb3b074e1fe2aef4f64b61d6f9.1734945104.git.zhengqi.arch@bytedance.com> X-Mailer: git-send-email 2.24.3 (Apple Git-128) In-Reply-To: References: MIME-Version: 1.0 X-CRM114-Version: 20100106-BlameMichelson ( TRE 0.8.0 (BSD) ) MR-646709E3 X-CRM114-CacheID: sfid-20241223_014446_165131_77DC2A7E X-CRM114-Status: GOOD ( 10.11 ) X-BeenThere: linux-riscv@lists.infradead.org X-Mailman-Version: 2.1.34 Precedence: list List-Id: List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Sender: "linux-riscv" Errors-To: linux-riscv-bounces+linux-riscv=archiver.kernel.org@lists.infradead.org Move pagetable_dtor() to __tlb_remove_table(), so that ptlock and page table pages can be freed together (regardless of whether RCU is used). This prevents the use-after-free problem where the ptlock is freed immediately but the page table pages is freed later via RCU. Page tables shouldn't have swap cache, so use pagetable_free() instead of free_page_and_swap_cache() to free page table pages. Signed-off-by: Qi Zheng Suggested-by: Peter Zijlstra (Intel) Cc: linux-arm-kernel@lists.infradead.org --- arch/arm/include/asm/tlb.h | 9 ++++----- 1 file changed, 4 insertions(+), 5 deletions(-) diff --git a/arch/arm/include/asm/tlb.h b/arch/arm/include/asm/tlb.h index ef79bf1e8563f..264ab635e807a 100644 --- a/arch/arm/include/asm/tlb.h +++ b/arch/arm/include/asm/tlb.h @@ -26,12 +26,14 @@ #else /* !CONFIG_MMU */ -#include #include static inline void __tlb_remove_table(void *_table) { - free_page_and_swap_cache((struct page *)_table); + struct ptdesc *ptdesc = (struct ptdesc *)_table; + + pagetable_dtor(ptdesc); + pagetable_free(ptdesc); } #include @@ -41,8 +43,6 @@ __pte_free_tlb(struct mmu_gather *tlb, pgtable_t pte, unsigned long addr) { struct ptdesc *ptdesc = page_ptdesc(pte); - pagetable_dtor(ptdesc); - #ifndef CONFIG_ARM_LPAE /* * With the classic ARM MMU, a pte page has two corresponding pmd @@ -61,7 +61,6 @@ __pmd_free_tlb(struct mmu_gather *tlb, pmd_t *pmdp, unsigned long addr) #ifdef CONFIG_ARM_LPAE struct ptdesc *ptdesc = virt_to_ptdesc(pmdp); - pagetable_dtor(ptdesc); tlb_remove_ptdesc(tlb, ptdesc); #endif } From patchwork Mon Dec 23 09:40:55 2024 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Qi Zheng X-Patchwork-Id: 13918761 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from bombadil.infradead.org (bombadil.infradead.org [198.137.202.133]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (No client certificate requested) by smtp.lore.kernel.org (Postfix) with ESMTPS id E7D94E7718D for ; Mon, 23 Dec 2024 11:06:15 +0000 (UTC) DKIM-Signature: v=1; a=rsa-sha256; q=dns/txt; c=relaxed/relaxed; d=lists.infradead.org; s=bombadil.20210309; h=Sender: Content-Transfer-Encoding:Content-Type:List-Subscribe:List-Help:List-Post: List-Archive:List-Unsubscribe:List-Id:MIME-Version:References:In-Reply-To: Message-Id:Date:Subject:Cc:To:From:Reply-To:Content-ID:Content-Description: Resent-Date:Resent-From:Resent-Sender:Resent-To:Resent-Cc:Resent-Message-ID: List-Owner; bh=AuerR80fM9OR46P7HQ4om1xY9e6+fwG/rfzRhMKd8CI=; b=n+HEgX5Tn5DJRA VPUw3dCRm40ZnwMLPu4cdl39A/HiMiKMu7EmrfirqP67vy0GBDjOjSjfFmAL16nmwC/T7GvrTeJYL G4lHlySKR5Gs85nIkFWSvYudLMPHogvtMsqRFWwcBhwtEnA8Yfsh37nASsITV9KMJZWgv4+vbv7qc FtB2BahmhohQZVGVfm5OL0lwfq2OUDFfbq0HLvpiVWUVwZaJ96zDd1OiencIep14fDLjob+OtZpn8 fr+/7e8Gv3VUxZt3oCZR4jes38qhQ6g4LGsRKscmLmTNPXEm9ebVoVHHpbcMctDrqbh5yj1zbZAwP 540a8Ik3ANg1GFvE4J2A==; Received: from localhost ([::1] helo=bombadil.infradead.org) by bombadil.infradead.org with esmtp (Exim 4.98 #2 (Red Hat Linux)) id 1tPgGJ-00000009qK3-05f0; Mon, 23 Dec 2024 11:06:11 +0000 Received: from mail-pf1-x435.google.com ([2607:f8b0:4864:20::435]) by bombadil.infradead.org with esmtps (Exim 4.98 #2 (Red Hat Linux)) id 1tPezi-00000009gwv-2K6H for linux-riscv@lists.infradead.org; Mon, 23 Dec 2024 09:45:00 +0000 Received: by mail-pf1-x435.google.com with SMTP id d2e1a72fcca58-728eccf836bso3361976b3a.1 for ; Mon, 23 Dec 2024 01:44:58 -0800 (PST) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=bytedance.com; s=google; t=1734947098; x=1735551898; darn=lists.infradead.org; h=content-transfer-encoding:mime-version:references:in-reply-to :message-id:date:subject:cc:to:from:from:to:cc:subject:date :message-id:reply-to; bh=0nJ0bdEaVLB0+13gaCQHdoBfFkuXXT3Pqta7GOjv8Bo=; b=S3at9zfz3yUNjlkjUPKzJaN+AAsKK1Vjgg9/OZxOi6o6ncGXF71uYtR1xjjE4iIlRM jCyJY4/7RZftYZlSP+Pu+NLnygsyt5CnpCsauB7PeSR42NaK+x3Ug6U32tjyLztSjgOP IjPHyvUv/dndwIhRKLqP/H+dypNNYwJVxSw6+YFpgfZKrzxuUPLEQz1CQJUjnusysbqd CsHV2BqsPnYri9kZQRgJyocOawSVNiVIxXvA8l/UumDNxo5dj6SbiuNVFO3WsEYbeW0y p9HHclfdzzShjjOQ6EB/FeaCn0ZxvCWL5jLZsFs8vD6TO48Sf14PVWSCnzoQ/OYdfOP8 3hMw== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20230601; t=1734947098; x=1735551898; h=content-transfer-encoding:mime-version:references:in-reply-to :message-id:date:subject:cc:to:from:x-gm-message-state:from:to:cc :subject:date:message-id:reply-to; bh=0nJ0bdEaVLB0+13gaCQHdoBfFkuXXT3Pqta7GOjv8Bo=; b=JAfg99tN9Voes59ce37dIMUYLv9XXMzZF9FA2AppPHgjYdDwNRaJEuNcWfqTHAMH+B 9eKrQ5BEjknqfHjiD2LXnFyS9zk5UROnuao3q61uu4goW6BVjCWkm3wppTtE/Umb9yHP a97UmrmDefOD1W+WznJzpRnNUjOAldeAiCtCw5FN/bNm/Kb4RP/DwCllBmOlO8/VRk4K G9tqRBB16bT21uh7WIU7+Uvt5kFxK/MteYiFWapB5ayNnjwKGjuPM2lzdXf36UGPNYk6 jwKleCT3xX1zou/+Ahj9dUWnDXeuRF3jAz/k1tJaXWhNXNSDn8KTEukVqnASe6YSBRZB w6jQ== X-Forwarded-Encrypted: i=1; AJvYcCX6QMngNmnWJqIWLXoTD20ffltDaoalSGYWJCxFhY73braqd20b7iqOvpjYi7Jgf2Me7oNYaB0O38N3Ug==@lists.infradead.org X-Gm-Message-State: AOJu0YzSLAfJ770bc7gudjqmYTLnwnOamnLO1AM9zb8Tbd2M4U1OyDjP X5PeXCgeMZ9Q/DgqYbYO0F06+IvKaOJt4o1wHDNljv1v4AkiXJLhm2Ys5Ogdzo53zWc4Bo0QEoy t X-Gm-Gg: ASbGncsTsnn2BXtFzB0aaJwJNDvsmrXC9nDjxgmfeTBASAc4Lmbd0Dqx05l4zdKSgUb zhqaeLRczhIVwnOvcxFzf4cYcCShKRBF2OSHSeprDB+q8/FTWL1668EiYlXqqxu8bjiXEBlSB/V m9C1rSvuAmwITMb5aIFmen5+mUiR2hRrTnKh+n5x2ROwZBMR/IXZoaqhuts+nTeiYKCofpydSz1 lyIvTON5/iQfJRcQ+argHGNM1RO5jWRH6QGhZWX7bGb3fJb3Rf3UURERMrQwTe14CkNTIgia2MU YzhRY0RgX82HuEW4wG7sWQ== X-Google-Smtp-Source: AGHT+IH27a1l8kT42WYYi2KTFhW6smXjjCcrkYYj/EF7HLDGUXcMfllZ3ayHCCSFU+IsHMUxZvkRwQ== X-Received: by 2002:a05:6a21:6f87:b0:1db:c20f:2c4d with SMTP id adf61e73a8af0-1e5e044b1c1mr19165305637.2.1734947098042; Mon, 23 Dec 2024 01:44:58 -0800 (PST) Received: from C02DW0BEMD6R.bytedance.net ([203.208.167.150]) by smtp.gmail.com with ESMTPSA id d2e1a72fcca58-72aad8486c6sm7468309b3a.85.2024.12.23.01.44.45 (version=TLS1_3 cipher=TLS_AES_256_GCM_SHA384 bits=256/256); Mon, 23 Dec 2024 01:44:57 -0800 (PST) From: Qi Zheng To: peterz@infradead.org, agordeev@linux.ibm.com, kevin.brodsky@arm.com, tglx@linutronix.de, david@redhat.com, jannh@google.com, hughd@google.com, yuzhao@google.com, willy@infradead.org, muchun.song@linux.dev, vbabka@kernel.org, lorenzo.stoakes@oracle.com, akpm@linux-foundation.org, rientjes@google.com, vishal.moola@gmail.com, arnd@arndb.de, will@kernel.org, aneesh.kumar@kernel.org, npiggin@gmail.com, dave.hansen@linux.intel.com, rppt@kernel.org, ryan.roberts@arm.com Cc: linux-mm@kvack.org, linux-arm-kernel@lists.infradead.org, linuxppc-dev@lists.ozlabs.org, linux-riscv@lists.infradead.org, linux-s390@vger.kernel.org, sparclinux@vger.kernel.org, linux-kernel@vger.kernel.org, x86@kernel.org, linux-arch@vger.kernel.org, linux-csky@vger.kernel.org, linux-hexagon@vger.kernel.org, loongarch@lists.linux.dev, linux-m68k@lists.linux-m68k.org, linux-mips@vger.kernel.org, linux-openrisc@vger.kernel.org, linux-sh@vger.kernel.org, linux-um@lists.infradead.org, Qi Zheng Subject: [PATCH v3 09/17] arm64: pgtable: move pagetable_dtor() to __tlb_remove_table() Date: Mon, 23 Dec 2024 17:40:55 +0800 Message-Id: X-Mailer: git-send-email 2.24.3 (Apple Git-128) In-Reply-To: References: MIME-Version: 1.0 X-CRM114-Version: 20100106-BlameMichelson ( TRE 0.8.0 (BSD) ) MR-646709E3 X-CRM114-CacheID: sfid-20241223_014458_600414_9D5BA21D X-CRM114-Status: UNSURE ( 9.52 ) X-CRM114-Notice: Please train this message. X-BeenThere: linux-riscv@lists.infradead.org X-Mailman-Version: 2.1.34 Precedence: list List-Id: List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Sender: "linux-riscv" Errors-To: linux-riscv-bounces+linux-riscv=archiver.kernel.org@lists.infradead.org Move pagetable_dtor() to __tlb_remove_table(), so that ptlock and page table pages can be freed together (regardless of whether RCU is used). This prevents the use-after-free problem where the ptlock is freed immediately but the page table pages is freed later via RCU. Page tables shouldn't have swap cache, so use pagetable_free() instead of free_page_and_swap_cache() to free page table pages. Signed-off-by: Qi Zheng Suggested-by: Peter Zijlstra (Intel) Cc: linux-arm-kernel@lists.infradead.org --- arch/arm64/include/asm/tlb.h | 10 ++++------ 1 file changed, 4 insertions(+), 6 deletions(-) diff --git a/arch/arm64/include/asm/tlb.h b/arch/arm64/include/asm/tlb.h index 408d0f36a8a8f..93591a80b5bfb 100644 --- a/arch/arm64/include/asm/tlb.h +++ b/arch/arm64/include/asm/tlb.h @@ -9,11 +9,13 @@ #define __ASM_TLB_H #include -#include static inline void __tlb_remove_table(void *_table) { - free_page_and_swap_cache((struct page *)_table); + struct ptdesc *ptdesc = (struct ptdesc *)_table; + + pagetable_dtor(ptdesc); + pagetable_free(ptdesc); } #define tlb_flush tlb_flush @@ -82,7 +84,6 @@ static inline void __pte_free_tlb(struct mmu_gather *tlb, pgtable_t pte, { struct ptdesc *ptdesc = page_ptdesc(pte); - pagetable_dtor(ptdesc); tlb_remove_ptdesc(tlb, ptdesc); } @@ -92,7 +93,6 @@ static inline void __pmd_free_tlb(struct mmu_gather *tlb, pmd_t *pmdp, { struct ptdesc *ptdesc = virt_to_ptdesc(pmdp); - pagetable_dtor(ptdesc); tlb_remove_ptdesc(tlb, ptdesc); } #endif @@ -106,7 +106,6 @@ static inline void __pud_free_tlb(struct mmu_gather *tlb, pud_t *pudp, if (!pgtable_l4_enabled()) return; - pagetable_dtor(ptdesc); tlb_remove_ptdesc(tlb, ptdesc); } #endif @@ -120,7 +119,6 @@ static inline void __p4d_free_tlb(struct mmu_gather *tlb, p4d_t *p4dp, if (!pgtable_l5_enabled()) return; - pagetable_dtor(ptdesc); tlb_remove_ptdesc(tlb, ptdesc); } #endif From patchwork Mon Dec 23 09:40:56 2024 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Qi Zheng X-Patchwork-Id: 13918707 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from bombadil.infradead.org (bombadil.infradead.org [198.137.202.133]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (No client certificate requested) by smtp.lore.kernel.org (Postfix) with ESMTPS id 0D17AE7718B for ; Mon, 23 Dec 2024 09:53:20 +0000 (UTC) DKIM-Signature: v=1; a=rsa-sha256; q=dns/txt; c=relaxed/relaxed; d=lists.infradead.org; s=bombadil.20210309; h=Sender: Content-Transfer-Encoding:Content-Type:List-Subscribe:List-Help:List-Post: List-Archive:List-Unsubscribe:List-Id:MIME-Version:References:In-Reply-To: Message-Id:Date:Subject:Cc:To:From:Reply-To:Content-ID:Content-Description: Resent-Date:Resent-From:Resent-Sender:Resent-To:Resent-Cc:Resent-Message-ID: List-Owner; bh=t7gkfLpjBj6mSQOVd5iJQO9/lpl92JTglkFs7ApRUL4=; b=QQudTUzaeRpsqn PGwZ2IPlDB1rrWizulTzj1cREBG5YbcCn6CraqfwhV/b46ZB3LqBFCNRKV8dvjvyHqD6DE8YGQhb9 s1vYQ4OJlXDdN/hMU2vmMsbGkMK+sT5hg2SlP0R/1xLeLw4HxHnOlSBRn6gbeOHXXltT1UYCdlWqT GG2xepebzCOocVLNsQFBpbGL9jAmen5DyZUE1ddrZF/Zo1U0Xeha5VqVuInxraG+zUE91BFsoA2tR YrkDuRyz8T9flT5enCxM3dJvvKOpIs4a5YrvI4apZ/pBzR0SWfhWDQFpv6vrFU6IXONqkxl+ou70z Arb7f5QW8eGM5nFrNWdQ==; Received: from localhost ([::1] helo=bombadil.infradead.org) by bombadil.infradead.org with esmtp (Exim 4.98 #2 (Red Hat Linux)) id 1tPf7j-00000009jDh-0T4n; Mon, 23 Dec 2024 09:53:15 +0000 Received: from mail-pf1-x42c.google.com ([2607:f8b0:4864:20::42c]) by bombadil.infradead.org with esmtps (Exim 4.98 #2 (Red Hat Linux)) id 1tPezv-00000009h3B-3HC5 for linux-riscv@lists.infradead.org; Mon, 23 Dec 2024 09:45:12 +0000 Received: by mail-pf1-x42c.google.com with SMTP id d2e1a72fcca58-725e71a11f7so3868100b3a.1 for ; Mon, 23 Dec 2024 01:45:11 -0800 (PST) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=bytedance.com; s=google; t=1734947111; x=1735551911; darn=lists.infradead.org; h=content-transfer-encoding:mime-version:references:in-reply-to :message-id:date:subject:cc:to:from:from:to:cc:subject:date :message-id:reply-to; bh=76YtbImY0jPwdXnQQVs9oYKl35yZZ81uc4A2YpilpOk=; b=E2DELmPx1gh7ATCS+MsxQuTw9Zt5dLbPt3km3Jh5X6/dpuimQwe1pDWRuejsOT1cmj tVIHcX7bEvqFIGE4ELMMSYxY3mLIwPCx4sHCBClXcBH3qCeID2jqqcmFHzdPk1aw7z0O QjgeiMkGfOATn42HGjFYcfnVcQnXpnbJERejUn5wdqU5PZfO/ezutvTDcN3RgusX9RjV 4m3YXJMCUh+YIsS2uzmd5EIe2JrdnbVLAKeLxphKj58ixN4DwZlQGzbwcS2WspiAEwKS dmfrGuc86Jnb9+WWxYGPhiBsMRzBfewGcRVyl48XmP3LVWy3GLNmEcfmzCCYHzglNjnS 2QvQ== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20230601; t=1734947111; x=1735551911; h=content-transfer-encoding:mime-version:references:in-reply-to :message-id:date:subject:cc:to:from:x-gm-message-state:from:to:cc :subject:date:message-id:reply-to; bh=76YtbImY0jPwdXnQQVs9oYKl35yZZ81uc4A2YpilpOk=; b=WouAnd0aXt/WgfE3FGfuCtEhpN2klpTjfXiUgTbiqhXsJTgUO8S5Q9Qd9Gdr4qD5i5 5iergf3Zl/27zh+MBRTBdl3pfzY7OmNtefkMbzS6U6zT3WeCLHiPJcWHYsKLqJGrZHs2 DdnoiICQYOaKxR4On45oeeSMpgDiX7NiY5S8/F8keYWcdSC5Gs1WOGP/mg+utkgiqmEL p0gnOg2LantIHsm9vI8CXbGkCLrKuiFu5Ck5qdlNt0X3bnu3qv2LiQT/m9Am7xkwwvtZ gC8bmH+7LeXHq0VuVBI/mipC4XsyY1ruTRpgGWaVirUUf422Mbg8Ng+z/Ubx05pZ86LA YrLw== X-Forwarded-Encrypted: i=1; AJvYcCUlixmtaZH3cTd0HXWIqBSwtYZkSg8wZNOD9yVsClyxuq51bDs3FjDIqQeqEuMVqPd8Z/NCG5XXJH9l4Q==@lists.infradead.org X-Gm-Message-State: AOJu0YzlOawiB3MJExzG+6DR7203pO/CwUAH/zFgy+zrXGpzuh6yguR+ TdDKJ/rn7pm4gAQ+pqnp0Qr+zdblTiFdb+WM2bMHS/+H8XXy5aHWvoYHR+yVbAc= X-Gm-Gg: ASbGncu7qW6b7WcGKDSneNw5eqshTWvx/qd64CGtDkwk95S6P/Rq5hi8509DYOb+XIr lhUWGnb98jF/LixFn9CNpt/YSyMexHL/sb8Co9NPDzwvTeM8QodYkLMWhyJYuvvZfpszwp+XzBK IyP3i8QFGEWe2DhoYZbYG/oZsBUtAghx/lbUUeeiYxGeQneGTS3QxNi3Ogd/GCwVizXwWg1YIQZ Zw8ngLaiTogmk2xFYcpm67c96nxOldsY/HgXmymmD2W89Uv/LPNNbcXdduPSxZpe1jZ0F86oTo5 0WtvktEAewge9kujU8IqnA== X-Google-Smtp-Source: AGHT+IFO8x3oURIM+5JbbGjULQ8gSrXSXvX9ZJiJMgilE8gLJBNqFvKuKmavbyH4ASOcC8EYHQP7xQ== X-Received: by 2002:a05:6a20:1593:b0:1e1:ffec:b1a9 with SMTP id adf61e73a8af0-1e5c6ec6f11mr24851906637.3.1734947111156; Mon, 23 Dec 2024 01:45:11 -0800 (PST) Received: from C02DW0BEMD6R.bytedance.net ([203.208.167.150]) by smtp.gmail.com with ESMTPSA id d2e1a72fcca58-72aad8486c6sm7468309b3a.85.2024.12.23.01.44.58 (version=TLS1_3 cipher=TLS_AES_256_GCM_SHA384 bits=256/256); Mon, 23 Dec 2024 01:45:10 -0800 (PST) From: Qi Zheng To: peterz@infradead.org, agordeev@linux.ibm.com, kevin.brodsky@arm.com, tglx@linutronix.de, david@redhat.com, jannh@google.com, hughd@google.com, yuzhao@google.com, willy@infradead.org, muchun.song@linux.dev, vbabka@kernel.org, lorenzo.stoakes@oracle.com, akpm@linux-foundation.org, rientjes@google.com, vishal.moola@gmail.com, arnd@arndb.de, will@kernel.org, aneesh.kumar@kernel.org, npiggin@gmail.com, dave.hansen@linux.intel.com, rppt@kernel.org, ryan.roberts@arm.com Cc: linux-mm@kvack.org, linux-arm-kernel@lists.infradead.org, linuxppc-dev@lists.ozlabs.org, linux-riscv@lists.infradead.org, linux-s390@vger.kernel.org, sparclinux@vger.kernel.org, linux-kernel@vger.kernel.org, x86@kernel.org, linux-arch@vger.kernel.org, linux-csky@vger.kernel.org, linux-hexagon@vger.kernel.org, loongarch@lists.linux.dev, linux-m68k@lists.linux-m68k.org, linux-mips@vger.kernel.org, linux-openrisc@vger.kernel.org, linux-sh@vger.kernel.org, linux-um@lists.infradead.org, Qi Zheng Subject: [PATCH v3 10/17] riscv: pgtable: move pagetable_dtor() to __tlb_remove_table() Date: Mon, 23 Dec 2024 17:40:56 +0800 Message-Id: <0e8f0b3835c15e99145e0006ac1020ae45a2b166.1734945104.git.zhengqi.arch@bytedance.com> X-Mailer: git-send-email 2.24.3 (Apple Git-128) In-Reply-To: References: MIME-Version: 1.0 X-CRM114-Version: 20100106-BlameMichelson ( TRE 0.8.0 (BSD) ) MR-646709E3 X-CRM114-CacheID: sfid-20241223_014511_827089_C34FA2F7 X-CRM114-Status: GOOD ( 13.00 ) X-BeenThere: linux-riscv@lists.infradead.org X-Mailman-Version: 2.1.34 Precedence: list List-Id: List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Sender: "linux-riscv" Errors-To: linux-riscv-bounces+linux-riscv=archiver.kernel.org@lists.infradead.org Move pagetable_dtor() to __tlb_remove_table(), so that ptlock and page table pages can be freed together (regardless of whether RCU is used). This prevents the use-after-free problem where the ptlock is freed immediately but the page table pages is freed later via RCU. Page tables shouldn't have swap cache, so use pagetable_free() instead of free_page_and_swap_cache() to free page table pages. By the way, move the comment above __tlb_remove_table() to riscv_tlb_remove_ptdesc(), it will be more appropriate. Signed-off-by: Qi Zheng Suggested-by: Peter Zijlstra (Intel) Cc: linux-riscv@lists.infradead.org --- arch/riscv/include/asm/pgalloc.h | 38 ++++++++++++++------------------ arch/riscv/include/asm/tlb.h | 14 ++++-------- 2 files changed, 21 insertions(+), 31 deletions(-) diff --git a/arch/riscv/include/asm/pgalloc.h b/arch/riscv/include/asm/pgalloc.h index b6793c5c99296..c8907b8317115 100644 --- a/arch/riscv/include/asm/pgalloc.h +++ b/arch/riscv/include/asm/pgalloc.h @@ -15,12 +15,22 @@ #define __HAVE_ARCH_PUD_FREE #include +/* + * While riscv platforms with riscv_ipi_for_rfence as true require an IPI to + * perform TLB shootdown, some platforms with riscv_ipi_for_rfence as false use + * SBI to perform TLB shootdown. To keep software pagetable walkers safe in this + * case we switch to RCU based table free (MMU_GATHER_RCU_TABLE_FREE). See the + * comment below 'ifdef CONFIG_MMU_GATHER_RCU_TABLE_FREE' in include/asm-generic/tlb.h + * for more details. + */ static inline void riscv_tlb_remove_ptdesc(struct mmu_gather *tlb, void *pt) { - if (riscv_use_sbi_for_rfence()) + if (riscv_use_sbi_for_rfence()) { tlb_remove_ptdesc(tlb, pt); - else + } else { + pagetable_dtor(pt); tlb_remove_page_ptdesc(tlb, pt); + } } static inline void pmd_populate_kernel(struct mm_struct *mm, @@ -97,23 +107,15 @@ static inline void pud_free(struct mm_struct *mm, pud_t *pud) static inline void __pud_free_tlb(struct mmu_gather *tlb, pud_t *pud, unsigned long addr) { - if (pgtable_l4_enabled) { - struct ptdesc *ptdesc = virt_to_ptdesc(pud); - - pagetable_dtor(ptdesc); - riscv_tlb_remove_ptdesc(tlb, ptdesc); - } + if (pgtable_l4_enabled) + riscv_tlb_remove_ptdesc(tlb, virt_to_ptdesc(pud)); } static inline void __p4d_free_tlb(struct mmu_gather *tlb, p4d_t *p4d, unsigned long addr) { - if (pgtable_l5_enabled) { - struct ptdesc *ptdesc = virt_to_ptdesc(p4d); - - pagetable_dtor(ptdesc); + if (pgtable_l5_enabled) riscv_tlb_remove_ptdesc(tlb, virt_to_ptdesc(p4d)); - } } #endif /* __PAGETABLE_PMD_FOLDED */ @@ -142,10 +144,7 @@ static inline pgd_t *pgd_alloc(struct mm_struct *mm) static inline void __pmd_free_tlb(struct mmu_gather *tlb, pmd_t *pmd, unsigned long addr) { - struct ptdesc *ptdesc = virt_to_ptdesc(pmd); - - pagetable_dtor(ptdesc); - riscv_tlb_remove_ptdesc(tlb, ptdesc); + riscv_tlb_remove_ptdesc(tlb, virt_to_ptdesc(pmd)); } #endif /* __PAGETABLE_PMD_FOLDED */ @@ -153,10 +152,7 @@ static inline void __pmd_free_tlb(struct mmu_gather *tlb, pmd_t *pmd, static inline void __pte_free_tlb(struct mmu_gather *tlb, pgtable_t pte, unsigned long addr) { - struct ptdesc *ptdesc = page_ptdesc(pte); - - pagetable_dtor(ptdesc); - riscv_tlb_remove_ptdesc(tlb, ptdesc); + riscv_tlb_remove_ptdesc(tlb, page_ptdesc(pte)); } #endif /* CONFIG_MMU */ diff --git a/arch/riscv/include/asm/tlb.h b/arch/riscv/include/asm/tlb.h index 1f6c38420d8e0..ded8724b3c4f7 100644 --- a/arch/riscv/include/asm/tlb.h +++ b/arch/riscv/include/asm/tlb.h @@ -11,19 +11,13 @@ struct mmu_gather; static void tlb_flush(struct mmu_gather *tlb); #ifdef CONFIG_MMU -#include -/* - * While riscv platforms with riscv_ipi_for_rfence as true require an IPI to - * perform TLB shootdown, some platforms with riscv_ipi_for_rfence as false use - * SBI to perform TLB shootdown. To keep software pagetable walkers safe in this - * case we switch to RCU based table free (MMU_GATHER_RCU_TABLE_FREE). See the - * comment below 'ifdef CONFIG_MMU_GATHER_RCU_TABLE_FREE' in include/asm-generic/tlb.h - * for more details. - */ static inline void __tlb_remove_table(void *table) { - free_page_and_swap_cache(table); + struct ptdesc *ptdesc = (struct ptdesc *)table; + + pagetable_dtor(ptdesc); + pagetable_free(ptdesc); } #endif /* CONFIG_MMU */ From patchwork Mon Dec 23 09:40:57 2024 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Qi Zheng X-Patchwork-Id: 13918713 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from bombadil.infradead.org (bombadil.infradead.org [198.137.202.133]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (No client certificate requested) by smtp.lore.kernel.org (Postfix) with ESMTPS id 8973FE7718B for ; Mon, 23 Dec 2024 09:54:33 +0000 (UTC) DKIM-Signature: v=1; a=rsa-sha256; q=dns/txt; c=relaxed/relaxed; d=lists.infradead.org; s=bombadil.20210309; h=Sender: Content-Transfer-Encoding:Content-Type:List-Subscribe:List-Help:List-Post: List-Archive:List-Unsubscribe:List-Id:MIME-Version:References:In-Reply-To: Message-Id:Date:Subject:Cc:To:From:Reply-To:Content-ID:Content-Description: Resent-Date:Resent-From:Resent-Sender:Resent-To:Resent-Cc:Resent-Message-ID: List-Owner; bh=iI6VTMAbuYCeibJsxRexDHlzqAFQingHNA2lYm+lNck=; b=t2qcLgbjDWdyFM 8OABzHfRJE3bjnYl/Gl4xwX9fzEMEn8wrc8Ct++vTPUZq8q87pmtgi7iTEsS+gZpgtot8iPVEn6Gz KaPdG+Vz8tUpnePFVdNJ27XLjc/z5YLAUfU88vk/Dni4kuYr4CJObNMZxfH8ohMBg7iCFSUqVnlD2 A/SBClmEqejZ/hzJgIcLjd0BLvaOJN6nmlz2OlSKP2tXx3vQcW5duExp8VFfeqGpm3Ga/SYfuCcJH /GzeMnuwzDRq90wpAfaxd6LmPiKIpD5BwuuTcsH6JXYpv8ZlA07XtAR+WheD5Q0TUNAXXqoVUnFpN WX/UrWXD2BRhMZDH9U1Q==; Received: from localhost ([::1] helo=bombadil.infradead.org) by bombadil.infradead.org with esmtp (Exim 4.98 #2 (Red Hat Linux)) id 1tPf8t-00000009jKb-3hNg; Mon, 23 Dec 2024 09:54:27 +0000 Received: from mail-pf1-x430.google.com ([2607:f8b0:4864:20::430]) by bombadil.infradead.org with esmtps (Exim 4.98 #2 (Red Hat Linux)) id 1tPf08-00000009h94-25Ar for linux-riscv@lists.infradead.org; Mon, 23 Dec 2024 09:45:25 +0000 Received: by mail-pf1-x430.google.com with SMTP id d2e1a72fcca58-725ef0397aeso3097451b3a.2 for ; Mon, 23 Dec 2024 01:45:24 -0800 (PST) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=bytedance.com; s=google; t=1734947124; x=1735551924; darn=lists.infradead.org; h=content-transfer-encoding:mime-version:references:in-reply-to :message-id:date:subject:cc:to:from:from:to:cc:subject:date :message-id:reply-to; bh=Rs6Lkouarv58ZoKb+WtHLVo9G63kikGgJGrD7ursbxc=; b=DD42x19JvQOPD930qrXn43L81Q0zFimemVRQP44YlFBqhPZfD31PxjHf7h7s3RTLBm 3PSk4U7GbIwpl/bTcvRKqtqYVl/3T5ywX5vJDK9xTGtOqsi7EM7PF+OdM7wKPPmssFKD OD7vPiWbqk5zcNqhNuDLJZrasuudAI326gLAf16tdKYBs78g1VOX/oG2KGQhFfxk0E8Z wtiDJYwukOB9UH0grkpF1RArKCNYpONlk3OSUMQfo46Gz9NttZw5nEsdJzuyC5vWrRpv kx1qrhqxLfJjUuJIW7IhLvyMDjhdA7EQanvgy9ljGUHh+U4mzSLWB8fe0+vuS8DeSk6U sRtg== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20230601; t=1734947124; x=1735551924; h=content-transfer-encoding:mime-version:references:in-reply-to :message-id:date:subject:cc:to:from:x-gm-message-state:from:to:cc :subject:date:message-id:reply-to; bh=Rs6Lkouarv58ZoKb+WtHLVo9G63kikGgJGrD7ursbxc=; b=JKa/SZ+M6ylrOikWG3I3GccEHuSeshVZ6XwIFEIyYHlYqQ3POvFzEDlk/7PdBtipU6 VeRl/L1adgO6PEUyfq4XDw+SW4B/0way/ugWFgPkOL2KTv+/oS4Hz6bSDVkQd5x9umRy xxWrb85mrXywsH76oorphASffmI9DLJaYJn3oHawuGLz1/Iu8q97cibbgRPbQQl/FxzF 7JrJESGc15SIFs7iEJHwkrCppvPG/yMMUEFdFjZ1zIpUs7bwSROY/o48L8omSr+tL37d EpA8i4UwRxYoOxhfpqOkeQez/SEQJ3jsjiLeX2j+jI6N2k6ZzPq48p8eFPQLQpNvI5m9 nVMA== X-Forwarded-Encrypted: i=1; AJvYcCWmO0zjETxzh/oDqlqCwujLI12FmJDfpw+V9+8UTj8gmh4pPY8X8tN6HLdytkrAg5/LxxgP80ItPiNIkQ==@lists.infradead.org X-Gm-Message-State: AOJu0YylJKdBLGs2JXZz8eBjWUAR9nTt7+LXnjeTd3Qexsfifd5BM9kp sZzeKShE0PBmYznWldgHfDw7cMcQxt0GxPPJ8Z5mz0iFoeQavRcAQ4z+o6K/0Yg= X-Gm-Gg: ASbGnctqHgiU75yLQyn5eauVjNnjPQT2fxBvvqYf5SIOYYtAdn9cUc/v6XlZyQ854Tz vQOcfE+CmT0H0MX7baFUFDGZQpprDLkZrr+N69W4VWv56M706/jKVydzRwsbN8fHPfQw1SnFijD VDS29Y7gmCyd0lzSJ/5Xphiye/bpYZGc9snmrnGEE7xYl8M3deo+RsLjwAB0mK2BxP9M6y8VGIc +AcsSucxlHuYQFAXshzy8beMR6QGqux5bc2Wr7r7tNUkmgvnzoq6O86VzVcWkmSVSiujFPWTxgb YDsS1SOsCXQIlNF7KE2P0g== X-Google-Smtp-Source: AGHT+IEsUkA9RBo0Vmvp6m94/gMVFQ4PHl6SoN42Z+rLx+q01B5oEKa1sTTPOWHs1mFCNObFPbuRxA== X-Received: by 2002:a05:6a20:6a25:b0:1d9:18af:d150 with SMTP id adf61e73a8af0-1e5e05a9e39mr19676430637.21.1734947123919; Mon, 23 Dec 2024 01:45:23 -0800 (PST) Received: from C02DW0BEMD6R.bytedance.net ([203.208.167.150]) by smtp.gmail.com with ESMTPSA id d2e1a72fcca58-72aad8486c6sm7468309b3a.85.2024.12.23.01.45.11 (version=TLS1_3 cipher=TLS_AES_256_GCM_SHA384 bits=256/256); Mon, 23 Dec 2024 01:45:23 -0800 (PST) From: Qi Zheng To: peterz@infradead.org, agordeev@linux.ibm.com, kevin.brodsky@arm.com, tglx@linutronix.de, david@redhat.com, jannh@google.com, hughd@google.com, yuzhao@google.com, willy@infradead.org, muchun.song@linux.dev, vbabka@kernel.org, lorenzo.stoakes@oracle.com, akpm@linux-foundation.org, rientjes@google.com, vishal.moola@gmail.com, arnd@arndb.de, will@kernel.org, aneesh.kumar@kernel.org, npiggin@gmail.com, dave.hansen@linux.intel.com, rppt@kernel.org, ryan.roberts@arm.com Cc: linux-mm@kvack.org, linux-arm-kernel@lists.infradead.org, linuxppc-dev@lists.ozlabs.org, linux-riscv@lists.infradead.org, linux-s390@vger.kernel.org, sparclinux@vger.kernel.org, linux-kernel@vger.kernel.org, x86@kernel.org, linux-arch@vger.kernel.org, linux-csky@vger.kernel.org, linux-hexagon@vger.kernel.org, loongarch@lists.linux.dev, linux-m68k@lists.linux-m68k.org, linux-mips@vger.kernel.org, linux-openrisc@vger.kernel.org, linux-sh@vger.kernel.org, linux-um@lists.infradead.org, Qi Zheng Subject: [PATCH v3 11/17] x86: pgtable: move pagetable_dtor() to __tlb_remove_table() Date: Mon, 23 Dec 2024 17:40:57 +0800 Message-Id: <0dc5a3bf5a692e24379c1d3b879a6d4396f0dbbd.1734945104.git.zhengqi.arch@bytedance.com> X-Mailer: git-send-email 2.24.3 (Apple Git-128) In-Reply-To: References: MIME-Version: 1.0 X-CRM114-Version: 20100106-BlameMichelson ( TRE 0.8.0 (BSD) ) MR-646709E3 X-CRM114-CacheID: sfid-20241223_014524_542218_E2D0E0BE X-CRM114-Status: GOOD ( 11.28 ) X-BeenThere: linux-riscv@lists.infradead.org X-Mailman-Version: 2.1.34 Precedence: list List-Id: List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Sender: "linux-riscv" Errors-To: linux-riscv-bounces+linux-riscv=archiver.kernel.org@lists.infradead.org Move pagetable_dtor() to __tlb_remove_table(), so that ptlock and page table pages can be freed together (regardless of whether RCU is used). This prevents the use-after-free problem where the ptlock is freed immediately but the page table pages is freed later via RCU. Page tables shouldn't have swap cache, so use pagetable_free() instead of free_page_and_swap_cache() to free page table pages. Signed-off-by: Qi Zheng Suggested-by: Peter Zijlstra (Intel) Cc: x86@kernel.org --- arch/x86/include/asm/tlb.h | 17 ++++++++++------- arch/x86/kernel/paravirt.c | 1 + arch/x86/mm/pgtable.c | 12 ++---------- 3 files changed, 13 insertions(+), 17 deletions(-) diff --git a/arch/x86/include/asm/tlb.h b/arch/x86/include/asm/tlb.h index 73f0786181cc9..f64730be5ad67 100644 --- a/arch/x86/include/asm/tlb.h +++ b/arch/x86/include/asm/tlb.h @@ -31,24 +31,27 @@ static inline void tlb_flush(struct mmu_gather *tlb) */ static inline void __tlb_remove_table(void *table) { - free_page_and_swap_cache(table); + struct ptdesc *ptdesc = (struct ptdesc *)table; + + pagetable_dtor(ptdesc); + pagetable_free(ptdesc); } #ifdef CONFIG_PT_RECLAIM static inline void __tlb_remove_table_one_rcu(struct rcu_head *head) { - struct page *page; + struct ptdesc *ptdesc; - page = container_of(head, struct page, rcu_head); - put_page(page); + ptdesc = container_of(head, struct ptdesc, pt_rcu_head); + __tlb_remove_table(ptdesc); } static inline void __tlb_remove_table_one(void *table) { - struct page *page; + struct ptdesc *ptdesc; - page = table; - call_rcu(&page->rcu_head, __tlb_remove_table_one_rcu); + ptdesc = table; + call_rcu(&ptdesc->pt_rcu_head, __tlb_remove_table_one_rcu); } #define __tlb_remove_table_one __tlb_remove_table_one #endif /* CONFIG_PT_RECLAIM */ diff --git a/arch/x86/kernel/paravirt.c b/arch/x86/kernel/paravirt.c index 7bdcf152778c0..46d5d325483b0 100644 --- a/arch/x86/kernel/paravirt.c +++ b/arch/x86/kernel/paravirt.c @@ -62,6 +62,7 @@ void __init native_pv_lock_init(void) #ifndef CONFIG_PT_RECLAIM static void native_tlb_remove_table(struct mmu_gather *tlb, void *table) { + pagetable_dtor(table); tlb_remove_page(tlb, table); } #else diff --git a/arch/x86/mm/pgtable.c b/arch/x86/mm/pgtable.c index a6cd9660e29ec..a0b0e501ba663 100644 --- a/arch/x86/mm/pgtable.c +++ b/arch/x86/mm/pgtable.c @@ -23,6 +23,7 @@ EXPORT_SYMBOL(physical_mask); static inline void paravirt_tlb_remove_table(struct mmu_gather *tlb, void *table) { + pagetable_dtor(table); tlb_remove_page(tlb, table); } #else @@ -60,7 +61,6 @@ early_param("userpte", setup_userpte); void ___pte_free_tlb(struct mmu_gather *tlb, struct page *pte) { - pagetable_dtor(page_ptdesc(pte)); paravirt_release_pte(page_to_pfn(pte)); paravirt_tlb_remove_table(tlb, pte); } @@ -68,7 +68,6 @@ void ___pte_free_tlb(struct mmu_gather *tlb, struct page *pte) #if CONFIG_PGTABLE_LEVELS > 2 void ___pmd_free_tlb(struct mmu_gather *tlb, pmd_t *pmd) { - struct ptdesc *ptdesc = virt_to_ptdesc(pmd); paravirt_release_pmd(__pa(pmd) >> PAGE_SHIFT); /* * NOTE! For PAE, any changes to the top page-directory-pointer-table @@ -77,16 +76,12 @@ void ___pmd_free_tlb(struct mmu_gather *tlb, pmd_t *pmd) #ifdef CONFIG_X86_PAE tlb->need_flush_all = 1; #endif - pagetable_dtor(ptdesc); - paravirt_tlb_remove_table(tlb, ptdesc_page(ptdesc)); + paravirt_tlb_remove_table(tlb, virt_to_page(pmd)); } #if CONFIG_PGTABLE_LEVELS > 3 void ___pud_free_tlb(struct mmu_gather *tlb, pud_t *pud) { - struct ptdesc *ptdesc = virt_to_ptdesc(pud); - - pagetable_dtor(ptdesc); paravirt_release_pud(__pa(pud) >> PAGE_SHIFT); paravirt_tlb_remove_table(tlb, virt_to_page(pud)); } @@ -94,9 +89,6 @@ void ___pud_free_tlb(struct mmu_gather *tlb, pud_t *pud) #if CONFIG_PGTABLE_LEVELS > 4 void ___p4d_free_tlb(struct mmu_gather *tlb, p4d_t *p4d) { - struct ptdesc *ptdesc = virt_to_ptdesc(p4d); - - pagetable_dtor(ptdesc); paravirt_release_p4d(__pa(p4d) >> PAGE_SHIFT); paravirt_tlb_remove_table(tlb, virt_to_page(p4d)); } From patchwork Mon Dec 23 09:40:58 2024 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Qi Zheng X-Patchwork-Id: 13918714 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from bombadil.infradead.org (bombadil.infradead.org [198.137.202.133]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (No client certificate requested) by smtp.lore.kernel.org (Postfix) with ESMTPS id 5858AE7718D for ; Mon, 23 Dec 2024 09:55:49 +0000 (UTC) DKIM-Signature: v=1; a=rsa-sha256; q=dns/txt; c=relaxed/relaxed; d=lists.infradead.org; s=bombadil.20210309; h=Sender: Content-Transfer-Encoding:Content-Type:List-Subscribe:List-Help:List-Post: List-Archive:List-Unsubscribe:List-Id:MIME-Version:References:In-Reply-To: Message-Id:Date:Subject:Cc:To:From:Reply-To:Content-ID:Content-Description: Resent-Date:Resent-From:Resent-Sender:Resent-To:Resent-Cc:Resent-Message-ID: List-Owner; bh=rwXVFhlXslS1ZqogietsCOZDnMu0wrIfoJHcoODLu3E=; b=ApyR0TlWUOSVHG OxXb8pzcnslA5VeEV/fuVklbyruwqZhv5SzwQLJP6Bg6bTCCvMIhvXWzvNw+YHabEUnKezBtFH6va Qn6vXwl6apUbIFgbUB4z6XepUYKenFzQp9KvuiUgylo+In+TKeLdEsNtR+n5uFSlv83yhHY/F5V+w 52C5VZdGtRb1gBGZLN5Xp23QiccPF46IRmYt7+QYayUErw9gaZTvYMEOO2LTz+TMP3E+ejxhmvtq+ NaXcMSlmQSJ1c5KooPqXq6Rh3yDcMsGTv2Xlr8YizbvRyNGmmIgK7unuz+ve39Z20zfBVESmu4zZG 6RN38E7ug2VCCmEAFT/A==; Received: from localhost ([::1] helo=bombadil.infradead.org) by bombadil.infradead.org with esmtp (Exim 4.98 #2 (Red Hat Linux)) id 1tPfA9-00000009jZr-0sBt; Mon, 23 Dec 2024 09:55:45 +0000 Received: from mail-pf1-x42a.google.com ([2607:f8b0:4864:20::42a]) by bombadil.infradead.org with esmtps (Exim 4.98 #2 (Red Hat Linux)) id 1tPf0L-00000009hGB-0plF for linux-riscv@lists.infradead.org; Mon, 23 Dec 2024 09:45:38 +0000 Received: by mail-pf1-x42a.google.com with SMTP id d2e1a72fcca58-728f337a921so4004071b3a.3 for ; Mon, 23 Dec 2024 01:45:37 -0800 (PST) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=bytedance.com; s=google; t=1734947136; x=1735551936; darn=lists.infradead.org; h=content-transfer-encoding:mime-version:references:in-reply-to :message-id:date:subject:cc:to:from:from:to:cc:subject:date :message-id:reply-to; bh=jyEzElX2/Cl2Hb0vACePBo9dDp7jwROhL3d0Mq9f7ig=; b=dHLqc/emEC32IdhbnFyJBVGn0kpxazeZeV79vtASzlkWZm7HDal4Jno5VPoETQ+K9e sugnFzXcWwF9ggYIDSZPbg9j5cn/dFkIsoLUZSN2Z4jWHEG6Rs/YrSCe+Ds69N9eck4N LltUJiJBEMbebWkHHoJebCuOybzlGGHXjlnxuN5mG/4BL7Z06byIbPffZaOxUbc+e000 3Tz076Q5lUtb0AiHHV9R9lyc92pIzOZRUzNP2Pi0R1fMFgjYM2ocfX3JFlZcjwCSnO5F Pso5//zrMdNIVtnCnKvBnH46uTZk4zvJPx0HocH9EQq2VvkhJg71UKB/lk1ulWmxem/1 rDuA== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20230601; t=1734947136; x=1735551936; h=content-transfer-encoding:mime-version:references:in-reply-to :message-id:date:subject:cc:to:from:x-gm-message-state:from:to:cc :subject:date:message-id:reply-to; bh=jyEzElX2/Cl2Hb0vACePBo9dDp7jwROhL3d0Mq9f7ig=; b=ao2lyb+zLiPPBmkwNAAcXPO5aj1C+IadqMd2Yb/jRWotHhx+HQjzEtHJMypO2eX8pN 48LdvEqh1bkpY2ptPjFly48wtdOreBapTQ+UzxwxWNYukd0YnN+fSfyzoqgJXC51elzS Mqor4HZ3zOfXAvefGT7h6+pClOnXZeQXtjCTNIzX7rqMHsoM4zKzhUdJRwG+eQqbEPvw U+wawFOMfeZ/Dnf/NBgLDlCU5jDFe0KShQlAoUS4qEI8+KTMhBO3f+2U0ZGG6echzdXt qdHUgl6U4nZhSa/037s+KUREPhTvCdQFWyJmDZoMoUHF/QwRmFJAyv99GBQy7LprBL6k LoNw== X-Forwarded-Encrypted: i=1; AJvYcCUlhbFRaX8sTZI4E/4rWNTodcTlmMmtE0WrxUSxXgF6e+oEKqbu5ONz83D9Xbb5GBW76fEGuXt/JH0wnQ==@lists.infradead.org X-Gm-Message-State: AOJu0YzInQT9lznbhHciAyvcbwrvVKhDVDKUfAttHotZmweRkjkvQwVX /35Fu6sLjxLfFcmARVNwgcXz/QwouGc1Vu7MIy7DBV1NYMbb9Ez7WcIhmvVARPY= X-Gm-Gg: ASbGncsG1MC4c8YQ+3yEZfl5QZY7C3H7fglgog+HOQgFn9d4UdQpHwgYrKolBkHULwN vSt+vHaWEv7ztan2wXBKCYI7lYeO/1GLalPTE1/TzdPkyqIIBUCpU2uecuNls6oyFlv7MkK0F1j 6Spw+pxkJUM2yG2DTyNh1x/3wh43Bqmkbc4dAx8+s8rZYgxzjTTztdROwRlwgxhYBJZptPCr8GD TH1Z1x1bkwv1qq8JzkhT5lNWjJ/byVhocQv3c65SGoViS4iog/FdI+x1fv/25A3nWVvo3YHSDbG OC3568Dc/qq57sj7vrr6dw== X-Google-Smtp-Source: AGHT+IFFxsa0Y8D8QMIcsR8+VA1yaCfovsoDpuS31clip0UDjebZ+1TP8q/TeJDt5eTrdc0wzKH4iQ== X-Received: by 2002:a05:6a00:399a:b0:725:b7dd:e668 with SMTP id d2e1a72fcca58-72abdebb868mr13954007b3a.17.1734947136545; Mon, 23 Dec 2024 01:45:36 -0800 (PST) Received: from C02DW0BEMD6R.bytedance.net ([203.208.167.150]) by smtp.gmail.com with ESMTPSA id d2e1a72fcca58-72aad8486c6sm7468309b3a.85.2024.12.23.01.45.24 (version=TLS1_3 cipher=TLS_AES_256_GCM_SHA384 bits=256/256); Mon, 23 Dec 2024 01:45:36 -0800 (PST) From: Qi Zheng To: peterz@infradead.org, agordeev@linux.ibm.com, kevin.brodsky@arm.com, tglx@linutronix.de, david@redhat.com, jannh@google.com, hughd@google.com, yuzhao@google.com, willy@infradead.org, muchun.song@linux.dev, vbabka@kernel.org, lorenzo.stoakes@oracle.com, akpm@linux-foundation.org, rientjes@google.com, vishal.moola@gmail.com, arnd@arndb.de, will@kernel.org, aneesh.kumar@kernel.org, npiggin@gmail.com, dave.hansen@linux.intel.com, rppt@kernel.org, ryan.roberts@arm.com Cc: linux-mm@kvack.org, linux-arm-kernel@lists.infradead.org, linuxppc-dev@lists.ozlabs.org, linux-riscv@lists.infradead.org, linux-s390@vger.kernel.org, sparclinux@vger.kernel.org, linux-kernel@vger.kernel.org, x86@kernel.org, linux-arch@vger.kernel.org, linux-csky@vger.kernel.org, linux-hexagon@vger.kernel.org, loongarch@lists.linux.dev, linux-m68k@lists.linux-m68k.org, linux-mips@vger.kernel.org, linux-openrisc@vger.kernel.org, linux-sh@vger.kernel.org, linux-um@lists.infradead.org, Qi Zheng Subject: [PATCH v3 12/17] s390: pgtable: also move pagetable_dtor() of PxD to __tlb_remove_table() Date: Mon, 23 Dec 2024 17:40:58 +0800 Message-Id: X-Mailer: git-send-email 2.24.3 (Apple Git-128) In-Reply-To: References: MIME-Version: 1.0 X-CRM114-Version: 20100106-BlameMichelson ( TRE 0.8.0 (BSD) ) MR-646709E3 X-CRM114-CacheID: sfid-20241223_014537_251706_AB2A310B X-CRM114-Status: UNSURE ( 9.58 ) X-CRM114-Notice: Please train this message. X-BeenThere: linux-riscv@lists.infradead.org X-Mailman-Version: 2.1.34 Precedence: list List-Id: List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Sender: "linux-riscv" Errors-To: linux-riscv-bounces+linux-riscv=archiver.kernel.org@lists.infradead.org To unify the PxD and PTE TLB free path, also move the pagetable_dtor() of PMD|PUD|P4D to __tlb_remove_table(). Signed-off-by: Qi Zheng Suggested-by: Peter Zijlstra (Intel) Cc: linux-s390@vger.kernel.org --- arch/s390/include/asm/tlb.h | 3 --- arch/s390/mm/pgalloc.c | 14 ++++---------- 2 files changed, 4 insertions(+), 13 deletions(-) diff --git a/arch/s390/include/asm/tlb.h b/arch/s390/include/asm/tlb.h index 74b6fba4c2ee3..79df7c0932c56 100644 --- a/arch/s390/include/asm/tlb.h +++ b/arch/s390/include/asm/tlb.h @@ -102,7 +102,6 @@ static inline void pmd_free_tlb(struct mmu_gather *tlb, pmd_t *pmd, { if (mm_pmd_folded(tlb->mm)) return; - pagetable_dtor(virt_to_ptdesc(pmd)); __tlb_adjust_range(tlb, address, PAGE_SIZE); tlb->mm->context.flush_mm = 1; tlb->freed_tables = 1; @@ -122,7 +121,6 @@ static inline void pud_free_tlb(struct mmu_gather *tlb, pud_t *pud, { if (mm_pud_folded(tlb->mm)) return; - pagetable_dtor(virt_to_ptdesc(pud)); tlb->mm->context.flush_mm = 1; tlb->freed_tables = 1; tlb->cleared_p4ds = 1; @@ -141,7 +139,6 @@ static inline void p4d_free_tlb(struct mmu_gather *tlb, p4d_t *p4d, { if (mm_p4d_folded(tlb->mm)) return; - pagetable_dtor(virt_to_ptdesc(p4d)); __tlb_adjust_range(tlb, address, PAGE_SIZE); tlb->mm->context.flush_mm = 1; tlb->freed_tables = 1; diff --git a/arch/s390/mm/pgalloc.c b/arch/s390/mm/pgalloc.c index 569de24d33761..c73b89811a264 100644 --- a/arch/s390/mm/pgalloc.c +++ b/arch/s390/mm/pgalloc.c @@ -180,7 +180,7 @@ unsigned long *page_table_alloc(struct mm_struct *mm) return table; } -static void pagetable_pte_dtor_free(struct ptdesc *ptdesc) +static void pagetable_dtor_free(struct ptdesc *ptdesc) { pagetable_dtor(ptdesc); pagetable_free(ptdesc); @@ -190,20 +190,14 @@ void page_table_free(struct mm_struct *mm, unsigned long *table) { struct ptdesc *ptdesc = virt_to_ptdesc(table); - pagetable_pte_dtor_free(ptdesc); + pagetable_dtor_free(ptdesc); } void __tlb_remove_table(void *table) { struct ptdesc *ptdesc = virt_to_ptdesc(table); - struct page *page = ptdesc_page(ptdesc); - if (compound_order(page) == CRST_ALLOC_ORDER) { - /* pmd, pud, or p4d */ - pagetable_free(ptdesc); - return; - } - pagetable_pte_dtor_free(ptdesc); + pagetable_dtor_free(ptdesc); } #ifdef CONFIG_TRANSPARENT_HUGEPAGE @@ -211,7 +205,7 @@ static void pte_free_now(struct rcu_head *head) { struct ptdesc *ptdesc = container_of(head, struct ptdesc, pt_rcu_head); - pagetable_pte_dtor_free(ptdesc); + pagetable_dtor_free(ptdesc); } void pte_free_defer(struct mm_struct *mm, pgtable_t pgtable) From patchwork Mon Dec 23 09:40:59 2024 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Qi Zheng X-Patchwork-Id: 13918715 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from bombadil.infradead.org (bombadil.infradead.org [198.137.202.133]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (No client certificate requested) by smtp.lore.kernel.org (Postfix) with ESMTPS id 281D4E7718B for ; Mon, 23 Dec 2024 09:57:00 +0000 (UTC) DKIM-Signature: v=1; a=rsa-sha256; q=dns/txt; c=relaxed/relaxed; d=lists.infradead.org; s=bombadil.20210309; h=Sender: Content-Transfer-Encoding:Content-Type:List-Subscribe:List-Help:List-Post: List-Archive:List-Unsubscribe:List-Id:MIME-Version:References:In-Reply-To: Message-Id:Date:Subject:Cc:To:From:Reply-To:Content-ID:Content-Description: Resent-Date:Resent-From:Resent-Sender:Resent-To:Resent-Cc:Resent-Message-ID: List-Owner; bh=eedrNzAO5XaYHtJdwhCeBkp619iU9877RZ00/pYjsno=; b=qXJ34rW3sdmiSl GGtt5EJz0JVFYXsonMjWcnzTJEeZc+D5NdG7paGqnAm+kPr14kQiE4BE27raCHhEkVzB5mNbKAJJl 5fDIsb/yb76WvVGUf+kGq8TV47zDe6VQR9X8KqSGpXBHnEk7D6VEmpaYUpuFbEG7AZI7XmPU7XCp2 GGYoxB8SSLkCu37UHRHamJek0x5FTEE7+4K3gGJHRSZ/Ny7TGUtODcxapt7dIM9p6bb6pZGGmPM1E HtXA+ZCUUqkQ71xqf9Z+3XuzL+7hdynQVes3Vf4rEO/VCDQGyQR5CeTtuGqznceCcPAC4cLiy7MSf YNdMXy/pzaUOs4nAmcSA==; Received: from localhost ([::1] helo=bombadil.infradead.org) by bombadil.infradead.org with esmtp (Exim 4.98 #2 (Red Hat Linux)) id 1tPfBH-00000009jjZ-2rks; Mon, 23 Dec 2024 09:56:55 +0000 Received: from mail-pf1-x430.google.com ([2607:f8b0:4864:20::430]) by bombadil.infradead.org with esmtps (Exim 4.98 #2 (Red Hat Linux)) id 1tPf0X-00000009hQ8-3kxv for linux-riscv@lists.infradead.org; Mon, 23 Dec 2024 09:45:51 +0000 Received: by mail-pf1-x430.google.com with SMTP id d2e1a72fcca58-72764c995e5so2421225b3a.2 for ; Mon, 23 Dec 2024 01:45:49 -0800 (PST) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=bytedance.com; s=google; t=1734947149; x=1735551949; darn=lists.infradead.org; h=content-transfer-encoding:mime-version:references:in-reply-to :message-id:date:subject:cc:to:from:from:to:cc:subject:date :message-id:reply-to; bh=m/+0uIvp7xUeTeFwO63TH1y++vtASeDM7X35qnuktEM=; b=aj1RsLX9syCardp0apiJK5ItIycllpxMHizlIW5C54znV3aBO79iF4LKdiEYCILCIs 9ksXkiL7z1mvWXpo8nnhVRAEHkUaYAUTEdKeEwvTIK3ousCBD1WTzrbZ5t206nGBxkhH yqT/cjv1M4qXKTl5/LwyJcOMhRRJsP4B05Da1DpvOQEb3vaA1HybdAx2AAHCzpQW1PTV UVmT1wZdKtmrBMUtGeGzDaMm81lv9ExrzhgWIkp53sA2P05GQdx4gQbdvImbTuo4CVI8 Cye1/O91tZNQeyN+ZQCxte7hZfzgHFCwohhX1bJkm6LR4kZtGUmVSiMQzrA5/2IjQ4BO +LQw== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20230601; t=1734947149; x=1735551949; h=content-transfer-encoding:mime-version:references:in-reply-to :message-id:date:subject:cc:to:from:x-gm-message-state:from:to:cc :subject:date:message-id:reply-to; bh=m/+0uIvp7xUeTeFwO63TH1y++vtASeDM7X35qnuktEM=; b=OrSZilPD4yp66iSAnKFVNENVLOTR6Lkx0YroFVDpvOk/xmSERbSST68ac0FMiX3Pat /D4P+I0wXSNqDjC73gTKeJeJnUxItBqqYrByr0OzbbD5UzjbsUaQDOn3/RKe2DLFysMO +XWN99Yhjkg0KH5NeH+/o8Vf0YmxZkd2Owguaq1mW0RmrtTIT4svn/0ch+7He0BjgZkw w9p/UkV+vQjrh79GbhACi2sc2NNTBTtlJ3ehWRwL9XNuBz/KohD7NcDME/pxvvZDco0P G37dRDc+gEvzWW3Yl1r6NWWVcurjzXjsmAP2HCOKPJkAcpdSecv9loOWRvOhZHXxIiUV LEhQ== X-Forwarded-Encrypted: i=1; AJvYcCW9U7YFwIVFfjG4+TWKFzKTYf+u+2dhBk/FuvKusx1XgS/9/QXB5cn8vwFMIKbVjckX5LllBGID1lh6TQ==@lists.infradead.org X-Gm-Message-State: AOJu0YxcU/g9N109OtoLnw5Kw8YbL13aMhDr41rOlwLUwdREpx85177d 0qo05tS7gqK7XXfBe2GdYwiyeID/29m96losedaf4I9yE7n4tSaEZ/ni70s6Sj8= X-Gm-Gg: ASbGncs41S6bpnpvVtKJv2r+NkntipaUcy8Vzp9JDXWjeaOkROIMdRU4WEnVtzNKw2e YxoSvQ5eY/ewwWXhxTnw1j4XIGXKd7EOABHRpcNctW0SebvW6cXfGofMnH/kwt3nBoaJ5wbcT4P ExZxHONS2VcY+7SQCM2vKdIuiUJdmIE2DLLD3qd79CyOf/I5bwQ+zhj2yS6shg9gU7KRl3K13nQ Q8sIlrR5+PC3SjRzKn22kLnpDNJZB+RkCTMlItfJJgI7wkCZtUInrPm73cBHOva0G6BXhBra1oM l7e8qp2bkPVCKueMSe7PNQ== X-Google-Smtp-Source: AGHT+IHlutX536vagq2BMCvNjirBm1uJlF1dzaH3GgOc0ZHj3lAXm7kK/AuGYqDlsmkXqpl9hOqJ3g== X-Received: by 2002:a05:6a00:92a4:b0:728:e1f9:b680 with SMTP id d2e1a72fcca58-72abdd7ac89mr17103004b3a.6.1734947149240; Mon, 23 Dec 2024 01:45:49 -0800 (PST) Received: from C02DW0BEMD6R.bytedance.net ([203.208.167.150]) by smtp.gmail.com with ESMTPSA id d2e1a72fcca58-72aad8486c6sm7468309b3a.85.2024.12.23.01.45.37 (version=TLS1_3 cipher=TLS_AES_256_GCM_SHA384 bits=256/256); Mon, 23 Dec 2024 01:45:48 -0800 (PST) From: Qi Zheng To: peterz@infradead.org, agordeev@linux.ibm.com, kevin.brodsky@arm.com, tglx@linutronix.de, david@redhat.com, jannh@google.com, hughd@google.com, yuzhao@google.com, willy@infradead.org, muchun.song@linux.dev, vbabka@kernel.org, lorenzo.stoakes@oracle.com, akpm@linux-foundation.org, rientjes@google.com, vishal.moola@gmail.com, arnd@arndb.de, will@kernel.org, aneesh.kumar@kernel.org, npiggin@gmail.com, dave.hansen@linux.intel.com, rppt@kernel.org, ryan.roberts@arm.com Cc: linux-mm@kvack.org, linux-arm-kernel@lists.infradead.org, linuxppc-dev@lists.ozlabs.org, linux-riscv@lists.infradead.org, linux-s390@vger.kernel.org, sparclinux@vger.kernel.org, linux-kernel@vger.kernel.org, x86@kernel.org, linux-arch@vger.kernel.org, linux-csky@vger.kernel.org, linux-hexagon@vger.kernel.org, loongarch@lists.linux.dev, linux-m68k@lists.linux-m68k.org, linux-mips@vger.kernel.org, linux-openrisc@vger.kernel.org, linux-sh@vger.kernel.org, linux-um@lists.infradead.org, Qi Zheng Subject: [PATCH v3 13/17] mm: pgtable: introduce generic __tlb_remove_table() Date: Mon, 23 Dec 2024 17:40:59 +0800 Message-Id: X-Mailer: git-send-email 2.24.3 (Apple Git-128) In-Reply-To: References: MIME-Version: 1.0 X-CRM114-Version: 20100106-BlameMichelson ( TRE 0.8.0 (BSD) ) MR-646709E3 X-CRM114-CacheID: sfid-20241223_014549_973518_ED64AECE X-CRM114-Status: GOOD ( 15.10 ) X-BeenThere: linux-riscv@lists.infradead.org X-Mailman-Version: 2.1.34 Precedence: list List-Id: List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Sender: "linux-riscv" Errors-To: linux-riscv-bounces+linux-riscv=archiver.kernel.org@lists.infradead.org Several architectures (arm, arm64, riscv and x86) define exactly the same __tlb_remove_table(), just introduce generic __tlb_remove_table() to eliminate these duplications. The s390 __tlb_remove_table() is nearly the same, so also make s390 __tlb_remove_table() version generic. Signed-off-by: Qi Zheng --- arch/arm/include/asm/tlb.h | 9 --------- arch/arm64/include/asm/tlb.h | 7 ------- arch/powerpc/include/asm/tlb.h | 1 + arch/riscv/include/asm/tlb.h | 12 ------------ arch/s390/include/asm/tlb.h | 9 ++++----- arch/s390/mm/pgalloc.c | 7 ------- arch/sparc/include/asm/tlb_32.h | 1 + arch/sparc/include/asm/tlb_64.h | 1 + arch/x86/include/asm/tlb.h | 17 ----------------- include/asm-generic/tlb.h | 15 +++++++++++++-- 10 files changed, 20 insertions(+), 59 deletions(-) diff --git a/arch/arm/include/asm/tlb.h b/arch/arm/include/asm/tlb.h index 264ab635e807a..ea4fbe7b17f6f 100644 --- a/arch/arm/include/asm/tlb.h +++ b/arch/arm/include/asm/tlb.h @@ -27,15 +27,6 @@ #else /* !CONFIG_MMU */ #include - -static inline void __tlb_remove_table(void *_table) -{ - struct ptdesc *ptdesc = (struct ptdesc *)_table; - - pagetable_dtor(ptdesc); - pagetable_free(ptdesc); -} - #include static inline void diff --git a/arch/arm64/include/asm/tlb.h b/arch/arm64/include/asm/tlb.h index 93591a80b5bfb..8d762607285cc 100644 --- a/arch/arm64/include/asm/tlb.h +++ b/arch/arm64/include/asm/tlb.h @@ -10,13 +10,6 @@ #include -static inline void __tlb_remove_table(void *_table) -{ - struct ptdesc *ptdesc = (struct ptdesc *)_table; - - pagetable_dtor(ptdesc); - pagetable_free(ptdesc); -} #define tlb_flush tlb_flush static void tlb_flush(struct mmu_gather *tlb); diff --git a/arch/powerpc/include/asm/tlb.h b/arch/powerpc/include/asm/tlb.h index 1ca7d4c4b90db..2058e8d3e0138 100644 --- a/arch/powerpc/include/asm/tlb.h +++ b/arch/powerpc/include/asm/tlb.h @@ -37,6 +37,7 @@ extern void tlb_flush(struct mmu_gather *tlb); */ #define tlb_needs_table_invalidate() radix_enabled() +#define __HAVE_ARCH_TLB_REMOVE_TABLE /* Get the generic bits... */ #include diff --git a/arch/riscv/include/asm/tlb.h b/arch/riscv/include/asm/tlb.h index ded8724b3c4f7..50b63b5c15bd8 100644 --- a/arch/riscv/include/asm/tlb.h +++ b/arch/riscv/include/asm/tlb.h @@ -10,18 +10,6 @@ struct mmu_gather; static void tlb_flush(struct mmu_gather *tlb); -#ifdef CONFIG_MMU - -static inline void __tlb_remove_table(void *table) -{ - struct ptdesc *ptdesc = (struct ptdesc *)table; - - pagetable_dtor(ptdesc); - pagetable_free(ptdesc); -} - -#endif /* CONFIG_MMU */ - #define tlb_flush tlb_flush #include diff --git a/arch/s390/include/asm/tlb.h b/arch/s390/include/asm/tlb.h index 79df7c0932c56..da4a7d175f69c 100644 --- a/arch/s390/include/asm/tlb.h +++ b/arch/s390/include/asm/tlb.h @@ -22,7 +22,6 @@ * Pages used for the page tables is a different story. FIXME: more */ -void __tlb_remove_table(void *_table); static inline void tlb_flush(struct mmu_gather *tlb); static inline bool __tlb_remove_page_size(struct mmu_gather *tlb, struct page *page, bool delay_rmap, int page_size); @@ -87,7 +86,7 @@ static inline void pte_free_tlb(struct mmu_gather *tlb, pgtable_t pte, tlb->cleared_pmds = 1; if (mm_alloc_pgste(tlb->mm)) gmap_unlink(tlb->mm, (unsigned long *)pte, address); - tlb_remove_ptdesc(tlb, pte); + tlb_remove_ptdesc(tlb, virt_to_ptdesc(pte)); } /* @@ -106,7 +105,7 @@ static inline void pmd_free_tlb(struct mmu_gather *tlb, pmd_t *pmd, tlb->mm->context.flush_mm = 1; tlb->freed_tables = 1; tlb->cleared_puds = 1; - tlb_remove_ptdesc(tlb, pmd); + tlb_remove_ptdesc(tlb, virt_to_ptdesc(pmd)); } /* @@ -124,7 +123,7 @@ static inline void pud_free_tlb(struct mmu_gather *tlb, pud_t *pud, tlb->mm->context.flush_mm = 1; tlb->freed_tables = 1; tlb->cleared_p4ds = 1; - tlb_remove_ptdesc(tlb, pud); + tlb_remove_ptdesc(tlb, virt_to_ptdesc(pud)); } /* @@ -142,7 +141,7 @@ static inline void p4d_free_tlb(struct mmu_gather *tlb, p4d_t *p4d, __tlb_adjust_range(tlb, address, PAGE_SIZE); tlb->mm->context.flush_mm = 1; tlb->freed_tables = 1; - tlb_remove_ptdesc(tlb, p4d); + tlb_remove_ptdesc(tlb, virt_to_ptdesc(p4d)); } #endif /* _S390_TLB_H */ diff --git a/arch/s390/mm/pgalloc.c b/arch/s390/mm/pgalloc.c index c73b89811a264..3e002dea6278f 100644 --- a/arch/s390/mm/pgalloc.c +++ b/arch/s390/mm/pgalloc.c @@ -193,13 +193,6 @@ void page_table_free(struct mm_struct *mm, unsigned long *table) pagetable_dtor_free(ptdesc); } -void __tlb_remove_table(void *table) -{ - struct ptdesc *ptdesc = virt_to_ptdesc(table); - - pagetable_dtor_free(ptdesc); -} - #ifdef CONFIG_TRANSPARENT_HUGEPAGE static void pte_free_now(struct rcu_head *head) { diff --git a/arch/sparc/include/asm/tlb_32.h b/arch/sparc/include/asm/tlb_32.h index 5cd28a8793e39..910254867dfbd 100644 --- a/arch/sparc/include/asm/tlb_32.h +++ b/arch/sparc/include/asm/tlb_32.h @@ -2,6 +2,7 @@ #ifndef _SPARC_TLB_H #define _SPARC_TLB_H +#define __HAVE_ARCH_TLB_REMOVE_TABLE #include #endif /* _SPARC_TLB_H */ diff --git a/arch/sparc/include/asm/tlb_64.h b/arch/sparc/include/asm/tlb_64.h index 3037187482db7..1a6e694418e39 100644 --- a/arch/sparc/include/asm/tlb_64.h +++ b/arch/sparc/include/asm/tlb_64.h @@ -33,6 +33,7 @@ void flush_tlb_pending(void); #define tlb_needs_table_invalidate() (false) #endif +#define __HAVE_ARCH_TLB_REMOVE_TABLE #include #endif /* _SPARC64_TLB_H */ diff --git a/arch/x86/include/asm/tlb.h b/arch/x86/include/asm/tlb.h index f64730be5ad67..3858dbf75880e 100644 --- a/arch/x86/include/asm/tlb.h +++ b/arch/x86/include/asm/tlb.h @@ -20,23 +20,6 @@ static inline void tlb_flush(struct mmu_gather *tlb) flush_tlb_mm_range(tlb->mm, start, end, stride_shift, tlb->freed_tables); } -/* - * While x86 architecture in general requires an IPI to perform TLB - * shootdown, enablement code for several hypervisors overrides - * .flush_tlb_others hook in pv_mmu_ops and implements it by issuing - * a hypercall. To keep software pagetable walkers safe in this case we - * switch to RCU based table free (MMU_GATHER_RCU_TABLE_FREE). See the comment - * below 'ifdef CONFIG_MMU_GATHER_RCU_TABLE_FREE' in include/asm-generic/tlb.h - * for more details. - */ -static inline void __tlb_remove_table(void *table) -{ - struct ptdesc *ptdesc = (struct ptdesc *)table; - - pagetable_dtor(ptdesc); - pagetable_free(ptdesc); -} - #ifdef CONFIG_PT_RECLAIM static inline void __tlb_remove_table_one_rcu(struct rcu_head *head) { diff --git a/include/asm-generic/tlb.h b/include/asm-generic/tlb.h index 709830274b756..69de47c7ef3c5 100644 --- a/include/asm-generic/tlb.h +++ b/include/asm-generic/tlb.h @@ -153,8 +153,9 @@ * * Useful if your architecture has non-page page directories. * - * When used, an architecture is expected to provide __tlb_remove_table() - * which does the actual freeing of these pages. + * When used, an architecture is expected to provide __tlb_remove_table() or + * use the generic __tlb_remove_table(), which does the actual freeing of these + * pages. * * MMU_GATHER_RCU_TABLE_FREE * @@ -207,6 +208,16 @@ struct mmu_table_batch { #define MAX_TABLE_BATCH \ ((PAGE_SIZE - sizeof(struct mmu_table_batch)) / sizeof(void *)) +#ifndef __HAVE_ARCH_TLB_REMOVE_TABLE +static inline void __tlb_remove_table(void *table) +{ + struct ptdesc *ptdesc = (struct ptdesc *)table; + + pagetable_dtor(ptdesc); + pagetable_free(ptdesc); +} +#endif + extern void tlb_remove_table(struct mmu_gather *tlb, void *table); #else /* !CONFIG_MMU_GATHER_HAVE_TABLE_FREE */ From patchwork Mon Dec 23 09:41:00 2024 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Qi Zheng X-Patchwork-Id: 13918727 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from bombadil.infradead.org (bombadil.infradead.org [198.137.202.133]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (No client certificate requested) by smtp.lore.kernel.org (Postfix) with ESMTPS id E602BE7718D for ; Mon, 23 Dec 2024 09:59:24 +0000 (UTC) DKIM-Signature: v=1; a=rsa-sha256; q=dns/txt; c=relaxed/relaxed; d=lists.infradead.org; s=bombadil.20210309; h=Sender: Content-Transfer-Encoding:Content-Type:List-Subscribe:List-Help:List-Post: List-Archive:List-Unsubscribe:List-Id:MIME-Version:References:In-Reply-To: Message-Id:Date:Subject:Cc:To:From:Reply-To:Content-ID:Content-Description: Resent-Date:Resent-From:Resent-Sender:Resent-To:Resent-Cc:Resent-Message-ID: List-Owner; bh=eLF1bvpK+f+aX3mGJ5gDYGE+kAdA6sGstjzK1JWKAas=; b=2ag9fXaxpF+RSH B+AXhqa2N55cQrnxFcVrihAF06qJbA/FROvsbp+iqKZwOa3JFcFIyFwxtQAH+KlDdXzAseI18pA8W +IGG80j4aM6dWjhwsbl7/RIyQO9Tqh1PWs1CAlfmmqCPwwBcKgxQAskYKRKWf501Q33tUQg2zcAoD TZSAbilEjMao3c0oNxZLMmCBXghmD1QN2CdDN9uDfE/zdzWGzfm2e6u/LO32FUqT0kFKrVzocjB2X XtPSDxD3pZlVxPEpC8XwpioUSIm1uzKjxLch6BYJd9qdZOBzkd7V0t5QG8JotFjUfuAowmQ1cjYki njk05wzgP0JLQipWSxlA==; Received: from localhost ([::1] helo=bombadil.infradead.org) by bombadil.infradead.org with esmtp (Exim 4.98 #2 (Red Hat Linux)) id 1tPfDa-00000009k3s-2QCq; Mon, 23 Dec 2024 09:59:18 +0000 Received: from mail-pf1-x435.google.com ([2607:f8b0:4864:20::435]) by bombadil.infradead.org with esmtps (Exim 4.98 #2 (Red Hat Linux)) id 1tPf0k-00000009hWr-1spq for linux-riscv@lists.infradead.org; Mon, 23 Dec 2024 09:46:04 +0000 Received: by mail-pf1-x435.google.com with SMTP id d2e1a72fcca58-728e729562fso3164700b3a.0 for ; Mon, 23 Dec 2024 01:46:02 -0800 (PST) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=bytedance.com; s=google; t=1734947162; x=1735551962; darn=lists.infradead.org; h=content-transfer-encoding:mime-version:references:in-reply-to :message-id:date:subject:cc:to:from:from:to:cc:subject:date :message-id:reply-to; bh=VAJKsDdGNb+ALwQvMYs9Oz9Vq8VGrLJsA8GxB+kZmZc=; b=hTtSLAP5sAPY0TF89JKr3Rb2m5Xr6h4ihoj9/EC79sNLnuyRgZkm4qjmVV8G/3rG4E 8so3/CdL78cmmycHIETwLNKHqdlbVS22UFokPJWJPytUII3Nn3BtdsZVJHVRpZoBgfL7 a22akR0Ll/MlRBmQZFibM8Aw7lsgF5BQbb4n2CS6QwKAG2RHk5IneY7Im1bdIUanhrGX Ndvw+mUpt/NHZV1Nkw/CbqasqywKqC29jcGP57q7FlJGMbxGBCAse54oMiYST5PL+zbD Dkdr+oA7FiEUeYce51l0yaMn98ow0mLH4wvphuDoTaoNBHQlBy+YXAY6taBBi9BkcbHx hYrQ== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20230601; t=1734947162; x=1735551962; h=content-transfer-encoding:mime-version:references:in-reply-to :message-id:date:subject:cc:to:from:x-gm-message-state:from:to:cc :subject:date:message-id:reply-to; bh=VAJKsDdGNb+ALwQvMYs9Oz9Vq8VGrLJsA8GxB+kZmZc=; b=k8Q26GZFrD9AfOLx5w+alZhTUELfuGQ4GxzysXE1ANSM77PdOfFdjICc8P8ftGGhRT A9tmath4ACX4k5I7ZZEts6BbI145MvRAUxjVot2x/4j+lOw7mJ9tiLsWUmOA92VbdRGm SjUUkDuOtf69pnG2gbdv+C1Txs1JAxO1V0pdH3bbKOGplWekq3WKIEYAmR0B05Yz0H2N J8JQQiknGDSXHDvutAbMr2ZCLjGTt2gXCw8WO6GpT4wsMh2naP3EqK0sS7FoMbTTLUu9 f8cmCH/4up/9K5EGZAR00YcGYAtB5p/yRUPxAilCF1va8mQXuN8IoUU5xJ9uBBDcbOFz TiwQ== X-Forwarded-Encrypted: i=1; AJvYcCXaLFIG6BvoYiLMbvNg5LUz7U7Ds7AN9pZ3KnxmhMkAldnXWGOB2xeFRmLcm35gdnPSRd4r0UGalOjdQQ==@lists.infradead.org X-Gm-Message-State: AOJu0YylpGgFnXHReqWflKNVcNw80ThL3Ghxp2UAhl/S2itxmXA/AkLn /z6hb1rwaWFrLWoxbtG4qCDCOcs7VUcHuuUyyWPNJ16Nw9z75JuEA0dYalMqjCQ= X-Gm-Gg: ASbGnctKOGAmoBkjNwV7tlOc74mUSismS9onvqqcg26pWZAGbC5ndJ0DNPJZaZZG82W 1nSApXF/0ECNjfmkUOFP4M/2w7XRMV/0bhGQ5do0QksoCE7xb9pW775UahMBTdtExR/T1sNLJ88 NuiJDT4L6ZwDzHC/Btg+H+6g/F7VGaig+EhHHcEhi1E6zP7MayqjwtoyMS0kJ+o7f721lLNs1zJ fa7HHhMiNoeuHQVDtS/0mzmbqs5Xsz1/Ju23UtiUrEC1L8UGJfwyKEyMxOgVhmD/tAyfWMMMIqJ eBdKrNv0zrtQ/go4F57BNg== X-Google-Smtp-Source: AGHT+IEiKQiNoCztX+vEmXJ78+SVJHcoRjQcReSYKDpgexw8hG6RQW+fGx0o5GN+OmUz8JwpFgmgkg== X-Received: by 2002:a05:6a20:c88f:b0:1d5:10d6:92b9 with SMTP id adf61e73a8af0-1e5e07f88demr18787949637.30.1734947161832; Mon, 23 Dec 2024 01:46:01 -0800 (PST) Received: from C02DW0BEMD6R.bytedance.net ([203.208.167.150]) by smtp.gmail.com with ESMTPSA id d2e1a72fcca58-72aad8486c6sm7468309b3a.85.2024.12.23.01.45.49 (version=TLS1_3 cipher=TLS_AES_256_GCM_SHA384 bits=256/256); Mon, 23 Dec 2024 01:46:01 -0800 (PST) From: Qi Zheng To: peterz@infradead.org, agordeev@linux.ibm.com, kevin.brodsky@arm.com, tglx@linutronix.de, david@redhat.com, jannh@google.com, hughd@google.com, yuzhao@google.com, willy@infradead.org, muchun.song@linux.dev, vbabka@kernel.org, lorenzo.stoakes@oracle.com, akpm@linux-foundation.org, rientjes@google.com, vishal.moola@gmail.com, arnd@arndb.de, will@kernel.org, aneesh.kumar@kernel.org, npiggin@gmail.com, dave.hansen@linux.intel.com, rppt@kernel.org, ryan.roberts@arm.com Cc: linux-mm@kvack.org, linux-arm-kernel@lists.infradead.org, linuxppc-dev@lists.ozlabs.org, linux-riscv@lists.infradead.org, linux-s390@vger.kernel.org, sparclinux@vger.kernel.org, linux-kernel@vger.kernel.org, x86@kernel.org, linux-arch@vger.kernel.org, linux-csky@vger.kernel.org, linux-hexagon@vger.kernel.org, loongarch@lists.linux.dev, linux-m68k@lists.linux-m68k.org, linux-mips@vger.kernel.org, linux-openrisc@vger.kernel.org, linux-sh@vger.kernel.org, linux-um@lists.infradead.org, Qi Zheng Subject: [PATCH v3 14/17] mm: pgtable: move __tlb_remove_table_one() in x86 to generic file Date: Mon, 23 Dec 2024 17:41:00 +0800 Message-Id: <286e9777dd266dc610de20120fae453b84d3a868.1734945104.git.zhengqi.arch@bytedance.com> X-Mailer: git-send-email 2.24.3 (Apple Git-128) In-Reply-To: References: MIME-Version: 1.0 X-CRM114-Version: 20100106-BlameMichelson ( TRE 0.8.0 (BSD) ) MR-646709E3 X-CRM114-CacheID: sfid-20241223_014602_487226_34C799BC X-CRM114-Status: UNSURE ( 8.24 ) X-CRM114-Notice: Please train this message. X-BeenThere: linux-riscv@lists.infradead.org X-Mailman-Version: 2.1.34 Precedence: list List-Id: List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Sender: "linux-riscv" Errors-To: linux-riscv-bounces+linux-riscv=archiver.kernel.org@lists.infradead.org The __tlb_remove_table_one() in x86 does not contain architecture-specific content, so move it to the generic file. Signed-off-by: Qi Zheng --- arch/x86/include/asm/tlb.h | 19 ------------------- mm/mmu_gather.c | 20 ++++++++++++++++++-- 2 files changed, 18 insertions(+), 21 deletions(-) diff --git a/arch/x86/include/asm/tlb.h b/arch/x86/include/asm/tlb.h index 3858dbf75880e..77f52bc1578a7 100644 --- a/arch/x86/include/asm/tlb.h +++ b/arch/x86/include/asm/tlb.h @@ -20,25 +20,6 @@ static inline void tlb_flush(struct mmu_gather *tlb) flush_tlb_mm_range(tlb->mm, start, end, stride_shift, tlb->freed_tables); } -#ifdef CONFIG_PT_RECLAIM -static inline void __tlb_remove_table_one_rcu(struct rcu_head *head) -{ - struct ptdesc *ptdesc; - - ptdesc = container_of(head, struct ptdesc, pt_rcu_head); - __tlb_remove_table(ptdesc); -} - -static inline void __tlb_remove_table_one(void *table) -{ - struct ptdesc *ptdesc; - - ptdesc = table; - call_rcu(&ptdesc->pt_rcu_head, __tlb_remove_table_one_rcu); -} -#define __tlb_remove_table_one __tlb_remove_table_one -#endif /* CONFIG_PT_RECLAIM */ - static inline void invlpg(unsigned long addr) { asm volatile("invlpg (%0)" ::"r" (addr) : "memory"); diff --git a/mm/mmu_gather.c b/mm/mmu_gather.c index 1e21022bcf339..7aa6f18c500b2 100644 --- a/mm/mmu_gather.c +++ b/mm/mmu_gather.c @@ -311,13 +311,29 @@ static inline void tlb_table_invalidate(struct mmu_gather *tlb) } } -#ifndef __tlb_remove_table_one +#ifdef CONFIG_PT_RECLAIM +static inline void __tlb_remove_table_one_rcu(struct rcu_head *head) +{ + struct ptdesc *ptdesc; + + ptdesc = container_of(head, struct ptdesc, pt_rcu_head); + __tlb_remove_table(ptdesc); +} + +static inline void __tlb_remove_table_one(void *table) +{ + struct ptdesc *ptdesc; + + ptdesc = table; + call_rcu(&ptdesc->pt_rcu_head, __tlb_remove_table_one_rcu); +} +#else static inline void __tlb_remove_table_one(void *table) { tlb_remove_table_sync_one(); __tlb_remove_table(table); } -#endif +#endif /* CONFIG_PT_RECLAIM */ static void tlb_remove_table_one(void *table) { From patchwork Mon Dec 23 09:41:01 2024 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Qi Zheng X-Patchwork-Id: 13918726 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from bombadil.infradead.org (bombadil.infradead.org [198.137.202.133]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (No client certificate requested) by smtp.lore.kernel.org (Postfix) with ESMTPS id A98A9E7718B for ; Mon, 23 Dec 2024 09:59:24 +0000 (UTC) DKIM-Signature: v=1; a=rsa-sha256; q=dns/txt; c=relaxed/relaxed; d=lists.infradead.org; s=bombadil.20210309; h=Sender: Content-Transfer-Encoding:Content-Type:List-Subscribe:List-Help:List-Post: List-Archive:List-Unsubscribe:List-Id:MIME-Version:References:In-Reply-To: Message-Id:Date:Subject:Cc:To:From:Reply-To:Content-ID:Content-Description: Resent-Date:Resent-From:Resent-Sender:Resent-To:Resent-Cc:Resent-Message-ID: List-Owner; bh=nDwBXTjT7x9ABEJCekyxy8CDN32u8Lc0/gjIjEBqoB4=; b=P7+hShAPOMRUOX 3VvmCyQRd8Kb5wz/8ur08uWf/GB4BXzfSUMGyKVur5Yaqf7SLYo+58FTh1+HUwyrw2X4Bi9sA3zLs 1ICECE5OC3o+AcHmq+y87eDNBGtRhVhyjkBbq/CjoB2dIR0nGGFPz2q8NKSoC09TLiLp/5w5T7dwk wqyJA7PbqAxM8Tij+UFZ9SMQ2+jjP/3/V3IdyLOOpKdViA9js/KaU9TybLM3cuSeYzjluM76QrrgM /hSR9NtJp5OwbkghEcicrqjiqrMwtKdSQ7elinvMR4oDCbzgSAVccDqHp8xGOs2Ry5u97RoheJAKw LNN58LK9kNNmwEhRu1IA==; Received: from localhost ([::1] helo=bombadil.infradead.org) by bombadil.infradead.org with esmtp (Exim 4.98 #2 (Red Hat Linux)) id 1tPfDc-00000009k5A-1QL8; Mon, 23 Dec 2024 09:59:20 +0000 Received: from mail-pf1-x431.google.com ([2607:f8b0:4864:20::431]) by bombadil.infradead.org with esmtps (Exim 4.98 #2 (Red Hat Linux)) id 1tPf0x-00000009hdO-2xlH for linux-riscv@lists.infradead.org; Mon, 23 Dec 2024 09:46:18 +0000 Received: by mail-pf1-x431.google.com with SMTP id d2e1a72fcca58-728ea1573c0so3238674b3a.0 for ; Mon, 23 Dec 2024 01:46:15 -0800 (PST) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=bytedance.com; s=google; t=1734947175; x=1735551975; darn=lists.infradead.org; h=content-transfer-encoding:mime-version:references:in-reply-to :message-id:date:subject:cc:to:from:from:to:cc:subject:date :message-id:reply-to; bh=FItpERzyh1wDMphf2PSqriZlspFkmpQlPhA2jGTCtDU=; b=cvhQTvk+P6/NrNoWUSbOzNZIhKg5lC3LAk3e1SlQwYqW27tVT93QSBbWONXTjl/S3l Pmdty/KHleD+fykgGnpBuScEX9bg6bCqw0PSbHVO5dakxmDnNnSznHxdVY9dh5o7z9gJ JfFWV58m5v2FQfSbWndt3HjieB3Rf0Upwk6aXq9Ra78HeZNSUZ3lDNdHsyzYGZZOOoOy Yu+brNQzgc09Kp6RrskAtuQWOnWHXyyDprlPHx+AV6OmTtL7s95IOCznyOfdc5wix9gv 3q0eEGgVNiYeemTNDbSrMiID0X07zy6axYEesuJkdzGXQjs+livqD3OnBZWQkNM9Fgy4 2xUA== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20230601; t=1734947175; x=1735551975; h=content-transfer-encoding:mime-version:references:in-reply-to :message-id:date:subject:cc:to:from:x-gm-message-state:from:to:cc :subject:date:message-id:reply-to; bh=FItpERzyh1wDMphf2PSqriZlspFkmpQlPhA2jGTCtDU=; b=K2g2Lxsn0+UWLH521nzNHbMpTKTk9jF1Ed5gnjxVTKsgYOoo7GpOSeQzD4YZBCBFNV RP9sck3XiHwv4dN/WEprnwTV8fl3LzTU5sfSnCmCMQRUxhvVegBdGnLQQ2BXJkJEZCET ngVllK+oBsA3o0tKuO/v7G2mm2ipe5uFUDfOraClsQKJB9Ec6VvuaNX+pADrseT4tSDc YUAzFVkEIotfl/+P2IeZ8R2V/2WSTw8Qwp3JyVLg+0s4iKCJwKXxyRHXXbxb4aqzDfJZ y+/AMjRVLNKD+Mgq/hYwtl9/6VZpILLZFhWaH8EJlzVRYbR9pA8+acwLx7YUwRPfzxeV 4dxQ== X-Forwarded-Encrypted: i=1; AJvYcCXd7fqLSS6xI7wgACO0cuKVoO1kIZvn0lwJH+5bRD/sq0J2VD1a5u4o26MK/T2J6WJQHdQrDmSX8P8Cyg==@lists.infradead.org X-Gm-Message-State: AOJu0YzyfNh6xcMqK4Gp9ByuRnUdc9jSSInkq6UeOp21LpHfX1hPQoj8 u2Fp/KQ7OhJnEhKITBjfYCm9FGYRtSPIkEWuSZoEP/9OWqfEya6av60fhv3tDvQ= X-Gm-Gg: ASbGncu4IiFZEUQxAXRPc1ZrFbZc5eP3s63vhQx99J1W8cahJ704h+y8Krwy+Z5R2B3 1FqewUHkA5U9/+9mkHppvk78NTauFodVA08eR6jdsenHs4WS8QGbgp8oBmUuv6Fu0LnNe4dMtOV I4RrLHkQCQeljVUBwSMYO4437Q0F6TlJFlcbOfMWjBAywDpnq1jD5XeCLmbyycVV46lYkNJ1Cc8 6AxmdD+Zz/823Db8f0O45WdK3DtH52o1ewHBUsom5WBEbj5bSAbvAFhtg/w+OXAjTnuhKv61Wfc t21CADMa3pLKdHuRfnRFKA== X-Google-Smtp-Source: AGHT+IF9RJvbCaq38Te31r1RKGqLS5POHI3EPqaSbNpVA67C9keEJqX0jQ2rnqy+EikMhfv/vLwTVQ== X-Received: by 2002:a05:6a20:12d2:b0:1e0:c0fa:e088 with SMTP id adf61e73a8af0-1e5e059b19cmr20289317637.18.1734947175103; Mon, 23 Dec 2024 01:46:15 -0800 (PST) Received: from C02DW0BEMD6R.bytedance.net ([203.208.167.150]) by smtp.gmail.com with ESMTPSA id d2e1a72fcca58-72aad8486c6sm7468309b3a.85.2024.12.23.01.46.02 (version=TLS1_3 cipher=TLS_AES_256_GCM_SHA384 bits=256/256); Mon, 23 Dec 2024 01:46:14 -0800 (PST) From: Qi Zheng To: peterz@infradead.org, agordeev@linux.ibm.com, kevin.brodsky@arm.com, tglx@linutronix.de, david@redhat.com, jannh@google.com, hughd@google.com, yuzhao@google.com, willy@infradead.org, muchun.song@linux.dev, vbabka@kernel.org, lorenzo.stoakes@oracle.com, akpm@linux-foundation.org, rientjes@google.com, vishal.moola@gmail.com, arnd@arndb.de, will@kernel.org, aneesh.kumar@kernel.org, npiggin@gmail.com, dave.hansen@linux.intel.com, rppt@kernel.org, ryan.roberts@arm.com Cc: linux-mm@kvack.org, linux-arm-kernel@lists.infradead.org, linuxppc-dev@lists.ozlabs.org, linux-riscv@lists.infradead.org, linux-s390@vger.kernel.org, sparclinux@vger.kernel.org, linux-kernel@vger.kernel.org, x86@kernel.org, linux-arch@vger.kernel.org, linux-csky@vger.kernel.org, linux-hexagon@vger.kernel.org, loongarch@lists.linux.dev, linux-m68k@lists.linux-m68k.org, linux-mips@vger.kernel.org, linux-openrisc@vger.kernel.org, linux-sh@vger.kernel.org, linux-um@lists.infradead.org, Qi Zheng Subject: [PATCH v3 15/17] mm: pgtable: remove tlb_remove_page_ptdesc() Date: Mon, 23 Dec 2024 17:41:01 +0800 Message-Id: X-Mailer: git-send-email 2.24.3 (Apple Git-128) In-Reply-To: References: MIME-Version: 1.0 X-CRM114-Version: 20100106-BlameMichelson ( TRE 0.8.0 (BSD) ) MR-646709E3 X-CRM114-CacheID: sfid-20241223_014615_783598_B78066FC X-CRM114-Status: UNSURE ( 9.88 ) X-CRM114-Notice: Please train this message. X-BeenThere: linux-riscv@lists.infradead.org X-Mailman-Version: 2.1.34 Precedence: list List-Id: List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Sender: "linux-riscv" Errors-To: linux-riscv-bounces+linux-riscv=archiver.kernel.org@lists.infradead.org Here we are explicitly dealing with struct page, and the following logic semms strange: tlb_remove_page_ptdesc((tlb), (page_ptdesc(pte))); tlb_remove_page_ptdesc --> tlb_remove_page(tlb, ptdesc_page(pt)); So remove tlb_remove_page_ptdesc() and make callers call tlb_remove_page() directly. Signed-off-by: Qi Zheng Originally-by: Peter Zijlstra (Intel) --- arch/csky/include/asm/pgalloc.h | 2 +- arch/hexagon/include/asm/pgalloc.h | 2 +- arch/loongarch/include/asm/pgalloc.h | 2 +- arch/m68k/include/asm/sun3_pgalloc.h | 2 +- arch/mips/include/asm/pgalloc.h | 2 +- arch/nios2/include/asm/pgalloc.h | 2 +- arch/openrisc/include/asm/pgalloc.h | 2 +- arch/riscv/include/asm/pgalloc.h | 2 +- arch/sh/include/asm/pgalloc.h | 2 +- arch/um/include/asm/pgalloc.h | 8 ++++---- include/asm-generic/tlb.h | 6 ------ 11 files changed, 13 insertions(+), 19 deletions(-) diff --git a/arch/csky/include/asm/pgalloc.h b/arch/csky/include/asm/pgalloc.h index f1ce5b7b28f22..936a43a49e704 100644 --- a/arch/csky/include/asm/pgalloc.h +++ b/arch/csky/include/asm/pgalloc.h @@ -64,7 +64,7 @@ static inline pgd_t *pgd_alloc(struct mm_struct *mm) #define __pte_free_tlb(tlb, pte, address) \ do { \ pagetable_dtor(page_ptdesc(pte)); \ - tlb_remove_page_ptdesc(tlb, page_ptdesc(pte)); \ + tlb_remove_page(tlb, (pte)); \ } while (0) extern void pagetable_init(void); diff --git a/arch/hexagon/include/asm/pgalloc.h b/arch/hexagon/include/asm/pgalloc.h index 40e42a0e71673..8b1550498f1bf 100644 --- a/arch/hexagon/include/asm/pgalloc.h +++ b/arch/hexagon/include/asm/pgalloc.h @@ -90,7 +90,7 @@ static inline void pmd_populate_kernel(struct mm_struct *mm, pmd_t *pmd, #define __pte_free_tlb(tlb, pte, addr) \ do { \ pagetable_dtor((page_ptdesc(pte))); \ - tlb_remove_page_ptdesc((tlb), (page_ptdesc(pte))); \ + tlb_remove_page((tlb), (pte)); \ } while (0) #endif diff --git a/arch/loongarch/include/asm/pgalloc.h b/arch/loongarch/include/asm/pgalloc.h index 7211dff8c969e..5a4f22aeb6189 100644 --- a/arch/loongarch/include/asm/pgalloc.h +++ b/arch/loongarch/include/asm/pgalloc.h @@ -58,7 +58,7 @@ static inline pte_t *pte_alloc_one_kernel(struct mm_struct *mm) #define __pte_free_tlb(tlb, pte, address) \ do { \ pagetable_dtor(page_ptdesc(pte)); \ - tlb_remove_page_ptdesc((tlb), page_ptdesc(pte)); \ + tlb_remove_page((tlb), (pte)); \ } while (0) #ifndef __PAGETABLE_PMD_FOLDED diff --git a/arch/m68k/include/asm/sun3_pgalloc.h b/arch/m68k/include/asm/sun3_pgalloc.h index 2b626cb3ad0ae..63d9f95f5e3dd 100644 --- a/arch/m68k/include/asm/sun3_pgalloc.h +++ b/arch/m68k/include/asm/sun3_pgalloc.h @@ -20,7 +20,7 @@ extern const char bad_pmd_string[]; #define __pte_free_tlb(tlb, pte, addr) \ do { \ pagetable_dtor(page_ptdesc(pte)); \ - tlb_remove_page_ptdesc((tlb), page_ptdesc(pte)); \ + tlb_remove_page((tlb), (pte)); \ } while (0) static inline void pmd_populate_kernel(struct mm_struct *mm, pmd_t *pmd, pte_t *pte) diff --git a/arch/mips/include/asm/pgalloc.h b/arch/mips/include/asm/pgalloc.h index 36d9805033c4b..bbee21345154b 100644 --- a/arch/mips/include/asm/pgalloc.h +++ b/arch/mips/include/asm/pgalloc.h @@ -57,7 +57,7 @@ static inline void pgd_free(struct mm_struct *mm, pgd_t *pgd) #define __pte_free_tlb(tlb, pte, address) \ do { \ pagetable_dtor(page_ptdesc(pte)); \ - tlb_remove_page_ptdesc((tlb), page_ptdesc(pte)); \ + tlb_remove_page((tlb), (pte)); \ } while (0) #ifndef __PAGETABLE_PMD_FOLDED diff --git a/arch/nios2/include/asm/pgalloc.h b/arch/nios2/include/asm/pgalloc.h index 12a536b7bfbd4..641cec8fb2a22 100644 --- a/arch/nios2/include/asm/pgalloc.h +++ b/arch/nios2/include/asm/pgalloc.h @@ -31,7 +31,7 @@ extern pgd_t *pgd_alloc(struct mm_struct *mm); #define __pte_free_tlb(tlb, pte, addr) \ do { \ pagetable_dtor(page_ptdesc(pte)); \ - tlb_remove_page_ptdesc((tlb), (page_ptdesc(pte))); \ + tlb_remove_page((tlb), (pte)); \ } while (0) #endif /* _ASM_NIOS2_PGALLOC_H */ diff --git a/arch/openrisc/include/asm/pgalloc.h b/arch/openrisc/include/asm/pgalloc.h index 596e2355824e3..e9b9bc53ece0b 100644 --- a/arch/openrisc/include/asm/pgalloc.h +++ b/arch/openrisc/include/asm/pgalloc.h @@ -69,7 +69,7 @@ extern pte_t *pte_alloc_one_kernel(struct mm_struct *mm); #define __pte_free_tlb(tlb, pte, addr) \ do { \ pagetable_dtor(page_ptdesc(pte)); \ - tlb_remove_page_ptdesc((tlb), (page_ptdesc(pte))); \ + tlb_remove_page((tlb), (pte)); \ } while (0) #endif diff --git a/arch/riscv/include/asm/pgalloc.h b/arch/riscv/include/asm/pgalloc.h index c8907b8317115..ab4f9b2cf9e11 100644 --- a/arch/riscv/include/asm/pgalloc.h +++ b/arch/riscv/include/asm/pgalloc.h @@ -29,7 +29,7 @@ static inline void riscv_tlb_remove_ptdesc(struct mmu_gather *tlb, void *pt) tlb_remove_ptdesc(tlb, pt); } else { pagetable_dtor(pt); - tlb_remove_page_ptdesc(tlb, pt); + tlb_remove_page(tlb, ptdesc_page((struct ptdesc *)pt)); } } diff --git a/arch/sh/include/asm/pgalloc.h b/arch/sh/include/asm/pgalloc.h index 96d938fdf2244..43812b2363efd 100644 --- a/arch/sh/include/asm/pgalloc.h +++ b/arch/sh/include/asm/pgalloc.h @@ -35,7 +35,7 @@ static inline void pmd_populate(struct mm_struct *mm, pmd_t *pmd, #define __pte_free_tlb(tlb, pte, addr) \ do { \ pagetable_dtor(page_ptdesc(pte)); \ - tlb_remove_page_ptdesc((tlb), (page_ptdesc(pte))); \ + tlb_remove_page((tlb), (pte)); \ } while (0) #endif /* __ASM_SH_PGALLOC_H */ diff --git a/arch/um/include/asm/pgalloc.h b/arch/um/include/asm/pgalloc.h index f0af23c3aeb2b..98190c318a8e9 100644 --- a/arch/um/include/asm/pgalloc.h +++ b/arch/um/include/asm/pgalloc.h @@ -28,7 +28,7 @@ extern pgd_t *pgd_alloc(struct mm_struct *); #define __pte_free_tlb(tlb, pte, address) \ do { \ pagetable_dtor(page_ptdesc(pte)); \ - tlb_remove_page_ptdesc((tlb), (page_ptdesc(pte))); \ + tlb_remove_page((tlb), (pte)); \ } while (0) #if CONFIG_PGTABLE_LEVELS > 2 @@ -36,15 +36,15 @@ do { \ #define __pmd_free_tlb(tlb, pmd, address) \ do { \ pagetable_dtor(virt_to_ptdesc(pmd)); \ - tlb_remove_page_ptdesc((tlb), virt_to_ptdesc(pmd)); \ + tlb_remove_page((tlb), virt_to_page(pmd)); \ } while (0) #if CONFIG_PGTABLE_LEVELS > 3 #define __pud_free_tlb(tlb, pud, address) \ do { \ - pagetable_dtor(virt_to_ptdesc(pud)); \ - tlb_remove_page_ptdesc((tlb), virt_to_ptdesc(pud)); \ + pagetable_dtor(virt_to_ptdesc(pud)); \ + tlb_remove_page((tlb), virt_to_page(pud)); \ } while (0) #endif diff --git a/include/asm-generic/tlb.h b/include/asm-generic/tlb.h index 69de47c7ef3c5..8d6cfe5058543 100644 --- a/include/asm-generic/tlb.h +++ b/include/asm-generic/tlb.h @@ -504,12 +504,6 @@ static inline void tlb_remove_ptdesc(struct mmu_gather *tlb, void *pt) tlb_remove_table(tlb, pt); } -/* Like tlb_remove_ptdesc, but for page-like page directories. */ -static inline void tlb_remove_page_ptdesc(struct mmu_gather *tlb, struct ptdesc *pt) -{ - tlb_remove_page(tlb, ptdesc_page(pt)); -} - static inline void tlb_change_page_size(struct mmu_gather *tlb, unsigned int page_size) { From patchwork Mon Dec 23 09:41:02 2024 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Qi Zheng X-Patchwork-Id: 13918728 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from bombadil.infradead.org (bombadil.infradead.org [198.137.202.133]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (No client certificate requested) by smtp.lore.kernel.org (Postfix) with ESMTPS id 1366AE7718B for ; Mon, 23 Dec 2024 10:00:52 +0000 (UTC) DKIM-Signature: v=1; a=rsa-sha256; q=dns/txt; c=relaxed/relaxed; d=lists.infradead.org; s=bombadil.20210309; h=Sender: Content-Transfer-Encoding:Content-Type:List-Subscribe:List-Help:List-Post: List-Archive:List-Unsubscribe:List-Id:MIME-Version:References:In-Reply-To: Message-Id:Date:Subject:Cc:To:From:Reply-To:Content-ID:Content-Description: Resent-Date:Resent-From:Resent-Sender:Resent-To:Resent-Cc:Resent-Message-ID: List-Owner; bh=hYq7kNzOVoPjwgNwsFLXNtBJmbtyBPArAIk/Q6e+Bd0=; b=0zlOLE+XKoD6XR Mq++B+OQQF7fSr6RcPo/JhiTn00gT4+3ZmNp1T4adEk1Q5Qqv89SD+Dpr84xbBZLLwkJPfvfUF7gk MCmxQPT4sGR41wEXv1ejFP3lHKZYfmN8UM9Nogua7cZoishF18RSklkdJ/bKg1nuz19FbXH4w7hs0 7PEOAEwxyqdMG/bKv3B8YbOS5xwdzMs6FF/k56CLqZjD138jTRzbWE/Tv5hIz4cMAOIGVqvv+s21a wTOT5C/Cw+9POcuVXFKkyQmRrPFuD+9nlWr9toqNKc8jB2To3yOFqDpkDWZG0EBd3p5j5Dpc9s8bs plyMiRtNw19MP7goIq2Q==; Received: from localhost ([::1] helo=bombadil.infradead.org) by bombadil.infradead.org with esmtp (Exim 4.98 #2 (Red Hat Linux)) id 1tPfF0-00000009kKb-0m8l; Mon, 23 Dec 2024 10:00:46 +0000 Received: from mail-pf1-x429.google.com ([2607:f8b0:4864:20::429]) by bombadil.infradead.org with esmtps (Exim 4.98 #2 (Red Hat Linux)) id 1tPf1A-00000009hjj-1ktu for linux-riscv@lists.infradead.org; Mon, 23 Dec 2024 09:46:30 +0000 Received: by mail-pf1-x429.google.com with SMTP id d2e1a72fcca58-728e3826211so3072137b3a.0 for ; Mon, 23 Dec 2024 01:46:28 -0800 (PST) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=bytedance.com; s=google; t=1734947188; x=1735551988; darn=lists.infradead.org; h=content-transfer-encoding:mime-version:references:in-reply-to :message-id:date:subject:cc:to:from:from:to:cc:subject:date :message-id:reply-to; bh=6vzAxma0H0oGKVF+0yjfVv8ni44DJQgUTcLWSpNFGuY=; b=Uh+Za0YsDnaqLpQVwGT483cvqnHOKZB56ya0wu2izglNI90Jl3Ije75oaiu+0xn/x1 JmTtX2vbZ3vLYuy2DZ6G9AyXxmbiMwhRKZ8mcp3IFKGAAWIuM/FZjPUjiE8h/KJE0JDF hL+s9N+dOWmPvHm4XFwrBCUmei8wMIyIX0HuZaHzVe+Oih7yQMGmGZ074OnHe+bbrPJK MTHHCMWBkTYnncm6CgHL45Qes9RwTh3Jajma5c7OjXaqwm9/J7NW4ppEa2r9OwIuFEVI wnNal6o41hb8NbkzgjbR4jBMmqbyHCwhpFrMO2w3FBoCQuPuESSzxSh1FDyayoFhnmCR i6YA== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20230601; t=1734947188; x=1735551988; h=content-transfer-encoding:mime-version:references:in-reply-to :message-id:date:subject:cc:to:from:x-gm-message-state:from:to:cc :subject:date:message-id:reply-to; bh=6vzAxma0H0oGKVF+0yjfVv8ni44DJQgUTcLWSpNFGuY=; b=E2zEYZ6GDSv2lsesgn49SInumYUfyxwB3O/DFNPMTnIt6IASKCUrxmGkTe224JX8cs 5Mos3pAVHj18PkofIkdzzO0EB0aV8KR8Odrys2zxHepU/eW0tVe51ST46DO5KU6nE0AC 1BW3bK7SkLYw5uk3OZjGWUs8rrdltmBTRA0NfMqtdiWB5YBdwA5MBJWA2d/Y3ZO07UIQ WX2Bqkv31/j5aOkCKNYpcW5NMb2PZsW/cnzLBIzAHK2fsZSI2htCc+FGszaLNwcvYMmv WWjO4mk6GA9MQN0WDrno4YJu/ebcXkne+BwYE4k7NypxNPeQ3OtiLBgSgE6oDBMT7qcq P0Yg== X-Forwarded-Encrypted: i=1; AJvYcCU5WVVkRqn/ap0Bv9CqRE4p0cbqJ+fF+X3EGvrcS1uxbynKh6L8b4Va+W73yX/GwZrITGyERB4h6RLruw==@lists.infradead.org X-Gm-Message-State: AOJu0YxHbAImElzdfoFj3TU40jJeqBunwFTMyva5wurDwRIsM6RX3ksR 2cPaSfCVfR7eEj/mT6c4QNJbsTGQ9OtQBHzDiyFiA/72UiBiTQYlWEo8hefuBIA= X-Gm-Gg: ASbGncs5Rwcoe5YJHh7NrVKYXBV8SeugPQvKlwDSNXACyB2auDyo1cDAGTPDYhZw2pJ rKl3zgajjigDuxI8gHPopdnLA4P0ibLoJAwDSphrfCM8WzOEN34D2QWdhXkpILivg5jhZ0J56xH 9Km6drNMzpipo3lHPGllykiupBydhJLhV4vy7h/oCqJ+Vt0RLmwskggy1SFXaRzDydzQH4l42Ng tpE37Lhdvhy601Q06tylh0GmKGA019cMYQdh4qqZbrkrSg2AT9SFZnaDnYLcGTIO4svz2ORdu4K 7qe2pjxW/6ZKtHTQBOH/dg== X-Google-Smtp-Source: AGHT+IGqxRLXXsgzbTBUt4P8wMbb/CAKHFN0NH0iCzAzNT/ELUdunTp3cz3Ieuj7L8l/gUjxiL3RMw== X-Received: by 2002:aa7:888c:0:b0:72a:a9b5:ed91 with SMTP id d2e1a72fcca58-72abde0e6b7mr17804649b3a.13.1734947187839; Mon, 23 Dec 2024 01:46:27 -0800 (PST) Received: from C02DW0BEMD6R.bytedance.net ([203.208.167.150]) by smtp.gmail.com with ESMTPSA id d2e1a72fcca58-72aad8486c6sm7468309b3a.85.2024.12.23.01.46.15 (version=TLS1_3 cipher=TLS_AES_256_GCM_SHA384 bits=256/256); Mon, 23 Dec 2024 01:46:27 -0800 (PST) From: Qi Zheng To: peterz@infradead.org, agordeev@linux.ibm.com, kevin.brodsky@arm.com, tglx@linutronix.de, david@redhat.com, jannh@google.com, hughd@google.com, yuzhao@google.com, willy@infradead.org, muchun.song@linux.dev, vbabka@kernel.org, lorenzo.stoakes@oracle.com, akpm@linux-foundation.org, rientjes@google.com, vishal.moola@gmail.com, arnd@arndb.de, will@kernel.org, aneesh.kumar@kernel.org, npiggin@gmail.com, dave.hansen@linux.intel.com, rppt@kernel.org, ryan.roberts@arm.com Cc: linux-mm@kvack.org, linux-arm-kernel@lists.infradead.org, linuxppc-dev@lists.ozlabs.org, linux-riscv@lists.infradead.org, linux-s390@vger.kernel.org, sparclinux@vger.kernel.org, linux-kernel@vger.kernel.org, x86@kernel.org, linux-arch@vger.kernel.org, linux-csky@vger.kernel.org, linux-hexagon@vger.kernel.org, loongarch@lists.linux.dev, linux-m68k@lists.linux-m68k.org, linux-mips@vger.kernel.org, linux-openrisc@vger.kernel.org, linux-sh@vger.kernel.org, linux-um@lists.infradead.org, Qi Zheng Subject: [PATCH v3 16/17] mm: pgtable: remove tlb_remove_ptdesc() Date: Mon, 23 Dec 2024 17:41:02 +0800 Message-Id: <93cce93bf8be04f3a5cd828cc0a48750fb90af44.1734945104.git.zhengqi.arch@bytedance.com> X-Mailer: git-send-email 2.24.3 (Apple Git-128) In-Reply-To: References: MIME-Version: 1.0 X-CRM114-Version: 20100106-BlameMichelson ( TRE 0.8.0 (BSD) ) MR-646709E3 X-CRM114-CacheID: sfid-20241223_014628_659442_96E0B753 X-CRM114-Status: GOOD ( 11.15 ) X-BeenThere: linux-riscv@lists.infradead.org X-Mailman-Version: 2.1.34 Precedence: list List-Id: List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Sender: "linux-riscv" Errors-To: linux-riscv-bounces+linux-riscv=archiver.kernel.org@lists.infradead.org Just like removing tlb_remove_page_ptdesc(), remove tlb_remove_ptdesc() as well, and make callers call tlb_remove_table() directly. Signed-off-by: Qi Zheng Originally-by: Peter Zijlstra (Intel) --- arch/arm/include/asm/tlb.h | 8 ++------ arch/arm64/include/asm/tlb.h | 16 ++++------------ arch/riscv/include/asm/pgalloc.h | 14 +++++++------- arch/s390/include/asm/tlb.h | 8 ++++---- include/asm-generic/tlb.h | 7 +------ mm/mmu_gather.c | 11 +++++------ 6 files changed, 23 insertions(+), 41 deletions(-) diff --git a/arch/arm/include/asm/tlb.h b/arch/arm/include/asm/tlb.h index ea4fbe7b17f6f..ac3881ec342f1 100644 --- a/arch/arm/include/asm/tlb.h +++ b/arch/arm/include/asm/tlb.h @@ -32,8 +32,6 @@ static inline void __pte_free_tlb(struct mmu_gather *tlb, pgtable_t pte, unsigned long addr) { - struct ptdesc *ptdesc = page_ptdesc(pte); - #ifndef CONFIG_ARM_LPAE /* * With the classic ARM MMU, a pte page has two corresponding pmd @@ -43,16 +41,14 @@ __pte_free_tlb(struct mmu_gather *tlb, pgtable_t pte, unsigned long addr) __tlb_adjust_range(tlb, addr - PAGE_SIZE, 2 * PAGE_SIZE); #endif - tlb_remove_ptdesc(tlb, ptdesc); + tlb_remove_table(tlb, pte); } static inline void __pmd_free_tlb(struct mmu_gather *tlb, pmd_t *pmdp, unsigned long addr) { #ifdef CONFIG_ARM_LPAE - struct ptdesc *ptdesc = virt_to_ptdesc(pmdp); - - tlb_remove_ptdesc(tlb, ptdesc); + tlb_remove_table(tlb, virt_to_page(pmdp)); #endif } diff --git a/arch/arm64/include/asm/tlb.h b/arch/arm64/include/asm/tlb.h index 8d762607285cc..4a60569fed696 100644 --- a/arch/arm64/include/asm/tlb.h +++ b/arch/arm64/include/asm/tlb.h @@ -75,18 +75,14 @@ static inline void tlb_flush(struct mmu_gather *tlb) static inline void __pte_free_tlb(struct mmu_gather *tlb, pgtable_t pte, unsigned long addr) { - struct ptdesc *ptdesc = page_ptdesc(pte); - - tlb_remove_ptdesc(tlb, ptdesc); + tlb_remove_table(tlb, pte); } #if CONFIG_PGTABLE_LEVELS > 2 static inline void __pmd_free_tlb(struct mmu_gather *tlb, pmd_t *pmdp, unsigned long addr) { - struct ptdesc *ptdesc = virt_to_ptdesc(pmdp); - - tlb_remove_ptdesc(tlb, ptdesc); + tlb_remove_table(tlb, virt_to_page(pmdp)); } #endif @@ -94,12 +90,10 @@ static inline void __pmd_free_tlb(struct mmu_gather *tlb, pmd_t *pmdp, static inline void __pud_free_tlb(struct mmu_gather *tlb, pud_t *pudp, unsigned long addr) { - struct ptdesc *ptdesc = virt_to_ptdesc(pudp); - if (!pgtable_l4_enabled()) return; - tlb_remove_ptdesc(tlb, ptdesc); + tlb_remove_table(tlb, virt_to_page(pudp)); } #endif @@ -107,12 +101,10 @@ static inline void __pud_free_tlb(struct mmu_gather *tlb, pud_t *pudp, static inline void __p4d_free_tlb(struct mmu_gather *tlb, p4d_t *p4dp, unsigned long addr) { - struct ptdesc *ptdesc = virt_to_ptdesc(p4dp); - if (!pgtable_l5_enabled()) return; - tlb_remove_ptdesc(tlb, ptdesc); + tlb_remove_table(tlb, virt_to_page(p4dp)); } #endif diff --git a/arch/riscv/include/asm/pgalloc.h b/arch/riscv/include/asm/pgalloc.h index ab4f9b2cf9e11..25c2e2f262810 100644 --- a/arch/riscv/include/asm/pgalloc.h +++ b/arch/riscv/include/asm/pgalloc.h @@ -26,10 +26,10 @@ static inline void riscv_tlb_remove_ptdesc(struct mmu_gather *tlb, void *pt) { if (riscv_use_sbi_for_rfence()) { - tlb_remove_ptdesc(tlb, pt); + tlb_remove_table(tlb, pt); } else { - pagetable_dtor(pt); - tlb_remove_page(tlb, ptdesc_page((struct ptdesc *)pt)); + pagetable_dtor(page_ptdesc((struct page *)pt)); + tlb_remove_page(tlb, pt); } } @@ -108,14 +108,14 @@ static inline void __pud_free_tlb(struct mmu_gather *tlb, pud_t *pud, unsigned long addr) { if (pgtable_l4_enabled) - riscv_tlb_remove_ptdesc(tlb, virt_to_ptdesc(pud)); + riscv_tlb_remove_ptdesc(tlb, virt_to_page(pud)); } static inline void __p4d_free_tlb(struct mmu_gather *tlb, p4d_t *p4d, unsigned long addr) { if (pgtable_l5_enabled) - riscv_tlb_remove_ptdesc(tlb, virt_to_ptdesc(p4d)); + riscv_tlb_remove_ptdesc(tlb, virt_to_page(p4d)); } #endif /* __PAGETABLE_PMD_FOLDED */ @@ -144,7 +144,7 @@ static inline pgd_t *pgd_alloc(struct mm_struct *mm) static inline void __pmd_free_tlb(struct mmu_gather *tlb, pmd_t *pmd, unsigned long addr) { - riscv_tlb_remove_ptdesc(tlb, virt_to_ptdesc(pmd)); + riscv_tlb_remove_ptdesc(tlb, virt_to_page(pmd)); } #endif /* __PAGETABLE_PMD_FOLDED */ @@ -152,7 +152,7 @@ static inline void __pmd_free_tlb(struct mmu_gather *tlb, pmd_t *pmd, static inline void __pte_free_tlb(struct mmu_gather *tlb, pgtable_t pte, unsigned long addr) { - riscv_tlb_remove_ptdesc(tlb, page_ptdesc(pte)); + riscv_tlb_remove_ptdesc(tlb, pte); } #endif /* CONFIG_MMU */ diff --git a/arch/s390/include/asm/tlb.h b/arch/s390/include/asm/tlb.h index da4a7d175f69c..5eed6300f3d72 100644 --- a/arch/s390/include/asm/tlb.h +++ b/arch/s390/include/asm/tlb.h @@ -86,7 +86,7 @@ static inline void pte_free_tlb(struct mmu_gather *tlb, pgtable_t pte, tlb->cleared_pmds = 1; if (mm_alloc_pgste(tlb->mm)) gmap_unlink(tlb->mm, (unsigned long *)pte, address); - tlb_remove_ptdesc(tlb, virt_to_ptdesc(pte)); + tlb_remove_table(tlb, virt_to_page(pte)); } /* @@ -105,7 +105,7 @@ static inline void pmd_free_tlb(struct mmu_gather *tlb, pmd_t *pmd, tlb->mm->context.flush_mm = 1; tlb->freed_tables = 1; tlb->cleared_puds = 1; - tlb_remove_ptdesc(tlb, virt_to_ptdesc(pmd)); + tlb_remove_table(tlb, virt_to_page(pmd)); } /* @@ -123,7 +123,7 @@ static inline void pud_free_tlb(struct mmu_gather *tlb, pud_t *pud, tlb->mm->context.flush_mm = 1; tlb->freed_tables = 1; tlb->cleared_p4ds = 1; - tlb_remove_ptdesc(tlb, virt_to_ptdesc(pud)); + tlb_remove_table(tlb, virt_to_page(pud)); } /* @@ -141,7 +141,7 @@ static inline void p4d_free_tlb(struct mmu_gather *tlb, p4d_t *p4d, __tlb_adjust_range(tlb, address, PAGE_SIZE); tlb->mm->context.flush_mm = 1; tlb->freed_tables = 1; - tlb_remove_ptdesc(tlb, virt_to_ptdesc(p4d)); + tlb_remove_table(tlb, virt_to_page(p4d)); } #endif /* _S390_TLB_H */ diff --git a/include/asm-generic/tlb.h b/include/asm-generic/tlb.h index 8d6cfe5058543..583e95568f52b 100644 --- a/include/asm-generic/tlb.h +++ b/include/asm-generic/tlb.h @@ -211,7 +211,7 @@ struct mmu_table_batch { #ifndef __HAVE_ARCH_TLB_REMOVE_TABLE static inline void __tlb_remove_table(void *table) { - struct ptdesc *ptdesc = (struct ptdesc *)table; + struct ptdesc *ptdesc = page_ptdesc((struct page *)table); pagetable_dtor(ptdesc); pagetable_free(ptdesc); @@ -499,11 +499,6 @@ static inline void tlb_remove_page(struct mmu_gather *tlb, struct page *page) return tlb_remove_page_size(tlb, page, PAGE_SIZE); } -static inline void tlb_remove_ptdesc(struct mmu_gather *tlb, void *pt) -{ - tlb_remove_table(tlb, pt); -} - static inline void tlb_change_page_size(struct mmu_gather *tlb, unsigned int page_size) { diff --git a/mm/mmu_gather.c b/mm/mmu_gather.c index 7aa6f18c500b2..c58ce4539c56f 100644 --- a/mm/mmu_gather.c +++ b/mm/mmu_gather.c @@ -314,18 +314,17 @@ static inline void tlb_table_invalidate(struct mmu_gather *tlb) #ifdef CONFIG_PT_RECLAIM static inline void __tlb_remove_table_one_rcu(struct rcu_head *head) { - struct ptdesc *ptdesc; + struct page *page; - ptdesc = container_of(head, struct ptdesc, pt_rcu_head); - __tlb_remove_table(ptdesc); + page = container_of(head, struct page, rcu_head); + __tlb_remove_table(page); } static inline void __tlb_remove_table_one(void *table) { - struct ptdesc *ptdesc; + struct page *page = (struct page *)table; - ptdesc = table; - call_rcu(&ptdesc->pt_rcu_head, __tlb_remove_table_one_rcu); + call_rcu(&page->rcu_head, __tlb_remove_table_one_rcu); } #else static inline void __tlb_remove_table_one(void *table) From patchwork Mon Dec 23 09:41:03 2024 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Qi Zheng X-Patchwork-Id: 13918729 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from bombadil.infradead.org (bombadil.infradead.org [198.137.202.133]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (No client certificate requested) by smtp.lore.kernel.org (Postfix) with ESMTPS id D55F8E7718B for ; Mon, 23 Dec 2024 10:02:03 +0000 (UTC) DKIM-Signature: v=1; a=rsa-sha256; q=dns/txt; c=relaxed/relaxed; d=lists.infradead.org; s=bombadil.20210309; h=Sender: Content-Transfer-Encoding:Content-Type:List-Subscribe:List-Help:List-Post: List-Archive:List-Unsubscribe:List-Id:MIME-Version:References:In-Reply-To: Message-Id:Date:Subject:Cc:To:From:Reply-To:Content-ID:Content-Description: Resent-Date:Resent-From:Resent-Sender:Resent-To:Resent-Cc:Resent-Message-ID: List-Owner; bh=3wGat9EQBcoYGkP0FZK1ZUuXT06ndFR5T/zBS6zQ7EE=; b=AErQR36PxiWj6r opZGnPIu9NGtWE3dYWCYPhh8/VCqImhCwO35KUhQyGa+rQfy24o5OMtUYg7uFUoVCxXCDdr21ugDU hUkEMFvsNMIq5WMn/rjmZvG6Fmpu+nEdV9Z7E964yAS3H/FEpE7AG3IEi79YvnlILht0eJOzPbpNP QHfyY9BIx5pQTjzzmSBs+d1wRj+ahHta67u5MW0Y06RLhyMGFHDyuEM5e+/grhx8RcBzj970O+Tnb fnkyNEBEBT7z4O8bi6rIcCESlTvMbrnxps7ArtxAqZkhv+T2GElk5BjyTMDptY5kRX8G1WTb7cSoV 0s1JrEItL+7ECZe6up7A==; Received: from localhost ([::1] helo=bombadil.infradead.org) by bombadil.infradead.org with esmtp (Exim 4.98 #2 (Red Hat Linux)) id 1tPfGA-00000009kZp-2D0l; Mon, 23 Dec 2024 10:01:58 +0000 Received: from desiato.infradead.org ([2001:8b0:10b:1:d65d:64ff:fe57:4e05]) by bombadil.infradead.org with esmtps (Exim 4.98 #2 (Red Hat Linux)) id 1tPf1Y-00000009huc-1Gmm for linux-riscv@bombadil.infradead.org; Mon, 23 Dec 2024 09:46:52 +0000 DKIM-Signature: v=1; a=rsa-sha256; q=dns/txt; c=relaxed/relaxed; d=infradead.org; s=desiato.20200630; h=Content-Transfer-Encoding:MIME-Version :References:In-Reply-To:Message-Id:Date:Subject:Cc:To:From:Sender:Reply-To: Content-Type:Content-ID:Content-Description; bh=/Vy7m3PQdTp9s3cHn2TFUEZi29ZbV6wwwxQw/bfJltk=; b=oMg8rIs3MyoHBzFX/LyMFRSZFb ZFLwwFV2I7KkG54Rr1Gfktl1NSOy1TQrdz2zhblYHfq21hmZORw8vz/eX6x4BJDRfezU4G7T9Bnlg y37G6cG/BDVnu4t0lyDa/ZC4wAq+AK46xA7cyx58mNQvyY+evKFlqIQKZmRIJGQtnEe9P4ElHjLg/ s4dEZsBlfJslzO8lK1aABGRnVlp6L/p8L1qgC+62wPhuT2iHEDxzgQAotMmjvQLfup8EtcPfXl5NF CLN38OAGn8D0wk1wrDqOZRtVIHkORL9fy17SmGyw0XUIBfn66Qe4rzlzSx5IXIGVBtWriaYtbg12d 9cBOC0og==; Received: from mail-pf1-x431.google.com ([2607:f8b0:4864:20::431]) by desiato.infradead.org with esmtps (Exim 4.98 #2 (Red Hat Linux)) id 1tPf1V-00000006J30-0ie5 for linux-riscv@lists.infradead.org; Mon, 23 Dec 2024 09:46:51 +0000 Received: by mail-pf1-x431.google.com with SMTP id d2e1a72fcca58-725ed193c9eso3397022b3a.1 for ; Mon, 23 Dec 2024 01:46:42 -0800 (PST) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=bytedance.com; s=google; t=1734947200; x=1735552000; darn=lists.infradead.org; h=content-transfer-encoding:mime-version:references:in-reply-to :message-id:date:subject:cc:to:from:from:to:cc:subject:date :message-id:reply-to; bh=/Vy7m3PQdTp9s3cHn2TFUEZi29ZbV6wwwxQw/bfJltk=; b=dWHNm1fTC93qpo09q3RNXZsq9eLA2tvIe04O6rsuA0e7WL8Vw8wY/D2DCJkVz1BO3S ++Uvx2GyRgkN8k68dWAx9tPXO1lGU3ZttjXoEOF3nPOyqsJJW3LGAR+kAC4xAPvkHlhY GI8GW7HKNdxlc8I1A4iRnhxSmSn2PaMe4AgUaed97R0eXEQRoxNTdqAlQOL+wFIQo87t /jhxUu/wewlWqDi6pArvsDpAjxopyZH9n2KKKsh9n72c60W5ScmRywDJ0dc0XIwP/5Y3 6IFJ54sTcQPbszozMeyTGwSsuR182X3HqEunzg8nuhgpSYTlETxwz3kvUnxAn98eUXgn e15g== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20230601; t=1734947200; x=1735552000; h=content-transfer-encoding:mime-version:references:in-reply-to :message-id:date:subject:cc:to:from:x-gm-message-state:from:to:cc :subject:date:message-id:reply-to; bh=/Vy7m3PQdTp9s3cHn2TFUEZi29ZbV6wwwxQw/bfJltk=; b=CJUnWPcQn9VoERZYSKNMjk2/0d1KjKz+Srs17fbd1lltVZbK3Mi5teUb7JZzViiMoq HyYO3TFhbv/PW7bzIPODRxNUzmfO68Kzk6BqNtYA1hDYIYLDTtZS6a/+VTjheJH+2KW8 57TU4NhT6pQTjSHMukjwm1rMz3RRXP/7Z3BAqi/ESMC4xhs3am2GsZPRBp1tv+JxxDyO ZrA/RIph0XMq6/3kc7bn8d3yq5Bv5WkFfTzm8BFEK5YDywdFDwDxBZWJ5UqvkknK9jmU Dlvdk25f2ZNu3V3P3p9hTssHcyl6XHPEgTZMfe4tMIU7sV7zJcfArugK/ARhmogMbYLR Kqkg== X-Forwarded-Encrypted: i=1; AJvYcCUtkjw8mH0r/R1SeiqvYdrfHb2kiTCVQu9arQ2VM2a+Gu/MPkhqldZHpmFUcXbK+f4t3aPkrM/7CmHzpg==@lists.infradead.org X-Gm-Message-State: AOJu0YxHSIhUb9ndcHvnId0yCVAv+EkF7KIMeMdxa1vbTMR5sUyYv4Fk ljX+5SpvzVUFzeZJwKlH2Xq8KsH0gDnVsvK+M/7ydgvB5X62EHWNvC1MLmHGzxo= X-Gm-Gg: ASbGncsmDZj8yYVcBgQqmY+JB1ZfMGTca/QZ2bfM8OQXTw2gvBOY1xR3B0udZhpwuCh P/WVfsD112VKyLtdR9iBxQkvtBQv4cG5bZqrveUB7/T6IS6oIvu9O7rHzEXtd42xnwMlZt/tgFQ GVnbuJNRQL/BVJjcb9t+7T9YcT7rzZgBbHvBwvTg9/F2SNSHDb59iEzWJixmn7gHvpkpM89uuYP B5gVOmKoRb7rDcM+bDy9oQ0t3ZsV9eH96X8M7ksSQ8nzCmqW5VFDL4ffwcK3UFGDKjN9/wHECYv It5CUAksX1zruW6JGzlQLg== X-Google-Smtp-Source: AGHT+IHTlP+hesMysY/dnyw4k1T/aSGpJ0FSxzWG+ektln9VxnC2FaHlf9zeRwOgabkwOIXU2HSkPg== X-Received: by 2002:a05:6a00:2407:b0:729:597:4faa with SMTP id d2e1a72fcca58-72abde9c004mr15027146b3a.16.1734947200520; Mon, 23 Dec 2024 01:46:40 -0800 (PST) Received: from C02DW0BEMD6R.bytedance.net ([203.208.167.150]) by smtp.gmail.com with ESMTPSA id d2e1a72fcca58-72aad8486c6sm7468309b3a.85.2024.12.23.01.46.28 (version=TLS1_3 cipher=TLS_AES_256_GCM_SHA384 bits=256/256); Mon, 23 Dec 2024 01:46:40 -0800 (PST) From: Qi Zheng To: peterz@infradead.org, agordeev@linux.ibm.com, kevin.brodsky@arm.com, tglx@linutronix.de, david@redhat.com, jannh@google.com, hughd@google.com, yuzhao@google.com, willy@infradead.org, muchun.song@linux.dev, vbabka@kernel.org, lorenzo.stoakes@oracle.com, akpm@linux-foundation.org, rientjes@google.com, vishal.moola@gmail.com, arnd@arndb.de, will@kernel.org, aneesh.kumar@kernel.org, npiggin@gmail.com, dave.hansen@linux.intel.com, rppt@kernel.org, ryan.roberts@arm.com Cc: linux-mm@kvack.org, linux-arm-kernel@lists.infradead.org, linuxppc-dev@lists.ozlabs.org, linux-riscv@lists.infradead.org, linux-s390@vger.kernel.org, sparclinux@vger.kernel.org, linux-kernel@vger.kernel.org, x86@kernel.org, linux-arch@vger.kernel.org, linux-csky@vger.kernel.org, linux-hexagon@vger.kernel.org, loongarch@lists.linux.dev, linux-m68k@lists.linux-m68k.org, linux-mips@vger.kernel.org, linux-openrisc@vger.kernel.org, linux-sh@vger.kernel.org, linux-um@lists.infradead.org, Qi Zheng Subject: [PATCH v3 17/17] mm: pgtable: introduce generic pagetable_dtor_free() Date: Mon, 23 Dec 2024 17:41:03 +0800 Message-Id: <3ade33c5049f465dc2f0b95edc2d68c80f2048c9.1734945104.git.zhengqi.arch@bytedance.com> X-Mailer: git-send-email 2.24.3 (Apple Git-128) In-Reply-To: References: MIME-Version: 1.0 X-CRM114-Version: 20100106-BlameMichelson ( TRE 0.8.0 (BSD) ) MR-646709E3 X-CRM114-CacheID: sfid-20241223_094649_327762_A315B356 X-CRM114-Status: GOOD ( 10.83 ) X-BeenThere: linux-riscv@lists.infradead.org X-Mailman-Version: 2.1.34 Precedence: list List-Id: List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Sender: "linux-riscv" Errors-To: linux-riscv-bounces+linux-riscv=archiver.kernel.org@lists.infradead.org The pte_free(), pmd_free(), __pud_free() and __p4d_free() in asm-generic/pgalloc.h and the generic __tlb_remove_table() are basically the same, so let's introduce pagetable_dtor_free() to deduplicate them. In addition, the pagetable_dtor_free() in s390 actually does the same thing, so let's s390 also calls generic pagetable_dtor_free(). Signed-off-by: Qi Zheng Suggested-by: Peter Zijlstra (Intel) --- arch/s390/mm/pgalloc.c | 18 ++++++------------ include/asm-generic/pgalloc.h | 23 ++++------------------- include/asm-generic/tlb.h | 5 +---- include/linux/mm.h | 8 ++++++++ 4 files changed, 19 insertions(+), 35 deletions(-) diff --git a/arch/s390/mm/pgalloc.c b/arch/s390/mm/pgalloc.c index 3e002dea6278f..1e0727be48eaf 100644 --- a/arch/s390/mm/pgalloc.c +++ b/arch/s390/mm/pgalloc.c @@ -180,32 +180,26 @@ unsigned long *page_table_alloc(struct mm_struct *mm) return table; } -static void pagetable_dtor_free(struct ptdesc *ptdesc) -{ - pagetable_dtor(ptdesc); - pagetable_free(ptdesc); -} - void page_table_free(struct mm_struct *mm, unsigned long *table) { - struct ptdesc *ptdesc = virt_to_ptdesc(table); + struct page *page = virt_to_page(table); - pagetable_dtor_free(ptdesc); + pagetable_dtor_free(page); } #ifdef CONFIG_TRANSPARENT_HUGEPAGE static void pte_free_now(struct rcu_head *head) { - struct ptdesc *ptdesc = container_of(head, struct ptdesc, pt_rcu_head); + struct page *page = container_of(head, struct page, rcu_head); - pagetable_dtor_free(ptdesc); + pagetable_dtor_free(page); } void pte_free_defer(struct mm_struct *mm, pgtable_t pgtable) { - struct ptdesc *ptdesc = virt_to_ptdesc(pgtable); + struct page *page = virt_to_page(pgtable); - call_rcu(&ptdesc->pt_rcu_head, pte_free_now); + call_rcu(&page->rcu_head, pte_free_now); /* * THPs are not allowed for KVM guests. Warn if pgste ever reaches here. * Turn to the generic pte_free_defer() version once gmap is removed. diff --git a/include/asm-generic/pgalloc.h b/include/asm-generic/pgalloc.h index 4afb346eae255..7d327889df306 100644 --- a/include/asm-generic/pgalloc.h +++ b/include/asm-generic/pgalloc.h @@ -107,10 +107,7 @@ static inline pgtable_t pte_alloc_one_noprof(struct mm_struct *mm) */ static inline void pte_free(struct mm_struct *mm, struct page *pte_page) { - struct ptdesc *ptdesc = page_ptdesc(pte_page); - - pagetable_dtor(ptdesc); - pagetable_free(ptdesc); + pagetable_dtor_free(pte_page); } @@ -150,11 +147,7 @@ static inline pmd_t *pmd_alloc_one_noprof(struct mm_struct *mm, unsigned long ad #ifndef __HAVE_ARCH_PMD_FREE static inline void pmd_free(struct mm_struct *mm, pmd_t *pmd) { - struct ptdesc *ptdesc = virt_to_ptdesc(pmd); - - BUG_ON((unsigned long)pmd & (PAGE_SIZE-1)); - pagetable_dtor(ptdesc); - pagetable_free(ptdesc); + pagetable_dtor_free(virt_to_page(pmd)); } #endif @@ -199,11 +192,7 @@ static inline pud_t *pud_alloc_one_noprof(struct mm_struct *mm, unsigned long ad static inline void __pud_free(struct mm_struct *mm, pud_t *pud) { - struct ptdesc *ptdesc = virt_to_ptdesc(pud); - - BUG_ON((unsigned long)pud & (PAGE_SIZE-1)); - pagetable_dtor(ptdesc); - pagetable_free(ptdesc); + pagetable_dtor_free(virt_to_page(pud)); } #ifndef __HAVE_ARCH_PUD_FREE @@ -245,11 +234,7 @@ static inline p4d_t *p4d_alloc_one_noprof(struct mm_struct *mm, unsigned long ad static inline void __p4d_free(struct mm_struct *mm, p4d_t *p4d) { - struct ptdesc *ptdesc = virt_to_ptdesc(p4d); - - BUG_ON((unsigned long)p4d & (PAGE_SIZE-1)); - pagetable_dtor(ptdesc); - pagetable_free(ptdesc); + pagetable_dtor_free(virt_to_page(p4d)); } #ifndef __HAVE_ARCH_P4D_FREE diff --git a/include/asm-generic/tlb.h b/include/asm-generic/tlb.h index 583e95568f52b..ef25169523602 100644 --- a/include/asm-generic/tlb.h +++ b/include/asm-generic/tlb.h @@ -211,10 +211,7 @@ struct mmu_table_batch { #ifndef __HAVE_ARCH_TLB_REMOVE_TABLE static inline void __tlb_remove_table(void *table) { - struct ptdesc *ptdesc = page_ptdesc((struct page *)table); - - pagetable_dtor(ptdesc); - pagetable_free(ptdesc); + pagetable_dtor_free(table); } #endif diff --git a/include/linux/mm.h b/include/linux/mm.h index cad11fa10c192..cd078d51f47c7 100644 --- a/include/linux/mm.h +++ b/include/linux/mm.h @@ -3001,6 +3001,14 @@ static inline void pagetable_dtor(struct ptdesc *ptdesc) lruvec_stat_sub_folio(folio, NR_PAGETABLE); } +static inline void pagetable_dtor_free(void *table) +{ + struct ptdesc *ptdesc = page_ptdesc((struct page *)table); + + pagetable_dtor(ptdesc); + pagetable_free(ptdesc); +} + static inline bool pagetable_pte_ctor(struct ptdesc *ptdesc) { struct folio *folio = ptdesc_folio(ptdesc);