From patchwork Tue Aug 6 02:21:13 2024 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Yu Zhao X-Patchwork-Id: 13754342 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from kanga.kvack.org (kanga.kvack.org [205.233.56.17]) by smtp.lore.kernel.org (Postfix) with ESMTP id B34FDC52D6F for ; Tue, 6 Aug 2024 02:21:30 +0000 (UTC) Received: by kanga.kvack.org (Postfix) id 425B26B00C3; Mon, 5 Aug 2024 22:21:30 -0400 (EDT) Received: by kanga.kvack.org (Postfix, from userid 40) id 3AF866B00C4; Mon, 5 Aug 2024 22:21:30 -0400 (EDT) X-Delivered-To: int-list-linux-mm@kvack.org Received: by kanga.kvack.org (Postfix, from userid 63042) id 1DB146B00C5; Mon, 5 Aug 2024 22:21:30 -0400 (EDT) X-Delivered-To: linux-mm@kvack.org Received: from relay.hostedemail.com (smtprelay0013.hostedemail.com [216.40.44.13]) by kanga.kvack.org (Postfix) with ESMTP id EA2606B00C3 for ; Mon, 5 Aug 2024 22:21:29 -0400 (EDT) Received: from smtpin10.hostedemail.com (a10.router.float.18 [10.200.18.1]) by unirelay09.hostedemail.com (Postfix) with ESMTP id 964F6804FC for ; Tue, 6 Aug 2024 02:21:29 +0000 (UTC) X-FDA: 82420219098.10.B58608E Received: from mail-yw1-f202.google.com (mail-yw1-f202.google.com [209.85.128.202]) by imf24.hostedemail.com (Postfix) with ESMTP id CE91018000F for ; Tue, 6 Aug 2024 02:21:27 +0000 (UTC) Authentication-Results: imf24.hostedemail.com; dkim=pass header.d=google.com header.s=20230601 header.b=PEtNHwlt; spf=pass (imf24.hostedemail.com: domain of 3poixZgYKCLkxtygZnfnnfkd.bnlkhmtw-lljuZbj.nqf@flex--yuzhao.bounces.google.com designates 209.85.128.202 as permitted sender) smtp.mailfrom=3poixZgYKCLkxtygZnfnnfkd.bnlkhmtw-lljuZbj.nqf@flex--yuzhao.bounces.google.com; dmarc=pass (policy=reject) header.from=google.com ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=hostedemail.com; s=arc-20220608; t=1722910825; h=from:from:sender:reply-to:subject:subject:date:date: message-id:message-id:to:to:cc:cc:mime-version:mime-version: content-type:content-type:content-transfer-encoding: in-reply-to:in-reply-to:references:references:dkim-signature; bh=Dz9+L9Nas5YdR+3aU4+X/tR5OCfJ0AKY3iAQMLv0S5k=; b=XisR+LWXpvV0MYFtL2pfZac+lEWNQbS86FGKj5S4Rxr+qocAl/IsW4hGf1/FGSQflYdx9T azj+r6UJIsb45iqmOrAcvn1q5HeT4HXQfPITvNkcaUd/rbfcSW5anxH5GnviI4/jcGOfHo JLBTQ8/iGLHj88DZUKC5xbbY1cbKp8E= ARC-Seal: i=1; s=arc-20220608; d=hostedemail.com; t=1722910825; a=rsa-sha256; cv=none; b=FCnEiwEfYAaWEc3HH/4j+JaiIcKfWpIEwPGaiESPs+ir+cgFSxCW6/eNIpWs4BsmTC9I2E RdpzR+W5zYgKQv/eFC2nidNf5jOq/rlX47rAJWgBWPtp9B1KtzRMe1XY4lZhTfHOGtnrYZ 71dLs2ICzi4zJcUG30Ohl8VYgULpK+o= ARC-Authentication-Results: i=1; imf24.hostedemail.com; dkim=pass header.d=google.com header.s=20230601 header.b=PEtNHwlt; spf=pass (imf24.hostedemail.com: domain of 3poixZgYKCLkxtygZnfnnfkd.bnlkhmtw-lljuZbj.nqf@flex--yuzhao.bounces.google.com designates 209.85.128.202 as permitted sender) smtp.mailfrom=3poixZgYKCLkxtygZnfnnfkd.bnlkhmtw-lljuZbj.nqf@flex--yuzhao.bounces.google.com; dmarc=pass (policy=reject) header.from=google.com Received: by mail-yw1-f202.google.com with SMTP id 00721157ae682-690404fd34eso534357b3.1 for ; Mon, 05 Aug 2024 19:21:27 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=google.com; s=20230601; t=1722910887; x=1723515687; darn=kvack.org; h=cc:to:from:subject:message-id:references:mime-version:in-reply-to :date:from:to:cc:subject:date:message-id:reply-to; bh=Dz9+L9Nas5YdR+3aU4+X/tR5OCfJ0AKY3iAQMLv0S5k=; b=PEtNHwlt9D0n7j//BgajQRwPbiCGfxhC495MUF/BUei51TzojKWz+g33/3qwgQOlk6 Z3gB8FcA9KZHXNjhe22iqQqvBEj0LyI0VD6giOh2A2T/YehkmwXxwPmqAeg+m52PD21H Y5D3EWAkpw4WsksueVJ2zeAMXH0vL/gtE9oU4BNxA4oO/KvrR+ErYdqiPqWIvqlGfD60 32felkPWW9L9RjIahksZ1y0bPm/fiToqAYOQDuWzOkXi+B95lDCXmCVpkpR1+uFPAYcb qqPvcO/dLYFN4jOFWhU764Dj26BtsAoW0spMu/e8RUFlXVXPHNdmaEmKwxNY6yTXfckq dVCA== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20230601; t=1722910887; x=1723515687; h=cc:to:from:subject:message-id:references:mime-version:in-reply-to :date:x-gm-message-state:from:to:cc:subject:date:message-id:reply-to; bh=Dz9+L9Nas5YdR+3aU4+X/tR5OCfJ0AKY3iAQMLv0S5k=; b=JZcd6jFDUgRKGkIRZin9GTgJckxg+7FYEPuI7Wcli0w94lcuQF5nCZc2f6OO9ZLFpp S72+nVT3VmbwNhGtoR7xv8+EboQAYwD6BAyYzdjx5amL8hPTq/gY0xiJ+pwSp5bDhhGM 6KIL14fsAqzXKMDI0YGwaRa/G+gT8PpptLJ/PAV6NYhcfkLw7nXvVZMJPzNgIJmKLFYA FFhW0OJ+XHyqneZtuIFd/2Ejtk5qpPrn5uns6JQ0ZUdog7ckUUSfmQHXm5ksOX/P4CYY iQBrCEz3lo5gfl6YgDHPT7JBi8AWQg5B/8cX+WaJStqmDwWkoydF0HKLo/4huPA966NA aH7A== X-Forwarded-Encrypted: i=1; AJvYcCVYhg85DHNx3TYXLC4+zxmLMhkFB69U8EOZah0+bVnwSUgKNi5C6mnWW9LeWKAQo/zAZo8E1RtW9m9w4u1m0xvb0ec= X-Gm-Message-State: AOJu0Yz1j97GEQyQjDZrxjp4xcN+9tCGAGb4K1jcDRse+XLd0lroF+bT HVccIxMtFAZVfpdPxVv/MSBZO7Im13pi7Vqe0W0JiPUMYxSPhVzPjDbT4Orfidyn4cjLuUQWtW1 RLw== X-Google-Smtp-Source: AGHT+IE5Ot4MCUehL2BALtzt0Nbbacrl8HCjMPwdvfemN8nL8qevC8oWRD8zmA2efNk4jGDFZ//D943lM9s= X-Received: from yuzhao2.bld.corp.google.com ([2a00:79e0:2e28:6:261c:802b:6b55:e09c]) (user=yuzhao job=sendgmr) by 2002:a05:690c:113:b0:648:3f93:61f2 with SMTP id 00721157ae682-689641a414emr5295867b3.6.1722910886865; Mon, 05 Aug 2024 19:21:26 -0700 (PDT) Date: Mon, 5 Aug 2024 20:21:13 -0600 In-Reply-To: <20240806022114.3320543-1-yuzhao@google.com> Mime-Version: 1.0 References: <20240806022114.3320543-1-yuzhao@google.com> X-Mailer: git-send-email 2.46.0.rc2.264.g509ed76dc8-goog Message-ID: <20240806022114.3320543-4-yuzhao@google.com> Subject: [RFC PATCH 3/4] arm64: pause remote CPUs to update vmemmap From: Yu Zhao To: Catalin Marinas , Will Deacon Cc: Andrew Morton , David Rientjes , Douglas Anderson , Frank van der Linden , Mark Rutland , Muchun Song , Nanyong Sun , Yang Shi , linux-arm-kernel@lists.infradead.org, linux-mm@kvack.org, linux-kernel@vger.kernel.org, Yu Zhao X-Stat-Signature: njyg7fp6m5sfwnxhgot3emj67t1okfu7 X-Rspamd-Queue-Id: CE91018000F X-Rspam-User: X-Rspamd-Server: rspam08 X-HE-Tag: 1722910887-703919 X-HE-Meta: U2FsdGVkX1+Vf3w+4g0NdHA4xAemJKPn8ks4uU0ErTswztnZbDByXmMeZ/j/JmEKgUBrqBN+LnuQbw1i3/jbYalS5uDSuoDSmUPxCZpwMkTv4YDQMy8cP4Ug8MZ6StrhJI/g4F+CcZSoWuYOQszNDMwx8mRl+OcTfAHrvSXspTQzb1GDkRtujhRKTo5zCy/4/xwvQL3IJCWhEOKHraOk3pe9kRwiIyICcdYsTsc35m0jNYGlIfCteRAZNuw92IXbOm9+OBSOLIUPfE0dHViBdEhNNmN5hoNl5FPkJl7HLfoTnCm2avZUBOATKBz5jE31WFQ4y/y6XA9ktRb88L1TzMPIwg5OP9NbeJ9R8ZC0DW5NwlNaIRvX04d67Lth28VdT+nZP5DFfKtuMTNwqITnV8S17XuLtBLGpRSvtXK+GLfux/OZewsIELFXDPuCy87B4FBQhuJgvrs3einCUhL2dqMY+1CljZJVUZzg2HCOrWYwJpuTfbk2VoOIK/DxOCMcklQJQa6GPMVpPiqTVxhFUOm6CZ7vBh264TUf21ym6m2LgfzBul8ukM7q4KLBUwpyXWy1wo0IOOWluuicQ4tPgnophSOkF6vr3Rn1EwYjTewcZNOJHNnku/nlIKhV1L8oqsgiobJyvDZI77DbuqkODmyyy/u7dib8I6HyumMR5dXMjAyfYf9ZgCx5No3fJdmoHQaxCNx8PeipF+QrHYocu1Sxq99du6sCaPphFUVh/EVQEFK6madXs3IpASQnJ858Jo2WpqLW2CbxL8F8UArT+I/DgJl6yL/7p9SnxF5WhtA8PNbUWE9F+DTwp5k4zIsnFAvI394/C93DUeH6Gm2eU2BK38YXVPEr1UdnncskoTmpsw3Y3zXMrE4tHG/FxlNMdVrpd/rZW9rXiZBUThXnt4NJkTdTqtSzD9XWrhk8xsq5rj/Ay33Nv6/jV5in+4bXrgWgdg78hYOOiV0NloQ Qlkr5ZfS J/kB4g82oC8YBBTNvOPUVRyAHH6banok2Ff/x6BXUyXZ9Uk+IYxAOriVou47GeVfkYJc94i3z/D+dRuLqnHA+w/1K41lgkDwEZ0TkZ3u8qYZx4EfcKi9ENiwQFVbs5IIRDvE3WF4yVRb3bS4B0lh+kAp2peFw0s08W7lhaJzzcCkCNKf23pSoZcIqS5lvmEHf3ZYM3OLZimqSCh+H97QgSd+tjH9ilJdEBdJ2NttAoVoUhlmK/D+hR+bVlg4otVudZdLbrPkiy1jpPkB+AiOprHNjlUg70Nhaa1cq/3ZgiRmFJjOrRHL8xqAE50Q82rLG87R9OLJo9HGg3hqu2xfmtNxhSserAcQhtfZQlqRkyuEYElOETq4vDaGvLlpGr3TBu0bRgBcpWDki6l43vAbGGixVOnfoNS/EbKrmEWO5cSc/z6vu3YKvZyXMMSRSgonorAm05emuf9VtFwIoS4DI4YZzQYLXd/Mw0t4i0wdteopMHadv6T4PLDDZbJUgqWkaiqM06LhW99ikHo43BjEsIZgwH3EH3IoIUrIiu1IDLJXaIOTFxndj7OJoBg== X-Bogosity: Ham, tests=bogofilter, spamicity=0.000000, version=1.2.4 Sender: owner-linux-mm@kvack.org Precedence: bulk X-Loop: owner-majordomo@kvack.org List-ID: List-Subscribe: List-Unsubscribe: Pause remote CPUs so that the local CPU can follow the proper BBM sequence to safely update the vmemmap mapping `struct page` areas. While updating the vmemmap, it is guaranteed that neither the local CPU nor the remote ones will access the `struct page` area being updated, and therefore they will not trigger kernel PFs. Signed-off-by: Yu Zhao --- arch/arm64/include/asm/pgalloc.h | 55 ++++++++++++++++++++++++++++++++ mm/hugetlb_vmemmap.c | 14 ++++++++ 2 files changed, 69 insertions(+) diff --git a/arch/arm64/include/asm/pgalloc.h b/arch/arm64/include/asm/pgalloc.h index 8ff5f2a2579e..1af1aa34a351 100644 --- a/arch/arm64/include/asm/pgalloc.h +++ b/arch/arm64/include/asm/pgalloc.h @@ -12,6 +12,7 @@ #include #include #include +#include #define __HAVE_ARCH_PGD_FREE #define __HAVE_ARCH_PUD_FREE @@ -137,4 +138,58 @@ pmd_populate(struct mm_struct *mm, pmd_t *pmdp, pgtable_t ptep) __pmd_populate(pmdp, page_to_phys(ptep), PMD_TYPE_TABLE | PMD_TABLE_PXN); } +#ifdef CONFIG_HUGETLB_PAGE_OPTIMIZE_VMEMMAP + +#define vmemmap_update_lock vmemmap_update_lock +static inline void vmemmap_update_lock(void) +{ + cpus_read_lock(); +} + +#define vmemmap_update_unlock vmemmap_update_unlock +static inline void vmemmap_update_unlock(void) +{ + cpus_read_unlock(); +} + +#define vmemmap_update_pte vmemmap_update_pte +static inline void vmemmap_update_pte(unsigned long addr, pte_t *ptep, pte_t pte) +{ + preempt_disable(); + pause_remote_cpus(); + + pte_clear(&init_mm, addr, ptep); + flush_tlb_kernel_range(addr, addr + PAGE_SIZE); + set_pte_at(&init_mm, addr, ptep, pte); + + resume_remote_cpus(); + preempt_enable(); +} + +#define vmemmap_update_pmd vmemmap_update_pmd +static inline void vmemmap_update_pmd(unsigned long addr, pmd_t *pmdp, pte_t *ptep) +{ + preempt_disable(); + pause_remote_cpus(); + + pmd_clear(pmdp); + flush_tlb_kernel_range(addr, addr + PMD_SIZE); + pmd_populate_kernel(&init_mm, pmdp, ptep); + + resume_remote_cpus(); + preempt_enable(); +} + +#define vmemmap_flush_tlb_all vmemmap_flush_tlb_all +static inline void vmemmap_flush_tlb_all(void) +{ +} + +#define vmemmap_flush_tlb_range vmemmap_flush_tlb_range +static inline void vmemmap_flush_tlb_range(unsigned long start, unsigned long end) +{ +} + +#endif /* CONFIG_HUGETLB_PAGE_OPTIMIZE_VMEMMAP */ + #endif diff --git a/mm/hugetlb_vmemmap.c b/mm/hugetlb_vmemmap.c index 2dd92e58f304..893c73493d9c 100644 --- a/mm/hugetlb_vmemmap.c +++ b/mm/hugetlb_vmemmap.c @@ -46,6 +46,18 @@ struct vmemmap_remap_walk { unsigned long flags; }; +#ifndef vmemmap_update_lock +static void vmemmap_update_lock(void) +{ +} +#endif + +#ifndef vmemmap_update_unlock +static void vmemmap_update_unlock(void) +{ +} +#endif + #ifndef vmemmap_update_pmd static inline void vmemmap_update_pmd(unsigned long addr, pmd_t *pmdp, pte_t *ptep) @@ -194,10 +206,12 @@ static int vmemmap_remap_range(unsigned long start, unsigned long end, VM_BUG_ON(!PAGE_ALIGNED(start | end)); + vmemmap_update_lock(); mmap_read_lock(&init_mm); ret = walk_page_range_novma(&init_mm, start, end, &vmemmap_remap_ops, NULL, walk); mmap_read_unlock(&init_mm); + vmemmap_update_unlock(); if (ret) return ret;