From patchwork Tue Aug 6 02:21:11 2024 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Yu Zhao X-Patchwork-Id: 13754340 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from kanga.kvack.org (kanga.kvack.org [205.233.56.17]) by smtp.lore.kernel.org (Postfix) with ESMTP id 996D3C3DA4A for ; Tue, 6 Aug 2024 02:21:25 +0000 (UTC) Received: by kanga.kvack.org (Postfix) id 2C6BF6B00BE; Mon, 5 Aug 2024 22:21:25 -0400 (EDT) Received: by kanga.kvack.org (Postfix, from userid 40) id 24F116B00C2; Mon, 5 Aug 2024 22:21:25 -0400 (EDT) X-Delivered-To: int-list-linux-mm@kvack.org Received: by kanga.kvack.org (Postfix, from userid 63042) id 0A0936B00C1; Mon, 5 Aug 2024 22:21:24 -0400 (EDT) X-Delivered-To: linux-mm@kvack.org Received: from relay.hostedemail.com (smtprelay0016.hostedemail.com [216.40.44.16]) by kanga.kvack.org (Postfix) with ESMTP id D32E56B00BE for ; Mon, 5 Aug 2024 22:21:24 -0400 (EDT) Received: from smtpin09.hostedemail.com (a10.router.float.18 [10.200.18.1]) by unirelay02.hostedemail.com (Postfix) with ESMTP id 9212A12088E for ; Tue, 6 Aug 2024 02:21:24 +0000 (UTC) X-FDA: 82420218888.09.C9B43A2 Received: from mail-yb1-f201.google.com (mail-yb1-f201.google.com [209.85.219.201]) by imf05.hostedemail.com (Postfix) with ESMTP id CAACA10000B for ; Tue, 6 Aug 2024 02:21:22 +0000 (UTC) Authentication-Results: imf05.hostedemail.com; dkim=pass header.d=google.com header.s=20230601 header.b=pLQle4Uu; spf=pass (imf05.hostedemail.com: domain of 3oYixZgYKCLQsotbUiaiiafY.Wigfchor-ggepUWe.ila@flex--yuzhao.bounces.google.com designates 209.85.219.201 as permitted sender) smtp.mailfrom=3oYixZgYKCLQsotbUiaiiafY.Wigfchor-ggepUWe.ila@flex--yuzhao.bounces.google.com; dmarc=pass (policy=reject) header.from=google.com ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=hostedemail.com; s=arc-20220608; t=1722910834; h=from:from:sender:reply-to:subject:subject:date:date: message-id:message-id:to:to:cc:cc:mime-version:mime-version: content-type:content-type:content-transfer-encoding: in-reply-to:in-reply-to:references:references:dkim-signature; bh=yV7dY4XDiIXGgrOmDcJWCpFmJrDLDUqsW20nfMTrSyE=; b=WopLgW22B93L8Zh9kSh9uTGI2xcI4Q0/5M2X/1YQNGRCWsbsN/9aVP9DzKmZkTpPgN0V+e 8oZ0SgF6qzytjs2PyYMEQ6O3cDPYE50+tW+VA0GIAPnEaIXl0kM7iULOdfDRATBbEEkzXA g2pDqDyZfwZannNhdsihANuSpzL2EPI= ARC-Authentication-Results: i=1; imf05.hostedemail.com; dkim=pass header.d=google.com header.s=20230601 header.b=pLQle4Uu; spf=pass (imf05.hostedemail.com: domain of 3oYixZgYKCLQsotbUiaiiafY.Wigfchor-ggepUWe.ila@flex--yuzhao.bounces.google.com designates 209.85.219.201 as permitted sender) smtp.mailfrom=3oYixZgYKCLQsotbUiaiiafY.Wigfchor-ggepUWe.ila@flex--yuzhao.bounces.google.com; dmarc=pass (policy=reject) header.from=google.com ARC-Seal: i=1; s=arc-20220608; d=hostedemail.com; t=1722910834; a=rsa-sha256; cv=none; b=ecRwLfolAxaeqG2tfPu/Kqgmh3bk1E8i13b9bobJQ56VtJ+vqtFl/OGOwYCJf2ydlyMxJL nUJuV4I3PiZKaNV247y+gDXln7tHsc0nZOmYwoU8LDAvTOaZDQZcLJQR+iGZBRx8OLK0L2 YPnuiCxckuQPhd4HJBI4X/IWTKkq/gU= Received: by mail-yb1-f201.google.com with SMTP id 3f1490d57ef6-e0e3eb3feaeso340153276.2 for ; Mon, 05 Aug 2024 19:21:22 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=google.com; s=20230601; t=1722910882; x=1723515682; darn=kvack.org; h=cc:to:from:subject:message-id:references:mime-version:in-reply-to :date:from:to:cc:subject:date:message-id:reply-to; bh=yV7dY4XDiIXGgrOmDcJWCpFmJrDLDUqsW20nfMTrSyE=; b=pLQle4Uuk4Kp4rRUF5o+aoIelLlmUdPrnAaq3KHm50iFwJc9/b47tnjcQwkXsw6N6K mJBdCfsEkJZlxKnCHl4p/GBqAMROVpaEc7CRb7GQadxj8pGy84G8/rqFiPvZyL1KPuva IJ7HGeQkJIIrgTT5tydBe5T7t9aYBcjtXzIbfiKRyASwz3I3qaGgOSAyWyDL/XOroU1s a8UW0qQUh6Vawev6rXwDhOh/yPmyp1NDs9WIUIsdmtVocHXSi61A/zOtfiurlMrPzz2y dKvf0O/vry4LcdbFtnGWxl6EfgOkXcsiS+wfRQ1YycmAAxpoCpz/SG6TeTg3uFpNj97F bPQw== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20230601; t=1722910882; x=1723515682; h=cc:to:from:subject:message-id:references:mime-version:in-reply-to :date:x-gm-message-state:from:to:cc:subject:date:message-id:reply-to; bh=yV7dY4XDiIXGgrOmDcJWCpFmJrDLDUqsW20nfMTrSyE=; b=hY/UyFAV2vCBITBwc5EovRsYzsCbK94W7L7UhiCSGzPMZUZaud7pim7JgqyUjo6qOR wgsFlBjfwE7Wp9ER07fuPs20SmYVvji/cpfrlaZpUetmdM2vJZEopfa5fle17gjPJYc1 3MxBSD9NZ+FLeHEImxhfb2FOkghswXPEgDklOk9xV5W2eG5KzdKPM4nDTOv/5N4Gv6hx ZYwRdxaCh0qXyLQkPfoZHygeiZa+Y4WTBTnEUSzfYTYnHY5SiITeJ0VuDgApNxdGCgnL 2rrL4PkyellNZ/ESOXhqkdxd/+4XJnp55nc+8Ds/VSPQgMvYiIlqYrsLtjzciY8G1NhE Jq2Q== X-Forwarded-Encrypted: i=1; AJvYcCUnt3oBIyybQyJrrgnc1Sm9nygVyPRqA2WdRezJ2DzgRu1LJ6jPgqS7kiNDI4LT42jNOTyk9euCXt/YH2kZrUt2KvE= X-Gm-Message-State: AOJu0YzqWJMg3/w9bDAAWaXntZOZ77OIg7BWKBEKLhnkVDlvoMhSxhFA mC/HcoHYNpw7R1/A9yPRFAeTxjhEyJpLU/PiE+o3m4kXSUuwoPpTuarTOe4iF6qoxzLCE3HZpPt +EA== X-Google-Smtp-Source: AGHT+IGSY6zrX4Qs7JrUctkrSdbtisepSykseFX5DdXlWxT8TG2QUbuQRw2CO7r/i/yRAh7feSGbV18ovH8= X-Received: from yuzhao2.bld.corp.google.com ([2a00:79e0:2e28:6:261c:802b:6b55:e09c]) (user=yuzhao job=sendgmr) by 2002:a05:6902:2b84:b0:e0b:cce3:45c7 with SMTP id 3f1490d57ef6-e0bde3ef075mr24855276.9.1722910881783; Mon, 05 Aug 2024 19:21:21 -0700 (PDT) Date: Mon, 5 Aug 2024 20:21:11 -0600 In-Reply-To: <20240806022114.3320543-1-yuzhao@google.com> Mime-Version: 1.0 References: <20240806022114.3320543-1-yuzhao@google.com> X-Mailer: git-send-email 2.46.0.rc2.264.g509ed76dc8-goog Message-ID: <20240806022114.3320543-2-yuzhao@google.com> Subject: [RFC PATCH 1/4] mm: HVO: introduce helper function to update and flush pgtable From: Yu Zhao To: Catalin Marinas , Will Deacon Cc: Andrew Morton , David Rientjes , Douglas Anderson , Frank van der Linden , Mark Rutland , Muchun Song , Nanyong Sun , Yang Shi , linux-arm-kernel@lists.infradead.org, linux-mm@kvack.org, linux-kernel@vger.kernel.org, Muchun Song , Yu Zhao X-Stat-Signature: dtkhpqanhmsmk6ompj5w68ctzn8skgtw X-Rspam-User: X-Rspamd-Queue-Id: CAACA10000B X-Rspamd-Server: rspam02 X-HE-Tag: 1722910882-537532 X-HE-Meta: U2FsdGVkX1+OZUi+jya3l77OACxF+pMtZDDo4GMYeTHMQiufBOESfHZ6UcaSG81CDWLMjaimyRMTWc1lLaVy0xCzbKAlL/4D9DqMZUXABAyrnrH/pyESH4b8Wkos4TsmA2S7rG3uxIBqd3Z36C02SwLISvD1HCaLiGFy3TWouQC6ijrkMAZHvAeFECslIPQpmV+0MyD4HZb+/+use0hJYyolTz9xZMCn2HyTkbCCN03j2nt5X0/KhL+iGqVTtPvb7tXun6bW4eGKtj9G8kP2+g9NyTEM6ImcXcUxhX6fhMfA9vlBH91BUuUQakgfSk6RmxR9sGfxz+Ptneym5vEhW5nixwFQKT2PFSfKLyj9eHLshOFuASR0gNpGcdVGV6rNI6Vq38A8CxB+X2AXb0BOwXLiPjf90Zg5rf3wZQxpDJU6S2CWJjliR0y4PFult7Xie5I1pLqceIyt3wP5Mp/8E4eFRkI6vQG/vzquAj1f6ZdovaHenqiwWEy4PKYslj6kZnYPXwF/2+sxVe6BebB0OjRhw9Rr3knKhxVl1hnMs1KudjQx/t/H3nB941163XmOYn6Qjjl4cLmELZICLeT+KnnssGr3DW5hZnit0QKdackEfFrPIh32m70leod65h0UN5gkHxPXEySCFYIoxAk8hDp5NBUk2NcaUT5e15cvrGmW/26vJoDHMVKdX4L5nVftHtsPz0h1eTtHI6N2P9SbAyJ+MKOU1I5V6d7SVkmFIO+jkl0stoCsrnP/90LYvmd4/PXflHRD1+MTYOfmbuF/nOrotEIMQb31k3q/44ewHDnymax8VFHglG5gJOkx/o1vvicPJxTh7y3X8RwhVG4SgD/I2sAPETjmkQ86fj42/qJshr/wkOQEV8TIleqvxk2KyxMgHoQ7g2X1KBN74FsICl1By/GFWH2jX3avEknI+IH38bvBPYlwXocbYtXCzY5gPcCjpfVY8fpw/qbQyDF csEt6BOn SGjK7WNxDxOzTYwgmT7KpobZQL7k5uq0gOsiVic8zLnGLuAPNuPZAM+2CZZvqe/qiHmHPpa91QuzoxcJu7GihpzyX28hcwP1pdWtS8GlYy1Rdj0NFdgMG8i3FZudu39VTAS7HycVXE3iESSofivr/h0Uglpgo+xJmiOamvR2LsQh+ROmEXWNcozpeP1sMWeZuYAGcWl1B50A5XxnKBEh9LNUXyDbl78SyfYk0yH6XuYOxYU99eKJq1LaMcpw0Su/UZsMUEwUfv/EgEgLjDaOL+Cz9mO5DVdVDKoAv4CWR+36psDTdJKDqlxU3s24QQRFEw3bDj9cPRbdszfp1OdW80oa5yn6hpjgImXp/YXEFkTKuVsnr4gbOGTjs78Tmrz50kMwwGOJB9zfdtWIbWrf+TeJKExU5900TKauylMGJSGjIPPqsa5S0bKhBKEwIkHl/yIzSRqvftABdb1qTapEh9mvUfm+GYfsNSS0jAG1oa5C2EzCwqKDGHSHg6oGaKgeplRx3vIj1/pPWXc7sV0u479ozeTMZnEES7ez1QNyrpvsexsQfZW/CeOodpdttJe6sZO/GZ+3POW6VozJ5Vdhg6SNuDQ== X-Bogosity: Ham, tests=bogofilter, spamicity=0.000000, version=1.2.4 Sender: owner-linux-mm@kvack.org Precedence: bulk X-Loop: owner-majordomo@kvack.org List-ID: List-Subscribe: List-Unsubscribe: From: Nanyong Sun Add pmd/pte update and tlb flush helper function to update page table. This refactoring patch is designed to facilitate each architecture to implement its own special logic in preparation for the arm64 architecture to follow the necessary break-before-make sequence when updating page tables. Signed-off-by: Nanyong Sun Reviewed-by: Muchun Song Signed-off-by: Yu Zhao --- mm/hugetlb_vmemmap.c | 55 ++++++++++++++++++++++++++++++++++---------- 1 file changed, 43 insertions(+), 12 deletions(-) diff --git a/mm/hugetlb_vmemmap.c b/mm/hugetlb_vmemmap.c index 829112b0a914..2dd92e58f304 100644 --- a/mm/hugetlb_vmemmap.c +++ b/mm/hugetlb_vmemmap.c @@ -46,6 +46,37 @@ struct vmemmap_remap_walk { unsigned long flags; }; +#ifndef vmemmap_update_pmd +static inline void vmemmap_update_pmd(unsigned long addr, + pmd_t *pmdp, pte_t *ptep) +{ + pmd_populate_kernel(&init_mm, pmdp, ptep); +} +#endif + +#ifndef vmemmap_update_pte +static inline void vmemmap_update_pte(unsigned long addr, + pte_t *ptep, pte_t pte) +{ + set_pte_at(&init_mm, addr, ptep, pte); +} +#endif + +#ifndef vmemmap_flush_tlb_all +static inline void vmemmap_flush_tlb_all(void) +{ + flush_tlb_all(); +} +#endif + +#ifndef vmemmap_flush_tlb_range +static inline void vmemmap_flush_tlb_range(unsigned long start, + unsigned long end) +{ + flush_tlb_kernel_range(start, end); +} +#endif + static int vmemmap_split_pmd(pmd_t *pmd, struct page *head, unsigned long start, struct vmemmap_remap_walk *walk) { @@ -81,9 +112,9 @@ static int vmemmap_split_pmd(pmd_t *pmd, struct page *head, unsigned long start, /* Make pte visible before pmd. See comment in pmd_install(). */ smp_wmb(); - pmd_populate_kernel(&init_mm, pmd, pgtable); + vmemmap_update_pmd(start, pmd, pgtable); if (!(walk->flags & VMEMMAP_SPLIT_NO_TLB_FLUSH)) - flush_tlb_kernel_range(start, start + PMD_SIZE); + vmemmap_flush_tlb_range(start, start + PMD_SIZE); } else { pte_free_kernel(&init_mm, pgtable); } @@ -171,7 +202,7 @@ static int vmemmap_remap_range(unsigned long start, unsigned long end, return ret; if (walk->remap_pte && !(walk->flags & VMEMMAP_REMAP_NO_TLB_FLUSH)) - flush_tlb_kernel_range(start, end); + vmemmap_flush_tlb_range(start, end); return 0; } @@ -220,15 +251,15 @@ static void vmemmap_remap_pte(pte_t *pte, unsigned long addr, /* * Makes sure that preceding stores to the page contents from - * vmemmap_remap_free() become visible before the set_pte_at() - * write. + * vmemmap_remap_free() become visible before the + * vmemmap_update_pte() write. */ smp_wmb(); } entry = mk_pte(walk->reuse_page, pgprot); list_add(&page->lru, walk->vmemmap_pages); - set_pte_at(&init_mm, addr, pte, entry); + vmemmap_update_pte(addr, pte, entry); } /* @@ -267,10 +298,10 @@ static void vmemmap_restore_pte(pte_t *pte, unsigned long addr, /* * Makes sure that preceding stores to the page contents become visible - * before the set_pte_at() write. + * before the vmemmap_update_pte() write. */ smp_wmb(); - set_pte_at(&init_mm, addr, pte, mk_pte(page, pgprot)); + vmemmap_update_pte(addr, pte, mk_pte(page, pgprot)); } /** @@ -536,7 +567,7 @@ long hugetlb_vmemmap_restore_folios(const struct hstate *h, } if (restored) - flush_tlb_all(); + vmemmap_flush_tlb_all(); if (!ret) ret = restored; return ret; @@ -664,7 +695,7 @@ void hugetlb_vmemmap_optimize_folios(struct hstate *h, struct list_head *folio_l break; } - flush_tlb_all(); + vmemmap_flush_tlb_all(); /* avoid writes from page_ref_add_unless() while folding vmemmap */ synchronize_rcu(); @@ -684,7 +715,7 @@ void hugetlb_vmemmap_optimize_folios(struct hstate *h, struct list_head *folio_l * allowing more vmemmap remaps to occur. */ if (ret == -ENOMEM && !list_empty(&vmemmap_pages)) { - flush_tlb_all(); + vmemmap_flush_tlb_all(); free_vmemmap_page_list(&vmemmap_pages); INIT_LIST_HEAD(&vmemmap_pages); __hugetlb_vmemmap_optimize_folio(h, folio, &vmemmap_pages, @@ -692,7 +723,7 @@ void hugetlb_vmemmap_optimize_folios(struct hstate *h, struct list_head *folio_l } } - flush_tlb_all(); + vmemmap_flush_tlb_all(); free_vmemmap_page_list(&vmemmap_pages); } From patchwork Tue Aug 6 02:21:12 2024 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Yu Zhao X-Patchwork-Id: 13754341 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from kanga.kvack.org (kanga.kvack.org [205.233.56.17]) by smtp.lore.kernel.org (Postfix) with ESMTP id 28EBBC3DA7F for ; Tue, 6 Aug 2024 02:21:28 +0000 (UTC) Received: by kanga.kvack.org (Postfix) id B50A36B00C1; Mon, 5 Aug 2024 22:21:27 -0400 (EDT) Received: by kanga.kvack.org (Postfix, from userid 40) id AD9AF6B00C2; Mon, 5 Aug 2024 22:21:27 -0400 (EDT) X-Delivered-To: int-list-linux-mm@kvack.org Received: by kanga.kvack.org (Postfix, from userid 63042) id 952B16B00C3; Mon, 5 Aug 2024 22:21:27 -0400 (EDT) X-Delivered-To: linux-mm@kvack.org Received: from relay.hostedemail.com (smtprelay0015.hostedemail.com [216.40.44.15]) by kanga.kvack.org (Postfix) with ESMTP id 7202B6B00C1 for ; Mon, 5 Aug 2024 22:21:27 -0400 (EDT) Received: from smtpin20.hostedemail.com (a10.router.float.18 [10.200.18.1]) by unirelay02.hostedemail.com (Postfix) with ESMTP id 2325D1208A3 for ; Tue, 6 Aug 2024 02:21:27 +0000 (UTC) X-FDA: 82420219014.20.5B2A147 Received: from mail-yw1-f201.google.com (mail-yw1-f201.google.com [209.85.128.201]) by imf02.hostedemail.com (Postfix) with ESMTP id 5C8EF80019 for ; Tue, 6 Aug 2024 02:21:25 +0000 (UTC) Authentication-Results: imf02.hostedemail.com; dkim=pass header.d=google.com header.s=20230601 header.b=f9G5LQF0; dmarc=pass (policy=reject) header.from=google.com; spf=pass (imf02.hostedemail.com: domain of 3pIixZgYKCLcvrweXldlldib.Zljifkru-jjhsXZh.lod@flex--yuzhao.bounces.google.com designates 209.85.128.201 as permitted sender) smtp.mailfrom=3pIixZgYKCLcvrweXldlldib.Zljifkru-jjhsXZh.lod@flex--yuzhao.bounces.google.com ARC-Seal: i=1; s=arc-20220608; d=hostedemail.com; t=1722910834; a=rsa-sha256; cv=none; b=BvtGgjaVHvfFdNunHtRvmJxwpKqNGSJpKsC8dqKKRd03Iz2GE1YXf7RQHzINvA5gxhj79J rrpsV3BD8kS/GQfB5SPJym3I4vgqkywdDZo+Eo0vMzW+09vGdQo7SfIB1mkocqqWiV7fHK iAl3XX3JXS7AjAPEiAPbwZthtDGj9a0= ARC-Authentication-Results: i=1; imf02.hostedemail.com; dkim=pass header.d=google.com header.s=20230601 header.b=f9G5LQF0; dmarc=pass (policy=reject) header.from=google.com; spf=pass (imf02.hostedemail.com: domain of 3pIixZgYKCLcvrweXldlldib.Zljifkru-jjhsXZh.lod@flex--yuzhao.bounces.google.com designates 209.85.128.201 as permitted sender) smtp.mailfrom=3pIixZgYKCLcvrweXldlldib.Zljifkru-jjhsXZh.lod@flex--yuzhao.bounces.google.com ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=hostedemail.com; s=arc-20220608; t=1722910834; h=from:from:sender:reply-to:subject:subject:date:date: message-id:message-id:to:to:cc:cc:mime-version:mime-version: content-type:content-type:content-transfer-encoding: in-reply-to:in-reply-to:references:references:dkim-signature; bh=O2mshx39uv2zkvLi95oIpi0tXJ9/QGv1zrpIuDCw18w=; b=1bgjzXDIYiDsxkbApF9Bwu3OuhdcQeDkJb9UK6OrQUp+l/XejWA2hFrV0huou24+5vM/Yv gbdwEgEXp6H/tI4ZMmnGeqLLVl/lmGa/Vx1wR1VAXO65fbYcbk/AXO4OnQurpIxN7VrIzH Cs2Vh4j33lc86zmIW5vZ2+LfYfDw6zE= Received: by mail-yw1-f201.google.com with SMTP id 00721157ae682-6643016423fso4214647b3.3 for ; Mon, 05 Aug 2024 19:21:25 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=google.com; s=20230601; t=1722910884; x=1723515684; darn=kvack.org; h=cc:to:from:subject:message-id:references:mime-version:in-reply-to :date:from:to:cc:subject:date:message-id:reply-to; bh=O2mshx39uv2zkvLi95oIpi0tXJ9/QGv1zrpIuDCw18w=; b=f9G5LQF0CHdgQR/dLOvxx3lT71UxTXlqbb2g25/Z/sW9EUtBXiVXjkyygnQvGQGAEM 3vWZeewjriAufEybuY2RVWOxtHN2I0Aw4nh2ODtnFjOxiKHFyuwbn7ADhWRQ5R/fXhdq XxQxh8YVShwB6fGPbA3Ex6lb3aWYgCPc6mcRr76phQS7zVULUmXH8nWZaCWEuelocOhP qdrax1eVCmUGlgH76FWKYEEYcab889flxFqNwQ+R3EXAUOVmOAJlxHwy6j6jYbgap5fq UPR2C3QZkJ0CT9UGxobdEnPdrn8obarpqKWFDtK6JHRvvlOPOKDFo4LTE415lz5L4PL6 2qUA== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20230601; t=1722910884; x=1723515684; h=cc:to:from:subject:message-id:references:mime-version:in-reply-to :date:x-gm-message-state:from:to:cc:subject:date:message-id:reply-to; bh=O2mshx39uv2zkvLi95oIpi0tXJ9/QGv1zrpIuDCw18w=; b=KJRWZ4TqJk9ODVLVe8FVCCHrFRL7xVZQF5OXO37S5nIjG38NXdY93eT87Ya1/B/PAH ZPIZqIRHpJaY2IO2hwXB4R1fPefiWxKs8cpC+zuADeRfdIg2Bs4T/dl9AE7+9Adsv17w fXDN8fwPNISuIn7nLwNXWmiUg9wC2f3E0C05pzZ8f5G7uYWZ0dvzaYh8v1dWSVpGXY4E JagnsAtgJ8eR6Jqj8a4BA2ZpiqvwqdY2eyIEKdZCxfhMv/S3rdFE9aJvOQC+xUxMbimo A2hcCrYEGcSlmqE0/lFjfvaz6aovLWEPh9wpliR5gp36JTsfLKc617z8SCOz7kXfu31h FkaA== X-Forwarded-Encrypted: i=1; AJvYcCV9GgzLJP0X2iaKBRoYHv2UFI5iu05ROlGiy3TID1Pobiegl9burUBE19B6Zy/w4ngjjfIjmZxQa7xgGtAFrnh8NRM= X-Gm-Message-State: AOJu0YwB3D44mLYJpnhrT+kQvXHXOM7/8fqXvodgrqvApX20nMqtQG+K 5Gw1R/8EAlvmykTodFsQbt6ihUyjWISXB852E6j3ZRxz7pVMYIonNIfkgLxvPmkXXuBLtqVncjf 3zQ== X-Google-Smtp-Source: AGHT+IG70ROskvb4vGAVk7HFhpn0VwxmpNbq5+lHHNoIlP/q1TqLemnSQjn2n7kwU3OW0QixZ2DVJNXlB8A= X-Received: from yuzhao2.bld.corp.google.com ([2a00:79e0:2e28:6:261c:802b:6b55:e09c]) (user=yuzhao job=sendgmr) by 2002:a05:690c:288:b0:673:b39a:9291 with SMTP id 00721157ae682-6895ffbd30dmr4163187b3.3.1722910884345; Mon, 05 Aug 2024 19:21:24 -0700 (PDT) Date: Mon, 5 Aug 2024 20:21:12 -0600 In-Reply-To: <20240806022114.3320543-1-yuzhao@google.com> Mime-Version: 1.0 References: <20240806022114.3320543-1-yuzhao@google.com> X-Mailer: git-send-email 2.46.0.rc2.264.g509ed76dc8-goog Message-ID: <20240806022114.3320543-3-yuzhao@google.com> Subject: [RFC PATCH 2/4] arm64: use IPIs to pause/resume remote CPUs From: Yu Zhao To: Catalin Marinas , Will Deacon Cc: Andrew Morton , David Rientjes , Douglas Anderson , Frank van der Linden , Mark Rutland , Muchun Song , Nanyong Sun , Yang Shi , linux-arm-kernel@lists.infradead.org, linux-mm@kvack.org, linux-kernel@vger.kernel.org, Yu Zhao X-Rspamd-Server: rspam12 X-Rspamd-Queue-Id: 5C8EF80019 X-Stat-Signature: mrrojqabnabk6mxx1serjw1f7yp9fzi8 X-Rspam-User: X-HE-Tag: 1722910885-481938 X-HE-Meta: U2FsdGVkX1+wnP9tkDwGW798nolkR1oYGlvtkUygDKbRg44XeP931lqXwp/ac1iZ5EPgD3P/MIllPIs9608BTn2XoHSJQxq3AYXyKooKmyIE2o2UErlHgFg0Q2BgYxVAQWTu4Z/8cYez2KSnT0shEmJOZMEFRr63Tr8v2twsbyyNzY49tTUF0QY03vN40jJia3ZMiFlFtGI0vPfsTcn9BMISXduXXZg1OBXkYMh5fJC4E1UPX17H/U3P0hBLPaCxAKSwmTa7Euz3Tm4V+r3sa/QaFMJjZ2sGRmv/63edWrUVtwomTWLgrYY845M5ABqhrIbpv/MVcZmR6qKWVILCfu5bheHkg2oIrTsxnqtTQDaEaPQejZtNGY6oRt49iJiJ5xrpRAGQJX3MTFwUEcuHOzCTf6LWALxbXWyb3ZPka/FoEq/HDSF0OXOeo8zfkk7Qd1ZG2Imsh1FsGsXq0CoeJlxHDzgR+Osix/UzZk2HQxnQjRBNeWN/wywgZj6gRxfO6GIeNFez3b0aqfptBIAE8I5kidqgE65RDk0nKV+lLTiOxL25L8xBqD+jv4LvoVT57pmejAtUU/9g1Ddx9qSV8GQr4C6YXMKzPB4OgEcyW8l1VE8H0m7ZStNdtJJFz+y2GmEZ1WNM2XmESqWrRq/pIWsdwN6RJECaUIdHvum0P0Qcm5CLpi6vImvlISKw90mMaY5eq7n+ZwSBHkt/t1+8457BjqLZlo/JZ50/D+0OXC6BeG8mVMMjLrl5OTtdbj67fqJiqK9IbQdHZ7pGpFNt/kXAPPhCDYDUkYjfk0BMJMVkVv2woxSReWg1YwnR9gQL23M56F4Pt9cW+7huBnEoCn/sqNkEHvRF4EIanSvZLkanD9ve13xapS8unq8H8krD+HTy+Vc7Uxlwpyc3UsLHAf1BYvZwg+zsYxySzhLYj5XYnBnglGnAX6SVkDw1ngq/PHRORuOqYwKJAqlHrXd 2WLqp9wc HcydP+/3HgWCpgWTjfkhfMxK6w0vNxs+wp+aPC+e8rjdBn3z2X7lYyjUM0d+Lmf0BYawkpx3mZ2W5NstCVxo5B3chlUA2e/aHZQHGnn5Ikiv0lo4b9aiQJjCyLFr2zgc0bCj3msWSdP8Mysom7TTIeax5Of5xnbcKvCQUF2u6IYx1J0YwgphipKHYgc5XkmNrPeeJXzXgiwg14+6zRjHrEFiJltgsXCv1nYAVcv03SwT5c/P8w/+I+AkxnoEiL3v/4pWZduDHU+J09g9uOoWA+Wmow8JUzd5rp9wVoKFsWwKuFBn+u+NTjQ932cB59bDzPDGROX7UOlA/ecyHj/KpSbysZ5u4BQU1Pbk+FeY7XZsR43cWs0iIYFWwklOBDevx1eKVj9ARlxRt97/nq036PweEY7fmETcQqmFchQG4c6dm+TkJWKPe/vSNj7iXgjimPFKr52nzAKWGa3EW2XdVgnh24hWMgQ5vyc9wpsBthjgw/nNW5ajoe68qmivCBi9SspZbAlIeF11LlOI= X-Bogosity: Ham, tests=bogofilter, spamicity=0.000000, version=1.2.4 Sender: owner-linux-mm@kvack.org Precedence: bulk X-Loop: owner-majordomo@kvack.org List-ID: List-Subscribe: List-Unsubscribe: Use pseudo-NMI IPIs to pause remote CPUs for a short period of time, and then reliably resume them when the local CPU exits critical sections that preclude the execution of remote CPUs. A typical example of such critical sections is BBM on kernel PTEs. HugeTLB Vmemmap Optimization (HVO) on arm64 was disabled by commit 060a2c92d1b6 ("arm64: mm: hugetlb: Disable HUGETLB_PAGE_OPTIMIZE_VMEMMAP") due to the folllowing reason: This is deemed UNPREDICTABLE by the Arm architecture without a break-before-make sequence (make the PTE invalid, TLBI, write the new valid PTE). However, such sequence is not possible since the vmemmap may be concurrently accessed by the kernel. Supporting BBM on kernel PTEs is one of the approaches that can potentially make arm64 support HVO. Signed-off-by: Yu Zhao --- arch/arm64/include/asm/smp.h | 3 + arch/arm64/kernel/smp.c | 110 +++++++++++++++++++++++++++++++++++ 2 files changed, 113 insertions(+) diff --git a/arch/arm64/include/asm/smp.h b/arch/arm64/include/asm/smp.h index 2510eec026f7..cffb0cfed961 100644 --- a/arch/arm64/include/asm/smp.h +++ b/arch/arm64/include/asm/smp.h @@ -133,6 +133,9 @@ bool cpus_are_stuck_in_kernel(void); extern void crash_smp_send_stop(void); extern bool smp_crash_stop_failed(void); +void pause_remote_cpus(void); +void resume_remote_cpus(void); + #endif /* ifndef __ASSEMBLY__ */ #endif /* ifndef __ASM_SMP_H */ diff --git a/arch/arm64/kernel/smp.c b/arch/arm64/kernel/smp.c index 5e18fbcee9a2..aa80266e5c9d 100644 --- a/arch/arm64/kernel/smp.c +++ b/arch/arm64/kernel/smp.c @@ -68,16 +68,25 @@ enum ipi_msg_type { IPI_RESCHEDULE, IPI_CALL_FUNC, IPI_CPU_STOP, + IPI_CPU_PAUSE, +#ifdef CONFIG_KEXEC_CORE IPI_CPU_CRASH_STOP, +#endif +#ifdef CONFIG_GENERIC_CLOCKEVENTS_BROADCAST IPI_TIMER, +#endif +#ifdef CONFIG_IRQ_WORK IPI_IRQ_WORK, +#endif NR_IPI, /* * Any enum >= NR_IPI and < MAX_IPI is special and not tracable * with trace_ipi_* */ IPI_CPU_BACKTRACE = NR_IPI, +#ifdef CONFIG_KGDB IPI_KGDB_ROUNDUP, +#endif MAX_IPI }; @@ -821,11 +830,20 @@ static const char *ipi_types[MAX_IPI] __tracepoint_string = { [IPI_RESCHEDULE] = "Rescheduling interrupts", [IPI_CALL_FUNC] = "Function call interrupts", [IPI_CPU_STOP] = "CPU stop interrupts", + [IPI_CPU_PAUSE] = "CPU pause interrupts", +#ifdef CONFIG_KEXEC_CORE [IPI_CPU_CRASH_STOP] = "CPU stop (for crash dump) interrupts", +#endif +#ifdef CONFIG_GENERIC_CLOCKEVENTS_BROADCAST [IPI_TIMER] = "Timer broadcast interrupts", +#endif +#ifdef CONFIG_IRQ_WORK [IPI_IRQ_WORK] = "IRQ work interrupts", +#endif [IPI_CPU_BACKTRACE] = "CPU backtrace interrupts", +#ifdef CONFIG_KGDB [IPI_KGDB_ROUNDUP] = "KGDB roundup interrupts", +#endif }; static void smp_cross_call(const struct cpumask *target, unsigned int ipinr); @@ -884,6 +902,85 @@ void __noreturn panic_smp_self_stop(void) local_cpu_stop(); } +static DEFINE_SPINLOCK(cpu_pause_lock); +static cpumask_t paused_cpus; +static cpumask_t resumed_cpus; + +static void pause_local_cpu(void) +{ + int cpu = smp_processor_id(); + + cpumask_clear_cpu(cpu, &resumed_cpus); + /* + * Paired with pause_remote_cpus() to confirm that this CPU not only + * will be paused but also can be reliably resumed. + */ + smp_wmb(); + cpumask_set_cpu(cpu, &paused_cpus); + /* A typical example for sleep and wake-up functions. */ + smp_mb(); + while (!cpumask_test_cpu(cpu, &resumed_cpus)) { + wfe(); + barrier(); + } + barrier(); + cpumask_clear_cpu(cpu, &paused_cpus); +} + +void pause_remote_cpus(void) +{ + cpumask_t cpus_to_pause; + + lockdep_assert_cpus_held(); + lockdep_assert_preemption_disabled(); + + cpumask_copy(&cpus_to_pause, cpu_online_mask); + cpumask_clear_cpu(smp_processor_id(), &cpus_to_pause); + + spin_lock(&cpu_pause_lock); + + WARN_ON_ONCE(!cpumask_empty(&paused_cpus)); + + smp_cross_call(&cpus_to_pause, IPI_CPU_PAUSE); + + while (!cpumask_equal(&cpus_to_pause, &paused_cpus)) { + cpu_relax(); + barrier(); + } + /* + * Paired with pause_local_cpu() to confirm that all CPUs not only will + * be paused but also can be reliably resumed. + */ + smp_rmb(); + WARN_ON_ONCE(cpumask_intersects(&cpus_to_pause, &resumed_cpus)); + + spin_unlock(&cpu_pause_lock); +} + +void resume_remote_cpus(void) +{ + cpumask_t cpus_to_resume; + + lockdep_assert_cpus_held(); + lockdep_assert_preemption_disabled(); + + cpumask_copy(&cpus_to_resume, cpu_online_mask); + cpumask_clear_cpu(smp_processor_id(), &cpus_to_resume); + + spin_lock(&cpu_pause_lock); + + cpumask_setall(&resumed_cpus); + /* A typical example for sleep and wake-up functions. */ + smp_mb(); + while (cpumask_intersects(&cpus_to_resume, &paused_cpus)) { + sev(); + cpu_relax(); + barrier(); + } + + spin_unlock(&cpu_pause_lock); +} + #ifdef CONFIG_KEXEC_CORE static atomic_t waiting_for_crash_ipi = ATOMIC_INIT(0); #endif @@ -963,6 +1060,11 @@ static void do_handle_IPI(int ipinr) local_cpu_stop(); break; + case IPI_CPU_PAUSE: + pause_local_cpu(); + break; + +#ifdef CONFIG_KEXEC_CORE case IPI_CPU_CRASH_STOP: if (IS_ENABLED(CONFIG_KEXEC_CORE)) { ipi_cpu_crash_stop(cpu, get_irq_regs()); @@ -970,6 +1072,7 @@ static void do_handle_IPI(int ipinr) unreachable(); } break; +#endif #ifdef CONFIG_GENERIC_CLOCKEVENTS_BROADCAST case IPI_TIMER: @@ -991,9 +1094,11 @@ static void do_handle_IPI(int ipinr) nmi_cpu_backtrace(get_irq_regs()); break; +#ifdef CONFIG_KGDB case IPI_KGDB_ROUNDUP: kgdb_nmicallback(cpu, get_irq_regs()); break; +#endif default: pr_crit("CPU%u: Unknown IPI message 0x%x\n", cpu, ipinr); @@ -1023,9 +1128,14 @@ static bool ipi_should_be_nmi(enum ipi_msg_type ipi) switch (ipi) { case IPI_CPU_STOP: + case IPI_CPU_PAUSE: +#ifdef CONFIG_KEXEC_CORE case IPI_CPU_CRASH_STOP: +#endif case IPI_CPU_BACKTRACE: +#ifdef CONFIG_KGDB case IPI_KGDB_ROUNDUP: +#endif return true; default: return false; From patchwork Tue Aug 6 02:21:13 2024 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Yu Zhao X-Patchwork-Id: 13754342 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from kanga.kvack.org (kanga.kvack.org [205.233.56.17]) by smtp.lore.kernel.org (Postfix) with ESMTP id B34FDC52D6F for ; Tue, 6 Aug 2024 02:21:30 +0000 (UTC) Received: by kanga.kvack.org (Postfix) id 425B26B00C3; Mon, 5 Aug 2024 22:21:30 -0400 (EDT) Received: by kanga.kvack.org (Postfix, from userid 40) id 3AF866B00C4; Mon, 5 Aug 2024 22:21:30 -0400 (EDT) X-Delivered-To: int-list-linux-mm@kvack.org Received: by kanga.kvack.org (Postfix, from userid 63042) id 1DB146B00C5; Mon, 5 Aug 2024 22:21:30 -0400 (EDT) X-Delivered-To: linux-mm@kvack.org Received: from relay.hostedemail.com (smtprelay0013.hostedemail.com [216.40.44.13]) by kanga.kvack.org (Postfix) with ESMTP id EA2606B00C3 for ; Mon, 5 Aug 2024 22:21:29 -0400 (EDT) Received: from smtpin10.hostedemail.com (a10.router.float.18 [10.200.18.1]) by unirelay09.hostedemail.com (Postfix) with ESMTP id 964F6804FC for ; Tue, 6 Aug 2024 02:21:29 +0000 (UTC) X-FDA: 82420219098.10.B58608E Received: from mail-yw1-f202.google.com (mail-yw1-f202.google.com [209.85.128.202]) by imf24.hostedemail.com (Postfix) with ESMTP id CE91018000F for ; Tue, 6 Aug 2024 02:21:27 +0000 (UTC) Authentication-Results: imf24.hostedemail.com; dkim=pass header.d=google.com header.s=20230601 header.b=PEtNHwlt; spf=pass (imf24.hostedemail.com: domain of 3poixZgYKCLkxtygZnfnnfkd.bnlkhmtw-lljuZbj.nqf@flex--yuzhao.bounces.google.com designates 209.85.128.202 as permitted sender) smtp.mailfrom=3poixZgYKCLkxtygZnfnnfkd.bnlkhmtw-lljuZbj.nqf@flex--yuzhao.bounces.google.com; dmarc=pass (policy=reject) header.from=google.com ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=hostedemail.com; s=arc-20220608; t=1722910825; h=from:from:sender:reply-to:subject:subject:date:date: message-id:message-id:to:to:cc:cc:mime-version:mime-version: content-type:content-type:content-transfer-encoding: in-reply-to:in-reply-to:references:references:dkim-signature; bh=Dz9+L9Nas5YdR+3aU4+X/tR5OCfJ0AKY3iAQMLv0S5k=; b=XisR+LWXpvV0MYFtL2pfZac+lEWNQbS86FGKj5S4Rxr+qocAl/IsW4hGf1/FGSQflYdx9T azj+r6UJIsb45iqmOrAcvn1q5HeT4HXQfPITvNkcaUd/rbfcSW5anxH5GnviI4/jcGOfHo JLBTQ8/iGLHj88DZUKC5xbbY1cbKp8E= ARC-Seal: i=1; s=arc-20220608; d=hostedemail.com; t=1722910825; a=rsa-sha256; cv=none; b=FCnEiwEfYAaWEc3HH/4j+JaiIcKfWpIEwPGaiESPs+ir+cgFSxCW6/eNIpWs4BsmTC9I2E RdpzR+W5zYgKQv/eFC2nidNf5jOq/rlX47rAJWgBWPtp9B1KtzRMe1XY4lZhTfHOGtnrYZ 71dLs2ICzi4zJcUG30Ohl8VYgULpK+o= ARC-Authentication-Results: i=1; imf24.hostedemail.com; dkim=pass header.d=google.com header.s=20230601 header.b=PEtNHwlt; spf=pass (imf24.hostedemail.com: domain of 3poixZgYKCLkxtygZnfnnfkd.bnlkhmtw-lljuZbj.nqf@flex--yuzhao.bounces.google.com designates 209.85.128.202 as permitted sender) smtp.mailfrom=3poixZgYKCLkxtygZnfnnfkd.bnlkhmtw-lljuZbj.nqf@flex--yuzhao.bounces.google.com; dmarc=pass (policy=reject) header.from=google.com Received: by mail-yw1-f202.google.com with SMTP id 00721157ae682-690404fd34eso534357b3.1 for ; Mon, 05 Aug 2024 19:21:27 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=google.com; s=20230601; t=1722910887; x=1723515687; darn=kvack.org; h=cc:to:from:subject:message-id:references:mime-version:in-reply-to :date:from:to:cc:subject:date:message-id:reply-to; bh=Dz9+L9Nas5YdR+3aU4+X/tR5OCfJ0AKY3iAQMLv0S5k=; b=PEtNHwlt9D0n7j//BgajQRwPbiCGfxhC495MUF/BUei51TzojKWz+g33/3qwgQOlk6 Z3gB8FcA9KZHXNjhe22iqQqvBEj0LyI0VD6giOh2A2T/YehkmwXxwPmqAeg+m52PD21H Y5D3EWAkpw4WsksueVJ2zeAMXH0vL/gtE9oU4BNxA4oO/KvrR+ErYdqiPqWIvqlGfD60 32felkPWW9L9RjIahksZ1y0bPm/fiToqAYOQDuWzOkXi+B95lDCXmCVpkpR1+uFPAYcb qqPvcO/dLYFN4jOFWhU764Dj26BtsAoW0spMu/e8RUFlXVXPHNdmaEmKwxNY6yTXfckq dVCA== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20230601; t=1722910887; x=1723515687; h=cc:to:from:subject:message-id:references:mime-version:in-reply-to :date:x-gm-message-state:from:to:cc:subject:date:message-id:reply-to; bh=Dz9+L9Nas5YdR+3aU4+X/tR5OCfJ0AKY3iAQMLv0S5k=; b=JZcd6jFDUgRKGkIRZin9GTgJckxg+7FYEPuI7Wcli0w94lcuQF5nCZc2f6OO9ZLFpp S72+nVT3VmbwNhGtoR7xv8+EboQAYwD6BAyYzdjx5amL8hPTq/gY0xiJ+pwSp5bDhhGM 6KIL14fsAqzXKMDI0YGwaRa/G+gT8PpptLJ/PAV6NYhcfkLw7nXvVZMJPzNgIJmKLFYA FFhW0OJ+XHyqneZtuIFd/2Ejtk5qpPrn5uns6JQ0ZUdog7ckUUSfmQHXm5ksOX/P4CYY iQBrCEz3lo5gfl6YgDHPT7JBi8AWQg5B/8cX+WaJStqmDwWkoydF0HKLo/4huPA966NA aH7A== X-Forwarded-Encrypted: i=1; AJvYcCVYhg85DHNx3TYXLC4+zxmLMhkFB69U8EOZah0+bVnwSUgKNi5C6mnWW9LeWKAQo/zAZo8E1RtW9m9w4u1m0xvb0ec= X-Gm-Message-State: AOJu0Yz1j97GEQyQjDZrxjp4xcN+9tCGAGb4K1jcDRse+XLd0lroF+bT HVccIxMtFAZVfpdPxVv/MSBZO7Im13pi7Vqe0W0JiPUMYxSPhVzPjDbT4Orfidyn4cjLuUQWtW1 RLw== X-Google-Smtp-Source: AGHT+IE5Ot4MCUehL2BALtzt0Nbbacrl8HCjMPwdvfemN8nL8qevC8oWRD8zmA2efNk4jGDFZ//D943lM9s= X-Received: from yuzhao2.bld.corp.google.com ([2a00:79e0:2e28:6:261c:802b:6b55:e09c]) (user=yuzhao job=sendgmr) by 2002:a05:690c:113:b0:648:3f93:61f2 with SMTP id 00721157ae682-689641a414emr5295867b3.6.1722910886865; Mon, 05 Aug 2024 19:21:26 -0700 (PDT) Date: Mon, 5 Aug 2024 20:21:13 -0600 In-Reply-To: <20240806022114.3320543-1-yuzhao@google.com> Mime-Version: 1.0 References: <20240806022114.3320543-1-yuzhao@google.com> X-Mailer: git-send-email 2.46.0.rc2.264.g509ed76dc8-goog Message-ID: <20240806022114.3320543-4-yuzhao@google.com> Subject: [RFC PATCH 3/4] arm64: pause remote CPUs to update vmemmap From: Yu Zhao To: Catalin Marinas , Will Deacon Cc: Andrew Morton , David Rientjes , Douglas Anderson , Frank van der Linden , Mark Rutland , Muchun Song , Nanyong Sun , Yang Shi , linux-arm-kernel@lists.infradead.org, linux-mm@kvack.org, linux-kernel@vger.kernel.org, Yu Zhao X-Stat-Signature: njyg7fp6m5sfwnxhgot3emj67t1okfu7 X-Rspamd-Queue-Id: CE91018000F X-Rspam-User: X-Rspamd-Server: rspam08 X-HE-Tag: 1722910887-703919 X-HE-Meta: U2FsdGVkX1+Vf3w+4g0NdHA4xAemJKPn8ks4uU0ErTswztnZbDByXmMeZ/j/JmEKgUBrqBN+LnuQbw1i3/jbYalS5uDSuoDSmUPxCZpwMkTv4YDQMy8cP4Ug8MZ6StrhJI/g4F+CcZSoWuYOQszNDMwx8mRl+OcTfAHrvSXspTQzb1GDkRtujhRKTo5zCy/4/xwvQL3IJCWhEOKHraOk3pe9kRwiIyICcdYsTsc35m0jNYGlIfCteRAZNuw92IXbOm9+OBSOLIUPfE0dHViBdEhNNmN5hoNl5FPkJl7HLfoTnCm2avZUBOATKBz5jE31WFQ4y/y6XA9ktRb88L1TzMPIwg5OP9NbeJ9R8ZC0DW5NwlNaIRvX04d67Lth28VdT+nZP5DFfKtuMTNwqITnV8S17XuLtBLGpRSvtXK+GLfux/OZewsIELFXDPuCy87B4FBQhuJgvrs3einCUhL2dqMY+1CljZJVUZzg2HCOrWYwJpuTfbk2VoOIK/DxOCMcklQJQa6GPMVpPiqTVxhFUOm6CZ7vBh264TUf21ym6m2LgfzBul8ukM7q4KLBUwpyXWy1wo0IOOWluuicQ4tPgnophSOkF6vr3Rn1EwYjTewcZNOJHNnku/nlIKhV1L8oqsgiobJyvDZI77DbuqkODmyyy/u7dib8I6HyumMR5dXMjAyfYf9ZgCx5No3fJdmoHQaxCNx8PeipF+QrHYocu1Sxq99du6sCaPphFUVh/EVQEFK6madXs3IpASQnJ858Jo2WpqLW2CbxL8F8UArT+I/DgJl6yL/7p9SnxF5WhtA8PNbUWE9F+DTwp5k4zIsnFAvI394/C93DUeH6Gm2eU2BK38YXVPEr1UdnncskoTmpsw3Y3zXMrE4tHG/FxlNMdVrpd/rZW9rXiZBUThXnt4NJkTdTqtSzD9XWrhk8xsq5rj/Ay33Nv6/jV5in+4bXrgWgdg78hYOOiV0NloQ Qlkr5ZfS J/kB4g82oC8YBBTNvOPUVRyAHH6banok2Ff/x6BXUyXZ9Uk+IYxAOriVou47GeVfkYJc94i3z/D+dRuLqnHA+w/1K41lgkDwEZ0TkZ3u8qYZx4EfcKi9ENiwQFVbs5IIRDvE3WF4yVRb3bS4B0lh+kAp2peFw0s08W7lhaJzzcCkCNKf23pSoZcIqS5lvmEHf3ZYM3OLZimqSCh+H97QgSd+tjH9ilJdEBdJ2NttAoVoUhlmK/D+hR+bVlg4otVudZdLbrPkiy1jpPkB+AiOprHNjlUg70Nhaa1cq/3ZgiRmFJjOrRHL8xqAE50Q82rLG87R9OLJo9HGg3hqu2xfmtNxhSserAcQhtfZQlqRkyuEYElOETq4vDaGvLlpGr3TBu0bRgBcpWDki6l43vAbGGixVOnfoNS/EbKrmEWO5cSc/z6vu3YKvZyXMMSRSgonorAm05emuf9VtFwIoS4DI4YZzQYLXd/Mw0t4i0wdteopMHadv6T4PLDDZbJUgqWkaiqM06LhW99ikHo43BjEsIZgwH3EH3IoIUrIiu1IDLJXaIOTFxndj7OJoBg== X-Bogosity: Ham, tests=bogofilter, spamicity=0.000000, version=1.2.4 Sender: owner-linux-mm@kvack.org Precedence: bulk X-Loop: owner-majordomo@kvack.org List-ID: List-Subscribe: List-Unsubscribe: Pause remote CPUs so that the local CPU can follow the proper BBM sequence to safely update the vmemmap mapping `struct page` areas. While updating the vmemmap, it is guaranteed that neither the local CPU nor the remote ones will access the `struct page` area being updated, and therefore they will not trigger kernel PFs. Signed-off-by: Yu Zhao --- arch/arm64/include/asm/pgalloc.h | 55 ++++++++++++++++++++++++++++++++ mm/hugetlb_vmemmap.c | 14 ++++++++ 2 files changed, 69 insertions(+) diff --git a/arch/arm64/include/asm/pgalloc.h b/arch/arm64/include/asm/pgalloc.h index 8ff5f2a2579e..1af1aa34a351 100644 --- a/arch/arm64/include/asm/pgalloc.h +++ b/arch/arm64/include/asm/pgalloc.h @@ -12,6 +12,7 @@ #include #include #include +#include #define __HAVE_ARCH_PGD_FREE #define __HAVE_ARCH_PUD_FREE @@ -137,4 +138,58 @@ pmd_populate(struct mm_struct *mm, pmd_t *pmdp, pgtable_t ptep) __pmd_populate(pmdp, page_to_phys(ptep), PMD_TYPE_TABLE | PMD_TABLE_PXN); } +#ifdef CONFIG_HUGETLB_PAGE_OPTIMIZE_VMEMMAP + +#define vmemmap_update_lock vmemmap_update_lock +static inline void vmemmap_update_lock(void) +{ + cpus_read_lock(); +} + +#define vmemmap_update_unlock vmemmap_update_unlock +static inline void vmemmap_update_unlock(void) +{ + cpus_read_unlock(); +} + +#define vmemmap_update_pte vmemmap_update_pte +static inline void vmemmap_update_pte(unsigned long addr, pte_t *ptep, pte_t pte) +{ + preempt_disable(); + pause_remote_cpus(); + + pte_clear(&init_mm, addr, ptep); + flush_tlb_kernel_range(addr, addr + PAGE_SIZE); + set_pte_at(&init_mm, addr, ptep, pte); + + resume_remote_cpus(); + preempt_enable(); +} + +#define vmemmap_update_pmd vmemmap_update_pmd +static inline void vmemmap_update_pmd(unsigned long addr, pmd_t *pmdp, pte_t *ptep) +{ + preempt_disable(); + pause_remote_cpus(); + + pmd_clear(pmdp); + flush_tlb_kernel_range(addr, addr + PMD_SIZE); + pmd_populate_kernel(&init_mm, pmdp, ptep); + + resume_remote_cpus(); + preempt_enable(); +} + +#define vmemmap_flush_tlb_all vmemmap_flush_tlb_all +static inline void vmemmap_flush_tlb_all(void) +{ +} + +#define vmemmap_flush_tlb_range vmemmap_flush_tlb_range +static inline void vmemmap_flush_tlb_range(unsigned long start, unsigned long end) +{ +} + +#endif /* CONFIG_HUGETLB_PAGE_OPTIMIZE_VMEMMAP */ + #endif diff --git a/mm/hugetlb_vmemmap.c b/mm/hugetlb_vmemmap.c index 2dd92e58f304..893c73493d9c 100644 --- a/mm/hugetlb_vmemmap.c +++ b/mm/hugetlb_vmemmap.c @@ -46,6 +46,18 @@ struct vmemmap_remap_walk { unsigned long flags; }; +#ifndef vmemmap_update_lock +static void vmemmap_update_lock(void) +{ +} +#endif + +#ifndef vmemmap_update_unlock +static void vmemmap_update_unlock(void) +{ +} +#endif + #ifndef vmemmap_update_pmd static inline void vmemmap_update_pmd(unsigned long addr, pmd_t *pmdp, pte_t *ptep) @@ -194,10 +206,12 @@ static int vmemmap_remap_range(unsigned long start, unsigned long end, VM_BUG_ON(!PAGE_ALIGNED(start | end)); + vmemmap_update_lock(); mmap_read_lock(&init_mm); ret = walk_page_range_novma(&init_mm, start, end, &vmemmap_remap_ops, NULL, walk); mmap_read_unlock(&init_mm); + vmemmap_update_unlock(); if (ret) return ret; From patchwork Tue Aug 6 02:21:14 2024 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Yu Zhao X-Patchwork-Id: 13754343 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from kanga.kvack.org (kanga.kvack.org [205.233.56.17]) by smtp.lore.kernel.org (Postfix) with ESMTP id 03524C3DA7F for ; Tue, 6 Aug 2024 02:21:33 +0000 (UTC) Received: by kanga.kvack.org (Postfix) id 8B9456B00C5; Mon, 5 Aug 2024 22:21:32 -0400 (EDT) Received: by kanga.kvack.org (Postfix, from userid 40) id 841606B00C6; Mon, 5 Aug 2024 22:21:32 -0400 (EDT) X-Delivered-To: int-list-linux-mm@kvack.org Received: by kanga.kvack.org (Postfix, from userid 63042) id 6E46A6B00C7; Mon, 5 Aug 2024 22:21:32 -0400 (EDT) X-Delivered-To: linux-mm@kvack.org Received: from relay.hostedemail.com (smtprelay0016.hostedemail.com [216.40.44.16]) by kanga.kvack.org (Postfix) with ESMTP id 445866B00C5 for ; Mon, 5 Aug 2024 22:21:32 -0400 (EDT) Received: from smtpin07.hostedemail.com (a10.router.float.18 [10.200.18.1]) by unirelay10.hostedemail.com (Postfix) with ESMTP id 098D6C1E80 for ; Tue, 6 Aug 2024 02:21:32 +0000 (UTC) X-FDA: 82420219224.07.1A60FB5 Received: from mail-yw1-f202.google.com (mail-yw1-f202.google.com [209.85.128.202]) by imf14.hostedemail.com (Postfix) with ESMTP id 36DFF100008 for ; Tue, 6 Aug 2024 02:21:30 +0000 (UTC) Authentication-Results: imf14.hostedemail.com; dkim=pass header.d=google.com header.s=20230601 header.b=2Wd54qYh; spf=pass (imf14.hostedemail.com: domain of 3qYixZgYKCLw0w1jcqiqqing.eqonkpwz-oomxcem.qti@flex--yuzhao.bounces.google.com designates 209.85.128.202 as permitted sender) smtp.mailfrom=3qYixZgYKCLw0w1jcqiqqing.eqonkpwz-oomxcem.qti@flex--yuzhao.bounces.google.com; dmarc=pass (policy=reject) header.from=google.com ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=hostedemail.com; s=arc-20220608; t=1722910882; h=from:from:sender:reply-to:subject:subject:date:date: message-id:message-id:to:to:cc:cc:mime-version:mime-version: content-type:content-type:content-transfer-encoding: in-reply-to:in-reply-to:references:references:dkim-signature; bh=klWRv7jv+zR9aL4I7OIDRLjBsHfroErcwBIt7k9CqK8=; b=oraDzgjTktdwD23OxSp7EQ2dgSRKJzYSCm2O1NqCialAg3LwOyAVDYThdhx+w2XhQpyTtH 4w5OzJNoZUOkzhS2L66tizPQ2xHdQ1Yz8bgPrbwxcWTfrTs9lazbbahCF+sOGWcS9dRcin /2WkRXGg7OUEUwPsfJq0CdTrO/TJiUI= ARC-Authentication-Results: i=1; imf14.hostedemail.com; dkim=pass header.d=google.com header.s=20230601 header.b=2Wd54qYh; spf=pass (imf14.hostedemail.com: domain of 3qYixZgYKCLw0w1jcqiqqing.eqonkpwz-oomxcem.qti@flex--yuzhao.bounces.google.com designates 209.85.128.202 as permitted sender) smtp.mailfrom=3qYixZgYKCLw0w1jcqiqqing.eqonkpwz-oomxcem.qti@flex--yuzhao.bounces.google.com; dmarc=pass (policy=reject) header.from=google.com ARC-Seal: i=1; s=arc-20220608; d=hostedemail.com; t=1722910882; a=rsa-sha256; cv=none; b=RJWPABXnHL3Z/4MwYNV1cJf44d04Ygb6JtXLb1O46bXp8iAUXrsnO2OLuLpT8wUl9e2LMw McEN5+wf7njBqYZlOoqGYbUNTjPnGoiXGG1GLB9S5G0ADGINzYQkTU6KSfn5JchaDM6PZn OZ8O8PtWwb0yeaUFM0LAy+ghRgk7bQs= Received: by mail-yw1-f202.google.com with SMTP id 00721157ae682-650ab31aabdso3619367b3.3 for ; Mon, 05 Aug 2024 19:21:29 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=google.com; s=20230601; t=1722910889; x=1723515689; darn=kvack.org; h=cc:to:from:subject:message-id:references:mime-version:in-reply-to :date:from:to:cc:subject:date:message-id:reply-to; bh=klWRv7jv+zR9aL4I7OIDRLjBsHfroErcwBIt7k9CqK8=; b=2Wd54qYhHT4kyGZY/fBH3PUqDUGtVzmyzXbuxAwzWIBI8Ctx1G8wVkKxDAJL8g0G2P l2ln9KZU0R+E1byc5sR1irs3DpOO+dmxHKLgnf/dBjF1zpowTkjX5JS0QvoTw3ZZmgBJ xCkYD7sDprBIExscEIKLO54mM2DVTM33hM4PI87cnaELIOQyjVthIu/8jIZHdZX86mdR Hc/LEdJe5w6A57jMc/ViPijd+AAS7Xpr7JGUoJVCgRTGvrF10Us6wkIBfNVDXr/vaXf6 A4S03ZB59w2m/k4F8DBNJt5KheNLGk8U0hhw9BmK6fuSEdDkPHWHaLTjQTl9eEoXurbl uZ9g== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20230601; t=1722910889; x=1723515689; h=cc:to:from:subject:message-id:references:mime-version:in-reply-to :date:x-gm-message-state:from:to:cc:subject:date:message-id:reply-to; bh=klWRv7jv+zR9aL4I7OIDRLjBsHfroErcwBIt7k9CqK8=; b=J+Bpx/i7AS2NJq19++r51xF+ZZqHxyjx2D/Mmy5q85mKpkpRomnKIv7UVLi92qAWM/ qwg7Kby1b+7AYUan3dSz0551WnpBhXIl4WNy2HG++Ja+WHdBxz8ZCWC2YTfeXMIEg+hw F4q57HViHkmYxYqCSU7xETeDh1DqZv2avjuy8OLjPmKdchkrePygR5YmDlSAX2ARMmug nQ/EJt+AM74hN2szmCiA3kvsYSPXV0s6cfvVRGV/dLIUOZDG2zgO6l2rcX+ArTjm2g4O 06a22pNNd4/ZNTWpS72jx1LReSxdleerWnTRvSlU9HWR4grzEsjg/iKDw6S6tmuBcBX1 4A1A== X-Forwarded-Encrypted: i=1; AJvYcCW6riluHlxlXh22nd0XV6G5TMZa4W8giFuEKLrOaEyI1fzh8nL3ZD0//Sam9i7pxLuP0UgCkOxKKznKWsr0HvEFFUE= X-Gm-Message-State: AOJu0Yxb3uPBZeM9Kk4/N7Dlk4eESd9LChaLieSMCmVZDEXnwoiT9ZPl 4VbcGFOXSeAS1qiJS/eVk+tVYg2VXmQItV49QR6uyGL6Q7miy7O71wh8RRzeyCqIROkdeRd4tcO lIA== X-Google-Smtp-Source: AGHT+IGBeIkNUsFUwrlBhpMmppS6FEO5hkHGCy4Rnvwll+98qZLKAoOlt7677TqFz3cp98ge6KT5O3g57gw= X-Received: from yuzhao2.bld.corp.google.com ([2a00:79e0:2e28:6:261c:802b:6b55:e09c]) (user=yuzhao job=sendgmr) by 2002:a05:6902:2b90:b0:e05:eccb:95dc with SMTP id 3f1490d57ef6-e0bde283222mr64471276.6.1722910889235; Mon, 05 Aug 2024 19:21:29 -0700 (PDT) Date: Mon, 5 Aug 2024 20:21:14 -0600 In-Reply-To: <20240806022114.3320543-1-yuzhao@google.com> Mime-Version: 1.0 References: <20240806022114.3320543-1-yuzhao@google.com> X-Mailer: git-send-email 2.46.0.rc2.264.g509ed76dc8-goog Message-ID: <20240806022114.3320543-5-yuzhao@google.com> Subject: [RFC PATCH 4/4] arm64: mm: Re-enable OPTIMIZE_HUGETLB_VMEMMAP From: Yu Zhao To: Catalin Marinas , Will Deacon Cc: Andrew Morton , David Rientjes , Douglas Anderson , Frank van der Linden , Mark Rutland , Muchun Song , Nanyong Sun , Yang Shi , linux-arm-kernel@lists.infradead.org, linux-mm@kvack.org, linux-kernel@vger.kernel.org, Muchun Song , Yu Zhao X-Rspam-User: X-Stat-Signature: shyfyqf3w6nqbj17kk35od5dbh874wyk X-Rspamd-Queue-Id: 36DFF100008 X-Rspamd-Server: rspam11 X-HE-Tag: 1722910890-329675 X-HE-Meta: U2FsdGVkX1/asUcVmfNTiFrX6go58BL6h/QJ+BbZkisY15R2mvbXuhKFfZDWfJrLXYZdtiXXTAwzuKOeKO7ZF/Ps9jcRp4sM8yaWIY5/NxDot2qgBRYoQOSTT6/xfMWjffmuvjjuSFYoZPBS899fHsLj06lNAlQumkgNDNrGrBn6ewlF8mqbIyeb0bqQkQpP+gb9b3v7SfU7U6dGypdoCTsplFZSu6e8gtNopYp50GwLREmnuL9tq+KJorEU54qIwW+SSU+MiPzEuO+m9q+72U3tHRp1cTfmdZZqK6x5YGt8hNK4OVpuEEbpVc7xpLN3uOz4gt/aXNwgk92BMP1qDOx4EJ72Aa0PjPvOeBYSBICdbPtoRJ2VJIJuo4hOnVNRyFlGJY+YZytTb1gLOUUsygY4Viq1/vHLHkYoyb16nb7S5zbT1Wt0LSKocLGKkX4FygPayceiNF1nkosgiu6iI3zwXmsgspqQe3fvpwhXCrQSzSS7vvjAiUxDWqKx33+OjKs20BZn34DfJj4uYkgNMUUDooctD2BviYRrUpICcy+ABzTBR+SV8mqUsxiMXNJbaRXSK2anQjKBp4J7bq8Oh4gnUhaOXLbgukf4I9AvaQQtRDjk4ldigeW6cbKSe2VsOYzM0H6N32N5qPiXUveVDHuHi8LxWaFBGwdMkfiliCEwPYURM9y02eNOymTzkhjRWgvQqRdxhqQav9fi1lhKwYmsfRhK/EvnhOnn07LEOk/iOeqLbjGQiPP8luF1pw4jCTDGRblULk+EGjKN82iedBavmVnT+SzF5zJgJT/nQADtEaTKk/zPbi+2SmC2oRzzSLB4x5pJs6pGAMi2+WDJfmXdHZyl04XO1Zqdykq2LAjUsgK5TdjFDkaRwfyHzfMaHCBRpnRIDmSK7eFX8gnFtWLwJsGOK5p/U2HBOAjWid9wmKqbc8UZvDGqEJ5wMCZKCOW+fu4N0E7IUH2LiW2 1wI8K+wr Y1MMFxJ3Mt/r+Jv20LXaWGI2pW0kqWyAGVcisoMdSluLwz6bkpILG52q4+0HogF4dhNQEwKk2MMam1JL+KNh3CcT8NiaN4i+wi9wFsInWZNdvkjaxoskCWj9X/VbkqVVoPkPbTTay6E3bFxX9AlKTogXXp9YSvz3lYQMF4gzOTyZBtcioRsugiHUxX39WaKHQ7fpyXoTMDyDxxhEz+R6qnvOjwe4QfGbWqwQ/IKzcv+JnG7WNJwlFlOJ2aDFbb15o4+mCSHirm9ej9VdtxTgVrC/aJjiJaRGybx/ir06c+ST0MrD7kpKGSk8v6439WqkgvxHrynjT2Lg0ngLuyTbVZCc4TnB/vEQ4jbxce8aaL0SJqqyRngA0n82p1LzuQdwk3bFUtX8JYHGZOVeOqhw3i/GBPmmhQdj8ZFKheJmNjhadvFW4bXWAwcCM4Cqac+zBjRYLy9DLD6dm9rzqbXXX/Qjsp1vVPIm6hw9euPP++tTr+LqkOefbyUyAl+Eg/evB+U/nAm1OlcFWwJF8q/NkDMafrXJTcFhnPbe60J0iqbCqCrDEq5lMaFbU7tyZqTgOJgMSLQXb3M+W/oeqExRcpA4N0ioCRo7tmdT3twA9l6rJTzc2FHfMBlcFe8xn0IrJyGv3 X-Bogosity: Ham, tests=bogofilter, spamicity=0.000014, version=1.2.4 Sender: owner-linux-mm@kvack.org Precedence: bulk X-Loop: owner-majordomo@kvack.org List-ID: List-Subscribe: List-Unsubscribe: From: Nanyong Sun Now update of vmemmap page table can follow the rule of break-before-make safely for arm64 architecture, re-enable HVO on arm64. Signed-off-by: Nanyong Sun Reviewed-by: Muchun Song Signed-off-by: Yu Zhao --- arch/arm64/Kconfig | 1 + 1 file changed, 1 insertion(+) diff --git a/arch/arm64/Kconfig b/arch/arm64/Kconfig index a2f8ff354ca6..25ff026cdaf5 100644 --- a/arch/arm64/Kconfig +++ b/arch/arm64/Kconfig @@ -105,6 +105,7 @@ config ARM64 select ARCH_WANT_DEFAULT_TOPDOWN_MMAP_LAYOUT select ARCH_WANT_FRAME_POINTERS select ARCH_WANT_HUGE_PMD_SHARE if ARM64_4K_PAGES || (ARM64_16K_PAGES && !ARM64_VA_BITS_36) + select ARCH_WANT_OPTIMIZE_HUGETLB_VMEMMAP select ARCH_WANT_LD_ORPHAN_WARN select ARCH_WANTS_EXECMEM_LATE if EXECMEM select ARCH_WANTS_NO_INSTR