From patchwork Mon Oct 21 04:22:13 2024 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Yu Zhao X-Patchwork-Id: 13843540 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from kanga.kvack.org (kanga.kvack.org [205.233.56.17]) by smtp.lore.kernel.org (Postfix) with ESMTP id 5F5EBD3C93E for ; Mon, 21 Oct 2024 04:22:29 +0000 (UTC) Received: by kanga.kvack.org (Postfix) id B9ADA6B0083; Mon, 21 Oct 2024 00:22:28 -0400 (EDT) Received: by kanga.kvack.org (Postfix, from userid 40) id B71686B0089; Mon, 21 Oct 2024 00:22:28 -0400 (EDT) X-Delivered-To: int-list-linux-mm@kvack.org Received: by kanga.kvack.org (Postfix, from userid 63042) id 99F9D6B0088; Mon, 21 Oct 2024 00:22:28 -0400 (EDT) X-Delivered-To: linux-mm@kvack.org Received: from relay.hostedemail.com (smtprelay0012.hostedemail.com [216.40.44.12]) by kanga.kvack.org (Postfix) with ESMTP id 7C6D86B0082 for ; Mon, 21 Oct 2024 00:22:28 -0400 (EDT) Received: from smtpin01.hostedemail.com (a10.router.float.18 [10.200.18.1]) by unirelay03.hostedemail.com (Postfix) with ESMTP id F0AF3A145C for ; Mon, 21 Oct 2024 04:22:00 +0000 (UTC) X-FDA: 82696312104.01.75EE3DE Received: from mail-yw1-f201.google.com (mail-yw1-f201.google.com [209.85.128.201]) by imf24.hostedemail.com (Postfix) with ESMTP id D12FC18000C for ; Mon, 21 Oct 2024 04:22:23 +0000 (UTC) Authentication-Results: imf24.hostedemail.com; dkim=pass header.d=google.com header.s=20230601 header.b="MXjg+g2/"; spf=pass (imf24.hostedemail.com: domain of 3AdcVZwYKCEY627piwowwotm.kwutqv25-uus3iks.wzo@flex--yuzhao.bounces.google.com designates 209.85.128.201 as permitted sender) smtp.mailfrom=3AdcVZwYKCEY627piwowwotm.kwutqv25-uus3iks.wzo@flex--yuzhao.bounces.google.com; dmarc=pass (policy=reject) header.from=google.com ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=hostedemail.com; s=arc-20220608; t=1729484398; h=from:from:sender:reply-to:subject:subject:date:date: message-id:message-id:to:to:cc:cc:mime-version:mime-version: content-type:content-type:content-transfer-encoding: in-reply-to:in-reply-to:references:references:dkim-signature; bh=KalWPiT5Cmq0awZpJ2ht7HZzxnfLYA4lEUcwS3DKUno=; b=A+LPPsHzkhCMtkZEdwOMiVBwDdMoxwVIfBqGMnpyc4mBUuvhMNRDVD2tFC4rrbGrJ3somN KJNZ8bKxlV9OraazDRBlpFQamhZzZVPwIH4nZiOLji3+TbLP3Lr6YSdg5qUCU+X4EUHGJg qr5fQDmsHLmA1RuhC05SvFEQKR/hN34= ARC-Seal: i=1; s=arc-20220608; d=hostedemail.com; t=1729484398; a=rsa-sha256; cv=none; b=DNWjT+0jtOHQpG4eV3xegamCkbtWmzldkEKmUrq3gfeRzbtzKyPZOnFHTmG6UcRxsXDPu2 GGu4zEMDTmcVxYK3xTZv0eJk9udLfxJo2aGgbeezpNVc0AS6qgHMfvEDpXFiipgcZ6USA9 WmIPRvsjsCc/sDkNrTEBnkR1Qw8LFAA= ARC-Authentication-Results: i=1; imf24.hostedemail.com; dkim=pass header.d=google.com header.s=20230601 header.b="MXjg+g2/"; spf=pass (imf24.hostedemail.com: domain of 3AdcVZwYKCEY627piwowwotm.kwutqv25-uus3iks.wzo@flex--yuzhao.bounces.google.com designates 209.85.128.201 as permitted sender) smtp.mailfrom=3AdcVZwYKCEY627piwowwotm.kwutqv25-uus3iks.wzo@flex--yuzhao.bounces.google.com; dmarc=pass (policy=reject) header.from=google.com Received: by mail-yw1-f201.google.com with SMTP id 00721157ae682-6e1fbe2a6b1so70600117b3.2 for ; Sun, 20 Oct 2024 21:22:26 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=google.com; s=20230601; t=1729484545; x=1730089345; darn=kvack.org; h=cc:to:from:subject:message-id:references:mime-version:in-reply-to :date:from:to:cc:subject:date:message-id:reply-to; bh=KalWPiT5Cmq0awZpJ2ht7HZzxnfLYA4lEUcwS3DKUno=; b=MXjg+g2/6p0EHl3uZTAuVCGDsc9dt3WICOC0Mgr8TIQ/MGHXeVFMRhaMSDKGDK29Hs UoQ97scqCPrBdJwPUrW6Br5RCcYR/eX3P5bqphTU9B/k8HOZ4qfXxE2UZE2TuXwqqTNs 3gyK2vH9CWRjtwlMnlOU0BvWFie1XI/hFf8dpbhruSHAv2iNjoxSGw+8hmOtmSNgX8vh iMp56h4Qmeh3C/tD9XUde8DCAfZ0y2T0U+vy+MB9GWy9G5w4XforMEmZLCB5GbInl8G5 tW8l4okUMQUEWzxlqPSOTfuj+TBEPATZ/2afwgbbkP+kKr98bckbTBHqac3JUHARBgbJ 6NsA== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20230601; t=1729484545; x=1730089345; h=cc:to:from:subject:message-id:references:mime-version:in-reply-to :date:x-gm-message-state:from:to:cc:subject:date:message-id:reply-to; bh=KalWPiT5Cmq0awZpJ2ht7HZzxnfLYA4lEUcwS3DKUno=; b=LMX915dSvMGZu+9ENxHmQeSWqru+9GV5Am8QSw2D9hwI+EOrRZ+c0zxon6QM3AHTbH bR/VjdNbVDpZK0WMWt71gm/9DteKhVjvLUGoqSLH29LPPQ34yBCc2XtUDap3z+M6Xm8V TDO3D4hwwlRABhlWb/yi+0YD8xV3aDERxB3WIjoyW9fZNLuz7wqluhrt98zYLaopOaiv xkkPzhKY+dKGEiQ06YFrF7HXh6Cck/RW/Txsj8KOlpm6x1gMIazlJJyD21m79oJTGNu9 XR+ANM05VaW6MTVFVIc/DlH2QodgM/w0j1WmKqJcYnNA2ZND9WDh3/GoCQO1MaQ8rqxq EhzA== X-Forwarded-Encrypted: i=1; AJvYcCWh4fdYzXHJ4kA4Ou+gHxACn5OE4ai9ovO3vCOEd5JqYdMlmePhjD1nX1H3ohqhyu42QLja/JKNpw==@kvack.org X-Gm-Message-State: AOJu0Yxf9G6KBMVl3El6XxK9SIVcH1k+jwJr3/vZHjcGqa9A2ikyS16p BasRq18nyQBejDR0vCQ4dMVOzBhAQmjiigerz1m/jF4BSF/ei8HvfTDFtqxE6dVOduMNaZBPG/I KZA== X-Google-Smtp-Source: AGHT+IH2Ur7HHiXBaY4bDEA69MOZ2p1T90XrFcEHYsQ0uSyMb8pWEEibt70U8jRDqfM7NgoMPydl9BV9rAw= X-Received: from yuzhao2.bld.corp.google.com ([2a00:79e0:2e28:6:1569:9ef4:20ab:abf9]) (user=yuzhao job=sendgmr) by 2002:a25:dc41:0:b0:e25:caa3:9c2e with SMTP id 3f1490d57ef6-e2bb16d9206mr22332276.11.1729484545011; Sun, 20 Oct 2024 21:22:25 -0700 (PDT) Date: Sun, 20 Oct 2024 22:22:13 -0600 In-Reply-To: <20241021042218.746659-1-yuzhao@google.com> Mime-Version: 1.0 References: <20241021042218.746659-1-yuzhao@google.com> X-Mailer: git-send-email 2.47.0.rc1.288.g06298d1525-goog Message-ID: <20241021042218.746659-2-yuzhao@google.com> Subject: [PATCH v1 1/6] mm/hugetlb_vmemmap: batch update PTEs From: Yu Zhao To: Andrew Morton , Catalin Marinas , Marc Zyngier , Muchun Song , Thomas Gleixner , Will Deacon Cc: Douglas Anderson , Mark Rutland , Nanyong Sun , linux-arm-kernel@lists.infradead.org, linux-kernel@vger.kernel.org, linux-mm@kvack.org, Yu Zhao X-Rspamd-Queue-Id: D12FC18000C X-Stat-Signature: qqekg4py4rhhmw43qodsrh99pa611h6e X-Rspamd-Server: rspam09 X-Rspam-User: X-HE-Tag: 1729484543-195516 X-HE-Meta: U2FsdGVkX18I84GVENTMxSAoD4fEI9Rrq8epJA7WXjHgxLh1aC0mPEmgeLFIGsqk12Nf8B1iCcEIdskuxCgJcIeTB2rjmKCAJyHucBCnTTtHIa+9tJnxaMguImq+vpql94yJxvI7Gc7LQkTZCYtEgIpPrElZp/XRld2JtS6JMKXHuJVE6SrzvWkN1Vb0MNUZ8h5fPZa+tf439GGMBo9k1Kyd7gMO22uIu9Z47vM1PFIEqIt2TpX09wfnQp/679AfZtjmjz3fng903yGEVZaCOt9GLu6dPpC6TvwfABiIDZjbn9exvp6aTw+j+thH1g+t/vowd38UKyNERDXFp1Y3ajhHVx7tdyqm8MyCPXN94oQ7x+MUGVDs6AowSpUh6sayqdUCXVjLahH644rQnZN0nTanzxX0LrinLy/oI37DOrbZLGIa5dKWkjhupOMgCqb8cD4pKTBXAeK2Ujqd682pfPUqWl91WWu+rnl5feOLlBmziMU0iOMqVGTiZY+Kn5No0Uc/44nkOhdKoD+UZ6gbiMDIu7yeZvBcDfOKPYQQABVAnYE5kstmHLuR0ZnsbsL9ONntaRiMVv9suMa7oRvuB0N7OnMI3ZM2BGtcTwC8WPMG83J9MXP85+P4v2ek4YdWEvqupx1zW+ZOjOfLAZR47qHuWPDoROQctgRXsgaZ8uTrLIWSUzd9CD9QFOXRngpaqXG/B28ZEYmRPWs3fRTxNBIHkRJpJ/iXzEabijK6b71Odi1yf/bFvV4AjZXg3UiDS/UocOubkJmKaE9zpNF4TLkfGynmodhazSTq7W8tCz9dfLuNq5Kefj69Rv4R8HeSCTl5NUn15CxQH24+bZCx41d0j5mjtqYbZGOqEZPRRiKZrKZJnJUYav3sTUBRuE+inNW/n5g4jB4rmXejU2MRL0gMPq2bEIsIsRvOIOn/Dm+ekMyP422ngSRo5NmrCj366Cm5HjMxOCFcY+EUTWr ++sdMVAE R576NptPMbfzjmEeUBSOtAkohqV9OK1bQrMLUvbArDPnWFvuwTJ/u0WagbYOll+F0CqI9Tl4ZBMka+MuLY4FKFL+ZctHplkCYxxx95fwHgr7AEQyAJ6JKhq+l3o4YlQHZKThOATtkKbsUMeYCxMxb/1tr3rqFLYz4i4rI5WGYdKWnja7l+1JuuNAw1W01nfCniOHfDOxDwrFrrdNtccf9heOjzeHDu+qQT7amkHjlZ12tM1LPOqIsGdbEaR8n/lUUr77atN8Y4xaIOeQNDaKZR7zZSlbEEBcdFIszNsPYywiXSdHzpIONYU31vCKw0Fu+3fhwq2Cy7WXurLXBEJ9MI62T42XSzros/9rNLwoJRYAUhpDp+dD5xE/UAeH61nKeV/iXcFwKk+JwVJ45sTp/bUcWrbCyBlEOm7a3LGZ3SrxslJgRCoNwpBdwFGYTXIXnxH6IzWP2opmx4d1CQMP91AlUlqXpqFUq3RZHKzVLl9DusuNlkb2TeWZZYZp6/G545KMlH0XLk/c6bA0Ygjdpsw6QvRrWCoL1DP/oOMPJsJP7orcht07pOkubKpEAk0Phd0K+lX7NTFgAki4YhpWlpDdYEg== X-Bogosity: Ham, tests=bogofilter, spamicity=0.000000, version=1.2.4 Sender: owner-linux-mm@kvack.org Precedence: bulk X-Loop: owner-majordomo@kvack.org List-ID: List-Subscribe: List-Unsubscribe: Convert vmemmap_remap_walk->remap_pte to ->remap_pte_range so that vmemmap remap walks can batch update PTEs. The goal of this conversion is to allow architectures to implement their own optimizations if possible, e.g., only to stop remote CPUs once for each batch when updating vmemmap on arm64. It is not intended to change the remap workflow nor should it by itself have any side effects on performance. Signed-off-by: Yu Zhao --- mm/hugetlb_vmemmap.c | 163 ++++++++++++++++++++++++------------------- 1 file changed, 91 insertions(+), 72 deletions(-) diff --git a/mm/hugetlb_vmemmap.c b/mm/hugetlb_vmemmap.c index 57b7f591eee8..46befab48d41 100644 --- a/mm/hugetlb_vmemmap.c +++ b/mm/hugetlb_vmemmap.c @@ -22,7 +22,7 @@ /** * struct vmemmap_remap_walk - walk vmemmap page table * - * @remap_pte: called for each lowest-level entry (PTE). + * @remap_pte_range: called on a range of PTEs. * @nr_walked: the number of walked pte. * @reuse_page: the page which is reused for the tail vmemmap pages. * @reuse_addr: the virtual address of the @reuse_page page. @@ -32,8 +32,8 @@ * operations. */ struct vmemmap_remap_walk { - void (*remap_pte)(pte_t *pte, unsigned long addr, - struct vmemmap_remap_walk *walk); + void (*remap_pte_range)(pte_t *pte, unsigned long start, + unsigned long end, struct vmemmap_remap_walk *walk); unsigned long nr_walked; struct page *reuse_page; unsigned long reuse_addr; @@ -101,10 +101,6 @@ static int vmemmap_pmd_entry(pmd_t *pmd, unsigned long addr, struct page *head; struct vmemmap_remap_walk *vmemmap_walk = walk->private; - /* Only splitting, not remapping the vmemmap pages. */ - if (!vmemmap_walk->remap_pte) - walk->action = ACTION_CONTINUE; - spin_lock(&init_mm.page_table_lock); head = pmd_leaf(*pmd) ? pmd_page(*pmd) : NULL; /* @@ -129,33 +125,36 @@ static int vmemmap_pmd_entry(pmd_t *pmd, unsigned long addr, ret = -ENOTSUPP; } spin_unlock(&init_mm.page_table_lock); - if (!head || ret) + if (ret) return ret; - return vmemmap_split_pmd(pmd, head, addr & PMD_MASK, vmemmap_walk); -} + if (head) { + ret = vmemmap_split_pmd(pmd, head, addr & PMD_MASK, vmemmap_walk); + if (ret) + return ret; + } -static int vmemmap_pte_entry(pte_t *pte, unsigned long addr, - unsigned long next, struct mm_walk *walk) -{ - struct vmemmap_remap_walk *vmemmap_walk = walk->private; + if (vmemmap_walk->remap_pte_range) { + pte_t *pte = pte_offset_kernel(pmd, addr); - /* - * The reuse_page is found 'first' in page table walking before - * starting remapping. - */ - if (!vmemmap_walk->reuse_page) - vmemmap_walk->reuse_page = pte_page(ptep_get(pte)); - else - vmemmap_walk->remap_pte(pte, addr, vmemmap_walk); - vmemmap_walk->nr_walked++; + vmemmap_walk->nr_walked += (next - addr) / PAGE_SIZE; + /* + * The reuse_page is found 'first' in page table walking before + * starting remapping. + */ + if (!vmemmap_walk->reuse_page) { + vmemmap_walk->reuse_page = pte_page(ptep_get(pte)); + pte++; + addr += PAGE_SIZE; + } + vmemmap_walk->remap_pte_range(pte, addr, next, vmemmap_walk); + } return 0; } static const struct mm_walk_ops vmemmap_remap_ops = { .pmd_entry = vmemmap_pmd_entry, - .pte_entry = vmemmap_pte_entry, }; static int vmemmap_remap_range(unsigned long start, unsigned long end, @@ -172,7 +171,7 @@ static int vmemmap_remap_range(unsigned long start, unsigned long end, if (ret) return ret; - if (walk->remap_pte && !(walk->flags & VMEMMAP_REMAP_NO_TLB_FLUSH)) + if (walk->remap_pte_range && !(walk->flags & VMEMMAP_REMAP_NO_TLB_FLUSH)) flush_tlb_kernel_range(start, end); return 0; @@ -204,33 +203,45 @@ static void free_vmemmap_page_list(struct list_head *list) free_vmemmap_page(page); } -static void vmemmap_remap_pte(pte_t *pte, unsigned long addr, - struct vmemmap_remap_walk *walk) +static void vmemmap_remap_pte_range(pte_t *pte, unsigned long start, unsigned long end, + struct vmemmap_remap_walk *walk) { - /* - * Remap the tail pages as read-only to catch illegal write operation - * to the tail pages. - */ - pgprot_t pgprot = PAGE_KERNEL_RO; - struct page *page = pte_page(ptep_get(pte)); - pte_t entry; - - /* Remapping the head page requires r/w */ - if (unlikely(addr == walk->reuse_addr)) { - pgprot = PAGE_KERNEL; - list_del(&walk->reuse_page->lru); + int i; + struct page *page; + int nr_pages = (end - start) / PAGE_SIZE; + for (i = 0; i < nr_pages; i++) { + page = pte_page(ptep_get(pte + i)); + + list_add(&page->lru, walk->vmemmap_pages); + } + + page = walk->reuse_page; + + if (start == walk->reuse_addr) { + list_del(&page->lru); + copy_page(page_to_virt(page), (void *)walk->reuse_addr); /* - * Makes sure that preceding stores to the page contents from - * vmemmap_remap_free() become visible before the set_pte_at() - * write. + * Makes sure that preceding stores to the page contents become + * visible before set_pte_at(). */ smp_wmb(); } - entry = mk_pte(walk->reuse_page, pgprot); - list_add(&page->lru, walk->vmemmap_pages); - set_pte_at(&init_mm, addr, pte, entry); + for (i = 0; i < nr_pages; i++) { + pte_t val; + + /* + * The head page must be mapped read-write; the tail pages are + * mapped read-only to catch illegal modifications. + */ + if (!i && start == walk->reuse_addr) + val = mk_pte(page, PAGE_KERNEL); + else + val = mk_pte(page, PAGE_KERNEL_RO); + + set_pte_at(&init_mm, start + PAGE_SIZE * i, pte + i, val); + } } /* @@ -252,27 +263,39 @@ static inline void reset_struct_pages(struct page *start) memcpy(start, from, sizeof(*from) * NR_RESET_STRUCT_PAGE); } -static void vmemmap_restore_pte(pte_t *pte, unsigned long addr, - struct vmemmap_remap_walk *walk) +static void vmemmap_restore_pte_range(pte_t *pte, unsigned long start, unsigned long end, + struct vmemmap_remap_walk *walk) { - pgprot_t pgprot = PAGE_KERNEL; + int i; struct page *page; - void *to; - - BUG_ON(pte_page(ptep_get(pte)) != walk->reuse_page); + int nr_pages = (end - start) / PAGE_SIZE; page = list_first_entry(walk->vmemmap_pages, struct page, lru); - list_del(&page->lru); - to = page_to_virt(page); - copy_page(to, (void *)walk->reuse_addr); - reset_struct_pages(to); + + for (i = 0; i < nr_pages; i++) { + BUG_ON(pte_page(ptep_get(pte + i)) != walk->reuse_page); + + copy_page(page_to_virt(page), (void *)walk->reuse_addr); + reset_struct_pages(page_to_virt(page)); + + page = list_next_entry(page, lru); + } /* * Makes sure that preceding stores to the page contents become visible - * before the set_pte_at() write. + * before set_pte_at(). */ smp_wmb(); - set_pte_at(&init_mm, addr, pte, mk_pte(page, pgprot)); + + for (i = 0; i < nr_pages; i++) { + pte_t val; + + page = list_first_entry(walk->vmemmap_pages, struct page, lru); + list_del(&page->lru); + + val = mk_pte(page, PAGE_KERNEL); + set_pte_at(&init_mm, start + PAGE_SIZE * i, pte + i, val); + } } /** @@ -290,7 +313,6 @@ static int vmemmap_remap_split(unsigned long start, unsigned long end, unsigned long reuse) { struct vmemmap_remap_walk walk = { - .remap_pte = NULL, .flags = VMEMMAP_SPLIT_NO_TLB_FLUSH, }; @@ -322,10 +344,10 @@ static int vmemmap_remap_free(unsigned long start, unsigned long end, { int ret; struct vmemmap_remap_walk walk = { - .remap_pte = vmemmap_remap_pte, - .reuse_addr = reuse, - .vmemmap_pages = vmemmap_pages, - .flags = flags, + .remap_pte_range = vmemmap_remap_pte_range, + .reuse_addr = reuse, + .vmemmap_pages = vmemmap_pages, + .flags = flags, }; int nid = page_to_nid((struct page *)reuse); gfp_t gfp_mask = GFP_KERNEL | __GFP_NORETRY | __GFP_NOWARN; @@ -340,8 +362,6 @@ static int vmemmap_remap_free(unsigned long start, unsigned long end, */ walk.reuse_page = alloc_pages_node(nid, gfp_mask, 0); if (walk.reuse_page) { - copy_page(page_to_virt(walk.reuse_page), - (void *)walk.reuse_addr); list_add(&walk.reuse_page->lru, vmemmap_pages); memmap_pages_add(1); } @@ -371,10 +391,9 @@ static int vmemmap_remap_free(unsigned long start, unsigned long end, * They will be restored in the following call. */ walk = (struct vmemmap_remap_walk) { - .remap_pte = vmemmap_restore_pte, - .reuse_addr = reuse, - .vmemmap_pages = vmemmap_pages, - .flags = 0, + .remap_pte_range = vmemmap_restore_pte_range, + .reuse_addr = reuse, + .vmemmap_pages = vmemmap_pages, }; vmemmap_remap_range(reuse, end, &walk); @@ -425,10 +444,10 @@ static int vmemmap_remap_alloc(unsigned long start, unsigned long end, { LIST_HEAD(vmemmap_pages); struct vmemmap_remap_walk walk = { - .remap_pte = vmemmap_restore_pte, - .reuse_addr = reuse, - .vmemmap_pages = &vmemmap_pages, - .flags = flags, + .remap_pte_range = vmemmap_restore_pte_range, + .reuse_addr = reuse, + .vmemmap_pages = &vmemmap_pages, + .flags = flags, }; /* See the comment in the vmemmap_remap_free(). */ From patchwork Mon Oct 21 04:22:14 2024 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Yu Zhao X-Patchwork-Id: 13843541 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from kanga.kvack.org (kanga.kvack.org [205.233.56.17]) by smtp.lore.kernel.org (Postfix) with ESMTP id 7A294D3C93D for ; Mon, 21 Oct 2024 04:22:31 +0000 (UTC) Received: by kanga.kvack.org (Postfix) id 02E086B0088; Mon, 21 Oct 2024 00:22:31 -0400 (EDT) Received: by kanga.kvack.org (Postfix, from userid 40) id EA6706B0089; Mon, 21 Oct 2024 00:22:30 -0400 (EDT) X-Delivered-To: int-list-linux-mm@kvack.org Received: by kanga.kvack.org (Postfix, from userid 63042) id CD19D6B008A; Mon, 21 Oct 2024 00:22:30 -0400 (EDT) X-Delivered-To: linux-mm@kvack.org Received: from relay.hostedemail.com (smtprelay0011.hostedemail.com [216.40.44.11]) by kanga.kvack.org (Postfix) with ESMTP id AF7C16B0088 for ; Mon, 21 Oct 2024 00:22:30 -0400 (EDT) Received: from smtpin24.hostedemail.com (a10.router.float.18 [10.200.18.1]) by unirelay02.hostedemail.com (Postfix) with ESMTP id DA72A12153F for ; Mon, 21 Oct 2024 04:22:16 +0000 (UTC) X-FDA: 82696312440.24.FB69079 Received: from mail-yw1-f202.google.com (mail-yw1-f202.google.com [209.85.128.202]) by imf19.hostedemail.com (Postfix) with ESMTP id BBF0E1A0003 for ; Mon, 21 Oct 2024 04:22:10 +0000 (UTC) Authentication-Results: imf19.hostedemail.com; dkim=pass header.d=google.com header.s=20230601 header.b=qiC2Uaxr; spf=pass (imf19.hostedemail.com: domain of 3A9cVZwYKCEg849rkyqyyqvo.mywvsx47-wwu5kmu.y1q@flex--yuzhao.bounces.google.com designates 209.85.128.202 as permitted sender) smtp.mailfrom=3A9cVZwYKCEg849rkyqyyqvo.mywvsx47-wwu5kmu.y1q@flex--yuzhao.bounces.google.com; dmarc=pass (policy=reject) header.from=google.com ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=hostedemail.com; s=arc-20220608; t=1729484350; h=from:from:sender:reply-to:subject:subject:date:date: message-id:message-id:to:to:cc:cc:mime-version:mime-version: content-type:content-type:content-transfer-encoding: in-reply-to:in-reply-to:references:references:dkim-signature; bh=kQVfzlUfyOXzQNiM6rrgf1IxWXC/TuVQ/9/mg1gNSUk=; b=Hr9fqPVXrGNtyaadnZzaiNgs53BVf8ATgfEsrka6p2MGIMvTMNyFwsvK2jloV9l1+ijcVr fIi0D/7SbVboRDS7R/2LUz3rid6WxDTMjtZYW2T0mboFS03o3p3wb2JN69D1Ady0XJrAqS I0Qkk9qwvTYypC/3BGhkpg8OBt2kGxk= ARC-Authentication-Results: i=1; imf19.hostedemail.com; dkim=pass header.d=google.com header.s=20230601 header.b=qiC2Uaxr; spf=pass (imf19.hostedemail.com: domain of 3A9cVZwYKCEg849rkyqyyqvo.mywvsx47-wwu5kmu.y1q@flex--yuzhao.bounces.google.com designates 209.85.128.202 as permitted sender) smtp.mailfrom=3A9cVZwYKCEg849rkyqyyqvo.mywvsx47-wwu5kmu.y1q@flex--yuzhao.bounces.google.com; dmarc=pass (policy=reject) header.from=google.com ARC-Seal: i=1; s=arc-20220608; d=hostedemail.com; t=1729484350; a=rsa-sha256; cv=none; b=YJMpgbjO+j94UNCs/2+o0/5mriwxuDeht7H3xDKrEqx3T8nvr5zMBezPIwllqTZTWZG3kz WCoYLoQMoPq+TwdD5jOrhhfhgW3SJrUI8hZ4q+GQp/8SNrY8DVZ0hvDjm3TxlI4NRPJpvR Dxr/JVxeYoWm4VmW7TyOkAZcexJ5RsQ= Received: by mail-yw1-f202.google.com with SMTP id 00721157ae682-6e315a5b199so64481337b3.2 for ; Sun, 20 Oct 2024 21:22:28 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=google.com; s=20230601; t=1729484548; x=1730089348; darn=kvack.org; h=cc:to:from:subject:message-id:references:mime-version:in-reply-to :date:from:to:cc:subject:date:message-id:reply-to; bh=kQVfzlUfyOXzQNiM6rrgf1IxWXC/TuVQ/9/mg1gNSUk=; b=qiC2UaxrWj+cpqLemRVCyqmi9lqpQevy1nqq3sVjNNpwJMuS2jXGiPX/YqjY+IKCcP H7fRnddBaupa7cX5MfgXWP5mW/0DLwYQxXuoK2ok2SqNRuPlzQz3wNABkwd5Me+AfSz3 VXlGZrsclulFONZZ2y5c5O16n5i0fxFQiHLV4bBqsjTNN0n2KR2tIzFC0rjPt6x9ZnmO aMa3qG7egaEPpBZvbeh6AvT2kixR4y239ZxRc+nRzFRTELYIHAlizvh8EP2j3zrj4CwR 8lzMqw+G/DMdIoQfLmn52mbbXchAKhkIJcVqOqYWBCPIpg2qsxHaiybHpeYjWx/uMyha E3CQ== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20230601; t=1729484548; x=1730089348; h=cc:to:from:subject:message-id:references:mime-version:in-reply-to :date:x-gm-message-state:from:to:cc:subject:date:message-id:reply-to; bh=kQVfzlUfyOXzQNiM6rrgf1IxWXC/TuVQ/9/mg1gNSUk=; b=CI9KRRm3ll2DJfAdnZwez8GUFPkpY2tibPDoFHCA+951Z590a+6fS8C7Xo18dXIlHl mAvPg15OAXnbbiL5zI6y9aR6xIZM8HVvbVZDeHga3fCZeC6OYXxB4ZAY9RScISp2mO/E nQ7SGvcJDUTvFyTkBGchaYmeg9QQsjbDAAtLasSlNDddPdHn12Wzzq99v3yf/oVKycU7 YqUchQxIwLmA02qR9jL6/Vwndo0c5RSg3fQuTioF2Z+ncjhl/fWA4vdYsFPLNhabklj4 aMb12+eRshcQ+iUZ7ODW0WI8qNGjg6hn1wqLWNaDkcTDX/NkuVlKalMzOluc0tuLB7CF eW7w== X-Forwarded-Encrypted: i=1; AJvYcCWuS+iT4E++JZ0N5i3TaphJ7Ltusnj1BDZhEbKYtypfTWqDj6tv31I45Uy9pGOXNSzXmXb5972IZA==@kvack.org X-Gm-Message-State: AOJu0YyUUoHOeBB35Xk8l5E73sAO3buu44wvSTj72REDBMsyzoNMC7xS Y7fKOyW/oNm05y5iOFarZsGhE0sN8SQDq2qhsGvW+jWDJZ4JP3cN7Feg4a7jaGVqbuRPDXkCOy7 LGA== X-Google-Smtp-Source: AGHT+IFa+RAhMnCTBGawKhIBMzr1i4wNva1osbWPtOgVISCaAsioJIQoe2S7YEzQTgFhpJ04zIjZfcK8uDQ= X-Received: from yuzhao2.bld.corp.google.com ([2a00:79e0:2e28:6:1569:9ef4:20ab:abf9]) (user=yuzhao job=sendgmr) by 2002:a05:690c:6383:b0:663:ddc1:eab8 with SMTP id 00721157ae682-6e5bfbeb2e0mr2568607b3.4.1729484547736; Sun, 20 Oct 2024 21:22:27 -0700 (PDT) Date: Sun, 20 Oct 2024 22:22:14 -0600 In-Reply-To: <20241021042218.746659-1-yuzhao@google.com> Mime-Version: 1.0 References: <20241021042218.746659-1-yuzhao@google.com> X-Mailer: git-send-email 2.47.0.rc1.288.g06298d1525-goog Message-ID: <20241021042218.746659-3-yuzhao@google.com> Subject: [PATCH v1 2/6] mm/hugetlb_vmemmap: add arch-independent helpers From: Yu Zhao To: Andrew Morton , Catalin Marinas , Marc Zyngier , Muchun Song , Thomas Gleixner , Will Deacon Cc: Douglas Anderson , Mark Rutland , Nanyong Sun , linux-arm-kernel@lists.infradead.org, linux-kernel@vger.kernel.org, linux-mm@kvack.org, Yu Zhao X-Rspamd-Server: rspam06 X-Rspamd-Queue-Id: BBF0E1A0003 X-Stat-Signature: ptu7hhxrajc9eo1if4p37i4e6p3pwqyh X-Rspam-User: X-HE-Tag: 1729484530-899358 X-HE-Meta: U2FsdGVkX19H8ZWLhC37mjmxqeQGijvs5Del5Vow0ItuDKdKpizi7/q9c3HBPRJcUGqMGJbRRepfoQJ99ebZKlTWSz1/dvXvWpXKTbDmDgAacRs2y9wtxLTkCTj17D3LU8rD4zC2eRAF4E55m9CLfeyNMoRuXb/gHM5PFRB+QXMKnDZ5hkPsntKW+YHgpyoYITRQ7GzeJrxxY4iHK9yeSI6rQLPhdZe6x64W/HyqpTEAttfnmdG4SFHVWr+mPetKI7zJx4eJlmKvH9Y/BVeD056HqTX28Hpn1n+oPckvhArYdMvBvb89FdxjMvxMg0teq0haMr/U68vNl7I84EkYOynR6E+xDWPPXT0E1RmSyieVuutXBp+cztFMco5j1ZtuDQBtepHZWWFY6Id0mrkpMDfx0ynlyqj2OQcXKVkUDnhskwyd6hDXzMsiBUdmXduZ2OmboXApjMgHS1sZc+1IcP7VDnOqjesAWvA+KS2Vnf4C8krxbfJBZ1XvbNEnQ6JpBnKhGrZQoYRlOgTXcfT4XIhiXHyoqJhau5a7L7vC7sYP5mkeUEDSxtu9TtuSQNakXQipGvI46M2liNINOPzaNkO6Jt6evgOhoBITvoKrLumRs2MAKNBl9C6vCMzUtYXQa6wHogQC39JmuoPCmMaAEGAQhhyuzW45nx+iFRs0yzQpSgFPUCPfGDuSUhOJQNfnJYEFrgYgSf+/orOLDRW5we/L7X64nofVQ+KZ59/KSSo03qrpORqCBlg5SVA6/kUHx5tpdL0B36L6mrBf/wJu+k8xopf1NM6A3dN71yZNz6cSlYIjYhJZVUmbkAIU6mfsRH9b3E02bDs3cMeORPj8pbWjjM52KVRR0bKLIAZehv9+wm14Zi0ZiVAkp1yskD6Ya1YN27PcM/7nddTqFgX5Am+63c30+KIIFFEdc/dNnHTHOBgTyVjDcFAvb7tY+Evn86pMbybzE+xFc1hKmI9 h147oQKG R1pyvxykalm4PPtP56qRjiJ2mM0eyk2zfA/oO1mTIFqWVzflhAOIGe0VmkrJxn6MwHoAq8Hj28jfgm15rLI1Ed0cl/hvseyn2luB0UdIl0fr30xATsBXGugdDyFHxHWCTpijA0Y7ksJDMlV/PTYQerDT5JnXhC9vHzmvF+JMrLrXPpRoCMxk4zt7ZZNJp7nJGb2yNOYnhw0C9k/inlTuQ631YslVzE/8ayAEy6BYOkjCo110/RyxMFuDh6thEFINiOML5r2ALN9aMrBegdZeL5q919u+r9gkC9kYwcN0UB/9tnUfa015jtVj/8rIpePxQa3zPrCsgiiTpesJbPl52rgs7kNPrNqIH2K2YLJ50FIg+JMyHEdJo62/Mrrxa8DzeKPB7DPorqdmpqHloMJ9ryOy82hAuGFEBEdWqOHRycbW9KHhrNXY3vauHF8ZfVgMWH06SDEkgSZTV9CcjOp/cf377RSwGlQYbtsHOwyxjU2knrRHC4WjE1EJy4X2Lff6cqfEDJfZZvGPXOXsyiRB5w8+Jkrylp8TTghqFLAsnmP488t/EEsYwob8EJqDYW5V/mCY8ntTGDBunZbM+BKdlgNbcMA== X-Bogosity: Ham, tests=bogofilter, spamicity=0.000000, version=1.2.4 Sender: owner-linux-mm@kvack.org Precedence: bulk X-Loop: owner-majordomo@kvack.org List-ID: List-Subscribe: List-Unsubscribe: Add architecture-independent helpers to allow individual architectures to work around their own limitations when updating vmemmap. Specifically, the current remap workflow requires break-before-make (BBM) on arm64. By overriding the default helpers later in this series, arm64 will be able to support the current HVO implementation. Signed-off-by: Yu Zhao --- include/linux/mm_types.h | 7 +++ mm/hugetlb_vmemmap.c | 99 ++++++++++++++++++++++++++++++++++------ 2 files changed, 92 insertions(+), 14 deletions(-) diff --git a/include/linux/mm_types.h b/include/linux/mm_types.h index 6e3bdf8e38bc..0f3ae6e173f6 100644 --- a/include/linux/mm_types.h +++ b/include/linux/mm_types.h @@ -1499,4 +1499,11 @@ enum { /* See also internal only FOLL flags in mm/internal.h */ }; +/* Skip the TLB flush when we split the PMD */ +#define VMEMMAP_SPLIT_NO_TLB_FLUSH BIT(0) +/* Skip the TLB flush when we remap the PTE */ +#define VMEMMAP_REMAP_NO_TLB_FLUSH BIT(1) +/* synchronize_rcu() to avoid writes from page_ref_add_unless() */ +#define VMEMMAP_SYNCHRONIZE_RCU BIT(2) + #endif /* _LINUX_MM_TYPES_H */ diff --git a/mm/hugetlb_vmemmap.c b/mm/hugetlb_vmemmap.c index 46befab48d41..e50a196399f5 100644 --- a/mm/hugetlb_vmemmap.c +++ b/mm/hugetlb_vmemmap.c @@ -38,16 +38,56 @@ struct vmemmap_remap_walk { struct page *reuse_page; unsigned long reuse_addr; struct list_head *vmemmap_pages; - -/* Skip the TLB flush when we split the PMD */ -#define VMEMMAP_SPLIT_NO_TLB_FLUSH BIT(0) -/* Skip the TLB flush when we remap the PTE */ -#define VMEMMAP_REMAP_NO_TLB_FLUSH BIT(1) -/* synchronize_rcu() to avoid writes from page_ref_add_unless() */ -#define VMEMMAP_SYNCHRONIZE_RCU BIT(2) unsigned long flags; }; +#ifndef VMEMMAP_ARCH_TLB_FLUSH_FLAGS +#define VMEMMAP_ARCH_TLB_FLUSH_FLAGS 0 +#endif + +#ifndef vmemmap_update_supported +static bool vmemmap_update_supported(void) +{ + return true; +} +#endif + +#ifndef vmemmap_update_lock +static void vmemmap_update_lock(void) +{ +} +#endif + +#ifndef vmemmap_update_unlock +static void vmemmap_update_unlock(void) +{ +} +#endif + +#ifndef vmemmap_update_pte_range_start +static void vmemmap_update_pte_range_start(pte_t *pte, unsigned long start, unsigned long end) +{ +} +#endif + +#ifndef vmemmap_update_pte_range_end +static void vmemmap_update_pte_range_end(void) +{ +} +#endif + +#ifndef vmemmap_update_pmd_range_start +static void vmemmap_update_pmd_range_start(pmd_t *pmd, unsigned long start, unsigned long end) +{ +} +#endif + +#ifndef vmemmap_update_pmd_range_end +static void vmemmap_update_pmd_range_end(void) +{ +} +#endif + static int vmemmap_split_pmd(pmd_t *pmd, struct page *head, unsigned long start, struct vmemmap_remap_walk *walk) { @@ -83,7 +123,9 @@ static int vmemmap_split_pmd(pmd_t *pmd, struct page *head, unsigned long start, /* Make pte visible before pmd. See comment in pmd_install(). */ smp_wmb(); + vmemmap_update_pmd_range_start(pmd, start, start + PMD_SIZE); pmd_populate_kernel(&init_mm, pmd, pgtable); + vmemmap_update_pmd_range_end(); if (!(walk->flags & VMEMMAP_SPLIT_NO_TLB_FLUSH)) flush_tlb_kernel_range(start, start + PMD_SIZE); } else { @@ -164,10 +206,12 @@ static int vmemmap_remap_range(unsigned long start, unsigned long end, VM_BUG_ON(!PAGE_ALIGNED(start | end)); + vmemmap_update_lock(); mmap_read_lock(&init_mm); ret = walk_page_range_novma(&init_mm, start, end, &vmemmap_remap_ops, NULL, walk); mmap_read_unlock(&init_mm); + vmemmap_update_unlock(); if (ret) return ret; @@ -228,6 +272,8 @@ static void vmemmap_remap_pte_range(pte_t *pte, unsigned long start, unsigned lo smp_wmb(); } + vmemmap_update_pte_range_start(pte, start, end); + for (i = 0; i < nr_pages; i++) { pte_t val; @@ -242,6 +288,8 @@ static void vmemmap_remap_pte_range(pte_t *pte, unsigned long start, unsigned lo set_pte_at(&init_mm, start + PAGE_SIZE * i, pte + i, val); } + + vmemmap_update_pte_range_end(); } /* @@ -287,6 +335,8 @@ static void vmemmap_restore_pte_range(pte_t *pte, unsigned long start, unsigned */ smp_wmb(); + vmemmap_update_pte_range_start(pte, start, end); + for (i = 0; i < nr_pages; i++) { pte_t val; @@ -296,6 +346,8 @@ static void vmemmap_restore_pte_range(pte_t *pte, unsigned long start, unsigned val = mk_pte(page, PAGE_KERNEL); set_pte_at(&init_mm, start + PAGE_SIZE * i, pte + i, val); } + + vmemmap_update_pte_range_end(); } /** @@ -513,7 +565,8 @@ static int __hugetlb_vmemmap_restore_folio(const struct hstate *h, */ int hugetlb_vmemmap_restore_folio(const struct hstate *h, struct folio *folio) { - return __hugetlb_vmemmap_restore_folio(h, folio, VMEMMAP_SYNCHRONIZE_RCU); + return __hugetlb_vmemmap_restore_folio(h, folio, + VMEMMAP_SYNCHRONIZE_RCU | VMEMMAP_ARCH_TLB_FLUSH_FLAGS); } /** @@ -553,7 +606,7 @@ long hugetlb_vmemmap_restore_folios(const struct hstate *h, list_move(&folio->lru, non_hvo_folios); } - if (restored) + if (restored && !(VMEMMAP_ARCH_TLB_FLUSH_FLAGS & VMEMMAP_REMAP_NO_TLB_FLUSH)) flush_tlb_all(); if (!ret) ret = restored; @@ -641,7 +694,8 @@ void hugetlb_vmemmap_optimize_folio(const struct hstate *h, struct folio *folio) { LIST_HEAD(vmemmap_pages); - __hugetlb_vmemmap_optimize_folio(h, folio, &vmemmap_pages, VMEMMAP_SYNCHRONIZE_RCU); + __hugetlb_vmemmap_optimize_folio(h, folio, &vmemmap_pages, + VMEMMAP_SYNCHRONIZE_RCU | VMEMMAP_ARCH_TLB_FLUSH_FLAGS); free_vmemmap_page_list(&vmemmap_pages); } @@ -683,7 +737,8 @@ void hugetlb_vmemmap_optimize_folios(struct hstate *h, struct list_head *folio_l break; } - flush_tlb_all(); + if (!(VMEMMAP_ARCH_TLB_FLUSH_FLAGS & VMEMMAP_SPLIT_NO_TLB_FLUSH)) + flush_tlb_all(); list_for_each_entry(folio, folio_list, lru) { int ret; @@ -701,24 +756,35 @@ void hugetlb_vmemmap_optimize_folios(struct hstate *h, struct list_head *folio_l * allowing more vmemmap remaps to occur. */ if (ret == -ENOMEM && !list_empty(&vmemmap_pages)) { - flush_tlb_all(); + if (!(VMEMMAP_ARCH_TLB_FLUSH_FLAGS & VMEMMAP_REMAP_NO_TLB_FLUSH)) + flush_tlb_all(); free_vmemmap_page_list(&vmemmap_pages); INIT_LIST_HEAD(&vmemmap_pages); __hugetlb_vmemmap_optimize_folio(h, folio, &vmemmap_pages, flags); } } - flush_tlb_all(); + if (!(VMEMMAP_ARCH_TLB_FLUSH_FLAGS & VMEMMAP_REMAP_NO_TLB_FLUSH)) + flush_tlb_all(); free_vmemmap_page_list(&vmemmap_pages); } +static int hugetlb_vmemmap_sysctl(const struct ctl_table *ctl, int write, + void *buffer, size_t *lenp, loff_t *ppos) +{ + if (!vmemmap_update_supported()) + return -ENODEV; + + return proc_dobool(ctl, write, buffer, lenp, ppos); +} + static struct ctl_table hugetlb_vmemmap_sysctls[] = { { .procname = "hugetlb_optimize_vmemmap", .data = &vmemmap_optimize_enabled, .maxlen = sizeof(vmemmap_optimize_enabled), .mode = 0644, - .proc_handler = proc_dobool, + .proc_handler = hugetlb_vmemmap_sysctl, }, }; @@ -729,6 +795,11 @@ static int __init hugetlb_vmemmap_init(void) /* HUGETLB_VMEMMAP_RESERVE_SIZE should cover all used struct pages */ BUILD_BUG_ON(__NR_USED_SUBPAGE > HUGETLB_VMEMMAP_RESERVE_PAGES); + if (READ_ONCE(vmemmap_optimize_enabled) && !vmemmap_update_supported()) { + pr_warn("HugeTLB: disabling HVO due to missing support.\n"); + WRITE_ONCE(vmemmap_optimize_enabled, false); + } + for_each_hstate(h) { if (hugetlb_vmemmap_optimizable(h)) { register_sysctl_init("vm", hugetlb_vmemmap_sysctls); From patchwork Mon Oct 21 04:22:15 2024 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Yu Zhao X-Patchwork-Id: 13843542 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from kanga.kvack.org (kanga.kvack.org [205.233.56.17]) by smtp.lore.kernel.org (Postfix) with ESMTP id 593E0D3C933 for ; Mon, 21 Oct 2024 04:22:34 +0000 (UTC) Received: by kanga.kvack.org (Postfix) id CD5126B008A; Mon, 21 Oct 2024 00:22:33 -0400 (EDT) Received: by kanga.kvack.org (Postfix, from userid 40) id C83C46B008C; Mon, 21 Oct 2024 00:22:33 -0400 (EDT) X-Delivered-To: int-list-linux-mm@kvack.org Received: by kanga.kvack.org (Postfix, from userid 63042) id AB0BE6B0092; Mon, 21 Oct 2024 00:22:33 -0400 (EDT) X-Delivered-To: linux-mm@kvack.org Received: from relay.hostedemail.com (smtprelay0013.hostedemail.com [216.40.44.13]) by kanga.kvack.org (Postfix) with ESMTP id 746466B008A for ; Mon, 21 Oct 2024 00:22:33 -0400 (EDT) Received: from smtpin04.hostedemail.com (a10.router.float.18 [10.200.18.1]) by unirelay03.hostedemail.com (Postfix) with ESMTP id C7F2AA145C for ; Mon, 21 Oct 2024 04:22:05 +0000 (UTC) X-FDA: 82696312692.04.A661831 Received: from mail-yb1-f201.google.com (mail-yb1-f201.google.com [209.85.219.201]) by imf01.hostedemail.com (Postfix) with ESMTP id A5D4340011 for ; Mon, 21 Oct 2024 04:22:19 +0000 (UTC) Authentication-Results: imf01.hostedemail.com; dkim=pass header.d=google.com header.s=20230601 header.b=LR76z8Fp; spf=pass (imf01.hostedemail.com: domain of 3BtcVZwYKCEsB7Cun1t11tyr.p1zyv07A-zzx8npx.14t@flex--yuzhao.bounces.google.com designates 209.85.219.201 as permitted sender) smtp.mailfrom=3BtcVZwYKCEsB7Cun1t11tyr.p1zyv07A-zzx8npx.14t@flex--yuzhao.bounces.google.com; dmarc=pass (policy=reject) header.from=google.com ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=hostedemail.com; s=arc-20220608; t=1729484501; h=from:from:sender:reply-to:subject:subject:date:date: message-id:message-id:to:to:cc:cc:mime-version:mime-version: content-type:content-type:content-transfer-encoding: in-reply-to:in-reply-to:references:references:dkim-signature; bh=rF+TQYGEDP6+t2V2Dte+w8ImgbG43FZ1I4yOJyMV9jY=; b=0eHllPlWoiqM5KnpjZrnOUwRuDCAzeKir1jCuK564AJSFA2eSwDa9+OFC7538nn2KQGMGQ pprbCJUiTX9g/Ewf9OOHhYAM5ejmMk+RbhJRz2cGnnIe911gKKO+fXtwFuSfCqfRey94O6 tpb0QE9t+puWJ9AJksKhWDKctzhrH2Y= ARC-Authentication-Results: i=1; imf01.hostedemail.com; dkim=pass header.d=google.com header.s=20230601 header.b=LR76z8Fp; spf=pass (imf01.hostedemail.com: domain of 3BtcVZwYKCEsB7Cun1t11tyr.p1zyv07A-zzx8npx.14t@flex--yuzhao.bounces.google.com designates 209.85.219.201 as permitted sender) smtp.mailfrom=3BtcVZwYKCEsB7Cun1t11tyr.p1zyv07A-zzx8npx.14t@flex--yuzhao.bounces.google.com; dmarc=pass (policy=reject) header.from=google.com ARC-Seal: i=1; s=arc-20220608; d=hostedemail.com; t=1729484501; a=rsa-sha256; cv=none; b=m9CF9TmDLDqjT2Je7xNBhSLdmGINe8AW74ie5NCIGnvyr0/pyOYSDDP0mnIkh71CTz4yD9 A9EoHrA/BBxn0/VdD3/9gWztr0sjuc48FoXEnVxfXkMonC7EgN1EUuxmTPjFWvAgkYvilG fKjpHZLObZCIPA64zoRSJ08hDQArrlA= Received: by mail-yb1-f201.google.com with SMTP id 3f1490d57ef6-e2939e25402so6261955276.2 for ; Sun, 20 Oct 2024 21:22:30 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=google.com; s=20230601; t=1729484550; x=1730089350; darn=kvack.org; h=cc:to:from:subject:message-id:references:mime-version:in-reply-to :date:from:to:cc:subject:date:message-id:reply-to; bh=rF+TQYGEDP6+t2V2Dte+w8ImgbG43FZ1I4yOJyMV9jY=; b=LR76z8FpVonfntEgmlk0BnpHEik25crT+toL5oBeoQITMMVe6PNoa+8EQ5hUTcJm4U HyWYzkPpI+RLO5dQaBJRxOnWj7RPhwBGy8nHuC22SEmwtJ6C04TSmuT6RD64w0vlzxuC bg8EIfDzC3dEES4PnT+lZMHUOT+c3+mrA71CiFiY9hR8dazp3f0Zk4SIxyowQ0CyzeRQ 0XbYhmb1aY3PP7uhkPS4xh7Unm29RW07mIRN6NSksS/SL11/Pwsre3eQvk18BXDb59Ii AZwXWuxPc+HGTa3l4snxUH4NpMyxUQzrPdNYMJ0ZQDUgycBw9Jmql7UkmRuYYhsvlt9C Os4w== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20230601; t=1729484550; x=1730089350; h=cc:to:from:subject:message-id:references:mime-version:in-reply-to :date:x-gm-message-state:from:to:cc:subject:date:message-id:reply-to; bh=rF+TQYGEDP6+t2V2Dte+w8ImgbG43FZ1I4yOJyMV9jY=; b=vSNDd+8qZSJ3rYH54FLADXwu1ZpqSm+P4NfTDGQsqm5m1JjitJmm1vK6Qll8s7vnDA dIr/DBT+8sARGIuCkAayXW5wZ0hhDpZHfMFC3wvC2wz8XR3FoTICW9yeaCL8f11VKf3c 5RacCnf3zdb7zf8xvu61MsCL54Du7lwqEyBvB/5C7yUt3BYujgznQhE2qeqS7yR6c4W7 RGaSWQdY2jW/CvrHoj+wkjHkFpkx3gip6ZRLQbwRQMXBnT4DHI5NwtQNMvbesDq1NVGE RU0u1b6/1/PAc106KNZ2EyDc52wpv4D9YMVLrTXk9k5KB4p9bBaJ2nNgYaOda3zg2tVM mSBA== X-Forwarded-Encrypted: i=1; AJvYcCWCkmTKXdN1O1y2CQnotBthS/SMJY9pXoUy+ur0JXtu6sksCUVoM36K5Zq+L202BxEDIfI/wJOoWQ==@kvack.org X-Gm-Message-State: AOJu0YzvDVPJM7Lz9wtGaDjGfvv4v7XwWd7KdrkbjHoNhKNGgpgZUM8c 5+0wPwPZXKblFjP79+VdziuukKTVvdQiHjsN9JEbpNUjPGzBUrqCxRiHoJ2j5iZxYM9QwXwx3Jb SpQ== X-Google-Smtp-Source: AGHT+IEeIIe5Xg98lfzs9VJWYOJ07F6ek87y8El+d0cpjxlVvqWrNUTXfkoh0Qdzc1+S/vFogSJ2meafeF0= X-Received: from yuzhao2.bld.corp.google.com ([2a00:79e0:2e28:6:1569:9ef4:20ab:abf9]) (user=yuzhao job=sendgmr) by 2002:a25:800f:0:b0:e05:6532:166 with SMTP id 3f1490d57ef6-e2bb11cccaemr14439276.1.1729484550141; Sun, 20 Oct 2024 21:22:30 -0700 (PDT) Date: Sun, 20 Oct 2024 22:22:15 -0600 In-Reply-To: <20241021042218.746659-1-yuzhao@google.com> Mime-Version: 1.0 References: <20241021042218.746659-1-yuzhao@google.com> X-Mailer: git-send-email 2.47.0.rc1.288.g06298d1525-goog Message-ID: <20241021042218.746659-4-yuzhao@google.com> Subject: [PATCH v1 3/6] irqchip/gic-v3: support SGI broadcast From: Yu Zhao To: Andrew Morton , Catalin Marinas , Marc Zyngier , Muchun Song , Thomas Gleixner , Will Deacon Cc: Douglas Anderson , Mark Rutland , Nanyong Sun , linux-arm-kernel@lists.infradead.org, linux-kernel@vger.kernel.org, linux-mm@kvack.org, Yu Zhao X-Rspam-User: X-Stat-Signature: qfzibruy4thsbc3ecai88u3wr8ab6msj X-Rspamd-Queue-Id: A5D4340011 X-Rspamd-Server: rspam11 X-HE-Tag: 1729484539-845459 X-HE-Meta: U2FsdGVkX1+W9APWGIaXvDLLfTh4IGertmIYpR3IsBs3rAA0F+X3eihuHHmsY3weUNvYP4rlWk5qYVbRWgzxGnLUAuSIYhDOuJV9qrM1WwEsBxIDHXMUNgW4nSnoIVJ4TDm7eSmkYCKMoUm3Y/nMJGW1jwJFUtn0h6Teomyty5L+a+52BERq+OvPU6jQJlvfef37f9ptkQ5rcGHO+uLUGsfaiigok+B10g+0SJnr4Fv9ZD/pMPmPv7i6O/4kDi/VLQB4ar+s9Rdnz3DwjUxzl+FwAeABDrE/WWwmu+vh5XGGIS1Ihf9DBHeKPq96UqdYR2WCLJ0HrBpn+e2g6MRrrzBO9tc2weHysOXb14VgPpoPba+7dhXwpsjZAQiZ0/acprxCiK3Nk5lIXdwPJl1R1NInKTV0FXnZ0RlnAwPg4kax1zMAe0Qfq0KEQdB2BbM+zFL3lUeAAbrovSGKjCRbNfkhuOrwTt67HKqa+HMXTc1wjPQlpgzSYZ+To0DQc/tlAmM21YcB+UJbKAHejwotpiz8QcM5wXlv52z/N15B9Wh3wrqs4vx3UCpOfRG4KaT/Bgb0P2pSeuCFXW+9hCPn6g1YXUmsp6GfBn3Hq+XvaqZPVz0uWz1S7CQQD33zsop6BPXzhWKHrh0By6XnD4UdCyOkKHjAL9oG0AZz9KkOEQOmvsDChlSR1dmxrSOOcn/VgaH0ShD08ygop8V83tybDPMaFAfdAZ4M5jybYMdK9td1gsyCMptMfZalnS/HYyYR2vffo9SkUDOru+C83nz0y1xVccZBj2y+16L+7uqZn9r0+3rHl9g9h/GhjT2+nwOEqW1wANr9tbm+Qqd3v2sFtZiAClOSX5TeRWxjZFHafx91y7SYvVFfTdFPGRPfeNOpTeVY/fKrqvq9+vnCeOm64k8mCdVnIgioFybBiecibWChKn8ggHcPpfG0OSQXgS8QsezCWS/FMRPrBABsyiF NhOk9434 F+W0Q4t/CZQLVTBblB1/3vD2oQmFHVjrBVUpxcolMS72s3YaSScTlD6R1GCcSEzLerVp6GjblAUutOxq4xJSv99/3xvk2AxlNGvsiReMa2O4VHsx+mpqz1zoNKez4FNtJ3+gihdvsc32bqTsOTGUnUKhZj4gAtG4Pvlt+SduuMlz2gYu+Yn3yQh2BDm2alpXdfoJRrEiQFF+NflrM/XkCn3jj8F9I9exAp5zoy3gQBDPUHSOqT7bjbItLPCpQntEtP4m60RTueFP5Ys+kEzCgK44Whm6w3uR8+0WIW5WPczrz8gAlODw4OyKFUZEoJEmqS/NM1cWVbiQET9oJShi8UyLeHytM+AIvuxGR+lq9h4S4JrMimhcWGN/5CKznV4bLNDFgyMhD18zqhIMcxxAsyy7zVS+dLWE6tZjsoEJCVOtb25qeTdpsuTUkPQHQRtVqs0MMlD2dv4ZjbJ2lo9v1fcl4HpCFd9Pe5WMaOrT6kmonFOFYb6bEzcEmx43ipORXPuM5NzXP7pR7nhrxV/mWyIN0F0RGoxeTDvXMygkvS2ok2as= X-Bogosity: Ham, tests=bogofilter, spamicity=0.000000, version=1.2.4 Sender: owner-linux-mm@kvack.org Precedence: bulk X-Loop: owner-majordomo@kvack.org List-ID: List-Subscribe: List-Unsubscribe: GIC v3 and later support SGI broadcast, i.e., the mode that routes interrupts to all PEs in the system excluding the local CPU. Supporting this mode can avoid looping through all the remote CPUs when broadcasting SGIs, especially for systems with 200+ CPUs. The performance improvement can be measured with the rest of this series booted with "hugetlb_free_vmemmap=on irqchip.gicv3_pseudo_nmi=1": cd /sys/kernel/mm/hugepages/ echo 600 >hugepages-1048576kB/nr_hugepages echo 2048kB >hugepages-1048576kB/demote_size perf record -g -- bash -c "echo 600 >hugepages-1048576kB/demote" gic_ipi_send_mask() bash sys time Before: 38.14% 0m10.513s After: 0.20% 0m5.132s Signed-off-by: Yu Zhao --- drivers/irqchip/irq-gic-v3.c | 20 +++++++++++++++++++- 1 file changed, 19 insertions(+), 1 deletion(-) diff --git a/drivers/irqchip/irq-gic-v3.c b/drivers/irqchip/irq-gic-v3.c index ce87205e3e82..42c39385e1b9 100644 --- a/drivers/irqchip/irq-gic-v3.c +++ b/drivers/irqchip/irq-gic-v3.c @@ -1394,9 +1394,20 @@ static void gic_send_sgi(u64 cluster_id, u16 tlist, unsigned int irq) gic_write_sgi1r(val); } +static void gic_broadcast_sgi(unsigned int irq) +{ + u64 val; + + val = BIT(ICC_SGI1R_IRQ_ROUTING_MODE_BIT) | (irq << ICC_SGI1R_SGI_ID_SHIFT); + + pr_devel("CPU %d: broadcasting SGI %u\n", smp_processor_id(), irq); + gic_write_sgi1r(val); +} + static void gic_ipi_send_mask(struct irq_data *d, const struct cpumask *mask) { int cpu; + cpumask_t broadcast; if (WARN_ON(d->hwirq >= 16)) return; @@ -1407,6 +1418,13 @@ static void gic_ipi_send_mask(struct irq_data *d, const struct cpumask *mask) */ dsb(ishst); + cpumask_copy(&broadcast, cpu_present_mask); + cpumask_clear_cpu(smp_processor_id(), &broadcast); + if (cpumask_equal(&broadcast, mask)) { + gic_broadcast_sgi(d->hwirq); + goto done; + } + for_each_cpu(cpu, mask) { u64 cluster_id = MPIDR_TO_SGI_CLUSTER_ID(gic_cpu_to_affinity(cpu)); u16 tlist; @@ -1414,7 +1432,7 @@ static void gic_ipi_send_mask(struct irq_data *d, const struct cpumask *mask) tlist = gic_compute_target_list(&cpu, mask, cluster_id); gic_send_sgi(cluster_id, tlist, d->hwirq); } - +done: /* Force the above writes to ICC_SGI1R_EL1 to be executed */ isb(); } From patchwork Mon Oct 21 04:22:16 2024 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Yu Zhao X-Patchwork-Id: 13843543 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from kanga.kvack.org (kanga.kvack.org [205.233.56.17]) by smtp.lore.kernel.org (Postfix) with ESMTP id 633EBD3C93E for ; Mon, 21 Oct 2024 04:22:36 +0000 (UTC) Received: by kanga.kvack.org (Postfix) id B4ECC6B0092; Mon, 21 Oct 2024 00:22:35 -0400 (EDT) Received: by kanga.kvack.org (Postfix, from userid 40) id AFDF16B0093; Mon, 21 Oct 2024 00:22:35 -0400 (EDT) X-Delivered-To: int-list-linux-mm@kvack.org Received: by kanga.kvack.org (Postfix, from userid 63042) id 928F16B0095; Mon, 21 Oct 2024 00:22:35 -0400 (EDT) X-Delivered-To: linux-mm@kvack.org Received: from relay.hostedemail.com (smtprelay0013.hostedemail.com [216.40.44.13]) by kanga.kvack.org (Postfix) with ESMTP id 6E55B6B0092 for ; Mon, 21 Oct 2024 00:22:35 -0400 (EDT) Received: from smtpin28.hostedemail.com (a10.router.float.18 [10.200.18.1]) by unirelay03.hostedemail.com (Postfix) with ESMTP id F1D61A0E91 for ; Mon, 21 Oct 2024 04:22:07 +0000 (UTC) X-FDA: 82696312902.28.F41F0FA Received: from mail-yw1-f201.google.com (mail-yw1-f201.google.com [209.85.128.201]) by imf12.hostedemail.com (Postfix) with ESMTP id B1FE04000E for ; Mon, 21 Oct 2024 04:22:26 +0000 (UTC) Authentication-Results: imf12.hostedemail.com; dkim=pass header.d=google.com header.s=20230601 header.b=QLln8vYr; dmarc=pass (policy=reject) header.from=google.com; spf=pass (imf12.hostedemail.com: domain of 3CNcVZwYKCE0D9Ewp3v33v0t.r310x29C-11zAprz.36v@flex--yuzhao.bounces.google.com designates 209.85.128.201 as permitted sender) smtp.mailfrom=3CNcVZwYKCE0D9Ewp3v33v0t.r310x29C-11zAprz.36v@flex--yuzhao.bounces.google.com ARC-Seal: i=1; s=arc-20220608; d=hostedemail.com; t=1729484503; a=rsa-sha256; cv=none; b=wnXVvXMYWz0vEpQwwmhu2BvMl/OoTl4fxBhLrZpTqQ5Mx9EZ3kJUXTSf0XxnYZqQ8qqPTy cZ9widnRwFoiYr74kNN57rOICqaVE4DcwXQS2fDmwK3xAKMu9sjmjBsHa0EIKHVpiEbch2 cNlTb2ca2OitmLW23BVuYB1z/3enaN8= ARC-Authentication-Results: i=1; imf12.hostedemail.com; dkim=pass header.d=google.com header.s=20230601 header.b=QLln8vYr; dmarc=pass (policy=reject) header.from=google.com; spf=pass (imf12.hostedemail.com: domain of 3CNcVZwYKCE0D9Ewp3v33v0t.r310x29C-11zAprz.36v@flex--yuzhao.bounces.google.com designates 209.85.128.201 as permitted sender) smtp.mailfrom=3CNcVZwYKCE0D9Ewp3v33v0t.r310x29C-11zAprz.36v@flex--yuzhao.bounces.google.com ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=hostedemail.com; s=arc-20220608; t=1729484503; h=from:from:sender:reply-to:subject:subject:date:date: message-id:message-id:to:to:cc:cc:mime-version:mime-version: content-type:content-type:content-transfer-encoding: in-reply-to:in-reply-to:references:references:dkim-signature; bh=KJDQv9BsbNqPzAGIVHBRqdKJMgT8Ksx982Co8Pglskw=; b=sSo0ysh4vmRbjJHbg0S03cNF8vEZN0LS5wFsZeLHX0pqgzdPGk8IrZK8IgDUHfKuqPireT WpinQZPrxNLh7X8IN7XayuZCjGGuNy3qZu2MQDfo5sCiUtegDmI03KaVmk0jMpstq3v00X Wr73QMTBjdijKdM8KlxTFBY2xgys4EU= Received: by mail-yw1-f201.google.com with SMTP id 00721157ae682-6e3c638cc27so68212947b3.0 for ; Sun, 20 Oct 2024 21:22:33 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=google.com; s=20230601; t=1729484552; x=1730089352; darn=kvack.org; h=cc:to:from:subject:message-id:references:mime-version:in-reply-to :date:from:to:cc:subject:date:message-id:reply-to; bh=KJDQv9BsbNqPzAGIVHBRqdKJMgT8Ksx982Co8Pglskw=; b=QLln8vYr2JTUk/Mr/30ZYBQwucENSjMn5LmGItj6oUWvW+051UlP1t9mJH0xaBmuWB dqDS73D2qkB4xSfrDjMatD7AtvVn8xaPHHknl01GBoMEPjd9Up8JScx+bzWSz2067sHK oqrUzetghU6kvNrGLoOjp+/RNybS6B3jxJL52Tf+nuDlulYxpuA4oJxyXXlhdjmVPk2x 0A3UvjOwtTJD9MNu9R9jeGxMYX7q/Y8NoZBUDwR8loNt4Ax1HunNACyMxOdtFmFoxxse P2EBkGfiEJUVIDBgVirZnPSZG0GKL79BDTmMwcLR4olfnqEL1jLdeCmFaN64X+RdD++t blKQ== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20230601; t=1729484552; x=1730089352; h=cc:to:from:subject:message-id:references:mime-version:in-reply-to :date:x-gm-message-state:from:to:cc:subject:date:message-id:reply-to; bh=KJDQv9BsbNqPzAGIVHBRqdKJMgT8Ksx982Co8Pglskw=; b=Zm7lNcrmRnL+FDblnauBzjamS+4k1fu6hi8y+DUiW2LZTNdd96dA6grU7NN1VkBZ2E zzmpDOiCXN9wSj1p/76kAdj2PVCMLdqt24NQouiyvVweoSDyWuJHRU0IfACYWrqKefen 7VPMBZ18JC1aOHJlL1+FVA58azRTtBnUIyUVHpKvDWT+8XV+zYcH/oUzKcUHqcIC5RD6 xCxaN1B2zUNjYLq6f+KoxWRCukORLSEl9AnpkC0wUDjAcRnpU2+V1gvd8QAc/bYrHioA 7FEL3F+0VVoX+oCdc700n7i1j3jOVK5Be0hGgYZYy67IYoaB1MQFnAM8BIkfb9muQ5pn AeAA== X-Forwarded-Encrypted: i=1; AJvYcCUj4hAq40J8dUsmIUXeSwthi1NswNfLieVx/5CUxzSPL/X1u5XB9yQvpxQIBNq9kLzLdu7zjJbJaw==@kvack.org X-Gm-Message-State: AOJu0YyX49cPpYtLySuxm1a5FXtnChgXxYRFdrdaoyM8GM/a/AMmVBNw MOdWwcZCNzo8QhJJYuuYBz2htFeG8CoED0V7YUzNfUE7MuqlZk93LTRRy2uboDqhlYnNnOQR7l8 kQA== X-Google-Smtp-Source: AGHT+IGEZGtfLFTAo+zlX7imoG9sifk97tpSM8aK6pq0aGlMOVsYXw6fthvIZGCv/EPFEWt1vDf5ZDyKg70= X-Received: from yuzhao2.bld.corp.google.com ([2a00:79e0:2e28:6:1569:9ef4:20ab:abf9]) (user=yuzhao job=sendgmr) by 2002:a05:690c:fc2:b0:6e2:70e:e82e with SMTP id 00721157ae682-6e5bfc0c757mr2067897b3.6.1729484552429; Sun, 20 Oct 2024 21:22:32 -0700 (PDT) Date: Sun, 20 Oct 2024 22:22:16 -0600 In-Reply-To: <20241021042218.746659-1-yuzhao@google.com> Mime-Version: 1.0 References: <20241021042218.746659-1-yuzhao@google.com> X-Mailer: git-send-email 2.47.0.rc1.288.g06298d1525-goog Message-ID: <20241021042218.746659-5-yuzhao@google.com> Subject: [PATCH v1 4/6] arm64: broadcast IPIs to pause remote CPUs From: Yu Zhao To: Andrew Morton , Catalin Marinas , Marc Zyngier , Muchun Song , Thomas Gleixner , Will Deacon Cc: Douglas Anderson , Mark Rutland , Nanyong Sun , linux-arm-kernel@lists.infradead.org, linux-kernel@vger.kernel.org, linux-mm@kvack.org, Yu Zhao X-Rspam-User: X-Rspamd-Queue-Id: B1FE04000E X-Rspamd-Server: rspam01 X-Stat-Signature: ibkq1teetysghmgyi7cnkoky8nsbg3qp X-HE-Tag: 1729484546-827453 X-HE-Meta: U2FsdGVkX18scOwQuyiVLF9K+3Z1V49HS121bd75Z/r5bOA0f+B4+z6pwVz+C3MQKhz++WaN6+r2hFIv6MgIHA6bDybHOOJcLTrckDYn8RDS9KogO5BKZF+s/+we1160o6QA9Y1ZghLlF1W4NLguh29SqjE26vVpXTKNdi2uhbBpzJowW5rhQ8jhxniMo8vknNsfXCkfrJAA2QaPMUneyaI/3upNNxLxLU3PauemmUzSHwTSPSDB3Rf9O83jgjTdR2/0eD7c44KyBPrtC7H+cydj+3AKRPgcwVgFkYV2ZD+FoKKfjY4LYbFgfkmNXHZcTD8TV4a3FJB8B9HJL+czhlu6VSsVeTR6xNL8GDUTncsrhOi+p/6cCanO6AZ86MSJHIoBA9I5VNGlbHwixCfP3KE8S2Yy0+Dhhz94Lp36z4DQhMZi6ArawxOuKGuCNJjGxEOclLr19LM58osSNtO4hYjVF4SqtxDOvvSGpBTJw6q7WPJj29AOuVwLB2lgcQuZiIcgmAr4Um6h63T85o5+mFXgGjpPmutiVWAFKT2+uB1cuM+rAzB8Z9Nkd7eqE5mcLbcQQYQmXOMcqNO3fm/AGLT5GgiINyQq0nuFmMB23Q/PYFwYA6Er0q6ge12SQ7tWPmo7Mwga5Ot0feHx3aGBCgoBaZl3fyHxkNW6j5m+H4n+d6ep9chZnUv0DhdTjoypSm0ZrYZ+pq8nh3Z6tBHXcJ+bKxhuCJQz2Wdc8tS+Vnx6Vafooc8cbzptWs3O2dAiihLBmx0npR9kj1Skn05aBl2pZiG41vpAFGqVK6Nv0t501uIEpna+C00FQsSuDoUknZSd4g1mR5ZL+G6VSQuYftH4ZP4WreJALcaCMsWDAZFkKOK5Y6EITFel4oBPZ4sgJWgKU7THeKRnkuXGqasfykxGHDrvyI+0lxGRX4ygrdlyFaqKL7OWHTE9n/hov4TL+roScIYIw0eX8kXyR30 oZQYX4eV ZZkvX9oRlTD6BalCdyfcqgKB+kTFM1rIFKnIgfIUx1ddArWU3hP5coDoWAMOVm6t+DLsGd2H/3wu3OxfGk20/Ub2yKrAbkWlqbO+/t5C/DorQfXuFkhCOea4enXq8jZU4uAtKBczp52fr2/oYlfTB/rqbXxOTvntRkWQwK+Ki5OSn3FldMFM/YcTD2vgqxRdz10NC/Hem2pTIXo1MNCYxW5bwNlZW/P7p0mq6KGUWPMnREBHYn5OCP3MT0NMDswy9ZIiNee2mS4NjRaKudze9uClZI5406x+rrzSVqN2sIbJxTcsKxow/HiKucSx5cVfbZpVG1dGcTd+cKb/szFDkpnT+x6VvHSWswo+G1B5ALKJl45fKDPQ0zqSREIeeEzD31suZpjwxDOMAa/6TFQSI4u0Er54IWGWvj6KOoBZcy07mv7c8m/TW2z1S0/e52rix3dA6YAkJ8VonP8PN7fh+pQb2MRhsEspb0rNy0zwBnSnJ/YjJTFV/wXM0jvXwy/cyL7Sts4I7eshd68Aoc8NHBIOWALvkMqGySX5RK0y7y2tx5XY= X-Bogosity: Ham, tests=bogofilter, spamicity=0.000000, version=1.2.4 Sender: owner-linux-mm@kvack.org Precedence: bulk X-Loop: owner-majordomo@kvack.org List-ID: List-Subscribe: List-Unsubscribe: Broadcast pseudo-NMI IPIs to pause remote CPUs for a short period of time, and then reliably resume them when the local CPU exits critical sections that preclude the execution of remote CPUs. A typical example of such critical sections is BBM on kernel PTEs. HugeTLB Vmemmap Optimization (HVO) on arm64 was disabled by commit 060a2c92d1b6 ("arm64: mm: hugetlb: Disable HUGETLB_PAGE_OPTIMIZE_VMEMMAP") due to the folllowing reason: This is deemed UNPREDICTABLE by the Arm architecture without a break-before-make sequence (make the PTE invalid, TLBI, write the new valid PTE). However, such sequence is not possible since the vmemmap may be concurrently accessed by the kernel. Supporting BBM on kernel PTEs is one of the approaches that can make HVO theoretically safe on arm64. Note that it is still possible for the paused CPUs to perform speculative translations. Such translations would cause spurious kernel PFs, which should be properly handled by is_spurious_el1_translation_fault(). Signed-off-by: Yu Zhao --- arch/arm64/include/asm/smp.h | 3 ++ arch/arm64/kernel/smp.c | 92 +++++++++++++++++++++++++++++++++--- 2 files changed, 88 insertions(+), 7 deletions(-) diff --git a/arch/arm64/include/asm/smp.h b/arch/arm64/include/asm/smp.h index 2510eec026f7..cffb0cfed961 100644 --- a/arch/arm64/include/asm/smp.h +++ b/arch/arm64/include/asm/smp.h @@ -133,6 +133,9 @@ bool cpus_are_stuck_in_kernel(void); extern void crash_smp_send_stop(void); extern bool smp_crash_stop_failed(void); +void pause_remote_cpus(void); +void resume_remote_cpus(void); + #endif /* ifndef __ASSEMBLY__ */ #endif /* ifndef __ASM_SMP_H */ diff --git a/arch/arm64/kernel/smp.c b/arch/arm64/kernel/smp.c index 3b3f6b56e733..68829c6de1b1 100644 --- a/arch/arm64/kernel/smp.c +++ b/arch/arm64/kernel/smp.c @@ -85,7 +85,12 @@ static int ipi_irq_base __ro_after_init; static int nr_ipi __ro_after_init = NR_IPI; static struct irq_desc *ipi_desc[MAX_IPI] __ro_after_init; -static bool crash_stop; +enum { + SEND_STOP = BIT(0), + CRASH_STOP = BIT(1), +}; + +static unsigned long stop_in_progress; static void ipi_setup(int cpu); @@ -917,6 +922,79 @@ static void __noreturn ipi_cpu_crash_stop(unsigned int cpu, struct pt_regs *regs #endif } +static DEFINE_SPINLOCK(cpu_pause_lock); +static cpumask_t paused_cpus; +static cpumask_t resumed_cpus; + +static void pause_local_cpu(void) +{ + int cpu = smp_processor_id(); + + cpumask_clear_cpu(cpu, &resumed_cpus); + /* + * Paired with pause_remote_cpus() to confirm that this CPU not only + * will be paused but also can be reliably resumed. + */ + smp_wmb(); + cpumask_set_cpu(cpu, &paused_cpus); + /* paused_cpus must be set before waiting on resumed_cpus. */ + barrier(); + while (!cpumask_test_cpu(cpu, &resumed_cpus)) + cpu_relax(); + /* A typical example for sleep and wake-up functions. */ + smp_mb(); + cpumask_clear_cpu(cpu, &paused_cpus); +} + +void pause_remote_cpus(void) +{ + cpumask_t cpus_to_pause; + + lockdep_assert_cpus_held(); + lockdep_assert_preemption_disabled(); + + cpumask_copy(&cpus_to_pause, cpu_online_mask); + cpumask_clear_cpu(smp_processor_id(), &cpus_to_pause); + + spin_lock(&cpu_pause_lock); + + WARN_ON_ONCE(!cpumask_empty(&paused_cpus)); + + smp_cross_call(&cpus_to_pause, IPI_CPU_STOP_NMI); + + while (!cpumask_equal(&cpus_to_pause, &paused_cpus)) + cpu_relax(); + /* + * Paired with pause_local_cpu() to confirm that all CPUs not only will + * be paused but also can be reliably resumed. + */ + smp_rmb(); + WARN_ON_ONCE(cpumask_intersects(&cpus_to_pause, &resumed_cpus)); + + spin_unlock(&cpu_pause_lock); +} + +void resume_remote_cpus(void) +{ + cpumask_t cpus_to_resume; + + lockdep_assert_cpus_held(); + lockdep_assert_preemption_disabled(); + + cpumask_copy(&cpus_to_resume, cpu_online_mask); + cpumask_clear_cpu(smp_processor_id(), &cpus_to_resume); + + spin_lock(&cpu_pause_lock); + + cpumask_setall(&resumed_cpus); + /* A typical example for sleep and wake-up functions. */ + smp_mb(); + while (cpumask_intersects(&cpus_to_resume, &paused_cpus)) + cpu_relax(); + + spin_unlock(&cpu_pause_lock); +} + static void arm64_backtrace_ipi(cpumask_t *mask) { __ipi_send_mask(ipi_desc[IPI_CPU_BACKTRACE], mask); @@ -970,7 +1048,9 @@ static void do_handle_IPI(int ipinr) case IPI_CPU_STOP: case IPI_CPU_STOP_NMI: - if (IS_ENABLED(CONFIG_KEXEC_CORE) && crash_stop) { + if (!test_bit(SEND_STOP, &stop_in_progress)) { + pause_local_cpu(); + } else if (test_bit(CRASH_STOP, &stop_in_progress)) { ipi_cpu_crash_stop(cpu, get_irq_regs()); unreachable(); } else { @@ -1142,7 +1222,6 @@ static inline unsigned int num_other_online_cpus(void) void smp_send_stop(void) { - static unsigned long stop_in_progress; cpumask_t mask; unsigned long timeout; @@ -1154,7 +1233,7 @@ void smp_send_stop(void) goto skip_ipi; /* Only proceed if this is the first CPU to reach this code */ - if (test_and_set_bit(0, &stop_in_progress)) + if (test_and_set_bit(SEND_STOP, &stop_in_progress)) return; /* @@ -1230,12 +1309,11 @@ void crash_smp_send_stop(void) * This function can be called twice in panic path, but obviously * we execute this only once. * - * We use this same boolean to tell whether the IPI we send was a + * We use the CRASH_STOP bit to tell whether the IPI we send was a * stop or a "crash stop". */ - if (crash_stop) + if (test_and_set_bit(CRASH_STOP, &stop_in_progress)) return; - crash_stop = 1; smp_send_stop(); From patchwork Mon Oct 21 04:22:17 2024 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Yu Zhao X-Patchwork-Id: 13843544 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from kanga.kvack.org (kanga.kvack.org [205.233.56.17]) by smtp.lore.kernel.org (Postfix) with ESMTP id 0F0B8D3C933 for ; Mon, 21 Oct 2024 04:22:39 +0000 (UTC) Received: by kanga.kvack.org (Postfix) id 113ED6B0095; Mon, 21 Oct 2024 00:22:38 -0400 (EDT) Received: by kanga.kvack.org (Postfix, from userid 40) id 0C5526B0096; Mon, 21 Oct 2024 00:22:38 -0400 (EDT) X-Delivered-To: int-list-linux-mm@kvack.org Received: by kanga.kvack.org (Postfix, from userid 63042) id EA6E36B0098; Mon, 21 Oct 2024 00:22:37 -0400 (EDT) X-Delivered-To: linux-mm@kvack.org Received: from relay.hostedemail.com (smtprelay0016.hostedemail.com [216.40.44.16]) by kanga.kvack.org (Postfix) with ESMTP id C298B6B0095 for ; Mon, 21 Oct 2024 00:22:37 -0400 (EDT) Received: from smtpin13.hostedemail.com (a10.router.float.18 [10.200.18.1]) by unirelay08.hostedemail.com (Postfix) with ESMTP id E911C141605 for ; Mon, 21 Oct 2024 04:22:21 +0000 (UTC) X-FDA: 82696312482.13.92ED3AD Received: from mail-yb1-f201.google.com (mail-yb1-f201.google.com [209.85.219.201]) by imf12.hostedemail.com (Postfix) with ESMTP id 156F14000E for ; Mon, 21 Oct 2024 04:22:28 +0000 (UTC) Authentication-Results: imf12.hostedemail.com; dkim=pass header.d=google.com header.s=20230601 header.b=XxSsBpXJ; dmarc=pass (policy=reject) header.from=google.com; spf=pass (imf12.hostedemail.com: domain of 3CtcVZwYKCE8FBGyr5x55x2v.t532z4BE-331Crt1.58x@flex--yuzhao.bounces.google.com designates 209.85.219.201 as permitted sender) smtp.mailfrom=3CtcVZwYKCE8FBGyr5x55x2v.t532z4BE-331Crt1.58x@flex--yuzhao.bounces.google.com ARC-Seal: i=1; s=arc-20220608; d=hostedemail.com; t=1729484519; a=rsa-sha256; cv=none; b=RAxiArMo6O6XNskFidA2RDGJZ7RcJcDSoBI89QD220zfNKy+yuf4gp0vmfWtNtbDTNk2tW jd5M5qenOVdm4YLspFmlhU3s5snpyXS2xJ99WBVOyS/R6gySH8BL3b5vis+hNT4apyhtPa HhgT9TM6EIuxXdTMD3DpcKlZUMddLKo= ARC-Authentication-Results: i=1; imf12.hostedemail.com; dkim=pass header.d=google.com header.s=20230601 header.b=XxSsBpXJ; dmarc=pass (policy=reject) header.from=google.com; spf=pass (imf12.hostedemail.com: domain of 3CtcVZwYKCE8FBGyr5x55x2v.t532z4BE-331Crt1.58x@flex--yuzhao.bounces.google.com designates 209.85.219.201 as permitted sender) smtp.mailfrom=3CtcVZwYKCE8FBGyr5x55x2v.t532z4BE-331Crt1.58x@flex--yuzhao.bounces.google.com ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=hostedemail.com; s=arc-20220608; t=1729484519; h=from:from:sender:reply-to:subject:subject:date:date: message-id:message-id:to:to:cc:cc:mime-version:mime-version: content-type:content-type:content-transfer-encoding: in-reply-to:in-reply-to:references:references:dkim-signature; bh=toVG9Bv1ZFXM9w/cWVF/q6arTJn/fxOOJss0eISMw6g=; b=VGgqNntGH62+Ch0Ws7c41iFo+jdsjWHyU3uopkqOmhyZ9AKsheIXeAUnpYpVpw7znyi2f2 wtm3EDxr+DNq5iqWP1owcvrVGgszYmowb39bhLP6iSLlS3yMeWCgewEmzqM4m9TisT+tzk wkH1oIcLm0fkC7xSHSK9M6m8Dk7rhik= Received: by mail-yb1-f201.google.com with SMTP id 3f1490d57ef6-e293150c2c6so8378662276.1 for ; Sun, 20 Oct 2024 21:22:35 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=google.com; s=20230601; t=1729484555; x=1730089355; darn=kvack.org; h=cc:to:from:subject:message-id:references:mime-version:in-reply-to :date:from:to:cc:subject:date:message-id:reply-to; bh=toVG9Bv1ZFXM9w/cWVF/q6arTJn/fxOOJss0eISMw6g=; b=XxSsBpXJn0hWnEbW6lIoteEbYB+8hG6wQwBVXRxsMoAtJVuCJKjbKxXuygnYvYYa22 u5R2OjwlIHCkqonFX7VnY4avcviA2F0EkkUboF50hhYy7iYAUJ9serlCo0TGjsGDQO/y mrtl2p9IK27ZZT1q/3iGrcgszFVCA1TGvaZ36kzAvul1mugJ0ENYntVQ71fxIh5XB5zC 6veOa4y+rfsZDIO4PxQ5AbA5NyPCyPXu6igPuFeAnSB5uaJAfTgpgvnyx7bq9MSTmLEu v1sdWpCljeVYeSpq+o8IYHCkt+gpvISWG4L9muU+HUezt5TmefpIy1DXXuf9N15H/+Ku WPAA== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20230601; t=1729484555; x=1730089355; h=cc:to:from:subject:message-id:references:mime-version:in-reply-to :date:x-gm-message-state:from:to:cc:subject:date:message-id:reply-to; bh=toVG9Bv1ZFXM9w/cWVF/q6arTJn/fxOOJss0eISMw6g=; b=b8KdonWmZVOspyAutlWnhkA4h1P8Vj/6798PGzxSrA17UCeIh6nk4gIIAxOClNooiX I+8VVRt8muidsuom/zMNLh+hfvmG/RXu1cWNrEc8vUUyP4wSB0f27wY0vuPG15fmt1yJ Ss1/Z18HvR7Jq0Dk9JXREIZexSUi2zEayjehYHSCDv27cf0eLFwPfWAHb+u2nMuTK14S Cf89yKx1bdNZc5QHoAd++CP+9lKtEOs7hvqz2nPA8d4ij0amUPO3IHE0cIgvMFQ2DJIs oxoyfK3d/644vBZvcYvi8kVubiZDl9HA1+qcedYpkaji86AN7CJ7xV6HzTZg073221Si ZKGg== X-Forwarded-Encrypted: i=1; AJvYcCUJJ9VEYFRCCD9OWyxTJOdzEYEy5+fuQ/k6ZrBYYi3NweHPzFjdBDemE0WGLQUkM63PFDtpC914VQ==@kvack.org X-Gm-Message-State: AOJu0Yz59sDXQjBUI49jdvcKkQlsk/XdCdFVT+ndcG/Niy+bZ8GdJ5ee Gi39EhGEQ4aJFYOyauGysMDtRP0hQMBNIcAHHmwGy3T2O1yxR6zhRoRaLSkjWXoUbgUzB6SjRzh lYA== X-Google-Smtp-Source: AGHT+IFk9uamhIyVPiPVkC9AAkl4R2cZvpHrX9RuV4B2/rBtrreDBZ8gNEYfOVDCArI/WEZD+WVYAn+KZ5o= X-Received: from yuzhao2.bld.corp.google.com ([2a00:79e0:2e28:6:1569:9ef4:20ab:abf9]) (user=yuzhao job=sendgmr) by 2002:a25:7453:0:b0:e28:eee0:aaa1 with SMTP id 3f1490d57ef6-e2bb12db7e5mr5668276.4.1729484554722; Sun, 20 Oct 2024 21:22:34 -0700 (PDT) Date: Sun, 20 Oct 2024 22:22:17 -0600 In-Reply-To: <20241021042218.746659-1-yuzhao@google.com> Mime-Version: 1.0 References: <20241021042218.746659-1-yuzhao@google.com> X-Mailer: git-send-email 2.47.0.rc1.288.g06298d1525-goog Message-ID: <20241021042218.746659-6-yuzhao@google.com> Subject: [PATCH v1 5/6] arm64: pause remote CPUs to update vmemmap From: Yu Zhao To: Andrew Morton , Catalin Marinas , Marc Zyngier , Muchun Song , Thomas Gleixner , Will Deacon Cc: Douglas Anderson , Mark Rutland , Nanyong Sun , linux-arm-kernel@lists.infradead.org, linux-kernel@vger.kernel.org, linux-mm@kvack.org, Yu Zhao X-Rspam-User: X-Stat-Signature: 1ycjaynp8c9bzapuc1jzo8om6kq6sfjs X-Rspamd-Queue-Id: 156F14000E X-Rspamd-Server: rspam02 X-HE-Tag: 1729484548-926349 X-HE-Meta: U2FsdGVkX1+i9sIgUN+sL1LdXqT6/OGELYaZ2Xpb2jIGDz3N4Qur5G5+SwayiGgvsGZzHcnSFmXRKEtqy7kPbkgcp0ajZJRK0UCmSH8Bds2hXOzXr2FkyGtcFYMQT7sH9yKhVagSOV+zhY5nSeSvGf+yLfKZSrblQ18NE4Yzy4WH9MIdw0JrHHijqm3BhOB8ZJ6XeKHjNFIWz6wjyPuRRKvkqFNWqY/GT2jMyW2SCjsA4MkrOG8HOHjP1ZK9lL7Xvt+dXRfUCkSG7DkRBtT7otjUj1pW+gG7YbuCDIkH9/mJA8WcUoJMgt5mLVbWlMtYDlkYLq/1ZZx+HFsW5y3DjB53s9vDjJyl71UmjaGfL4sXMBTLv1NT3w8erXIa8A7yMJcY3HTeV6duvGVttu20OVnwVhv4lii/zVFu18RYlafskKXxuUEIPbhm/uJhqsvd4UkKxJXF8tJU9dd0r1yxhv8K1eaPj9yq7JYyKE9bqSajjYs5yNh2K3HCtkRUka6C2FeAzuCLCe2I6+tkUicZlJSxwWFas4P9BzvanB061PNytQqe0t3YqAMYU5llP1peYSgR9McqfSLPb9TmpW078oqJFPEIs1RMmtRMoeTiz/q9V1kjvIHXT01iIUNvtm0RCJXiRbvFiAOQOO1Sv57RYeqUmpKzDYPm7Y195VhWlowY7esXV7cTNoekNM1rjLKIubwolTiSyp64jXVFG7jQl0mRYuiEXn+8giuM7ckiVo3unWkZm4RMDGhexU0CBdrbjjTZpoRMJqfIGZH9nPOsJlFB9oO6zhWERhECxHj044hUDbFHu3BuTQbd8schJ3gI1dV5jNotdixFITK4rhOQV1muhMJV1j4fJR6QO5griuNHsSY/F6dR7S/Jl2d8DYnN+sjWwYF8W72ismCJN/R27DJiylxOu3Kfqma+FA8xasPD0i4ezzoo1qj/yyEiqiQWnLbd8N2zjepMJmm1Y8P vdLTkoBR LbOey4kuppD3PR18dj/wn9RL0suJtf7GkBqpjYUMwcP9KJ1lptVu7sXJKaprXqJRjDGtoHsgEwn2A2pya3s/Z91GuuYNfGRia5VUa2tDSW/MxVXLqAL4Ql+Q+mxyuN/ZWQT23dmqN+Wh6VygloHPbJNOqnz8lstKAq37H2dYgDciDLu2RblNURAy2NN2dCNcRgnG6d/J08Gs3eNK56ZSFckZd8Hc6r8vhWD06djjSczoD+7T3lGg+HO/8EvrFLz2gKCxLRz1LjjcwGOj8qmKXBtHwHVEBl0shP21jR/8e8YKrGIFvjtm8so2bqmM0EdaBqG82izVo/EMMq+obiLEQjJqTW2IOJ5klsTkvMQUOO0yqqs67nRkiAAwbyAlC77HLq1fUCcB8wl590IBcjRF/6X1eQxRmUBX28Wahh59WXqn6YafMaxpU3WMzNvFHYg6OEiYg/p45kTmYWGoT8JXL24cn3GaUNTM1p+fBJz0t0OPh2k5K2wnOCc0M5OXbou5CSCFUAusn2mdcCutTJRQKknwdTKd+3wtidBugr5BnCgdoBXq5iS//hAUCDkMcwHudEz8tUMCDICjl1YE= X-Bogosity: Ham, tests=bogofilter, spamicity=0.000000, version=1.2.4 Sender: owner-linux-mm@kvack.org Precedence: bulk X-Loop: owner-majordomo@kvack.org List-ID: List-Subscribe: List-Unsubscribe: Pause remote CPUs so that the local CPU can follow the proper BBM sequence to safely update the vmemmap mapping `struct page` areas. While updating the vmemmap, it is guaranteed that neither the local CPU nor the remote ones will access the `struct page` area being updated, and therefore they should not trigger (non-spurious) kernel PFs. Signed-off-by: Yu Zhao --- arch/arm64/include/asm/pgalloc.h | 69 ++++++++++++++++++++++++++++++++ 1 file changed, 69 insertions(+) diff --git a/arch/arm64/include/asm/pgalloc.h b/arch/arm64/include/asm/pgalloc.h index 8ff5f2a2579e..f50f79f57c1e 100644 --- a/arch/arm64/include/asm/pgalloc.h +++ b/arch/arm64/include/asm/pgalloc.h @@ -12,6 +12,7 @@ #include #include #include +#include #define __HAVE_ARCH_PGD_FREE #define __HAVE_ARCH_PUD_FREE @@ -137,4 +138,72 @@ pmd_populate(struct mm_struct *mm, pmd_t *pmdp, pgtable_t ptep) __pmd_populate(pmdp, page_to_phys(ptep), PMD_TYPE_TABLE | PMD_TABLE_PXN); } +#ifdef CONFIG_HUGETLB_PAGE_OPTIMIZE_VMEMMAP + +#define VMEMMAP_ARCH_TLB_FLUSH_FLAGS (VMEMMAP_SPLIT_NO_TLB_FLUSH | VMEMMAP_REMAP_NO_TLB_FLUSH) + +#define vmemmap_update_supported vmemmap_update_supported +static inline bool vmemmap_update_supported(void) +{ + return system_uses_irq_prio_masking(); +} + +#define vmemmap_update_lock vmemmap_update_lock +static inline void vmemmap_update_lock(void) +{ + cpus_read_lock(); +} + +#define vmemmap_update_unlock vmemmap_update_unlock +static inline void vmemmap_update_unlock(void) +{ + cpus_read_unlock(); +} + +#define vmemmap_update_pte_range_start vmemmap_update_pte_range_start +static inline void vmemmap_update_pte_range_start(pte_t *pte, + unsigned long start, unsigned long end) +{ + unsigned long addr; + + local_irq_disable(); + pause_remote_cpus(); + + for (addr = start; addr != end; addr += PAGE_SIZE, pte++) + pte_clear(&init_mm, addr, pte); + + flush_tlb_kernel_range(start, end); +} + +#define vmemmap_update_pte_range_end vmemmap_update_pte_range_end +static inline void vmemmap_update_pte_range_end(void) +{ + resume_remote_cpus(); + local_irq_enable(); +} + +#define vmemmap_update_pmd_range_start vmemmap_update_pmd_range_start +static inline void vmemmap_update_pmd_range_start(pmd_t *pmd, + unsigned long start, unsigned long end) +{ + unsigned long addr; + + local_irq_disable(); + pause_remote_cpus(); + + for (addr = start; addr != end; addr += PMD_SIZE, pmd++) + pmd_clear(pmd); + + flush_tlb_kernel_range(start, end); +} + +#define vmemmap_update_pmd_range_end vmemmap_update_pmd_range_end +static inline void vmemmap_update_pmd_range_end(void) +{ + resume_remote_cpus(); + local_irq_enable(); +} + +#endif /* CONFIG_HUGETLB_PAGE_OPTIMIZE_VMEMMAP */ + #endif From patchwork Mon Oct 21 04:22:18 2024 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Yu Zhao X-Patchwork-Id: 13843545 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from kanga.kvack.org (kanga.kvack.org [205.233.56.17]) by smtp.lore.kernel.org (Postfix) with ESMTP id C302CD3C93E for ; Mon, 21 Oct 2024 04:22:41 +0000 (UTC) Received: by kanga.kvack.org (Postfix) id 902AB6B0098; Mon, 21 Oct 2024 00:22:40 -0400 (EDT) Received: by kanga.kvack.org (Postfix, from userid 40) id 8B3F46B0099; Mon, 21 Oct 2024 00:22:40 -0400 (EDT) X-Delivered-To: int-list-linux-mm@kvack.org Received: by kanga.kvack.org (Postfix, from userid 63042) id 707F36B009A; Mon, 21 Oct 2024 00:22:40 -0400 (EDT) X-Delivered-To: linux-mm@kvack.org Received: from relay.hostedemail.com (smtprelay0011.hostedemail.com [216.40.44.11]) by kanga.kvack.org (Postfix) with ESMTP id 4F88B6B0098 for ; Mon, 21 Oct 2024 00:22:40 -0400 (EDT) Received: from smtpin19.hostedemail.com (a10.router.float.18 [10.200.18.1]) by unirelay09.hostedemail.com (Postfix) with ESMTP id 2C7CD815A1 for ; Mon, 21 Oct 2024 04:22:27 +0000 (UTC) X-FDA: 82696312650.19.DC91C3A Received: from mail-yw1-f202.google.com (mail-yw1-f202.google.com [209.85.128.202]) by imf15.hostedemail.com (Postfix) with ESMTP id D0252A000A for ; Mon, 21 Oct 2024 04:22:23 +0000 (UTC) Authentication-Results: imf15.hostedemail.com; dkim=pass header.d=google.com header.s=20230601 header.b=Ya8qsin8; spf=pass (imf15.hostedemail.com: domain of 3DdcVZwYKCFIIEJ1u808805y.w86527EH-664Fuw4.8B0@flex--yuzhao.bounces.google.com designates 209.85.128.202 as permitted sender) smtp.mailfrom=3DdcVZwYKCFIIEJ1u808805y.w86527EH-664Fuw4.8B0@flex--yuzhao.bounces.google.com; dmarc=pass (policy=reject) header.from=google.com ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=hostedemail.com; s=arc-20220608; t=1729484408; h=from:from:sender:reply-to:subject:subject:date:date: message-id:message-id:to:to:cc:cc:mime-version:mime-version: content-type:content-type:content-transfer-encoding: in-reply-to:in-reply-to:references:references:dkim-signature; bh=0qOoUc7S7Ph4MJcNIkBb7u3tw8zxiypBu0RBb7iGSGQ=; b=JwNtFJM9DgJO4ugn3sdFfK1R36FiuvKNo+RdCTDIbRD6dQMMezdNrjczTHWkwDhCckgEci bUPx4bMRq9Gea/XVmjpRqs3RAVkjjbSqmdQ0WtZtba64cHf0jgvQIi81znll1/2GJJRuYW sB1ZhXyvU1p803YKXeeUMT+dL/Fuwe4= ARC-Seal: i=1; s=arc-20220608; d=hostedemail.com; t=1729484408; a=rsa-sha256; cv=none; b=6rADBf0dEosr9c6P5lispk47oRaKrTJ4JjPu+ZN3zoxHTS3iTGiU71bDBda0ia6QF0rH34 hgbOrZi1XdzW+Je1a6i9/zOESLCxoZh9+3VnP0f42w8OjU/SyOZWbVU+/oMAAaVb1H+FSJ PCXN1nt4G2NubOWcTFJspMbQALZOnHM= ARC-Authentication-Results: i=1; imf15.hostedemail.com; dkim=pass header.d=google.com header.s=20230601 header.b=Ya8qsin8; spf=pass (imf15.hostedemail.com: domain of 3DdcVZwYKCFIIEJ1u808805y.w86527EH-664Fuw4.8B0@flex--yuzhao.bounces.google.com designates 209.85.128.202 as permitted sender) smtp.mailfrom=3DdcVZwYKCFIIEJ1u808805y.w86527EH-664Fuw4.8B0@flex--yuzhao.bounces.google.com; dmarc=pass (policy=reject) header.from=google.com Received: by mail-yw1-f202.google.com with SMTP id 00721157ae682-6e383ba6763so67843667b3.3 for ; Sun, 20 Oct 2024 21:22:37 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=google.com; s=20230601; t=1729484557; x=1730089357; darn=kvack.org; h=cc:to:from:subject:message-id:references:mime-version:in-reply-to :date:from:to:cc:subject:date:message-id:reply-to; bh=0qOoUc7S7Ph4MJcNIkBb7u3tw8zxiypBu0RBb7iGSGQ=; b=Ya8qsin8MM4gX/lhIVoPGnnYcZAOgg4rO9hMHMiK2zZ7o60skffA9/x8vH8kvslKTZ /cjVkoADnPVG6ntkMC8cdc756XujYmpw8geo1Px20mzwDRizO1/sUokX0H4MOcl5pd0q KG42naNs2X/UYn5hml2OXYZdbdcs4gvrl2u1onaHSsWJhQVXqdz7OIXTFp3TmJh0hLjO vPksx3yON1NQX0uOYduUPKKuUFtxFwe/jdLNxWP+7plyhjpdIMGpg/pCZzVykubM8Lmn 4xz4xAHr1VAyeJRGKiVFHStAO91P7pWBoJhw5iMleHbeUz4XtJCxXSCMppsmHVPU+47w 1UMA== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20230601; t=1729484557; x=1730089357; h=cc:to:from:subject:message-id:references:mime-version:in-reply-to :date:x-gm-message-state:from:to:cc:subject:date:message-id:reply-to; bh=0qOoUc7S7Ph4MJcNIkBb7u3tw8zxiypBu0RBb7iGSGQ=; b=pm5KunCJIPnq1nDfldCjOPBcUYtHZfZhpp7/vG7ZnOrnWzLOqlLAl38YKM66jQqrX/ 3UJzjT+LUAc2K+djtNQ31R6nBKhGou8/idVdtrEMwsu1N76KfK+9cz++9WDz4s3hFf+Q KIRuUGA6x9f2mqgvotiquiAoqvrKBcw+W2cNe3APV01AMV+kYBqsdmHXMiJlzovIhwJj J82TWGnlb4jmRVUp0qT8uuV/Wf2vYkrskSkcakUVRF/PD4LvzRxO5zoJWgTQAgydJbjW 6qYl359o3WuWbtaMJ+2k7Psk0+BPrkpAJkv/8ME8uUjpVcduqTYdvyiACHJq4wcNsFNA FEPg== X-Forwarded-Encrypted: i=1; AJvYcCW01otjxn5CFrSNUkjmRacZPLuMUdw/yLsYbhhI39crsgIL5DFOJv8e5v8mdZreH+k9/jIsQOGEdw==@kvack.org X-Gm-Message-State: AOJu0YwfcWzeP64UqTaVBGMUpk9LM3Vahj+M68IRt7edizczCExTi4Z1 0gVHIYffvjuRiakoOGJiWNf1jNYqv7yj61e47FGCH5kfiZCXjPpTUqd4XLc8kRekc/C0man8CCD y8g== X-Google-Smtp-Source: AGHT+IHT0kbssKqnqvzVXYJ6z8GdpmqmvaKobmcxDyj7IjWvvjuKhF8UkhPdehMt/EKyLWoXTgeCtaMmNCE= X-Received: from yuzhao2.bld.corp.google.com ([2a00:79e0:2e28:6:1569:9ef4:20ab:abf9]) (user=yuzhao job=sendgmr) by 2002:a25:c781:0:b0:e29:7cd6:593b with SMTP id 3f1490d57ef6-e2bb169afa4mr24157276.8.1729484557262; Sun, 20 Oct 2024 21:22:37 -0700 (PDT) Date: Sun, 20 Oct 2024 22:22:18 -0600 In-Reply-To: <20241021042218.746659-1-yuzhao@google.com> Mime-Version: 1.0 References: <20241021042218.746659-1-yuzhao@google.com> X-Mailer: git-send-email 2.47.0.rc1.288.g06298d1525-goog Message-ID: <20241021042218.746659-7-yuzhao@google.com> Subject: [PATCH v1 6/6] arm64: select ARCH_WANT_OPTIMIZE_HUGETLB_VMEMMAP From: Yu Zhao To: Andrew Morton , Catalin Marinas , Marc Zyngier , Muchun Song , Thomas Gleixner , Will Deacon Cc: Douglas Anderson , Mark Rutland , Nanyong Sun , linux-arm-kernel@lists.infradead.org, linux-kernel@vger.kernel.org, linux-mm@kvack.org, Yu Zhao X-Stat-Signature: njb5tqs3jzhmkkbu5cp45oxp4qmh5onh X-Rspamd-Queue-Id: D0252A000A X-Rspam-User: X-Rspamd-Server: rspam08 X-HE-Tag: 1729484543-880123 X-HE-Meta: U2FsdGVkX1/FDaUFS1ENxi5V5dMUP66pNrHVLkIY7l3IClzrZYhn/LxzjjdnyksSLcYIJ80kLRZ5h755auZ3etf81HxF2TfdHOissEDHsvgk1sfPEHNhdtfIB9qDjcAyGVzzUl0nyCMxLsUoM9Bh50vXewi/D74VlgPyuGApujECk7WEIZOJApIhfEwabmX7KbhhwFWidSX12YC1sWWYMs+AMwAscv80tT5+nm3EgySqY5yLLrMxNyFaveyjb5dCujd1F8BNZiJxUDp9Li9LWlVXp5Ox7wgDoOBlyw6sfymV2bwTSPh8zO0+X4b+sLc1f5SOIxXrSjydMe5Xq1agVKOzrKaZ/CMidNgJNQ8z4LwuAcXsXnaMgNlIBDG7Q0fPnMf4Ju4HImYAyxRTlwmzJbPB8v4Zm1YwlmNKjL0GbJZDVCsz1sxHduZybIUO2k/+BjcDYo05a73L46hQdlcHaOtQ5BLob/hdd5PwKKjF++RkAYr28SH2QDDZNFOVlgSSfLFVoGN9oOvORLerqAG6nJyZH7ClpWvyp9i0Jkq1TmJuDq6AWzANzVPIqzWPkyeh4GGYmInqkjPtQLFfuc3dkVnaGDeNSlWWyaBue2TOqHly6HqgmedPX2LsvqTLQ+RVQiwNbPdDEnTVIfBcmXWy4wpB6bI+5qjSv3wcrAWx8hU0lcIBPGMHR4wrHpSGRKUBLMN9P71VVlr5XobvI6MIpXnZHqy4vTfM4AJ1l1LKkj4sLh+xJWZPq0VN4xmOX4/vK3FZlrBIK0dCZzhG3bGxnVlafByBrArbudyfz8FH9OJ/3H3pUZEEv3Rm3hl9CiIGnRZFFn/ARyZhgzmchmsIXYPv64UT/EUgFugjp01CRzuQjmeByy+V29zmlyyGHMNtp8D1+/kGVFRw2B28FJK73KPVfGOK34KIOSGCZl2JIcd6TeEhAG/31yW2vV+D8Gk3LvyJz7ebjWlXQkaGz3I GhnnOi+V 3hzE/A7BpCXbnpnErUGvZOWhPtFfvjddcrohyc7X8DfJSeEIzhOqIfeVeI2uiNBghopfQ8n+t7z2cI3glS6dIksjUKiaImGLBPCsfdDb5b6fpH+01S+qIJHB7e2+EzxYMpZrHwEXqAIac+P3VpPLeij3qZIz2mYL0BykTM7hEiCuFOGjAp+KQYXJI+jyRsF78LR3kWs5H9zSQff9IR2YSkwEd9AMn6PzHpqqEngU0vM6UJbjezTZvFbFwv8R3WUZHAjLUsCrA4qbMC+X4MrXOTf/5Jm+IMvgR5DbdiYRi9wpT9tYLebDmmN2dfcANhox7vWRLskP6HUTxGoqDszZV0IKp2M6Y+9L/LSaJustIIg2OwLA+L1se/ZMD64LkuyLifLnCGaWl6uBW3BZE8QngoWQ7TbjicEyXF9PyahO+wSQiZqohUDXSTdQCU7SWpFDWPbr81jSlXgouOniOOek73hmnllbyMZID6mOP662HjHRg6/bKD+c6jubRG0L096BS5sKqGNQkrcMaE7aWUCJLJGogx2ePh9NenstE/5YP430qbLXgD0Nkr5nqm6mHkWkfiOjLE/YEgFK2hhmwWHhS5w0Ipw== X-Bogosity: Ham, tests=bogofilter, spamicity=0.000000, version=1.2.4 Sender: owner-linux-mm@kvack.org Precedence: bulk X-Loop: owner-majordomo@kvack.org List-ID: List-Subscribe: List-Unsubscribe: To use HVO, make sure that the kernel is booted with pseudo-NMI enabled by "irqchip.gicv3_pseudo_nmi=1", as well as "hugetlb_free_vmemmap=on" unless HVO is enabled by default. Note that HVO checks the pseudo-NMI capability and is disabled at runtime if the capability turns out not supported. Successfully enabling HVO should have the following: # dmesg | grep NMI GICv3: Pseudo-NMIs enabled using ... # sysctl vm.hugetlb_optimize_vmemmap vm.hugetlb_optimize_vmemmap = 1 Signed-off-by: Yu Zhao --- arch/arm64/Kconfig | 1 + 1 file changed, 1 insertion(+) diff --git a/arch/arm64/Kconfig b/arch/arm64/Kconfig index fd9df6dcc593..e93745f819d9 100644 --- a/arch/arm64/Kconfig +++ b/arch/arm64/Kconfig @@ -109,6 +109,7 @@ config ARM64 select ARCH_WANT_DEFAULT_TOPDOWN_MMAP_LAYOUT select ARCH_WANT_FRAME_POINTERS select ARCH_WANT_HUGE_PMD_SHARE if ARM64_4K_PAGES || (ARM64_16K_PAGES && !ARM64_VA_BITS_36) + select ARCH_WANT_OPTIMIZE_HUGETLB_VMEMMAP select ARCH_WANT_LD_ORPHAN_WARN select ARCH_WANTS_EXECMEM_LATE if EXECMEM select ARCH_WANTS_NO_INSTR