From patchwork Mon Nov 27 08:46:43 2023 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Muchun Song X-Patchwork-Id: 13469265 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from kanga.kvack.org (kanga.kvack.org [205.233.56.17]) by smtp.lore.kernel.org (Postfix) with ESMTP id E29E3C4167B for ; Mon, 27 Nov 2023 08:47:29 +0000 (UTC) Received: by kanga.kvack.org (Postfix) id 7F00A6B0327; Mon, 27 Nov 2023 03:47:29 -0500 (EST) Received: by kanga.kvack.org (Postfix, from userid 40) id 79F8F6B0328; Mon, 27 Nov 2023 03:47:29 -0500 (EST) X-Delivered-To: int-list-linux-mm@kvack.org Received: by kanga.kvack.org (Postfix, from userid 63042) id 6690B6B0329; Mon, 27 Nov 2023 03:47:29 -0500 (EST) X-Delivered-To: linux-mm@kvack.org Received: from relay.hostedemail.com (smtprelay0010.hostedemail.com [216.40.44.10]) by kanga.kvack.org (Postfix) with ESMTP id 538DF6B0327 for ; Mon, 27 Nov 2023 03:47:29 -0500 (EST) Received: from smtpin20.hostedemail.com (a10.router.float.18 [10.200.18.1]) by unirelay05.hostedemail.com (Postfix) with ESMTP id 143B24017D for ; Mon, 27 Nov 2023 08:47:29 +0000 (UTC) X-FDA: 81503105418.20.0085930 Received: from mail-pf1-f169.google.com (mail-pf1-f169.google.com [209.85.210.169]) by imf13.hostedemail.com (Postfix) with ESMTP id 41D2C20003 for ; Mon, 27 Nov 2023 08:47:27 +0000 (UTC) Authentication-Results: imf13.hostedemail.com; dkim=pass header.d=bytedance.com header.s=google header.b=hb4hRqz0; spf=pass (imf13.hostedemail.com: domain of songmuchun@bytedance.com designates 209.85.210.169 as permitted sender) smtp.mailfrom=songmuchun@bytedance.com; dmarc=pass (policy=quarantine) header.from=bytedance.com ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=hostedemail.com; s=arc-20220608; t=1701074847; h=from:from:sender:reply-to:subject:subject:date:date: message-id:message-id:to:to:cc:cc:mime-version:mime-version: content-type:content-transfer-encoding:content-transfer-encoding: in-reply-to:in-reply-to:references:references:dkim-signature; bh=u6GtJdFrAMiD9TMiTJ2tpU0p4EP2T7FpthDyFgbUgV8=; b=M/2k9mI9m4Hz35Q73oNjAHGS/RtDpvg+gL8I+6Fok10+nkOvy7lZxMO+f4Fdmx4r3zMDl6 gzyLvAceRErJuOFzWZHjROaKZjd5guOkzEEeFP0wCHm2cmEaMSwn6SARqRZdqAHUKbM/eL jUqB/WpQCCzXszWYg2BpaiGpaQcLfVo= ARC-Seal: i=1; s=arc-20220608; d=hostedemail.com; t=1701074847; a=rsa-sha256; cv=none; b=lO4HdmkPFylHl4EsKiCjNi1iB+ZmF1HH0TSRRkJD+bo4vaooAhPaM5ktYl2Pyufy26TCLu 9qmV38YBA+hH3kH64xGwm2i5LJqZpE8ff90IRVW4HjgTTHWbM/lQGdDqsVEoWMBb4grTgj Oc+oCI6DOLlJ47rTtUOBtXmfoRIUefo= ARC-Authentication-Results: i=1; imf13.hostedemail.com; dkim=pass header.d=bytedance.com header.s=google header.b=hb4hRqz0; spf=pass (imf13.hostedemail.com: domain of songmuchun@bytedance.com designates 209.85.210.169 as permitted sender) smtp.mailfrom=songmuchun@bytedance.com; dmarc=pass (policy=quarantine) header.from=bytedance.com Received: by mail-pf1-f169.google.com with SMTP id d2e1a72fcca58-6cbd24d9557so2680813b3a.1 for ; Mon, 27 Nov 2023 00:47:26 -0800 (PST) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=bytedance.com; s=google; t=1701074846; x=1701679646; darn=kvack.org; h=content-transfer-encoding:mime-version:references:in-reply-to :message-id:date:subject:cc:to:from:from:to:cc:subject:date :message-id:reply-to; bh=u6GtJdFrAMiD9TMiTJ2tpU0p4EP2T7FpthDyFgbUgV8=; b=hb4hRqz0rPjXOJVHbV7RZFT3FHug6s8Q8Fmt/8FgyPm+JMzH4Lf8roD+JVHSEyMDaG se2qU0sP8Z5y9jIrhH2rvOzJkPiAupmTjuxDooL/NRcOY4CUSkDSYoOqCYtf2Bo4u+IT F0EUQRF7C3aIf1eR4/PsV9rDEfl4vyI2MTHTYZxx7uXPgVNvAObGh3iaQlzNUU+HXJVj 1QKNOWd/sUBiBNF++Bq8Y1zGyFYRx98DfCte76VPwiIGt/47skvNC0GL2VYgy5QFYdsm FOzpDgXno0QFLVXAA3hd1/Vn6KIr68KJScJl4vbclV0zq3ncgrTq7N3b/qXbJVtTmyZc lztA== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20230601; t=1701074846; x=1701679646; h=content-transfer-encoding:mime-version:references:in-reply-to :message-id:date:subject:cc:to:from:x-gm-message-state:from:to:cc :subject:date:message-id:reply-to; bh=u6GtJdFrAMiD9TMiTJ2tpU0p4EP2T7FpthDyFgbUgV8=; b=MYsSP3h5X/tXpBIT5TWR5ak6LVo60wW3kvvQYVXZcIqX1MpHaLKnxa5tWiot5isJGT whr7ztCrbWyJSHAp98//VZp73sSImclxK3pg7fV2C6/wBaJqzgpZLhf5PECTxoVTlTn8 F7ZDkYE2vA7SPnb8/5yz0BcBssHzH41yXAjMsplRIoOyeZvLJbFNL0YB9ZyJnuRhCkwP 1IuMo2QHJwDc30ZIXVtDRL0JUnS56Oq0RokEXk20zfD2Su+UR76aMlILz+FrUP4InxRk K0X1zJ50NzzdVdCwrKfzktbSD/47WPhlxij4Xt/h+lMs2i5KtODanVcJ6iMxZyPVel0o tpKw== X-Gm-Message-State: AOJu0YyqiwRSahEvldw8FtQvSNpAV1fwQ9jnG0WvKaQNhzqSALe+TM87 6uSsqeyXb099wdcnabLWy4e96g== X-Google-Smtp-Source: AGHT+IH8dlIQBo06oYLZpoUw9SLI1+YxmO7I/y/nkByjLUJqY3NGvBpQZM1zhaq6gplUxS6RY2JFSg== X-Received: by 2002:a05:6a20:9390:b0:18c:5178:9649 with SMTP id x16-20020a056a20939000b0018c51789649mr7667438pzh.14.1701074846205; Mon, 27 Nov 2023 00:47:26 -0800 (PST) Received: from PXLDJ45XCM.bytedance.net ([139.177.225.230]) by smtp.gmail.com with ESMTPSA id e22-20020aa78c56000000b006c875abecbcsm6686932pfd.121.2023.11.27.00.47.23 (version=TLS1_3 cipher=TLS_CHACHA20_POLY1305_SHA256 bits=256/256); Mon, 27 Nov 2023 00:47:25 -0800 (PST) From: Muchun Song To: mike.kravetz@oracle.com, muchun.song@linux.dev, akpm@linux-foundation.org Cc: linux-mm@kvack.org, linux-kernel@vger.kernel.org, Muchun Song Subject: [PATCH 2/4] mm: hugetlb_vmemmap: use walk_page_range_novma() to simplify the code Date: Mon, 27 Nov 2023 16:46:43 +0800 Message-Id: <20231127084645.27017-3-songmuchun@bytedance.com> X-Mailer: git-send-email 2.39.3 (Apple Git-145) In-Reply-To: <20231127084645.27017-1-songmuchun@bytedance.com> References: <20231127084645.27017-1-songmuchun@bytedance.com> MIME-Version: 1.0 X-Rspamd-Queue-Id: 41D2C20003 X-Rspam-User: X-Rspamd-Server: rspam11 X-Stat-Signature: 1a85diep4kyf44sg5f6ismn5bmh73jyw X-HE-Tag: 1701074847-58755 X-HE-Meta: U2FsdGVkX1+Ck+gbJ+nvaBoFCxTDFX/s1/7mPMilhMzW4gr72XNnSzHDlj2ozgab9pzS2rSUinKLXctiL8Xr8Ociibue90s8VWgmd6EBFE5ujY04JpHtuoZiN9tUW5baTb5Laa9ldliZZgLhZkhDi3IZMjDmcPwbxw5cNE5GK+xJY5eQ+H9cbAhvsSdPWSFRCXHa/33DLFVeVnLA1ACFbQ6ifEyipVTQaGvthhS5GeVL+3e2WWA1nbpYPUYA9jhTXq1XAqlyF6iQ0zEClGcoEQ+lxkSFt7H1hVMUPwKHe96IlA+6rCoaD47sppuubfjCfcixiWHtg3Hnov++/Kf/Etj/hNye2xj12ZImvY/LpGtMlecnB4TKV81JiOG1EEojHldfffcPD50P19OBf2c1JwPHJF20qO1gT4BOmimVxOYjLj0hrBgQYMTi0iYbpyfYnDVyF5lzX+hQGV8wer9g2DMNpWSm9ku+WwoHB2054xZI6P3LeQl8YGepRDaQlo9bf/U9StDw8JDGrnsLIlYt1gHqCQK4uC74HliUrlC7rqD8ua7+/fb4EI2gwmRFTjKqkXc8roFMApfnFAneA2KlUZCcB7qXCDHPTHinEhdSZR7vRmJDtun+d/LhVHeW8wfH2MdI7S6lXJRfrWtmfgR6Tlhvx1UaELkxUl/QsXrquWZR63/RfOFudZ98LPw8Dd8/c0nJB79ZWpgTuM2HMv60oGXyWontSbHcLTjHh6s910z//WghuR1t8AWF7YRJFQwQhOn5kmzfr9rkbNbmEKdHt5Li1kf2/n3xQWr+p8Fn+RSYtDWy36BUtJz1gsaK1diQbb7pPQjC4coNyjaB7COt8npSU4ScPPvViLIjaPeD0U++ZebomX2qIxOb8xlKIhFlEFU5QC+Fld7fBQJ3z5O+aLbeb2Azm6EX1azch6vvJY3vbz356odfRUbQIKfY7/Wzya5seAIWn/VPsdU5BN1 RJmY5dPF XTLhsluM0mGaML5uwnRS0Ivl8avWApPMMgkWIdsgchFyBAIrUXVu+R4GBetHsxjbb46gr/vbGmPH/YmthDjPAwVn+P2mwvvjv3HjDkgddkqfBS7JGTip6CvnH2sfr3FLCsK/53lrl1SJFqv42B5yn6ndsqbI9eGKSJUnuNKRDnj/oGD02LzK/ekjgkdGI+HN0NrwjMYfZa62ogXfPmLkbTRMyfwotmXbQSSgmLMBEApOTbJugyL1M6k0NL7ft+0YfzzcFhLp1QA+MXNkKyI1N23MJc2UkW65szt8yK2k0AQLVzojHjjQjY2yr+g3/zvPB8LTSXE6s12K4RD5Pf7SB4BhI3N9c5+ZHMQjGqYVx27hiGNG7mRNmoTjh45I//MoSIz1/e/nLYcHvd4q7NiEzEn4Cc63A4myxl6SqDqTg/vFZUYvJV7ZtQy2ukvDQq06My/y8kO4rpv5wF60= X-Bogosity: Ham, tests=bogofilter, spamicity=0.000000, version=1.2.4 Sender: owner-linux-mm@kvack.org Precedence: bulk X-Loop: owner-majordomo@kvack.org List-ID: List-Subscribe: List-Unsubscribe: It is unnecessary to implement a series of dedicated page table walking helpers since there is already a general one walk_page_range_novma(). So use it to simplify the code. Signed-off-by: Muchun Song Reviewed-by: Mike Kravetz --- mm/hugetlb_vmemmap.c | 148 ++++++++++++------------------------------- 1 file changed, 39 insertions(+), 109 deletions(-) diff --git a/mm/hugetlb_vmemmap.c b/mm/hugetlb_vmemmap.c index 87818ee7f01d7..ef14356855d13 100644 --- a/mm/hugetlb_vmemmap.c +++ b/mm/hugetlb_vmemmap.c @@ -14,6 +14,7 @@ #include #include #include +#include #include #include #include "hugetlb_vmemmap.h" @@ -45,21 +46,14 @@ struct vmemmap_remap_walk { unsigned long flags; }; -static int split_vmemmap_huge_pmd(pmd_t *pmd, unsigned long start, bool flush) +static int vmemmap_split_pmd(pmd_t *pmd, struct page *head, unsigned long start, + struct vmemmap_remap_walk *walk) { pmd_t __pmd; int i; unsigned long addr = start; - struct page *head; pte_t *pgtable; - spin_lock(&init_mm.page_table_lock); - head = pmd_leaf(*pmd) ? pmd_page(*pmd) : NULL; - spin_unlock(&init_mm.page_table_lock); - - if (!head) - return 0; - pgtable = pte_alloc_one_kernel(&init_mm); if (!pgtable) return -ENOMEM; @@ -88,7 +82,7 @@ static int split_vmemmap_huge_pmd(pmd_t *pmd, unsigned long start, bool flush) /* Make pte visible before pmd. See comment in pmd_install(). */ smp_wmb(); pmd_populate_kernel(&init_mm, pmd, pgtable); - if (flush) + if (!(walk->flags & VMEMMAP_SPLIT_NO_TLB_FLUSH)) flush_tlb_kernel_range(start, start + PMD_SIZE); } else { pte_free_kernel(&init_mm, pgtable); @@ -98,123 +92,59 @@ static int split_vmemmap_huge_pmd(pmd_t *pmd, unsigned long start, bool flush) return 0; } -static void vmemmap_pte_range(pmd_t *pmd, unsigned long addr, - unsigned long end, - struct vmemmap_remap_walk *walk) -{ - pte_t *pte = pte_offset_kernel(pmd, addr); - - /* - * The reuse_page is found 'first' in table walk before we start - * remapping (which is calling @walk->remap_pte). - */ - if (!walk->reuse_page) { - walk->reuse_page = pte_page(ptep_get(pte)); - /* - * Because the reuse address is part of the range that we are - * walking, skip the reuse address range. - */ - addr += PAGE_SIZE; - pte++; - walk->nr_walked++; - } - - for (; addr != end; addr += PAGE_SIZE, pte++) { - walk->remap_pte(pte, addr, walk); - walk->nr_walked++; - } -} - -static int vmemmap_pmd_range(pud_t *pud, unsigned long addr, - unsigned long end, - struct vmemmap_remap_walk *walk) +static int vmemmap_pmd_entry(pmd_t *pmd, unsigned long addr, + unsigned long next, struct mm_walk *walk) { - pmd_t *pmd; - unsigned long next; - - pmd = pmd_offset(pud, addr); - do { - int ret; - - ret = split_vmemmap_huge_pmd(pmd, addr & PMD_MASK, - !(walk->flags & VMEMMAP_SPLIT_NO_TLB_FLUSH)); - if (ret) - return ret; + struct page *head; + struct vmemmap_remap_walk *vmemmap_walk = walk->private; - next = pmd_addr_end(addr, end); + /* Only splitting, not remapping the vmemmap pages. */ + if (!vmemmap_walk->remap_pte) + walk->action = ACTION_CONTINUE; - /* - * We are only splitting, not remapping the hugetlb vmemmap - * pages. - */ - if (!walk->remap_pte) - continue; - - vmemmap_pte_range(pmd, addr, next, walk); - } while (pmd++, addr = next, addr != end); + spin_lock(&init_mm.page_table_lock); + head = pmd_leaf(*pmd) ? pmd_page(*pmd) : NULL; + spin_unlock(&init_mm.page_table_lock); + if (!head) + return 0; - return 0; + return vmemmap_split_pmd(pmd, head, addr & PMD_MASK, vmemmap_walk); } -static int vmemmap_pud_range(p4d_t *p4d, unsigned long addr, - unsigned long end, - struct vmemmap_remap_walk *walk) +static int vmemmap_pte_entry(pte_t *pte, unsigned long addr, + unsigned long next, struct mm_walk *walk) { - pud_t *pud; - unsigned long next; - - pud = pud_offset(p4d, addr); - do { - int ret; + struct vmemmap_remap_walk *vmemmap_walk = walk->private; - next = pud_addr_end(addr, end); - ret = vmemmap_pmd_range(pud, addr, next, walk); - if (ret) - return ret; - } while (pud++, addr = next, addr != end); + /* + * The reuse_page is found 'first' in page table walking before + * starting remapping. + */ + if (!vmemmap_walk->reuse_page) + vmemmap_walk->reuse_page = pte_page(ptep_get(pte)); + else + vmemmap_walk->remap_pte(pte, addr, vmemmap_walk); + vmemmap_walk->nr_walked++; return 0; } -static int vmemmap_p4d_range(pgd_t *pgd, unsigned long addr, - unsigned long end, - struct vmemmap_remap_walk *walk) -{ - p4d_t *p4d; - unsigned long next; - - p4d = p4d_offset(pgd, addr); - do { - int ret; - - next = p4d_addr_end(addr, end); - ret = vmemmap_pud_range(p4d, addr, next, walk); - if (ret) - return ret; - } while (p4d++, addr = next, addr != end); - - return 0; -} +static const struct mm_walk_ops vmemmap_remap_ops = { + .pmd_entry = vmemmap_pmd_entry, + .pte_entry = vmemmap_pte_entry, +}; static int vmemmap_remap_range(unsigned long start, unsigned long end, struct vmemmap_remap_walk *walk) { - unsigned long addr = start; - unsigned long next; - pgd_t *pgd; - - VM_BUG_ON(!PAGE_ALIGNED(start)); - VM_BUG_ON(!PAGE_ALIGNED(end)); + int ret; - pgd = pgd_offset_k(addr); - do { - int ret; + VM_BUG_ON(!PAGE_ALIGNED(start | end)); - next = pgd_addr_end(addr, end); - ret = vmemmap_p4d_range(pgd, addr, next, walk); - if (ret) - return ret; - } while (pgd++, addr = next, addr != end); + ret = walk_page_range_novma(&init_mm, start, end, &vmemmap_remap_ops, + NULL, walk); + if (ret) + return ret; if (walk->remap_pte && !(walk->flags & VMEMMAP_REMAP_NO_TLB_FLUSH)) flush_tlb_kernel_range(start, end);