From patchwork Thu Jan 5 10:18:09 2023 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: James Houghton X-Patchwork-Id: 13089646 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from kanga.kvack.org (kanga.kvack.org [205.233.56.17]) by smtp.lore.kernel.org (Postfix) with ESMTP id 9B527C53210 for ; Thu, 5 Jan 2023 10:19:12 +0000 (UTC) Received: by kanga.kvack.org (Postfix) id 3C500940009; Thu, 5 Jan 2023 05:19:12 -0500 (EST) Received: by kanga.kvack.org (Postfix, from userid 40) id 34E3B940008; Thu, 5 Jan 2023 05:19:12 -0500 (EST) X-Delivered-To: int-list-linux-mm@kvack.org Received: by kanga.kvack.org (Postfix, from userid 63042) id 1C773940009; Thu, 5 Jan 2023 05:19:12 -0500 (EST) X-Delivered-To: linux-mm@kvack.org Received: from relay.hostedemail.com (smtprelay0010.hostedemail.com [216.40.44.10]) by kanga.kvack.org (Postfix) with ESMTP id 0DCC2940008 for ; Thu, 5 Jan 2023 05:19:12 -0500 (EST) Received: from smtpin01.hostedemail.com (a10.router.float.18 [10.200.18.1]) by unirelay10.hostedemail.com (Postfix) with ESMTP id CDD00C060F for ; Thu, 5 Jan 2023 10:19:11 +0000 (UTC) X-FDA: 80320347702.01.83606DC Received: from mail-yb1-f202.google.com (mail-yb1-f202.google.com [209.85.219.202]) by imf20.hostedemail.com (Postfix) with ESMTP id 41CCA1C0014 for ; Thu, 5 Jan 2023 10:19:10 +0000 (UTC) Authentication-Results: imf20.hostedemail.com; dkim=pass header.d=google.com header.s=20210112 header.b=iSz2ooNk; dmarc=pass (policy=reject) header.from=google.com; spf=pass (imf20.hostedemail.com: domain of 3HaS2YwoKCGQLVJQWIJVQPIQQING.EQONKPWZ-OOMXCEM.QTI@flex--jthoughton.bounces.google.com designates 209.85.219.202 as permitted sender) smtp.mailfrom=3HaS2YwoKCGQLVJQWIJVQPIQQING.EQONKPWZ-OOMXCEM.QTI@flex--jthoughton.bounces.google.com ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=hostedemail.com; s=arc-20220608; t=1672913950; h=from:from:sender:reply-to:subject:subject:date:date: message-id:message-id:to:to:cc:cc:mime-version:mime-version: content-type:content-type:content-transfer-encoding: in-reply-to:in-reply-to:references:references:dkim-signature; bh=HlgWOjaAJz0zfdNn9cG/30nMIJO7QLF2mBB2gxe7Hq8=; b=fwr9m1AslEwuL23NUlS8ML2eOvfGnTgkDP4OWStl6q4RiRFWZLO0rncuglDNpBSR/3qat2 6lOBMP2/BdLsiypuD/Xz69KM8PUGpwhEeNU/jGb4lajnmTHHF2f7uZymXajtdwlV8FAV0i c6NOOjSB9lVkNsMUtsjLGVZanPgwE44= ARC-Authentication-Results: i=1; imf20.hostedemail.com; dkim=pass header.d=google.com header.s=20210112 header.b=iSz2ooNk; dmarc=pass (policy=reject) header.from=google.com; spf=pass (imf20.hostedemail.com: domain of 3HaS2YwoKCGQLVJQWIJVQPIQQING.EQONKPWZ-OOMXCEM.QTI@flex--jthoughton.bounces.google.com designates 209.85.219.202 as permitted sender) smtp.mailfrom=3HaS2YwoKCGQLVJQWIJVQPIQQING.EQONKPWZ-OOMXCEM.QTI@flex--jthoughton.bounces.google.com ARC-Seal: i=1; s=arc-20220608; d=hostedemail.com; t=1672913950; a=rsa-sha256; cv=none; b=mUTYhASDmGVUen8aUtwHyST+yi6NTVq+1IfIuH30f0zUH/76t0hnIB587fYknyVtUuSGHX xFWmttiuiGJU0tWixWCd9pphcHfkJt66vX5Q4HsoV6/Nw2M4IM9T4qU2qIB7TNDSKtTe3G kYyYhs3VY4jpboE60E/mky8KE8ij1l4= Received: by mail-yb1-f202.google.com with SMTP id w9-20020a05690210c900b007b20e8d0c99so4516671ybu.0 for ; Thu, 05 Jan 2023 02:19:10 -0800 (PST) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=google.com; s=20210112; h=cc:to:from:subject:message-id:references:mime-version:in-reply-to :date:from:to:cc:subject:date:message-id:reply-to; bh=HlgWOjaAJz0zfdNn9cG/30nMIJO7QLF2mBB2gxe7Hq8=; b=iSz2ooNkUZGnH87sDmHD2gCBj2Hey3YSf423VRaH6AnWyafV22fMV2Sw9PR3D4uqic CllJH2wRMj7/tkZJT/L4HrkszREidz8xo8elOcL/ubDPrXdbjKqcvSwNuBpfIvAGVtvR A5ldBvPdHR0ntED8Zu2n7yVUxfVZ9fS2FUgGqGsf9I7fh5DevYo9DWvQ14UI5opS2Ndf 9pyDdSpxBL6nohGJDqj9v21g+a96iXBzAfKohulMwY8yNne3UYL+LmR1Lm/zk3AAVII7 QovdH+ZXsH6oWLiYCENZhk3nLT5IEtiB909u4krXh79gZLNok0fXXDBeFzffTBDi9IPZ p31Q== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20210112; h=cc:to:from:subject:message-id:references:mime-version:in-reply-to :date:x-gm-message-state:from:to:cc:subject:date:message-id:reply-to; bh=HlgWOjaAJz0zfdNn9cG/30nMIJO7QLF2mBB2gxe7Hq8=; b=p9sdLwdhO3CJnQ4ECVB2lrQjGXqOIqeZbb/ppwKCNIFzKO6xkgsDfAXoNOVLf8DPqs JsbCpZ5QAu81dTZRENskxQzhYQrel5lEIiAOmQGqPYPOn0HGQiUx/1bIz+3W1KYjkOi4 7Oirt8WoJFENEZMZhveK8aAxH/kiMuywJd9O85Q5wjLQqtgyY5e4H/lUSHpbIfdvKXJg BCQCvu9c6cTSetqBHqi0IpNh2LRZAXlgS8/oHhtTte7GAwFJ/nZ+CqBAaM1y6P3lfn7Q YPb1mSo1kIct7GQ34rmyFJZkjkeaGeySZqtaNHqZu8bHJMpaGUx2NCBRx4NOL+tNq/Xp Xa8w== X-Gm-Message-State: AFqh2kpQhBqDd7JbYiuJKo+ZSk09HKzA8cuHy6aPQZ3iIeL5sT3CCH7D EAL0SDNoJ/FoszRJ3jsCrJVlkKmO0w7rYQ9n X-Google-Smtp-Source: AMrXdXuWXwTFKt3tEGwcWkM+8Z5EJbSUAOSB4/uJq2eHaPtcD22bcM4r//hqZZqHWIknAkMmLTX/CoDS7t5HOrA7 X-Received: from jthoughton.c.googlers.com ([fda3:e722:ac3:cc00:14:4d90:c0a8:2a4f]) (user=jthoughton job=sendgmr) by 2002:a25:42c2:0:b0:758:bd9:8a70 with SMTP id p185-20020a2542c2000000b007580bd98a70mr4286087yba.377.1672913949470; Thu, 05 Jan 2023 02:19:09 -0800 (PST) Date: Thu, 5 Jan 2023 10:18:09 +0000 In-Reply-To: <20230105101844.1893104-1-jthoughton@google.com> Mime-Version: 1.0 References: <20230105101844.1893104-1-jthoughton@google.com> X-Mailer: git-send-email 2.39.0.314.g84b9a713c41-goog Message-ID: <20230105101844.1893104-12-jthoughton@google.com> Subject: [PATCH 11/46] hugetlb: add hugetlb_pte to track HugeTLB page table entries From: James Houghton To: Mike Kravetz , Muchun Song , Peter Xu Cc: David Hildenbrand , David Rientjes , Axel Rasmussen , Mina Almasry , "Zach O'Keefe" , Manish Mishra , Naoya Horiguchi , "Dr . David Alan Gilbert" , "Matthew Wilcox (Oracle)" , Vlastimil Babka , Baolin Wang , Miaohe Lin , Yang Shi , Andrew Morton , linux-mm@kvack.org, linux-kernel@vger.kernel.org, James Houghton X-Rspamd-Queue-Id: 41CCA1C0014 X-Rspamd-Server: rspam09 X-Rspam-User: X-Stat-Signature: qrgdk736cg4pin4yh9z65gzxhzsowmt6 X-HE-Tag: 1672913950-588659 X-HE-Meta: U2FsdGVkX1+22A9UQDJ2t92ttjG/Ui2yOowcOkJSE8ddUdaIjZgrONc1ncAiQUjQPfz9EyiKe6j+qijotuFmxJjd9uqk9NwsAVqdgrDdvztWKffSWG+ZJwZleAUl3leTlP4XBnFkZeCBkZ3jcbn7hX69TIVV46odS6MJsLFBtmdN6aIVW+oPYWhj+XBhTlbfUoWPtVOcdD3SVt8d9iRvp5rXAKBTZ1kdFnm62AJC0s8bbOQ4TMn0uRbGrzObVhU+tbb+RCefD+s7UsPS3v1vrf36Y499fqmFWlh8wAOnuLWodAcfcZDoegcY7GRwi6bfB8K5e2qVJPtROrnDBnp2RhoU5L8QSpcqU6jRYptw8L4xaCt3uyNUxArJFVSfMrQwS0SwHY/Lz1bIfTit3+8jgiNOjS6V1K/nfm8oiy/5GwDDMeCNiqg1qKnGWAxSVP4VnvLp2tNnyU/rMT7rmNGjMZYvtwdK3rXp5T3rzp/jQTI77PC0BRnTROOo6VACo6u4Sby0x26WGg1M3+GyPeS1wqrHzoA8OQJE+fMlKxIefy50CmB1+Leq9R4X7NQH3SaM12N3Bw3dzHQhNgY0ibHCAoaVMC0HsEfDZAP18Mpy6FbTU3N5fYLNwpMIV0U5TLDjC7oD9WKag+qmcDDbEEehRUSSZuvtC/1R1DorJBLIi6rgJ1SS1SdM8zON2x7KIRzIaKGejrbSoJrJny0fXyrsIhuJWRQdeabP+0IFgLokxte80tAM1DMitMR2gEexKI6SZ0Enip2hW4bAbXVC6Yx5pgceBClee5VW7TLMJNw2DcRbVX8hcjjvaXJVhHZGZEj0gRr/A9R5uWEdgpCVdfaIKuKLUqezSSoVY0GReS2rNrw7P4iKUq3w/sJtCag1iLR3WFcQ5L+oZsXAmUw/GQDwqwroT8GBD3hEhWfklnb0ZctF45nqE/vffPt0GDLx8ebIFA2P6+CyU1/FX9H4FDC bsR/sgiF YwGTrQvVL12R++vmNxUh+PFkUX0RSOCGh6TAgKsKWqJt5wgR5zpMAQaobkvUwXZYg5Uorb+sbCNvUY9syDgU7LCgJly8X6qQFJt4SsPfTMLbB2pPspZFjSwg8IDYU20BsthLSMGloxbH/eL6r2D+7ttfS3GI8dot23jwNTxA5CutJCiVUHxbwMo9C3kxmhENzKuoa25n1Fw6XuWWj1mphPH/Qhnet1Z+WWATy8Gft0/CaiucGr7Npc8jm55Nms2chPHyX2d+F86vmmnVvnEfBKPlIvj1p8KlO5XJv/3IflizR8dvq/0q5BUOBdhuAV/3e7D3MT1bLjNiB3AU8lTNJ02mlLk+uhHtZKSWJAHzqvd+VP+4NvLfn46wstcj4yO120z2wZtPuNYSZIpbhuHZ7qEdiBYj67CP0R1A0Cv25LAV8aPa3cO9jS4C9/i7J2mHKp3D6 X-Bogosity: Ham, tests=bogofilter, spamicity=0.000000, version=1.2.4 Sender: owner-linux-mm@kvack.org Precedence: bulk X-Loop: owner-majordomo@kvack.org List-ID: After high-granularity mapping, page table entries for HugeTLB pages can be of any size/type. (For example, we can have a 1G page mapped with a mix of PMDs and PTEs.) This struct is to help keep track of a HugeTLB PTE after we have done a page table walk. Without this, we'd have to pass around the "size" of the PTE everywhere. We effectively did this before; it could be fetched from the hstate, which we pass around pretty much everywhere. hugetlb_pte_present_leaf is included here as a helper function that will be used frequently later on. Signed-off-by: James Houghton --- include/linux/hugetlb.h | 72 +++++++++++++++++++++++++++++++++++++++++ mm/hugetlb.c | 29 +++++++++++++++++ 2 files changed, 101 insertions(+) diff --git a/include/linux/hugetlb.h b/include/linux/hugetlb.h index 3f098363cd6e..bf441d8a1b52 100644 --- a/include/linux/hugetlb.h +++ b/include/linux/hugetlb.h @@ -38,6 +38,54 @@ typedef struct { unsigned long pd; } hugepd_t; */ #define __NR_USED_SUBPAGE 3 +enum hugetlb_level { + HUGETLB_LEVEL_PTE = 1, + /* + * We always include PMD, PUD, and P4D in this enum definition so that, + * when logged as an integer, we can easily tell which level it is. + */ + HUGETLB_LEVEL_PMD, + HUGETLB_LEVEL_PUD, + HUGETLB_LEVEL_P4D, + HUGETLB_LEVEL_PGD, +}; + +struct hugetlb_pte { + pte_t *ptep; + unsigned int shift; + enum hugetlb_level level; + spinlock_t *ptl; +}; + +static inline +void __hugetlb_pte_populate(struct hugetlb_pte *hpte, pte_t *ptep, + unsigned int shift, enum hugetlb_level level, + spinlock_t *ptl) +{ + /* + * If 'shift' indicates that this PTE is contiguous, then @ptep must + * be the first pte of the contiguous bunch. + */ + hpte->ptl = ptl; + hpte->ptep = ptep; + hpte->shift = shift; + hpte->level = level; +} + +static inline +unsigned long hugetlb_pte_size(const struct hugetlb_pte *hpte) +{ + return 1UL << hpte->shift; +} + +static inline +unsigned long hugetlb_pte_mask(const struct hugetlb_pte *hpte) +{ + return ~(hugetlb_pte_size(hpte) - 1); +} + +bool hugetlb_pte_present_leaf(const struct hugetlb_pte *hpte, pte_t pte); + struct hugepage_subpool { spinlock_t lock; long count; @@ -1232,6 +1280,30 @@ static inline spinlock_t *huge_pte_lock(struct hstate *h, return ptl; } +static inline +spinlock_t *hugetlb_pte_lockptr(struct hugetlb_pte *hpte) +{ + return hpte->ptl; +} + +static inline +spinlock_t *hugetlb_pte_lock(struct hugetlb_pte *hpte) +{ + spinlock_t *ptl = hugetlb_pte_lockptr(hpte); + + spin_lock(ptl); + return ptl; +} + +static inline +void hugetlb_pte_populate(struct mm_struct *mm, struct hugetlb_pte *hpte, + pte_t *ptep, unsigned int shift, + enum hugetlb_level level) +{ + __hugetlb_pte_populate(hpte, ptep, shift, level, + huge_pte_lockptr(shift, mm, ptep)); +} + #if defined(CONFIG_HUGETLB_PAGE) && defined(CONFIG_CMA) extern void __init hugetlb_cma_reserve(int order); #else diff --git a/mm/hugetlb.c b/mm/hugetlb.c index 4db38dc79d0e..2d83a2c359a2 100644 --- a/mm/hugetlb.c +++ b/mm/hugetlb.c @@ -1266,6 +1266,35 @@ static bool vma_has_reserves(struct vm_area_struct *vma, long chg) return false; } +bool hugetlb_pte_present_leaf(const struct hugetlb_pte *hpte, pte_t pte) +{ + pgd_t pgd; + p4d_t p4d; + pud_t pud; + pmd_t pmd; + + switch (hpte->level) { + case HUGETLB_LEVEL_PGD: + pgd = __pgd(pte_val(pte)); + return pgd_present(pgd) && pgd_leaf(pgd); + case HUGETLB_LEVEL_P4D: + p4d = __p4d(pte_val(pte)); + return p4d_present(p4d) && p4d_leaf(p4d); + case HUGETLB_LEVEL_PUD: + pud = __pud(pte_val(pte)); + return pud_present(pud) && pud_leaf(pud); + case HUGETLB_LEVEL_PMD: + pmd = __pmd(pte_val(pte)); + return pmd_present(pmd) && pmd_leaf(pmd); + case HUGETLB_LEVEL_PTE: + return pte_present(pte); + default: + WARN_ON_ONCE(1); + return false; + } +} + + static void enqueue_hugetlb_folio(struct hstate *h, struct folio *folio) { int nid = folio_nid(folio);