From patchwork Sat Feb 18 00:28:09 2023 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: James Houghton X-Patchwork-Id: 13145402 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from kanga.kvack.org (kanga.kvack.org [205.233.56.17]) by smtp.lore.kernel.org (Postfix) with ESMTP id 00162C05027 for ; Sat, 18 Feb 2023 00:29:49 +0000 (UTC) Received: by kanga.kvack.org (Postfix) id 5229E280021; Fri, 17 Feb 2023 19:29:23 -0500 (EST) Received: by kanga.kvack.org (Postfix, from userid 40) id 4D474280002; Fri, 17 Feb 2023 19:29:23 -0500 (EST) X-Delivered-To: int-list-linux-mm@kvack.org Received: by kanga.kvack.org (Postfix, from userid 63042) id 2841F280021; Fri, 17 Feb 2023 19:29:23 -0500 (EST) X-Delivered-To: linux-mm@kvack.org Received: from relay.hostedemail.com (smtprelay0015.hostedemail.com [216.40.44.15]) by kanga.kvack.org (Postfix) with ESMTP id 15794280002 for ; Fri, 17 Feb 2023 19:29:23 -0500 (EST) Received: from smtpin08.hostedemail.com (a10.router.float.18 [10.200.18.1]) by unirelay10.hostedemail.com (Postfix) with ESMTP id E861BC0271 for ; Sat, 18 Feb 2023 00:29:22 +0000 (UTC) X-FDA: 80478528564.08.7910E00 Received: from mail-yb1-f202.google.com (mail-yb1-f202.google.com [209.85.219.202]) by imf22.hostedemail.com (Postfix) with ESMTP id 38990C000C for ; Sat, 18 Feb 2023 00:29:21 +0000 (UTC) Authentication-Results: imf22.hostedemail.com; dkim=pass header.d=google.com header.s=20210112 header.b=Kd84UzoU; spf=pass (imf22.hostedemail.com: domain of 34BvwYwoKCP0oymtzlmytslttlqj.htrqnsz2-rrp0fhp.twl@flex--jthoughton.bounces.google.com designates 209.85.219.202 as permitted sender) smtp.mailfrom=34BvwYwoKCP0oymtzlmytslttlqj.htrqnsz2-rrp0fhp.twl@flex--jthoughton.bounces.google.com; dmarc=pass (policy=reject) header.from=google.com ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=hostedemail.com; s=arc-20220608; t=1676680161; h=from:from:sender:reply-to:subject:subject:date:date: message-id:message-id:to:to:cc:cc:mime-version:mime-version: content-type:content-type:content-transfer-encoding: in-reply-to:in-reply-to:references:references:dkim-signature; bh=HPj8If6i9Ge054c2yi88NtPUxX7MkhEBgezIKsLW8z8=; b=aAHmP7SZ5G1Y6WWJBjzfTktxNZaLdE34CPAU7hguYS0P4p5tiMLUVKdgUF3RUPfwXovOyn ZWxEM+Vqk/behy6gI3lEaHVp//ef58SPKuFuFNBI9EwdfDHvkxCAZUNF6bxmOoeVwMIJTu EkiiBj3KNj0GPYzwLb58sVAdbZ3FyVM= ARC-Authentication-Results: i=1; imf22.hostedemail.com; dkim=pass header.d=google.com header.s=20210112 header.b=Kd84UzoU; spf=pass (imf22.hostedemail.com: domain of 34BvwYwoKCP0oymtzlmytslttlqj.htrqnsz2-rrp0fhp.twl@flex--jthoughton.bounces.google.com designates 209.85.219.202 as permitted sender) smtp.mailfrom=34BvwYwoKCP0oymtzlmytslttlqj.htrqnsz2-rrp0fhp.twl@flex--jthoughton.bounces.google.com; dmarc=pass (policy=reject) header.from=google.com ARC-Seal: i=1; s=arc-20220608; d=hostedemail.com; t=1676680161; a=rsa-sha256; cv=none; b=eNF/HbgIG3EEjG9cvRUsl1Xh6aUbsMOr7NfVlpmLk+RFwgv+5k+pN3NEImVySyfaCGv9Ud RHQGfAY2QNEwmy9xMI3YxsboYloAvG8bVEtQPmhTnIpJ+KoisHFXN42mGD0b2nBwtm2R5g KX843IAebe5TzwIIGFWUyNkSOrr1urg= Received: by mail-yb1-f202.google.com with SMTP id e128-20020a253786000000b009433a21be0dso2008651yba.19 for ; Fri, 17 Feb 2023 16:29:20 -0800 (PST) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=google.com; s=20210112; h=cc:to:from:subject:message-id:references:mime-version:in-reply-to :date:from:to:cc:subject:date:message-id:reply-to; bh=HPj8If6i9Ge054c2yi88NtPUxX7MkhEBgezIKsLW8z8=; b=Kd84UzoUTRPGM2pruNTbVhhDlhlYmwG9wMjKAkrrxmwn/CJAyxsS6jZ/LzaQVwYC3z S69dpVgrGNsnQ2VWeasqchjAp3jDCn9n8jxsJDYFL1UCwLksmO5Kw+6uUwl7zSSgvaCl CihuI8Yf+eChioQnTWfweqD4fr36yuVoZcedwO4ze5S8nNa5G9axS81vDsPaWdtLrroV 8UuCTi7HBc8+DG1q9YpGK/r09QKm4gOgG0V2UzM7mQ71xE6K5hJlSxLD1cNbFDXLleV5 yEaFpVguoifalvGf8MAWgne1cnF863zM85vjZWzRVitIqTT0Ms2eNQkJwx5cA7n7LbdD 9FRQ== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20210112; h=cc:to:from:subject:message-id:references:mime-version:in-reply-to :date:x-gm-message-state:from:to:cc:subject:date:message-id:reply-to; bh=HPj8If6i9Ge054c2yi88NtPUxX7MkhEBgezIKsLW8z8=; b=C1AF1COEO2OrUd/s6xqzlM1cc+Pm4HcpHE1805jt2MpinE+Hr9FJzT9jCIUm/My3SF t19QC12ro0EEQWq/Q7aARviyYhdad310swQwxFuq0VO/zfKENdiEeID3oMHbhMObO6CR phqNHEJdT2kjDsxLOkNilbjB9K1IVOm3P7HMTwAQt6EJ8VTRWrxaRzIciSmuZkHG9mIp pLxYaoVNRrmc+KJ2FjPMc3pyJqDk43YgY8RXfqDYoQFeu7NHYNstgFRDNZDFbyTAd31h PejASGcWLBicHM252WiIYIQMdFkyMrw+66qrnZ9J/CRWq/+9FN5zvGlzS7A7bifZdHkB pWtQ== X-Gm-Message-State: AO0yUKVcLJdEk3I1SUZV+Ef1anKNWODzvle0z+LVpKzqqXdzh0Hy3Xyp lWN7rLAOAaT/HHoJo1LwLr0DN7q4XTWMQn2N X-Google-Smtp-Source: AK7set/mcEFSHmue9cMCcVJMB1Ku9p/0QTBXluCyDE4PFP+ttIIRE+dDut302gTdIA8Yp2aY7RngxiVOCfe3YAYP X-Received: from jthoughton.c.googlers.com ([fda3:e722:ac3:cc00:14:4d90:c0a8:2a4f]) (user=jthoughton job=sendgmr) by 2002:a05:6902:1cd:b0:985:3b30:f27 with SMTP id u13-20020a05690201cd00b009853b300f27mr245191ybh.13.1676680160446; Fri, 17 Feb 2023 16:29:20 -0800 (PST) Date: Sat, 18 Feb 2023 00:28:09 +0000 In-Reply-To: <20230218002819.1486479-1-jthoughton@google.com> Mime-Version: 1.0 References: <20230218002819.1486479-1-jthoughton@google.com> X-Mailer: git-send-email 2.39.2.637.g21b0678d19-goog Message-ID: <20230218002819.1486479-37-jthoughton@google.com> Subject: [PATCH v2 36/46] hugetlb: remove huge_pte_lock and huge_pte_lockptr From: James Houghton To: Mike Kravetz , Muchun Song , Peter Xu , Andrew Morton Cc: David Hildenbrand , David Rientjes , Axel Rasmussen , Mina Almasry , "Zach O'Keefe" , Manish Mishra , Naoya Horiguchi , "Dr . David Alan Gilbert" , "Matthew Wilcox (Oracle)" , Vlastimil Babka , Baolin Wang , Miaohe Lin , Yang Shi , Frank van der Linden , Jiaqi Yan , linux-mm@kvack.org, linux-kernel@vger.kernel.org, James Houghton X-Stat-Signature: pq51umbmmmks8ojtmae4re3p45n4jx19 X-Rspam-User: X-Rspamd-Queue-Id: 38990C000C X-Rspamd-Server: rspam06 X-HE-Tag: 1676680161-75918 X-HE-Meta: U2FsdGVkX19bcilqFFaZJdWA8cUWPHdtbyrcL/LSQEP7/2v2M6kgm4c4ZNpFTNwAQFM+Fq+lT6BioEKA9IDuJUrWzgnB6tnERgun7LEFoSm2nhQd8hF1IpSzipzEMEEyZzXlYhs5VyEZOLU1l7ixubwaZTP39jV64ZJzmpeo0i2vgE48qGTXlNj/VwfoZ/IahGRqi7/UMsDTgbL2bet5PwxMNR4vRAdAwHFGV6rA6G+0G3wHcwaZ070+vIXo0qpAmtoqISGrThdDu8dqvEIrUIdYDd25A1zVBhM4hPAAfwYF8d3933VLA5waQUuivFlKzj4F/60I4hySnkpnQgZ/TeMrn4oqD3kQQEYGAX1qztqdGVw/QP7lv+uz4RsqyGfJyK9EsJFpKESoPhrBpgEbFypCRYHG0FGIH3DRNyvZeN8PH1wRb+srVQ4X3Qn9c01ivYhi7MDc5FsdTvw9gn3smSgqAGhisLwkP4aLEJGzyV/DNJTlebgzAX7ZjbbJlS/Ei24GAqH2l5CSivYyfZBOKgOiHe1gvUtnMoRoHNYUxKXYNq0Iwwto9VUnpollBFpq/Y7qyRpInOpeSjRJohATYVrViAdIMa6Vppm5DQ10s5+fwQ5rKMGj2MHU5TO8p+tSdR46SAxJgiQlKOhg7eb/hs9zL26PyrnXk2GzXa151faoWtHPKqf9sQrXO+yIFkLB3d8nzPq1bae/lSuuh59V4tGRa8Lg3WAW1Dx1HxyzX05WDBbCndn6cYtZoO/NKMuZudQwKNba6AQGSGB7DSwjXtsILJ4VWm5gn/Rc8fbzHTQqJrSPW7T95wdwyRXlfD+YqwUMR2c+zb65s8jvc/CooV3wvVRWVN90thKYXbC4ghjNuQYF9Vw/7/zyYgSrnp9TrCdtu3FrgxD11bbLQQ2dXbDHUPtKR99eeFBl4D2HkJVBiFVv8S3owRkJR4fKdMdQWTUtM5RG+SUE34XqsT2 DEgDkNPz RiUQQIQ2cx5v9iNVLLCmWACgOxDglMm8rMZ8vnq4GFAgKV9cuwHTLIMEuNxS9pf/0PubnbmEbzmGawn6qwD3YuIEU+nX5LkVmVk/NRgCm6U64iZImdi4l8nfRMyBIW0eQkLwjFvJHDkBDHSSbk3OlpuOmxZs5YYWM1mzsv2VZ1YvvU89IIMWV2fRkbpZ8KLIn3cK6i+bVgbvUbt8fkKo1G9VQHe1d7S8t5su78aZ3zTmh/9jpumWf60p7PnPrYlG/F8ZFf56IA4O2pUnMTN6qaQc6CmUflL+GqyUJzuVOZzZOVyS5xX9KPl+YEoWL2ZVFoqNaTm934obwZarjYxdKjQ6X010Y8uMdu1n3uqFn6v0whLvjEuYUzHV1273H0SfN6ubQ++0zfanmQ8MW0H+egeycjjrmZ3EfPZYy+nTBllP8tvKTyWqG4WiCJTMuQjus6ZRx2MnPqYs6aT31o5HdtIA506vnl8+DFdTc85stPFtLsabozZGs+T/G6AV1Jbjb1Mr+PqA3K8vJ/agxHDAvrW0W2S+D+SMthjztqt+2aUjHd0m5G7DGGV/DIKhP1j3TFQC89Ve/K4DWCL/pu3YREVfuWcQJfjeyD/vD/Bmz+htZwik= X-Bogosity: Ham, tests=bogofilter, spamicity=0.000000, version=1.2.4 Sender: owner-linux-mm@kvack.org Precedence: bulk X-Loop: owner-majordomo@kvack.org List-ID: They are replaced with hugetlb_pte_lock{,ptr}. All callers that haven't already been replaced don't get called when using HGM, so we handle them by populating hugetlb_ptes with the standard, hstate-sized huge PTEs. Signed-off-by: James Houghton diff --git a/arch/powerpc/mm/pgtable.c b/arch/powerpc/mm/pgtable.c index 035a0df47af0..c90ac06dc8d9 100644 --- a/arch/powerpc/mm/pgtable.c +++ b/arch/powerpc/mm/pgtable.c @@ -258,11 +258,14 @@ int huge_ptep_set_access_flags(struct vm_area_struct *vma, #ifdef CONFIG_PPC_BOOK3S_64 struct hstate *h = hstate_vma(vma); + struct hugetlb_pte hpte; psize = hstate_get_psize(h); #ifdef CONFIG_DEBUG_VM - assert_spin_locked(huge_pte_lockptr(huge_page_shift(h), - vma->vm_mm, ptep)); + /* HGM is not supported for powerpc yet. */ + hugetlb_pte_init(&hpte, ptep, huge_page_shift(h), + hpage_size_to_level(psize)); + assert_spin_locked(hpte.ptl); #endif #else diff --git a/include/linux/hugetlb.h b/include/linux/hugetlb.h index 6cd4ae08d84d..742e7f2cb170 100644 --- a/include/linux/hugetlb.h +++ b/include/linux/hugetlb.h @@ -1012,14 +1012,6 @@ static inline gfp_t htlb_modify_alloc_mask(struct hstate *h, gfp_t gfp_mask) return modified_mask; } -static inline spinlock_t *huge_pte_lockptr(unsigned int shift, - struct mm_struct *mm, pte_t *pte) -{ - if (shift == PMD_SHIFT) - return pmd_lockptr(mm, (pmd_t *) pte); - return &mm->page_table_lock; -} - #ifndef hugepages_supported /* * Some platform decide whether they support huge pages at boot @@ -1228,12 +1220,6 @@ static inline gfp_t htlb_modify_alloc_mask(struct hstate *h, gfp_t gfp_mask) return 0; } -static inline spinlock_t *huge_pte_lockptr(unsigned int shift, - struct mm_struct *mm, pte_t *pte) -{ - return &mm->page_table_lock; -} - static inline void hugetlb_count_init(struct mm_struct *mm) { } @@ -1308,16 +1294,6 @@ int hugetlb_collapse(struct mm_struct *mm, unsigned long start, } #endif -static inline spinlock_t *huge_pte_lock(struct hstate *h, - struct mm_struct *mm, pte_t *pte) -{ - spinlock_t *ptl; - - ptl = huge_pte_lockptr(huge_page_shift(h), mm, pte); - spin_lock(ptl); - return ptl; -} - static inline spinlock_t *hugetlb_pte_lockptr(struct hugetlb_pte *hpte) { @@ -1353,8 +1329,22 @@ void hugetlb_pte_init(struct mm_struct *mm, struct hugetlb_pte *hpte, pte_t *ptep, unsigned int shift, enum hugetlb_level level) { - __hugetlb_pte_init(hpte, ptep, shift, level, - huge_pte_lockptr(shift, mm, ptep)); + spinlock_t *ptl; + + /* + * For contiguous HugeTLB PTEs that can contain other HugeTLB PTEs + * on the same level, the same PTL for both must be used. + * + * For some architectures that implement hugetlb_walk_step, this + * version of hugetlb_pte_populate() may not be correct to use for + * high-granularity PTEs. Instead, call __hugetlb_pte_populate() + * directly. + */ + if (level == HUGETLB_LEVEL_PMD) + ptl = pmd_lockptr(mm, (pmd_t *) ptep); + else + ptl = &mm->page_table_lock; + __hugetlb_pte_init(hpte, ptep, shift, level, ptl); } #if defined(CONFIG_HUGETLB_PAGE) && defined(CONFIG_CMA) diff --git a/mm/hugetlb.c b/mm/hugetlb.c index 34368072dabe..e0a92e7c1755 100644 --- a/mm/hugetlb.c +++ b/mm/hugetlb.c @@ -5454,9 +5454,8 @@ int copy_hugetlb_page_range(struct mm_struct *dst, struct mm_struct *src, put_page(hpage); /* Install the new hugetlb folio if src pte stable */ - dst_ptl = huge_pte_lock(h, dst, dst_pte); - src_ptl = huge_pte_lockptr(huge_page_shift(h), - src, src_pte); + dst_ptl = hugetlb_pte_lock(&dst_hpte); + src_ptl = hugetlb_pte_lockptr(&src_hpte); spin_lock_nested(src_ptl, SINGLE_DEPTH_NESTING); entry = huge_ptep_get(src_pte); if (!pte_same(src_pte_old, entry)) { @@ -7582,7 +7581,8 @@ pte_t *huge_pmd_share(struct mm_struct *mm, struct vm_area_struct *vma, unsigned long saddr; pte_t *spte = NULL; pte_t *pte; - spinlock_t *ptl; + struct hugetlb_pte hpte; + struct hstate *shstate; i_mmap_lock_read(mapping); vma_interval_tree_foreach(svma, &mapping->i_mmap, idx, idx) { @@ -7603,7 +7603,11 @@ pte_t *huge_pmd_share(struct mm_struct *mm, struct vm_area_struct *vma, if (!spte) goto out; - ptl = huge_pte_lock(hstate_vma(vma), mm, spte); + shstate = hstate_vma(svma); + + hugetlb_pte_init(mm, &hpte, spte, huge_page_shift(shstate), + hpage_size_to_level(huge_page_size(shstate))); + spin_lock(hpte.ptl); if (pud_none(*pud)) { pud_populate(mm, pud, (pmd_t *)((unsigned long)spte & PAGE_MASK)); @@ -7611,7 +7615,7 @@ pte_t *huge_pmd_share(struct mm_struct *mm, struct vm_area_struct *vma, } else { put_page(virt_to_page(spte)); } - spin_unlock(ptl); + spin_unlock(hpte.ptl); out: pte = (pte_t *)pmd_alloc(mm, pud, addr); i_mmap_unlock_read(mapping); @@ -8315,6 +8319,7 @@ static void hugetlb_unshare_pmds(struct vm_area_struct *vma, unsigned long address; spinlock_t *ptl; pte_t *ptep; + struct hugetlb_pte hpte; if (!(vma->vm_flags & VM_MAYSHARE)) return; @@ -8336,7 +8341,10 @@ static void hugetlb_unshare_pmds(struct vm_area_struct *vma, ptep = hugetlb_walk(vma, address, sz); if (!ptep) continue; - ptl = huge_pte_lock(h, mm, ptep); + + hugetlb_pte_init(mm, &hpte, ptep, huge_page_shift(h), + hpage_size_to_level(sz)); + ptl = hugetlb_pte_lock(&hpte); huge_pmd_unshare(mm, vma, address, ptep); spin_unlock(ptl); }