From patchwork Thu Jan 5 10:18:08 2023 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: James Houghton X-Patchwork-Id: 13089645 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from kanga.kvack.org (kanga.kvack.org [205.233.56.17]) by smtp.lore.kernel.org (Postfix) with ESMTP id 60294C54E76 for ; Thu, 5 Jan 2023 10:19:11 +0000 (UTC) Received: by kanga.kvack.org (Postfix) id F0F168E0001; Thu, 5 Jan 2023 05:19:10 -0500 (EST) Received: by kanga.kvack.org (Postfix, from userid 40) id EBFF4940008; Thu, 5 Jan 2023 05:19:10 -0500 (EST) X-Delivered-To: int-list-linux-mm@kvack.org Received: by kanga.kvack.org (Postfix, from userid 63042) id D3A9F8E0005; Thu, 5 Jan 2023 05:19:10 -0500 (EST) X-Delivered-To: linux-mm@kvack.org Received: from relay.hostedemail.com (smtprelay0013.hostedemail.com [216.40.44.13]) by kanga.kvack.org (Postfix) with ESMTP id C467D8E0001 for ; Thu, 5 Jan 2023 05:19:10 -0500 (EST) Received: from smtpin10.hostedemail.com (a10.router.float.18 [10.200.18.1]) by unirelay01.hostedemail.com (Postfix) with ESMTP id 788F91C64F9 for ; Thu, 5 Jan 2023 10:19:10 +0000 (UTC) X-FDA: 80320347660.10.E7039C7 Received: from mail-vk1-f202.google.com (mail-vk1-f202.google.com [209.85.221.202]) by imf19.hostedemail.com (Postfix) with ESMTP id E2F591A0010 for ; Thu, 5 Jan 2023 10:19:08 +0000 (UTC) Authentication-Results: imf19.hostedemail.com; dkim=pass header.d=google.com header.s=20210112 header.b=kKVOVndA; spf=pass (imf19.hostedemail.com: domain of 3HKS2YwoKCGMKUIPVHIUPOHPPHMF.DPNMJOVY-NNLWBDL.PSH@flex--jthoughton.bounces.google.com designates 209.85.221.202 as permitted sender) smtp.mailfrom=3HKS2YwoKCGMKUIPVHIUPOHPPHMF.DPNMJOVY-NNLWBDL.PSH@flex--jthoughton.bounces.google.com; dmarc=pass (policy=reject) header.from=google.com ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=hostedemail.com; s=arc-20220608; t=1672913948; h=from:from:sender:reply-to:subject:subject:date:date: message-id:message-id:to:to:cc:cc:mime-version:mime-version: content-type:content-type:content-transfer-encoding: in-reply-to:in-reply-to:references:references:dkim-signature; bh=YeLoaIulUhCWWXAMbuNF9M+t6oo8aI+KwJNM9mGseGc=; b=hWZvSKxlLF+NorKs/l6jAbv1u6y+zRrc4Hpkk5/EDr8BJlaMzsr8UDH72PtnrjSMPLisOB /3gQRT9V1kFDo8TqIeg9CIvJO/r5+ysa3puofseg3lpnC2zMefV7OO+CVzjB0Ljd1s15GG YtKowVBbDGpbKzrW4/zrLLS7QAIQolc= ARC-Authentication-Results: i=1; imf19.hostedemail.com; dkim=pass header.d=google.com header.s=20210112 header.b=kKVOVndA; spf=pass (imf19.hostedemail.com: domain of 3HKS2YwoKCGMKUIPVHIUPOHPPHMF.DPNMJOVY-NNLWBDL.PSH@flex--jthoughton.bounces.google.com designates 209.85.221.202 as permitted sender) smtp.mailfrom=3HKS2YwoKCGMKUIPVHIUPOHPPHMF.DPNMJOVY-NNLWBDL.PSH@flex--jthoughton.bounces.google.com; dmarc=pass (policy=reject) header.from=google.com ARC-Seal: i=1; s=arc-20220608; d=hostedemail.com; t=1672913948; a=rsa-sha256; cv=none; b=EKcsilmBpIlsSH5wUtWaTX4dTQwQt1WCh1z2J+N3cfM1NpKHEP/BeEJNURXk2R+MMtQttE Nea6ytg52FRf/pOPIchAeITknr5HRLGFtZ8A4igpd1bSAo2KpvGDk5FhzIZd9TEQRpU4Sf kuNonIoFDO6TDU1nFLdMMLAeRfLMCIA= Received: by mail-vk1-f202.google.com with SMTP id b72-20020a1fb24b000000b003d58fdc33fcso5749886vkf.10 for ; Thu, 05 Jan 2023 02:19:08 -0800 (PST) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=google.com; s=20210112; h=cc:to:from:subject:message-id:references:mime-version:in-reply-to :date:from:to:cc:subject:date:message-id:reply-to; bh=YeLoaIulUhCWWXAMbuNF9M+t6oo8aI+KwJNM9mGseGc=; b=kKVOVndAgw/m6ANRlyIyxTS08GniA9U67oeruSczuNHOB6SYI/n6pFU+HOjz+szql2 PWk4++j2ti0R10lgALWNsE1fhYxnWxM7JukY7mIAe7pCJ5g5YXsry19Ma9iPvK6znPtU jMYy+OpdWtqzX+Nek/p3045z5ujRR/oRPI7EUBPhqyf1BWSaZmKdKoPtQI+kGu8Ti537 3J1Z6ngNzsR80E2A85x/f+xTfzkF6gI9uOpsAHaLaSIpG6B2w5XFihWl8tN3nkAnSfct 1xZKvd3M+YUP+kgur9AeYOYhuRYRgatI3fPPdF/dj+Kh+YLKqpI/TnSWPjyTPhxjRiqI rjMg== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20210112; h=cc:to:from:subject:message-id:references:mime-version:in-reply-to :date:x-gm-message-state:from:to:cc:subject:date:message-id:reply-to; bh=YeLoaIulUhCWWXAMbuNF9M+t6oo8aI+KwJNM9mGseGc=; b=NU2ZLCGtIN1PglFdoRDOjTkg+6tlQYECW6gXI6M/Q1QlrCfsUBv9FzPouC4d8FcHec zEahGHQuE/joNMMD1/7P0ZMsl2D+qMXhYSo6M8AVhJikCQk4at8gEHeBO58/dA20jfnV KKi9YcM2QcXRSaY2ILvLEtp+SpWQY62B54ADSr1rFRRt4/UPP7ccqypOJ3ScyIBIRV4W 8eytKjHW9MYTkERfCSNe+ZPZuE569dVLvcH/DlCsWp9WB+Vxlo3kZRxnGKwRjbJfVMMb rFDsDpODZ/BzvwuQKCCngbY0ldxo4rv8qlJNmseENclWGLw48xS8owwYWU2f/Xv44ZQn 1wLg== X-Gm-Message-State: AFqh2koiuqOHhcZ0QS1J0OfGwY+b3al47B+zX+/1quWkqhJKtHrEA3VW IJwUfrpaC8UTppDk6GIQK7ZcT5n7WjgiyZNE X-Google-Smtp-Source: AMrXdXtkUHqLKKc7LXMFTV+aWMn8L7Sl+RR7fShIQHxbdJZ0UldDgF+0b7JMDk/710jpas3HLfAgCwlTWk0Lhcfi X-Received: from jthoughton.c.googlers.com ([fda3:e722:ac3:cc00:14:4d90:c0a8:2a4f]) (user=jthoughton job=sendgmr) by 2002:ab0:2e84:0:b0:419:2056:34b8 with SMTP id f4-20020ab02e84000000b00419205634b8mr5177046uaa.85.1672913948093; Thu, 05 Jan 2023 02:19:08 -0800 (PST) Date: Thu, 5 Jan 2023 10:18:08 +0000 In-Reply-To: <20230105101844.1893104-1-jthoughton@google.com> Mime-Version: 1.0 References: <20230105101844.1893104-1-jthoughton@google.com> X-Mailer: git-send-email 2.39.0.314.g84b9a713c41-goog Message-ID: <20230105101844.1893104-11-jthoughton@google.com> Subject: [PATCH 10/46] hugetlb: make huge_pte_lockptr take an explicit shift argument From: James Houghton To: Mike Kravetz , Muchun Song , Peter Xu Cc: David Hildenbrand , David Rientjes , Axel Rasmussen , Mina Almasry , "Zach O'Keefe" , Manish Mishra , Naoya Horiguchi , "Dr . David Alan Gilbert" , "Matthew Wilcox (Oracle)" , Vlastimil Babka , Baolin Wang , Miaohe Lin , Yang Shi , Andrew Morton , linux-mm@kvack.org, linux-kernel@vger.kernel.org, James Houghton X-Rspamd-Server: rspam05 X-Rspamd-Queue-Id: E2F591A0010 X-Stat-Signature: ykzka14hw5x3jbur53c9fm71ofwktnsu X-Rspam-User: X-HE-Tag: 1672913948-70558 X-HE-Meta: U2FsdGVkX1+0wctXVB0bTeO168WjzpYMDuxaWrvgd0S+ONVyPhSIe8nUUU9OmA7Dt84PvmErPu1iSpcZ/VKEQuRqZT1dww9wxs1PPwTJYBzOoO78wezvfwquWGewUHMieUnPB54M4OLcrVASGtFQhNjfGjBdapIECyZcvnjtK+hjBDImiTAYZvqpnfIkolZfHy1ngZ15pe8Zz7N9FeP9t/6KxLpi73gGfLHnTF0t/V74jcuZrI9FzyHjjLwWBQeledZq42FBDZxL2LNbO1aU4I/pqAhE/BR74nFv6NUEgjGSpCZ+BPXMLge6/h3OqZtS6DelMMUbvUfDM3Tci56408jf354/GDZ5J/W0abvB3mr7dJp7CuQIDar3J0EynyvTTuVU5xfMZ1pUKdawXh/JGFhWZifvZ1KDEbBW3ien9RQ9UFFrJ/Hle5CYgNCTVpireGeazyWHM09rXwiHf1wtJdRqgV5SN6sf5z4SDGXtjPaG03+bPLDoBPwQFfWQCqSrmHjUrqKsWryS18QXjpsbmGocgCoeXRaIGMsiUp5Go8lnFE3G9e4VCywzJ8NDuEUw651KukE3EBAi5mx96vNd0jK4KAusIGcEdWEz8sKq6DvX8q4sQAsWMcDqLAj0u8nVasli+98hwJAM1C43BH4QlRTYbp4lBGmzqsVX/V36VMcu6KocVuWyqDVTd9qaXnvXEmSf9q1zfiX8FSSXvG6rgoq3F5lQNvzT5durX+aVf/p6XSbLiyaq7Jovq8FPfDp0E4jr+9ZFrsZG0Sfdt74vtRYhOzrX1RafQalWD/EajoNcsLgK0iaEqqyANsVnQ+i4yww/9IELu9llQzIfX8xwWObYEDpTFWdIN1zeFMHIutND1oaVDHSndltol7uM0j0JCFpp4xr3o/uCPQ7MFRzyhzf3wh5aRwNxf1FoTsO2M0DDjoaU9SIHESIXSTWOLzMx+Smib0EJxTh3tU8hbiu ayB34rnU 11TVpvv0fodgC+8AtiMNMSumpS+Hep0rG8IHpgW0hMdKlENLSROvzw3GBObSaRfYMHUCPAB8HaDvTEyHmNhfkUlIQEYY1n1/mv60oLRbbOfhITgEuAMyk5GCHl9AeyiZ6lInG8FYcg42Djyh3NuBaPDhA0QtXtnTcv6M7iRYhiidgjlqZaK0edteOS2O5guWtpsclSIgkGlRi5TzMCErNvA0zUONZ41TIuPu3FjY2IONBp7BLKI2/vzxDc8bfwoMoXm4pUMxuKSL6zQEUt51g7hiIkBsc6WVYN07nQcbnJMb8uR/hbiyiVYz3+kwwhhdLpByBcBS4fPWATku6VLnReboDkna2oFjZO36gqMFeg7GaSjSxS/bf9N2CG34OpS5Q37ghhyuiNsXT+O12IvByJR4XwpeCXI44bXJAHMvYXjr4HZvb8333JC0TqupI/8hx+7qV X-Bogosity: Ham, tests=bogofilter, spamicity=0.000000, version=1.2.4 Sender: owner-linux-mm@kvack.org Precedence: bulk X-Loop: owner-majordomo@kvack.org List-ID: This is needed to handle PTL locking with high-granularity mapping. We won't always be using the PMD-level PTL even if we're using the 2M hugepage hstate. It's possible that we're dealing with 4K PTEs, in which case, we need to lock the PTL for the 4K PTE. Reviewed-by: Mina Almasry Acked-by: Mike Kravetz Signed-off-by: James Houghton --- arch/powerpc/mm/pgtable.c | 3 ++- include/linux/hugetlb.h | 9 ++++----- mm/hugetlb.c | 7 ++++--- mm/migrate.c | 3 ++- 4 files changed, 12 insertions(+), 10 deletions(-) diff --git a/arch/powerpc/mm/pgtable.c b/arch/powerpc/mm/pgtable.c index cb2dcdb18f8e..035a0df47af0 100644 --- a/arch/powerpc/mm/pgtable.c +++ b/arch/powerpc/mm/pgtable.c @@ -261,7 +261,8 @@ int huge_ptep_set_access_flags(struct vm_area_struct *vma, psize = hstate_get_psize(h); #ifdef CONFIG_DEBUG_VM - assert_spin_locked(huge_pte_lockptr(h, vma->vm_mm, ptep)); + assert_spin_locked(huge_pte_lockptr(huge_page_shift(h), + vma->vm_mm, ptep)); #endif #else diff --git a/include/linux/hugetlb.h b/include/linux/hugetlb.h index 16fc3e381801..3f098363cd6e 100644 --- a/include/linux/hugetlb.h +++ b/include/linux/hugetlb.h @@ -956,12 +956,11 @@ static inline gfp_t htlb_modify_alloc_mask(struct hstate *h, gfp_t gfp_mask) return modified_mask; } -static inline spinlock_t *huge_pte_lockptr(struct hstate *h, +static inline spinlock_t *huge_pte_lockptr(unsigned int shift, struct mm_struct *mm, pte_t *pte) { - if (huge_page_size(h) == PMD_SIZE) + if (shift == PMD_SHIFT) return pmd_lockptr(mm, (pmd_t *) pte); - VM_BUG_ON(huge_page_size(h) == PAGE_SIZE); return &mm->page_table_lock; } @@ -1171,7 +1170,7 @@ static inline gfp_t htlb_modify_alloc_mask(struct hstate *h, gfp_t gfp_mask) return 0; } -static inline spinlock_t *huge_pte_lockptr(struct hstate *h, +static inline spinlock_t *huge_pte_lockptr(unsigned int shift, struct mm_struct *mm, pte_t *pte) { return &mm->page_table_lock; @@ -1228,7 +1227,7 @@ static inline spinlock_t *huge_pte_lock(struct hstate *h, { spinlock_t *ptl; - ptl = huge_pte_lockptr(h, mm, pte); + ptl = huge_pte_lockptr(huge_page_shift(h), mm, pte); spin_lock(ptl); return ptl; } diff --git a/mm/hugetlb.c b/mm/hugetlb.c index 5bd53ae8ca4b..4db38dc79d0e 100644 --- a/mm/hugetlb.c +++ b/mm/hugetlb.c @@ -4987,7 +4987,7 @@ int copy_hugetlb_page_range(struct mm_struct *dst, struct mm_struct *src, } dst_ptl = huge_pte_lock(h, dst, dst_pte); - src_ptl = huge_pte_lockptr(h, src, src_pte); + src_ptl = huge_pte_lockptr(huge_page_shift(h), src, src_pte); spin_lock_nested(src_ptl, SINGLE_DEPTH_NESTING); entry = huge_ptep_get(src_pte); again: @@ -5068,7 +5068,8 @@ int copy_hugetlb_page_range(struct mm_struct *dst, struct mm_struct *src, /* Install the new huge page if src pte stable */ dst_ptl = huge_pte_lock(h, dst, dst_pte); - src_ptl = huge_pte_lockptr(h, src, src_pte); + src_ptl = huge_pte_lockptr(huge_page_shift(h), + src, src_pte); spin_lock_nested(src_ptl, SINGLE_DEPTH_NESTING); entry = huge_ptep_get(src_pte); if (!pte_same(src_pte_old, entry)) { @@ -5122,7 +5123,7 @@ static void move_huge_pte(struct vm_area_struct *vma, unsigned long old_addr, pte_t pte; dst_ptl = huge_pte_lock(h, mm, dst_pte); - src_ptl = huge_pte_lockptr(h, mm, src_pte); + src_ptl = huge_pte_lockptr(huge_page_shift(h), mm, src_pte); /* * We don't have to worry about the ordering of src and dst ptlocks diff --git a/mm/migrate.c b/mm/migrate.c index b5032c3e940a..832f639fc49a 100644 --- a/mm/migrate.c +++ b/mm/migrate.c @@ -360,7 +360,8 @@ void __migration_entry_wait_huge(struct vm_area_struct *vma, void migration_entry_wait_huge(struct vm_area_struct *vma, pte_t *pte) { - spinlock_t *ptl = huge_pte_lockptr(hstate_vma(vma), vma->vm_mm, pte); + spinlock_t *ptl = huge_pte_lockptr(huge_page_shift(hstate_vma(vma)), + vma->vm_mm, pte); __migration_entry_wait_huge(vma, pte, ptl); }