From patchwork Sat Feb 18 00:27:43 2023 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: James Houghton X-Patchwork-Id: 13145376 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from kanga.kvack.org (kanga.kvack.org [205.233.56.17]) by smtp.lore.kernel.org (Postfix) with ESMTP id CF561C6379F for ; Sat, 18 Feb 2023 00:29:05 +0000 (UTC) Received: by kanga.kvack.org (Postfix) id 8C7D5280006; Fri, 17 Feb 2023 19:28:56 -0500 (EST) Received: by kanga.kvack.org (Postfix, from userid 40) id 828DD280002; Fri, 17 Feb 2023 19:28:56 -0500 (EST) X-Delivered-To: int-list-linux-mm@kvack.org Received: by kanga.kvack.org (Postfix, from userid 63042) id 6F255280006; Fri, 17 Feb 2023 19:28:56 -0500 (EST) X-Delivered-To: linux-mm@kvack.org Received: from relay.hostedemail.com (smtprelay0015.hostedemail.com [216.40.44.15]) by kanga.kvack.org (Postfix) with ESMTP id 60C65280002 for ; Fri, 17 Feb 2023 19:28:56 -0500 (EST) Received: from smtpin16.hostedemail.com (a10.router.float.18 [10.200.18.1]) by unirelay03.hostedemail.com (Postfix) with ESMTP id 3C363A0694 for ; Sat, 18 Feb 2023 00:28:56 +0000 (UTC) X-FDA: 80478527472.16.F5E4554 Received: from mail-yb1-f202.google.com (mail-yb1-f202.google.com [209.85.219.202]) by imf28.hostedemail.com (Postfix) with ESMTP id 80AE8C000E for ; Sat, 18 Feb 2023 00:28:54 +0000 (UTC) Authentication-Results: imf28.hostedemail.com; dkim=pass header.d=google.com header.s=20210112 header.b=G9OKNC7M; spf=pass (imf28.hostedemail.com: domain of 3xRvwYwoKCOINXLSYKLXSRKSSKPI.GSQPMRYb-QQOZEGO.SVK@flex--jthoughton.bounces.google.com designates 209.85.219.202 as permitted sender) smtp.mailfrom=3xRvwYwoKCOINXLSYKLXSRKSSKPI.GSQPMRYb-QQOZEGO.SVK@flex--jthoughton.bounces.google.com; dmarc=pass (policy=reject) header.from=google.com ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=hostedemail.com; s=arc-20220608; t=1676680134; h=from:from:sender:reply-to:subject:subject:date:date: message-id:message-id:to:to:cc:cc:mime-version:mime-version: content-type:content-type:content-transfer-encoding: in-reply-to:in-reply-to:references:references:dkim-signature; bh=5Xd63Fapv1zuFib4b0fw8zL6DTisPv2GzpUTcimolMg=; b=iCC1hPTGQrh20HwwfaltR4Zlij1ZD9d4UZ4t82sFLn4Dnz9Q4gdP2U9tntZFrxlVscgHIH n8wrd1CGpr9uvtpd5J1lcjD6PtXiYKeAsoRWiMVsek1UzSy5em1KPgwV+AMrA1iCB0tzNc kyQpRVZcIoItP+5oMGwsdqAtFEdnSmA= ARC-Authentication-Results: i=1; imf28.hostedemail.com; dkim=pass header.d=google.com header.s=20210112 header.b=G9OKNC7M; spf=pass (imf28.hostedemail.com: domain of 3xRvwYwoKCOINXLSYKLXSRKSSKPI.GSQPMRYb-QQOZEGO.SVK@flex--jthoughton.bounces.google.com designates 209.85.219.202 as permitted sender) smtp.mailfrom=3xRvwYwoKCOINXLSYKLXSRKSSKPI.GSQPMRYb-QQOZEGO.SVK@flex--jthoughton.bounces.google.com; dmarc=pass (policy=reject) header.from=google.com ARC-Seal: i=1; s=arc-20220608; d=hostedemail.com; t=1676680134; a=rsa-sha256; cv=none; b=sGKfwTVl7zmNfC0J+qX8g4DEPBwKaAsBZHMLe16MO4kojbYAG6Ka9506/qfs9bNtanAWTg WrETPKxuCJC2TYbTvy8GRRBsXw1aUZTx8B43hj0HTe0BL9CchrkPRfcNLg/ijkPwOKBDd/ B2TQJbASNG03J06HU+C6jjEcjLXvIW8= Received: by mail-yb1-f202.google.com with SMTP id y187-20020a2532c4000000b008f257b16d71so2333709yby.15 for ; Fri, 17 Feb 2023 16:28:54 -0800 (PST) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=google.com; s=20210112; h=cc:to:from:subject:message-id:references:mime-version:in-reply-to :date:from:to:cc:subject:date:message-id:reply-to; bh=5Xd63Fapv1zuFib4b0fw8zL6DTisPv2GzpUTcimolMg=; b=G9OKNC7MB5QZ1WtbCVd/b4GGEEQxFMkkCyh9YPN/SaI+Iv0UY6xpXV4C/bjLuR5GX/ xT5jqUIS0kJvgcXBGXG6eGMOl41EQ+9+EPXqDEN8/E+QrPDYAERgHP5tENbu09gPxEEE J2plHmBfhFP7cLD9TJAsfNCeQfwyLgPfEiNyKU7bNFt6UToPyOfI/Tv35qpqCPqZOx+3 BkeCahUvSvDUjoiOquOpSnIGEQ48uVtj2zGVrhp8k95YXggqw2b91kgnZMkucKn+Y1SW QrvcwU7NPBdC0gLOUvUCZFlWQklPf0wCQDXXAGua2AKBXeE+nHdPEk2ToJ76Lq1odat1 0YNw== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20210112; h=cc:to:from:subject:message-id:references:mime-version:in-reply-to :date:x-gm-message-state:from:to:cc:subject:date:message-id:reply-to; bh=5Xd63Fapv1zuFib4b0fw8zL6DTisPv2GzpUTcimolMg=; b=0tTxjof6x++0KEuBY6TPJAP8mzqt1dI7ptBJaExwPVC+28EdiMwazBvb9g+vD10jUz NnR8OMFps6B39G8hrY/zxG5zvuANC4R59mioKVcAjSU+8nqMvFZOT9NMPyIXJEY/j/LX TilZcBclR4lu5EfMY+S3+E3qBQ1mYIIv8gNPsknM9fHKaiQuEVLdx/mKD8pekygTUN+9 DrXb9IBlkzGQkw424mEHjSRSLRGv4ppAVvPTnXdNgMMgUYfZq2yIuYL+aFWQ0+ZJHND9 bVy4ESFFfxuUIyfuVD0WLx3bQeb0IE7g3A8RUXlauj+OMuaYxnOotp+/KZ/iJ0gRG2p+ ItLA== X-Gm-Message-State: AO0yUKVqgSwBIFTOvcJcfPSeMELPYryYEVmRo9HKFvedwVZ1ghjIN28J MbHy1IXLZbQzNpRTifBXG1UT0nuVzva+I/Vo X-Google-Smtp-Source: AK7set8eABNjJa8kuKJfW7WVM4L/kIrgHT4nRE22RZ7aTpjJYSh+BZra+YA1xFFUMi8kyRc4Y+q2EPk2y7TmuTgs X-Received: from jthoughton.c.googlers.com ([fda3:e722:ac3:cc00:14:4d90:c0a8:2a4f]) (user=jthoughton job=sendgmr) by 2002:a81:b60f:0:b0:52a:92e9:27c1 with SMTP id u15-20020a81b60f000000b0052a92e927c1mr279953ywh.10.1676680133715; Fri, 17 Feb 2023 16:28:53 -0800 (PST) Date: Sat, 18 Feb 2023 00:27:43 +0000 In-Reply-To: <20230218002819.1486479-1-jthoughton@google.com> Mime-Version: 1.0 References: <20230218002819.1486479-1-jthoughton@google.com> X-Mailer: git-send-email 2.39.2.637.g21b0678d19-goog Message-ID: <20230218002819.1486479-11-jthoughton@google.com> Subject: [PATCH v2 10/46] hugetlb: make huge_pte_lockptr take an explicit shift argument From: James Houghton To: Mike Kravetz , Muchun Song , Peter Xu , Andrew Morton Cc: David Hildenbrand , David Rientjes , Axel Rasmussen , Mina Almasry , "Zach O'Keefe" , Manish Mishra , Naoya Horiguchi , "Dr . David Alan Gilbert" , "Matthew Wilcox (Oracle)" , Vlastimil Babka , Baolin Wang , Miaohe Lin , Yang Shi , Frank van der Linden , Jiaqi Yan , linux-mm@kvack.org, linux-kernel@vger.kernel.org, James Houghton X-Rspam-User: X-Rspamd-Server: rspam04 X-Rspamd-Queue-Id: 80AE8C000E X-Stat-Signature: 89xzhwi3nshymic1p7hithjmd1oscem6 X-HE-Tag: 1676680134-832340 X-HE-Meta: U2FsdGVkX1/YdYpHc3VWKWczsAvEIOlpFsnwXuvnoJfEgZG3l34mzTMZvZkHWsySfrVUbYQiMGGY92unDjwbOLMueAAQu+olTQ9isBcPKBA5dUQSSXNYyAsRMXMDv+bTzBkfzFq6z0VUsEipTRg52Lgb+5uZ9WFjlt4K1xUR60nceex/3Sa45YjWyuPb9bZn8eJoKBAw2DMlXvY8qJCZUuY6OH9Y+3+S+I5rvVJQknrj35/DYTky6fI5O2lAfoh7xPFBCcFtzmLYZwwrf7YmI5MHnD6nEranKzHe0y+jkWNHskqq++xz9sjuhPPlKNOS2Dm7FlNncV3edWWs64TOXYkLrKoKN3joERFAdbhNmU45KiyXOrtRizeN41FpYP4+CydCgqM4+KY+jwwkEg4tp+G3TJ48yyLToaCaY+P3gLF3Iopjn0upBhp/W29ZoQhMuZOXCSP3ApfudC7IrT/el1zMtv8CVK4X1ffVmaqndojmR7SH59KY89wGUZSA3Hi01lTAKkT8oQmd2gOrsQIFcKoxHlVDiN4imAnGz16RxwW/MmKISreDnjqHd+vxy1A/L9WjYZad/swIW2j0QHGGmgUNSxMWfMPtbYIveYAJRLO883mPQtbv1rxC3vuhySZ7pgChTdzR43RSUMmuJ/Zc12SyCdKaO3HyGQSYS+YbXLOzAt+CIWVCLOzof5vGmISHnMhLl2isKkzn2IaVY6Y5VoZIQcpxaXKl5em0bSgrTB5F6k1RcsoijZm4wo+kWWH8ANg+Rum3a8Vlf5WDJSBKW/34rNnYSFpshbq34c8DNzrcCGK6l73+oHo0BmAOC9/V0wFW7CAAjCL2XCvTu99IXPiIS2gzLNYnH9/fiz3Fqqp0cBGQS4gd8w/MEN9rzlXq7ggdcm5tNknZ73xBbhEysTqSDW2uBV5LtBEQ3eoDnvGOpX+cE0p6pa721rPC3XHHaWP+orU2dqaCexFOZFB CNUm5G7k TVzBl4APfOl+Vb+v/9T9LGkWimum152V+/ORWo6zF6I77KQWIjgipCXz65KCozhn3wr+6PkFanno6VkB+nXByJfzpXsLbjdGPn1WB/Fu0OnNuhKiWEamKCVM5l/IYNKsQZtIbNP35IwWJUfIvle27BBlOn1ssyHJ8nPW78TVXi2oeP36/O0v0URCtPQGf7TtZy9Nwrsu77NSGg5NkMBVnUT7pUoYvewjzyQJVzh29i9yRPVtmv/f6BFDyX3enNRzuR6otbpf3HWTArd2mIlH/zU14akNSfGE9ENCvZfcEOVubnw84hFRWpAvTJscwEhkKE54OOBmdCk5Wf2NFtjmlodyxhMFOnKd2jNHP4xVuMMLu3mODxJOnwnZ05e4NbW8s8CgaTkGg8xfOxsq0ZApaZtRTB/OpbLRbz+MVsT9CVIzKmB0EWDuWpVfEMQT78gYHApdQ6Gj8/qaPYT68xi2DvoPCR1BSNPVido0uRsaUzlIOLf/lD2ZZMSSmsQ4Uq3caeEW0YFrb45NmLw0trmZOiukWlwDbg+leHfNOd9/qqc50E7GwxyyDndDD0LiJPfqCSv1ADVnZO/buU1O5pmT1DWfzuE/UAMt6aZr5lZLMwLH28h1Zhij6JXPIhhVJxsn3xfRc46NbWwvAuhs= X-Bogosity: Ham, tests=bogofilter, spamicity=0.000000, version=1.2.4 Sender: owner-linux-mm@kvack.org Precedence: bulk X-Loop: owner-majordomo@kvack.org List-ID: This is needed to handle PTL locking with high-granularity mapping. We won't always be using the PMD-level PTL even if we're using the 2M hugepage hstate. It's possible that we're dealing with 4K PTEs, in which case, we need to lock the PTL for the 4K PTE. Reviewed-by: Mina Almasry Acked-by: Mike Kravetz Signed-off-by: James Houghton diff --git a/arch/powerpc/mm/pgtable.c b/arch/powerpc/mm/pgtable.c index cb2dcdb18f8e..035a0df47af0 100644 --- a/arch/powerpc/mm/pgtable.c +++ b/arch/powerpc/mm/pgtable.c @@ -261,7 +261,8 @@ int huge_ptep_set_access_flags(struct vm_area_struct *vma, psize = hstate_get_psize(h); #ifdef CONFIG_DEBUG_VM - assert_spin_locked(huge_pte_lockptr(h, vma->vm_mm, ptep)); + assert_spin_locked(huge_pte_lockptr(huge_page_shift(h), + vma->vm_mm, ptep)); #endif #else diff --git a/include/linux/hugetlb.h b/include/linux/hugetlb.h index efd2635a87f5..a1ceb9417f01 100644 --- a/include/linux/hugetlb.h +++ b/include/linux/hugetlb.h @@ -958,12 +958,11 @@ static inline gfp_t htlb_modify_alloc_mask(struct hstate *h, gfp_t gfp_mask) return modified_mask; } -static inline spinlock_t *huge_pte_lockptr(struct hstate *h, +static inline spinlock_t *huge_pte_lockptr(unsigned int shift, struct mm_struct *mm, pte_t *pte) { - if (huge_page_size(h) == PMD_SIZE) + if (shift == PMD_SHIFT) return pmd_lockptr(mm, (pmd_t *) pte); - VM_BUG_ON(huge_page_size(h) == PAGE_SIZE); return &mm->page_table_lock; } @@ -1173,7 +1172,7 @@ static inline gfp_t htlb_modify_alloc_mask(struct hstate *h, gfp_t gfp_mask) return 0; } -static inline spinlock_t *huge_pte_lockptr(struct hstate *h, +static inline spinlock_t *huge_pte_lockptr(unsigned int shift, struct mm_struct *mm, pte_t *pte) { return &mm->page_table_lock; @@ -1230,7 +1229,7 @@ static inline spinlock_t *huge_pte_lock(struct hstate *h, { spinlock_t *ptl; - ptl = huge_pte_lockptr(h, mm, pte); + ptl = huge_pte_lockptr(huge_page_shift(h), mm, pte); spin_lock(ptl); return ptl; } diff --git a/mm/hugetlb.c b/mm/hugetlb.c index 0576dcc98044..5ca9eae0ac42 100644 --- a/mm/hugetlb.c +++ b/mm/hugetlb.c @@ -5017,7 +5017,7 @@ int copy_hugetlb_page_range(struct mm_struct *dst, struct mm_struct *src, } dst_ptl = huge_pte_lock(h, dst, dst_pte); - src_ptl = huge_pte_lockptr(h, src, src_pte); + src_ptl = huge_pte_lockptr(huge_page_shift(h), src, src_pte); spin_lock_nested(src_ptl, SINGLE_DEPTH_NESTING); entry = huge_ptep_get(src_pte); again: @@ -5098,7 +5098,8 @@ int copy_hugetlb_page_range(struct mm_struct *dst, struct mm_struct *src, /* Install the new hugetlb folio if src pte stable */ dst_ptl = huge_pte_lock(h, dst, dst_pte); - src_ptl = huge_pte_lockptr(h, src, src_pte); + src_ptl = huge_pte_lockptr(huge_page_shift(h), + src, src_pte); spin_lock_nested(src_ptl, SINGLE_DEPTH_NESTING); entry = huge_ptep_get(src_pte); if (!pte_same(src_pte_old, entry)) { @@ -5152,7 +5153,7 @@ static void move_huge_pte(struct vm_area_struct *vma, unsigned long old_addr, pte_t pte; dst_ptl = huge_pte_lock(h, mm, dst_pte); - src_ptl = huge_pte_lockptr(h, mm, src_pte); + src_ptl = huge_pte_lockptr(huge_page_shift(h), mm, src_pte); /* * We don't have to worry about the ordering of src and dst ptlocks diff --git a/mm/migrate.c b/mm/migrate.c index b0f87f19b536..9b4a7e75f6e6 100644 --- a/mm/migrate.c +++ b/mm/migrate.c @@ -363,7 +363,8 @@ void __migration_entry_wait_huge(struct vm_area_struct *vma, void migration_entry_wait_huge(struct vm_area_struct *vma, pte_t *pte) { - spinlock_t *ptl = huge_pte_lockptr(hstate_vma(vma), vma->vm_mm, pte); + spinlock_t *ptl = huge_pte_lockptr(huge_page_shift(hstate_vma(vma)), + vma->vm_mm, pte); __migration_entry_wait_huge(vma, pte, ptl); }