From patchwork Thu Jan 5 10:18:05 2023 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: James Houghton X-Patchwork-Id: 13089685 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from kanga.kvack.org (kanga.kvack.org [205.233.56.17]) by smtp.lore.kernel.org (Postfix) with ESMTP id 39203C3DA7A for ; Thu, 5 Jan 2023 10:24:11 +0000 (UTC) Received: by kanga.kvack.org (Postfix) id C25928E0005; Thu, 5 Jan 2023 05:24:10 -0500 (EST) Received: by kanga.kvack.org (Postfix, from userid 40) id BAEEE8E0002; Thu, 5 Jan 2023 05:24:10 -0500 (EST) X-Delivered-To: int-list-linux-mm@kvack.org Received: by kanga.kvack.org (Postfix, from userid 63042) id A29358E0005; Thu, 5 Jan 2023 05:24:10 -0500 (EST) X-Delivered-To: linux-mm@kvack.org Received: from relay.hostedemail.com (smtprelay0014.hostedemail.com [216.40.44.14]) by kanga.kvack.org (Postfix) with ESMTP id 8E9358E0002 for ; Thu, 5 Jan 2023 05:24:10 -0500 (EST) Received: from smtpin25.hostedemail.com (a10.router.float.18 [10.200.18.1]) by unirelay02.hostedemail.com (Postfix) with ESMTP id 6A6D0120D39 for ; Thu, 5 Jan 2023 10:24:10 +0000 (UTC) X-FDA: 80320360260.25.483A7EB Received: from mail-yw1-f202.google.com (mail-yw1-f202.google.com [209.85.128.202]) by imf01.hostedemail.com (Postfix) with ESMTP id C342540006 for ; Thu, 5 Jan 2023 10:24:08 +0000 (UTC) Authentication-Results: imf01.hostedemail.com; dkim=pass header.d=google.com header.s=20210112 header.b=Wlu7XQk1; spf=pass (imf01.hostedemail.com: domain of 3F6S2YwoKCF4FPDKQCDPKJCKKCHA.8KIHEJQT-IIGR68G.KNC@flex--jthoughton.bounces.google.com designates 209.85.128.202 as permitted sender) smtp.mailfrom=3F6S2YwoKCF4FPDKQCDPKJCKKCHA.8KIHEJQT-IIGR68G.KNC@flex--jthoughton.bounces.google.com; dmarc=pass (policy=reject) header.from=google.com ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=hostedemail.com; s=arc-20220608; t=1672914248; h=from:from:sender:reply-to:subject:subject:date:date: message-id:message-id:to:to:cc:cc:mime-version:mime-version: content-type:content-type:content-transfer-encoding: in-reply-to:in-reply-to:references:references:dkim-signature; bh=/8tZn9tkRZbaiVi1nuTHJ8rnAzFnEyMSAOvoIDK0LiA=; b=kmj5Lekuz/dohHK5KPeAhFPozQ5Ac1iaS52j2STGpb02Wfr+hl1cfiUb2mQ3rrbgMEt6mi svGeZugP0tLuqrWacSee2OVg2FagQW6HPHiVlHE/i4ME8uzCjF4iBp9VHBoK7SO5v2CmPU on9t8IKrD86jfR4DBssZbuO9O0MQFEM= ARC-Authentication-Results: i=1; imf01.hostedemail.com; dkim=pass header.d=google.com header.s=20210112 header.b=Wlu7XQk1; spf=pass (imf01.hostedemail.com: domain of 3F6S2YwoKCF4FPDKQCDPKJCKKCHA.8KIHEJQT-IIGR68G.KNC@flex--jthoughton.bounces.google.com designates 209.85.128.202 as permitted sender) smtp.mailfrom=3F6S2YwoKCF4FPDKQCDPKJCKKCHA.8KIHEJQT-IIGR68G.KNC@flex--jthoughton.bounces.google.com; dmarc=pass (policy=reject) header.from=google.com ARC-Seal: i=1; s=arc-20220608; d=hostedemail.com; t=1672914248; a=rsa-sha256; cv=none; b=GkMx9UuW/eyIPm4VJkHVOmXJtsTB+1CTcsZD93Br5ScsqLjOmybawrkiyVkvLgl7+EALia KrbXRZqry2gHNo6g5p9mqxWDYq4Z4sjfkCt7YDH4LDM2BerMTtONIa6NKJfVBXPPQPq3Y8 6NbgbdPpQfyiulpV2qa39rhOU+1WEAw= Received: by mail-yw1-f202.google.com with SMTP id 00721157ae682-46839d9ca5dso358704307b3.16 for ; Thu, 05 Jan 2023 02:24:08 -0800 (PST) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=google.com; s=20210112; h=cc:to:from:subject:message-id:references:mime-version:in-reply-to :date:from:to:cc:subject:date:message-id:reply-to; bh=/8tZn9tkRZbaiVi1nuTHJ8rnAzFnEyMSAOvoIDK0LiA=; b=Wlu7XQk16tqMmg+gNIiIDB1JA/O/qtq05ltp8iTMsYdjH00SBVdUFxa+cBdSZqAQcb d1Wubwm9cgGb/JWbc53zSO6qZR6u0tqT/uhjkvshXe4Nbc29/Mx0FTWQepbVo5VkK/f4 iZzimrA+jYG5bRVUIknjFLaPXJryFDe1l0StPyym3HfhV9kGhFLJmjpADMJIHuvbrMU2 TBM7F+ry7k4c0I//IisI9senk9jQmTO6g/9v4uatar6m1pi5ieYC8Z12sX5T0E2ZspUi kbNXjBMVADs1kunBuGHYbySRRHfw6oNiaeB7XCT6DI2AIDizBxl6F3hmyK+58dSXDshb CYtg== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20210112; h=cc:to:from:subject:message-id:references:mime-version:in-reply-to :date:x-gm-message-state:from:to:cc:subject:date:message-id:reply-to; bh=/8tZn9tkRZbaiVi1nuTHJ8rnAzFnEyMSAOvoIDK0LiA=; b=mXi7M/JixJyGfbHuol1SFCuRJJJT0nCR5ArBBMhNu/eR1QJN2jCfFEau4LkxprsNfM I2T4vE/GZtZy+EFAwT/eqLZm9z8GrMGq3e1pBTfPDdCsvYLKYE+nE7bd4wTZXK39UmKc Hq/FIYbduMYMBaIiZh5SZwZcvR7A+dRzgQYcRVHfU1OEAS/edip7Nr83+zIGeA9qJ053 jgccMsvEG1T8l/WkOOCBnmdFnXHtEH2AbLkLfreRX+FjjoQthqJxrmD3vm1IMzw4sVr0 yMgceVjV3IxtTMTPN7jRxXUKILeAb+uYyjakDGIP2fhnPPZKs1vRF82pWZ0Q91JZ9eBu kb0Q== X-Gm-Message-State: AFqh2kqGR4/B7G4GBbWv7f8eHjeAO0quUXughYtpd9TvwM4O1myE+L7g Na+xxY6W79B1euHirF0II0ZIBLF2dU9UXEK8 X-Google-Smtp-Source: AMrXdXvzTfWe8MomGisuHc0GNsvUXDDVMXnXUti7atm5wfvLHi6RqA6sGIYMNCo6vky1zzYRo3Z5sRweZqrg8aL0 X-Received: from jthoughton.c.googlers.com ([fda3:e722:ac3:cc00:14:4d90:c0a8:2a4f]) (user=jthoughton job=sendgmr) by 2002:a25:afce:0:b0:6ca:ecb9:9fc0 with SMTP id d14-20020a25afce000000b006caecb99fc0mr6101337ybj.199.1672913943280; Thu, 05 Jan 2023 02:19:03 -0800 (PST) Date: Thu, 5 Jan 2023 10:18:05 +0000 In-Reply-To: <20230105101844.1893104-1-jthoughton@google.com> Mime-Version: 1.0 References: <20230105101844.1893104-1-jthoughton@google.com> X-Mailer: git-send-email 2.39.0.314.g84b9a713c41-goog Message-ID: <20230105101844.1893104-8-jthoughton@google.com> Subject: [PATCH 07/46] hugetlb: rename __vma_shareable_flags_pmd to __vma_has_hugetlb_vma_lock From: James Houghton To: Mike Kravetz , Muchun Song , Peter Xu Cc: David Hildenbrand , David Rientjes , Axel Rasmussen , Mina Almasry , "Zach O'Keefe" , Manish Mishra , Naoya Horiguchi , "Dr . David Alan Gilbert" , "Matthew Wilcox (Oracle)" , Vlastimil Babka , Baolin Wang , Miaohe Lin , Yang Shi , Andrew Morton , linux-mm@kvack.org, linux-kernel@vger.kernel.org, James Houghton X-Stat-Signature: 77ufbxnicr5t7sr9mf99rnf1d14j4wgf X-Rspam-User: X-Rspamd-Queue-Id: C342540006 X-Rspamd-Server: rspam06 X-HE-Tag: 1672914248-879650 X-HE-Meta: U2FsdGVkX18ojLVmNVhc7WuDPYgu+IMYxw0a++AeKyQu9yPDcN4xAD9YHmSb3CunHU/O7RNbgFXeR+B7ziRUfymMnLsmZJt/VQ8nUiMSvkHapzLFHpE2XzFUY/Gkc++ATQ5SjuZoGzBS9KYeekgOKnC0Lk0N2pBl39+JJYwSMP3qxAg1oWr4Xt5Jg2eA3JYsKo88DqmcgAlk3ZwgG6TY7u7fAzif8K3KCvIkmo+xtFx7iht+8GSdygaIJzPOTbBZEimu7RrVapY8HRxITbvQ3N5m6DHkNwqDdg9GK77TuK9N1RDo8BMy2V08M8hLwFyD6FB5aKqaidS4gL4pSDdh8+ZdFIcdEwAUeBMc4n8zC9HyMZRJVVSOT15Jk9sgIPNB4E4csucgFlN7bGgypaWJNXFNOY5GlXKdYVOjFjw1Cv6g9bRiHf1TyMOrqaNnFPynMbtSaFLv3/yytmr3Pdljk7idiStpNxCF+0qcnn0P5fDJzpfcPk+iTAN8UgY0ticBSxO8KHU8bjEzUpdLL+qn7UBZ/icZQMeQRm6e9X4XPseNeE8v3B4e2BeJnSP30Xs8EOF+988VZSGtPqGATuNdRCbBHelO5EmIFcj/pugg1nGdoK2jYN8E9uMV1Dv8+SJYdx5hs087d2rz4kIrCNVJ9x1IVIqLI8lvPC6qOtvGLrBUrG0UWWIC2yYgH9Q1GQ6GVkJNzcPCXaUgzCCk2oqbbw4GSnRbnaLPnpqPbQImmB4eoluLP8qBIe9DJwVH6/WPBqCI82zfeIf472Ag7U1ALJoOj22seaYxj5VuV4TLgNmMjV5QA4yZrS0beW/RWoHnPEtgM0tkPAIWpYTS1TPm1nyqt//Z7k1mP3Ue/2Je+/H8/z53wscbgvNpzV+igFFHfNSZiTh8zGsqglMQ8Bx8b80tz96tef5+ksquT56nkdALFDlaXdzE8iGEHqLD30ncL1wDuE+Lawr2RTrk0EO fof6iVim jzObmpzIrsgCesStk9ynfzt9pWIjQNmpG2BALnKuj+AFLXxOmdbsNIcm6gaQ441M2WOBRLTfnrJZRRspi+G/S7bhosoYKL37iCxT+/YFEi3UxZ/F31haVx8osWd/GlZ0KK8cLdgd8zdYyFb9AWCBa37gyvL2iRDUm5+S1brpi7G//Fb3DJ+Tz7nCvTpLHsTe+BvsAF1Cqr33GaJPnm3ckmBG1VtrS7g3+X7NED36kEN7eg0X8XSh0a9nc29/KnWBBgb5MZ0Arl6CA/P+kERbfcv3+lYo3lTOddh4b+ts1QN0dKPKPUtq5jCrGQYEcTdhzUeAQg+a9nECKrRWahDezd95nndz7C3UUCM2uByZGPptixE0uL28/lbwUwBuBO8Ag13a17wS/BdRUyjZJZMuqiHXdPkhRzWWycTL+VVXdq/VqoMK1HXy41cVwQ6JyyYD5PWr2 X-Bogosity: Ham, tests=bogofilter, spamicity=0.000000, version=1.2.4 Sender: owner-linux-mm@kvack.org Precedence: bulk X-Loop: owner-majordomo@kvack.org List-ID: Previously, if the hugetlb VMA lock was present, that meant that the VMA was PMD-shareable. Now it is possible that the VMA lock is allocated but the VMA is not PMD-shareable: if the VMA is a high-granularity VMA. It is possible for a high-granularity VMA not to have a VMA lock; in this case, MADV_COLLAPSE will not be able to collapse the mappings. Signed-off-by: James Houghton --- include/linux/hugetlb.h | 15 ++++++++++----- mm/hugetlb.c | 16 ++++++++-------- 2 files changed, 18 insertions(+), 13 deletions(-) diff --git a/include/linux/hugetlb.h b/include/linux/hugetlb.h index b6b10101bea7..aa49fd8cb47c 100644 --- a/include/linux/hugetlb.h +++ b/include/linux/hugetlb.h @@ -1235,7 +1235,8 @@ bool want_pmd_share(struct vm_area_struct *vma, unsigned long addr); #define flush_hugetlb_tlb_range(vma, addr, end) flush_tlb_range(vma, addr, end) #endif -static inline bool __vma_shareable_lock(struct vm_area_struct *vma) +static inline bool +__vma_has_hugetlb_vma_lock(struct vm_area_struct *vma) { return (vma->vm_flags & VM_MAYSHARE) && vma->vm_private_data; } @@ -1252,13 +1253,17 @@ hugetlb_walk(struct vm_area_struct *vma, unsigned long addr, unsigned long sz) struct hugetlb_vma_lock *vma_lock = vma->vm_private_data; /* - * If pmd sharing possible, locking needed to safely walk the - * hugetlb pgtables. More information can be found at the comment - * above huge_pte_offset() in the same file. + * If the VMA has the hugetlb vma lock (PMD sharable or HGM + * collapsible), locking needed to safely walk the hugetlb pgtables. + * More information can be found at the comment above huge_pte_offset() + * in the same file. + * + * This doesn't do a full high-granularity walk, so we are concerned + * only with PMD unsharing. * * NOTE: lockdep_is_held() is only defined with CONFIG_LOCKDEP. */ - if (__vma_shareable_lock(vma)) + if (__vma_has_hugetlb_vma_lock(vma)) WARN_ON_ONCE(!lockdep_is_held(&vma_lock->rw_sema) && !lockdep_is_held( &vma->vm_file->f_mapping->i_mmap_rwsem)); diff --git a/mm/hugetlb.c b/mm/hugetlb.c index 99fadd7680ec..2f86fedef283 100644 --- a/mm/hugetlb.c +++ b/mm/hugetlb.c @@ -260,7 +260,7 @@ static inline struct hugepage_subpool *subpool_vma(struct vm_area_struct *vma) */ void hugetlb_vma_lock_read(struct vm_area_struct *vma) { - if (__vma_shareable_lock(vma)) { + if (__vma_has_hugetlb_vma_lock(vma)) { struct hugetlb_vma_lock *vma_lock = vma->vm_private_data; down_read(&vma_lock->rw_sema); @@ -269,7 +269,7 @@ void hugetlb_vma_lock_read(struct vm_area_struct *vma) void hugetlb_vma_unlock_read(struct vm_area_struct *vma) { - if (__vma_shareable_lock(vma)) { + if (__vma_has_hugetlb_vma_lock(vma)) { struct hugetlb_vma_lock *vma_lock = vma->vm_private_data; up_read(&vma_lock->rw_sema); @@ -278,7 +278,7 @@ void hugetlb_vma_unlock_read(struct vm_area_struct *vma) void hugetlb_vma_lock_write(struct vm_area_struct *vma) { - if (__vma_shareable_lock(vma)) { + if (__vma_has_hugetlb_vma_lock(vma)) { struct hugetlb_vma_lock *vma_lock = vma->vm_private_data; down_write(&vma_lock->rw_sema); @@ -287,7 +287,7 @@ void hugetlb_vma_lock_write(struct vm_area_struct *vma) void hugetlb_vma_unlock_write(struct vm_area_struct *vma) { - if (__vma_shareable_lock(vma)) { + if (__vma_has_hugetlb_vma_lock(vma)) { struct hugetlb_vma_lock *vma_lock = vma->vm_private_data; up_write(&vma_lock->rw_sema); @@ -298,7 +298,7 @@ int hugetlb_vma_trylock_write(struct vm_area_struct *vma) { struct hugetlb_vma_lock *vma_lock = vma->vm_private_data; - if (!__vma_shareable_lock(vma)) + if (!__vma_has_hugetlb_vma_lock(vma)) return 1; return down_write_trylock(&vma_lock->rw_sema); @@ -306,7 +306,7 @@ int hugetlb_vma_trylock_write(struct vm_area_struct *vma) void hugetlb_vma_assert_locked(struct vm_area_struct *vma) { - if (__vma_shareable_lock(vma)) { + if (__vma_has_hugetlb_vma_lock(vma)) { struct hugetlb_vma_lock *vma_lock = vma->vm_private_data; lockdep_assert_held(&vma_lock->rw_sema); @@ -338,7 +338,7 @@ static void __hugetlb_vma_unlock_write_put(struct hugetlb_vma_lock *vma_lock) static void __hugetlb_vma_unlock_write_free(struct vm_area_struct *vma) { - if (__vma_shareable_lock(vma)) { + if (__vma_has_hugetlb_vma_lock(vma)) { struct hugetlb_vma_lock *vma_lock = vma->vm_private_data; __hugetlb_vma_unlock_write_put(vma_lock); @@ -350,7 +350,7 @@ static void hugetlb_vma_lock_free(struct vm_area_struct *vma) /* * Only present in sharable vmas. */ - if (!vma || !__vma_shareable_lock(vma)) + if (!vma || !__vma_has_hugetlb_vma_lock(vma)) return; if (vma->vm_private_data) {