From patchwork Thu Nov 16 01:28:57 2023 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Peter Xu X-Patchwork-Id: 13457476 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from kanga.kvack.org (kanga.kvack.org [205.233.56.17]) by smtp.lore.kernel.org (Postfix) with ESMTP id 1A12AC2BB3F for ; Thu, 16 Nov 2023 01:29:18 +0000 (UTC) Received: by kanga.kvack.org (Postfix) id 7C2D66B03BD; Wed, 15 Nov 2023 20:29:18 -0500 (EST) Received: by kanga.kvack.org (Postfix, from userid 40) id 771CF6B03DF; Wed, 15 Nov 2023 20:29:18 -0500 (EST) X-Delivered-To: int-list-linux-mm@kvack.org Received: by kanga.kvack.org (Postfix, from userid 63042) id 5C4C56B03E0; Wed, 15 Nov 2023 20:29:18 -0500 (EST) X-Delivered-To: linux-mm@kvack.org Received: from relay.hostedemail.com (smtprelay0011.hostedemail.com [216.40.44.11]) by kanga.kvack.org (Postfix) with ESMTP id 3A41F6B03BD for ; Wed, 15 Nov 2023 20:29:18 -0500 (EST) Received: from smtpin19.hostedemail.com (a10.router.float.18 [10.200.18.1]) by unirelay08.hostedemail.com (Postfix) with ESMTP id 12F6B1401E4 for ; Thu, 16 Nov 2023 01:29:18 +0000 (UTC) X-FDA: 81462084396.19.FF5DE68 Received: from us-smtp-delivery-124.mimecast.com (us-smtp-delivery-124.mimecast.com [170.10.129.124]) by imf30.hostedemail.com (Postfix) with ESMTP id EA1F98000B for ; Thu, 16 Nov 2023 01:29:15 +0000 (UTC) Authentication-Results: imf30.hostedemail.com; dkim=pass header.d=redhat.com header.s=mimecast20190719 header.b=IRb4U1PO; spf=pass (imf30.hostedemail.com: domain of peterx@redhat.com designates 170.10.129.124 as permitted sender) smtp.mailfrom=peterx@redhat.com; dmarc=pass (policy=none) header.from=redhat.com ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=hostedemail.com; s=arc-20220608; t=1700098156; h=from:from:sender:reply-to:subject:subject:date:date: message-id:message-id:to:to:cc:cc:mime-version:mime-version: content-type:content-type: content-transfer-encoding:content-transfer-encoding: in-reply-to:in-reply-to:references:references:dkim-signature; bh=j75oQuDXytIzx0fg9CVxcXlN+sYlHIe4GGC/3d4Gln4=; b=JknCO2oGdz3Qv/n2461/FnMkgb4V851T55VPs4LIfVka4fHgrgHfxGHILbDOTiftxENVYc /QPrDzHYn5w1LOe4XhBZfZS9hG1ttx6fQI9v9/zYN7BqGK1sF1yT6tPnDTaywN261CVN1j vTs6Br30AarL2XOmqtZ30p8gUHZd6bM= ARC-Seal: i=1; s=arc-20220608; d=hostedemail.com; t=1700098156; a=rsa-sha256; cv=none; b=0Oz2T90LtrPifq9JL4By3DZIgK24Z/CKUqK1nYEN4q69vZfM1bxdbucZC8d+Zr0bmyTSem Kr44RSkmKJlS6QiP+IKXD3tMfu2paDr61osgIT46il2yS4x9uJwaE/vVR3ufvWw5Mn6mnm Pbj4P0fyOZYB3Oj08IorNvSZyaoJZjw= ARC-Authentication-Results: i=1; imf30.hostedemail.com; dkim=pass header.d=redhat.com header.s=mimecast20190719 header.b=IRb4U1PO; spf=pass (imf30.hostedemail.com: domain of peterx@redhat.com designates 170.10.129.124 as permitted sender) smtp.mailfrom=peterx@redhat.com; dmarc=pass (policy=none) header.from=redhat.com DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=redhat.com; s=mimecast20190719; t=1700098155; h=from:from:reply-to:subject:subject:date:date:message-id:message-id: to:to:cc:cc:mime-version:mime-version:content-type:content-type: content-transfer-encoding:content-transfer-encoding: in-reply-to:in-reply-to:references:references; bh=j75oQuDXytIzx0fg9CVxcXlN+sYlHIe4GGC/3d4Gln4=; b=IRb4U1POu0WhPCvZObb40AU6wqvMkjWqlqNcfb+fWiH9dO/Xg6pivWiKW35BIRfNKQcrdn J9N7nahPWJ4ovqqVeNv6CGvpVJeES0+RGpknCOifbrDc7y+3D6VM8ui5nGD/NgBVEZtwMd n9rW4MLZ0VYGWlJV40cXro+kZDh5uS0= Received: from mail-qt1-f200.google.com (mail-qt1-f200.google.com [209.85.160.200]) by relay.mimecast.com with ESMTP with STARTTLS (version=TLSv1.3, cipher=TLS_AES_256_GCM_SHA384) id us-mta-185-i_Az9E7QMaWrKt5H0Pln6w-1; Wed, 15 Nov 2023 20:29:13 -0500 X-MC-Unique: i_Az9E7QMaWrKt5H0Pln6w-1 Received: by mail-qt1-f200.google.com with SMTP id d75a77b69052e-421ac2bad66so928341cf.0 for ; Wed, 15 Nov 2023 17:29:13 -0800 (PST) X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20230601; t=1700098153; x=1700702953; h=content-transfer-encoding:mime-version:references:in-reply-to :message-id:date:subject:cc:to:from:x-gm-message-state:from:to:cc :subject:date:message-id:reply-to; bh=j75oQuDXytIzx0fg9CVxcXlN+sYlHIe4GGC/3d4Gln4=; b=QHsTpqSfMk4hEc/jRdjZN9GEl5Nl0P/WpNkP9qWTzufiBoBgUOb1WNg99fLQzHmIQ8 I6/a6Vuc/+RQvbNQkdjCM4NBd94/02FNerltYnXalTtE4zvj73mgGiqRMa4goRZ2KYmn tftwONzSU0HefAhwX3QZi16Z5+6rvxYKWON8WW58vOMl0UPFasokgmHBIoVRDwtZRo9J RHZWgxBiWvnmEtNf+MXlSHhANVzLAgKNnFBN5wTnffC2Pz8fjnO96haeLFxksh+5XZaB Wt067+89jkSjUbnRr6rYncG1R3BEKYkRY/9pvDDmMH/ilBvmiMJaTUZ0aWczRz/sr1vG 1Cpg== X-Gm-Message-State: AOJu0Yzt7GM6GXFGRfBIh3SukTgmR2z/yrctLFiUPhZuI8mJg5igMxP2 svbxYXuwK99pR3oU435t6YQ7cMCO/OC6nXoAzZiLbq5lwvnNIFhK0P3dcU84uUAvgK4z49Bho0r rw47YHWtsAGI= X-Received: by 2002:ac8:7d13:0:b0:41e:4be2:d3eb with SMTP id g19-20020ac87d13000000b0041e4be2d3ebmr10348612qtb.1.1700098153059; Wed, 15 Nov 2023 17:29:13 -0800 (PST) X-Google-Smtp-Source: AGHT+IGyA/ppnbzE3SBmMhc1vSGcpOQYT5DWDMM4yQxJwgB1XG6+271v4ghGsJVWP9HBrNNkwutL1g== X-Received: by 2002:ac8:7d13:0:b0:41e:4be2:d3eb with SMTP id g19-20020ac87d13000000b0041e4be2d3ebmr10348586qtb.1.1700098152747; Wed, 15 Nov 2023 17:29:12 -0800 (PST) Received: from x1n.redhat.com (cpe5c7695f3aee0-cm5c7695f3aede.cpe.net.cable.rogers.com. [99.254.144.39]) by smtp.gmail.com with ESMTPSA id c24-20020ac85198000000b0041e383d527esm3922598qtn.66.2023.11.15.17.29.11 (version=TLS1_3 cipher=TLS_AES_256_GCM_SHA384 bits=256/256); Wed, 15 Nov 2023 17:29:12 -0800 (PST) From: Peter Xu To: linux-kernel@vger.kernel.org, linux-mm@kvack.org Cc: Mike Kravetz , "Kirill A . Shutemov" , Lorenzo Stoakes , Axel Rasmussen , Matthew Wilcox , John Hubbard , Mike Rapoport , peterx@redhat.com, Hugh Dickins , David Hildenbrand , Andrea Arcangeli , Rik van Riel , James Houghton , Yang Shi , Jason Gunthorpe , Vlastimil Babka , Andrew Morton Subject: [PATCH RFC 01/12] mm/hugetlb: Export hugetlbfs_pagecache_present() Date: Wed, 15 Nov 2023 20:28:57 -0500 Message-ID: <20231116012908.392077-2-peterx@redhat.com> X-Mailer: git-send-email 2.41.0 In-Reply-To: <20231116012908.392077-1-peterx@redhat.com> References: <20231116012908.392077-1-peterx@redhat.com> MIME-Version: 1.0 X-Mimecast-Spam-Score: 0 X-Mimecast-Originator: redhat.com X-Stat-Signature: ykaqxnhnqf69rb4w9fz78rww5wskc9qw X-Rspamd-Server: rspam10 X-Rspamd-Queue-Id: EA1F98000B X-Rspam-User: X-HE-Tag: 1700098155-354664 X-HE-Meta: U2FsdGVkX1/1syfZLgmeyS70MOwz0RVwpl89szpZ27e0TACZ2/PbUpsC9RAu9/ssYLSBOoAxHe67Cb0jROsw/9aPUXYeH+3vozP8SJ8+jbbbtZipZpGsPkq7RRl+4C0bVr/sJXyix+q3iLXJs4Jxpr5RePuCgUqeJh0EJ4zCOOmQHVvCbJlQ8mubrcW2uQDd9F26iJqBYdjPbOXD/In4Pwtzmaz/MU7541I1MbnDq0qI8AR9feWbUdSw7vXFVp1AqleCJgjI7dxLwhRDhncIakivLpu2Q8LqPUUxVqCvOwwhFp65DJ+Htn6TudrQMeC/xR3UztBSFkCHuIqKVQUZZQqo24wqoBQyqIrndTVDEh8tqnroPR8WDQH41m4U6nK9kuaMSgXbCZjzdv1IONA1SIKd6Z0SJnW7KK+jKAqFVMSev8++VqjQhJcJkxPZZMhMWPud+wMb9UVwZ0xGHfLka9e2NOcieX8KLs7nu3sc6wT+unIq1LqnFIYbitM1YxmC7kpL4wcESQUhrXWGzVGnYtQ3JIxl/mtKuqnKyhmoNLHxUjmf35QXfUrkSJWMsc7GKD6P4mxNlzccYnROPKj1nYbkApi/oqAMi65QZO9qk0acYrbZWJdc/8957H4v/AQtq/RkEIc47Cl2m4rclR/RZ9Gvus8VX/B/U1f/Bx+Tz1mj0UTH/2VNlSYyfqhDAnCVlL8yVkz3B2F3tDXbVJ8IG4H8kbWeO21f4B0YiLYtd1n1lMVAFBPUL3hXEHa8FbPjNaEieHFK6ZVyywkjYLqZ4+8viXEkNb3uCMcSBlpfbbOZyZEltSSc3BRXk1Jrfn4lfBzXPSPEccNI9/+fQtDsm8iqu8bnxSJU0CI86VYVPrBN72BBfkKIurJtrVSvLvNevGwFvatMVQzOzDQlWWjN56kLngXnTQ0fy28N/WNpLT2VVE40OX20UrG2j1NYdOb0XRkXkMscrTsbKtE2zOl KYBVREVj 7tneaWYBto+x0Zyoq5SBNWh/3QJFKnXjnGfKmSljMyAJFVDxlFBtk+zqiLlHc8JzD5TUXI8yKbfXPEjBWlhixqhBORhSphpZi56jnzEwkrYcIFK2diPwvTcwJUiYlG9CxEDJe6mK8/Kyml04PwPRt4hcg8N4CaOLnSGFvRsYlBIEzMYhWVhoCn7KQw2sF2O1UMbj8Q3Dbn9lNiKd0DmVFoYAp/80KYdRIEyok4uqrf0ujo2hpvmPwD2r4UIdy7TH0NtEqXJiHJ5X0JIK7/lvj7HW8tVEunbEu0S9RnPfWf9Yir4k6ZqOaHZenPiz6jU7Dp7FDRps9RIx+jTxTW/v/KBd2MwSR5d83KQqjB1n1OMB+NbOyYtpiG1t/71uMzIdNbueix9jwHVRHU2qoN5LdtqVCLp3bUR/s4ipbKGJb5Oo2kJOHxeNA3PlMdQ== X-Bogosity: Ham, tests=bogofilter, spamicity=0.000000, version=1.2.4 Sender: owner-linux-mm@kvack.org Precedence: bulk X-Loop: owner-majordomo@kvack.org List-ID: List-Subscribe: List-Unsubscribe: It will be used outside hugetlb.c soon. Signed-off-by: Peter Xu --- include/linux/hugetlb.h | 3 +++ mm/hugetlb.c | 4 ++-- 2 files changed, 5 insertions(+), 2 deletions(-) diff --git a/include/linux/hugetlb.h b/include/linux/hugetlb.h index 236ec7b63c54..bb07279b8991 100644 --- a/include/linux/hugetlb.h +++ b/include/linux/hugetlb.h @@ -174,6 +174,9 @@ u32 hugetlb_fault_mutex_hash(struct address_space *mapping, pgoff_t idx); pte_t *huge_pmd_share(struct mm_struct *mm, struct vm_area_struct *vma, unsigned long addr, pud_t *pud); +bool hugetlbfs_pagecache_present(struct hstate *h, + struct vm_area_struct *vma, + unsigned long address); struct address_space *hugetlb_page_mapping_lock_write(struct page *hpage); diff --git a/mm/hugetlb.c b/mm/hugetlb.c index 6feb3e0630d1..29705e5c6f40 100644 --- a/mm/hugetlb.c +++ b/mm/hugetlb.c @@ -6018,8 +6018,8 @@ static vm_fault_t hugetlb_wp(struct mm_struct *mm, struct vm_area_struct *vma, /* * Return whether there is a pagecache page to back given address within VMA. */ -static bool hugetlbfs_pagecache_present(struct hstate *h, - struct vm_area_struct *vma, unsigned long address) +bool hugetlbfs_pagecache_present(struct hstate *h, + struct vm_area_struct *vma, unsigned long address) { struct address_space *mapping = vma->vm_file->f_mapping; pgoff_t idx = linear_page_index(vma, address); From patchwork Thu Nov 16 01:28:58 2023 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Peter Xu X-Patchwork-Id: 13457477 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from kanga.kvack.org (kanga.kvack.org [205.233.56.17]) by smtp.lore.kernel.org (Postfix) with ESMTP id 0D74EC072A2 for ; Thu, 16 Nov 2023 01:29:21 +0000 (UTC) Received: by kanga.kvack.org (Postfix) id 502826B03DF; Wed, 15 Nov 2023 20:29:20 -0500 (EST) Received: by kanga.kvack.org (Postfix, from userid 40) id 48B106B03E1; Wed, 15 Nov 2023 20:29:20 -0500 (EST) X-Delivered-To: int-list-linux-mm@kvack.org Received: by kanga.kvack.org (Postfix, from userid 63042) id 2DDBB6B03E2; Wed, 15 Nov 2023 20:29:20 -0500 (EST) X-Delivered-To: linux-mm@kvack.org Received: from relay.hostedemail.com (smtprelay0010.hostedemail.com [216.40.44.10]) by kanga.kvack.org (Postfix) with ESMTP id 18E0D6B03DF for ; Wed, 15 Nov 2023 20:29:20 -0500 (EST) Received: from smtpin09.hostedemail.com (a10.router.float.18 [10.200.18.1]) by unirelay03.hostedemail.com (Postfix) with ESMTP id EE530A0482 for ; Thu, 16 Nov 2023 01:29:19 +0000 (UTC) X-FDA: 81462084438.09.95D75EC Received: from us-smtp-delivery-124.mimecast.com (us-smtp-delivery-124.mimecast.com [170.10.133.124]) by imf17.hostedemail.com (Postfix) with ESMTP id D871940009 for ; Thu, 16 Nov 2023 01:29:17 +0000 (UTC) Authentication-Results: imf17.hostedemail.com; dkim=pass header.d=redhat.com header.s=mimecast20190719 header.b=LvzUWBEI; dmarc=pass (policy=none) header.from=redhat.com; spf=pass (imf17.hostedemail.com: domain of peterx@redhat.com designates 170.10.133.124 as permitted sender) smtp.mailfrom=peterx@redhat.com ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=hostedemail.com; s=arc-20220608; t=1700098157; h=from:from:sender:reply-to:subject:subject:date:date: message-id:message-id:to:to:cc:cc:mime-version:mime-version: content-type:content-type: content-transfer-encoding:content-transfer-encoding: in-reply-to:in-reply-to:references:references:dkim-signature; bh=Rr9/qVNLgFGWDGzcAqjV00T9t0NzgxXDD6w387jcEYI=; b=3jGa/+cWeX7qtsnvn+xEXYIvcPJLw2QM3ORVUnWgps0ZGrsG0P4xyDfaRnXoDwndAkDajy di7VxKWNfBO9kRfaA+a5ZXOaUvDy+Bhnc1y6ekN2+Ep2/2u08FPddlpRSQzjDUNON1hr0c E3+peHRw1RY0sG+IYBfUhneT4c11lXw= ARC-Authentication-Results: i=1; imf17.hostedemail.com; dkim=pass header.d=redhat.com header.s=mimecast20190719 header.b=LvzUWBEI; dmarc=pass (policy=none) header.from=redhat.com; spf=pass (imf17.hostedemail.com: domain of peterx@redhat.com designates 170.10.133.124 as permitted sender) smtp.mailfrom=peterx@redhat.com ARC-Seal: i=1; s=arc-20220608; d=hostedemail.com; t=1700098157; a=rsa-sha256; cv=none; b=ElO7cCCG7cghuYH+gDJvYCrLA2GswtPeCWo7kpUAGMHn4hWCy8n3rn3dVkAJ7RmT5ZUdgJ ZsP5wXV6vkI6L92L3y+tpAVD/WsFl0zh9Mw/7HMZEX0wIkPjCVVOdrJwwBCKUS3OItsmyo G7QARBEqB6tgon2oVYIhT149a4M7bTM= DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=redhat.com; s=mimecast20190719; t=1700098157; h=from:from:reply-to:subject:subject:date:date:message-id:message-id: to:to:cc:cc:mime-version:mime-version:content-type:content-type: content-transfer-encoding:content-transfer-encoding: in-reply-to:in-reply-to:references:references; bh=Rr9/qVNLgFGWDGzcAqjV00T9t0NzgxXDD6w387jcEYI=; b=LvzUWBEIUIV+qlm7ECl/EkQelJFUv93LiT047th0Ogirsg5Hbh++Yn0WVMWSKlo266eDj/ 6A4MyNXP8ghbviPrXumA1sxN3sftCUqCcWym+xgg+HMFZ+9oPsuDxT3+mZ2o7E6g5ayjpt Rohg1g/XBtOXnppsKx1Gm/oezP3bT5w= Received: from mail-qk1-f198.google.com (mail-qk1-f198.google.com [209.85.222.198]) by relay.mimecast.com with ESMTP with STARTTLS (version=TLSv1.3, cipher=TLS_AES_256_GCM_SHA384) id us-mta-606-j3q-Fc-8PEKNpXeinqKNCQ-1; Wed, 15 Nov 2023 20:29:15 -0500 X-MC-Unique: j3q-Fc-8PEKNpXeinqKNCQ-1 Received: by mail-qk1-f198.google.com with SMTP id af79cd13be357-77a02ceef95so2901085a.0 for ; Wed, 15 Nov 2023 17:29:15 -0800 (PST) X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20230601; t=1700098154; x=1700702954; h=content-transfer-encoding:mime-version:references:in-reply-to :message-id:date:subject:cc:to:from:x-gm-message-state:from:to:cc :subject:date:message-id:reply-to; bh=Rr9/qVNLgFGWDGzcAqjV00T9t0NzgxXDD6w387jcEYI=; b=VeM+hc4beo7e0U2fgWPiG4u/eiy2cwVzKnNWU45PwWjAadepL1jQlNQJhxjwADVru/ xajUQG3xIFMg38LSjKKe/9YbWodsnhe+q0xyrMI6Qla+SAXn1oyJRHbBO4Lm8fhnB0g/ mLjuYFW/aoPV42wDRkEbPysKRE1Qs6Bgn3msnrgxftLAgbHWOZoDUgx68yEliH8AZxYl SAq/MDxGqTXilMslPmEPyaVF7HWQm0YWf6h4H4X/j+srlcPK0WR9tAX6tYB/jFBy5DiX zc3a/qcxJzNmW5w+TVXNrWsy21ZIuPRx7n9+4mOGx2VKZ5qUDFDutsUrgi08ljulM0/1 IxIg== X-Gm-Message-State: AOJu0Yyenq4Lm/JxZnDBNxTr+vxS3r3vSsyCN6mcpbKNWNHydnl182fQ 95M/ZMjxQW8aTNSgVCJKZ56JC29WcjsgXtLgDuvVoL+YcBmzU+JrYG6YcXaj61ycXWY3UPebZzN t1y3YuL0BKuY= X-Received: by 2002:ac8:5189:0:b0:421:c275:f917 with SMTP id c9-20020ac85189000000b00421c275f917mr7201788qtn.6.1700098154664; Wed, 15 Nov 2023 17:29:14 -0800 (PST) X-Google-Smtp-Source: AGHT+IFImkFHmltxmevzx1fu9EK+pMFz+bfHbyIeYa7FxbalfbUu9NzmbSIrMhaQwUJeT0EwTIDDnQ== X-Received: by 2002:ac8:5189:0:b0:421:c275:f917 with SMTP id c9-20020ac85189000000b00421c275f917mr7201781qtn.6.1700098154444; Wed, 15 Nov 2023 17:29:14 -0800 (PST) Received: from x1n.redhat.com (cpe5c7695f3aee0-cm5c7695f3aede.cpe.net.cable.rogers.com. [99.254.144.39]) by smtp.gmail.com with ESMTPSA id c24-20020ac85198000000b0041e383d527esm3922598qtn.66.2023.11.15.17.29.13 (version=TLS1_3 cipher=TLS_AES_256_GCM_SHA384 bits=256/256); Wed, 15 Nov 2023 17:29:14 -0800 (PST) From: Peter Xu To: linux-kernel@vger.kernel.org, linux-mm@kvack.org Cc: Mike Kravetz , "Kirill A . Shutemov" , Lorenzo Stoakes , Axel Rasmussen , Matthew Wilcox , John Hubbard , Mike Rapoport , peterx@redhat.com, Hugh Dickins , David Hildenbrand , Andrea Arcangeli , Rik van Riel , James Houghton , Yang Shi , Jason Gunthorpe , Vlastimil Babka , Andrew Morton Subject: [PATCH RFC 02/12] mm: Provide generic pmd_thp_or_huge() Date: Wed, 15 Nov 2023 20:28:58 -0500 Message-ID: <20231116012908.392077-3-peterx@redhat.com> X-Mailer: git-send-email 2.41.0 In-Reply-To: <20231116012908.392077-1-peterx@redhat.com> References: <20231116012908.392077-1-peterx@redhat.com> MIME-Version: 1.0 X-Mimecast-Spam-Score: 0 X-Mimecast-Originator: redhat.com X-Rspamd-Queue-Id: D871940009 X-Rspam-User: X-Rspamd-Server: rspam02 X-Stat-Signature: sotxjxzx7cm6ynw9osun8ztkph4o1ojs X-HE-Tag: 1700098157-50759 X-HE-Meta: U2FsdGVkX18glz/QwOnIXbQ8rIKjWXveWyVDIzTqx8IeqKV+a8ts5cl4DSSrZ3xjl1rWUQC5Np4uVgI54vrRL9C0GxzobrVtt6hcCdLfbtmHBZX99Dy2gMbY2oNb9aodDJo1Q0K4p9rfRdT/l5suqYYp2eWLMOB7ZskENjIiYllmM1couziTbOKdLuY7HUp/Q+OtS6CfHIwd1WUBQR+Cyy0j0sMYtbaAovpuUsVixFFWUt95GiSJ60FAPQfRoJCatla93FeOPyFVe8auE4H8avTGGdk8ppVug8xCMUWYKfhVKPJzqxcl3wpI71jIcdA3VY4ZgkdOjspf/Z/V9eqimzZqC5TeQnmCKPtlp4opX2MjLQnzjRd04vll61Faq6OgMimp5gAAIe/zrLO8yZBPKRkRNlUCkswBrnAyAuQ4DvN+ezKjNdopQKOunkZ8aAkg13yhyMVLgaH6cu82Cq+Ug9Xnl1mGf7JgfO1cukBQ0/aSET9qAYgfj39mgSYk2pWy5qXfoN6dr5OQY1jpffnoxUuGUnQQTFdborbVGaNTKG2GdQ0oxWiubbKdkFX7vWw7tSeSntoB/TXFk551f1+qYahTbyHp++ZfYOECENJ3z10hDbfnv/Nnd5+dw6AZ0wwLxEFskOBwxDk3QUNPsl92jyUQPVbSGrf5vENCZNrHbCXmINbeQ2kMfq9z3N/Reurb2LIHwHj44Z4u/hHrHW1GQAuO7Dz901EHe/v8DXWMx0UzRJUe5RAYvYGMX01XHaK2ikbT2qHjlHLYlLsTuqaLaPZrSY+rMYVGVYPoaCAx6q7KHXRttDkQ/DFli5K4ATRIQKxFGzqQvq3ALyM48+6ZcfAZnLkFGkKa97TVb4Cw8yKbz580KColRFqBUP+xOB1EzwsDASyARPdfQ+u+bVhiGUPtROk/ZLbvF7ahHwaybNzVMvbeHkYQMgPNuPDGLdKL+XP1w7+SxbtcfWHPB3H zqNybZr7 Pq4AAmaaoawiOFvn6d4m74+B0QFkRpi6WZTJyJjQ+MaF6AVodGSGS3XGEkR9ijzLNlQX2P0j3LQXnkjbBPCDVGpiZ94WnUWcGgJRdsdy+lH+IxOy4U/LG8Sg5+W2XH7LNqcW+cwGDLlBm8rpniBNMvDzdgv3WH4oQ83HUHJgdtpWfmSTLzAwLasSiP9KJO/ccQWmfYerUochBfpi0ZSpYyuTrLuo6P2WCZXig7qZxHu5KTZp7aFDe6omuneIYI4njxZk32JZaUK90QD03abxB+t2AAotpCLBPBsEtMtbIBVeGQS5I1irSsf7UebiQXH89WLu7wkKC+dEOcePRdiudcNHoOrlNr+vf5DnOGuxdDCfQEY5pQRZHGlD8CX/I4zfwx/64VCJ2cJMpn6Vn5mav2IjLAHN/Sg7Tqb3iaQ9mBbaa9I9eutgJjYfL5Q== X-Bogosity: Ham, tests=bogofilter, spamicity=0.000000, version=1.2.4 Sender: owner-linux-mm@kvack.org Precedence: bulk X-Loop: owner-majordomo@kvack.org List-ID: List-Subscribe: List-Unsubscribe: ARM defines pmd_thp_or_huge(), detecting either a THP or a huge PMD. It can be a helpful helper if we want to merge more THP and hugetlb code paths. Make it a generic default implementation, only exist when CONFIG_MMU. Arch can overwrite it by defining its own version. For example, ARM's pgtable-2level.h defines it to always return false. Signed-off-by: Peter Xu --- include/linux/pgtable.h | 4 ++++ mm/gup.c | 3 +-- 2 files changed, 5 insertions(+), 2 deletions(-) diff --git a/include/linux/pgtable.h b/include/linux/pgtable.h index af7639c3b0a3..6f2fa1977b8a 100644 --- a/include/linux/pgtable.h +++ b/include/linux/pgtable.h @@ -1355,6 +1355,10 @@ static inline int pmd_write(pmd_t pmd) #endif /* pmd_write */ #endif /* CONFIG_TRANSPARENT_HUGEPAGE */ +#ifndef pmd_thp_or_huge +#define pmd_thp_or_huge(pmd) (pmd_huge(pmd) || pmd_trans_huge(pmd)) +#endif + #ifndef pud_write static inline int pud_write(pud_t pud) { diff --git a/mm/gup.c b/mm/gup.c index 231711efa390..a8b73a8289ad 100644 --- a/mm/gup.c +++ b/mm/gup.c @@ -3002,8 +3002,7 @@ static int gup_pmd_range(pud_t *pudp, pud_t pud, unsigned long addr, unsigned lo if (!pmd_present(pmd)) return 0; - if (unlikely(pmd_trans_huge(pmd) || pmd_huge(pmd) || - pmd_devmap(pmd))) { + if (unlikely(pmd_thp_or_huge(pmd) || pmd_devmap(pmd))) { /* See gup_pte_range() */ if (pmd_protnone(pmd)) return 0; From patchwork Thu Nov 16 01:28:59 2023 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Peter Xu X-Patchwork-Id: 13457478 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from kanga.kvack.org (kanga.kvack.org [205.233.56.17]) by smtp.lore.kernel.org (Postfix) with ESMTP id 9B74AC2BB3F for ; Thu, 16 Nov 2023 01:29:23 +0000 (UTC) Received: by kanga.kvack.org (Postfix) id 2DF986B03E1; Wed, 15 Nov 2023 20:29:22 -0500 (EST) Received: by kanga.kvack.org (Postfix, from userid 40) id 28FA16B03E3; Wed, 15 Nov 2023 20:29:22 -0500 (EST) X-Delivered-To: int-list-linux-mm@kvack.org Received: by kanga.kvack.org (Postfix, from userid 63042) id 0E29C6B03E4; Wed, 15 Nov 2023 20:29:21 -0500 (EST) X-Delivered-To: linux-mm@kvack.org Received: from relay.hostedemail.com (smtprelay0017.hostedemail.com [216.40.44.17]) by kanga.kvack.org (Postfix) with ESMTP id E684A6B03E1 for ; Wed, 15 Nov 2023 20:29:21 -0500 (EST) Received: from smtpin01.hostedemail.com (a10.router.float.18 [10.200.18.1]) by unirelay08.hostedemail.com (Postfix) with ESMTP id C24EE14030F for ; Thu, 16 Nov 2023 01:29:21 +0000 (UTC) X-FDA: 81462084522.01.E86070C Received: from us-smtp-delivery-124.mimecast.com (us-smtp-delivery-124.mimecast.com [170.10.129.124]) by imf10.hostedemail.com (Postfix) with ESMTP id C3DCCC000D for ; Thu, 16 Nov 2023 01:29:19 +0000 (UTC) Authentication-Results: imf10.hostedemail.com; dkim=pass header.d=redhat.com header.s=mimecast20190719 header.b="WwyG/LGz"; dmarc=pass (policy=none) header.from=redhat.com; spf=pass (imf10.hostedemail.com: domain of peterx@redhat.com designates 170.10.129.124 as permitted sender) smtp.mailfrom=peterx@redhat.com ARC-Seal: i=1; s=arc-20220608; d=hostedemail.com; t=1700098159; a=rsa-sha256; cv=none; b=qdgtgGcdvfeb1lVSFGpWayEmbzOGIawCEKVHrgK4hGwekZV4KQusL6US0th3VpFz7UGiMd OfoBCKAlGcKM9279Jq6rdz84qHFqLp/NZHY5geZnxFBxvkviMT9Vc1fiw7hbgObMepCxhP 0JKZ/yE1tZTYJ7y9uKwLNd7hmqMwyGM= ARC-Authentication-Results: i=1; imf10.hostedemail.com; dkim=pass header.d=redhat.com header.s=mimecast20190719 header.b="WwyG/LGz"; dmarc=pass (policy=none) header.from=redhat.com; spf=pass (imf10.hostedemail.com: domain of peterx@redhat.com designates 170.10.129.124 as permitted sender) smtp.mailfrom=peterx@redhat.com ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=hostedemail.com; s=arc-20220608; t=1700098159; h=from:from:sender:reply-to:subject:subject:date:date: message-id:message-id:to:to:cc:cc:mime-version:mime-version: content-type:content-type: content-transfer-encoding:content-transfer-encoding: in-reply-to:in-reply-to:references:references:dkim-signature; bh=PGiD5LrOPbNAVIZVL9CsLOYGNogun3F0ZNXBV5lOZSk=; b=1aog08BGM91dqQgEZ2BN0pXJWpeaR9OrHLoASu9SSs6PmFPhuiTUs4kxRCZzv5se1ZckD9 1RDt8WZreemrW+ny4R+xK4a3RaWme/HWoir9js/cpscMnvpfiD5ULE5Yfud7xXGiV09g4Q 3u3ZKqFmnSW1m+i0+ThmJXJJbDDrkoY= DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=redhat.com; s=mimecast20190719; t=1700098159; h=from:from:reply-to:subject:subject:date:date:message-id:message-id: to:to:cc:cc:mime-version:mime-version:content-type:content-type: content-transfer-encoding:content-transfer-encoding: in-reply-to:in-reply-to:references:references; bh=PGiD5LrOPbNAVIZVL9CsLOYGNogun3F0ZNXBV5lOZSk=; b=WwyG/LGzQp4vKzMrnRAVdNhXyLw9/9Z5cn2YgptouI4zc+tIpAlEu8wyXKqb8eMzqcZkv5 5y6cOWj+s/7fe5wHkIG8C5j5cXzl3OFUBHG6Ax8BvX/+NBYYuVWCE9IV+3tBQ2Gmw8tnKr c11TLgt/unDsH2WGCxZal9nJ/+es9nI= Received: from mail-qt1-f198.google.com (mail-qt1-f198.google.com [209.85.160.198]) by relay.mimecast.com with ESMTP with STARTTLS (version=TLSv1.3, cipher=TLS_AES_256_GCM_SHA384) id us-mta-637-ZQ9xhaPjOmKUZZjttFcSRg-1; Wed, 15 Nov 2023 20:29:16 -0500 X-MC-Unique: ZQ9xhaPjOmKUZZjttFcSRg-1 Received: by mail-qt1-f198.google.com with SMTP id d75a77b69052e-41cd9e05c8bso881081cf.1 for ; Wed, 15 Nov 2023 17:29:16 -0800 (PST) X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20230601; t=1700098156; x=1700702956; h=content-transfer-encoding:mime-version:references:in-reply-to :message-id:date:subject:cc:to:from:x-gm-message-state:from:to:cc :subject:date:message-id:reply-to; bh=PGiD5LrOPbNAVIZVL9CsLOYGNogun3F0ZNXBV5lOZSk=; b=gNjGxWDEaYdYrRNqIrxvcYbmcP5KEJRMilwSLPKAnR1FeqgLMM3oa9Pp28qo1FrYC4 8IynIA1iqrrzNG2DtDIouKniQbvnw7b/XJefkBrW3orGIvm4MkCBXFAVn8sEJFie8Ige SaGoRc66n8kcCvgfBKkbACoxKXw+0IT9e611InpErfnhOXp6iXYk0m6Dl8AAOaWIm0Nx iGkw5KIF6/BVFwpAhPTRLCTu4Z2Z1Td10kO7yIN5dHeDj1pj8kIr+i81U7086UfRyPft +8EdIgbEXz8IXHr3Vns+Sm+bhGRO81gcbt4bHYGixKSZ8Dl6csA2xte++ZhjKFP4+/w9 jb6Q== X-Gm-Message-State: AOJu0Yz4WYGhE7ZDY8KGQfZNPZFxRyykfrw+3bn7mXBTSrPLofMsyZZo ssPc2E5ADbHSR1d/U6bHlIXUpvjPoPdo9uM84fesyI9aHTmVmO/Yw9Tp1awLwf6F0Jjs1KUQcqM yffw245BtOuc= X-Received: by 2002:a05:622a:8008:b0:41e:a62b:3d28 with SMTP id jr8-20020a05622a800800b0041ea62b3d28mr6714090qtb.4.1700098156300; Wed, 15 Nov 2023 17:29:16 -0800 (PST) X-Google-Smtp-Source: AGHT+IEODK+EO2T6pc5X1e/ytZXEP5MAlQvhJeor+MetW91HDS49FOIp5Z13gLBn7JzEPU2jgEp9Cg== X-Received: by 2002:a05:622a:8008:b0:41e:a62b:3d28 with SMTP id jr8-20020a05622a800800b0041ea62b3d28mr6714076qtb.4.1700098155869; Wed, 15 Nov 2023 17:29:15 -0800 (PST) Received: from x1n.redhat.com (cpe5c7695f3aee0-cm5c7695f3aede.cpe.net.cable.rogers.com. [99.254.144.39]) by smtp.gmail.com with ESMTPSA id c24-20020ac85198000000b0041e383d527esm3922598qtn.66.2023.11.15.17.29.14 (version=TLS1_3 cipher=TLS_AES_256_GCM_SHA384 bits=256/256); Wed, 15 Nov 2023 17:29:15 -0800 (PST) From: Peter Xu To: linux-kernel@vger.kernel.org, linux-mm@kvack.org Cc: Mike Kravetz , "Kirill A . Shutemov" , Lorenzo Stoakes , Axel Rasmussen , Matthew Wilcox , John Hubbard , Mike Rapoport , peterx@redhat.com, Hugh Dickins , David Hildenbrand , Andrea Arcangeli , Rik van Riel , James Houghton , Yang Shi , Jason Gunthorpe , Vlastimil Babka , Andrew Morton Subject: [PATCH RFC 03/12] mm: Export HPAGE_PXD_* macros even if !THP Date: Wed, 15 Nov 2023 20:28:59 -0500 Message-ID: <20231116012908.392077-4-peterx@redhat.com> X-Mailer: git-send-email 2.41.0 In-Reply-To: <20231116012908.392077-1-peterx@redhat.com> References: <20231116012908.392077-1-peterx@redhat.com> MIME-Version: 1.0 X-Mimecast-Spam-Score: 0 X-Mimecast-Originator: redhat.com X-Rspam-User: X-Rspamd-Server: rspam06 X-Rspamd-Queue-Id: C3DCCC000D X-Stat-Signature: msjyzi9sqt8k1ubn1x3h4ojqat7pewcu X-HE-Tag: 1700098159-213681 X-HE-Meta: U2FsdGVkX1/Ovi3In6FyXW1To8rw15QGFjnJm17U7j0NYeLTQW13QeRBO+8pYyZKzt6gkP/jHXxRYC52LHoeuShxT76UoA8wplXLMKFyAipdEv8dFwdOyxwouoIpGPbQAwb0Hm10kNiNOW2smmEbOL/Tj/z5TvBDmJLSC95Z07un/ATFffoaRESXuO3NY7qbaHZpKucd7V8gPcqjgG0pfew9pL6X150uJ1QNVtB8vija22QtQPMGYfLs4fM8g3ybfzdHmewES6Lu7grKiblWizj8/FtjZ8h2jCyKPALhBkqyrM1HUKIBMY5zi7bBLBvbVq7N5TyhD8mpQcpydTcGcAhfbdpux7Xx2zdUi7WKQ0I7QjoVVDLtjfBzddKeVCwjYOX1DwXk9/Dc2NWGr8CixRGL4UzadwAzfj0eGRCIl1EMBg+eCxCaEhiWCaL1GzoeUe96iXNtlTvWCA+MEuG4Eg6EBUZUr0IUibIWikxugnin+T60s7mGH9v4nCjQc7rxqX1zTA7BCawWwP/dyY90xnX2QM10K/FwlbphVZJw6cDScNd9t5H5bAgOzWi/8WuuYmFun7OY1TnT01umy6uNZsAlb6PT+Iaz5jcZ5wk3S2/kWh+6stnzfy6YtuYi3KcJtKzR+KjTBM+2hhOlpCy3NfKgsvenYJsqmrl2D4WyrG2DWe764pguUpR26CzM5E2lzmttZDNE/5VmtTMtWz3yPZ6hPTnqIxdRF3FI2kC49RerFmykqn9T/+1rxJvMD6QX2XdXmyC2x2xF5CbMW99ENCqrNd8u3BdM0HZfr5Ylqh5zVtH9aUwYxJXrUzZ0vN+MRRjztx27kfQwm7tkuv74JYhasXBPNCs8toZpfGVnNaYKc0sSSyOM86Uyul7PEhA2RHCV3jHsIS1lCInY+bTvgAu9BMH7IxVQYzttEwVrcwqBX/iMHtnHCRE0K9WCqKCuJ8JQV2SIxP6jn5OaROJ wviZfSMZ 2/y++avnfd5WSyJKn9ItAU2QfOOiA/xFJaUiW8lxzE3owBko0mBkU2ukUqIyP4GumubpFePP5jdtZm0YEbaDcw0tOzJk+7e2k9bxd+0NTY8vld/ewh08VZieBC3d3K0l72+NMSA3vgGt8l/+QA0R6SEUkAyM8VJvMcOEoYwJ8YW7WH1BmMcONK82EYWjgNyYxpiZGEh32saCOeQKrqeX5yk1fIYeNozv/2e9kCKFlGUwombZzd0UUKbfXjVcZCULEfsvQA16rHYSNZtpTG8B9mdHYh9cMAzmkh12vh5J2iN1argMdLeGZRo7mtoxHKqHfy7VujqzOQqK2Y/FANS+4+eDIC//PP8m1hnNCZax4mo5d2N0DVjmroYAldgD+oZCUoO2O1a9VhHRY4R/qz7z8aJqf2fdw/YkGaPm4or6T+jXFVlxMe0dh4ypmUjMBjQqB5vM637SNE4cwz9jdV1klQToOK80dzkCMR5SV X-Bogosity: Ham, tests=bogofilter, spamicity=0.000000, version=1.2.4 Sender: owner-linux-mm@kvack.org Precedence: bulk X-Loop: owner-majordomo@kvack.org List-ID: List-Subscribe: List-Unsubscribe: These macros can be helpful when we plan to merge hugetlb code into generic code. Move them out and define them even if !THP. We actually already defined HPAGE_PMD_NR for other reasons even if !THP. Reorganize these macros. Signed-off-by: Peter Xu Reviewed-by: Christoph Hellwig --- include/linux/huge_mm.h | 17 ++++++----------- 1 file changed, 6 insertions(+), 11 deletions(-) diff --git a/include/linux/huge_mm.h b/include/linux/huge_mm.h index fa0350b0812a..ec463410aecc 100644 --- a/include/linux/huge_mm.h +++ b/include/linux/huge_mm.h @@ -64,17 +64,19 @@ ssize_t single_hugepage_flag_show(struct kobject *kobj, enum transparent_hugepage_flag flag); extern struct kobj_attribute shmem_enabled_attr; -#define HPAGE_PMD_ORDER (HPAGE_PMD_SHIFT-PAGE_SHIFT) -#define HPAGE_PMD_NR (1< X-Patchwork-Id: 13457479 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from kanga.kvack.org (kanga.kvack.org [205.233.56.17]) by smtp.lore.kernel.org (Postfix) with ESMTP id 067FBC5ACB2 for ; Thu, 16 Nov 2023 01:29:26 +0000 (UTC) Received: by kanga.kvack.org (Postfix) id C810B6B03E3; Wed, 15 Nov 2023 20:29:22 -0500 (EST) Received: by kanga.kvack.org (Postfix, from userid 40) id C07896B03E5; Wed, 15 Nov 2023 20:29:22 -0500 (EST) X-Delivered-To: int-list-linux-mm@kvack.org Received: by kanga.kvack.org (Postfix, from userid 63042) id A08BA6B03E6; Wed, 15 Nov 2023 20:29:22 -0500 (EST) X-Delivered-To: linux-mm@kvack.org Received: from relay.hostedemail.com (smtprelay0014.hostedemail.com [216.40.44.14]) by kanga.kvack.org (Postfix) with ESMTP id 8C6B86B03E3 for ; Wed, 15 Nov 2023 20:29:22 -0500 (EST) Received: from smtpin30.hostedemail.com (a10.router.float.18 [10.200.18.1]) by unirelay04.hostedemail.com (Postfix) with ESMTP id 6CD761A017D for ; Thu, 16 Nov 2023 01:29:22 +0000 (UTC) X-FDA: 81462084564.30.F8B2ACD Received: from us-smtp-delivery-124.mimecast.com (us-smtp-delivery-124.mimecast.com [170.10.133.124]) by imf04.hostedemail.com (Postfix) with ESMTP id 63C564000A for ; Thu, 16 Nov 2023 01:29:20 +0000 (UTC) Authentication-Results: imf04.hostedemail.com; dkim=pass header.d=redhat.com header.s=mimecast20190719 header.b=MDL2fyfx; dmarc=pass (policy=none) header.from=redhat.com; spf=pass (imf04.hostedemail.com: domain of peterx@redhat.com designates 170.10.133.124 as permitted sender) smtp.mailfrom=peterx@redhat.com ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=hostedemail.com; s=arc-20220608; t=1700098160; h=from:from:sender:reply-to:subject:subject:date:date: message-id:message-id:to:to:cc:cc:mime-version:mime-version: content-type:content-type: content-transfer-encoding:content-transfer-encoding: in-reply-to:in-reply-to:references:references:dkim-signature; bh=i0M65PApL8h6gQcrG6fLgH2zAjMUA+gwn5/0n3MZLTE=; b=BqpJpTps627EHhu7lbwneQkv+KhWqDV9tIUbibeV/WWDcUOjncKnabkJMx8pxNS68JmQvS HJ2nxoXCEfg1pyul7uAEcMTnQuprCMcCm8jGlmfEHO4VvISJS/LJZ2HFi+zBtfK76LTgMD RG+yrWf53B3hX//Qa1hLWgxKGxjaWO0= ARC-Authentication-Results: i=1; imf04.hostedemail.com; dkim=pass header.d=redhat.com header.s=mimecast20190719 header.b=MDL2fyfx; dmarc=pass (policy=none) header.from=redhat.com; spf=pass (imf04.hostedemail.com: domain of peterx@redhat.com designates 170.10.133.124 as permitted sender) smtp.mailfrom=peterx@redhat.com ARC-Seal: i=1; s=arc-20220608; d=hostedemail.com; t=1700098160; a=rsa-sha256; cv=none; b=rEzHautrM5CP0DTKd+F13+259gonkk9OQSGrMI9k8MpdUyWqlBkQaPBo2+eIhrYZ18AXXZ +yECySpq64ZVX+omfMmJ5K7agFDbHfyta7K87RvCwCIboYkC2cEsdJDs3oyzf84klHuC3P GatyGT6quf93v/SQxrWxPdNA6neSPTw= DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=redhat.com; s=mimecast20190719; t=1700098159; h=from:from:reply-to:subject:subject:date:date:message-id:message-id: to:to:cc:cc:mime-version:mime-version:content-type:content-type: content-transfer-encoding:content-transfer-encoding: in-reply-to:in-reply-to:references:references; bh=i0M65PApL8h6gQcrG6fLgH2zAjMUA+gwn5/0n3MZLTE=; b=MDL2fyfxPG+pvu3smhKh3tlxnwBFbIlET5vr7Ll3chZiQr2rkIalilgHiD8C8gDdMKke+8 rrhQRYnr/gTUqAg9jU60u7CEcreI6JnwI0TUgj5Tc2bXJ7k8Mh+I/jkcZ+uBu4bGhlnplp YUx+bljIOIpXytFecX+Q3ceyl1xE7ng= Received: from mail-qt1-f199.google.com (mail-qt1-f199.google.com [209.85.160.199]) by relay.mimecast.com with ESMTP with STARTTLS (version=TLSv1.3, cipher=TLS_AES_256_GCM_SHA384) id us-mta-612-UhXfdOTROVO4J-3itagVxA-1; Wed, 15 Nov 2023 20:29:18 -0500 X-MC-Unique: UhXfdOTROVO4J-3itagVxA-1 Received: by mail-qt1-f199.google.com with SMTP id d75a77b69052e-421be23b19aso883431cf.1 for ; Wed, 15 Nov 2023 17:29:18 -0800 (PST) X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20230601; t=1700098158; x=1700702958; h=content-transfer-encoding:mime-version:references:in-reply-to :message-id:date:subject:cc:to:from:x-gm-message-state:from:to:cc :subject:date:message-id:reply-to; bh=i0M65PApL8h6gQcrG6fLgH2zAjMUA+gwn5/0n3MZLTE=; b=G9yKD+r2GfczKxRNQG2qYnrOVqVbzFma4cC+f3m4pftS+RgaJ5vYCcR+pPcjb28j9v WcdR65ZiZwRAsmXYc4zZTHnhf7DE9bX9YLmnZGxVhnK5tp6xpbgPBEzX37Y+/NGQtr1L xD/3+2AvNN/iL4GkCiDYuU9bDb9npdxuo/6Zky9U3n1MT+dqpTjTckOA5rx2Sjnbwq3+ gkPqayNPz53YPY9Ea6RHH4Y/8P9P4u2XccWqDhKXGz/g4LkJIlixu9F7EWs3aKDZNYfa cj6ZoB4guFibztS8C9D6K90hhWbEI4ffhukVGMDQM9d/DKrVFW9jc+d5cQ0nq1fYHo4p 5AFA== X-Gm-Message-State: AOJu0YxtDrGnU8Mk4crq7en8CnDukMP1OOcChd8TLhUvxKT1oU3oLp/q 2RqMlcc4CB/CcHcme4lfHSta7nxBntk9OxiadDGYteNus4aPHn9xFm8cfiqjMQJWR3z6Xxd/wvq kFTUadkY/gWw= X-Received: by 2002:ac8:4716:0:b0:421:abb7:1eac with SMTP id f22-20020ac84716000000b00421abb71eacmr7303882qtp.0.1700098157802; Wed, 15 Nov 2023 17:29:17 -0800 (PST) X-Google-Smtp-Source: AGHT+IG+ZhOFntlbofY44x0fylWHFWXMzqejypTQt/uP49OXaw97pBVx5pY9jGkMxKB+HQGuvx0qxQ== X-Received: by 2002:ac8:4716:0:b0:421:abb7:1eac with SMTP id f22-20020ac84716000000b00421abb71eacmr7303852qtp.0.1700098157463; Wed, 15 Nov 2023 17:29:17 -0800 (PST) Received: from x1n.redhat.com (cpe5c7695f3aee0-cm5c7695f3aede.cpe.net.cable.rogers.com. [99.254.144.39]) by smtp.gmail.com with ESMTPSA id c24-20020ac85198000000b0041e383d527esm3922598qtn.66.2023.11.15.17.29.15 (version=TLS1_3 cipher=TLS_AES_256_GCM_SHA384 bits=256/256); Wed, 15 Nov 2023 17:29:16 -0800 (PST) From: Peter Xu To: linux-kernel@vger.kernel.org, linux-mm@kvack.org Cc: Mike Kravetz , "Kirill A . Shutemov" , Lorenzo Stoakes , Axel Rasmussen , Matthew Wilcox , John Hubbard , Mike Rapoport , peterx@redhat.com, Hugh Dickins , David Hildenbrand , Andrea Arcangeli , Rik van Riel , James Houghton , Yang Shi , Jason Gunthorpe , Vlastimil Babka , Andrew Morton Subject: [PATCH RFC 04/12] mm: Introduce vma_pgtable_walk_{begin|end}() Date: Wed, 15 Nov 2023 20:29:00 -0500 Message-ID: <20231116012908.392077-5-peterx@redhat.com> X-Mailer: git-send-email 2.41.0 In-Reply-To: <20231116012908.392077-1-peterx@redhat.com> References: <20231116012908.392077-1-peterx@redhat.com> MIME-Version: 1.0 X-Mimecast-Spam-Score: 0 X-Mimecast-Originator: redhat.com X-Rspamd-Queue-Id: 63C564000A X-Rspam-User: X-Rspamd-Server: rspam05 X-Stat-Signature: gn9d1id5w6kxcs7cgtfkepfe8ywbsxfn X-HE-Tag: 1700098160-315803 X-HE-Meta: U2FsdGVkX19N+3VgBtlWU2tRGiDEm1y5L4uSqJ4MNeWywQnoQGBMijnJ/Wu/CMqPkkaUuqBMx9xa/X5Ej89Xid52d7d3dxotmIppkLnOIXpsbJtDPLElQULlDC42Fc8R/ThNluK9Yq530x4ULInR5wbXzh4KV0zJ4mkYYr/HDbrPaXWGlsl9NXqUCYPilzD9ulUNSCeb5kNToN5yOQ87hOSuO8nCk5Slv7lPo7nYKPU5cGeCgLv4rWl/AjpUh1G4P3GhuAYhnGdAwrWHT/RlnNjHP0KU8XxZdFOrgZrzNFLuP86CWmzUbtLdp6vu6NO4P9uPCDMznPHN4z4LY54Mi7DmMFTMz92rfEQrokx9Bpd/ECkH2Pc+0VnxLUpUwRsBNzk0qzLU0pCOCOQyC8vW0+aawByu4R3cuM8i5+4S8RnbIY6ieU16I+pxSi+U8aETOhp+f7KbRjXOHOW1CUbLxEcH0vVmgQ0yzM55xVhKCAHLrW0amB0JrRYZhn+Y75Zejn4v7I7jdNCT9BKtahOQ9UGuuHfXnUEzGCTi4+pPoRIB475TEP7Ob13LLTiuvEkD6Bp3YfwJ7kni8aYns8YNgt06/i5BAT0WtikDxOfhUneB9pkBj9vvHLwY3aRWUjTa+0OpOel14BTIHQAZ3/Ym9s3mOy1sKWHLje7Dv+2fVp/5kRpHvlVGTCY6Oeh9Jc1ndF11ZzLtUbOIDv1pJoc0tW7NyeZXW2DiLFX50A3APoSkLvimH+7k/OpOVmLA91wcpfpU+m5bGnlnYg/O/tzJjIg9crlBVtiFQGShzNa8UE55WMKFdv7UQU+TK9HYJq0sqte/pnQ4wesuB9f5Sr9CW9pGkdGBDXRRIq2XrZaBcdZqgxauBpPjkaw9Mz3ouCqDPgNCK3726KlkbbM7sWJ79WpOL+lorQuMsJi6Ql3gqBxuvvUwp5BJHIn7m+/Iw8uS1Qu1brX4XqvYcbylBzp arecgU8w 8PQ4WtKdSWE2Z1MbFyKrqYWVfR4BdHikhSNGug8Un20z+4xfudz6eeosMQjpWZlAGDZyGdYAvYjGDSj9iqyTFAI9ymSlvoIZJ+f+ygLsEXfFO3bIInp0TRxdXMb3z22nnXFlIQE+xLeRSluarChnyD55whl0h8p0leP2n2pLQno//vFvqMM6+r1L+JHi5sxiuZSuWFD4D0LMmGdhlM1a0hswY4vUcS3tTtLXjp0JGJgW1yCY4oXWHbDx1e4hdd/m4rN6zrELan+6RVuuGAG1X5hU0j4NUyhgfw0m1DPzmy4Zs13+EozxvU6ztEJgcYH8ikmqt+JS8/PrQE8BwOs6agnHtv4H7+XUYYV8SqH5NbxoROBrboFpFOjnCICabrAncr3YV21yolQSEPopfWUD3wdglrsAhNH9nZ0DiLTjwJZgwBrlhErYcGKj7fQ== X-Bogosity: Ham, tests=bogofilter, spamicity=0.000000, version=1.2.4 Sender: owner-linux-mm@kvack.org Precedence: bulk X-Loop: owner-majordomo@kvack.org List-ID: List-Subscribe: List-Unsubscribe: Introduce per-vma begin()/end() helpers for pgtable walks. This is a preparation work to merge hugetlb pgtable walkers with generic mm. The helpers need to be called before and after a pgtable walk, will start to be needed if the pgtable walker code supports hugetlb pages. It's a hook point for any type of VMA, but for now only hugetlb uses it to stablize the pgtable pages from getting away (due to possible pmd unsharing). Signed-off-by: Peter Xu Reviewed-by: Christoph Hellwig --- include/linux/mm.h | 3 +++ mm/memory.c | 12 ++++++++++++ 2 files changed, 15 insertions(+) diff --git a/include/linux/mm.h b/include/linux/mm.h index 64cd1ee4aacc..349232dd20fb 100644 --- a/include/linux/mm.h +++ b/include/linux/mm.h @@ -4154,4 +4154,7 @@ static inline bool pfn_is_unaccepted_memory(unsigned long pfn) return range_contains_unaccepted_memory(paddr, paddr + PAGE_SIZE); } +void vma_pgtable_walk_begin(struct vm_area_struct *vma); +void vma_pgtable_walk_end(struct vm_area_struct *vma); + #endif /* _LINUX_MM_H */ diff --git a/mm/memory.c b/mm/memory.c index e27e2e5beb3f..3a6434b40d87 100644 --- a/mm/memory.c +++ b/mm/memory.c @@ -6180,3 +6180,15 @@ void ptlock_free(struct ptdesc *ptdesc) kmem_cache_free(page_ptl_cachep, ptdesc->ptl); } #endif + +void vma_pgtable_walk_begin(struct vm_area_struct *vma) +{ + if (is_vm_hugetlb_page(vma)) + hugetlb_vma_lock_read(vma); +} + +void vma_pgtable_walk_end(struct vm_area_struct *vma) +{ + if (is_vm_hugetlb_page(vma)) + hugetlb_vma_unlock_read(vma); +} From patchwork Thu Nov 16 01:29:01 2023 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Peter Xu X-Patchwork-Id: 13457480 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from kanga.kvack.org (kanga.kvack.org [205.233.56.17]) by smtp.lore.kernel.org (Postfix) with ESMTP id B2C6FC2BB3F for ; Thu, 16 Nov 2023 01:29:28 +0000 (UTC) Received: by kanga.kvack.org (Postfix) id 48ED56B03E5; Wed, 15 Nov 2023 20:29:24 -0500 (EST) Received: by kanga.kvack.org (Postfix, from userid 40) id 418EF6B03E7; Wed, 15 Nov 2023 20:29:24 -0500 (EST) X-Delivered-To: int-list-linux-mm@kvack.org Received: by kanga.kvack.org (Postfix, from userid 63042) id 26BA36B03E8; Wed, 15 Nov 2023 20:29:24 -0500 (EST) X-Delivered-To: linux-mm@kvack.org Received: from relay.hostedemail.com (smtprelay0014.hostedemail.com [216.40.44.14]) by kanga.kvack.org (Postfix) with ESMTP id 0F6BD6B03E5 for ; Wed, 15 Nov 2023 20:29:24 -0500 (EST) Received: from smtpin22.hostedemail.com (a10.router.float.18 [10.200.18.1]) by unirelay02.hostedemail.com (Postfix) with ESMTP id D6FCF12041C for ; Thu, 16 Nov 2023 01:29:23 +0000 (UTC) X-FDA: 81462084606.22.C1F54B8 Received: from us-smtp-delivery-124.mimecast.com (us-smtp-delivery-124.mimecast.com [170.10.129.124]) by imf15.hostedemail.com (Postfix) with ESMTP id BCF1CA0011 for ; Thu, 16 Nov 2023 01:29:21 +0000 (UTC) Authentication-Results: imf15.hostedemail.com; dkim=pass header.d=redhat.com header.s=mimecast20190719 header.b=aaMlRdgz; dmarc=pass (policy=none) header.from=redhat.com; spf=pass (imf15.hostedemail.com: domain of peterx@redhat.com designates 170.10.129.124 as permitted sender) smtp.mailfrom=peterx@redhat.com ARC-Seal: i=1; s=arc-20220608; d=hostedemail.com; t=1700098161; a=rsa-sha256; cv=none; b=7hdLAPbqYKCKqdaLV0Pe7XH3FyS0qpqbIufyW5Rk7YlGjH5sd+yvzmEXrECFRBRQHQYzAG UDF5plrCVBJOIi8+ZALobhU6Ci8W2Gvznx4YfciXC4xQ2UKiInq2EYDinDR8t5eSgzD6S6 ZzhscZfNAumo8njTj7UWoWsP5ADm6ko= ARC-Authentication-Results: i=1; imf15.hostedemail.com; dkim=pass header.d=redhat.com header.s=mimecast20190719 header.b=aaMlRdgz; dmarc=pass (policy=none) header.from=redhat.com; spf=pass (imf15.hostedemail.com: domain of peterx@redhat.com designates 170.10.129.124 as permitted sender) smtp.mailfrom=peterx@redhat.com ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=hostedemail.com; s=arc-20220608; t=1700098161; h=from:from:sender:reply-to:subject:subject:date:date: message-id:message-id:to:to:cc:cc:mime-version:mime-version: content-type:content-type: content-transfer-encoding:content-transfer-encoding: in-reply-to:in-reply-to:references:references:dkim-signature; bh=iVPh5vUSg15R+EjRvyUTGLnvTLGftOOyrePNaUJdD/U=; b=nwg6e/M01+LSBcHR6e8Q9wUpDnwppetpjz48ZIrMqXyliWMebVwnlNbOrAZ98oGhxuO2bJ FXtcKTOabgiHwYQj7ngZifFMIRunlh9U5HYbKHKRqiXZzZwg4dCvNEK/igeThieMI6VnWT S8mD493+BRXsmqKNKuiTzPPclERwCMc= DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=redhat.com; s=mimecast20190719; t=1700098161; h=from:from:reply-to:subject:subject:date:date:message-id:message-id: to:to:cc:cc:mime-version:mime-version:content-type:content-type: content-transfer-encoding:content-transfer-encoding: in-reply-to:in-reply-to:references:references; bh=iVPh5vUSg15R+EjRvyUTGLnvTLGftOOyrePNaUJdD/U=; b=aaMlRdgzjqAA+phZ9tlYXBcK2YG/2nG3Fuf9K895ifmdXGs4k7aH1nv/zT9ZzQZzRfwjdm +iV4sSQEAg1FlYBJpu3RBZ5eyZfI5LaefZJK71N5IT/AFjRtny4K3bfgUP1kD+2VW3zbdk KOwOpBL+RfQ0qDGB7ZBJH5B3dZOa1h8= Received: from mail-qt1-f197.google.com (mail-qt1-f197.google.com [209.85.160.197]) by relay.mimecast.com with ESMTP with STARTTLS (version=TLSv1.3, cipher=TLS_AES_256_GCM_SHA384) id us-mta-689-iGh0QhGVOaCGhy8CXKvh5Q-1; Wed, 15 Nov 2023 20:29:19 -0500 X-MC-Unique: iGh0QhGVOaCGhy8CXKvh5Q-1 Received: by mail-qt1-f197.google.com with SMTP id d75a77b69052e-41eb42115e9so899281cf.1 for ; Wed, 15 Nov 2023 17:29:19 -0800 (PST) X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20230601; t=1700098159; x=1700702959; h=content-transfer-encoding:mime-version:references:in-reply-to :message-id:date:subject:cc:to:from:x-gm-message-state:from:to:cc :subject:date:message-id:reply-to; bh=iVPh5vUSg15R+EjRvyUTGLnvTLGftOOyrePNaUJdD/U=; b=P1c1tRJ+men8QgtmxN7moj0uYy94nyP0VdkqxyjACiRr5S6DlPnz8Pmzv0JsjOqW+m s/r8mzUIorbpCJairoCPY9F1i/1Rdx4T/cubqcf+IbW1NNS2VP7mkmOKBaUn12HOw2k1 Xje7pdh58JvAaMjEFgItS7SrTlVsDyLjDzp6q3NW0n+aQZRwL/zhDnOuy5I6AShmXyo2 YmFWdPPHtNDk7o6n6JVJOYYeEyi6OYo0TzbRnL4N/j1aWSAiRQedb/jTfVfqsMiqO9Jk 6lT3eYwzX//isuyUGW7B632tpR9u2NBe4kEyMD2dd4zzPbYh2yKkegfT9EWIJMrBAouL TVDA== X-Gm-Message-State: AOJu0Yy239YObwbuPKu5Gcef2YJGVDeFxhk17kjTduGC3xc3Dyt5HMnZ vHCdD7UtTEpDZNIIm7CdRPXQFpdEVSLNB7+iOyUisEyNkjWazEWqRqJ0hf8roeZpjCYDvhVnLxo s6X2NlfqJdJY= X-Received: by 2002:ac8:4716:0:b0:421:b9be:d4a4 with SMTP id f22-20020ac84716000000b00421b9bed4a4mr6891657qtp.0.1700098159444; Wed, 15 Nov 2023 17:29:19 -0800 (PST) X-Google-Smtp-Source: AGHT+IGJxMhrfIQJHLleHfYWSj3v7NIOAggKaUbuvyUHKkhcdknXN7d9JWqJ2v9ElBIqwX+6WGc0WQ== X-Received: by 2002:ac8:4716:0:b0:421:b9be:d4a4 with SMTP id f22-20020ac84716000000b00421b9bed4a4mr6891630qtp.0.1700098159208; Wed, 15 Nov 2023 17:29:19 -0800 (PST) Received: from x1n.redhat.com (cpe5c7695f3aee0-cm5c7695f3aede.cpe.net.cable.rogers.com. [99.254.144.39]) by smtp.gmail.com with ESMTPSA id c24-20020ac85198000000b0041e383d527esm3922598qtn.66.2023.11.15.17.29.17 (version=TLS1_3 cipher=TLS_AES_256_GCM_SHA384 bits=256/256); Wed, 15 Nov 2023 17:29:18 -0800 (PST) From: Peter Xu To: linux-kernel@vger.kernel.org, linux-mm@kvack.org Cc: Mike Kravetz , "Kirill A . Shutemov" , Lorenzo Stoakes , Axel Rasmussen , Matthew Wilcox , John Hubbard , Mike Rapoport , peterx@redhat.com, Hugh Dickins , David Hildenbrand , Andrea Arcangeli , Rik van Riel , James Houghton , Yang Shi , Jason Gunthorpe , Vlastimil Babka , Andrew Morton Subject: [PATCH RFC 05/12] mm/gup: Fix follow_devmap_p[mu]d() to return even if NULL Date: Wed, 15 Nov 2023 20:29:01 -0500 Message-ID: <20231116012908.392077-6-peterx@redhat.com> X-Mailer: git-send-email 2.41.0 In-Reply-To: <20231116012908.392077-1-peterx@redhat.com> References: <20231116012908.392077-1-peterx@redhat.com> MIME-Version: 1.0 X-Mimecast-Spam-Score: 0 X-Mimecast-Originator: redhat.com X-Rspam-User: X-Rspamd-Server: rspam06 X-Rspamd-Queue-Id: BCF1CA0011 X-Stat-Signature: 6b9wj3qb4csomas3tmdkqjnnyhboca5d X-HE-Tag: 1700098161-817415 X-HE-Meta: U2FsdGVkX1/FRtLs/KjXJVgSs7AfCppbbaN85pn16FlJN1hLONc0ijNPMunfQLSt1MFlIYSq6XZUyNTnxJckhIYxbBvLpkCFvS/VT3L4zzCEXXXwofA4nKugGR2Vq9pOqA1JrnkyCImcl+VSlIKeP5tM/IytTefaT/htNhoZeh7A9jkbygBKiTo5Li/Hik25RHVwcvShUnB4NkYZSUBPXC3eLZvxEzaAdklJEE/KPSfwaAVxI7ZRRm1SIbhG4FTxNyQMWCiNmT4+ZdgkK3xzn48ss6ekRGi319Z6gHRjoSVWHEy+Fkv/3URRYkcCaDjt0IQIiJasY2qORBg7m+7mn9HnH5sXWkPUdpzui26KSqcpi8KFhlffA6A1H84Kg8JEsJ1nvat1HkP8bsnCVJjTz+xU7WyOlEsgeqtVnV2FcTSobkmQLvg3tMlhI9INPVCbUvJQQ2dMM2l/U5ENtDCdJDvEVtRz1dHZjUbjBduw0QyyoTzNNoLN1d2dE+/GaiXqyUkQQoq5hQg0lKyKdzqV/xbAQ1RPj/OmMXAtzB5+Y5bvxXtFgFXNaWkdM4FEhcdNap5sP1a4gkzi9kBBWkUC/TprumYvXyAfaHKlu6yE7j9FQ7kjDIKIAXZcWVRFc9G5h+0K31WoUWjRcxzm3AhySDHzYKNWnlDrz+2HOFIhuNeiYzlgo/kZg1RAexuWq+34TVZ9GKdxsO0Qbl46ZyZ6biC3Qm4zZ8sX3Xgz3gmCkx9gtRvR5uP937VTbsyU7KTU67E83UxMo9UC280Zl+NflKSUwuFmpXS8dYQeh37ERQ8SZ7nlGLRhDRu1nXH5u3rCWH+suUC9pLK6VG5f2Apb3AAQu+hLswiNRfQF186VOl+o68JwmCTWKMN185Zr2pm+hqu0BAzDjojaYcwKwIYEJzG5LkLoP3QdcD7j2l0qYviWEA5340K7h8XdeaktvqLyw72vTgm7rC+60pItkE/ lD/joUM3 qBCCJMpHPs2aqP3nhfph/D6SriAlmDql4FjBg6blGKIvDayqm9IDVq1KssqRooKlxa86fuSWluNxwdb0dk+zDir4o5waOHtQeuEZBovjZMLNwlFCKQTHskOTl9r3P+lhkShY7S/CNsg1KdABM4IX/H/gEWg1oygw5on9qMIPjgY5y+QaqPN/IkgiVsrFBpYLZiQ+wzqdFCss7lJDzVzPfTOz9znHyYE9iV27QYgcTEJm2IJfa7xip0pAd239eDf42G7eBE/ORlwIGIO5lIpWLwyMx4rJtaMosjtVkn3eP38lNl8Q+DjU+fkAK57V+LNvDRZMSsMfgnjEIK51i6o0Fgs/pcZ2cjWcJbbKVf9VpnKSDghTMcN0LRERwEND1ug06rSBvI2/0xhWfVUzngCYvI//ztGRZ1GLnPcCeIWdi0sfVdJxVxjEpvAeuLg== X-Bogosity: Ham, tests=bogofilter, spamicity=0.000000, version=1.2.4 Sender: owner-linux-mm@kvack.org Precedence: bulk X-Loop: owner-majordomo@kvack.org List-ID: List-Subscribe: List-Unsubscribe: This seems to be a bug not by any report but by code observations. When GUP sees a devpmd or devpud, it should return whatever value returned from follow_devmap_p[mu]d(). If page==NULL returned, it means a fault is probably required. Skipping return the NULL should allow the code to fall through, which can cause unexpected behavior. It was there at least before the follow page rework (080dbb618b) in 2017, so 6 years. Not yet digging for a Fixes, assuming it can hardly trigger even if the logical bug does exist. Signed-off-by: Peter Xu Reviewed-by: Christoph Hellwig --- mm/gup.c | 6 ++---- 1 file changed, 2 insertions(+), 4 deletions(-) diff --git a/mm/gup.c b/mm/gup.c index a8b73a8289ad..0e00204761d2 100644 --- a/mm/gup.c +++ b/mm/gup.c @@ -708,8 +708,7 @@ static struct page *follow_pmd_mask(struct vm_area_struct *vma, ptl = pmd_lock(mm, pmd); page = follow_devmap_pmd(vma, address, pmd, flags, &ctx->pgmap); spin_unlock(ptl); - if (page) - return page; + return page; } if (likely(!pmd_trans_huge(pmdval))) return follow_page_pte(vma, address, pmd, flags, &ctx->pgmap); @@ -756,8 +755,7 @@ static struct page *follow_pud_mask(struct vm_area_struct *vma, ptl = pud_lock(mm, pud); page = follow_devmap_pud(vma, address, pud, flags, &ctx->pgmap); spin_unlock(ptl); - if (page) - return page; + return page; } if (unlikely(pud_bad(*pud))) return no_page_table(vma, flags); From patchwork Thu Nov 16 01:29:02 2023 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Peter Xu X-Patchwork-Id: 13457481 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from kanga.kvack.org (kanga.kvack.org [205.233.56.17]) by smtp.lore.kernel.org (Postfix) with ESMTP id 34ECFC2BB3F for ; Thu, 16 Nov 2023 01:29:31 +0000 (UTC) Received: by kanga.kvack.org (Postfix) id D6C3A6B03E7; Wed, 15 Nov 2023 20:29:25 -0500 (EST) Received: by kanga.kvack.org (Postfix, from userid 40) id CF33D6B03E9; Wed, 15 Nov 2023 20:29:25 -0500 (EST) X-Delivered-To: int-list-linux-mm@kvack.org Received: by kanga.kvack.org (Postfix, from userid 63042) id B43356B03EA; Wed, 15 Nov 2023 20:29:25 -0500 (EST) X-Delivered-To: linux-mm@kvack.org Received: from relay.hostedemail.com (smtprelay0014.hostedemail.com [216.40.44.14]) by kanga.kvack.org (Postfix) with ESMTP id 9F5366B03E7 for ; Wed, 15 Nov 2023 20:29:25 -0500 (EST) Received: from smtpin14.hostedemail.com (a10.router.float.18 [10.200.18.1]) by unirelay01.hostedemail.com (Postfix) with ESMTP id 78AC01CAD62 for ; Thu, 16 Nov 2023 01:29:25 +0000 (UTC) X-FDA: 81462084690.14.41A20ED Received: from us-smtp-delivery-124.mimecast.com (us-smtp-delivery-124.mimecast.com [170.10.133.124]) by imf05.hostedemail.com (Postfix) with ESMTP id 6810610000F for ; Thu, 16 Nov 2023 01:29:23 +0000 (UTC) Authentication-Results: imf05.hostedemail.com; dkim=pass header.d=redhat.com header.s=mimecast20190719 header.b=ISpriMu8; dmarc=pass (policy=none) header.from=redhat.com; spf=pass (imf05.hostedemail.com: domain of peterx@redhat.com designates 170.10.133.124 as permitted sender) smtp.mailfrom=peterx@redhat.com ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=hostedemail.com; s=arc-20220608; t=1700098163; h=from:from:sender:reply-to:subject:subject:date:date: message-id:message-id:to:to:cc:cc:mime-version:mime-version: content-type:content-type: content-transfer-encoding:content-transfer-encoding: in-reply-to:in-reply-to:references:references:dkim-signature; bh=bUKDadnp+1aD4MuuhezpyppaSOr8lCSuMGXbTAXddpQ=; b=gi+c4EXzHvIzMbw8p9PCuP+rh4KEqlg3iA5tHBP8H8o99GNIRIGvcV5FfdL1TSIr4VZ80R 06LLDzeuCQGqr7km/nihMN6TaMRG8aBKbmr2quu2zy47EQ+y9mvU2OXXPQzJZdkb/Tv6MS qv3FpoiY+/SKAf2hWByQvtc+igbebtQ= ARC-Authentication-Results: i=1; imf05.hostedemail.com; dkim=pass header.d=redhat.com header.s=mimecast20190719 header.b=ISpriMu8; dmarc=pass (policy=none) header.from=redhat.com; spf=pass (imf05.hostedemail.com: domain of peterx@redhat.com designates 170.10.133.124 as permitted sender) smtp.mailfrom=peterx@redhat.com ARC-Seal: i=1; s=arc-20220608; d=hostedemail.com; t=1700098163; a=rsa-sha256; cv=none; b=0H8KpnN0Oqcllc2IF5TtRSd1koxoBjN5L84Ll+Uagd276v5JzvqNtZLhquybzHqwdev4fl rOfW5PB0WnB2tlknKD3bkdGo/2zCqRddOPY3yvSf6vmnJQC+16ylt6m6ce3442KXVQZUwW 1Dg3OEg2T4QwwP7wneJVzMunV7syv7c= DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=redhat.com; s=mimecast20190719; t=1700098162; h=from:from:reply-to:subject:subject:date:date:message-id:message-id: to:to:cc:cc:mime-version:mime-version:content-type:content-type: content-transfer-encoding:content-transfer-encoding: in-reply-to:in-reply-to:references:references; bh=bUKDadnp+1aD4MuuhezpyppaSOr8lCSuMGXbTAXddpQ=; b=ISpriMu8yGjWIlMf/FTN2qDnDbzOacO8LwZIbhWXXo58Hnn9Nc6gSZMcWhOBS1wIao8ku0 bnVByu6CqmwzDz+epiELqETzjevXFEhH82E9kupK1KMpBGzkQ1nKxo39GZC+s1Z7F8zxXD kTjF8XI+hdybutIHwveXzVtK8Kpm0Hg= Received: from mail-qt1-f200.google.com (mail-qt1-f200.google.com [209.85.160.200]) by relay.mimecast.com with ESMTP with STARTTLS (version=TLSv1.3, cipher=TLS_AES_256_GCM_SHA384) id us-mta-182-5m6WC-yBNEiffqFXqkbbPA-1; Wed, 15 Nov 2023 20:29:21 -0500 X-MC-Unique: 5m6WC-yBNEiffqFXqkbbPA-1 Received: by mail-qt1-f200.google.com with SMTP id d75a77b69052e-41eb42115e9so899461cf.1 for ; Wed, 15 Nov 2023 17:29:21 -0800 (PST) X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20230601; t=1700098161; x=1700702961; h=content-transfer-encoding:mime-version:references:in-reply-to :message-id:date:subject:cc:to:from:x-gm-message-state:from:to:cc :subject:date:message-id:reply-to; bh=bUKDadnp+1aD4MuuhezpyppaSOr8lCSuMGXbTAXddpQ=; b=Zun1N2iSTRyQMMtvFOJtMWvVITYwhGAZ1069npVDthJeSHhxkNV/+KYYe+OyeWpl+4 mwqZrwTpHWTgEud8AW2yKzlm8XMnkLEzezIvKxZwdphkgkajX2cib2OqhEeD8XjxqqHW JyXmymC2/JWX376rST4SS2XsQ9ZF7YAAPdE84cCTO4jKrmNaAApR/w9VUS7JG0dN/arO 9tS9ZDmIxIU7pmSVCHFaT/2EjZCfIqraW95v8P8Y6jK2gQOR+xVn6wrfff1hoOEUsD1q ElRMIpFG686pUBbtwyljf0Yl6fD7rXC7Rb5W2VEoJIl1dNdKHfEh5XujclDWExBdzytL onIg== X-Gm-Message-State: AOJu0Yxvaa8mNVzyskyh8kWW2ZBif+GbOUBl0TP464HUgGfFLe+PJso8 9xaaZTynMkxYRLFv5tr+S03Q6o+vh7zoI9RQ3eolV6ACBdtG8JEVQ4QDlCpTW9qeU1Yb+O5LXUa Iej0Zyo5b72w= X-Received: by 2002:a05:622a:810e:b0:41c:d433:6c86 with SMTP id jx14-20020a05622a810e00b0041cd4336c86mr6482552qtb.4.1700098160985; Wed, 15 Nov 2023 17:29:20 -0800 (PST) X-Google-Smtp-Source: AGHT+IEBd1/VL7OzaVonjYuLiYORLZv0OSCh6ZT9Z6PevvXUdWDeqJg/IL5dpgLaUCzFthgUgScdWA== X-Received: by 2002:a05:622a:810e:b0:41c:d433:6c86 with SMTP id jx14-20020a05622a810e00b0041cd4336c86mr6482544qtb.4.1700098160768; Wed, 15 Nov 2023 17:29:20 -0800 (PST) Received: from x1n.redhat.com (cpe5c7695f3aee0-cm5c7695f3aede.cpe.net.cable.rogers.com. [99.254.144.39]) by smtp.gmail.com with ESMTPSA id c24-20020ac85198000000b0041e383d527esm3922598qtn.66.2023.11.15.17.29.19 (version=TLS1_3 cipher=TLS_AES_256_GCM_SHA384 bits=256/256); Wed, 15 Nov 2023 17:29:20 -0800 (PST) From: Peter Xu To: linux-kernel@vger.kernel.org, linux-mm@kvack.org Cc: Mike Kravetz , "Kirill A . Shutemov" , Lorenzo Stoakes , Axel Rasmussen , Matthew Wilcox , John Hubbard , Mike Rapoport , peterx@redhat.com, Hugh Dickins , David Hildenbrand , Andrea Arcangeli , Rik van Riel , James Houghton , Yang Shi , Jason Gunthorpe , Vlastimil Babka , Andrew Morton , Michael Ellerman , linuxppc-dev@lists.ozlabs.org Subject: [PATCH RFC 06/12] mm/gup: Drop folio_fast_pin_allowed() in hugepd processing Date: Wed, 15 Nov 2023 20:29:02 -0500 Message-ID: <20231116012908.392077-7-peterx@redhat.com> X-Mailer: git-send-email 2.41.0 In-Reply-To: <20231116012908.392077-1-peterx@redhat.com> References: <20231116012908.392077-1-peterx@redhat.com> MIME-Version: 1.0 X-Mimecast-Spam-Score: 0 X-Mimecast-Originator: redhat.com X-Rspamd-Queue-Id: 6810610000F X-Rspam-User: X-Rspamd-Server: rspam04 X-Stat-Signature: 6e6jgwjhkbqc4ty8m7a65373uccwjuxq X-HE-Tag: 1700098163-887940 X-HE-Meta: U2FsdGVkX1/I9zIdcLxtq92ShYpu65BckFivakyW9XuZiZ2ZhbwGXQ5OgnuP7I85EKawxVybpNf7WOXE21FweHfbfbSw2bN7vNCbb/oH93WvkfyGthrI72lpx7vm1uzFGFPKiGQnwrSglXvljkrkEYxIhUReDiUf1LNK+IM7MwkJSitEvSQTrjq78pcekb3Z+nunkGvNR6HLgzo+gl2tV9FkF9UiGECFEo5t+ublLIKz0bT6tefb/LnpOAljg1w6iKEMeeShod6rbZ+F8QFog9qJFhZUPa1/t9QDAMr1whXiWPcv1bMgN3tSmWIn02/bUvT0qlLb5cjQ9hQBz3ia8br85hnC+Vnb4Kkir2HcZPO/YhoOBdKqx0tuWjo2nz84X3VRjnERgsyLg6jpwK5EY7WDK+RfmvRgBxQzN4w/b5oCAq+yer0IJuPzc06H8vXOjbWCIhJr1mekb2HX5lkAa8eECwM80NNGGlLgDFmAXO+ewpCyu53VFkLSJtz7oUvTxgBfwXYVEBtyj8r4ngYMPTOKgl4Bxe/hfngKZdwX6rTlRtHYubRW7McJDxDRpdw0HthzdGNomI5n/wZlS46W+WFOeag8WOH9ypRWJOAhiXPcuPnenxnI8NDX9C2rr84DTawDzQP54Wqcd8/kh2rpA8ob1pMfjSzAhn/hOuUx4BaMPruIjraWvAe067OMY50VApdWDpAaTFHBkeAR66GPhjzifdKEHNUbX25JCtQ//MTQBsqZxnzVIKBLTcyiVEuYUJgDDSH05KpE1yXojRfXgE5XP3Z3ZQa+niIjv+/0zdEHABwRePKZUJyhbAAr3AmzYs84awg10vDRc3OUnvWxOshoL5VMtWYpyoHZeOEbr57gSTtC545I1weT7X/OVEnrt3oYsxWylvcwT/VA3Z9j/+uLL9gSn8mQepxZuAg+PPw1Q6Qeu4ISJ1Ms8M2FPFnJ07uHrDbBVwwq01GzgFO dp+jtn2c N4m7L2C73KjPTlcmPHaNN/XN3nZwZwTzGBVFy+Q2PAM+n8u+nVUWmgOvu8kpiXsEJjwxtNlzHwpzzDaRLFowd+NDEhYxtP+qIIJxXorNZnofscrsUDKpVA+4Id6WMS9LwKO1ePZeJKBO9I9bOEPtMbRW38F+cgoeVD6EgiGwbryGldkcMJPPFmg6a9u34b6VVm24ck1c+pCjLQXzbJTe1q4swW38q8vWvoIsyGuaX1dmNkiSVJhLxhnf/KEAzMAhOGF/qea7BaeuD7sodwDGjY9gKI+indY4/YGM0i1rSsjR5IdLkoDVRyYkfQMlN2kH1Tf2RK9Wr2LZMq6mCedxBjvVIDxVOhss1x+3G/44lQlaZaTN1gegiQuy0um4ArBQi0gAF6jXaSkQlZLMM2er4Qw3xxQQudX3vu/LXSXKHfbq6BxSIRobV/dwl/2ldAvWSvpWKAERuuNSzf17A2LRih4uWWx11NAmmzGIRFMRMbHNTJX5ejbpMFnWJ++2uFoFyrOnd5uM3IRYgCOA= X-Bogosity: Ham, tests=bogofilter, spamicity=0.000000, version=1.2.4 Sender: owner-linux-mm@kvack.org Precedence: bulk X-Loop: owner-majordomo@kvack.org List-ID: List-Subscribe: List-Unsubscribe: Hugepd format is only used in PowerPC with hugetlbfs. In commit a6e79df92e4a ("mm/gup: disallow FOLL_LONGTERM GUP-fast writing to file-backed mappings"), we added a check to fail gup-fast if there's potential risk of violating GUP over writeback file systems. That should never apply to hugepd. Drop that check, not only because it'll never be true for hugepd, but also it paves way for reusing the function outside fast-gup. Cc: Lorenzo Stoakes Cc: Michael Ellerman Cc: linuxppc-dev@lists.ozlabs.org Signed-off-by: Peter Xu --- mm/gup.c | 5 ----- 1 file changed, 5 deletions(-) diff --git a/mm/gup.c b/mm/gup.c index 0e00204761d2..424d45e1afb3 100644 --- a/mm/gup.c +++ b/mm/gup.c @@ -2816,11 +2816,6 @@ static int gup_hugepte(pte_t *ptep, unsigned long sz, unsigned long addr, return 0; } - if (!folio_fast_pin_allowed(folio, flags)) { - gup_put_folio(folio, refs, flags); - return 0; - } - if (!pte_write(pte) && gup_must_unshare(NULL, flags, &folio->page)) { gup_put_folio(folio, refs, flags); return 0; From patchwork Thu Nov 16 01:29:03 2023 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Peter Xu X-Patchwork-Id: 13457485 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from kanga.kvack.org (kanga.kvack.org [205.233.56.17]) by smtp.lore.kernel.org (Postfix) with ESMTP id F0754C072A2 for ; Thu, 16 Nov 2023 01:29:41 +0000 (UTC) Received: by kanga.kvack.org (Postfix) id 998FB6B03F0; Wed, 15 Nov 2023 20:29:33 -0500 (EST) Received: by kanga.kvack.org (Postfix, from userid 40) id 9215A6B03F1; Wed, 15 Nov 2023 20:29:33 -0500 (EST) X-Delivered-To: int-list-linux-mm@kvack.org Received: by kanga.kvack.org (Postfix, from userid 63042) id 778036B03F2; Wed, 15 Nov 2023 20:29:33 -0500 (EST) X-Delivered-To: linux-mm@kvack.org Received: from relay.hostedemail.com (smtprelay0016.hostedemail.com [216.40.44.16]) by kanga.kvack.org (Postfix) with ESMTP id 622B46B03F0 for ; Wed, 15 Nov 2023 20:29:33 -0500 (EST) Received: from smtpin06.hostedemail.com (a10.router.float.18 [10.200.18.1]) by unirelay10.hostedemail.com (Postfix) with ESMTP id 25F04C01A6 for ; Thu, 16 Nov 2023 01:29:33 +0000 (UTC) X-FDA: 81462085026.06.642552D Received: from us-smtp-delivery-124.mimecast.com (us-smtp-delivery-124.mimecast.com [170.10.133.124]) by imf23.hostedemail.com (Postfix) with ESMTP id 1A18A14000F for ; Thu, 16 Nov 2023 01:29:28 +0000 (UTC) Authentication-Results: imf23.hostedemail.com; dkim=pass header.d=redhat.com header.s=mimecast20190719 header.b=AaZXdQAj; dmarc=pass (policy=none) header.from=redhat.com; spf=pass (imf23.hostedemail.com: domain of peterx@redhat.com designates 170.10.133.124 as permitted sender) smtp.mailfrom=peterx@redhat.com ARC-Seal: i=1; s=arc-20220608; d=hostedemail.com; t=1700098171; a=rsa-sha256; cv=none; b=WlAhYWB3NDAM6T4eGR7FyStPpQdhDoUBTbJwL46d0dkBQElVSzAYlDLB9z8vAi61EjyGAD QQoQfdPU1BCHmsLocJTv7eKzaPCUC076g+ouVhM9YuMAoWVddcaHfP6tlXjlV7BNSiz8/8 ShwwFOfOHPExWfnUhzW5Tw3mzN/+l14= ARC-Authentication-Results: i=1; imf23.hostedemail.com; dkim=pass header.d=redhat.com header.s=mimecast20190719 header.b=AaZXdQAj; dmarc=pass (policy=none) header.from=redhat.com; spf=pass (imf23.hostedemail.com: domain of peterx@redhat.com designates 170.10.133.124 as permitted sender) smtp.mailfrom=peterx@redhat.com ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=hostedemail.com; s=arc-20220608; t=1700098171; h=from:from:sender:reply-to:subject:subject:date:date: message-id:message-id:to:to:cc:cc:mime-version:mime-version: content-type:content-type: content-transfer-encoding:content-transfer-encoding: in-reply-to:in-reply-to:references:references:dkim-signature; bh=/m4Nmkq0cr3qM8eU/2c8RM1XZmbD+yMVfcEDigP2nYM=; b=dchGU9j0wwI46E4UX2SxCM5pXWDeuEiYVFjJADAoPPGzonc0Di7uKnu9PxjnJdocN5wXCn +FXsIidWyvPx9cP3QSrH4qeBkrLLHMHjZ24HQRPK/dKBjo67Nm819HmNOgwAUlljWapWdc 3f63QGZE6+Bf9gXxL1sz4E9CWM62KyU= DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=redhat.com; s=mimecast20190719; t=1700098167; h=from:from:reply-to:subject:subject:date:date:message-id:message-id: to:to:cc:cc:mime-version:mime-version:content-type:content-type: content-transfer-encoding:content-transfer-encoding: in-reply-to:in-reply-to:references:references; bh=/m4Nmkq0cr3qM8eU/2c8RM1XZmbD+yMVfcEDigP2nYM=; b=AaZXdQAjUiZsWmiDJsnE2gWSIvItsTcUG0/jF35lGa55OU8f6z0ukCzb1K4ECWvQ9o3NRo nRZQfgVhl5Bu7PAr/U0zfNUkeV6+U0hnDqCMAwi4OokNOfEaa1uIbLy1cO9cFxRXWTtSND OvWJhy9Ld8btIKfVNugE+jQa81ZuM04= Received: from mail-qt1-f198.google.com (mail-qt1-f198.google.com [209.85.160.198]) by relay.mimecast.com with ESMTP with STARTTLS (version=TLSv1.3, cipher=TLS_AES_256_GCM_SHA384) id us-mta-330-egE_FXqrPhW-05S524YTPQ-1; Wed, 15 Nov 2023 20:29:24 -0500 X-MC-Unique: egE_FXqrPhW-05S524YTPQ-1 Received: by mail-qt1-f198.google.com with SMTP id d75a77b69052e-420c7034fddso915871cf.0 for ; Wed, 15 Nov 2023 17:29:24 -0800 (PST) X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20230601; t=1700098163; x=1700702963; h=content-transfer-encoding:mime-version:references:in-reply-to :message-id:date:subject:cc:to:from:x-gm-message-state:from:to:cc :subject:date:message-id:reply-to; bh=/m4Nmkq0cr3qM8eU/2c8RM1XZmbD+yMVfcEDigP2nYM=; b=qOWbLe66429WIejW0KaK8Y0UTCKln3vBQDaM8XgICRUgbJI6f7o7Y4ktnVDbzMsUYb 43wo7x2dzlQH6PlncWdi7/0bXRWzLq9Bf+9t9DQKBAE5LKcW1urxbMAwpd8OAcD6EAbn vr89oQVAd7BIF+b7aHaGWKBClNLIselTY9DssoIJyoD4lborj8ODnNphsiu9sI8hIJEj NDTbjoUYP46CKMIEAyzf89SFGmld9yHNp4Xy/VJz9YNVhULVGBC+v209d4K6TbEY/YHx TK2EAv4JS1Yw4YPopSKk0UH7ySgOCYw3+oidRR27XIKqifytkhCo96M8UfxInAwaWHvM RECA== X-Gm-Message-State: AOJu0Yww7pzg1+5u8eFYDZWB9U6RSc7s9o8zLH7x+nEWvzIAPe1617iv +MXmHKZBV4OAW8nyrNVeGzljTUOCuZhXTuy2HpHheXghcP/56GI277lqzhngqD3eH1zyXm2yROM 202tvtQS0M2M= X-Received: by 2002:ac8:1496:0:b0:421:9fe5:6dc1 with SMTP id l22-20020ac81496000000b004219fe56dc1mr6484666qtj.3.1700098162895; Wed, 15 Nov 2023 17:29:22 -0800 (PST) X-Google-Smtp-Source: AGHT+IFEPgMVSzfIxtQlfIF+wS0OHZ+NUQWxFmPrijPw2JerykvyjqCin/P2xZqUrm2jMmgcDcZfrw== X-Received: by 2002:ac8:1496:0:b0:421:9fe5:6dc1 with SMTP id l22-20020ac81496000000b004219fe56dc1mr6484654qtj.3.1700098162561; Wed, 15 Nov 2023 17:29:22 -0800 (PST) Received: from x1n.redhat.com (cpe5c7695f3aee0-cm5c7695f3aede.cpe.net.cable.rogers.com. [99.254.144.39]) by smtp.gmail.com with ESMTPSA id c24-20020ac85198000000b0041e383d527esm3922598qtn.66.2023.11.15.17.29.20 (version=TLS1_3 cipher=TLS_AES_256_GCM_SHA384 bits=256/256); Wed, 15 Nov 2023 17:29:22 -0800 (PST) From: Peter Xu To: linux-kernel@vger.kernel.org, linux-mm@kvack.org Cc: Mike Kravetz , "Kirill A . Shutemov" , Lorenzo Stoakes , Axel Rasmussen , Matthew Wilcox , John Hubbard , Mike Rapoport , peterx@redhat.com, Hugh Dickins , David Hildenbrand , Andrea Arcangeli , Rik van Riel , James Houghton , Yang Shi , Jason Gunthorpe , Vlastimil Babka , Andrew Morton Subject: [PATCH RFC 07/12] mm/gup: Refactor record_subpages() to find 1st small page Date: Wed, 15 Nov 2023 20:29:03 -0500 Message-ID: <20231116012908.392077-8-peterx@redhat.com> X-Mailer: git-send-email 2.41.0 In-Reply-To: <20231116012908.392077-1-peterx@redhat.com> References: <20231116012908.392077-1-peterx@redhat.com> MIME-Version: 1.0 X-Mimecast-Spam-Score: 0 X-Mimecast-Originator: redhat.com X-Rspam-User: X-Rspamd-Server: rspam06 X-Rspamd-Queue-Id: 1A18A14000F X-Stat-Signature: srnwxk3w9uzjci3ii96kkoye9hxeohcs X-HE-Tag: 1700098168-898148 X-HE-Meta: U2FsdGVkX19uA7riHIn5atXgS1sVbh9gi4pIV2WC4WjeHX5ABLXEAHgH39XtGweP/kIQ59v5SPa/RqBpUrfJCEKlDYqtGst4EWyH8raoOHGNHV3mqEa8hyxGpiWyxQgS7pi6uN4Tm7g2+IbEvI3sM1bNcKKgm0PPaPdOdWCvmMdwBTzx+A9gGqLr1K76cbHuzGYiDo6SGYAFo+YSQ4RQzcxDpFUUJgFIYLsWvOZ8SFtdrDfI1VrtePwH3rmYy66Qcu/dt25K2yiaJhsuaiIIhPQ1YD2a0Jtzm+0f6GwfDqDF/6x9xaXZOL8CZ9jvU21rRv/ssXm78q9lOV2E7Lvo/rQsJx/XEfveFgNtH+q20Lt475HjAqrccIjrXUwWPbOPFbITKY3jaMcvmApBkxBRcNhoZszO7ptStfbxkZJmu4xLbvDg9jXqPEvqADP5IG+ULTqjXL2mnnHlCVu77ji55aMqxwZBEaP+XGU8Mm4qLxCbfcbIoPJ9MNhryvmrAVQR+MYm/hb8uPS4DzJ4E3gkuwW4izqk20cfk8c7LFwB/FZeYRMgvMdljPAw4rhUGKx07w7SufxleN/RSg/NWFx+qMNEVaDfZTtYzYITn3J5iwEwkjUMKrtbSSjQydcS8FxI5TAnN2dByXvU7RF6sS85kNaZSE3mnGgb50v6HMtc6q3kVqDEdXLLtpfqNRHXpyL6Duzxb7OOUZJoXuNzxn2S8PlxGDLgPvkDXVhT6JdAv53ne0m2WNpee/lxKzzZEe5rxOfJEkdYtvBD/qqxQeIoTmoeqlmVgSoEvDXRu/tA7/QvU5lrZh9Cff0/Ygl18+PbeyOmR1Mc+FaYt6B8UuJBPCrJ9UaCCRQwxcC+ZuqmTG6ggNMtpHobakZbGFYAFsTEj/gSOckveVhW8wQQE/mr3csPZ52Btizm5JSFVkUR1dZipVgl/ooBUSUXdOEGZZbdrVlzyYxLUZtD19kYQua LfutFIZW ivPKg6xzO+0kD4OZ/R/dSb4fC0EiiP23zz4JKsv/ZFxN0QjY13auB2hoCnR6KerX0jnb/xvWI3w+Mk7Ul8hancEpOkhWx+V5As+KLbSDnIE6WDpMwY9uc4ChZJQXjj5fvN0EYzdS4t7iWYclsf14Q/NwrxOCq+zWf2dzP5nJ86+19lqcRCq7TQ2Eb46+UZSUth2QZX0TU0y/qAPc6KbYR4xJz6/SzOFy2bR5rZminuyQ25XWRahjUEbKrdWSmyqTiXkaN8oROj05jixNp8QvK2YMOtvEWbvJqToCivgsfpNcBLyJueh8ztIezJKDp7j1F+/bI4gx5oWJT1F7WTwXz6k5H2vFOaVCHQ3XF0mIAdkkWOd0qnUtbjYC8pgijgUs6GOT0wmnZs3jwTIB0hHF+zl8rgFOKbGmcTCBIfRU9ofjaOAczogCl0lsl9A== X-Bogosity: Ham, tests=bogofilter, spamicity=0.000000, version=1.2.4 Sender: owner-linux-mm@kvack.org Precedence: bulk X-Loop: owner-majordomo@kvack.org List-ID: List-Subscribe: List-Unsubscribe: All the fast-gup functions take a tail page to operate, always need to do page mask calculations before feeding that into record_subpages(). Merge that logic into record_subpages(), so that we always take a head page, and leave the rest calculation to record_subpages(). Signed-off-by: Peter Xu --- mm/gup.c | 23 +++++++++++++---------- 1 file changed, 13 insertions(+), 10 deletions(-) diff --git a/mm/gup.c b/mm/gup.c index 424d45e1afb3..69dae51f3eb1 100644 --- a/mm/gup.c +++ b/mm/gup.c @@ -2763,11 +2763,14 @@ static int __gup_device_huge_pud(pud_t pud, pud_t *pudp, unsigned long addr, } #endif -static int record_subpages(struct page *page, unsigned long addr, - unsigned long end, struct page **pages) +static int record_subpages(struct page *head, unsigned long sz, + unsigned long addr, unsigned long end, + struct page **pages) { + struct page *page; int nr; + page = nth_page(head, (addr & (sz - 1)) >> PAGE_SHIFT); for (nr = 0; addr != end; nr++, addr += PAGE_SIZE) pages[nr] = nth_page(page, nr); @@ -2804,8 +2807,8 @@ static int gup_hugepte(pte_t *ptep, unsigned long sz, unsigned long addr, /* hugepages are never "special" */ VM_BUG_ON(!pfn_valid(pte_pfn(pte))); - page = nth_page(pte_page(pte), (addr & (sz - 1)) >> PAGE_SHIFT); - refs = record_subpages(page, addr, end, pages + *nr); + page = pte_page(pte); + refs = record_subpages(page, sz, addr, end, pages + *nr); folio = try_grab_folio(page, refs, flags); if (!folio) @@ -2870,8 +2873,8 @@ static int gup_huge_pmd(pmd_t orig, pmd_t *pmdp, unsigned long addr, pages, nr); } - page = nth_page(pmd_page(orig), (addr & ~PMD_MASK) >> PAGE_SHIFT); - refs = record_subpages(page, addr, end, pages + *nr); + page = pmd_page(orig); + refs = record_subpages(page, PMD_SIZE, addr, end, pages + *nr); folio = try_grab_folio(page, refs, flags); if (!folio) @@ -2914,8 +2917,8 @@ static int gup_huge_pud(pud_t orig, pud_t *pudp, unsigned long addr, pages, nr); } - page = nth_page(pud_page(orig), (addr & ~PUD_MASK) >> PAGE_SHIFT); - refs = record_subpages(page, addr, end, pages + *nr); + page = pud_page(orig); + refs = record_subpages(page, PUD_SIZE, addr, end, pages + *nr); folio = try_grab_folio(page, refs, flags); if (!folio) @@ -2954,8 +2957,8 @@ static int gup_huge_pgd(pgd_t orig, pgd_t *pgdp, unsigned long addr, BUILD_BUG_ON(pgd_devmap(orig)); - page = nth_page(pgd_page(orig), (addr & ~PGDIR_MASK) >> PAGE_SHIFT); - refs = record_subpages(page, addr, end, pages + *nr); + page = pgd_page(orig); + refs = record_subpages(page, PGDIR_SIZE, addr, end, pages + *nr); folio = try_grab_folio(page, refs, flags); if (!folio) From patchwork Thu Nov 16 01:29:04 2023 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Peter Xu X-Patchwork-Id: 13457482 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from kanga.kvack.org (kanga.kvack.org [205.233.56.17]) by smtp.lore.kernel.org (Postfix) with ESMTP id 9C710C5AE4A for ; Thu, 16 Nov 2023 01:29:33 +0000 (UTC) Received: by kanga.kvack.org (Postfix) id D8FA06B03E9; Wed, 15 Nov 2023 20:29:29 -0500 (EST) Received: by kanga.kvack.org (Postfix, from userid 40) id D186A6B03EB; Wed, 15 Nov 2023 20:29:29 -0500 (EST) X-Delivered-To: int-list-linux-mm@kvack.org Received: by kanga.kvack.org (Postfix, from userid 63042) id B41576B03EC; Wed, 15 Nov 2023 20:29:29 -0500 (EST) X-Delivered-To: linux-mm@kvack.org Received: from relay.hostedemail.com (smtprelay0015.hostedemail.com [216.40.44.15]) by kanga.kvack.org (Postfix) with ESMTP id 9C4126B03E9 for ; Wed, 15 Nov 2023 20:29:29 -0500 (EST) Received: from smtpin25.hostedemail.com (a10.router.float.18 [10.200.18.1]) by unirelay09.hostedemail.com (Postfix) with ESMTP id 6FDEF804B7 for ; Thu, 16 Nov 2023 01:29:29 +0000 (UTC) X-FDA: 81462084858.25.0AB74A4 Received: from us-smtp-delivery-124.mimecast.com (us-smtp-delivery-124.mimecast.com [170.10.129.124]) by imf03.hostedemail.com (Postfix) with ESMTP id 6D4CB20006 for ; Thu, 16 Nov 2023 01:29:27 +0000 (UTC) Authentication-Results: imf03.hostedemail.com; dkim=pass header.d=redhat.com header.s=mimecast20190719 header.b=Nvp2qtJX; spf=pass (imf03.hostedemail.com: domain of peterx@redhat.com designates 170.10.129.124 as permitted sender) smtp.mailfrom=peterx@redhat.com; dmarc=pass (policy=none) header.from=redhat.com ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=hostedemail.com; s=arc-20220608; t=1700098167; h=from:from:sender:reply-to:subject:subject:date:date: message-id:message-id:to:to:cc:cc:mime-version:mime-version: content-type:content-type: content-transfer-encoding:content-transfer-encoding: in-reply-to:in-reply-to:references:references:dkim-signature; bh=hxqjSAwi8aTfo4zdaLYLd0e+DlQCcl09BvXUJR2cui0=; b=6AW7hBR7CI9RFnU8m1xe0n5e3HLLXA5jFjY/2c1REeQq1ZXXZWsmKvgPRdCAI3OxXcihQV E5bjifMLm8+g9msXxiMiTNx7oPtYwfvRRpII+/TNbnb8JYq9F4L8Pjcf2N3HfmIFhoo0fU Y/HEueupoLSrzlqaEjsqHxv4WHMZ3AE= ARC-Seal: i=1; s=arc-20220608; d=hostedemail.com; t=1700098167; a=rsa-sha256; cv=none; b=y6VKonOxI4yT2U3VlbDKszqvrhYFM5Fi8zBtwFf4dOrhO8l81kEGtVQ8RdsZ75c1YQzfaZ ohF4ntWoRvusN6R/RmyJAXVGehG5SenZnHhVU92lOhDERTt7kKOLZrSfjQne9iROV3CUFK 4MnteLLEyDh+CbZEMFjHJyUVprCvEaI= ARC-Authentication-Results: i=1; imf03.hostedemail.com; dkim=pass header.d=redhat.com header.s=mimecast20190719 header.b=Nvp2qtJX; spf=pass (imf03.hostedemail.com: domain of peterx@redhat.com designates 170.10.129.124 as permitted sender) smtp.mailfrom=peterx@redhat.com; dmarc=pass (policy=none) header.from=redhat.com DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=redhat.com; s=mimecast20190719; t=1700098166; h=from:from:reply-to:subject:subject:date:date:message-id:message-id: to:to:cc:cc:mime-version:mime-version:content-type:content-type: content-transfer-encoding:content-transfer-encoding: in-reply-to:in-reply-to:references:references; bh=hxqjSAwi8aTfo4zdaLYLd0e+DlQCcl09BvXUJR2cui0=; b=Nvp2qtJXwSymt57zTo3T1jyHUAW/TexmQ25H4l6H0t5L7jpbsCrjLiciEDZ8/toCHOPFD4 LvcadK6yDX0Y/AMxv7Cv0cfvBDflQGU1z9TomHsATXcZBapnCC0EOCQeltRtduln0kFYKG 4izUGfxwWEAndFUrWxIGbG8Ak950Kx8= Received: from mail-qv1-f70.google.com (mail-qv1-f70.google.com [209.85.219.70]) by relay.mimecast.com with ESMTP with STARTTLS (version=TLSv1.3, cipher=TLS_AES_256_GCM_SHA384) id us-mta-21-nAhfH7V0O9G7ofKJvQTx9A-1; Wed, 15 Nov 2023 20:29:25 -0500 X-MC-Unique: nAhfH7V0O9G7ofKJvQTx9A-1 Received: by mail-qv1-f70.google.com with SMTP id 6a1803df08f44-66d7b75c854so807686d6.1 for ; Wed, 15 Nov 2023 17:29:25 -0800 (PST) X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20230601; t=1700098164; x=1700702964; h=content-transfer-encoding:mime-version:references:in-reply-to :message-id:date:subject:cc:to:from:x-gm-message-state:from:to:cc :subject:date:message-id:reply-to; bh=hxqjSAwi8aTfo4zdaLYLd0e+DlQCcl09BvXUJR2cui0=; b=SrUyeDynWACTvuzX/IjvZ0hPra1RyK7KjFV1W/slVCV5YsEYzfrt8jDyw2KLNo8q7D /ZrIBdBCwrUwlz8DJqLQHZAa2aqxx5amqEpE8SRMT1z1DtBLU0yNjfO6TVPCSt3DM8Q9 65Fj0zmQk/k69XKfCCLB/8NKP44NCOL00GibmdMWmnv0mZ8G0i76lhSyoK5OprrTpU/L s1zQsu7fk8l4boKys5KLzTmZ8YLJvfC/hJ8XfCdXtxshy9XNlfZ+0rqHcWk0SsSXyRC0 U8rkOgP2n2mrvLsp9VvYeKuXifEilpmCEHG8l38sLDr4stGR5lCbvEF+ZVFrbPfqZcgo 3nJQ== X-Gm-Message-State: AOJu0Yw68rb4oIuvRcixgLE/plhExExuRe9tP85wH+4Hh7Wsw5XVqNa8 jnlhuV3cMeCsSvo25fVHk3hBaSh4IK853kMGIPR0ytkOkQwJ+Va7GYaiB9zU41koeKCZIosjE82 CGCAJhocbVOg= X-Received: by 2002:a05:622a:4291:b0:41c:d528:6589 with SMTP id cr17-20020a05622a429100b0041cd5286589mr8639212qtb.4.1700098164580; Wed, 15 Nov 2023 17:29:24 -0800 (PST) X-Google-Smtp-Source: AGHT+IHJwc+nHfVii0rwj2VF/0wkfwbvSOBYLSzNseCq1M0oxHWvF5z7fbEvQHxGk1DiYrhq4t4KkQ== X-Received: by 2002:a05:622a:4291:b0:41c:d528:6589 with SMTP id cr17-20020a05622a429100b0041cd5286589mr8639189qtb.4.1700098164258; Wed, 15 Nov 2023 17:29:24 -0800 (PST) Received: from x1n.redhat.com (cpe5c7695f3aee0-cm5c7695f3aede.cpe.net.cable.rogers.com. [99.254.144.39]) by smtp.gmail.com with ESMTPSA id c24-20020ac85198000000b0041e383d527esm3922598qtn.66.2023.11.15.17.29.22 (version=TLS1_3 cipher=TLS_AES_256_GCM_SHA384 bits=256/256); Wed, 15 Nov 2023 17:29:23 -0800 (PST) From: Peter Xu To: linux-kernel@vger.kernel.org, linux-mm@kvack.org Cc: Mike Kravetz , "Kirill A . Shutemov" , Lorenzo Stoakes , Axel Rasmussen , Matthew Wilcox , John Hubbard , Mike Rapoport , peterx@redhat.com, Hugh Dickins , David Hildenbrand , Andrea Arcangeli , Rik van Riel , James Houghton , Yang Shi , Jason Gunthorpe , Vlastimil Babka , Andrew Morton Subject: [PATCH RFC 08/12] mm/gup: Handle hugetlb for no_page_table() Date: Wed, 15 Nov 2023 20:29:04 -0500 Message-ID: <20231116012908.392077-9-peterx@redhat.com> X-Mailer: git-send-email 2.41.0 In-Reply-To: <20231116012908.392077-1-peterx@redhat.com> References: <20231116012908.392077-1-peterx@redhat.com> MIME-Version: 1.0 X-Mimecast-Spam-Score: 0 X-Mimecast-Originator: redhat.com X-Stat-Signature: sf1sxsgocx6p1uengytihgrex1u6rfez X-Rspamd-Server: rspam10 X-Rspamd-Queue-Id: 6D4CB20006 X-Rspam-User: X-HE-Tag: 1700098167-535881 X-HE-Meta: U2FsdGVkX1/0XIMrVC6zkyOn6p1qdpG0gyHm9yOjAStWf/6Bdc1Yt165aQ5pbIJL7P+C9SMQNWh1xwRPzNxsJVmRDV8sscWe8pMCOhzAPNhD/jMR1uwAwgo1dKob1db4QpkhaN1yJ3wJJrzhUO4TEgSTKT+TLdqha9p9l6lKhrAMViVcoyRXOSqYbci+Tkgw3MLtCLXOf9nlkK0F/D0Thttk0uQPsCOGGrdALlxxe9oEWkPeA1VoPUC+/vmiwbMzHSqdgs3pYZan1uvV5pG7GLQB+YoVL+ZEBm2W2Jie/LFe+T1UqUUJuL8Vm/Hs6WnccVajMBVM/GIbi0GSPzX5JuSsEm8FmuWDPkYNpNxq52PyA0tk7BmDuI0cfw0CNAZLs4+T62p0E8cPtm5H6puuTHE3YIPI5lC7FOt65cr+C3Juokq0YUYEIZG68SQrkWKC0zvwAzJcdI55fcKnEEmW5lGG5cMiaLWFdj+gp16uqJr3m25cjw87Ki5ps8LrY4NKk4T+s89S3FwsQgsbAQP/FhW+oYd3zprgZBXkjI6RryzV/yMvrPYIYPl5XS92aBlB6WGkHGm4UaVSBLCxOO9LzEgsybKdqBz0aBnidEJmddIJc36DHpt6bvd5k5D7mtj/tBquZhMaoxLpJZ2NYBu5UxpZhYuD2xqkYDl7PueNv3fuSd0Fj0SzHfdIeBMfiBgd18hYNiP9KMVQXRluUxKZILQN1ZsU4WRmdmg4FKmb1hMKibN2BO8ymE6sslJdfbQg2fqDOvuKwN62IoihCTmY6LSaXidcLFRXLyYc5gq0vgxqcS6vn35HYtpd09Al61HgKlS3byGjnLCEUf4yGI0q4buLIcL5TxVFQFv1sdCcaBkAaLMQrDmcwtZ75Fsz25+h8vxWCFOsJnJ8rmJ+fccWb7/31aNOi/Kb+tjD2Pz3iNIemFqHr0ezuTx5nVrAJik5pcDc/xMbKJZcpkN88Ou gOkgFheV x8OeGU01k77rP+yEJLUcv/8VOS7ZvZ+6mo1x2AAo+takHLduXssaQ/bxDYxil8iBQd4y/lU00lx+cbxVs6mpBO0lhiPez3Kv2lsYajt3ewruTnn2QaV35eySg+hON+N6I1r5C2L/BrfoM+mKylU0utGlJkP56QJjwObZiHDAbgcn4oJoPwzx5bK4dGGhS0mlKrN6ngIemxyCvvAviROwdGddVww2bhOD3tm8/7OCeTNs5H4ze9PSbh2EXB4o4iaEds/+WCOlbEEchvPA5mRNMAnizdtSAlusUlD6LZbHEG90E27lg6S7asaWibUTmeDJCzryWFTmzjs1I7/0AuRrZTJbsduMBEmc8KOi9chDzUNz7dZv7A2DCIvbop3e6bebPkI2Pn7N1tU+CKnTqTimh+ww5N3C9M0FfEMcivAhiJNTnUIREkSXsZHEIVA== X-Bogosity: Ham, tests=bogofilter, spamicity=0.000000, version=1.2.4 Sender: owner-linux-mm@kvack.org Precedence: bulk X-Loop: owner-majordomo@kvack.org List-ID: List-Subscribe: List-Unsubscribe: no_page_table() is not yet used for hugetlb code paths. Make it prepared. The major difference here is hugetlb will return -EFAULT as long as page cache does not exist, even if VM_SHARED. See hugetlb_follow_page_mask(). Pass "address" into no_page_table() too, as hugetlb will need it. Signed-off-by: Peter Xu Reviewed-by: Christoph Hellwig --- mm/gup.c | 40 ++++++++++++++++++++++++---------------- 1 file changed, 24 insertions(+), 16 deletions(-) diff --git a/mm/gup.c b/mm/gup.c index 69dae51f3eb1..89c1584d68f0 100644 --- a/mm/gup.c +++ b/mm/gup.c @@ -501,19 +501,27 @@ static inline void mm_set_has_pinned_flag(unsigned long *mm_flags) #ifdef CONFIG_MMU static struct page *no_page_table(struct vm_area_struct *vma, - unsigned int flags) + unsigned int flags, unsigned long address) { + if (!(flags & FOLL_DUMP)) + return NULL; + /* - * When core dumping an enormous anonymous area that nobody - * has touched so far, we don't want to allocate unnecessary pages or + * When core dumping, we don't want to allocate unnecessary pages or * page tables. Return error instead of NULL to skip handle_mm_fault, * then get_dump_page() will return NULL to leave a hole in the dump. * But we can only make this optimization where a hole would surely * be zero-filled if handle_mm_fault() actually did handle it. */ - if ((flags & FOLL_DUMP) && - (vma_is_anonymous(vma) || !vma->vm_ops->fault)) + if (is_vm_hugetlb_page(vma)) { + struct hstate *h = hstate_vma(vma); + + if (!hugetlbfs_pagecache_present(h, vma, address)) + return ERR_PTR(-EFAULT); + } else if ((vma_is_anonymous(vma) || !vma->vm_ops->fault)) { return ERR_PTR(-EFAULT); + } + return NULL; } @@ -593,7 +601,7 @@ static struct page *follow_page_pte(struct vm_area_struct *vma, ptep = pte_offset_map_lock(mm, pmd, address, &ptl); if (!ptep) - return no_page_table(vma, flags); + return no_page_table(vma, flags, address); pte = ptep_get(ptep); if (!pte_present(pte)) goto no_page; @@ -685,7 +693,7 @@ static struct page *follow_page_pte(struct vm_area_struct *vma, pte_unmap_unlock(ptep, ptl); if (!pte_none(pte)) return NULL; - return no_page_table(vma, flags); + return no_page_table(vma, flags, address); } static struct page *follow_pmd_mask(struct vm_area_struct *vma, @@ -701,9 +709,9 @@ static struct page *follow_pmd_mask(struct vm_area_struct *vma, pmd = pmd_offset(pudp, address); pmdval = pmdp_get_lockless(pmd); if (pmd_none(pmdval)) - return no_page_table(vma, flags); + return no_page_table(vma, flags, address); if (!pmd_present(pmdval)) - return no_page_table(vma, flags); + return no_page_table(vma, flags, address); if (pmd_devmap(pmdval)) { ptl = pmd_lock(mm, pmd); page = follow_devmap_pmd(vma, address, pmd, flags, &ctx->pgmap); @@ -714,12 +722,12 @@ static struct page *follow_pmd_mask(struct vm_area_struct *vma, return follow_page_pte(vma, address, pmd, flags, &ctx->pgmap); if (pmd_protnone(pmdval) && !gup_can_follow_protnone(vma, flags)) - return no_page_table(vma, flags); + return no_page_table(vma, flags, address); ptl = pmd_lock(mm, pmd); if (unlikely(!pmd_present(*pmd))) { spin_unlock(ptl); - return no_page_table(vma, flags); + return no_page_table(vma, flags, address); } if (unlikely(!pmd_trans_huge(*pmd))) { spin_unlock(ptl); @@ -750,7 +758,7 @@ static struct page *follow_pud_mask(struct vm_area_struct *vma, pud = pud_offset(p4dp, address); if (pud_none(*pud)) - return no_page_table(vma, flags); + return no_page_table(vma, flags, address); if (pud_devmap(*pud)) { ptl = pud_lock(mm, pud); page = follow_devmap_pud(vma, address, pud, flags, &ctx->pgmap); @@ -758,7 +766,7 @@ static struct page *follow_pud_mask(struct vm_area_struct *vma, return page; } if (unlikely(pud_bad(*pud))) - return no_page_table(vma, flags); + return no_page_table(vma, flags, address); return follow_pmd_mask(vma, address, pud, flags, ctx); } @@ -772,10 +780,10 @@ static struct page *follow_p4d_mask(struct vm_area_struct *vma, p4d = p4d_offset(pgdp, address); if (p4d_none(*p4d)) - return no_page_table(vma, flags); + return no_page_table(vma, flags, address); BUILD_BUG_ON(p4d_huge(*p4d)); if (unlikely(p4d_bad(*p4d))) - return no_page_table(vma, flags); + return no_page_table(vma, flags, address); return follow_pud_mask(vma, address, p4d, flags, ctx); } @@ -825,7 +833,7 @@ static struct page *follow_page_mask(struct vm_area_struct *vma, pgd = pgd_offset(mm, address); if (pgd_none(*pgd) || unlikely(pgd_bad(*pgd))) - return no_page_table(vma, flags); + return no_page_table(vma, flags, address); return follow_p4d_mask(vma, address, pgd, flags, ctx); } From patchwork Thu Nov 16 01:29:05 2023 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Peter Xu X-Patchwork-Id: 13457483 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from kanga.kvack.org (kanga.kvack.org [205.233.56.17]) by smtp.lore.kernel.org (Postfix) with ESMTP id 430C2C2BB3F for ; Thu, 16 Nov 2023 01:29:36 +0000 (UTC) Received: by kanga.kvack.org (Postfix) id 7CB8E6B03EB; Wed, 15 Nov 2023 20:29:31 -0500 (EST) Received: by kanga.kvack.org (Postfix, from userid 40) id 754856B03ED; Wed, 15 Nov 2023 20:29:31 -0500 (EST) X-Delivered-To: int-list-linux-mm@kvack.org Received: by kanga.kvack.org (Postfix, from userid 63042) id 499236B03EE; Wed, 15 Nov 2023 20:29:31 -0500 (EST) X-Delivered-To: linux-mm@kvack.org Received: from relay.hostedemail.com (smtprelay0011.hostedemail.com [216.40.44.11]) by kanga.kvack.org (Postfix) with ESMTP id 2E88D6B03EB for ; Wed, 15 Nov 2023 20:29:31 -0500 (EST) Received: from smtpin18.hostedemail.com (a10.router.float.18 [10.200.18.1]) by unirelay07.hostedemail.com (Postfix) with ESMTP id 0CCBE160213 for ; Thu, 16 Nov 2023 01:29:31 +0000 (UTC) X-FDA: 81462084942.18.D696E7E Received: from us-smtp-delivery-124.mimecast.com (us-smtp-delivery-124.mimecast.com [170.10.129.124]) by imf22.hostedemail.com (Postfix) with ESMTP id EBD17C000F for ; Thu, 16 Nov 2023 01:29:28 +0000 (UTC) Authentication-Results: imf22.hostedemail.com; dkim=pass header.d=redhat.com header.s=mimecast20190719 header.b=Sp5+Rq+g; dmarc=pass (policy=none) header.from=redhat.com; spf=pass (imf22.hostedemail.com: domain of peterx@redhat.com designates 170.10.129.124 as permitted sender) smtp.mailfrom=peterx@redhat.com ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=hostedemail.com; s=arc-20220608; t=1700098169; h=from:from:sender:reply-to:subject:subject:date:date: message-id:message-id:to:to:cc:cc:mime-version:mime-version: content-type:content-type: content-transfer-encoding:content-transfer-encoding: in-reply-to:in-reply-to:references:references:dkim-signature; bh=gQy6k5CEHGKiOJI0ugKFVUkk8XVFLkY9zLGCNjIpOps=; b=33zwD4J2Q5ZxEay49i8aNUVovy+jKiaF+VBGJLpsQy+4bwyobpk5oiTEIOssoVSICwRVvp PsCoioXk1jpk6j6P3gIPx89ouSlWUh/BZMGrdtRfmdF6uN4FKhXNTRHJOr9LS+JslfgTS+ e+kPucAp5EADrEGeDHAWD1CYH4mVGnI= ARC-Authentication-Results: i=1; imf22.hostedemail.com; dkim=pass header.d=redhat.com header.s=mimecast20190719 header.b=Sp5+Rq+g; dmarc=pass (policy=none) header.from=redhat.com; spf=pass (imf22.hostedemail.com: domain of peterx@redhat.com designates 170.10.129.124 as permitted sender) smtp.mailfrom=peterx@redhat.com ARC-Seal: i=1; s=arc-20220608; d=hostedemail.com; t=1700098169; a=rsa-sha256; cv=none; b=ArFbVJHoWGoDfpcI9VWcrhhnlnreo58MWArUdZRxRZhbJhQXbtS46RuPXmR7cQdXleuoF9 +7xjXze4EAcqrCilVnG1aROxg+cdi0wNSBufBoSslbX/kQd1n4AQ3CqIv7yX5PHiB0xDVU NWQFPb+Br0/JDSKMrcev98duVEu2a3o= DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=redhat.com; s=mimecast20190719; t=1700098168; h=from:from:reply-to:subject:subject:date:date:message-id:message-id: to:to:cc:cc:mime-version:mime-version:content-type:content-type: content-transfer-encoding:content-transfer-encoding: in-reply-to:in-reply-to:references:references; bh=gQy6k5CEHGKiOJI0ugKFVUkk8XVFLkY9zLGCNjIpOps=; b=Sp5+Rq+gsh9x5CH4Pj6Cbe5AD2W5jrFJ8Sb477unfTkNOPMFU+rIvyaLr7zdCu0db++s3i laG7QpqJf1uzoxb6oyngln0N9Sq9bTW4zc8O6OrEPXk1P404LkwrephDrGeM1CP237+AmB QXMeGFnjKsf46GQXSRcLneurnAO64DE= Received: from mail-vk1-f199.google.com (mail-vk1-f199.google.com [209.85.221.199]) by relay.mimecast.com with ESMTP with STARTTLS (version=TLSv1.3, cipher=TLS_AES_256_GCM_SHA384) id us-mta-45-wPQuVHxbPyWPY-sZWgXHqA-1; Wed, 15 Nov 2023 20:29:26 -0500 X-MC-Unique: wPQuVHxbPyWPY-sZWgXHqA-1 Received: by mail-vk1-f199.google.com with SMTP id 71dfb90a1353d-4abee584390so39591e0c.1 for ; Wed, 15 Nov 2023 17:29:26 -0800 (PST) X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20230601; t=1700098166; x=1700702966; h=content-transfer-encoding:mime-version:references:in-reply-to :message-id:date:subject:cc:to:from:x-gm-message-state:from:to:cc :subject:date:message-id:reply-to; bh=gQy6k5CEHGKiOJI0ugKFVUkk8XVFLkY9zLGCNjIpOps=; b=Gxw+uWVMabo3FoWhX2jJkLSKausQKwqk1mJxZlKKWWPec+vEIo6p5cZGBsuV/U6qVW WnP87npltmTCNeAZNcI2JogVchzFZHLO7VgVkw7QrLjM4bNjDc1cD6Dh+ihLqeGUfCLZ 2sQ7rNepde+/7zjUahqrMUl2IhyC7D9a4ze8KpQgio2WXYFORKNKfglcSQApqvJ0Dbc2 zDXetvwTLpCCpe+0ZnMalTZa7GPGBzdxMJIJI4zRiGaI+IlgG11Gp270D3fKKAMrqQGI rEMydfsXgjfCF2EzohlfbJNVqJP9e0DGLvg4EAYdn+Ep9Ce0mrlkwqwm1cIdnTAQlFWL iA5g== X-Gm-Message-State: AOJu0YyaPhH3vsMYQINXNn88nQ7DZeHUleYP9cKa3Nhqr/4tRzwc+ItN CMqC0U6KY0TrqzSK6Lgo2xPzseCcQH/NS8DwUv+SE1HDyo5nkceCyvULHgwWprFkL9jVMVCsbjo M3gyhQdEauTk= X-Received: by 2002:a05:6122:1484:b0:4ab:f831:98d4 with SMTP id z4-20020a056122148400b004abf83198d4mr7555123vkp.0.1700098166248; Wed, 15 Nov 2023 17:29:26 -0800 (PST) X-Google-Smtp-Source: AGHT+IHNrGR90/jKzfekxV/YscaCCcqT0aN5ZcRXVaKloa9cNGZ2HnRMduZYMn8+Y4FcR5oknUprWw== X-Received: by 2002:a05:6122:1484:b0:4ab:f831:98d4 with SMTP id z4-20020a056122148400b004abf83198d4mr7555100vkp.0.1700098165901; Wed, 15 Nov 2023 17:29:25 -0800 (PST) Received: from x1n.redhat.com (cpe5c7695f3aee0-cm5c7695f3aede.cpe.net.cable.rogers.com. [99.254.144.39]) by smtp.gmail.com with ESMTPSA id c24-20020ac85198000000b0041e383d527esm3922598qtn.66.2023.11.15.17.29.24 (version=TLS1_3 cipher=TLS_AES_256_GCM_SHA384 bits=256/256); Wed, 15 Nov 2023 17:29:25 -0800 (PST) From: Peter Xu To: linux-kernel@vger.kernel.org, linux-mm@kvack.org Cc: Mike Kravetz , "Kirill A . Shutemov" , Lorenzo Stoakes , Axel Rasmussen , Matthew Wilcox , John Hubbard , Mike Rapoport , peterx@redhat.com, Hugh Dickins , David Hildenbrand , Andrea Arcangeli , Rik van Riel , James Houghton , Yang Shi , Jason Gunthorpe , Vlastimil Babka , Andrew Morton Subject: [PATCH RFC 09/12] mm/gup: Handle huge pud for follow_pud_mask() Date: Wed, 15 Nov 2023 20:29:05 -0500 Message-ID: <20231116012908.392077-10-peterx@redhat.com> X-Mailer: git-send-email 2.41.0 In-Reply-To: <20231116012908.392077-1-peterx@redhat.com> References: <20231116012908.392077-1-peterx@redhat.com> MIME-Version: 1.0 X-Mimecast-Spam-Score: 0 X-Mimecast-Originator: redhat.com X-Rspamd-Queue-Id: EBD17C000F X-Rspam-User: X-Rspamd-Server: rspam05 X-Stat-Signature: gsbrfbh3sjek1yh1zdmp7xhdiicpi9ij X-HE-Tag: 1700098168-452744 X-HE-Meta: U2FsdGVkX18kOsDhDdcFyM0k8GbrB4dHWz34UObW76nISHO6+Kyag4DjnRkoUs4cfA1zWQOOaAdpVCCghSD0H/czJuSItZ9WNxi2OZtmoMoC+pWiUWsTEcUNtvBgk5o75WoU1HRHKykLXw+jOJJBkHSO/KMBRO0ly0jwasQJnkAMzSFnjXYdf7oQpoVyYbCa8TuyCWBnX1U5gR/0QBkMhY7dVkwWH15De9KAu8PXvV3HsSOiqMe4RHHqlYBDre9lLzz1K4wp+xXgy6umUpqoU2ltMSKFhmw7B69btTwsbT/CA9uiai8t/JQQVIGGzYAl7uoKS/bFDmX8om7pycDQHrSJ8xexvGQQMEfj1Oa01A3N6NVXfSpqGrUfVxKc6UuRc3yPdM+x2fLhfn0HTOwKSMAnQBecgBZbdtTdgy1SAR6iqjuhDzG161JyeKkhhp/QAJCLYJYat5YThWUG8rjXF70YP19vZ2FbszeVjQg/h+PO7+orxrYF6GX6YVF84/DeVHLPlJzntuhHS3KdSo/CNacZPtjBk5vN5cDFuhJ78lWVzuslcqGOMwaJII6h4A8BdC1QyXnR/F4YnDD11rLAhAQ7mrJ+CVnmGSWwYLhKDlNEsxOeYK+yOSAMtcnJo88if0yiCsdD53lyJDtOV1PWEiOzgSJoyyDOyEdpQz77eZASIdfOvWz/CvMdtBErIS53efX81RsW4v8Q1WwO7n5CEi9xZN4gk6gKWO9V9KF/EEkTQlTV65QYc7OWwu7SDFo9NogcCqjD2xgAGSAGZFMvaXBF1EjoUl+eKIIXlBY7vNFqEXEF6iQDHwSbMGTYLaU+kxgThh7rpPuYOlY2FiGSa0SG8kSe3ieatTbUlDBE5tCARZcc1rTL0l6pVgtwCXTq5MNz8F2SjUQ5MlzcPmvsD65zfG0SDuosDCyLSiJdXYpv040VDWMTkUrQwUW1ewVg1ZyrBqscizF0EptLBxY XEKUuX3I 5TG+4NylrZiiKblUXEbGH96kRdGfMTIRzaTFcsrStgfIycYuOXbgSxG6vxbMSAP1xzXGqWTdmFT/a22gCMDz/BHxmgQiIx1cDApOwWb/U4WbmBOca48OjNU5XOEmztxUw6EClgXpUpj18kY2zL9TgBCI6G8iGamm3nqI9NkBqQq+XTN7n9Te2ucBne/YRi1h+tUwLeuy0q4dQtVUqm1gafUkyHX2dehfOI/A80qqKY9O3wZFGOmgBGngBVknPL4QKMDgj6AnznBPKKzquMMHue4gjLq2/qzC9Ik2+dd/WYM6MooqKeSlGTyjk0cihY135x6DMKVWIIfKIk7AM4JaNEssiu4zpSmdMqEi6sKoiqWEexeQLtCpBzsjwLD7ioIMOnsdv8DQh1nGLwF+LU7Eee3TaGo3jW4KhoVwReelnx49AqwuHJd6SnUfDWQ== X-Bogosity: Ham, tests=bogofilter, spamicity=0.000000, version=1.2.4 Sender: owner-linux-mm@kvack.org Precedence: bulk X-Loop: owner-majordomo@kvack.org List-ID: List-Subscribe: List-Unsubscribe: Teach follow_pud_mask() to be able to handle normal PUD pages like hugetlb. Rename follow_devmap_pud() to follow_huge_pud(), move it out of config CONFIG_HAVE_ARCH_TRANSPARENT_HUGEPAGE_PUD, instead let that macro only covers the devmap special operations, like pgmap. In the new follow_huge_pud(), taking care of possible CoR for hugetlb if necessary. Since at it, optimize the non-present check by adding a pud_present() early check before taking the pgtable lock, failing the follow_page() early if PUD is not present: that is required by both devmap or hugetlb. Use pud_huge() to also cover the pud_devmap() case. We need to export "struct follow_page_context" along the way, so that huge_memory.c can understand it. One trivial more thing to mention is, introduce "pud_t pud" in the code paths along the way, so the code doesn't dereference *pudp multiple time. Not only because that looks less straightforward, but also because if the dereference really happened, it's not clear whether there can be race to see different *pudp values when it's being modified at the same time. Setting ctx->page_mask properly for a PUD entry. As a side effect, this should also be able to optimize devmap GUP on PUD to be able to jump over the whole PUD range, but not yet verified. Signed-off-by: Peter Xu --- include/linux/huge_mm.h | 17 +++---- mm/gup.c | 22 ++++----- mm/huge_memory.c | 98 +++++++++++++++++++++++------------------ 3 files changed, 73 insertions(+), 64 deletions(-) diff --git a/include/linux/huge_mm.h b/include/linux/huge_mm.h index ec463410aecc..84815012d3cf 100644 --- a/include/linux/huge_mm.h +++ b/include/linux/huge_mm.h @@ -7,6 +7,11 @@ #include /* only for vma_is_dax() */ +struct follow_page_context { + struct dev_pagemap *pgmap; + unsigned int page_mask; +}; + vm_fault_t do_huge_pmd_anonymous_page(struct vm_fault *vmf); int copy_huge_pmd(struct mm_struct *dst_mm, struct mm_struct *src_mm, pmd_t *dst_pmd, pmd_t *src_pmd, unsigned long addr, @@ -222,8 +227,6 @@ static inline bool folio_test_pmd_mappable(struct folio *folio) struct page *follow_devmap_pmd(struct vm_area_struct *vma, unsigned long addr, pmd_t *pmd, int flags, struct dev_pagemap **pgmap); -struct page *follow_devmap_pud(struct vm_area_struct *vma, unsigned long addr, - pud_t *pud, int flags, struct dev_pagemap **pgmap); vm_fault_t do_huge_pmd_numa_page(struct vm_fault *vmf); @@ -372,18 +375,16 @@ static inline struct page *follow_devmap_pmd(struct vm_area_struct *vma, return NULL; } -static inline struct page *follow_devmap_pud(struct vm_area_struct *vma, - unsigned long addr, pud_t *pud, int flags, struct dev_pagemap **pgmap) -{ - return NULL; -} - static inline bool thp_migration_supported(void) { return false; } #endif /* CONFIG_TRANSPARENT_HUGEPAGE */ +struct page *follow_huge_pud(struct vm_area_struct *vma, unsigned long addr, + pud_t *pud, int flags, + struct follow_page_context *ctx); + static inline int split_folio_to_list(struct folio *folio, struct list_head *list) { diff --git a/mm/gup.c b/mm/gup.c index 89c1584d68f0..55a2ae55f00f 100644 --- a/mm/gup.c +++ b/mm/gup.c @@ -25,11 +25,6 @@ #include "internal.h" -struct follow_page_context { - struct dev_pagemap *pgmap; - unsigned int page_mask; -}; - static inline void sanity_check_pinned_pages(struct page **pages, unsigned long npages) { @@ -751,24 +746,25 @@ static struct page *follow_pud_mask(struct vm_area_struct *vma, unsigned int flags, struct follow_page_context *ctx) { - pud_t *pud; + pud_t *pudp, pud; spinlock_t *ptl; struct page *page; struct mm_struct *mm = vma->vm_mm; - pud = pud_offset(p4dp, address); - if (pud_none(*pud)) + pudp = pud_offset(p4dp, address); + pud = *pudp; + if (pud_none(pud) || !pud_present(pud)) return no_page_table(vma, flags, address); - if (pud_devmap(*pud)) { - ptl = pud_lock(mm, pud); - page = follow_devmap_pud(vma, address, pud, flags, &ctx->pgmap); + if (pud_huge(pud)) { + ptl = pud_lock(mm, pudp); + page = follow_huge_pud(vma, address, pudp, flags, ctx); spin_unlock(ptl); return page; } - if (unlikely(pud_bad(*pud))) + if (unlikely(pud_bad(pud))) return no_page_table(vma, flags, address); - return follow_pmd_mask(vma, address, pud, flags, ctx); + return follow_pmd_mask(vma, address, pudp, flags, ctx); } static struct page *follow_p4d_mask(struct vm_area_struct *vma, diff --git a/mm/huge_memory.c b/mm/huge_memory.c index 6eb55f97a3d2..6748ef5f3fd9 100644 --- a/mm/huge_memory.c +++ b/mm/huge_memory.c @@ -1207,49 +1207,6 @@ static void touch_pud(struct vm_area_struct *vma, unsigned long addr, update_mmu_cache_pud(vma, addr, pud); } -struct page *follow_devmap_pud(struct vm_area_struct *vma, unsigned long addr, - pud_t *pud, int flags, struct dev_pagemap **pgmap) -{ - unsigned long pfn = pud_pfn(*pud); - struct mm_struct *mm = vma->vm_mm; - struct page *page; - int ret; - - assert_spin_locked(pud_lockptr(mm, pud)); - - if (flags & FOLL_WRITE && !pud_write(*pud)) - return NULL; - - if (pud_present(*pud) && pud_devmap(*pud)) - /* pass */; - else - return NULL; - - if (flags & FOLL_TOUCH) - touch_pud(vma, addr, pud, flags & FOLL_WRITE); - - /* - * device mapped pages can only be returned if the - * caller will manage the page reference count. - * - * At least one of FOLL_GET | FOLL_PIN must be set, so assert that here: - */ - if (!(flags & (FOLL_GET | FOLL_PIN))) - return ERR_PTR(-EEXIST); - - pfn += (addr & ~PUD_MASK) >> PAGE_SHIFT; - *pgmap = get_dev_pagemap(pfn, *pgmap); - if (!*pgmap) - return ERR_PTR(-EFAULT); - page = pfn_to_page(pfn); - - ret = try_grab_page(page, flags); - if (ret) - page = ERR_PTR(ret); - - return page; -} - int copy_huge_pud(struct mm_struct *dst_mm, struct mm_struct *src_mm, pud_t *dst_pud, pud_t *src_pud, unsigned long addr, struct vm_area_struct *vma) @@ -1305,6 +1262,61 @@ void huge_pud_set_accessed(struct vm_fault *vmf, pud_t orig_pud) } #endif /* CONFIG_HAVE_ARCH_TRANSPARENT_HUGEPAGE_PUD */ +struct page *follow_huge_pud(struct vm_area_struct *vma, unsigned long addr, + pud_t *pudp, int flags, + struct follow_page_context *ctx) +{ + struct mm_struct *mm = vma->vm_mm; + struct page *page; + pud_t pud = *pudp; + unsigned long pfn = pud_pfn(pud); + int ret; + + assert_spin_locked(pud_lockptr(mm, pudp)); + + if ((flags & FOLL_WRITE) && !pud_write(pud)) + return NULL; + + if (!pud_present(pud)) + return NULL; + + pfn += (addr & ~PUD_MASK) >> PAGE_SHIFT; + +#ifdef CONFIG_HAVE_ARCH_TRANSPARENT_HUGEPAGE_PUD + if (pud_devmap(pud)) { + /* + * device mapped pages can only be returned if the caller + * will manage the page reference count. + * + * At least one of FOLL_GET | FOLL_PIN must be set, so + * assert that here: + */ + if (!(flags & (FOLL_GET | FOLL_PIN))) + return ERR_PTR(-EEXIST); + + if (flags & FOLL_TOUCH) + touch_pud(vma, addr, pudp, flags & FOLL_WRITE); + + ctx->pgmap = get_dev_pagemap(pfn, ctx->pgmap); + if (!ctx->pgmap) + return ERR_PTR(-EFAULT); + } +#endif + page = pfn_to_page(pfn); + + if (!pud_devmap(pud) && !pud_write(pud) && + gup_must_unshare(vma, flags, page)) + return ERR_PTR(-EMLINK); + + ret = try_grab_page(page, flags); + if (ret) + page = ERR_PTR(ret); + else + ctx->page_mask = HPAGE_PUD_NR - 1; + + return page; +} + void huge_pmd_set_accessed(struct vm_fault *vmf) { bool write = vmf->flags & FAULT_FLAG_WRITE; From patchwork Thu Nov 16 01:29:06 2023 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Peter Xu X-Patchwork-Id: 13457484 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from kanga.kvack.org (kanga.kvack.org [205.233.56.17]) by smtp.lore.kernel.org (Postfix) with ESMTP id 2B520C072A2 for ; Thu, 16 Nov 2023 01:29:39 +0000 (UTC) Received: by kanga.kvack.org (Postfix) id E02D96B03ED; Wed, 15 Nov 2023 20:29:32 -0500 (EST) Received: by kanga.kvack.org (Postfix, from userid 40) id DB3916B03EF; Wed, 15 Nov 2023 20:29:32 -0500 (EST) X-Delivered-To: int-list-linux-mm@kvack.org Received: by kanga.kvack.org (Postfix, from userid 63042) id C05BF6B03F0; Wed, 15 Nov 2023 20:29:32 -0500 (EST) X-Delivered-To: linux-mm@kvack.org Received: from relay.hostedemail.com (smtprelay0010.hostedemail.com [216.40.44.10]) by kanga.kvack.org (Postfix) with ESMTP id A861F6B03ED for ; Wed, 15 Nov 2023 20:29:32 -0500 (EST) Received: from smtpin12.hostedemail.com (a10.router.float.18 [10.200.18.1]) by unirelay01.hostedemail.com (Postfix) with ESMTP id 89EDC1CACBB for ; Thu, 16 Nov 2023 01:29:32 +0000 (UTC) X-FDA: 81462084984.12.B72DEFB Received: from us-smtp-delivery-124.mimecast.com (us-smtp-delivery-124.mimecast.com [170.10.133.124]) by imf02.hostedemail.com (Postfix) with ESMTP id 77C168000B for ; Thu, 16 Nov 2023 01:29:30 +0000 (UTC) Authentication-Results: imf02.hostedemail.com; dkim=pass header.d=redhat.com header.s=mimecast20190719 header.b=IdcAM6XS; spf=pass (imf02.hostedemail.com: domain of peterx@redhat.com designates 170.10.133.124 as permitted sender) smtp.mailfrom=peterx@redhat.com; dmarc=pass (policy=none) header.from=redhat.com ARC-Seal: i=1; s=arc-20220608; d=hostedemail.com; t=1700098170; a=rsa-sha256; cv=none; b=Jq8lwPJxgi4gT2NR+Wv9PoVVwDcxUbOsau6t8c2xEKa06j9BakH2TTy7RWgyJonZKoXn9I jZ2CnUhmDC2I/1Zd0yedsyZjTG5hZBRqMWGpKcZmFF+dod9v1/XBWmAwrrZzX503sLWJZ0 6YFM14p1T4W70ZGp7BB2glRWLJiNYmU= ARC-Authentication-Results: i=1; imf02.hostedemail.com; dkim=pass header.d=redhat.com header.s=mimecast20190719 header.b=IdcAM6XS; spf=pass (imf02.hostedemail.com: domain of peterx@redhat.com designates 170.10.133.124 as permitted sender) smtp.mailfrom=peterx@redhat.com; dmarc=pass (policy=none) header.from=redhat.com ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=hostedemail.com; s=arc-20220608; t=1700098170; h=from:from:sender:reply-to:subject:subject:date:date: message-id:message-id:to:to:cc:cc:mime-version:mime-version: content-type:content-type: content-transfer-encoding:content-transfer-encoding: in-reply-to:in-reply-to:references:references:dkim-signature; bh=KEE21en6sv5Z4IJFnP9A/1kLw7BwwEfESnJrJ/3CqcU=; b=g50iO3OhW38skWOPguqZjhuiE/gHew2t5HAjYQ1hJQUGf/yqX7UwiYunN557LSv2h2UC7S wkbKjLs5nuCtayN3nG6rm489qC2s7EaCoNdj+61EB0NlLmtFleIuYmhvLQcSpgWppVngQp /cHvRkVOqWx+8yi7DFeEL6nr8cwwS7A= DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=redhat.com; s=mimecast20190719; t=1700098169; h=from:from:reply-to:subject:subject:date:date:message-id:message-id: to:to:cc:cc:mime-version:mime-version:content-type:content-type: content-transfer-encoding:content-transfer-encoding: in-reply-to:in-reply-to:references:references; bh=KEE21en6sv5Z4IJFnP9A/1kLw7BwwEfESnJrJ/3CqcU=; b=IdcAM6XSAAXf9GmhU2ShqxdgWgHYGdhPdzRBZue366+0/HHABAW3e4WXiZtrg2P1Gyg25S 8KjwlII7f2TjBCRUOY9CeaUGjZ4KarOQ9MF7RHqPtbH6uX05f52QcZDRx1yGJvV7xQNkba gA0ln+k6BUTmOYNqUtYrYnDgcZwippo= Received: from mail-qt1-f197.google.com (mail-qt1-f197.google.com [209.85.160.197]) by relay.mimecast.com with ESMTP with STARTTLS (version=TLSv1.3, cipher=TLS_AES_256_GCM_SHA384) id us-mta-568-DxV7d27JOwGgcUg1kkMqPg-1; Wed, 15 Nov 2023 20:29:28 -0500 X-MC-Unique: DxV7d27JOwGgcUg1kkMqPg-1 Received: by mail-qt1-f197.google.com with SMTP id d75a77b69052e-420c7034fddso916111cf.0 for ; Wed, 15 Nov 2023 17:29:28 -0800 (PST) X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20230601; t=1700098168; x=1700702968; h=content-transfer-encoding:mime-version:references:in-reply-to :message-id:date:subject:cc:to:from:x-gm-message-state:from:to:cc :subject:date:message-id:reply-to; bh=KEE21en6sv5Z4IJFnP9A/1kLw7BwwEfESnJrJ/3CqcU=; b=lTcPHdDvn4xWvtXIcYczOWWDMX7FImnA4W6LVx3NBQjYrLoaXTaZHV29cZ2R1c5mR7 B6t1wSPeXj7S0iUNpYsxTsgXR3H57QfOETuFNH783flSUY7katHnFa/BCHU7THnYQd89 JxLXrsuSRz3ZuYQtJX0ZrjCsTND7zQ9xEeThJJPJJeIMf5xPpWIUo2mTXKVpV0IkdIQ5 XAZvVDv8HgIv46mjOLbfr7t9nhn5xDIbDuePTNKpG0ygC+8pbruiuaQgxEpGgzi8CZpq mcKbBgy0+cic12bDg6aBFyuMYHlm4egvXW0o6DSGUS5RznlBJPVRue5pIENXusCys2Gm t4sw== X-Gm-Message-State: AOJu0YxD3Ill8YFgqS7JZ8G1CulhWv6OPHycorD90hFyomtlbC7ctGS8 m+WYugGKprCdO3SITGN1BZTr8p3FuZZ1O5tZo9Zr0HeS2txRs6Ss9hgMNdc1fKZUsgOacxgW30l xgP1RIllv2Y0= X-Received: by 2002:ac8:5254:0:b0:421:c7ad:bb08 with SMTP id y20-20020ac85254000000b00421c7adbb08mr8123201qtn.6.1700098167769; Wed, 15 Nov 2023 17:29:27 -0800 (PST) X-Google-Smtp-Source: AGHT+IG0pY05bEW1kuLTvTDTDFSfvrB9/wLj2Toyy995bIe7hSkDprXdYQMw37m3Xz3vWwWvq8gVXw== X-Received: by 2002:ac8:5254:0:b0:421:c7ad:bb08 with SMTP id y20-20020ac85254000000b00421c7adbb08mr8123183qtn.6.1700098167489; Wed, 15 Nov 2023 17:29:27 -0800 (PST) Received: from x1n.redhat.com (cpe5c7695f3aee0-cm5c7695f3aede.cpe.net.cable.rogers.com. [99.254.144.39]) by smtp.gmail.com with ESMTPSA id c24-20020ac85198000000b0041e383d527esm3922598qtn.66.2023.11.15.17.29.26 (version=TLS1_3 cipher=TLS_AES_256_GCM_SHA384 bits=256/256); Wed, 15 Nov 2023 17:29:27 -0800 (PST) From: Peter Xu To: linux-kernel@vger.kernel.org, linux-mm@kvack.org Cc: Mike Kravetz , "Kirill A . Shutemov" , Lorenzo Stoakes , Axel Rasmussen , Matthew Wilcox , John Hubbard , Mike Rapoport , peterx@redhat.com, Hugh Dickins , David Hildenbrand , Andrea Arcangeli , Rik van Riel , James Houghton , Yang Shi , Jason Gunthorpe , Vlastimil Babka , Andrew Morton Subject: [PATCH RFC 10/12] mm/gup: Handle huge pmd for follow_pmd_mask() Date: Wed, 15 Nov 2023 20:29:06 -0500 Message-ID: <20231116012908.392077-11-peterx@redhat.com> X-Mailer: git-send-email 2.41.0 In-Reply-To: <20231116012908.392077-1-peterx@redhat.com> References: <20231116012908.392077-1-peterx@redhat.com> MIME-Version: 1.0 X-Mimecast-Spam-Score: 0 X-Mimecast-Originator: redhat.com X-Rspamd-Server: rspam08 X-Rspamd-Queue-Id: 77C168000B X-Stat-Signature: 1sukcss5y1hj4qnmea8cxgzp496nmnjb X-Rspam-User: X-HE-Tag: 1700098170-225130 X-HE-Meta: U2FsdGVkX1/fig331/bMlWb+KlDnCv2hhQZcPAyq3owQKKjcfQsUnd0xtxe5cZhPNm+XE+wvQa5mJnYX1PQ9kla8R9XZ+3KYmMhN9OWoHWMSdInMq4xYcaS5TdzOiCGGpD5xCvak4eF7mZ0fc4sC05UrzqJhhQGyqOH+jt2ysmW3KjUij9GoqxU1lpwaUZfXNpOB4JDyKl4D5i8OefTUQQKBoXmyCCO8MG+LbHiCbf4BvnmEw4v3f8oPgITmwCxGpg6F1GC4XPwXBxKyX/1EgJQ7zv93iUTqz50eishFt47Z9jilQhVr2OTjnHBumIaFVmDfjfZ1fabTFHlhlftDJeFEKASd7fCmrCSx8q6SP4xngm6kAZh3KxAYIOP0DLxJqpIzkBlwLWc5EsEbFfk0ja3OZRtaqBfpLmnEXDTviMQXIrYDbnzFexPd/iWofwMxg5xOV7fKMHl68KfFFDcBW+SF14xq5ICyNG1kAkSDbk2ZV2ug+BLfwlTmPjRWPPUMmWopV0aw/P2Ppczkj7OhBxvAuRcynxbJp1pObyOsM/RMkWxKh3au58xta+U7gUJ4IpFMTAueZY8V8mLi3gN45UjrTrvhI8ehKpDFRqgME1X/M/CvqbN3XZ/jAXHACvX4QVv8UZj+HOFPvNUfFX6LHL6GPOkWb7WqcPraX2HKCCTZrPh0a9tjo/XunDb9b1riZ6TCTdGsVy5OIMgQJ4BL88UgA7y6ISaoz0aZ50cTvldruTw0KhTSt4XIZ/+m+YRFOyUeyWVskgwSDCiyw/aeFOlwgxlEjmxET0VHqN1qY0XvXPnn/BcQLro490CdjecOJpD9wF9erpOdpDxexaQz5tvQlZHEM1vAmEktn/LdyLaiGRPc0wEdaoAJaKax30uwNRwAAn+sPGS30VHxMcPna78rIPkIU3ZUeAda1r+e+JBqHN6ahZYF89NfvnzBknrIH/BK8rFM7dAwBgy1EXz 9DATADTI 7hDusmY4KEPTquu5xAjq0gvC1LrdO49zQ4AUs4LlBdrvN/xm4aJ+EyuyZh3aoMJqFgUnbvCDh8R5HwY/VsQPr0mpm9CvIfwvRom98wDQes1VKrX3HtZXjnd2Y/0eIrSTKGRWPVeX2U1dowffS8Twt140W49BDAHvABDxGWUtghwOhDlwi+Lv4hPy9dn3pFUJMYYjS/q5PcxHtYmNqenB5g799ZbZgaVzMQewZNCUWiJaszMR5S/GbWh8RtfM7rU1rMKJ/4Fb0rPdwbiO0favE0h2JLJGvEjtgyk+WM09XbtUaPvuTuRYxedFQRgdHe8Q8MGqqQlnu8h8Lbxn2IigMEOrxiq5szi4bIhuPBS0ifvFW9J+kdCgKrS/uS5UU/mjwn2CfDZfwEHcuICSCs12IspGokmrxgAiFTIqA1U8QeU3CBzpztTb1xRHmeA== X-Bogosity: Ham, tests=bogofilter, spamicity=0.000000, version=1.2.4 Sender: owner-linux-mm@kvack.org Precedence: bulk X-Loop: owner-majordomo@kvack.org List-ID: List-Subscribe: List-Unsubscribe: Do proper replacement of pmd_trans_huge() by using pmd_thp_or_huge() to also cover pmd_huge() as long as enabled. FOLL_TOUCH and FOLL_SPLIT_PMD only apply to THP, not yet huge. Since now follow_trans_huge_pmd() can handle hugetlb pages, renaming it into follow_huge_pmd() to match what it does. When at it, move the ctx->page_mask setup into follow_huge_pmd(), only set it when the page is valid. It was not a bug to set it before even if GUP failed (page==NULL), because follow_page_mask() callers always ignores page_mask if so. But doing so makes the code cleaner. Signed-off-by: Peter Xu --- mm/gup.c | 12 ++++++------ mm/huge_memory.c | 19 ++++++++++--------- mm/internal.h | 6 +++--- 3 files changed, 19 insertions(+), 18 deletions(-) diff --git a/mm/gup.c b/mm/gup.c index 55a2ae55f00f..7c210206470f 100644 --- a/mm/gup.c +++ b/mm/gup.c @@ -713,31 +713,31 @@ static struct page *follow_pmd_mask(struct vm_area_struct *vma, spin_unlock(ptl); return page; } - if (likely(!pmd_trans_huge(pmdval))) + if (likely(!pmd_thp_or_huge(pmdval))) return follow_page_pte(vma, address, pmd, flags, &ctx->pgmap); if (pmd_protnone(pmdval) && !gup_can_follow_protnone(vma, flags)) return no_page_table(vma, flags, address); ptl = pmd_lock(mm, pmd); - if (unlikely(!pmd_present(*pmd))) { + pmdval = *pmd; + if (unlikely(!pmd_present(pmdval))) { spin_unlock(ptl); return no_page_table(vma, flags, address); } - if (unlikely(!pmd_trans_huge(*pmd))) { + if (unlikely(!pmd_thp_or_huge(pmdval))) { spin_unlock(ptl); return follow_page_pte(vma, address, pmd, flags, &ctx->pgmap); } - if (flags & FOLL_SPLIT_PMD) { + if (pmd_trans_huge(pmdval) && (flags & FOLL_SPLIT_PMD)) { spin_unlock(ptl); split_huge_pmd(vma, pmd, address); /* If pmd was left empty, stuff a page table in there quickly */ return pte_alloc(mm, pmd) ? ERR_PTR(-ENOMEM) : follow_page_pte(vma, address, pmd, flags, &ctx->pgmap); } - page = follow_trans_huge_pmd(vma, address, pmd, flags); + page = follow_huge_pmd(vma, address, pmd, flags, ctx); spin_unlock(ptl); - ctx->page_mask = HPAGE_PMD_NR - 1; return page; } diff --git a/mm/huge_memory.c b/mm/huge_memory.c index 6748ef5f3fd9..43fb81218c5e 100644 --- a/mm/huge_memory.c +++ b/mm/huge_memory.c @@ -1486,32 +1486,32 @@ static inline bool can_follow_write_pmd(pmd_t pmd, struct page *page, return !userfaultfd_huge_pmd_wp(vma, pmd); } -struct page *follow_trans_huge_pmd(struct vm_area_struct *vma, - unsigned long addr, - pmd_t *pmd, - unsigned int flags) +struct page *follow_huge_pmd(struct vm_area_struct *vma, unsigned long addr, + pmd_t *pmd, unsigned int flags, + struct follow_page_context *ctx) { struct mm_struct *mm = vma->vm_mm; + pmd_t pmdval = *pmd; struct page *page; int ret; assert_spin_locked(pmd_lockptr(mm, pmd)); - page = pmd_page(*pmd); + page = pmd_page(pmdval); VM_BUG_ON_PAGE(!PageHead(page) && !is_zone_device_page(page), page); if ((flags & FOLL_WRITE) && - !can_follow_write_pmd(*pmd, page, vma, flags)) + !can_follow_write_pmd(pmdval, page, vma, flags)) return NULL; /* Avoid dumping huge zero page */ - if ((flags & FOLL_DUMP) && is_huge_zero_pmd(*pmd)) + if ((flags & FOLL_DUMP) && is_huge_zero_pmd(pmdval)) return ERR_PTR(-EFAULT); if (pmd_protnone(*pmd) && !gup_can_follow_protnone(vma, flags)) return NULL; - if (!pmd_write(*pmd) && gup_must_unshare(vma, flags, page)) + if (!pmd_write(pmdval) && gup_must_unshare(vma, flags, page)) return ERR_PTR(-EMLINK); VM_BUG_ON_PAGE((flags & FOLL_PIN) && PageAnon(page) && @@ -1521,10 +1521,11 @@ struct page *follow_trans_huge_pmd(struct vm_area_struct *vma, if (ret) return ERR_PTR(ret); - if (flags & FOLL_TOUCH) + if (pmd_trans_huge(pmdval) && (flags & FOLL_TOUCH)) touch_pmd(vma, addr, pmd, flags & FOLL_WRITE); page += (addr & ~HPAGE_PMD_MASK) >> PAGE_SHIFT; + ctx->page_mask = HPAGE_PMD_NR - 1; VM_BUG_ON_PAGE(!PageCompound(page) && !is_zone_device_page(page), page); return page; diff --git a/mm/internal.h b/mm/internal.h index 8450562744cf..bf0dc896c274 100644 --- a/mm/internal.h +++ b/mm/internal.h @@ -1007,9 +1007,9 @@ int __must_check try_grab_page(struct page *page, unsigned int flags); /* * mm/huge_memory.c */ -struct page *follow_trans_huge_pmd(struct vm_area_struct *vma, - unsigned long addr, pmd_t *pmd, - unsigned int flags); +struct page *follow_huge_pmd(struct vm_area_struct *vma, unsigned long addr, + pmd_t *pmd, unsigned int flags, + struct follow_page_context *ctx); /* * mm/mmap.c From patchwork Thu Nov 16 01:29:07 2023 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Peter Xu X-Patchwork-Id: 13457486 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from kanga.kvack.org (kanga.kvack.org [205.233.56.17]) by smtp.lore.kernel.org (Postfix) with ESMTP id 974ADC072A2 for ; Thu, 16 Nov 2023 01:29:44 +0000 (UTC) Received: by kanga.kvack.org (Postfix) id 5A19F6B03F1; Wed, 15 Nov 2023 20:29:36 -0500 (EST) Received: by kanga.kvack.org (Postfix, from userid 40) id 511E36B03F3; Wed, 15 Nov 2023 20:29:36 -0500 (EST) X-Delivered-To: int-list-linux-mm@kvack.org Received: by kanga.kvack.org (Postfix, from userid 63042) id 306906B03F4; Wed, 15 Nov 2023 20:29:36 -0500 (EST) X-Delivered-To: linux-mm@kvack.org Received: from relay.hostedemail.com (smtprelay0012.hostedemail.com [216.40.44.12]) by kanga.kvack.org (Postfix) with ESMTP id 1987F6B03F1 for ; Wed, 15 Nov 2023 20:29:36 -0500 (EST) Received: from smtpin22.hostedemail.com (a10.router.float.18 [10.200.18.1]) by unirelay10.hostedemail.com (Postfix) with ESMTP id E699CC01D6 for ; Thu, 16 Nov 2023 01:29:35 +0000 (UTC) X-FDA: 81462085110.22.A6D2CEF Received: from us-smtp-delivery-124.mimecast.com (us-smtp-delivery-124.mimecast.com [170.10.133.124]) by imf15.hostedemail.com (Postfix) with ESMTP id D5809A0005 for ; Thu, 16 Nov 2023 01:29:33 +0000 (UTC) Authentication-Results: imf15.hostedemail.com; dkim=pass header.d=redhat.com header.s=mimecast20190719 header.b=MSVFoLe4; dmarc=pass (policy=none) header.from=redhat.com; spf=pass (imf15.hostedemail.com: domain of peterx@redhat.com designates 170.10.133.124 as permitted sender) smtp.mailfrom=peterx@redhat.com ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=hostedemail.com; s=arc-20220608; t=1700098173; h=from:from:sender:reply-to:subject:subject:date:date: message-id:message-id:to:to:cc:cc:mime-version:mime-version: content-type:content-type: content-transfer-encoding:content-transfer-encoding: in-reply-to:in-reply-to:references:references:dkim-signature; bh=ltEuMSvLIUou1PxvqED3QFklmTWz+odfLbJoVHfc4cY=; b=2b10LAAj4x5KUwQnmz0gbjjDswJmb52XSfIYsQVJ2Y58gEQQf3BSIbu36Jackh5W+KKNHi WEoDVSwOYXWQHeGbfHaX2cecIVO+hr1a30KORqYabHU8eQj0NDf4F1P4RZjOIDTBK8+uP4 utXd9K5tqsa2oajap8ThfjyKn2Hwbcw= ARC-Authentication-Results: i=1; imf15.hostedemail.com; dkim=pass header.d=redhat.com header.s=mimecast20190719 header.b=MSVFoLe4; dmarc=pass (policy=none) header.from=redhat.com; spf=pass (imf15.hostedemail.com: domain of peterx@redhat.com designates 170.10.133.124 as permitted sender) smtp.mailfrom=peterx@redhat.com ARC-Seal: i=1; s=arc-20220608; d=hostedemail.com; t=1700098173; a=rsa-sha256; cv=none; b=lu0FLmn+hgGZsNTiIKN074RM7mpO2DtUFn9Vu4jnb1aEzAV6dwSlP71/Rb7xSjHoA0kDNR TqbEuJPr55yvam3KAQa8uRr92v9RNBG7cVaFCo/ARC5IztQLOjIPT+lcfyZReU4NPUUaJj p9JGgZoIC+gcywBCUTPYX0I/F8NQmBY= DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=redhat.com; s=mimecast20190719; t=1700098173; h=from:from:reply-to:subject:subject:date:date:message-id:message-id: to:to:cc:cc:mime-version:mime-version:content-type:content-type: content-transfer-encoding:content-transfer-encoding: in-reply-to:in-reply-to:references:references; bh=ltEuMSvLIUou1PxvqED3QFklmTWz+odfLbJoVHfc4cY=; b=MSVFoLe4543+AAleX94WuN/MXL1ArtQ7HP5/l5Wg0bIFtcesqhDRGs5/I7LVPTondLkx1c 2m9s2FC8dt3lA2zIciuFxPOSoe23P5WIu+MkUc1yQilbr5P/YCzRhfbbwEMqXSisRWiIfl 472cMgd+vgiZFJpJrjW9ChQUye/mlB4= Received: from mail-vs1-f71.google.com (mail-vs1-f71.google.com [209.85.217.71]) by relay.mimecast.com with ESMTP with STARTTLS (version=TLSv1.3, cipher=TLS_AES_256_GCM_SHA384) id us-mta-136-Y70yDB4gMPCRN9jjRwrrJA-1; Wed, 15 Nov 2023 20:29:30 -0500 X-MC-Unique: Y70yDB4gMPCRN9jjRwrrJA-1 Received: by mail-vs1-f71.google.com with SMTP id ada2fe7eead31-46107bc7936so12676137.1 for ; Wed, 15 Nov 2023 17:29:30 -0800 (PST) X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20230601; t=1700098169; x=1700702969; h=content-transfer-encoding:mime-version:references:in-reply-to :message-id:date:subject:cc:to:from:x-gm-message-state:from:to:cc :subject:date:message-id:reply-to; bh=ltEuMSvLIUou1PxvqED3QFklmTWz+odfLbJoVHfc4cY=; b=VJb+rcMORmwgNNFC7mLDdp/Hxb/iFA4MfsgWSZqn7LD3ZUpTAVUCXe0uIn3gyco1MT Wkl1FBNyKhApJAhWBkgEdTYlygkyVMfe7M8mDAQo3jhc7sbMw6hgkgcx+R1n0sNXBmiK wA8Hu2BMzn645vS/l4OgSn56t7LqA0RAAjT7SFAkyBtT4bYexzcc8SU+SXhnqiUAfsoT mkbS4ys6xtAlu8SPaox/VLN6sEq1HJDA3tHoFdSt0ihnHEXvxf9va2+kHpxDndt6hEF2 z/t2H1hder18JPl9Er/gQU3DvGVMBAMybqlujXkg4FQa2Uar/CrZnTCFXg1yYWk4NcqM FKjw== X-Gm-Message-State: AOJu0YyFYkDNLvO5hOqiEeVKEXC6kLZ+dWRVSfZ8zAYQazMfFmH5KP+q 1mtZVp4SmPS2lvRwxYeL+imTwzG4uc5JfCKK++jV82RbXhbaB3Dp2SY4Oz3wXz1zePPb1moKYjR WPqmGZ6VTVErhBhUhYTk= X-Received: by 2002:a1f:1f95:0:b0:4ac:3094:fc14 with SMTP id f143-20020a1f1f95000000b004ac3094fc14mr7205521vkf.1.1700098169361; Wed, 15 Nov 2023 17:29:29 -0800 (PST) X-Google-Smtp-Source: AGHT+IFmlcK0wHn56i8+DaE6cZQcDPiXYda86sHdfw6XVjm04MvVaxCRJjJBn7Y0oU98rdXaw93h4g== X-Received: by 2002:a1f:1f95:0:b0:4ac:3094:fc14 with SMTP id f143-20020a1f1f95000000b004ac3094fc14mr7205497vkf.1.1700098169016; Wed, 15 Nov 2023 17:29:29 -0800 (PST) Received: from x1n.redhat.com (cpe5c7695f3aee0-cm5c7695f3aede.cpe.net.cable.rogers.com. [99.254.144.39]) by smtp.gmail.com with ESMTPSA id c24-20020ac85198000000b0041e383d527esm3922598qtn.66.2023.11.15.17.29.27 (version=TLS1_3 cipher=TLS_AES_256_GCM_SHA384 bits=256/256); Wed, 15 Nov 2023 17:29:28 -0800 (PST) From: Peter Xu To: linux-kernel@vger.kernel.org, linux-mm@kvack.org Cc: Mike Kravetz , "Kirill A . Shutemov" , Lorenzo Stoakes , Axel Rasmussen , Matthew Wilcox , John Hubbard , Mike Rapoport , peterx@redhat.com, Hugh Dickins , David Hildenbrand , Andrea Arcangeli , Rik van Riel , James Houghton , Yang Shi , Jason Gunthorpe , Vlastimil Babka , Andrew Morton Subject: [PATCH RFC 11/12] mm/gup: Handle hugepd for follow_page() Date: Wed, 15 Nov 2023 20:29:07 -0500 Message-ID: <20231116012908.392077-12-peterx@redhat.com> X-Mailer: git-send-email 2.41.0 In-Reply-To: <20231116012908.392077-1-peterx@redhat.com> References: <20231116012908.392077-1-peterx@redhat.com> MIME-Version: 1.0 X-Mimecast-Spam-Score: 0 X-Mimecast-Originator: redhat.com X-Rspamd-Queue-Id: D5809A0005 X-Rspam-User: X-Rspamd-Server: rspam05 X-Stat-Signature: 5k6g6wd7sz13rrm46w4op54cq5fgzwen X-HE-Tag: 1700098173-178794 X-HE-Meta: U2FsdGVkX1/XTpApeG9qY2oi0mhqodpqWeeIE27XzQm66D+CtvqHztfUk/19Uhx2Slk+uRb/LRPMTn0xhmx6w469AxfPNhlTje1udJwvc5baKJ4kk20lkAEKyYxYxKzKys2guqrfqKjp8u2Llimwm5/w1Q8NPoSyL6ZVDFH8ZPMhKUAY/esEvbeDGP/rJouJeKUf41l3f5qJv67f6vmJhYuT0RuRgD4LS78kE+gtehMx2TqQiNCBH+RteW0wK/itZPr8C9cwqCbDir0gvD1d3teMg5grKnsY91leyXmrdP+gBRxQtYO7h/RRDc5z7kPaa7VIntR5FpGCh7g/kSwOKBTYK0hFrW6wMsg269Ry0lp3mo6nwL5HYFZP/MOkARLlC7WIWbcGggVpnmdqq141VUtBPDUWadSbnivN1EzIuwOh8A7L7lQYZpvjPwCdLKqn7G/xS44nK9+ZccnrC4TSRuhdHE8Kx/HNgMIYdyJ3hek0vfdN9RFAfSLDJ1BUrKYZg0t32Iae7z8/usfsXerRkRutv7D86lvIVrd7ouqcyflMkwBysJcD7NTvPcUQtqM0EnSafsehz/BBFvUnMdxBqSdCcuGI3ZjpYtWnx2jo9X6D2f1ERIm1pBA+8VCs9WBBWG8ujWVY1q2ToH20IkziJf1vH1E0nOkSAjywVpL99bYqgylp1CRLcFGFsEKRw/pT3FeddWdD5mIAuMXTTPZ0QpIqp2pobyluNHwbdhSnZik4SrhFfwaVgkhgc6h/tCFtp4pwbJARgn8odWwYn7VMS+ju7kuy8epJTNvCa1FwpgwX8HGGHYJET0gTboY+eEMTmS9LfLWr3ka1eegwMb58cxy8G9n9XASysW5x7IiFJZVAgcD4u1ZegEBuN8cbq4b9+ZyFwHgG3D67YG4aURhjpDb/E/XEo7vkZYXEevIulKLVpVRVTTFsWiYzbCwytahz8VIpjAZB5ePDws0M0dk CNncc4HO 0t1bj5qDWunHIHqGMo1JIrGcJJ5VP25/EGgGfCPWuWhxELcWtGpszF4zyIqbY6TtHtAjqjNMpd4ylXnvJvc20rJ6c82xMv2td4TwLC+z9IVx5MvB/Dv3QsN2XJZf77b+Yvu+k4F6b/jwK0siYpT9c08Ug1ShfPx4mWyZx1OzdB6+OUntLH/QfJ05eAnq+fNZMbXnJr7DpwAR293Y0j+iJHuFOj6jp8guyLhRi93UNnHEHdNr1wRQWAk+xe0gSdX8E0OofaEHTlvqcDHpqWnOU7M5UjK+j5MyEhOU472StpnKzids7D3a4t01D9IZV4Ks2p72Vqi3DpIyTi6VvUNGTI6rxrESS1KQvdtGj6MCH+cFDDI1JAJj2x3fkCmkT7M1G2hLcE/mCrCP/F5qvkkZiV7bx8xsXRiwmbsnhelajBgvG6T6qOGXgg5EnlQ== X-Bogosity: Ham, tests=bogofilter, spamicity=0.000000, version=1.2.4 Sender: owner-linux-mm@kvack.org Precedence: bulk X-Loop: owner-majordomo@kvack.org List-ID: List-Subscribe: List-Unsubscribe: Hugepd is only used in PowerPC's hugetlbfs. follow_page_mask() used to leverage hugetlb APIs to access hugepd entries. Teach follow_page_mask() itself on hugepd. With previous refactors on fast-gup gup_huge_pd(), most of the code can be easily leveraged. Since follow_page() always only fetch one page, set the end to "address + PAGE_SIZE" should suffice. Signed-off-by: Peter Xu --- mm/gup.c | 77 ++++++++++++++++++++++++++++++++++++++++++++++++++------ 1 file changed, 69 insertions(+), 8 deletions(-) diff --git a/mm/gup.c b/mm/gup.c index 7c210206470f..e635278f65f9 100644 --- a/mm/gup.c +++ b/mm/gup.c @@ -25,6 +25,11 @@ #include "internal.h" +static struct page *follow_hugepd(struct vm_area_struct *vma, hugepd_t hugepd, + unsigned long addr, unsigned int pdshift, + unsigned int flags, + struct follow_page_context *ctx); + static inline void sanity_check_pinned_pages(struct page **pages, unsigned long npages) { @@ -713,6 +718,9 @@ static struct page *follow_pmd_mask(struct vm_area_struct *vma, spin_unlock(ptl); return page; } + if (unlikely(is_hugepd(__hugepd(pmd_val(pmdval))))) + return follow_hugepd(vma, __hugepd(pmd_val(pmdval)), + address, PMD_SHIFT, flags, ctx); if (likely(!pmd_thp_or_huge(pmdval))) return follow_page_pte(vma, address, pmd, flags, &ctx->pgmap); @@ -764,6 +772,10 @@ static struct page *follow_pud_mask(struct vm_area_struct *vma, if (unlikely(pud_bad(pud))) return no_page_table(vma, flags, address); + if (unlikely(is_hugepd(__hugepd(pud_val(pud))))) + return follow_hugepd(vma, __hugepd(pud_val(pud)), + address, PUD_SHIFT, flags, ctx); + return follow_pmd_mask(vma, address, pudp, flags, ctx); } @@ -772,15 +784,19 @@ static struct page *follow_p4d_mask(struct vm_area_struct *vma, unsigned int flags, struct follow_page_context *ctx) { - p4d_t *p4d; + p4d_t *p4d, p4dval; p4d = p4d_offset(pgdp, address); - if (p4d_none(*p4d)) - return no_page_table(vma, flags, address); - BUILD_BUG_ON(p4d_huge(*p4d)); - if (unlikely(p4d_bad(*p4d))) + p4dval = *p4d; + BUILD_BUG_ON(p4d_huge(p4dval)); + + if (p4d_none(p4dval) || unlikely(p4d_bad(p4dval))) return no_page_table(vma, flags, address); + if (unlikely(is_hugepd(__hugepd(p4d_val(p4dval))))) + return follow_hugepd(vma, __hugepd(p4d_val(p4dval)), + address, P4D_SHIFT, flags, ctx); + return follow_pud_mask(vma, address, p4d, flags, ctx); } @@ -812,7 +828,7 @@ static struct page *follow_page_mask(struct vm_area_struct *vma, unsigned long address, unsigned int flags, struct follow_page_context *ctx) { - pgd_t *pgd; + pgd_t *pgd, pgdval; struct mm_struct *mm = vma->vm_mm; ctx->page_mask = 0; @@ -827,11 +843,17 @@ static struct page *follow_page_mask(struct vm_area_struct *vma, &ctx->page_mask); pgd = pgd_offset(mm, address); + pgdval = *pgd; if (pgd_none(*pgd) || unlikely(pgd_bad(*pgd))) - return no_page_table(vma, flags, address); + page = no_page_table(vma, flags, address); + else if (unlikely(is_hugepd(__hugepd(pgd_val(pgdval))))) + page = follow_hugepd(vma, __hugepd(pgd_val(pgdval)), + address, PGDIR_SHIFT, flags, ctx); + else + page = follow_p4d_mask(vma, address, pgd, flags, ctx); - return follow_p4d_mask(vma, address, pgd, flags, ctx); + return page; } struct page *follow_page(struct vm_area_struct *vma, unsigned long address, @@ -2850,6 +2872,37 @@ static int gup_huge_pd(hugepd_t hugepd, unsigned long addr, return 1; } + +static struct page *follow_hugepd(struct vm_area_struct *vma, hugepd_t hugepd, + unsigned long addr, unsigned int pdshift, + unsigned int flags, + struct follow_page_context *ctx) +{ + struct page *page; + struct hstate *h; + spinlock_t *ptl; + int nr = 0, ret; + pte_t *ptep; + + /* Only hugetlb supports hugepd */ + if (WARN_ON_ONCE(!is_vm_hugetlb_page(vma))) + return ERR_PTR(-EFAULT); + + h = hstate_vma(vma); + ptep = hugepte_offset(hugepd, addr, pdshift); + ptl = huge_pte_lock(h, mm, ptep); + ret = gup_huge_pd(hugepd, addr, pdshift, addr + PAGE_SIZE, + flags, &page, &nr)) { + spin_unlock(ptl); + + if (ret) { + WARN_ON_ONCE(nr != 1); + ctx->page_mask = (1U << huge_page_order(h)) - 1; + return page; + } + + return NULL; +} #else static inline int gup_huge_pd(hugepd_t hugepd, unsigned long addr, unsigned int pdshift, unsigned long end, unsigned int flags, @@ -2857,6 +2910,14 @@ static inline int gup_huge_pd(hugepd_t hugepd, unsigned long addr, { return 0; } + +static struct page *follow_hugepd(struct vm_area_struct *vma, hugepd_t hugepd, + unsigned long addr, unsigned int pdshift, + unsigned int flags, + struct follow_page_context *ctx) +{ + return NULL; +} #endif /* CONFIG_ARCH_HAS_HUGEPD */ static int gup_huge_pmd(pmd_t orig, pmd_t *pmdp, unsigned long addr, From patchwork Thu Nov 16 01:29:08 2023 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Peter Xu X-Patchwork-Id: 13457487 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from kanga.kvack.org (kanga.kvack.org [205.233.56.17]) by smtp.lore.kernel.org (Postfix) with ESMTP id 5B0FBC5ACB2 for ; Thu, 16 Nov 2023 01:29:47 +0000 (UTC) Received: by kanga.kvack.org (Postfix) id 6BB6F6B03F3; Wed, 15 Nov 2023 20:29:38 -0500 (EST) Received: by kanga.kvack.org (Postfix, from userid 40) id 6171A6B03F5; Wed, 15 Nov 2023 20:29:38 -0500 (EST) X-Delivered-To: int-list-linux-mm@kvack.org Received: by kanga.kvack.org (Postfix, from userid 63042) id 43FC66B03F6; Wed, 15 Nov 2023 20:29:38 -0500 (EST) X-Delivered-To: linux-mm@kvack.org Received: from relay.hostedemail.com (smtprelay0017.hostedemail.com [216.40.44.17]) by kanga.kvack.org (Postfix) with ESMTP id 2EE186B03F3 for ; Wed, 15 Nov 2023 20:29:38 -0500 (EST) Received: from smtpin09.hostedemail.com (a10.router.float.18 [10.200.18.1]) by unirelay05.hostedemail.com (Postfix) with ESMTP id 09810402CA for ; Thu, 16 Nov 2023 01:29:38 +0000 (UTC) X-FDA: 81462085236.09.463704F Received: from us-smtp-delivery-124.mimecast.com (us-smtp-delivery-124.mimecast.com [170.10.129.124]) by imf16.hostedemail.com (Postfix) with ESMTP id D6FF6180004 for ; Thu, 16 Nov 2023 01:29:35 +0000 (UTC) Authentication-Results: imf16.hostedemail.com; dkim=pass header.d=redhat.com header.s=mimecast20190719 header.b=GcNuMlIM; spf=pass (imf16.hostedemail.com: domain of peterx@redhat.com designates 170.10.129.124 as permitted sender) smtp.mailfrom=peterx@redhat.com; dmarc=pass (policy=none) header.from=redhat.com ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=hostedemail.com; s=arc-20220608; t=1700098175; h=from:from:sender:reply-to:subject:subject:date:date: message-id:message-id:to:to:cc:cc:mime-version:mime-version: content-type:content-type: content-transfer-encoding:content-transfer-encoding: in-reply-to:in-reply-to:references:references:dkim-signature; bh=k0e00AejVp7ba16+GtnRW4DXLpdJhvAKN3LJp7S4MZU=; b=y3oQ83CkEPukgdGG/qaYWqeukz6ZRNSCX0zmBCJAKNwbAdviSCOBpGw0pVwymn6Plo6pap 5q/qjxqHOf1F2PaEcjozVvgHDjt4kqTpFJZwnY2wbslTD8A3rsVjuOLdxGFt/cXED0IWmJ HFPsBfx2XQkNA8AR8lV11XNh26pGd+0= ARC-Authentication-Results: i=1; imf16.hostedemail.com; dkim=pass header.d=redhat.com header.s=mimecast20190719 header.b=GcNuMlIM; spf=pass (imf16.hostedemail.com: domain of peterx@redhat.com designates 170.10.129.124 as permitted sender) smtp.mailfrom=peterx@redhat.com; dmarc=pass (policy=none) header.from=redhat.com ARC-Seal: i=1; s=arc-20220608; d=hostedemail.com; t=1700098175; a=rsa-sha256; cv=none; b=cyBdfP47ryDHAf87AuoxYxnAVG3o5LaQ2UGHoAR+2jZxTCCvIVGM7ScbPdPB2SP+y28cBh sd6nyXlXhwPh1kbzvtrM0lweifZlL+iyLzQO7gtnHd1V6yIKiTV67iNqq1VSdqtgBF6lVV Uk8EX9RIQZT+NldjUfpG5fuwOL28P9Q= DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=redhat.com; s=mimecast20190719; t=1700098175; h=from:from:reply-to:subject:subject:date:date:message-id:message-id: to:to:cc:cc:mime-version:mime-version:content-type:content-type: content-transfer-encoding:content-transfer-encoding: in-reply-to:in-reply-to:references:references; bh=k0e00AejVp7ba16+GtnRW4DXLpdJhvAKN3LJp7S4MZU=; b=GcNuMlIMDy0ZbF4dYiiZzR1SqgbTNuB0FhHx2OHFT1d6nhRqnvxV/FNe1IG1eoeWYImY7p /tgoJ4XX3PhXhpwuC4D8TyU46bVyNC1gHwVzl5kNk56t2HXx9eMpguP82YIqPzYZR+k1HK n0r5idvp7WwbLCirNLhmcTv6tXIe6dU= Received: from mail-vs1-f71.google.com (mail-vs1-f71.google.com [209.85.217.71]) by relay.mimecast.com with ESMTP with STARTTLS (version=TLSv1.3, cipher=TLS_AES_256_GCM_SHA384) id us-mta-389-jCoTNCXgObma1NLozEOjCA-1; Wed, 15 Nov 2023 20:29:34 -0500 X-MC-Unique: jCoTNCXgObma1NLozEOjCA-1 Received: by mail-vs1-f71.google.com with SMTP id ada2fe7eead31-45fb8f2166fso12484137.0 for ; Wed, 15 Nov 2023 17:29:33 -0800 (PST) X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20230601; t=1700098171; x=1700702971; h=content-transfer-encoding:mime-version:references:in-reply-to :message-id:date:subject:cc:to:from:x-gm-message-state:from:to:cc :subject:date:message-id:reply-to; bh=k0e00AejVp7ba16+GtnRW4DXLpdJhvAKN3LJp7S4MZU=; b=B52wZ2in5PKhS29Iic4IlIebP8SfYnxRtFEc+ncy5/SSiUD5dtVdvTNkdcC7tFji49 e4YZxvciik4nQScyiCyiqdyHNgyWk2fZZuub+eayiKusDmqJV56SU3FtDjwS9wxFPj9Q OwRKjG7EKbUrsULfVzrDdsep6cez77kuIsPPbxEETX67pkYPIxZz9k2oiHQQm0H6piwZ 2g5Z9GZPUoxAFb81O4DjilyK8btw7kx7ZuoM6OMTrYfpNxR5D5+Y/a9inHaMaYNvKim4 mz+FMZenO7Nbqz0eo7r1mtHX1VXMyqyVZkE1xTBJYtULFRPeOqndWVdM0A+wQYVW+c+E +jPg== X-Gm-Message-State: AOJu0YzJy0s8dckssDlO0LTMc6wipKI0GM5MuChZ1oRGnbhydnmqLE8Y 0ZHKqH5fjGO41KsbWnnWjoJ+/QcZiF1SZbWvPPoFRO0cLfhlmxCswKiFvo98oxRH7wldgxxFq44 Ci74244wnLIg= X-Received: by 2002:a1f:a017:0:b0:4ac:3694:7011 with SMTP id j23-20020a1fa017000000b004ac36947011mr7269393vke.0.1700098171275; Wed, 15 Nov 2023 17:29:31 -0800 (PST) X-Google-Smtp-Source: AGHT+IFewvoOFObJgHzAs8T/gdoGZhF8fsIKsBFTyxvttvo0eTXlsdk7Fy6qUthUD5DfsgEESegFFg== X-Received: by 2002:a1f:a017:0:b0:4ac:3694:7011 with SMTP id j23-20020a1fa017000000b004ac36947011mr7269371vke.0.1700098170969; Wed, 15 Nov 2023 17:29:30 -0800 (PST) Received: from x1n.redhat.com (cpe5c7695f3aee0-cm5c7695f3aede.cpe.net.cable.rogers.com. [99.254.144.39]) by smtp.gmail.com with ESMTPSA id c24-20020ac85198000000b0041e383d527esm3922598qtn.66.2023.11.15.17.29.29 (version=TLS1_3 cipher=TLS_AES_256_GCM_SHA384 bits=256/256); Wed, 15 Nov 2023 17:29:30 -0800 (PST) From: Peter Xu To: linux-kernel@vger.kernel.org, linux-mm@kvack.org Cc: Mike Kravetz , "Kirill A . Shutemov" , Lorenzo Stoakes , Axel Rasmussen , Matthew Wilcox , John Hubbard , Mike Rapoport , peterx@redhat.com, Hugh Dickins , David Hildenbrand , Andrea Arcangeli , Rik van Riel , James Houghton , Yang Shi , Jason Gunthorpe , Vlastimil Babka , Andrew Morton Subject: [PATCH RFC 12/12] mm/gup: Merge hugetlb into generic mm code Date: Wed, 15 Nov 2023 20:29:08 -0500 Message-ID: <20231116012908.392077-13-peterx@redhat.com> X-Mailer: git-send-email 2.41.0 In-Reply-To: <20231116012908.392077-1-peterx@redhat.com> References: <20231116012908.392077-1-peterx@redhat.com> MIME-Version: 1.0 X-Mimecast-Spam-Score: 0 X-Mimecast-Originator: redhat.com X-Rspamd-Queue-Id: D6FF6180004 X-Rspam-User: X-Stat-Signature: oh4x5q3terc4i5ob31wz9ddwo6f7kksr X-Rspamd-Server: rspam01 X-HE-Tag: 1700098175-22472 X-HE-Meta: U2FsdGVkX18cQdkLlJtIcebiGEpXTC8cWo0r3JAszeDx3rjVWPRw9lJgvhojFWSXitan/+UPSzMjOZ6nORHDCuGbrarA76VMc4BtRl8XR1diZlBjkR+ZXzxfzVawlgS7IC0pXZj7DRqstBlfvdpnSOtN2l5rFZijZ1BQKqH/EGX8BZWFedyNTBArstizplfeh67OTmLMd+20hrq8D8esT7npdivAhdUWMG7mBNCoUXc9Iy4Bpc04OMPY844YY3Mg9MOHqGNeCbMuVKkBZb/J4hELiu0zJhxbiqO2QPc0nB405aYzRRaJQdFXgy8xrxg9YtNiz5XrQBIac1Ih3+GwB27I6fOSvJaizLuoKyCzAR9EgzuUAXSgm4t36Jzp0LXDiDnGAlmqpeM3wtL0t8nx9ryUpE0ZQdZd7JC7zhPd5wen0zY5AEm5ak27+rDxejrGm22daJccrwwXKdoKDACfvb7QW/iT1niDp+YzUvl/ji8HjRasDzPNnki+UEpuHTdo72WV7BClcEBXArAIXTbk8T4MZg/WKWlT35MXuCOE7yUO82HojmMLmZtE+pPdqVoNgTZqnCNZIfz0X8TZ/wY6pNJa5xpGbpYwyb3Tlf1o/pLht+geFm4XzPH91V2aYSkv0EEYEe2DZmsxcDNm0yIu+udAKPAvaboHU+kLUgaLmVKK+zYgCSOzbBerHRy/vyzBAli2bGa+DIhMPCf5fcCgMUJ3k9HbWzYDm1dP6JxUAtVImRCqCoRaP6waNZK1SByHwSJmbw642k1vD44I6JEO0283R7/p0wQuPNLNIikMpHdVjagG1fNJ7X6n+kWtkmU93FVFUx6y+iT1ujJDhpHhUSIna8t2zpwCBM/IrTrwMhk50v2YwhYFIk8INgbERdswsn5NVkTIgl1ChlYWV2a00RW21Ac+QydZMh37aqM87WSyYEM7aidR5+a9ovBApuNWUlc1BOXQcwVwUy59e0v e65W7Ukw 8b80CsdpnKbFHW6OkBZxYJtGGwSUr7sABN0eO97kLl6qnFfFo9/NhK1qMKICTGXBVn/tKBIZEN/o8O7iV/6zRQvN3NmkTqgeIcgu8hs+0lEi6tZmNXoXEmLfdx1QZZ/mCbe6j7ZevWYDMQe7OyzQstAgG2I27+xYPj3nAK+40blFNmA+bg4jy79syujlmG5jF2yn9F28tu6QFk0I5bSWYGG0yDqiT9evJOh/O4x8H/hapPuKiMImI42Pk/m/AQd/uPWup2ykET16MxxnX6n91Y7ujMKKiuFjspwkPQq8I2sBD1TN1a/ZAzrAclbuRK3shJK8tm4Ts1S9EIjQVmouKj4b4rg+fkAWEvLAT3oRYlyXIuRINkwOQHL7q/VtTYnd/prTcMmBtqKz4Gpg= X-Bogosity: Ham, tests=bogofilter, spamicity=0.000000, version=1.2.4 Sender: owner-linux-mm@kvack.org Precedence: bulk X-Loop: owner-majordomo@kvack.org List-ID: List-Subscribe: List-Unsubscribe: Now follow_page() is ready to handle hugetlb pages in whatever form, and over all architectures. Switch to the generic code path. Time to retire hugetlb_follow_page_mask(), following the previous retirement of follow_hugetlb_page() in 4849807114b8. There may be a slight difference of how the loops run when processing GUP over a large hugetlb range on either ARM64 (e.g. CONT_PMD) or RISCV (mostly its Svnapot extension on 64K huge pages): each loop of __get_user_pages() will resolve one pgtable entry with the patch applied, rather than relying on the size of hugetlb hstate, the latter may cover multiple entries in one loop. However, the performance difference should hopefully not be a major concern, considering that GUP just yet got 57edfcfd3419 ("mm/gup: accelerate thp gup even for "pages != NULL""), and that's not part of a performance analysis but a side dish. If the performance will be a concern, we can consider handle CONT_PTE in follow_page(), for example. Before that is justified to be necessary, keep everything clean and simple. Signed-off-by: Peter Xu --- include/linux/hugetlb.h | 7 ---- mm/gup.c | 15 +++------ mm/hugetlb.c | 71 ----------------------------------------- 3 files changed, 5 insertions(+), 88 deletions(-) diff --git a/include/linux/hugetlb.h b/include/linux/hugetlb.h index bb07279b8991..87630a185acf 100644 --- a/include/linux/hugetlb.h +++ b/include/linux/hugetlb.h @@ -332,13 +332,6 @@ static inline void hugetlb_zap_end( { } -static inline struct page *hugetlb_follow_page_mask( - struct vm_area_struct *vma, unsigned long address, unsigned int flags, - unsigned int *page_mask) -{ - BUILD_BUG(); /* should never be compiled in if !CONFIG_HUGETLB_PAGE*/ -} - static inline int copy_hugetlb_page_range(struct mm_struct *dst, struct mm_struct *src, struct vm_area_struct *dst_vma, diff --git a/mm/gup.c b/mm/gup.c index e635278f65f9..23fcac5aa3db 100644 --- a/mm/gup.c +++ b/mm/gup.c @@ -830,18 +830,11 @@ static struct page *follow_page_mask(struct vm_area_struct *vma, { pgd_t *pgd, pgdval; struct mm_struct *mm = vma->vm_mm; + struct page *page; - ctx->page_mask = 0; - - /* - * Call hugetlb_follow_page_mask for hugetlb vmas as it will use - * special hugetlb page table walking code. This eliminates the - * need to check for hugetlb entries in the general walking code. - */ - if (is_vm_hugetlb_page(vma)) - return hugetlb_follow_page_mask(vma, address, flags, - &ctx->page_mask); + vma_pgtable_walk_begin(vma); + ctx->page_mask = 0; pgd = pgd_offset(mm, address); pgdval = *pgd; @@ -853,6 +846,8 @@ static struct page *follow_page_mask(struct vm_area_struct *vma, else page = follow_p4d_mask(vma, address, pgd, flags, ctx); + vma_pgtable_walk_end(vma); + return page; } diff --git a/mm/hugetlb.c b/mm/hugetlb.c index 29705e5c6f40..3013122a739f 100644 --- a/mm/hugetlb.c +++ b/mm/hugetlb.c @@ -6783,77 +6783,6 @@ int hugetlb_mfill_atomic_pte(pte_t *dst_pte, } #endif /* CONFIG_USERFAULTFD */ -struct page *hugetlb_follow_page_mask(struct vm_area_struct *vma, - unsigned long address, unsigned int flags, - unsigned int *page_mask) -{ - struct hstate *h = hstate_vma(vma); - struct mm_struct *mm = vma->vm_mm; - unsigned long haddr = address & huge_page_mask(h); - struct page *page = NULL; - spinlock_t *ptl; - pte_t *pte, entry; - int ret; - - hugetlb_vma_lock_read(vma); - pte = hugetlb_walk(vma, haddr, huge_page_size(h)); - if (!pte) - goto out_unlock; - - ptl = huge_pte_lock(h, mm, pte); - entry = huge_ptep_get(pte); - if (pte_present(entry)) { - page = pte_page(entry); - - if (!huge_pte_write(entry)) { - if (flags & FOLL_WRITE) { - page = NULL; - goto out; - } - - if (gup_must_unshare(vma, flags, page)) { - /* Tell the caller to do unsharing */ - page = ERR_PTR(-EMLINK); - goto out; - } - } - - page = nth_page(page, ((address & ~huge_page_mask(h)) >> PAGE_SHIFT)); - - /* - * Note that page may be a sub-page, and with vmemmap - * optimizations the page struct may be read only. - * try_grab_page() will increase the ref count on the - * head page, so this will be OK. - * - * try_grab_page() should always be able to get the page here, - * because we hold the ptl lock and have verified pte_present(). - */ - ret = try_grab_page(page, flags); - - if (WARN_ON_ONCE(ret)) { - page = ERR_PTR(ret); - goto out; - } - - *page_mask = (1U << huge_page_order(h)) - 1; - } -out: - spin_unlock(ptl); -out_unlock: - hugetlb_vma_unlock_read(vma); - - /* - * Fixup retval for dump requests: if pagecache doesn't exist, - * don't try to allocate a new page but just skip it. - */ - if (!page && (flags & FOLL_DUMP) && - !hugetlbfs_pagecache_present(h, vma, address)) - page = ERR_PTR(-EFAULT); - - return page; -} - long hugetlb_change_protection(struct vm_area_struct *vma, unsigned long address, unsigned long end, pgprot_t newprot, unsigned long cp_flags)