From patchwork Wed Mar 27 15:23:20 2024 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Peter Xu X-Patchwork-Id: 13606782 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from kanga.kvack.org (kanga.kvack.org [205.233.56.17]) by smtp.lore.kernel.org (Postfix) with ESMTP id 43B79C54E67 for ; Wed, 27 Mar 2024 15:23:47 +0000 (UTC) Received: by kanga.kvack.org (Postfix) id BFF216B008A; Wed, 27 Mar 2024 11:23:46 -0400 (EDT) Received: by kanga.kvack.org (Postfix, from userid 40) id B83386B0093; Wed, 27 Mar 2024 11:23:46 -0400 (EDT) X-Delivered-To: int-list-linux-mm@kvack.org Received: by kanga.kvack.org (Postfix, from userid 63042) id 9C9DB6B0092; Wed, 27 Mar 2024 11:23:46 -0400 (EDT) X-Delivered-To: linux-mm@kvack.org Received: from relay.hostedemail.com (smtprelay0013.hostedemail.com [216.40.44.13]) by kanga.kvack.org (Postfix) with ESMTP id 7E5156B0082 for ; Wed, 27 Mar 2024 11:23:46 -0400 (EDT) Received: from smtpin02.hostedemail.com (a10.router.float.18 [10.200.18.1]) by unirelay09.hostedemail.com (Postfix) with ESMTP id 4DA63806A5 for ; Wed, 27 Mar 2024 15:23:46 +0000 (UTC) X-FDA: 81943188852.02.0C2621A Received: from us-smtp-delivery-124.mimecast.com (us-smtp-delivery-124.mimecast.com [170.10.133.124]) by imf20.hostedemail.com (Postfix) with ESMTP id 171341C0026 for ; Wed, 27 Mar 2024 15:23:43 +0000 (UTC) Authentication-Results: imf20.hostedemail.com; dkim=pass header.d=redhat.com header.s=mimecast20190719 header.b=UN1YbEx6; spf=pass (imf20.hostedemail.com: domain of peterx@redhat.com designates 170.10.133.124 as permitted sender) smtp.mailfrom=peterx@redhat.com; dmarc=pass (policy=none) header.from=redhat.com ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=hostedemail.com; s=arc-20220608; t=1711553024; h=from:from:sender:reply-to:subject:subject:date:date: message-id:message-id:to:to:cc:cc:mime-version:mime-version: content-type:content-type: content-transfer-encoding:content-transfer-encoding: in-reply-to:in-reply-to:references:references:dkim-signature; bh=bzTkuuEUZ2f7C9YouF2ORWZwnTLSeJ8Y5AJhy7kkISQ=; b=fiOZPYWie5WMZmm3Uy+uHjw5nbmkGdLsHp+Bq9XqGvF+bGEFq9Wli5WIaAwZOxd8myIUMq npHD1jXu4n8sWaHjfqshyiv0BVvjre/3G62I7DXkFPl6r7fHLykYaiJQYK/aATsu3JbjIa ix+ts+W7CGH7ns9jbAApiuYLgMy9a0Y= ARC-Authentication-Results: i=1; imf20.hostedemail.com; dkim=pass header.d=redhat.com header.s=mimecast20190719 header.b=UN1YbEx6; spf=pass (imf20.hostedemail.com: domain of peterx@redhat.com designates 170.10.133.124 as permitted sender) smtp.mailfrom=peterx@redhat.com; dmarc=pass (policy=none) header.from=redhat.com ARC-Seal: i=1; s=arc-20220608; d=hostedemail.com; t=1711553024; a=rsa-sha256; cv=none; b=68+5twMAcOTlUexmr6jWNLOU61rGAl+QF7lvz9KRojGJn3KWIr+XKFWTUBa/LOPwHLL7D5 QUBnFnKuHRteMbFqeIvsZC5LdC5kO6/hQz2XKRXVps8mD4qmdP4M19/U40jF/mqnwvod0d gRi6l1AWsvukb//b/H0zw5yZqsUa6eM= DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=redhat.com; s=mimecast20190719; t=1711553023; h=from:from:reply-to:subject:subject:date:date:message-id:message-id: to:to:cc:cc:mime-version:mime-version:content-type:content-type: content-transfer-encoding:content-transfer-encoding: in-reply-to:in-reply-to:references:references; bh=bzTkuuEUZ2f7C9YouF2ORWZwnTLSeJ8Y5AJhy7kkISQ=; b=UN1YbEx6Jsm/AzvLT6ULyL6dHufIChZamumgV5TgOi3E7CFJpZmdhS4Tsp0W3H/C++mKGZ SoJx2cfZMfAWdNzcZytDi1MH4dsW7ZIbRoXEU4RtrDM3LPgYspfLx9osrxTRqoWFBRlTAQ SjlcSCr1A3uwU6lpq1BwEvUkweRbe3Q= Received: from mail-qv1-f71.google.com (mail-qv1-f71.google.com [209.85.219.71]) by relay.mimecast.com with ESMTP with STARTTLS (version=TLSv1.3, cipher=TLS_AES_256_GCM_SHA384) id us-mta-28-hx4n8dwsNg6mkoOV4Y4FvQ-1; Wed, 27 Mar 2024 11:23:39 -0400 X-MC-Unique: hx4n8dwsNg6mkoOV4Y4FvQ-1 Received: by mail-qv1-f71.google.com with SMTP id 6a1803df08f44-696a9c482a4so3595506d6.0 for ; Wed, 27 Mar 2024 08:23:39 -0700 (PDT) X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20230601; t=1711553018; x=1712157818; h=content-transfer-encoding:mime-version:references:in-reply-to :message-id:date:subject:cc:to:from:x-gm-message-state:from:to:cc :subject:date:message-id:reply-to; bh=bzTkuuEUZ2f7C9YouF2ORWZwnTLSeJ8Y5AJhy7kkISQ=; b=AjqA5TgEHK/8TalTIfmaoHBy6hk4wcZUiiFEUDueR1/t3KHsxILFtKQgu2V1zElVu3 lJAjCe2HYq59JiLdBjo9QJnrK1PdQcKX9PmhZQMPuJahYiHdVrulprjQHGRbcf9/V3Hk Ql9J+rDb5jnLAPY5S90upeSxS51js+ZcYzgJt6JSp/w4J/ETo0iKz2p6doUyH8zkqFXd ztV9HVuYUKS5Z1PJoMqzwDQeIWLEj6syBugQMsN2FMLotin30g06Vwmfc8iwNKn/Ti9d 2/bADi3UBrFcJXTRAB+wB7QoMF6xUuEQ7yaz4F+QCttBp7CogpsVLbbuXrGSmHU3MnXK 4evw== X-Gm-Message-State: AOJu0Yxq6iPEWAiPqhwehccTt94pDYSpKZBoVGzuFMu4DtiQkL0RUPAr QieCbw4NzrU7954RkG6g7rTxJytFpt3/loSmBbMyVe3jt2GY63pXHFz9+m99Jke3HXCL6uAFhwA YDtc6rUwz4mGjFY6J2GAb4yX0EgLcwpDmKEkVXrMzUPfKPPurX+PSmTte7YSpUm/r8bTCtJ1eKK 9W928ISG+TssTSPN1/5CsvxcpKRePPDQ== X-Received: by 2002:a05:6214:5d11:b0:690:3c85:c5b with SMTP id me17-20020a0562145d1100b006903c850c5bmr14918747qvb.3.1711553017836; Wed, 27 Mar 2024 08:23:37 -0700 (PDT) X-Google-Smtp-Source: AGHT+IFplzIHsd2VycAjj5oUNWAa5qW9RFPqJwKtRaT5aqzv2S9fkhcwA1AKc/2h1PXS7yh/3x/6aw== X-Received: by 2002:a05:6214:5d11:b0:690:3c85:c5b with SMTP id me17-20020a0562145d1100b006903c850c5bmr14918682qvb.3.1711553016992; Wed, 27 Mar 2024 08:23:36 -0700 (PDT) Received: from x1n.redhat.com ([99.254.121.117]) by smtp.gmail.com with ESMTPSA id hu4-20020a056214234400b00690dd47a41csm6412639qvb.86.2024.03.27.08.23.35 (version=TLS1_3 cipher=TLS_AES_256_GCM_SHA384 bits=256/256); Wed, 27 Mar 2024 08:23:36 -0700 (PDT) From: peterx@redhat.com To: linux-mm@kvack.org, linux-kernel@vger.kernel.org Cc: Yang Shi , "Kirill A . Shutemov" , Mike Kravetz , John Hubbard , Michael Ellerman , peterx@redhat.com, Andrew Jones , Muchun Song , linux-riscv@lists.infradead.org, linuxppc-dev@lists.ozlabs.org, Christophe Leroy , Andrew Morton , Christoph Hellwig , Lorenzo Stoakes , Matthew Wilcox , Rik van Riel , linux-arm-kernel@lists.infradead.org, Andrea Arcangeli , David Hildenbrand , "Aneesh Kumar K . V" , Vlastimil Babka , James Houghton , Jason Gunthorpe , Mike Rapoport , Axel Rasmussen Subject: [PATCH v4 01/13] mm/Kconfig: CONFIG_PGTABLE_HAS_HUGE_LEAVES Date: Wed, 27 Mar 2024 11:23:20 -0400 Message-ID: <20240327152332.950956-2-peterx@redhat.com> X-Mailer: git-send-email 2.44.0 In-Reply-To: <20240327152332.950956-1-peterx@redhat.com> References: <20240327152332.950956-1-peterx@redhat.com> MIME-Version: 1.0 X-Mimecast-Spam-Score: 0 X-Mimecast-Originator: redhat.com X-Rspamd-Queue-Id: 171341C0026 X-Rspam-User: X-Stat-Signature: f9mzahnpbdwwuseqizndppdfnj3w51nb X-Rspamd-Server: rspam01 X-HE-Tag: 1711553023-944894 X-HE-Meta: U2FsdGVkX18aNLFef+fdUpQ1ihvPLcPB434YC1OWVJdnow8VukYDvN9Lj8BNqHgNWL/9PJVOUx7VW5acHwZDvbyJs5QG07nclOqpIbU+3cRy+oa4ROH2GO4IpV48QawIeh6KkafC/FFU40XTaPrfTrIeOMhJh1q2DZBiPbvW0UYMWO61ip0vujqProxwu/h+p4Q3JvF/IOZu50/2RJEF6168W0asrbhNHHb8feS1MQ0URGedXjTE/sFJWCjoVMtry28in/lpZJ1aryFtFfDAwoKJymbltNEJcgHDvwPCte6r+Thv3x1hTO64heRDLlhie5pzMA/qp8jemVpcvM4GFERYAFTLPbwM1VLPamuS/sbFOq5BZ1wtjS5ME48qdcnuJZlab8rZk5hIOJsvQOQHKRWK1HNxP5e1HMSU5InCH3WRbamFqIX4qXPREVswJoKd7oYLVFfE4ZAYu3mrmP166hpqxKXEPaduIU9UsvZjgSy9sUAH/bqzIdgekOkG1RsJq9pIaPJD6EWW3bGuypbnCvPQHTd0fJB5CqLFn2+eH1+UJ11sLCZss4RF+FnK99KYwDtwYhnaJ1V6s6pnDdjRJ+6F20bpH6N4PzFFdhQYwjBZk0wp0xB2lM/3mlmhN46CQiQGHPpLekpd362PkNCOz3HpjuytFt094yoaqU5weKkYk8dul57vLe9X1/MxrLKorCChhLHBauiLR66V/h7rX+sXdaIMa/JhlvmMAqNmUa4nqKVe/NeG1ktattilIQtQsOzLP1DlmhLNNOizdcOIN/K5CgHqU4mzprZWyxsX9HmBef9BrREyowsOfsFi63Z5t827eADs2ZO0rAMQRBRDfqQ1p4kakfh3TSuMAN9RIqzU2HM4GSneHM/9BKnq4WfhTCJ4JRBueXIn8JcF8zG1Jvy/8EyXHTpVZNu8hhEfZOYMnuczj54pejEwd+jv6xfXD4gMYb3lg+XIryNuUJP kFJFpr/H PTsUW6DQd2wBX6VJATYrqvJcUDUvRCOOsBpM5gH9REtRBcn0WKe0B99OeMpM8TRM19cIfM49GzN6sTToZc4RBrjGoAAfQ6eztUjpKKfCaY9AARlfehzRy7XlarTg73rRcmdzWDfqYk2gEeztnctNwpa6ZXH8Yb37pdA1prWVIsK8jVXRWVOuyv6JsUxjF2G3jQ3nrX94/5cAGIZnZx5761UjEAXX2nVB1nQy936NJKY+jJG1LceKYIcLN5zKS6hs8i0BygQmsmGpgW/MOKa7LxDzqK7cvBqkV9Jph0lBwH/8XwT1+puvkhfvRDTDDXhu9bV0FiBKgkKsLqiEaAGtTpVWoYPLXp++L9CKY5P1cE+3KMtk1evadjsaAan/C31wbwy7HlFBgw4Rn19akhO3ulki2/wE/EKecgSh8BAOVVeO1lvEnJ11OTcvAVh8MWMLUEGX0mJqpZ3o7ep8= X-Bogosity: Ham, tests=bogofilter, spamicity=0.000000, version=1.2.4 Sender: owner-linux-mm@kvack.org Precedence: bulk X-Loop: owner-majordomo@kvack.org List-ID: List-Subscribe: List-Unsubscribe: From: Peter Xu Introduce a config option that will be selected as long as huge leaves are involved in pgtable (thp or hugetlbfs). It would be useful to mark any code with this new config that can process either hugetlb or thp pages in any level that is higher than pte level. Reviewed-by: Jason Gunthorpe Signed-off-by: Peter Xu --- mm/Kconfig | 6 ++++++ 1 file changed, 6 insertions(+) diff --git a/mm/Kconfig b/mm/Kconfig index b924f4a5a3ef..497cdf4d8ebf 100644 --- a/mm/Kconfig +++ b/mm/Kconfig @@ -850,6 +850,12 @@ config READ_ONLY_THP_FOR_FS endif # TRANSPARENT_HUGEPAGE +# +# The architecture supports pgtable leaves that is larger than PAGE_SIZE +# +config PGTABLE_HAS_HUGE_LEAVES + def_bool TRANSPARENT_HUGEPAGE || HUGETLB_PAGE + # # UP and nommu archs use km based percpu allocator # From patchwork Wed Mar 27 15:23:21 2024 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Peter Xu X-Patchwork-Id: 13606783 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from kanga.kvack.org (kanga.kvack.org [205.233.56.17]) by smtp.lore.kernel.org (Postfix) with ESMTP id 8BA31C47DD9 for ; Wed, 27 Mar 2024 15:23:48 +0000 (UTC) Received: by kanga.kvack.org (Postfix) id 11B246B0082; Wed, 27 Mar 2024 11:23:47 -0400 (EDT) Received: by kanga.kvack.org (Postfix, from userid 40) id 0C8966B0092; Wed, 27 Mar 2024 11:23:47 -0400 (EDT) X-Delivered-To: int-list-linux-mm@kvack.org Received: by kanga.kvack.org (Postfix, from userid 63042) id DF3486B0096; Wed, 27 Mar 2024 11:23:46 -0400 (EDT) X-Delivered-To: linux-mm@kvack.org Received: from relay.hostedemail.com (smtprelay0016.hostedemail.com [216.40.44.16]) by kanga.kvack.org (Postfix) with ESMTP id A90EC6B0082 for ; Wed, 27 Mar 2024 11:23:46 -0400 (EDT) Received: from smtpin08.hostedemail.com (a10.router.float.18 [10.200.18.1]) by unirelay08.hostedemail.com (Postfix) with ESMTP id 69F5C140DD7 for ; Wed, 27 Mar 2024 15:23:46 +0000 (UTC) X-FDA: 81943188852.08.4406278 Received: from us-smtp-delivery-124.mimecast.com (us-smtp-delivery-124.mimecast.com [170.10.129.124]) by imf27.hostedemail.com (Postfix) with ESMTP id 5FBC340003 for ; Wed, 27 Mar 2024 15:23:44 +0000 (UTC) Authentication-Results: imf27.hostedemail.com; dkim=pass header.d=redhat.com header.s=mimecast20190719 header.b=XlUc7hCJ; dmarc=pass (policy=none) header.from=redhat.com; spf=pass (imf27.hostedemail.com: domain of peterx@redhat.com designates 170.10.129.124 as permitted sender) smtp.mailfrom=peterx@redhat.com ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=hostedemail.com; s=arc-20220608; t=1711553024; h=from:from:sender:reply-to:subject:subject:date:date: message-id:message-id:to:to:cc:cc:mime-version:mime-version: content-type:content-type: content-transfer-encoding:content-transfer-encoding: in-reply-to:in-reply-to:references:references:dkim-signature; bh=/DKy2rskdC2Yvy4b+7pXCefd1yMSUUO/lg/gxYCRDGA=; b=UJXnegEp9WfcNyx7mcSM1KsqMVbUQsFyRH1/U2M9JbR/w2FuCDWAgMA+kpC2kOnM44Hxsn v/f0fpv31mUv0hWGV8Sew7zWStEu/oaX9LDMrs3Yw33LSqmPXRljcPR9HeQLPu2Fifyyn3 naLvHXt34YqSZMbk3qjx1wmJEF75ilk= ARC-Authentication-Results: i=1; imf27.hostedemail.com; dkim=pass header.d=redhat.com header.s=mimecast20190719 header.b=XlUc7hCJ; dmarc=pass (policy=none) header.from=redhat.com; spf=pass (imf27.hostedemail.com: domain of peterx@redhat.com designates 170.10.129.124 as permitted sender) smtp.mailfrom=peterx@redhat.com ARC-Seal: i=1; s=arc-20220608; d=hostedemail.com; t=1711553024; a=rsa-sha256; cv=none; b=I7giWj+5ywt80eBy0w/T0GW19bITlpoWJJEs7Y30NIFecHRBXTch8QQTS8o4bxaivxo+SC jSFIHc00ah+SzKEBujz03lzr8nLrHQOnj3qIPv0wGerTLx6FIEMe+NfXLf0thARpiW6g4Y Hfo1vo2jv64piq4DjkH9ePxpNsq5jbA= DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=redhat.com; s=mimecast20190719; t=1711553023; h=from:from:reply-to:subject:subject:date:date:message-id:message-id: to:to:cc:cc:mime-version:mime-version:content-type:content-type: content-transfer-encoding:content-transfer-encoding: in-reply-to:in-reply-to:references:references; bh=/DKy2rskdC2Yvy4b+7pXCefd1yMSUUO/lg/gxYCRDGA=; b=XlUc7hCJlcVqjBmKtdtMMEJJKeTdwY7QKS4DuNNHANJzCg/A8NHvCcm5PLfbl8xwOjmthD OcGdI/91Qt0YEaRWlkIxuUdpJoThP4ExCF0L73NG/mltslt/KY0IiEnbU2Iuu2NaZWMehV PaY2JzPbCJBPSMPMpMtLQKrLEmE0awU= Received: from mail-oi1-f198.google.com (mail-oi1-f198.google.com [209.85.167.198]) by relay.mimecast.com with ESMTP with STARTTLS (version=TLSv1.3, cipher=TLS_AES_256_GCM_SHA384) id us-mta-527-S2JESGfUNS-rExgcMtnFvQ-1; Wed, 27 Mar 2024 11:23:41 -0400 X-MC-Unique: S2JESGfUNS-rExgcMtnFvQ-1 Received: by mail-oi1-f198.google.com with SMTP id 5614622812f47-3c377528ca3so585634b6e.1 for ; Wed, 27 Mar 2024 08:23:41 -0700 (PDT) X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20230601; t=1711553020; x=1712157820; h=content-transfer-encoding:mime-version:references:in-reply-to :message-id:date:subject:cc:to:from:x-gm-message-state:from:to:cc :subject:date:message-id:reply-to; bh=/DKy2rskdC2Yvy4b+7pXCefd1yMSUUO/lg/gxYCRDGA=; b=mXkyfApmueHup13MeaYYmAw3lCgVDBrwbBr9lys5e6ARZHpmcypnGtW+7gAlftHBc6 In0Qe8FzmErWAxoYRmoUOBNfzbmF3JpakD+q3ysVikRrv1g2JRGjyde616MglLkTjbxQ mJZekGIKZ0r/7B9wP+54bPt+1FHYUIe5FEyUQAh8OdFp/NGKGHhqSFmLlDFDkKooO1MW dv4ylAf+jBJ3W7PSivGX4mrbfYQIUOpw7Cwp56T869vBMN0JXytFMxMcA+6EIoxPArA3 IRGO3AWQbaqkfrxAjJWyQBYUzBSlN1/OgP4fmslimPPlI95aPknHgFaKnwEgxAUkP2bK EpZA== X-Gm-Message-State: AOJu0YzwSZbJeKHcAp+sEb2cu2qWIiouYXiioqYux78+18p01BvzjbOR CmcfVkoKSL4Rjw+SxKEC38SMhgyLNhHwCjqI8OrsOqigsWIZJeXuYZYQtBp+nJ9/8ysY3F/8ICs cJP5bfvZGZxC/IUiC3VAATq+6qJ3R3vfsct+Smv2MbrNhJIoWmEEnThI3qAc79T5VBPI5avay4l 1xe0Kg+HmIvM1fb/Z9waLzehXaLHCAkA== X-Received: by 2002:a05:6808:1814:b0:3c3:d28a:b1a8 with SMTP id bh20-20020a056808181400b003c3d28ab1a8mr370375oib.0.1711553020238; Wed, 27 Mar 2024 08:23:40 -0700 (PDT) X-Google-Smtp-Source: AGHT+IEUSCRMAKzeIW51sHYtQ4kGfBJ1LCcy7xm5bPm4kr5W7dAQlFQDGhJMDaaF/OcaFnro9iqkpQ== X-Received: by 2002:a05:6808:1814:b0:3c3:d28a:b1a8 with SMTP id bh20-20020a056808181400b003c3d28ab1a8mr370311oib.0.1711553019443; Wed, 27 Mar 2024 08:23:39 -0700 (PDT) Received: from x1n.redhat.com ([99.254.121.117]) by smtp.gmail.com with ESMTPSA id hu4-20020a056214234400b00690dd47a41csm6412639qvb.86.2024.03.27.08.23.37 (version=TLS1_3 cipher=TLS_AES_256_GCM_SHA384 bits=256/256); Wed, 27 Mar 2024 08:23:38 -0700 (PDT) From: peterx@redhat.com To: linux-mm@kvack.org, linux-kernel@vger.kernel.org Cc: Yang Shi , "Kirill A . Shutemov" , Mike Kravetz , John Hubbard , Michael Ellerman , peterx@redhat.com, Andrew Jones , Muchun Song , linux-riscv@lists.infradead.org, linuxppc-dev@lists.ozlabs.org, Christophe Leroy , Andrew Morton , Christoph Hellwig , Lorenzo Stoakes , Matthew Wilcox , Rik van Riel , linux-arm-kernel@lists.infradead.org, Andrea Arcangeli , David Hildenbrand , "Aneesh Kumar K . V" , Vlastimil Babka , James Houghton , Jason Gunthorpe , Mike Rapoport , Axel Rasmussen Subject: [PATCH v4 02/13] mm/hugetlb: Declare hugetlbfs_pagecache_present() non-static Date: Wed, 27 Mar 2024 11:23:21 -0400 Message-ID: <20240327152332.950956-3-peterx@redhat.com> X-Mailer: git-send-email 2.44.0 In-Reply-To: <20240327152332.950956-1-peterx@redhat.com> References: <20240327152332.950956-1-peterx@redhat.com> MIME-Version: 1.0 X-Mimecast-Spam-Score: 0 X-Mimecast-Originator: redhat.com X-Rspamd-Queue-Id: 5FBC340003 X-Rspam-User: X-Rspamd-Server: rspam04 X-Stat-Signature: xnfa93qep5j5t19cxpphjax6qk9o1tj5 X-HE-Tag: 1711553024-826 X-HE-Meta: U2FsdGVkX19QqPTNqr3+xxj7bpKbJLVmJd5VQSX76jGfHVJXcgH4TkbAVg7DZLAxxYhjRYPUsUd6FO7bddxZw7Q4M664HCLI0lF8qBXcAu0639rGTqDz6n4DRJmQo/i9qMlkMIEzdlRZG/kq/9r36plo+xHxu1QHj54OqkrGJncJXgUxny6gopb0RA1isnWIm7+NbQAycr2xrUTiKyIa2v5m375kuWfLJrNeMeKmQcULsDuquboTMSSXnPkHFN4gQV1wR98ECpKuaXYNspyO0QSJxF9lE0fxhvOno1N14+NQNjeI4hDzozuwjbN6MAa0mUfhzUuHK1cpRF4uQR9IXkEvEomJvCFzb+v86Z5wZ2/Pl74NJ4oWRQRVhJ2neMsOVg8/jtNlJDd1KWYVuEw0PaWUqf2iPHi+85yLwrzEzKPTZNsq1eayyEflo0ncZDyVDe706dVnEdlLnTBEtINe7puGDWw+41Fgf+XvqJu5cAiQgCOcPURanLYLOItKyk+dTBridbaiSp+SjyQ+6PIhZ6Z0qYcwqLLxfWkkwCV59IZBR/v84ByaKtMUocryvXdWjLNzcujo/NeX2wak9ksbKqwO43Zvci1yTqym84HtxwasiFRBiPTBzlcEJyC5NILv9DJxdtLUDt3gQmoMQnQTKWE0oyhd8IJCPgnvaz451srt5bgzoXsBBxGvlNSqUE3fKDke9h/14ajs+f56yuSndDdI8K4KxRm9otTPIaq3NIIBjmM+PSfRPk4p9aUUTyVBATVlxk9uVph9F2RBH160MXznrkm4GviweytxZcYw1HZpC5ILW9ib4q9SFXFkXosvD4Puij1U2mbjhIw22W+vhWOudYhh5DTRXDMR9Qmiq0oA49ommSOiAyolMm0eD2PPkxgJx/VWYejfUb6bH9trETQ+T/VbL5hLzBzvXLroHfM/EeMiSGuvy+f4RiwKSVROVA1uOFMR7SiuqSUBW4+ dz+hmGT0 30vPDxYBnsKdhWM1xaY9JX8Zm7ChrXuKREsbupft1KWt4L7QRP5eX5Ol5Ml/JuRhtil5ghFMBSuF963ErpfYYMZ+Y2M0gwq6OZFGGsA+cozzAbUwtM+SG7HCycgqjPBJrxfS7nL3Fgefg4kFD9JQTylNH7gmbuiEQS0n5IHY0NyjhAO4tlWNiiy6KPnnGLDgCPIauxiYIaoW1fdNeUsX+JNxx7ovZYujd1lRonSjXfyDYv1XCHX0BN45cqIwe19kMX6MD5HYmfHNRICBn4lRGdMFhdNTark1qwdMPup7Ar6sK7qG4e1ZPRTjLBXKNqhvPw81tPsVJYrkZ3/hLkaJp12s1opBnwoJQ1kjc8+bbBZUxkoZ21LqII9PLuwSE5BW+KrZvGsdCpVyyoaJahVwa6xCB+G7vJBe1Trz0p2M97p3siBxwVgw3x6MYlw== X-Bogosity: Ham, tests=bogofilter, spamicity=0.000000, version=1.2.4 Sender: owner-linux-mm@kvack.org Precedence: bulk X-Loop: owner-majordomo@kvack.org List-ID: List-Subscribe: List-Unsubscribe: From: Peter Xu It will be used outside hugetlb.c soon. Signed-off-by: Peter Xu --- include/linux/hugetlb.h | 9 +++++++++ mm/hugetlb.c | 4 ++-- 2 files changed, 11 insertions(+), 2 deletions(-) diff --git a/include/linux/hugetlb.h b/include/linux/hugetlb.h index d748628efc5e..294c78b3549f 100644 --- a/include/linux/hugetlb.h +++ b/include/linux/hugetlb.h @@ -174,6 +174,9 @@ u32 hugetlb_fault_mutex_hash(struct address_space *mapping, pgoff_t idx); pte_t *huge_pmd_share(struct mm_struct *mm, struct vm_area_struct *vma, unsigned long addr, pud_t *pud); +bool hugetlbfs_pagecache_present(struct hstate *h, + struct vm_area_struct *vma, + unsigned long address); struct address_space *hugetlb_page_mapping_lock_write(struct page *hpage); @@ -1228,6 +1231,12 @@ static inline void hugetlb_register_node(struct node *node) static inline void hugetlb_unregister_node(struct node *node) { } + +static inline bool hugetlbfs_pagecache_present( + struct hstate *h, struct vm_area_struct *vma, unsigned long address) +{ + return false; +} #endif /* CONFIG_HUGETLB_PAGE */ static inline spinlock_t *huge_pte_lock(struct hstate *h, diff --git a/mm/hugetlb.c b/mm/hugetlb.c index f9640a81226e..65b9c9a48fd2 100644 --- a/mm/hugetlb.c +++ b/mm/hugetlb.c @@ -6110,8 +6110,8 @@ static vm_fault_t hugetlb_wp(struct mm_struct *mm, struct vm_area_struct *vma, /* * Return whether there is a pagecache page to back given address within VMA. */ -static bool hugetlbfs_pagecache_present(struct hstate *h, - struct vm_area_struct *vma, unsigned long address) +bool hugetlbfs_pagecache_present(struct hstate *h, + struct vm_area_struct *vma, unsigned long address) { struct address_space *mapping = vma->vm_file->f_mapping; pgoff_t idx = linear_page_index(vma, address); From patchwork Wed Mar 27 15:23:22 2024 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Peter Xu X-Patchwork-Id: 13606784 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from kanga.kvack.org (kanga.kvack.org [205.233.56.17]) by smtp.lore.kernel.org (Postfix) with ESMTP id 01195C54E67 for ; Wed, 27 Mar 2024 15:23:50 +0000 (UTC) Received: by kanga.kvack.org (Postfix) id 49BF56B0093; Wed, 27 Mar 2024 11:23:48 -0400 (EDT) Received: by kanga.kvack.org (Postfix, from userid 40) id 44BA36B0098; Wed, 27 Mar 2024 11:23:48 -0400 (EDT) X-Delivered-To: int-list-linux-mm@kvack.org Received: by kanga.kvack.org (Postfix, from userid 63042) id 202256B0096; Wed, 27 Mar 2024 11:23:48 -0400 (EDT) X-Delivered-To: linux-mm@kvack.org Received: from relay.hostedemail.com (smtprelay0013.hostedemail.com [216.40.44.13]) by kanga.kvack.org (Postfix) with ESMTP id 034A06B0092 for ; Wed, 27 Mar 2024 11:23:47 -0400 (EDT) Received: from smtpin06.hostedemail.com (a10.router.float.18 [10.200.18.1]) by unirelay07.hostedemail.com (Postfix) with ESMTP id C9AFC16072A for ; Wed, 27 Mar 2024 15:23:47 +0000 (UTC) X-FDA: 81943188894.06.1F89E5F Received: from us-smtp-delivery-124.mimecast.com (us-smtp-delivery-124.mimecast.com [170.10.129.124]) by imf12.hostedemail.com (Postfix) with ESMTP id 9A1884002D for ; Wed, 27 Mar 2024 15:23:45 +0000 (UTC) Authentication-Results: imf12.hostedemail.com; dkim=pass header.d=redhat.com header.s=mimecast20190719 header.b=bkvBedbz; spf=pass (imf12.hostedemail.com: domain of peterx@redhat.com designates 170.10.129.124 as permitted sender) smtp.mailfrom=peterx@redhat.com; dmarc=pass (policy=none) header.from=redhat.com ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=hostedemail.com; s=arc-20220608; t=1711553025; h=from:from:sender:reply-to:subject:subject:date:date: message-id:message-id:to:to:cc:cc:mime-version:mime-version: content-type:content-type: content-transfer-encoding:content-transfer-encoding: in-reply-to:in-reply-to:references:references:dkim-signature; bh=/g69WjpU0rXeMHNQq0wHJ20ikWbVhFRACidICzEeTCE=; b=rfWmkqWKxeABA8KgS9UD5ugHuVX6tvxkwlx83TiZ0K9FNn0odSLmHsx065QdVsU8dgUTli U4RIa0qw7PAZ7Mn4n8LYxaDdAN7hnprGmSuY0SFcjNrB9Bu81gv6fdgCnwKz1r+RJ1Ogrl LGqVltmsBSe1Ni+Lx7ZOrxo1htfEBM0= ARC-Authentication-Results: i=1; imf12.hostedemail.com; dkim=pass header.d=redhat.com header.s=mimecast20190719 header.b=bkvBedbz; spf=pass (imf12.hostedemail.com: domain of peterx@redhat.com designates 170.10.129.124 as permitted sender) smtp.mailfrom=peterx@redhat.com; dmarc=pass (policy=none) header.from=redhat.com ARC-Seal: i=1; s=arc-20220608; d=hostedemail.com; t=1711553025; a=rsa-sha256; cv=none; b=Q3xyewurb6JH85FMlyRgjccgb/027JHaKhIesdbH4b4b6rFSqp5J8xSXLFp9WOimm3c2Ls 7cf6eYQBAiIxJlhWnxeRNjYTTG5HWjdPJzfq8nHhXy/TeKdaSiZx9H8uAx3Chq1nA+dAp/ 7iO3XnibFuG06ImTQa50QhuaIbWTFVk= DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=redhat.com; s=mimecast20190719; t=1711553024; h=from:from:reply-to:subject:subject:date:date:message-id:message-id: to:to:cc:cc:mime-version:mime-version:content-type:content-type: content-transfer-encoding:content-transfer-encoding: in-reply-to:in-reply-to:references:references; bh=/g69WjpU0rXeMHNQq0wHJ20ikWbVhFRACidICzEeTCE=; b=bkvBedbze8RVNVsEW+6xD5cM6TwuRhHYlCAAPcieiEM5mVWDPZ/MUSSoZ8et5+FudChwbr ll8o6EhgzahcYrgnrDilfqGB8eYJn6ogSYlPOLu6QF06a32wzXjE5AWzt5YD7LGKMXIRWl rFOHvUuSH01eEIpR8piDboLY6uhQGGs= Received: from mail-qv1-f72.google.com (mail-qv1-f72.google.com [209.85.219.72]) by relay.mimecast.com with ESMTP with STARTTLS (version=TLSv1.3, cipher=TLS_AES_256_GCM_SHA384) id us-mta-381-SGdhaBABMwCtlOLzqu5r0A-1; Wed, 27 Mar 2024 11:23:43 -0400 X-MC-Unique: SGdhaBABMwCtlOLzqu5r0A-1 Received: by mail-qv1-f72.google.com with SMTP id 6a1803df08f44-69672754479so14006336d6.1 for ; Wed, 27 Mar 2024 08:23:43 -0700 (PDT) X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20230601; t=1711553022; x=1712157822; h=content-transfer-encoding:mime-version:references:in-reply-to :message-id:date:subject:cc:to:from:x-gm-message-state:from:to:cc :subject:date:message-id:reply-to; bh=/g69WjpU0rXeMHNQq0wHJ20ikWbVhFRACidICzEeTCE=; b=RgJqATNSVBqwLXej0+bUvKa90YmvWkI4vO45aB4iAM8ke9yuraAhNtJ/yBxhlHY2fZ 2vtlP1EPXUyw/E3sCKXKdOlxYi1EhVSnbTtGtj7iBAdtwXbZ0mHaprPFRqUNoKU/K3Py NLK+qHZMF4JQHFj1liPjLfWQxellqv0Ly7yIlBDpqZz9X+cgHZoIGRmoyU3TrAMlaMIo 7wIZ4I5cEF2ls42nItxw1yZjnqorDCTTKVYRPCAW5RkB4o6VpmIK88H3iVEhxAUlvTz4 mYejHbuFsktG9Thsepqx4AxJrvxqK/bNByCmKH9tAXLZ8I7fckgrUfXW3SIEVbgJ8M53 8sEg== X-Gm-Message-State: AOJu0Yw1AoUXZaqods7tzTL4bXji9lO1IzOFzQnKoAv78zCNWSzIIkAD 8VuJxkesXQNbLGWv/HwJB2E8UeciXDazBD8P0giFruDG4h25UQzFKu+ndadPKY4WcL5QbCRkyo5 OVayR8Z7c+rw7vN5IYWHMt801V/P+24BOPmhUospgNymlEU1TXT1unnSvcJ21eMBOkYcAwMVJgC TnTTfqC2oITY2jyVneXW5odpDet6oOhg== X-Received: by 2002:a05:6214:3187:b0:691:456f:415a with SMTP id lb7-20020a056214318700b00691456f415amr14772484qvb.4.1711553022332; Wed, 27 Mar 2024 08:23:42 -0700 (PDT) X-Google-Smtp-Source: AGHT+IFwFABTpt0oYC5XP41rgqaavJwjTuxekGLogedD16Vy17R/DxfwS2XqF10qrZ+lSq8ZmFvRzQ== X-Received: by 2002:a05:6214:3187:b0:691:456f:415a with SMTP id lb7-20020a056214318700b00691456f415amr14772425qvb.4.1711553021719; Wed, 27 Mar 2024 08:23:41 -0700 (PDT) Received: from x1n.redhat.com ([99.254.121.117]) by smtp.gmail.com with ESMTPSA id hu4-20020a056214234400b00690dd47a41csm6412639qvb.86.2024.03.27.08.23.39 (version=TLS1_3 cipher=TLS_AES_256_GCM_SHA384 bits=256/256); Wed, 27 Mar 2024 08:23:40 -0700 (PDT) From: peterx@redhat.com To: linux-mm@kvack.org, linux-kernel@vger.kernel.org Cc: Yang Shi , "Kirill A . Shutemov" , Mike Kravetz , John Hubbard , Michael Ellerman , peterx@redhat.com, Andrew Jones , Muchun Song , linux-riscv@lists.infradead.org, linuxppc-dev@lists.ozlabs.org, Christophe Leroy , Andrew Morton , Christoph Hellwig , Lorenzo Stoakes , Matthew Wilcox , Rik van Riel , linux-arm-kernel@lists.infradead.org, Andrea Arcangeli , David Hildenbrand , "Aneesh Kumar K . V" , Vlastimil Babka , James Houghton , Jason Gunthorpe , Mike Rapoport , Axel Rasmussen Subject: [PATCH v4 03/13] mm: Make HPAGE_PXD_* macros even if !THP Date: Wed, 27 Mar 2024 11:23:22 -0400 Message-ID: <20240327152332.950956-4-peterx@redhat.com> X-Mailer: git-send-email 2.44.0 In-Reply-To: <20240327152332.950956-1-peterx@redhat.com> References: <20240327152332.950956-1-peterx@redhat.com> MIME-Version: 1.0 X-Mimecast-Spam-Score: 0 X-Mimecast-Originator: redhat.com X-Rspamd-Queue-Id: 9A1884002D X-Rspam-User: X-Stat-Signature: ubgfzgpee6du359hezjtxxny5edjuk5d X-Rspamd-Server: rspam01 X-HE-Tag: 1711553025-904328 X-HE-Meta: U2FsdGVkX1+9dOn8v//FQ+Ma/kI5i32J+jYpVBLnjV/KgOl9nNi0y5TX6Rf2nxzP4rxbTsyTV7YV5JJkAPjwbKb72btN3HFB4JXuw7tZAGSRDB5sROE+RCSicS8Y9NwE3Bctqs3kTvBMNooo8jf8ALzMMDa8u12NXZfSM4ByBsnLfJowhSN6uqLbC8DBgKvHD2hOJdxGBsJEJ156mWQq6en139CGSZhmTFlnvNjH+bBNS7TupXExjixpF/xSg++FyRrY4PLMgmesqOLSVp3wwzu0TISENh/jSkiZ6tOOM6Cb26XqzkYh93/Ep2tDw44GX7lPxuucRBN5E2P9DxjhTuCt3Rmo7ZRZHLImfHe1NENc8160bN8VhRWIebIFdVkY3x65r09F0ZDzoboi8eK8P2mLAlHvSs2eaP8mwjh5U/9Ft7Har/PXyIDfI6QWTc+yhqIJXxdrkvGoia7aOfj5UIIyheksCxqiUO7bazTvcMiU80O5w8F59QW08t2qYpH2MIX3K9I92bc3fGtD/5Z5XWqxAA3XDMT/s0Fwmv00555knpsS4ReM0JXWGr5HrCf9rPXv1RVn416FYDcE1lU4g51ZnrZrwa2wWiAd9yv7dZKvBPlIz3QnP3FHEiImrE+WE6lIdj32tBviDy4EEWKrsxiRaqmM4iDX5RVisTn8Lor1+qXGBNECmw/X+UCjgl5qkVbzYt/Z0Jvk71m40dRwRvaQ3TOHdUMk89p1hvb8a0zsSUvZmUdWBhi2CFVD3mBilAAC2AK+QmHAjJzlPiNrw6co0EE/b6mPdmYsovpDh4npjF/W02OI1+7+qkN5UXKiE5cBuQr0Xf/5toN07WBDVW3tZ+cR+pVYRvRrE27eCnNp6b5NIlb+CHa/P4eur9JTRgbRaFusUXMLBKsRmMTCESfK5E4J8I+iygV6+7qxT2C5Wkh11apLpA+7E0N6us80pebnBMcsV2YCtKzmeez aP/LgJ+T OnL2elAyYe4R/SS0981b3oZUsCnUUZt9nWs0Z2o2am5qP1v4DroxVvSSlzLiPeASUB8FjC5Tyth9+jnymY4ZKsoZFNVW0JFAngo7hNh/rMWEZuEF0J+J+6/7nbJ8rSvY+4xtDbI5vYq0O4jUfzWhJYaVhdUIPvMUZznrOhsJrP/xxZwyb6CbPKBlopD5vmd7LwwZv0bk48vHHeonYzAF22fdbXdl5beEqZyQ0TxB06BYRaPL0d79zkdnx5dYkhYGa7DhQYRZa4kmwieViCicB0jwLIWhhSELeTmEXBeyjK94kcnYFciVOFA71gqBWEs5sZQaLmEL/1R8xuD9Vicz8zd9mnWXVtmrUypdAmkX0m/7FglX5xuLiAQgWnNHI4lUKQeSMAF6O/CWsQe2Rxm29cokEaN6bA6Uz7NVGhPqphtuxuZmkmVJx58XHMIySLoKilNNdTi1tRLJ2EMmhExxkbT5fzr3y93QFGLya X-Bogosity: Ham, tests=bogofilter, spamicity=0.000000, version=1.2.4 Sender: owner-linux-mm@kvack.org Precedence: bulk X-Loop: owner-majordomo@kvack.org List-ID: List-Subscribe: List-Unsubscribe: From: Peter Xu These macros can be helpful when we plan to merge hugetlb code into generic code. Move them out and define them as long as PGTABLE_HAS_HUGE_LEAVES is selected, because there are systems that only define HUGETLB_PAGE not THP. One note here is HPAGE_PMD_SHIFT must be defined even if PMD_SHIFT is not defined (e.g. !CONFIG_MMU case); it (or in other forms, like HPAGE_PMD_NR) is already used in lots of common codes without ifdef guards. Use the old trick to let complations work. Here we only need to differenciate HPAGE_PXD_SHIFT definitions. All the rest macros will be defined based on it. When at it, move HPAGE_PMD_NR / HPAGE_PMD_ORDER over together. Signed-off-by: Peter Xu --- include/linux/huge_mm.h | 29 +++++++++++++++-------------- 1 file changed, 15 insertions(+), 14 deletions(-) diff --git a/include/linux/huge_mm.h b/include/linux/huge_mm.h index 7576025db55d..d3bb25c39482 100644 --- a/include/linux/huge_mm.h +++ b/include/linux/huge_mm.h @@ -64,9 +64,6 @@ ssize_t single_hugepage_flag_show(struct kobject *kobj, enum transparent_hugepage_flag flag); extern struct kobj_attribute shmem_enabled_attr; -#define HPAGE_PMD_ORDER (HPAGE_PMD_SHIFT-PAGE_SHIFT) -#define HPAGE_PMD_NR (1< X-Patchwork-Id: 13606789 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from kanga.kvack.org (kanga.kvack.org [205.233.56.17]) by smtp.lore.kernel.org (Postfix) with ESMTP id 00EF7CD1280 for ; Wed, 27 Mar 2024 15:24:04 +0000 (UTC) Received: by kanga.kvack.org (Postfix) id E8B0A6B009F; Wed, 27 Mar 2024 11:23:59 -0400 (EDT) Received: by kanga.kvack.org (Postfix, from userid 40) id E35906B00A0; Wed, 27 Mar 2024 11:23:59 -0400 (EDT) X-Delivered-To: int-list-linux-mm@kvack.org Received: by kanga.kvack.org (Postfix, from userid 63042) id C8A276B00A1; Wed, 27 Mar 2024 11:23:59 -0400 (EDT) X-Delivered-To: linux-mm@kvack.org Received: from relay.hostedemail.com (smtprelay0011.hostedemail.com [216.40.44.11]) by kanga.kvack.org (Postfix) with ESMTP id A60D56B009F for ; Wed, 27 Mar 2024 11:23:59 -0400 (EDT) Received: from smtpin24.hostedemail.com (a10.router.float.18 [10.200.18.1]) by unirelay04.hostedemail.com (Postfix) with ESMTP id 3B91B1A0DD4 for ; Wed, 27 Mar 2024 15:23:59 +0000 (UTC) X-FDA: 81943189398.24.2EEDA3B Received: from us-smtp-delivery-124.mimecast.com (us-smtp-delivery-124.mimecast.com [170.10.129.124]) by imf11.hostedemail.com (Postfix) with ESMTP id 151A14001A for ; Wed, 27 Mar 2024 15:23:56 +0000 (UTC) Authentication-Results: imf11.hostedemail.com; dkim=pass header.d=redhat.com header.s=mimecast20190719 header.b=OzgduuCE; spf=pass (imf11.hostedemail.com: domain of peterx@redhat.com designates 170.10.129.124 as permitted sender) smtp.mailfrom=peterx@redhat.com; dmarc=pass (policy=none) header.from=redhat.com ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=hostedemail.com; s=arc-20220608; t=1711553037; h=from:from:sender:reply-to:subject:subject:date:date: message-id:message-id:to:to:cc:cc:mime-version:mime-version: content-type:content-type: content-transfer-encoding:content-transfer-encoding: in-reply-to:in-reply-to:references:references:dkim-signature; bh=iN7hVKxpU9yGNoa0FvV6QQfaV15Sbc5QZKf/pqr08oc=; b=Y81U8eHhLibXGIGvaeFo5LLoANECbqQSQCcu7gIE+XIfb0HiHdj3+WkGyeGis83gW53ymd 31Qea5fHUlN/T6/TGWKCoPr/j1MUNrSTq1lLFDnTno4RNqHpWM1wMK7Ea8q6G8GxJzVGoD YF55JRj5Gxf4dBRU8KAavTnL7SQn3e4= ARC-Seal: i=1; s=arc-20220608; d=hostedemail.com; t=1711553037; a=rsa-sha256; cv=none; b=XnXKEMq58uy7DzCUxEnQRtvPz7SvO7n0h7V6peusUmCuakqcx3z66ysT+r9J9Q4IGfrFHg 6wshETEa7R+TTCCVkROKTmdwWS5OA3y1JebnUqg1er0FF9KNvljE6gbqk1ubN+LdysX0Ra D6nTpAzfdAtpwxe6ge83SHdVUjM5l9Y= ARC-Authentication-Results: i=1; imf11.hostedemail.com; dkim=pass header.d=redhat.com header.s=mimecast20190719 header.b=OzgduuCE; spf=pass (imf11.hostedemail.com: domain of peterx@redhat.com designates 170.10.129.124 as permitted sender) smtp.mailfrom=peterx@redhat.com; dmarc=pass (policy=none) header.from=redhat.com DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=redhat.com; s=mimecast20190719; t=1711553036; h=from:from:reply-to:subject:subject:date:date:message-id:message-id: to:to:cc:cc:mime-version:mime-version:content-type:content-type: content-transfer-encoding:content-transfer-encoding: in-reply-to:in-reply-to:references:references; bh=iN7hVKxpU9yGNoa0FvV6QQfaV15Sbc5QZKf/pqr08oc=; b=OzgduuCE5ijan7+anPU0v3ATulf3xn3Bk7Gk3JYuhoBm3ZS1gIcqJVt2pAesMXzNJJDgWE rEKjZuM7hbLQjp1Z2MVlmx/cUWy3UY90lsp+S+50j6tqO3UmvYbfEs2+2Fphs2IepJQLmq tRbBPSVjZ4Xb1ZzQo+sSp3dXf0khzr0= Received: from mail-qv1-f69.google.com (mail-qv1-f69.google.com [209.85.219.69]) by relay.mimecast.com with ESMTP with STARTTLS (version=TLSv1.3, cipher=TLS_AES_256_GCM_SHA384) id us-mta-611-lvZlrj4dMEenZ5lBG6ACHQ-1; Wed, 27 Mar 2024 11:23:45 -0400 X-MC-Unique: lvZlrj4dMEenZ5lBG6ACHQ-1 Received: by mail-qv1-f69.google.com with SMTP id 6a1803df08f44-696a9c482a4so3595976d6.0 for ; Wed, 27 Mar 2024 08:23:45 -0700 (PDT) X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20230601; t=1711553025; x=1712157825; h=content-transfer-encoding:mime-version:references:in-reply-to :message-id:date:subject:cc:to:from:x-gm-message-state:from:to:cc :subject:date:message-id:reply-to; bh=iN7hVKxpU9yGNoa0FvV6QQfaV15Sbc5QZKf/pqr08oc=; b=e1DTHvUMVEXj6wDelgDY9znswepFH/tZWBK6FG8bn8SiLBlN2XGTdBWcYk1+OnmLZq 4hSoxm2lLpX/J7eyOrAgFda5hPGur9PXHu9Xh8Jx7FRPEzNlf6VW06cLc5ww5+oujSeB LBgBdhMjiYWZ/nzjKrpHvqthuqyVfmbV4EpslSDPu5I7if5Vji+hlK1Jd5YZLjNoK3F0 OhQRZpr5/+oT7oD4h8++yi82mNIiZLWWAMFgNK08KOkdSJQlHk0oqZb8dunF1KlA8md8 U3oCBELKlPKYlEG8rMUIUovneLklXzKDX3maOnKd5RRJ61qTvaobC0E/OrELYH3uSd1b yLUA== X-Gm-Message-State: AOJu0YxEcoYoDxwyXrrJNKfIwWFtOUEzZlLIxXeNZiTSb8GfHNCfUsVV vvCEiF/9a8VMGjW+1jKrKgCduY1/gZOGDRUk+XK3u+7zM9wb4XBid7qB/ia7W1qWVPbQpPLdKnN +I0w+QlJbVCC+HUvHZuP3vfu37BPiqIlb6ITbxrI6IPdqCb01lHAv4ep1+I/cSx8YI2mCB85ESL ClkFHYAb2qYSUTaOXPqeCPmAdxggiYbw== X-Received: by 2002:a05:6214:4598:b0:696:7b32:cceb with SMTP id op24-20020a056214459800b006967b32ccebmr13869575qvb.6.1711553024769; Wed, 27 Mar 2024 08:23:44 -0700 (PDT) X-Google-Smtp-Source: AGHT+IFS0Dr3UNRPnI4el/D5x8gcrtCXLMsQwIvd4dOzHJoXuoNg03xhCSCfsu9VpmJMAiYn3G+95A== X-Received: by 2002:a05:6214:4598:b0:696:7b32:cceb with SMTP id op24-20020a056214459800b006967b32ccebmr13869513qvb.6.1711553024070; Wed, 27 Mar 2024 08:23:44 -0700 (PDT) Received: from x1n.redhat.com ([99.254.121.117]) by smtp.gmail.com with ESMTPSA id hu4-20020a056214234400b00690dd47a41csm6412639qvb.86.2024.03.27.08.23.41 (version=TLS1_3 cipher=TLS_AES_256_GCM_SHA384 bits=256/256); Wed, 27 Mar 2024 08:23:43 -0700 (PDT) From: peterx@redhat.com To: linux-mm@kvack.org, linux-kernel@vger.kernel.org Cc: Yang Shi , "Kirill A . Shutemov" , Mike Kravetz , John Hubbard , Michael Ellerman , peterx@redhat.com, Andrew Jones , Muchun Song , linux-riscv@lists.infradead.org, linuxppc-dev@lists.ozlabs.org, Christophe Leroy , Andrew Morton , Christoph Hellwig , Lorenzo Stoakes , Matthew Wilcox , Rik van Riel , linux-arm-kernel@lists.infradead.org, Andrea Arcangeli , David Hildenbrand , "Aneesh Kumar K . V" , Vlastimil Babka , James Houghton , Jason Gunthorpe , Mike Rapoport , Axel Rasmussen Subject: [PATCH v4 04/13] mm: Introduce vma_pgtable_walk_{begin|end}() Date: Wed, 27 Mar 2024 11:23:23 -0400 Message-ID: <20240327152332.950956-5-peterx@redhat.com> X-Mailer: git-send-email 2.44.0 In-Reply-To: <20240327152332.950956-1-peterx@redhat.com> References: <20240327152332.950956-1-peterx@redhat.com> MIME-Version: 1.0 X-Mimecast-Spam-Score: 0 X-Mimecast-Originator: redhat.com X-Rspamd-Queue-Id: 151A14001A X-Rspam-User: X-Rspamd-Server: rspam11 X-Stat-Signature: yfnhrdisatua895kc7mdkupt8tygzm4d X-HE-Tag: 1711553036-710980 X-HE-Meta: U2FsdGVkX1/ZVUkhOEulavnL4HKywEjbDgKrGtWpXp2LF9FHDqaIzflLM7TAbunDE6nnEUu2gBK3s7pVigHkQhebacAJIPougLLxZeBHWekrfJ0qsUZahf0x4Lt2PWADos9W+Uoxb/DBEeSnA93rAI2nhDN6c1CJ/KwIgGIMvSDLVJIWgXQC/9342YbNOou9kTIBp01FivjlFLSPvEJt6De7fefvwLAnAjqSSmBRres3TdbmL0HtLTKNU13xVPj6/9ai7eorS1rteBBHue5jnyfDhDddDVFYAd5tfoVTxPdxMve8CZQOs9nSOScspZfb+VNTOBi4Ra/HOT+QcYqZfBHiWboIOTmBxuuBYxpdENGPNRTK/WuH1tR3uRmXGqYHp/yLhuW8MVs7kvgwKLqRB2T2APx0aBAoHhfYC5khR7YBuACtOTmocW+jIriLzg1u4sxW0qtQZrbIE3gYdTPK28j7CO8d+ZYcueF6YYb0PzWEsTJWVctbYG15nDtIySeg6SRFhss1K3hlyV027BgIveZIqcs/mN6sM+vIIvtG+UuV5ejSurKY02L3NyvfkugILht9taII2/B22zNLlRCWvmp8dVBPBU1VTQc0HRtgMu5kXzgJhB3GsQB73plF5c9gLBJEKtrnOLDqOOC2cF7rlvJW8XtRby5KvoaLcHeuXKqBrmX4O8W3BKvIGngepkSug4fps6RVzCLC1MR7Ej5EDA0h4pcqXYX2C+ZF8n9uGpWqP/DtX8lYsUwBSfKhuZm4760oLclSmIG0y1kx3TdyiTOjOKObJ1m7dZwG6/U01rGmQk98JuZ456qo61D3rRHwom4IqU0yxB/q8E8Ygq5q9eKuJ23X39coepYRACbhMlDErSiRwRwzUIeRU9TSxKzFuRNzmjKc+MqJZT1icbTP3bZluPHlf8ZmKTc0voWbG7jEr/Rg9tcL1dK+1gnfgG/KE0cxYth4mITL6ZTHlft 8t817T/T 7MJ9MYuOLT6BGW/Athqe6OehcnhxQMgpmbv1VrzVkNRXpyxUhx/NY6osgNzjja90s76tiju0yekPLTXdogfDsocWnxzfBMF2fm4R/Xf6d7v5eCVbjWXKLZFHBoHukMtzu7a2VxWezUhQF4nFMQkF4z7MfrxGCYIQrfsAvffpCGr+oNXBLv+FLNJu1Z2dibn0BLh8QluM4Ojy6yc2wiD7TboVJkiEi6+orM/e11w/fzecr5Bu9DfixE71A150JreZmISzXtYQ0fkv0DRVciQt+3Cmb2pYklrxlxWsRgqNrrK/3A+pc4sG3pnGbslO7wNPQpGds9tZ3+30N9qCeXYm3MTcO3dPwhAx7IHmWNdUWxjBmBop7Rk89uRgT5z9ETgP/pLtP0Sfb01+qP1t7H2VxoKezwO3mKJfib+bAFu5nT3vfqz5ec6vwgK6qFvUEAv1U+NGHaG74AtCZQPBGvhwaOp33rPb+i1I6YV8JRjLvidahbds= X-Bogosity: Ham, tests=bogofilter, spamicity=0.000000, version=1.2.4 Sender: owner-linux-mm@kvack.org Precedence: bulk X-Loop: owner-majordomo@kvack.org List-ID: List-Subscribe: List-Unsubscribe: From: Peter Xu Introduce per-vma begin()/end() helpers for pgtable walks. This is a preparation work to merge hugetlb pgtable walkers with generic mm. The helpers need to be called before and after a pgtable walk, will start to be needed if the pgtable walker code supports hugetlb pages. It's a hook point for any type of VMA, but for now only hugetlb uses it to stablize the pgtable pages from getting away (due to possible pmd unsharing). Reviewed-by: Christoph Hellwig Reviewed-by: Muchun Song Signed-off-by: Peter Xu --- include/linux/mm.h | 3 +++ mm/memory.c | 12 ++++++++++++ 2 files changed, 15 insertions(+) diff --git a/include/linux/mm.h b/include/linux/mm.h index afe27ff3fa94..d8f78017d271 100644 --- a/include/linux/mm.h +++ b/include/linux/mm.h @@ -4233,4 +4233,7 @@ static inline bool pfn_is_unaccepted_memory(unsigned long pfn) return range_contains_unaccepted_memory(paddr, paddr + PAGE_SIZE); } +void vma_pgtable_walk_begin(struct vm_area_struct *vma); +void vma_pgtable_walk_end(struct vm_area_struct *vma); + #endif /* _LINUX_MM_H */ diff --git a/mm/memory.c b/mm/memory.c index 3d0c0cc33c57..27d173f9a521 100644 --- a/mm/memory.c +++ b/mm/memory.c @@ -6438,3 +6438,15 @@ void ptlock_free(struct ptdesc *ptdesc) kmem_cache_free(page_ptl_cachep, ptdesc->ptl); } #endif + +void vma_pgtable_walk_begin(struct vm_area_struct *vma) +{ + if (is_vm_hugetlb_page(vma)) + hugetlb_vma_lock_read(vma); +} + +void vma_pgtable_walk_end(struct vm_area_struct *vma) +{ + if (is_vm_hugetlb_page(vma)) + hugetlb_vma_unlock_read(vma); +} From patchwork Wed Mar 27 15:23:24 2024 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Peter Xu X-Patchwork-Id: 13606785 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from kanga.kvack.org (kanga.kvack.org [205.233.56.17]) by smtp.lore.kernel.org (Postfix) with ESMTP id C8562C54E67 for ; Wed, 27 Mar 2024 15:23:53 +0000 (UTC) Received: by kanga.kvack.org (Postfix) id 90A2B6B0099; Wed, 27 Mar 2024 11:23:52 -0400 (EDT) Received: by kanga.kvack.org (Postfix, from userid 40) id 8BB046B009A; Wed, 27 Mar 2024 11:23:52 -0400 (EDT) X-Delivered-To: int-list-linux-mm@kvack.org Received: by kanga.kvack.org (Postfix, from userid 63042) id 696A56B009B; Wed, 27 Mar 2024 11:23:52 -0400 (EDT) X-Delivered-To: linux-mm@kvack.org Received: from relay.hostedemail.com (smtprelay0014.hostedemail.com [216.40.44.14]) by kanga.kvack.org (Postfix) with ESMTP id 4989A6B0099 for ; Wed, 27 Mar 2024 11:23:52 -0400 (EDT) Received: from smtpin26.hostedemail.com (a10.router.float.18 [10.200.18.1]) by unirelay04.hostedemail.com (Postfix) with ESMTP id 0E2C71A0E13 for ; Wed, 27 Mar 2024 15:23:52 +0000 (UTC) X-FDA: 81943189104.26.7DAAAE8 Received: from us-smtp-delivery-124.mimecast.com (us-smtp-delivery-124.mimecast.com [170.10.129.124]) by imf24.hostedemail.com (Postfix) with ESMTP id 03A6218000F for ; Wed, 27 Mar 2024 15:23:49 +0000 (UTC) Authentication-Results: imf24.hostedemail.com; dkim=pass header.d=redhat.com header.s=mimecast20190719 header.b=AVsJQ3ox; dmarc=pass (policy=none) header.from=redhat.com; spf=pass (imf24.hostedemail.com: domain of peterx@redhat.com designates 170.10.129.124 as permitted sender) smtp.mailfrom=peterx@redhat.com ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=hostedemail.com; s=arc-20220608; t=1711553030; h=from:from:sender:reply-to:subject:subject:date:date: message-id:message-id:to:to:cc:cc:mime-version:mime-version: content-type:content-type: content-transfer-encoding:content-transfer-encoding: in-reply-to:in-reply-to:references:references:dkim-signature; bh=VkfUQlJiw2YmDi669w4oXgCkjfbnGDZMVpc2LJneNK4=; b=W5eykt6K1Rlaj9ohohWHBvje75V+qfFvfkJd7RLFjwIfC7ccDd0J4fDGsfu7EtRBNJ+ehX +56BhbqU+2C7QKrrqByZUZVp94xlpCPwCWW8y/+bTS0wl3gjduFgRSd9v4g3q7MniOKy36 QSNwMcRviaj5TqA0qyrb0AEHMIUzDog= ARC-Authentication-Results: i=1; imf24.hostedemail.com; dkim=pass header.d=redhat.com header.s=mimecast20190719 header.b=AVsJQ3ox; dmarc=pass (policy=none) header.from=redhat.com; spf=pass (imf24.hostedemail.com: domain of peterx@redhat.com designates 170.10.129.124 as permitted sender) smtp.mailfrom=peterx@redhat.com ARC-Seal: i=1; s=arc-20220608; d=hostedemail.com; t=1711553030; a=rsa-sha256; cv=none; b=UsRUQXn++3n+DPFOlmr+QzHsgdzue4PDxpvSGAWA5kO4jguO246I+3I00nugMd2uxiDJCI 7kLd/dBW7Rv4shNlD4VJipUAsMsh3g+jYDDoeQWU7xHNMysmqZfIhwReKKgHrQ9SrQSkM+ c46EgPywp6jvhIyFU0KGSBNCbb+L7Y4= DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=redhat.com; s=mimecast20190719; t=1711553029; h=from:from:reply-to:subject:subject:date:date:message-id:message-id: to:to:cc:cc:mime-version:mime-version:content-type:content-type: content-transfer-encoding:content-transfer-encoding: in-reply-to:in-reply-to:references:references; bh=VkfUQlJiw2YmDi669w4oXgCkjfbnGDZMVpc2LJneNK4=; b=AVsJQ3oxZori/0jbiU/hEUHGTPyOIfF2KtXrxRs+X9qPY6bpKKg3pU3mS80iG+/rUMPxLq aQKTRNxV677HRcYDhyG05TlV3K7yqcgzl97J0robQAfWSrJEq7Qlz53NpFNy5DqVt8zadu G+7oNcVo6BMi+6axzYrp57Fo5oxz+yY= Received: from mail-qv1-f72.google.com (mail-qv1-f72.google.com [209.85.219.72]) by relay.mimecast.com with ESMTP with STARTTLS (version=TLSv1.3, cipher=TLS_AES_256_GCM_SHA384) id us-mta-607-X4n5eIOTPlWScapNvWyJzw-1; Wed, 27 Mar 2024 11:23:48 -0400 X-MC-Unique: X4n5eIOTPlWScapNvWyJzw-1 Received: by mail-qv1-f72.google.com with SMTP id 6a1803df08f44-690c19ee50bso2421946d6.0 for ; Wed, 27 Mar 2024 08:23:48 -0700 (PDT) X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20230601; t=1711553027; x=1712157827; h=content-transfer-encoding:mime-version:references:in-reply-to :message-id:date:subject:cc:to:from:x-gm-message-state:from:to:cc :subject:date:message-id:reply-to; bh=VkfUQlJiw2YmDi669w4oXgCkjfbnGDZMVpc2LJneNK4=; b=ECQz9JpO/hDmHvFH+JRpx//t5Z2Qod+MG3DcmhmNMHbrOuvD7DcQnafxjREInljGEp 56OfrOX3Bu6UOhzGoaiWAOyb3JO9LlqLWtK1wegtiNQgvvwDcHShabbCKOdqv7x6RKBd /mAobm31z0FH5nixmcuekbuN5/XW2+W8IVGGWYHvsicnzwvxNLmLCcL5uULUvi+nYYOG Sp0eJMqfZ56Pf69z5efk+GEfe2Gntg+xZF18mzmJAkktiypw7NwV/i5q3GVwUsR10pm7 +PYx8lwHatD4jGX+4ARu9AREuc44835vwMS+iIvOGm/O8pkbw1sOo/A+qUTITccZpX3D HLrg== X-Gm-Message-State: AOJu0YyCrSBiOauMk9GTTZgVu0q+M1c7tP2EwFfmjA3vwcbAzXJrSS5q wIDLzM1vAeF6EfkeXXpgtBOZnEIfA7e4uLocVnAxdF34JIjxDhpZS0hJlqM5uR6jVeb9QDHO5bP +aap4Z/q62rq0iEPPQPKli9jEftJ5FJ4anWZQwn5YApNQKanW3Ct1Mq9cUsh56LziVU6UPRBUf2 YRnVJsodkbGBuOgQcyi2z4E2SxFpKw6A== X-Received: by 2002:a05:6214:3105:b0:696:6f59:4d19 with SMTP id ks5-20020a056214310500b006966f594d19mr14917350qvb.1.1711553027130; Wed, 27 Mar 2024 08:23:47 -0700 (PDT) X-Google-Smtp-Source: AGHT+IG/cvaZbNzejgmgOgxRck5pBSr/g3hsLSZETHWpBhBPVR+VZkVf7lYhXdSr9+e3t1xSA2Wfog== X-Received: by 2002:a05:6214:3105:b0:696:6f59:4d19 with SMTP id ks5-20020a056214310500b006966f594d19mr14917279qvb.1.1711553026413; Wed, 27 Mar 2024 08:23:46 -0700 (PDT) Received: from x1n.redhat.com ([99.254.121.117]) by smtp.gmail.com with ESMTPSA id hu4-20020a056214234400b00690dd47a41csm6412639qvb.86.2024.03.27.08.23.44 (version=TLS1_3 cipher=TLS_AES_256_GCM_SHA384 bits=256/256); Wed, 27 Mar 2024 08:23:45 -0700 (PDT) From: peterx@redhat.com To: linux-mm@kvack.org, linux-kernel@vger.kernel.org Cc: Yang Shi , "Kirill A . Shutemov" , Mike Kravetz , John Hubbard , Michael Ellerman , peterx@redhat.com, Andrew Jones , Muchun Song , linux-riscv@lists.infradead.org, linuxppc-dev@lists.ozlabs.org, Christophe Leroy , Andrew Morton , Christoph Hellwig , Lorenzo Stoakes , Matthew Wilcox , Rik van Riel , linux-arm-kernel@lists.infradead.org, Andrea Arcangeli , David Hildenbrand , "Aneesh Kumar K . V" , Vlastimil Babka , James Houghton , Jason Gunthorpe , Mike Rapoport , Axel Rasmussen Subject: [PATCH v4 05/13] mm/arch: Provide pud_pfn() fallback Date: Wed, 27 Mar 2024 11:23:24 -0400 Message-ID: <20240327152332.950956-6-peterx@redhat.com> X-Mailer: git-send-email 2.44.0 In-Reply-To: <20240327152332.950956-1-peterx@redhat.com> References: <20240327152332.950956-1-peterx@redhat.com> MIME-Version: 1.0 X-Mimecast-Spam-Score: 0 X-Mimecast-Originator: redhat.com X-Rspamd-Server: rspam09 X-Rspamd-Queue-Id: 03A6218000F X-Stat-Signature: ybs43r9dpi1pwz89c3kmf6f3rwpgoth9 X-Rspam-User: X-HE-Tag: 1711553029-993992 X-HE-Meta: U2FsdGVkX19vQOgKu6hpk2rfz+V8IY/IkSFgaAruESuy7uxDBCpdyWfD5ZT3/pSysjTb/aP71hWTdSKOjinWEjHDRZdnUGC8S59+aybkDuXIfrQatXIOHXhGoz/qyP2wTmM9tOm7cM3aeyl2LQA8JhH21o6mJ0dmAmpR0oRBqpUnItDFtkePJprvUe2AhGijkbhnEx/QfdwCKysHMGeFbADrhC94lEmImBpcbWu1RJCwLXV9aP6sK2Sd1PE6cvr7diCXtmZQej9i0uBiaNPvLk1yLcaUSIF0HnZWpYVFXTuByngRqlGlwvsbT1wtoK8ktNQ5op81w+BfMNt0HTLkVCDh/nQB5SWkClALvRKgpFaa3wtwLPV0TUaXq2+yVaBschbpAGFSJlUBTWEiida0loSfQ1WzpFxa+97m7QSSfsYjFu8UcshuqXbJu0jU7RwJGtDrAVAxWEB34NOpg/QeThZtU8Xj7TtvafSEwDvgpxMdcbAhOriDjMSvMaECQ83P8zOSKERL/qHcH0GXM7iuBgv4i/Q9fesM/DcTQH0ZMWECv0jEdYSlADhNrNfTCz7hgt///T0HGCYWm9dzmDE8v8C86pE6d1eizMwBDWIpM8228EAczAhHLNjYJsbEv1rIJik2/dcIlmsii6ntv6Mg55wi/QW3YDjM6ty/QDOWnIfAe/e67JS4rSLJmsN+3xY+jKaIdCCYoszOqPUrJ+skH8our3i5Yh8A2VQmWJ8/mVNxEObLoP3kRVkMEMppW8XIjZ4OZVXuDK5VuBKY3zIweTX3Ky0u8q0xxhGYoq0psNcC4syRmJyxnf+UvY081uoLVSXuKxeQZ3+QDA/On7x0lcm4kp0RDYD0L8l0ed7D+twBQUiM58o9jII6whtv0Ei2Ng2BGlubrZNFfZCyNk7VglIujKRX85/Y5imQdh1RHUrUtan7ZlBMG++IX8qsHjpXQQnX0y7dsCRsr9RkHbO JAEmPYMZ Jp4M2HFe2ivKAI/3AtYniNHRnmapWWpcFACf2yWQ0tSPKV4CiSZavN+iXXgjh7bs1NDd7d4/iHwdM+IL0Kyp7/R5sbiNh6ZT/XWCDLE0V2uIp38DeY9tKjyEF4Cpu6olPOrljSIVK2EMFbvCaTZhwyilW5GSpWUVA48CfMC9N+rnSasFXT15Gk+Uo75GAKYltbdo78AG0F7OcqGFjGpnYDsyTRnIVIsMgzBs7pFQs9tcrAgCiBjW0p9kAA/4D0bns1MGs3l0sI3PflAD25pS0xaTCADNLf5z8xpbemXFh4+BVHNXpuvYd8IiN6jQ4kR92UzTMJzk2pJ2VoaPCe/TrDaDsNYI+GIA6cp0cayKtZLuDltEarMSGWadfqT7PJsw7IzwVk5aAEXbZMb1W7UNJ/9HFcTR0SeAMccpqGbovjacwU4PiSINcbb/IiGl60inq7Sa808AHekAyyVvggWSNu9klsSrc4An5qWZeqeeAwn7b13eyrwYgaycEUESxVsZhoD01yZ7ZzNCyTKKDFRCtbu0UuSaGqA0d22MxQODdaj8sCKORw5R5cvPFJ5blz8SxXqYTWOP/Jx5Vy1JO1Lq0NUcEAzWkysZhttELSvibJ1GGQy7Ck0M7YJW2Ew== X-Bogosity: Ham, tests=bogofilter, spamicity=0.000000, version=1.2.4 Sender: owner-linux-mm@kvack.org Precedence: bulk X-Loop: owner-majordomo@kvack.org List-ID: List-Subscribe: List-Unsubscribe: From: Peter Xu The comment in the code explains the reasons. We took a different approach comparing to pmd_pfn() by providing a fallback function. Another option is to provide some lower level config options (compare to HUGETLB_PAGE or THP) to identify which layer an arch can support for such huge mappings. However that can be an overkill. Cc: Mike Rapoport (IBM) Cc: Matthew Wilcox Reviewed-by: Jason Gunthorpe Signed-off-by: Peter Xu --- arch/riscv/include/asm/pgtable.h | 1 + arch/s390/include/asm/pgtable.h | 1 + arch/sparc/include/asm/pgtable_64.h | 1 + arch/x86/include/asm/pgtable.h | 1 + include/linux/pgtable.h | 10 ++++++++++ 5 files changed, 14 insertions(+) diff --git a/arch/riscv/include/asm/pgtable.h b/arch/riscv/include/asm/pgtable.h index 20242402fc11..0ca28cc8e3fa 100644 --- a/arch/riscv/include/asm/pgtable.h +++ b/arch/riscv/include/asm/pgtable.h @@ -646,6 +646,7 @@ static inline unsigned long pmd_pfn(pmd_t pmd) #define __pud_to_phys(pud) (__page_val_to_pfn(pud_val(pud)) << PAGE_SHIFT) +#define pud_pfn pud_pfn static inline unsigned long pud_pfn(pud_t pud) { return ((__pud_to_phys(pud) & PUD_MASK) >> PAGE_SHIFT); diff --git a/arch/s390/include/asm/pgtable.h b/arch/s390/include/asm/pgtable.h index 1a71cb19c089..6cbbe473f680 100644 --- a/arch/s390/include/asm/pgtable.h +++ b/arch/s390/include/asm/pgtable.h @@ -1414,6 +1414,7 @@ static inline unsigned long pud_deref(pud_t pud) return (unsigned long)__va(pud_val(pud) & origin_mask); } +#define pud_pfn pud_pfn static inline unsigned long pud_pfn(pud_t pud) { return __pa(pud_deref(pud)) >> PAGE_SHIFT; diff --git a/arch/sparc/include/asm/pgtable_64.h b/arch/sparc/include/asm/pgtable_64.h index 4d1bafaba942..26efc9bb644a 100644 --- a/arch/sparc/include/asm/pgtable_64.h +++ b/arch/sparc/include/asm/pgtable_64.h @@ -875,6 +875,7 @@ static inline bool pud_leaf(pud_t pud) return pte_val(pte) & _PAGE_PMD_HUGE; } +#define pud_pfn pud_pfn static inline unsigned long pud_pfn(pud_t pud) { pte_t pte = __pte(pud_val(pud)); diff --git a/arch/x86/include/asm/pgtable.h b/arch/x86/include/asm/pgtable.h index cefc7a84f7a4..273f7557218c 100644 --- a/arch/x86/include/asm/pgtable.h +++ b/arch/x86/include/asm/pgtable.h @@ -234,6 +234,7 @@ static inline unsigned long pmd_pfn(pmd_t pmd) return (pfn & pmd_pfn_mask(pmd)) >> PAGE_SHIFT; } +#define pud_pfn pud_pfn static inline unsigned long pud_pfn(pud_t pud) { phys_addr_t pfn = pud_val(pud); diff --git a/include/linux/pgtable.h b/include/linux/pgtable.h index 600e17d03659..75fe309a4e10 100644 --- a/include/linux/pgtable.h +++ b/include/linux/pgtable.h @@ -1817,6 +1817,16 @@ typedef unsigned int pgtbl_mod_mask; #define pte_leaf_size(x) PAGE_SIZE #endif +/* + * We always define pmd_pfn for all archs as it's used in lots of generic + * code. Now it happens too for pud_pfn (and can happen for larger + * mappings too in the future; we're not there yet). Instead of defining + * it for all archs (like pmd_pfn), provide a fallback. + */ +#ifndef pud_pfn +#define pud_pfn(x) ({ BUILD_BUG(); 0; }) +#endif + /* * Some architectures have MMUs that are configurable or selectable at boot * time. These lead to variable PTRS_PER_x. For statically allocated arrays it From patchwork Wed Mar 27 15:23:25 2024 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Peter Xu X-Patchwork-Id: 13606786 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from kanga.kvack.org (kanga.kvack.org [205.233.56.17]) by smtp.lore.kernel.org (Postfix) with ESMTP id 9C03DC47DD9 for ; Wed, 27 Mar 2024 15:23:56 +0000 (UTC) Received: by kanga.kvack.org (Postfix) id F262B6B009A; Wed, 27 Mar 2024 11:23:53 -0400 (EDT) Received: by kanga.kvack.org (Postfix, from userid 40) id E84B16B009B; Wed, 27 Mar 2024 11:23:53 -0400 (EDT) X-Delivered-To: int-list-linux-mm@kvack.org Received: by kanga.kvack.org (Postfix, from userid 63042) id CD8F66B009C; Wed, 27 Mar 2024 11:23:53 -0400 (EDT) X-Delivered-To: linux-mm@kvack.org Received: from relay.hostedemail.com (smtprelay0014.hostedemail.com [216.40.44.14]) by kanga.kvack.org (Postfix) with ESMTP id ABB8B6B009A for ; Wed, 27 Mar 2024 11:23:53 -0400 (EDT) Received: from smtpin30.hostedemail.com (a10.router.float.18 [10.200.18.1]) by unirelay10.hostedemail.com (Postfix) with ESMTP id 76F46C0CFE for ; Wed, 27 Mar 2024 15:23:53 +0000 (UTC) X-FDA: 81943189146.30.C96F7C9 Received: from us-smtp-delivery-124.mimecast.com (us-smtp-delivery-124.mimecast.com [170.10.133.124]) by imf17.hostedemail.com (Postfix) with ESMTP id 9381740003 for ; Wed, 27 Mar 2024 15:23:51 +0000 (UTC) Authentication-Results: imf17.hostedemail.com; dkim=pass header.d=redhat.com header.s=mimecast20190719 header.b=MmVWvOTm; dmarc=pass (policy=none) header.from=redhat.com; spf=pass (imf17.hostedemail.com: domain of peterx@redhat.com designates 170.10.133.124 as permitted sender) smtp.mailfrom=peterx@redhat.com ARC-Seal: i=1; s=arc-20220608; d=hostedemail.com; t=1711553031; a=rsa-sha256; cv=none; b=lPBjk4e69PH5/kecs+GZK+1Uddk+/YcGXD3EhQ7IxVb70CSuLxFNACavhaftaC79Xfg0qo K7G4WhVlJPXOELLrf2M1p0RpyA2jEz0spX+K/HW/cCCeJ/ehUKvSPodVo4Q1KXs9P382n8 2ytioV9JIBBfyOt88sA0uA3evC/Zul8= ARC-Authentication-Results: i=1; imf17.hostedemail.com; dkim=pass header.d=redhat.com header.s=mimecast20190719 header.b=MmVWvOTm; dmarc=pass (policy=none) header.from=redhat.com; spf=pass (imf17.hostedemail.com: domain of peterx@redhat.com designates 170.10.133.124 as permitted sender) smtp.mailfrom=peterx@redhat.com ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=hostedemail.com; s=arc-20220608; t=1711553031; h=from:from:sender:reply-to:subject:subject:date:date: message-id:message-id:to:to:cc:cc:mime-version:mime-version: content-type:content-type: content-transfer-encoding:content-transfer-encoding: in-reply-to:in-reply-to:references:references:dkim-signature; bh=gD68z8/n0HSMW69QF19tjjGBtDgz4n3uypdcneHmRPc=; b=CU2EFhWREtLh++opZAuTs0DpPFytXBTz1l51vmLD7QFKF+GFS7T05nJpRgI0Ikiq8IZGid UvCf1GjSxVlmEMJYZv2mn4ZGGosYhGv8CqErENpHKWHxud3OTJbP5Q3JBMxflKrh+BT+mL HX/trS1lOun8bqVOI7CH/5GHF8dUK5Y= DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=redhat.com; s=mimecast20190719; t=1711553031; h=from:from:reply-to:subject:subject:date:date:message-id:message-id: to:to:cc:cc:mime-version:mime-version:content-type:content-type: content-transfer-encoding:content-transfer-encoding: in-reply-to:in-reply-to:references:references; bh=gD68z8/n0HSMW69QF19tjjGBtDgz4n3uypdcneHmRPc=; b=MmVWvOTmTU6nujh5vqCWOvh40F2hfZGnujaJOTLKSL1yFnAGxQ4g3uLfFiByr/zlkmcOQY ilQbcnWPImt4PEV4bgatJGE6JiNqgsB0sGlqJhncvrw/Z5NtDTHzinR+sWw6JzChrB51nU VvMjCL3Zn3JP7dFwbrsQZySMmR31jtA= Received: from mail-qv1-f70.google.com (mail-qv1-f70.google.com [209.85.219.70]) by relay.mimecast.com with ESMTP with STARTTLS (version=TLSv1.3, cipher=TLS_AES_256_GCM_SHA384) id us-mta-310-ThSDHml2MlGi7AMbtsoiGw-1; Wed, 27 Mar 2024 11:23:49 -0400 X-MC-Unique: ThSDHml2MlGi7AMbtsoiGw-1 Received: by mail-qv1-f70.google.com with SMTP id 6a1803df08f44-690c19ee50bso2422136d6.0 for ; Wed, 27 Mar 2024 08:23:49 -0700 (PDT) X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20230601; t=1711553029; x=1712157829; h=content-transfer-encoding:mime-version:references:in-reply-to :message-id:date:subject:cc:to:from:x-gm-message-state:from:to:cc :subject:date:message-id:reply-to; bh=gD68z8/n0HSMW69QF19tjjGBtDgz4n3uypdcneHmRPc=; b=cdhdF6PvJ1y14eGyABli776mNJMJOiEoj99Rr7/cQ0NHcQoIvaPjCM/zwhUH280HZ/ p8rduN+D5K/LFL/UiT3tCh+GOou4e3loWdu6+bjgZMfj0G1rbaXJpDiBfUZ9JcbQCM6Y GMJey6W8Km3QdakMez07A4lsNUsBGMvgp6SjzT16wxOiEogDzwwEiBYUOV9ymA76TVaU 94cQOs4b6Ro4sblFU7gZM2wyaM5saGvBptj0mVhr33GZTWZpuiMaFpNwRcGbTnijJt6q dmcVQINcqw/6wgzsWmvtPi/8e4b3mcMkiGQT7usiSXiAOhezwLYSv8+ga9bxOJshFnZi vWPA== X-Gm-Message-State: AOJu0YyDptpkU5qD7nSbJMJjTdWiNyM9QZa7PSfRIWFNfuQvGVZIFHli DFNRrU/msMrJJtVtsfgEQ/YoTGCaa386u63mB7sjdhjoQw5Qkck7N6oklqdXENMu6pTjO9PYrlS E3NiwcKybCWIk1g9xIJkTLuu8ieg4fvUxkDdWRy95O07RbGca14r8AbzU57ZgMyre083K8xoesb 0a5cM46t6lexmxLdTze8YEqj+Az4tKxw== X-Received: by 2002:a05:6214:3a01:b0:696:81b8:a462 with SMTP id nw1-20020a0562143a0100b0069681b8a462mr13234485qvb.0.1711553028878; Wed, 27 Mar 2024 08:23:48 -0700 (PDT) X-Google-Smtp-Source: AGHT+IFaIIFSwEPr6utJECaPIJXCgmHTkHVcLUQEzRJpciko/79sjH/yOlRx7KIof08fJkHOKzrhqQ== X-Received: by 2002:a05:6214:3a01:b0:696:81b8:a462 with SMTP id nw1-20020a0562143a0100b0069681b8a462mr13234423qvb.0.1711553028305; Wed, 27 Mar 2024 08:23:48 -0700 (PDT) Received: from x1n.redhat.com ([99.254.121.117]) by smtp.gmail.com with ESMTPSA id hu4-20020a056214234400b00690dd47a41csm6412639qvb.86.2024.03.27.08.23.46 (version=TLS1_3 cipher=TLS_AES_256_GCM_SHA384 bits=256/256); Wed, 27 Mar 2024 08:23:47 -0700 (PDT) From: peterx@redhat.com To: linux-mm@kvack.org, linux-kernel@vger.kernel.org Cc: Yang Shi , "Kirill A . Shutemov" , Mike Kravetz , John Hubbard , Michael Ellerman , peterx@redhat.com, Andrew Jones , Muchun Song , linux-riscv@lists.infradead.org, linuxppc-dev@lists.ozlabs.org, Christophe Leroy , Andrew Morton , Christoph Hellwig , Lorenzo Stoakes , Matthew Wilcox , Rik van Riel , linux-arm-kernel@lists.infradead.org, Andrea Arcangeli , David Hildenbrand , "Aneesh Kumar K . V" , Vlastimil Babka , James Houghton , Jason Gunthorpe , Mike Rapoport , Axel Rasmussen Subject: [PATCH v4 06/13] mm/gup: Drop folio_fast_pin_allowed() in hugepd processing Date: Wed, 27 Mar 2024 11:23:25 -0400 Message-ID: <20240327152332.950956-7-peterx@redhat.com> X-Mailer: git-send-email 2.44.0 In-Reply-To: <20240327152332.950956-1-peterx@redhat.com> References: <20240327152332.950956-1-peterx@redhat.com> MIME-Version: 1.0 X-Mimecast-Spam-Score: 0 X-Mimecast-Originator: redhat.com X-Rspam-User: X-Rspamd-Server: rspam06 X-Rspamd-Queue-Id: 9381740003 X-Stat-Signature: awdicszm3yr5yiryo8gxk4q3s6tj3fhz X-HE-Tag: 1711553031-357983 X-HE-Meta: U2FsdGVkX1/OjX1mOkxDxUssWpaZS6S6t3qRAYcTmnPsATkROA87ukHyywcN5sCRV8fxLEVYdVMTvxFZs5RA1hluNNQLQp7phAeS2ZmfIle+y7A6RgB2nzuU7+Esz74lqdQ3cgGRm7DmTaRdwgnUiGtOJyJIG+lM5y+Hz5irIJc72oYq2MH9h01ZWOWIoGCCDE7510s9acf270erbMhpKv3Zanb7UAQliOLhTqCk3r4yBl5T1Yl5RM/lYfc/FD3npVYGS0CMil4EAnYKgEfhd4mqdtjVfvBHOkzn3+8AnTR2JGS9mSx/PDifQA2H61QkE3VQB0kpDpF4hqIIvDLFVlZil2wWYjY2sU5LK635fOW1VB6D8SVVwLX/VTML1ZPPfc/0Uwk9ccROIMUFgxPsFzG50n1G+wck4wOMC1TCOHlIl645go1imFf2OgUo3yvn1WYhIle9O+mn92ls0FBmx5X9JGQkpNXnZfriLCn/fRmxzUA1t6oTHX3TU1VfpwX1egjyqZtpzHbRzkNu3RmxMwW0g66dbLnWS5G4BjLdw48U3c9+NqqIJTCMBPYCwIGtsXZuulF6Fu2NP6tCLupfUuYbe0vkg8mz8C1rx6lNKndge7Mj7dkA2zg2sRziIiCEgbHlSbB+BJ8lV4r6KmWMi5gfgVy3olgPLG72psSU3eXGsn1WmV9u3R+b0OmULdWY8mXWlKRhh9NKX5H7np8OyEQvFgFeBdSUG8+3B/T4bzqcRLg8jjVwqXyg/7zHsp4xmTFFAAiTCK/xiC8d8MtRbAofRTNUu+hytYkCiCwMV2edoYoIgHIQu3J3UT7aR6em6fCC2DbUFQjf+k+nqTSUSuinchkuTCeGwAd+J0kxc+75wqd/zqQkvaq8tZ3yHVOXzMZolCWR0lQTGIXgfmTSofgg20uVO4xjVuGDBQSfrUVMxxQ4fZQljuSLvQHaYaR0UUvmTkge0jpMmNZ9WCL 6J8FFNm3 XjXA12a2X5D4HzyWMKeyg+6msTmumKnQtjLPH660olp2n7O/SCzHdvCQFS2HYkfxux1Kua9ze6yTpOZpXH15t2xjApZAEbVPvyqsVrb+5l0oQc0SmIql/i6h+IB9RQd83pIw9POyq1T9T5zI= X-Bogosity: Ham, tests=bogofilter, spamicity=0.000000, version=1.2.4 Sender: owner-linux-mm@kvack.org Precedence: bulk X-Loop: owner-majordomo@kvack.org List-ID: List-Subscribe: List-Unsubscribe: From: Peter Xu Hugepd format for GUP is only used in PowerPC with hugetlbfs. There are some kernel usage of hugepd (can refer to hugepd_populate_kernel() for PPC_8XX), however those pages are not candidates for GUP. Commit a6e79df92e4a ("mm/gup: disallow FOLL_LONGTERM GUP-fast writing to file-backed mappings") added a check to fail gup-fast if there's potential risk of violating GUP over writeback file systems. That should never apply to hugepd. Considering that hugepd is an old format (and even software-only), there's no plan to extend hugepd into other file typed memories that is prone to the same issue. Drop that check, not only because it'll never be true for hugepd per any known plan, but also it paves way for reusing the function outside fast-gup. To make sure we'll still remember this issue just in case hugepd will be extended to support non-hugetlbfs memories, add a rich comment above gup_huge_pd(), explaining the issue with proper references. Cc: Christoph Hellwig Cc: Lorenzo Stoakes Cc: Michael Ellerman Cc: linuxppc-dev@lists.ozlabs.org Signed-off-by: Peter Xu --- mm/gup.c | 13 ++++++++----- 1 file changed, 8 insertions(+), 5 deletions(-) diff --git a/mm/gup.c b/mm/gup.c index e7510b6ce765..db35b056fc9a 100644 --- a/mm/gup.c +++ b/mm/gup.c @@ -2832,11 +2832,6 @@ static int gup_hugepte(pte_t *ptep, unsigned long sz, unsigned long addr, return 0; } - if (!folio_fast_pin_allowed(folio, flags)) { - gup_put_folio(folio, refs, flags); - return 0; - } - if (!pte_write(pte) && gup_must_unshare(NULL, flags, &folio->page)) { gup_put_folio(folio, refs, flags); return 0; @@ -2847,6 +2842,14 @@ static int gup_hugepte(pte_t *ptep, unsigned long sz, unsigned long addr, return 1; } +/* + * NOTE: currently GUP for a hugepd is only possible on hugetlbfs file + * systems on Power, which does not have issue with folio writeback against + * GUP updates. When hugepd will be extended to support non-hugetlbfs or + * even anonymous memory, we need to do extra check as what we do with most + * of the other folios. See writable_file_mapping_allowed() and + * folio_fast_pin_allowed() for more information. + */ static int gup_huge_pd(hugepd_t hugepd, unsigned long addr, unsigned int pdshift, unsigned long end, unsigned int flags, struct page **pages, int *nr) From patchwork Wed Mar 27 15:23:26 2024 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Peter Xu X-Patchwork-Id: 13606787 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from kanga.kvack.org (kanga.kvack.org [205.233.56.17]) by smtp.lore.kernel.org (Postfix) with ESMTP id B2C05CD1284 for ; Wed, 27 Mar 2024 15:23:59 +0000 (UTC) Received: by kanga.kvack.org (Postfix) id EFC656B009E; Wed, 27 Mar 2024 11:23:56 -0400 (EDT) Received: by kanga.kvack.org (Postfix, from userid 40) id EAC5A6B009D; Wed, 27 Mar 2024 11:23:56 -0400 (EDT) X-Delivered-To: int-list-linux-mm@kvack.org Received: by kanga.kvack.org (Postfix, from userid 63042) id D26B56B009E; Wed, 27 Mar 2024 11:23:56 -0400 (EDT) X-Delivered-To: linux-mm@kvack.org Received: from relay.hostedemail.com (smtprelay0013.hostedemail.com [216.40.44.13]) by kanga.kvack.org (Postfix) with ESMTP id B20416B009C for ; Wed, 27 Mar 2024 11:23:56 -0400 (EDT) Received: from smtpin09.hostedemail.com (a10.router.float.18 [10.200.18.1]) by unirelay08.hostedemail.com (Postfix) with ESMTP id 7C2EC1401A6 for ; Wed, 27 Mar 2024 15:23:56 +0000 (UTC) X-FDA: 81943189272.09.EFCBD8D Received: from us-smtp-delivery-124.mimecast.com (us-smtp-delivery-124.mimecast.com [170.10.133.124]) by imf09.hostedemail.com (Postfix) with ESMTP id 8F2EA140026 for ; Wed, 27 Mar 2024 15:23:54 +0000 (UTC) Authentication-Results: imf09.hostedemail.com; dkim=pass header.d=redhat.com header.s=mimecast20190719 header.b="NB2Jw+1/"; spf=pass (imf09.hostedemail.com: domain of peterx@redhat.com designates 170.10.133.124 as permitted sender) smtp.mailfrom=peterx@redhat.com; dmarc=pass (policy=none) header.from=redhat.com ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=hostedemail.com; s=arc-20220608; t=1711553034; h=from:from:sender:reply-to:subject:subject:date:date: message-id:message-id:to:to:cc:cc:mime-version:mime-version: content-type:content-type: content-transfer-encoding:content-transfer-encoding: in-reply-to:in-reply-to:references:references:dkim-signature; bh=QDTtuz/iBLnx3uLIw0aNQbjjBf12gK6PRqFn//yUSp8=; b=bYUfMcOvEqc3QbbZUqlmAUy+NKsknbyJAoVnjeYiOF8x8xkDiVZgvvl2CDqdyazpZAna/b l5ofmnQJ8zTO+idADkDSxGxkfP4nh3ndnvgAxqELjmJRemMrBr80mFb/Ao1CfeMKATKgvp kv3tJhlIRI523/D02Iv0oaoKny/sbrM= ARC-Seal: i=1; s=arc-20220608; d=hostedemail.com; t=1711553034; a=rsa-sha256; cv=none; b=3qLG6IioCQhkVLC6vGAR6zq6QKdn6TagrWlTfR3keBtDEk2OCBq1KMwFDjdkzp426pY7uL pdTJaXWuY9ge54Jh3XpOKPaCQlj3Pe03Xo7g7/jbU5wdg6GxPrYfUfC7vbg8bck8McgE+f wtJWMeNVI5QqXNMNYkI7b9iawq9cJdA= ARC-Authentication-Results: i=1; imf09.hostedemail.com; dkim=pass header.d=redhat.com header.s=mimecast20190719 header.b="NB2Jw+1/"; spf=pass (imf09.hostedemail.com: domain of peterx@redhat.com designates 170.10.133.124 as permitted sender) smtp.mailfrom=peterx@redhat.com; dmarc=pass (policy=none) header.from=redhat.com DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=redhat.com; s=mimecast20190719; t=1711553033; h=from:from:reply-to:subject:subject:date:date:message-id:message-id: to:to:cc:cc:mime-version:mime-version:content-type:content-type: content-transfer-encoding:content-transfer-encoding: in-reply-to:in-reply-to:references:references; bh=QDTtuz/iBLnx3uLIw0aNQbjjBf12gK6PRqFn//yUSp8=; b=NB2Jw+1/zvAUDFua4KIxw3i7AD/PZA6GWDGWzJ0BOo8NhKNLhoVNZ4t5ZOYTqMM4jwjott akyruVwwIJz9HdM2lfZuNGu9BD+q/8XAPf8tG0uVT5Lz2v4TIIbBqJ0lsIq6yzdmNVIk/C k2fko7R2N/jgiNf81F2rjkwGTTTbOWo= Received: from mail-qv1-f71.google.com (mail-qv1-f71.google.com [209.85.219.71]) by relay.mimecast.com with ESMTP with STARTTLS (version=TLSv1.3, cipher=TLS_AES_256_GCM_SHA384) id us-mta-288-P5uOxmQsOSisN7-fay1htg-1; Wed, 27 Mar 2024 11:23:51 -0400 X-MC-Unique: P5uOxmQsOSisN7-fay1htg-1 Received: by mail-qv1-f71.google.com with SMTP id 6a1803df08f44-69672754479so14006816d6.1 for ; Wed, 27 Mar 2024 08:23:51 -0700 (PDT) X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20230601; t=1711553031; x=1712157831; h=content-transfer-encoding:mime-version:references:in-reply-to :message-id:date:subject:cc:to:from:x-gm-message-state:from:to:cc :subject:date:message-id:reply-to; bh=QDTtuz/iBLnx3uLIw0aNQbjjBf12gK6PRqFn//yUSp8=; b=myjt5bpALoRRCo78UFOnhYzMoQVi/j6V5CyPAsTNrhok3BIB/F2LGbLmcvysE2P5js DFmxrdpV8L1j6BCFhpX3cqhfbQ3vfvJdTrLyGTjb71qg22BWXisyWJLtZOM402OsveEC pyVXtZgvpYb4PqTUzQ7E65kEuHGDgeWdNE/j7mktJEHX1CIQMsS+JSrs0EenfOz7UQf+ 66q3VUEjTxDfeHQ/UL6cG5Fi2GnSZUiQQhs9frMJn4BbMAjyCW+ERT1liIHIvmoD8744 GCF05XO4MB9kXDlcbpunVhOUaPqtJ6oW5laAwQFtDBDG58z/1o49sVmyBiN+OpQWPqVE duBA== X-Gm-Message-State: AOJu0Yx1s0chN5PllTn6AAQjETHHIcHbEu14W7adJT7eg92soqYibNVn No7IuCO7lD1E0sbhUWHy3dcR7tWhDkwcPrqo7rdSbRj9RZccoQuhJ4r/nlBO+WtzuNKNrIC0D5W WX3gAeq7QXZzoX3k2d7bstEyssx2ic72Pk3LAT4o/8vDbr8DVWE3kufwPQ3AUBKLY+dubwNo3Ma 4DqaLzSncZvt+p+468y5M+gGQoNg4l+w== X-Received: by 2002:a05:6214:2b86:b0:690:de72:316f with SMTP id kr6-20020a0562142b8600b00690de72316fmr14582177qvb.1.1711553030824; Wed, 27 Mar 2024 08:23:50 -0700 (PDT) X-Google-Smtp-Source: AGHT+IEWXp2F9loH7HSDfuOw6HflxlC5J0wiBYnMNwM1iqpFTIhT0c0cqQhWkIb89hM3i5euenCqpA== X-Received: by 2002:a05:6214:2b86:b0:690:de72:316f with SMTP id kr6-20020a0562142b8600b00690de72316fmr14582105qvb.1.1711553030174; Wed, 27 Mar 2024 08:23:50 -0700 (PDT) Received: from x1n.redhat.com ([99.254.121.117]) by smtp.gmail.com with ESMTPSA id hu4-20020a056214234400b00690dd47a41csm6412639qvb.86.2024.03.27.08.23.48 (version=TLS1_3 cipher=TLS_AES_256_GCM_SHA384 bits=256/256); Wed, 27 Mar 2024 08:23:49 -0700 (PDT) From: peterx@redhat.com To: linux-mm@kvack.org, linux-kernel@vger.kernel.org Cc: Yang Shi , "Kirill A . Shutemov" , Mike Kravetz , John Hubbard , Michael Ellerman , peterx@redhat.com, Andrew Jones , Muchun Song , linux-riscv@lists.infradead.org, linuxppc-dev@lists.ozlabs.org, Christophe Leroy , Andrew Morton , Christoph Hellwig , Lorenzo Stoakes , Matthew Wilcox , Rik van Riel , linux-arm-kernel@lists.infradead.org, Andrea Arcangeli , David Hildenbrand , "Aneesh Kumar K . V" , Vlastimil Babka , James Houghton , Jason Gunthorpe , Mike Rapoport , Axel Rasmussen Subject: [PATCH v4 07/13] mm/gup: Refactor record_subpages() to find 1st small page Date: Wed, 27 Mar 2024 11:23:26 -0400 Message-ID: <20240327152332.950956-8-peterx@redhat.com> X-Mailer: git-send-email 2.44.0 In-Reply-To: <20240327152332.950956-1-peterx@redhat.com> References: <20240327152332.950956-1-peterx@redhat.com> MIME-Version: 1.0 X-Mimecast-Spam-Score: 0 X-Mimecast-Originator: redhat.com X-Rspamd-Queue-Id: 8F2EA140026 X-Rspam-User: X-Stat-Signature: pyiufysnsogyrh1ij86riq7ww51rsjn3 X-Rspamd-Server: rspam03 X-HE-Tag: 1711553034-277285 X-HE-Meta: U2FsdGVkX1/l51ITm7OUtBSrQHbIsXqn8ioLhULPMPOb8Shn3YLxTlK2P8rQXgOLkwMARP3k6x8zUEQ79cTZp4vhrX6NC1ro80VTPUBtzLJB1NIBLtL7cgBneXoaROAQfsOnAykevDu7tj8mCxLQavAzgKkOnRzr+nu1YwLUm/U3Gn/zDrZrG66EFTtr3wRcbQy6eJ0IEPis5JH4dmNs1M1vfk+RgwB0E0EJV608bnwpz07jN6ZX9RDKHtjbeahl6GaDgAh/KR7ombMs+YImsEGAly3zZbZiRddVgc6NXBlHWduCWmpK/r8LXH9nzsmEe3BYEw6mUdGBvvc2oxO7R4zhJOhveavoMU4DXKdD+oFAV3Hp/F9Jlx/gO8ZKUwsME1h5MMBJVA5QD58PdgVQJrqiGM40723j0YH341aGIQrqjx6YIFITXJYqVEcnsYnvjDBrmVaQVjvMHiJ2m5E4Uz0fujfW5RLwDDzCLG+YEcRwkpBXKpiXSGoHuK+zAKPuXvpERB0og9Ia99I6dnYt8fcHiJV36yfvKhfQjVtSBjGREAYQgtDIxr4r8uVY8mSJIM5DBbWNVtsd55YABcjABUt1bpVPwxnwQPRGq8ueZQCb3fGxGeUQ6rjS9/ttRk3FUmgmK0VqYSBH/Nou8CUuBopSuG5W01QGTemww20YEwujH4VVaVIplKP60GcMgUWa8Yo4xl4K0x9+hTsxEnFmt53ILVOjkeBaSOc9z/qJz0hbiSkR8Qajiy/meSAGbt8IOEOv4DNvW9NaO3tiwIqipdx+gOXAQtffj5vSMfDAUzSj+EeoWZSA23pJ3X9cu3p1QggNp86xQ/u80bqDX2yuTipZ91Y9n4NjTCfn50/KdW59I9AVNBakXYrlljKdR0slO82NBZyhGODUrQSPvo+uihl+Lf6wEqe12vTgScFs+1X+OzKI0tkwYjRFB2DITG6RWzYzQKqnD/GB8lwcYAH KwY31Jkm EWqHxEPQHH9bwBq5SmsbgW8Yf36QZcxY7Gz7ucSwsJbUEZwvYLbDVlhBweASbq05MVTPYeBiw8t0hAiHhen9VqxXqGYBDHThelAkWzABJGupdH+2WSt6TdPWfQJIZNRIRzF9qgIvLq+uU86E= X-Bogosity: Ham, tests=bogofilter, spamicity=0.000000, version=1.2.4 Sender: owner-linux-mm@kvack.org Precedence: bulk X-Loop: owner-majordomo@kvack.org List-ID: List-Subscribe: List-Unsubscribe: From: Peter Xu All the fast-gup functions take a tail page to operate, always need to do page mask calculations before feeding that into record_subpages(). Merge that logic into record_subpages(), so that it will do the nth_page() calculation. Reviewed-by: Jason Gunthorpe Signed-off-by: Peter Xu --- mm/gup.c | 25 ++++++++++++++----------- 1 file changed, 14 insertions(+), 11 deletions(-) diff --git a/mm/gup.c b/mm/gup.c index db35b056fc9a..c2881772216b 100644 --- a/mm/gup.c +++ b/mm/gup.c @@ -2779,13 +2779,16 @@ static int __gup_device_huge_pud(pud_t pud, pud_t *pudp, unsigned long addr, } #endif -static int record_subpages(struct page *page, unsigned long addr, - unsigned long end, struct page **pages) +static int record_subpages(struct page *page, unsigned long sz, + unsigned long addr, unsigned long end, + struct page **pages) { + struct page *start_page; int nr; + start_page = nth_page(page, (addr & (sz - 1)) >> PAGE_SHIFT); for (nr = 0; addr != end; nr++, addr += PAGE_SIZE) - pages[nr] = nth_page(page, nr); + pages[nr] = nth_page(start_page, nr); return nr; } @@ -2820,8 +2823,8 @@ static int gup_hugepte(pte_t *ptep, unsigned long sz, unsigned long addr, /* hugepages are never "special" */ VM_BUG_ON(!pfn_valid(pte_pfn(pte))); - page = nth_page(pte_page(pte), (addr & (sz - 1)) >> PAGE_SHIFT); - refs = record_subpages(page, addr, end, pages + *nr); + page = pte_page(pte); + refs = record_subpages(page, sz, addr, end, pages + *nr); folio = try_grab_folio(page, refs, flags); if (!folio) @@ -2894,8 +2897,8 @@ static int gup_huge_pmd(pmd_t orig, pmd_t *pmdp, unsigned long addr, pages, nr); } - page = nth_page(pmd_page(orig), (addr & ~PMD_MASK) >> PAGE_SHIFT); - refs = record_subpages(page, addr, end, pages + *nr); + page = pmd_page(orig); + refs = record_subpages(page, PMD_SIZE, addr, end, pages + *nr); folio = try_grab_folio(page, refs, flags); if (!folio) @@ -2938,8 +2941,8 @@ static int gup_huge_pud(pud_t orig, pud_t *pudp, unsigned long addr, pages, nr); } - page = nth_page(pud_page(orig), (addr & ~PUD_MASK) >> PAGE_SHIFT); - refs = record_subpages(page, addr, end, pages + *nr); + page = pud_page(orig); + refs = record_subpages(page, PUD_SIZE, addr, end, pages + *nr); folio = try_grab_folio(page, refs, flags); if (!folio) @@ -2978,8 +2981,8 @@ static int gup_huge_pgd(pgd_t orig, pgd_t *pgdp, unsigned long addr, BUILD_BUG_ON(pgd_devmap(orig)); - page = nth_page(pgd_page(orig), (addr & ~PGDIR_MASK) >> PAGE_SHIFT); - refs = record_subpages(page, addr, end, pages + *nr); + page = pgd_page(orig); + refs = record_subpages(page, PGDIR_SIZE, addr, end, pages + *nr); folio = try_grab_folio(page, refs, flags); if (!folio) From patchwork Wed Mar 27 15:23:27 2024 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Peter Xu X-Patchwork-Id: 13606788 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from kanga.kvack.org (kanga.kvack.org [205.233.56.17]) by smtp.lore.kernel.org (Postfix) with ESMTP id 3D75EC54E67 for ; Wed, 27 Mar 2024 15:24:02 +0000 (UTC) Received: by kanga.kvack.org (Postfix) id 64ADC6B009D; Wed, 27 Mar 2024 11:23:58 -0400 (EDT) Received: by kanga.kvack.org (Postfix, from userid 40) id 560096B009F; Wed, 27 Mar 2024 11:23:58 -0400 (EDT) X-Delivered-To: int-list-linux-mm@kvack.org Received: by kanga.kvack.org (Postfix, from userid 63042) id 3B5136B00A0; Wed, 27 Mar 2024 11:23:58 -0400 (EDT) X-Delivered-To: linux-mm@kvack.org Received: from relay.hostedemail.com (smtprelay0012.hostedemail.com [216.40.44.12]) by kanga.kvack.org (Postfix) with ESMTP id 1BEBD6B009D for ; Wed, 27 Mar 2024 11:23:58 -0400 (EDT) Received: from smtpin16.hostedemail.com (a10.router.float.18 [10.200.18.1]) by unirelay03.hostedemail.com (Postfix) with ESMTP id CEA5CA03EC for ; Wed, 27 Mar 2024 15:23:57 +0000 (UTC) X-FDA: 81943189314.16.E10E9FF Received: from us-smtp-delivery-124.mimecast.com (us-smtp-delivery-124.mimecast.com [170.10.133.124]) by imf02.hostedemail.com (Postfix) with ESMTP id EC3BC80017 for ; Wed, 27 Mar 2024 15:23:55 +0000 (UTC) Authentication-Results: imf02.hostedemail.com; dkim=pass header.d=redhat.com header.s=mimecast20190719 header.b=De2WPSLD; dmarc=pass (policy=none) header.from=redhat.com; spf=pass (imf02.hostedemail.com: domain of peterx@redhat.com designates 170.10.133.124 as permitted sender) smtp.mailfrom=peterx@redhat.com ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=hostedemail.com; s=arc-20220608; t=1711553036; h=from:from:sender:reply-to:subject:subject:date:date: message-id:message-id:to:to:cc:cc:mime-version:mime-version: content-type:content-type: content-transfer-encoding:content-transfer-encoding: in-reply-to:in-reply-to:references:references:dkim-signature; bh=2/UoYTg8lz4bAl/L4a1cbQCJuXO8WYRlAVPmeKHJK2E=; b=Qx4l3jYiYJouVW8VK1tSiNbLVQxK1vpQOG4MebKYcrUbBNzuWGbKJgsWFyAUePx3ybLPie TjlblM5XRENIUj2/cx9Z/1VzHo+FNvYOyrSgP7t1ExhmIx302a7VWJWlxDOQzAPNlsTbKs lw5v9UeEtBOggLTrf6nwQAePkw4Crxk= ARC-Authentication-Results: i=1; imf02.hostedemail.com; dkim=pass header.d=redhat.com header.s=mimecast20190719 header.b=De2WPSLD; dmarc=pass (policy=none) header.from=redhat.com; spf=pass (imf02.hostedemail.com: domain of peterx@redhat.com designates 170.10.133.124 as permitted sender) smtp.mailfrom=peterx@redhat.com ARC-Seal: i=1; s=arc-20220608; d=hostedemail.com; t=1711553036; a=rsa-sha256; cv=none; b=BdD/L3FmdvdOgc5loTiklg6gPmtxDArgH7SMdAm2UC9cS6izdY6yUvisPiNDY+aSiCsVnB M3A+YfqUXZtQ/KfGXDkSEXnGnmfjHbrtEPzxqJmEgTGReF843oQ1etNTuMOKVT8KKVLN9G IrrtQR2ujRW2CurDn4+G4vVNoL1QhB4= DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=redhat.com; s=mimecast20190719; t=1711553035; h=from:from:reply-to:subject:subject:date:date:message-id:message-id: to:to:cc:cc:mime-version:mime-version:content-type:content-type: content-transfer-encoding:content-transfer-encoding: in-reply-to:in-reply-to:references:references; bh=2/UoYTg8lz4bAl/L4a1cbQCJuXO8WYRlAVPmeKHJK2E=; b=De2WPSLDQBlryuPjCl/FjvTSJQdZboyR9+Z0CJEllaeg7GCX/IxfdKPDsm1xQ6TWFSwe4A q577qKZm5SRfqW1bp4tJ0pDkwCWRoVE3teBMkenAQVJtVWUKCTUeL1nWFGmTBVFMP273dx 5/P8nZNjJV8wHbM/Xw7leV7MJJGuCII= Received: from mail-qk1-f198.google.com (mail-qk1-f198.google.com [209.85.222.198]) by relay.mimecast.com with ESMTP with STARTTLS (version=TLSv1.3, cipher=TLS_AES_256_GCM_SHA384) id us-mta-508-xKCepNNQNqCHX1TapyOK7Q-1; Wed, 27 Mar 2024 11:23:54 -0400 X-MC-Unique: xKCepNNQNqCHX1TapyOK7Q-1 Received: by mail-qk1-f198.google.com with SMTP id af79cd13be357-7830635331bso149215685a.1 for ; Wed, 27 Mar 2024 08:23:54 -0700 (PDT) X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20230601; t=1711553033; x=1712157833; h=content-transfer-encoding:mime-version:references:in-reply-to :message-id:date:subject:cc:to:from:x-gm-message-state:from:to:cc :subject:date:message-id:reply-to; bh=2/UoYTg8lz4bAl/L4a1cbQCJuXO8WYRlAVPmeKHJK2E=; b=E65I1WeN/X7AWT/LzWsPqGwQpF8xhS+5LIEU9Fw2cBAGSwn6UCs6Lx5mfBk5YutccC S3KqCvKk3u79dzcflpigh9GyzQdtLY9bGPqNcE55NzJHY+bUq4qL1qw7853IFweFfuiF hsSMQEVDIlZdWVx6AZFjjswRXzr3qBklxUKE66Ks8K9Ckwk/37u9ZOv4XMv5k9749Hwq Oi6Ye6ph4Yo+gXLFyqLOvmT+zY2fMXqoYauvGeHHy3oaRWQIR/Vw5AYoDr/wLiKva0Xz yStjNaM3bj4h80I2UzvgHcBMMMHQkOLHA2o/Yod6kQMeM4Qp9rtKKHjykIKxQzJzWuwq tbRA== X-Gm-Message-State: AOJu0YyNTrTRWAL78cuZf3b1fRHPWkQQPZP35yUiJl8gJgiJZFtXhBfq 0p+R4tvbzKpXXbhBGC8NaSs7kY0hBDTlaHEBglIDZdnfpDBajLXJiv1vq6luI5nTcaiRw/Z1JBU oiHTsIXJWqghsk4vCIbVlcaEitXujBrC5bO8Po9cFhImVcV4al0CR1lNxxb1aS+YHw9GB0XH5uQ DT1hYtdjRnB3N7y3ujPzvpecGm8UgH+g== X-Received: by 2002:a05:6214:4a8a:b0:696:ad00:7deb with SMTP id pi10-20020a0562144a8a00b00696ad007debmr2078377qvb.2.1711553033327; Wed, 27 Mar 2024 08:23:53 -0700 (PDT) X-Google-Smtp-Source: AGHT+IH8Fh1FPXUmWlSE4Ssu3coIMvek7Et1xEVKx+8QTY39hYxqq3mRFRgZ7bz+i1V9Jmd/zYMv6Q== X-Received: by 2002:a05:6214:4a8a:b0:696:ad00:7deb with SMTP id pi10-20020a0562144a8a00b00696ad007debmr2078325qvb.2.1711553032687; Wed, 27 Mar 2024 08:23:52 -0700 (PDT) Received: from x1n.redhat.com ([99.254.121.117]) by smtp.gmail.com with ESMTPSA id hu4-20020a056214234400b00690dd47a41csm6412639qvb.86.2024.03.27.08.23.50 (version=TLS1_3 cipher=TLS_AES_256_GCM_SHA384 bits=256/256); Wed, 27 Mar 2024 08:23:52 -0700 (PDT) From: peterx@redhat.com To: linux-mm@kvack.org, linux-kernel@vger.kernel.org Cc: Yang Shi , "Kirill A . Shutemov" , Mike Kravetz , John Hubbard , Michael Ellerman , peterx@redhat.com, Andrew Jones , Muchun Song , linux-riscv@lists.infradead.org, linuxppc-dev@lists.ozlabs.org, Christophe Leroy , Andrew Morton , Christoph Hellwig , Lorenzo Stoakes , Matthew Wilcox , Rik van Riel , linux-arm-kernel@lists.infradead.org, Andrea Arcangeli , David Hildenbrand , "Aneesh Kumar K . V" , Vlastimil Babka , James Houghton , Jason Gunthorpe , Mike Rapoport , Axel Rasmussen Subject: [PATCH v4 08/13] mm/gup: Handle hugetlb for no_page_table() Date: Wed, 27 Mar 2024 11:23:27 -0400 Message-ID: <20240327152332.950956-9-peterx@redhat.com> X-Mailer: git-send-email 2.44.0 In-Reply-To: <20240327152332.950956-1-peterx@redhat.com> References: <20240327152332.950956-1-peterx@redhat.com> MIME-Version: 1.0 X-Mimecast-Spam-Score: 0 X-Mimecast-Originator: redhat.com X-Rspam-User: X-Stat-Signature: wqyu57crcw9wwwzpaest38obstcwq6cs X-Rspamd-Server: rspam07 X-Rspamd-Queue-Id: EC3BC80017 X-HE-Tag: 1711553035-335269 X-HE-Meta: U2FsdGVkX19FhfA+0TUImFFo9oLuPrZq/reHnKL4M2XFyCuaif3AIIAv0yoLduw8QG7Qa4gJ2zU4MiFFDslctRqW/b67BOa0HamCzJJcTnVKRE0jhaDtOtXzXd4faOuuJLAxNPxLNNlGG3aVAakE33756IiabjIIRT+TrFb4R1XdgT3VizlGo4rmlBSTpsst8YKvQ2ZvvAOOzAMV24J0CHeoqPkWdNmKQhk8ghB+wL3J5Y0D5ytq0KnPkR23b8cFPJiN2lVRfBIVSycZeTOkiQn+ejLdIWBqWKVJD2gbk+opaEgIC1aY1q17XiBrzKOGXlZz6Sjc1FJtSz2GKutoj8BM4o2O8gIZ+KIG8YJMKeqOp9QO5iA3bQCN2XBE2O63J/Zj6fZFcSDrO0fJ739eYJZBd7rMMA7smfY4dbQ5LBVyIfmGAkT08B0U+LNYEe4rXMdZdpr/UtuhGnAGwqcAWjurTucU6KQZek2RXH4GUJhNFAmIXXXk1kRz8fyCKvDKyTe7ShJpHMMq01dG9kEx5Bh8dNvUXUj+hrnE12srfO3RU/Uvr/ml2OdQxWmG6jWmf/+Nh7BWiYVm/DnN8630uuYsbQk/uPeiYw9CvFWTiWWmWc2mDVdhg/D/s7JQsEGs5RLD0OEKec7mfKQRBVOQ6qapSnYDxM7YUWIylkbz8R8XSuIIgxDYSFneLUVB+Fs80F3azUUBcTuKk2OXDDqdDRoyICT0AJ2HMQvFijm7YLdlzy0/n1Pi2pEkKgRWxD8YhDBeOKFG2pqRQ4FtlCGx0MnOP9ZgvMvvK2IcEngavXj/hQTFBKKs4unxcjBxi5i0R90BXYXQJ3yIw7Z2qcJuWCmFngyaqfeQTi06LGYPROO771HNUxBW3j6LliEKrpwXF/7iI/6PAUMnjHa5jhBoby39zW2Y6kbv6ciZ7dBkiBXlN0d4VYXpZIcePFpq4aOfgDL67a9xjsSYilac4cX j4Pctjy+ VyGD64tAb9qGw7iUH4cSeX9jN+7+9OkTvibNAthlRZ9mfOHM+VlYgiX6+H13OR4kNHqUAQqBsEHn4Oe/5G8mCJVzHw+ZgZHVRM+0hCS+N2drl2557YMIbRsUU52EnMRfRKU3rqRe6nPXIR90= X-Bogosity: Ham, tests=bogofilter, spamicity=0.000000, version=1.2.4 Sender: owner-linux-mm@kvack.org Precedence: bulk X-Loop: owner-majordomo@kvack.org List-ID: List-Subscribe: List-Unsubscribe: From: Peter Xu no_page_table() is not yet used for hugetlb code paths. Make it prepared. The major difference here is hugetlb will return -EFAULT as long as page cache does not exist, even if VM_SHARED. See hugetlb_follow_page_mask(). Pass "address" into no_page_table() too, as hugetlb will need it. Reviewed-by: Christoph Hellwig Reviewed-by: Jason Gunthorpe Signed-off-by: Peter Xu --- mm/gup.c | 44 ++++++++++++++++++++++++++------------------ 1 file changed, 26 insertions(+), 18 deletions(-) diff --git a/mm/gup.c b/mm/gup.c index c2881772216b..ef46a7053e16 100644 --- a/mm/gup.c +++ b/mm/gup.c @@ -501,19 +501,27 @@ static inline void mm_set_has_pinned_flag(unsigned long *mm_flags) #ifdef CONFIG_MMU static struct page *no_page_table(struct vm_area_struct *vma, - unsigned int flags) + unsigned int flags, unsigned long address) { + if (!(flags & FOLL_DUMP)) + return NULL; + /* - * When core dumping an enormous anonymous area that nobody - * has touched so far, we don't want to allocate unnecessary pages or + * When core dumping, we don't want to allocate unnecessary pages or * page tables. Return error instead of NULL to skip handle_mm_fault, * then get_dump_page() will return NULL to leave a hole in the dump. * But we can only make this optimization where a hole would surely * be zero-filled if handle_mm_fault() actually did handle it. */ - if ((flags & FOLL_DUMP) && - (vma_is_anonymous(vma) || !vma->vm_ops->fault)) + if (is_vm_hugetlb_page(vma)) { + struct hstate *h = hstate_vma(vma); + + if (!hugetlbfs_pagecache_present(h, vma, address)) + return ERR_PTR(-EFAULT); + } else if ((vma_is_anonymous(vma) || !vma->vm_ops->fault)) { return ERR_PTR(-EFAULT); + } + return NULL; } @@ -593,7 +601,7 @@ static struct page *follow_page_pte(struct vm_area_struct *vma, ptep = pte_offset_map_lock(mm, pmd, address, &ptl); if (!ptep) - return no_page_table(vma, flags); + return no_page_table(vma, flags, address); pte = ptep_get(ptep); if (!pte_present(pte)) goto no_page; @@ -685,7 +693,7 @@ static struct page *follow_page_pte(struct vm_area_struct *vma, pte_unmap_unlock(ptep, ptl); if (!pte_none(pte)) return NULL; - return no_page_table(vma, flags); + return no_page_table(vma, flags, address); } static struct page *follow_pmd_mask(struct vm_area_struct *vma, @@ -701,27 +709,27 @@ static struct page *follow_pmd_mask(struct vm_area_struct *vma, pmd = pmd_offset(pudp, address); pmdval = pmdp_get_lockless(pmd); if (pmd_none(pmdval)) - return no_page_table(vma, flags); + return no_page_table(vma, flags, address); if (!pmd_present(pmdval)) - return no_page_table(vma, flags); + return no_page_table(vma, flags, address); if (pmd_devmap(pmdval)) { ptl = pmd_lock(mm, pmd); page = follow_devmap_pmd(vma, address, pmd, flags, &ctx->pgmap); spin_unlock(ptl); if (page) return page; - return no_page_table(vma, flags); + return no_page_table(vma, flags, address); } if (likely(!pmd_trans_huge(pmdval))) return follow_page_pte(vma, address, pmd, flags, &ctx->pgmap); if (pmd_protnone(pmdval) && !gup_can_follow_protnone(vma, flags)) - return no_page_table(vma, flags); + return no_page_table(vma, flags, address); ptl = pmd_lock(mm, pmd); if (unlikely(!pmd_present(*pmd))) { spin_unlock(ptl); - return no_page_table(vma, flags); + return no_page_table(vma, flags, address); } if (unlikely(!pmd_trans_huge(*pmd))) { spin_unlock(ptl); @@ -752,17 +760,17 @@ static struct page *follow_pud_mask(struct vm_area_struct *vma, pud = pud_offset(p4dp, address); if (pud_none(*pud)) - return no_page_table(vma, flags); + return no_page_table(vma, flags, address); if (pud_devmap(*pud)) { ptl = pud_lock(mm, pud); page = follow_devmap_pud(vma, address, pud, flags, &ctx->pgmap); spin_unlock(ptl); if (page) return page; - return no_page_table(vma, flags); + return no_page_table(vma, flags, address); } if (unlikely(pud_bad(*pud))) - return no_page_table(vma, flags); + return no_page_table(vma, flags, address); return follow_pmd_mask(vma, address, pud, flags, ctx); } @@ -777,10 +785,10 @@ static struct page *follow_p4d_mask(struct vm_area_struct *vma, p4dp = p4d_offset(pgdp, address); p4d = READ_ONCE(*p4dp); if (!p4d_present(p4d)) - return no_page_table(vma, flags); + return no_page_table(vma, flags, address); BUILD_BUG_ON(p4d_leaf(p4d)); if (unlikely(p4d_bad(p4d))) - return no_page_table(vma, flags); + return no_page_table(vma, flags, address); return follow_pud_mask(vma, address, p4dp, flags, ctx); } @@ -830,7 +838,7 @@ static struct page *follow_page_mask(struct vm_area_struct *vma, pgd = pgd_offset(mm, address); if (pgd_none(*pgd) || unlikely(pgd_bad(*pgd))) - return no_page_table(vma, flags); + return no_page_table(vma, flags, address); return follow_p4d_mask(vma, address, pgd, flags, ctx); } From patchwork Wed Mar 27 15:23:28 2024 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Peter Xu X-Patchwork-Id: 13606790 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from kanga.kvack.org (kanga.kvack.org [205.233.56.17]) by smtp.lore.kernel.org (Postfix) with ESMTP id AE8A1C47DD9 for ; Wed, 27 Mar 2024 15:24:07 +0000 (UTC) Received: by kanga.kvack.org (Postfix) id E13426B00A0; Wed, 27 Mar 2024 11:24:00 -0400 (EDT) Received: by kanga.kvack.org (Postfix, from userid 40) id D9BE66B00A1; Wed, 27 Mar 2024 11:24:00 -0400 (EDT) X-Delivered-To: int-list-linux-mm@kvack.org Received: by kanga.kvack.org (Postfix, from userid 63042) id BC7896B00A2; Wed, 27 Mar 2024 11:24:00 -0400 (EDT) X-Delivered-To: linux-mm@kvack.org Received: from relay.hostedemail.com (smtprelay0016.hostedemail.com [216.40.44.16]) by kanga.kvack.org (Postfix) with ESMTP id 98A856B00A0 for ; Wed, 27 Mar 2024 11:24:00 -0400 (EDT) Received: from smtpin04.hostedemail.com (a10.router.float.18 [10.200.18.1]) by unirelay02.hostedemail.com (Postfix) with ESMTP id 6DC34120767 for ; Wed, 27 Mar 2024 15:24:00 +0000 (UTC) X-FDA: 81943189440.04.9D09155 Received: from us-smtp-delivery-124.mimecast.com (us-smtp-delivery-124.mimecast.com [170.10.133.124]) by imf11.hostedemail.com (Postfix) with ESMTP id 58B1D40026 for ; Wed, 27 Mar 2024 15:23:58 +0000 (UTC) Authentication-Results: imf11.hostedemail.com; dkim=pass header.d=redhat.com header.s=mimecast20190719 header.b=eey1M4R5; dmarc=pass (policy=none) header.from=redhat.com; spf=pass (imf11.hostedemail.com: domain of peterx@redhat.com designates 170.10.133.124 as permitted sender) smtp.mailfrom=peterx@redhat.com ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=hostedemail.com; s=arc-20220608; t=1711553038; h=from:from:sender:reply-to:subject:subject:date:date: message-id:message-id:to:to:cc:cc:mime-version:mime-version: content-type:content-type: content-transfer-encoding:content-transfer-encoding: in-reply-to:in-reply-to:references:references:dkim-signature; bh=0ClTZB7edrSwahK3rulsC63VROBcD1QOdNuRXa61Ddg=; b=jGC+pi4jabcEo+O+1RkKIOiYbaCYQURKKYTaH2yAmS2mTTeG2Kk5DrnP7M/H5oqF8nIxCk L+Xnrecew7rMsmMgz7071l9vFiqlPC85R1a6+o7SBOVbA0UFimH+fI3T2HZkwI+Ed0RmiT IUivXMgPo7wO7PW0uzgu+2z6WD5uvtE= ARC-Authentication-Results: i=1; imf11.hostedemail.com; dkim=pass header.d=redhat.com header.s=mimecast20190719 header.b=eey1M4R5; dmarc=pass (policy=none) header.from=redhat.com; spf=pass (imf11.hostedemail.com: domain of peterx@redhat.com designates 170.10.133.124 as permitted sender) smtp.mailfrom=peterx@redhat.com ARC-Seal: i=1; s=arc-20220608; d=hostedemail.com; t=1711553038; a=rsa-sha256; cv=none; b=FZGxuDIXJunZyz50FogRaVXdsSwUDbFQB5fGTxePgRYbqt2daV2apsa5fuN8sDpXiz/jny lnfJvOFc8WfRTsAQtasfR1CbeDF+oSujzp59m420aQ3i43xid5hKwIG9dWhRUEZpZaOiim JieWncBtkKso8TKyA3yN/6SGZnyd2mk= DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=redhat.com; s=mimecast20190719; t=1711553037; h=from:from:reply-to:subject:subject:date:date:message-id:message-id: to:to:cc:cc:mime-version:mime-version:content-type:content-type: content-transfer-encoding:content-transfer-encoding: in-reply-to:in-reply-to:references:references; bh=0ClTZB7edrSwahK3rulsC63VROBcD1QOdNuRXa61Ddg=; b=eey1M4R5QdNvghzCyAdpKgxR3UFQrRHlhz9ErR5bJO0cPAVauYXEwkGN6HpoM+aBd2wFoC 9wWwEbFNUz4HELmc1fbrJj3MTjTXVnOwYJobAwPCNJH7XJatZhm9aJmpUAyKkjBVnpaElq OCsQp2h8LA6Ph/R33xnjDmPxFILHH+o= Received: from mail-qv1-f70.google.com (mail-qv1-f70.google.com [209.85.219.70]) by relay.mimecast.com with ESMTP with STARTTLS (version=TLSv1.3, cipher=TLS_AES_256_GCM_SHA384) id us-mta-70-D6PYM3rFOqK5OvUKrE_gPQ-1; Wed, 27 Mar 2024 11:23:56 -0400 X-MC-Unique: D6PYM3rFOqK5OvUKrE_gPQ-1 Received: by mail-qv1-f70.google.com with SMTP id 6a1803df08f44-69672754479so14007386d6.1 for ; Wed, 27 Mar 2024 08:23:56 -0700 (PDT) X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20230601; t=1711553035; x=1712157835; h=content-transfer-encoding:mime-version:references:in-reply-to :message-id:date:subject:cc:to:from:x-gm-message-state:from:to:cc :subject:date:message-id:reply-to; bh=0ClTZB7edrSwahK3rulsC63VROBcD1QOdNuRXa61Ddg=; b=QcV5E6ZHuyBSI79Zvwxr35B+NS4kGH2V2oS6eIyFck1jRmcV10acspgmfnwZjfuTaM oKGJgBAa//OdxK01aZZR0QjTtZtPuv4OLMiaSeKVTzIl4AxwKpJVxRv5jEmD8uRLLBmZ uEZ8ULurRxB98ozurCTAa+guu7sL/+an0aXAdXAs15j96RhGc6s/Tgkmhx1EfvxvR7HW rerzMcaKvCaRBBI1Ex0sgTgnijY+DaebAN7dRKKyLwuA2xgyG+HA/YUtN9LOpo55aMdF fjGZx6DDaT2a1qYYDAN7M8Fe9Ev1lasIY4O1JSq7QeG6osQC6hFa7lQyy3Xw34Z8Sa8t NsPg== X-Gm-Message-State: AOJu0Ywkyx4hQJl+wpNPQR+yT2Ykrh20XpbZCqGtHU7WQ++6E/uQejBC AwBPSx8XMplkcQ+R8qYLa4FBL3gWFujlqf/M1QZiJwciS0dpGmv6bvMkQ4gx+ntDSoQgC9vmNzI 6gfIFkxJnTxpgB+A2gYalLJdHtU3hUF2uQg9k0ZOXIddxGMq7uWjv72uL5Vmu+mJl/FEq3HQ4kU V215g63xOzYEJcscpZWSV9SSUZdiox1A== X-Received: by 2002:a05:6214:3d8c:b0:696:1892:c19f with SMTP id om12-20020a0562143d8c00b006961892c19fmr15035567qvb.3.1711553035194; Wed, 27 Mar 2024 08:23:55 -0700 (PDT) X-Google-Smtp-Source: AGHT+IFT3GjlhPAIM0f0tWP3PNax9bFYBunh5winrPqMP8FazkZOSJ/P00xdKeX4lX2Up3S+10Z8OQ== X-Received: by 2002:a05:6214:3d8c:b0:696:1892:c19f with SMTP id om12-20020a0562143d8c00b006961892c19fmr15035509qvb.3.1711553034516; Wed, 27 Mar 2024 08:23:54 -0700 (PDT) Received: from x1n.redhat.com ([99.254.121.117]) by smtp.gmail.com with ESMTPSA id hu4-20020a056214234400b00690dd47a41csm6412639qvb.86.2024.03.27.08.23.52 (version=TLS1_3 cipher=TLS_AES_256_GCM_SHA384 bits=256/256); Wed, 27 Mar 2024 08:23:54 -0700 (PDT) From: peterx@redhat.com To: linux-mm@kvack.org, linux-kernel@vger.kernel.org Cc: Yang Shi , "Kirill A . Shutemov" , Mike Kravetz , John Hubbard , Michael Ellerman , peterx@redhat.com, Andrew Jones , Muchun Song , linux-riscv@lists.infradead.org, linuxppc-dev@lists.ozlabs.org, Christophe Leroy , Andrew Morton , Christoph Hellwig , Lorenzo Stoakes , Matthew Wilcox , Rik van Riel , linux-arm-kernel@lists.infradead.org, Andrea Arcangeli , David Hildenbrand , "Aneesh Kumar K . V" , Vlastimil Babka , James Houghton , Jason Gunthorpe , Mike Rapoport , Axel Rasmussen Subject: [PATCH v4 09/13] mm/gup: Cache *pudp in follow_pud_mask() Date: Wed, 27 Mar 2024 11:23:28 -0400 Message-ID: <20240327152332.950956-10-peterx@redhat.com> X-Mailer: git-send-email 2.44.0 In-Reply-To: <20240327152332.950956-1-peterx@redhat.com> References: <20240327152332.950956-1-peterx@redhat.com> MIME-Version: 1.0 X-Mimecast-Spam-Score: 0 X-Mimecast-Originator: redhat.com X-Rspamd-Queue-Id: 58B1D40026 X-Rspam-User: X-Rspamd-Server: rspam05 X-Stat-Signature: d8yhtizhcw7f7k7rzfnzintssenxrtzu X-HE-Tag: 1711553038-54553 X-HE-Meta: U2FsdGVkX1/dZhCZwpFdog4luMThc5gWgox+0Ye0waxmtX2VNuqIlNdhDoK9t9aIAaxVdotLNNLojg6HY7Tlo5AuovGyAWsujpLzak0TvJkP62saT4+PN+icKXHuxc3YD3ecE50VxUWhfpVeXBiiD3xE0ICDcATdCfJe8N9JxmkqDlA2n9zJ9b+72d75XegAxYAeqy08oD0WYpF6gMMGY//BI7TxuOFdM8/NE8ilmbrOhDkjAresYuV+MOv23trHnCJanNAB+yKjGddLi8rkMrzfqy7xCtBbJbNG+n96iQhD2yDEN3p30jZ9DuPYwbBydd1A/4bAcHFnUkjb7QlTbyuIJUTKqTn5PwR18Xddcosvy68l7VnXcbOncQXkcTp8/vonX8UwFmBuXnaXmPcAxCto1in7upgiqbEYge/Zp6QGVkKMIfUvd929PcGwzMx1yMx2uF9OZ0YcYXiY7OQC5wXtCsU159lAA6bk+UCj/08yoQEi/FFdsUaNbRBfSQFJMuiAGLsF6poeDGAlO6YD5zpOgabuR9UDatAD4MROXr9Ltbe3S+i7pnXiqeCeYykZ9g0+zPM5tQj/Ht80oeqQTTt0PglJFnvw/slOSMtYXA1WCVZbFNn4NcVVeS1g6jQayDb2CmbeJR6qXseRScdBgpCAlhf7bF6nbsEbTl86J1K5hJFdsuykYT72DbKva+75dRIPKDUN2nAtK5tfVaWRp2pulkwGtum4zTVeyukSrftaAwBj5VX0VpfD43KuWqZkS8/S9PE+/iLf9fMM38zTsrdr24YUDXIc9LQbgxUemAurya/VoCrAYetcLvluLs/wKrQuWSIVmDPpTMc4RiHtivvjSK4QHogNsN/iHAbd2BZLRcehrv6o79U3rXzs8+7HS7/92nH/tALR+RBRtEGVe8VKKSNEsA+RP3FioYiy0T3FoidVpSUcYJSN1G0mUUoX7eBaZbSEeS9mHKO8qiE hil+C6IV sG2GJ0rJKDuOFAk8VMLgd5L75Vh0dadMITKYnheqZQoeWnVukNW0dPO9L9LhUE+M3/HHqAKP60MFaQqDeoIZfJuU+9c7Ea3oLx3EnC7bfaV2zM+7ptmj1sFkDnmg30G1HNoBe4J+8OdGVgfLqiN7zvM8+TWadPlzSp42ZxnmeZCLEgnDax8l8GVwULrnuV2Er+xa+LSyJrws+AB19OVsOwfqBaPMECZwYlke6apo0xrJRj6SfkpV6hupJVMJT4dqrZCf2K419VXxlBh49Qwt0NmiL1MqwGKPPLdT7slPxFxPTEzkhlptZ3wX0TDYEpcy6SlyjcNxPDBn95TnW2wcztndH25xR117/ttTojYFg2xjaslkbzCEvvXcouVEN2VShbNHPw6snsF6r5PBKVHVsjksYGcxqQVmGcAtAYOsPh1zg8Py0YymfXmcHkVye0lWgKMc5j6/rhAn3xt6HvBOOvP3LKxvv6BJ2tTH5 X-Bogosity: Ham, tests=bogofilter, spamicity=0.000000, version=1.2.4 Sender: owner-linux-mm@kvack.org Precedence: bulk X-Loop: owner-majordomo@kvack.org List-ID: List-Subscribe: List-Unsubscribe: From: Peter Xu Introduce "pud_t pud" in the function, so the code won't dereference *pudp multiple time. Not only because that looks less straightforward, but also because if the dereference really happened, it's not clear whether there can be race to see different *pudp values if it's being modified at the same time. Acked-by: James Houghton Reviewed-by: Jason Gunthorpe Signed-off-by: Peter Xu --- mm/gup.c | 17 +++++++++-------- 1 file changed, 9 insertions(+), 8 deletions(-) diff --git a/mm/gup.c b/mm/gup.c index ef46a7053e16..26b8cca24077 100644 --- a/mm/gup.c +++ b/mm/gup.c @@ -753,26 +753,27 @@ static struct page *follow_pud_mask(struct vm_area_struct *vma, unsigned int flags, struct follow_page_context *ctx) { - pud_t *pud; + pud_t *pudp, pud; spinlock_t *ptl; struct page *page; struct mm_struct *mm = vma->vm_mm; - pud = pud_offset(p4dp, address); - if (pud_none(*pud)) + pudp = pud_offset(p4dp, address); + pud = READ_ONCE(*pudp); + if (pud_none(pud)) return no_page_table(vma, flags, address); - if (pud_devmap(*pud)) { - ptl = pud_lock(mm, pud); - page = follow_devmap_pud(vma, address, pud, flags, &ctx->pgmap); + if (pud_devmap(pud)) { + ptl = pud_lock(mm, pudp); + page = follow_devmap_pud(vma, address, pudp, flags, &ctx->pgmap); spin_unlock(ptl); if (page) return page; return no_page_table(vma, flags, address); } - if (unlikely(pud_bad(*pud))) + if (unlikely(pud_bad(pud))) return no_page_table(vma, flags, address); - return follow_pmd_mask(vma, address, pud, flags, ctx); + return follow_pmd_mask(vma, address, pudp, flags, ctx); } static struct page *follow_p4d_mask(struct vm_area_struct *vma, From patchwork Wed Mar 27 15:23:29 2024 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Peter Xu X-Patchwork-Id: 13606791 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from kanga.kvack.org (kanga.kvack.org [205.233.56.17]) by smtp.lore.kernel.org (Postfix) with ESMTP id CCC6DC47DD9 for ; Wed, 27 Mar 2024 15:24:10 +0000 (UTC) Received: by kanga.kvack.org (Postfix) id 737296B00A1; Wed, 27 Mar 2024 11:24:02 -0400 (EDT) Received: by kanga.kvack.org (Postfix, from userid 40) id 6BD676B00A3; Wed, 27 Mar 2024 11:24:02 -0400 (EDT) X-Delivered-To: int-list-linux-mm@kvack.org Received: by kanga.kvack.org (Postfix, from userid 63042) id 4EA106B00A4; Wed, 27 Mar 2024 11:24:02 -0400 (EDT) X-Delivered-To: linux-mm@kvack.org Received: from relay.hostedemail.com (smtprelay0011.hostedemail.com [216.40.44.11]) by kanga.kvack.org (Postfix) with ESMTP id 2BC9B6B00A1 for ; Wed, 27 Mar 2024 11:24:02 -0400 (EDT) Received: from smtpin16.hostedemail.com (a10.router.float.18 [10.200.18.1]) by unirelay09.hostedemail.com (Postfix) with ESMTP id E562E80A6B for ; Wed, 27 Mar 2024 15:24:01 +0000 (UTC) X-FDA: 81943189482.16.AB3DF6A Received: from us-smtp-delivery-124.mimecast.com (us-smtp-delivery-124.mimecast.com [170.10.133.124]) by imf26.hostedemail.com (Postfix) with ESMTP id 00413140020 for ; Wed, 27 Mar 2024 15:23:59 +0000 (UTC) Authentication-Results: imf26.hostedemail.com; dkim=pass header.d=redhat.com header.s=mimecast20190719 header.b=BSGJl9wx; spf=pass (imf26.hostedemail.com: domain of peterx@redhat.com designates 170.10.133.124 as permitted sender) smtp.mailfrom=peterx@redhat.com; dmarc=pass (policy=none) header.from=redhat.com ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=hostedemail.com; s=arc-20220608; t=1711553040; h=from:from:sender:reply-to:subject:subject:date:date: message-id:message-id:to:to:cc:cc:mime-version:mime-version: content-type:content-type: content-transfer-encoding:content-transfer-encoding: in-reply-to:in-reply-to:references:references:dkim-signature; bh=ldvmB2LhfsIRddCCGOUltuspZLyBYJ6Ab85tSc3WzTs=; b=tS49UKCYQRUkiW+CougM+/wzc3Fex+S5iyL668F1lJ8OKMJbKPMd185QU/h7f6A81oq9PN IzuLLeJyNH23ZY+Oewuo7VJril/QjwmeQ1XQqjZxszbl60B/i6T2p7FVdbXqjmM9EysP+0 GzZ5C0yzbYEw1xJoOERHJnfU6NTrs84= ARC-Authentication-Results: i=1; imf26.hostedemail.com; dkim=pass header.d=redhat.com header.s=mimecast20190719 header.b=BSGJl9wx; spf=pass (imf26.hostedemail.com: domain of peterx@redhat.com designates 170.10.133.124 as permitted sender) smtp.mailfrom=peterx@redhat.com; dmarc=pass (policy=none) header.from=redhat.com ARC-Seal: i=1; s=arc-20220608; d=hostedemail.com; t=1711553040; a=rsa-sha256; cv=none; b=VtkgEyeQw/K5JC/GxJbT3o53MvyHeEVYXsET0QZ/SjBGJLVp+8A3m3csTro9n78nO3qEEH WZJaue7aUcaLbm/VZZp8NSg2U28ATMyx6V4olmDxAmZI1/lq5F1gSxKztXLRQHsvrtVwLc C75xp53GQ8WwvPRunI5VTZ4bPnECP44= DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=redhat.com; s=mimecast20190719; t=1711553039; h=from:from:reply-to:subject:subject:date:date:message-id:message-id: to:to:cc:cc:mime-version:mime-version:content-type:content-type: content-transfer-encoding:content-transfer-encoding: in-reply-to:in-reply-to:references:references; bh=ldvmB2LhfsIRddCCGOUltuspZLyBYJ6Ab85tSc3WzTs=; b=BSGJl9wxcjjAjeAVZsrZtKz/inf8oY4eNgmD7SzvW9j8nX1r6qONCLUACsEcD9j+p6vZx5 bJf3lXQ1oePSHc+cLgmQW+EBgjyHATvgXsiQO2kQvT0cwBCVyiWVGXTarhhLhQ+bpSKDBA hZWmAnR3ci3QXQuYPOW46iUQhgBcBsM= Received: from mail-qv1-f71.google.com (mail-qv1-f71.google.com [209.85.219.71]) by relay.mimecast.com with ESMTP with STARTTLS (version=TLSv1.3, cipher=TLS_AES_256_GCM_SHA384) id us-mta-266-ltTAhUhMP7u23L-pAkafqw-1; Wed, 27 Mar 2024 11:23:58 -0400 X-MC-Unique: ltTAhUhMP7u23L-pAkafqw-1 Received: by mail-qv1-f71.google.com with SMTP id 6a1803df08f44-690ab2b0de1so15530976d6.0 for ; Wed, 27 Mar 2024 08:23:58 -0700 (PDT) X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20230601; t=1711553037; x=1712157837; h=content-transfer-encoding:mime-version:references:in-reply-to :message-id:date:subject:cc:to:from:x-gm-message-state:from:to:cc :subject:date:message-id:reply-to; bh=ldvmB2LhfsIRddCCGOUltuspZLyBYJ6Ab85tSc3WzTs=; b=EgYNK9HcVmJk4Ov8ZuimIn/qsTAuA9NB3cjMMikYmic+3N6ujMAAgssjgxe7lmC1cM nghVN3tWJN+ykbwZMcOrYlgeOkWAW+56kOFhVT+gjAmazJuwrVbwvJQZOHS7Pxs3Rwkh PI4zMuEmDzrKrEbK7mKTKn0mF/hxXtPNVgrRjF8hr8J0LN5q8L92pCZqtsjA5pjIjwsi MjQ/L/u7gqtMZ8N2bsdda5mhHOSi3j5Z46x3Vq5xK5roeU0i9WnxRqyo1r05G7D2RRnw eTUHaLBB9UzPE8u+64dq1072lsuvDLt3sgxVMcdi8XAQHKEGV9gMbitf4C/adgMPJim1 sZ7w== X-Gm-Message-State: AOJu0YydNZnvsS+ySriug4w4p6XyRe8iW5cVU9aNmjsD0UJYmGxqGAAu iDoK8PRJ7hVm5KZTGfj0XJODz0Rjh88ZFOu7FQ5wYeMQqIl2LBiGpO1z2KdZqrE6fZChwFcm85k kXKlCO4MB8JXNr4MTklH8Uv0bdC7hpQjVF4lio6uP72Vhj4U2J6fXE6ukLJQFCLKhZt4BD0pFc4 Y4AZzZlsELTPkNL0niaCt6VCw9QWCOcA== X-Received: by 2002:a05:6214:3d8c:b0:696:6f95:4421 with SMTP id om12-20020a0562143d8c00b006966f954421mr14386007qvb.1.1711553037086; Wed, 27 Mar 2024 08:23:57 -0700 (PDT) X-Google-Smtp-Source: AGHT+IEnSeTu/B6EMntVMOCEyhcTmSAZJByvhuxAj5BPZo0z5woD7z6vaVN6Kd8/ZBw0a5bsOP9d6g== X-Received: by 2002:a05:6214:3d8c:b0:696:6f95:4421 with SMTP id om12-20020a0562143d8c00b006966f954421mr14385952qvb.1.1711553036462; Wed, 27 Mar 2024 08:23:56 -0700 (PDT) Received: from x1n.redhat.com ([99.254.121.117]) by smtp.gmail.com with ESMTPSA id hu4-20020a056214234400b00690dd47a41csm6412639qvb.86.2024.03.27.08.23.54 (version=TLS1_3 cipher=TLS_AES_256_GCM_SHA384 bits=256/256); Wed, 27 Mar 2024 08:23:56 -0700 (PDT) From: peterx@redhat.com To: linux-mm@kvack.org, linux-kernel@vger.kernel.org Cc: Yang Shi , "Kirill A . Shutemov" , Mike Kravetz , John Hubbard , Michael Ellerman , peterx@redhat.com, Andrew Jones , Muchun Song , linux-riscv@lists.infradead.org, linuxppc-dev@lists.ozlabs.org, Christophe Leroy , Andrew Morton , Christoph Hellwig , Lorenzo Stoakes , Matthew Wilcox , Rik van Riel , linux-arm-kernel@lists.infradead.org, Andrea Arcangeli , David Hildenbrand , "Aneesh Kumar K . V" , Vlastimil Babka , James Houghton , Jason Gunthorpe , Mike Rapoport , Axel Rasmussen Subject: [PATCH v4 10/13] mm/gup: Handle huge pud for follow_pud_mask() Date: Wed, 27 Mar 2024 11:23:29 -0400 Message-ID: <20240327152332.950956-11-peterx@redhat.com> X-Mailer: git-send-email 2.44.0 In-Reply-To: <20240327152332.950956-1-peterx@redhat.com> References: <20240327152332.950956-1-peterx@redhat.com> MIME-Version: 1.0 X-Mimecast-Spam-Score: 0 X-Mimecast-Originator: redhat.com X-Rspamd-Queue-Id: 00413140020 X-Rspam-User: X-Stat-Signature: aih5dknr3tca6bti9m6ctd78cuj1fxbj X-Rspamd-Server: rspam01 X-HE-Tag: 1711553039-323136 X-HE-Meta: U2FsdGVkX18ErmHLJw0pB4lUVycyfkZ7AvA8Ln4CfrGcquOrFkjksU9cHkPlmegdqON6FLnMnf/wtZzC+n5TcE9nR/EjKn+TOivBcvFZfVdJFUXwxOY3qjRnAjbMI6y/B011zHO5mKLTv7ghthAWr3LoE71J+qCnqvxe4zomv/dXyHMmeKvvXAJwOpLJJkxzy2x/+DkpyZitcEeF3QDm/7+O3V/vOLb14+kuAhO1VS10/Gz/Z4BuK1x/nk5QKKbQ30FJGqy8Z7MQoNghxeD3czIki53A6WS31DSdKqoWWuDJsdhEr9DeXrBVkuyJv0GOClaTc/8pbsJRnTY6ax2twRgTTVX5lO0TmUoXahQ3sqIKbDHdkoPmqrRCtk6DP8r4PHhQfYJVl6V8ZbElApjoUUHsJS0p4CdoxbqZQmrDGc4IiziELmBPc+nejhIVxEEO4bShRJv+uCxND8V1CutPVpVrfDYP5qfR/LvmiIdBqXFszdDSF+kIniZvKw9TgBuOSnAcjLkQcteusOsFiwd2UCCul31/s7SzZ9FrHPfPr5dMvIW82r4QR8XiG7lbo0cMWaczt1HzD7yu5FINy+AFH1osycFCKUM8XJdjrDAcKNV+ZAlK+ilITksES0hkH5YEsvfCQ+pTArGeFySfZtxoaRrqkAnRaE3N2Np/HJ8nbrqmOORhpKhfuGbYcLVrcD4NnDhjgW6KUdBFIIyiqlDXYnVtFZSFA1RMtbR+WfZX/HLCL/KZYaxVRi6FKKRDzctTSC1hz4UMedT/+/EHR9/xvcVxv3CkRfRNGVPooIIa03yQG5a6Mh5LDNlkx30cprFfSMiGoCAPpJCXImPoH16I/tx5SSqOH+/JjmUZM1RG9FSwSZxcrdTEIhpMsY3s/GiuTDfy0fznTXX+mY+foujlmqeTkvkc1a46NfwVlgwSQEM42PgMWSxLe5ysGYqYqpSNun3OVGqkP1WJ8mRzJ8T GmAKKaNe oaPPRq7Vm4av65M+XJMRxOGyZYdB66KMsSPqdN2xU9yGxjjAYzYSrCxDGbCeGBItI/Oabe9wzyQgxr/qL6sp8vIzSOSDhk7WVLTixzU1ps2iUXDlMht2kOEygj2mR6ZLSfoNY1zJ6OXb/2Io= X-Bogosity: Ham, tests=bogofilter, spamicity=0.000000, version=1.2.4 Sender: owner-linux-mm@kvack.org Precedence: bulk X-Loop: owner-majordomo@kvack.org List-ID: List-Subscribe: List-Unsubscribe: From: Peter Xu Teach follow_pud_mask() to be able to handle normal PUD pages like hugetlb. Rename follow_devmap_pud() to follow_huge_pud() so that it can process either huge devmap or hugetlb. Move it out of TRANSPARENT_HUGEPAGE_PUD and and huge_memory.c (which relies on CONFIG_THP). Switch to pud_leaf() to detect both cases in the slow gup. In the new follow_huge_pud(), taking care of possible CoR for hugetlb if necessary. touch_pud() needs to be moved out of huge_memory.c to be accessable from gup.c even if !THP. Since at it, optimize the non-present check by adding a pud_present() early check before taking the pgtable lock, failing the follow_page() early if PUD is not present: that is required by both devmap or hugetlb. Use pud_huge() to also cover the pud_devmap() case. One more trivial thing to mention is, introduce "pud_t pud" in the code paths along the way, so the code doesn't dereference *pudp multiple time. Not only because that looks less straightforward, but also because if the dereference really happened, it's not clear whether there can be race to see different *pudp values when it's being modified at the same time. Setting ctx->page_mask properly for a PUD entry. As a side effect, this patch should also be able to optimize devmap GUP on PUD to be able to jump over the whole PUD range, but not yet verified. Hugetlb already can do so prior to this patch. Reviewed-by: Jason Gunthorpe Signed-off-by: Peter Xu --- include/linux/huge_mm.h | 8 ----- mm/gup.c | 70 +++++++++++++++++++++++++++++++++++++++-- mm/huge_memory.c | 47 ++------------------------- mm/internal.h | 2 ++ 4 files changed, 71 insertions(+), 56 deletions(-) diff --git a/include/linux/huge_mm.h b/include/linux/huge_mm.h index d3bb25c39482..3f36511bdc02 100644 --- a/include/linux/huge_mm.h +++ b/include/linux/huge_mm.h @@ -351,8 +351,6 @@ static inline bool folio_test_pmd_mappable(struct folio *folio) struct page *follow_devmap_pmd(struct vm_area_struct *vma, unsigned long addr, pmd_t *pmd, int flags, struct dev_pagemap **pgmap); -struct page *follow_devmap_pud(struct vm_area_struct *vma, unsigned long addr, - pud_t *pud, int flags, struct dev_pagemap **pgmap); vm_fault_t do_huge_pmd_numa_page(struct vm_fault *vmf); @@ -507,12 +505,6 @@ static inline struct page *follow_devmap_pmd(struct vm_area_struct *vma, return NULL; } -static inline struct page *follow_devmap_pud(struct vm_area_struct *vma, - unsigned long addr, pud_t *pud, int flags, struct dev_pagemap **pgmap) -{ - return NULL; -} - static inline bool thp_migration_supported(void) { return false; diff --git a/mm/gup.c b/mm/gup.c index 26b8cca24077..1e5d42211bb4 100644 --- a/mm/gup.c +++ b/mm/gup.c @@ -525,6 +525,70 @@ static struct page *no_page_table(struct vm_area_struct *vma, return NULL; } +#ifdef CONFIG_PGTABLE_HAS_HUGE_LEAVES +static struct page *follow_huge_pud(struct vm_area_struct *vma, + unsigned long addr, pud_t *pudp, + int flags, struct follow_page_context *ctx) +{ + struct mm_struct *mm = vma->vm_mm; + struct page *page; + pud_t pud = *pudp; + unsigned long pfn = pud_pfn(pud); + int ret; + + assert_spin_locked(pud_lockptr(mm, pudp)); + + if ((flags & FOLL_WRITE) && !pud_write(pud)) + return NULL; + + if (!pud_present(pud)) + return NULL; + + pfn += (addr & ~PUD_MASK) >> PAGE_SHIFT; + + if (IS_ENABLED(CONFIG_HAVE_ARCH_TRANSPARENT_HUGEPAGE_PUD) && + pud_devmap(pud)) { + /* + * device mapped pages can only be returned if the caller + * will manage the page reference count. + * + * At least one of FOLL_GET | FOLL_PIN must be set, so + * assert that here: + */ + if (!(flags & (FOLL_GET | FOLL_PIN))) + return ERR_PTR(-EEXIST); + + if (flags & FOLL_TOUCH) + touch_pud(vma, addr, pudp, flags & FOLL_WRITE); + + ctx->pgmap = get_dev_pagemap(pfn, ctx->pgmap); + if (!ctx->pgmap) + return ERR_PTR(-EFAULT); + } + + page = pfn_to_page(pfn); + + if (!pud_devmap(pud) && !pud_write(pud) && + gup_must_unshare(vma, flags, page)) + return ERR_PTR(-EMLINK); + + ret = try_grab_page(page, flags); + if (ret) + page = ERR_PTR(ret); + else + ctx->page_mask = HPAGE_PUD_NR - 1; + + return page; +} +#else /* CONFIG_PGTABLE_HAS_HUGE_LEAVES */ +static struct page *follow_huge_pud(struct vm_area_struct *vma, + unsigned long addr, pud_t *pudp, + int flags, struct follow_page_context *ctx) +{ + return NULL; +} +#endif /* CONFIG_PGTABLE_HAS_HUGE_LEAVES */ + static int follow_pfn_pte(struct vm_area_struct *vma, unsigned long address, pte_t *pte, unsigned int flags) { @@ -760,11 +824,11 @@ static struct page *follow_pud_mask(struct vm_area_struct *vma, pudp = pud_offset(p4dp, address); pud = READ_ONCE(*pudp); - if (pud_none(pud)) + if (!pud_present(pud)) return no_page_table(vma, flags, address); - if (pud_devmap(pud)) { + if (pud_leaf(pud)) { ptl = pud_lock(mm, pudp); - page = follow_devmap_pud(vma, address, pudp, flags, &ctx->pgmap); + page = follow_huge_pud(vma, address, pudp, flags, ctx); spin_unlock(ptl); if (page) return page; diff --git a/mm/huge_memory.c b/mm/huge_memory.c index bc6fa82d9815..2979198d7b71 100644 --- a/mm/huge_memory.c +++ b/mm/huge_memory.c @@ -1377,8 +1377,8 @@ int copy_huge_pmd(struct mm_struct *dst_mm, struct mm_struct *src_mm, } #ifdef CONFIG_HAVE_ARCH_TRANSPARENT_HUGEPAGE_PUD -static void touch_pud(struct vm_area_struct *vma, unsigned long addr, - pud_t *pud, bool write) +void touch_pud(struct vm_area_struct *vma, unsigned long addr, + pud_t *pud, bool write) { pud_t _pud; @@ -1390,49 +1390,6 @@ static void touch_pud(struct vm_area_struct *vma, unsigned long addr, update_mmu_cache_pud(vma, addr, pud); } -struct page *follow_devmap_pud(struct vm_area_struct *vma, unsigned long addr, - pud_t *pud, int flags, struct dev_pagemap **pgmap) -{ - unsigned long pfn = pud_pfn(*pud); - struct mm_struct *mm = vma->vm_mm; - struct page *page; - int ret; - - assert_spin_locked(pud_lockptr(mm, pud)); - - if (flags & FOLL_WRITE && !pud_write(*pud)) - return NULL; - - if (pud_present(*pud) && pud_devmap(*pud)) - /* pass */; - else - return NULL; - - if (flags & FOLL_TOUCH) - touch_pud(vma, addr, pud, flags & FOLL_WRITE); - - /* - * device mapped pages can only be returned if the - * caller will manage the page reference count. - * - * At least one of FOLL_GET | FOLL_PIN must be set, so assert that here: - */ - if (!(flags & (FOLL_GET | FOLL_PIN))) - return ERR_PTR(-EEXIST); - - pfn += (addr & ~PUD_MASK) >> PAGE_SHIFT; - *pgmap = get_dev_pagemap(pfn, *pgmap); - if (!*pgmap) - return ERR_PTR(-EFAULT); - page = pfn_to_page(pfn); - - ret = try_grab_page(page, flags); - if (ret) - page = ERR_PTR(ret); - - return page; -} - int copy_huge_pud(struct mm_struct *dst_mm, struct mm_struct *src_mm, pud_t *dst_pud, pud_t *src_pud, unsigned long addr, struct vm_area_struct *vma) diff --git a/mm/internal.h b/mm/internal.h index 6c8d3844b6a3..eee8c82740b5 100644 --- a/mm/internal.h +++ b/mm/internal.h @@ -1111,6 +1111,8 @@ int __must_check try_grab_page(struct page *page, unsigned int flags); /* * mm/huge_memory.c */ +void touch_pud(struct vm_area_struct *vma, unsigned long addr, + pud_t *pud, bool write); struct page *follow_trans_huge_pmd(struct vm_area_struct *vma, unsigned long addr, pmd_t *pmd, unsigned int flags); From patchwork Wed Mar 27 15:23:30 2024 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Peter Xu X-Patchwork-Id: 13606793 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from kanga.kvack.org (kanga.kvack.org [205.233.56.17]) by smtp.lore.kernel.org (Postfix) with ESMTP id BC075C54E67 for ; Wed, 27 Mar 2024 15:24:17 +0000 (UTC) Received: by kanga.kvack.org (Postfix) id 22CAF6B00A6; Wed, 27 Mar 2024 11:24:09 -0400 (EDT) Received: by kanga.kvack.org (Postfix, from userid 40) id 1DE006B00A7; Wed, 27 Mar 2024 11:24:09 -0400 (EDT) X-Delivered-To: int-list-linux-mm@kvack.org Received: by kanga.kvack.org (Postfix, from userid 63042) id 02EE46B00A9; Wed, 27 Mar 2024 11:24:08 -0400 (EDT) X-Delivered-To: linux-mm@kvack.org Received: from relay.hostedemail.com (smtprelay0015.hostedemail.com [216.40.44.15]) by kanga.kvack.org (Postfix) with ESMTP id D57FF6B00A6 for ; Wed, 27 Mar 2024 11:24:08 -0400 (EDT) Received: from smtpin14.hostedemail.com (a10.router.float.18 [10.200.18.1]) by unirelay06.hostedemail.com (Postfix) with ESMTP id 9BC8AA12EA for ; Wed, 27 Mar 2024 15:24:08 +0000 (UTC) X-FDA: 81943189776.14.C90D781 Received: from us-smtp-delivery-124.mimecast.com (us-smtp-delivery-124.mimecast.com [170.10.133.124]) by imf15.hostedemail.com (Postfix) with ESMTP id 827F5A0012 for ; Wed, 27 Mar 2024 15:24:06 +0000 (UTC) Authentication-Results: imf15.hostedemail.com; dkim=pass header.d=redhat.com header.s=mimecast20190719 header.b=aR2vUTXE; spf=pass (imf15.hostedemail.com: domain of peterx@redhat.com designates 170.10.133.124 as permitted sender) smtp.mailfrom=peterx@redhat.com; dmarc=pass (policy=none) header.from=redhat.com ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=hostedemail.com; s=arc-20220608; t=1711553046; h=from:from:sender:reply-to:subject:subject:date:date: message-id:message-id:to:to:cc:cc:mime-version:mime-version: content-type:content-type: content-transfer-encoding:content-transfer-encoding: in-reply-to:in-reply-to:references:references:dkim-signature; bh=h7p860Pwfa5oWvGOHiVson4UsvvLX6Zsz0nCOpl6KoI=; b=QYx+peDzb97cLNhqlbYQuxaplzDv5AXa52rMULWaBR3ObnZp7ZAF6Nf/h7W4aPkuYOAYzR jRdCXoQU0kbr8oZU1qlxwhAeoejjNZCjLgL55cLP+cPHjHllytFrnSMqvurccIeRHZYs8i nQrK49htk4l/VvLmjnSR1pTuXr8h/qI= ARC-Seal: i=1; s=arc-20220608; d=hostedemail.com; t=1711553046; a=rsa-sha256; cv=none; b=3h/uUcG92naEbZF2/H3MPZs+oMKLN2P3LBuATVDkxcN2rqkaoO0H1ULSeWnyAgxP1X/SJ0 R/G7tiKON4Dc6spM7ax4sOoi2Mw/Bh6LHeV2X+lsB3VHsImStZYU6OEkaHOPy/YfMlnQVY gnr7U7UfJ8EPtjVcVTnuEbSrQmxUbO4= ARC-Authentication-Results: i=1; imf15.hostedemail.com; dkim=pass header.d=redhat.com header.s=mimecast20190719 header.b=aR2vUTXE; spf=pass (imf15.hostedemail.com: domain of peterx@redhat.com designates 170.10.133.124 as permitted sender) smtp.mailfrom=peterx@redhat.com; dmarc=pass (policy=none) header.from=redhat.com DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=redhat.com; s=mimecast20190719; t=1711553045; h=from:from:reply-to:subject:subject:date:date:message-id:message-id: to:to:cc:cc:mime-version:mime-version:content-type:content-type: content-transfer-encoding:content-transfer-encoding: in-reply-to:in-reply-to:references:references; bh=h7p860Pwfa5oWvGOHiVson4UsvvLX6Zsz0nCOpl6KoI=; b=aR2vUTXE1JmyVrZPV7R2A9z7VaeQmXL4darXFLG/h8QavsZ8PmT1NEGAvwbMBgo9SjVkhn aumW98T9gkqdrNfMwuDd/q8LdMhJYKgwFQje7hT7JtmfF1ZGCL4/fW9zoxsbDIEKGCgcMy 8jXnbSkWup1j6uoW1RdPDyiUauYpDG0= Received: from mail-qv1-f72.google.com (mail-qv1-f72.google.com [209.85.219.72]) by relay.mimecast.com with ESMTP with STARTTLS (version=TLSv1.3, cipher=TLS_AES_256_GCM_SHA384) id us-mta-396-sVUPtgWRPpGsibhPgmmQTA-1; Wed, 27 Mar 2024 11:24:01 -0400 X-MC-Unique: sVUPtgWRPpGsibhPgmmQTA-1 Received: by mail-qv1-f72.google.com with SMTP id 6a1803df08f44-69152af7760so25773136d6.1 for ; Wed, 27 Mar 2024 08:24:01 -0700 (PDT) X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20230601; t=1711553041; x=1712157841; h=content-transfer-encoding:mime-version:references:in-reply-to :message-id:date:subject:cc:to:from:x-gm-message-state:from:to:cc :subject:date:message-id:reply-to; bh=h7p860Pwfa5oWvGOHiVson4UsvvLX6Zsz0nCOpl6KoI=; b=oI3FEQepwesORMxGG/KsoCqNRmRspRk8cEdJ64g3SNCPh4p31PBadGQf56hxDCmqsQ +gp+ujgJ3QGzVAv3HKirSxz5jRR+TeN/zM5IoNpo8iUwhQrNKKwh86POPpWskcC5azDM fXgzy4cBJPPZ6SYPy7CodF9rJxUH+ZTSVpRBtWzda3iiCX2VVtevyt7icF409TIXkLNX 1RdQA5q4WoixjJ1S8//QwS4o+9wTt+lLFnmPTOs/qAFv4543lQd7Un/fBlP+DB8RX3Ag ZB78fm9s4iLRZY648AMZEK66ikhk2p5fWf7eaQodPklF4GotINBD5IL8yuXYpbLQD8h2 rnfQ== X-Gm-Message-State: AOJu0YxjuP4Uxts4jNlC31ZtzWBaxk1RzIrwEikEp3OZQhzmz8nqqxL+ 1R6UVoLDjs1rQ3l/czzwo5HV9tvMQ27TOZ58ZFZ/rN6m0TYgdHcJNb29ksppQUauTSIDNXHFrZP r5lUr2xmlpAnec5yS6zZ2YQZFo1kI5cchR2EdGaSyaNwizubE9zCSNza8vLK+aAdoG1cL7Kx4b0 PcNx9Fwye4YhES+B0J+CCXXTn8J3TI4g== X-Received: by 2002:a05:6214:3d13:b0:696:7b12:3744 with SMTP id ol19-20020a0562143d1300b006967b123744mr13941613qvb.0.1711553040447; Wed, 27 Mar 2024 08:24:00 -0700 (PDT) X-Google-Smtp-Source: AGHT+IGf1gHn7pWGOliI2n3iseJ1Y4KDSmrGTREwaUpEtNei2O8hiTLwONxqZARC/3SfeDcLLHxfTA== X-Received: by 2002:a05:6214:3d13:b0:696:7b12:3744 with SMTP id ol19-20020a0562143d1300b006967b123744mr13941496qvb.0.1711553038757; Wed, 27 Mar 2024 08:23:58 -0700 (PDT) Received: from x1n.redhat.com ([99.254.121.117]) by smtp.gmail.com with ESMTPSA id hu4-20020a056214234400b00690dd47a41csm6412639qvb.86.2024.03.27.08.23.56 (version=TLS1_3 cipher=TLS_AES_256_GCM_SHA384 bits=256/256); Wed, 27 Mar 2024 08:23:58 -0700 (PDT) From: peterx@redhat.com To: linux-mm@kvack.org, linux-kernel@vger.kernel.org Cc: Yang Shi , "Kirill A . Shutemov" , Mike Kravetz , John Hubbard , Michael Ellerman , peterx@redhat.com, Andrew Jones , Muchun Song , linux-riscv@lists.infradead.org, linuxppc-dev@lists.ozlabs.org, Christophe Leroy , Andrew Morton , Christoph Hellwig , Lorenzo Stoakes , Matthew Wilcox , Rik van Riel , linux-arm-kernel@lists.infradead.org, Andrea Arcangeli , David Hildenbrand , "Aneesh Kumar K . V" , Vlastimil Babka , James Houghton , Jason Gunthorpe , Mike Rapoport , Axel Rasmussen Subject: [PATCH v4 11/13] mm/gup: Handle huge pmd for follow_pmd_mask() Date: Wed, 27 Mar 2024 11:23:30 -0400 Message-ID: <20240327152332.950956-12-peterx@redhat.com> X-Mailer: git-send-email 2.44.0 In-Reply-To: <20240327152332.950956-1-peterx@redhat.com> References: <20240327152332.950956-1-peterx@redhat.com> MIME-Version: 1.0 X-Mimecast-Spam-Score: 0 X-Mimecast-Originator: redhat.com X-Rspamd-Queue-Id: 827F5A0012 X-Rspam-User: X-Rspamd-Server: rspam11 X-Stat-Signature: ug9zir6t6pstou36zho1grxjxtwg1e4y X-HE-Tag: 1711553046-766991 X-HE-Meta: U2FsdGVkX183sVqKv7wwex82Yc48rlY0jeaUfZsWrI48/0238JxnwHLBUBQs11upfUBeKHpHS6zkvTg/k8dxRdoQ2OmYw6txm6XEFVnkAF+bbRaGstQ2Op7os8MSQoqAL7rOtKrSJta5rRHJUkflsZhIEhgws+o4n7LeLLIbAmX8gWQEPxEPWwr5g/g2lOVXCpS4O2hiaqIOjekMj5Qawqo9YEOJr7MhnaOgiQ38bRbG1x8/zqX/nFLDMGryxWlzvtXOsMpoavaBQ8SEkb8o/YkBhC8oSKF4t8wv+LUBi1EUb+/6+5P62/GvDzsaAeN/Y0nmiS8krgn8AzHKcJa3LNaRURZsajanDXkSOpFmnd3d/bB6VlP+LQvG252LLYTxiZbkd/bp+83xt623Gd8SbauZZvP6G/Ub+6V4R0GYByJzungoqb3hMu1sMKveTy0BHtYhlBdTZgNYbJnn3HBeCommMGtKZR+/kpXqgtCp6QCZ/UsCEzH5KA5sdPIsFql7AYVIyHA+ivrEtxJijbL5ZpZYs2JJPpsdEN5ndmuM3BlQ5C7UIwTuR2oyjpjIxUIOl2QsNrrht8RfwmDt4GfF4+j27nUwgRIeHTAdjoXUOyOTQBmY8B6Qx4lmDP1qzkjuI1Sca/0hHD6K8rvijN5flCLZJfhWM3TViVhVYCLHn+egkccT2UCXmzst9kf4IH+NsF8qdfekfBCxuYzCQP6kR3jvKq8JkNFuuU1fLr3SSC9irzBsTS4SusVmDQ9k+IuHqrmx0Lbdp1OPKqmY/Ol7NgezsjkS6WmpBeQ6eSElryToYcgM71ELQH7nDXn0OcdFNXwxCoV7wIDC84MSOcenEb5ePgp5wagg9+h1HCrLOj4SbWkJgQCoJEPdQZqiP8c7pOoB+S1abzReFcr1SIaVF9t5Be7aiiAGGHPaVV+eLww9zcWpYE9qMdebUmVkpHFdLDiYGTtsNuTb9P9QJOr WTyuoryL AA7j5UYm7pUx5aNic37oaet2Qvd+sQoHAlQPRqdAtE5HK5fTjiVHn8W7VRFJZgHXjyUfJFtOCkwrWJE965DkEqCmBJ9OECgVSbJrRNFkGiYBOyC7JN05luxwVxAa/BlrwwOe02LiVhgkRbH6yqqIkw3FScFAbzKYiahYt2lY6gXaG/9vGDdyxrfVefFyV/CZyGowOl/q1toDse/z3/AI0nm5OJUvtBsw2LUrqRQGNiWa8/ME454yQS8PvoQwZiI3ttPkdTuP/G3ZaMi0mW/E294z2PC8FbF/BxRteLPGMcJMEUSmf+5mgSipb9Ei/YOj8OO9/z7LXhfsZ373bEM/047B/sEb6H77HCPTKk7x7nZMwaF3DJyku4RksMnVHTERTdHOySA0eeOZe6AXOq3eWq7jcu/FROuw5XoV94usczAckHzrXzKJBp4RYBoCRu0qv9aLxZVu3SK/5WZI= X-Bogosity: Ham, tests=bogofilter, spamicity=0.000000, version=1.2.4 Sender: owner-linux-mm@kvack.org Precedence: bulk X-Loop: owner-majordomo@kvack.org List-ID: List-Subscribe: List-Unsubscribe: From: Peter Xu Replace pmd_trans_huge() with pmd_leaf() to also cover pmd_huge() as long as enabled. FOLL_TOUCH and FOLL_SPLIT_PMD only apply to THP, not yet huge. Since now follow_trans_huge_pmd() can process hugetlb pages, renaming it into follow_huge_pmd() to match what it does. Move it into gup.c so not depend on CONFIG_THP. When at it, move the ctx->page_mask setup into follow_huge_pmd(), only set it when the page is valid. It was not a bug to set it before even if GUP failed (page==NULL), because follow_page_mask() callers always ignores page_mask if so. But doing so makes the code cleaner. Reviewed-by: Jason Gunthorpe Signed-off-by: Peter Xu --- mm/gup.c | 107 ++++++++++++++++++++++++++++++++++++++++++++--- mm/huge_memory.c | 86 +------------------------------------ mm/internal.h | 5 +-- 3 files changed, 105 insertions(+), 93 deletions(-) diff --git a/mm/gup.c b/mm/gup.c index 1e5d42211bb4..a81184b01276 100644 --- a/mm/gup.c +++ b/mm/gup.c @@ -580,6 +580,93 @@ static struct page *follow_huge_pud(struct vm_area_struct *vma, return page; } + +/* FOLL_FORCE can write to even unwritable PMDs in COW mappings. */ +static inline bool can_follow_write_pmd(pmd_t pmd, struct page *page, + struct vm_area_struct *vma, + unsigned int flags) +{ + /* If the pmd is writable, we can write to the page. */ + if (pmd_write(pmd)) + return true; + + /* Maybe FOLL_FORCE is set to override it? */ + if (!(flags & FOLL_FORCE)) + return false; + + /* But FOLL_FORCE has no effect on shared mappings */ + if (vma->vm_flags & (VM_MAYSHARE | VM_SHARED)) + return false; + + /* ... or read-only private ones */ + if (!(vma->vm_flags & VM_MAYWRITE)) + return false; + + /* ... or already writable ones that just need to take a write fault */ + if (vma->vm_flags & VM_WRITE) + return false; + + /* + * See can_change_pte_writable(): we broke COW and could map the page + * writable if we have an exclusive anonymous page ... + */ + if (!page || !PageAnon(page) || !PageAnonExclusive(page)) + return false; + + /* ... and a write-fault isn't required for other reasons. */ + if (vma_soft_dirty_enabled(vma) && !pmd_soft_dirty(pmd)) + return false; + return !userfaultfd_huge_pmd_wp(vma, pmd); +} + +static struct page *follow_huge_pmd(struct vm_area_struct *vma, + unsigned long addr, pmd_t *pmd, + unsigned int flags, + struct follow_page_context *ctx) +{ + struct mm_struct *mm = vma->vm_mm; + pmd_t pmdval = *pmd; + struct page *page; + int ret; + + assert_spin_locked(pmd_lockptr(mm, pmd)); + + page = pmd_page(pmdval); + VM_BUG_ON_PAGE(!PageHead(page) && !is_zone_device_page(page), page); + + if ((flags & FOLL_WRITE) && + !can_follow_write_pmd(pmdval, page, vma, flags)) + return NULL; + + /* Avoid dumping huge zero page */ + if ((flags & FOLL_DUMP) && is_huge_zero_pmd(pmdval)) + return ERR_PTR(-EFAULT); + + if (pmd_protnone(*pmd) && !gup_can_follow_protnone(vma, flags)) + return NULL; + + if (!pmd_write(pmdval) && gup_must_unshare(vma, flags, page)) + return ERR_PTR(-EMLINK); + + VM_BUG_ON_PAGE((flags & FOLL_PIN) && PageAnon(page) && + !PageAnonExclusive(page), page); + + ret = try_grab_page(page, flags); + if (ret) + return ERR_PTR(ret); + +#ifdef CONFIG_TRANSPARENT_HUGEPAGE + if (pmd_trans_huge(pmdval) && (flags & FOLL_TOUCH)) + touch_pmd(vma, addr, pmd, flags & FOLL_WRITE); +#endif /* CONFIG_TRANSPARENT_HUGEPAGE */ + + page += (addr & ~HPAGE_PMD_MASK) >> PAGE_SHIFT; + ctx->page_mask = HPAGE_PMD_NR - 1; + VM_BUG_ON_PAGE(!PageCompound(page) && !is_zone_device_page(page), page); + + return page; +} + #else /* CONFIG_PGTABLE_HAS_HUGE_LEAVES */ static struct page *follow_huge_pud(struct vm_area_struct *vma, unsigned long addr, pud_t *pudp, @@ -587,6 +674,14 @@ static struct page *follow_huge_pud(struct vm_area_struct *vma, { return NULL; } + +static struct page *follow_huge_pmd(struct vm_area_struct *vma, + unsigned long addr, pmd_t *pmd, + unsigned int flags, + struct follow_page_context *ctx) +{ + return NULL; +} #endif /* CONFIG_PGTABLE_HAS_HUGE_LEAVES */ static int follow_pfn_pte(struct vm_area_struct *vma, unsigned long address, @@ -784,31 +879,31 @@ static struct page *follow_pmd_mask(struct vm_area_struct *vma, return page; return no_page_table(vma, flags, address); } - if (likely(!pmd_trans_huge(pmdval))) + if (likely(!pmd_leaf(pmdval))) return follow_page_pte(vma, address, pmd, flags, &ctx->pgmap); if (pmd_protnone(pmdval) && !gup_can_follow_protnone(vma, flags)) return no_page_table(vma, flags, address); ptl = pmd_lock(mm, pmd); - if (unlikely(!pmd_present(*pmd))) { + pmdval = *pmd; + if (unlikely(!pmd_present(pmdval))) { spin_unlock(ptl); return no_page_table(vma, flags, address); } - if (unlikely(!pmd_trans_huge(*pmd))) { + if (unlikely(!pmd_leaf(pmdval))) { spin_unlock(ptl); return follow_page_pte(vma, address, pmd, flags, &ctx->pgmap); } - if (flags & FOLL_SPLIT_PMD) { + if (pmd_trans_huge(pmdval) && (flags & FOLL_SPLIT_PMD)) { spin_unlock(ptl); split_huge_pmd(vma, pmd, address); /* If pmd was left empty, stuff a page table in there quickly */ return pte_alloc(mm, pmd) ? ERR_PTR(-ENOMEM) : follow_page_pte(vma, address, pmd, flags, &ctx->pgmap); } - page = follow_trans_huge_pmd(vma, address, pmd, flags); + page = follow_huge_pmd(vma, address, pmd, flags, ctx); spin_unlock(ptl); - ctx->page_mask = HPAGE_PMD_NR - 1; return page; } diff --git a/mm/huge_memory.c b/mm/huge_memory.c index 2979198d7b71..ed0d82c4b829 100644 --- a/mm/huge_memory.c +++ b/mm/huge_memory.c @@ -1220,8 +1220,8 @@ vm_fault_t vmf_insert_pfn_pud(struct vm_fault *vmf, pfn_t pfn, bool write) EXPORT_SYMBOL_GPL(vmf_insert_pfn_pud); #endif /* CONFIG_HAVE_ARCH_TRANSPARENT_HUGEPAGE_PUD */ -static void touch_pmd(struct vm_area_struct *vma, unsigned long addr, - pmd_t *pmd, bool write) +void touch_pmd(struct vm_area_struct *vma, unsigned long addr, + pmd_t *pmd, bool write) { pmd_t _pmd; @@ -1576,88 +1576,6 @@ static inline bool can_change_pmd_writable(struct vm_area_struct *vma, return pmd_dirty(pmd); } -/* FOLL_FORCE can write to even unwritable PMDs in COW mappings. */ -static inline bool can_follow_write_pmd(pmd_t pmd, struct page *page, - struct vm_area_struct *vma, - unsigned int flags) -{ - /* If the pmd is writable, we can write to the page. */ - if (pmd_write(pmd)) - return true; - - /* Maybe FOLL_FORCE is set to override it? */ - if (!(flags & FOLL_FORCE)) - return false; - - /* But FOLL_FORCE has no effect on shared mappings */ - if (vma->vm_flags & (VM_MAYSHARE | VM_SHARED)) - return false; - - /* ... or read-only private ones */ - if (!(vma->vm_flags & VM_MAYWRITE)) - return false; - - /* ... or already writable ones that just need to take a write fault */ - if (vma->vm_flags & VM_WRITE) - return false; - - /* - * See can_change_pte_writable(): we broke COW and could map the page - * writable if we have an exclusive anonymous page ... - */ - if (!page || !PageAnon(page) || !PageAnonExclusive(page)) - return false; - - /* ... and a write-fault isn't required for other reasons. */ - if (vma_soft_dirty_enabled(vma) && !pmd_soft_dirty(pmd)) - return false; - return !userfaultfd_huge_pmd_wp(vma, pmd); -} - -struct page *follow_trans_huge_pmd(struct vm_area_struct *vma, - unsigned long addr, - pmd_t *pmd, - unsigned int flags) -{ - struct mm_struct *mm = vma->vm_mm; - struct page *page; - int ret; - - assert_spin_locked(pmd_lockptr(mm, pmd)); - - page = pmd_page(*pmd); - VM_BUG_ON_PAGE(!PageHead(page) && !is_zone_device_page(page), page); - - if ((flags & FOLL_WRITE) && - !can_follow_write_pmd(*pmd, page, vma, flags)) - return NULL; - - /* Avoid dumping huge zero page */ - if ((flags & FOLL_DUMP) && is_huge_zero_pmd(*pmd)) - return ERR_PTR(-EFAULT); - - if (pmd_protnone(*pmd) && !gup_can_follow_protnone(vma, flags)) - return NULL; - - if (!pmd_write(*pmd) && gup_must_unshare(vma, flags, page)) - return ERR_PTR(-EMLINK); - - VM_BUG_ON_PAGE((flags & FOLL_PIN) && PageAnon(page) && - !PageAnonExclusive(page), page); - - ret = try_grab_page(page, flags); - if (ret) - return ERR_PTR(ret); - - if (flags & FOLL_TOUCH) - touch_pmd(vma, addr, pmd, flags & FOLL_WRITE); - - page += (addr & ~HPAGE_PMD_MASK) >> PAGE_SHIFT; - VM_BUG_ON_PAGE(!PageCompound(page) && !is_zone_device_page(page), page); - - return page; -} - /* NUMA hinting page fault entry point for trans huge pmds */ vm_fault_t do_huge_pmd_numa_page(struct vm_fault *vmf) { diff --git a/mm/internal.h b/mm/internal.h index eee8c82740b5..e10ecc6594f1 100644 --- a/mm/internal.h +++ b/mm/internal.h @@ -1113,9 +1113,8 @@ int __must_check try_grab_page(struct page *page, unsigned int flags); */ void touch_pud(struct vm_area_struct *vma, unsigned long addr, pud_t *pud, bool write); -struct page *follow_trans_huge_pmd(struct vm_area_struct *vma, - unsigned long addr, pmd_t *pmd, - unsigned int flags); +void touch_pmd(struct vm_area_struct *vma, unsigned long addr, + pmd_t *pmd, bool write); /* * mm/mmap.c From patchwork Wed Mar 27 15:23:31 2024 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Peter Xu X-Patchwork-Id: 13606792 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from kanga.kvack.org (kanga.kvack.org [205.233.56.17]) by smtp.lore.kernel.org (Postfix) with ESMTP id 60958C47DD9 for ; Wed, 27 Mar 2024 15:24:14 +0000 (UTC) Received: by kanga.kvack.org (Postfix) id 5D8086B00A5; Wed, 27 Mar 2024 11:24:07 -0400 (EDT) Received: by kanga.kvack.org (Postfix, from userid 40) id 585AF6B00A6; Wed, 27 Mar 2024 11:24:07 -0400 (EDT) X-Delivered-To: int-list-linux-mm@kvack.org Received: by kanga.kvack.org (Postfix, from userid 63042) id 3D8936B00A7; Wed, 27 Mar 2024 11:24:07 -0400 (EDT) X-Delivered-To: linux-mm@kvack.org Received: from relay.hostedemail.com (smtprelay0010.hostedemail.com [216.40.44.10]) by kanga.kvack.org (Postfix) with ESMTP id 13C2F6B00A5 for ; Wed, 27 Mar 2024 11:24:07 -0400 (EDT) Received: from smtpin01.hostedemail.com (a10.router.float.18 [10.200.18.1]) by unirelay05.hostedemail.com (Postfix) with ESMTP id 97D59405CD for ; Wed, 27 Mar 2024 15:24:06 +0000 (UTC) X-FDA: 81943189692.01.437B7C3 Received: from us-smtp-delivery-124.mimecast.com (us-smtp-delivery-124.mimecast.com [170.10.133.124]) by imf01.hostedemail.com (Postfix) with ESMTP id 8638040026 for ; Wed, 27 Mar 2024 15:24:04 +0000 (UTC) Authentication-Results: imf01.hostedemail.com; dkim=pass header.d=redhat.com header.s=mimecast20190719 header.b=TfsyBFxE; spf=pass (imf01.hostedemail.com: domain of peterx@redhat.com designates 170.10.133.124 as permitted sender) smtp.mailfrom=peterx@redhat.com; dmarc=pass (policy=none) header.from=redhat.com ARC-Seal: i=1; s=arc-20220608; d=hostedemail.com; t=1711553044; a=rsa-sha256; cv=none; b=gll8EACD4zETK5xUqhR1MtISW4acPkvtywKDiS+nCbS/0rp9a5FVNSfpPlbq1Me5Ajq8sT DihxIQTXeWGS/s1AYCDbFSR4jBan3bFEPa3PJ6W5N7Yr4HoZ1+geAjJypHAhFInQOsJzdm onzevSpU1ZaM3Z3UxApoI+c17EDx2QE= ARC-Authentication-Results: i=1; imf01.hostedemail.com; dkim=pass header.d=redhat.com header.s=mimecast20190719 header.b=TfsyBFxE; spf=pass (imf01.hostedemail.com: domain of peterx@redhat.com designates 170.10.133.124 as permitted sender) smtp.mailfrom=peterx@redhat.com; dmarc=pass (policy=none) header.from=redhat.com ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=hostedemail.com; s=arc-20220608; t=1711553044; h=from:from:sender:reply-to:subject:subject:date:date: message-id:message-id:to:to:cc:cc:mime-version:mime-version: content-type:content-type: content-transfer-encoding:content-transfer-encoding: in-reply-to:in-reply-to:references:references:dkim-signature; bh=cjVJdBQT9/0ffkgcc4+ae3LmOxlOj4cpHe90WMWbkEY=; b=Jqyx+UmHbBAo8fITcZhinL/DkVrETfs5XW3fBob2WbF6D7EtG5sF/FSXXONuSRQ9y64q+P Obpdqr8NHdpXticTWX1UvwkgTGaZivZYKa28e9wYUo5s4TPUQKV7oFnxTLj7oYYQsZt/Cu Q3DW0Xf5O/aRUgsZ6q+DJQ14XCIi42c= DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=redhat.com; s=mimecast20190719; t=1711553043; h=from:from:reply-to:subject:subject:date:date:message-id:message-id: to:to:cc:cc:mime-version:mime-version:content-type:content-type: content-transfer-encoding:content-transfer-encoding: in-reply-to:in-reply-to:references:references; bh=cjVJdBQT9/0ffkgcc4+ae3LmOxlOj4cpHe90WMWbkEY=; b=TfsyBFxEFrT6b4KrBNyvRlxbsZp4OZkyKTCpTtCBk8ymfeZAveKqzgIMEyCAe3HhiFN1iX 9RPvS+k4j1WEOo2hE5L/0oIrkRDWMr+YkbUWM4w1nfFimtZmvcBrt70q80HZ+Y8FQ/WyPf csy8AmVY5hwQZnLx30zb+Bx3vBcGXiE= Received: from mail-oi1-f200.google.com (mail-oi1-f200.google.com [209.85.167.200]) by relay.mimecast.com with ESMTP with STARTTLS (version=TLSv1.3, cipher=TLS_AES_256_GCM_SHA384) id us-mta-620-AVTZfM8vP6WPMbNJVouU3w-1; Wed, 27 Mar 2024 11:24:02 -0400 X-MC-Unique: AVTZfM8vP6WPMbNJVouU3w-1 Received: by mail-oi1-f200.google.com with SMTP id 5614622812f47-3c3c43470d7so1243378b6e.0 for ; Wed, 27 Mar 2024 08:24:02 -0700 (PDT) X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20230601; t=1711553041; x=1712157841; h=content-transfer-encoding:mime-version:references:in-reply-to :message-id:date:subject:cc:to:from:x-gm-message-state:from:to:cc :subject:date:message-id:reply-to; bh=cjVJdBQT9/0ffkgcc4+ae3LmOxlOj4cpHe90WMWbkEY=; b=mgCEhHORpfgZhO/DN3nKiGuuV0yZQSQquXxFQP8+AjAzQola8UXiD+vxDPvYI0wj2u 14HQyFKwRKsItPRuvmK0U9UPy+a0yvjzUqfvZZFWI3NJeNYSPsFehe3dKacH6oI+eqG2 iKNoGwAiaE2DNp7Jy7EJi51/U3x7U/avMcyYJTUmcE6V72quCndI/TEN77Ew7d50q9eM zpylE4CDvglkFuHnYTsrXFUeNbuLE31tfaHN2innFsBydY5Bbq6GxZufHLOgejeR99SC jc7GgQgtvBUOUzipkjdNvF6qHwIqS9FeQIkVAX4sKhS0Lz1mJnXcX7sDqB3qkkuM7zyP PWhA== X-Gm-Message-State: AOJu0YybvyN2x15582F8TFeNJp4uwz1QX7zYgi+RyFeqX+Y6BxyjS+Bs AHEZ6oOA9qL9annRk8lmrmuxEthBshiDMPHmDtehqev6Qg8/FUqhK+JfCwhYOMKr9sX3XNJYCmh p9safQt+Q1bd41ZWGf21VEz0TyxcCTfMWOm5QAYzoc6N5flSVi0BFVt2U66LzCkksbK5zXl092J G3c1wOU/RXNSYfTws+UJpq4W0zRut6iQ== X-Received: by 2002:a05:6808:1829:b0:3c3:c913:2709 with SMTP id bh41-20020a056808182900b003c3c9132709mr293817oib.2.1711553041315; Wed, 27 Mar 2024 08:24:01 -0700 (PDT) X-Google-Smtp-Source: AGHT+IHAanQflKq3pmTo1+xDGdmPcZlLuONtp7wiA2xhvVSTNiy0v1ZL/DhUjtngy2OWwjyBQX1+lw== X-Received: by 2002:a05:6808:1829:b0:3c3:c913:2709 with SMTP id bh41-20020a056808182900b003c3c9132709mr293761oib.2.1711553040651; Wed, 27 Mar 2024 08:24:00 -0700 (PDT) Received: from x1n.redhat.com ([99.254.121.117]) by smtp.gmail.com with ESMTPSA id hu4-20020a056214234400b00690dd47a41csm6412639qvb.86.2024.03.27.08.23.59 (version=TLS1_3 cipher=TLS_AES_256_GCM_SHA384 bits=256/256); Wed, 27 Mar 2024 08:24:00 -0700 (PDT) From: peterx@redhat.com To: linux-mm@kvack.org, linux-kernel@vger.kernel.org Cc: Yang Shi , "Kirill A . Shutemov" , Mike Kravetz , John Hubbard , Michael Ellerman , peterx@redhat.com, Andrew Jones , Muchun Song , linux-riscv@lists.infradead.org, linuxppc-dev@lists.ozlabs.org, Christophe Leroy , Andrew Morton , Christoph Hellwig , Lorenzo Stoakes , Matthew Wilcox , Rik van Riel , linux-arm-kernel@lists.infradead.org, Andrea Arcangeli , David Hildenbrand , "Aneesh Kumar K . V" , Vlastimil Babka , James Houghton , Jason Gunthorpe , Mike Rapoport , Axel Rasmussen Subject: [PATCH v4 12/13] mm/gup: Handle hugepd for follow_page() Date: Wed, 27 Mar 2024 11:23:31 -0400 Message-ID: <20240327152332.950956-13-peterx@redhat.com> X-Mailer: git-send-email 2.44.0 In-Reply-To: <20240327152332.950956-1-peterx@redhat.com> References: <20240327152332.950956-1-peterx@redhat.com> MIME-Version: 1.0 X-Mimecast-Spam-Score: 0 X-Mimecast-Originator: redhat.com X-Rspamd-Server: rspam08 X-Rspamd-Queue-Id: 8638040026 X-Stat-Signature: 3z77r91pzuodq1etw43csbms5ouqkwu7 X-Rspam-User: X-HE-Tag: 1711553044-6166 X-HE-Meta: U2FsdGVkX1+jggi/6vhFyYQLO0xl7154mj0rQEVeWNuB8z7B0+5lpiUesb4CacEqM5XYXheBLknxoLrZwGgkCYK/aNQqbdGYvZcrqjInhS4T/AOEWJaRwXGsyBjsTTU8QNYFw5Jrt0MAaTVnQLb6gm+rqBQ2Byz6Qst8BniG0NZkkVw0vsSNaEpVDSOkQX8qkW97G9qQIhwP02QMjs2qlBe+DrNqIbg4h5p/jubt1vqXWabPexgyrTpNYlYE3nCGNI9HNv51ihxNEsVJlRXngA/QgvtyktrI5zm08VcaLjqs20HSkkXaiDZ6Z4OClJbZFB2NQOlvOCj2g0YI9VWDgU2JahSfSJ2SKZYUYGux82fTxFDlnmhAmb6nZIcy4xea0qTPmkWQ1+lQlR22IWaisVw7c7zzIoVQIQlvSzZJoobLDBNKC0+JEUsA7+CdajmmPcZlc2k0+D9JVkqLWSjYQROI+VHIhWcTy4Swk65eLZlMf+RtesFiOGenO+VMTLj9XfqVW0ltpEQPnDZMk3jvlolCuK97F/BvNNKe4DZSjChSRar4fbEzm3g1NmhP70G1S5M2wuWk1kFq7tkI1tNnSxC1qGaNJT/hhka9G3eqUpSNTJv8Bi/pTxkBU/6UW7y9J72glyxOcxf+or42vh8ZKl1Aqh+Gp6xLxVJIhOO7gQ1bi94ma5oxsZ78XYLDB3v/ngNTszCyDbnP/68yiTvIP2iICPKPEyqpEue+qKX7laC7WuakxXVDi9i84HBWsjMIHpcAekVJmHMID13QOk0NadxmjC/KZPPY7rBfxFABx6szrOg0Kk2gt4XUFRHTGyiPI+IycE3hJ8kHMVK+1CwVKoFKlPsJ1FQDL2EPwClG7+ftVJ0gabcUuHE92kprO9uyl2c+DtXSS3qHyfSIUc4DXdrxBNBpOtG6T3xeLGTTFTYd5SB/3p7Et9r3Y483+8aNZEnuFpwl5G/2dF4SFjP 0Fs4AMup 0yCjJfiWYtrZimGAr9joOWbx1qMi65mASfXPvLef196eSZQ+IVp4u9x5B6YJWQkRtaGDdfzGLAJF/C2sgLhm4eu/ydmON3y3/xhJ0EuqQL7PjniBr7D7xfiBr/G3d3jRazteqf66yzmwBDc/oiUmTjlGK0ndjSKiceykDwls+FWCymlnzxddMwp8drbWViNqPdw0EZZaHoUBvR2Kg6SlZs6Q4tbFZBQsYeX8bTWmAVmju6wRccfchP/Y0f1lYgse9WOUuVg+C8DbTMwyO/Cj6LQNw6lC6Q8+gMvn/1BOpirTQr1lntzlUjFw60BjDJa94jc1icFpTbfpkxXfzB8Izfy1phwUK091iND0N9++ZYICZBQPq6GAbQlDOA8l1ggZuR8wqf7S5teB4eD85DnPMtR6cKjLrOLHchicweeEa7zW9tm7sUoa0z97dIA== X-Bogosity: Ham, tests=bogofilter, spamicity=0.000000, version=1.2.4 Sender: owner-linux-mm@kvack.org Precedence: bulk X-Loop: owner-majordomo@kvack.org List-ID: List-Subscribe: List-Unsubscribe: From: Peter Xu Hugepd is only used in PowerPC so far on 4K page size kernels where hash mmu is used. follow_page_mask() used to leverage hugetlb APIs to access hugepd entries. Teach follow_page_mask() itself on hugepd. With previous refactors on fast-gup gup_huge_pd(), most of the code can be leveraged. There's something not needed for follow page, for example, gup_hugepte() tries to detect pgtable entry change which will never happen with slow gup (which has the pgtable lock held), but that's not a problem to check. Since follow_page() always only fetch one page, set the end to "address + PAGE_SIZE" should suffice. We will still do the pgtable walk once for each hugetlb page by setting ctx->page_mask properly. One thing worth mentioning is that some level of pgtable's _bad() helper will report is_hugepd() entries as TRUE on Power8 hash MMUs. I think it at least applies to PUD on Power8 with 4K pgsize. It means feeding a hugepd entry to pud_bad() will report a false positive. Let's leave that for now because it can be arch-specific where I am a bit declined to touch. In this patch it's not a problem as long as hugepd is detected before any bad pgtable entries. To allow slow gup like follow_*_page() to access hugepd helpers, hugepd codes are moved to the top. Besides that, the helper record_subpages() will be used by either hugepd or fast-gup now. To avoid "unused function" warnings we must provide a "#ifdef" for it, unfortunately. Signed-off-by: Peter Xu --- mm/gup.c | 269 +++++++++++++++++++++++++++++++++---------------------- 1 file changed, 163 insertions(+), 106 deletions(-) diff --git a/mm/gup.c b/mm/gup.c index a81184b01276..a02463c9420e 100644 --- a/mm/gup.c +++ b/mm/gup.c @@ -500,6 +500,149 @@ static inline void mm_set_has_pinned_flag(unsigned long *mm_flags) } #ifdef CONFIG_MMU + +#if defined(CONFIG_ARCH_HAS_HUGEPD) || defined(CONFIG_HAVE_FAST_GUP) +static int record_subpages(struct page *page, unsigned long sz, + unsigned long addr, unsigned long end, + struct page **pages) +{ + struct page *start_page; + int nr; + + start_page = nth_page(page, (addr & (sz - 1)) >> PAGE_SHIFT); + for (nr = 0; addr != end; nr++, addr += PAGE_SIZE) + pages[nr] = nth_page(start_page, nr); + + return nr; +} +#endif /* CONFIG_ARCH_HAS_HUGEPD || CONFIG_HAVE_FAST_GUP */ + +#ifdef CONFIG_ARCH_HAS_HUGEPD +static unsigned long hugepte_addr_end(unsigned long addr, unsigned long end, + unsigned long sz) +{ + unsigned long __boundary = (addr + sz) & ~(sz-1); + return (__boundary - 1 < end - 1) ? __boundary : end; +} + +static int gup_hugepte(pte_t *ptep, unsigned long sz, unsigned long addr, + unsigned long end, unsigned int flags, + struct page **pages, int *nr) +{ + unsigned long pte_end; + struct page *page; + struct folio *folio; + pte_t pte; + int refs; + + pte_end = (addr + sz) & ~(sz-1); + if (pte_end < end) + end = pte_end; + + pte = huge_ptep_get(ptep); + + if (!pte_access_permitted(pte, flags & FOLL_WRITE)) + return 0; + + /* hugepages are never "special" */ + VM_BUG_ON(!pfn_valid(pte_pfn(pte))); + + page = pte_page(pte); + refs = record_subpages(page, sz, addr, end, pages + *nr); + + folio = try_grab_folio(page, refs, flags); + if (!folio) + return 0; + + if (unlikely(pte_val(pte) != pte_val(ptep_get(ptep)))) { + gup_put_folio(folio, refs, flags); + return 0; + } + + if (!pte_write(pte) && gup_must_unshare(NULL, flags, &folio->page)) { + gup_put_folio(folio, refs, flags); + return 0; + } + + *nr += refs; + folio_set_referenced(folio); + return 1; +} + +/* + * NOTE: currently GUP for a hugepd is only possible on hugetlbfs file + * systems on Power, which does not have issue with folio writeback against + * GUP updates. When hugepd will be extended to support non-hugetlbfs or + * even anonymous memory, we need to do extra check as what we do with most + * of the other folios. See writable_file_mapping_allowed() and + * folio_fast_pin_allowed() for more information. + */ +static int gup_huge_pd(hugepd_t hugepd, unsigned long addr, + unsigned int pdshift, unsigned long end, unsigned int flags, + struct page **pages, int *nr) +{ + pte_t *ptep; + unsigned long sz = 1UL << hugepd_shift(hugepd); + unsigned long next; + + ptep = hugepte_offset(hugepd, addr, pdshift); + do { + next = hugepte_addr_end(addr, end, sz); + if (!gup_hugepte(ptep, sz, addr, end, flags, pages, nr)) + return 0; + } while (ptep++, addr = next, addr != end); + + return 1; +} + +static struct page *follow_hugepd(struct vm_area_struct *vma, hugepd_t hugepd, + unsigned long addr, unsigned int pdshift, + unsigned int flags, + struct follow_page_context *ctx) +{ + struct page *page; + struct hstate *h; + spinlock_t *ptl; + int nr = 0, ret; + pte_t *ptep; + + /* Only hugetlb supports hugepd */ + if (WARN_ON_ONCE(!is_vm_hugetlb_page(vma))) + return ERR_PTR(-EFAULT); + + h = hstate_vma(vma); + ptep = hugepte_offset(hugepd, addr, pdshift); + ptl = huge_pte_lock(h, vma->vm_mm, ptep); + ret = gup_huge_pd(hugepd, addr, pdshift, addr + PAGE_SIZE, + flags, &page, &nr); + spin_unlock(ptl); + + if (ret) { + WARN_ON_ONCE(nr != 1); + ctx->page_mask = (1U << huge_page_order(h)) - 1; + return page; + } + + return NULL; +} +#else /* CONFIG_ARCH_HAS_HUGEPD */ +static inline int gup_huge_pd(hugepd_t hugepd, unsigned long addr, + unsigned int pdshift, unsigned long end, unsigned int flags, + struct page **pages, int *nr) +{ + return 0; +} + +static struct page *follow_hugepd(struct vm_area_struct *vma, hugepd_t hugepd, + unsigned long addr, unsigned int pdshift, + unsigned int flags, + struct follow_page_context *ctx) +{ + return NULL; +} +#endif /* CONFIG_ARCH_HAS_HUGEPD */ + + static struct page *no_page_table(struct vm_area_struct *vma, unsigned int flags, unsigned long address) { @@ -871,6 +1014,9 @@ static struct page *follow_pmd_mask(struct vm_area_struct *vma, return no_page_table(vma, flags, address); if (!pmd_present(pmdval)) return no_page_table(vma, flags, address); + if (unlikely(is_hugepd(__hugepd(pmd_val(pmdval))))) + return follow_hugepd(vma, __hugepd(pmd_val(pmdval)), + address, PMD_SHIFT, flags, ctx); if (pmd_devmap(pmdval)) { ptl = pmd_lock(mm, pmd); page = follow_devmap_pmd(vma, address, pmd, flags, &ctx->pgmap); @@ -921,6 +1067,9 @@ static struct page *follow_pud_mask(struct vm_area_struct *vma, pud = READ_ONCE(*pudp); if (!pud_present(pud)) return no_page_table(vma, flags, address); + if (unlikely(is_hugepd(__hugepd(pud_val(pud))))) + return follow_hugepd(vma, __hugepd(pud_val(pud)), + address, PUD_SHIFT, flags, ctx); if (pud_leaf(pud)) { ptl = pud_lock(mm, pudp); page = follow_huge_pud(vma, address, pudp, flags, ctx); @@ -944,10 +1093,13 @@ static struct page *follow_p4d_mask(struct vm_area_struct *vma, p4dp = p4d_offset(pgdp, address); p4d = READ_ONCE(*p4dp); - if (!p4d_present(p4d)) - return no_page_table(vma, flags, address); BUILD_BUG_ON(p4d_leaf(p4d)); - if (unlikely(p4d_bad(p4d))) + + if (unlikely(is_hugepd(__hugepd(p4d_val(p4d))))) + return follow_hugepd(vma, __hugepd(p4d_val(p4d)), + address, P4D_SHIFT, flags, ctx); + + if (!p4d_present(p4d) || p4d_bad(p4d)) return no_page_table(vma, flags, address); return follow_pud_mask(vma, address, p4dp, flags, ctx); @@ -997,10 +1149,15 @@ static struct page *follow_page_mask(struct vm_area_struct *vma, pgd = pgd_offset(mm, address); - if (pgd_none(*pgd) || unlikely(pgd_bad(*pgd))) - return no_page_table(vma, flags, address); + if (unlikely(is_hugepd(__hugepd(pgd_val(*pgd))))) + page = follow_hugepd(vma, __hugepd(pgd_val(*pgd)), + address, PGDIR_SHIFT, flags, ctx); + else if (pgd_none(*pgd) || unlikely(pgd_bad(*pgd))) + page = no_page_table(vma, flags, address); + else + page = follow_p4d_mask(vma, address, pgd, flags, ctx); - return follow_p4d_mask(vma, address, pgd, flags, ctx); + return page; } struct page *follow_page(struct vm_area_struct *vma, unsigned long address, @@ -2947,106 +3104,6 @@ static int __gup_device_huge_pud(pud_t pud, pud_t *pudp, unsigned long addr, } #endif -static int record_subpages(struct page *page, unsigned long sz, - unsigned long addr, unsigned long end, - struct page **pages) -{ - struct page *start_page; - int nr; - - start_page = nth_page(page, (addr & (sz - 1)) >> PAGE_SHIFT); - for (nr = 0; addr != end; nr++, addr += PAGE_SIZE) - pages[nr] = nth_page(start_page, nr); - - return nr; -} - -#ifdef CONFIG_ARCH_HAS_HUGEPD -static unsigned long hugepte_addr_end(unsigned long addr, unsigned long end, - unsigned long sz) -{ - unsigned long __boundary = (addr + sz) & ~(sz-1); - return (__boundary - 1 < end - 1) ? __boundary : end; -} - -static int gup_hugepte(pte_t *ptep, unsigned long sz, unsigned long addr, - unsigned long end, unsigned int flags, - struct page **pages, int *nr) -{ - unsigned long pte_end; - struct page *page; - struct folio *folio; - pte_t pte; - int refs; - - pte_end = (addr + sz) & ~(sz-1); - if (pte_end < end) - end = pte_end; - - pte = huge_ptep_get(ptep); - - if (!pte_access_permitted(pte, flags & FOLL_WRITE)) - return 0; - - /* hugepages are never "special" */ - VM_BUG_ON(!pfn_valid(pte_pfn(pte))); - - page = pte_page(pte); - refs = record_subpages(page, sz, addr, end, pages + *nr); - - folio = try_grab_folio(page, refs, flags); - if (!folio) - return 0; - - if (unlikely(pte_val(pte) != pte_val(ptep_get(ptep)))) { - gup_put_folio(folio, refs, flags); - return 0; - } - - if (!pte_write(pte) && gup_must_unshare(NULL, flags, &folio->page)) { - gup_put_folio(folio, refs, flags); - return 0; - } - - *nr += refs; - folio_set_referenced(folio); - return 1; -} - -/* - * NOTE: currently GUP for a hugepd is only possible on hugetlbfs file - * systems on Power, which does not have issue with folio writeback against - * GUP updates. When hugepd will be extended to support non-hugetlbfs or - * even anonymous memory, we need to do extra check as what we do with most - * of the other folios. See writable_file_mapping_allowed() and - * folio_fast_pin_allowed() for more information. - */ -static int gup_huge_pd(hugepd_t hugepd, unsigned long addr, - unsigned int pdshift, unsigned long end, unsigned int flags, - struct page **pages, int *nr) -{ - pte_t *ptep; - unsigned long sz = 1UL << hugepd_shift(hugepd); - unsigned long next; - - ptep = hugepte_offset(hugepd, addr, pdshift); - do { - next = hugepte_addr_end(addr, end, sz); - if (!gup_hugepte(ptep, sz, addr, end, flags, pages, nr)) - return 0; - } while (ptep++, addr = next, addr != end); - - return 1; -} -#else -static inline int gup_huge_pd(hugepd_t hugepd, unsigned long addr, - unsigned int pdshift, unsigned long end, unsigned int flags, - struct page **pages, int *nr) -{ - return 0; -} -#endif /* CONFIG_ARCH_HAS_HUGEPD */ - static int gup_huge_pmd(pmd_t orig, pmd_t *pmdp, unsigned long addr, unsigned long end, unsigned int flags, struct page **pages, int *nr)