From patchwork Thu Mar 21 22:07:51 2024 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Peter Xu X-Patchwork-Id: 13599431 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from bombadil.infradead.org (bombadil.infradead.org [198.137.202.133]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (No client certificate requested) by smtp.lore.kernel.org (Postfix) with ESMTPS id 848EDC54E58 for ; Thu, 21 Mar 2024 22:08:21 +0000 (UTC) DKIM-Signature: v=1; a=rsa-sha256; q=dns/txt; c=relaxed/relaxed; d=lists.infradead.org; s=bombadil.20210309; h=Sender: Content-Transfer-Encoding:Content-Type:List-Subscribe:List-Help:List-Post: List-Archive:List-Unsubscribe:List-Id:MIME-Version:References:In-Reply-To: Message-ID:Date:Subject:Cc:To:From:Reply-To:Content-ID:Content-Description: Resent-Date:Resent-From:Resent-Sender:Resent-To:Resent-Cc:Resent-Message-ID: List-Owner; bh=2pXNUEigY5EVQ0GMfUUCeStwsXw//3pd6DKGjVzSM1Y=; b=OvmL5CJvD58Ggd TIejLctipRY1wp5sWgEIJ9KSzuUnHnqxdp797X5OXVlgyzCLd5wUpu/8YOPsYSALTTkRon2kxKHfV V+CN/2jSYrJ7xFIkJpGN6eJkgOhF0W8EmtR6uqQ+1x+liiu6O1LhfhkQA/Qs+LKnfnibCU4YX91T8 lcwDhDlet1ocywucS8oN6i2g5mvLCwxZXJt28ApDHi6mHQ/KcBQ9h0pLAkPvUUx8xviZrP3cMb6g5 Q0zdBXHypDpmM9AFFl2rfINgsyv9yTa4lgvF7f9jtn4ds0X667tUe7MlPzFIIF4kL33UWwXS/qhiI VAcbVwGGxydKN9VBkcTw==; Received: from localhost ([::1] helo=bombadil.infradead.org) by bombadil.infradead.org with esmtp (Exim 4.97.1 #2 (Red Hat Linux)) id 1rnQa7-00000004qJa-3nzR; Thu, 21 Mar 2024 22:08:15 +0000 Received: from us-smtp-delivery-124.mimecast.com ([170.10.133.124]) by bombadil.infradead.org with esmtps (Exim 4.97.1 #2 (Red Hat Linux)) id 1rnQa2-00000004qEO-46EE for linux-riscv@lists.infradead.org; Thu, 21 Mar 2024 22:08:12 +0000 DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=redhat.com; s=mimecast20190719; t=1711058889; h=from:from:reply-to:subject:subject:date:date:message-id:message-id: to:to:cc:cc:mime-version:mime-version:content-type:content-type: content-transfer-encoding:content-transfer-encoding: in-reply-to:in-reply-to:references:references; bh=bzTkuuEUZ2f7C9YouF2ORWZwnTLSeJ8Y5AJhy7kkISQ=; b=Z1QOr5VHwlnXVmwfQMhze8UZ86wYuDJEJRmDmm7Gd/MNIoSjQjosnXLK1+/tX4cfMTLOkW /grErCiT6duLwQtG6YQabPQH0HPqGZbdIpPnBmmZOlDv3FbocaTE0IBk1AYiZw6fGACvtE V0tbvz6WS6cNd0AC3nUqdbM63IySHWk= Received: from mail-qk1-f198.google.com (mail-qk1-f198.google.com [209.85.222.198]) by relay.mimecast.com with ESMTP with STARTTLS (version=TLSv1.3, cipher=TLS_AES_256_GCM_SHA384) id us-mta-488-JR-UATuJN1aJwyGbCJFGcg-1; Thu, 21 Mar 2024 18:08:08 -0400 X-MC-Unique: JR-UATuJN1aJwyGbCJFGcg-1 Received: by mail-qk1-f198.google.com with SMTP id af79cd13be357-78a0e8a60f8so22222985a.0 for ; Thu, 21 Mar 2024 15:08:08 -0700 (PDT) X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20230601; t=1711058887; x=1711663687; h=content-transfer-encoding:mime-version:references:in-reply-to :message-id:date:subject:cc:to:from:x-gm-message-state:from:to:cc :subject:date:message-id:reply-to; bh=bzTkuuEUZ2f7C9YouF2ORWZwnTLSeJ8Y5AJhy7kkISQ=; b=bXbgmonzUXAVnhdvGG7/xixOPXpRP9qRuzkcHV7VYmdUgVw4Go4nXV1wrwGPmy4tqx sdH/5CJh4D3QWZ2wnjhAUMT3XnPA6QuD4uM4C+0AGLZ/0fcWxAfI3WlXza2EltwDUyGv yTAD61BOaLpKNREiIPnliIawhINpJ/ikDiKKEf1TI9lbNkE7YbAMAE0eijUQ0Jdg/tQX YCft73m16lcijiCqjDr/ewaW/ozsITcF0DT8f6qsJoESO9QR/TkT6FmTXgO5DDG2o+rV nfEt1nWX5qk/M3YXVIXDW4M4PxTZbPuDkwuziZbO6VVH1hgdn3HadSjx5azP8kmGwr4B Qb6g== X-Forwarded-Encrypted: i=1; AJvYcCWJ8jIRhyHBLm7tDUyMqB2NRPDMLKEoSP/vjQo9dnpBwjiffZbb2g2bFHTG1XEvr2qCgjLWzhgoNelYjcO+cfSEMUEGZ46Qj4ttAHJFykK4 X-Gm-Message-State: AOJu0YxYanb9xVLkAi4nSI3IeDhsM3+gl/YpZB3x0IO0e0akBg6T053W RJs29cmEXbKWAA9fR/Xz0pbJgKnwgcn4qz42wm9Z08HpEtHxasOrajkOvO4M5a+PBmCqXgYNKND jTHq57osMzlzS+cgjO4izILKwQdjKtDTPIT6XwA6eWL039gbGwc6leqX8YYkjKQ+ptQ== X-Received: by 2002:a0c:e58b:0:b0:691:87dd:f836 with SMTP id t11-20020a0ce58b000000b0069187ddf836mr428470qvm.5.1711058887525; Thu, 21 Mar 2024 15:08:07 -0700 (PDT) X-Google-Smtp-Source: AGHT+IGbvsPPVNStAWHyU+pKNLEFDZ4j8yXA8+9tn+eCe8z7LjIA9ieYt5evzB7ZRQzGTHD3gHSYEw== X-Received: by 2002:a0c:e58b:0:b0:691:87dd:f836 with SMTP id t11-20020a0ce58b000000b0069187ddf836mr428451qvm.5.1711058887085; Thu, 21 Mar 2024 15:08:07 -0700 (PDT) Received: from x1n.redhat.com ([99.254.121.117]) by smtp.gmail.com with ESMTPSA id o6-20020a0562140e4600b00690baf5cde9sm351663qvc.118.2024.03.21.15.08.05 (version=TLS1_3 cipher=TLS_AES_256_GCM_SHA384 bits=256/256); Thu, 21 Mar 2024 15:08:06 -0700 (PDT) From: peterx@redhat.com To: linux-mm@kvack.org, linux-kernel@vger.kernel.org Cc: linuxppc-dev@lists.ozlabs.org, Michael Ellerman , Christophe Leroy , Matthew Wilcox , Rik van Riel , Lorenzo Stoakes , Axel Rasmussen , peterx@redhat.com, Yang Shi , John Hubbard , linux-arm-kernel@lists.infradead.org, "Kirill A . Shutemov" , Andrew Jones , Vlastimil Babka , Mike Rapoport , Andrew Morton , Muchun Song , Christoph Hellwig , linux-riscv@lists.infradead.org, James Houghton , David Hildenbrand , Jason Gunthorpe , Andrea Arcangeli , "Aneesh Kumar K . V" , Mike Kravetz Subject: [PATCH v3 01/12] mm/Kconfig: CONFIG_PGTABLE_HAS_HUGE_LEAVES Date: Thu, 21 Mar 2024 18:07:51 -0400 Message-ID: <20240321220802.679544-2-peterx@redhat.com> X-Mailer: git-send-email 2.44.0 In-Reply-To: <20240321220802.679544-1-peterx@redhat.com> References: <20240321220802.679544-1-peterx@redhat.com> MIME-Version: 1.0 X-Mimecast-Spam-Score: 0 X-Mimecast-Originator: redhat.com X-CRM114-Version: 20100106-BlameMichelson ( TRE 0.8.0 (BSD) ) MR-646709E3 X-CRM114-CacheID: sfid-20240321_150811_100677_012718D6 X-CRM114-Status: UNSURE ( 9.61 ) X-CRM114-Notice: Please train this message. X-BeenThere: linux-riscv@lists.infradead.org X-Mailman-Version: 2.1.34 Precedence: list List-Id: List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Sender: "linux-riscv" Errors-To: linux-riscv-bounces+linux-riscv=archiver.kernel.org@lists.infradead.org From: Peter Xu Introduce a config option that will be selected as long as huge leaves are involved in pgtable (thp or hugetlbfs). It would be useful to mark any code with this new config that can process either hugetlb or thp pages in any level that is higher than pte level. Reviewed-by: Jason Gunthorpe Signed-off-by: Peter Xu --- mm/Kconfig | 6 ++++++ 1 file changed, 6 insertions(+) diff --git a/mm/Kconfig b/mm/Kconfig index b924f4a5a3ef..497cdf4d8ebf 100644 --- a/mm/Kconfig +++ b/mm/Kconfig @@ -850,6 +850,12 @@ config READ_ONLY_THP_FOR_FS endif # TRANSPARENT_HUGEPAGE +# +# The architecture supports pgtable leaves that is larger than PAGE_SIZE +# +config PGTABLE_HAS_HUGE_LEAVES + def_bool TRANSPARENT_HUGEPAGE || HUGETLB_PAGE + # # UP and nommu archs use km based percpu allocator # From patchwork Thu Mar 21 22:07:52 2024 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Peter Xu X-Patchwork-Id: 13599433 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from bombadil.infradead.org (bombadil.infradead.org [198.137.202.133]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (No client certificate requested) by smtp.lore.kernel.org (Postfix) with ESMTPS id 5D3DBCD11BF for ; Thu, 21 Mar 2024 22:08:28 +0000 (UTC) DKIM-Signature: v=1; a=rsa-sha256; q=dns/txt; c=relaxed/relaxed; d=lists.infradead.org; s=bombadil.20210309; h=Sender: Content-Transfer-Encoding:Content-Type:List-Subscribe:List-Help:List-Post: List-Archive:List-Unsubscribe:List-Id:MIME-Version:References:In-Reply-To: Message-ID:Date:Subject:Cc:To:From:Reply-To:Content-ID:Content-Description: Resent-Date:Resent-From:Resent-Sender:Resent-To:Resent-Cc:Resent-Message-ID: List-Owner; bh=4gCaR2alkvlkeRRtFYl57iSfcBJvLC0kH2z44UvugW4=; b=EeYEm2QaMOhOii mWUgqFK/hgw/dEDuOPrPNOLCRylIIacuSRYbU8VAC9us3GrdzQ6bJCG4ul+JcgFN7aWetEICDaaBh L7JC6CEHFqBvf6tnABrUGhnsLwLCc/ITC01TJT4cFMXGsqzaRoV0KK195HBOaLIYapybgf/lkzb8+ 2I+rASfhJjCw2srUEgrvtrAiwiejVQK8RicRn5acwQJAv5c4bJ6ZYsgenCmUb9mVyNamGbnO6iNAQ zaAERdl+uN7f7qRR85TT/saNvRNt6m8c/kQh6ehMNgg/zr8XBTM3KCS7R32E5j5606vufmoTQkfny 2nV6fubeQx/9I83Ornfw==; Received: from localhost ([::1] helo=bombadil.infradead.org) by bombadil.infradead.org with esmtp (Exim 4.97.1 #2 (Red Hat Linux)) id 1rnQaE-00000004qRi-3PB7; Thu, 21 Mar 2024 22:08:22 +0000 Received: from us-smtp-delivery-124.mimecast.com ([170.10.129.124]) by bombadil.infradead.org with esmtps (Exim 4.97.1 #2 (Red Hat Linux)) id 1rnQa5-00000004qGO-1liD for linux-riscv@lists.infradead.org; Thu, 21 Mar 2024 22:08:15 +0000 DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=redhat.com; s=mimecast20190719; t=1711058892; h=from:from:reply-to:subject:subject:date:date:message-id:message-id: to:to:cc:cc:mime-version:mime-version:content-type:content-type: content-transfer-encoding:content-transfer-encoding: in-reply-to:in-reply-to:references:references; bh=Jz9rMLZ3uXx9gi3+G5qYwalRYpIGEA6fjYlj3iYlq7A=; b=Y5OGlwCohUm85fF8pPE2ItbRk/dkc2xD/b+FM39jWqS9yqFh9RalXOBoTGsVYXywhGziP/ fn8MLhnjSBgMR81+YDen+oMZ/BiMj5WIOf4l9vDu6Nmy/GS+f0uGoU7QabkNtkQar0CaPs hf72RG2rO9AntZdM2JMVAzKF7mrJXKw= Received: from mail-vs1-f70.google.com (mail-vs1-f70.google.com [209.85.217.70]) by relay.mimecast.com with ESMTP with STARTTLS (version=TLSv1.3, cipher=TLS_AES_256_GCM_SHA384) id us-mta-287-CJG1rIkzP_WAvGO59ZLq_g-1; Thu, 21 Mar 2024 18:08:10 -0400 X-MC-Unique: CJG1rIkzP_WAvGO59ZLq_g-1 Received: by mail-vs1-f70.google.com with SMTP id ada2fe7eead31-4769277a5fbso157084137.1 for ; Thu, 21 Mar 2024 15:08:10 -0700 (PDT) X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20230601; t=1711058889; x=1711663689; h=content-transfer-encoding:mime-version:references:in-reply-to :message-id:date:subject:cc:to:from:x-gm-message-state:from:to:cc :subject:date:message-id:reply-to; bh=Jz9rMLZ3uXx9gi3+G5qYwalRYpIGEA6fjYlj3iYlq7A=; b=npmOEgIyE2XIIOn3Lq9eo/Mf9LhvNptEhCn29dwcY86Iish1NtzjqrfePjivX3LZL/ C6aZSSut4l7gtJgqFqo/OZ5l9I4T5XdnNU8PaRgPQDmuE0JqtNWEBEqa2eCcyr9DRFdx 1GW4cigDJWf/NiIX66IILJKiUI8tYWD292AIPHTG+HMU1ft3jFfViiv8XyUajOCFUoAo FfY6FOcMNUICdyyaC9juJ7qQq8ZLaQwFKKHt6x2oSsehYzZsRLmUrMoVL/KPV+aRliru cMcHvdDzt9sZNPJ2JdyAVJ+1Ier62ufp4no/ciK/evbA0QDY/4AWyogOAjF6R5uihtwj 0VTQ== X-Forwarded-Encrypted: i=1; AJvYcCV6o13AI7y/AByF9vr962+5FiXvHYXB47SKHgjTer0UMon7I+IWTPdog9v02NRw6/TV+WroenqgS1OX3ozKyE8GirmHpAm10qAD0JMEdYdD X-Gm-Message-State: AOJu0YyI1/lzA3fifZgjnOrc88kmWj67vxFwqU9ARELO5Y+MhVFYovRf DthgNzBQOOP16jAC5L+q33X8xJAOQ5EC3xULOb3eTLnQEUMIDpsA5vLcur4NLKDcmGO1jZ/zcZy pzRs9dgwST1tPHPErTWLLTu9jzhl84+gFRDX2jfy/uEQOR/mCIv9oSWtf+WLCwFQgAg== X-Received: by 2002:a05:6122:6b2:b0:4d3:3a78:60e2 with SMTP id r18-20020a05612206b200b004d33a7860e2mr108409vkq.1.1711058889431; Thu, 21 Mar 2024 15:08:09 -0700 (PDT) X-Google-Smtp-Source: AGHT+IHYDbaW1J9E1O0bYtn9bzXmgf1WJ4RRo+0b7nGhM7jp1NUFRItVBXPR4HtJOs8C30IHezp50w== X-Received: by 2002:a05:6122:6b2:b0:4d3:3a78:60e2 with SMTP id r18-20020a05612206b200b004d33a7860e2mr108396vkq.1.1711058889030; Thu, 21 Mar 2024 15:08:09 -0700 (PDT) Received: from x1n.redhat.com ([99.254.121.117]) by smtp.gmail.com with ESMTPSA id o6-20020a0562140e4600b00690baf5cde9sm351663qvc.118.2024.03.21.15.08.07 (version=TLS1_3 cipher=TLS_AES_256_GCM_SHA384 bits=256/256); Thu, 21 Mar 2024 15:08:08 -0700 (PDT) From: peterx@redhat.com To: linux-mm@kvack.org, linux-kernel@vger.kernel.org Cc: linuxppc-dev@lists.ozlabs.org, Michael Ellerman , Christophe Leroy , Matthew Wilcox , Rik van Riel , Lorenzo Stoakes , Axel Rasmussen , peterx@redhat.com, Yang Shi , John Hubbard , linux-arm-kernel@lists.infradead.org, "Kirill A . Shutemov" , Andrew Jones , Vlastimil Babka , Mike Rapoport , Andrew Morton , Muchun Song , Christoph Hellwig , linux-riscv@lists.infradead.org, James Houghton , David Hildenbrand , Jason Gunthorpe , Andrea Arcangeli , "Aneesh Kumar K . V" , Mike Kravetz Subject: [PATCH v3 02/12] mm/hugetlb: Declare hugetlbfs_pagecache_present() non-static Date: Thu, 21 Mar 2024 18:07:52 -0400 Message-ID: <20240321220802.679544-3-peterx@redhat.com> X-Mailer: git-send-email 2.44.0 In-Reply-To: <20240321220802.679544-1-peterx@redhat.com> References: <20240321220802.679544-1-peterx@redhat.com> MIME-Version: 1.0 X-Mimecast-Spam-Score: 0 X-Mimecast-Originator: redhat.com X-CRM114-Version: 20100106-BlameMichelson ( TRE 0.8.0 (BSD) ) MR-646709E3 X-CRM114-CacheID: sfid-20240321_150813_607855_7A9FC032 X-CRM114-Status: GOOD ( 11.95 ) X-BeenThere: linux-riscv@lists.infradead.org X-Mailman-Version: 2.1.34 Precedence: list List-Id: List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Sender: "linux-riscv" Errors-To: linux-riscv-bounces+linux-riscv=archiver.kernel.org@lists.infradead.org From: Peter Xu It will be used outside hugetlb.c soon. Signed-off-by: Peter Xu --- include/linux/hugetlb.h | 9 +++++++++ mm/hugetlb.c | 4 ++-- 2 files changed, 11 insertions(+), 2 deletions(-) diff --git a/include/linux/hugetlb.h b/include/linux/hugetlb.h index 300de33c6fde..52d9efcf1edf 100644 --- a/include/linux/hugetlb.h +++ b/include/linux/hugetlb.h @@ -174,6 +174,9 @@ u32 hugetlb_fault_mutex_hash(struct address_space *mapping, pgoff_t idx); pte_t *huge_pmd_share(struct mm_struct *mm, struct vm_area_struct *vma, unsigned long addr, pud_t *pud); +bool hugetlbfs_pagecache_present(struct hstate *h, + struct vm_area_struct *vma, + unsigned long address); struct address_space *hugetlb_page_mapping_lock_write(struct page *hpage); @@ -1197,6 +1200,12 @@ static inline void hugetlb_register_node(struct node *node) static inline void hugetlb_unregister_node(struct node *node) { } + +static inline bool hugetlbfs_pagecache_present( + struct hstate *h, struct vm_area_struct *vma, unsigned long address) +{ + return false; +} #endif /* CONFIG_HUGETLB_PAGE */ static inline spinlock_t *huge_pte_lock(struct hstate *h, diff --git a/mm/hugetlb.c b/mm/hugetlb.c index 23ef240ba48a..abec04575c89 100644 --- a/mm/hugetlb.c +++ b/mm/hugetlb.c @@ -6129,8 +6129,8 @@ static vm_fault_t hugetlb_wp(struct mm_struct *mm, struct vm_area_struct *vma, /* * Return whether there is a pagecache page to back given address within VMA. */ -static bool hugetlbfs_pagecache_present(struct hstate *h, - struct vm_area_struct *vma, unsigned long address) +bool hugetlbfs_pagecache_present(struct hstate *h, + struct vm_area_struct *vma, unsigned long address) { struct address_space *mapping = vma->vm_file->f_mapping; pgoff_t idx = linear_page_index(vma, address); From patchwork Thu Mar 21 22:07:53 2024 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Peter Xu X-Patchwork-Id: 13599434 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from bombadil.infradead.org (bombadil.infradead.org [198.137.202.133]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (No client certificate requested) by smtp.lore.kernel.org (Postfix) with ESMTPS id 4FBFFC54E58 for ; Thu, 21 Mar 2024 22:09:00 +0000 (UTC) DKIM-Signature: v=1; a=rsa-sha256; q=dns/txt; c=relaxed/relaxed; d=lists.infradead.org; s=bombadil.20210309; h=Sender: Content-Transfer-Encoding:Content-Type:List-Subscribe:List-Help:List-Post: List-Archive:List-Unsubscribe:List-Id:MIME-Version:References:In-Reply-To: Message-ID:Date:Subject:Cc:To:From:Reply-To:Content-ID:Content-Description: Resent-Date:Resent-From:Resent-Sender:Resent-To:Resent-Cc:Resent-Message-ID: List-Owner; bh=JcOJXOXaFIEC/46sHgfdOCL+K4dYCLDuGWEpwyKp/kg=; b=c4/MO+LLWHIvWx 1vTiq1e5zzHiwFD1hztQgTNjfEx3gof34R9TN17nXoF3IaS1LNgZK/qH3RZlrqSEDNOZsj9xQQhVs JcaYF7VtqmOSrVg4xY8hXASIq7GmEo3DQfUXm+wk0s0yk0YbVhJ9X7Z/vuEJ6QGozgOPYvyNtch0K z4QggmBD62uLKSMyXg6DeqK2kUe+Cyw7npVS38oPzJrOHpxv2B/0fVcjoaSmMGya9TqghowiVwiyQ AAezk0yoZQPYm8lS9Qhe6uaRElzNHOVg9if6dN9nfWw702nUe+5A76m9cuc9oBfeT2NL/h9ogrCPx hzCe7o9828eqSX5a1shQ==; Received: from localhost ([::1] helo=bombadil.infradead.org) by bombadil.infradead.org with esmtp (Exim 4.97.1 #2 (Red Hat Linux)) id 1rnQaj-00000004qqZ-46WN; Thu, 21 Mar 2024 22:08:53 +0000 Received: from us-smtp-delivery-124.mimecast.com ([170.10.133.124]) by bombadil.infradead.org with esmtps (Exim 4.97.1 #2 (Red Hat Linux)) id 1rnQa6-00000004qHd-1m6b for linux-riscv@lists.infradead.org; Thu, 21 Mar 2024 22:08:16 +0000 DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=redhat.com; s=mimecast20190719; t=1711058893; h=from:from:reply-to:subject:subject:date:date:message-id:message-id: to:to:cc:cc:mime-version:mime-version:content-type:content-type: content-transfer-encoding:content-transfer-encoding: in-reply-to:in-reply-to:references:references; bh=wdRUTOTGji9K1EOtUzCYNHXUg1STGKkafiwUdqA6oYA=; b=ddEbQRS7uISRVpiE543XZzi76iOwqW75Ahq9OpyxnA4p7RAtKWfYimvu9giRHxIXv1lSeG FwmMHbjjH/SueJq9l4PB75WWNQNbpI3vOjhMoH1iiFewI0pLk6+AwYIhrJLf3T7ysiC/8l MO5Ao+4uqii5sZIjbsGBImV1lzH6cOo= Received: from mail-qt1-f199.google.com (mail-qt1-f199.google.com [209.85.160.199]) by relay.mimecast.com with ESMTP with STARTTLS (version=TLSv1.3, cipher=TLS_AES_256_GCM_SHA384) id us-mta-114-N_zkJ6hOM5qRPd-AIqUwDg-1; Thu, 21 Mar 2024 18:08:12 -0400 X-MC-Unique: N_zkJ6hOM5qRPd-AIqUwDg-1 Received: by mail-qt1-f199.google.com with SMTP id d75a77b69052e-430c76df729so4253841cf.0 for ; Thu, 21 Mar 2024 15:08:11 -0700 (PDT) X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20230601; t=1711058891; x=1711663691; h=content-transfer-encoding:mime-version:references:in-reply-to :message-id:date:subject:cc:to:from:x-gm-message-state:from:to:cc :subject:date:message-id:reply-to; bh=wdRUTOTGji9K1EOtUzCYNHXUg1STGKkafiwUdqA6oYA=; b=kBlOjodsvJxLNhK8+rFBRDg0y1b0qpGRN000fCpNuYcuoxfzuK51hYEOMZXj84AVc1 +CacFcEZTZlHCNoW/Z11+CCl9GgKEwqiKYUEJrY2lbmIPvszNDLB9IbMTcC1kqKXJPwg GcShy18Vc+NQ6yS9XNgSNvBegnINyGY4LJ8XPntzPMDP46WLkwfM1XdG7B5uhes9K+o2 lNVvVJwULcv5urjK1UgRslHv4HI0ge82//6VXWTiwSCdw2c6kHvZc+3TRtTWJ+I2SODr duGbg8ZqtZ7H+h/yfCJLiqmu7izcuuCLvVF0tsXSLlAetdYIhjmWk1txHv8Gjf/Dfr8G 75Aw== X-Forwarded-Encrypted: i=1; AJvYcCUGixmQdqFgzcsXdWm6LEfDoemHSioYyBt/maYLTaGnAWzaN+vygsYEWOeMfqLrHuFRWcxh+6CcAnkVrLEFPAmi7t1o4yDQv+JiMGH7OEdt X-Gm-Message-State: AOJu0Yxhz7yVdRHYeMCMLHTLnfFp19vCk2Zv/XYNyLjVupU/tkcHGqna LOhfpCVMDfsnQ05/htG//bwFr/MHuS6tXrOpN1h5d/awxsAoGY/K5OHeRauB84xvKDsCf4vVJdH iMfVQX41tEBryb0woU8q4l4DTPQe0YJdWmpmAbwXAYpvDQO7Xf8AGAWEdyHkk01P8Xg== X-Received: by 2002:a05:6214:3a8a:b0:690:3c85:c5b with SMTP id nz10-20020a0562143a8a00b006903c850c5bmr195553qvb.3.1711058891331; Thu, 21 Mar 2024 15:08:11 -0700 (PDT) X-Google-Smtp-Source: AGHT+IG16ns9jJ9MJ80ThL7hh1B1qvLdYLgSNyXX5iyvBQC6076/qVHetDei+F5v2+719IrZ5asQ9g== X-Received: by 2002:a05:6214:3a8a:b0:690:3c85:c5b with SMTP id nz10-20020a0562143a8a00b006903c850c5bmr195527qvb.3.1711058890838; Thu, 21 Mar 2024 15:08:10 -0700 (PDT) Received: from x1n.redhat.com ([99.254.121.117]) by smtp.gmail.com with ESMTPSA id o6-20020a0562140e4600b00690baf5cde9sm351663qvc.118.2024.03.21.15.08.09 (version=TLS1_3 cipher=TLS_AES_256_GCM_SHA384 bits=256/256); Thu, 21 Mar 2024 15:08:10 -0700 (PDT) From: peterx@redhat.com To: linux-mm@kvack.org, linux-kernel@vger.kernel.org Cc: linuxppc-dev@lists.ozlabs.org, Michael Ellerman , Christophe Leroy , Matthew Wilcox , Rik van Riel , Lorenzo Stoakes , Axel Rasmussen , peterx@redhat.com, Yang Shi , John Hubbard , linux-arm-kernel@lists.infradead.org, "Kirill A . Shutemov" , Andrew Jones , Vlastimil Babka , Mike Rapoport , Andrew Morton , Muchun Song , Christoph Hellwig , linux-riscv@lists.infradead.org, James Houghton , David Hildenbrand , Jason Gunthorpe , Andrea Arcangeli , "Aneesh Kumar K . V" , Mike Kravetz Subject: [PATCH v3 03/12] mm: Make HPAGE_PXD_* macros even if !THP Date: Thu, 21 Mar 2024 18:07:53 -0400 Message-ID: <20240321220802.679544-4-peterx@redhat.com> X-Mailer: git-send-email 2.44.0 In-Reply-To: <20240321220802.679544-1-peterx@redhat.com> References: <20240321220802.679544-1-peterx@redhat.com> MIME-Version: 1.0 X-Mimecast-Spam-Score: 0 X-Mimecast-Originator: redhat.com X-CRM114-Version: 20100106-BlameMichelson ( TRE 0.8.0 (BSD) ) MR-646709E3 X-CRM114-CacheID: sfid-20240321_150814_687070_51B15BD5 X-CRM114-Status: GOOD ( 12.29 ) X-BeenThere: linux-riscv@lists.infradead.org X-Mailman-Version: 2.1.34 Precedence: list List-Id: List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Sender: "linux-riscv" Errors-To: linux-riscv-bounces+linux-riscv=archiver.kernel.org@lists.infradead.org From: Peter Xu These macros can be helpful when we plan to merge hugetlb code into generic code. Move them out and define them even if !THP. We actually already defined HPAGE_PMD_NR for other reasons even if !THP. Reorganize these macros. Reviewed-by: Christoph Hellwig Reviewed-by: Jason Gunthorpe Signed-off-by: Peter Xu --- include/linux/huge_mm.h | 17 ++++++----------- 1 file changed, 6 insertions(+), 11 deletions(-) diff --git a/include/linux/huge_mm.h b/include/linux/huge_mm.h index de0c89105076..3bcdfc7e5d57 100644 --- a/include/linux/huge_mm.h +++ b/include/linux/huge_mm.h @@ -64,9 +64,6 @@ ssize_t single_hugepage_flag_show(struct kobject *kobj, enum transparent_hugepage_flag flag); extern struct kobj_attribute shmem_enabled_attr; -#define HPAGE_PMD_ORDER (HPAGE_PMD_SHIFT-PAGE_SHIFT) -#define HPAGE_PMD_NR (1< X-Patchwork-Id: 13599436 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from bombadil.infradead.org (bombadil.infradead.org [198.137.202.133]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (No client certificate requested) by smtp.lore.kernel.org (Postfix) with ESMTPS id 9B2A1C54E58 for ; Thu, 21 Mar 2024 22:09:18 +0000 (UTC) DKIM-Signature: v=1; a=rsa-sha256; q=dns/txt; c=relaxed/relaxed; d=lists.infradead.org; s=bombadil.20210309; h=Sender: Content-Transfer-Encoding:Content-Type:List-Subscribe:List-Help:List-Post: List-Archive:List-Unsubscribe:List-Id:MIME-Version:References:In-Reply-To: Message-ID:Date:Subject:Cc:To:From:Reply-To:Content-ID:Content-Description: Resent-Date:Resent-From:Resent-Sender:Resent-To:Resent-Cc:Resent-Message-ID: List-Owner; bh=9ZUekCjm+W9czQvKVgLGuYUCLzVd0EJdJcuos5jbRSs=; b=PClFEUzhOAXQG3 OPAUylAnGW5vmyhVPU7flHaV2tG7WeBeBjizIuZwC5Fx+u8Mf6jT87qQzFBIt674ZXxipliWeFdXL Ae/nkYdZq9mvCSJHrcIUqY68cpAKMi+GWLnluEp+0Jdl9H5YX+4eExXBa9SOjb72jeWI6voVcgBDO Mj888ONfMjvvy6hQKlKkq4v51sMlhrIzauATkIMYA71/8+1/uM5synnAdkTf2yry30f5rxKGsZmgM i9N+vrTwqn4QGpvUtfFpjbhya9Y28Y9B0eUiUoVvunqxJh9tDu5XkJaYMhPW//q/0ANGelGVQPwU9 AlyVMVP0zKMzACCWIjAA==; Received: from localhost ([::1] helo=bombadil.infradead.org) by bombadil.infradead.org with esmtp (Exim 4.97.1 #2 (Red Hat Linux)) id 1rnQar-00000004qv6-2JNU; Thu, 21 Mar 2024 22:09:01 +0000 Received: from us-smtp-delivery-124.mimecast.com ([170.10.133.124]) by bombadil.infradead.org with esmtps (Exim 4.97.1 #2 (Red Hat Linux)) id 1rnQa8-00000004qJC-07Kw for linux-riscv@lists.infradead.org; Thu, 21 Mar 2024 22:08:17 +0000 DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=redhat.com; s=mimecast20190719; t=1711058895; h=from:from:reply-to:subject:subject:date:date:message-id:message-id: to:to:cc:cc:mime-version:mime-version:content-type:content-type: content-transfer-encoding:content-transfer-encoding: in-reply-to:in-reply-to:references:references; bh=TwZ/C3Y+mq5srgVBOLaWiB530mwVuOVA1xifIUFcqIU=; b=e17x3FP9D4AjpkF8LFJ+st+8sb6zzeFymsTgfF4db2f4ry3muxxWuknm8+wrrmMpFWEjwc HSdUhHJaTeTr+cTELps/QGqfmuIll7HbwOdaGjwT+HSDIOoG8ngfht6+S9h83subskWafc cV5cIQbZlH0LSX8BbREmBn7uumvjWtI= Received: from mail-qv1-f70.google.com (mail-qv1-f70.google.com [209.85.219.70]) by relay.mimecast.com with ESMTP with STARTTLS (version=TLSv1.3, cipher=TLS_AES_256_GCM_SHA384) id us-mta-76-kwUSFjIGPae_PNzUGBg4kA-1; Thu, 21 Mar 2024 18:08:13 -0400 X-MC-Unique: kwUSFjIGPae_PNzUGBg4kA-1 Received: by mail-qv1-f70.google.com with SMTP id 6a1803df08f44-69651ab4c4fso3556856d6.0 for ; Thu, 21 Mar 2024 15:08:13 -0700 (PDT) X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20230601; t=1711058893; x=1711663693; h=content-transfer-encoding:mime-version:references:in-reply-to :message-id:date:subject:cc:to:from:x-gm-message-state:from:to:cc :subject:date:message-id:reply-to; bh=TwZ/C3Y+mq5srgVBOLaWiB530mwVuOVA1xifIUFcqIU=; b=Kmi1LgLDMD/rNFPgxOR7yvl23m6Ux2Au7L8tZW8GEqrFGIpnxfgdIQfSx+K9rZZfUF U+aO9Ts7Tc/enCwTBwvFhjUM1kKyBFamBPfUjpExteiWot4XvVZhVI35ej1Z1OYI1Icq OWBdFiraR70s7aNFXTvtmATp+p4mEQ05y3GETCGuqaNZyH5J7bxc9tz47ApFFW3TsoQh G+4fJSwLotRwzfLII5D7GzkbTF2C+2rei18zREn62KhsEWTd8rV2Hf6NAnITrJipwfL6 XpZsTwwyGFMUgidWgFHGnJHlZl5BsmlbrQ1k5PII9obvqs9ww77iEBEOh7Vdu2oohmKE q7kg== X-Forwarded-Encrypted: i=1; AJvYcCXZUJ5Jjl/0lMxAeiT6iftgOUFMZsFTL8RBy5XbcxWwRNLGUP+Ll5u6KCDoFgkjAY9XQb1njenq8gljiAP1x153aOaLdAgE8hciIQJnj1Ip X-Gm-Message-State: AOJu0YyV9ka8NDX9g+2bn1jvIOZT4IViq9l4+6NfftsnOfWueBHm87Qw 8Lhbx1L7CqQ06fUGvyl74ZPRahAOVX+nm/H+pkvkOPJ/jAqJRDmLhhQE6eYrHn9oGRLZycASSKk zHrodo3jndPAHn+s5zt0ivvgRf+jz+f+dY4OQkZvfW4EPGrs6uUgshFMROPIoIC07iA== X-Received: by 2002:a05:6214:3f89:b0:690:c35c:7590 with SMTP id ow9-20020a0562143f8900b00690c35c7590mr218140qvb.2.1711058893171; Thu, 21 Mar 2024 15:08:13 -0700 (PDT) X-Google-Smtp-Source: AGHT+IGl1Qmf0wBzWRoAAACR5bZA/I9hFYAFMFu/sm/LF6+WMAng7TWyPkikdWZMgBZCHnmtXnvmwg== X-Received: by 2002:a05:6214:3f89:b0:690:c35c:7590 with SMTP id ow9-20020a0562143f8900b00690c35c7590mr218110qvb.2.1711058892727; Thu, 21 Mar 2024 15:08:12 -0700 (PDT) Received: from x1n.redhat.com ([99.254.121.117]) by smtp.gmail.com with ESMTPSA id o6-20020a0562140e4600b00690baf5cde9sm351663qvc.118.2024.03.21.15.08.11 (version=TLS1_3 cipher=TLS_AES_256_GCM_SHA384 bits=256/256); Thu, 21 Mar 2024 15:08:12 -0700 (PDT) From: peterx@redhat.com To: linux-mm@kvack.org, linux-kernel@vger.kernel.org Cc: linuxppc-dev@lists.ozlabs.org, Michael Ellerman , Christophe Leroy , Matthew Wilcox , Rik van Riel , Lorenzo Stoakes , Axel Rasmussen , peterx@redhat.com, Yang Shi , John Hubbard , linux-arm-kernel@lists.infradead.org, "Kirill A . Shutemov" , Andrew Jones , Vlastimil Babka , Mike Rapoport , Andrew Morton , Muchun Song , Christoph Hellwig , linux-riscv@lists.infradead.org, James Houghton , David Hildenbrand , Jason Gunthorpe , Andrea Arcangeli , "Aneesh Kumar K . V" , Mike Kravetz Subject: [PATCH v3 04/12] mm: Introduce vma_pgtable_walk_{begin|end}() Date: Thu, 21 Mar 2024 18:07:54 -0400 Message-ID: <20240321220802.679544-5-peterx@redhat.com> X-Mailer: git-send-email 2.44.0 In-Reply-To: <20240321220802.679544-1-peterx@redhat.com> References: <20240321220802.679544-1-peterx@redhat.com> MIME-Version: 1.0 X-Mimecast-Spam-Score: 0 X-Mimecast-Originator: redhat.com X-CRM114-Version: 20100106-BlameMichelson ( TRE 0.8.0 (BSD) ) MR-646709E3 X-CRM114-CacheID: sfid-20240321_150816_243331_F6612D24 X-CRM114-Status: GOOD ( 11.38 ) X-BeenThere: linux-riscv@lists.infradead.org X-Mailman-Version: 2.1.34 Precedence: list List-Id: List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Sender: "linux-riscv" Errors-To: linux-riscv-bounces+linux-riscv=archiver.kernel.org@lists.infradead.org From: Peter Xu Introduce per-vma begin()/end() helpers for pgtable walks. This is a preparation work to merge hugetlb pgtable walkers with generic mm. The helpers need to be called before and after a pgtable walk, will start to be needed if the pgtable walker code supports hugetlb pages. It's a hook point for any type of VMA, but for now only hugetlb uses it to stablize the pgtable pages from getting away (due to possible pmd unsharing). Reviewed-by: Christoph Hellwig Reviewed-by: Muchun Song Signed-off-by: Peter Xu Reviewed-by: Jason Gunthorpe --- include/linux/mm.h | 3 +++ mm/memory.c | 12 ++++++++++++ 2 files changed, 15 insertions(+) diff --git a/include/linux/mm.h b/include/linux/mm.h index 8147b1302413..d10eb89f4096 100644 --- a/include/linux/mm.h +++ b/include/linux/mm.h @@ -4198,4 +4198,7 @@ static inline bool pfn_is_unaccepted_memory(unsigned long pfn) return range_contains_unaccepted_memory(paddr, paddr + PAGE_SIZE); } +void vma_pgtable_walk_begin(struct vm_area_struct *vma); +void vma_pgtable_walk_end(struct vm_area_struct *vma); + #endif /* _LINUX_MM_H */ diff --git a/mm/memory.c b/mm/memory.c index 9bce1fa76dd7..4f2caf1c3c4d 100644 --- a/mm/memory.c +++ b/mm/memory.c @@ -6438,3 +6438,15 @@ void ptlock_free(struct ptdesc *ptdesc) kmem_cache_free(page_ptl_cachep, ptdesc->ptl); } #endif + +void vma_pgtable_walk_begin(struct vm_area_struct *vma) +{ + if (is_vm_hugetlb_page(vma)) + hugetlb_vma_lock_read(vma); +} + +void vma_pgtable_walk_end(struct vm_area_struct *vma) +{ + if (is_vm_hugetlb_page(vma)) + hugetlb_vma_unlock_read(vma); +} From patchwork Thu Mar 21 22:07:55 2024 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Peter Xu X-Patchwork-Id: 13599435 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from bombadil.infradead.org (bombadil.infradead.org [198.137.202.133]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (No client certificate requested) by smtp.lore.kernel.org (Postfix) with ESMTPS id 57A2EC6FD1F for ; Thu, 21 Mar 2024 22:09:18 +0000 (UTC) DKIM-Signature: v=1; a=rsa-sha256; q=dns/txt; c=relaxed/relaxed; d=lists.infradead.org; s=bombadil.20210309; h=Sender: Content-Transfer-Encoding:Content-Type:List-Subscribe:List-Help:List-Post: List-Archive:List-Unsubscribe:List-Id:MIME-Version:References:In-Reply-To: Message-ID:Date:Subject:Cc:To:From:Reply-To:Content-ID:Content-Description: Resent-Date:Resent-From:Resent-Sender:Resent-To:Resent-Cc:Resent-Message-ID: List-Owner; bh=yfZnqd4TffXVg89S++E+3o/cWViSYQW9xFcT938oyXA=; b=YjW2f00Qhpfv5Y WFpdX8pRVnTkYOEmrmNfKsezdGo0R+x65tXFY+XNF4Ka9dd1N4WqYpgf1d36QYz8UbKYgp+3N+bEB o7KVTTHpmUt9njGj47iZAilofcnHjb0lpLDgagt7p5s5gy1dl7SoF5uZOhLarMBwn26X3wcdqhHeD J9HRR8t8HaTWvbrWqh4W6pYkGcKJdBLG8h25zNqwCQY2zF5a3fIjK1VK98TWvIiWl6GznM8p3gU0x EQ1rVMtCMk0N+gyWOf09krQJIV82/Osxxq+pFevb/mW2Gp9SaiERq3J2Kw7pQGgH5DZFFiK5VAiYl h2VqycbrP+CUlLpZ3rWQ==; Received: from localhost ([::1] helo=bombadil.infradead.org) by bombadil.infradead.org with esmtp (Exim 4.97.1 #2 (Red Hat Linux)) id 1rnQav-00000004qxo-1ghu; Thu, 21 Mar 2024 22:09:09 +0000 Received: from us-smtp-delivery-124.mimecast.com ([170.10.129.124]) by bombadil.infradead.org with esmtps (Exim 4.97.1 #2 (Red Hat Linux)) id 1rnQaA-00000004qM1-0o5k for linux-riscv@lists.infradead.org; Thu, 21 Mar 2024 22:08:20 +0000 DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=redhat.com; s=mimecast20190719; t=1711058897; h=from:from:reply-to:subject:subject:date:date:message-id:message-id: to:to:cc:cc:mime-version:mime-version:content-type:content-type: content-transfer-encoding:content-transfer-encoding: in-reply-to:in-reply-to:references:references; bh=SUPAT0rXQJh3Nc2mzYzsdwstyixoWF7iCZ8XsZkm6u0=; b=M1RrXhhRspIvZd+9r5uYKzQTvq6cv6BXIuBBSvlblcR2Wju6znfH3x1j97oO3y9Cp2snab gSA8mzWZ74gbVx4BfU3AIhB6G0RBrgp1OcP3Og1uvCqcVf+eV+/i6ZP4TKg7e4EBUetifi jOSQ3PpRQ4xYQ5guZZgeypFcZjPdguk= Received: from mail-qk1-f197.google.com (mail-qk1-f197.google.com [209.85.222.197]) by relay.mimecast.com with ESMTP with STARTTLS (version=TLSv1.3, cipher=TLS_AES_256_GCM_SHA384) id us-mta-281-9R6p4aOdPZm8GOsuQWZ2Aw-1; Thu, 21 Mar 2024 18:08:15 -0400 X-MC-Unique: 9R6p4aOdPZm8GOsuQWZ2Aw-1 Received: by mail-qk1-f197.google.com with SMTP id af79cd13be357-78a3986e95bso1842385a.0 for ; Thu, 21 Mar 2024 15:08:15 -0700 (PDT) X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20230601; t=1711058895; x=1711663695; h=content-transfer-encoding:mime-version:references:in-reply-to :message-id:date:subject:cc:to:from:x-gm-message-state:from:to:cc :subject:date:message-id:reply-to; bh=SUPAT0rXQJh3Nc2mzYzsdwstyixoWF7iCZ8XsZkm6u0=; b=GslnUZtuuhMVOij8SBlDlTeiJzZvTOIceN/yCpblCc90qMI8oVQeQn0a8yr+t/aOhv VELBk72PkRiaqWPq76aHUqBEWp6N+QLpl87nRym7KS4eM75+GgDAJaTI/A1UjQddOtUH y7Ythm8zmiTNTaOreH6G+WIOSNj1FCojeozW0BDodgu8OSvZ576XJ1V9MArOkvHUQXX4 N+3fVbWUjLxsbCvLpkbzkT5zIs/fhUSGqdJK2WF+ZuyLWwTMwlSx0QnOnRVGzYeevuhP aTkQ9gjshhn6BaCyfWMr9KcRi/nDqbgkjVJz11RDO4uIZw4ydGL+npM01A++b1qZ4IUL ynYQ== X-Forwarded-Encrypted: i=1; AJvYcCWZmCZDU/LwUQmqq/2KadqU+woCKci7rlggVAONPixKFyndzc3Pugp/WRUgTrx+ZdQcvVNvz2JpY/7PkCYyNF7dyQ5CRIHLNXfsO7WN0E8p X-Gm-Message-State: AOJu0Yy9sqA8AjlnQLMKJnakdP8jul6b2UU2TuLzhBR7ehdIhOQUPFww eeZs7B5IqHHIglpXEo53122QDJwF+cx0SOBgNiUBsHJJhMNetSu4UfNymwIDp1grYpfLn0MoyAt uyuvkPqxIhGpMAtx/DqqnzEs0Eqz4BMFnazdeVV2vzfxr6eE8bfdEpdStwBlTvm6EjQ== X-Received: by 2002:a05:6214:3f89:b0:691:2524:2d87 with SMTP id ow9-20020a0562143f8900b0069125242d87mr214727qvb.2.1711058895193; Thu, 21 Mar 2024 15:08:15 -0700 (PDT) X-Google-Smtp-Source: AGHT+IHuIdayMB3K1ezoBSvCD0gJVl53LosR8J8AUw6cJS5L0fxLggs0ufmC9b8hoGFmk2jAJqKnIA== X-Received: by 2002:a05:6214:3f89:b0:691:2524:2d87 with SMTP id ow9-20020a0562143f8900b0069125242d87mr214690qvb.2.1711058894780; Thu, 21 Mar 2024 15:08:14 -0700 (PDT) Received: from x1n.redhat.com ([99.254.121.117]) by smtp.gmail.com with ESMTPSA id o6-20020a0562140e4600b00690baf5cde9sm351663qvc.118.2024.03.21.15.08.13 (version=TLS1_3 cipher=TLS_AES_256_GCM_SHA384 bits=256/256); Thu, 21 Mar 2024 15:08:14 -0700 (PDT) From: peterx@redhat.com To: linux-mm@kvack.org, linux-kernel@vger.kernel.org Cc: linuxppc-dev@lists.ozlabs.org, Michael Ellerman , Christophe Leroy , Matthew Wilcox , Rik van Riel , Lorenzo Stoakes , Axel Rasmussen , peterx@redhat.com, Yang Shi , John Hubbard , linux-arm-kernel@lists.infradead.org, "Kirill A . Shutemov" , Andrew Jones , Vlastimil Babka , Mike Rapoport , Andrew Morton , Muchun Song , Christoph Hellwig , linux-riscv@lists.infradead.org, James Houghton , David Hildenbrand , Jason Gunthorpe , Andrea Arcangeli , "Aneesh Kumar K . V" , Mike Kravetz Subject: [PATCH v3 05/12] mm/gup: Drop folio_fast_pin_allowed() in hugepd processing Date: Thu, 21 Mar 2024 18:07:55 -0400 Message-ID: <20240321220802.679544-6-peterx@redhat.com> X-Mailer: git-send-email 2.44.0 In-Reply-To: <20240321220802.679544-1-peterx@redhat.com> References: <20240321220802.679544-1-peterx@redhat.com> MIME-Version: 1.0 X-Mimecast-Spam-Score: 0 X-Mimecast-Originator: redhat.com X-CRM114-Version: 20100106-BlameMichelson ( TRE 0.8.0 (BSD) ) MR-646709E3 X-CRM114-CacheID: sfid-20240321_150818_522894_70130E88 X-CRM114-Status: GOOD ( 15.70 ) X-BeenThere: linux-riscv@lists.infradead.org X-Mailman-Version: 2.1.34 Precedence: list List-Id: List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Sender: "linux-riscv" Errors-To: linux-riscv-bounces+linux-riscv=archiver.kernel.org@lists.infradead.org From: Peter Xu Hugepd format for GUP is only used in PowerPC with hugetlbfs. There are some kernel usage of hugepd (can refer to hugepd_populate_kernel() for PPC_8XX), however those pages are not candidates for GUP. Commit a6e79df92e4a ("mm/gup: disallow FOLL_LONGTERM GUP-fast writing to file-backed mappings") added a check to fail gup-fast if there's potential risk of violating GUP over writeback file systems. That should never apply to hugepd. Considering that hugepd is an old format (and even software-only), there's no plan to extend hugepd into other file typed memories that is prone to the same issue. Drop that check, not only because it'll never be true for hugepd per any known plan, but also it paves way for reusing the function outside fast-gup. To make sure we'll still remember this issue just in case hugepd will be extended to support non-hugetlbfs memories, add a rich comment above gup_huge_pd(), explaining the issue with proper references. Cc: Christoph Hellwig Cc: Lorenzo Stoakes Cc: Michael Ellerman Cc: linuxppc-dev@lists.ozlabs.org Signed-off-by: Peter Xu Reviewed-by: Jason Gunthorpe --- mm/gup.c | 13 ++++++++----- 1 file changed, 8 insertions(+), 5 deletions(-) diff --git a/mm/gup.c b/mm/gup.c index 484a7c70d121..9127ec5515ac 100644 --- a/mm/gup.c +++ b/mm/gup.c @@ -2831,11 +2831,6 @@ static int gup_hugepte(pte_t *ptep, unsigned long sz, unsigned long addr, return 0; } - if (!folio_fast_pin_allowed(folio, flags)) { - gup_put_folio(folio, refs, flags); - return 0; - } - if (!pte_write(pte) && gup_must_unshare(NULL, flags, &folio->page)) { gup_put_folio(folio, refs, flags); return 0; @@ -2846,6 +2841,14 @@ static int gup_hugepte(pte_t *ptep, unsigned long sz, unsigned long addr, return 1; } +/* + * NOTE: currently GUP for a hugepd is only possible on hugetlbfs file + * systems on Power, which does not have issue with folio writeback against + * GUP updates. When hugepd will be extended to support non-hugetlbfs or + * even anonymous memory, we need to do extra check as what we do with most + * of the other folios. See writable_file_mapping_allowed() and + * folio_fast_pin_allowed() for more information. + */ static int gup_huge_pd(hugepd_t hugepd, unsigned long addr, unsigned int pdshift, unsigned long end, unsigned int flags, struct page **pages, int *nr) From patchwork Thu Mar 21 22:07:56 2024 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Peter Xu X-Patchwork-Id: 13599437 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from bombadil.infradead.org (bombadil.infradead.org [198.137.202.133]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (No client certificate requested) by smtp.lore.kernel.org (Postfix) with ESMTPS id 5AF8AC6FD1F for ; Thu, 21 Mar 2024 22:09:44 +0000 (UTC) DKIM-Signature: v=1; a=rsa-sha256; q=dns/txt; c=relaxed/relaxed; d=lists.infradead.org; s=bombadil.20210309; h=Sender: Content-Transfer-Encoding:Content-Type:List-Subscribe:List-Help:List-Post: List-Archive:List-Unsubscribe:List-Id:MIME-Version:References:In-Reply-To: Message-ID:Date:Subject:Cc:To:From:Reply-To:Content-ID:Content-Description: Resent-Date:Resent-From:Resent-Sender:Resent-To:Resent-Cc:Resent-Message-ID: List-Owner; bh=TIdSYgnG+6Rt5e6kRhKaO2Jjf5GkDLKg5iwaJO4mQCc=; b=eRYC1mNsqzjdmr 5jyxIQIolawJ21KNQ8jfKHKxvVLWKHgj1sWqrFO4b/mV1hdIUFhLAYpNGl/SMvYqvZd+JIcMc519X vO9tLosunPJEiKS/Jn5lQ2/iIUBvFrS4bKM1/SM/bK7C1Oyk8K10xpkY36QkpKbfyVTApC81UHhGH WXWNRiyAuDxbW16SXmtsgebnHIzbgyIL29krHk0v2BOCRAP6t5cESmXrlMOTMva5VBsR0xIGYQCEL tlk5ywJCQAEvJcYXqXHTiB6bGnxOwa/kefoaLACx5M20S4fimFcYFzxMHVnEaJ16joes5QZ7qEDp+ Qg16RrL039TkDHh1EQJQ==; Received: from localhost ([::1] helo=bombadil.infradead.org) by bombadil.infradead.org with esmtp (Exim 4.97.1 #2 (Red Hat Linux)) id 1rnQbS-00000004rFi-1Mex; Thu, 21 Mar 2024 22:09:38 +0000 Received: from us-smtp-delivery-124.mimecast.com ([170.10.133.124]) by bombadil.infradead.org with esmtps (Exim 4.97.1 #2 (Red Hat Linux)) id 1rnQaC-00000004qOT-2vvy for linux-riscv@lists.infradead.org; Thu, 21 Mar 2024 22:08:23 +0000 DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=redhat.com; s=mimecast20190719; t=1711058899; h=from:from:reply-to:subject:subject:date:date:message-id:message-id: to:to:cc:cc:mime-version:mime-version:content-type:content-type: content-transfer-encoding:content-transfer-encoding: in-reply-to:in-reply-to:references:references; bh=dI3BkO2nCtQEnBDt3PLoa+NL/Gig0J3ZsiByK6YiGEU=; b=Gq0YrZaZJDba3/oiLHQ5k0QmoQm24cQ+2uRUbyM7sQ1AQDcS7Dy2L/3pkR7wVjlqpxGD3z mcAbI1PWYLUrjrMR4xJKymVdTk2i//qVxoi6PSNXD+zK4MGzMyJGLIS+lin8FeDhv0Hmcl 3dxdDIBhC+tRORZawc/LBLVRrZXzBG8= Received: from mail-qk1-f199.google.com (mail-qk1-f199.google.com [209.85.222.199]) by relay.mimecast.com with ESMTP with STARTTLS (version=TLSv1.3, cipher=TLS_AES_256_GCM_SHA384) id us-mta-342-kRwCnjsYOpqLXS7UsPO25A-1; Thu, 21 Mar 2024 18:08:17 -0400 X-MC-Unique: kRwCnjsYOpqLXS7UsPO25A-1 Received: by mail-qk1-f199.google.com with SMTP id af79cd13be357-7830635331bso30344785a.1 for ; Thu, 21 Mar 2024 15:08:17 -0700 (PDT) X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20230601; t=1711058897; x=1711663697; h=content-transfer-encoding:mime-version:references:in-reply-to :message-id:date:subject:cc:to:from:x-gm-message-state:from:to:cc :subject:date:message-id:reply-to; bh=dI3BkO2nCtQEnBDt3PLoa+NL/Gig0J3ZsiByK6YiGEU=; b=QKZGcb0LrD1wE3N2jKJJLVse+UdTbLq4CeiIvoWXfWZRN8ln7Yu+nSjsvr4JiZyFET tQzWBbGhyQA1/WvfQUO4Gk10wsmNWVhplSGhdoQgR1oxkNkyEtPkNYdobaNyRfFwDxH0 J7A0HulcIGLKtaXwUrFmR2wjteTQI2BUSn2G/Hz3EZ7Y0akn3Ulmfw0JkPxdkbWGY3kD bF+VcPsw7sieblfRJ9wadyIUZDZSR6DRvfv+BS4Z/VvAATloglKK9WRmmj5a0ZOxRKvk NN3MrrDCsEZcCjE5uk7HBS2SBrAW8rXdlBExdCAYkil0QNl0iWXqm82jL4b7HqMB0ofc mV0A== X-Forwarded-Encrypted: i=1; AJvYcCULMZ/ZlYtFEuZF2w+qgMuaZ3CZr3N1F8EDgH/2rzIX0XuwotI0IYtihnftREQx8Z9uy8KNa2ke1XZptM8n6Gn2CltvdBm2gqt3W1kiPfNg X-Gm-Message-State: AOJu0Yy1ImjQEwlbYjjj5ASk5mRDToU9aK8LrDi3P0binHq+cc6CZjyP RJCM+7g0tNrY/K7rVR3FF75xq8J7m0ckcs7KT9PE6Jo2WKPkam8uXWc0pBJN5GO9k7dZwfxXbDZ PqKWxj3oh1c6a0m3WdH9qlfPC0wAgnhQQZKaYDwikjKS2qExspHs20bboUbX1ov4mLQ== X-Received: by 2002:a05:6214:4489:b0:68f:e779:70f2 with SMTP id on9-20020a056214448900b0068fe77970f2mr443538qvb.3.1711058897151; Thu, 21 Mar 2024 15:08:17 -0700 (PDT) X-Google-Smtp-Source: AGHT+IHGuIUGmtGDilXkO+Txm1IkzpifWGQBAYBHvb3QP196cSPWL36y2snARbq2rVPrLWqoQw6FQA== X-Received: by 2002:a05:6214:4489:b0:68f:e779:70f2 with SMTP id on9-20020a056214448900b0068fe77970f2mr443515qvb.3.1711058896774; Thu, 21 Mar 2024 15:08:16 -0700 (PDT) Received: from x1n.redhat.com ([99.254.121.117]) by smtp.gmail.com with ESMTPSA id o6-20020a0562140e4600b00690baf5cde9sm351663qvc.118.2024.03.21.15.08.14 (version=TLS1_3 cipher=TLS_AES_256_GCM_SHA384 bits=256/256); Thu, 21 Mar 2024 15:08:16 -0700 (PDT) From: peterx@redhat.com To: linux-mm@kvack.org, linux-kernel@vger.kernel.org Cc: linuxppc-dev@lists.ozlabs.org, Michael Ellerman , Christophe Leroy , Matthew Wilcox , Rik van Riel , Lorenzo Stoakes , Axel Rasmussen , peterx@redhat.com, Yang Shi , John Hubbard , linux-arm-kernel@lists.infradead.org, "Kirill A . Shutemov" , Andrew Jones , Vlastimil Babka , Mike Rapoport , Andrew Morton , Muchun Song , Christoph Hellwig , linux-riscv@lists.infradead.org, James Houghton , David Hildenbrand , Jason Gunthorpe , Andrea Arcangeli , "Aneesh Kumar K . V" , Mike Kravetz Subject: [PATCH v3 06/12] mm/gup: Refactor record_subpages() to find 1st small page Date: Thu, 21 Mar 2024 18:07:56 -0400 Message-ID: <20240321220802.679544-7-peterx@redhat.com> X-Mailer: git-send-email 2.44.0 In-Reply-To: <20240321220802.679544-1-peterx@redhat.com> References: <20240321220802.679544-1-peterx@redhat.com> MIME-Version: 1.0 X-Mimecast-Spam-Score: 0 X-Mimecast-Originator: redhat.com X-CRM114-Version: 20100106-BlameMichelson ( TRE 0.8.0 (BSD) ) MR-646709E3 X-CRM114-CacheID: sfid-20240321_150820_997565_0E6EAAAC X-CRM114-Status: GOOD ( 10.93 ) X-BeenThere: linux-riscv@lists.infradead.org X-Mailman-Version: 2.1.34 Precedence: list List-Id: List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Sender: "linux-riscv" Errors-To: linux-riscv-bounces+linux-riscv=archiver.kernel.org@lists.infradead.org From: Peter Xu All the fast-gup functions take a tail page to operate, always need to do page mask calculations before feeding that into record_subpages(). Merge that logic into record_subpages(), so that it will do the nth_page() calculation. Reviewed-by: Jason Gunthorpe Signed-off-by: Peter Xu --- mm/gup.c | 25 ++++++++++++++----------- 1 file changed, 14 insertions(+), 11 deletions(-) diff --git a/mm/gup.c b/mm/gup.c index 9127ec5515ac..f3ae8f6ce8a4 100644 --- a/mm/gup.c +++ b/mm/gup.c @@ -2778,13 +2778,16 @@ static int __gup_device_huge_pud(pud_t pud, pud_t *pudp, unsigned long addr, } #endif -static int record_subpages(struct page *page, unsigned long addr, - unsigned long end, struct page **pages) +static int record_subpages(struct page *page, unsigned long sz, + unsigned long addr, unsigned long end, + struct page **pages) { + struct page *start_page; int nr; + start_page = nth_page(page, (addr & (sz - 1)) >> PAGE_SHIFT); for (nr = 0; addr != end; nr++, addr += PAGE_SIZE) - pages[nr] = nth_page(page, nr); + pages[nr] = nth_page(start_page, nr); return nr; } @@ -2819,8 +2822,8 @@ static int gup_hugepte(pte_t *ptep, unsigned long sz, unsigned long addr, /* hugepages are never "special" */ VM_BUG_ON(!pfn_valid(pte_pfn(pte))); - page = nth_page(pte_page(pte), (addr & (sz - 1)) >> PAGE_SHIFT); - refs = record_subpages(page, addr, end, pages + *nr); + page = pte_page(pte); + refs = record_subpages(page, sz, addr, end, pages + *nr); folio = try_grab_folio(page, refs, flags); if (!folio) @@ -2893,8 +2896,8 @@ static int gup_huge_pmd(pmd_t orig, pmd_t *pmdp, unsigned long addr, pages, nr); } - page = nth_page(pmd_page(orig), (addr & ~PMD_MASK) >> PAGE_SHIFT); - refs = record_subpages(page, addr, end, pages + *nr); + page = pmd_page(orig); + refs = record_subpages(page, PMD_SIZE, addr, end, pages + *nr); folio = try_grab_folio(page, refs, flags); if (!folio) @@ -2937,8 +2940,8 @@ static int gup_huge_pud(pud_t orig, pud_t *pudp, unsigned long addr, pages, nr); } - page = nth_page(pud_page(orig), (addr & ~PUD_MASK) >> PAGE_SHIFT); - refs = record_subpages(page, addr, end, pages + *nr); + page = pud_page(orig); + refs = record_subpages(page, PUD_SIZE, addr, end, pages + *nr); folio = try_grab_folio(page, refs, flags); if (!folio) @@ -2977,8 +2980,8 @@ static int gup_huge_pgd(pgd_t orig, pgd_t *pgdp, unsigned long addr, BUILD_BUG_ON(pgd_devmap(orig)); - page = nth_page(pgd_page(orig), (addr & ~PGDIR_MASK) >> PAGE_SHIFT); - refs = record_subpages(page, addr, end, pages + *nr); + page = pgd_page(orig); + refs = record_subpages(page, PGDIR_SIZE, addr, end, pages + *nr); folio = try_grab_folio(page, refs, flags); if (!folio) From patchwork Thu Mar 21 22:07:57 2024 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Peter Xu X-Patchwork-Id: 13599438 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from bombadil.infradead.org (bombadil.infradead.org [198.137.202.133]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (No client certificate requested) by smtp.lore.kernel.org (Postfix) with ESMTPS id A6F1EC6FD1F for ; Thu, 21 Mar 2024 22:10:25 +0000 (UTC) DKIM-Signature: v=1; a=rsa-sha256; q=dns/txt; c=relaxed/relaxed; d=lists.infradead.org; s=bombadil.20210309; h=Sender: Content-Transfer-Encoding:Content-Type:List-Subscribe:List-Help:List-Post: List-Archive:List-Unsubscribe:List-Id:MIME-Version:References:In-Reply-To: Message-ID:Date:Subject:Cc:To:From:Reply-To:Content-ID:Content-Description: Resent-Date:Resent-From:Resent-Sender:Resent-To:Resent-Cc:Resent-Message-ID: List-Owner; bh=buUekmyP1ORhLJ0yhWKax++Y+pzOwrNHijceCs/2MVU=; b=0PrXwRFz2thML4 Rd0gdmCcgQgFdgkgSm8iZycuimSPyveSq2H+bK87XUa6O0SA0WZ0cSXHN/ViX3V0qk9EV/DctIpKu twvlyckeEqkMLLjeqIZMV9l4VBfU9PQFqNzxUtYwNGSLRv2+x4Ozo0dZvTCJqTX+gbIh6DiJsSZqb lCSPTIwYvQ/fPBCvr6RLo2lol+Ky8GtdjzVCn341aAqVm3ZIuAx/xXm94yJJLxTUGIbEKqICyndIV 681IOHymr5OgQctZUgH3UU8HqufN5aehNmMpuQEN7DQuipEFHlw1zGprHBieioD+2PU8d+xeUKzer t1BxP+nhNDLm+n3H8elA==; Received: from localhost ([::1] helo=bombadil.infradead.org) by bombadil.infradead.org with esmtp (Exim 4.97.1 #2 (Red Hat Linux)) id 1rnQc8-00000004rmZ-20dP; Thu, 21 Mar 2024 22:10:20 +0000 Received: from us-smtp-delivery-124.mimecast.com ([170.10.129.124]) by bombadil.infradead.org with esmtps (Exim 4.97.1 #2 (Red Hat Linux)) id 1rnQaG-00000004qSQ-0R8I for linux-riscv@lists.infradead.org; Thu, 21 Mar 2024 22:08:29 +0000 DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=redhat.com; s=mimecast20190719; t=1711058903; h=from:from:reply-to:subject:subject:date:date:message-id:message-id: to:to:cc:cc:mime-version:mime-version:content-type:content-type: content-transfer-encoding:content-transfer-encoding: in-reply-to:in-reply-to:references:references; bh=U+6dxwHwA9JOroUFOiyHgWbjpU2ZpFB4CR6YlbtDU08=; b=L5b2zZqw83lTFjcAAV0+mINOwkPlN3+SGs569sETtf0Pweq9rdG7bK+pwZoZYYbRTgTHCS p7f16rbOQ7rxC1Tuss6fIFflnCLfqA+PZcXtKmUkKbzuB/HXubDzES2KI7FiKv9rC+8Z+d n/WHpOBprRbefeCxrqwJBf/I/mvT5tU= Received: from mail-qv1-f69.google.com (mail-qv1-f69.google.com [209.85.219.69]) by relay.mimecast.com with ESMTP with STARTTLS (version=TLSv1.3, cipher=TLS_AES_256_GCM_SHA384) id us-mta-647-2Iq9AuaDNt6nhaAWhrcNjA-1; Thu, 21 Mar 2024 18:08:19 -0400 X-MC-Unique: 2Iq9AuaDNt6nhaAWhrcNjA-1 Received: by mail-qv1-f69.google.com with SMTP id 6a1803df08f44-6961a54234cso2674336d6.1 for ; Thu, 21 Mar 2024 15:08:19 -0700 (PDT) X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20230601; t=1711058899; x=1711663699; h=content-transfer-encoding:mime-version:references:in-reply-to :message-id:date:subject:cc:to:from:x-gm-message-state:from:to:cc :subject:date:message-id:reply-to; bh=U+6dxwHwA9JOroUFOiyHgWbjpU2ZpFB4CR6YlbtDU08=; b=a05dcXkkdTZ7LesFPVFPZe0Euk+oEnUB8+Geit1YwGYhKQFdNye4p47Fhc57O8BWZl CMotXubCCzbToIjmBqfxwssvL7U399qg5A5wdYBMVShOT/0J3zCp/NDRuLC6J/GRvOZw New/8tXlZwVPqG/AtOS+N3WCKqOwwRrRP0MZt6f/zn7V3lIOCFextjh61+jHuXuCX0kv JYjwKCQ2fJwlMZiVFL21guQevLpMd7J2Yk0aBw61YyVAQlns25RowScoDQGCPLVV0NpY je1kDTT9XCXON3nKqbL0dMIFmNlHODr45l35ApsXVbSn3kUlPVmNl5z8PauguKKXS82p gEgg== X-Forwarded-Encrypted: i=1; AJvYcCV4YJJdmukmU2M0iQv+qyiYf5bcQfcCaqyScYxpfdZIwwqYGiImeGHrJ54DKGdoy3GQc1l68QybD/HfxLO02UhAt0//3feiSBKdb4QpcOuu X-Gm-Message-State: AOJu0Yxwq3tff3Ozlvk0yxLQYgVj8SQMfbLuMvv77elTFDLIf6/BGYG8 OQEaR4e5iA+dzYZv4l9n2IS4GghdhtWOmnJajhiFU2xbyEjdIBiYtO88PNNWCfVteykeReUJmaX dUcEuhZCuPRkj7tw5P5plwxH/3BxXijd2cQVfq2VphXh12ud2V4cWdWqPLfWme70rSg== X-Received: by 2002:a05:6214:3912:b0:691:456f:415a with SMTP id nh18-20020a056214391200b00691456f415amr179393qvb.4.1711058899092; Thu, 21 Mar 2024 15:08:19 -0700 (PDT) X-Google-Smtp-Source: AGHT+IFnkSxmKz4azbUELRR0cRTvXrQr4udCnJcoTFPGMx5l/qOyVfxe6G6Q7JjdIdQ6Zo0lZ2QFGA== X-Received: by 2002:a05:6214:3912:b0:691:456f:415a with SMTP id nh18-20020a056214391200b00691456f415amr179365qvb.4.1711058898616; Thu, 21 Mar 2024 15:08:18 -0700 (PDT) Received: from x1n.redhat.com ([99.254.121.117]) by smtp.gmail.com with ESMTPSA id o6-20020a0562140e4600b00690baf5cde9sm351663qvc.118.2024.03.21.15.08.16 (version=TLS1_3 cipher=TLS_AES_256_GCM_SHA384 bits=256/256); Thu, 21 Mar 2024 15:08:18 -0700 (PDT) From: peterx@redhat.com To: linux-mm@kvack.org, linux-kernel@vger.kernel.org Cc: linuxppc-dev@lists.ozlabs.org, Michael Ellerman , Christophe Leroy , Matthew Wilcox , Rik van Riel , Lorenzo Stoakes , Axel Rasmussen , peterx@redhat.com, Yang Shi , John Hubbard , linux-arm-kernel@lists.infradead.org, "Kirill A . Shutemov" , Andrew Jones , Vlastimil Babka , Mike Rapoport , Andrew Morton , Muchun Song , Christoph Hellwig , linux-riscv@lists.infradead.org, James Houghton , David Hildenbrand , Jason Gunthorpe , Andrea Arcangeli , "Aneesh Kumar K . V" , Mike Kravetz Subject: [PATCH v3 07/12] mm/gup: Handle hugetlb for no_page_table() Date: Thu, 21 Mar 2024 18:07:57 -0400 Message-ID: <20240321220802.679544-8-peterx@redhat.com> X-Mailer: git-send-email 2.44.0 In-Reply-To: <20240321220802.679544-1-peterx@redhat.com> References: <20240321220802.679544-1-peterx@redhat.com> MIME-Version: 1.0 X-Mimecast-Spam-Score: 0 X-Mimecast-Originator: redhat.com X-CRM114-Version: 20100106-BlameMichelson ( TRE 0.8.0 (BSD) ) MR-646709E3 X-CRM114-CacheID: sfid-20240321_150824_482747_99F6CC77 X-CRM114-Status: GOOD ( 14.49 ) X-BeenThere: linux-riscv@lists.infradead.org X-Mailman-Version: 2.1.34 Precedence: list List-Id: List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Sender: "linux-riscv" Errors-To: linux-riscv-bounces+linux-riscv=archiver.kernel.org@lists.infradead.org From: Peter Xu no_page_table() is not yet used for hugetlb code paths. Make it prepared. The major difference here is hugetlb will return -EFAULT as long as page cache does not exist, even if VM_SHARED. See hugetlb_follow_page_mask(). Pass "address" into no_page_table() too, as hugetlb will need it. Reviewed-by: Christoph Hellwig Reviewed-by: Jason Gunthorpe Signed-off-by: Peter Xu --- mm/gup.c | 44 ++++++++++++++++++++++++++------------------ 1 file changed, 26 insertions(+), 18 deletions(-) diff --git a/mm/gup.c b/mm/gup.c index f3ae8f6ce8a4..943671736080 100644 --- a/mm/gup.c +++ b/mm/gup.c @@ -501,19 +501,27 @@ static inline void mm_set_has_pinned_flag(unsigned long *mm_flags) #ifdef CONFIG_MMU static struct page *no_page_table(struct vm_area_struct *vma, - unsigned int flags) + unsigned int flags, unsigned long address) { + if (!(flags & FOLL_DUMP)) + return NULL; + /* - * When core dumping an enormous anonymous area that nobody - * has touched so far, we don't want to allocate unnecessary pages or + * When core dumping, we don't want to allocate unnecessary pages or * page tables. Return error instead of NULL to skip handle_mm_fault, * then get_dump_page() will return NULL to leave a hole in the dump. * But we can only make this optimization where a hole would surely * be zero-filled if handle_mm_fault() actually did handle it. */ - if ((flags & FOLL_DUMP) && - (vma_is_anonymous(vma) || !vma->vm_ops->fault)) + if (is_vm_hugetlb_page(vma)) { + struct hstate *h = hstate_vma(vma); + + if (!hugetlbfs_pagecache_present(h, vma, address)) + return ERR_PTR(-EFAULT); + } else if ((vma_is_anonymous(vma) || !vma->vm_ops->fault)) { return ERR_PTR(-EFAULT); + } + return NULL; } @@ -593,7 +601,7 @@ static struct page *follow_page_pte(struct vm_area_struct *vma, ptep = pte_offset_map_lock(mm, pmd, address, &ptl); if (!ptep) - return no_page_table(vma, flags); + return no_page_table(vma, flags, address); pte = ptep_get(ptep); if (!pte_present(pte)) goto no_page; @@ -685,7 +693,7 @@ static struct page *follow_page_pte(struct vm_area_struct *vma, pte_unmap_unlock(ptep, ptl); if (!pte_none(pte)) return NULL; - return no_page_table(vma, flags); + return no_page_table(vma, flags, address); } static struct page *follow_pmd_mask(struct vm_area_struct *vma, @@ -701,27 +709,27 @@ static struct page *follow_pmd_mask(struct vm_area_struct *vma, pmd = pmd_offset(pudp, address); pmdval = pmdp_get_lockless(pmd); if (pmd_none(pmdval)) - return no_page_table(vma, flags); + return no_page_table(vma, flags, address); if (!pmd_present(pmdval)) - return no_page_table(vma, flags); + return no_page_table(vma, flags, address); if (pmd_devmap(pmdval)) { ptl = pmd_lock(mm, pmd); page = follow_devmap_pmd(vma, address, pmd, flags, &ctx->pgmap); spin_unlock(ptl); if (page) return page; - return no_page_table(vma, flags); + return no_page_table(vma, flags, address); } if (likely(!pmd_trans_huge(pmdval))) return follow_page_pte(vma, address, pmd, flags, &ctx->pgmap); if (pmd_protnone(pmdval) && !gup_can_follow_protnone(vma, flags)) - return no_page_table(vma, flags); + return no_page_table(vma, flags, address); ptl = pmd_lock(mm, pmd); if (unlikely(!pmd_present(*pmd))) { spin_unlock(ptl); - return no_page_table(vma, flags); + return no_page_table(vma, flags, address); } if (unlikely(!pmd_trans_huge(*pmd))) { spin_unlock(ptl); @@ -752,17 +760,17 @@ static struct page *follow_pud_mask(struct vm_area_struct *vma, pud = pud_offset(p4dp, address); if (pud_none(*pud)) - return no_page_table(vma, flags); + return no_page_table(vma, flags, address); if (pud_devmap(*pud)) { ptl = pud_lock(mm, pud); page = follow_devmap_pud(vma, address, pud, flags, &ctx->pgmap); spin_unlock(ptl); if (page) return page; - return no_page_table(vma, flags); + return no_page_table(vma, flags, address); } if (unlikely(pud_bad(*pud))) - return no_page_table(vma, flags); + return no_page_table(vma, flags, address); return follow_pmd_mask(vma, address, pud, flags, ctx); } @@ -777,10 +785,10 @@ static struct page *follow_p4d_mask(struct vm_area_struct *vma, p4dp = p4d_offset(pgdp, address); p4d = READ_ONCE(*p4dp); if (!p4d_present(p4d)) - return no_page_table(vma, flags); + return no_page_table(vma, flags, address); BUILD_BUG_ON(p4d_leaf(p4d)); if (unlikely(p4d_bad(p4d))) - return no_page_table(vma, flags); + return no_page_table(vma, flags, address); return follow_pud_mask(vma, address, p4dp, flags, ctx); } @@ -830,7 +838,7 @@ static struct page *follow_page_mask(struct vm_area_struct *vma, pgd = pgd_offset(mm, address); if (pgd_none(*pgd) || unlikely(pgd_bad(*pgd))) - return no_page_table(vma, flags); + return no_page_table(vma, flags, address); return follow_p4d_mask(vma, address, pgd, flags, ctx); } From patchwork Thu Mar 21 22:07:58 2024 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Peter Xu X-Patchwork-Id: 13599439 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from bombadil.infradead.org (bombadil.infradead.org [198.137.202.133]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (No client certificate requested) by smtp.lore.kernel.org (Postfix) with ESMTPS id 8CC4AC6FD1F for ; Thu, 21 Mar 2024 22:10:36 +0000 (UTC) DKIM-Signature: v=1; a=rsa-sha256; q=dns/txt; c=relaxed/relaxed; d=lists.infradead.org; s=bombadil.20210309; h=Sender: Content-Transfer-Encoding:Content-Type:List-Subscribe:List-Help:List-Post: List-Archive:List-Unsubscribe:List-Id:MIME-Version:References:In-Reply-To: Message-ID:Date:Subject:Cc:To:From:Reply-To:Content-ID:Content-Description: Resent-Date:Resent-From:Resent-Sender:Resent-To:Resent-Cc:Resent-Message-ID: List-Owner; bh=qoU13PGoZpDRXkHUBotxWowri83FVemOj7s01kOpx9k=; b=m62R/rW7PyzkZW fyMnmhxr98JYVOrvxYF86h47AB9kPbUJuYn/PY30ntp4VBEFk10sKd9AYlt8kqlth3VkpRxd68x5w NX1nhndeb8jxKF0vk+99k0KxPr4tdbGkqCxth+2ErwYay+IABABGwhE+Fjf2SANnpPmOqeYZIGqlq j1vqoiRlECjz0ZRvPycMCbRhdkk4wslGU++XSInc8Y2dIziA4fcWmy35Kh7nCwlwZRQWFtJUp9tI6 gRiFW4Rm3tMzrV2ISZ+LYz8AOvb3hdNPtPF8Nk46RWWoh3jqVKvNztmIInEvehMpbKDZr9lRg71Oj zCR3J3bF8UzzUBKaUnlg==; Received: from localhost ([::1] helo=bombadil.infradead.org) by bombadil.infradead.org with esmtp (Exim 4.97.1 #2 (Red Hat Linux)) id 1rnQcJ-00000004rv8-1H19; Thu, 21 Mar 2024 22:10:31 +0000 Received: from desiato.infradead.org ([2001:8b0:10b:1:d65d:64ff:fe57:4e05]) by bombadil.infradead.org with esmtps (Exim 4.97.1 #2 (Red Hat Linux)) id 1rnQaR-00000004qas-1dBG for linux-riscv@bombadil.infradead.org; Thu, 21 Mar 2024 22:08:37 +0000 DKIM-Signature: v=1; a=rsa-sha256; q=dns/txt; c=relaxed/relaxed; d=infradead.org; s=desiato.20200630; h=Content-Type:Content-Transfer-Encoding :MIME-Version:References:In-Reply-To:Message-ID:Date:Subject:Cc:To:From: Sender:Reply-To:Content-ID:Content-Description; bh=ha5DGM0IN3r6kIFCrMZt60pb/ho2ZGKh4AhFb2qqPrM=; b=g1MUvdV3W8YxFTpOQXPvVxbOFz 3LX97qUU8vfzXKD6pFDFDB3UTCUPw+dj8rG73zKVOgXZxs2bmO2Z7cCu56D5i75wadNGx7Kn++jYp e3EY4967XMhyoqqy6RSkrnupGkoy9LHvhLWqc1raue10UnGX9Xy22wiexjrzbczABvS8QEl37C7p0 x+jMIj3i+xhsZrY4hU2z9WOz1P0aCVUVjZJZNOL1BYSuY0EdlwPwJzr2protw62QG/UI1aWLIBCtY uvuA6aQGB1HJbvL5s7a1xO3edd/m02HIWL00yK8dNY/GXZxGqsNkiwDHYr/fDSs4LcYeLWnN89tdE ThKGcnfw==; Received: from us-smtp-delivery-124.mimecast.com ([170.10.129.124]) by desiato.infradead.org with esmtps (Exim 4.97.1 #2 (Red Hat Linux)) id 1rnQaL-0000000ESZu-3l4A for linux-riscv@lists.infradead.org; Thu, 21 Mar 2024 22:08:33 +0000 DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=redhat.com; s=mimecast20190719; t=1711058905; h=from:from:reply-to:subject:subject:date:date:message-id:message-id: to:to:cc:cc:mime-version:mime-version:content-type:content-type: content-transfer-encoding:content-transfer-encoding: in-reply-to:in-reply-to:references:references; bh=ha5DGM0IN3r6kIFCrMZt60pb/ho2ZGKh4AhFb2qqPrM=; b=ckOUdmzDPoMlV6BrrHRXMzbwIZQpg350yNfe4fPr6sUofOytxXlRCgta8KmyAO6kIPWEyY emsuB/M7x3Iu412R86bpBHI5PR9HbMlk0gUAo1zn7+9mXIVyPy0YoP4H8rcu4VYriu7Dib 2FLUXXUkr2FE7ZJNqz74q1YW4u28wl8= DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=redhat.com; s=mimecast20190719; t=1711058905; h=from:from:reply-to:subject:subject:date:date:message-id:message-id: to:to:cc:cc:mime-version:mime-version:content-type:content-type: content-transfer-encoding:content-transfer-encoding: in-reply-to:in-reply-to:references:references; bh=ha5DGM0IN3r6kIFCrMZt60pb/ho2ZGKh4AhFb2qqPrM=; b=ckOUdmzDPoMlV6BrrHRXMzbwIZQpg350yNfe4fPr6sUofOytxXlRCgta8KmyAO6kIPWEyY emsuB/M7x3Iu412R86bpBHI5PR9HbMlk0gUAo1zn7+9mXIVyPy0YoP4H8rcu4VYriu7Dib 2FLUXXUkr2FE7ZJNqz74q1YW4u28wl8= Received: from mail-qv1-f71.google.com (mail-qv1-f71.google.com [209.85.219.71]) by relay.mimecast.com with ESMTP with STARTTLS (version=TLSv1.3, cipher=TLS_AES_256_GCM_SHA384) id us-mta-367-xWdZ55pTMqGCZJ_o9LCD6w-1; Thu, 21 Mar 2024 18:08:21 -0400 X-MC-Unique: xWdZ55pTMqGCZJ_o9LCD6w-1 Received: by mail-qv1-f71.google.com with SMTP id 6a1803df08f44-690d3f5af86so3056926d6.0 for ; Thu, 21 Mar 2024 15:08:21 -0700 (PDT) X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20230601; t=1711058901; x=1711663701; h=content-transfer-encoding:mime-version:references:in-reply-to :message-id:date:subject:cc:to:from:x-gm-message-state:from:to:cc :subject:date:message-id:reply-to; bh=ha5DGM0IN3r6kIFCrMZt60pb/ho2ZGKh4AhFb2qqPrM=; b=mQSN1Qk7lMwe6QmUZHqVjqC4rEBNKzvsYJzubtk01D6TFrpgI8VgDo4T+JuIJfmVX+ gtjL1C94C/Js2wTcDFd06Q5w9mqLpmkX0R47sq32wWXmj3p8Pdxc8by7QHzivI9B7khi vLF0m1CQvjL0vlly+6+ilkcHKXiosLztKzl6/elYhGStRyUE/x1VSZM5JtX/HMyZX4nb ZCckOtx3y1x8M1LodGL0DjcZ4mW4O7m/rjzgO4E4kg+yocHKPOhDBYkXTeVduxdoHXQ7 So9AqpdqhOw05jon8a5G9Jk9o4hW5rrQrlVlgLlBRqgMNhAuohY/j/sH45py0VH5scoG JNUg== X-Forwarded-Encrypted: i=1; AJvYcCWvn0ES/Zd3sDWhVS5Ca4DbSvFg6MbuFp4eQzWGyVn212IbgT4242iRSzCL1dPnkD49UHpCWMI8gpwKzk5y/K+/fL8gmU4BqUZkjzynj/c2 X-Gm-Message-State: AOJu0YwwuNAIdA3eHaV+fWoFwziCso8MtiQcjPZmbLVy+lUZxmcTQmQE 2jCRCEOvQYUNa5K4MtHPbcPVWW1pOaL194+5pBjjKNCwCv6DfKgVi6UbLupu1wS9hPEBuDeCMb1 go70tuI0YqapTieN8DQFPWFDDPDp4N7zc3gfhB09far0qgH6rHc9tg2U/gs/cUfGWig== X-Received: by 2002:a05:6214:3f89:b0:690:de72:316f with SMTP id ow9-20020a0562143f8900b00690de72316fmr214818qvb.1.1711058900918; Thu, 21 Mar 2024 15:08:20 -0700 (PDT) X-Google-Smtp-Source: AGHT+IG9Op81dSYI2sDIbFIL6Ox+mlvTFr9echbLfQb+zyBeeZ6FRPjC6M2m9xSIjZkrEy5V/lebFg== X-Received: by 2002:a05:6214:3f89:b0:690:de72:316f with SMTP id ow9-20020a0562143f8900b00690de72316fmr214791qvb.1.1711058900463; Thu, 21 Mar 2024 15:08:20 -0700 (PDT) Received: from x1n.redhat.com ([99.254.121.117]) by smtp.gmail.com with ESMTPSA id o6-20020a0562140e4600b00690baf5cde9sm351663qvc.118.2024.03.21.15.08.18 (version=TLS1_3 cipher=TLS_AES_256_GCM_SHA384 bits=256/256); Thu, 21 Mar 2024 15:08:20 -0700 (PDT) From: peterx@redhat.com To: linux-mm@kvack.org, linux-kernel@vger.kernel.org Cc: linuxppc-dev@lists.ozlabs.org, Michael Ellerman , Christophe Leroy , Matthew Wilcox , Rik van Riel , Lorenzo Stoakes , Axel Rasmussen , peterx@redhat.com, Yang Shi , John Hubbard , linux-arm-kernel@lists.infradead.org, "Kirill A . Shutemov" , Andrew Jones , Vlastimil Babka , Mike Rapoport , Andrew Morton , Muchun Song , Christoph Hellwig , linux-riscv@lists.infradead.org, James Houghton , David Hildenbrand , Jason Gunthorpe , Andrea Arcangeli , "Aneesh Kumar K . V" , Mike Kravetz Subject: [PATCH v3 08/12] mm/gup: Cache *pudp in follow_pud_mask() Date: Thu, 21 Mar 2024 18:07:58 -0400 Message-ID: <20240321220802.679544-9-peterx@redhat.com> X-Mailer: git-send-email 2.44.0 In-Reply-To: <20240321220802.679544-1-peterx@redhat.com> References: <20240321220802.679544-1-peterx@redhat.com> MIME-Version: 1.0 X-Mimecast-Spam-Score: 0 X-Mimecast-Originator: redhat.com X-CRM114-Version: 20100106-BlameMichelson ( TRE 0.8.0 (BSD) ) MR-646709E3 X-CRM114-CacheID: sfid-20240321_220830_573486_B5681F9F X-CRM114-Status: GOOD ( 12.73 ) X-BeenThere: linux-riscv@lists.infradead.org X-Mailman-Version: 2.1.34 Precedence: list List-Id: List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Sender: "linux-riscv" Errors-To: linux-riscv-bounces+linux-riscv=archiver.kernel.org@lists.infradead.org From: Peter Xu Introduce "pud_t pud" in the function, so the code won't dereference *pudp multiple time. Not only because that looks less straightforward, but also because if the dereference really happened, it's not clear whether there can be race to see different *pudp values if it's being modified at the same time. Acked-by: James Houghton Reviewed-by: Jason Gunthorpe Signed-off-by: Peter Xu --- mm/gup.c | 17 +++++++++-------- 1 file changed, 9 insertions(+), 8 deletions(-) diff --git a/mm/gup.c b/mm/gup.c index 943671736080..a338944e4425 100644 --- a/mm/gup.c +++ b/mm/gup.c @@ -753,26 +753,27 @@ static struct page *follow_pud_mask(struct vm_area_struct *vma, unsigned int flags, struct follow_page_context *ctx) { - pud_t *pud; + pud_t *pudp, pud; spinlock_t *ptl; struct page *page; struct mm_struct *mm = vma->vm_mm; - pud = pud_offset(p4dp, address); - if (pud_none(*pud)) + pudp = pud_offset(p4dp, address); + pud = READ_ONCE(*pudp); + if (pud_none(pud)) return no_page_table(vma, flags, address); - if (pud_devmap(*pud)) { - ptl = pud_lock(mm, pud); - page = follow_devmap_pud(vma, address, pud, flags, &ctx->pgmap); + if (pud_devmap(pud)) { + ptl = pud_lock(mm, pudp); + page = follow_devmap_pud(vma, address, pudp, flags, &ctx->pgmap); spin_unlock(ptl); if (page) return page; return no_page_table(vma, flags, address); } - if (unlikely(pud_bad(*pud))) + if (unlikely(pud_bad(pud))) return no_page_table(vma, flags, address); - return follow_pmd_mask(vma, address, pud, flags, ctx); + return follow_pmd_mask(vma, address, pudp, flags, ctx); } static struct page *follow_p4d_mask(struct vm_area_struct *vma, From patchwork Thu Mar 21 22:07:59 2024 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Peter Xu X-Patchwork-Id: 13599441 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from bombadil.infradead.org (bombadil.infradead.org [198.137.202.133]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (No client certificate requested) by smtp.lore.kernel.org (Postfix) with ESMTPS id B283BC6FD1F for ; Thu, 21 Mar 2024 22:10:50 +0000 (UTC) DKIM-Signature: v=1; a=rsa-sha256; q=dns/txt; c=relaxed/relaxed; d=lists.infradead.org; s=bombadil.20210309; h=Sender: Content-Transfer-Encoding:Content-Type:List-Subscribe:List-Help:List-Post: List-Archive:List-Unsubscribe:List-Id:MIME-Version:References:In-Reply-To: Message-ID:Date:Subject:Cc:To:From:Reply-To:Content-ID:Content-Description: Resent-Date:Resent-From:Resent-Sender:Resent-To:Resent-Cc:Resent-Message-ID: List-Owner; bh=+zRtEJgxO8NhcOg0nUQCG2u22McfYixTePQ8lG2IURA=; b=C9klOSg7L4PFmb S6XQwo3UkO6OvmUsLhCHc/kjTUULtygTkw6ZYo06MCY5A3mFlxqWeeBUT2uJPaBYtoIKc2ZMfQDaG 1NrK22UNkxAgx3g7Gq/wIp8+7RgPn78aM4uo0dYbdUZZlEvfPyvk+gtHBb1UsviHH6hvGTsAnTjJi Ro8vs+dGw5mU955cekHt0QDqTy3N7OWD0EtAzbUo1DFD/7OmPBNUNjg5HfE4H2vZK2dK5TWSfTazz j1eik8Sh4rgQU6b4vXgy6RtlDG9wH0TJp+Fh37uCyyZPf1SoxjwUMxFbZPuq4b6DeMeqsVGdfIrnn JwQABeXCD6tsshLlMQ1w==; Received: from localhost ([::1] helo=bombadil.infradead.org) by bombadil.infradead.org with esmtp (Exim 4.97.1 #2 (Red Hat Linux)) id 1rnQcX-00000004s64-0cCC; Thu, 21 Mar 2024 22:10:45 +0000 Received: from desiato.infradead.org ([2001:8b0:10b:1:d65d:64ff:fe57:4e05]) by bombadil.infradead.org with esmtps (Exim 4.97.1 #2 (Red Hat Linux)) id 1rnQaR-00000004qar-1Zr1 for linux-riscv@bombadil.infradead.org; Thu, 21 Mar 2024 22:08:37 +0000 DKIM-Signature: v=1; a=rsa-sha256; q=dns/txt; c=relaxed/relaxed; d=infradead.org; s=desiato.20200630; h=Content-Type:Content-Transfer-Encoding :MIME-Version:References:In-Reply-To:Message-ID:Date:Subject:Cc:To:From: Sender:Reply-To:Content-ID:Content-Description; bh=JwtVKmcef1NV2SEc6Dd7pe+XzH6c3RDiTdVw8lXpTc0=; b=A+cIGCzHtLhFNB06Zk3hjO6Ru4 xC+aJU1/pkzvY8l+B4/1HlF1dByNpMvHU8rBLNvvOk6tWJBDF6KbU4kUj0twTgkT5n2yTViUCOFBD LWrkLRN0GMiA+cossHx6RLBa66ePdFFm4ZddZdzWwhm50118/H/+40CriyNOENPx9/JBns+qke0K6 DQa99AiTb1oGCXWP8B5NHtpu8OJn0FMuc3baWulgI6w8ETN9yyHt0Ejnc61JKKRXdtFPuKYo7L7hh Zw6MZl8VCfjnsRan10UYmKjD27l81pJFNDjvOFVnJHYpxNo69MtfpoZPPEB666GWUbrKWM600oUHK +mQrO3eg==; Received: from us-smtp-delivery-124.mimecast.com ([170.10.129.124]) by desiato.infradead.org with esmtps (Exim 4.97.1 #2 (Red Hat Linux)) id 1rnQaL-0000000ESZe-3NFx for linux-riscv@lists.infradead.org; Thu, 21 Mar 2024 22:08:33 +0000 DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=redhat.com; s=mimecast20190719; t=1711058904; h=from:from:reply-to:subject:subject:date:date:message-id:message-id: to:to:cc:cc:mime-version:mime-version:content-type:content-type: content-transfer-encoding:content-transfer-encoding: in-reply-to:in-reply-to:references:references; bh=JwtVKmcef1NV2SEc6Dd7pe+XzH6c3RDiTdVw8lXpTc0=; b=YulDxqZlYF8zbWj7Gm5KUiqaWX5R3raWvsWRr3q8DryctAdcJmFpon0eVYsia5gxrR1UUN P1aZ/6mdwgqVZ4oinMlmNsuqZbBO7Zo+HMKbtXkOQDlVlWIHXMXW0T4Sw8zpQdcQJ3bV5k 05+o4svLKyVKvyx+BPZyH37C0fVJVkc= DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=redhat.com; s=mimecast20190719; t=1711058905; h=from:from:reply-to:subject:subject:date:date:message-id:message-id: to:to:cc:cc:mime-version:mime-version:content-type:content-type: content-transfer-encoding:content-transfer-encoding: in-reply-to:in-reply-to:references:references; bh=JwtVKmcef1NV2SEc6Dd7pe+XzH6c3RDiTdVw8lXpTc0=; b=hB4+bvBNOg9naYc25t9fXECr0IEmJ1v/tdJ54roV7oq+7hRL2LFS6zyNOG/9q0tfZHUZAJ xxj7d+Hss6T3Jmyncrd3dP/naobb7ELqmMNdxt9+zeANP2Zned/3GRzziONicJ2bhIzsPl sjysMEO4CA6sU2kdOtFPaWjxhInum/w= Received: from mail-qv1-f71.google.com (mail-qv1-f71.google.com [209.85.219.71]) by relay.mimecast.com with ESMTP with STARTTLS (version=TLSv1.3, cipher=TLS_AES_256_GCM_SHA384) id us-mta-511-PLP8cqoKMwy526o-9Np_ng-1; Thu, 21 Mar 2024 18:08:23 -0400 X-MC-Unique: PLP8cqoKMwy526o-9Np_ng-1 Received: by mail-qv1-f71.google.com with SMTP id 6a1803df08f44-690d3f5af86so3057176d6.0 for ; Thu, 21 Mar 2024 15:08:23 -0700 (PDT) X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20230601; t=1711058903; x=1711663703; h=content-transfer-encoding:mime-version:references:in-reply-to :message-id:date:subject:cc:to:from:x-gm-message-state:from:to:cc :subject:date:message-id:reply-to; bh=JwtVKmcef1NV2SEc6Dd7pe+XzH6c3RDiTdVw8lXpTc0=; b=icX+lNbpHAr5B19HHFAdhmb48vKNzxmNNaBBmO6XrqREOeHSDLg0x29uqEdQ4me9Q5 tvEUI/eHG2XfyGYwr4KutT5vvqhvJgL50O35WFeMlT11pSd7FNDp8jjwPnD+wpyRKgps aU79Lv1WJWOad4cheWSmbVYQi48aQ6g3S0l57zJl1dgEna9pBKPnDAMtJsXuVSC14mUk OvhSA+o2N/YZY/V7Yb4l65TsUo7Mb1DiLi30tDyeUMSPak8DsWgoA/BD7aUsWudcOn6F nJZLjW4SkA4B1OqoVqwUsWYsmGyUECc0TvZa/XI6522hwye129UHyXjaDM5WIpEVFbT1 xKGw== X-Forwarded-Encrypted: i=1; AJvYcCXML5qBqOITozxgdIxwSDLlcHu24KjGkGCHWLjU5uIGdlFRcQYtTyM5fi4Fi5hUWe/5FRo+HdnMay1SCubQw5rESXxZSibOaHGE19ue5ObK X-Gm-Message-State: AOJu0Yxfm/bmm8dayZn87/cxYANQ9DDnVwCBb/y2C+W/e7m2fUxWjPdb qWLMLVyNm0zYkhQVaCZvrC3edtvrCYwg4TEPA9UpeqYexzDRzOUcnQGznzINkh2W2ZrE5KWpQi1 GygUhpiRnDWHpfccwchUHi1+oWBYmxX0eRd4q583iobydnfbsOKnpgFhiAWOL+x9hZw== X-Received: by 2002:a05:6214:4403:b0:690:b47e:60e3 with SMTP id oj3-20020a056214440300b00690b47e60e3mr171649qvb.5.1711058902748; Thu, 21 Mar 2024 15:08:22 -0700 (PDT) X-Google-Smtp-Source: AGHT+IFefVkn71VpIW4uNQ3LlUv67q0iftDvBfI3myHkSibZanv1/17GTXunJ1XxyTP0oQ7cl/tpbA== X-Received: by 2002:a05:6214:4403:b0:690:b47e:60e3 with SMTP id oj3-20020a056214440300b00690b47e60e3mr171615qvb.5.1711058902326; Thu, 21 Mar 2024 15:08:22 -0700 (PDT) Received: from x1n.redhat.com ([99.254.121.117]) by smtp.gmail.com with ESMTPSA id o6-20020a0562140e4600b00690baf5cde9sm351663qvc.118.2024.03.21.15.08.20 (version=TLS1_3 cipher=TLS_AES_256_GCM_SHA384 bits=256/256); Thu, 21 Mar 2024 15:08:21 -0700 (PDT) From: peterx@redhat.com To: linux-mm@kvack.org, linux-kernel@vger.kernel.org Cc: linuxppc-dev@lists.ozlabs.org, Michael Ellerman , Christophe Leroy , Matthew Wilcox , Rik van Riel , Lorenzo Stoakes , Axel Rasmussen , peterx@redhat.com, Yang Shi , John Hubbard , linux-arm-kernel@lists.infradead.org, "Kirill A . Shutemov" , Andrew Jones , Vlastimil Babka , Mike Rapoport , Andrew Morton , Muchun Song , Christoph Hellwig , linux-riscv@lists.infradead.org, James Houghton , David Hildenbrand , Jason Gunthorpe , Andrea Arcangeli , "Aneesh Kumar K . V" , Mike Kravetz Subject: [PATCH v3 09/12] mm/gup: Handle huge pud for follow_pud_mask() Date: Thu, 21 Mar 2024 18:07:59 -0400 Message-ID: <20240321220802.679544-10-peterx@redhat.com> X-Mailer: git-send-email 2.44.0 In-Reply-To: <20240321220802.679544-1-peterx@redhat.com> References: <20240321220802.679544-1-peterx@redhat.com> MIME-Version: 1.0 X-Mimecast-Spam-Score: 0 X-Mimecast-Originator: redhat.com X-CRM114-Version: 20100106-BlameMichelson ( TRE 0.8.0 (BSD) ) MR-646709E3 X-CRM114-CacheID: sfid-20240321_220830_568056_B78A187E X-CRM114-Status: GOOD ( 24.14 ) X-BeenThere: linux-riscv@lists.infradead.org X-Mailman-Version: 2.1.34 Precedence: list List-Id: List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Sender: "linux-riscv" Errors-To: linux-riscv-bounces+linux-riscv=archiver.kernel.org@lists.infradead.org From: Peter Xu Teach follow_pud_mask() to be able to handle normal PUD pages like hugetlb. Rename follow_devmap_pud() to follow_huge_pud() so that it can process either huge devmap or hugetlb. Move it out of TRANSPARENT_HUGEPAGE_PUD and and huge_memory.c (which relies on CONFIG_THP). Switch to pud_leaf() to detect both cases in the slow gup. In the new follow_huge_pud(), taking care of possible CoR for hugetlb if necessary. touch_pud() needs to be moved out of huge_memory.c to be accessable from gup.c even if !THP. Since at it, optimize the non-present check by adding a pud_present() early check before taking the pgtable lock, failing the follow_page() early if PUD is not present: that is required by both devmap or hugetlb. Use pud_huge() to also cover the pud_devmap() case. One more trivial thing to mention is, introduce "pud_t pud" in the code paths along the way, so the code doesn't dereference *pudp multiple time. Not only because that looks less straightforward, but also because if the dereference really happened, it's not clear whether there can be race to see different *pudp values when it's being modified at the same time. Setting ctx->page_mask properly for a PUD entry. As a side effect, this patch should also be able to optimize devmap GUP on PUD to be able to jump over the whole PUD range, but not yet verified. Hugetlb already can do so prior to this patch. Reviewed-by: Jason Gunthorpe Signed-off-by: Peter Xu --- include/linux/huge_mm.h | 8 ----- mm/gup.c | 70 +++++++++++++++++++++++++++++++++++++++-- mm/huge_memory.c | 47 ++------------------------- mm/internal.h | 2 ++ 4 files changed, 71 insertions(+), 56 deletions(-) diff --git a/include/linux/huge_mm.h b/include/linux/huge_mm.h index 3bcdfc7e5d57..39195f7c5269 100644 --- a/include/linux/huge_mm.h +++ b/include/linux/huge_mm.h @@ -346,8 +346,6 @@ static inline bool folio_test_pmd_mappable(struct folio *folio) struct page *follow_devmap_pmd(struct vm_area_struct *vma, unsigned long addr, pmd_t *pmd, int flags, struct dev_pagemap **pgmap); -struct page *follow_devmap_pud(struct vm_area_struct *vma, unsigned long addr, - pud_t *pud, int flags, struct dev_pagemap **pgmap); vm_fault_t do_huge_pmd_numa_page(struct vm_fault *vmf); @@ -504,12 +502,6 @@ static inline struct page *follow_devmap_pmd(struct vm_area_struct *vma, return NULL; } -static inline struct page *follow_devmap_pud(struct vm_area_struct *vma, - unsigned long addr, pud_t *pud, int flags, struct dev_pagemap **pgmap) -{ - return NULL; -} - static inline bool thp_migration_supported(void) { return false; diff --git a/mm/gup.c b/mm/gup.c index a338944e4425..ae21afb9434e 100644 --- a/mm/gup.c +++ b/mm/gup.c @@ -525,6 +525,70 @@ static struct page *no_page_table(struct vm_area_struct *vma, return NULL; } +#ifdef CONFIG_PGTABLE_HAS_HUGE_LEAVES +static struct page *follow_huge_pud(struct vm_area_struct *vma, + unsigned long addr, pud_t *pudp, + int flags, struct follow_page_context *ctx) +{ + struct mm_struct *mm = vma->vm_mm; + struct page *page; + pud_t pud = *pudp; + unsigned long pfn = pud_pfn(pud); + int ret; + + assert_spin_locked(pud_lockptr(mm, pudp)); + + if ((flags & FOLL_WRITE) && !pud_write(pud)) + return NULL; + + if (!pud_present(pud)) + return NULL; + + pfn += (addr & ~PUD_MASK) >> PAGE_SHIFT; + + if (IS_ENABLED(CONFIG_HAVE_ARCH_TRANSPARENT_HUGEPAGE_PUD) && + pud_devmap(pud)) { + /* + * device mapped pages can only be returned if the caller + * will manage the page reference count. + * + * At least one of FOLL_GET | FOLL_PIN must be set, so + * assert that here: + */ + if (!(flags & (FOLL_GET | FOLL_PIN))) + return ERR_PTR(-EEXIST); + + if (flags & FOLL_TOUCH) + touch_pud(vma, addr, pudp, flags & FOLL_WRITE); + + ctx->pgmap = get_dev_pagemap(pfn, ctx->pgmap); + if (!ctx->pgmap) + return ERR_PTR(-EFAULT); + } + + page = pfn_to_page(pfn); + + if (!pud_devmap(pud) && !pud_write(pud) && + gup_must_unshare(vma, flags, page)) + return ERR_PTR(-EMLINK); + + ret = try_grab_page(page, flags); + if (ret) + page = ERR_PTR(ret); + else + ctx->page_mask = HPAGE_PUD_NR - 1; + + return page; +} +#else /* CONFIG_PGTABLE_HAS_HUGE_LEAVES */ +static struct page *follow_huge_pud(struct vm_area_struct *vma, + unsigned long addr, pud_t *pudp, + int flags, struct follow_page_context *ctx) +{ + return NULL; +} +#endif /* CONFIG_PGTABLE_HAS_HUGE_LEAVES */ + static int follow_pfn_pte(struct vm_area_struct *vma, unsigned long address, pte_t *pte, unsigned int flags) { @@ -760,11 +824,11 @@ static struct page *follow_pud_mask(struct vm_area_struct *vma, pudp = pud_offset(p4dp, address); pud = READ_ONCE(*pudp); - if (pud_none(pud)) + if (!pud_present(pud)) return no_page_table(vma, flags, address); - if (pud_devmap(pud)) { + if (pud_leaf(pud)) { ptl = pud_lock(mm, pudp); - page = follow_devmap_pud(vma, address, pudp, flags, &ctx->pgmap); + page = follow_huge_pud(vma, address, pudp, flags, ctx); spin_unlock(ptl); if (page) return page; diff --git a/mm/huge_memory.c b/mm/huge_memory.c index c77cedf45f3a..f8bd2012bc27 100644 --- a/mm/huge_memory.c +++ b/mm/huge_memory.c @@ -1363,8 +1363,8 @@ int copy_huge_pmd(struct mm_struct *dst_mm, struct mm_struct *src_mm, } #ifdef CONFIG_HAVE_ARCH_TRANSPARENT_HUGEPAGE_PUD -static void touch_pud(struct vm_area_struct *vma, unsigned long addr, - pud_t *pud, bool write) +void touch_pud(struct vm_area_struct *vma, unsigned long addr, + pud_t *pud, bool write) { pud_t _pud; @@ -1376,49 +1376,6 @@ static void touch_pud(struct vm_area_struct *vma, unsigned long addr, update_mmu_cache_pud(vma, addr, pud); } -struct page *follow_devmap_pud(struct vm_area_struct *vma, unsigned long addr, - pud_t *pud, int flags, struct dev_pagemap **pgmap) -{ - unsigned long pfn = pud_pfn(*pud); - struct mm_struct *mm = vma->vm_mm; - struct page *page; - int ret; - - assert_spin_locked(pud_lockptr(mm, pud)); - - if (flags & FOLL_WRITE && !pud_write(*pud)) - return NULL; - - if (pud_present(*pud) && pud_devmap(*pud)) - /* pass */; - else - return NULL; - - if (flags & FOLL_TOUCH) - touch_pud(vma, addr, pud, flags & FOLL_WRITE); - - /* - * device mapped pages can only be returned if the - * caller will manage the page reference count. - * - * At least one of FOLL_GET | FOLL_PIN must be set, so assert that here: - */ - if (!(flags & (FOLL_GET | FOLL_PIN))) - return ERR_PTR(-EEXIST); - - pfn += (addr & ~PUD_MASK) >> PAGE_SHIFT; - *pgmap = get_dev_pagemap(pfn, *pgmap); - if (!*pgmap) - return ERR_PTR(-EFAULT); - page = pfn_to_page(pfn); - - ret = try_grab_page(page, flags); - if (ret) - page = ERR_PTR(ret); - - return page; -} - int copy_huge_pud(struct mm_struct *dst_mm, struct mm_struct *src_mm, pud_t *dst_pud, pud_t *src_pud, unsigned long addr, struct vm_area_struct *vma) diff --git a/mm/internal.h b/mm/internal.h index f8b31234c130..63e4f6e001be 100644 --- a/mm/internal.h +++ b/mm/internal.h @@ -1102,6 +1102,8 @@ int __must_check try_grab_page(struct page *page, unsigned int flags); /* * mm/huge_memory.c */ +void touch_pud(struct vm_area_struct *vma, unsigned long addr, + pud_t *pud, bool write); struct page *follow_trans_huge_pmd(struct vm_area_struct *vma, unsigned long addr, pmd_t *pmd, unsigned int flags); From patchwork Thu Mar 21 22:08:00 2024 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Peter Xu X-Patchwork-Id: 13599440 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from bombadil.infradead.org (bombadil.infradead.org [198.137.202.133]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (No client certificate requested) by smtp.lore.kernel.org (Postfix) with ESMTPS id 7448BC6FD1F for ; Thu, 21 Mar 2024 22:10:45 +0000 (UTC) DKIM-Signature: v=1; a=rsa-sha256; q=dns/txt; c=relaxed/relaxed; d=lists.infradead.org; s=bombadil.20210309; h=Sender: Content-Transfer-Encoding:Content-Type:List-Subscribe:List-Help:List-Post: List-Archive:List-Unsubscribe:List-Id:MIME-Version:References:In-Reply-To: Message-ID:Date:Subject:Cc:To:From:Reply-To:Content-ID:Content-Description: Resent-Date:Resent-From:Resent-Sender:Resent-To:Resent-Cc:Resent-Message-ID: List-Owner; bh=Jvqw5eJEf2fhJ7f6p4STY/a6I19Wa1MCPjNvXu7y+Dg=; b=bTDGKxcHKHO7GS e1kp4zWRPglkARddgKAvnnZLf66F2y637iDZqUJNqIJ1LGfQY9B2X10b1RKB63KwJNV9w2sVN1Vwe fC5tGmUH05oeflcH9GrWq4E+qhjVydHrJ0Jz1Yc5qE6EdUyWcp9XPLoyG2EpneQpf2FdUSvnotqqO 2SLFOCT26aIyIAc5XjVVfSPBS1wSaLzOR485yI7mYgdyJ+4ZyBIk9AiFxoKzaYefk5yh2+TtddHxO FcxPySmIC14XP2lCVu8y7FeRaTASF5PmtzA6OaM97ovKBdo4hShGoXPfBcsT//ajqNb/VF9oGcFWW cKHr3IYbWKN6ut04eXdg==; Received: from localhost ([::1] helo=bombadil.infradead.org) by bombadil.infradead.org with esmtp (Exim 4.97.1 #2 (Red Hat Linux)) id 1rnQcR-00000004s0k-2owa; Thu, 21 Mar 2024 22:10:40 +0000 Received: from desiato.infradead.org ([2001:8b0:10b:1:d65d:64ff:fe57:4e05]) by bombadil.infradead.org with esmtps (Exim 4.97.1 #2 (Red Hat Linux)) id 1rnQaS-00000004qau-0DCA for linux-riscv@bombadil.infradead.org; Thu, 21 Mar 2024 22:08:37 +0000 DKIM-Signature: v=1; a=rsa-sha256; q=dns/txt; c=relaxed/relaxed; d=infradead.org; s=desiato.20200630; h=Content-Type:Content-Transfer-Encoding :MIME-Version:References:In-Reply-To:Message-ID:Date:Subject:Cc:To:From: Sender:Reply-To:Content-ID:Content-Description; bh=ESTHlGxPxGuUFdX3p2oamk2h4NMABnT27/X9cpNES3o=; b=ipcCfDPUVdIrc1Wgp/ebe1uNMS DI/wnPBbvaDXGjskzN6BTc9ZMZwKd4wWjs1WyjLXRPLZl5Srcvt32D25rwuG5cB0lY0Azthb/XZ8X p79EQfTs2ZB7xBfVRaGr284gh1xK60GK1UtsXEtPEh4qGfLV935QnuezErIkazV3wlgRQiNdFQ55X KSyY3qqopSN8AorsOzBAXrJxuCDIN4Edzb71bG+MXqTOIzB/KnJEfb/Une6IFdL2SV8VBpCw0ivxj eIvGGcxKvj+VuOqFwSMC3KNnRKzF4b8O2VIHPGyr8EUNsbNsYW4EqWdVhe3nMgR2NpzProkAdvdia idtmHxRQ==; Received: from us-smtp-delivery-124.mimecast.com ([170.10.133.124]) by desiato.infradead.org with esmtps (Exim 4.97.1 #2 (Red Hat Linux)) id 1rnQaM-0000000ESbV-224I for linux-riscv@lists.infradead.org; Thu, 21 Mar 2024 22:08:34 +0000 DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=redhat.com; s=mimecast20190719; t=1711058907; h=from:from:reply-to:subject:subject:date:date:message-id:message-id: to:to:cc:cc:mime-version:mime-version:content-type:content-type: content-transfer-encoding:content-transfer-encoding: in-reply-to:in-reply-to:references:references; bh=ESTHlGxPxGuUFdX3p2oamk2h4NMABnT27/X9cpNES3o=; b=GVXnj9Ckm80zVguA9+LlKknu6DyFn+RzheXegfbuvO7XNvPDejVroweD0lryLpGf25BYY2 gsZTZBkwKAZsU0m/xqo2y5FUGFXgRfoYd9e+KI2B9Jx1oIQSw85n42J8q3MxdKMy1GPWbL 9yKLMnHcNzZEe8KCt+ysvdswMJQKFoo= DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=redhat.com; s=mimecast20190719; t=1711058907; h=from:from:reply-to:subject:subject:date:date:message-id:message-id: to:to:cc:cc:mime-version:mime-version:content-type:content-type: content-transfer-encoding:content-transfer-encoding: in-reply-to:in-reply-to:references:references; bh=ESTHlGxPxGuUFdX3p2oamk2h4NMABnT27/X9cpNES3o=; b=GVXnj9Ckm80zVguA9+LlKknu6DyFn+RzheXegfbuvO7XNvPDejVroweD0lryLpGf25BYY2 gsZTZBkwKAZsU0m/xqo2y5FUGFXgRfoYd9e+KI2B9Jx1oIQSw85n42J8q3MxdKMy1GPWbL 9yKLMnHcNzZEe8KCt+ysvdswMJQKFoo= Received: from mail-ot1-f71.google.com (mail-ot1-f71.google.com [209.85.210.71]) by relay.mimecast.com with ESMTP with STARTTLS (version=TLSv1.3, cipher=TLS_AES_256_GCM_SHA384) id us-mta-590-SgWLzvrgOfCDBQ6zRZ6Y0A-1; Thu, 21 Mar 2024 18:08:25 -0400 X-MC-Unique: SgWLzvrgOfCDBQ6zRZ6Y0A-1 Received: by mail-ot1-f71.google.com with SMTP id 46e09a7af769-6e691df3cd6so567538a34.0 for ; Thu, 21 Mar 2024 15:08:25 -0700 (PDT) X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20230601; t=1711058905; x=1711663705; h=content-transfer-encoding:mime-version:references:in-reply-to :message-id:date:subject:cc:to:from:x-gm-message-state:from:to:cc :subject:date:message-id:reply-to; bh=ESTHlGxPxGuUFdX3p2oamk2h4NMABnT27/X9cpNES3o=; b=IhK6x+ZKkadcWgGbARy5qKcTtLdxbejZd2FwExckuA9rxNY0l15EPQWJrkYIvw5Wfw p/FEWAdCnEoTJtsnqL54sdpskjUnubzdPnx+9pyxMBaQpDgFQlOEi16fropHNTk7vMpP hrlVhA7LRgEeN0Du7af30sOqcOydx8jFXVGW3yTXuU8AzsIzowcmR7S3/spnjNCIxPna TgAi4tSVrOb9danLhasJoL1z6AhpiGH2D6/Oq3Wc03HHqD+Zm7a8wt3FUtAK6euE6Uuz LK0lypJEdtXObxHYMP2OAalDTBGbpQcw9/g5Cy9hOfdpF7LSTtsLrWCJie4kgjtgBwzp p1oQ== X-Forwarded-Encrypted: i=1; AJvYcCVgucdzyaqzwVAwRBLBvi6uJ01JerTSiQMcKfzGUHMe6uKZboNgBq/dOKMgLX6GRXhtoEsQtRdKpg15iimvNof38g8Vxz9LWsrsygUxtxAk X-Gm-Message-State: AOJu0YzQ/S+uchKewlgK6Th3JS/cp2wVzftB+t/o/xUY+SusadXKPPy9 C4/JcAaynJOmDNdc/J7Haz9t/zaE/zK8SeIm/Z1tRbDJ021zC7/uGRUJjZ6GtladhM2nVXRtdEC AgpVsBK6gpfXxTExNJGf02vh14/7Cy2heRKDseJiUxD/iGPG174Tgcu7flcrVVJnOpw== X-Received: by 2002:a4a:ca85:0:b0:5a5:1645:90f4 with SMTP id x5-20020a4aca85000000b005a5164590f4mr622772ooq.1.1711058904617; Thu, 21 Mar 2024 15:08:24 -0700 (PDT) X-Google-Smtp-Source: AGHT+IE4H/bBBZLy/REcdrgCXmaDlClCIl1Ey/SRhhRBkoU7cfytCZoQGoym/o3YpdiOnUWtv8D4UQ== X-Received: by 2002:a4a:ca85:0:b0:5a5:1645:90f4 with SMTP id x5-20020a4aca85000000b005a5164590f4mr622720ooq.1.1711058904123; Thu, 21 Mar 2024 15:08:24 -0700 (PDT) Received: from x1n.redhat.com ([99.254.121.117]) by smtp.gmail.com with ESMTPSA id o6-20020a0562140e4600b00690baf5cde9sm351663qvc.118.2024.03.21.15.08.22 (version=TLS1_3 cipher=TLS_AES_256_GCM_SHA384 bits=256/256); Thu, 21 Mar 2024 15:08:23 -0700 (PDT) From: peterx@redhat.com To: linux-mm@kvack.org, linux-kernel@vger.kernel.org Cc: linuxppc-dev@lists.ozlabs.org, Michael Ellerman , Christophe Leroy , Matthew Wilcox , Rik van Riel , Lorenzo Stoakes , Axel Rasmussen , peterx@redhat.com, Yang Shi , John Hubbard , linux-arm-kernel@lists.infradead.org, "Kirill A . Shutemov" , Andrew Jones , Vlastimil Babka , Mike Rapoport , Andrew Morton , Muchun Song , Christoph Hellwig , linux-riscv@lists.infradead.org, James Houghton , David Hildenbrand , Jason Gunthorpe , Andrea Arcangeli , "Aneesh Kumar K . V" , Mike Kravetz Subject: [PATCH v3 10/12] mm/gup: Handle huge pmd for follow_pmd_mask() Date: Thu, 21 Mar 2024 18:08:00 -0400 Message-ID: <20240321220802.679544-11-peterx@redhat.com> X-Mailer: git-send-email 2.44.0 In-Reply-To: <20240321220802.679544-1-peterx@redhat.com> References: <20240321220802.679544-1-peterx@redhat.com> MIME-Version: 1.0 X-Mimecast-Spam-Score: 0 X-Mimecast-Originator: redhat.com X-CRM114-Version: 20100106-BlameMichelson ( TRE 0.8.0 (BSD) ) MR-646709E3 X-CRM114-CacheID: sfid-20240321_220831_199020_82EA459B X-CRM114-Status: GOOD ( 23.58 ) X-BeenThere: linux-riscv@lists.infradead.org X-Mailman-Version: 2.1.34 Precedence: list List-Id: List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Sender: "linux-riscv" Errors-To: linux-riscv-bounces+linux-riscv=archiver.kernel.org@lists.infradead.org From: Peter Xu Replace pmd_trans_huge() with pmd_leaf() to also cover pmd_huge() as long as enabled. FOLL_TOUCH and FOLL_SPLIT_PMD only apply to THP, not yet huge. Since now follow_trans_huge_pmd() can process hugetlb pages, renaming it into follow_huge_pmd() to match what it does. Move it into gup.c so not depend on CONFIG_THP. When at it, move the ctx->page_mask setup into follow_huge_pmd(), only set it when the page is valid. It was not a bug to set it before even if GUP failed (page==NULL), because follow_page_mask() callers always ignores page_mask if so. But doing so makes the code cleaner. Reviewed-by: Jason Gunthorpe Signed-off-by: Peter Xu --- mm/gup.c | 107 ++++++++++++++++++++++++++++++++++++++++++++--- mm/huge_memory.c | 86 +------------------------------------ mm/internal.h | 5 +-- 3 files changed, 105 insertions(+), 93 deletions(-) diff --git a/mm/gup.c b/mm/gup.c index ae21afb9434e..00cdf4cb0cd4 100644 --- a/mm/gup.c +++ b/mm/gup.c @@ -580,6 +580,93 @@ static struct page *follow_huge_pud(struct vm_area_struct *vma, return page; } + +/* FOLL_FORCE can write to even unwritable PMDs in COW mappings. */ +static inline bool can_follow_write_pmd(pmd_t pmd, struct page *page, + struct vm_area_struct *vma, + unsigned int flags) +{ + /* If the pmd is writable, we can write to the page. */ + if (pmd_write(pmd)) + return true; + + /* Maybe FOLL_FORCE is set to override it? */ + if (!(flags & FOLL_FORCE)) + return false; + + /* But FOLL_FORCE has no effect on shared mappings */ + if (vma->vm_flags & (VM_MAYSHARE | VM_SHARED)) + return false; + + /* ... or read-only private ones */ + if (!(vma->vm_flags & VM_MAYWRITE)) + return false; + + /* ... or already writable ones that just need to take a write fault */ + if (vma->vm_flags & VM_WRITE) + return false; + + /* + * See can_change_pte_writable(): we broke COW and could map the page + * writable if we have an exclusive anonymous page ... + */ + if (!page || !PageAnon(page) || !PageAnonExclusive(page)) + return false; + + /* ... and a write-fault isn't required for other reasons. */ + if (vma_soft_dirty_enabled(vma) && !pmd_soft_dirty(pmd)) + return false; + return !userfaultfd_huge_pmd_wp(vma, pmd); +} + +static struct page *follow_huge_pmd(struct vm_area_struct *vma, + unsigned long addr, pmd_t *pmd, + unsigned int flags, + struct follow_page_context *ctx) +{ + struct mm_struct *mm = vma->vm_mm; + pmd_t pmdval = *pmd; + struct page *page; + int ret; + + assert_spin_locked(pmd_lockptr(mm, pmd)); + + page = pmd_page(pmdval); + VM_BUG_ON_PAGE(!PageHead(page) && !is_zone_device_page(page), page); + + if ((flags & FOLL_WRITE) && + !can_follow_write_pmd(pmdval, page, vma, flags)) + return NULL; + + /* Avoid dumping huge zero page */ + if ((flags & FOLL_DUMP) && is_huge_zero_pmd(pmdval)) + return ERR_PTR(-EFAULT); + + if (pmd_protnone(*pmd) && !gup_can_follow_protnone(vma, flags)) + return NULL; + + if (!pmd_write(pmdval) && gup_must_unshare(vma, flags, page)) + return ERR_PTR(-EMLINK); + + VM_BUG_ON_PAGE((flags & FOLL_PIN) && PageAnon(page) && + !PageAnonExclusive(page), page); + + ret = try_grab_page(page, flags); + if (ret) + return ERR_PTR(ret); + +#ifdef CONFIG_TRANSPARENT_HUGEPAGE + if (pmd_trans_huge(pmdval) && (flags & FOLL_TOUCH)) + touch_pmd(vma, addr, pmd, flags & FOLL_WRITE); +#endif /* CONFIG_TRANSPARENT_HUGEPAGE */ + + page += (addr & ~HPAGE_PMD_MASK) >> PAGE_SHIFT; + ctx->page_mask = HPAGE_PMD_NR - 1; + VM_BUG_ON_PAGE(!PageCompound(page) && !is_zone_device_page(page), page); + + return page; +} + #else /* CONFIG_PGTABLE_HAS_HUGE_LEAVES */ static struct page *follow_huge_pud(struct vm_area_struct *vma, unsigned long addr, pud_t *pudp, @@ -587,6 +674,14 @@ static struct page *follow_huge_pud(struct vm_area_struct *vma, { return NULL; } + +static struct page *follow_huge_pmd(struct vm_area_struct *vma, + unsigned long addr, pmd_t *pmd, + unsigned int flags, + struct follow_page_context *ctx) +{ + return NULL; +} #endif /* CONFIG_PGTABLE_HAS_HUGE_LEAVES */ static int follow_pfn_pte(struct vm_area_struct *vma, unsigned long address, @@ -784,31 +879,31 @@ static struct page *follow_pmd_mask(struct vm_area_struct *vma, return page; return no_page_table(vma, flags, address); } - if (likely(!pmd_trans_huge(pmdval))) + if (likely(!pmd_leaf(pmdval))) return follow_page_pte(vma, address, pmd, flags, &ctx->pgmap); if (pmd_protnone(pmdval) && !gup_can_follow_protnone(vma, flags)) return no_page_table(vma, flags, address); ptl = pmd_lock(mm, pmd); - if (unlikely(!pmd_present(*pmd))) { + pmdval = *pmd; + if (unlikely(!pmd_present(pmdval))) { spin_unlock(ptl); return no_page_table(vma, flags, address); } - if (unlikely(!pmd_trans_huge(*pmd))) { + if (unlikely(!pmd_leaf(pmdval))) { spin_unlock(ptl); return follow_page_pte(vma, address, pmd, flags, &ctx->pgmap); } - if (flags & FOLL_SPLIT_PMD) { + if (pmd_trans_huge(pmdval) && (flags & FOLL_SPLIT_PMD)) { spin_unlock(ptl); split_huge_pmd(vma, pmd, address); /* If pmd was left empty, stuff a page table in there quickly */ return pte_alloc(mm, pmd) ? ERR_PTR(-ENOMEM) : follow_page_pte(vma, address, pmd, flags, &ctx->pgmap); } - page = follow_trans_huge_pmd(vma, address, pmd, flags); + page = follow_huge_pmd(vma, address, pmd, flags, ctx); spin_unlock(ptl); - ctx->page_mask = HPAGE_PMD_NR - 1; return page; } diff --git a/mm/huge_memory.c b/mm/huge_memory.c index f8bd2012bc27..e747dacb5051 100644 --- a/mm/huge_memory.c +++ b/mm/huge_memory.c @@ -1206,8 +1206,8 @@ vm_fault_t vmf_insert_pfn_pud(struct vm_fault *vmf, pfn_t pfn, bool write) EXPORT_SYMBOL_GPL(vmf_insert_pfn_pud); #endif /* CONFIG_HAVE_ARCH_TRANSPARENT_HUGEPAGE_PUD */ -static void touch_pmd(struct vm_area_struct *vma, unsigned long addr, - pmd_t *pmd, bool write) +void touch_pmd(struct vm_area_struct *vma, unsigned long addr, + pmd_t *pmd, bool write) { pmd_t _pmd; @@ -1562,88 +1562,6 @@ static inline bool can_change_pmd_writable(struct vm_area_struct *vma, return pmd_dirty(pmd); } -/* FOLL_FORCE can write to even unwritable PMDs in COW mappings. */ -static inline bool can_follow_write_pmd(pmd_t pmd, struct page *page, - struct vm_area_struct *vma, - unsigned int flags) -{ - /* If the pmd is writable, we can write to the page. */ - if (pmd_write(pmd)) - return true; - - /* Maybe FOLL_FORCE is set to override it? */ - if (!(flags & FOLL_FORCE)) - return false; - - /* But FOLL_FORCE has no effect on shared mappings */ - if (vma->vm_flags & (VM_MAYSHARE | VM_SHARED)) - return false; - - /* ... or read-only private ones */ - if (!(vma->vm_flags & VM_MAYWRITE)) - return false; - - /* ... or already writable ones that just need to take a write fault */ - if (vma->vm_flags & VM_WRITE) - return false; - - /* - * See can_change_pte_writable(): we broke COW and could map the page - * writable if we have an exclusive anonymous page ... - */ - if (!page || !PageAnon(page) || !PageAnonExclusive(page)) - return false; - - /* ... and a write-fault isn't required for other reasons. */ - if (vma_soft_dirty_enabled(vma) && !pmd_soft_dirty(pmd)) - return false; - return !userfaultfd_huge_pmd_wp(vma, pmd); -} - -struct page *follow_trans_huge_pmd(struct vm_area_struct *vma, - unsigned long addr, - pmd_t *pmd, - unsigned int flags) -{ - struct mm_struct *mm = vma->vm_mm; - struct page *page; - int ret; - - assert_spin_locked(pmd_lockptr(mm, pmd)); - - page = pmd_page(*pmd); - VM_BUG_ON_PAGE(!PageHead(page) && !is_zone_device_page(page), page); - - if ((flags & FOLL_WRITE) && - !can_follow_write_pmd(*pmd, page, vma, flags)) - return NULL; - - /* Avoid dumping huge zero page */ - if ((flags & FOLL_DUMP) && is_huge_zero_pmd(*pmd)) - return ERR_PTR(-EFAULT); - - if (pmd_protnone(*pmd) && !gup_can_follow_protnone(vma, flags)) - return NULL; - - if (!pmd_write(*pmd) && gup_must_unshare(vma, flags, page)) - return ERR_PTR(-EMLINK); - - VM_BUG_ON_PAGE((flags & FOLL_PIN) && PageAnon(page) && - !PageAnonExclusive(page), page); - - ret = try_grab_page(page, flags); - if (ret) - return ERR_PTR(ret); - - if (flags & FOLL_TOUCH) - touch_pmd(vma, addr, pmd, flags & FOLL_WRITE); - - page += (addr & ~HPAGE_PMD_MASK) >> PAGE_SHIFT; - VM_BUG_ON_PAGE(!PageCompound(page) && !is_zone_device_page(page), page); - - return page; -} - /* NUMA hinting page fault entry point for trans huge pmds */ vm_fault_t do_huge_pmd_numa_page(struct vm_fault *vmf) { diff --git a/mm/internal.h b/mm/internal.h index 63e4f6e001be..d47862e6d968 100644 --- a/mm/internal.h +++ b/mm/internal.h @@ -1104,9 +1104,8 @@ int __must_check try_grab_page(struct page *page, unsigned int flags); */ void touch_pud(struct vm_area_struct *vma, unsigned long addr, pud_t *pud, bool write); -struct page *follow_trans_huge_pmd(struct vm_area_struct *vma, - unsigned long addr, pmd_t *pmd, - unsigned int flags); +void touch_pmd(struct vm_area_struct *vma, unsigned long addr, + pmd_t *pmd, bool write); #ifdef CONFIG_MEMCG static inline From patchwork Thu Mar 21 22:08:01 2024 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Peter Xu X-Patchwork-Id: 13599442 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from bombadil.infradead.org (bombadil.infradead.org [198.137.202.133]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (No client certificate requested) by smtp.lore.kernel.org (Postfix) with ESMTPS id 0D8BEC6FD1F for ; Thu, 21 Mar 2024 22:10:56 +0000 (UTC) DKIM-Signature: v=1; a=rsa-sha256; q=dns/txt; c=relaxed/relaxed; d=lists.infradead.org; s=bombadil.20210309; h=Sender: Content-Transfer-Encoding:Content-Type:List-Subscribe:List-Help:List-Post: List-Archive:List-Unsubscribe:List-Id:MIME-Version:References:In-Reply-To: Message-ID:Date:Subject:Cc:To:From:Reply-To:Content-ID:Content-Description: Resent-Date:Resent-From:Resent-Sender:Resent-To:Resent-Cc:Resent-Message-ID: List-Owner; bh=kKCot220rU0GpIzyX181ihwdu/B8KRmTY6R1sqObL4I=; b=IYeDy12ZRQf7rq nC+hn99oMhmsIOe3yfBCFP+c5fZClLoLvh+O/7+g2xylQk9eQQYTjP/dnec+manBIRihqtlHcvhn/ UWVNoAqHyhlXzdPNsc/axu0eN8uamAILjKhcsPFy2ER/Asj3YWyJswUmV5ULpdt8xmMsMLXDhRC6g 4S2tlSEC7vZeZ9+3AQH//todNOh1MX5A3pXCsZxD7zkg7kjznzYYEyCh5ALA7Pv5EXgrNKqrL/F6f F+TXUq18H1NllLtGfEYV3M9CaGj/7mge9zWoqw7goQD8MI/A0N6mN0GCulVsKJZWyLwzH7h9tHi9h LRaC1mktI5QEA9ItgOQw==; Received: from localhost ([::1] helo=bombadil.infradead.org) by bombadil.infradead.org with esmtp (Exim 4.97.1 #2 (Red Hat Linux)) id 1rnQcc-00000004sBZ-40VD; Thu, 21 Mar 2024 22:10:50 +0000 Received: from desiato.infradead.org ([2001:8b0:10b:1:d65d:64ff:fe57:4e05]) by bombadil.infradead.org with esmtps (Exim 4.97.1 #2 (Red Hat Linux)) id 1rnQaV-00000004qdO-0Cfg for linux-riscv@bombadil.infradead.org; Thu, 21 Mar 2024 22:08:39 +0000 DKIM-Signature: v=1; a=rsa-sha256; q=dns/txt; c=relaxed/relaxed; d=infradead.org; s=desiato.20200630; h=Content-Type:Content-Transfer-Encoding :MIME-Version:References:In-Reply-To:Message-ID:Date:Subject:Cc:To:From: Sender:Reply-To:Content-ID:Content-Description; bh=o0KbP6b9RlV02LlNjzHhHyMPZ1vhd/KMB6w1UdD4tH4=; b=P070YU9qM/4RTORQpZoLH/LGmt nAFGGykgbk3nnytvo/tb9wjjjCVIezQNarGoqxZ5nrUOuvVVe5ANZ2hkeGCGrgGmVHLYgC8IUGsjt XHB54iQe3Smg9kCxtjg9JCwpK7voie3IzHOeDr1b0A2iAMOx46vYYKWUlFIFjmEeuO96rU5pIzwH0 +a/V4ZYtfnwsGBOOljcvANV9Ojg6lWWFZycRj9MKSAEc1W99xoDvtynwoCC2Cyyim/D0nfGZjm3Bk wkXGCyvjXU2swDib+2ab4m/i2d6d5ApSRDha4VBJRXOIyGugkJLzidVd3y6dI5aSlJ1ESixQD/mNc y2CxYkzw==; Received: from us-smtp-delivery-124.mimecast.com ([170.10.133.124]) by desiato.infradead.org with esmtps (Exim 4.97.1 #2 (Red Hat Linux)) id 1rnQaN-0000000EScX-2ZRU for linux-riscv@lists.infradead.org; Thu, 21 Mar 2024 22:08:37 +0000 DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=redhat.com; s=mimecast20190719; t=1711058908; h=from:from:reply-to:subject:subject:date:date:message-id:message-id: to:to:cc:cc:mime-version:mime-version:content-type:content-type: content-transfer-encoding:content-transfer-encoding: in-reply-to:in-reply-to:references:references; bh=o0KbP6b9RlV02LlNjzHhHyMPZ1vhd/KMB6w1UdD4tH4=; b=WJk1gJkIocWn8iGjJhbZctKaNdmfMdtg1DczD3yCneMzGNUG4NuKff+JG17EN+n/Ypkarh Q/FQeUTM7y5nNOtLVBKDCt6krlSwk8OlI3J8RepdkKkpOnMFSdBlp4pBD9DLTUoSq6nrqJ n1FJYgvgdZJChBkVmXXFjixaREk1+bw= DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=redhat.com; s=mimecast20190719; t=1711058909; h=from:from:reply-to:subject:subject:date:date:message-id:message-id: to:to:cc:cc:mime-version:mime-version:content-type:content-type: content-transfer-encoding:content-transfer-encoding: in-reply-to:in-reply-to:references:references; bh=o0KbP6b9RlV02LlNjzHhHyMPZ1vhd/KMB6w1UdD4tH4=; b=gfW2DZiAYEDL1n0+OxC9iZ/ZLiiZT41o+MizICXgaq7HLOVhTcJYayO4UxEqDdE6p+OoNh qpsQHk10vh/Uz4i7A3wU4hmF7Nrqp2rAcd62omd3+6STDr/3i8AMNNLKggrRzyq8zCeWdC xYu/Ldf8XeVEyD9KPMGoKRtWxOITPKY= Received: from mail-oa1-f70.google.com (mail-oa1-f70.google.com [209.85.160.70]) by relay.mimecast.com with ESMTP with STARTTLS (version=TLSv1.3, cipher=TLS_AES_256_GCM_SHA384) id us-mta-316-o9y883vAORC3awLgkVbcxw-1; Thu, 21 Mar 2024 18:08:27 -0400 X-MC-Unique: o9y883vAORC3awLgkVbcxw-1 Received: by mail-oa1-f70.google.com with SMTP id 586e51a60fabf-222b55202f4so525968fac.1 for ; Thu, 21 Mar 2024 15:08:27 -0700 (PDT) X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20230601; t=1711058907; x=1711663707; h=content-transfer-encoding:mime-version:references:in-reply-to :message-id:date:subject:cc:to:from:x-gm-message-state:from:to:cc :subject:date:message-id:reply-to; bh=o0KbP6b9RlV02LlNjzHhHyMPZ1vhd/KMB6w1UdD4tH4=; b=xHSPL42kd2AlnjlWQAS9KJoEMWVhM/2yM/heqYWlfSc5xGjBpOaSI6G7gMcfJRPTk6 IS7EgZ/hPafwK/fJmCshydP7TqaRNu4wSmM/cV2PJd/ND/4cht78qDtXbg1wy82TPFVs fpkBV4tKVvdL8MdCSm3CagTXhdSyUW76ES49zAbhFlEjcYlLeo0h1IF6Nx2AT5hn98ZO ESWLCjnr/MH5qeRyC6hObrL01MZ2odjdOcXghhhBwkrtHoXwayggsigK9vcp5SzED9M+ cSkTWk2A6oDmlqG4f90Q67PBOUt8aJwXvWv6/Qnc0+zK0k+Kp83IYTY0C7NWQw/6md5m wPwQ== X-Forwarded-Encrypted: i=1; AJvYcCWiGX4YM9ohd0kgS9uVJbAOCuXbPHG9XtAPYG6zF7dYA5tZaxPHTaUfKQJMrp9zxTMGZXVibj+vecsEMS91l3ROtcmKJGln+UrO6iwPf09t X-Gm-Message-State: AOJu0YzBSSFOkwZLpOLafS8fr149sH4uCkb/+n4IjSYQJ0IcjaJyr1CP gAmVp1OmqjYnwTQfn6iKhmgm9jlfWYaBjTfnQaegCE5GEy83SMgRjwCmzMAG7nZOmL8+yylpmxf 2qF4pAmzkcrcRKLFuR8fdMTq7K2wBPApTQxlh3+yFaBaGpCrpLE4+zogGMYUJqGDJsw== X-Received: by 2002:a05:6870:4189:b0:229:8575:875b with SMTP id y9-20020a056870418900b002298575875bmr520315oac.5.1711058906675; Thu, 21 Mar 2024 15:08:26 -0700 (PDT) X-Google-Smtp-Source: AGHT+IHyRVGIH/+XEvquuQvEBSJicroe5v4UAFWGp7nk+sM4AQMJKL66g328LmVlireG12paxfzEyQ== X-Received: by 2002:a05:6870:4189:b0:229:8575:875b with SMTP id y9-20020a056870418900b002298575875bmr520279oac.5.1711058906111; Thu, 21 Mar 2024 15:08:26 -0700 (PDT) Received: from x1n.redhat.com ([99.254.121.117]) by smtp.gmail.com with ESMTPSA id o6-20020a0562140e4600b00690baf5cde9sm351663qvc.118.2024.03.21.15.08.24 (version=TLS1_3 cipher=TLS_AES_256_GCM_SHA384 bits=256/256); Thu, 21 Mar 2024 15:08:25 -0700 (PDT) From: peterx@redhat.com To: linux-mm@kvack.org, linux-kernel@vger.kernel.org Cc: linuxppc-dev@lists.ozlabs.org, Michael Ellerman , Christophe Leroy , Matthew Wilcox , Rik van Riel , Lorenzo Stoakes , Axel Rasmussen , peterx@redhat.com, Yang Shi , John Hubbard , linux-arm-kernel@lists.infradead.org, "Kirill A . Shutemov" , Andrew Jones , Vlastimil Babka , Mike Rapoport , Andrew Morton , Muchun Song , Christoph Hellwig , linux-riscv@lists.infradead.org, James Houghton , David Hildenbrand , Jason Gunthorpe , Andrea Arcangeli , "Aneesh Kumar K . V" , Mike Kravetz Subject: [PATCH v3 11/12] mm/gup: Handle hugepd for follow_page() Date: Thu, 21 Mar 2024 18:08:01 -0400 Message-ID: <20240321220802.679544-12-peterx@redhat.com> X-Mailer: git-send-email 2.44.0 In-Reply-To: <20240321220802.679544-1-peterx@redhat.com> References: <20240321220802.679544-1-peterx@redhat.com> MIME-Version: 1.0 X-Mimecast-Spam-Score: 0 X-Mimecast-Originator: redhat.com X-CRM114-Version: 20100106-BlameMichelson ( TRE 0.8.0 (BSD) ) MR-646709E3 X-CRM114-CacheID: sfid-20240321_220832_149179_9B63D685 X-CRM114-Status: GOOD ( 21.31 ) X-BeenThere: linux-riscv@lists.infradead.org X-Mailman-Version: 2.1.34 Precedence: list List-Id: List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Sender: "linux-riscv" Errors-To: linux-riscv-bounces+linux-riscv=archiver.kernel.org@lists.infradead.org From: Peter Xu Hugepd is only used in PowerPC so far on 4K page size kernels where hash mmu is used. follow_page_mask() used to leverage hugetlb APIs to access hugepd entries. Teach follow_page_mask() itself on hugepd. With previous refactors on fast-gup gup_huge_pd(), most of the code can be easily leveraged. There's something not needed for follow page, for example, gup_hugepte() tries to detect pgtable entry change which will never happen with slow gup (which has the pgtable lock held), but that's not a problem to check. Since follow_page() always only fetch one page, set the end to "address + PAGE_SIZE" should suffice. We will still do the pgtable walk once for each hugetlb page by setting ctx->page_mask properly. One thing worth mentioning is that some level of pgtable's _bad() helper will report is_hugepd() entries as TRUE on Power8 hash MMUs. I think it at least applies to PUD on Power8 with 4K pgsize. It means feeding a hugepd entry to pud_bad() will report a false positive. Let's leave that for now because it can be arch-specific where I am a bit declined to touch. In this patch it's not a problem as long as hugepd is detected before any bad pgtable entries. Signed-off-by: Peter Xu --- mm/gup.c | 73 ++++++++++++++++++++++++++++++++++++++++++++++++++------ 1 file changed, 66 insertions(+), 7 deletions(-) diff --git a/mm/gup.c b/mm/gup.c index 00cdf4cb0cd4..43a2e0a203cd 100644 --- a/mm/gup.c +++ b/mm/gup.c @@ -30,6 +30,11 @@ struct follow_page_context { unsigned int page_mask; }; +static struct page *follow_hugepd(struct vm_area_struct *vma, hugepd_t hugepd, + unsigned long addr, unsigned int pdshift, + unsigned int flags, + struct follow_page_context *ctx); + static inline void sanity_check_pinned_pages(struct page **pages, unsigned long npages) { @@ -871,6 +876,9 @@ static struct page *follow_pmd_mask(struct vm_area_struct *vma, return no_page_table(vma, flags, address); if (!pmd_present(pmdval)) return no_page_table(vma, flags, address); + if (unlikely(is_hugepd(__hugepd(pmd_val(pmdval))))) + return follow_hugepd(vma, __hugepd(pmd_val(pmdval)), + address, PMD_SHIFT, flags, ctx); if (pmd_devmap(pmdval)) { ptl = pmd_lock(mm, pmd); page = follow_devmap_pmd(vma, address, pmd, flags, &ctx->pgmap); @@ -921,6 +929,9 @@ static struct page *follow_pud_mask(struct vm_area_struct *vma, pud = READ_ONCE(*pudp); if (!pud_present(pud)) return no_page_table(vma, flags, address); + if (unlikely(is_hugepd(__hugepd(pud_val(pud))))) + return follow_hugepd(vma, __hugepd(pud_val(pud)), + address, PUD_SHIFT, flags, ctx); if (pud_leaf(pud)) { ptl = pud_lock(mm, pudp); page = follow_huge_pud(vma, address, pudp, flags, ctx); @@ -944,10 +955,13 @@ static struct page *follow_p4d_mask(struct vm_area_struct *vma, p4dp = p4d_offset(pgdp, address); p4d = READ_ONCE(*p4dp); - if (!p4d_present(p4d)) - return no_page_table(vma, flags, address); BUILD_BUG_ON(p4d_leaf(p4d)); - if (unlikely(p4d_bad(p4d))) + + if (unlikely(is_hugepd(__hugepd(p4d_val(p4d))))) + return follow_hugepd(vma, __hugepd(p4d_val(p4d)), + address, P4D_SHIFT, flags, ctx); + + if (!p4d_present(p4d) || p4d_bad(p4d)) return no_page_table(vma, flags, address); return follow_pud_mask(vma, address, p4dp, flags, ctx); @@ -981,7 +995,7 @@ static struct page *follow_page_mask(struct vm_area_struct *vma, unsigned long address, unsigned int flags, struct follow_page_context *ctx) { - pgd_t *pgd; + pgd_t *pgd, pgdval; struct mm_struct *mm = vma->vm_mm; ctx->page_mask = 0; @@ -996,11 +1010,17 @@ static struct page *follow_page_mask(struct vm_area_struct *vma, &ctx->page_mask); pgd = pgd_offset(mm, address); + pgdval = *pgd; - if (pgd_none(*pgd) || unlikely(pgd_bad(*pgd))) - return no_page_table(vma, flags, address); + if (unlikely(is_hugepd(__hugepd(pgd_val(pgdval))))) + page = follow_hugepd(vma, __hugepd(pgd_val(pgdval)), + address, PGDIR_SHIFT, flags, ctx); + else if (pgd_none(*pgd) || unlikely(pgd_bad(*pgd))) + page = no_page_table(vma, flags, address); + else + page = follow_p4d_mask(vma, address, pgd, flags, ctx); - return follow_p4d_mask(vma, address, pgd, flags, ctx); + return page; } struct page *follow_page(struct vm_area_struct *vma, unsigned long address, @@ -3037,6 +3057,37 @@ static int gup_huge_pd(hugepd_t hugepd, unsigned long addr, return 1; } + +static struct page *follow_hugepd(struct vm_area_struct *vma, hugepd_t hugepd, + unsigned long addr, unsigned int pdshift, + unsigned int flags, + struct follow_page_context *ctx) +{ + struct page *page; + struct hstate *h; + spinlock_t *ptl; + int nr = 0, ret; + pte_t *ptep; + + /* Only hugetlb supports hugepd */ + if (WARN_ON_ONCE(!is_vm_hugetlb_page(vma))) + return ERR_PTR(-EFAULT); + + h = hstate_vma(vma); + ptep = hugepte_offset(hugepd, addr, pdshift); + ptl = huge_pte_lock(h, vma->vm_mm, ptep); + ret = gup_huge_pd(hugepd, addr, pdshift, addr + PAGE_SIZE, + flags, &page, &nr); + spin_unlock(ptl); + + if (ret) { + WARN_ON_ONCE(nr != 1); + ctx->page_mask = (1U << huge_page_order(h)) - 1; + return page; + } + + return NULL; +} #else static inline int gup_huge_pd(hugepd_t hugepd, unsigned long addr, unsigned int pdshift, unsigned long end, unsigned int flags, @@ -3044,6 +3095,14 @@ static inline int gup_huge_pd(hugepd_t hugepd, unsigned long addr, { return 0; } + +static struct page *follow_hugepd(struct vm_area_struct *vma, hugepd_t hugepd, + unsigned long addr, unsigned int pdshift, + unsigned int flags, + struct follow_page_context *ctx) +{ + return NULL; +} #endif /* CONFIG_ARCH_HAS_HUGEPD */ static int gup_huge_pmd(pmd_t orig, pmd_t *pmdp, unsigned long addr, From patchwork Thu Mar 21 22:08:02 2024 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Peter Xu X-Patchwork-Id: 13599443 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from bombadil.infradead.org (bombadil.infradead.org [198.137.202.133]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (No client certificate requested) by smtp.lore.kernel.org (Postfix) with ESMTPS id AF83EC6FD1F for ; Thu, 21 Mar 2024 22:10:59 +0000 (UTC) DKIM-Signature: v=1; a=rsa-sha256; q=dns/txt; c=relaxed/relaxed; d=lists.infradead.org; s=bombadil.20210309; h=Sender: Content-Transfer-Encoding:Content-Type:List-Subscribe:List-Help:List-Post: List-Archive:List-Unsubscribe:List-Id:MIME-Version:References:In-Reply-To: Message-ID:Date:Subject:Cc:To:From:Reply-To:Content-ID:Content-Description: Resent-Date:Resent-From:Resent-Sender:Resent-To:Resent-Cc:Resent-Message-ID: List-Owner; bh=0r/O7papj+oui8wyVXtigDiLfbSDFjB/plndC0hIfmI=; b=zKf2fzDry86+S8 q6cGShZrCpZVnJ3hEIHZR6RlzlTM9aYlpshoP1lDPAIwiFxtyRD7g8y6IuQvc+mXtFo4bcjA0UY/6 TTN8S9pwIEXSg+9Y0I/YA96GnnpukHvcc9AuAcTEg9/m6e5jB6h+IG9dNiiiFdiGoeX3wi8RF5X7A EURs37HGBNd75l1e4MxyR6uswsYrPbBGhw2wFmZC92QX1JTajH+FU1fidpLHWeyurCc1wmuAjyMvm WHsGnePTuTnqDZSkIFc2RKCwDnbkuWan6bpjBUR1BRerMIlXAyQrG0xhWJCJBQuSSrSYnp7rgGft7 qksi2VKVMB6Tvhzn+D6A==; Received: from localhost ([::1] helo=bombadil.infradead.org) by bombadil.infradead.org with esmtp (Exim 4.97.1 #2 (Red Hat Linux)) id 1rnQcg-00000004sFP-4487; Thu, 21 Mar 2024 22:10:54 +0000 Received: from us-smtp-delivery-124.mimecast.com ([170.10.133.124]) by bombadil.infradead.org with esmtps (Exim 4.97.1 #2 (Red Hat Linux)) id 1rnQaU-00000004qac-0Dqt for linux-riscv@lists.infradead.org; Thu, 21 Mar 2024 22:08:43 +0000 DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=redhat.com; s=mimecast20190719; t=1711058912; h=from:from:reply-to:subject:subject:date:date:message-id:message-id: to:to:cc:cc:mime-version:mime-version:content-type:content-type: content-transfer-encoding:content-transfer-encoding: in-reply-to:in-reply-to:references:references; bh=kKAkS039JEm9eSlNjnxGInSB8mOTV3IHPzDUFjGHTVk=; b=fKM4VqYAhMay1/b4NGyU6dSXL/rrOm2sU1lH2xiw8x5BjFUKE7kpClPgNf8K1RMEsgeP95 0x0lmTu+mQNBLKRflDVJnz9GjVxn5yCzjBcsFr1WuwGCJQyxqJEMIlehapF07wKLdKIUAR Tm8lU6Nyz+Q5OSk+4S4GH2BS38x71Oo= Received: from mail-ot1-f71.google.com (mail-ot1-f71.google.com [209.85.210.71]) by relay.mimecast.com with ESMTP with STARTTLS (version=TLSv1.3, cipher=TLS_AES_256_GCM_SHA384) id us-mta-204-uUz_8c9zNW2IZi-rs-s0Tw-1; Thu, 21 Mar 2024 18:08:30 -0400 X-MC-Unique: uUz_8c9zNW2IZi-rs-s0Tw-1 Received: by mail-ot1-f71.google.com with SMTP id 46e09a7af769-6e691df3cd6so567557a34.0 for ; Thu, 21 Mar 2024 15:08:30 -0700 (PDT) X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20230601; t=1711058909; x=1711663709; h=content-transfer-encoding:mime-version:references:in-reply-to :message-id:date:subject:cc:to:from:x-gm-message-state:from:to:cc :subject:date:message-id:reply-to; bh=kKAkS039JEm9eSlNjnxGInSB8mOTV3IHPzDUFjGHTVk=; b=OBJWYkbNsTxk4oM4smaX9heOcj7hhxZJypJbDO9mty5RbRf78H81NsaUf3Fa9UXc4E eJ920eBKMiTVTwxhz52IE6rbe9CH4Pf6B611yK1vSTWUbG2AwdEMLMoKnUUiZJ7+QsAT zIjkc5aWLmiGurLwmvA0xlNyDzWVGfc0JzfQqXEVKca5nksUMgl1F4wjEaiCp2YxI7Q3 g+y2UoBf/+bjppDzpx7S643Hz/ISl3vPefdqh5lIt3PjXpbsepg066TgwcN7YhSFWEs6 UmV+9VpJJgY1wa7iZyqOn/SfGGg4K4YEPVjJkg0/37ju9lN6w9ULzimjfnbbHbDzSjxR wIfw== X-Forwarded-Encrypted: i=1; AJvYcCWJa6SuuyDINyBSCTNqCsJt+T/wqKbqoEULHHLZe7ovrR8Owa14QdD+7vToQVW1Qbsj93SLzrEbJHaocGjqxmQV3Fb7pLoJE2wOL1BqK5FZ X-Gm-Message-State: AOJu0YyKLr/TaWOOP1u5nA0ZML7k1JyoFlmOl/fpuN9MjlRd6K8IgOEj BrluBut+8bZxn/IlKsn2HmTOFVNfwtMSlBt5PnE2UQs3YIxHFy83Um3bwosnW7eoiZCEG5jJQE6 zHJZwbc5rE9qs8MoWPmbnSqZTj0mbcpfDtd6+133LyNRbQIAvOlBPicBmx3TC1o/ieQ== X-Received: by 2002:a4a:d5ce:0:b0:5a2:26c7:397b with SMTP id a14-20020a4ad5ce000000b005a226c7397bmr912009oot.0.1711058909577; Thu, 21 Mar 2024 15:08:29 -0700 (PDT) X-Google-Smtp-Source: AGHT+IH7bJa0sTEC4z3o8Ei9oO8UNdi6+GHr5ekBXamqtFS26QUeqg4nthASLLtCNNWPAVCCH5AJXA== X-Received: by 2002:a4a:d5ce:0:b0:5a2:26c7:397b with SMTP id a14-20020a4ad5ce000000b005a226c7397bmr911987oot.0.1711058909121; Thu, 21 Mar 2024 15:08:29 -0700 (PDT) Received: from x1n.redhat.com ([99.254.121.117]) by smtp.gmail.com with ESMTPSA id o6-20020a0562140e4600b00690baf5cde9sm351663qvc.118.2024.03.21.15.08.26 (version=TLS1_3 cipher=TLS_AES_256_GCM_SHA384 bits=256/256); Thu, 21 Mar 2024 15:08:28 -0700 (PDT) From: peterx@redhat.com To: linux-mm@kvack.org, linux-kernel@vger.kernel.org Cc: linuxppc-dev@lists.ozlabs.org, Michael Ellerman , Christophe Leroy , Matthew Wilcox , Rik van Riel , Lorenzo Stoakes , Axel Rasmussen , peterx@redhat.com, Yang Shi , John Hubbard , linux-arm-kernel@lists.infradead.org, "Kirill A . Shutemov" , Andrew Jones , Vlastimil Babka , Mike Rapoport , Andrew Morton , Muchun Song , Christoph Hellwig , linux-riscv@lists.infradead.org, James Houghton , David Hildenbrand , Jason Gunthorpe , Andrea Arcangeli , "Aneesh Kumar K . V" , Mike Kravetz Subject: [PATCH v3 12/12] mm/gup: Handle hugetlb in the generic follow_page_mask code Date: Thu, 21 Mar 2024 18:08:02 -0400 Message-ID: <20240321220802.679544-13-peterx@redhat.com> X-Mailer: git-send-email 2.44.0 In-Reply-To: <20240321220802.679544-1-peterx@redhat.com> References: <20240321220802.679544-1-peterx@redhat.com> MIME-Version: 1.0 X-Mimecast-Spam-Score: 0 X-Mimecast-Originator: redhat.com X-CRM114-Version: 20100106-BlameMichelson ( TRE 0.8.0 (BSD) ) MR-646709E3 X-CRM114-CacheID: sfid-20240321_150838_915304_16480EE5 X-CRM114-Status: GOOD ( 23.06 ) X-BeenThere: linux-riscv@lists.infradead.org X-Mailman-Version: 2.1.34 Precedence: list List-Id: List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Sender: "linux-riscv" Errors-To: linux-riscv-bounces+linux-riscv=archiver.kernel.org@lists.infradead.org From: Peter Xu Now follow_page() is ready to handle hugetlb pages in whatever form, and over all architectures. Switch to the generic code path. Time to retire hugetlb_follow_page_mask(), following the previous retirement of follow_hugetlb_page() in 4849807114b8. There may be a slight difference of how the loops run when processing slow GUP over a large hugetlb range on cont_pte/cont_pmd supported archs: each loop of __get_user_pages() will resolve one pgtable entry with the patch applied, rather than relying on the size of hugetlb hstate, the latter may cover multiple entries in one loop. A quick performance test on an aarch64 VM on M1 chip shows 15% degrade over a tight loop of slow gup after the path switched. That shouldn't be a problem because slow-gup should not be a hot path for GUP in general: when page is commonly present, fast-gup will already succeed, while when the page is indeed missing and require a follow up page fault, the slow gup degrade will probably buried in the fault paths anyway. It also explains why slow gup for THP used to be very slow before 57edfcfd3419 ("mm/gup: accelerate thp gup even for "pages != NULL"") lands, the latter not part of a performance analysis but a side benefit. If the performance will be a concern, we can consider handle CONT_PTE in follow_page(). Before that is justified to be necessary, keep everything clean and simple. Signed-off-by: Peter Xu Reviewed-by: Jason Gunthorpe --- include/linux/hugetlb.h | 7 ---- mm/gup.c | 15 +++------ mm/hugetlb.c | 71 ----------------------------------------- 3 files changed, 5 insertions(+), 88 deletions(-) diff --git a/include/linux/hugetlb.h b/include/linux/hugetlb.h index 52d9efcf1edf..85e1c9931ae5 100644 --- a/include/linux/hugetlb.h +++ b/include/linux/hugetlb.h @@ -328,13 +328,6 @@ static inline void hugetlb_zap_end( { } -static inline struct page *hugetlb_follow_page_mask( - struct vm_area_struct *vma, unsigned long address, unsigned int flags, - unsigned int *page_mask) -{ - BUILD_BUG(); /* should never be compiled in if !CONFIG_HUGETLB_PAGE*/ -} - static inline int copy_hugetlb_page_range(struct mm_struct *dst, struct mm_struct *src, struct vm_area_struct *dst_vma, diff --git a/mm/gup.c b/mm/gup.c index 43a2e0a203cd..2eb5911ba849 100644 --- a/mm/gup.c +++ b/mm/gup.c @@ -997,18 +997,11 @@ static struct page *follow_page_mask(struct vm_area_struct *vma, { pgd_t *pgd, pgdval; struct mm_struct *mm = vma->vm_mm; + struct page *page; - ctx->page_mask = 0; - - /* - * Call hugetlb_follow_page_mask for hugetlb vmas as it will use - * special hugetlb page table walking code. This eliminates the - * need to check for hugetlb entries in the general walking code. - */ - if (is_vm_hugetlb_page(vma)) - return hugetlb_follow_page_mask(vma, address, flags, - &ctx->page_mask); + vma_pgtable_walk_begin(vma); + ctx->page_mask = 0; pgd = pgd_offset(mm, address); pgdval = *pgd; @@ -1020,6 +1013,8 @@ static struct page *follow_page_mask(struct vm_area_struct *vma, else page = follow_p4d_mask(vma, address, pgd, flags, ctx); + vma_pgtable_walk_end(vma); + return page; } diff --git a/mm/hugetlb.c b/mm/hugetlb.c index abec04575c89..2e320757501b 100644 --- a/mm/hugetlb.c +++ b/mm/hugetlb.c @@ -6883,77 +6883,6 @@ int hugetlb_mfill_atomic_pte(pte_t *dst_pte, } #endif /* CONFIG_USERFAULTFD */ -struct page *hugetlb_follow_page_mask(struct vm_area_struct *vma, - unsigned long address, unsigned int flags, - unsigned int *page_mask) -{ - struct hstate *h = hstate_vma(vma); - struct mm_struct *mm = vma->vm_mm; - unsigned long haddr = address & huge_page_mask(h); - struct page *page = NULL; - spinlock_t *ptl; - pte_t *pte, entry; - int ret; - - hugetlb_vma_lock_read(vma); - pte = hugetlb_walk(vma, haddr, huge_page_size(h)); - if (!pte) - goto out_unlock; - - ptl = huge_pte_lock(h, mm, pte); - entry = huge_ptep_get(pte); - if (pte_present(entry)) { - page = pte_page(entry); - - if (!huge_pte_write(entry)) { - if (flags & FOLL_WRITE) { - page = NULL; - goto out; - } - - if (gup_must_unshare(vma, flags, page)) { - /* Tell the caller to do unsharing */ - page = ERR_PTR(-EMLINK); - goto out; - } - } - - page = nth_page(page, ((address & ~huge_page_mask(h)) >> PAGE_SHIFT)); - - /* - * Note that page may be a sub-page, and with vmemmap - * optimizations the page struct may be read only. - * try_grab_page() will increase the ref count on the - * head page, so this will be OK. - * - * try_grab_page() should always be able to get the page here, - * because we hold the ptl lock and have verified pte_present(). - */ - ret = try_grab_page(page, flags); - - if (WARN_ON_ONCE(ret)) { - page = ERR_PTR(ret); - goto out; - } - - *page_mask = (1U << huge_page_order(h)) - 1; - } -out: - spin_unlock(ptl); -out_unlock: - hugetlb_vma_unlock_read(vma); - - /* - * Fixup retval for dump requests: if pagecache doesn't exist, - * don't try to allocate a new page but just skip it. - */ - if (!page && (flags & FOLL_DUMP) && - !hugetlbfs_pagecache_present(h, vma, address)) - page = ERR_PTR(-EFAULT); - - return page; -} - long hugetlb_change_protection(struct vm_area_struct *vma, unsigned long address, unsigned long end, pgprot_t newprot, unsigned long cp_flags)