From patchwork Mon Aug 26 20:43:35 2024 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Peter Xu X-Patchwork-Id: 13778434 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from bombadil.infradead.org (bombadil.infradead.org [198.137.202.133]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (No client certificate requested) by smtp.lore.kernel.org (Postfix) with ESMTPS id 5E779C5472C for ; Mon, 26 Aug 2024 20:45:16 +0000 (UTC) DKIM-Signature: v=1; a=rsa-sha256; q=dns/txt; c=relaxed/relaxed; d=lists.infradead.org; s=bombadil.20210309; h=Sender:List-Subscribe:List-Help :List-Post:List-Archive:List-Unsubscribe:List-Id:Content-Type: Content-Transfer-Encoding:MIME-Version:References:In-Reply-To:Message-ID:Date :Subject:Cc:To:From:Reply-To:Content-ID:Content-Description:Resent-Date: Resent-From:Resent-Sender:Resent-To:Resent-Cc:Resent-Message-ID:List-Owner; bh=2iIRbDNVyj1lHjlop0tZMw2JlHFFftRtxMxE0Q8z7r8=; b=pzmI0vjfTUjygiDyM8oQVWczsE 4wV5xGmETpwPuZfc0g/W8J01K40LBedtMKlDHDaMj8g2DzVx1lIByGoq/UzDUiRtoqc1HOtmJIrnu BeZQDS2bFaANUMpFThdbtVxYKFgROuy6eVWH7m3bcNRp2OAJnDj4LpOqIpmNeB7agm4D60T1LL0OA vfQ+xNGfbaX/X1BdXmD9xD5VzLBfwFj1aSB42wsAbCN1qEvuTfhCUGnybotl/ZgPIvXL2WCtNS5hO 0C1mnuGZYqrrT04smiMu6bevTBAbq0gHvZRXonFK6BxOdg6P2I+F4llTIQ6swK+kdAT+Kr7rZ8Hxt XiU+h4KQ==; Received: from localhost ([::1] helo=bombadil.infradead.org) by bombadil.infradead.org with esmtp (Exim 4.97.1 #2 (Red Hat Linux)) id 1siga9-00000008epa-2XbI; Mon, 26 Aug 2024 20:44:57 +0000 Received: from us-smtp-delivery-124.mimecast.com ([170.10.129.124]) by bombadil.infradead.org with esmtps (Exim 4.97.1 #2 (Red Hat Linux)) id 1sigZG-00000008eKW-2yWI for linux-arm-kernel@lists.infradead.org; Mon, 26 Aug 2024 20:44:04 +0000 DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=redhat.com; s=mimecast20190719; t=1724705040; h=from:from:reply-to:subject:subject:date:date:message-id:message-id: to:to:cc:cc:mime-version:mime-version:content-type:content-type: content-transfer-encoding:content-transfer-encoding: in-reply-to:in-reply-to:references:references; bh=2iIRbDNVyj1lHjlop0tZMw2JlHFFftRtxMxE0Q8z7r8=; b=CA7zNfCF0xUY8jKJ9JgY/kSX2J8YK01F40vOchM0sYYa4NDNpDHxg9BeNO4hagmivubtyG YlZAgkrrQI78kEmy0yTimIsepL55GhwXOU5gUt2wPO9I3SedP/HrTTKKZ9SceYnSXIA30L e1107/ltOvoiEiQ0f5QPulh9HawiSEQ= Received: from mail-oo1-f70.google.com (mail-oo1-f70.google.com [209.85.161.70]) by relay.mimecast.com with ESMTP with STARTTLS (version=TLSv1.3, cipher=TLS_AES_256_GCM_SHA384) id us-mta-347-T7iKttrrONKZ__wYvhEJTQ-1; Mon, 26 Aug 2024 16:43:59 -0400 X-MC-Unique: T7iKttrrONKZ__wYvhEJTQ-1 Received: by mail-oo1-f70.google.com with SMTP id 006d021491bc7-5de842f4435so3259461eaf.1 for ; Mon, 26 Aug 2024 13:43:59 -0700 (PDT) X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20230601; t=1724705038; x=1725309838; h=content-transfer-encoding:mime-version:references:in-reply-to :message-id:date:subject:cc:to:from:x-gm-message-state:from:to:cc :subject:date:message-id:reply-to; bh=2iIRbDNVyj1lHjlop0tZMw2JlHFFftRtxMxE0Q8z7r8=; b=b7wgZo1cZBG8c9A0Vg04rV3oSwCZsYCS+eoVzT4/m0CVw068k9d/oFyZHh/8s2bpX1 1LzNOtNdo97+UhMwqIv5DnJriXgBelv/EFOWEVP2Osmrb1wSIo6Y+5Dj+v71zPeuEhPH f1Ttg+u8/v+rmVjK6C6GgpEHsjBi/SwvwRmkS+pEzUmq7NHErQUdR8SPURY5wfsgMXfl INY+AzbLsqku0Obtz2+6AEWlblB+RQZS+GvLjMjry1YM6rl0mvCBMuTOBI4slxjZjpmE V3KQFFkHFdwxUyRKdcL1bBsizLSVKBIw9JJwwfDf32Hw6maqnLYhsxAtnfg3JHGhdXuX qQrg== X-Forwarded-Encrypted: i=1; AJvYcCWD4a5qWzOq0P9z3eAO9FjeB2sdl+tVNCy6Vl57mN7j9eN5nG0AiJXbKxKnMZPFTpyWHBd0G9zjWHOXBswghZXT@lists.infradead.org X-Gm-Message-State: AOJu0Yw5jtKeA9KOXC6SVL/p4Hp9FXLzfk2/1Gfaa81To/eaFj0wEIM0 qx3hjCg9oZ2AY9BI8nEGfEwZK4seVHRpBUfySpzwtjXmhL+nhhXZQSKHJJLwLPzcCD088tpLNw+ 3VLDeO29jgylBPQlekQVDKCACFSM3gTyAF/y7/mPkD0NQIXBPAZECp09cnV2/+AHZl9dvFF8E X-Received: by 2002:a05:6358:52ca:b0:1b5:a032:9994 with SMTP id e5c5f4694b2df-1b5ebe5b1b3mr113534655d.0.1724705038386; Mon, 26 Aug 2024 13:43:58 -0700 (PDT) X-Google-Smtp-Source: AGHT+IHkrFRhmUpfOOmJ2YwtqClR5K8A29/GQ8sXabHeZ52sv0b77XENUJCQ9LJLwxTjS7HZUYg0sQ== X-Received: by 2002:a05:6358:52ca:b0:1b5:a032:9994 with SMTP id e5c5f4694b2df-1b5ebe5b1b3mr113532855d.0.1724705038035; Mon, 26 Aug 2024 13:43:58 -0700 (PDT) Received: from x1n.redhat.com (pool-99-254-121-117.cpe.net.cable.rogers.com. [99.254.121.117]) by smtp.gmail.com with ESMTPSA id af79cd13be357-7a67f3fd6c1sm491055185a.121.2024.08.26.13.43.56 (version=TLS1_3 cipher=TLS_AES_256_GCM_SHA384 bits=256/256); Mon, 26 Aug 2024 13:43:57 -0700 (PDT) From: Peter Xu To: linux-kernel@vger.kernel.org, linux-mm@kvack.org Cc: Gavin Shan , Catalin Marinas , x86@kernel.org, Ingo Molnar , Andrew Morton , Paolo Bonzini , Dave Hansen , Thomas Gleixner , Alistair Popple , kvm@vger.kernel.org, linux-arm-kernel@lists.infradead.org, Sean Christopherson , peterx@redhat.com, Oscar Salvador , Jason Gunthorpe , Borislav Petkov , Zi Yan , Axel Rasmussen , David Hildenbrand , Yan Zhao , Will Deacon , Kefeng Wang , Alex Williamson Subject: [PATCH v2 01/19] mm: Introduce ARCH_SUPPORTS_HUGE_PFNMAP and special bits to pmd/pud Date: Mon, 26 Aug 2024 16:43:35 -0400 Message-ID: <20240826204353.2228736-2-peterx@redhat.com> X-Mailer: git-send-email 2.45.0 In-Reply-To: <20240826204353.2228736-1-peterx@redhat.com> References: <20240826204353.2228736-1-peterx@redhat.com> MIME-Version: 1.0 X-Mimecast-Spam-Score: 0 X-Mimecast-Originator: redhat.com X-CRM114-Version: 20100106-BlameMichelson ( TRE 0.8.0 (BSD) ) MR-646709E3 X-CRM114-CacheID: sfid-20240826_134402_842947_714D659B X-CRM114-Status: GOOD ( 12.69 ) X-BeenThere: linux-arm-kernel@lists.infradead.org X-Mailman-Version: 2.1.34 Precedence: list List-Id: List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Sender: "linux-arm-kernel" Errors-To: linux-arm-kernel-bounces+linux-arm-kernel=archiver.kernel.org@lists.infradead.org This patch introduces the option to introduce special pte bit into pmd/puds. Archs can start to define pmd_special / pud_special when supported by selecting the new option. Per-arch support will be added later. Before that, create fallbacks for these helpers so that they are always available. Signed-off-by: Peter Xu --- include/linux/mm.h | 24 ++++++++++++++++++++++++ mm/Kconfig | 13 +++++++++++++ 2 files changed, 37 insertions(+) diff --git a/include/linux/mm.h b/include/linux/mm.h index b0ff06d18c71..d900f15b7650 100644 --- a/include/linux/mm.h +++ b/include/linux/mm.h @@ -2643,6 +2643,30 @@ static inline pte_t pte_mkspecial(pte_t pte) } #endif +#ifndef CONFIG_ARCH_SUPPORTS_PMD_PFNMAP +static inline bool pmd_special(pmd_t pmd) +{ + return false; +} + +static inline pmd_t pmd_mkspecial(pmd_t pmd) +{ + return pmd; +} +#endif /* CONFIG_ARCH_SUPPORTS_PMD_PFNMAP */ + +#ifndef CONFIG_ARCH_SUPPORTS_PUD_PFNMAP +static inline bool pud_special(pud_t pud) +{ + return false; +} + +static inline pud_t pud_mkspecial(pud_t pud) +{ + return pud; +} +#endif /* CONFIG_ARCH_SUPPORTS_PUD_PFNMAP */ + #ifndef CONFIG_ARCH_HAS_PTE_DEVMAP static inline int pte_devmap(pte_t pte) { diff --git a/mm/Kconfig b/mm/Kconfig index 8078a4b3c509..b23913d4e47e 100644 --- a/mm/Kconfig +++ b/mm/Kconfig @@ -875,6 +875,19 @@ endif # TRANSPARENT_HUGEPAGE config PGTABLE_HAS_HUGE_LEAVES def_bool TRANSPARENT_HUGEPAGE || HUGETLB_PAGE +# TODO: Allow to be enabled without THP +config ARCH_SUPPORTS_HUGE_PFNMAP + def_bool n + depends on TRANSPARENT_HUGEPAGE + +config ARCH_SUPPORTS_PMD_PFNMAP + def_bool y + depends on ARCH_SUPPORTS_HUGE_PFNMAP && HAVE_ARCH_TRANSPARENT_HUGEPAGE + +config ARCH_SUPPORTS_PUD_PFNMAP + def_bool y + depends on ARCH_SUPPORTS_HUGE_PFNMAP && HAVE_ARCH_TRANSPARENT_HUGEPAGE_PUD + # # UP and nommu archs use km based percpu allocator # From patchwork Mon Aug 26 20:43:36 2024 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Peter Xu X-Patchwork-Id: 13778436 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from bombadil.infradead.org (bombadil.infradead.org [198.137.202.133]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (No client certificate requested) by smtp.lore.kernel.org (Postfix) with ESMTPS id 462BDC5321D for ; Mon, 26 Aug 2024 20:46:46 +0000 (UTC) DKIM-Signature: v=1; a=rsa-sha256; q=dns/txt; c=relaxed/relaxed; d=lists.infradead.org; s=bombadil.20210309; h=Sender:List-Subscribe:List-Help :List-Post:List-Archive:List-Unsubscribe:List-Id:Content-Type: Content-Transfer-Encoding:MIME-Version:References:In-Reply-To:Message-ID:Date :Subject:Cc:To:From:Reply-To:Content-ID:Content-Description:Resent-Date: Resent-From:Resent-Sender:Resent-To:Resent-Cc:Resent-Message-ID:List-Owner; bh=rCV2q6t08l8ndW0pGxUU3F/GDhRRI1SX3G16E23iQDg=; b=3QbseDLbLc3wn5Nb/tewSI7SR7 6jH3lnu+b7qG0dYIoA7+Uar06KLI+7WoJ1VX0XdtsexMdqFHnVIl7rlvFzAPexFh1rLwS+6xbzlPg yc4pIMHE4pd6sfLJ3BbwEI9TxiOaVTlUFNHoSp1OD+vBmKL4hHvTtnUgpx13bXS+q7IwTcuqbvmcF BIA69JLQ2zdkeT2Zx5F59lP4kPvrfOfLZiBvXJOZNDGeaDvrEr5esAKeJyBcbnd4Lv4D+ffFcNeb4 m0YFN+bNsERly1hqXhIKfzCnPyc+W8GRuXMPJdUABtlkXGHxWhSRFv6miCtsEbRZ7BQsiLYW2Tew5 CSCEViVg==; Received: from localhost ([::1] helo=bombadil.infradead.org) by bombadil.infradead.org with esmtp (Exim 4.97.1 #2 (Red Hat Linux)) id 1sigbi-00000008fLD-0AjG; Mon, 26 Aug 2024 20:46:34 +0000 Received: from us-smtp-delivery-124.mimecast.com ([170.10.133.124]) by bombadil.infradead.org with esmtps (Exim 4.97.1 #2 (Red Hat Linux)) id 1sigZH-00000008eKg-2UVm for linux-arm-kernel@lists.infradead.org; Mon, 26 Aug 2024 20:44:05 +0000 DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=redhat.com; s=mimecast20190719; t=1724705042; h=from:from:reply-to:subject:subject:date:date:message-id:message-id: to:to:cc:cc:mime-version:mime-version:content-type:content-type: content-transfer-encoding:content-transfer-encoding: in-reply-to:in-reply-to:references:references; bh=rCV2q6t08l8ndW0pGxUU3F/GDhRRI1SX3G16E23iQDg=; b=Tl3d8A2AhgRkTO/Om/ffHNuLk5Tvtkwr+ZhC4vFyHmijx1Um9A9tO5MSWww9vCXXI9JjG2 +xcizImJhRhT03d9MrKB3hA6TpMywsDmYH1QvbCUb42nsK7xyqSwTEalUlagAedDWdcQul 9vnc/5WqJILWqKjB2TV9u8DRG+NfD6o= Received: from mail-qk1-f199.google.com (mail-qk1-f199.google.com [209.85.222.199]) by relay.mimecast.com with ESMTP with STARTTLS (version=TLSv1.3, cipher=TLS_AES_256_GCM_SHA384) id us-mta-488-vZ5H_cfANKq11zxQP9STfw-1; Mon, 26 Aug 2024 16:44:01 -0400 X-MC-Unique: vZ5H_cfANKq11zxQP9STfw-1 Received: by mail-qk1-f199.google.com with SMTP id af79cd13be357-7a1d4238d65so631891385a.0 for ; Mon, 26 Aug 2024 13:44:01 -0700 (PDT) X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20230601; t=1724705040; x=1725309840; h=content-transfer-encoding:mime-version:references:in-reply-to :message-id:date:subject:cc:to:from:x-gm-message-state:from:to:cc :subject:date:message-id:reply-to; bh=rCV2q6t08l8ndW0pGxUU3F/GDhRRI1SX3G16E23iQDg=; b=uWi95pPzCiz/FXX2qihOJvIg06N6lYbajBKZJ/IOinpmAAAX9IPPGLGQtdD9plEsb8 W66wKOGYwMwgtZGvk0PqCFgObKoY826jdD5Bu6GjhpZxZ/KTyNmvdkRiAj8i9ilT0dU+ W5i6AhuPKhGNW98hGdlzXAU+NRDBhEFUowkrSsVupxUydEhBAi3tXz3PQg85Z9B2WB44 IGG1c1yGL+DRHnkGOLVSzwr7swqBZkveioxVqAutc3zhjdj7FUqbWSw9aI15futBnQSQ Xff0RGBUaePFcln54MY2ju1ccLwmM8xcelE4sK34qvDbMhun7LyL1KXMjablMY3Vttdi 2SiQ== X-Forwarded-Encrypted: i=1; AJvYcCXlWC3DUZKmFnuAjUT33395INcMlcvYy5DpwibQnpRyO826A9vss1jfCNe4vyp3gGC0CNRPTiWkx5AAOin/wSzE@lists.infradead.org X-Gm-Message-State: AOJu0Yz4QNcyEKfaz7gUXEeHOC9RY32rLp1Jt0t3DgsZsf8gK3b7pI/e IabNLLPIoXBzZg1gdPcN+WjdbntLGIHo36s0FzKyaYRJxnLAGIwqK7LKmllOdafApY73PSwrNL7 DSdi4ywswbq+FnzJFzezXkz8054A7C/PNDwVNXqSi0owan1mNPvUIFTpLcqxE0aU8X7sruJZp X-Received: by 2002:a05:620a:28c8:b0:7a1:62ad:9d89 with SMTP id af79cd13be357-7a7e4e6d3c1mr92956385a.64.1724705040436; Mon, 26 Aug 2024 13:44:00 -0700 (PDT) X-Google-Smtp-Source: AGHT+IEW5yVxsMo5HJEI7stPZogQTAiTuzeFOof5FGrO8bEsBYthlHw7SRF31ZNT2saoeMAKCiiqzA== X-Received: by 2002:a05:620a:28c8:b0:7a1:62ad:9d89 with SMTP id af79cd13be357-7a7e4e6d3c1mr92952485a.64.1724705040096; Mon, 26 Aug 2024 13:44:00 -0700 (PDT) Received: from x1n.redhat.com (pool-99-254-121-117.cpe.net.cable.rogers.com. [99.254.121.117]) by smtp.gmail.com with ESMTPSA id af79cd13be357-7a67f3fd6c1sm491055185a.121.2024.08.26.13.43.58 (version=TLS1_3 cipher=TLS_AES_256_GCM_SHA384 bits=256/256); Mon, 26 Aug 2024 13:43:59 -0700 (PDT) From: Peter Xu To: linux-kernel@vger.kernel.org, linux-mm@kvack.org Cc: Gavin Shan , Catalin Marinas , x86@kernel.org, Ingo Molnar , Andrew Morton , Paolo Bonzini , Dave Hansen , Thomas Gleixner , Alistair Popple , kvm@vger.kernel.org, linux-arm-kernel@lists.infradead.org, Sean Christopherson , peterx@redhat.com, Oscar Salvador , Jason Gunthorpe , Borislav Petkov , Zi Yan , Axel Rasmussen , David Hildenbrand , Yan Zhao , Will Deacon , Kefeng Wang , Alex Williamson , Matthew Wilcox , "Aneesh Kumar K . V" Subject: [PATCH v2 02/19] mm: Drop is_huge_zero_pud() Date: Mon, 26 Aug 2024 16:43:36 -0400 Message-ID: <20240826204353.2228736-3-peterx@redhat.com> X-Mailer: git-send-email 2.45.0 In-Reply-To: <20240826204353.2228736-1-peterx@redhat.com> References: <20240826204353.2228736-1-peterx@redhat.com> MIME-Version: 1.0 X-Mimecast-Spam-Score: 0 X-Mimecast-Originator: redhat.com X-CRM114-Version: 20100106-BlameMichelson ( TRE 0.8.0 (BSD) ) MR-646709E3 X-CRM114-CacheID: sfid-20240826_134403_733981_C16236A3 X-CRM114-Status: GOOD ( 16.97 ) X-BeenThere: linux-arm-kernel@lists.infradead.org X-Mailman-Version: 2.1.34 Precedence: list List-Id: List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Sender: "linux-arm-kernel" Errors-To: linux-arm-kernel-bounces+linux-arm-kernel=archiver.kernel.org@lists.infradead.org It constantly returns false since 2017. One assertion is added in 2019 but it should never have triggered, IOW it means what is checked should be asserted instead. If it didn't exist for 7 years maybe it's good idea to remove it and only add it when it comes. Cc: Matthew Wilcox Cc: Aneesh Kumar K.V Acked-by: David Hildenbrand Reviewed-by: Jason Gunthorpe Signed-off-by: Peter Xu --- include/linux/huge_mm.h | 10 ---------- mm/huge_memory.c | 13 +------------ 2 files changed, 1 insertion(+), 22 deletions(-) diff --git a/include/linux/huge_mm.h b/include/linux/huge_mm.h index 4902e2f7e896..b550b5a248bb 100644 --- a/include/linux/huge_mm.h +++ b/include/linux/huge_mm.h @@ -433,11 +433,6 @@ static inline bool is_huge_zero_pmd(pmd_t pmd) return pmd_present(pmd) && READ_ONCE(huge_zero_pfn) == pmd_pfn(pmd); } -static inline bool is_huge_zero_pud(pud_t pud) -{ - return false; -} - struct folio *mm_get_huge_zero_folio(struct mm_struct *mm); void mm_put_huge_zero_folio(struct mm_struct *mm); @@ -578,11 +573,6 @@ static inline bool is_huge_zero_pmd(pmd_t pmd) return false; } -static inline bool is_huge_zero_pud(pud_t pud) -{ - return false; -} - static inline void mm_put_huge_zero_folio(struct mm_struct *mm) { return; diff --git a/mm/huge_memory.c b/mm/huge_memory.c index a81eab98d6b8..3f74b09ada38 100644 --- a/mm/huge_memory.c +++ b/mm/huge_memory.c @@ -1429,10 +1429,8 @@ static void insert_pfn_pud(struct vm_area_struct *vma, unsigned long addr, ptl = pud_lock(mm, pud); if (!pud_none(*pud)) { if (write) { - if (pud_pfn(*pud) != pfn_t_to_pfn(pfn)) { - WARN_ON_ONCE(!is_huge_zero_pud(*pud)); + if (WARN_ON_ONCE(pud_pfn(*pud) != pfn_t_to_pfn(pfn))) goto out_unlock; - } entry = pud_mkyoung(*pud); entry = maybe_pud_mkwrite(pud_mkdirty(entry), vma); if (pudp_set_access_flags(vma, addr, pud, entry, 1)) @@ -1680,15 +1678,6 @@ int copy_huge_pud(struct mm_struct *dst_mm, struct mm_struct *src_mm, if (unlikely(!pud_trans_huge(pud) && !pud_devmap(pud))) goto out_unlock; - /* - * When page table lock is held, the huge zero pud should not be - * under splitting since we don't split the page itself, only pud to - * a page table. - */ - if (is_huge_zero_pud(pud)) { - /* No huge zero pud yet */ - } - /* * TODO: once we support anonymous pages, use * folio_try_dup_anon_rmap_*() and split if duplicating fails. From patchwork Mon Aug 26 20:43:37 2024 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Peter Xu X-Patchwork-Id: 13778437 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from bombadil.infradead.org (bombadil.infradead.org [198.137.202.133]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (No client certificate requested) by smtp.lore.kernel.org (Postfix) with ESMTPS id E2E49C5472C for ; Mon, 26 Aug 2024 20:47:31 +0000 (UTC) DKIM-Signature: v=1; a=rsa-sha256; q=dns/txt; c=relaxed/relaxed; d=lists.infradead.org; s=bombadil.20210309; h=Sender:List-Subscribe:List-Help :List-Post:List-Archive:List-Unsubscribe:List-Id:Content-Type: Content-Transfer-Encoding:MIME-Version:References:In-Reply-To:Message-ID:Date :Subject:Cc:To:From:Reply-To:Content-ID:Content-Description:Resent-Date: Resent-From:Resent-Sender:Resent-To:Resent-Cc:Resent-Message-ID:List-Owner; bh=gD9ggS8OhpZf/NsZoH3fZjUIikns33eAnWwzmGyUsEI=; b=VIuXGhaXABkMUnteQ2PYcck6xL DC5T+Wtqw27FiAHP9mCKu+dJg6glwZcFZI+yFjXyqFef+NtqC/WBaFnaL6pragBdq4ZaHeUkgdDdp fqVQxElP1sQvd/2HvG0UHgGDCTAHkFr07cSdIk4W27jyOW23ekF6VQLS5HXGCSLHcPHV9YvxHmgxq Sm9zs5ayabsI0LWgmApPJPalSECYUmzaMLn7ViZ8ySDCBukDm/KjQlWYGJGwqTbR0gcyV843RuoWO m6ktSKAFJhft14V4J0SLcGrqiRLU3uEZxtG+ijZ38EbYqQSIWA0bs2tPcPgC04MjnxvHBv20Tz46f 0TyCWUAQ==; Received: from localhost ([::1] helo=bombadil.infradead.org) by bombadil.infradead.org with esmtp (Exim 4.97.1 #2 (Red Hat Linux)) id 1sigcU-00000008fck-2tZP; Mon, 26 Aug 2024 20:47:22 +0000 Received: from us-smtp-delivery-124.mimecast.com ([170.10.129.124]) by bombadil.infradead.org with esmtps (Exim 4.97.1 #2 (Red Hat Linux)) id 1sigZJ-00000008eLw-1eet for linux-arm-kernel@lists.infradead.org; Mon, 26 Aug 2024 20:44:06 +0000 DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=redhat.com; s=mimecast20190719; t=1724705044; h=from:from:reply-to:subject:subject:date:date:message-id:message-id: to:to:cc:cc:mime-version:mime-version:content-type:content-type: content-transfer-encoding:content-transfer-encoding: in-reply-to:in-reply-to:references:references; bh=gD9ggS8OhpZf/NsZoH3fZjUIikns33eAnWwzmGyUsEI=; b=Q82x8+X872PXGlQbC3y0v165wX4+CBhWfdMjlxbb5gyJBg3sWC+VmG56BDyVDwPge//L4r 2AAhQMj6yIt7eTaxQ/0tAUTtdn2VLO4nYJDKAKVdidhju9qJ5T3T2H9IE4QB75mOkD9C++ kJ/gQPPHLXAv/nQHi2JoxTHcXufRmW8= Received: from mail-qk1-f200.google.com (mail-qk1-f200.google.com [209.85.222.200]) by relay.mimecast.com with ESMTP with STARTTLS (version=TLSv1.3, cipher=TLS_AES_256_GCM_SHA384) id us-mta-367-8E2RMll4NoutypmRop0dlQ-1; Mon, 26 Aug 2024 16:44:03 -0400 X-MC-Unique: 8E2RMll4NoutypmRop0dlQ-1 Received: by mail-qk1-f200.google.com with SMTP id af79cd13be357-7a66bf35402so589826585a.3 for ; Mon, 26 Aug 2024 13:44:03 -0700 (PDT) X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20230601; t=1724705042; x=1725309842; h=content-transfer-encoding:mime-version:references:in-reply-to :message-id:date:subject:cc:to:from:x-gm-message-state:from:to:cc :subject:date:message-id:reply-to; bh=gD9ggS8OhpZf/NsZoH3fZjUIikns33eAnWwzmGyUsEI=; b=DDIzv7ZfZuoDm6xIP2ILiOZ0Gk1RVlJG7PNrLXPQZXCqYxg4Jrd2hQt59TjU6Njvr8 ib5Fwb7NzsiB2p9BfDAleBfnIuk0zI2e1MfiLkN1o++yFuo7V4jOeofDrjDFdBsR5kwT /8CawOGtYLalj7Na5uc4PxQuiNc9h/NRenz4Qf5+q/vHaagIwy7Q+OfKJGiZfA5Rq+3W AYZzyDg94eldc1lKhELHT4RR8cfabp37H+gk4E3XHeqnhXRSGaxrvboMspt5pecQ5zZY wmTm1S2oqfLLRfdmLLlAoOyrsdEGSle5b+LuWkL0G1BpUUoQUQP+FsWv/JiNsD9JtLXE mGmA== X-Forwarded-Encrypted: i=1; AJvYcCXYzQBOwfHmx5h4b8Kee14OV4TPcO7ao8oY6RF3Z1f4hCd6vlbfMkjQv0vLUHpJoSf1fcOkwux2TlMjCcz5ZEuS@lists.infradead.org X-Gm-Message-State: AOJu0YzH1rjjCrFH0b9ih1WJO6MucL72CYYYKtPf5smTwqkMj4IcIucj Ama+LU1A7O154AMnWVqvKnYyyv9w+xKTLl5w15FbRwGF9sTgd19AJ3VURGkmf7IB8S88IOPjoth plBzFXfnRQQNyGHMnIDwjVQSLn0BuT/qazMpu0iZt6q1ssjMSEIc6S7gQ7JAsMTqbl8JEk14q X-Received: by 2002:a05:620a:24d5:b0:7a3:6dd9:ef9d with SMTP id af79cd13be357-7a6897c6e76mr1277666585a.62.1724705042591; Mon, 26 Aug 2024 13:44:02 -0700 (PDT) X-Google-Smtp-Source: AGHT+IHIqrjqvJMs9Ym5fhhSHDCZDk2wQ6104PL5R9Vk0hTAFm1CUlIqFGQmzGdyDHGQhWdDyxioXg== X-Received: by 2002:a05:620a:24d5:b0:7a3:6dd9:ef9d with SMTP id af79cd13be357-7a6897c6e76mr1277664185a.62.1724705042003; Mon, 26 Aug 2024 13:44:02 -0700 (PDT) Received: from x1n.redhat.com (pool-99-254-121-117.cpe.net.cable.rogers.com. [99.254.121.117]) by smtp.gmail.com with ESMTPSA id af79cd13be357-7a67f3fd6c1sm491055185a.121.2024.08.26.13.44.00 (version=TLS1_3 cipher=TLS_AES_256_GCM_SHA384 bits=256/256); Mon, 26 Aug 2024 13:44:01 -0700 (PDT) From: Peter Xu To: linux-kernel@vger.kernel.org, linux-mm@kvack.org Cc: Gavin Shan , Catalin Marinas , x86@kernel.org, Ingo Molnar , Andrew Morton , Paolo Bonzini , Dave Hansen , Thomas Gleixner , Alistair Popple , kvm@vger.kernel.org, linux-arm-kernel@lists.infradead.org, Sean Christopherson , peterx@redhat.com, Oscar Salvador , Jason Gunthorpe , Borislav Petkov , Zi Yan , Axel Rasmussen , David Hildenbrand , Yan Zhao , Will Deacon , Kefeng Wang , Alex Williamson Subject: [PATCH v2 03/19] mm: Mark special bits for huge pfn mappings when inject Date: Mon, 26 Aug 2024 16:43:37 -0400 Message-ID: <20240826204353.2228736-4-peterx@redhat.com> X-Mailer: git-send-email 2.45.0 In-Reply-To: <20240826204353.2228736-1-peterx@redhat.com> References: <20240826204353.2228736-1-peterx@redhat.com> MIME-Version: 1.0 X-Mimecast-Spam-Score: 0 X-Mimecast-Originator: redhat.com X-CRM114-Version: 20100106-BlameMichelson ( TRE 0.8.0 (BSD) ) MR-646709E3 X-CRM114-CacheID: sfid-20240826_134405_529893_C9596016 X-CRM114-Status: GOOD ( 12.52 ) X-BeenThere: linux-arm-kernel@lists.infradead.org X-Mailman-Version: 2.1.34 Precedence: list List-Id: List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Sender: "linux-arm-kernel" Errors-To: linux-arm-kernel-bounces+linux-arm-kernel=archiver.kernel.org@lists.infradead.org We need these special bits to be around on pfnmaps. Mark properly for !devmap case, reflecting that there's no page struct backing the entry. Reviewed-by: Jason Gunthorpe Signed-off-by: Peter Xu Acked-by: David Hildenbrand --- mm/huge_memory.c | 4 ++++ 1 file changed, 4 insertions(+) diff --git a/mm/huge_memory.c b/mm/huge_memory.c index 3f74b09ada38..dec17d09390f 100644 --- a/mm/huge_memory.c +++ b/mm/huge_memory.c @@ -1346,6 +1346,8 @@ static void insert_pfn_pmd(struct vm_area_struct *vma, unsigned long addr, entry = pmd_mkhuge(pfn_t_pmd(pfn, prot)); if (pfn_t_devmap(pfn)) entry = pmd_mkdevmap(entry); + else + entry = pmd_mkspecial(entry); if (write) { entry = pmd_mkyoung(pmd_mkdirty(entry)); entry = maybe_pmd_mkwrite(entry, vma); @@ -1442,6 +1444,8 @@ static void insert_pfn_pud(struct vm_area_struct *vma, unsigned long addr, entry = pud_mkhuge(pfn_t_pud(pfn, prot)); if (pfn_t_devmap(pfn)) entry = pud_mkdevmap(entry); + else + entry = pud_mkspecial(entry); if (write) { entry = pud_mkyoung(pud_mkdirty(entry)); entry = maybe_pud_mkwrite(entry, vma); From patchwork Mon Aug 26 20:43:38 2024 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Peter Xu X-Patchwork-Id: 13778438 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from bombadil.infradead.org (bombadil.infradead.org [198.137.202.133]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (No client certificate requested) by smtp.lore.kernel.org (Postfix) with ESMTPS id 49E34C5472C for ; Mon, 26 Aug 2024 20:48:21 +0000 (UTC) DKIM-Signature: v=1; a=rsa-sha256; q=dns/txt; c=relaxed/relaxed; d=lists.infradead.org; s=bombadil.20210309; h=Sender:List-Subscribe:List-Help :List-Post:List-Archive:List-Unsubscribe:List-Id:Content-Type: Content-Transfer-Encoding:MIME-Version:References:In-Reply-To:Message-ID:Date :Subject:Cc:To:From:Reply-To:Content-ID:Content-Description:Resent-Date: Resent-From:Resent-Sender:Resent-To:Resent-Cc:Resent-Message-ID:List-Owner; bh=p41Ko1t0SYR+aQ+NnkPoKcQjWVRbF5iAoso+I7QNClw=; b=pN3U05GjUcweRvL5heTUAFdeaU aMVGBWy29sH1qIGQG7HVilnIRDWiwl98+v95rsavnmbR8kMfnNmX1+0n3ZStLW/N31gpm113E7TuM GggX5pRFxl7sMYMEsvXnGgRQJHjs3vfVbEPoGW7dkjtM9zfhggDZiDWG4NFMxY4JBzl0BhZk7cs8Q 2AmA/bCYyjGy1Fy/qtb1UOy2Cs2vmTxDvyya7Exv0RTVNZcwfOhHSgRgSeGi97i5u6Bo4HkRnWjX0 lMp0HJ8IDYMFZXxevvkR0UW3fQS3zSu7XYUhAlNXauhbGf8+FYqjoYREnfyKzcyLUes25qSnCULqq azb4Pgog==; Received: from localhost ([::1] helo=bombadil.infradead.org) by bombadil.infradead.org with esmtp (Exim 4.97.1 #2 (Red Hat Linux)) id 1sigdF-00000008frz-3bDl; Mon, 26 Aug 2024 20:48:09 +0000 Received: from us-smtp-delivery-124.mimecast.com ([170.10.129.124]) by bombadil.infradead.org with esmtps (Exim 4.97.1 #2 (Red Hat Linux)) id 1sigZL-00000008eN5-2jIh for linux-arm-kernel@lists.infradead.org; Mon, 26 Aug 2024 20:44:09 +0000 DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=redhat.com; s=mimecast20190719; t=1724705046; h=from:from:reply-to:subject:subject:date:date:message-id:message-id: to:to:cc:cc:mime-version:mime-version:content-type:content-type: content-transfer-encoding:content-transfer-encoding: in-reply-to:in-reply-to:references:references; bh=p41Ko1t0SYR+aQ+NnkPoKcQjWVRbF5iAoso+I7QNClw=; b=LCF+o8RT9taIiLwfMAqK5M5Ylj+viyK+Apsw/ZlBTKag2Mph/2nbTL4kd+KAh5GGHkTUY5 gpiD7qtxQaZXCCRWPIGISOGV+oSglWgCdT3PWRfRibrGOD4XMG6fRa4gQ/Jzv5syo81bGC 5Xz0Rtw6gx9m4ng0NBrN6Vim7bCAiVY= Received: from mail-oa1-f72.google.com (mail-oa1-f72.google.com [209.85.160.72]) by relay.mimecast.com with ESMTP with STARTTLS (version=TLSv1.3, cipher=TLS_AES_256_GCM_SHA384) id us-mta-687-byQKyuB4PPuNdpPx_FAfHw-1; Mon, 26 Aug 2024 16:44:05 -0400 X-MC-Unique: byQKyuB4PPuNdpPx_FAfHw-1 Received: by mail-oa1-f72.google.com with SMTP id 586e51a60fabf-26823e7e16eso10889792fac.3 for ; Mon, 26 Aug 2024 13:44:05 -0700 (PDT) X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20230601; t=1724705044; x=1725309844; h=content-transfer-encoding:mime-version:references:in-reply-to :message-id:date:subject:cc:to:from:x-gm-message-state:from:to:cc :subject:date:message-id:reply-to; bh=p41Ko1t0SYR+aQ+NnkPoKcQjWVRbF5iAoso+I7QNClw=; b=qOkgE2kBOKIRmlUhNOVJGBhWvHyPZXoPLUghVQIj2kf7UDCkHyG2/LaYj8zD25H640 Qs769UAuLe3zTbSHrrlQ5gcSZC5nAo7euFoCGpODfhjyJ2YrevJ4MN4Su8GUI4Gn0J0M 2mAnJfQIGwjbBvIfZVNSG9qYLftMqtWm2DZsAWYPI/w0GsUrfKcpIlouj4ikWrkdGYz/ U/TJILNvFLAfWQdxTJoYGeXU6yeSZik7jt7y+GgYtc0DjiEPVR6lVDRnXF/W8GkUbgbm gLHL9RJFbkXAOWJtudMqklVAr0yYi0XrZYcurzVYNoRGqhG2dhHDfHZb9GhC7uE2AzY/ wqog== X-Forwarded-Encrypted: i=1; AJvYcCVMNTWP4Kql3sjE+0dWwMVm6L5x767fZkwVKIoqRvIPocGDE9Sr/jmRzv/ai7RPtMsTv1Lp+h5AC0EzZrmicav6@lists.infradead.org X-Gm-Message-State: AOJu0Yw42V9vBKZWskhbNesePRjoIECGDDVhWb9AJTMOcykEi78oKtOa R203ZGS8LPa4WEdwcil4ndE3eCYagVyP7qpntil4BGKmU0uR2+wP3l7xMbS/hOUtGbvByQgvq4C 56x70nAxOAC9RQdw98yrADG80HQnV1/LrZHOh2jOuJWlxj1FpH6EJyk4oFrWIQFU7ESb8mYdy X-Received: by 2002:a05:6358:ca5:b0:1b5:ecce:b760 with SMTP id e5c5f4694b2df-1b5ecceb805mr68656255d.21.1724705044542; Mon, 26 Aug 2024 13:44:04 -0700 (PDT) X-Google-Smtp-Source: AGHT+IGqSprkmyPyKIDES0Mxu2D+0b60kqL8Leav+5Mf5w5kuR6qgHz2l36Zs0Y1NhYRXD/bnn90Fg== X-Received: by 2002:a05:6358:ca5:b0:1b5:ecce:b760 with SMTP id e5c5f4694b2df-1b5ecceb805mr68653955d.21.1724705044223; Mon, 26 Aug 2024 13:44:04 -0700 (PDT) Received: from x1n.redhat.com (pool-99-254-121-117.cpe.net.cable.rogers.com. [99.254.121.117]) by smtp.gmail.com with ESMTPSA id af79cd13be357-7a67f3fd6c1sm491055185a.121.2024.08.26.13.44.02 (version=TLS1_3 cipher=TLS_AES_256_GCM_SHA384 bits=256/256); Mon, 26 Aug 2024 13:44:03 -0700 (PDT) From: Peter Xu To: linux-kernel@vger.kernel.org, linux-mm@kvack.org Cc: Gavin Shan , Catalin Marinas , x86@kernel.org, Ingo Molnar , Andrew Morton , Paolo Bonzini , Dave Hansen , Thomas Gleixner , Alistair Popple , kvm@vger.kernel.org, linux-arm-kernel@lists.infradead.org, Sean Christopherson , peterx@redhat.com, Oscar Salvador , Jason Gunthorpe , Borislav Petkov , Zi Yan , Axel Rasmussen , David Hildenbrand , Yan Zhao , Will Deacon , Kefeng Wang , Alex Williamson , Matthew Wilcox , Ryan Roberts Subject: [PATCH v2 04/19] mm: Allow THP orders for PFNMAPs Date: Mon, 26 Aug 2024 16:43:38 -0400 Message-ID: <20240826204353.2228736-5-peterx@redhat.com> X-Mailer: git-send-email 2.45.0 In-Reply-To: <20240826204353.2228736-1-peterx@redhat.com> References: <20240826204353.2228736-1-peterx@redhat.com> MIME-Version: 1.0 X-Mimecast-Spam-Score: 0 X-Mimecast-Originator: redhat.com X-CRM114-Version: 20100106-BlameMichelson ( TRE 0.8.0 (BSD) ) MR-646709E3 X-CRM114-CacheID: sfid-20240826_134407_784043_7745525B X-CRM114-Status: GOOD ( 17.49 ) X-BeenThere: linux-arm-kernel@lists.infradead.org X-Mailman-Version: 2.1.34 Precedence: list List-Id: List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Sender: "linux-arm-kernel" Errors-To: linux-arm-kernel-bounces+linux-arm-kernel=archiver.kernel.org@lists.infradead.org This enables PFNMAPs to be mapped at either pmd/pud layers. Generalize the dax case into vma_is_special_huge() so as to cover both. Meanwhile, rename the macro to THP_ORDERS_ALL_SPECIAL. Cc: Matthew Wilcox Cc: Gavin Shan Cc: Ryan Roberts Cc: David Hildenbrand Cc: Zi Yan Reviewed-by: Jason Gunthorpe Signed-off-by: Peter Xu Acked-by: David Hildenbrand --- include/linux/huge_mm.h | 6 +++--- mm/huge_memory.c | 4 ++-- 2 files changed, 5 insertions(+), 5 deletions(-) diff --git a/include/linux/huge_mm.h b/include/linux/huge_mm.h index b550b5a248bb..4da102b74a8c 100644 --- a/include/linux/huge_mm.h +++ b/include/linux/huge_mm.h @@ -76,9 +76,9 @@ extern struct kobj_attribute thpsize_shmem_enabled_attr; /* * Mask of all large folio orders supported for file THP. Folios in a DAX * file is never split and the MAX_PAGECACHE_ORDER limit does not apply to - * it. + * it. Same to PFNMAPs where there's neither page* nor pagecache. */ -#define THP_ORDERS_ALL_FILE_DAX \ +#define THP_ORDERS_ALL_SPECIAL \ (BIT(PMD_ORDER) | BIT(PUD_ORDER)) #define THP_ORDERS_ALL_FILE_DEFAULT \ ((BIT(MAX_PAGECACHE_ORDER + 1) - 1) & ~BIT(0)) @@ -87,7 +87,7 @@ extern struct kobj_attribute thpsize_shmem_enabled_attr; * Mask of all large folio orders supported for THP. */ #define THP_ORDERS_ALL \ - (THP_ORDERS_ALL_ANON | THP_ORDERS_ALL_FILE_DAX | THP_ORDERS_ALL_FILE_DEFAULT) + (THP_ORDERS_ALL_ANON | THP_ORDERS_ALL_SPECIAL | THP_ORDERS_ALL_FILE_DEFAULT) #define TVA_SMAPS (1 << 0) /* Will be used for procfs */ #define TVA_IN_PF (1 << 1) /* Page fault handler */ diff --git a/mm/huge_memory.c b/mm/huge_memory.c index dec17d09390f..e2c314f631f3 100644 --- a/mm/huge_memory.c +++ b/mm/huge_memory.c @@ -96,8 +96,8 @@ unsigned long __thp_vma_allowable_orders(struct vm_area_struct *vma, /* Check the intersection of requested and supported orders. */ if (vma_is_anonymous(vma)) supported_orders = THP_ORDERS_ALL_ANON; - else if (vma_is_dax(vma)) - supported_orders = THP_ORDERS_ALL_FILE_DAX; + else if (vma_is_special_huge(vma)) + supported_orders = THP_ORDERS_ALL_SPECIAL; else supported_orders = THP_ORDERS_ALL_FILE_DEFAULT; From patchwork Mon Aug 26 20:43:39 2024 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Peter Xu X-Patchwork-Id: 13778439 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from bombadil.infradead.org (bombadil.infradead.org [198.137.202.133]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (No client certificate requested) by smtp.lore.kernel.org (Postfix) with ESMTPS id 3AEA5C5472C for ; Mon, 26 Aug 2024 20:49:07 +0000 (UTC) DKIM-Signature: v=1; a=rsa-sha256; q=dns/txt; c=relaxed/relaxed; d=lists.infradead.org; s=bombadil.20210309; h=Sender:List-Subscribe:List-Help :List-Post:List-Archive:List-Unsubscribe:List-Id:Content-Type: Content-Transfer-Encoding:MIME-Version:References:In-Reply-To:Message-ID:Date :Subject:Cc:To:From:Reply-To:Content-ID:Content-Description:Resent-Date: Resent-From:Resent-Sender:Resent-To:Resent-Cc:Resent-Message-ID:List-Owner; bh=SFHgAMnpeJHqDMa0ADr1pyvSduy44GuxF04BOe5aqcw=; b=N9JDqXq548pGbLvy63j22ZJu02 2ePjiETF/ah6PpEIgycwCfBqz6pyRzsvsffFdL746oBq78RLWGpXmS0EAW2JndmFkf39FUGB3UPqw Kbkw0xgOBe+ZEOXA8N/Sy5r0Pu5siUpgv0rsrTzGy1ZT9jP+AAv3tdvgoghIwt7eCdSwgzFy8obNh C9dKbo2ao3d1W/u+DVxO5F3//RCFJ2Q991nL2pLbf/LHdVQaGn/kPwt4KJY9x7ghFYvxRrz3Jx0TL FxBaPGlKyFAr4Cry6i4cnUooTxuuOlhHY3zfYNeMBZ1gsBU7d+pvFExAlVIPLg65PgePZqWVBSTNN iV5XTRMw==; Received: from localhost ([::1] helo=bombadil.infradead.org) by bombadil.infradead.org with esmtp (Exim 4.97.1 #2 (Red Hat Linux)) id 1sige2-00000008g8P-09Dx; Mon, 26 Aug 2024 20:48:58 +0000 Received: from us-smtp-delivery-124.mimecast.com ([170.10.133.124]) by bombadil.infradead.org with esmtps (Exim 4.97.1 #2 (Red Hat Linux)) id 1sigZO-00000008eOJ-1Z9i for linux-arm-kernel@lists.infradead.org; Mon, 26 Aug 2024 20:44:12 +0000 DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=redhat.com; s=mimecast20190719; t=1724705049; h=from:from:reply-to:subject:subject:date:date:message-id:message-id: to:to:cc:cc:mime-version:mime-version:content-type:content-type: content-transfer-encoding:content-transfer-encoding: in-reply-to:in-reply-to:references:references; bh=SFHgAMnpeJHqDMa0ADr1pyvSduy44GuxF04BOe5aqcw=; b=dtTbA9uKpTvjGFS0JZf60dx1PumcxPN8MyfLw0XI+pduVKl79+5dyti6KkGbZSULdVyekz yvp3eebj5MU83nyajfD1/edzpeMbp+gJW9i4lH1F6KJ79QbC1ClcYhecnPtcG1+TOmF+85 q0TkT4kTpEb9Z9Cz+HtK46IrxJ/xyu4= Received: from mail-qk1-f197.google.com (mail-qk1-f197.google.com [209.85.222.197]) by relay.mimecast.com with ESMTP with STARTTLS (version=TLSv1.3, cipher=TLS_AES_256_GCM_SHA384) id us-mta-194-9IFjm5D8Oq6-84A60BXPnA-1; Mon, 26 Aug 2024 16:44:07 -0400 X-MC-Unique: 9IFjm5D8Oq6-84A60BXPnA-1 Received: by mail-qk1-f197.google.com with SMTP id af79cd13be357-7a1dbbe6d6fso633435085a.3 for ; Mon, 26 Aug 2024 13:44:07 -0700 (PDT) X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20230601; t=1724705047; x=1725309847; h=content-transfer-encoding:mime-version:references:in-reply-to :message-id:date:subject:cc:to:from:x-gm-message-state:from:to:cc :subject:date:message-id:reply-to; bh=SFHgAMnpeJHqDMa0ADr1pyvSduy44GuxF04BOe5aqcw=; b=Rv8W523hnEYFlnSXEY6NBwAD3ZorhUqDjPjx4hZV/xpsc3RVCHypGoraCDmE4Cv7h5 Aa1MSZOayhfoDxT9x6Qf2GJfjQANHCPx1w70RikKHN/yoWvDCg0g7WEBvbV8Ufjlnxdb m5TfHcc04s/Xd7eshUVN6hLm7O+O59VokCWogpsVIeMmceW9Cnk7Z6on8yEPUs1FJjkq IKF4i3wyQYncbIqQd0G3LPzz7ZBEntoflPKGxcbj+UiHV15fH7xs74BvDMidX5eC3xsX 5ZI0ZIV9EYX48Qsd1k92UEC6CqPFdByUNTdZXms5/aENI2ZgSlXt42GEFynxv0TU2pKM LsGQ== X-Forwarded-Encrypted: i=1; AJvYcCWfIe0kb6f3hdCAfwU+J4GNmtfGWwPGpfhMyZFJ2HM6hbQRkI7dnEkYKUiCvNiFdM1AcieQUXvQuFx0+YpTrRMH@lists.infradead.org X-Gm-Message-State: AOJu0YwFDDJshI3fslyS0cocTkK5Mld7AmRlXKq5DWRooz75YUhZlzFL YXhD760bH600jeyPliw7EhOD0lkHrMVIyNRM1kAs966cS8RNKs+8FNMwGIL+M1zhoMBuNF9E9dX WbPSebptQ1dKjQ8NJTiCAbfFRV01ilhV/pzq6ILhm/OjOtV2JHl6EDJ3UiHqe7aXIdQACyugn X-Received: by 2002:a05:620a:2a14:b0:7a2:ccd:9672 with SMTP id af79cd13be357-7a7e4e6144fmr67568485a.67.1724705046879; Mon, 26 Aug 2024 13:44:06 -0700 (PDT) X-Google-Smtp-Source: AGHT+IG++rZwRsHPi3LEX2hBpbUb+Bhnwx97nv6F7Fy5QJm5RgQYxXJ7nF8aD8ykhSm+rXvWrvbUzQ== X-Received: by 2002:a05:620a:2a14:b0:7a2:ccd:9672 with SMTP id af79cd13be357-7a7e4e6144fmr67565885a.67.1724705046455; Mon, 26 Aug 2024 13:44:06 -0700 (PDT) Received: from x1n.redhat.com (pool-99-254-121-117.cpe.net.cable.rogers.com. [99.254.121.117]) by smtp.gmail.com with ESMTPSA id af79cd13be357-7a67f3fd6c1sm491055185a.121.2024.08.26.13.44.04 (version=TLS1_3 cipher=TLS_AES_256_GCM_SHA384 bits=256/256); Mon, 26 Aug 2024 13:44:05 -0700 (PDT) From: Peter Xu To: linux-kernel@vger.kernel.org, linux-mm@kvack.org Cc: Gavin Shan , Catalin Marinas , x86@kernel.org, Ingo Molnar , Andrew Morton , Paolo Bonzini , Dave Hansen , Thomas Gleixner , Alistair Popple , kvm@vger.kernel.org, linux-arm-kernel@lists.infradead.org, Sean Christopherson , peterx@redhat.com, Oscar Salvador , Jason Gunthorpe , Borislav Petkov , Zi Yan , Axel Rasmussen , David Hildenbrand , Yan Zhao , Will Deacon , Kefeng Wang , Alex Williamson Subject: [PATCH v2 05/19] mm/gup: Detect huge pfnmap entries in gup-fast Date: Mon, 26 Aug 2024 16:43:39 -0400 Message-ID: <20240826204353.2228736-6-peterx@redhat.com> X-Mailer: git-send-email 2.45.0 In-Reply-To: <20240826204353.2228736-1-peterx@redhat.com> References: <20240826204353.2228736-1-peterx@redhat.com> MIME-Version: 1.0 X-Mimecast-Spam-Score: 0 X-Mimecast-Originator: redhat.com X-CRM114-Version: 20100106-BlameMichelson ( TRE 0.8.0 (BSD) ) MR-646709E3 X-CRM114-CacheID: sfid-20240826_134410_665193_A25BD6FE X-CRM114-Status: GOOD ( 12.56 ) X-BeenThere: linux-arm-kernel@lists.infradead.org X-Mailman-Version: 2.1.34 Precedence: list List-Id: List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Sender: "linux-arm-kernel" Errors-To: linux-arm-kernel-bounces+linux-arm-kernel=archiver.kernel.org@lists.infradead.org Since gup-fast doesn't have the vma reference, teach it to detect such huge pfnmaps by checking the special bit for pmd/pud too, just like ptes. Acked-by: David Hildenbrand Reviewed-by: Jason Gunthorpe Signed-off-by: Peter Xu --- mm/gup.c | 6 ++++++ 1 file changed, 6 insertions(+) diff --git a/mm/gup.c b/mm/gup.c index d19884e097fd..a49f67a512ee 100644 --- a/mm/gup.c +++ b/mm/gup.c @@ -3038,6 +3038,9 @@ static int gup_fast_pmd_leaf(pmd_t orig, pmd_t *pmdp, unsigned long addr, if (!pmd_access_permitted(orig, flags & FOLL_WRITE)) return 0; + if (pmd_special(orig)) + return 0; + if (pmd_devmap(orig)) { if (unlikely(flags & FOLL_LONGTERM)) return 0; @@ -3082,6 +3085,9 @@ static int gup_fast_pud_leaf(pud_t orig, pud_t *pudp, unsigned long addr, if (!pud_access_permitted(orig, flags & FOLL_WRITE)) return 0; + if (pud_special(orig)) + return 0; + if (pud_devmap(orig)) { if (unlikely(flags & FOLL_LONGTERM)) return 0; From patchwork Mon Aug 26 20:43:40 2024 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Peter Xu X-Patchwork-Id: 13778440 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from bombadil.infradead.org (bombadil.infradead.org [198.137.202.133]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (No client certificate requested) by smtp.lore.kernel.org (Postfix) with ESMTPS id CF2E9C5321D for ; Mon, 26 Aug 2024 20:49:57 +0000 (UTC) DKIM-Signature: v=1; a=rsa-sha256; q=dns/txt; c=relaxed/relaxed; d=lists.infradead.org; s=bombadil.20210309; h=Sender:List-Subscribe:List-Help :List-Post:List-Archive:List-Unsubscribe:List-Id:Content-Type: Content-Transfer-Encoding:MIME-Version:References:In-Reply-To:Message-ID:Date :Subject:Cc:To:From:Reply-To:Content-ID:Content-Description:Resent-Date: Resent-From:Resent-Sender:Resent-To:Resent-Cc:Resent-Message-ID:List-Owner; bh=UaLXo2idJup/Oy4OwRPy2aK364ayUQ63MrUET5wBpvQ=; b=N2t8mnKY/hABGYJI90Cf7Z6Nku 1++BG4U5NEyMczAonkQHNpd3IhjHf7d6C5Q+DO2PpXAb+2Fdssm3G4/wtvMxWwgXcaoa+Tu4Jop/W /FmzghjGTYQL1Do+iRFNLk1NVbr/Zw9DBnKn7iSd/6LwOtb6cmqcWyvV5uoPdegYyryvF2+0C8IU4 5P7J6EGURD22d67NacIviifexMq4+sf8eo0cFzvPsIego9kTobvfZypA751CdQ41FakHXO2iRCyoX 50oZeo3aXx2I/Gi98CE94JOhn59XJB+jdJjzcdXwYv7G41qH/WQr9HWBb3RjHTphiMFY/omiTK7BH 7dHimKfA==; Received: from localhost ([::1] helo=bombadil.infradead.org) by bombadil.infradead.org with esmtp (Exim 4.97.1 #2 (Red Hat Linux)) id 1sigen-00000008gPD-41NT; Mon, 26 Aug 2024 20:49:45 +0000 Received: from us-smtp-delivery-124.mimecast.com ([170.10.133.124]) by bombadil.infradead.org with esmtps (Exim 4.97.1 #2 (Red Hat Linux)) id 1sigZQ-00000008ePr-0oIc for linux-arm-kernel@lists.infradead.org; Mon, 26 Aug 2024 20:44:13 +0000 DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=redhat.com; s=mimecast20190719; t=1724705051; h=from:from:reply-to:subject:subject:date:date:message-id:message-id: to:to:cc:cc:mime-version:mime-version:content-type:content-type: content-transfer-encoding:content-transfer-encoding: in-reply-to:in-reply-to:references:references; bh=UaLXo2idJup/Oy4OwRPy2aK364ayUQ63MrUET5wBpvQ=; b=VvSY6pkvdoBZR75VdSwrCXL41Ztxpb9sxBHfdhY1xyQ9FTIzLlFvJe2N827g+cLdbXveRS Lwc/9rtaxin1Lh9fU+Vxa4dz6k7DpopmW+Ff8gMRfm2neGduTqzujKJAoilpj+uK0YWRkd f3SCG/mG83+FK8OwdFk0J0XdZjisQFY= Received: from mail-qk1-f197.google.com (mail-qk1-f197.google.com [209.85.222.197]) by relay.mimecast.com with ESMTP with STARTTLS (version=TLSv1.3, cipher=TLS_AES_256_GCM_SHA384) id us-mta-562-XgI7vkohOLqhEW4mRYFRZQ-1; Mon, 26 Aug 2024 16:44:09 -0400 X-MC-Unique: XgI7vkohOLqhEW4mRYFRZQ-1 Received: by mail-qk1-f197.google.com with SMTP id af79cd13be357-7a1df704ef8so599548585a.2 for ; Mon, 26 Aug 2024 13:44:09 -0700 (PDT) X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20230601; t=1724705049; x=1725309849; h=content-transfer-encoding:mime-version:references:in-reply-to :message-id:date:subject:cc:to:from:x-gm-message-state:from:to:cc :subject:date:message-id:reply-to; bh=UaLXo2idJup/Oy4OwRPy2aK364ayUQ63MrUET5wBpvQ=; b=L+VKCoyU0i86mwAnvKHjVc9J6THUSPIFAJe6RSdVi+KoKuLnqgRWeThWs8U88XASLP 8Nfd4ygWLsBHfy2ZPxp5QumgsTBBOMXmoihTKdXGM04vtP09A2p8JfSX3eaNK09KtncH B5qv/wfnRF0wVp57XR6bYp+CoH1zQz0T4DTyxNgBhKlkxCC0ALCZJremaroGW1ByaqOx qIptyqnNtE6N+3Xat6uLb59fkwR/BWx/7EF+VAII8eSWZAtXM7SggXJz+WJBvQsPcZj5 p5hXUpJOaNpmYFD7v+9LkuO5BFG5ExP9sM8As1n4stBKSHr7oA/RIsauMQxGsLK/VMIL gM5w== X-Forwarded-Encrypted: i=1; AJvYcCXYtXGJop0pnJi240vxUJSVEtUDF9p+/H2SnipZX7D7L5ln+/1Y+UXKFddFsUZ6xf42SxfbF5v3fHLr2akKf/pZ@lists.infradead.org X-Gm-Message-State: AOJu0YxSf9iS1qn9GiKvU2HKOoyNZ/NtC5DVm8nCtbcL8NIoO0eCdnst nq19aWZofcdEpcobTAWXfumhSTZZRsDjGo5r/52gjxfFRIakSmZFaNp2jM4yWS/oZb5pb4ncD0u c6cv6V8WcRdidinnRBWZbPbvhBsU36Blhdce7+bNu0pLov1Y1ZvtsA5ZMH1KbfzILZ4F6dzdW X-Received: by 2002:a05:620a:4112:b0:7a3:49dc:e6e3 with SMTP id af79cd13be357-7a6897a6f65mr1244451985a.53.1724705048900; Mon, 26 Aug 2024 13:44:08 -0700 (PDT) X-Google-Smtp-Source: AGHT+IFAEqST5igJBEO0jhxDT4OAwN3xGv5rzMyL4N+pVFoujxFDEyZ/hbdQR2zB1kD+DQ4JLF652A== X-Received: by 2002:a05:620a:4112:b0:7a3:49dc:e6e3 with SMTP id af79cd13be357-7a6897a6f65mr1244449385a.53.1724705048557; Mon, 26 Aug 2024 13:44:08 -0700 (PDT) Received: from x1n.redhat.com (pool-99-254-121-117.cpe.net.cable.rogers.com. [99.254.121.117]) by smtp.gmail.com with ESMTPSA id af79cd13be357-7a67f3fd6c1sm491055185a.121.2024.08.26.13.44.06 (version=TLS1_3 cipher=TLS_AES_256_GCM_SHA384 bits=256/256); Mon, 26 Aug 2024 13:44:07 -0700 (PDT) From: Peter Xu To: linux-kernel@vger.kernel.org, linux-mm@kvack.org Cc: Gavin Shan , Catalin Marinas , x86@kernel.org, Ingo Molnar , Andrew Morton , Paolo Bonzini , Dave Hansen , Thomas Gleixner , Alistair Popple , kvm@vger.kernel.org, linux-arm-kernel@lists.infradead.org, Sean Christopherson , peterx@redhat.com, Oscar Salvador , Jason Gunthorpe , Borislav Petkov , Zi Yan , Axel Rasmussen , David Hildenbrand , Yan Zhao , Will Deacon , Kefeng Wang , Alex Williamson Subject: [PATCH v2 06/19] mm/pagewalk: Check pfnmap for folio_walk_start() Date: Mon, 26 Aug 2024 16:43:40 -0400 Message-ID: <20240826204353.2228736-7-peterx@redhat.com> X-Mailer: git-send-email 2.45.0 In-Reply-To: <20240826204353.2228736-1-peterx@redhat.com> References: <20240826204353.2228736-1-peterx@redhat.com> MIME-Version: 1.0 X-Mimecast-Spam-Score: 0 X-Mimecast-Originator: redhat.com X-CRM114-Version: 20100106-BlameMichelson ( TRE 0.8.0 (BSD) ) MR-646709E3 X-CRM114-CacheID: sfid-20240826_134412_457359_ED5AEC7C X-CRM114-Status: GOOD ( 13.80 ) X-BeenThere: linux-arm-kernel@lists.infradead.org X-Mailman-Version: 2.1.34 Precedence: list List-Id: List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Sender: "linux-arm-kernel" Errors-To: linux-arm-kernel-bounces+linux-arm-kernel=archiver.kernel.org@lists.infradead.org Teach folio_walk_start() to recognize special pmd/pud mappings, and fail them properly as it means there's no folio backing them. Cc: David Hildenbrand Signed-off-by: Peter Xu --- mm/pagewalk.c | 4 ++-- 1 file changed, 2 insertions(+), 2 deletions(-) diff --git a/mm/pagewalk.c b/mm/pagewalk.c index cd79fb3b89e5..12be5222d70e 100644 --- a/mm/pagewalk.c +++ b/mm/pagewalk.c @@ -753,7 +753,7 @@ struct folio *folio_walk_start(struct folio_walk *fw, fw->pudp = pudp; fw->pud = pud; - if (!pud_present(pud) || pud_devmap(pud)) { + if (!pud_present(pud) || pud_devmap(pud) || pud_special(pud)) { spin_unlock(ptl); goto not_found; } else if (!pud_leaf(pud)) { @@ -783,7 +783,7 @@ struct folio *folio_walk_start(struct folio_walk *fw, fw->pmdp = pmdp; fw->pmd = pmd; - if (pmd_none(pmd)) { + if (pmd_none(pmd) || pmd_special(pmd)) { spin_unlock(ptl); goto not_found; } else if (!pmd_leaf(pmd)) { From patchwork Mon Aug 26 20:43:41 2024 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Peter Xu X-Patchwork-Id: 13778441 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from bombadil.infradead.org (bombadil.infradead.org [198.137.202.133]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (No client certificate requested) by smtp.lore.kernel.org (Postfix) with ESMTPS id 612F1C5321D for ; Mon, 26 Aug 2024 20:50:44 +0000 (UTC) DKIM-Signature: v=1; a=rsa-sha256; q=dns/txt; c=relaxed/relaxed; d=lists.infradead.org; s=bombadil.20210309; h=Sender:List-Subscribe:List-Help :List-Post:List-Archive:List-Unsubscribe:List-Id:Content-Type: Content-Transfer-Encoding:MIME-Version:References:In-Reply-To:Message-ID:Date :Subject:Cc:To:From:Reply-To:Content-ID:Content-Description:Resent-Date: Resent-From:Resent-Sender:Resent-To:Resent-Cc:Resent-Message-ID:List-Owner; bh=xat90K+7laktWlKWrEYj2NKDd4TQak32OHmU5MtaKmE=; b=Le6qpt1I+KZqF9j/05/dmxKUo1 PqFRDicHUGmg5bde393v+EtYuTyH0kjUJZnOBIgU0WVSsAuQ69Is1lgAAg0bpjQdkpmD76W8AvgqT PqQB9UgehBDlNDIvMYQq9Lc2VOC9rHS/7rewFPcVujbbo+qlGVlJxgut4fMbTO5AZJJ7C7RqWhmPc uF7XXXodVfAZNS15DYvNSqZKBB7s0dBigZJYeIhdV5kxlG9KdvpuIbQ4oG45iI8mlPMrkUNl2/A18 sh3cAWZtWZI2yDtWDpXQgqJHKrSAiK0NfX7E7TnZ5/ls9YxHnldME5CfL42HBUhwQeV1jc7/5Lddt UKJF9maA==; Received: from localhost ([::1] helo=bombadil.infradead.org) by bombadil.infradead.org with esmtp (Exim 4.97.1 #2 (Red Hat Linux)) id 1sigfb-00000008gfb-2Q8z; Mon, 26 Aug 2024 20:50:35 +0000 Received: from us-smtp-delivery-124.mimecast.com ([170.10.133.124]) by bombadil.infradead.org with esmtps (Exim 4.97.1 #2 (Red Hat Linux)) id 1sigZS-00000008eRa-2KQT for linux-arm-kernel@lists.infradead.org; Mon, 26 Aug 2024 20:44:16 +0000 DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=redhat.com; s=mimecast20190719; t=1724705053; h=from:from:reply-to:subject:subject:date:date:message-id:message-id: to:to:cc:cc:mime-version:mime-version:content-type:content-type: content-transfer-encoding:content-transfer-encoding: in-reply-to:in-reply-to:references:references; bh=xat90K+7laktWlKWrEYj2NKDd4TQak32OHmU5MtaKmE=; b=FuaBypCIXrFi10ERjQFXZYz/rimq+Umn9ysq94nh+m7z4OCjzc1GUgK9YqfQlNl6i/yBF9 HfmlnHn5nDlgnM4CnmiwLxiLYleQCsbctDlBdFufREvIFdVPQl1pI37xQhwDNzgUFsFDWq bpBleLqR9rYZ3CWjsisMipFIgN5w2Pw= Received: from mail-yw1-f198.google.com (mail-yw1-f198.google.com [209.85.128.198]) by relay.mimecast.com with ESMTP with STARTTLS (version=TLSv1.3, cipher=TLS_AES_256_GCM_SHA384) id us-mta-468-KyL1SZMvOy6rM9JY8vyW5w-1; Mon, 26 Aug 2024 16:44:11 -0400 X-MC-Unique: KyL1SZMvOy6rM9JY8vyW5w-1 Received: by mail-yw1-f198.google.com with SMTP id 00721157ae682-6adcb88da08so100203997b3.3 for ; Mon, 26 Aug 2024 13:44:11 -0700 (PDT) X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20230601; t=1724705051; x=1725309851; h=content-transfer-encoding:mime-version:references:in-reply-to :message-id:date:subject:cc:to:from:x-gm-message-state:from:to:cc :subject:date:message-id:reply-to; bh=xat90K+7laktWlKWrEYj2NKDd4TQak32OHmU5MtaKmE=; b=kAB600Kcfj66z6mtR3wAOD+R7iplHYygYgixiQlu41iAQNqc/07dlWHcpIbqM3Qb97 zOPkiZO6rhdtw58jVRH5ohU10fyPYmPhyc11pnhCzG+a4pQ7vMHihDPaBD0aMOHOA0ml L/Zb+t244wkNmz21+gQKikyTykHsrvm/POc2OdeR5xA/jCZCFPxNiVKVW3116rbQJcPX lAiPFObKFZ40Bd2OSlnq3rzzBKJ9AUryT5zpDm8ern1k3OlBIyAEcrE2nDCa4bzlpNvf j44HSXt6eotUZiH30CW5XTa7MAfUCMMaRY8gYTrxfCwLAOeOHy9AbqCKKRgtScMNWRLN HH7w== X-Forwarded-Encrypted: i=1; AJvYcCX9nq+c+9tEisnUJvsChVE1y7nSuVVWfrosw5Lan01Qv7AWJFIv5S6CmhDzwyxq5wPnMAuspHm2S4V/UWBCAgJE@lists.infradead.org X-Gm-Message-State: AOJu0YxopSIHZoLD8gCDLi+K/lp7Gyd6ECdfwy+Krcf+7pk7unb429DX BgOLjfilkTZ5FcEIpjpasEM4iuAtBEOjmawUFsZa7QTMLditKJWqzIcigfD2kwQwzB2eVvZwdbK YfL5jrNgg3jUvPbL+alV2c2wefcCDk/+YO9dR9y/Haj+Zsy1af+6ZL1UlRIHOUWBwL2+Ew0Ej X-Received: by 2002:a05:690c:380b:b0:6c1:2b6d:1964 with SMTP id 00721157ae682-6c62906557cmr136660677b3.38.1724705050751; Mon, 26 Aug 2024 13:44:10 -0700 (PDT) X-Google-Smtp-Source: AGHT+IGVQdSmnkARvKpk+I4g97hETTq/PaNGcO/OaJoau0YQSMQm37evWRXkCU2N5l4bob/h1UXyLw== X-Received: by 2002:a05:690c:380b:b0:6c1:2b6d:1964 with SMTP id 00721157ae682-6c62906557cmr136660497b3.38.1724705050419; Mon, 26 Aug 2024 13:44:10 -0700 (PDT) Received: from x1n.redhat.com (pool-99-254-121-117.cpe.net.cable.rogers.com. [99.254.121.117]) by smtp.gmail.com with ESMTPSA id af79cd13be357-7a67f3fd6c1sm491055185a.121.2024.08.26.13.44.08 (version=TLS1_3 cipher=TLS_AES_256_GCM_SHA384 bits=256/256); Mon, 26 Aug 2024 13:44:09 -0700 (PDT) From: Peter Xu To: linux-kernel@vger.kernel.org, linux-mm@kvack.org Cc: Gavin Shan , Catalin Marinas , x86@kernel.org, Ingo Molnar , Andrew Morton , Paolo Bonzini , Dave Hansen , Thomas Gleixner , Alistair Popple , kvm@vger.kernel.org, linux-arm-kernel@lists.infradead.org, Sean Christopherson , peterx@redhat.com, Oscar Salvador , Jason Gunthorpe , Borislav Petkov , Zi Yan , Axel Rasmussen , David Hildenbrand , Yan Zhao , Will Deacon , Kefeng Wang , Alex Williamson Subject: [PATCH v2 07/19] mm/fork: Accept huge pfnmap entries Date: Mon, 26 Aug 2024 16:43:41 -0400 Message-ID: <20240826204353.2228736-8-peterx@redhat.com> X-Mailer: git-send-email 2.45.0 In-Reply-To: <20240826204353.2228736-1-peterx@redhat.com> References: <20240826204353.2228736-1-peterx@redhat.com> MIME-Version: 1.0 X-Mimecast-Spam-Score: 0 X-Mimecast-Originator: redhat.com X-CRM114-Version: 20100106-BlameMichelson ( TRE 0.8.0 (BSD) ) MR-646709E3 X-CRM114-CacheID: sfid-20240826_134414_744675_74E37A6F X-CRM114-Status: GOOD ( 17.17 ) X-BeenThere: linux-arm-kernel@lists.infradead.org X-Mailman-Version: 2.1.34 Precedence: list List-Id: List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Sender: "linux-arm-kernel" Errors-To: linux-arm-kernel-bounces+linux-arm-kernel=archiver.kernel.org@lists.infradead.org Teach the fork code to properly copy pfnmaps for pmd/pud levels. Pud is much easier, the write bit needs to be persisted though for writable and shared pud mappings like PFNMAP ones, otherwise a follow up write in either parent or child process will trigger a write fault. Do the same for pmd level. Signed-off-by: Peter Xu --- mm/huge_memory.c | 29 ++++++++++++++++++++++++++--- 1 file changed, 26 insertions(+), 3 deletions(-) diff --git a/mm/huge_memory.c b/mm/huge_memory.c index e2c314f631f3..15418ffdd377 100644 --- a/mm/huge_memory.c +++ b/mm/huge_memory.c @@ -1559,6 +1559,24 @@ int copy_huge_pmd(struct mm_struct *dst_mm, struct mm_struct *src_mm, pgtable_t pgtable = NULL; int ret = -ENOMEM; + pmd = pmdp_get_lockless(src_pmd); + if (unlikely(pmd_special(pmd))) { + dst_ptl = pmd_lock(dst_mm, dst_pmd); + src_ptl = pmd_lockptr(src_mm, src_pmd); + spin_lock_nested(src_ptl, SINGLE_DEPTH_NESTING); + /* + * No need to recheck the pmd, it can't change with write + * mmap lock held here. + * + * Meanwhile, making sure it's not a CoW VMA with writable + * mapping, otherwise it means either the anon page wrongly + * applied special bit, or we made the PRIVATE mapping be + * able to wrongly write to the backend MMIO. + */ + VM_WARN_ON_ONCE(is_cow_mapping(src_vma->vm_flags) && pmd_write(pmd)); + goto set_pmd; + } + /* Skip if can be re-fill on fault */ if (!vma_is_anonymous(dst_vma)) return 0; @@ -1640,7 +1658,9 @@ int copy_huge_pmd(struct mm_struct *dst_mm, struct mm_struct *src_mm, pmdp_set_wrprotect(src_mm, addr, src_pmd); if (!userfaultfd_wp(dst_vma)) pmd = pmd_clear_uffd_wp(pmd); - pmd = pmd_mkold(pmd_wrprotect(pmd)); + pmd = pmd_wrprotect(pmd); +set_pmd: + pmd = pmd_mkold(pmd); set_pmd_at(dst_mm, addr, dst_pmd, pmd); ret = 0; @@ -1686,8 +1706,11 @@ int copy_huge_pud(struct mm_struct *dst_mm, struct mm_struct *src_mm, * TODO: once we support anonymous pages, use * folio_try_dup_anon_rmap_*() and split if duplicating fails. */ - pudp_set_wrprotect(src_mm, addr, src_pud); - pud = pud_mkold(pud_wrprotect(pud)); + if (is_cow_mapping(vma->vm_flags) && pud_write(pud)) { + pudp_set_wrprotect(src_mm, addr, src_pud); + pud = pud_wrprotect(pud); + } + pud = pud_mkold(pud); set_pud_at(dst_mm, addr, dst_pud, pud); ret = 0; From patchwork Mon Aug 26 20:43:42 2024 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Peter Xu X-Patchwork-Id: 13778442 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from bombadil.infradead.org (bombadil.infradead.org [198.137.202.133]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (No client certificate requested) by smtp.lore.kernel.org (Postfix) with ESMTPS id 41E00C5472C for ; Mon, 26 Aug 2024 20:51:33 +0000 (UTC) DKIM-Signature: v=1; a=rsa-sha256; q=dns/txt; c=relaxed/relaxed; d=lists.infradead.org; s=bombadil.20210309; h=Sender:List-Subscribe:List-Help :List-Post:List-Archive:List-Unsubscribe:List-Id:Content-Type: Content-Transfer-Encoding:MIME-Version:References:In-Reply-To:Message-ID:Date :Subject:Cc:To:From:Reply-To:Content-ID:Content-Description:Resent-Date: Resent-From:Resent-Sender:Resent-To:Resent-Cc:Resent-Message-ID:List-Owner; bh=vK1PMhUcFVpQCvjQRaUqQOYBcILrsdr8yUlCuFWve7Q=; b=sPwVZky2cF1H+B+ZmZzVLgSa7z K0TVXAJZx0nuYhFDlCRuF4A8hZN0oHgTb9Zbu1l8wx3jBYRvniygKe7x5rNisEHbHZsFWQr+JqWPc twj1G5kW57EJpgac8mRjJAbM4GQPamG3EcPVEKn0h8krrRGf3OFxpW7BtboQ4Cpyne4y+CFmIypMR 66HPAVo3YNYx8ZSgD2b2LkzjSESWw2zVEW96H6UNwbkqppCMUsk6dn3oY4+VBCYLZgBI7G/9fJ8LC WFVvkqog0ivxw0FID1NohHYhypz0hTMMinRtSwYXKNseYopKyD3ErMNcOO7SHcf3ZiWLXhEcPYJgy 7iZuwBkw==; Received: from localhost ([::1] helo=bombadil.infradead.org) by bombadil.infradead.org with esmtp (Exim 4.97.1 #2 (Red Hat Linux)) id 1siggM-00000008grT-24zq; Mon, 26 Aug 2024 20:51:22 +0000 Received: from us-smtp-delivery-124.mimecast.com ([170.10.133.124]) by bombadil.infradead.org with esmtps (Exim 4.97.1 #2 (Red Hat Linux)) id 1sigZT-00000008eSO-2QWW for linux-arm-kernel@lists.infradead.org; Mon, 26 Aug 2024 20:44:17 +0000 DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=redhat.com; s=mimecast20190719; t=1724705054; h=from:from:reply-to:subject:subject:date:date:message-id:message-id: to:to:cc:cc:mime-version:mime-version:content-type:content-type: content-transfer-encoding:content-transfer-encoding: in-reply-to:in-reply-to:references:references; bh=vK1PMhUcFVpQCvjQRaUqQOYBcILrsdr8yUlCuFWve7Q=; b=hvw1mw4GDSi7fHiyMKnX+ltRFECaI7tOq50d/tKbjRUmFMS/WHeJDbUZOTCviecD8wJmvX 5ReIfTKW6e/oVMD3A0utpYYgJkWXX8MVfDr/gi6QKSe0E2CqSfaPDletRen8MbtX6H1bFb Iqj4Nvk13vd24yNK2pJWiQUDHVvo2yE= Received: from mail-yw1-f199.google.com (mail-yw1-f199.google.com [209.85.128.199]) by relay.mimecast.com with ESMTP with STARTTLS (version=TLSv1.3, cipher=TLS_AES_256_GCM_SHA384) id us-mta-582-OBQhkkLpOmOIn5PCjSCpWQ-1; Mon, 26 Aug 2024 16:44:13 -0400 X-MC-Unique: OBQhkkLpOmOIn5PCjSCpWQ-1 Received: by mail-yw1-f199.google.com with SMTP id 00721157ae682-6c3982a0c65so90371097b3.1 for ; Mon, 26 Aug 2024 13:44:13 -0700 (PDT) X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20230601; t=1724705052; x=1725309852; h=content-transfer-encoding:mime-version:references:in-reply-to :message-id:date:subject:cc:to:from:x-gm-message-state:from:to:cc :subject:date:message-id:reply-to; bh=vK1PMhUcFVpQCvjQRaUqQOYBcILrsdr8yUlCuFWve7Q=; b=AbghNWRdTwNaC6iYzriOum/9CsxJolSRbKEGUjFDmTCfmVh4f3vP1lbDbRZACJtUWn wcGTsZDcqpy3gC2Zbr6SLTzpR4K5UTRaQQtFszErQm+tmf5VcF0TLJVOf8Vwjxk6UZyX az0hU8lIdCJqVdc9DLHCZ+QxX9Q5S7Go0fN3wDHIJD9cZPtvRn2o6z5+De6FovrpS50w TmVyYAmd5s2gQyMVQOCwrnkJBc22xCFCqlYHh10ELXbUrDYBharuYk3k6HD48LDV5Yqg vB8/dEGoq6pXlpE3sYrc540ytbd3lhx4OWqnki2KqdMB6hLLge2RaFX6r4UL9cy8WYAN GO0Q== X-Forwarded-Encrypted: i=1; AJvYcCWNfrxEb7OPsygSu+6BguYCJFYNQfuYYrOxld0xGrCh2fOax6LBOeUlnmfK4k49siedMqCYtr0InY/wO85mkRiY@lists.infradead.org X-Gm-Message-State: AOJu0Yx/WdpqmZ5cpOqUCSn8Qr91CgXGeKBtKuyLC/zjM4bw8GVXbm0P vdXmY2M5rkOBW0wE7VUSPzp1n3UiX0HANzsl89VgeKYVw6yFfaF+4GvzDCCiDJQU/3xI7nDNnXk GhvKKK9iolp5H0oOWqddZOUjsTvXOuG78utb+6kbuioWn2g7mTx5Y7v+uuAg6tvj/DC/eqmaF X-Received: by 2002:a05:690c:60c2:b0:6bd:fdd3:e170 with SMTP id 00721157ae682-6c624dc7042mr142338827b3.10.1724705052641; Mon, 26 Aug 2024 13:44:12 -0700 (PDT) X-Google-Smtp-Source: AGHT+IEDLUzP2hp4LBST6IRuRTdz5MRV8z70t30cnU5hPcRwmZ021NQU+FxAT+ZUF1I58Qs4qkGJLg== X-Received: by 2002:a05:690c:60c2:b0:6bd:fdd3:e170 with SMTP id 00721157ae682-6c624dc7042mr142338617b3.10.1724705052349; Mon, 26 Aug 2024 13:44:12 -0700 (PDT) Received: from x1n.redhat.com (pool-99-254-121-117.cpe.net.cable.rogers.com. [99.254.121.117]) by smtp.gmail.com with ESMTPSA id af79cd13be357-7a67f3fd6c1sm491055185a.121.2024.08.26.13.44.10 (version=TLS1_3 cipher=TLS_AES_256_GCM_SHA384 bits=256/256); Mon, 26 Aug 2024 13:44:11 -0700 (PDT) From: Peter Xu To: linux-kernel@vger.kernel.org, linux-mm@kvack.org Cc: Gavin Shan , Catalin Marinas , x86@kernel.org, Ingo Molnar , Andrew Morton , Paolo Bonzini , Dave Hansen , Thomas Gleixner , Alistair Popple , kvm@vger.kernel.org, linux-arm-kernel@lists.infradead.org, Sean Christopherson , peterx@redhat.com, Oscar Salvador , Jason Gunthorpe , Borislav Petkov , Zi Yan , Axel Rasmussen , David Hildenbrand , Yan Zhao , Will Deacon , Kefeng Wang , Alex Williamson Subject: [PATCH v2 08/19] mm: Always define pxx_pgprot() Date: Mon, 26 Aug 2024 16:43:42 -0400 Message-ID: <20240826204353.2228736-9-peterx@redhat.com> X-Mailer: git-send-email 2.45.0 In-Reply-To: <20240826204353.2228736-1-peterx@redhat.com> References: <20240826204353.2228736-1-peterx@redhat.com> MIME-Version: 1.0 X-Mimecast-Spam-Score: 0 X-Mimecast-Originator: redhat.com X-CRM114-Version: 20100106-BlameMichelson ( TRE 0.8.0 (BSD) ) MR-646709E3 X-CRM114-CacheID: sfid-20240826_134415_741956_D4AD59E7 X-CRM114-Status: GOOD ( 14.13 ) X-BeenThere: linux-arm-kernel@lists.infradead.org X-Mailman-Version: 2.1.34 Precedence: list List-Id: List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Sender: "linux-arm-kernel" Errors-To: linux-arm-kernel-bounces+linux-arm-kernel=archiver.kernel.org@lists.infradead.org There're: - 8 archs (arc, arm64, include, mips, powerpc, s390, sh, x86) that support pte_pgprot(). - 2 archs (x86, sparc) that support pmd_pgprot(). - 1 arch (x86) that support pud_pgprot(). Always define them to be used in generic code, and then we don't need to fiddle with "#ifdef"s when doing so. Reviewed-by: Jason Gunthorpe Signed-off-by: Peter Xu --- arch/arm64/include/asm/pgtable.h | 1 + arch/powerpc/include/asm/pgtable.h | 1 + arch/s390/include/asm/pgtable.h | 1 + arch/sparc/include/asm/pgtable_64.h | 1 + include/linux/pgtable.h | 12 ++++++++++++ 5 files changed, 16 insertions(+) diff --git a/arch/arm64/include/asm/pgtable.h b/arch/arm64/include/asm/pgtable.h index 7a4f5604be3f..b78cc4a6758b 100644 --- a/arch/arm64/include/asm/pgtable.h +++ b/arch/arm64/include/asm/pgtable.h @@ -384,6 +384,7 @@ static inline void __sync_cache_and_tags(pte_t pte, unsigned int nr_pages) /* * Select all bits except the pfn */ +#define pte_pgprot pte_pgprot static inline pgprot_t pte_pgprot(pte_t pte) { unsigned long pfn = pte_pfn(pte); diff --git a/arch/powerpc/include/asm/pgtable.h b/arch/powerpc/include/asm/pgtable.h index 264a6c09517a..2f72ad885332 100644 --- a/arch/powerpc/include/asm/pgtable.h +++ b/arch/powerpc/include/asm/pgtable.h @@ -65,6 +65,7 @@ static inline unsigned long pte_pfn(pte_t pte) /* * Select all bits except the pfn */ +#define pte_pgprot pte_pgprot static inline pgprot_t pte_pgprot(pte_t pte) { unsigned long pte_flags; diff --git a/arch/s390/include/asm/pgtable.h b/arch/s390/include/asm/pgtable.h index 3fa280d0672a..0ffbaf741955 100644 --- a/arch/s390/include/asm/pgtable.h +++ b/arch/s390/include/asm/pgtable.h @@ -955,6 +955,7 @@ static inline int pte_unused(pte_t pte) * young/old accounting is not supported, i.e _PAGE_PROTECT and _PAGE_INVALID * must not be set. */ +#define pte_pgprot pte_pgprot static inline pgprot_t pte_pgprot(pte_t pte) { unsigned long pte_flags = pte_val(pte) & _PAGE_CHG_MASK; diff --git a/arch/sparc/include/asm/pgtable_64.h b/arch/sparc/include/asm/pgtable_64.h index 3fe429d73a65..2b7f358762c1 100644 --- a/arch/sparc/include/asm/pgtable_64.h +++ b/arch/sparc/include/asm/pgtable_64.h @@ -783,6 +783,7 @@ static inline pmd_t pmd_mkwrite_novma(pmd_t pmd) return __pmd(pte_val(pte)); } +#define pmd_pgprot pmd_pgprot static inline pgprot_t pmd_pgprot(pmd_t entry) { unsigned long val = pmd_val(entry); diff --git a/include/linux/pgtable.h b/include/linux/pgtable.h index 780f3b439d98..e8b2ac6bd2ae 100644 --- a/include/linux/pgtable.h +++ b/include/linux/pgtable.h @@ -1956,6 +1956,18 @@ typedef unsigned int pgtbl_mod_mask; #define MAX_PTRS_PER_P4D PTRS_PER_P4D #endif +#ifndef pte_pgprot +#define pte_pgprot(x) ((pgprot_t) {0}) +#endif + +#ifndef pmd_pgprot +#define pmd_pgprot(x) ((pgprot_t) {0}) +#endif + +#ifndef pud_pgprot +#define pud_pgprot(x) ((pgprot_t) {0}) +#endif + /* description of effects of mapping type and prot in current implementation. * this is due to the limited x86 page protection hardware. The expected * behavior is in parens: From patchwork Mon Aug 26 20:43:43 2024 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Peter Xu X-Patchwork-Id: 13778443 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from bombadil.infradead.org (bombadil.infradead.org [198.137.202.133]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (No client certificate requested) by smtp.lore.kernel.org (Postfix) with ESMTPS id CA061C5321D for ; Mon, 26 Aug 2024 20:52:17 +0000 (UTC) DKIM-Signature: v=1; a=rsa-sha256; q=dns/txt; c=relaxed/relaxed; d=lists.infradead.org; s=bombadil.20210309; h=Sender:List-Subscribe:List-Help :List-Post:List-Archive:List-Unsubscribe:List-Id:Content-Type: Content-Transfer-Encoding:MIME-Version:References:In-Reply-To:Message-ID:Date :Subject:Cc:To:From:Reply-To:Content-ID:Content-Description:Resent-Date: Resent-From:Resent-Sender:Resent-To:Resent-Cc:Resent-Message-ID:List-Owner; bh=XzWn17lHZnGD5Tr66iu6y8/zqhTRy72agTNrtdvxzwE=; b=vYKm3d2g9/uHQizB5XpMfwpr9q aylsywNXtM+YWsRb0UBZABxdLPP5iqTfWfFE3zKLoLpl0SR1y8ZyOYjnjcFaNiUgjL5dldvWKyr2A mFoq2LL6gHKpLPAmdEr6jEL0sLyR9pf9iViKe4v3hCsRamQdr9MVnQ9+lTykJybXk1Juyj8BsjdrD WF6wlXuAxwk6jQ+80HGzCW/ZA4qVv8gnBYh6sp56c9N79PRG9A4Ke0wFndrIgy3JfzKx/n7z2twYV cdShuzV902+UUPjwqOqH1TLYLM6G08WuE79puvcTqzlq7Bo3rgbxgulUjqlwT20GGS85dWkVcc0IK J1Oc8JAQ==; Received: from localhost ([::1] helo=bombadil.infradead.org) by bombadil.infradead.org with esmtp (Exim 4.97.1 #2 (Red Hat Linux)) id 1sigh6-00000008gy0-1v0T; Mon, 26 Aug 2024 20:52:08 +0000 Received: from us-smtp-delivery-124.mimecast.com ([170.10.129.124]) by bombadil.infradead.org with esmtps (Exim 4.97.1 #2 (Red Hat Linux)) id 1sigZV-00000008eTm-3B76 for linux-arm-kernel@lists.infradead.org; Mon, 26 Aug 2024 20:44:19 +0000 DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=redhat.com; s=mimecast20190719; t=1724705056; h=from:from:reply-to:subject:subject:date:date:message-id:message-id: to:to:cc:cc:mime-version:mime-version:content-type:content-type: content-transfer-encoding:content-transfer-encoding: in-reply-to:in-reply-to:references:references; bh=XzWn17lHZnGD5Tr66iu6y8/zqhTRy72agTNrtdvxzwE=; b=GB51IwsUFGz9QiMYb1aeCVArQshtX9/BPqs22vaRFMbgUPbZna0dbgweERVBax6aL419Md lsk2E8/+C62PpHINsxJHpsHa7XVHdtx3ZQR7ceynlKMx7H+BPLTi1ab8FvLjNePv/RzKwv +wsyS6nN6mCeHLCbYt7eNpBfge5Til4= Received: from mail-qk1-f199.google.com (mail-qk1-f199.google.com [209.85.222.199]) by relay.mimecast.com with ESMTP with STARTTLS (version=TLSv1.3, cipher=TLS_AES_256_GCM_SHA384) id us-mta-507-LQXsegmlNfOm8sS4AW-ltw-1; Mon, 26 Aug 2024 16:44:15 -0400 X-MC-Unique: LQXsegmlNfOm8sS4AW-ltw-1 Received: by mail-qk1-f199.google.com with SMTP id af79cd13be357-7a20ce27a6fso629050885a.0 for ; Mon, 26 Aug 2024 13:44:15 -0700 (PDT) X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20230601; t=1724705055; x=1725309855; h=content-transfer-encoding:mime-version:references:in-reply-to :message-id:date:subject:cc:to:from:x-gm-message-state:from:to:cc :subject:date:message-id:reply-to; bh=XzWn17lHZnGD5Tr66iu6y8/zqhTRy72agTNrtdvxzwE=; b=TiNwSblVD57lj9UzXeUoiFHoh/eXkvyEFMg5jO07da2pcYRgGKYfkS2DJbWbXjmv46 8phX8py+ET7Puyv4TfkhJ5GCWZWROVLIa0WAFTErL6h/D29AX3G83zji8EzOjctLSFrT oLLs6vV3NqoZ43LoW+is+deAcqgjr3NevUv3yryEp5n6nszTDVXxhgfdcV8de03Vjt3c w9aWdMzhkyRJuX0ihPSHBpW6eZ6GPxtZ+SlNLPzwrKfhrcMN3+meqyK+TZ0v7IuH5g3u j+wf+XKFXArjBMyqxtHSO+e2w2Ulv0eXBCiStJJNXDecbWCFyUMTo4s5jYvVp15ENcOY tgjg== X-Forwarded-Encrypted: i=1; AJvYcCX1UocJqR3/sddxOXRNjwpjrScjPwlHlIlKWYDrvpccgq9W2UKJNufeD+KfZ5vaTxQpClvsuGgg053sn7bpab9d@lists.infradead.org X-Gm-Message-State: AOJu0YxwTICCzOb4MHVa/gpKp48DPaURVkwXcjrTXfYxc1eMNC+V6YWa hQhyfzPjVo6zSYtFpU1VDI9kingZNRjgfiCPjOzI4WBTjqzlDPdDWYLthcA0tBjKZM2hcOp91+m KljjNyNqIwkpvfhI+HlHLYEX6WXPseV4hmJajLTSJGurwZVkonTmdGZ/lYtMfnDlrortKbDLp X-Received: by 2002:a05:620a:408d:b0:7a1:e93c:cd04 with SMTP id af79cd13be357-7a6896e0e1cmr1465306385a.9.1724705054684; Mon, 26 Aug 2024 13:44:14 -0700 (PDT) X-Google-Smtp-Source: AGHT+IH4kwi4qTohFBFu7kWFKQGbZI396sZSJGmlCvwBl5XfpN0sU8TboRNIhnEyWXLKkBzi4MYP4w== X-Received: by 2002:a05:620a:408d:b0:7a1:e93c:cd04 with SMTP id af79cd13be357-7a6896e0e1cmr1465302885a.9.1724705054299; Mon, 26 Aug 2024 13:44:14 -0700 (PDT) Received: from x1n.redhat.com (pool-99-254-121-117.cpe.net.cable.rogers.com. [99.254.121.117]) by smtp.gmail.com with ESMTPSA id af79cd13be357-7a67f3fd6c1sm491055185a.121.2024.08.26.13.44.12 (version=TLS1_3 cipher=TLS_AES_256_GCM_SHA384 bits=256/256); Mon, 26 Aug 2024 13:44:13 -0700 (PDT) From: Peter Xu To: linux-kernel@vger.kernel.org, linux-mm@kvack.org Cc: Gavin Shan , Catalin Marinas , x86@kernel.org, Ingo Molnar , Andrew Morton , Paolo Bonzini , Dave Hansen , Thomas Gleixner , Alistair Popple , kvm@vger.kernel.org, linux-arm-kernel@lists.infradead.org, Sean Christopherson , peterx@redhat.com, Oscar Salvador , Jason Gunthorpe , Borislav Petkov , Zi Yan , Axel Rasmussen , David Hildenbrand , Yan Zhao , Will Deacon , Kefeng Wang , Alex Williamson Subject: [PATCH v2 09/19] mm: New follow_pfnmap API Date: Mon, 26 Aug 2024 16:43:43 -0400 Message-ID: <20240826204353.2228736-10-peterx@redhat.com> X-Mailer: git-send-email 2.45.0 In-Reply-To: <20240826204353.2228736-1-peterx@redhat.com> References: <20240826204353.2228736-1-peterx@redhat.com> MIME-Version: 1.0 X-Mimecast-Spam-Score: 0 X-Mimecast-Originator: redhat.com X-CRM114-Version: 20100106-BlameMichelson ( TRE 0.8.0 (BSD) ) MR-646709E3 X-CRM114-CacheID: sfid-20240826_134417_912516_552BF126 X-CRM114-Status: GOOD ( 22.05 ) X-BeenThere: linux-arm-kernel@lists.infradead.org X-Mailman-Version: 2.1.34 Precedence: list List-Id: List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Sender: "linux-arm-kernel" Errors-To: linux-arm-kernel-bounces+linux-arm-kernel=archiver.kernel.org@lists.infradead.org Introduce a pair of APIs to follow pfn mappings to get entry information. It's very similar to what follow_pte() does before, but different in that it recognizes huge pfn mappings. Signed-off-by: Peter Xu --- include/linux/mm.h | 31 ++++++++++ mm/memory.c | 150 +++++++++++++++++++++++++++++++++++++++++++++ 2 files changed, 181 insertions(+) diff --git a/include/linux/mm.h b/include/linux/mm.h index d900f15b7650..161d496bfd18 100644 --- a/include/linux/mm.h +++ b/include/linux/mm.h @@ -2373,6 +2373,37 @@ int follow_pte(struct vm_area_struct *vma, unsigned long address, int generic_access_phys(struct vm_area_struct *vma, unsigned long addr, void *buf, int len, int write); +struct follow_pfnmap_args { + /** + * Inputs: + * @vma: Pointer to @vm_area_struct struct + * @address: the virtual address to walk + */ + struct vm_area_struct *vma; + unsigned long address; + /** + * Internals: + * + * The caller shouldn't touch any of these. + */ + spinlock_t *lock; + pte_t *ptep; + /** + * Outputs: + * + * @pfn: the PFN of the address + * @pgprot: the pgprot_t of the mapping + * @writable: whether the mapping is writable + * @special: whether the mapping is a special mapping (real PFN maps) + */ + unsigned long pfn; + pgprot_t pgprot; + bool writable; + bool special; +}; +int follow_pfnmap_start(struct follow_pfnmap_args *args); +void follow_pfnmap_end(struct follow_pfnmap_args *args); + extern void truncate_pagecache(struct inode *inode, loff_t new); extern void truncate_setsize(struct inode *inode, loff_t newsize); void pagecache_isize_extended(struct inode *inode, loff_t from, loff_t to); diff --git a/mm/memory.c b/mm/memory.c index 93c0c25433d0..0b136c398257 100644 --- a/mm/memory.c +++ b/mm/memory.c @@ -6173,6 +6173,156 @@ int follow_pte(struct vm_area_struct *vma, unsigned long address, } EXPORT_SYMBOL_GPL(follow_pte); +static inline void pfnmap_args_setup(struct follow_pfnmap_args *args, + spinlock_t *lock, pte_t *ptep, + pgprot_t pgprot, unsigned long pfn_base, + unsigned long addr_mask, bool writable, + bool special) +{ + args->lock = lock; + args->ptep = ptep; + args->pfn = pfn_base + ((args->address & ~addr_mask) >> PAGE_SHIFT); + args->pgprot = pgprot; + args->writable = writable; + args->special = special; +} + +static inline void pfnmap_lockdep_assert(struct vm_area_struct *vma) +{ +#ifdef CONFIG_LOCKDEP + struct address_space *mapping = vma->vm_file->f_mapping; + + if (mapping) + lockdep_assert(lockdep_is_held(&vma->vm_file->f_mapping->i_mmap_rwsem) || + lockdep_is_held(&vma->vm_mm->mmap_lock)); + else + lockdep_assert(lockdep_is_held(&vma->vm_mm->mmap_lock)); +#endif +} + +/** + * follow_pfnmap_start() - Look up a pfn mapping at a user virtual address + * @args: Pointer to struct @follow_pfnmap_args + * + * The caller needs to setup args->vma and args->address to point to the + * virtual address as the target of such lookup. On a successful return, + * the results will be put into other output fields. + * + * After the caller finished using the fields, the caller must invoke + * another follow_pfnmap_end() to proper releases the locks and resources + * of such look up request. + * + * During the start() and end() calls, the results in @args will be valid + * as proper locks will be held. After the end() is called, all the fields + * in @follow_pfnmap_args will be invalid to be further accessed. Further + * use of such information after end() may require proper synchronizations + * by the caller with page table updates, otherwise it can create a + * security bug. + * + * If the PTE maps a refcounted page, callers are responsible to protect + * against invalidation with MMU notifiers; otherwise access to the PFN at + * a later point in time can trigger use-after-free. + * + * Only IO mappings and raw PFN mappings are allowed. The mmap semaphore + * should be taken for read, and the mmap semaphore cannot be released + * before the end() is invoked. + * + * This function must not be used to modify PTE content. + * + * Return: zero on success, negative otherwise. + */ +int follow_pfnmap_start(struct follow_pfnmap_args *args) +{ + struct vm_area_struct *vma = args->vma; + unsigned long address = args->address; + struct mm_struct *mm = vma->vm_mm; + spinlock_t *lock; + pgd_t *pgdp; + p4d_t *p4dp, p4d; + pud_t *pudp, pud; + pmd_t *pmdp, pmd; + pte_t *ptep, pte; + + pfnmap_lockdep_assert(vma); + + if (unlikely(address < vma->vm_start || address >= vma->vm_end)) + goto out; + + if (!(vma->vm_flags & (VM_IO | VM_PFNMAP))) + goto out; +retry: + pgdp = pgd_offset(mm, address); + if (pgd_none(*pgdp) || unlikely(pgd_bad(*pgdp))) + goto out; + + p4dp = p4d_offset(pgdp, address); + p4d = READ_ONCE(*p4dp); + if (p4d_none(p4d) || unlikely(p4d_bad(p4d))) + goto out; + + pudp = pud_offset(p4dp, address); + pud = READ_ONCE(*pudp); + if (pud_none(pud)) + goto out; + if (pud_leaf(pud)) { + lock = pud_lock(mm, pudp); + if (!unlikely(pud_leaf(pud))) { + spin_unlock(lock); + goto retry; + } + pfnmap_args_setup(args, lock, NULL, pud_pgprot(pud), + pud_pfn(pud), PUD_MASK, pud_write(pud), + pud_special(pud)); + return 0; + } + + pmdp = pmd_offset(pudp, address); + pmd = pmdp_get_lockless(pmdp); + if (pmd_leaf(pmd)) { + lock = pmd_lock(mm, pmdp); + if (!unlikely(pmd_leaf(pmd))) { + spin_unlock(lock); + goto retry; + } + pfnmap_args_setup(args, lock, NULL, pmd_pgprot(pmd), + pmd_pfn(pmd), PMD_MASK, pmd_write(pmd), + pmd_special(pmd)); + return 0; + } + + ptep = pte_offset_map_lock(mm, pmdp, address, &lock); + if (!ptep) + goto out; + pte = ptep_get(ptep); + if (!pte_present(pte)) + goto unlock; + pfnmap_args_setup(args, lock, ptep, pte_pgprot(pte), + pte_pfn(pte), PAGE_MASK, pte_write(pte), + pte_special(pte)); + return 0; +unlock: + pte_unmap_unlock(ptep, lock); +out: + return -EINVAL; +} +EXPORT_SYMBOL_GPL(follow_pfnmap_start); + +/** + * follow_pfnmap_end(): End a follow_pfnmap_start() process + * @args: Pointer to struct @follow_pfnmap_args + * + * Must be used in pair of follow_pfnmap_start(). See the start() function + * above for more information. + */ +void follow_pfnmap_end(struct follow_pfnmap_args *args) +{ + if (args->lock) + spin_unlock(args->lock); + if (args->ptep) + pte_unmap(args->ptep); +} +EXPORT_SYMBOL_GPL(follow_pfnmap_end); + #ifdef CONFIG_HAVE_IOREMAP_PROT /** * generic_access_phys - generic implementation for iomem mmap access From patchwork Mon Aug 26 20:43:44 2024 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Peter Xu X-Patchwork-Id: 13778447 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from bombadil.infradead.org (bombadil.infradead.org [198.137.202.133]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (No client certificate requested) by smtp.lore.kernel.org (Postfix) with ESMTPS id A2BF0C5472C for ; Mon, 26 Aug 2024 20:53:08 +0000 (UTC) DKIM-Signature: v=1; a=rsa-sha256; q=dns/txt; c=relaxed/relaxed; d=lists.infradead.org; s=bombadil.20210309; h=Sender:List-Subscribe:List-Help :List-Post:List-Archive:List-Unsubscribe:List-Id:Content-Type: Content-Transfer-Encoding:MIME-Version:References:In-Reply-To:Message-ID:Date :Subject:Cc:To:From:Reply-To:Content-ID:Content-Description:Resent-Date: Resent-From:Resent-Sender:Resent-To:Resent-Cc:Resent-Message-ID:List-Owner; bh=RcTv8PG6WUMces5zPq/dQ8qXZuImd+6bX+nNmvbIpbY=; b=q8IKFch+bOw59/B+RZvZ6tlehl YJQa5hYyKE8ieDgeiN/pF7AvxgegUsOo0dS9cXlnzpqkwVWHodUATGKNbIE01mC85RU/g7JkFJObC EW8y0SjVwmwoTFgn/fZH9ia668JTr9qOi77hUoer6eOpbkqvhH+TzYLXPQz6W4XC4s0xDkC7vH8lt XpszVe6rz4AQwQPFbjLagZ8uCwjIYZSS/LXKkrTLYvNZlJLByf8ZRa+1OjT/gEiyL+AvZqMhjZZ2V b7qLolH0Ff9S3rQpKasP6jC56zFy5VbR4qEgB07Y3uIKukqZWrpMBoBB4g4KPR/XBkG2h1ufQOhSR SytLN8OA==; Received: from localhost ([::1] helo=bombadil.infradead.org) by bombadil.infradead.org with esmtp (Exim 4.97.1 #2 (Red Hat Linux)) id 1sight-00000008hBi-2m05; Mon, 26 Aug 2024 20:52:57 +0000 Received: from us-smtp-delivery-124.mimecast.com ([170.10.129.124]) by bombadil.infradead.org with esmtps (Exim 4.97.1 #2 (Red Hat Linux)) id 1sigZZ-00000008eWF-130i for linux-arm-kernel@lists.infradead.org; Mon, 26 Aug 2024 20:44:23 +0000 DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=redhat.com; s=mimecast20190719; t=1724705060; h=from:from:reply-to:subject:subject:date:date:message-id:message-id: to:to:cc:cc:mime-version:mime-version:content-type:content-type: content-transfer-encoding:content-transfer-encoding: in-reply-to:in-reply-to:references:references; bh=RcTv8PG6WUMces5zPq/dQ8qXZuImd+6bX+nNmvbIpbY=; b=aImJ/sPVFaMTyKxDJTQyeEr1p+PVDhpZF9SBROg3pY4qThG78FNjAD0tu0lW8pFSHDD+oQ fxrXyhvx1pvkg+MFb+4JefMaqimP2SFXFFQOPMkJ31qfrf/fxpQJhI10io3iRBopTom5+t kemLFZIV/XJlABZfAVJCIKyBVj4dOj4= Received: from mail-yw1-f199.google.com (mail-yw1-f199.google.com [209.85.128.199]) by relay.mimecast.com with ESMTP with STARTTLS (version=TLSv1.3, cipher=TLS_AES_256_GCM_SHA384) id us-mta-673-sL1TmLxSOEarDfsJdmEtsw-1; Mon, 26 Aug 2024 16:44:17 -0400 X-MC-Unique: sL1TmLxSOEarDfsJdmEtsw-1 Received: by mail-yw1-f199.google.com with SMTP id 00721157ae682-6b41e02c293so94041127b3.0 for ; Mon, 26 Aug 2024 13:44:17 -0700 (PDT) X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20230601; t=1724705056; x=1725309856; h=content-transfer-encoding:mime-version:references:in-reply-to :message-id:date:subject:cc:to:from:x-gm-message-state:from:to:cc :subject:date:message-id:reply-to; bh=RcTv8PG6WUMces5zPq/dQ8qXZuImd+6bX+nNmvbIpbY=; b=TGmYCKE5r+YLNUHKn31L3e/2NGWz/a67DDfOITEcFNqLpIZ68pJmzK50xY4wRKWU06 cyoORwjC1+ZdLxqVCGmTXn/VIKMgxtNolyelMoNXQncaN4OATcNrQd41AEzWBZupSuDS epNc5CqSy7Z2/QhGE4X4l7gPO9APvL89Xu5rPOSU5jBY51vOrzqop7E7khZAxM5jYNnz sIUj9dQBrJnAt8ELiwJVTb12iVhLYVSJVXBt4T8BL1Woby8AF9SH6x64BXCFqgEsXCke ++wLbeCUkEDgDuA2t9TyQE5PkxOJJUIcJCPG7JGKXIEOdTggNvaLq/s+S8hWtIEBcidR fiTg== X-Forwarded-Encrypted: i=1; AJvYcCUJUXgvNXTIosCSNySztr2+fhgIfU/SYb89aBTWlKJbe+GIs54WIzODt00ierM9Qcvslf5i9nSz/gWJ2amt+iq2@lists.infradead.org X-Gm-Message-State: AOJu0YzubN3Zx0IgJe+yZm95WFbCp8r94o1F4PabAH73PrPa3ODYgujp z3baZvJhAtREmSdVaWhMG6qB8vGuXMTuk6Z8vJ8pdz1IgjEb8hXYxYFQTfHICm3DeQ49z6YwctF O7eTXsWPgF5zVdhmC1KTskjnuQOZJovJobsqSjwLuJJwxD5msIfoBeM9tN/F4FyYEONyGvueh X-Received: by 2002:a05:690c:4246:b0:6b2:4154:e598 with SMTP id 00721157ae682-6c6262f43e4mr113946477b3.26.1724705056538; Mon, 26 Aug 2024 13:44:16 -0700 (PDT) X-Google-Smtp-Source: AGHT+IGCZvjuJnmUbtCZKJF68AHOtk4V9xJHAv9nFBXLLIOg3bv7tZNlMwBWnRJYIMuW8OX1o2s1rQ== X-Received: by 2002:a05:690c:4246:b0:6b2:4154:e598 with SMTP id 00721157ae682-6c6262f43e4mr113946177b3.26.1724705056224; Mon, 26 Aug 2024 13:44:16 -0700 (PDT) Received: from x1n.redhat.com (pool-99-254-121-117.cpe.net.cable.rogers.com. [99.254.121.117]) by smtp.gmail.com with ESMTPSA id af79cd13be357-7a67f3fd6c1sm491055185a.121.2024.08.26.13.44.14 (version=TLS1_3 cipher=TLS_AES_256_GCM_SHA384 bits=256/256); Mon, 26 Aug 2024 13:44:15 -0700 (PDT) From: Peter Xu To: linux-kernel@vger.kernel.org, linux-mm@kvack.org Cc: Gavin Shan , Catalin Marinas , x86@kernel.org, Ingo Molnar , Andrew Morton , Paolo Bonzini , Dave Hansen , Thomas Gleixner , Alistair Popple , kvm@vger.kernel.org, linux-arm-kernel@lists.infradead.org, Sean Christopherson , peterx@redhat.com, Oscar Salvador , Jason Gunthorpe , Borislav Petkov , Zi Yan , Axel Rasmussen , David Hildenbrand , Yan Zhao , Will Deacon , Kefeng Wang , Alex Williamson Subject: [PATCH v2 10/19] KVM: Use follow_pfnmap API Date: Mon, 26 Aug 2024 16:43:44 -0400 Message-ID: <20240826204353.2228736-11-peterx@redhat.com> X-Mailer: git-send-email 2.45.0 In-Reply-To: <20240826204353.2228736-1-peterx@redhat.com> References: <20240826204353.2228736-1-peterx@redhat.com> MIME-Version: 1.0 X-Mimecast-Spam-Score: 0 X-Mimecast-Originator: redhat.com X-CRM114-Version: 20100106-BlameMichelson ( TRE 0.8.0 (BSD) ) MR-646709E3 X-CRM114-CacheID: sfid-20240826_134421_396251_46BB5A76 X-CRM114-Status: GOOD ( 15.59 ) X-BeenThere: linux-arm-kernel@lists.infradead.org X-Mailman-Version: 2.1.34 Precedence: list List-Id: List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Sender: "linux-arm-kernel" Errors-To: linux-arm-kernel-bounces+linux-arm-kernel=archiver.kernel.org@lists.infradead.org Use the new pfnmap API to allow huge MMIO mappings for VMs. The rest work is done perfectly on the other side (host_pfn_mapping_level()). Cc: Paolo Bonzini Cc: Sean Christopherson Signed-off-by: Peter Xu --- virt/kvm/kvm_main.c | 19 +++++++------------ 1 file changed, 7 insertions(+), 12 deletions(-) diff --git a/virt/kvm/kvm_main.c b/virt/kvm/kvm_main.c index cb2b78e92910..f416d5e3f9c0 100644 --- a/virt/kvm/kvm_main.c +++ b/virt/kvm/kvm_main.c @@ -2860,13 +2860,11 @@ static int hva_to_pfn_remapped(struct vm_area_struct *vma, unsigned long addr, bool write_fault, bool *writable, kvm_pfn_t *p_pfn) { + struct follow_pfnmap_args args = { .vma = vma, .address = addr }; kvm_pfn_t pfn; - pte_t *ptep; - pte_t pte; - spinlock_t *ptl; int r; - r = follow_pte(vma, addr, &ptep, &ptl); + r = follow_pfnmap_start(&args); if (r) { /* * get_user_pages fails for VM_IO and VM_PFNMAP vmas and does @@ -2881,21 +2879,19 @@ static int hva_to_pfn_remapped(struct vm_area_struct *vma, if (r) return r; - r = follow_pte(vma, addr, &ptep, &ptl); + r = follow_pfnmap_start(&args); if (r) return r; } - pte = ptep_get(ptep); - - if (write_fault && !pte_write(pte)) { + if (write_fault && !args.writable) { pfn = KVM_PFN_ERR_RO_FAULT; goto out; } if (writable) - *writable = pte_write(pte); - pfn = pte_pfn(pte); + *writable = args.writable; + pfn = args.pfn; /* * Get a reference here because callers of *hva_to_pfn* and @@ -2916,9 +2912,8 @@ static int hva_to_pfn_remapped(struct vm_area_struct *vma, */ if (!kvm_try_get_pfn(pfn)) r = -EFAULT; - out: - pte_unmap_unlock(ptep, ptl); + follow_pfnmap_end(&args); *p_pfn = pfn; return r; From patchwork Mon Aug 26 20:43:45 2024 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Peter Xu X-Patchwork-Id: 13778448 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from bombadil.infradead.org (bombadil.infradead.org [198.137.202.133]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (No client certificate requested) by smtp.lore.kernel.org (Postfix) with ESMTPS id 13B19C5472C for ; Mon, 26 Aug 2024 20:53:55 +0000 (UTC) DKIM-Signature: v=1; a=rsa-sha256; q=dns/txt; c=relaxed/relaxed; d=lists.infradead.org; s=bombadil.20210309; h=Sender:List-Subscribe:List-Help :List-Post:List-Archive:List-Unsubscribe:List-Id:Content-Type: Content-Transfer-Encoding:MIME-Version:References:In-Reply-To:Message-ID:Date :Subject:Cc:To:From:Reply-To:Content-ID:Content-Description:Resent-Date: Resent-From:Resent-Sender:Resent-To:Resent-Cc:Resent-Message-ID:List-Owner; bh=kaY3EfeA+k09YWgS+MfGM8qkjlv58srLyvVau/9jXnU=; b=0RURGm0CoO0sDREijYXIH/L8QP pSBL/Y6Saq8XQoa9UPV3Zf9/afaJQwQE2R0UxKJ6i6/bpwyz2eC5f8C0NHuNQ4RN9y8zw02xK0/yu ID0hJKHcDg/Z2eSzl9sdewWuFQ7EZi9dbIqS3GOlGFQYb159alcgaGjOcs7DRcLk7hM1UgDVHSdGO h6O4ShjPGknJr1s6hJ4lx/wPooZ3X/4/uTwZglmLTk5PJYyRsZhsJyXVSc0qtRtFK/Ph29zanchpl h6ovzMw6BkkfkqPq/tkhqL1BN58787O3w/KBqSFfMUfvzG0GTcLG+Alh4ppdi/SvuVCuLh7WGCJWa hMq5g+Pg==; Received: from localhost ([::1] helo=bombadil.infradead.org) by bombadil.infradead.org with esmtp (Exim 4.97.1 #2 (Red Hat Linux)) id 1sigif-00000008hMe-3BIY; Mon, 26 Aug 2024 20:53:45 +0000 Received: from us-smtp-delivery-124.mimecast.com ([170.10.133.124]) by bombadil.infradead.org with esmtps (Exim 4.97.1 #2 (Red Hat Linux)) id 1sigZa-00000008eWg-0oAk for linux-arm-kernel@lists.infradead.org; Mon, 26 Aug 2024 20:44:24 +0000 DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=redhat.com; s=mimecast20190719; t=1724705061; h=from:from:reply-to:subject:subject:date:date:message-id:message-id: to:to:cc:cc:mime-version:mime-version:content-type:content-type: content-transfer-encoding:content-transfer-encoding: in-reply-to:in-reply-to:references:references; bh=kaY3EfeA+k09YWgS+MfGM8qkjlv58srLyvVau/9jXnU=; b=J8eqF3l5TjJLjwTcZd/Ijl+hgiFskRSTSPxTeafDfoRjyRPTXsvUVxl+mkbsgRCnwTAR57 mOLdcYe3xHhChX/b6Jwv/wU8KEmwnmv1zrAuMEVG06m24BYvoBMZaNOc/dZfbuJ6UBX1Sb YBXUUqstBndX9WmK5YTpOC7tkbWfqXA= Received: from mail-oo1-f71.google.com (mail-oo1-f71.google.com [209.85.161.71]) by relay.mimecast.com with ESMTP with STARTTLS (version=TLSv1.3, cipher=TLS_AES_256_GCM_SHA384) id us-mta-522-lIbvTnzYMdyAF9RvOKvLQw-1; Mon, 26 Aug 2024 16:44:19 -0400 X-MC-Unique: lIbvTnzYMdyAF9RvOKvLQw-1 Received: by mail-oo1-f71.google.com with SMTP id 006d021491bc7-5d5cec5ab62so6030827eaf.3 for ; Mon, 26 Aug 2024 13:44:19 -0700 (PDT) X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20230601; t=1724705059; x=1725309859; h=content-transfer-encoding:mime-version:references:in-reply-to :message-id:date:subject:cc:to:from:x-gm-message-state:from:to:cc :subject:date:message-id:reply-to; bh=kaY3EfeA+k09YWgS+MfGM8qkjlv58srLyvVau/9jXnU=; b=vbGXeKKZaa2ycJ3HzjerTdY9dQBgcRUW+DoGxNPlbqgCI8DdS1czGfrcyTah5N06PT Bmm8LK5vBBJNTwNesAdMnu/okqne/BcoDbC9PSijLhIaQ10yba4YTvVxFVth2QvdfBIj IuWLml7wENJALtFEKSvnyckk5dtcUmxBfksFSw51QW1GTAOJUEh4QVgthAUiZi7eGc+c YDnB4Yp239T984RRrzULPpudz2srj9HuklqYoD/jTEWPXSRR1yXiwK43Mjgv+oEjxVib +Q7chrX19Bi3RGqcd2aS/P3QuIoCgGwUWYiXhRs+6ivKgLKxA267i/cmguwrb2BUmrNd 2PDw== X-Forwarded-Encrypted: i=1; AJvYcCUUfstQ6PEUSzyshlRPjUOHK6iu9PR8PAULosfwYwQmLNp8jSgqu54GCF0d/sQcrRI6anPIZtXxwNjoQRWOnoJZ@lists.infradead.org X-Gm-Message-State: AOJu0YxI+zt/5j2Lhncc7R3EAbO/Y3LDiEFm7gAZyG/iLlWOf4jJGdVE U+cuxRhvnE+fnNGFqGoLmpnzHB7pq5INdZWkB6ChR0JsDJIzFQB9RWTzcruCWy3BgLoOqbx2ZC7 qLPAZPgBMaUetH5aeCEJEFgEqVG7BNFsYeAdPZ2ftJmZ+C0IXh+ouShPfFx7oZU0bLrgXYPdA X-Received: by 2002:a05:6358:7201:b0:1ac:f3df:3bde with SMTP id e5c5f4694b2df-1b5ebf3abdcmr117737655d.5.1724705058971; Mon, 26 Aug 2024 13:44:18 -0700 (PDT) X-Google-Smtp-Source: AGHT+IHztBftH8jfc2wrFWJQjc0BC5uJ0IwQWlV9Rpv0YGeitSfcFoHRh3KGrU0k5GaslF87BwNh3Q== X-Received: by 2002:a05:6358:7201:b0:1ac:f3df:3bde with SMTP id e5c5f4694b2df-1b5ebf3abdcmr117734755d.5.1724705058531; Mon, 26 Aug 2024 13:44:18 -0700 (PDT) Received: from x1n.redhat.com (pool-99-254-121-117.cpe.net.cable.rogers.com. [99.254.121.117]) by smtp.gmail.com with ESMTPSA id af79cd13be357-7a67f3fd6c1sm491055185a.121.2024.08.26.13.44.16 (version=TLS1_3 cipher=TLS_AES_256_GCM_SHA384 bits=256/256); Mon, 26 Aug 2024 13:44:18 -0700 (PDT) From: Peter Xu To: linux-kernel@vger.kernel.org, linux-mm@kvack.org Cc: Gavin Shan , Catalin Marinas , x86@kernel.org, Ingo Molnar , Andrew Morton , Paolo Bonzini , Dave Hansen , Thomas Gleixner , Alistair Popple , kvm@vger.kernel.org, linux-arm-kernel@lists.infradead.org, Sean Christopherson , peterx@redhat.com, Oscar Salvador , Jason Gunthorpe , Borislav Petkov , Zi Yan , Axel Rasmussen , David Hildenbrand , Yan Zhao , Will Deacon , Kefeng Wang , Alex Williamson , Niklas Schnelle , Gerald Schaefer , Heiko Carstens , Vasily Gorbik , Alexander Gordeev , Christian Borntraeger , Sven Schnelle , linux-s390@vger.kernel.org Subject: [PATCH v2 11/19] s390/pci_mmio: Use follow_pfnmap API Date: Mon, 26 Aug 2024 16:43:45 -0400 Message-ID: <20240826204353.2228736-12-peterx@redhat.com> X-Mailer: git-send-email 2.45.0 In-Reply-To: <20240826204353.2228736-1-peterx@redhat.com> References: <20240826204353.2228736-1-peterx@redhat.com> MIME-Version: 1.0 X-Mimecast-Spam-Score: 0 X-Mimecast-Originator: redhat.com X-CRM114-Version: 20100106-BlameMichelson ( TRE 0.8.0 (BSD) ) MR-646709E3 X-CRM114-CacheID: sfid-20240826_134422_523929_0B125817 X-CRM114-Status: GOOD ( 13.24 ) X-BeenThere: linux-arm-kernel@lists.infradead.org X-Mailman-Version: 2.1.34 Precedence: list List-Id: List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Sender: "linux-arm-kernel" Errors-To: linux-arm-kernel-bounces+linux-arm-kernel=archiver.kernel.org@lists.infradead.org Use the new API that can understand huge pfn mappings. Cc: Niklas Schnelle Cc: Gerald Schaefer Cc: Heiko Carstens Cc: Vasily Gorbik Cc: Alexander Gordeev Cc: Christian Borntraeger Cc: Sven Schnelle Cc: linux-s390@vger.kernel.org Signed-off-by: Peter Xu --- arch/s390/pci/pci_mmio.c | 22 ++++++++++++---------- 1 file changed, 12 insertions(+), 10 deletions(-) diff --git a/arch/s390/pci/pci_mmio.c b/arch/s390/pci/pci_mmio.c index 5398729bfe1b..de5c0b389a3e 100644 --- a/arch/s390/pci/pci_mmio.c +++ b/arch/s390/pci/pci_mmio.c @@ -118,12 +118,11 @@ static inline int __memcpy_toio_inuser(void __iomem *dst, SYSCALL_DEFINE3(s390_pci_mmio_write, unsigned long, mmio_addr, const void __user *, user_buffer, size_t, length) { + struct follow_pfnmap_args args = { }; u8 local_buf[64]; void __iomem *io_addr; void *buf; struct vm_area_struct *vma; - pte_t *ptep; - spinlock_t *ptl; long ret; if (!zpci_is_enabled()) @@ -169,11 +168,13 @@ SYSCALL_DEFINE3(s390_pci_mmio_write, unsigned long, mmio_addr, if (!(vma->vm_flags & VM_WRITE)) goto out_unlock_mmap; - ret = follow_pte(vma, mmio_addr, &ptep, &ptl); + args.address = mmio_addr; + args.vma = vma; + ret = follow_pfnmap_start(&args); if (ret) goto out_unlock_mmap; - io_addr = (void __iomem *)((pte_pfn(*ptep) << PAGE_SHIFT) | + io_addr = (void __iomem *)((args.pfn << PAGE_SHIFT) | (mmio_addr & ~PAGE_MASK)); if ((unsigned long) io_addr < ZPCI_IOMAP_ADDR_BASE) @@ -181,7 +182,7 @@ SYSCALL_DEFINE3(s390_pci_mmio_write, unsigned long, mmio_addr, ret = zpci_memcpy_toio(io_addr, buf, length); out_unlock_pt: - pte_unmap_unlock(ptep, ptl); + follow_pfnmap_end(&args); out_unlock_mmap: mmap_read_unlock(current->mm); out_free: @@ -260,12 +261,11 @@ static inline int __memcpy_fromio_inuser(void __user *dst, SYSCALL_DEFINE3(s390_pci_mmio_read, unsigned long, mmio_addr, void __user *, user_buffer, size_t, length) { + struct follow_pfnmap_args args = { }; u8 local_buf[64]; void __iomem *io_addr; void *buf; struct vm_area_struct *vma; - pte_t *ptep; - spinlock_t *ptl; long ret; if (!zpci_is_enabled()) @@ -308,11 +308,13 @@ SYSCALL_DEFINE3(s390_pci_mmio_read, unsigned long, mmio_addr, if (!(vma->vm_flags & VM_WRITE)) goto out_unlock_mmap; - ret = follow_pte(vma, mmio_addr, &ptep, &ptl); + args.vma = vma; + args.address = mmio_addr; + ret = follow_pfnmap_start(&args); if (ret) goto out_unlock_mmap; - io_addr = (void __iomem *)((pte_pfn(*ptep) << PAGE_SHIFT) | + io_addr = (void __iomem *)((args.pfn << PAGE_SHIFT) | (mmio_addr & ~PAGE_MASK)); if ((unsigned long) io_addr < ZPCI_IOMAP_ADDR_BASE) { @@ -322,7 +324,7 @@ SYSCALL_DEFINE3(s390_pci_mmio_read, unsigned long, mmio_addr, ret = zpci_memcpy_fromio(buf, io_addr, length); out_unlock_pt: - pte_unmap_unlock(ptep, ptl); + follow_pfnmap_end(&args); out_unlock_mmap: mmap_read_unlock(current->mm); From patchwork Mon Aug 26 20:43:46 2024 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Peter Xu X-Patchwork-Id: 13778449 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from bombadil.infradead.org (bombadil.infradead.org [198.137.202.133]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (No client certificate requested) by smtp.lore.kernel.org (Postfix) with ESMTPS id BB463C5321D for ; Mon, 26 Aug 2024 20:54:46 +0000 (UTC) DKIM-Signature: v=1; a=rsa-sha256; q=dns/txt; c=relaxed/relaxed; d=lists.infradead.org; s=bombadil.20210309; h=Sender:List-Subscribe:List-Help :List-Post:List-Archive:List-Unsubscribe:List-Id:Content-Type: Content-Transfer-Encoding:MIME-Version:References:In-Reply-To:Message-ID:Date :Subject:Cc:To:From:Reply-To:Content-ID:Content-Description:Resent-Date: Resent-From:Resent-Sender:Resent-To:Resent-Cc:Resent-Message-ID:List-Owner; bh=moprxbUn5M7+L+63SZwwyqeq4iL0OHeeIJZbsfnkOho=; b=g5oJrE4kzSjAOOtAwMi/ugQCa0 hGxtrsdio9o3xMps9qy9sdvQXv7yken/UDh0IkdRWyFIwlkOT1SBys/th/0fWmV0Cuh3z1T//jzur z4EJN3IXHvQyRxw5jfXWHie22T9FPYAQPgtMZSHgyNi7FEjrmgEP62QGxJifmTz+ptcgImqkaG6yY dD4sqmrURZzFC1f+SXxub8oRs1xf8jiTPxn0WI96ZT1Mh0DZ2HgRhQQhbbqsm6TaSQYI8CgG+y6ez yatQ2aNs8GnX0NjkS/majSYgOp5johyE+jcPRDBTqQx6gx/vDsc96zFramgw5jVPZDYkPBcOWEq4C WXarmFjQ==; Received: from localhost ([::1] helo=bombadil.infradead.org) by bombadil.infradead.org with esmtp (Exim 4.97.1 #2 (Red Hat Linux)) id 1sigjR-00000008hYV-114X; Mon, 26 Aug 2024 20:54:33 +0000 Received: from us-smtp-delivery-124.mimecast.com ([170.10.129.124]) by bombadil.infradead.org with esmtps (Exim 4.97.1 #2 (Red Hat Linux)) id 1sigZb-00000008eXy-2YL5 for linux-arm-kernel@lists.infradead.org; Mon, 26 Aug 2024 20:44:25 +0000 DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=redhat.com; s=mimecast20190719; t=1724705062; h=from:from:reply-to:subject:subject:date:date:message-id:message-id: to:to:cc:cc:mime-version:mime-version:content-type:content-type: content-transfer-encoding:content-transfer-encoding: in-reply-to:in-reply-to:references:references; bh=moprxbUn5M7+L+63SZwwyqeq4iL0OHeeIJZbsfnkOho=; b=U3Hm2rYD+TDFjedSX/ywSr3TBwLzavI/njXNAx2PRr3kBvrgbXYEsimWA9V8GdFgkvgTqI QHFkl3NbnN/CADQ7NIKFB44/ot9+5/3OCazsr0aRTnROLZGe27rIkiOclJhDj6D96tU5e8 e6H2RLrc9sAJ0WTxvRViRz8JODo/Ed4= Received: from mail-qk1-f199.google.com (mail-qk1-f199.google.com [209.85.222.199]) by relay.mimecast.com with ESMTP with STARTTLS (version=TLSv1.3, cipher=TLS_AES_256_GCM_SHA384) id us-mta-245-YWmJGXzvPQeDshp8CGgD0Q-1; Mon, 26 Aug 2024 16:44:21 -0400 X-MC-Unique: YWmJGXzvPQeDshp8CGgD0Q-1 Received: by mail-qk1-f199.google.com with SMTP id af79cd13be357-7a1d9a712bcso1119803385a.1 for ; Mon, 26 Aug 2024 13:44:21 -0700 (PDT) X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20230601; t=1724705060; x=1725309860; h=content-transfer-encoding:mime-version:references:in-reply-to :message-id:date:subject:cc:to:from:x-gm-message-state:from:to:cc :subject:date:message-id:reply-to; bh=moprxbUn5M7+L+63SZwwyqeq4iL0OHeeIJZbsfnkOho=; b=KqcZU/iCDk3Ew4wo2Axd/Kx9x1jZzFxzWvNDBC+B+8OawVAeLZLSnpT+N/zusE6+7s TKkXBRIFavvxvz1v5xanOO0XEt2FPEqs1ox86aHu4yxC32pLlKTC7ouFRqyaIQGEPf7I CyPmqT+qNWJjwn68FADH/PqGPkyi4R7jBFS+lWpSZXZB5btV1tfSRu+G5+y93vYWlEBC GNaUkyfau+TTTpC+n9cH8sP1/uGX+qPjALTCtG1PSQPDpl2p9gmwyVyNmQ2gzouQPN5B WzeRG5SobQ5CpmjeD/5JOYBJ8f5t6nKhviyWqwYOocd47ycRyuYp83gDWERkVwcnzH7N gaSw== X-Forwarded-Encrypted: i=1; AJvYcCUa0p16KWKss/DwvFGvwHo0L11oLEVzDB7rCSuNTgkQZHK3cWMx0AA7ybPXY4nOyitE/w/T5EdnzR5dvq34HE8B@lists.infradead.org X-Gm-Message-State: AOJu0YyvUFUCPaM7Lwol9FhmcCLLh9pFxg2ljQ+6R/86R7zjGE59yYl/ 5MLiPcrqno6b4P9oB4ccPJlzg/r0Zc1CU6pcMA0KUGC8nZvLk1PUa4PTLrtg1nL1iyJsG+fC17O 0y7lnnJ0XikmIpGQb8a3DlFWrTJYKEXPphGD3eMmxsiUrKxk5ysnIuczI1XCCJAV5wdofaFom X-Received: by 2002:a05:620a:1a92:b0:7a1:e341:d543 with SMTP id af79cd13be357-7a7e4cf3a3fmr158797685a.28.1724705060674; Mon, 26 Aug 2024 13:44:20 -0700 (PDT) X-Google-Smtp-Source: AGHT+IF8L+Naz3FkxDynSQ3yMse2qHglqFkf4LqBPi/5cD3sQEuFfYs6yWPMbT2Qk9XtpK98A+7trg== X-Received: by 2002:a05:620a:1a92:b0:7a1:e341:d543 with SMTP id af79cd13be357-7a7e4cf3a3fmr158793685a.28.1724705060390; Mon, 26 Aug 2024 13:44:20 -0700 (PDT) Received: from x1n.redhat.com (pool-99-254-121-117.cpe.net.cable.rogers.com. [99.254.121.117]) by smtp.gmail.com with ESMTPSA id af79cd13be357-7a67f3fd6c1sm491055185a.121.2024.08.26.13.44.18 (version=TLS1_3 cipher=TLS_AES_256_GCM_SHA384 bits=256/256); Mon, 26 Aug 2024 13:44:19 -0700 (PDT) From: Peter Xu To: linux-kernel@vger.kernel.org, linux-mm@kvack.org Cc: Gavin Shan , Catalin Marinas , x86@kernel.org, Ingo Molnar , Andrew Morton , Paolo Bonzini , Dave Hansen , Thomas Gleixner , Alistair Popple , kvm@vger.kernel.org, linux-arm-kernel@lists.infradead.org, Sean Christopherson , peterx@redhat.com, Oscar Salvador , Jason Gunthorpe , Borislav Petkov , Zi Yan , Axel Rasmussen , David Hildenbrand , Yan Zhao , Will Deacon , Kefeng Wang , Alex Williamson Subject: [PATCH v2 12/19] mm/x86/pat: Use the new follow_pfnmap API Date: Mon, 26 Aug 2024 16:43:46 -0400 Message-ID: <20240826204353.2228736-13-peterx@redhat.com> X-Mailer: git-send-email 2.45.0 In-Reply-To: <20240826204353.2228736-1-peterx@redhat.com> References: <20240826204353.2228736-1-peterx@redhat.com> MIME-Version: 1.0 X-Mimecast-Spam-Score: 0 X-Mimecast-Originator: redhat.com X-CRM114-Version: 20100106-BlameMichelson ( TRE 0.8.0 (BSD) ) MR-646709E3 X-CRM114-CacheID: sfid-20240826_134423_865067_11C49000 X-CRM114-Status: GOOD ( 14.48 ) X-BeenThere: linux-arm-kernel@lists.infradead.org X-Mailman-Version: 2.1.34 Precedence: list List-Id: List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Sender: "linux-arm-kernel" Errors-To: linux-arm-kernel-bounces+linux-arm-kernel=archiver.kernel.org@lists.infradead.org Use the new API that can understand huge pfn mappings. Cc: x86@kernel.org Cc: Thomas Gleixner Cc: Ingo Molnar Cc: Borislav Petkov Cc: Dave Hansen Signed-off-by: Peter Xu --- arch/x86/mm/pat/memtype.c | 17 +++++++---------- 1 file changed, 7 insertions(+), 10 deletions(-) diff --git a/arch/x86/mm/pat/memtype.c b/arch/x86/mm/pat/memtype.c index 1fa0bf6ed295..f73b5ce270b3 100644 --- a/arch/x86/mm/pat/memtype.c +++ b/arch/x86/mm/pat/memtype.c @@ -951,23 +951,20 @@ static void free_pfn_range(u64 paddr, unsigned long size) static int follow_phys(struct vm_area_struct *vma, unsigned long *prot, resource_size_t *phys) { - pte_t *ptep, pte; - spinlock_t *ptl; + struct follow_pfnmap_args args = { .vma = vma, .address = vma->vm_start }; - if (follow_pte(vma, vma->vm_start, &ptep, &ptl)) + if (follow_pfnmap_start(&args)) return -EINVAL; - pte = ptep_get(ptep); - /* Never return PFNs of anon folios in COW mappings. */ - if (vm_normal_folio(vma, vma->vm_start, pte)) { - pte_unmap_unlock(ptep, ptl); + if (!args.special) { + follow_pfnmap_end(&args); return -EINVAL; } - *prot = pgprot_val(pte_pgprot(pte)); - *phys = (resource_size_t)pte_pfn(pte) << PAGE_SHIFT; - pte_unmap_unlock(ptep, ptl); + *prot = pgprot_val(args.pgprot); + *phys = (resource_size_t)args.pfn << PAGE_SHIFT; + follow_pfnmap_end(&args); return 0; } From patchwork Mon Aug 26 20:43:47 2024 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Peter Xu X-Patchwork-Id: 13778450 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from bombadil.infradead.org (bombadil.infradead.org [198.137.202.133]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (No client certificate requested) by smtp.lore.kernel.org (Postfix) with ESMTPS id 97951C5472C for ; Mon, 26 Aug 2024 20:55:31 +0000 (UTC) DKIM-Signature: v=1; a=rsa-sha256; q=dns/txt; c=relaxed/relaxed; d=lists.infradead.org; s=bombadil.20210309; h=Sender:List-Subscribe:List-Help :List-Post:List-Archive:List-Unsubscribe:List-Id:Content-Type: Content-Transfer-Encoding:MIME-Version:References:In-Reply-To:Message-ID:Date :Subject:Cc:To:From:Reply-To:Content-ID:Content-Description:Resent-Date: Resent-From:Resent-Sender:Resent-To:Resent-Cc:Resent-Message-ID:List-Owner; bh=HLBqit68L/U2mdwYaFkydFK3txHa+LkmL8n2YTQXnnw=; b=yHNu4PyuJDOiWdCGHmntXodtdk JgtQGnD+pmcfTiEsOjppehVKRMIFZQ4W8c2yKo9yGxz2p5uy43fJO7ObIp7CrtRmo2kRT7B+h/9EZ X5jhi8xtF2NQobLsnkT1oLmwaUtnJCEcDiK/bFIZR/betT7WqfEUpOgp6QDfB9Ow3girwouhvwHqd SoQ3Jr9/UaYCDh+7v9VYh0bzYDaSUj9ReqZ9Xoqlv+WMIxH0/H9D5ZBSbw7ZIlBoFLvuRDTJdDVkr LRt39eWTyRukLKpbKU59coZC9hxS+tPBPevCswyZBxENAC6miSxLXgCL7jErrZ7lLXmCYm11ZW03G 2cAX/2Fg==; Received: from localhost ([::1] helo=bombadil.infradead.org) by bombadil.infradead.org with esmtp (Exim 4.97.1 #2 (Red Hat Linux)) id 1sigkD-00000008hmh-1Oc5; Mon, 26 Aug 2024 20:55:21 +0000 Received: from us-smtp-delivery-124.mimecast.com ([170.10.129.124]) by bombadil.infradead.org with esmtps (Exim 4.97.1 #2 (Red Hat Linux)) id 1sigZd-00000008eYx-1J7i for linux-arm-kernel@lists.infradead.org; Mon, 26 Aug 2024 20:44:26 +0000 DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=redhat.com; s=mimecast20190719; t=1724705064; h=from:from:reply-to:subject:subject:date:date:message-id:message-id: to:to:cc:cc:mime-version:mime-version:content-type:content-type: content-transfer-encoding:content-transfer-encoding: in-reply-to:in-reply-to:references:references; bh=HLBqit68L/U2mdwYaFkydFK3txHa+LkmL8n2YTQXnnw=; b=hDMkfdUFa2+crLVYgkKa35feEWHHtuXyYrUwt5xQQkbcedtdoZAfa642vdy0qaR4daPto0 M0tgUiRz1zwJyv4J56pMuE9Qgm+nzRxRLpk1t6GkiHWk7iJBMmWPEpgcXfEbk3SB0N1oXP J/oTRCYdXknDa+beaM2vnXsntJBiWW8= Received: from mail-yw1-f199.google.com (mail-yw1-f199.google.com [209.85.128.199]) by relay.mimecast.com with ESMTP with STARTTLS (version=TLSv1.3, cipher=TLS_AES_256_GCM_SHA384) id us-mta-663-C7DnQqkVNf-F4ut0tfZDCA-1; Mon, 26 Aug 2024 16:44:23 -0400 X-MC-Unique: C7DnQqkVNf-F4ut0tfZDCA-1 Received: by mail-yw1-f199.google.com with SMTP id 00721157ae682-6b3fec974e5so90045387b3.1 for ; Mon, 26 Aug 2024 13:44:23 -0700 (PDT) X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20230601; t=1724705063; x=1725309863; h=content-transfer-encoding:mime-version:references:in-reply-to :message-id:date:subject:cc:to:from:x-gm-message-state:from:to:cc :subject:date:message-id:reply-to; bh=HLBqit68L/U2mdwYaFkydFK3txHa+LkmL8n2YTQXnnw=; b=BUx4/v4Se7lFbQ005aZ/ZVTBZW0RPCBObl5B3j7k0L3eNpYC7wWwVennvkU6LHxhq7 BbgjSo8b+Ks5s/nhawIUfkqsHpLnmjl9Lw6pe0yTS5KEYfidYMGU8C7LcQIg6nkIMBJi IZvdVL461BGX/yUVdrj6EDl320sN0k4+GxDtOMO4o3ToiSpbaRvtnhXnGDTot1vwyL8j uZwVLGLwBHwdj0bTuwD+WiRk0sRbcnlNjAgzb2+2y4dAYQVqN6sfHfG1o97IjKuVJ48Z JlRshbFyTSuQzTsMudlvWuacpotY+gGQQclDlY5IVALubr+9f9F6S6RKepI8Xyu5sXKN WzJg== X-Forwarded-Encrypted: i=1; AJvYcCV4XAW9pisT94k6kumPv6jfoQ9vbKqELWgF5nmUKPctKebrH4OsgAwmzgCaHwN2YefxmZpD0Trno6OBdFa79Xuc@lists.infradead.org X-Gm-Message-State: AOJu0YzQbLkcTyL4Z7UWCETOquz2z0Bud2RBH1cqQf181nV3xyEy9tM7 SjGlJw0WqQNpkJ0lGZfQnDqy8fe7awVCLJUmDeurOUIYmYoNs4RdbRcKwikoZVh09Wf8V2/ORPl Yn5aOR++LGIT4euClVazFbztD3r6fK+qK5fcavJQg7CAJbruUdCIBeglO6U6b3cdSZLT3YEBm X-Received: by 2002:a05:690c:39b:b0:62f:aaaa:187a with SMTP id 00721157ae682-6c625a4cc55mr151106927b3.14.1724705062737; Mon, 26 Aug 2024 13:44:22 -0700 (PDT) X-Google-Smtp-Source: AGHT+IGeaoJv/EbB8r7mSzacjacGidgTJuNMUQKi5I30qkj93MEO5BseX564RMMiUfXmELj5ALvNSw== X-Received: by 2002:a05:690c:39b:b0:62f:aaaa:187a with SMTP id 00721157ae682-6c625a4cc55mr151106717b3.14.1724705062426; Mon, 26 Aug 2024 13:44:22 -0700 (PDT) Received: from x1n.redhat.com (pool-99-254-121-117.cpe.net.cable.rogers.com. [99.254.121.117]) by smtp.gmail.com with ESMTPSA id af79cd13be357-7a67f3fd6c1sm491055185a.121.2024.08.26.13.44.20 (version=TLS1_3 cipher=TLS_AES_256_GCM_SHA384 bits=256/256); Mon, 26 Aug 2024 13:44:21 -0700 (PDT) From: Peter Xu To: linux-kernel@vger.kernel.org, linux-mm@kvack.org Cc: Gavin Shan , Catalin Marinas , x86@kernel.org, Ingo Molnar , Andrew Morton , Paolo Bonzini , Dave Hansen , Thomas Gleixner , Alistair Popple , kvm@vger.kernel.org, linux-arm-kernel@lists.infradead.org, Sean Christopherson , peterx@redhat.com, Oscar Salvador , Jason Gunthorpe , Borislav Petkov , Zi Yan , Axel Rasmussen , David Hildenbrand , Yan Zhao , Will Deacon , Kefeng Wang , Alex Williamson Subject: [PATCH v2 13/19] vfio: Use the new follow_pfnmap API Date: Mon, 26 Aug 2024 16:43:47 -0400 Message-ID: <20240826204353.2228736-14-peterx@redhat.com> X-Mailer: git-send-email 2.45.0 In-Reply-To: <20240826204353.2228736-1-peterx@redhat.com> References: <20240826204353.2228736-1-peterx@redhat.com> MIME-Version: 1.0 X-Mimecast-Spam-Score: 0 X-Mimecast-Originator: redhat.com X-CRM114-Version: 20100106-BlameMichelson ( TRE 0.8.0 (BSD) ) MR-646709E3 X-CRM114-CacheID: sfid-20240826_134425_456081_E4021BE4 X-CRM114-Status: GOOD ( 14.29 ) X-BeenThere: linux-arm-kernel@lists.infradead.org X-Mailman-Version: 2.1.34 Precedence: list List-Id: List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Sender: "linux-arm-kernel" Errors-To: linux-arm-kernel-bounces+linux-arm-kernel=archiver.kernel.org@lists.infradead.org Use the new API that can understand huge pfn mappings. Cc: Alex Williamson Cc: Jason Gunthorpe Signed-off-by: Peter Xu --- drivers/vfio/vfio_iommu_type1.c | 16 ++++++---------- 1 file changed, 6 insertions(+), 10 deletions(-) diff --git a/drivers/vfio/vfio_iommu_type1.c b/drivers/vfio/vfio_iommu_type1.c index 0960699e7554..bf391b40e576 100644 --- a/drivers/vfio/vfio_iommu_type1.c +++ b/drivers/vfio/vfio_iommu_type1.c @@ -513,12 +513,10 @@ static int follow_fault_pfn(struct vm_area_struct *vma, struct mm_struct *mm, unsigned long vaddr, unsigned long *pfn, bool write_fault) { - pte_t *ptep; - pte_t pte; - spinlock_t *ptl; + struct follow_pfnmap_args args = { .vma = vma, .address = vaddr }; int ret; - ret = follow_pte(vma, vaddr, &ptep, &ptl); + ret = follow_pfnmap_start(&args); if (ret) { bool unlocked = false; @@ -532,19 +530,17 @@ static int follow_fault_pfn(struct vm_area_struct *vma, struct mm_struct *mm, if (ret) return ret; - ret = follow_pte(vma, vaddr, &ptep, &ptl); + ret = follow_pfnmap_start(&args); if (ret) return ret; } - pte = ptep_get(ptep); - - if (write_fault && !pte_write(pte)) + if (write_fault && !args.writable) ret = -EFAULT; else - *pfn = pte_pfn(pte); + *pfn = args.pfn; - pte_unmap_unlock(ptep, ptl); + follow_pfnmap_end(&args); return ret; } From patchwork Mon Aug 26 20:43:48 2024 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Peter Xu X-Patchwork-Id: 13778451 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from bombadil.infradead.org (bombadil.infradead.org [198.137.202.133]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (No client certificate requested) by smtp.lore.kernel.org (Postfix) with ESMTPS id 79587C5321D for ; Mon, 26 Aug 2024 20:56:20 +0000 (UTC) DKIM-Signature: v=1; a=rsa-sha256; q=dns/txt; c=relaxed/relaxed; d=lists.infradead.org; s=bombadil.20210309; h=Sender:List-Subscribe:List-Help :List-Post:List-Archive:List-Unsubscribe:List-Id:Content-Type: Content-Transfer-Encoding:MIME-Version:References:In-Reply-To:Message-ID:Date :Subject:Cc:To:From:Reply-To:Content-ID:Content-Description:Resent-Date: Resent-From:Resent-Sender:Resent-To:Resent-Cc:Resent-Message-ID:List-Owner; bh=d0kI5he4gLOJUpsF87CpEeiS1FOCa69CaYI81su9df0=; b=nf45MNTlBbWJt6TRzfjfErGky+ U56J67jHikL5JaKGyOnSaXPt8J784uWdMFyTpmDc5RnBOz+3gpa2xCLvnVLe5c0n0bBofR5sqS5fx lFbvtRQJXOAkK4wgCzs5b3VrJcaO2utd7QW1VtdtcNigISTzsuu/3HL3T33IdQ8I/8EaCpordpaZs D/MY22pns2r7NbXZP0UfiCSTM1ApG2WH4MZazY/Mzau3NYaDptXZAmbIQRzfCngRmRJhZeCIarhkZ pBHmYCp/RDwxQt+QgycrpZLAx0fWCGlTf0Q1Z/TMKo+y35GbHOs8p/C2f0U/kfru77ukyeJ8/y7wA BNFY4b8g==; Received: from localhost ([::1] helo=bombadil.infradead.org) by bombadil.infradead.org with esmtp (Exim 4.97.1 #2 (Red Hat Linux)) id 1sigkz-00000008i2x-3OAi; Mon, 26 Aug 2024 20:56:09 +0000 Received: from us-smtp-delivery-124.mimecast.com ([170.10.129.124]) by bombadil.infradead.org with esmtps (Exim 4.97.1 #2 (Red Hat Linux)) id 1sigZf-00000008eaI-0DTf for linux-arm-kernel@lists.infradead.org; Mon, 26 Aug 2024 20:44:28 +0000 DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=redhat.com; s=mimecast20190719; t=1724705066; h=from:from:reply-to:subject:subject:date:date:message-id:message-id: to:to:cc:cc:mime-version:mime-version:content-type:content-type: content-transfer-encoding:content-transfer-encoding: in-reply-to:in-reply-to:references:references; bh=d0kI5he4gLOJUpsF87CpEeiS1FOCa69CaYI81su9df0=; b=TlIhbTc1ZhoKtSipLHoNMaY3r6WAyvfQrLyXrWCZMyjW0PuyK76PazUM0aXMsUUNgUt/aV Vwfew9Y/My5oOmRSoXxa4ksZp7QN1OQzmPovdvWOemzLE2upwV8v2SNke4ZWaIwxOQMWxP 5xDmbeWlYxfuOH4PGnj54jGqTAzVkvY= Received: from mail-qk1-f198.google.com (mail-qk1-f198.google.com [209.85.222.198]) by relay.mimecast.com with ESMTP with STARTTLS (version=TLSv1.3, cipher=TLS_AES_256_GCM_SHA384) id us-mta-331-IQvGC-OoOZehu6XeuruFUw-1; Mon, 26 Aug 2024 16:44:25 -0400 X-MC-Unique: IQvGC-OoOZehu6XeuruFUw-1 Received: by mail-qk1-f198.google.com with SMTP id af79cd13be357-7a1de8a2adbso643993385a.1 for ; Mon, 26 Aug 2024 13:44:25 -0700 (PDT) X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20230601; t=1724705064; x=1725309864; h=content-transfer-encoding:mime-version:references:in-reply-to :message-id:date:subject:cc:to:from:x-gm-message-state:from:to:cc :subject:date:message-id:reply-to; bh=d0kI5he4gLOJUpsF87CpEeiS1FOCa69CaYI81su9df0=; b=QhRCbHvwyrf7TZiWh+BECytsT9Y0kjv4T/yiDafZp9RE7JyDWMXt8tcFIec9poS82C xMK9RtGD9IHZMnEXI7RLcSAKJot5m9I4RqizDepI1V2QoUvRfIPjehb1wLTJ9TwXlf96 TwqI77SQUET0kvhRNq2bPj107NHZ1FYfwp6DdMPcPy+9xNUy6mi5Qb7n1jNsbmi5HlaB VAO5YSDzGTl/x15erk1r9LCNQans7v4DZuuPalXdJc7DGSJzdqXBeHEeQmtroYtvr8Yv mjEP/RO4U3Wey+sD5F9LTkEK5RTMasj9xJUuOYLSrUelgBLoZCyZC+c4OPgTG2vLcH8a kGwQ== X-Forwarded-Encrypted: i=1; AJvYcCWCGfxv8j9ZXG6b1NBx4Ls5Cf9wgTO5vy9oDLKlFlzVxQTuzJUzaUvjjARwjDs+zIfu2cEedKvu4B3Xyovd8vYu@lists.infradead.org X-Gm-Message-State: AOJu0Yx+H+lSKcdM4dd60vjuGITsam9jcAUEaTWeWTyIkGT5N5kQ1wZ3 /oOtq82tgAefv6tDq9Ijyzq+QYp+JigMAWICdUwdpXlwth9/b3kD2r8iisjzcglB5DFslhh4mqX 4cafRMhZAelYVxr+ns7YR7uI/vuAQLlce+K7cknzSo7ot36IcgPGxlIGNIK28vrY8ReQYf/OP X-Received: by 2002:a05:620a:2484:b0:79e:ff41:fd47 with SMTP id af79cd13be357-7a7e4dc9528mr93288485a.28.1724705064650; Mon, 26 Aug 2024 13:44:24 -0700 (PDT) X-Google-Smtp-Source: AGHT+IEloa5qz2QVMofE8PCWJJ6gTfdMgC7cjn+h2anZ9qsldjcOaIzzaCXqVOAeeqkMcyU6B13FuA== X-Received: by 2002:a05:620a:2484:b0:79e:ff41:fd47 with SMTP id af79cd13be357-7a7e4dc9528mr93286485a.28.1724705064315; Mon, 26 Aug 2024 13:44:24 -0700 (PDT) Received: from x1n.redhat.com (pool-99-254-121-117.cpe.net.cable.rogers.com. [99.254.121.117]) by smtp.gmail.com with ESMTPSA id af79cd13be357-7a67f3fd6c1sm491055185a.121.2024.08.26.13.44.22 (version=TLS1_3 cipher=TLS_AES_256_GCM_SHA384 bits=256/256); Mon, 26 Aug 2024 13:44:23 -0700 (PDT) From: Peter Xu To: linux-kernel@vger.kernel.org, linux-mm@kvack.org Cc: Gavin Shan , Catalin Marinas , x86@kernel.org, Ingo Molnar , Andrew Morton , Paolo Bonzini , Dave Hansen , Thomas Gleixner , Alistair Popple , kvm@vger.kernel.org, linux-arm-kernel@lists.infradead.org, Sean Christopherson , peterx@redhat.com, Oscar Salvador , Jason Gunthorpe , Borislav Petkov , Zi Yan , Axel Rasmussen , David Hildenbrand , Yan Zhao , Will Deacon , Kefeng Wang , Alex Williamson Subject: [PATCH v2 14/19] acrn: Use the new follow_pfnmap API Date: Mon, 26 Aug 2024 16:43:48 -0400 Message-ID: <20240826204353.2228736-15-peterx@redhat.com> X-Mailer: git-send-email 2.45.0 In-Reply-To: <20240826204353.2228736-1-peterx@redhat.com> References: <20240826204353.2228736-1-peterx@redhat.com> MIME-Version: 1.0 X-Mimecast-Spam-Score: 0 X-Mimecast-Originator: redhat.com X-CRM114-Version: 20100106-BlameMichelson ( TRE 0.8.0 (BSD) ) MR-646709E3 X-CRM114-CacheID: sfid-20240826_134427_188292_D24E82A6 X-CRM114-Status: GOOD ( 14.53 ) X-BeenThere: linux-arm-kernel@lists.infradead.org X-Mailman-Version: 2.1.34 Precedence: list List-Id: List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Sender: "linux-arm-kernel" Errors-To: linux-arm-kernel-bounces+linux-arm-kernel=archiver.kernel.org@lists.infradead.org Use the new API that can understand huge pfn mappings. Signed-off-by: Peter Xu --- drivers/virt/acrn/mm.c | 16 +++++++++------- 1 file changed, 9 insertions(+), 7 deletions(-) diff --git a/drivers/virt/acrn/mm.c b/drivers/virt/acrn/mm.c index db8ff1d0ac23..4c2f28715b70 100644 --- a/drivers/virt/acrn/mm.c +++ b/drivers/virt/acrn/mm.c @@ -177,9 +177,7 @@ int acrn_vm_ram_map(struct acrn_vm *vm, struct acrn_vm_memmap *memmap) vma = vma_lookup(current->mm, memmap->vma_base); if (vma && ((vma->vm_flags & VM_PFNMAP) != 0)) { unsigned long start_pfn, cur_pfn; - spinlock_t *ptl; bool writable; - pte_t *ptep; if ((memmap->vma_base + memmap->len) > vma->vm_end) { mmap_read_unlock(current->mm); @@ -187,16 +185,20 @@ int acrn_vm_ram_map(struct acrn_vm *vm, struct acrn_vm_memmap *memmap) } for (i = 0; i < nr_pages; i++) { - ret = follow_pte(vma, memmap->vma_base + i * PAGE_SIZE, - &ptep, &ptl); + struct follow_pfnmap_args args = { + .vma = vma, + .address = memmap->vma_base + i * PAGE_SIZE, + }; + + ret = follow_pfnmap_start(&args); if (ret) break; - cur_pfn = pte_pfn(ptep_get(ptep)); + cur_pfn = args.pfn; if (i == 0) start_pfn = cur_pfn; - writable = !!pte_write(ptep_get(ptep)); - pte_unmap_unlock(ptep, ptl); + writable = args.writable; + follow_pfnmap_end(&args); /* Disallow write access if the PTE is not writable. */ if (!writable && From patchwork Mon Aug 26 20:43:49 2024 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Peter Xu X-Patchwork-Id: 13778452 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from bombadil.infradead.org (bombadil.infradead.org [198.137.202.133]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (No client certificate requested) by smtp.lore.kernel.org (Postfix) with ESMTPS id 5B2DFC5321D for ; Mon, 26 Aug 2024 20:57:08 +0000 (UTC) DKIM-Signature: v=1; a=rsa-sha256; q=dns/txt; c=relaxed/relaxed; d=lists.infradead.org; s=bombadil.20210309; h=Sender:List-Subscribe:List-Help :List-Post:List-Archive:List-Unsubscribe:List-Id:Content-Type: Content-Transfer-Encoding:MIME-Version:References:In-Reply-To:Message-ID:Date :Subject:Cc:To:From:Reply-To:Content-ID:Content-Description:Resent-Date: Resent-From:Resent-Sender:Resent-To:Resent-Cc:Resent-Message-ID:List-Owner; bh=t75nynj+5ccZ6qy3UooqAYOOLyCQeJyCwmKKtTf1Ss4=; b=U5JGa2D8WBaVXr2jnQ+LQz5IMb RCjFgPfOt+hzYvQhYzvw/tId8vdiXo/dUav5UXlI5wCqVhRYREwfCyjFF4wXTbng+DZkmScuzFQ2R McbljlkLsMR4b+7eKh2MLyoGUwo8aiwgdvhi8IhMqwaizvrr1cNudHuTixKEhevjolmNXa+VbRIsv VtSzgGN3LmG3sNAVuTZof6aEdXg3YuFOkXufLUpJtAeho1AV/ceZj2HjiaB3POcbuo4w8pGJVvWMl GDQLC+uI9BzAV3hWgLPVhNRmJMrEiOG0tjpeVCDPQLnrKd1FAq4/jrHKexwiDoahtmfx72JoCuu4x 8wx0B5Yw==; Received: from localhost ([::1] helo=bombadil.infradead.org) by bombadil.infradead.org with esmtp (Exim 4.97.1 #2 (Red Hat Linux)) id 1siglm-00000008iHt-481R; Mon, 26 Aug 2024 20:56:58 +0000 Received: from us-smtp-delivery-124.mimecast.com ([170.10.133.124]) by bombadil.infradead.org with esmtps (Exim 4.97.1 #2 (Red Hat Linux)) id 1sigZh-00000008ebO-1jWO for linux-arm-kernel@lists.infradead.org; Mon, 26 Aug 2024 20:44:31 +0000 DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=redhat.com; s=mimecast20190719; t=1724705068; h=from:from:reply-to:subject:subject:date:date:message-id:message-id: to:to:cc:cc:mime-version:mime-version:content-type:content-type: content-transfer-encoding:content-transfer-encoding: in-reply-to:in-reply-to:references:references; bh=t75nynj+5ccZ6qy3UooqAYOOLyCQeJyCwmKKtTf1Ss4=; b=OewNm48nnEsXZ4g7ozbw/jYqwhX4mt6Y1hsL0mgpqyH6uTCnw2csiExh1uqsIAGtmZ2ORi q6yIMQDyO3onAN7MYTSsFszZfl/15J7Ksbhpyrxb/csEbZpaXX9icONu/RLJt78+tsNruR Q+dfJh3xwxLClRnVm9k00S+MxiD0aTg= Received: from mail-oa1-f70.google.com (mail-oa1-f70.google.com [209.85.160.70]) by relay.mimecast.com with ESMTP with STARTTLS (version=TLSv1.3, cipher=TLS_AES_256_GCM_SHA384) id us-mta-52-FVIMtCrjN-KYpjbvIJ4QLg-1; Mon, 26 Aug 2024 16:44:27 -0400 X-MC-Unique: FVIMtCrjN-KYpjbvIJ4QLg-1 Received: by mail-oa1-f70.google.com with SMTP id 586e51a60fabf-2700946649eso6410941fac.0 for ; Mon, 26 Aug 2024 13:44:27 -0700 (PDT) X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20230601; t=1724705066; x=1725309866; h=content-transfer-encoding:mime-version:references:in-reply-to :message-id:date:subject:cc:to:from:x-gm-message-state:from:to:cc :subject:date:message-id:reply-to; bh=t75nynj+5ccZ6qy3UooqAYOOLyCQeJyCwmKKtTf1Ss4=; b=PlNWShSLHIg7CXcYI8+c3CT3hWKn8JPaXzd3bcMBbACHG2BT9SPPKOit/aNo56hK1a jo2OhTHJJLp0X2kq3z2mLpmRMGhhvzyFiEYj83ggaBrVaPmZj7He9nToOVz1dsk3A0/F /9ibqwC8b7tQ9DqK8tgNLIqfuBl2uJb6pm1AcJ2a6wsX2p9asSDUUtFOOmQKcFRsgfg4 3kgy7QMUGWWxI0vZXx9hP2xIfTKkfWhv2iKxKTYM8uLDYKHuhJ8WB26pGjPwpuGQ2S1W ZKunCRsn7sjibkZqnVTDuNPeeaXHv+nfDROmsTRhekQgdtNN20x4dYrKNi5HIJ3sHUXw 0qJg== X-Forwarded-Encrypted: i=1; AJvYcCX6ATV8SXGr4qa2nEf+FJ9uWBvmvgJ7lmAfq9EmDDeaQDTKHJtz3mn7NPXnq48J5T56kFcvkqVWkPrQB3dbkvRp@lists.infradead.org X-Gm-Message-State: AOJu0YwvikHwaTS3jUtnO6MWYR5Hxmv55Q4pLbckxNvOUTSBBAtGTXm/ VE4/gAgmcqrMlw7x9NlqTL8vLMge/ML5dL5WeOHPnKPgyscEIhApve5KK4WQu7AyVFL9E/+Nzox 07Uj2zieBBHeM6a5MvYSTHXC1iV5Bx8gdlDzbnYHZjTp2zihTHS1FTbOB2GEFc27WZ9ard/L9 X-Received: by 2002:a05:6358:3a07:b0:1ad:10ff:341e with SMTP id e5c5f4694b2df-1b5c3a3b69dmr1457842955d.4.1724705066538; Mon, 26 Aug 2024 13:44:26 -0700 (PDT) X-Google-Smtp-Source: AGHT+IGoPajws+HFHWojCbb3D8qBEyBE/zIeN3nkxAxD5pVBn23+qUQ20h6jqSHyvqmBY4DTTGYw+g== X-Received: by 2002:a05:6358:3a07:b0:1ad:10ff:341e with SMTP id e5c5f4694b2df-1b5c3a3b69dmr1457841255d.4.1724705066240; Mon, 26 Aug 2024 13:44:26 -0700 (PDT) Received: from x1n.redhat.com (pool-99-254-121-117.cpe.net.cable.rogers.com. [99.254.121.117]) by smtp.gmail.com with ESMTPSA id af79cd13be357-7a67f3fd6c1sm491055185a.121.2024.08.26.13.44.24 (version=TLS1_3 cipher=TLS_AES_256_GCM_SHA384 bits=256/256); Mon, 26 Aug 2024 13:44:25 -0700 (PDT) From: Peter Xu To: linux-kernel@vger.kernel.org, linux-mm@kvack.org Cc: Gavin Shan , Catalin Marinas , x86@kernel.org, Ingo Molnar , Andrew Morton , Paolo Bonzini , Dave Hansen , Thomas Gleixner , Alistair Popple , kvm@vger.kernel.org, linux-arm-kernel@lists.infradead.org, Sean Christopherson , peterx@redhat.com, Oscar Salvador , Jason Gunthorpe , Borislav Petkov , Zi Yan , Axel Rasmussen , David Hildenbrand , Yan Zhao , Will Deacon , Kefeng Wang , Alex Williamson Subject: [PATCH v2 15/19] mm/access_process_vm: Use the new follow_pfnmap API Date: Mon, 26 Aug 2024 16:43:49 -0400 Message-ID: <20240826204353.2228736-16-peterx@redhat.com> X-Mailer: git-send-email 2.45.0 In-Reply-To: <20240826204353.2228736-1-peterx@redhat.com> References: <20240826204353.2228736-1-peterx@redhat.com> MIME-Version: 1.0 X-Mimecast-Spam-Score: 0 X-Mimecast-Originator: redhat.com X-CRM114-Version: 20100106-BlameMichelson ( TRE 0.8.0 (BSD) ) MR-646709E3 X-CRM114-CacheID: sfid-20240826_134429_555428_E41CE6DE X-CRM114-Status: GOOD ( 13.74 ) X-BeenThere: linux-arm-kernel@lists.infradead.org X-Mailman-Version: 2.1.34 Precedence: list List-Id: List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Sender: "linux-arm-kernel" Errors-To: linux-arm-kernel-bounces+linux-arm-kernel=archiver.kernel.org@lists.infradead.org Use the new API that can understand huge pfn mappings. Signed-off-by: Peter Xu --- mm/memory.c | 28 ++++++++++++++-------------- 1 file changed, 14 insertions(+), 14 deletions(-) diff --git a/mm/memory.c b/mm/memory.c index 0b136c398257..b5d07f493d5d 100644 --- a/mm/memory.c +++ b/mm/memory.c @@ -6342,34 +6342,34 @@ int generic_access_phys(struct vm_area_struct *vma, unsigned long addr, resource_size_t phys_addr; unsigned long prot = 0; void __iomem *maddr; - pte_t *ptep, pte; - spinlock_t *ptl; int offset = offset_in_page(addr); int ret = -EINVAL; + bool writable; + struct follow_pfnmap_args args = { .vma = vma, .address = addr }; retry: - if (follow_pte(vma, addr, &ptep, &ptl)) + if (follow_pfnmap_start(&args)) return -EINVAL; - pte = ptep_get(ptep); - pte_unmap_unlock(ptep, ptl); + prot = pgprot_val(args.pgprot); + phys_addr = (resource_size_t)args.pfn << PAGE_SHIFT; + writable = args.writable; + follow_pfnmap_end(&args); - prot = pgprot_val(pte_pgprot(pte)); - phys_addr = (resource_size_t)pte_pfn(pte) << PAGE_SHIFT; - - if ((write & FOLL_WRITE) && !pte_write(pte)) + if ((write & FOLL_WRITE) && !writable) return -EINVAL; maddr = ioremap_prot(phys_addr, PAGE_ALIGN(len + offset), prot); if (!maddr) return -ENOMEM; - if (follow_pte(vma, addr, &ptep, &ptl)) + if (follow_pfnmap_start(&args)) goto out_unmap; - if (!pte_same(pte, ptep_get(ptep))) { - pte_unmap_unlock(ptep, ptl); + if ((prot != pgprot_val(args.pgprot)) || + (phys_addr != (args.pfn << PAGE_SHIFT)) || + (writable != args.writable)) { + follow_pfnmap_end(&args); iounmap(maddr); - goto retry; } @@ -6378,7 +6378,7 @@ int generic_access_phys(struct vm_area_struct *vma, unsigned long addr, else memcpy_fromio(buf, maddr + offset, len); ret = len; - pte_unmap_unlock(ptep, ptl); + follow_pfnmap_end(&args); out_unmap: iounmap(maddr); From patchwork Mon Aug 26 20:43:50 2024 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Peter Xu X-Patchwork-Id: 13778453 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from bombadil.infradead.org (bombadil.infradead.org [198.137.202.133]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (No client certificate requested) by smtp.lore.kernel.org (Postfix) with ESMTPS id D8778C5321D for ; Mon, 26 Aug 2024 20:57:57 +0000 (UTC) DKIM-Signature: v=1; a=rsa-sha256; q=dns/txt; c=relaxed/relaxed; d=lists.infradead.org; s=bombadil.20210309; h=Sender:List-Subscribe:List-Help :List-Post:List-Archive:List-Unsubscribe:List-Id:Content-Type: Content-Transfer-Encoding:MIME-Version:References:In-Reply-To:Message-ID:Date :Subject:Cc:To:From:Reply-To:Content-ID:Content-Description:Resent-Date: Resent-From:Resent-Sender:Resent-To:Resent-Cc:Resent-Message-ID:List-Owner; bh=X8WHCJ7kNzF2XCQH70BbHdio2suKNHxAf6rhsHMkPbs=; b=Zze/M+Td/cPAAXXPolKXVY09k7 zidWWSrbLic40fSJGsr6OvK991VvFbL+HFUTw8pgyBecSmfT8brkT/RTZ8Lu2nFywr5uOM7kMfran U7bZ1d+GfW4OP+YStng9aiQt7nqYfXXaAL+P1nSno9EQs5TcXOZaFphmBhbMs6QD4qZSfqn2+LHxZ 60ZxY1/WyS+04Bqr6rX3wD5SH3L2WJzU6nfQICN13xCeakBDiTA8Yk2loKsYBzmrYYV2GoF6421DK JMIkePBagjb4JxUcTAZic8O2vWpnCArWqiGQFuuJ4ixqAgie2pjr4sJi0VIQRxne0CzHEIDlV8vts 4uKD7YQg==; Received: from localhost ([::1] helo=bombadil.infradead.org) by bombadil.infradead.org with esmtp (Exim 4.97.1 #2 (Red Hat Linux)) id 1sigmZ-00000008iXG-2L2W; Mon, 26 Aug 2024 20:57:47 +0000 Received: from us-smtp-delivery-124.mimecast.com ([170.10.133.124]) by bombadil.infradead.org with esmtps (Exim 4.97.1 #2 (Red Hat Linux)) id 1sigZj-00000008ecU-0Qro for linux-arm-kernel@lists.infradead.org; Mon, 26 Aug 2024 20:44:33 +0000 DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=redhat.com; s=mimecast20190719; t=1724705070; h=from:from:reply-to:subject:subject:date:date:message-id:message-id: to:to:cc:cc:mime-version:mime-version:content-type:content-type: content-transfer-encoding:content-transfer-encoding: in-reply-to:in-reply-to:references:references; bh=X8WHCJ7kNzF2XCQH70BbHdio2suKNHxAf6rhsHMkPbs=; b=Y8SkvWu2LOubSTzyFjhJG+uaffHluqMJERUI1Eu1Uw/9ljdkiwXGV1hGP183x9gafw8i3I j7tSUSw3ajrzFTK6pgtF3iqMCJsMlviAcKYcA7qksr2PXh+k1frHWfPTuz/PccEMarvGs6 4RIvB1FJ+sHyXnt2nwT3G80QRCHjWwI= Received: from mail-ua1-f71.google.com (mail-ua1-f71.google.com [209.85.222.71]) by relay.mimecast.com with ESMTP with STARTTLS (version=TLSv1.3, cipher=TLS_AES_256_GCM_SHA384) id us-mta-134-AyC8uNfGNT-OVY0Yr0vTLQ-1; Mon, 26 Aug 2024 16:44:29 -0400 X-MC-Unique: AyC8uNfGNT-OVY0Yr0vTLQ-1 Received: by mail-ua1-f71.google.com with SMTP id a1e0cc1a2514c-842f7fc5cb5so1351923241.0 for ; Mon, 26 Aug 2024 13:44:28 -0700 (PDT) X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20230601; t=1724705068; x=1725309868; h=content-transfer-encoding:mime-version:references:in-reply-to :message-id:date:subject:cc:to:from:x-gm-message-state:from:to:cc :subject:date:message-id:reply-to; bh=X8WHCJ7kNzF2XCQH70BbHdio2suKNHxAf6rhsHMkPbs=; b=OYMen8+mgrYl51B4CIj3LFvn+ajxV6KSH471z5o3bJVwWOTXvQycFKBrz60SS13I4T A5LXs4y2cV3ZmTVaW8D2ZdyzTEKuctv5Clkb0vAxIWvnQE1ekKBs/w7iRHqQ7a68zv+9 72y9NTVYUxWmzEUN+7fn8l8k+eWHGRhXusVAbb4YilfEc2FWxUOr/XCtQZrsXtp3ei0O t/onknxfzTecl+rn84cI5cB5hOLkl01Y9+Tuy9b4Kk1g1DSz2Vd1i9UU6MS2rlAx8Ee1 3O4OH09vuetn1V5bMDytiNA7K9BwepT7obhatSDZ5otoUuUWYyRPM5USc8KoBZQ6DG7u Sd9Q== X-Forwarded-Encrypted: i=1; AJvYcCUdURFxe09QmlBRjmAF5qp6iXBmA3nIFM8ETIfvAPxfOxqApof4qUMDD3rMTKsyFlgwcPfsDk0LQm+LG6WUCOq1@lists.infradead.org X-Gm-Message-State: AOJu0Yx3Kw+U3tEnAzXFcMelzDheW4L9ysecmalXBh11J312gplJucNE eIfp/lRxcy8zOmMZdaS3W9YQISaRp4CXGHmN/E5OkB6fmHUesi6aCnSATiM093ANnfp6beCoMwD Kw/tbtCrtVFbMowkp+wr1LB0aBLXWc0+ISDnHOcpElJHeG4ZzwpCRoewkXbDRjotUhfJg/8fX X-Received: by 2002:a05:6102:3e94:b0:493:e585:6ce3 with SMTP id ada2fe7eead31-49a3bd1cd83mr1005192137.31.1724705068475; Mon, 26 Aug 2024 13:44:28 -0700 (PDT) X-Google-Smtp-Source: AGHT+IGKOe4NKlkU/HSIr+h3CCvXyG5C70MeRvYsCyQPr65oWdFZ+m1iMwzpLlsyBV/cJVa8qCplQQ== X-Received: by 2002:a05:6102:3e94:b0:493:e585:6ce3 with SMTP id ada2fe7eead31-49a3bd1cd83mr1005176137.31.1724705068153; Mon, 26 Aug 2024 13:44:28 -0700 (PDT) Received: from x1n.redhat.com (pool-99-254-121-117.cpe.net.cable.rogers.com. [99.254.121.117]) by smtp.gmail.com with ESMTPSA id af79cd13be357-7a67f3fd6c1sm491055185a.121.2024.08.26.13.44.26 (version=TLS1_3 cipher=TLS_AES_256_GCM_SHA384 bits=256/256); Mon, 26 Aug 2024 13:44:27 -0700 (PDT) From: Peter Xu To: linux-kernel@vger.kernel.org, linux-mm@kvack.org Cc: Gavin Shan , Catalin Marinas , x86@kernel.org, Ingo Molnar , Andrew Morton , Paolo Bonzini , Dave Hansen , Thomas Gleixner , Alistair Popple , kvm@vger.kernel.org, linux-arm-kernel@lists.infradead.org, Sean Christopherson , peterx@redhat.com, Oscar Salvador , Jason Gunthorpe , Borislav Petkov , Zi Yan , Axel Rasmussen , David Hildenbrand , Yan Zhao , Will Deacon , Kefeng Wang , Alex Williamson Subject: [PATCH v2 16/19] mm: Remove follow_pte() Date: Mon, 26 Aug 2024 16:43:50 -0400 Message-ID: <20240826204353.2228736-17-peterx@redhat.com> X-Mailer: git-send-email 2.45.0 In-Reply-To: <20240826204353.2228736-1-peterx@redhat.com> References: <20240826204353.2228736-1-peterx@redhat.com> MIME-Version: 1.0 X-Mimecast-Spam-Score: 0 X-Mimecast-Originator: redhat.com X-CRM114-Version: 20100106-BlameMichelson ( TRE 0.8.0 (BSD) ) MR-646709E3 X-CRM114-CacheID: sfid-20240826_134431_321113_77496AE5 X-CRM114-Status: GOOD ( 19.09 ) X-BeenThere: linux-arm-kernel@lists.infradead.org X-Mailman-Version: 2.1.34 Precedence: list List-Id: List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Sender: "linux-arm-kernel" Errors-To: linux-arm-kernel-bounces+linux-arm-kernel=archiver.kernel.org@lists.infradead.org follow_pte() users have been converted to follow_pfnmap*(). Remove the API. Signed-off-by: Peter Xu --- include/linux/mm.h | 2 -- mm/memory.c | 73 ---------------------------------------------- 2 files changed, 75 deletions(-) diff --git a/include/linux/mm.h b/include/linux/mm.h index 161d496bfd18..b31d4bdd65ad 100644 --- a/include/linux/mm.h +++ b/include/linux/mm.h @@ -2368,8 +2368,6 @@ void free_pgd_range(struct mmu_gather *tlb, unsigned long addr, unsigned long end, unsigned long floor, unsigned long ceiling); int copy_page_range(struct vm_area_struct *dst_vma, struct vm_area_struct *src_vma); -int follow_pte(struct vm_area_struct *vma, unsigned long address, - pte_t **ptepp, spinlock_t **ptlp); int generic_access_phys(struct vm_area_struct *vma, unsigned long addr, void *buf, int len, int write); diff --git a/mm/memory.c b/mm/memory.c index b5d07f493d5d..288f81a8698e 100644 --- a/mm/memory.c +++ b/mm/memory.c @@ -6100,79 +6100,6 @@ int __pmd_alloc(struct mm_struct *mm, pud_t *pud, unsigned long address) } #endif /* __PAGETABLE_PMD_FOLDED */ -/** - * follow_pte - look up PTE at a user virtual address - * @vma: the memory mapping - * @address: user virtual address - * @ptepp: location to store found PTE - * @ptlp: location to store the lock for the PTE - * - * On a successful return, the pointer to the PTE is stored in @ptepp; - * the corresponding lock is taken and its location is stored in @ptlp. - * - * The contents of the PTE are only stable until @ptlp is released using - * pte_unmap_unlock(). This function will fail if the PTE is non-present. - * Present PTEs may include PTEs that map refcounted pages, such as - * anonymous folios in COW mappings. - * - * Callers must be careful when relying on PTE content after - * pte_unmap_unlock(). Especially if the PTE maps a refcounted page, - * callers must protect against invalidation with MMU notifiers; otherwise - * access to the PFN at a later point in time can trigger use-after-free. - * - * Only IO mappings and raw PFN mappings are allowed. The mmap semaphore - * should be taken for read. - * - * This function must not be used to modify PTE content. - * - * Return: zero on success, -ve otherwise. - */ -int follow_pte(struct vm_area_struct *vma, unsigned long address, - pte_t **ptepp, spinlock_t **ptlp) -{ - struct mm_struct *mm = vma->vm_mm; - pgd_t *pgd; - p4d_t *p4d; - pud_t *pud; - pmd_t *pmd; - pte_t *ptep; - - mmap_assert_locked(mm); - if (unlikely(address < vma->vm_start || address >= vma->vm_end)) - goto out; - - if (!(vma->vm_flags & (VM_IO | VM_PFNMAP))) - goto out; - - pgd = pgd_offset(mm, address); - if (pgd_none(*pgd) || unlikely(pgd_bad(*pgd))) - goto out; - - p4d = p4d_offset(pgd, address); - if (p4d_none(*p4d) || unlikely(p4d_bad(*p4d))) - goto out; - - pud = pud_offset(p4d, address); - if (pud_none(*pud) || unlikely(pud_bad(*pud))) - goto out; - - pmd = pmd_offset(pud, address); - VM_BUG_ON(pmd_trans_huge(*pmd)); - - ptep = pte_offset_map_lock(mm, pmd, address, ptlp); - if (!ptep) - goto out; - if (!pte_present(ptep_get(ptep))) - goto unlock; - *ptepp = ptep; - return 0; -unlock: - pte_unmap_unlock(ptep, *ptlp); -out: - return -EINVAL; -} -EXPORT_SYMBOL_GPL(follow_pte); - static inline void pfnmap_args_setup(struct follow_pfnmap_args *args, spinlock_t *lock, pte_t *ptep, pgprot_t pgprot, unsigned long pfn_base, From patchwork Mon Aug 26 20:43:51 2024 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Peter Xu X-Patchwork-Id: 13778456 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from bombadil.infradead.org (bombadil.infradead.org [198.137.202.133]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (No client certificate requested) by smtp.lore.kernel.org (Postfix) with ESMTPS id 0E5C6C5321D for ; Mon, 26 Aug 2024 20:58:42 +0000 (UTC) DKIM-Signature: v=1; a=rsa-sha256; q=dns/txt; c=relaxed/relaxed; d=lists.infradead.org; s=bombadil.20210309; h=Sender:List-Subscribe:List-Help :List-Post:List-Archive:List-Unsubscribe:List-Id:Content-Type: Content-Transfer-Encoding:MIME-Version:References:In-Reply-To:Message-ID:Date :Subject:Cc:To:From:Reply-To:Content-ID:Content-Description:Resent-Date: Resent-From:Resent-Sender:Resent-To:Resent-Cc:Resent-Message-ID:List-Owner; bh=IC5gYR+7Z4BwAPnZp8TN8XZEGypCYGRNkok3NV40z5s=; b=s801DTdL1a4t3I+9id2q2/Scw6 JI14hyScmPWOQL4tIscNwpZpCzWWc9TDXdl0rsEEC7A37A5DTo73zcbqY1Df7sl7nbS+D0RIOu7/o VODngHHFYbp5Hrn/R4lExzWEpmrMANvNfQYxFqOCjIBZAqUeGdkRbBk9UjDyo8ockC2Fhrh2su4pa GKurOtoA4zI3WLTZ1EY/sa8ck+a2poW95zUvTgWt/Ekwiq2VXQvVxuHupaUrLBWELjbBr9/FpjMOJ IIzw0swzmJi2FZvhHQhrdvAvJvYaRVZ0+o3mGV1s81fDFCB0iGynrxl5M1C2JJuEuHdvlku98kFGa ujlgqAvw==; Received: from localhost ([::1] helo=bombadil.infradead.org) by bombadil.infradead.org with esmtp (Exim 4.97.1 #2 (Red Hat Linux)) id 1signJ-00000008idJ-1S9V; Mon, 26 Aug 2024 20:58:33 +0000 Received: from us-smtp-delivery-124.mimecast.com ([170.10.129.124]) by bombadil.infradead.org with esmtps (Exim 4.97.1 #2 (Red Hat Linux)) id 1sigZl-00000008eeF-2FLK for linux-arm-kernel@lists.infradead.org; Mon, 26 Aug 2024 20:44:34 +0000 DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=redhat.com; s=mimecast20190719; t=1724705072; h=from:from:reply-to:subject:subject:date:date:message-id:message-id: to:to:cc:cc:mime-version:mime-version:content-type:content-type: content-transfer-encoding:content-transfer-encoding: in-reply-to:in-reply-to:references:references; bh=IC5gYR+7Z4BwAPnZp8TN8XZEGypCYGRNkok3NV40z5s=; b=Bx48/gEy8HcNPSGTC1saLoQpFtRlpAqaHob2hDvDJssP+38CWMrcsGJHc9yq14J+BcKZKB pTw4U6vVcGmGAmneL4EgB6tKASr//AaS5oPgIPuaZj7qp/UMLGNCE8ZZmCxgMsFCl3iKHN nipTN8TOh7lp2ZJCuTHVmvEytRghhrI= Received: from mail-oo1-f71.google.com (mail-oo1-f71.google.com [209.85.161.71]) by relay.mimecast.com with ESMTP with STARTTLS (version=TLSv1.3, cipher=TLS_AES_256_GCM_SHA384) id us-mta-221--qr-l5kHPcOMCu7xaMUoaw-1; Mon, 26 Aug 2024 16:44:31 -0400 X-MC-Unique: -qr-l5kHPcOMCu7xaMUoaw-1 Received: by mail-oo1-f71.google.com with SMTP id 006d021491bc7-5da5516c615so6122345eaf.1 for ; Mon, 26 Aug 2024 13:44:31 -0700 (PDT) X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20230601; t=1724705070; x=1725309870; h=content-transfer-encoding:mime-version:references:in-reply-to :message-id:date:subject:cc:to:from:x-gm-message-state:from:to:cc :subject:date:message-id:reply-to; bh=IC5gYR+7Z4BwAPnZp8TN8XZEGypCYGRNkok3NV40z5s=; b=K5qyswp1GVqDYaye25KXEiI3mAWzZwRJEH6QF0JtAqzyW4TtveAo++8+6DAoUxEGCz OJEjlL8WwbY9xCxGKRoddJ0odxCTK20dPxauzHk1iqcagm9UAi1u9gKkSxJXoRhEkqLD AqieP4S8yDlgRJQFNFO/Nh5vYX4rEjIo55G52mD0d30KbvYyaRFJBCATEsRnVDHGaHTC GWf3n05IEOUgIsyjZuxjQdSGES7qvPNXM7eenFGeJGM68iI6g33MDAXwM2bgbNkrqiL6 8bYj/pqyqvD5/GXdYgixUxQdu7gLB5dpPHrxV59fIH/SM0jf/4RtuL14CKFgi20Ph/I0 teiQ== X-Forwarded-Encrypted: i=1; AJvYcCX/RhSFLF6rAq8/O0+rmB/PxQB7RMinvq9iL0if60HidYovxnXlKO2ga62GJEu20xItATqFvQbJ3NNcoWmF7jsm@lists.infradead.org X-Gm-Message-State: AOJu0YxUctnJTYNrC4ti0vrRKV+nIGEh3nU4Q5gaUiJpzX+fb7czVfQP VhCkI4qtWlHnQCHHAReINGZOUXkELx7ia1x4BoJRO/Jtdqh5TfBr/00WJF7TCIeXoQb62KL6Y8F TllObpHRI3/xLhGs8N4qiuBRAshWjKG4llc5KaN0z0WvV9E6RdU6AUoH/jWS64SmLMKpsE3sB X-Received: by 2002:a05:6358:70c3:b0:1ac:f00d:c8c6 with SMTP id e5c5f4694b2df-1b5c22ebd07mr1493542355d.27.1724705070549; Mon, 26 Aug 2024 13:44:30 -0700 (PDT) X-Google-Smtp-Source: AGHT+IHWRaHJzCfl4jAxAoYVG6HBrfG1+v3SSmaZHfZSlba6wG4sR68STMcFscyvu7mDZcazVEGWKw== X-Received: by 2002:a05:6358:70c3:b0:1ac:f00d:c8c6 with SMTP id e5c5f4694b2df-1b5c22ebd07mr1493538355d.27.1724705070146; Mon, 26 Aug 2024 13:44:30 -0700 (PDT) Received: from x1n.redhat.com (pool-99-254-121-117.cpe.net.cable.rogers.com. [99.254.121.117]) by smtp.gmail.com with ESMTPSA id af79cd13be357-7a67f3fd6c1sm491055185a.121.2024.08.26.13.44.28 (version=TLS1_3 cipher=TLS_AES_256_GCM_SHA384 bits=256/256); Mon, 26 Aug 2024 13:44:29 -0700 (PDT) From: Peter Xu To: linux-kernel@vger.kernel.org, linux-mm@kvack.org Cc: Gavin Shan , Catalin Marinas , x86@kernel.org, Ingo Molnar , Andrew Morton , Paolo Bonzini , Dave Hansen , Thomas Gleixner , Alistair Popple , kvm@vger.kernel.org, linux-arm-kernel@lists.infradead.org, Sean Christopherson , peterx@redhat.com, Oscar Salvador , Jason Gunthorpe , Borislav Petkov , Zi Yan , Axel Rasmussen , David Hildenbrand , Yan Zhao , Will Deacon , Kefeng Wang , Alex Williamson Subject: [PATCH v2 17/19] mm/x86: Support large pfn mappings Date: Mon, 26 Aug 2024 16:43:51 -0400 Message-ID: <20240826204353.2228736-18-peterx@redhat.com> X-Mailer: git-send-email 2.45.0 In-Reply-To: <20240826204353.2228736-1-peterx@redhat.com> References: <20240826204353.2228736-1-peterx@redhat.com> MIME-Version: 1.0 X-Mimecast-Spam-Score: 0 X-Mimecast-Originator: redhat.com X-CRM114-Version: 20100106-BlameMichelson ( TRE 0.8.0 (BSD) ) MR-646709E3 X-CRM114-CacheID: sfid-20240826_134433_678685_2B771542 X-CRM114-Status: GOOD ( 17.85 ) X-BeenThere: linux-arm-kernel@lists.infradead.org X-Mailman-Version: 2.1.34 Precedence: list List-Id: List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Sender: "linux-arm-kernel" Errors-To: linux-arm-kernel-bounces+linux-arm-kernel=archiver.kernel.org@lists.infradead.org Helpers to install and detect special pmd/pud entries. In short, bit 9 on x86 is not used for pmd/pud, so we can directly define them the same as the pte level. One note is that it's also used in _PAGE_BIT_CPA_TEST but that is only used in the debug test, and shouldn't conflict in this case. One note is that pxx_set|clear_flags() for pmd/pud will need to be moved upper so that they can be referenced by the new special bit helpers. There's no change in the code that was moved. Cc: x86@kernel.org Cc: Thomas Gleixner Cc: Ingo Molnar Cc: Borislav Petkov Cc: Dave Hansen Signed-off-by: Peter Xu --- arch/x86/Kconfig | 1 + arch/x86/include/asm/pgtable.h | 80 ++++++++++++++++++++++------------ 2 files changed, 53 insertions(+), 28 deletions(-) diff --git a/arch/x86/Kconfig b/arch/x86/Kconfig index b74b9ee484da..d4dbe9717e96 100644 --- a/arch/x86/Kconfig +++ b/arch/x86/Kconfig @@ -28,6 +28,7 @@ config X86_64 select ARCH_HAS_GIGANTIC_PAGE select ARCH_SUPPORTS_INT128 if CC_HAS_INT128 select ARCH_SUPPORTS_PER_VMA_LOCK + select ARCH_SUPPORTS_HUGE_PFNMAP if TRANSPARENT_HUGEPAGE select HAVE_ARCH_SOFT_DIRTY select MODULES_USE_ELF_RELA select NEED_DMA_MAP_STATE diff --git a/arch/x86/include/asm/pgtable.h b/arch/x86/include/asm/pgtable.h index 8d12bfad6a1d..4c2d080d26b4 100644 --- a/arch/x86/include/asm/pgtable.h +++ b/arch/x86/include/asm/pgtable.h @@ -120,6 +120,34 @@ extern pmdval_t early_pmd_flags; #define arch_end_context_switch(prev) do {} while(0) #endif /* CONFIG_PARAVIRT_XXL */ +static inline pmd_t pmd_set_flags(pmd_t pmd, pmdval_t set) +{ + pmdval_t v = native_pmd_val(pmd); + + return native_make_pmd(v | set); +} + +static inline pmd_t pmd_clear_flags(pmd_t pmd, pmdval_t clear) +{ + pmdval_t v = native_pmd_val(pmd); + + return native_make_pmd(v & ~clear); +} + +static inline pud_t pud_set_flags(pud_t pud, pudval_t set) +{ + pudval_t v = native_pud_val(pud); + + return native_make_pud(v | set); +} + +static inline pud_t pud_clear_flags(pud_t pud, pudval_t clear) +{ + pudval_t v = native_pud_val(pud); + + return native_make_pud(v & ~clear); +} + /* * The following only work if pte_present() is true. * Undefined behaviour if not.. @@ -317,6 +345,30 @@ static inline int pud_devmap(pud_t pud) } #endif +#ifdef CONFIG_ARCH_SUPPORTS_PMD_PFNMAP +static inline bool pmd_special(pmd_t pmd) +{ + return pmd_flags(pmd) & _PAGE_SPECIAL; +} + +static inline pmd_t pmd_mkspecial(pmd_t pmd) +{ + return pmd_set_flags(pmd, _PAGE_SPECIAL); +} +#endif /* CONFIG_ARCH_SUPPORTS_PMD_PFNMAP */ + +#ifdef CONFIG_ARCH_SUPPORTS_PUD_PFNMAP +static inline bool pud_special(pud_t pud) +{ + return pud_flags(pud) & _PAGE_SPECIAL; +} + +static inline pud_t pud_mkspecial(pud_t pud) +{ + return pud_set_flags(pud, _PAGE_SPECIAL); +} +#endif /* CONFIG_ARCH_SUPPORTS_PUD_PFNMAP */ + static inline int pgd_devmap(pgd_t pgd) { return 0; @@ -487,20 +539,6 @@ static inline pte_t pte_mkdevmap(pte_t pte) return pte_set_flags(pte, _PAGE_SPECIAL|_PAGE_DEVMAP); } -static inline pmd_t pmd_set_flags(pmd_t pmd, pmdval_t set) -{ - pmdval_t v = native_pmd_val(pmd); - - return native_make_pmd(v | set); -} - -static inline pmd_t pmd_clear_flags(pmd_t pmd, pmdval_t clear) -{ - pmdval_t v = native_pmd_val(pmd); - - return native_make_pmd(v & ~clear); -} - /* See comments above mksaveddirty_shift() */ static inline pmd_t pmd_mksaveddirty(pmd_t pmd) { @@ -595,20 +633,6 @@ static inline pmd_t pmd_mkwrite_novma(pmd_t pmd) pmd_t pmd_mkwrite(pmd_t pmd, struct vm_area_struct *vma); #define pmd_mkwrite pmd_mkwrite -static inline pud_t pud_set_flags(pud_t pud, pudval_t set) -{ - pudval_t v = native_pud_val(pud); - - return native_make_pud(v | set); -} - -static inline pud_t pud_clear_flags(pud_t pud, pudval_t clear) -{ - pudval_t v = native_pud_val(pud); - - return native_make_pud(v & ~clear); -} - /* See comments above mksaveddirty_shift() */ static inline pud_t pud_mksaveddirty(pud_t pud) { From patchwork Mon Aug 26 20:43:52 2024 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Peter Xu X-Patchwork-Id: 13778457 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from bombadil.infradead.org (bombadil.infradead.org [198.137.202.133]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (No client certificate requested) by smtp.lore.kernel.org (Postfix) with ESMTPS id B560DC5321D for ; Mon, 26 Aug 2024 20:59:31 +0000 (UTC) DKIM-Signature: v=1; a=rsa-sha256; q=dns/txt; c=relaxed/relaxed; d=lists.infradead.org; s=bombadil.20210309; h=Sender:List-Subscribe:List-Help :List-Post:List-Archive:List-Unsubscribe:List-Id:Content-Type: Content-Transfer-Encoding:MIME-Version:References:In-Reply-To:Message-ID:Date :Subject:Cc:To:From:Reply-To:Content-ID:Content-Description:Resent-Date: Resent-From:Resent-Sender:Resent-To:Resent-Cc:Resent-Message-ID:List-Owner; bh=FEAM9DK74WZYySVYhL7YnC2YOY0xQ42haB8uzH2x2BY=; b=zU4NSVXe5VrlgZ3OY8DjCBxFKh M0dO0g91GvT+zytjGjes2I8G6ZyaEQ5n5Gr/sjpAM5m2mQtG6guKhsskkzehLwdCmZUW/Ngoop6SM ivDRU17/CSbViUE2ns6dzfRc5+tm3frZuKBqKPzGimTvObWNlly9O2s0JfczYOE/JE5yrNE6WpKj9 Bk4Dg8Nmeonzfx6pA1nkd/Ta2ksjw7n0WE+/CcR/ncXw/JP588Sq+/Brl7ehUiRntQhgWhHvmOsK6 vc1HsDzCsla+Y7TSVX8De1OfwXVf66QPVUrM6DB0v8DHosnzrZXyTOhRlAKKiKwqRppI/rXTi3Ge6 BzXaqm1w==; Received: from localhost ([::1] helo=bombadil.infradead.org) by bombadil.infradead.org with esmtp (Exim 4.97.1 #2 (Red Hat Linux)) id 1sigo4-00000008ijj-06De; Mon, 26 Aug 2024 20:59:20 +0000 Received: from us-smtp-delivery-124.mimecast.com ([170.10.133.124]) by bombadil.infradead.org with esmtps (Exim 4.97.1 #2 (Red Hat Linux)) id 1sigZn-00000008efy-2rmx for linux-arm-kernel@lists.infradead.org; Mon, 26 Aug 2024 20:44:37 +0000 DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=redhat.com; s=mimecast20190719; t=1724705074; h=from:from:reply-to:subject:subject:date:date:message-id:message-id: to:to:cc:cc:mime-version:mime-version:content-type:content-type: content-transfer-encoding:content-transfer-encoding: in-reply-to:in-reply-to:references:references; bh=FEAM9DK74WZYySVYhL7YnC2YOY0xQ42haB8uzH2x2BY=; b=YYLVoWiGteZHco1yz/MpCpfQF0yuSjBl1/+aBlhmwR1vGYC/tdjrJcfQZlZrF0e9fqAAqF 7VTkx7WYGYneWcCf+1sMX+ik/PDLezaALHGApYH03Wjeo7mkXZc1u5YTk2nTbMLFbzsYUf YnhJl9U7lzcQndaKRAOubldo2RhP3AA= Received: from mail-qt1-f198.google.com (mail-qt1-f198.google.com [209.85.160.198]) by relay.mimecast.com with ESMTP with STARTTLS (version=TLSv1.3, cipher=TLS_AES_256_GCM_SHA384) id us-mta-552-SwpVj3TuNauJ4yQHfwbDyg-1; Mon, 26 Aug 2024 16:44:33 -0400 X-MC-Unique: SwpVj3TuNauJ4yQHfwbDyg-1 Received: by mail-qt1-f198.google.com with SMTP id d75a77b69052e-44fe1a88f85so67570111cf.1 for ; Mon, 26 Aug 2024 13:44:33 -0700 (PDT) X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20230601; t=1724705072; x=1725309872; h=content-transfer-encoding:mime-version:references:in-reply-to :message-id:date:subject:cc:to:from:x-gm-message-state:from:to:cc :subject:date:message-id:reply-to; bh=FEAM9DK74WZYySVYhL7YnC2YOY0xQ42haB8uzH2x2BY=; b=b1Hb3iVAl+Gf/d+B930eftA2xcnc1vbPjQ6o7oEuqQVnSme1iFdUKg+oEa2cS4Pje4 sR2WP/DcDPjLRqThD1BtP+c0k97ds9gUk4vaSf3K/wVqJ/ZpMAcW9h6fdVTG1BNsQ+gs aFOsuXXh5b4NZRmEpGvhWipGyjix1Ce+7c1w+rmDV8f3yfiaJ9bA3RnD5MgsgmxSF2oV MuAba96X8pamV+/s1VmrmJqPaNEcqUcvO/B0zLm/loyBUUM0EZAwxX6Vfp+bevQge63S x3aa2fXPRooQSh+U3SRBYWgku2ziwCmj640YN2chhpRkEEPLr7NuZ94JfvCnpesiNpWS mKZA== X-Forwarded-Encrypted: i=1; AJvYcCXK9NMibpx4Ev+TtryPsEpqXEfkeZneXKp68iWC6DtJ5UMmmgTYb3CEip82ynrmhV/8jKMsXIyPNie07IkBekxx@lists.infradead.org X-Gm-Message-State: AOJu0YxISfNYAlorIHKMoLVyhmBdLeU0rdpm/qt+lEFYJ9rP+0czKt6m S1VhVWe8rt4/lnoyJcF8JKoFnpCWzml6YSQQUgQmTgEwOY6RN2ZZakjMGesjs/VD3BEvBU+yk2K qhzHxJfZ1u+jXy+aVX3QCOlK3HlD+tIG3XdSoecdrNZ81TUR8dNNNj+FluHJ2eBNOIxpx/kPp X-Received: by 2002:a05:620a:430a:b0:7a4:faab:fc79 with SMTP id af79cd13be357-7a6896d1835mr1392501485a.8.1724705072557; Mon, 26 Aug 2024 13:44:32 -0700 (PDT) X-Google-Smtp-Source: AGHT+IHDiJ56urcvTKu8a4/ieO25ZyreTkMfmYYALFQqTS8OY3wwzzWFQolk9khXQC+xv0hRJc4KwA== X-Received: by 2002:a05:620a:430a:b0:7a4:faab:fc79 with SMTP id af79cd13be357-7a6896d1835mr1392497485a.8.1724705072141; Mon, 26 Aug 2024 13:44:32 -0700 (PDT) Received: from x1n.redhat.com (pool-99-254-121-117.cpe.net.cable.rogers.com. [99.254.121.117]) by smtp.gmail.com with ESMTPSA id af79cd13be357-7a67f3fd6c1sm491055185a.121.2024.08.26.13.44.30 (version=TLS1_3 cipher=TLS_AES_256_GCM_SHA384 bits=256/256); Mon, 26 Aug 2024 13:44:31 -0700 (PDT) From: Peter Xu To: linux-kernel@vger.kernel.org, linux-mm@kvack.org Cc: Gavin Shan , Catalin Marinas , x86@kernel.org, Ingo Molnar , Andrew Morton , Paolo Bonzini , Dave Hansen , Thomas Gleixner , Alistair Popple , kvm@vger.kernel.org, linux-arm-kernel@lists.infradead.org, Sean Christopherson , peterx@redhat.com, Oscar Salvador , Jason Gunthorpe , Borislav Petkov , Zi Yan , Axel Rasmussen , David Hildenbrand , Yan Zhao , Will Deacon , Kefeng Wang , Alex Williamson Subject: [PATCH v2 18/19] mm/arm64: Support large pfn mappings Date: Mon, 26 Aug 2024 16:43:52 -0400 Message-ID: <20240826204353.2228736-19-peterx@redhat.com> X-Mailer: git-send-email 2.45.0 In-Reply-To: <20240826204353.2228736-1-peterx@redhat.com> References: <20240826204353.2228736-1-peterx@redhat.com> MIME-Version: 1.0 X-Mimecast-Spam-Score: 0 X-Mimecast-Originator: redhat.com X-CRM114-Version: 20100106-BlameMichelson ( TRE 0.8.0 (BSD) ) MR-646709E3 X-CRM114-CacheID: sfid-20240826_134435_824332_34820C3D X-CRM114-Status: GOOD ( 14.23 ) X-BeenThere: linux-arm-kernel@lists.infradead.org X-Mailman-Version: 2.1.34 Precedence: list List-Id: List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Sender: "linux-arm-kernel" Errors-To: linux-arm-kernel-bounces+linux-arm-kernel=archiver.kernel.org@lists.infradead.org Support huge pfnmaps by using bit 56 (PTE_SPECIAL) for "special" on pmds/puds. Provide the pmd/pud helpers to set/get special bit. There's one more thing missing for arm64 which is the pxx_pgprot() for pmd/pud. Add them too, which is mostly the same as the pte version by dropping the pfn field. These helpers are essential to be used in the new follow_pfnmap*() API to report valid pgprot_t results. Note that arm64 doesn't yet support huge PUD yet, but it's still straightforward to provide the pud helpers that we need altogether. Only PMD helpers will make an immediate benefit until arm64 will support huge PUDs first in general (e.g. in THPs). Cc: linux-arm-kernel@lists.infradead.org Cc: Catalin Marinas Cc: Will Deacon Signed-off-by: Peter Xu --- arch/arm64/Kconfig | 1 + arch/arm64/include/asm/pgtable.h | 29 +++++++++++++++++++++++++++++ 2 files changed, 30 insertions(+) diff --git a/arch/arm64/Kconfig b/arch/arm64/Kconfig index 6494848019a0..6607ed8fdbb4 100644 --- a/arch/arm64/Kconfig +++ b/arch/arm64/Kconfig @@ -99,6 +99,7 @@ config ARM64 select ARCH_SUPPORTS_NUMA_BALANCING select ARCH_SUPPORTS_PAGE_TABLE_CHECK select ARCH_SUPPORTS_PER_VMA_LOCK + select ARCH_SUPPORTS_HUGE_PFNMAP if TRANSPARENT_HUGEPAGE select ARCH_WANT_BATCHED_UNMAP_TLB_FLUSH select ARCH_WANT_COMPAT_IPC_PARSE_VERSION if COMPAT select ARCH_WANT_DEFAULT_BPF_JIT diff --git a/arch/arm64/include/asm/pgtable.h b/arch/arm64/include/asm/pgtable.h index b78cc4a6758b..2faecc033a19 100644 --- a/arch/arm64/include/asm/pgtable.h +++ b/arch/arm64/include/asm/pgtable.h @@ -578,6 +578,14 @@ static inline pmd_t pmd_mkdevmap(pmd_t pmd) return pte_pmd(set_pte_bit(pmd_pte(pmd), __pgprot(PTE_DEVMAP))); } +#ifdef CONFIG_ARCH_SUPPORTS_PMD_PFNMAP +#define pmd_special(pte) (!!((pmd_val(pte) & PTE_SPECIAL))) +static inline pmd_t pmd_mkspecial(pmd_t pmd) +{ + return set_pmd_bit(pmd, __pgprot(PTE_SPECIAL)); +} +#endif + #define __pmd_to_phys(pmd) __pte_to_phys(pmd_pte(pmd)) #define __phys_to_pmd_val(phys) __phys_to_pte_val(phys) #define pmd_pfn(pmd) ((__pmd_to_phys(pmd) & PMD_MASK) >> PAGE_SHIFT) @@ -595,6 +603,27 @@ static inline pmd_t pmd_mkdevmap(pmd_t pmd) #define pud_pfn(pud) ((__pud_to_phys(pud) & PUD_MASK) >> PAGE_SHIFT) #define pfn_pud(pfn,prot) __pud(__phys_to_pud_val((phys_addr_t)(pfn) << PAGE_SHIFT) | pgprot_val(prot)) +#ifdef CONFIG_ARCH_SUPPORTS_PUD_PFNMAP +#define pud_special(pte) pte_special(pud_pte(pud)) +#define pud_mkspecial(pte) pte_pud(pte_mkspecial(pud_pte(pud))) +#endif + +#define pmd_pgprot pmd_pgprot +static inline pgprot_t pmd_pgprot(pmd_t pmd) +{ + unsigned long pfn = pmd_pfn(pmd); + + return __pgprot(pmd_val(pfn_pmd(pfn, __pgprot(0))) ^ pmd_val(pmd)); +} + +#define pud_pgprot pud_pgprot +static inline pgprot_t pud_pgprot(pud_t pud) +{ + unsigned long pfn = pud_pfn(pud); + + return __pgprot(pud_val(pfn_pud(pfn, __pgprot(0))) ^ pud_val(pud)); +} + static inline void __set_pte_at(struct mm_struct *mm, unsigned long __always_unused addr, pte_t *ptep, pte_t pte, unsigned int nr) From patchwork Mon Aug 26 20:43:53 2024 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Peter Xu X-Patchwork-Id: 13778458 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from bombadil.infradead.org (bombadil.infradead.org [198.137.202.133]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (No client certificate requested) by smtp.lore.kernel.org (Postfix) with ESMTPS id 18ED7C5472E for ; Mon, 26 Aug 2024 21:00:22 +0000 (UTC) DKIM-Signature: v=1; a=rsa-sha256; q=dns/txt; c=relaxed/relaxed; d=lists.infradead.org; s=bombadil.20210309; h=Sender:List-Subscribe:List-Help :List-Post:List-Archive:List-Unsubscribe:List-Id:Content-Type: Content-Transfer-Encoding:MIME-Version:References:In-Reply-To:Message-ID:Date :Subject:Cc:To:From:Reply-To:Content-ID:Content-Description:Resent-Date: Resent-From:Resent-Sender:Resent-To:Resent-Cc:Resent-Message-ID:List-Owner; bh=9MPOki3YF/wyEgBnejzItxwjIF1moGifGifGb5OwzY4=; b=qSDbG1fbeYVn3M7AO/PeYVW8Qy Gmjzk8QSipPsX5BaAxjiAgPxpk8kACzLaBL+CzTm8gM6POX9gHokFy97uDJ6QayidDhkgfmJHqwAg 74DoNWJfqvC7+M3xyGsBIibKxqCprcgnhbhoP26M0fONhejwSQm7vgpSXnJgxgpsxpa0YOjQYCqGe o3kpQF3RfTYa629wf0pffTG0GRCCYZdlLqqm5Pc+Mf6oH6nyqQXWRkNvch7M0Yh00QtF9TDwVJD9h 5gUFhNaoWzv1CZTZTJPRanGhxP4RQjeNxwh6eCZd0jEIPDst0qe5lSwJowdy339Xbhe8GSfnisOux BetlGQpA==; Received: from localhost ([::1] helo=bombadil.infradead.org) by bombadil.infradead.org with esmtp (Exim 4.97.1 #2 (Red Hat Linux)) id 1sigoo-00000008isH-3orW; Mon, 26 Aug 2024 21:00:06 +0000 Received: from us-smtp-delivery-124.mimecast.com ([170.10.129.124]) by bombadil.infradead.org with esmtps (Exim 4.97.1 #2 (Red Hat Linux)) id 1sigZp-00000008eh5-2Fsc for linux-arm-kernel@lists.infradead.org; Mon, 26 Aug 2024 20:44:38 +0000 DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=redhat.com; s=mimecast20190719; t=1724705076; h=from:from:reply-to:subject:subject:date:date:message-id:message-id: to:to:cc:cc:mime-version:mime-version:content-type:content-type: content-transfer-encoding:content-transfer-encoding: in-reply-to:in-reply-to:references:references; bh=9MPOki3YF/wyEgBnejzItxwjIF1moGifGifGb5OwzY4=; b=FDIAs+ZALbgen0k8cu0tRJgAA+EKXDxjQUttBBD+BpXE+wKtY3Qxb/eO5q1BAPGjAFnNSv mkY5io3dvWZf41lPLORMvb3ES8DbvLBnmbk6SLl/D5+qIgk3ghQkRZdjAsViHpgg5SbktJ mpRWwQPYXj0rZNG92At3Cw/CAWshGIw= Received: from mail-qk1-f199.google.com (mail-qk1-f199.google.com [209.85.222.199]) by relay.mimecast.com with ESMTP with STARTTLS (version=TLSv1.3, cipher=TLS_AES_256_GCM_SHA384) id us-mta-349-jrtGFzGlMhi8AHXCeUrGJA-1; Mon, 26 Aug 2024 16:44:35 -0400 X-MC-Unique: jrtGFzGlMhi8AHXCeUrGJA-1 Received: by mail-qk1-f199.google.com with SMTP id af79cd13be357-7a1dc1e5662so663447285a.2 for ; Mon, 26 Aug 2024 13:44:35 -0700 (PDT) X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20230601; t=1724705075; x=1725309875; h=content-transfer-encoding:mime-version:references:in-reply-to :message-id:date:subject:cc:to:from:x-gm-message-state:from:to:cc :subject:date:message-id:reply-to; bh=9MPOki3YF/wyEgBnejzItxwjIF1moGifGifGb5OwzY4=; b=fR77PybOZZFDEjkX2boiqrx2oxSadRrAAvGt7Mug4rfHIsx/xbKSpcSEG9FfebdBXR Bxh+OfcVPjGFlXFYO5uwHzeqUIqEcNRxzpDdF4RseHCCybWgAWPj5SX/GoafuyBtgz0k mfPTDCCrFieMnLUqLKKuRNR4T24BDsjEh1ubEbfztnlQQD8SG5axCTTgN4Gx4BI+iJnG Ew3w3GVe3kt0VUxho4ty7bMQ3w3trfaC14VkEtRNDH8t7PCeg++02Xj1a9W0pLbuYM82 LWxuq+zZXYtSXZqFjMFAuqqpf/3hCgWlhIWWgO7Ka0lhX3U0eI8wa8ftzURvwvqa8Gjy YDyg== X-Forwarded-Encrypted: i=1; AJvYcCXi7TL5d6lol8tW0RWdF+BGz83QWiJBgaOhyi5XCdIBGHx1p3bPTQAO56cZ4HuzONZo8Q8ZSMf9LtOq//oWv/iE@lists.infradead.org X-Gm-Message-State: AOJu0YyqQe2Xc1heD3V61JJUNhAHrGbE2MTnX0/IikOwlu+WnQ6Pddhl K97r97hlTLNh0Q+EsznLInxJSP0S+vM5Bn52GM4XK/ConJ0kI30w5iFizTexWTxZWJxLkLyN0gF C5ix65E/oZhzdnmLPgAmbtSOTKdDxkWvSV3xNf2lmB6ulHD3keWeqQQ85juI5JE2Cvq6IX/mb X-Received: by 2002:a05:620a:4007:b0:79f:595:f64a with SMTP id af79cd13be357-7a6897b7b14mr1477181785a.58.1724705074680; Mon, 26 Aug 2024 13:44:34 -0700 (PDT) X-Google-Smtp-Source: AGHT+IHpx2qE8XXNwFA/VbcshgEEhM7utf/nDzzwdfyOyUpm1p8Kz8GxiojqL+PSarI5vXlPZn99hQ== X-Received: by 2002:a05:620a:4007:b0:79f:595:f64a with SMTP id af79cd13be357-7a6897b7b14mr1477180385a.58.1724705074328; Mon, 26 Aug 2024 13:44:34 -0700 (PDT) Received: from x1n.redhat.com (pool-99-254-121-117.cpe.net.cable.rogers.com. [99.254.121.117]) by smtp.gmail.com with ESMTPSA id af79cd13be357-7a67f3fd6c1sm491055185a.121.2024.08.26.13.44.32 (version=TLS1_3 cipher=TLS_AES_256_GCM_SHA384 bits=256/256); Mon, 26 Aug 2024 13:44:33 -0700 (PDT) From: Peter Xu To: linux-kernel@vger.kernel.org, linux-mm@kvack.org Cc: Gavin Shan , Catalin Marinas , x86@kernel.org, Ingo Molnar , Andrew Morton , Paolo Bonzini , Dave Hansen , Thomas Gleixner , Alistair Popple , kvm@vger.kernel.org, linux-arm-kernel@lists.infradead.org, Sean Christopherson , peterx@redhat.com, Oscar Salvador , Jason Gunthorpe , Borislav Petkov , Zi Yan , Axel Rasmussen , David Hildenbrand , Yan Zhao , Will Deacon , Kefeng Wang , Alex Williamson Subject: [PATCH v2 19/19] vfio/pci: Implement huge_fault support Date: Mon, 26 Aug 2024 16:43:53 -0400 Message-ID: <20240826204353.2228736-20-peterx@redhat.com> X-Mailer: git-send-email 2.45.0 In-Reply-To: <20240826204353.2228736-1-peterx@redhat.com> References: <20240826204353.2228736-1-peterx@redhat.com> MIME-Version: 1.0 X-Mimecast-Spam-Score: 0 X-Mimecast-Originator: redhat.com X-CRM114-Version: 20100106-BlameMichelson ( TRE 0.8.0 (BSD) ) MR-646709E3 X-CRM114-CacheID: sfid-20240826_134437_688373_48B5D911 X-CRM114-Status: GOOD ( 18.59 ) X-BeenThere: linux-arm-kernel@lists.infradead.org X-Mailman-Version: 2.1.34 Precedence: list List-Id: List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Sender: "linux-arm-kernel" Errors-To: linux-arm-kernel-bounces+linux-arm-kernel=archiver.kernel.org@lists.infradead.org From: Alex Williamson With the addition of pfnmap support in vmf_insert_pfn_{pmd,pud}() we can take advantage of PMD and PUD faults to PCI BAR mmaps and create more efficient mappings. PCI BARs are always a power of two and will typically get at least PMD alignment without userspace even trying. Userspace alignment for PUD mappings is also not too difficult. Consolidate faults through a single handler with a new wrapper for standard single page faults. The pre-faulting behavior of commit d71a989cf5d9 ("vfio/pci: Insert full vma on mmap'd MMIO fault") is removed in this refactoring since huge_fault will cover the bulk of the faults and results in more efficient page table usage. We also want to avoid that pre-faulted single page mappings preempt huge page mappings. Cc: kvm@vger.kernel.org Signed-off-by: Alex Williamson Signed-off-by: Peter Xu --- drivers/vfio/pci/vfio_pci_core.c | 60 +++++++++++++++++++++++--------- 1 file changed, 43 insertions(+), 17 deletions(-) diff --git a/drivers/vfio/pci/vfio_pci_core.c b/drivers/vfio/pci/vfio_pci_core.c index ba0ce0075b2f..2d7478e9a62d 100644 --- a/drivers/vfio/pci/vfio_pci_core.c +++ b/drivers/vfio/pci/vfio_pci_core.c @@ -20,6 +20,7 @@ #include #include #include +#include #include #include #include @@ -1657,14 +1658,20 @@ static unsigned long vma_to_pfn(struct vm_area_struct *vma) return (pci_resource_start(vdev->pdev, index) >> PAGE_SHIFT) + pgoff; } -static vm_fault_t vfio_pci_mmap_fault(struct vm_fault *vmf) +static vm_fault_t vfio_pci_mmap_huge_fault(struct vm_fault *vmf, + unsigned int order) { struct vm_area_struct *vma = vmf->vma; struct vfio_pci_core_device *vdev = vma->vm_private_data; unsigned long pfn, pgoff = vmf->pgoff - vma->vm_pgoff; - unsigned long addr = vma->vm_start; vm_fault_t ret = VM_FAULT_SIGBUS; + if (order && (vmf->address & ((PAGE_SIZE << order) - 1) || + vmf->address + (PAGE_SIZE << order) > vma->vm_end)) { + ret = VM_FAULT_FALLBACK; + goto out; + } + pfn = vma_to_pfn(vma); down_read(&vdev->memory_lock); @@ -1672,30 +1679,49 @@ static vm_fault_t vfio_pci_mmap_fault(struct vm_fault *vmf) if (vdev->pm_runtime_engaged || !__vfio_pci_memory_enabled(vdev)) goto out_unlock; - ret = vmf_insert_pfn(vma, vmf->address, pfn + pgoff); - if (ret & VM_FAULT_ERROR) - goto out_unlock; - - /* - * Pre-fault the remainder of the vma, abort further insertions and - * supress error if fault is encountered during pre-fault. - */ - for (; addr < vma->vm_end; addr += PAGE_SIZE, pfn++) { - if (addr == vmf->address) - continue; - - if (vmf_insert_pfn(vma, addr, pfn) & VM_FAULT_ERROR) - break; + switch (order) { + case 0: + ret = vmf_insert_pfn(vma, vmf->address, pfn + pgoff); + break; +#ifdef CONFIG_ARCH_SUPPORTS_PMD_PFNMAP + case PMD_ORDER: + ret = vmf_insert_pfn_pmd(vmf, __pfn_to_pfn_t(pfn + pgoff, + PFN_DEV), false); + break; +#endif +#ifdef CONFIG_ARCH_SUPPORTS_PUD_PFNMAP + case PUD_ORDER: + ret = vmf_insert_pfn_pud(vmf, __pfn_to_pfn_t(pfn + pgoff, + PFN_DEV), false); + break; +#endif + default: + ret = VM_FAULT_FALLBACK; } out_unlock: up_read(&vdev->memory_lock); +out: + dev_dbg_ratelimited(&vdev->pdev->dev, + "%s(,order = %d) BAR %ld page offset 0x%lx: 0x%x\n", + __func__, order, + vma->vm_pgoff >> + (VFIO_PCI_OFFSET_SHIFT - PAGE_SHIFT), + pgoff, (unsigned int)ret); return ret; } +static vm_fault_t vfio_pci_mmap_page_fault(struct vm_fault *vmf) +{ + return vfio_pci_mmap_huge_fault(vmf, 0); +} + static const struct vm_operations_struct vfio_pci_mmap_ops = { - .fault = vfio_pci_mmap_fault, + .fault = vfio_pci_mmap_page_fault, +#ifdef CONFIG_ARCH_SUPPORTS_HUGE_PFNMAP + .huge_fault = vfio_pci_mmap_huge_fault, +#endif }; int vfio_pci_core_mmap(struct vfio_device *core_vdev, struct vm_area_struct *vma)