From patchwork Mon Aug 26 20:43:35 2024 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Peter Xu X-Patchwork-Id: 13778415 Received: from us-smtp-delivery-124.mimecast.com (us-smtp-delivery-124.mimecast.com [170.10.133.124]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (No client certificate requested) by smtp.subspace.kernel.org (Postfix) with ESMTPS id 51DF61991AD for ; Mon, 26 Aug 2024 20:44:00 +0000 (UTC) Authentication-Results: smtp.subspace.kernel.org; arc=none smtp.client-ip=170.10.133.124 ARC-Seal: i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1724705043; cv=none; b=G3UIFCtHNDLdFR6n43BptXm3ZMphorlUoo317yhAK+y0kV5zi80p9CNGOrCsdqO3WHYX0dIHAGKCq5np4ll0J6cr6tx6CyOSO6+arcORTCl6joNBdJhwRnOYZ3BOugB10ZoN4lwcX2DnCuY/BZCgKjjA1hKCA0NT+sNf8SP6JNI= ARC-Message-Signature: i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1724705043; c=relaxed/simple; bh=bqt9KW78i0VdjWfmtysLtEN3zBnedLCNShRvlblkAoU=; h=From:To:Cc:Subject:Date:Message-ID:In-Reply-To:References: MIME-Version; b=iszhFdfJrp86GDQo6vnyLtbGxXimY988Lm6tzslYc6vJZi9CJ8OveIsUYRNT7dPKDFaM8m+SdY5HuI+ROktwjzOFf/iBROs/zBwgIaOgJPxdKd5fG8TlOPewloAzHfO/ZwviLMwQ0+E2ji0h1Lfz/a77aSHI1CNo2URxSO1BDUI= ARC-Authentication-Results: i=1; smtp.subspace.kernel.org; dmarc=pass (p=none dis=none) header.from=redhat.com; spf=pass smtp.mailfrom=redhat.com; dkim=pass (1024-bit key) header.d=redhat.com header.i=@redhat.com header.b=c3fg5EpP; arc=none smtp.client-ip=170.10.133.124 Authentication-Results: smtp.subspace.kernel.org; dmarc=pass (p=none dis=none) header.from=redhat.com Authentication-Results: smtp.subspace.kernel.org; spf=pass smtp.mailfrom=redhat.com Authentication-Results: smtp.subspace.kernel.org; dkim=pass (1024-bit key) header.d=redhat.com header.i=@redhat.com header.b="c3fg5EpP" DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=redhat.com; s=mimecast20190719; t=1724705040; h=from:from:reply-to:subject:subject:date:date:message-id:message-id: to:to:cc:cc:mime-version:mime-version: content-transfer-encoding:content-transfer-encoding: in-reply-to:in-reply-to:references:references; bh=2iIRbDNVyj1lHjlop0tZMw2JlHFFftRtxMxE0Q8z7r8=; b=c3fg5EpPMAJICZzAXDP+/qtDO0C1D4KaRU1POn9CQfncMnMJ+Kx85jVXZSq3Ur1J6oriyh dfXh0yGD7IIURES8z8TJSkMoKXNayXjuox17jSONloLARvB378svLZyawivjAR8ITBw8tg 925LPXb66qEnH20xdjxHAQg2R5QOpSg= Received: from mail-oo1-f72.google.com (mail-oo1-f72.google.com [209.85.161.72]) by relay.mimecast.com with ESMTP with STARTTLS (version=TLSv1.3, cipher=TLS_AES_256_GCM_SHA384) id us-mta-638-YAVOKIf2PJewiJp4brwPIg-1; Mon, 26 Aug 2024 16:43:59 -0400 X-MC-Unique: YAVOKIf2PJewiJp4brwPIg-1 Received: by mail-oo1-f72.google.com with SMTP id 006d021491bc7-5de842f4435so3259462eaf.1 for ; Mon, 26 Aug 2024 13:43:59 -0700 (PDT) X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20230601; t=1724705038; x=1725309838; h=content-transfer-encoding:mime-version:references:in-reply-to :message-id:date:subject:cc:to:from:x-gm-message-state:from:to:cc :subject:date:message-id:reply-to; bh=2iIRbDNVyj1lHjlop0tZMw2JlHFFftRtxMxE0Q8z7r8=; b=ni9jrMPDsVMBM3jF5Gt2K48IILY0IB173bqxKmOh6Nh/kQ4Trp5zKQt9cUWt4ySFui KVqHx/ZBiuQ+ngfqXwmZwMg+X/KWrbyzRA1iQya93B3LSVKniXTp5IGcDZhYdy+rKxUn qHIhxVfhCzzmwtUCRR0CNWdZqhNBL9Ra8dEVJ5C9fi2wkQgDi9Qx3jE8zdYX8wnpek0a 9Fbo2dodO+EfnljNCYPuED4H5hlqlicxaXiadU47m9HqFvncnRUreRGwAW5HAJ5F4Fof GYRdB3764v1mdxIWR+0R40B3Nzs7VQvm9eK6zA2u8ylQAltGxuqEX4V081I0ZrW33MWX 1xvw== X-Forwarded-Encrypted: i=1; AJvYcCWX1QEb8KBQclr8XgzVONV4BGw6YNtHMwh1912x3kAkByhSQiJKPx44ebD2vi2/JF2dGN0=@vger.kernel.org X-Gm-Message-State: AOJu0YwAQ2re69/8ACY9V6fIeF0FD+l3/x5PhoVPJC03HeOasgJuQhmf HjeE+MsfA5zmlcVwunCuhDDN72vaBdb0Pp3udxhJGk4VA9a3F3uzyldJqgak8DqT/MauTZ87+va peTPa+fiGgTe/Q5mBHixA0bqfm6+P+H5tKoeYprxC905p0+UBrQ== X-Received: by 2002:a05:6358:52ca:b0:1b5:a032:9994 with SMTP id e5c5f4694b2df-1b5ebe5b1b3mr113535555d.0.1724705038384; Mon, 26 Aug 2024 13:43:58 -0700 (PDT) X-Google-Smtp-Source: AGHT+IHkrFRhmUpfOOmJ2YwtqClR5K8A29/GQ8sXabHeZ52sv0b77XENUJCQ9LJLwxTjS7HZUYg0sQ== X-Received: by 2002:a05:6358:52ca:b0:1b5:a032:9994 with SMTP id e5c5f4694b2df-1b5ebe5b1b3mr113532855d.0.1724705038035; Mon, 26 Aug 2024 13:43:58 -0700 (PDT) Received: from x1n.redhat.com (pool-99-254-121-117.cpe.net.cable.rogers.com. [99.254.121.117]) by smtp.gmail.com with ESMTPSA id af79cd13be357-7a67f3fd6c1sm491055185a.121.2024.08.26.13.43.56 (version=TLS1_3 cipher=TLS_AES_256_GCM_SHA384 bits=256/256); Mon, 26 Aug 2024 13:43:57 -0700 (PDT) From: Peter Xu To: linux-kernel@vger.kernel.org, linux-mm@kvack.org Cc: Gavin Shan , Catalin Marinas , x86@kernel.org, Ingo Molnar , Andrew Morton , Paolo Bonzini , Dave Hansen , Thomas Gleixner , Alistair Popple , kvm@vger.kernel.org, linux-arm-kernel@lists.infradead.org, Sean Christopherson , peterx@redhat.com, Oscar Salvador , Jason Gunthorpe , Borislav Petkov , Zi Yan , Axel Rasmussen , David Hildenbrand , Yan Zhao , Will Deacon , Kefeng Wang , Alex Williamson Subject: [PATCH v2 01/19] mm: Introduce ARCH_SUPPORTS_HUGE_PFNMAP and special bits to pmd/pud Date: Mon, 26 Aug 2024 16:43:35 -0400 Message-ID: <20240826204353.2228736-2-peterx@redhat.com> X-Mailer: git-send-email 2.45.0 In-Reply-To: <20240826204353.2228736-1-peterx@redhat.com> References: <20240826204353.2228736-1-peterx@redhat.com> Precedence: bulk X-Mailing-List: kvm@vger.kernel.org List-Id: List-Subscribe: List-Unsubscribe: MIME-Version: 1.0 This patch introduces the option to introduce special pte bit into pmd/puds. Archs can start to define pmd_special / pud_special when supported by selecting the new option. Per-arch support will be added later. Before that, create fallbacks for these helpers so that they are always available. Signed-off-by: Peter Xu --- include/linux/mm.h | 24 ++++++++++++++++++++++++ mm/Kconfig | 13 +++++++++++++ 2 files changed, 37 insertions(+) diff --git a/include/linux/mm.h b/include/linux/mm.h index b0ff06d18c71..d900f15b7650 100644 --- a/include/linux/mm.h +++ b/include/linux/mm.h @@ -2643,6 +2643,30 @@ static inline pte_t pte_mkspecial(pte_t pte) } #endif +#ifndef CONFIG_ARCH_SUPPORTS_PMD_PFNMAP +static inline bool pmd_special(pmd_t pmd) +{ + return false; +} + +static inline pmd_t pmd_mkspecial(pmd_t pmd) +{ + return pmd; +} +#endif /* CONFIG_ARCH_SUPPORTS_PMD_PFNMAP */ + +#ifndef CONFIG_ARCH_SUPPORTS_PUD_PFNMAP +static inline bool pud_special(pud_t pud) +{ + return false; +} + +static inline pud_t pud_mkspecial(pud_t pud) +{ + return pud; +} +#endif /* CONFIG_ARCH_SUPPORTS_PUD_PFNMAP */ + #ifndef CONFIG_ARCH_HAS_PTE_DEVMAP static inline int pte_devmap(pte_t pte) { diff --git a/mm/Kconfig b/mm/Kconfig index 8078a4b3c509..b23913d4e47e 100644 --- a/mm/Kconfig +++ b/mm/Kconfig @@ -875,6 +875,19 @@ endif # TRANSPARENT_HUGEPAGE config PGTABLE_HAS_HUGE_LEAVES def_bool TRANSPARENT_HUGEPAGE || HUGETLB_PAGE +# TODO: Allow to be enabled without THP +config ARCH_SUPPORTS_HUGE_PFNMAP + def_bool n + depends on TRANSPARENT_HUGEPAGE + +config ARCH_SUPPORTS_PMD_PFNMAP + def_bool y + depends on ARCH_SUPPORTS_HUGE_PFNMAP && HAVE_ARCH_TRANSPARENT_HUGEPAGE + +config ARCH_SUPPORTS_PUD_PFNMAP + def_bool y + depends on ARCH_SUPPORTS_HUGE_PFNMAP && HAVE_ARCH_TRANSPARENT_HUGEPAGE_PUD + # # UP and nommu archs use km based percpu allocator # From patchwork Mon Aug 26 20:43:36 2024 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Peter Xu X-Patchwork-Id: 13778416 Received: from us-smtp-delivery-124.mimecast.com (us-smtp-delivery-124.mimecast.com [170.10.133.124]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (No client certificate requested) by smtp.subspace.kernel.org (Postfix) with ESMTPS id 3954619AA57 for ; Mon, 26 Aug 2024 20:44:03 +0000 (UTC) Authentication-Results: smtp.subspace.kernel.org; arc=none smtp.client-ip=170.10.133.124 ARC-Seal: i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1724705044; cv=none; b=OVww9AWDFZ/92L1R6KgWATu1Y+ZMST3vjyXOFCdQYAeFXYXUyV51v5dA/vVTCUhh/o1n0oSLi8VIPrstqhNgdky34CHbU57EVoFZNFBElvMvgoJz5gbEJNqMupsKZhY8xK6qFX3YM1d0vDLTBne+EEIgb2LLrscz0hk7+TVxisY= ARC-Message-Signature: i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1724705044; c=relaxed/simple; bh=34MqcYAeedS1HMCphQeI6ozj3aHeCly2AzsezqBdon4=; h=From:To:Cc:Subject:Date:Message-ID:In-Reply-To:References: MIME-Version; b=pvB3I4YgHaqpnm8frItXTg0eCWJZ93zrCiSoqBZQLgFqcPho6IWYIaNrR0gAZOSZHzaoCnFqDKns/olF2x5o61kkhfK83sKkDCEnGLrd3DKHj6wAk9leC1KquOtQ+onI5oHMJuQGgdAQwhXXxOYyWLmk6EnRl381GGJGBgciwB8= ARC-Authentication-Results: i=1; smtp.subspace.kernel.org; dmarc=pass (p=none dis=none) header.from=redhat.com; spf=pass smtp.mailfrom=redhat.com; dkim=pass (1024-bit key) header.d=redhat.com header.i=@redhat.com header.b=IfR9KALu; arc=none smtp.client-ip=170.10.133.124 Authentication-Results: smtp.subspace.kernel.org; dmarc=pass (p=none dis=none) header.from=redhat.com Authentication-Results: smtp.subspace.kernel.org; spf=pass smtp.mailfrom=redhat.com Authentication-Results: smtp.subspace.kernel.org; dkim=pass (1024-bit key) header.d=redhat.com header.i=@redhat.com header.b="IfR9KALu" DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=redhat.com; s=mimecast20190719; t=1724705042; h=from:from:reply-to:subject:subject:date:date:message-id:message-id: to:to:cc:cc:mime-version:mime-version: content-transfer-encoding:content-transfer-encoding: in-reply-to:in-reply-to:references:references; bh=rCV2q6t08l8ndW0pGxUU3F/GDhRRI1SX3G16E23iQDg=; b=IfR9KALuKb4OfuYAomKCC9JuIVvP/pWCaCcefLaHyxeDS1EdHPwXpMC6HPG8JA1vsECQe9 /fhj6D43nXi18gvck8/ZM1OWuTc0Zl+TaX+rwUIAlIJrJI0H6O+SyKREHC3aDah4nq/gwz 7wfKhnI/5tuYhfkJKYoMgZNC1i1Purg= Received: from mail-qk1-f200.google.com (mail-qk1-f200.google.com [209.85.222.200]) by relay.mimecast.com with ESMTP with STARTTLS (version=TLSv1.3, cipher=TLS_AES_256_GCM_SHA384) id us-mta-488-cOXSj4YUO16SKLLpOTjVpw-1; Mon, 26 Aug 2024 16:44:01 -0400 X-MC-Unique: cOXSj4YUO16SKLLpOTjVpw-1 Received: by mail-qk1-f200.google.com with SMTP id af79cd13be357-7a1d44099a3so628033885a.3 for ; Mon, 26 Aug 2024 13:44:01 -0700 (PDT) X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20230601; t=1724705040; x=1725309840; h=content-transfer-encoding:mime-version:references:in-reply-to :message-id:date:subject:cc:to:from:x-gm-message-state:from:to:cc :subject:date:message-id:reply-to; bh=rCV2q6t08l8ndW0pGxUU3F/GDhRRI1SX3G16E23iQDg=; b=p8OocfDrLG+yT/2DannDUNrveR2n7Wy7JE/fRx2ny+skXruIFc7vQY/ljdKSSlnway DKgDDANJEfTzX3QPMdjFUcLqR18fwPTGMhIE4nJt/iTcfOLrLPGz4k1iTwPT2tQC2Bpy RloDd27e4DiJ2JtvvXGK8h5H9PcsmXobGVvIdA2FdbnmDFsk2v2Z53Sps84sd8j6dfpv 1Ao7G0rkhNOHVfpk1Jyv9YWTiOQn/BaKkFffQ99C/0Lz0jksJJe34JiNwibPZDI73/RF oX50t+vtSx6aRx3y6eZGkZ+TPbSzMkw8jwoB0v54wCmP0YY66uINNsWge05h3mvxD0de nPYg== X-Forwarded-Encrypted: i=1; AJvYcCVtJ1AIAmlWQa1fWPacAVKhQZscryg8y3HUFwoToCF2dmNeD8KDO/XnAjsLmZwjQhA6f+k=@vger.kernel.org X-Gm-Message-State: AOJu0YxC2zN7nVTx06sQUzZFfkG5OA0I1mn3x4sDGUR20SuPc+AjUU8J Tezs95nu8Dz2vOG0TorQfnNeNeK+rdmGVd9VtyoHfF1EJUFFhK81MO0OCLv8OtSbm6I+uixTgO1 Wgx/VlgmVcva9/fk9UhG6d8ZmZ0SvIgxgbnGSsy625Y6h8NWVEg== X-Received: by 2002:a05:620a:28c8:b0:7a1:62ad:9d89 with SMTP id af79cd13be357-7a7e4e6d3c1mr92954885a.64.1724705040432; Mon, 26 Aug 2024 13:44:00 -0700 (PDT) X-Google-Smtp-Source: AGHT+IEW5yVxsMo5HJEI7stPZogQTAiTuzeFOof5FGrO8bEsBYthlHw7SRF31ZNT2saoeMAKCiiqzA== X-Received: by 2002:a05:620a:28c8:b0:7a1:62ad:9d89 with SMTP id af79cd13be357-7a7e4e6d3c1mr92952485a.64.1724705040096; Mon, 26 Aug 2024 13:44:00 -0700 (PDT) Received: from x1n.redhat.com (pool-99-254-121-117.cpe.net.cable.rogers.com. [99.254.121.117]) by smtp.gmail.com with ESMTPSA id af79cd13be357-7a67f3fd6c1sm491055185a.121.2024.08.26.13.43.58 (version=TLS1_3 cipher=TLS_AES_256_GCM_SHA384 bits=256/256); Mon, 26 Aug 2024 13:43:59 -0700 (PDT) From: Peter Xu To: linux-kernel@vger.kernel.org, linux-mm@kvack.org Cc: Gavin Shan , Catalin Marinas , x86@kernel.org, Ingo Molnar , Andrew Morton , Paolo Bonzini , Dave Hansen , Thomas Gleixner , Alistair Popple , kvm@vger.kernel.org, linux-arm-kernel@lists.infradead.org, Sean Christopherson , peterx@redhat.com, Oscar Salvador , Jason Gunthorpe , Borislav Petkov , Zi Yan , Axel Rasmussen , David Hildenbrand , Yan Zhao , Will Deacon , Kefeng Wang , Alex Williamson , Matthew Wilcox , "Aneesh Kumar K . V" Subject: [PATCH v2 02/19] mm: Drop is_huge_zero_pud() Date: Mon, 26 Aug 2024 16:43:36 -0400 Message-ID: <20240826204353.2228736-3-peterx@redhat.com> X-Mailer: git-send-email 2.45.0 In-Reply-To: <20240826204353.2228736-1-peterx@redhat.com> References: <20240826204353.2228736-1-peterx@redhat.com> Precedence: bulk X-Mailing-List: kvm@vger.kernel.org List-Id: List-Subscribe: List-Unsubscribe: MIME-Version: 1.0 It constantly returns false since 2017. One assertion is added in 2019 but it should never have triggered, IOW it means what is checked should be asserted instead. If it didn't exist for 7 years maybe it's good idea to remove it and only add it when it comes. Cc: Matthew Wilcox Cc: Aneesh Kumar K.V Acked-by: David Hildenbrand Reviewed-by: Jason Gunthorpe Signed-off-by: Peter Xu --- include/linux/huge_mm.h | 10 ---------- mm/huge_memory.c | 13 +------------ 2 files changed, 1 insertion(+), 22 deletions(-) diff --git a/include/linux/huge_mm.h b/include/linux/huge_mm.h index 4902e2f7e896..b550b5a248bb 100644 --- a/include/linux/huge_mm.h +++ b/include/linux/huge_mm.h @@ -433,11 +433,6 @@ static inline bool is_huge_zero_pmd(pmd_t pmd) return pmd_present(pmd) && READ_ONCE(huge_zero_pfn) == pmd_pfn(pmd); } -static inline bool is_huge_zero_pud(pud_t pud) -{ - return false; -} - struct folio *mm_get_huge_zero_folio(struct mm_struct *mm); void mm_put_huge_zero_folio(struct mm_struct *mm); @@ -578,11 +573,6 @@ static inline bool is_huge_zero_pmd(pmd_t pmd) return false; } -static inline bool is_huge_zero_pud(pud_t pud) -{ - return false; -} - static inline void mm_put_huge_zero_folio(struct mm_struct *mm) { return; diff --git a/mm/huge_memory.c b/mm/huge_memory.c index a81eab98d6b8..3f74b09ada38 100644 --- a/mm/huge_memory.c +++ b/mm/huge_memory.c @@ -1429,10 +1429,8 @@ static void insert_pfn_pud(struct vm_area_struct *vma, unsigned long addr, ptl = pud_lock(mm, pud); if (!pud_none(*pud)) { if (write) { - if (pud_pfn(*pud) != pfn_t_to_pfn(pfn)) { - WARN_ON_ONCE(!is_huge_zero_pud(*pud)); + if (WARN_ON_ONCE(pud_pfn(*pud) != pfn_t_to_pfn(pfn))) goto out_unlock; - } entry = pud_mkyoung(*pud); entry = maybe_pud_mkwrite(pud_mkdirty(entry), vma); if (pudp_set_access_flags(vma, addr, pud, entry, 1)) @@ -1680,15 +1678,6 @@ int copy_huge_pud(struct mm_struct *dst_mm, struct mm_struct *src_mm, if (unlikely(!pud_trans_huge(pud) && !pud_devmap(pud))) goto out_unlock; - /* - * When page table lock is held, the huge zero pud should not be - * under splitting since we don't split the page itself, only pud to - * a page table. - */ - if (is_huge_zero_pud(pud)) { - /* No huge zero pud yet */ - } - /* * TODO: once we support anonymous pages, use * folio_try_dup_anon_rmap_*() and split if duplicating fails. From patchwork Mon Aug 26 20:43:37 2024 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Peter Xu X-Patchwork-Id: 13778417 Received: from us-smtp-delivery-124.mimecast.com (us-smtp-delivery-124.mimecast.com [170.10.133.124]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (No client certificate requested) by smtp.subspace.kernel.org (Postfix) with ESMTPS id 06DE219B5A3 for ; Mon, 26 Aug 2024 20:44:05 +0000 (UTC) Authentication-Results: smtp.subspace.kernel.org; arc=none smtp.client-ip=170.10.133.124 ARC-Seal: i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1724705047; cv=none; b=Ft5Mrv8Jjz8HZpof4VEJbAmIzLgp7uUkLDcf0oujKa9Z8HGy82ElTUZwVO6PqW3zHRoe/0uE2xTilyifokPCBomed/hxCIm/5ugyAN+fqe/JfYRp+VZnu+M+AIUvM/aTu7NKGexEf4iCRBVeSgL28pcLDVkqAYhwAAEgZtcZPJs= ARC-Message-Signature: i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1724705047; c=relaxed/simple; bh=cBR1CRPKBp17TA49iECdr5UwUokaTW4WPHX9nl57T+I=; h=From:To:Cc:Subject:Date:Message-ID:In-Reply-To:References: MIME-Version; b=GIhY2LRzTqEfiq4Sn3ajQ3UU+fnAtIDd7Ssy5mjAMkLxM2edUC7Q5nm7b4lm6te+EmXFFGi0UxAysSEzVrSGeiZ5TwbOqMV854Ss5kLeJWZqjW8KqXNWRyxpjXCqI9gs/7TY/S9/kl955CUAIxAKmgxNLKOzdc6C4wJhdaLtx28= ARC-Authentication-Results: i=1; smtp.subspace.kernel.org; dmarc=pass (p=none dis=none) header.from=redhat.com; spf=pass smtp.mailfrom=redhat.com; dkim=pass (1024-bit key) header.d=redhat.com header.i=@redhat.com header.b=bvNdCgRH; arc=none smtp.client-ip=170.10.133.124 Authentication-Results: smtp.subspace.kernel.org; dmarc=pass (p=none dis=none) header.from=redhat.com Authentication-Results: smtp.subspace.kernel.org; spf=pass smtp.mailfrom=redhat.com Authentication-Results: smtp.subspace.kernel.org; dkim=pass (1024-bit key) header.d=redhat.com header.i=@redhat.com header.b="bvNdCgRH" DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=redhat.com; s=mimecast20190719; t=1724705045; h=from:from:reply-to:subject:subject:date:date:message-id:message-id: to:to:cc:cc:mime-version:mime-version: content-transfer-encoding:content-transfer-encoding: in-reply-to:in-reply-to:references:references; bh=gD9ggS8OhpZf/NsZoH3fZjUIikns33eAnWwzmGyUsEI=; b=bvNdCgRHgeQDfPqTbXy7AOUtvm0fWfrTmB5EjOt3NmkIiJxJA40jKncg7xorr+Rj266uPC E0a9fSq06CDyW12F62o+QzYyGibPC/6PVULu0Q0IaiXSKNNh2HZfbDn0bENMaFACv8QVvq zByNYMCE1/rDAndJdHjwHuzNPpWfbOo= Received: from mail-qk1-f198.google.com (mail-qk1-f198.google.com [209.85.222.198]) by relay.mimecast.com with ESMTP with STARTTLS (version=TLSv1.3, cipher=TLS_AES_256_GCM_SHA384) id us-mta-230-ELkz5TTNOPC7Seit0yP3-A-1; Mon, 26 Aug 2024 16:44:03 -0400 X-MC-Unique: ELkz5TTNOPC7Seit0yP3-A-1 Received: by mail-qk1-f198.google.com with SMTP id af79cd13be357-7a66bf35402so589827285a.3 for ; Mon, 26 Aug 2024 13:44:03 -0700 (PDT) X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20230601; t=1724705043; x=1725309843; h=content-transfer-encoding:mime-version:references:in-reply-to :message-id:date:subject:cc:to:from:x-gm-message-state:from:to:cc :subject:date:message-id:reply-to; bh=gD9ggS8OhpZf/NsZoH3fZjUIikns33eAnWwzmGyUsEI=; b=E0n5urttRui6J55hTbWmU5SWe/lqSifv+7mez8xxLc+Rw0WRrY6cPzw5BijENnfTP7 9n7Dzo1+Gk9e6crfV8DtgrWqCfmyOinteQ8j7jGnFd2EoJP/e3aIOp7SOI3WFxD+9Z4T 7fQOLPsmhCCTTTS6mhlLMQvalIDqPkxOZ3U0ZEgcxplpS6drvaPEXonGzCnlF0Er0I/f NucI3Vj5E29zej0MPzSNL32+2Y08dw8/h2b+PnJDVCAmOKomRVQkmKwB37PvvwPbiykI PKoZidBMJLS4ka3kP22BDlO/xWsSxNSul0Skset4ugdyauQuB8BKaE7vbwqnqAL/kYkv 9dfQ== X-Forwarded-Encrypted: i=1; AJvYcCUugjB1TKPmwc9g36yk1kWv6Gewh2VN+Hc3mnl5pQ2+B6qUzi+9VOLHUcM8+x6ZomM02zo=@vger.kernel.org X-Gm-Message-State: AOJu0Yxdn0QAr4x+MJq5ErEJgLfLKjiTkYSuBskpeYVp4RSveFQKWi1Y ZqeDyfYUK9O+7s2N0je7HNqV27Wt2KOlruFFhP0sBQxfsAaPhN/03LWpSOZSzt7EXIEF6fML/8b GKzprfLop/+JOr4NFsaCUy7GvIPn1eb3Nkr2BiF1/WIIuiBWKew== X-Received: by 2002:a05:620a:24d5:b0:7a3:6dd9:ef9d with SMTP id af79cd13be357-7a6897c6e76mr1277666885a.62.1724705042591; Mon, 26 Aug 2024 13:44:02 -0700 (PDT) X-Google-Smtp-Source: AGHT+IHIqrjqvJMs9Ym5fhhSHDCZDk2wQ6104PL5R9Vk0hTAFm1CUlIqFGQmzGdyDHGQhWdDyxioXg== X-Received: by 2002:a05:620a:24d5:b0:7a3:6dd9:ef9d with SMTP id af79cd13be357-7a6897c6e76mr1277664185a.62.1724705042003; Mon, 26 Aug 2024 13:44:02 -0700 (PDT) Received: from x1n.redhat.com (pool-99-254-121-117.cpe.net.cable.rogers.com. [99.254.121.117]) by smtp.gmail.com with ESMTPSA id af79cd13be357-7a67f3fd6c1sm491055185a.121.2024.08.26.13.44.00 (version=TLS1_3 cipher=TLS_AES_256_GCM_SHA384 bits=256/256); Mon, 26 Aug 2024 13:44:01 -0700 (PDT) From: Peter Xu To: linux-kernel@vger.kernel.org, linux-mm@kvack.org Cc: Gavin Shan , Catalin Marinas , x86@kernel.org, Ingo Molnar , Andrew Morton , Paolo Bonzini , Dave Hansen , Thomas Gleixner , Alistair Popple , kvm@vger.kernel.org, linux-arm-kernel@lists.infradead.org, Sean Christopherson , peterx@redhat.com, Oscar Salvador , Jason Gunthorpe , Borislav Petkov , Zi Yan , Axel Rasmussen , David Hildenbrand , Yan Zhao , Will Deacon , Kefeng Wang , Alex Williamson Subject: [PATCH v2 03/19] mm: Mark special bits for huge pfn mappings when inject Date: Mon, 26 Aug 2024 16:43:37 -0400 Message-ID: <20240826204353.2228736-4-peterx@redhat.com> X-Mailer: git-send-email 2.45.0 In-Reply-To: <20240826204353.2228736-1-peterx@redhat.com> References: <20240826204353.2228736-1-peterx@redhat.com> Precedence: bulk X-Mailing-List: kvm@vger.kernel.org List-Id: List-Subscribe: List-Unsubscribe: MIME-Version: 1.0 We need these special bits to be around on pfnmaps. Mark properly for !devmap case, reflecting that there's no page struct backing the entry. Reviewed-by: Jason Gunthorpe Signed-off-by: Peter Xu Acked-by: David Hildenbrand --- mm/huge_memory.c | 4 ++++ 1 file changed, 4 insertions(+) diff --git a/mm/huge_memory.c b/mm/huge_memory.c index 3f74b09ada38..dec17d09390f 100644 --- a/mm/huge_memory.c +++ b/mm/huge_memory.c @@ -1346,6 +1346,8 @@ static void insert_pfn_pmd(struct vm_area_struct *vma, unsigned long addr, entry = pmd_mkhuge(pfn_t_pmd(pfn, prot)); if (pfn_t_devmap(pfn)) entry = pmd_mkdevmap(entry); + else + entry = pmd_mkspecial(entry); if (write) { entry = pmd_mkyoung(pmd_mkdirty(entry)); entry = maybe_pmd_mkwrite(entry, vma); @@ -1442,6 +1444,8 @@ static void insert_pfn_pud(struct vm_area_struct *vma, unsigned long addr, entry = pud_mkhuge(pfn_t_pud(pfn, prot)); if (pfn_t_devmap(pfn)) entry = pud_mkdevmap(entry); + else + entry = pud_mkspecial(entry); if (write) { entry = pud_mkyoung(pud_mkdirty(entry)); entry = maybe_pud_mkwrite(entry, vma); From patchwork Mon Aug 26 20:43:38 2024 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Peter Xu X-Patchwork-Id: 13778418 Received: from us-smtp-delivery-124.mimecast.com (us-smtp-delivery-124.mimecast.com [170.10.129.124]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (No client certificate requested) by smtp.subspace.kernel.org (Postfix) with ESMTPS id 4B10C19D06D for ; Mon, 26 Aug 2024 20:44:08 +0000 (UTC) Authentication-Results: smtp.subspace.kernel.org; arc=none smtp.client-ip=170.10.129.124 ARC-Seal: i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1724705049; cv=none; b=sQRJmjTuno/yq3KdAwYkOw2hbQQRqmtrclQW3J8tGWujHXo+qLPQpIqDYO28i6VcjtFb02VpEWpEIxfNxebvfjxq1iGk5EHuQz7r7ziK1vTCJaRdA9Qw5nMTcZsPP+5V8HC0gLOxzPzX8gFLWGmutzBTSqgynVutT6rq9wyAdgY= ARC-Message-Signature: i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1724705049; c=relaxed/simple; bh=aAVG1dQkK4CK+bGKDFmGiwLRBBmSB50x/ZhpxKIy1+Q=; h=From:To:Cc:Subject:Date:Message-ID:In-Reply-To:References: MIME-Version; b=YhUsL5grAJQDnCgtC6sp258PaOPzpbvtaAE8h5pj/jkc2/+HX69hvTCTxXdtE+BlCMD57W2EHHiWeZlsRKUlXkzRNZpJCu+OE6dhGvfNMWPlbHBlyxiQo/ZKUS2h5DDpLON2kJt92je6zfuPcIMOmpw9PZpTJK124qDnT9ioy3A= ARC-Authentication-Results: i=1; smtp.subspace.kernel.org; dmarc=pass (p=none dis=none) header.from=redhat.com; spf=pass smtp.mailfrom=redhat.com; dkim=pass (1024-bit key) header.d=redhat.com header.i=@redhat.com header.b=U/IvSmSi; arc=none smtp.client-ip=170.10.129.124 Authentication-Results: smtp.subspace.kernel.org; dmarc=pass (p=none dis=none) header.from=redhat.com Authentication-Results: smtp.subspace.kernel.org; spf=pass smtp.mailfrom=redhat.com Authentication-Results: smtp.subspace.kernel.org; dkim=pass (1024-bit key) header.d=redhat.com header.i=@redhat.com header.b="U/IvSmSi" DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=redhat.com; s=mimecast20190719; t=1724705047; h=from:from:reply-to:subject:subject:date:date:message-id:message-id: to:to:cc:cc:mime-version:mime-version: content-transfer-encoding:content-transfer-encoding: in-reply-to:in-reply-to:references:references; bh=p41Ko1t0SYR+aQ+NnkPoKcQjWVRbF5iAoso+I7QNClw=; b=U/IvSmSirmaWE7Ef8vFO/UCoqdpHgwXgcB21EQ0xYAHCKX4ww5M7QJgkYHrqhIRaZVKKDt oQsRiF5dePjh3ynn/KKuf9gP/RuOtqkhCP+sSewgmtLnNcouhljEO/itrzqKrSM+9bj9kx oM5DnUZpkoWk8wulDD2hQSsA9asi9Sk= Received: from mail-oo1-f71.google.com (mail-oo1-f71.google.com [209.85.161.71]) by relay.mimecast.com with ESMTP with STARTTLS (version=TLSv1.3, cipher=TLS_AES_256_GCM_SHA384) id us-mta-25-DObyiIkoOo-hFW8LKiW-RQ-1; Mon, 26 Aug 2024 16:44:05 -0400 X-MC-Unique: DObyiIkoOo-hFW8LKiW-RQ-1 Received: by mail-oo1-f71.google.com with SMTP id 006d021491bc7-5d5cec5ab62so6030656eaf.3 for ; Mon, 26 Aug 2024 13:44:05 -0700 (PDT) X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20230601; t=1724705044; x=1725309844; h=content-transfer-encoding:mime-version:references:in-reply-to :message-id:date:subject:cc:to:from:x-gm-message-state:from:to:cc :subject:date:message-id:reply-to; bh=p41Ko1t0SYR+aQ+NnkPoKcQjWVRbF5iAoso+I7QNClw=; b=t4WSflEGld92yFA6kAml/rQJ554joG/l472j4gAscmoursftaWxuAlOWpsa/TQWHxE b9bIEome23YISSCff93JmxMCtsZvLDZkDzSdDNcP1GaRsivkYvCmTM1oXxew6m0XYrsd Kxv3MI35xBouyIUHao+sBk5IDlmCHlamIlS2ntwwfeGZPke/6uq+o860LxJdH8lttCaN LaPgsCu8TUMno4/eP9/DhuS/45zHKnMml2N9lJEqzJTpSJIBeGVA+7yYQN7J1+jxLEnW RAo+nOPBXuGdPqW8wffwQqi5yYT2/ZVgjfMjc3hD+3xRCJZV7mVgoru4weXUltzaemcC glMQ== X-Forwarded-Encrypted: i=1; AJvYcCXtB3zXajqySTRSvxhfwZP3MMHJa56ynYhyhkY66lPgIIRKw9cym2KJT7+dmEE74Sknn3Y=@vger.kernel.org X-Gm-Message-State: AOJu0YwkIj2kBOuRbimzw+kHCdmjEhkpFgICpvPft8NsvhPymTDwFrxS w1rrqKy7wBmDUljPFUj0r445lIz9OjbroaPIpI4H469bBOcbOwCQOLKQ216+BZ7CQ9TVYF8TAiG jVz77EnFSY83z0SmAIc0cBPTTRdgg0tIyejapfmZuxYeD7j/AiA== X-Received: by 2002:a05:6358:ca5:b0:1b5:ecce:b760 with SMTP id e5c5f4694b2df-1b5ecceb805mr68656955d.21.1724705044544; Mon, 26 Aug 2024 13:44:04 -0700 (PDT) X-Google-Smtp-Source: AGHT+IGqSprkmyPyKIDES0Mxu2D+0b60kqL8Leav+5Mf5w5kuR6qgHz2l36Zs0Y1NhYRXD/bnn90Fg== X-Received: by 2002:a05:6358:ca5:b0:1b5:ecce:b760 with SMTP id e5c5f4694b2df-1b5ecceb805mr68653955d.21.1724705044223; Mon, 26 Aug 2024 13:44:04 -0700 (PDT) Received: from x1n.redhat.com (pool-99-254-121-117.cpe.net.cable.rogers.com. [99.254.121.117]) by smtp.gmail.com with ESMTPSA id af79cd13be357-7a67f3fd6c1sm491055185a.121.2024.08.26.13.44.02 (version=TLS1_3 cipher=TLS_AES_256_GCM_SHA384 bits=256/256); Mon, 26 Aug 2024 13:44:03 -0700 (PDT) From: Peter Xu To: linux-kernel@vger.kernel.org, linux-mm@kvack.org Cc: Gavin Shan , Catalin Marinas , x86@kernel.org, Ingo Molnar , Andrew Morton , Paolo Bonzini , Dave Hansen , Thomas Gleixner , Alistair Popple , kvm@vger.kernel.org, linux-arm-kernel@lists.infradead.org, Sean Christopherson , peterx@redhat.com, Oscar Salvador , Jason Gunthorpe , Borislav Petkov , Zi Yan , Axel Rasmussen , David Hildenbrand , Yan Zhao , Will Deacon , Kefeng Wang , Alex Williamson , Matthew Wilcox , Ryan Roberts Subject: [PATCH v2 04/19] mm: Allow THP orders for PFNMAPs Date: Mon, 26 Aug 2024 16:43:38 -0400 Message-ID: <20240826204353.2228736-5-peterx@redhat.com> X-Mailer: git-send-email 2.45.0 In-Reply-To: <20240826204353.2228736-1-peterx@redhat.com> References: <20240826204353.2228736-1-peterx@redhat.com> Precedence: bulk X-Mailing-List: kvm@vger.kernel.org List-Id: List-Subscribe: List-Unsubscribe: MIME-Version: 1.0 This enables PFNMAPs to be mapped at either pmd/pud layers. Generalize the dax case into vma_is_special_huge() so as to cover both. Meanwhile, rename the macro to THP_ORDERS_ALL_SPECIAL. Cc: Matthew Wilcox Cc: Gavin Shan Cc: Ryan Roberts Cc: David Hildenbrand Cc: Zi Yan Reviewed-by: Jason Gunthorpe Signed-off-by: Peter Xu Acked-by: David Hildenbrand --- include/linux/huge_mm.h | 6 +++--- mm/huge_memory.c | 4 ++-- 2 files changed, 5 insertions(+), 5 deletions(-) diff --git a/include/linux/huge_mm.h b/include/linux/huge_mm.h index b550b5a248bb..4da102b74a8c 100644 --- a/include/linux/huge_mm.h +++ b/include/linux/huge_mm.h @@ -76,9 +76,9 @@ extern struct kobj_attribute thpsize_shmem_enabled_attr; /* * Mask of all large folio orders supported for file THP. Folios in a DAX * file is never split and the MAX_PAGECACHE_ORDER limit does not apply to - * it. + * it. Same to PFNMAPs where there's neither page* nor pagecache. */ -#define THP_ORDERS_ALL_FILE_DAX \ +#define THP_ORDERS_ALL_SPECIAL \ (BIT(PMD_ORDER) | BIT(PUD_ORDER)) #define THP_ORDERS_ALL_FILE_DEFAULT \ ((BIT(MAX_PAGECACHE_ORDER + 1) - 1) & ~BIT(0)) @@ -87,7 +87,7 @@ extern struct kobj_attribute thpsize_shmem_enabled_attr; * Mask of all large folio orders supported for THP. */ #define THP_ORDERS_ALL \ - (THP_ORDERS_ALL_ANON | THP_ORDERS_ALL_FILE_DAX | THP_ORDERS_ALL_FILE_DEFAULT) + (THP_ORDERS_ALL_ANON | THP_ORDERS_ALL_SPECIAL | THP_ORDERS_ALL_FILE_DEFAULT) #define TVA_SMAPS (1 << 0) /* Will be used for procfs */ #define TVA_IN_PF (1 << 1) /* Page fault handler */ diff --git a/mm/huge_memory.c b/mm/huge_memory.c index dec17d09390f..e2c314f631f3 100644 --- a/mm/huge_memory.c +++ b/mm/huge_memory.c @@ -96,8 +96,8 @@ unsigned long __thp_vma_allowable_orders(struct vm_area_struct *vma, /* Check the intersection of requested and supported orders. */ if (vma_is_anonymous(vma)) supported_orders = THP_ORDERS_ALL_ANON; - else if (vma_is_dax(vma)) - supported_orders = THP_ORDERS_ALL_FILE_DAX; + else if (vma_is_special_huge(vma)) + supported_orders = THP_ORDERS_ALL_SPECIAL; else supported_orders = THP_ORDERS_ALL_FILE_DEFAULT; From patchwork Mon Aug 26 20:43:39 2024 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Peter Xu X-Patchwork-Id: 13778419 Received: from us-smtp-delivery-124.mimecast.com (us-smtp-delivery-124.mimecast.com [170.10.133.124]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (No client certificate requested) by smtp.subspace.kernel.org (Postfix) with ESMTPS id DDD4219AD7E for ; Mon, 26 Aug 2024 20:44:09 +0000 (UTC) Authentication-Results: smtp.subspace.kernel.org; arc=none smtp.client-ip=170.10.133.124 ARC-Seal: i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1724705051; cv=none; b=H0Ey26y//mOI+up/9/eJPfPArxmQBaOR849qQRpwKDxusXHfPW9GfbVR7ANW/it407b6rY5lsO+nyxFv4z12rTbcV9TniDCI6RoH7QmO/cceqQuUCizhEYHWpJnpuzlEW7TzQpWf6RIWodI0hflMQ6FC9Vg1curR2F+y2hqbOoU= ARC-Message-Signature: i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1724705051; c=relaxed/simple; bh=B4CjEPEBNhW/7ay+iPhYCq//7mfGSaIpgiOs2kdAYJU=; h=From:To:Cc:Subject:Date:Message-ID:In-Reply-To:References: MIME-Version; b=H71KcEso55ilbzznE3kSKc7wI92YoBoA6jbhirUbuI1vSS+IOxQ2k8izrlB0IuCGXwC0WCdzsxQkOcWYyfo0pUMG5v9RLgs4t7qnwaH1rFj6gWuvcQO9Q4C9LhqhNpfaV9AhX6rW4x8CtjP7x0JlqBCjF8bLQLpwg3Z59/QHZCc= ARC-Authentication-Results: i=1; smtp.subspace.kernel.org; dmarc=pass (p=none dis=none) header.from=redhat.com; spf=pass smtp.mailfrom=redhat.com; dkim=pass (1024-bit key) header.d=redhat.com header.i=@redhat.com header.b=PRhodrri; arc=none smtp.client-ip=170.10.133.124 Authentication-Results: smtp.subspace.kernel.org; dmarc=pass (p=none dis=none) header.from=redhat.com Authentication-Results: smtp.subspace.kernel.org; spf=pass smtp.mailfrom=redhat.com Authentication-Results: smtp.subspace.kernel.org; dkim=pass (1024-bit key) header.d=redhat.com header.i=@redhat.com header.b="PRhodrri" DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=redhat.com; s=mimecast20190719; t=1724705048; h=from:from:reply-to:subject:subject:date:date:message-id:message-id: to:to:cc:cc:mime-version:mime-version: content-transfer-encoding:content-transfer-encoding: in-reply-to:in-reply-to:references:references; bh=SFHgAMnpeJHqDMa0ADr1pyvSduy44GuxF04BOe5aqcw=; b=PRhodrriEksh5vkQhp6TNKCFjCEmQviivZkIYfyfSdsu3iLOiFE3rLCiat840hn8AsUYtI mrAahOnD5KnoS7Cv2yguSMHLF3+tnjOkxihh2ch9kSDjoi8Z6kckxVcLmrDBJ98/B+UpiI ryPMwZscBjKd09hfP+G0iK90KMPIiwo= Received: from mail-qk1-f200.google.com (mail-qk1-f200.google.com [209.85.222.200]) by relay.mimecast.com with ESMTP with STARTTLS (version=TLSv1.3, cipher=TLS_AES_256_GCM_SHA384) id us-mta-548-WwII-ndHOpCLLgneJNl2JA-1; Mon, 26 Aug 2024 16:44:07 -0400 X-MC-Unique: WwII-ndHOpCLLgneJNl2JA-1 Received: by mail-qk1-f200.google.com with SMTP id af79cd13be357-7a1de8a2adbso643965085a.1 for ; Mon, 26 Aug 2024 13:44:07 -0700 (PDT) X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20230601; t=1724705047; x=1725309847; h=content-transfer-encoding:mime-version:references:in-reply-to :message-id:date:subject:cc:to:from:x-gm-message-state:from:to:cc :subject:date:message-id:reply-to; bh=SFHgAMnpeJHqDMa0ADr1pyvSduy44GuxF04BOe5aqcw=; b=PrgL+ZFpeWG0wqoAfesuI5qV5Hjs4moM87X8B6umKErJXAxswBaDKEi3weVDCdj4TF MliencXTVkW1bJn6a3lPwWdrkOy3QFeDH96VhG37IgXK2yWricCrzRyF38r7X/xHpH4Q ivg6sTReY6UlulYrY5hsXud87+w0EHQowQDs127QDljPgW2zYgFtj8miA9pihzwEseKQ 5oxZWYbuFRVyn08H04FTKVR7ZE3PGpE7R6V9Q3Hbyva09JctgMw7rTOY+1qK03u0RzEt IhtzIGT7sOL7l2Vp6eYbU8K5fMG9+U9A1xbTlCS99W9upHQuY0YddxuhXEyL2sfbKqm4 a9JA== X-Forwarded-Encrypted: i=1; AJvYcCXseu7q2rckjU+LUzy+llv3PPpysXh+YEckhLDMG1koqjQbHvqOLZo9DMoEe+uTITOCt9s=@vger.kernel.org X-Gm-Message-State: AOJu0YybjM3AXWBnzhBfdvoQU1exwkmv9b41Xgx9bIvJ6v1UbKlaUhhJ K0H0ki+Yu7xP3hCW+KhxRGh4SJcx4XqdUnRlKZgOSR+2ghsTU4KW3YPTT+urgXYc82I9qsoA0Ur OYnZ7CPYlgvKcTyHELMyqt5VZjl7wyDXi2hRkgxUi8zHvF8eQhg== X-Received: by 2002:a05:620a:2a14:b0:7a2:ccd:9672 with SMTP id af79cd13be357-7a7e4e6144fmr67568685a.67.1724705046879; Mon, 26 Aug 2024 13:44:06 -0700 (PDT) X-Google-Smtp-Source: AGHT+IG++rZwRsHPi3LEX2hBpbUb+Bhnwx97nv6F7Fy5QJm5RgQYxXJ7nF8aD8ykhSm+rXvWrvbUzQ== X-Received: by 2002:a05:620a:2a14:b0:7a2:ccd:9672 with SMTP id af79cd13be357-7a7e4e6144fmr67565885a.67.1724705046455; Mon, 26 Aug 2024 13:44:06 -0700 (PDT) Received: from x1n.redhat.com (pool-99-254-121-117.cpe.net.cable.rogers.com. [99.254.121.117]) by smtp.gmail.com with ESMTPSA id af79cd13be357-7a67f3fd6c1sm491055185a.121.2024.08.26.13.44.04 (version=TLS1_3 cipher=TLS_AES_256_GCM_SHA384 bits=256/256); Mon, 26 Aug 2024 13:44:05 -0700 (PDT) From: Peter Xu To: linux-kernel@vger.kernel.org, linux-mm@kvack.org Cc: Gavin Shan , Catalin Marinas , x86@kernel.org, Ingo Molnar , Andrew Morton , Paolo Bonzini , Dave Hansen , Thomas Gleixner , Alistair Popple , kvm@vger.kernel.org, linux-arm-kernel@lists.infradead.org, Sean Christopherson , peterx@redhat.com, Oscar Salvador , Jason Gunthorpe , Borislav Petkov , Zi Yan , Axel Rasmussen , David Hildenbrand , Yan Zhao , Will Deacon , Kefeng Wang , Alex Williamson Subject: [PATCH v2 05/19] mm/gup: Detect huge pfnmap entries in gup-fast Date: Mon, 26 Aug 2024 16:43:39 -0400 Message-ID: <20240826204353.2228736-6-peterx@redhat.com> X-Mailer: git-send-email 2.45.0 In-Reply-To: <20240826204353.2228736-1-peterx@redhat.com> References: <20240826204353.2228736-1-peterx@redhat.com> Precedence: bulk X-Mailing-List: kvm@vger.kernel.org List-Id: List-Subscribe: List-Unsubscribe: MIME-Version: 1.0 Since gup-fast doesn't have the vma reference, teach it to detect such huge pfnmaps by checking the special bit for pmd/pud too, just like ptes. Acked-by: David Hildenbrand Reviewed-by: Jason Gunthorpe Signed-off-by: Peter Xu --- mm/gup.c | 6 ++++++ 1 file changed, 6 insertions(+) diff --git a/mm/gup.c b/mm/gup.c index d19884e097fd..a49f67a512ee 100644 --- a/mm/gup.c +++ b/mm/gup.c @@ -3038,6 +3038,9 @@ static int gup_fast_pmd_leaf(pmd_t orig, pmd_t *pmdp, unsigned long addr, if (!pmd_access_permitted(orig, flags & FOLL_WRITE)) return 0; + if (pmd_special(orig)) + return 0; + if (pmd_devmap(orig)) { if (unlikely(flags & FOLL_LONGTERM)) return 0; @@ -3082,6 +3085,9 @@ static int gup_fast_pud_leaf(pud_t orig, pud_t *pudp, unsigned long addr, if (!pud_access_permitted(orig, flags & FOLL_WRITE)) return 0; + if (pud_special(orig)) + return 0; + if (pud_devmap(orig)) { if (unlikely(flags & FOLL_LONGTERM)) return 0; From patchwork Mon Aug 26 20:43:40 2024 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Peter Xu X-Patchwork-Id: 13778420 Received: from us-smtp-delivery-124.mimecast.com (us-smtp-delivery-124.mimecast.com [170.10.129.124]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (No client certificate requested) by smtp.subspace.kernel.org (Postfix) with ESMTPS id EB08A19D89D for ; Mon, 26 Aug 2024 20:44:11 +0000 (UTC) Authentication-Results: smtp.subspace.kernel.org; arc=none smtp.client-ip=170.10.129.124 ARC-Seal: i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1724705053; cv=none; b=OYycL/zJlcqsmC+cExvAj26UAMZRETkf2uYCJLKsenvZ4Pr8RrXqO/LegovL2aWlWAG9fIQkFQaHHGCP1xigUhuINIG35pxaRfB6q9j14JJxHiSOrzvLtR+PXtdpX2o+BiE0LBXvi0VM8seQK9g6xdfu78MXBCF/DoM7j4SAcZA= ARC-Message-Signature: i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1724705053; c=relaxed/simple; bh=0JOvh9d6Ck9MlI1IecLNRwn4z9Son10cjOiVCsL6GIo=; h=From:To:Cc:Subject:Date:Message-ID:In-Reply-To:References: MIME-Version; b=UAXVa5WNFnJgD3NoEMuuVgeQpn+cYQTpmhpT3kKaAIolnxglOjnqkf9M1Sh6IvpyQPKFjbchLGC/tthJAYrkYY1L20Qu0TLb7QsqskEZshiGU1J3R4KBJjLU/CTJdY5O4fCmRmZnnv7vnP3R51p4Efr5uTyDaECS53JUQTN4Kdw= ARC-Authentication-Results: i=1; smtp.subspace.kernel.org; dmarc=pass (p=none dis=none) header.from=redhat.com; spf=pass smtp.mailfrom=redhat.com; dkim=pass (1024-bit key) header.d=redhat.com header.i=@redhat.com header.b=WlORUi7T; arc=none smtp.client-ip=170.10.129.124 Authentication-Results: smtp.subspace.kernel.org; dmarc=pass (p=none dis=none) header.from=redhat.com Authentication-Results: smtp.subspace.kernel.org; spf=pass smtp.mailfrom=redhat.com Authentication-Results: smtp.subspace.kernel.org; dkim=pass (1024-bit key) header.d=redhat.com header.i=@redhat.com header.b="WlORUi7T" DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=redhat.com; s=mimecast20190719; t=1724705050; h=from:from:reply-to:subject:subject:date:date:message-id:message-id: to:to:cc:cc:mime-version:mime-version: content-transfer-encoding:content-transfer-encoding: in-reply-to:in-reply-to:references:references; bh=UaLXo2idJup/Oy4OwRPy2aK364ayUQ63MrUET5wBpvQ=; b=WlORUi7T0GKEw0Z+Bsp2hqnlPrQ8BrVlWB8Z/I0dj7H1ijrvKlOSPg/nR8SpEiCphyDV2M nV4LZlZ4N5F0CcYttpgMjUxqONZ/0iDJN4BZmAp38p3GqzhnCHRPTwLhi3thr4i6BBIR8B 5sGjT/GMJWi9ON1SjQdIw7dFPBB6k+I= Received: from mail-qk1-f199.google.com (mail-qk1-f199.google.com [209.85.222.199]) by relay.mimecast.com with ESMTP with STARTTLS (version=TLSv1.3, cipher=TLS_AES_256_GCM_SHA384) id us-mta-541-4r5Ox5mmMumN5XhKyMo6zA-1; Mon, 26 Aug 2024 16:44:09 -0400 X-MC-Unique: 4r5Ox5mmMumN5XhKyMo6zA-1 Received: by mail-qk1-f199.google.com with SMTP id af79cd13be357-7a1df704ef8so599548885a.2 for ; Mon, 26 Aug 2024 13:44:09 -0700 (PDT) X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20230601; t=1724705049; x=1725309849; h=content-transfer-encoding:mime-version:references:in-reply-to :message-id:date:subject:cc:to:from:x-gm-message-state:from:to:cc :subject:date:message-id:reply-to; bh=UaLXo2idJup/Oy4OwRPy2aK364ayUQ63MrUET5wBpvQ=; b=T7dupp4l2PSeQ5MN8oAuih+y8z2B3VGOSit/6cWhZTuqodt/PnTwGm1LwXrhZDmLhC 5U5bMqYjsX8YCxk8wQxAIfkNIIgkKAnCWdMUo8H1sqYEiE6ZO6X8JUgZ2XCi90ypm8Yk viyd9HUkpr2D4uWJa7LRg549pVin0cQHfHiM7ixmqYyzHQS/5tLs7i/LtZFOHHQ6Qdb2 0dbZcVrZWWYcASVIqkNW2hvJ+XaAILmExVsn4C07aBKepAJCWX8ppARMEMzV4WLWxGe4 L7PhjzVzOOspudfP0SwATYfOXQnuMGn5kt18V2eaiBrs+0/yJadCq9dbQ1pbtbUa2GQ+ zCLw== X-Forwarded-Encrypted: i=1; AJvYcCWsaTDuiLWtw7gNXiX7VoAgftiGOZXzcmlmo9Bq/XXc3VoHig+3VM2rfjzem4q6UvuwSJI=@vger.kernel.org X-Gm-Message-State: AOJu0YyGyUGpuGnFAsVNJFkh1WYKdFoUiEui08m1bRElYXqmOyWoLepx lKwlejh2thfjiTiP05POowCYSU+87qE+B3FcC859gQSRfZ55TKruqSMxTlWbxCJTGafUIjsJmqa chS9H/M1TnItZVsAeDd/T5gg0yRtAm0qrcg46OPEJFOb5Uec8mw== X-Received: by 2002:a05:620a:4112:b0:7a3:49dc:e6e3 with SMTP id af79cd13be357-7a6897a6f65mr1244452485a.53.1724705048904; Mon, 26 Aug 2024 13:44:08 -0700 (PDT) X-Google-Smtp-Source: AGHT+IFAEqST5igJBEO0jhxDT4OAwN3xGv5rzMyL4N+pVFoujxFDEyZ/hbdQR2zB1kD+DQ4JLF652A== X-Received: by 2002:a05:620a:4112:b0:7a3:49dc:e6e3 with SMTP id af79cd13be357-7a6897a6f65mr1244449385a.53.1724705048557; Mon, 26 Aug 2024 13:44:08 -0700 (PDT) Received: from x1n.redhat.com (pool-99-254-121-117.cpe.net.cable.rogers.com. [99.254.121.117]) by smtp.gmail.com with ESMTPSA id af79cd13be357-7a67f3fd6c1sm491055185a.121.2024.08.26.13.44.06 (version=TLS1_3 cipher=TLS_AES_256_GCM_SHA384 bits=256/256); Mon, 26 Aug 2024 13:44:07 -0700 (PDT) From: Peter Xu To: linux-kernel@vger.kernel.org, linux-mm@kvack.org Cc: Gavin Shan , Catalin Marinas , x86@kernel.org, Ingo Molnar , Andrew Morton , Paolo Bonzini , Dave Hansen , Thomas Gleixner , Alistair Popple , kvm@vger.kernel.org, linux-arm-kernel@lists.infradead.org, Sean Christopherson , peterx@redhat.com, Oscar Salvador , Jason Gunthorpe , Borislav Petkov , Zi Yan , Axel Rasmussen , David Hildenbrand , Yan Zhao , Will Deacon , Kefeng Wang , Alex Williamson Subject: [PATCH v2 06/19] mm/pagewalk: Check pfnmap for folio_walk_start() Date: Mon, 26 Aug 2024 16:43:40 -0400 Message-ID: <20240826204353.2228736-7-peterx@redhat.com> X-Mailer: git-send-email 2.45.0 In-Reply-To: <20240826204353.2228736-1-peterx@redhat.com> References: <20240826204353.2228736-1-peterx@redhat.com> Precedence: bulk X-Mailing-List: kvm@vger.kernel.org List-Id: List-Subscribe: List-Unsubscribe: MIME-Version: 1.0 Teach folio_walk_start() to recognize special pmd/pud mappings, and fail them properly as it means there's no folio backing them. Cc: David Hildenbrand Signed-off-by: Peter Xu --- mm/pagewalk.c | 4 ++-- 1 file changed, 2 insertions(+), 2 deletions(-) diff --git a/mm/pagewalk.c b/mm/pagewalk.c index cd79fb3b89e5..12be5222d70e 100644 --- a/mm/pagewalk.c +++ b/mm/pagewalk.c @@ -753,7 +753,7 @@ struct folio *folio_walk_start(struct folio_walk *fw, fw->pudp = pudp; fw->pud = pud; - if (!pud_present(pud) || pud_devmap(pud)) { + if (!pud_present(pud) || pud_devmap(pud) || pud_special(pud)) { spin_unlock(ptl); goto not_found; } else if (!pud_leaf(pud)) { @@ -783,7 +783,7 @@ struct folio *folio_walk_start(struct folio_walk *fw, fw->pmdp = pmdp; fw->pmd = pmd; - if (pmd_none(pmd)) { + if (pmd_none(pmd) || pmd_special(pmd)) { spin_unlock(ptl); goto not_found; } else if (!pmd_leaf(pmd)) { From patchwork Mon Aug 26 20:43:41 2024 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Peter Xu X-Patchwork-Id: 13778421 Received: from us-smtp-delivery-124.mimecast.com (us-smtp-delivery-124.mimecast.com [170.10.133.124]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (No client certificate requested) by smtp.subspace.kernel.org (Postfix) with ESMTPS id 4BA3C19DF49 for ; Mon, 26 Aug 2024 20:44:13 +0000 (UTC) Authentication-Results: smtp.subspace.kernel.org; arc=none smtp.client-ip=170.10.133.124 ARC-Seal: i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1724705054; cv=none; b=cYK8CGkJnJ1PYY9ZS2P27thKBtCZnMt+mIDwodTaLPsOPNGVRfjAiAqo7hmXyjxPTmyfg/QPdXpLKH72ZT6YOdhbPL21u4qBE3IMwpm4kfUd8OjaTukCQL9hr70MCS53NkpZWApQPS8LLOwWWHA9QW9dUF/YRoSxNGCJuqxzFEo= ARC-Message-Signature: i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1724705054; c=relaxed/simple; bh=2o5q081kpM9gqW6xM+rZ8YlKEO3D1caJMlb8NRd33tM=; h=From:To:Cc:Subject:Date:Message-ID:In-Reply-To:References: MIME-Version; b=K2POPiZkO6yrdh9+UPW7Bujx6X/cGHu4GryRCShSTJdMcoswoKxAKFNJAD5ESoA7FH/I6sW0Fw7mLw+2pCoXOaZCHPd2mojQnGsK5AaJ27wansnEXh+tqhWybp7i1IV8Vr0eXeP9wNocVTr/9n8KoceApEDvk3m5MLMe6ZuMxn4= ARC-Authentication-Results: i=1; smtp.subspace.kernel.org; dmarc=pass (p=none dis=none) header.from=redhat.com; spf=pass smtp.mailfrom=redhat.com; dkim=pass (1024-bit key) header.d=redhat.com header.i=@redhat.com header.b=Mp/CL9LT; arc=none smtp.client-ip=170.10.133.124 Authentication-Results: smtp.subspace.kernel.org; dmarc=pass (p=none dis=none) header.from=redhat.com Authentication-Results: smtp.subspace.kernel.org; spf=pass smtp.mailfrom=redhat.com Authentication-Results: smtp.subspace.kernel.org; dkim=pass (1024-bit key) header.d=redhat.com header.i=@redhat.com header.b="Mp/CL9LT" DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=redhat.com; s=mimecast20190719; t=1724705052; h=from:from:reply-to:subject:subject:date:date:message-id:message-id: to:to:cc:cc:mime-version:mime-version: content-transfer-encoding:content-transfer-encoding: in-reply-to:in-reply-to:references:references; bh=xat90K+7laktWlKWrEYj2NKDd4TQak32OHmU5MtaKmE=; b=Mp/CL9LTo0Q5866CqBhTpD+usRUcv+G+3B8fcVrnRFs0M7AuVpdqj1MYmrydyyaFdfDKet WEvWYGlLWht27HJeMAR9/OgDGmhIGaAx3YfuE5ffPzmZwi4Cp4Phr96/kU/68de2NTopkI VOQ5HfjQ0Xn2ZbiE267Tm9pyRfj+gic= Received: from mail-yw1-f200.google.com (mail-yw1-f200.google.com [209.85.128.200]) by relay.mimecast.com with ESMTP with STARTTLS (version=TLSv1.3, cipher=TLS_AES_256_GCM_SHA384) id us-mta-528-vtdB0HNFOTSmQWSCHj8mfg-1; Mon, 26 Aug 2024 16:44:11 -0400 X-MC-Unique: vtdB0HNFOTSmQWSCHj8mfg-1 Received: by mail-yw1-f200.google.com with SMTP id 00721157ae682-6ad26d5b061so84748757b3.2 for ; Mon, 26 Aug 2024 13:44:11 -0700 (PDT) X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20230601; t=1724705051; x=1725309851; h=content-transfer-encoding:mime-version:references:in-reply-to :message-id:date:subject:cc:to:from:x-gm-message-state:from:to:cc :subject:date:message-id:reply-to; bh=xat90K+7laktWlKWrEYj2NKDd4TQak32OHmU5MtaKmE=; b=H1m9xa2Q+8q+rwykUzTPMtDRtjK5FTSPaS1wIzcT3ZtZqm2wR4wlah9y9rZr+SZCVV NQP/2VC7/NaO4wOALCFRBPUMVuT1VnZg3p0NXY/Vpl7/B4Mod82FC/mLARlANpBRq+xB nCqXEJfQ86w38UcF71MIDX2YG2lYf3xbsFW43lxKW9nHAGO0UgLtLzXalUcWEerncI5a qATcLywGi79X5BmzGLj9y+ib+tuda6X5ZhRwkMwzKJRm8jEhoHkGF0fYHfc3ZgycFWCA yqlLOqnH7CfW1E8O/DKzTJSIk6rZb8678aPBO483aMZf22Ztk2iAhZagkd9QMIZ4zMnl MfLg== X-Forwarded-Encrypted: i=1; AJvYcCWi4H0PBf8tAoZERBWEEiBqn+sjVwK7es9LeUTPf4Q3tT/2yfRsUyJxm4kxLn0o7UZv2wc=@vger.kernel.org X-Gm-Message-State: AOJu0Yy5aKmJtgdWw+a5+VkwGFzocczQLMPZN8xf3U7R6E7VBm3PB5QL AAePQjnEG1FLBwvrZzMP5EzP4g4KfyrkNbJ2XcnrY2fRXe1/DtJAnxI5pJTysjPbn1CSEQb2Gr1 VgvRoN9MGxDJkLJo1a3s1/JqgBKaldpegnotQT+vhjORZRujZug== X-Received: by 2002:a05:690c:380b:b0:6c1:2b6d:1964 with SMTP id 00721157ae682-6c62906557cmr136660647b3.38.1724705050750; Mon, 26 Aug 2024 13:44:10 -0700 (PDT) X-Google-Smtp-Source: AGHT+IGVQdSmnkARvKpk+I4g97hETTq/PaNGcO/OaJoau0YQSMQm37evWRXkCU2N5l4bob/h1UXyLw== X-Received: by 2002:a05:690c:380b:b0:6c1:2b6d:1964 with SMTP id 00721157ae682-6c62906557cmr136660497b3.38.1724705050419; Mon, 26 Aug 2024 13:44:10 -0700 (PDT) Received: from x1n.redhat.com (pool-99-254-121-117.cpe.net.cable.rogers.com. [99.254.121.117]) by smtp.gmail.com with ESMTPSA id af79cd13be357-7a67f3fd6c1sm491055185a.121.2024.08.26.13.44.08 (version=TLS1_3 cipher=TLS_AES_256_GCM_SHA384 bits=256/256); Mon, 26 Aug 2024 13:44:09 -0700 (PDT) From: Peter Xu To: linux-kernel@vger.kernel.org, linux-mm@kvack.org Cc: Gavin Shan , Catalin Marinas , x86@kernel.org, Ingo Molnar , Andrew Morton , Paolo Bonzini , Dave Hansen , Thomas Gleixner , Alistair Popple , kvm@vger.kernel.org, linux-arm-kernel@lists.infradead.org, Sean Christopherson , peterx@redhat.com, Oscar Salvador , Jason Gunthorpe , Borislav Petkov , Zi Yan , Axel Rasmussen , David Hildenbrand , Yan Zhao , Will Deacon , Kefeng Wang , Alex Williamson Subject: [PATCH v2 07/19] mm/fork: Accept huge pfnmap entries Date: Mon, 26 Aug 2024 16:43:41 -0400 Message-ID: <20240826204353.2228736-8-peterx@redhat.com> X-Mailer: git-send-email 2.45.0 In-Reply-To: <20240826204353.2228736-1-peterx@redhat.com> References: <20240826204353.2228736-1-peterx@redhat.com> Precedence: bulk X-Mailing-List: kvm@vger.kernel.org List-Id: List-Subscribe: List-Unsubscribe: MIME-Version: 1.0 Teach the fork code to properly copy pfnmaps for pmd/pud levels. Pud is much easier, the write bit needs to be persisted though for writable and shared pud mappings like PFNMAP ones, otherwise a follow up write in either parent or child process will trigger a write fault. Do the same for pmd level. Signed-off-by: Peter Xu --- mm/huge_memory.c | 29 ++++++++++++++++++++++++++--- 1 file changed, 26 insertions(+), 3 deletions(-) diff --git a/mm/huge_memory.c b/mm/huge_memory.c index e2c314f631f3..15418ffdd377 100644 --- a/mm/huge_memory.c +++ b/mm/huge_memory.c @@ -1559,6 +1559,24 @@ int copy_huge_pmd(struct mm_struct *dst_mm, struct mm_struct *src_mm, pgtable_t pgtable = NULL; int ret = -ENOMEM; + pmd = pmdp_get_lockless(src_pmd); + if (unlikely(pmd_special(pmd))) { + dst_ptl = pmd_lock(dst_mm, dst_pmd); + src_ptl = pmd_lockptr(src_mm, src_pmd); + spin_lock_nested(src_ptl, SINGLE_DEPTH_NESTING); + /* + * No need to recheck the pmd, it can't change with write + * mmap lock held here. + * + * Meanwhile, making sure it's not a CoW VMA with writable + * mapping, otherwise it means either the anon page wrongly + * applied special bit, or we made the PRIVATE mapping be + * able to wrongly write to the backend MMIO. + */ + VM_WARN_ON_ONCE(is_cow_mapping(src_vma->vm_flags) && pmd_write(pmd)); + goto set_pmd; + } + /* Skip if can be re-fill on fault */ if (!vma_is_anonymous(dst_vma)) return 0; @@ -1640,7 +1658,9 @@ int copy_huge_pmd(struct mm_struct *dst_mm, struct mm_struct *src_mm, pmdp_set_wrprotect(src_mm, addr, src_pmd); if (!userfaultfd_wp(dst_vma)) pmd = pmd_clear_uffd_wp(pmd); - pmd = pmd_mkold(pmd_wrprotect(pmd)); + pmd = pmd_wrprotect(pmd); +set_pmd: + pmd = pmd_mkold(pmd); set_pmd_at(dst_mm, addr, dst_pmd, pmd); ret = 0; @@ -1686,8 +1706,11 @@ int copy_huge_pud(struct mm_struct *dst_mm, struct mm_struct *src_mm, * TODO: once we support anonymous pages, use * folio_try_dup_anon_rmap_*() and split if duplicating fails. */ - pudp_set_wrprotect(src_mm, addr, src_pud); - pud = pud_mkold(pud_wrprotect(pud)); + if (is_cow_mapping(vma->vm_flags) && pud_write(pud)) { + pudp_set_wrprotect(src_mm, addr, src_pud); + pud = pud_wrprotect(pud); + } + pud = pud_mkold(pud); set_pud_at(dst_mm, addr, dst_pud, pud); ret = 0; From patchwork Mon Aug 26 20:43:42 2024 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Peter Xu X-Patchwork-Id: 13778422 Received: from us-smtp-delivery-124.mimecast.com (us-smtp-delivery-124.mimecast.com [170.10.129.124]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (No client certificate requested) by smtp.subspace.kernel.org (Postfix) with ESMTPS id 6BA7019DFB6 for ; Mon, 26 Aug 2024 20:44:16 +0000 (UTC) Authentication-Results: smtp.subspace.kernel.org; arc=none smtp.client-ip=170.10.129.124 ARC-Seal: i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1724705057; cv=none; b=dmmlf0P7Ng/O31zaCSzVPw+aof/t2y3+G2DvhkVb/p26ELKCX99ypUR+BBBZCsGp/kKdkSueYYztidSwoC3owVBJbgvxEPiBdCGLvbqw/9c0juTPXO//LZXxnEMTZtVCDNXI0hkBWc8N9tNiaoLGc4v2gY191XNnqlAAqGC/uHg= ARC-Message-Signature: i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1724705057; c=relaxed/simple; bh=5m+a/sFeWuT8yBU8zTVl6UkOZQKWMf5wpgIpcmHb57A=; h=From:To:Cc:Subject:Date:Message-ID:In-Reply-To:References: MIME-Version; b=bkPLZWwX6YE2PPZ8ng7ClwiEXQrekkixGOfkOdXPyUk7xwbheg45FDvD40hL1mzDQzzoNWQdYSrW/8Q2/g5dvnJn0peKMOPJAhbI06WdtHVoxCAE91gxhTUBvGWpWr6D9nb10NKWvNMMkRFnOFowAoD1t6mfUxQp1zMCsnRWlCQ= ARC-Authentication-Results: i=1; smtp.subspace.kernel.org; dmarc=pass (p=none dis=none) header.from=redhat.com; spf=pass smtp.mailfrom=redhat.com; dkim=pass (1024-bit key) header.d=redhat.com header.i=@redhat.com header.b=cqn1o6Yk; arc=none smtp.client-ip=170.10.129.124 Authentication-Results: smtp.subspace.kernel.org; dmarc=pass (p=none dis=none) header.from=redhat.com Authentication-Results: smtp.subspace.kernel.org; spf=pass smtp.mailfrom=redhat.com Authentication-Results: smtp.subspace.kernel.org; dkim=pass (1024-bit key) header.d=redhat.com header.i=@redhat.com header.b="cqn1o6Yk" DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=redhat.com; s=mimecast20190719; t=1724705055; h=from:from:reply-to:subject:subject:date:date:message-id:message-id: to:to:cc:cc:mime-version:mime-version: content-transfer-encoding:content-transfer-encoding: in-reply-to:in-reply-to:references:references; bh=vK1PMhUcFVpQCvjQRaUqQOYBcILrsdr8yUlCuFWve7Q=; b=cqn1o6YkXtY2Da0jJ6CZFzfGjc/aMwAX/KqsM+od6aT9Llk2e2iSMpXjqrBOzYoQwwI6OZ RcM3dXMW0h38Nfa/pRkhQNGOrmtPudjG3Hij1X7bDT460Sl53PfjCOuyrVRCYyrsCuq3rF MekJBDmgUhW5JSeGQDqtbjMZ4jCjDcs= Received: from mail-yw1-f198.google.com (mail-yw1-f198.google.com [209.85.128.198]) by relay.mimecast.com with ESMTP with STARTTLS (version=TLSv1.3, cipher=TLS_AES_256_GCM_SHA384) id us-mta-275-VQ5c9UaOOeKev-t9Zt5tcw-1; Mon, 26 Aug 2024 16:44:14 -0400 X-MC-Unique: VQ5c9UaOOeKev-t9Zt5tcw-1 Received: by mail-yw1-f198.google.com with SMTP id 00721157ae682-6b351a76beeso102720317b3.0 for ; Mon, 26 Aug 2024 13:44:13 -0700 (PDT) X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20230601; t=1724705052; x=1725309852; h=content-transfer-encoding:mime-version:references:in-reply-to :message-id:date:subject:cc:to:from:x-gm-message-state:from:to:cc :subject:date:message-id:reply-to; bh=vK1PMhUcFVpQCvjQRaUqQOYBcILrsdr8yUlCuFWve7Q=; b=VWJEB+rDpRGrRumsM4XR4i+5BrB1AYOepJRlPZTZPc5GVUuH5NDHzfxlaF8aSuhPzh mBtH23MnZAPGfVJzCVbabMTY+30/kPY/ve53whKwbaYwNKkUGUXoR8Gv8dZ6XczL/+OO cwSsWckbq+r/sIr+qFgq+j+PFg+rRT0GujzbOxIC9ETIyRgJWKiWzf6noYmPtPqpcLW5 uP+6roc2NXbYwOS/OiWi1SweozVMIERnawggJJFhnKhcvmTuyOZF6VKVTcPwbBrTsdPK +vpxP+107bCKpa9ozXEICYyn1PIH7LEsj2WA2bw8SZJyE1Qvi0M+7ckyzioH36VL4hIb O6HQ== X-Forwarded-Encrypted: i=1; AJvYcCVFfp1Mg1ezQkWuiPhKAjPjE6QRjOhE3Wd4cHR4EeBM9iu2Z34ptragfWMVkZFRVYDcXsQ=@vger.kernel.org X-Gm-Message-State: AOJu0YwcltE/fZ95T/zSiMxBv3hgvqJX10hVzKpc8/lOtRKS1A4Ivceu jJqNwl92Fq1XCXgZ672iwK4a21aeOXwv3M4EiSkVlLSwx03172RdgWtP2e1E4EmJD4Q2qlcciZw o0/ucus5YeLe7DYjD+CYOwiwPPKknIEQQclIsOsG6fDFlWe11yQ== X-Received: by 2002:a05:690c:60c2:b0:6bd:fdd3:e170 with SMTP id 00721157ae682-6c624dc7042mr142338917b3.10.1724705052647; Mon, 26 Aug 2024 13:44:12 -0700 (PDT) X-Google-Smtp-Source: AGHT+IEDLUzP2hp4LBST6IRuRTdz5MRV8z70t30cnU5hPcRwmZ021NQU+FxAT+ZUF1I58Qs4qkGJLg== X-Received: by 2002:a05:690c:60c2:b0:6bd:fdd3:e170 with SMTP id 00721157ae682-6c624dc7042mr142338617b3.10.1724705052349; Mon, 26 Aug 2024 13:44:12 -0700 (PDT) Received: from x1n.redhat.com (pool-99-254-121-117.cpe.net.cable.rogers.com. [99.254.121.117]) by smtp.gmail.com with ESMTPSA id af79cd13be357-7a67f3fd6c1sm491055185a.121.2024.08.26.13.44.10 (version=TLS1_3 cipher=TLS_AES_256_GCM_SHA384 bits=256/256); Mon, 26 Aug 2024 13:44:11 -0700 (PDT) From: Peter Xu To: linux-kernel@vger.kernel.org, linux-mm@kvack.org Cc: Gavin Shan , Catalin Marinas , x86@kernel.org, Ingo Molnar , Andrew Morton , Paolo Bonzini , Dave Hansen , Thomas Gleixner , Alistair Popple , kvm@vger.kernel.org, linux-arm-kernel@lists.infradead.org, Sean Christopherson , peterx@redhat.com, Oscar Salvador , Jason Gunthorpe , Borislav Petkov , Zi Yan , Axel Rasmussen , David Hildenbrand , Yan Zhao , Will Deacon , Kefeng Wang , Alex Williamson Subject: [PATCH v2 08/19] mm: Always define pxx_pgprot() Date: Mon, 26 Aug 2024 16:43:42 -0400 Message-ID: <20240826204353.2228736-9-peterx@redhat.com> X-Mailer: git-send-email 2.45.0 In-Reply-To: <20240826204353.2228736-1-peterx@redhat.com> References: <20240826204353.2228736-1-peterx@redhat.com> Precedence: bulk X-Mailing-List: kvm@vger.kernel.org List-Id: List-Subscribe: List-Unsubscribe: MIME-Version: 1.0 There're: - 8 archs (arc, arm64, include, mips, powerpc, s390, sh, x86) that support pte_pgprot(). - 2 archs (x86, sparc) that support pmd_pgprot(). - 1 arch (x86) that support pud_pgprot(). Always define them to be used in generic code, and then we don't need to fiddle with "#ifdef"s when doing so. Reviewed-by: Jason Gunthorpe Signed-off-by: Peter Xu --- arch/arm64/include/asm/pgtable.h | 1 + arch/powerpc/include/asm/pgtable.h | 1 + arch/s390/include/asm/pgtable.h | 1 + arch/sparc/include/asm/pgtable_64.h | 1 + include/linux/pgtable.h | 12 ++++++++++++ 5 files changed, 16 insertions(+) diff --git a/arch/arm64/include/asm/pgtable.h b/arch/arm64/include/asm/pgtable.h index 7a4f5604be3f..b78cc4a6758b 100644 --- a/arch/arm64/include/asm/pgtable.h +++ b/arch/arm64/include/asm/pgtable.h @@ -384,6 +384,7 @@ static inline void __sync_cache_and_tags(pte_t pte, unsigned int nr_pages) /* * Select all bits except the pfn */ +#define pte_pgprot pte_pgprot static inline pgprot_t pte_pgprot(pte_t pte) { unsigned long pfn = pte_pfn(pte); diff --git a/arch/powerpc/include/asm/pgtable.h b/arch/powerpc/include/asm/pgtable.h index 264a6c09517a..2f72ad885332 100644 --- a/arch/powerpc/include/asm/pgtable.h +++ b/arch/powerpc/include/asm/pgtable.h @@ -65,6 +65,7 @@ static inline unsigned long pte_pfn(pte_t pte) /* * Select all bits except the pfn */ +#define pte_pgprot pte_pgprot static inline pgprot_t pte_pgprot(pte_t pte) { unsigned long pte_flags; diff --git a/arch/s390/include/asm/pgtable.h b/arch/s390/include/asm/pgtable.h index 3fa280d0672a..0ffbaf741955 100644 --- a/arch/s390/include/asm/pgtable.h +++ b/arch/s390/include/asm/pgtable.h @@ -955,6 +955,7 @@ static inline int pte_unused(pte_t pte) * young/old accounting is not supported, i.e _PAGE_PROTECT and _PAGE_INVALID * must not be set. */ +#define pte_pgprot pte_pgprot static inline pgprot_t pte_pgprot(pte_t pte) { unsigned long pte_flags = pte_val(pte) & _PAGE_CHG_MASK; diff --git a/arch/sparc/include/asm/pgtable_64.h b/arch/sparc/include/asm/pgtable_64.h index 3fe429d73a65..2b7f358762c1 100644 --- a/arch/sparc/include/asm/pgtable_64.h +++ b/arch/sparc/include/asm/pgtable_64.h @@ -783,6 +783,7 @@ static inline pmd_t pmd_mkwrite_novma(pmd_t pmd) return __pmd(pte_val(pte)); } +#define pmd_pgprot pmd_pgprot static inline pgprot_t pmd_pgprot(pmd_t entry) { unsigned long val = pmd_val(entry); diff --git a/include/linux/pgtable.h b/include/linux/pgtable.h index 780f3b439d98..e8b2ac6bd2ae 100644 --- a/include/linux/pgtable.h +++ b/include/linux/pgtable.h @@ -1956,6 +1956,18 @@ typedef unsigned int pgtbl_mod_mask; #define MAX_PTRS_PER_P4D PTRS_PER_P4D #endif +#ifndef pte_pgprot +#define pte_pgprot(x) ((pgprot_t) {0}) +#endif + +#ifndef pmd_pgprot +#define pmd_pgprot(x) ((pgprot_t) {0}) +#endif + +#ifndef pud_pgprot +#define pud_pgprot(x) ((pgprot_t) {0}) +#endif + /* description of effects of mapping type and prot in current implementation. * this is due to the limited x86 page protection hardware. The expected * behavior is in parens: From patchwork Mon Aug 26 20:43:43 2024 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Peter Xu X-Patchwork-Id: 13778423 Received: from us-smtp-delivery-124.mimecast.com (us-smtp-delivery-124.mimecast.com [170.10.133.124]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (No client certificate requested) by smtp.subspace.kernel.org (Postfix) with ESMTPS id 1F66B19E7E5 for ; Mon, 26 Aug 2024 20:44:17 +0000 (UTC) Authentication-Results: smtp.subspace.kernel.org; arc=none smtp.client-ip=170.10.133.124 ARC-Seal: i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1724705059; cv=none; b=cCmjMEC4SVyp6KOQrzNJXLUjaKNoMlJ++s7qB8ASyD+PHUldyjPoiRWaCiwJrIykB1YplX6+jdefYtpOlGJ2IZpTj83sBKVsuHBKD7Lizezt8L0/BsnP6U+AHf9UOyZ0eymkWVQqCKoIdAaUfhnOLuke/FQRXRDD+VeQAqDNigY= ARC-Message-Signature: i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1724705059; c=relaxed/simple; bh=OG1HOirCXbGLRvA8SpEinM1gGzP9lRsOJkqbcyKglh8=; h=From:To:Cc:Subject:Date:Message-ID:In-Reply-To:References: MIME-Version; b=ORDZ+8Swn9V+y3K9Ll5Qfv8/JpbgqKAggp6A8pdDrRuN2VjJ83aHLrH8borVP4GRnVM8xGm7R2xGuF2B+K4v5+Qvy/u9fWOSDqM0tuzygDewhhoMCrAsCkDO9V7ksAFxVqLUrGQW9IZTmZwY+FHVclhOA3NJuxLtkXEi/QD6krw= ARC-Authentication-Results: i=1; smtp.subspace.kernel.org; dmarc=pass (p=none dis=none) header.from=redhat.com; spf=pass smtp.mailfrom=redhat.com; dkim=pass (1024-bit key) header.d=redhat.com header.i=@redhat.com header.b=Y7DgqhFi; arc=none smtp.client-ip=170.10.133.124 Authentication-Results: smtp.subspace.kernel.org; dmarc=pass (p=none dis=none) header.from=redhat.com Authentication-Results: smtp.subspace.kernel.org; spf=pass smtp.mailfrom=redhat.com Authentication-Results: smtp.subspace.kernel.org; dkim=pass (1024-bit key) header.d=redhat.com header.i=@redhat.com header.b="Y7DgqhFi" DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=redhat.com; s=mimecast20190719; t=1724705057; h=from:from:reply-to:subject:subject:date:date:message-id:message-id: to:to:cc:cc:mime-version:mime-version: content-transfer-encoding:content-transfer-encoding: in-reply-to:in-reply-to:references:references; bh=XzWn17lHZnGD5Tr66iu6y8/zqhTRy72agTNrtdvxzwE=; b=Y7DgqhFiU9py8NcaEACQZfKOvi9JTLUlm3mwv9T3grgx/EUGBfkjCso6/IKeYYRt1HtKXL XiVkbLQvNckgR3M9urfRaDThtsuXbTmImGekdIXQxtsDR6+9/byPkPiubIq5wSKrBM6Wgv K7xmMZQl073267MLHpqeFDd22o5i3n0= Received: from mail-qk1-f200.google.com (mail-qk1-f200.google.com [209.85.222.200]) by relay.mimecast.com with ESMTP with STARTTLS (version=TLSv1.3, cipher=TLS_AES_256_GCM_SHA384) id us-mta-398-_fChLC89OyS18BA2mVsauA-1; Mon, 26 Aug 2024 16:44:15 -0400 X-MC-Unique: _fChLC89OyS18BA2mVsauA-1 Received: by mail-qk1-f200.google.com with SMTP id af79cd13be357-7a1dc1e5662so663406385a.2 for ; Mon, 26 Aug 2024 13:44:15 -0700 (PDT) X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20230601; t=1724705055; x=1725309855; h=content-transfer-encoding:mime-version:references:in-reply-to :message-id:date:subject:cc:to:from:x-gm-message-state:from:to:cc :subject:date:message-id:reply-to; bh=XzWn17lHZnGD5Tr66iu6y8/zqhTRy72agTNrtdvxzwE=; b=DsSsCILTllUdyhuvwdJF344x/2fw/c4OJc58ye1okDIOlGP1jAdLi5q4CuCG2dXfF+ RGpfbEQ6TRvfSBa4D18b3pClTeYLBGT1puF+1LDjsMa7DvIMrWXTFBM12AgbTgtTndnD OJJujP0wOHdScUTKjlYbZr84OAhmy6P9gigBOOfR70rqkJUrieDNVTLjSBmK2AiQ74+8 iS6TGSQKETVY821+fqtf0/QxMBobNkGJENtWeiLcHzqEfOKigo/nE6wRuFQutAvWjwVX mSF40pO/GSVTsuwXUkmfWIojNmszXQrg+cSSVS8xPvLIVaOOjG0N8SgqCLS/WoV+/eYb J22g== X-Forwarded-Encrypted: i=1; AJvYcCUqCUOl2HBP1jDO+SDTGkhF7UPCf7OQUMMNhcgv+sfdGGeYb4RnrddI8E8ghjKosnLIjKw=@vger.kernel.org X-Gm-Message-State: AOJu0Yzm6zTEGz19Wmu9kVm/WOvDIWPtRNR3CTGEIKKde8vEDky3rTOI afhQRB3pskey46o/Vjbr1QTNqHQ981SttDFiJx48Md7zJB6jin/QF6xPGjuprkmV59lnV7PlGLP /jq2NKXeC6iDXVXGCBuS2+NbM5DjgpPDLt7iwYjuMXoeLgSFv5Q== X-Received: by 2002:a05:620a:408d:b0:7a1:e93c:cd04 with SMTP id af79cd13be357-7a6896e0e1cmr1465304585a.9.1724705054677; Mon, 26 Aug 2024 13:44:14 -0700 (PDT) X-Google-Smtp-Source: AGHT+IH4kwi4qTohFBFu7kWFKQGbZI396sZSJGmlCvwBl5XfpN0sU8TboRNIhnEyWXLKkBzi4MYP4w== X-Received: by 2002:a05:620a:408d:b0:7a1:e93c:cd04 with SMTP id af79cd13be357-7a6896e0e1cmr1465302885a.9.1724705054299; Mon, 26 Aug 2024 13:44:14 -0700 (PDT) Received: from x1n.redhat.com (pool-99-254-121-117.cpe.net.cable.rogers.com. [99.254.121.117]) by smtp.gmail.com with ESMTPSA id af79cd13be357-7a67f3fd6c1sm491055185a.121.2024.08.26.13.44.12 (version=TLS1_3 cipher=TLS_AES_256_GCM_SHA384 bits=256/256); Mon, 26 Aug 2024 13:44:13 -0700 (PDT) From: Peter Xu To: linux-kernel@vger.kernel.org, linux-mm@kvack.org Cc: Gavin Shan , Catalin Marinas , x86@kernel.org, Ingo Molnar , Andrew Morton , Paolo Bonzini , Dave Hansen , Thomas Gleixner , Alistair Popple , kvm@vger.kernel.org, linux-arm-kernel@lists.infradead.org, Sean Christopherson , peterx@redhat.com, Oscar Salvador , Jason Gunthorpe , Borislav Petkov , Zi Yan , Axel Rasmussen , David Hildenbrand , Yan Zhao , Will Deacon , Kefeng Wang , Alex Williamson Subject: [PATCH v2 09/19] mm: New follow_pfnmap API Date: Mon, 26 Aug 2024 16:43:43 -0400 Message-ID: <20240826204353.2228736-10-peterx@redhat.com> X-Mailer: git-send-email 2.45.0 In-Reply-To: <20240826204353.2228736-1-peterx@redhat.com> References: <20240826204353.2228736-1-peterx@redhat.com> Precedence: bulk X-Mailing-List: kvm@vger.kernel.org List-Id: List-Subscribe: List-Unsubscribe: MIME-Version: 1.0 Introduce a pair of APIs to follow pfn mappings to get entry information. It's very similar to what follow_pte() does before, but different in that it recognizes huge pfn mappings. Signed-off-by: Peter Xu --- include/linux/mm.h | 31 ++++++++++ mm/memory.c | 150 +++++++++++++++++++++++++++++++++++++++++++++ 2 files changed, 181 insertions(+) diff --git a/include/linux/mm.h b/include/linux/mm.h index d900f15b7650..161d496bfd18 100644 --- a/include/linux/mm.h +++ b/include/linux/mm.h @@ -2373,6 +2373,37 @@ int follow_pte(struct vm_area_struct *vma, unsigned long address, int generic_access_phys(struct vm_area_struct *vma, unsigned long addr, void *buf, int len, int write); +struct follow_pfnmap_args { + /** + * Inputs: + * @vma: Pointer to @vm_area_struct struct + * @address: the virtual address to walk + */ + struct vm_area_struct *vma; + unsigned long address; + /** + * Internals: + * + * The caller shouldn't touch any of these. + */ + spinlock_t *lock; + pte_t *ptep; + /** + * Outputs: + * + * @pfn: the PFN of the address + * @pgprot: the pgprot_t of the mapping + * @writable: whether the mapping is writable + * @special: whether the mapping is a special mapping (real PFN maps) + */ + unsigned long pfn; + pgprot_t pgprot; + bool writable; + bool special; +}; +int follow_pfnmap_start(struct follow_pfnmap_args *args); +void follow_pfnmap_end(struct follow_pfnmap_args *args); + extern void truncate_pagecache(struct inode *inode, loff_t new); extern void truncate_setsize(struct inode *inode, loff_t newsize); void pagecache_isize_extended(struct inode *inode, loff_t from, loff_t to); diff --git a/mm/memory.c b/mm/memory.c index 93c0c25433d0..0b136c398257 100644 --- a/mm/memory.c +++ b/mm/memory.c @@ -6173,6 +6173,156 @@ int follow_pte(struct vm_area_struct *vma, unsigned long address, } EXPORT_SYMBOL_GPL(follow_pte); +static inline void pfnmap_args_setup(struct follow_pfnmap_args *args, + spinlock_t *lock, pte_t *ptep, + pgprot_t pgprot, unsigned long pfn_base, + unsigned long addr_mask, bool writable, + bool special) +{ + args->lock = lock; + args->ptep = ptep; + args->pfn = pfn_base + ((args->address & ~addr_mask) >> PAGE_SHIFT); + args->pgprot = pgprot; + args->writable = writable; + args->special = special; +} + +static inline void pfnmap_lockdep_assert(struct vm_area_struct *vma) +{ +#ifdef CONFIG_LOCKDEP + struct address_space *mapping = vma->vm_file->f_mapping; + + if (mapping) + lockdep_assert(lockdep_is_held(&vma->vm_file->f_mapping->i_mmap_rwsem) || + lockdep_is_held(&vma->vm_mm->mmap_lock)); + else + lockdep_assert(lockdep_is_held(&vma->vm_mm->mmap_lock)); +#endif +} + +/** + * follow_pfnmap_start() - Look up a pfn mapping at a user virtual address + * @args: Pointer to struct @follow_pfnmap_args + * + * The caller needs to setup args->vma and args->address to point to the + * virtual address as the target of such lookup. On a successful return, + * the results will be put into other output fields. + * + * After the caller finished using the fields, the caller must invoke + * another follow_pfnmap_end() to proper releases the locks and resources + * of such look up request. + * + * During the start() and end() calls, the results in @args will be valid + * as proper locks will be held. After the end() is called, all the fields + * in @follow_pfnmap_args will be invalid to be further accessed. Further + * use of such information after end() may require proper synchronizations + * by the caller with page table updates, otherwise it can create a + * security bug. + * + * If the PTE maps a refcounted page, callers are responsible to protect + * against invalidation with MMU notifiers; otherwise access to the PFN at + * a later point in time can trigger use-after-free. + * + * Only IO mappings and raw PFN mappings are allowed. The mmap semaphore + * should be taken for read, and the mmap semaphore cannot be released + * before the end() is invoked. + * + * This function must not be used to modify PTE content. + * + * Return: zero on success, negative otherwise. + */ +int follow_pfnmap_start(struct follow_pfnmap_args *args) +{ + struct vm_area_struct *vma = args->vma; + unsigned long address = args->address; + struct mm_struct *mm = vma->vm_mm; + spinlock_t *lock; + pgd_t *pgdp; + p4d_t *p4dp, p4d; + pud_t *pudp, pud; + pmd_t *pmdp, pmd; + pte_t *ptep, pte; + + pfnmap_lockdep_assert(vma); + + if (unlikely(address < vma->vm_start || address >= vma->vm_end)) + goto out; + + if (!(vma->vm_flags & (VM_IO | VM_PFNMAP))) + goto out; +retry: + pgdp = pgd_offset(mm, address); + if (pgd_none(*pgdp) || unlikely(pgd_bad(*pgdp))) + goto out; + + p4dp = p4d_offset(pgdp, address); + p4d = READ_ONCE(*p4dp); + if (p4d_none(p4d) || unlikely(p4d_bad(p4d))) + goto out; + + pudp = pud_offset(p4dp, address); + pud = READ_ONCE(*pudp); + if (pud_none(pud)) + goto out; + if (pud_leaf(pud)) { + lock = pud_lock(mm, pudp); + if (!unlikely(pud_leaf(pud))) { + spin_unlock(lock); + goto retry; + } + pfnmap_args_setup(args, lock, NULL, pud_pgprot(pud), + pud_pfn(pud), PUD_MASK, pud_write(pud), + pud_special(pud)); + return 0; + } + + pmdp = pmd_offset(pudp, address); + pmd = pmdp_get_lockless(pmdp); + if (pmd_leaf(pmd)) { + lock = pmd_lock(mm, pmdp); + if (!unlikely(pmd_leaf(pmd))) { + spin_unlock(lock); + goto retry; + } + pfnmap_args_setup(args, lock, NULL, pmd_pgprot(pmd), + pmd_pfn(pmd), PMD_MASK, pmd_write(pmd), + pmd_special(pmd)); + return 0; + } + + ptep = pte_offset_map_lock(mm, pmdp, address, &lock); + if (!ptep) + goto out; + pte = ptep_get(ptep); + if (!pte_present(pte)) + goto unlock; + pfnmap_args_setup(args, lock, ptep, pte_pgprot(pte), + pte_pfn(pte), PAGE_MASK, pte_write(pte), + pte_special(pte)); + return 0; +unlock: + pte_unmap_unlock(ptep, lock); +out: + return -EINVAL; +} +EXPORT_SYMBOL_GPL(follow_pfnmap_start); + +/** + * follow_pfnmap_end(): End a follow_pfnmap_start() process + * @args: Pointer to struct @follow_pfnmap_args + * + * Must be used in pair of follow_pfnmap_start(). See the start() function + * above for more information. + */ +void follow_pfnmap_end(struct follow_pfnmap_args *args) +{ + if (args->lock) + spin_unlock(args->lock); + if (args->ptep) + pte_unmap(args->ptep); +} +EXPORT_SYMBOL_GPL(follow_pfnmap_end); + #ifdef CONFIG_HAVE_IOREMAP_PROT /** * generic_access_phys - generic implementation for iomem mmap access From patchwork Mon Aug 26 20:43:44 2024 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Peter Xu X-Patchwork-Id: 13778424 Received: from us-smtp-delivery-124.mimecast.com (us-smtp-delivery-124.mimecast.com [170.10.129.124]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (No client certificate requested) by smtp.subspace.kernel.org (Postfix) with ESMTPS id 249D019E804 for ; Mon, 26 Aug 2024 20:44:19 +0000 (UTC) Authentication-Results: smtp.subspace.kernel.org; arc=none smtp.client-ip=170.10.129.124 ARC-Seal: i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1724705060; cv=none; b=SDeY4Ro5stDheAPj8WZosJdnkbNnNb8KhrDI8bR9aoOrG6Hb9dTmHFr2zt+Spoyb1IzZHwW6HvstQDDIqt5c4GEdcP8jIoqBPxaTW8DSUuwyYWyE5ijGyrADdJpVfSVR+AojMsc4QDwbUduvWGg3tKQAsHU9NX2tBep6Nm6S5c8= ARC-Message-Signature: i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1724705060; c=relaxed/simple; bh=Aa8NqcDK98lGFy3crPXQ+TfEjjdi6Ds7uzzU5PCZnys=; h=From:To:Cc:Subject:Date:Message-ID:In-Reply-To:References: MIME-Version; b=LfRYdCe0z1NGRow4ERoD3UfJob3IYkRZoecEnetCl7GFBshAbN/OwC8TzFQuPM/3PKR3G7FwbHum6FK2JcG9xDuM+Bzos7RPWXhth97QB4UNl8gP98QxzHwsOn14P49Ql59iJ1OvwrM6VrJyLeqpWBn+SRWksT2VymKwsshs//8= ARC-Authentication-Results: i=1; smtp.subspace.kernel.org; dmarc=pass (p=none dis=none) header.from=redhat.com; spf=pass smtp.mailfrom=redhat.com; dkim=pass (1024-bit key) header.d=redhat.com header.i=@redhat.com header.b=MTLyfwYW; arc=none smtp.client-ip=170.10.129.124 Authentication-Results: smtp.subspace.kernel.org; dmarc=pass (p=none dis=none) header.from=redhat.com Authentication-Results: smtp.subspace.kernel.org; spf=pass smtp.mailfrom=redhat.com Authentication-Results: smtp.subspace.kernel.org; dkim=pass (1024-bit key) header.d=redhat.com header.i=@redhat.com header.b="MTLyfwYW" DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=redhat.com; s=mimecast20190719; t=1724705058; h=from:from:reply-to:subject:subject:date:date:message-id:message-id: to:to:cc:cc:mime-version:mime-version: content-transfer-encoding:content-transfer-encoding: in-reply-to:in-reply-to:references:references; bh=RcTv8PG6WUMces5zPq/dQ8qXZuImd+6bX+nNmvbIpbY=; b=MTLyfwYWhsMNf4pb8kDAIarW8+NJrX1jRYCOcCCbMnphHbPayWDQNhkXv+WbsSFVFDxAaU faWdGVgSt1FLMgroQvR8PefnTKq3LP7LuAT7w6shXP2qaNNOoqLv+l2IrZ2lefKUKDjAYt k5MJntAJ+WsiM6ZzFDmViVA3hIP7cWY= Received: from mail-yw1-f199.google.com (mail-yw1-f199.google.com [209.85.128.199]) by relay.mimecast.com with ESMTP with STARTTLS (version=TLSv1.3, cipher=TLS_AES_256_GCM_SHA384) id us-mta-643-6ZHkzoHXMkycCmARbOEhfA-1; Mon, 26 Aug 2024 16:44:17 -0400 X-MC-Unique: 6ZHkzoHXMkycCmARbOEhfA-1 Received: by mail-yw1-f199.google.com with SMTP id 00721157ae682-6b41e02c293so94041137b3.0 for ; Mon, 26 Aug 2024 13:44:17 -0700 (PDT) X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20230601; t=1724705056; x=1725309856; h=content-transfer-encoding:mime-version:references:in-reply-to :message-id:date:subject:cc:to:from:x-gm-message-state:from:to:cc :subject:date:message-id:reply-to; bh=RcTv8PG6WUMces5zPq/dQ8qXZuImd+6bX+nNmvbIpbY=; b=DOXzrREKS3YOt/v2mkO1gKntLXDFyTRacZYZ0SVt90JwVlDo6/65gOnSnIZrqLbIZV SEwUFB4ZKkfvQrv9YGO6tNaTj9vI0/0ibVJ6BMBytYh2V8dKBYEliQgb8iqGtI3le/JR bTPNSTOff7t25kLveHkHGQbFomAxaadLGLGtnhCUDoFD3XcqQweJABLciZ47+JlEJjjE LTEHMZftz/G5PSR18yl3mHKhsZXuY35f3aKO0BDaV9hkvOjdcA3s5IXx4wrPK5wY79zn gn2jtpTB5cS7DIKz21xjQzaO5o8B1dWihVXYgdQeXOwCCtfcdhicqPhIr3HNob8MDHhe UYag== X-Forwarded-Encrypted: i=1; AJvYcCVWsB7BJJpyqKCr8eaKFk7zI29ahEueMBF8rOQhjH4D6n0EwjMlNvL0otmRlbVgXeiAwWg=@vger.kernel.org X-Gm-Message-State: AOJu0YybSjAZyd1EkLmXoT4UP6fHpqlFjL7WcjG3qfi7gzMJ8oFn/926 0vc4M4kEXolbHH/Lvseqx12YfYjetsLtO42I1T7C50qrpXg3P7NFId1OJauh6hbRRGl1HvYlJNW gBt4go6yFK1IQZm/rnDLwVTiMsB+jUTdWhl5hn12WtRMdJ8TD/A== X-Received: by 2002:a05:690c:4246:b0:6b2:4154:e598 with SMTP id 00721157ae682-6c6262f43e4mr113946487b3.26.1724705056539; Mon, 26 Aug 2024 13:44:16 -0700 (PDT) X-Google-Smtp-Source: AGHT+IGCZvjuJnmUbtCZKJF68AHOtk4V9xJHAv9nFBXLLIOg3bv7tZNlMwBWnRJYIMuW8OX1o2s1rQ== X-Received: by 2002:a05:690c:4246:b0:6b2:4154:e598 with SMTP id 00721157ae682-6c6262f43e4mr113946177b3.26.1724705056224; Mon, 26 Aug 2024 13:44:16 -0700 (PDT) Received: from x1n.redhat.com (pool-99-254-121-117.cpe.net.cable.rogers.com. [99.254.121.117]) by smtp.gmail.com with ESMTPSA id af79cd13be357-7a67f3fd6c1sm491055185a.121.2024.08.26.13.44.14 (version=TLS1_3 cipher=TLS_AES_256_GCM_SHA384 bits=256/256); Mon, 26 Aug 2024 13:44:15 -0700 (PDT) From: Peter Xu To: linux-kernel@vger.kernel.org, linux-mm@kvack.org Cc: Gavin Shan , Catalin Marinas , x86@kernel.org, Ingo Molnar , Andrew Morton , Paolo Bonzini , Dave Hansen , Thomas Gleixner , Alistair Popple , kvm@vger.kernel.org, linux-arm-kernel@lists.infradead.org, Sean Christopherson , peterx@redhat.com, Oscar Salvador , Jason Gunthorpe , Borislav Petkov , Zi Yan , Axel Rasmussen , David Hildenbrand , Yan Zhao , Will Deacon , Kefeng Wang , Alex Williamson Subject: [PATCH v2 10/19] KVM: Use follow_pfnmap API Date: Mon, 26 Aug 2024 16:43:44 -0400 Message-ID: <20240826204353.2228736-11-peterx@redhat.com> X-Mailer: git-send-email 2.45.0 In-Reply-To: <20240826204353.2228736-1-peterx@redhat.com> References: <20240826204353.2228736-1-peterx@redhat.com> Precedence: bulk X-Mailing-List: kvm@vger.kernel.org List-Id: List-Subscribe: List-Unsubscribe: MIME-Version: 1.0 Use the new pfnmap API to allow huge MMIO mappings for VMs. The rest work is done perfectly on the other side (host_pfn_mapping_level()). Cc: Paolo Bonzini Cc: Sean Christopherson Signed-off-by: Peter Xu --- virt/kvm/kvm_main.c | 19 +++++++------------ 1 file changed, 7 insertions(+), 12 deletions(-) diff --git a/virt/kvm/kvm_main.c b/virt/kvm/kvm_main.c index cb2b78e92910..f416d5e3f9c0 100644 --- a/virt/kvm/kvm_main.c +++ b/virt/kvm/kvm_main.c @@ -2860,13 +2860,11 @@ static int hva_to_pfn_remapped(struct vm_area_struct *vma, unsigned long addr, bool write_fault, bool *writable, kvm_pfn_t *p_pfn) { + struct follow_pfnmap_args args = { .vma = vma, .address = addr }; kvm_pfn_t pfn; - pte_t *ptep; - pte_t pte; - spinlock_t *ptl; int r; - r = follow_pte(vma, addr, &ptep, &ptl); + r = follow_pfnmap_start(&args); if (r) { /* * get_user_pages fails for VM_IO and VM_PFNMAP vmas and does @@ -2881,21 +2879,19 @@ static int hva_to_pfn_remapped(struct vm_area_struct *vma, if (r) return r; - r = follow_pte(vma, addr, &ptep, &ptl); + r = follow_pfnmap_start(&args); if (r) return r; } - pte = ptep_get(ptep); - - if (write_fault && !pte_write(pte)) { + if (write_fault && !args.writable) { pfn = KVM_PFN_ERR_RO_FAULT; goto out; } if (writable) - *writable = pte_write(pte); - pfn = pte_pfn(pte); + *writable = args.writable; + pfn = args.pfn; /* * Get a reference here because callers of *hva_to_pfn* and @@ -2916,9 +2912,8 @@ static int hva_to_pfn_remapped(struct vm_area_struct *vma, */ if (!kvm_try_get_pfn(pfn)) r = -EFAULT; - out: - pte_unmap_unlock(ptep, ptl); + follow_pfnmap_end(&args); *p_pfn = pfn; return r; From patchwork Mon Aug 26 20:43:45 2024 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Peter Xu X-Patchwork-Id: 13778425 Received: from us-smtp-delivery-124.mimecast.com (us-smtp-delivery-124.mimecast.com [170.10.133.124]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (No client certificate requested) by smtp.subspace.kernel.org (Postfix) with ESMTPS id 4A75E19AD70 for ; Mon, 26 Aug 2024 20:44:22 +0000 (UTC) Authentication-Results: smtp.subspace.kernel.org; arc=none smtp.client-ip=170.10.133.124 ARC-Seal: i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1724705063; cv=none; b=R1dcVDTKh8oBiB5KALP2ctKf8PRk9Zw7Ao9ECuC3AwCPyykZlPNSsdg6KhGVBlMX/5H+yvC6V9T5wSjHSfLf5KIV/lR7mma3or9mrZNhLoig8Uv4t7+l+2cWjJOHCG/aLgBHwGEH7HFOAQGwAuAjY80Ke4PD33Th2zOBiGeQb2w= ARC-Message-Signature: i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1724705063; c=relaxed/simple; bh=F3zrw7GW4L3DDhmifn0KqTitiZnPFDtSbk8TCndoYpY=; h=From:To:Cc:Subject:Date:Message-ID:In-Reply-To:References: MIME-Version; b=TKKWG2XhbTqF1y1NozHH3RSa5K2VksDuDSoruIUG+qCCaR4tRubwOC1R4wFfnFtlXJxz4vuEnlr+XtjzppcwJGcuvkVwFJYlWyayaG+5kScQtKZSh/DKend8fRdCykGWRWDMZwOWL2xtaIHTgP5gv+QQZIgAShNHvMfrI/n1Y/s= ARC-Authentication-Results: i=1; smtp.subspace.kernel.org; dmarc=pass (p=none dis=none) header.from=redhat.com; spf=pass smtp.mailfrom=redhat.com; dkim=pass (1024-bit key) header.d=redhat.com header.i=@redhat.com header.b=CEepY3px; arc=none smtp.client-ip=170.10.133.124 Authentication-Results: smtp.subspace.kernel.org; dmarc=pass (p=none dis=none) header.from=redhat.com Authentication-Results: smtp.subspace.kernel.org; spf=pass smtp.mailfrom=redhat.com Authentication-Results: smtp.subspace.kernel.org; dkim=pass (1024-bit key) header.d=redhat.com header.i=@redhat.com header.b="CEepY3px" DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=redhat.com; s=mimecast20190719; t=1724705061; h=from:from:reply-to:subject:subject:date:date:message-id:message-id: to:to:cc:cc:mime-version:mime-version: content-transfer-encoding:content-transfer-encoding: in-reply-to:in-reply-to:references:references; bh=kaY3EfeA+k09YWgS+MfGM8qkjlv58srLyvVau/9jXnU=; b=CEepY3pxevilkMXy6m+MxhWH0JyC8nmTA/XhdMqGH/2HlbICCUCsx3wz7L8oYUtDoN7ZmX WSmiJbH3wx9P2OwWeBA9Jehz4kKiyaTWg19O2JaiUhERxxq06l8szczNhJip6oCwmXdrYp 2SyxlT9JDzvecjG7BJa/cX6ZkUChiA4= Received: from mail-oa1-f71.google.com (mail-oa1-f71.google.com [209.85.160.71]) by relay.mimecast.com with ESMTP with STARTTLS (version=TLSv1.3, cipher=TLS_AES_256_GCM_SHA384) id us-mta-522-LmGor_ZEO-WoBfIOuuKR6A-1; Mon, 26 Aug 2024 16:44:19 -0400 X-MC-Unique: LmGor_ZEO-WoBfIOuuKR6A-1 Received: by mail-oa1-f71.google.com with SMTP id 586e51a60fabf-270475308abso7796439fac.0 for ; Mon, 26 Aug 2024 13:44:19 -0700 (PDT) X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20230601; t=1724705059; x=1725309859; h=content-transfer-encoding:mime-version:references:in-reply-to :message-id:date:subject:cc:to:from:x-gm-message-state:from:to:cc :subject:date:message-id:reply-to; bh=kaY3EfeA+k09YWgS+MfGM8qkjlv58srLyvVau/9jXnU=; b=LNOXd2g2Vj2EdWntGDx9shZ/2NKAMJErLgf6TUthOiL70QflD4XraK4NIwulyZPKX8 0bxDzfbmWGSl6Q2QdhHt1hHQSyTDyLuUwilFKXXBKIAVSxt8ouTFOyRzquioqgP/5YBc jSJ0gIWrC/PmqKrtna6GjZqKXUDQ9jCgoB+HNoy4rtAjqzBIs1olhPu5fQdal6V8YQl2 OkpGweQpV6aDaOIx+sqmfYPiDfYPw8RkT4KoHhSMxM2yOrs39EIdJ7dFPSPelSRgzu4Q 61CLzpQFU4hatm9ZA/nVmyREAQKdEM5ksX5RjcbRzC2Df5wrg/gzvEsRIzAzFZB1K3WG F46A== X-Forwarded-Encrypted: i=1; AJvYcCXijHdFlLdCn7w4Hozh5m7WmA4Te2MUBeDfGa7d3l/5974HZSlkl7FH0/SME01YUwgJCoM=@vger.kernel.org X-Gm-Message-State: AOJu0YzKsbdnmq2/c8GIzjHHuNbCiYJdNVRLh6epXNydtTwXcSt8gfju TmjoN+0wBIfc5p5fXDJelgDwL+mXXcwfWICe4iGi+s6d3XivTDkJr7n8B0bR6ldofuCHGtBo5jQ MB9W0IT3sGwefddeT1p2Sv8zgJCUN7gqG78p8uygdS3ZG8LFG2w== X-Received: by 2002:a05:6358:7201:b0:1ac:f3df:3bde with SMTP id e5c5f4694b2df-1b5ebf3abdcmr117738655d.5.1724705058975; Mon, 26 Aug 2024 13:44:18 -0700 (PDT) X-Google-Smtp-Source: AGHT+IHztBftH8jfc2wrFWJQjc0BC5uJ0IwQWlV9Rpv0YGeitSfcFoHRh3KGrU0k5GaslF87BwNh3Q== X-Received: by 2002:a05:6358:7201:b0:1ac:f3df:3bde with SMTP id e5c5f4694b2df-1b5ebf3abdcmr117734755d.5.1724705058531; Mon, 26 Aug 2024 13:44:18 -0700 (PDT) Received: from x1n.redhat.com (pool-99-254-121-117.cpe.net.cable.rogers.com. [99.254.121.117]) by smtp.gmail.com with ESMTPSA id af79cd13be357-7a67f3fd6c1sm491055185a.121.2024.08.26.13.44.16 (version=TLS1_3 cipher=TLS_AES_256_GCM_SHA384 bits=256/256); Mon, 26 Aug 2024 13:44:18 -0700 (PDT) From: Peter Xu To: linux-kernel@vger.kernel.org, linux-mm@kvack.org Cc: Gavin Shan , Catalin Marinas , x86@kernel.org, Ingo Molnar , Andrew Morton , Paolo Bonzini , Dave Hansen , Thomas Gleixner , Alistair Popple , kvm@vger.kernel.org, linux-arm-kernel@lists.infradead.org, Sean Christopherson , peterx@redhat.com, Oscar Salvador , Jason Gunthorpe , Borislav Petkov , Zi Yan , Axel Rasmussen , David Hildenbrand , Yan Zhao , Will Deacon , Kefeng Wang , Alex Williamson , Niklas Schnelle , Gerald Schaefer , Heiko Carstens , Vasily Gorbik , Alexander Gordeev , Christian Borntraeger , Sven Schnelle , linux-s390@vger.kernel.org Subject: [PATCH v2 11/19] s390/pci_mmio: Use follow_pfnmap API Date: Mon, 26 Aug 2024 16:43:45 -0400 Message-ID: <20240826204353.2228736-12-peterx@redhat.com> X-Mailer: git-send-email 2.45.0 In-Reply-To: <20240826204353.2228736-1-peterx@redhat.com> References: <20240826204353.2228736-1-peterx@redhat.com> Precedence: bulk X-Mailing-List: kvm@vger.kernel.org List-Id: List-Subscribe: List-Unsubscribe: MIME-Version: 1.0 Use the new API that can understand huge pfn mappings. Cc: Niklas Schnelle Cc: Gerald Schaefer Cc: Heiko Carstens Cc: Vasily Gorbik Cc: Alexander Gordeev Cc: Christian Borntraeger Cc: Sven Schnelle Cc: linux-s390@vger.kernel.org Signed-off-by: Peter Xu --- arch/s390/pci/pci_mmio.c | 22 ++++++++++++---------- 1 file changed, 12 insertions(+), 10 deletions(-) diff --git a/arch/s390/pci/pci_mmio.c b/arch/s390/pci/pci_mmio.c index 5398729bfe1b..de5c0b389a3e 100644 --- a/arch/s390/pci/pci_mmio.c +++ b/arch/s390/pci/pci_mmio.c @@ -118,12 +118,11 @@ static inline int __memcpy_toio_inuser(void __iomem *dst, SYSCALL_DEFINE3(s390_pci_mmio_write, unsigned long, mmio_addr, const void __user *, user_buffer, size_t, length) { + struct follow_pfnmap_args args = { }; u8 local_buf[64]; void __iomem *io_addr; void *buf; struct vm_area_struct *vma; - pte_t *ptep; - spinlock_t *ptl; long ret; if (!zpci_is_enabled()) @@ -169,11 +168,13 @@ SYSCALL_DEFINE3(s390_pci_mmio_write, unsigned long, mmio_addr, if (!(vma->vm_flags & VM_WRITE)) goto out_unlock_mmap; - ret = follow_pte(vma, mmio_addr, &ptep, &ptl); + args.address = mmio_addr; + args.vma = vma; + ret = follow_pfnmap_start(&args); if (ret) goto out_unlock_mmap; - io_addr = (void __iomem *)((pte_pfn(*ptep) << PAGE_SHIFT) | + io_addr = (void __iomem *)((args.pfn << PAGE_SHIFT) | (mmio_addr & ~PAGE_MASK)); if ((unsigned long) io_addr < ZPCI_IOMAP_ADDR_BASE) @@ -181,7 +182,7 @@ SYSCALL_DEFINE3(s390_pci_mmio_write, unsigned long, mmio_addr, ret = zpci_memcpy_toio(io_addr, buf, length); out_unlock_pt: - pte_unmap_unlock(ptep, ptl); + follow_pfnmap_end(&args); out_unlock_mmap: mmap_read_unlock(current->mm); out_free: @@ -260,12 +261,11 @@ static inline int __memcpy_fromio_inuser(void __user *dst, SYSCALL_DEFINE3(s390_pci_mmio_read, unsigned long, mmio_addr, void __user *, user_buffer, size_t, length) { + struct follow_pfnmap_args args = { }; u8 local_buf[64]; void __iomem *io_addr; void *buf; struct vm_area_struct *vma; - pte_t *ptep; - spinlock_t *ptl; long ret; if (!zpci_is_enabled()) @@ -308,11 +308,13 @@ SYSCALL_DEFINE3(s390_pci_mmio_read, unsigned long, mmio_addr, if (!(vma->vm_flags & VM_WRITE)) goto out_unlock_mmap; - ret = follow_pte(vma, mmio_addr, &ptep, &ptl); + args.vma = vma; + args.address = mmio_addr; + ret = follow_pfnmap_start(&args); if (ret) goto out_unlock_mmap; - io_addr = (void __iomem *)((pte_pfn(*ptep) << PAGE_SHIFT) | + io_addr = (void __iomem *)((args.pfn << PAGE_SHIFT) | (mmio_addr & ~PAGE_MASK)); if ((unsigned long) io_addr < ZPCI_IOMAP_ADDR_BASE) { @@ -322,7 +324,7 @@ SYSCALL_DEFINE3(s390_pci_mmio_read, unsigned long, mmio_addr, ret = zpci_memcpy_fromio(buf, io_addr, length); out_unlock_pt: - pte_unmap_unlock(ptep, ptl); + follow_pfnmap_end(&args); out_unlock_mmap: mmap_read_unlock(current->mm); From patchwork Mon Aug 26 20:43:46 2024 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Peter Xu X-Patchwork-Id: 13778426 Received: from us-smtp-delivery-124.mimecast.com (us-smtp-delivery-124.mimecast.com [170.10.129.124]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (No client certificate requested) by smtp.subspace.kernel.org (Postfix) with ESMTPS id 3E71B19F47A for ; Mon, 26 Aug 2024 20:44:23 +0000 (UTC) Authentication-Results: smtp.subspace.kernel.org; arc=none smtp.client-ip=170.10.129.124 ARC-Seal: i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1724705064; cv=none; b=JauRQJ/mZxSeLW0mFhbMWj97D5+/MJOjpD2dX3A1+tl3ii2GC2oq4lIFFZJLpbu/mA7AEDUWg+T7XnEchjR8GO0u8jIlEdwrtUrin0bthzDhuntZsTFVFOuqC1/VaV6LNRs24rK0yh+BwWat1ZvCXLYsj7ZcttMZgtfPm877biE= ARC-Message-Signature: i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1724705064; c=relaxed/simple; bh=tylq1muqjeOnFYCjzZOC/g07tmHzw2/ZVGGdJsJupkM=; h=From:To:Cc:Subject:Date:Message-ID:In-Reply-To:References: MIME-Version; b=SsMoxEA2dQqhpHYQCzWhSriLSleh4mDSImqUsrrzDdZeEn9+7KHQNQb/TWspn/XLT0a2grFRI25pKcg+sI/UbUmsyn0ZZ94Zz5EoN7H3FfY4dKQaTp0uaCJjjcdyqNylXlGr/MJFlHclYhOzquazS0L6ThFk10M8IuXEnqEoUKA= ARC-Authentication-Results: i=1; smtp.subspace.kernel.org; dmarc=pass (p=none dis=none) header.from=redhat.com; spf=pass smtp.mailfrom=redhat.com; dkim=pass (1024-bit key) header.d=redhat.com header.i=@redhat.com header.b=PafE/RZG; arc=none smtp.client-ip=170.10.129.124 Authentication-Results: smtp.subspace.kernel.org; dmarc=pass (p=none dis=none) header.from=redhat.com Authentication-Results: smtp.subspace.kernel.org; spf=pass smtp.mailfrom=redhat.com Authentication-Results: smtp.subspace.kernel.org; dkim=pass (1024-bit key) header.d=redhat.com header.i=@redhat.com header.b="PafE/RZG" DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=redhat.com; s=mimecast20190719; t=1724705062; h=from:from:reply-to:subject:subject:date:date:message-id:message-id: to:to:cc:cc:mime-version:mime-version: content-transfer-encoding:content-transfer-encoding: in-reply-to:in-reply-to:references:references; bh=moprxbUn5M7+L+63SZwwyqeq4iL0OHeeIJZbsfnkOho=; b=PafE/RZGFVTYMxQTpjZoYnyhDgjwSIbUlcuYbqOJj5uf1Cn+leiVrI+5fxn80s9scRnU4D ytXSUtmLaKapmz2+dRy6A0bEB+pj1At4oU+ZmBzwxjgIXKSahSZunouU+K8KljZpaKF5HJ Z5eoBzjHwAGQHnGqxehQe8nirXQcQGY= Received: from mail-qk1-f197.google.com (mail-qk1-f197.google.com [209.85.222.197]) by relay.mimecast.com with ESMTP with STARTTLS (version=TLSv1.3, cipher=TLS_AES_256_GCM_SHA384) id us-mta-677-5HcWGlDaO2uz19YfwB3Y0g-1; Mon, 26 Aug 2024 16:44:21 -0400 X-MC-Unique: 5HcWGlDaO2uz19YfwB3Y0g-1 Received: by mail-qk1-f197.google.com with SMTP id af79cd13be357-7a1d690cef9so882239385a.0 for ; Mon, 26 Aug 2024 13:44:21 -0700 (PDT) X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20230601; t=1724705061; x=1725309861; h=content-transfer-encoding:mime-version:references:in-reply-to :message-id:date:subject:cc:to:from:x-gm-message-state:from:to:cc :subject:date:message-id:reply-to; bh=moprxbUn5M7+L+63SZwwyqeq4iL0OHeeIJZbsfnkOho=; b=o7k1pPtfLr5xxKfWvtWjh+WpW8sTz7WBbLfybGv/wMD5MimyT1vpmuELn4VLbp1SLQ F6gsZcXH+vz0guObDydWL7edtQ6rTBLOvqft7Ts483cokO7wiIGrtdaaH6CByUAWorVs /G+yFGNPtHrtkHhf01jOwbwqcDxeSY+HLnViexnFXmJiexoG/Au2WBtFmyDnVYc9BLmU /nXnj+v8IrspDNZcz/9TZhX8yBCBU7x2/gOls0+RquhrpzO71msGRAITzDn5AZLIrxBj OxMAanJyURvgGLy1y6qWPDB9tMyXLdt3LRk0zuTNY/XJYeJxjOMaXGi6tKPVv+0w9j0J sW7w== X-Forwarded-Encrypted: i=1; AJvYcCXS2ulYZvaoxoMZgR78JK6+DUo8IGGpi/rJUn7bwgExO7oLkJB43jmhmuYKzVgIMw1ohdI=@vger.kernel.org X-Gm-Message-State: AOJu0YxoHz+chLG8JtnA/bGO79M246k/5K7xzh+iL5ePwXplvQbemdQg +9MMoKL+GsDSS0glkcqH33vFGB4homrXgODa0b8ZLX0Qnr+OhXjdNT/juIQgX91f9YnV4Q9Ihtd UNrheZusfj/mp+NrKYWv+mlK892NauBG9SW+0BtY3pC6GFXkbeg== X-Received: by 2002:a05:620a:1a92:b0:7a1:e341:d543 with SMTP id af79cd13be357-7a7e4cf3a3fmr158797885a.28.1724705060677; Mon, 26 Aug 2024 13:44:20 -0700 (PDT) X-Google-Smtp-Source: AGHT+IF8L+Naz3FkxDynSQ3yMse2qHglqFkf4LqBPi/5cD3sQEuFfYs6yWPMbT2Qk9XtpK98A+7trg== X-Received: by 2002:a05:620a:1a92:b0:7a1:e341:d543 with SMTP id af79cd13be357-7a7e4cf3a3fmr158793685a.28.1724705060390; Mon, 26 Aug 2024 13:44:20 -0700 (PDT) Received: from x1n.redhat.com (pool-99-254-121-117.cpe.net.cable.rogers.com. [99.254.121.117]) by smtp.gmail.com with ESMTPSA id af79cd13be357-7a67f3fd6c1sm491055185a.121.2024.08.26.13.44.18 (version=TLS1_3 cipher=TLS_AES_256_GCM_SHA384 bits=256/256); Mon, 26 Aug 2024 13:44:19 -0700 (PDT) From: Peter Xu To: linux-kernel@vger.kernel.org, linux-mm@kvack.org Cc: Gavin Shan , Catalin Marinas , x86@kernel.org, Ingo Molnar , Andrew Morton , Paolo Bonzini , Dave Hansen , Thomas Gleixner , Alistair Popple , kvm@vger.kernel.org, linux-arm-kernel@lists.infradead.org, Sean Christopherson , peterx@redhat.com, Oscar Salvador , Jason Gunthorpe , Borislav Petkov , Zi Yan , Axel Rasmussen , David Hildenbrand , Yan Zhao , Will Deacon , Kefeng Wang , Alex Williamson Subject: [PATCH v2 12/19] mm/x86/pat: Use the new follow_pfnmap API Date: Mon, 26 Aug 2024 16:43:46 -0400 Message-ID: <20240826204353.2228736-13-peterx@redhat.com> X-Mailer: git-send-email 2.45.0 In-Reply-To: <20240826204353.2228736-1-peterx@redhat.com> References: <20240826204353.2228736-1-peterx@redhat.com> Precedence: bulk X-Mailing-List: kvm@vger.kernel.org List-Id: List-Subscribe: List-Unsubscribe: MIME-Version: 1.0 Use the new API that can understand huge pfn mappings. Cc: x86@kernel.org Cc: Thomas Gleixner Cc: Ingo Molnar Cc: Borislav Petkov Cc: Dave Hansen Signed-off-by: Peter Xu --- arch/x86/mm/pat/memtype.c | 17 +++++++---------- 1 file changed, 7 insertions(+), 10 deletions(-) diff --git a/arch/x86/mm/pat/memtype.c b/arch/x86/mm/pat/memtype.c index 1fa0bf6ed295..f73b5ce270b3 100644 --- a/arch/x86/mm/pat/memtype.c +++ b/arch/x86/mm/pat/memtype.c @@ -951,23 +951,20 @@ static void free_pfn_range(u64 paddr, unsigned long size) static int follow_phys(struct vm_area_struct *vma, unsigned long *prot, resource_size_t *phys) { - pte_t *ptep, pte; - spinlock_t *ptl; + struct follow_pfnmap_args args = { .vma = vma, .address = vma->vm_start }; - if (follow_pte(vma, vma->vm_start, &ptep, &ptl)) + if (follow_pfnmap_start(&args)) return -EINVAL; - pte = ptep_get(ptep); - /* Never return PFNs of anon folios in COW mappings. */ - if (vm_normal_folio(vma, vma->vm_start, pte)) { - pte_unmap_unlock(ptep, ptl); + if (!args.special) { + follow_pfnmap_end(&args); return -EINVAL; } - *prot = pgprot_val(pte_pgprot(pte)); - *phys = (resource_size_t)pte_pfn(pte) << PAGE_SHIFT; - pte_unmap_unlock(ptep, ptl); + *prot = pgprot_val(args.pgprot); + *phys = (resource_size_t)args.pfn << PAGE_SHIFT; + follow_pfnmap_end(&args); return 0; } From patchwork Mon Aug 26 20:43:47 2024 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Peter Xu X-Patchwork-Id: 13778427 Received: from us-smtp-delivery-124.mimecast.com (us-smtp-delivery-124.mimecast.com [170.10.129.124]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (No client certificate requested) by smtp.subspace.kernel.org (Postfix) with ESMTPS id 68C3419FA91 for ; Mon, 26 Aug 2024 20:44:25 +0000 (UTC) Authentication-Results: smtp.subspace.kernel.org; arc=none smtp.client-ip=170.10.129.124 ARC-Seal: i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1724705066; cv=none; b=JZWVIMMg6XKN6wVajwpusTUb8KN16qkHT9mzFX0U440MkiKgEy0v0+1QaDyoaEA/DVJ3Tyb0ejGLJ9JMb5q4wDr38YqX0Ji21UbOwP5QlbxkqaZxQm0MuhNIN1hARqDltYANGmLdDemRvu8tJmpFPh0VyytizSuV8KnP1YuW3ek= ARC-Message-Signature: i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1724705066; c=relaxed/simple; bh=BQuf5gTrb+LFvvLLmklTnMaR8Ti7tRzNRxtQXaWJ5ko=; h=From:To:Cc:Subject:Date:Message-ID:In-Reply-To:References: MIME-Version; b=VaeCeV6bmoxJkHzneKgvP77QcM5R0blVluwoq9UJrybbebckDmLXpCq1I6ehYIrMm4igatU8JyvMl42OCJrHgZdJCrKROsmd1zWQIvC/P4ykeaVSqTuUVNgrk84NCBi4eRok+rLpbrMNKw1VdmD+a2vAdawPvgQkK619hFrc178= ARC-Authentication-Results: i=1; smtp.subspace.kernel.org; dmarc=pass (p=none dis=none) header.from=redhat.com; spf=pass smtp.mailfrom=redhat.com; dkim=pass (1024-bit key) header.d=redhat.com header.i=@redhat.com header.b=T411QiG3; arc=none smtp.client-ip=170.10.129.124 Authentication-Results: smtp.subspace.kernel.org; dmarc=pass (p=none dis=none) header.from=redhat.com Authentication-Results: smtp.subspace.kernel.org; spf=pass smtp.mailfrom=redhat.com Authentication-Results: smtp.subspace.kernel.org; dkim=pass (1024-bit key) header.d=redhat.com header.i=@redhat.com header.b="T411QiG3" DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=redhat.com; s=mimecast20190719; t=1724705064; h=from:from:reply-to:subject:subject:date:date:message-id:message-id: to:to:cc:cc:mime-version:mime-version: content-transfer-encoding:content-transfer-encoding: in-reply-to:in-reply-to:references:references; bh=HLBqit68L/U2mdwYaFkydFK3txHa+LkmL8n2YTQXnnw=; b=T411QiG3k0KnR/Jp0Y6Zo5OLSjptvrt0k8Gu+LCM8i79bNd8XItHKQ0L1NCpbH9j/KwLan 51Q6bQUCYhpvIAS4WXuLHiF/5tUcIkUXo2J8tyPM18QnR2eGY2fs0pEt3LeMYp7yAWv/Mz MkSV8/taXD+LQlEn7Pmx5vfr9jzcOVE= Received: from mail-yw1-f197.google.com (mail-yw1-f197.google.com [209.85.128.197]) by relay.mimecast.com with ESMTP with STARTTLS (version=TLSv1.3, cipher=TLS_AES_256_GCM_SHA384) id us-mta-391-OgZPF1ScPCajcp384GWIMg-1; Mon, 26 Aug 2024 16:44:23 -0400 X-MC-Unique: OgZPF1ScPCajcp384GWIMg-1 Received: by mail-yw1-f197.google.com with SMTP id 00721157ae682-6b3fec974e5so90045407b3.1 for ; Mon, 26 Aug 2024 13:44:23 -0700 (PDT) X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20230601; t=1724705063; x=1725309863; h=content-transfer-encoding:mime-version:references:in-reply-to :message-id:date:subject:cc:to:from:x-gm-message-state:from:to:cc :subject:date:message-id:reply-to; bh=HLBqit68L/U2mdwYaFkydFK3txHa+LkmL8n2YTQXnnw=; b=YpIRjmD0T5IjMf+JC08SdrrY6U08Iuivjr78iamrph8HVewyVFG01+qAOdlq6CDmi3 S13zsgcIAeZpXgX65dlXL6PetjJWigKbhSWiPcjAgReLJLsOcZIufu1DZyXXt/DmopAb coQGkkGawZG2ISWYYGgI9k+sb9wRKkCmmN2jJclQ3qf/nzMCC6hYiKhd/KcPmMCBAw3D lZBQFR+ly76OZzYUjo4HYRINDw0T0g1N8/i35fgO1rUWDmMnO2PeQWbMiyOXaClgnMlU TRcJTPHzzx+OoFrL78+61GsYGGozUjW/lVd7ULOLcwY4r5ZqJrWrDG0PObU69CCqKD4e u7Jg== X-Forwarded-Encrypted: i=1; AJvYcCVKuWZOw6q+//gIFG2b6lMLzNYiOlPsHV96iYDCcWlViUV1PsAwGUma0Uk/vRCMOv5CdlM=@vger.kernel.org X-Gm-Message-State: AOJu0YzOgR1lcejjN1mTk0yEohW4NY0k/+W+43XQsGTnixkAYouneD+D agr+a3tJx3kJAexo7WwcfWLmL+5+zEHkUgzxKE9WXEcWSq788HVe4J81L9AeJxio/3qtjCkzskq eAU6g2W1d5upR8O5f8/fGIx7jX2eqBfcyD2rPz+a8kfkEEHladw== X-Received: by 2002:a05:690c:39b:b0:62f:aaaa:187a with SMTP id 00721157ae682-6c625a4cc55mr151107017b3.14.1724705062742; Mon, 26 Aug 2024 13:44:22 -0700 (PDT) X-Google-Smtp-Source: AGHT+IGeaoJv/EbB8r7mSzacjacGidgTJuNMUQKi5I30qkj93MEO5BseX564RMMiUfXmELj5ALvNSw== X-Received: by 2002:a05:690c:39b:b0:62f:aaaa:187a with SMTP id 00721157ae682-6c625a4cc55mr151106717b3.14.1724705062426; Mon, 26 Aug 2024 13:44:22 -0700 (PDT) Received: from x1n.redhat.com (pool-99-254-121-117.cpe.net.cable.rogers.com. [99.254.121.117]) by smtp.gmail.com with ESMTPSA id af79cd13be357-7a67f3fd6c1sm491055185a.121.2024.08.26.13.44.20 (version=TLS1_3 cipher=TLS_AES_256_GCM_SHA384 bits=256/256); Mon, 26 Aug 2024 13:44:21 -0700 (PDT) From: Peter Xu To: linux-kernel@vger.kernel.org, linux-mm@kvack.org Cc: Gavin Shan , Catalin Marinas , x86@kernel.org, Ingo Molnar , Andrew Morton , Paolo Bonzini , Dave Hansen , Thomas Gleixner , Alistair Popple , kvm@vger.kernel.org, linux-arm-kernel@lists.infradead.org, Sean Christopherson , peterx@redhat.com, Oscar Salvador , Jason Gunthorpe , Borislav Petkov , Zi Yan , Axel Rasmussen , David Hildenbrand , Yan Zhao , Will Deacon , Kefeng Wang , Alex Williamson Subject: [PATCH v2 13/19] vfio: Use the new follow_pfnmap API Date: Mon, 26 Aug 2024 16:43:47 -0400 Message-ID: <20240826204353.2228736-14-peterx@redhat.com> X-Mailer: git-send-email 2.45.0 In-Reply-To: <20240826204353.2228736-1-peterx@redhat.com> References: <20240826204353.2228736-1-peterx@redhat.com> Precedence: bulk X-Mailing-List: kvm@vger.kernel.org List-Id: List-Subscribe: List-Unsubscribe: MIME-Version: 1.0 Use the new API that can understand huge pfn mappings. Cc: Alex Williamson Cc: Jason Gunthorpe Signed-off-by: Peter Xu --- drivers/vfio/vfio_iommu_type1.c | 16 ++++++---------- 1 file changed, 6 insertions(+), 10 deletions(-) diff --git a/drivers/vfio/vfio_iommu_type1.c b/drivers/vfio/vfio_iommu_type1.c index 0960699e7554..bf391b40e576 100644 --- a/drivers/vfio/vfio_iommu_type1.c +++ b/drivers/vfio/vfio_iommu_type1.c @@ -513,12 +513,10 @@ static int follow_fault_pfn(struct vm_area_struct *vma, struct mm_struct *mm, unsigned long vaddr, unsigned long *pfn, bool write_fault) { - pte_t *ptep; - pte_t pte; - spinlock_t *ptl; + struct follow_pfnmap_args args = { .vma = vma, .address = vaddr }; int ret; - ret = follow_pte(vma, vaddr, &ptep, &ptl); + ret = follow_pfnmap_start(&args); if (ret) { bool unlocked = false; @@ -532,19 +530,17 @@ static int follow_fault_pfn(struct vm_area_struct *vma, struct mm_struct *mm, if (ret) return ret; - ret = follow_pte(vma, vaddr, &ptep, &ptl); + ret = follow_pfnmap_start(&args); if (ret) return ret; } - pte = ptep_get(ptep); - - if (write_fault && !pte_write(pte)) + if (write_fault && !args.writable) ret = -EFAULT; else - *pfn = pte_pfn(pte); + *pfn = args.pfn; - pte_unmap_unlock(ptep, ptl); + follow_pfnmap_end(&args); return ret; } From patchwork Mon Aug 26 20:43:48 2024 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Peter Xu X-Patchwork-Id: 13778428 Received: from us-smtp-delivery-124.mimecast.com (us-smtp-delivery-124.mimecast.com [170.10.129.124]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (No client certificate requested) by smtp.subspace.kernel.org (Postfix) with ESMTPS id 11BB71A00F1 for ; Mon, 26 Aug 2024 20:44:26 +0000 (UTC) Authentication-Results: smtp.subspace.kernel.org; arc=none smtp.client-ip=170.10.129.124 ARC-Seal: i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1724705068; cv=none; b=EqwdTUYsl46jjGClcv6XAjvThJ1aAP/V1perb6BMKXaAcRSGL1XddQUArH7lzqhIrCtpyRw5EjRnYeCc18bJqYrTEtIYdmVPPlZhCCP+FDjkythebpr70aN0SMQu4BRveBN62PqAMY3PskSOVHA9q6JhuqzK5Jh2d1h1vx90m+Q= ARC-Message-Signature: i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1724705068; c=relaxed/simple; bh=RxaKGf99KcIXWtv8SyMqO8KfjLx2VTZrfelSTDMLhuM=; h=From:To:Cc:Subject:Date:Message-ID:In-Reply-To:References: MIME-Version; b=jDnK9pttdOAVJk/R04FIlnAtvRO3iXpFVBN47ej8lNg42U3OeKtUUKT8ncWihNcR/pdDd4oV6umpVPU8ZrFF0bMsHpac0u68uQENeEyIvGdJVGz3oGGpUJggSLkolQkluwc4PkYpOeqAReaI5BCI3dhUhaHMqA5KaJ7UgGRGmhE= ARC-Authentication-Results: i=1; smtp.subspace.kernel.org; dmarc=pass (p=none dis=none) header.from=redhat.com; spf=pass smtp.mailfrom=redhat.com; dkim=pass (1024-bit key) header.d=redhat.com header.i=@redhat.com header.b=XHPLT0oc; arc=none smtp.client-ip=170.10.129.124 Authentication-Results: smtp.subspace.kernel.org; dmarc=pass (p=none dis=none) header.from=redhat.com Authentication-Results: smtp.subspace.kernel.org; spf=pass smtp.mailfrom=redhat.com Authentication-Results: smtp.subspace.kernel.org; dkim=pass (1024-bit key) header.d=redhat.com header.i=@redhat.com header.b="XHPLT0oc" DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=redhat.com; s=mimecast20190719; t=1724705066; h=from:from:reply-to:subject:subject:date:date:message-id:message-id: to:to:cc:cc:mime-version:mime-version: content-transfer-encoding:content-transfer-encoding: in-reply-to:in-reply-to:references:references; bh=d0kI5he4gLOJUpsF87CpEeiS1FOCa69CaYI81su9df0=; b=XHPLT0ocyeIQ6bYspsizMfyy1L6CijBqwhoWZYQQKXNk5pXtlAAmZMWKWiCexzSL0kNvjI +TbjBiSfURXsDzVHKba+Jtk3fRwp2PyRZmfY4iq+qek+n77XJpc/GLCkS2p1OUneD2wf70 i3Zj+AJdEOOwMpGLsPMKrRUv81iBb1s= Received: from mail-qk1-f197.google.com (mail-qk1-f197.google.com [209.85.222.197]) by relay.mimecast.com with ESMTP with STARTTLS (version=TLSv1.3, cipher=TLS_AES_256_GCM_SHA384) id us-mta-679-l84EIBdLPK-WoGI5AGF9uQ-1; Mon, 26 Aug 2024 16:44:25 -0400 X-MC-Unique: l84EIBdLPK-WoGI5AGF9uQ-1 Received: by mail-qk1-f197.google.com with SMTP id af79cd13be357-7a66c19d33fso676637985a.0 for ; Mon, 26 Aug 2024 13:44:25 -0700 (PDT) X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20230601; t=1724705064; x=1725309864; h=content-transfer-encoding:mime-version:references:in-reply-to :message-id:date:subject:cc:to:from:x-gm-message-state:from:to:cc :subject:date:message-id:reply-to; bh=d0kI5he4gLOJUpsF87CpEeiS1FOCa69CaYI81su9df0=; b=frtEy8/12PTcVx8HhWPTtWol4+aPRK7wCKO22VYzcNYgMvQjq9Ih8T6CQ8PqSXN2Po 15MOiGwIn8o5N84+tAYjiJwRRRv2uM/i8w6jzmw8gZqnf0spb04T5TIWCO0xur1USq1F 3EOM8ZLDY+RvYnpgA2SVpZmAEtz9MR9SthoAeImQPDgIVr8qXT4JkjBhP6rbrE8jDeUQ 0yNdgS+IV73UVqfyKW1O26bNesf/HMQqorHlP7FrtkWxeLp+D3PSOqncE8ZmehO+lYoy sTrFpROJ3VvVk+mXac4Knk0L7pbNQuLLebxqlTev8yS7H6Sl1BjzAhPv6V7qR9NcB0P+ WY3g== X-Forwarded-Encrypted: i=1; AJvYcCWlMBwwETXNWP466CqoQ8CFmYgLCFK40e3jfSdh12MwxWU/ajDGX7L6PM5tQYtxKlPSItE=@vger.kernel.org X-Gm-Message-State: AOJu0Yx747qNVGFVKva/Q1up9Rhd9JOvkt3f6yD9D74QyYAl5nlf3s0H WRx9xZQU9iZPlpXCXvNo2uCC2Juh+WUWQMJfH3cmUU9y4ldZ2ScZKq3uPflBD7Vc5Oealm4xmxN llXdnwEfPu80mb3C14J1ZTCsrHGaBLLpuCzAqH/xwBLW3ILhKHQ== X-Received: by 2002:a05:620a:2484:b0:79e:ff41:fd47 with SMTP id af79cd13be357-7a7e4dc9528mr93288085a.28.1724705064647; Mon, 26 Aug 2024 13:44:24 -0700 (PDT) X-Google-Smtp-Source: AGHT+IEloa5qz2QVMofE8PCWJJ6gTfdMgC7cjn+h2anZ9qsldjcOaIzzaCXqVOAeeqkMcyU6B13FuA== X-Received: by 2002:a05:620a:2484:b0:79e:ff41:fd47 with SMTP id af79cd13be357-7a7e4dc9528mr93286485a.28.1724705064315; Mon, 26 Aug 2024 13:44:24 -0700 (PDT) Received: from x1n.redhat.com (pool-99-254-121-117.cpe.net.cable.rogers.com. [99.254.121.117]) by smtp.gmail.com with ESMTPSA id af79cd13be357-7a67f3fd6c1sm491055185a.121.2024.08.26.13.44.22 (version=TLS1_3 cipher=TLS_AES_256_GCM_SHA384 bits=256/256); Mon, 26 Aug 2024 13:44:23 -0700 (PDT) From: Peter Xu To: linux-kernel@vger.kernel.org, linux-mm@kvack.org Cc: Gavin Shan , Catalin Marinas , x86@kernel.org, Ingo Molnar , Andrew Morton , Paolo Bonzini , Dave Hansen , Thomas Gleixner , Alistair Popple , kvm@vger.kernel.org, linux-arm-kernel@lists.infradead.org, Sean Christopherson , peterx@redhat.com, Oscar Salvador , Jason Gunthorpe , Borislav Petkov , Zi Yan , Axel Rasmussen , David Hildenbrand , Yan Zhao , Will Deacon , Kefeng Wang , Alex Williamson Subject: [PATCH v2 14/19] acrn: Use the new follow_pfnmap API Date: Mon, 26 Aug 2024 16:43:48 -0400 Message-ID: <20240826204353.2228736-15-peterx@redhat.com> X-Mailer: git-send-email 2.45.0 In-Reply-To: <20240826204353.2228736-1-peterx@redhat.com> References: <20240826204353.2228736-1-peterx@redhat.com> Precedence: bulk X-Mailing-List: kvm@vger.kernel.org List-Id: List-Subscribe: List-Unsubscribe: MIME-Version: 1.0 Use the new API that can understand huge pfn mappings. Signed-off-by: Peter Xu --- drivers/virt/acrn/mm.c | 16 +++++++++------- 1 file changed, 9 insertions(+), 7 deletions(-) diff --git a/drivers/virt/acrn/mm.c b/drivers/virt/acrn/mm.c index db8ff1d0ac23..4c2f28715b70 100644 --- a/drivers/virt/acrn/mm.c +++ b/drivers/virt/acrn/mm.c @@ -177,9 +177,7 @@ int acrn_vm_ram_map(struct acrn_vm *vm, struct acrn_vm_memmap *memmap) vma = vma_lookup(current->mm, memmap->vma_base); if (vma && ((vma->vm_flags & VM_PFNMAP) != 0)) { unsigned long start_pfn, cur_pfn; - spinlock_t *ptl; bool writable; - pte_t *ptep; if ((memmap->vma_base + memmap->len) > vma->vm_end) { mmap_read_unlock(current->mm); @@ -187,16 +185,20 @@ int acrn_vm_ram_map(struct acrn_vm *vm, struct acrn_vm_memmap *memmap) } for (i = 0; i < nr_pages; i++) { - ret = follow_pte(vma, memmap->vma_base + i * PAGE_SIZE, - &ptep, &ptl); + struct follow_pfnmap_args args = { + .vma = vma, + .address = memmap->vma_base + i * PAGE_SIZE, + }; + + ret = follow_pfnmap_start(&args); if (ret) break; - cur_pfn = pte_pfn(ptep_get(ptep)); + cur_pfn = args.pfn; if (i == 0) start_pfn = cur_pfn; - writable = !!pte_write(ptep_get(ptep)); - pte_unmap_unlock(ptep, ptl); + writable = args.writable; + follow_pfnmap_end(&args); /* Disallow write access if the PTE is not writable. */ if (!writable && From patchwork Mon Aug 26 20:43:49 2024 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Peter Xu X-Patchwork-Id: 13778429 Received: from us-smtp-delivery-124.mimecast.com (us-smtp-delivery-124.mimecast.com [170.10.133.124]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (No client certificate requested) by smtp.subspace.kernel.org (Postfix) with ESMTPS id 741091A01D4 for ; Mon, 26 Aug 2024 20:44:29 +0000 (UTC) Authentication-Results: smtp.subspace.kernel.org; arc=none smtp.client-ip=170.10.133.124 ARC-Seal: i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1724705071; cv=none; b=HxgmESalJihDU2kghNU1ZDBMpJvGpR7ohcz6yXE9nflOZ/eRyrJ1IDEixkawLHQonzRKiz65sESRKdGdv4rTvntwm2VtRuV6pnAKyBYiUVuxbUhewjoNdY1C3g7WUk2yMIqc5PfWU4NaIO2zmDxpS403LhsDf6mDeOdlDTk88Ro= ARC-Message-Signature: i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1724705071; c=relaxed/simple; bh=hpLmEYhsByAEmqEdBsP006I9UauZryxW/FlKEPWnJc4=; h=From:To:Cc:Subject:Date:Message-ID:In-Reply-To:References: MIME-Version; b=foliOG1ioSZ/GpC5VLARFpBkC3YFkbvdOgHZU5LarTYOgoAr3wvqiAZ0JC7qcxjgHFUrKOtDjB1pJux78JezjakFAFWyytFULk1zCG/7zbYLybu8eX4b8Ai1T/v40wclXGgKBqljyvl8wNwQSPTXJa+JjkWInEGjDNWuA4YT7c0= ARC-Authentication-Results: i=1; smtp.subspace.kernel.org; dmarc=pass (p=none dis=none) header.from=redhat.com; spf=pass smtp.mailfrom=redhat.com; dkim=pass (1024-bit key) header.d=redhat.com header.i=@redhat.com header.b=RE2//aLs; arc=none smtp.client-ip=170.10.133.124 Authentication-Results: smtp.subspace.kernel.org; dmarc=pass (p=none dis=none) header.from=redhat.com Authentication-Results: smtp.subspace.kernel.org; spf=pass smtp.mailfrom=redhat.com Authentication-Results: smtp.subspace.kernel.org; dkim=pass (1024-bit key) header.d=redhat.com header.i=@redhat.com header.b="RE2//aLs" DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=redhat.com; s=mimecast20190719; t=1724705068; h=from:from:reply-to:subject:subject:date:date:message-id:message-id: to:to:cc:cc:mime-version:mime-version: content-transfer-encoding:content-transfer-encoding: in-reply-to:in-reply-to:references:references; bh=t75nynj+5ccZ6qy3UooqAYOOLyCQeJyCwmKKtTf1Ss4=; b=RE2//aLsidYqxI/Vq6eXXVa8odw+UOLLRE4cBut+e9JGmNhrdEGZav1WTRmgPolejdkoS2 GXNceF38y1X7f85RMpx7fRHAn8o442O4PKfKb9LIQkO53QWXQISHWxIDypra9ZkyCgoLm1 ySXdhrh0LQKdxI7FLpsx0tu4+ft/VvY= Received: from mail-oa1-f71.google.com (mail-oa1-f71.google.com [209.85.160.71]) by relay.mimecast.com with ESMTP with STARTTLS (version=TLSv1.3, cipher=TLS_AES_256_GCM_SHA384) id us-mta-138-inRu7sl3OE2epc-tPpi9xA-1; Mon, 26 Aug 2024 16:44:27 -0400 X-MC-Unique: inRu7sl3OE2epc-tPpi9xA-1 Received: by mail-oa1-f71.google.com with SMTP id 586e51a60fabf-27061a48e70so6268033fac.1 for ; Mon, 26 Aug 2024 13:44:27 -0700 (PDT) X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20230601; t=1724705066; x=1725309866; h=content-transfer-encoding:mime-version:references:in-reply-to :message-id:date:subject:cc:to:from:x-gm-message-state:from:to:cc :subject:date:message-id:reply-to; bh=t75nynj+5ccZ6qy3UooqAYOOLyCQeJyCwmKKtTf1Ss4=; b=Gp6QmrQcUJ7V3aoTeEnydLXXV6j6SpZThDM1GAo6sqYG6cthlU8FrsGldADOuYJq9W BKYf5Vmu6m5bPtDA+OQ3BWu0LoSZyPr5syYi4kxZcO8zmPoTAAgFKb+hVj5IuCWiSqTK NfcJDFsDFJ3jchHam2hM/priUjEKSvgm5re4WZFzRTOBq4xzLp1g6Taj/reLoS9twyBH sBMjfzMQiAQAjFM5jVEsrdk3F02DzX8hjdKdK6mhAaxvdXOaPhXW6E3tosu/HQ2RuGnS seil02Nr3De9+87rlgT2Q1m0SPn8+mPCeO9jZpLZsiz48056lU6fIHxBs4ULnb6/OzFC g3vA== X-Forwarded-Encrypted: i=1; AJvYcCVGHywL/ikKt5pHcV+PI34V3yuaf6fJnonem1xnhJIaQxVBrNg1cNlzq+IGJnUIf8u4i4E=@vger.kernel.org X-Gm-Message-State: AOJu0YybJpkE1hhOL1FIBNevDGBLoC3JLb9krFfBD8y3hbF/aNQYOaoJ xetXrzQL0q/btjj6OlZyDHR8iCBy+pysJHgqQslB89AqG9SlxSrmKT8Xd/dF7Qvp9gjW93gV5nT XOq4iJo0CtJX1qtF5Q3siGI+DUrSSss6WScbCqMHARoWwpitPKA== X-Received: by 2002:a05:6358:3a07:b0:1ad:10ff:341e with SMTP id e5c5f4694b2df-1b5c3a3b69dmr1457843355d.4.1724705066539; Mon, 26 Aug 2024 13:44:26 -0700 (PDT) X-Google-Smtp-Source: AGHT+IGoPajws+HFHWojCbb3D8qBEyBE/zIeN3nkxAxD5pVBn23+qUQ20h6jqSHyvqmBY4DTTGYw+g== X-Received: by 2002:a05:6358:3a07:b0:1ad:10ff:341e with SMTP id e5c5f4694b2df-1b5c3a3b69dmr1457841255d.4.1724705066240; Mon, 26 Aug 2024 13:44:26 -0700 (PDT) Received: from x1n.redhat.com (pool-99-254-121-117.cpe.net.cable.rogers.com. [99.254.121.117]) by smtp.gmail.com with ESMTPSA id af79cd13be357-7a67f3fd6c1sm491055185a.121.2024.08.26.13.44.24 (version=TLS1_3 cipher=TLS_AES_256_GCM_SHA384 bits=256/256); Mon, 26 Aug 2024 13:44:25 -0700 (PDT) From: Peter Xu To: linux-kernel@vger.kernel.org, linux-mm@kvack.org Cc: Gavin Shan , Catalin Marinas , x86@kernel.org, Ingo Molnar , Andrew Morton , Paolo Bonzini , Dave Hansen , Thomas Gleixner , Alistair Popple , kvm@vger.kernel.org, linux-arm-kernel@lists.infradead.org, Sean Christopherson , peterx@redhat.com, Oscar Salvador , Jason Gunthorpe , Borislav Petkov , Zi Yan , Axel Rasmussen , David Hildenbrand , Yan Zhao , Will Deacon , Kefeng Wang , Alex Williamson Subject: [PATCH v2 15/19] mm/access_process_vm: Use the new follow_pfnmap API Date: Mon, 26 Aug 2024 16:43:49 -0400 Message-ID: <20240826204353.2228736-16-peterx@redhat.com> X-Mailer: git-send-email 2.45.0 In-Reply-To: <20240826204353.2228736-1-peterx@redhat.com> References: <20240826204353.2228736-1-peterx@redhat.com> Precedence: bulk X-Mailing-List: kvm@vger.kernel.org List-Id: List-Subscribe: List-Unsubscribe: MIME-Version: 1.0 Use the new API that can understand huge pfn mappings. Signed-off-by: Peter Xu --- mm/memory.c | 28 ++++++++++++++-------------- 1 file changed, 14 insertions(+), 14 deletions(-) diff --git a/mm/memory.c b/mm/memory.c index 0b136c398257..b5d07f493d5d 100644 --- a/mm/memory.c +++ b/mm/memory.c @@ -6342,34 +6342,34 @@ int generic_access_phys(struct vm_area_struct *vma, unsigned long addr, resource_size_t phys_addr; unsigned long prot = 0; void __iomem *maddr; - pte_t *ptep, pte; - spinlock_t *ptl; int offset = offset_in_page(addr); int ret = -EINVAL; + bool writable; + struct follow_pfnmap_args args = { .vma = vma, .address = addr }; retry: - if (follow_pte(vma, addr, &ptep, &ptl)) + if (follow_pfnmap_start(&args)) return -EINVAL; - pte = ptep_get(ptep); - pte_unmap_unlock(ptep, ptl); + prot = pgprot_val(args.pgprot); + phys_addr = (resource_size_t)args.pfn << PAGE_SHIFT; + writable = args.writable; + follow_pfnmap_end(&args); - prot = pgprot_val(pte_pgprot(pte)); - phys_addr = (resource_size_t)pte_pfn(pte) << PAGE_SHIFT; - - if ((write & FOLL_WRITE) && !pte_write(pte)) + if ((write & FOLL_WRITE) && !writable) return -EINVAL; maddr = ioremap_prot(phys_addr, PAGE_ALIGN(len + offset), prot); if (!maddr) return -ENOMEM; - if (follow_pte(vma, addr, &ptep, &ptl)) + if (follow_pfnmap_start(&args)) goto out_unmap; - if (!pte_same(pte, ptep_get(ptep))) { - pte_unmap_unlock(ptep, ptl); + if ((prot != pgprot_val(args.pgprot)) || + (phys_addr != (args.pfn << PAGE_SHIFT)) || + (writable != args.writable)) { + follow_pfnmap_end(&args); iounmap(maddr); - goto retry; } @@ -6378,7 +6378,7 @@ int generic_access_phys(struct vm_area_struct *vma, unsigned long addr, else memcpy_fromio(buf, maddr + offset, len); ret = len; - pte_unmap_unlock(ptep, ptl); + follow_pfnmap_end(&args); out_unmap: iounmap(maddr); From patchwork Mon Aug 26 20:43:50 2024 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Peter Xu X-Patchwork-Id: 13778430 Received: from us-smtp-delivery-124.mimecast.com (us-smtp-delivery-124.mimecast.com [170.10.133.124]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (No client certificate requested) by smtp.subspace.kernel.org (Postfix) with ESMTPS id 38A381A01BB for ; Mon, 26 Aug 2024 20:44:33 +0000 (UTC) Authentication-Results: smtp.subspace.kernel.org; arc=none smtp.client-ip=170.10.133.124 ARC-Seal: i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1724705074; cv=none; b=url4E0M46otMHAJOFluahFk2+2L8Bz228nd3cI6TppxWd1FwhaTeQMytkKXMxsbLS6kjo5zxBsoE1UQ7JwNPmwYl6dynjPpl2sZpTHwXrhAHXEFeVaMcdAFTpcqAqP1lPGQBtFkFHn1z/6mHUSZVwZBsT1WSj71hlp8dA3Y17Bw= ARC-Message-Signature: i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1724705074; c=relaxed/simple; bh=Eh2leUymwhy+QIbtX8/+/NdJiGHoDFC3uu2OHYBcoWg=; h=From:To:Cc:Subject:Date:Message-ID:In-Reply-To:References: MIME-Version; b=rRielPX2pk5bT1oQJQMAHyaDVQX/DHZat/vYAOv6JJ0vXVBMBL0E98OUX3sOB2iyTQRNYAzOMCWPLBuI/EJlH+X9VJ31ztWpp5unOQGeKBn6ubp8Pa6BLUdLcDHOnXoY/3IU6ruRVRqfbsjt5V8A7rvyAGpQF/7Vb/bw5wZAIEY= ARC-Authentication-Results: i=1; smtp.subspace.kernel.org; dmarc=pass (p=none dis=none) header.from=redhat.com; spf=pass smtp.mailfrom=redhat.com; dkim=pass (1024-bit key) header.d=redhat.com header.i=@redhat.com header.b=GLRXzQZt; arc=none smtp.client-ip=170.10.133.124 Authentication-Results: smtp.subspace.kernel.org; dmarc=pass (p=none dis=none) header.from=redhat.com Authentication-Results: smtp.subspace.kernel.org; spf=pass smtp.mailfrom=redhat.com Authentication-Results: smtp.subspace.kernel.org; dkim=pass (1024-bit key) header.d=redhat.com header.i=@redhat.com header.b="GLRXzQZt" DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=redhat.com; s=mimecast20190719; t=1724705072; h=from:from:reply-to:subject:subject:date:date:message-id:message-id: to:to:cc:cc:mime-version:mime-version: content-transfer-encoding:content-transfer-encoding: in-reply-to:in-reply-to:references:references; bh=X8WHCJ7kNzF2XCQH70BbHdio2suKNHxAf6rhsHMkPbs=; b=GLRXzQZtGfDxos/RYkF6U37Ksutcne5zEVq5cZMb/h2eBdKI79hxQoXBOwfNTTSHUoXyHE fhf8ifPAPzZlcUkyGU66GaouUN+FRya69xbOO44n0PnMPj6UaRML5iEpE8MxbEQJIIDCdU XDjbeHfC1cqSgrSq81hze5JpfT/Amj0= Received: from mail-ua1-f69.google.com (mail-ua1-f69.google.com [209.85.222.69]) by relay.mimecast.com with ESMTP with STARTTLS (version=TLSv1.3, cipher=TLS_AES_256_GCM_SHA384) id us-mta-66-etn_Z8nVOgOfYUK9M1_ijA-1; Mon, 26 Aug 2024 16:44:29 -0400 X-MC-Unique: etn_Z8nVOgOfYUK9M1_ijA-1 Received: by mail-ua1-f69.google.com with SMTP id a1e0cc1a2514c-842f9f7509eso1359979241.2 for ; Mon, 26 Aug 2024 13:44:28 -0700 (PDT) X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20230601; t=1724705068; x=1725309868; h=content-transfer-encoding:mime-version:references:in-reply-to :message-id:date:subject:cc:to:from:x-gm-message-state:from:to:cc :subject:date:message-id:reply-to; bh=X8WHCJ7kNzF2XCQH70BbHdio2suKNHxAf6rhsHMkPbs=; b=obXMh20bNYR29p6iGFj2Aacbs+0S64OtbkVG5mm00QFaozSg3xQg5vhK6TtXvMTO8K SyKRnCRltyBHC5dCiJgNwAhv+Ytl5VXxOq1mE64nGxqal230/jLXSF6gxodvjg3FpIVf 7j9LyTQmXDuw5fWDlNLc676lXJ+AYyu/CfJZpu3kEp68hMkx7Vfx9OuPjzNIagOAVFYe g+mMIyPty3AWO+DnZX7VBTOy9ASAm8jcx0PX1lrp38FM2ghvJ2CqEVPmhyeEs+fQTvg7 Ymox3UkaTBY27b8p1ijZw8BjNo4TVrYSIzC5lbsZtxr3Az3mdU6qjltLGocmQLbQXBsg r4Aw== X-Forwarded-Encrypted: i=1; AJvYcCUBqRgN+uLy+5068dSUkHPu2LpaBfOjVJuDou2DxzRZ2hOCnFF4SX9DtQwhFvBELGc4HKM=@vger.kernel.org X-Gm-Message-State: AOJu0YxRCkAynaQMZ8mr8qs89yweaA3RpJelGx/CVJGevY7kpb6esoRg zV0JDnJLyexCROr565nDOnk73oz1kI7uX4h8J1zeZTpX8Lks6+xA7xEOM11uyFTJlQS6WIB+vkj AISY3ll3HLeoroJ8rCbVeTE6A1mjToQC+W+STZ82q+UdhG3uQTQ== X-Received: by 2002:a05:6102:3e94:b0:493:e585:6ce3 with SMTP id ada2fe7eead31-49a3bd1cd83mr1005200137.31.1724705068479; Mon, 26 Aug 2024 13:44:28 -0700 (PDT) X-Google-Smtp-Source: AGHT+IGKOe4NKlkU/HSIr+h3CCvXyG5C70MeRvYsCyQPr65oWdFZ+m1iMwzpLlsyBV/cJVa8qCplQQ== X-Received: by 2002:a05:6102:3e94:b0:493:e585:6ce3 with SMTP id ada2fe7eead31-49a3bd1cd83mr1005176137.31.1724705068153; Mon, 26 Aug 2024 13:44:28 -0700 (PDT) Received: from x1n.redhat.com (pool-99-254-121-117.cpe.net.cable.rogers.com. [99.254.121.117]) by smtp.gmail.com with ESMTPSA id af79cd13be357-7a67f3fd6c1sm491055185a.121.2024.08.26.13.44.26 (version=TLS1_3 cipher=TLS_AES_256_GCM_SHA384 bits=256/256); Mon, 26 Aug 2024 13:44:27 -0700 (PDT) From: Peter Xu To: linux-kernel@vger.kernel.org, linux-mm@kvack.org Cc: Gavin Shan , Catalin Marinas , x86@kernel.org, Ingo Molnar , Andrew Morton , Paolo Bonzini , Dave Hansen , Thomas Gleixner , Alistair Popple , kvm@vger.kernel.org, linux-arm-kernel@lists.infradead.org, Sean Christopherson , peterx@redhat.com, Oscar Salvador , Jason Gunthorpe , Borislav Petkov , Zi Yan , Axel Rasmussen , David Hildenbrand , Yan Zhao , Will Deacon , Kefeng Wang , Alex Williamson Subject: [PATCH v2 16/19] mm: Remove follow_pte() Date: Mon, 26 Aug 2024 16:43:50 -0400 Message-ID: <20240826204353.2228736-17-peterx@redhat.com> X-Mailer: git-send-email 2.45.0 In-Reply-To: <20240826204353.2228736-1-peterx@redhat.com> References: <20240826204353.2228736-1-peterx@redhat.com> Precedence: bulk X-Mailing-List: kvm@vger.kernel.org List-Id: List-Subscribe: List-Unsubscribe: MIME-Version: 1.0 follow_pte() users have been converted to follow_pfnmap*(). Remove the API. Signed-off-by: Peter Xu --- include/linux/mm.h | 2 -- mm/memory.c | 73 ---------------------------------------------- 2 files changed, 75 deletions(-) diff --git a/include/linux/mm.h b/include/linux/mm.h index 161d496bfd18..b31d4bdd65ad 100644 --- a/include/linux/mm.h +++ b/include/linux/mm.h @@ -2368,8 +2368,6 @@ void free_pgd_range(struct mmu_gather *tlb, unsigned long addr, unsigned long end, unsigned long floor, unsigned long ceiling); int copy_page_range(struct vm_area_struct *dst_vma, struct vm_area_struct *src_vma); -int follow_pte(struct vm_area_struct *vma, unsigned long address, - pte_t **ptepp, spinlock_t **ptlp); int generic_access_phys(struct vm_area_struct *vma, unsigned long addr, void *buf, int len, int write); diff --git a/mm/memory.c b/mm/memory.c index b5d07f493d5d..288f81a8698e 100644 --- a/mm/memory.c +++ b/mm/memory.c @@ -6100,79 +6100,6 @@ int __pmd_alloc(struct mm_struct *mm, pud_t *pud, unsigned long address) } #endif /* __PAGETABLE_PMD_FOLDED */ -/** - * follow_pte - look up PTE at a user virtual address - * @vma: the memory mapping - * @address: user virtual address - * @ptepp: location to store found PTE - * @ptlp: location to store the lock for the PTE - * - * On a successful return, the pointer to the PTE is stored in @ptepp; - * the corresponding lock is taken and its location is stored in @ptlp. - * - * The contents of the PTE are only stable until @ptlp is released using - * pte_unmap_unlock(). This function will fail if the PTE is non-present. - * Present PTEs may include PTEs that map refcounted pages, such as - * anonymous folios in COW mappings. - * - * Callers must be careful when relying on PTE content after - * pte_unmap_unlock(). Especially if the PTE maps a refcounted page, - * callers must protect against invalidation with MMU notifiers; otherwise - * access to the PFN at a later point in time can trigger use-after-free. - * - * Only IO mappings and raw PFN mappings are allowed. The mmap semaphore - * should be taken for read. - * - * This function must not be used to modify PTE content. - * - * Return: zero on success, -ve otherwise. - */ -int follow_pte(struct vm_area_struct *vma, unsigned long address, - pte_t **ptepp, spinlock_t **ptlp) -{ - struct mm_struct *mm = vma->vm_mm; - pgd_t *pgd; - p4d_t *p4d; - pud_t *pud; - pmd_t *pmd; - pte_t *ptep; - - mmap_assert_locked(mm); - if (unlikely(address < vma->vm_start || address >= vma->vm_end)) - goto out; - - if (!(vma->vm_flags & (VM_IO | VM_PFNMAP))) - goto out; - - pgd = pgd_offset(mm, address); - if (pgd_none(*pgd) || unlikely(pgd_bad(*pgd))) - goto out; - - p4d = p4d_offset(pgd, address); - if (p4d_none(*p4d) || unlikely(p4d_bad(*p4d))) - goto out; - - pud = pud_offset(p4d, address); - if (pud_none(*pud) || unlikely(pud_bad(*pud))) - goto out; - - pmd = pmd_offset(pud, address); - VM_BUG_ON(pmd_trans_huge(*pmd)); - - ptep = pte_offset_map_lock(mm, pmd, address, ptlp); - if (!ptep) - goto out; - if (!pte_present(ptep_get(ptep))) - goto unlock; - *ptepp = ptep; - return 0; -unlock: - pte_unmap_unlock(ptep, *ptlp); -out: - return -EINVAL; -} -EXPORT_SYMBOL_GPL(follow_pte); - static inline void pfnmap_args_setup(struct follow_pfnmap_args *args, spinlock_t *lock, pte_t *ptep, pgprot_t pgprot, unsigned long pfn_base, From patchwork Mon Aug 26 20:43:51 2024 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Peter Xu X-Patchwork-Id: 13778431 Received: from us-smtp-delivery-124.mimecast.com (us-smtp-delivery-124.mimecast.com [170.10.133.124]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (No client certificate requested) by smtp.subspace.kernel.org (Postfix) with ESMTPS id B248E1A01DB for ; Mon, 26 Aug 2024 20:44:34 +0000 (UTC) Authentication-Results: smtp.subspace.kernel.org; arc=none smtp.client-ip=170.10.133.124 ARC-Seal: i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1724705076; cv=none; b=TbiQw1lYOjJfnKRbZpQCU2wp20qjIw6KHXvGVCZrMfptluMf+bTOXjvmeCaV4ufKE2JJ5DeAQ/F9mxXyIaLCSDXExYHxntSTHEDvgFq6U7F6ARRTTxTpQZU/GER8bny9mj3EIytm4MQY8B4PVkGRSeNrLoBG74ZyAXHwFmN4+rg= ARC-Message-Signature: i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1724705076; c=relaxed/simple; bh=wO6hxJPqfj9KTLTi6qw+v+hbYJLMemjraAGsqutmV4c=; h=From:To:Cc:Subject:Date:Message-ID:In-Reply-To:References: MIME-Version; b=naHwl/kO+MMEBPD67qa2JqeIa/OshpMZJM0sbxkMo1YBFbx6qh+BqXcjbb8Sasj0Iy30kT+bTRTTqDGCJYyMtX2G02Ao6sRAAzBfnafTDuzLXfTwnyo8K3ZEhmM/8Jja5cTSrYOMyxA8FdNUscpFK+ZjIMYMhm1yqN2LUUQDhB8= ARC-Authentication-Results: i=1; smtp.subspace.kernel.org; dmarc=pass (p=none dis=none) header.from=redhat.com; spf=pass smtp.mailfrom=redhat.com; dkim=pass (1024-bit key) header.d=redhat.com header.i=@redhat.com header.b=Xnryqcbg; arc=none smtp.client-ip=170.10.133.124 Authentication-Results: smtp.subspace.kernel.org; dmarc=pass (p=none dis=none) header.from=redhat.com Authentication-Results: smtp.subspace.kernel.org; spf=pass smtp.mailfrom=redhat.com Authentication-Results: smtp.subspace.kernel.org; dkim=pass (1024-bit key) header.d=redhat.com header.i=@redhat.com header.b="Xnryqcbg" DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=redhat.com; s=mimecast20190719; t=1724705073; h=from:from:reply-to:subject:subject:date:date:message-id:message-id: to:to:cc:cc:mime-version:mime-version: content-transfer-encoding:content-transfer-encoding: in-reply-to:in-reply-to:references:references; bh=IC5gYR+7Z4BwAPnZp8TN8XZEGypCYGRNkok3NV40z5s=; b=XnryqcbgleFQu7wsb3ArLf6EE3uUSWK//GXhMsmI9jW89K3rTSNw8kygWmA0SGqz0athRd P0i+HBcJyIOLsfUVtZBF9Mdevx+gcNdBbkaevSQ3/zDJYfmURFjtntrnfHi53v0EQHdB73 oIVlDZAi35dyCKAnpTAe3PGb1GScXck= Received: from mail-oo1-f70.google.com (mail-oo1-f70.google.com [209.85.161.70]) by relay.mimecast.com with ESMTP with STARTTLS (version=TLSv1.3, cipher=TLS_AES_256_GCM_SHA384) id us-mta-306-lrBThTgZOiuBEcvW-3fRfg-1; Mon, 26 Aug 2024 16:44:32 -0400 X-MC-Unique: lrBThTgZOiuBEcvW-3fRfg-1 Received: by mail-oo1-f70.google.com with SMTP id 006d021491bc7-5daa93afe1bso5814885eaf.0 for ; Mon, 26 Aug 2024 13:44:32 -0700 (PDT) X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20230601; t=1724705070; x=1725309870; h=content-transfer-encoding:mime-version:references:in-reply-to :message-id:date:subject:cc:to:from:x-gm-message-state:from:to:cc :subject:date:message-id:reply-to; bh=IC5gYR+7Z4BwAPnZp8TN8XZEGypCYGRNkok3NV40z5s=; b=uQjlsE56F7hkcCWBaNE1xWo5RdEcCsG1VkkQqW6JEUkie2es1gAvL1JRKuHuaelB2k PFkiTGKKo28AZgYI2SWGaAdyV30Img/OgZH9pnt4drGOvBOziU7R62TUNU5DhMroYAsV piC2NElakSXPwbE9+BwEJ0B2Y/xAGpDXS7eDIPW8bNfnCkm83ULuO6LeaewssjpUrgY7 yzonpSZdWmPHJgiR8+PAyojIHBiNrMfnQuNd7s2eIjeWa1eJkflxhcqwtem/7sArk+/i c3OSZfMRDUQRwX6t12KfyxbvYGCUQg1GBYaJZCx5mz/49Dm4p8isdxsO41/QkPqwMEC5 mxmA== X-Forwarded-Encrypted: i=1; AJvYcCV4gDPJMfgLT7lX9mrwpFduNzoxp9kLZv/P2CLA2JwN1ugSqncWG+7Lo2oQyPvkCVhDAlI=@vger.kernel.org X-Gm-Message-State: AOJu0YyrFnHadzQzSugNdg8LCo/Ft4tCIcqKrBF9TFsDeRRzQIP7runE hAgY12Hs+KbClh4t5wyDpMmKNpmyBEFyCq6jx3cxmfVXgGCRsLej1tPJT8QGJ1yuPtDFgpPG3xI Eury6IvBE8R0vx4dJF1q0eguz+yBFy8PIEDAjFftxXOsRILfI1w== X-Received: by 2002:a05:6358:70c3:b0:1ac:f00d:c8c6 with SMTP id e5c5f4694b2df-1b5c22ebd07mr1493541955d.27.1724705070547; Mon, 26 Aug 2024 13:44:30 -0700 (PDT) X-Google-Smtp-Source: AGHT+IHWRaHJzCfl4jAxAoYVG6HBrfG1+v3SSmaZHfZSlba6wG4sR68STMcFscyvu7mDZcazVEGWKw== X-Received: by 2002:a05:6358:70c3:b0:1ac:f00d:c8c6 with SMTP id e5c5f4694b2df-1b5c22ebd07mr1493538355d.27.1724705070146; Mon, 26 Aug 2024 13:44:30 -0700 (PDT) Received: from x1n.redhat.com (pool-99-254-121-117.cpe.net.cable.rogers.com. [99.254.121.117]) by smtp.gmail.com with ESMTPSA id af79cd13be357-7a67f3fd6c1sm491055185a.121.2024.08.26.13.44.28 (version=TLS1_3 cipher=TLS_AES_256_GCM_SHA384 bits=256/256); Mon, 26 Aug 2024 13:44:29 -0700 (PDT) From: Peter Xu To: linux-kernel@vger.kernel.org, linux-mm@kvack.org Cc: Gavin Shan , Catalin Marinas , x86@kernel.org, Ingo Molnar , Andrew Morton , Paolo Bonzini , Dave Hansen , Thomas Gleixner , Alistair Popple , kvm@vger.kernel.org, linux-arm-kernel@lists.infradead.org, Sean Christopherson , peterx@redhat.com, Oscar Salvador , Jason Gunthorpe , Borislav Petkov , Zi Yan , Axel Rasmussen , David Hildenbrand , Yan Zhao , Will Deacon , Kefeng Wang , Alex Williamson Subject: [PATCH v2 17/19] mm/x86: Support large pfn mappings Date: Mon, 26 Aug 2024 16:43:51 -0400 Message-ID: <20240826204353.2228736-18-peterx@redhat.com> X-Mailer: git-send-email 2.45.0 In-Reply-To: <20240826204353.2228736-1-peterx@redhat.com> References: <20240826204353.2228736-1-peterx@redhat.com> Precedence: bulk X-Mailing-List: kvm@vger.kernel.org List-Id: List-Subscribe: List-Unsubscribe: MIME-Version: 1.0 Helpers to install and detect special pmd/pud entries. In short, bit 9 on x86 is not used for pmd/pud, so we can directly define them the same as the pte level. One note is that it's also used in _PAGE_BIT_CPA_TEST but that is only used in the debug test, and shouldn't conflict in this case. One note is that pxx_set|clear_flags() for pmd/pud will need to be moved upper so that they can be referenced by the new special bit helpers. There's no change in the code that was moved. Cc: x86@kernel.org Cc: Thomas Gleixner Cc: Ingo Molnar Cc: Borislav Petkov Cc: Dave Hansen Signed-off-by: Peter Xu --- arch/x86/Kconfig | 1 + arch/x86/include/asm/pgtable.h | 80 ++++++++++++++++++++++------------ 2 files changed, 53 insertions(+), 28 deletions(-) diff --git a/arch/x86/Kconfig b/arch/x86/Kconfig index b74b9ee484da..d4dbe9717e96 100644 --- a/arch/x86/Kconfig +++ b/arch/x86/Kconfig @@ -28,6 +28,7 @@ config X86_64 select ARCH_HAS_GIGANTIC_PAGE select ARCH_SUPPORTS_INT128 if CC_HAS_INT128 select ARCH_SUPPORTS_PER_VMA_LOCK + select ARCH_SUPPORTS_HUGE_PFNMAP if TRANSPARENT_HUGEPAGE select HAVE_ARCH_SOFT_DIRTY select MODULES_USE_ELF_RELA select NEED_DMA_MAP_STATE diff --git a/arch/x86/include/asm/pgtable.h b/arch/x86/include/asm/pgtable.h index 8d12bfad6a1d..4c2d080d26b4 100644 --- a/arch/x86/include/asm/pgtable.h +++ b/arch/x86/include/asm/pgtable.h @@ -120,6 +120,34 @@ extern pmdval_t early_pmd_flags; #define arch_end_context_switch(prev) do {} while(0) #endif /* CONFIG_PARAVIRT_XXL */ +static inline pmd_t pmd_set_flags(pmd_t pmd, pmdval_t set) +{ + pmdval_t v = native_pmd_val(pmd); + + return native_make_pmd(v | set); +} + +static inline pmd_t pmd_clear_flags(pmd_t pmd, pmdval_t clear) +{ + pmdval_t v = native_pmd_val(pmd); + + return native_make_pmd(v & ~clear); +} + +static inline pud_t pud_set_flags(pud_t pud, pudval_t set) +{ + pudval_t v = native_pud_val(pud); + + return native_make_pud(v | set); +} + +static inline pud_t pud_clear_flags(pud_t pud, pudval_t clear) +{ + pudval_t v = native_pud_val(pud); + + return native_make_pud(v & ~clear); +} + /* * The following only work if pte_present() is true. * Undefined behaviour if not.. @@ -317,6 +345,30 @@ static inline int pud_devmap(pud_t pud) } #endif +#ifdef CONFIG_ARCH_SUPPORTS_PMD_PFNMAP +static inline bool pmd_special(pmd_t pmd) +{ + return pmd_flags(pmd) & _PAGE_SPECIAL; +} + +static inline pmd_t pmd_mkspecial(pmd_t pmd) +{ + return pmd_set_flags(pmd, _PAGE_SPECIAL); +} +#endif /* CONFIG_ARCH_SUPPORTS_PMD_PFNMAP */ + +#ifdef CONFIG_ARCH_SUPPORTS_PUD_PFNMAP +static inline bool pud_special(pud_t pud) +{ + return pud_flags(pud) & _PAGE_SPECIAL; +} + +static inline pud_t pud_mkspecial(pud_t pud) +{ + return pud_set_flags(pud, _PAGE_SPECIAL); +} +#endif /* CONFIG_ARCH_SUPPORTS_PUD_PFNMAP */ + static inline int pgd_devmap(pgd_t pgd) { return 0; @@ -487,20 +539,6 @@ static inline pte_t pte_mkdevmap(pte_t pte) return pte_set_flags(pte, _PAGE_SPECIAL|_PAGE_DEVMAP); } -static inline pmd_t pmd_set_flags(pmd_t pmd, pmdval_t set) -{ - pmdval_t v = native_pmd_val(pmd); - - return native_make_pmd(v | set); -} - -static inline pmd_t pmd_clear_flags(pmd_t pmd, pmdval_t clear) -{ - pmdval_t v = native_pmd_val(pmd); - - return native_make_pmd(v & ~clear); -} - /* See comments above mksaveddirty_shift() */ static inline pmd_t pmd_mksaveddirty(pmd_t pmd) { @@ -595,20 +633,6 @@ static inline pmd_t pmd_mkwrite_novma(pmd_t pmd) pmd_t pmd_mkwrite(pmd_t pmd, struct vm_area_struct *vma); #define pmd_mkwrite pmd_mkwrite -static inline pud_t pud_set_flags(pud_t pud, pudval_t set) -{ - pudval_t v = native_pud_val(pud); - - return native_make_pud(v | set); -} - -static inline pud_t pud_clear_flags(pud_t pud, pudval_t clear) -{ - pudval_t v = native_pud_val(pud); - - return native_make_pud(v & ~clear); -} - /* See comments above mksaveddirty_shift() */ static inline pud_t pud_mksaveddirty(pud_t pud) { From patchwork Mon Aug 26 20:43:52 2024 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Peter Xu X-Patchwork-Id: 13778432 Received: from us-smtp-delivery-124.mimecast.com (us-smtp-delivery-124.mimecast.com [170.10.133.124]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (No client certificate requested) by smtp.subspace.kernel.org (Postfix) with ESMTPS id 671C81A0B05 for ; Mon, 26 Aug 2024 20:44:37 +0000 (UTC) Authentication-Results: smtp.subspace.kernel.org; arc=none smtp.client-ip=170.10.133.124 ARC-Seal: i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1724705078; cv=none; b=TNJ1jZ38ZxNfxnNec317Y49o/1KmqMJ1265ywTJNJhRuQm9JZjRZGAXek05o/sgRdgS7098qBfUUBHCVWioAnabBpxpKK42ABV8vmok22akaDnw+sMALhBtOWmVQ+2Fu7uPs3qplfrlJ16CfWcMzLTTB8B8JrZrHfIH//Y+lBo0= ARC-Message-Signature: i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1724705078; c=relaxed/simple; bh=5/RkLDpH7n+0lRZdC6d7Fr9a2r7GbmT3sDiSd48lk+g=; h=From:To:Cc:Subject:Date:Message-ID:In-Reply-To:References: MIME-Version; b=EauaLnJvIn5nr+ThMaaoYB5WWFQWFcBeSzC+2Xd6UgGOjQTorG0msBtKMWqMgUYL5jmWMWKeP46GtMkwKsUzx0ZVKqVHFO5rMdKng5RzLedYeJL+m9fziOODVZ3oYOH5phIkPSImC9+iztNC0pZNyxrdm7fl3o7WppQ8wk/d5+g= ARC-Authentication-Results: i=1; smtp.subspace.kernel.org; dmarc=pass (p=none dis=none) header.from=redhat.com; spf=pass smtp.mailfrom=redhat.com; dkim=pass (1024-bit key) header.d=redhat.com header.i=@redhat.com header.b=GCF8AjKH; arc=none smtp.client-ip=170.10.133.124 Authentication-Results: smtp.subspace.kernel.org; dmarc=pass (p=none dis=none) header.from=redhat.com Authentication-Results: smtp.subspace.kernel.org; spf=pass smtp.mailfrom=redhat.com Authentication-Results: smtp.subspace.kernel.org; dkim=pass (1024-bit key) header.d=redhat.com header.i=@redhat.com header.b="GCF8AjKH" DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=redhat.com; s=mimecast20190719; t=1724705076; h=from:from:reply-to:subject:subject:date:date:message-id:message-id: to:to:cc:cc:mime-version:mime-version: content-transfer-encoding:content-transfer-encoding: in-reply-to:in-reply-to:references:references; bh=FEAM9DK74WZYySVYhL7YnC2YOY0xQ42haB8uzH2x2BY=; b=GCF8AjKHkwg96mBkzB2sBebZgDfnSkfAC0Za1lGOGjnaARDiYvn7h31mMK4R43Te9VqYmQ Uc30PlxK0U5SpwL9R4mfZ/JwKMxa4scuSYHxRyDNSGbSpQv+CYMbapSu2r9wC9bx1XUOJc g2gy2flvaF1+ohE+gwB/A0Dx0QH2dec= Received: from mail-qk1-f198.google.com (mail-qk1-f198.google.com [209.85.222.198]) by relay.mimecast.com with ESMTP with STARTTLS (version=TLSv1.3, cipher=TLS_AES_256_GCM_SHA384) id us-mta-126-f4fLNMtDP7efYHp5vxZkAg-1; Mon, 26 Aug 2024 16:44:33 -0400 X-MC-Unique: f4fLNMtDP7efYHp5vxZkAg-1 Received: by mail-qk1-f198.google.com with SMTP id af79cd13be357-7a1e9a3825aso606412085a.2 for ; Mon, 26 Aug 2024 13:44:33 -0700 (PDT) X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20230601; t=1724705072; x=1725309872; h=content-transfer-encoding:mime-version:references:in-reply-to :message-id:date:subject:cc:to:from:x-gm-message-state:from:to:cc :subject:date:message-id:reply-to; bh=FEAM9DK74WZYySVYhL7YnC2YOY0xQ42haB8uzH2x2BY=; b=D9gi/WTJRtmaaG6itass9Cb0eiThguXzdubRNbjFMQ/LSTPpYGflmtMutMKs6LM2TY SbcrqXhmAiYOELVaaI5Atho4AaWdDa/NI21zPqJxn753/ufDqcyYc2ZqZXUxGZlfrOD/ WzfVhoVI/jd1L5O5q7YyBmnZF+vWfHot12FIF7M0eU2BF5QYuQHRV31zXp15JX9Aikwa +xjBEKX6+CMuXFbCEkZMk+XmEmcWfpPlW9U4MUUXpU0xHmGPpaVuKu93uXjF0MAO7uQl AaJDTrmZZq8rKCvdLznjIe4HyqwtZpKlILvi9BQ/48HVTRRiX/4s3E1wLtK6vKb92Ofc eopA== X-Forwarded-Encrypted: i=1; AJvYcCU2wMGfmeH23+qEA1vq2yCcW1xGI9YGrQrGx95yq+JMPQPi5voNaJuw9lKcAKJnZ2WLv3s=@vger.kernel.org X-Gm-Message-State: AOJu0YyrAsmfRoZUgUscjrjxdXdIMNbd4vJKxcD+Q/AoG39KoVqWwA8G lyNulllykyPdAPue9PijdNhMlNIDNAmbjMxjajmkiopzDvsXzQRFxQmTNry48l+HHGiX1pwqEK/ +a1bNIW/D7/ruceViRSArJks61uS3xPqg9zAQxjfi4FcRWtQ0JQ== X-Received: by 2002:a05:620a:430a:b0:7a4:faab:fc79 with SMTP id af79cd13be357-7a6896d1835mr1392500685a.8.1724705072553; Mon, 26 Aug 2024 13:44:32 -0700 (PDT) X-Google-Smtp-Source: AGHT+IHDiJ56urcvTKu8a4/ieO25ZyreTkMfmYYALFQqTS8OY3wwzzWFQolk9khXQC+xv0hRJc4KwA== X-Received: by 2002:a05:620a:430a:b0:7a4:faab:fc79 with SMTP id af79cd13be357-7a6896d1835mr1392497485a.8.1724705072141; Mon, 26 Aug 2024 13:44:32 -0700 (PDT) Received: from x1n.redhat.com (pool-99-254-121-117.cpe.net.cable.rogers.com. [99.254.121.117]) by smtp.gmail.com with ESMTPSA id af79cd13be357-7a67f3fd6c1sm491055185a.121.2024.08.26.13.44.30 (version=TLS1_3 cipher=TLS_AES_256_GCM_SHA384 bits=256/256); Mon, 26 Aug 2024 13:44:31 -0700 (PDT) From: Peter Xu To: linux-kernel@vger.kernel.org, linux-mm@kvack.org Cc: Gavin Shan , Catalin Marinas , x86@kernel.org, Ingo Molnar , Andrew Morton , Paolo Bonzini , Dave Hansen , Thomas Gleixner , Alistair Popple , kvm@vger.kernel.org, linux-arm-kernel@lists.infradead.org, Sean Christopherson , peterx@redhat.com, Oscar Salvador , Jason Gunthorpe , Borislav Petkov , Zi Yan , Axel Rasmussen , David Hildenbrand , Yan Zhao , Will Deacon , Kefeng Wang , Alex Williamson Subject: [PATCH v2 18/19] mm/arm64: Support large pfn mappings Date: Mon, 26 Aug 2024 16:43:52 -0400 Message-ID: <20240826204353.2228736-19-peterx@redhat.com> X-Mailer: git-send-email 2.45.0 In-Reply-To: <20240826204353.2228736-1-peterx@redhat.com> References: <20240826204353.2228736-1-peterx@redhat.com> Precedence: bulk X-Mailing-List: kvm@vger.kernel.org List-Id: List-Subscribe: List-Unsubscribe: MIME-Version: 1.0 Support huge pfnmaps by using bit 56 (PTE_SPECIAL) for "special" on pmds/puds. Provide the pmd/pud helpers to set/get special bit. There's one more thing missing for arm64 which is the pxx_pgprot() for pmd/pud. Add them too, which is mostly the same as the pte version by dropping the pfn field. These helpers are essential to be used in the new follow_pfnmap*() API to report valid pgprot_t results. Note that arm64 doesn't yet support huge PUD yet, but it's still straightforward to provide the pud helpers that we need altogether. Only PMD helpers will make an immediate benefit until arm64 will support huge PUDs first in general (e.g. in THPs). Cc: linux-arm-kernel@lists.infradead.org Cc: Catalin Marinas Cc: Will Deacon Signed-off-by: Peter Xu --- arch/arm64/Kconfig | 1 + arch/arm64/include/asm/pgtable.h | 29 +++++++++++++++++++++++++++++ 2 files changed, 30 insertions(+) diff --git a/arch/arm64/Kconfig b/arch/arm64/Kconfig index 6494848019a0..6607ed8fdbb4 100644 --- a/arch/arm64/Kconfig +++ b/arch/arm64/Kconfig @@ -99,6 +99,7 @@ config ARM64 select ARCH_SUPPORTS_NUMA_BALANCING select ARCH_SUPPORTS_PAGE_TABLE_CHECK select ARCH_SUPPORTS_PER_VMA_LOCK + select ARCH_SUPPORTS_HUGE_PFNMAP if TRANSPARENT_HUGEPAGE select ARCH_WANT_BATCHED_UNMAP_TLB_FLUSH select ARCH_WANT_COMPAT_IPC_PARSE_VERSION if COMPAT select ARCH_WANT_DEFAULT_BPF_JIT diff --git a/arch/arm64/include/asm/pgtable.h b/arch/arm64/include/asm/pgtable.h index b78cc4a6758b..2faecc033a19 100644 --- a/arch/arm64/include/asm/pgtable.h +++ b/arch/arm64/include/asm/pgtable.h @@ -578,6 +578,14 @@ static inline pmd_t pmd_mkdevmap(pmd_t pmd) return pte_pmd(set_pte_bit(pmd_pte(pmd), __pgprot(PTE_DEVMAP))); } +#ifdef CONFIG_ARCH_SUPPORTS_PMD_PFNMAP +#define pmd_special(pte) (!!((pmd_val(pte) & PTE_SPECIAL))) +static inline pmd_t pmd_mkspecial(pmd_t pmd) +{ + return set_pmd_bit(pmd, __pgprot(PTE_SPECIAL)); +} +#endif + #define __pmd_to_phys(pmd) __pte_to_phys(pmd_pte(pmd)) #define __phys_to_pmd_val(phys) __phys_to_pte_val(phys) #define pmd_pfn(pmd) ((__pmd_to_phys(pmd) & PMD_MASK) >> PAGE_SHIFT) @@ -595,6 +603,27 @@ static inline pmd_t pmd_mkdevmap(pmd_t pmd) #define pud_pfn(pud) ((__pud_to_phys(pud) & PUD_MASK) >> PAGE_SHIFT) #define pfn_pud(pfn,prot) __pud(__phys_to_pud_val((phys_addr_t)(pfn) << PAGE_SHIFT) | pgprot_val(prot)) +#ifdef CONFIG_ARCH_SUPPORTS_PUD_PFNMAP +#define pud_special(pte) pte_special(pud_pte(pud)) +#define pud_mkspecial(pte) pte_pud(pte_mkspecial(pud_pte(pud))) +#endif + +#define pmd_pgprot pmd_pgprot +static inline pgprot_t pmd_pgprot(pmd_t pmd) +{ + unsigned long pfn = pmd_pfn(pmd); + + return __pgprot(pmd_val(pfn_pmd(pfn, __pgprot(0))) ^ pmd_val(pmd)); +} + +#define pud_pgprot pud_pgprot +static inline pgprot_t pud_pgprot(pud_t pud) +{ + unsigned long pfn = pud_pfn(pud); + + return __pgprot(pud_val(pfn_pud(pfn, __pgprot(0))) ^ pud_val(pud)); +} + static inline void __set_pte_at(struct mm_struct *mm, unsigned long __always_unused addr, pte_t *ptep, pte_t pte, unsigned int nr) From patchwork Mon Aug 26 20:43:53 2024 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Peter Xu X-Patchwork-Id: 13778433 Received: from us-smtp-delivery-124.mimecast.com (us-smtp-delivery-124.mimecast.com [170.10.133.124]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (No client certificate requested) by smtp.subspace.kernel.org (Postfix) with ESMTPS id 498A31A0B16 for ; Mon, 26 Aug 2024 20:44:37 +0000 (UTC) Authentication-Results: smtp.subspace.kernel.org; arc=none smtp.client-ip=170.10.133.124 ARC-Seal: i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1724705079; cv=none; b=W5zAbWsvkIn62ZdbxbOhP7Y93b90HThhMyKoBesSp5ZYMdxxXx+Sd/smDtpfGd//UqhOUVkoPGtWWusCw+oPLi7DshZTYHqA9J/lDqKf/3Qt7UsswsqYYkq31k/+UAaU1tucgxlmZu8ncbVGhF1Zzu9XTL7AGj/oKsp4kk5zF/4= ARC-Message-Signature: i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1724705079; c=relaxed/simple; bh=Dmoew7i8XEW+gHBA0zJyGyEzte9fLwrtktcen/8enVY=; h=From:To:Cc:Subject:Date:Message-ID:In-Reply-To:References: MIME-Version; b=siQWQxqAAidqgmIoQVZj4PlgMjqPqD5ILXB+SRDBw/8DebIzBPGNuftx6MbjnaB/vlvjtSrQdN1c4v9tUFnq7m8srwuz7W1idHRP1mKKnj7hs5G4UFg3pGpW+iCRVGknG8RYpjWxyHFDAkxyfwk64Z5LgJv1GeGGjD5TTL62X7o= ARC-Authentication-Results: i=1; smtp.subspace.kernel.org; dmarc=pass (p=none dis=none) header.from=redhat.com; spf=pass smtp.mailfrom=redhat.com; dkim=pass (1024-bit key) header.d=redhat.com header.i=@redhat.com header.b=FhySGljM; arc=none smtp.client-ip=170.10.133.124 Authentication-Results: smtp.subspace.kernel.org; dmarc=pass (p=none dis=none) header.from=redhat.com Authentication-Results: smtp.subspace.kernel.org; spf=pass smtp.mailfrom=redhat.com Authentication-Results: smtp.subspace.kernel.org; dkim=pass (1024-bit key) header.d=redhat.com header.i=@redhat.com header.b="FhySGljM" DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=redhat.com; s=mimecast20190719; t=1724705077; h=from:from:reply-to:subject:subject:date:date:message-id:message-id: to:to:cc:cc:mime-version:mime-version: content-transfer-encoding:content-transfer-encoding: in-reply-to:in-reply-to:references:references; bh=9MPOki3YF/wyEgBnejzItxwjIF1moGifGifGb5OwzY4=; b=FhySGljMbpHAB9+fOXckwB3PHKcnCG4nxvJpUlFUCVLdYAj/nbkeYKgNMqPIEpnfvUoNb9 dH7JrygPjxonljj4/QaAFCfZbclT/wN7GDnyG3JBM6s9TgUfmHPPoX4LUJSoOs0TrhEIBg 9R71iuO6fqzROWWEB1IGt67NCtMuvuM= Received: from mail-qk1-f197.google.com (mail-qk1-f197.google.com [209.85.222.197]) by relay.mimecast.com with ESMTP with STARTTLS (version=TLSv1.3, cipher=TLS_AES_256_GCM_SHA384) id us-mta-214-E_U0CEgeM8WhKhO0OFpY-w-1; Mon, 26 Aug 2024 16:44:35 -0400 X-MC-Unique: E_U0CEgeM8WhKhO0OFpY-w-1 Received: by mail-qk1-f197.google.com with SMTP id af79cd13be357-7a1dc1e5662so663447185a.2 for ; Mon, 26 Aug 2024 13:44:35 -0700 (PDT) X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20230601; t=1724705075; x=1725309875; h=content-transfer-encoding:mime-version:references:in-reply-to :message-id:date:subject:cc:to:from:x-gm-message-state:from:to:cc :subject:date:message-id:reply-to; bh=9MPOki3YF/wyEgBnejzItxwjIF1moGifGifGb5OwzY4=; b=GGxrv98hA0zVxL6T7a56iwxcydkJ+HQSSjLaUuJBKY+lnhLbWasJ+fjtu/RESeXqmL azAxG4YK5H3/VIH47O1ZmrqFgsPm1o75tff3PCkeY0BxfW7BNtXUSNyCat6NUFDg9GIv 6lGT/9AxzfV6ub4eD/M4VfZRyqhNd6hbPMC+yC11cXFk8B84p6K/pypp2uqECxgZg0Xp M9GUiCNMfqvF4vWoJfaFg1HqTGwwJXhOOu44JaN4GxNlyb3zlFlk0S2tEVrVS1Oh3YF2 Y4VpgHnnIn2NwMwwJ2ji7E4lkZeifP/6//GgrEraRopqX4YUc2eGQ1GRVrxkozw/e/0/ 6ZHQ== X-Forwarded-Encrypted: i=1; AJvYcCVo25Aq14vqQuLOV5j72nH3vq6aIIlmm0nSMva8glPfkm/BW06ypKaGjTILv+WRevw6wu4=@vger.kernel.org X-Gm-Message-State: AOJu0YzpO2GbQLx+XrFdybGqHzdz3xbxGX9ZWW++TQl9T+rSUP5V6Qm3 i8H0IkaXk8dp+FuBXmTUKjx7tZy/BCFk+Lz4lhSV/GZKYOhfdh6qMf3feEKXKiXMEFF0nJVsLFW NqkaiAwhaSt6AWknCA9V6ueJ4dwbKKRQSt1RU6pdOsDobzL6jLQ== X-Received: by 2002:a05:620a:4007:b0:79f:595:f64a with SMTP id af79cd13be357-7a6897b7b14mr1477183485a.58.1724705074689; Mon, 26 Aug 2024 13:44:34 -0700 (PDT) X-Google-Smtp-Source: AGHT+IHpx2qE8XXNwFA/VbcshgEEhM7utf/nDzzwdfyOyUpm1p8Kz8GxiojqL+PSarI5vXlPZn99hQ== X-Received: by 2002:a05:620a:4007:b0:79f:595:f64a with SMTP id af79cd13be357-7a6897b7b14mr1477180385a.58.1724705074328; Mon, 26 Aug 2024 13:44:34 -0700 (PDT) Received: from x1n.redhat.com (pool-99-254-121-117.cpe.net.cable.rogers.com. [99.254.121.117]) by smtp.gmail.com with ESMTPSA id af79cd13be357-7a67f3fd6c1sm491055185a.121.2024.08.26.13.44.32 (version=TLS1_3 cipher=TLS_AES_256_GCM_SHA384 bits=256/256); Mon, 26 Aug 2024 13:44:33 -0700 (PDT) From: Peter Xu To: linux-kernel@vger.kernel.org, linux-mm@kvack.org Cc: Gavin Shan , Catalin Marinas , x86@kernel.org, Ingo Molnar , Andrew Morton , Paolo Bonzini , Dave Hansen , Thomas Gleixner , Alistair Popple , kvm@vger.kernel.org, linux-arm-kernel@lists.infradead.org, Sean Christopherson , peterx@redhat.com, Oscar Salvador , Jason Gunthorpe , Borislav Petkov , Zi Yan , Axel Rasmussen , David Hildenbrand , Yan Zhao , Will Deacon , Kefeng Wang , Alex Williamson Subject: [PATCH v2 19/19] vfio/pci: Implement huge_fault support Date: Mon, 26 Aug 2024 16:43:53 -0400 Message-ID: <20240826204353.2228736-20-peterx@redhat.com> X-Mailer: git-send-email 2.45.0 In-Reply-To: <20240826204353.2228736-1-peterx@redhat.com> References: <20240826204353.2228736-1-peterx@redhat.com> Precedence: bulk X-Mailing-List: kvm@vger.kernel.org List-Id: List-Subscribe: List-Unsubscribe: MIME-Version: 1.0 From: Alex Williamson With the addition of pfnmap support in vmf_insert_pfn_{pmd,pud}() we can take advantage of PMD and PUD faults to PCI BAR mmaps and create more efficient mappings. PCI BARs are always a power of two and will typically get at least PMD alignment without userspace even trying. Userspace alignment for PUD mappings is also not too difficult. Consolidate faults through a single handler with a new wrapper for standard single page faults. The pre-faulting behavior of commit d71a989cf5d9 ("vfio/pci: Insert full vma on mmap'd MMIO fault") is removed in this refactoring since huge_fault will cover the bulk of the faults and results in more efficient page table usage. We also want to avoid that pre-faulted single page mappings preempt huge page mappings. Cc: kvm@vger.kernel.org Signed-off-by: Alex Williamson Signed-off-by: Peter Xu --- drivers/vfio/pci/vfio_pci_core.c | 60 +++++++++++++++++++++++--------- 1 file changed, 43 insertions(+), 17 deletions(-) diff --git a/drivers/vfio/pci/vfio_pci_core.c b/drivers/vfio/pci/vfio_pci_core.c index ba0ce0075b2f..2d7478e9a62d 100644 --- a/drivers/vfio/pci/vfio_pci_core.c +++ b/drivers/vfio/pci/vfio_pci_core.c @@ -20,6 +20,7 @@ #include #include #include +#include #include #include #include @@ -1657,14 +1658,20 @@ static unsigned long vma_to_pfn(struct vm_area_struct *vma) return (pci_resource_start(vdev->pdev, index) >> PAGE_SHIFT) + pgoff; } -static vm_fault_t vfio_pci_mmap_fault(struct vm_fault *vmf) +static vm_fault_t vfio_pci_mmap_huge_fault(struct vm_fault *vmf, + unsigned int order) { struct vm_area_struct *vma = vmf->vma; struct vfio_pci_core_device *vdev = vma->vm_private_data; unsigned long pfn, pgoff = vmf->pgoff - vma->vm_pgoff; - unsigned long addr = vma->vm_start; vm_fault_t ret = VM_FAULT_SIGBUS; + if (order && (vmf->address & ((PAGE_SIZE << order) - 1) || + vmf->address + (PAGE_SIZE << order) > vma->vm_end)) { + ret = VM_FAULT_FALLBACK; + goto out; + } + pfn = vma_to_pfn(vma); down_read(&vdev->memory_lock); @@ -1672,30 +1679,49 @@ static vm_fault_t vfio_pci_mmap_fault(struct vm_fault *vmf) if (vdev->pm_runtime_engaged || !__vfio_pci_memory_enabled(vdev)) goto out_unlock; - ret = vmf_insert_pfn(vma, vmf->address, pfn + pgoff); - if (ret & VM_FAULT_ERROR) - goto out_unlock; - - /* - * Pre-fault the remainder of the vma, abort further insertions and - * supress error if fault is encountered during pre-fault. - */ - for (; addr < vma->vm_end; addr += PAGE_SIZE, pfn++) { - if (addr == vmf->address) - continue; - - if (vmf_insert_pfn(vma, addr, pfn) & VM_FAULT_ERROR) - break; + switch (order) { + case 0: + ret = vmf_insert_pfn(vma, vmf->address, pfn + pgoff); + break; +#ifdef CONFIG_ARCH_SUPPORTS_PMD_PFNMAP + case PMD_ORDER: + ret = vmf_insert_pfn_pmd(vmf, __pfn_to_pfn_t(pfn + pgoff, + PFN_DEV), false); + break; +#endif +#ifdef CONFIG_ARCH_SUPPORTS_PUD_PFNMAP + case PUD_ORDER: + ret = vmf_insert_pfn_pud(vmf, __pfn_to_pfn_t(pfn + pgoff, + PFN_DEV), false); + break; +#endif + default: + ret = VM_FAULT_FALLBACK; } out_unlock: up_read(&vdev->memory_lock); +out: + dev_dbg_ratelimited(&vdev->pdev->dev, + "%s(,order = %d) BAR %ld page offset 0x%lx: 0x%x\n", + __func__, order, + vma->vm_pgoff >> + (VFIO_PCI_OFFSET_SHIFT - PAGE_SHIFT), + pgoff, (unsigned int)ret); return ret; } +static vm_fault_t vfio_pci_mmap_page_fault(struct vm_fault *vmf) +{ + return vfio_pci_mmap_huge_fault(vmf, 0); +} + static const struct vm_operations_struct vfio_pci_mmap_ops = { - .fault = vfio_pci_mmap_fault, + .fault = vfio_pci_mmap_page_fault, +#ifdef CONFIG_ARCH_SUPPORTS_HUGE_PFNMAP + .huge_fault = vfio_pci_mmap_huge_fault, +#endif }; int vfio_pci_core_mmap(struct vfio_device *core_vdev, struct vm_area_struct *vma)