From patchwork Fri Aug 9 16:08:51 2024 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Peter Xu X-Patchwork-Id: 13758917 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from kanga.kvack.org (kanga.kvack.org [205.233.56.17]) by smtp.lore.kernel.org (Postfix) with ESMTP id AFFA2C52D7C for ; Fri, 9 Aug 2024 16:09:23 +0000 (UTC) Received: by kanga.kvack.org (Postfix) id 40EBF6B009A; Fri, 9 Aug 2024 12:09:23 -0400 (EDT) Received: by kanga.kvack.org (Postfix, from userid 40) id 3C0556B009E; Fri, 9 Aug 2024 12:09:23 -0400 (EDT) X-Delivered-To: int-list-linux-mm@kvack.org Received: by kanga.kvack.org (Postfix, from userid 63042) id 212A26B009F; Fri, 9 Aug 2024 12:09:23 -0400 (EDT) X-Delivered-To: linux-mm@kvack.org Received: from relay.hostedemail.com (smtprelay0013.hostedemail.com [216.40.44.13]) by kanga.kvack.org (Postfix) with ESMTP id EC5486B009A for ; Fri, 9 Aug 2024 12:09:22 -0400 (EDT) Received: from smtpin08.hostedemail.com (a10.router.float.18 [10.200.18.1]) by unirelay05.hostedemail.com (Postfix) with ESMTP id 8DE48412C2 for ; Fri, 9 Aug 2024 16:09:22 +0000 (UTC) X-FDA: 82433191764.08.B3EEB76 Received: from us-smtp-delivery-124.mimecast.com (us-smtp-delivery-124.mimecast.com [170.10.129.124]) by imf26.hostedemail.com (Postfix) with ESMTP id 6B174140014 for ; Fri, 9 Aug 2024 16:09:19 +0000 (UTC) Authentication-Results: imf26.hostedemail.com; dkim=pass header.d=redhat.com header.s=mimecast20190719 header.b=WQmcrt4G; spf=pass (imf26.hostedemail.com: domain of peterx@redhat.com designates 170.10.129.124 as permitted sender) smtp.mailfrom=peterx@redhat.com; dmarc=pass (policy=none) header.from=redhat.com ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=hostedemail.com; s=arc-20220608; t=1723219693; h=from:from:sender:reply-to:subject:subject:date:date: message-id:message-id:to:to:cc:cc:mime-version:mime-version: content-type:content-type: content-transfer-encoding:content-transfer-encoding: in-reply-to:in-reply-to:references:references:dkim-signature; bh=fKim6v00HbGyszP90Y55Ixe8BGOuNlK0Kt5OSqhmIXk=; b=TtOGcpKGE9HB8sUpuFs+ANKtrTDYtvl5TapTVuNBFUCHTesdGsqVVxtXilo0/mI39BfyRI 7Rn7MaWolg/kAwcXKF1QN5eOCN4hbNS23N9yK+beTnJaIXopiXMVtGgY1FhVgTg1HF5f5k XWmd1YIoQLD7mM2RbmtTHGGPRSIBQSo= ARC-Seal: i=1; s=arc-20220608; d=hostedemail.com; t=1723219693; a=rsa-sha256; cv=none; b=tNS7AOTJnD+OEzziMGuADxeXV5iGCq26dfQRyrqWGrzW4zn6c0sq84OkkzegJFPuzrsrk2 HQAhYw2YegEvTiSi6o4jGYf3T5RbtBCd7L+lzlT6FFY2xgUgwqXqnFXx2RSy2NmCcI/t9f TIWdQqqSGVJ7mVBGa5MryEqK1iyYRKc= ARC-Authentication-Results: i=1; imf26.hostedemail.com; dkim=pass header.d=redhat.com header.s=mimecast20190719 header.b=WQmcrt4G; spf=pass (imf26.hostedemail.com: domain of peterx@redhat.com designates 170.10.129.124 as permitted sender) smtp.mailfrom=peterx@redhat.com; dmarc=pass (policy=none) header.from=redhat.com DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=redhat.com; s=mimecast20190719; t=1723219758; h=from:from:reply-to:subject:subject:date:date:message-id:message-id: to:to:cc:cc:mime-version:mime-version:content-type:content-type: content-transfer-encoding:content-transfer-encoding: in-reply-to:in-reply-to:references:references; bh=fKim6v00HbGyszP90Y55Ixe8BGOuNlK0Kt5OSqhmIXk=; b=WQmcrt4GzDvpomcCmmo5ToSnVO+INctBuXCOUQ3auzuvhxMYhACHwmP6D2GBDboupjq6HE YuMgs/UHQaUL4xWQ7A+X9qwagzP3gN93+XlfwJxYc29WzvqODbkXjw5/QmLXAhM1NSAqds lu+LyExRnkakYClNc5vUpHYUh4HJCb8= Received: from mail-qk1-f197.google.com (mail-qk1-f197.google.com [209.85.222.197]) by relay.mimecast.com with ESMTP with STARTTLS (version=TLSv1.3, cipher=TLS_AES_256_GCM_SHA384) id us-mta-617-Qmd3C24qOpiinUJEEOUMOw-1; Fri, 09 Aug 2024 12:09:17 -0400 X-MC-Unique: Qmd3C24qOpiinUJEEOUMOw-1 Received: by mail-qk1-f197.google.com with SMTP id af79cd13be357-7a1d3c02fcfso7764685a.0 for ; Fri, 09 Aug 2024 09:09:17 -0700 (PDT) X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20230601; t=1723219757; x=1723824557; h=content-transfer-encoding:mime-version:references:in-reply-to :message-id:date:subject:cc:to:from:x-gm-message-state:from:to:cc :subject:date:message-id:reply-to; bh=fKim6v00HbGyszP90Y55Ixe8BGOuNlK0Kt5OSqhmIXk=; b=SkcrRan5dDVYKCHi8QyKdkYx3o1uVbEFEfWZfuL1nvKbZ1M51RNL/WvUG9xTQ1ecQr OEUH3LKGQO/7u23QOWDvGXwh4lHcM4Pa6/dHLFFokax4hG7pf7dsfckQDKswp2DUpDOQ RXiz3umSwmhemYqg8qmMw8vQNGPx+pLLqfOjUOxX5adNMBXvwsgDkxqz1+d85RKSCHvY pKdOrbipHC0ctxUwgAZ0vEHFBfRVTj69REpLKOFOOysE4ouc4DaCvKqb5DHldInfozF2 j1npLyDb7S44D0CdmZK4BJ4zO2Ksa9oXGfKM8aZFazHuAnoJWIHUXyuq9OPXUZmDInY0 Miwg== X-Gm-Message-State: AOJu0Yy/4z10/JN5xaU1/o2D2Zhoqq+bF3L1tjmBJceVtHoXykpmi8uy lrJKG04wmn7q+iWA81QpAyr8uzepySTfTR0mCoYdxpBy0i9z1mBl0uAN67An9iCa9KE4LXYqG0c 0/yCKh0ecnfxs6+UxCE8Id9XILPbxqmbYYbyzs+mxzIFlIyd2eSDI1oG9ShMXzzvd4RFxVfyWli hZaMm+PWou+NU8AkcUw2LjosLmcUhof8R+ X-Received: by 2002:ac8:58c4:0:b0:450:1e5e:3f3f with SMTP id d75a77b69052e-4531255bb38mr13611261cf.5.1723219756764; Fri, 09 Aug 2024 09:09:16 -0700 (PDT) X-Google-Smtp-Source: AGHT+IEAN0G3ozf6o5j8NgDrSR8g+6fB/Zkz8yhBLk8I4VWUl++aRhUduuiwR699bOA3QLxmHpo6kg== X-Received: by 2002:ac8:58c4:0:b0:450:1e5e:3f3f with SMTP id d75a77b69052e-4531255bb38mr13610781cf.5.1723219756198; Fri, 09 Aug 2024 09:09:16 -0700 (PDT) Received: from x1n.redhat.com (pool-99-254-121-117.cpe.net.cable.rogers.com. [99.254.121.117]) by smtp.gmail.com with ESMTPSA id d75a77b69052e-451c870016csm22526741cf.19.2024.08.09.09.09.13 (version=TLS1_3 cipher=TLS_AES_256_GCM_SHA384 bits=256/256); Fri, 09 Aug 2024 09:09:15 -0700 (PDT) From: Peter Xu To: linux-mm@kvack.org, linux-kernel@vger.kernel.org Cc: Sean Christopherson , Oscar Salvador , Jason Gunthorpe , Axel Rasmussen , linux-arm-kernel@lists.infradead.org, x86@kernel.org, peterx@redhat.com, Will Deacon , Gavin Shan , Paolo Bonzini , Zi Yan , Andrew Morton , Catalin Marinas , Ingo Molnar , Alistair Popple , Borislav Petkov , David Hildenbrand , Thomas Gleixner , kvm@vger.kernel.org, Dave Hansen , Alex Williamson , Yan Zhao Subject: [PATCH 01/19] mm: Introduce ARCH_SUPPORTS_HUGE_PFNMAP and special bits to pmd/pud Date: Fri, 9 Aug 2024 12:08:51 -0400 Message-ID: <20240809160909.1023470-2-peterx@redhat.com> X-Mailer: git-send-email 2.45.0 In-Reply-To: <20240809160909.1023470-1-peterx@redhat.com> References: <20240809160909.1023470-1-peterx@redhat.com> MIME-Version: 1.0 X-Mimecast-Spam-Score: 0 X-Mimecast-Originator: redhat.com X-Rspam-User: X-Rspamd-Server: rspam04 X-Rspamd-Queue-Id: 6B174140014 X-Stat-Signature: ren19cqf9h4min68wno6ggt31expgwse X-HE-Tag: 1723219759-765670 X-HE-Meta: U2FsdGVkX1/k+smkUzu/B4ot38UMYG5bXYF7s1VI0ZwjIyeMa19MONwhwCRLPCfqT63AQXsxyQp2TPuqDz8IDIRrLia6F6ASW3QfrgkR3e4rgxoTOyBWRAF1NPRxU33HRhBunGiu0wjprc2bJY+X3YcIPGvHekzKSGrh2q0Rcq4c/j31MrG50kcJ2ur8tTeTjk39NxBbJpuIijZHNMGz9dkVBw36vIupHxiHkRZHhCVMzNkrTuX1oisvx4SSDqR1co8sCijp2PnA8a1MWNtJXiS585PQ8cpuMiLmoM0bwHmVKshz+uyAcTSWosIK+2cEk6o4N+zLzUbfQoFCf4KZXPY5LC0pJQmezn/WP+InStHWSPyaCYbODW4BWzw8PMuXvoeC3eB+wt0Jx/UEfhc3bjTOFGWmigYL4f/JUFrx3FPJwL0hk7SLqWQ010uj7Vcn91ylzT36DZKgFbvUo0Pqk7PpBFJ+Yn/Z0/RT2FSuTJKYisqhjRebgMEg9Gsgf9ZS/Dv3gzkdIkvo9f7cxG0Ld94E0NebemwEUIK21suZP7/npvpF41uBJAewayWLbTumHUtq1B01mcUxFpeDnZxZSGVPS7OvGvRapkCzkGSJKNpEhkpbttW5BCcpfQhCx2mltNoiXlCJxv7fV3nr25B1lsH20bl0Ojz4yqITfAUI7qdeG0GhG+1YgZYyJIKt75w02tpd3b8ami5QwoxDdf4YodebelZMciTI2H3Cc4bNjPJCm6wJ43fi2w+6Voj2dcP6hv8dT2DmC6b93M4+VBDF8FWW9HgOTQwQvQn6pXJfkZU8E5snRxXngU0Y+jpws+Zw8X3IJr5Uktu6Vx7VO9RVEBeAUD3MRnYXI65dOxlNE7HboeUlGxUkbqanirrQOXfidHinJJxIXHdhAJOP4WP8CId8pFuri/ZNbUbnXW5YNs2BlFzcF9SqMS6eO8hHq+2KrNqDHDELI8MOXD5ckLt We98TiWS C6g/i8EnbFhJC593TzPxd+FUdphkc7woYFnf42nqqz6effrW2b3tyhXZLixDiJ/ZPysr/2PvTYamnNo8kq3vt4nBc9Ah9eUsSGu2K8IGETfN7K/L3l+AUGQCnZ1oofP7NiHXQCA4Urk69ece2sGwbqF1SfbqmngWUqLWr3+Vd331vuaVs+UFYmaXlk6u/vaKTHmFc+wgH8Lb1OiBFxfY1I6sdEJ2GDDtKJzZK/BOjASxJ4Q+KSrC/yY2j0WAoKgoMja8v9idrnrlOetnzms86X6zCC4I7/zfot8TyPTs+JsFVx4Wz7ndus0jdb+wzvb+pdPbhAH8MA0LS1J22z+C45HqG5oMSbswgG8ZSuaOFp1xqK+oY9wNMkk1nGhJpywH/mGi8p1RrJtt2y09JksQQuDPyY9Q3OBSuZwx7RZkEcNUmkQWmmss/b7OP92wM2z8oxGtEJSHDdQWbGM0LNXbNMaxwoarJo4TScX9Vo73K8EmPmAmgP6saJkvT8r/agOY80VVI X-Bogosity: Ham, tests=bogofilter, spamicity=0.000000, version=1.2.4 Sender: owner-linux-mm@kvack.org Precedence: bulk X-Loop: owner-majordomo@kvack.org List-ID: List-Subscribe: List-Unsubscribe: This patch introduces the option to introduce special pte bit into pmd/puds. Archs can start to define pmd_special / pud_special when supported by selecting the new option. Per-arch support will be added later. Before that, create fallbacks for these helpers so that they are always available. Signed-off-by: Peter Xu --- include/linux/mm.h | 24 ++++++++++++++++++++++++ mm/Kconfig | 13 +++++++++++++ 2 files changed, 37 insertions(+) diff --git a/include/linux/mm.h b/include/linux/mm.h index 43b40334e9b2..90ca84200800 100644 --- a/include/linux/mm.h +++ b/include/linux/mm.h @@ -2644,6 +2644,30 @@ static inline pte_t pte_mkspecial(pte_t pte) } #endif +#ifndef CONFIG_ARCH_SUPPORTS_PMD_PFNMAP +static inline bool pmd_special(pmd_t pmd) +{ + return false; +} + +static inline pmd_t pmd_mkspecial(pmd_t pmd) +{ + return pmd; +} +#endif /* CONFIG_ARCH_SUPPORTS_PMD_PFNMAP */ + +#ifndef CONFIG_ARCH_SUPPORTS_PUD_PFNMAP +static inline bool pud_special(pud_t pud) +{ + return false; +} + +static inline pud_t pud_mkspecial(pud_t pud) +{ + return pud; +} +#endif /* CONFIG_ARCH_SUPPORTS_PUD_PFNMAP */ + #ifndef CONFIG_ARCH_HAS_PTE_DEVMAP static inline int pte_devmap(pte_t pte) { diff --git a/mm/Kconfig b/mm/Kconfig index 3936fe4d26d9..3db0eebb53e2 100644 --- a/mm/Kconfig +++ b/mm/Kconfig @@ -881,6 +881,19 @@ endif # TRANSPARENT_HUGEPAGE config PGTABLE_HAS_HUGE_LEAVES def_bool TRANSPARENT_HUGEPAGE || HUGETLB_PAGE +# TODO: Allow to be enabled without THP +config ARCH_SUPPORTS_HUGE_PFNMAP + def_bool n + depends on TRANSPARENT_HUGEPAGE + +config ARCH_SUPPORTS_PMD_PFNMAP + def_bool y + depends on ARCH_SUPPORTS_HUGE_PFNMAP && HAVE_ARCH_TRANSPARENT_HUGEPAGE + +config ARCH_SUPPORTS_PUD_PFNMAP + def_bool y + depends on ARCH_SUPPORTS_HUGE_PFNMAP && HAVE_ARCH_TRANSPARENT_HUGEPAGE_PUD + # # UP and nommu archs use km based percpu allocator # From patchwork Fri Aug 9 16:08:52 2024 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Peter Xu X-Patchwork-Id: 13758918 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from kanga.kvack.org (kanga.kvack.org [205.233.56.17]) by smtp.lore.kernel.org (Postfix) with ESMTP id 30242C52D7C for ; Fri, 9 Aug 2024 16:09:27 +0000 (UTC) Received: by kanga.kvack.org (Postfix) id AEE766B009E; Fri, 9 Aug 2024 12:09:26 -0400 (EDT) Received: by kanga.kvack.org (Postfix, from userid 40) id A9E1F6B00A0; Fri, 9 Aug 2024 12:09:26 -0400 (EDT) X-Delivered-To: int-list-linux-mm@kvack.org Received: by kanga.kvack.org (Postfix, from userid 63042) id 8F12F6B00A1; Fri, 9 Aug 2024 12:09:26 -0400 (EDT) X-Delivered-To: linux-mm@kvack.org Received: from relay.hostedemail.com (smtprelay0016.hostedemail.com [216.40.44.16]) by kanga.kvack.org (Postfix) with ESMTP id 6D24B6B009E for ; Fri, 9 Aug 2024 12:09:26 -0400 (EDT) Received: from smtpin27.hostedemail.com (a10.router.float.18 [10.200.18.1]) by unirelay01.hostedemail.com (Postfix) with ESMTP id DE96D1C4A95 for ; Fri, 9 Aug 2024 16:09:25 +0000 (UTC) X-FDA: 82433191890.27.44C2D2B Received: from us-smtp-delivery-124.mimecast.com (us-smtp-delivery-124.mimecast.com [170.10.133.124]) by imf29.hostedemail.com (Postfix) with ESMTP id D59F3120030 for ; Fri, 9 Aug 2024 16:09:23 +0000 (UTC) Authentication-Results: imf29.hostedemail.com; dkim=pass header.d=redhat.com header.s=mimecast20190719 header.b=GkUZLCvu; dmarc=pass (policy=none) header.from=redhat.com; spf=pass (imf29.hostedemail.com: domain of peterx@redhat.com designates 170.10.133.124 as permitted sender) smtp.mailfrom=peterx@redhat.com ARC-Seal: i=1; s=arc-20220608; d=hostedemail.com; t=1723219714; a=rsa-sha256; cv=none; b=6Aifdn4bkzuZl2/544iw0RX9zol3QuL94Ah/nmTZ8Po6ovrFTVgw6I+66CcqGgjjvk1iJD 58Yp6IL763qP3cQ1FswVHVlG0kqd/6ExE+Qq9tm9jh6FObsVHTnybasXbvn98MwkZM5JuH cxAyXcf0Av2jGpMWVcJC2ku95PimmG8= ARC-Authentication-Results: i=1; imf29.hostedemail.com; dkim=pass header.d=redhat.com header.s=mimecast20190719 header.b=GkUZLCvu; dmarc=pass (policy=none) header.from=redhat.com; spf=pass (imf29.hostedemail.com: domain of peterx@redhat.com designates 170.10.133.124 as permitted sender) smtp.mailfrom=peterx@redhat.com ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=hostedemail.com; s=arc-20220608; t=1723219714; h=from:from:sender:reply-to:subject:subject:date:date: message-id:message-id:to:to:cc:cc:mime-version:mime-version: content-type:content-type: content-transfer-encoding:content-transfer-encoding: in-reply-to:in-reply-to:references:references:dkim-signature; bh=xVA/urUlESz7em6kHS5QV39/c1UeLmTRXD3YGysY/mU=; b=xCi8n3o3ARILdlkLR8RtjVLV34OCMSV9b+4m+FcQlnc+3vZcczL/hs8D6vY+JR45rk48oo c7g+G2qBhlR+Qa6n9Vghp9HpG53A2prpQnt+/SG0DEBEE7pjpjjJPeIHQDnsJmZk0oOy+T M6yGD3EFFUPYopk4uIHtZTcDCnfAW/w= DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=redhat.com; s=mimecast20190719; t=1723219763; h=from:from:reply-to:subject:subject:date:date:message-id:message-id: to:to:cc:cc:mime-version:mime-version:content-type:content-type: content-transfer-encoding:content-transfer-encoding: in-reply-to:in-reply-to:references:references; bh=xVA/urUlESz7em6kHS5QV39/c1UeLmTRXD3YGysY/mU=; b=GkUZLCvuopddhJkdpuui/li0AeWpjBB0pphMwO3InMZmGgUt+ufb5FgXMAvDTfjLoV4ekK lcOdarg1KHIEKWAwhYu8sveewkIBxreZghQTpnLZbapTllKmv+IiNRArXEPHw/796uXk57 ZQSop8f2WPBihcApK03r3llVWBtVSwc= Received: from mail-qk1-f199.google.com (mail-qk1-f199.google.com [209.85.222.199]) by relay.mimecast.com with ESMTP with STARTTLS (version=TLSv1.3, cipher=TLS_AES_256_GCM_SHA384) id us-mta-518-kQIgGgLJPoiystyRlBt1Yw-1; Fri, 09 Aug 2024 12:09:20 -0400 X-MC-Unique: kQIgGgLJPoiystyRlBt1Yw-1 Received: by mail-qk1-f199.google.com with SMTP id af79cd13be357-7a1d3c02fcfso7766185a.0 for ; Fri, 09 Aug 2024 09:09:20 -0700 (PDT) X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20230601; t=1723219759; x=1723824559; h=content-transfer-encoding:mime-version:references:in-reply-to :message-id:date:subject:cc:to:from:x-gm-message-state:from:to:cc :subject:date:message-id:reply-to; bh=xVA/urUlESz7em6kHS5QV39/c1UeLmTRXD3YGysY/mU=; b=u2LhdcE35HkoGtfla6isTJ/f7F+w9bN9I+7ByRCLKGgDIzmKQBKlTevAuGqiprSNx4 R0wEW6mo96yKRzYLaVwE6G8Kp+b2+pmFefCiqiZBl0UxV3N/5o9WOAZ2SsneRr1W+3RS Cf+rqAUL58FmOl7WFvhYvV9ttIZqRHbug84xzv6fmjfdL6TKhZSsBmj8hIenLoRx4Vu5 +OFlETphfIQZ/hWV48cN//PzBukC7NcxrUhM+5w5HY8shY5t+HnEzagzNYnJ7q7PDWcl BwLj+Y6DgaKILEVTonBzB3KY/dHfOp9J3cW86dyDc/Ng4wxlRvNLynB/6ltVvqPbsRpl 8lxA== X-Gm-Message-State: AOJu0YwKBdrbu6P6npO2lXdPYFGF1M9gKU6MsXvFshmNZtBlYfhlfPXg EdbNNCJ7Mjz5Uu81mju7OyAFq6TlLSThdHDM9Fm1RXqR30tMmS7zmEYAnLTwK32EFZ8pBvZqMuI GWfyQ3gJTPGyTpKhssiMRO1R+GQf6a7ZK+1qkMKpn6pUvlRNh0ZaS0MRtFcB1ereOdHcnu+ywhx Lx5N+nIJEFwThIwQxEE0J5UOJxC8yWS9Xe X-Received: by 2002:a05:620a:2906:b0:7a1:3ff9:1e1 with SMTP id af79cd13be357-7a4c16af689mr138620085a.0.1723219759145; Fri, 09 Aug 2024 09:09:19 -0700 (PDT) X-Google-Smtp-Source: AGHT+IHesXKAH7m1pKPVMriR5+O5cSg1VUsP7Em6MavFjtwcHLSKYNYbUt0UMVXsf+PUo1OYJsvsTg== X-Received: by 2002:a05:620a:2906:b0:7a1:3ff9:1e1 with SMTP id af79cd13be357-7a4c16af689mr138614585a.0.1723219758659; Fri, 09 Aug 2024 09:09:18 -0700 (PDT) Received: from x1n.redhat.com (pool-99-254-121-117.cpe.net.cable.rogers.com. [99.254.121.117]) by smtp.gmail.com with ESMTPSA id d75a77b69052e-451c870016csm22526741cf.19.2024.08.09.09.09.16 (version=TLS1_3 cipher=TLS_AES_256_GCM_SHA384 bits=256/256); Fri, 09 Aug 2024 09:09:18 -0700 (PDT) From: Peter Xu To: linux-mm@kvack.org, linux-kernel@vger.kernel.org Cc: Sean Christopherson , Oscar Salvador , Jason Gunthorpe , Axel Rasmussen , linux-arm-kernel@lists.infradead.org, x86@kernel.org, peterx@redhat.com, Will Deacon , Gavin Shan , Paolo Bonzini , Zi Yan , Andrew Morton , Catalin Marinas , Ingo Molnar , Alistair Popple , Borislav Petkov , David Hildenbrand , Thomas Gleixner , kvm@vger.kernel.org, Dave Hansen , Alex Williamson , Yan Zhao , Matthew Wilcox , "Aneesh Kumar K . V" Subject: [PATCH 02/19] mm: Drop is_huge_zero_pud() Date: Fri, 9 Aug 2024 12:08:52 -0400 Message-ID: <20240809160909.1023470-3-peterx@redhat.com> X-Mailer: git-send-email 2.45.0 In-Reply-To: <20240809160909.1023470-1-peterx@redhat.com> References: <20240809160909.1023470-1-peterx@redhat.com> MIME-Version: 1.0 X-Mimecast-Spam-Score: 0 X-Mimecast-Originator: redhat.com X-Rspamd-Queue-Id: D59F3120030 X-Rspam-User: X-Rspamd-Server: rspam05 X-Stat-Signature: oc8e6brx648k9doqcgiwgsya3sfupx3c X-HE-Tag: 1723219763-477353 X-HE-Meta: U2FsdGVkX19bGGNas2gD6Zm8SI02uNXSNH5WrsjyUd3RF8q6dHtrS/hJWVnj26QyYsAj85mR0+b5ae0PtV7eW6kfEZ2FCn8z/vHO3y2EzeIpeyWmq+SPHnEEALqmFqg8+3Wj+NAykulu8pWlfKqrnrTzFGFtLMmi/Ue2D/wzdXK0HSAfnwQLddf9mBD0sy9UC14U851SqXt3/ksBaFefvqmSEsfONpzzuquqJCm+yUBHXKtOdCefLO0uDqAxNx/yOq0Ff3j4pLtFcAaJV/awQqlckFF+lG1YXe10Auz2DrCKKfdUJod8k+1VR5lhl4bW9akusXOYytsatBpmn4d3qAAlvO5Ar8xllqWw6qOxSBDEHGkIA20X60L/8DEtvs2LY51FLlsFd4bm4x7RMzqdU8jBcdlZeSAKdF3HNKXOIddIoOt1hqEHhH1HlGBAVh8LAd8ytc2UNllKos5BwQKkhLRxzOM6qemw0loETgrgAXMzQPjLSlBfZwcR3WGJJ3uo6glWuUtasMAYJsypeKa6FSuj1PbGeAE10ENFV78TYaNkLjM5wHNFQKaapkfcpofeK0KRx65f1hu1QXHL0Rd6LVYHTVzauWkLQkJRlIdV1VIF75aqwJuDunK5pZJ+qz1YXiNMqo5WSQEX/QDYB4R3/EgdTfVyGQ5ZBiXMkiCTaWHWSmRcKLwdYHpznJBg3jFpbqRDBPLnf2dCPV2hTzu+NLMtpsDMBIAqbSm9v9ejc0H5fhRL6SLluTfrN7McOtTW8LIWoppwO1tryyAqX6WXw1VptXidjEwUhtKg1ZyvWfnyXG4TBXHp6kuPz42z2OveLIY3rkaGC5UjxuVHz+PqEAFwlhDCOOk+qg5etumpmZWb0YPzWP6jvTQVJCKP65MeYGbfkKmry/Y/CyenVHpmqNYpA19JU65GVS9fwocHLP7DfyMcE9cmbccRc4fX36S4uRG7T8UxuFTgB0br+oR UT0tU0MC zTxba3aMfFuJu8SLBKTcLTZgEEuvon1eiUgaEeagWBOo489ySSSFsRxYJdzIcSzIDgHtGZOiNSk3RqjqXMJXI8QLjZ6ahuhMFhxfXdoQpX1f4d5e4oQb6XRddMwTCnFpXwXScCviaqzeKz11ZDDdSE8lKs8JXJ9enQLtGnr/W1m9KBdmbD/dioBfSfedlu/vMYlFRg5c/Njgrt3X7GpfnuhqRW66lpiPrAS9QnAYO2x6dM10WjZfBpXx60iq4wBBhm+qAUbYVKZcmfqfLUn5dX9i36daO4Xw6lEA9WwTFZDtzu3q+dNX2W5EmtTEM00veUp5IkPPu44HMRpJ1pVoOzc9yrcFYdcccei8z/XVyePXLhTb2DOSgtVIVvijZDHl6QON0FPY0KFuhs33OlsFnOY08odQuMsTHBHB+NqCgvPxXAFM8QxdtaoaV5TqjOtvqmvSnpE93wuh9S/Qp40r4jasSU86VRt5HJhUhVb+my96kk+WS96akJ2f2WNvYNSIh6uAt0ah6hjkx/gn1Kup+hziVYvo2Y8ZzHu8E X-Bogosity: Ham, tests=bogofilter, spamicity=0.000000, version=1.2.4 Sender: owner-linux-mm@kvack.org Precedence: bulk X-Loop: owner-majordomo@kvack.org List-ID: List-Subscribe: List-Unsubscribe: It constantly returns false since 2017. One assertion is added in 2019 but it should never have triggered, IOW it means what is checked should be asserted instead. If it didn't exist for 7 years maybe it's good idea to remove it and only add it when it comes. Cc: Matthew Wilcox Cc: Aneesh Kumar K.V Signed-off-by: Peter Xu Acked-by: David Hildenbrand Reviewed-by: Jason Gunthorpe --- include/linux/huge_mm.h | 10 ---------- mm/huge_memory.c | 13 +------------ 2 files changed, 1 insertion(+), 22 deletions(-) diff --git a/include/linux/huge_mm.h b/include/linux/huge_mm.h index 6370026689e0..2121060232ce 100644 --- a/include/linux/huge_mm.h +++ b/include/linux/huge_mm.h @@ -421,11 +421,6 @@ static inline bool is_huge_zero_pmd(pmd_t pmd) return pmd_present(pmd) && READ_ONCE(huge_zero_pfn) == pmd_pfn(pmd); } -static inline bool is_huge_zero_pud(pud_t pud) -{ - return false; -} - struct folio *mm_get_huge_zero_folio(struct mm_struct *mm); void mm_put_huge_zero_folio(struct mm_struct *mm); @@ -566,11 +561,6 @@ static inline bool is_huge_zero_pmd(pmd_t pmd) return false; } -static inline bool is_huge_zero_pud(pud_t pud) -{ - return false; -} - static inline void mm_put_huge_zero_folio(struct mm_struct *mm) { return; diff --git a/mm/huge_memory.c b/mm/huge_memory.c index 0aafd26d7a53..39c401a62e87 100644 --- a/mm/huge_memory.c +++ b/mm/huge_memory.c @@ -1245,10 +1245,8 @@ static void insert_pfn_pud(struct vm_area_struct *vma, unsigned long addr, ptl = pud_lock(mm, pud); if (!pud_none(*pud)) { if (write) { - if (pud_pfn(*pud) != pfn_t_to_pfn(pfn)) { - WARN_ON_ONCE(!is_huge_zero_pud(*pud)); + if (WARN_ON_ONCE(pud_pfn(*pud) != pfn_t_to_pfn(pfn))) goto out_unlock; - } entry = pud_mkyoung(*pud); entry = maybe_pud_mkwrite(pud_mkdirty(entry), vma); if (pudp_set_access_flags(vma, addr, pud, entry, 1)) @@ -1496,15 +1494,6 @@ int copy_huge_pud(struct mm_struct *dst_mm, struct mm_struct *src_mm, if (unlikely(!pud_trans_huge(pud) && !pud_devmap(pud))) goto out_unlock; - /* - * When page table lock is held, the huge zero pud should not be - * under splitting since we don't split the page itself, only pud to - * a page table. - */ - if (is_huge_zero_pud(pud)) { - /* No huge zero pud yet */ - } - /* * TODO: once we support anonymous pages, use * folio_try_dup_anon_rmap_*() and split if duplicating fails. From patchwork Fri Aug 9 16:08:53 2024 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Peter Xu X-Patchwork-Id: 13758919 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from kanga.kvack.org (kanga.kvack.org [205.233.56.17]) by smtp.lore.kernel.org (Postfix) with ESMTP id EC222C3DA4A for ; Fri, 9 Aug 2024 16:09:28 +0000 (UTC) Received: by kanga.kvack.org (Postfix) id 412036B00A0; Fri, 9 Aug 2024 12:09:27 -0400 (EDT) Received: by kanga.kvack.org (Postfix, from userid 40) id 3BF876B00A1; Fri, 9 Aug 2024 12:09:27 -0400 (EDT) X-Delivered-To: int-list-linux-mm@kvack.org Received: by kanga.kvack.org (Postfix, from userid 63042) id 1EC056B00A2; Fri, 9 Aug 2024 12:09:27 -0400 (EDT) X-Delivered-To: linux-mm@kvack.org Received: from relay.hostedemail.com (smtprelay0012.hostedemail.com [216.40.44.12]) by kanga.kvack.org (Postfix) with ESMTP id F25026B00A0 for ; Fri, 9 Aug 2024 12:09:26 -0400 (EDT) Received: from smtpin02.hostedemail.com (a10.router.float.18 [10.200.18.1]) by unirelay05.hostedemail.com (Postfix) with ESMTP id A6A8F412A6 for ; Fri, 9 Aug 2024 16:09:26 +0000 (UTC) X-FDA: 82433191932.02.9635CC4 Received: from us-smtp-delivery-124.mimecast.com (us-smtp-delivery-124.mimecast.com [170.10.133.124]) by imf19.hostedemail.com (Postfix) with ESMTP id 855421A0026 for ; Fri, 9 Aug 2024 16:09:24 +0000 (UTC) Authentication-Results: imf19.hostedemail.com; dkim=pass header.d=redhat.com header.s=mimecast20190719 header.b=AIkLRe78; dmarc=pass (policy=none) header.from=redhat.com; spf=pass (imf19.hostedemail.com: domain of peterx@redhat.com designates 170.10.133.124 as permitted sender) smtp.mailfrom=peterx@redhat.com ARC-Seal: i=1; s=arc-20220608; d=hostedemail.com; t=1723219710; a=rsa-sha256; cv=none; b=xJtOf3lKG51d3vUYilqm3hMTXsRR+Rj6V/Mdf07bjy/NV+0upSS2cp8Ja35JvS4CNDF+FY YM0zjZ2DvUIxLNjzxg8mXDZhngJ06MauO3vCIo0uyxvjzMQ2NmA4CmYCyh0fylyxhUawsJ q54ODlmi5o+XrsnIMz6j1zWikpJGI8w= ARC-Authentication-Results: i=1; imf19.hostedemail.com; dkim=pass header.d=redhat.com header.s=mimecast20190719 header.b=AIkLRe78; dmarc=pass (policy=none) header.from=redhat.com; spf=pass (imf19.hostedemail.com: domain of peterx@redhat.com designates 170.10.133.124 as permitted sender) smtp.mailfrom=peterx@redhat.com ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=hostedemail.com; s=arc-20220608; t=1723219710; h=from:from:sender:reply-to:subject:subject:date:date: message-id:message-id:to:to:cc:cc:mime-version:mime-version: content-type:content-type: content-transfer-encoding:content-transfer-encoding: in-reply-to:in-reply-to:references:references:dkim-signature; bh=pOevS25kl1+p5C9bnayjnL4IWCIfturozNfzs94vCz8=; b=GYSfXKV6XgvKePto0hn8uYOn2riTiAbG2BaC0V/2XrKOBLukf3x+Ecmc1npcXBbk8WdkBT nH/YeLe2eQcoKJ71HUVnovaZT4FHBHxUA3O0l+tfoCspWdHJ8gx0eobnkM6AEwMfsGC4w0 1rDEzQkZhh6+ztwP2F1bCzh6UgPY75E= DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=redhat.com; s=mimecast20190719; t=1723219763; h=from:from:reply-to:subject:subject:date:date:message-id:message-id: to:to:cc:cc:mime-version:mime-version:content-type:content-type: content-transfer-encoding:content-transfer-encoding: in-reply-to:in-reply-to:references:references; bh=pOevS25kl1+p5C9bnayjnL4IWCIfturozNfzs94vCz8=; b=AIkLRe78Zxz80FdBVVF94Ok0iKX6PlppauloyrnCDhoRbT+1guSOkX9X42aP94brCJe12a NoqdEgbelGqZckfmJHl7mqnvKHmIQJjnd3miBCiJfR1Az3sUNGJUYHjyQuZRJrblqsWaBC 2AXDbRkHHI9NIA3Z6gxYJaHz4YaPMDQ= Received: from mail-qv1-f70.google.com (mail-qv1-f70.google.com [209.85.219.70]) by relay.mimecast.com with ESMTP with STARTTLS (version=TLSv1.3, cipher=TLS_AES_256_GCM_SHA384) id us-mta-442-ATh9A-zFM0-5lWbJvGWAIQ-1; Fri, 09 Aug 2024 12:09:22 -0400 X-MC-Unique: ATh9A-zFM0-5lWbJvGWAIQ-1 Received: by mail-qv1-f70.google.com with SMTP id 6a1803df08f44-6b7678caf7dso6678686d6.1 for ; Fri, 09 Aug 2024 09:09:22 -0700 (PDT) X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20230601; t=1723219761; x=1723824561; h=content-transfer-encoding:mime-version:references:in-reply-to :message-id:date:subject:cc:to:from:x-gm-message-state:from:to:cc :subject:date:message-id:reply-to; bh=pOevS25kl1+p5C9bnayjnL4IWCIfturozNfzs94vCz8=; b=bvL9K7lvwYzhOHLVBT6QOBj4F0YSXlX+9O/+vsG4cfJNq1EDTtajILzw+opvS0hxOY 8p4P0VR9RM5OTySEszH3yrjLbLI9/KVEcbbd+8+GjXyO+Slcf5y0ScG7eJuQnleBevx7 +SQlM8uE/hMY3BnYAvYwnpLoR8Z512MS3wRT+cgRtXBrvnaW24KxFnOLHLwFlzlNR45R K96fzbB+myugMWwtswlzDlaPUm2ZQCsiNtY/WljIpuc40kPMHhz+TkHu/hQc/RPFZeaf bEln3cSCe8bbsysDYEBL7PICwXLhxLZXW7zHn/74zZmmr4vtxnqd+ugZAcHz7cIYXZS6 8uCg== X-Gm-Message-State: AOJu0YwhtDUYYya5v2MlrMMPwwJjTGO+WtqEiNmi/80IL9P8acm3V0wv MeVu9rghHwz9+QftZkGWhheU4SbpjTsYom2cXoYj6MNjDvoqn3NlrCWbnj7zArTQzQntJfF3NNF qo5TRBir60vPGsXqJPBsb4hiR0WDnPDOAGK5eWNiQZuXmIV9YHhwxL33oaYlER5CFr6tPLLFSJG Lwh9WW/Iez1CMa+RM3nJTu3fSXINFHlTZv X-Received: by 2002:a05:622a:1a8b:b0:44f:89e3:d119 with SMTP id d75a77b69052e-453126babcemr11885761cf.12.1723219761733; Fri, 09 Aug 2024 09:09:21 -0700 (PDT) X-Google-Smtp-Source: AGHT+IGHUujWoHeQHvUCs2OAJSjyQdbxlWd9FX1yi8fb3RZKwv0O5abV6g6G64GeuNGe2uyjoJD48A== X-Received: by 2002:a05:622a:1a8b:b0:44f:89e3:d119 with SMTP id d75a77b69052e-453126babcemr11885331cf.12.1723219761262; Fri, 09 Aug 2024 09:09:21 -0700 (PDT) Received: from x1n.redhat.com (pool-99-254-121-117.cpe.net.cable.rogers.com. [99.254.121.117]) by smtp.gmail.com with ESMTPSA id d75a77b69052e-451c870016csm22526741cf.19.2024.08.09.09.09.18 (version=TLS1_3 cipher=TLS_AES_256_GCM_SHA384 bits=256/256); Fri, 09 Aug 2024 09:09:20 -0700 (PDT) From: Peter Xu To: linux-mm@kvack.org, linux-kernel@vger.kernel.org Cc: Sean Christopherson , Oscar Salvador , Jason Gunthorpe , Axel Rasmussen , linux-arm-kernel@lists.infradead.org, x86@kernel.org, peterx@redhat.com, Will Deacon , Gavin Shan , Paolo Bonzini , Zi Yan , Andrew Morton , Catalin Marinas , Ingo Molnar , Alistair Popple , Borislav Petkov , David Hildenbrand , Thomas Gleixner , kvm@vger.kernel.org, Dave Hansen , Alex Williamson , Yan Zhao Subject: [PATCH 03/19] mm: Mark special bits for huge pfn mappings when inject Date: Fri, 9 Aug 2024 12:08:53 -0400 Message-ID: <20240809160909.1023470-4-peterx@redhat.com> X-Mailer: git-send-email 2.45.0 In-Reply-To: <20240809160909.1023470-1-peterx@redhat.com> References: <20240809160909.1023470-1-peterx@redhat.com> MIME-Version: 1.0 X-Mimecast-Spam-Score: 0 X-Mimecast-Originator: redhat.com X-Rspamd-Server: rspam12 X-Rspamd-Queue-Id: 855421A0026 X-Stat-Signature: fay3uthgbzk5ni4njriyreznqwzxihim X-Rspam-User: X-HE-Tag: 1723219764-553049 X-HE-Meta: U2FsdGVkX18xAymM046ybIm6d2q9j0dLlMmxGcys9lHDLErnE+o8kOlKRHPbgyYknBodjkkEdXTJP7KNyPUdPfg6jI8uvPJVQRACQMrNVV8UiePyWNFnLsMJIRPQ9x5NM46DEPZGtoYoz7PASqFtS2VRBbUHSPDpxMXHA9dTv3llia9G+VtkeApbtmrt0NbmjMyAfYxPFcgBxB6iK6Ye5e4uO79RNzPLfxLRoh0tdZS9GZxgS/mY/dKu1ciwaAFvzUxGstm5BKV2UQH0T2/8GdswoCeiRS23tIBRvII7Lpsue+KJRgJL89Z4SZayeC2rPQnY9EEJle5togWBx8NZYXtsuoxHYvKXNlICsuyAfH3JIkoBYuMS26VFpv0smCZA9hzgf2NhDvkoCUeSOVCxv22tCQYoaSgUrv5dfuy2iYmsVuOV+Bxf/VDlsAvrYKLrAdhGjoT/Bd48gSwJY6jl9sfmKk3qzuxG6DafrstASALbFYcGkmQH+YaVJKQ31lyj9yem78Y/VYQs37BxiPOTmVahJugOEEB+9AqAld3mYOnKQYWCQc4kjhDYQOfKjW2l34niUWhIU1knvq9ipK73oVDrEY2Cqf6KrdTkhkIOlQZK9uql1B0/O2PpO8pn9BvTvjhJf/daDs5wHjh89VAmfomIi17QU3InwrIrj3nSZzsjC+fI3TCf0dzObFnphvHu9Hp6LW2+DfkkGEeJqqZEtOmq3pW3Zxmbu0OYmIKIxcq+ORAaVfZzoRUz6VtI8X88D+QPZmlZqD0KNUttrIUTsB1hjY1YSpZtrXN91rUrRyGrmHj+oLn25hxNdiWC4Mp5ts6ywWANt3p9mwjMNCRaECBaxsPUcsXryKB7tIkKS6DsxsBBQEk7vGv2d6jFlQsO3kfKXBarpheLV2vEWgsi56UXg99a0h7eYsKflFb59NOFLhz/OVHvkCyEFg/gb1Hqg+92sXyTeMJ7DOTLNRj XIEd4y5r 072BuNrGHF920Op+xazCdD/XbJczyWjly8soJiQQoSI/jJnjuqGUUM5weJVRKZMNdyWMSQy5f+b28BI0x3wokZkfmg/x/d3BPpDZ+YZutkVyIdNZy38n4iPe01Q0allLCKS+JMyPJkrJHIT4cWFDXvTp4iGAUQEHJkoJa/hNaM+kz28sj89fFk61zWfEPKbxO5cd5+fRcu1qrreg6am2MfCdBmKLoKSlsG+9b+qjfHfWRhFyGNiyfjjf2S2SoKeeMw8yHeod/MjDYyN1zGtKe460DHJxnvy0YgaxDDDwMqX7HnV2Xn7wx0BAC4q7Gms/jhTPnaPn7H33ECgbNaFSWoDpfZA== X-Bogosity: Ham, tests=bogofilter, spamicity=0.000000, version=1.2.4 Sender: owner-linux-mm@kvack.org Precedence: bulk X-Loop: owner-majordomo@kvack.org List-ID: List-Subscribe: List-Unsubscribe: We need these special bits to be around to enable gup-fast on pfnmaps. Mark properly for !devmap case, reflecting that there's no page struct backing the entry. Signed-off-by: Peter Xu Reviewed-by: Jason Gunthorpe --- mm/huge_memory.c | 4 ++++ 1 file changed, 4 insertions(+) diff --git a/mm/huge_memory.c b/mm/huge_memory.c index 39c401a62e87..e95b3a468aee 100644 --- a/mm/huge_memory.c +++ b/mm/huge_memory.c @@ -1162,6 +1162,8 @@ static void insert_pfn_pmd(struct vm_area_struct *vma, unsigned long addr, entry = pmd_mkhuge(pfn_t_pmd(pfn, prot)); if (pfn_t_devmap(pfn)) entry = pmd_mkdevmap(entry); + else + entry = pmd_mkspecial(entry); if (write) { entry = pmd_mkyoung(pmd_mkdirty(entry)); entry = maybe_pmd_mkwrite(entry, vma); @@ -1258,6 +1260,8 @@ static void insert_pfn_pud(struct vm_area_struct *vma, unsigned long addr, entry = pud_mkhuge(pfn_t_pud(pfn, prot)); if (pfn_t_devmap(pfn)) entry = pud_mkdevmap(entry); + else + entry = pud_mkspecial(entry); if (write) { entry = pud_mkyoung(pud_mkdirty(entry)); entry = maybe_pud_mkwrite(entry, vma); From patchwork Fri Aug 9 16:08:54 2024 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Peter Xu X-Patchwork-Id: 13758930 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from kanga.kvack.org (kanga.kvack.org [205.233.56.17]) by smtp.lore.kernel.org (Postfix) with ESMTP id 93C96C52D7F for ; Fri, 9 Aug 2024 16:09:31 +0000 (UTC) Received: by kanga.kvack.org (Postfix) id B666B6B00A2; Fri, 9 Aug 2024 12:09:30 -0400 (EDT) Received: by kanga.kvack.org (Postfix, from userid 40) id B134A6B00A3; Fri, 9 Aug 2024 12:09:30 -0400 (EDT) X-Delivered-To: int-list-linux-mm@kvack.org Received: by kanga.kvack.org (Postfix, from userid 63042) id 93E9D6B00A4; Fri, 9 Aug 2024 12:09:30 -0400 (EDT) X-Delivered-To: linux-mm@kvack.org Received: from relay.hostedemail.com (smtprelay0016.hostedemail.com [216.40.44.16]) by kanga.kvack.org (Postfix) with ESMTP id 75C706B00A2 for ; Fri, 9 Aug 2024 12:09:30 -0400 (EDT) Received: from smtpin21.hostedemail.com (a10.router.float.18 [10.200.18.1]) by unirelay02.hostedemail.com (Postfix) with ESMTP id 36C82121117 for ; Fri, 9 Aug 2024 16:09:30 +0000 (UTC) X-FDA: 82433192100.21.9512529 Received: from us-smtp-delivery-124.mimecast.com (us-smtp-delivery-124.mimecast.com [170.10.129.124]) by imf29.hostedemail.com (Postfix) with ESMTP id 3060B12003D for ; Fri, 9 Aug 2024 16:09:27 +0000 (UTC) Authentication-Results: imf29.hostedemail.com; dkim=pass header.d=redhat.com header.s=mimecast20190719 header.b=LpGFTbVt; dmarc=pass (policy=none) header.from=redhat.com; spf=pass (imf29.hostedemail.com: domain of peterx@redhat.com designates 170.10.129.124 as permitted sender) smtp.mailfrom=peterx@redhat.com ARC-Seal: i=1; s=arc-20220608; d=hostedemail.com; t=1723219757; a=rsa-sha256; cv=none; b=6/VMhNbM/WFP+owklYsHcinr+LMA+JebbdsLvi5IOvtyg8INPVrdr3oBkKweaMHSZKQphC yGKqVZq7ss8y4YcFHo8+Q/kLnZZvXU/SD/65/WoRvbepwPMtQc23inOLsdj5Tsc2IAmEyA l8CW0P8+kPDLqEmbevMDPUJkDXDZtwA= ARC-Authentication-Results: i=1; imf29.hostedemail.com; dkim=pass header.d=redhat.com header.s=mimecast20190719 header.b=LpGFTbVt; dmarc=pass (policy=none) header.from=redhat.com; spf=pass (imf29.hostedemail.com: domain of peterx@redhat.com designates 170.10.129.124 as permitted sender) smtp.mailfrom=peterx@redhat.com ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=hostedemail.com; s=arc-20220608; t=1723219757; h=from:from:sender:reply-to:subject:subject:date:date: message-id:message-id:to:to:cc:cc:mime-version:mime-version: content-type:content-type: content-transfer-encoding:content-transfer-encoding: in-reply-to:in-reply-to:references:references:dkim-signature; bh=ip3F/lMuK2on6ujO8FbOHDG+DYlKv6bK2Y5nLIm/Zzk=; b=z4ft6APio9Ex9mhaZbEO2Bdygqkm/Awub4uvlNQ/dYLtbTk8hk71IXn/Cc8fFj7oq1aDNm 6Ysr9Bu8pFIrf5VsDoUMhLvTAPXvHV5GszDhMbkdOqVexCkqCRZDbk9Pg8jG2oP8YO28aP u3HxrTliqgKq5Xc36ma0yXyPYurXzCQ= DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=redhat.com; s=mimecast20190719; t=1723219766; h=from:from:reply-to:subject:subject:date:date:message-id:message-id: to:to:cc:cc:mime-version:mime-version:content-type:content-type: content-transfer-encoding:content-transfer-encoding: in-reply-to:in-reply-to:references:references; bh=ip3F/lMuK2on6ujO8FbOHDG+DYlKv6bK2Y5nLIm/Zzk=; b=LpGFTbVtTohrLvzxpIXLTw/68JaA/e2bvqtH9OdcyDr4M3dL7GL0wehytF2O3k0zkI27Z1 qy43mt1rF+mGCAqaH+810FzBaH6C7OI0f6hRhxo11qmogUg3RIrYODPxQONNBEmkmxN+Xn cUBhgL/Hf6g25S/f4zRD93+En9e0/ZE= Received: from mail-qt1-f197.google.com (mail-qt1-f197.google.com [209.85.160.197]) by relay.mimecast.com with ESMTP with STARTTLS (version=TLSv1.3, cipher=TLS_AES_256_GCM_SHA384) id us-mta-563-ddLL0_QROwaWL8nL1tAlsg-1; Fri, 09 Aug 2024 12:09:25 -0400 X-MC-Unique: ddLL0_QROwaWL8nL1tAlsg-1 Received: by mail-qt1-f197.google.com with SMTP id d75a77b69052e-4531adc3c76so35721cf.2 for ; Fri, 09 Aug 2024 09:09:25 -0700 (PDT) X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20230601; t=1723219764; x=1723824564; h=content-transfer-encoding:mime-version:references:in-reply-to :message-id:date:subject:cc:to:from:x-gm-message-state:from:to:cc :subject:date:message-id:reply-to; bh=ip3F/lMuK2on6ujO8FbOHDG+DYlKv6bK2Y5nLIm/Zzk=; b=f6frIQMMe0dUcAF05B+d2vl2pDtDVvNjyl9ePTp3ZJBhGLWPBWyF1l1MTcWxtxUmEs L2AOYy/9irf7lu+DqtfXXb4Cdgoq8XjwDyQVYhNifXU0IPD9iLA7nocMBK179i2t8y2V qtFwM/sZTkpSLhX0XFzyRVqZnBQij+zt/Ld/oDNb0i74Q64YMelKicFwkq5nz/a6hdCp 61qpZe905bId7VHhEBM4eKJ6SpoPVdzO2bzRSQFLCFk//mhikT6I4XMU3tAylPKJMtQk tHIopkR7TfF3zNbEV8NMVTRZfrmXXnp358R9h7B+p9SfiqYJZwWR6sPJsdtGhEH8glTw 7W7g== X-Gm-Message-State: AOJu0Yy9MI8bDufywam1pxJuXsmCmm5BRzFhnlQjKlP4M8O5aRpTZMfp CvJnxrgl9vZ0BE4YH3jHAQCmfipB4B5/rOdeVKYyo6LmVni0+2ZrxGwUs3on7GFLMldn4edkQ29 RsnCE6O0jne7Wgrh0aseSAMty5S8pW23ckNcss0dOluIKDdAl3MueeRPn/UnvrYvvr8MBryDFtB D2swdCqcphIdlwopKCaypSwqkbqfjKr7gS X-Received: by 2002:a05:622a:1b8e:b0:447:e636:9ea9 with SMTP id d75a77b69052e-4531251c724mr14423361cf.3.1723219764323; Fri, 09 Aug 2024 09:09:24 -0700 (PDT) X-Google-Smtp-Source: AGHT+IHps50LqsfKl8qXevdYdePG8tewmjaKqGlRovYXRf84yciplQ1RIErbjQjT+BAlToj6EpXuWQ== X-Received: by 2002:a05:622a:1b8e:b0:447:e636:9ea9 with SMTP id d75a77b69052e-4531251c724mr14422831cf.3.1723219763777; Fri, 09 Aug 2024 09:09:23 -0700 (PDT) Received: from x1n.redhat.com (pool-99-254-121-117.cpe.net.cable.rogers.com. [99.254.121.117]) by smtp.gmail.com with ESMTPSA id d75a77b69052e-451c870016csm22526741cf.19.2024.08.09.09.09.21 (version=TLS1_3 cipher=TLS_AES_256_GCM_SHA384 bits=256/256); Fri, 09 Aug 2024 09:09:22 -0700 (PDT) From: Peter Xu To: linux-mm@kvack.org, linux-kernel@vger.kernel.org Cc: Sean Christopherson , Oscar Salvador , Jason Gunthorpe , Axel Rasmussen , linux-arm-kernel@lists.infradead.org, x86@kernel.org, peterx@redhat.com, Will Deacon , Gavin Shan , Paolo Bonzini , Zi Yan , Andrew Morton , Catalin Marinas , Ingo Molnar , Alistair Popple , Borislav Petkov , David Hildenbrand , Thomas Gleixner , kvm@vger.kernel.org, Dave Hansen , Alex Williamson , Yan Zhao , Matthew Wilcox , Ryan Roberts Subject: [PATCH 04/19] mm: Allow THP orders for PFNMAPs Date: Fri, 9 Aug 2024 12:08:54 -0400 Message-ID: <20240809160909.1023470-5-peterx@redhat.com> X-Mailer: git-send-email 2.45.0 In-Reply-To: <20240809160909.1023470-1-peterx@redhat.com> References: <20240809160909.1023470-1-peterx@redhat.com> MIME-Version: 1.0 X-Mimecast-Spam-Score: 0 X-Mimecast-Originator: redhat.com X-Rspam-User: X-Rspamd-Queue-Id: 3060B12003D X-Rspamd-Server: rspam01 X-Stat-Signature: miebahp6ih37psum18ndx5e7bwgyren5 X-HE-Tag: 1723219767-472056 X-HE-Meta: U2FsdGVkX19B33PqKEB80BAXLSdgQ22eeqlmKfTxfVa544MjBi6Wffqn+hBGCumFMVYy0hH5suQ1TUaRdzFmZYL6vSAFSk77pStrOeV7cQJqkRg+3Ud7hT9+l4qfXMbf2iCgqV8nenz3rYG50nwenCzXECMoGdFlc0ELilPDbxnWL2EcLG/o6icBHUjGYdZhfjWytxDcKh9PIHLkbr7Wh6GXgdq0qdd2j/GKbBXzUvqQ3lhrYl9WmkAjYuM+MrPAASNWa4ia+QEKym3BEyvBGylMGupeNnC34vSVOafwNE+95bFrtS2q/hI0KZAlx2tzgrw8OwE+uWEwgNQu06Cr7J9iOy/nZ8tcxJMRKeE2pVAy7R9YIW0X2cD9W19cT5VnzW5d9QyivNQD5YwS2FtX/0953C7XTxVhezJ7fFSc7O180OyoZYQBqQOQds2GsjRA3RiNM6ehRdKX5ZKRd/iefJ/kTt2RS87+mN9C2pdxjmdorII2JZgB7lmWtQlT+slBaTZ7PfGF9KMksVf1JV6L27sRI3tBShIYtXEesBcmrSNfZhVMGjzzExHbJ6IvokDArU3ncGKpF4bVy7/KcOYGhmPLb/xWvaesqL7rwwso2Z8Sm970ZZ8nfXPTih9jzMQ/05dRSdiNpF9+OGOMsmjT9QECdstoc3PE9Mkkz3HpAfq2LJzezEsJFreNMhM7vgSBX2gJuCue68Puxdi84/+7lT3gcE9AUE61m7qJgxLZV0yLDmSGwFC7P+JeUW5TpW9RhRzH1pPPS2ldq+/uNDpgfNHrbW+0y9SXCkhLLHWkFiby6+deP6XTVye2rKvbe4BIfN6b88oGVWhu8+W23G6h+6iCA4nlvQCdA7YAYdAyKZY6kxFAqmF8iCWaA5U7gtQZjEoq2+j4tamiCPdKBxezmtkLG14m5hW/WGu1kL+6c5qV5jJxBm5s1BD1gsDgKbV8DQCnjTqkXkPiXONUwmz yyF8iuET 9vcaMiK/dWAwlMb+D8C3q5Pe7f7BmkeeUF2U8svA7QSBnslGiYoAt4Pc3jRFEXmcJLLxnoObCGUynd51rqNHpbhcWvOmRTTyw2EF0mSafNXJMoNkXnrFJFyzMRcOENff+FBRpQnQz2ifppQ7gMbiWHc+UDqyQmAG0swqg61NnumlU2GRlsFE4xpa211aB0qCnmt3v2CAvLkmQTtfifMsweay+fhp0dxHdwmUiUcQsYyXtxL/qlHVjadpEzJN6qPe8ai+4LuLmcOQne9PPc6xcrHCuEcAER0myR5yqq6X9eSMJ3JN2pMupjg448EOq0QLYi8LuojmJQ9JOH4B69Thz+JwQ8c2wsZNyS0LtQYXM+Emwe7PeNT4doCtOgROjTmjjMREShRiet+ho4vkH07julPH/PqlvvxJKvjqGd5f3IqmMwcelZ47gsVT64RI7WjNGI0aoFvOAxc/MGJrq0P5uVnPoLtxqFpfzzAT2egSnXnZ5IEGVz0eCoG4ZNjK0QkAwhTgtVEkzW6NlBr2Qds5Aww1bTfI8rR6MJe3hwn8QjaJAGj/wtzM8k6LzAGv+sCwTcF0I X-Bogosity: Ham, tests=bogofilter, spamicity=0.000000, version=1.2.4 Sender: owner-linux-mm@kvack.org Precedence: bulk X-Loop: owner-majordomo@kvack.org List-ID: List-Subscribe: List-Unsubscribe: This enables PFNMAPs to be mapped at either pmd/pud layers. Generalize the dax case into vma_is_special_huge() so as to cover both. Meanwhile, rename the macro to THP_ORDERS_ALL_SPECIAL. Cc: Matthew Wilcox Cc: Gavin Shan Cc: Ryan Roberts Cc: David Hildenbrand Cc: Zi Yan Signed-off-by: Peter Xu Reviewed-by: Jason Gunthorpe --- include/linux/huge_mm.h | 6 +++--- mm/huge_memory.c | 4 ++-- 2 files changed, 5 insertions(+), 5 deletions(-) diff --git a/include/linux/huge_mm.h b/include/linux/huge_mm.h index 2121060232ce..984cbc960d8b 100644 --- a/include/linux/huge_mm.h +++ b/include/linux/huge_mm.h @@ -76,9 +76,9 @@ extern struct kobj_attribute thpsize_shmem_enabled_attr; /* * Mask of all large folio orders supported for file THP. Folios in a DAX * file is never split and the MAX_PAGECACHE_ORDER limit does not apply to - * it. + * it. Same to PFNMAPs where there's neither page* nor pagecache. */ -#define THP_ORDERS_ALL_FILE_DAX \ +#define THP_ORDERS_ALL_SPECIAL \ (BIT(PMD_ORDER) | BIT(PUD_ORDER)) #define THP_ORDERS_ALL_FILE_DEFAULT \ ((BIT(MAX_PAGECACHE_ORDER + 1) - 1) & ~BIT(0)) @@ -87,7 +87,7 @@ extern struct kobj_attribute thpsize_shmem_enabled_attr; * Mask of all large folio orders supported for THP. */ #define THP_ORDERS_ALL \ - (THP_ORDERS_ALL_ANON | THP_ORDERS_ALL_FILE_DAX | THP_ORDERS_ALL_FILE_DEFAULT) + (THP_ORDERS_ALL_ANON | THP_ORDERS_ALL_SPECIAL | THP_ORDERS_ALL_FILE_DEFAULT) #define TVA_SMAPS (1 << 0) /* Will be used for procfs */ #define TVA_IN_PF (1 << 1) /* Page fault handler */ diff --git a/mm/huge_memory.c b/mm/huge_memory.c index e95b3a468aee..6568586b21ab 100644 --- a/mm/huge_memory.c +++ b/mm/huge_memory.c @@ -95,8 +95,8 @@ unsigned long __thp_vma_allowable_orders(struct vm_area_struct *vma, /* Check the intersection of requested and supported orders. */ if (vma_is_anonymous(vma)) supported_orders = THP_ORDERS_ALL_ANON; - else if (vma_is_dax(vma)) - supported_orders = THP_ORDERS_ALL_FILE_DAX; + else if (vma_is_special_huge(vma)) + supported_orders = THP_ORDERS_ALL_SPECIAL; else supported_orders = THP_ORDERS_ALL_FILE_DEFAULT; From patchwork Fri Aug 9 16:08:55 2024 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Peter Xu X-Patchwork-Id: 13758931 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from kanga.kvack.org (kanga.kvack.org [205.233.56.17]) by smtp.lore.kernel.org (Postfix) with ESMTP id 530BDC52D7C for ; Fri, 9 Aug 2024 16:09:34 +0000 (UTC) Received: by kanga.kvack.org (Postfix) id A57306B00A4; Fri, 9 Aug 2024 12:09:32 -0400 (EDT) Received: by kanga.kvack.org (Postfix, from userid 40) id 9E0F56B00A5; Fri, 9 Aug 2024 12:09:32 -0400 (EDT) X-Delivered-To: int-list-linux-mm@kvack.org Received: by kanga.kvack.org (Postfix, from userid 63042) id 82E4B6B00A6; Fri, 9 Aug 2024 12:09:32 -0400 (EDT) X-Delivered-To: linux-mm@kvack.org Received: from relay.hostedemail.com (smtprelay0015.hostedemail.com [216.40.44.15]) by kanga.kvack.org (Postfix) with ESMTP id 638D16B00A4 for ; Fri, 9 Aug 2024 12:09:32 -0400 (EDT) Received: from smtpin06.hostedemail.com (a10.router.float.18 [10.200.18.1]) by unirelay07.hostedemail.com (Postfix) with ESMTP id D88821611DC for ; Fri, 9 Aug 2024 16:09:31 +0000 (UTC) X-FDA: 82433192142.06.524A70C Received: from us-smtp-delivery-124.mimecast.com (us-smtp-delivery-124.mimecast.com [170.10.133.124]) by imf18.hostedemail.com (Postfix) with ESMTP id C10551C0019 for ; Fri, 9 Aug 2024 16:09:29 +0000 (UTC) Authentication-Results: imf18.hostedemail.com; dkim=pass header.d=redhat.com header.s=mimecast20190719 header.b=gBd4TJUh; dmarc=pass (policy=none) header.from=redhat.com; spf=pass (imf18.hostedemail.com: domain of peterx@redhat.com designates 170.10.133.124 as permitted sender) smtp.mailfrom=peterx@redhat.com ARC-Seal: i=1; s=arc-20220608; d=hostedemail.com; t=1723219760; a=rsa-sha256; cv=none; b=TQrtKJCCwO/wZEUipItbHbig9hDX/qbPEwSL0lzReN9F8QiBD1N3T+PNmxDOTMx2wnO6sr 5ki7izgY4oG1bdu0JHTcOutxm9Z8A4P+MO4dpwSJSPuccquvpMiwpnQX8cG0i3CBrSA5yo dXWSb8S6uPZI+c4uH7ABHKRki4Ofh+8= ARC-Authentication-Results: i=1; imf18.hostedemail.com; dkim=pass header.d=redhat.com header.s=mimecast20190719 header.b=gBd4TJUh; dmarc=pass (policy=none) header.from=redhat.com; spf=pass (imf18.hostedemail.com: domain of peterx@redhat.com designates 170.10.133.124 as permitted sender) smtp.mailfrom=peterx@redhat.com ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=hostedemail.com; s=arc-20220608; t=1723219760; h=from:from:sender:reply-to:subject:subject:date:date: message-id:message-id:to:to:cc:cc:mime-version:mime-version: content-type:content-type: content-transfer-encoding:content-transfer-encoding: in-reply-to:in-reply-to:references:references:dkim-signature; bh=bfMsI243pXvbDIYk3LIHdKnZKRp0h0WHhTX8nwPWLTI=; b=E6lnnL4y01RX3OkhxgQU+If21p/p70dFYSiw8T3ZaQxo1EkdT9fIlYL4v9CpG0XJS974p0 k2RiK0Y1DlgV7EyEumVwc1ozdfrmvbaqNugI6G2zhFfjJT16WjkTHYC1U8E4HOCdlAwrh5 g/knA7t/M/PsGMcJv2w5Oubz1nWiKPI= DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=redhat.com; s=mimecast20190719; t=1723219769; h=from:from:reply-to:subject:subject:date:date:message-id:message-id: to:to:cc:cc:mime-version:mime-version:content-type:content-type: content-transfer-encoding:content-transfer-encoding: in-reply-to:in-reply-to:references:references; bh=bfMsI243pXvbDIYk3LIHdKnZKRp0h0WHhTX8nwPWLTI=; b=gBd4TJUhLvNQDYh1PUDUgel3bAtJU85/efPnUWHoaHH9sH1Agh0YmjeA2xolzfHxVEWxaj jXi8Qent0c1uqpyE4/KJasL8B1HC+HLtctnOFbrSP5O8JGu3QD6iC6fDKjolgG0ZX4XpW9 5PbJfmW8O0jT8Ttx8EEUJCWm3MeZzOs= Received: from mail-qt1-f199.google.com (mail-qt1-f199.google.com [209.85.160.199]) by relay.mimecast.com with ESMTP with STARTTLS (version=TLSv1.3, cipher=TLS_AES_256_GCM_SHA384) id us-mta-360-s7knL5dfMNSYK9X8TR5dnw-1; Fri, 09 Aug 2024 12:09:28 -0400 X-MC-Unique: s7knL5dfMNSYK9X8TR5dnw-1 Received: by mail-qt1-f199.google.com with SMTP id d75a77b69052e-4531adc3c76so35861cf.2 for ; Fri, 09 Aug 2024 09:09:27 -0700 (PDT) X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20230601; t=1723219767; x=1723824567; h=content-transfer-encoding:mime-version:references:in-reply-to :message-id:date:subject:cc:to:from:x-gm-message-state:from:to:cc :subject:date:message-id:reply-to; bh=bfMsI243pXvbDIYk3LIHdKnZKRp0h0WHhTX8nwPWLTI=; b=Pun9UwELgT7qvSz4Ihz2f/6n5fRX6cRkwLtwUT1YFKlBtqNcyzdl4Os4w+PPMuG9E6 yYd6fwkkidQzQ3nYjaBXcpG3NNe0YP1tQt2v3yEqQ7uqomk8y1GFSOULTNfBZR/OT1UP m1Cfpb2Mn/fzfijYgW+Bgi8dlKQ3b5GIInN+UGHUDTOujDAk11ObcYzQ2TAw2d8DRyhV FJ3rOGHaIahMIqixYMOj0P3FEKiK4jO4HV89d/lH6zUEHr5BJQ4HfLA4C482Qk1TaAEo sJMunSy4SxNMsSmpew4Mwxe5e8Z+1vFBjKDxqIKJJeutyjaKK2bYkWiJkWRAx9bljwRp 7mow== X-Gm-Message-State: AOJu0YwmqE6a+cLvNQgPd9s/OAjaYx3ldlAvE040I3ILkZFhsgPdbRqm Ybgf/w8KiukODhtQjSRaMF837ScS55NmQ2m8TyK1B7VlNMIPJeveABJxCM1uQtGilKs9djN/9fN VZDN/IgQY+6R/5yD5bUZy0qjnhq1VJozya30Gowlf1SIHrYKjYxavylsxJCh33hKzQ22rQU46F9 WXTlKtSPOeUpspQ4coc4eEpx1N/DGJ8Scg X-Received: by 2002:ac8:59cc:0:b0:44f:c953:290d with SMTP id d75a77b69052e-4531251d17bmr12886321cf.2.1723219767164; Fri, 09 Aug 2024 09:09:27 -0700 (PDT) X-Google-Smtp-Source: AGHT+IElkbLJ8q6bD88BtBGZOYKdaksYNPrmA3JWwjtOEQ9+GQ5ZX+OpUCG/V8WuVaLDdrJFvk6CQQ== X-Received: by 2002:ac8:59cc:0:b0:44f:c953:290d with SMTP id d75a77b69052e-4531251d17bmr12885751cf.2.1723219766412; Fri, 09 Aug 2024 09:09:26 -0700 (PDT) Received: from x1n.redhat.com (pool-99-254-121-117.cpe.net.cable.rogers.com. [99.254.121.117]) by smtp.gmail.com with ESMTPSA id d75a77b69052e-451c870016csm22526741cf.19.2024.08.09.09.09.24 (version=TLS1_3 cipher=TLS_AES_256_GCM_SHA384 bits=256/256); Fri, 09 Aug 2024 09:09:25 -0700 (PDT) From: Peter Xu To: linux-mm@kvack.org, linux-kernel@vger.kernel.org Cc: Sean Christopherson , Oscar Salvador , Jason Gunthorpe , Axel Rasmussen , linux-arm-kernel@lists.infradead.org, x86@kernel.org, peterx@redhat.com, Will Deacon , Gavin Shan , Paolo Bonzini , Zi Yan , Andrew Morton , Catalin Marinas , Ingo Molnar , Alistair Popple , Borislav Petkov , David Hildenbrand , Thomas Gleixner , kvm@vger.kernel.org, Dave Hansen , Alex Williamson , Yan Zhao Subject: [PATCH 05/19] mm/gup: Detect huge pfnmap entries in gup-fast Date: Fri, 9 Aug 2024 12:08:55 -0400 Message-ID: <20240809160909.1023470-6-peterx@redhat.com> X-Mailer: git-send-email 2.45.0 In-Reply-To: <20240809160909.1023470-1-peterx@redhat.com> References: <20240809160909.1023470-1-peterx@redhat.com> MIME-Version: 1.0 X-Mimecast-Spam-Score: 0 X-Mimecast-Originator: redhat.com X-Rspam-User: X-Rspamd-Queue-Id: C10551C0019 X-Rspamd-Server: rspam01 X-Stat-Signature: ey9yyd37m1p8noir37yboj7xd64gqn4h X-HE-Tag: 1723219769-844956 X-HE-Meta: U2FsdGVkX18G2WipcPfthMCx8CWl9Cu1vCG6x+zOC56shVuumUdIDSEij5WQBWhCiP9UGQzidw7klcSIQrYswsuDXgbRHhsltfxNdetSKQmTs6Mu5MpT/nYo4zXcAKCMek3qf81Ta3unjn+jPLu8cAOEPbMcY/nyQtcdDhMkL0aZt01qVd3cllsCgaJ2ExKx7fFLj7xUjIM/jcNsQBkNTzcQql7Czz9fbpjNA03yTw3Xg31k89Vhm8Vz6I6CxInKOrRM/VI+9h1mEh0QfGz3i72LQYYFR3oHrotXYZpqCmdfzt1aHZxCXjnPXyclriia+5x1wV7fBTp6/jSHGhdRWRqc526KRtsEm+oYldxowcWB6pHhCwM3LByq2e2Z/5hZ4cZIBQK3FzoN9QNi8Ef1FbUbfIuBCw/yLXvw7J9ozWy9f9FBYnaqoYmJhcZlGfyXcrRqu53Mm5OgjNIHQL8DY17O7JLxtNPLNHXrRJVdpc9yLW0aQcnn2GUnY811eRKKpNysT6PP+XaeRfMHyro6qjHBBpWAsYi5u9L8B4mXvFe71deODeUh1E4Ty8IqIBQOaAd/JtPfskC9IzaegNrjJvUQJCPEQCDOV0uhs8MNMhBS2AzBB98V75ChCJyFgi+MUMJP4Fj+rOTETgx+AddWzZtjZa65H9aW/KU8CuEkouYkkxtGgBLB26031Cif0EAXVGoojNK3pK1u5VOKa7gsXPGsxuK+qIaYVyz7pEUAwNdWtf30gzYHHB/HRAej/V/w9Q2R9P2oOE1n+aerULdZVTGVlP+8TDBYplWZvhvvlBxMxnAHw6A9/zvFOuJ2COuasRGPYvJfF+r9QAnTY2T0532fq7/yUTyYXi9foB00OdWVxO4q+x0od9yU0pk/ild7PsDIW4ay6ayGbRYSAdyiZ5vYyWSXBBjUK8jFbu9wTdjUSs3/oG1IcDQWOxGCJKmEbH8nD8J9EMvc2oXncl/ I/AzD+3u gO+odi30zN+26Ysr0S0Z4zbYyLd0N0aYUNwjlRy8IhC+FIlFCl5IKWY3uGt0GQ0QX44yzMJje3FzlvLKiLiEAJsFJTCbfiVMg0Eu6Lfkx3C/wBED8EweMmIWzFxaFVRoVyXoKwYpkIUAs8lD5+bi0gXetG5J3/4F67cnuhjV9re4T7jSU+18flJ2RrMpV9vu1xAilK4k7kl2iAYx+0cxAfuVgc8Rzkm2kw0MQnH9fnZcXA5t8JMy/yM3TsUTWvQHfahDQn/QtUB5lEsECooTYDp8Zpn2Inef/xR1AGYeZfNFbvYfFx1JzJsqtO15d3JL+apxQ4wWV2lgtSVgYDwNULqry1EO2O22erbeLFS9G6tOKSKPN/XLWWWIbiV7NC/KB0zVbx1K7UPXaGtR7NbeCGTjLHmikLKI233EFomn9ovPNjcslyrGQyPVJBIpZICVB/y6JuzvBbBvO3jE= X-Bogosity: Ham, tests=bogofilter, spamicity=0.000002, version=1.2.4 Sender: owner-linux-mm@kvack.org Precedence: bulk X-Loop: owner-majordomo@kvack.org List-ID: List-Subscribe: List-Unsubscribe: Since gup-fast doesn't have the vma reference, teach it to detect such huge pfnmaps by checking the special bit for pmd/pud too, just like ptes. Signed-off-by: Peter Xu Acked-by: David Hildenbrand Reviewed-by: Jason Gunthorpe --- mm/gup.c | 6 ++++++ 1 file changed, 6 insertions(+) diff --git a/mm/gup.c b/mm/gup.c index d19884e097fd..a49f67a512ee 100644 --- a/mm/gup.c +++ b/mm/gup.c @@ -3038,6 +3038,9 @@ static int gup_fast_pmd_leaf(pmd_t orig, pmd_t *pmdp, unsigned long addr, if (!pmd_access_permitted(orig, flags & FOLL_WRITE)) return 0; + if (pmd_special(orig)) + return 0; + if (pmd_devmap(orig)) { if (unlikely(flags & FOLL_LONGTERM)) return 0; @@ -3082,6 +3085,9 @@ static int gup_fast_pud_leaf(pud_t orig, pud_t *pudp, unsigned long addr, if (!pud_access_permitted(orig, flags & FOLL_WRITE)) return 0; + if (pud_special(orig)) + return 0; + if (pud_devmap(orig)) { if (unlikely(flags & FOLL_LONGTERM)) return 0; From patchwork Fri Aug 9 16:08:57 2024 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Peter Xu X-Patchwork-Id: 13758932 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from kanga.kvack.org (kanga.kvack.org [205.233.56.17]) by smtp.lore.kernel.org (Postfix) with ESMTP id 2F48EC52D7C for ; Fri, 9 Aug 2024 16:09:37 +0000 (UTC) Received: by kanga.kvack.org (Postfix) id 7B9C66B00A6; Fri, 9 Aug 2024 12:09:36 -0400 (EDT) Received: by kanga.kvack.org (Postfix, from userid 40) id 741C96B00A7; Fri, 9 Aug 2024 12:09:36 -0400 (EDT) X-Delivered-To: int-list-linux-mm@kvack.org Received: by kanga.kvack.org (Postfix, from userid 63042) id 56EA66B00A8; Fri, 9 Aug 2024 12:09:36 -0400 (EDT) X-Delivered-To: linux-mm@kvack.org Received: from relay.hostedemail.com (smtprelay0014.hostedemail.com [216.40.44.14]) by kanga.kvack.org (Postfix) with ESMTP id 2F7406B00A6 for ; Fri, 9 Aug 2024 12:09:36 -0400 (EDT) Received: from smtpin28.hostedemail.com (a10.router.float.18 [10.200.18.1]) by unirelay04.hostedemail.com (Postfix) with ESMTP id EE6831A11CB for ; Fri, 9 Aug 2024 16:09:35 +0000 (UTC) X-FDA: 82433192310.28.567775F Received: from us-smtp-delivery-124.mimecast.com (us-smtp-delivery-124.mimecast.com [170.10.133.124]) by imf02.hostedemail.com (Postfix) with ESMTP id DBED38001F for ; Fri, 9 Aug 2024 16:09:33 +0000 (UTC) Authentication-Results: imf02.hostedemail.com; dkim=pass header.d=redhat.com header.s=mimecast20190719 header.b=NEl6YNjs; dmarc=pass (policy=none) header.from=redhat.com; spf=pass (imf02.hostedemail.com: domain of peterx@redhat.com designates 170.10.133.124 as permitted sender) smtp.mailfrom=peterx@redhat.com ARC-Seal: i=1; s=arc-20220608; d=hostedemail.com; t=1723219720; a=rsa-sha256; cv=none; b=MlvLZP1eKXRONrkWH6qhGzO/sw77R6c8MJiQxd4X4KzU+CJ82e6y1pCnGO9QW/DGk8Z9H2 X5Ix4ZYo+AoZ8cu/1lUCIO+D6iGmO/6ntwMNr9VlD1iscr8KbLl5NsCeFCNc+iP5WABLc6 DYp8PtvBJCW9gpijIv5eSCugJK07eFw= ARC-Authentication-Results: i=1; imf02.hostedemail.com; dkim=pass header.d=redhat.com header.s=mimecast20190719 header.b=NEl6YNjs; dmarc=pass (policy=none) header.from=redhat.com; spf=pass (imf02.hostedemail.com: domain of peterx@redhat.com designates 170.10.133.124 as permitted sender) smtp.mailfrom=peterx@redhat.com ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=hostedemail.com; s=arc-20220608; t=1723219720; h=from:from:sender:reply-to:subject:subject:date:date: message-id:message-id:to:to:cc:cc:mime-version:mime-version: content-type:content-type: content-transfer-encoding:content-transfer-encoding: in-reply-to:in-reply-to:references:references:dkim-signature; bh=GAUMBfy0kdSy8bea6Q2/adQQTYYkrWuiUWfctkd/rW8=; b=kcyQNY7fGtTu6fZDXtUwIYdnzQCiUYLQSlD/8RCZMKI2y9j1i8uZsbqzi9qlrOZjl2ZFjg P1TBTGIy0UbIyzWKyvEf99nmlR2Fl96+m+VR//ZhA8/7BOoXM8w1UsDgf6pMDnjNea3bUn /tRGtP02BukLzSxnhDPM2H5gcdf030Q= DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=redhat.com; s=mimecast20190719; t=1723219773; h=from:from:reply-to:subject:subject:date:date:message-id:message-id: to:to:cc:cc:mime-version:mime-version:content-type:content-type: content-transfer-encoding:content-transfer-encoding: in-reply-to:in-reply-to:references:references; bh=GAUMBfy0kdSy8bea6Q2/adQQTYYkrWuiUWfctkd/rW8=; b=NEl6YNjsZGDTrFfDymRE51xbhcfb4tlH5mhszKGFLWFgzdOKhrswl/Epk85m7TRsOSJFep PwmaPzlOFT5RT9ZfjpHaGHLvUNMBoswV7n4s0Nx/Rw7G16siLE/aSmJ6dmHYrLwtDI7GcG Jbetw5bya204NFUbNMI7eHC/Gn/ixjY= Received: from mail-qt1-f199.google.com (mail-qt1-f199.google.com [209.85.160.199]) by relay.mimecast.com with ESMTP with STARTTLS (version=TLSv1.3, cipher=TLS_AES_256_GCM_SHA384) id us-mta-190-80W5_HAMNamnPtrtIMjWSg-1; Fri, 09 Aug 2024 12:09:32 -0400 X-MC-Unique: 80W5_HAMNamnPtrtIMjWSg-1 Received: by mail-qt1-f199.google.com with SMTP id d75a77b69052e-4519ec19cb6so810441cf.1 for ; Fri, 09 Aug 2024 09:09:32 -0700 (PDT) X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20230601; t=1723219771; x=1723824571; h=content-transfer-encoding:mime-version:references:in-reply-to :message-id:date:subject:cc:to:from:x-gm-message-state:from:to:cc :subject:date:message-id:reply-to; bh=GAUMBfy0kdSy8bea6Q2/adQQTYYkrWuiUWfctkd/rW8=; b=mtfTGdX3qsk9Zj2y+lm147lEpNMFMGkk3Ftwfi0aH4T3cxnuL5fyIifBD/a46kabos d7Il8DtF6ihwwHuijWt2BT9brp8W6/q6LLvnCCluvcxtafnjAI2hDDsV5oqn3YTB9UQ2 fwV1v+aYcla+HbKI3E33rDpouAeaZ02QiE5ikkutYiRd+LeUqX2Ha2APEkIlUa03M9OK fujuB6xuQW5zORlS+XeQubIgcH+MDb3fVgVSPWChDOyw916kQuDgUTxbmZ5KDVDU5h0q sUzhgbC9ssL9ylXi/5aArecXec08U1XTtNNnrZjXk93zz4SWgY0mdnaY8qQzCoHeEa+q VxmA== X-Gm-Message-State: AOJu0YyTNS6DEFqFnzv8Stm67Hpuje4AdEVxASGT2+nJ0653XckXsjgt 5FV0DGq3IfEGD7RF/buZKr3nWZo5OoCweDcaNm/7HE49MtKDCOX7dFxOri51gfMpsnAPIv0fgkS yo80OM7DX6w+8ew4T1Krjo+2uPhiVXnLDRGXiXs53TkEzM96Pzl4xLEONBMm9zRBLrhqsf69w8A TCBnhs2hz9OQ1Gpq/X4pfvs8ql2J+awYex X-Received: by 2002:a05:622a:180e:b0:44e:d016:ef7 with SMTP id d75a77b69052e-45312652444mr12909511cf.7.1723219771202; Fri, 09 Aug 2024 09:09:31 -0700 (PDT) X-Google-Smtp-Source: AGHT+IH2AJ498flXqsf/RjsZizkWR5q4LBdwGhozxnqN6XMiL4rwNH8ESKXDSym68P6o8cpdHObfXw== X-Received: by 2002:a05:622a:180e:b0:44e:d016:ef7 with SMTP id d75a77b69052e-45312652444mr12908941cf.7.1723219770588; Fri, 09 Aug 2024 09:09:30 -0700 (PDT) Received: from x1n.redhat.com (pool-99-254-121-117.cpe.net.cable.rogers.com. [99.254.121.117]) by smtp.gmail.com with ESMTPSA id d75a77b69052e-451c870016csm22526741cf.19.2024.08.09.09.09.28 (version=TLS1_3 cipher=TLS_AES_256_GCM_SHA384 bits=256/256); Fri, 09 Aug 2024 09:09:30 -0700 (PDT) From: Peter Xu To: linux-mm@kvack.org, linux-kernel@vger.kernel.org Cc: Sean Christopherson , Oscar Salvador , Jason Gunthorpe , Axel Rasmussen , linux-arm-kernel@lists.infradead.org, x86@kernel.org, peterx@redhat.com, Will Deacon , Gavin Shan , Paolo Bonzini , Zi Yan , Andrew Morton , Catalin Marinas , Ingo Molnar , Alistair Popple , Borislav Petkov , David Hildenbrand , Thomas Gleixner , kvm@vger.kernel.org, Dave Hansen , Alex Williamson , Yan Zhao Subject: [PATCH 07/19] mm/fork: Accept huge pfnmap entries Date: Fri, 9 Aug 2024 12:08:57 -0400 Message-ID: <20240809160909.1023470-8-peterx@redhat.com> X-Mailer: git-send-email 2.45.0 In-Reply-To: <20240809160909.1023470-1-peterx@redhat.com> References: <20240809160909.1023470-1-peterx@redhat.com> MIME-Version: 1.0 X-Mimecast-Spam-Score: 0 X-Mimecast-Originator: redhat.com X-Rspamd-Server: rspam12 X-Rspamd-Queue-Id: DBED38001F X-Stat-Signature: ng4xz8h19yag844nrb968sokck5g85zf X-Rspam-User: X-HE-Tag: 1723219773-182903 X-HE-Meta: U2FsdGVkX1/d7BZ31OaaIVXAfUpHiYw3Ma8lcPqIYfw0BSgVucNePSWQ2Du64q4L3Wu4DpaY1wpARCPiAZGZ2VMeEc1Er5mkZ9EMr2I2/m1lHt8FFayL4sDGXAqFBU9TqhXae5Yyn/x9tiVYpN0o2mEBUhPSyVkYJ8bTiO7lX8f6bSjMiY3h3B8EaFc0yHA4uZ6vzPWab60v5DgWQI3nK8NxLjWVZuHvPqozCwSNPjIfDzBTd4D9RGUuH0zEPqT50cYlXcE7YA+JD2BX35CcxphYTdII1LTVsICQqIl/89jtsE4qfFSiZGap+kQzSD+H+RDm9sZRCKKACQrziCETPMhTtm2AdUWf/wasoRs25wtkdN64DOz28XGFyI3n0zQfoWm/ZAnE5CcC6djpOLHjCSTHQopQQqQVfBAzDEAhEj2FHxzBCPMSMLkRYcxq7Dldok4OFTKTDlCt+YPIQFC6BXbp5h0y43AcgKxsuDk/RWi7GyjOd3UdD+V1vxiJJqUPcre8E/L/RDkMj8v3dYUoCY2LUIgaz3sxOEJfGV+mCgIOQtN8N07or3KC5vUp9G1o3sI2/V2WV/wfjbL6c/A0cvGb5hUnY6N8GcPbgGKORDkA0qarkRVj86AhnslgQkVrSLKVQtWcgh0juwfvjFxUie15U3YzDveVoKOcE6RbysKlOFUMuN9GcLDkeiJEYDW+WgQweLZzoC/hkqkPKlzw72k7nYF0e0ZpyRm+s8N4Ck5b8W6vHZkQot9x0/9rYk8M66QJx+Ya8h4tmjWASCHMA0GoQFyhph9hOLhwBbeinvWVKhH5XJXseLWpkGyRW3wFandDOztcdMjUv/KUGy67l0wLurrPkq8sV7vfMfNJ2KZ9AAVB2Gnv8++GHl0Y5ZjP+tM92ZDsBL/176XZS9m9wyf95NuNbkKSMO02qWGZOg0OeNB5gho43wk+HIpaJfRifIifOi87nStNcS5ysG+ pYm/JzxV dC1A1SL1I3ZNzxTgv4DQ24YjezHt4StAJP6jihZ90i4+9kGEPmNgRrDB225p16eTOvXCWDeg9nwRjWQQ5PvT2YjvjSpIasIq1Fhk3ZOIBM2480BMNd4tX11y1Pi2aDjgIAQrxXHNqFkTOHU/noewjeubEAzjrsHh8T4EMc0x2SoqSYNYiZ5meIx7p8trVOIiEZLpP+qCqnEYUA2Buh05wlNuVKq1oRZ3gSkP/eyO5Xfe5mZgaBcJEx93sAmLDRJacRdpFKhPsSap40jtQibdztSchiLUgAbdm/I97PSDfE0NIP7TCyIDztzBRqo5ghMusPa+ynOxnpY87Nqz/vtgeVyWZtoqQEO0Cu3ClFlbBcvoZtTB4bDlOdBl2VBWGQCK6vFAF6GQWIPJS9GPRl+2YIhfL6k1Vi2yDQUDqM71xfP0JAk/8OF0q9KBLHg== X-Bogosity: Ham, tests=bogofilter, spamicity=0.000000, version=1.2.4 Sender: owner-linux-mm@kvack.org Precedence: bulk X-Loop: owner-majordomo@kvack.org List-ID: List-Subscribe: List-Unsubscribe: Teach the fork code to properly copy pfnmaps for pmd/pud levels. Pud is much easier, the write bit needs to be persisted though for writable and shared pud mappings like PFNMAP ones, otherwise a follow up write in either parent or child process will trigger a write fault. Do the same for pmd level. Signed-off-by: Peter Xu --- mm/huge_memory.c | 27 ++++++++++++++++++++++++--- 1 file changed, 24 insertions(+), 3 deletions(-) diff --git a/mm/huge_memory.c b/mm/huge_memory.c index 6568586b21ab..015c9468eed5 100644 --- a/mm/huge_memory.c +++ b/mm/huge_memory.c @@ -1375,6 +1375,22 @@ int copy_huge_pmd(struct mm_struct *dst_mm, struct mm_struct *src_mm, pgtable_t pgtable = NULL; int ret = -ENOMEM; + pmd = pmdp_get_lockless(src_pmd); + if (unlikely(pmd_special(pmd))) { + dst_ptl = pmd_lock(dst_mm, dst_pmd); + src_ptl = pmd_lockptr(src_mm, src_pmd); + spin_lock_nested(src_ptl, SINGLE_DEPTH_NESTING); + /* + * No need to recheck the pmd, it can't change with write + * mmap lock held here. + */ + if (is_cow_mapping(src_vma->vm_flags) && pmd_write(pmd)) { + pmdp_set_wrprotect(src_mm, addr, src_pmd); + pmd = pmd_wrprotect(pmd); + } + goto set_pmd; + } + /* Skip if can be re-fill on fault */ if (!vma_is_anonymous(dst_vma)) return 0; @@ -1456,7 +1472,9 @@ int copy_huge_pmd(struct mm_struct *dst_mm, struct mm_struct *src_mm, pmdp_set_wrprotect(src_mm, addr, src_pmd); if (!userfaultfd_wp(dst_vma)) pmd = pmd_clear_uffd_wp(pmd); - pmd = pmd_mkold(pmd_wrprotect(pmd)); + pmd = pmd_wrprotect(pmd); +set_pmd: + pmd = pmd_mkold(pmd); set_pmd_at(dst_mm, addr, dst_pmd, pmd); ret = 0; @@ -1502,8 +1520,11 @@ int copy_huge_pud(struct mm_struct *dst_mm, struct mm_struct *src_mm, * TODO: once we support anonymous pages, use * folio_try_dup_anon_rmap_*() and split if duplicating fails. */ - pudp_set_wrprotect(src_mm, addr, src_pud); - pud = pud_mkold(pud_wrprotect(pud)); + if (is_cow_mapping(vma->vm_flags) && pud_write(pud)) { + pudp_set_wrprotect(src_mm, addr, src_pud); + pud = pud_wrprotect(pud); + } + pud = pud_mkold(pud); set_pud_at(dst_mm, addr, dst_pud, pud); ret = 0; From patchwork Fri Aug 9 16:08:58 2024 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Peter Xu X-Patchwork-Id: 13758933 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from kanga.kvack.org (kanga.kvack.org [205.233.56.17]) by smtp.lore.kernel.org (Postfix) with ESMTP id 3D0DDC52D7C for ; Fri, 9 Aug 2024 16:09:40 +0000 (UTC) Received: by kanga.kvack.org (Postfix) id 4FCDB6B00A7; Fri, 9 Aug 2024 12:09:38 -0400 (EDT) Received: by kanga.kvack.org (Postfix, from userid 40) id 436516B00A8; Fri, 9 Aug 2024 12:09:38 -0400 (EDT) X-Delivered-To: int-list-linux-mm@kvack.org Received: by kanga.kvack.org (Postfix, from userid 63042) id 23B5E6B00AA; Fri, 9 Aug 2024 12:09:38 -0400 (EDT) X-Delivered-To: linux-mm@kvack.org Received: from relay.hostedemail.com (smtprelay0013.hostedemail.com [216.40.44.13]) by kanga.kvack.org (Postfix) with ESMTP id E613E6B00A7 for ; Fri, 9 Aug 2024 12:09:37 -0400 (EDT) Received: from smtpin26.hostedemail.com (a10.router.float.18 [10.200.18.1]) by unirelay07.hostedemail.com (Postfix) with ESMTP id 9CE27160F9A for ; Fri, 9 Aug 2024 16:09:37 +0000 (UTC) X-FDA: 82433192394.26.468F66A Received: from us-smtp-delivery-124.mimecast.com (us-smtp-delivery-124.mimecast.com [170.10.133.124]) by imf27.hostedemail.com (Postfix) with ESMTP id 8E4D34000C for ; Fri, 9 Aug 2024 16:09:35 +0000 (UTC) Authentication-Results: imf27.hostedemail.com; dkim=pass header.d=redhat.com header.s=mimecast20190719 header.b=Gth6H0Ji; dmarc=pass (policy=none) header.from=redhat.com; spf=pass (imf27.hostedemail.com: domain of peterx@redhat.com designates 170.10.133.124 as permitted sender) smtp.mailfrom=peterx@redhat.com ARC-Seal: i=1; s=arc-20220608; d=hostedemail.com; t=1723219721; a=rsa-sha256; cv=none; b=Se5Ww7B0ZVAIjVm6ZHGNknkSPmr7WVsrShVlsgMAnJnZkW2NSmOpACxH/qHhKgOB9/gxmq RECg18UZRLWZnuLOXfW+POTp9bgEGFUgaaI9RvzlBJTQuMKjGC614oR/ce4++WLp1HnZOX p+YzslIOVZIkesm8tefH03LzE6+64Zc= ARC-Authentication-Results: i=1; imf27.hostedemail.com; dkim=pass header.d=redhat.com header.s=mimecast20190719 header.b=Gth6H0Ji; dmarc=pass (policy=none) header.from=redhat.com; spf=pass (imf27.hostedemail.com: domain of peterx@redhat.com designates 170.10.133.124 as permitted sender) smtp.mailfrom=peterx@redhat.com ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=hostedemail.com; s=arc-20220608; t=1723219721; h=from:from:sender:reply-to:subject:subject:date:date: message-id:message-id:to:to:cc:cc:mime-version:mime-version: content-type:content-type: content-transfer-encoding:content-transfer-encoding: in-reply-to:in-reply-to:references:references:dkim-signature; bh=3aM3oVPeXmFPFf4Q68bjufzjsUq2ESmscQPnLEkaBB8=; b=MaQJEdVNEHM6ZdWjEm1HA+tL3XO/diJZpqcaYvD1tpK2q/NKqbCoyNNbK+HjK65eA+ZM5t gJ90FER6fXNsEPqaU4bFwBmUTvQbROPZiK+AQe26bdAT4yJhTyxphtRsfQYEbmF+dW9GzS PWt58PP72gxLATnm02n+riyntfrVB88= DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=redhat.com; s=mimecast20190719; t=1723219774; h=from:from:reply-to:subject:subject:date:date:message-id:message-id: to:to:cc:cc:mime-version:mime-version:content-type:content-type: content-transfer-encoding:content-transfer-encoding: in-reply-to:in-reply-to:references:references; bh=3aM3oVPeXmFPFf4Q68bjufzjsUq2ESmscQPnLEkaBB8=; b=Gth6H0JiQmUa8JZCcN9EQQ4jGMoDek3jzRrpm7yccOdxJ2dcj/2xCrQIsPWVsOb8zIQcUc 3UNr9ELVDTKlsc6cTCYFqgalUjdEjHya6cuvgLql25OW6DiiXOwpn2N1Wb8lxre4wtt+ts omVi9S15ZmZbdyWjRSRSnLA04AOLiIo= Received: from mail-qt1-f198.google.com (mail-qt1-f198.google.com [209.85.160.198]) by relay.mimecast.com with ESMTP with STARTTLS (version=TLSv1.3, cipher=TLS_AES_256_GCM_SHA384) id us-mta-218-4OQMLml-M5yGbcJ7ato0iA-1; Fri, 09 Aug 2024 12:09:33 -0400 X-MC-Unique: 4OQMLml-M5yGbcJ7ato0iA-1 Received: by mail-qt1-f198.google.com with SMTP id d75a77b69052e-45009e27b83so4905411cf.1 for ; Fri, 09 Aug 2024 09:09:33 -0700 (PDT) X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20230601; t=1723219773; x=1723824573; h=content-transfer-encoding:mime-version:references:in-reply-to :message-id:date:subject:cc:to:from:x-gm-message-state:from:to:cc :subject:date:message-id:reply-to; bh=3aM3oVPeXmFPFf4Q68bjufzjsUq2ESmscQPnLEkaBB8=; b=hlQ3THaxKTDNA+P+saUnvQrab6De0EqkJu6zY8L+MNk2c/l7jIJv464BXzq7e/d/1f N6R+o1syVjWCqxtRfrvEvBf8qwbH3O3IWE7JKquvRoBgiyef8qhow9QXLF7vvY3A4hNk PlpvTH4NH0/udGqjMCa2axQpiYMHDeZgmWdB4GuyvVa80W6FvlzG7O09kuASSrcWmTBg fd8EnWVMLm76IGHe6ua9k8jlmGXGRUKlAf5Ptj2Towpt1uHLOo5EqAa5P7Zj72yL0stE 9Z+vg+O3gyYQqo37ecpOBh9dyy6456hNouDPFE9Hu24Y05nqTpw2Ot4xbdZQ6IkMKK5C 4zlQ== X-Gm-Message-State: AOJu0YxSMlIzk+Y9Unf8UjkO5Y2WQHT8i85mLYThb0h3ZasJ/puiUxKc W6+ycX+oG6BedYvVdttl9ruMyRYZ+deKOHPFn1Poq8pBPHoGZM5TY/eWo1niub1fNWhAWXAOLee 2WSVxi+PEfyva1Ykh4knvk2zaWtxnCQU9+2hGEI25yWSNGeIho4PjEJqbVjYmHgbeBYPYknC6aU OGWGLmeGbXjLqRCPWAca8MSpO2N3z+dtYg X-Received: by 2002:ac8:5f10:0:b0:450:349:1170 with SMTP id d75a77b69052e-4531255c62dmr14739881cf.6.1723219773131; Fri, 09 Aug 2024 09:09:33 -0700 (PDT) X-Google-Smtp-Source: AGHT+IE6GyuXgyukqToIMYwmYCyyJms6T0RBSwsqvRpSKO6DWSQsB926zH9/mqcqwcpfSE6Nqd5Tsw== X-Received: by 2002:ac8:5f10:0:b0:450:349:1170 with SMTP id d75a77b69052e-4531255c62dmr14739431cf.6.1723219772696; Fri, 09 Aug 2024 09:09:32 -0700 (PDT) Received: from x1n.redhat.com (pool-99-254-121-117.cpe.net.cable.rogers.com. [99.254.121.117]) by smtp.gmail.com with ESMTPSA id d75a77b69052e-451c870016csm22526741cf.19.2024.08.09.09.09.30 (version=TLS1_3 cipher=TLS_AES_256_GCM_SHA384 bits=256/256); Fri, 09 Aug 2024 09:09:32 -0700 (PDT) From: Peter Xu To: linux-mm@kvack.org, linux-kernel@vger.kernel.org Cc: Sean Christopherson , Oscar Salvador , Jason Gunthorpe , Axel Rasmussen , linux-arm-kernel@lists.infradead.org, x86@kernel.org, peterx@redhat.com, Will Deacon , Gavin Shan , Paolo Bonzini , Zi Yan , Andrew Morton , Catalin Marinas , Ingo Molnar , Alistair Popple , Borislav Petkov , David Hildenbrand , Thomas Gleixner , kvm@vger.kernel.org, Dave Hansen , Alex Williamson , Yan Zhao Subject: [PATCH 08/19] mm: Always define pxx_pgprot() Date: Fri, 9 Aug 2024 12:08:58 -0400 Message-ID: <20240809160909.1023470-9-peterx@redhat.com> X-Mailer: git-send-email 2.45.0 In-Reply-To: <20240809160909.1023470-1-peterx@redhat.com> References: <20240809160909.1023470-1-peterx@redhat.com> MIME-Version: 1.0 X-Mimecast-Spam-Score: 0 X-Mimecast-Originator: redhat.com X-Rspamd-Server: rspam12 X-Rspamd-Queue-Id: 8E4D34000C X-Stat-Signature: xbhr45hhrgicqzuoujw7e94btamb9ija X-Rspam-User: X-HE-Tag: 1723219775-790388 X-HE-Meta: U2FsdGVkX1/iaUx86YtEPNeDHynZHOsNuJgjSqSckVEjGolFDDfWakivn6Lc6kppujT9fjy3Or5wjQ4SQ534G7adSgPakcfPN2+/iN4uU9cMZ2vDAhT04YVzhrB72szq+6MEwEaTpcvvGHlPLZnoI9FUmSZGlKC5TH5xh1AzJo9rpCagLlTyPCxaZ2cQaM16Cazf3+LhV1ExEn7cIyE9qFtUi5EWI0nJl5LIOhQYDZbfpzTHl0jeIYpbNHrhsvq3d0TIcSlSl6EL0mssLo5N3tE7I6NQENPqGfY5Vo8ax+JaDizYiypGU2rZOkHclWvlHJukh6UF/9RmwZuAQTMxrtxH4Hf1nRrmtYreh2/aqzp0eTBVxxvKugGMK1GjsrvKcdljqT2EQshaspdV1EoROWMuFPUUAA2s1pERp6i66uDnhu/ViJiPEkoo2WDOoXaBWeyup/fz8MpBYQj6OT/4oWGot3VJTBegxWRyBp+eeEIeWGqd88YFEap1pETh1uTxq/CF9lqsZ58ii1knGRaERqaOy5J5onZavaqPrHDX942YaB+wf/dxkxhugrplPFujOo+HNNwfdZcwOYWEFkphFZLlyFprcXcQCNYqMI0ApkIvNUTEUrbRP4lpYjZz2DsrncA/iWs861iIziavmiwxhUmHNNdN5OjzNpIpT0bUR8IpIi+WgzghrGJm1ubHGkCIpIFHZlj1dhd3AfcfoI2aBjTF2DN/CeoqE8xsF4WuG6H/59jrF+ckAE28KrMWAYqSd1MBp9f1/sky4ta46bv24wZlPtUZ6gNtmnC7qPXEH4Qk4gH54AyRgWBOMQ2mC1Vh+z+UBVmSZi6vgOBgdneKc6OiILdJh4FyzIo9r4KYciQVvs8GsyHchlGc37Yyv12MYbAI6mcMkzsJ4DFRhjcajt3TUPyuIyW/pQ1WmHTD1d4gJ1CAyEYKoE87YFZ3mVidDcj6f02oYPJEkwM3Fzn dXjATJvP Xggxw9beoyIJ7CbKYfXj4SGjGK4vJjvm6KeRT2fjL5JOjv/b4RXoToRv1zBl/eNU0ZljncWJs2L9TBLYOBfgXFixGJo6qOvi2+pah4CqpqNknm+TdL2V92LmBKP5/hvXQZFxi0G5meGcHaHpLrxXC1nxQGIv1m+l7imeU5a0KHt068GJ6FcdG/YGeZ1DWoq2TW6vrS7m5CLUJLRNXliqehVLYbjCfg7iQJGohL9fEXLDaNvBw6uwrtCB3Q0+Qs+KCdAvUikPThS4jiQezhtb7Mz5aomtCi1TgDtn3NfPCaIbXa9HTEP2TxuMzoHJMF9So8t9RiqcCArnrp9BvG9TzuDOtql9N6v/AqD/EjfvHPOTAqSs+bQ02r2QkOFsTb8deKfWkBT3DH9ftqHl4za29OU5qVJTkSEhjjzB7Lk7lXy/jvMW/RuXih9OhW3Y0j89WJvqWnttuCzq9vwElxZsUnR9lC29Aqgg7OLFSZ5DDhWrjC4fxhljjB7iL5S9quW/Z3N4n X-Bogosity: Ham, tests=bogofilter, spamicity=0.000000, version=1.2.4 Sender: owner-linux-mm@kvack.org Precedence: bulk X-Loop: owner-majordomo@kvack.org List-ID: List-Subscribe: List-Unsubscribe: There're: - 8 archs (arc, arm64, include, mips, powerpc, s390, sh, x86) that support pte_pgprot(). - 2 archs (x86, sparc) that support pmd_pgprot(). - 1 arch (x86) that support pud_pgprot(). Always define them to be used in generic code, and then we don't need to fiddle with "#ifdef"s when doing so. Signed-off-by: Peter Xu Reviewed-by: Jason Gunthorpe --- arch/arm64/include/asm/pgtable.h | 1 + arch/powerpc/include/asm/pgtable.h | 1 + arch/s390/include/asm/pgtable.h | 1 + arch/sparc/include/asm/pgtable_64.h | 1 + include/linux/pgtable.h | 12 ++++++++++++ 5 files changed, 16 insertions(+) diff --git a/arch/arm64/include/asm/pgtable.h b/arch/arm64/include/asm/pgtable.h index 7a4f5604be3f..b78cc4a6758b 100644 --- a/arch/arm64/include/asm/pgtable.h +++ b/arch/arm64/include/asm/pgtable.h @@ -384,6 +384,7 @@ static inline void __sync_cache_and_tags(pte_t pte, unsigned int nr_pages) /* * Select all bits except the pfn */ +#define pte_pgprot pte_pgprot static inline pgprot_t pte_pgprot(pte_t pte) { unsigned long pfn = pte_pfn(pte); diff --git a/arch/powerpc/include/asm/pgtable.h b/arch/powerpc/include/asm/pgtable.h index 264a6c09517a..2f72ad885332 100644 --- a/arch/powerpc/include/asm/pgtable.h +++ b/arch/powerpc/include/asm/pgtable.h @@ -65,6 +65,7 @@ static inline unsigned long pte_pfn(pte_t pte) /* * Select all bits except the pfn */ +#define pte_pgprot pte_pgprot static inline pgprot_t pte_pgprot(pte_t pte) { unsigned long pte_flags; diff --git a/arch/s390/include/asm/pgtable.h b/arch/s390/include/asm/pgtable.h index 3fa280d0672a..0ffbaf741955 100644 --- a/arch/s390/include/asm/pgtable.h +++ b/arch/s390/include/asm/pgtable.h @@ -955,6 +955,7 @@ static inline int pte_unused(pte_t pte) * young/old accounting is not supported, i.e _PAGE_PROTECT and _PAGE_INVALID * must not be set. */ +#define pte_pgprot pte_pgprot static inline pgprot_t pte_pgprot(pte_t pte) { unsigned long pte_flags = pte_val(pte) & _PAGE_CHG_MASK; diff --git a/arch/sparc/include/asm/pgtable_64.h b/arch/sparc/include/asm/pgtable_64.h index 3fe429d73a65..2b7f358762c1 100644 --- a/arch/sparc/include/asm/pgtable_64.h +++ b/arch/sparc/include/asm/pgtable_64.h @@ -783,6 +783,7 @@ static inline pmd_t pmd_mkwrite_novma(pmd_t pmd) return __pmd(pte_val(pte)); } +#define pmd_pgprot pmd_pgprot static inline pgprot_t pmd_pgprot(pmd_t entry) { unsigned long val = pmd_val(entry); diff --git a/include/linux/pgtable.h b/include/linux/pgtable.h index 780f3b439d98..e8b2ac6bd2ae 100644 --- a/include/linux/pgtable.h +++ b/include/linux/pgtable.h @@ -1956,6 +1956,18 @@ typedef unsigned int pgtbl_mod_mask; #define MAX_PTRS_PER_P4D PTRS_PER_P4D #endif +#ifndef pte_pgprot +#define pte_pgprot(x) ((pgprot_t) {0}) +#endif + +#ifndef pmd_pgprot +#define pmd_pgprot(x) ((pgprot_t) {0}) +#endif + +#ifndef pud_pgprot +#define pud_pgprot(x) ((pgprot_t) {0}) +#endif + /* description of effects of mapping type and prot in current implementation. * this is due to the limited x86 page protection hardware. The expected * behavior is in parens: From patchwork Fri Aug 9 16:08:59 2024 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Peter Xu X-Patchwork-Id: 13758934 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from kanga.kvack.org (kanga.kvack.org [205.233.56.17]) by smtp.lore.kernel.org (Postfix) with ESMTP id AD915C52D7C for ; Fri, 9 Aug 2024 16:09:43 +0000 (UTC) Received: by kanga.kvack.org (Postfix) id 615EA6B00A8; Fri, 9 Aug 2024 12:09:41 -0400 (EDT) Received: by kanga.kvack.org (Postfix, from userid 40) id 5BB8D6B00AA; Fri, 9 Aug 2024 12:09:41 -0400 (EDT) X-Delivered-To: int-list-linux-mm@kvack.org Received: by kanga.kvack.org (Postfix, from userid 63042) id 3C22F6B00AB; Fri, 9 Aug 2024 12:09:41 -0400 (EDT) X-Delivered-To: linux-mm@kvack.org Received: from relay.hostedemail.com (smtprelay0014.hostedemail.com [216.40.44.14]) by kanga.kvack.org (Postfix) with ESMTP id 1BB906B00A8 for ; Fri, 9 Aug 2024 12:09:41 -0400 (EDT) Received: from smtpin22.hostedemail.com (a10.router.float.18 [10.200.18.1]) by unirelay10.hostedemail.com (Postfix) with ESMTP id C46B2C1275 for ; Fri, 9 Aug 2024 16:09:40 +0000 (UTC) X-FDA: 82433192520.22.18AD2AC Received: from us-smtp-delivery-124.mimecast.com (us-smtp-delivery-124.mimecast.com [170.10.133.124]) by imf02.hostedemail.com (Postfix) with ESMTP id ADC428001F for ; Fri, 9 Aug 2024 16:09:38 +0000 (UTC) Authentication-Results: imf02.hostedemail.com; dkim=pass header.d=redhat.com header.s=mimecast20190719 header.b=gNU0me8v; spf=pass (imf02.hostedemail.com: domain of peterx@redhat.com designates 170.10.133.124 as permitted sender) smtp.mailfrom=peterx@redhat.com; dmarc=pass (policy=none) header.from=redhat.com ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=hostedemail.com; s=arc-20220608; t=1723219746; h=from:from:sender:reply-to:subject:subject:date:date: message-id:message-id:to:to:cc:cc:mime-version:mime-version: content-type:content-type: content-transfer-encoding:content-transfer-encoding: in-reply-to:in-reply-to:references:references:dkim-signature; bh=IXGL3lb2RETUVA/g/GpjABQk9U9hrAJ/lFeyUGjNHTs=; b=PsqBdWeUL//PCjl7Yxj5BnnBHdNVh2+DMFhR0483zu+xvYn1UNS6OhFQpaLnFLDs0CpxbU CsjFHNGc1UkOFBdDe+x8UX1zTzS1jGfFeoI/vf697HBc4JMm4bxlK6C5ERFBcrXYvBfuB3 LuW1peOhLrQ/vmToe2oFe7WS6aAw/pg= ARC-Authentication-Results: i=1; imf02.hostedemail.com; dkim=pass header.d=redhat.com header.s=mimecast20190719 header.b=gNU0me8v; spf=pass (imf02.hostedemail.com: domain of peterx@redhat.com designates 170.10.133.124 as permitted sender) smtp.mailfrom=peterx@redhat.com; dmarc=pass (policy=none) header.from=redhat.com ARC-Seal: i=1; s=arc-20220608; d=hostedemail.com; t=1723219746; a=rsa-sha256; cv=none; b=r3Ir0GcltIxNg45PJPX570voBwyNjXqd6QP6GMBAj2eoaew11GQlYcl/JZ+Ga0avx3Ou/2 3zBI9yHgmuH8HBXEuEGjahP5Axh+lM8pPMg1YgB9vrAJMPqYvhg7qbL4KF5btLefnOmcvK q34A/tn8xTg8g5QyFCyDZo6CQ/qaazk= DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=redhat.com; s=mimecast20190719; t=1723219778; h=from:from:reply-to:subject:subject:date:date:message-id:message-id: to:to:cc:cc:mime-version:mime-version:content-type:content-type: content-transfer-encoding:content-transfer-encoding: in-reply-to:in-reply-to:references:references; bh=IXGL3lb2RETUVA/g/GpjABQk9U9hrAJ/lFeyUGjNHTs=; b=gNU0me8vMRbbVC9DF2lfOPdArJ/GIHar3HMxA0n1QtMxL/zS3tkBH+UWyflm0fKxXHAOhN 8KTIaMuhrlK6p3ylA6ZpX1smihZda9eh5jweJYo+MSMV/e6Be7xzri0gmjCsBIxkR32Odg zTCyCtk5dngqimv5yVIPW7sVKmr9V60= Received: from mail-qt1-f199.google.com (mail-qt1-f199.google.com [209.85.160.199]) by relay.mimecast.com with ESMTP with STARTTLS (version=TLSv1.3, cipher=TLS_AES_256_GCM_SHA384) id us-mta-470-nfNbIargOVqoDv8noLiS6Q-1; Fri, 09 Aug 2024 12:09:36 -0400 X-MC-Unique: nfNbIargOVqoDv8noLiS6Q-1 Received: by mail-qt1-f199.google.com with SMTP id d75a77b69052e-44ff25bbfe1so3070151cf.3 for ; Fri, 09 Aug 2024 09:09:36 -0700 (PDT) X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20230601; t=1723219776; x=1723824576; h=content-transfer-encoding:mime-version:references:in-reply-to :message-id:date:subject:cc:to:from:x-gm-message-state:from:to:cc :subject:date:message-id:reply-to; bh=IXGL3lb2RETUVA/g/GpjABQk9U9hrAJ/lFeyUGjNHTs=; b=EULe+W3KQ9/Q1i38EGYd1kj6C0+rNWe+tEwiOzjt3u7X2F0QEmkz2Etss3RgXoQMdv UQBCuq9TYhpShtrMoA8oCeZxqrXAyYPdZtWTA7sB+U7LZz/YlJPRAbeJVNYK615Xmu13 qdK9idI3v9NF7yczO4VJH3tVJbiJJmMnyYf5VoUmxQhVSQSTfDxVNq6hQbPy1GcS1QvA UCw2Kma/Ufdi+kkul4fp6Bf0egkCjLoNKQyIAjmBJN6Z/+tfAte6NeA3h7E1mKovUUtB pcg+Oz6ozT6dWW7JfmlDXVoXW1DKIhNWD1Qm1u/ZXKIjjQOKIgnjuB2f8A2od51aJEt5 OzDQ== X-Gm-Message-State: AOJu0Yw8VecfAqNljopvtV6Og/6P7c0aw6qhDq+6wPDD+hW+sbKTtlsJ YCEjH6FSVp+NWgdEBLQIIHP3tTb4zLz8+FcCHeq9bFtEPwq86EHCAfUOoixs1Mz1rfUT8YhXaFE XdBFYFru1V9WpL2AIqVcsNZIJjkW9HYZUvK3Fj1KCs+zIi4aYzjoD21+WrQmzd+ZpFMa99xkxAO +xTAr14REGv3d2kktPaVAxFAPbPTn46H3/ X-Received: by 2002:a05:622a:1b8e:b0:447:e636:9ea9 with SMTP id d75a77b69052e-4531251c724mr14428861cf.3.1723219775827; Fri, 09 Aug 2024 09:09:35 -0700 (PDT) X-Google-Smtp-Source: AGHT+IFUrJX3vSa/2Mr5PpFf2V5KDcJEixKltLA8oaEJqLbElyFRMCegYAzgvcdg3qK/Zl0Af+cLBA== X-Received: by 2002:a05:622a:1b8e:b0:447:e636:9ea9 with SMTP id d75a77b69052e-4531251c724mr14428311cf.3.1723219775158; Fri, 09 Aug 2024 09:09:35 -0700 (PDT) Received: from x1n.redhat.com (pool-99-254-121-117.cpe.net.cable.rogers.com. [99.254.121.117]) by smtp.gmail.com with ESMTPSA id d75a77b69052e-451c870016csm22526741cf.19.2024.08.09.09.09.32 (version=TLS1_3 cipher=TLS_AES_256_GCM_SHA384 bits=256/256); Fri, 09 Aug 2024 09:09:34 -0700 (PDT) From: Peter Xu To: linux-mm@kvack.org, linux-kernel@vger.kernel.org Cc: Sean Christopherson , Oscar Salvador , Jason Gunthorpe , Axel Rasmussen , linux-arm-kernel@lists.infradead.org, x86@kernel.org, peterx@redhat.com, Will Deacon , Gavin Shan , Paolo Bonzini , Zi Yan , Andrew Morton , Catalin Marinas , Ingo Molnar , Alistair Popple , Borislav Petkov , David Hildenbrand , Thomas Gleixner , kvm@vger.kernel.org, Dave Hansen , Alex Williamson , Yan Zhao Subject: [PATCH 09/19] mm: New follow_pfnmap API Date: Fri, 9 Aug 2024 12:08:59 -0400 Message-ID: <20240809160909.1023470-10-peterx@redhat.com> X-Mailer: git-send-email 2.45.0 In-Reply-To: <20240809160909.1023470-1-peterx@redhat.com> References: <20240809160909.1023470-1-peterx@redhat.com> MIME-Version: 1.0 X-Mimecast-Spam-Score: 0 X-Mimecast-Originator: redhat.com X-Rspamd-Server: rspam03 X-Rspam-User: X-Rspamd-Queue-Id: ADC428001F X-Stat-Signature: uzhbbxz368tyuy34opcimxostnq191b4 X-HE-Tag: 1723219778-1973 X-HE-Meta: U2FsdGVkX18AQFWjbM5ocGaXmc/9uIOwdwtJQTHKtD/Go3dFqjsYdjlAlW8KQh82x3iOPl633goWRqimAN4X/0fokEVxEltUuHDYrMpzpw7Wp5ZckV/sk8qlXKawh43Hko+Tp24nva6+jT7VacydFElsVo49PpNUCS1IfNnKyZ2/5HmGcX6I/Gz0mbLTQwW8xIErsXkgFqxZs9TyhUFJP0J7A3E5Q11YrPTM//S4I4u2c83FN8IPbKVSFTFzA0lv65kNGLVaw2ScgHwVhiCXuH6+9uxNvm3vqsvPjWWH/TiWE4lbvvip3F6uiZMsRuHhl77ESBNAeb3SPXHpmzeddNbVisF9fZjdf9bRp6FghCAGFaYAa1l4lfWMLUqftutJxDo4NhF8pqFJcOeU2SV/rGagycujZvSehdfVAUujqDyWLdqN0MhUnZwedzBhySfQ6to4DaLrNDaN33yv2sFjj6UUS8Z/kJeqo3YRcuvkWnMVSrKmluWE3MhVycZA6dNlBqbbxWTgTghfLKovhLt6WMDa7SKstiYqOc4qq1Mj+n11WFZRInoEMilZ6HucjC1PH+l0XF7SD49nKoX3aL01+0iIdUaOZmevzRj8njJMhPp3JghBKTVx+owr7oXZeP6xhgdNSL1Wv01a6oWhnLiVqqdMfwfskmlekefFyOiy9JfswnZ9hUDpjcfQPIx7Ne+b0K4t8ctutCNgPhNMXLGZWY8x9I1PLtKi9d4Vxr7xD6Y4n+NV+EdcSiU280v358jHVxV6xui+B2BAmXMe0KM1dGWfmM+jKxfnk2t9bGOw/FziGPPDbLXacPuMTkCKTjeEQR9ciBJEI+L9WM5yGWQlhx8qmlIG/TY8wbxxYlxWUIaOICLCqppBmDnVHsxHHSxwhd9SoWP7IHk2Xm7RWy9lm4hmJKfPpER12Y2cJbGNAjpfvfdJfLJRdBaij3Vmn9NxyavnbUdnfA4KlvZjZgt DhJSeQ2y Z5FSW52CRAVjjt9mkaZv+fAbmqyXGCGPg45Ick6EBdFJFAqli/8K647aWdx6JsKn8VY8Qm3rKJRiXBhrVLHXAOGyFJt2w9SWVfQ0mTvWAkbAsokORcKLHKZufso90yzm+ig+salmXDSMnNTQ1DqAh/Sf0p8x4vnMnfeo40l2nhXY++Gm9mvN0UExUrxpGIa4HJXNapDVjMqThMbOUFMZJixALTnXTD1UU2qmBUpG5nSEiKwYLzx2k+5BEDcBaLs85Il8gF21TKSEjoDqY8A621f3dT7E6wIskxFpz+TfA9ff71B/C4DTxM9C7/osSJkmcJIz6h+KFUv58hU0+kvdA6XV+THaUh4T8OFqEC4fBbHf1yM2Sv7+CcBj7BWONb+w8Xcv2OgOGIp9XUKJfik7zryA5nd4i2GoCNf1MmjVm9Ma3Bmw5FZc8YHzejjjWEyg1YU9ZV6h3bt+Y4vserjGBxXEsDmUfHzSWgOKe/6s7nHlJCNaswyyggzeHfa1Zzm6qdEmb X-Bogosity: Ham, tests=bogofilter, spamicity=0.000000, version=1.2.4 Sender: owner-linux-mm@kvack.org Precedence: bulk X-Loop: owner-majordomo@kvack.org List-ID: List-Subscribe: List-Unsubscribe: Introduce a pair of APIs to follow pfn mappings to get entry information. It's very similar to what follow_pte() does before, but different in that it recognizes huge pfn mappings. Signed-off-by: Peter Xu --- include/linux/mm.h | 31 ++++++++++ mm/memory.c | 147 +++++++++++++++++++++++++++++++++++++++++++++ 2 files changed, 178 insertions(+) diff --git a/include/linux/mm.h b/include/linux/mm.h index 90ca84200800..7471302658af 100644 --- a/include/linux/mm.h +++ b/include/linux/mm.h @@ -2374,6 +2374,37 @@ int follow_pte(struct vm_area_struct *vma, unsigned long address, int generic_access_phys(struct vm_area_struct *vma, unsigned long addr, void *buf, int len, int write); +struct follow_pfnmap_args { + /** + * Inputs: + * @vma: Pointer to @vm_area_struct struct + * @address: the virtual address to walk + */ + struct vm_area_struct *vma; + unsigned long address; + /** + * Internals: + * + * The caller shouldn't touch any of these. + */ + spinlock_t *lock; + pte_t *ptep; + /** + * Outputs: + * + * @pfn: the PFN of the address + * @pgprot: the pgprot_t of the mapping + * @writable: whether the mapping is writable + * @special: whether the mapping is a special mapping (real PFN maps) + */ + unsigned long pfn; + pgprot_t pgprot; + bool writable; + bool special; +}; +int follow_pfnmap_start(struct follow_pfnmap_args *args); +void follow_pfnmap_end(struct follow_pfnmap_args *args); + extern void truncate_pagecache(struct inode *inode, loff_t new); extern void truncate_setsize(struct inode *inode, loff_t newsize); void pagecache_isize_extended(struct inode *inode, loff_t from, loff_t to); diff --git a/mm/memory.c b/mm/memory.c index 67496dc5064f..2194e0f9f541 100644 --- a/mm/memory.c +++ b/mm/memory.c @@ -6338,6 +6338,153 @@ int follow_pte(struct vm_area_struct *vma, unsigned long address, } EXPORT_SYMBOL_GPL(follow_pte); +static inline void pfnmap_args_setup(struct follow_pfnmap_args *args, + spinlock_t *lock, pte_t *ptep, + pgprot_t pgprot, unsigned long pfn_base, + unsigned long addr_mask, bool writable, + bool special) +{ + args->lock = lock; + args->ptep = ptep; + args->pfn = pfn_base + ((args->address & ~addr_mask) >> PAGE_SHIFT); + args->pgprot = pgprot; + args->writable = writable; + args->special = special; +} + +static inline void pfnmap_lockdep_assert(struct vm_area_struct *vma) +{ +#ifdef CONFIG_LOCKDEP + struct address_space *mapping = vma->vm_file->f_mapping; + + if (mapping) + lockdep_assert(lockdep_is_held(&vma->vm_file->f_mapping->i_mmap_rwsem) || + lockdep_is_held(&vma->vm_mm->mmap_lock)); + else + lockdep_assert(lockdep_is_held(&vma->vm_mm->mmap_lock)); +#endif +} + +/** + * follow_pfnmap_start() - Look up a pfn mapping at a user virtual address + * @args: Pointer to struct @follow_pfnmap_args + * + * The caller needs to setup args->vma and args->address to point to the + * virtual address as the target of such lookup. On a successful return, + * the results will be put into other output fields. + * + * After the caller finished using the fields, the caller must invoke + * another follow_pfnmap_end() to proper releases the locks and resources + * of such look up request. + * + * During the start() and end() calls, the results in @args will be valid + * as proper locks will be held. After the end() is called, all the fields + * in @follow_pfnmap_args will be invalid to be further accessed. + * + * If the PTE maps a refcounted page, callers are responsible to protect + * against invalidation with MMU notifiers; otherwise access to the PFN at + * a later point in time can trigger use-after-free. + * + * Only IO mappings and raw PFN mappings are allowed. The mmap semaphore + * should be taken for read, and the mmap semaphore cannot be released + * before the end() is invoked. + * + * This function must not be used to modify PTE content. + * + * Return: zero on success, -ve otherwise. + */ +int follow_pfnmap_start(struct follow_pfnmap_args *args) +{ + struct vm_area_struct *vma = args->vma; + unsigned long address = args->address; + struct mm_struct *mm = vma->vm_mm; + spinlock_t *lock; + pgd_t *pgdp; + p4d_t *p4dp, p4d; + pud_t *pudp, pud; + pmd_t *pmdp, pmd; + pte_t *ptep, pte; + + pfnmap_lockdep_assert(vma); + + if (unlikely(address < vma->vm_start || address >= vma->vm_end)) + goto out; + + if (!(vma->vm_flags & (VM_IO | VM_PFNMAP))) + goto out; +retry: + pgdp = pgd_offset(mm, address); + if (pgd_none(*pgdp) || unlikely(pgd_bad(*pgdp))) + goto out; + + p4dp = p4d_offset(pgdp, address); + p4d = READ_ONCE(*p4dp); + if (p4d_none(p4d) || unlikely(p4d_bad(p4d))) + goto out; + + pudp = pud_offset(p4dp, address); + pud = READ_ONCE(*pudp); + if (pud_none(pud)) + goto out; + if (pud_leaf(pud)) { + lock = pud_lock(mm, pudp); + if (!unlikely(pud_leaf(pud))) { + spin_unlock(lock); + goto retry; + } + pfnmap_args_setup(args, lock, NULL, pud_pgprot(pud), + pud_pfn(pud), PUD_MASK, pud_write(pud), + pud_special(pud)); + return 0; + } + + pmdp = pmd_offset(pudp, address); + pmd = pmdp_get_lockless(pmdp); + if (pmd_leaf(pmd)) { + lock = pmd_lock(mm, pmdp); + if (!unlikely(pmd_leaf(pmd))) { + spin_unlock(lock); + goto retry; + } + pfnmap_args_setup(args, lock, NULL, pmd_pgprot(pmd), + pmd_pfn(pmd), PMD_MASK, pmd_write(pmd), + pmd_special(pmd)); + return 0; + } + + ptep = pte_offset_map_lock(mm, pmdp, address, &lock); + if (!ptep) + goto out; + pte = ptep_get(ptep); + if (!pte_present(pte)) + goto unlock; + pfnmap_args_setup(args, lock, ptep, pte_pgprot(pte), + pte_pfn(pte), PAGE_MASK, pte_write(pte), + pte_special(pte)); + return 0; +unlock: + pte_unmap_unlock(ptep, lock); +out: + return -EINVAL; +} +EXPORT_SYMBOL_GPL(follow_pfnmap_start); + +/** + * follow_pfnmap_end(): End a follow_pfnmap_start() process + * @args: Pointer to struct @follow_pfnmap_args + * + * Must be used in pair of follow_pfnmap_start(). See the start() function + * above for more information. + */ +void follow_pfnmap_end(struct follow_pfnmap_args *args) +{ + if (args->lock) + spin_unlock(args->lock); + if (args->ptep) + pte_unmap(args->ptep); +} +EXPORT_SYMBOL_GPL(follow_pfnmap_end); + #ifdef CONFIG_HAVE_IOREMAP_PROT /** * generic_access_phys - generic implementation for iomem mmap access From patchwork Fri Aug 9 16:09:00 2024 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Peter Xu X-Patchwork-Id: 13758935 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from kanga.kvack.org (kanga.kvack.org [205.233.56.17]) by smtp.lore.kernel.org (Postfix) with ESMTP id 5A7DAC52D7C for ; Fri, 9 Aug 2024 16:09:47 +0000 (UTC) Received: by kanga.kvack.org (Postfix) id 940296B00AB; Fri, 9 Aug 2024 12:09:43 -0400 (EDT) Received: by kanga.kvack.org (Postfix, from userid 40) id 8EF966B00AC; Fri, 9 Aug 2024 12:09:43 -0400 (EDT) X-Delivered-To: int-list-linux-mm@kvack.org Received: by kanga.kvack.org (Postfix, from userid 63042) id 71E396B00AD; Fri, 9 Aug 2024 12:09:43 -0400 (EDT) X-Delivered-To: linux-mm@kvack.org Received: from relay.hostedemail.com (smtprelay0014.hostedemail.com [216.40.44.14]) by kanga.kvack.org (Postfix) with ESMTP id 4DB7D6B00AB for ; Fri, 9 Aug 2024 12:09:43 -0400 (EDT) Received: from smtpin05.hostedemail.com (a10.router.float.18 [10.200.18.1]) by unirelay02.hostedemail.com (Postfix) with ESMTP id 0CD6212117E for ; Fri, 9 Aug 2024 16:09:43 +0000 (UTC) X-FDA: 82433192646.05.CA91536 Received: from us-smtp-delivery-124.mimecast.com (us-smtp-delivery-124.mimecast.com [170.10.133.124]) by imf13.hostedemail.com (Postfix) with ESMTP id 00C042002B for ; Fri, 9 Aug 2024 16:09:40 +0000 (UTC) Authentication-Results: imf13.hostedemail.com; dkim=pass header.d=redhat.com header.s=mimecast20190719 header.b=YPhU4q7v; dmarc=pass (policy=none) header.from=redhat.com; spf=pass (imf13.hostedemail.com: domain of peterx@redhat.com designates 170.10.133.124 as permitted sender) smtp.mailfrom=peterx@redhat.com ARC-Seal: i=1; s=arc-20220608; d=hostedemail.com; t=1723219732; a=rsa-sha256; cv=none; b=gHpxqasm2At56Gptgx1nX62hxJkou6hR1Xkz4xxZ242DZ84nqU8Iy6yknCYh6zDupWEJ2M Uo+kqRB2bi4WJ963t17rx4qnJrFQSaQk0K95Nqry4fQG9GNJ+wJAVifRQG/bOy8UT5/9wx EY2tWGRHUTyuX0aZu/XwcwHTfpIQOhM= ARC-Authentication-Results: i=1; imf13.hostedemail.com; dkim=pass header.d=redhat.com header.s=mimecast20190719 header.b=YPhU4q7v; dmarc=pass (policy=none) header.from=redhat.com; spf=pass (imf13.hostedemail.com: domain of peterx@redhat.com designates 170.10.133.124 as permitted sender) smtp.mailfrom=peterx@redhat.com ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=hostedemail.com; s=arc-20220608; t=1723219732; h=from:from:sender:reply-to:subject:subject:date:date: message-id:message-id:to:to:cc:cc:mime-version:mime-version: content-type:content-type: content-transfer-encoding:content-transfer-encoding: in-reply-to:in-reply-to:references:references:dkim-signature; bh=XgMG3z25w/FU4W74BEAmfGyy8QhkBSj3TNUPa/vYriM=; b=FQKhaftHAnj/rJkn11t2iu77Lypbby+gtMjrzry2b9JFi53JMv9fPUpMbC4fKzUKnM3Jir N16FQ5it9g2koqQ5ZKlb3XNwm3GDVWhY7EMUnwUuRxotAeXKlPyKEhI1MAlSgkd+qE7Q4D 2c5ijApKfz+hJnQH9/R5umwvQC8QDmg= DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=redhat.com; s=mimecast20190719; t=1723219780; h=from:from:reply-to:subject:subject:date:date:message-id:message-id: to:to:cc:cc:mime-version:mime-version:content-type:content-type: content-transfer-encoding:content-transfer-encoding: in-reply-to:in-reply-to:references:references; bh=XgMG3z25w/FU4W74BEAmfGyy8QhkBSj3TNUPa/vYriM=; b=YPhU4q7vBmE+shzeD+qcwbh8e578X+hGr3vuwNeBm5qYqMxsMBnME+zKlxZN7JnCarnqL2 yTKc1TIdFJl2CKURUSHheXT3gTeIMavlJlMCO1vMsS7VtZKDZ4ZLtL6g1VrU9fF135a2D6 YhTZ1Ot52w7rd8XGzCds6VOcOi3H04U= Received: from mail-qv1-f71.google.com (mail-qv1-f71.google.com [209.85.219.71]) by relay.mimecast.com with ESMTP with STARTTLS (version=TLSv1.3, cipher=TLS_AES_256_GCM_SHA384) id us-mta-636-F6oSDlBEMLS1mxBgvznuag-1; Fri, 09 Aug 2024 12:09:39 -0400 X-MC-Unique: F6oSDlBEMLS1mxBgvznuag-1 Received: by mail-qv1-f71.google.com with SMTP id 6a1803df08f44-6b7a47a271cso6056356d6.2 for ; Fri, 09 Aug 2024 09:09:39 -0700 (PDT) X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20230601; t=1723219778; x=1723824578; h=content-transfer-encoding:mime-version:references:in-reply-to :message-id:date:subject:cc:to:from:x-gm-message-state:from:to:cc :subject:date:message-id:reply-to; bh=XgMG3z25w/FU4W74BEAmfGyy8QhkBSj3TNUPa/vYriM=; b=sc8wJxr1ZOA/mllAaTIYMC+6QPVZSnRxOa4Xveq4jzQrvj8cSUrHA8NYEOFXurZAHH 3SLlUV7GAzfALiwYlP9rwJX2PRQIbuiEC7hSu61sSsqtctZwte+XNl0zpoRGe1gNNjj/ qAaahfhH7ASQviQqVMP8wJGqEjVFdJpivtOqTS+ZJr40Uq/+CaIzeFJigFJhK7DKV/3n EB2Q0X+67+lCv09AQ9Y/2wL8zSL1Exd8Og7SpnFRelrR2kq7cYXV4D70qI5GCcY2N93I Oq4fbHbqdJ1LflI1LC7pGI3KkjVmG9FJnEMwgVexqyzAs5kA3CuETqtd36dGHMWQLkkn kSuA== X-Gm-Message-State: AOJu0YxB7YHCOBqELVhz8WCrjwKulP//lpKmDsvfM86sRuP76EQNhSQG YRjyn9L7PsTGX24WO4d2iY+Wy0U0aObcMEJjJCZigFDqS47npysBk3b6HWai7m8CdmFy0HrjHoX rQ2FioOLY2d8B5zMEbl5mddzKw/sgGW9tsjHUN0lyWTijhKtHpvhfqu0EMtm0awRWV/fPisQmlQ vqRu+uEyHBFMBVbXsGC3LHV/5G1f67yVwe X-Received: by 2002:ac8:5dd3:0:b0:447:dfa0:283e with SMTP id d75a77b69052e-4531251fe34mr11225581cf.1.1723219778067; Fri, 09 Aug 2024 09:09:38 -0700 (PDT) X-Google-Smtp-Source: AGHT+IE6MoN+Dd8Hdt5Y2cXuQIqSY6AztxI8VpmGorOFfK4YC2KlFlbKhz5m1DVHT1NSvONycoblfw== X-Received: by 2002:ac8:5dd3:0:b0:447:dfa0:283e with SMTP id d75a77b69052e-4531251fe34mr11225161cf.1.1723219777518; Fri, 09 Aug 2024 09:09:37 -0700 (PDT) Received: from x1n.redhat.com (pool-99-254-121-117.cpe.net.cable.rogers.com. [99.254.121.117]) by smtp.gmail.com with ESMTPSA id d75a77b69052e-451c870016csm22526741cf.19.2024.08.09.09.09.35 (version=TLS1_3 cipher=TLS_AES_256_GCM_SHA384 bits=256/256); Fri, 09 Aug 2024 09:09:36 -0700 (PDT) From: Peter Xu To: linux-mm@kvack.org, linux-kernel@vger.kernel.org Cc: Sean Christopherson , Oscar Salvador , Jason Gunthorpe , Axel Rasmussen , linux-arm-kernel@lists.infradead.org, x86@kernel.org, peterx@redhat.com, Will Deacon , Gavin Shan , Paolo Bonzini , Zi Yan , Andrew Morton , Catalin Marinas , Ingo Molnar , Alistair Popple , Borislav Petkov , David Hildenbrand , Thomas Gleixner , kvm@vger.kernel.org, Dave Hansen , Alex Williamson , Yan Zhao Subject: [PATCH 10/19] KVM: Use follow_pfnmap API Date: Fri, 9 Aug 2024 12:09:00 -0400 Message-ID: <20240809160909.1023470-11-peterx@redhat.com> X-Mailer: git-send-email 2.45.0 In-Reply-To: <20240809160909.1023470-1-peterx@redhat.com> References: <20240809160909.1023470-1-peterx@redhat.com> MIME-Version: 1.0 X-Mimecast-Spam-Score: 0 X-Mimecast-Originator: redhat.com X-Rspamd-Queue-Id: 00C042002B X-Rspam-User: X-Rspamd-Server: rspam05 X-Stat-Signature: 7omxtahtdda4b5mcjauq1mxns4ykybf9 X-HE-Tag: 1723219780-878016 X-HE-Meta: U2FsdGVkX1+V/4eHDNRs58NrRvd9/LoVMky0tRgxA1EBJdz2MLDg8XDsTPA72PXWVBksBRdcXMPrKGn6EnySHR7bmhOryRyS4VsbmBNEWPdoNxSOlCrh3ZOvd0tYenS9o79z5JQnhJdEXZgXouuyhhel92tUPFqhI5+jtlzJTlhZQzaT/DcjH5TVAc6iIiqgAlK20mMcMucVXpwDdHR0IXPxdP0G23Ek4c0nrkch8ak6ese0DXAuEZmXrkFEW2sr4tuIgOOpuOg6XnZhlnSq7cOgBjOOHLDOr+73ooDb4GYZrtBSfwUvFnLB4iw1GEbSJ7LxNbkti9muzUEp6CX+p8w2v1xHu+qhAgiTcmF3MYmcBni5zRhSeIDWokARMqxIlcrMe0GPpn+3VhaHiVGv7E7tppSdt2heLDAbZitT/igmd8wfIa0m6FCJNVMIw7gBh+5KKtj7ZqSmkcrODsTnLPD7EIrjwIj7iYHKXV/2lrNSLmt7SXG3Qlsrj1JbOnoJCB1zZTshmRuADjBruu0OBOyP7xzYzllr1s187jBtVRX8o3QE5OMFsbvN0/shDyRmprQp4cwSRDK2vpgFSmVFdixm+T0hm4Cp+gAXaR3HWPSJsuVqupUMDOY+5M+NmLUX0ap6eUcdGYTDWVACp9eLzguwSapNS3UlD5KV2dXh3RefcaHuCx/YsGHGh49lJkIAchPAV7muI4JM0RThMMtaxSHbz4T2xg/8p+EszkKkA8RX0VynHYUmh7nCMu6VUyiHG8hG/1fZmvNmC4HCW6EytJNGoqFcWOo8YqIniMGDo0w1tVJ83ewwePZZY+EimCUgRMm3S8XuL2P3rJTavvfJixiSdsE9b/a/Gg/WZI6duTHj2DwVDGokfouSXcJGh0oilVmdAZvaphYpB8SM41/it98ZSy59Oma20h7wxI4ihFRE8kKkrs47b2cxO+Dj7rm6BVvFkMxblZFqPiEuHJP 0yAM8s+R ytm+6AAGFw7CexoQG69yC+2sICpx2Vdyk2QrjXVm5WKn27UZdh5BJNgnxccvnHYiyTQ1Iw4Q1tVgibE8/h9lOfniAsJByUZCXOtUqV7acOGA90LzRQA41Dh5gQPVnUJN9mZq/bPQcHFBMpG1J6I4QC0TZ5jiEPga41KVnvKufu+tsiEuFCXz86KiocFVi1z/QYm55pLA4yDYANx1n/8n7BHXDY6+4wTZKl9q5NCfWA1OKL327fqhZ0q9+NHA8M4Z1pH2b9cEqSlVGb6GLNsLNisKj4DOc+IYIP4yh0exVoIHkaUm+wAn41wPsP0C+gDs7v43/Wf4IRizIRznBBVj+6Ac0s0KzXbKY5ZORnlHvApERR9wBa1dqaNdTU2zEitxJz0JKOYenPDGs9JHTMs7AABUW8TPF1fNu8u7L1cTou8BDOBdl9Wj1lFOurOlo60IjaridTQo+CE7BdJvrl1A9PFueY1chrlArTe9KxEkg5/SBeV8= X-Bogosity: Ham, tests=bogofilter, spamicity=0.000000, version=1.2.4 Sender: owner-linux-mm@kvack.org Precedence: bulk X-Loop: owner-majordomo@kvack.org List-ID: List-Subscribe: List-Unsubscribe: Use the new pfnmap API to allow huge MMIO mappings for VMs. The rest work is done perfectly on the other side (host_pfn_mapping_level()). Cc: Paolo Bonzini Cc: Sean Christopherson Signed-off-by: Peter Xu --- virt/kvm/kvm_main.c | 19 +++++++------------ 1 file changed, 7 insertions(+), 12 deletions(-) diff --git a/virt/kvm/kvm_main.c b/virt/kvm/kvm_main.c index d0788d0a72cc..9fb1c527a8e1 100644 --- a/virt/kvm/kvm_main.c +++ b/virt/kvm/kvm_main.c @@ -2862,13 +2862,11 @@ static int hva_to_pfn_remapped(struct vm_area_struct *vma, unsigned long addr, bool write_fault, bool *writable, kvm_pfn_t *p_pfn) { + struct follow_pfnmap_args args = { .vma = vma, .address = addr }; kvm_pfn_t pfn; - pte_t *ptep; - pte_t pte; - spinlock_t *ptl; int r; - r = follow_pte(vma, addr, &ptep, &ptl); + r = follow_pfnmap_start(&args); if (r) { /* * get_user_pages fails for VM_IO and VM_PFNMAP vmas and does @@ -2883,21 +2881,19 @@ static int hva_to_pfn_remapped(struct vm_area_struct *vma, if (r) return r; - r = follow_pte(vma, addr, &ptep, &ptl); + r = follow_pfnmap_start(&args); if (r) return r; } - pte = ptep_get(ptep); - - if (write_fault && !pte_write(pte)) { + if (write_fault && !args.writable) { pfn = KVM_PFN_ERR_RO_FAULT; goto out; } if (writable) - *writable = pte_write(pte); - pfn = pte_pfn(pte); + *writable = args.writable; + pfn = args.pfn; /* * Get a reference here because callers of *hva_to_pfn* and @@ -2918,9 +2914,8 @@ static int hva_to_pfn_remapped(struct vm_area_struct *vma, */ if (!kvm_try_get_pfn(pfn)) r = -EFAULT; - out: - pte_unmap_unlock(ptep, ptl); + follow_pfnmap_end(&args); *p_pfn = pfn; return r; From patchwork Fri Aug 9 16:09:01 2024 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Peter Xu X-Patchwork-Id: 13758936 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from kanga.kvack.org (kanga.kvack.org [205.233.56.17]) by smtp.lore.kernel.org (Postfix) with ESMTP id AC28DC3DA4A for ; Fri, 9 Aug 2024 16:09:50 +0000 (UTC) Received: by kanga.kvack.org (Postfix) id 3C39E6B00AD; Fri, 9 Aug 2024 12:09:46 -0400 (EDT) Received: by kanga.kvack.org (Postfix, from userid 40) id 372106B00AE; Fri, 9 Aug 2024 12:09:46 -0400 (EDT) X-Delivered-To: int-list-linux-mm@kvack.org Received: by kanga.kvack.org (Postfix, from userid 63042) id 12AB26B00AF; Fri, 9 Aug 2024 12:09:46 -0400 (EDT) X-Delivered-To: linux-mm@kvack.org Received: from relay.hostedemail.com (smtprelay0015.hostedemail.com [216.40.44.15]) by kanga.kvack.org (Postfix) with ESMTP id DF2A56B00AD for ; Fri, 9 Aug 2024 12:09:45 -0400 (EDT) Received: from smtpin02.hostedemail.com (a10.router.float.18 [10.200.18.1]) by unirelay09.hostedemail.com (Postfix) with ESMTP id 8DA338118C for ; Fri, 9 Aug 2024 16:09:45 +0000 (UTC) X-FDA: 82433192730.02.5DFC523 Received: from us-smtp-delivery-124.mimecast.com (us-smtp-delivery-124.mimecast.com [170.10.129.124]) by imf06.hostedemail.com (Postfix) with ESMTP id 764C0180003 for ; Fri, 9 Aug 2024 16:09:43 +0000 (UTC) Authentication-Results: imf06.hostedemail.com; dkim=pass header.d=redhat.com header.s=mimecast20190719 header.b=esEb7RxG; spf=pass (imf06.hostedemail.com: domain of peterx@redhat.com designates 170.10.129.124 as permitted sender) smtp.mailfrom=peterx@redhat.com; dmarc=pass (policy=none) header.from=redhat.com ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=hostedemail.com; s=arc-20220608; t=1723219718; h=from:from:sender:reply-to:subject:subject:date:date: message-id:message-id:to:to:cc:cc:mime-version:mime-version: content-type:content-type: content-transfer-encoding:content-transfer-encoding: in-reply-to:in-reply-to:references:references:dkim-signature; bh=kaY3EfeA+k09YWgS+MfGM8qkjlv58srLyvVau/9jXnU=; b=IfD0Iv4E1v9UtZn3h+Um7k1CQPZ82okpsplopbBKWj31m62MXiMBir1SmVMyYySXUKGYin Wu9nPctYzGkdCJs1yjnXgre0jeuYso956jbbhM78JM6EZ+w2GI6b4V3tFtBTVj3hh/hAw/ VmVr87BjkLLdDDgCeOQ1LkjE75XYDbU= ARC-Seal: i=1; s=arc-20220608; d=hostedemail.com; t=1723219718; a=rsa-sha256; cv=none; b=d9DpXbrd15Qic/lQ7Vb69RmaDwWNAwZe1/H69Ficp/ijqmDgKQ5fAE55P4MMgwvaM8j9F0 bkiCHYuGADxoDQydYv3SKseWY7VuEO4xl2g7k8+18RYxDXH2+m4Wlk2B0qEY71mpPHj76v Uc53nKJrAnvZ6imstd4SmnKEMgWHB5I= ARC-Authentication-Results: i=1; imf06.hostedemail.com; dkim=pass header.d=redhat.com header.s=mimecast20190719 header.b=esEb7RxG; spf=pass (imf06.hostedemail.com: domain of peterx@redhat.com designates 170.10.129.124 as permitted sender) smtp.mailfrom=peterx@redhat.com; dmarc=pass (policy=none) header.from=redhat.com DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=redhat.com; s=mimecast20190719; t=1723219782; h=from:from:reply-to:subject:subject:date:date:message-id:message-id: to:to:cc:cc:mime-version:mime-version:content-type:content-type: content-transfer-encoding:content-transfer-encoding: in-reply-to:in-reply-to:references:references; bh=kaY3EfeA+k09YWgS+MfGM8qkjlv58srLyvVau/9jXnU=; b=esEb7RxG1OyNGuXULMwbkHisXkJiTy4f7Gr1RZYOWuxPxZLfmXbbewqKJq1MGTnfMa5sEe nD4zRgC6zv0aiU9gHZ2y4KRvfU5cWwGs6wIr5pIUSotNGQZruG6lLVwmgIehOMnlncZX5y FeSpNhDEXQ2v5g5I49a2jONl8vGP9L8= Received: from mail-qt1-f198.google.com (mail-qt1-f198.google.com [209.85.160.198]) by relay.mimecast.com with ESMTP with STARTTLS (version=TLSv1.3, cipher=TLS_AES_256_GCM_SHA384) id us-mta-647-zq2mnokHM06WzSH82zDuTQ-1; Fri, 09 Aug 2024 12:09:41 -0400 X-MC-Unique: zq2mnokHM06WzSH82zDuTQ-1 Received: by mail-qt1-f198.google.com with SMTP id d75a77b69052e-44fea369811so4530291cf.0 for ; Fri, 09 Aug 2024 09:09:41 -0700 (PDT) X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20230601; t=1723219780; x=1723824580; h=content-transfer-encoding:mime-version:references:in-reply-to :message-id:date:subject:cc:to:from:x-gm-message-state:from:to:cc :subject:date:message-id:reply-to; bh=kaY3EfeA+k09YWgS+MfGM8qkjlv58srLyvVau/9jXnU=; b=ezmC9ddxeZbs5e79pP2h4qKC7eZOBzKka3pvoGjuVONDStFf+vNCmeh3w1puHFX2zZ ScYcEhBcYtA8tFv6qhpT728LT4Flx6kNAjwX0yDOJSODkyYbgU7q5sKHTMaF59+5RdR3 NxTLfCw7vt4KxaZk20yKggPHxMcyEKVgYgESauHpP2Y9KWgIpZyKfJ5vO8DV6OSAWdof TVY3bzSVPouiJm40JY9TzNuEqTHXWUnTNTKfhyO+kbaBjEBmIcWYqB+sGOgmH8NxUaiF TmqHlqqhOxXErN64wp91hoCXttt2u8M1WYzIaV14NYT17gjLQehmr6/QlOQGnq0adPIS YvSg== X-Gm-Message-State: AOJu0YzH9D/MwGy5dpCAHLL8tJ4MzXfTguf9fSf4mGUbDYu4FY4UvImj XCz+uHXSKhYcjlWYuaLPLqRdKsmn+YWmA4tzcT2jVQi9wOEyO0eOKz20oZhL9xIeW0X/2xZwvQl n72xsc5Bc7QZzzEgTC44VEdws6v2CIFuSqMdJBxpMKHUOTsJVCTBH2f6rMq6bhEVg+JBO/ODMQ4 rEG9YHLYUCA3U7/Ps8CYgNTBZemeZ9ammM X-Received: by 2002:ac8:5912:0:b0:450:efd:723 with SMTP id d75a77b69052e-453125552fbmr13163501cf.4.1723219780623; Fri, 09 Aug 2024 09:09:40 -0700 (PDT) X-Google-Smtp-Source: AGHT+IFtbIkutydCKQHHfrahTRflQkyMYGSh3zYrObmthmw14T659Ni251Xo72HIqIFQZU+hooLp2A== X-Received: by 2002:ac8:5912:0:b0:450:efd:723 with SMTP id d75a77b69052e-453125552fbmr13162951cf.4.1723219780083; Fri, 09 Aug 2024 09:09:40 -0700 (PDT) Received: from x1n.redhat.com (pool-99-254-121-117.cpe.net.cable.rogers.com. [99.254.121.117]) by smtp.gmail.com with ESMTPSA id d75a77b69052e-451c870016csm22526741cf.19.2024.08.09.09.09.37 (version=TLS1_3 cipher=TLS_AES_256_GCM_SHA384 bits=256/256); Fri, 09 Aug 2024 09:09:39 -0700 (PDT) From: Peter Xu To: linux-mm@kvack.org, linux-kernel@vger.kernel.org Cc: Sean Christopherson , Oscar Salvador , Jason Gunthorpe , Axel Rasmussen , linux-arm-kernel@lists.infradead.org, x86@kernel.org, peterx@redhat.com, Will Deacon , Gavin Shan , Paolo Bonzini , Zi Yan , Andrew Morton , Catalin Marinas , Ingo Molnar , Alistair Popple , Borislav Petkov , David Hildenbrand , Thomas Gleixner , kvm@vger.kernel.org, Dave Hansen , Alex Williamson , Yan Zhao , Niklas Schnelle , Gerald Schaefer , Heiko Carstens , Vasily Gorbik , Alexander Gordeev , Christian Borntraeger , Sven Schnelle , linux-s390@vger.kernel.org Subject: [PATCH 11/19] s390/pci_mmio: Use follow_pfnmap API Date: Fri, 9 Aug 2024 12:09:01 -0400 Message-ID: <20240809160909.1023470-12-peterx@redhat.com> X-Mailer: git-send-email 2.45.0 In-Reply-To: <20240809160909.1023470-1-peterx@redhat.com> References: <20240809160909.1023470-1-peterx@redhat.com> MIME-Version: 1.0 X-Mimecast-Spam-Score: 0 X-Mimecast-Originator: redhat.com X-Rspamd-Queue-Id: 764C0180003 X-Stat-Signature: ekiskewyapkkom7p8dtoip9ybbqzyyf3 X-Rspamd-Server: rspam09 X-Rspam-User: X-HE-Tag: 1723219783-990954 X-HE-Meta: U2FsdGVkX18Idoq6VD41Poi7zJCIQoA2uxXcUUW8OrVJDbEZXwCStWqBURhPAvycyeWoFAhHjpmFed/SWjAwH/Yqe6kGf4s+EWXwJdcD+uL4fSkS4KmiLyiKRAY2nAkKbfOZVzVJvwiJbHJVO2oOqlyW6MqxymRLq2G2hjSEn6VEPHrUWNPC/rpYTRY2LREVXHPmn1TDI8YOcB3/vXPG/yV8pO7n4TWGgInZdgdvBuNL2NkqFWbmxCFtRySzlZwjNpSe/Zbxc2qNgkz34lzkDZNxXfXsF6LLCNC9rHXWfI0WElEpi7T3bTN+Q+lbCfjDHXROCDNvZ44L9TOF0wJaDLHcOtG8tIlE4dy47KjdSiU+zK4PbC8FobwvwX2uYa6iDL3nieRP5mW5D1Oz2VZ+zInu+0bj4YDEFtBW1LzFj7MJfYGrjqxVyWTOxckM/WOa2RFy/c2d5oFOFyylApOHFTJRy3YPamB6LkQDSGtX6Qc3t87W4Bon1mMbQSt8ufzytqBMadDPsVWLpaM9r8DwuIqJyw1hMh6m9P10cYyE3FuSi6NVn1PoMEAdZ85DTZ8T8nNcn5cIYesyEXFo/tXmHClQyfkCUdqMYDshQ3i1ws/SNGPuCsuFSwHVacfMRE7PqYK6rCqxox1afprTfnxh2nC3mZkd3ALCMYtm/eCI7HgsomefbHfCwZ2jlWfQRqG4TR9DY/ngd0jA3MMqy0afjI3X+rNx8RcZEWV9DyF3dGwZvw0ydjURtsNra5NjY6aOhPLhcmaxkS9/TiGfYkVW+zlZwHjbfOEIFM1LZ1ng7NOCPols3vG4LLgCTC/zauUDPUdrqFqxy/Hbk0XzUAU12BzfotzDFjWvx0U3N1E4Gc+SSbb3SWq/l2Zyq1IT6n/rKwJlgoqHv1haxNuOl3/+tPhaVKBHI8usKUUq7krbf184/swmafU6rItRd25GCAq5F7Qn6IVjKzsJodGrjBY 9SCp8V0p gFSDl+IHYd6vZv+/rro4c9vIv4dzDr9+l/BOBpLnWjFPJ8r124SALPJr3ZCRey2DlBVCAAIgszU3JZQxeWkvNYcqu5Dmgcgq9bigaJzqnlcnECIoEUsw73FEu2hxlE/9srmACOPWRwm4zq9ToGpYsBSMmLBUkcH7atCwHWqAyQ1WEl+ehR6PshmJqAxq3yAQWJXRxLsU6TyMeDk/a0M7BREOBvNfRYtMLHiMAPnYZLaeru/nu9fvBdq0J3NMeEx8IV5Vfsr/DFDpCbv62CWYH1dcvhGl5FrK6ox9Wd7jYRUZcPPvvoPGroU51VW+9Eh0qGPydvpkq6Hah8Rm0oYWkCSi4yT+I/Lpfrjkj0WuctTV/9mpXr57UpTzJXyWAKuUglBgb/7Z7tnODDCpO5+/V4Pttp5nsjDOxGAvfcV5dGiEIk1EKXRHO9YBMT8MJSevaQ2J8xv0ZexoBKSAYaGEK5id8EmBXukro9d7qmDN3axAWp6elviZ4SJP6cvmh/gHcv6p8yUHZ/6GfPGCJaPiFGtReLrhabmXTOCsudbnupEPbbYS8ugh5RvCTpU55MYxZBVo7HdPBWWFZVZX443wr23AxBA== X-Bogosity: Ham, tests=bogofilter, spamicity=0.000000, version=1.2.4 Sender: owner-linux-mm@kvack.org Precedence: bulk X-Loop: owner-majordomo@kvack.org List-ID: List-Subscribe: List-Unsubscribe: Use the new API that can understand huge pfn mappings. Cc: Niklas Schnelle Cc: Gerald Schaefer Cc: Heiko Carstens Cc: Vasily Gorbik Cc: Alexander Gordeev Cc: Christian Borntraeger Cc: Sven Schnelle Cc: linux-s390@vger.kernel.org Signed-off-by: Peter Xu --- arch/s390/pci/pci_mmio.c | 22 ++++++++++++---------- 1 file changed, 12 insertions(+), 10 deletions(-) diff --git a/arch/s390/pci/pci_mmio.c b/arch/s390/pci/pci_mmio.c index 5398729bfe1b..de5c0b389a3e 100644 --- a/arch/s390/pci/pci_mmio.c +++ b/arch/s390/pci/pci_mmio.c @@ -118,12 +118,11 @@ static inline int __memcpy_toio_inuser(void __iomem *dst, SYSCALL_DEFINE3(s390_pci_mmio_write, unsigned long, mmio_addr, const void __user *, user_buffer, size_t, length) { + struct follow_pfnmap_args args = { }; u8 local_buf[64]; void __iomem *io_addr; void *buf; struct vm_area_struct *vma; - pte_t *ptep; - spinlock_t *ptl; long ret; if (!zpci_is_enabled()) @@ -169,11 +168,13 @@ SYSCALL_DEFINE3(s390_pci_mmio_write, unsigned long, mmio_addr, if (!(vma->vm_flags & VM_WRITE)) goto out_unlock_mmap; - ret = follow_pte(vma, mmio_addr, &ptep, &ptl); + args.address = mmio_addr; + args.vma = vma; + ret = follow_pfnmap_start(&args); if (ret) goto out_unlock_mmap; - io_addr = (void __iomem *)((pte_pfn(*ptep) << PAGE_SHIFT) | + io_addr = (void __iomem *)((args.pfn << PAGE_SHIFT) | (mmio_addr & ~PAGE_MASK)); if ((unsigned long) io_addr < ZPCI_IOMAP_ADDR_BASE) @@ -181,7 +182,7 @@ SYSCALL_DEFINE3(s390_pci_mmio_write, unsigned long, mmio_addr, ret = zpci_memcpy_toio(io_addr, buf, length); out_unlock_pt: - pte_unmap_unlock(ptep, ptl); + follow_pfnmap_end(&args); out_unlock_mmap: mmap_read_unlock(current->mm); out_free: @@ -260,12 +261,11 @@ static inline int __memcpy_fromio_inuser(void __user *dst, SYSCALL_DEFINE3(s390_pci_mmio_read, unsigned long, mmio_addr, void __user *, user_buffer, size_t, length) { + struct follow_pfnmap_args args = { }; u8 local_buf[64]; void __iomem *io_addr; void *buf; struct vm_area_struct *vma; - pte_t *ptep; - spinlock_t *ptl; long ret; if (!zpci_is_enabled()) @@ -308,11 +308,13 @@ SYSCALL_DEFINE3(s390_pci_mmio_read, unsigned long, mmio_addr, if (!(vma->vm_flags & VM_WRITE)) goto out_unlock_mmap; - ret = follow_pte(vma, mmio_addr, &ptep, &ptl); + args.vma = vma; + args.address = mmio_addr; + ret = follow_pfnmap_start(&args); if (ret) goto out_unlock_mmap; - io_addr = (void __iomem *)((pte_pfn(*ptep) << PAGE_SHIFT) | + io_addr = (void __iomem *)((args.pfn << PAGE_SHIFT) | (mmio_addr & ~PAGE_MASK)); if ((unsigned long) io_addr < ZPCI_IOMAP_ADDR_BASE) { @@ -322,7 +324,7 @@ SYSCALL_DEFINE3(s390_pci_mmio_read, unsigned long, mmio_addr, ret = zpci_memcpy_fromio(buf, io_addr, length); out_unlock_pt: - pte_unmap_unlock(ptep, ptl); + follow_pfnmap_end(&args); out_unlock_mmap: mmap_read_unlock(current->mm); From patchwork Fri Aug 9 16:09:02 2024 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Peter Xu X-Patchwork-Id: 13758937 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from kanga.kvack.org (kanga.kvack.org [205.233.56.17]) by smtp.lore.kernel.org (Postfix) with ESMTP id 2E53BC3DA4A for ; Fri, 9 Aug 2024 16:09:54 +0000 (UTC) Received: by kanga.kvack.org (Postfix) id 7E26A6B0085; Fri, 9 Aug 2024 12:09:48 -0400 (EDT) Received: by kanga.kvack.org (Postfix, from userid 40) id 792866B008A; Fri, 9 Aug 2024 12:09:48 -0400 (EDT) X-Delivered-To: int-list-linux-mm@kvack.org Received: by kanga.kvack.org (Postfix, from userid 63042) id 5BE496B008C; Fri, 9 Aug 2024 12:09:48 -0400 (EDT) X-Delivered-To: linux-mm@kvack.org Received: from relay.hostedemail.com (smtprelay0013.hostedemail.com [216.40.44.13]) by kanga.kvack.org (Postfix) with ESMTP id 368E16B0085 for ; Fri, 9 Aug 2024 12:09:48 -0400 (EDT) Received: from smtpin13.hostedemail.com (a10.router.float.18 [10.200.18.1]) by unirelay06.hostedemail.com (Postfix) with ESMTP id ED7B3A7B10 for ; Fri, 9 Aug 2024 16:09:47 +0000 (UTC) X-FDA: 82433192814.13.970C691 Received: from us-smtp-delivery-124.mimecast.com (us-smtp-delivery-124.mimecast.com [170.10.129.124]) by imf17.hostedemail.com (Postfix) with ESMTP id E97C340034 for ; Fri, 9 Aug 2024 16:09:45 +0000 (UTC) Authentication-Results: imf17.hostedemail.com; dkim=pass header.d=redhat.com header.s=mimecast20190719 header.b=M4WnckVP; spf=pass (imf17.hostedemail.com: domain of peterx@redhat.com designates 170.10.129.124 as permitted sender) smtp.mailfrom=peterx@redhat.com; dmarc=pass (policy=none) header.from=redhat.com ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=hostedemail.com; s=arc-20220608; t=1723219719; h=from:from:sender:reply-to:subject:subject:date:date: message-id:message-id:to:to:cc:cc:mime-version:mime-version: content-type:content-type: content-transfer-encoding:content-transfer-encoding: in-reply-to:in-reply-to:references:references:dkim-signature; bh=dHUdq0ZI2fvSzmSbu56kn/TFI4mY4GVPslnR6ZzQETQ=; b=0XsdGpyJefWZlPDE/A8xVxHge7ecHZzj9oF68WkWbknNOmO0wOqrdtb4B2F6T8UuX4QjjJ 6do1yVJlMdKi8zZufINnDQUOc4PR7ybz+kSpNl+tCn0qF23L4AGqJYJd683zycBpn9ZYJS k2JSICHq0BCRIiryfxwlwprOv1pTxSk= ARC-Seal: i=1; s=arc-20220608; d=hostedemail.com; t=1723219719; a=rsa-sha256; cv=none; b=5/QZDtFxF0zOMGDPZd8GmLNlHToM7kVt0GO3fOnOYE9nayHCLryI21hdb8bue2QIkvd5pN Kn87+esDXQE+psEFKdjgEeMY887yxs/GVcaUsJmFqVB8coVO6/ZCqth71oj/BDefWzdziH UGIaSdKSV9mRSwIRPXp98QzZ52cin9g= ARC-Authentication-Results: i=1; imf17.hostedemail.com; dkim=pass header.d=redhat.com header.s=mimecast20190719 header.b=M4WnckVP; spf=pass (imf17.hostedemail.com: domain of peterx@redhat.com designates 170.10.129.124 as permitted sender) smtp.mailfrom=peterx@redhat.com; dmarc=pass (policy=none) header.from=redhat.com DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=redhat.com; s=mimecast20190719; t=1723219785; h=from:from:reply-to:subject:subject:date:date:message-id:message-id: to:to:cc:cc:mime-version:mime-version:content-type:content-type: content-transfer-encoding:content-transfer-encoding: in-reply-to:in-reply-to:references:references; bh=dHUdq0ZI2fvSzmSbu56kn/TFI4mY4GVPslnR6ZzQETQ=; b=M4WnckVPRZ/ed4FeZ2sm8w1yEUZCH8/WLlOIC647Sjaa+FfBBX72x2WRm1DZJHZnFUDpi6 4LJVRasHq7NwEzmVNMcZxO0Vlcvq0hAeX7ECIt2TkX8wrbuFQZkKZrCLxS5W+KD5goc/sC DZnt1GzIxUpWV29dpgcS2s2mk0vDDgI= Received: from mail-qv1-f70.google.com (mail-qv1-f70.google.com [209.85.219.70]) by relay.mimecast.com with ESMTP with STARTTLS (version=TLSv1.3, cipher=TLS_AES_256_GCM_SHA384) id us-mta-137-GzvSfFNLMHGBRB4U3x0ypA-1; Fri, 09 Aug 2024 12:09:44 -0400 X-MC-Unique: GzvSfFNLMHGBRB4U3x0ypA-1 Received: by mail-qv1-f70.google.com with SMTP id 6a1803df08f44-6b79c5c972eso2605496d6.1 for ; Fri, 09 Aug 2024 09:09:44 -0700 (PDT) X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20230601; t=1723219783; x=1723824583; h=content-transfer-encoding:mime-version:references:in-reply-to :message-id:date:subject:cc:to:from:x-gm-message-state:from:to:cc :subject:date:message-id:reply-to; bh=dHUdq0ZI2fvSzmSbu56kn/TFI4mY4GVPslnR6ZzQETQ=; b=PITtNq4dGvISeRKQulLxAk3vPduiBzhdxAiud4awygS+IR6plFJdii9EFC5HopHpNf xwas6TvWehoIkf/2Ah2bousHvmUyVCjAzvnPqmmiTfwhdqggau07QJTSYBVLIdvyGnbe whFDSPeEo9fxHaxyyIsYU4t0+s/HpjbHV47H24Sgtr1XRYSCyDkXF4+bBT1oqM3iBLNR zw1/RR5EF4CzglATukvccEE1f7381GP+kN4TQq9FENL7eN67lIdD2ia9x7ydi0YeWUFk QYMJ/ZYd5lGL2vKA/IFrwh9Deg8vuuW5sUf+xZ12L5wPhM4YlsHIMhU7bca8rmOUpPe8 6wag== X-Gm-Message-State: AOJu0Ywq5QjX1+J7AIQf8qA3Tv9VuXwWMnsF21yZVQnaD5D0TO+taHVF j512Yary/tbb6bHGzwJU38svS+ezn5+oeoKWsUFu1iVPBscm5KvgXRNlmT3Tdn1Pmg6z6xjw12w PwDIvOhfUgPWlYorF1kxlVo5G5etZY/pAHmJT8d6xwa9Jd46OQ4oSw0g7HLXalEVjlWT6J0YUMW Nu6baO9ilJbjwNBV4xWm2dfR+XBBPo2t8r X-Received: by 2002:a05:620a:460d:b0:7a3:5e3f:cbf4 with SMTP id af79cd13be357-7a4c18467b6mr127416985a.8.1723219782888; Fri, 09 Aug 2024 09:09:42 -0700 (PDT) X-Google-Smtp-Source: AGHT+IFqMMArTR19FgmAq+7tz35tycS6tSaD4MMjthIpuW8WNcff+L3Yj5M2c5Huoz5dZxDipeszqw== X-Received: by 2002:a05:620a:460d:b0:7a3:5e3f:cbf4 with SMTP id af79cd13be357-7a4c18467b6mr127412385a.8.1723219782335; Fri, 09 Aug 2024 09:09:42 -0700 (PDT) Received: from x1n.redhat.com (pool-99-254-121-117.cpe.net.cable.rogers.com. [99.254.121.117]) by smtp.gmail.com with ESMTPSA id d75a77b69052e-451c870016csm22526741cf.19.2024.08.09.09.09.40 (version=TLS1_3 cipher=TLS_AES_256_GCM_SHA384 bits=256/256); Fri, 09 Aug 2024 09:09:41 -0700 (PDT) From: Peter Xu To: linux-mm@kvack.org, linux-kernel@vger.kernel.org Cc: Sean Christopherson , Oscar Salvador , Jason Gunthorpe , Axel Rasmussen , linux-arm-kernel@lists.infradead.org, x86@kernel.org, peterx@redhat.com, Will Deacon , Gavin Shan , Paolo Bonzini , Zi Yan , Andrew Morton , Catalin Marinas , Ingo Molnar , Alistair Popple , Borislav Petkov , David Hildenbrand , Thomas Gleixner , kvm@vger.kernel.org, Dave Hansen , Alex Williamson , Yan Zhao Subject: [PATCH 12/19] mm/x86/pat: Use the new follow_pfnmap API Date: Fri, 9 Aug 2024 12:09:02 -0400 Message-ID: <20240809160909.1023470-13-peterx@redhat.com> X-Mailer: git-send-email 2.45.0 In-Reply-To: <20240809160909.1023470-1-peterx@redhat.com> References: <20240809160909.1023470-1-peterx@redhat.com> MIME-Version: 1.0 X-Mimecast-Spam-Score: 0 X-Mimecast-Originator: redhat.com X-Rspam-User: X-Rspamd-Server: rspam04 X-Rspamd-Queue-Id: E97C340034 X-Stat-Signature: uy8s16aehcncw3ufk66xn89eog7x5dbx X-HE-Tag: 1723219785-236205 X-HE-Meta: U2FsdGVkX184VJbSR5HOJDytVQ/hmCfHB+WvBI7HuCWlWMQEGO1bWL4iBGNGkcY3J23oBo8RF1GQfI3Xo6fReYaws5Jv5FxqMFAeghd0ClUJb7khfXwvhsVo8qxOoqxnhROW/Gw7622AsaHG5Q5UrKfyRBx9/krdmS3ZQ7fqeY3bPbZf/8pJn+Z9Wd/NFNUJL8sgCTSRz/dP17fuhfS+wjC699KdnUlru0/srxMdcZzUDMyoGJf2+d2b7WSqxi7Hn2W8opCyFqVPgEWVDpEqaoGqjlMD6To8ehSMW8e6zpPmRYrXeqlYkokBDJ+IT+s8ahTq5m7XB/b0+EKteREEvlLsYHMbhVid3PM8MSxj5yp6de6ST3TlT1oD4kgRMoavcV32uJUWh0sY2pOqIbZIx5CQdK/zI5SAEjH3w7/PKZ76fX/9TgxQOE/OQTLu1YymbuG4s8k4qIYsA2jF/oNLOcrWTWp/BW9gDY8slerYHpY9gktX/qGR9bXbv4dio6bRu2YfSn88oGoJnkwgcm9iNIlskkytP925swqB/Z9r4bmMyz/I+7DhpNBO67FgodROidxQsUA/HgKNQowkrz0719/s2pTft7JO4e1BsjD61W6iniUY8yW5MxMYUwnNKUrMnpxUdS9AwYLXGYKGCW2Y2RO/0Ck/nFJf3sj1rhepYncdWBEsw4t1brL+tioJGFUhHgZMGPZBolDlMu+sVtQVGv/xvmWkS0ItJ9QnA/RcBn1OLnejHyJ08SuFpFLXpwAFVYHGc8maUv2TUyo4rh8a29yftFcDgwveh6rtt1YfqRX2N+pyFA6FDuhgeaEy3WoogvkQc6fGlOK1Eq1nsR7L8QEnlMl0enuHAXnYV+ZYVZGOasQPNgLIanFH1UgseGivOe3ywZRbE7JL5GUwimCz95VfQJ16CdK3Mg4WHG87AGMYsRKxi9nTurshuJIpWGpM9yn89angYL1SrJTNKgh 4Ifjl92o 65oMhPZP4jUDpCJpHVIrDBbxFf6GztWQ5agPCj502jCAJjh4aRtMSDvG0PdySkDxGVMbKOntIslk1BzbKMJwccK6Rr1Nr8jLB+di8bsZUmRxWP6eHb7jDNxKu77PxAxUb4B9h6CpPgl76qpKU/mcUbrfMxlurHkyPg4jjuJfS4tue0LmoyHJk1IU3p72LeRHJE72gBtUlzIQdVSeWDI1yIfXJyAe5C+WNY5Oc2UHeGkY6mEFvMUn1OXAmydEw6n/0snFjaTgQsB1tiZTjThRnwebWrCCKjAUdpJ0ihZU1fzfxXma2kFoSqMq2SlbcyQzo4N7I7itROwpcCTZZvK9yHydRsxy24r+PwsAq08lA+Z8ynGQpz/mREqBv+LX4TsK5MEqPfRSxQyuk3pE0gqx9ec726qNFfP5mznlrDh9nWSmKaMc= X-Bogosity: Ham, tests=bogofilter, spamicity=0.000000, version=1.2.4 Sender: owner-linux-mm@kvack.org Precedence: bulk X-Loop: owner-majordomo@kvack.org List-ID: List-Subscribe: List-Unsubscribe: Use the new API that can understand huge pfn mappings. Cc: x86@kernel.org Cc: Thomas Gleixner Cc: Ingo Molnar Cc: Borislav Petkov Cc: Dave Hansen Signed-off-by: Peter Xu --- arch/x86/mm/pat/memtype.c | 17 +++++++---------- 1 file changed, 7 insertions(+), 10 deletions(-) diff --git a/arch/x86/mm/pat/memtype.c b/arch/x86/mm/pat/memtype.c index bdc2a240c2aa..fd210b362a04 100644 --- a/arch/x86/mm/pat/memtype.c +++ b/arch/x86/mm/pat/memtype.c @@ -951,23 +951,20 @@ static void free_pfn_range(u64 paddr, unsigned long size) static int follow_phys(struct vm_area_struct *vma, unsigned long *prot, resource_size_t *phys) { - pte_t *ptep, pte; - spinlock_t *ptl; + struct follow_pfnmap_args args = { .vma = vma, .address = vma->vm_start }; - if (follow_pte(vma, vma->vm_start, &ptep, &ptl)) + if (follow_pfnmap_start(&args)) return -EINVAL; - pte = ptep_get(ptep); - /* Never return PFNs of anon folios in COW mappings. */ - if (vm_normal_folio(vma, vma->vm_start, pte)) { - pte_unmap_unlock(ptep, ptl); + if (!args.special) { + follow_pfnmap_end(&args); return -EINVAL; } - *prot = pgprot_val(pte_pgprot(pte)); - *phys = (resource_size_t)pte_pfn(pte) << PAGE_SHIFT; - pte_unmap_unlock(ptep, ptl); + *prot = pgprot_val(args.pgprot); + *phys = (resource_size_t)args.pfn << PAGE_SHIFT; + follow_pfnmap_end(&args); return 0; } From patchwork Fri Aug 9 16:09:03 2024 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Peter Xu X-Patchwork-Id: 13758938 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from kanga.kvack.org (kanga.kvack.org [205.233.56.17]) by smtp.lore.kernel.org (Postfix) with ESMTP id 7AA27C3DA4A for ; Fri, 9 Aug 2024 16:09:57 +0000 (UTC) Received: by kanga.kvack.org (Postfix) id 682496B008C; Fri, 9 Aug 2024 12:09:50 -0400 (EDT) Received: by kanga.kvack.org (Postfix, from userid 40) id 5E3956B00AF; Fri, 9 Aug 2024 12:09:50 -0400 (EDT) X-Delivered-To: int-list-linux-mm@kvack.org Received: by kanga.kvack.org (Postfix, from userid 63042) id 3EAF06B00B0; Fri, 9 Aug 2024 12:09:50 -0400 (EDT) X-Delivered-To: linux-mm@kvack.org Received: from relay.hostedemail.com (smtprelay0010.hostedemail.com [216.40.44.10]) by kanga.kvack.org (Postfix) with ESMTP id 1C8FF6B008C for ; Fri, 9 Aug 2024 12:09:50 -0400 (EDT) Received: from smtpin19.hostedemail.com (a10.router.float.18 [10.200.18.1]) by unirelay08.hostedemail.com (Postfix) with ESMTP id BA37314113B for ; Fri, 9 Aug 2024 16:09:49 +0000 (UTC) X-FDA: 82433192898.19.2E6484D Received: from us-smtp-delivery-124.mimecast.com (us-smtp-delivery-124.mimecast.com [170.10.129.124]) by imf23.hostedemail.com (Postfix) with ESMTP id 7D5BA140016 for ; Fri, 9 Aug 2024 16:09:47 +0000 (UTC) Authentication-Results: imf23.hostedemail.com; dkim=pass header.d=redhat.com header.s=mimecast20190719 header.b=O60UKNzU; dmarc=pass (policy=none) header.from=redhat.com; spf=pass (imf23.hostedemail.com: domain of peterx@redhat.com designates 170.10.129.124 as permitted sender) smtp.mailfrom=peterx@redhat.com ARC-Seal: i=1; s=arc-20220608; d=hostedemail.com; t=1723219777; a=rsa-sha256; cv=none; b=GDEXrfyLRFZ/3tWUo9nmvHtju+p0P0l7bCDJJIaJYsO2wIEh0eUFIs0nBj1q0rS5g+g7l5 tD9tAzBXmSNwjPj9MypYlLFryMrZo6cniYebDOfFasOatlU3j/R5h9wXzNgNIFjnx+d4AG gYuKg9vObKVmhCql7UjMbCUuB7f4T/E= ARC-Authentication-Results: i=1; imf23.hostedemail.com; dkim=pass header.d=redhat.com header.s=mimecast20190719 header.b=O60UKNzU; dmarc=pass (policy=none) header.from=redhat.com; spf=pass (imf23.hostedemail.com: domain of peterx@redhat.com designates 170.10.129.124 as permitted sender) smtp.mailfrom=peterx@redhat.com ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=hostedemail.com; s=arc-20220608; t=1723219777; h=from:from:sender:reply-to:subject:subject:date:date: message-id:message-id:to:to:cc:cc:mime-version:mime-version: content-type:content-type: content-transfer-encoding:content-transfer-encoding: in-reply-to:in-reply-to:references:references:dkim-signature; bh=HLBqit68L/U2mdwYaFkydFK3txHa+LkmL8n2YTQXnnw=; b=bR4+gyt468CIVyp9BL8/u+SwJUWJLW0IQBcloPyRyNG4jAPDDPd1x7b+1CFqoD5d83i6QG EFFSYoHDe5ZbjJp3s6tYZjU5J6zWsr3tcKWk+mEnH4K0rkAJ4KbmRqArzO2ZgeM9rw8zNb oLGY26OIgN7qvJDugF8sIMofMkArRDY= DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=redhat.com; s=mimecast20190719; t=1723219786; h=from:from:reply-to:subject:subject:date:date:message-id:message-id: to:to:cc:cc:mime-version:mime-version:content-type:content-type: content-transfer-encoding:content-transfer-encoding: in-reply-to:in-reply-to:references:references; bh=HLBqit68L/U2mdwYaFkydFK3txHa+LkmL8n2YTQXnnw=; b=O60UKNzUmBZ03y1iKh7v9QxiN8SPmfZ0APY24e/9nMjRxsiRiG8IaSFEGOJSr7eHCpwpKe 2+89+RpXi55Tpi4vxaixMF0e2gZKGkBrpYo5egTMQnSxUP3ezoIksxcBFqHn7DDNKlHmtY JzGuS0ZKWptjDxYdw4/mAM41C0mohzM= Received: from mail-qt1-f197.google.com (mail-qt1-f197.google.com [209.85.160.197]) by relay.mimecast.com with ESMTP with STARTTLS (version=TLSv1.3, cipher=TLS_AES_256_GCM_SHA384) id us-mta-557-USvTRhHsP5GrU_EuTZT-8w-1; Fri, 09 Aug 2024 12:09:45 -0400 X-MC-Unique: USvTRhHsP5GrU_EuTZT-8w-1 Received: by mail-qt1-f197.google.com with SMTP id d75a77b69052e-45009e27b83so4905811cf.1 for ; Fri, 09 Aug 2024 09:09:45 -0700 (PDT) X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20230601; t=1723219785; x=1723824585; h=content-transfer-encoding:mime-version:references:in-reply-to :message-id:date:subject:cc:to:from:x-gm-message-state:from:to:cc :subject:date:message-id:reply-to; bh=HLBqit68L/U2mdwYaFkydFK3txHa+LkmL8n2YTQXnnw=; b=Z05lWvR4Ge1G2+XgT19/nawSYL6BCn86Ofi2lSTETeBYuzTmzd6gDO5LzpugCvJoZM PZn8Jm9yVfAlx4TmNmd3FC2Bu2iBBQ5R+dvmd/F9tQ6qZrpqYcDwF4BrNBA9Z39VelRj jUYMx0BAgY5EvmwiSirhI7DR2Fnqrt6UqcVrFAlaWmVz+zTF4qgt/+a8AcAUW1PLR62x 5LLg+VqgVjkFUbvGJD9j9fSUhE51NXIb9WlLKLwCgdd5PfIs+sVJOOj4e21RcWFdctF1 0Z2VhxayhVbh8nJrd1MkV/2LFLOqAULifLP2ma+K2so9vyx1yTpT2jyxatscnmIuAaBd n3Tg== X-Gm-Message-State: AOJu0Yx7dN50o2Y6Nn7iYwzDrCp4JshRux+iS2XWh6KkoWRJfywJqHeo 8QgjHkRlNAGm4H5VMwkX+ZOuHSwVxOm0W8n277p+Ap6HIhSAx4Mpu2PB02G+ReebPYlTAYsOV7B Vg9iJwYYYaewf3yiS5D5IaT42fV0d7LJRFCF97rAM23QTeRzMsBA1mX9Iy4gEne6qgeTqVN6CE5 NCngCauN5mLJHsqEOis+IGzScf+CmfQEHY X-Received: by 2002:ac8:5f10:0:b0:450:349:1170 with SMTP id d75a77b69052e-4531255c62dmr14745591cf.6.1723219784710; Fri, 09 Aug 2024 09:09:44 -0700 (PDT) X-Google-Smtp-Source: AGHT+IEFbC3kPrc0fRPufwvcIMbfwNKbDcEyDmf8f6qiGpsAJzwvtAb4AIbgx0osGWWLWwEBOAE+Pw== X-Received: by 2002:ac8:5f10:0:b0:450:349:1170 with SMTP id d75a77b69052e-4531255c62dmr14745171cf.6.1723219784295; Fri, 09 Aug 2024 09:09:44 -0700 (PDT) Received: from x1n.redhat.com (pool-99-254-121-117.cpe.net.cable.rogers.com. [99.254.121.117]) by smtp.gmail.com with ESMTPSA id d75a77b69052e-451c870016csm22526741cf.19.2024.08.09.09.09.42 (version=TLS1_3 cipher=TLS_AES_256_GCM_SHA384 bits=256/256); Fri, 09 Aug 2024 09:09:43 -0700 (PDT) From: Peter Xu To: linux-mm@kvack.org, linux-kernel@vger.kernel.org Cc: Sean Christopherson , Oscar Salvador , Jason Gunthorpe , Axel Rasmussen , linux-arm-kernel@lists.infradead.org, x86@kernel.org, peterx@redhat.com, Will Deacon , Gavin Shan , Paolo Bonzini , Zi Yan , Andrew Morton , Catalin Marinas , Ingo Molnar , Alistair Popple , Borislav Petkov , David Hildenbrand , Thomas Gleixner , kvm@vger.kernel.org, Dave Hansen , Alex Williamson , Yan Zhao Subject: [PATCH 13/19] vfio: Use the new follow_pfnmap API Date: Fri, 9 Aug 2024 12:09:03 -0400 Message-ID: <20240809160909.1023470-14-peterx@redhat.com> X-Mailer: git-send-email 2.45.0 In-Reply-To: <20240809160909.1023470-1-peterx@redhat.com> References: <20240809160909.1023470-1-peterx@redhat.com> MIME-Version: 1.0 X-Mimecast-Spam-Score: 0 X-Mimecast-Originator: redhat.com X-Rspam-User: X-Rspamd-Queue-Id: 7D5BA140016 X-Rspamd-Server: rspam01 X-Stat-Signature: fruc3t48geg96drthbeticu8smn9dqxr X-HE-Tag: 1723219787-579020 X-HE-Meta: U2FsdGVkX180YjtvONE/l0bPVFvx7hiw7VbI29tqu41exYLNiG5JM6HLb9bb9IrIYouoiEulZzu9S24fJUUB6dMe+axrngSa2gTnDrc03hcTgPBiqPnVkHcYA8voeNL0xd0XLuY8iSOUAG7dwyOcVuVFrV6oyy61qHo/ZwYvPbU3WX2MqC8cFROUugG8HEBpXSxQHJP4r811m3N+QNkcPXVr64FG1PysjqRq+AdD9GyZrLK1yjXaMOROJS5YWpNS+SCpZyzpYV2slZOpUrOhfx4LWGoYwk5T0w2Cyf+zjcQZaFXEkLdTQCUewUgqHc3BQ3VrRX0c9BlcJCNZ/QNpq2wjuurFcY72BvpwMfutux1c1wP+t2Yo2L2ftsFdt7K4u2AgnIIBbmc0Pbiac4UfDQf2iHqYureliSfNihsK6fSj/0zPpjdJAha7224vdMCs+4wNeMvuSk7LAk9yiQeZlGgzPoH1Igov9n41TWb7RoiTG4QnfehKclKKmMQaC8K7VG1Ldd21DXTOgaMpp2P/XEuzoEWm9W138elaA8YNiovs8NEiUiKvoeimSTaYBfV8h565q6WCbM1Nc5VXulquDcvzI83N99MsdNsqnNM0TK361/DMxQlD/Zo7niefiKU3wqbBBKtlW10QiHXFLgC5SBj52x6vpW+m43N1C8QKjfvitIc85SE9/p+jad+DGWIwqbzlYEZ6DHbDgYSYRURep0f6nBEnoW0Swp22rYJFZsfVsgZQ4EVez1r/dBTIgsmbjCEFPmj7cVx0jwbTyk6q8n2KgzbjXRD0EFfy1VXskKQfvtH66Lh3n1I7cgfQmmokFXXmnH/vEO1UedwGbI/5ZKLk+yHFXj6jKLGBSWyLW65dPfPFn5Y6xpq1MKK4676w3OAR0R31ldAONWksLStjwJA7SA6DF9lOHht85iMWmYidhAbIFbFIDUnOR9t+ouJbPs4xcFpxnXDgStnKrN8 eX5TYnsa O821mXSroI+2h1OIhwpUr5EdRPX7+HyP4LnP0B8KhzQCkR65tSTrk3TWvwebVNioK6+r5dsLO8YBjLe3vo8U8lg119OSEp3t2UWJ1AQc1UDkWJHphwquFOzQiJe8ks99D1rf5/VbZkBexmDhJTls3OzM5EiBXP+vTaZbdg2wgU3FNm4uVA6yKW+sJLMmeZKJ5UHAxiWbFe8ePPrKc3rVDBaCXE4KwzHQuWomj4UqKuMsW113aBRlxv21zRTDFXYXTqcNZDbPXiHyFfJLHr22EhutA7HsUjzcotMQ7CUFY4DPpGKKV/tSgBta8zJfBO0A1IX5BcwZhxgPY5t7aP3NuYFj5I9L7u7gYAVaeW5/Qkruuh6Pu7b4wQtgCNkJ6vpzxoCEAL+Zsk4a7wlSIqORJppBwbj75XdBnI/FTUUsxE4WjmyoyH57AbzKixC5nUwnKLymxRwf7aw29ach9EsnK/ZNegIXFq3UZ2jWno/23Hpz51zVGJKYbvYW+vRbLyY+vj082h0nNChFV0AhqxHmE1QAggA== X-Bogosity: Ham, tests=bogofilter, spamicity=0.000000, version=1.2.4 Sender: owner-linux-mm@kvack.org Precedence: bulk X-Loop: owner-majordomo@kvack.org List-ID: List-Subscribe: List-Unsubscribe: Use the new API that can understand huge pfn mappings. Cc: Alex Williamson Cc: Jason Gunthorpe Signed-off-by: Peter Xu --- drivers/vfio/vfio_iommu_type1.c | 16 ++++++---------- 1 file changed, 6 insertions(+), 10 deletions(-) diff --git a/drivers/vfio/vfio_iommu_type1.c b/drivers/vfio/vfio_iommu_type1.c index 0960699e7554..bf391b40e576 100644 --- a/drivers/vfio/vfio_iommu_type1.c +++ b/drivers/vfio/vfio_iommu_type1.c @@ -513,12 +513,10 @@ static int follow_fault_pfn(struct vm_area_struct *vma, struct mm_struct *mm, unsigned long vaddr, unsigned long *pfn, bool write_fault) { - pte_t *ptep; - pte_t pte; - spinlock_t *ptl; + struct follow_pfnmap_args args = { .vma = vma, .address = vaddr }; int ret; - ret = follow_pte(vma, vaddr, &ptep, &ptl); + ret = follow_pfnmap_start(&args); if (ret) { bool unlocked = false; @@ -532,19 +530,17 @@ static int follow_fault_pfn(struct vm_area_struct *vma, struct mm_struct *mm, if (ret) return ret; - ret = follow_pte(vma, vaddr, &ptep, &ptl); + ret = follow_pfnmap_start(&args); if (ret) return ret; } - pte = ptep_get(ptep); - - if (write_fault && !pte_write(pte)) + if (write_fault && !args.writable) ret = -EFAULT; else - *pfn = pte_pfn(pte); + *pfn = args.pfn; - pte_unmap_unlock(ptep, ptl); + follow_pfnmap_end(&args); return ret; } From patchwork Fri Aug 9 16:09:04 2024 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Peter Xu X-Patchwork-Id: 13758940 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from kanga.kvack.org (kanga.kvack.org [205.233.56.17]) by smtp.lore.kernel.org (Postfix) with ESMTP id 0D120C3DA4A for ; Fri, 9 Aug 2024 16:10:04 +0000 (UTC) Received: by kanga.kvack.org (Postfix) id 6F94F6B00B1; Fri, 9 Aug 2024 12:09:55 -0400 (EDT) Received: by kanga.kvack.org (Postfix, from userid 40) id 67F5E6B00B2; Fri, 9 Aug 2024 12:09:55 -0400 (EDT) X-Delivered-To: int-list-linux-mm@kvack.org Received: by kanga.kvack.org (Postfix, from userid 63042) id 4D23B6B00B3; Fri, 9 Aug 2024 12:09:55 -0400 (EDT) X-Delivered-To: linux-mm@kvack.org Received: from relay.hostedemail.com (smtprelay0010.hostedemail.com [216.40.44.10]) by kanga.kvack.org (Postfix) with ESMTP id 2E8B56B00B1 for ; Fri, 9 Aug 2024 12:09:55 -0400 (EDT) Received: from smtpin12.hostedemail.com (a10.router.float.18 [10.200.18.1]) by unirelay06.hostedemail.com (Postfix) with ESMTP id D3FFAA7F33 for ; Fri, 9 Aug 2024 16:09:54 +0000 (UTC) X-FDA: 82433193108.12.E8B2326 Received: from us-smtp-delivery-124.mimecast.com (us-smtp-delivery-124.mimecast.com [170.10.133.124]) by imf07.hostedemail.com (Postfix) with ESMTP id B546F40019 for ; Fri, 9 Aug 2024 16:09:50 +0000 (UTC) Authentication-Results: imf07.hostedemail.com; dkim=pass header.d=redhat.com header.s=mimecast20190719 header.b=NOHMZTEY; spf=pass (imf07.hostedemail.com: domain of peterx@redhat.com designates 170.10.133.124 as permitted sender) smtp.mailfrom=peterx@redhat.com; dmarc=pass (policy=none) header.from=redhat.com ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=hostedemail.com; s=arc-20220608; t=1723219726; h=from:from:sender:reply-to:subject:subject:date:date: message-id:message-id:to:to:cc:cc:mime-version:mime-version: content-type:content-type: content-transfer-encoding:content-transfer-encoding: in-reply-to:in-reply-to:references:references:dkim-signature; bh=d0kI5he4gLOJUpsF87CpEeiS1FOCa69CaYI81su9df0=; b=mRlMp2y5lYtn3ub3vg8QUhgEGz7Brudpt5T5RLPXKO++AqX5+zJhBOZ5iey1Rl8Qjp3Iyp PH1Sje42QCFSLXnKUuMmJ7KzB784lAHUgMEKjRh7FbGP7b6cQWruMZUVlgf5SecSo1Xlje ONPgbiRQmEwGynwxC+zZCUxZT5fuC+U= ARC-Seal: i=1; s=arc-20220608; d=hostedemail.com; t=1723219726; a=rsa-sha256; cv=none; b=VTxm7dlCNIqggil4s7C7fGJkQjGE3Vb960bub9uZ2XdGvwEZUsPgizX548aZ+nA2Sjyy9A XTVBVD3tnxafh9l4y2SOjdXJLnb0KhEDtY3whzapHWh1UwxCuHd7yxRNk5Mq6dgtymxbIK LPYpOl9tiN51poTH6lDonCFJPTHsPLI= ARC-Authentication-Results: i=1; imf07.hostedemail.com; dkim=pass header.d=redhat.com header.s=mimecast20190719 header.b=NOHMZTEY; spf=pass (imf07.hostedemail.com: domain of peterx@redhat.com designates 170.10.133.124 as permitted sender) smtp.mailfrom=peterx@redhat.com; dmarc=pass (policy=none) header.from=redhat.com DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=redhat.com; s=mimecast20190719; t=1723219789; h=from:from:reply-to:subject:subject:date:date:message-id:message-id: to:to:cc:cc:mime-version:mime-version:content-type:content-type: content-transfer-encoding:content-transfer-encoding: in-reply-to:in-reply-to:references:references; bh=d0kI5he4gLOJUpsF87CpEeiS1FOCa69CaYI81su9df0=; b=NOHMZTEYQRZqXwCjooRaZ4IBrccp5xFAecuBdzIBmUzItE2s8qswsmxXAd7sJkMm95Z2Rj wo/el++TUrw6Et7EjRKczGEgM0Isme3b3qR7okM/lMbz0qjKQmNofY+SE1lLrlqh1NDQDv jvN0gr9oP/r839zRpBkg5ZkldV5i3Yg= Received: from mail-qt1-f200.google.com (mail-qt1-f200.google.com [209.85.160.200]) by relay.mimecast.com with ESMTP with STARTTLS (version=TLSv1.3, cipher=TLS_AES_256_GCM_SHA384) id us-mta-330-KzGBYIAYNNGhMlD_CdVPsg-1; Fri, 09 Aug 2024 12:09:48 -0400 X-MC-Unique: KzGBYIAYNNGhMlD_CdVPsg-1 Received: by mail-qt1-f200.google.com with SMTP id d75a77b69052e-44fe28cd027so3758511cf.0 for ; Fri, 09 Aug 2024 09:09:48 -0700 (PDT) X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20230601; t=1723219787; x=1723824587; h=content-transfer-encoding:mime-version:references:in-reply-to :message-id:date:subject:cc:to:from:x-gm-message-state:from:to:cc :subject:date:message-id:reply-to; bh=d0kI5he4gLOJUpsF87CpEeiS1FOCa69CaYI81su9df0=; b=oGHMPAUrkYid0Pgs5cwSTgM1zIIyAoG+q31yWIPKVdiu1ZsIypE/bxlnziHPUiEqDL zQJf2l0L8ZBb5LgKubmyJOq0h6uOCBy6yDeXWCParDuTy1CdwNujirUdxbG6fd3A0TBE NjVur4hn9Zrzega8oFiFQm1Z97a8zoGUj3cNcsWkuFC35757L+g+lUYoX7Np021b8rAf 6z8FLisjM0wXeo9RBxAJbcJSTMGHRDE7Fu6LdiVBGgIorwA4bmzA3r7A7qyUxqgUDsra q5Gkr2WSUMhyR305a3WMreK2KlcGz6HfnmUSaPetYE5l0guFsrCrP2U8p9gcm4ejwp7S l1AQ== X-Gm-Message-State: AOJu0YyYlYBXxTFjMWlsfb+wdVnB+2yltkQd4Wg5eKKO2pAMU1f1eigB TzbScHCAdcEb/Ul6qcut2uJhKNnMTzpybJRs8y6YWUm6zfV/BeJkufcq3gw51hi3ccom2xXsMAJ qiqogGtGcmlOYrQQ7kv5QC6noIuGU/wEnxeK10F5WQYaOSELVBu1Uts0aFQixxG7wt9fxDXcR+L Fq/6qhGg/KRat/AuU5F6cx/yWlRMeyRVjH X-Received: by 2002:ac8:5acc:0:b0:450:26ca:9170 with SMTP id d75a77b69052e-4531255b48dmr13596881cf.5.1723219787344; Fri, 09 Aug 2024 09:09:47 -0700 (PDT) X-Google-Smtp-Source: AGHT+IGxpG5xPFbbw3Y9PVyV1kJ47S21qHA1+Fquj+FmgsVP6Zo0aX1jKS3YWr/9l77XMEl3bpPFxw== X-Received: by 2002:ac8:5acc:0:b0:450:26ca:9170 with SMTP id d75a77b69052e-4531255b48dmr13596351cf.5.1723219786668; Fri, 09 Aug 2024 09:09:46 -0700 (PDT) Received: from x1n.redhat.com (pool-99-254-121-117.cpe.net.cable.rogers.com. [99.254.121.117]) by smtp.gmail.com with ESMTPSA id d75a77b69052e-451c870016csm22526741cf.19.2024.08.09.09.09.44 (version=TLS1_3 cipher=TLS_AES_256_GCM_SHA384 bits=256/256); Fri, 09 Aug 2024 09:09:45 -0700 (PDT) From: Peter Xu To: linux-mm@kvack.org, linux-kernel@vger.kernel.org Cc: Sean Christopherson , Oscar Salvador , Jason Gunthorpe , Axel Rasmussen , linux-arm-kernel@lists.infradead.org, x86@kernel.org, peterx@redhat.com, Will Deacon , Gavin Shan , Paolo Bonzini , Zi Yan , Andrew Morton , Catalin Marinas , Ingo Molnar , Alistair Popple , Borislav Petkov , David Hildenbrand , Thomas Gleixner , kvm@vger.kernel.org, Dave Hansen , Alex Williamson , Yan Zhao Subject: [PATCH 14/19] acrn: Use the new follow_pfnmap API Date: Fri, 9 Aug 2024 12:09:04 -0400 Message-ID: <20240809160909.1023470-15-peterx@redhat.com> X-Mailer: git-send-email 2.45.0 In-Reply-To: <20240809160909.1023470-1-peterx@redhat.com> References: <20240809160909.1023470-1-peterx@redhat.com> MIME-Version: 1.0 X-Mimecast-Spam-Score: 0 X-Mimecast-Originator: redhat.com X-Rspam-User: X-Rspamd-Server: rspam04 X-Rspamd-Queue-Id: B546F40019 X-Stat-Signature: nctzdc6satapes4q5ki9abjn4aidz9bb X-HE-Tag: 1723219790-201884 X-HE-Meta: U2FsdGVkX1+tkXw6AWolG8GNSeoJGAHfl6dP7ZshSm480an2wYzEhvYxpTawEq9M91r2ANBaREPwMaYLrEe1mzYIn8eAe8kQA/CGqi7+yk5kR0F0mWYmvL2xcHuTFGiQK2ITfO7qtiKZavZu1l32Ed1IIcQKEJVVCwVyxQjq1RCKR7Kbw8VHb7DUWlqwWokKKd/9mBcddRsqrH2/Sjgbe57Z5D4oIz/rHlzUNLJJgxAY60If6dloEgZwgy7lgiy16UtUbAKqzOiD+I1Eyny6TnUX5U0fRn5XLSGC1BUbkSdlAJ/OOGRLWmqqtCHnfYDwLyDcxzHjV2DT/BaPSGcODaNY1GYs6+IvjLiB9nX12kNi/qCL/0buEqsgKBngAYPy0M01Kh6BlxC7EOEMgTu0/BObvd6ABFYirvMvO4zOtnouORD7ltrzxKhpD0El0ZKfeqfkK7YmH9SZiOYq1kyKJOlLDcLT7JGonyOwPKj0Ey6EL9mRK47XdlWJ7PQMT3/LtLfbMws1EhZHNuSr9qu96MO9m46kUUVe+lr5QsS22lbuPnDYNhaCt5o7x/idc5Ah8CvzIGQN2hD47C1w4ILIuuYIR0ohE5j2eM6CeH8QY9OXbrHhnX+ngOwvR/Ir0Co8oa4ESV6XWTNJT+LmMhc6/XbGWGknQPJ7yY/usvxyPyyq6mzcjxpteEPGgbfmCdpmkkr8wwml6UiN27sZMYqEURWJ88Smn011Gl0Hpjr0lTykIg4WHrh29ODuWEV3QHUPF3NYe3h1+Bo9Atep0jOuWX6OICDfCl8GlXMBRplswMNirkAjYvWG/3J9FHOcLCiUORa9ksPxO5sGEK9CvwtRCXM3GeUk9iAr1+2DP2tYiEraV3IXnoVLUbI/MP17CUfffWAWpagxcWWQSmgN2pfej8c/sCzz/06h7+4f7CU1n2ftqVgphzHefw1Z+EBIidTB6UslMkwdL9HT7QRMoZy B4XwekZv /Hkd09SAxWvDo8QuO2o3OizGs92LjmV0tRmzi9RhOD2PeYk44PH5mSQSmkvWWIWSdEvdytzeOlQ3SUTFxLfn8AQY2rtOXk3hNTIuIXB3v1yUL9JIiY43NCoo5UXoC3HQpC0YoyEIwmVKl7KdXBF1m7PJ/nt6znboDZFscSFpRhK14cTGtoOuASRnU32y+wyIR0qckz6ms1iCMl5pheWnEEB0+/30GUqbco/OJUeYYaCeX7cnU4G7/b5syJm+2WyaqZXuE18qzsOcZlSJNcQqAJQn/97yqd70qGR3S1c3dFd+ozbgb4684pkuat9gvkB3vaEUJb6pudDV2kh2qosyDmhbzfjmlkL9jnQrfSOKWQmOy1ORB2Ia9sUWzf5VR9tKAYeVfqhK+huH9e+5rFtz17hwdkiEeAO/gJif5o2WEUw+FOLQjeutQj4J6YDbHovUjM0oO4hrzhhbt+VijApnCwcCyxS9Q/QFo6qOtFXCoI0vB35NZd1NcG6BP3aBOtRVVluN8 X-Bogosity: Ham, tests=bogofilter, spamicity=0.000000, version=1.2.4 Sender: owner-linux-mm@kvack.org Precedence: bulk X-Loop: owner-majordomo@kvack.org List-ID: List-Subscribe: List-Unsubscribe: Use the new API that can understand huge pfn mappings. Signed-off-by: Peter Xu --- drivers/virt/acrn/mm.c | 16 +++++++++------- 1 file changed, 9 insertions(+), 7 deletions(-) diff --git a/drivers/virt/acrn/mm.c b/drivers/virt/acrn/mm.c index db8ff1d0ac23..4c2f28715b70 100644 --- a/drivers/virt/acrn/mm.c +++ b/drivers/virt/acrn/mm.c @@ -177,9 +177,7 @@ int acrn_vm_ram_map(struct acrn_vm *vm, struct acrn_vm_memmap *memmap) vma = vma_lookup(current->mm, memmap->vma_base); if (vma && ((vma->vm_flags & VM_PFNMAP) != 0)) { unsigned long start_pfn, cur_pfn; - spinlock_t *ptl; bool writable; - pte_t *ptep; if ((memmap->vma_base + memmap->len) > vma->vm_end) { mmap_read_unlock(current->mm); @@ -187,16 +185,20 @@ int acrn_vm_ram_map(struct acrn_vm *vm, struct acrn_vm_memmap *memmap) } for (i = 0; i < nr_pages; i++) { - ret = follow_pte(vma, memmap->vma_base + i * PAGE_SIZE, - &ptep, &ptl); + struct follow_pfnmap_args args = { + .vma = vma, + .address = memmap->vma_base + i * PAGE_SIZE, + }; + + ret = follow_pfnmap_start(&args); if (ret) break; - cur_pfn = pte_pfn(ptep_get(ptep)); + cur_pfn = args.pfn; if (i == 0) start_pfn = cur_pfn; - writable = !!pte_write(ptep_get(ptep)); - pte_unmap_unlock(ptep, ptl); + writable = args.writable; + follow_pfnmap_end(&args); /* Disallow write access if the PTE is not writable. */ if (!writable && From patchwork Fri Aug 9 16:09:05 2024 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Peter Xu X-Patchwork-Id: 13758939 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from kanga.kvack.org (kanga.kvack.org [205.233.56.17]) by smtp.lore.kernel.org (Postfix) with ESMTP id 86C16C52D7F for ; Fri, 9 Aug 2024 16:10:00 +0000 (UTC) Received: by kanga.kvack.org (Postfix) id 19F736B00B0; Fri, 9 Aug 2024 12:09:54 -0400 (EDT) Received: by kanga.kvack.org (Postfix, from userid 40) id 14FFC6B00B1; Fri, 9 Aug 2024 12:09:54 -0400 (EDT) X-Delivered-To: int-list-linux-mm@kvack.org Received: by kanga.kvack.org (Postfix, from userid 63042) id F0AD06B00B2; Fri, 9 Aug 2024 12:09:53 -0400 (EDT) X-Delivered-To: linux-mm@kvack.org Received: from relay.hostedemail.com (smtprelay0015.hostedemail.com [216.40.44.15]) by kanga.kvack.org (Postfix) with ESMTP id D089F6B00B0 for ; Fri, 9 Aug 2024 12:09:53 -0400 (EDT) Received: from smtpin27.hostedemail.com (a10.router.float.18 [10.200.18.1]) by unirelay04.hostedemail.com (Postfix) with ESMTP id A027C1A102A for ; Fri, 9 Aug 2024 16:09:53 +0000 (UTC) X-FDA: 82433193066.27.7080CFA Received: from us-smtp-delivery-124.mimecast.com (us-smtp-delivery-124.mimecast.com [170.10.129.124]) by imf26.hostedemail.com (Postfix) with ESMTP id 65556140035 for ; Fri, 9 Aug 2024 16:09:51 +0000 (UTC) Authentication-Results: imf26.hostedemail.com; dkim=pass header.d=redhat.com header.s=mimecast20190719 header.b=AhY5TU29; spf=pass (imf26.hostedemail.com: domain of peterx@redhat.com designates 170.10.129.124 as permitted sender) smtp.mailfrom=peterx@redhat.com; dmarc=pass (policy=none) header.from=redhat.com ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=hostedemail.com; s=arc-20220608; t=1723219781; h=from:from:sender:reply-to:subject:subject:date:date: message-id:message-id:to:to:cc:cc:mime-version:mime-version: content-type:content-type: content-transfer-encoding:content-transfer-encoding: in-reply-to:in-reply-to:references:references:dkim-signature; bh=9FTs1XtnGv67Q34nCFPbcUqDR1Dyk8Tfq0P0nZ4zw7U=; b=OJ48B5zeP4GB6Vsn8Ju2K+eCxD6F/GtjmKAmXFE7U0UlAsTVpFJtdozffyTJTVjdi1lxUa 5Z4HFaxeE2LAY8phPyTgmLvJMp3nCrmXvAfWEwO8jx0WyyFUvU/+W3EF+WmAqEby0QGkHc BtFM7Gy1WwHtQ00J/NVleuHdGTXheu4= ARC-Authentication-Results: i=1; imf26.hostedemail.com; dkim=pass header.d=redhat.com header.s=mimecast20190719 header.b=AhY5TU29; spf=pass (imf26.hostedemail.com: domain of peterx@redhat.com designates 170.10.129.124 as permitted sender) smtp.mailfrom=peterx@redhat.com; dmarc=pass (policy=none) header.from=redhat.com ARC-Seal: i=1; s=arc-20220608; d=hostedemail.com; t=1723219781; a=rsa-sha256; cv=none; b=NQNsDGnCLzeTUsbO/Dtzjs+qGYH0ckZp7ejAE9lms2jcrpUgCizsBnrSKgY98Pu2JDSFil bIbh1gwzo+Xqn+zARJ0cAy9xZ+f9GIUZ5Rhm4C+lQBPJuF+uW9ihel7OIl0W2VSmwRtOOj TbN51imyIIYpaf9fXWNo8yBKUHNXIKA= DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=redhat.com; s=mimecast20190719; t=1723219790; h=from:from:reply-to:subject:subject:date:date:message-id:message-id: to:to:cc:cc:mime-version:mime-version:content-type:content-type: content-transfer-encoding:content-transfer-encoding: in-reply-to:in-reply-to:references:references; bh=9FTs1XtnGv67Q34nCFPbcUqDR1Dyk8Tfq0P0nZ4zw7U=; b=AhY5TU29KFIoZUAFt/vE8ansnAn+BAds844IseN4ZW7ZlfEEg6HVU/Ofrf3rMIloDrE/oQ 7dkyXdosofR3mHRinQl6vYZWqOuDncvMK4G7tc8qmh60a8liCa9q7SLQrbXklzCDhl3rwE wUL1MrjqBIRo/k9Z4KvkktrsWNnA/fI= Received: from mail-qk1-f197.google.com (mail-qk1-f197.google.com [209.85.222.197]) by relay.mimecast.com with ESMTP with STARTTLS (version=TLSv1.3, cipher=TLS_AES_256_GCM_SHA384) id us-mta-631-mckn3LVINVuKgskhPPS9GQ-1; Fri, 09 Aug 2024 12:09:49 -0400 X-MC-Unique: mckn3LVINVuKgskhPPS9GQ-1 Received: by mail-qk1-f197.google.com with SMTP id af79cd13be357-7a1dd7e9a72so5256985a.2 for ; Fri, 09 Aug 2024 09:09:49 -0700 (PDT) X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20230601; t=1723219789; x=1723824589; h=content-transfer-encoding:mime-version:references:in-reply-to :message-id:date:subject:cc:to:from:x-gm-message-state:from:to:cc :subject:date:message-id:reply-to; bh=9FTs1XtnGv67Q34nCFPbcUqDR1Dyk8Tfq0P0nZ4zw7U=; b=ocqXQs7ym/zZizHHbwBUwMUszbpnK3AdZz2Awq+eMTYAxORizCfUfifM4DVo3oE31k T3nbT54Boly3+odRvVjgGgYyr6xlIf6eUVV9wko1uP+esuEagzo/lDgzg+5J7VNC+XNT 0d0utCcckvC/1yErqBtV4CKE1bWasNU4Edtlg2No5r7CFJQP9KLowkbHQgdD/cbW06J/ vivIrqJWawVgs0UklO4GG4Fc2D+dFuzcn1iEu1DlIBqfj1X0X9bzt0oSyyG1M1/xPyIR ofZLPBR8n/AOp0dyPGJsQF91b12hAyOesNKuFXilhoLF+Fn0Cex0kzi2ectRdhus/km+ ZjWQ== X-Gm-Message-State: AOJu0YyeSe7gbL7nZpcjjju/596ikuy3AdGVANLmjA+RPlOidEZC4yZV WCvIVMW0NTeF9dMzCnm4qZD2b51JfYvbJ92fPoDcMcLYp44/l/+wu95nDvqAiioTuG5jpMtmEqo 8q0HS4pOgEaq9CVdjnNWzFKK19jz0vUgeXX6LxEzGpPMCBAnFspBtVfw6CQgu5fKf15nA9seM8F RiWtzDtD1fBxND/ahkb2yY/M+QCKI0iLxP X-Received: by 2002:a05:622a:11d1:b0:446:5b0a:e778 with SMTP id d75a77b69052e-4531235a44cmr11675251cf.0.1723219788899; Fri, 09 Aug 2024 09:09:48 -0700 (PDT) X-Google-Smtp-Source: AGHT+IH3kKOCpVIG6mSA6dMC4Xb4vB8Z6SmUroM26OhSq4nP1v0KE/gOqkuWVbJLnRl3s3aUYCeIEQ== X-Received: by 2002:a05:622a:11d1:b0:446:5b0a:e778 with SMTP id d75a77b69052e-4531235a44cmr11674811cf.0.1723219788421; Fri, 09 Aug 2024 09:09:48 -0700 (PDT) Received: from x1n.redhat.com (pool-99-254-121-117.cpe.net.cable.rogers.com. [99.254.121.117]) by smtp.gmail.com with ESMTPSA id d75a77b69052e-451c870016csm22526741cf.19.2024.08.09.09.09.46 (version=TLS1_3 cipher=TLS_AES_256_GCM_SHA384 bits=256/256); Fri, 09 Aug 2024 09:09:48 -0700 (PDT) From: Peter Xu To: linux-mm@kvack.org, linux-kernel@vger.kernel.org Cc: Sean Christopherson , Oscar Salvador , Jason Gunthorpe , Axel Rasmussen , linux-arm-kernel@lists.infradead.org, x86@kernel.org, peterx@redhat.com, Will Deacon , Gavin Shan , Paolo Bonzini , Zi Yan , Andrew Morton , Catalin Marinas , Ingo Molnar , Alistair Popple , Borislav Petkov , David Hildenbrand , Thomas Gleixner , kvm@vger.kernel.org, Dave Hansen , Alex Williamson , Yan Zhao Subject: [PATCH 15/19] mm/access_process_vm: Use the new follow_pfnmap API Date: Fri, 9 Aug 2024 12:09:05 -0400 Message-ID: <20240809160909.1023470-16-peterx@redhat.com> X-Mailer: git-send-email 2.45.0 In-Reply-To: <20240809160909.1023470-1-peterx@redhat.com> References: <20240809160909.1023470-1-peterx@redhat.com> MIME-Version: 1.0 X-Mimecast-Spam-Score: 0 X-Mimecast-Originator: redhat.com X-Rspam-User: X-Stat-Signature: kng8w4oghpk96ynrhfekgzpcu48c7gk1 X-Rspamd-Queue-Id: 65556140035 X-Rspamd-Server: rspam11 X-HE-Tag: 1723219791-923616 X-HE-Meta: U2FsdGVkX19qKTCW/xOPmaP8Tqz4qQnI4kDAIs4ZxdcLwbU6A2V6wUYc0ZLvEOiYeYcyMU7b+uO0fA4jw5jHU6NtqTQmwjm1K7HhM501LDwm/5wmCawdpvQKEztz23Fy/56K4xs5PvenLhR7D3CfeGRMcddoOz/xP+pN2+Jn7Zd8RGu7CyJOUzaQv2HxU6ZNAjeQGgWxlmuVncpeSWWX68hF438P/ioozAf/lhMElpbrchFSy/VX3v/Q4zZkEdnc0k+U3/BCUZVe/j5jaKLHOUN9nJrKsHbF881G6hH+iAnJafz+NFwGEdjYjKFy0rT3cImxLUCB4rzWdXFmPlQvjwP+s2pVYQ8L/d5OesiGZCkUQTVSgMzw/+ko1Wje5nTmdQxQtoBeD6DqghUwE/pvYjiu0B7baAf2a7Cf6UIY5imeNURMYFBTnqZfItKQBkC2tapAWUMxe78iaYQWFbIVwc/6Uruy4pCRtm068slYV3h4zo76je9/KtNxIaRNozdmUca4Q21GqnN1kQ3JYINYGZi4ijLor9ZFwyM/DQ/Ltv3EQbw3kHUI+hp2aBE+nXJ4SvLkqi6UJEv6KgBI26nwrYrR2N/kUNY5mlrNwvgBI+RClV0kfo168/5TemhKyE0AYfnqBm82Hyxn/i0EwcdYugnv1r6mOoua3VYc26KPXFO2AHx6ftPpbAD50HLuLbDOu5AIQv7sSk/KV4iCp5Ugh2M4VnzemN05G5avyS/dJ/Bo9qOc/b9I57aNERrj2T98K09sQhIHW2ghYY4P0yLvfGBfvYB9zBzn20q0azc95vbaT97RKkfJxervvAxXHg5sBY7j1gZrIzd2zCUf9ind5gNw7OMOGCJCTAJlr1Ew0dLIS1lf3rVtFlm+4x6BxmImSVP2vE7owgRCJb52P/ee4z/+/L3nWFwRLZA+hUvAinpAf4STdviN/LIGCOOexSiieiYOZ8BzLl8ywlAlX5U oA40nush vgmpHkX3WGuwv0CkcBDnsMNPDgnYqnrM558HQO5i+2gcwoxxQxJM/9yTNPniLM7rGsyHjgNsla81cxPFDJZZz753ADNj3ZgnL/Hhn6UiWgw3clrFnVhdFHFpzMR2r7hnk16h6wAtbQhLMsLQzGvZkZgncnFQdsi1ytGlxricQYHyX6iuDH3IyX69aYEYsynyfw3ulNqqSdZSYD6seaiwEiM1XmTWwLF/3PdzZXTiJtnSsS3GrCnFAYlvm0flVLGYNZAPozDDrgLGtzPrPucQcmEGjFYF4hN3pk3Qrn3j4IhF5q2yq96Uty3ESgUuRg2/Oxi0XZsYscPG80ylSeZQbkyrOlNzVrT5dyJLvj/9kXIzCoUsJ19ColyMWWvP5KVBielSQHer4btghff4AS4s4rGiTvK5nTqk47gM21OAIPse9EhwyfzJXZsCSIQlY/NUTQKgHQBDyL7nRD7Ld1DScDPuuDOD+Qj6SZ1e6nP+vcWy9BnBdJ3bu6dKD0zXfr5d/zCUQ X-Bogosity: Ham, tests=bogofilter, spamicity=0.000000, version=1.2.4 Sender: owner-linux-mm@kvack.org Precedence: bulk X-Loop: owner-majordomo@kvack.org List-ID: List-Subscribe: List-Unsubscribe: Use the new API that can understand huge pfn mappings. Signed-off-by: Peter Xu --- mm/memory.c | 28 ++++++++++++++-------------- 1 file changed, 14 insertions(+), 14 deletions(-) diff --git a/mm/memory.c b/mm/memory.c index 2194e0f9f541..313c17eedf56 100644 --- a/mm/memory.c +++ b/mm/memory.c @@ -6504,34 +6504,34 @@ int generic_access_phys(struct vm_area_struct *vma, unsigned long addr, resource_size_t phys_addr; unsigned long prot = 0; void __iomem *maddr; - pte_t *ptep, pte; - spinlock_t *ptl; int offset = offset_in_page(addr); int ret = -EINVAL; + bool writable; + struct follow_pfnmap_args args = { .vma = vma, .address = addr }; retry: - if (follow_pte(vma, addr, &ptep, &ptl)) + if (follow_pfnmap_start(&args)) return -EINVAL; - pte = ptep_get(ptep); - pte_unmap_unlock(ptep, ptl); + prot = pgprot_val(args.pgprot); + phys_addr = (resource_size_t)args.pfn << PAGE_SHIFT; + writable = args.writable; + follow_pfnmap_end(&args); - prot = pgprot_val(pte_pgprot(pte)); - phys_addr = (resource_size_t)pte_pfn(pte) << PAGE_SHIFT; - - if ((write & FOLL_WRITE) && !pte_write(pte)) + if ((write & FOLL_WRITE) && !writable) return -EINVAL; maddr = ioremap_prot(phys_addr, PAGE_ALIGN(len + offset), prot); if (!maddr) return -ENOMEM; - if (follow_pte(vma, addr, &ptep, &ptl)) + if (follow_pfnmap_start(&args)) goto out_unmap; - if (!pte_same(pte, ptep_get(ptep))) { - pte_unmap_unlock(ptep, ptl); + if ((prot != pgprot_val(args.pgprot)) || + (phys_addr != (args.pfn << PAGE_SHIFT)) || + (writable != args.writable)) { + follow_pfnmap_end(&args); iounmap(maddr); - goto retry; } @@ -6540,7 +6540,7 @@ int generic_access_phys(struct vm_area_struct *vma, unsigned long addr, else memcpy_fromio(buf, maddr + offset, len); ret = len; - pte_unmap_unlock(ptep, ptl); + follow_pfnmap_end(&args); out_unmap: iounmap(maddr); From patchwork Fri Aug 9 16:09:06 2024 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Peter Xu X-Patchwork-Id: 13758941 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from kanga.kvack.org (kanga.kvack.org [205.233.56.17]) by smtp.lore.kernel.org (Postfix) with ESMTP id 6E597C52D7C for ; Fri, 9 Aug 2024 16:10:07 +0000 (UTC) Received: by kanga.kvack.org (Postfix) id C266B6B00B3; Fri, 9 Aug 2024 12:09:56 -0400 (EDT) Received: by kanga.kvack.org (Postfix, from userid 40) id BD76E6B00B4; Fri, 9 Aug 2024 12:09:56 -0400 (EDT) X-Delivered-To: int-list-linux-mm@kvack.org Received: by kanga.kvack.org (Postfix, from userid 63042) id A2B626B00B5; Fri, 9 Aug 2024 12:09:56 -0400 (EDT) X-Delivered-To: linux-mm@kvack.org Received: from relay.hostedemail.com (smtprelay0010.hostedemail.com [216.40.44.10]) by kanga.kvack.org (Postfix) with ESMTP id 6937F6B00B3 for ; Fri, 9 Aug 2024 12:09:56 -0400 (EDT) Received: from smtpin18.hostedemail.com (a10.router.float.18 [10.200.18.1]) by unirelay04.hostedemail.com (Postfix) with ESMTP id 205F01A10A7 for ; Fri, 9 Aug 2024 16:09:56 +0000 (UTC) X-FDA: 82433193192.18.46F404D Received: from us-smtp-delivery-124.mimecast.com (us-smtp-delivery-124.mimecast.com [170.10.133.124]) by imf09.hostedemail.com (Postfix) with ESMTP id EA2DF14001B for ; Fri, 9 Aug 2024 16:09:53 +0000 (UTC) Authentication-Results: imf09.hostedemail.com; dkim=pass header.d=redhat.com header.s=mimecast20190719 header.b="D/yHJWk5"; spf=pass (imf09.hostedemail.com: domain of peterx@redhat.com designates 170.10.133.124 as permitted sender) smtp.mailfrom=peterx@redhat.com; dmarc=pass (policy=none) header.from=redhat.com ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=hostedemail.com; s=arc-20220608; t=1723219728; h=from:from:sender:reply-to:subject:subject:date:date: message-id:message-id:to:to:cc:cc:mime-version:mime-version: content-type:content-type: content-transfer-encoding:content-transfer-encoding: in-reply-to:in-reply-to:references:references:dkim-signature; bh=rQ5uhZ2uiWdH1/UlOzJ8LVGX9FzFBDg5i1FarWNBfgY=; b=L/TVFcN3b/NyCzzesVpz2/2lE56ZXQihkt+fp6x8+sNH8FE1yNg6pzenOZqmydvdaSVmoZ /H1K61zgW1IKB6cxB0sRRH74+rBsTI71lXz75l7NLZPtyQ4/bndh2ny5kyHk3L6aKRAFQm xJAVihtmYkACUjpPsloUenWjPD4eZwo= ARC-Seal: i=1; s=arc-20220608; d=hostedemail.com; t=1723219728; a=rsa-sha256; cv=none; b=Ww5UYP+KjIMxkSirixv473LE1+eBE9v8fbhUrEicEVDlyd0WFmw9LYWAyMKjOGBBNu9UJj OL02F74N3bWNSQ3ldViscllT9mH3COSvqPscGoMnOtcJSuufuaGbb5xBNdRZgNXfqsp3BV P7ti+9Gz2ebvu1SZMEdJlCm3lQv619I= ARC-Authentication-Results: i=1; imf09.hostedemail.com; dkim=pass header.d=redhat.com header.s=mimecast20190719 header.b="D/yHJWk5"; spf=pass (imf09.hostedemail.com: domain of peterx@redhat.com designates 170.10.133.124 as permitted sender) smtp.mailfrom=peterx@redhat.com; dmarc=pass (policy=none) header.from=redhat.com DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=redhat.com; s=mimecast20190719; t=1723219793; h=from:from:reply-to:subject:subject:date:date:message-id:message-id: to:to:cc:cc:mime-version:mime-version:content-type:content-type: content-transfer-encoding:content-transfer-encoding: in-reply-to:in-reply-to:references:references; bh=rQ5uhZ2uiWdH1/UlOzJ8LVGX9FzFBDg5i1FarWNBfgY=; b=D/yHJWk542YQuqv98dJ1Lz39VKd6FMmXeCkRhZgOQ0ZAhJctotI2zvckv8tJzeK8F7q8QM VWPAydnVlKDIGI4XuAa+n6e5EWr1eV/5fOzRRgx8zPrVnaHUNNNY/GxTYsXct8i6C6VE5l ZnFZoNM50MBLM65mwY2UQJyHgycaQGU= Received: from mail-qt1-f198.google.com (mail-qt1-f198.google.com [209.85.160.198]) by relay.mimecast.com with ESMTP with STARTTLS (version=TLSv1.3, cipher=TLS_AES_256_GCM_SHA384) id us-mta-124-GVWj-RqTPIWetj5oBnv1bA-1; Fri, 09 Aug 2024 12:09:51 -0400 X-MC-Unique: GVWj-RqTPIWetj5oBnv1bA-1 Received: by mail-qt1-f198.google.com with SMTP id d75a77b69052e-44fe28cd027so3758651cf.0 for ; Fri, 09 Aug 2024 09:09:51 -0700 (PDT) X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20230601; t=1723219791; x=1723824591; h=content-transfer-encoding:mime-version:references:in-reply-to :message-id:date:subject:cc:to:from:x-gm-message-state:from:to:cc :subject:date:message-id:reply-to; bh=rQ5uhZ2uiWdH1/UlOzJ8LVGX9FzFBDg5i1FarWNBfgY=; b=SuvKu7yk/9aw3ksoPH+aOtVhqVRDeoL8d32B81FhNt7jSmsX5KgG0oaMM9/WZfGlKc 2GfdsTJyZrmLmoqnQ8CTZ3Zv0SfIMws/zFP3Kkq115ANdkBvbBre3BZ8cpINx2LqIIck xSm7WIe6dY4fBBKalPd/1kugeaywipif8IwWykY4i1+ZuSw9XcEBfSiyKZjh/3plUTAj ZQzz2mNSIReJVDCwKIvTkwPKXHvXqK4OZ7gg2r9oMV2iProOfuyPCNbNYgDEI3HOqOR9 5l7i5J3TQIERx8H/d4cZv3JlCFlEcncBftbyhJwmpCAf837hRdbonMejwJqbI/iFHwVA JFPA== X-Gm-Message-State: AOJu0YxjPyxhl+xEidPVb4Rrq+ywVs1MBTlx6OUTKi7zrsYUjTQHUrB8 6o2W61MzrPaQnwqZrDtYRmbBxcqQMhbk4RGF+OGzywnU2ErexTB8xax+gBfoGSEi3M3ZVeI5NZT t7n3lvgmmeHzjvocqpMuFQVpRouOfqjFwnS7Mq4kt9emDOvt8EckfEIwrhMs67Y4hp00Dqnfg4h +VLv4WMKsk6WnDHmwlsuyfs13QQaILR5dj X-Received: by 2002:a05:622a:412:b0:444:b935:ebb5 with SMTP id d75a77b69052e-45312359fbcmr13373751cf.0.1723219790923; Fri, 09 Aug 2024 09:09:50 -0700 (PDT) X-Google-Smtp-Source: AGHT+IH2UCzm8Bb1pveediLGLwWSJm3e36K8ZWtfh9bSQqsAnCTBAfRwieQENSqzRtCTpHEz3lk41Q== X-Received: by 2002:a05:622a:412:b0:444:b935:ebb5 with SMTP id d75a77b69052e-45312359fbcmr13373301cf.0.1723219790408; Fri, 09 Aug 2024 09:09:50 -0700 (PDT) Received: from x1n.redhat.com (pool-99-254-121-117.cpe.net.cable.rogers.com. [99.254.121.117]) by smtp.gmail.com with ESMTPSA id d75a77b69052e-451c870016csm22526741cf.19.2024.08.09.09.09.48 (version=TLS1_3 cipher=TLS_AES_256_GCM_SHA384 bits=256/256); Fri, 09 Aug 2024 09:09:49 -0700 (PDT) From: Peter Xu To: linux-mm@kvack.org, linux-kernel@vger.kernel.org Cc: Sean Christopherson , Oscar Salvador , Jason Gunthorpe , Axel Rasmussen , linux-arm-kernel@lists.infradead.org, x86@kernel.org, peterx@redhat.com, Will Deacon , Gavin Shan , Paolo Bonzini , Zi Yan , Andrew Morton , Catalin Marinas , Ingo Molnar , Alistair Popple , Borislav Petkov , David Hildenbrand , Thomas Gleixner , kvm@vger.kernel.org, Dave Hansen , Alex Williamson , Yan Zhao Subject: [PATCH 16/19] mm: Remove follow_pte() Date: Fri, 9 Aug 2024 12:09:06 -0400 Message-ID: <20240809160909.1023470-17-peterx@redhat.com> X-Mailer: git-send-email 2.45.0 In-Reply-To: <20240809160909.1023470-1-peterx@redhat.com> References: <20240809160909.1023470-1-peterx@redhat.com> MIME-Version: 1.0 X-Mimecast-Spam-Score: 0 X-Mimecast-Originator: redhat.com X-Rspamd-Queue-Id: EA2DF14001B X-Stat-Signature: huzxyrdd9xu9b86pjkgoaqsdgpxwpbto X-Rspamd-Server: rspam09 X-Rspam-User: X-HE-Tag: 1723219793-586956 X-HE-Meta: U2FsdGVkX18vrNvUq6BabEY6yftH1ok7GSVehN8EL+6+OoAEextDNNzm9LOLZrSQUTosYV3lSa2n6+CWZPPpVavpG+Ptc0iB3GDN5L6LQpPWOZGRMR1Wt4Cy8mOxSqE+HDATqypRt0nYzUtBJyzYp6/YEhDlEbF9wnU91ZC6Atksva1iYyxoWvXOyDxTVLXuJqZ6MdQiaOJ+LzrtBREHiNaCE/ReGNlOThwrUPMsJ+p5JFjpJvtp2vcdP2DKMmkFvT1g3ZSMMn9iTUoGq6ukw1qqS/6IOAqzOuLnkqqmyYTVMI/+6slxXZvivUnCufA4CjU2YG8agB4KlpuwliLweQJw9WPj7tIkiJ7ESZrGv5c2fA2VhzxyoesJyxICAYLUT3dHU84h9ypx342iNbGdzJYJyYnyhcCJAeAhZqrk9ilzXm6NUSrh4URtxP+gFDtWpUCieMwP9mlbTgBtQdFpeIFhT/DnowN/94X5V5sTu1f4Y73a7gvw4Qdqa2FMPmqLZ8FIge4b2KaEMv2KnezPNuheicfTwE+yEGpwqXXEVvzD3X1AtMpEWDwJt2d3Jm7bkRX7fQG4nP3CQS9sSl0D3uDfpm4EoM9u2o55oVy6eGx8FDhPgy8jqSuc19f2KJEgXF/6h13FyxRptXfbjnXKoYooXX5Tte77acHB5N+87K2iMPfvpktwsj/YDxnPkVgCrWPcDtrnO6m25uOuG5Fs8qiiERzHuvkuyxtYEEgvPbvh02A2nnHuSpMQu9yntG5Lj2wuH7FEZpWVI/Rz9AidPLewNicNwbyPS6KAefUr5GBTgPbLSt3k7yMeii0iPOLFODFPvp+Yq5eS5YZL3Zm3K8e0KLJcVotVEy0Z7sRMtawLNZYy0LmVq/ZofehqPKyuoihcQQtkKcAZwRYy31Qxl/8w222CBByNNP3C8NgWM+///Kg7IryJazzHtgdMEyvGm9hN4loYgyqSEdTfiYB CSO2+cgD TUMR+sH308Anq1C+HaYIwWSFdTD39LIY3QDzArSkMwGmo3IH2fg7xglGPPrwvrW0q2dfORFCtvd6ybuYf098pfn7aKs7fNsB6Qzj3Iugltrp3do+SILm/v2YxR7+UyxYL1AcI241yU7hl1SipF8OVSLrATByPDD2rd6/k3zS7ijdjKiuQUlG8exjPaKkME3uhmisU0ZyFLcOC20FStsfXLPCWU0afItVmXizmmh+o4+YZ2/79z8h99+YZzM/ALYheUXzbztZG+5nCgCg4Ax/hTSxnF0BrlRpjV3qRUUtAjF0sYh0Hl33ypXkOirLbPPctdudlfeVCAbyWlqGs5m8Dc4AqKHXSAx/2ANNt/YbbPpwytibwfv18HRVE6Wl4uPTGNbEUMEE1p2NoyMOnTDz75w1tfal/mlWFhp0RrOP8f4C/3IEOz49XqTV1rQnz1KP8UFklGnW2rtcrIZ7mid49XnQjJg== X-Bogosity: Ham, tests=bogofilter, spamicity=0.000000, version=1.2.4 Sender: owner-linux-mm@kvack.org Precedence: bulk X-Loop: owner-majordomo@kvack.org List-ID: List-Subscribe: List-Unsubscribe: follow_pte() users have been converted to follow_pfnmap*(). Remove the API. Signed-off-by: Peter Xu --- include/linux/mm.h | 2 -- mm/memory.c | 73 ---------------------------------------------- 2 files changed, 75 deletions(-) diff --git a/include/linux/mm.h b/include/linux/mm.h index 7471302658af..c5949b8052c6 100644 --- a/include/linux/mm.h +++ b/include/linux/mm.h @@ -2369,8 +2369,6 @@ void free_pgd_range(struct mmu_gather *tlb, unsigned long addr, unsigned long end, unsigned long floor, unsigned long ceiling); int copy_page_range(struct vm_area_struct *dst_vma, struct vm_area_struct *src_vma); -int follow_pte(struct vm_area_struct *vma, unsigned long address, - pte_t **ptepp, spinlock_t **ptlp); int generic_access_phys(struct vm_area_struct *vma, unsigned long addr, void *buf, int len, int write); diff --git a/mm/memory.c b/mm/memory.c index 313c17eedf56..72f61fffdda2 100644 --- a/mm/memory.c +++ b/mm/memory.c @@ -6265,79 +6265,6 @@ int __pmd_alloc(struct mm_struct *mm, pud_t *pud, unsigned long address) } #endif /* __PAGETABLE_PMD_FOLDED */ -/** - * follow_pte - look up PTE at a user virtual address - * @vma: the memory mapping - * @address: user virtual address - * @ptepp: location to store found PTE - * @ptlp: location to store the lock for the PTE - * - * On a successful return, the pointer to the PTE is stored in @ptepp; - * the corresponding lock is taken and its location is stored in @ptlp. - * - * The contents of the PTE are only stable until @ptlp is released using - * pte_unmap_unlock(). This function will fail if the PTE is non-present. - * Present PTEs may include PTEs that map refcounted pages, such as - * anonymous folios in COW mappings. - * - * Callers must be careful when relying on PTE content after - * pte_unmap_unlock(). Especially if the PTE maps a refcounted page, - * callers must protect against invalidation with MMU notifiers; otherwise - * access to the PFN at a later point in time can trigger use-after-free. - * - * Only IO mappings and raw PFN mappings are allowed. The mmap semaphore - * should be taken for read. - * - * This function must not be used to modify PTE content. - * - * Return: zero on success, -ve otherwise. - */ -int follow_pte(struct vm_area_struct *vma, unsigned long address, - pte_t **ptepp, spinlock_t **ptlp) -{ - struct mm_struct *mm = vma->vm_mm; - pgd_t *pgd; - p4d_t *p4d; - pud_t *pud; - pmd_t *pmd; - pte_t *ptep; - - mmap_assert_locked(mm); - if (unlikely(address < vma->vm_start || address >= vma->vm_end)) - goto out; - - if (!(vma->vm_flags & (VM_IO | VM_PFNMAP))) - goto out; - - pgd = pgd_offset(mm, address); - if (pgd_none(*pgd) || unlikely(pgd_bad(*pgd))) - goto out; - - p4d = p4d_offset(pgd, address); - if (p4d_none(*p4d) || unlikely(p4d_bad(*p4d))) - goto out; - - pud = pud_offset(p4d, address); - if (pud_none(*pud) || unlikely(pud_bad(*pud))) - goto out; - - pmd = pmd_offset(pud, address); - VM_BUG_ON(pmd_trans_huge(*pmd)); - - ptep = pte_offset_map_lock(mm, pmd, address, ptlp); - if (!ptep) - goto out; - if (!pte_present(ptep_get(ptep))) - goto unlock; - *ptepp = ptep; - return 0; -unlock: - pte_unmap_unlock(ptep, *ptlp); -out: - return -EINVAL; -} -EXPORT_SYMBOL_GPL(follow_pte); - static inline void pfnmap_args_setup(struct follow_pfnmap_args *args, spinlock_t *lock, pte_t *ptep, pgprot_t pgprot, unsigned long pfn_base, From patchwork Fri Aug 9 16:09:07 2024 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Peter Xu X-Patchwork-Id: 13758942 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from kanga.kvack.org (kanga.kvack.org [205.233.56.17]) by smtp.lore.kernel.org (Postfix) with ESMTP id CD971C3DA4A for ; Fri, 9 Aug 2024 16:10:10 +0000 (UTC) Received: by kanga.kvack.org (Postfix) id 289A06B00B5; Fri, 9 Aug 2024 12:10:00 -0400 (EDT) Received: by kanga.kvack.org (Postfix, from userid 40) id 213476B00B6; Fri, 9 Aug 2024 12:10:00 -0400 (EDT) X-Delivered-To: int-list-linux-mm@kvack.org Received: by kanga.kvack.org (Postfix, from userid 63042) id 08E8C6B00B7; Fri, 9 Aug 2024 12:09:59 -0400 (EDT) X-Delivered-To: linux-mm@kvack.org Received: from relay.hostedemail.com (smtprelay0016.hostedemail.com [216.40.44.16]) by kanga.kvack.org (Postfix) with ESMTP id C4BAC6B00B5 for ; Fri, 9 Aug 2024 12:09:59 -0400 (EDT) Received: from smtpin18.hostedemail.com (a10.router.float.18 [10.200.18.1]) by unirelay09.hostedemail.com (Postfix) with ESMTP id 790C681167 for ; Fri, 9 Aug 2024 16:09:59 +0000 (UTC) X-FDA: 82433193318.18.31182F8 Received: from us-smtp-delivery-124.mimecast.com (us-smtp-delivery-124.mimecast.com [170.10.129.124]) by imf06.hostedemail.com (Postfix) with ESMTP id 6399E180032 for ; Fri, 9 Aug 2024 16:09:57 +0000 (UTC) Authentication-Results: imf06.hostedemail.com; dkim=pass header.d=redhat.com header.s=mimecast20190719 header.b=DCqMayQp; spf=pass (imf06.hostedemail.com: domain of peterx@redhat.com designates 170.10.129.124 as permitted sender) smtp.mailfrom=peterx@redhat.com; dmarc=pass (policy=none) header.from=redhat.com ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=hostedemail.com; s=arc-20220608; t=1723219764; h=from:from:sender:reply-to:subject:subject:date:date: message-id:message-id:to:to:cc:cc:mime-version:mime-version: content-type:content-type: content-transfer-encoding:content-transfer-encoding: in-reply-to:in-reply-to:references:references:dkim-signature; bh=YoGoBxg1RIhRUWN8551lXVuosUhfDUfYXVK/Rz8tB4Q=; b=sTrtgFCrrUANpMUrDpsFQM/tOlsncYDRTCU5AORBQoofRlbujcrGDhl77GBujKH4noCZmy BrfiU3nIX/o3xySKgZ5UBYospqr4m/J3M26GHl3d0kNJorAHkhE9nV92SzAXPFR0MbiVqU sp3zpBJXjnzgASB/leUmD7rDFkkz6Zo= ARC-Authentication-Results: i=1; imf06.hostedemail.com; dkim=pass header.d=redhat.com header.s=mimecast20190719 header.b=DCqMayQp; spf=pass (imf06.hostedemail.com: domain of peterx@redhat.com designates 170.10.129.124 as permitted sender) smtp.mailfrom=peterx@redhat.com; dmarc=pass (policy=none) header.from=redhat.com ARC-Seal: i=1; s=arc-20220608; d=hostedemail.com; t=1723219764; a=rsa-sha256; cv=none; b=vbad3NU05rs+mDrO3g36L68kqUd0PiiuWsj3GGDGXucFIbtYruwWk2kGQ75ZZ8NPusFP+o CNyEBHVSERzGdx7UEy6PVx0VhkczQNKgdwHaM9ujWXmTQFoU+Wn5U+tRqQJXdgCCDmPUY6 kIJuSIO3r9XsHN+34kSybr0cW374sF0= DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=redhat.com; s=mimecast20190719; t=1723219796; h=from:from:reply-to:subject:subject:date:date:message-id:message-id: to:to:cc:cc:mime-version:mime-version:content-type:content-type: content-transfer-encoding:content-transfer-encoding: in-reply-to:in-reply-to:references:references; bh=YoGoBxg1RIhRUWN8551lXVuosUhfDUfYXVK/Rz8tB4Q=; b=DCqMayQpcVuHowf05xL+L7KKv4A/9fk/wXxHCwd/hJQqV9Cy05P4BPDCF5oa2ovdX0v6OK uFxM8f6uiW9v6oy9K9ifiCe4TYkk/f8Ji6uB3MI8nTOlIPU2nf9Z/xS9UaBMfaKcKEsDSI Qw70woQpQ9w8K/zBWCFkv7waYojf8Y0= Received: from mail-qt1-f200.google.com (mail-qt1-f200.google.com [209.85.160.200]) by relay.mimecast.com with ESMTP with STARTTLS (version=TLSv1.3, cipher=TLS_AES_256_GCM_SHA384) id us-mta-687-I2RmvRsvOn-srxzu3sVoMg-1; Fri, 09 Aug 2024 12:09:54 -0400 X-MC-Unique: I2RmvRsvOn-srxzu3sVoMg-1 Received: by mail-qt1-f200.google.com with SMTP id d75a77b69052e-4503ee1b5d6so5447661cf.3 for ; Fri, 09 Aug 2024 09:09:54 -0700 (PDT) X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20230601; t=1723219793; x=1723824593; h=content-transfer-encoding:mime-version:references:in-reply-to :message-id:date:subject:cc:to:from:x-gm-message-state:from:to:cc :subject:date:message-id:reply-to; bh=YoGoBxg1RIhRUWN8551lXVuosUhfDUfYXVK/Rz8tB4Q=; b=ZILmGWjb8SN+J9l2SY+l+9Q9fwrwshNq7ZQrXXZqWy5BaHGD0zSPBwh0EytquxpmVI SkWRUQ9O9mK26Pof3BHJT1dWPVOAR5Vf0YUTskbDjcU2+H/1gWuTYJDvgxvleutL3Nem Oon6SYmjPXrmCt/5IKm54Z/1zE2x/WoHOCftW2Hj4i49ipJ7Z1zEvWbn1VZoF39p1qGv aVaEw//oGSS9UigOfWJRHTBdomIHhIBFfRjivBOJdqKGybX38Gf9KWN1k3ms3OZXzvx8 /VI4EeeJZNi/R1I+GYI+dSGcQ9KC+pBB2jp8XKlyP9l9NyGaqwugpHRmjFl4hJv4QxS4 YR8Q== X-Gm-Message-State: AOJu0YwnPZVzflELRE1NfBD7Yno/E6sUsSPdSogMegNbNrl9p0HU/awh R0fixmxbyw/YSP08VTHLSWsKOymNQ9Bb4SOx9LCt1hJqN6FSs5ooICCxSy5sNsxc5gsG2pqgE9F dul+N8KmHkXL27lJeJcvjpohI6hMdp5SAFwDGRLzaiCzvZlGqonSE+02KIVuTubp8juISaW0FE6 9LdciAbGZfmNueNLz2EfoPo1gNEqim2/CS X-Received: by 2002:ac8:5e07:0:b0:44e:cff7:3741 with SMTP id d75a77b69052e-45312646da2mr14005871cf.7.1723219793423; Fri, 09 Aug 2024 09:09:53 -0700 (PDT) X-Google-Smtp-Source: AGHT+IHyC9JbjVCeQC7LVvckqh3sKVh2QUlZ0IzLJIVdarr6jR5vj53aOO53iHwQIsrAsjgq+0MAjA== X-Received: by 2002:ac8:5e07:0:b0:44e:cff7:3741 with SMTP id d75a77b69052e-45312646da2mr14005321cf.7.1723219792685; Fri, 09 Aug 2024 09:09:52 -0700 (PDT) Received: from x1n.redhat.com (pool-99-254-121-117.cpe.net.cable.rogers.com. [99.254.121.117]) by smtp.gmail.com with ESMTPSA id d75a77b69052e-451c870016csm22526741cf.19.2024.08.09.09.09.50 (version=TLS1_3 cipher=TLS_AES_256_GCM_SHA384 bits=256/256); Fri, 09 Aug 2024 09:09:51 -0700 (PDT) From: Peter Xu To: linux-mm@kvack.org, linux-kernel@vger.kernel.org Cc: Sean Christopherson , Oscar Salvador , Jason Gunthorpe , Axel Rasmussen , linux-arm-kernel@lists.infradead.org, x86@kernel.org, peterx@redhat.com, Will Deacon , Gavin Shan , Paolo Bonzini , Zi Yan , Andrew Morton , Catalin Marinas , Ingo Molnar , Alistair Popple , Borislav Petkov , David Hildenbrand , Thomas Gleixner , kvm@vger.kernel.org, Dave Hansen , Alex Williamson , Yan Zhao Subject: [PATCH 17/19] mm/x86: Support large pfn mappings Date: Fri, 9 Aug 2024 12:09:07 -0400 Message-ID: <20240809160909.1023470-18-peterx@redhat.com> X-Mailer: git-send-email 2.45.0 In-Reply-To: <20240809160909.1023470-1-peterx@redhat.com> References: <20240809160909.1023470-1-peterx@redhat.com> MIME-Version: 1.0 X-Mimecast-Spam-Score: 0 X-Mimecast-Originator: redhat.com X-Rspamd-Server: rspam03 X-Rspam-User: X-Rspamd-Queue-Id: 6399E180032 X-Stat-Signature: 6ur7k4zjia76qunwi1cnqxs8o36id95s X-HE-Tag: 1723219797-417230 X-HE-Meta: U2FsdGVkX19+JIG1mJrO7vkyOPX5x034IUVmxzZzH47TKB/Shc64oKfJZMEclh+GGqlzMXF0I/zQSjk+d8aYtz0qs2ZTtatuipJ7QY6jBqLagF4njOZMCmQM897Q7Zd5AURPzwKwCL0GShPwveLunDFQcTixBrHkWJ5IMEk4sW6BsVBNwkaqCRXVXbGQFm3bjLIVwmUfgXna0d7KT/gaK0jjI3fuNaU4APZkAmkRS09AslBVXmU7A6wJPv1/SdkUz70pvkMMAj6sdlkta17r8XQunibE1ZrOF/9odv1cOWxhGPgQkEqDXiOsVOLF2y8zaIKer/A3J75Z35RzEum/SmOcIUhQnzq0bOQm6cQ890H/z1w0jZPco1I0dcZUbYnFX44GSV4V+M1/aqeTLdkWU8UJkqb7ZNZjxtkQwMzMzZfS9nElW++oGYanCYS9L4PZE02KulxA8a1ZU1QXN0HVQSWJ+IxYUKGb53kLzGR/+R911OwXEIC+4Ctjh8klAk3G5mxZsYBwfSjjQxmu+FH7PxrVMZYNp7BG38ctJFBF+RYsaupXr0doR0rmBnAcC+tCMRGp8XQHQn8tqpfznEFOQ1J/+3PfEGcmmHlEGXIk3w5m8PpBhsKRYZZMBdmcIhfJvNIqUCN683MsuZfWlXxo2PO/bhV3d7A94hT/lNJ5wA5wynRrgvddwrLs+xifTXwqwa8TkrE2IHO3qPashjvwvRDfaZlsexfbrEErB922mlOgGyryt3jUMS5u4chepNqTBWr/a0vQmbyJNjYIBtLzGiSHuj2PE5CWb9MGr2zp+Jv7Ck5EYuDo5NBJTIASBSxfhzz1ifRgCUHaoeGM3w/eBSRoJES1UFWUMLyVXr1ez+uY21nwLhHPQ0RnMjfhjQPHc64TAM87pgGubj/X/KmenR1GyPFUqvGCFpkCmmopRcllvqTD5djtLWlK04C2w6r8t17vVyN4SK5+pJnaLIQ QCYGmZLF rg+xv1rUoETslZf/tP2X03m4p61/R9cLRodXV7ZJg7ZzOOVMgvArj5Gx8SBkJ7xkLb0nWcLpgys+mghAlJ47S+yO+1zscXjkQ6UqUxENBA1NcfqLMZtycmP9pA5fq44tZxgBokcZfYEaJiTy8w+metd8jvR9KSPyGjxCGUyD0UDRBJPwfbKTNweboEHQy3hE/ccMNXk3JyLD/bm8LswYuVwlGpmkrZJLqv9StmP6TWuT9d4ju2A45N1vUBOSjeV98Yp4EUuNZKF8yvUCuXGW5SeJNqA4vfS1+Mu9LG3U3960vaKwcICDtSciqF7Zonsw7vEPho5u9laXVrooAQUzOCUdzNgG10J0MtLvmqNHUveJxG1y4QHMMOzzdNC3Qt9cTXaoKnRUeFqjyD5YzfKbQckn3RbJOc/pxY1BZX0aYyzFYnYOQ/z8ATXLMZ+kdX6eGIMBnGIDj9U3nfo2SmN1Mza+lOe48D9IqJZSuD87W4+xRdSQkyyvCzknWbbyiG0ZEX44sZCAUkaaJgzw= X-Bogosity: Ham, tests=bogofilter, spamicity=0.000000, version=1.2.4 Sender: owner-linux-mm@kvack.org Precedence: bulk X-Loop: owner-majordomo@kvack.org List-ID: List-Subscribe: List-Unsubscribe: Helpers to install and detect special pmd/pud entries. In short, bit 9 on x86 is not used for pmd/pud, so we can directly define them the same as the pte level. One note is that it's also used in _PAGE_BIT_CPA_TEST but that is only used in the debug test, and shouldn't conflict in this case. One note is that pxx_set|clear_flags() for pmd/pud will need to be moved upper so that they can be referenced by the new special bit helpers. There's no change in the code that was moved. Cc: x86@kernel.org Cc: Thomas Gleixner Cc: Ingo Molnar Cc: Borislav Petkov Cc: Dave Hansen Signed-off-by: Peter Xu --- arch/x86/Kconfig | 1 + arch/x86/include/asm/pgtable.h | 80 ++++++++++++++++++++++------------ 2 files changed, 53 insertions(+), 28 deletions(-) diff --git a/arch/x86/Kconfig b/arch/x86/Kconfig index acd9745bf2ae..7a3fb2ff3e72 100644 --- a/arch/x86/Kconfig +++ b/arch/x86/Kconfig @@ -28,6 +28,7 @@ config X86_64 select ARCH_HAS_GIGANTIC_PAGE select ARCH_SUPPORTS_INT128 if CC_HAS_INT128 select ARCH_SUPPORTS_PER_VMA_LOCK + select ARCH_SUPPORTS_HUGE_PFNMAP if TRANSPARENT_HUGEPAGE select HAVE_ARCH_SOFT_DIRTY select MODULES_USE_ELF_RELA select NEED_DMA_MAP_STATE diff --git a/arch/x86/include/asm/pgtable.h b/arch/x86/include/asm/pgtable.h index a7c1e9cfea41..1e463c9a650f 100644 --- a/arch/x86/include/asm/pgtable.h +++ b/arch/x86/include/asm/pgtable.h @@ -120,6 +120,34 @@ extern pmdval_t early_pmd_flags; #define arch_end_context_switch(prev) do {} while(0) #endif /* CONFIG_PARAVIRT_XXL */ +static inline pmd_t pmd_set_flags(pmd_t pmd, pmdval_t set) +{ + pmdval_t v = native_pmd_val(pmd); + + return native_make_pmd(v | set); +} + +static inline pmd_t pmd_clear_flags(pmd_t pmd, pmdval_t clear) +{ + pmdval_t v = native_pmd_val(pmd); + + return native_make_pmd(v & ~clear); +} + +static inline pud_t pud_set_flags(pud_t pud, pudval_t set) +{ + pudval_t v = native_pud_val(pud); + + return native_make_pud(v | set); +} + +static inline pud_t pud_clear_flags(pud_t pud, pudval_t clear) +{ + pudval_t v = native_pud_val(pud); + + return native_make_pud(v & ~clear); +} + /* * The following only work if pte_present() is true. * Undefined behaviour if not.. @@ -317,6 +345,30 @@ static inline int pud_devmap(pud_t pud) } #endif +#ifdef CONFIG_ARCH_SUPPORTS_PMD_PFNMAP +static inline bool pmd_special(pmd_t pmd) +{ + return pmd_flags(pmd) & _PAGE_SPECIAL; +} + +static inline pmd_t pmd_mkspecial(pmd_t pmd) +{ + return pmd_set_flags(pmd, _PAGE_SPECIAL); +} +#endif /* CONFIG_ARCH_SUPPORTS_PMD_PFNMAP */ + +#ifdef CONFIG_ARCH_SUPPORTS_PUD_PFNMAP +static inline bool pud_special(pud_t pud) +{ + return pud_flags(pud) & _PAGE_SPECIAL; +} + +static inline pud_t pud_mkspecial(pud_t pud) +{ + return pud_set_flags(pud, _PAGE_SPECIAL); +} +#endif /* CONFIG_ARCH_SUPPORTS_PUD_PFNMAP */ + static inline int pgd_devmap(pgd_t pgd) { return 0; @@ -487,20 +539,6 @@ static inline pte_t pte_mkdevmap(pte_t pte) return pte_set_flags(pte, _PAGE_SPECIAL|_PAGE_DEVMAP); } -static inline pmd_t pmd_set_flags(pmd_t pmd, pmdval_t set) -{ - pmdval_t v = native_pmd_val(pmd); - - return native_make_pmd(v | set); -} - -static inline pmd_t pmd_clear_flags(pmd_t pmd, pmdval_t clear) -{ - pmdval_t v = native_pmd_val(pmd); - - return native_make_pmd(v & ~clear); -} - /* See comments above mksaveddirty_shift() */ static inline pmd_t pmd_mksaveddirty(pmd_t pmd) { @@ -595,20 +633,6 @@ static inline pmd_t pmd_mkwrite_novma(pmd_t pmd) pmd_t pmd_mkwrite(pmd_t pmd, struct vm_area_struct *vma); #define pmd_mkwrite pmd_mkwrite -static inline pud_t pud_set_flags(pud_t pud, pudval_t set) -{ - pudval_t v = native_pud_val(pud); - - return native_make_pud(v | set); -} - -static inline pud_t pud_clear_flags(pud_t pud, pudval_t clear) -{ - pudval_t v = native_pud_val(pud); - - return native_make_pud(v & ~clear); -} - /* See comments above mksaveddirty_shift() */ static inline pud_t pud_mksaveddirty(pud_t pud) { From patchwork Fri Aug 9 16:09:08 2024 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Peter Xu X-Patchwork-Id: 13758943 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from kanga.kvack.org (kanga.kvack.org [205.233.56.17]) by smtp.lore.kernel.org (Postfix) with ESMTP id 51028C52D7C for ; Fri, 9 Aug 2024 16:10:14 +0000 (UTC) Received: by kanga.kvack.org (Postfix) id EE7536B0088; Fri, 9 Aug 2024 12:10:01 -0400 (EDT) Received: by kanga.kvack.org (Postfix, from userid 40) id E74F86B00B6; Fri, 9 Aug 2024 12:10:01 -0400 (EDT) X-Delivered-To: int-list-linux-mm@kvack.org Received: by kanga.kvack.org (Postfix, from userid 63042) id C4CD66B00B7; Fri, 9 Aug 2024 12:10:01 -0400 (EDT) X-Delivered-To: linux-mm@kvack.org Received: from relay.hostedemail.com (smtprelay0017.hostedemail.com [216.40.44.17]) by kanga.kvack.org (Postfix) with ESMTP id A2C516B0088 for ; Fri, 9 Aug 2024 12:10:01 -0400 (EDT) Received: from smtpin26.hostedemail.com (a10.router.float.18 [10.200.18.1]) by unirelay05.hostedemail.com (Postfix) with ESMTP id 47ED84129F for ; Fri, 9 Aug 2024 16:10:01 +0000 (UTC) X-FDA: 82433193402.26.FAF89DE Received: from us-smtp-delivery-124.mimecast.com (us-smtp-delivery-124.mimecast.com [170.10.129.124]) by imf21.hostedemail.com (Postfix) with ESMTP id 422EB1C001E for ; Fri, 9 Aug 2024 16:09:59 +0000 (UTC) Authentication-Results: imf21.hostedemail.com; dkim=pass header.d=redhat.com header.s=mimecast20190719 header.b="OCiZ3c/J"; dmarc=pass (policy=none) header.from=redhat.com; spf=pass (imf21.hostedemail.com: domain of peterx@redhat.com designates 170.10.129.124 as permitted sender) smtp.mailfrom=peterx@redhat.com ARC-Seal: i=1; s=arc-20220608; d=hostedemail.com; t=1723219745; a=rsa-sha256; cv=none; b=pC/3BAI1BJiSBWVH+uM6Bqa9G7fS+1WDpPi8zauaU4eZim2M4dUOoLshezaf2OQ14fHiCk fySIYFRws62hKqEp5j52gV+0c5qupeeDuJZM83BexcwOnrneLX7UzSHQUITJ+hKS4J0Pkl mG/AbYootRnxPww/UE5oaH2qSyLxS30= ARC-Authentication-Results: i=1; imf21.hostedemail.com; dkim=pass header.d=redhat.com header.s=mimecast20190719 header.b="OCiZ3c/J"; dmarc=pass (policy=none) header.from=redhat.com; spf=pass (imf21.hostedemail.com: domain of peterx@redhat.com designates 170.10.129.124 as permitted sender) smtp.mailfrom=peterx@redhat.com ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=hostedemail.com; s=arc-20220608; t=1723219745; h=from:from:sender:reply-to:subject:subject:date:date: message-id:message-id:to:to:cc:cc:mime-version:mime-version: content-type:content-type: content-transfer-encoding:content-transfer-encoding: in-reply-to:in-reply-to:references:references:dkim-signature; bh=UM853E440qfXKYz7avWh673RydKsfl7XbVKvi0Id2+c=; b=21VcYP8EQQlEq0BmzOshQ83u0k2rfSsI+QGYjD87BqzYtLIzsD0Y5YJNVrvIaxkJ9xTXkB NH2nyNeCZHLJm7IzHDmhBNeM3N6sg5utTittLvWU5g9ycoZJ2GhLvxq4KWb5HDALsYx7Th JoLSUFRB0MwTYjFJhPNGmfkj8euvwKs= DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=redhat.com; s=mimecast20190719; t=1723219798; h=from:from:reply-to:subject:subject:date:date:message-id:message-id: to:to:cc:cc:mime-version:mime-version:content-type:content-type: content-transfer-encoding:content-transfer-encoding: in-reply-to:in-reply-to:references:references; bh=UM853E440qfXKYz7avWh673RydKsfl7XbVKvi0Id2+c=; b=OCiZ3c/Jna1NjWYCT1GAw6hpwYfDsY7rNuFD414H3z9wlkAn7GoPOGaxcVhGsZN988kBvD 9fc8t8SI6qnvsdyQs2OIbC3F3Ej0xcCSqSvzmsXCt13weLjmwGWDnGdFhWwf6UYi+Ahl2b pt4TngbBDCubf2UKv2KT3JR2b+NUt8I= Received: from mail-qt1-f199.google.com (mail-qt1-f199.google.com [209.85.160.199]) by relay.mimecast.com with ESMTP with STARTTLS (version=TLSv1.3, cipher=TLS_AES_256_GCM_SHA384) id us-mta-221-T8p-ZL9NNT6GhjBeOVOwNg-1; Fri, 09 Aug 2024 12:09:56 -0400 X-MC-Unique: T8p-ZL9NNT6GhjBeOVOwNg-1 Received: by mail-qt1-f199.google.com with SMTP id d75a77b69052e-44ff585970bso4757231cf.2 for ; Fri, 09 Aug 2024 09:09:56 -0700 (PDT) X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20230601; t=1723219795; x=1723824595; h=content-transfer-encoding:mime-version:references:in-reply-to :message-id:date:subject:cc:to:from:x-gm-message-state:from:to:cc :subject:date:message-id:reply-to; bh=UM853E440qfXKYz7avWh673RydKsfl7XbVKvi0Id2+c=; b=KnoLcT8gVTOem67cFGJvHDWuysP6v88iQ4s6leC4q+VWoWRianX9HZfSiCHbpjzdqW qm6tv0gpK2OuZMbtC7CLwFOUHoSW3KmoirohHHe2U8HzJ42Ez9MAUu0IjFpWhE2DdOQn qRIBIsa/SaImoOvzaxulfMjha9dQPdXgJ7RZtiIn+P/+VoReV3jx5MLUbd+rKwAWULJU TBQmTdiHP/SweyrZJ9AlojnCsj3qbkcXyDIyLkEsxzTUC4ukJ9qN0WWvSQs9EC7Zyasx f6M0PjfTBHIAvdAMc1u/yTVMtfXCMsC+MRxUZZ9IChi8eTOJbPLVGMPH4SzvwWctOAZE 42QQ== X-Gm-Message-State: AOJu0Yw4/05SAY72RRck4diQf1UJpsAjK+B+M5/dGzlaqX8nS08CyPdL EyoPQZZzjhIQ2FDCeFM3Vjlxw8aLEXXRGEwZ8XrpY8kzz2++sfuQ9Hkd2wy46scJBpSRS7ma4je BMj7Tqp2kt4PXZJEOhostLxPOW76pZuaIY/nPSbGuKoK8MIobFO9tzejiL25UEsYtClIcMG9bQL WMxVR+4MCWuRfqV7O4+jJHZJwCyWCswz36 X-Received: by 2002:ac8:5e07:0:b0:44e:cff7:3741 with SMTP id d75a77b69052e-45312646da2mr14006871cf.7.1723219795409; Fri, 09 Aug 2024 09:09:55 -0700 (PDT) X-Google-Smtp-Source: AGHT+IEzDYjFq+v9SYAdrw2G1PaR/i1nnMK5UGwVdyzJwk9mK3TcvaabWRsZT43a6L+rPfA/Jc6OAg== X-Received: by 2002:ac8:5e07:0:b0:44e:cff7:3741 with SMTP id d75a77b69052e-45312646da2mr14006381cf.7.1723219794745; Fri, 09 Aug 2024 09:09:54 -0700 (PDT) Received: from x1n.redhat.com (pool-99-254-121-117.cpe.net.cable.rogers.com. [99.254.121.117]) by smtp.gmail.com with ESMTPSA id d75a77b69052e-451c870016csm22526741cf.19.2024.08.09.09.09.52 (version=TLS1_3 cipher=TLS_AES_256_GCM_SHA384 bits=256/256); Fri, 09 Aug 2024 09:09:54 -0700 (PDT) From: Peter Xu To: linux-mm@kvack.org, linux-kernel@vger.kernel.org Cc: Sean Christopherson , Oscar Salvador , Jason Gunthorpe , Axel Rasmussen , linux-arm-kernel@lists.infradead.org, x86@kernel.org, peterx@redhat.com, Will Deacon , Gavin Shan , Paolo Bonzini , Zi Yan , Andrew Morton , Catalin Marinas , Ingo Molnar , Alistair Popple , Borislav Petkov , David Hildenbrand , Thomas Gleixner , kvm@vger.kernel.org, Dave Hansen , Alex Williamson , Yan Zhao Subject: [PATCH 18/19] mm/arm64: Support large pfn mappings Date: Fri, 9 Aug 2024 12:09:08 -0400 Message-ID: <20240809160909.1023470-19-peterx@redhat.com> X-Mailer: git-send-email 2.45.0 In-Reply-To: <20240809160909.1023470-1-peterx@redhat.com> References: <20240809160909.1023470-1-peterx@redhat.com> MIME-Version: 1.0 X-Mimecast-Spam-Score: 0 X-Mimecast-Originator: redhat.com X-Rspamd-Server: rspam12 X-Rspamd-Queue-Id: 422EB1C001E X-Stat-Signature: 1k3oywozmzeku9bhkwdteg6d5wa3omw1 X-Rspam-User: X-HE-Tag: 1723219799-416921 X-HE-Meta: U2FsdGVkX19M20AEtRr5/NE0S87c65NIUEi0dG7pUNpKVd7M2PB0UvMl0qqrUTN1Fcp3XtdhlOXL8jDweMyehrPL5VbdOCfPcDFAnufCV8wX8mCnQSVlDLRWRPhoKUI1t1rdGBOeJe8Jb8FtBNgvTJm68mosBQTaEw/BLkL7h32mC+P84+pohltvqTXzR/xTyEBejOJYhnpaQLGv7h+NhhYJpQ83ZM1eN9X6hWrWIssYnoB10XrjYDjz7oXxSLPVcNRkbXsbRn+f4jVk6XzRiMOOZkVb3HDladSo5FLOm8hTNVfVPan08AamV4chi/AwjDzX3cpykdMDdlWw5OefTPJ9k22F1kUEil9VWegJVLuGsESWGsw4ohWMwsYxjynMhgMsFOxEcKNhBr68pIKq6tWnPQ3goLKdGj6zCQJ8cTA99if56wtc0S/kqpa+/m/DuPNcwSuHFuxsbMa9hPqJKLN9FXMTh7QnAJjLzDUOJlAi7cXfvkjBWBbTT9xGx/9dCliRZ93eK8Y7YRPc4Pf3HYZt63LA2RCL/qp+oiG1snB0u/LsFJwWqSs084cNIfRt4ml9wli2JtDG+UnPBy2ZmYsX7sXmoqgT8DwAHH6l6q0krAmWSoe2LffqrkziCgUH9MvxobqvNugHKTKQTDon8ApQ4LEbFxUlr9uXwKNbDVV7e1bXRolb/R2Slx0zNQWhtxjUwEGxVaCnBZXncpGymiJU63jElpn9MjVZ+bF+MbEZ38eOI2gsHtemKOWjK5O+ILUdbpYakraifXVUzRIwnq1ZaSspmU4HeR93XHg1H5AQ5Q4HDY9n3Cn2EKRucCAe3KSV/NPOwbzylZyEON0ALt8PHN7p72168lDN2x8UA32G47OcLxZVSe9aEa+TXp8hxa21OgSdTWWci+f9EHiVh0XUfBNpwqXvzRVT5/9TUccOU7lWKW/RivulOk1GaMPR554ZLlQCgtjWJj1BpaI pZrkRPd6 mLhuQ8IP5gO73HVRealdT+DD76ekM35Jeacee4/OTqwiCT2NLF3qicouUj9rT1Qyi+1um8CiDdb076UCaB1yUSAN027rjceVCTLAbWjLD/TTNSSZx0lKh0Cu4XDrnHwoYFtzmZg5qPkleHDbPVuRinlxgdlJ3PSwuttHRQY6d1mf3zQH3disXzfJldfa7s1MiN72NSzqh0HLQ0cJzUBxYkOcGllxO/98aKJFEAYYnr0uHJiqSi+Cu4+fXgD9H+IPbc8OquekrRwOPkHT+vSSJY3GrHoofnkJ9SMcpdV+WkSSR5S1vFkIzbq+yBRHZJQQlu/begCd0ypwZMmgoO0lDCL09MlgyC/u8YV2+7CVbCfOgKGD8yOZd/r85QN1ZOnbDPJRCFRg+vLPVDrhiAgAiKhmgAwRfjQPc4atwfE4Y726R/Ml8wMM4WZ4c+8o7e1aDpDKQO5RSzaW/Wgxc0HXqJxKSiEAwiDSoLo4b0fFKvXuxHv4obn0gDREunlhloKsFEgIvldhO5iY97JgTA6hAmSmRT7Ptz8Lqb+jaRcUFZptEsELT+mFjE8o1E6gmokTLP4MYLvVLkomXKEkxTVk/Q/mDyCUDBDdK9OUHj/zJcCHcj68= X-Bogosity: Ham, tests=bogofilter, spamicity=0.000000, version=1.2.4 Sender: owner-linux-mm@kvack.org Precedence: bulk X-Loop: owner-majordomo@kvack.org List-ID: List-Subscribe: List-Unsubscribe: Support huge pfnmaps by using bit 56 (PTE_SPECIAL) for "special" on pmds/puds. Provide the pmd/pud helpers to set/get special bit. There's one more thing missing for arm64 which is the pxx_pgprot() for pmd/pud. Add them too, which is mostly the same as the pte version by dropping the pfn field. These helpers are essential to be used in the new follow_pfnmap*() API to report valid pgprot_t results. Note that arm64 doesn't yet support huge PUD yet, but it's still straightforward to provide the pud helpers that we need altogether. Only PMD helpers will make an immediate benefit until arm64 will support huge PUDs first in general (e.g. in THPs). Cc: linux-arm-kernel@lists.infradead.org Cc: Catalin Marinas Cc: Will Deacon Signed-off-by: Peter Xu --- arch/arm64/Kconfig | 1 + arch/arm64/include/asm/pgtable.h | 29 +++++++++++++++++++++++++++++ 2 files changed, 30 insertions(+) diff --git a/arch/arm64/Kconfig b/arch/arm64/Kconfig index b3fc891f1544..5f026b95f309 100644 --- a/arch/arm64/Kconfig +++ b/arch/arm64/Kconfig @@ -99,6 +99,7 @@ config ARM64 select ARCH_SUPPORTS_NUMA_BALANCING select ARCH_SUPPORTS_PAGE_TABLE_CHECK select ARCH_SUPPORTS_PER_VMA_LOCK + select ARCH_SUPPORTS_HUGE_PFNMAP if TRANSPARENT_HUGEPAGE select ARCH_WANT_BATCHED_UNMAP_TLB_FLUSH select ARCH_WANT_COMPAT_IPC_PARSE_VERSION if COMPAT select ARCH_WANT_DEFAULT_BPF_JIT diff --git a/arch/arm64/include/asm/pgtable.h b/arch/arm64/include/asm/pgtable.h index b78cc4a6758b..2faecc033a19 100644 --- a/arch/arm64/include/asm/pgtable.h +++ b/arch/arm64/include/asm/pgtable.h @@ -578,6 +578,14 @@ static inline pmd_t pmd_mkdevmap(pmd_t pmd) return pte_pmd(set_pte_bit(pmd_pte(pmd), __pgprot(PTE_DEVMAP))); } +#ifdef CONFIG_ARCH_SUPPORTS_PMD_PFNMAP +#define pmd_special(pte) (!!((pmd_val(pte) & PTE_SPECIAL))) +static inline pmd_t pmd_mkspecial(pmd_t pmd) +{ + return set_pmd_bit(pmd, __pgprot(PTE_SPECIAL)); +} +#endif + #define __pmd_to_phys(pmd) __pte_to_phys(pmd_pte(pmd)) #define __phys_to_pmd_val(phys) __phys_to_pte_val(phys) #define pmd_pfn(pmd) ((__pmd_to_phys(pmd) & PMD_MASK) >> PAGE_SHIFT) @@ -595,6 +603,27 @@ static inline pmd_t pmd_mkdevmap(pmd_t pmd) #define pud_pfn(pud) ((__pud_to_phys(pud) & PUD_MASK) >> PAGE_SHIFT) #define pfn_pud(pfn,prot) __pud(__phys_to_pud_val((phys_addr_t)(pfn) << PAGE_SHIFT) | pgprot_val(prot)) +#ifdef CONFIG_ARCH_SUPPORTS_PUD_PFNMAP +#define pud_special(pte) pte_special(pud_pte(pud)) +#define pud_mkspecial(pte) pte_pud(pte_mkspecial(pud_pte(pud))) +#endif + +#define pmd_pgprot pmd_pgprot +static inline pgprot_t pmd_pgprot(pmd_t pmd) +{ + unsigned long pfn = pmd_pfn(pmd); + + return __pgprot(pmd_val(pfn_pmd(pfn, __pgprot(0))) ^ pmd_val(pmd)); +} + +#define pud_pgprot pud_pgprot +static inline pgprot_t pud_pgprot(pud_t pud) +{ + unsigned long pfn = pud_pfn(pud); + + return __pgprot(pud_val(pfn_pud(pfn, __pgprot(0))) ^ pud_val(pud)); +} + static inline void __set_pte_at(struct mm_struct *mm, unsigned long __always_unused addr, pte_t *ptep, pte_t pte, unsigned int nr) From patchwork Fri Aug 9 16:09:09 2024 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Peter Xu X-Patchwork-Id: 13758944 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from kanga.kvack.org (kanga.kvack.org [205.233.56.17]) by smtp.lore.kernel.org (Postfix) with ESMTP id D4B49C52D7C for ; Fri, 9 Aug 2024 16:10:17 +0000 (UTC) Received: by kanga.kvack.org (Postfix) id B8A376B00B9; Fri, 9 Aug 2024 12:10:08 -0400 (EDT) Received: by kanga.kvack.org (Postfix, from userid 40) id AEB186B00BA; Fri, 9 Aug 2024 12:10:08 -0400 (EDT) X-Delivered-To: int-list-linux-mm@kvack.org Received: by kanga.kvack.org (Postfix, from userid 63042) id 93DC66B00BB; Fri, 9 Aug 2024 12:10:08 -0400 (EDT) X-Delivered-To: linux-mm@kvack.org Received: from relay.hostedemail.com (smtprelay0013.hostedemail.com [216.40.44.13]) by kanga.kvack.org (Postfix) with ESMTP id 6F7F26B00B9 for ; Fri, 9 Aug 2024 12:10:08 -0400 (EDT) Received: from smtpin11.hostedemail.com (a10.router.float.18 [10.200.18.1]) by unirelay07.hostedemail.com (Postfix) with ESMTP id 2C0D1161242 for ; Fri, 9 Aug 2024 16:10:08 +0000 (UTC) X-FDA: 82433193696.11.6BC8EC8 Received: from us-smtp-delivery-124.mimecast.com (us-smtp-delivery-124.mimecast.com [170.10.133.124]) by imf13.hostedemail.com (Postfix) with ESMTP id F15922002E for ; Fri, 9 Aug 2024 16:10:05 +0000 (UTC) Authentication-Results: imf13.hostedemail.com; dkim=pass header.d=redhat.com header.s=mimecast20190719 header.b=OzyMs3Hq; spf=pass (imf13.hostedemail.com: domain of peterx@redhat.com designates 170.10.133.124 as permitted sender) smtp.mailfrom=peterx@redhat.com; dmarc=pass (policy=none) header.from=redhat.com ARC-Seal: i=1; s=arc-20220608; d=hostedemail.com; t=1723219772; a=rsa-sha256; cv=none; b=AOFSf9agwj5MoLghfhc9m2s28ZdawyMLq3DMqRHWRd47krrUwek+BnwbaJ+kD94ugkKb79 pMHWfe4KsOL6a1LejjzVqSVO/m/lV5XSRaMmcBHdoa3l/JgLZIkQUum7VNBQt+Z3SYlwVg foYW6Tda1LLqfyXSd4JvoPElItM+BzE= ARC-Authentication-Results: i=1; imf13.hostedemail.com; dkim=pass header.d=redhat.com header.s=mimecast20190719 header.b=OzyMs3Hq; spf=pass (imf13.hostedemail.com: domain of peterx@redhat.com designates 170.10.133.124 as permitted sender) smtp.mailfrom=peterx@redhat.com; dmarc=pass (policy=none) header.from=redhat.com ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=hostedemail.com; s=arc-20220608; t=1723219772; h=from:from:sender:reply-to:subject:subject:date:date: message-id:message-id:to:to:cc:cc:mime-version:mime-version: content-type:content-type: content-transfer-encoding:content-transfer-encoding: in-reply-to:in-reply-to:references:references:dkim-signature; bh=9MPOki3YF/wyEgBnejzItxwjIF1moGifGifGb5OwzY4=; b=ZuxbhJTEmyvVitdF8dMwph69q8J8jFGoN74z1I241hpXcswSn0RUi05DChOaVMrP8eN8AX 6NUAYKxvb5np8PE27ZmloGXOYBV9Qz6Oeo6Tp3/px8ODumENTq2ex6+OxLDHpHpfqwnau2 IurbF5H7jDmm1drYDP58LwJ1t1Qgnds= DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=redhat.com; s=mimecast20190719; t=1723219805; h=from:from:reply-to:subject:subject:date:date:message-id:message-id: to:to:cc:cc:mime-version:mime-version:content-type:content-type: content-transfer-encoding:content-transfer-encoding: in-reply-to:in-reply-to:references:references; bh=9MPOki3YF/wyEgBnejzItxwjIF1moGifGifGb5OwzY4=; b=OzyMs3HqYEtNjYS/7rcDP1LadYYk4UWvwse0/DbNl2Jk3onuCjvZ5N3ULszOvEOw2NjGl7 UoMOJ6pYQCMlU3UeO2LIldMuU+xT6nvrdo7PLTmXS2hqbRrIIXn+2n4afxJnHEXqEY6oio JGfHTAx9OAFYM5X7dtE/B06GT7kVuUE= Received: from mail-qk1-f198.google.com (mail-qk1-f198.google.com [209.85.222.198]) by relay.mimecast.com with ESMTP with STARTTLS (version=TLSv1.3, cipher=TLS_AES_256_GCM_SHA384) id us-mta-198-Hlxief1hPo6gZVkAOH9FWg-1; Fri, 09 Aug 2024 12:10:02 -0400 X-MC-Unique: Hlxief1hPo6gZVkAOH9FWg-1 Received: by mail-qk1-f198.google.com with SMTP id af79cd13be357-7a1e1828321so6887985a.1 for ; Fri, 09 Aug 2024 09:10:02 -0700 (PDT) X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20230601; t=1723219797; x=1723824597; h=content-transfer-encoding:mime-version:references:in-reply-to :message-id:date:subject:cc:to:from:x-gm-message-state:from:to:cc :subject:date:message-id:reply-to; bh=9MPOki3YF/wyEgBnejzItxwjIF1moGifGifGb5OwzY4=; b=KHsPbLh9/V/Ng230BErFuSgKsHG7+OH2Kju9d2g3xdED+6wuJBK1DLyOmN6yuaH3RI Dox+IZBPSGWaGB3En+kjvI8KfjXxPnzUptpCBELH+nJjhC743qTuYV310nETLvwMp6Py 8I7VbvbH1Mzc8S+gmRLsFLeeND6+fuPD6/OzkttDdyTCsjRVIGFt7dQ82kXkPg9sE8kf uMuYksuzkcjWaIF+Q7TedvOaU6/N3YeHU34DHzXoIYzPVzyA4+3OlnKd3vzGj53FT/VY ZUHhPiYxPIeAo7cFRCgTenhsAmOlOPgiqhDBL3gCYVHjTSOty/g23SoY1ftyJjT2iVbn LV2A== X-Gm-Message-State: AOJu0Yw2AlwD5b4s3ixVmA2M40je/mYRewG3ADyzInYZBP+1K/HuhqId 2fdEHKmxOyjCVQ4NpwiQQ2g9xR0M5fzuwnSRIUfvHVU6F9ktU1n4o70y6NlnQdnIncSKC3yY4zg pAbOLTfduSPEZUDy1hQgQtjslAmaYVRLnbYwQeWgU4qhQXsjKAFwQDzyYLhGhkiquUxlafzuhm3 bSasnxWCNYOXPtr00xJxxQZD0sKC/LjCO5 X-Received: by 2002:a05:622a:1b8d:b0:446:58f5:e6ff with SMTP id d75a77b69052e-4531251d52cmr13545211cf.2.1723219797424; Fri, 09 Aug 2024 09:09:57 -0700 (PDT) X-Google-Smtp-Source: AGHT+IHJY3GNDsLRixrwL3Yd/1vgrL/iS2SbMHaD3fo6KrtERFLLt1PJ4ncszNtj/1mVMrjvOJRHSw== X-Received: by 2002:a05:622a:1b8d:b0:446:58f5:e6ff with SMTP id d75a77b69052e-4531251d52cmr13544641cf.2.1723219796830; Fri, 09 Aug 2024 09:09:56 -0700 (PDT) Received: from x1n.redhat.com (pool-99-254-121-117.cpe.net.cable.rogers.com. [99.254.121.117]) by smtp.gmail.com with ESMTPSA id d75a77b69052e-451c870016csm22526741cf.19.2024.08.09.09.09.55 (version=TLS1_3 cipher=TLS_AES_256_GCM_SHA384 bits=256/256); Fri, 09 Aug 2024 09:09:56 -0700 (PDT) From: Peter Xu To: linux-mm@kvack.org, linux-kernel@vger.kernel.org Cc: Sean Christopherson , Oscar Salvador , Jason Gunthorpe , Axel Rasmussen , linux-arm-kernel@lists.infradead.org, x86@kernel.org, peterx@redhat.com, Will Deacon , Gavin Shan , Paolo Bonzini , Zi Yan , Andrew Morton , Catalin Marinas , Ingo Molnar , Alistair Popple , Borislav Petkov , David Hildenbrand , Thomas Gleixner , kvm@vger.kernel.org, Dave Hansen , Alex Williamson , Yan Zhao Subject: [PATCH 19/19] vfio/pci: Implement huge_fault support Date: Fri, 9 Aug 2024 12:09:09 -0400 Message-ID: <20240809160909.1023470-20-peterx@redhat.com> X-Mailer: git-send-email 2.45.0 In-Reply-To: <20240809160909.1023470-1-peterx@redhat.com> References: <20240809160909.1023470-1-peterx@redhat.com> MIME-Version: 1.0 X-Mimecast-Spam-Score: 0 X-Mimecast-Originator: redhat.com X-Stat-Signature: 1ub5zdx4ywobbtjsyppt1wbefnw74x5c X-Rspamd-Queue-Id: F15922002E X-Rspam-User: X-Rspamd-Server: rspam10 X-HE-Tag: 1723219805-21848 X-HE-Meta: U2FsdGVkX18l8ie4JyPP45xnnFlox2TmaBBBK0G2pU3OQ62LYBAFIZcLW9xItZG+DP7WL3FvDQ9AzqDwhTZAxImRwd1dHA0BPWXAT9ai1yi2hDthcorFmWPRwuNTMdLi8bwOH/TULBlmRG5ZarY/bzkGfL4+UKBgKX311Z9+fKI6bR727O4wI3Z/JGyldBR7KKeIwwf9IdO3g3GcDz367nMPihEHUWQt5UsZ8BmypKY02m6d7+0+WaFgEEDVMnTfNrl/lKhE2kxcC50OAzZO+Y08k8b5ZCd38MNPJe+O+tY3R4nA9+X98341hlRHBoYuiXyybdSEIO0pkkU4K7AeC6du+R48JfNIR+kx8izf4fE/F/SwdvuZUm6SIJqcSIvOJkBU8zZhDljVwzMBB8ziHCUPdns/cYSofUSWWxGrycYMubxj50MwSxFZcpY9CaJqlA3YfPWksvEpXytKRyYtpY7iEwER3b5Udjb83lDT8suNL5lYQEwoqmrGjYwUz1rRlci00jsci4E5OxbNogO7wZ7RxK9dkwRq+dsHHOjGe351D8revDERUBWDsscHr30uoVEzxzOAVp2xLmbO/cBoU5BFsCiMPGJqkA4SgK/Qcj1zRm6I/YdplbvzgmzfD5zrMwq2hvLT7lJwhZGNgQupfvGHvhYtv/3HmDaQB2cv9Uaa5p99HncAJlykMN2xpU6DfvCANFe2km+Hi6LXj7aI167oOHwvm1LkmC5IiSaOqczDVsIJUwVpSPqZ2KgZG7yl2I4o5N/boCZj7+sQV8ty2lhnL1iLOlz+T1KWH7EbTxKh7aZJkV8cNH2QMsV1ujCjtrAGYDFR77+DuFqHY05ebCfXYvTEccG4zdGM7OO73LSNetPgXUA+wPDoyOrQiEpOGypezabllahhKRHmGchVShyJyg0gTbV/OO41BUeNEae5TzWVLGWTgpTjvcn/Jm2fEY3nPXu4/It/i/PsQXM YMRJOH3n scRrh/10Ei6R5BUI7fAq8aVzCaUTgzlkEIsvRsk9RbTeYsrtinJxYNC3ItpiSjNiwq2C/dkXyLB+k9lm3Py4/kzL+T1F3eLmcpnpHd8diGtkajt1SjSMdEVNYBQmm10V7HNrn/hUiqnJxnMh6O9i0eqqMKfOfqQYtIuP3QSzjY8ZUijSJbA38fW9XQo/dDvEG7dJ0NR4ChPi0Trf1HQuTee4qw1JZomyW2HeHngzuQE42LWRZbCQVhmHa/4LMk1/5wW/yRA1WypNDmktDgw6gDNhhkIQo4XKPuQ9bbJfhfzv922kcDxSonkK1LtcUUpk9gifmI6qbwj/vtNpE8YIn0Wg7uHZnHtwJA90DdrIiGT1BwUi9leAIGYxw0oRfUlHHUScIGA8dr46hiD8X2HsLrfxRXZqz8zdakPjsT/kWQMwuOcp802sAaQ18ccuuYWqfOL3/eeM9EeVc+ZzFmczCFl6asrYi/BSLyUjBi75P3GAktfTGPAmabUU77JYKYo2k05StIDkYvFH0/gt7LbthWthvQ0KaZYAVWQxbik2X0QCxoHmJjklUoxOuLyiUIRXaUxwh X-Bogosity: Ham, tests=bogofilter, spamicity=0.000000, version=1.2.4 Sender: owner-linux-mm@kvack.org Precedence: bulk X-Loop: owner-majordomo@kvack.org List-ID: List-Subscribe: List-Unsubscribe: From: Alex Williamson With the addition of pfnmap support in vmf_insert_pfn_{pmd,pud}() we can take advantage of PMD and PUD faults to PCI BAR mmaps and create more efficient mappings. PCI BARs are always a power of two and will typically get at least PMD alignment without userspace even trying. Userspace alignment for PUD mappings is also not too difficult. Consolidate faults through a single handler with a new wrapper for standard single page faults. The pre-faulting behavior of commit d71a989cf5d9 ("vfio/pci: Insert full vma on mmap'd MMIO fault") is removed in this refactoring since huge_fault will cover the bulk of the faults and results in more efficient page table usage. We also want to avoid that pre-faulted single page mappings preempt huge page mappings. Cc: kvm@vger.kernel.org Signed-off-by: Alex Williamson Signed-off-by: Peter Xu --- drivers/vfio/pci/vfio_pci_core.c | 60 +++++++++++++++++++++++--------- 1 file changed, 43 insertions(+), 17 deletions(-) diff --git a/drivers/vfio/pci/vfio_pci_core.c b/drivers/vfio/pci/vfio_pci_core.c index ba0ce0075b2f..2d7478e9a62d 100644 --- a/drivers/vfio/pci/vfio_pci_core.c +++ b/drivers/vfio/pci/vfio_pci_core.c @@ -20,6 +20,7 @@ #include #include #include +#include #include #include #include @@ -1657,14 +1658,20 @@ static unsigned long vma_to_pfn(struct vm_area_struct *vma) return (pci_resource_start(vdev->pdev, index) >> PAGE_SHIFT) + pgoff; } -static vm_fault_t vfio_pci_mmap_fault(struct vm_fault *vmf) +static vm_fault_t vfio_pci_mmap_huge_fault(struct vm_fault *vmf, + unsigned int order) { struct vm_area_struct *vma = vmf->vma; struct vfio_pci_core_device *vdev = vma->vm_private_data; unsigned long pfn, pgoff = vmf->pgoff - vma->vm_pgoff; - unsigned long addr = vma->vm_start; vm_fault_t ret = VM_FAULT_SIGBUS; + if (order && (vmf->address & ((PAGE_SIZE << order) - 1) || + vmf->address + (PAGE_SIZE << order) > vma->vm_end)) { + ret = VM_FAULT_FALLBACK; + goto out; + } + pfn = vma_to_pfn(vma); down_read(&vdev->memory_lock); @@ -1672,30 +1679,49 @@ static vm_fault_t vfio_pci_mmap_fault(struct vm_fault *vmf) if (vdev->pm_runtime_engaged || !__vfio_pci_memory_enabled(vdev)) goto out_unlock; - ret = vmf_insert_pfn(vma, vmf->address, pfn + pgoff); - if (ret & VM_FAULT_ERROR) - goto out_unlock; - - /* - * Pre-fault the remainder of the vma, abort further insertions and - * supress error if fault is encountered during pre-fault. - */ - for (; addr < vma->vm_end; addr += PAGE_SIZE, pfn++) { - if (addr == vmf->address) - continue; - - if (vmf_insert_pfn(vma, addr, pfn) & VM_FAULT_ERROR) - break; + switch (order) { + case 0: + ret = vmf_insert_pfn(vma, vmf->address, pfn + pgoff); + break; +#ifdef CONFIG_ARCH_SUPPORTS_PMD_PFNMAP + case PMD_ORDER: + ret = vmf_insert_pfn_pmd(vmf, __pfn_to_pfn_t(pfn + pgoff, + PFN_DEV), false); + break; +#endif +#ifdef CONFIG_ARCH_SUPPORTS_PUD_PFNMAP + case PUD_ORDER: + ret = vmf_insert_pfn_pud(vmf, __pfn_to_pfn_t(pfn + pgoff, + PFN_DEV), false); + break; +#endif + default: + ret = VM_FAULT_FALLBACK; } out_unlock: up_read(&vdev->memory_lock); +out: + dev_dbg_ratelimited(&vdev->pdev->dev, + "%s(,order = %d) BAR %ld page offset 0x%lx: 0x%x\n", + __func__, order, + vma->vm_pgoff >> + (VFIO_PCI_OFFSET_SHIFT - PAGE_SHIFT), + pgoff, (unsigned int)ret); return ret; } +static vm_fault_t vfio_pci_mmap_page_fault(struct vm_fault *vmf) +{ + return vfio_pci_mmap_huge_fault(vmf, 0); +} + static const struct vm_operations_struct vfio_pci_mmap_ops = { - .fault = vfio_pci_mmap_fault, + .fault = vfio_pci_mmap_page_fault, +#ifdef CONFIG_ARCH_SUPPORTS_HUGE_PFNMAP + .huge_fault = vfio_pci_mmap_huge_fault, +#endif }; int vfio_pci_core_mmap(struct vfio_device *core_vdev, struct vm_area_struct *vma)