From patchwork Mon Aug 26 20:43:35 2024 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Peter Xu X-Patchwork-Id: 13778394 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from kanga.kvack.org (kanga.kvack.org [205.233.56.17]) by smtp.lore.kernel.org (Postfix) with ESMTP id C87C4C5472C for ; Mon, 26 Aug 2024 20:44:05 +0000 (UTC) Received: by kanga.kvack.org (Postfix) id 1C6026B0082; Mon, 26 Aug 2024 16:44:04 -0400 (EDT) Received: by kanga.kvack.org (Postfix, from userid 40) id 12E716B0088; Mon, 26 Aug 2024 16:44:04 -0400 (EDT) X-Delivered-To: int-list-linux-mm@kvack.org Received: by kanga.kvack.org (Postfix, from userid 63042) id D865E6B0083; Mon, 26 Aug 2024 16:44:03 -0400 (EDT) X-Delivered-To: linux-mm@kvack.org Received: from relay.hostedemail.com (smtprelay0015.hostedemail.com [216.40.44.15]) by kanga.kvack.org (Postfix) with ESMTP id B342F6B0082 for ; Mon, 26 Aug 2024 16:44:03 -0400 (EDT) Received: from smtpin17.hostedemail.com (a10.router.float.18 [10.200.18.1]) by unirelay09.hostedemail.com (Postfix) with ESMTP id 24DEA813F8 for ; Mon, 26 Aug 2024 20:44:03 +0000 (UTC) X-FDA: 82495573566.17.D8FDDB1 Received: from us-smtp-delivery-124.mimecast.com (us-smtp-delivery-124.mimecast.com [170.10.129.124]) by imf22.hostedemail.com (Postfix) with ESMTP id F39BBC0016 for ; Mon, 26 Aug 2024 20:44:00 +0000 (UTC) Authentication-Results: imf22.hostedemail.com; dkim=pass header.d=redhat.com header.s=mimecast20190719 header.b=CA7zNfCF; spf=pass (imf22.hostedemail.com: domain of peterx@redhat.com designates 170.10.129.124 as permitted sender) smtp.mailfrom=peterx@redhat.com; dmarc=pass (policy=none) header.from=redhat.com ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=hostedemail.com; s=arc-20220608; t=1724704954; h=from:from:sender:reply-to:subject:subject:date:date: message-id:message-id:to:to:cc:cc:mime-version:mime-version: content-type:content-type: content-transfer-encoding:content-transfer-encoding: in-reply-to:in-reply-to:references:references:dkim-signature; bh=2iIRbDNVyj1lHjlop0tZMw2JlHFFftRtxMxE0Q8z7r8=; b=OCTniikZddJpJLc8fJliw5ZJ7tpEkcKJchEWaN/kIUfGSMR093EawSfVDNKqBisaS5D0wq ImcItjFveDGY0ZNo+QLnlw1FZanPIaGDjdBB+t67RiF10R3lYCPOb/bbCUXGeROCKgRw3+ zT+jjdkalfIBDiynidKgDj+gWiopf08= ARC-Seal: i=1; s=arc-20220608; d=hostedemail.com; t=1724704954; a=rsa-sha256; cv=none; b=QjNQFK4FgClozr/jY/NYNnhqZR+9b5wiyOuC9baI0GK6c0yGXcjfppjp2m/BqEobZuJXP7 Kd1JZ2CozeJVNUc14FzbReAQL2As0rGTAsmF3+RNRQTzJ4ScwbYjElYXonYig2GbnWDy9f cSnPcaQEoKq6thEZYGRd89ZwahFlv6Q= ARC-Authentication-Results: i=1; imf22.hostedemail.com; dkim=pass header.d=redhat.com header.s=mimecast20190719 header.b=CA7zNfCF; spf=pass (imf22.hostedemail.com: domain of peterx@redhat.com designates 170.10.129.124 as permitted sender) smtp.mailfrom=peterx@redhat.com; dmarc=pass (policy=none) header.from=redhat.com DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=redhat.com; s=mimecast20190719; t=1724705040; h=from:from:reply-to:subject:subject:date:date:message-id:message-id: to:to:cc:cc:mime-version:mime-version:content-type:content-type: content-transfer-encoding:content-transfer-encoding: in-reply-to:in-reply-to:references:references; bh=2iIRbDNVyj1lHjlop0tZMw2JlHFFftRtxMxE0Q8z7r8=; b=CA7zNfCF0xUY8jKJ9JgY/kSX2J8YK01F40vOchM0sYYa4NDNpDHxg9BeNO4hagmivubtyG YlZAgkrrQI78kEmy0yTimIsepL55GhwXOU5gUt2wPO9I3SedP/HrTTKKZ9SceYnSXIA30L e1107/ltOvoiEiQ0f5QPulh9HawiSEQ= Received: from mail-oo1-f70.google.com (mail-oo1-f70.google.com [209.85.161.70]) by relay.mimecast.com with ESMTP with STARTTLS (version=TLSv1.3, cipher=TLS_AES_256_GCM_SHA384) id us-mta-355-y9rnmEz8NU2Ygaoy134OUA-1; Mon, 26 Aug 2024 16:43:59 -0400 X-MC-Unique: y9rnmEz8NU2Ygaoy134OUA-1 Received: by mail-oo1-f70.google.com with SMTP id 006d021491bc7-5d5ba96af29so6521244eaf.0 for ; Mon, 26 Aug 2024 13:43:59 -0700 (PDT) X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20230601; t=1724705038; x=1725309838; h=content-transfer-encoding:mime-version:references:in-reply-to :message-id:date:subject:cc:to:from:x-gm-message-state:from:to:cc :subject:date:message-id:reply-to; bh=2iIRbDNVyj1lHjlop0tZMw2JlHFFftRtxMxE0Q8z7r8=; b=kk0PHQ/vwRxKjwp3fmjHqjWFlfkB6wTBXpwRnPqEYhRCQKSMkqRdWN+RPbdC2qoPgE 2tnC3QaQJb6BONsOwxKbHyL9865ykNapW6MryEuBjeO62VWp4uPthThMOPJwtrri50D9 q3Rkz4wtvnwSE0A4YNopV5Fk/HhiDHvUEifRYVpepY0hDwWIH3oYuvuPVmXb1oy0y8Hz Z9dMYQHkPuJyA4fQ/l8b4K6MLlMy6j8JGVvIIVJ4Yn6UAHONS5PQfgJ8M8AXKc/vKqPR 0UJMKFRLmBurVikbrOrtiCG/ZdsfsTBUC9g4A6Or531BumZoE9PlHZyWaz2UcilIVdMI 39zQ== X-Forwarded-Encrypted: i=1; AJvYcCXOG9MCHglDOy+PQ3skybupEUc3CX+ovkw7ecpuozoOwsfpBhyh0YMhXvtSMjncMD7JCB+KPfB5yA==@kvack.org X-Gm-Message-State: AOJu0YzDKmjgnu8LPxB6Sr5eo/DfKez8lSzqro53gYPOdNRpJdTMPO+q T4+OENnNiYmHgqQlISsuRTBr3JSJ+kbVm7BQdz7WbFDnHk13jDyGfGuDgropuEYe0cbRLcpRbAB wBSqSGcX0LXmFUE2C4xK/qo2/CREJ6CmhignNTcV4y7e511M/ X-Received: by 2002:a05:6358:52ca:b0:1b5:a032:9994 with SMTP id e5c5f4694b2df-1b5ebe5b1b3mr113536255d.0.1724705038386; Mon, 26 Aug 2024 13:43:58 -0700 (PDT) X-Google-Smtp-Source: AGHT+IHkrFRhmUpfOOmJ2YwtqClR5K8A29/GQ8sXabHeZ52sv0b77XENUJCQ9LJLwxTjS7HZUYg0sQ== X-Received: by 2002:a05:6358:52ca:b0:1b5:a032:9994 with SMTP id e5c5f4694b2df-1b5ebe5b1b3mr113532855d.0.1724705038035; Mon, 26 Aug 2024 13:43:58 -0700 (PDT) Received: from x1n.redhat.com (pool-99-254-121-117.cpe.net.cable.rogers.com. [99.254.121.117]) by smtp.gmail.com with ESMTPSA id af79cd13be357-7a67f3fd6c1sm491055185a.121.2024.08.26.13.43.56 (version=TLS1_3 cipher=TLS_AES_256_GCM_SHA384 bits=256/256); Mon, 26 Aug 2024 13:43:57 -0700 (PDT) From: Peter Xu To: linux-kernel@vger.kernel.org, linux-mm@kvack.org Cc: Gavin Shan , Catalin Marinas , x86@kernel.org, Ingo Molnar , Andrew Morton , Paolo Bonzini , Dave Hansen , Thomas Gleixner , Alistair Popple , kvm@vger.kernel.org, linux-arm-kernel@lists.infradead.org, Sean Christopherson , peterx@redhat.com, Oscar Salvador , Jason Gunthorpe , Borislav Petkov , Zi Yan , Axel Rasmussen , David Hildenbrand , Yan Zhao , Will Deacon , Kefeng Wang , Alex Williamson Subject: [PATCH v2 01/19] mm: Introduce ARCH_SUPPORTS_HUGE_PFNMAP and special bits to pmd/pud Date: Mon, 26 Aug 2024 16:43:35 -0400 Message-ID: <20240826204353.2228736-2-peterx@redhat.com> X-Mailer: git-send-email 2.45.0 In-Reply-To: <20240826204353.2228736-1-peterx@redhat.com> References: <20240826204353.2228736-1-peterx@redhat.com> MIME-Version: 1.0 X-Mimecast-Spam-Score: 0 X-Mimecast-Originator: redhat.com X-Stat-Signature: km6uekbzhx64ckxmoygj6wxnyen9xu11 X-Rspamd-Queue-Id: F39BBC0016 X-Rspam-User: X-Rspamd-Server: rspam08 X-HE-Tag: 1724705040-672687 X-HE-Meta: U2FsdGVkX18hXbVwUgKuzTt1Wzwt2EynbH7Rs3DWrCiUa17s+FJzpLdZwD/3nMHc3FhWXXj3oG3lyh60Zv5PHzmIgzeSkPlFTEvg5yh1DLrHHcD3d3IjqjDJLoWnojHP3qHnmN2t5JfckLxgFWaMGq11mqTTAtYmmjk5nduRZ+XSSKEa3CUQxc0z6AqrKhLzx0yL72svAbi0tRKOnPfFvZpYV+MCmdBaEaTAxhoJOXCewQd7CBIgpScPgHk/x/ho4s5LUthCaS0gtp/FPFleeF+NI0YE9+HXFQPb2SXeLtcyaKSGjlnifPPKRguTkxORTPis4f+pyzypMbTOXbHrZOI7ps70aOifVlD4sDiOxUmvCSCEOB7ndv7NCYXyfbQCr1vddPnqcNae1YoAhyWBs7j1QCka2/4+LpCOhT6nw20cxeawpwaB+rZDG3Y2aQYPnxZmZxWR4c86g6+CjTeAqLMFTIRWy5zHptisi1fVZeMJbWJKWVrIPF1EyUeR7FzwAAiZsccwRQNg5zNNQq4snlfAyvHeoZO3O7DjVAsRt5jiPG/AzePjHD8SEAJxiCViw3ijem1MUBtE5ov6Pg2jBuKbaEkNwOBmsI3ZFu3dQ7Wigw4/BmSWk58m7+mjJuJS8v0zh6F4yYC/BxehLl4WL5H6DtakiocADErSK6VGQbaMQxMRGOlYrhByboFqdTfwanrI0K+oT/AbY1VQS4btLLpKiui9YYvO4C3dFLip6olnXPGIDCSjbpqEFuUR+PfV30/2pyzs9gckZPbajUQCVe4hyfF4C+A6WfpIIVLVBEGKYyX5vDhAO8Z1iZgcL591gTHWiUhfmU5vAeUQ+kk4YeS3b76HsRtz/0no6osLhb3orWDt/TCcOXs97h7F6oCDSlGiU1KjFYHmYC2VfVx/+vGGH/0tVQTkhk7fq3gz/NGiOodQMK8q+X4jdMP0Uf6xY5+fIV2k7mtT37z4yXu ZUtP4TpU vR3WiN7cYH5+uNPip+LXUlQb9nFhcD9vPHnX/jv7zRss+iEXGE2F9zEmRnReA1l5w8QVxdDGFmyDJNsaGFpQ5f5ezRMs1iNkSu1lMfMHIpVkdh1gAc/1RIv9w4SqhMG2hQ16J3U71YfgjwuOxxsjN7Pp8FMlSwrX4yvAR16H2M24wOIil27QSv/GC/d3GM5ZvlOo8KX9a/xWxImGmC/SLg0QBt9ENkYkhJmz8q+xc8I/n/5M1RqEuf/Ukucw3Y3QxIUjxvFiIf97PE8PHjbhUAd3OJQoAO7K4fUBUmRQ3BPxhrpqJM8QrPQN2c2xm/J5T+bDErtQN9wzezCilSU5DhfiuP0wOscvN5mEABzblEsb5eY3ma/4PXU0Xtp10O1eMvhP5 X-Bogosity: Ham, tests=bogofilter, spamicity=0.000000, version=1.2.4 Sender: owner-linux-mm@kvack.org Precedence: bulk X-Loop: owner-majordomo@kvack.org List-ID: List-Subscribe: List-Unsubscribe: This patch introduces the option to introduce special pte bit into pmd/puds. Archs can start to define pmd_special / pud_special when supported by selecting the new option. Per-arch support will be added later. Before that, create fallbacks for these helpers so that they are always available. Signed-off-by: Peter Xu --- include/linux/mm.h | 24 ++++++++++++++++++++++++ mm/Kconfig | 13 +++++++++++++ 2 files changed, 37 insertions(+) diff --git a/include/linux/mm.h b/include/linux/mm.h index b0ff06d18c71..d900f15b7650 100644 --- a/include/linux/mm.h +++ b/include/linux/mm.h @@ -2643,6 +2643,30 @@ static inline pte_t pte_mkspecial(pte_t pte) } #endif +#ifndef CONFIG_ARCH_SUPPORTS_PMD_PFNMAP +static inline bool pmd_special(pmd_t pmd) +{ + return false; +} + +static inline pmd_t pmd_mkspecial(pmd_t pmd) +{ + return pmd; +} +#endif /* CONFIG_ARCH_SUPPORTS_PMD_PFNMAP */ + +#ifndef CONFIG_ARCH_SUPPORTS_PUD_PFNMAP +static inline bool pud_special(pud_t pud) +{ + return false; +} + +static inline pud_t pud_mkspecial(pud_t pud) +{ + return pud; +} +#endif /* CONFIG_ARCH_SUPPORTS_PUD_PFNMAP */ + #ifndef CONFIG_ARCH_HAS_PTE_DEVMAP static inline int pte_devmap(pte_t pte) { diff --git a/mm/Kconfig b/mm/Kconfig index 8078a4b3c509..b23913d4e47e 100644 --- a/mm/Kconfig +++ b/mm/Kconfig @@ -875,6 +875,19 @@ endif # TRANSPARENT_HUGEPAGE config PGTABLE_HAS_HUGE_LEAVES def_bool TRANSPARENT_HUGEPAGE || HUGETLB_PAGE +# TODO: Allow to be enabled without THP +config ARCH_SUPPORTS_HUGE_PFNMAP + def_bool n + depends on TRANSPARENT_HUGEPAGE + +config ARCH_SUPPORTS_PMD_PFNMAP + def_bool y + depends on ARCH_SUPPORTS_HUGE_PFNMAP && HAVE_ARCH_TRANSPARENT_HUGEPAGE + +config ARCH_SUPPORTS_PUD_PFNMAP + def_bool y + depends on ARCH_SUPPORTS_HUGE_PFNMAP && HAVE_ARCH_TRANSPARENT_HUGEPAGE_PUD + # # UP and nommu archs use km based percpu allocator # From patchwork Mon Aug 26 20:43:36 2024 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Peter Xu X-Patchwork-Id: 13778396 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from kanga.kvack.org (kanga.kvack.org [205.233.56.17]) by smtp.lore.kernel.org (Postfix) with ESMTP id F06BAC54731 for ; Mon, 26 Aug 2024 20:44:07 +0000 (UTC) Received: by kanga.kvack.org (Postfix) id CEC736B0083; Mon, 26 Aug 2024 16:44:05 -0400 (EDT) Received: by kanga.kvack.org (Postfix, from userid 40) id C74C76B0085; Mon, 26 Aug 2024 16:44:05 -0400 (EDT) X-Delivered-To: int-list-linux-mm@kvack.org Received: by kanga.kvack.org (Postfix, from userid 63042) id AA1436B0088; Mon, 26 Aug 2024 16:44:05 -0400 (EDT) X-Delivered-To: linux-mm@kvack.org Received: from relay.hostedemail.com (smtprelay0015.hostedemail.com [216.40.44.15]) by kanga.kvack.org (Postfix) with ESMTP id 8A5E16B0083 for ; Mon, 26 Aug 2024 16:44:05 -0400 (EDT) Received: from smtpin09.hostedemail.com (a10.router.float.18 [10.200.18.1]) by unirelay10.hostedemail.com (Postfix) with ESMTP id 4952FC0F3F for ; Mon, 26 Aug 2024 20:44:05 +0000 (UTC) X-FDA: 82495573650.09.969D9E2 Received: from us-smtp-delivery-124.mimecast.com (us-smtp-delivery-124.mimecast.com [170.10.133.124]) by imf20.hostedemail.com (Postfix) with ESMTP id 1C1071C001E for ; Mon, 26 Aug 2024 20:44:02 +0000 (UTC) Authentication-Results: imf20.hostedemail.com; dkim=pass header.d=redhat.com header.s=mimecast20190719 header.b=Tl3d8A2A; spf=pass (imf20.hostedemail.com: domain of peterx@redhat.com designates 170.10.133.124 as permitted sender) smtp.mailfrom=peterx@redhat.com; dmarc=pass (policy=none) header.from=redhat.com ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=hostedemail.com; s=arc-20220608; t=1724704928; h=from:from:sender:reply-to:subject:subject:date:date: message-id:message-id:to:to:cc:cc:mime-version:mime-version: content-type:content-type: content-transfer-encoding:content-transfer-encoding: in-reply-to:in-reply-to:references:references:dkim-signature; bh=rCV2q6t08l8ndW0pGxUU3F/GDhRRI1SX3G16E23iQDg=; b=dYlVhY0ugg29smahEruwhvPrEEo+Vsg1D6qsThIRQawRCcHwJvTSnszEMSEj20znlk5WFd U3Px/C+yHtuIsYEi5Bo8GcdAz84JtByIjvjpv91tkN/Wxpv2oI8exKnlqKseFWp9Tv9nqU 01hX3x7xbepaTwFhm0GXZmQVk01fYLg= ARC-Authentication-Results: i=1; imf20.hostedemail.com; dkim=pass header.d=redhat.com header.s=mimecast20190719 header.b=Tl3d8A2A; spf=pass (imf20.hostedemail.com: domain of peterx@redhat.com designates 170.10.133.124 as permitted sender) smtp.mailfrom=peterx@redhat.com; dmarc=pass (policy=none) header.from=redhat.com ARC-Seal: i=1; s=arc-20220608; d=hostedemail.com; t=1724704929; a=rsa-sha256; cv=none; b=MCqn8NuvVaBGgbnriK2DXgR1pc7zH3gHx2EHRC1Vz6WSIe47Yr0bOSj0td83eDH2dNUFZw oysmZz43v4AlvXUGA9/XXs2X+D2MkEexKaRcxYIa7l8+NdsYFNH5Dd0ZQBGUnchlTbesWv Bfeaf2lDOEK4VACfFTX6nRxOb0bCLts= DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=redhat.com; s=mimecast20190719; t=1724705042; h=from:from:reply-to:subject:subject:date:date:message-id:message-id: to:to:cc:cc:mime-version:mime-version:content-type:content-type: content-transfer-encoding:content-transfer-encoding: in-reply-to:in-reply-to:references:references; bh=rCV2q6t08l8ndW0pGxUU3F/GDhRRI1SX3G16E23iQDg=; b=Tl3d8A2AhgRkTO/Om/ffHNuLk5Tvtkwr+ZhC4vFyHmijx1Um9A9tO5MSWww9vCXXI9JjG2 +xcizImJhRhT03d9MrKB3hA6TpMywsDmYH1QvbCUb42nsK7xyqSwTEalUlagAedDWdcQul 9vnc/5WqJILWqKjB2TV9u8DRG+NfD6o= Received: from mail-qk1-f197.google.com (mail-qk1-f197.google.com [209.85.222.197]) by relay.mimecast.com with ESMTP with STARTTLS (version=TLSv1.3, cipher=TLS_AES_256_GCM_SHA384) id us-mta-616-xJKtw6tZPfadOM6QvLcQIw-1; Mon, 26 Aug 2024 16:44:00 -0400 X-MC-Unique: xJKtw6tZPfadOM6QvLcQIw-1 Received: by mail-qk1-f197.google.com with SMTP id af79cd13be357-7a66b52c944so520438785a.1 for ; Mon, 26 Aug 2024 13:44:00 -0700 (PDT) X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20230601; t=1724705040; x=1725309840; h=content-transfer-encoding:mime-version:references:in-reply-to :message-id:date:subject:cc:to:from:x-gm-message-state:from:to:cc :subject:date:message-id:reply-to; bh=rCV2q6t08l8ndW0pGxUU3F/GDhRRI1SX3G16E23iQDg=; b=QoZ22u1wcxhhLX/mw+Pr8D9XvI/jfXC+bnh4TUF3l4azik47bEXoMWEdE4xTDzyaAR IjkC+c/rZoUZiSrDKD3fA5GD4Nz/azWgdkXfctlrTuEa/QQGeRszZPugMfION6QDHTml VR3AmBil26iyFk4HYXfgd32WQDLueb4LrmzNTz1gm4rkJWpY0EKvBn8lX3xKJpCLF8Zc 6KIMkEl47Om9f9Vj7YpVKQtt7jBg452Xc55hetdzyBqGbc8/L8mtkSRDf66EJvd81P7Y uVEw0N/AXRXGwbb4nE9qYU2rUCnjBHXJ4uuGDeybw2Y8lY69PGr0NBbImR39Kwdx7oXe xlEQ== X-Forwarded-Encrypted: i=1; AJvYcCVAkIjH+bO9/XxTyWCJEb5FB1kaXZD5tsyEdwPReO786y+qRaChNLce+Vu9mk4Ar+GuyGI2iKWpfA==@kvack.org X-Gm-Message-State: AOJu0Yw5JtwD2Hzk1SW9H+NINq7FUH2xvRjO0mR2xKBZZmLG6yLwcwEV /b8MVDoOqVEN9ED8khZFLvyS+pXYZ757jRvk4Dj4XnVUpJDbvNb+f9fBvOLAymLWz5hYBU5X53I NutMt7pJYmxsophGZrc0fXjxN//hhGt2lVP4YFTeulkR77E9+ X-Received: by 2002:a05:620a:28c8:b0:7a1:62ad:9d89 with SMTP id af79cd13be357-7a7e4e6d3c1mr92954385a.64.1724705040431; Mon, 26 Aug 2024 13:44:00 -0700 (PDT) X-Google-Smtp-Source: AGHT+IEW5yVxsMo5HJEI7stPZogQTAiTuzeFOof5FGrO8bEsBYthlHw7SRF31ZNT2saoeMAKCiiqzA== X-Received: by 2002:a05:620a:28c8:b0:7a1:62ad:9d89 with SMTP id af79cd13be357-7a7e4e6d3c1mr92952485a.64.1724705040096; Mon, 26 Aug 2024 13:44:00 -0700 (PDT) Received: from x1n.redhat.com (pool-99-254-121-117.cpe.net.cable.rogers.com. [99.254.121.117]) by smtp.gmail.com with ESMTPSA id af79cd13be357-7a67f3fd6c1sm491055185a.121.2024.08.26.13.43.58 (version=TLS1_3 cipher=TLS_AES_256_GCM_SHA384 bits=256/256); Mon, 26 Aug 2024 13:43:59 -0700 (PDT) From: Peter Xu To: linux-kernel@vger.kernel.org, linux-mm@kvack.org Cc: Gavin Shan , Catalin Marinas , x86@kernel.org, Ingo Molnar , Andrew Morton , Paolo Bonzini , Dave Hansen , Thomas Gleixner , Alistair Popple , kvm@vger.kernel.org, linux-arm-kernel@lists.infradead.org, Sean Christopherson , peterx@redhat.com, Oscar Salvador , Jason Gunthorpe , Borislav Petkov , Zi Yan , Axel Rasmussen , David Hildenbrand , Yan Zhao , Will Deacon , Kefeng Wang , Alex Williamson , Matthew Wilcox , "Aneesh Kumar K . V" Subject: [PATCH v2 02/19] mm: Drop is_huge_zero_pud() Date: Mon, 26 Aug 2024 16:43:36 -0400 Message-ID: <20240826204353.2228736-3-peterx@redhat.com> X-Mailer: git-send-email 2.45.0 In-Reply-To: <20240826204353.2228736-1-peterx@redhat.com> References: <20240826204353.2228736-1-peterx@redhat.com> MIME-Version: 1.0 X-Mimecast-Spam-Score: 0 X-Mimecast-Originator: redhat.com X-Rspamd-Server: rspam06 X-Rspamd-Queue-Id: 1C1071C001E X-Stat-Signature: btizhhd94tisgoi7jcc1wjeqkp46metk X-Rspam-User: X-HE-Tag: 1724705042-506633 X-HE-Meta: U2FsdGVkX18imHxFO9pfASj/G7OGmUwa4Ek+RwFJ1o1UJIf2ll7vttw3QVHp5E/dcQptIEHcvZVhpoWntx2PHjX2XZbd4E2yPwK1k+nzRZ6Q42toQU+hNmzZFXbJdpkj+k3RZXZXAcF15CyUP9DcUYuJ65gZ/lg2GkR7Sq2+mkcGRKMA0UVq/T3ZFxtCBoeqzd2alLuGDNe2eBohXNbiLDBH67O3EHgNbnDYwqBeRt2TwJNQi/FVPnsBBQd1M+SBHmELwCiNMw3INUgSng5XP2LeUY2z6P4IGgyAV6H2hlV/HEtWn7BCFXVXqfW9GZFU6sT1zH/khC2TTRQ5ZuIoHsnWXd61Pa50h0UMOY7533hC8Cazdj8+e9HBRClhsXwoqV78t3sc8h5BdkS7KWXTGqaiP/cxlAqPY2/gdyVKKg+57WmoFgnaQJLN2NccBhu08KQOAM4eZSqN9d5K+cxP5DbsmNMtHxpgTOF2bXKGRgvQff92bE3SWHxU65x5jtHxRVJRlezoK8p2UD4xetJLmh1AFSIuK0v3hG7krR/5G2JlSEoZed82WYD+UOqCauvBHFm8lW7t4e9qlm5j00nG1xyF6z13lArvsme2zMiLdyUFYnCs/y5GjHqtgllqLOQQYBJLOPUUqURo1bbXH9jpGd4JX2nCU7wXU7l0BR9JbVKp5UijWIBvR/s51EMfF6IqSvDqihIay9Pzbr5D70/AvLrthEN1MOlw42iCne6hAYRi0Mlkxtjp/LMJimj/MkywkGtiBHvkn4WAD0fFRBbPuWWiVPg7Yz7PvJrwleL3vN1jXbfURmYiwY3l/6TUpz+edBTm9zKE1PT+jxWqmDNFgWjw7iHwZwld6Zt32+kNQvPHnc3X+9eBtrhOpRRRXB4u1/CsPk6gLicL4KVK9bnYdN1GndtoK1U7zy1YHcCSOkYGqcecxIA5VruaKHu8rH4Qirptlp2vadf/oChHc2N +qe1Me2f Q48O9qeXyaM2duCuSb27+8aqhbfixDU2Wp08cpCnkCF5g9uM1s5zexC+UpnqqSPHv/xSz0ULOmIQjWHFQMApOrd28/D+Jd8rl8Z/RkxYTuajnWAxekNxzS2UvehwbADH9+rLuiAceQ1U5ANPIYgv+YmZlv800U+spNVqym8OAlex1RFUTDWP5RObSXWj05wkeLAG53XApI9THVK062i+JRuWVjHM4CekQ46eUw4iAblEVdmVKN2CTKEvPEBXiPCgZOk6XJ05mt+Tw/alGuns6+Ii7TP1lx16lVjuPFpb8RSybVkGMvpUDdC/VcW8+MwKHp9wv7JcVlBbvG+MEp8kZorfLJ4OYFnUHXpzz6P4oC014J8jhl2hLNJpQTfdDxAlAS6oOd7hAUyypCCbzxSOQCBsvzovT2Tbtr1SdrBZBepP4MW8eo1cXX2SzyXIkoPHBxvp+Rw15uBd1L0fWezHriyzzSi04G6K9x7t8qUOgK/9UtMA= X-Bogosity: Ham, tests=bogofilter, spamicity=0.000000, version=1.2.4 Sender: owner-linux-mm@kvack.org Precedence: bulk X-Loop: owner-majordomo@kvack.org List-ID: List-Subscribe: List-Unsubscribe: It constantly returns false since 2017. One assertion is added in 2019 but it should never have triggered, IOW it means what is checked should be asserted instead. If it didn't exist for 7 years maybe it's good idea to remove it and only add it when it comes. Cc: Matthew Wilcox Cc: Aneesh Kumar K.V Acked-by: David Hildenbrand Reviewed-by: Jason Gunthorpe Signed-off-by: Peter Xu --- include/linux/huge_mm.h | 10 ---------- mm/huge_memory.c | 13 +------------ 2 files changed, 1 insertion(+), 22 deletions(-) diff --git a/include/linux/huge_mm.h b/include/linux/huge_mm.h index 4902e2f7e896..b550b5a248bb 100644 --- a/include/linux/huge_mm.h +++ b/include/linux/huge_mm.h @@ -433,11 +433,6 @@ static inline bool is_huge_zero_pmd(pmd_t pmd) return pmd_present(pmd) && READ_ONCE(huge_zero_pfn) == pmd_pfn(pmd); } -static inline bool is_huge_zero_pud(pud_t pud) -{ - return false; -} - struct folio *mm_get_huge_zero_folio(struct mm_struct *mm); void mm_put_huge_zero_folio(struct mm_struct *mm); @@ -578,11 +573,6 @@ static inline bool is_huge_zero_pmd(pmd_t pmd) return false; } -static inline bool is_huge_zero_pud(pud_t pud) -{ - return false; -} - static inline void mm_put_huge_zero_folio(struct mm_struct *mm) { return; diff --git a/mm/huge_memory.c b/mm/huge_memory.c index a81eab98d6b8..3f74b09ada38 100644 --- a/mm/huge_memory.c +++ b/mm/huge_memory.c @@ -1429,10 +1429,8 @@ static void insert_pfn_pud(struct vm_area_struct *vma, unsigned long addr, ptl = pud_lock(mm, pud); if (!pud_none(*pud)) { if (write) { - if (pud_pfn(*pud) != pfn_t_to_pfn(pfn)) { - WARN_ON_ONCE(!is_huge_zero_pud(*pud)); + if (WARN_ON_ONCE(pud_pfn(*pud) != pfn_t_to_pfn(pfn))) goto out_unlock; - } entry = pud_mkyoung(*pud); entry = maybe_pud_mkwrite(pud_mkdirty(entry), vma); if (pudp_set_access_flags(vma, addr, pud, entry, 1)) @@ -1680,15 +1678,6 @@ int copy_huge_pud(struct mm_struct *dst_mm, struct mm_struct *src_mm, if (unlikely(!pud_trans_huge(pud) && !pud_devmap(pud))) goto out_unlock; - /* - * When page table lock is held, the huge zero pud should not be - * under splitting since we don't split the page itself, only pud to - * a page table. - */ - if (is_huge_zero_pud(pud)) { - /* No huge zero pud yet */ - } - /* * TODO: once we support anonymous pages, use * folio_try_dup_anon_rmap_*() and split if duplicating fails. From patchwork Mon Aug 26 20:43:37 2024 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Peter Xu X-Patchwork-Id: 13778397 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from kanga.kvack.org (kanga.kvack.org [205.233.56.17]) by smtp.lore.kernel.org (Postfix) with ESMTP id 9756CC5472C for ; Mon, 26 Aug 2024 20:44:13 +0000 (UTC) Received: by kanga.kvack.org (Postfix) id 1CB506B0088; Mon, 26 Aug 2024 16:44:08 -0400 (EDT) Received: by kanga.kvack.org (Postfix, from userid 40) id 0DF1E6B008C; Mon, 26 Aug 2024 16:44:08 -0400 (EDT) X-Delivered-To: int-list-linux-mm@kvack.org Received: by kanga.kvack.org (Postfix, from userid 63042) id E740D6B0092; Mon, 26 Aug 2024 16:44:07 -0400 (EDT) X-Delivered-To: linux-mm@kvack.org Received: from relay.hostedemail.com (smtprelay0010.hostedemail.com [216.40.44.10]) by kanga.kvack.org (Postfix) with ESMTP id BD8006B0088 for ; Mon, 26 Aug 2024 16:44:07 -0400 (EDT) Received: from smtpin24.hostedemail.com (a10.router.float.18 [10.200.18.1]) by unirelay04.hostedemail.com (Postfix) with ESMTP id 72D131A13AC for ; Mon, 26 Aug 2024 20:44:07 +0000 (UTC) X-FDA: 82495573734.24.4108886 Received: from us-smtp-delivery-124.mimecast.com (us-smtp-delivery-124.mimecast.com [170.10.133.124]) by imf09.hostedemail.com (Postfix) with ESMTP id 4B428140022 for ; Mon, 26 Aug 2024 20:44:05 +0000 (UTC) Authentication-Results: imf09.hostedemail.com; dkim=pass header.d=redhat.com header.s=mimecast20190719 header.b=Q82x8+X8; spf=pass (imf09.hostedemail.com: domain of peterx@redhat.com designates 170.10.133.124 as permitted sender) smtp.mailfrom=peterx@redhat.com; dmarc=pass (policy=none) header.from=redhat.com ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=hostedemail.com; s=arc-20220608; t=1724704959; h=from:from:sender:reply-to:subject:subject:date:date: message-id:message-id:to:to:cc:cc:mime-version:mime-version: content-type:content-type: content-transfer-encoding:content-transfer-encoding: in-reply-to:in-reply-to:references:references:dkim-signature; bh=gD9ggS8OhpZf/NsZoH3fZjUIikns33eAnWwzmGyUsEI=; b=ZpkYAC4MQwJYimCTT8TB59FgIoStZAhEXYzNUuCFYqJmisctzFvzwtv8Jd6M2c/jfFH3jH KaXJ8C+wifB/rbP0vGPE7xyUXZg7CilX/qoJmq1Sp8Rjhv1Ix2epKBtbUrneUFDeNTZi6I KB8uxhY6qoz338oCEmjPPx/BqHEB64A= ARC-Seal: i=1; s=arc-20220608; d=hostedemail.com; t=1724704959; a=rsa-sha256; cv=none; b=CyCTrKygf8uHzqjmV3sZQOhyWqHoSj3F2qWM+QPNuKygS5EPTNy3xQm/o6kVjKkUzGW25+ nKBrzpm/mTEe/fJN2eW9nyRo5PWQBWAsR9Cw99qtOFHhS2sFBqjHBl144lUwnWqRW5+pYs 5AldnExKm9OfovIluXLvADNNpbuOIXQ= ARC-Authentication-Results: i=1; imf09.hostedemail.com; dkim=pass header.d=redhat.com header.s=mimecast20190719 header.b=Q82x8+X8; spf=pass (imf09.hostedemail.com: domain of peterx@redhat.com designates 170.10.133.124 as permitted sender) smtp.mailfrom=peterx@redhat.com; dmarc=pass (policy=none) header.from=redhat.com DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=redhat.com; s=mimecast20190719; t=1724705044; h=from:from:reply-to:subject:subject:date:date:message-id:message-id: to:to:cc:cc:mime-version:mime-version:content-type:content-type: content-transfer-encoding:content-transfer-encoding: in-reply-to:in-reply-to:references:references; bh=gD9ggS8OhpZf/NsZoH3fZjUIikns33eAnWwzmGyUsEI=; b=Q82x8+X872PXGlQbC3y0v165wX4+CBhWfdMjlxbb5gyJBg3sWC+VmG56BDyVDwPge//L4r 2AAhQMj6yIt7eTaxQ/0tAUTtdn2VLO4nYJDKAKVdidhju9qJ5T3T2H9IE4QB75mOkD9C++ kJ/gQPPHLXAv/nQHi2JoxTHcXufRmW8= Received: from mail-qk1-f197.google.com (mail-qk1-f197.google.com [209.85.222.197]) by relay.mimecast.com with ESMTP with STARTTLS (version=TLSv1.3, cipher=TLS_AES_256_GCM_SHA384) id us-mta-670-RDiWEq-dPdOs3yy6ysqrxA-1; Mon, 26 Aug 2024 16:44:03 -0400 X-MC-Unique: RDiWEq-dPdOs3yy6ysqrxA-1 Received: by mail-qk1-f197.google.com with SMTP id af79cd13be357-7a1de8a2af2so745173685a.0 for ; Mon, 26 Aug 2024 13:44:03 -0700 (PDT) X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20230601; t=1724705042; x=1725309842; h=content-transfer-encoding:mime-version:references:in-reply-to :message-id:date:subject:cc:to:from:x-gm-message-state:from:to:cc :subject:date:message-id:reply-to; bh=gD9ggS8OhpZf/NsZoH3fZjUIikns33eAnWwzmGyUsEI=; b=cDRuHPbrxB7Vtfahnfk/hqeYbLweyaq7JB8rdBbJZ3/g6UDYqfHzoplKKDJRfGKYj5 qktM9rDlUwKSMOO7skWFko7MIh34V83BjYEd9XV/813jzMYwpc9hh/ITdxiR9OxuW2aI kApcil1aOWu+XwmcTBAIWGF/U7bDYZyPDfz/OVWTkBEyO2h6X9c7wX1lsoPhRHXShoQR Q3ClAgm3tAdCp/+F5wcVQLBresDXVLYZNeqM286a96R8VKYeAfbD0gKoOMQIn6ydn1Oz CVrQYOuGZL1x4vuQ87zw/Ja/VO6VAdLoOo3+12RZ7vdu4GXrsxLlln44KMlRNyBFPL22 hdcg== X-Forwarded-Encrypted: i=1; AJvYcCUP9sHw9cffarSrFafM76EOEwfd/o+YWsYmPMCxtqKTM38G3/SAvFOQQuRRcvVAvpuKD5ix2YfI/w==@kvack.org X-Gm-Message-State: AOJu0Ywzm3x/lFaMH8F+CkoloGEsizETyG53xlAf+INCNC2xPN7dNVTi DavS2/qo+eYDU1oDLSXwryq7xzLD+FIWRUiZX71Gu4jZVRLA2RNmsVwStIEIuIvbS0AHxXua9Qc KrY238/hiEm04cuRHPrz86dcvoLKbkvupc15twRsOpel4sfLY X-Received: by 2002:a05:620a:24d5:b0:7a3:6dd9:ef9d with SMTP id af79cd13be357-7a6897c6e76mr1277666185a.62.1724705042583; Mon, 26 Aug 2024 13:44:02 -0700 (PDT) X-Google-Smtp-Source: AGHT+IHIqrjqvJMs9Ym5fhhSHDCZDk2wQ6104PL5R9Vk0hTAFm1CUlIqFGQmzGdyDHGQhWdDyxioXg== X-Received: by 2002:a05:620a:24d5:b0:7a3:6dd9:ef9d with SMTP id af79cd13be357-7a6897c6e76mr1277664185a.62.1724705042003; Mon, 26 Aug 2024 13:44:02 -0700 (PDT) Received: from x1n.redhat.com (pool-99-254-121-117.cpe.net.cable.rogers.com. [99.254.121.117]) by smtp.gmail.com with ESMTPSA id af79cd13be357-7a67f3fd6c1sm491055185a.121.2024.08.26.13.44.00 (version=TLS1_3 cipher=TLS_AES_256_GCM_SHA384 bits=256/256); Mon, 26 Aug 2024 13:44:01 -0700 (PDT) From: Peter Xu To: linux-kernel@vger.kernel.org, linux-mm@kvack.org Cc: Gavin Shan , Catalin Marinas , x86@kernel.org, Ingo Molnar , Andrew Morton , Paolo Bonzini , Dave Hansen , Thomas Gleixner , Alistair Popple , kvm@vger.kernel.org, linux-arm-kernel@lists.infradead.org, Sean Christopherson , peterx@redhat.com, Oscar Salvador , Jason Gunthorpe , Borislav Petkov , Zi Yan , Axel Rasmussen , David Hildenbrand , Yan Zhao , Will Deacon , Kefeng Wang , Alex Williamson Subject: [PATCH v2 03/19] mm: Mark special bits for huge pfn mappings when inject Date: Mon, 26 Aug 2024 16:43:37 -0400 Message-ID: <20240826204353.2228736-4-peterx@redhat.com> X-Mailer: git-send-email 2.45.0 In-Reply-To: <20240826204353.2228736-1-peterx@redhat.com> References: <20240826204353.2228736-1-peterx@redhat.com> MIME-Version: 1.0 X-Mimecast-Spam-Score: 0 X-Mimecast-Originator: redhat.com X-Rspam-User: X-Rspamd-Server: rspam04 X-Rspamd-Queue-Id: 4B428140022 X-Stat-Signature: nwxqnifw93p7y5u93uygejfag1cf5epx X-HE-Tag: 1724705045-679792 X-HE-Meta: U2FsdGVkX18git0ZviYHVAcOWx0AWYLWKPoS2Mu0L9Ij7wjNdz81yuAAsZeM6upzPtxMLzNxu5q2opOS2KAHF+FhPjLZ1wc5g+LFbJAzKmKDVm7jVziV/z7cPDplCAv6P43k2VvV/jykxaM+p+4Mv5Ka+dLAE67/yleOTgY1YQWDrr/fAN3y4sTCVo4Tpn5yyKbelLxfxlD4ecOlFkXHzjjuniD1JS8OtRNBKhFY7rq3MLZpt/crTUTGuvtvqa3CedUWAKfhGSnAQFt+Tdw07DrvAwmn3dYddQ3rVUFojhbxE1bPkcrLJHNwWAEYSq7HgDKySeFRBu69Vouy6CDxXeKWccouwLGeTSS0sf6h3M/dAEII+DhWA5w4bU7juOwS+CzoonhfXfog6Oz7Q574joFawQ2myibP5MQC8MRhpPW1foHI/99tqJexRzs9jf7B7st7eCgzF+uG2s1nzABUJoTkQg0gnnYiUTXl9+wjCDGNn5QgTsypT6Thgt5Jd4DypAkBE0FJ8Uv2UkS7vCXoP5YT7oimK3OJqdDBDm6CHND0dU0z9IdR+An7UR3KsXO7bN9o0iVRv2ePwZiKFou1e0ityNzfPTXdpah8tnI672mXS8NZ5li5v1mc0aDeBnSmgH0zeQ+Tj9bsVVbDwuu1pHgU0VgHIljQz7o0anxYhs9QmtsfksmHJhSo+1aiCzkKdyZEF2+lQRIPCtWOhApf1JyN0RreuHYSucrEZh3pWcppW+gVITqvTqltXkcHFsTT+NMBzeZKzpFixztT3WS/XTtsnLRiyaw1KefMbgJ0nyM6XkKWXNfLzdCG6SqVUnq7u+Y2JsdxG4uOSxX9+erYdpbPlsX7+B6deQFGKp8a+xAM3e5YWw9fJ126yQQhePbRe/KGECCiMN98SJ+xF4ppxtiLK2CfTS7Osw7OAZeA4a2EqGEjoSAbBZUb0ES7E5E26jG9C1bslMzBhTl1g+8 6yMyca9/ zuqQDjRiuuiQU2TGhKup2WJDnD46fjX4W1/B4v2aQRKwOijDxsEsHl+NYKl8Y5dxFSHGV6mAtLhLgWJN1sVPRuRz9a1ur1gJOZFw7qTgUtIT9aRyKiowlULV67dO5UdGlmQAZhZR59zBPdBBOC6prJ+Gt4XrzttZOARWX9RzLwEmIHdw29QQ/SYpmGQqK8F9mv+AMP9ddzJk7WCOpgpFgW8Sws+jgVRcffLB7IQy8Ho7aHiNiL+2isbco4wWBs2vxIHnPflkIdQcsfVe2dyB2wSYEJK+/UCPg/hOln15OfMzd/stUTDrM9ToG4X3l9LjMnYXk X-Bogosity: Ham, tests=bogofilter, spamicity=0.000000, version=1.2.4 Sender: owner-linux-mm@kvack.org Precedence: bulk X-Loop: owner-majordomo@kvack.org List-ID: List-Subscribe: List-Unsubscribe: We need these special bits to be around on pfnmaps. Mark properly for !devmap case, reflecting that there's no page struct backing the entry. Reviewed-by: Jason Gunthorpe Signed-off-by: Peter Xu Acked-by: David Hildenbrand --- mm/huge_memory.c | 4 ++++ 1 file changed, 4 insertions(+) diff --git a/mm/huge_memory.c b/mm/huge_memory.c index 3f74b09ada38..dec17d09390f 100644 --- a/mm/huge_memory.c +++ b/mm/huge_memory.c @@ -1346,6 +1346,8 @@ static void insert_pfn_pmd(struct vm_area_struct *vma, unsigned long addr, entry = pmd_mkhuge(pfn_t_pmd(pfn, prot)); if (pfn_t_devmap(pfn)) entry = pmd_mkdevmap(entry); + else + entry = pmd_mkspecial(entry); if (write) { entry = pmd_mkyoung(pmd_mkdirty(entry)); entry = maybe_pmd_mkwrite(entry, vma); @@ -1442,6 +1444,8 @@ static void insert_pfn_pud(struct vm_area_struct *vma, unsigned long addr, entry = pud_mkhuge(pfn_t_pud(pfn, prot)); if (pfn_t_devmap(pfn)) entry = pud_mkdevmap(entry); + else + entry = pud_mkspecial(entry); if (write) { entry = pud_mkyoung(pud_mkdirty(entry)); entry = maybe_pud_mkwrite(entry, vma); From patchwork Mon Aug 26 20:43:38 2024 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Peter Xu X-Patchwork-Id: 13778398 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from kanga.kvack.org (kanga.kvack.org [205.233.56.17]) by smtp.lore.kernel.org (Postfix) with ESMTP id 1C956C5321D for ; Mon, 26 Aug 2024 20:44:16 +0000 (UTC) Received: by kanga.kvack.org (Postfix) id 241EC6B0093; Mon, 26 Aug 2024 16:44:13 -0400 (EDT) Received: by kanga.kvack.org (Postfix, from userid 40) id 1C9CC6B0095; Mon, 26 Aug 2024 16:44:13 -0400 (EDT) X-Delivered-To: int-list-linux-mm@kvack.org Received: by kanga.kvack.org (Postfix, from userid 63042) id F36936B0096; Mon, 26 Aug 2024 16:44:12 -0400 (EDT) X-Delivered-To: linux-mm@kvack.org Received: from relay.hostedemail.com (smtprelay0017.hostedemail.com [216.40.44.17]) by kanga.kvack.org (Postfix) with ESMTP id CDF456B0093 for ; Mon, 26 Aug 2024 16:44:12 -0400 (EDT) Received: from smtpin09.hostedemail.com (a10.router.float.18 [10.200.18.1]) by unirelay09.hostedemail.com (Postfix) with ESMTP id 79C3D81445 for ; Mon, 26 Aug 2024 20:44:12 +0000 (UTC) X-FDA: 82495573944.09.CEB06B5 Received: from us-smtp-delivery-124.mimecast.com (us-smtp-delivery-124.mimecast.com [170.10.129.124]) by imf16.hostedemail.com (Postfix) with ESMTP id 55566180006 for ; Mon, 26 Aug 2024 20:44:09 +0000 (UTC) Authentication-Results: imf16.hostedemail.com; dkim=pass header.d=redhat.com header.s=mimecast20190719 header.b=ANhLXbK+; spf=pass (imf16.hostedemail.com: domain of peterx@redhat.com designates 170.10.129.124 as permitted sender) smtp.mailfrom=peterx@redhat.com; dmarc=pass (policy=none) header.from=redhat.com ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=hostedemail.com; s=arc-20220608; t=1724704963; h=from:from:sender:reply-to:subject:subject:date:date: message-id:message-id:to:to:cc:cc:mime-version:mime-version: content-type:content-type: content-transfer-encoding:content-transfer-encoding: in-reply-to:in-reply-to:references:references:dkim-signature; bh=p41Ko1t0SYR+aQ+NnkPoKcQjWVRbF5iAoso+I7QNClw=; b=flqfbxYp96EojJ6QAEhG38j2/CykRZh6zcmdxmaftR9Hw4tjzY0cZxZVrfh1oyWS26+Y3J B7aj9OEz15q34s2W9s8oTBpgqMn9hz5FdLRMfafN/FA5ecj2pOLiyvvibjtzQ/jD1bjbKs 04aVJl1fCUNcs4mIFP5DqO9tOL/U4oY= ARC-Seal: i=1; s=arc-20220608; d=hostedemail.com; t=1724704963; a=rsa-sha256; cv=none; b=MTqdJX0snTXl0BA7UAo4mHsW+q8XbzMHXVTApgGgskKQtnzffHymC7XF9sQR8e/foUVuXD 97HoJykLgOc+g5Jjfm6uE95FQnJ6jjJpkOGXZliJnip1YNja3fRWaUM3ATQGztJ4CiKxaA 98O6tn0RimS8JrFAKhGTm1qv3oX0N94= ARC-Authentication-Results: i=1; imf16.hostedemail.com; dkim=pass header.d=redhat.com header.s=mimecast20190719 header.b=ANhLXbK+; spf=pass (imf16.hostedemail.com: domain of peterx@redhat.com designates 170.10.129.124 as permitted sender) smtp.mailfrom=peterx@redhat.com; dmarc=pass (policy=none) header.from=redhat.com DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=redhat.com; s=mimecast20190719; t=1724705048; h=from:from:reply-to:subject:subject:date:date:message-id:message-id: to:to:cc:cc:mime-version:mime-version:content-type:content-type: content-transfer-encoding:content-transfer-encoding: in-reply-to:in-reply-to:references:references; bh=p41Ko1t0SYR+aQ+NnkPoKcQjWVRbF5iAoso+I7QNClw=; b=ANhLXbK+9pIbuwML79qjnFRIdr/V/LQLR07EtLdGzgb+JQj+HKRqlCGrHHGP4PnGWcbTGK hc3cM5dR61C7n4uMTmKoZgHXwBHAp58XiZu/mbOSbCwywauwVCDnYd1aw5HMi6NJCuoBvH 3wlZ2gRE58o+SDnMX2/JWji2x+tewzE= Received: from mail-oo1-f70.google.com (mail-oo1-f70.google.com [209.85.161.70]) by relay.mimecast.com with ESMTP with STARTTLS (version=TLSv1.3, cipher=TLS_AES_256_GCM_SHA384) id us-mta-139-jnUBCwXrP-elFPqWB1fdxQ-1; Mon, 26 Aug 2024 16:44:05 -0400 X-MC-Unique: jnUBCwXrP-elFPqWB1fdxQ-1 Received: by mail-oo1-f70.google.com with SMTP id 006d021491bc7-5dcaed7de57so6506921eaf.0 for ; Mon, 26 Aug 2024 13:44:05 -0700 (PDT) X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20230601; t=1724705044; x=1725309844; h=content-transfer-encoding:mime-version:references:in-reply-to :message-id:date:subject:cc:to:from:x-gm-message-state:from:to:cc :subject:date:message-id:reply-to; bh=p41Ko1t0SYR+aQ+NnkPoKcQjWVRbF5iAoso+I7QNClw=; b=ctei8GjlEQYyoW8NXAvX2mxps2Av/TTMOHBMN2AAhadk9rO7J0RH+giMzJxnqTCfOn JdFIAKlIOo9SWMHHc1Ih8LyTBdfpRDvjCHSlhzG2Uw2kA0tm39vjJoq22sDhpK69ekWX JnSuv6t98AhC8RASTqv1YMj1MmCbDICOmvQ+wPT2ipgFMkegn0YoIAW//78UTzhoPfU5 D3B/7B2ALUtKQSjOCL+xDutjX7pKIaFoH9GHoaaJ1bXY2dpFtl7rtT0COSBKiofiICYq 0EUJUoRFySRSrzp20MYqUGiEZf5zPxlVRih0/cTDBDP6KnAxcESDU7wbiVANHNBv4lDG HFQQ== X-Forwarded-Encrypted: i=1; AJvYcCVLQJ9ZYFz4lMeTVxm5ngAf8CnHsl3+5O1R60QRYF9RUocmBQ9GTroqxVOUCUAIq/Zcpgs7KDwoig==@kvack.org X-Gm-Message-State: AOJu0YxNgUBblTsLAeAETH8w0sSvlKWUGpXN27dKnkVXq8YPgfoMYbYS HgeHiocF+c5VVOS05ALHI9bYogDKC0GMUzhIDO/SKdDBzDYp1hmJp6Be1SNWWxzE0cR02StL5qr xTh9ni3N4yaomxJshQxqBB/KYJYG+itirDVUWP4e4RaEgbun5fMCdjAN+ X-Received: by 2002:a05:6358:ca5:b0:1b5:ecce:b760 with SMTP id e5c5f4694b2df-1b5ecceb805mr68655355d.21.1724705044538; Mon, 26 Aug 2024 13:44:04 -0700 (PDT) X-Google-Smtp-Source: AGHT+IGqSprkmyPyKIDES0Mxu2D+0b60kqL8Leav+5Mf5w5kuR6qgHz2l36Zs0Y1NhYRXD/bnn90Fg== X-Received: by 2002:a05:6358:ca5:b0:1b5:ecce:b760 with SMTP id e5c5f4694b2df-1b5ecceb805mr68653955d.21.1724705044223; Mon, 26 Aug 2024 13:44:04 -0700 (PDT) Received: from x1n.redhat.com (pool-99-254-121-117.cpe.net.cable.rogers.com. [99.254.121.117]) by smtp.gmail.com with ESMTPSA id af79cd13be357-7a67f3fd6c1sm491055185a.121.2024.08.26.13.44.02 (version=TLS1_3 cipher=TLS_AES_256_GCM_SHA384 bits=256/256); Mon, 26 Aug 2024 13:44:03 -0700 (PDT) From: Peter Xu To: linux-kernel@vger.kernel.org, linux-mm@kvack.org Cc: Gavin Shan , Catalin Marinas , x86@kernel.org, Ingo Molnar , Andrew Morton , Paolo Bonzini , Dave Hansen , Thomas Gleixner , Alistair Popple , kvm@vger.kernel.org, linux-arm-kernel@lists.infradead.org, Sean Christopherson , peterx@redhat.com, Oscar Salvador , Jason Gunthorpe , Borislav Petkov , Zi Yan , Axel Rasmussen , David Hildenbrand , Yan Zhao , Will Deacon , Kefeng Wang , Alex Williamson , Matthew Wilcox , Ryan Roberts Subject: [PATCH v2 04/19] mm: Allow THP orders for PFNMAPs Date: Mon, 26 Aug 2024 16:43:38 -0400 Message-ID: <20240826204353.2228736-5-peterx@redhat.com> X-Mailer: git-send-email 2.45.0 In-Reply-To: <20240826204353.2228736-1-peterx@redhat.com> References: <20240826204353.2228736-1-peterx@redhat.com> MIME-Version: 1.0 X-Mimecast-Spam-Score: 0 X-Mimecast-Originator: redhat.com X-Stat-Signature: xatemx363oxy74u4pn9mmcnhdy9wx1mw X-Rspamd-Queue-Id: 55566180006 X-Rspam-User: X-Rspamd-Server: rspam08 X-HE-Tag: 1724705049-60535 X-HE-Meta: U2FsdGVkX1/4Igj2RZCM0j1znKE72Qf7L/4ysiA35uGsGDqJ0wfJoW7+7sn81JhY486FD5e7M9FOUavyQFbq9EaIBF1YWQ03ufnI/pbEv1Kn+FX1wy/5MEjyMr99do6Xz2Rz7flLdBZsRWQXtOhpzPj80Q8+za5hiAZNhFnJkH4UAZr/A4cAnYTDwmb7YgHFa22vkpzE5MCUEQMOf7nB3GhMIs9mn6/tc3Vpnksi6OHLhlbNdrSEFQwX/OsInZ5n0zLnoP6bsqKEfP7rt8chOl0Dh5CpuK1UPAs0rxEuNTDoAzTfhUqJIholnT8itQn/y891Bgrz94Zc+go2X4KUCRzCz/duqOD3gXSCXjjAbPbREcm/t9f1mb4jz7cE8eTOPC0+9OValaszGk1Pww1gNxpA2SKcNpzSu7zXASIWFsQ+ZDV+MGj1yybmLngUPilNU2iQ+zxFcKAelf9SVIRJTrzHyjM7gAlYrqE2WFIHcOiuLPT6irpYOsVzuOUOxMeuzETdCc0Nvc3HJRvu5qmYbQn25SUsWcch1M+OIhzV/RA7c/RnS1cLQybaybJckwxQqhwi8Q8lPQ8pDYpBD9Pcoti0oWT/qCz/ETpcqLUMaacyrSAFYyKlaYjRzoSETkjaL5SNpqout7YejZmPxZWoGFwYv7h+hrKInR4tovJ98vhEjkTRkdTBVXqm0jCsrKh5yd4wD3xAkJeUoVqknjmtRBP/hTZ8zHGgqSVp86AK6Qh4gn9U7kQvSd4oSH8U0RaTbrTjACGp8dpjBYeOlcm0pl/MvMMWB1lO3ma7KNQ+MR58nnxhBmpLcH3WodHebEY9AAkFNZbXJgsUUx/VvxDbcyHCUs6c7pCLbeguFVQl+D8sqYVooeLCzsLYadATAguR4UbOEXfOTftdqpqcwOL/qsudT9Q9IlAipP8A6TGssLEdyVNAyy+zz6R4Qt2dPc0ZoNwEjrlkF0lQ5CFqCmK BbJnzDP5 iWy+SUUYTofHsGB22TLQ/7Qfn7EzJ25Upvm7wULTblnmRkx9zsV/1bsMP+6+ptXKYmNWzPcrLAvgGC7u7W1wu6i0CwZw/Zf2er9TLMmy1jAURJCkiiCGQLiuD0hQD0276ROBz2+0pWrkthzXTdWGDh686o4uFE6JKl1Hc0dwHKwFybllap3y1eTmnj9M/De6vPHfMt7gKT3EVdbHptHRtKekaRiFa2BoE9ZHoNPuBpp8OmO9LLIm9dfGyt04xnBcK+cqvtfhiNGzMrtPGLVnw1QKaQMBT6u2fsCy7Rh7yA7QgAz9/W5WGkFirpwGXen7+WycB6hABD0VpQa7A9cCAJkDRecA4Q2yeD5r1JTAXiT2fl1trD0YoNQB1N6fWKuB9jj3v4JeXlVF7Q1PYJ1WVorRStMb2IdwY3Ap9+5/75gyiNqO6myNlyxF/t2nTGqNNk9pwV/B6gy4gsnB+wpkbgCD0a40HAiOXCiu4 X-Bogosity: Ham, tests=bogofilter, spamicity=0.000000, version=1.2.4 Sender: owner-linux-mm@kvack.org Precedence: bulk X-Loop: owner-majordomo@kvack.org List-ID: List-Subscribe: List-Unsubscribe: This enables PFNMAPs to be mapped at either pmd/pud layers. Generalize the dax case into vma_is_special_huge() so as to cover both. Meanwhile, rename the macro to THP_ORDERS_ALL_SPECIAL. Cc: Matthew Wilcox Cc: Gavin Shan Cc: Ryan Roberts Cc: David Hildenbrand Cc: Zi Yan Reviewed-by: Jason Gunthorpe Signed-off-by: Peter Xu Acked-by: David Hildenbrand --- include/linux/huge_mm.h | 6 +++--- mm/huge_memory.c | 4 ++-- 2 files changed, 5 insertions(+), 5 deletions(-) diff --git a/include/linux/huge_mm.h b/include/linux/huge_mm.h index b550b5a248bb..4da102b74a8c 100644 --- a/include/linux/huge_mm.h +++ b/include/linux/huge_mm.h @@ -76,9 +76,9 @@ extern struct kobj_attribute thpsize_shmem_enabled_attr; /* * Mask of all large folio orders supported for file THP. Folios in a DAX * file is never split and the MAX_PAGECACHE_ORDER limit does not apply to - * it. + * it. Same to PFNMAPs where there's neither page* nor pagecache. */ -#define THP_ORDERS_ALL_FILE_DAX \ +#define THP_ORDERS_ALL_SPECIAL \ (BIT(PMD_ORDER) | BIT(PUD_ORDER)) #define THP_ORDERS_ALL_FILE_DEFAULT \ ((BIT(MAX_PAGECACHE_ORDER + 1) - 1) & ~BIT(0)) @@ -87,7 +87,7 @@ extern struct kobj_attribute thpsize_shmem_enabled_attr; * Mask of all large folio orders supported for THP. */ #define THP_ORDERS_ALL \ - (THP_ORDERS_ALL_ANON | THP_ORDERS_ALL_FILE_DAX | THP_ORDERS_ALL_FILE_DEFAULT) + (THP_ORDERS_ALL_ANON | THP_ORDERS_ALL_SPECIAL | THP_ORDERS_ALL_FILE_DEFAULT) #define TVA_SMAPS (1 << 0) /* Will be used for procfs */ #define TVA_IN_PF (1 << 1) /* Page fault handler */ diff --git a/mm/huge_memory.c b/mm/huge_memory.c index dec17d09390f..e2c314f631f3 100644 --- a/mm/huge_memory.c +++ b/mm/huge_memory.c @@ -96,8 +96,8 @@ unsigned long __thp_vma_allowable_orders(struct vm_area_struct *vma, /* Check the intersection of requested and supported orders. */ if (vma_is_anonymous(vma)) supported_orders = THP_ORDERS_ALL_ANON; - else if (vma_is_dax(vma)) - supported_orders = THP_ORDERS_ALL_FILE_DAX; + else if (vma_is_special_huge(vma)) + supported_orders = THP_ORDERS_ALL_SPECIAL; else supported_orders = THP_ORDERS_ALL_FILE_DEFAULT; From patchwork Mon Aug 26 20:43:39 2024 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Peter Xu X-Patchwork-Id: 13778399 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from kanga.kvack.org (kanga.kvack.org [205.233.56.17]) by smtp.lore.kernel.org (Postfix) with ESMTP id A67C0C5472E for ; Mon, 26 Aug 2024 20:44:18 +0000 (UTC) Received: by kanga.kvack.org (Postfix) id BDC0B6B0095; Mon, 26 Aug 2024 16:44:13 -0400 (EDT) Received: by kanga.kvack.org (Postfix, from userid 40) id B3D946B0096; Mon, 26 Aug 2024 16:44:13 -0400 (EDT) X-Delivered-To: int-list-linux-mm@kvack.org Received: by kanga.kvack.org (Postfix, from userid 63042) id 969536B0098; Mon, 26 Aug 2024 16:44:13 -0400 (EDT) X-Delivered-To: linux-mm@kvack.org Received: from relay.hostedemail.com (smtprelay0015.hostedemail.com [216.40.44.15]) by kanga.kvack.org (Postfix) with ESMTP id 77FF56B0095 for ; Mon, 26 Aug 2024 16:44:13 -0400 (EDT) Received: from smtpin12.hostedemail.com (a10.router.float.18 [10.200.18.1]) by unirelay02.hostedemail.com (Postfix) with ESMTP id 85432121396 for ; Mon, 26 Aug 2024 20:44:11 +0000 (UTC) X-FDA: 82495573902.12.908F827 Received: from us-smtp-delivery-124.mimecast.com (us-smtp-delivery-124.mimecast.com [170.10.129.124]) by imf14.hostedemail.com (Postfix) with ESMTP id 6A08F100010 for ; Mon, 26 Aug 2024 20:44:09 +0000 (UTC) Authentication-Results: imf14.hostedemail.com; dkim=pass header.d=redhat.com header.s=mimecast20190719 header.b=RDyPqAPJ; spf=pass (imf14.hostedemail.com: domain of peterx@redhat.com designates 170.10.129.124 as permitted sender) smtp.mailfrom=peterx@redhat.com; dmarc=pass (policy=none) header.from=redhat.com ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=hostedemail.com; s=arc-20220608; t=1724705030; h=from:from:sender:reply-to:subject:subject:date:date: message-id:message-id:to:to:cc:cc:mime-version:mime-version: content-type:content-type: content-transfer-encoding:content-transfer-encoding: in-reply-to:in-reply-to:references:references:dkim-signature; bh=SFHgAMnpeJHqDMa0ADr1pyvSduy44GuxF04BOe5aqcw=; b=tk3F/dzoxBNHnXoj2anCJP23mKs9Ou2aL8I2xbw38wIGivyi5B+PU/+AtcHRnpmEEpJL5J M7jo/8RrOYW2BrKBXZsvJrVjt1gSKefFdYtlx7haWP6YhwwiMm+TPq3k5mgT2VxaSVvCd8 6c10NKladnuPPSSdVdRFWO8M8mhbun8= ARC-Authentication-Results: i=1; imf14.hostedemail.com; dkim=pass header.d=redhat.com header.s=mimecast20190719 header.b=RDyPqAPJ; spf=pass (imf14.hostedemail.com: domain of peterx@redhat.com designates 170.10.129.124 as permitted sender) smtp.mailfrom=peterx@redhat.com; dmarc=pass (policy=none) header.from=redhat.com ARC-Seal: i=1; s=arc-20220608; d=hostedemail.com; t=1724705030; a=rsa-sha256; cv=none; b=KYloQ05GN37RI1ZpnueCZf0JnWZLeuprdFZ7XjdM1KkaWkgN35mrSMI/1PYqaCP/BYgtcU bjoAWfGiZ8yEIDh2/HCdlxVQSTOVKWdiy6UcMmzh9sLjk6lvch48uyhAP16QCs7HGbOv3F S9cFyZTEr7jC8rM3+X9tvqHDmHhO9k0= DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=redhat.com; s=mimecast20190719; t=1724705048; h=from:from:reply-to:subject:subject:date:date:message-id:message-id: to:to:cc:cc:mime-version:mime-version:content-type:content-type: content-transfer-encoding:content-transfer-encoding: in-reply-to:in-reply-to:references:references; bh=SFHgAMnpeJHqDMa0ADr1pyvSduy44GuxF04BOe5aqcw=; b=RDyPqAPJHjgBfMYKmTCGMBroxlyMfFK3yWgJNGM01HLl/lo7WpU0QM2/UERzvl+jtYRAX7 fhRHO3tv15uHXbqjSV/AaoHESndNf+7fTrlijp+/PQWyLuNm3wXxDlDXFNSdcEq1WFOyUZ y3XXA5TOL6ttYJmuzXcqDdXUWv8Ct9g= Received: from mail-qk1-f199.google.com (mail-qk1-f199.google.com [209.85.222.199]) by relay.mimecast.com with ESMTP with STARTTLS (version=TLSv1.3, cipher=TLS_AES_256_GCM_SHA384) id us-mta-369-zByFxE3OPJGmx1HM3_02jw-1; Mon, 26 Aug 2024 16:44:07 -0400 X-MC-Unique: zByFxE3OPJGmx1HM3_02jw-1 Received: by mail-qk1-f199.google.com with SMTP id af79cd13be357-7a1db42803eso699490285a.2 for ; Mon, 26 Aug 2024 13:44:07 -0700 (PDT) X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20230601; t=1724705047; x=1725309847; h=content-transfer-encoding:mime-version:references:in-reply-to :message-id:date:subject:cc:to:from:x-gm-message-state:from:to:cc :subject:date:message-id:reply-to; bh=SFHgAMnpeJHqDMa0ADr1pyvSduy44GuxF04BOe5aqcw=; b=om8YnGrJL4lKKxVeKmZ41BPj8bXUimHdBEHgyWoKQY2HQSAUIPhikuwFmW6oWBHYIr k6Mp0gKEbz2zstmsWj0Qmp8q7o0ninDj7RIxcBYFB739Q2E35neUhQvAAELIaloegzYD gdB0h+KjK2v4GOquCJoxXWxmeN0BI5VcjEr3y/BSdP2Md/vdlG9JBXfqyIXwdr4Z1Jre ws3VGhKONL0J6XPFeu6GZw+Wig9Rkz/A0D+8b0xwP4MJevNoQhFoVbK92zCMTqWqr7mz sCNeJtnT61OQEY55Cw73MCvUc6EUwr832GDpD1CiYp8AIlRfCwBwblQt8lXp3/hwNMHB Z2uw== X-Forwarded-Encrypted: i=1; AJvYcCXbIORkNULKn0JLO0GnnCjOdc3E087R3y4PS1IbrYgc17O3JbQduwR3UmiJVbCYSyQ8Fs7R2iT9rQ==@kvack.org X-Gm-Message-State: AOJu0YxX0j9R8Sf4IroMatQ/HyFyYRejOMhLiHtHLzyK/lQ4mbxhQGbR 1aBQhTjF/AWJKBAlN/VHoyocrnIt4V6wH8NpLXlMSUOkN+qJodvbY4Wqp8cgFX6b87Zc9IlVb3M 5dsAp16AgSO+HCOYD8ssRSoAjQkOUlxe/lmQ/tf006zwSe5yX X-Received: by 2002:a05:620a:2a14:b0:7a2:ccd:9672 with SMTP id af79cd13be357-7a7e4e6144fmr67568785a.67.1724705046879; Mon, 26 Aug 2024 13:44:06 -0700 (PDT) X-Google-Smtp-Source: AGHT+IG++rZwRsHPi3LEX2hBpbUb+Bhnwx97nv6F7Fy5QJm5RgQYxXJ7nF8aD8ykhSm+rXvWrvbUzQ== X-Received: by 2002:a05:620a:2a14:b0:7a2:ccd:9672 with SMTP id af79cd13be357-7a7e4e6144fmr67565885a.67.1724705046455; Mon, 26 Aug 2024 13:44:06 -0700 (PDT) Received: from x1n.redhat.com (pool-99-254-121-117.cpe.net.cable.rogers.com. [99.254.121.117]) by smtp.gmail.com with ESMTPSA id af79cd13be357-7a67f3fd6c1sm491055185a.121.2024.08.26.13.44.04 (version=TLS1_3 cipher=TLS_AES_256_GCM_SHA384 bits=256/256); Mon, 26 Aug 2024 13:44:05 -0700 (PDT) From: Peter Xu To: linux-kernel@vger.kernel.org, linux-mm@kvack.org Cc: Gavin Shan , Catalin Marinas , x86@kernel.org, Ingo Molnar , Andrew Morton , Paolo Bonzini , Dave Hansen , Thomas Gleixner , Alistair Popple , kvm@vger.kernel.org, linux-arm-kernel@lists.infradead.org, Sean Christopherson , peterx@redhat.com, Oscar Salvador , Jason Gunthorpe , Borislav Petkov , Zi Yan , Axel Rasmussen , David Hildenbrand , Yan Zhao , Will Deacon , Kefeng Wang , Alex Williamson Subject: [PATCH v2 05/19] mm/gup: Detect huge pfnmap entries in gup-fast Date: Mon, 26 Aug 2024 16:43:39 -0400 Message-ID: <20240826204353.2228736-6-peterx@redhat.com> X-Mailer: git-send-email 2.45.0 In-Reply-To: <20240826204353.2228736-1-peterx@redhat.com> References: <20240826204353.2228736-1-peterx@redhat.com> MIME-Version: 1.0 X-Mimecast-Spam-Score: 0 X-Mimecast-Originator: redhat.com X-Rspam-User: X-Stat-Signature: 54u5ngu44bherffao56z6ba3h97ms3cx X-Rspamd-Queue-Id: 6A08F100010 X-Rspamd-Server: rspam11 X-HE-Tag: 1724705049-669219 X-HE-Meta: U2FsdGVkX1+AK9SFeImQvfHuQV3rfh9/zEGqdu2mTh3kl9HVqEibu3DMd8y7T/MapT0El2+mar8Y41BDM6oaoydfWQn8ZrOJVG/YW3vuSRe/FvtNmv9+v2ftvZF3wL6co7O+BepGUn4QpCTHbjre2RLRbqp+IRhGbbD/11AjvWZvYRGB/b97isExM+eGf7SUYmJjWLY/b9Fv4D3uPQkZj/FWiSptnc2y16mslh4RKhupXEuRYWQmkdGfkx54bgT2IXw18rCimEKG9ByPUgFgoSLX3Od60lFa3V1p7puzngMxYSfBT4MJILI8/oo7xT1cXqeOBiIHgTLHKrdPAHPC7JwlezVvaOtWN5hDZpHdvw0dAjxVtLBFUfVmpuuL60GG04JRxSskdz05eH/uE3nNi5XjQd6i57VqqSFpikRtjA0quuBVdKI8UKqa3FNSVNmSCzH8nHkZ813v3OK4dHcXszw9s7Ta4KHkV3MPhOzuPJW3K+v6n7z/MPTkCczp1axRuERBbGlpjucckek46macSHgbwotyKxiJC0swoUNUa9KsUE7Cm2mFEBauP/M0tupqKRGspJmqD33paqOqDjPETyVYJxU31nOe9yMKDURGQ3WFNkJ1AFAQT7/+G57l6g6Pnv8aFPSlaUbzw5Dn5J5hfZun+p66/dmKaKpqH/8OWaC1on4u8UxE5mBvvi0Zag01Z3BZN7NJeN3BSyzoyTnI4cCVMX2JjlzkCaCUGbL1ogunYgs92W1UhZN3gRJKkKY5Vnyz2qlK7V2QWSoUDl7WPgOh20j4RoCDf0+bk7ioagwEVE2PJnasBnQmPfIGADFaYYeSx4QNOEJqyb79f0kCxJdiQ560GbCfa8SbRsyCD1y+mYjeSxp6c6P5JggUUQtkZ4RhezW0LX1axX3dZ5sSM3LPUS7iUvSUfDKMgQLfWSBUFyxgCzCv7BUdGxCSd3vOHp9oqVGIoSbiRpfOLbk HPVT2S+t zDNAI4xZv8TVL5XS/skScZEf/1z2imvSX8uCMFYn+ElyZS+y5AooVCB/iCaZ63mHLqcbiDNa4o9t4/LAMu5QAn22NEuBeP0P9SvoFpnsfAjq0o3Yirevyw0zm5mzt8yUa4DMrwFwSDJZl9peAf3+9M4z5+Vi8Ub5zheKLYc4BAOU94UDyefkruemSe0GFV8Qp1jJ14P/cfYzlDZWo/tVZmm1f4+yztKb6A+4SZFIoT79Zcza+OdjGMu4ant4ifEwcLHwi1ddOrLeMVdzCszBFQtYe4p/uMDD7HfFVo4v5Zb4IMBfJ6dbniWKi+sl7oW+Uvm+DgGyUbbYu8ZSxmfgz2a+JmhCNaOQj6oa9WtJnwANOU2U= X-Bogosity: Ham, tests=bogofilter, spamicity=0.000000, version=1.2.4 Sender: owner-linux-mm@kvack.org Precedence: bulk X-Loop: owner-majordomo@kvack.org List-ID: List-Subscribe: List-Unsubscribe: Since gup-fast doesn't have the vma reference, teach it to detect such huge pfnmaps by checking the special bit for pmd/pud too, just like ptes. Acked-by: David Hildenbrand Reviewed-by: Jason Gunthorpe Signed-off-by: Peter Xu --- mm/gup.c | 6 ++++++ 1 file changed, 6 insertions(+) diff --git a/mm/gup.c b/mm/gup.c index d19884e097fd..a49f67a512ee 100644 --- a/mm/gup.c +++ b/mm/gup.c @@ -3038,6 +3038,9 @@ static int gup_fast_pmd_leaf(pmd_t orig, pmd_t *pmdp, unsigned long addr, if (!pmd_access_permitted(orig, flags & FOLL_WRITE)) return 0; + if (pmd_special(orig)) + return 0; + if (pmd_devmap(orig)) { if (unlikely(flags & FOLL_LONGTERM)) return 0; @@ -3082,6 +3085,9 @@ static int gup_fast_pud_leaf(pud_t orig, pud_t *pudp, unsigned long addr, if (!pud_access_permitted(orig, flags & FOLL_WRITE)) return 0; + if (pud_special(orig)) + return 0; + if (pud_devmap(orig)) { if (unlikely(flags & FOLL_LONGTERM)) return 0; From patchwork Mon Aug 26 20:43:40 2024 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Peter Xu X-Patchwork-Id: 13778400 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from kanga.kvack.org (kanga.kvack.org [205.233.56.17]) by smtp.lore.kernel.org (Postfix) with ESMTP id 322AAC54731 for ; Mon, 26 Aug 2024 20:44:21 +0000 (UTC) Received: by kanga.kvack.org (Postfix) id 3505A6B0096; Mon, 26 Aug 2024 16:44:14 -0400 (EDT) Received: by kanga.kvack.org (Postfix, from userid 40) id 2FFD96B009A; Mon, 26 Aug 2024 16:44:14 -0400 (EDT) X-Delivered-To: int-list-linux-mm@kvack.org Received: by kanga.kvack.org (Postfix, from userid 63042) id 152296B0099; Mon, 26 Aug 2024 16:44:14 -0400 (EDT) X-Delivered-To: linux-mm@kvack.org Received: from relay.hostedemail.com (smtprelay0011.hostedemail.com [216.40.44.11]) by kanga.kvack.org (Postfix) with ESMTP id E9C786B0098 for ; Mon, 26 Aug 2024 16:44:13 -0400 (EDT) Received: from smtpin05.hostedemail.com (a10.router.float.18 [10.200.18.1]) by unirelay01.hostedemail.com (Postfix) with ESMTP id 9F4F01C455C for ; Mon, 26 Aug 2024 20:44:13 +0000 (UTC) X-FDA: 82495573986.05.D1C8CF1 Received: from us-smtp-delivery-124.mimecast.com (us-smtp-delivery-124.mimecast.com [170.10.133.124]) by imf02.hostedemail.com (Postfix) with ESMTP id 93B8580005 for ; Mon, 26 Aug 2024 20:44:11 +0000 (UTC) Authentication-Results: imf02.hostedemail.com; dkim=pass header.d=redhat.com header.s=mimecast20190719 header.b=VvSY6pkv; dmarc=pass (policy=none) header.from=redhat.com; spf=pass (imf02.hostedemail.com: domain of peterx@redhat.com designates 170.10.133.124 as permitted sender) smtp.mailfrom=peterx@redhat.com ARC-Seal: i=1; s=arc-20220608; d=hostedemail.com; t=1724705032; a=rsa-sha256; cv=none; b=Zee+s4ISGqj/z7ezJFdRHu6f1FPPJGcT1SFYg5hD37pvb5Z50GZUmWyjZ3q048jNFxb30+ MjMUy4OxWZWUt0aXYwDewoLTwo3VQwhRyh0EZy41UuKSv45KWfTZgcaM7rcIT0swP2279c Z7FYd8zjPmPT+f08RUYDkVfhdC0B5TY= ARC-Authentication-Results: i=1; imf02.hostedemail.com; dkim=pass header.d=redhat.com header.s=mimecast20190719 header.b=VvSY6pkv; dmarc=pass (policy=none) header.from=redhat.com; spf=pass (imf02.hostedemail.com: domain of peterx@redhat.com designates 170.10.133.124 as permitted sender) smtp.mailfrom=peterx@redhat.com ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=hostedemail.com; s=arc-20220608; t=1724705032; h=from:from:sender:reply-to:subject:subject:date:date: message-id:message-id:to:to:cc:cc:mime-version:mime-version: content-type:content-type: content-transfer-encoding:content-transfer-encoding: in-reply-to:in-reply-to:references:references:dkim-signature; bh=UaLXo2idJup/Oy4OwRPy2aK364ayUQ63MrUET5wBpvQ=; b=2Tl2I70aV9wDf4RkJC4TcQkbLZV/hAOMLbpd2UCRBqgS31gOnXPOcCWhftrxsXyYurv+O0 xrDs7p4KwBJHe5tqw/LaIi9prtiEdNxoBz7BkKCTM46USA5e2LqAXctUT/HW5qkboSpe+I wctLwnZvsIx/FvJKhKtMmFfyhuvT6fo= DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=redhat.com; s=mimecast20190719; t=1724705051; h=from:from:reply-to:subject:subject:date:date:message-id:message-id: to:to:cc:cc:mime-version:mime-version:content-type:content-type: content-transfer-encoding:content-transfer-encoding: in-reply-to:in-reply-to:references:references; bh=UaLXo2idJup/Oy4OwRPy2aK364ayUQ63MrUET5wBpvQ=; b=VvSY6pkvdoBZR75VdSwrCXL41Ztxpb9sxBHfdhY1xyQ9FTIzLlFvJe2N827g+cLdbXveRS Lwc/9rtaxin1Lh9fU+Vxa4dz6k7DpopmW+Ff8gMRfm2neGduTqzujKJAoilpj+uK0YWRkd f3SCG/mG83+FK8OwdFk0J0XdZjisQFY= Received: from mail-qk1-f198.google.com (mail-qk1-f198.google.com [209.85.222.198]) by relay.mimecast.com with ESMTP with STARTTLS (version=TLSv1.3, cipher=TLS_AES_256_GCM_SHA384) id us-mta-284-8WfFysqLMESwmLOiE5Bx8A-1; Mon, 26 Aug 2024 16:44:09 -0400 X-MC-Unique: 8WfFysqLMESwmLOiE5Bx8A-1 Received: by mail-qk1-f198.google.com with SMTP id af79cd13be357-7a1d06f8e78so523627285a.3 for ; Mon, 26 Aug 2024 13:44:09 -0700 (PDT) X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20230601; t=1724705049; x=1725309849; h=content-transfer-encoding:mime-version:references:in-reply-to :message-id:date:subject:cc:to:from:x-gm-message-state:from:to:cc :subject:date:message-id:reply-to; bh=UaLXo2idJup/Oy4OwRPy2aK364ayUQ63MrUET5wBpvQ=; b=oxW/5ruuFu0bpvTAPvaKqjJlmtj96wMvQVpxnXdXRcmAMgQ3gZdeOrpp3/lB0399VG +BXfmbQY+NH4Fx53qB83VfnXPO+HYFUK7SisRuvGIRtPf0F9CY01h7WLOWo0+jZWTNFY hVk8jv+Yq+3tBAwDdFFKn4KmVjI64CZjThiI2VLeRs/PQgrpduxI+LzhGhyCY6p2CbOc wYAb4fg9k3kHAW/AhE066HPeeGj6I7TP8IMNhfCsQSMvhyvyQrsc43sItbuUWX7/o+n9 OI5pWEjB6ppgXbsTtwbaMLIfNBkArhrQlgEW/PGnMREv/2V9ONx1MpZQW4qFseF1qPaa /EUw== X-Forwarded-Encrypted: i=1; AJvYcCXqmqIy6k53Ter5ulkfbrTIxVch/GubaVg9eBeZ+7SdtlyAvztdLtRW+W4d8XZen4vAdTtz94JhUg==@kvack.org X-Gm-Message-State: AOJu0Yyvai/iK8Jt4VdZBdRMUQmwJAW31RGR9CAuANUsBxgloq3Ku30S KpKmCx9PvMEbM+P5a+ae09lxuMWvVicQOC6WXcjoS8TwWA4Gfs26hhSPj7A1wEb7SEegLu/iGNk 8CNZSfrgx/pfugDVjuwrCSNGaunFBWxREDGsrTGEMkDZHcj0x X-Received: by 2002:a05:620a:4112:b0:7a3:49dc:e6e3 with SMTP id af79cd13be357-7a6897a6f65mr1244452985a.53.1724705048903; Mon, 26 Aug 2024 13:44:08 -0700 (PDT) X-Google-Smtp-Source: AGHT+IFAEqST5igJBEO0jhxDT4OAwN3xGv5rzMyL4N+pVFoujxFDEyZ/hbdQR2zB1kD+DQ4JLF652A== X-Received: by 2002:a05:620a:4112:b0:7a3:49dc:e6e3 with SMTP id af79cd13be357-7a6897a6f65mr1244449385a.53.1724705048557; Mon, 26 Aug 2024 13:44:08 -0700 (PDT) Received: from x1n.redhat.com (pool-99-254-121-117.cpe.net.cable.rogers.com. [99.254.121.117]) by smtp.gmail.com with ESMTPSA id af79cd13be357-7a67f3fd6c1sm491055185a.121.2024.08.26.13.44.06 (version=TLS1_3 cipher=TLS_AES_256_GCM_SHA384 bits=256/256); Mon, 26 Aug 2024 13:44:07 -0700 (PDT) From: Peter Xu To: linux-kernel@vger.kernel.org, linux-mm@kvack.org Cc: Gavin Shan , Catalin Marinas , x86@kernel.org, Ingo Molnar , Andrew Morton , Paolo Bonzini , Dave Hansen , Thomas Gleixner , Alistair Popple , kvm@vger.kernel.org, linux-arm-kernel@lists.infradead.org, Sean Christopherson , peterx@redhat.com, Oscar Salvador , Jason Gunthorpe , Borislav Petkov , Zi Yan , Axel Rasmussen , David Hildenbrand , Yan Zhao , Will Deacon , Kefeng Wang , Alex Williamson Subject: [PATCH v2 06/19] mm/pagewalk: Check pfnmap for folio_walk_start() Date: Mon, 26 Aug 2024 16:43:40 -0400 Message-ID: <20240826204353.2228736-7-peterx@redhat.com> X-Mailer: git-send-email 2.45.0 In-Reply-To: <20240826204353.2228736-1-peterx@redhat.com> References: <20240826204353.2228736-1-peterx@redhat.com> MIME-Version: 1.0 X-Mimecast-Spam-Score: 0 X-Mimecast-Originator: redhat.com X-Rspam-User: X-Rspamd-Queue-Id: 93B8580005 X-Rspamd-Server: rspam01 X-Stat-Signature: reewkazajenbdh68jz4ojzei8ryq3e78 X-HE-Tag: 1724705051-326705 X-HE-Meta: U2FsdGVkX18jMZkjOlWx5ofhbZhdU8M3kLFZ0ZPNK9c6wyw2LiV8oHpE11C6EXjSbsx3kmrBSRXPZmj4LoFQ6GVEU4YuzY9hqEEZ6Xv/1NmA+OR3nt9yisXZj6Z8GCQuRJBx/fo8OcRevEkTrd/cry601vWrp0j+OCEJAYFYHowjFGiI7KXDB6QZIHxRQTMTPNURt5vqjZvhSd3TKXXmzz8jsYAUf+pkF2C4XSN1xGz4WusbE66n4tM/Vcmst0COCI9n0PmWKjLxMFZrG1mHfPJCV70BVjPYgHC1AF+oo8c5RlULO5sIMXFOoJ5HTFk2Gqk+zXZncDuw45pqBReLvciggAwrsVKqp9BjKQqA4Ta36mqxI3IIBBRGmA24WlZf4EdP9C/2Hm4fSnD0ddRi6NYNmh7gXOXza00V7KQ7/jIwBQ+yCtdOYYx96a7W3yXC4Kq1YGH80pgeQUbnZosnvCCLZmfTKGIHG+hNGmzIaZT5wbDBm2rp9fP5pmIb4D3s11FYcsBZtNWxzVZKyGwCdAT+2ShMtFXFaBVJaNxpjkMQkJbOKeHls/VXy+UX7kRDK2URl5QCOEJ0zaEPGHIp2qiI/K1SaL3KFJtC+I4AuqfY06OlXaCMSjG3krXEfQwsc+NpTJtfWueO72EZDbk2YlFg0FSrrrmq/Iq2gQ+Ig4DiYyNLB/hw9cbtg0OR11TwhwIHn6W7m0Tyww1Q+RHvI2wWTy1AJfv/nlWJ0eb6VBObp+vFOKE5cBmQ6Ge/wXEMOi4DyV4VDFCb8NlnDQ9s2zfosj6SAYoQ7PtPtGQ+GPF/ijC7uur9u8F9RpMYOrb+5CiSJiwslT+s9Yh01nKjkAhe4GBfQEnX8Vbk0+wYvfJmNWpOWk83mpgB8Yy/jl/3809eYdlEMxgPZzBhhLdtvsF6Xr9JcuLF28BH45dfs+i90hTWxPbDwZp7Z5veMRA5ypYYu9d5bXgT/vpAN8m bfohO5vG PZsfog753xMGDtyrZw6iRCgMyusSgMQ+ejV80KwullDPpKySHkqGLeJ1MMYgzZNGRYAvoFnvztHduEqEITSlCvWOdR89g399TLHqrlSBbV4+JtmI5pG8lpapHatsRK8EHOonolb8jhXem2lK2dTTE0M2LC3Ph4zK9jTgb+6Uptn2SL2uYvrXJiq9ZMFGTdE9JvVRngnaoNWdOkubqNHmhQmCXn76ky2BBHWUN4xpBsxtw+HXtZulArSk8OnvsuwehKEV8Hp3pmjV6YfCExX22BAeYjGH5U3bWB0lOuW+bWshDSAqWd/W3qtc6JzgG0hjCtwJQMS3+uJ0qHyOOzmzhtA6rjlqFf78jyKCRWp/Dq+8YyRgaKbiUpJYXnql1Bd5NjxPAAK0vrrFQ3Y1LCXe+9tuy4E7eeESc21o8zPBnZdzrRXaxhGfzu/P1GzNL36/K5JzdR9hZtcxIQOnIXTRkf8hOVOBQxkcMp1rnKeEm0b7hO0//44HZlP3iTCSzpNAhcjUq X-Bogosity: Ham, tests=bogofilter, spamicity=0.000000, version=1.2.4 Sender: owner-linux-mm@kvack.org Precedence: bulk X-Loop: owner-majordomo@kvack.org List-ID: List-Subscribe: List-Unsubscribe: Teach folio_walk_start() to recognize special pmd/pud mappings, and fail them properly as it means there's no folio backing them. Cc: David Hildenbrand Signed-off-by: Peter Xu --- mm/pagewalk.c | 4 ++-- 1 file changed, 2 insertions(+), 2 deletions(-) diff --git a/mm/pagewalk.c b/mm/pagewalk.c index cd79fb3b89e5..12be5222d70e 100644 --- a/mm/pagewalk.c +++ b/mm/pagewalk.c @@ -753,7 +753,7 @@ struct folio *folio_walk_start(struct folio_walk *fw, fw->pudp = pudp; fw->pud = pud; - if (!pud_present(pud) || pud_devmap(pud)) { + if (!pud_present(pud) || pud_devmap(pud) || pud_special(pud)) { spin_unlock(ptl); goto not_found; } else if (!pud_leaf(pud)) { @@ -783,7 +783,7 @@ struct folio *folio_walk_start(struct folio_walk *fw, fw->pmdp = pmdp; fw->pmd = pmd; - if (pmd_none(pmd)) { + if (pmd_none(pmd) || pmd_special(pmd)) { spin_unlock(ptl); goto not_found; } else if (!pmd_leaf(pmd)) { From patchwork Mon Aug 26 20:43:41 2024 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Peter Xu X-Patchwork-Id: 13778401 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from kanga.kvack.org (kanga.kvack.org [205.233.56.17]) by smtp.lore.kernel.org (Postfix) with ESMTP id B7BEDC5472C for ; Mon, 26 Aug 2024 20:44:23 +0000 (UTC) Received: by kanga.kvack.org (Postfix) id C2A236B009A; Mon, 26 Aug 2024 16:44:17 -0400 (EDT) Received: by kanga.kvack.org (Postfix, from userid 40) id B63D26B009B; Mon, 26 Aug 2024 16:44:17 -0400 (EDT) X-Delivered-To: int-list-linux-mm@kvack.org Received: by kanga.kvack.org (Postfix, from userid 63042) id 9522C6B009C; Mon, 26 Aug 2024 16:44:17 -0400 (EDT) X-Delivered-To: linux-mm@kvack.org Received: from relay.hostedemail.com (smtprelay0017.hostedemail.com [216.40.44.17]) by kanga.kvack.org (Postfix) with ESMTP id 6FCC26B009A for ; Mon, 26 Aug 2024 16:44:17 -0400 (EDT) Received: from smtpin20.hostedemail.com (a10.router.float.18 [10.200.18.1]) by unirelay04.hostedemail.com (Postfix) with ESMTP id 3836E1A1330 for ; Mon, 26 Aug 2024 20:44:17 +0000 (UTC) X-FDA: 82495574154.20.938BAAC Received: from us-smtp-delivery-124.mimecast.com (us-smtp-delivery-124.mimecast.com [170.10.129.124]) by imf25.hostedemail.com (Postfix) with ESMTP id 2F6AAA0012 for ; Mon, 26 Aug 2024 20:44:15 +0000 (UTC) Authentication-Results: imf25.hostedemail.com; dkim=pass header.d=redhat.com header.s=mimecast20190719 header.b="Mxon/CzT"; dmarc=pass (policy=none) header.from=redhat.com; spf=pass (imf25.hostedemail.com: domain of peterx@redhat.com designates 170.10.129.124 as permitted sender) smtp.mailfrom=peterx@redhat.com ARC-Seal: i=1; s=arc-20220608; d=hostedemail.com; t=1724704985; a=rsa-sha256; cv=none; b=bKn3hAu/MXZ4zTLNOyEB69bdTTr3JqsAKOLx79k8nW42bZDjiLLlAnQfErWNBt1JE74nMa KWJFMc73NyICY/sHfwsLDbL1Wj/+qAiXxOy5skEO9hudCw+KAYOg68m5EGMrUDdOCHfhk2 RR+5zuC/S0CXEoA1gfVJt9+Us1NPSQ0= ARC-Authentication-Results: i=1; imf25.hostedemail.com; dkim=pass header.d=redhat.com header.s=mimecast20190719 header.b="Mxon/CzT"; dmarc=pass (policy=none) header.from=redhat.com; spf=pass (imf25.hostedemail.com: domain of peterx@redhat.com designates 170.10.129.124 as permitted sender) smtp.mailfrom=peterx@redhat.com ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=hostedemail.com; s=arc-20220608; t=1724704985; h=from:from:sender:reply-to:subject:subject:date:date: message-id:message-id:to:to:cc:cc:mime-version:mime-version: content-type:content-type: content-transfer-encoding:content-transfer-encoding: in-reply-to:in-reply-to:references:references:dkim-signature; bh=xat90K+7laktWlKWrEYj2NKDd4TQak32OHmU5MtaKmE=; b=hCcxiOUUOpDWR79QtyipBr7u+M8KvEgQVIrhdjQ4cpL7yVfAduj4z7rMHuqEeShIrefhFH qJFtSICjFqZpWWBDFE5iUnqHq4TQLN9n4VE76Yg2gi3c74kHNP7BhpUpbW6+ZrIu7RpBds x1sjuQr9dS1sGJV/hU9j/W7PDtFMcwE= DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=redhat.com; s=mimecast20190719; t=1724705054; h=from:from:reply-to:subject:subject:date:date:message-id:message-id: to:to:cc:cc:mime-version:mime-version:content-type:content-type: content-transfer-encoding:content-transfer-encoding: in-reply-to:in-reply-to:references:references; bh=xat90K+7laktWlKWrEYj2NKDd4TQak32OHmU5MtaKmE=; b=Mxon/CzTlrAUDYEC4tybCxt8knx5/r2HXsOm/8An5nLpqHxljJBr/sxpals4Ak84FhtDPW wxgqD1q4qxpCNpKU5PuWaIBLZXKyTnXna4aqGCxytVTBx01rn8U3MKOjLuizy2iPRtsC24 h/NvGX9Zbuw3ViIbYLAoiTd9lFp9x6c= Received: from mail-yw1-f198.google.com (mail-yw1-f198.google.com [209.85.128.198]) by relay.mimecast.com with ESMTP with STARTTLS (version=TLSv1.3, cipher=TLS_AES_256_GCM_SHA384) id us-mta-91-9fwqpGmyNIO6g-0E-lMKJA-1; Mon, 26 Aug 2024 16:44:11 -0400 X-MC-Unique: 9fwqpGmyNIO6g-0E-lMKJA-1 Received: by mail-yw1-f198.google.com with SMTP id 00721157ae682-6adcb88da08so100204027b3.3 for ; Mon, 26 Aug 2024 13:44:11 -0700 (PDT) X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20230601; t=1724705051; x=1725309851; h=content-transfer-encoding:mime-version:references:in-reply-to :message-id:date:subject:cc:to:from:x-gm-message-state:from:to:cc :subject:date:message-id:reply-to; bh=xat90K+7laktWlKWrEYj2NKDd4TQak32OHmU5MtaKmE=; b=Fq2oYFApJeP79cAKjRKdAp40/TkV5XbV7Ke+U5TBgC6Y0tV/1VZIrAmHTKPv5JVlYY ne5FF21G+rifc6afurMwbq0dOsiYZDXrG1o+HOy0+7LCPzIXhrfR1nKPuL69xaR4AKGg gzqP8hrbfsVYARcK0sNyAyKIqy9QKtXRTDOXCdESlnUareeApAub7vGv5eee3rnbmTQD QTTCKWx0d33YxTYYZU5u8oPaysoxQKE1Uxq7t93kvV6Iri6yTsEfM/1IN2Zba2FP7/rw Z4EGfQ0GjM80L1m2CDsnQyNF/T3QYtlRM8/+xMb8hBggMU3XYvJufBS5fgspsP9Tsv6k IqwA== X-Forwarded-Encrypted: i=1; AJvYcCUtgtxZVFSeSvSswgEjXWdbavSIOo70Ntpw62IcjC0fM95iv8CJDUz6tQ67YnKCNp0vcKk8K01O0w==@kvack.org X-Gm-Message-State: AOJu0Yyl4QBgBOG2H9qeLPDJ+u+AhEUKFWwrtZCjk6Zu5Q2XP9uF8yV2 TSAsr0uB75bT4KqzNjPugsgSfA7tSVK/lIjxImt1By6lf6HCOmJPFv9+ZVBNY67sqd0famM9/zL D55y+1zRCVToY/57aPabAU3qXjAY+lq7eByKRxTrPIncRwPDQ X-Received: by 2002:a05:690c:380b:b0:6c1:2b6d:1964 with SMTP id 00721157ae682-6c62906557cmr136660707b3.38.1724705050751; Mon, 26 Aug 2024 13:44:10 -0700 (PDT) X-Google-Smtp-Source: AGHT+IGVQdSmnkARvKpk+I4g97hETTq/PaNGcO/OaJoau0YQSMQm37evWRXkCU2N5l4bob/h1UXyLw== X-Received: by 2002:a05:690c:380b:b0:6c1:2b6d:1964 with SMTP id 00721157ae682-6c62906557cmr136660497b3.38.1724705050419; Mon, 26 Aug 2024 13:44:10 -0700 (PDT) Received: from x1n.redhat.com (pool-99-254-121-117.cpe.net.cable.rogers.com. [99.254.121.117]) by smtp.gmail.com with ESMTPSA id af79cd13be357-7a67f3fd6c1sm491055185a.121.2024.08.26.13.44.08 (version=TLS1_3 cipher=TLS_AES_256_GCM_SHA384 bits=256/256); Mon, 26 Aug 2024 13:44:09 -0700 (PDT) From: Peter Xu To: linux-kernel@vger.kernel.org, linux-mm@kvack.org Cc: Gavin Shan , Catalin Marinas , x86@kernel.org, Ingo Molnar , Andrew Morton , Paolo Bonzini , Dave Hansen , Thomas Gleixner , Alistair Popple , kvm@vger.kernel.org, linux-arm-kernel@lists.infradead.org, Sean Christopherson , peterx@redhat.com, Oscar Salvador , Jason Gunthorpe , Borislav Petkov , Zi Yan , Axel Rasmussen , David Hildenbrand , Yan Zhao , Will Deacon , Kefeng Wang , Alex Williamson Subject: [PATCH v2 07/19] mm/fork: Accept huge pfnmap entries Date: Mon, 26 Aug 2024 16:43:41 -0400 Message-ID: <20240826204353.2228736-8-peterx@redhat.com> X-Mailer: git-send-email 2.45.0 In-Reply-To: <20240826204353.2228736-1-peterx@redhat.com> References: <20240826204353.2228736-1-peterx@redhat.com> MIME-Version: 1.0 X-Mimecast-Spam-Score: 0 X-Mimecast-Originator: redhat.com X-Rspamd-Server: rspam12 X-Rspamd-Queue-Id: 2F6AAA0012 X-Stat-Signature: 7o8ik9t5snsf115jghgoasta1nwist4d X-Rspam-User: X-HE-Tag: 1724705055-70743 X-HE-Meta: U2FsdGVkX182Lv31Q/FUCQNfhmKfEAJi7P9nM9kGf+251GMghc5ZpXLZ8DJ3En4syUYXC2OE1Tt4BpD/pBlNQk/pycnDGWB0eq+LDvwgyLEHRI7gLsYM4idzqgwrzj0s+gORIWC9DjHXK1THDnWPU75SkR21d5TRMxpT3+kZO9M7XHCJv2WTAYqrr9TN3X+mFd27/1RvKxheEdvx3hd0mXNOvxmdZrV5FZ57g4ThDk0rOOwV5j/a5yHVCANdU2sKxWFAXNWuBtRHD3cnn9fp2MLNIpb7Rz755Dcr1J8LG03Vw4MgXC2yuQTzuDsZS48qk3jSihfWOEY5lNrB3phNCmUwm3YqPCp8zOD7/NP6VJTophWTt4p/IR3SvFn4eB918BtEUcQXGhbuLW0KmmtUMUfHVHCmyfXkiQdzdIP91+L9U4OqtCREsLxbJv3isAYRACYvhIef7oTzqV9VdRNvG2k+aMdh5P1Mfltal/EQ0Za95p+PS0n/I7AIs+aIKjdYPGbbLZiW8LQ4PaIL3cS05zkbBZ0gzy19Wi/zlppXVdvXjcnrHdmlirRAt4qipQ/pksNyn2Q1w6L4QZBuCIuGMqAHr1RZ60s+q1lyjFRssuA3dWtcbWkDuDuD7cD040+IaXfQsNV15xa45pKooEZGJYXjeZZ060m7hlrq/PYiog5i7d+Fyso2EnTYmaMyVlqoKR/XS8Hei869HIDynOqkDMwJovJm1Am3+y6ToCqLqZ5s7B6an4+wS5x6hFyT+vR1NvjMe5nBtCcqkyYPclE32UwYi6ZsKwK1CjY6xjEgmsrqG1GSpsR7DCuiW2uQDPv97GvOLn9nkaLylP2uZ9qbIiM9517PsXKk0B5a+n+dQyj+cBXuGvvZjmacLiQjERA3N3/M5mDymdaqQKCthahidJrC9svUmu8AL4y0kkB//ACWlcyJR/QfhDTpFKBrFMpMc0bv7so+AeEs7YZo0hd b+rkymtG 8MzihfJltPHXfX7CKGYIswcQ70hIrC1ZoRm0KV2mvz3pzL2WoLKMhxrB3IYBgXUAJXFDkhIXfxYsJDQTV9dr4S2Sm7UabQNFxMvOedu94iTGs/v+3tIcDfMIbwZsyCu+B5UO+cFXbLhWVBifcicvydZCoLAD7QYxlohjcImQSw9nTpWsfMYJMDvRxd9c8EdqCEk2vai7bDiFntMoS3ePLGMJmqFQAGLRt9Zk//5Mm1BFGkfcHOM8MNtXegt1XnjibbFVS8R+535qTc8eIIzfzLM/K6wAlAc97LrzpGZx0O9WhTphqISRg1wf5pQSAkBd5euQqWqJcxDrNvEoaQNivqzlUxQ== X-Bogosity: Ham, tests=bogofilter, spamicity=0.000000, version=1.2.4 Sender: owner-linux-mm@kvack.org Precedence: bulk X-Loop: owner-majordomo@kvack.org List-ID: List-Subscribe: List-Unsubscribe: Teach the fork code to properly copy pfnmaps for pmd/pud levels. Pud is much easier, the write bit needs to be persisted though for writable and shared pud mappings like PFNMAP ones, otherwise a follow up write in either parent or child process will trigger a write fault. Do the same for pmd level. Signed-off-by: Peter Xu --- mm/huge_memory.c | 29 ++++++++++++++++++++++++++--- 1 file changed, 26 insertions(+), 3 deletions(-) diff --git a/mm/huge_memory.c b/mm/huge_memory.c index e2c314f631f3..15418ffdd377 100644 --- a/mm/huge_memory.c +++ b/mm/huge_memory.c @@ -1559,6 +1559,24 @@ int copy_huge_pmd(struct mm_struct *dst_mm, struct mm_struct *src_mm, pgtable_t pgtable = NULL; int ret = -ENOMEM; + pmd = pmdp_get_lockless(src_pmd); + if (unlikely(pmd_special(pmd))) { + dst_ptl = pmd_lock(dst_mm, dst_pmd); + src_ptl = pmd_lockptr(src_mm, src_pmd); + spin_lock_nested(src_ptl, SINGLE_DEPTH_NESTING); + /* + * No need to recheck the pmd, it can't change with write + * mmap lock held here. + * + * Meanwhile, making sure it's not a CoW VMA with writable + * mapping, otherwise it means either the anon page wrongly + * applied special bit, or we made the PRIVATE mapping be + * able to wrongly write to the backend MMIO. + */ + VM_WARN_ON_ONCE(is_cow_mapping(src_vma->vm_flags) && pmd_write(pmd)); + goto set_pmd; + } + /* Skip if can be re-fill on fault */ if (!vma_is_anonymous(dst_vma)) return 0; @@ -1640,7 +1658,9 @@ int copy_huge_pmd(struct mm_struct *dst_mm, struct mm_struct *src_mm, pmdp_set_wrprotect(src_mm, addr, src_pmd); if (!userfaultfd_wp(dst_vma)) pmd = pmd_clear_uffd_wp(pmd); - pmd = pmd_mkold(pmd_wrprotect(pmd)); + pmd = pmd_wrprotect(pmd); +set_pmd: + pmd = pmd_mkold(pmd); set_pmd_at(dst_mm, addr, dst_pmd, pmd); ret = 0; @@ -1686,8 +1706,11 @@ int copy_huge_pud(struct mm_struct *dst_mm, struct mm_struct *src_mm, * TODO: once we support anonymous pages, use * folio_try_dup_anon_rmap_*() and split if duplicating fails. */ - pudp_set_wrprotect(src_mm, addr, src_pud); - pud = pud_mkold(pud_wrprotect(pud)); + if (is_cow_mapping(vma->vm_flags) && pud_write(pud)) { + pudp_set_wrprotect(src_mm, addr, src_pud); + pud = pud_wrprotect(pud); + } + pud = pud_mkold(pud); set_pud_at(dst_mm, addr, dst_pud, pud); ret = 0; From patchwork Mon Aug 26 20:43:42 2024 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Peter Xu X-Patchwork-Id: 13778402 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from kanga.kvack.org (kanga.kvack.org [205.233.56.17]) by smtp.lore.kernel.org (Postfix) with ESMTP id 6B0D7C5472E for ; Mon, 26 Aug 2024 20:44:26 +0000 (UTC) Received: by kanga.kvack.org (Postfix) id 9BA606B009B; Mon, 26 Aug 2024 16:44:18 -0400 (EDT) Received: by kanga.kvack.org (Postfix, from userid 40) id 969166B009C; Mon, 26 Aug 2024 16:44:18 -0400 (EDT) X-Delivered-To: int-list-linux-mm@kvack.org Received: by kanga.kvack.org (Postfix, from userid 63042) id 76E926B009D; Mon, 26 Aug 2024 16:44:18 -0400 (EDT) X-Delivered-To: linux-mm@kvack.org Received: from relay.hostedemail.com (smtprelay0017.hostedemail.com [216.40.44.17]) by kanga.kvack.org (Postfix) with ESMTP id 569346B009B for ; Mon, 26 Aug 2024 16:44:18 -0400 (EDT) Received: from smtpin21.hostedemail.com (a10.router.float.18 [10.200.18.1]) by unirelay08.hostedemail.com (Postfix) with ESMTP id 0DD8D1413A5 for ; Mon, 26 Aug 2024 20:44:18 +0000 (UTC) X-FDA: 82495574196.21.075C215 Received: from us-smtp-delivery-124.mimecast.com (us-smtp-delivery-124.mimecast.com [170.10.129.124]) by imf18.hostedemail.com (Postfix) with ESMTP id DF4F51C0017 for ; Mon, 26 Aug 2024 20:44:15 +0000 (UTC) Authentication-Results: imf18.hostedemail.com; dkim=pass header.d=redhat.com header.s=mimecast20190719 header.b=MjeN0CQp; dmarc=pass (policy=none) header.from=redhat.com; spf=pass (imf18.hostedemail.com: domain of peterx@redhat.com designates 170.10.129.124 as permitted sender) smtp.mailfrom=peterx@redhat.com ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=hostedemail.com; s=arc-20220608; t=1724704960; h=from:from:sender:reply-to:subject:subject:date:date: message-id:message-id:to:to:cc:cc:mime-version:mime-version: content-type:content-type: content-transfer-encoding:content-transfer-encoding: in-reply-to:in-reply-to:references:references:dkim-signature; bh=vK1PMhUcFVpQCvjQRaUqQOYBcILrsdr8yUlCuFWve7Q=; b=M4J4qTP5ujeop1pJi6sqeHJNzfmeliZxKjv5vebvgAF42eArvNDIv2J0wayIzaZ5bUFNBU FBBBgCgyBRT9J8eCeB5/UdMBcNbNbbFUv8+yjToXSnt6zlA0uK5BVyEc99ITs65zYmz8Fv V4Ih+hggoCh50HgtnBwp4voY6ANc268= ARC-Seal: i=1; s=arc-20220608; d=hostedemail.com; t=1724704960; a=rsa-sha256; cv=none; b=s2JIuR3gvheU43vUnSqKr1zwOU+eRlEqI3Zb86hbAaz1CMtGite3eP/ONSeGbpGtWvq9A9 w+DuA6wsFpiyJqfgd3AqpjMI2DfWggUuVnoQjA4zmM4AhT//50k/D70fdKTAkX7Z7UPfhQ eFLAaG95SXiOvON81KCw3AOta7X52bo= ARC-Authentication-Results: i=1; imf18.hostedemail.com; dkim=pass header.d=redhat.com header.s=mimecast20190719 header.b=MjeN0CQp; dmarc=pass (policy=none) header.from=redhat.com; spf=pass (imf18.hostedemail.com: domain of peterx@redhat.com designates 170.10.129.124 as permitted sender) smtp.mailfrom=peterx@redhat.com DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=redhat.com; s=mimecast20190719; t=1724705055; h=from:from:reply-to:subject:subject:date:date:message-id:message-id: to:to:cc:cc:mime-version:mime-version:content-type:content-type: content-transfer-encoding:content-transfer-encoding: in-reply-to:in-reply-to:references:references; bh=vK1PMhUcFVpQCvjQRaUqQOYBcILrsdr8yUlCuFWve7Q=; b=MjeN0CQpFhyybJqWUDBzLl0VSCnEEdrkbztDtUb53F7Gu+g85RycsMkC0aGEl2xcygzlaW clJ5IR9Cg2mBDWz0ykQdXWrRwY3po08ds4dHS7ZJ/uEfgI0X+L3hpGkV1hyRg2+gvPmHWL FiNjq1/2B/aKtnGCyBXS1EoraxB06rA= Received: from mail-yw1-f198.google.com (mail-yw1-f198.google.com [209.85.128.198]) by relay.mimecast.com with ESMTP with STARTTLS (version=TLSv1.3, cipher=TLS_AES_256_GCM_SHA384) id us-mta-283-5GkLVTDnO9-BIbWb5ljY5g-1; Mon, 26 Aug 2024 16:44:14 -0400 X-MC-Unique: 5GkLVTDnO9-BIbWb5ljY5g-1 Received: by mail-yw1-f198.google.com with SMTP id 00721157ae682-6b351a76beeso102720347b3.0 for ; Mon, 26 Aug 2024 13:44:13 -0700 (PDT) X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20230601; t=1724705052; x=1725309852; h=content-transfer-encoding:mime-version:references:in-reply-to :message-id:date:subject:cc:to:from:x-gm-message-state:from:to:cc :subject:date:message-id:reply-to; bh=vK1PMhUcFVpQCvjQRaUqQOYBcILrsdr8yUlCuFWve7Q=; b=aIQHuLEUjVns9NU2ecLZwEKD18DmGL9nDfeLJhqQTMhquj1KJePRJGWRSScQHjUha2 Oe4bv3wVIytXR+qCVbed1Clf3tW+CJuQk9sMthxjb9hGj++8fyi+2ZgNUTbsFPFY2sJS R5q3hHo/Al34gD0NaAucrSPTlPiwXzBFFDHvaBO+Y+V2e7G+wp2iONQux4GA7ug7xEeB WwFF+1kSvCrHordC+zNWOl5EJupZuLo/21ZSc7MMz4wcYdjRsYZ6J0ws8m6k+JtMJAUU PUuvSrc9NHUcWHFJGdIjeT3jcjObfDmibxuQwyDLEmLzxYGqD+sDBVnLRgOvMejKfkuX T8OQ== X-Forwarded-Encrypted: i=1; AJvYcCX1piqvdaOGUURokDjrdrpnH+6Pa7I8tzP7RQd+LlFNEVSPXesv091pKAtmHk7JVxrnz3pJBeE2IA==@kvack.org X-Gm-Message-State: AOJu0Yzk2wAZFDO1G5od4lFJWn7MbX97hNSqVSt1cYsomHdsv9rmEoNn 3SA3jCIktRGAgfMNeBD2S/rMU4CnuP5N2UxlqDnI2NoHclo+5vN3b7gSj/sLygjja4KM3VKq1iT ybbChy+l3Kg0clmhAcRr8nDBRh1k1EudAlxS8PNeof6L8uwZg X-Received: by 2002:a05:690c:60c2:b0:6bd:fdd3:e170 with SMTP id 00721157ae682-6c624dc7042mr142338947b3.10.1724705052648; Mon, 26 Aug 2024 13:44:12 -0700 (PDT) X-Google-Smtp-Source: AGHT+IEDLUzP2hp4LBST6IRuRTdz5MRV8z70t30cnU5hPcRwmZ021NQU+FxAT+ZUF1I58Qs4qkGJLg== X-Received: by 2002:a05:690c:60c2:b0:6bd:fdd3:e170 with SMTP id 00721157ae682-6c624dc7042mr142338617b3.10.1724705052349; Mon, 26 Aug 2024 13:44:12 -0700 (PDT) Received: from x1n.redhat.com (pool-99-254-121-117.cpe.net.cable.rogers.com. [99.254.121.117]) by smtp.gmail.com with ESMTPSA id af79cd13be357-7a67f3fd6c1sm491055185a.121.2024.08.26.13.44.10 (version=TLS1_3 cipher=TLS_AES_256_GCM_SHA384 bits=256/256); Mon, 26 Aug 2024 13:44:11 -0700 (PDT) From: Peter Xu To: linux-kernel@vger.kernel.org, linux-mm@kvack.org Cc: Gavin Shan , Catalin Marinas , x86@kernel.org, Ingo Molnar , Andrew Morton , Paolo Bonzini , Dave Hansen , Thomas Gleixner , Alistair Popple , kvm@vger.kernel.org, linux-arm-kernel@lists.infradead.org, Sean Christopherson , peterx@redhat.com, Oscar Salvador , Jason Gunthorpe , Borislav Petkov , Zi Yan , Axel Rasmussen , David Hildenbrand , Yan Zhao , Will Deacon , Kefeng Wang , Alex Williamson Subject: [PATCH v2 08/19] mm: Always define pxx_pgprot() Date: Mon, 26 Aug 2024 16:43:42 -0400 Message-ID: <20240826204353.2228736-9-peterx@redhat.com> X-Mailer: git-send-email 2.45.0 In-Reply-To: <20240826204353.2228736-1-peterx@redhat.com> References: <20240826204353.2228736-1-peterx@redhat.com> MIME-Version: 1.0 X-Mimecast-Spam-Score: 0 X-Mimecast-Originator: redhat.com X-Rspamd-Server: rspam07 X-Rspamd-Queue-Id: DF4F51C0017 X-Stat-Signature: uatpfa5ec8p9etspjmeo6c8ho9e7hr8e X-Rspam-User: X-HE-Tag: 1724705055-323524 X-HE-Meta: U2FsdGVkX18Vnjq0yJzN1lj/2ebYHeuOfxd+jIj2NvD7lktp5Sl5BL98NGjITlXE6ty91lSRT5cvzMe//LWF5nvsDs0wAoYjG/FzuUYMiQvk1cEbpAMWT1CTJo1cFRCxt+uo/ongul8uyLYXBLF1OWkwdzl66E8+gco2Fu1b4eezD5AdLbcSdgdQbvMf1XacBHN58GG58JRjrxEDxvcBDsPiJjvRikpGwZ2wGx/1HJy2jy1b1GEVIJYZ4y8eUzwm/9o8lFnJyBriLzzS57XkjNlBAKBMo9S/AFDc3lNQztbtzU7uZq8K9jdxR0YByEiVdul+oJQbrp9fUKdU8m4LUZbktbgorXDdUKgiwOoRykSXKID1XRBo7ovovgG+99m5KgwPM8ASY+RNxrd8eQOwiuFo+Hr28VjUyspVIszFKIfIGGHW0ConxGLfS+mreZgP2+Yd5xbc6ANQzsqmefh7aNZ/5ca5PxZPuXhEvZFbWmFgGq7sbU5YB2qLgwlzkjSSnrpYKidYjYA1hEO48TBVVUYekpowhMGFMPX3T+jN1vzkzHQfCVW0mIbJbC/D9u9ZlSM0QuXHq6DuXHzAfkAg//kuBDQaE83r/CaxQkNEBCgOCUQ58r88bCVrMxG6m9qUh/WdAjgFZKDBi/iHqpjuGl5jG6F9AxY66b/T3WVJ5sJ3LfYctRx8drr3wzg+eylUgGxtNt2zaCCXLRiXfRoO8Q7OHfznFVRL6tC8H51hoJBR1/27srNb8dtY0mJiMiv943ApgNjvz0pmPhewb3B7hvR70r0CMrNElXoNxkGlRY5ltoqiHDCgtMRbI7MamJLMt+wfty0TVPyvPrCM5Bz2mZeAK4uB034HSn9tFjiQujzyAEFS26L3piFQWQ0ELaa5Z1eqNJKgqq/aYTGGL/i0NXrClkMN1NjKjK+LZJwrePTpymZy/WEvd366S3u5Uyp+y0HBN3RwNwW9EYbAVEn MVVNto3C mPoRoCxZ8poDQwbZE4FPhTXK9kB66xYUYO6s5i69jGEMyGmZD4K1IzX2PG1TbcUjQRlnZzEDfc82lNVvVG2LFMjOplDtgdT1t7r7Wfk/I737teOZRFOypE26FMfqp+Ap//gTFVZ5yUBtO08+h/SiB6OOHRmCAYaBhgoR3v3JTVsE/9wP225GTLs263EJ2TCmEU7B5G35E6ooq+lSuI5Tfp1Mldhnx/74seYRvYV3Shhe1Um1KgE0XTVIt+330GURg2xBptgf2jZRO7v3Bk+l8gZsnOv+g8srezFn42Ezh/ALyha5RjTi/WM+PZAHltFUrwKuKLnbUHSFwRhiAPKCBo8F1oWUmnFjGXfIDqHM6s3B/mNgDuPnvK/IEzALjjeLl7czaQrgLdvnJFxMbqduu+4xVsX+6AiG27hdF X-Bogosity: Ham, tests=bogofilter, spamicity=0.000000, version=1.2.4 Sender: owner-linux-mm@kvack.org Precedence: bulk X-Loop: owner-majordomo@kvack.org List-ID: List-Subscribe: List-Unsubscribe: There're: - 8 archs (arc, arm64, include, mips, powerpc, s390, sh, x86) that support pte_pgprot(). - 2 archs (x86, sparc) that support pmd_pgprot(). - 1 arch (x86) that support pud_pgprot(). Always define them to be used in generic code, and then we don't need to fiddle with "#ifdef"s when doing so. Reviewed-by: Jason Gunthorpe Signed-off-by: Peter Xu --- arch/arm64/include/asm/pgtable.h | 1 + arch/powerpc/include/asm/pgtable.h | 1 + arch/s390/include/asm/pgtable.h | 1 + arch/sparc/include/asm/pgtable_64.h | 1 + include/linux/pgtable.h | 12 ++++++++++++ 5 files changed, 16 insertions(+) diff --git a/arch/arm64/include/asm/pgtable.h b/arch/arm64/include/asm/pgtable.h index 7a4f5604be3f..b78cc4a6758b 100644 --- a/arch/arm64/include/asm/pgtable.h +++ b/arch/arm64/include/asm/pgtable.h @@ -384,6 +384,7 @@ static inline void __sync_cache_and_tags(pte_t pte, unsigned int nr_pages) /* * Select all bits except the pfn */ +#define pte_pgprot pte_pgprot static inline pgprot_t pte_pgprot(pte_t pte) { unsigned long pfn = pte_pfn(pte); diff --git a/arch/powerpc/include/asm/pgtable.h b/arch/powerpc/include/asm/pgtable.h index 264a6c09517a..2f72ad885332 100644 --- a/arch/powerpc/include/asm/pgtable.h +++ b/arch/powerpc/include/asm/pgtable.h @@ -65,6 +65,7 @@ static inline unsigned long pte_pfn(pte_t pte) /* * Select all bits except the pfn */ +#define pte_pgprot pte_pgprot static inline pgprot_t pte_pgprot(pte_t pte) { unsigned long pte_flags; diff --git a/arch/s390/include/asm/pgtable.h b/arch/s390/include/asm/pgtable.h index 3fa280d0672a..0ffbaf741955 100644 --- a/arch/s390/include/asm/pgtable.h +++ b/arch/s390/include/asm/pgtable.h @@ -955,6 +955,7 @@ static inline int pte_unused(pte_t pte) * young/old accounting is not supported, i.e _PAGE_PROTECT and _PAGE_INVALID * must not be set. */ +#define pte_pgprot pte_pgprot static inline pgprot_t pte_pgprot(pte_t pte) { unsigned long pte_flags = pte_val(pte) & _PAGE_CHG_MASK; diff --git a/arch/sparc/include/asm/pgtable_64.h b/arch/sparc/include/asm/pgtable_64.h index 3fe429d73a65..2b7f358762c1 100644 --- a/arch/sparc/include/asm/pgtable_64.h +++ b/arch/sparc/include/asm/pgtable_64.h @@ -783,6 +783,7 @@ static inline pmd_t pmd_mkwrite_novma(pmd_t pmd) return __pmd(pte_val(pte)); } +#define pmd_pgprot pmd_pgprot static inline pgprot_t pmd_pgprot(pmd_t entry) { unsigned long val = pmd_val(entry); diff --git a/include/linux/pgtable.h b/include/linux/pgtable.h index 780f3b439d98..e8b2ac6bd2ae 100644 --- a/include/linux/pgtable.h +++ b/include/linux/pgtable.h @@ -1956,6 +1956,18 @@ typedef unsigned int pgtbl_mod_mask; #define MAX_PTRS_PER_P4D PTRS_PER_P4D #endif +#ifndef pte_pgprot +#define pte_pgprot(x) ((pgprot_t) {0}) +#endif + +#ifndef pmd_pgprot +#define pmd_pgprot(x) ((pgprot_t) {0}) +#endif + +#ifndef pud_pgprot +#define pud_pgprot(x) ((pgprot_t) {0}) +#endif + /* description of effects of mapping type and prot in current implementation. * this is due to the limited x86 page protection hardware. The expected * behavior is in parens: From patchwork Mon Aug 26 20:43:43 2024 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Peter Xu X-Patchwork-Id: 13778403 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from kanga.kvack.org (kanga.kvack.org [205.233.56.17]) by smtp.lore.kernel.org (Postfix) with ESMTP id 24ECCC5472C for ; Mon, 26 Aug 2024 20:44:29 +0000 (UTC) Received: by kanga.kvack.org (Postfix) id 9E0616B009D; Mon, 26 Aug 2024 16:44:19 -0400 (EDT) Received: by kanga.kvack.org (Postfix, from userid 40) id 992866B009E; Mon, 26 Aug 2024 16:44:19 -0400 (EDT) X-Delivered-To: int-list-linux-mm@kvack.org Received: by kanga.kvack.org (Postfix, from userid 63042) id 808D86B009F; Mon, 26 Aug 2024 16:44:19 -0400 (EDT) X-Delivered-To: linux-mm@kvack.org Received: from relay.hostedemail.com (smtprelay0017.hostedemail.com [216.40.44.17]) by kanga.kvack.org (Postfix) with ESMTP id 5B13E6B009D for ; Mon, 26 Aug 2024 16:44:19 -0400 (EDT) Received: from smtpin05.hostedemail.com (a10.router.float.18 [10.200.18.1]) by unirelay07.hostedemail.com (Postfix) with ESMTP id 16BC31613C1 for ; Mon, 26 Aug 2024 20:44:19 +0000 (UTC) X-FDA: 82495574238.05.7EBCE80 Received: from us-smtp-delivery-124.mimecast.com (us-smtp-delivery-124.mimecast.com [170.10.133.124]) by imf10.hostedemail.com (Postfix) with ESMTP id F0997C001E for ; Mon, 26 Aug 2024 20:44:16 +0000 (UTC) Authentication-Results: imf10.hostedemail.com; dkim=pass header.d=redhat.com header.s=mimecast20190719 header.b=GB51IwsU; spf=pass (imf10.hostedemail.com: domain of peterx@redhat.com designates 170.10.133.124 as permitted sender) smtp.mailfrom=peterx@redhat.com; dmarc=pass (policy=none) header.from=redhat.com ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=hostedemail.com; s=arc-20220608; t=1724704970; h=from:from:sender:reply-to:subject:subject:date:date: message-id:message-id:to:to:cc:cc:mime-version:mime-version: content-type:content-type: content-transfer-encoding:content-transfer-encoding: in-reply-to:in-reply-to:references:references:dkim-signature; bh=XzWn17lHZnGD5Tr66iu6y8/zqhTRy72agTNrtdvxzwE=; b=fUty0eAqWEpKKlL40qM2qq2fJ6OmrIPuAlNtjQoRlEFaKQTRMxvP6WTU+VyDxXaJTFnl85 LPo0BNWpRM8mFLd6pGYvrlpeQFhgn0YOHHusOkRNn93ya3UZ1SowKq8ZNjE+hTeaYVIQqj jfwkiPTuzKmOuxz3m1+3BcBDV6HVN/8= ARC-Seal: i=1; s=arc-20220608; d=hostedemail.com; t=1724704970; a=rsa-sha256; cv=none; b=PAzLufNIdPrRXb1FWBDrFm6/M5HWDLqsyJBwk/e0PdAxVBQpwzY1+3Hrhu2/NyusbDMLQP uGUX8FgHqFp5gFnsl92gX15RIzWlQote91SJjs9K+1XMlttoQWRXipbSUMw1TbeblVS8cG UDRp6JSNYYYbDPZAA+f0b1O0kAohYV8= ARC-Authentication-Results: i=1; imf10.hostedemail.com; dkim=pass header.d=redhat.com header.s=mimecast20190719 header.b=GB51IwsU; spf=pass (imf10.hostedemail.com: domain of peterx@redhat.com designates 170.10.133.124 as permitted sender) smtp.mailfrom=peterx@redhat.com; dmarc=pass (policy=none) header.from=redhat.com DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=redhat.com; s=mimecast20190719; t=1724705056; h=from:from:reply-to:subject:subject:date:date:message-id:message-id: to:to:cc:cc:mime-version:mime-version:content-type:content-type: content-transfer-encoding:content-transfer-encoding: in-reply-to:in-reply-to:references:references; bh=XzWn17lHZnGD5Tr66iu6y8/zqhTRy72agTNrtdvxzwE=; b=GB51IwsUFGz9QiMYb1aeCVArQshtX9/BPqs22vaRFMbgUPbZna0dbgweERVBax6aL419Md lsk2E8/+C62PpHINsxJHpsHa7XVHdtx3ZQR7ceynlKMx7H+BPLTi1ab8FvLjNePv/RzKwv +wsyS6nN6mCeHLCbYt7eNpBfge5Til4= Received: from mail-qk1-f198.google.com (mail-qk1-f198.google.com [209.85.222.198]) by relay.mimecast.com with ESMTP with STARTTLS (version=TLSv1.3, cipher=TLS_AES_256_GCM_SHA384) id us-mta-678-zsijSCRTOGqEWOseYysFXA-1; Mon, 26 Aug 2024 16:44:15 -0400 X-MC-Unique: zsijSCRTOGqEWOseYysFXA-1 Received: by mail-qk1-f198.google.com with SMTP id af79cd13be357-7a1df6b3a26so630267885a.1 for ; Mon, 26 Aug 2024 13:44:15 -0700 (PDT) X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20230601; t=1724705055; x=1725309855; h=content-transfer-encoding:mime-version:references:in-reply-to :message-id:date:subject:cc:to:from:x-gm-message-state:from:to:cc :subject:date:message-id:reply-to; bh=XzWn17lHZnGD5Tr66iu6y8/zqhTRy72agTNrtdvxzwE=; b=SV5Ydji92PtPI3ogZ6fYuYR+rdOGmbMsNY9Gl+DLBXAJRComAoTTMeBzMA2zo3HtRY wR0d0xlSSpwMArStu0PJILuO2r36nwzojMLBv/S1qUUopy6df9+i/NEZn4AxmQXTONyo icAFG3xxdbXfXigd2ONZr7+yo9gomWwA/j9jVTrpGI2c8Y7KzeebLlwuUm6Bld4Tf+k2 Ptu+0Ak1oFDQmDwvd2MMaUu5XSCwEO5dRVPt2fR7XP4DLK0IBF1+GjCaMA7LP78RlgNa /m/Tkm/HzI+VspH78nu6FpOZJmXItxObg2CAtop60z/lFQpu5S+EnbuX59WBmLaaxpeO 01/w== X-Forwarded-Encrypted: i=1; AJvYcCXzugR89WqdOHj4ycbTYJ/fu4QAlx3GLdcZL3mj1mc76c9G7+0wux2dKbokV7XXR+1fn0rU1KJJnw==@kvack.org X-Gm-Message-State: AOJu0Ywh4rMgz1oVJZOwLYxADB1qd04tuLhkjEYlgZ2QgFNQytmdaEnS a0L5QQhzYd8Hs1q6st1GdBIyLHlziEQp7Gk09ToWmS8YV1vf77H8eaw7x/0C+zd3R/Zwf8mPEnI PwdXnVa5BZLUcArkcRcgf8D8TlohyD+td2fp4GKSDNBeMTz4r X-Received: by 2002:a05:620a:408d:b0:7a1:e93c:cd04 with SMTP id af79cd13be357-7a6896e0e1cmr1465305785a.9.1724705054682; Mon, 26 Aug 2024 13:44:14 -0700 (PDT) X-Google-Smtp-Source: AGHT+IH4kwi4qTohFBFu7kWFKQGbZI396sZSJGmlCvwBl5XfpN0sU8TboRNIhnEyWXLKkBzi4MYP4w== X-Received: by 2002:a05:620a:408d:b0:7a1:e93c:cd04 with SMTP id af79cd13be357-7a6896e0e1cmr1465302885a.9.1724705054299; Mon, 26 Aug 2024 13:44:14 -0700 (PDT) Received: from x1n.redhat.com (pool-99-254-121-117.cpe.net.cable.rogers.com. [99.254.121.117]) by smtp.gmail.com with ESMTPSA id af79cd13be357-7a67f3fd6c1sm491055185a.121.2024.08.26.13.44.12 (version=TLS1_3 cipher=TLS_AES_256_GCM_SHA384 bits=256/256); Mon, 26 Aug 2024 13:44:13 -0700 (PDT) From: Peter Xu To: linux-kernel@vger.kernel.org, linux-mm@kvack.org Cc: Gavin Shan , Catalin Marinas , x86@kernel.org, Ingo Molnar , Andrew Morton , Paolo Bonzini , Dave Hansen , Thomas Gleixner , Alistair Popple , kvm@vger.kernel.org, linux-arm-kernel@lists.infradead.org, Sean Christopherson , peterx@redhat.com, Oscar Salvador , Jason Gunthorpe , Borislav Petkov , Zi Yan , Axel Rasmussen , David Hildenbrand , Yan Zhao , Will Deacon , Kefeng Wang , Alex Williamson Subject: [PATCH v2 09/19] mm: New follow_pfnmap API Date: Mon, 26 Aug 2024 16:43:43 -0400 Message-ID: <20240826204353.2228736-10-peterx@redhat.com> X-Mailer: git-send-email 2.45.0 In-Reply-To: <20240826204353.2228736-1-peterx@redhat.com> References: <20240826204353.2228736-1-peterx@redhat.com> MIME-Version: 1.0 X-Mimecast-Spam-Score: 0 X-Mimecast-Originator: redhat.com X-Stat-Signature: 1muzcdosamimitpc9iu544mkom1g6kga X-Rspamd-Queue-Id: F0997C001E X-Rspam-User: X-Rspamd-Server: rspam08 X-HE-Tag: 1724705056-683710 X-HE-Meta: U2FsdGVkX1+NthvfzNDO5vfzwiS49exXfY5q/CExCMBqWKg19kCoF/BZeizi+DLIlc2MtVdYZGDeR0xXAzVezh3FZDQdBqm+5K9KMrQ6eOmu/AbGLkTLYHm6ZZGVAXMG4OeBhVIKGdRNrtds6bal8CZR4+jFyATaTxEPAXR/j3dG7yIY2qvxEvTIdpSmMvehOwIg2bd/5lDDuZecPO9uOPHdnL0dkVbJ2Kh7/RMskBu7ktkYVa1JksCkp9T1qGU2ZdDd85AUrXzmfmTCdtYT+ZoOcM17Su6J6tAS2y/jnmkvUOjdVtOngQ54NmOE6CS2LaV7+L2Pi52qhsCUzXd6wt4/jdSKSPIVS4OsovUU2q+qWW7FhvJknqoTR/X5P+2cIfa3FWlk8GPL86VpNupcfjpx5bL1rmjn/PVnk6VsVgzlV788eKnofgiKeIkrdAVNz+G+rKaQzfxgn20c/BdvA8N0M9I2k+gm6B7OETsyk9dgvZCEShvevPxp8wsDvKD+6WNSDwdBu3jk3mPfYVUG1DoobnWDlHNo7mJdUQnLn1VaCp9pzz5I/dfIjAc9io570j+5N3EDCFXQUTi9JHNHvDFAaU6hdJfa8fLiAJBIS1at5t2Ge1o7ZofJiRkzeEja2D/NyOvPJ4HBNbRXlh9L8R0hNNaA41wR/6a1QjuRFeXADLVYKjZoMbka6OT/GZFj5l7Avi50HJnEKGMAW6ynJ9THeuu/1vXd30fxcN0mQ1jv1oAC1NPk5rFlvBLIQYUSjrk0Yf9/L1mCMOVZS6/FK5KNXrh5DrwXRFIoZ84XRqhkKSmrItzSJbwod6sip3yuvbQXQA3pKSzTxBM0j7qGCTTKNIaiIT6W4/y9RmxfWPRn5DgSa+3dlAf4JtQ436ympoDL1N3i4bEz9COkb8n6+6kvS0ny7WQsbz6UMIRErjhy6qumEAoKmz0R4Mxi9KmUYyQHfTQKslfBjotqHQe Kw46gOLh iC5Z6gaFh5M/rzc0NPCDDakBq5OqImny43lGy/HbUkSPuiDDELV2/u41O5DTzSwnOQfuyOZjmq01RkxK8CSj9XcFkzT4FVMNJSuaqAOb9WDM+XG2mS7Y8MCS4cnUVSYZzwGVBeT3R3jVSee9NTQGcWuZrGTPwDepJ72gvkhE8zkvkvp0a1kTGCaAPZiLNMHdf4hHYoMKrIs3ZQRwTTZzv32Vr9EG8xqJjlgj25clz6acHPQgF8sHSzl4c3Fl4h/btiUGCb3QQtfz0BfW8iMX+l1zZT8UCLv6h+ysL7IL+htxVOBqyn+PRJcmSfR00OtoE+lpw7JBiIeEGn/FZUDgAZrs7GEauEo7Ot8dV7p4HMKf+QPoIP0JZvBtv9XIfdek/w/XKmdlYQ0BGGH4MG/iXF8MJkdcKhPCHJYT2zrFJ2GjFFhADIlk7gsyLBXVQYOvUInOUZX/Vt2it790rDe3ADHiKFVevbKlgJSfsj+Dx/6shPhLmMFxzO/xUkOYfwUD9sUEU X-Bogosity: Ham, tests=bogofilter, spamicity=0.000000, version=1.2.4 Sender: owner-linux-mm@kvack.org Precedence: bulk X-Loop: owner-majordomo@kvack.org List-ID: List-Subscribe: List-Unsubscribe: Introduce a pair of APIs to follow pfn mappings to get entry information. It's very similar to what follow_pte() does before, but different in that it recognizes huge pfn mappings. Signed-off-by: Peter Xu --- include/linux/mm.h | 31 ++++++++++ mm/memory.c | 150 +++++++++++++++++++++++++++++++++++++++++++++ 2 files changed, 181 insertions(+) diff --git a/include/linux/mm.h b/include/linux/mm.h index d900f15b7650..161d496bfd18 100644 --- a/include/linux/mm.h +++ b/include/linux/mm.h @@ -2373,6 +2373,37 @@ int follow_pte(struct vm_area_struct *vma, unsigned long address, int generic_access_phys(struct vm_area_struct *vma, unsigned long addr, void *buf, int len, int write); +struct follow_pfnmap_args { + /** + * Inputs: + * @vma: Pointer to @vm_area_struct struct + * @address: the virtual address to walk + */ + struct vm_area_struct *vma; + unsigned long address; + /** + * Internals: + * + * The caller shouldn't touch any of these. + */ + spinlock_t *lock; + pte_t *ptep; + /** + * Outputs: + * + * @pfn: the PFN of the address + * @pgprot: the pgprot_t of the mapping + * @writable: whether the mapping is writable + * @special: whether the mapping is a special mapping (real PFN maps) + */ + unsigned long pfn; + pgprot_t pgprot; + bool writable; + bool special; +}; +int follow_pfnmap_start(struct follow_pfnmap_args *args); +void follow_pfnmap_end(struct follow_pfnmap_args *args); + extern void truncate_pagecache(struct inode *inode, loff_t new); extern void truncate_setsize(struct inode *inode, loff_t newsize); void pagecache_isize_extended(struct inode *inode, loff_t from, loff_t to); diff --git a/mm/memory.c b/mm/memory.c index 93c0c25433d0..0b136c398257 100644 --- a/mm/memory.c +++ b/mm/memory.c @@ -6173,6 +6173,156 @@ int follow_pte(struct vm_area_struct *vma, unsigned long address, } EXPORT_SYMBOL_GPL(follow_pte); +static inline void pfnmap_args_setup(struct follow_pfnmap_args *args, + spinlock_t *lock, pte_t *ptep, + pgprot_t pgprot, unsigned long pfn_base, + unsigned long addr_mask, bool writable, + bool special) +{ + args->lock = lock; + args->ptep = ptep; + args->pfn = pfn_base + ((args->address & ~addr_mask) >> PAGE_SHIFT); + args->pgprot = pgprot; + args->writable = writable; + args->special = special; +} + +static inline void pfnmap_lockdep_assert(struct vm_area_struct *vma) +{ +#ifdef CONFIG_LOCKDEP + struct address_space *mapping = vma->vm_file->f_mapping; + + if (mapping) + lockdep_assert(lockdep_is_held(&vma->vm_file->f_mapping->i_mmap_rwsem) || + lockdep_is_held(&vma->vm_mm->mmap_lock)); + else + lockdep_assert(lockdep_is_held(&vma->vm_mm->mmap_lock)); +#endif +} + +/** + * follow_pfnmap_start() - Look up a pfn mapping at a user virtual address + * @args: Pointer to struct @follow_pfnmap_args + * + * The caller needs to setup args->vma and args->address to point to the + * virtual address as the target of such lookup. On a successful return, + * the results will be put into other output fields. + * + * After the caller finished using the fields, the caller must invoke + * another follow_pfnmap_end() to proper releases the locks and resources + * of such look up request. + * + * During the start() and end() calls, the results in @args will be valid + * as proper locks will be held. After the end() is called, all the fields + * in @follow_pfnmap_args will be invalid to be further accessed. Further + * use of such information after end() may require proper synchronizations + * by the caller with page table updates, otherwise it can create a + * security bug. + * + * If the PTE maps a refcounted page, callers are responsible to protect + * against invalidation with MMU notifiers; otherwise access to the PFN at + * a later point in time can trigger use-after-free. + * + * Only IO mappings and raw PFN mappings are allowed. The mmap semaphore + * should be taken for read, and the mmap semaphore cannot be released + * before the end() is invoked. + * + * This function must not be used to modify PTE content. + * + * Return: zero on success, negative otherwise. + */ +int follow_pfnmap_start(struct follow_pfnmap_args *args) +{ + struct vm_area_struct *vma = args->vma; + unsigned long address = args->address; + struct mm_struct *mm = vma->vm_mm; + spinlock_t *lock; + pgd_t *pgdp; + p4d_t *p4dp, p4d; + pud_t *pudp, pud; + pmd_t *pmdp, pmd; + pte_t *ptep, pte; + + pfnmap_lockdep_assert(vma); + + if (unlikely(address < vma->vm_start || address >= vma->vm_end)) + goto out; + + if (!(vma->vm_flags & (VM_IO | VM_PFNMAP))) + goto out; +retry: + pgdp = pgd_offset(mm, address); + if (pgd_none(*pgdp) || unlikely(pgd_bad(*pgdp))) + goto out; + + p4dp = p4d_offset(pgdp, address); + p4d = READ_ONCE(*p4dp); + if (p4d_none(p4d) || unlikely(p4d_bad(p4d))) + goto out; + + pudp = pud_offset(p4dp, address); + pud = READ_ONCE(*pudp); + if (pud_none(pud)) + goto out; + if (pud_leaf(pud)) { + lock = pud_lock(mm, pudp); + if (!unlikely(pud_leaf(pud))) { + spin_unlock(lock); + goto retry; + } + pfnmap_args_setup(args, lock, NULL, pud_pgprot(pud), + pud_pfn(pud), PUD_MASK, pud_write(pud), + pud_special(pud)); + return 0; + } + + pmdp = pmd_offset(pudp, address); + pmd = pmdp_get_lockless(pmdp); + if (pmd_leaf(pmd)) { + lock = pmd_lock(mm, pmdp); + if (!unlikely(pmd_leaf(pmd))) { + spin_unlock(lock); + goto retry; + } + pfnmap_args_setup(args, lock, NULL, pmd_pgprot(pmd), + pmd_pfn(pmd), PMD_MASK, pmd_write(pmd), + pmd_special(pmd)); + return 0; + } + + ptep = pte_offset_map_lock(mm, pmdp, address, &lock); + if (!ptep) + goto out; + pte = ptep_get(ptep); + if (!pte_present(pte)) + goto unlock; + pfnmap_args_setup(args, lock, ptep, pte_pgprot(pte), + pte_pfn(pte), PAGE_MASK, pte_write(pte), + pte_special(pte)); + return 0; +unlock: + pte_unmap_unlock(ptep, lock); +out: + return -EINVAL; +} +EXPORT_SYMBOL_GPL(follow_pfnmap_start); + +/** + * follow_pfnmap_end(): End a follow_pfnmap_start() process + * @args: Pointer to struct @follow_pfnmap_args + * + * Must be used in pair of follow_pfnmap_start(). See the start() function + * above for more information. + */ +void follow_pfnmap_end(struct follow_pfnmap_args *args) +{ + if (args->lock) + spin_unlock(args->lock); + if (args->ptep) + pte_unmap(args->ptep); +} +EXPORT_SYMBOL_GPL(follow_pfnmap_end); + #ifdef CONFIG_HAVE_IOREMAP_PROT /** * generic_access_phys - generic implementation for iomem mmap access From patchwork Mon Aug 26 20:43:44 2024 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Peter Xu X-Patchwork-Id: 13778404 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from kanga.kvack.org (kanga.kvack.org [205.233.56.17]) by smtp.lore.kernel.org (Postfix) with ESMTP id C1AECC5321D for ; Mon, 26 Aug 2024 20:44:31 +0000 (UTC) Received: by kanga.kvack.org (Postfix) id ACC696B009F; Mon, 26 Aug 2024 16:44:21 -0400 (EDT) Received: by kanga.kvack.org (Postfix, from userid 40) id A60336B00A1; Mon, 26 Aug 2024 16:44:21 -0400 (EDT) X-Delivered-To: int-list-linux-mm@kvack.org Received: by kanga.kvack.org (Postfix, from userid 63042) id 7E0EE6B00A0; Mon, 26 Aug 2024 16:44:21 -0400 (EDT) X-Delivered-To: linux-mm@kvack.org Received: from relay.hostedemail.com (smtprelay0013.hostedemail.com [216.40.44.13]) by kanga.kvack.org (Postfix) with ESMTP id 58F006B009E for ; Mon, 26 Aug 2024 16:44:21 -0400 (EDT) Received: from smtpin21.hostedemail.com (a10.router.float.18 [10.200.18.1]) by unirelay02.hostedemail.com (Postfix) with ESMTP id 09041121396 for ; Mon, 26 Aug 2024 20:44:21 +0000 (UTC) X-FDA: 82495574322.21.E2D831C Received: from us-smtp-delivery-124.mimecast.com (us-smtp-delivery-124.mimecast.com [170.10.133.124]) by imf23.hostedemail.com (Postfix) with ESMTP id E4AC0140008 for ; Mon, 26 Aug 2024 20:44:18 +0000 (UTC) Authentication-Results: imf23.hostedemail.com; dkim=pass header.d=redhat.com header.s=mimecast20190719 header.b=I8EyohvP; spf=pass (imf23.hostedemail.com: domain of peterx@redhat.com designates 170.10.133.124 as permitted sender) smtp.mailfrom=peterx@redhat.com; dmarc=pass (policy=none) header.from=redhat.com ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=hostedemail.com; s=arc-20220608; t=1724704972; h=from:from:sender:reply-to:subject:subject:date:date: message-id:message-id:to:to:cc:cc:mime-version:mime-version: content-type:content-type: content-transfer-encoding:content-transfer-encoding: in-reply-to:in-reply-to:references:references:dkim-signature; bh=RcTv8PG6WUMces5zPq/dQ8qXZuImd+6bX+nNmvbIpbY=; b=O7UlkbYOy2rcHisnn+6+dOHCHPXQKvnOvtkFuADH8WL6Gz2uVvaF6THtFa+ucQMBClNIdU m29gz/b/Z60Cetkx8nQhMiUHavXoi9FIytOcv3muaTO/gNp/nM2UtNlL2kHumarBDAa7Ty Hu4BSFsaL9Sy7Pa1Ww3oD2gujw8SREg= ARC-Seal: i=1; s=arc-20220608; d=hostedemail.com; t=1724704972; a=rsa-sha256; cv=none; b=Rgtfi8QVwzm+oJI+Npr44xhbSYDeMfDaUhnVWyarHXhLLlgw4jLA0QR+X/ehXFsW2LawhO yySZKGqyvLDHmRUDAaCNdIK7RQ27woYafe/x4waBVjrJNFCrKdoKXlzEaE0MYF+TBOhTLv ktHbhhCbRof9My6DVjTzIWW9abZznQk= ARC-Authentication-Results: i=1; imf23.hostedemail.com; dkim=pass header.d=redhat.com header.s=mimecast20190719 header.b=I8EyohvP; spf=pass (imf23.hostedemail.com: domain of peterx@redhat.com designates 170.10.133.124 as permitted sender) smtp.mailfrom=peterx@redhat.com; dmarc=pass (policy=none) header.from=redhat.com DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=redhat.com; s=mimecast20190719; t=1724705058; h=from:from:reply-to:subject:subject:date:date:message-id:message-id: to:to:cc:cc:mime-version:mime-version:content-type:content-type: content-transfer-encoding:content-transfer-encoding: in-reply-to:in-reply-to:references:references; bh=RcTv8PG6WUMces5zPq/dQ8qXZuImd+6bX+nNmvbIpbY=; b=I8EyohvPdwcIg3kArl1LTRYt9zwIvuGxiAQJP+1J0tgK5vMuVxP4bpdqVL8va3o2o/ybJr uVSc2joH5AWRJRW/AKI6SbWnIUxWE3K1m4W7N5TMW4QVCBLi7/bGthYrzVd598iQdt38KB pX3PDZzAhNdgf35Cvnhvlag7IGnnfBI= Received: from mail-yw1-f199.google.com (mail-yw1-f199.google.com [209.85.128.199]) by relay.mimecast.com with ESMTP with STARTTLS (version=TLSv1.3, cipher=TLS_AES_256_GCM_SHA384) id us-mta-54-gM-UhamaMICb4kv_IvaKQA-1; Mon, 26 Aug 2024 16:44:17 -0400 X-MC-Unique: gM-UhamaMICb4kv_IvaKQA-1 Received: by mail-yw1-f199.google.com with SMTP id 00721157ae682-6b41e02c293so94041157b3.0 for ; Mon, 26 Aug 2024 13:44:17 -0700 (PDT) X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20230601; t=1724705056; x=1725309856; h=content-transfer-encoding:mime-version:references:in-reply-to :message-id:date:subject:cc:to:from:x-gm-message-state:from:to:cc :subject:date:message-id:reply-to; bh=RcTv8PG6WUMces5zPq/dQ8qXZuImd+6bX+nNmvbIpbY=; b=k7mQc59uUjNuYUhVHjUid35gkQng1YaMq3u0+LdNO4gZeH5c9pU6rNvyoNCDwoYmOr E1MrJ7N1cfldpkZiGaakBPU7Zk4Th8urLe10Kij+ll45qnMKXB/CaiAYI3GuotJlgs8W yMx9K/2ilh5NCopo0Ym633g9bvEc0+Sz3ZaMOVniFZeo3Zoc5pibZVE8EYSJUvVr/tk1 FjDnPAx1iYlJcOMQKnmG/Obmbe9tgo78ZVAu0NEBNBYLnyOGQMpt4tEE5sOO7KYzGgBl pBVNy3LSGV7hIugYWWRkOJpeOAsMqqotYwUrcz+tgkXzUy7OOXDHGXz/OhpIv48lGDtv 80og== X-Forwarded-Encrypted: i=1; AJvYcCUKIV/s0S/VMywJcSIJ1BmMCk379zUY17Ng2LNiTgiHe8hY+EJMyN60LpCgl7suvR+s1LbFDknzqQ==@kvack.org X-Gm-Message-State: AOJu0YxbdwvE0NYeachgx1pEhL429PudXGYbbLt9FccEzeb2+/BE6gM7 7rQvHb3tiniCm6aq2eWEYwt9jcAMPtuPeoIgd06kRrsiSZRBzJxynC9JhSmcmidg1u7JqT+eL1H OkRINaukr9petJXGMdtOmz7CUeo+ioKq9p/64YB5U3M7t4iad X-Received: by 2002:a05:690c:4246:b0:6b2:4154:e598 with SMTP id 00721157ae682-6c6262f43e4mr113946257b3.26.1724705056533; Mon, 26 Aug 2024 13:44:16 -0700 (PDT) X-Google-Smtp-Source: AGHT+IGCZvjuJnmUbtCZKJF68AHOtk4V9xJHAv9nFBXLLIOg3bv7tZNlMwBWnRJYIMuW8OX1o2s1rQ== X-Received: by 2002:a05:690c:4246:b0:6b2:4154:e598 with SMTP id 00721157ae682-6c6262f43e4mr113946177b3.26.1724705056224; Mon, 26 Aug 2024 13:44:16 -0700 (PDT) Received: from x1n.redhat.com (pool-99-254-121-117.cpe.net.cable.rogers.com. [99.254.121.117]) by smtp.gmail.com with ESMTPSA id af79cd13be357-7a67f3fd6c1sm491055185a.121.2024.08.26.13.44.14 (version=TLS1_3 cipher=TLS_AES_256_GCM_SHA384 bits=256/256); Mon, 26 Aug 2024 13:44:15 -0700 (PDT) From: Peter Xu To: linux-kernel@vger.kernel.org, linux-mm@kvack.org Cc: Gavin Shan , Catalin Marinas , x86@kernel.org, Ingo Molnar , Andrew Morton , Paolo Bonzini , Dave Hansen , Thomas Gleixner , Alistair Popple , kvm@vger.kernel.org, linux-arm-kernel@lists.infradead.org, Sean Christopherson , peterx@redhat.com, Oscar Salvador , Jason Gunthorpe , Borislav Petkov , Zi Yan , Axel Rasmussen , David Hildenbrand , Yan Zhao , Will Deacon , Kefeng Wang , Alex Williamson Subject: [PATCH v2 10/19] KVM: Use follow_pfnmap API Date: Mon, 26 Aug 2024 16:43:44 -0400 Message-ID: <20240826204353.2228736-11-peterx@redhat.com> X-Mailer: git-send-email 2.45.0 In-Reply-To: <20240826204353.2228736-1-peterx@redhat.com> References: <20240826204353.2228736-1-peterx@redhat.com> MIME-Version: 1.0 X-Mimecast-Spam-Score: 0 X-Mimecast-Originator: redhat.com X-Stat-Signature: r7i4qitkutw78q1bpbikja9uyc3g8yfm X-Rspamd-Queue-Id: E4AC0140008 X-Rspam-User: X-Rspamd-Server: rspam08 X-HE-Tag: 1724705058-975170 X-HE-Meta: U2FsdGVkX1/rdOKzr2iiVSOWneSbLZzyIELAuqndDNBWQrQFKIN2gq/g+lcdE8F3UjZuJdDUowLGoaUr1g09qut+Rv5Jdng+c3AFQpmqax7CcgN0BTWMQzZ1XXjr1E448lkq6q7vMyIT0RrKJNIudwaWJFqyGqsH0TUNPmrkc3LfUCGYJCU3/CyJVo9dCfMMp6tH6WrwWxWQm/ypmj/ywWVZ0quIghVnTYxLNARO6K8JHCnA9Jz0411u5++S9rXkqDEHVmqGf5sUoKs3EusMl1KRbtBR0kb06Sc2LuVwShudSK9rdyrtLX5S554zxSRosl6QfwUKa0oJOSJ9TpPLdBMTDqIWpP6vwjH35SAka5ZOhLosduN1Srp2YlTPDQWtDpKdnmM7T+cyd1NckzxIgRP4MH6FwSirDf/xSy/v/dd9N7YCddkAYWyd2Ix8uP2yZfgruuJDX1uYwccpYMFV/dJoV8c2pKy4kLdAjn+xAlg2UoeA6VTmsm5WAuCC6jfi0xNuVZ0LHoRoc6kj1p8IvXsj3NczOnvbHWtaiPZDGS+bEnps0MSyU66g7T+1M4Ia9+cu5x2bFL0np1qUOCQJ/XNf6VwawcCoeKwXJ5Ak6k6guWt0Lp7YMG/Wmpc6SlJrG5RJlba+mr/7sLjvJtPVk0AsHOAInlYm4j5f9z5zRCofoT3Vv0OYfGGKW0z40rmxr3JZaVGJxbl6gK4rHKAhwEpOUw8d/PN9pYuiA8XzUcdKoeYZUjCx2GBZeRhToNjUTJU27g766j/LgLdLEUdJoReNcUssXhDtVBPN0Pmetqk5BCTpGMjBbZuH+hOvvJWNvYT9vm83qV0OjMlr+miMkmnKhZgQ6xy+wMgWMba9e363kfIFnKUK54fowXiHbH5Tmbsjf+J1gAs1yVX5gONG/tHica5fEXK83l43HpaEj6LQ/RdsHcyucaF7FOjABy/fmyIjqPf0Xfg9JV0lkUC H4Hrl9ad 4Jkf8Rb+8gaOdsA8I/IQdbTPv4IrhlocVexxYVcLwOSu0xMfpWR7+6U9YnkWNQ0hX5YLrOO54PWA2lW1A1lPnFwclYyHTf+PnQoXNfQl8BqTHeM3u9WhZZYC1hFnpzSbQnY7PWtZXztPM2EUeJgOCEOxJhtgNcEKoVZOQ2PCWJQCaGaRx7HDc9eYfM9CuPhF4wRVJVNoP5/tVSjebxBggRMmPUsczrZnYGHl8GmhNeOMmecKiHVSE/vdKtFrH7HXu6x4eRw3n5mYqtylqbU7fHFVO8gI3BWb2JF9xBGShALZ7jKeaf6JaZt5qhh1mrXhB2y8wOBfHSP/YTUgYZPsiFOT9PpQ1wz6avBAbwAcrsMh1YmmUhSP+aSDSBj3ZDXW1NDhyaw9xZpTN9Hq8nNurTII0O1MsghkD9Ngu X-Bogosity: Ham, tests=bogofilter, spamicity=0.000000, version=1.2.4 Sender: owner-linux-mm@kvack.org Precedence: bulk X-Loop: owner-majordomo@kvack.org List-ID: List-Subscribe: List-Unsubscribe: Use the new pfnmap API to allow huge MMIO mappings for VMs. The rest work is done perfectly on the other side (host_pfn_mapping_level()). Cc: Paolo Bonzini Cc: Sean Christopherson Signed-off-by: Peter Xu --- virt/kvm/kvm_main.c | 19 +++++++------------ 1 file changed, 7 insertions(+), 12 deletions(-) diff --git a/virt/kvm/kvm_main.c b/virt/kvm/kvm_main.c index cb2b78e92910..f416d5e3f9c0 100644 --- a/virt/kvm/kvm_main.c +++ b/virt/kvm/kvm_main.c @@ -2860,13 +2860,11 @@ static int hva_to_pfn_remapped(struct vm_area_struct *vma, unsigned long addr, bool write_fault, bool *writable, kvm_pfn_t *p_pfn) { + struct follow_pfnmap_args args = { .vma = vma, .address = addr }; kvm_pfn_t pfn; - pte_t *ptep; - pte_t pte; - spinlock_t *ptl; int r; - r = follow_pte(vma, addr, &ptep, &ptl); + r = follow_pfnmap_start(&args); if (r) { /* * get_user_pages fails for VM_IO and VM_PFNMAP vmas and does @@ -2881,21 +2879,19 @@ static int hva_to_pfn_remapped(struct vm_area_struct *vma, if (r) return r; - r = follow_pte(vma, addr, &ptep, &ptl); + r = follow_pfnmap_start(&args); if (r) return r; } - pte = ptep_get(ptep); - - if (write_fault && !pte_write(pte)) { + if (write_fault && !args.writable) { pfn = KVM_PFN_ERR_RO_FAULT; goto out; } if (writable) - *writable = pte_write(pte); - pfn = pte_pfn(pte); + *writable = args.writable; + pfn = args.pfn; /* * Get a reference here because callers of *hva_to_pfn* and @@ -2916,9 +2912,8 @@ static int hva_to_pfn_remapped(struct vm_area_struct *vma, */ if (!kvm_try_get_pfn(pfn)) r = -EFAULT; - out: - pte_unmap_unlock(ptep, ptl); + follow_pfnmap_end(&args); *p_pfn = pfn; return r; From patchwork Mon Aug 26 20:43:45 2024 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Peter Xu X-Patchwork-Id: 13778405 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from kanga.kvack.org (kanga.kvack.org [205.233.56.17]) by smtp.lore.kernel.org (Postfix) with ESMTP id 7D5CAC5321D for ; Mon, 26 Aug 2024 20:44:34 +0000 (UTC) Received: by kanga.kvack.org (Postfix) id 809D86B00A0; Mon, 26 Aug 2024 16:44:24 -0400 (EDT) Received: by kanga.kvack.org (Postfix, from userid 40) id 7BB9D6B00A1; Mon, 26 Aug 2024 16:44:24 -0400 (EDT) X-Delivered-To: int-list-linux-mm@kvack.org Received: by kanga.kvack.org (Postfix, from userid 63042) id 5BE996B00A2; Mon, 26 Aug 2024 16:44:24 -0400 (EDT) X-Delivered-To: linux-mm@kvack.org Received: from relay.hostedemail.com (smtprelay0010.hostedemail.com [216.40.44.10]) by kanga.kvack.org (Postfix) with ESMTP id 3E0076B00A0 for ; Mon, 26 Aug 2024 16:44:24 -0400 (EDT) Received: from smtpin03.hostedemail.com (a10.router.float.18 [10.200.18.1]) by unirelay05.hostedemail.com (Postfix) with ESMTP id 04405413CB for ; Mon, 26 Aug 2024 20:44:23 +0000 (UTC) X-FDA: 82495574448.03.CF9705C Received: from us-smtp-delivery-124.mimecast.com (us-smtp-delivery-124.mimecast.com [170.10.129.124]) by imf21.hostedemail.com (Postfix) with ESMTP id E2CCA1C0010 for ; Mon, 26 Aug 2024 20:44:21 +0000 (UTC) Authentication-Results: imf21.hostedemail.com; dkim=pass header.d=redhat.com header.s=mimecast20190719 header.b=J8eqF3l5; spf=pass (imf21.hostedemail.com: domain of peterx@redhat.com designates 170.10.129.124 as permitted sender) smtp.mailfrom=peterx@redhat.com; dmarc=pass (policy=none) header.from=redhat.com ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=hostedemail.com; s=arc-20220608; t=1724704977; h=from:from:sender:reply-to:subject:subject:date:date: message-id:message-id:to:to:cc:cc:mime-version:mime-version: content-type:content-type: content-transfer-encoding:content-transfer-encoding: in-reply-to:in-reply-to:references:references:dkim-signature; bh=kaY3EfeA+k09YWgS+MfGM8qkjlv58srLyvVau/9jXnU=; b=kdtWdaIbZ20IQKqFWJmr1XtWhvr99W64NEq5SN8fJDDUsJZM4w5I7vugHH4aXgf2T+WNDv jLMJl9qYQqXI+a8oGtCsB1MUSYzEn0Ub2x9PsgILok/KsmMs2QRb+5sYqe8xQzKoBYhjEu G5bZcYwvX5Wk4r2vB7tm4RZv6Thg5ZQ= ARC-Seal: i=1; s=arc-20220608; d=hostedemail.com; t=1724704977; a=rsa-sha256; cv=none; b=mUIMhYEFA36TcFCXs3X+ooGm7MensiwhGTdNAEqtDSvmz/Ud8YSp2DSIjcsVx8+Dc9k0Zy LmkxTE7vE/9H1t3k+u7NAMYIagMxWJSZVDB9hcjK9OaECVdTOBM30RePlEKDT4SOJQXMX6 xOKljABX1C4Rb13KYuL4XliQ896eX60= ARC-Authentication-Results: i=1; imf21.hostedemail.com; dkim=pass header.d=redhat.com header.s=mimecast20190719 header.b=J8eqF3l5; spf=pass (imf21.hostedemail.com: domain of peterx@redhat.com designates 170.10.129.124 as permitted sender) smtp.mailfrom=peterx@redhat.com; dmarc=pass (policy=none) header.from=redhat.com DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=redhat.com; s=mimecast20190719; t=1724705061; h=from:from:reply-to:subject:subject:date:date:message-id:message-id: to:to:cc:cc:mime-version:mime-version:content-type:content-type: content-transfer-encoding:content-transfer-encoding: in-reply-to:in-reply-to:references:references; bh=kaY3EfeA+k09YWgS+MfGM8qkjlv58srLyvVau/9jXnU=; b=J8eqF3l5TjJLjwTcZd/Ijl+hgiFskRSTSPxTeafDfoRjyRPTXsvUVxl+mkbsgRCnwTAR57 mOLdcYe3xHhChX/b6Jwv/wU8KEmwnmv1zrAuMEVG06m24BYvoBMZaNOc/dZfbuJ6UBX1Sb YBXUUqstBndX9WmK5YTpOC7tkbWfqXA= Received: from mail-oa1-f70.google.com (mail-oa1-f70.google.com [209.85.160.70]) by relay.mimecast.com with ESMTP with STARTTLS (version=TLSv1.3, cipher=TLS_AES_256_GCM_SHA384) id us-mta-633-z1SjHFiYPkGTUQUinnhufQ-1; Mon, 26 Aug 2024 16:44:20 -0400 X-MC-Unique: z1SjHFiYPkGTUQUinnhufQ-1 Received: by mail-oa1-f70.google.com with SMTP id 586e51a60fabf-26823e7e16eso10890276fac.3 for ; Mon, 26 Aug 2024 13:44:19 -0700 (PDT) X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20230601; t=1724705059; x=1725309859; h=content-transfer-encoding:mime-version:references:in-reply-to :message-id:date:subject:cc:to:from:x-gm-message-state:from:to:cc :subject:date:message-id:reply-to; bh=kaY3EfeA+k09YWgS+MfGM8qkjlv58srLyvVau/9jXnU=; b=Qn4SFV+ilkm2VGftTiUECIiaNS8FXmjjbWNf/ldqJxemb9+xmUROWn6b3yAjf+96yX txqbRMtSbE5xszCqjXWwdt+C5X6OAy6W729ujyE+k7jDeYSqsCAs0qL2GWWWoqFouTF5 cFSpfQdZuYPNuzwEHx+ctMn/NI2QDc3Pxvl+JPTVxeWlPVeXGTs3aDeTyc5eQy3oZSGq cu/Mahya2joBENtQD04TS8pYV+iUXu4k61Cctko4XPJuR1sTMfVTbd2SVpnvHvv090MI Gy76RS/VEtStLpxAiReHzgPcius6JV23GaK2BLN3TMmsYhWHd65DuxEToe3kZowyApVa 9uZA== X-Forwarded-Encrypted: i=1; AJvYcCVRIoADMcrisT+MuanTwYwhIjevE68Wqo0rcUXGEjZ8a8UvWL/aJ3I7MzZ6gE3ArJRhnShj54gnEg==@kvack.org X-Gm-Message-State: AOJu0YyM2ZY/NOcE+DGgxtV6y0Zfl6fhpn+fCoMXp+X3Ds/47uiAjCI1 Uh9NEZvs8ITT4Dim99ZA1OmLucbRWUwCOl+dNGlKTKPp3mK/SX4YrH3tJNzheBmGF6xyZOTT0aB QvwPNdeevNVpYAWtV/I9oXNw59EnyJp+zFjgqJhsKgvzHYoNtVeGXZ9+Z X-Received: by 2002:a05:6358:7201:b0:1ac:f3df:3bde with SMTP id e5c5f4694b2df-1b5ebf3abdcmr117740055d.5.1724705058981; Mon, 26 Aug 2024 13:44:18 -0700 (PDT) X-Google-Smtp-Source: AGHT+IHztBftH8jfc2wrFWJQjc0BC5uJ0IwQWlV9Rpv0YGeitSfcFoHRh3KGrU0k5GaslF87BwNh3Q== X-Received: by 2002:a05:6358:7201:b0:1ac:f3df:3bde with SMTP id e5c5f4694b2df-1b5ebf3abdcmr117734755d.5.1724705058531; Mon, 26 Aug 2024 13:44:18 -0700 (PDT) Received: from x1n.redhat.com (pool-99-254-121-117.cpe.net.cable.rogers.com. [99.254.121.117]) by smtp.gmail.com with ESMTPSA id af79cd13be357-7a67f3fd6c1sm491055185a.121.2024.08.26.13.44.16 (version=TLS1_3 cipher=TLS_AES_256_GCM_SHA384 bits=256/256); Mon, 26 Aug 2024 13:44:18 -0700 (PDT) From: Peter Xu To: linux-kernel@vger.kernel.org, linux-mm@kvack.org Cc: Gavin Shan , Catalin Marinas , x86@kernel.org, Ingo Molnar , Andrew Morton , Paolo Bonzini , Dave Hansen , Thomas Gleixner , Alistair Popple , kvm@vger.kernel.org, linux-arm-kernel@lists.infradead.org, Sean Christopherson , peterx@redhat.com, Oscar Salvador , Jason Gunthorpe , Borislav Petkov , Zi Yan , Axel Rasmussen , David Hildenbrand , Yan Zhao , Will Deacon , Kefeng Wang , Alex Williamson , Niklas Schnelle , Gerald Schaefer , Heiko Carstens , Vasily Gorbik , Alexander Gordeev , Christian Borntraeger , Sven Schnelle , linux-s390@vger.kernel.org Subject: [PATCH v2 11/19] s390/pci_mmio: Use follow_pfnmap API Date: Mon, 26 Aug 2024 16:43:45 -0400 Message-ID: <20240826204353.2228736-12-peterx@redhat.com> X-Mailer: git-send-email 2.45.0 In-Reply-To: <20240826204353.2228736-1-peterx@redhat.com> References: <20240826204353.2228736-1-peterx@redhat.com> MIME-Version: 1.0 X-Mimecast-Spam-Score: 0 X-Mimecast-Originator: redhat.com X-Rspamd-Queue-Id: E2CCA1C0010 X-Stat-Signature: 3ith431i1cngwiuecnwk9j8j9k4eaqfa X-Rspamd-Server: rspam09 X-Rspam-User: X-HE-Tag: 1724705061-998465 X-HE-Meta: U2FsdGVkX18qr14F77ogpSIsHPpYGWztnf/mySsD8rMixQucY5btDmxa/BKEJM0I6hhirTizNoxdcno1Hch67wCNLrfocudm8W+RmX5mjDhsdDl3qZD+/NWdGC0DDszaN5kmnABT//pzzt7dFzTWm8dY9YeAXXQQqHeeYj8/Qr1zZHdIQ40beQ9j/GldpUgNeV8FEK6zbVJzRU2A6F5Zm4NYS1XdKIo2/wvnhx9vnrySunJYqTMTtwpRA2D2fMDpqZ4euXNMWsxtKB3ZWy2I6D8aVWDyqQ66btm+IJyWwh3c5MFFCx3222Ymit+D0GkiWTk+q5qX4+arPliHI7YiqZ/MJtIP1/gUkkSq0iIjALMjYmVVWBjYvy+1n4kp2Qx2LeJZRm7gf7vbr6SqaTep3oiVeLeT+r0suPWsunFC2mp93YdbD1oGqhR0GOejX3xesXrzJ0suo97T4VwaJW5eyHzmUsefMTlJagPxfb2V9/5QWEgHuWEqvbfxRxxsj3w4TQGbHN8Y9auSsLBo/O2o8y8EDgh2vegoaR67IlW9H7gfAaxryOzatXWdildZbnIAEGcl9TYN4pP4UH5D3xrv0gEvF6iKoxjcXjRrg2HCv8ulbKpOurNJkPI1N9Eh6SrbZUtJPYsgRuZKo4e9uGHia2yw1QZZ6oVct+fMhi94OopOPkLy5BqJHmHiklXAoIprJ9y0eLGzIOhnMbsJPb+o5Y6zJD3h+k+HGir2hxOrJpVOCslDlhIC3iD8/orMUCW6Ngo5GO/oP8VZKsjCwc3A67RqUHS+v0fDZP/Ts6TuxkeatdYWJZiZwpcN86y5N4prdnQcBvjZEaHD5BVAEL7WEcN59Vsx/XB5+m3sO3YbT1HBQAKHYsz8HHRatSr4Emwh2Ad1nMAXgyEvUxGf6ERACHBOc6SLMT3hVHl5AgbTebF01K0vV++eK4RjhS7T79rG7E3EnnzCrvltpwoleiy jTHQOeFW Y3UEYgKAhFKPHeL0wRljzPpHhe0uNTBU+wYYdpV8BCRF1LejMtJkag28mlUE77f4sKA6KCg84nudGJHgWXzrDti3MdybMW6h/4fGW/YyQk43lI2Rl4/rwwgN5CnBSHbQgG/caIS6XCylWzwffRBbBcZuQUE6Y5rVpCPY5p+MVwISQGms7/6Qku1Az6Mwvzx+V8aJ2Sd27o78qzYsJgOfOglnVwLyxqBdhMqeG2huKYrOAMmlpGjnrx9MjwWM4H6SoB/g+BV45HthDXe16etQhnVO+AedPObdT8oGBWxCiZmLJ3WisESGSMkbegFM39p3ILytamDYuaYtp7yQZAg5kEkDq6JiQPmFLy7dvJP4BANpBHw6xKn7ytq9FAQn1HeuItiqUiAbRZOrbFOMSYhNmSALNnbY+0FscQvlXXrddifBEKWp+E3kPfsJxcUXixg9XrJp5qii3/miLMZ+ddFZMKk4/bA== X-Bogosity: Ham, tests=bogofilter, spamicity=0.000000, version=1.2.4 Sender: owner-linux-mm@kvack.org Precedence: bulk X-Loop: owner-majordomo@kvack.org List-ID: List-Subscribe: List-Unsubscribe: Use the new API that can understand huge pfn mappings. Cc: Niklas Schnelle Cc: Gerald Schaefer Cc: Heiko Carstens Cc: Vasily Gorbik Cc: Alexander Gordeev Cc: Christian Borntraeger Cc: Sven Schnelle Cc: linux-s390@vger.kernel.org Signed-off-by: Peter Xu --- arch/s390/pci/pci_mmio.c | 22 ++++++++++++---------- 1 file changed, 12 insertions(+), 10 deletions(-) diff --git a/arch/s390/pci/pci_mmio.c b/arch/s390/pci/pci_mmio.c index 5398729bfe1b..de5c0b389a3e 100644 --- a/arch/s390/pci/pci_mmio.c +++ b/arch/s390/pci/pci_mmio.c @@ -118,12 +118,11 @@ static inline int __memcpy_toio_inuser(void __iomem *dst, SYSCALL_DEFINE3(s390_pci_mmio_write, unsigned long, mmio_addr, const void __user *, user_buffer, size_t, length) { + struct follow_pfnmap_args args = { }; u8 local_buf[64]; void __iomem *io_addr; void *buf; struct vm_area_struct *vma; - pte_t *ptep; - spinlock_t *ptl; long ret; if (!zpci_is_enabled()) @@ -169,11 +168,13 @@ SYSCALL_DEFINE3(s390_pci_mmio_write, unsigned long, mmio_addr, if (!(vma->vm_flags & VM_WRITE)) goto out_unlock_mmap; - ret = follow_pte(vma, mmio_addr, &ptep, &ptl); + args.address = mmio_addr; + args.vma = vma; + ret = follow_pfnmap_start(&args); if (ret) goto out_unlock_mmap; - io_addr = (void __iomem *)((pte_pfn(*ptep) << PAGE_SHIFT) | + io_addr = (void __iomem *)((args.pfn << PAGE_SHIFT) | (mmio_addr & ~PAGE_MASK)); if ((unsigned long) io_addr < ZPCI_IOMAP_ADDR_BASE) @@ -181,7 +182,7 @@ SYSCALL_DEFINE3(s390_pci_mmio_write, unsigned long, mmio_addr, ret = zpci_memcpy_toio(io_addr, buf, length); out_unlock_pt: - pte_unmap_unlock(ptep, ptl); + follow_pfnmap_end(&args); out_unlock_mmap: mmap_read_unlock(current->mm); out_free: @@ -260,12 +261,11 @@ static inline int __memcpy_fromio_inuser(void __user *dst, SYSCALL_DEFINE3(s390_pci_mmio_read, unsigned long, mmio_addr, void __user *, user_buffer, size_t, length) { + struct follow_pfnmap_args args = { }; u8 local_buf[64]; void __iomem *io_addr; void *buf; struct vm_area_struct *vma; - pte_t *ptep; - spinlock_t *ptl; long ret; if (!zpci_is_enabled()) @@ -308,11 +308,13 @@ SYSCALL_DEFINE3(s390_pci_mmio_read, unsigned long, mmio_addr, if (!(vma->vm_flags & VM_WRITE)) goto out_unlock_mmap; - ret = follow_pte(vma, mmio_addr, &ptep, &ptl); + args.vma = vma; + args.address = mmio_addr; + ret = follow_pfnmap_start(&args); if (ret) goto out_unlock_mmap; - io_addr = (void __iomem *)((pte_pfn(*ptep) << PAGE_SHIFT) | + io_addr = (void __iomem *)((args.pfn << PAGE_SHIFT) | (mmio_addr & ~PAGE_MASK)); if ((unsigned long) io_addr < ZPCI_IOMAP_ADDR_BASE) { @@ -322,7 +324,7 @@ SYSCALL_DEFINE3(s390_pci_mmio_read, unsigned long, mmio_addr, ret = zpci_memcpy_fromio(buf, io_addr, length); out_unlock_pt: - pte_unmap_unlock(ptep, ptl); + follow_pfnmap_end(&args); out_unlock_mmap: mmap_read_unlock(current->mm); From patchwork Mon Aug 26 20:43:46 2024 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Peter Xu X-Patchwork-Id: 13778406 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from kanga.kvack.org (kanga.kvack.org [205.233.56.17]) by smtp.lore.kernel.org (Postfix) with ESMTP id 27216C5472E for ; Mon, 26 Aug 2024 20:44:37 +0000 (UTC) Received: by kanga.kvack.org (Postfix) id AA5396B00A2; Mon, 26 Aug 2024 16:44:25 -0400 (EDT) Received: by kanga.kvack.org (Postfix, from userid 40) id 9DE346B00A3; Mon, 26 Aug 2024 16:44:25 -0400 (EDT) X-Delivered-To: int-list-linux-mm@kvack.org Received: by kanga.kvack.org (Postfix, from userid 63042) id 830026B00A4; Mon, 26 Aug 2024 16:44:25 -0400 (EDT) X-Delivered-To: linux-mm@kvack.org Received: from relay.hostedemail.com (smtprelay0015.hostedemail.com [216.40.44.15]) by kanga.kvack.org (Postfix) with ESMTP id 63BAE6B00A2 for ; Mon, 26 Aug 2024 16:44:25 -0400 (EDT) Received: from smtpin27.hostedemail.com (a10.router.float.18 [10.200.18.1]) by unirelay01.hostedemail.com (Postfix) with ESMTP id 2C2C01C5063 for ; Mon, 26 Aug 2024 20:44:25 +0000 (UTC) X-FDA: 82495574490.27.8847D14 Received: from us-smtp-delivery-124.mimecast.com (us-smtp-delivery-124.mimecast.com [170.10.129.124]) by imf11.hostedemail.com (Postfix) with ESMTP id 18F184001E for ; Mon, 26 Aug 2024 20:44:22 +0000 (UTC) Authentication-Results: imf11.hostedemail.com; dkim=pass header.d=redhat.com header.s=mimecast20190719 header.b=U3Hm2rYD; spf=pass (imf11.hostedemail.com: domain of peterx@redhat.com designates 170.10.129.124 as permitted sender) smtp.mailfrom=peterx@redhat.com; dmarc=pass (policy=none) header.from=redhat.com ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=hostedemail.com; s=arc-20220608; t=1724704976; h=from:from:sender:reply-to:subject:subject:date:date: message-id:message-id:to:to:cc:cc:mime-version:mime-version: content-type:content-type: content-transfer-encoding:content-transfer-encoding: in-reply-to:in-reply-to:references:references:dkim-signature; bh=moprxbUn5M7+L+63SZwwyqeq4iL0OHeeIJZbsfnkOho=; b=jKOUtB676QmPGwuQnBANOow534L0GStwa73a9J1u9INrz5+jCF11Lj2U7J/YhTRmtnTWxK r1DEFoVPsTdQ1WV38cDyM4H3FcgcZb5f5W08d3p+55Dfs6D/tPuNE8kVhuuG65VDtQlhHl 7Ehoh5yvxrYFbnt4cU+pXuP++EvMGwU= ARC-Seal: i=1; s=arc-20220608; d=hostedemail.com; t=1724704976; a=rsa-sha256; cv=none; b=m2UEfHBYax/3wfbkKLRCUg5BFucqmQcRxvz4yzUPBMJyFm5GTmT7a7pP+a9OsIb+GtXsLP IULeLmFpMK7sgE3ZeSbU2LEJu3SOGcRn62aqdb1LtEyK75YaIUufGvZfeFZwjjMVXj4xrH +lwNeelnPi4Z7ZuQRNawIo6DLp35S0A= ARC-Authentication-Results: i=1; imf11.hostedemail.com; dkim=pass header.d=redhat.com header.s=mimecast20190719 header.b=U3Hm2rYD; spf=pass (imf11.hostedemail.com: domain of peterx@redhat.com designates 170.10.129.124 as permitted sender) smtp.mailfrom=peterx@redhat.com; dmarc=pass (policy=none) header.from=redhat.com DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=redhat.com; s=mimecast20190719; t=1724705062; h=from:from:reply-to:subject:subject:date:date:message-id:message-id: to:to:cc:cc:mime-version:mime-version:content-type:content-type: content-transfer-encoding:content-transfer-encoding: in-reply-to:in-reply-to:references:references; bh=moprxbUn5M7+L+63SZwwyqeq4iL0OHeeIJZbsfnkOho=; b=U3Hm2rYD+TDFjedSX/ywSr3TBwLzavI/njXNAx2PRr3kBvrgbXYEsimWA9V8GdFgkvgTqI QHFkl3NbnN/CADQ7NIKFB44/ot9+5/3OCazsr0aRTnROLZGe27rIkiOclJhDj6D96tU5e8 e6H2RLrc9sAJ0WTxvRViRz8JODo/Ed4= Received: from mail-qk1-f198.google.com (mail-qk1-f198.google.com [209.85.222.198]) by relay.mimecast.com with ESMTP with STARTTLS (version=TLSv1.3, cipher=TLS_AES_256_GCM_SHA384) id us-mta-597-PfjDT15aMICk5SKflzJl0g-1; Mon, 26 Aug 2024 16:44:21 -0400 X-MC-Unique: PfjDT15aMICk5SKflzJl0g-1 Received: by mail-qk1-f198.google.com with SMTP id af79cd13be357-7a1d4335cceso884881285a.1 for ; Mon, 26 Aug 2024 13:44:21 -0700 (PDT) X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20230601; t=1724705061; x=1725309861; h=content-transfer-encoding:mime-version:references:in-reply-to :message-id:date:subject:cc:to:from:x-gm-message-state:from:to:cc :subject:date:message-id:reply-to; bh=moprxbUn5M7+L+63SZwwyqeq4iL0OHeeIJZbsfnkOho=; b=MAj5iEtT3iTFw1ROSX/i4IG90VkfshCtre5HJZYXHY9HOoHHi/KOTiis2jQnk1CqqG dZE7vx/wDSmDcL2e73lVfJPm/G7hmrBp05XmitZwHEL+hzTv6xut8zUS572SYFnyfE5c 1dCUd2TcPTDEdcyOxP7Mj0OKFdgrCK7o+gJhU9k0VtNVZme+g2W8U46WbzaLIahMN3Co 01WTsRlCiTuMBpVKcfm742xQtzuQFUIGW3BeSv+3DW7R/7PoCtNEfgmkQZoUy2n1u1tO cdMxgEfCs+AnVF+ODfZxFFADmZJDT0/aVVLhZXh6cQtx5IlSDf1khjZsGH7qGaZDMAGf iGpQ== X-Forwarded-Encrypted: i=1; AJvYcCWmsHmWu2hjbDJ1udz72lmX3x4K6hfq14k4ienRUeSjW5ZZkSsxjesw2DTTA35tNNE0gv0JDJBBjw==@kvack.org X-Gm-Message-State: AOJu0YzBdr1KwoEOEEYvdF93lQkPXjuGnVy7qZUyb2OsAvLyFhZMDSiW LeUVN3zq7RcTAO7M2l2mnsVx6DSUvfhvDqQiXzFeCKnK6LirdKTZjOVJ1iQXcJFKcMS1lVr5011 dxdeSk91hJ+RzD3s4RQRnOlS2sLNpzCwdviN2/Uvt8hTI3BI5 X-Received: by 2002:a05:620a:1a92:b0:7a1:e341:d543 with SMTP id af79cd13be357-7a7e4cf3a3fmr158797585a.28.1724705060674; Mon, 26 Aug 2024 13:44:20 -0700 (PDT) X-Google-Smtp-Source: AGHT+IF8L+Naz3FkxDynSQ3yMse2qHglqFkf4LqBPi/5cD3sQEuFfYs6yWPMbT2Qk9XtpK98A+7trg== X-Received: by 2002:a05:620a:1a92:b0:7a1:e341:d543 with SMTP id af79cd13be357-7a7e4cf3a3fmr158793685a.28.1724705060390; Mon, 26 Aug 2024 13:44:20 -0700 (PDT) Received: from x1n.redhat.com (pool-99-254-121-117.cpe.net.cable.rogers.com. [99.254.121.117]) by smtp.gmail.com with ESMTPSA id af79cd13be357-7a67f3fd6c1sm491055185a.121.2024.08.26.13.44.18 (version=TLS1_3 cipher=TLS_AES_256_GCM_SHA384 bits=256/256); Mon, 26 Aug 2024 13:44:19 -0700 (PDT) From: Peter Xu To: linux-kernel@vger.kernel.org, linux-mm@kvack.org Cc: Gavin Shan , Catalin Marinas , x86@kernel.org, Ingo Molnar , Andrew Morton , Paolo Bonzini , Dave Hansen , Thomas Gleixner , Alistair Popple , kvm@vger.kernel.org, linux-arm-kernel@lists.infradead.org, Sean Christopherson , peterx@redhat.com, Oscar Salvador , Jason Gunthorpe , Borislav Petkov , Zi Yan , Axel Rasmussen , David Hildenbrand , Yan Zhao , Will Deacon , Kefeng Wang , Alex Williamson Subject: [PATCH v2 12/19] mm/x86/pat: Use the new follow_pfnmap API Date: Mon, 26 Aug 2024 16:43:46 -0400 Message-ID: <20240826204353.2228736-13-peterx@redhat.com> X-Mailer: git-send-email 2.45.0 In-Reply-To: <20240826204353.2228736-1-peterx@redhat.com> References: <20240826204353.2228736-1-peterx@redhat.com> MIME-Version: 1.0 X-Mimecast-Spam-Score: 0 X-Mimecast-Originator: redhat.com X-Stat-Signature: ehhem9k5d9h5t9yzo63i8b37nrsamimz X-Rspamd-Queue-Id: 18F184001E X-Rspam-User: X-Rspamd-Server: rspam08 X-HE-Tag: 1724705062-887955 X-HE-Meta: U2FsdGVkX1+c0eoaRGKfPKOeV75IY2er5X0tx79cQnUiNeUlB7D4EYa9v3Mx+PSWK1CUaII4X0dCimT4nQ7Tj8kwApseEXM3/iUwv3lmeNjJfQ9snynRD7HcvT6ieEPamrP42c0mcDJAiB09g1n3ygWIHIdhu+GKcJsrXTKKbVkqNJfQg43/8R7Y4Z07daki2VrGwClQYadJ3W1MQbv2tR1Y1ZXfe1d7h8uUmoid57tAEKW16ibab6fFnLoIPoQrmxQtiSjwiTFWCnSemO+yrUQY5HEhtfl7tkfg79oJzyMA+On0S5zZIWp1NYfTg+BecBKo3qNxnaArA33cWt3tIL/I8fJiJucWN2gyz8AL53bRlaPBAwQ2f+GL9Z8fcAG+dtA1d5kMkkXz9TsklC6tPAHwejhfAyhN19d6Vu8s4p85XtYnFVLmKlDjrvFhrhLetWEhS69myR6pNMKsniCNH9J2UjDcrtdEYm3B4+4CeqfupaSvrZJFN2qz4QLTEXb/4xxHX5Ltv9dMb6/tbiVD72L4kPNrhw88X7Co1hmlCyiC+2aqspazdJvLY6WF7mU2cIS0dGmIEoz9Z4AcZ3mFqdnBi/WEIX0gLoYbPznGqicFpUrkbnPlipPHJuyfDuKH2NfHAOySy02mAWrkygTqZa4INCqMlSgzsWP8C8SKDbL/QL2GnTPlTt626rUlmEeHaxQnJrR0zdI7vcaHWDTRH0y/27FoaXekxs6h84whNtjTdK53Q5lxdSVOLPmQCBTtoIHSED7Or6HOzEImHWY7nBuO10MwaVuYjySaToHsyUt6DFbzk5zyUYNeh0ra7p9xs43c4vK1CoDU5neQJhNUqLJOTe7AS/5BMS7v1ZSQdoyv5KgmKAQkM3OrVJXuqSJfwcA5+91p7cXDh9IL1qOMwgZkw3lnLeBaufyHMx2ObJybKlT8jkB5cmiNHmImSpEJpKWShEm99Wb6zx4TVBO +vmNek00 YRIzxwxKjrjQNV1ZlAPfqfiaMKJ5skrKs3wQQ10qR9M+grRbeebVxNFeRdRHmfFLX/stbbD3+H3c51i7rSLaaImd6hAR33X2YqnS/9jS9oKz6x9D6J7JS04wE201Z4WhQPm1qLL5GSppnzp/zBxQ/HaYFpjT/WMYsY00v3YJ+zKnksOrgTcq5OAJfn3tAdTdmINcg8zrMuDItR+949RIjk3HuhwrKKGPXlT+/e0huTHUPGMcpqLWA6YkbKx4XscuzJwIkxiro3Yal6N38xN2Nv6HX9ByH5+aVi96hld4Ergx+q960pRXWT86AYWNhzyE4O6yTwK9Uy078FLG34+U6xWD+EaOel7avbo6eU4g8AbODxrLIiBqPDyLbx98rWrvbu+tSuPlcC33hk5RUV7Y4EDyLa7cdjPKUM5jtr8e0d8P24VRZJ+JupHAbM24JfuJ/ckAjh4io7g01BqFGUaLsaoigbPPWcdMkHmnI3o/lYV/Fa5MJub4jbFB7zkyBXRoMDELn6wesmGumT9vD02an41b6tHhZ97eQyWEjdKTNkajYSG8ecEQzOONGLMmEQ308GLptiiY60PlA4oz0yFHx9HkptA== X-Bogosity: Ham, tests=bogofilter, spamicity=0.000000, version=1.2.4 Sender: owner-linux-mm@kvack.org Precedence: bulk X-Loop: owner-majordomo@kvack.org List-ID: List-Subscribe: List-Unsubscribe: Use the new API that can understand huge pfn mappings. Cc: x86@kernel.org Cc: Thomas Gleixner Cc: Ingo Molnar Cc: Borislav Petkov Cc: Dave Hansen Signed-off-by: Peter Xu --- arch/x86/mm/pat/memtype.c | 17 +++++++---------- 1 file changed, 7 insertions(+), 10 deletions(-) diff --git a/arch/x86/mm/pat/memtype.c b/arch/x86/mm/pat/memtype.c index 1fa0bf6ed295..f73b5ce270b3 100644 --- a/arch/x86/mm/pat/memtype.c +++ b/arch/x86/mm/pat/memtype.c @@ -951,23 +951,20 @@ static void free_pfn_range(u64 paddr, unsigned long size) static int follow_phys(struct vm_area_struct *vma, unsigned long *prot, resource_size_t *phys) { - pte_t *ptep, pte; - spinlock_t *ptl; + struct follow_pfnmap_args args = { .vma = vma, .address = vma->vm_start }; - if (follow_pte(vma, vma->vm_start, &ptep, &ptl)) + if (follow_pfnmap_start(&args)) return -EINVAL; - pte = ptep_get(ptep); - /* Never return PFNs of anon folios in COW mappings. */ - if (vm_normal_folio(vma, vma->vm_start, pte)) { - pte_unmap_unlock(ptep, ptl); + if (!args.special) { + follow_pfnmap_end(&args); return -EINVAL; } - *prot = pgprot_val(pte_pgprot(pte)); - *phys = (resource_size_t)pte_pfn(pte) << PAGE_SHIFT; - pte_unmap_unlock(ptep, ptl); + *prot = pgprot_val(args.pgprot); + *phys = (resource_size_t)args.pfn << PAGE_SHIFT; + follow_pfnmap_end(&args); return 0; } From patchwork Mon Aug 26 20:43:47 2024 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Peter Xu X-Patchwork-Id: 13778407 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from kanga.kvack.org (kanga.kvack.org [205.233.56.17]) by smtp.lore.kernel.org (Postfix) with ESMTP id E1E95C5472C for ; Mon, 26 Aug 2024 20:44:39 +0000 (UTC) Received: by kanga.kvack.org (Postfix) id C7D596B00A4; Mon, 26 Aug 2024 16:44:27 -0400 (EDT) Received: by kanga.kvack.org (Postfix, from userid 40) id BB1946B00A5; Mon, 26 Aug 2024 16:44:27 -0400 (EDT) X-Delivered-To: int-list-linux-mm@kvack.org Received: by kanga.kvack.org (Postfix, from userid 63042) id 9B94F6B00A6; Mon, 26 Aug 2024 16:44:27 -0400 (EDT) X-Delivered-To: linux-mm@kvack.org Received: from relay.hostedemail.com (smtprelay0014.hostedemail.com [216.40.44.14]) by kanga.kvack.org (Postfix) with ESMTP id 797A56B00A4 for ; Mon, 26 Aug 2024 16:44:27 -0400 (EDT) Received: from smtpin28.hostedemail.com (a10.router.float.18 [10.200.18.1]) by unirelay07.hostedemail.com (Postfix) with ESMTP id 31C06161384 for ; Mon, 26 Aug 2024 20:44:27 +0000 (UTC) X-FDA: 82495574574.28.86C0DFA Received: from us-smtp-delivery-124.mimecast.com (us-smtp-delivery-124.mimecast.com [170.10.133.124]) by imf11.hostedemail.com (Postfix) with ESMTP id 266964002A for ; Mon, 26 Aug 2024 20:44:25 +0000 (UTC) Authentication-Results: imf11.hostedemail.com; dkim=pass header.d=redhat.com header.s=mimecast20190719 header.b=hDMkfdUF; dmarc=pass (policy=none) header.from=redhat.com; spf=pass (imf11.hostedemail.com: domain of peterx@redhat.com designates 170.10.133.124 as permitted sender) smtp.mailfrom=peterx@redhat.com ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=hostedemail.com; s=arc-20220608; t=1724704969; h=from:from:sender:reply-to:subject:subject:date:date: message-id:message-id:to:to:cc:cc:mime-version:mime-version: content-type:content-type: content-transfer-encoding:content-transfer-encoding: in-reply-to:in-reply-to:references:references:dkim-signature; bh=HLBqit68L/U2mdwYaFkydFK3txHa+LkmL8n2YTQXnnw=; b=AZc14GZICIprQ36xb936tsYZT1EiNO2Jaz/VCEQvp3A+qFG1FWd6K9J2CqgCD8h2YCYnVx vS2E18qR4GCndWW9rKYShUwWhV+w2CK7/fYc7W2nB5shtzLC2WTle+9Ue6Ah+e96CFsKmb duQP+m8yVdqKxpk9C+zMZA/azrANwdk= ARC-Seal: i=1; s=arc-20220608; d=hostedemail.com; t=1724704969; a=rsa-sha256; cv=none; b=kKyP1q+eESIR88EOKXuCm56BObAKws8d20oBTBkyTHR6HiDS0Iiuq08xDC5tRCwl+osZqh 7ydx10mIwagZqrv3xoYQzsx7czFXrQblA/wehf5HnC1wMG/lW6L4ZtfBmm5zTohTC47b4K r+KUJQpnUdN6L5cQ8xF5D/QY8l0/u0k= ARC-Authentication-Results: i=1; imf11.hostedemail.com; dkim=pass header.d=redhat.com header.s=mimecast20190719 header.b=hDMkfdUF; dmarc=pass (policy=none) header.from=redhat.com; spf=pass (imf11.hostedemail.com: domain of peterx@redhat.com designates 170.10.133.124 as permitted sender) smtp.mailfrom=peterx@redhat.com DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=redhat.com; s=mimecast20190719; t=1724705064; h=from:from:reply-to:subject:subject:date:date:message-id:message-id: to:to:cc:cc:mime-version:mime-version:content-type:content-type: content-transfer-encoding:content-transfer-encoding: in-reply-to:in-reply-to:references:references; bh=HLBqit68L/U2mdwYaFkydFK3txHa+LkmL8n2YTQXnnw=; b=hDMkfdUFa2+crLVYgkKa35feEWHHtuXyYrUwt5xQQkbcedtdoZAfa642vdy0qaR4daPto0 M0tgUiRz1zwJyv4J56pMuE9Qgm+nzRxRLpk1t6GkiHWk7iJBMmWPEpgcXfEbk3SB0N1oXP J/oTRCYdXknDa+beaM2vnXsntJBiWW8= Received: from mail-yw1-f197.google.com (mail-yw1-f197.google.com [209.85.128.197]) by relay.mimecast.com with ESMTP with STARTTLS (version=TLSv1.3, cipher=TLS_AES_256_GCM_SHA384) id us-mta-374-QJ2XF0_JNIWBv40XbTA59g-1; Mon, 26 Aug 2024 16:44:23 -0400 X-MC-Unique: QJ2XF0_JNIWBv40XbTA59g-1 Received: by mail-yw1-f197.google.com with SMTP id 00721157ae682-6b43e6b9c82so112909007b3.0 for ; Mon, 26 Aug 2024 13:44:23 -0700 (PDT) X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20230601; t=1724705062; x=1725309862; h=content-transfer-encoding:mime-version:references:in-reply-to :message-id:date:subject:cc:to:from:x-gm-message-state:from:to:cc :subject:date:message-id:reply-to; bh=HLBqit68L/U2mdwYaFkydFK3txHa+LkmL8n2YTQXnnw=; b=gOVJovX+mZY7ofjecCDHh27t4SO2Dy7Sq5IZ+CZIIY36xYGCbkPhLCR88I1dEdoDxl L5Tq9/5SZ0JRGY2dzcL5Z+6gOt2Dh9u7zFltrD8FX+n/DdNVJpeHxinw1o2xXjXSmBza yAOwLvGGJdpioEI0gUfxf8crRQsJ+csdbAtfVhMW5FmJhKvN+GExFpdm0A73f6ook4oY 6fzfmhTihR1dOLotFTRAAqV1uCaA1jbpNxoiryNmDAocYCtYbfdU1P7XMqDnz7fCs5Ui znXB2RJCxEhOCaMCHGUBRPBDRhm+8QAUjVwRJCpm+oqibbgOGhQ1kG4M6ueQ/ap3ZZhS PJ7A== X-Forwarded-Encrypted: i=1; AJvYcCUPqal5Tsg9WHHYoRMi+ufYNU7adk4yGPMCq2Elym0Bh3kKyuujZU987ZIIMhpV69tzG5eZv7hMew==@kvack.org X-Gm-Message-State: AOJu0Yyccir0ImNSajWG4BofKas8MgLCYgw1doyCy0H24KI3ax709axA v0WC3OJrU7b9hydinpQLVIEt3XX+fp6KREoZWaqFq19IeJNF1vFZIJvhSIIFCAwBa/MbB3/+SM2 jplfnv0+e3z9sl7F6I4k4X8P45eyQ20rbKOjRFBRv0eOydIsc X-Received: by 2002:a05:690c:39b:b0:62f:aaaa:187a with SMTP id 00721157ae682-6c625a4cc55mr151107077b3.14.1724705062748; Mon, 26 Aug 2024 13:44:22 -0700 (PDT) X-Google-Smtp-Source: AGHT+IGeaoJv/EbB8r7mSzacjacGidgTJuNMUQKi5I30qkj93MEO5BseX564RMMiUfXmELj5ALvNSw== X-Received: by 2002:a05:690c:39b:b0:62f:aaaa:187a with SMTP id 00721157ae682-6c625a4cc55mr151106717b3.14.1724705062426; Mon, 26 Aug 2024 13:44:22 -0700 (PDT) Received: from x1n.redhat.com (pool-99-254-121-117.cpe.net.cable.rogers.com. [99.254.121.117]) by smtp.gmail.com with ESMTPSA id af79cd13be357-7a67f3fd6c1sm491055185a.121.2024.08.26.13.44.20 (version=TLS1_3 cipher=TLS_AES_256_GCM_SHA384 bits=256/256); Mon, 26 Aug 2024 13:44:21 -0700 (PDT) From: Peter Xu To: linux-kernel@vger.kernel.org, linux-mm@kvack.org Cc: Gavin Shan , Catalin Marinas , x86@kernel.org, Ingo Molnar , Andrew Morton , Paolo Bonzini , Dave Hansen , Thomas Gleixner , Alistair Popple , kvm@vger.kernel.org, linux-arm-kernel@lists.infradead.org, Sean Christopherson , peterx@redhat.com, Oscar Salvador , Jason Gunthorpe , Borislav Petkov , Zi Yan , Axel Rasmussen , David Hildenbrand , Yan Zhao , Will Deacon , Kefeng Wang , Alex Williamson Subject: [PATCH v2 13/19] vfio: Use the new follow_pfnmap API Date: Mon, 26 Aug 2024 16:43:47 -0400 Message-ID: <20240826204353.2228736-14-peterx@redhat.com> X-Mailer: git-send-email 2.45.0 In-Reply-To: <20240826204353.2228736-1-peterx@redhat.com> References: <20240826204353.2228736-1-peterx@redhat.com> MIME-Version: 1.0 X-Mimecast-Spam-Score: 0 X-Mimecast-Originator: redhat.com X-Rspamd-Server: rspam07 X-Rspamd-Queue-Id: 266964002A X-Stat-Signature: zu5jgo34mwy5ig7kogo8aey66b3ztgdo X-Rspam-User: X-HE-Tag: 1724705064-51033 X-HE-Meta: U2FsdGVkX18UMbQO5ATNSPNh1LwLn/YiZoM10Wzg7kz4oovztj34MnV7c/+O+WrT9R3wMJ1CSeohCWlvrdYTIb2xoxTxkY/5QGzcE+4bdzd8BUvIPEZ7ww8XdtKaByGCK1l1Mq8wMoq5w7Tpy7Hke+qr3eVCZNJ66dOyBA1E0havZ9apwoVHosSVE4dZc5eeBT/ptgREkNpvl0P4eepiS8dkVC8uUGEY1lab/l5GUasCryBCYuy2UheIKWhlO+w3+C+m29BQIPyMnbYVwWh/rGGnK0ZUsOdHz4XyDkZeG19gmMkGZp8Gs01e02LJY/bHL3UymYpVVOsksLf9S4mPq0O+x6Nwk9HYG+JKT5xORMdErhQUr0LZCBJ6uNfs2P4AWdfZM4Zp+UR+vfPXh/2A+4gVPm4rVnsokgq2sbMn9sI6TqfByOD38ECkjSxyGCkLL843KyKhbma3pbLIymh/s69QPyQzYOlEUsz0W/5qDnPKWg6HJvNI/nbkfK49QepBRm/SnTcpSGXaPgz8sw6XK+Kyrx7QynHzWP30rQPLWW8X02oFKvnc2s1zGtdX/2Q7AskNpOuGMIDkKXzx4OOeRXEgyKO1vTnxYRjbXn22fLtgx4FCxKc1qw6jIJdkELEYpWXBdA589MC17Udh06/6uXfyw8zWOsMUy02+jBhA7+9RUnhUyTCijiaVkkRNGvHmAZAz9sz0NxNCZMCHmM/ea5wMWRyb9nrO5n/5wn8gszndZpnfC+1Nytdp6f3jsZ5X1StZ1fdNz2URfukDgwsSQL34R25eLEMmNX9W7HX1R+jaYS4/CCED4nCRvK8su+H1m+O7jECjpBhufX3K96nJN54XK7JCmjGYRjsQnGISbd6oYS4dZCnD4PsAmJTjmE41X1Q8LFVN1a08oY53Nb+DGhsIFYKKxbzgdmluhlhM/OVfyRdoATpDY1AERqlnNA/W1ArbIAs1NC/eLwqdWFF V0/Wie0K v4xfgnxI5TLtODWGIprsmNRb76JF+D9G3q8PaQdFS5aXvFva9pVfSjL2rM+r+2/Pxpppa3qffYXUk69pfJVOC37kLD6zXzVm5nBoRq7AZReW16qhQ6iRVy8xF/WLro2sKPpBRtcIBDNiDt2wNA2PeuTgrQ2/oYfaGeN78LwlcCg9dS/Lnc1XiNB5DpGz8kjSqWjLzvaeakv52DkFJjQDZMy7Kfw+wAVv48PcV1htAJPd9FOgYqcs701sgLsWPoHyFY4gMLPA2uHymrddAl6PjJOdlnTgksCSNYvKSsU6ZKCKVNKbFMVaBj0IIOjW9O3flthrG6VJoHPRNS8aOnrVWE4HjH+EZYBp+HQqDmvRJRNR3Y6qj19Ein1IX6Dozs2SR+/0i92c3ArRTfGTWNhfBnDz9rouP3U2/I26i X-Bogosity: Ham, tests=bogofilter, spamicity=0.000000, version=1.2.4 Sender: owner-linux-mm@kvack.org Precedence: bulk X-Loop: owner-majordomo@kvack.org List-ID: List-Subscribe: List-Unsubscribe: Use the new API that can understand huge pfn mappings. Cc: Alex Williamson Cc: Jason Gunthorpe Signed-off-by: Peter Xu --- drivers/vfio/vfio_iommu_type1.c | 16 ++++++---------- 1 file changed, 6 insertions(+), 10 deletions(-) diff --git a/drivers/vfio/vfio_iommu_type1.c b/drivers/vfio/vfio_iommu_type1.c index 0960699e7554..bf391b40e576 100644 --- a/drivers/vfio/vfio_iommu_type1.c +++ b/drivers/vfio/vfio_iommu_type1.c @@ -513,12 +513,10 @@ static int follow_fault_pfn(struct vm_area_struct *vma, struct mm_struct *mm, unsigned long vaddr, unsigned long *pfn, bool write_fault) { - pte_t *ptep; - pte_t pte; - spinlock_t *ptl; + struct follow_pfnmap_args args = { .vma = vma, .address = vaddr }; int ret; - ret = follow_pte(vma, vaddr, &ptep, &ptl); + ret = follow_pfnmap_start(&args); if (ret) { bool unlocked = false; @@ -532,19 +530,17 @@ static int follow_fault_pfn(struct vm_area_struct *vma, struct mm_struct *mm, if (ret) return ret; - ret = follow_pte(vma, vaddr, &ptep, &ptl); + ret = follow_pfnmap_start(&args); if (ret) return ret; } - pte = ptep_get(ptep); - - if (write_fault && !pte_write(pte)) + if (write_fault && !args.writable) ret = -EFAULT; else - *pfn = pte_pfn(pte); + *pfn = args.pfn; - pte_unmap_unlock(ptep, ptl); + follow_pfnmap_end(&args); return ret; } From patchwork Mon Aug 26 20:43:48 2024 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Peter Xu X-Patchwork-Id: 13778409 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from kanga.kvack.org (kanga.kvack.org [205.233.56.17]) by smtp.lore.kernel.org (Postfix) with ESMTP id 8A737C5321D for ; Mon, 26 Aug 2024 20:44:44 +0000 (UTC) Received: by kanga.kvack.org (Postfix) id 5C49F6B00A7; Mon, 26 Aug 2024 16:44:32 -0400 (EDT) Received: by kanga.kvack.org (Postfix, from userid 40) id 549636B00A8; Mon, 26 Aug 2024 16:44:32 -0400 (EDT) X-Delivered-To: int-list-linux-mm@kvack.org Received: by kanga.kvack.org (Postfix, from userid 63042) id 3C4D36B00A9; Mon, 26 Aug 2024 16:44:32 -0400 (EDT) X-Delivered-To: linux-mm@kvack.org Received: from relay.hostedemail.com (smtprelay0014.hostedemail.com [216.40.44.14]) by kanga.kvack.org (Postfix) with ESMTP id 1B4026B00A7 for ; Mon, 26 Aug 2024 16:44:32 -0400 (EDT) Received: from smtpin17.hostedemail.com (a10.router.float.18 [10.200.18.1]) by unirelay10.hostedemail.com (Postfix) with ESMTP id C9055C1358 for ; Mon, 26 Aug 2024 20:44:31 +0000 (UTC) X-FDA: 82495574742.17.19DF950 Received: from us-smtp-delivery-124.mimecast.com (us-smtp-delivery-124.mimecast.com [170.10.129.124]) by imf26.hostedemail.com (Postfix) with ESMTP id B2EE714000B for ; Mon, 26 Aug 2024 20:44:29 +0000 (UTC) Authentication-Results: imf26.hostedemail.com; dkim=pass header.d=redhat.com header.s=mimecast20190719 header.b=hXCemyk1; dmarc=pass (policy=none) header.from=redhat.com; spf=pass (imf26.hostedemail.com: domain of peterx@redhat.com designates 170.10.129.124 as permitted sender) smtp.mailfrom=peterx@redhat.com ARC-Seal: i=1; s=arc-20220608; d=hostedemail.com; t=1724705006; a=rsa-sha256; cv=none; b=FiXVwkVsulABrJe9PlgcB1MpEZw0iLDCtfJyIdCJuKYuML8r0pzy7GShVSHNUAvskmpLgx QBmizZJ68d/PKoe+3rN1tiJE6FRnYnsYrHxargTBOv7hl2rSFI1M+95rzFySi8NG8EzxBB huE/sXpS/+WhlI2+nzokZfBLq4ktl5o= ARC-Authentication-Results: i=1; imf26.hostedemail.com; dkim=pass header.d=redhat.com header.s=mimecast20190719 header.b=hXCemyk1; dmarc=pass (policy=none) header.from=redhat.com; spf=pass (imf26.hostedemail.com: domain of peterx@redhat.com designates 170.10.129.124 as permitted sender) smtp.mailfrom=peterx@redhat.com ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=hostedemail.com; s=arc-20220608; t=1724705006; h=from:from:sender:reply-to:subject:subject:date:date: message-id:message-id:to:to:cc:cc:mime-version:mime-version: content-type:content-type: content-transfer-encoding:content-transfer-encoding: in-reply-to:in-reply-to:references:references:dkim-signature; bh=d0kI5he4gLOJUpsF87CpEeiS1FOCa69CaYI81su9df0=; b=rCH+TOmWjwVpZypnkhwuBUdq/dRh92DUalWR8DWpssZy0jcAWe3B4GD/b918lLVTxT0cG8 VAGvtywPM0JTYfBlY3Ciws3eK2v9hfOSFlLhBhBELCGY+1EDHCx3kH9i7gdldn4pIHyF3b bSgMmtOQXOiw1R4VzJeZaKdCXB6k2RU= DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=redhat.com; s=mimecast20190719; t=1724705069; h=from:from:reply-to:subject:subject:date:date:message-id:message-id: to:to:cc:cc:mime-version:mime-version:content-type:content-type: content-transfer-encoding:content-transfer-encoding: in-reply-to:in-reply-to:references:references; bh=d0kI5he4gLOJUpsF87CpEeiS1FOCa69CaYI81su9df0=; b=hXCemyk1jGznDibqf6fEE5WhXyLIPBzuQamm+d+2ykB5JBeYWgIzTcqDb3btvIJ1pHbUSZ glzqM8n/lPORLYONupETHtQdYEZsoQbly4/wGB+e5NkbvFrQiSwH8ddIDHRtrQd0AbaCGF daz4+Qj1i3AGYcmkc4qpjLJ7MVKwPis= Received: from mail-qk1-f199.google.com (mail-qk1-f199.google.com [209.85.222.199]) by relay.mimecast.com with ESMTP with STARTTLS (version=TLSv1.3, cipher=TLS_AES_256_GCM_SHA384) id us-mta-343-LU42H9xCMjKfUaqc1KxNAg-1; Mon, 26 Aug 2024 16:44:26 -0400 X-MC-Unique: LU42H9xCMjKfUaqc1KxNAg-1 Received: by mail-qk1-f199.google.com with SMTP id af79cd13be357-7a66c19d33fso676638685a.0 for ; Mon, 26 Aug 2024 13:44:25 -0700 (PDT) X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20230601; t=1724705064; x=1725309864; h=content-transfer-encoding:mime-version:references:in-reply-to :message-id:date:subject:cc:to:from:x-gm-message-state:from:to:cc :subject:date:message-id:reply-to; bh=d0kI5he4gLOJUpsF87CpEeiS1FOCa69CaYI81su9df0=; b=JAazJMgIQb8VuEE3AshEAKQYYMWZDgufWsrBrl1lOkblP1CWBsV32/E5WRvfkjqp8v uSjLD7poFSMuJ8y+JQL6iXMmCuT/n6i6MnjlmotgtAxLez9zKX0uDfQXLoyUb89q/cuD thR8zEVNJ0aWx/UlKkJLzNlCFAnIH4bT5oX6eSMkTY7N9XrtGqn2vwdV/t6BzI2e1LdB Tzi7x6zZdUyrm/lRf2eSaFy6WAQMKV9WzJCZ01w24Jt9phzXFHf5gr/3mgPYRtO+PnjS n9bJ9LNNmT9cNPTXKCBYUVaFXELd+S9PC/5/BkU8NMa055/7FhAAEw53WaDQC7edGVIx ocug== X-Forwarded-Encrypted: i=1; AJvYcCXJL+IxFKwXRRfAOJUWTGKgJm+i9bklzomzwu/jTRmUu2CCFVe3HOjBBs+R+ht8Xa9y+JYsWWIykw==@kvack.org X-Gm-Message-State: AOJu0YyjHw8onzctyydg1GFEYhlsw/QBFbhM9w3D+Yg9El84c6UfT5HR BnClCvPQ4lAFiLZC1E+gYEin0JYG7foGZDi2Kyig2hFM9qjCcN1LdrabblabbpBoJQc4KAkxCiI Abpo/9A8bdLgozmTN1fis+ZLD6qJw3q1MS36d7nKsYEbjwkvYPWwye0cG X-Received: by 2002:a05:620a:2484:b0:79e:ff41:fd47 with SMTP id af79cd13be357-7a7e4dc9528mr93288585a.28.1724705064650; Mon, 26 Aug 2024 13:44:24 -0700 (PDT) X-Google-Smtp-Source: AGHT+IEloa5qz2QVMofE8PCWJJ6gTfdMgC7cjn+h2anZ9qsldjcOaIzzaCXqVOAeeqkMcyU6B13FuA== X-Received: by 2002:a05:620a:2484:b0:79e:ff41:fd47 with SMTP id af79cd13be357-7a7e4dc9528mr93286485a.28.1724705064315; Mon, 26 Aug 2024 13:44:24 -0700 (PDT) Received: from x1n.redhat.com (pool-99-254-121-117.cpe.net.cable.rogers.com. [99.254.121.117]) by smtp.gmail.com with ESMTPSA id af79cd13be357-7a67f3fd6c1sm491055185a.121.2024.08.26.13.44.22 (version=TLS1_3 cipher=TLS_AES_256_GCM_SHA384 bits=256/256); Mon, 26 Aug 2024 13:44:23 -0700 (PDT) From: Peter Xu To: linux-kernel@vger.kernel.org, linux-mm@kvack.org Cc: Gavin Shan , Catalin Marinas , x86@kernel.org, Ingo Molnar , Andrew Morton , Paolo Bonzini , Dave Hansen , Thomas Gleixner , Alistair Popple , kvm@vger.kernel.org, linux-arm-kernel@lists.infradead.org, Sean Christopherson , peterx@redhat.com, Oscar Salvador , Jason Gunthorpe , Borislav Petkov , Zi Yan , Axel Rasmussen , David Hildenbrand , Yan Zhao , Will Deacon , Kefeng Wang , Alex Williamson Subject: [PATCH v2 14/19] acrn: Use the new follow_pfnmap API Date: Mon, 26 Aug 2024 16:43:48 -0400 Message-ID: <20240826204353.2228736-15-peterx@redhat.com> X-Mailer: git-send-email 2.45.0 In-Reply-To: <20240826204353.2228736-1-peterx@redhat.com> References: <20240826204353.2228736-1-peterx@redhat.com> MIME-Version: 1.0 X-Mimecast-Spam-Score: 0 X-Mimecast-Originator: redhat.com X-Rspamd-Queue-Id: B2EE714000B X-Rspam-User: X-Rspamd-Server: rspam05 X-Stat-Signature: xjjfe84sqoqkt8d5zgmuwgrsbc58y1h6 X-HE-Tag: 1724705069-842489 X-HE-Meta: U2FsdGVkX19BWuR8m/xxxzM+00UrP0GGDizAoK0+3BwYyHukyl+DtckAr//+wGWuwyHtQHl6zWYE46diE9aFFebYYCIQvFGUPJCG5V+aaGP0QYxzg3Lklv8YHw4S2cOB3Ebk+ogIDbE8kcEDWGYIzriVHcKhDFYUmKSgce5nrNZ7E3k1pJEonTUEkE1p6mqQ+XfMSHZ4UGQtSLrKHCCs6dSns9DRmoFjstuHefPaHnBWdcCPnOCbzP/qI2IgVDU9eyue2H7xZsDbAGTf6SUHaLCIMQxV3s0PxbCM0jqgj5lS98CcAEalp6cXlT6aeruOSQWeh16veW350PoxIrxiN/cBu6zIwY5lqgNo7GJxMYhlZdZ420LRwVZpWCfTptX8NkphfIkQwFWqN2MsA8cNZQKusL39CVrXfiXectbDUDhw34DDbT+0c9lfRHH/kH7FRjGvFw/2fG1KKaWKHt/Fv1arXeaeSrGEEzzd10lTcgGSiQv1rTQ1lqu/YcfNz1AVF/jn8OEt86dSKPK0HUzfd370AYLXzjzdvC8pEj+yEKje2Qcp8D7A4ketmYMLJszfmj5XDRS6WoU0VGHfXJzRX1OIhFbLZKI14/CYSTevJ1m1b5fpL6zppWDAsiqCO63miWD43KSiibXm6OFF4INCEDiqeyJycMmu1A4Dpraen6rh5bOrZWjD5dMgMw00O2+/jV0hRMVXPliLjSkR4pBvcjPGF9rIIsvcQtZfPGbicZntnGN+K6OUH+b+uCHPPzPR8jPpHR9piYv538bAJCJLIWSFwsXryFOQmDQKZY3M0Gwkacqcty/9DPinTUgqQRanT/D36y8F27OYQxhRUozE1ifpeHsfHYzIbzmEQveCjIkTD+zI04IrbAe3l2ERZThv6H65OKLqgMNl2rSUFKZV9kC4QfAdghtUHx+IxRKEI5JhuMKik6jiOfpS1PWq9gfPkLi1e9xecleOE2P2qoD DdICrl5N 9j/uYxivykq856fyCO1qxHabcsShfjBfcRNVNVpA1WB+nINAWWAK+PsfM8ukbPYU5+i9LLKagOeBrWkNLWghL3MABg2eaoHe1xIM9mMnhvocpTq/5Mt4rO79ujFaoNwd0L7VhvgnhapCreEn2sEgjptDENrmrB5n53mWOycZ9t0BiG340WkZ2DCUIWDf3JY6HFd9KzW3xviWIJ1xfwPfA10+0P3x0HO7rnzbeKGlw0diPXuZiyJ5GNvmhAooFpZu2mc1WAQCukWoH2VfWmKuv0/MOhcOzsFpFvhRS0GgqrRzr9AeCcjlwbWNLoCeU2j5uIC5o5IGWgg2aQA71j3CMhsWD+g== X-Bogosity: Ham, tests=bogofilter, spamicity=0.000000, version=1.2.4 Sender: owner-linux-mm@kvack.org Precedence: bulk X-Loop: owner-majordomo@kvack.org List-ID: List-Subscribe: List-Unsubscribe: Use the new API that can understand huge pfn mappings. Signed-off-by: Peter Xu --- drivers/virt/acrn/mm.c | 16 +++++++++------- 1 file changed, 9 insertions(+), 7 deletions(-) diff --git a/drivers/virt/acrn/mm.c b/drivers/virt/acrn/mm.c index db8ff1d0ac23..4c2f28715b70 100644 --- a/drivers/virt/acrn/mm.c +++ b/drivers/virt/acrn/mm.c @@ -177,9 +177,7 @@ int acrn_vm_ram_map(struct acrn_vm *vm, struct acrn_vm_memmap *memmap) vma = vma_lookup(current->mm, memmap->vma_base); if (vma && ((vma->vm_flags & VM_PFNMAP) != 0)) { unsigned long start_pfn, cur_pfn; - spinlock_t *ptl; bool writable; - pte_t *ptep; if ((memmap->vma_base + memmap->len) > vma->vm_end) { mmap_read_unlock(current->mm); @@ -187,16 +185,20 @@ int acrn_vm_ram_map(struct acrn_vm *vm, struct acrn_vm_memmap *memmap) } for (i = 0; i < nr_pages; i++) { - ret = follow_pte(vma, memmap->vma_base + i * PAGE_SIZE, - &ptep, &ptl); + struct follow_pfnmap_args args = { + .vma = vma, + .address = memmap->vma_base + i * PAGE_SIZE, + }; + + ret = follow_pfnmap_start(&args); if (ret) break; - cur_pfn = pte_pfn(ptep_get(ptep)); + cur_pfn = args.pfn; if (i == 0) start_pfn = cur_pfn; - writable = !!pte_write(ptep_get(ptep)); - pte_unmap_unlock(ptep, ptl); + writable = args.writable; + follow_pfnmap_end(&args); /* Disallow write access if the PTE is not writable. */ if (!writable && From patchwork Mon Aug 26 20:43:49 2024 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Peter Xu X-Patchwork-Id: 13778408 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from kanga.kvack.org (kanga.kvack.org [205.233.56.17]) by smtp.lore.kernel.org (Postfix) with ESMTP id 34460C5472E for ; Mon, 26 Aug 2024 20:44:42 +0000 (UTC) Received: by kanga.kvack.org (Postfix) id A55A96B00A6; Mon, 26 Aug 2024 16:44:31 -0400 (EDT) Received: by kanga.kvack.org (Postfix, from userid 40) id A07486B00A7; Mon, 26 Aug 2024 16:44:31 -0400 (EDT) X-Delivered-To: int-list-linux-mm@kvack.org Received: by kanga.kvack.org (Postfix, from userid 63042) id 7E4866B00A8; Mon, 26 Aug 2024 16:44:31 -0400 (EDT) X-Delivered-To: linux-mm@kvack.org Received: from relay.hostedemail.com (smtprelay0012.hostedemail.com [216.40.44.12]) by kanga.kvack.org (Postfix) with ESMTP id 5D4B96B00A6 for ; Mon, 26 Aug 2024 16:44:31 -0400 (EDT) Received: from smtpin06.hostedemail.com (a10.router.float.18 [10.200.18.1]) by unirelay05.hostedemail.com (Postfix) with ESMTP id 168E6411DB for ; Mon, 26 Aug 2024 20:44:31 +0000 (UTC) X-FDA: 82495574742.06.1254C96 Received: from us-smtp-delivery-124.mimecast.com (us-smtp-delivery-124.mimecast.com [170.10.129.124]) by imf19.hostedemail.com (Postfix) with ESMTP id 0768E1A0007 for ; Mon, 26 Aug 2024 20:44:28 +0000 (UTC) Authentication-Results: imf19.hostedemail.com; dkim=pass header.d=redhat.com header.s=mimecast20190719 header.b=OewNm48n; dmarc=pass (policy=none) header.from=redhat.com; spf=pass (imf19.hostedemail.com: domain of peterx@redhat.com designates 170.10.129.124 as permitted sender) smtp.mailfrom=peterx@redhat.com ARC-Seal: i=1; s=arc-20220608; d=hostedemail.com; t=1724705049; a=rsa-sha256; cv=none; b=Inr5EcRjsnTj1VKjRr13K0raSieewrY/UlVV6FBcb2/Nt9SVc3tpmYAoX6M2EuZ0cknVi4 ty2vvRZHXoCcKMKSoEMy+vZFLD1b+6E5KizcDPHrb2NCLAxeHLU3QFNfI6xXysCLwfWn7r hPOowIiT8jMc+NsDZs8Gzn1XChb+ixs= ARC-Authentication-Results: i=1; imf19.hostedemail.com; dkim=pass header.d=redhat.com header.s=mimecast20190719 header.b=OewNm48n; dmarc=pass (policy=none) header.from=redhat.com; spf=pass (imf19.hostedemail.com: domain of peterx@redhat.com designates 170.10.129.124 as permitted sender) smtp.mailfrom=peterx@redhat.com ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=hostedemail.com; s=arc-20220608; t=1724705049; h=from:from:sender:reply-to:subject:subject:date:date: message-id:message-id:to:to:cc:cc:mime-version:mime-version: content-type:content-type: content-transfer-encoding:content-transfer-encoding: in-reply-to:in-reply-to:references:references:dkim-signature; bh=t75nynj+5ccZ6qy3UooqAYOOLyCQeJyCwmKKtTf1Ss4=; b=2H8FNyY0fEn041fX7S2dQj69MpD/NKpUGQx65pHZSEIoNTPeo0airyBNi5rqVf1i+fETJR m+qL+imj+2nbjBweY5Von6yR2fDaGCAPeaZml8SIgir3iYRDV5B3jPNc4ashVvwUOUTGpY 1jCGAHGLUB04MbSQ24j4ktzih+CbgTg= DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=redhat.com; s=mimecast20190719; t=1724705068; h=from:from:reply-to:subject:subject:date:date:message-id:message-id: to:to:cc:cc:mime-version:mime-version:content-type:content-type: content-transfer-encoding:content-transfer-encoding: in-reply-to:in-reply-to:references:references; bh=t75nynj+5ccZ6qy3UooqAYOOLyCQeJyCwmKKtTf1Ss4=; b=OewNm48nnEsXZ4g7ozbw/jYqwhX4mt6Y1hsL0mgpqyH6uTCnw2csiExh1uqsIAGtmZ2ORi q6yIMQDyO3onAN7MYTSsFszZfl/15J7Ksbhpyrxb/csEbZpaXX9icONu/RLJt78+tsNruR Q+dfJh3xwxLClRnVm9k00S+MxiD0aTg= Received: from mail-oa1-f71.google.com (mail-oa1-f71.google.com [209.85.160.71]) by relay.mimecast.com with ESMTP with STARTTLS (version=TLSv1.3, cipher=TLS_AES_256_GCM_SHA384) id us-mta-479-M8nyVohvMpqCa_5kQDGLrA-1; Mon, 26 Aug 2024 16:44:27 -0400 X-MC-Unique: M8nyVohvMpqCa_5kQDGLrA-1 Received: by mail-oa1-f71.google.com with SMTP id 586e51a60fabf-2701a253946so7887382fac.3 for ; Mon, 26 Aug 2024 13:44:27 -0700 (PDT) X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20230601; t=1724705066; x=1725309866; h=content-transfer-encoding:mime-version:references:in-reply-to :message-id:date:subject:cc:to:from:x-gm-message-state:from:to:cc :subject:date:message-id:reply-to; bh=t75nynj+5ccZ6qy3UooqAYOOLyCQeJyCwmKKtTf1Ss4=; b=Y07W5rO02I5pQDvgIK7mElU0R8utHQd1Go90wWd6PGkOJhNYciBDiEItgi783NRSFK pfYhkGH2q5HgO1z6Z/T/8pMxL3peCUd+Q6mpAAepqvGwA45I9j6vkU80+6XKEFlmhJp2 NMZv6rzVmNPHZ+INhyYELXqV13HRa//nAenVT3VtTsKOot+1fpVbKHJoeT7c9b5GjE9Y TXN4EqHuX1D6HyiWac9RXMtIFcczEEmZw2Vn7FmeIbWSd/BvBa78P5MTMYoFLvItZllO jAA5yAEXXf59ycWRH3apgu88x7030gQVPQ0Lgb+4TMEXMpZ5SRGN/wFWl1TfWiGTdUd6 izvA== X-Forwarded-Encrypted: i=1; AJvYcCVhmvWuHFnCv3bugtts16N1rwCWv+hIdZv3EaSdkyNyl5aR7aEP/QaqexuZjldaK0s6jQnFuYIxlQ==@kvack.org X-Gm-Message-State: AOJu0YwkOv2CgyZh0tTzDf4J9iEMa8TpQg7lqWqOZAdlScptk1GrIR2e z3sfJ1C0UDdcj9QirWYNa5MXsF+f+89lyUxuyHfWO7Xh6Zbyl5UlM3zzz5xkyP0+yZBX8oB7YDJ O3WDSnPdkB/LIaxynLV88ro8i6UD7DJrm/SLMAmMK+S4QJLox X-Received: by 2002:a05:6358:3a07:b0:1ad:10ff:341e with SMTP id e5c5f4694b2df-1b5c3a3b69dmr1457844155d.4.1724705066543; Mon, 26 Aug 2024 13:44:26 -0700 (PDT) X-Google-Smtp-Source: AGHT+IGoPajws+HFHWojCbb3D8qBEyBE/zIeN3nkxAxD5pVBn23+qUQ20h6jqSHyvqmBY4DTTGYw+g== X-Received: by 2002:a05:6358:3a07:b0:1ad:10ff:341e with SMTP id e5c5f4694b2df-1b5c3a3b69dmr1457841255d.4.1724705066240; Mon, 26 Aug 2024 13:44:26 -0700 (PDT) Received: from x1n.redhat.com (pool-99-254-121-117.cpe.net.cable.rogers.com. [99.254.121.117]) by smtp.gmail.com with ESMTPSA id af79cd13be357-7a67f3fd6c1sm491055185a.121.2024.08.26.13.44.24 (version=TLS1_3 cipher=TLS_AES_256_GCM_SHA384 bits=256/256); Mon, 26 Aug 2024 13:44:25 -0700 (PDT) From: Peter Xu To: linux-kernel@vger.kernel.org, linux-mm@kvack.org Cc: Gavin Shan , Catalin Marinas , x86@kernel.org, Ingo Molnar , Andrew Morton , Paolo Bonzini , Dave Hansen , Thomas Gleixner , Alistair Popple , kvm@vger.kernel.org, linux-arm-kernel@lists.infradead.org, Sean Christopherson , peterx@redhat.com, Oscar Salvador , Jason Gunthorpe , Borislav Petkov , Zi Yan , Axel Rasmussen , David Hildenbrand , Yan Zhao , Will Deacon , Kefeng Wang , Alex Williamson Subject: [PATCH v2 15/19] mm/access_process_vm: Use the new follow_pfnmap API Date: Mon, 26 Aug 2024 16:43:49 -0400 Message-ID: <20240826204353.2228736-16-peterx@redhat.com> X-Mailer: git-send-email 2.45.0 In-Reply-To: <20240826204353.2228736-1-peterx@redhat.com> References: <20240826204353.2228736-1-peterx@redhat.com> MIME-Version: 1.0 X-Mimecast-Spam-Score: 0 X-Mimecast-Originator: redhat.com X-Rspam-User: X-Rspamd-Queue-Id: 0768E1A0007 X-Rspamd-Server: rspam01 X-Stat-Signature: 99br51sgeccxcfamwsmn6omahaeijr1o X-HE-Tag: 1724705068-378920 X-HE-Meta: U2FsdGVkX1+RqPM80klDqu4QM19XNXim3UTjALVIxCUMXlIvHORUsiVh9AAq9rodlszjOp2l6YHU18PNAtKy3FyveQE3K/og25VuB7Dd75zJeKhqXfdUjPPCmFBemavr5XqXYcuH9My4HkP5pIxYVTJS2RQhkkZhjYjmaB5eK8KyIemCFjuo5Os+nl+wtTYWNAwPBpuWNL8oEcl7lIUbeoHOI4k2mt4YS+nsn2oFM/DfV7fArjYz7IFQhpmQCNr/p3nPYiEZAjoldrTeiPxdaKCmYW7liEFekkTPRUYXpoSAlA/oxb/tHlsuE+/kyp8v/wpf+i1gMfmiu4B5BHlVQIZJTcLYeRahh1TDhNGzPLf5gM5HeS9fKGi+rdb8xz7ZUcdmrGWtOIuCKmb71b4qMsm252hEHUh5Ur3HjYiJRsKf2qv3LQGIYNxZbc6X7DfGQupf1Go8KjqM2VhwPJUJKdUcXyd6zKfCOyFx86sIeVdzHya/ebXYDybpwSioaNmqJNF1jN4sTix/mYwHLV5VzZ3GWd1dsvrPMMhMU/wLMj/WmrDjAc44Lm9hr8OOKB9pPeWYw9uYKs0/8uKk91UkZA+S+MBQ5w+360h2B8EJG2kQsWUGtjhZlUB5kkOta4/Xx/ySPjmtoPFYN2nBWX1TDaz5lD55uEy9vd164Zv5G+u7G6dj73XcGL6vnxNhSxvKL8NJwS3V5X73wy6UJmHhyKfFI/d2nC817n7xv/M8pVT/HqED9o5viEIK0EHvU7RBK2+8uLcbj8LWg5FJLGAGEAmhlA2TVvVh9bQ2HP/h3MFNkkQbMj3zBZ4aKhiyOr5GdV7iHhIb34tx2xRy+SIriWVTuo53sMAz6o+9iJi3fHWtPbzUb7vsYWLZPQm+xvoRcJHTPnF2e19lFJpa3LvYZ1qAEw7J04jwMtm5k3//gwncVrU2vbnL5IIonayexgL0tA8gJqWBtZ5C8tmmrJG ADpt5OSK yoC7igmxrbIhefxx3MXjsiS3AwxoZjz0tVBG5Y9kPhO2ZH9ab+aBMTpfaabv0yTrsFxI6b4ah4ALcUPyA6yfzXCuSq0M+cdbJgMf1zeu5BAkRXlJVZ/YyUMfZ71eeD8DQ+qYW4Wi34p7gyqaMS5JS+e4O0xFvOByBoqrzGLX/vow8V0lWc3tCLJvByogc+gZ5BT/J8MKJOBT0ifDMLjrkZBF2vIe8dG+ynoh/ASUvc2ewe4cd7W4wMOqJ2KrddL+bhQzPSwRL4splRyRieeVF4FCc40WEv+50Opp1Nu26IahoC7OXP/xIb7CoVEOt9KNPmt4eqs/cCQD1rdkMZNmIS0y/bQ== X-Bogosity: Ham, tests=bogofilter, spamicity=0.000000, version=1.2.4 Sender: owner-linux-mm@kvack.org Precedence: bulk X-Loop: owner-majordomo@kvack.org List-ID: List-Subscribe: List-Unsubscribe: Use the new API that can understand huge pfn mappings. Signed-off-by: Peter Xu --- mm/memory.c | 28 ++++++++++++++-------------- 1 file changed, 14 insertions(+), 14 deletions(-) diff --git a/mm/memory.c b/mm/memory.c index 0b136c398257..b5d07f493d5d 100644 --- a/mm/memory.c +++ b/mm/memory.c @@ -6342,34 +6342,34 @@ int generic_access_phys(struct vm_area_struct *vma, unsigned long addr, resource_size_t phys_addr; unsigned long prot = 0; void __iomem *maddr; - pte_t *ptep, pte; - spinlock_t *ptl; int offset = offset_in_page(addr); int ret = -EINVAL; + bool writable; + struct follow_pfnmap_args args = { .vma = vma, .address = addr }; retry: - if (follow_pte(vma, addr, &ptep, &ptl)) + if (follow_pfnmap_start(&args)) return -EINVAL; - pte = ptep_get(ptep); - pte_unmap_unlock(ptep, ptl); + prot = pgprot_val(args.pgprot); + phys_addr = (resource_size_t)args.pfn << PAGE_SHIFT; + writable = args.writable; + follow_pfnmap_end(&args); - prot = pgprot_val(pte_pgprot(pte)); - phys_addr = (resource_size_t)pte_pfn(pte) << PAGE_SHIFT; - - if ((write & FOLL_WRITE) && !pte_write(pte)) + if ((write & FOLL_WRITE) && !writable) return -EINVAL; maddr = ioremap_prot(phys_addr, PAGE_ALIGN(len + offset), prot); if (!maddr) return -ENOMEM; - if (follow_pte(vma, addr, &ptep, &ptl)) + if (follow_pfnmap_start(&args)) goto out_unmap; - if (!pte_same(pte, ptep_get(ptep))) { - pte_unmap_unlock(ptep, ptl); + if ((prot != pgprot_val(args.pgprot)) || + (phys_addr != (args.pfn << PAGE_SHIFT)) || + (writable != args.writable)) { + follow_pfnmap_end(&args); iounmap(maddr); - goto retry; } @@ -6378,7 +6378,7 @@ int generic_access_phys(struct vm_area_struct *vma, unsigned long addr, else memcpy_fromio(buf, maddr + offset, len); ret = len; - pte_unmap_unlock(ptep, ptl); + follow_pfnmap_end(&args); out_unmap: iounmap(maddr); From patchwork Mon Aug 26 20:43:50 2024 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Peter Xu X-Patchwork-Id: 13778410 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from kanga.kvack.org (kanga.kvack.org [205.233.56.17]) by smtp.lore.kernel.org (Postfix) with ESMTP id 117FCC5472C for ; Mon, 26 Aug 2024 20:44:47 +0000 (UTC) Received: by kanga.kvack.org (Postfix) id 747EB6B00A9; Mon, 26 Aug 2024 16:44:33 -0400 (EDT) Received: by kanga.kvack.org (Postfix, from userid 40) id 6A9896B00AA; Mon, 26 Aug 2024 16:44:33 -0400 (EDT) X-Delivered-To: int-list-linux-mm@kvack.org Received: by kanga.kvack.org (Postfix, from userid 63042) id 4D5F96B00AB; Mon, 26 Aug 2024 16:44:33 -0400 (EDT) X-Delivered-To: linux-mm@kvack.org Received: from relay.hostedemail.com (smtprelay0012.hostedemail.com [216.40.44.12]) by kanga.kvack.org (Postfix) with ESMTP id 28DAE6B00A9 for ; Mon, 26 Aug 2024 16:44:33 -0400 (EDT) Received: from smtpin28.hostedemail.com (a10.router.float.18 [10.200.18.1]) by unirelay08.hostedemail.com (Postfix) with ESMTP id CD93314121F for ; Mon, 26 Aug 2024 20:44:32 +0000 (UTC) X-FDA: 82495574784.28.07E7E7C Received: from us-smtp-delivery-124.mimecast.com (us-smtp-delivery-124.mimecast.com [170.10.129.124]) by imf16.hostedemail.com (Postfix) with ESMTP id B75DF180020 for ; Mon, 26 Aug 2024 20:44:30 +0000 (UTC) Authentication-Results: imf16.hostedemail.com; dkim=pass header.d=redhat.com header.s=mimecast20190719 header.b=Y8SkvWu2; spf=pass (imf16.hostedemail.com: domain of peterx@redhat.com designates 170.10.129.124 as permitted sender) smtp.mailfrom=peterx@redhat.com; dmarc=pass (policy=none) header.from=redhat.com ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=hostedemail.com; s=arc-20220608; t=1724705002; h=from:from:sender:reply-to:subject:subject:date:date: message-id:message-id:to:to:cc:cc:mime-version:mime-version: content-type:content-type: content-transfer-encoding:content-transfer-encoding: in-reply-to:in-reply-to:references:references:dkim-signature; bh=X8WHCJ7kNzF2XCQH70BbHdio2suKNHxAf6rhsHMkPbs=; b=Ylsb6x4zUJmBHXVNlH6hQvHjfgFGPwddJzKn9nvoZu/6qYnV8XCOdpzvMYEPAuHXqK1Rpr hLqoq/Rwvu2W9HN8t7hOTuBGaYL+NIIyb2RCcUCyQlOdZyY5AnICRFOTiXICqjNIF+nP7a j3KN7ndgN50nDYG6276ZgpoZqeeejCk= ARC-Authentication-Results: i=1; imf16.hostedemail.com; dkim=pass header.d=redhat.com header.s=mimecast20190719 header.b=Y8SkvWu2; spf=pass (imf16.hostedemail.com: domain of peterx@redhat.com designates 170.10.129.124 as permitted sender) smtp.mailfrom=peterx@redhat.com; dmarc=pass (policy=none) header.from=redhat.com ARC-Seal: i=1; s=arc-20220608; d=hostedemail.com; t=1724705002; a=rsa-sha256; cv=none; b=5ux2P2TofTNIDctrI8qoWxcQRg0RtbmZVVu6S7oupT5xZTdq4bDEgy2oEYWc5LtBLWE2ZF ePbPmyV0IEYvnIzWGJO3FCfrwvkbrb6RSfUnnGwaGELfQrESjTf3KY9RqJ88SdulLzVbiP sg3mejfC+YRgeO8JMXCfJ08AHWa0wbA= DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=redhat.com; s=mimecast20190719; t=1724705070; h=from:from:reply-to:subject:subject:date:date:message-id:message-id: to:to:cc:cc:mime-version:mime-version:content-type:content-type: content-transfer-encoding:content-transfer-encoding: in-reply-to:in-reply-to:references:references; bh=X8WHCJ7kNzF2XCQH70BbHdio2suKNHxAf6rhsHMkPbs=; b=Y8SkvWu2LOubSTzyFjhJG+uaffHluqMJERUI1Eu1Uw/9ljdkiwXGV1hGP183x9gafw8i3I j7tSUSw3ajrzFTK6pgtF3iqMCJsMlviAcKYcA7qksr2PXh+k1frHWfPTuz/PccEMarvGs6 4RIvB1FJ+sHyXnt2nwT3G80QRCHjWwI= Received: from mail-ua1-f72.google.com (mail-ua1-f72.google.com [209.85.222.72]) by relay.mimecast.com with ESMTP with STARTTLS (version=TLSv1.3, cipher=TLS_AES_256_GCM_SHA384) id us-mta-455-718lCt5dM7a8WjbAifUNaw-1; Mon, 26 Aug 2024 16:44:29 -0400 X-MC-Unique: 718lCt5dM7a8WjbAifUNaw-1 Received: by mail-ua1-f72.google.com with SMTP id a1e0cc1a2514c-842f7fc5cb5so1351924241.0 for ; Mon, 26 Aug 2024 13:44:28 -0700 (PDT) X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20230601; t=1724705068; x=1725309868; h=content-transfer-encoding:mime-version:references:in-reply-to :message-id:date:subject:cc:to:from:x-gm-message-state:from:to:cc :subject:date:message-id:reply-to; bh=X8WHCJ7kNzF2XCQH70BbHdio2suKNHxAf6rhsHMkPbs=; b=npQtJeBPQGiFox2cRFW2wVg5UylMVgfgeExKidRcKQyl8IXV0I6I8enuffQKMhAJWM SUMlgsHkUDft1Fj8YaKuBnXv4cac8TYFkmuJrco3EmFFuZyb5hbI9Yyyy88eIKysHJJD orJeuFJSVs2c45O70sG81Kciig4nm7vb5wP3b/7OGpsNoQ09eX/T43INzw25cjpL+SdO BQ0Pyz8zXsHUP+CVgT0CCseYzDFIZDGx/CEx8VJM+z1nH0wuRHyqs2TZkqdKnzzFU0WG K3lcWN6SWcwiGZQ0x+5G3RG/XT9kc4MWY80MtoupHHzQngbQpzo8FVFGrfNYDw7rch9y NoNA== X-Forwarded-Encrypted: i=1; AJvYcCV/L/LJWJ8h28WdtqIShs5dzHyNVpUUcID/9wi+IqKvzKc9XQm2mLOYHNEHzLBTFT2fbmaoCOEgmQ==@kvack.org X-Gm-Message-State: AOJu0YxSxz2HkoWPpoq34U6y56x819cAXG6O1A2Mt6KpasWtCHFGld36 t7AM/pU2Mt/jaShhrZ4B+B3Ho2enNL2YpoJXviZZRhb6IhVWGXiVhUTJwYy9kjHbyiEAfAn2FJA yxNCE08dQ8WQesO8o/VpwnwZBbbL+ALKFHdP9YWftiJe89miO X-Received: by 2002:a05:6102:3e94:b0:493:e585:6ce3 with SMTP id ada2fe7eead31-49a3bd1cd83mr1005184137.31.1724705068473; Mon, 26 Aug 2024 13:44:28 -0700 (PDT) X-Google-Smtp-Source: AGHT+IGKOe4NKlkU/HSIr+h3CCvXyG5C70MeRvYsCyQPr65oWdFZ+m1iMwzpLlsyBV/cJVa8qCplQQ== X-Received: by 2002:a05:6102:3e94:b0:493:e585:6ce3 with SMTP id ada2fe7eead31-49a3bd1cd83mr1005176137.31.1724705068153; Mon, 26 Aug 2024 13:44:28 -0700 (PDT) Received: from x1n.redhat.com (pool-99-254-121-117.cpe.net.cable.rogers.com. [99.254.121.117]) by smtp.gmail.com with ESMTPSA id af79cd13be357-7a67f3fd6c1sm491055185a.121.2024.08.26.13.44.26 (version=TLS1_3 cipher=TLS_AES_256_GCM_SHA384 bits=256/256); Mon, 26 Aug 2024 13:44:27 -0700 (PDT) From: Peter Xu To: linux-kernel@vger.kernel.org, linux-mm@kvack.org Cc: Gavin Shan , Catalin Marinas , x86@kernel.org, Ingo Molnar , Andrew Morton , Paolo Bonzini , Dave Hansen , Thomas Gleixner , Alistair Popple , kvm@vger.kernel.org, linux-arm-kernel@lists.infradead.org, Sean Christopherson , peterx@redhat.com, Oscar Salvador , Jason Gunthorpe , Borislav Petkov , Zi Yan , Axel Rasmussen , David Hildenbrand , Yan Zhao , Will Deacon , Kefeng Wang , Alex Williamson Subject: [PATCH v2 16/19] mm: Remove follow_pte() Date: Mon, 26 Aug 2024 16:43:50 -0400 Message-ID: <20240826204353.2228736-17-peterx@redhat.com> X-Mailer: git-send-email 2.45.0 In-Reply-To: <20240826204353.2228736-1-peterx@redhat.com> References: <20240826204353.2228736-1-peterx@redhat.com> MIME-Version: 1.0 X-Mimecast-Spam-Score: 0 X-Mimecast-Originator: redhat.com X-Stat-Signature: fobd1n9dtxaufj6nkoxgz3u85m1babtk X-Rspam-User: X-Rspamd-Queue-Id: B75DF180020 X-Rspamd-Server: rspam02 X-HE-Tag: 1724705070-662681 X-HE-Meta: U2FsdGVkX194q9wsiAup6hYZtb3cCwVpJD6JNTly1zFe07gY0Uju9k3ck7+aCZQCvzlhYm2bvnymEEtLFzHPj0MjblQ/rMPx/R1IRzZbdrJq70VYcd3oDH2GEEQ6yxwm6yqrTbpcVvSfCjsah/w5uIeia6gW0IMEO5cz+nH9+CnAp3+zmf7lNVU3Q10/obcBWKcbRmuUBgataf+1RpW1ArwGMhj8PWaTt0Ek69SUbmCzy8h/mvLfg//6keTEgk0zpyVmY5f3DnvQbQB2MOSP3vFAOloytygGxWBHNDbxrXYTbw8Cwjsa4+r6sLshJzBAEZ4BAtTXmjZrIJV1yQZ8fZq8GxoBDO5sXK5auuoO7v4xQ7m3yg6U9qz99z+P+DZobQ7LfBw1KoRYBus66xz6zWzaQTpKla7IJhOu+yQ9QUjhi05Da3LSKo01Xa6wUIhDScUni/LJD82IDmXHtnoKWRdygLeD+uwQXYGLJ2AYJD1yM4RcH7P7uwef81413phZJ0ovRa0+lBq2xc08git95RqSG0NnFRJO9SU/1zf+nrIpE1U1Yf7VOlSbPqalVeO72MPjwur4jE2clJqakmqe/KYKQDDIBaovS8YvL+hwbARW4Mm1Gv9bvGtItZW4OgX4Z+O9au/QC2VzQU1rT5kMSW64N8ADoq3yIOBvU5MJL7+pR946PKr/uTwGD6eIAUi9LvO0a4rjLgN2B+90zmK2J/XlNO8P14zbrbaSNkMZkdAR8qW8b14m8vpmyufy6FpdBZ4QJIqmWT7PbOjTD8fPjdulEaDef65+2RG60mLTamHaV8uBeR0GeT+14R9F2TgJm+cv/BwgZXBjAWE5HT4umS2p70WnH7iaDwavdJ4r6f1ttpZ6xuCWuB2xfOBF6RWG3i+aBlhYQCEiJLFLHLUpZoJUD4WuHXnCgaRP6DiBRkAYOjLG7NAAPVhE7OwohbDNzYpDi2LMI6CmbhnznmY s5MaQIhN 0EyIMm6QxbV7yyEjDklpp3FF8tzWJuA1f6RUa5tFi+liWx3lXCkPzF+Q7pySacvYYo9nsqpPPyj5orBDmKMajrU+J31UjTaZnpkPlcjII/0MGDN4/ep0vlFZmU+NL9VbrLxDsgFpaGM+UfB8+ier/vXCxkrKjZbWhcdoaNG7CiC2Ai9X9apx2GxUv4cmX4fjjWN/tKXyIc5ySA7Qzqp3RUSKhCPsDvSTM3E1PncIsNfa7f+cMsLMIuzA9BjAXGu7m9kNL+COatwZYs/lkNajFEkBTNurvv+THckK3GQOZdNL09TrziIhtHte7jZI9kCZTyJCbioQVD3qoJ8AJPD+j9Y82Dg== X-Bogosity: Ham, tests=bogofilter, spamicity=0.000000, version=1.2.4 Sender: owner-linux-mm@kvack.org Precedence: bulk X-Loop: owner-majordomo@kvack.org List-ID: List-Subscribe: List-Unsubscribe: follow_pte() users have been converted to follow_pfnmap*(). Remove the API. Signed-off-by: Peter Xu --- include/linux/mm.h | 2 -- mm/memory.c | 73 ---------------------------------------------- 2 files changed, 75 deletions(-) diff --git a/include/linux/mm.h b/include/linux/mm.h index 161d496bfd18..b31d4bdd65ad 100644 --- a/include/linux/mm.h +++ b/include/linux/mm.h @@ -2368,8 +2368,6 @@ void free_pgd_range(struct mmu_gather *tlb, unsigned long addr, unsigned long end, unsigned long floor, unsigned long ceiling); int copy_page_range(struct vm_area_struct *dst_vma, struct vm_area_struct *src_vma); -int follow_pte(struct vm_area_struct *vma, unsigned long address, - pte_t **ptepp, spinlock_t **ptlp); int generic_access_phys(struct vm_area_struct *vma, unsigned long addr, void *buf, int len, int write); diff --git a/mm/memory.c b/mm/memory.c index b5d07f493d5d..288f81a8698e 100644 --- a/mm/memory.c +++ b/mm/memory.c @@ -6100,79 +6100,6 @@ int __pmd_alloc(struct mm_struct *mm, pud_t *pud, unsigned long address) } #endif /* __PAGETABLE_PMD_FOLDED */ -/** - * follow_pte - look up PTE at a user virtual address - * @vma: the memory mapping - * @address: user virtual address - * @ptepp: location to store found PTE - * @ptlp: location to store the lock for the PTE - * - * On a successful return, the pointer to the PTE is stored in @ptepp; - * the corresponding lock is taken and its location is stored in @ptlp. - * - * The contents of the PTE are only stable until @ptlp is released using - * pte_unmap_unlock(). This function will fail if the PTE is non-present. - * Present PTEs may include PTEs that map refcounted pages, such as - * anonymous folios in COW mappings. - * - * Callers must be careful when relying on PTE content after - * pte_unmap_unlock(). Especially if the PTE maps a refcounted page, - * callers must protect against invalidation with MMU notifiers; otherwise - * access to the PFN at a later point in time can trigger use-after-free. - * - * Only IO mappings and raw PFN mappings are allowed. The mmap semaphore - * should be taken for read. - * - * This function must not be used to modify PTE content. - * - * Return: zero on success, -ve otherwise. - */ -int follow_pte(struct vm_area_struct *vma, unsigned long address, - pte_t **ptepp, spinlock_t **ptlp) -{ - struct mm_struct *mm = vma->vm_mm; - pgd_t *pgd; - p4d_t *p4d; - pud_t *pud; - pmd_t *pmd; - pte_t *ptep; - - mmap_assert_locked(mm); - if (unlikely(address < vma->vm_start || address >= vma->vm_end)) - goto out; - - if (!(vma->vm_flags & (VM_IO | VM_PFNMAP))) - goto out; - - pgd = pgd_offset(mm, address); - if (pgd_none(*pgd) || unlikely(pgd_bad(*pgd))) - goto out; - - p4d = p4d_offset(pgd, address); - if (p4d_none(*p4d) || unlikely(p4d_bad(*p4d))) - goto out; - - pud = pud_offset(p4d, address); - if (pud_none(*pud) || unlikely(pud_bad(*pud))) - goto out; - - pmd = pmd_offset(pud, address); - VM_BUG_ON(pmd_trans_huge(*pmd)); - - ptep = pte_offset_map_lock(mm, pmd, address, ptlp); - if (!ptep) - goto out; - if (!pte_present(ptep_get(ptep))) - goto unlock; - *ptepp = ptep; - return 0; -unlock: - pte_unmap_unlock(ptep, *ptlp); -out: - return -EINVAL; -} -EXPORT_SYMBOL_GPL(follow_pte); - static inline void pfnmap_args_setup(struct follow_pfnmap_args *args, spinlock_t *lock, pte_t *ptep, pgprot_t pgprot, unsigned long pfn_base, From patchwork Mon Aug 26 20:43:51 2024 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Peter Xu X-Patchwork-Id: 13778411 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from kanga.kvack.org (kanga.kvack.org [205.233.56.17]) by smtp.lore.kernel.org (Postfix) with ESMTP id 523A7C5321D for ; Mon, 26 Aug 2024 20:44:49 +0000 (UTC) Received: by kanga.kvack.org (Postfix) id B6D6F6B00AB; Mon, 26 Aug 2024 16:44:35 -0400 (EDT) Received: by kanga.kvack.org (Postfix, from userid 40) id B15F66B00AC; Mon, 26 Aug 2024 16:44:35 -0400 (EDT) X-Delivered-To: int-list-linux-mm@kvack.org Received: by kanga.kvack.org (Postfix, from userid 63042) id 943416B00AD; Mon, 26 Aug 2024 16:44:35 -0400 (EDT) X-Delivered-To: linux-mm@kvack.org Received: from relay.hostedemail.com (smtprelay0017.hostedemail.com [216.40.44.17]) by kanga.kvack.org (Postfix) with ESMTP id 66F9A6B00AB for ; Mon, 26 Aug 2024 16:44:35 -0400 (EDT) Received: from smtpin13.hostedemail.com (a10.router.float.18 [10.200.18.1]) by unirelay08.hostedemail.com (Postfix) with ESMTP id 1CCA5141099 for ; Mon, 26 Aug 2024 20:44:35 +0000 (UTC) X-FDA: 82495574910.13.0EC1FB8 Received: from us-smtp-delivery-124.mimecast.com (us-smtp-delivery-124.mimecast.com [170.10.129.124]) by imf07.hostedemail.com (Postfix) with ESMTP id E785D40015 for ; Mon, 26 Aug 2024 20:44:32 +0000 (UTC) Authentication-Results: imf07.hostedemail.com; dkim=pass header.d=redhat.com header.s=mimecast20190719 header.b="Bx48/gEy"; dmarc=pass (policy=none) header.from=redhat.com; spf=pass (imf07.hostedemail.com: domain of peterx@redhat.com designates 170.10.129.124 as permitted sender) smtp.mailfrom=peterx@redhat.com ARC-Seal: i=1; s=arc-20220608; d=hostedemail.com; t=1724705053; a=rsa-sha256; cv=none; b=H2DDQ4C0Ak1F/DXNDU7H05EDOtAuskwezBUigjws0LHRnfHmB+v9vfNwTJgl4mDMx9vTOd rZ9b5kdjkrTEPjeTGQ8PzWo1IG838mOf6VMPsBfXoi2BpuYubEWitWi0Z1BLO0F0SDcpjr C9WTjpNirIIMg4XjHjB4lEGnmsvkdAg= ARC-Authentication-Results: i=1; imf07.hostedemail.com; dkim=pass header.d=redhat.com header.s=mimecast20190719 header.b="Bx48/gEy"; dmarc=pass (policy=none) header.from=redhat.com; spf=pass (imf07.hostedemail.com: domain of peterx@redhat.com designates 170.10.129.124 as permitted sender) smtp.mailfrom=peterx@redhat.com ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=hostedemail.com; s=arc-20220608; t=1724705053; h=from:from:sender:reply-to:subject:subject:date:date: message-id:message-id:to:to:cc:cc:mime-version:mime-version: content-type:content-type: content-transfer-encoding:content-transfer-encoding: in-reply-to:in-reply-to:references:references:dkim-signature; bh=IC5gYR+7Z4BwAPnZp8TN8XZEGypCYGRNkok3NV40z5s=; b=YlHz6DjO+8vApZL6iuAFf4vkfNS5qHeO84Kvycrd2knDjJpfogMMOTloUsZqe8YHe4LTKw zaiYvsc09EWREJMyAb7OClNviWYZI5Myvjo9XMiExlCCmUWtWdR0CZ12OLSBk25O9VTgZu FVuXnf3zY6c4C4PPFbmTMU2kS1WVs6w= DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=redhat.com; s=mimecast20190719; t=1724705072; h=from:from:reply-to:subject:subject:date:date:message-id:message-id: to:to:cc:cc:mime-version:mime-version:content-type:content-type: content-transfer-encoding:content-transfer-encoding: in-reply-to:in-reply-to:references:references; bh=IC5gYR+7Z4BwAPnZp8TN8XZEGypCYGRNkok3NV40z5s=; b=Bx48/gEy8HcNPSGTC1saLoQpFtRlpAqaHob2hDvDJssP+38CWMrcsGJHc9yq14J+BcKZKB pTw4U6vVcGmGAmneL4EgB6tKASr//AaS5oPgIPuaZj7qp/UMLGNCE8ZZmCxgMsFCl3iKHN nipTN8TOh7lp2ZJCuTHVmvEytRghhrI= Received: from mail-oo1-f69.google.com (mail-oo1-f69.google.com [209.85.161.69]) by relay.mimecast.com with ESMTP with STARTTLS (version=TLSv1.3, cipher=TLS_AES_256_GCM_SHA384) id us-mta-195-KZWYpV3qPxyI8b7b-OtlNQ-1; Mon, 26 Aug 2024 16:44:31 -0400 X-MC-Unique: KZWYpV3qPxyI8b7b-OtlNQ-1 Received: by mail-oo1-f69.google.com with SMTP id 006d021491bc7-5d5c7bfd8aaso6416841eaf.2 for ; Mon, 26 Aug 2024 13:44:31 -0700 (PDT) X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20230601; t=1724705070; x=1725309870; h=content-transfer-encoding:mime-version:references:in-reply-to :message-id:date:subject:cc:to:from:x-gm-message-state:from:to:cc :subject:date:message-id:reply-to; bh=IC5gYR+7Z4BwAPnZp8TN8XZEGypCYGRNkok3NV40z5s=; b=vi1h+ZONASayu04LF0WAeIvMIP37v1tp7RB57ftdfB3+6NH1pxh9GVSq5GEBVabyMN Dxn7Q3+CYyOF/Mugw+xn/hf95I2LkL/OX1GOiaD2DRzdB1ilM8QhWA53cHoPxZXSQn+y BplSygofFQdqspm20O0aj/OGOf9rscaDQcjYg04YXWqzQWYHPeccaxF9YA/vQKtQOIoa UEAtXnodhjcuCztqPoWy+1bNjyR9FEY9wAcriu1MsT0QIdSgUYdw4DDGKNKchQqaEKKQ Nth5CP2NvDtnCfMGlh152txFwfVtMsy6nOj84WrnV6mWBixBcNKkepXv9S4bKkH7wfSa JjlQ== X-Forwarded-Encrypted: i=1; AJvYcCUSJwuibA5+YBQlgevrLxjhDiIspjypB4A8GtGzkipfZz6zzQFd/x5ab7L1UtxZdR3FRbuhJM7k6w==@kvack.org X-Gm-Message-State: AOJu0YwbCns+BusA7sVsEw3zq4Eeh3XI8PAnxRuQb50BP/Kc2OEGzpgS NXAsc+za8M4JDDj3tiuhOcHWmh2qVFDwvQOQRsmfksDuRmoHQLHqih0V4GF5gvDLxCehxvPbqpa 7Q7mTsJQ6BTdd4/w/ykzBf//XX9DxSNdDNREGIgYU4r7PXo5o X-Received: by 2002:a05:6358:70c3:b0:1ac:f00d:c8c6 with SMTP id e5c5f4694b2df-1b5c22ebd07mr1493541755d.27.1724705070546; Mon, 26 Aug 2024 13:44:30 -0700 (PDT) X-Google-Smtp-Source: AGHT+IHWRaHJzCfl4jAxAoYVG6HBrfG1+v3SSmaZHfZSlba6wG4sR68STMcFscyvu7mDZcazVEGWKw== X-Received: by 2002:a05:6358:70c3:b0:1ac:f00d:c8c6 with SMTP id e5c5f4694b2df-1b5c22ebd07mr1493538355d.27.1724705070146; Mon, 26 Aug 2024 13:44:30 -0700 (PDT) Received: from x1n.redhat.com (pool-99-254-121-117.cpe.net.cable.rogers.com. [99.254.121.117]) by smtp.gmail.com with ESMTPSA id af79cd13be357-7a67f3fd6c1sm491055185a.121.2024.08.26.13.44.28 (version=TLS1_3 cipher=TLS_AES_256_GCM_SHA384 bits=256/256); Mon, 26 Aug 2024 13:44:29 -0700 (PDT) From: Peter Xu To: linux-kernel@vger.kernel.org, linux-mm@kvack.org Cc: Gavin Shan , Catalin Marinas , x86@kernel.org, Ingo Molnar , Andrew Morton , Paolo Bonzini , Dave Hansen , Thomas Gleixner , Alistair Popple , kvm@vger.kernel.org, linux-arm-kernel@lists.infradead.org, Sean Christopherson , peterx@redhat.com, Oscar Salvador , Jason Gunthorpe , Borislav Petkov , Zi Yan , Axel Rasmussen , David Hildenbrand , Yan Zhao , Will Deacon , Kefeng Wang , Alex Williamson Subject: [PATCH v2 17/19] mm/x86: Support large pfn mappings Date: Mon, 26 Aug 2024 16:43:51 -0400 Message-ID: <20240826204353.2228736-18-peterx@redhat.com> X-Mailer: git-send-email 2.45.0 In-Reply-To: <20240826204353.2228736-1-peterx@redhat.com> References: <20240826204353.2228736-1-peterx@redhat.com> MIME-Version: 1.0 X-Mimecast-Spam-Score: 0 X-Mimecast-Originator: redhat.com X-Rspam-User: X-Rspamd-Queue-Id: E785D40015 X-Rspamd-Server: rspam01 X-Stat-Signature: z4y33b3yo9a1b4x4r1dnq8h55sigij5f X-HE-Tag: 1724705072-842191 X-HE-Meta: U2FsdGVkX19TfaqS+uPGqtRC4yW6I+ZhrevcZ+huT38zLxLehliH4hMqS34BvCFJAloLFrjcDqq22BXJ51bg7Uto6Y8RJfFsScIsXTbEUwvMThLdkFs1VNwuNpudVGrCtyo3gYRML9YfHFLAj+CzOkQi6euFjaGNE0OL5L92YE+zQ0XL/y1gt6YTckVayxmdi5QkM2WtzngZnwi1F2mf+jzqW8VOMwW1TlcqvJ9CkJhTg9lVcz5dNiPKumuFs26GoCvdFNEc6W90oZiDVzrDe5B6yCpNCrhQOLKbDfgsNR3Bhhi9/6yt/A5wR3QSPkNPlzopvTQIB6jLTDmAryR5BFaYYmWyKrKk4uiDEdZfM/KYA6WR1ZAcZwmZ0RbY25DEUgx4ZB+wnKgqwr/ojTNQyo/+yRpQiRkt3y+vdHew7SXFsgl5K8BCD9KhCVvm/8y5ggB4XeG3d13hzNkfcrXIOZdWsycrQ5DiEcyXBf/wOqoJsqt2aC9V6kIL7cA7KL++ARobe988t2jVt8bUrqipcafGxfjcvUOwnPyFkf+nNlFvycjbTqadfEymc+hpSf5ALMDMKRNR7w4/M/DsuXdJOfQDu+DA+FZz1i8tG108XCkNuUTc0+8uak4ujItCJVGCQtz4WS6Q8BVc1uyrFxry4tzXBAa9Q1UFmgT/TyToAd/VMyrbUvkAQN475AbE0a8UdQTgrVY4pmeVFADkh+SF7w+rzd4O/SJy8WH5YjJ4L7WecnQc0NSRXH3EoY0vH6hw9RpFK2W7WCCOg3KxfTKkid6eo391WlV0lNwz0+clBzLe/2OukVDqdrK5I3DgTRwWAUZuqkg4jIIjcjaoUKupx1+gYV03UIbkeS8NIXlIXoVho9Y2tzFB9HBrgiZo/PmfpUCaF7Gsz5fu0kKIEHJQnNqHKwaovQxmC6Bn/ivsHXpD1DDfyOGoqCB7WITcuktJefsmGZZVRg0xaHF3SWj PZjk3xe9 umBmw7Ug0x2dhPIu9llBRhraakcIsU1q4rYKZmKkNylroQvEt5iV3zWQClcnOTfssOMCBPhXRztf7KFPEYtY2iCTe9MV83LLijNyoJsl5auSw+P2NXtySP5I3grHNZS78pD3BLZkp0pOIvWVDgyg/uPYJ5pG4xdITcHxJ9yuljLxelFw03nP+vxumQw08MvB5qmvfE2j6b356Ih/ZeONd/Y9jOjiHwo7AXPmKXekMyajEyfWYE4Y/9t2QXUzIcpJ+4Wt/osGa8LrZNL6QpYbYc6AVnxRMnIo6HCq4OutGBYdO86jexEfp02n+TlZaoG3/Jq0vchVUTzOdOXnL3q/FExeNp7HtQRATAldfVt0Nwqf+WRrNrFrq2WXdOJ5js+IZHjnxUMMCtiNSXRF3gnrjmQxbSSG7QNoWo11+YXE+lOsn6h8= X-Bogosity: Ham, tests=bogofilter, spamicity=0.000000, version=1.2.4 Sender: owner-linux-mm@kvack.org Precedence: bulk X-Loop: owner-majordomo@kvack.org List-ID: List-Subscribe: List-Unsubscribe: Helpers to install and detect special pmd/pud entries. In short, bit 9 on x86 is not used for pmd/pud, so we can directly define them the same as the pte level. One note is that it's also used in _PAGE_BIT_CPA_TEST but that is only used in the debug test, and shouldn't conflict in this case. One note is that pxx_set|clear_flags() for pmd/pud will need to be moved upper so that they can be referenced by the new special bit helpers. There's no change in the code that was moved. Cc: x86@kernel.org Cc: Thomas Gleixner Cc: Ingo Molnar Cc: Borislav Petkov Cc: Dave Hansen Signed-off-by: Peter Xu --- arch/x86/Kconfig | 1 + arch/x86/include/asm/pgtable.h | 80 ++++++++++++++++++++++------------ 2 files changed, 53 insertions(+), 28 deletions(-) diff --git a/arch/x86/Kconfig b/arch/x86/Kconfig index b74b9ee484da..d4dbe9717e96 100644 --- a/arch/x86/Kconfig +++ b/arch/x86/Kconfig @@ -28,6 +28,7 @@ config X86_64 select ARCH_HAS_GIGANTIC_PAGE select ARCH_SUPPORTS_INT128 if CC_HAS_INT128 select ARCH_SUPPORTS_PER_VMA_LOCK + select ARCH_SUPPORTS_HUGE_PFNMAP if TRANSPARENT_HUGEPAGE select HAVE_ARCH_SOFT_DIRTY select MODULES_USE_ELF_RELA select NEED_DMA_MAP_STATE diff --git a/arch/x86/include/asm/pgtable.h b/arch/x86/include/asm/pgtable.h index 8d12bfad6a1d..4c2d080d26b4 100644 --- a/arch/x86/include/asm/pgtable.h +++ b/arch/x86/include/asm/pgtable.h @@ -120,6 +120,34 @@ extern pmdval_t early_pmd_flags; #define arch_end_context_switch(prev) do {} while(0) #endif /* CONFIG_PARAVIRT_XXL */ +static inline pmd_t pmd_set_flags(pmd_t pmd, pmdval_t set) +{ + pmdval_t v = native_pmd_val(pmd); + + return native_make_pmd(v | set); +} + +static inline pmd_t pmd_clear_flags(pmd_t pmd, pmdval_t clear) +{ + pmdval_t v = native_pmd_val(pmd); + + return native_make_pmd(v & ~clear); +} + +static inline pud_t pud_set_flags(pud_t pud, pudval_t set) +{ + pudval_t v = native_pud_val(pud); + + return native_make_pud(v | set); +} + +static inline pud_t pud_clear_flags(pud_t pud, pudval_t clear) +{ + pudval_t v = native_pud_val(pud); + + return native_make_pud(v & ~clear); +} + /* * The following only work if pte_present() is true. * Undefined behaviour if not.. @@ -317,6 +345,30 @@ static inline int pud_devmap(pud_t pud) } #endif +#ifdef CONFIG_ARCH_SUPPORTS_PMD_PFNMAP +static inline bool pmd_special(pmd_t pmd) +{ + return pmd_flags(pmd) & _PAGE_SPECIAL; +} + +static inline pmd_t pmd_mkspecial(pmd_t pmd) +{ + return pmd_set_flags(pmd, _PAGE_SPECIAL); +} +#endif /* CONFIG_ARCH_SUPPORTS_PMD_PFNMAP */ + +#ifdef CONFIG_ARCH_SUPPORTS_PUD_PFNMAP +static inline bool pud_special(pud_t pud) +{ + return pud_flags(pud) & _PAGE_SPECIAL; +} + +static inline pud_t pud_mkspecial(pud_t pud) +{ + return pud_set_flags(pud, _PAGE_SPECIAL); +} +#endif /* CONFIG_ARCH_SUPPORTS_PUD_PFNMAP */ + static inline int pgd_devmap(pgd_t pgd) { return 0; @@ -487,20 +539,6 @@ static inline pte_t pte_mkdevmap(pte_t pte) return pte_set_flags(pte, _PAGE_SPECIAL|_PAGE_DEVMAP); } -static inline pmd_t pmd_set_flags(pmd_t pmd, pmdval_t set) -{ - pmdval_t v = native_pmd_val(pmd); - - return native_make_pmd(v | set); -} - -static inline pmd_t pmd_clear_flags(pmd_t pmd, pmdval_t clear) -{ - pmdval_t v = native_pmd_val(pmd); - - return native_make_pmd(v & ~clear); -} - /* See comments above mksaveddirty_shift() */ static inline pmd_t pmd_mksaveddirty(pmd_t pmd) { @@ -595,20 +633,6 @@ static inline pmd_t pmd_mkwrite_novma(pmd_t pmd) pmd_t pmd_mkwrite(pmd_t pmd, struct vm_area_struct *vma); #define pmd_mkwrite pmd_mkwrite -static inline pud_t pud_set_flags(pud_t pud, pudval_t set) -{ - pudval_t v = native_pud_val(pud); - - return native_make_pud(v | set); -} - -static inline pud_t pud_clear_flags(pud_t pud, pudval_t clear) -{ - pudval_t v = native_pud_val(pud); - - return native_make_pud(v & ~clear); -} - /* See comments above mksaveddirty_shift() */ static inline pud_t pud_mksaveddirty(pud_t pud) { From patchwork Mon Aug 26 20:43:52 2024 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Peter Xu X-Patchwork-Id: 13778412 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from kanga.kvack.org (kanga.kvack.org [205.233.56.17]) by smtp.lore.kernel.org (Postfix) with ESMTP id CBA61C5472E for ; Mon, 26 Aug 2024 20:44:51 +0000 (UTC) Received: by kanga.kvack.org (Postfix) id C51BC6B00AD; Mon, 26 Aug 2024 16:44:37 -0400 (EDT) Received: by kanga.kvack.org (Postfix, from userid 40) id C00F56B00AE; Mon, 26 Aug 2024 16:44:37 -0400 (EDT) X-Delivered-To: int-list-linux-mm@kvack.org Received: by kanga.kvack.org (Postfix, from userid 63042) id A05556B00AF; Mon, 26 Aug 2024 16:44:37 -0400 (EDT) X-Delivered-To: linux-mm@kvack.org Received: from relay.hostedemail.com (smtprelay0012.hostedemail.com [216.40.44.12]) by kanga.kvack.org (Postfix) with ESMTP id 7EBBB6B00AD for ; Mon, 26 Aug 2024 16:44:37 -0400 (EDT) Received: from smtpin20.hostedemail.com (a10.router.float.18 [10.200.18.1]) by unirelay01.hostedemail.com (Postfix) with ESMTP id 3A0C81C4E25 for ; Mon, 26 Aug 2024 20:44:37 +0000 (UTC) X-FDA: 82495574994.20.53C969C Received: from us-smtp-delivery-124.mimecast.com (us-smtp-delivery-124.mimecast.com [170.10.133.124]) by imf17.hostedemail.com (Postfix) with ESMTP id 2773C40008 for ; Mon, 26 Aug 2024 20:44:35 +0000 (UTC) Authentication-Results: imf17.hostedemail.com; dkim=pass header.d=redhat.com header.s=mimecast20190719 header.b=YYLVoWiG; dmarc=pass (policy=none) header.from=redhat.com; spf=pass (imf17.hostedemail.com: domain of peterx@redhat.com designates 170.10.133.124 as permitted sender) smtp.mailfrom=peterx@redhat.com ARC-Seal: i=1; s=arc-20220608; d=hostedemail.com; t=1724705056; a=rsa-sha256; cv=none; b=VPuKczshfG50cZbSkqK5og0OdEoh/uDF7aRwu/6YDjxgVNL6RAZVNFZIZCNhWInmxrS9/R 4xI5XCSa/rl5YWUVXdkSt/Is1M+G3KJLCpZQMTysRYP2oXgjlofQQWWHui42sE+Ldvdr7Q a7Fke+VgUH0fSNtRDUcz5lc7ctbgqz8= ARC-Authentication-Results: i=1; imf17.hostedemail.com; dkim=pass header.d=redhat.com header.s=mimecast20190719 header.b=YYLVoWiG; dmarc=pass (policy=none) header.from=redhat.com; spf=pass (imf17.hostedemail.com: domain of peterx@redhat.com designates 170.10.133.124 as permitted sender) smtp.mailfrom=peterx@redhat.com ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=hostedemail.com; s=arc-20220608; t=1724705056; h=from:from:sender:reply-to:subject:subject:date:date: message-id:message-id:to:to:cc:cc:mime-version:mime-version: content-type:content-type: content-transfer-encoding:content-transfer-encoding: in-reply-to:in-reply-to:references:references:dkim-signature; bh=FEAM9DK74WZYySVYhL7YnC2YOY0xQ42haB8uzH2x2BY=; b=55fMtcQ9Z+qnxxjW1gKhY5fqeL4MfdKyYALAEDBWELXiVdf3Fys4KX6FhWCa9g9kd1Pbe+ yfFl5Z2iYTl6BrP19Xdspoor2MRVqycvm9IwnLo2yXGQrfWCmbj5oAbOSIRZIB44laWBSu 0TqoLdeEdSPTKC/vgERkv8rK14K4Dd8= DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=redhat.com; s=mimecast20190719; t=1724705074; h=from:from:reply-to:subject:subject:date:date:message-id:message-id: to:to:cc:cc:mime-version:mime-version:content-type:content-type: content-transfer-encoding:content-transfer-encoding: in-reply-to:in-reply-to:references:references; bh=FEAM9DK74WZYySVYhL7YnC2YOY0xQ42haB8uzH2x2BY=; b=YYLVoWiGteZHco1yz/MpCpfQF0yuSjBl1/+aBlhmwR1vGYC/tdjrJcfQZlZrF0e9fqAAqF 7VTkx7WYGYneWcCf+1sMX+ik/PDLezaALHGApYH03Wjeo7mkXZc1u5YTk2nTbMLFbzsYUf YnhJl9U7lzcQndaKRAOubldo2RhP3AA= Received: from mail-qk1-f199.google.com (mail-qk1-f199.google.com [209.85.222.199]) by relay.mimecast.com with ESMTP with STARTTLS (version=TLSv1.3, cipher=TLS_AES_256_GCM_SHA384) id us-mta-554-rfjIgXKePW6xZG6plbQN4A-1; Mon, 26 Aug 2024 16:44:33 -0400 X-MC-Unique: rfjIgXKePW6xZG6plbQN4A-1 Received: by mail-qk1-f199.google.com with SMTP id af79cd13be357-7a1e9a3825aso606412385a.2 for ; Mon, 26 Aug 2024 13:44:33 -0700 (PDT) X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20230601; t=1724705072; x=1725309872; h=content-transfer-encoding:mime-version:references:in-reply-to :message-id:date:subject:cc:to:from:x-gm-message-state:from:to:cc :subject:date:message-id:reply-to; bh=FEAM9DK74WZYySVYhL7YnC2YOY0xQ42haB8uzH2x2BY=; b=joD02D9qF/uMllT4ir6sM5PQ0M5Eqbf753Vw7xr/rM5Z95RN9Z1iOUhpeVFHRZd+uM 7En3Ub3KjOaSfbCYHnv71AVrkUZwUTjFXF7i6Toy1VAAfE1GpOGSVcB7U+38CylgbJJ8 TiGVHJPBUCRpVStMbI75qhv7YH5P043OClr5nSgUCDW+mrVFi2W64GdIwzPhxg9ooeA9 0bbFCOd258QruZYrzLYNf/LlRsWoW5RgV47L9ffgq3tdrYWajdrWNDAySwuSyvsGryej qoGtTcVcsF85jku7eGHACAixzdk2jdftKwjuQQ275vmmKJZeQFl4bATKww2+td2BDjvV bMFQ== X-Forwarded-Encrypted: i=1; AJvYcCVr3zcd8a8hKb5ROs4Gf58BIwc4sg8eO/INcR0BB/UVKJCNac11yVMCGBe12IQVh4l7okq3pva20Q==@kvack.org X-Gm-Message-State: AOJu0YwTBGDKkY5mc/z6wAFvsj/gxCROU9yHibC/OeTbdfpmlLQALVeO hyEt5k9LkO5Dfu3+myhLd0oW9vA1Eksn5Ka2dWfRb6INCS2hOt2+g3t3WmrPfiTDMBL4wz9aCWJ hJsVcBPGysW+NZrS6kily94gE4fOND6kJ8K0En98RA1EUWwmmELCIkMPi X-Received: by 2002:a05:620a:430a:b0:7a4:faab:fc79 with SMTP id af79cd13be357-7a6896d1835mr1392500285a.8.1724705072549; Mon, 26 Aug 2024 13:44:32 -0700 (PDT) X-Google-Smtp-Source: AGHT+IHDiJ56urcvTKu8a4/ieO25ZyreTkMfmYYALFQqTS8OY3wwzzWFQolk9khXQC+xv0hRJc4KwA== X-Received: by 2002:a05:620a:430a:b0:7a4:faab:fc79 with SMTP id af79cd13be357-7a6896d1835mr1392497485a.8.1724705072141; Mon, 26 Aug 2024 13:44:32 -0700 (PDT) Received: from x1n.redhat.com (pool-99-254-121-117.cpe.net.cable.rogers.com. [99.254.121.117]) by smtp.gmail.com with ESMTPSA id af79cd13be357-7a67f3fd6c1sm491055185a.121.2024.08.26.13.44.30 (version=TLS1_3 cipher=TLS_AES_256_GCM_SHA384 bits=256/256); Mon, 26 Aug 2024 13:44:31 -0700 (PDT) From: Peter Xu To: linux-kernel@vger.kernel.org, linux-mm@kvack.org Cc: Gavin Shan , Catalin Marinas , x86@kernel.org, Ingo Molnar , Andrew Morton , Paolo Bonzini , Dave Hansen , Thomas Gleixner , Alistair Popple , kvm@vger.kernel.org, linux-arm-kernel@lists.infradead.org, Sean Christopherson , peterx@redhat.com, Oscar Salvador , Jason Gunthorpe , Borislav Petkov , Zi Yan , Axel Rasmussen , David Hildenbrand , Yan Zhao , Will Deacon , Kefeng Wang , Alex Williamson Subject: [PATCH v2 18/19] mm/arm64: Support large pfn mappings Date: Mon, 26 Aug 2024 16:43:52 -0400 Message-ID: <20240826204353.2228736-19-peterx@redhat.com> X-Mailer: git-send-email 2.45.0 In-Reply-To: <20240826204353.2228736-1-peterx@redhat.com> References: <20240826204353.2228736-1-peterx@redhat.com> MIME-Version: 1.0 X-Mimecast-Spam-Score: 0 X-Mimecast-Originator: redhat.com X-Rspam-User: X-Rspamd-Queue-Id: 2773C40008 X-Rspamd-Server: rspam01 X-Stat-Signature: z934tg434mckxjxiwc4j8nu6rzwy1ukn X-HE-Tag: 1724705075-103837 X-HE-Meta: U2FsdGVkX1/zPijBPSqeZ5cPemstG+1dHMTSaMKaQyu1mXf5OVq6AnLnjixSJbGBGXrAlitdDBT9Obvr31LNsUFp9U+yr9OsR0ByGrRgz+fdktl+QPJg/XalzDg9RdXTFOY7KHdxJGls9LgWrzxIbUXfSvtQfaK/cq5l/Iq4bKx5KBSSECa1SGpyOcPssqqT7wUn/2bRCw779G28lJwCeIG4/kTvpvdWfqaR8pzx4bgfRL4BB6ftQf+g0cIisupYaTrpVF/clCJy9Nh9HFj31/VkgZ5FaJWfJH5csFuTk1iySlQj2lped62Oq0LuamnZBJljhC/vX7ucKMLOYvfjbM3UdbOYLTCZml1CvVjL+hHZ6aH4v4u4BPX4lALvUupZrRCNF52nE/tMHfmYvAGOIPuG4Kr4hoeyA5cKI0AzQFLxKv4WcsU5kV+0NihSxEmTCj6G7j7Xc+DWUmEiyV3t++vgLVGQc/w+4wVgezAG9cTd7R81j09kTh6oRHi3GmL+Sjd/oljAzTjXjUGU6AL+KIRjKuyEQyO6cieWzsLF7AstC9UONbV0kd+aKlmphqIVXjk/K7+3bae7Wg3AbA0RIjc/WvaytTRSyIAqj6Kvxavda2HYK/yR0POX/KByY0CHiNnCLHmNUZ7ibC+SmazD7OJWzCfFFkbvRfwid1ygFJWYkbVNeAyrn7SjJkctgT+xjo+ArHNR6cquZVECa6xJ6uAy5p4uAlNP8uzqoDLpDs5tWudCYNX1r+ddMPPRdZn2JHvgI+P3LYwId9qZ9u8e0sDRwmmkgclNCK8hM88+77jCb7fQLfwSYby5/4FsGGKPBokexjvGaL1vupNJqqptsBenmwzJ0qcAX3003cB19gvCMueLyJQngAv23TUknZBhzR5AtlJpCv7O4QlccqcA0H8CJgJTVOQGb6/lOkTbXwQSWw2ZI6kyojC44+1MqdPzzfz0k1BAmGCJEUKW8GW 8MvgGU/N eHVg3p1W2XWwqfo/5o3vOHST2jBLS3XEKZ+K6biNo8XqcPVtZ5XBLlje1Hqg4q4tCWAErGzpqX2lLoHqQpIEoAXZfHMgv1uVTcEwv7XNeeHeuBf6BketW9jb/zajWOIHZ73Fgh5WGRkgtKukhSFWvk18JegXzZ6L6PUX+tG0AQOeyl2KD3Ti+KI7nvm3w2JDEBgw4Cu9VQHte6Ngf/KCHobBHoCjz+pJn/A8XHdbUZs/pWwoYj/NDQESSfMm7kOfNqsLRfNrhIeAwWZ+uSzBtqLDHmrkfLjRSABUOCGc88npYhN4HqANQQUHigaRduC+tvVAW/qHmPAvA7GIr0w6O+YlUtwAuDhlrL8jR4RUdstnWpCBzXXtMTZ8FOf8ntmlNPctRzh+EzPgYgRAuB3Wt79hdgJ/xK1u/YOesjRTgk5z3XGFbqiH/iMQoj84fQmIdrddeD3Bb9O+/lHR9redrME1QghL2saJn+KX9zHFwiCXtObuPHY+A8Qir8xVGVfAyQynJcmStbDHDKCo= X-Bogosity: Ham, tests=bogofilter, spamicity=0.000000, version=1.2.4 Sender: owner-linux-mm@kvack.org Precedence: bulk X-Loop: owner-majordomo@kvack.org List-ID: List-Subscribe: List-Unsubscribe: Support huge pfnmaps by using bit 56 (PTE_SPECIAL) for "special" on pmds/puds. Provide the pmd/pud helpers to set/get special bit. There's one more thing missing for arm64 which is the pxx_pgprot() for pmd/pud. Add them too, which is mostly the same as the pte version by dropping the pfn field. These helpers are essential to be used in the new follow_pfnmap*() API to report valid pgprot_t results. Note that arm64 doesn't yet support huge PUD yet, but it's still straightforward to provide the pud helpers that we need altogether. Only PMD helpers will make an immediate benefit until arm64 will support huge PUDs first in general (e.g. in THPs). Cc: linux-arm-kernel@lists.infradead.org Cc: Catalin Marinas Cc: Will Deacon Signed-off-by: Peter Xu --- arch/arm64/Kconfig | 1 + arch/arm64/include/asm/pgtable.h | 29 +++++++++++++++++++++++++++++ 2 files changed, 30 insertions(+) diff --git a/arch/arm64/Kconfig b/arch/arm64/Kconfig index 6494848019a0..6607ed8fdbb4 100644 --- a/arch/arm64/Kconfig +++ b/arch/arm64/Kconfig @@ -99,6 +99,7 @@ config ARM64 select ARCH_SUPPORTS_NUMA_BALANCING select ARCH_SUPPORTS_PAGE_TABLE_CHECK select ARCH_SUPPORTS_PER_VMA_LOCK + select ARCH_SUPPORTS_HUGE_PFNMAP if TRANSPARENT_HUGEPAGE select ARCH_WANT_BATCHED_UNMAP_TLB_FLUSH select ARCH_WANT_COMPAT_IPC_PARSE_VERSION if COMPAT select ARCH_WANT_DEFAULT_BPF_JIT diff --git a/arch/arm64/include/asm/pgtable.h b/arch/arm64/include/asm/pgtable.h index b78cc4a6758b..2faecc033a19 100644 --- a/arch/arm64/include/asm/pgtable.h +++ b/arch/arm64/include/asm/pgtable.h @@ -578,6 +578,14 @@ static inline pmd_t pmd_mkdevmap(pmd_t pmd) return pte_pmd(set_pte_bit(pmd_pte(pmd), __pgprot(PTE_DEVMAP))); } +#ifdef CONFIG_ARCH_SUPPORTS_PMD_PFNMAP +#define pmd_special(pte) (!!((pmd_val(pte) & PTE_SPECIAL))) +static inline pmd_t pmd_mkspecial(pmd_t pmd) +{ + return set_pmd_bit(pmd, __pgprot(PTE_SPECIAL)); +} +#endif + #define __pmd_to_phys(pmd) __pte_to_phys(pmd_pte(pmd)) #define __phys_to_pmd_val(phys) __phys_to_pte_val(phys) #define pmd_pfn(pmd) ((__pmd_to_phys(pmd) & PMD_MASK) >> PAGE_SHIFT) @@ -595,6 +603,27 @@ static inline pmd_t pmd_mkdevmap(pmd_t pmd) #define pud_pfn(pud) ((__pud_to_phys(pud) & PUD_MASK) >> PAGE_SHIFT) #define pfn_pud(pfn,prot) __pud(__phys_to_pud_val((phys_addr_t)(pfn) << PAGE_SHIFT) | pgprot_val(prot)) +#ifdef CONFIG_ARCH_SUPPORTS_PUD_PFNMAP +#define pud_special(pte) pte_special(pud_pte(pud)) +#define pud_mkspecial(pte) pte_pud(pte_mkspecial(pud_pte(pud))) +#endif + +#define pmd_pgprot pmd_pgprot +static inline pgprot_t pmd_pgprot(pmd_t pmd) +{ + unsigned long pfn = pmd_pfn(pmd); + + return __pgprot(pmd_val(pfn_pmd(pfn, __pgprot(0))) ^ pmd_val(pmd)); +} + +#define pud_pgprot pud_pgprot +static inline pgprot_t pud_pgprot(pud_t pud) +{ + unsigned long pfn = pud_pfn(pud); + + return __pgprot(pud_val(pfn_pud(pfn, __pgprot(0))) ^ pud_val(pud)); +} + static inline void __set_pte_at(struct mm_struct *mm, unsigned long __always_unused addr, pte_t *ptep, pte_t pte, unsigned int nr) From patchwork Mon Aug 26 20:43:53 2024 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Peter Xu X-Patchwork-Id: 13778413 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from kanga.kvack.org (kanga.kvack.org [205.233.56.17]) by smtp.lore.kernel.org (Postfix) with ESMTP id 2080BC5472C for ; Mon, 26 Aug 2024 20:44:54 +0000 (UTC) Received: by kanga.kvack.org (Postfix) id DD9D06B00AF; Mon, 26 Aug 2024 16:44:39 -0400 (EDT) Received: by kanga.kvack.org (Postfix, from userid 40) id D3B336B00B0; Mon, 26 Aug 2024 16:44:39 -0400 (EDT) X-Delivered-To: int-list-linux-mm@kvack.org Received: by kanga.kvack.org (Postfix, from userid 63042) id B65C06B00B1; Mon, 26 Aug 2024 16:44:39 -0400 (EDT) X-Delivered-To: linux-mm@kvack.org Received: from relay.hostedemail.com (smtprelay0016.hostedemail.com [216.40.44.16]) by kanga.kvack.org (Postfix) with ESMTP id 8CC316B00AF for ; Mon, 26 Aug 2024 16:44:39 -0400 (EDT) Received: from smtpin15.hostedemail.com (a10.router.float.18 [10.200.18.1]) by unirelay05.hostedemail.com (Postfix) with ESMTP id 0E907413CB for ; Mon, 26 Aug 2024 20:44:39 +0000 (UTC) X-FDA: 82495575078.15.03E67AF Received: from us-smtp-delivery-124.mimecast.com (us-smtp-delivery-124.mimecast.com [170.10.129.124]) by imf13.hostedemail.com (Postfix) with ESMTP id EF4AB20019 for ; Mon, 26 Aug 2024 20:44:36 +0000 (UTC) Authentication-Results: imf13.hostedemail.com; dkim=pass header.d=redhat.com header.s=mimecast20190719 header.b=FDIAs+ZA; dmarc=pass (policy=none) header.from=redhat.com; spf=pass (imf13.hostedemail.com: domain of peterx@redhat.com designates 170.10.129.124 as permitted sender) smtp.mailfrom=peterx@redhat.com ARC-Seal: i=1; s=arc-20220608; d=hostedemail.com; t=1724705007; a=rsa-sha256; cv=none; b=nvaIGY9KSNqJH7olxLLv847cGAkh97/G063CWVt2ZJVN2Fro0kHoQVDwn6TThhGL8tB9w1 2XU9mrb5iaTQqexMRxnjZp6ZGc39wiJeMaZfNxxzEer3RKFWOqnS5T/GVthY6uLQ1w5Zyr 3Zz14aDjqcXIiM6Om6eFMmGks0diD9s= ARC-Authentication-Results: i=1; imf13.hostedemail.com; dkim=pass header.d=redhat.com header.s=mimecast20190719 header.b=FDIAs+ZA; dmarc=pass (policy=none) header.from=redhat.com; spf=pass (imf13.hostedemail.com: domain of peterx@redhat.com designates 170.10.129.124 as permitted sender) smtp.mailfrom=peterx@redhat.com ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=hostedemail.com; s=arc-20220608; t=1724705007; h=from:from:sender:reply-to:subject:subject:date:date: message-id:message-id:to:to:cc:cc:mime-version:mime-version: content-type:content-type: content-transfer-encoding:content-transfer-encoding: in-reply-to:in-reply-to:references:references:dkim-signature; bh=9MPOki3YF/wyEgBnejzItxwjIF1moGifGifGb5OwzY4=; b=VzykeHYWvlLjkpv729jdn14O+XhAKuOhECX8Cnz278scRoxXcBQd4EjiMAftUPYt2z4L7K D8bD5d5B8vQUORnAxDo/wmfopDje3N123gMLnLKGL1RgPSaGj2zqLpkGdsyrnCrLU28Ksy 7nktJ0YFel8WRkCQA+BOG68uLqtN4Ng= DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=redhat.com; s=mimecast20190719; t=1724705076; h=from:from:reply-to:subject:subject:date:date:message-id:message-id: to:to:cc:cc:mime-version:mime-version:content-type:content-type: content-transfer-encoding:content-transfer-encoding: in-reply-to:in-reply-to:references:references; bh=9MPOki3YF/wyEgBnejzItxwjIF1moGifGifGb5OwzY4=; b=FDIAs+ZALbgen0k8cu0tRJgAA+EKXDxjQUttBBD+BpXE+wKtY3Qxb/eO5q1BAPGjAFnNSv mkY5io3dvWZf41lPLORMvb3ES8DbvLBnmbk6SLl/D5+qIgk3ghQkRZdjAsViHpgg5SbktJ mpRWwQPYXj0rZNG92At3Cw/CAWshGIw= Received: from mail-qk1-f199.google.com (mail-qk1-f199.google.com [209.85.222.199]) by relay.mimecast.com with ESMTP with STARTTLS (version=TLSv1.3, cipher=TLS_AES_256_GCM_SHA384) id us-mta-563-8gTl8bUsN6CArjyc04SRFw-1; Mon, 26 Aug 2024 16:44:35 -0400 X-MC-Unique: 8gTl8bUsN6CArjyc04SRFw-1 Received: by mail-qk1-f199.google.com with SMTP id af79cd13be357-7a1dbbe6e4cso581918985a.3 for ; Mon, 26 Aug 2024 13:44:35 -0700 (PDT) X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20230601; t=1724705074; x=1725309874; h=content-transfer-encoding:mime-version:references:in-reply-to :message-id:date:subject:cc:to:from:x-gm-message-state:from:to:cc :subject:date:message-id:reply-to; bh=9MPOki3YF/wyEgBnejzItxwjIF1moGifGifGb5OwzY4=; b=ldJXNRVBGHMkjP/3WT+qFsADHEvjze/KO84x56zu+ds85V7+R+59uOAvs5GqtDn0aq Epjzh2Q7SjJ2u0dSQpyvI6zavfj5cXqyESmfAlWJ8YttAnuFu/oIa8wwDmYoIhLfaT/z YShfGilDGdf4UfKDCBTOerl3D1voxpF9k3VkNqayG31RxeK6nS7+5Q8YHWRxDG8dsvvo jJN8vlt0Lt+C6O1Rw7SFHHVZXpAAiTTd5vl1o/mGJdMWmcfS6H89CIjBjRIQVtgeZWY7 NX1egHAHb3RKY8soBnIQ8jdav1OSYpbUV7lCOuTEdEVZkhVwDoud6QSvbKnHX5lzn4xU tGEQ== X-Forwarded-Encrypted: i=1; AJvYcCVrFPJsMKN4bgZHA+NHP0LaAVpGwF6ZE5DoXlqQapvwSIADInSt7N3MSMDtnPtepnnczPgNNUrntg==@kvack.org X-Gm-Message-State: AOJu0YzDFliqePPbdhGfF6Lg2g8cQkRJJ6Lm+WNSbFfPUbmIovMNK8MH YA5UZOChAboseFI70CVrVLBvoxcjgwTEHWo4iQ7DF0/TxYOQfzlZw9K8oPnjTJdkUg/qSNbTN+z 4yvWoWY0ERxtItaUJx7uaT818M7z0IzUOyUaqFnEEs1mCIBPhxtXsk7By X-Received: by 2002:a05:620a:4007:b0:79f:595:f64a with SMTP id af79cd13be357-7a6897b7b14mr1477183385a.58.1724705074688; Mon, 26 Aug 2024 13:44:34 -0700 (PDT) X-Google-Smtp-Source: AGHT+IHpx2qE8XXNwFA/VbcshgEEhM7utf/nDzzwdfyOyUpm1p8Kz8GxiojqL+PSarI5vXlPZn99hQ== X-Received: by 2002:a05:620a:4007:b0:79f:595:f64a with SMTP id af79cd13be357-7a6897b7b14mr1477180385a.58.1724705074328; Mon, 26 Aug 2024 13:44:34 -0700 (PDT) Received: from x1n.redhat.com (pool-99-254-121-117.cpe.net.cable.rogers.com. [99.254.121.117]) by smtp.gmail.com with ESMTPSA id af79cd13be357-7a67f3fd6c1sm491055185a.121.2024.08.26.13.44.32 (version=TLS1_3 cipher=TLS_AES_256_GCM_SHA384 bits=256/256); Mon, 26 Aug 2024 13:44:33 -0700 (PDT) From: Peter Xu To: linux-kernel@vger.kernel.org, linux-mm@kvack.org Cc: Gavin Shan , Catalin Marinas , x86@kernel.org, Ingo Molnar , Andrew Morton , Paolo Bonzini , Dave Hansen , Thomas Gleixner , Alistair Popple , kvm@vger.kernel.org, linux-arm-kernel@lists.infradead.org, Sean Christopherson , peterx@redhat.com, Oscar Salvador , Jason Gunthorpe , Borislav Petkov , Zi Yan , Axel Rasmussen , David Hildenbrand , Yan Zhao , Will Deacon , Kefeng Wang , Alex Williamson Subject: [PATCH v2 19/19] vfio/pci: Implement huge_fault support Date: Mon, 26 Aug 2024 16:43:53 -0400 Message-ID: <20240826204353.2228736-20-peterx@redhat.com> X-Mailer: git-send-email 2.45.0 In-Reply-To: <20240826204353.2228736-1-peterx@redhat.com> References: <20240826204353.2228736-1-peterx@redhat.com> MIME-Version: 1.0 X-Mimecast-Spam-Score: 0 X-Mimecast-Originator: redhat.com X-Rspamd-Server: rspam12 X-Rspamd-Queue-Id: EF4AB20019 X-Stat-Signature: s6styomhng9bsoeeebe65kawky3k1af3 X-Rspam-User: X-HE-Tag: 1724705076-20634 X-HE-Meta: U2FsdGVkX1/1v4n4EHBQmirFfTZJqQm1h1uip83oUjuEmwhsSDZiBQhQSXdb4NDT6SUKOziVmiv1NVH4bMaRMO8FEdl+tkBrzL0dkNQCjEVVE9MNNRF2jHTRRwEZ5bljgoCFaaOKdntD7ac5ajnXQmAqCGwOecgOBVnFc0ILH3Bevt8oVftRf+CrukoYYIXPwYvx/zHQzLP20lIE742YZr2ChHW8DvsV2l/OkDqHGVaR8NRt+goBvzVCWaF4zw1aTuDSJopFnE+wop9ZG0ajLJHMARLruAOwXfPNUmBxW4GHaPhYfHhp7dxSMwoFVqZzpOBiek+5vyR8XdiyBSQbAa0QvotsZyworN25GjEkyyZvFJx/mx0JLcYyMAfwetSIDtvjp9EizytEZRxK0b/o7w0GPDucGNsLng+X0yO6ublXbW3Q79ldoToUx9TXDZylUUxp83IxY6q+6VZ2medZ1bQ+aV+oP3ZzI1iUUz0ISX+7MZw6ULMOcm0zFiOTqO1AoLu5UvseGxL+EGwXKF5ut81/QzyK+pgJuJ2LQ+Mqn6UFRu+db71wVi7KEdyU8wt8V0SWzKeXO/RjFtkUaqYvINGrXi41Afn3mqCCxGSBV1G9IWFATXmdCjSj3yHWdmJHn1j8kyoC5u4s596xGx0FvDCWTa3dZ/frYc+8y0PkkRYXCqlgVjfJ6tmw2ch1IV1/OB/J8Ztpe4E7LEaPM1P8tHCLMO/+ce/2v4lA3zizzBU6ThM4z2GeP2Jf9b5QupxoAaiF7WxIMI1mgzAeF6bMiy8mR+IJ/jfGD/3iOF9CHjp9u3GiwMSsOZ7jSasCthLWZCE3WfGVxofmS4K0xVK+FI63lHW4PVNTrIy46PJkPi7Cr2q9oUYnVP1wvTANfflx1hvmniFvYOArNzkTMg9Snff2dRMvMbfj5If/l7ZTI0LVe8ojfyJqz6+FvUiLcJ+xTIgh9nRAuhWNasBLF7J IqnqV5Hm MBcp8IPkE7MDqVnDcaiRCG/cM9qeznscM9uWQj6SVJOj6vuWnfF7GTZpNyW1iR+X/yyGFCoAdpik1FYc+6h2XL7X6pVOlzGU/+alypFxmr1glt6uXSZ11phHRsFNFUIzc+6ZsK6odIZ6/BOOpPQomRi2v/XWkRdVp6WWf75dTOkctZa1Usyy4O424I5so6Nd36+q8+ae7ctxhLeEz94HKRFPNeuGpZFRtv+m5/36qaoeVtpnr4jq3Y1SGMnwf7YEjkLI3WtukQ40/ZCHdQPJ7jA5OdwsFOzL3ygQYYaXFpKsFUP3YnzBxTtZkv0REB9Y1kQXgz4nswkkoha/+KbvFvbUPF26VZjPyHcYLUlUjbxQZGpvOsIObgkIsBE7wMMukVMac+kjKW5osXuQp/EwTV2h/PlwNy6ix+EJ9EvEGHMTpM/vI+7BoSMdYgpYywEJwfAKt X-Bogosity: Ham, tests=bogofilter, spamicity=0.000000, version=1.2.4 Sender: owner-linux-mm@kvack.org Precedence: bulk X-Loop: owner-majordomo@kvack.org List-ID: List-Subscribe: List-Unsubscribe: From: Alex Williamson With the addition of pfnmap support in vmf_insert_pfn_{pmd,pud}() we can take advantage of PMD and PUD faults to PCI BAR mmaps and create more efficient mappings. PCI BARs are always a power of two and will typically get at least PMD alignment without userspace even trying. Userspace alignment for PUD mappings is also not too difficult. Consolidate faults through a single handler with a new wrapper for standard single page faults. The pre-faulting behavior of commit d71a989cf5d9 ("vfio/pci: Insert full vma on mmap'd MMIO fault") is removed in this refactoring since huge_fault will cover the bulk of the faults and results in more efficient page table usage. We also want to avoid that pre-faulted single page mappings preempt huge page mappings. Cc: kvm@vger.kernel.org Signed-off-by: Alex Williamson Signed-off-by: Peter Xu --- drivers/vfio/pci/vfio_pci_core.c | 60 +++++++++++++++++++++++--------- 1 file changed, 43 insertions(+), 17 deletions(-) diff --git a/drivers/vfio/pci/vfio_pci_core.c b/drivers/vfio/pci/vfio_pci_core.c index ba0ce0075b2f..2d7478e9a62d 100644 --- a/drivers/vfio/pci/vfio_pci_core.c +++ b/drivers/vfio/pci/vfio_pci_core.c @@ -20,6 +20,7 @@ #include #include #include +#include #include #include #include @@ -1657,14 +1658,20 @@ static unsigned long vma_to_pfn(struct vm_area_struct *vma) return (pci_resource_start(vdev->pdev, index) >> PAGE_SHIFT) + pgoff; } -static vm_fault_t vfio_pci_mmap_fault(struct vm_fault *vmf) +static vm_fault_t vfio_pci_mmap_huge_fault(struct vm_fault *vmf, + unsigned int order) { struct vm_area_struct *vma = vmf->vma; struct vfio_pci_core_device *vdev = vma->vm_private_data; unsigned long pfn, pgoff = vmf->pgoff - vma->vm_pgoff; - unsigned long addr = vma->vm_start; vm_fault_t ret = VM_FAULT_SIGBUS; + if (order && (vmf->address & ((PAGE_SIZE << order) - 1) || + vmf->address + (PAGE_SIZE << order) > vma->vm_end)) { + ret = VM_FAULT_FALLBACK; + goto out; + } + pfn = vma_to_pfn(vma); down_read(&vdev->memory_lock); @@ -1672,30 +1679,49 @@ static vm_fault_t vfio_pci_mmap_fault(struct vm_fault *vmf) if (vdev->pm_runtime_engaged || !__vfio_pci_memory_enabled(vdev)) goto out_unlock; - ret = vmf_insert_pfn(vma, vmf->address, pfn + pgoff); - if (ret & VM_FAULT_ERROR) - goto out_unlock; - - /* - * Pre-fault the remainder of the vma, abort further insertions and - * supress error if fault is encountered during pre-fault. - */ - for (; addr < vma->vm_end; addr += PAGE_SIZE, pfn++) { - if (addr == vmf->address) - continue; - - if (vmf_insert_pfn(vma, addr, pfn) & VM_FAULT_ERROR) - break; + switch (order) { + case 0: + ret = vmf_insert_pfn(vma, vmf->address, pfn + pgoff); + break; +#ifdef CONFIG_ARCH_SUPPORTS_PMD_PFNMAP + case PMD_ORDER: + ret = vmf_insert_pfn_pmd(vmf, __pfn_to_pfn_t(pfn + pgoff, + PFN_DEV), false); + break; +#endif +#ifdef CONFIG_ARCH_SUPPORTS_PUD_PFNMAP + case PUD_ORDER: + ret = vmf_insert_pfn_pud(vmf, __pfn_to_pfn_t(pfn + pgoff, + PFN_DEV), false); + break; +#endif + default: + ret = VM_FAULT_FALLBACK; } out_unlock: up_read(&vdev->memory_lock); +out: + dev_dbg_ratelimited(&vdev->pdev->dev, + "%s(,order = %d) BAR %ld page offset 0x%lx: 0x%x\n", + __func__, order, + vma->vm_pgoff >> + (VFIO_PCI_OFFSET_SHIFT - PAGE_SHIFT), + pgoff, (unsigned int)ret); return ret; } +static vm_fault_t vfio_pci_mmap_page_fault(struct vm_fault *vmf) +{ + return vfio_pci_mmap_huge_fault(vmf, 0); +} + static const struct vm_operations_struct vfio_pci_mmap_ops = { - .fault = vfio_pci_mmap_fault, + .fault = vfio_pci_mmap_page_fault, +#ifdef CONFIG_ARCH_SUPPORTS_HUGE_PFNMAP + .huge_fault = vfio_pci_mmap_huge_fault, +#endif }; int vfio_pci_core_mmap(struct vfio_device *core_vdev, struct vm_area_struct *vma)