From patchwork Fri Jun 21 14:24:58 2024 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Peter Xu X-Patchwork-Id: 13707653 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from kanga.kvack.org (kanga.kvack.org [205.233.56.17]) by smtp.lore.kernel.org (Postfix) with ESMTP id 396DDC27C4F for ; Fri, 21 Jun 2024 14:25:16 +0000 (UTC) Received: by kanga.kvack.org (Postfix) id E2E778D0164; Fri, 21 Jun 2024 10:25:14 -0400 (EDT) Received: by kanga.kvack.org (Postfix, from userid 40) id CC9FA8D0174; Fri, 21 Jun 2024 10:25:14 -0400 (EDT) X-Delivered-To: int-list-linux-mm@kvack.org Received: by kanga.kvack.org (Postfix, from userid 63042) id 9BC028D0164; Fri, 21 Jun 2024 10:25:14 -0400 (EDT) X-Delivered-To: linux-mm@kvack.org Received: from relay.hostedemail.com (smtprelay0015.hostedemail.com [216.40.44.15]) by kanga.kvack.org (Postfix) with ESMTP id 7BFA16B0411 for ; Fri, 21 Jun 2024 10:25:14 -0400 (EDT) Received: from smtpin20.hostedemail.com (a10.router.float.18 [10.200.18.1]) by unirelay04.hostedemail.com (Postfix) with ESMTP id 3B8AC1A0FDF for ; Fri, 21 Jun 2024 14:25:14 +0000 (UTC) X-FDA: 82255118148.20.4C8A267 Received: from us-smtp-delivery-124.mimecast.com (us-smtp-delivery-124.mimecast.com [170.10.129.124]) by imf11.hostedemail.com (Postfix) with ESMTP id 2231E4002A for ; Fri, 21 Jun 2024 14:25:11 +0000 (UTC) Authentication-Results: imf11.hostedemail.com; dkim=pass header.d=redhat.com header.s=mimecast20190719 header.b=HrQPIRZR; spf=pass (imf11.hostedemail.com: domain of peterx@redhat.com designates 170.10.129.124 as permitted sender) smtp.mailfrom=peterx@redhat.com; dmarc=pass (policy=none) header.from=redhat.com ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=hostedemail.com; s=arc-20220608; t=1718979904; h=from:from:sender:reply-to:subject:subject:date:date: message-id:message-id:to:to:cc:cc:mime-version:mime-version: content-type:content-type: content-transfer-encoding:content-transfer-encoding: in-reply-to:in-reply-to:references:references:dkim-signature; bh=xrFUBbcc9D0ORnx0qoNz81Y3Ewmb+xGo+hBykZs/BiM=; b=P51yhcKDKU3qg4Tfy0mYjrjsSX7q+RB9M6SkiZP9PZS1E15tbdpLbv0SqFszS7DFLz5D46 pmZCTi3gc1OmsXRjwYrh89MQmKVG12gcvsQI+xvjiqzJX8kpQEiC6g+TmI1nf0bpmmZCVM 9f/baM08QTWXar0rf50oJ8vvYBkxOCk= ARC-Authentication-Results: i=1; imf11.hostedemail.com; dkim=pass header.d=redhat.com header.s=mimecast20190719 header.b=HrQPIRZR; spf=pass (imf11.hostedemail.com: domain of peterx@redhat.com designates 170.10.129.124 as permitted sender) smtp.mailfrom=peterx@redhat.com; dmarc=pass (policy=none) header.from=redhat.com ARC-Seal: i=1; s=arc-20220608; d=hostedemail.com; t=1718979904; a=rsa-sha256; cv=none; b=JnmbSbADSNBqKPw547ca+dC7OlGhkK1KlvgxW4zaSei1aHhDq3ir0ReGS7tcd79pFHwn8p QPlN8EJmdgl2u5B4kHEw/8kdvj8wRG8Vr99+lPD2DP+jFIS8dJTXd8Jea1DbBkHZqABugZ HTGIX6dRopW2JEAysTxs3Ep0u0Nkvfk= DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=redhat.com; s=mimecast20190719; t=1718979911; h=from:from:reply-to:subject:subject:date:date:message-id:message-id: to:to:cc:cc:mime-version:mime-version:content-type:content-type: content-transfer-encoding:content-transfer-encoding: in-reply-to:in-reply-to:references:references; bh=xrFUBbcc9D0ORnx0qoNz81Y3Ewmb+xGo+hBykZs/BiM=; b=HrQPIRZREKg1SmRTJOIjwS2VdprYxMDQ8hApM+uI2q5g0TV0nIGO66wIPF9pClFLg2+1yT DKhTgYddP0G3vfi5hiCkM37A5deBmDoQeNtVwOl5sj9FiVGbh34uKh1YOJmzVjnLVqKuW6 SrWI0zzYmFsJFXmfwlF15NZYk7EadBI= Received: from mail-qk1-f200.google.com (mail-qk1-f200.google.com [209.85.222.200]) by relay.mimecast.com with ESMTP with STARTTLS (version=TLSv1.3, cipher=TLS_AES_256_GCM_SHA384) id us-mta-589-aAj8-o_JNW6Oh-E_aVVPPQ-1; Fri, 21 Jun 2024 10:25:10 -0400 X-MC-Unique: aAj8-o_JNW6Oh-E_aVVPPQ-1 Received: by mail-qk1-f200.google.com with SMTP id af79cd13be357-797a903cf8fso21893585a.2 for ; Fri, 21 Jun 2024 07:25:10 -0700 (PDT) X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20230601; t=1718979909; x=1719584709; h=content-transfer-encoding:mime-version:references:in-reply-to :message-id:date:subject:cc:to:from:x-gm-message-state:from:to:cc :subject:date:message-id:reply-to; bh=xrFUBbcc9D0ORnx0qoNz81Y3Ewmb+xGo+hBykZs/BiM=; b=mFNOpGco82c8ZGKAf5Z6LL/rmGjd/nAzX+9iX+AOseoq6eN+KQ0RQgxPGe5CRfJuKQ nR0U8Dgfbava2PmwEY1o2HPTCnpT0exJ2FPX55flk9w8r9lJ9rLv683OYEkrmbUgtb2q Fi6u/J7exNsIaQ+Y37omS9WIbxn9Uk85edNgAn2HNLLPOhyoexdqoAZGl3dFiosb1yWJ GylnYB5LwYAtoByW+lW4Bznb5e4HsISjP/uLVWKS1VHRyZjn+2M+g/m9c2Ibjit9w3FY ocf8d59sETuGcKhP00ZfA8rufCO6+SJajKD/Lwv/L7OcuhmZKHPXscF5OW95CPZoKG7T VBAQ== X-Forwarded-Encrypted: i=1; AJvYcCV/qKqC086Q3LbFGdjw2ooyAYBroimaZQ2PjbJdniHiXLRr/sSto0vcjbTgfW7m5zpZWVmnRiW+me2YecKY+0vOcrY= X-Gm-Message-State: AOJu0YzTs9sJKOGR9T4iHtXBCR4GJNrMR3ng7xLAwtJirnKVb5nxEmUP wfPjbEy8/O0U7yPlSKVJvxLG11G7Nnmc4GBMiy2AkCucQBiglHSHWsAKuxoBuOI2+gs/ZUv4WL3 Zf2b+fabQNS5B+BFO9jFqY0uKyd92Xg+WoZDV4GTAx0iMsL0v X-Received: by 2002:a05:620a:1a1f:b0:795:c5a1:cbac with SMTP id af79cd13be357-79bb3ee371bmr869907885a.5.1718979909464; Fri, 21 Jun 2024 07:25:09 -0700 (PDT) X-Google-Smtp-Source: AGHT+IHBrJ7N3pmBZU3UlBHH/5YVIoMtOfK+G61+YRt17HXeINLiGOOyCA8EM9L/R3OdwJcsLTRnFw== X-Received: by 2002:a05:620a:1a1f:b0:795:c5a1:cbac with SMTP id af79cd13be357-79bb3ee371bmr869904185a.5.1718979908847; Fri, 21 Jun 2024 07:25:08 -0700 (PDT) Received: from x1n.redhat.com (pool-99-254-121-117.cpe.net.cable.rogers.com. [99.254.121.117]) by smtp.gmail.com with ESMTPSA id af79cd13be357-79bce944cb2sm90564785a.125.2024.06.21.07.25.06 (version=TLS1_3 cipher=TLS_AES_256_GCM_SHA384 bits=256/256); Fri, 21 Jun 2024 07:25:08 -0700 (PDT) From: Peter Xu To: linux-kernel@vger.kernel.org, linux-mm@kvack.org Cc: x86@kernel.org, Borislav Petkov , Dave Jiang , "Kirill A . Shutemov" , Ingo Molnar , Oscar Salvador , peterx@redhat.com, Matthew Wilcox , Vlastimil Babka , Dan Williams , Andrew Morton , Hugh Dickins , Michael Ellerman , Dave Hansen , Thomas Gleixner , linuxppc-dev@lists.ozlabs.org, Christophe Leroy , Rik van Riel , Mel Gorman , "Aneesh Kumar K . V" , Nicholas Piggin , Huang Ying Subject: [PATCH 1/7] mm/dax: Dump start address in fault handler Date: Fri, 21 Jun 2024 10:24:58 -0400 Message-ID: <20240621142504.1940209-2-peterx@redhat.com> X-Mailer: git-send-email 2.45.0 In-Reply-To: <20240621142504.1940209-1-peterx@redhat.com> References: <20240621142504.1940209-1-peterx@redhat.com> MIME-Version: 1.0 X-Mimecast-Spam-Score: 0 X-Mimecast-Originator: redhat.com X-Stat-Signature: cpfc7c1gu56n7x9gar3orw1rae7qrry3 X-Rspam-User: X-Rspamd-Queue-Id: 2231E4002A X-Rspamd-Server: rspam02 X-HE-Tag: 1718979911-963907 X-HE-Meta: U2FsdGVkX1+Nr0MbjmZBdLaTo4djdlWxJy/SrdjPbswIMD8DsHWKAN4jOfKDe7jzBdfmyCoUDg/C8NURX1SzgYigHRbj88FIOLlzUCtrIXqmrnROIk6LmxxVDnzDMW416iymzXJ7gR5fCT5Tk+q0NyVHgstGmxS/lmTPmZq8cWfCPcC/BRMDHzzgUTdNjxKn6ytzuEz009pR7pL4TlMnHbbuiPQ/Yj0pZQvitCd3OJ9cPjl8RaDsuperRGURu13F5w+EXpSdbFu1d8KqSnT9SVMRrM+xj0+LFzt+wAT90NtoxVqc0L+XamdpKT6Ak8V5xn+ERgp1Ti9G13KN/nWxAA+7M5hGOujkltCpJiQrtWBPZe1VpkLkN9MX3UNz7/+JY9e0x5Nc/CaUb3yUfh38mDop02absKq7zZdxkwYH7PqJgPEexvLoTBScS5hSp09j5zuLfg4em08ufPohZZNhoL4IPxD49CaIdn/5FHGjjnqNOghtXqnkOzAVdckm+TsxyolGFeAEi7IKle+/qISrm5/RmpbxhQJ481IJx9vahcISNYAQEDsI01lmldVsJUVEl+TX/4dGCfw7pneSxXMRfOeYAIcgMrymjX+N9YoWUXL2dc4tgLNFJTM3+65kDiAomN0LaBfKxm9TpyeG51xKPSLixrkWk4QvL5d0mf6b3qryqiGYmv/Y6DPd7lFvDZ9KXGAn9UyTdD77fONyDqe2K5TYcHnPiQLC4RtExAZ4tk54Amtg7tUjy6hVwPgpARAmWrQQNo3etJMojLfmmZYPKO50YNHu+ZGQKFpU/QXl30VV5WqvshcGPolJ6x7/ZvO22pRF0X9FVD/LThzMr1qYSGwWD/GrT0n2FMQWggxxfpdlXJUjxH1R+KMgY7w6AekMbKhGlvilvKXXKmochoIT+qiYgLZdeu+sU2yMwpfgYfKkk+ak1OzJTtwKHOXzUnIHVXljVEpez+JcWQc7pda t3Bln3Jh Pkhp5GZ3x/Fb44V1WcUvKM7NV8jpS9vNdAcG1Wb4AbCSQTMH8/sXUw3xG+CYve3AQnYBjVc7JEyAqK8+d7XRuqlOt6wvFV/x1smfAFl0dukRLCiBHaM7fUz9xAmCbe6NA/i9EKjgZ1NmeFt3SUb+DFBWjCTatVhq+X+Xr61n+r+Boo26bE0ZiMjrWEQomA0hHeYbj5H4l+Y87cRLw426xCWbY9n+KtfRT2c+u9BEBoYYCUzInEofLi855fM6+2UzNnBlfH6RIUSVGxv83WctEW6Z+xS/V/3JWfnNfRhPbCs+FG2kOyq3m06f6ydqRWsWgnl8ycC3CYGqCvKKUD+V+D/tZhQ== X-Bogosity: Ham, tests=bogofilter, spamicity=0.000000, version=1.2.4 Sender: owner-linux-mm@kvack.org Precedence: bulk X-Loop: owner-majordomo@kvack.org List-ID: List-Subscribe: List-Unsubscribe: Currently the dax fault handler dumps the vma range when dynamic debugging enabled. That's mostly not useful. Dump the (aligned) address instead with the order info. Signed-off-by: Peter Xu --- drivers/dax/device.c | 6 +++--- 1 file changed, 3 insertions(+), 3 deletions(-) diff --git a/drivers/dax/device.c b/drivers/dax/device.c index eb61598247a9..714174844ca5 100644 --- a/drivers/dax/device.c +++ b/drivers/dax/device.c @@ -235,9 +235,9 @@ static vm_fault_t dev_dax_huge_fault(struct vm_fault *vmf, unsigned int order) int id; struct dev_dax *dev_dax = filp->private_data; - dev_dbg(&dev_dax->dev, "%s: %s (%#lx - %#lx) order:%d\n", current->comm, - (vmf->flags & FAULT_FLAG_WRITE) ? "write" : "read", - vmf->vma->vm_start, vmf->vma->vm_end, order); + dev_dbg(&dev_dax->dev, "%s: op=%s addr=%#lx order=%d\n", current->comm, + (vmf->flags & FAULT_FLAG_WRITE) ? "write" : "read", + vmf->address & ~((1UL << (order + PAGE_SHIFT)) - 1), order); id = dax_read_lock(); if (order == 0) From patchwork Fri Jun 21 14:24:59 2024 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Peter Xu X-Patchwork-Id: 13707655 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from kanga.kvack.org (kanga.kvack.org [205.233.56.17]) by smtp.lore.kernel.org (Postfix) with ESMTP id 15F45C2BBCA for ; Fri, 21 Jun 2024 14:25:23 +0000 (UTC) Received: by kanga.kvack.org (Postfix) id BEC398D0175; Fri, 21 Jun 2024 10:25:19 -0400 (EDT) Received: by kanga.kvack.org (Postfix, from userid 40) id 5C5498D0176; Fri, 21 Jun 2024 10:25:19 -0400 (EDT) X-Delivered-To: int-list-linux-mm@kvack.org Received: by kanga.kvack.org (Postfix, from userid 63042) id 0E75B8D0174; Fri, 21 Jun 2024 10:25:19 -0400 (EDT) X-Delivered-To: linux-mm@kvack.org Received: from relay.hostedemail.com (smtprelay0015.hostedemail.com [216.40.44.15]) by kanga.kvack.org (Postfix) with ESMTP id DF1388D0175 for ; Fri, 21 Jun 2024 10:25:18 -0400 (EDT) Received: from smtpin16.hostedemail.com (a10.router.float.18 [10.200.18.1]) by unirelay02.hostedemail.com (Postfix) with ESMTP id A2F07120F9E for ; Fri, 21 Jun 2024 14:25:18 +0000 (UTC) X-FDA: 82255118316.16.D79EC13 Received: from us-smtp-delivery-124.mimecast.com (us-smtp-delivery-124.mimecast.com [170.10.129.124]) by imf07.hostedemail.com (Postfix) with ESMTP id 3F6D640007 for ; Fri, 21 Jun 2024 14:25:16 +0000 (UTC) Authentication-Results: imf07.hostedemail.com; dkim=pass header.d=redhat.com header.s=mimecast20190719 header.b=BPQNmZsd; spf=pass (imf07.hostedemail.com: domain of peterx@redhat.com designates 170.10.129.124 as permitted sender) smtp.mailfrom=peterx@redhat.com; dmarc=pass (policy=none) header.from=redhat.com ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=hostedemail.com; s=arc-20220608; t=1718979908; h=from:from:sender:reply-to:subject:subject:date:date: message-id:message-id:to:to:cc:cc:mime-version:mime-version: content-type:content-type: content-transfer-encoding:content-transfer-encoding: in-reply-to:in-reply-to:references:references:dkim-signature; bh=MJ/z3oUyfvIT45NLtsICo4unaxlXJrC1yYznR3c5j0M=; b=tZzktPgehqUETGTc+OzwsYiE8UxkkAqp66EtwD2+F3gr0C4nB/Md38VCHYllDnqPN6INob cno5fQRf3GdacRKdlAvJU5ZegWp5saiE20eErgCQtU+9BO8DumRi8xXhInwdvgAjOtXSng OqvWJpEHxoCxaMuZAFO/imtstBo3Jao= ARC-Authentication-Results: i=1; imf07.hostedemail.com; dkim=pass header.d=redhat.com header.s=mimecast20190719 header.b=BPQNmZsd; spf=pass (imf07.hostedemail.com: domain of peterx@redhat.com designates 170.10.129.124 as permitted sender) smtp.mailfrom=peterx@redhat.com; dmarc=pass (policy=none) header.from=redhat.com ARC-Seal: i=1; s=arc-20220608; d=hostedemail.com; t=1718979908; a=rsa-sha256; cv=none; b=3bPgVeT8J+kYO2ckhcilBLpwgpcxcYQqbUB7so8DS3SC6c4y1mPMDkbn7kdr9fkQbyFx5l q9U+o+yBv7OG4ehoQEDuLZQ+M0Z+iXBlsxDAead2hCPS8Bzku+MVg4hJw+ZjL06e9WR3df s+DoBZ7iD93tDp6w6cRHEMMpXuz1/R0= DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=redhat.com; s=mimecast20190719; t=1718979915; h=from:from:reply-to:subject:subject:date:date:message-id:message-id: to:to:cc:cc:mime-version:mime-version:content-type:content-type: content-transfer-encoding:content-transfer-encoding: in-reply-to:in-reply-to:references:references; bh=MJ/z3oUyfvIT45NLtsICo4unaxlXJrC1yYznR3c5j0M=; b=BPQNmZsdO0WoRm/VCXOJpG0k9qIztnE/gmqf0kxL8HNySvuLI0pa6d7kE1UsvUTlV94iU/ h59C2e1LAJOJpwY9HQyF5c0xJjSLuumapYErq8ffRf/+OtAXA6KkRhBjNsJ9n7rtCzbCLn /TbyXGGR6/PMSNj7OER4QiCwNMu47UA= Received: from mail-qk1-f199.google.com (mail-qk1-f199.google.com [209.85.222.199]) by relay.mimecast.com with ESMTP with STARTTLS (version=TLSv1.3, cipher=TLS_AES_256_GCM_SHA384) id us-mta-661-s31wfH44OY-UMNJccsgR6g-1; Fri, 21 Jun 2024 10:25:12 -0400 X-MC-Unique: s31wfH44OY-UMNJccsgR6g-1 Received: by mail-qk1-f199.google.com with SMTP id af79cd13be357-79560b25fc4so14964985a.3 for ; Fri, 21 Jun 2024 07:25:11 -0700 (PDT) X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20230601; t=1718979911; x=1719584711; h=content-transfer-encoding:mime-version:references:in-reply-to :message-id:date:subject:cc:to:from:x-gm-message-state:from:to:cc :subject:date:message-id:reply-to; bh=MJ/z3oUyfvIT45NLtsICo4unaxlXJrC1yYznR3c5j0M=; b=YP/1dobzDciaMSXIzCr1MUs+C08siTwQ5pRLsh2WY23dX/yWAB17CFq9bNLUx1pW+Q MHO1MyywMlqF52B6u5NKb+mUB+z7OxJ2a50AmOrELln/azt6xst63zndceQuTIfm8QjR dCN1hk2StEzOqqipG3YVjMIlD2tvaiP7j3TqeYS4JPU7FMNCVW+vYkjK6hkUFBRLqT3q G7h/TuzeqOAhACxXl59frwRiAHbA7XDcU6PiJ65ZZNC6O2Esak7pZ5eJy9PzqdTVdPxT qXdW4/fMiC3ZbvT4Fx98P0sJAX3U//WgnLHEHtmLMa93pZG0CQyns35X0hunrGZmU/1V bWQQ== X-Forwarded-Encrypted: i=1; AJvYcCVsTnvIFfxWLwZE9UKm+2abPG9ViJJkuFzUFdbn5gfrXNzZFAnk7VBqeZAYXgMvQRrR6HfI45lCfRlrxS4m+0oGjHw= X-Gm-Message-State: AOJu0Yyl4/J6fNCKxPc+4YoL/lZpNwJGH5LqKQSA1fQK0Nwh21WfixDD NEUA9kaakW2Ddaj6LOEne0KJq7n9fabf88cPSQLbfWYr2qYghXhiIMbRCJXQMDMTJknt5F2cYF9 KeHwdetPglv89BsLBFV/L5835Si6+cDeJadp/lBwOPuMs0F1F X-Received: by 2002:a05:620a:4610:b0:79b:be0b:7815 with SMTP id af79cd13be357-79bbe0b7c0fmr735910685a.1.1718979911259; Fri, 21 Jun 2024 07:25:11 -0700 (PDT) X-Google-Smtp-Source: AGHT+IE1JVg4Gm5x1JzvXAIal9/bHLuF/aojpegzJFLEbRVA7h1M8KnWp/bZUR1LNlUwbmLSWaEiNQ== X-Received: by 2002:a05:620a:4610:b0:79b:be0b:7815 with SMTP id af79cd13be357-79bbe0b7c0fmr735907585a.1.1718979910823; Fri, 21 Jun 2024 07:25:10 -0700 (PDT) Received: from x1n.redhat.com (pool-99-254-121-117.cpe.net.cable.rogers.com. [99.254.121.117]) by smtp.gmail.com with ESMTPSA id af79cd13be357-79bce944cb2sm90564785a.125.2024.06.21.07.25.09 (version=TLS1_3 cipher=TLS_AES_256_GCM_SHA384 bits=256/256); Fri, 21 Jun 2024 07:25:10 -0700 (PDT) From: Peter Xu To: linux-kernel@vger.kernel.org, linux-mm@kvack.org Cc: x86@kernel.org, Borislav Petkov , Dave Jiang , "Kirill A . Shutemov" , Ingo Molnar , Oscar Salvador , peterx@redhat.com, Matthew Wilcox , Vlastimil Babka , Dan Williams , Andrew Morton , Hugh Dickins , Michael Ellerman , Dave Hansen , Thomas Gleixner , linuxppc-dev@lists.ozlabs.org, Christophe Leroy , Rik van Riel , Mel Gorman , "Aneesh Kumar K . V" , Nicholas Piggin , Huang Ying , Alex Thorlton Subject: [PATCH 2/7] mm/mprotect: Remove NUMA_HUGE_PTE_UPDATES Date: Fri, 21 Jun 2024 10:24:59 -0400 Message-ID: <20240621142504.1940209-3-peterx@redhat.com> X-Mailer: git-send-email 2.45.0 In-Reply-To: <20240621142504.1940209-1-peterx@redhat.com> References: <20240621142504.1940209-1-peterx@redhat.com> MIME-Version: 1.0 X-Mimecast-Spam-Score: 0 X-Mimecast-Originator: redhat.com X-Stat-Signature: afmfdohchyrcfndd6jthg4eutpp8mddq X-Rspam-User: X-Rspamd-Queue-Id: 3F6D640007 X-Rspamd-Server: rspam02 X-HE-Tag: 1718979916-271813 X-HE-Meta: U2FsdGVkX1/e9U4BqnN2V1434iGHpoqiNHw9OjFxXrqpWlgw+iqLASQr7edltv9CE45X1UHpEGIFgjcXuVAQcTXIiG0EfzInOf4D0ABfLPSFE7No4IV/scoOnrbZVSnwlAsAvOD5A47Ib6rp2cgzxhefZiAMj9rBombVUKisXB5D5+pzLvq022w/WCaOiJa3qET8v/ON/1FE0B1o2P0gFNEEps2Wdh0y8zbwCI7e/gwCFen7VFjSQE/5zi2gKgM0g4790DTYrnFiAcCb+Cf/CIQqyIQ5IguTIpYPcrdU4dVSpBFoC8Hau9UCg+b7eD9qbDKg241vRGFd8djUTdUhVsuI2C+48clZZasQh7GyAU98GvGbP5qNBPJgsKweubGPwWGSTRgN0ctXpLUFCB9FRi6LEM7PxtGmZY5LvJSKq/6NUHkNqcQ2b2f8+wKx0V1gcgK+Imx4WDaf6/gu/yQ7KM0behJu42rqoa10Q6xHgO6iMzqNKPtKNe50omEmIJQ+Vak0juaJBFGEjg5j43c7wUQPZLQL8mhx/t9BMm0sr0UDlIxHRaPVHnmHJzHzHqnLynU0xcLS5lj5Wdc9qOPXG5zF9mODW1qDAF8+jdbJhq8e7Ihean27JS5QSSRQtJKl0a+VQijP/r8ZDaPgwIFNIbdSLEcjc2vzsm/yAOSGll15KLqftfNOQv3tTlC0sMV3P7MjaYDuvwIZhNWdE/Xr6jXIanocwb0X/bC6hTMHz0SRXtSjJe72ysN2Zmn0jp183li1qZoh3iaRLqDDoQpePVOoQ1aJfY4uojWT5xehcz/UdwSycBiRGQYuhf1WbGnFsF8ZvD2TZAqA68KWvVj3BV0MKfeJZZ3UMm0SHXgLcc77INMUSlEyZHxI/43BfqlnMkbDVXLW+6ge8j+rOplf/jDQbBAMi48TsskqJU4hIDwxGKXdKmbFGOVFelLvd9aToE4cS491ksBDwhRqcuQ ekuBNmMZ lSS1BnnTerxUMO/UZh04QKJWcg6mxKYNJHKKb3Zzzp6d80f7MlkC+0QsVLRMN7JjPP46kJ75jARr0VFMTqLqrPpxqw94jvOmaiDLyne27ZaSS6f5PjhsjbwB04bI8URr8XYg//d4aqqJnwJ/TxLgI/yoMmpXbF5YXxk+ZVfXxxE52w0zI0b60nIj1wdtxq5VI2Z3i9a/eW/7F5Ihzpoe93rDUo0Z7Skn5kuOtNTb18vJDboB2eF5ZJhG+G6HBadSEPa4RpXY/qkuY8m9loeMIYBieq869KsOfNKcfSUojaFem+yZMe5nAlTuY7SR8YnPQHcaq1E8snkJCVgG+aA+OnE30F7yRQEMhmKLnKOOQAe4jP5CTnxsG9rBnqpPZlsoYOyJuU2TrisMta4bWrzZP+xFbaL/FLSuoGczseCRv47t27QfnImzN/bT64M/k/MvixRHZuKjJizb0kwisHNJFg/PYjQQhHReHSkhgdmOZGtopRiX08JQ7zaHSUx1OfqYYMJskLmxmcr6OPMeIDwbFhAMuyIS6yQs/Iv0eUkHfmfn2rnrrBt0Wm9ym8F1rDgQ24ysfZj7Z6UBFkCsabp1sTOv0GQ== X-Bogosity: Ham, tests=bogofilter, spamicity=0.000000, version=1.2.4 Sender: owner-linux-mm@kvack.org Precedence: bulk X-Loop: owner-majordomo@kvack.org List-ID: List-Subscribe: List-Unsubscribe: In 2013, commit 72403b4a0fbd ("mm: numa: return the number of base pages altered by protection changes") introduced "numa_huge_pte_updates" vmstat entry, trying to capture how many huge ptes (in reality, PMD thps at that time) are marked by NUMA balancing. This patch proposes to remove it for some reasons. Firstly, the name is misleading. We can have more than one way to have a "huge pte" at least nowadays, and that's also the major goal of this patch, where it paves way for PUD handling in change protection code paths. PUDs are coming not only for dax (which has already came and yet broken..), but also for pfnmaps and hugetlb pages. The name will simply stop making sense when PUD will start to be involved in mprotect() world. It'll also make it not reasonable either if we boost the counter for both pmd/puds. In short, current accounting won't be right when PUD comes, so the scheme was only suitable at that point in time where PUD wasn't even possible. Secondly, the accounting was simply not right from the start as long as it was also affected by other call sites besides NUMA. mprotect() is one, while userfaultfd-wp also leverages change protection path to modify pgtables. If it wants to do right it needs to check the caller but it never did; at least mprotect() should be there even in 2013. It gives me the impression that nobody is seriously using this field, and it's also impossible to be serious. We may want to do it right if any NUMA developers would like it to exist, but we should do that with all above resolved, on both considering PUDs, but also on correct accountings. That should be able to be done on top when there's a real need of such. Cc: Huang Ying Cc: Mel Gorman Cc: Alex Thorlton Cc: Rik van Riel Signed-off-by: Peter Xu --- include/linux/vm_event_item.h | 1 - mm/mprotect.c | 8 +------- mm/vmstat.c | 1 - 3 files changed, 1 insertion(+), 9 deletions(-) diff --git a/include/linux/vm_event_item.h b/include/linux/vm_event_item.h index 73fa5fbf33a3..a76b75d6a8f4 100644 --- a/include/linux/vm_event_item.h +++ b/include/linux/vm_event_item.h @@ -59,7 +59,6 @@ enum vm_event_item { PGPGIN, PGPGOUT, PSWPIN, PSWPOUT, OOM_KILL, #ifdef CONFIG_NUMA_BALANCING NUMA_PTE_UPDATES, - NUMA_HUGE_PTE_UPDATES, NUMA_HINT_FAULTS, NUMA_HINT_FAULTS_LOCAL, NUMA_PAGE_MIGRATE, diff --git a/mm/mprotect.c b/mm/mprotect.c index 222ab434da54..21172272695e 100644 --- a/mm/mprotect.c +++ b/mm/mprotect.c @@ -363,7 +363,6 @@ static inline long change_pmd_range(struct mmu_gather *tlb, pmd_t *pmd; unsigned long next; long pages = 0; - unsigned long nr_huge_updates = 0; struct mmu_notifier_range range; range.start = 0; @@ -411,11 +410,8 @@ static inline long change_pmd_range(struct mmu_gather *tlb, ret = change_huge_pmd(tlb, vma, pmd, addr, newprot, cp_flags); if (ret) { - if (ret == HPAGE_PMD_NR) { + if (ret == HPAGE_PMD_NR) pages += HPAGE_PMD_NR; - nr_huge_updates++; - } - /* huge pmd was handled */ goto next; } @@ -435,8 +431,6 @@ static inline long change_pmd_range(struct mmu_gather *tlb, if (range.start) mmu_notifier_invalidate_range_end(&range); - if (nr_huge_updates) - count_vm_numa_events(NUMA_HUGE_PTE_UPDATES, nr_huge_updates); return pages; } diff --git a/mm/vmstat.c b/mm/vmstat.c index 6e3347789eb2..be774893f69e 100644 --- a/mm/vmstat.c +++ b/mm/vmstat.c @@ -1313,7 +1313,6 @@ const char * const vmstat_text[] = { #ifdef CONFIG_NUMA_BALANCING "numa_pte_updates", - "numa_huge_pte_updates", "numa_hint_faults", "numa_hint_faults_local", "numa_pages_migrated", From patchwork Fri Jun 21 14:25:00 2024 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Peter Xu X-Patchwork-Id: 13707654 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from kanga.kvack.org (kanga.kvack.org [205.233.56.17]) by smtp.lore.kernel.org (Postfix) with ESMTP id 4A711C27C4F for ; Fri, 21 Jun 2024 14:25:20 +0000 (UTC) Received: by kanga.kvack.org (Postfix) id 622F58D0174; Fri, 21 Jun 2024 10:25:19 -0400 (EDT) Received: by kanga.kvack.org (Postfix, from userid 40) id 32C708D0175; Fri, 21 Jun 2024 10:25:19 -0400 (EDT) X-Delivered-To: int-list-linux-mm@kvack.org Received: by kanga.kvack.org (Postfix, from userid 63042) id F3CBC8D0176; Fri, 21 Jun 2024 10:25:18 -0400 (EDT) X-Delivered-To: linux-mm@kvack.org Received: from relay.hostedemail.com (smtprelay0012.hostedemail.com [216.40.44.12]) by kanga.kvack.org (Postfix) with ESMTP id D14D18D0174 for ; Fri, 21 Jun 2024 10:25:18 -0400 (EDT) Received: from smtpin04.hostedemail.com (a10.router.float.18 [10.200.18.1]) by unirelay07.hostedemail.com (Postfix) with ESMTP id 85854160F86 for ; Fri, 21 Jun 2024 14:25:18 +0000 (UTC) X-FDA: 82255118316.04.A85EDF2 Received: from us-smtp-delivery-124.mimecast.com (us-smtp-delivery-124.mimecast.com [170.10.129.124]) by imf03.hostedemail.com (Postfix) with ESMTP id 660E42000C for ; Fri, 21 Jun 2024 14:25:16 +0000 (UTC) Authentication-Results: imf03.hostedemail.com; dkim=pass header.d=redhat.com header.s=mimecast20190719 header.b=A4XGJQWf; spf=pass (imf03.hostedemail.com: domain of peterx@redhat.com designates 170.10.129.124 as permitted sender) smtp.mailfrom=peterx@redhat.com; dmarc=pass (policy=none) header.from=redhat.com ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=hostedemail.com; s=arc-20220608; t=1718979908; h=from:from:sender:reply-to:subject:subject:date:date: message-id:message-id:to:to:cc:cc:mime-version:mime-version: content-type:content-type: content-transfer-encoding:content-transfer-encoding: in-reply-to:in-reply-to:references:references:dkim-signature; bh=VpPwnVRxGccST937wjacfAT3ndiJzOREHKlh4Jm8zSE=; b=iBb8I2lmDT9WwVPda/2HvzMEZGB7JJAXLDpH36J5OJOFhX6BB+4hjGqdgIz3TIzrTlVgnk Wv/SbLiWPA48Mqj7qY9XAw3NRnfmD3ZBaY8r6IhTriiZRJ2MYwaU8p5ksSfHtUQwUU2M3Z qfQDbEd6iQ2af3trBZsJCxrt9WWBCWw= ARC-Authentication-Results: i=1; imf03.hostedemail.com; dkim=pass header.d=redhat.com header.s=mimecast20190719 header.b=A4XGJQWf; spf=pass (imf03.hostedemail.com: domain of peterx@redhat.com designates 170.10.129.124 as permitted sender) smtp.mailfrom=peterx@redhat.com; dmarc=pass (policy=none) header.from=redhat.com ARC-Seal: i=1; s=arc-20220608; d=hostedemail.com; t=1718979908; a=rsa-sha256; cv=none; b=ZGFWoTmS0nuUC+NbyECZz60hOV5CdvrIFvyF6uxjXiSxNc0sppo3JLnPAKJ8eTt+zO33S5 c0xq/HakuAa1o4DiTqj/1F+1ZcSqn0ULZm2AV1Mh0b+mfEDdHA0vykWsoc9vpNJukswIyw wqFVjopUzt7UH8kmxPnwXTDfJyXZjug= DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=redhat.com; s=mimecast20190719; t=1718979915; h=from:from:reply-to:subject:subject:date:date:message-id:message-id: to:to:cc:cc:mime-version:mime-version:content-type:content-type: content-transfer-encoding:content-transfer-encoding: in-reply-to:in-reply-to:references:references; bh=VpPwnVRxGccST937wjacfAT3ndiJzOREHKlh4Jm8zSE=; b=A4XGJQWf1W3bYUzhFwiq5AApcOb+kPKpusr4oguxDixyLpSw2lfhCJYz26/lsriMvP5aYK GcG7WNAqaaeDdTGmswkUaRJ3ZGpJ8DiQ2NPGpaQdKZYHKQgw9Ngnz1xTORWtRVfuS2Y0l0 ZA/nSt6B4+eyKYgSuQorh9/hedpqX+4= Received: from mail-ot1-f69.google.com (mail-ot1-f69.google.com [209.85.210.69]) by relay.mimecast.com with ESMTP with STARTTLS (version=TLSv1.3, cipher=TLS_AES_256_GCM_SHA384) id us-mta-609-FxUGhy4hPka5dWdSzub6Aw-1; Fri, 21 Jun 2024 10:25:14 -0400 X-MC-Unique: FxUGhy4hPka5dWdSzub6Aw-1 Received: by mail-ot1-f69.google.com with SMTP id 46e09a7af769-6f961a1baeeso476272a34.2 for ; Fri, 21 Jun 2024 07:25:14 -0700 (PDT) X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20230601; t=1718979913; x=1719584713; h=content-transfer-encoding:mime-version:references:in-reply-to :message-id:date:subject:cc:to:from:x-gm-message-state:from:to:cc :subject:date:message-id:reply-to; bh=VpPwnVRxGccST937wjacfAT3ndiJzOREHKlh4Jm8zSE=; b=gDky9othnBYac4bM76dmsykYpwjC4nqhzhkqfpOspr4T3y4ql9O9p5ICMdY01kOjzl SPjUQrebvyJ4wbrf+nIYNkpk9iXHvt6lkUdRiT9laQmNnRH/IRsICgS69vBoXwEOpHq+ 2nHI5CyYxP10YdrnnrULQIPH6R5XeCCTIA+t+g9H50g2+ctcMtLCPgakWdxGK4d9uY8y /KxCGBOEAOxQ3zBM3BmBai4Gj0gM0mm5YwnpyaM7G/92T+RnpY+aWtgQzQ248ZPfHJ1J cxd6UBMj73ZuCaPD6ZJWG2qqytpCT4mclC7FV3VaQdZy3J2LQDvlrH2aeX5VoO1k2E44 F/TA== X-Forwarded-Encrypted: i=1; AJvYcCWCCwVyLxV63HXTw2Ng1HssI3GaFpPINzXVLnCDt1x4X/iKGAMjGjr3aWU6nUDHNkuf50/r5dFHsAnETe4aq1RO/po= X-Gm-Message-State: AOJu0Ywh+WDbc7tyVSulb71Kw6eRQIeQZnutzPBZiRfpZtI09RJITnCc gZza+v2NTJiZJnAdgZvPYhZkwjMja1lv9+m4q0ZlqWxhwgTvvZ0ZgR9ndxSoSOedi7qIhbUJgDY DNCo1InU/np5EYIUkOs4OxuCwg5G21K9jQVfaAbOtX2f6VYQC X-Received: by 2002:a4a:c60f:0:b0:5bd:af39:c9d9 with SMTP id 006d021491bc7-5c1ad9093ebmr9601658eaf.0.1718979913413; Fri, 21 Jun 2024 07:25:13 -0700 (PDT) X-Google-Smtp-Source: AGHT+IFna6HwoEVcDVzacrmJp70i2Ye9mZl03svl3PALStAHNV8VRqUBmNMnAivUzvRTiL6Q97gHJQ== X-Received: by 2002:a4a:c60f:0:b0:5bd:af39:c9d9 with SMTP id 006d021491bc7-5c1ad9093ebmr9601627eaf.0.1718979912731; Fri, 21 Jun 2024 07:25:12 -0700 (PDT) Received: from x1n.redhat.com (pool-99-254-121-117.cpe.net.cable.rogers.com. [99.254.121.117]) by smtp.gmail.com with ESMTPSA id af79cd13be357-79bce944cb2sm90564785a.125.2024.06.21.07.25.11 (version=TLS1_3 cipher=TLS_AES_256_GCM_SHA384 bits=256/256); Fri, 21 Jun 2024 07:25:12 -0700 (PDT) From: Peter Xu To: linux-kernel@vger.kernel.org, linux-mm@kvack.org Cc: x86@kernel.org, Borislav Petkov , Dave Jiang , "Kirill A . Shutemov" , Ingo Molnar , Oscar Salvador , peterx@redhat.com, Matthew Wilcox , Vlastimil Babka , Dan Williams , Andrew Morton , Hugh Dickins , Michael Ellerman , Dave Hansen , Thomas Gleixner , linuxppc-dev@lists.ozlabs.org, Christophe Leroy , Rik van Riel , Mel Gorman , "Aneesh Kumar K . V" , Nicholas Piggin , Huang Ying , kvm@vger.kernel.org, Sean Christopherson , Paolo Bonzini , David Rientjes Subject: [PATCH 3/7] mm/mprotect: Push mmu notifier to PUDs Date: Fri, 21 Jun 2024 10:25:00 -0400 Message-ID: <20240621142504.1940209-4-peterx@redhat.com> X-Mailer: git-send-email 2.45.0 In-Reply-To: <20240621142504.1940209-1-peterx@redhat.com> References: <20240621142504.1940209-1-peterx@redhat.com> MIME-Version: 1.0 X-Mimecast-Spam-Score: 0 X-Mimecast-Originator: redhat.com X-Stat-Signature: mf6kqjp885p81rmsyb1kx113z3toegzh X-Rspam-User: X-Rspamd-Queue-Id: 660E42000C X-Rspamd-Server: rspam02 X-HE-Tag: 1718979916-768109 X-HE-Meta: U2FsdGVkX18A2b6I7cS+p4axSYlPNmFIFxOgbWzxAEe6a8Kgd+6xCCuQvdQgxQkfLr05IUMBO2E/Lik5i1eN0XBF5bXvRhj8nhaGr/HJCgwrGE3FhUj9mOJK2kphu6ikHY2GE/GkKgrKCDsMdei6wq/ixNFJzY5HGw+w65y/SkjQgaMp+W4NFaGAbA/I4EIXusj+yxN6FV7muSsbfDrA/l49V1vR8+5tR4yU4Oq8+XxUNNt2GqkkW+ZtV6wn1Kuum3J8ztZyTbK7dxLTz0MKwr8WMjsLHPRSIdxVcw1FAaYSuSarNAw9HInOiVEhqAxL/Q1sXaTJPYqkm+BpRkzGZpF+BV96VIUXMjD/Bno2TKBiaxzc8/Wu/S3gzMsbIyh6XWYNm8+v9E3hUthNqI4q2HXpW1lca1z3vsvzWx/Csh9iK1pTCXdxo+1lHnR1TArPpnlTr/8oCv6TbEv4VhZjQyziwrP5FCdWJYgHqVzPXGZeHgLQBXk5jFKXzxDy71ue5UDyf9YtKk8iFdT9GBxBLKQQm39nLarwCt7l1hRi0E3cJxpD/S4jtLpS4OyTJQUrG0XTH+kLAqS87iYKdJtPJPL30KKh4YF3lRD3rOlfZRtvzNGkMNJPL79nOVebOZermN5ckGZegGb8++sb/cKqV8ulgIV1SJJ1HXMeD7nZylnNlZCxLqvLLjMnWyW6N6s6ddZWuHcMkUJF2E1VdGmie/hMPwKO5FEzp6p71ER8cr+W9msPiaCbzvW/Yb4yyRc0/iuVUh/wkvSqVOvo716lDisXXNBoJ0G6umKWxRkSKRG/akQ3mQCzXkHEurDY7Q/2VuF9WaYS5s6dsueT49q3Ix48Nb5AdCxL4rQvP6V607+ux/wxlzqHeqdJhI27GoMkyV7RJy2OEkIfd/v6ScrZ3PwUhSYqf49mbrinnjTbxG/KHxY0qRdT3P7YkaAreLIwU/X9aC8UqUJ/VIRTlF9 KxKpmDIB iLzCa/+HukXSthAEodIT0L6HaiObQi74HNWhmKGbw02N2/vt+dUwBt3aLi3rbgZrT92adaqCddrCFHJSsTkOkRvV4Wt1Md+jm1iD2CxOxHhGq9ietZ6GDUkXWzsKNqXIhgBDYMA9bWHBNmm0Crkw/PoWIqLzscUswAPma3wZrXwWJf5V49k4zcvcov/QiubMxWm2vvRYEmunKKZIGOuCuZ0fkhN5WNPHS4F/USfi7E3UwjW+yBDvCpC/veqhEvg3VzIF2wROK4jeEpavRrg77gZ0TnOaOF8p3eZJ+VBIO9SavAIB56Vb7bNSZua7Jfs7TeP31DAaXRzrcuHFs1+PV9WCYkrGulq66JOuyO99GVNRquYdasxsd43n3aCyOQZ2eX7WctNuq0s79go/Xg6/U8tMKv+31d5qpX8oMfRhMpSVck/gJeYgdS+MsSqz8s/SIspmZJ+9VfG1oL/YKV6Vrg+9iTHL1x1wZGQt9iQB2OhqRszvIhqjNQUCdQQlJ2IyHZ77Dugl0KNDXBhuulWkxtHmSl6JcWtAmWAR0jFUvwJzGagWKzBLXMZAfn/ZdZDa/Zm1w X-Bogosity: Ham, tests=bogofilter, spamicity=0.000000, version=1.2.4 Sender: owner-linux-mm@kvack.org Precedence: bulk X-Loop: owner-majordomo@kvack.org List-ID: List-Subscribe: List-Unsubscribe: mprotect() does mmu notifiers in PMD levels. It's there since 2014 of commit a5338093bfb4 ("mm: move mmu notifier call from change_protection to change_pmd_range"). At that time, the issue was that NUMA balancing can be applied on a huge range of VM memory, even if nothing was populated. The notification can be avoided in this case if no valid pmd detected, which includes either THP or a PTE pgtable page. Now to pave way for PUD handling, this isn't enough. We need to generate mmu notifications even on PUD entries properly. mprotect() is currently broken on PUD (e.g., one can easily trigger kernel error with dax 1G mappings already), this is the start to fix it. To fix that, this patch proposes to push such notifications to the PUD layers. There is risk on regressing the problem Rik wanted to resolve before, but I think it shouldn't really happen, and I still chose this solution because of a few reasons: 1) Consider a large VM that should definitely contain more than GBs of memory, it's highly likely that PUDs are also none. In this case there will have no regression. 2) KVM has evolved a lot over the years to get rid of rmap walks, which might be the major cause of the previous soft-lockup. At least TDP MMU already got rid of rmap as long as not nested (which should be the major use case, IIUC), then the TDP MMU pgtable walker will simply see empty VM pgtable (e.g. EPT on x86), the invalidation of a full empty region in most cases could be pretty fast now, comparing to 2014. 3) KVM has explicit code paths now to even give way for mmu notifiers just like this one, e.g. in commit d02c357e5bfa ("KVM: x86/mmu: Retry fault before acquiring mmu_lock if mapping is changing"). It'll also avoid contentions that may also contribute to a soft-lockup. 4) Stick with PMD layer simply don't work when PUD is there... We need one way or another to fix PUD mappings on mprotect(). Pushing it to PUD should be the safest approach as of now, e.g. there's yet no sign of huge P4D coming on any known archs. Cc: kvm@vger.kernel.org Cc: Sean Christopherson Cc: Paolo Bonzini Cc: David Rientjes Cc: Rik van Riel Signed-off-by: Peter Xu --- mm/mprotect.c | 26 ++++++++++++-------------- 1 file changed, 12 insertions(+), 14 deletions(-) diff --git a/mm/mprotect.c b/mm/mprotect.c index 21172272695e..fb8bf3ff7cd9 100644 --- a/mm/mprotect.c +++ b/mm/mprotect.c @@ -363,9 +363,6 @@ static inline long change_pmd_range(struct mmu_gather *tlb, pmd_t *pmd; unsigned long next; long pages = 0; - struct mmu_notifier_range range; - - range.start = 0; pmd = pmd_offset(pud, addr); do { @@ -383,14 +380,6 @@ static inline long change_pmd_range(struct mmu_gather *tlb, if (pmd_none(*pmd)) goto next; - /* invoke the mmu notifier if the pmd is populated */ - if (!range.start) { - mmu_notifier_range_init(&range, - MMU_NOTIFY_PROTECTION_VMA, 0, - vma->vm_mm, addr, end); - mmu_notifier_invalidate_range_start(&range); - } - _pmd = pmdp_get_lockless(pmd); if (is_swap_pmd(_pmd) || pmd_trans_huge(_pmd) || pmd_devmap(_pmd)) { if ((next - addr != HPAGE_PMD_SIZE) || @@ -428,9 +417,6 @@ static inline long change_pmd_range(struct mmu_gather *tlb, cond_resched(); } while (pmd++, addr = next, addr != end); - if (range.start) - mmu_notifier_invalidate_range_end(&range); - return pages; } @@ -438,10 +424,13 @@ static inline long change_pud_range(struct mmu_gather *tlb, struct vm_area_struct *vma, p4d_t *p4d, unsigned long addr, unsigned long end, pgprot_t newprot, unsigned long cp_flags) { + struct mmu_notifier_range range; pud_t *pud; unsigned long next; long pages = 0, ret; + range.start = 0; + pud = pud_offset(p4d, addr); do { next = pud_addr_end(addr, end); @@ -450,10 +439,19 @@ static inline long change_pud_range(struct mmu_gather *tlb, return ret; if (pud_none_or_clear_bad(pud)) continue; + if (!range.start) { + mmu_notifier_range_init(&range, + MMU_NOTIFY_PROTECTION_VMA, 0, + vma->vm_mm, addr, end); + mmu_notifier_invalidate_range_start(&range); + } pages += change_pmd_range(tlb, vma, pud, addr, next, newprot, cp_flags); } while (pud++, addr = next, addr != end); + if (range.start) + mmu_notifier_invalidate_range_end(&range); + return pages; } From patchwork Fri Jun 21 14:25:01 2024 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Peter Xu X-Patchwork-Id: 13707656 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from kanga.kvack.org (kanga.kvack.org [205.233.56.17]) by smtp.lore.kernel.org (Postfix) with ESMTP id A6037C27C4F for ; Fri, 21 Jun 2024 14:25:26 +0000 (UTC) Received: by kanga.kvack.org (Postfix) id D44998D0177; Fri, 21 Jun 2024 10:25:20 -0400 (EDT) Received: by kanga.kvack.org (Postfix, from userid 40) id CF7158D0176; Fri, 21 Jun 2024 10:25:20 -0400 (EDT) X-Delivered-To: int-list-linux-mm@kvack.org Received: by kanga.kvack.org (Postfix, from userid 63042) id AF9B38D0177; Fri, 21 Jun 2024 10:25:20 -0400 (EDT) X-Delivered-To: linux-mm@kvack.org Received: from relay.hostedemail.com (smtprelay0012.hostedemail.com [216.40.44.12]) by kanga.kvack.org (Postfix) with ESMTP id 84A488D0176 for ; Fri, 21 Jun 2024 10:25:20 -0400 (EDT) Received: from smtpin05.hostedemail.com (a10.router.float.18 [10.200.18.1]) by unirelay02.hostedemail.com (Postfix) with ESMTP id 4193A120F2E for ; Fri, 21 Jun 2024 14:25:20 +0000 (UTC) X-FDA: 82255118400.05.78BC067 Received: from us-smtp-delivery-124.mimecast.com (us-smtp-delivery-124.mimecast.com [170.10.129.124]) by imf23.hostedemail.com (Postfix) with ESMTP id 238B214001F for ; Fri, 21 Jun 2024 14:25:17 +0000 (UTC) Authentication-Results: imf23.hostedemail.com; dkim=pass header.d=redhat.com header.s=mimecast20190719 header.b=OBFvTQTp; spf=pass (imf23.hostedemail.com: domain of peterx@redhat.com designates 170.10.129.124 as permitted sender) smtp.mailfrom=peterx@redhat.com; dmarc=pass (policy=none) header.from=redhat.com ARC-Seal: i=1; s=arc-20220608; d=hostedemail.com; t=1718979912; a=rsa-sha256; cv=none; b=JdBwcaGI/hHRZItbYcADWrKW7XGcsGJUbqdpOOQtsU4zx3SNE5DMg1+cJ2Sx4mhshrMsvd 0twaUmI7bmmkR8FUQrh0MqZhyPxGuIXdQujz5lNKthhGXAHZWLlYRsAQzuEWmmlcWtIFnn pvBtElao8xle1QmBUwdTDrsbKmUetoQ= ARC-Authentication-Results: i=1; imf23.hostedemail.com; dkim=pass header.d=redhat.com header.s=mimecast20190719 header.b=OBFvTQTp; spf=pass (imf23.hostedemail.com: domain of peterx@redhat.com designates 170.10.129.124 as permitted sender) smtp.mailfrom=peterx@redhat.com; dmarc=pass (policy=none) header.from=redhat.com ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=hostedemail.com; s=arc-20220608; t=1718979912; h=from:from:sender:reply-to:subject:subject:date:date: message-id:message-id:to:to:cc:cc:mime-version:mime-version: content-type:content-type: content-transfer-encoding:content-transfer-encoding: in-reply-to:in-reply-to:references:references:dkim-signature; bh=sMhnR88Gj51dgjDcNyzJVzhu8osLx9zEmj3hidZ3w4E=; b=2lm+IHoslkcwDRkwQOIUINmgt/3p061JnH5B9g4xfviqums/J2jSkGsWIuLydAVbtdyI8A 8Fn86jCaDAdtCyrnpu0tQOAsMxcCyg5eXi+5eOt0U/RrKUAY6ol7dhql5pWYrRc8/gfUWL zh7K/zi2m+XqHYjF9TbPCKvAw0nJsQw= DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=redhat.com; s=mimecast20190719; t=1718979917; h=from:from:reply-to:subject:subject:date:date:message-id:message-id: to:to:cc:cc:mime-version:mime-version:content-type:content-type: content-transfer-encoding:content-transfer-encoding: in-reply-to:in-reply-to:references:references; bh=sMhnR88Gj51dgjDcNyzJVzhu8osLx9zEmj3hidZ3w4E=; b=OBFvTQTp8UEOINw9sBFyJdIdvS7mNXFR8yPBYqkPn4GR8qYbtUgKMYXVd5k5BZo2qtWXbh XdSsjI/b5v9glHJGYIJIEPLfchBzP4HglAGfcLtk3RFKnhpENWxhirtDR6A0lFxKsKbgrm 1I5Yqb8cM8dfoI5X/ch7O9Y/qjcwczw= Received: from mail-qk1-f198.google.com (mail-qk1-f198.google.com [209.85.222.198]) by relay.mimecast.com with ESMTP with STARTTLS (version=TLSv1.3, cipher=TLS_AES_256_GCM_SHA384) id us-mta-517-aIbT7vPTMzKis3si18EI7w-1; Fri, 21 Jun 2024 10:25:16 -0400 X-MC-Unique: aIbT7vPTMzKis3si18EI7w-1 Received: by mail-qk1-f198.google.com with SMTP id af79cd13be357-797cde3c2f8so18604185a.1 for ; Fri, 21 Jun 2024 07:25:16 -0700 (PDT) X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20230601; t=1718979916; x=1719584716; h=content-transfer-encoding:mime-version:references:in-reply-to :message-id:date:subject:cc:to:from:x-gm-message-state:from:to:cc :subject:date:message-id:reply-to; bh=sMhnR88Gj51dgjDcNyzJVzhu8osLx9zEmj3hidZ3w4E=; b=kfJXHj+NWEw5QRZud7+n6yHMhBkB+xJajqybhJCbv5KRQUpDpDSt91kv7u8yNV7p2N Q9yFL3TM/1r96LIM+YKotHxZpQzgpAfZXq6gPwVwOX9PRjwOzJ1hY6Wx2H+2Um4CShyo 9XE8JBNGaC7yrhB9zw+ueVsWLb6eO+XomuEeKqzGFu+1D6d38supIjlajvEl6yDu2QfV ePJfCrBnuB/pqskEFwP/96OwbFbXRaRCVJQ+FNNzo+2Mf5eQQIhhmjvWWM1jaAfOItis glHZ14W3I7uM7uK4O0egZm255YlNfQWf+W8TA1MSgpb3E6mwI84E7/OzK9EuI1u4jzkk mfMA== X-Forwarded-Encrypted: i=1; AJvYcCV/VhQ+wiSCx2Ca+pEX4E2Ohvud5b3bEwoCkc38Md1JEvqs0Jh/YSsVRMz250/YTvxDzmFOKJt2pNBNu/gixkLLrSM= X-Gm-Message-State: AOJu0YzcvAO6g+VFJP9HkiFxQfaIrh8OicBXDn3GgOu1UzcWV/dMoepR pD0l+NA0n0P+9yzXCyX9E/BS/i9D7Qnq4xAHEKD68kb9UDqQIyVXq6QPtfaswoE7d2EhqtH3Z2B ZZ6nOMnBSoUl3qyEGN4QFLnLJOGCzxDpnSPe+LqgLsidnAQSb X-Received: by 2002:a05:620a:3951:b0:79b:be0b:77e9 with SMTP id af79cd13be357-79bbe0b7cbemr703657385a.1.1718979915479; Fri, 21 Jun 2024 07:25:15 -0700 (PDT) X-Google-Smtp-Source: AGHT+IEKR5sMdbnn7x0tNnlB82FrZESiADWgCE3S9rPF+Z4OitOxuRlE3iKP8psvTE41pTKeNbOqHw== X-Received: by 2002:a05:620a:3951:b0:79b:be0b:77e9 with SMTP id af79cd13be357-79bbe0b7cbemr703653085a.1.1718979914845; Fri, 21 Jun 2024 07:25:14 -0700 (PDT) Received: from x1n.redhat.com (pool-99-254-121-117.cpe.net.cable.rogers.com. [99.254.121.117]) by smtp.gmail.com with ESMTPSA id af79cd13be357-79bce944cb2sm90564785a.125.2024.06.21.07.25.12 (version=TLS1_3 cipher=TLS_AES_256_GCM_SHA384 bits=256/256); Fri, 21 Jun 2024 07:25:14 -0700 (PDT) From: Peter Xu To: linux-kernel@vger.kernel.org, linux-mm@kvack.org Cc: x86@kernel.org, Borislav Petkov , Dave Jiang , "Kirill A . Shutemov" , Ingo Molnar , Oscar Salvador , peterx@redhat.com, Matthew Wilcox , Vlastimil Babka , Dan Williams , Andrew Morton , Hugh Dickins , Michael Ellerman , Dave Hansen , Thomas Gleixner , linuxppc-dev@lists.ozlabs.org, Christophe Leroy , Rik van Riel , Mel Gorman , "Aneesh Kumar K . V" , Nicholas Piggin , Huang Ying Subject: [PATCH 4/7] mm/powerpc: Add missing pud helpers Date: Fri, 21 Jun 2024 10:25:01 -0400 Message-ID: <20240621142504.1940209-5-peterx@redhat.com> X-Mailer: git-send-email 2.45.0 In-Reply-To: <20240621142504.1940209-1-peterx@redhat.com> References: <20240621142504.1940209-1-peterx@redhat.com> MIME-Version: 1.0 X-Mimecast-Spam-Score: 0 X-Mimecast-Originator: redhat.com X-Stat-Signature: yhtph49tid3wpygxi17ngwcutid97fed X-Rspamd-Queue-Id: 238B214001F X-Rspam-User: X-Rspamd-Server: rspam10 X-HE-Tag: 1718979917-488877 X-HE-Meta: U2FsdGVkX19rMMSVRpgyC9VNEOfTr9gdWMR7df7rnqLiLv3IK8AOlqRo0QUV9G4tsXXeGZ51Mp4pOBZmu1mim7kaee0VV7/b9VW+maY+CFzXuDzWQL1fQeizVVrLy1Phei0oTRzTPFiiowDDi53M+FRGYRwN/R7iolJWTbv7gqQxR2ib2HgD6u2zIF2mfSo16stWOCjcSLEZ4uO5Efo9pUrq3tTPlQwy0nC3y0M4zhZwIj9fxeA4YipzXIsnbRusc+OnXfU1++YJo05o9+dn7d7ANYCnfpjIOjqjMdgVVGBrguDt5bxBXAdjiOew1TctVzh/rpJsfN7hF/zZqCUfyGQGJkQKagBTb5Y6lvo0tu8b9/75OPJIoZjtptlSuJ9l/+GMkxZxmCTtCyBdJdiIVMem6ZXQm21tU7c4KR9S2NV96DjiClN9J3w+IhJMBjNHTNlWjNtLw3kZYa6hMmsZ/qxKa20AJGtcD/IhEZagyp+5FdEXwgRx8VFIfUXVRiQjAupm0qg7+D+9POGzn00IuTtlobD3LI3UoG6Q9qnJG2AVkgtIjsz7KHwRqtOV2jWLc8UwXHqymsg7dil/epIsIkAsoho39TtSMh3GRHZK7Uo8TIK20+IbtmaWWk3pKtFL3HuBcjWkTDqheUGliWxQY1s21KYvu6OgJrvTB4maN9EGrWOUoVeW/lfHv5VSGI1vJEEnbAym9PUf43qUCDEWzJBIFVHVGszUqmQfqrtpq/iv3us4+5H+FNWLzs35WSO/cA9019cSz02YuLAP1WO7oZ9i3sDXoJ3mhjbcg2HX6xULNiXnvHLMucToIVTwVfX+obN5a428JoiJEQpXJLa9ec4P6vxQVuwRQsUsdpJjefyh5HAAJN2uO3pA8Fda9nFa29KA+XUiYAR8S4EOQSegImvSLXP/2fE9qV5J8FC98QXGzkSj9CK0QYQs/1gch5C0WabJXlJUYic+1zPwgqK dHTIoIJC vTwxGbIxG67i3uQFXEa3HGQb6oQXLjSuERmOr9laDh1n8g8XFtqovgjG6XO0NU5qd9hvJUE1uXWKd4ADRS2LXIPe+B3agEUyKH6oGuhCb0GssZUC49FD658vhH668mG4Go7QgKiRQJNGiKoOee1ZBT4xpRpVMICggRQM1epd830ovoCIhqSipiuIUr5X974sXcJPBlFWQaf0JQRMPlpYX8iKn1YYzB9Jg9BGtEujMEKJk2IjAt7bXmPdKQb8QESneIj6KIp0Hpne5Vw93oo51zSXLWu7iMk72opDtYbPxvJ9pxjG9s3YTMU+4ZEVrt/6gM481ngrkr38QBxyeDYVFSOBhf/uBF7YzfaY2UlIkfciZm87SaDE84jgbNrQZkSwPAo34IenhEvvdjzM+k0F4QnesfeFEMGR3D+HV+z9UL/MpM4BMxRl0BXcDBr3yKK2K0HSW+v188MCEmKzyP5o6zKhG8QLMjRhDdloA8icULaIsc1d0BBCGOY3fGi7UIAvcrHFq1j1h4WA7qg/5CGH7Hpp4D22yBc1E+jr3AoHR9fgzxYAzBn1ljlxHQyqRY9I3q2jjsQ9aN/Fx0YYxeHNtkUx/Zw== X-Bogosity: Ham, tests=bogofilter, spamicity=0.000000, version=1.2.4 Sender: owner-linux-mm@kvack.org Precedence: bulk X-Loop: owner-majordomo@kvack.org List-ID: List-Subscribe: List-Unsubscribe: These new helpers will be needed for pud entry updates soon. Namely: - pudp_invalidate() - pud_modify() Cc: Michael Ellerman Cc: Nicholas Piggin Cc: Christophe Leroy Cc: linuxppc-dev@lists.ozlabs.org Cc: Aneesh Kumar K.V Signed-off-by: Peter Xu --- arch/powerpc/include/asm/book3s/64/pgtable.h | 3 +++ arch/powerpc/mm/book3s64/pgtable.c | 20 ++++++++++++++++++++ 2 files changed, 23 insertions(+) diff --git a/arch/powerpc/include/asm/book3s/64/pgtable.h b/arch/powerpc/include/asm/book3s/64/pgtable.h index 8f9432e3855a..fc628a984669 100644 --- a/arch/powerpc/include/asm/book3s/64/pgtable.h +++ b/arch/powerpc/include/asm/book3s/64/pgtable.h @@ -1108,6 +1108,7 @@ extern pmd_t pfn_pmd(unsigned long pfn, pgprot_t pgprot); extern pud_t pfn_pud(unsigned long pfn, pgprot_t pgprot); extern pmd_t mk_pmd(struct page *page, pgprot_t pgprot); extern pmd_t pmd_modify(pmd_t pmd, pgprot_t newprot); +extern pud_t pud_modify(pud_t pud, pgprot_t newprot); extern void set_pmd_at(struct mm_struct *mm, unsigned long addr, pmd_t *pmdp, pmd_t pmd); extern void set_pud_at(struct mm_struct *mm, unsigned long addr, @@ -1368,6 +1369,8 @@ static inline pgtable_t pgtable_trans_huge_withdraw(struct mm_struct *mm, #define __HAVE_ARCH_PMDP_INVALIDATE extern pmd_t pmdp_invalidate(struct vm_area_struct *vma, unsigned long address, pmd_t *pmdp); +extern pud_t pudp_invalidate(struct vm_area_struct *vma, unsigned long address, + pud_t *pudp); #define pmd_move_must_withdraw pmd_move_must_withdraw struct spinlock; diff --git a/arch/powerpc/mm/book3s64/pgtable.c b/arch/powerpc/mm/book3s64/pgtable.c index 2975ea0841ba..c6ae969020e0 100644 --- a/arch/powerpc/mm/book3s64/pgtable.c +++ b/arch/powerpc/mm/book3s64/pgtable.c @@ -176,6 +176,17 @@ pmd_t pmdp_invalidate(struct vm_area_struct *vma, unsigned long address, return __pmd(old_pmd); } +pmd_t pudp_invalidate(struct vm_area_struct *vma, unsigned long address, + pud_t *pudp) +{ + unsigned long old_pud; + + VM_WARN_ON_ONCE(!pmd_present(*pmdp)); + old_pmd = pmd_hugepage_update(vma->vm_mm, address, pmdp, _PAGE_PRESENT, _PAGE_INVALID); + flush_pmd_tlb_range(vma, address, address + HPAGE_PMD_SIZE); + return __pmd(old_pmd); +} + pmd_t pmdp_huge_get_and_clear_full(struct vm_area_struct *vma, unsigned long addr, pmd_t *pmdp, int full) { @@ -259,6 +270,15 @@ pmd_t pmd_modify(pmd_t pmd, pgprot_t newprot) pmdv &= _HPAGE_CHG_MASK; return pmd_set_protbits(__pmd(pmdv), newprot); } + +pud_t pud_modify(pud_t pud, pgprot_t newprot) +{ + unsigned long pudv; + + pudv = pud_val(pud); + pudv &= _HPAGE_CHG_MASK; + return pud_set_protbits(__pud(pudv), newprot); +} #endif /* CONFIG_TRANSPARENT_HUGEPAGE */ /* For use by kexec, called with MMU off */ From patchwork Fri Jun 21 14:25:02 2024 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Peter Xu X-Patchwork-Id: 13707657 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from kanga.kvack.org (kanga.kvack.org [205.233.56.17]) by smtp.lore.kernel.org (Postfix) with ESMTP id 29CF2C27C4F for ; Fri, 21 Jun 2024 14:25:31 +0000 (UTC) Received: by kanga.kvack.org (Postfix) id 923E18D0178; Fri, 21 Jun 2024 10:25:23 -0400 (EDT) Received: by kanga.kvack.org (Postfix, from userid 40) id 85E208D0176; Fri, 21 Jun 2024 10:25:23 -0400 (EDT) X-Delivered-To: int-list-linux-mm@kvack.org Received: by kanga.kvack.org (Postfix, from userid 63042) id 6D6E78D0178; Fri, 21 Jun 2024 10:25:23 -0400 (EDT) X-Delivered-To: linux-mm@kvack.org Received: from relay.hostedemail.com (smtprelay0014.hostedemail.com [216.40.44.14]) by kanga.kvack.org (Postfix) with ESMTP id 4968A8D0176 for ; Fri, 21 Jun 2024 10:25:23 -0400 (EDT) Received: from smtpin20.hostedemail.com (a10.router.float.18 [10.200.18.1]) by unirelay02.hostedemail.com (Postfix) with ESMTP id E9E1D120F99 for ; Fri, 21 Jun 2024 14:25:22 +0000 (UTC) X-FDA: 82255118484.20.26AD6C3 Received: from us-smtp-delivery-124.mimecast.com (us-smtp-delivery-124.mimecast.com [170.10.133.124]) by imf20.hostedemail.com (Postfix) with ESMTP id D66711C0007 for ; Fri, 21 Jun 2024 14:25:20 +0000 (UTC) Authentication-Results: imf20.hostedemail.com; dkim=pass header.d=redhat.com header.s=mimecast20190719 header.b=MNDVFhGd; spf=pass (imf20.hostedemail.com: domain of peterx@redhat.com designates 170.10.133.124 as permitted sender) smtp.mailfrom=peterx@redhat.com; dmarc=pass (policy=none) header.from=redhat.com ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=hostedemail.com; s=arc-20220608; t=1718979911; h=from:from:sender:reply-to:subject:subject:date:date: message-id:message-id:to:to:cc:cc:mime-version:mime-version: content-type:content-type: content-transfer-encoding:content-transfer-encoding: in-reply-to:in-reply-to:references:references:dkim-signature; bh=k2ztUKph00cBWbOmplix2Q4QYxHwnhS7TcGpCnDIOcM=; b=LjD0yErpJEkkr1ECPS/8t4CFUqaDhSUkbVtyYonPyEQajjqTsYcMQnmzFDTG6IWyxKvLD7 X7UpK/hXXPCbZd+ua9lrUcD1Gedw4TDFr5Db0YBW+66twXT3IrknygcvBw0RvcBAs6XOuy q/7LtT1DYRjydfXbTiZPIyIx5karcUI= ARC-Seal: i=1; s=arc-20220608; d=hostedemail.com; t=1718979911; a=rsa-sha256; cv=none; b=RMq/kakwfv+23GB1CXDBMyJ2lMtHG8OEVzFp8Snwd3hmi2ib9WAzKORUcNiIUBlOwO6d2O 4ZscvI4xSo35qafRwfGPrP7KZ4as5w+Mk3Ym6Wl5ug+w3iei5EmhYk/K8rVjxnfezU7lvA qOJET0whMX5yrCafOP3O6yAm+BGF4rY= ARC-Authentication-Results: i=1; imf20.hostedemail.com; dkim=pass header.d=redhat.com header.s=mimecast20190719 header.b=MNDVFhGd; spf=pass (imf20.hostedemail.com: domain of peterx@redhat.com designates 170.10.133.124 as permitted sender) smtp.mailfrom=peterx@redhat.com; dmarc=pass (policy=none) header.from=redhat.com DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=redhat.com; s=mimecast20190719; t=1718979920; h=from:from:reply-to:subject:subject:date:date:message-id:message-id: to:to:cc:cc:mime-version:mime-version:content-type:content-type: content-transfer-encoding:content-transfer-encoding: in-reply-to:in-reply-to:references:references; bh=k2ztUKph00cBWbOmplix2Q4QYxHwnhS7TcGpCnDIOcM=; b=MNDVFhGd7Yxed17ZEE7bDdfEt9tuVHoTDmEYpBH/JRDVSb6r3nZN6uQtdyu/qqrhr5UKl/ j2yXuerZLYaDyE9OMqyRSK9esBLBbbt0ES6ayw2os9ZEXYtsMvQ7pF3mQGEhNZsanPdAIc E+JRZxrCrhly+c8p3gZH79tcH+e6VIE= Received: from mail-oa1-f70.google.com (mail-oa1-f70.google.com [209.85.160.70]) by relay.mimecast.com with ESMTP with STARTTLS (version=TLSv1.3, cipher=TLS_AES_256_GCM_SHA384) id us-mta-570-Fjt1vY9BMZuQRd-RNN3E5A-1; Fri, 21 Jun 2024 10:25:18 -0400 X-MC-Unique: Fjt1vY9BMZuQRd-RNN3E5A-1 Received: by mail-oa1-f70.google.com with SMTP id 586e51a60fabf-25cc7a26048so509902fac.2 for ; Fri, 21 Jun 2024 07:25:18 -0700 (PDT) X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20230601; t=1718979918; x=1719584718; h=content-transfer-encoding:mime-version:references:in-reply-to :message-id:date:subject:cc:to:from:x-gm-message-state:from:to:cc :subject:date:message-id:reply-to; bh=k2ztUKph00cBWbOmplix2Q4QYxHwnhS7TcGpCnDIOcM=; b=ggs+Hd/DaL96R+7sxB3BB7vRzqQnYJ41MHMcEprryk7yzafie2YP+74GBaKQNllsMp pXfU2tHBRn+PjwUcwWoACsHU58iTePwBsgRdSUkPXsOflgQ2zqclN6ipfgbCDwQrD+1y 55WD4PrfUUrlEwO9X/QYsQfMS0SJnoISE6Of2Xz3qZLaTCizcprtAowINqO8YeBADr5h lonSupciZFQS+UvYnHDmIOu7BfY085bSSrjnkbAt6m+z/aRqG4K1GG3y2aO4VW1OvGXS 9P23HAXqUV2Nq49VTMiSRIJNv8ViGomLN/9XYYLvuGa7HTQlMO3HxqFShMoOr1jhzH8u UTKQ== X-Forwarded-Encrypted: i=1; AJvYcCUfvyZC6pC5aUojnsuILQJlyv1am9r51/QsSSCZxA1IEdb5SvhF8yGtM0I0URR8rAvgBci+9RGjs3avo2iXE2Am2Fo= X-Gm-Message-State: AOJu0YwhSvSZDxL3Ai1vxJQOIYfGTAKirFkaqMlzFbJcmWy/QNwa5djj VN5Nm968aYYzW6xI02Yk7tcrnJiwEcppHBxUs0rcqIIlFqV07z/zHwRhTljiPGg/eK+6+ZXC8XB Im6FaxkUjT+wDJ/uwXsZpTtNVmHRoqoojdCnbbfiHAUL+sngH X-Received: by 2002:a05:6871:14f:b0:254:cae6:a812 with SMTP id 586e51a60fabf-25c94d8a94emr9065105fac.3.1718979917840; Fri, 21 Jun 2024 07:25:17 -0700 (PDT) X-Google-Smtp-Source: AGHT+IEpfu3cPye3+7RpVIl2JRxNk+CTOiRpoiTdFmb6dpb/ERi8sD9JFsePfCkLHiFMYZR2Pz0U9w== X-Received: by 2002:a05:6871:14f:b0:254:cae6:a812 with SMTP id 586e51a60fabf-25c94d8a94emr9065078fac.3.1718979917305; Fri, 21 Jun 2024 07:25:17 -0700 (PDT) Received: from x1n.redhat.com (pool-99-254-121-117.cpe.net.cable.rogers.com. [99.254.121.117]) by smtp.gmail.com with ESMTPSA id af79cd13be357-79bce944cb2sm90564785a.125.2024.06.21.07.25.15 (version=TLS1_3 cipher=TLS_AES_256_GCM_SHA384 bits=256/256); Fri, 21 Jun 2024 07:25:16 -0700 (PDT) From: Peter Xu To: linux-kernel@vger.kernel.org, linux-mm@kvack.org Cc: x86@kernel.org, Borislav Petkov , Dave Jiang , "Kirill A . Shutemov" , Ingo Molnar , Oscar Salvador , peterx@redhat.com, Matthew Wilcox , Vlastimil Babka , Dan Williams , Andrew Morton , Hugh Dickins , Michael Ellerman , Dave Hansen , Thomas Gleixner , linuxppc-dev@lists.ozlabs.org, Christophe Leroy , Rik van Riel , Mel Gorman , "Aneesh Kumar K . V" , Nicholas Piggin , Huang Ying Subject: [PATCH 5/7] mm/x86: Make pud_leaf() only cares about PSE bit Date: Fri, 21 Jun 2024 10:25:02 -0400 Message-ID: <20240621142504.1940209-6-peterx@redhat.com> X-Mailer: git-send-email 2.45.0 In-Reply-To: <20240621142504.1940209-1-peterx@redhat.com> References: <20240621142504.1940209-1-peterx@redhat.com> MIME-Version: 1.0 X-Mimecast-Spam-Score: 0 X-Mimecast-Originator: redhat.com X-Rspamd-Queue-Id: D66711C0007 X-Stat-Signature: qhz4kt4yi5twxrae6jakz1bpt1rqw34s X-Rspamd-Server: rspam09 X-Rspam-User: X-HE-Tag: 1718979920-475229 X-HE-Meta: U2FsdGVkX18p1NFupIVDuQEzlESC55YNPXny6LVGr7ccz/NObYiRt+StmxcZIfr3FOjttazMIkqZ16+nN+R4q917yhdNnmF0c3mu6dq+NADXXwozHlRhFb7JjSrBl4MEZEpyKMTa/lqj5H5c7y7tZKLWIsxcdfBQW81R4AxXo5CJbbKeNKGq+tfIEZuXViiWWd5WNcsW0og1m1DyhqH5I7ZeNYZF79H/lv7cyxtbjnwqo+CmX0EyuqCxlhYUvfRm6YBZ+kZESFtgbQZjeODMKkDZIHVKkar1/dqfV/rgXumERdaCettulgSYlFLvLnegr1yDu+o3bAJ81sCDPy7av5iG+oAftdiN77RbfH3aA9kq0QJV+/5uW+Tt+NLvNoK++MrLpC6T+5h8uedp3ikT28JLobdspPhMvm1aTKZOSrHN4aOX5rWFAxUfpK2gvuJfPu7INtk3MQ6wfg4WGfRUD13FBHamG7EyzqM2/p+9E74ODvL02SIDVR+37CwVEt8QC70yQN849Q6S5/lbWWz4PAtyKCfMH81B5M5D85K80EmjP04X3GHgvH5Aj7nRLDOx1fuIQWqBYmv7LdpzZ4SZGUwIlIT8/ewIOLmo347dTU4uYdtLMz6DUTlgTE1O2FQCMdVpHe214KarT7ERMxMx7f5gcKXSm+gWdcdDYFTL2fhUMdofikwZU0CjJqU3Vv4Y5FYBoZj8b1glMpyQctQ/YmuP3b2D4wr2I6K0MR/pWoNJZFzQcdbKI2TwqTwpC3YbsTwGVPspJtL4p4qHIjafAao2sqFjECXArf8LDjATIziq1XzQeXS9wMM7+/0lIK1CNU3CmjuVH63Hm+DGXWucJTf3U3rovTOFWkJACZY6tBPkTsNTTTrZ5HDcKyMM9hrxjI/U+htntLsbijR6LrgldUN03Haff5KnSTKHHBNIijon467IR+nXZu4zuHTSbITlexiepWQQPmOd2QxNhJD yF2gE0i8 6HXh8lvvXCEveyD7TCF09q+YZhFC21VcN9q7r/K/Avh6iEEVOGRbPLv2MFe9vB2NmMQVvy9NAM/oBdEPw4PGIZZNgGyEpA6EolfrtnH7SC/WdEK76hDqTzUXaz7G4IkBQDIxBMHyfiO+W28TRYuK20vuyjky0BdqI/HxIRhBVlbH4LyIfEVvEswG3CoCggoI6nC1vKBHv4T3/uHvEsC2nrr/XEyIu0dNNEryrpD3MoI43aukRc3EtBcoEv98M6L4cMYgGG5ksUCwCEBr0LCwrGc+Lg1ICgGRDsPmx3OzgTUPffeXn9JagVbBKwIdBiZQpuGRr4pwZX2qfCT6ZXvmRgpRXHAHEcc+bBErImUg36pC/J7Ill1N1vRZVtkP1L/NKlq18FB6y6gbxw4P5eUeOQfm+YQrDQQcwJiMl3Bqw2ZlgGf9dWEEJJIsXTXHcWDYpNKbI0GOvjVUg9ir/zI/LvDyfhw== X-Bogosity: Ham, tests=bogofilter, spamicity=0.000000, version=1.2.4 Sender: owner-linux-mm@kvack.org Precedence: bulk X-Loop: owner-majordomo@kvack.org List-ID: List-Subscribe: List-Unsubscribe: An entry should be reported as PUD leaf even if it's PROT_NONE, in which case PRESENT bit isn't there. I hit bad pud without this when testing dax 1G on zapping a PROT_NONE PUD. Cc: Thomas Gleixner Cc: Ingo Molnar Cc: Borislav Petkov Cc: Dave Hansen Cc: x86@kernel.org Signed-off-by: Peter Xu Acked-by: Dave Hansen --- arch/x86/include/asm/pgtable.h | 3 +-- 1 file changed, 1 insertion(+), 2 deletions(-) diff --git a/arch/x86/include/asm/pgtable.h b/arch/x86/include/asm/pgtable.h index 65b8e5bb902c..25fc6d809572 100644 --- a/arch/x86/include/asm/pgtable.h +++ b/arch/x86/include/asm/pgtable.h @@ -1073,8 +1073,7 @@ static inline pmd_t *pud_pgtable(pud_t pud) #define pud_leaf pud_leaf static inline bool pud_leaf(pud_t pud) { - return (pud_val(pud) & (_PAGE_PSE | _PAGE_PRESENT)) == - (_PAGE_PSE | _PAGE_PRESENT); + return pud_val(pud) & _PAGE_PSE; } static inline int pud_bad(pud_t pud) From patchwork Fri Jun 21 14:25:03 2024 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Peter Xu X-Patchwork-Id: 13707658 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from kanga.kvack.org (kanga.kvack.org [205.233.56.17]) by smtp.lore.kernel.org (Postfix) with ESMTP id 27258C27C4F for ; Fri, 21 Jun 2024 14:25:35 +0000 (UTC) Received: by kanga.kvack.org (Postfix) id 2BA9B6B00FD; Fri, 21 Jun 2024 10:25:27 -0400 (EDT) Received: by kanga.kvack.org (Postfix, from userid 40) id 273FD6B0103; Fri, 21 Jun 2024 10:25:27 -0400 (EDT) X-Delivered-To: int-list-linux-mm@kvack.org Received: by kanga.kvack.org (Postfix, from userid 63042) id F15346B00FD; Fri, 21 Jun 2024 10:25:26 -0400 (EDT) X-Delivered-To: linux-mm@kvack.org Received: from relay.hostedemail.com (smtprelay0017.hostedemail.com [216.40.44.17]) by kanga.kvack.org (Postfix) with ESMTP id CFA676B00FA for ; Fri, 21 Jun 2024 10:25:26 -0400 (EDT) Received: from smtpin21.hostedemail.com (a10.router.float.18 [10.200.18.1]) by unirelay02.hostedemail.com (Postfix) with ESMTP id 4CD90120BF4 for ; Fri, 21 Jun 2024 14:25:26 +0000 (UTC) X-FDA: 82255118652.21.78C00C0 Received: from us-smtp-delivery-124.mimecast.com (us-smtp-delivery-124.mimecast.com [170.10.129.124]) by imf15.hostedemail.com (Postfix) with ESMTP id 45D51A001B for ; Fri, 21 Jun 2024 14:25:24 +0000 (UTC) Authentication-Results: imf15.hostedemail.com; dkim=pass header.d=redhat.com header.s=mimecast20190719 header.b="Q/PWYrIF"; spf=pass (imf15.hostedemail.com: domain of peterx@redhat.com designates 170.10.129.124 as permitted sender) smtp.mailfrom=peterx@redhat.com; dmarc=pass (policy=none) header.from=redhat.com ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=hostedemail.com; s=arc-20220608; t=1718979919; h=from:from:sender:reply-to:subject:subject:date:date: message-id:message-id:to:to:cc:cc:mime-version:mime-version: content-type:content-type: content-transfer-encoding:content-transfer-encoding: in-reply-to:in-reply-to:references:references:dkim-signature; bh=tqSfUWMuuFFpoXlX71uD6ModhPCWC1FSZp1xmlIx7N4=; b=mxoLWAO3uYuMWiUNmqNzb/HKbhZr6U/1s40f68LCTcc5kMx9ICoW1NzmVHhdRd73Htpq8a irm3zSvTuYsMQo4myhQoJ699SdRfBGKLpe4GHamUBxbQTxn3zA3YXL0/fP91p6MAHRv5eb HYObF15D2Ogn4buA+PG4z2cYlisIo/U= ARC-Authentication-Results: i=1; imf15.hostedemail.com; dkim=pass header.d=redhat.com header.s=mimecast20190719 header.b="Q/PWYrIF"; spf=pass (imf15.hostedemail.com: domain of peterx@redhat.com designates 170.10.129.124 as permitted sender) smtp.mailfrom=peterx@redhat.com; dmarc=pass (policy=none) header.from=redhat.com ARC-Seal: i=1; s=arc-20220608; d=hostedemail.com; t=1718979919; a=rsa-sha256; cv=none; b=yoRZG+EgBo6QwYRAWT27y3KeCEfJdmTp+sXrhtrMRlAshgpVGNCz6CIghBow5mo0jR7LGJ iv7ykvGq2A4AAG3VxUjVBMITHwjaK9UqNobePRgW85nGrZBd+rAvXtrXgKtJpsIZT2oh88 lvYIFogsHfBLuJjN8DBm9oNu+QSjSCE= DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=redhat.com; s=mimecast20190719; t=1718979923; h=from:from:reply-to:subject:subject:date:date:message-id:message-id: to:to:cc:cc:mime-version:mime-version:content-type:content-type: content-transfer-encoding:content-transfer-encoding: in-reply-to:in-reply-to:references:references; bh=tqSfUWMuuFFpoXlX71uD6ModhPCWC1FSZp1xmlIx7N4=; b=Q/PWYrIFwYsOQkwMj1/MqEliOkiM84OOsJxRrWn8JcYYjP6HFD9bKJpbwBuLblvT+rsfn8 W3FJHqXZVnWfyWUE6zv3izzW+OWDo7yv7y9sqSsGWFxJ5kdqCrvCvWY1azTf0qJ2fgWC48 e1NetC2eUOWygigfS2alSkX1q24D8AU= Received: from mail-ot1-f70.google.com (mail-ot1-f70.google.com [209.85.210.70]) by relay.mimecast.com with ESMTP with STARTTLS (version=TLSv1.3, cipher=TLS_AES_256_GCM_SHA384) id us-mta-635-CHQTQU0KOPuZOfxdZ_517Q-1; Fri, 21 Jun 2024 10:25:21 -0400 X-MC-Unique: CHQTQU0KOPuZOfxdZ_517Q-1 Received: by mail-ot1-f70.google.com with SMTP id 46e09a7af769-6f908f002c2so339989a34.1 for ; Fri, 21 Jun 2024 07:25:21 -0700 (PDT) X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20230601; t=1718979920; x=1719584720; h=content-transfer-encoding:mime-version:references:in-reply-to :message-id:date:subject:cc:to:from:x-gm-message-state:from:to:cc :subject:date:message-id:reply-to; bh=tqSfUWMuuFFpoXlX71uD6ModhPCWC1FSZp1xmlIx7N4=; b=ZYzw87uGVrqfea5wnzoMjIvPFYe7JMQzAVq8qReXUW6HIOd6fxmsPtiP8LUo2qVNpX P5KULsGECLiL3Cmc5qCuGTdc8TdmyIOwtwp7dj6wBJLCRvVfHSeDvkSmKp6M2wtC59yd GIrzRBHziZolsO9b4z35lPLIIF7q7t61Xdza7r/N6ibd7nd/P1F3R1ZmFOiyLXhnQwCg wgKi/fVzM9zvQ62dHW/xC2ItPo3xnUmDZOSdyDm4WoDSgRlcDevV6lBeOcEkb0SQqMfk R4sjBrM+Nr5d0Wn7m6KMnMpyCxz4YKHkxOIUddBWZyueh5l1sGWRnMzwN7/N+Iu/fRNU VYxQ== X-Forwarded-Encrypted: i=1; AJvYcCXjso0qo9yWbminJcphEH9wiTMn5xSOqL/Q6z8a21RK0b+CyAeJPOCxwhiUsy6AeR0xpKGR8+Qj9uBfzlVVIwvG2g4= X-Gm-Message-State: AOJu0YwlLjyyyDxg9Oh3KAm77uJkAeHB2PQMoc/LT5438JDflQV3rNj5 3h8xzoOK+8XYWgBdoxZA88BIcHd/PVx5i1aKqhCtiQbOVxTancwDo4xYhfiNiPyGPFSWg8KayTC OZh7QnZw7xx4PMoJB7k/BXq95s8KoA9RHvqh9vcp5GTlnp2tm X-Received: by 2002:a05:6358:8096:b0:19f:81fb:131c with SMTP id e5c5f4694b2df-1a1fd1d513bmr911568055d.0.1718979920234; Fri, 21 Jun 2024 07:25:20 -0700 (PDT) X-Google-Smtp-Source: AGHT+IHOaIBFrX4PIBPbd7NrgasVtJd+jvRQiQvrolpQX5hE+bY+MZMCcoynbRsNLrRaLePwagbsiQ== X-Received: by 2002:a05:6358:8096:b0:19f:81fb:131c with SMTP id e5c5f4694b2df-1a1fd1d513bmr911563455d.0.1718979919599; Fri, 21 Jun 2024 07:25:19 -0700 (PDT) Received: from x1n.redhat.com (pool-99-254-121-117.cpe.net.cable.rogers.com. [99.254.121.117]) by smtp.gmail.com with ESMTPSA id af79cd13be357-79bce944cb2sm90564785a.125.2024.06.21.07.25.17 (version=TLS1_3 cipher=TLS_AES_256_GCM_SHA384 bits=256/256); Fri, 21 Jun 2024 07:25:18 -0700 (PDT) From: Peter Xu To: linux-kernel@vger.kernel.org, linux-mm@kvack.org Cc: x86@kernel.org, Borislav Petkov , Dave Jiang , "Kirill A . Shutemov" , Ingo Molnar , Oscar Salvador , peterx@redhat.com, Matthew Wilcox , Vlastimil Babka , Dan Williams , Andrew Morton , Hugh Dickins , Michael Ellerman , Dave Hansen , Thomas Gleixner , linuxppc-dev@lists.ozlabs.org, Christophe Leroy , Rik van Riel , Mel Gorman , "Aneesh Kumar K . V" , Nicholas Piggin , Huang Ying Subject: [PATCH 6/7] mm/x86: Add missing pud helpers Date: Fri, 21 Jun 2024 10:25:03 -0400 Message-ID: <20240621142504.1940209-7-peterx@redhat.com> X-Mailer: git-send-email 2.45.0 In-Reply-To: <20240621142504.1940209-1-peterx@redhat.com> References: <20240621142504.1940209-1-peterx@redhat.com> MIME-Version: 1.0 X-Mimecast-Spam-Score: 0 X-Mimecast-Originator: redhat.com X-Rspamd-Server: rspam03 X-Rspam-User: X-Rspamd-Queue-Id: 45D51A001B X-Stat-Signature: iny73pfp7p4ptix8jhmg8w7zza9hie76 X-HE-Tag: 1718979924-145272 X-HE-Meta: U2FsdGVkX180cvweJCHfdlvygvwtPmWAjyL22miZAihVp8VFCA2JQqne3Bzf7yjAUr102bogQOdZRK2YbeNd0VotRoBkgaSEGjY/xbcAewKBo9kp3Gw0qhFQe7hBgNr1FCSkPvCZK03oXVl29fI6LsS1zHt/swK4iJ9DRpnVekkMogon4um0hH2wJxl2J31DQtY0mCvMGwJXhU4UXREOCzSvBDxDelo8KAUkS/EduTFtmIzAVLPRahJ1zbRK7nw4id+rqK4wdWz/S0T/teGD/BYuf5Yw2pRVUtbKofTR6g6La3Y9j2FWmcCX4V376Apo8sM/zy1kHFkh4KvZueb8ZpcCFn1L4gJETIThGk3t+hCp8rYMIwrqAtPrcd4ejPakng3H7ILbwXkDh5am4MZnG2UXB6M3U1CelSVN26qVu1bcfSxKNf4osvU5pSzUSkYbt/CNQGgWf4KTzIGeY2AqEMnmKvubSJKcPEYftna3ljoxq2zybEiG57T0o2htZDTK8Vfhj2kA4zTzIkk00WBySUOW6i2ul7Aq/mBpbts/CBMkJdywZQeT+sssjSnQ3/sff0NCoq/gHtc332Bq8506oyuOaFKdsybOZ3xSdL4fccSd/edVa3aMuOeB7ko9mNrFfXzpjTXhWOvkNvoUraJntCVNTtt4P8Quqpq/cFhkcxVYNwge1ILk1nOCGsAo/AWUgBM/eVYtM/99ARotEQjxlNCe4gNgo88gpt+ipQB2qYtAMY8CHnGs6IJ+aCjrp8nok9f8FikAXtkfPZWlZUct+R5SU5newP4xuyVk+N294L8y4ur4242UIsrPC1R97Dz+Qc2FH/D+UAOK7NBA/2hTWGtTVc6JF5P2d+Xm8tnX/GnZXZzk6ppkx0i1VK3e0EcE1peK/GBf1ZR1eFZ1Wz7z+/wxRNnIaq7PaA7Rhq6kF/H1MWqY6k2FqB5Gat6nGgnfOQix2TOaE8WtcWSvkLk 1+51cM+F zwSlqMQshO5Dgt4lnixJKHQAy6BMbXfZJwY47g3Gxk6oHX/wHGGoSogtDPDcGssfaoFcoydKx2lGdXZv3+4KTxTIGn4NfMTsJ7fOGZjmcsL97OY2H0G+ZuEhrO+OLDsRoaW84OrXB4u0pQS6S0Wn7xeqnom3VNF6/gT+zLXmR1DvHUoSO+d/g16kFGUkgnM2FJQ0Hag0RIHEIUfUBZqPz+MN1VRFaJ9yaKP0RD3ilaRIY+Bxpy1z6ZTy9cf9zJKCMDK+c+4ayoA+ybCK7yok6Rg4UvfTL7pYVeHexdGtk1N2jGk8Ug5EHpr4K7wlIJ/inqL++w28X2k/MvqApregI8DIhu2+XJFLYWyTT3eMmQByv23j+FpIfvCWAaumB+MXjbpwMxgHJZI+irLc6jOCXtNL80EFAXBRipOYl6VvnDwYB2Fo/eo5YKwRnU5CcoW9zLkokiotx2AU0Kj2jsMMJi3UiaB8spikuxBF3TsdXFe01sO2pMsP1gl6N2rcgQYNBOO7b X-Bogosity: Ham, tests=bogofilter, spamicity=0.000000, version=1.2.4 Sender: owner-linux-mm@kvack.org Precedence: bulk X-Loop: owner-majordomo@kvack.org List-ID: List-Subscribe: List-Unsubscribe: These new helpers will be needed for pud entry updates soon. Namely: - pudp_invalidate() - pud_modify() Cc: Thomas Gleixner Cc: Ingo Molnar Cc: Borislav Petkov Cc: Dave Hansen Cc: x86@kernel.org Signed-off-by: Peter Xu --- arch/x86/include/asm/pgtable.h | 36 ++++++++++++++++++++++++++++++++++ arch/x86/mm/pgtable.c | 11 +++++++++++ 2 files changed, 47 insertions(+) diff --git a/arch/x86/include/asm/pgtable.h b/arch/x86/include/asm/pgtable.h index 25fc6d809572..3c23077adca6 100644 --- a/arch/x86/include/asm/pgtable.h +++ b/arch/x86/include/asm/pgtable.h @@ -775,6 +775,12 @@ static inline pmd_t pmd_mkinvalid(pmd_t pmd) __pgprot(pmd_flags(pmd) & ~(_PAGE_PRESENT|_PAGE_PROTNONE))); } +static inline pud_t pud_mkinvalid(pud_t pud) +{ + return pfn_pud(pud_pfn(pud), + __pgprot(pud_flags(pud) & ~(_PAGE_PRESENT|_PAGE_PROTNONE))); +} + static inline u64 flip_protnone_guard(u64 oldval, u64 val, u64 mask); static inline pte_t pte_modify(pte_t pte, pgprot_t newprot) @@ -839,6 +845,21 @@ static inline pmd_t pmd_modify(pmd_t pmd, pgprot_t newprot) return pmd_result; } +static inline pud_t pud_modify(pud_t pud, pgprot_t newprot) +{ + pudval_t val = pud_val(pud), oldval = val; + + /* + * NOTE: no need to consider shadow stack complexities because it + * doesn't support 1G mappings. + */ + val &= _HPAGE_CHG_MASK; + val |= check_pgprot(newprot) & ~_HPAGE_CHG_MASK; + val = flip_protnone_guard(oldval, val, PHYSICAL_PUD_PAGE_MASK); + + return __pud(val); +} + /* * mprotect needs to preserve PAT and encryption bits when updating * vm_page_prot @@ -1377,10 +1398,25 @@ static inline pmd_t pmdp_establish(struct vm_area_struct *vma, } #endif +static inline pud_t pudp_establish(struct vm_area_struct *vma, + unsigned long address, pud_t *pudp, pud_t pud) +{ + if (IS_ENABLED(CONFIG_SMP)) { + return xchg(pudp, pud); + } else { + pud_t old = *pudp; + WRITE_ONCE(*pudp, pud); + return old; + } +} + #define __HAVE_ARCH_PMDP_INVALIDATE_AD extern pmd_t pmdp_invalidate_ad(struct vm_area_struct *vma, unsigned long address, pmd_t *pmdp); +pud_t pudp_invalidate(struct vm_area_struct *vma, unsigned long address, + pud_t *pudp); + /* * Page table pages are page-aligned. The lower half of the top * level is used for userspace and the top half for the kernel. diff --git a/arch/x86/mm/pgtable.c b/arch/x86/mm/pgtable.c index 93e54ba91fbf..4e245a1526ad 100644 --- a/arch/x86/mm/pgtable.c +++ b/arch/x86/mm/pgtable.c @@ -641,6 +641,17 @@ pmd_t pmdp_invalidate_ad(struct vm_area_struct *vma, unsigned long address, } #endif +#ifdef CONFIG_TRANSPARENT_HUGEPAGE +pud_t pudp_invalidate(struct vm_area_struct *vma, unsigned long address, + pud_t *pudp) +{ + VM_WARN_ON_ONCE(!pud_present(*pudp)); + pud_t old = pudp_establish(vma, address, pudp, pud_mkinvalid(*pudp)); + flush_pud_tlb_range(vma, address, address + HPAGE_PUD_SIZE); + return old; +} +#endif + /** * reserve_top_address - reserves a hole in the top of kernel address space * @reserve - size of hole to reserve From patchwork Fri Jun 21 14:25:04 2024 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Peter Xu X-Patchwork-Id: 13707659 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from kanga.kvack.org (kanga.kvack.org [205.233.56.17]) by smtp.lore.kernel.org (Postfix) with ESMTP id 11B32C27C4F for ; Fri, 21 Jun 2024 14:25:39 +0000 (UTC) Received: by kanga.kvack.org (Postfix) id 9EB286B0431; Fri, 21 Jun 2024 10:25:29 -0400 (EDT) Received: by kanga.kvack.org (Postfix, from userid 40) id 96F956B0432; Fri, 21 Jun 2024 10:25:29 -0400 (EDT) X-Delivered-To: int-list-linux-mm@kvack.org Received: by kanga.kvack.org (Postfix, from userid 63042) id 775196B042D; Fri, 21 Jun 2024 10:25:29 -0400 (EDT) X-Delivered-To: linux-mm@kvack.org Received: from relay.hostedemail.com (smtprelay0016.hostedemail.com [216.40.44.16]) by kanga.kvack.org (Postfix) with ESMTP id 502BF6B0428 for ; Fri, 21 Jun 2024 10:25:29 -0400 (EDT) Received: from smtpin08.hostedemail.com (a10.router.float.18 [10.200.18.1]) by unirelay09.hostedemail.com (Postfix) with ESMTP id 124CB80F74 for ; Fri, 21 Jun 2024 14:25:29 +0000 (UTC) X-FDA: 82255118778.08.2F41511 Received: from us-smtp-delivery-124.mimecast.com (us-smtp-delivery-124.mimecast.com [170.10.133.124]) by imf24.hostedemail.com (Postfix) with ESMTP id E2DC9180013 for ; Fri, 21 Jun 2024 14:25:26 +0000 (UTC) Authentication-Results: imf24.hostedemail.com; dkim=pass header.d=redhat.com header.s=mimecast20190719 header.b=XQ0BTdFB; spf=pass (imf24.hostedemail.com: domain of peterx@redhat.com designates 170.10.133.124 as permitted sender) smtp.mailfrom=peterx@redhat.com; dmarc=pass (policy=none) header.from=redhat.com ARC-Seal: i=1; s=arc-20220608; d=hostedemail.com; t=1718979921; a=rsa-sha256; cv=none; b=78r3DkqphGxnA34rcccSHdmiW6LKhHb3imE+ffl2C0DVzI1bQ3sUzzxa7W6wOeFevIdgWU rbwP3LyBJVEc/Bu52tDFIcok4BgY1f6fxOqi9fnPziZjduTHUhHVSCTZA93u2xoJGrI6c6 1pGCzilXxDn9rPfPtKkdF7hzqspipRk= ARC-Authentication-Results: i=1; imf24.hostedemail.com; dkim=pass header.d=redhat.com header.s=mimecast20190719 header.b=XQ0BTdFB; spf=pass (imf24.hostedemail.com: domain of peterx@redhat.com designates 170.10.133.124 as permitted sender) smtp.mailfrom=peterx@redhat.com; dmarc=pass (policy=none) header.from=redhat.com ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=hostedemail.com; s=arc-20220608; t=1718979921; h=from:from:sender:reply-to:subject:subject:date:date: message-id:message-id:to:to:cc:cc:mime-version:mime-version: content-type:content-type: content-transfer-encoding:content-transfer-encoding: in-reply-to:in-reply-to:references:references:dkim-signature; bh=FloNdHDAMCUmgBKRbs+OcXuVLR4NlkKUkgqw/Oorwhs=; b=SLI+25IEqEkSgd+iUTDU4OMQcGTTYjpnzpslXxiuFEbhe/1kQgRahFvS0D4kwa8fw/BO6a syvlcakGlU1l/ae5Zf12RVDulB5Y7qRdDnDOMKFzydmJfQwWFizIHP7mHMx3ATUeXV6IVX dC3N/tjcmUK3TnEXEF6PZw9jYPEQueY= DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=redhat.com; s=mimecast20190719; t=1718979926; h=from:from:reply-to:subject:subject:date:date:message-id:message-id: to:to:cc:cc:mime-version:mime-version:content-type:content-type: content-transfer-encoding:content-transfer-encoding: in-reply-to:in-reply-to:references:references; bh=FloNdHDAMCUmgBKRbs+OcXuVLR4NlkKUkgqw/Oorwhs=; b=XQ0BTdFBI+ykbnJvxk9XobUoMX7glCe/nTMu5tQNPSjjOyg1Ib/oigX9Q8JKzR+5CZQmP2 1Fdz9zB7jCUYE+FcdlWp7f14Uqdxcif7+VXEWKGdOixpDyRnu1V8/KHe2wZ/UbUCwKRLJB 0gLfbVANPMK4lJiiB89H5H6lpRdx2a8= Received: from mail-qt1-f198.google.com (mail-qt1-f198.google.com [209.85.160.198]) by relay.mimecast.com with ESMTP with STARTTLS (version=TLSv1.3, cipher=TLS_AES_256_GCM_SHA384) id us-mta-316-DnUxS-eaOiae2_EsbYnU-Q-1; Fri, 21 Jun 2024 10:25:22 -0400 X-MC-Unique: DnUxS-eaOiae2_EsbYnU-Q-1 Received: by mail-qt1-f198.google.com with SMTP id d75a77b69052e-43fd537e6a6so3296671cf.0 for ; Fri, 21 Jun 2024 07:25:22 -0700 (PDT) X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20230601; t=1718979922; x=1719584722; h=content-transfer-encoding:mime-version:references:in-reply-to :message-id:date:subject:cc:to:from:x-gm-message-state:from:to:cc :subject:date:message-id:reply-to; bh=FloNdHDAMCUmgBKRbs+OcXuVLR4NlkKUkgqw/Oorwhs=; b=YJEpkLfmPF4RIqEgUNt5DpmgTlbO/pGQ02AtNhGW7gvq4/5h5+2+In/QhFeS6beHHe 9W1e9UoocejKasSUybTX0fnxwKtSwa2wXXtR3kxUx8oPA6NjS5dB0zWrA8Oulmw1moDs Px6unGuC9ERyGRvo7nBK+WFNqQPsSPWv5mQJr2x9UXl9hLp9zuIaDrPqYlVWzgKPOr61 WFPOUxhasIU585HT/MToGpFEs+rb064tj7IpR19pC5akg/FRllwS6NRaHWCGEa27bkvF Wo6RFd2Fxs+OcXnGYABIMr82oKjHM/zbFyoWJZoBVK8bEiOe+cr73VFwxAdTJTRJyru2 KTHQ== X-Forwarded-Encrypted: i=1; AJvYcCVo3+DhU7ZcRDYza5+boaRyQh/wjAbm+ChwmUhsEgvuAdjP+lM0oQrczID89kSWuhh2QgAIy81S27w35IrDQ9QvLRM= X-Gm-Message-State: AOJu0YzdwBpEfYZbQTNvDisVPCzKarsD4/FOoBa+jDtyD1q3dNOKLwmf XjYyECUYZeuub8no9iFHYI6pRepO/47IlXyDYhRhwU2CQZBIh0+nUN9POksQaxCHfgNRFHjw9N8 HUnKPf7UaTNsb+0L9z/2Bn4HZAEt4i0eLJGRZeYd08FkSkuks X-Received: by 2002:a05:620a:4725:b0:79b:b8ef:160f with SMTP id af79cd13be357-79bb8ef1a25mr832756885a.0.1718979922070; Fri, 21 Jun 2024 07:25:22 -0700 (PDT) X-Google-Smtp-Source: AGHT+IHKznxYW2FLJ0xJwWKaEtD8Tk/BHH+OIQZWLkXgxtzT/k39w+HZ9hDMrU5i6fKaXRUQoupH+Q== X-Received: by 2002:a05:620a:4725:b0:79b:b8ef:160f with SMTP id af79cd13be357-79bb8ef1a25mr832752985a.0.1718979921429; Fri, 21 Jun 2024 07:25:21 -0700 (PDT) Received: from x1n.redhat.com (pool-99-254-121-117.cpe.net.cable.rogers.com. [99.254.121.117]) by smtp.gmail.com with ESMTPSA id af79cd13be357-79bce944cb2sm90564785a.125.2024.06.21.07.25.19 (version=TLS1_3 cipher=TLS_AES_256_GCM_SHA384 bits=256/256); Fri, 21 Jun 2024 07:25:21 -0700 (PDT) From: Peter Xu To: linux-kernel@vger.kernel.org, linux-mm@kvack.org Cc: x86@kernel.org, Borislav Petkov , Dave Jiang , "Kirill A . Shutemov" , Ingo Molnar , Oscar Salvador , peterx@redhat.com, Matthew Wilcox , Vlastimil Babka , Dan Williams , Andrew Morton , Hugh Dickins , Michael Ellerman , Dave Hansen , Thomas Gleixner , linuxppc-dev@lists.ozlabs.org, Christophe Leroy , Rik van Riel , Mel Gorman , "Aneesh Kumar K . V" , Nicholas Piggin , Huang Ying Subject: [PATCH 7/7] mm/mprotect: fix dax pud handlings Date: Fri, 21 Jun 2024 10:25:04 -0400 Message-ID: <20240621142504.1940209-8-peterx@redhat.com> X-Mailer: git-send-email 2.45.0 In-Reply-To: <20240621142504.1940209-1-peterx@redhat.com> References: <20240621142504.1940209-1-peterx@redhat.com> MIME-Version: 1.0 X-Mimecast-Spam-Score: 0 X-Mimecast-Originator: redhat.com X-Stat-Signature: fkx33tsthobuegdiw4oj7gbugmikgc7u X-Rspamd-Queue-Id: E2DC9180013 X-Rspam-User: X-Rspamd-Server: rspam10 X-HE-Tag: 1718979926-555283 X-HE-Meta: U2FsdGVkX1/isRBzSSklgTHnuJyyvOlgFosZPutaLqdTCAZplMvrX439YbzeNfa1sPtfagsE6OVXGrkhqI0AJE8uZbyhZQhB3O3QaNC8CvM9um4SEgC12cFBiYZMiDoz4OHscTUCyXAdMjxVFD72OKvm7gUZSqImbPcEWpxjs5ifSh8USpl6ocg5/ulG1Pokg6namguBmgBmXaDLyd0FVwKTYOh2rbDXwEIa0Ik12iFnqfCyhyUHbDuarNKWz0LHODIvLFFTn4lhLV7PLHQata5wyQhKJPK95lc+COrQlIAhxnfz+5Vj6Al0u2TU8Th9Pw3HuMT6CYObCWk89AsTVZKdI6S6bZUwlVcMGVDoU1Rers555P+1baGRIjEDLrDn2jzZdpfTbsVsht6d2Xo6+rVGmQpWAR2+yFauOzABfF51z/q31asqpwCM+lzJhVrtrMLMYtH7W+HOkXEcVNdoXr9Af7fmZ37fdJ4fa3lBFVaHTzowiFzifyVni98gpLAt9trtwU5nLS6EhmMNMsTXQqZHT+QK5olZCus4J/JjHsIBoH7cXWdZJX9gBd8dBcW8RLNbjUuY0DQVlnQeVX+HVaRN1I8ryi6PbxdFPgo6Z45XZDeRyQzYy19lH575MzzccFcwGDF5c0J8s7YkaDL86CEsMA8x0WaDr/XTJE+gkCHQeRBHmGEre1afw/7QWRFV6204/23Zl6KJg3LnHgDPciE65INCzbj0fmkMFh/tZ5Ubb9IvTzCIBvyUsAGv0tM2TG8zrqo6p2B6vi5He+q26mtMiA2ZF6CbxOu8ZliF6zzIirf1npyqW+yJc2xsq4aDdpfUUCzACe72t6P/xydmbO53m8JWArgVl7ezVPJf9F403OsBy7BT4+qTa1JNwGtRSaxsJ61mNWj4chqw3KAhgCsrePnCjCbpSUmxIJopPp0Qwiz4Vas4+tPBhZW6uBuzIsI/A33s1MCf3JI6a6x gs56GJL7 A7EgTZVcAc+/M3r1OGPYYj1E74f2DXKQn61C3unYhy4s8V4ZDYGmIty1RJb/aUstSht5HJgJPZvYYOe85vrXUCtetEEkNMTfaDYK4T7iK7nuLbS5cNm+7dRfGx/xZvOU5uirA9yHo/h2ZPxnm+WzKD9tZMM/RfJY3iyUb3geUExio1zm1qgBxk9b2a3sO1sziEUM2RXPzapebUgabuxC3IjzzsVsY2xc5dmh/trvUrMSISHOJei8RQXRAG1Ef0H0xe/OkNcYKkHGQjb+kkNEgr1HBiF06TkrXcHaGN0d9evLWkAqdTZB7HHhEXlZmaGnZzqio9JmE8wUcYYpjOdXOHwAnk80Axup5InMfrF+vi19XkVf0pzjy9WaZxOwZOq3hehhgogXmXMIk8JEsGKXzaCQPg9uKlyzvvbftq/PoqK6heAEO9+QTjkAGwW+x7PyToEJmJ4X+7v767HfXjzzZN1srdszm41irmtS5/rVM1wVtnLNYYii7737SidXckLlooCl6L9I+qw6zRe4lkTtFHZPMhHXPmPLSUMPohy8hBE9CbS/8cCnSOMuF7A1OJLuf2VxqzXySc9l4lAc= X-Bogosity: Ham, tests=bogofilter, spamicity=0.000000, version=1.2.4 Sender: owner-linux-mm@kvack.org Precedence: bulk X-Loop: owner-majordomo@kvack.org List-ID: List-Subscribe: List-Unsubscribe: This is only relevant to the two archs that support PUD dax, aka, x86_64 and ppc64. PUD THPs do not yet exist elsewhere, and hugetlb PUDs do not count in this case. DAX have had PUD mappings for years, but change protection path never worked. When the path is triggered in any form (a simple test program would be: call mprotect() on a 1G dev_dax mapping), the kernel will report "bad pud". This patch should fix that. The new change_huge_pud() tries to keep everything simple. For example, it doesn't optimize write bit as that will need even more PUD helpers. It's not too bad anyway to have one more write fault in the worst case once for 1G range; may be a bigger thing for each PAGE_SIZE, though. Neither does it support userfault-wp bits, as there isn't such PUD mappings that is supported; file mappings always need a split there. The same to TLB shootdown: the pmd path (which was for x86 only) has the trick of using _ad() version of pmdp_invalidate*() which can avoid one redundant TLB, but let's also leave that for later. Again, the larger the mapping, the smaller of such effect. Another thing worth mention is this path needs to be careful on handling "retry" event for change_huge_pud() (where it can return 0): it isn't like change_huge_pmd(), as the pmd version is safe with all conditions handled in change_pte_range() later, thanks to Hugh's new pte_offset_map_lock(). In short, change_pte_range() is simply smarter than change_pmd_range() now after the shmem thp collapse rework. For that reason, change_pud_range() will need proper retry if it races with something else when a huge PUD changed from under us. Cc: Dan Williams Cc: Matthew Wilcox Cc: Dave Jiang Cc: Hugh Dickins Cc: Kirill A. Shutemov Cc: Vlastimil Babka Cc: Thomas Gleixner Cc: Ingo Molnar Cc: Borislav Petkov Cc: Dave Hansen Cc: Michael Ellerman Cc: Aneesh Kumar K.V Cc: Oscar Salvador Cc: x86@kernel.org Cc: linuxppc-dev@lists.ozlabs.org Fixes: a00cc7d9dd93 ("mm, x86: add support for PUD-sized transparent hugepages") Fixes: 27af67f35631 ("powerpc/book3s64/mm: enable transparent pud hugepage") Signed-off-by: Peter Xu --- include/linux/huge_mm.h | 24 +++++++++++++++++++ mm/huge_memory.c | 52 +++++++++++++++++++++++++++++++++++++++++ mm/mprotect.c | 40 ++++++++++++++++++++++++------- 3 files changed, 108 insertions(+), 8 deletions(-) diff --git a/include/linux/huge_mm.h b/include/linux/huge_mm.h index 212cca384d7e..b46673689f19 100644 --- a/include/linux/huge_mm.h +++ b/include/linux/huge_mm.h @@ -345,6 +345,17 @@ void split_huge_pmd_address(struct vm_area_struct *vma, unsigned long address, void __split_huge_pud(struct vm_area_struct *vma, pud_t *pud, unsigned long address); +#ifdef CONFIG_HAVE_ARCH_TRANSPARENT_HUGEPAGE_PUD +int change_huge_pud(struct mmu_gather *tlb, struct vm_area_struct *vma, + pud_t *pudp, unsigned long addr, pgprot_t newprot, + unsigned long cp_flags); +#else +static inline int +change_huge_pud(struct mmu_gather *tlb, struct vm_area_struct *vma, + pud_t *pudp, unsigned long addr, pgprot_t newprot, + unsigned long cp_flags) { return 0; } +#endif + #define split_huge_pud(__vma, __pud, __address) \ do { \ pud_t *____pud = (__pud); \ @@ -588,6 +599,19 @@ static inline int next_order(unsigned long *orders, int prev) { return 0; } + +static inline void __split_huge_pud(struct vm_area_struct *vma, pud_t *pud, + unsigned long address) +{ +} + +static inline int change_huge_pud(struct mmu_gather *tlb, + struct vm_area_struct *vma, pud_t *pudp, + unsigned long addr, pgprot_t newprot, + unsigned long cp_flags) +{ + return 0; +} #endif /* CONFIG_TRANSPARENT_HUGEPAGE */ static inline int split_folio_to_list_to_order(struct folio *folio, diff --git a/mm/huge_memory.c b/mm/huge_memory.c index 0fffaa58a47a..af8d83f4ce02 100644 --- a/mm/huge_memory.c +++ b/mm/huge_memory.c @@ -2091,6 +2091,53 @@ int change_huge_pmd(struct mmu_gather *tlb, struct vm_area_struct *vma, return ret; } +/* + * Returns: + * + * - 0: if pud leaf changed from under us + * - 1: if pud can be skipped + * - HPAGE_PUD_NR: if pud was successfully processed + */ +#ifdef CONFIG_HAVE_ARCH_TRANSPARENT_HUGEPAGE_PUD +int change_huge_pud(struct mmu_gather *tlb, struct vm_area_struct *vma, + pud_t *pudp, unsigned long addr, pgprot_t newprot, + unsigned long cp_flags) +{ + struct mm_struct *mm = vma->vm_mm; + pud_t oldpud, entry; + spinlock_t *ptl; + + tlb_change_page_size(tlb, HPAGE_PUD_SIZE); + + /* NUMA balancing doesn't apply to dax */ + if (cp_flags & MM_CP_PROT_NUMA) + return 1; + + /* + * Huge entries on userfault-wp only works with anonymous, while we + * don't have anonymous PUDs yet. + */ + if (WARN_ON_ONCE(cp_flags & MM_CP_UFFD_WP_ALL)) + return 1; + + ptl = __pud_trans_huge_lock(pudp, vma); + if (!ptl) + return 0; + + /* + * Can't clear PUD or it can race with concurrent zapping. See + * change_huge_pmd(). + */ + oldpud = pudp_invalidate(vma, addr, pudp); + entry = pud_modify(oldpud, newprot); + set_pud_at(mm, addr, pudp, entry); + tlb_flush_pud_range(tlb, addr, HPAGE_PUD_SIZE); + + spin_unlock(ptl); + return HPAGE_PUD_NR; +} +#endif + #ifdef CONFIG_USERFAULTFD /* * The PT lock for src_pmd and dst_vma/src_vma (for reading) are locked by @@ -2319,6 +2366,11 @@ void __split_huge_pud(struct vm_area_struct *vma, pud_t *pud, spin_unlock(ptl); mmu_notifier_invalidate_range_end(&range); } +#else +void __split_huge_pud(struct vm_area_struct *vma, pud_t *pud, + unsigned long address) +{ +} #endif /* CONFIG_HAVE_ARCH_TRANSPARENT_HUGEPAGE_PUD */ static void __split_huge_zero_page_pmd(struct vm_area_struct *vma, diff --git a/mm/mprotect.c b/mm/mprotect.c index fb8bf3ff7cd9..694f13b83864 100644 --- a/mm/mprotect.c +++ b/mm/mprotect.c @@ -425,29 +425,53 @@ static inline long change_pud_range(struct mmu_gather *tlb, unsigned long end, pgprot_t newprot, unsigned long cp_flags) { struct mmu_notifier_range range; - pud_t *pud; + pud_t *pudp, pud; unsigned long next; long pages = 0, ret; range.start = 0; - pud = pud_offset(p4d, addr); + pudp = pud_offset(p4d, addr); do { +again: next = pud_addr_end(addr, end); - ret = change_prepare(vma, pud, pmd, addr, cp_flags); - if (ret) - return ret; - if (pud_none_or_clear_bad(pud)) + ret = change_prepare(vma, pudp, pmd, addr, cp_flags); + if (ret) { + pages = ret; + break; + } + + pud = READ_ONCE(*pudp); + if (pud_none(pud)) continue; + if (!range.start) { mmu_notifier_range_init(&range, MMU_NOTIFY_PROTECTION_VMA, 0, vma->vm_mm, addr, end); mmu_notifier_invalidate_range_start(&range); } - pages += change_pmd_range(tlb, vma, pud, addr, next, newprot, + + if (pud_leaf(pud)) { + if ((next - addr != PUD_SIZE) || + pgtable_split_needed(vma, cp_flags)) { + __split_huge_pud(vma, pudp, addr); + goto again; + } else { + ret = change_huge_pud(tlb, vma, pudp, + addr, newprot, cp_flags); + if (ret == 0) + goto again; + /* huge pud was handled */ + if (ret == HPAGE_PUD_NR) + pages += HPAGE_PUD_NR; + continue; + } + } + + pages += change_pmd_range(tlb, vma, pudp, addr, next, newprot, cp_flags); - } while (pud++, addr = next, addr != end); + } while (pudp++, addr = next, addr != end); if (range.start) mmu_notifier_invalidate_range_end(&range);