From patchwork Wed Jul 3 21:29:11 2024 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Peter Xu X-Patchwork-Id: 13722797 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from kanga.kvack.org (kanga.kvack.org [205.233.56.17]) by smtp.lore.kernel.org (Postfix) with ESMTP id 59227C2BD09 for ; Wed, 3 Jul 2024 21:29:31 +0000 (UTC) Received: by kanga.kvack.org (Postfix) id 486CD6B0088; Wed, 3 Jul 2024 17:29:30 -0400 (EDT) Received: by kanga.kvack.org (Postfix, from userid 40) id 40EC46B008A; Wed, 3 Jul 2024 17:29:30 -0400 (EDT) X-Delivered-To: int-list-linux-mm@kvack.org Received: by kanga.kvack.org (Postfix, from userid 63042) id 2B0586B008C; Wed, 3 Jul 2024 17:29:30 -0400 (EDT) X-Delivered-To: linux-mm@kvack.org Received: from relay.hostedemail.com (smtprelay0010.hostedemail.com [216.40.44.10]) by kanga.kvack.org (Postfix) with ESMTP id 059496B0088 for ; Wed, 3 Jul 2024 17:29:29 -0400 (EDT) Received: from smtpin02.hostedemail.com (a10.router.float.18 [10.200.18.1]) by unirelay07.hostedemail.com (Postfix) with ESMTP id A30841606D3 for ; Wed, 3 Jul 2024 21:29:29 +0000 (UTC) X-FDA: 82299732858.02.168FBCA Received: from us-smtp-delivery-124.mimecast.com (us-smtp-delivery-124.mimecast.com [170.10.129.124]) by imf15.hostedemail.com (Postfix) with ESMTP id 93CB1A0005 for ; Wed, 3 Jul 2024 21:29:26 +0000 (UTC) Authentication-Results: imf15.hostedemail.com; dkim=pass header.d=redhat.com header.s=mimecast20190719 header.b="fesftc6/"; spf=pass (imf15.hostedemail.com: domain of peterx@redhat.com designates 170.10.129.124 as permitted sender) smtp.mailfrom=peterx@redhat.com; dmarc=pass (policy=none) header.from=redhat.com ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=hostedemail.com; s=arc-20220608; t=1720042142; h=from:from:sender:reply-to:subject:subject:date:date: message-id:message-id:to:to:cc:cc:mime-version:mime-version: content-type:content-type: content-transfer-encoding:content-transfer-encoding: in-reply-to:in-reply-to:references:references:dkim-signature; bh=xrFUBbcc9D0ORnx0qoNz81Y3Ewmb+xGo+hBykZs/BiM=; b=2v9GYzbj4Gq+Fjp9O2t/yC03iJQ1xRaLhJcwnqJ3xqrkHbMCCAtmuUFKb2V2pPrz8KAWAR 57y+aVwi1pnAzKUSaFJxbYfBRd2+i0GrG0XVAJgPS0xsum20xv6/cAcxSbBcbkf3yz2UxJ 6wWM9NijYRyOHEJBz7wFZ3WB4CG4fTA= ARC-Seal: i=1; s=arc-20220608; d=hostedemail.com; t=1720042142; a=rsa-sha256; cv=none; b=3fqvOak7thRchiWcl5mt2BcHNmEfBLpTVvz4TGGiYbfdyuj3+y/U5brolQttbrkYvCQ2yE 1W43ET4Ot5Y+AVDtN873PMfEecyIWLqbYkfjn0vL23hkPJhFaQtWfnMaXfnMBtOkOxIyQU RQJMROsfYXub5mFYxfNlJ36ujnx37aw= ARC-Authentication-Results: i=1; imf15.hostedemail.com; dkim=pass header.d=redhat.com header.s=mimecast20190719 header.b="fesftc6/"; spf=pass (imf15.hostedemail.com: domain of peterx@redhat.com designates 170.10.129.124 as permitted sender) smtp.mailfrom=peterx@redhat.com; dmarc=pass (policy=none) header.from=redhat.com DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=redhat.com; s=mimecast20190719; t=1720042165; h=from:from:reply-to:subject:subject:date:date:message-id:message-id: to:to:cc:cc:mime-version:mime-version:content-type:content-type: content-transfer-encoding:content-transfer-encoding: in-reply-to:in-reply-to:references:references; bh=xrFUBbcc9D0ORnx0qoNz81Y3Ewmb+xGo+hBykZs/BiM=; b=fesftc6/Q1IKz3esFPW/XG/zN1kDCswfyRrfI7Wu2mHmRw91mQxvMBIAEI1o4men3YwgPf sfjwCZNLDmyRPe1xi45QUIqwu3IlqyKySckwQyVfITVX2863QFYar7yh0X/8wNbB/f8SWU Rf0eIp3A+094Jn1jy49b+FXPvSqFA6k= Received: from mail-qt1-f198.google.com (mail-qt1-f198.google.com [209.85.160.198]) by relay.mimecast.com with ESMTP with STARTTLS (version=TLSv1.3, cipher=TLS_AES_256_GCM_SHA384) id us-mta-3-zroENq4WM6iNumvo7Rx7eg-1; Wed, 03 Jul 2024 17:29:24 -0400 X-MC-Unique: zroENq4WM6iNumvo7Rx7eg-1 Received: by mail-qt1-f198.google.com with SMTP id d75a77b69052e-446405b39baso4234271cf.0 for ; Wed, 03 Jul 2024 14:29:24 -0700 (PDT) X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20230601; t=1720042164; x=1720646964; h=content-transfer-encoding:mime-version:references:in-reply-to :message-id:date:subject:cc:to:from:x-gm-message-state:from:to:cc :subject:date:message-id:reply-to; bh=xrFUBbcc9D0ORnx0qoNz81Y3Ewmb+xGo+hBykZs/BiM=; b=F88ioBV+9v+EGVoJLCTe8/f853I31gZ8CfWOLcdRQrYSVc9x14TUgVgOPgnRZ+dUeb +tPXztAGo6x2sy8pwiNKHgRskVVTWJ8eZ8p4cCOYHAHrL9tY51owoEdA/UYPkpkBivQi CH+cFvy+q5M9/ASCN1X1Yz8ieWKnQskolrcq7mHiTSU4y3ZRn+QjAo9lUtz02YKJdUtP IxvzKFBnAhvp6pKS76D6f0oKFuUdH/tjHjRJCcoH631UFLDGy0QGC30BH/PCziw6Yc/T ahNdiaFumsYi4KoOa/5dWIiNrzXkcIeubW1hzDuaDMt9OZxFY+Bvjix4aK6aIs5KgOkV cM8Q== X-Forwarded-Encrypted: i=1; AJvYcCXKioGm5dnfuuyjfWULf9O1zarGSvRCCr3bMVslLjc2KrbUVkOYjAQkX6p+RaW6ryIrj5CJCkorLa/yVdh0yoD/BrQ= X-Gm-Message-State: AOJu0YxaKaEa1RX9MvNXdoYUY8D3e73sgylV9hG8KSG8A0vXmpJg911l 9nsk+sd4jLZasrd8jPpqg2I25/VGLfL1J4B2J61bmr2aqRFSTEgbMB2G+9/9Lg8QgUmZw2JzVfx VJYoU/4gcZmhnDcIV+eh55/ICgD4K1FKF28pOQs/aMB/iF8b7 X-Received: by 2002:a05:622a:20f:b0:446:395a:37c9 with SMTP id d75a77b69052e-44662e3a591mr137075611cf.4.1720042163850; Wed, 03 Jul 2024 14:29:23 -0700 (PDT) X-Google-Smtp-Source: AGHT+IHMTQtKzW3HcLgwCHDxSHbcORdQ9cM6uS4LGBZ0J8HIBqay3DdOq6gf2nTNP/Qtw7gCSEI+OA== X-Received: by 2002:a05:622a:20f:b0:446:395a:37c9 with SMTP id d75a77b69052e-44662e3a591mr137075451cf.4.1720042163442; Wed, 03 Jul 2024 14:29:23 -0700 (PDT) Received: from x1n.redhat.com (pool-99-254-121-117.cpe.net.cable.rogers.com. [99.254.121.117]) by smtp.gmail.com with ESMTPSA id d75a77b69052e-4465143eb1csm54337481cf.57.2024.07.03.14.29.21 (version=TLS1_3 cipher=TLS_AES_256_GCM_SHA384 bits=256/256); Wed, 03 Jul 2024 14:29:22 -0700 (PDT) From: Peter Xu To: linux-kernel@vger.kernel.org, linux-mm@kvack.org Cc: Matthew Wilcox , Mel Gorman , Dave Jiang , linuxppc-dev@lists.ozlabs.org, Michael Ellerman , Rik van Riel , Vlastimil Babka , Nicholas Piggin , Christophe Leroy , Andrew Morton , Huang Ying , Oscar Salvador , "Aneesh Kumar K . V" , Thomas Gleixner , Dave Hansen , x86@kernel.org, Ingo Molnar , "Kirill A . Shutemov" , Dan Williams , Borislav Petkov , peterx@redhat.com, Hugh Dickins , Rick P Edgecombe Subject: [PATCH v2 1/8] mm/dax: Dump start address in fault handler Date: Wed, 3 Jul 2024 17:29:11 -0400 Message-ID: <20240703212918.2417843-2-peterx@redhat.com> X-Mailer: git-send-email 2.45.0 In-Reply-To: <20240703212918.2417843-1-peterx@redhat.com> References: <20240703212918.2417843-1-peterx@redhat.com> MIME-Version: 1.0 X-Mimecast-Spam-Score: 0 X-Mimecast-Originator: redhat.com X-Rspam-User: X-Rspamd-Server: rspam04 X-Rspamd-Queue-Id: 93CB1A0005 X-Stat-Signature: nadp3yx6hpsgbhgbtmxsj4ggdoqfxipn X-HE-Tag: 1720042166-703463 X-HE-Meta: U2FsdGVkX1/+QFwQ2Do/tKYZ3phBlEbvfR8RV5fpWDoiqFI00HUUM1/CDLs+ZZRGy3SBgMC3dGw8YGTmTD50ZybuX0rBTlJh6cL0jpF8UfGiUvvwP+71jzLNa68olyVnoaf1TCKxzBXk9ITAxnixrQsvDUPUJT/2jN5IfiLEbCPm0GIxxIO4o7jGWL9e/vb4oAmfzmHzNGqExN+lTynBkiBawA2dUnBnuVQS9ssdcPIi3ZAwy/M07X8mnaNTwgaIOXiS62S+KYd6GXKAJM1g7Kp0iXNIW5m30JN55p9guANQ77FdhNJKWoB6NGPYvtSlasn8kysapRI3T+7lRRoNZLLxJC1djLKBEhVbOmTgNOdCgsusgioQD7xByhy3+mWtBxveQeUFpRaiU0ejuVi6Zl494j/VSrSEqIHId3FzSkpDDJ+C4VYVkQ/+Dv/UlO1QclA5jAU4yvOdqM2F8LF2y68p212snGf9w/9UPozfq9mhCxpjJx0AH6yEwqoHXgzPwxVgA9N3J54oK+UBdyZPQBZaCvT8mYPEHWxJOxiqZIi/DY9ln+x6E30u9VVcre1r5Tyei92PfUF4Q/CkOiSJmvu6OsfQtYwPFyWBPy77NJvuv4oPEFjidxKNmiBrtvblOHlgnidk8yUC8ee62lUUrfUhjKujVW0puBJvzmBw0rvL6pQF+aE/qIhc40iGvFbyJYuRRElorP3fsFU/QknIRnpBQZVXvWltDGqMcc3cjbEfYBbqFrAPGZ0jLRqEjDHOiM+VAx8WIKu4uiXUISHGGnZ7cA/x+BarT4xt9CTO0Z1O+k8L+E5UIxuEcLDKJZA5nGWkP9Uzrr9bGd+RGdIo21v3QhtcGwqKiGBhIPc7gC1GlHeYxPuzVyz3Boqg4yeFPzS9YWxmKuycCyKwPNcXtcGZs9JxfgRvZ/Pharpb4xQuXcUxyNrt9iag4x7/AYgyRfiryL3HGoKjRzktiLa oTPO0CE7 4cTatIZ6681jZEpB7Pc+HBjRbox0IKBonI54Kxtf2Ka3aIraQYtrZ2fplOJKE8lSM3qTKHQ1JLqI0Sh9s8JKWIRhQpWwZatxdB2Yqx6KiTb82Pooh49WC+RENxPUf0hWR2knPGLgD+tOv6AQBXGSJLfit2zg4/ph0FUX05+Ww8rEE7ImH8hHkI8diZnRCO/onSvtmG2pisCakG8lQ8uk1Ti/9JDoK+iFHTTKkZ+JYRjaHNgV2cmZ/uG53BBNJE8u5nlrp/bI+EkENRioKVqQbg+XzB5EV8ktQk+EMMZbtWdy3TF46SR4GRlMe8bvmdcFSUNsTtA8LNezemjpMfaxi0/AAMg== X-Bogosity: Ham, tests=bogofilter, spamicity=0.000000, version=1.2.4 Sender: owner-linux-mm@kvack.org Precedence: bulk X-Loop: owner-majordomo@kvack.org List-ID: List-Subscribe: List-Unsubscribe: Currently the dax fault handler dumps the vma range when dynamic debugging enabled. That's mostly not useful. Dump the (aligned) address instead with the order info. Signed-off-by: Peter Xu --- drivers/dax/device.c | 6 +++--- 1 file changed, 3 insertions(+), 3 deletions(-) diff --git a/drivers/dax/device.c b/drivers/dax/device.c index eb61598247a9..714174844ca5 100644 --- a/drivers/dax/device.c +++ b/drivers/dax/device.c @@ -235,9 +235,9 @@ static vm_fault_t dev_dax_huge_fault(struct vm_fault *vmf, unsigned int order) int id; struct dev_dax *dev_dax = filp->private_data; - dev_dbg(&dev_dax->dev, "%s: %s (%#lx - %#lx) order:%d\n", current->comm, - (vmf->flags & FAULT_FLAG_WRITE) ? "write" : "read", - vmf->vma->vm_start, vmf->vma->vm_end, order); + dev_dbg(&dev_dax->dev, "%s: op=%s addr=%#lx order=%d\n", current->comm, + (vmf->flags & FAULT_FLAG_WRITE) ? "write" : "read", + vmf->address & ~((1UL << (order + PAGE_SHIFT)) - 1), order); id = dax_read_lock(); if (order == 0) From patchwork Wed Jul 3 21:29:12 2024 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Peter Xu X-Patchwork-Id: 13722798 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from kanga.kvack.org (kanga.kvack.org [205.233.56.17]) by smtp.lore.kernel.org (Postfix) with ESMTP id 78716C31D97 for ; Wed, 3 Jul 2024 21:29:33 +0000 (UTC) Received: by kanga.kvack.org (Postfix) id 98EA26B008A; Wed, 3 Jul 2024 17:29:31 -0400 (EDT) Received: by kanga.kvack.org (Postfix, from userid 40) id 966F66B008C; Wed, 3 Jul 2024 17:29:31 -0400 (EDT) X-Delivered-To: int-list-linux-mm@kvack.org Received: by kanga.kvack.org (Postfix, from userid 63042) id 79CD96B0092; Wed, 3 Jul 2024 17:29:31 -0400 (EDT) X-Delivered-To: linux-mm@kvack.org Received: from relay.hostedemail.com (smtprelay0010.hostedemail.com [216.40.44.10]) by kanga.kvack.org (Postfix) with ESMTP id 612A86B008A for ; Wed, 3 Jul 2024 17:29:31 -0400 (EDT) Received: from smtpin18.hostedemail.com (a10.router.float.18 [10.200.18.1]) by unirelay07.hostedemail.com (Postfix) with ESMTP id 1CC6A160E9E for ; Wed, 3 Jul 2024 21:29:31 +0000 (UTC) X-FDA: 82299732942.18.E63A358 Received: from us-smtp-delivery-124.mimecast.com (us-smtp-delivery-124.mimecast.com [170.10.133.124]) by imf25.hostedemail.com (Postfix) with ESMTP id 0B06EA0017 for ; Wed, 3 Jul 2024 21:29:28 +0000 (UTC) Authentication-Results: imf25.hostedemail.com; dkim=pass header.d=redhat.com header.s=mimecast20190719 header.b=FPAwxZhl; spf=pass (imf25.hostedemail.com: domain of peterx@redhat.com designates 170.10.133.124 as permitted sender) smtp.mailfrom=peterx@redhat.com; dmarc=pass (policy=none) header.from=redhat.com ARC-Seal: i=1; s=arc-20220608; d=hostedemail.com; t=1720042156; a=rsa-sha256; cv=none; b=L3EmKIE22jDs/4KLZJM331S/AINOfjTP1CuVNT5gGslMquk3s5NibHkmDXCnfdzFMuapft UVn+cIY2EGstOqZJhrfaCQzYVYx0heuHpDDKCROEHC8DI2xDqG4y/A+Mybb0O+sbWwyH83 tX6yZmhbTxKXAJVp0lBlqigFwFJoqEU= ARC-Authentication-Results: i=1; imf25.hostedemail.com; dkim=pass header.d=redhat.com header.s=mimecast20190719 header.b=FPAwxZhl; spf=pass (imf25.hostedemail.com: domain of peterx@redhat.com designates 170.10.133.124 as permitted sender) smtp.mailfrom=peterx@redhat.com; dmarc=pass (policy=none) header.from=redhat.com ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=hostedemail.com; s=arc-20220608; t=1720042156; h=from:from:sender:reply-to:subject:subject:date:date: message-id:message-id:to:to:cc:cc:mime-version:mime-version: content-type:content-type: content-transfer-encoding:content-transfer-encoding: in-reply-to:in-reply-to:references:references:dkim-signature; bh=uDSHiwJLCIYvvzv2DOw1kewa0irBZ6HqHLZp2ORNUK8=; b=kBb1WRgJKUF/BrRD+S08V86bZ4GMpbM7jlZuutVuXiIOvsXGymeSzR2Xo5gEwpRwWpvU5x 4f4rApXNOyMZOJa/+UYw2T9jUZO/GMsKorcqrXWo65hg4QOtsGhRS3fB01dsjiqrtRtZkh 7zG2tefqqwzvZQhSIvHq0dFIcKpCOps= DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=redhat.com; s=mimecast20190719; t=1720042168; h=from:from:reply-to:subject:subject:date:date:message-id:message-id: to:to:cc:cc:mime-version:mime-version:content-type:content-type: content-transfer-encoding:content-transfer-encoding: in-reply-to:in-reply-to:references:references; bh=uDSHiwJLCIYvvzv2DOw1kewa0irBZ6HqHLZp2ORNUK8=; b=FPAwxZhlakaX8irDqeiSJUbBAK0a6XO/kZJUkmo/l1fIh/Emb7bdnHlyv61MnNjx9Soq1i UQ6obd3e6s68xJifAIP+2S6ucK5Z53P+59Y4s4ks5RStsSgHGFkeWV7Q3zErbknu9vSzrL CIhQu5un+ieWcWMJyM2Hw1NHPRQUYrw= Received: from mail-qt1-f198.google.com (mail-qt1-f198.google.com [209.85.160.198]) by relay.mimecast.com with ESMTP with STARTTLS (version=TLSv1.3, cipher=TLS_AES_256_GCM_SHA384) id us-mta-516-8kQ0VApiPgypKsWY1gy0pg-1; Wed, 03 Jul 2024 17:29:26 -0400 X-MC-Unique: 8kQ0VApiPgypKsWY1gy0pg-1 Received: by mail-qt1-f198.google.com with SMTP id d75a77b69052e-4464454e329so1957721cf.1 for ; Wed, 03 Jul 2024 14:29:26 -0700 (PDT) X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20230601; t=1720042166; x=1720646966; h=content-transfer-encoding:mime-version:references:in-reply-to :message-id:date:subject:cc:to:from:x-gm-message-state:from:to:cc :subject:date:message-id:reply-to; bh=uDSHiwJLCIYvvzv2DOw1kewa0irBZ6HqHLZp2ORNUK8=; b=GKexdPEncSrcXbpefNcOovB/cKVgqbikGSrtbi4cqFpQFmEUrLCPAO2iWPmjqFW+DS G6tv05s0yiQdje5RICoel/nQgznGttQcsxfTEszyq9iZ0iNxWBscl8pvQcbKzIu05pXs +1AnSq+0q3a7fyMo4tXy0byRe5uFpRz0j0VKHGMSbKzTTUrjjOAhrQETvf+Rh/DB/FpQ Er3qRD5FxNQXZBw6ELCDpIQzCy145oAba8Flxsk2UKEcK7Wa31ZDyPCjeLa6AEPKxXou FbrT4q2A99e5RI8OuxOpZNq49yMZVTBY65dUn8j0G+dr6OM7KdC1IYKLJgCopku8DoBD A7Bw== X-Forwarded-Encrypted: i=1; AJvYcCXnQtuw0Kza0Him4lx2yqyGTr29lIIkgO4FhLarUN3joq7L5//6cPmykbY+bk4f4c0KbqAH5U4Qf14OQzmrLqPRZkI= X-Gm-Message-State: AOJu0YwO/CICJ766kp4vsMQHLVKe7AitZQXUknh8adBsqTbc6atYFGTm iNTYStkyfH5HQSai8SI7XQXL6/rd4pQhbgrM7VLZebHV/qvHNhPmaIxb8TmrJZ0HYQV0om9/pCl dlAlvvhWtgDSY8cow4FdxdC9juval85JbplaSmu4Qdoreeko0 X-Received: by 2002:a05:622a:1a29:b0:446:5c96:3e39 with SMTP id d75a77b69052e-44662c7b9b6mr149928221cf.2.1720042165791; Wed, 03 Jul 2024 14:29:25 -0700 (PDT) X-Google-Smtp-Source: AGHT+IFMMhz5U/uaY0swHE/aSBQQusLb0Xfs1HUkfM825Vkmolo2nubsLeOfdQnkewhNugg6U8gw9g== X-Received: by 2002:a05:622a:1a29:b0:446:5c96:3e39 with SMTP id d75a77b69052e-44662c7b9b6mr149928031cf.2.1720042165404; Wed, 03 Jul 2024 14:29:25 -0700 (PDT) Received: from x1n.redhat.com (pool-99-254-121-117.cpe.net.cable.rogers.com. [99.254.121.117]) by smtp.gmail.com with ESMTPSA id d75a77b69052e-4465143eb1csm54337481cf.57.2024.07.03.14.29.23 (version=TLS1_3 cipher=TLS_AES_256_GCM_SHA384 bits=256/256); Wed, 03 Jul 2024 14:29:25 -0700 (PDT) From: Peter Xu To: linux-kernel@vger.kernel.org, linux-mm@kvack.org Cc: Matthew Wilcox , Mel Gorman , Dave Jiang , linuxppc-dev@lists.ozlabs.org, Michael Ellerman , Rik van Riel , Vlastimil Babka , Nicholas Piggin , Christophe Leroy , Andrew Morton , Huang Ying , Oscar Salvador , "Aneesh Kumar K . V" , Thomas Gleixner , Dave Hansen , x86@kernel.org, Ingo Molnar , "Kirill A . Shutemov" , Dan Williams , Borislav Petkov , peterx@redhat.com, Hugh Dickins , Rick P Edgecombe , Alex Thorlton Subject: [PATCH v2 2/8] mm/mprotect: Remove NUMA_HUGE_PTE_UPDATES Date: Wed, 3 Jul 2024 17:29:12 -0400 Message-ID: <20240703212918.2417843-3-peterx@redhat.com> X-Mailer: git-send-email 2.45.0 In-Reply-To: <20240703212918.2417843-1-peterx@redhat.com> References: <20240703212918.2417843-1-peterx@redhat.com> MIME-Version: 1.0 X-Mimecast-Spam-Score: 0 X-Mimecast-Originator: redhat.com X-Stat-Signature: d14jrsiohfu4iddwhzbk1e9iq8y9g5ng X-Rspamd-Queue-Id: 0B06EA0017 X-Rspam-User: X-Rspamd-Server: rspam10 X-HE-Tag: 1720042168-456845 X-HE-Meta: U2FsdGVkX19BrY6bsKlUsqwHissgtvqXwSU5wwromwalC8eNPbjChyRgmn4m576jMJYMkwD4dbR5o0nFXRTIbNKnDdGuHeor5kK191NSDLc33avq8WklHsyuGO8l0ppRnPS5VPgIehJIBvimVEjwYlJsoihdeVJ3AXFNEuU7BpAlnQYvOueQeGrwtQNqtwWKYyojAUj5uVkGEdDF1CqwX4WMZUbeqDM7V9gp2c1PODoKuNSrQznErlXKI21/WuWEyHP7c+V5aSLURAcllujPjaH60NO114K5PX73oU/ACC3mNIkG7y1kRWH/XykFYf9z6VnxRWFMqB85azzOSwx6tV4F7L+5hEagdRtsq+tiPeBSk/T7dN/QIkQcg9pxzLBXf9gaSJD3nsg1tLmNt/0ydCSIj/7wd2uMrbEcq8GDBh4RZWXmGRdSPZLw8qyeE6QH3ImntJ5/MT7XN7kmgaixJZskL0y6XWdnKsQe5rNvm5OLmLjVosMNDnfFKAjrh8mXz5murVvdBk7SYfNk7JpDCmuZkoAtBMMl2/FXMtVUaGH8ymPIYpCGnA23YG4kZ0GZna8lf56f4hKNMwOnHzCVTd2GNqAmcTOLzD0lNE2ma7n+vzK9JiNV9S1b6Tv+2LmlT8GeWU7jpfxPG0BVOX4M1Sy3ffTysQRg5sw7YEkDz8FOoQNIZntpqKH8umL0Rn7ewWFIqGVwMtrjv3He6x9nRj0UVrdRlDnwxEE0/aGnYKz3+sML1sgR8zQBJxBIeyNPZVdYyUIq88Dm2s7zie8ZjUp2y4r+M3/6Br7VP3zDpLBfRJfyxOO+dp4NGEEhInGqsdrnOK9XDN6sAY1JY5zgCmhksQiG+QGCTG6R+Cjd7Sar06Wfil0stRqd5efnJDICHIuECSBGFY0OVM9AZiaMdE0rW2Vj0aIGM35UneH+iRO3CCP+LfP6SiGqnlXafoanxYtlwN0AGyf6GBn09Ce tKgIcDhg DsjhCPnZ+DFvfUV5To1uhISz5AVctOcUFbsIHE55mLprpM+9LAqeOk0DtwOjDrur6eCG9vAvcna9NdYkTcKEE+q+tcbyKs6zlZ0mkWzIeZJArBcbWcX+itr2pahr0tFyOwTP04l9x6lG43jV7s6KQPdFKf3JqW3VLvj+TB1mzV2anKm1GpY1p1FpSDbhsqklh3oA6C7i0XuX4FibFznEe/bu1xb6gnGh3h0ehMLrFv+GnbGdQy8XBJJriF74z65vEbVtSHEIPpSDRekzTfiIN2IZY1kdVAfik7USnOt29UXZFAz2iBtXJwqJAKu1Ruj+Rdn+y9kH6UghTbMEnk6SIg4VHi7eskWsI5N+g4+E4d+VfCuRuK9WbpKIIp8zdZJgK5meefWdbeK29qg/09czFDM/UzluT95P6zGnB0J9kAs/+WEoBOISedg/GjR9F4UuYin8ToBH+sBU/Owa8Jf00fV1wnt4noXIdomVdGQpn9wabD+wqzy0O95+U1pOGaIwpzoYoOsoi90yyYXbT1nfHb0KmWL9vWqLMFORo X-Bogosity: Ham, tests=bogofilter, spamicity=0.000000, version=1.2.4 Sender: owner-linux-mm@kvack.org Precedence: bulk X-Loop: owner-majordomo@kvack.org List-ID: List-Subscribe: List-Unsubscribe: In 2013, commit 72403b4a0fbd ("mm: numa: return the number of base pages altered by protection changes") introduced "numa_huge_pte_updates" vmstat entry, trying to capture how many huge ptes (in reality, PMD thps at that time) are marked by NUMA balancing. This patch proposes to remove it for some reasons. Firstly, the name is misleading. We can have more than one way to have a "huge pte" at least nowadays, and that's also the major goal of this patch, where it paves way for PUD handling in change protection code paths. PUDs are coming not only for dax (which has already came and yet broken..), but also for pfnmaps and hugetlb pages. The name will simply stop making sense when PUD will start to be involved in mprotect() world. It'll also make it not reasonable either if we boost the counter for both pmd/puds. In short, current accounting won't be right when PUD comes, so the scheme was only suitable at that point in time where PUD wasn't even possible. Secondly, the accounting was simply not right from the start as long as it was also affected by other call sites besides NUMA. mprotect() is one, while userfaultfd-wp also leverages change protection path to modify pgtables. If it wants to do right it needs to check the caller but it never did; at least mprotect() should be there even in 2013. It gives me the impression that nobody is seriously using this field, and it's also impossible to be serious. We may want to do it right if any NUMA developers would like it to exist, but we should do that with all above resolved, on both considering PUDs, but also on correct accountings. That should be able to be done on top when there's a real need of such. Cc: Huang Ying Cc: Mel Gorman Cc: Alex Thorlton Cc: Rik van Riel Signed-off-by: Peter Xu --- include/linux/vm_event_item.h | 1 - mm/mprotect.c | 8 +------- mm/vmstat.c | 1 - 3 files changed, 1 insertion(+), 9 deletions(-) diff --git a/include/linux/vm_event_item.h b/include/linux/vm_event_item.h index 747943bc8cc2..2a3797fb6742 100644 --- a/include/linux/vm_event_item.h +++ b/include/linux/vm_event_item.h @@ -59,7 +59,6 @@ enum vm_event_item { PGPGIN, PGPGOUT, PSWPIN, PSWPOUT, OOM_KILL, #ifdef CONFIG_NUMA_BALANCING NUMA_PTE_UPDATES, - NUMA_HUGE_PTE_UPDATES, NUMA_HINT_FAULTS, NUMA_HINT_FAULTS_LOCAL, NUMA_PAGE_MIGRATE, diff --git a/mm/mprotect.c b/mm/mprotect.c index 222ab434da54..21172272695e 100644 --- a/mm/mprotect.c +++ b/mm/mprotect.c @@ -363,7 +363,6 @@ static inline long change_pmd_range(struct mmu_gather *tlb, pmd_t *pmd; unsigned long next; long pages = 0; - unsigned long nr_huge_updates = 0; struct mmu_notifier_range range; range.start = 0; @@ -411,11 +410,8 @@ static inline long change_pmd_range(struct mmu_gather *tlb, ret = change_huge_pmd(tlb, vma, pmd, addr, newprot, cp_flags); if (ret) { - if (ret == HPAGE_PMD_NR) { + if (ret == HPAGE_PMD_NR) pages += HPAGE_PMD_NR; - nr_huge_updates++; - } - /* huge pmd was handled */ goto next; } @@ -435,8 +431,6 @@ static inline long change_pmd_range(struct mmu_gather *tlb, if (range.start) mmu_notifier_invalidate_range_end(&range); - if (nr_huge_updates) - count_vm_numa_events(NUMA_HUGE_PTE_UPDATES, nr_huge_updates); return pages; } diff --git a/mm/vmstat.c b/mm/vmstat.c index 73d791d1caad..53656227f70d 100644 --- a/mm/vmstat.c +++ b/mm/vmstat.c @@ -1313,7 +1313,6 @@ const char * const vmstat_text[] = { #ifdef CONFIG_NUMA_BALANCING "numa_pte_updates", - "numa_huge_pte_updates", "numa_hint_faults", "numa_hint_faults_local", "numa_pages_migrated", From patchwork Wed Jul 3 21:29:13 2024 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Peter Xu X-Patchwork-Id: 13722799 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from kanga.kvack.org (kanga.kvack.org [205.233.56.17]) by smtp.lore.kernel.org (Postfix) with ESMTP id 75C77C2BD09 for ; Wed, 3 Jul 2024 21:29:36 +0000 (UTC) Received: by kanga.kvack.org (Postfix) id 85A356B008C; Wed, 3 Jul 2024 17:29:33 -0400 (EDT) Received: by kanga.kvack.org (Postfix, from userid 40) id 7DFC86B0092; Wed, 3 Jul 2024 17:29:33 -0400 (EDT) X-Delivered-To: int-list-linux-mm@kvack.org Received: by kanga.kvack.org (Postfix, from userid 63042) id 5E3FC6B0093; Wed, 3 Jul 2024 17:29:33 -0400 (EDT) X-Delivered-To: linux-mm@kvack.org Received: from relay.hostedemail.com (smtprelay0011.hostedemail.com [216.40.44.11]) by kanga.kvack.org (Postfix) with ESMTP id 3D6CC6B008C for ; Wed, 3 Jul 2024 17:29:33 -0400 (EDT) Received: from smtpin09.hostedemail.com (a10.router.float.18 [10.200.18.1]) by unirelay08.hostedemail.com (Postfix) with ESMTP id E4025140A9C for ; Wed, 3 Jul 2024 21:29:32 +0000 (UTC) X-FDA: 82299732984.09.6EDF4ED Received: from us-smtp-delivery-124.mimecast.com (us-smtp-delivery-124.mimecast.com [170.10.129.124]) by imf17.hostedemail.com (Postfix) with ESMTP id CBFCC4000F for ; Wed, 3 Jul 2024 21:29:30 +0000 (UTC) Authentication-Results: imf17.hostedemail.com; dkim=pass header.d=redhat.com header.s=mimecast20190719 header.b="iyegTmQ/"; spf=pass (imf17.hostedemail.com: domain of peterx@redhat.com designates 170.10.129.124 as permitted sender) smtp.mailfrom=peterx@redhat.com; dmarc=pass (policy=none) header.from=redhat.com ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=hostedemail.com; s=arc-20220608; t=1720042147; h=from:from:sender:reply-to:subject:subject:date:date: message-id:message-id:to:to:cc:cc:mime-version:mime-version: content-type:content-type: content-transfer-encoding:content-transfer-encoding: in-reply-to:in-reply-to:references:references:dkim-signature; bh=VpPwnVRxGccST937wjacfAT3ndiJzOREHKlh4Jm8zSE=; b=VZohN9q32WJvL4xGHjsrQ0t0JqMuV9egnFGqJv49ogaJm4uPxyPpgR4HZSOE2ecEkH414l lgMjjwNNPQjV9+cBtouzD/CQQJrwLBCKzJoOdK2nFojiWR60mFRYK+JmNnC/XPMhOEX//X TMU0lyLlCZcugCFYeYdRk2O+NIKbRjY= ARC-Seal: i=1; s=arc-20220608; d=hostedemail.com; t=1720042147; a=rsa-sha256; cv=none; b=xNT3hs8p3v/KexEIfPNeA2kMYVU1vShfKpRuODjE4A5IJ4Wy/4UcfFoxI71xIi+bMfTiZD UznPGTTrw04xebxtMDmENzx0t0uOC600Q5ZOnn9zVehW3D7poI4pqKgwktQZLSlkku2KOC cargu+O67q7j/9q/P6jbMzMYCvefEzI= ARC-Authentication-Results: i=1; imf17.hostedemail.com; dkim=pass header.d=redhat.com header.s=mimecast20190719 header.b="iyegTmQ/"; spf=pass (imf17.hostedemail.com: domain of peterx@redhat.com designates 170.10.129.124 as permitted sender) smtp.mailfrom=peterx@redhat.com; dmarc=pass (policy=none) header.from=redhat.com DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=redhat.com; s=mimecast20190719; t=1720042170; h=from:from:reply-to:subject:subject:date:date:message-id:message-id: to:to:cc:cc:mime-version:mime-version:content-type:content-type: content-transfer-encoding:content-transfer-encoding: in-reply-to:in-reply-to:references:references; bh=VpPwnVRxGccST937wjacfAT3ndiJzOREHKlh4Jm8zSE=; b=iyegTmQ/+x//h1iLh/xkzxuRmTa0+vAZP4ASN1cINCV5f2zK6uVPV8byCH2inkTtA1VVCj ctFCfqluv8ijojQ7RkBONQMVRCE9gvKDTICF9LQ6B2NVZQILZ5KH+wZe6KlmB/rjvgsCNc S05SrR0SKPfhiuPMyqzjHaWba4rDS84= Received: from mail-oo1-f71.google.com (mail-oo1-f71.google.com [209.85.161.71]) by relay.mimecast.com with ESMTP with STARTTLS (version=TLSv1.3, cipher=TLS_AES_256_GCM_SHA384) id us-mta-669-MYDrrYEwOHSX9txzf9KJtw-1; Wed, 03 Jul 2024 17:29:29 -0400 X-MC-Unique: MYDrrYEwOHSX9txzf9KJtw-1 Received: by mail-oo1-f71.google.com with SMTP id 006d021491bc7-5c21b6db74dso1147837eaf.3 for ; Wed, 03 Jul 2024 14:29:28 -0700 (PDT) X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20230601; t=1720042168; x=1720646968; h=content-transfer-encoding:mime-version:references:in-reply-to :message-id:date:subject:cc:to:from:x-gm-message-state:from:to:cc :subject:date:message-id:reply-to; bh=VpPwnVRxGccST937wjacfAT3ndiJzOREHKlh4Jm8zSE=; b=PcKjamLUgRMXGrTKLpP24VKB1XhXJIUyvG28iaPS9UC57/tyRS4gyMPGH+mb6zo3YA tdLRvkNshnJ05qMt1T3LzgWg9K4rY5iGCzmQovvRSzMZ/NHsB/GEA7MF+ra9OMWKow54 TBtk+QabjP0MHppK0l3abIqyAdZ9U+5iOidD06hElNkgM0A9XMKHkVa3CAaZAaoKfMMZ EG+/Lx4cJggYKLhk8TUE7ATvbjP9uHuSAltq7PvpqpEurWdDSXhZzbC4HdEQSeZBlgx3 9Ey38y7FgqelG2GO3NOywVsPYf7EAEyh+IQo1BTf8udgxtn82x/OEF+4PlfY4KxKtWN+ Ukow== X-Forwarded-Encrypted: i=1; AJvYcCVQonNRvPFmHu7K0w3oVVpFna4JlIBNa3p/bSxc77c/0eUnAZ3xGCI5ERuY4Qti7PMtIYcrqou+akS+liSc30wlwb8= X-Gm-Message-State: AOJu0YyxVuQ2uKcI94APxuLvj/TWTNuMhRk3E4kpyzbAZpfbv5cHxFV9 QqUUjHxFDPEv7b+rKjom5Et1FYoirx1+1Y0eU4JRQTZNc3X7y+F6jjToP52cK1Nu4ZJFeLK2oVT +l/N1VOoStYrQYEScLLM6dzIrbxVdPOw9G7iu5SqXej80j/+v X-Received: by 2002:a05:6870:f112:b0:25e:15e1:35c6 with SMTP id 586e51a60fabf-25e15e13956mr2491359fac.0.1720042168250; Wed, 03 Jul 2024 14:29:28 -0700 (PDT) X-Google-Smtp-Source: AGHT+IFy7qnRAv+DeYDujdDipaXvn8J32B6JV16U/qU1i2FGfDDVRUJs42vciaW1uTjUnBxFRGp+Ow== X-Received: by 2002:a05:6870:f112:b0:25e:15e1:35c6 with SMTP id 586e51a60fabf-25e15e13956mr2491322fac.0.1720042167878; Wed, 03 Jul 2024 14:29:27 -0700 (PDT) Received: from x1n.redhat.com (pool-99-254-121-117.cpe.net.cable.rogers.com. [99.254.121.117]) by smtp.gmail.com with ESMTPSA id d75a77b69052e-4465143eb1csm54337481cf.57.2024.07.03.14.29.25 (version=TLS1_3 cipher=TLS_AES_256_GCM_SHA384 bits=256/256); Wed, 03 Jul 2024 14:29:27 -0700 (PDT) From: Peter Xu To: linux-kernel@vger.kernel.org, linux-mm@kvack.org Cc: Matthew Wilcox , Mel Gorman , Dave Jiang , linuxppc-dev@lists.ozlabs.org, Michael Ellerman , Rik van Riel , Vlastimil Babka , Nicholas Piggin , Christophe Leroy , Andrew Morton , Huang Ying , Oscar Salvador , "Aneesh Kumar K . V" , Thomas Gleixner , Dave Hansen , x86@kernel.org, Ingo Molnar , "Kirill A . Shutemov" , Dan Williams , Borislav Petkov , peterx@redhat.com, Hugh Dickins , Rick P Edgecombe , kvm@vger.kernel.org, Sean Christopherson , Paolo Bonzini , David Rientjes Subject: [PATCH v2 3/8] mm/mprotect: Push mmu notifier to PUDs Date: Wed, 3 Jul 2024 17:29:13 -0400 Message-ID: <20240703212918.2417843-4-peterx@redhat.com> X-Mailer: git-send-email 2.45.0 In-Reply-To: <20240703212918.2417843-1-peterx@redhat.com> References: <20240703212918.2417843-1-peterx@redhat.com> MIME-Version: 1.0 X-Mimecast-Spam-Score: 0 X-Mimecast-Originator: redhat.com X-Rspamd-Queue-Id: CBFCC4000F X-Stat-Signature: qq9ehi7y8hfa4be3jku6m3pkzjnhz3hg X-Rspamd-Server: rspam09 X-Rspam-User: X-HE-Tag: 1720042170-956880 X-HE-Meta: U2FsdGVkX19VCZ7JHSVaEez0KNoLYjzLrn6owz4wpoM6JBDDwS4AYcYA6vwm7vKu+qMi9oz1fV9tF0k98ZUoP0lNL1XKIX1YmdGYhIyXCJmaNJ+rEW5piB6e8n9TJvHSqcEKBU6+3qQ3IGqNZeKSA3Aja9FP3SVdsWUTwBRzJ/XRCH1io0PJ/4Vakjdpc+8rGB1ZFC0BdObkvTgoWsdOPFDTFKZiqIFFJbCYqjggtQfslSADHMOaMi1jkHpQ247Sef8JHOZ+b8Yx45RPvCDVcHI60BVQ7LKPrUUqEkpkcH9ioUMeeQJefkSAJ/P2g/XiWuqZCjPYfPKoTFcLuD9koKR/mi3/DHIyWGqZD98sh/zGugYzKeWs6k0lQl74AdH+H72zhfxXnvF+kmqIPZpHpKVfXIv4aeCIXjYl4AKZlHH8qEsUKaub5kOmyHaGGJvwgQZ9xWglax0wSBb2/IAtJ1G1sFAfHcrx0vGJwUyrIJYdo7phOPzWW5AAMoTPoStv/dIjjlLi6b5g3wJ1QmIkAnkyBXCYKodIdgOPivuIu4mNRErYV7bdp0gUoO7yO80/0Vi74BDKiWCWI9ZcWkqisZXz7nRkozGhfMwiM82ISEnMqEnmeKE9s+KTFx0/i0fhBrZUQYESJK46yAO9zTuohxCD2jYr+3nBpofJ88s/9yy216XGJfQ8XGKL58NGyyXeehUziuVfxjlTVlfDFPCiYoVC+cHPcqf9bRaseuTfcqWnN1WaWjJDeGZBnlOh8sUFvMGYyNUa52Q4rsdesyTWLqXea+7i+Kd/XW6yxxCBpoRh//qAFNBXmgKOEpuWC7wjUYib4CzGBGHE0Ea8aPxQm+6o1a/wrfAllww0IgGHeXkXXnSbaRgHuhSZDus1ZTL3j6OTofUTsanRNdH3tv2b7Bu2yOke8cUZi8T4ta5fb34thBrSG/+HclyEnWbHvrvFd/7fBkY+VDDzPx13D7a e3GOmMad vPpXp7j/8y5PwDlO9y+PQ6UhtCxN6Z+uvlfJhartzM1a7Z4SCqoLcf/hQ3i4DBIn6W7LAAUzn+BpteTzndbeysUH0vUGwok8UzbkVWAHd7rC8BaR6TdS3G9glmoifE7JZb8B6fWLyQiPdek54cdtTyW0wtsGb7ISkZVWNIbRlLDg0sblezxuWGM4O2d55uK2j24wvK1jyxzlBEsllnxSMoxPaWSh3wWhcN75MaQkHcFuDwuchy43xDe6pXAHVDDjT5OJhtKM8wvVtpGo0ElUe9tqWjZxUkID4wMVcLZ5gGqBVYB0lp7R8Gdhc6FJp1kOdxpcm6MajRU1Q7WPGgP6itNHzbh6cx8b4KJo9+Wbzpk6irm1zYEaU+FvzcjV0QgDQ6UvPi5I0WFryXy9a0L0jlaG2ZuhRdfzdEGGa/7EqquL6EteioKBaxMF3BVqg54cXXgznLPPLkC8PdoUS8OqEvBItGto+BsyWZDny X-Bogosity: Ham, tests=bogofilter, spamicity=0.000000, version=1.2.4 Sender: owner-linux-mm@kvack.org Precedence: bulk X-Loop: owner-majordomo@kvack.org List-ID: List-Subscribe: List-Unsubscribe: mprotect() does mmu notifiers in PMD levels. It's there since 2014 of commit a5338093bfb4 ("mm: move mmu notifier call from change_protection to change_pmd_range"). At that time, the issue was that NUMA balancing can be applied on a huge range of VM memory, even if nothing was populated. The notification can be avoided in this case if no valid pmd detected, which includes either THP or a PTE pgtable page. Now to pave way for PUD handling, this isn't enough. We need to generate mmu notifications even on PUD entries properly. mprotect() is currently broken on PUD (e.g., one can easily trigger kernel error with dax 1G mappings already), this is the start to fix it. To fix that, this patch proposes to push such notifications to the PUD layers. There is risk on regressing the problem Rik wanted to resolve before, but I think it shouldn't really happen, and I still chose this solution because of a few reasons: 1) Consider a large VM that should definitely contain more than GBs of memory, it's highly likely that PUDs are also none. In this case there will have no regression. 2) KVM has evolved a lot over the years to get rid of rmap walks, which might be the major cause of the previous soft-lockup. At least TDP MMU already got rid of rmap as long as not nested (which should be the major use case, IIUC), then the TDP MMU pgtable walker will simply see empty VM pgtable (e.g. EPT on x86), the invalidation of a full empty region in most cases could be pretty fast now, comparing to 2014. 3) KVM has explicit code paths now to even give way for mmu notifiers just like this one, e.g. in commit d02c357e5bfa ("KVM: x86/mmu: Retry fault before acquiring mmu_lock if mapping is changing"). It'll also avoid contentions that may also contribute to a soft-lockup. 4) Stick with PMD layer simply don't work when PUD is there... We need one way or another to fix PUD mappings on mprotect(). Pushing it to PUD should be the safest approach as of now, e.g. there's yet no sign of huge P4D coming on any known archs. Cc: kvm@vger.kernel.org Cc: Sean Christopherson Cc: Paolo Bonzini Cc: David Rientjes Cc: Rik van Riel Signed-off-by: Peter Xu --- mm/mprotect.c | 26 ++++++++++++-------------- 1 file changed, 12 insertions(+), 14 deletions(-) diff --git a/mm/mprotect.c b/mm/mprotect.c index 21172272695e..fb8bf3ff7cd9 100644 --- a/mm/mprotect.c +++ b/mm/mprotect.c @@ -363,9 +363,6 @@ static inline long change_pmd_range(struct mmu_gather *tlb, pmd_t *pmd; unsigned long next; long pages = 0; - struct mmu_notifier_range range; - - range.start = 0; pmd = pmd_offset(pud, addr); do { @@ -383,14 +380,6 @@ static inline long change_pmd_range(struct mmu_gather *tlb, if (pmd_none(*pmd)) goto next; - /* invoke the mmu notifier if the pmd is populated */ - if (!range.start) { - mmu_notifier_range_init(&range, - MMU_NOTIFY_PROTECTION_VMA, 0, - vma->vm_mm, addr, end); - mmu_notifier_invalidate_range_start(&range); - } - _pmd = pmdp_get_lockless(pmd); if (is_swap_pmd(_pmd) || pmd_trans_huge(_pmd) || pmd_devmap(_pmd)) { if ((next - addr != HPAGE_PMD_SIZE) || @@ -428,9 +417,6 @@ static inline long change_pmd_range(struct mmu_gather *tlb, cond_resched(); } while (pmd++, addr = next, addr != end); - if (range.start) - mmu_notifier_invalidate_range_end(&range); - return pages; } @@ -438,10 +424,13 @@ static inline long change_pud_range(struct mmu_gather *tlb, struct vm_area_struct *vma, p4d_t *p4d, unsigned long addr, unsigned long end, pgprot_t newprot, unsigned long cp_flags) { + struct mmu_notifier_range range; pud_t *pud; unsigned long next; long pages = 0, ret; + range.start = 0; + pud = pud_offset(p4d, addr); do { next = pud_addr_end(addr, end); @@ -450,10 +439,19 @@ static inline long change_pud_range(struct mmu_gather *tlb, return ret; if (pud_none_or_clear_bad(pud)) continue; + if (!range.start) { + mmu_notifier_range_init(&range, + MMU_NOTIFY_PROTECTION_VMA, 0, + vma->vm_mm, addr, end); + mmu_notifier_invalidate_range_start(&range); + } pages += change_pmd_range(tlb, vma, pud, addr, next, newprot, cp_flags); } while (pud++, addr = next, addr != end); + if (range.start) + mmu_notifier_invalidate_range_end(&range); + return pages; } From patchwork Wed Jul 3 21:29:14 2024 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Peter Xu X-Patchwork-Id: 13722800 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from kanga.kvack.org (kanga.kvack.org [205.233.56.17]) by smtp.lore.kernel.org (Postfix) with ESMTP id D0CA9C31D97 for ; Wed, 3 Jul 2024 21:29:39 +0000 (UTC) Received: by kanga.kvack.org (Postfix) id 856C86B0093; Wed, 3 Jul 2024 17:29:36 -0400 (EDT) Received: by kanga.kvack.org (Postfix, from userid 40) id 8071A6B0095; Wed, 3 Jul 2024 17:29:36 -0400 (EDT) X-Delivered-To: int-list-linux-mm@kvack.org Received: by kanga.kvack.org (Postfix, from userid 63042) id 5E49A6B0096; Wed, 3 Jul 2024 17:29:36 -0400 (EDT) X-Delivered-To: linux-mm@kvack.org Received: from relay.hostedemail.com (smtprelay0012.hostedemail.com [216.40.44.12]) by kanga.kvack.org (Postfix) with ESMTP id 3DED36B0093 for ; Wed, 3 Jul 2024 17:29:36 -0400 (EDT) Received: from smtpin18.hostedemail.com (a10.router.float.18 [10.200.18.1]) by unirelay06.hostedemail.com (Postfix) with ESMTP id BC9E7A1F9E for ; Wed, 3 Jul 2024 21:29:35 +0000 (UTC) X-FDA: 82299733110.18.AA02829 Received: from us-smtp-delivery-124.mimecast.com (us-smtp-delivery-124.mimecast.com [170.10.129.124]) by imf01.hostedemail.com (Postfix) with ESMTP id A0F4B4000C for ; Wed, 3 Jul 2024 21:29:32 +0000 (UTC) Authentication-Results: imf01.hostedemail.com; dkim=pass header.d=redhat.com header.s=mimecast20190719 header.b="Km3/jtgH"; spf=pass (imf01.hostedemail.com: domain of peterx@redhat.com designates 170.10.129.124 as permitted sender) smtp.mailfrom=peterx@redhat.com; dmarc=pass (policy=none) header.from=redhat.com ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=hostedemail.com; s=arc-20220608; t=1720042161; h=from:from:sender:reply-to:subject:subject:date:date: message-id:message-id:to:to:cc:cc:mime-version:mime-version: content-type:content-type: content-transfer-encoding:content-transfer-encoding: in-reply-to:in-reply-to:references:references:dkim-signature; bh=DPSgdrK+/9txvlBVeWURx0wHDXuvLt3nB8aMlUhEkV8=; b=Euo9iBtlaj7KZ/svDCbOhcfV7IHUabstr0wWq0fBtoexqZ9dkw4doUMXV4N07hv//i8hLl 1K+dZeiI5WYSEeAVMqmUhgqwLT1lGk8gq1Hoi/snRRxvlqUHEpz0d3jLYYIUjODMVk+N6h LH43azyJlAR5+KLOPJqYKAWyr67yqgo= ARC-Authentication-Results: i=1; imf01.hostedemail.com; dkim=pass header.d=redhat.com header.s=mimecast20190719 header.b="Km3/jtgH"; spf=pass (imf01.hostedemail.com: domain of peterx@redhat.com designates 170.10.129.124 as permitted sender) smtp.mailfrom=peterx@redhat.com; dmarc=pass (policy=none) header.from=redhat.com ARC-Seal: i=1; s=arc-20220608; d=hostedemail.com; t=1720042161; a=rsa-sha256; cv=none; b=16VlfOwdTAL3pInHVZ3CRskPRaekf7zrHf3umOH3zuNWptLek8qKqKzGfymOYDAbLgo8YA oq9tTLRnbeVXQIb8uXIOEo9WkeUkL4rwpCYgN5fTuQzr8n0E3YMPky9Pwx0JReW17tj9/G qt4upIy4Gd0CGGoOv/cV1KnGwQVMWq4= DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=redhat.com; s=mimecast20190719; t=1720042172; h=from:from:reply-to:subject:subject:date:date:message-id:message-id: to:to:cc:cc:mime-version:mime-version:content-type:content-type: content-transfer-encoding:content-transfer-encoding: in-reply-to:in-reply-to:references:references; bh=DPSgdrK+/9txvlBVeWURx0wHDXuvLt3nB8aMlUhEkV8=; b=Km3/jtgHsTRSi5gGVw/t0cmhyKsIi3WMvVh5hIdyIIa4diuninNPt5mtDnxCM4R1wQjBDm XS/+H3qzqQ3v4pCBvJkLl+N+2Gb3BlDPoCxYGYrVhgsf9Th8GRfH4gc4b6gzcyC46ZhA8X ToTP6VI4hOTbi6X11ytEfZV2gaIbJgA= Received: from mail-qt1-f200.google.com (mail-qt1-f200.google.com [209.85.160.200]) by relay.mimecast.com with ESMTP with STARTTLS (version=TLSv1.3, cipher=TLS_AES_256_GCM_SHA384) id us-mta-383-YYFbZh5vOVeU94vn7whZMg-1; Wed, 03 Jul 2024 17:29:30 -0400 X-MC-Unique: YYFbZh5vOVeU94vn7whZMg-1 Received: by mail-qt1-f200.google.com with SMTP id d75a77b69052e-4466fe59a3dso2086181cf.3 for ; Wed, 03 Jul 2024 14:29:30 -0700 (PDT) X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20230601; t=1720042170; x=1720646970; h=content-transfer-encoding:mime-version:references:in-reply-to :message-id:date:subject:cc:to:from:x-gm-message-state:from:to:cc :subject:date:message-id:reply-to; bh=DPSgdrK+/9txvlBVeWURx0wHDXuvLt3nB8aMlUhEkV8=; b=OrANW+UcRxbOXO3fs0tzkAwExvuvFzAtdGckrS4oDfoYpYOTDyYS8SOVu8wLakUuQu JC1NMfwMvAmv5q5oU8U8mrwdWeBbG7MqSL3Huy8iaQrBfFSGdU7H1YuCgEdabk1y0ETe ALl3c/2Vozc/3KdxzI4PTFsuMdNSXPUXbslLBdVDxawsmB+tm2cL9oflp6MGRcNEDe6T vfdAJSlg+HtqRA8gQfS6jQgri7a1i/CIFHuPIFtnV8zJYkQOuKktpKmHYFdRVmcQGRsV uMC1npsa8QQNjsNqP6TdfD0X5cbsHJwTb0YKSLi9564miyBpIxg77ZuZtp1lJnuJairR 5czg== X-Forwarded-Encrypted: i=1; AJvYcCXp/9ObcG5nBX9O2dN+LRudwnnEg+o3SQ4B9+GoQw4JFXGSCKZO08GR6v+lIq2V6tGlYyAbLhl1+hsoR8vt0GvPqRA= X-Gm-Message-State: AOJu0Yw3qhITeWxxHZnBbRH8OfU9O7LGUaNnWs9mFhO9R9dWuSEqCDZn IkbOVpQ/j8GoS+o2uSSTmobwj4njaRWZC7RBieykuIOXww+124h9NRTFmRRTj0JdJDDKXpLuc34 Nymb7z2dfezs/sjS7SG228augKaQpf6nU4AYas05XClZskzs6 X-Received: by 2002:a05:622a:102:b0:446:5d1b:5a2e with SMTP id d75a77b69052e-44662f544a3mr145792471cf.5.1720042170211; Wed, 03 Jul 2024 14:29:30 -0700 (PDT) X-Google-Smtp-Source: AGHT+IHbwAQwNTOhld77jh3m6+0aceNgmx9fLMFpnLFNTJ0A2GCbsDu4E9EVwwSQY1Fjy1mOFnq/rQ== X-Received: by 2002:a05:622a:102:b0:446:5d1b:5a2e with SMTP id d75a77b69052e-44662f544a3mr145792111cf.5.1720042169860; Wed, 03 Jul 2024 14:29:29 -0700 (PDT) Received: from x1n.redhat.com (pool-99-254-121-117.cpe.net.cable.rogers.com. [99.254.121.117]) by smtp.gmail.com with ESMTPSA id d75a77b69052e-4465143eb1csm54337481cf.57.2024.07.03.14.29.28 (version=TLS1_3 cipher=TLS_AES_256_GCM_SHA384 bits=256/256); Wed, 03 Jul 2024 14:29:29 -0700 (PDT) From: Peter Xu To: linux-kernel@vger.kernel.org, linux-mm@kvack.org Cc: Matthew Wilcox , Mel Gorman , Dave Jiang , linuxppc-dev@lists.ozlabs.org, Michael Ellerman , Rik van Riel , Vlastimil Babka , Nicholas Piggin , Christophe Leroy , Andrew Morton , Huang Ying , Oscar Salvador , "Aneesh Kumar K . V" , Thomas Gleixner , Dave Hansen , x86@kernel.org, Ingo Molnar , "Kirill A . Shutemov" , Dan Williams , Borislav Petkov , peterx@redhat.com, Hugh Dickins , Rick P Edgecombe Subject: [PATCH v2 4/8] mm/powerpc: Add missing pud helpers Date: Wed, 3 Jul 2024 17:29:14 -0400 Message-ID: <20240703212918.2417843-5-peterx@redhat.com> X-Mailer: git-send-email 2.45.0 In-Reply-To: <20240703212918.2417843-1-peterx@redhat.com> References: <20240703212918.2417843-1-peterx@redhat.com> MIME-Version: 1.0 X-Mimecast-Spam-Score: 0 X-Mimecast-Originator: redhat.com X-Rspam-User: X-Rspamd-Server: rspam01 X-Rspamd-Queue-Id: A0F4B4000C X-Stat-Signature: pds4zc5nmcq84k3r8xsj3a917kqq3r4c X-HE-Tag: 1720042172-3112 X-HE-Meta: U2FsdGVkX1+85W7BK9UuBATWMUYX0YiIZ+oGAmP7etjVluRWDH56oKl0elluSL4YrYnhQ8jfxWy8kDrXNKMh1LWay2xScEQq7BwQebbbaiL6HKghst5XaIziy3fSCO623c7W+IohGib7xRuuZipdu4RPP/sEEbZKJI2Z+Wd2iuZR01iKxoQgENz4zAB0AZepVwtYazXRtdotBmT99zMcldlUBSs5cvwHH0E5nku0EPakErWqsYnAM6ztNaJ0FyxwZ+t1Im7rMthP/rM+yj+XHOiJWOfszSKIyZOcYHjtwZ+Bb+ticlWAS7VZQPxfALdUpd8VtXIEGk3vtFEaTRe7XWQte01RvZR5j1rQAMiGn4Ny0H14eyBufjuf9vB00m6S96Q51UJkwOp9FIn1HLSDykpiz6yKQY+crEevY8p1sNmMOWNtCoLFum9ry2Dt8R9PeinLhFqpzCr1YCrxTZ0f5I1t+dhXPriu7xqpLgz4O/PhJFoAessbu+TAuczmNAeg4ja6JlrZL3zwS2c7xZtkxx5mgTv9bV323qKH4kuNrbRysoK0DFSXheLcdhUe9HtzyzgOEjm9O93Fqx8Wh/Qt0v7mbxFQbv5iBvKXwRxisn6UMT0ytzJfQE3p40M/DmmsUeAVa2kPwOMFk3NOQ39CD6DO8tRMevZTEDFVL7w0kZbhYI/QOWF8Sl4SDj+aThq87kVWsbp+yLSAO5+C60CMIoZA7UX2D8hUbhhVsW6HVyxw9upov7geDQZTPSqIKcCe/OgaSH0hKKHUrsHQubutvty9vw0jSinOiwyOomb4/vupw98UUCRAcHX8dtSauktmfStHv0ie76kHoBLfKTD+iWx7lgQBEvZTYUSnDdUODd6PXMUPoA5o/bm33OSXKQkHzZ7KZ7dBDAEVAKjlI/wS4YdbCTo1e4CXlW0hLf8MArXFOP6Mw16XynFHZvWgbUFRyLsQ2g4Bsi8iU7QMT+T GykW1BTS e4mVpT9aey8QYwsYzD2Rc6LdDkDbgJS+fLOcG870cmfSOHk5kKF45Y1LovohJv7l8oEauX1O85SBFqqEDrVzJAs59eCmbh/mfxCgUp/Z3xPq//LamOhFvRxJ5tWP8TtNxcC9azhwETgHL9UWG0cH4upyKCdLZMw3P3+oVbUFHvKNe3JcgabMpG4f/FlBiJcJJM6QGPBwOyMaQBAo5MtRjgY9Ewq53BWa7uj/tAwfHdIEoE6tO6INotDOu7fdC+/dnPPK/BjcbeUWqHiEH0eO6GrvlUn1C4uBkLpAhDjaHtcbn9xfz/pF7FFQueRUlJvEeK9RJpC33EAMEIQf6WJqh3tmutRxFcgdG8M3wtEh4bzsNxSFXHGIqhP4Y5GQWzfcIbVM3otYEip2q/qB5kWiHdqZcrKNOX78N/2yeyJmLTPrKn3GCaVIUOJ/kmmNKfi21IvOjsz7OmDpofo9EWiiJ2dcYwz32Jl19ce/H1voRO/RKhSFqxXxnqG0XAA== X-Bogosity: Ham, tests=bogofilter, spamicity=0.000000, version=1.2.4 Sender: owner-linux-mm@kvack.org Precedence: bulk X-Loop: owner-majordomo@kvack.org List-ID: List-Subscribe: List-Unsubscribe: These new helpers will be needed for pud entry updates soon. Introduce them by referencing the pmd ones. Namely: - pudp_invalidate() - pud_modify() Cc: Michael Ellerman Cc: Nicholas Piggin Cc: Christophe Leroy Cc: linuxppc-dev@lists.ozlabs.org Cc: Aneesh Kumar K.V Signed-off-by: Peter Xu --- arch/powerpc/include/asm/book3s/64/pgtable.h | 3 +++ arch/powerpc/mm/book3s64/pgtable.c | 20 ++++++++++++++++++++ 2 files changed, 23 insertions(+) diff --git a/arch/powerpc/include/asm/book3s/64/pgtable.h b/arch/powerpc/include/asm/book3s/64/pgtable.h index 519b1743a0f4..5da92ba68a45 100644 --- a/arch/powerpc/include/asm/book3s/64/pgtable.h +++ b/arch/powerpc/include/asm/book3s/64/pgtable.h @@ -1124,6 +1124,7 @@ extern pmd_t pfn_pmd(unsigned long pfn, pgprot_t pgprot); extern pud_t pfn_pud(unsigned long pfn, pgprot_t pgprot); extern pmd_t mk_pmd(struct page *page, pgprot_t pgprot); extern pmd_t pmd_modify(pmd_t pmd, pgprot_t newprot); +extern pud_t pud_modify(pud_t pud, pgprot_t newprot); extern void set_pmd_at(struct mm_struct *mm, unsigned long addr, pmd_t *pmdp, pmd_t pmd); extern void set_pud_at(struct mm_struct *mm, unsigned long addr, @@ -1384,6 +1385,8 @@ static inline pgtable_t pgtable_trans_huge_withdraw(struct mm_struct *mm, #define __HAVE_ARCH_PMDP_INVALIDATE extern pmd_t pmdp_invalidate(struct vm_area_struct *vma, unsigned long address, pmd_t *pmdp); +extern pud_t pudp_invalidate(struct vm_area_struct *vma, unsigned long address, + pud_t *pudp); #define pmd_move_must_withdraw pmd_move_must_withdraw struct spinlock; diff --git a/arch/powerpc/mm/book3s64/pgtable.c b/arch/powerpc/mm/book3s64/pgtable.c index f4d8d3c40e5c..5a4a75369043 100644 --- a/arch/powerpc/mm/book3s64/pgtable.c +++ b/arch/powerpc/mm/book3s64/pgtable.c @@ -176,6 +176,17 @@ pmd_t pmdp_invalidate(struct vm_area_struct *vma, unsigned long address, return __pmd(old_pmd); } +pud_t pudp_invalidate(struct vm_area_struct *vma, unsigned long address, + pud_t *pudp) +{ + unsigned long old_pud; + + VM_WARN_ON_ONCE(!pud_present(*pudp)); + old_pud = pud_hugepage_update(vma->vm_mm, address, pudp, _PAGE_PRESENT, _PAGE_INVALID); + flush_pud_tlb_range(vma, address, address + HPAGE_PUD_SIZE); + return __pud(old_pud); +} + pmd_t pmdp_huge_get_and_clear_full(struct vm_area_struct *vma, unsigned long addr, pmd_t *pmdp, int full) { @@ -259,6 +270,15 @@ pmd_t pmd_modify(pmd_t pmd, pgprot_t newprot) pmdv &= _HPAGE_CHG_MASK; return pmd_set_protbits(__pmd(pmdv), newprot); } + +pud_t pud_modify(pud_t pud, pgprot_t newprot) +{ + unsigned long pudv; + + pudv = pud_val(pud); + pudv &= _HPAGE_CHG_MASK; + return pud_set_protbits(__pud(pudv), newprot); +} #endif /* CONFIG_TRANSPARENT_HUGEPAGE */ /* For use by kexec, called with MMU off */ From patchwork Wed Jul 3 21:29:15 2024 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Peter Xu X-Patchwork-Id: 13722801 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from kanga.kvack.org (kanga.kvack.org [205.233.56.17]) by smtp.lore.kernel.org (Postfix) with ESMTP id 8B88EC2BD09 for ; Wed, 3 Jul 2024 21:29:42 +0000 (UTC) Received: by kanga.kvack.org (Postfix) id 32A7C6B0098; Wed, 3 Jul 2024 17:29:39 -0400 (EDT) Received: by kanga.kvack.org (Postfix, from userid 40) id 1EEB16B0096; Wed, 3 Jul 2024 17:29:39 -0400 (EDT) X-Delivered-To: int-list-linux-mm@kvack.org Received: by kanga.kvack.org (Postfix, from userid 63042) id E70CF6B009A; Wed, 3 Jul 2024 17:29:38 -0400 (EDT) X-Delivered-To: linux-mm@kvack.org Received: from relay.hostedemail.com (smtprelay0016.hostedemail.com [216.40.44.16]) by kanga.kvack.org (Postfix) with ESMTP id C59C46B0096 for ; Wed, 3 Jul 2024 17:29:38 -0400 (EDT) Received: from smtpin25.hostedemail.com (a10.router.float.18 [10.200.18.1]) by unirelay02.hostedemail.com (Postfix) with ESMTP id 6C194120ECA for ; Wed, 3 Jul 2024 21:29:38 +0000 (UTC) X-FDA: 82299733236.25.DFFD267 Received: from us-smtp-delivery-124.mimecast.com (us-smtp-delivery-124.mimecast.com [170.10.133.124]) by imf25.hostedemail.com (Postfix) with ESMTP id 5F226A000E for ; Wed, 3 Jul 2024 21:29:36 +0000 (UTC) Authentication-Results: imf25.hostedemail.com; dkim=pass header.d=redhat.com header.s=mimecast20190719 header.b=GpFf9Kkv; dmarc=pass (policy=none) header.from=redhat.com; spf=pass (imf25.hostedemail.com: domain of peterx@redhat.com designates 170.10.133.124 as permitted sender) smtp.mailfrom=peterx@redhat.com ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=hostedemail.com; s=arc-20220608; t=1720042149; h=from:from:sender:reply-to:subject:subject:date:date: message-id:message-id:to:to:cc:cc:mime-version:mime-version: content-type:content-type: content-transfer-encoding:content-transfer-encoding: in-reply-to:in-reply-to:references:references:dkim-signature; bh=TFEuzrgphkpyKgRaMn/StFmQ1IWZKcW4qj9eIjoToHw=; b=z7cqXGG5R2QxDZ9VCB1SK9FvWKbf7IzVWsOEYGtxnFnjkbjx0+FT1sDt5T1nLnMxWUg73v y42f+SFg7TQ17LlhuCuDod6NvzsEssoFosDkfIhkLUcO0GTId4x0NlnCyWWF2kv04lRpsO tHfKISYOVyh5rZPP4HQyBJMq+4MpDh0= ARC-Seal: i=1; s=arc-20220608; d=hostedemail.com; t=1720042149; a=rsa-sha256; cv=none; b=7tnaLv+M5OT4BMuOtKPNWOxEe1GL2YjZ7EvYn5q6rpvj89rvxHYUW8wuSvZWjcscxbFbto 4WgN8DsrAyUNliOsSHgIynVajtSuN+s6M7GIzUBm1wui0lSXHxMXM/SzVYI9B0FTr/oKpb ijCVUXFeflFlF+zY8Vyk+3QmUzm5V/I= ARC-Authentication-Results: i=1; imf25.hostedemail.com; dkim=pass header.d=redhat.com header.s=mimecast20190719 header.b=GpFf9Kkv; dmarc=pass (policy=none) header.from=redhat.com; spf=pass (imf25.hostedemail.com: domain of peterx@redhat.com designates 170.10.133.124 as permitted sender) smtp.mailfrom=peterx@redhat.com DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=redhat.com; s=mimecast20190719; t=1720042175; h=from:from:reply-to:subject:subject:date:date:message-id:message-id: to:to:cc:cc:mime-version:mime-version:content-type:content-type: content-transfer-encoding:content-transfer-encoding: in-reply-to:in-reply-to:references:references; bh=TFEuzrgphkpyKgRaMn/StFmQ1IWZKcW4qj9eIjoToHw=; b=GpFf9KkvOZXx5qmQNGd4rITI5B2ggnpYAwljMLL6bBkHCh0taS3O41TlYHw3l8X4QvAfeY LuBBt4ihMWawcHBWBPmJuxA/Ohk+hpOu/VnAXNqwPfpW7Ftq1S+X2Zl9E43x2UKBTdebh6 mRUu8Tcmm7CfXoLrxbVPeHL9pbuZ7hk= Received: from mail-qt1-f199.google.com (mail-qt1-f199.google.com [209.85.160.199]) by relay.mimecast.com with ESMTP with STARTTLS (version=TLSv1.3, cipher=TLS_AES_256_GCM_SHA384) id us-mta-550-XHg48IzXP8iq1QAOifv9OA-1; Wed, 03 Jul 2024 17:29:32 -0400 X-MC-Unique: XHg48IzXP8iq1QAOifv9OA-1 Received: by mail-qt1-f199.google.com with SMTP id d75a77b69052e-4466fe59a3dso2086321cf.3 for ; Wed, 03 Jul 2024 14:29:32 -0700 (PDT) X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20230601; t=1720042172; x=1720646972; h=content-transfer-encoding:mime-version:references:in-reply-to :message-id:date:subject:cc:to:from:x-gm-message-state:from:to:cc :subject:date:message-id:reply-to; bh=TFEuzrgphkpyKgRaMn/StFmQ1IWZKcW4qj9eIjoToHw=; b=Ww1qnw8MSd7XfI6kMyuUaAipI3XW4nD2PIkFq5ZIelVgRfQyjfLRfyzSJp49HBE5Og kd4qkR/Plu78pHGmGaw47fufSLgAjEwb4iKb+md0P5ACLtc6lq8tKEjz2c2NIqy8uyJO FUiT+eU6n4gpPDXu0nznaZkXraETeQHZyxJ52aWKimYRO+DlhDHDi/BG7HNXxpxRW9Pk CSRutvOu37fmGzpfd1JX7NJPKTn7oC/0FG5kfNExGucgIBvxcA4ZkP98s9+k9ey5ZCdm um/HHtqzr5cHkFC2x+5mwCcMSuHXBXTuQHo7SJBGUe9wW/Ce3EvVgPrcMbBguJQIAbKL 8s2Q== X-Forwarded-Encrypted: i=1; AJvYcCWmrydXwxk4ub8+j1JpbD2eEUNziaAPGZ5z8wcA5AKM0rb/nhT/Ff0nmeHOabEcLt+G9JUsORgk1zu5e9x4rjaLABM= X-Gm-Message-State: AOJu0YyCZLLUuNkUz/WcXsmHFAovrN3Wawnv4Tub+fxYqv2NPgP5FYDG pZ5QeZK637S03TAjauGLNfyT/6bNoV6vgBj3uF/2fVb0Anw5j06sk9SJO2rV0LEegrn12LvMzsV EgNwYtyQi0IAAJmI9fodwefz16UgAT1kXhYmDzQYFJjrSZ2Jg X-Received: by 2002:ac8:7c4f:0:b0:446:4428:2d8f with SMTP id d75a77b69052e-44662fe8d5dmr138289811cf.6.1720042171898; Wed, 03 Jul 2024 14:29:31 -0700 (PDT) X-Google-Smtp-Source: AGHT+IHMHO5UTPVYkVSEJYbLjxCcukt7PlLC0h81KBHiBOgwFBj/wiCFjtQJqk/7gb0k9Iu8HJBYNg== X-Received: by 2002:ac8:7c4f:0:b0:446:4428:2d8f with SMTP id d75a77b69052e-44662fe8d5dmr138289521cf.6.1720042171603; Wed, 03 Jul 2024 14:29:31 -0700 (PDT) Received: from x1n.redhat.com (pool-99-254-121-117.cpe.net.cable.rogers.com. [99.254.121.117]) by smtp.gmail.com with ESMTPSA id d75a77b69052e-4465143eb1csm54337481cf.57.2024.07.03.14.29.30 (version=TLS1_3 cipher=TLS_AES_256_GCM_SHA384 bits=256/256); Wed, 03 Jul 2024 14:29:31 -0700 (PDT) From: Peter Xu To: linux-kernel@vger.kernel.org, linux-mm@kvack.org Cc: Matthew Wilcox , Mel Gorman , Dave Jiang , linuxppc-dev@lists.ozlabs.org, Michael Ellerman , Rik van Riel , Vlastimil Babka , Nicholas Piggin , Christophe Leroy , Andrew Morton , Huang Ying , Oscar Salvador , "Aneesh Kumar K . V" , Thomas Gleixner , Dave Hansen , x86@kernel.org, Ingo Molnar , "Kirill A . Shutemov" , Dan Williams , Borislav Petkov , peterx@redhat.com, Hugh Dickins , Rick P Edgecombe Subject: [PATCH v2 5/8] mm/x86: Make pud_leaf() only cares about PSE bit Date: Wed, 3 Jul 2024 17:29:15 -0400 Message-ID: <20240703212918.2417843-6-peterx@redhat.com> X-Mailer: git-send-email 2.45.0 In-Reply-To: <20240703212918.2417843-1-peterx@redhat.com> References: <20240703212918.2417843-1-peterx@redhat.com> MIME-Version: 1.0 X-Mimecast-Spam-Score: 0 X-Mimecast-Originator: redhat.com X-Rspamd-Server: rspam07 X-Rspamd-Queue-Id: 5F226A000E X-Stat-Signature: yy94gaf59sg8eqddyk8p6sxti83o97fe X-Rspam-User: X-HE-Tag: 1720042176-511879 X-HE-Meta: U2FsdGVkX1+ORBcdsO6FKx9CvT0vFyL8avQS+eSdM+MMoX7qV/w5rsFYro/JBhZTmy2hk1PuQDZ+pubxud76HAjbBTef8sCTnVac7z5uVd5BtYmVkfUKUpB5a2Mh9uQPq3y51DCnhOtkD9k81dCK3IW0prDo0qseB4fXMSbdNHJAdvw9ipIlm5AOSfo73gUcn2Z5qLCbRh+D5e1tO8EGidrPkp1NI1I8qLHLGkrKx+hdk+p+6J+ls+evdKUruKiVBRIJ6eLtCYhVdfC3RLtAb3q3LYN2Zh4lLyd4pl2EelNiK3TBFtt90g1Wn1E0AIdwChQXUQQrPVuUP2np4VsrZ43BNOu3TLCZtHWx0yp8OIWlojeRMuF/5Mzs2OJV5f6TOMGldLTBM3U2n9G2fa+kW9OiOC6Aiwu5urnH+MSn2y6geJNSDyz0/OomEH1VfMuJXAJISCVdLB/Vbv1N7r/oLgmT0uYnMxu6ikL5fkronzrs1E1W7m8Fek914oFh2COmNWjknSolxygK0bam7LzxKmhgtn9R0nx53gPoo5p5RpuSlUuQeuAwMieXnG7RPH3dnTamolXO7AcLN3fBGdeEeCGtTOejv8lGu8sN3hrxzcjWdzRKUsBO/TIoz0IDsHyn30Aqo2mRHoaq7ss+Fi7SPnLFA2JcMN2fDtKFpIJDbf46GM2cRDYJ+SvymiIQQKL5eV9V5XZiSJe+ft6s0IoLX5KTchtmnsaVBl1J/YMSlvjbHHtfshFd/bPCPK2SgRxmfxqZNyN9WWq+6XmH45Y8ZPOYcVfMoMqCDk6keh55hm+4ngq+i3iM2XvJek51QH1V0UCAK/faaXrnFVSGDEJ41bHctXfD39XHVg4FbXzL7R3qtoRgq+pednvON2/d83FugvzjXcGyKJY1PfleQX2b1LT8ogq5Uqv1wTfFIOdRFYVK63ywe9vTsZYIpFGpw5B5wclcPbNevL22sMsaWPG DhO/+nhx YReL4B9+SAYOUTP0qrj3OgI6026EiY6Mnx/U6oXUV/IJ6zepD1ctc4rE4n64FBMYLS4dkgYnPV1Le0wECPaYlmlh5aY1tQiLeJK29/SFgsNjtGqb2aErk0R6cl1QaRFC3URtH9qBIN5J7uYiiZm2KUGBGMpaYo7Ctuu7J6wtaQ6sdrhzIPS48VkMKHQ7OAex+0w43gGmu/TRQfd5LtLZXR+/Db+itAoKf6B6+IhcN+Alm48C9yXhxi9Gp2tLn6TJJdlzKlwIwQYhUlz15Zy99aPjz9fBtWev6JTQ7C6W+/RapnEmfKwttPW5Tza7axoQSAgtiq6CPTjLnLHX8lKSm4Y85WTEt7C/+TnA/w8XwPp1H7jS8ukPnTVY1MII69EwMS+D7GkAgMGbvO0531NYo9xQuLYwxH2Xkpl+jc1PjJhknniVzcVsjNuy9EqKvtzVBDoC6RdPEMhk1lOUQWSRinfxTow== X-Bogosity: Ham, tests=bogofilter, spamicity=0.000000, version=1.2.4 Sender: owner-linux-mm@kvack.org Precedence: bulk X-Loop: owner-majordomo@kvack.org List-ID: List-Subscribe: List-Unsubscribe: An entry should be reported as PUD leaf even if it's PROT_NONE, in which case PRESENT bit isn't there. I hit bad pud without this when testing dax 1G on zapping a PROT_NONE PUD. Cc: Thomas Gleixner Cc: Ingo Molnar Cc: Borislav Petkov Cc: Dave Hansen Cc: x86@kernel.org Acked-by: Dave Hansen Signed-off-by: Peter Xu --- arch/x86/include/asm/pgtable.h | 3 +-- 1 file changed, 1 insertion(+), 2 deletions(-) diff --git a/arch/x86/include/asm/pgtable.h b/arch/x86/include/asm/pgtable.h index 65b8e5bb902c..25fc6d809572 100644 --- a/arch/x86/include/asm/pgtable.h +++ b/arch/x86/include/asm/pgtable.h @@ -1073,8 +1073,7 @@ static inline pmd_t *pud_pgtable(pud_t pud) #define pud_leaf pud_leaf static inline bool pud_leaf(pud_t pud) { - return (pud_val(pud) & (_PAGE_PSE | _PAGE_PRESENT)) == - (_PAGE_PSE | _PAGE_PRESENT); + return pud_val(pud) & _PAGE_PSE; } static inline int pud_bad(pud_t pud) From patchwork Wed Jul 3 21:29:16 2024 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Peter Xu X-Patchwork-Id: 13722802 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from kanga.kvack.org (kanga.kvack.org [205.233.56.17]) by smtp.lore.kernel.org (Postfix) with ESMTP id 29346C2BD09 for ; Wed, 3 Jul 2024 21:29:45 +0000 (UTC) Received: by kanga.kvack.org (Postfix) id 5999E6B0096; Wed, 3 Jul 2024 17:29:39 -0400 (EDT) Received: by kanga.kvack.org (Postfix, from userid 40) id 4D2F96B009B; Wed, 3 Jul 2024 17:29:39 -0400 (EDT) X-Delivered-To: int-list-linux-mm@kvack.org Received: by kanga.kvack.org (Postfix, from userid 63042) id 12E936B0099; Wed, 3 Jul 2024 17:29:39 -0400 (EDT) X-Delivered-To: linux-mm@kvack.org Received: from relay.hostedemail.com (smtprelay0017.hostedemail.com [216.40.44.17]) by kanga.kvack.org (Postfix) with ESMTP id D556C6B0098 for ; Wed, 3 Jul 2024 17:29:38 -0400 (EDT) Received: from smtpin22.hostedemail.com (a10.router.float.18 [10.200.18.1]) by unirelay01.hostedemail.com (Postfix) with ESMTP id 79DF91C1C40 for ; Wed, 3 Jul 2024 21:29:38 +0000 (UTC) X-FDA: 82299733236.22.1DE5838 Received: from us-smtp-delivery-124.mimecast.com (us-smtp-delivery-124.mimecast.com [170.10.133.124]) by imf20.hostedemail.com (Postfix) with ESMTP id 5C7E51C001D for ; Wed, 3 Jul 2024 21:29:36 +0000 (UTC) Authentication-Results: imf20.hostedemail.com; dkim=pass header.d=redhat.com header.s=mimecast20190719 header.b=QjPdYvAF; spf=pass (imf20.hostedemail.com: domain of peterx@redhat.com designates 170.10.133.124 as permitted sender) smtp.mailfrom=peterx@redhat.com; dmarc=pass (policy=none) header.from=redhat.com ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=hostedemail.com; s=arc-20220608; t=1720042164; h=from:from:sender:reply-to:subject:subject:date:date: message-id:message-id:to:to:cc:cc:mime-version:mime-version: content-type:content-type: content-transfer-encoding:content-transfer-encoding: in-reply-to:in-reply-to:references:references:dkim-signature; bh=8uNV81qP1pkF0zqyhqKWAPXX0CBAvqkfZkOGu2qHMfo=; b=PsrXWBQn9bPZ/oprQBYOLFOugIMLmL5JoT8DLZJdukd6zh1ziMcO+cKHEK8JQvcu83NyVg 5vGOY30FJq8pGngbCvn7jSv4f3eMrs3ceuL6VyLnZswnSJZdXOQUiMcafJc1Pdo5Y4Lrwo 35JXAhNDJPXAWBdmTz55V7CDuov0KVQ= ARC-Authentication-Results: i=1; imf20.hostedemail.com; dkim=pass header.d=redhat.com header.s=mimecast20190719 header.b=QjPdYvAF; spf=pass (imf20.hostedemail.com: domain of peterx@redhat.com designates 170.10.133.124 as permitted sender) smtp.mailfrom=peterx@redhat.com; dmarc=pass (policy=none) header.from=redhat.com ARC-Seal: i=1; s=arc-20220608; d=hostedemail.com; t=1720042164; a=rsa-sha256; cv=none; b=72x1jkMMEWmvm9cy2gzhz1BZ2EgYBTYzlgRvO9wQdR3g6PsnkK2V2t6jo1fQPD7Pni4WYD 3NRXaY7kL7Gtbt2k0asPmeQEQH68r2vADuIYb/99LXBqE8W0cQOUd+cITT+F8weJ+IhbK0 AicNHWm098pS/GfYotiw43jC4O28lWk= DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=redhat.com; s=mimecast20190719; t=1720042175; h=from:from:reply-to:subject:subject:date:date:message-id:message-id: to:to:cc:cc:mime-version:mime-version:content-type:content-type: content-transfer-encoding:content-transfer-encoding: in-reply-to:in-reply-to:references:references; bh=8uNV81qP1pkF0zqyhqKWAPXX0CBAvqkfZkOGu2qHMfo=; b=QjPdYvAFiko/2nij1EoOB/nUFnIBnDIcKIrG8cBPvuMQcVnUbaN0kObfi6miv6QXO/r2rW nzTNouNbD2TsWXXpgGoqYFSPFBd1TwGkkRYMJzgsen/PTQJgZewu+ih+kLVGLzPCiWXmJo O4YcF0e841agdU2H8Gt9jUcNw20cvCk= Received: from mail-qt1-f198.google.com (mail-qt1-f198.google.com [209.85.160.198]) by relay.mimecast.com with ESMTP with STARTTLS (version=TLSv1.3, cipher=TLS_AES_256_GCM_SHA384) id us-mta-168-lj_zvRZCMTeW3YSxl-uFuw-1; Wed, 03 Jul 2024 17:29:34 -0400 X-MC-Unique: lj_zvRZCMTeW3YSxl-uFuw-1 Received: by mail-qt1-f198.google.com with SMTP id d75a77b69052e-444fbad91f3so3806611cf.3 for ; Wed, 03 Jul 2024 14:29:34 -0700 (PDT) X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20230601; t=1720042174; x=1720646974; h=content-transfer-encoding:mime-version:references:in-reply-to :message-id:date:subject:cc:to:from:x-gm-message-state:from:to:cc :subject:date:message-id:reply-to; bh=8uNV81qP1pkF0zqyhqKWAPXX0CBAvqkfZkOGu2qHMfo=; b=iSW5CegkQr0IsmjDp1Ok3Dt+b21IVOSBjtlzzNY+g4nNKqklnqsUGXX54ZOr3mSrWm GEXbTOTxxZNVJrAJjmf8jX169gzImjo69FtxhC8JTPcYjCcY7n7fXICSwH/5th8yk1bQ MKY2pn52t2gQ49b143FS04LkkomxrAy/+0N0p6zyi1rmT2jQzi2MlNBz9wjSjPYL8CVQ ixHo6TZJMvHc+AELE0LZpPKs/hG36aV/Lo2PVuraqGukP7wRyUbC5H2LDrIrQvEIatWE cRs5JLkvLH4pKPGqVaP5Qj8306Q9M/k+O9SCJBTIn8ElLB0dSZOfhbH9Rd7KxWuU43AZ ENmA== X-Forwarded-Encrypted: i=1; AJvYcCUA6Rbrv0dZao9SIEXxsBE97aLSjgn3dRcFu7Ci8oRpsLkOcp9dwyGec+4S3NZ70XpE68CP04PWQB++ivRXUmcrB4c= X-Gm-Message-State: AOJu0YzrqHdsmjWXrsb9bv0WEl9jXdXYOOG3RoiOnvgmzQBylBIQ47b1 Z7bDuXE+34tMOkUschNPKhynSLQPSaH0C+8hk5KNuaIMIUetT+DG7v/fAXP/RdhWQmymoX6qGbe sefTTQ6su1Sy+wXJj0sgbAvqWCs1rnzm3eQSSMQdQ4NDXeTrQ X-Received: by 2002:a05:620a:470c:b0:79d:8ea3:9d24 with SMTP id af79cd13be357-79d8ea39f72mr1046536785a.2.1720042173826; Wed, 03 Jul 2024 14:29:33 -0700 (PDT) X-Google-Smtp-Source: AGHT+IFduTwlKnISGNZIByqQ7Sp6Ih/1fAT1ztK1vNR6ntfPREjYbbEVcJ6gOgRtr7E4arLOH2JOwA== X-Received: by 2002:a05:620a:470c:b0:79d:8ea3:9d24 with SMTP id af79cd13be357-79d8ea39f72mr1046534585a.2.1720042173510; Wed, 03 Jul 2024 14:29:33 -0700 (PDT) Received: from x1n.redhat.com (pool-99-254-121-117.cpe.net.cable.rogers.com. [99.254.121.117]) by smtp.gmail.com with ESMTPSA id d75a77b69052e-4465143eb1csm54337481cf.57.2024.07.03.14.29.31 (version=TLS1_3 cipher=TLS_AES_256_GCM_SHA384 bits=256/256); Wed, 03 Jul 2024 14:29:33 -0700 (PDT) From: Peter Xu To: linux-kernel@vger.kernel.org, linux-mm@kvack.org Cc: Matthew Wilcox , Mel Gorman , Dave Jiang , linuxppc-dev@lists.ozlabs.org, Michael Ellerman , Rik van Riel , Vlastimil Babka , Nicholas Piggin , Christophe Leroy , Andrew Morton , Huang Ying , Oscar Salvador , "Aneesh Kumar K . V" , Thomas Gleixner , Dave Hansen , x86@kernel.org, Ingo Molnar , "Kirill A . Shutemov" , Dan Williams , Borislav Petkov , peterx@redhat.com, Hugh Dickins , Rick P Edgecombe Subject: [PATCH v2 6/8] mm/x86: arch_check_zapped_pud() Date: Wed, 3 Jul 2024 17:29:16 -0400 Message-ID: <20240703212918.2417843-7-peterx@redhat.com> X-Mailer: git-send-email 2.45.0 In-Reply-To: <20240703212918.2417843-1-peterx@redhat.com> References: <20240703212918.2417843-1-peterx@redhat.com> MIME-Version: 1.0 X-Mimecast-Spam-Score: 0 X-Mimecast-Originator: redhat.com X-Rspam-User: X-Rspamd-Server: rspam01 X-Rspamd-Queue-Id: 5C7E51C001D X-Stat-Signature: yhuqzgbke3nwz71mxdrjyrdbctjt1hhn X-HE-Tag: 1720042176-691439 X-HE-Meta: U2FsdGVkX197Nr3ut1S1RgZWP0n2OpQ1w4dUj7WkqXvG8sZc5boMiL/Vb0zqDW2186OWOC0xm0fogA/kI4ZjtwznebW1T/nULy+rE+aUXtrjJbpdZlckm0V+wbnY7yl3ZX5BAiX1r3d6mXshNcifW1gpqKeXCNIol3NEmGZOnjXRkSXa74Is0+3T3gAlQCv2TMxNe+QRO1/51x3NtFkShe5igKFAzOnFP8kf79JXA4d3m20owW8YApa8cP0RIffiuNNUI9aJwvfP9oDm9zvqK8WJAk2A2V7euCR5mSKgTKjMtF41KvCCljLXMY4C1f259boBktj2718S7imzQpasmJ5Ww+JJMlMinowA0siHLlQdNwasrIfzBIKicqF4NiAm0/I8TuSPxY7MS4yWheYaeLgGw7tOAfi8YBCvjjYU0RnCGNPJayAeQ3Tu7M14QU3lzDZ+LRKgiK55Uz3H3JNPYeOhVAUb++KziRHVmYnI+fsnbZmREdYAp1/ThTjkJfJh7gC+uMhmX84gUjEZrLxSTD9rJ5id8u/i+MjMQUGM+avSrg8MSNN6bHNTIyVEwwQn9mW0hgLQFjdu81O31KfUxUTbIkwl0DwehH4yCdf+gsVL0qA6JVEsXGPZEJ48VBKYchTKTagaNNk8BULkpbgGE5x2FLDzuXEEK1tsq0GAcRCOBuHhhQighgM0UKlhLXGqVfSNktRqDfi/6osjsHcYVi7lNllLdoFn70JKOSYQ1YZRNAz1Wo4Iz+R3ax1JXZ7ijLZ3hYa2eC0gd0ITkjeu5whfEEzL6IHCveTYoyD3225UnB3qOU/oLBclPPGWRhLn85O56o49Fb3psLjokInhDlfJaoDiq93LZ++mEGaOGo5efwf8EGpux6HtfJhc7bXCl5mYFYlSh6VqGBTrZLxBgXDdh/B7BkQVT34v+AiMseis5R/9B09oTI5Yx1rR0iHPKAb/83flXOzr7PGdcyi J2o8LEao tcnW3sdgROJCz2nPgPrJGTf52sdoCypeaY5tTzRDZl6a+e1prF/u3HdDAIT4sbTCegWdSMQmz9cq8FjimyajXyyK9/j18QsKcXmsedmTfgWa1vKRnJsgpf3g1ag9zq/aJEi/oYWL91PsBsAjDIWOsNkIlQpP46Tdq6bS66LvrNSzfIg3wKk6tbv+WIUgUtqh33dbsQFYSl8DcY+CAWe0vcv1JIsfAUXQOQqEJJvDQBQsRp8onwQguuN7sb+l4vmrTLBQGzgW+/24OQ1pLWlvoLroMe3gU28d9uH0jE8S6ohCOAgzagxSTnsG1zPTpDQqxqjyx562zj9SknKtXkJ8XqQC3Z3LHdc4ij7jZezmQsA48rUcHXmFPG+0ILtIPF6Etw+fEy8qO43amPaut+QULCTPjYRqfS/ZPiBNlVYekXsnVy13EAFCBr1V69ypdJH1doT022/c+3ae7cjTrYl5zW9GR3Q== X-Bogosity: Ham, tests=bogofilter, spamicity=0.000000, version=1.2.4 Sender: owner-linux-mm@kvack.org Precedence: bulk X-Loop: owner-majordomo@kvack.org List-ID: List-Subscribe: List-Unsubscribe: Introduce arch_check_zapped_pud() to sanity check shadow stack on PUD zaps. It has the same logic of the PMD helper. One thing to mention is, it might be a good idea to use page_table_check in the future for trapping wrong setups of shadow stack pgtable entries [1]. That is left for the future as a separate effort. [1] https://lore.kernel.org/all/59d518698f664e07c036a5098833d7b56b953305.camel@intel.com Cc: "Edgecombe, Rick P" Cc: Thomas Gleixner Cc: Ingo Molnar Cc: Borislav Petkov Cc: Dave Hansen Cc: x86@kernel.org Signed-off-by: Peter Xu --- arch/x86/include/asm/pgtable.h | 10 ++++++++++ arch/x86/mm/pgtable.c | 7 +++++++ include/linux/pgtable.h | 7 +++++++ mm/huge_memory.c | 4 +++- 4 files changed, 27 insertions(+), 1 deletion(-) diff --git a/arch/x86/include/asm/pgtable.h b/arch/x86/include/asm/pgtable.h index 25fc6d809572..cdf044c2ad6e 100644 --- a/arch/x86/include/asm/pgtable.h +++ b/arch/x86/include/asm/pgtable.h @@ -169,6 +169,13 @@ static inline int pud_young(pud_t pud) return pud_flags(pud) & _PAGE_ACCESSED; } +static inline bool pud_shstk(pud_t pud) +{ + return cpu_feature_enabled(X86_FEATURE_SHSTK) && + (pud_flags(pud) & (_PAGE_RW | _PAGE_DIRTY | _PAGE_PSE)) == + (_PAGE_DIRTY | _PAGE_PSE); +} + static inline int pte_write(pte_t pte) { /* @@ -1662,6 +1669,9 @@ void arch_check_zapped_pte(struct vm_area_struct *vma, pte_t pte); #define arch_check_zapped_pmd arch_check_zapped_pmd void arch_check_zapped_pmd(struct vm_area_struct *vma, pmd_t pmd); +#define arch_check_zapped_pud arch_check_zapped_pud +void arch_check_zapped_pud(struct vm_area_struct *vma, pud_t pud); + #ifdef CONFIG_XEN_PV #define arch_has_hw_nonleaf_pmd_young arch_has_hw_nonleaf_pmd_young static inline bool arch_has_hw_nonleaf_pmd_young(void) diff --git a/arch/x86/mm/pgtable.c b/arch/x86/mm/pgtable.c index 93e54ba91fbf..564b8945951e 100644 --- a/arch/x86/mm/pgtable.c +++ b/arch/x86/mm/pgtable.c @@ -926,3 +926,10 @@ void arch_check_zapped_pmd(struct vm_area_struct *vma, pmd_t pmd) VM_WARN_ON_ONCE(!(vma->vm_flags & VM_SHADOW_STACK) && pmd_shstk(pmd)); } + +void arch_check_zapped_pud(struct vm_area_struct *vma, pud_t pud) +{ + /* See note in arch_check_zapped_pte() */ + VM_WARN_ON_ONCE(!(vma->vm_flags & VM_SHADOW_STACK) && + pud_shstk(pud)); +} diff --git a/include/linux/pgtable.h b/include/linux/pgtable.h index 2a6a3cccfc36..2289e9f7aa1b 100644 --- a/include/linux/pgtable.h +++ b/include/linux/pgtable.h @@ -447,6 +447,13 @@ static inline void arch_check_zapped_pmd(struct vm_area_struct *vma, } #endif +#ifndef arch_check_zapped_pud +static inline void arch_check_zapped_pud(struct vm_area_struct *vma, + pud_t pud) +{ +} +#endif + #ifndef __HAVE_ARCH_PTEP_GET_AND_CLEAR static inline pte_t ptep_get_and_clear(struct mm_struct *mm, unsigned long address, diff --git a/mm/huge_memory.c b/mm/huge_memory.c index 251d6932130f..017377920d18 100644 --- a/mm/huge_memory.c +++ b/mm/huge_memory.c @@ -2278,12 +2278,14 @@ int zap_huge_pud(struct mmu_gather *tlb, struct vm_area_struct *vma, pud_t *pud, unsigned long addr) { spinlock_t *ptl; + pud_t orig_pud; ptl = __pud_trans_huge_lock(pud, vma); if (!ptl) return 0; - pudp_huge_get_and_clear_full(vma, addr, pud, tlb->fullmm); + orig_pud = pudp_huge_get_and_clear_full(vma, addr, pud, tlb->fullmm); + arch_check_zapped_pud(vma, orig_pud); tlb_remove_pud_tlb_entry(tlb, pud, addr); if (vma_is_special_huge(vma)) { spin_unlock(ptl); From patchwork Wed Jul 3 21:29:17 2024 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Peter Xu X-Patchwork-Id: 13722803 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from kanga.kvack.org (kanga.kvack.org [205.233.56.17]) by smtp.lore.kernel.org (Postfix) with ESMTP id E99F8C31D97 for ; Wed, 3 Jul 2024 21:29:47 +0000 (UTC) Received: by kanga.kvack.org (Postfix) id 1F2396B009A; Wed, 3 Jul 2024 17:29:43 -0400 (EDT) Received: by kanga.kvack.org (Postfix, from userid 40) id 1A0A56B009B; Wed, 3 Jul 2024 17:29:43 -0400 (EDT) X-Delivered-To: int-list-linux-mm@kvack.org Received: by kanga.kvack.org (Postfix, from userid 63042) id EBF526B009C; Wed, 3 Jul 2024 17:29:42 -0400 (EDT) X-Delivered-To: linux-mm@kvack.org Received: from relay.hostedemail.com (smtprelay0017.hostedemail.com [216.40.44.17]) by kanga.kvack.org (Postfix) with ESMTP id CC1346B009A for ; Wed, 3 Jul 2024 17:29:42 -0400 (EDT) Received: from smtpin12.hostedemail.com (a10.router.float.18 [10.200.18.1]) by unirelay02.hostedemail.com (Postfix) with ESMTP id 90CCA120852 for ; Wed, 3 Jul 2024 21:29:42 +0000 (UTC) X-FDA: 82299733404.12.40806B8 Received: from us-smtp-delivery-124.mimecast.com (us-smtp-delivery-124.mimecast.com [170.10.129.124]) by imf07.hostedemail.com (Postfix) with ESMTP id 82E4B40015 for ; Wed, 3 Jul 2024 21:29:40 +0000 (UTC) Authentication-Results: imf07.hostedemail.com; dkim=pass header.d=redhat.com header.s=mimecast20190719 header.b=jLtMgAo0; spf=pass (imf07.hostedemail.com: domain of peterx@redhat.com designates 170.10.129.124 as permitted sender) smtp.mailfrom=peterx@redhat.com; dmarc=pass (policy=none) header.from=redhat.com ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=hostedemail.com; s=arc-20220608; t=1720042169; h=from:from:sender:reply-to:subject:subject:date:date: message-id:message-id:to:to:cc:cc:mime-version:mime-version: content-type:content-type: content-transfer-encoding:content-transfer-encoding: in-reply-to:in-reply-to:references:references:dkim-signature; bh=VBkUbHh5FhI05CAEJEjFs15FFfGPI8FVTV6ICfAryLc=; b=ZboMKlWYyrC5muiDhql0DiinwXDEGeyfCNBWyNrZf98wEtep41vKeudKWEFTFxIqNU9lBq vPkf8c3pNqtSEVFzc+Fw9sq1RMpqrGfJOcxQLyCamK+5mbVSpzkdPhgX5KNjKn35EH1x0m TUCQYp0z2sDfoiVdFdq/v3yvxWBxQf0= ARC-Authentication-Results: i=1; imf07.hostedemail.com; dkim=pass header.d=redhat.com header.s=mimecast20190719 header.b=jLtMgAo0; spf=pass (imf07.hostedemail.com: domain of peterx@redhat.com designates 170.10.129.124 as permitted sender) smtp.mailfrom=peterx@redhat.com; dmarc=pass (policy=none) header.from=redhat.com ARC-Seal: i=1; s=arc-20220608; d=hostedemail.com; t=1720042169; a=rsa-sha256; cv=none; b=SFQLzztygCh//FkUQyzDILO07quz3PpzuCFvmRcvgJARNzEXDm0UUzfm4BE1/u6Aej3Gj1 BrZY/arAGcnRO0OHS+OV8U2R2QwPHbcnUmDDBE4v9VWQ/kVFnTj4W/OVDyx8+W2Z/af/O6 zUyRmi2cPFQJKHQ9kChLjMLO4U3XGYI= DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=redhat.com; s=mimecast20190719; t=1720042179; h=from:from:reply-to:subject:subject:date:date:message-id:message-id: to:to:cc:cc:mime-version:mime-version:content-type:content-type: content-transfer-encoding:content-transfer-encoding: in-reply-to:in-reply-to:references:references; bh=VBkUbHh5FhI05CAEJEjFs15FFfGPI8FVTV6ICfAryLc=; b=jLtMgAo0xKJ3azfLtIdgREGcNQHoH9aoyUJMsUnjb2tgjcac5T5sucMGVACx5TeKwOfDrQ Xnc1kubsEHo2Dvru01CbuagFLR2Vv7iLw8LK3KP7VCEQSBN2ByhElQTE2sKDa+A34ywkzr axEsdP4uCJwkjWSYzZjNgLK3HyjhCiU= Received: from mail-oo1-f72.google.com (mail-oo1-f72.google.com [209.85.161.72]) by relay.mimecast.com with ESMTP with STARTTLS (version=TLSv1.3, cipher=TLS_AES_256_GCM_SHA384) id us-mta-437-LOrlaqfoN6e-hb39ELTxWg-1; Wed, 03 Jul 2024 17:29:38 -0400 X-MC-Unique: LOrlaqfoN6e-hb39ELTxWg-1 Received: by mail-oo1-f72.google.com with SMTP id 006d021491bc7-5c479ad9c8cso171261eaf.2 for ; Wed, 03 Jul 2024 14:29:38 -0700 (PDT) X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20230601; t=1720042178; x=1720646978; h=content-transfer-encoding:mime-version:references:in-reply-to :message-id:date:subject:cc:to:from:x-gm-message-state:from:to:cc :subject:date:message-id:reply-to; bh=VBkUbHh5FhI05CAEJEjFs15FFfGPI8FVTV6ICfAryLc=; b=VDitrU8Za8sJ07tHqDkHfkPGazLtu4EcLCfD0qRQzjNCbgpm1f4SyG57qL4yUfw14G /EbKE9XAhXgjHyo/rLrgmnSSLUrECE7etBtOHPYNmC05V1sDmXIdfdicHRF7r1ULPRxn 8OFauPHEOApPw2xsWAXIhUFpHfFjCaJjyOBCY5qA3yP/FQMZkhsJlAxPtdDKyVh3qkod hoKZw5PVXyoMmZSfP31hHlN0tETKquXmKCrQZrcObM6EeFKBdaTpQLgSdZISdcOXxiYJ my2xnM7FNO0h2ToF1RdNoUne2ZOlsZ4BYLVlKVXIwe7nr8/VfnbQv2vW7QsXJFm7HeVu jJRw== X-Forwarded-Encrypted: i=1; AJvYcCWOIduALJsy3lTFZ2A+3ayK2ev0IPv9Lp9xHQCvivb4Jc9CSZX0zo3WoFB46Y02/sljdWUEYeH5ONb6ILFWWiWaNsk= X-Gm-Message-State: AOJu0YwE0rdlBnop22oIwJXcIgYT+im+lmjtlXKz7smowp8eFsMuKUEh gUJ18n4gMbMe4lF3R036aDpQrtFjFuhmpx0cuU8bqd86BqRDMQ207U35i0ftbIfml1/mGkafGdG g1P2VpcSCF1Jm2tPBgXI73Fs99vdFpasdolJuSgfRgfel388s X-Received: by 2002:a05:6358:7f0d:b0:1a2:3bde:1425 with SMTP id e5c5f4694b2df-1a6acf18b33mr1469440055d.2.1720042177858; Wed, 03 Jul 2024 14:29:37 -0700 (PDT) X-Google-Smtp-Source: AGHT+IFh/EdB/xSzquRBRoGgcbcKdNPV887AakpSVBbTc4I0BiDNq1hD4sw71GTYj4zNAp5JLHYzHw== X-Received: by 2002:a05:6358:7f0d:b0:1a2:3bde:1425 with SMTP id e5c5f4694b2df-1a6acf18b33mr1469428955d.2.1720042175430; Wed, 03 Jul 2024 14:29:35 -0700 (PDT) Received: from x1n.redhat.com (pool-99-254-121-117.cpe.net.cable.rogers.com. [99.254.121.117]) by smtp.gmail.com with ESMTPSA id d75a77b69052e-4465143eb1csm54337481cf.57.2024.07.03.14.29.33 (version=TLS1_3 cipher=TLS_AES_256_GCM_SHA384 bits=256/256); Wed, 03 Jul 2024 14:29:34 -0700 (PDT) From: Peter Xu To: linux-kernel@vger.kernel.org, linux-mm@kvack.org Cc: Matthew Wilcox , Mel Gorman , Dave Jiang , linuxppc-dev@lists.ozlabs.org, Michael Ellerman , Rik van Riel , Vlastimil Babka , Nicholas Piggin , Christophe Leroy , Andrew Morton , Huang Ying , Oscar Salvador , "Aneesh Kumar K . V" , Thomas Gleixner , Dave Hansen , x86@kernel.org, Ingo Molnar , "Kirill A . Shutemov" , Dan Williams , Borislav Petkov , peterx@redhat.com, Hugh Dickins , Rick P Edgecombe Subject: [PATCH v2 7/8] mm/x86: Add missing pud helpers Date: Wed, 3 Jul 2024 17:29:17 -0400 Message-ID: <20240703212918.2417843-8-peterx@redhat.com> X-Mailer: git-send-email 2.45.0 In-Reply-To: <20240703212918.2417843-1-peterx@redhat.com> References: <20240703212918.2417843-1-peterx@redhat.com> MIME-Version: 1.0 X-Mimecast-Spam-Score: 0 X-Mimecast-Originator: redhat.com X-Rspam-User: X-Rspamd-Server: rspam01 X-Rspamd-Queue-Id: 82E4B40015 X-Stat-Signature: bjap5g3ocsamw4picbfqpanjc31i6xy7 X-HE-Tag: 1720042180-551706 X-HE-Meta: U2FsdGVkX1/LQyrZpSqPLzdFitDnKpk2DLElcTW8gM+0Qyxr6s8eZ0RlymlSLvu9G9H+xd9K+LTp8u0dqN1/IABfMcj783uTlLlR41+6iMR/7+NJKEfm3R/nWW/hlZY4J7rZBKVURgjO7e0WAfQzhg2JM31qJsup4MTE5/5BbzDLwaXPXCU5t9HN6oZ3V/2u3rU83AFmYRhwqTwM7dWT+P+ALONa24gyKM0E2pbllwTCbOqfTWpdejDHTGX8Fz+L0VLsZB+bsTsY5icikYJ15c1N7EyrXnCBsloEVhs2sSkYVaI/WdFo+LaEnKxmV8pX6UkjIp65Yv/JyhS3FVP+45j1y9Ho03f71tBNoWfCy0CTQRjFt7rHmF/aJI1GJJn1s6eQKvhMS9pk9L3km7ocfR3n0AcHq4wFkQqx6MgefqGWRv+/dNNg52p4xAVcuc9cxZ6XJmw4XXv4Hv2ZW6R7IDuzWxZYuH1VbC0oaUg4Z9PjqeoN/hrr8lFQNjoK0vOFbg1LUIcdS6eTpWvWRyHK06jLR2Eh49QFuqQawnU3yjtRtaAtOrh7+P56y0PPJzQtemqxkz+CWOAIF5ZS3QViLUTmnIL6dfaZh3+7hTZVz+fUenJZS4729Ld9lKsUC5CAalTFD9lAkomQGN9fVaXCqiXe7Q8eewuLGkJJgMqzr6dbol6SJxTw3WwtGJTwxLYXszub5RTe09kRl5E2qKl35NnKz4Z3ltOaQuLmR73iK4sLqFKO4OoajbIl5rNNdeETPp4YlUVxkInTtemrhd34u25drpVDb9PncUIj/3lzTAZe2VweozO6oeENDDdMLw/fMSxAyCpQG2X6v5dRFEvrCSiAShTCIXHy+iT8SPz06YMQZCT3N5J8P9a//6sWQtCQMJY7XPwn0ILA431maV1pf2n5GW26Wis2vqqH8vaKVsMrUL5n27MAsk9XaroGHHFfpMOxsMVU09VJjxG71sz 2n20HbJi CNMfRpdnqRAmU/MGlWgk5QAjW6OqNw/0yhShKnXXj7owvBBIl+FoH2pM+OurTCTTa0RKiNZkHfhGhYhFa0PRDrBzY3s/t0GshlpJ/7h72/H6PSP3qiwWsRN8Fyb+HTZq9HLaxksNDcv8GGyOGt878RuOs1Nti/jDWT0edeK/uYQ2TPCuw04vMf+QnR5NQIUluduvOixDN7yCjYavoB7dqh0hxQfxXA10zlnncbc7MBqgArNrCki96rM8jSpLHBkZAksDoZyLlGyIir7Pt82rVu647UhaaTwwdb8jPqk7TME0aMeNV5By0ao6yWfGVuCoizQXr8GsaBVAMZp9Hofq3MPzifRppLvHBMeddQ8ani8FSWXjtpj3vivS7Uw== X-Bogosity: Ham, tests=bogofilter, spamicity=0.000000, version=1.2.4 Sender: owner-linux-mm@kvack.org Precedence: bulk X-Loop: owner-majordomo@kvack.org List-ID: List-Subscribe: List-Unsubscribe: These new helpers will be needed for pud entry updates soon. Introduce these helpers by referencing the pmd ones. Namely: - pudp_invalidate() - pud_modify() Cc: Thomas Gleixner Cc: Ingo Molnar Cc: Borislav Petkov Cc: Dave Hansen Cc: x86@kernel.org Signed-off-by: Peter Xu --- arch/x86/include/asm/pgtable.h | 55 +++++++++++++++++++++++++++++----- arch/x86/mm/pgtable.c | 11 +++++++ 2 files changed, 58 insertions(+), 8 deletions(-) diff --git a/arch/x86/include/asm/pgtable.h b/arch/x86/include/asm/pgtable.h index cdf044c2ad6e..701593c53f3b 100644 --- a/arch/x86/include/asm/pgtable.h +++ b/arch/x86/include/asm/pgtable.h @@ -782,6 +782,12 @@ static inline pmd_t pmd_mkinvalid(pmd_t pmd) __pgprot(pmd_flags(pmd) & ~(_PAGE_PRESENT|_PAGE_PROTNONE))); } +static inline pud_t pud_mkinvalid(pud_t pud) +{ + return pfn_pud(pud_pfn(pud), + __pgprot(pud_flags(pud) & ~(_PAGE_PRESENT|_PAGE_PROTNONE))); +} + static inline u64 flip_protnone_guard(u64 oldval, u64 val, u64 mask); static inline pte_t pte_modify(pte_t pte, pgprot_t newprot) @@ -829,14 +835,8 @@ static inline pmd_t pmd_modify(pmd_t pmd, pgprot_t newprot) pmd_result = __pmd(val); /* - * To avoid creating Write=0,Dirty=1 PMDs, pte_modify() needs to avoid: - * 1. Marking Write=0 PMDs Dirty=1 - * 2. Marking Dirty=1 PMDs Write=0 - * - * The first case cannot happen because the _PAGE_CHG_MASK will filter - * out any Dirty bit passed in newprot. Handle the second case by - * going through the mksaveddirty exercise. Only do this if the old - * value was Write=1 to avoid doing this on Shadow Stack PTEs. + * Avoid creating shadow stack PMD by accident. See comment in + * pte_modify(). */ if (oldval & _PAGE_RW) pmd_result = pmd_mksaveddirty(pmd_result); @@ -846,6 +846,29 @@ static inline pmd_t pmd_modify(pmd_t pmd, pgprot_t newprot) return pmd_result; } +static inline pud_t pud_modify(pud_t pud, pgprot_t newprot) +{ + pudval_t val = pud_val(pud), oldval = val; + pud_t pud_result; + + val &= _HPAGE_CHG_MASK; + val |= check_pgprot(newprot) & ~_HPAGE_CHG_MASK; + val = flip_protnone_guard(oldval, val, PHYSICAL_PUD_PAGE_MASK); + + pud_result = __pud(val); + + /* + * Avoid creating shadow stack PUD by accident. See comment in + * pte_modify(). + */ + if (oldval & _PAGE_RW) + pud_result = pud_mksaveddirty(pud_result); + else + pud_result = pud_clear_saveddirty(pud_result); + + return pud_result; +} + /* * mprotect needs to preserve PAT and encryption bits when updating * vm_page_prot @@ -1384,10 +1407,26 @@ static inline pmd_t pmdp_establish(struct vm_area_struct *vma, } #endif +static inline pud_t pudp_establish(struct vm_area_struct *vma, + unsigned long address, pud_t *pudp, pud_t pud) +{ + page_table_check_pud_set(vma->vm_mm, pudp, pud); + if (IS_ENABLED(CONFIG_SMP)) { + return xchg(pudp, pud); + } else { + pud_t old = *pudp; + WRITE_ONCE(*pudp, pud); + return old; + } +} + #define __HAVE_ARCH_PMDP_INVALIDATE_AD extern pmd_t pmdp_invalidate_ad(struct vm_area_struct *vma, unsigned long address, pmd_t *pmdp); +pud_t pudp_invalidate(struct vm_area_struct *vma, unsigned long address, + pud_t *pudp); + /* * Page table pages are page-aligned. The lower half of the top * level is used for userspace and the top half for the kernel. diff --git a/arch/x86/mm/pgtable.c b/arch/x86/mm/pgtable.c index 564b8945951e..9d97a1b99160 100644 --- a/arch/x86/mm/pgtable.c +++ b/arch/x86/mm/pgtable.c @@ -641,6 +641,17 @@ pmd_t pmdp_invalidate_ad(struct vm_area_struct *vma, unsigned long address, } #endif +#ifdef CONFIG_TRANSPARENT_HUGEPAGE +pud_t pudp_invalidate(struct vm_area_struct *vma, unsigned long address, + pud_t *pudp) +{ + VM_WARN_ON_ONCE(!pud_present(*pudp)); + pud_t old = pudp_establish(vma, address, pudp, pud_mkinvalid(*pudp)); + flush_pud_tlb_range(vma, address, address + HPAGE_PUD_SIZE); + return old; +} +#endif + /** * reserve_top_address - reserves a hole in the top of kernel address space * @reserve - size of hole to reserve From patchwork Wed Jul 3 21:29:18 2024 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Peter Xu X-Patchwork-Id: 13722804 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from kanga.kvack.org (kanga.kvack.org [205.233.56.17]) by smtp.lore.kernel.org (Postfix) with ESMTP id C65A6C2BD09 for ; Wed, 3 Jul 2024 21:29:50 +0000 (UTC) Received: by kanga.kvack.org (Postfix) id 439606B009C; Wed, 3 Jul 2024 17:29:45 -0400 (EDT) Received: by kanga.kvack.org (Postfix, from userid 40) id 3ECFB6B009D; Wed, 3 Jul 2024 17:29:45 -0400 (EDT) X-Delivered-To: int-list-linux-mm@kvack.org Received: by kanga.kvack.org (Postfix, from userid 63042) id 213226B009E; Wed, 3 Jul 2024 17:29:45 -0400 (EDT) X-Delivered-To: linux-mm@kvack.org Received: from relay.hostedemail.com (smtprelay0014.hostedemail.com [216.40.44.14]) by kanga.kvack.org (Postfix) with ESMTP id 05E876B009C for ; Wed, 3 Jul 2024 17:29:45 -0400 (EDT) Received: from smtpin26.hostedemail.com (a10.router.float.18 [10.200.18.1]) by unirelay01.hostedemail.com (Postfix) with ESMTP id 7BFBD1C1A20 for ; Wed, 3 Jul 2024 21:29:44 +0000 (UTC) X-FDA: 82299733488.26.A6848B2 Received: from us-smtp-delivery-124.mimecast.com (us-smtp-delivery-124.mimecast.com [170.10.129.124]) by imf27.hostedemail.com (Postfix) with ESMTP id 3F14B4000F for ; Wed, 3 Jul 2024 21:29:42 +0000 (UTC) Authentication-Results: imf27.hostedemail.com; dkim=pass header.d=redhat.com header.s=mimecast20190719 header.b=LoRntBCu; spf=pass (imf27.hostedemail.com: domain of peterx@redhat.com designates 170.10.129.124 as permitted sender) smtp.mailfrom=peterx@redhat.com; dmarc=pass (policy=none) header.from=redhat.com ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=hostedemail.com; s=arc-20220608; t=1720042169; h=from:from:sender:reply-to:subject:subject:date:date: message-id:message-id:to:to:cc:cc:mime-version:mime-version: content-type:content-type: content-transfer-encoding:content-transfer-encoding: in-reply-to:in-reply-to:references:references:dkim-signature; bh=j2Zxgw15onbd0eMS/03kg+dHZsOcAz6IsQR1l1z1OuE=; b=rMJz4eh/spwXUKMZzrg4KKKiV2jMtQ98GSV0GqykveoYdNXy1gkgtX/8e/duA10PMNsvUC lBm/yprQO3x4+hfATcJW8mYQ69JCILg3fRadDEQFtseJYR9hQlGhXvPOp92Fb8MqtVaYZz hv4BGJ6UsQ6fhvMjVURBvAuWIRcU6zc= ARC-Authentication-Results: i=1; imf27.hostedemail.com; dkim=pass header.d=redhat.com header.s=mimecast20190719 header.b=LoRntBCu; spf=pass (imf27.hostedemail.com: domain of peterx@redhat.com designates 170.10.129.124 as permitted sender) smtp.mailfrom=peterx@redhat.com; dmarc=pass (policy=none) header.from=redhat.com ARC-Seal: i=1; s=arc-20220608; d=hostedemail.com; t=1720042169; a=rsa-sha256; cv=none; b=UTAu/aytr2OmBtzyAi2aqyuwOWaKh1izjCxhMvtoWtA8HrFYVU30w6BoQbsDrNy+jHc2gf xGVIcSASkjF/wumK7L/QYFWnZkV0Y7Z2mdwkkSNih+jx9qthbvpBcwRs3axvP2Nt2IYPID VDhsWEHp0b9qdD1fMNqK21vqXRhvh/w= DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=redhat.com; s=mimecast20190719; t=1720042181; h=from:from:reply-to:subject:subject:date:date:message-id:message-id: to:to:cc:cc:mime-version:mime-version:content-type:content-type: content-transfer-encoding:content-transfer-encoding: in-reply-to:in-reply-to:references:references; bh=j2Zxgw15onbd0eMS/03kg+dHZsOcAz6IsQR1l1z1OuE=; b=LoRntBCuK5FTB0c0NjbQ+TQkcVOdJNntHrpJ3DNVwSOiLMgMUPYR5N7XjW1iWuUBrWz+Su QAgKCsYFRGZ6gHaDCaPaC6OeR4cznyUr/fjCr7RdGmRvgbPHfKgIDgD+isO4ffDeJ6aTtn 0Tf8X7/JK07wPYt6Tzpf47viTWkPqCs= Received: from mail-qt1-f200.google.com (mail-qt1-f200.google.com [209.85.160.200]) by relay.mimecast.com with ESMTP with STARTTLS (version=TLSv1.3, cipher=TLS_AES_256_GCM_SHA384) id us-mta-253-AQm0WAoXM864bmyG2xCcbg-1; Wed, 03 Jul 2024 17:29:40 -0400 X-MC-Unique: AQm0WAoXM864bmyG2xCcbg-1 Received: by mail-qt1-f200.google.com with SMTP id d75a77b69052e-4464454e329so1958251cf.1 for ; Wed, 03 Jul 2024 14:29:40 -0700 (PDT) X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20230601; t=1720042180; x=1720646980; h=content-transfer-encoding:mime-version:references:in-reply-to :message-id:date:subject:cc:to:from:x-gm-message-state:from:to:cc :subject:date:message-id:reply-to; bh=j2Zxgw15onbd0eMS/03kg+dHZsOcAz6IsQR1l1z1OuE=; b=Vk2AwMRxiM2sMq0NcJvFuXxYPS/IGojwsGTQug37t2vF0+VN1xNEj+V2wfTWB44e5K rMA2QWZkBl71q+RWcbfUP2tBo6/ZLko3hE6t3s1avVbuZs9PDpZndoBs/hKcgwM+gDUQ WW1SP42p2BY/jY7/Iy77STdT2w1iYHcyWvhwzdQ0fdWIKlyrxuTXfVayy6vm8kG+0KGN NkZKfyVHmb7ElQ+p9nrwCX+ZCWh2QTiYVZrIokyblsuDEWxatI1PTPbRs7w8SjDprKhd FNYanR/zHnI4BEKMQ4LDp/117mbBECTAtG64i5iMkC2TDHbArGkRw2FVxURLB9VakOe7 Ottg== X-Forwarded-Encrypted: i=1; AJvYcCVK/f6UfnBQnBPNQdnhIucFJpOSbB0VJdDn4ieY6gxku7+f+ui/kKER9Nrd7kxwKVYom+OFqrgd9cpLI/9Vpqfhr4I= X-Gm-Message-State: AOJu0Ywo4wyjKqZOIcYYzvm99qwP1WaTnPH3ztDAIuv+mduDcBEtNQih W3YSSXCUi+F6jSmDUrzK4fDajL24eFWR2xSv7vEps4nAwYTI0KERhjNrz9CluEqcdIJW1kX9hDA m58fmPnuhfn8atbbu4ZJbUQssoFpBMZ+0htR8IpIbFNB/1r/v X-Received: by 2002:a05:622a:8f:b0:446:3c97:d3ab with SMTP id d75a77b69052e-44662eb7436mr151118481cf.4.1720042179676; Wed, 03 Jul 2024 14:29:39 -0700 (PDT) X-Google-Smtp-Source: AGHT+IG9nYtpxfPIDPA/kSFmDjfHMlL7UyJZQtNoCKoRdFeJRABCZlINs1O4dDPbOXn5gaxSQwwtsA== X-Received: by 2002:a05:622a:8f:b0:446:3c97:d3ab with SMTP id d75a77b69052e-44662eb7436mr151118391cf.4.1720042179335; Wed, 03 Jul 2024 14:29:39 -0700 (PDT) Received: from x1n.redhat.com (pool-99-254-121-117.cpe.net.cable.rogers.com. [99.254.121.117]) by smtp.gmail.com with ESMTPSA id d75a77b69052e-4465143eb1csm54337481cf.57.2024.07.03.14.29.37 (version=TLS1_3 cipher=TLS_AES_256_GCM_SHA384 bits=256/256); Wed, 03 Jul 2024 14:29:39 -0700 (PDT) From: Peter Xu To: linux-kernel@vger.kernel.org, linux-mm@kvack.org Cc: Matthew Wilcox , Mel Gorman , Dave Jiang , linuxppc-dev@lists.ozlabs.org, Michael Ellerman , Rik van Riel , Vlastimil Babka , Nicholas Piggin , Christophe Leroy , Andrew Morton , Huang Ying , Oscar Salvador , "Aneesh Kumar K . V" , Thomas Gleixner , Dave Hansen , x86@kernel.org, Ingo Molnar , "Kirill A . Shutemov" , Dan Williams , Borislav Petkov , peterx@redhat.com, Hugh Dickins , Rick P Edgecombe Subject: [PATCH v2 8/8] mm/mprotect: fix dax pud handlings Date: Wed, 3 Jul 2024 17:29:18 -0400 Message-ID: <20240703212918.2417843-9-peterx@redhat.com> X-Mailer: git-send-email 2.45.0 In-Reply-To: <20240703212918.2417843-1-peterx@redhat.com> References: <20240703212918.2417843-1-peterx@redhat.com> MIME-Version: 1.0 X-Mimecast-Spam-Score: 0 X-Mimecast-Originator: redhat.com X-Rspamd-Server: rspam11 X-Rspamd-Queue-Id: 3F14B4000F X-Stat-Signature: 6r8dsr1c59bqu6foot11ia4djt6x6x96 X-Rspam-User: X-HE-Tag: 1720042182-809562 X-HE-Meta: U2FsdGVkX1+fqVPzSPqc4xUg+Iz1nVlkD7+A7rwElEoY275v+Y8hFFDRF4ut7VpTX5oiv4ynmDgDsxlliYVPY97io/GUiYUcD/RMafSyP1ELa6SYCzbH+RaVJArEtxMAIV6W1FNJJhnpidEB9SNMS6shUp6V4pOV47sV9hhCAPEwlM4gy1DbHiLtvf8msReMnH0JP167ZCAHsajXpmxqZtCYGY1CZYigDvhGHuSX4bfovQX3eVTy8O1K5x0THlQj9dDyfQ3yowBaZVOewpc8AnbF7imFjp7RJybb+6rSdSrVWBasW6OYed62G2iiJLROZ0NpuvWJop/QTJOmHp9SgLfxnyYjq2/ODSj1FRwl8ZEnAOwQ0vKOh0IRU85kiL7Aya1tiI/pyO9Q4NZHOkX3k3ckl5WlYHm95QU6WNcGd4DqF4/ENrZAlhGwfkBbvRd3ocJkxDPi9uEB/Lk+Het8ZRxoYuT23XJ+tbSDZnDcP+ej571NVA0rOtcpwet7r3iGV27cwQuX8oDaWt+pJTh+fwkK0G73oYBlav/JWTn2Y2Rg5DHF/SMDW0hUzonTtL9+qKGNYxh4uaG6aF+GC7/RNmoffGyDkc2fCjAQyKGlSdeoAh6fhIxHu5MCSE9nGEv/SIKuEOuAb+6H15grPuCGKMmvgCeSoBkpD1gI4bwLc2cXsUDnoiUxJtJjLzR3pI17DzDsX6DYCidiZwC75B/f31HBPGT18MguHsQkBpFwKZyAa/0YMS8fpvNs/fbyh3+DVPWdziegOBmSNxVmXMOiXf2fUtDyqDhDuAdGgNsQErEPvUbdIDSL4p3mAiNIu88u+LCZaCRGMK9k2/sJ7o45Xqa8NpxjZUffeM/0ObIvcgGbIghj4zGc+B1Jcc9zRQWF2vvnQ2cL7mdpJapHZAzDj08iYxCpNxLGz55ORUxWrRZG0ch/QaqzSPb0B1j0jeEi+RC+Mgjvu+adPS190Qz zQkUZKFn 030pYrLIcaVSvvKMggJV4hi3Ux9cHNljO2RTBgus7JRw1Lxbzdxi0jo1b02A/t3nFyJpqSppaZkbVxMXcm+hvwRTyWh+HpsO3u0OrjRfGuLuXnNQN8Uo6SXGKeBAmX4I2W6vFtXM1TWYrX/pBry9bQC3Mhcbo7RoXnMFqiZ50J8zh850JHLyZJWcLBc3iwz9ggZ6cgFGcUu4oTGqLnC+wkyQgvtVbh5l2piRQwl2bfyzjT2VLxOtrIqhduQOlZULUIX6xMeeeEkVF+B7l2qWVcg4v5jb4FrLHYpp7xJ1y97LC69MyKtJz9HMMNB6cSNnfVxtUpsvfqCZ2rFmj/hmP+Xwyl8gYSqGbFKcR9T+Ng/sp5kTEPyE+t1UwuWkN2zc6Ktqw9pSZO7i6ZTx/BX6zD7I+zTpzpNRK69H+xXI0k7MlIc6CGEOi/CkdzQ03+Y/0tFqojmCxsKZpjjUW0pd9ltnJdkVnpI63XqnM9/FSHJTVoszVWm9l9dXEuKeSP7S+c4pfSnjbjoJJdQBgDBEGGJHaluYU8bRZIjiE20T9iwY5D15G0dI9UDV8S1th/GtHxi0Z28K86CY/Wo4= X-Bogosity: Ham, tests=bogofilter, spamicity=0.000000, version=1.2.4 Sender: owner-linux-mm@kvack.org Precedence: bulk X-Loop: owner-majordomo@kvack.org List-ID: List-Subscribe: List-Unsubscribe: This is only relevant to the two archs that support PUD dax, aka, x86_64 and ppc64. PUD THPs do not yet exist elsewhere, and hugetlb PUDs do not count in this case. DAX have had PUD mappings for years, but change protection path never worked. When the path is triggered in any form (a simple test program would be: call mprotect() on a 1G dev_dax mapping), the kernel will report "bad pud". This patch should fix that. The new change_huge_pud() tries to keep everything simple. For example, it doesn't optimize write bit as that will need even more PUD helpers. It's not too bad anyway to have one more write fault in the worst case once for 1G range; may be a bigger thing for each PAGE_SIZE, though. Neither does it support userfault-wp bits, as there isn't such PUD mappings that is supported; file mappings always need a split there. The same to TLB shootdown: the pmd path (which was for x86 only) has the trick of using _ad() version of pmdp_invalidate*() which can avoid one redundant TLB, but let's also leave that for later. Again, the larger the mapping, the smaller of such effect. Another thing worth mention is this path needs to be careful on handling "retry" event for change_huge_pud() (where it can return 0): it isn't like change_huge_pmd(), as the pmd version is safe with all conditions handled in change_pte_range() later, thanks to Hugh's new pte_offset_map_lock(). In short, change_pte_range() is simply smarter than change_pmd_range() now after the shmem thp collapse rework. For that reason, change_pud_range() will need proper retry if it races with something else when a huge PUD changed from under us. Cc: Dan Williams Cc: Matthew Wilcox Cc: Dave Jiang Cc: Hugh Dickins Cc: Kirill A. Shutemov Cc: Vlastimil Babka Cc: Thomas Gleixner Cc: Ingo Molnar Cc: Borislav Petkov Cc: Dave Hansen Cc: Michael Ellerman Cc: Aneesh Kumar K.V Cc: Oscar Salvador Cc: x86@kernel.org Cc: linuxppc-dev@lists.ozlabs.org Fixes: a00cc7d9dd93 ("mm, x86: add support for PUD-sized transparent hugepages") Fixes: 27af67f35631 ("powerpc/book3s64/mm: enable transparent pud hugepage") Signed-off-by: Peter Xu --- include/linux/huge_mm.h | 24 +++++++++++++++++++ mm/huge_memory.c | 52 +++++++++++++++++++++++++++++++++++++++++ mm/mprotect.c | 40 ++++++++++++++++++++++++------- 3 files changed, 108 insertions(+), 8 deletions(-) diff --git a/include/linux/huge_mm.h b/include/linux/huge_mm.h index cee3c5da8f0e..2ebebadfbe47 100644 --- a/include/linux/huge_mm.h +++ b/include/linux/huge_mm.h @@ -348,6 +348,17 @@ void split_huge_pmd_address(struct vm_area_struct *vma, unsigned long address, void __split_huge_pud(struct vm_area_struct *vma, pud_t *pud, unsigned long address); +#ifdef CONFIG_HAVE_ARCH_TRANSPARENT_HUGEPAGE_PUD +int change_huge_pud(struct mmu_gather *tlb, struct vm_area_struct *vma, + pud_t *pudp, unsigned long addr, pgprot_t newprot, + unsigned long cp_flags); +#else +static inline int +change_huge_pud(struct mmu_gather *tlb, struct vm_area_struct *vma, + pud_t *pudp, unsigned long addr, pgprot_t newprot, + unsigned long cp_flags) { return 0; } +#endif + #define split_huge_pud(__vma, __pud, __address) \ do { \ pud_t *____pud = (__pud); \ @@ -591,6 +602,19 @@ static inline int next_order(unsigned long *orders, int prev) { return 0; } + +static inline void __split_huge_pud(struct vm_area_struct *vma, pud_t *pud, + unsigned long address) +{ +} + +static inline int change_huge_pud(struct mmu_gather *tlb, + struct vm_area_struct *vma, pud_t *pudp, + unsigned long addr, pgprot_t newprot, + unsigned long cp_flags) +{ + return 0; +} #endif /* CONFIG_TRANSPARENT_HUGEPAGE */ static inline int split_folio_to_list_to_order(struct folio *folio, diff --git a/mm/huge_memory.c b/mm/huge_memory.c index 017377920d18..4998452f7c49 100644 --- a/mm/huge_memory.c +++ b/mm/huge_memory.c @@ -2099,6 +2099,53 @@ int change_huge_pmd(struct mmu_gather *tlb, struct vm_area_struct *vma, return ret; } +/* + * Returns: + * + * - 0: if pud leaf changed from under us + * - 1: if pud can be skipped + * - HPAGE_PUD_NR: if pud was successfully processed + */ +#ifdef CONFIG_HAVE_ARCH_TRANSPARENT_HUGEPAGE_PUD +int change_huge_pud(struct mmu_gather *tlb, struct vm_area_struct *vma, + pud_t *pudp, unsigned long addr, pgprot_t newprot, + unsigned long cp_flags) +{ + struct mm_struct *mm = vma->vm_mm; + pud_t oldpud, entry; + spinlock_t *ptl; + + tlb_change_page_size(tlb, HPAGE_PUD_SIZE); + + /* NUMA balancing doesn't apply to dax */ + if (cp_flags & MM_CP_PROT_NUMA) + return 1; + + /* + * Huge entries on userfault-wp only works with anonymous, while we + * don't have anonymous PUDs yet. + */ + if (WARN_ON_ONCE(cp_flags & MM_CP_UFFD_WP_ALL)) + return 1; + + ptl = __pud_trans_huge_lock(pudp, vma); + if (!ptl) + return 0; + + /* + * Can't clear PUD or it can race with concurrent zapping. See + * change_huge_pmd(). + */ + oldpud = pudp_invalidate(vma, addr, pudp); + entry = pud_modify(oldpud, newprot); + set_pud_at(mm, addr, pudp, entry); + tlb_flush_pud_range(tlb, addr, HPAGE_PUD_SIZE); + + spin_unlock(ptl); + return HPAGE_PUD_NR; +} +#endif + #ifdef CONFIG_USERFAULTFD /* * The PT lock for src_pmd and dst_vma/src_vma (for reading) are locked by @@ -2329,6 +2376,11 @@ void __split_huge_pud(struct vm_area_struct *vma, pud_t *pud, spin_unlock(ptl); mmu_notifier_invalidate_range_end(&range); } +#else +void __split_huge_pud(struct vm_area_struct *vma, pud_t *pud, + unsigned long address) +{ +} #endif /* CONFIG_HAVE_ARCH_TRANSPARENT_HUGEPAGE_PUD */ static void __split_huge_zero_page_pmd(struct vm_area_struct *vma, diff --git a/mm/mprotect.c b/mm/mprotect.c index fb8bf3ff7cd9..694f13b83864 100644 --- a/mm/mprotect.c +++ b/mm/mprotect.c @@ -425,29 +425,53 @@ static inline long change_pud_range(struct mmu_gather *tlb, unsigned long end, pgprot_t newprot, unsigned long cp_flags) { struct mmu_notifier_range range; - pud_t *pud; + pud_t *pudp, pud; unsigned long next; long pages = 0, ret; range.start = 0; - pud = pud_offset(p4d, addr); + pudp = pud_offset(p4d, addr); do { +again: next = pud_addr_end(addr, end); - ret = change_prepare(vma, pud, pmd, addr, cp_flags); - if (ret) - return ret; - if (pud_none_or_clear_bad(pud)) + ret = change_prepare(vma, pudp, pmd, addr, cp_flags); + if (ret) { + pages = ret; + break; + } + + pud = READ_ONCE(*pudp); + if (pud_none(pud)) continue; + if (!range.start) { mmu_notifier_range_init(&range, MMU_NOTIFY_PROTECTION_VMA, 0, vma->vm_mm, addr, end); mmu_notifier_invalidate_range_start(&range); } - pages += change_pmd_range(tlb, vma, pud, addr, next, newprot, + + if (pud_leaf(pud)) { + if ((next - addr != PUD_SIZE) || + pgtable_split_needed(vma, cp_flags)) { + __split_huge_pud(vma, pudp, addr); + goto again; + } else { + ret = change_huge_pud(tlb, vma, pudp, + addr, newprot, cp_flags); + if (ret == 0) + goto again; + /* huge pud was handled */ + if (ret == HPAGE_PUD_NR) + pages += HPAGE_PUD_NR; + continue; + } + } + + pages += change_pmd_range(tlb, vma, pudp, addr, next, newprot, cp_flags); - } while (pud++, addr = next, addr != end); + } while (pudp++, addr = next, addr != end); if (range.start) mmu_notifier_invalidate_range_end(&range);