From patchwork Mon Aug 12 18:12:19 2024 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Peter Xu X-Patchwork-Id: 13760906 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from kanga.kvack.org (kanga.kvack.org [205.233.56.17]) by smtp.lore.kernel.org (Postfix) with ESMTP id 46492C52D7C for ; Mon, 12 Aug 2024 18:12:37 +0000 (UTC) Received: by kanga.kvack.org (Postfix) id B23196B008A; Mon, 12 Aug 2024 14:12:36 -0400 (EDT) Received: by kanga.kvack.org (Postfix, from userid 40) id AD2886B008C; Mon, 12 Aug 2024 14:12:36 -0400 (EDT) X-Delivered-To: int-list-linux-mm@kvack.org Received: by kanga.kvack.org (Postfix, from userid 63042) id 973806B0095; Mon, 12 Aug 2024 14:12:36 -0400 (EDT) X-Delivered-To: linux-mm@kvack.org Received: from relay.hostedemail.com (smtprelay0017.hostedemail.com [216.40.44.17]) by kanga.kvack.org (Postfix) with ESMTP id 7A33D6B008A for ; Mon, 12 Aug 2024 14:12:36 -0400 (EDT) Received: from smtpin05.hostedemail.com (a10.router.float.18 [10.200.18.1]) by unirelay03.hostedemail.com (Postfix) with ESMTP id 2366AA032A for ; Mon, 12 Aug 2024 18:12:36 +0000 (UTC) X-FDA: 82444388712.05.9F3AECC Received: from us-smtp-delivery-124.mimecast.com (us-smtp-delivery-124.mimecast.com [170.10.129.124]) by imf08.hostedemail.com (Postfix) with ESMTP id 96FD6160013 for ; Mon, 12 Aug 2024 18:12:33 +0000 (UTC) Authentication-Results: imf08.hostedemail.com; dkim=pass header.d=redhat.com header.s=mimecast20190719 header.b=QOVdmOce; spf=pass (imf08.hostedemail.com: domain of peterx@redhat.com designates 170.10.129.124 as permitted sender) smtp.mailfrom=peterx@redhat.com; dmarc=pass (policy=none) header.from=redhat.com ARC-Seal: i=1; s=arc-20220608; d=hostedemail.com; t=1723486318; a=rsa-sha256; cv=none; b=gt+xSlx9XV+2d4dCnChhJZaxw3cJBD2PGqYpcIZaN4O1lT01x6S3ItSC5+vWK8cqGbZ/2a 3Zu9AdWJ7iDf3eT9FQwUgZ6P22wtfz0Igi5rBoRRfFsc7aVGTxlZbjANY+iJHIUC4GtTGc XFb2uisdnTncm6AZCXkPXAsWmTc0I94= ARC-Authentication-Results: i=1; imf08.hostedemail.com; dkim=pass header.d=redhat.com header.s=mimecast20190719 header.b=QOVdmOce; spf=pass (imf08.hostedemail.com: domain of peterx@redhat.com designates 170.10.129.124 as permitted sender) smtp.mailfrom=peterx@redhat.com; dmarc=pass (policy=none) header.from=redhat.com ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=hostedemail.com; s=arc-20220608; t=1723486318; h=from:from:sender:reply-to:subject:subject:date:date: message-id:message-id:to:to:cc:cc:mime-version:mime-version: content-type:content-type: content-transfer-encoding:content-transfer-encoding: in-reply-to:in-reply-to:references:references:dkim-signature; bh=9Kq72BfB1Q9PD26Udwsuz2Gt7rCvKvwunl3vkpU7op8=; b=TrHOLuhMuRpGwaIiWeZEkDrb2UG/rciJ3bz4Y0aQnvw36ZsnlWyWgfIxfGE28lBCj/ACKv EcmepIAMgNBZK/SyNibOTXFfteYdGBDx5HSJhXZ6vBXs7gDZnn4127QbXDh67slmsc0rCG 48XOwRgzmu4Ylb2eLRxa+X7PYRTiycM= DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=redhat.com; s=mimecast20190719; t=1723486353; h=from:from:reply-to:subject:subject:date:date:message-id:message-id: to:to:cc:cc:mime-version:mime-version:content-type:content-type: content-transfer-encoding:content-transfer-encoding: in-reply-to:in-reply-to:references:references; bh=9Kq72BfB1Q9PD26Udwsuz2Gt7rCvKvwunl3vkpU7op8=; b=QOVdmOce2ZaDIdP/Ry9Xt6G9PauDrffoP6DVk56imp6RchXFBqE6v0n0VbjggQ4ThUB482 A2lEvxPMn/8u4PPf1qTlLWCbAxMLA/mdV4i9WqMy0c9GLuoPHUo8RRMFBMdosJJgPd4RNN ugKfTugLE4mf2evWkHqvh8pgZnkBTLY= Received: from mail-qv1-f72.google.com (mail-qv1-f72.google.com [209.85.219.72]) by relay.mimecast.com with ESMTP with STARTTLS (version=TLSv1.3, cipher=TLS_AES_256_GCM_SHA384) id us-mta-147-1P-d02t3OEG4D7S_nzh6ew-1; Mon, 12 Aug 2024 14:12:31 -0400 X-MC-Unique: 1P-d02t3OEG4D7S_nzh6ew-1 Received: by mail-qv1-f72.google.com with SMTP id 6a1803df08f44-6b7ad98c1f8so10692226d6.1 for ; Mon, 12 Aug 2024 11:12:31 -0700 (PDT) X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20230601; t=1723486351; x=1724091151; h=content-transfer-encoding:mime-version:references:in-reply-to :message-id:date:subject:cc:to:from:x-gm-message-state:from:to:cc :subject:date:message-id:reply-to; bh=9Kq72BfB1Q9PD26Udwsuz2Gt7rCvKvwunl3vkpU7op8=; b=AwDp2NJoFZFdzL+P84uaZkw3yNE7GPav7eAhlvcSs4/eS65dQPHZhiA5yXIJ099XZJ /J3NbgEwVk/zZAr+BR3l3BuT1mwV+PgJcCb3XJY14L+4rRIFqz/28RRcRBgMPhNuvSLr zPTtmJOF47y94cp75egMj/uLNQ+rH/DT+JTRytGCRzPp7lHkQHrMiqAzFmbvGpw7bG7J pmwW9+CXBbxl1OtX2OOloXfs4DDWmPq3m5huWS7ZO3tl2dgjbWpoHeRMH3TEmkCcgZlk U1KmRcd5l2L1zpreFAri6KccI7IW/A6Y0SWso26ymoaq78ygOvepDrztw561GNKcvTzX FsMw== X-Forwarded-Encrypted: i=1; AJvYcCU0wlTisz9la1xCszoUqRX6P9nBKA07nXByfA5ia008tGkf4EHn2Ox0Qa/PxkJXScZ95tdibRzg4Q==@kvack.org X-Gm-Message-State: AOJu0YwtPxwEDXqMhpJjIaWvZGZXPGC/D2eBjxsdDxnZjEsLLe4oYO1G u8xx+UDHus4vgNByrf5dvCC7OT1XZspYqJsitenBD1kAmaoFw3DusZXAv3qdefg0Cv8rQJG9aPl q+ror0Zx16/P/euoTz5gwX7S4T3Eckm1loXWIXAC4vOpYgbJX X-Received: by 2002:a05:620a:4005:b0:7a3:49e2:d5 with SMTP id af79cd13be357-7a4e15b7f45mr75248485a.5.1723486351141; Mon, 12 Aug 2024 11:12:31 -0700 (PDT) X-Google-Smtp-Source: AGHT+IGINV1er94KHea7itgcrlskkFbFJC5xxX91ZMY2mT7mbgIaqz9jLoR0x5JTp0pkA3b+65JpVA== X-Received: by 2002:a05:620a:4005:b0:7a3:49e2:d5 with SMTP id af79cd13be357-7a4e15b7f45mr75245485a.5.1723486350655; Mon, 12 Aug 2024 11:12:30 -0700 (PDT) Received: from x1n.redhat.com (pool-99-254-121-117.cpe.net.cable.rogers.com. [99.254.121.117]) by smtp.gmail.com with ESMTPSA id af79cd13be357-7a4c7dee013sm268663985a.84.2024.08.12.11.12.28 (version=TLS1_3 cipher=TLS_AES_256_GCM_SHA384 bits=256/256); Mon, 12 Aug 2024 11:12:30 -0700 (PDT) From: Peter Xu To: linux-kernel@vger.kernel.org, linux-mm@kvack.org Cc: "Kirill A . Shutemov" , Nicholas Piggin , David Hildenbrand , Matthew Wilcox , Andrew Morton , James Houghton , Huang Ying , "Aneesh Kumar K . V" , peterx@redhat.com, Vlastimil Babka , Rick P Edgecombe , Hugh Dickins , Borislav Petkov , Christophe Leroy , Michael Ellerman , Rik van Riel , Dan Williams , Mel Gorman , x86@kernel.org, Ingo Molnar , linuxppc-dev@lists.ozlabs.org, Dave Hansen , Dave Jiang , Oscar Salvador , Thomas Gleixner Subject: [PATCH v5 1/7] mm/dax: Dump start address in fault handler Date: Mon, 12 Aug 2024 14:12:19 -0400 Message-ID: <20240812181225.1360970-2-peterx@redhat.com> X-Mailer: git-send-email 2.45.0 In-Reply-To: <20240812181225.1360970-1-peterx@redhat.com> References: <20240812181225.1360970-1-peterx@redhat.com> MIME-Version: 1.0 X-Mimecast-Spam-Score: 0 X-Mimecast-Originator: redhat.com X-Stat-Signature: r946n1qpbd71gyor5gct4tussstgyx43 X-Rspamd-Queue-Id: 96FD6160013 X-Rspam-User: X-Rspamd-Server: rspam10 X-HE-Tag: 1723486353-476877 X-HE-Meta: U2FsdGVkX19HQ+kX9vZrk9XIZILXrk0KncLzd6WpUrJO2H9+zcT2tpyEMgaM5ICGOPeFSfdN7spfCHyPQjnfd9F2yNhWkNdmc39/dTl6UpsHUhUQ0ONTtC01I0CInchz6JBxyPRu/bsMVImN9yB8Y6Fzg/dl/dND82RmUm0IIYMROFpIprx+48ShZbEWM75OHWUOYi07OODYv4y0lnKJhu686hNBRcdsed62rq5KrPdddBDxztLG70kqMwXmQKTVf54nDhIY4Eai/YLieJGYkOVstS7JeD3rpnaMKOVqJrDdDNnvt/6yR1MN4b4lmEQEmGWdC+k0H65AqLsXPC5EUleeN89N9aC8i5vYN5nN+gxVucaiwHcE80w+kbP57V1SzDGQ078+hGd5JWAkEeCOWlUxudzS4Xw6+aa2Nuh28gYKfQBGgexpDCVnSjovQZaWJxeSBWHWRYIrQntljzk/6WzidRsR4nUvoIx4eA6ebTvS8WznvUxjSjs7VgvdBWgASJpU1slk5sjvuVbuWpNXOIAAY7SMSagmRaI5YU8YTyBa1AM+PPV5KzjfAP3bGro+NzKX4yEOPHOU5nUe2VOICfJY5lfdT6r6KPF/uow1d72So+2ADgei7749L5xdzGzq3g3+kneVYDpQIuRcbh5dfXiJShb6b7Ugq7hs+lL7DbrtidDld5n8CRr1xdKn/Pwu7j9bxCLWJ90Mw06w7bbnvMBhMdwVb3BoKDngbKGO6qpS03A4vxHAaV4ZaK8nLK7eh7kGSAYWhzyIlTg1djf75gSDNRd91hZ20TaZU0gvgOP0iFtLYLP80/KILLV3P30aJveoqtKjd+ZMLtERAvIlWdcd/QlgaHTGSi3nMb8y7fJ1pH8IRG/avbupowstfPjlk204gvMD9xLvLHuCGPaXllkO2DCamG70OghnOPxWoDr7jFWdM+eEVb2OlJu7a9aTQxxfa0bmu3RvEPp6GuZ prGl5KWc aIKcc/7noja7gfN81u5e2Hw5dRK+J3V0ozDPRayS/GF6nAYl0z/LT5XZhmNyMRZ5mk62/0qOG0XHuAuhC0vRkzVHlU6hEywBguf1qnobaYsgYX49RZB8+uteVnumWEx6V/FtIcDvbETaATmBmp2ZMTL3kgu5j7cnaEK2q/anbkjPNlKOfeupBMUF7b9EDbFjP6I7eGRG62ZFN7JcXqb6OdftQQwHGxyfLb2e+/39iwTCwHdiDekNm9NOLWnyUMKiiLZ+xPLPglbX2pvkch4unZPfdZ5HVH0YHOF8qxA7gct1zJ0qUuAPE51Mt6ewVJ8SHoBA+1O8lkJks//M+Nz8X8mPuNg== X-Bogosity: Ham, tests=bogofilter, spamicity=0.000000, version=1.2.4 Sender: owner-linux-mm@kvack.org Precedence: bulk X-Loop: owner-majordomo@kvack.org List-ID: List-Subscribe: List-Unsubscribe: Currently the dax fault handler dumps the vma range when dynamic debugging enabled. That's mostly not useful. Dump the (aligned) address instead with the order info. Acked-by: David Hildenbrand Signed-off-by: Peter Xu --- drivers/dax/device.c | 6 +++--- 1 file changed, 3 insertions(+), 3 deletions(-) diff --git a/drivers/dax/device.c b/drivers/dax/device.c index 2051e4f73c8a..9c1a729cd77e 100644 --- a/drivers/dax/device.c +++ b/drivers/dax/device.c @@ -235,9 +235,9 @@ static vm_fault_t dev_dax_huge_fault(struct vm_fault *vmf, unsigned int order) int id; struct dev_dax *dev_dax = filp->private_data; - dev_dbg(&dev_dax->dev, "%s: %s (%#lx - %#lx) order:%d\n", current->comm, - (vmf->flags & FAULT_FLAG_WRITE) ? "write" : "read", - vmf->vma->vm_start, vmf->vma->vm_end, order); + dev_dbg(&dev_dax->dev, "%s: op=%s addr=%#lx order=%d\n", current->comm, + (vmf->flags & FAULT_FLAG_WRITE) ? "write" : "read", + vmf->address & ~((1UL << (order + PAGE_SHIFT)) - 1), order); id = dax_read_lock(); if (order == 0) From patchwork Mon Aug 12 18:12:20 2024 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Peter Xu X-Patchwork-Id: 13760909 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from kanga.kvack.org (kanga.kvack.org [205.233.56.17]) by smtp.lore.kernel.org (Postfix) with ESMTP id 8C41BC52D7C for ; Mon, 12 Aug 2024 18:12:40 +0000 (UTC) Received: by kanga.kvack.org (Postfix) id 2BEC36B0095; Mon, 12 Aug 2024 14:12:39 -0400 (EDT) Received: by kanga.kvack.org (Postfix, from userid 40) id 26E7E6B0098; Mon, 12 Aug 2024 14:12:39 -0400 (EDT) X-Delivered-To: int-list-linux-mm@kvack.org Received: by kanga.kvack.org (Postfix, from userid 63042) id 0E8776B009A; Mon, 12 Aug 2024 14:12:39 -0400 (EDT) X-Delivered-To: linux-mm@kvack.org Received: from relay.hostedemail.com (smtprelay0016.hostedemail.com [216.40.44.16]) by kanga.kvack.org (Postfix) with ESMTP id E371C6B0095 for ; Mon, 12 Aug 2024 14:12:38 -0400 (EDT) Received: from smtpin02.hostedemail.com (a10.router.float.18 [10.200.18.1]) by unirelay06.hostedemail.com (Postfix) with ESMTP id 9B527A51A6 for ; Mon, 12 Aug 2024 18:12:38 +0000 (UTC) X-FDA: 82444388796.02.6E096F8 Received: from us-smtp-delivery-124.mimecast.com (us-smtp-delivery-124.mimecast.com [170.10.133.124]) by imf23.hostedemail.com (Postfix) with ESMTP id 7A5B1140018 for ; Mon, 12 Aug 2024 18:12:36 +0000 (UTC) Authentication-Results: imf23.hostedemail.com; dkim=pass header.d=redhat.com header.s=mimecast20190719 header.b=bb7ChQBZ; spf=pass (imf23.hostedemail.com: domain of peterx@redhat.com designates 170.10.133.124 as permitted sender) smtp.mailfrom=peterx@redhat.com; dmarc=pass (policy=none) header.from=redhat.com ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=hostedemail.com; s=arc-20220608; t=1723486263; h=from:from:sender:reply-to:subject:subject:date:date: message-id:message-id:to:to:cc:cc:mime-version:mime-version: content-type:content-type: content-transfer-encoding:content-transfer-encoding: in-reply-to:in-reply-to:references:references:dkim-signature; bh=6aC2RF+//p0GJfjgH/iZrlW9ATLJl4mVfe8Rdc4jHgk=; b=eN4cChNeXMZExph9JfQ+Mm/8kIZkfQLDG5LFAn77MnAsEOBzmqGuEaedOqYbTNG2rqS+/D 0qDtLx0+YdGCwAfG7kq4pGzUwg8M1B4bKcgQvz8lg/3dWhQ8KjMBY0T76TxxfAV6+f7ukA ER73EMVGeSYjg20RQVhzuke13SycffM= ARC-Authentication-Results: i=1; imf23.hostedemail.com; dkim=pass header.d=redhat.com header.s=mimecast20190719 header.b=bb7ChQBZ; spf=pass (imf23.hostedemail.com: domain of peterx@redhat.com designates 170.10.133.124 as permitted sender) smtp.mailfrom=peterx@redhat.com; dmarc=pass (policy=none) header.from=redhat.com ARC-Seal: i=1; s=arc-20220608; d=hostedemail.com; t=1723486263; a=rsa-sha256; cv=none; b=8K8qUOapwMDsLELiIcA6aoRE0On4Qvr4lXDINTMHIxRpeBY+4ckSIQOOExFjoN1SIMO4S6 v5Y5N7WKRkj3JT2jGcXV+WG6hejA8fTxT/0UPLQ1MfoCHwv9RUZY2bNTFTprDmZB5nVTGv TiVv/YaROZyYKy7/sBjikCHkWiZKZL8= DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=redhat.com; s=mimecast20190719; t=1723486355; h=from:from:reply-to:subject:subject:date:date:message-id:message-id: to:to:cc:cc:mime-version:mime-version:content-type:content-type: content-transfer-encoding:content-transfer-encoding: in-reply-to:in-reply-to:references:references; bh=6aC2RF+//p0GJfjgH/iZrlW9ATLJl4mVfe8Rdc4jHgk=; b=bb7ChQBZ34guu+6NgVUqrcZ4p2b7xHoIyLPnUD4fbKiMTybEDIVP/J7xKfUK5KtJZizm36 XLK3ba00vK5iOg7AKxui/hOfGmpWzIQbuyjrsU7b7i/Tn0M4i/gKY5CZxbkdmt2w0cDx1p 1vYQ6t8oFWyRuKnxje95pyAXm0eNYq4= Received: from mail-ot1-f70.google.com (mail-ot1-f70.google.com [209.85.210.70]) by relay.mimecast.com with ESMTP with STARTTLS (version=TLSv1.3, cipher=TLS_AES_256_GCM_SHA384) id us-mta-620-DD7E5LD0NWyc597mxy-oXg-1; Mon, 12 Aug 2024 14:12:34 -0400 X-MC-Unique: DD7E5LD0NWyc597mxy-oXg-1 Received: by mail-ot1-f70.google.com with SMTP id 46e09a7af769-7093f4569b3so148706a34.1 for ; Mon, 12 Aug 2024 11:12:34 -0700 (PDT) X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20230601; t=1723486354; x=1724091154; h=content-transfer-encoding:mime-version:references:in-reply-to :message-id:date:subject:cc:to:from:x-gm-message-state:from:to:cc :subject:date:message-id:reply-to; bh=6aC2RF+//p0GJfjgH/iZrlW9ATLJl4mVfe8Rdc4jHgk=; b=WmQuEkqBVHG3KLW3Wu6XrpC/9kwUBlIMc5dd+YuApwJ0I9R2SFOW7dTaInfrgTUlu8 N/8NXhfm7DKtIEpBH1r2sGIS1pv50aDlfSjbA1Ortucn02eK4PeqHC/IfuWXJMYRgFL1 wTbks//QnJ+5ri1vjbK9q1rTnocX51xmXCXMsj3TRmkJvWomtPhxcNnQRFIf9Frn5K9h ErWe5AasBbhSSus4hQWMdAFQ+K3eBBTOqFOkOrfc/ukQhlRa7cy4JZXbAkgxRyygl0xC MJzlXbTl370V96xbS1wEBpCk/PTNZFlbsfMnovcrDICk0o9mlwk1qlquNg2jTjCS9jNK WHPQ== X-Forwarded-Encrypted: i=1; AJvYcCXPD41xqXPJerhkAinWH+ycZ086qZU4b8hk/Q7s/kD/Laf3THC0jh/FC4DNLekcS3FxKOkf2ht6KWWQt4IpNTgFiSM= X-Gm-Message-State: AOJu0Yw9IZBhEONVV9Je6uKoCBdoIM3JQs4O1xv/xumPOCNkHYzzYNpA sL74+7ZeIFSfmc238qEJbrWEfgoUxpt2TqKz4B2HsbvZRbTM/dTwN6BIg5/rR1c2vt+Afn1/Ed0 qN/0z5JL4Luo96iuzBMlix5HIiTRO046RhYklUcGmVQWcNZwn X-Received: by 2002:a05:6358:d25:b0:1ac:a26c:a07a with SMTP id e5c5f4694b2df-1b1a02f2896mr3555d.4.1723486353704; Mon, 12 Aug 2024 11:12:33 -0700 (PDT) X-Google-Smtp-Source: AGHT+IFKB0h9tqBDznq7jPudw+rKB1F4j/lDBjBP5vz8Ij/mXfeVHheDU0KW5XKnOwJHdaPAJFxqTQ== X-Received: by 2002:a05:6358:d25:b0:1ac:a26c:a07a with SMTP id e5c5f4694b2df-1b1a02f2896mr2555d.4.1723486353186; Mon, 12 Aug 2024 11:12:33 -0700 (PDT) Received: from x1n.redhat.com (pool-99-254-121-117.cpe.net.cable.rogers.com. [99.254.121.117]) by smtp.gmail.com with ESMTPSA id af79cd13be357-7a4c7dee013sm268663985a.84.2024.08.12.11.12.30 (version=TLS1_3 cipher=TLS_AES_256_GCM_SHA384 bits=256/256); Mon, 12 Aug 2024 11:12:32 -0700 (PDT) From: Peter Xu To: linux-kernel@vger.kernel.org, linux-mm@kvack.org Cc: "Kirill A . Shutemov" , Nicholas Piggin , David Hildenbrand , Matthew Wilcox , Andrew Morton , James Houghton , Huang Ying , "Aneesh Kumar K . V" , peterx@redhat.com, Vlastimil Babka , Rick P Edgecombe , Hugh Dickins , Borislav Petkov , Christophe Leroy , Michael Ellerman , Rik van Riel , Dan Williams , Mel Gorman , x86@kernel.org, Ingo Molnar , linuxppc-dev@lists.ozlabs.org, Dave Hansen , Dave Jiang , Oscar Salvador , Thomas Gleixner , kvm@vger.kernel.org, Sean Christopherson , Paolo Bonzini , David Rientjes Subject: [PATCH v5 2/7] mm/mprotect: Push mmu notifier to PUDs Date: Mon, 12 Aug 2024 14:12:20 -0400 Message-ID: <20240812181225.1360970-3-peterx@redhat.com> X-Mailer: git-send-email 2.45.0 In-Reply-To: <20240812181225.1360970-1-peterx@redhat.com> References: <20240812181225.1360970-1-peterx@redhat.com> MIME-Version: 1.0 X-Mimecast-Spam-Score: 0 X-Mimecast-Originator: redhat.com X-Rspamd-Server: rspam06 X-Rspamd-Queue-Id: 7A5B1140018 X-Stat-Signature: iny98uj6seqf45k9hocd6n331uhfog18 X-Rspam-User: X-HE-Tag: 1723486356-604144 X-HE-Meta: U2FsdGVkX19rYTua3V7J7kLGEGrO9NcodFM7GvZFO+yBh1Tmg61GhQzkJCIUyeviWjkCTXh3GDPyKPg6ObRVHByONMW+q5JU7SQIjC/ua8GWYRQxEQ9HvqPQ3YNp+41V4wsT2ArEBmHecXhBsbEhNjtEb411KUQOCL6MKTDUMuJoHeTvxHkwTFWwgHh6f98FseWipIM2nti+V8ZqGMP3TYYFo3lXsAJWZb95koARRAzQWDR+Kl9QR5GyqI5CXNlYb1UAut+bt3t87wXUEvjfcP8Pi7ogO4+FJdksPtGRNNTpwLfB975qZgXE5ooepJLHwO7S+bF3R/nr2O7sTzr2aDNv/Np6bWt/UUO9kqsyXNNLBNzNpgm8GVKfOA2RyhY2SxWl72NbIOdINW8X/GaqbxtV9vA1bPZKiaSiSQICfLEqBmTo5maOLBGK966r9HADOilkSLc1vpv84ro0W7kZ4dbK6KmOEvkE8X6yllxWuSw8t7prf/yoOeq2SM7t9VcXmhfghtfXl11IM38NSNQIPaNIHmq7spbrI7IjCealBpbHdLpEoDLtxnKJgndiCpouiAa6nrFT6lLDDbF+WgKwTrAOs2qfrujfua9envn5kUjttoqVtdZVfz5/hqlZKHjrXyP378mGa8fKrPwVUhRIL8oxw/vKHCZ0FjO9RAQCR739eNXtZyun/z/cw0K+xB+QJYSIiVsbyr7hQeLrQ84a0s0b1sl1PumM5h8hKf2+3s+A1hFoQin8IuIPBUK0ltmF850BwqUDJtc+sQouDK8CJgDKBVRTIkk7qfcyQ8HPBo/cbxKuPu41rBtOffF8OU+jPM1LrRkvmXymW9OI5x9DZONjfcVNBkmjTub9EbZ3fK6uiG3rTsmGkeNjPT/Y6BusE9K8ddefuSC1PG/S6Z4k4wwZbbOLKjW8ZGTSIMzkMJJSONCrHxtQiqiz8LymtUu0MbWLjzjer+8yFEK7iPw XwQug8Tg FEoO3RyXvRUPLr2zF5v+Nq8ruHy+lvK7a8asTolypfSE9Q5DhF3v2YyAI95ghOXLq4jkHD3wQrleUfuml9gjcNAYgv0g2EVEnc7JvFI8H/2eW6+PRe8nbtUcHYau9bI8tkzpgRQ72IWMAaZF/uFjfeUSk48xhkwZspW00h5hKQp2amem0nrkIxMH2NwM6AV8jXm+MPk9/N/dajbSK3QW50mWwYtvIGl2Oin38ivT2XFExx1u11ELxL+wlvJ2cXWrLffzLlJHDs+knHYOVzx4dDOV2gkXQbdwaj8mpL/bJutZukfI6HZZL3zC32wbvgC4WsQWCTFXzOeHtocfsiAn4FEEwy5TvfrAbXKJgcv4Y8kNmnm01pTECdKvw/IW7kebKZXTVJsTcaYsN+7jykmCOmJruYMC3HD2Uh2k2/Z9Doi9XHVD/nVp3FeHwWbPmC66PHBM/4WhLUf9fml524EzdqGPZGjqCrXJLCPXa3i9dphZmjgpNg6mgGxTXJmH8GuFTZ2SxNz3VFErfOkMvnysvlFtWsHEvbvbDThZk1y8zCw7460XopCCC9tBd4crOHyEi7SqQ X-Bogosity: Ham, tests=bogofilter, spamicity=0.000000, version=1.2.4 Sender: owner-linux-mm@kvack.org Precedence: bulk X-Loop: owner-majordomo@kvack.org List-ID: List-Subscribe: List-Unsubscribe: mprotect() does mmu notifiers in PMD levels. It's there since 2014 of commit a5338093bfb4 ("mm: move mmu notifier call from change_protection to change_pmd_range"). At that time, the issue was that NUMA balancing can be applied on a huge range of VM memory, even if nothing was populated. The notification can be avoided in this case if no valid pmd detected, which includes either THP or a PTE pgtable page. Now to pave way for PUD handling, this isn't enough. We need to generate mmu notifications even on PUD entries properly. mprotect() is currently broken on PUD (e.g., one can easily trigger kernel error with dax 1G mappings already), this is the start to fix it. To fix that, this patch proposes to push such notifications to the PUD layers. There is risk on regressing the problem Rik wanted to resolve before, but I think it shouldn't really happen, and I still chose this solution because of a few reasons: 1) Consider a large VM that should definitely contain more than GBs of memory, it's highly likely that PUDs are also none. In this case there will have no regression. 2) KVM has evolved a lot over the years to get rid of rmap walks, which might be the major cause of the previous soft-lockup. At least TDP MMU already got rid of rmap as long as not nested (which should be the major use case, IIUC), then the TDP MMU pgtable walker will simply see empty VM pgtable (e.g. EPT on x86), the invalidation of a full empty region in most cases could be pretty fast now, comparing to 2014. 3) KVM has explicit code paths now to even give way for mmu notifiers just like this one, e.g. in commit d02c357e5bfa ("KVM: x86/mmu: Retry fault before acquiring mmu_lock if mapping is changing"). It'll also avoid contentions that may also contribute to a soft-lockup. 4) Stick with PMD layer simply don't work when PUD is there... We need one way or another to fix PUD mappings on mprotect(). Pushing it to PUD should be the safest approach as of now, e.g. there's yet no sign of huge P4D coming on any known archs. Cc: kvm@vger.kernel.org Cc: Sean Christopherson Cc: Paolo Bonzini Cc: David Rientjes Cc: Rik van Riel Signed-off-by: Peter Xu --- mm/mprotect.c | 32 ++++++++++++++++---------------- 1 file changed, 16 insertions(+), 16 deletions(-) diff --git a/mm/mprotect.c b/mm/mprotect.c index 37cf8d249405..d423080e6509 100644 --- a/mm/mprotect.c +++ b/mm/mprotect.c @@ -363,9 +363,6 @@ static inline long change_pmd_range(struct mmu_gather *tlb, unsigned long next; long pages = 0; unsigned long nr_huge_updates = 0; - struct mmu_notifier_range range; - - range.start = 0; pmd = pmd_offset(pud, addr); do { @@ -383,14 +380,6 @@ static inline long change_pmd_range(struct mmu_gather *tlb, if (pmd_none(*pmd)) goto next; - /* invoke the mmu notifier if the pmd is populated */ - if (!range.start) { - mmu_notifier_range_init(&range, - MMU_NOTIFY_PROTECTION_VMA, 0, - vma->vm_mm, addr, end); - mmu_notifier_invalidate_range_start(&range); - } - _pmd = pmdp_get_lockless(pmd); if (is_swap_pmd(_pmd) || pmd_trans_huge(_pmd) || pmd_devmap(_pmd)) { if ((next - addr != HPAGE_PMD_SIZE) || @@ -431,9 +420,6 @@ static inline long change_pmd_range(struct mmu_gather *tlb, cond_resched(); } while (pmd++, addr = next, addr != end); - if (range.start) - mmu_notifier_invalidate_range_end(&range); - if (nr_huge_updates) count_vm_numa_events(NUMA_HUGE_PTE_UPDATES, nr_huge_updates); return pages; @@ -443,22 +429,36 @@ static inline long change_pud_range(struct mmu_gather *tlb, struct vm_area_struct *vma, p4d_t *p4d, unsigned long addr, unsigned long end, pgprot_t newprot, unsigned long cp_flags) { + struct mmu_notifier_range range; pud_t *pud; unsigned long next; long pages = 0, ret; + range.start = 0; + pud = pud_offset(p4d, addr); do { next = pud_addr_end(addr, end); ret = change_prepare(vma, pud, pmd, addr, cp_flags); - if (ret) - return ret; + if (ret) { + pages = ret; + break; + } if (pud_none_or_clear_bad(pud)) continue; + if (!range.start) { + mmu_notifier_range_init(&range, + MMU_NOTIFY_PROTECTION_VMA, 0, + vma->vm_mm, addr, end); + mmu_notifier_invalidate_range_start(&range); + } pages += change_pmd_range(tlb, vma, pud, addr, next, newprot, cp_flags); } while (pud++, addr = next, addr != end); + if (range.start) + mmu_notifier_invalidate_range_end(&range); + return pages; } From patchwork Mon Aug 12 18:12:21 2024 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Peter Xu X-Patchwork-Id: 13760908 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from kanga.kvack.org (kanga.kvack.org [205.233.56.17]) by smtp.lore.kernel.org (Postfix) with ESMTP id 7E5C6C531DC for ; Mon, 12 Aug 2024 18:12:43 +0000 (UTC) Received: by kanga.kvack.org (Postfix) id 57F8F6B009E; Mon, 12 Aug 2024 14:12:42 -0400 (EDT) Received: by kanga.kvack.org (Postfix, from userid 40) id 530DA6B009F; Mon, 12 Aug 2024 14:12:42 -0400 (EDT) X-Delivered-To: int-list-linux-mm@kvack.org Received: by kanga.kvack.org (Postfix, from userid 63042) id 2E6A46B00A0; Mon, 12 Aug 2024 14:12:42 -0400 (EDT) X-Delivered-To: linux-mm@kvack.org Received: from relay.hostedemail.com (smtprelay0010.hostedemail.com [216.40.44.10]) by kanga.kvack.org (Postfix) with ESMTP id 0F44C6B009E for ; Mon, 12 Aug 2024 14:12:42 -0400 (EDT) Received: from smtpin19.hostedemail.com (a10.router.float.18 [10.200.18.1]) by unirelay07.hostedemail.com (Postfix) with ESMTP id 838E716035A for ; Mon, 12 Aug 2024 18:12:41 +0000 (UTC) X-FDA: 82444388922.19.66794A7 Received: from us-smtp-delivery-124.mimecast.com (us-smtp-delivery-124.mimecast.com [170.10.133.124]) by imf13.hostedemail.com (Postfix) with ESMTP id EC9242001F for ; Mon, 12 Aug 2024 18:12:37 +0000 (UTC) Authentication-Results: imf13.hostedemail.com; dkim=pass header.d=redhat.com header.s=mimecast20190719 header.b=hqTqk2Ug; dmarc=pass (policy=none) header.from=redhat.com; spf=pass (imf13.hostedemail.com: domain of peterx@redhat.com designates 170.10.133.124 as permitted sender) smtp.mailfrom=peterx@redhat.com ARC-Seal: i=1; s=arc-20220608; d=hostedemail.com; t=1723486306; a=rsa-sha256; cv=none; b=MqDUOrEnwOX0DOCiy+M+uANqfua9JRkQV/AYUx+3KHeIF2yHTaCuI2GMHng8q2c3BUGpF7 qhfRH37UzzeluVS8g+r8iAMxtoY+BmebO/67S2Y2inWzIm0SXLWDGQk/OkUzTJJfNzgH/w Ew9ORRnSR2uROCPwwTzPzlGKy2plhGg= ARC-Authentication-Results: i=1; imf13.hostedemail.com; dkim=pass header.d=redhat.com header.s=mimecast20190719 header.b=hqTqk2Ug; dmarc=pass (policy=none) header.from=redhat.com; spf=pass (imf13.hostedemail.com: domain of peterx@redhat.com designates 170.10.133.124 as permitted sender) smtp.mailfrom=peterx@redhat.com ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=hostedemail.com; s=arc-20220608; t=1723486306; h=from:from:sender:reply-to:subject:subject:date:date: message-id:message-id:to:to:cc:cc:mime-version:mime-version: content-type:content-type: content-transfer-encoding:content-transfer-encoding: in-reply-to:in-reply-to:references:references:dkim-signature; bh=wAu0M0+1UWqmiItjD8rL7YnlMQXKv8wW1V5JPENrD98=; b=KYVLRUd4AMHVoGLv3RnNT5SlnuQcyINXavkSrTP4MRa9K4j7E+7BGQF5fBNQ5QAj2PsFQf VDbR3PKCQLz7IoiDJ79A2c4tDdpNWXlnPS04P+q6eyqRjPr4koHyhL0ta9VZMBN2SaPq2I gzIFxMjb7LccJJhMytKmF89PCwfj+H4= DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=redhat.com; s=mimecast20190719; t=1723486357; h=from:from:reply-to:subject:subject:date:date:message-id:message-id: to:to:cc:cc:mime-version:mime-version:content-type:content-type: content-transfer-encoding:content-transfer-encoding: in-reply-to:in-reply-to:references:references; bh=wAu0M0+1UWqmiItjD8rL7YnlMQXKv8wW1V5JPENrD98=; b=hqTqk2UgY/UVvg9dsRH6hq38kv1/HPOoepA1N9p8s3VIpUOE4hdPDxj9O10Ok+4XJAj4Wo 03HvrK4wPpN6IyrOxQt2MdVS3OuD99NXBxrWLa3zP/5VAeFjtPIjurcMjdu4/J7+Im7wQW kxdKbjPMvaub9uUtXigMUrzFZRfELKo= Received: from mail-qk1-f199.google.com (mail-qk1-f199.google.com [209.85.222.199]) by relay.mimecast.com with ESMTP with STARTTLS (version=TLSv1.3, cipher=TLS_AES_256_GCM_SHA384) id us-mta-682-ZhZR4BwvOUqvLag7JIzsVQ-1; Mon, 12 Aug 2024 14:12:36 -0400 X-MC-Unique: ZhZR4BwvOUqvLag7JIzsVQ-1 Received: by mail-qk1-f199.google.com with SMTP id af79cd13be357-7a1d3c02fcfso18662385a.0 for ; Mon, 12 Aug 2024 11:12:36 -0700 (PDT) X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20230601; t=1723486356; x=1724091156; h=content-transfer-encoding:mime-version:references:in-reply-to :message-id:date:subject:cc:to:from:x-gm-message-state:from:to:cc :subject:date:message-id:reply-to; bh=wAu0M0+1UWqmiItjD8rL7YnlMQXKv8wW1V5JPENrD98=; b=U3SUfTQT4I4VExgESo0voAwzSTwr5FE9GSYMBsE9Zp/ntPM+6CppLkVo0weIpcdxDv rbsMM2TvtBiG39USHdc4xpekQtUdNsA1MwEpMriIwPTOK1UxpZsfl5xwnH+efN/WOQiX YEiM9avnXyYEU7Yar9rdo6YS8fz188SQlfJoJdNwgWVLrNcGhfb+YPERvs5PkhkdR5Mx opUVVHbmfqrFU44mKkamvDZMdv3O1mZeEvWkV06F2OsoCq+Dx530pAf8Gldye6nkj61m zLgVtTCnItSoUUXmcwF8IYyWzBGOomSMF9Z7VA10lg4lseWwuGg+YbcGiCJuqoYLdNiI zszg== X-Forwarded-Encrypted: i=1; AJvYcCVJhzYj7dLax2LpQ6Hj39N2oUw9HU31cGijuC+xSbcq1BqWbWcGJ+iXsH+7bbnG6vZfvoaZWzeD3Ldls5a6Bi8sL6Y= X-Gm-Message-State: AOJu0Yz3E8O4DesW75czdSc12DxLUU50VmBa0B1Y+6yyNX2KsFik8TyO Yxo0PRKFpkZfuprELF5HSM+8tCrKUfggRz98BhkI7cQQcKJsZVDcFw4ia30Qfck+ivB11NjABTf 1Nb/liMFhtevPuTEBHBelhqxRkDXT4CWA5TTZGIMTCX95t++F X-Received: by 2002:a05:620a:4005:b0:79f:84f:80b1 with SMTP id af79cd13be357-7a4e15d46a6mr74880985a.7.1723486355729; Mon, 12 Aug 2024 11:12:35 -0700 (PDT) X-Google-Smtp-Source: AGHT+IFpNsU7cY2l6XFvfi/NYSnYqbYktrA9n3rcHeLoBtv5kBl17wu9V8UOiRcR1CmKk37Q2l0U8w== X-Received: by 2002:a05:620a:4005:b0:79f:84f:80b1 with SMTP id af79cd13be357-7a4e15d46a6mr74878085a.7.1723486355340; Mon, 12 Aug 2024 11:12:35 -0700 (PDT) Received: from x1n.redhat.com (pool-99-254-121-117.cpe.net.cable.rogers.com. [99.254.121.117]) by smtp.gmail.com with ESMTPSA id af79cd13be357-7a4c7dee013sm268663985a.84.2024.08.12.11.12.33 (version=TLS1_3 cipher=TLS_AES_256_GCM_SHA384 bits=256/256); Mon, 12 Aug 2024 11:12:34 -0700 (PDT) From: Peter Xu To: linux-kernel@vger.kernel.org, linux-mm@kvack.org Cc: "Kirill A . Shutemov" , Nicholas Piggin , David Hildenbrand , Matthew Wilcox , Andrew Morton , James Houghton , Huang Ying , "Aneesh Kumar K . V" , peterx@redhat.com, Vlastimil Babka , Rick P Edgecombe , Hugh Dickins , Borislav Petkov , Christophe Leroy , Michael Ellerman , Rik van Riel , Dan Williams , Mel Gorman , x86@kernel.org, Ingo Molnar , linuxppc-dev@lists.ozlabs.org, Dave Hansen , Dave Jiang , Oscar Salvador , Thomas Gleixner Subject: [PATCH v5 3/7] mm/powerpc: Add missing pud helpers Date: Mon, 12 Aug 2024 14:12:21 -0400 Message-ID: <20240812181225.1360970-4-peterx@redhat.com> X-Mailer: git-send-email 2.45.0 In-Reply-To: <20240812181225.1360970-1-peterx@redhat.com> References: <20240812181225.1360970-1-peterx@redhat.com> MIME-Version: 1.0 X-Mimecast-Spam-Score: 0 X-Mimecast-Originator: redhat.com X-Rspamd-Queue-Id: EC9242001F X-Rspam-User: X-Rspamd-Server: rspam05 X-Stat-Signature: 1c9dytq6urigfry8j1phaw977hjdq1kj X-HE-Tag: 1723486357-821458 X-HE-Meta: U2FsdGVkX19mH6C7TeXI6ZEKCJI6oWBo0rKxDqijk8d2SNvURf2+fII4oOeY2/tzb+TKfqsJOIjoDc6JCROOnZ6iKUdsP37Iar32m0ttVNtzLM5ecY0ky2GR/bQDp+IaTkEqNXUFJ1z8l+J7ZLASvGTYJNlCeo1icMDye1MYo/QWwAnS/c4IHfEC8utueziZ6FNI4SsF+eHyEG3vI6YVEjxnp5j72UNtMVyfGvyfJZbsZEXwbuvygRfMkcCFAsoyj9vQQAfFmzeGlbGiVl9/dxiy7NMJQq3Jy8vnhOR1OmsxNPhvUq6VqAUAg8g1pISUgDDVsaGKa7aquC8qR1rBGPsgH/sB7yHxBmguX1uYQtVJqVbFBI74POq1BLKUOVolMIYMegDBby3VMB/Whhy2GPT76WF3F732CDW1axhBYjG7zaLDkFQc/3DUIZWgZ8Om87+82jT1z317k8slHZlJ1kUcTR92lJVpil2ZpwmL7KCRB9lMjCBrY/3/hJqgXAHCL/Vi8i7Y0bNDdlxytV0V1amogWQLWId3+Fc/QoRPxGCeOcbVD1D4XmjdKuQQaIU0B90csWzAU46qk0b8Fyn0nRz/xH9IetybeGQbAY/AzxW5rGls+XAU8tetxxjx9iqspUAE7tCR+Y8tHD8nNVBbs+yqjzR3WnlXKoM1p0uGggtbKoMux41IvHeJzSjtFo2mc+Dpn4ce0BClHgTNtgUKQRzVr28mgWS/8Usfm/mv0U7Z/BygXDVnhDDIWdWgCmMp5N9BONCm4Ql6WWY1lOwnYLgUifXEdxC/7m3tk2Bwo3Tc6hDO2hOcR/NjNaWxufwLMI8WmyuL6FWq76brrrc5hqiINa8QEwyo6/lCPgYn3dUsjYGwCpYcfZebtdtvGh9aEaZAWuXeyqe/4AMoIdJ5THIC/NHrmO0lROxkAwpm4b0xIBlkEU69e0ijWrM8HZOuUWriMID67q81VD833n3 Iqh4dKVQ t8FoDpDEr3JkCDYg5IIowpg9QgXYHuK2WSSbBQj0q6cQlb7FK8zCzb2kEH/c/dM0sskIxp0aE3X3bFGD5Ogm5gBKn3EUndtzO5ZGfTzBkuGoSwJQBh4tXOmbocDq9hFyHeOtJVRZ0QEWt6tXElS3M9BZ1QUMz/uKp/iaHStY0pb31D+CSVjS9bKL2QSLAEvAArc2bLEV5vrQDG+EtTlNSsMldO6Q5dIBM1kZnuMmdZWdWFSxkXuQ6+/OV79mDENwKmxoZHBCLn2XfunTnvNVQNTUZ/Ogq+qnYk581LDHCoV3fv+QXmF6GAE6/zOhd2vLwPnT74R+JM4UFbGbAUqDpOrORmGvzV49p4GqP+xBNPXBXSUOCQR8rwIq2X9hHTgtduQqbZauYWa50+S7JvJgy6VUS2cYOq5M78GSn7IbaWaw7f2C3v/zqWuenwdul5UvENcQgEPYkq8OsDwinPc2FfUYRAcZQcRZhpN61G/rW6kYW6io/IZwPS+tPKbstDZotUPBHifzASGwg31TKZzU1TnUrMpOsHsNxZjhTvieIE/2CKLl+cjrTZMEfEFjuVPDKyP8F/pbJ3ERc86jrQAN/0y0moQ== X-Bogosity: Ham, tests=bogofilter, spamicity=0.000000, version=1.2.4 Sender: owner-linux-mm@kvack.org Precedence: bulk X-Loop: owner-majordomo@kvack.org List-ID: List-Subscribe: List-Unsubscribe: Some new helpers will be needed for pud entry updates soon. Introduce these helpers by referencing the pmd ones. Namely: - pudp_invalidate(): this helper invalidates a huge pud before a split happens, so that the invalidated pud entry will make sure no race will happen (either with software, like a concurrent zap, or hardware, like a/d bit lost). - pud_modify(): this helper applies a new pgprot to an existing huge pud mapping. For more information on why we need these two helpers, please refer to the corresponding pmd helpers in the mprotect() code path. Cc: Michael Ellerman Cc: Nicholas Piggin Cc: Christophe Leroy Cc: linuxppc-dev@lists.ozlabs.org Cc: Aneesh Kumar K.V Signed-off-by: Peter Xu --- arch/powerpc/include/asm/book3s/64/pgtable.h | 3 +++ arch/powerpc/mm/book3s64/pgtable.c | 20 ++++++++++++++++++++ 2 files changed, 23 insertions(+) diff --git a/arch/powerpc/include/asm/book3s/64/pgtable.h b/arch/powerpc/include/asm/book3s/64/pgtable.h index 519b1743a0f4..5da92ba68a45 100644 --- a/arch/powerpc/include/asm/book3s/64/pgtable.h +++ b/arch/powerpc/include/asm/book3s/64/pgtable.h @@ -1124,6 +1124,7 @@ extern pmd_t pfn_pmd(unsigned long pfn, pgprot_t pgprot); extern pud_t pfn_pud(unsigned long pfn, pgprot_t pgprot); extern pmd_t mk_pmd(struct page *page, pgprot_t pgprot); extern pmd_t pmd_modify(pmd_t pmd, pgprot_t newprot); +extern pud_t pud_modify(pud_t pud, pgprot_t newprot); extern void set_pmd_at(struct mm_struct *mm, unsigned long addr, pmd_t *pmdp, pmd_t pmd); extern void set_pud_at(struct mm_struct *mm, unsigned long addr, @@ -1384,6 +1385,8 @@ static inline pgtable_t pgtable_trans_huge_withdraw(struct mm_struct *mm, #define __HAVE_ARCH_PMDP_INVALIDATE extern pmd_t pmdp_invalidate(struct vm_area_struct *vma, unsigned long address, pmd_t *pmdp); +extern pud_t pudp_invalidate(struct vm_area_struct *vma, unsigned long address, + pud_t *pudp); #define pmd_move_must_withdraw pmd_move_must_withdraw struct spinlock; diff --git a/arch/powerpc/mm/book3s64/pgtable.c b/arch/powerpc/mm/book3s64/pgtable.c index f4d8d3c40e5c..5a4a75369043 100644 --- a/arch/powerpc/mm/book3s64/pgtable.c +++ b/arch/powerpc/mm/book3s64/pgtable.c @@ -176,6 +176,17 @@ pmd_t pmdp_invalidate(struct vm_area_struct *vma, unsigned long address, return __pmd(old_pmd); } +pud_t pudp_invalidate(struct vm_area_struct *vma, unsigned long address, + pud_t *pudp) +{ + unsigned long old_pud; + + VM_WARN_ON_ONCE(!pud_present(*pudp)); + old_pud = pud_hugepage_update(vma->vm_mm, address, pudp, _PAGE_PRESENT, _PAGE_INVALID); + flush_pud_tlb_range(vma, address, address + HPAGE_PUD_SIZE); + return __pud(old_pud); +} + pmd_t pmdp_huge_get_and_clear_full(struct vm_area_struct *vma, unsigned long addr, pmd_t *pmdp, int full) { @@ -259,6 +270,15 @@ pmd_t pmd_modify(pmd_t pmd, pgprot_t newprot) pmdv &= _HPAGE_CHG_MASK; return pmd_set_protbits(__pmd(pmdv), newprot); } + +pud_t pud_modify(pud_t pud, pgprot_t newprot) +{ + unsigned long pudv; + + pudv = pud_val(pud); + pudv &= _HPAGE_CHG_MASK; + return pud_set_protbits(__pud(pudv), newprot); +} #endif /* CONFIG_TRANSPARENT_HUGEPAGE */ /* For use by kexec, called with MMU off */ From patchwork Mon Aug 12 18:12:22 2024 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Peter Xu X-Patchwork-Id: 13760910 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from kanga.kvack.org (kanga.kvack.org [205.233.56.17]) by smtp.lore.kernel.org (Postfix) with ESMTP id 63CFBC3DA7F for ; Mon, 12 Aug 2024 18:12:46 +0000 (UTC) Received: by kanga.kvack.org (Postfix) id C84AD6B009F; Mon, 12 Aug 2024 14:12:44 -0400 (EDT) Received: by kanga.kvack.org (Postfix, from userid 40) id C0F1B6B00A0; Mon, 12 Aug 2024 14:12:44 -0400 (EDT) X-Delivered-To: int-list-linux-mm@kvack.org Received: by kanga.kvack.org (Postfix, from userid 63042) id 9EAEE6B00A1; Mon, 12 Aug 2024 14:12:44 -0400 (EDT) X-Delivered-To: linux-mm@kvack.org Received: from relay.hostedemail.com (smtprelay0014.hostedemail.com [216.40.44.14]) by kanga.kvack.org (Postfix) with ESMTP id 7C9096B009F for ; Mon, 12 Aug 2024 14:12:44 -0400 (EDT) Received: from smtpin13.hostedemail.com (a10.router.float.18 [10.200.18.1]) by unirelay06.hostedemail.com (Postfix) with ESMTP id 268F6A5B5F for ; Mon, 12 Aug 2024 18:12:44 +0000 (UTC) X-FDA: 82444389048.13.26F8F9F Received: from us-smtp-delivery-124.mimecast.com (us-smtp-delivery-124.mimecast.com [170.10.133.124]) by imf28.hostedemail.com (Postfix) with ESMTP id 1335BC0014 for ; Mon, 12 Aug 2024 18:12:41 +0000 (UTC) Authentication-Results: imf28.hostedemail.com; dkim=pass header.d=redhat.com header.s=mimecast20190719 header.b=Jnn7w6n0; spf=pass (imf28.hostedemail.com: domain of peterx@redhat.com designates 170.10.133.124 as permitted sender) smtp.mailfrom=peterx@redhat.com; dmarc=pass (policy=none) header.from=redhat.com ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=hostedemail.com; s=arc-20220608; t=1723486307; h=from:from:sender:reply-to:subject:subject:date:date: message-id:message-id:to:to:cc:cc:mime-version:mime-version: content-type:content-type: content-transfer-encoding:content-transfer-encoding: in-reply-to:in-reply-to:references:references:dkim-signature; bh=51AsM4wPs8z1MhF0DnEEQJIHR/NiBzvmhfDXZZK2UA8=; b=VEEv2ygapol25q94EXr6Sqon9vBfYcl8IsZNSuVpqeMof8TlBq4jf6mPz8juMEgwkcYZ67 WAaLgFEf5V/H/KR1u8At0r10ySOVm2znoz20VhPJQvbwu0yh0gYs8/8+Slw55y4wZhMxhn M+Uj1Y626xMhnnpTLNGYOqgO9Yb4agU= ARC-Authentication-Results: i=1; imf28.hostedemail.com; dkim=pass header.d=redhat.com header.s=mimecast20190719 header.b=Jnn7w6n0; spf=pass (imf28.hostedemail.com: domain of peterx@redhat.com designates 170.10.133.124 as permitted sender) smtp.mailfrom=peterx@redhat.com; dmarc=pass (policy=none) header.from=redhat.com ARC-Seal: i=1; s=arc-20220608; d=hostedemail.com; t=1723486307; a=rsa-sha256; cv=none; b=y3h+eDyv3QEfWqNk5mRBJBC/Lsm/ABQv+5fJbXLbdobUzRD2cUo21mBJVkiKnAPF7NHFEu Xyhmx1uBi75/aOYSRV7rXOH7weZmc6nYg2Np60vuPTMdV3TA2H37lWZwcyufxrSNSFt82a qHHWxc+TdvO9VJumg976pwPAjjISNK0= DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=redhat.com; s=mimecast20190719; t=1723486361; h=from:from:reply-to:subject:subject:date:date:message-id:message-id: to:to:cc:cc:mime-version:mime-version:content-type:content-type: content-transfer-encoding:content-transfer-encoding: in-reply-to:in-reply-to:references:references; bh=51AsM4wPs8z1MhF0DnEEQJIHR/NiBzvmhfDXZZK2UA8=; b=Jnn7w6n0gwtjJfI2DZ8EB0wjG/1sqPNoKtV3FWMJBvINwdiupXN9WDMrfk37pxCBGCF/wf hWQJILdqi8jzaQxDY2lPygYHsPbfh4OPj9ob3aqYdmK3doRD3eNtzPmvI905lSkHVFDLHL XbeVrMvl7Qx5N+/qtl2XQt/65bs9x7A= Received: from mail-vs1-f71.google.com (mail-vs1-f71.google.com [209.85.217.71]) by relay.mimecast.com with ESMTP with STARTTLS (version=TLSv1.3, cipher=TLS_AES_256_GCM_SHA384) id us-mta-90-gnnl6X5nPmKlrU9ktxCHZg-1; Mon, 12 Aug 2024 14:12:39 -0400 X-MC-Unique: gnnl6X5nPmKlrU9ktxCHZg-1 Received: by mail-vs1-f71.google.com with SMTP id ada2fe7eead31-492a3346e4bso216482137.3 for ; Mon, 12 Aug 2024 11:12:39 -0700 (PDT) X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20230601; t=1723486358; x=1724091158; h=content-transfer-encoding:mime-version:references:in-reply-to :message-id:date:subject:cc:to:from:x-gm-message-state:from:to:cc :subject:date:message-id:reply-to; bh=51AsM4wPs8z1MhF0DnEEQJIHR/NiBzvmhfDXZZK2UA8=; b=LzkaztRiOPWFSA+rXCvqyK9pX4v6NMoxEKBwqpGTpc8uBA+u9aSqZKTdIuy3qFEHh1 ijl0YvVd3FjwbnJ5i4p/fjEuyqqA8yzC8DRsUoU8wWiWWrZ9+9o8TbWaf8iS+0IggaYw G2G2O6OCfG7gVL0FQOSyhJ5RxyeH3qyNSmJ7GCj0Lq0j4kLlzobcUHA7MDlgfrqBW9lf 7RrqflfKJzfc7OZXFSZcl+A9wlIoPsJVGl/ev04t1S/Ibz2+F4EAOkPYUvI2jCK4ARF+ Wjb1NQArGaP7s6WFzU/d0Dwlt+gsgJqBvGxbBw+xTH3ZC8eJtYvtSvuvjPr7v9DSj2OQ SuSQ== X-Forwarded-Encrypted: i=1; AJvYcCWnJnz7hl2+PcnfbiNCj8zjcSGZiTPhHZWxpEzJ+hPh8ooD7uOvev61FyNcrz3ZqCCoipIA5rqZeQ==@kvack.org X-Gm-Message-State: AOJu0YyiZm0zvBh+kE95Nykhu5Mi/8rCfM/uTiBmgzAwEOXDrKgPp0Ti Z0ztac5IrVhtCFUYXTWHTsUmIuJ6K4zWHMKIBtSPENuT0G2w/oWf2jQ55PXMntmh3QcqA/MSsIy G7Uc4jg4Qom3VxYrdQIcYDJp8LGYu9B3Rfk0qlJtLs2rxTzgmx0xsYPCg X-Received: by 2002:a05:6102:3ecb:b0:48f:1db0:e268 with SMTP id ada2fe7eead31-49743b3bcadmr808464137.3.1723486358024; Mon, 12 Aug 2024 11:12:38 -0700 (PDT) X-Google-Smtp-Source: AGHT+IHqoB9/0mESDqYSEBNTj9AtHyzcMfWYv34FSY6okF36kkcu051ygwC1i1HRAV7+0kWzcEuDbQ== X-Received: by 2002:a05:6102:3ecb:b0:48f:1db0:e268 with SMTP id ada2fe7eead31-49743b3bcadmr808432137.3.1723486357628; Mon, 12 Aug 2024 11:12:37 -0700 (PDT) Received: from x1n.redhat.com (pool-99-254-121-117.cpe.net.cable.rogers.com. [99.254.121.117]) by smtp.gmail.com with ESMTPSA id af79cd13be357-7a4c7dee013sm268663985a.84.2024.08.12.11.12.35 (version=TLS1_3 cipher=TLS_AES_256_GCM_SHA384 bits=256/256); Mon, 12 Aug 2024 11:12:37 -0700 (PDT) From: Peter Xu To: linux-kernel@vger.kernel.org, linux-mm@kvack.org Cc: "Kirill A . Shutemov" , Nicholas Piggin , David Hildenbrand , Matthew Wilcox , Andrew Morton , James Houghton , Huang Ying , "Aneesh Kumar K . V" , peterx@redhat.com, Vlastimil Babka , Rick P Edgecombe , Hugh Dickins , Borislav Petkov , Christophe Leroy , Michael Ellerman , Rik van Riel , Dan Williams , Mel Gorman , x86@kernel.org, Ingo Molnar , linuxppc-dev@lists.ozlabs.org, Dave Hansen , Dave Jiang , Oscar Salvador , Thomas Gleixner Subject: [PATCH v5 4/7] mm/x86: Make pud_leaf() only care about PSE bit Date: Mon, 12 Aug 2024 14:12:22 -0400 Message-ID: <20240812181225.1360970-5-peterx@redhat.com> X-Mailer: git-send-email 2.45.0 In-Reply-To: <20240812181225.1360970-1-peterx@redhat.com> References: <20240812181225.1360970-1-peterx@redhat.com> MIME-Version: 1.0 X-Mimecast-Spam-Score: 0 X-Mimecast-Originator: redhat.com X-Stat-Signature: a8cma61td7yandrhwo3imqfgnfsqczqy X-Rspam-User: X-Rspamd-Queue-Id: 1335BC0014 X-Rspamd-Server: rspam02 X-HE-Tag: 1723486361-142697 X-HE-Meta: U2FsdGVkX18QsxC52KJn2jGmoiFNXq/4qSw+cW77LUV6b3HhPuT1UzmA2P/2xh1Zsiaegi/3D1wxS+ENhJnVBu7yGzInM1mGUD0otHy+DbJKMvVLYthMhwrGr+5mCt9gjSjRUh5ca2Cr7KO6+NygTfhOw2dWjw1UBeu6g2x9d1zhRoMYuWbltzx6tNaX10XtU7flPNSZ9qITmce9SyMO8KBFZgQbKa3qX/qQqcCuBQ7NwBpGfAQVrdkntKjRTtuI0KheEpCpcO8y7qf4PjzbbTyI0/chnQKGt1dk1wWYdJKsE1UUjdzJW2GsYQNIN62QV62MDwDE4uhyDpjBj8PegX2mfR6D+roW/aVAyK2OSs/CQZ2Gpvi9y24DjMBDMHo644KWDm/BYy09N0ffYzA8wMG4dYQHDtXzsrzdq7VJrErjDLtXy+TWwC1F/ZFMp9a3zOl+zE7rmCq/VkctL23dKZ2fLH4ssl0ZaWolXQOn7mImF89qyxLthizUxNT88BYkQucmsZlKh3ONNRIuc12sfDH1tgUH+JpV7eV+32R/rC8nCy6qSyuC7h5m+DQ23zs4BVeEjQKmMLSciLjs/aKt0YA7YlIVdkiu792+x/4kjrRLnFcATSreMSI6d6wozZg3vxL6kmeXHW+BCHqPTJHuaqF7JtioXvY/nwJzixq1LM/4OUgSTbjvAv4JKTyCjv2HPHH9vEqNnA2zIq2dOY2tVMyasgzxTrGkHZgWOy/55Wv1Qbs5C2HOZEsTZHjrQF3YgxVJiyl3a1VHxGKj7g06cBRz+KAq6NjJAXi9fM2Dh9abCHei/W1NYyXcYd/w3c71LE34trT7Qk0Rt5nWIWLCKNEacHstJ+eUmgG1o8MavfZeq0NnvAzSDHp/3zoa/8flHWvp3mHghtjNgGNRLmnQ/FY7eLITrM2vaAffm33hlp36N/OLRrLCKhIh28ukk7EdthlBSqV91hpGy9YIiOO b5NS+KfT ctYwciUkm7aiSF8ZvZxvhog143IcRsWG+Z8xM9o3StATBhPW8Jv/0gpD2pxjARiw7e019ke4g8W8u42Ua+iugBq0lGxWXdJjWpSlUd6Zv+R7ApZfr9p+LaYathh0+e8oy5v20UXflpPYQMMWxRQnqmIWqPH+DvhDNWO2gSIc/BXJ0F7XTAkGqwzCP5I3DdnbAEN5Q6CchxmkfsP+3fCIzgna6hBngKouaMYI20xXYOr4OpRsETa0U/QyyS11EpvWcX55JGFAXUsEJKO8rBe9qQ7iKYQLO+md/uhpcFvwfcbbzT+9vMS1ZUICMNp9K5Ieu6dEPVIc14krBz+VqOWSLDQ0oUvsg8JrigEGQs8U1kU6Z8//e+QcGC1rIa9BEx+yWuyRXjWl7q4mQTWXHAl4t5i7pFXGiT4MXOhxHFxs0K8BkVwxRAj5UejectVP4qFvf6i4/Ymsc7Xh+A4HHZaFnu05qR0CWYUUEyutC2Sa2DqeUjUfmgoyR0+QSBcBCaqLNWzo5X8dTtnp4wgYGBf6t4LZeX36nMyi9b9w+048Iiz3IK9RbovRyP2ozOgR8EUEqIxWOsZhyE51VYGT0aguy3VTdpw== X-Bogosity: Ham, tests=bogofilter, spamicity=0.000000, version=1.2.4 Sender: owner-linux-mm@kvack.org Precedence: bulk X-Loop: owner-majordomo@kvack.org List-ID: List-Subscribe: List-Unsubscribe: When working on mprotect() on 1G dax entries, I hit an zap bad pud error when zapping a huge pud that is with PROT_NONE permission. Here the problem is x86's pud_leaf() requires both PRESENT and PSE bits set to report a pud entry as a leaf, but that doesn't look right, as it's not following the pXd_leaf() definition that we stick with so far, where PROT_NONE entries should be reported as leaves. To fix it, change x86's pud_leaf() implementation to only check against PSE bit to report a leaf, irrelevant of whether PRESENT bit is set. Cc: Thomas Gleixner Cc: Ingo Molnar Cc: Borislav Petkov Cc: Dave Hansen Cc: x86@kernel.org Acked-by: Dave Hansen Reviewed-by: David Hildenbrand Signed-off-by: Peter Xu --- arch/x86/include/asm/pgtable.h | 3 +-- 1 file changed, 1 insertion(+), 2 deletions(-) diff --git a/arch/x86/include/asm/pgtable.h b/arch/x86/include/asm/pgtable.h index e39311a89bf4..a2a3bd4c1bda 100644 --- a/arch/x86/include/asm/pgtable.h +++ b/arch/x86/include/asm/pgtable.h @@ -1078,8 +1078,7 @@ static inline pmd_t *pud_pgtable(pud_t pud) #define pud_leaf pud_leaf static inline bool pud_leaf(pud_t pud) { - return (pud_val(pud) & (_PAGE_PSE | _PAGE_PRESENT)) == - (_PAGE_PSE | _PAGE_PRESENT); + return pud_val(pud) & _PAGE_PSE; } static inline int pud_bad(pud_t pud) From patchwork Mon Aug 12 18:12:23 2024 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Peter Xu X-Patchwork-Id: 13760911 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from kanga.kvack.org (kanga.kvack.org [205.233.56.17]) by smtp.lore.kernel.org (Postfix) with ESMTP id 51106C3DA7F for ; Mon, 12 Aug 2024 18:12:49 +0000 (UTC) Received: by kanga.kvack.org (Postfix) id 669046B00A1; Mon, 12 Aug 2024 14:12:47 -0400 (EDT) Received: by kanga.kvack.org (Postfix, from userid 40) id 61C8B6B00A2; Mon, 12 Aug 2024 14:12:47 -0400 (EDT) X-Delivered-To: int-list-linux-mm@kvack.org Received: by kanga.kvack.org (Postfix, from userid 63042) id 492016B00A3; Mon, 12 Aug 2024 14:12:47 -0400 (EDT) X-Delivered-To: linux-mm@kvack.org Received: from relay.hostedemail.com (smtprelay0012.hostedemail.com [216.40.44.12]) by kanga.kvack.org (Postfix) with ESMTP id 254836B00A1 for ; Mon, 12 Aug 2024 14:12:47 -0400 (EDT) Received: from smtpin23.hostedemail.com (a10.router.float.18 [10.200.18.1]) by unirelay03.hostedemail.com (Postfix) with ESMTP id D573CA0335 for ; Mon, 12 Aug 2024 18:12:46 +0000 (UTC) X-FDA: 82444389132.23.3FB2DBB Received: from us-smtp-delivery-124.mimecast.com (us-smtp-delivery-124.mimecast.com [170.10.133.124]) by imf07.hostedemail.com (Postfix) with ESMTP id ACA2C4001B for ; Mon, 12 Aug 2024 18:12:44 +0000 (UTC) Authentication-Results: imf07.hostedemail.com; dkim=pass header.d=redhat.com header.s=mimecast20190719 header.b=U1P3j7EN; spf=pass (imf07.hostedemail.com: domain of peterx@redhat.com designates 170.10.133.124 as permitted sender) smtp.mailfrom=peterx@redhat.com; dmarc=pass (policy=none) header.from=redhat.com ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=hostedemail.com; s=arc-20220608; t=1723486353; h=from:from:sender:reply-to:subject:subject:date:date: message-id:message-id:to:to:cc:cc:mime-version:mime-version: content-type:content-type: content-transfer-encoding:content-transfer-encoding: in-reply-to:in-reply-to:references:references:dkim-signature; bh=vL+nA7AT6HH2fptU3fDtubaziCDql6jx+j3l3b3Xtg8=; b=VWfWr7+ItaxIf6oSTW/Nm0+qV6qbYcYrIIhoDnxOxuCwZSeJ+LkH5m4YmozAVSo+rRYrGc klANoGmn1ns9Kc81EKwzh62gr1+1ibH8LqJ9c9zioahJMulQyxjbmCxtUy8jXTWVgrNt8n RuEJBvVdSwPyVpcUg/LC6u148qP84Mc= ARC-Authentication-Results: i=1; imf07.hostedemail.com; dkim=pass header.d=redhat.com header.s=mimecast20190719 header.b=U1P3j7EN; spf=pass (imf07.hostedemail.com: domain of peterx@redhat.com designates 170.10.133.124 as permitted sender) smtp.mailfrom=peterx@redhat.com; dmarc=pass (policy=none) header.from=redhat.com ARC-Seal: i=1; s=arc-20220608; d=hostedemail.com; t=1723486353; a=rsa-sha256; cv=none; b=zb7ssfVff3Fy3TIsx7iYwwxBP2Wt4NnxdpxiPco0JD2+KNyxRZQXdLvoU8WfFx1jRRDwQv KiJm06ihOAixzmJx4rVKERAdnA6/Qnp8/xL2oKL6CyQMAf+MzrQ4EAeqmDLHpBlWc2tFll PeriSSLg/LqYKTcwkCmpy0M71hJfJ1E= DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=redhat.com; s=mimecast20190719; t=1723486364; h=from:from:reply-to:subject:subject:date:date:message-id:message-id: to:to:cc:cc:mime-version:mime-version:content-type:content-type: content-transfer-encoding:content-transfer-encoding: in-reply-to:in-reply-to:references:references; bh=vL+nA7AT6HH2fptU3fDtubaziCDql6jx+j3l3b3Xtg8=; b=U1P3j7ENeI7q8+7/oKuw8y8SjnekQ+qiuXrDp7o5JAkV2KFmW6PUPAJUQM7vpC6SRKhTOA 2ll13AGeGX0hmg8Rcso2nQSno3+5Spxv0+KSh9nuAp+CX56ZVM30M2wjEs0bMbnkrRLm5b eYqK3HLRl2SC9Vtssq5VpNvtbTyM1ec= Received: from mail-qk1-f198.google.com (mail-qk1-f198.google.com [209.85.222.198]) by relay.mimecast.com with ESMTP with STARTTLS (version=TLSv1.3, cipher=TLS_AES_256_GCM_SHA384) id us-mta-442-IDOXQsPXO-KhQQb0OnkSdQ-1; Mon, 12 Aug 2024 14:12:40 -0400 X-MC-Unique: IDOXQsPXO-KhQQb0OnkSdQ-1 Received: by mail-qk1-f198.google.com with SMTP id af79cd13be357-7a1e1828321so15941185a.1 for ; Mon, 12 Aug 2024 11:12:40 -0700 (PDT) X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20230601; t=1723486360; x=1724091160; h=content-transfer-encoding:mime-version:references:in-reply-to :message-id:date:subject:cc:to:from:x-gm-message-state:from:to:cc :subject:date:message-id:reply-to; bh=vL+nA7AT6HH2fptU3fDtubaziCDql6jx+j3l3b3Xtg8=; b=u2nZOiLTKHmeVQDekWbo/Nggf+dSp57wlaXpdAV0t4+ElW6JHb5dknjo/LY8j58Or+ 4yV+1q9hvodsYavOTORCfi4xMEbQqsObiYnHs+WNUgBRH4u/0wL0ippdsV5/OUPK/YNq rAJGn52H4wm8nQBJy0/0G2eB3sJZCCD/gDiY/X1HT63FH/lsbJA8TzSS5ITuYc/+TT0e ZylO0K7N9tF28XvTYPQNHJx7R93+LddPZBr6Olh4uHkYB6+5U0+YEvSwapuRQF+k27vt GaqqUjlHXGug9a7Lpc5Ilh/Uyw3GXDZ0O8MqbuG+Bndtwqz0sS7/VkA6p4j0CDBFmU2o W3Kg== X-Forwarded-Encrypted: i=1; AJvYcCWu8u1llPX+MpOv26Prp6M36cUdKO5Cd3dv//TV+CLjG9rvj7YQN+Qd3CxkSIKG9p6ayjvuXU7Hiw5MZkov66Ze9y8= X-Gm-Message-State: AOJu0YxSgdiCV8kG/HIBMYxHQnBoUbeZzDSHUlQalzgu0WehJjGjL6rD 5RQZOvUi5bc7/U7UflI8j749VuANODZKPoXCmaICRCaQG8t2ccMxy8sl36OEOrCGOfXrRQCwK0m PWAmucV+b/AdSBYQWo1auNaSij0FD0gCnDkD8bQG+0UWUM5xh X-Received: by 2002:a05:620a:d95:b0:7a1:5683:b04b with SMTP id af79cd13be357-7a4e1625378mr68903985a.9.1723486360175; Mon, 12 Aug 2024 11:12:40 -0700 (PDT) X-Google-Smtp-Source: AGHT+IHermK5CePcyhPQZfo7QtCxftIfNR9zVAKZznJ8AtnLMjyZZZpq0Zdfjnp4iDQgBs3zunmzLQ== X-Received: by 2002:a05:620a:d95:b0:7a1:5683:b04b with SMTP id af79cd13be357-7a4e1625378mr68901885a.9.1723486359781; Mon, 12 Aug 2024 11:12:39 -0700 (PDT) Received: from x1n.redhat.com (pool-99-254-121-117.cpe.net.cable.rogers.com. [99.254.121.117]) by smtp.gmail.com with ESMTPSA id af79cd13be357-7a4c7dee013sm268663985a.84.2024.08.12.11.12.37 (version=TLS1_3 cipher=TLS_AES_256_GCM_SHA384 bits=256/256); Mon, 12 Aug 2024 11:12:39 -0700 (PDT) From: Peter Xu To: linux-kernel@vger.kernel.org, linux-mm@kvack.org Cc: "Kirill A . Shutemov" , Nicholas Piggin , David Hildenbrand , Matthew Wilcox , Andrew Morton , James Houghton , Huang Ying , "Aneesh Kumar K . V" , peterx@redhat.com, Vlastimil Babka , Rick P Edgecombe , Hugh Dickins , Borislav Petkov , Christophe Leroy , Michael Ellerman , Rik van Riel , Dan Williams , Mel Gorman , x86@kernel.org, Ingo Molnar , linuxppc-dev@lists.ozlabs.org, Dave Hansen , Dave Jiang , Oscar Salvador , Thomas Gleixner Subject: [PATCH v5 5/7] mm/x86: Implement arch_check_zapped_pud() Date: Mon, 12 Aug 2024 14:12:23 -0400 Message-ID: <20240812181225.1360970-6-peterx@redhat.com> X-Mailer: git-send-email 2.45.0 In-Reply-To: <20240812181225.1360970-1-peterx@redhat.com> References: <20240812181225.1360970-1-peterx@redhat.com> MIME-Version: 1.0 X-Mimecast-Spam-Score: 0 X-Mimecast-Originator: redhat.com X-Rspam-User: X-Stat-Signature: a5psuw6ezxdbfg4d5yzogn4jm7hxrkqo X-Rspamd-Queue-Id: ACA2C4001B X-Rspamd-Server: rspam11 X-HE-Tag: 1723486364-365483 X-HE-Meta: U2FsdGVkX18QOHCbsw+hub+zwTNO+SURPRJHLlFR9oOS/wWeIyFzo/c5LtVY0kPIF65YMfHbfvfud3zg/1wTyzJQ2/Yh83/VvKCtwQY/oJqfSw8588PlKDnwCJgyxKGiMmE9Qt3HyprHbkxhEvhmgHKJ+Rox5eFX3/YLCrfJyT/UokRrrpUBiZngQdQYXHni649b/eu+fk5TdIG96KkDZOA4yeTyAF2TYuBseut1yfXoZi9i9i45fnXX0xiO0nsJgWenotQ9IKJjVKSCz6qExHLzVehu5FTpFZvxwNvakweoWQvzSBE/5GoyTdF58iR65SUXJCi7LvUsD/USN5ju+hj55HBUwO2yHVBgX/ZELghJbIBjmQTY96zIpxK3vG4n+GnsqStlnuSabIcGmuH0adzdONADTL8tbWxnXQ2eGerAnQAGMMeyd0r+gzfrW8iDMLab3Df9zOpHj0OE+dwEb4aGl2ELc06Ivfhn+m8+Wke36f2fuTxpnyuo1GQVsz+mdu+SwsvFn9A1TXkr2vazczj7SOD+Y2WJ+P18ejfy2TFN6c7M/kSeDnKTUfOMgetEvHRAwmKj1lvxAMEDUvIdcQaEBoaYy07uEoZ6ALAlzxCPBgdLKTd8oqgD3os986yhzaqgPgxKpeSY85Q9cPP5g+4nbRmBYuInVO1eowvWWvJYKrjejL3+5nCTDge84IhgFlCHL9Bunma1+JGdwnZIxQNXJj5ATtqGTKZEFwc1TWM9kLeUsVJfpdtiIEMWfK9S36u8DFiNrM2O88RxpM6s7nLQTDKQickTQIOn9xVSg05bCRIwEpDTuB+G+mSuVAZkgzsr/ctaEdcS5iw6+/HMtKobqecVRcKk7eFybPRCpCE2u/CrBgTcf0P2NhwdH6Ab4okCqmiL8VXyUM2+dEEaHDHz3JaVz4br3ENCdQi9D70iJED99d+KDp1YrSVdzhP36eslEdewUwILH/JCv98 c4cI5X5F TdJU52f1B3DRjLHuWaAx5rqDA3SveT2rA0fpSP2aMOP9VpssDWveZG3S5TDDj+RInNPqTOBZ04VjDwD0LMAMC+7zyJkfqjV0eS+pZBwkYGh6hsELBEHD04XgocyasWfXdrf9H8a8Hi8hZu7xS1AM+HGIZVQ2ZBzateGfVYZmZeszvpGUg1kTfQc9/cut8FfsyWDrN4/B2tNx6XbplYi6MGOodzUXdZYAAc8fXZgIAtRKV525bvDc1T1+9Li3xM9WCLK/LMUcnSz0PbDEWqFnbWFBZS9tpM+kpJcV5L00f/q5XdqxL74Op7kGvOUl1gvReCDVVgtTLNjEVEmSVzNZT25CzE073Ca5BJFamTwVA1nkd6dnQ7rPktp3+V1HtrT8wwZTvbh4cTteV6nFQN5449UPuB3UGlzDKGUoaF2zfLqJA+Do1U+HvQN918bkKQ33GP7Z9d4FZvNuDZmde1TYOInEYb4/PXhdaAdA5AR+XLz4gNJg4EgeMEfGHIsc1oYo6yE/mIJxnLo1YjgQ= X-Bogosity: Ham, tests=bogofilter, spamicity=0.000000, version=1.2.4 Sender: owner-linux-mm@kvack.org Precedence: bulk X-Loop: owner-majordomo@kvack.org List-ID: List-Subscribe: List-Unsubscribe: Introduce arch_check_zapped_pud() to sanity check shadow stack on PUD zaps. It has the same logic as the PMD helper. One thing to mention is, it might be a good idea to use page_table_check in the future for trapping wrong setups of shadow stack pgtable entries [1]. That is left for the future as a separate effort. [1] https://lore.kernel.org/all/59d518698f664e07c036a5098833d7b56b953305.camel@intel.com Cc: "Edgecombe, Rick P" Cc: Thomas Gleixner Cc: Ingo Molnar Cc: Borislav Petkov Cc: Dave Hansen Cc: x86@kernel.org Acked-by: David Hildenbrand Signed-off-by: Peter Xu --- arch/x86/include/asm/pgtable.h | 10 ++++++++++ arch/x86/mm/pgtable.c | 6 ++++++ include/linux/pgtable.h | 6 ++++++ mm/huge_memory.c | 4 +++- 4 files changed, 25 insertions(+), 1 deletion(-) diff --git a/arch/x86/include/asm/pgtable.h b/arch/x86/include/asm/pgtable.h index a2a3bd4c1bda..fdb8ac9e7030 100644 --- a/arch/x86/include/asm/pgtable.h +++ b/arch/x86/include/asm/pgtable.h @@ -174,6 +174,13 @@ static inline int pud_young(pud_t pud) return pud_flags(pud) & _PAGE_ACCESSED; } +static inline bool pud_shstk(pud_t pud) +{ + return cpu_feature_enabled(X86_FEATURE_SHSTK) && + (pud_flags(pud) & (_PAGE_RW | _PAGE_DIRTY | _PAGE_PSE)) == + (_PAGE_DIRTY | _PAGE_PSE); +} + static inline int pte_write(pte_t pte) { /* @@ -1667,6 +1674,9 @@ void arch_check_zapped_pte(struct vm_area_struct *vma, pte_t pte); #define arch_check_zapped_pmd arch_check_zapped_pmd void arch_check_zapped_pmd(struct vm_area_struct *vma, pmd_t pmd); +#define arch_check_zapped_pud arch_check_zapped_pud +void arch_check_zapped_pud(struct vm_area_struct *vma, pud_t pud); + #ifdef CONFIG_XEN_PV #define arch_has_hw_nonleaf_pmd_young arch_has_hw_nonleaf_pmd_young static inline bool arch_has_hw_nonleaf_pmd_young(void) diff --git a/arch/x86/mm/pgtable.c b/arch/x86/mm/pgtable.c index f5931499c2d6..36e7139a61d9 100644 --- a/arch/x86/mm/pgtable.c +++ b/arch/x86/mm/pgtable.c @@ -926,3 +926,9 @@ void arch_check_zapped_pmd(struct vm_area_struct *vma, pmd_t pmd) VM_WARN_ON_ONCE(!(vma->vm_flags & VM_SHADOW_STACK) && pmd_shstk(pmd)); } + +void arch_check_zapped_pud(struct vm_area_struct *vma, pud_t pud) +{ + /* See note in arch_check_zapped_pte() */ + VM_WARN_ON_ONCE(!(vma->vm_flags & VM_SHADOW_STACK) && pud_shstk(pud)); +} diff --git a/include/linux/pgtable.h b/include/linux/pgtable.h index 2a6a3cccfc36..780f3b439d98 100644 --- a/include/linux/pgtable.h +++ b/include/linux/pgtable.h @@ -447,6 +447,12 @@ static inline void arch_check_zapped_pmd(struct vm_area_struct *vma, } #endif +#ifndef arch_check_zapped_pud +static inline void arch_check_zapped_pud(struct vm_area_struct *vma, pud_t pud) +{ +} +#endif + #ifndef __HAVE_ARCH_PTEP_GET_AND_CLEAR static inline pte_t ptep_get_and_clear(struct mm_struct *mm, unsigned long address, diff --git a/mm/huge_memory.c b/mm/huge_memory.c index 0024266dea0a..81c5da0708ed 100644 --- a/mm/huge_memory.c +++ b/mm/huge_memory.c @@ -2293,12 +2293,14 @@ int zap_huge_pud(struct mmu_gather *tlb, struct vm_area_struct *vma, pud_t *pud, unsigned long addr) { spinlock_t *ptl; + pud_t orig_pud; ptl = __pud_trans_huge_lock(pud, vma); if (!ptl) return 0; - pudp_huge_get_and_clear_full(vma, addr, pud, tlb->fullmm); + orig_pud = pudp_huge_get_and_clear_full(vma, addr, pud, tlb->fullmm); + arch_check_zapped_pud(vma, orig_pud); tlb_remove_pud_tlb_entry(tlb, pud, addr); if (vma_is_special_huge(vma)) { spin_unlock(ptl); From patchwork Mon Aug 12 18:12:24 2024 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Peter Xu X-Patchwork-Id: 13760912 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from kanga.kvack.org (kanga.kvack.org [205.233.56.17]) by smtp.lore.kernel.org (Postfix) with ESMTP id 65439C3DA7F for ; Mon, 12 Aug 2024 18:12:52 +0000 (UTC) Received: by kanga.kvack.org (Postfix) id CF8BC6B00A2; Mon, 12 Aug 2024 14:12:49 -0400 (EDT) Received: by kanga.kvack.org (Postfix, from userid 40) id C323F6B00A4; Mon, 12 Aug 2024 14:12:49 -0400 (EDT) X-Delivered-To: int-list-linux-mm@kvack.org Received: by kanga.kvack.org (Postfix, from userid 63042) id AAC5A6B00A5; Mon, 12 Aug 2024 14:12:49 -0400 (EDT) X-Delivered-To: linux-mm@kvack.org Received: from relay.hostedemail.com (smtprelay0015.hostedemail.com [216.40.44.15]) by kanga.kvack.org (Postfix) with ESMTP id 8408B6B00A2 for ; Mon, 12 Aug 2024 14:12:49 -0400 (EDT) Received: from smtpin06.hostedemail.com (a10.router.float.18 [10.200.18.1]) by unirelay05.hostedemail.com (Postfix) with ESMTP id F2E854036A for ; Mon, 12 Aug 2024 18:12:48 +0000 (UTC) X-FDA: 82444389216.06.47D9967 Received: from us-smtp-delivery-124.mimecast.com (us-smtp-delivery-124.mimecast.com [170.10.129.124]) by imf28.hostedemail.com (Postfix) with ESMTP id E26F6C0013 for ; Mon, 12 Aug 2024 18:12:46 +0000 (UTC) Authentication-Results: imf28.hostedemail.com; dkim=pass header.d=redhat.com header.s=mimecast20190719 header.b=FqBb78o8; spf=pass (imf28.hostedemail.com: domain of peterx@redhat.com designates 170.10.129.124 as permitted sender) smtp.mailfrom=peterx@redhat.com; dmarc=pass (policy=none) header.from=redhat.com ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=hostedemail.com; s=arc-20220608; t=1723486312; h=from:from:sender:reply-to:subject:subject:date:date: message-id:message-id:to:to:cc:cc:mime-version:mime-version: content-type:content-type: content-transfer-encoding:content-transfer-encoding: in-reply-to:in-reply-to:references:references:dkim-signature; bh=j48ti3LHCtaDfvs8kVCshb+r8YjKqoQPQoRoRoS7ZWU=; b=bDo/LXYu48AuDG6oBqstxsE15Zf5iHf28eipZo75zYP8rnDCpCev808b2VQABfZLIQmJMX hQYiWQPER3gHF4zcQTPhe8AGBgRQ4ycjltJ3QsacKcALiDyv7V08m9kxqq8WKqOgBBcCuS E9E6dKsWQCh4suLhhivP+N7fIsLZjF8= ARC-Authentication-Results: i=1; imf28.hostedemail.com; dkim=pass header.d=redhat.com header.s=mimecast20190719 header.b=FqBb78o8; spf=pass (imf28.hostedemail.com: domain of peterx@redhat.com designates 170.10.129.124 as permitted sender) smtp.mailfrom=peterx@redhat.com; dmarc=pass (policy=none) header.from=redhat.com ARC-Seal: i=1; s=arc-20220608; d=hostedemail.com; t=1723486312; a=rsa-sha256; cv=none; b=cicVDhRbSOrf0kPrS39MXl+aTVQtcf3SYgdcGQEH+gZM4dFDkg86fFfYDd4xaaUv4Q7Pa/ J4023SW9MexIabTtPniLazjJqPwdMKc8K9zvmC/EEmRCa+dkhhA44xu5wixtFkpS8EJZzD zttrJyHl94mbXh30Hsd7okudUetO1FY= DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=redhat.com; s=mimecast20190719; t=1723486366; h=from:from:reply-to:subject:subject:date:date:message-id:message-id: to:to:cc:cc:mime-version:mime-version:content-type:content-type: content-transfer-encoding:content-transfer-encoding: in-reply-to:in-reply-to:references:references; bh=j48ti3LHCtaDfvs8kVCshb+r8YjKqoQPQoRoRoS7ZWU=; b=FqBb78o8PXUVqSjPeJBYDDx/gQ9Ck1yIyPzQEe7cCZS3h8iQhz+d3t93dZksd+7LXPKA95 IEnEmvc5BHs8Le4RK4AOuq8QZamR8pxcV9f5RiY2iYKgLFs2KJ3/17rznE6nMx2remSJ5u OzIpnTPdaXRPhCClknvXFb9qnscMQCY= Received: from mail-qt1-f198.google.com (mail-qt1-f198.google.com [209.85.160.198]) by relay.mimecast.com with ESMTP with STARTTLS (version=TLSv1.3, cipher=TLS_AES_256_GCM_SHA384) id us-mta-9-TJFp-bSnOZOY0DCXffwZOw-1; Mon, 12 Aug 2024 14:12:42 -0400 X-MC-Unique: TJFp-bSnOZOY0DCXffwZOw-1 Received: by mail-qt1-f198.google.com with SMTP id d75a77b69052e-44fecb3dfd7so3082641cf.2 for ; Mon, 12 Aug 2024 11:12:42 -0700 (PDT) X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20230601; t=1723486362; x=1724091162; h=content-transfer-encoding:mime-version:references:in-reply-to :message-id:date:subject:cc:to:from:x-gm-message-state:from:to:cc :subject:date:message-id:reply-to; bh=j48ti3LHCtaDfvs8kVCshb+r8YjKqoQPQoRoRoS7ZWU=; b=nZ7Mc0O1GdQJs0Cc5H5lo/sJts9SJA9f/78XdoFJeqZXEEeWoQg7wm/MfpbSDnl6pO HT99NF3VSjgkpNzuao41+YmhVyhqB8gAJrFQ54koza9xizDvMZxZFr/QAc21+0I3zCUK 6LFz5VhSmoeBargGQspI3Ky7w0kTDgp/4FOU/P7bjx3yFuYhY6OZol9gsDK9+PPmbZPR g4T9ufxkeDE9Sw0vKDDjotQR7Z50bMFI/6kM7iU7bhVb91Xxdgx19UvhNnd53fOmoqRS TJdoEIfoNo5XOXaksuKGem4fU+X9YDyAb1tPiZ1XWdSqauP6yE8u2IQ5TbeqaluQbP1f biWA== X-Forwarded-Encrypted: i=1; AJvYcCWGmElNJ8NzW0LN14IyGa2H736iKIY/EKMZQljHwJUW38LZHC+5EwuZB5Qfihf7dD/DJzVJ1mmtuw==@kvack.org X-Gm-Message-State: AOJu0YxJtny9EpExyMUYzW4m9EyquoNLLHoq/N/Mwz1qWBXYO5kYRqmS C3jpRSaWeJSQTyyN685A5WEdoBh86z5p9p9uz/8tKlwNRk8UbTsxZO83C5KagSw6wq4/PyrS4Ow 1k/lXS+o/Gc3QkvP2ttq+J5E994itEBCHfa5ufx7O46APWufT X-Received: by 2002:a05:620a:4609:b0:79d:6273:9993 with SMTP id af79cd13be357-7a4e15be5b1mr69031085a.6.1723486362333; Mon, 12 Aug 2024 11:12:42 -0700 (PDT) X-Google-Smtp-Source: AGHT+IFnMgvJOIkVw2s4SdK0L87wmxjCzEZ4UjgHz1yAaoa3V+0fWl6Gh90ffNusNtYhZ+374uYqxw== X-Received: by 2002:a05:620a:4609:b0:79d:6273:9993 with SMTP id af79cd13be357-7a4e15be5b1mr69028285a.6.1723486361844; Mon, 12 Aug 2024 11:12:41 -0700 (PDT) Received: from x1n.redhat.com (pool-99-254-121-117.cpe.net.cable.rogers.com. [99.254.121.117]) by smtp.gmail.com with ESMTPSA id af79cd13be357-7a4c7dee013sm268663985a.84.2024.08.12.11.12.39 (version=TLS1_3 cipher=TLS_AES_256_GCM_SHA384 bits=256/256); Mon, 12 Aug 2024 11:12:41 -0700 (PDT) From: Peter Xu To: linux-kernel@vger.kernel.org, linux-mm@kvack.org Cc: "Kirill A . Shutemov" , Nicholas Piggin , David Hildenbrand , Matthew Wilcox , Andrew Morton , James Houghton , Huang Ying , "Aneesh Kumar K . V" , peterx@redhat.com, Vlastimil Babka , Rick P Edgecombe , Hugh Dickins , Borislav Petkov , Christophe Leroy , Michael Ellerman , Rik van Riel , Dan Williams , Mel Gorman , x86@kernel.org, Ingo Molnar , linuxppc-dev@lists.ozlabs.org, Dave Hansen , Dave Jiang , Oscar Salvador , Thomas Gleixner Subject: [PATCH v5 6/7] mm/x86: Add missing pud helpers Date: Mon, 12 Aug 2024 14:12:24 -0400 Message-ID: <20240812181225.1360970-7-peterx@redhat.com> X-Mailer: git-send-email 2.45.0 In-Reply-To: <20240812181225.1360970-1-peterx@redhat.com> References: <20240812181225.1360970-1-peterx@redhat.com> MIME-Version: 1.0 X-Mimecast-Spam-Score: 0 X-Mimecast-Originator: redhat.com X-Stat-Signature: tbhbrmgdddtg8q3dpajce94uxuqztzq3 X-Rspam-User: X-Rspamd-Queue-Id: E26F6C0013 X-Rspamd-Server: rspam02 X-HE-Tag: 1723486366-452657 X-HE-Meta: U2FsdGVkX1/XkCadHRMM/BPXoxoNi4IAjO+DDV13jQEmn+XU+5yXdnYK7fwdM5WOZfm/LRoEznfFLGN7HYXCJ1RF8zDwCLcAr5MvpHbtsRf5vCQ22mjxMxHm8kFpHHfKf160FiDxxHkR4WjdklISaZ6yH9SDSlEik/5I6bkIPhrOMb3rS7A1ufmMliabewm6UugjZcA3nCfFphA6T41lYjVV8q7St6C7U0XQQboow4gDbl+aFKbuZZKgFRC4Ho3huc2jlKPuONxy5iY6SExNlKZUC5CnpwCbpyKH4vSdTJF8MM9q3hT9kJj3YaRSc8bOAWyDlV6x+9B5+oz3IPOw8DzUUPub/DBS9gwpph924I4WJW3cFVCFQUFg+ts/5gEVIEXw+Ase9HuSJmYwrK5sR1LyG14NI+R9FHn1bkLTdzegI4zHAuceqJh6LPTaMdfSgGvE55mqgCpd9qou7RihEBniED2OUDcPdfduVCYXv3bGF9KEwVbUf9DGB1v32Nd4zfydBpOprDhu5FMMv2GHeG8Vnth+0geTFDeGBjlifyLeqnhoHBMM1O1uSV1am7RptJCv5yUwfLWaoXJglP8dNdrQe/ohAjKPbzRYgZ4WlJ0UJsMkCdQDjSdn3ikFFqJHYcttKhJZwVVVXF/hh5y2MkJBFOeQ7jjXQ40mFkt41zLfw92nVpOBwbEKey5o+JiwcaxAMZNXK+oHAcyEXfGoccBLBWdgHA9R1ddnu6WW3o17YXeaahUNJdyvE0BBcFxHBA9D4jrZhO66nU580nd/BNrf0Brk7FwvHhb8VK+sOeqcc0fGyL5/JzgjNZucq3Q+ecL13/GHjkpW/CErOygaUbHXhtc9VhEbtNmtvblFDd214OOJgF8xeEcJFIsgUHnIL90ZSDlsaY9Y4phjpTMsgrQDv/xfi0J4GOa3JLIev7naEqsIxCoPc2Ui787vODC5hDuUMtCOb2Y0CzLICqp 5ekaTtTa bG6Qqof/Kjw+G4Da0IiCRZx3sstRmeV6w1ER77DDvubPVr4cqPdQsrTaDbHRuVjsm46XUqx8Q19lJD1Enskk4dOpsu2eWFnNbmgm+Hw/HgTh0xJn+T2AFdxII+3vpMZiRIP5pogcBn0kn5Za5LsuR0jHBQpH17QgdELjX2ePqlquP+2ZIzI3xD2O21xWQA/FW3YahpZg2pF3vhVwy6yWOeV9BAkESiCi20MQN8Pdf+51jCohkAFCHU4TYJSseaqrQbwumB+A5WgYUUzKV60+tIu6hSANgZT5G3wRWJzfUn7gRujxGwfiTg/R2pn0LP7ClQJqtx/gX26FGjtBj29AGyu+gvCzncu8M5ZrB+qhIUgSoH6d89EFUbzmOjQ== X-Bogosity: Ham, tests=bogofilter, spamicity=0.000000, version=1.2.4 Sender: owner-linux-mm@kvack.org Precedence: bulk X-Loop: owner-majordomo@kvack.org List-ID: List-Subscribe: List-Unsubscribe: Some new helpers will be needed for pud entry updates soon. Introduce these helpers by referencing the pmd ones. Namely: - pudp_invalidate(): this helper invalidates a huge pud before a split happens, so that the invalidated pud entry will make sure no race will happen (either with software, like a concurrent zap, or hardware, like a/d bit lost). - pud_modify(): this helper applies a new pgprot to an existing huge pud mapping. For more information on why we need these two helpers, please refer to the corresponding pmd helpers in the mprotect() code path. When at it, simplify the pud_modify()/pmd_modify() comments on shadow stack pgtable entries to reference pte_modify() to avoid duplicating the whole paragraph three times. Cc: Thomas Gleixner Cc: Ingo Molnar Cc: Borislav Petkov Cc: Dave Hansen Cc: x86@kernel.org Signed-off-by: Peter Xu --- arch/x86/include/asm/pgtable.h | 57 +++++++++++++++++++++++++++++----- arch/x86/mm/pgtable.c | 12 +++++++ 2 files changed, 61 insertions(+), 8 deletions(-) diff --git a/arch/x86/include/asm/pgtable.h b/arch/x86/include/asm/pgtable.h index fdb8ac9e7030..8d12bfad6a1d 100644 --- a/arch/x86/include/asm/pgtable.h +++ b/arch/x86/include/asm/pgtable.h @@ -787,6 +787,12 @@ static inline pmd_t pmd_mkinvalid(pmd_t pmd) __pgprot(pmd_flags(pmd) & ~(_PAGE_PRESENT|_PAGE_PROTNONE))); } +static inline pud_t pud_mkinvalid(pud_t pud) +{ + return pfn_pud(pud_pfn(pud), + __pgprot(pud_flags(pud) & ~(_PAGE_PRESENT|_PAGE_PROTNONE))); +} + static inline u64 flip_protnone_guard(u64 oldval, u64 val, u64 mask); static inline pte_t pte_modify(pte_t pte, pgprot_t newprot) @@ -834,14 +840,8 @@ static inline pmd_t pmd_modify(pmd_t pmd, pgprot_t newprot) pmd_result = __pmd(val); /* - * To avoid creating Write=0,Dirty=1 PMDs, pte_modify() needs to avoid: - * 1. Marking Write=0 PMDs Dirty=1 - * 2. Marking Dirty=1 PMDs Write=0 - * - * The first case cannot happen because the _PAGE_CHG_MASK will filter - * out any Dirty bit passed in newprot. Handle the second case by - * going through the mksaveddirty exercise. Only do this if the old - * value was Write=1 to avoid doing this on Shadow Stack PTEs. + * Avoid creating shadow stack PMD by accident. See comment in + * pte_modify(). */ if (oldval & _PAGE_RW) pmd_result = pmd_mksaveddirty(pmd_result); @@ -851,6 +851,29 @@ static inline pmd_t pmd_modify(pmd_t pmd, pgprot_t newprot) return pmd_result; } +static inline pud_t pud_modify(pud_t pud, pgprot_t newprot) +{ + pudval_t val = pud_val(pud), oldval = val; + pud_t pud_result; + + val &= _HPAGE_CHG_MASK; + val |= check_pgprot(newprot) & ~_HPAGE_CHG_MASK; + val = flip_protnone_guard(oldval, val, PHYSICAL_PUD_PAGE_MASK); + + pud_result = __pud(val); + + /* + * Avoid creating shadow stack PUD by accident. See comment in + * pte_modify(). + */ + if (oldval & _PAGE_RW) + pud_result = pud_mksaveddirty(pud_result); + else + pud_result = pud_clear_saveddirty(pud_result); + + return pud_result; +} + /* * mprotect needs to preserve PAT and encryption bits when updating * vm_page_prot @@ -1389,10 +1412,28 @@ static inline pmd_t pmdp_establish(struct vm_area_struct *vma, } #endif +#ifdef CONFIG_HAVE_ARCH_TRANSPARENT_HUGEPAGE_PUD +static inline pud_t pudp_establish(struct vm_area_struct *vma, + unsigned long address, pud_t *pudp, pud_t pud) +{ + page_table_check_pud_set(vma->vm_mm, pudp, pud); + if (IS_ENABLED(CONFIG_SMP)) { + return xchg(pudp, pud); + } else { + pud_t old = *pudp; + WRITE_ONCE(*pudp, pud); + return old; + } +} +#endif + #define __HAVE_ARCH_PMDP_INVALIDATE_AD extern pmd_t pmdp_invalidate_ad(struct vm_area_struct *vma, unsigned long address, pmd_t *pmdp); +pud_t pudp_invalidate(struct vm_area_struct *vma, unsigned long address, + pud_t *pudp); + /* * Page table pages are page-aligned. The lower half of the top * level is used for userspace and the top half for the kernel. diff --git a/arch/x86/mm/pgtable.c b/arch/x86/mm/pgtable.c index 36e7139a61d9..5745a354a241 100644 --- a/arch/x86/mm/pgtable.c +++ b/arch/x86/mm/pgtable.c @@ -641,6 +641,18 @@ pmd_t pmdp_invalidate_ad(struct vm_area_struct *vma, unsigned long address, } #endif +#if defined(CONFIG_TRANSPARENT_HUGEPAGE) && \ + defined(CONFIG_HAVE_ARCH_TRANSPARENT_HUGEPAGE_PUD) +pud_t pudp_invalidate(struct vm_area_struct *vma, unsigned long address, + pud_t *pudp) +{ + VM_WARN_ON_ONCE(!pud_present(*pudp)); + pud_t old = pudp_establish(vma, address, pudp, pud_mkinvalid(*pudp)); + flush_pud_tlb_range(vma, address, address + HPAGE_PUD_SIZE); + return old; +} +#endif + /** * reserve_top_address - reserves a hole in the top of kernel address space * @reserve - size of hole to reserve From patchwork Mon Aug 12 18:12:25 2024 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Peter Xu X-Patchwork-Id: 13760913 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from kanga.kvack.org (kanga.kvack.org [205.233.56.17]) by smtp.lore.kernel.org (Postfix) with ESMTP id 357D4C531DC for ; Mon, 12 Aug 2024 18:12:55 +0000 (UTC) Received: by kanga.kvack.org (Postfix) id 92A396B00A5; Mon, 12 Aug 2024 14:12:51 -0400 (EDT) Received: by kanga.kvack.org (Postfix, from userid 40) id 8D9DA6B00A6; Mon, 12 Aug 2024 14:12:51 -0400 (EDT) X-Delivered-To: int-list-linux-mm@kvack.org Received: by kanga.kvack.org (Postfix, from userid 63042) id 702A06B00A7; Mon, 12 Aug 2024 14:12:51 -0400 (EDT) X-Delivered-To: linux-mm@kvack.org Received: from relay.hostedemail.com (smtprelay0014.hostedemail.com [216.40.44.14]) by kanga.kvack.org (Postfix) with ESMTP id 50F886B00A5 for ; Mon, 12 Aug 2024 14:12:51 -0400 (EDT) Received: from smtpin01.hostedemail.com (a10.router.float.18 [10.200.18.1]) by unirelay03.hostedemail.com (Postfix) with ESMTP id 0DB1DA032A for ; Mon, 12 Aug 2024 18:12:51 +0000 (UTC) X-FDA: 82444389342.01.02C5D4B Received: from us-smtp-delivery-124.mimecast.com (us-smtp-delivery-124.mimecast.com [170.10.133.124]) by imf03.hostedemail.com (Postfix) with ESMTP id DB25020036 for ; Mon, 12 Aug 2024 18:12:48 +0000 (UTC) Authentication-Results: imf03.hostedemail.com; dkim=pass header.d=redhat.com header.s=mimecast20190719 header.b=LsUwHmgy; dmarc=pass (policy=none) header.from=redhat.com; spf=pass (imf03.hostedemail.com: domain of peterx@redhat.com designates 170.10.133.124 as permitted sender) smtp.mailfrom=peterx@redhat.com ARC-Seal: i=1; s=arc-20220608; d=hostedemail.com; t=1723486357; a=rsa-sha256; cv=none; b=0BVNRqex5vq2d6J2JuwSpYkdDcHE5cqgdRId0/W4QIqDPszxDqDtxt/MTMdzsGTj4K6Xg3 6mP8Kslj2d/3PbqnHDfwfOsJ74fCuB/7WCDv0At6HvatHQI1kqIwgHVAal6DUgRQAXTO/S rG4fvNq82GEsTNZBGylhDTzsKsymbaA= ARC-Authentication-Results: i=1; imf03.hostedemail.com; dkim=pass header.d=redhat.com header.s=mimecast20190719 header.b=LsUwHmgy; dmarc=pass (policy=none) header.from=redhat.com; spf=pass (imf03.hostedemail.com: domain of peterx@redhat.com designates 170.10.133.124 as permitted sender) smtp.mailfrom=peterx@redhat.com ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=hostedemail.com; s=arc-20220608; t=1723486357; h=from:from:sender:reply-to:subject:subject:date:date: message-id:message-id:to:to:cc:cc:mime-version:mime-version: content-type:content-type: content-transfer-encoding:content-transfer-encoding: in-reply-to:in-reply-to:references:references:dkim-signature; bh=A/6PEtzQTa+drbYKkEnodTZuUMYm4tSA4vStMaJG0Ro=; b=8jCb1N9VsT7PlX1jZBM/IW0iE1vyLcicf9r1ikvmnZ8psN/zXfgrLw7SV49XEo+gS8zNDH FT62IQESI+APaD22rIcg+JT+dFvg9FTj/Pny5bvYW4po0HqCMgLjuZ5DpJ8epeqCcyajH0 tC8s2Om87pLCOhnrV3457LAwBcqJbpo= DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=redhat.com; s=mimecast20190719; t=1723486368; h=from:from:reply-to:subject:subject:date:date:message-id:message-id: to:to:cc:cc:mime-version:mime-version:content-type:content-type: content-transfer-encoding:content-transfer-encoding: in-reply-to:in-reply-to:references:references; bh=A/6PEtzQTa+drbYKkEnodTZuUMYm4tSA4vStMaJG0Ro=; b=LsUwHmgySMm6mK2YuBCj95NSk3THe3p0E9aspsuhM4V3HIucDTCVMpn5OgtUJLB6SkQJGa 4PmN2e9RBzUomBydl0grWlM1vuBy0Pfb6xPzbO9HpXdK4neDvol50kn2lBfWcIRTayo9KM 7RLd7TeHItqADFMgts+MzqS0NA+/2ZI= Received: from mail-vs1-f72.google.com (mail-vs1-f72.google.com [209.85.217.72]) by relay.mimecast.com with ESMTP with STARTTLS (version=TLSv1.3, cipher=TLS_AES_256_GCM_SHA384) id us-mta-210-950vymddN5eIPu2s_SQwqQ-1; Mon, 12 Aug 2024 14:12:45 -0400 X-MC-Unique: 950vymddN5eIPu2s_SQwqQ-1 Received: by mail-vs1-f72.google.com with SMTP id ada2fe7eead31-49291c389b9so194146137.2 for ; Mon, 12 Aug 2024 11:12:44 -0700 (PDT) X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20230601; t=1723486364; x=1724091164; h=content-transfer-encoding:mime-version:references:in-reply-to :message-id:date:subject:cc:to:from:x-gm-message-state:from:to:cc :subject:date:message-id:reply-to; bh=A/6PEtzQTa+drbYKkEnodTZuUMYm4tSA4vStMaJG0Ro=; b=jDFXT3czRurOe2+lCnGgjSO4a1d/34/4Y77DGJ9MbjiNIYhz1FwCl8sG31CYrHdMYZ NxC8gWxWQ/X9kbKwwm1vBPQXGhFYyXHgCjYWwQAnOL5Pr8H+VSQlcniSOCHtUVG2wyaa NG4P7twuKZT7CwRF7VIuO0PdwsriN/QCghjGvaSMf3+tGU4XtddIjrEcjHj8rOiwHGDE h8ydHEoYHu8oBGZKtUIU/3KrL/hUoNeyAFK6UrdMYYWK/9zWlI/CYNZnn6eBL1uEDs3P Jqy6kFM1mU85dYI3zBGGxTXwULmqUWaQpx2q4cOLdZSML9qXk5lxkjTygto0OkiZSEai rxjQ== X-Forwarded-Encrypted: i=1; AJvYcCVHONA7Q/plzpLK1nSPR41aeJhydGKtXV0g/auQ0/nXVFb/oowLyc4xIDngd6TNLNSU+LDtFmiXog==@kvack.org X-Gm-Message-State: AOJu0Yx+dbvSgkEZfY0w0R5MI/5VchU73DC1DF+LJUs8LY9ioGOaWqpm wy600UtLgHtBUwIVsnkwdErae32VzHYrZ1qFuLzEHOXlY2qivGpReUj1SLFuUM5CchukOBIe+zG bSRT+0/pLDTd9Gy/huE/g81fzAn6VXaDyYL6Aqiafsp46klfc X-Received: by 2002:a05:6102:38d1:b0:493:31f9:d14e with SMTP id ada2fe7eead31-4974398cf6amr785254137.2.1723486364420; Mon, 12 Aug 2024 11:12:44 -0700 (PDT) X-Google-Smtp-Source: AGHT+IEGg++VQElyCH8BIocMaINRG5gLiISp0Xxb9L94RctCtM0eBivu+dVoRsXo8+SybwrNJQcpMw== X-Received: by 2002:a05:6102:38d1:b0:493:31f9:d14e with SMTP id ada2fe7eead31-4974398cf6amr785238137.2.1723486364018; Mon, 12 Aug 2024 11:12:44 -0700 (PDT) Received: from x1n.redhat.com (pool-99-254-121-117.cpe.net.cable.rogers.com. [99.254.121.117]) by smtp.gmail.com with ESMTPSA id af79cd13be357-7a4c7dee013sm268663985a.84.2024.08.12.11.12.42 (version=TLS1_3 cipher=TLS_AES_256_GCM_SHA384 bits=256/256); Mon, 12 Aug 2024 11:12:43 -0700 (PDT) From: Peter Xu To: linux-kernel@vger.kernel.org, linux-mm@kvack.org Cc: "Kirill A . Shutemov" , Nicholas Piggin , David Hildenbrand , Matthew Wilcox , Andrew Morton , James Houghton , Huang Ying , "Aneesh Kumar K . V" , peterx@redhat.com, Vlastimil Babka , Rick P Edgecombe , Hugh Dickins , Borislav Petkov , Christophe Leroy , Michael Ellerman , Rik van Riel , Dan Williams , Mel Gorman , x86@kernel.org, Ingo Molnar , linuxppc-dev@lists.ozlabs.org, Dave Hansen , Dave Jiang , Oscar Salvador , Thomas Gleixner Subject: [PATCH v5 7/7] mm/mprotect: fix dax pud handlings Date: Mon, 12 Aug 2024 14:12:25 -0400 Message-ID: <20240812181225.1360970-8-peterx@redhat.com> X-Mailer: git-send-email 2.45.0 In-Reply-To: <20240812181225.1360970-1-peterx@redhat.com> References: <20240812181225.1360970-1-peterx@redhat.com> MIME-Version: 1.0 X-Mimecast-Spam-Score: 0 X-Mimecast-Originator: redhat.com X-Rspam-User: X-Rspamd-Queue-Id: DB25020036 X-Rspamd-Server: rspam01 X-Stat-Signature: 71knigi4wuzc3n1eax5qgqjn5xz8y1aa X-HE-Tag: 1723486368-478517 X-HE-Meta: U2FsdGVkX18nLdu/AJTUKxtk3jh2Mm6DkefDIw3iFtQTZdaGFFXbKa2S2Zf4Kw6QmRcjBubR6krC8grjBmI32xOnOweYdIV3BrgkTSC4z61OAq28dV4Pedbfuy1SQ7u2Tjvc61UfqATCkWuXz0ABbNYHvFqXG0BrgVvrxQd6lQ+Ze2A2DeNZp9ERx8AEplh0TSyUKU16rbXKQm0kglJzF7c7VSvqsOUaIQFrJnd27tcHBcCO7WYh8iYSc/IWy+o0JsRzY6J7dytOsi8G95AXXVVwuQ5z0auNPaq8caqxHYoTfrZZ/AQ6lV6YWpbG5N8q9qZDm0G5JOsqkxgg7m0V/jMl1PoM6IiU3X4CEFW2utPjjSAS+tEkzzUWc/3Ddj55q7TGbMXoBGdQjZSBDD7seBByYviYQWSm8jN7P671c0fykjJoucoMdeS24duycIPb6AXp9YezV4Qy15Y9Kx9m+9mdNxU4WPrPIvHjkMk4VSWwa30gkhuXp0/PCIbj7L4V8aWKJLaDAnCzNAmZAlpy9ZnH8vA/oANRkNffwtuz6/Fnobkt3CE5JrLtpRbSkxTLJmzUu25ELYwTWUm9rDTl5okFZqFyXgLt7w7R1yJLyHTw6rNdQ7Sbvq+ymSIipXnb/eKK4rUvTmgmQ48j1BpkgQkYhicEP20WXD43BOBH+f2FgM912piDNpHrYsu3s+9EImCCGCG7PxruDhlMtgKycIyLJebnU3zCtGwS1kW2OtQXxDnh/vVGq43Lkj4mijT64Fouie7THXjiMkHpfWd24K82urhLBM+1upJ71HpKFf8q2D6zZioohcGeYtXaQNwjapej5ce4D1dKcHnzrZvUa0/wG9jYjXd7Ypm4kDNGt/VDzDcFUhutceY3XY4l09zF2InNmE4jOoKYZDwj9fLstax1LjgXyUInqr7dTtSLNJTo8DwDqrsLbzyy/pZNkFNgOiWQTp6QXrIDR4OEsXU mb1POJAy p+cRVEPlfpKuDw8r0BPhQhwowyg6Ni+TBO0FE1t11EuP9iQVOVNCEjeQ5XFfb2WH+guPU1RZ9cy6IA+fxx91XWEt9BDTip10C8uNPq+/4KFyLQ/QYOnOnYG0wJcLJgjo9o17FtYvYFz5jcnrLVG4cAPcUYTJUGslsNdcsIqfrkrQxEgllRcZBW9Lxa0KC9/6DCQGKGQi7ofafOTN/aCUKLr1VuvWfoKP/t1MKvhM++o9qIq4/ecdeX5x3uhjDzVo6Gkp2UXBEwwWSw7hCSioGY8qaRQESuQUoUpb0vCFqpFS5xpu/Sb9fMfWrZJFUvKy5kV8vojd0yP4fG6EhbaTw0qs9JNjB9T263F/Dvyo+SwKxtPZvQVavX6J3EhjTz9xydpkZadeOMwZjEm4Od8KRmFhiekPeHuAhB87FAzmKkbCVqZ3xI0rPncZMXXY5YdtE035KwXWRhigX0c7AhyJvpIdS2EZQm0f2kaypykw5w13ELQKZRB+1ifLklGNKTjPXuh3qLJQbD5hvMYA97sFGd47WaU8E8/KPxZIlSGcqKbARX+Dh35KYp4DtC7knJZcJnx58eNc5W/BRuVc= X-Bogosity: Ham, tests=bogofilter, spamicity=0.000000, version=1.2.4 Sender: owner-linux-mm@kvack.org Precedence: bulk X-Loop: owner-majordomo@kvack.org List-ID: List-Subscribe: List-Unsubscribe: This is only relevant to the two archs that support PUD dax, aka, x86_64 and ppc64. PUD THPs do not yet exist elsewhere, and hugetlb PUDs do not count in this case. DAX have had PUD mappings for years, but change protection path never worked. When the path is triggered in any form (a simple test program would be: call mprotect() on a 1G dev_dax mapping), the kernel will report "bad pud". This patch should fix that. The new change_huge_pud() tries to keep everything simple. For example, it doesn't optimize write bit as that will need even more PUD helpers. It's not too bad anyway to have one more write fault in the worst case once for 1G range; may be a bigger thing for each PAGE_SIZE, though. Neither does it support userfault-wp bits, as there isn't such PUD mappings that is supported; file mappings always need a split there. The same to TLB shootdown: the pmd path (which was for x86 only) has the trick of using _ad() version of pmdp_invalidate*() which can avoid one redundant TLB, but let's also leave that for later. Again, the larger the mapping, the smaller of such effect. There's some difference on handling "retry" for change_huge_pud() (where it can return 0): it isn't like change_huge_pmd(), as the pmd version is safe with all conditions handled in change_pte_range() later, thanks to Hugh's new pte_offset_map_lock(). In short, change_pte_range() is simply smarter. For that, change_pud_range() will need proper retry if it races with something else when a huge PUD changed from under us. The last thing to mention is currently the PUD path ignores the huge pte numa counter (NUMA_HUGE_PTE_UPDATES), not only because DAX is not applicable to NUMA, but also that it's ambiguous on its own to decide how to account pud in this case. In one earlier version of this patchset I proposed to remove the counter as it doesn't even look right to do the accounting as of now [1], but then a further discussion suggests we can leave that for later, as that doesn't block this series if we choose to ignore that counter. That's what this patch does, by ignoring it. When at it, touch up the comment in pgtable_split_needed() to make it generic to either pmd or pud file THPs. [1] https://lore.kernel.org/all/20240715192142.3241557-3-peterx@redhat.com/ [2] https://lore.kernel.org/r/added2d0-b8be-4108-82ca-1367a388d0b1@redhat.com Cc: Dan Williams Cc: Matthew Wilcox Cc: Dave Jiang Cc: Hugh Dickins Cc: Kirill A. Shutemov Cc: Vlastimil Babka Cc: Thomas Gleixner Cc: Ingo Molnar Cc: Borislav Petkov Cc: Dave Hansen Cc: Michael Ellerman Cc: Aneesh Kumar K.V Cc: Oscar Salvador Cc: x86@kernel.org Cc: linuxppc-dev@lists.ozlabs.org Fixes: a00cc7d9dd93 ("mm, x86: add support for PUD-sized transparent hugepages") Fixes: 27af67f35631 ("powerpc/book3s64/mm: enable transparent pud hugepage") Signed-off-by: Peter Xu --- include/linux/huge_mm.h | 24 +++++++++++++++++++ mm/huge_memory.c | 52 +++++++++++++++++++++++++++++++++++++++++ mm/mprotect.c | 39 ++++++++++++++++++++++++------- 3 files changed, 107 insertions(+), 8 deletions(-) diff --git a/include/linux/huge_mm.h b/include/linux/huge_mm.h index ce44caa40eed..6370026689e0 100644 --- a/include/linux/huge_mm.h +++ b/include/linux/huge_mm.h @@ -342,6 +342,17 @@ void split_huge_pmd_address(struct vm_area_struct *vma, unsigned long address, void __split_huge_pud(struct vm_area_struct *vma, pud_t *pud, unsigned long address); +#ifdef CONFIG_HAVE_ARCH_TRANSPARENT_HUGEPAGE_PUD +int change_huge_pud(struct mmu_gather *tlb, struct vm_area_struct *vma, + pud_t *pudp, unsigned long addr, pgprot_t newprot, + unsigned long cp_flags); +#else +static inline int +change_huge_pud(struct mmu_gather *tlb, struct vm_area_struct *vma, + pud_t *pudp, unsigned long addr, pgprot_t newprot, + unsigned long cp_flags) { return 0; } +#endif + #define split_huge_pud(__vma, __pud, __address) \ do { \ pud_t *____pud = (__pud); \ @@ -585,6 +596,19 @@ static inline int next_order(unsigned long *orders, int prev) { return 0; } + +static inline void __split_huge_pud(struct vm_area_struct *vma, pud_t *pud, + unsigned long address) +{ +} + +static inline int change_huge_pud(struct mmu_gather *tlb, + struct vm_area_struct *vma, pud_t *pudp, + unsigned long addr, pgprot_t newprot, + unsigned long cp_flags) +{ + return 0; +} #endif /* CONFIG_TRANSPARENT_HUGEPAGE */ static inline int split_folio_to_list_to_order(struct folio *folio, diff --git a/mm/huge_memory.c b/mm/huge_memory.c index 81c5da0708ed..0aafd26d7a53 100644 --- a/mm/huge_memory.c +++ b/mm/huge_memory.c @@ -2114,6 +2114,53 @@ int change_huge_pmd(struct mmu_gather *tlb, struct vm_area_struct *vma, return ret; } +/* + * Returns: + * + * - 0: if pud leaf changed from under us + * - 1: if pud can be skipped + * - HPAGE_PUD_NR: if pud was successfully processed + */ +#ifdef CONFIG_HAVE_ARCH_TRANSPARENT_HUGEPAGE_PUD +int change_huge_pud(struct mmu_gather *tlb, struct vm_area_struct *vma, + pud_t *pudp, unsigned long addr, pgprot_t newprot, + unsigned long cp_flags) +{ + struct mm_struct *mm = vma->vm_mm; + pud_t oldpud, entry; + spinlock_t *ptl; + + tlb_change_page_size(tlb, HPAGE_PUD_SIZE); + + /* NUMA balancing doesn't apply to dax */ + if (cp_flags & MM_CP_PROT_NUMA) + return 1; + + /* + * Huge entries on userfault-wp only works with anonymous, while we + * don't have anonymous PUDs yet. + */ + if (WARN_ON_ONCE(cp_flags & MM_CP_UFFD_WP_ALL)) + return 1; + + ptl = __pud_trans_huge_lock(pudp, vma); + if (!ptl) + return 0; + + /* + * Can't clear PUD or it can race with concurrent zapping. See + * change_huge_pmd(). + */ + oldpud = pudp_invalidate(vma, addr, pudp); + entry = pud_modify(oldpud, newprot); + set_pud_at(mm, addr, pudp, entry); + tlb_flush_pud_range(tlb, addr, HPAGE_PUD_SIZE); + + spin_unlock(ptl); + return HPAGE_PUD_NR; +} +#endif + #ifdef CONFIG_USERFAULTFD /* * The PT lock for src_pmd and dst_vma/src_vma (for reading) are locked by @@ -2344,6 +2391,11 @@ void __split_huge_pud(struct vm_area_struct *vma, pud_t *pud, spin_unlock(ptl); mmu_notifier_invalidate_range_end(&range); } +#else +void __split_huge_pud(struct vm_area_struct *vma, pud_t *pud, + unsigned long address) +{ +} #endif /* CONFIG_HAVE_ARCH_TRANSPARENT_HUGEPAGE_PUD */ static void __split_huge_zero_page_pmd(struct vm_area_struct *vma, diff --git a/mm/mprotect.c b/mm/mprotect.c index d423080e6509..446f8e5f10d9 100644 --- a/mm/mprotect.c +++ b/mm/mprotect.c @@ -302,8 +302,9 @@ pgtable_split_needed(struct vm_area_struct *vma, unsigned long cp_flags) { /* * pte markers only resides in pte level, if we need pte markers, - * we need to split. We cannot wr-protect shmem thp because file - * thp is handled differently when split by erasing the pmd so far. + * we need to split. For example, we cannot wr-protect a file thp + * (e.g. 2M shmem) because file thp is handled differently when + * split by erasing the pmd so far. */ return (cp_flags & MM_CP_UFFD_WP) && !vma_is_anonymous(vma); } @@ -430,31 +431,53 @@ static inline long change_pud_range(struct mmu_gather *tlb, unsigned long end, pgprot_t newprot, unsigned long cp_flags) { struct mmu_notifier_range range; - pud_t *pud; + pud_t *pudp, pud; unsigned long next; long pages = 0, ret; range.start = 0; - pud = pud_offset(p4d, addr); + pudp = pud_offset(p4d, addr); do { +again: next = pud_addr_end(addr, end); - ret = change_prepare(vma, pud, pmd, addr, cp_flags); + ret = change_prepare(vma, pudp, pmd, addr, cp_flags); if (ret) { pages = ret; break; } - if (pud_none_or_clear_bad(pud)) + + pud = READ_ONCE(*pudp); + if (pud_none(pud)) continue; + if (!range.start) { mmu_notifier_range_init(&range, MMU_NOTIFY_PROTECTION_VMA, 0, vma->vm_mm, addr, end); mmu_notifier_invalidate_range_start(&range); } - pages += change_pmd_range(tlb, vma, pud, addr, next, newprot, + + if (pud_leaf(pud)) { + if ((next - addr != PUD_SIZE) || + pgtable_split_needed(vma, cp_flags)) { + __split_huge_pud(vma, pudp, addr); + goto again; + } else { + ret = change_huge_pud(tlb, vma, pudp, + addr, newprot, cp_flags); + if (ret == 0) + goto again; + /* huge pud was handled */ + if (ret == HPAGE_PUD_NR) + pages += HPAGE_PUD_NR; + continue; + } + } + + pages += change_pmd_range(tlb, vma, pudp, addr, next, newprot, cp_flags); - } while (pud++, addr = next, addr != end); + } while (pudp++, addr = next, addr != end); if (range.start) mmu_notifier_invalidate_range_end(&range);