From patchwork Mon Jul 15 19:21:35 2024 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Peter Xu X-Patchwork-Id: 13733831 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from kanga.kvack.org (kanga.kvack.org [205.233.56.17]) by smtp.lore.kernel.org (Postfix) with ESMTP id B418FC3DA61 for ; Mon, 15 Jul 2024 19:21:55 +0000 (UTC) Received: by kanga.kvack.org (Postfix) id 000CD6B0095; Mon, 15 Jul 2024 15:21:55 -0400 (EDT) Received: by kanga.kvack.org (Postfix, from userid 40) id EF3BD6B0096; Mon, 15 Jul 2024 15:21:54 -0400 (EDT) X-Delivered-To: int-list-linux-mm@kvack.org Received: by kanga.kvack.org (Postfix, from userid 63042) id CD0856B0098; Mon, 15 Jul 2024 15:21:54 -0400 (EDT) X-Delivered-To: linux-mm@kvack.org Received: from relay.hostedemail.com (smtprelay0011.hostedemail.com [216.40.44.11]) by kanga.kvack.org (Postfix) with ESMTP id B06466B0095 for ; Mon, 15 Jul 2024 15:21:54 -0400 (EDT) Received: from smtpin18.hostedemail.com (a10.router.float.18 [10.200.18.1]) by unirelay06.hostedemail.com (Postfix) with ESMTP id 50385A5306 for ; Mon, 15 Jul 2024 19:21:54 +0000 (UTC) X-FDA: 82342956948.18.28DCA59 Received: from us-smtp-delivery-124.mimecast.com (us-smtp-delivery-124.mimecast.com [170.10.129.124]) by imf24.hostedemail.com (Postfix) with ESMTP id C1E8B180015 for ; Mon, 15 Jul 2024 19:21:51 +0000 (UTC) Authentication-Results: imf24.hostedemail.com; dkim=pass header.d=redhat.com header.s=mimecast20190719 header.b=IXILu6E5; dmarc=pass (policy=none) header.from=redhat.com; spf=pass (imf24.hostedemail.com: domain of peterx@redhat.com designates 170.10.129.124 as permitted sender) smtp.mailfrom=peterx@redhat.com ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=hostedemail.com; s=arc-20220608; t=1721071269; h=from:from:sender:reply-to:subject:subject:date:date: message-id:message-id:to:to:cc:cc:mime-version:mime-version: content-type:content-type: content-transfer-encoding:content-transfer-encoding: in-reply-to:in-reply-to:references:references:dkim-signature; bh=xrFUBbcc9D0ORnx0qoNz81Y3Ewmb+xGo+hBykZs/BiM=; b=NV/v7336NL1QyCmZu205ELof6o/ap61NbM2G3FIN/vmFU+r2rtfUdk3U8AGgSBNKktn6Kh 35Dc1F1EwFIqTS336cyYPhGy/hxGQi1rnrG1IElcalycbyCF51VMYzpeCtII+slbtMHnez ECHTZHboarok/tPnozqOsabThQbGaaU= ARC-Seal: i=1; s=arc-20220608; d=hostedemail.com; t=1721071269; a=rsa-sha256; cv=none; b=Ypn6K6L07g6E9HeGWWwkvPcEyrKk4cyifm6F5RNJ2DdyW5RvhPBA5prMmYjNECCSjcG8g5 qZgCg4UNLZ7BPG9wiINc6BYm1KS1K3uTlAZqjC3IlHq3bxssqYIgUYnKg469M/oPxEKum3 zZeF7uaPk+1grx/rKrcXJbzlb3eSy7s= ARC-Authentication-Results: i=1; imf24.hostedemail.com; dkim=pass header.d=redhat.com header.s=mimecast20190719 header.b=IXILu6E5; dmarc=pass (policy=none) header.from=redhat.com; spf=pass (imf24.hostedemail.com: domain of peterx@redhat.com designates 170.10.129.124 as permitted sender) smtp.mailfrom=peterx@redhat.com DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=redhat.com; s=mimecast20190719; t=1721071311; h=from:from:reply-to:subject:subject:date:date:message-id:message-id: to:to:cc:cc:mime-version:mime-version:content-type:content-type: content-transfer-encoding:content-transfer-encoding: in-reply-to:in-reply-to:references:references; bh=xrFUBbcc9D0ORnx0qoNz81Y3Ewmb+xGo+hBykZs/BiM=; b=IXILu6E5x3wA5zsonL8tx+nw5zAbj/4h66MD4lGDFIENTvebrBRz4jjQzyKFKuaNNLLP4t z0F5r9JRnur8+KOJJGg1UX6SZ89Ye4sazvhEy6fj7WEhtvR/qzErXuvK1hyaL/lMyRV4Df svhn6r90gjDiG8R9lfd07sa4gH0grXc= Received: from mail-qv1-f71.google.com (mail-qv1-f71.google.com [209.85.219.71]) by relay.mimecast.com with ESMTP with STARTTLS (version=TLSv1.3, cipher=TLS_AES_256_GCM_SHA384) id us-mta-343-V5CbpbQNM220fEYiYNIGcg-1; Mon, 15 Jul 2024 15:21:49 -0400 X-MC-Unique: V5CbpbQNM220fEYiYNIGcg-1 Received: by mail-qv1-f71.google.com with SMTP id 6a1803df08f44-6b5e99c06efso9781296d6.2 for ; Mon, 15 Jul 2024 12:21:49 -0700 (PDT) X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20230601; t=1721071308; x=1721676108; h=content-transfer-encoding:mime-version:references:in-reply-to :message-id:date:subject:cc:to:from:x-gm-message-state:from:to:cc :subject:date:message-id:reply-to; bh=xrFUBbcc9D0ORnx0qoNz81Y3Ewmb+xGo+hBykZs/BiM=; b=lB7/NjSs0VoFyiPbPtZWqV8urjReNwTISpQWnIqN7j8sa1TugNXX/qfaP96Ecbhr4b 2XlSriDU49YYZqRTigjGwF1iZK04hftZbPBH3VXh/qZifQsVjeYjdaoB1Vz+FxvG0fxS SBv31Ew0epRTIaqnOQ87JsEPpVlLGhI0Hvh+BirUR4d1gl3riw0O+ZPEaaFqBrH2rZqf 5gJOAqFKxmz25YMbBkDYA3rjT6vpaih+Go7UZIEoDrQhKAkXzVi4G93Hj8BJZwNjfQhb CxnzbwZmbOHoK4fRps69RLWL+BeuIg5dHvTbEKLpn25ap9eLQDv3gNpkBLXCNPNr3EZL b/CQ== X-Gm-Message-State: AOJu0Yz5nwaw07a1ukpR/xLRHe7vrwRgFPOBaY1pgc3gqz6QPjvFQ3OR LEeaxq9DdPKuk3dxYfCliWEyBpRrpoIcATmyvSXT4MEPvzlp/qRdDeW0K0j4orOuIN4HhxTEyTb vGgjGXGN4EpbRgtH150MbEfsa+pPljwPgQ1eglhaxkpHMZN5nkFpbvZIebYup5NBL3nh4oWrWeC wg/kE/Bv6ZexiRbcdOHtehUEyNcegxUQ== X-Received: by 2002:a05:6214:3209:b0:6b7:64a2:3189 with SMTP id 6a1803df08f44-6b77e17f47fmr4877996d6.8.1721071308507; Mon, 15 Jul 2024 12:21:48 -0700 (PDT) X-Google-Smtp-Source: AGHT+IFchV1JB+E55nHTPY9KJJNPGb9+S7LV2RfQetOl80mLgqA/kra292Zxs5L4jIR6tRho3Ee1yQ== X-Received: by 2002:a05:6214:3209:b0:6b7:64a2:3189 with SMTP id 6a1803df08f44-6b77e17f47fmr4877546d6.8.1721071308060; Mon, 15 Jul 2024 12:21:48 -0700 (PDT) Received: from x1n.redhat.com (pool-99-254-121-117.cpe.net.cable.rogers.com. [99.254.121.117]) by smtp.gmail.com with ESMTPSA id 6a1803df08f44-6b761978d30sm24039356d6.31.2024.07.15.12.21.46 (version=TLS1_3 cipher=TLS_AES_256_GCM_SHA384 bits=256/256); Mon, 15 Jul 2024 12:21:47 -0700 (PDT) From: Peter Xu To: linux-mm@kvack.org, linux-kernel@vger.kernel.org Cc: Dave Jiang , Rik van Riel , Dave Hansen , Michael Ellerman , linuxppc-dev@lists.ozlabs.org, Matthew Wilcox , Rick P Edgecombe , peterx@redhat.com, Oscar Salvador , Mel Gorman , Andrew Morton , Borislav Petkov , Christophe Leroy , Huang Ying , "Kirill A . Shutemov" , "Aneesh Kumar K . V" , Dan Williams , Thomas Gleixner , Hugh Dickins , x86@kernel.org, Nicholas Piggin , Vlastimil Babka , Ingo Molnar Subject: [PATCH v3 1/8] mm/dax: Dump start address in fault handler Date: Mon, 15 Jul 2024 15:21:35 -0400 Message-ID: <20240715192142.3241557-2-peterx@redhat.com> X-Mailer: git-send-email 2.45.0 In-Reply-To: <20240715192142.3241557-1-peterx@redhat.com> References: <20240715192142.3241557-1-peterx@redhat.com> MIME-Version: 1.0 X-Mimecast-Spam-Score: 0 X-Mimecast-Originator: redhat.com X-Rspamd-Server: rspam07 X-Rspamd-Queue-Id: C1E8B180015 X-Stat-Signature: zyrrp46e9whghyu1pk8fi31e6p1apmw9 X-Rspam-User: X-HE-Tag: 1721071311-336124 X-HE-Meta: U2FsdGVkX18ym9eX29/iJj6dYpZjMVBaj3KoMcAu8BwtDA6IkxffjTmy7BxTCtkyXkTfQFmpB7gyo7iPy34UUvAYRdIoIgr/FfqUCbMmBiZ10cEAVO0EpJqq2DwA5vHY0gWBVbuoRpOqPFEYEFU/yoobWWSW+fB3lIjEX7+EaNcGjF+7M/uqveO/YlyKbbucqeI7Q59ul2Gie2D+Rm1c4HN1Kq0xe7TvlCxxVUR9OBKTdHHXcy8FzwQKqxWYkmJ7zC8tHeftTO4X8ssz4SZCPHCNOrGmgzgeuIEhOKkPhpD4nsDcRj4TOuVZqZC4HHXjGuomRsXe0/mDvurAh9AP9tMXC8DGEtOeY6rHKCmY08jiHCPrVL4Q1ir6pqDrYzG1TvMX+XH0Ej8yI46igSYPmfNbBpdXJ2oeOvNaROS86BNnM2o1lFQYALkkdyPLWuLKB2rLUtGS0hImr12MIuPNagCTkYEryCvUJ0Ixe6APv1W6c1EgolwKg5Y+Upz9DkabNT8lgZUpyfTxyi466m/Jj8vXB1QuOPeMRkDStf7RRPoocPylkiwOh5ot8wDUozlZNp8lR/6sp8pU9qrq+phU4FVpEYsjHsyf2X/OLieA9aHtIZ2d0k3SVapzVNsyD4q+PifmOGVGxg5MlB/25uGzXNDiPfWNfqdUJ/mw+YB5z2+A2YbDa2/dR8pbSKc8vIfAoiiodBcACfL5a1bRol30B4x+Ocuti7Xu/2t+hUvMqk14bNG9+wVOdXXp5KLOz8ukWycKrP5ntFFxyX5BHbBiEBuVGHLwA6K6UpRSwab2orUuGApNVyM0i7JYkNv00yzms4U/VpVmqq2QJUTss0PU1UVsK+Yt+Y2dBsPm6sBURfSVSaYC+MzKl3kXCHCtOOE84aB4IBRZBSt8uphyMz/ZRGAvX3ICQhpzag9wfk4wMBQqK82ygqUf8hwifsRtp8Q1BQT0YA9xUd9UorCw1d+ jAGSX4t3 qL7zEOEWXH0zfxS2dbzUCswa++5U6F8FcEsIeN/g9lkoL3ajgdtiBCYsJMQ67waXtvjCLHazDB1jCbQ3Zbc1MgneEHvHTxEzzfr0RmsFhWjfEMSh12ySQcmFU+m84JTNG602rN3cy8yfnegF03mi8LwXQICofbUw/u0VoC1t/qY+8Xhke1JOpffk+Mmh0o5I6QqHUZUR3JB30Wdf94zaydhsIy4/Eb+8upKFFrx/uI5GLvo96mQNXiCUpW056qmgOROErU5oMF9GXqPJb9IODZLlKHCvAWk4CIb+V X-Bogosity: Ham, tests=bogofilter, spamicity=0.000000, version=1.2.4 Sender: owner-linux-mm@kvack.org Precedence: bulk X-Loop: owner-majordomo@kvack.org List-ID: List-Subscribe: List-Unsubscribe: Currently the dax fault handler dumps the vma range when dynamic debugging enabled. That's mostly not useful. Dump the (aligned) address instead with the order info. Signed-off-by: Peter Xu Acked-by: David Hildenbrand --- drivers/dax/device.c | 6 +++--- 1 file changed, 3 insertions(+), 3 deletions(-) diff --git a/drivers/dax/device.c b/drivers/dax/device.c index eb61598247a9..714174844ca5 100644 --- a/drivers/dax/device.c +++ b/drivers/dax/device.c @@ -235,9 +235,9 @@ static vm_fault_t dev_dax_huge_fault(struct vm_fault *vmf, unsigned int order) int id; struct dev_dax *dev_dax = filp->private_data; - dev_dbg(&dev_dax->dev, "%s: %s (%#lx - %#lx) order:%d\n", current->comm, - (vmf->flags & FAULT_FLAG_WRITE) ? "write" : "read", - vmf->vma->vm_start, vmf->vma->vm_end, order); + dev_dbg(&dev_dax->dev, "%s: op=%s addr=%#lx order=%d\n", current->comm, + (vmf->flags & FAULT_FLAG_WRITE) ? "write" : "read", + vmf->address & ~((1UL << (order + PAGE_SHIFT)) - 1), order); id = dax_read_lock(); if (order == 0) From patchwork Mon Jul 15 19:21:36 2024 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Peter Xu X-Patchwork-Id: 13733833 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from kanga.kvack.org (kanga.kvack.org [205.233.56.17]) by smtp.lore.kernel.org (Postfix) with ESMTP id CF549C3DA62 for ; Mon, 15 Jul 2024 19:21:58 +0000 (UTC) Received: by kanga.kvack.org (Postfix) id 41F746B0096; Mon, 15 Jul 2024 15:21:58 -0400 (EDT) Received: by kanga.kvack.org (Postfix, from userid 40) id 3A8CA6B0099; Mon, 15 Jul 2024 15:21:58 -0400 (EDT) X-Delivered-To: int-list-linux-mm@kvack.org Received: by kanga.kvack.org (Postfix, from userid 63042) id 221E86B009A; Mon, 15 Jul 2024 15:21:58 -0400 (EDT) X-Delivered-To: linux-mm@kvack.org Received: from relay.hostedemail.com (smtprelay0017.hostedemail.com [216.40.44.17]) by kanga.kvack.org (Postfix) with ESMTP id 052E36B0096 for ; Mon, 15 Jul 2024 15:21:57 -0400 (EDT) Received: from smtpin10.hostedemail.com (a10.router.float.18 [10.200.18.1]) by unirelay07.hostedemail.com (Postfix) with ESMTP id B09EB16020D for ; Mon, 15 Jul 2024 19:21:57 +0000 (UTC) X-FDA: 82342957074.10.8B65B71 Received: from us-smtp-delivery-124.mimecast.com (us-smtp-delivery-124.mimecast.com [170.10.133.124]) by imf18.hostedemail.com (Postfix) with ESMTP id A9B261C001F for ; Mon, 15 Jul 2024 19:21:53 +0000 (UTC) Authentication-Results: imf18.hostedemail.com; dkim=pass header.d=redhat.com header.s=mimecast20190719 header.b="OIoH/AOi"; dmarc=pass (policy=none) header.from=redhat.com; spf=pass (imf18.hostedemail.com: domain of peterx@redhat.com designates 170.10.133.124 as permitted sender) smtp.mailfrom=peterx@redhat.com ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=hostedemail.com; s=arc-20220608; t=1721071271; h=from:from:sender:reply-to:subject:subject:date:date: message-id:message-id:to:to:cc:cc:mime-version:mime-version: content-type:content-type: content-transfer-encoding:content-transfer-encoding: in-reply-to:in-reply-to:references:references:dkim-signature; bh=uDSHiwJLCIYvvzv2DOw1kewa0irBZ6HqHLZp2ORNUK8=; b=c/nRKj2W/545ko3nXsbiL/TvCRUQz/JXNi8+54jXzZGOmPTyxJ0DVR2pTATJIH3raIlKMx PjzkkUQEsB2z1jYTv4YRZL8FnBKFAf4K61OSEAYyLYaex7Gc0Y1LyZk7vH+BkNq7bEPYK9 pU1CHInIaJ3KgmpEjegRp3JNukzbIRc= ARC-Seal: i=1; s=arc-20220608; d=hostedemail.com; t=1721071271; a=rsa-sha256; cv=none; b=t4RsyckG0CA3avfM0Zcbusk/cqYvhVF3Zz76+AocgbPHd0+fmDQAkqCoM/HAq+ZE+J3mQH B7zKY+YIwxXo0KNgQ3CLqnlV1GlrhdXFbaaUHFhiJGVUNfwY9kk866VFEEgbfD1QW3A76I tDuDBasN8uIoxnWcQROIY5uR99o+f9c= ARC-Authentication-Results: i=1; imf18.hostedemail.com; dkim=pass header.d=redhat.com header.s=mimecast20190719 header.b="OIoH/AOi"; dmarc=pass (policy=none) header.from=redhat.com; spf=pass (imf18.hostedemail.com: domain of peterx@redhat.com designates 170.10.133.124 as permitted sender) smtp.mailfrom=peterx@redhat.com DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=redhat.com; s=mimecast20190719; t=1721071313; h=from:from:reply-to:subject:subject:date:date:message-id:message-id: to:to:cc:cc:mime-version:mime-version:content-type:content-type: content-transfer-encoding:content-transfer-encoding: in-reply-to:in-reply-to:references:references; bh=uDSHiwJLCIYvvzv2DOw1kewa0irBZ6HqHLZp2ORNUK8=; b=OIoH/AOiD3qYfJfwY44rEsCVdJMdanjLQczMJJVZPohWBgYl42+Cx/9BlJsAGASrEQAS6N B0CFIouvagTF6M7bR01iRf0EOH/ssQzFByKNEkoq33J6tRkFd+V+cagpeNZ7zt+6XpnkSS 1HpZoFQVe/x0ob8hM2yhEa/I0PYRD8A= Received: from mail-qv1-f69.google.com (mail-qv1-f69.google.com [209.85.219.69]) by relay.mimecast.com with ESMTP with STARTTLS (version=TLSv1.3, cipher=TLS_AES_256_GCM_SHA384) id us-mta-136-phhHxa1VOi6L7hToVnuusQ-1; Mon, 15 Jul 2024 15:21:51 -0400 X-MC-Unique: phhHxa1VOi6L7hToVnuusQ-1 Received: by mail-qv1-f69.google.com with SMTP id 6a1803df08f44-6b5e41e09b6so6993436d6.1 for ; Mon, 15 Jul 2024 12:21:51 -0700 (PDT) X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20230601; t=1721071311; x=1721676111; h=content-transfer-encoding:mime-version:references:in-reply-to :message-id:date:subject:cc:to:from:x-gm-message-state:from:to:cc :subject:date:message-id:reply-to; bh=uDSHiwJLCIYvvzv2DOw1kewa0irBZ6HqHLZp2ORNUK8=; b=RDCZLpyfYjFvgc0JJFUPz9XpUwBKXYZHxa6ps5EBR7hUQUru4xnvZlWZdpTHd++pgK t0SM8RnxORFut4DU3KLsQqbvejq66PidURmkHCbKZIO6ZjpylrFKlZa8xrJyd6elplg6 XrHvnKyeJDAxkm42gum48FiyMNRi0aHKRlCViXhwcBgeTvcIhzZ5gW8Wr1pT3tv5poEn 1u1JOII6AWmhHKUlK0axT4DFx4QebOqyNPdlQeowY488T1rQ4ntL3LUqLp/xmOCJHJuA d3XhQo55ADPrcXcHleYuTwN8oIjuToX6z/P+pwJQAdb5VJLDlfU2C8IfM0Sj+v73vG+V cCsg== X-Gm-Message-State: AOJu0YyNrndOBc2Uowl5k0NJEzp7mlpzZdkDqtjMT0LWrPTyjDdvRDEa fnDlvahjp8swY49XOXtARpHlQL1hdu+kltEE7Y00aA7Ijctlidx94ClHaT+p7GGzma6LegIH9hJ kmd0S23rcZhIgyFlTim63TFi1tr+DRqUXPeca+jKtnqPQyuxOXvAmCE4KRp69ghWCJQbBEbTsIf qnASb8Am7PeF2WQDW1rltVf5kMMNt6AA== X-Received: by 2002:a05:6214:2d44:b0:6b5:4bf:e0b2 with SMTP id 6a1803df08f44-6b77df230d0mr4738446d6.6.1721071310748; Mon, 15 Jul 2024 12:21:50 -0700 (PDT) X-Google-Smtp-Source: AGHT+IHXNZWcX+OGChgQMRYaFRNal2nxEfRlz+I/X26JLEslV+UpQEodY5JY9Fmq0XZviJ4grHRk9w== X-Received: by 2002:a05:6214:2d44:b0:6b5:4bf:e0b2 with SMTP id 6a1803df08f44-6b77df230d0mr4738076d6.6.1721071310396; Mon, 15 Jul 2024 12:21:50 -0700 (PDT) Received: from x1n.redhat.com (pool-99-254-121-117.cpe.net.cable.rogers.com. [99.254.121.117]) by smtp.gmail.com with ESMTPSA id 6a1803df08f44-6b761978d30sm24039356d6.31.2024.07.15.12.21.48 (version=TLS1_3 cipher=TLS_AES_256_GCM_SHA384 bits=256/256); Mon, 15 Jul 2024 12:21:49 -0700 (PDT) From: Peter Xu To: linux-mm@kvack.org, linux-kernel@vger.kernel.org Cc: Dave Jiang , Rik van Riel , Dave Hansen , Michael Ellerman , linuxppc-dev@lists.ozlabs.org, Matthew Wilcox , Rick P Edgecombe , peterx@redhat.com, Oscar Salvador , Mel Gorman , Andrew Morton , Borislav Petkov , Christophe Leroy , Huang Ying , "Kirill A . Shutemov" , "Aneesh Kumar K . V" , Dan Williams , Thomas Gleixner , Hugh Dickins , x86@kernel.org, Nicholas Piggin , Vlastimil Babka , Ingo Molnar , Alex Thorlton Subject: [PATCH v3 2/8] mm/mprotect: Remove NUMA_HUGE_PTE_UPDATES Date: Mon, 15 Jul 2024 15:21:36 -0400 Message-ID: <20240715192142.3241557-3-peterx@redhat.com> X-Mailer: git-send-email 2.45.0 In-Reply-To: <20240715192142.3241557-1-peterx@redhat.com> References: <20240715192142.3241557-1-peterx@redhat.com> MIME-Version: 1.0 X-Mimecast-Spam-Score: 0 X-Mimecast-Originator: redhat.com X-Rspamd-Server: rspam07 X-Rspamd-Queue-Id: A9B261C001F X-Stat-Signature: tpef3dz91ron9k6z8ccsm9rtrj7593ij X-Rspam-User: X-HE-Tag: 1721071313-733515 X-HE-Meta: U2FsdGVkX1/6Ii3sFNVwtyE4G8lJRQIhFLO+4IHq+UNKaO2hN+EnevvVP3ERMp52QuF4aA2XNBg5LeAF/XF+rOuw/4O4DgCpPPllNa0HkndZDjc64/JaorkzcLqCwlyFWojSeQelHtFUfv+Uf5jaCuNP76CgD2s9yyOIP3ehyTv4ywhhCRREnnLykIu0SLB/zidNogz5FswwGiVysKRLjyx+Eiy/ofkqWRR9MR1JdBoXrmi0NPtX5S5Y0MZIChY3sIz1Sv6Gu/OQNechp5YQLkn7ItUo8r6OXDsgCK2QRcHm5W+YULdR4HLM7MBjVKD3kc4v/ow8Vhet3kX8Jg5dL3LB8dSFn8UPmV73wl082SaKcr/w66xoisNEQKQYH/hbkY7KopNhYl92q0f0H8pq9Ej0AhZ58IGdsbNRMrS353Ed4Bz+m3/ZEB/X33cFnn1QuWcnyvVj3r/Q2lH40BUvhmVGFDsScvM+ugpVtj/mOIH/trz79YPqJzH8qOUwlN+BIcePp2rfTCaNMjsQpdwqPo7fYUbpgZDBob3zYzyHd4/El5NPVjWIJpbd8UjSqJ/v3FQjDmdOFTGIeRTAt5SrCROw0xVlvBowrIZkFhCy6BSQGH10+WWmxBvmrCEy1e74+ByB8c24OpBnwBxEg9IIwmN8vfkTZVhcaDSjAA41QfRhtK4TLVB/4r/7mFqsFFhTBRiH5mV0Tei2Un+ySb1V7HbnuFNUaowO/66463klw/r3q6Y7etLSDcf/QKkxGfF2qrvsQ9eEJuGSbUW1q4wx4DXv4HnHcEt7FWCdWpMdCfnjeRHPjSkkhDPMvy5wWAUgtfoKxmWrACXQ9Gv7w8CWRA8ClZLDPMzLN/CDNqfDql7a60BTbpBvWcPfmcfRXKClTZ3f+o8PQp+bVZHnqPJcdz6ulK7xqVjGW8It6mD0XcGBIE5HqP7ZZCwmSqDFXOjvJBv1T69vFz7PdU1eNCI 1PMpja3+ bAVvePaNLDLwCjWmoCtdplkVRRRFcffC1eaFUGRtY3ylVgU08pFweOhT9sunQGkwOIrgaeoycj8earuYcYzC7HvTWifKeshtcJaVY39UaBurscm/5d96J3HgnvPgR3g7JVExo7aSuxzdf+1mSz39dsgOXBWZXGAsd+o4Xget6wiMOpzCKNDMM7OnOERddl1LRpuXfGmXgOiB7p0MqVQEgCWbYigMpUqtf8x6iUZgq4DwI7HsKjHDxi3m+0wR+YtVOFLw6O2o0UVRzp4sumSVfQ9+WwaWXuMGBa278JxzbkiGrHmSDn1IY8CgWd9Bhm7XLgSfWzApzYL8DdyISOazriclc8QfmjAdSfQPiMJoB9cLgKMLIGV5xrUsw8HKsSr6Pnu76elQCDwHwOyWOMJSr4SgwLLV0vnbDzj8/6DkuVpoxzub10rTer8cnY6BoRV5ycMkAtrw7vYqTIleMcGTnyNlhp3lMp8AYk3yzS551r4saooC5SY5FLF6fKVxozxyYb02mAMmxTJaTgazXpzorZP/oKh+jLBw5f/3lcemf07wzPGfJA7TyYlLl7pD4jwBPkOLX7Cu+Jp4/vKWHrLueroyfLL3z7LfHSopbteDhVhqLeKZAWrOS6+Zk0CyC3mPNZgMs8APmi0u/pZrKf8AoFoZMBQuMJZzZj/33 X-Bogosity: Ham, tests=bogofilter, spamicity=0.000000, version=1.2.4 Sender: owner-linux-mm@kvack.org Precedence: bulk X-Loop: owner-majordomo@kvack.org List-ID: List-Subscribe: List-Unsubscribe: In 2013, commit 72403b4a0fbd ("mm: numa: return the number of base pages altered by protection changes") introduced "numa_huge_pte_updates" vmstat entry, trying to capture how many huge ptes (in reality, PMD thps at that time) are marked by NUMA balancing. This patch proposes to remove it for some reasons. Firstly, the name is misleading. We can have more than one way to have a "huge pte" at least nowadays, and that's also the major goal of this patch, where it paves way for PUD handling in change protection code paths. PUDs are coming not only for dax (which has already came and yet broken..), but also for pfnmaps and hugetlb pages. The name will simply stop making sense when PUD will start to be involved in mprotect() world. It'll also make it not reasonable either if we boost the counter for both pmd/puds. In short, current accounting won't be right when PUD comes, so the scheme was only suitable at that point in time where PUD wasn't even possible. Secondly, the accounting was simply not right from the start as long as it was also affected by other call sites besides NUMA. mprotect() is one, while userfaultfd-wp also leverages change protection path to modify pgtables. If it wants to do right it needs to check the caller but it never did; at least mprotect() should be there even in 2013. It gives me the impression that nobody is seriously using this field, and it's also impossible to be serious. We may want to do it right if any NUMA developers would like it to exist, but we should do that with all above resolved, on both considering PUDs, but also on correct accountings. That should be able to be done on top when there's a real need of such. Cc: Huang Ying Cc: Mel Gorman Cc: Alex Thorlton Cc: Rik van Riel Signed-off-by: Peter Xu --- include/linux/vm_event_item.h | 1 - mm/mprotect.c | 8 +------- mm/vmstat.c | 1 - 3 files changed, 1 insertion(+), 9 deletions(-) diff --git a/include/linux/vm_event_item.h b/include/linux/vm_event_item.h index 747943bc8cc2..2a3797fb6742 100644 --- a/include/linux/vm_event_item.h +++ b/include/linux/vm_event_item.h @@ -59,7 +59,6 @@ enum vm_event_item { PGPGIN, PGPGOUT, PSWPIN, PSWPOUT, OOM_KILL, #ifdef CONFIG_NUMA_BALANCING NUMA_PTE_UPDATES, - NUMA_HUGE_PTE_UPDATES, NUMA_HINT_FAULTS, NUMA_HINT_FAULTS_LOCAL, NUMA_PAGE_MIGRATE, diff --git a/mm/mprotect.c b/mm/mprotect.c index 222ab434da54..21172272695e 100644 --- a/mm/mprotect.c +++ b/mm/mprotect.c @@ -363,7 +363,6 @@ static inline long change_pmd_range(struct mmu_gather *tlb, pmd_t *pmd; unsigned long next; long pages = 0; - unsigned long nr_huge_updates = 0; struct mmu_notifier_range range; range.start = 0; @@ -411,11 +410,8 @@ static inline long change_pmd_range(struct mmu_gather *tlb, ret = change_huge_pmd(tlb, vma, pmd, addr, newprot, cp_flags); if (ret) { - if (ret == HPAGE_PMD_NR) { + if (ret == HPAGE_PMD_NR) pages += HPAGE_PMD_NR; - nr_huge_updates++; - } - /* huge pmd was handled */ goto next; } @@ -435,8 +431,6 @@ static inline long change_pmd_range(struct mmu_gather *tlb, if (range.start) mmu_notifier_invalidate_range_end(&range); - if (nr_huge_updates) - count_vm_numa_events(NUMA_HUGE_PTE_UPDATES, nr_huge_updates); return pages; } diff --git a/mm/vmstat.c b/mm/vmstat.c index 73d791d1caad..53656227f70d 100644 --- a/mm/vmstat.c +++ b/mm/vmstat.c @@ -1313,7 +1313,6 @@ const char * const vmstat_text[] = { #ifdef CONFIG_NUMA_BALANCING "numa_pte_updates", - "numa_huge_pte_updates", "numa_hint_faults", "numa_hint_faults_local", "numa_pages_migrated", From patchwork Mon Jul 15 19:21:37 2024 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Peter Xu X-Patchwork-Id: 13733834 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from kanga.kvack.org (kanga.kvack.org [205.233.56.17]) by smtp.lore.kernel.org (Postfix) with ESMTP id 45952C3DA4B for ; Mon, 15 Jul 2024 19:22:01 +0000 (UTC) Received: by kanga.kvack.org (Postfix) id AD9126B0099; Mon, 15 Jul 2024 15:21:58 -0400 (EDT) Received: by kanga.kvack.org (Postfix, from userid 40) id A86626B009A; Mon, 15 Jul 2024 15:21:58 -0400 (EDT) X-Delivered-To: int-list-linux-mm@kvack.org Received: by kanga.kvack.org (Postfix, from userid 63042) id 8B1856B009C; Mon, 15 Jul 2024 15:21:58 -0400 (EDT) X-Delivered-To: linux-mm@kvack.org Received: from relay.hostedemail.com (smtprelay0012.hostedemail.com [216.40.44.12]) by kanga.kvack.org (Postfix) with ESMTP id 6FAB86B0099 for ; Mon, 15 Jul 2024 15:21:58 -0400 (EDT) Received: from smtpin23.hostedemail.com (a10.router.float.18 [10.200.18.1]) by unirelay05.hostedemail.com (Postfix) with ESMTP id 1F2EC401AE for ; Mon, 15 Jul 2024 19:21:58 +0000 (UTC) X-FDA: 82342957116.23.2958E43 Received: from us-smtp-delivery-124.mimecast.com (us-smtp-delivery-124.mimecast.com [170.10.129.124]) by imf04.hostedemail.com (Postfix) with ESMTP id 0C2DA4001B for ; Mon, 15 Jul 2024 19:21:55 +0000 (UTC) Authentication-Results: imf04.hostedemail.com; dkim=pass header.d=redhat.com header.s=mimecast20190719 header.b=jIUx8xDl; dmarc=pass (policy=none) header.from=redhat.com; spf=pass (imf04.hostedemail.com: domain of peterx@redhat.com designates 170.10.129.124 as permitted sender) smtp.mailfrom=peterx@redhat.com ARC-Seal: i=1; s=arc-20220608; d=hostedemail.com; t=1721071285; a=rsa-sha256; cv=none; b=EIYn2rS0zomEfkWfaH7cadeeO90EpjogRk0DhKKv0JjIUODY4pP1vlNRS3Jbo7n7Z6BUKi J4PcSfojfBINVlksQA62zcZvMCuDA3DL6TxjUBefV/AzAiTSSFitybeePXGjWajFmXMIpp 2tnE4KP8hlR8C1wPoNktsYMIjtoF+Y4= ARC-Authentication-Results: i=1; imf04.hostedemail.com; dkim=pass header.d=redhat.com header.s=mimecast20190719 header.b=jIUx8xDl; dmarc=pass (policy=none) header.from=redhat.com; spf=pass (imf04.hostedemail.com: domain of peterx@redhat.com designates 170.10.129.124 as permitted sender) smtp.mailfrom=peterx@redhat.com ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=hostedemail.com; s=arc-20220608; t=1721071285; h=from:from:sender:reply-to:subject:subject:date:date: message-id:message-id:to:to:cc:cc:mime-version:mime-version: content-type:content-type: content-transfer-encoding:content-transfer-encoding: in-reply-to:in-reply-to:references:references:dkim-signature; bh=ZTG40gQ1/IqtnBedMzEchXvuwrXGVRlB32P0aAwFHGI=; b=7fgJlET8NhqBxztA9Lb7+RgooX5oOkUHcDtL35rq5vr8ImgXWVHwH9HBcejWyjcsCsveWa v5C+kX4UCbmqZ2l7w+rbXY+zqSlgX1h+EfEWwGsIZurSgkM06gUC8S3bCRNQi95YfwIqJ7 9GBPYasBMeNuWWPy6FsGZJHDPLPjRMo= DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=redhat.com; s=mimecast20190719; t=1721071315; h=from:from:reply-to:subject:subject:date:date:message-id:message-id: to:to:cc:cc:mime-version:mime-version:content-type:content-type: content-transfer-encoding:content-transfer-encoding: in-reply-to:in-reply-to:references:references; bh=ZTG40gQ1/IqtnBedMzEchXvuwrXGVRlB32P0aAwFHGI=; b=jIUx8xDlPvZx/GdZ/Ox8H53XLFxwOiPzfC545Lqfv3FzUSw3r8PCaRVE8Clzf9GcffERU3 7oI4uFoV6h0xbihvIHefc44c1qtaMDXkhtGPKZh90S2+2biyd5zz5qkqgNjO7S8UrrFgqe Fmnck6UZQaBEuqNVSr4e6UkK7Nf/S1g= Received: from mail-qv1-f71.google.com (mail-qv1-f71.google.com [209.85.219.71]) by relay.mimecast.com with ESMTP with STARTTLS (version=TLSv1.3, cipher=TLS_AES_256_GCM_SHA384) id us-mta-77-Sblj2wwbONmHjurpnYzelw-1; Mon, 15 Jul 2024 15:21:54 -0400 X-MC-Unique: Sblj2wwbONmHjurpnYzelw-1 Received: by mail-qv1-f71.google.com with SMTP id 6a1803df08f44-6b60afbf5d1so14885376d6.0 for ; Mon, 15 Jul 2024 12:21:54 -0700 (PDT) X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20230601; t=1721071313; x=1721676113; h=content-transfer-encoding:mime-version:references:in-reply-to :message-id:date:subject:cc:to:from:x-gm-message-state:from:to:cc :subject:date:message-id:reply-to; bh=ZTG40gQ1/IqtnBedMzEchXvuwrXGVRlB32P0aAwFHGI=; b=Tl13cTQ1vaD5cAAunMRg3vmVt3ptMFKAzMYx3ly17Njph9opo2He3YR5HgLFpViRKO ePEiYZjfTr2IBYtUQrel6yW/V+I9yYlXRyYbeMpn1k5tc9WK9+fg1AEkVX4zX7NntxjS WPpPq8AnMELcJ4DzWIFUch8o3Yuy7jOmkJHHL1FgFgAAtcCDfSlbizJQPT4piejPwBk7 JoPGh+mu9z4EvD6NLiSx+bcnfBXbyYd51FbZQ5x3enAe0nFxjDNr0AoDp97Y8zgdU/aH YVvYCqXCOJ5WIKdhKzq8E+27SjoZy4GTGEVID06epbZ7d0ytRPOjozkrivEju491k6r8 Lkqg== X-Gm-Message-State: AOJu0YyBCqmRSYiFUJBxlUegMxWQ2CxOLduTBkbMai6sU50XtbK4r/AV OtukmzSKc/SRXpIxUlL2iBqMkP7ThbQRnmPNTwOVFptF1mth/v7K6Tgy6uOrQu+IkzyJEjOJ1Xo SDOZZrYBL77TV/Rc4w29u3wxcbBGjAdYnNJmNitsOBXRz3wGNo1PeHQ76TNXN/WVPt6OR6sTCh6 mTD6RTbItTV9fLFz+zVRbAetYzkJ5BEA== X-Received: by 2002:a05:6214:3d9c:b0:6b7:586c:6db with SMTP id 6a1803df08f44-6b77e1a9bbbmr4344596d6.9.1721071313108; Mon, 15 Jul 2024 12:21:53 -0700 (PDT) X-Google-Smtp-Source: AGHT+IFuaD7Bvj1JQVrhIrOGjnKODWs3APjSgAvSuKVxjGp/QvXbyCQZIOrqbPCB4cG0KJw/WyvKgQ== X-Received: by 2002:a05:6214:3d9c:b0:6b7:586c:6db with SMTP id 6a1803df08f44-6b77e1a9bbbmr4344156d6.9.1721071312655; Mon, 15 Jul 2024 12:21:52 -0700 (PDT) Received: from x1n.redhat.com (pool-99-254-121-117.cpe.net.cable.rogers.com. [99.254.121.117]) by smtp.gmail.com with ESMTPSA id 6a1803df08f44-6b761978d30sm24039356d6.31.2024.07.15.12.21.50 (version=TLS1_3 cipher=TLS_AES_256_GCM_SHA384 bits=256/256); Mon, 15 Jul 2024 12:21:52 -0700 (PDT) From: Peter Xu To: linux-mm@kvack.org, linux-kernel@vger.kernel.org Cc: Dave Jiang , Rik van Riel , Dave Hansen , Michael Ellerman , linuxppc-dev@lists.ozlabs.org, Matthew Wilcox , Rick P Edgecombe , peterx@redhat.com, Oscar Salvador , Mel Gorman , Andrew Morton , Borislav Petkov , Christophe Leroy , Huang Ying , "Kirill A . Shutemov" , "Aneesh Kumar K . V" , Dan Williams , Thomas Gleixner , Hugh Dickins , x86@kernel.org, Nicholas Piggin , Vlastimil Babka , Ingo Molnar , kvm@vger.kernel.org, Sean Christopherson , Paolo Bonzini , David Rientjes Subject: [PATCH v3 3/8] mm/mprotect: Push mmu notifier to PUDs Date: Mon, 15 Jul 2024 15:21:37 -0400 Message-ID: <20240715192142.3241557-4-peterx@redhat.com> X-Mailer: git-send-email 2.45.0 In-Reply-To: <20240715192142.3241557-1-peterx@redhat.com> References: <20240715192142.3241557-1-peterx@redhat.com> MIME-Version: 1.0 X-Mimecast-Spam-Score: 0 X-Mimecast-Originator: redhat.com X-Rspamd-Server: rspam12 X-Rspamd-Queue-Id: 0C2DA4001B X-Stat-Signature: tdn5tyhsagdpccdmiyrhpjpzr3mhdwxw X-Rspam-User: X-HE-Tag: 1721071315-688924 X-HE-Meta: U2FsdGVkX1+IqlPO6ZFuGAtaq2/yR/jKXaQ4EzlCRikkBeNxzDNjfq2it5qhRiSoYyIZQUKkBcbgpuey0iwnO+WbLeoqPwk771c+zbNsDGNqu/BuC4DHA9Uv6j6PZGD1VC9fEmQE9zTkQGGThqjO8ICyBYOuvt29w7bO6Tj7oZsAZuUQCfbhte3m2amirut5PA8fmimpCCCvBJh5I0zQJJsnMpKgAM59rRCTqv+QiZYQKVUGbr4jgOxdhONf9L9E1eqTqWfdC9l2DQ7CRJyoyvHX3Sxo8iDcwn9rwZyTSWYytXOvE7dD+uNBJhlBcccclGoaS2nMANI2de/5KPi7vbLRPahPqcF7ykdIzo6YgA26Pa/sLl2NVot08WeH8pQWNcVRr1hnYtE5zwECGwizyUBAysbb+SXVZ7Qrgn0ku0+EFu1Tk8KVXFUblc5y/WMOtioYazIyKY7ZkCGIm5nhbLNTW1IsM6tBwOucrv7z3IlumTRFIlN+uKisDsANFXpOw9hbGeIiexSXSe8pMZc5HpE6o9ooAix0Td4j5IVL5f2D5MGosSnePsnKKbhe5Bv3blRuM0gzrxkRzZlAB8qryP/850BENRAr7Z6aO0iL1invjA5dSujl9LnOZbrjcLTr5K8Zv+EtDCufcRAkZswA8NXajrG+X5cypHTIh6mcg7DM9bH7a8nWmkBO2ZMcTV9HR3UZ8KBlozzIcZCBTMsoC2RRvToKRrPS+Kh5Tw0GJoTBKhK48atkN10MWUzWhXGcb72SYF1fkHpl9bqFkNOebBbIOS5CGIOf3qi60dyAyJquuziutzDFxo//tzfs/GHC1Da8U4hLXdc36pz2aORqbg473yeq+oiJUp6yqgL75v5+ffDwuhtzSSpFGE2hTb6NaFaKuNf+hOq4BV/qb2wkDE8DPGtveZa/9G72wH97NPJrCjtGOGMi3Y8VLTiOXgTR0ROskKQvgCuVJ+UhwwU 9YM7opOd aVfVHyi0AkBTvaxZxcr/RkMyIWe4+OsUavf0igY2aEVa0rg1lLLb9J7w6iKdY87pjJCQ5tTk4db1g2Ea1SH8jBQttB5xYoBzaMUM8LM8yxUXfEWd/Oo5rbz4mP3jHzIOTDTswNxAKa/5UJRxI83qkckkBcOSiEyAm1m+TP2ks9/EmRkyakyj//1bk24VArg5QHPNgPUMZR04OjVjC2IMRebX0f9D5kee2mfMhWQq83qwxvyogx3dXW/PLIgvTE3Wozu3A2gyWFrM0c8IMP3uoSBXWp8ZCFNcqCDm1YOYqtRwyn4bD6ohTQWQAALDyTa0wOuZlcVk4yqPw9S3twnoEwBHG1mtzfJtCspcQJEhEor9wU0lYtOAneu3cABo/lc5JZFiURmlG4DTBrehgHuexKS7n54Ww6fzEKSrayqRl+nqQGJ+9EOOJVtUfpQ== X-Bogosity: Ham, tests=bogofilter, spamicity=0.000000, version=1.2.4 Sender: owner-linux-mm@kvack.org Precedence: bulk X-Loop: owner-majordomo@kvack.org List-ID: List-Subscribe: List-Unsubscribe: mprotect() does mmu notifiers in PMD levels. It's there since 2014 of commit a5338093bfb4 ("mm: move mmu notifier call from change_protection to change_pmd_range"). At that time, the issue was that NUMA balancing can be applied on a huge range of VM memory, even if nothing was populated. The notification can be avoided in this case if no valid pmd detected, which includes either THP or a PTE pgtable page. Now to pave way for PUD handling, this isn't enough. We need to generate mmu notifications even on PUD entries properly. mprotect() is currently broken on PUD (e.g., one can easily trigger kernel error with dax 1G mappings already), this is the start to fix it. To fix that, this patch proposes to push such notifications to the PUD layers. There is risk on regressing the problem Rik wanted to resolve before, but I think it shouldn't really happen, and I still chose this solution because of a few reasons: 1) Consider a large VM that should definitely contain more than GBs of memory, it's highly likely that PUDs are also none. In this case there will have no regression. 2) KVM has evolved a lot over the years to get rid of rmap walks, which might be the major cause of the previous soft-lockup. At least TDP MMU already got rid of rmap as long as not nested (which should be the major use case, IIUC), then the TDP MMU pgtable walker will simply see empty VM pgtable (e.g. EPT on x86), the invalidation of a full empty region in most cases could be pretty fast now, comparing to 2014. 3) KVM has explicit code paths now to even give way for mmu notifiers just like this one, e.g. in commit d02c357e5bfa ("KVM: x86/mmu: Retry fault before acquiring mmu_lock if mapping is changing"). It'll also avoid contentions that may also contribute to a soft-lockup. 4) Stick with PMD layer simply don't work when PUD is there... We need one way or another to fix PUD mappings on mprotect(). Pushing it to PUD should be the safest approach as of now, e.g. there's yet no sign of huge P4D coming on any known archs. Cc: kvm@vger.kernel.org Cc: Sean Christopherson Cc: Paolo Bonzini Cc: David Rientjes Cc: Rik van Riel Signed-off-by: Peter Xu --- mm/mprotect.c | 32 ++++++++++++++++---------------- 1 file changed, 16 insertions(+), 16 deletions(-) diff --git a/mm/mprotect.c b/mm/mprotect.c index 21172272695e..2a81060b603d 100644 --- a/mm/mprotect.c +++ b/mm/mprotect.c @@ -363,9 +363,6 @@ static inline long change_pmd_range(struct mmu_gather *tlb, pmd_t *pmd; unsigned long next; long pages = 0; - struct mmu_notifier_range range; - - range.start = 0; pmd = pmd_offset(pud, addr); do { @@ -383,14 +380,6 @@ static inline long change_pmd_range(struct mmu_gather *tlb, if (pmd_none(*pmd)) goto next; - /* invoke the mmu notifier if the pmd is populated */ - if (!range.start) { - mmu_notifier_range_init(&range, - MMU_NOTIFY_PROTECTION_VMA, 0, - vma->vm_mm, addr, end); - mmu_notifier_invalidate_range_start(&range); - } - _pmd = pmdp_get_lockless(pmd); if (is_swap_pmd(_pmd) || pmd_trans_huge(_pmd) || pmd_devmap(_pmd)) { if ((next - addr != HPAGE_PMD_SIZE) || @@ -428,9 +417,6 @@ static inline long change_pmd_range(struct mmu_gather *tlb, cond_resched(); } while (pmd++, addr = next, addr != end); - if (range.start) - mmu_notifier_invalidate_range_end(&range); - return pages; } @@ -438,22 +424,36 @@ static inline long change_pud_range(struct mmu_gather *tlb, struct vm_area_struct *vma, p4d_t *p4d, unsigned long addr, unsigned long end, pgprot_t newprot, unsigned long cp_flags) { + struct mmu_notifier_range range; pud_t *pud; unsigned long next; long pages = 0, ret; + range.start = 0; + pud = pud_offset(p4d, addr); do { next = pud_addr_end(addr, end); ret = change_prepare(vma, pud, pmd, addr, cp_flags); - if (ret) - return ret; + if (ret) { + pages = ret; + break; + } if (pud_none_or_clear_bad(pud)) continue; + if (!range.start) { + mmu_notifier_range_init(&range, + MMU_NOTIFY_PROTECTION_VMA, 0, + vma->vm_mm, addr, end); + mmu_notifier_invalidate_range_start(&range); + } pages += change_pmd_range(tlb, vma, pud, addr, next, newprot, cp_flags); } while (pud++, addr = next, addr != end); + if (range.start) + mmu_notifier_invalidate_range_end(&range); + return pages; } From patchwork Mon Jul 15 19:21:38 2024 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Peter Xu X-Patchwork-Id: 13733835 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from kanga.kvack.org (kanga.kvack.org [205.233.56.17]) by smtp.lore.kernel.org (Postfix) with ESMTP id 21C27C3DA64 for ; Mon, 15 Jul 2024 19:22:04 +0000 (UTC) Received: by kanga.kvack.org (Postfix) id D485E6B009A; Mon, 15 Jul 2024 15:22:01 -0400 (EDT) Received: by kanga.kvack.org (Postfix, from userid 40) id CFA406B009C; Mon, 15 Jul 2024 15:22:01 -0400 (EDT) X-Delivered-To: int-list-linux-mm@kvack.org Received: by kanga.kvack.org (Postfix, from userid 63042) id B22FC6B009D; Mon, 15 Jul 2024 15:22:01 -0400 (EDT) X-Delivered-To: linux-mm@kvack.org Received: from relay.hostedemail.com (smtprelay0011.hostedemail.com [216.40.44.11]) by kanga.kvack.org (Postfix) with ESMTP id 931366B009A for ; Mon, 15 Jul 2024 15:22:01 -0400 (EDT) Received: from smtpin29.hostedemail.com (a10.router.float.18 [10.200.18.1]) by unirelay09.hostedemail.com (Postfix) with ESMTP id 45E9380157 for ; Mon, 15 Jul 2024 19:22:01 +0000 (UTC) X-FDA: 82342957242.29.DB81E3F Received: from us-smtp-delivery-124.mimecast.com (us-smtp-delivery-124.mimecast.com [170.10.129.124]) by imf14.hostedemail.com (Postfix) with ESMTP id 39DF7100015 for ; Mon, 15 Jul 2024 19:21:58 +0000 (UTC) Authentication-Results: imf14.hostedemail.com; dkim=pass header.d=redhat.com header.s=mimecast20190719 header.b=cGx7pQhS; spf=pass (imf14.hostedemail.com: domain of peterx@redhat.com designates 170.10.129.124 as permitted sender) smtp.mailfrom=peterx@redhat.com; dmarc=pass (policy=none) header.from=redhat.com ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=hostedemail.com; s=arc-20220608; t=1721071299; h=from:from:sender:reply-to:subject:subject:date:date: message-id:message-id:to:to:cc:cc:mime-version:mime-version: content-type:content-type: content-transfer-encoding:content-transfer-encoding: in-reply-to:in-reply-to:references:references:dkim-signature; bh=DPSgdrK+/9txvlBVeWURx0wHDXuvLt3nB8aMlUhEkV8=; b=JQsWXKlYARhd17zV/s6WeK2CL/vGMx3ZgiMAzEuZ0WknfdkY/vlkBpUK+Me/lci7pMfgYc N5QUu9cvSgk5J7GfKCtqsCRWwxQNq6zXbvZt4VtOSxeoawZDcAnTKnnAVUtWikWzvGsHP6 JpwYrqUOZs6WZnSuTZC6NxkpTVeC2co= ARC-Authentication-Results: i=1; imf14.hostedemail.com; dkim=pass header.d=redhat.com header.s=mimecast20190719 header.b=cGx7pQhS; spf=pass (imf14.hostedemail.com: domain of peterx@redhat.com designates 170.10.129.124 as permitted sender) smtp.mailfrom=peterx@redhat.com; dmarc=pass (policy=none) header.from=redhat.com ARC-Seal: i=1; s=arc-20220608; d=hostedemail.com; t=1721071299; a=rsa-sha256; cv=none; b=5N2igaBvQvzJ0gSQ3Prq9kT3IREggDzbPZdqB2GBnMa8phN+1U13XKQ6sR/LV1yCLWdtVP 6hK4eqCAqhyeoLKQzm10UJSvHp4lxzhLVWAB3/6GBAOuNe7PEWr4i++DtJcwgQcSRnG3BL rpuDpJnQJyXG+ZlcFL+Sor7wrJt/aMo= DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=redhat.com; s=mimecast20190719; t=1721071317; h=from:from:reply-to:subject:subject:date:date:message-id:message-id: to:to:cc:cc:mime-version:mime-version:content-type:content-type: content-transfer-encoding:content-transfer-encoding: in-reply-to:in-reply-to:references:references; bh=DPSgdrK+/9txvlBVeWURx0wHDXuvLt3nB8aMlUhEkV8=; b=cGx7pQhSGxG8NiooGlbcXbDFs3ikYT9ajrHI0k+S6iAH2RCABJ8FxJDqLrFbt2icnhfFSa C8ooNWzbvOmuQ0ZgOSntMfLP1RO0ph3WG3dlQDMLwBEHCoCJC5d4Y6Ixm+6L1jNLyY8o9m 4zhqvVhkb6bdCnvHC225jqfFkoOf1qs= Received: from mail-qv1-f71.google.com (mail-qv1-f71.google.com [209.85.219.71]) by relay.mimecast.com with ESMTP with STARTTLS (version=TLSv1.3, cipher=TLS_AES_256_GCM_SHA384) id us-mta-25-B9UFhhDqPeSISU9a9qcrOw-1; Mon, 15 Jul 2024 15:21:56 -0400 X-MC-Unique: B9UFhhDqPeSISU9a9qcrOw-1 Received: by mail-qv1-f71.google.com with SMTP id 6a1803df08f44-6b60b5b85deso13080186d6.3 for ; Mon, 15 Jul 2024 12:21:56 -0700 (PDT) X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20230601; t=1721071315; x=1721676115; h=content-transfer-encoding:mime-version:references:in-reply-to :message-id:date:subject:cc:to:from:x-gm-message-state:from:to:cc :subject:date:message-id:reply-to; bh=DPSgdrK+/9txvlBVeWURx0wHDXuvLt3nB8aMlUhEkV8=; b=e//iPaby34H2HJz7tTFmr9iQzGRX18ze6EubnTWrLwiInqsz03gEhgRKG/hTzMvvIF GIGqbF4Jl8ogR+TLhu0XFMitTIGFIx2jpYaENO112ZfPkYf26N3vWv8j5rfzG3ZL6v/A SoiJPXSeZ3VFRF8xFIquLXLzVuEiwsuzp3Jv9flTrIBlamRqCnbFfXNU3TcQlzu3fp+g G2LblN9VXJXuff/+jWcCYO6vPiqtZTuVEZoPZeH2I4eSrwf1TUwByflJGW3YFVWddGFn z+h/vLWEa9GiMOlyTaxsgiImaq2vhGryruF4HkocPEJEZi35aHMGi0afoat8+brLz/sN SxiA== X-Gm-Message-State: AOJu0YzT+ptsWxEAk/dW9/JH3Og08XSo+apTIDLtdkAGGaRM1QAQFvsw Xdm8AM7rG0/wQwau4IwvXsOmWnQskC8Vbx6pBUjwN4FLmZlFa02EAyjnm9Mdw4eG7Nw5PzoqIHe Na265alH6jtvOhHW7rZzNQdFKT+2AS+IWRGPQOLWIETyTfR0jTYHjjTwHMRuPcLciTOnf17+2Sa u88bynJf6qJbGpeZxxpBz5Krk8RqjfqQ== X-Received: by 2002:a05:6214:2dc1:b0:6b5:90eb:370f with SMTP id 6a1803df08f44-6b77de8150amr4905806d6.2.1721071315045; Mon, 15 Jul 2024 12:21:55 -0700 (PDT) X-Google-Smtp-Source: AGHT+IGFc1iq6cjowggxR08A9jWNh2nLNKZtlnYVNF070vv7/tCRn3zBtJYgG+pCX+rmBsWl/qf7Fw== X-Received: by 2002:a05:6214:2dc1:b0:6b5:90eb:370f with SMTP id 6a1803df08f44-6b77de8150amr4905476d6.2.1721071314625; Mon, 15 Jul 2024 12:21:54 -0700 (PDT) Received: from x1n.redhat.com (pool-99-254-121-117.cpe.net.cable.rogers.com. [99.254.121.117]) by smtp.gmail.com with ESMTPSA id 6a1803df08f44-6b761978d30sm24039356d6.31.2024.07.15.12.21.52 (version=TLS1_3 cipher=TLS_AES_256_GCM_SHA384 bits=256/256); Mon, 15 Jul 2024 12:21:54 -0700 (PDT) From: Peter Xu To: linux-mm@kvack.org, linux-kernel@vger.kernel.org Cc: Dave Jiang , Rik van Riel , Dave Hansen , Michael Ellerman , linuxppc-dev@lists.ozlabs.org, Matthew Wilcox , Rick P Edgecombe , peterx@redhat.com, Oscar Salvador , Mel Gorman , Andrew Morton , Borislav Petkov , Christophe Leroy , Huang Ying , "Kirill A . Shutemov" , "Aneesh Kumar K . V" , Dan Williams , Thomas Gleixner , Hugh Dickins , x86@kernel.org, Nicholas Piggin , Vlastimil Babka , Ingo Molnar Subject: [PATCH v3 4/8] mm/powerpc: Add missing pud helpers Date: Mon, 15 Jul 2024 15:21:38 -0400 Message-ID: <20240715192142.3241557-5-peterx@redhat.com> X-Mailer: git-send-email 2.45.0 In-Reply-To: <20240715192142.3241557-1-peterx@redhat.com> References: <20240715192142.3241557-1-peterx@redhat.com> MIME-Version: 1.0 X-Mimecast-Spam-Score: 0 X-Mimecast-Originator: redhat.com X-Rspamd-Server: rspam03 X-Rspam-User: X-Rspamd-Queue-Id: 39DF7100015 X-Stat-Signature: bs3zkox1uwdghgwrqpjxg37juu851nj8 X-HE-Tag: 1721071318-719341 X-HE-Meta: U2FsdGVkX19zzUa9prt1uq/zo1fPNKut4xqUzoI90I/JzX15cnJZjYJdKDl11gdfCCtXZE0vYZ8Qa1BmmUrGyCWhwIuFv/X8QIFRsfbaWPv4m6oOTrQMnQO7HqhMt+HBXe7u/aAxcItcC35y4DZoqPgEbqADC2ZoT1KmqNc16vnoN0X68jZs9J/D3YJmL3zGKIPpUlwUPZo163l0mxvQDkR95NqzIGEO+lTPAP95He9OryPT9yZWfEeafUxIH/agz1PXlVwFjILQHf94F1SjJ8InEKjWMp/myw8fdHxBL3U0aqbXg+SDWZ6Lhc+WLfle2QfrTdQjpeSn8PEFJyQ8v0Yr0rK7Zb1FYA4ncA3BnL5XDafFPlrzTT5tekGDM0yk8N4b8Kxw8JKLn7biPPWXjUsNMGatGOu+V4o8nPu8N3l6AVBheimXQhPcaWq6vzJP1YCUE4IQw2WUCtlHarnRf+QBjkj6X1XOXH2BAMsgNarEr88uBkeuPzwgW8YX40o/uuhcB4x+BB9Rp3x4exmvbUQnFWJS0Hp46VN+7AAClwrlE2CW8durBMk6WiS65/gW0GisK0dh+MCSjul3tPMv4c32ChOIdejFkQJfYQq0XeTQNqfsnH4fGNKbx+zZImNQScwBUBOLQWvGh5kukT7k29jowavGgP2ykv4u6KEzsvErLt7hpYo+GdrbbrTaDZ7Pp1SLdLMiIE+t1FQ2eDakMMZvutnj1TdeGJxx8tEmakMYaWwePjalAllCa4GhhUwaG2n1KNFmumTAdDHFdezpJw/P0kktwMbm0ZBZWoqnw8IrGMzXr1o5vWN+zcnqypm8XzXqlsOE1JsjnYjUJv0+hS+TAoKX4beXLp3evm3/by6vcHn0JNXU7P1Ky6IVjd3KZoQmrATAm9Za/C0R3l/NU8Va0CUUlcoE9JjUXSwizc+n6aMHHUaWnEiiWx0KQqdT1lMv05PuDRdfOpXuYtA 2gFDiVcV VOY6V1Z9uDkBtX1XK7h5Rcp6F/H3qQLH6JjzFyuuZa9kcNGu3Pk4jLW+pc02D0X1lqhmPxt1+XulQzYdFsj7SPTL5zzmQfMRcvNa2FK3NfCF7cpkV64tg0uocUrfdDhiIHBIIqWrNwQYdjmvNhasyzrv2kBdXgdmqwWygJumTUx/v5uUGbQCscdLU4iZpD2IJuc5/DzMfEY4P69VtPyBFJ++fQ83Q2IpQiloGKGqw+MZ4F5834Zt3QWlBH1hyIl3J+TWA0KWm5sXtq49OFPvyZlwma+cUjfUOi/2bO5IzlDyKRJ8ij6DcUwlAKZfdnDTbp1dzPLmMjNMUvByk9TKyftLlUWzy4EKgfNPMX7oHqVQoR96IdzpDK0rLKKZ2rndTCSmtGQoLWLhexlW2gdc4n67DY2LisGz738T3EOKfDD0vP80MSXpVB/S8xOeN37kHBWThasSTVSJRpsY= X-Bogosity: Ham, tests=bogofilter, spamicity=0.000000, version=1.2.4 Sender: owner-linux-mm@kvack.org Precedence: bulk X-Loop: owner-majordomo@kvack.org List-ID: List-Subscribe: List-Unsubscribe: These new helpers will be needed for pud entry updates soon. Introduce them by referencing the pmd ones. Namely: - pudp_invalidate() - pud_modify() Cc: Michael Ellerman Cc: Nicholas Piggin Cc: Christophe Leroy Cc: linuxppc-dev@lists.ozlabs.org Cc: Aneesh Kumar K.V Signed-off-by: Peter Xu --- arch/powerpc/include/asm/book3s/64/pgtable.h | 3 +++ arch/powerpc/mm/book3s64/pgtable.c | 20 ++++++++++++++++++++ 2 files changed, 23 insertions(+) diff --git a/arch/powerpc/include/asm/book3s/64/pgtable.h b/arch/powerpc/include/asm/book3s/64/pgtable.h index 519b1743a0f4..5da92ba68a45 100644 --- a/arch/powerpc/include/asm/book3s/64/pgtable.h +++ b/arch/powerpc/include/asm/book3s/64/pgtable.h @@ -1124,6 +1124,7 @@ extern pmd_t pfn_pmd(unsigned long pfn, pgprot_t pgprot); extern pud_t pfn_pud(unsigned long pfn, pgprot_t pgprot); extern pmd_t mk_pmd(struct page *page, pgprot_t pgprot); extern pmd_t pmd_modify(pmd_t pmd, pgprot_t newprot); +extern pud_t pud_modify(pud_t pud, pgprot_t newprot); extern void set_pmd_at(struct mm_struct *mm, unsigned long addr, pmd_t *pmdp, pmd_t pmd); extern void set_pud_at(struct mm_struct *mm, unsigned long addr, @@ -1384,6 +1385,8 @@ static inline pgtable_t pgtable_trans_huge_withdraw(struct mm_struct *mm, #define __HAVE_ARCH_PMDP_INVALIDATE extern pmd_t pmdp_invalidate(struct vm_area_struct *vma, unsigned long address, pmd_t *pmdp); +extern pud_t pudp_invalidate(struct vm_area_struct *vma, unsigned long address, + pud_t *pudp); #define pmd_move_must_withdraw pmd_move_must_withdraw struct spinlock; diff --git a/arch/powerpc/mm/book3s64/pgtable.c b/arch/powerpc/mm/book3s64/pgtable.c index f4d8d3c40e5c..5a4a75369043 100644 --- a/arch/powerpc/mm/book3s64/pgtable.c +++ b/arch/powerpc/mm/book3s64/pgtable.c @@ -176,6 +176,17 @@ pmd_t pmdp_invalidate(struct vm_area_struct *vma, unsigned long address, return __pmd(old_pmd); } +pud_t pudp_invalidate(struct vm_area_struct *vma, unsigned long address, + pud_t *pudp) +{ + unsigned long old_pud; + + VM_WARN_ON_ONCE(!pud_present(*pudp)); + old_pud = pud_hugepage_update(vma->vm_mm, address, pudp, _PAGE_PRESENT, _PAGE_INVALID); + flush_pud_tlb_range(vma, address, address + HPAGE_PUD_SIZE); + return __pud(old_pud); +} + pmd_t pmdp_huge_get_and_clear_full(struct vm_area_struct *vma, unsigned long addr, pmd_t *pmdp, int full) { @@ -259,6 +270,15 @@ pmd_t pmd_modify(pmd_t pmd, pgprot_t newprot) pmdv &= _HPAGE_CHG_MASK; return pmd_set_protbits(__pmd(pmdv), newprot); } + +pud_t pud_modify(pud_t pud, pgprot_t newprot) +{ + unsigned long pudv; + + pudv = pud_val(pud); + pudv &= _HPAGE_CHG_MASK; + return pud_set_protbits(__pud(pudv), newprot); +} #endif /* CONFIG_TRANSPARENT_HUGEPAGE */ /* For use by kexec, called with MMU off */ From patchwork Mon Jul 15 19:21:39 2024 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Peter Xu X-Patchwork-Id: 13733836 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from kanga.kvack.org (kanga.kvack.org [205.233.56.17]) by smtp.lore.kernel.org (Postfix) with ESMTP id F3A8FC3DA63 for ; Mon, 15 Jul 2024 19:22:06 +0000 (UTC) Received: by kanga.kvack.org (Postfix) id 07DE26B009D; Mon, 15 Jul 2024 15:22:05 -0400 (EDT) Received: by kanga.kvack.org (Postfix, from userid 40) id EF5AC6B009E; Mon, 15 Jul 2024 15:22:04 -0400 (EDT) X-Delivered-To: int-list-linux-mm@kvack.org Received: by kanga.kvack.org (Postfix, from userid 63042) id CF8126B00A0; Mon, 15 Jul 2024 15:22:04 -0400 (EDT) X-Delivered-To: linux-mm@kvack.org Received: from relay.hostedemail.com (smtprelay0014.hostedemail.com [216.40.44.14]) by kanga.kvack.org (Postfix) with ESMTP id ADFC86B009E for ; Mon, 15 Jul 2024 15:22:04 -0400 (EDT) Received: from smtpin20.hostedemail.com (a10.router.float.18 [10.200.18.1]) by unirelay08.hostedemail.com (Postfix) with ESMTP id 5E4D01401EA for ; Mon, 15 Jul 2024 19:22:04 +0000 (UTC) X-FDA: 82342957368.20.61272C9 Received: from us-smtp-delivery-124.mimecast.com (us-smtp-delivery-124.mimecast.com [170.10.133.124]) by imf18.hostedemail.com (Postfix) with ESMTP id 5713F1C0023 for ; Mon, 15 Jul 2024 19:22:02 +0000 (UTC) Authentication-Results: imf18.hostedemail.com; dkim=pass header.d=redhat.com header.s=mimecast20190719 header.b=YlaI9QPa; spf=pass (imf18.hostedemail.com: domain of peterx@redhat.com designates 170.10.133.124 as permitted sender) smtp.mailfrom=peterx@redhat.com; dmarc=pass (policy=none) header.from=redhat.com ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=hostedemail.com; s=arc-20220608; t=1721071284; h=from:from:sender:reply-to:subject:subject:date:date: message-id:message-id:to:to:cc:cc:mime-version:mime-version: content-type:content-type: content-transfer-encoding:content-transfer-encoding: in-reply-to:in-reply-to:references:references:dkim-signature; bh=TFEuzrgphkpyKgRaMn/StFmQ1IWZKcW4qj9eIjoToHw=; b=AhcQ0027ZOzKNnKpqZ6UJCRwFqMlS4F88YFVO2wPPHIB7w2H1bWM2qHZvK2p4A6qCwbICq OVfCEq5MQw1mmdgtInotUJ8Jhw9o31SGIOTRXxSHKiGfdHWXT1JAphPlkaIQKm/CVYbkth kFRw9iLC+lCh8uuMSRhyJ/F2u8zLhaM= ARC-Seal: i=1; s=arc-20220608; d=hostedemail.com; t=1721071284; a=rsa-sha256; cv=none; b=1S2mfdkHProlA/uZ6EFDewd7et24/myONOQoigpqvoZc3MdI/mE7P+l+zQytR9hLBzbWXb bHKrGb0bh7DlZ3Z/OXqkJsYjn3CDoS565cqQGPI3fTSSNe4L6aH0UbX1eS6BfrY2Mo/QtT gMkBIpn74cw6EhSD7N0z9IaDjbkVIqI= ARC-Authentication-Results: i=1; imf18.hostedemail.com; dkim=pass header.d=redhat.com header.s=mimecast20190719 header.b=YlaI9QPa; spf=pass (imf18.hostedemail.com: domain of peterx@redhat.com designates 170.10.133.124 as permitted sender) smtp.mailfrom=peterx@redhat.com; dmarc=pass (policy=none) header.from=redhat.com DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=redhat.com; s=mimecast20190719; t=1721071321; h=from:from:reply-to:subject:subject:date:date:message-id:message-id: to:to:cc:cc:mime-version:mime-version:content-type:content-type: content-transfer-encoding:content-transfer-encoding: in-reply-to:in-reply-to:references:references; bh=TFEuzrgphkpyKgRaMn/StFmQ1IWZKcW4qj9eIjoToHw=; b=YlaI9QPacGnYVr/+RnEiIT09nODOOTEdtPukn5sWIXwqUSuKDtXvuSrXRSwI+KJ19gY++7 DZtPAWVpO23v28zS0z9IpLf965eJgVdtxdBOGpp2JUZK5KUOBbAH/pxkssMwXVDSkRUFWn c41ttFOjsWy+MKnEbCXhlK0ruKTPEvA= Received: from mail-oo1-f70.google.com (mail-oo1-f70.google.com [209.85.161.70]) by relay.mimecast.com with ESMTP with STARTTLS (version=TLSv1.3, cipher=TLS_AES_256_GCM_SHA384) id us-mta-326-BFsfOldZNnKJnaMuUuJmlg-1; Mon, 15 Jul 2024 15:21:59 -0400 X-MC-Unique: BFsfOldZNnKJnaMuUuJmlg-1 Received: by mail-oo1-f70.google.com with SMTP id 006d021491bc7-5ce529cca68so557233eaf.2 for ; Mon, 15 Jul 2024 12:21:58 -0700 (PDT) X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20230601; t=1721071317; x=1721676117; h=content-transfer-encoding:mime-version:references:in-reply-to :message-id:date:subject:cc:to:from:x-gm-message-state:from:to:cc :subject:date:message-id:reply-to; bh=TFEuzrgphkpyKgRaMn/StFmQ1IWZKcW4qj9eIjoToHw=; b=bScSLWgUIJixlAbuBYfT7SMRyakBwYcYD9HxChuBv0SNdWSDydxP6Ya56veW7c7K77 r76R2yPzX9DdDiTiR0T14Sv17NRRMpWoB5EN4p/3m12SH5Y5xOAaXHqXdBYSZJxuORy9 UG7+o6auiQ4hQhatSGFz1hoLxNAU5b3cZhi3xA+AFvZPkdwJQuRVuO/a0IGPXzh1Z0L/ oo76Vy/7NvM3eWa7B2oHQs4A63yWepHS/6T+QybcSVsALbXRoq63/K1GJBermdpBw1Qp Ydsj083cAkBU4GmHvJIUwVEml+0sIUQTTVD0N/Q3rtHK4KGHeZXOsN1Hk1LqqTlRkbBd N1EQ== X-Gm-Message-State: AOJu0YxHo4GgAq8p5KdRSKXJMafcr75pL3sLlSiKP9jdsgLNN24dkkJa X5wuuyzhvuFpnHxnOOX6f0shrE7fQt0Ht7azt3aJ0+K61st/HQ5pdxIiYZAg64e4kHgImnWWwAX nQkTnAYX20oGmQePN5KIYTZ5M0lj+peP2p7Xx3ugt3m3TR0NnQisTZE64r6WuoI1q0GZBhvDfl7 vFxXPadHqGDbvxr1+MyJJr+olWKbdX7Q== X-Received: by 2002:a4a:4308:0:b0:5c7:aeba:77a0 with SMTP id 006d021491bc7-5d25035be58mr500674eaf.0.1721071317138; Mon, 15 Jul 2024 12:21:57 -0700 (PDT) X-Google-Smtp-Source: AGHT+IHclECHVFRdVYbFwnrtKH3lJcHIiAHM9jyDo1qTRx8+nslWI3GA72lYOtCld/HD9tYeAzQ33w== X-Received: by 2002:a4a:4308:0:b0:5c7:aeba:77a0 with SMTP id 006d021491bc7-5d25035be58mr500637eaf.0.1721071316695; Mon, 15 Jul 2024 12:21:56 -0700 (PDT) Received: from x1n.redhat.com (pool-99-254-121-117.cpe.net.cable.rogers.com. [99.254.121.117]) by smtp.gmail.com with ESMTPSA id 6a1803df08f44-6b761978d30sm24039356d6.31.2024.07.15.12.21.54 (version=TLS1_3 cipher=TLS_AES_256_GCM_SHA384 bits=256/256); Mon, 15 Jul 2024 12:21:56 -0700 (PDT) From: Peter Xu To: linux-mm@kvack.org, linux-kernel@vger.kernel.org Cc: Dave Jiang , Rik van Riel , Dave Hansen , Michael Ellerman , linuxppc-dev@lists.ozlabs.org, Matthew Wilcox , Rick P Edgecombe , peterx@redhat.com, Oscar Salvador , Mel Gorman , Andrew Morton , Borislav Petkov , Christophe Leroy , Huang Ying , "Kirill A . Shutemov" , "Aneesh Kumar K . V" , Dan Williams , Thomas Gleixner , Hugh Dickins , x86@kernel.org, Nicholas Piggin , Vlastimil Babka , Ingo Molnar Subject: [PATCH v3 5/8] mm/x86: Make pud_leaf() only cares about PSE bit Date: Mon, 15 Jul 2024 15:21:39 -0400 Message-ID: <20240715192142.3241557-6-peterx@redhat.com> X-Mailer: git-send-email 2.45.0 In-Reply-To: <20240715192142.3241557-1-peterx@redhat.com> References: <20240715192142.3241557-1-peterx@redhat.com> MIME-Version: 1.0 X-Mimecast-Spam-Score: 0 X-Mimecast-Originator: redhat.com X-Stat-Signature: 8jy5rgbg9qd4bmeoxag9fagz9sraog8w X-Rspamd-Queue-Id: 5713F1C0023 X-Rspam-User: X-Rspamd-Server: rspam08 X-HE-Tag: 1721071322-714452 X-HE-Meta: U2FsdGVkX19aEDoQwhT2ZX/FM0RI5gYzbUHiyywPFvdPiNnaelsjch/IwB7UbuURif8Oyhh/yABwvlpk6+wAS5stYipx1Ld8ixAu5dH5QehEH2+keoN+t1Df2Sp47cG7CrgUVH6SID3TAXGxuvHDqr7eVObTmjM5HDHjB2Y5Djn5XnnCzQYaUpbtq6f89iNEMVoPgqzANwjs5bZiqzQ7MUakiIaJnLpyWn0PGACF94kUIqMYxwTZLlWaXbSabJ77DyokYNHT4l7TTqN4cqS8rtRHetwkQG/8lNYw2b/+/rjldRohiaJQ2RIeJ8QmDrcU66ClTQOXhmxMjLNnURJzvH+Nro2ubdWnFc7dWA1yT4C8Iqk6bgL6qqtTdNzL0VRO0v8J/Fj0BUECy42cV7aknztCA55+ssC2gOfivLN3DnPWmHx7YLj/kOvEVielhg655EGTwyUuufVb+17G81zwJSH3ipB95qpLSaDk34FNOu75+rkzZ+/0ITIetUuzfDJAqLUXXcItmzZRm+T3PLTY+3DpnN+PEOYBbb/IfD6Pcc3X1pmFLg39Mm54mOl0oqPQ2JQ0mTuSHY1njq6K4Mtb9RT5Jy2JCVJVgIHQOmO2dexhhqpP0LasV1J7X8hxHFRzL43W19lBAnWc2+GaXFRDOi3OrWEbQjAtpH/Km+L89ihJCWfsYUYb7AMGsiluGpQi55NsBz9NkEovAd8Gx0S3+ylzCT90FyILZ9LpGc5KPS3WLDXHnNswZe2NGzvSq53psvdQhYiujez3+pQaSx2EAeP93UcPuSLKNeL8ZDn99zbwPbEA/RzbOxIcp5/3L6DtcQlxhzJH5JrUOSvmWy6uz/FIXgUR9CwGOL6hMhPmahW66Seck61NkeVjL4lrpsDtcvtSjQQCuJtGnBvPvn1QvUg9v3zmF3t5KJSnJxsQ3rCKEkXuxx96wtpxra02KIkeWPH1y+shIdG6BjSR5G0 kYtUMeSL xAM17SbDhx5yCvFkzs2AMj7n+EJTyC2ZDLS/ISUgu/sm5WNPkYrj/gehNETB+dvY/7+JFlrrGtoRn3WYhSlGYVk4Ev7B4BFtYqpaXyEH4G5TmkxOrE2pIhmigataLvVKJbZR+R4P//h99BBUm+lTgU3JWe0IRRni7CYliRGsCsueBl/9FHNX3zrMS8tx9T25dpKkFE7d/C42n31uFSGAEIz0sbcYDObN4MntiRtctLQVtAhbjI0/eQB21sJFEZsLHgjaA5yN/62dsnQYSqhTWryd/oin4PiL7KhwQo5mHNryBdAYaw4ZfGC/5G2/fRRxjcCUXKeDPVVSsiUaw1bcnvspVuL7nHXyt+/cFbIPpu2dkEZXsh0yeFtOlorPA8HG7YHsvWV4d4iz5b0/0MWsfTphwmy/lqxakkNP9ibis6yEdtn4iFeI48nZTvY29vKXyLHUC4z2XMOMJ7LVjgiugIJy2yg== X-Bogosity: Ham, tests=bogofilter, spamicity=0.000000, version=1.2.4 Sender: owner-linux-mm@kvack.org Precedence: bulk X-Loop: owner-majordomo@kvack.org List-ID: List-Subscribe: List-Unsubscribe: An entry should be reported as PUD leaf even if it's PROT_NONE, in which case PRESENT bit isn't there. I hit bad pud without this when testing dax 1G on zapping a PROT_NONE PUD. Cc: Thomas Gleixner Cc: Ingo Molnar Cc: Borislav Petkov Cc: Dave Hansen Cc: x86@kernel.org Acked-by: Dave Hansen Signed-off-by: Peter Xu Reviewed-by: David Hildenbrand --- arch/x86/include/asm/pgtable.h | 3 +-- 1 file changed, 1 insertion(+), 2 deletions(-) diff --git a/arch/x86/include/asm/pgtable.h b/arch/x86/include/asm/pgtable.h index 65b8e5bb902c..25fc6d809572 100644 --- a/arch/x86/include/asm/pgtable.h +++ b/arch/x86/include/asm/pgtable.h @@ -1073,8 +1073,7 @@ static inline pmd_t *pud_pgtable(pud_t pud) #define pud_leaf pud_leaf static inline bool pud_leaf(pud_t pud) { - return (pud_val(pud) & (_PAGE_PSE | _PAGE_PRESENT)) == - (_PAGE_PSE | _PAGE_PRESENT); + return pud_val(pud) & _PAGE_PSE; } static inline int pud_bad(pud_t pud) From patchwork Mon Jul 15 19:21:40 2024 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Peter Xu X-Patchwork-Id: 13733837 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from kanga.kvack.org (kanga.kvack.org [205.233.56.17]) by smtp.lore.kernel.org (Postfix) with ESMTP id 082E0C3DA59 for ; Mon, 15 Jul 2024 19:22:10 +0000 (UTC) Received: by kanga.kvack.org (Postfix) id 348B96B009E; Mon, 15 Jul 2024 15:22:05 -0400 (EDT) Received: by kanga.kvack.org (Postfix, from userid 40) id 2A2BD6B00A1; Mon, 15 Jul 2024 15:22:05 -0400 (EDT) X-Delivered-To: int-list-linux-mm@kvack.org Received: by kanga.kvack.org (Postfix, from userid 63042) id D96B56B009F; Mon, 15 Jul 2024 15:22:04 -0400 (EDT) X-Delivered-To: linux-mm@kvack.org Received: from relay.hostedemail.com (smtprelay0012.hostedemail.com [216.40.44.12]) by kanga.kvack.org (Postfix) with ESMTP id A07EA6B009D for ; Mon, 15 Jul 2024 15:22:04 -0400 (EDT) Received: from smtpin24.hostedemail.com (a10.router.float.18 [10.200.18.1]) by unirelay02.hostedemail.com (Postfix) with ESMTP id 50A7A1201B7 for ; Mon, 15 Jul 2024 19:22:04 +0000 (UTC) X-FDA: 82342957368.24.F8E1B2C Received: from us-smtp-delivery-124.mimecast.com (us-smtp-delivery-124.mimecast.com [170.10.133.124]) by imf30.hostedemail.com (Postfix) with ESMTP id 493EC8001B for ; Mon, 15 Jul 2024 19:22:02 +0000 (UTC) Authentication-Results: imf30.hostedemail.com; dkim=pass header.d=redhat.com header.s=mimecast20190719 header.b=ZQ1PeKSH; dmarc=pass (policy=none) header.from=redhat.com; spf=pass (imf30.hostedemail.com: domain of peterx@redhat.com designates 170.10.133.124 as permitted sender) smtp.mailfrom=peterx@redhat.com ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=hostedemail.com; s=arc-20220608; t=1721071280; h=from:from:sender:reply-to:subject:subject:date:date: message-id:message-id:to:to:cc:cc:mime-version:mime-version: content-type:content-type: content-transfer-encoding:content-transfer-encoding: in-reply-to:in-reply-to:references:references:dkim-signature; bh=9JbEXa1vvY/QVlVssvu52FPZK180R34zP6ifpD40mks=; b=4mOHncNIIKKPRCfIIFcu05oCTu5QYVysHJd9E3HqivHJ+mWDXPTpytnXI7Xg2Wyoc1u9tZ vHAG77TMFOOwmcr5AF6yIHvtDyBWSh3AabHpw/RuA0NteZux6qF2wmrRVSvFaBygRgmpYY p31gQc2Gq1/L7WGEVcwsN/ImzQA45Cg= ARC-Seal: i=1; s=arc-20220608; d=hostedemail.com; t=1721071280; a=rsa-sha256; cv=none; b=spQhQTdrqe+ILKyNRh4yV0qrarBJrxXgJ4Nh1Ilpl1AHpGdzqhvV/ImKgiy/Yfqda8VhfI hvuU6zbqqUsAYOXtm7WTYN4fcW6JDbeeDoF/TcCu+0OlHogE0HtbCdkO2ZSIW8f86GeKbU k7G460PN7L6lM1LCpgpLLo9gsKHG8ig= ARC-Authentication-Results: i=1; imf30.hostedemail.com; dkim=pass header.d=redhat.com header.s=mimecast20190719 header.b=ZQ1PeKSH; dmarc=pass (policy=none) header.from=redhat.com; spf=pass (imf30.hostedemail.com: domain of peterx@redhat.com designates 170.10.133.124 as permitted sender) smtp.mailfrom=peterx@redhat.com DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=redhat.com; s=mimecast20190719; t=1721071321; h=from:from:reply-to:subject:subject:date:date:message-id:message-id: to:to:cc:cc:mime-version:mime-version:content-type:content-type: content-transfer-encoding:content-transfer-encoding: in-reply-to:in-reply-to:references:references; bh=9JbEXa1vvY/QVlVssvu52FPZK180R34zP6ifpD40mks=; b=ZQ1PeKSHc8PZc6E5seHJ5P7PN3LdirgMsFVEYuMcc6YrUko6YSiZyF9pNUQJF0xpI68e0f f8kp0IFzn4VFUWWoM3uoU9BPix0eeBCxzqW7oVNqP956Ix/AW09NJ9+tjVOGyzZsfz7fcY pBDd7oK/GddPOs0Z7AgA/7l0klI9mQA= Received: from mail-yb1-f200.google.com (mail-yb1-f200.google.com [209.85.219.200]) by relay.mimecast.com with ESMTP with STARTTLS (version=TLSv1.3, cipher=TLS_AES_256_GCM_SHA384) id us-mta-488-GInvCMcxOqyjWo2KsVnA7w-1; Mon, 15 Jul 2024 15:22:00 -0400 X-MC-Unique: GInvCMcxOqyjWo2KsVnA7w-1 Received: by mail-yb1-f200.google.com with SMTP id 3f1490d57ef6-e05a1434e17so676212276.0 for ; Mon, 15 Jul 2024 12:22:00 -0700 (PDT) X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20230601; t=1721071319; x=1721676119; h=content-transfer-encoding:mime-version:references:in-reply-to :message-id:date:subject:cc:to:from:x-gm-message-state:from:to:cc :subject:date:message-id:reply-to; bh=9JbEXa1vvY/QVlVssvu52FPZK180R34zP6ifpD40mks=; b=fY9MR/BTp3+VjrjjTj24Ax5YEtLIgPwc0H2oWJGMs8B3Vm19AoyOHnst7q9Z1Wxmpx erjBJ46GT14hqOmu3J6+pzpc95ZfQIHPiAjN7613PdlnzkeOMBYtX34KeVBU3lS/Jygk dGuN7Y6KnUtKJRZgGqbIxjybRp7F/PiwEU+4TAyE6NGuUduf9puBvB2yY6F1PHQSZiYC 2Dk37u6N+mrq52aGpQ2CMtdOHNOm4jEKFgS6zlKgKRRG9U8xIEQPwusmgXFdcjNSMifc zQRddJp8K30JRzZK7Simkty3eunpI6EtK6KURNbJXitcmjwYP2+6B8sclvZixeAZPscC F5rA== X-Gm-Message-State: AOJu0YzYNfSS4IIUw42qnn4LuqyVKXN0HCdTrfJFOmLRPvNFcGx99G3+ 7qKyHen+DPESuD7Rg+68bAUplqRz5E55gHm55dau745qxSJdXUTT5W31C8G4nd26fy8TL4iBKPA eKFPNY9nt0uGoQJG/SQAE12ix8QogqmxaKUWUHKaI8Lelq6GIGxEXaDCVMBn1jtnzTwP6v6aD0Y pCys8SWTVqIbvkirOPvxmq8tvdugOf7g== X-Received: by 2002:a25:2d11:0:b0:e05:900d:32e6 with SMTP id 3f1490d57ef6-e05d3c17338mr350147276.8.1721071319301; Mon, 15 Jul 2024 12:21:59 -0700 (PDT) X-Google-Smtp-Source: AGHT+IFmDbyBksbC4qmpVrfmmZ0A/VGjIJHLQ2OfuPZrkWmoJhvnJgk6xUDZyR/TdOg4zaQrzDq1HQ== X-Received: by 2002:a25:2d11:0:b0:e05:900d:32e6 with SMTP id 3f1490d57ef6-e05d3c17338mr350112276.8.1721071318815; Mon, 15 Jul 2024 12:21:58 -0700 (PDT) Received: from x1n.redhat.com (pool-99-254-121-117.cpe.net.cable.rogers.com. [99.254.121.117]) by smtp.gmail.com with ESMTPSA id 6a1803df08f44-6b761978d30sm24039356d6.31.2024.07.15.12.21.57 (version=TLS1_3 cipher=TLS_AES_256_GCM_SHA384 bits=256/256); Mon, 15 Jul 2024 12:21:58 -0700 (PDT) From: Peter Xu To: linux-mm@kvack.org, linux-kernel@vger.kernel.org Cc: Dave Jiang , Rik van Riel , Dave Hansen , Michael Ellerman , linuxppc-dev@lists.ozlabs.org, Matthew Wilcox , Rick P Edgecombe , peterx@redhat.com, Oscar Salvador , Mel Gorman , Andrew Morton , Borislav Petkov , Christophe Leroy , Huang Ying , "Kirill A . Shutemov" , "Aneesh Kumar K . V" , Dan Williams , Thomas Gleixner , Hugh Dickins , x86@kernel.org, Nicholas Piggin , Vlastimil Babka , Ingo Molnar Subject: [PATCH v3 6/8] mm/x86: arch_check_zapped_pud() Date: Mon, 15 Jul 2024 15:21:40 -0400 Message-ID: <20240715192142.3241557-7-peterx@redhat.com> X-Mailer: git-send-email 2.45.0 In-Reply-To: <20240715192142.3241557-1-peterx@redhat.com> References: <20240715192142.3241557-1-peterx@redhat.com> MIME-Version: 1.0 X-Mimecast-Spam-Score: 0 X-Mimecast-Originator: redhat.com X-Rspamd-Server: rspam07 X-Rspamd-Queue-Id: 493EC8001B X-Stat-Signature: bx3a6hp9dhq9zcdfoqj8pazj7a49r1u1 X-Rspam-User: X-HE-Tag: 1721071322-963841 X-HE-Meta: U2FsdGVkX1/iX+62QAxYhCGw/Z3++F29KhC1Q2sZUBhZMhCZ3kKEYGG8SEUExKktk3WA1nwEJrP2EYn7SadGYHpurDYiDpWjdsd8GIDZKx9C7UAj65u2/o97kZdq4wmZtZ9G5x/uGyafWTf49/uCPZdcMzJ4nSCckGDOTMbYYslRmaQXEqvG3WYfQcaY3DOBAHRYf8g3Lh6asIdqAmP2bONAAEtTP1vFC/bI4YRqWI62Gc4EekyFU4xHMUrLENezmqzMy4yWNUx0cabyInFkZeqA63HTccxtvArgAdIOVEhns1FcdmRtiHPt6OEJf1yS/hatXxruL/zBSbPk8FHo7xkrDQsdswNmbaC4NaqZiMuWYMGoQZdCUH4S6BEXeaE7/QQ/iiZMH752qJKn/RPLKbkw1uWHN73ZVegGEw6dwBLXrJjAvkDXbjvF1dN7E5w29oqZyIi9wfsgfACe/IN5vM7dOgudVPCfCwsxJklEnxesW/wc1OX8SWRYmfUF3uy3HzZ6uNWiMju2WN+huICZ1LaWthQI0NM1ggxAY03/5Txci4vxZlQijvlME4zHxUeNhanPEUb741fn7MY6pw1Bo0KBOZFFvLIhkUGOK10A02HQJl8hefc3/PXLNG0BdtVMb1Dyl/OjRu2iQUgH+qJY6DTj55FTEiBTPvU7VYj8oln25Or/Vr1PiiKmnZOdyrCxRkn9ja5VOvTQQnhajOaLvXdKNzdq7OG/zi0q5mDvBJlubXOR9fbzTcXxE+ZbgkRkpYBYv/rjlcYB6YlvOfx0jP/2NnlJ3qeztxwcIcbx4zNIrXt/mVxYZ7tdut06gx1+rxYnKFsaDAEe68Uy7tPHPr6Ef4TxxXnlC/JtjfsiJ03qonXjyEkZU+2eIBkcuFm3AMMEKaBcjsst+2rzI9/8QpZduyo7B3kdz1/BgBYzEiC0H5S9BizVJgN7DZU9BRA8zW7+Je99VO7AOk2dVys oXeboiwO SFV72rgPLsA15jFELI3zzv1GZft0ltNn7CyvbXu7s2qJsqDO/Z9QS32SxF/QLMURxHODdhyuzEqz+70WdGy1SKKGjED8JsycwCvmpv9/2E3eEpPIo6AAliue/RG/sE/SkKiK0g9n1urSC0/8dFn3VouaEFjLsH8JdEUIAk3O7ebZ+UmY4dX1SMOdE4fGEeRDxIMRl5Y7AEP8zgf7QtqEfqBdntIQwKy6P2/k01xIp7KpNm0hFzV3mAUT7MOaC+v8xfMs/tkmTxkFfPZO0gAzJee5U1twEUPRBjp076ikEJW4xbTDd1v/MwZn5pndydDDJxaOw+DMHC8bnkTzgtIntKUFDXtuCj4JQ9ESbLZB8+vFJ45dh8ki5TpB5Angopp+Hyha6g1vBfVhEyQdGsdIvo5KaoAZcNlPV1ZvfzXj5MXWrHoosgggHRH6i4aalNYTdOgGLNdi+Vpi4CqAoLtUkYb66TKKgyghmeYzr4TJIFyy8e6onznHNvFggZKLpweEVJrzmLKCI1mC1Jsc= X-Bogosity: Ham, tests=bogofilter, spamicity=0.000000, version=1.2.4 Sender: owner-linux-mm@kvack.org Precedence: bulk X-Loop: owner-majordomo@kvack.org List-ID: List-Subscribe: List-Unsubscribe: Introduce arch_check_zapped_pud() to sanity check shadow stack on PUD zaps. It has the same logic of the PMD helper. One thing to mention is, it might be a good idea to use page_table_check in the future for trapping wrong setups of shadow stack pgtable entries [1]. That is left for the future as a separate effort. [1] https://lore.kernel.org/all/59d518698f664e07c036a5098833d7b56b953305.camel@intel.com Cc: "Edgecombe, Rick P" Cc: Thomas Gleixner Cc: Ingo Molnar Cc: Borislav Petkov Cc: Dave Hansen Cc: x86@kernel.org Signed-off-by: Peter Xu Acked-by: David Hildenbrand --- arch/x86/include/asm/pgtable.h | 10 ++++++++++ arch/x86/mm/pgtable.c | 7 +++++++ include/linux/pgtable.h | 7 +++++++ mm/huge_memory.c | 4 +++- 4 files changed, 27 insertions(+), 1 deletion(-) diff --git a/arch/x86/include/asm/pgtable.h b/arch/x86/include/asm/pgtable.h index 25fc6d809572..cdf044c2ad6e 100644 --- a/arch/x86/include/asm/pgtable.h +++ b/arch/x86/include/asm/pgtable.h @@ -169,6 +169,13 @@ static inline int pud_young(pud_t pud) return pud_flags(pud) & _PAGE_ACCESSED; } +static inline bool pud_shstk(pud_t pud) +{ + return cpu_feature_enabled(X86_FEATURE_SHSTK) && + (pud_flags(pud) & (_PAGE_RW | _PAGE_DIRTY | _PAGE_PSE)) == + (_PAGE_DIRTY | _PAGE_PSE); +} + static inline int pte_write(pte_t pte) { /* @@ -1662,6 +1669,9 @@ void arch_check_zapped_pte(struct vm_area_struct *vma, pte_t pte); #define arch_check_zapped_pmd arch_check_zapped_pmd void arch_check_zapped_pmd(struct vm_area_struct *vma, pmd_t pmd); +#define arch_check_zapped_pud arch_check_zapped_pud +void arch_check_zapped_pud(struct vm_area_struct *vma, pud_t pud); + #ifdef CONFIG_XEN_PV #define arch_has_hw_nonleaf_pmd_young arch_has_hw_nonleaf_pmd_young static inline bool arch_has_hw_nonleaf_pmd_young(void) diff --git a/arch/x86/mm/pgtable.c b/arch/x86/mm/pgtable.c index 93e54ba91fbf..564b8945951e 100644 --- a/arch/x86/mm/pgtable.c +++ b/arch/x86/mm/pgtable.c @@ -926,3 +926,10 @@ void arch_check_zapped_pmd(struct vm_area_struct *vma, pmd_t pmd) VM_WARN_ON_ONCE(!(vma->vm_flags & VM_SHADOW_STACK) && pmd_shstk(pmd)); } + +void arch_check_zapped_pud(struct vm_area_struct *vma, pud_t pud) +{ + /* See note in arch_check_zapped_pte() */ + VM_WARN_ON_ONCE(!(vma->vm_flags & VM_SHADOW_STACK) && + pud_shstk(pud)); +} diff --git a/include/linux/pgtable.h b/include/linux/pgtable.h index 2a6a3cccfc36..2289e9f7aa1b 100644 --- a/include/linux/pgtable.h +++ b/include/linux/pgtable.h @@ -447,6 +447,13 @@ static inline void arch_check_zapped_pmd(struct vm_area_struct *vma, } #endif +#ifndef arch_check_zapped_pud +static inline void arch_check_zapped_pud(struct vm_area_struct *vma, + pud_t pud) +{ +} +#endif + #ifndef __HAVE_ARCH_PTEP_GET_AND_CLEAR static inline pte_t ptep_get_and_clear(struct mm_struct *mm, unsigned long address, diff --git a/mm/huge_memory.c b/mm/huge_memory.c index 9fec5bd1c8b0..c10247bef08a 100644 --- a/mm/huge_memory.c +++ b/mm/huge_memory.c @@ -2291,12 +2291,14 @@ int zap_huge_pud(struct mmu_gather *tlb, struct vm_area_struct *vma, pud_t *pud, unsigned long addr) { spinlock_t *ptl; + pud_t orig_pud; ptl = __pud_trans_huge_lock(pud, vma); if (!ptl) return 0; - pudp_huge_get_and_clear_full(vma, addr, pud, tlb->fullmm); + orig_pud = pudp_huge_get_and_clear_full(vma, addr, pud, tlb->fullmm); + arch_check_zapped_pud(vma, orig_pud); tlb_remove_pud_tlb_entry(tlb, pud, addr); if (vma_is_special_huge(vma)) { spin_unlock(ptl); From patchwork Mon Jul 15 19:21:41 2024 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Peter Xu X-Patchwork-Id: 13733838 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from kanga.kvack.org (kanga.kvack.org [205.233.56.17]) by smtp.lore.kernel.org (Postfix) with ESMTP id 19A5DC3DA5E for ; Mon, 15 Jul 2024 19:22:13 +0000 (UTC) Received: by kanga.kvack.org (Postfix) id CFB756B00A2; Mon, 15 Jul 2024 15:22:06 -0400 (EDT) Received: by kanga.kvack.org (Postfix, from userid 40) id CAA2E6B00A1; Mon, 15 Jul 2024 15:22:06 -0400 (EDT) X-Delivered-To: int-list-linux-mm@kvack.org Received: by kanga.kvack.org (Postfix, from userid 63042) id AD51E6B00A2; Mon, 15 Jul 2024 15:22:06 -0400 (EDT) X-Delivered-To: linux-mm@kvack.org Received: from relay.hostedemail.com (smtprelay0013.hostedemail.com [216.40.44.13]) by kanga.kvack.org (Postfix) with ESMTP id 8E8176B00A0 for ; Mon, 15 Jul 2024 15:22:06 -0400 (EDT) Received: from smtpin20.hostedemail.com (a10.router.float.18 [10.200.18.1]) by unirelay02.hostedemail.com (Postfix) with ESMTP id 47AE7120176 for ; Mon, 15 Jul 2024 19:22:06 +0000 (UTC) X-FDA: 82342957452.20.478A2F2 Received: from us-smtp-delivery-124.mimecast.com (us-smtp-delivery-124.mimecast.com [170.10.129.124]) by imf27.hostedemail.com (Postfix) with ESMTP id 3A4C940020 for ; Mon, 15 Jul 2024 19:22:04 +0000 (UTC) Authentication-Results: imf27.hostedemail.com; dkim=pass header.d=redhat.com header.s=mimecast20190719 header.b=VP0+2JKp; spf=pass (imf27.hostedemail.com: domain of peterx@redhat.com designates 170.10.129.124 as permitted sender) smtp.mailfrom=peterx@redhat.com; dmarc=pass (policy=none) header.from=redhat.com ARC-Seal: i=1; s=arc-20220608; d=hostedemail.com; t=1721071305; a=rsa-sha256; cv=none; b=GENN8W7goUiRCF71x8aowFPR7e2d5u+Y34qMnCCtB4/6PCWFD9OXR6xpmuF/f0dCNEew4G 14q/nqOjhDBNtsMcoXyUxcz4+WUGsLV4udUm/6SXsy69Ix0heEs2T3OSd5UmXMTOhTTfJ0 DY3G07YYfvH2ynYhl5KLBXI7JlQJZms= ARC-Authentication-Results: i=1; imf27.hostedemail.com; dkim=pass header.d=redhat.com header.s=mimecast20190719 header.b=VP0+2JKp; spf=pass (imf27.hostedemail.com: domain of peterx@redhat.com designates 170.10.129.124 as permitted sender) smtp.mailfrom=peterx@redhat.com; dmarc=pass (policy=none) header.from=redhat.com ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=hostedemail.com; s=arc-20220608; t=1721071305; h=from:from:sender:reply-to:subject:subject:date:date: message-id:message-id:to:to:cc:cc:mime-version:mime-version: content-type:content-type: content-transfer-encoding:content-transfer-encoding: in-reply-to:in-reply-to:references:references:dkim-signature; bh=TIbZhmDyeGPrqbKoRiCTCUG6/nrhKTfNHRA57Ap3jGE=; b=HAL3UUryiZx7dytXw5T51YYrd+3s9zARew00dPyAnNM1isp+j+xwnPf2c/vdRUnM7+cntN Nm3rYjEKD1HCybzjqrpmRqFKDkNOE4Q9zlkmyId2+62d8Oh3tIoYWLYEqsunpIHHYutEgx lo0btG4W99WBf+xhPf07u778mz56NUA= DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=redhat.com; s=mimecast20190719; t=1721071323; h=from:from:reply-to:subject:subject:date:date:message-id:message-id: to:to:cc:cc:mime-version:mime-version:content-type:content-type: content-transfer-encoding:content-transfer-encoding: in-reply-to:in-reply-to:references:references; bh=TIbZhmDyeGPrqbKoRiCTCUG6/nrhKTfNHRA57Ap3jGE=; b=VP0+2JKpbQTNj9KfJP5+2/o9U+59W/bnl2oBVYQtsZzPgShp9Xtn6gXALbtgW0bBKKE7Zs gV0i2G5CtXFX0wCNy6D8X0unQo5p+MkzUm56XfckT4VpAaXafGVgvwvtNsx8p9Mlizcg53 sq5r1cjk5GFwdOX5q3EKz29GicLlvds= Received: from mail-qv1-f71.google.com (mail-qv1-f71.google.com [209.85.219.71]) by relay.mimecast.com with ESMTP with STARTTLS (version=TLSv1.3, cipher=TLS_AES_256_GCM_SHA384) id us-mta-53-R-qjuoU8MO2bzD2BYRue6w-1; Mon, 15 Jul 2024 15:22:02 -0400 X-MC-Unique: R-qjuoU8MO2bzD2BYRue6w-1 Received: by mail-qv1-f71.google.com with SMTP id 6a1803df08f44-6b7740643fbso2326926d6.1 for ; Mon, 15 Jul 2024 12:22:02 -0700 (PDT) X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20230601; t=1721071321; x=1721676121; h=content-transfer-encoding:mime-version:references:in-reply-to :message-id:date:subject:cc:to:from:x-gm-message-state:from:to:cc :subject:date:message-id:reply-to; bh=TIbZhmDyeGPrqbKoRiCTCUG6/nrhKTfNHRA57Ap3jGE=; b=wlF1lk98elgMsQvAoOOfU+2giJd3TE5AyNMlo/2GoE5eGc8brYgKx1aKYb+vOcHTTA L9eFuC+ujgZrygIbBZ/yYh1AbeGT5b3R6Sp9bBKVn2Fxb/oF4ImuJLdrs52QXYy3+hyf kI407SMbC2UWHEREwrkhTZNXkZXcx2gC4EV4Iw/iygEOx7PWawUtMBjT3TJTO5MQBxqh WUeePA2Bm4dPzKp9mLqMEgsxovydRxk87F8npS+4cysHmUYQ9YCDRAtu3rnP7qtA+hwE 8e37ukyApDWVFeHdWb7C5eC7DEwz9D3MpaTERD5MDqKJQwlxA7Iu26Ag2DAGllPwptqU KvWw== X-Gm-Message-State: AOJu0YwCwGcWbKhb9BalKGP7jpbVIUcJRAOhZMjIkz7Tj6o+iz5xitdR /Zqg4F4cX4/E0DXRThBZYjbQCqkpD3zGgUgwCZKMlOHnDGyCBte1mVkzHh/iuhSFML14/TJRLnb z+l3ES3Gq00kmNWVTLbjcrkklbbTKMVGVqoE8aI68LWSfdbDvZLPN332OGtq7KnVc1dpSFkbWxd Q+Zj+M7kHvDyefpS9sfMBv67tQlNFwrw== X-Received: by 2002:a05:6214:5d0b:b0:6b5:38d4:c9c5 with SMTP id 6a1803df08f44-6b77df20d64mr4194976d6.4.1721071321213; Mon, 15 Jul 2024 12:22:01 -0700 (PDT) X-Google-Smtp-Source: AGHT+IEuNyAqmev7NeecbFUrfKh3dH4xJxhydhorfZ+q73TgeQxZK3QM+o4zZ2VKk8paCO1QT0QczQ== X-Received: by 2002:a05:6214:5d0b:b0:6b5:38d4:c9c5 with SMTP id 6a1803df08f44-6b77df20d64mr4194626d6.4.1721071320743; Mon, 15 Jul 2024 12:22:00 -0700 (PDT) Received: from x1n.redhat.com (pool-99-254-121-117.cpe.net.cable.rogers.com. [99.254.121.117]) by smtp.gmail.com with ESMTPSA id 6a1803df08f44-6b761978d30sm24039356d6.31.2024.07.15.12.21.59 (version=TLS1_3 cipher=TLS_AES_256_GCM_SHA384 bits=256/256); Mon, 15 Jul 2024 12:22:00 -0700 (PDT) From: Peter Xu To: linux-mm@kvack.org, linux-kernel@vger.kernel.org Cc: Dave Jiang , Rik van Riel , Dave Hansen , Michael Ellerman , linuxppc-dev@lists.ozlabs.org, Matthew Wilcox , Rick P Edgecombe , peterx@redhat.com, Oscar Salvador , Mel Gorman , Andrew Morton , Borislav Petkov , Christophe Leroy , Huang Ying , "Kirill A . Shutemov" , "Aneesh Kumar K . V" , Dan Williams , Thomas Gleixner , Hugh Dickins , x86@kernel.org, Nicholas Piggin , Vlastimil Babka , Ingo Molnar Subject: [PATCH v3 7/8] mm/x86: Add missing pud helpers Date: Mon, 15 Jul 2024 15:21:41 -0400 Message-ID: <20240715192142.3241557-8-peterx@redhat.com> X-Mailer: git-send-email 2.45.0 In-Reply-To: <20240715192142.3241557-1-peterx@redhat.com> References: <20240715192142.3241557-1-peterx@redhat.com> MIME-Version: 1.0 X-Mimecast-Spam-Score: 0 X-Mimecast-Originator: redhat.com X-Stat-Signature: rq8x8ywghryt7et4yiz55k5hg5k9appb X-Rspamd-Queue-Id: 3A4C940020 X-Rspam-User: X-Rspamd-Server: rspam10 X-HE-Tag: 1721071324-963366 X-HE-Meta: U2FsdGVkX1/0TBFhT6a0c+E8l/pFg1enqlkXAfaEi6wwzXCR/ZRehR7nY8Q4JArUn3AKePf3Qomx5ZRcZ38K0EWxuXF2oYSo1p51FuYE9lSisEJSsN1zuKdimgL2B3w2fms1KGTLiqASzuaVX5z1lPXcO/m8fB+9wXMShhdDH9B/reEnwBD3E4j2BopOyMlkdwVmVaTxQ2EDiJaj1y2oftk3dFFd7UnnYB8ubuxr1XOFd0lMwPgPh3Uq8a94MYn+8I6hwOSLqbc6iQezdTN/oAqeuJuCifKJUGseLZ4+wzNU8Z4GZ1OJsodizDgYJlhmAxLe7F108IvZn/ihRfcvowOyxAXZnmdFMz8LTZkLWd+cdspgDbAGk03ToDYOUCWCObCSSsVxaFsOhKbA+E6icqt9oLMfRVSr0ZUMqqRhIc7bqf1oMcEgVn4jOpo1qjDrI5dc1c5Gu/f3sTybh4lZYliVkbFgcXpsTorSbhwwipi784lBchQMNp0MANj7wPGsfRlyxeOh6Bh7l3DVtH0TgUj1DvxCi7Yw2YvMVimC8YHZNJKHSU3yp6+7q8uQ1LtqrdBOQDO2FFCqYrV8eAxs7TChSDhptUezNxH0KPLJWelWmP7RuIChzQ6w51jJqZkWmOsu98ZkOOtMxVgm3Oy6g1o3HHQbkQJiJqXuuR+Dc1S4GsOsExsNVml0U1SfzIZQjzneI7a4IbuN47eXpSADkQpahbRSGqu2pAJ2MDR+qRr/JQWxeU+XhM3Dgduahx3z7ze+ZjcHzxCs+ayydewEgBC7cEMzQ7rYzw0kEpyJsTu3K3Th8FtmAZMBCkR0aoaHSOxeh715RrTv4PWv65NOUj4Ue5RS9f3bqbj6+SzwrX1PONA0rd7qMviaQSW4ppG+lgA7CiX8jJFDQ2n9tKi4+k1CuWrwHA1Ic9O6G85X0kHfvcSRAvm/zdwgrI/0tJu9RQlbJatN3MxAHMjJAV2 4EZg1wUT RQ2qwDf+OcMovxdRq3+YDqSZUNca6HV7BJ7qjaIQEaqTM82+L2fRAj/1iZlJ7dwDoMqixILWn9WBVS5WwcttogfVI1f0zHjiDPdZ2J7LVyeqRtpxGWqmBQae9SmmPxQZSCNLCNCz6yKOB93/Snj4XznufuQFU/O8nNTAvdtU8+14h/FY/6fDDQiVc77XtvE30Or2qf/Yax9jeGOUvbSj+klO81FhTnXcs0QvLOCBXGOHj0CFgoSlZ+IxKn7s36xFo81l39gZrucMwnRb/C9dcrrIGfjyKaPJ86vr39U7e843YWbFVh2lDbdlfR8ReuDnDwS/7XY6XntYI54a1SBINXs+Wmb+ehMvdhaVSLkUh+w/Y0MyUSBIuo3rBiFVvbzKamJaxQ6Q5cFcwO83uKhwevT2MSw1DAA/H1Nd+EIFrHTBYWj4= X-Bogosity: Ham, tests=bogofilter, spamicity=0.000000, version=1.2.4 Sender: owner-linux-mm@kvack.org Precedence: bulk X-Loop: owner-majordomo@kvack.org List-ID: List-Subscribe: List-Unsubscribe: These new helpers will be needed for pud entry updates soon. Introduce these helpers by referencing the pmd ones. Namely: - pudp_invalidate() - pud_modify() Cc: Thomas Gleixner Cc: Ingo Molnar Cc: Borislav Petkov Cc: Dave Hansen Cc: x86@kernel.org Signed-off-by: Peter Xu --- arch/x86/include/asm/pgtable.h | 55 +++++++++++++++++++++++++++++----- arch/x86/mm/pgtable.c | 12 ++++++++ 2 files changed, 59 insertions(+), 8 deletions(-) diff --git a/arch/x86/include/asm/pgtable.h b/arch/x86/include/asm/pgtable.h index cdf044c2ad6e..701593c53f3b 100644 --- a/arch/x86/include/asm/pgtable.h +++ b/arch/x86/include/asm/pgtable.h @@ -782,6 +782,12 @@ static inline pmd_t pmd_mkinvalid(pmd_t pmd) __pgprot(pmd_flags(pmd) & ~(_PAGE_PRESENT|_PAGE_PROTNONE))); } +static inline pud_t pud_mkinvalid(pud_t pud) +{ + return pfn_pud(pud_pfn(pud), + __pgprot(pud_flags(pud) & ~(_PAGE_PRESENT|_PAGE_PROTNONE))); +} + static inline u64 flip_protnone_guard(u64 oldval, u64 val, u64 mask); static inline pte_t pte_modify(pte_t pte, pgprot_t newprot) @@ -829,14 +835,8 @@ static inline pmd_t pmd_modify(pmd_t pmd, pgprot_t newprot) pmd_result = __pmd(val); /* - * To avoid creating Write=0,Dirty=1 PMDs, pte_modify() needs to avoid: - * 1. Marking Write=0 PMDs Dirty=1 - * 2. Marking Dirty=1 PMDs Write=0 - * - * The first case cannot happen because the _PAGE_CHG_MASK will filter - * out any Dirty bit passed in newprot. Handle the second case by - * going through the mksaveddirty exercise. Only do this if the old - * value was Write=1 to avoid doing this on Shadow Stack PTEs. + * Avoid creating shadow stack PMD by accident. See comment in + * pte_modify(). */ if (oldval & _PAGE_RW) pmd_result = pmd_mksaveddirty(pmd_result); @@ -846,6 +846,29 @@ static inline pmd_t pmd_modify(pmd_t pmd, pgprot_t newprot) return pmd_result; } +static inline pud_t pud_modify(pud_t pud, pgprot_t newprot) +{ + pudval_t val = pud_val(pud), oldval = val; + pud_t pud_result; + + val &= _HPAGE_CHG_MASK; + val |= check_pgprot(newprot) & ~_HPAGE_CHG_MASK; + val = flip_protnone_guard(oldval, val, PHYSICAL_PUD_PAGE_MASK); + + pud_result = __pud(val); + + /* + * Avoid creating shadow stack PUD by accident. See comment in + * pte_modify(). + */ + if (oldval & _PAGE_RW) + pud_result = pud_mksaveddirty(pud_result); + else + pud_result = pud_clear_saveddirty(pud_result); + + return pud_result; +} + /* * mprotect needs to preserve PAT and encryption bits when updating * vm_page_prot @@ -1384,10 +1407,26 @@ static inline pmd_t pmdp_establish(struct vm_area_struct *vma, } #endif +static inline pud_t pudp_establish(struct vm_area_struct *vma, + unsigned long address, pud_t *pudp, pud_t pud) +{ + page_table_check_pud_set(vma->vm_mm, pudp, pud); + if (IS_ENABLED(CONFIG_SMP)) { + return xchg(pudp, pud); + } else { + pud_t old = *pudp; + WRITE_ONCE(*pudp, pud); + return old; + } +} + #define __HAVE_ARCH_PMDP_INVALIDATE_AD extern pmd_t pmdp_invalidate_ad(struct vm_area_struct *vma, unsigned long address, pmd_t *pmdp); +pud_t pudp_invalidate(struct vm_area_struct *vma, unsigned long address, + pud_t *pudp); + /* * Page table pages are page-aligned. The lower half of the top * level is used for userspace and the top half for the kernel. diff --git a/arch/x86/mm/pgtable.c b/arch/x86/mm/pgtable.c index 564b8945951e..fa77411bb266 100644 --- a/arch/x86/mm/pgtable.c +++ b/arch/x86/mm/pgtable.c @@ -641,6 +641,18 @@ pmd_t pmdp_invalidate_ad(struct vm_area_struct *vma, unsigned long address, } #endif +#if defined(CONFIG_TRANSPARENT_HUGEPAGE) && \ + defined(CONFIG_HAVE_ARCH_TRANSPARENT_HUGEPAGE_PUD) +pud_t pudp_invalidate(struct vm_area_struct *vma, unsigned long address, + pud_t *pudp) +{ + VM_WARN_ON_ONCE(!pud_present(*pudp)); + pud_t old = pudp_establish(vma, address, pudp, pud_mkinvalid(*pudp)); + flush_pud_tlb_range(vma, address, address + HPAGE_PUD_SIZE); + return old; +} +#endif + /** * reserve_top_address - reserves a hole in the top of kernel address space * @reserve - size of hole to reserve From patchwork Mon Jul 15 19:21:42 2024 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Peter Xu X-Patchwork-Id: 13733839 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from kanga.kvack.org (kanga.kvack.org [205.233.56.17]) by smtp.lore.kernel.org (Postfix) with ESMTP id 851A8C3DA59 for ; Mon, 15 Jul 2024 19:22:16 +0000 (UTC) Received: by kanga.kvack.org (Postfix) id DE57D6B00A1; Mon, 15 Jul 2024 15:22:10 -0400 (EDT) Received: by kanga.kvack.org (Postfix, from userid 40) id D940E6B00A3; Mon, 15 Jul 2024 15:22:10 -0400 (EDT) X-Delivered-To: int-list-linux-mm@kvack.org Received: by kanga.kvack.org (Postfix, from userid 63042) id B9C3A6B00A4; Mon, 15 Jul 2024 15:22:10 -0400 (EDT) X-Delivered-To: linux-mm@kvack.org Received: from relay.hostedemail.com (smtprelay0016.hostedemail.com [216.40.44.16]) by kanga.kvack.org (Postfix) with ESMTP id 97C786B00A1 for ; Mon, 15 Jul 2024 15:22:10 -0400 (EDT) Received: from smtpin08.hostedemail.com (a10.router.float.18 [10.200.18.1]) by unirelay10.hostedemail.com (Postfix) with ESMTP id 3B246C01D7 for ; Mon, 15 Jul 2024 19:22:10 +0000 (UTC) X-FDA: 82342957620.08.1D1FEDC Received: from us-smtp-delivery-124.mimecast.com (us-smtp-delivery-124.mimecast.com [170.10.133.124]) by imf25.hostedemail.com (Postfix) with ESMTP id 20D46A0020 for ; Mon, 15 Jul 2024 19:22:06 +0000 (UTC) Authentication-Results: imf25.hostedemail.com; dkim=pass header.d=redhat.com header.s=mimecast20190719 header.b=UI+hJyqU; spf=pass (imf25.hostedemail.com: domain of peterx@redhat.com designates 170.10.133.124 as permitted sender) smtp.mailfrom=peterx@redhat.com; dmarc=pass (policy=none) header.from=redhat.com ARC-Seal: i=1; s=arc-20220608; d=hostedemail.com; t=1721071307; a=rsa-sha256; cv=none; b=KUCtqh515ilWm2XQKSodY/bbCr54+ICKfc3jE0ovRCODS36W8VDMFhpKFEvVpHYz3kcjBZ +5S7be88e737lsQzmE1DqeNpCT7EcRYQn8qVjBQ186vZF7S5fDfKjIp36wPCtUP7CZmg2M N9UJWM4kD81uscOxE1gLRuW52p21Fqs= ARC-Authentication-Results: i=1; imf25.hostedemail.com; dkim=pass header.d=redhat.com header.s=mimecast20190719 header.b=UI+hJyqU; spf=pass (imf25.hostedemail.com: domain of peterx@redhat.com designates 170.10.133.124 as permitted sender) smtp.mailfrom=peterx@redhat.com; dmarc=pass (policy=none) header.from=redhat.com ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=hostedemail.com; s=arc-20220608; t=1721071307; h=from:from:sender:reply-to:subject:subject:date:date: message-id:message-id:to:to:cc:cc:mime-version:mime-version: content-type:content-type: content-transfer-encoding:content-transfer-encoding: in-reply-to:in-reply-to:references:references:dkim-signature; bh=MX5pdN1kDqdO60LUKursnvEFOQP0WFBlMnoZkGHINQQ=; b=BJspfY2tOQNnEvxLjxHartxEiVOgl8IIceIGdj/vg+xkOWNwRxHQBPkfxin1ALezmEm3Vc 08AgW3yDTd7Elre9GFDuC97lzSP3MZ7I9K+PDN2jUYe+GHn3OoKLRNlfFEMC/fsu+qj4gm SAmPOV/81HNoNChnXaYJHH0H9mFPytM= DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=redhat.com; s=mimecast20190719; t=1721071326; h=from:from:reply-to:subject:subject:date:date:message-id:message-id: to:to:cc:cc:mime-version:mime-version:content-type:content-type: content-transfer-encoding:content-transfer-encoding: in-reply-to:in-reply-to:references:references; bh=MX5pdN1kDqdO60LUKursnvEFOQP0WFBlMnoZkGHINQQ=; b=UI+hJyqUVVdqJoEJLwsMduIF4ljku5V6zHUvCiT05Vyc9jNV9FYpJdznGV2azGx+XzfNQs 1v498JyRoen0kqFKybyFRtk3GOuUmAeApBvUYs7WJNNYAM15B2PmX04Zu+/A9hnIm1nZXs BZHAcrt9Nqc0kow0MdUJEeTz02Mxy3Y= Received: from mail-qv1-f72.google.com (mail-qv1-f72.google.com [209.85.219.72]) by relay.mimecast.com with ESMTP with STARTTLS (version=TLSv1.3, cipher=TLS_AES_256_GCM_SHA384) id us-mta-328-zaq30LGIMIC2uxjlzcFdBA-1; Mon, 15 Jul 2024 15:22:05 -0400 X-MC-Unique: zaq30LGIMIC2uxjlzcFdBA-1 Received: by mail-qv1-f72.google.com with SMTP id 6a1803df08f44-6b60afbf5d1so14885756d6.0 for ; Mon, 15 Jul 2024 12:22:05 -0700 (PDT) X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20230601; t=1721071323; x=1721676123; h=content-transfer-encoding:mime-version:references:in-reply-to :message-id:date:subject:cc:to:from:x-gm-message-state:from:to:cc :subject:date:message-id:reply-to; bh=MX5pdN1kDqdO60LUKursnvEFOQP0WFBlMnoZkGHINQQ=; b=NcmWHaIzmwHK633l/0++ZV7wn1wUYjTrsGfdI5CHYWj0ysZ7wD7sAxiOJXB0HBwDf0 n/x9Vai2/QSFpLsVIIFtAUFNpSCtsMclxl9ClXK4NgdID4RZANA/3d8Iep4/ZFu7Koo4 e3U3b0XKx6YN7hucaCTqRHvpu5KfEJCglUYCUsENmmfpoPPtquw9+Ekekm/YIAKNCf5C xs3bZx2rLJAmEkAjz8RTglwjh1T4sNEcGJ6C0EMYd8Qrz2pKMYW40SejayoghLiyEITx cWZNXbl/hSIM6Hip2Uu4oCwruRxssDvJOjNubwwmOPAmkP8BIYulAkEU31qvbtD5/8Pa +UtA== X-Gm-Message-State: AOJu0YyQ1jo67PR3JaZzaC1iEM43hsn+bZDn/a34JGojJ5oH6O4pYAdU vv4F0Thr9QYN5arI/a/fKnML/splF4CZ1bgyQbV+UobYfhDHdqmJzGO05a8fYoE/3ILDp+0uOrs aCzSN62sNcpIlAXgNsSbMz/C0j7L+P8rOwwBRxZNYj43TPTRjIpyaSXL5gKgpa2eYH+dEE6Mf6C R9IStRVmwlN40Ik7R7bF9LdmSaHqv+RA== X-Received: by 2002:a05:6214:3d9c:b0:6b7:586c:6db with SMTP id 6a1803df08f44-6b77e1a9bbbmr4347926d6.9.1721071323016; Mon, 15 Jul 2024 12:22:03 -0700 (PDT) X-Google-Smtp-Source: AGHT+IGQhnzZI26fHLI1GP/caKDLHgduuPT+eLqK9zXdARycJ8bkdfse6d6/hJ9cmtHwcRv8UZEfEQ== X-Received: by 2002:a05:6214:3d9c:b0:6b7:586c:6db with SMTP id 6a1803df08f44-6b77e1a9bbbmr4347496d6.9.1721071322604; Mon, 15 Jul 2024 12:22:02 -0700 (PDT) Received: from x1n.redhat.com (pool-99-254-121-117.cpe.net.cable.rogers.com. [99.254.121.117]) by smtp.gmail.com with ESMTPSA id 6a1803df08f44-6b761978d30sm24039356d6.31.2024.07.15.12.22.00 (version=TLS1_3 cipher=TLS_AES_256_GCM_SHA384 bits=256/256); Mon, 15 Jul 2024 12:22:02 -0700 (PDT) From: Peter Xu To: linux-mm@kvack.org, linux-kernel@vger.kernel.org Cc: Dave Jiang , Rik van Riel , Dave Hansen , Michael Ellerman , linuxppc-dev@lists.ozlabs.org, Matthew Wilcox , Rick P Edgecombe , peterx@redhat.com, Oscar Salvador , Mel Gorman , Andrew Morton , Borislav Petkov , Christophe Leroy , Huang Ying , "Kirill A . Shutemov" , "Aneesh Kumar K . V" , Dan Williams , Thomas Gleixner , Hugh Dickins , x86@kernel.org, Nicholas Piggin , Vlastimil Babka , Ingo Molnar Subject: [PATCH v3 8/8] mm/mprotect: fix dax pud handlings Date: Mon, 15 Jul 2024 15:21:42 -0400 Message-ID: <20240715192142.3241557-9-peterx@redhat.com> X-Mailer: git-send-email 2.45.0 In-Reply-To: <20240715192142.3241557-1-peterx@redhat.com> References: <20240715192142.3241557-1-peterx@redhat.com> MIME-Version: 1.0 X-Mimecast-Spam-Score: 0 X-Mimecast-Originator: redhat.com X-Stat-Signature: m5wpuoydjc55tif31pkxn7txocorhdpw X-Rspamd-Queue-Id: 20D46A0020 X-Rspam-User: X-Rspamd-Server: rspam10 X-HE-Tag: 1721071326-671807 X-HE-Meta: U2FsdGVkX1/zMB+rPA5GeG3I4dI+uZtf4VkrBqjbCFXvFPA4hV5rpyjf6Ekz+FBmTZmVig0uEDAbemidYjZRrJSSCYELeU4YYfBIgsWc71FykONVdK+FrEgZ1c3oukraCD9WDHhduWTTE3H/JXFyT+bUnyxiRoXgYbaiJLCiqXXWG1NE2CQC/nMRydfW6XRgTEr1etE1hLdlpo5l/jigiolHX/a3MYNIblzizsAlCr0Ht2DVGBCfpJP9YANYkPriwZbdKMKkEN7ZocQ/GW+5NfFigeCF8FfZgNjqGNUp2PZ3KVeh54ZZkJkNJa+awCYlUsb33a7H7lCtPm6rRRCrgvGFkgiZBIm8LgLXZBY1FV+GSRMPtiX3o7bexTgTU27kOqsFfQck+iYRT4OMERavXl2O2pME8O3C8uADU1U86Fo4frBElGhwHnlCykfOzz1CK1KrYYvQmXpVziNZxGv0rDeJoYsxFYemVkMrahspIrLCNQpW607T68PKY8Gv99NHVSfoGyzlTeY8ANkqNPX4gctL9uh1EqJH70AhCevEUyRtV7tASvgEmG4kaWZptrsSGba7XOBXSoV/5TflzuCjMOptiINiiGMXntHqso2Xq08coiFB9HADlipgA4FZA47z/61/ZD365cNPkXNoy/5sMJOcoFdbFroK+HsYbkZIu6wH1Tsrf35MdJzRfE6Q2OkXXqxZ1vzsJexlBYM6nGB5cQfYjKrCO+m+jwk1QEkCWrFIIU1+cunAKx/s9lD0GLbYQKjpXS2K+Jbdk44wYiHUqlhX3AcvhE9molmFnQ/4Tg78st9lNix2db6+t815NxokBZicqYpppHWeq+VPCM8NE/TMHCJh+RQ8KJBBRIhUTH8jsadrZIuErD8oWSxByYAlyKEQmOoyAg6igIPlB1VYPn0rP0aWQoMB62mq7nlY1nJKDz/yOWFHTeivZ74xQx2OmFuNLmM76OuvtIexPv+ oX1Spl4R NoDRsdwR7GW8NH5aKWyz4ClXExjgRVYRdwW6E9M2nd7MtORZqPyzi+wzzDzK/o507N+MnQPJutIesFDtOah1uRV8GJd7TIzFtg5sG5iBU3OU5X5BhJfzi3svvrId/c6KagoH/E4t25VTbjvYvzsPFi2af2/8bh7NN89k/U2hUfah19zCnwBwlfnsiIeiiOvnG7TLtb98oehnIxf2NsPr6tpLk8nKlJwSKdb4UHinyJmhYMZjJWYhmzASmMgFBlvvQlQoRc7O0ZYfZV8CJzXk9utNd7UjJ8hFyP78+jnKj1jawhpbodkKYb/j93SkhFk5Gtkv8oLKiZEF7t111JtrXwYD+AcEWoEKJxDkhDi2O7WBOdDweLWBWXfToMRqenGnrwiQvM2lk199CeD707Zf4mgGMKpnJgK5qt3Amad7SXALSNKZG3j9QfowrkMeAlG42MUBUGmjxiTO74MLS5nlrW+I/0AUFtZ6d3wzkpuGdrMvORE3eKKBdb6NG/d1VzoI1Lc+2ii1qK8/1P9CD4Ozjrv92wkii8Nb+HTwSCqXaOkIStr2Y+fwmWIDHNgOw8TK4d397rYhLALfLvhX3dc+hXzzCZ0iTs9MRYHbu/3V836NKgQ+lM3KVnjaGPX0ZzkaAmNuN X-Bogosity: Ham, tests=bogofilter, spamicity=0.000000, version=1.2.4 Sender: owner-linux-mm@kvack.org Precedence: bulk X-Loop: owner-majordomo@kvack.org List-ID: List-Subscribe: List-Unsubscribe: This is only relevant to the two archs that support PUD dax, aka, x86_64 and ppc64. PUD THPs do not yet exist elsewhere, and hugetlb PUDs do not count in this case. DAX have had PUD mappings for years, but change protection path never worked. When the path is triggered in any form (a simple test program would be: call mprotect() on a 1G dev_dax mapping), the kernel will report "bad pud". This patch should fix that. The new change_huge_pud() tries to keep everything simple. For example, it doesn't optimize write bit as that will need even more PUD helpers. It's not too bad anyway to have one more write fault in the worst case once for 1G range; may be a bigger thing for each PAGE_SIZE, though. Neither does it support userfault-wp bits, as there isn't such PUD mappings that is supported; file mappings always need a split there. The same to TLB shootdown: the pmd path (which was for x86 only) has the trick of using _ad() version of pmdp_invalidate*() which can avoid one redundant TLB, but let's also leave that for later. Again, the larger the mapping, the smaller of such effect. Another thing worth mention is this path needs to be careful on handling "retry" event for change_huge_pud() (where it can return 0): it isn't like change_huge_pmd(), as the pmd version is safe with all conditions handled in change_pte_range() later, thanks to Hugh's new pte_offset_map_lock(). In short, change_pte_range() is simply smarter than change_pmd_range() now after the shmem thp collapse rework. For that reason, change_pud_range() will need proper retry if it races with something else when a huge PUD changed from under us. Cc: Dan Williams Cc: Matthew Wilcox Cc: Dave Jiang Cc: Hugh Dickins Cc: Kirill A. Shutemov Cc: Vlastimil Babka Cc: Thomas Gleixner Cc: Ingo Molnar Cc: Borislav Petkov Cc: Dave Hansen Cc: Michael Ellerman Cc: Aneesh Kumar K.V Cc: Oscar Salvador Cc: x86@kernel.org Cc: linuxppc-dev@lists.ozlabs.org Fixes: a00cc7d9dd93 ("mm, x86: add support for PUD-sized transparent hugepages") Fixes: 27af67f35631 ("powerpc/book3s64/mm: enable transparent pud hugepage") Signed-off-by: Peter Xu --- include/linux/huge_mm.h | 24 +++++++++++++++++++ mm/huge_memory.c | 52 +++++++++++++++++++++++++++++++++++++++++ mm/mprotect.c | 34 ++++++++++++++++++++++----- 3 files changed, 104 insertions(+), 6 deletions(-) diff --git a/include/linux/huge_mm.h b/include/linux/huge_mm.h index cff002be83eb..6e742680590a 100644 --- a/include/linux/huge_mm.h +++ b/include/linux/huge_mm.h @@ -336,6 +336,17 @@ void split_huge_pmd_address(struct vm_area_struct *vma, unsigned long address, void __split_huge_pud(struct vm_area_struct *vma, pud_t *pud, unsigned long address); +#ifdef CONFIG_HAVE_ARCH_TRANSPARENT_HUGEPAGE_PUD +int change_huge_pud(struct mmu_gather *tlb, struct vm_area_struct *vma, + pud_t *pudp, unsigned long addr, pgprot_t newprot, + unsigned long cp_flags); +#else +static inline int +change_huge_pud(struct mmu_gather *tlb, struct vm_area_struct *vma, + pud_t *pudp, unsigned long addr, pgprot_t newprot, + unsigned long cp_flags) { return 0; } +#endif + #define split_huge_pud(__vma, __pud, __address) \ do { \ pud_t *____pud = (__pud); \ @@ -579,6 +590,19 @@ static inline int next_order(unsigned long *orders, int prev) { return 0; } + +static inline void __split_huge_pud(struct vm_area_struct *vma, pud_t *pud, + unsigned long address) +{ +} + +static inline int change_huge_pud(struct mmu_gather *tlb, + struct vm_area_struct *vma, pud_t *pudp, + unsigned long addr, pgprot_t newprot, + unsigned long cp_flags) +{ + return 0; +} #endif /* CONFIG_TRANSPARENT_HUGEPAGE */ static inline int split_folio_to_list_to_order(struct folio *folio, diff --git a/mm/huge_memory.c b/mm/huge_memory.c index c10247bef08a..9a00c5955c0c 100644 --- a/mm/huge_memory.c +++ b/mm/huge_memory.c @@ -2112,6 +2112,53 @@ int change_huge_pmd(struct mmu_gather *tlb, struct vm_area_struct *vma, return ret; } +/* + * Returns: + * + * - 0: if pud leaf changed from under us + * - 1: if pud can be skipped + * - HPAGE_PUD_NR: if pud was successfully processed + */ +#ifdef CONFIG_HAVE_ARCH_TRANSPARENT_HUGEPAGE_PUD +int change_huge_pud(struct mmu_gather *tlb, struct vm_area_struct *vma, + pud_t *pudp, unsigned long addr, pgprot_t newprot, + unsigned long cp_flags) +{ + struct mm_struct *mm = vma->vm_mm; + pud_t oldpud, entry; + spinlock_t *ptl; + + tlb_change_page_size(tlb, HPAGE_PUD_SIZE); + + /* NUMA balancing doesn't apply to dax */ + if (cp_flags & MM_CP_PROT_NUMA) + return 1; + + /* + * Huge entries on userfault-wp only works with anonymous, while we + * don't have anonymous PUDs yet. + */ + if (WARN_ON_ONCE(cp_flags & MM_CP_UFFD_WP_ALL)) + return 1; + + ptl = __pud_trans_huge_lock(pudp, vma); + if (!ptl) + return 0; + + /* + * Can't clear PUD or it can race with concurrent zapping. See + * change_huge_pmd(). + */ + oldpud = pudp_invalidate(vma, addr, pudp); + entry = pud_modify(oldpud, newprot); + set_pud_at(mm, addr, pudp, entry); + tlb_flush_pud_range(tlb, addr, HPAGE_PUD_SIZE); + + spin_unlock(ptl); + return HPAGE_PUD_NR; +} +#endif + #ifdef CONFIG_USERFAULTFD /* * The PT lock for src_pmd and dst_vma/src_vma (for reading) are locked by @@ -2342,6 +2389,11 @@ void __split_huge_pud(struct vm_area_struct *vma, pud_t *pud, spin_unlock(ptl); mmu_notifier_invalidate_range_end(&range); } +#else +void __split_huge_pud(struct vm_area_struct *vma, pud_t *pud, + unsigned long address) +{ +} #endif /* CONFIG_HAVE_ARCH_TRANSPARENT_HUGEPAGE_PUD */ static void __split_huge_zero_page_pmd(struct vm_area_struct *vma, diff --git a/mm/mprotect.c b/mm/mprotect.c index 2a81060b603d..694f13b83864 100644 --- a/mm/mprotect.c +++ b/mm/mprotect.c @@ -425,31 +425,53 @@ static inline long change_pud_range(struct mmu_gather *tlb, unsigned long end, pgprot_t newprot, unsigned long cp_flags) { struct mmu_notifier_range range; - pud_t *pud; + pud_t *pudp, pud; unsigned long next; long pages = 0, ret; range.start = 0; - pud = pud_offset(p4d, addr); + pudp = pud_offset(p4d, addr); do { +again: next = pud_addr_end(addr, end); - ret = change_prepare(vma, pud, pmd, addr, cp_flags); + ret = change_prepare(vma, pudp, pmd, addr, cp_flags); if (ret) { pages = ret; break; } - if (pud_none_or_clear_bad(pud)) + + pud = READ_ONCE(*pudp); + if (pud_none(pud)) continue; + if (!range.start) { mmu_notifier_range_init(&range, MMU_NOTIFY_PROTECTION_VMA, 0, vma->vm_mm, addr, end); mmu_notifier_invalidate_range_start(&range); } - pages += change_pmd_range(tlb, vma, pud, addr, next, newprot, + + if (pud_leaf(pud)) { + if ((next - addr != PUD_SIZE) || + pgtable_split_needed(vma, cp_flags)) { + __split_huge_pud(vma, pudp, addr); + goto again; + } else { + ret = change_huge_pud(tlb, vma, pudp, + addr, newprot, cp_flags); + if (ret == 0) + goto again; + /* huge pud was handled */ + if (ret == HPAGE_PUD_NR) + pages += HPAGE_PUD_NR; + continue; + } + } + + pages += change_pmd_range(tlb, vma, pudp, addr, next, newprot, cp_flags); - } while (pud++, addr = next, addr != end); + } while (pudp++, addr = next, addr != end); if (range.start) mmu_notifier_invalidate_range_end(&range);