From patchwork Fri Mar 4 05:17:00 2022 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Peter Xu X-Patchwork-Id: 12768470 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from kanga.kvack.org (kanga.kvack.org [205.233.56.17]) by smtp.lore.kernel.org (Postfix) with ESMTP id 9F72AC433EF for ; Fri, 4 Mar 2022 05:19:27 +0000 (UTC) Received: by kanga.kvack.org (Postfix) id 44DAF8D000D; Fri, 4 Mar 2022 00:19:27 -0500 (EST) Received: by kanga.kvack.org (Postfix, from userid 40) id 3FCFE8D0001; Fri, 4 Mar 2022 00:19:27 -0500 (EST) X-Delivered-To: int-list-linux-mm@kvack.org Received: by kanga.kvack.org (Postfix, from userid 63042) id 2ECE98D000D; Fri, 4 Mar 2022 00:19:27 -0500 (EST) X-Delivered-To: linux-mm@kvack.org Received: from forelay.hostedemail.com (smtprelay0055.hostedemail.com [216.40.44.55]) by kanga.kvack.org (Postfix) with ESMTP id 209788D0001 for ; Fri, 4 Mar 2022 00:19:27 -0500 (EST) Received: from smtpin31.hostedemail.com (10.5.19.251.rfc1918.com [10.5.19.251]) by forelay03.hostedemail.com (Postfix) with ESMTP id D338F8249980 for ; Fri, 4 Mar 2022 05:19:26 +0000 (UTC) X-FDA: 79205550732.31.D14A3BD Received: from us-smtp-delivery-124.mimecast.com (us-smtp-delivery-124.mimecast.com [170.10.129.124]) by imf20.hostedemail.com (Postfix) with ESMTP id 493EB1C000C for ; Fri, 4 Mar 2022 05:19:26 +0000 (UTC) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=redhat.com; s=mimecast20190719; t=1646371165; h=from:from:reply-to:subject:subject:date:date:message-id:message-id: to:to:cc:cc:mime-version:mime-version:content-type:content-type: content-transfer-encoding:content-transfer-encoding: in-reply-to:in-reply-to:references:references; bh=u8Oi5M9aJF+Nhh3uBfIH6zUjZCwWQp8p9uYRbzwSERc=; b=VJHQ/Qr2p5Pu2BfIbBazsGeXfgYavstxarpm67KIUkpMZ7sdomMXBtPc1VBNJwWRvyk2ID JKfS0B4MCISf0zCsNjnbJhN0iearYv37Vlkkq+dcvCqVZkm0mH6BVxhryhtItrpJBLK71g G1nVvd0i5s3+DP3yHbZLYWFCPB67D00= Received: from mail-pj1-f69.google.com (mail-pj1-f69.google.com [209.85.216.69]) by relay.mimecast.com with ESMTP with STARTTLS (version=TLSv1.2, cipher=TLS_ECDHE_RSA_WITH_AES_256_GCM_SHA384) id us-mta-53-AgjJz61wOCiUPigPer3ymg-1; Fri, 04 Mar 2022 00:19:24 -0500 X-MC-Unique: AgjJz61wOCiUPigPer3ymg-1 Received: by mail-pj1-f69.google.com with SMTP id ev5-20020a17090aeac500b001bc3cb23d4cso4314383pjb.1 for ; Thu, 03 Mar 2022 21:19:24 -0800 (PST) X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20210112; h=x-gm-message-state:from:to:cc:subject:date:message-id:in-reply-to :references:mime-version:content-transfer-encoding; bh=u8Oi5M9aJF+Nhh3uBfIH6zUjZCwWQp8p9uYRbzwSERc=; b=gCxHb8rpbrAOPj/235BBboc3YzDfWgULuclNPbHM3BddgelZ9CKmV75+PfTtnA5Hru WWzc/49hhaApcCiCOo62hmBWow20MQaNrfYa1TNuK/hGlNk4Twzv2NBN5Zr0Up5Z2qEh HxicDQNJn+0mdtGG/hKTtpxVi8tW4e2MycKeHnJfZfYW//54o50nCM30vLDlkISEqkjJ ipLRW5IVP/9RwmRM3Z+177TyChzqapkXCrrxgi77kIFqPm06EhPGtdDlaR0ac7xwh+Ok KOGOxmNAuVRDaBlgzyZ887S362hszJjWYWnyu5u7eNUfvkdWkYTJxHtLbmfz31//BksX dRZQ== X-Gm-Message-State: AOAM533IsaAp2P8Hs687sE3KBO8WCL0wG1gmpBD8VEy9pVUtVPL2MNxt deNQ5r1FXZoNx2vH/B5T/U+fp4ZR3IUsaUMY7ks4a/AISufQ4Shn4xt1B7FJINSYTggxycH6q5K sfN9dkTnfQ8oJ0HUe7VKEwrpN9auZUiL794tg1XtSXZWxlWg5JDkEYfXKcsaS X-Received: by 2002:a17:902:e80f:b0:151:bdd2:cabc with SMTP id u15-20020a170902e80f00b00151bdd2cabcmr970857plg.31.1646371163509; Thu, 03 Mar 2022 21:19:23 -0800 (PST) X-Google-Smtp-Source: ABdhPJx50hZU3JRaU/3TajwefuQOeadN49vMH2q5JxcC4kmufwLK+rjdl0sW8g3unVKCqbQjfQkoew== X-Received: by 2002:a17:902:e80f:b0:151:bdd2:cabc with SMTP id u15-20020a170902e80f00b00151bdd2cabcmr970827plg.31.1646371163102; Thu, 03 Mar 2022 21:19:23 -0800 (PST) Received: from localhost.localdomain ([94.177.118.59]) by smtp.gmail.com with ESMTPSA id p16-20020a056a000b5000b004f669806cd9sm4323865pfo.87.2022.03.03.21.19.15 (version=TLS1_3 cipher=TLS_CHACHA20_POLY1305_SHA256 bits=256/256); Thu, 03 Mar 2022 21:19:22 -0800 (PST) From: Peter Xu To: linux-mm@kvack.org, linux-kernel@vger.kernel.org Cc: peterx@redhat.com, Nadav Amit , Hugh Dickins , David Hildenbrand , Axel Rasmussen , Matthew Wilcox , Alistair Popple , Mike Rapoport , Andrew Morton , Jerome Glisse , Mike Kravetz , "Kirill A . Shutemov" , Andrea Arcangeli Subject: [PATCH v7 15/23] mm/hugetlb: Handle pte markers in page faults Date: Fri, 4 Mar 2022 13:17:00 +0800 Message-Id: <20220304051708.86193-16-peterx@redhat.com> X-Mailer: git-send-email 2.32.0 In-Reply-To: <20220304051708.86193-1-peterx@redhat.com> References: <20220304051708.86193-1-peterx@redhat.com> MIME-Version: 1.0 X-Mimecast-Spam-Score: 0 X-Mimecast-Originator: redhat.com X-Rspam-User: X-Rspamd-Server: rspam12 X-Rspamd-Queue-Id: 493EB1C000C X-Stat-Signature: 6573mace5if75q58ccmrueb8k4tcq5n8 Authentication-Results: imf20.hostedemail.com; dkim=pass header.d=redhat.com header.s=mimecast20190719 header.b="VJHQ/Qr2"; spf=none (imf20.hostedemail.com: domain of peterx@redhat.com has no SPF policy when checking 170.10.129.124) smtp.mailfrom=peterx@redhat.com; dmarc=pass (policy=none) header.from=redhat.com X-HE-Tag: 1646371166-398752 X-Bogosity: Ham, tests=bogofilter, spamicity=0.000000, version=1.2.4 Sender: owner-linux-mm@kvack.org Precedence: bulk X-Loop: owner-majordomo@kvack.org List-ID: Allow hugetlb code to handle pte markers just like none ptes. It's mostly there, we just need to make sure we don't assume hugetlb_no_page() only handles none pte, so when detecting pte change we should use pte_same() rather than pte_none(). We need to pass in the old_pte to do the comparison. Check the original pte to see whether it's a pte marker, if it is, we should recover uffd-wp bit on the new pte to be installed, so that the next write will be trapped by uffd. Signed-off-by: Peter Xu --- mm/hugetlb.c | 18 ++++++++++++++---- 1 file changed, 14 insertions(+), 4 deletions(-) diff --git a/mm/hugetlb.c b/mm/hugetlb.c index 396d5a516d05..afd3d93cfe9a 100644 --- a/mm/hugetlb.c +++ b/mm/hugetlb.c @@ -5383,7 +5383,8 @@ static inline vm_fault_t hugetlb_handle_userfault(struct vm_area_struct *vma, static vm_fault_t hugetlb_no_page(struct mm_struct *mm, struct vm_area_struct *vma, struct address_space *mapping, pgoff_t idx, - unsigned long address, pte_t *ptep, unsigned int flags) + unsigned long address, pte_t *ptep, + pte_t old_pte, unsigned int flags) { struct hstate *h = hstate_vma(vma); vm_fault_t ret = VM_FAULT_SIGBUS; @@ -5509,7 +5510,8 @@ static vm_fault_t hugetlb_no_page(struct mm_struct *mm, ptl = huge_pte_lock(h, mm, ptep); ret = 0; - if (!huge_pte_none(huge_ptep_get(ptep))) + /* If pte changed from under us, retry */ + if (!pte_same(huge_ptep_get(ptep), old_pte)) goto backout; if (anon_rmap) { @@ -5519,6 +5521,12 @@ static vm_fault_t hugetlb_no_page(struct mm_struct *mm, page_dup_rmap(page, true); new_pte = make_huge_pte(vma, page, ((vma->vm_flags & VM_WRITE) && (vma->vm_flags & VM_SHARED))); + /* + * If this pte was previously wr-protected, keep it wr-protected even + * if populated. + */ + if (unlikely(pte_marker_uffd_wp(old_pte))) + new_pte = huge_pte_wrprotect(huge_pte_mkuffd_wp(new_pte)); set_huge_pte_at(mm, haddr, ptep, new_pte); hugetlb_count_add(pages_per_huge_page(h), mm); @@ -5636,8 +5644,10 @@ vm_fault_t hugetlb_fault(struct mm_struct *mm, struct vm_area_struct *vma, mutex_lock(&hugetlb_fault_mutex_table[hash]); entry = huge_ptep_get(ptep); - if (huge_pte_none(entry)) { - ret = hugetlb_no_page(mm, vma, mapping, idx, address, ptep, flags); + /* PTE markers should be handled the same way as none pte */ + if (huge_pte_none_mostly(entry)) { + ret = hugetlb_no_page(mm, vma, mapping, idx, address, ptep, + entry, flags); goto out_mutex; }