From patchwork Wed Jul 14 22:20:52 2021 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Peter Xu X-Patchwork-Id: 12377937 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org X-Spam-Level: X-Spam-Status: No, score=-16.6 required=3.0 tests=BAYES_00,DKIM_INVALID, DKIM_SIGNED,HEADER_FROM_DIFFERENT_DOMAINS,INCLUDES_CR_TRAILER,INCLUDES_PATCH, MAILING_LIST_MULTI,SPF_HELO_NONE,SPF_PASS,USER_AGENT_GIT autolearn=ham autolearn_force=no version=3.4.0 Received: from mail.kernel.org (mail.kernel.org [198.145.29.99]) by smtp.lore.kernel.org (Postfix) with ESMTP id 1B2E9C12002 for ; Wed, 14 Jul 2021 22:21:27 +0000 (UTC) Received: from kanga.kvack.org (kanga.kvack.org [205.233.56.17]) by mail.kernel.org (Postfix) with ESMTP id C0FA6613C9 for ; Wed, 14 Jul 2021 22:21:26 +0000 (UTC) DMARC-Filter: OpenDMARC Filter v1.3.2 mail.kernel.org C0FA6613C9 Authentication-Results: mail.kernel.org; dmarc=fail (p=none dis=none) header.from=redhat.com Authentication-Results: mail.kernel.org; spf=pass smtp.mailfrom=owner-linux-mm@kvack.org Received: by kanga.kvack.org (Postfix) id 0556B6B00A6; Wed, 14 Jul 2021 18:21:27 -0400 (EDT) Received: by kanga.kvack.org (Postfix, from userid 40) id 02C406B00A7; Wed, 14 Jul 2021 18:21:26 -0400 (EDT) X-Delivered-To: int-list-linux-mm@kvack.org Received: by kanga.kvack.org (Postfix, from userid 63042) id DE8756B00A8; Wed, 14 Jul 2021 18:21:26 -0400 (EDT) X-Delivered-To: linux-mm@kvack.org Received: from forelay.hostedemail.com (smtprelay0107.hostedemail.com [216.40.44.107]) by kanga.kvack.org (Postfix) with ESMTP id BCFE76B00A6 for ; Wed, 14 Jul 2021 18:21:26 -0400 (EDT) Received: from smtpin14.hostedemail.com (10.5.19.251.rfc1918.com [10.5.19.251]) by forelay01.hostedemail.com (Postfix) with ESMTP id A36691809FEEC for ; Wed, 14 Jul 2021 22:21:25 +0000 (UTC) X-FDA: 78362615730.14.935D60E Received: from us-smtp-delivery-124.mimecast.com (us-smtp-delivery-124.mimecast.com [216.205.24.124]) by imf29.hostedemail.com (Postfix) with ESMTP id 3A76F900024F for ; Wed, 14 Jul 2021 22:21:25 +0000 (UTC) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=redhat.com; s=mimecast20190719; t=1626301284; h=from:from:reply-to:subject:subject:date:date:message-id:message-id: to:to:cc:cc:mime-version:mime-version:content-type:content-type: content-transfer-encoding:content-transfer-encoding: in-reply-to:in-reply-to:references:references; bh=2wghNqx4JftAFaj2ufqftGGQbCHL8Jx+CkAGBQbGsx4=; b=LSy3Mqf4wQXdQKpxieB+L9ySEPf402M1SzR1hd1k2rhRBRxe7Y55AhIs90rvrcSxGsdiVb +69io/S1mcn1sKs2nniA7Nw8BCrnBdFj7rZPVe5QUwVtEsMW69N+5yKfjQGS4SiV27baeO 2HJ4v5g1/h586F54XKINIYfoqWAu41U= Received: from mail-qt1-f199.google.com (mail-qt1-f199.google.com [209.85.160.199]) (Using TLS) by relay.mimecast.com with ESMTP id us-mta-425-r9w-lJaiN4irvtC87ygNGg-1; Wed, 14 Jul 2021 18:21:23 -0400 X-MC-Unique: r9w-lJaiN4irvtC87ygNGg-1 Received: by mail-qt1-f199.google.com with SMTP id w3-20020ac80ec30000b029024e8c2383c1so2833929qti.5 for ; Wed, 14 Jul 2021 15:21:23 -0700 (PDT) X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20161025; h=x-gm-message-state:from:to:cc:subject:date:message-id:in-reply-to :references:mime-version:content-transfer-encoding; bh=2wghNqx4JftAFaj2ufqftGGQbCHL8Jx+CkAGBQbGsx4=; b=aH2r4ZKiFQ6wUSoigqCTmMtN7kruzPFPz6Uyguty6/vpn5Dw8vbhVSbFhuoqhE4j0Y GN0Q4ammLu+lh1kR54jzC7Sdc2phhXxtU1e0ei/4TNYv5cZ5Tu3OcUdD2uT3uswWxVF8 JVvrEHz8W+sCiyTR3CMHikQnJ9gmA4G7SGPdFQYq5u4VO9GKiaZrGnOUczPZ37UfO7cB l2WbllG3mi9bHGUnTQ18u2EtsYRhmWfQB5gncWPOtgkmUgpQyAmsGJK4mZJLyMTLULlG PaNvMDYM10aYcddrPAZDQc+jORv9KLRJ+TOey407IURbCEdPnU5i4ekhUAcg/QkWpB7s 6EHQ== X-Gm-Message-State: AOAM533DNSFT7aM3OsR8koxGIQgOIOtEcb+VJTO63NPinEIMfFP2UUnz u0zTMaCnsToICQA0DSgkLB5eTED5cYxOfGiBtfqUi4QRNRktksmZLyQ60gVYN+91l0Kl+dJ/Kfh +rV+I+tJCBTQ= X-Received: by 2002:a05:620a:16b7:: with SMTP id s23mr187832qkj.495.1626301282799; Wed, 14 Jul 2021 15:21:22 -0700 (PDT) X-Google-Smtp-Source: ABdhPJyHBJUlax/lXVspNfFXeao9SZKZfEe/LZ0w2Bg7nDn4nRH7cKQmqRx3VxQaS8Lv2fCmBULSGw== X-Received: by 2002:a05:620a:16b7:: with SMTP id s23mr187816qkj.495.1626301282594; Wed, 14 Jul 2021 15:21:22 -0700 (PDT) Received: from localhost.localdomain (bras-base-toroon474qw-grc-65-184-144-111-238.dsl.bell.ca. [184.144.111.238]) by smtp.gmail.com with ESMTPSA id b25sm1625854qka.123.2021.07.14.15.21.20 (version=TLS1_3 cipher=TLS_AES_256_GCM_SHA384 bits=256/256); Wed, 14 Jul 2021 15:21:22 -0700 (PDT) From: Peter Xu To: linux-kernel@vger.kernel.org, linux-mm@kvack.org Cc: Jason Gunthorpe , peterx@redhat.com, Matthew Wilcox , Andrew Morton , Axel Rasmussen , Nadav Amit , Jerome Glisse , Mike Rapoport , Miaohe Lin , Hugh Dickins , Alistair Popple , Andrea Arcangeli , Mike Kravetz , "Kirill A . Shutemov" , David Hildenbrand Subject: [PATCH v4 01/26] mm/shmem: Unconditionally set pte dirty in mfill_atomic_install_pte Date: Wed, 14 Jul 2021 18:20:52 -0400 Message-Id: <20210714222117.47648-2-peterx@redhat.com> X-Mailer: git-send-email 2.31.1 In-Reply-To: <20210714222117.47648-1-peterx@redhat.com> References: <20210714222117.47648-1-peterx@redhat.com> MIME-Version: 1.0 X-Mimecast-Spam-Score: 0 X-Mimecast-Originator: redhat.com Authentication-Results: imf29.hostedemail.com; dkim=pass header.d=redhat.com header.s=mimecast20190719 header.b=LSy3Mqf4; spf=none (imf29.hostedemail.com: domain of peterx@redhat.com has no SPF policy when checking 216.205.24.124) smtp.mailfrom=peterx@redhat.com; dmarc=pass (policy=none) header.from=redhat.com X-Stat-Signature: s77b15gzx76omdwtbe5bdbfbw13sg8cu X-Rspamd-Queue-Id: 3A76F900024F X-Rspamd-Server: rspam01 X-HE-Tag: 1626301285-520925 X-Bogosity: Ham, tests=bogofilter, spamicity=0.000000, version=1.2.4 Sender: owner-linux-mm@kvack.org Precedence: bulk X-Loop: owner-majordomo@kvack.org List-ID: It was conditionally done previously, as there's one shmem special case that we use SetPageDirty() instead. However that's not necessary and it should be easier and cleaner to do it unconditionally in mfill_atomic_install_pte(). The most recent discussion about this is here, where Hugh explained the history of SetPageDirty() and why it's possible that it's not required at all: https://lore.kernel.org/lkml/alpine.LSU.2.11.2104121657050.1097@eggly.anvils/ Currently mfill_atomic_install_pte() has three callers: 1. shmem_mfill_atomic_pte 2. mcopy_atomic_pte 3. mcontinue_atomic_pte After the change: case (1) should have its SetPageDirty replaced by the dirty bit on pte (so we unify them together, finally), case (2) should have no functional change at all as it has page_in_cache==false, case (3) may add a dirty bit to the pte. However since case (3) is UFFDIO_CONTINUE for shmem, it's merely 100% sure the page is dirty after all, so should not make a real difference either. This should make it much easier to follow on which case will set dirty for uffd, as we'll simply set it all now for all uffd related ioctls. Meanwhile, no special handling of SetPageDirty() if there's no need. Cc: Hugh Dickins Cc: Axel Rasmussen Cc: Andrea Arcangeli Signed-off-by: Peter Xu --- mm/shmem.c | 1 - mm/userfaultfd.c | 3 +-- 2 files changed, 1 insertion(+), 3 deletions(-) diff --git a/mm/shmem.c b/mm/shmem.c index 70d9ce294bb4..dc9f95b5fb34 100644 --- a/mm/shmem.c +++ b/mm/shmem.c @@ -2449,7 +2449,6 @@ int shmem_mfill_atomic_pte(struct mm_struct *dst_mm, shmem_recalc_inode(inode); spin_unlock_irq(&info->lock); - SetPageDirty(page); unlock_page(page); return 0; out_delete_from_cache: diff --git a/mm/userfaultfd.c b/mm/userfaultfd.c index 0e2132834bc7..b30a3724c701 100644 --- a/mm/userfaultfd.c +++ b/mm/userfaultfd.c @@ -69,10 +69,9 @@ int mfill_atomic_install_pte(struct mm_struct *dst_mm, pmd_t *dst_pmd, pgoff_t offset, max_off; _dst_pte = mk_pte(page, dst_vma->vm_page_prot); + _dst_pte = pte_mkdirty(_dst_pte); if (page_in_cache && !vm_shared) writable = false; - if (writable || !page_in_cache) - _dst_pte = pte_mkdirty(_dst_pte); if (writable) { if (wp_copy) _dst_pte = pte_mkuffd_wp(_dst_pte); From patchwork Wed Jul 14 22:20:53 2021 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Peter Xu X-Patchwork-Id: 12377939 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org X-Spam-Level: X-Spam-Status: No, score=-16.6 required=3.0 tests=BAYES_00,DKIM_INVALID, DKIM_SIGNED,HEADER_FROM_DIFFERENT_DOMAINS,INCLUDES_CR_TRAILER,INCLUDES_PATCH, MAILING_LIST_MULTI,SPF_HELO_NONE,SPF_PASS,USER_AGENT_GIT autolearn=ham autolearn_force=no version=3.4.0 Received: from mail.kernel.org (mail.kernel.org [198.145.29.99]) by smtp.lore.kernel.org (Postfix) with ESMTP id A8242C47E4B for ; Wed, 14 Jul 2021 22:21:28 +0000 (UTC) Received: from kanga.kvack.org (kanga.kvack.org [205.233.56.17]) by mail.kernel.org (Postfix) with ESMTP id 55BEA613D0 for ; Wed, 14 Jul 2021 22:21:28 +0000 (UTC) DMARC-Filter: OpenDMARC Filter v1.3.2 mail.kernel.org 55BEA613D0 Authentication-Results: mail.kernel.org; dmarc=fail (p=none dis=none) header.from=redhat.com Authentication-Results: mail.kernel.org; spf=pass smtp.mailfrom=owner-linux-mm@kvack.org Received: by kanga.kvack.org (Postfix) id A67D46B00A7; Wed, 14 Jul 2021 18:21:28 -0400 (EDT) Received: by kanga.kvack.org (Postfix, from userid 40) id A3E896B00A8; Wed, 14 Jul 2021 18:21:28 -0400 (EDT) X-Delivered-To: int-list-linux-mm@kvack.org Received: by kanga.kvack.org (Postfix, from userid 63042) id 8433B6B00A9; Wed, 14 Jul 2021 18:21:28 -0400 (EDT) X-Delivered-To: linux-mm@kvack.org Received: from forelay.hostedemail.com (smtprelay0062.hostedemail.com [216.40.44.62]) by kanga.kvack.org (Postfix) with ESMTP id 5F21F6B00A7 for ; Wed, 14 Jul 2021 18:21:28 -0400 (EDT) Received: from smtpin29.hostedemail.com (10.5.19.251.rfc1918.com [10.5.19.251]) by forelay01.hostedemail.com (Postfix) with ESMTP id 4E0BD1809FEEC for ; Wed, 14 Jul 2021 22:21:27 +0000 (UTC) X-FDA: 78362615814.29.44DB8E2 Received: from us-smtp-delivery-124.mimecast.com (us-smtp-delivery-124.mimecast.com [216.205.24.124]) by imf11.hostedemail.com (Postfix) with ESMTP id B94E0F0000B1 for ; Wed, 14 Jul 2021 22:21:26 +0000 (UTC) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=redhat.com; s=mimecast20190719; t=1626301286; h=from:from:reply-to:subject:subject:date:date:message-id:message-id: to:to:cc:cc:mime-version:mime-version:content-type:content-type: content-transfer-encoding:content-transfer-encoding: in-reply-to:in-reply-to:references:references; bh=OlGOfsM9rm2m+cZAonb5FrW/IEnwBmumPPW7EcnV+EI=; b=RXLcnaedMpfKP9AIS5Kr1UCoMHnP/ZtRasyYwkX/vKgTIo58NpvUm41V5Z/i0BKTVvvaNb uboTHtq5qX0rOxRi0sYwnrsl1iuvl57XXZBdDkBXMspNRUaIKI6dSJrVg1wDcdrMzHqNsa K4meXFTHYzOr3qyjQxT9/CNrAPX4kfc= Received: from mail-qk1-f200.google.com (mail-qk1-f200.google.com [209.85.222.200]) (Using TLS) by relay.mimecast.com with ESMTP id us-mta-37-OzrLYlEfPIagzjdARLwKYg-1; Wed, 14 Jul 2021 18:21:25 -0400 X-MC-Unique: OzrLYlEfPIagzjdARLwKYg-1 Received: by mail-qk1-f200.google.com with SMTP id u14-20020a05620a120eb02903b8bd5c7d95so1744024qkj.12 for ; Wed, 14 Jul 2021 15:21:25 -0700 (PDT) X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20161025; h=x-gm-message-state:from:to:cc:subject:date:message-id:in-reply-to :references:mime-version:content-transfer-encoding; bh=OlGOfsM9rm2m+cZAonb5FrW/IEnwBmumPPW7EcnV+EI=; b=laFRHqMdZ8yO/y+ClquTFdX05eiu43WR1MhyyjkMj7mGc9PdO+tSbknz5Cj1m8Wm/R MXuLDtQ250jJI+d01TQRsaQGaNH7ohiKaXpBL8smHToUiLeuUZO7HIkoWA0mGOfWk6Jm JDHeNoOZaSZyZy7ZHpdeCmi08R7YPww5cY5BClaZGsruDquQxLjBN//AtgoNfAu5MW5F rLkFMeSptqT7oLzANCLWUSjdbhWC4jHn4jhCliBArLDpKn58RTJ+AT0jQwDHp0tFjuYF M7qZsq9jFwzlaQBZKrS6Ie1M2z2oG0qVnGhnYz5XbC8iGVViKnrsad12T5c2Ez/V2QUp aTzg== X-Gm-Message-State: AOAM532XN0kb2iAdY7rXCYqEPv2GfFfUFwnhyL2njMg2t+QHg6vsg3+A hJwJuR46VGkXu/stQDjwgjJiXuFYTE2cPAIOhtSBv7Gtrb8x/APswR9ZIKki42WsJF4tK/n7Phf fejrinjR0Vng= X-Received: by 2002:ae9:e644:: with SMTP id x4mr208187qkl.105.1626301284724; Wed, 14 Jul 2021 15:21:24 -0700 (PDT) X-Google-Smtp-Source: ABdhPJxZMIEbENGHhL6vjrq7lZny3uTxO1l6RhCF7LFn65Ziwb4pSrVyPurhPBGMM/glELOuwBDj8w== X-Received: by 2002:ae9:e644:: with SMTP id x4mr208178qkl.105.1626301284518; Wed, 14 Jul 2021 15:21:24 -0700 (PDT) Received: from localhost.localdomain (bras-base-toroon474qw-grc-65-184-144-111-238.dsl.bell.ca. [184.144.111.238]) by smtp.gmail.com with ESMTPSA id b25sm1625854qka.123.2021.07.14.15.21.22 (version=TLS1_3 cipher=TLS_AES_256_GCM_SHA384 bits=256/256); Wed, 14 Jul 2021 15:21:23 -0700 (PDT) From: Peter Xu To: linux-kernel@vger.kernel.org, linux-mm@kvack.org Cc: Jason Gunthorpe , peterx@redhat.com, Matthew Wilcox , Andrew Morton , Axel Rasmussen , Nadav Amit , Jerome Glisse , Mike Rapoport , Miaohe Lin , Hugh Dickins , Alistair Popple , Andrea Arcangeli , Mike Kravetz , "Kirill A . Shutemov" , David Hildenbrand Subject: [PATCH v4 02/26] shmem/userfaultfd: Take care of UFFDIO_COPY_MODE_WP Date: Wed, 14 Jul 2021 18:20:53 -0400 Message-Id: <20210714222117.47648-3-peterx@redhat.com> X-Mailer: git-send-email 2.31.1 In-Reply-To: <20210714222117.47648-1-peterx@redhat.com> References: <20210714222117.47648-1-peterx@redhat.com> MIME-Version: 1.0 X-Mimecast-Spam-Score: 0 X-Mimecast-Originator: redhat.com X-Rspamd-Server: rspam06 X-Rspamd-Queue-Id: B94E0F0000B1 X-Stat-Signature: tjgfjesm79fbmeh8ymsx7kcrd49g3fb1 Authentication-Results: imf11.hostedemail.com; dkim=pass header.d=redhat.com header.s=mimecast20190719 header.b=RXLcnaed; spf=none (imf11.hostedemail.com: domain of peterx@redhat.com has no SPF policy when checking 216.205.24.124) smtp.mailfrom=peterx@redhat.com; dmarc=pass (policy=none) header.from=redhat.com X-HE-Tag: 1626301286-19992 X-Bogosity: Ham, tests=bogofilter, spamicity=0.000000, version=1.2.4 Sender: owner-linux-mm@kvack.org Precedence: bulk X-Loop: owner-majordomo@kvack.org List-ID: Firstly, pass wp_copy into shmem_mfill_atomic_pte() through the stack. Then apply the UFFD_WP bit properly when the UFFDIO_COPY on shmem is with UFFDIO_COPY_MODE_WP, then wp_copy lands mfill_atomic_install_pte() which is newly introduced very recently. We need to make sure shmem_mfill_atomic_pte() will always set the dirty bit in pte even if UFFDIO_COPY_MODE_WP is set. After the rework of minor fault series on shmem we need to slightly touch up the logic there, since uffd-wp needs to be applied even if writable==false previously (e.g., for shmem private mapping). Note: we must do pte_wrprotect() if !writable in mfill_atomic_install_pte(), as mk_pte() could return a writable pte (e.g., when VM_SHARED on a shmem file). Signed-off-by: Peter Xu --- include/linux/shmem_fs.h | 4 ++-- mm/shmem.c | 4 ++-- mm/userfaultfd.c | 23 +++++++++++++++-------- 3 files changed, 19 insertions(+), 12 deletions(-) diff --git a/include/linux/shmem_fs.h b/include/linux/shmem_fs.h index 8e775ce517bb..ba6ea8c4d534 100644 --- a/include/linux/shmem_fs.h +++ b/include/linux/shmem_fs.h @@ -128,11 +128,11 @@ extern int shmem_mfill_atomic_pte(struct mm_struct *dst_mm, pmd_t *dst_pmd, struct vm_area_struct *dst_vma, unsigned long dst_addr, unsigned long src_addr, - bool zeropage, + bool zeropage, bool wp_copy, struct page **pagep); #else /* !CONFIG_SHMEM */ #define shmem_mfill_atomic_pte(dst_mm, dst_pmd, dst_vma, dst_addr, \ - src_addr, zeropage, pagep) ({ BUG(); 0; }) + src_addr, zeropage, wp_copy, pagep) ({ BUG(); 0; }) #endif /* CONFIG_SHMEM */ #endif /* CONFIG_USERFAULTFD */ diff --git a/mm/shmem.c b/mm/shmem.c index dc9f95b5fb34..9608200f8a64 100644 --- a/mm/shmem.c +++ b/mm/shmem.c @@ -2368,7 +2368,7 @@ int shmem_mfill_atomic_pte(struct mm_struct *dst_mm, struct vm_area_struct *dst_vma, unsigned long dst_addr, unsigned long src_addr, - bool zeropage, + bool zeropage, bool wp_copy, struct page **pagep) { struct inode *inode = file_inode(dst_vma->vm_file); @@ -2439,7 +2439,7 @@ int shmem_mfill_atomic_pte(struct mm_struct *dst_mm, goto out_release; ret = mfill_atomic_install_pte(dst_mm, dst_pmd, dst_vma, dst_addr, - page, true, false); + page, true, wp_copy); if (ret) goto out_delete_from_cache; diff --git a/mm/userfaultfd.c b/mm/userfaultfd.c index b30a3724c701..2a9c9e6eb876 100644 --- a/mm/userfaultfd.c +++ b/mm/userfaultfd.c @@ -70,14 +70,22 @@ int mfill_atomic_install_pte(struct mm_struct *dst_mm, pmd_t *dst_pmd, _dst_pte = mk_pte(page, dst_vma->vm_page_prot); _dst_pte = pte_mkdirty(_dst_pte); - if (page_in_cache && !vm_shared) + /* Don't write if uffd-wp wr-protected */ + if (wp_copy) { + _dst_pte = pte_mkuffd_wp(_dst_pte); writable = false; - if (writable) { - if (wp_copy) - _dst_pte = pte_mkuffd_wp(_dst_pte); - else - _dst_pte = pte_mkwrite(_dst_pte); } + /* Don't write if page cache privately mapped */ + if (page_in_cache && !vm_shared) + writable = false; + if (writable) + _dst_pte = pte_mkwrite(_dst_pte); + else + /* + * We need this to make sure write bit removed; as mk_pte() + * could return a pte with write bit set. + */ + _dst_pte = pte_wrprotect(_dst_pte); dst_pte = pte_offset_map_lock(dst_mm, dst_pmd, dst_addr, &ptl); @@ -467,11 +475,10 @@ static __always_inline ssize_t mfill_atomic_pte(struct mm_struct *dst_mm, err = mfill_zeropage_pte(dst_mm, dst_pmd, dst_vma, dst_addr); } else { - VM_WARN_ON_ONCE(wp_copy); err = shmem_mfill_atomic_pte(dst_mm, dst_pmd, dst_vma, dst_addr, src_addr, mode != MCOPY_ATOMIC_NORMAL, - page); + wp_copy, page); } return err; From patchwork Wed Jul 14 22:20:54 2021 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Peter Xu X-Patchwork-Id: 12377941 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org X-Spam-Level: X-Spam-Status: No, score=-16.6 required=3.0 tests=BAYES_00,DKIM_INVALID, DKIM_SIGNED,HEADER_FROM_DIFFERENT_DOMAINS,INCLUDES_CR_TRAILER,INCLUDES_PATCH, MAILING_LIST_MULTI,SPF_HELO_NONE,SPF_PASS,USER_AGENT_GIT autolearn=ham autolearn_force=no version=3.4.0 Received: from mail.kernel.org (mail.kernel.org [198.145.29.99]) by smtp.lore.kernel.org (Postfix) with ESMTP id 3FEF8C12002 for ; Wed, 14 Jul 2021 22:21:30 +0000 (UTC) Received: from kanga.kvack.org (kanga.kvack.org [205.233.56.17]) by mail.kernel.org (Postfix) with ESMTP id E3BB9613CA for ; Wed, 14 Jul 2021 22:21:29 +0000 (UTC) DMARC-Filter: OpenDMARC Filter v1.3.2 mail.kernel.org E3BB9613CA Authentication-Results: mail.kernel.org; dmarc=fail (p=none dis=none) header.from=redhat.com Authentication-Results: mail.kernel.org; spf=pass smtp.mailfrom=owner-linux-mm@kvack.org Received: by kanga.kvack.org (Postfix) id 3FF896B00A9; Wed, 14 Jul 2021 18:21:30 -0400 (EDT) Received: by kanga.kvack.org (Postfix, from userid 40) id 33CE96B00AB; Wed, 14 Jul 2021 18:21:30 -0400 (EDT) X-Delivered-To: int-list-linux-mm@kvack.org Received: by kanga.kvack.org (Postfix, from userid 63042) id 0C93D6B00AC; Wed, 14 Jul 2021 18:21:30 -0400 (EDT) X-Delivered-To: linux-mm@kvack.org Received: from forelay.hostedemail.com (smtprelay0091.hostedemail.com [216.40.44.91]) by kanga.kvack.org (Postfix) with ESMTP id D75FE6B00A9 for ; Wed, 14 Jul 2021 18:21:29 -0400 (EDT) Received: from smtpin31.hostedemail.com (10.5.19.251.rfc1918.com [10.5.19.251]) by forelay02.hostedemail.com (Postfix) with ESMTP id CF34D160AF for ; Wed, 14 Jul 2021 22:21:28 +0000 (UTC) X-FDA: 78362615856.31.8471523 Received: from us-smtp-delivery-124.mimecast.com (us-smtp-delivery-124.mimecast.com [216.205.24.124]) by imf16.hostedemail.com (Postfix) with ESMTP id 70B9FF00009D for ; Wed, 14 Jul 2021 22:21:28 +0000 (UTC) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=redhat.com; s=mimecast20190719; t=1626301288; h=from:from:reply-to:subject:subject:date:date:message-id:message-id: to:to:cc:cc:mime-version:mime-version:content-type:content-type: content-transfer-encoding:content-transfer-encoding: in-reply-to:in-reply-to:references:references; bh=aMR1rwc2VptPVcJogLZEh7U1JPTpYleS0uDSBMQzgKI=; b=fdYbPUin0TQJTOiyUFqdMvaTxohc8NEf6kidv5SA1AALsr5WHgScj7Nom45zCwJhKmoHBt vVDiGHDuJ1v6pFZj3tK4/at6s/5KjbejbMCXbVOF1BaESjtfc99dnU+f8FoMF9I7FRoSwj dcBICwGDARhPwY9dNsXGZ41BJS9n318= Received: from mail-qt1-f200.google.com (mail-qt1-f200.google.com [209.85.160.200]) (Using TLS) by relay.mimecast.com with ESMTP id us-mta-377-EPFsQOBdPEKmzg_K25gYNA-1; Wed, 14 Jul 2021 18:21:26 -0400 X-MC-Unique: EPFsQOBdPEKmzg_K25gYNA-1 Received: by mail-qt1-f200.google.com with SMTP id w3-20020ac80ec30000b029024e8c2383c1so2834013qti.5 for ; Wed, 14 Jul 2021 15:21:26 -0700 (PDT) X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20161025; h=x-gm-message-state:from:to:cc:subject:date:message-id:in-reply-to :references:mime-version:content-transfer-encoding; bh=aMR1rwc2VptPVcJogLZEh7U1JPTpYleS0uDSBMQzgKI=; b=ERSaF5jOpj8bbu5OGunrstnc4Gke4SZZgd3Lssywx9LopeDNMf5laobF2lHhzWFgVT phk/YJnydrIoTwhthH/DjJXzZS2Vv8+I5nQDssjPvSVamrafgxDpt6cWuD5kSNC+e9RM Rw1+4jdcBZHTUVVDqc1YJk/6kIIfm2sLntOzoheWnqTZo4SSn1a1vIJK915vkW02Z0Be Kt41QO07HlaoHpzmvtZELEGTL9QhNr3L3ipv6k9sYFVHgGrQETf8js0D27FuR/uQZLXj RqizD4KLpcfN9pQ+CnAiGO5026Xdn2DA04W0i82HP3NN7WN3QOzKF6dx9DH/rFpuxuux RYqA== X-Gm-Message-State: AOAM531KhCb//z6pj7Rgju7WGBfHW99sqX9+aQJihTEZ1M+C8hP4IyGM 5sgRUHhPxWuQLMMYtWkpPdqZu96/k6U3f2yUW0e/1Wps/Iz47AIwLCxkyNU548oZ42aQAz579gZ q4bwKdYqV+RA= X-Received: by 2002:a05:620a:684:: with SMTP id f4mr223902qkh.258.1626301286424; Wed, 14 Jul 2021 15:21:26 -0700 (PDT) X-Google-Smtp-Source: ABdhPJymnAJAG1s3IIuL8U1KCptLk3RkFUZv2CP2yXs2+mvIhz26/ER+UYtoKJw1cDncE9s3s+WTuw== X-Received: by 2002:a05:620a:684:: with SMTP id f4mr223873qkh.258.1626301286213; Wed, 14 Jul 2021 15:21:26 -0700 (PDT) Received: from localhost.localdomain (bras-base-toroon474qw-grc-65-184-144-111-238.dsl.bell.ca. [184.144.111.238]) by smtp.gmail.com with ESMTPSA id b25sm1625854qka.123.2021.07.14.15.21.24 (version=TLS1_3 cipher=TLS_AES_256_GCM_SHA384 bits=256/256); Wed, 14 Jul 2021 15:21:25 -0700 (PDT) From: Peter Xu To: linux-kernel@vger.kernel.org, linux-mm@kvack.org Cc: Jason Gunthorpe , peterx@redhat.com, Matthew Wilcox , Andrew Morton , Axel Rasmussen , Nadav Amit , Jerome Glisse , Mike Rapoport , Miaohe Lin , Hugh Dickins , Alistair Popple , Andrea Arcangeli , Mike Kravetz , "Kirill A . Shutemov" , David Hildenbrand Subject: [PATCH v4 03/26] mm: Clear vmf->pte after pte_unmap_same() returns Date: Wed, 14 Jul 2021 18:20:54 -0400 Message-Id: <20210714222117.47648-4-peterx@redhat.com> X-Mailer: git-send-email 2.31.1 In-Reply-To: <20210714222117.47648-1-peterx@redhat.com> References: <20210714222117.47648-1-peterx@redhat.com> MIME-Version: 1.0 X-Mimecast-Spam-Score: 0 X-Mimecast-Originator: redhat.com Authentication-Results: imf16.hostedemail.com; dkim=pass header.d=redhat.com header.s=mimecast20190719 header.b=fdYbPUin; spf=none (imf16.hostedemail.com: domain of peterx@redhat.com has no SPF policy when checking 216.205.24.124) smtp.mailfrom=peterx@redhat.com; dmarc=pass (policy=none) header.from=redhat.com X-Stat-Signature: p3y861werssu4nhqkg9y8nr94qomf3jk X-Rspamd-Queue-Id: 70B9FF00009D X-Rspamd-Server: rspam01 X-HE-Tag: 1626301288-987286 X-Bogosity: Ham, tests=bogofilter, spamicity=0.000000, version=1.2.4 Sender: owner-linux-mm@kvack.org Precedence: bulk X-Loop: owner-majordomo@kvack.org List-ID: pte_unmap_same() will always unmap the pte pointer. After the unmap, vmf->pte will not be valid any more. We should clear it. It was safe only because no one is accessing vmf->pte after pte_unmap_same() returns, since the only caller of pte_unmap_same() (so far) is do_swap_page(), where vmf->pte will in most cases be overwritten very soon. pte_unmap_same() will be used in other places in follow up patches, so that vmf->pte will not always be re-written. This patch enables us to call functions like finish_fault() because that'll conditionally unmap the pte by checking vmf->pte first. Or, alloc_set_pte() will make sure to allocate a new pte even after calling pte_unmap_same(). Since we'll need to modify vmf->pte, directly pass in vmf into pte_unmap_same() and then we can also avoid the long parameter list. Reviewed-by: Miaohe Lin Signed-off-by: Peter Xu --- mm/memory.c | 13 +++++++------ 1 file changed, 7 insertions(+), 6 deletions(-) diff --git a/mm/memory.c b/mm/memory.c index 747a01d495f2..0e0de08a2cd5 100644 --- a/mm/memory.c +++ b/mm/memory.c @@ -2724,19 +2724,20 @@ EXPORT_SYMBOL_GPL(apply_to_existing_page_range); * proceeding (but do_wp_page is only called after already making such a check; * and do_anonymous_page can safely check later on). */ -static inline int pte_unmap_same(struct mm_struct *mm, pmd_t *pmd, - pte_t *page_table, pte_t orig_pte) +static inline int pte_unmap_same(struct vm_fault *vmf) { int same = 1; #if defined(CONFIG_SMP) || defined(CONFIG_PREEMPTION) if (sizeof(pte_t) > sizeof(unsigned long)) { - spinlock_t *ptl = pte_lockptr(mm, pmd); + spinlock_t *ptl = pte_lockptr(vmf->vma->vm_mm, vmf->pmd); spin_lock(ptl); - same = pte_same(*page_table, orig_pte); + same = pte_same(*vmf->pte, vmf->orig_pte); spin_unlock(ptl); } #endif - pte_unmap(page_table); + pte_unmap(vmf->pte); + /* After unmap of pte, the pointer is invalid now - clear it. */ + vmf->pte = NULL; return same; } @@ -3487,7 +3488,7 @@ vm_fault_t do_swap_page(struct vm_fault *vmf) vm_fault_t ret = 0; void *shadow = NULL; - if (!pte_unmap_same(vma->vm_mm, vmf->pmd, vmf->pte, vmf->orig_pte)) + if (!pte_unmap_same(vmf)) goto out; entry = pte_to_swp_entry(vmf->orig_pte); From patchwork Wed Jul 14 22:20:55 2021 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Peter Xu X-Patchwork-Id: 12377945 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org X-Spam-Level: X-Spam-Status: No, score=-16.6 required=3.0 tests=BAYES_00,DKIM_INVALID, DKIM_SIGNED,HEADER_FROM_DIFFERENT_DOMAINS,INCLUDES_CR_TRAILER,INCLUDES_PATCH, MAILING_LIST_MULTI,SPF_HELO_NONE,SPF_PASS,USER_AGENT_GIT autolearn=ham autolearn_force=no version=3.4.0 Received: from mail.kernel.org (mail.kernel.org [198.145.29.99]) by smtp.lore.kernel.org (Postfix) with ESMTP id 195B1C1B08C for ; Wed, 14 Jul 2021 22:21:34 +0000 (UTC) Received: from kanga.kvack.org (kanga.kvack.org [205.233.56.17]) by mail.kernel.org (Postfix) with ESMTP id BF5A1613C5 for ; Wed, 14 Jul 2021 22:21:33 +0000 (UTC) DMARC-Filter: OpenDMARC Filter v1.3.2 mail.kernel.org BF5A1613C5 Authentication-Results: mail.kernel.org; dmarc=fail (p=none dis=none) header.from=redhat.com Authentication-Results: mail.kernel.org; spf=pass smtp.mailfrom=owner-linux-mm@kvack.org Received: by kanga.kvack.org (Postfix) id D26F36B00AD; Wed, 14 Jul 2021 18:21:31 -0400 (EDT) Received: by kanga.kvack.org (Postfix, from userid 40) id C8A2A6B00AE; Wed, 14 Jul 2021 18:21:31 -0400 (EDT) X-Delivered-To: int-list-linux-mm@kvack.org Received: by kanga.kvack.org (Postfix, from userid 63042) id B2A076B00AF; Wed, 14 Jul 2021 18:21:31 -0400 (EDT) X-Delivered-To: linux-mm@kvack.org Received: from forelay.hostedemail.com (smtprelay0106.hostedemail.com [216.40.44.106]) by kanga.kvack.org (Postfix) with ESMTP id 8BE986B00AD for ; Wed, 14 Jul 2021 18:21:31 -0400 (EDT) Received: from smtpin26.hostedemail.com (10.5.19.251.rfc1918.com [10.5.19.251]) by forelay01.hostedemail.com (Postfix) with ESMTP id 81CF418464D1D for ; Wed, 14 Jul 2021 22:21:30 +0000 (UTC) X-FDA: 78362615940.26.0AED6A4 Received: from us-smtp-delivery-124.mimecast.com (us-smtp-delivery-124.mimecast.com [216.205.24.124]) by imf20.hostedemail.com (Postfix) with ESMTP id 295DAD0000B8 for ; Wed, 14 Jul 2021 22:21:30 +0000 (UTC) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=redhat.com; s=mimecast20190719; t=1626301289; h=from:from:reply-to:subject:subject:date:date:message-id:message-id: to:to:cc:cc:mime-version:mime-version:content-type:content-type: content-transfer-encoding:content-transfer-encoding: in-reply-to:in-reply-to:references:references; bh=Y0JmFuDDelZXYWv/E+BZRvjlS5kkw3E6CXCgFUA2NEY=; b=aZwSWU6GsWpXyx6SJM21OBJD10SvZ77P6LKhm+/2HKI07q69Nrw+bsnnDGSaGhzq9f8u7K dTWFUl0dtXo+03hG9iXdptpj/gfOqg4PvLcwsoMCCp2KY0/ePuoLbwC59A3X3TKkkiGA/B HuOGapPXiqB9tKTS4IGAbudXCqs5x8E= Received: from mail-qt1-f198.google.com (mail-qt1-f198.google.com [209.85.160.198]) (Using TLS) by relay.mimecast.com with ESMTP id us-mta-447-uC6jtfvxN8-ahIg_6LHCWg-1; Wed, 14 Jul 2021 18:21:28 -0400 X-MC-Unique: uC6jtfvxN8-ahIg_6LHCWg-1 Received: by mail-qt1-f198.google.com with SMTP id b6-20020a05622a0206b029025eb726dd9bso385176qtx.8 for ; Wed, 14 Jul 2021 15:21:28 -0700 (PDT) X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20161025; h=x-gm-message-state:from:to:cc:subject:date:message-id:in-reply-to :references:mime-version:content-transfer-encoding; bh=Y0JmFuDDelZXYWv/E+BZRvjlS5kkw3E6CXCgFUA2NEY=; b=ehVBA9o75nviUvvUP96yv9BnIw7zDD4n3QYHC40Yid99fkDFZQIPNmjeEPBDNLj7JT S/AmsgDGjQaHtlfPA5z2/Ds4LEaW+39kkuloqYd0JTZ4ZBa6rCzTQh3rVjNxqoR3BFdM 6U2LxCySMDOF6TwtUlLfQstakdVvh/CGtivEgisXFeQbVGwv9mTA3IOvZU8k0r1z41AR 2dFbtMwvvS6f03FSt4wNLI3OhE8hDqGLjQFwZziOtcvNECmYq+dmtrSI6MlKgXps+hxR HeANOMbQPgCbUIkEJP5tZ/UiT1v6cEY94Oenidf9Q+sUgVMYfRfu6GqXMrAnMtZaSDPH NwuQ== X-Gm-Message-State: AOAM530Akmmjf512GtzbLv6S/Fap3PQFzWqzvxArhE/KAfHa/FS/Gxup 8rjFhlezvvGSuN581l8IwjWaMhpnG35QPQCpRlYvyxREypmyKX9cPGq45OLxqUUivf8DRFDL/uc Zw6jw6NkaBrM= X-Received: by 2002:a05:622a:1987:: with SMTP id u7mr308284qtc.45.1626301287970; Wed, 14 Jul 2021 15:21:27 -0700 (PDT) X-Google-Smtp-Source: ABdhPJw8a60fJ/Iga9Wm6qTFCnF80A2iIi32agCJqRunS6d3XLWes8fPNIWRySZtcDQS5RPr3NCouw== X-Received: by 2002:a05:622a:1987:: with SMTP id u7mr308258qtc.45.1626301287734; Wed, 14 Jul 2021 15:21:27 -0700 (PDT) Received: from localhost.localdomain (bras-base-toroon474qw-grc-65-184-144-111-238.dsl.bell.ca. [184.144.111.238]) by smtp.gmail.com with ESMTPSA id b25sm1625854qka.123.2021.07.14.15.21.26 (version=TLS1_3 cipher=TLS_AES_256_GCM_SHA384 bits=256/256); Wed, 14 Jul 2021 15:21:27 -0700 (PDT) From: Peter Xu To: linux-kernel@vger.kernel.org, linux-mm@kvack.org Cc: Jason Gunthorpe , peterx@redhat.com, Matthew Wilcox , Andrew Morton , Axel Rasmussen , Nadav Amit , Jerome Glisse , Mike Rapoport , Miaohe Lin , Hugh Dickins , Alistair Popple , Andrea Arcangeli , Mike Kravetz , "Kirill A . Shutemov" , David Hildenbrand Subject: [PATCH v4 04/26] mm/userfaultfd: Introduce special pte for unmapped file-backed mem Date: Wed, 14 Jul 2021 18:20:55 -0400 Message-Id: <20210714222117.47648-5-peterx@redhat.com> X-Mailer: git-send-email 2.31.1 In-Reply-To: <20210714222117.47648-1-peterx@redhat.com> References: <20210714222117.47648-1-peterx@redhat.com> MIME-Version: 1.0 X-Mimecast-Spam-Score: 0 X-Mimecast-Originator: redhat.com Authentication-Results: imf20.hostedemail.com; dkim=pass header.d=redhat.com header.s=mimecast20190719 header.b=aZwSWU6G; spf=none (imf20.hostedemail.com: domain of peterx@redhat.com has no SPF policy when checking 216.205.24.124) smtp.mailfrom=peterx@redhat.com; dmarc=pass (policy=none) header.from=redhat.com X-Rspamd-Server: rspam05 X-Stat-Signature: mwyk7docpomg96348emstatujam5qeph X-Rspamd-Queue-Id: 295DAD0000B8 X-HE-Tag: 1626301290-502903 X-Bogosity: Ham, tests=bogofilter, spamicity=0.000000, version=1.2.4 Sender: owner-linux-mm@kvack.org Precedence: bulk X-Loop: owner-majordomo@kvack.org List-ID: This patch introduces a very special swap-like pte for file-backed memories. Currently it's only defined for x86_64 only, but as long as any arch that can properly define the UFFD_WP_SWP_PTE_SPECIAL value as requested, it should conceptually work too. We will use this special pte to arm the ptes that got either unmapped or swapped out for a file-backed region that was previously wr-protected. This special pte could trigger a page fault just like swap entries, and as long as the page fault will satisfy pte_none()==false && pte_present()==false. Then we can revive the special pte into a normal pte backed by the page cache. This idea is greatly inspired by Hugh and Andrea in the discussion, which is referenced in the links below. The other idea (from Hugh) is that we use swp_type==1 and swp_offset=0 as the special pte. The current solution (as pointed out by Andrea) is slightly preferred in that we don't even need swp_entry_t knowledge at all in trapping these accesses. Meanwhile, we also reuse _PAGE_SWP_UFFD_WP from the anonymous swp entries. This patch only introduces the special pte and its operators. It's not yet applied to have any functional difference. Link: https://lore.kernel.org/lkml/20201126222359.8120-1-peterx@redhat.com/ Link: https://lore.kernel.org/lkml/20201130230603.46187-1-peterx@redhat.com/ Suggested-by: Andrea Arcangeli Suggested-by: Hugh Dickins Signed-off-by: Peter Xu --- arch/x86/include/asm/pgtable.h | 28 ++++++++++++++++++++++++++++ include/asm-generic/pgtable_uffd.h | 3 +++ include/linux/userfaultfd_k.h | 25 +++++++++++++++++++++++++ 3 files changed, 56 insertions(+) diff --git a/arch/x86/include/asm/pgtable.h b/arch/x86/include/asm/pgtable.h index 448cd01eb3ec..71b1e73d5b26 100644 --- a/arch/x86/include/asm/pgtable.h +++ b/arch/x86/include/asm/pgtable.h @@ -1300,6 +1300,34 @@ static inline pmd_t pmd_swp_clear_soft_dirty(pmd_t pmd) #endif #ifdef CONFIG_HAVE_ARCH_USERFAULTFD_WP + +/* + * This is a very special swap-like pte that marks this pte as "wr-protected" + * by userfaultfd-wp. It should only exist for file-backed memory where the + * page (previously got wr-protected) has been unmapped or swapped out. + * + * For anonymous memories, the userfaultfd-wp _PAGE_SWP_UFFD_WP bit is kept + * along with a real swp entry instead. + * + * Let's make some rules for this special pte: + * + * (1) pte_none()==false, so that it'll not trigger a missing page fault. + * + * (2) pte_present()==false, so that it's recognized as swap (is_swap_pte). + * + * (3) pte_swp_uffd_wp()==true, so it can be tested just like a swap pte that + * contains a valid swap entry, so that we can check a swap pte always + * using "is_swap_pte() && pte_swp_uffd_wp()" without caring about whether + * there's one swap entry inside of the pte. + * + * (4) It should not be a valid swap pte anywhere, so that when we see this pte + * we know it does not contain a swap entry. + * + * For x86, the simplest special pte which satisfies all of above should be the + * pte with only _PAGE_SWP_UFFD_WP bit set (where swp_type==swp_offset==0). + */ +#define UFFD_WP_SWP_PTE_SPECIAL __pte(_PAGE_SWP_UFFD_WP) + static inline pte_t pte_swp_mkuffd_wp(pte_t pte) { return pte_set_flags(pte, _PAGE_SWP_UFFD_WP); diff --git a/include/asm-generic/pgtable_uffd.h b/include/asm-generic/pgtable_uffd.h index 828966d4c281..95e9811ce9d1 100644 --- a/include/asm-generic/pgtable_uffd.h +++ b/include/asm-generic/pgtable_uffd.h @@ -2,6 +2,9 @@ #define _ASM_GENERIC_PGTABLE_UFFD_H #ifndef CONFIG_HAVE_ARCH_USERFAULTFD_WP + +#define UFFD_WP_SWP_PTE_SPECIAL __pte(0) + static __always_inline int pte_uffd_wp(pte_t pte) { return 0; diff --git a/include/linux/userfaultfd_k.h b/include/linux/userfaultfd_k.h index 331d2ccf0bcc..bb5a72a2b07a 100644 --- a/include/linux/userfaultfd_k.h +++ b/include/linux/userfaultfd_k.h @@ -145,6 +145,21 @@ extern int userfaultfd_unmap_prep(struct vm_area_struct *vma, extern void userfaultfd_unmap_complete(struct mm_struct *mm, struct list_head *uf); +static inline pte_t pte_swp_mkuffd_wp_special(struct vm_area_struct *vma) +{ + WARN_ON_ONCE(vma_is_anonymous(vma)); + return UFFD_WP_SWP_PTE_SPECIAL; +} + +static inline bool pte_swp_uffd_wp_special(pte_t pte) +{ +#ifdef CONFIG_HAVE_ARCH_USERFAULTFD_WP + return pte_same(pte, UFFD_WP_SWP_PTE_SPECIAL); +#else + return false; +#endif +} + #else /* CONFIG_USERFAULTFD */ /* mm helpers */ @@ -234,6 +249,16 @@ static inline void userfaultfd_unmap_complete(struct mm_struct *mm, { } +static inline pte_t pte_swp_mkuffd_wp_special(struct vm_area_struct *vma) +{ + return __pte(0); +} + +static inline bool pte_swp_uffd_wp_special(pte_t pte) +{ + return false; +} + #endif /* CONFIG_USERFAULTFD */ #endif /* _LINUX_USERFAULTFD_K_H */ From patchwork Wed Jul 14 22:20:56 2021 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Peter Xu X-Patchwork-Id: 12377947 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org X-Spam-Level: X-Spam-Status: No, score=-16.6 required=3.0 tests=BAYES_00,DKIM_INVALID, DKIM_SIGNED,HEADER_FROM_DIFFERENT_DOMAINS,INCLUDES_CR_TRAILER,INCLUDES_PATCH, MAILING_LIST_MULTI,SPF_HELO_NONE,SPF_PASS,USER_AGENT_GIT autolearn=ham autolearn_force=no version=3.4.0 Received: from mail.kernel.org (mail.kernel.org [198.145.29.99]) by smtp.lore.kernel.org (Postfix) with ESMTP id 07097C47E4B for ; Wed, 14 Jul 2021 22:21:36 +0000 (UTC) Received: from kanga.kvack.org (kanga.kvack.org [205.233.56.17]) by mail.kernel.org (Postfix) with ESMTP id 9F2F8613C5 for ; Wed, 14 Jul 2021 22:21:35 +0000 (UTC) DMARC-Filter: OpenDMARC Filter v1.3.2 mail.kernel.org 9F2F8613C5 Authentication-Results: mail.kernel.org; dmarc=fail (p=none dis=none) header.from=redhat.com Authentication-Results: mail.kernel.org; spf=pass smtp.mailfrom=owner-linux-mm@kvack.org Received: by kanga.kvack.org (Postfix) id 9CD6A6B00AF; Wed, 14 Jul 2021 18:21:33 -0400 (EDT) Received: by kanga.kvack.org (Postfix, from userid 40) id 97DED6B00B0; Wed, 14 Jul 2021 18:21:33 -0400 (EDT) X-Delivered-To: int-list-linux-mm@kvack.org Received: by kanga.kvack.org (Postfix, from userid 63042) id 7D0A08D0003; Wed, 14 Jul 2021 18:21:33 -0400 (EDT) X-Delivered-To: linux-mm@kvack.org Received: from forelay.hostedemail.com (smtprelay0016.hostedemail.com [216.40.44.16]) by kanga.kvack.org (Postfix) with ESMTP id 466E56B00AF for ; Wed, 14 Jul 2021 18:21:33 -0400 (EDT) Received: from smtpin01.hostedemail.com (10.5.19.251.rfc1918.com [10.5.19.251]) by forelay04.hostedemail.com (Postfix) with ESMTP id 307F1184A1 for ; Wed, 14 Jul 2021 22:21:32 +0000 (UTC) X-FDA: 78362616024.01.7763163 Received: from us-smtp-delivery-124.mimecast.com (us-smtp-delivery-124.mimecast.com [216.205.24.124]) by imf10.hostedemail.com (Postfix) with ESMTP id B5ADF60019B0 for ; Wed, 14 Jul 2021 22:21:31 +0000 (UTC) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=redhat.com; s=mimecast20190719; t=1626301291; h=from:from:reply-to:subject:subject:date:date:message-id:message-id: to:to:cc:cc:mime-version:mime-version:content-type:content-type: content-transfer-encoding:content-transfer-encoding: in-reply-to:in-reply-to:references:references; bh=tW4g1JQLYDPCRpXGiRYBmI8jE+7JZpiQ4Axmuiw/lnI=; b=E81g6Qc+JbsDdb6Jyg0X35+LQqTQxdot6dYNVQS+NpV8pFgLf/kxBjT0RdzUjosCtOYsma nh8/mOpCV+2BezsgIVj7y2odbzcrpawAOpVR79bUEfIk2c00Q+o3l5ECTKFy4JIaNVAQd5 OYa7oocD1kIcD1wINtWHVyFa5eFmcb0= Received: from mail-qk1-f200.google.com (mail-qk1-f200.google.com [209.85.222.200]) (Using TLS) by relay.mimecast.com with ESMTP id us-mta-291-xsjnsBzPO-yeQ72SWCxJDA-1; Wed, 14 Jul 2021 18:21:30 -0400 X-MC-Unique: xsjnsBzPO-yeQ72SWCxJDA-1 Received: by mail-qk1-f200.google.com with SMTP id x12-20020a05620a448cb02903b3dabbe857so2289485qkp.15 for ; Wed, 14 Jul 2021 15:21:30 -0700 (PDT) X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20161025; h=x-gm-message-state:from:to:cc:subject:date:message-id:in-reply-to :references:mime-version:content-transfer-encoding; bh=tW4g1JQLYDPCRpXGiRYBmI8jE+7JZpiQ4Axmuiw/lnI=; b=AAjVXgKD4qzcCsku6TQl/qIsxkNYazIukWgElWwy3sKZn2oqXmgUIRXfBlBJMa5KWG mSWl4fb97koIPB605POS+0tuyFWH4OAebeW6r/4RsiM0krycqlDu1R8E2sK3uAohSn7w Q5aRlm6aYCSbB7FJyC6J55A26lumD7btY6R06G9lH3noND4w6UGl+dGE+FkQMbA1rmSc Dqa7FZ5IjpsLk1q/IVpq0x9/FxmDf7HwQ7eS68kIk2y9v5NZo3LZcpAS6kdk/uqP8Nmo VzEYOzhzLy0YHV0piquHS3uN2Lwtrg9r28cIJN7N9fNg9TFq8i2mNP0MMnh8xFQkKGy9 N4bw== X-Gm-Message-State: AOAM533UKHghKS6LFxs8oN9IE6AweG2dHqOsUml+SdLXYG3+AlCSA79X c1cBE126QW44Ng18WHds84MnkfloVg4C6Y7exdDVKt58gv7q4k0fG5Gbjeb0Jeza+emjuUdBO5q Eqkmfccr4Ewo= X-Received: by 2002:a37:d09:: with SMTP id 9mr222711qkn.376.1626301289652; Wed, 14 Jul 2021 15:21:29 -0700 (PDT) X-Google-Smtp-Source: ABdhPJwQ4W1R1AMWsvRmdNb1onQySRSxxG/zbIp9k95xiMXq5G4QFzGxjwWdLimBjeR261G289jfhw== X-Received: by 2002:a37:d09:: with SMTP id 9mr222674qkn.376.1626301289351; Wed, 14 Jul 2021 15:21:29 -0700 (PDT) Received: from localhost.localdomain (bras-base-toroon474qw-grc-65-184-144-111-238.dsl.bell.ca. [184.144.111.238]) by smtp.gmail.com with ESMTPSA id b25sm1625854qka.123.2021.07.14.15.21.27 (version=TLS1_3 cipher=TLS_AES_256_GCM_SHA384 bits=256/256); Wed, 14 Jul 2021 15:21:28 -0700 (PDT) From: Peter Xu To: linux-kernel@vger.kernel.org, linux-mm@kvack.org Cc: Jason Gunthorpe , peterx@redhat.com, Matthew Wilcox , Andrew Morton , Axel Rasmussen , Nadav Amit , Jerome Glisse , Mike Rapoport , Miaohe Lin , Hugh Dickins , Alistair Popple , Andrea Arcangeli , Mike Kravetz , "Kirill A . Shutemov" , David Hildenbrand Subject: [PATCH v4 05/26] mm/swap: Introduce the idea of special swap ptes Date: Wed, 14 Jul 2021 18:20:56 -0400 Message-Id: <20210714222117.47648-6-peterx@redhat.com> X-Mailer: git-send-email 2.31.1 In-Reply-To: <20210714222117.47648-1-peterx@redhat.com> References: <20210714222117.47648-1-peterx@redhat.com> MIME-Version: 1.0 X-Mimecast-Spam-Score: 0 X-Mimecast-Originator: redhat.com Authentication-Results: imf10.hostedemail.com; dkim=pass header.d=redhat.com header.s=mimecast20190719 header.b=E81g6Qc+; spf=none (imf10.hostedemail.com: domain of peterx@redhat.com has no SPF policy when checking 216.205.24.124) smtp.mailfrom=peterx@redhat.com; dmarc=pass (policy=none) header.from=redhat.com X-Rspamd-Server: rspam02 X-Stat-Signature: z5fran6ns8uz1jow6qi8gy5b7k3down1 X-Rspamd-Queue-Id: B5ADF60019B0 X-HE-Tag: 1626301291-390949 X-Bogosity: Ham, tests=bogofilter, spamicity=0.000000, version=1.2.4 Sender: owner-linux-mm@kvack.org Precedence: bulk X-Loop: owner-majordomo@kvack.org List-ID: We used to have special swap entries, like migration entries, hw-poison entries, device private entries, etc. Those "special swap entries" reside in the range that they need to be at least swap entries first, and their types are decided by swp_type(entry). This patch introduces another idea called "special swap ptes". It's very easy to get confused against "special swap entries", but a speical swap pte should never contain a swap entry at all. It means, it's illegal to call pte_to_swp_entry() upon a special swap pte. Make the uffd-wp special pte to be the first special swap pte. Before this patch, is_swap_pte()==true means one of the below: (a.1) The pte has a normal swap entry (non_swap_entry()==false). For example, when an anonymous page got swapped out. (a.2) The pte has a special swap entry (non_swap_entry()==true). For example, a migration entry, a hw-poison entry, etc. After this patch, is_swap_pte()==true means one of the below, where case (b) is added: (a) The pte contains a swap entry. (a.1) The pte has a normal swap entry (non_swap_entry()==false). For example, when an anonymous page got swapped out. (a.2) The pte has a special swap entry (non_swap_entry()==true). For example, a migration entry, a hw-poison entry, etc. (b) The pte does not contain a swap entry at all (so it cannot be passed into pte_to_swp_entry()). For example, uffd-wp special swap pte. Teach the whole mm core about this new idea. It's done by introducing another helper called pte_has_swap_entry() which stands for case (a.1) and (a.2). Before this patch, it will be the same as is_swap_pte() because there's no special swap pte yet. Now for most of the previous use of is_swap_entry() in mm core, we'll need to use the new helper pte_has_swap_entry() instead, to make sure we won't try to parse a swap entry from a swap special pte (which does not contain a swap entry at all!). We either handle the swap special pte, or it'll naturally use the default "else" paths. Warn properly (e.g., in do_swap_page()) when we see a special swap pte - we should never call do_swap_page() upon those ptes, but just to bail out early if it happens. Signed-off-by: Peter Xu --- arch/arm64/kernel/mte.c | 2 +- fs/proc/task_mmu.c | 14 ++++++++------ include/linux/swapops.h | 39 ++++++++++++++++++++++++++++++++++++++- mm/gup.c | 2 +- mm/hmm.c | 2 +- mm/khugepaged.c | 11 ++++++++++- mm/madvise.c | 4 ++-- mm/memcontrol.c | 2 +- mm/memory.c | 7 +++++++ mm/migrate.c | 4 ++-- mm/mincore.c | 2 +- mm/mprotect.c | 2 +- mm/mremap.c | 2 +- mm/page_vma_mapped.c | 6 +++--- mm/swapfile.c | 2 +- 15 files changed, 78 insertions(+), 23 deletions(-) diff --git a/arch/arm64/kernel/mte.c b/arch/arm64/kernel/mte.c index 69b3fde8759e..841ff639b4b5 100644 --- a/arch/arm64/kernel/mte.c +++ b/arch/arm64/kernel/mte.c @@ -35,7 +35,7 @@ EXPORT_SYMBOL_GPL(mte_async_mode); static void mte_sync_page_tags(struct page *page, pte_t old_pte, bool check_swap, bool pte_is_tagged) { - if (check_swap && is_swap_pte(old_pte)) { + if (check_swap && pte_has_swap_entry(old_pte)) { swp_entry_t entry = pte_to_swp_entry(old_pte); if (!non_swap_entry(entry) && mte_restore_tags(entry, page)) diff --git a/fs/proc/task_mmu.c b/fs/proc/task_mmu.c index eb97468dfe4c..9c5af77b5290 100644 --- a/fs/proc/task_mmu.c +++ b/fs/proc/task_mmu.c @@ -498,7 +498,7 @@ static void smaps_pte_entry(pte_t *pte, unsigned long addr, if (pte_present(*pte)) { page = vm_normal_page(vma, addr, *pte); - } else if (is_swap_pte(*pte)) { + } else if (pte_has_swap_entry(*pte)) { swp_entry_t swpent = pte_to_swp_entry(*pte); if (!non_swap_entry(swpent)) { @@ -516,8 +516,10 @@ static void smaps_pte_entry(pte_t *pte, unsigned long addr, } } else if (is_pfn_swap_entry(swpent)) page = pfn_swap_entry_to_page(swpent); - } else if (unlikely(IS_ENABLED(CONFIG_SHMEM) && mss->check_shmem_swap - && pte_none(*pte))) { + } else if (unlikely(IS_ENABLED(CONFIG_SHMEM) && + mss->check_shmem_swap && + /* Here swap special pte is the same as none pte */ + (pte_none(*pte) || is_swap_special_pte(*pte)))) { page = xa_load(&vma->vm_file->f_mapping->i_pages, linear_page_index(vma, addr)); if (xa_is_value(page)) @@ -689,7 +691,7 @@ static int smaps_hugetlb_range(pte_t *pte, unsigned long hmask, if (pte_present(*pte)) { page = vm_normal_page(vma, addr, *pte); - } else if (is_swap_pte(*pte)) { + } else if (pte_has_swap_entry(*pte)) { swp_entry_t swpent = pte_to_swp_entry(*pte); if (is_pfn_swap_entry(swpent)) @@ -1071,7 +1073,7 @@ static inline void clear_soft_dirty(struct vm_area_struct *vma, ptent = pte_wrprotect(old_pte); ptent = pte_clear_soft_dirty(ptent); ptep_modify_prot_commit(vma, addr, pte, old_pte, ptent); - } else if (is_swap_pte(ptent)) { + } else if (pte_has_swap_entry(ptent)) { ptent = pte_swp_clear_soft_dirty(ptent); set_pte_at(vma->vm_mm, addr, pte, ptent); } @@ -1374,7 +1376,7 @@ static pagemap_entry_t pte_to_pagemap_entry(struct pagemapread *pm, flags |= PM_SOFT_DIRTY; if (pte_uffd_wp(pte)) flags |= PM_UFFD_WP; - } else if (is_swap_pte(pte)) { + } else if (pte_has_swap_entry(pte)) { swp_entry_t entry; if (pte_swp_soft_dirty(pte)) flags |= PM_SOFT_DIRTY; diff --git a/include/linux/swapops.h b/include/linux/swapops.h index d356ab4047f7..7f46dec3be1d 100644 --- a/include/linux/swapops.h +++ b/include/linux/swapops.h @@ -5,6 +5,7 @@ #include #include #include +#include #ifdef CONFIG_MMU @@ -62,12 +63,48 @@ static inline pgoff_t swp_offset(swp_entry_t entry) return entry.val & SWP_OFFSET_MASK; } -/* check whether a pte points to a swap entry */ +/* + * is_swap_pte() returns true for three cases: + * + * (a) The pte contains a swap entry. + * + * (a.1) The pte has a normal swap entry (non_swap_entry()==false). For + * example, when an anonymous page got swapped out. + * + * (a.2) The pte has a special swap entry (non_swap_entry()==true). For + * example, a migration entry, a hw-poison entry, etc. + * + * (b) The pte does not contain a swap entry at all (so it cannot be passed + * into pte_to_swp_entry()). For example, uffd-wp special swap pte. + */ static inline int is_swap_pte(pte_t pte) { return !pte_none(pte) && !pte_present(pte); } +/* + * A swap-like special pte should only be used as special marker to trigger a + * page fault. We should treat them similarly as pte_none() in most cases, + * except that it may contain some special information that can persist within + * the pte. Currently the only special swap pte is UFFD_WP_SWP_PTE_SPECIAL. + * + * Note: we should never call pte_to_swp_entry() upon a special swap pte, + * Because a swap special pte does not contain a swap entry! + */ +static inline bool is_swap_special_pte(pte_t pte) +{ + return pte_swp_uffd_wp_special(pte); +} + +/* + * Returns true if the pte contains a swap entry. This includes not only the + * normal swp entry case, but also for migration entries, etc. + */ +static inline bool pte_has_swap_entry(pte_t pte) +{ + return is_swap_pte(pte) && !is_swap_special_pte(pte); +} + /* * Convert the arch-dependent pte representation of a swp_entry_t into an * arch-independent swp_entry_t. diff --git a/mm/gup.c b/mm/gup.c index 42b8b1fa6521..425c08788921 100644 --- a/mm/gup.c +++ b/mm/gup.c @@ -513,7 +513,7 @@ static struct page *follow_page_pte(struct vm_area_struct *vma, */ if (likely(!(flags & FOLL_MIGRATION))) goto no_page; - if (pte_none(pte)) + if (!pte_has_swap_entry(pte)) goto no_page; entry = pte_to_swp_entry(pte); if (!is_migration_entry(entry)) diff --git a/mm/hmm.c b/mm/hmm.c index fad6be2bf072..aba1bf2c6742 100644 --- a/mm/hmm.c +++ b/mm/hmm.c @@ -239,7 +239,7 @@ static int hmm_vma_handle_pte(struct mm_walk *walk, unsigned long addr, pte_t pte = *ptep; uint64_t pfn_req_flags = *hmm_pfn; - if (pte_none(pte)) { + if (pte_none(pte) || is_swap_special_pte(pte)) { required_fault = hmm_pte_need_fault(hmm_vma_walk, pfn_req_flags, 0); if (required_fault) diff --git a/mm/khugepaged.c b/mm/khugepaged.c index b0412be08fa2..7376a9b5bfc9 100644 --- a/mm/khugepaged.c +++ b/mm/khugepaged.c @@ -1018,7 +1018,7 @@ static bool __collapse_huge_page_swapin(struct mm_struct *mm, vmf.pte = pte_offset_map(pmd, address); vmf.orig_pte = *vmf.pte; - if (!is_swap_pte(vmf.orig_pte)) { + if (!pte_has_swap_entry(vmf.orig_pte)) { pte_unmap(vmf.pte); continue; } @@ -1245,6 +1245,15 @@ static int khugepaged_scan_pmd(struct mm_struct *mm, _pte++, _address += PAGE_SIZE) { pte_t pteval = *_pte; if (is_swap_pte(pteval)) { + if (is_swap_special_pte(pteval)) { + /* + * Reuse SCAN_PTE_UFFD_WP. If there will be + * new users of is_swap_special_pte(), we'd + * better introduce a new result type. + */ + result = SCAN_PTE_UFFD_WP; + goto out_unmap; + } if (++unmapped <= khugepaged_max_ptes_swap) { /* * Always be strict with uffd-wp diff --git a/mm/madvise.c b/mm/madvise.c index 6d3d348b17f4..2a8d2a9fc514 100644 --- a/mm/madvise.c +++ b/mm/madvise.c @@ -204,7 +204,7 @@ static int swapin_walk_pmd_entry(pmd_t *pmd, unsigned long start, pte = *(orig_pte + ((index - start) / PAGE_SIZE)); pte_unmap_unlock(orig_pte, ptl); - if (pte_present(pte) || pte_none(pte)) + if (!pte_has_swap_entry(pte)) continue; entry = pte_to_swp_entry(pte); if (unlikely(non_swap_entry(entry))) @@ -596,7 +596,7 @@ static int madvise_free_pte_range(pmd_t *pmd, unsigned long addr, for (; addr != end; pte++, addr += PAGE_SIZE) { ptent = *pte; - if (pte_none(ptent)) + if (pte_none(ptent) || is_swap_special_pte(ptent)) continue; /* * If the pte has swp_entry, just clear page table to diff --git a/mm/memcontrol.c b/mm/memcontrol.c index ae1f5d0cb581..4b46c099ad94 100644 --- a/mm/memcontrol.c +++ b/mm/memcontrol.c @@ -5738,7 +5738,7 @@ static enum mc_target_type get_mctgt_type(struct vm_area_struct *vma, if (pte_present(ptent)) page = mc_handle_present_pte(vma, addr, ptent); - else if (is_swap_pte(ptent)) + else if (pte_has_swap_entry(ptent)) page = mc_handle_swap_pte(vma, ptent, &ent); else if (pte_none(ptent)) page = mc_handle_file_pte(vma, addr, ptent, &ent); diff --git a/mm/memory.c b/mm/memory.c index 0e0de08a2cd5..998a4f9a3744 100644 --- a/mm/memory.c +++ b/mm/memory.c @@ -3491,6 +3491,13 @@ vm_fault_t do_swap_page(struct vm_fault *vmf) if (!pte_unmap_same(vmf)) goto out; + /* + * We should never call do_swap_page upon a swap special pte; just be + * safe to bail out if it happens. + */ + if (WARN_ON_ONCE(is_swap_special_pte(vmf->orig_pte))) + goto out; + entry = pte_to_swp_entry(vmf->orig_pte); if (unlikely(non_swap_entry(entry))) { if (is_migration_entry(entry)) { diff --git a/mm/migrate.c b/mm/migrate.c index 23cbd9de030b..b477d0d5f911 100644 --- a/mm/migrate.c +++ b/mm/migrate.c @@ -294,7 +294,7 @@ void __migration_entry_wait(struct mm_struct *mm, pte_t *ptep, spin_lock(ptl); pte = *ptep; - if (!is_swap_pte(pte)) + if (!pte_has_swap_entry(pte)) goto out; entry = pte_to_swp_entry(pte); @@ -2276,7 +2276,7 @@ static int migrate_vma_collect_pmd(pmd_t *pmdp, pte = *ptep; - if (pte_none(pte)) { + if (pte_none(pte) || is_swap_special_pte(pte)) { if (vma_is_anonymous(vma)) { mpfn = MIGRATE_PFN_MIGRATE; migrate->cpages++; diff --git a/mm/mincore.c b/mm/mincore.c index 9122676b54d6..5728c3e6473f 100644 --- a/mm/mincore.c +++ b/mm/mincore.c @@ -121,7 +121,7 @@ static int mincore_pte_range(pmd_t *pmd, unsigned long addr, unsigned long end, for (; addr != end; ptep++, addr += PAGE_SIZE) { pte_t pte = *ptep; - if (pte_none(pte)) + if (pte_none(pte) || is_swap_special_pte(pte)) __mincore_unmapped_range(addr, addr + PAGE_SIZE, vma, vec); else if (pte_present(pte)) diff --git a/mm/mprotect.c b/mm/mprotect.c index 883e2cc85cad..4b743394afbe 100644 --- a/mm/mprotect.c +++ b/mm/mprotect.c @@ -139,7 +139,7 @@ static unsigned long change_pte_range(struct vm_area_struct *vma, pmd_t *pmd, } ptep_modify_prot_commit(vma, addr, pte, oldpte, ptent); pages++; - } else if (is_swap_pte(oldpte)) { + } else if (pte_has_swap_entry(oldpte)) { swp_entry_t entry = pte_to_swp_entry(oldpte); pte_t newpte; diff --git a/mm/mremap.c b/mm/mremap.c index 5989d3990020..122b279333ee 100644 --- a/mm/mremap.c +++ b/mm/mremap.c @@ -125,7 +125,7 @@ static pte_t move_soft_dirty_pte(pte_t pte) #ifdef CONFIG_MEM_SOFT_DIRTY if (pte_present(pte)) pte = pte_mksoft_dirty(pte); - else if (is_swap_pte(pte)) + else if (pte_has_swap_entry(pte)) pte = pte_swp_mksoft_dirty(pte); #endif return pte; diff --git a/mm/page_vma_mapped.c b/mm/page_vma_mapped.c index f7b331081791..ff57b67426af 100644 --- a/mm/page_vma_mapped.c +++ b/mm/page_vma_mapped.c @@ -36,7 +36,7 @@ static bool map_pte(struct page_vma_mapped_walk *pvmw) * For more details on device private memory see HMM * (include/linux/hmm.h or mm/hmm.c). */ - if (is_swap_pte(*pvmw->pte)) { + if (pte_has_swap_entry(*pvmw->pte)) { swp_entry_t entry; /* Handle un-addressable ZONE_DEVICE memory */ @@ -90,7 +90,7 @@ static bool check_pte(struct page_vma_mapped_walk *pvmw) if (pvmw->flags & PVMW_MIGRATION) { swp_entry_t entry; - if (!is_swap_pte(*pvmw->pte)) + if (!pte_has_swap_entry(*pvmw->pte)) return false; entry = pte_to_swp_entry(*pvmw->pte); @@ -99,7 +99,7 @@ static bool check_pte(struct page_vma_mapped_walk *pvmw) return false; pfn = swp_offset(entry); - } else if (is_swap_pte(*pvmw->pte)) { + } else if (pte_has_swap_entry(*pvmw->pte)) { swp_entry_t entry; /* Handle un-addressable ZONE_DEVICE memory */ diff --git a/mm/swapfile.c b/mm/swapfile.c index 1e07d1c776f2..4993b4454c13 100644 --- a/mm/swapfile.c +++ b/mm/swapfile.c @@ -1951,7 +1951,7 @@ static int unuse_pte_range(struct vm_area_struct *vma, pmd_t *pmd, si = swap_info[type]; pte = pte_offset_map(pmd, addr); do { - if (!is_swap_pte(*pte)) + if (!pte_has_swap_entry(*pte)) continue; entry = pte_to_swp_entry(*pte); From patchwork Wed Jul 14 22:20:57 2021 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Peter Xu X-Patchwork-Id: 12377949 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org X-Spam-Level: X-Spam-Status: No, score=-16.6 required=3.0 tests=BAYES_00,DKIM_INVALID, DKIM_SIGNED,HEADER_FROM_DIFFERENT_DOMAINS,INCLUDES_CR_TRAILER,INCLUDES_PATCH, MAILING_LIST_MULTI,SPF_HELO_NONE,SPF_PASS,USER_AGENT_GIT autolearn=ham autolearn_force=no version=3.4.0 Received: from mail.kernel.org (mail.kernel.org [198.145.29.99]) by smtp.lore.kernel.org (Postfix) with ESMTP id 005CDC1B08C for ; Wed, 14 Jul 2021 22:21:38 +0000 (UTC) Received: from kanga.kvack.org (kanga.kvack.org [205.233.56.17]) by mail.kernel.org (Postfix) with ESMTP id A3BB8613C5 for ; Wed, 14 Jul 2021 22:21:37 +0000 (UTC) DMARC-Filter: OpenDMARC Filter v1.3.2 mail.kernel.org A3BB8613C5 Authentication-Results: mail.kernel.org; dmarc=fail (p=none dis=none) header.from=redhat.com Authentication-Results: mail.kernel.org; spf=pass smtp.mailfrom=owner-linux-mm@kvack.org Received: by kanga.kvack.org (Postfix) id 5FCD36B00B1; Wed, 14 Jul 2021 18:21:35 -0400 (EDT) Received: by kanga.kvack.org (Postfix, from userid 40) id 5D2688D0003; Wed, 14 Jul 2021 18:21:35 -0400 (EDT) X-Delivered-To: int-list-linux-mm@kvack.org Received: by kanga.kvack.org (Postfix, from userid 63042) id 3D9286B00B3; Wed, 14 Jul 2021 18:21:35 -0400 (EDT) X-Delivered-To: linux-mm@kvack.org Received: from forelay.hostedemail.com (smtprelay0111.hostedemail.com [216.40.44.111]) by kanga.kvack.org (Postfix) with ESMTP id 1ACB66B00B1 for ; Wed, 14 Jul 2021 18:21:35 -0400 (EDT) Received: from smtpin32.hostedemail.com (10.5.19.251.rfc1918.com [10.5.19.251]) by forelay03.hostedemail.com (Postfix) with ESMTP id 1016B824556B for ; Wed, 14 Jul 2021 22:21:34 +0000 (UTC) X-FDA: 78362616108.32.C86A7FC Received: from us-smtp-delivery-124.mimecast.com (us-smtp-delivery-124.mimecast.com [216.205.24.124]) by imf01.hostedemail.com (Postfix) with ESMTP id A75B6500C72F for ; Wed, 14 Jul 2021 22:21:33 +0000 (UTC) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=redhat.com; s=mimecast20190719; t=1626301293; h=from:from:reply-to:subject:subject:date:date:message-id:message-id: to:to:cc:cc:mime-version:mime-version:content-type:content-type: content-transfer-encoding:content-transfer-encoding: in-reply-to:in-reply-to:references:references; bh=qDlI2UiTtU/QeHLwRdnThmo8cw3zlDxrqquxhgngB94=; b=BrDiX8FA8NHqeKZD62WpDEel6pos3JDUnGzhAq+58sHjnIPVXxPhblRP1l//u44rMn+/IY zp/b0QrMiTt6WQzwPhV3LU816hQAxWhlSg+MPrSRVbNuX8BIHVC4L1aOa0UcpToE7Lj7g6 eqmGsorP0dr1I2OQEXllQ4wPKGcYNvI= Received: from mail-qv1-f71.google.com (mail-qv1-f71.google.com [209.85.219.71]) (Using TLS) by relay.mimecast.com with ESMTP id us-mta-415-6L9EmHHIOES6c-qXFuXGyg-1; Wed, 14 Jul 2021 18:21:32 -0400 X-MC-Unique: 6L9EmHHIOES6c-qXFuXGyg-1 Received: by mail-qv1-f71.google.com with SMTP id z93-20020a0ca5e60000b02902e558bb7a04so2683807qvz.10 for ; Wed, 14 Jul 2021 15:21:32 -0700 (PDT) X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20161025; h=x-gm-message-state:from:to:cc:subject:date:message-id:in-reply-to :references:mime-version:content-transfer-encoding; bh=qDlI2UiTtU/QeHLwRdnThmo8cw3zlDxrqquxhgngB94=; b=FCDhoScdZFLnM4duJjbY1eaZkxLyaMKiH9lcP2ICmYuvZwmlGJapu9ORLp9xnXr4Qn GE8v253u/Uz3Aya7NQlxTp2kflinxPjfhU/iN+YhrQJvb6Gy5WP7Fo5yB9wHUVIytFCj nPDkTab3+LlnFW6bhv0QVqjChD9cRnrK2fMXEXmoZ/ZYpAoLI20fTxZar6Ukqyt8leeB MYvpEulf+1PCmxewqipNNFJGFgqwBvRbUOECY0fd4OhQ47gZHyw+2o1OBaLzehiVQev9 SEQdH47eF3y8mbd3egHiR0RduW7/UFFskXDFymnpEHmKuhH+kz3ET8xtUm2MsMkjLzZQ U7Mw== X-Gm-Message-State: AOAM530mMHgM0M4cREG6I16ae+Ous7/5gXk0aDvlPnFQBzEVD/ev00Oo Hk5YG6cU1PzMI3cBDf/tNQKRnagsfAhXc+I+J2LJ4DquNPdzDy/XEKsCahYcXm9mvTzAUe48qdh aiZ1zCWEE6+8= X-Received: by 2002:ae9:e606:: with SMTP id z6mr222025qkf.326.1626301291471; Wed, 14 Jul 2021 15:21:31 -0700 (PDT) X-Google-Smtp-Source: ABdhPJzSs58KofxTXjDcRFIhAxH5oh2lw5GmZzcqsYGel0HFBZYU0Vo400V5xN/B8WiWscEQ6rKeLA== X-Received: by 2002:ae9:e606:: with SMTP id z6mr221993qkf.326.1626301291213; Wed, 14 Jul 2021 15:21:31 -0700 (PDT) Received: from localhost.localdomain (bras-base-toroon474qw-grc-65-184-144-111-238.dsl.bell.ca. [184.144.111.238]) by smtp.gmail.com with ESMTPSA id b25sm1625854qka.123.2021.07.14.15.21.29 (version=TLS1_3 cipher=TLS_AES_256_GCM_SHA384 bits=256/256); Wed, 14 Jul 2021 15:21:30 -0700 (PDT) From: Peter Xu To: linux-kernel@vger.kernel.org, linux-mm@kvack.org Cc: Jason Gunthorpe , peterx@redhat.com, Matthew Wilcox , Andrew Morton , Axel Rasmussen , Nadav Amit , Jerome Glisse , Mike Rapoport , Miaohe Lin , Hugh Dickins , Alistair Popple , Andrea Arcangeli , Mike Kravetz , "Kirill A . Shutemov" , David Hildenbrand Subject: [PATCH v4 06/26] shmem/userfaultfd: Handle uffd-wp special pte in page fault handler Date: Wed, 14 Jul 2021 18:20:57 -0400 Message-Id: <20210714222117.47648-7-peterx@redhat.com> X-Mailer: git-send-email 2.31.1 In-Reply-To: <20210714222117.47648-1-peterx@redhat.com> References: <20210714222117.47648-1-peterx@redhat.com> MIME-Version: 1.0 X-Mimecast-Spam-Score: 0 X-Mimecast-Originator: redhat.com Authentication-Results: imf01.hostedemail.com; dkim=pass header.d=redhat.com header.s=mimecast20190719 header.b=BrDiX8FA; spf=none (imf01.hostedemail.com: domain of peterx@redhat.com has no SPF policy when checking 216.205.24.124) smtp.mailfrom=peterx@redhat.com; dmarc=pass (policy=none) header.from=redhat.com X-Rspamd-Server: rspam05 X-Stat-Signature: azmpit95jgkthuzhnk8196n4cgqufu3d X-Rspamd-Queue-Id: A75B6500C72F X-HE-Tag: 1626301293-637388 X-Bogosity: Ham, tests=bogofilter, spamicity=0.000000, version=1.2.4 Sender: owner-linux-mm@kvack.org Precedence: bulk X-Loop: owner-majordomo@kvack.org List-ID: File-backed memories are prone to unmap/swap so the ptes are always unstable. This could lead to userfaultfd-wp information got lost when unmapped or swapped out on such types of memory, for example, shmem. To keep such an information persistent, we will start to use the newly introduced swap-like special ptes to replace a null pte when those ptes were removed. Prepare this by handling such a special pte first before it is applied in the general page fault handler. The handling of this special pte page fault is similar to missing fault, but it should happen after the pte missing logic since the special pte is designed to be a swap-like pte. Meanwhile it should be handled before do_swap_page() so that the swap core logic won't be confused to see such an illegal swap pte. This is a slow path of uffd-wp handling, because unmap of wr-protected shmem ptes should be rare. So far it should only trigger in two conditions: (1) When trying to punch holes in shmem_fallocate(), there will be a pre-unmap optimization before evicting the page. That will create unmapped shmem ptes with wr-protected pages covered. (2) Swapping out of shmem pages Because of this, the page fault handling is simplifed too by not sending the wr-protect message in the 1st page fault, instead the page will be installed read-only, so the message will be generated until the next write, which will trigger the do_wp_page() path of general uffd-wp handling. Disable fault-around for all uffd-wp registered ranges for extra safety, and clean the code up a bit after we introduced MINOR fault. Signed-off-by: Peter Xu Reported-by: kernel test robot --- include/linux/userfaultfd_k.h | 12 +++++ mm/memory.c | 88 +++++++++++++++++++++++++++++++---- 2 files changed, 90 insertions(+), 10 deletions(-) diff --git a/include/linux/userfaultfd_k.h b/include/linux/userfaultfd_k.h index bb5a72a2b07a..47c74ef635ab 100644 --- a/include/linux/userfaultfd_k.h +++ b/include/linux/userfaultfd_k.h @@ -94,6 +94,18 @@ static inline bool uffd_disable_huge_pmd_share(struct vm_area_struct *vma) return vma->vm_flags & (VM_UFFD_WP | VM_UFFD_MINOR); } +/* + * Don't do fault around for either WP or MINOR registered uffd range. For + * MINOR registered range, fault around will be a total disaster and ptes can + * be installed without notifications; for WP it should mostly be fine as long + * as the fault around checks for pte_none() before the installation, however + * to be super safe we just forbid it. + */ +static inline bool uffd_disable_fault_around(struct vm_area_struct *vma) +{ + return vma->vm_flags & (VM_UFFD_WP | VM_UFFD_MINOR); +} + static inline bool userfaultfd_missing(struct vm_area_struct *vma) { return vma->vm_flags & VM_UFFD_MISSING; diff --git a/mm/memory.c b/mm/memory.c index 998a4f9a3744..ba8033ca6682 100644 --- a/mm/memory.c +++ b/mm/memory.c @@ -3964,6 +3964,7 @@ vm_fault_t do_set_pmd(struct vm_fault *vmf, struct page *page) void do_set_pte(struct vm_fault *vmf, struct page *page, unsigned long addr) { struct vm_area_struct *vma = vmf->vma; + bool uffd_wp = pte_swp_uffd_wp_special(vmf->orig_pte); bool write = vmf->flags & FAULT_FLAG_WRITE; bool prefault = vmf->address != addr; pte_t entry; @@ -3978,6 +3979,8 @@ void do_set_pte(struct vm_fault *vmf, struct page *page, unsigned long addr) if (write) entry = maybe_mkwrite(pte_mkdirty(entry), vma); + if (unlikely(uffd_wp)) + entry = pte_mkuffd_wp(pte_wrprotect(entry)); /* copy-on-write page */ if (write && !(vma->vm_flags & VM_SHARED)) { inc_mm_counter_fast(vma->vm_mm, MM_ANONPAGES); @@ -4045,8 +4048,12 @@ vm_fault_t finish_fault(struct vm_fault *vmf) vmf->pte = pte_offset_map_lock(vma->vm_mm, vmf->pmd, vmf->address, &vmf->ptl); ret = 0; - /* Re-check under ptl */ - if (likely(pte_none(*vmf->pte))) + + /* + * Re-check under ptl. Note: this will cover both none pte and + * uffd-wp-special swap pte + */ + if (likely(pte_same(*vmf->pte, vmf->orig_pte))) do_set_pte(vmf, page, vmf->address); else ret = VM_FAULT_NOPAGE; @@ -4150,9 +4157,21 @@ static vm_fault_t do_fault_around(struct vm_fault *vmf) return vmf->vma->vm_ops->map_pages(vmf, start_pgoff, end_pgoff); } +/* Return true if we should do read fault-around, false otherwise */ +static inline bool should_fault_around(struct vm_fault *vmf) +{ + /* No ->map_pages? No way to fault around... */ + if (!vmf->vma->vm_ops->map_pages) + return false; + + if (uffd_disable_fault_around(vmf->vma)) + return false; + + return fault_around_bytes >> PAGE_SHIFT > 1; +} + static vm_fault_t do_read_fault(struct vm_fault *vmf) { - struct vm_area_struct *vma = vmf->vma; vm_fault_t ret = 0; /* @@ -4160,12 +4179,10 @@ static vm_fault_t do_read_fault(struct vm_fault *vmf) * if page by the offset is not ready to be mapped (cold cache or * something). */ - if (vma->vm_ops->map_pages && fault_around_bytes >> PAGE_SHIFT > 1) { - if (likely(!userfaultfd_minor(vmf->vma))) { - ret = do_fault_around(vmf); - if (ret) - return ret; - } + if (should_fault_around(vmf)) { + ret = do_fault_around(vmf); + if (ret) + return ret; } ret = __do_fault(vmf); @@ -4484,6 +4501,57 @@ static vm_fault_t wp_huge_pud(struct vm_fault *vmf, pud_t orig_pud) return VM_FAULT_FALLBACK; } +static vm_fault_t uffd_wp_clear_special(struct vm_fault *vmf) +{ + vmf->pte = pte_offset_map_lock(vmf->vma->vm_mm, vmf->pmd, + vmf->address, &vmf->ptl); + /* + * Be careful so that we will only recover a special uffd-wp pte into a + * none pte. Otherwise it means the pte could have changed, so retry. + */ + if (pte_swp_uffd_wp_special(*vmf->pte)) + pte_clear(vmf->vma->vm_mm, vmf->address, vmf->pte); + pte_unmap_unlock(vmf->pte, vmf->ptl); + return 0; +} + +/* + * This is actually a page-missing access, but with uffd-wp special pte + * installed. It means this pte was wr-protected before being unmapped. + */ +static vm_fault_t uffd_wp_handle_special(struct vm_fault *vmf) +{ + /* Careful! vmf->pte unmapped after return */ + if (!pte_unmap_same(vmf)) + return 0; + + /* + * Just in case there're leftover special ptes even after the region + * got unregistered - we can simply clear them. + */ + if (unlikely(!userfaultfd_wp(vmf->vma) || vma_is_anonymous(vmf->vma))) + return uffd_wp_clear_special(vmf); + + /* + * Here we share most code with do_fault(), in which we can identify + * whether this is "none pte fault" or "uffd-wp-special fault" by + * checking the vmf->orig_pte. + */ + return do_fault(vmf); +} + +static vm_fault_t do_swap_pte(struct vm_fault *vmf) +{ + /* + * We need to handle special swap ptes before handling ptes that + * contain swap entries, always. + */ + if (unlikely(pte_swp_uffd_wp_special(vmf->orig_pte))) + return uffd_wp_handle_special(vmf); + + return do_swap_page(vmf); +} + /* * These routines also need to handle stuff like marking pages dirty * and/or accessed for architectures that don't do it in hardware (most @@ -4558,7 +4626,7 @@ static vm_fault_t handle_pte_fault(struct vm_fault *vmf) } if (!pte_present(vmf->orig_pte)) - return do_swap_page(vmf); + return do_swap_pte(vmf); if (pte_protnone(vmf->orig_pte) && vma_is_accessible(vmf->vma)) return do_numa_page(vmf); From patchwork Wed Jul 14 22:20:58 2021 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Peter Xu X-Patchwork-Id: 12377951 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org X-Spam-Level: X-Spam-Status: No, score=-16.6 required=3.0 tests=BAYES_00,DKIM_INVALID, DKIM_SIGNED,HEADER_FROM_DIFFERENT_DOMAINS,INCLUDES_CR_TRAILER,INCLUDES_PATCH, MAILING_LIST_MULTI,SPF_HELO_NONE,SPF_PASS,URIBL_BLOCKED,USER_AGENT_GIT autolearn=ham autolearn_force=no version=3.4.0 Received: from mail.kernel.org (mail.kernel.org [198.145.29.99]) by smtp.lore.kernel.org (Postfix) with ESMTP id D35D5C47E4B for ; Wed, 14 Jul 2021 22:21:39 +0000 (UTC) Received: from kanga.kvack.org (kanga.kvack.org [205.233.56.17]) by mail.kernel.org (Postfix) with ESMTP id 82D0A613C5 for ; Wed, 14 Jul 2021 22:21:39 +0000 (UTC) DMARC-Filter: OpenDMARC Filter v1.3.2 mail.kernel.org 82D0A613C5 Authentication-Results: mail.kernel.org; dmarc=fail (p=none dis=none) header.from=redhat.com Authentication-Results: mail.kernel.org; spf=pass smtp.mailfrom=owner-linux-mm@kvack.org Received: by kanga.kvack.org (Postfix) id 474916B00B3; Wed, 14 Jul 2021 18:21:37 -0400 (EDT) Received: by kanga.kvack.org (Postfix, from userid 40) id 44B686B00B4; Wed, 14 Jul 2021 18:21:37 -0400 (EDT) X-Delivered-To: int-list-linux-mm@kvack.org Received: by kanga.kvack.org (Postfix, from userid 63042) id 2C8BB6B00B5; Wed, 14 Jul 2021 18:21:37 -0400 (EDT) X-Delivered-To: linux-mm@kvack.org Received: from forelay.hostedemail.com (smtprelay0152.hostedemail.com [216.40.44.152]) by kanga.kvack.org (Postfix) with ESMTP id 0A8FF6B00B3 for ; Wed, 14 Jul 2021 18:21:37 -0400 (EDT) Received: from smtpin40.hostedemail.com (10.5.19.251.rfc1918.com [10.5.19.251]) by forelay02.hostedemail.com (Postfix) with ESMTP id E8BDA18F69 for ; Wed, 14 Jul 2021 22:21:35 +0000 (UTC) X-FDA: 78362616150.40.C1FFEF7 Received: from us-smtp-delivery-124.mimecast.com (us-smtp-delivery-124.mimecast.com [170.10.133.124]) by imf09.hostedemail.com (Postfix) with ESMTP id 9002C3000106 for ; Wed, 14 Jul 2021 22:21:35 +0000 (UTC) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=redhat.com; s=mimecast20190719; t=1626301295; h=from:from:reply-to:subject:subject:date:date:message-id:message-id: to:to:cc:cc:mime-version:mime-version:content-type:content-type: content-transfer-encoding:content-transfer-encoding: in-reply-to:in-reply-to:references:references; bh=qiHbtSxpM+Odw+UliFXdE+ZYHxP5T/ev+d+bKicm4LE=; b=QJlew4AxR82Vp22I3IhaDSpsrDf+tloDrR4Wp2mGwhyD5V0kNbqa5R9cTsW8gPHIpMyQ3V CfC8JIGVH8SmsosyRm9oYWBiSKN+bGwbbank1xffIBIe8cxYM5YSOcUebQW/V2wjzvY0D4 KRKjeOYxycoOpge3ZQ87YCmpBWgAxD0= Received: from mail-qv1-f70.google.com (mail-qv1-f70.google.com [209.85.219.70]) (Using TLS) by relay.mimecast.com with ESMTP id us-mta-496-5ZxCLssiNYK1UaA0m7A79w-1; Wed, 14 Jul 2021 18:21:33 -0400 X-MC-Unique: 5ZxCLssiNYK1UaA0m7A79w-1 Received: by mail-qv1-f70.google.com with SMTP id z1-20020a0cfec10000b02902dbb4e0a8f2so2706848qvs.6 for ; Wed, 14 Jul 2021 15:21:33 -0700 (PDT) X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20161025; h=x-gm-message-state:from:to:cc:subject:date:message-id:in-reply-to :references:mime-version:content-transfer-encoding; bh=qiHbtSxpM+Odw+UliFXdE+ZYHxP5T/ev+d+bKicm4LE=; b=jZ7P+9zPykd5sNtsUy3jA3SvO/uwbZVMI0KD2t7SNXthurDcYM6EGDqYConQPKHePt vxyP489oNCWGSO1z8pW1SRK4IUfSzjCEF5hpKow2qn1GsDS/SAxrFIZvuT14+GWxo0+z ayfV/qrYVeWVoYhXDENHrN0rj8SaCNXihZ8ELqZZ0RZu3rXnMXRoIJdhMNvice/rGjM6 F5u2k0t5JdrYOGtV+gPiEf/romCwefDPmXD8IAzyJRQ2Ol/uBJpAGmcQq7iFjF4rH2Pw DuFt/UYYUj8irhCjNp2shYOWOfM0BWTEhtK7ndyDwa1iLzjskHRS9BeoqHlqNxnHKnPD jK7w== X-Gm-Message-State: AOAM533vrK33MPGd/uGLo0YjtkR0xJ2o8NRiudzH99miJtTrRUf18p/z vieUtgxarf8kePgkeGm54fpMlwiFESqrYO1lmlbzRmQ4+27aPNcDBng/z58K+NNOnpbvayGg50i hzLSQ4DIxek8= X-Received: by 2002:a05:622a:180e:: with SMTP id t14mr295683qtc.375.1626301293256; Wed, 14 Jul 2021 15:21:33 -0700 (PDT) X-Google-Smtp-Source: ABdhPJxNGf06mrGz5E8vMivuFkNGw2IaWWFm6VxJucFi8ZYzevcV28jRdodL7+fGsf4xJxP2fA6pRQ== X-Received: by 2002:a05:622a:180e:: with SMTP id t14mr295658qtc.375.1626301293057; Wed, 14 Jul 2021 15:21:33 -0700 (PDT) Received: from localhost.localdomain (bras-base-toroon474qw-grc-65-184-144-111-238.dsl.bell.ca. [184.144.111.238]) by smtp.gmail.com with ESMTPSA id b25sm1625854qka.123.2021.07.14.15.21.31 (version=TLS1_3 cipher=TLS_AES_256_GCM_SHA384 bits=256/256); Wed, 14 Jul 2021 15:21:32 -0700 (PDT) From: Peter Xu To: linux-kernel@vger.kernel.org, linux-mm@kvack.org Cc: Jason Gunthorpe , peterx@redhat.com, Matthew Wilcox , Andrew Morton , Axel Rasmussen , Nadav Amit , Jerome Glisse , Mike Rapoport , Miaohe Lin , Hugh Dickins , Alistair Popple , Andrea Arcangeli , Mike Kravetz , "Kirill A . Shutemov" , David Hildenbrand Subject: [PATCH v4 07/26] mm: Drop first_index/last_index in zap_details Date: Wed, 14 Jul 2021 18:20:58 -0400 Message-Id: <20210714222117.47648-8-peterx@redhat.com> X-Mailer: git-send-email 2.31.1 In-Reply-To: <20210714222117.47648-1-peterx@redhat.com> References: <20210714222117.47648-1-peterx@redhat.com> MIME-Version: 1.0 X-Mimecast-Spam-Score: 0 X-Mimecast-Originator: redhat.com X-Rspamd-Server: rspam04 X-Rspamd-Queue-Id: 9002C3000106 X-Stat-Signature: owhiu8rnxea1fjbdgi3okgqhwxp6mqio Authentication-Results: imf09.hostedemail.com; dkim=pass header.d=redhat.com header.s=mimecast20190719 header.b=QJlew4Ax; dmarc=pass (policy=none) header.from=redhat.com; spf=none (imf09.hostedemail.com: domain of peterx@redhat.com has no SPF policy when checking 170.10.133.124) smtp.mailfrom=peterx@redhat.com X-HE-Tag: 1626301295-285699 X-Bogosity: Ham, tests=bogofilter, spamicity=0.000000, version=1.2.4 Sender: owner-linux-mm@kvack.org Precedence: bulk X-Loop: owner-majordomo@kvack.org List-ID: The first_index/last_index parameters in zap_details are actually only used in unmap_mapping_range_tree(). At the meantime, this function is only called by unmap_mapping_pages() once. Instead of passing these two variables through the whole stack of page zapping code, remove them from zap_details and let them simply be parameters of unmap_mapping_range_tree(), which is inlined. Reviewed-by: Alistair Popple Signed-off-by: Peter Xu --- include/linux/mm.h | 2 -- mm/memory.c | 29 ++++++++++++++++------------- 2 files changed, 16 insertions(+), 15 deletions(-) diff --git a/include/linux/mm.h b/include/linux/mm.h index 57453dba41b9..795b3cd643ca 100644 --- a/include/linux/mm.h +++ b/include/linux/mm.h @@ -1720,8 +1720,6 @@ extern void user_shm_unlock(size_t, struct ucounts *); */ struct zap_details { struct address_space *check_mapping; /* Check page->mapping if set */ - pgoff_t first_index; /* Lowest page->index to unmap */ - pgoff_t last_index; /* Highest page->index to unmap */ struct page *single_page; /* Locked page to be unmapped */ }; diff --git a/mm/memory.c b/mm/memory.c index ba8033ca6682..4c269d7b3d83 100644 --- a/mm/memory.c +++ b/mm/memory.c @@ -3322,20 +3322,20 @@ static void unmap_mapping_range_vma(struct vm_area_struct *vma, } static inline void unmap_mapping_range_tree(struct rb_root_cached *root, + pgoff_t first_index, + pgoff_t last_index, struct zap_details *details) { struct vm_area_struct *vma; pgoff_t vba, vea, zba, zea; - vma_interval_tree_foreach(vma, root, - details->first_index, details->last_index) { - + vma_interval_tree_foreach(vma, root, first_index, last_index) { vba = vma->vm_pgoff; vea = vba + vma_pages(vma) - 1; - zba = details->first_index; + zba = first_index; if (zba < vba) zba = vba; - zea = details->last_index; + zea = last_index; if (zea > vea) zea = vea; @@ -3361,18 +3361,21 @@ void unmap_mapping_page(struct page *page) { struct address_space *mapping = page->mapping; struct zap_details details = { }; + pgoff_t first_index, last_index; VM_BUG_ON(!PageLocked(page)); VM_BUG_ON(PageTail(page)); + first_index = page->index; + last_index = page->index + thp_nr_pages(page) - 1; + details.check_mapping = mapping; - details.first_index = page->index; - details.last_index = page->index + thp_nr_pages(page) - 1; details.single_page = page; i_mmap_lock_write(mapping); if (unlikely(!RB_EMPTY_ROOT(&mapping->i_mmap.rb_root))) - unmap_mapping_range_tree(&mapping->i_mmap, &details); + unmap_mapping_range_tree(&mapping->i_mmap, first_index, + last_index, &details); i_mmap_unlock_write(mapping); } @@ -3391,17 +3394,17 @@ void unmap_mapping_page(struct page *page) void unmap_mapping_pages(struct address_space *mapping, pgoff_t start, pgoff_t nr, bool even_cows) { + pgoff_t first_index = start, last_index = start + nr - 1; struct zap_details details = { }; details.check_mapping = even_cows ? NULL : mapping; - details.first_index = start; - details.last_index = start + nr - 1; - if (details.last_index < details.first_index) - details.last_index = ULONG_MAX; + if (last_index < first_index) + last_index = ULONG_MAX; i_mmap_lock_write(mapping); if (unlikely(!RB_EMPTY_ROOT(&mapping->i_mmap.rb_root))) - unmap_mapping_range_tree(&mapping->i_mmap, &details); + unmap_mapping_range_tree(&mapping->i_mmap, first_index, + last_index, &details); i_mmap_unlock_write(mapping); } From patchwork Wed Jul 14 22:20:59 2021 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Peter Xu X-Patchwork-Id: 12377953 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org X-Spam-Level: X-Spam-Status: No, score=-16.6 required=3.0 tests=BAYES_00,DKIM_INVALID, DKIM_SIGNED,HEADER_FROM_DIFFERENT_DOMAINS,INCLUDES_CR_TRAILER,INCLUDES_PATCH, MAILING_LIST_MULTI,SPF_HELO_NONE,SPF_PASS,USER_AGENT_GIT autolearn=ham autolearn_force=no version=3.4.0 Received: from mail.kernel.org (mail.kernel.org [198.145.29.99]) by smtp.lore.kernel.org (Postfix) with ESMTP id CAFBBC1B08C for ; Wed, 14 Jul 2021 22:21:41 +0000 (UTC) Received: from kanga.kvack.org (kanga.kvack.org [205.233.56.17]) by mail.kernel.org (Postfix) with ESMTP id 7E14F613C5 for ; Wed, 14 Jul 2021 22:21:41 +0000 (UTC) DMARC-Filter: OpenDMARC Filter v1.3.2 mail.kernel.org 7E14F613C5 Authentication-Results: mail.kernel.org; dmarc=fail (p=none dis=none) header.from=redhat.com Authentication-Results: mail.kernel.org; spf=pass smtp.mailfrom=owner-linux-mm@kvack.org Received: by kanga.kvack.org (Postfix) id 1DDA76B00B5; Wed, 14 Jul 2021 18:21:41 -0400 (EDT) Received: by kanga.kvack.org (Postfix, from userid 40) id 140448D0005; Wed, 14 Jul 2021 18:21:40 -0400 (EDT) X-Delivered-To: int-list-linux-mm@kvack.org Received: by kanga.kvack.org (Postfix, from userid 63042) id BD49F6B00B5; Wed, 14 Jul 2021 18:21:40 -0400 (EDT) X-Delivered-To: linux-mm@kvack.org Received: from forelay.hostedemail.com (smtprelay0066.hostedemail.com [216.40.44.66]) by kanga.kvack.org (Postfix) with ESMTP id 87CA06B00B5 for ; Wed, 14 Jul 2021 18:21:40 -0400 (EDT) Received: from smtpin04.hostedemail.com (10.5.19.251.rfc1918.com [10.5.19.251]) by forelay05.hostedemail.com (Postfix) with ESMTP id 67CAD182A627E for ; Wed, 14 Jul 2021 22:21:39 +0000 (UTC) X-FDA: 78362616318.04.5DA4863 Received: from us-smtp-delivery-124.mimecast.com (us-smtp-delivery-124.mimecast.com [170.10.133.124]) by imf11.hostedemail.com (Postfix) with ESMTP id 11093F00021B for ; Wed, 14 Jul 2021 22:21:38 +0000 (UTC) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=redhat.com; s=mimecast20190719; t=1626301298; h=from:from:reply-to:subject:subject:date:date:message-id:message-id: to:to:cc:cc:mime-version:mime-version:content-type:content-type: content-transfer-encoding:content-transfer-encoding: in-reply-to:in-reply-to:references:references; bh=EGgwjvpJa6wLNwtGExzq+9oKBigK+KwEv4SzKA2ZKkU=; b=e+Rrl8W2uhuWDsFRlNUPeTbNqFL3F/1L6BFfZjk+z8lEfAdh25KzwvwF3yave+enVr8+g5 9KXaVxVkmU80w8Lh1DCc0Xv1HL5Nv7Rtqi1ONwEjQuZBfLw0uGB01nc9014cazTE86hoFP RNHJ7eXAZqEV0oVrBQVuCsK1MXpcQWs= Received: from mail-qk1-f200.google.com (mail-qk1-f200.google.com [209.85.222.200]) (Using TLS) by relay.mimecast.com with ESMTP id us-mta-430-qV1Zwh8BObCctqDrt4LfgA-1; Wed, 14 Jul 2021 18:21:35 -0400 X-MC-Unique: qV1Zwh8BObCctqDrt4LfgA-1 Received: by mail-qk1-f200.google.com with SMTP id b6-20020a05620a1186b02903b429a7ee4bso2318851qkk.4 for ; Wed, 14 Jul 2021 15:21:35 -0700 (PDT) X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20161025; h=x-gm-message-state:from:to:cc:subject:date:message-id:in-reply-to :references:mime-version:content-transfer-encoding; bh=EGgwjvpJa6wLNwtGExzq+9oKBigK+KwEv4SzKA2ZKkU=; b=AJ1gGHF0+65MZBBNnQSBC0tbFmQf5HRWSMoy2QJYT91zX6oP/LjRM4+igl0Itu47YZ YSTIJBSHWy+DhCMDWhj8jWjD+9Z/GkuX7fFTwfRPF/rGcV33HlU105eSjeKCQLD0Qftx jZAns6Qng8kRo6Ro3iMxt/F2jCXkaJxfjdJTiST+HHkeOZPQCygieFSUewd15lYWYsWc wAW+OvZ9AtxB7GVkCreyvtLqyzcHgIMdTTDr07e9t3V5ZxeBv/NovN/MuoGy6bivYzex R/iVIZOHx+DXQLDkCAHmvHhHiZxDpJzQ0dS9E9uXEaMlfPNHRGFw3s2n6oqRvOQYWBjw S7JA== X-Gm-Message-State: AOAM530ARuqbAIpn3UsuNqB/R2tiHyDw1ijua5fFd6N6RGFzi1xCPnnh /PpdfdRv/v/pp96lb2J31HvzCQfTu5R9FEieXAhj2CGVMPaiCQmQmXNzoK/jUaPfhVZnoAVhumE mBkcij/ErP3s= X-Received: by 2002:a05:622a:606:: with SMTP id z6mr331745qta.274.1626301294937; Wed, 14 Jul 2021 15:21:34 -0700 (PDT) X-Google-Smtp-Source: ABdhPJznmkYMfpfk9MyPUyGhCbV3tJ5/PwvweoAj+ICAnQortiwZdCkTLIHXggzx/+vr1Cu0MJIdoA== X-Received: by 2002:a05:622a:606:: with SMTP id z6mr331721qta.274.1626301294698; Wed, 14 Jul 2021 15:21:34 -0700 (PDT) Received: from localhost.localdomain (bras-base-toroon474qw-grc-65-184-144-111-238.dsl.bell.ca. [184.144.111.238]) by smtp.gmail.com with ESMTPSA id b25sm1625854qka.123.2021.07.14.15.21.33 (version=TLS1_3 cipher=TLS_AES_256_GCM_SHA384 bits=256/256); Wed, 14 Jul 2021 15:21:34 -0700 (PDT) From: Peter Xu To: linux-kernel@vger.kernel.org, linux-mm@kvack.org Cc: Jason Gunthorpe , peterx@redhat.com, Matthew Wilcox , Andrew Morton , Axel Rasmussen , Nadav Amit , Jerome Glisse , Mike Rapoport , Miaohe Lin , Hugh Dickins , Alistair Popple , Andrea Arcangeli , Mike Kravetz , "Kirill A . Shutemov" , David Hildenbrand Subject: [PATCH v4 08/26] mm: Introduce zap_details.zap_flags Date: Wed, 14 Jul 2021 18:20:59 -0400 Message-Id: <20210714222117.47648-9-peterx@redhat.com> X-Mailer: git-send-email 2.31.1 In-Reply-To: <20210714222117.47648-1-peterx@redhat.com> References: <20210714222117.47648-1-peterx@redhat.com> MIME-Version: 1.0 X-Mimecast-Spam-Score: 0 X-Mimecast-Originator: redhat.com X-Rspamd-Server: rspam04 X-Rspamd-Queue-Id: 11093F00021B X-Stat-Signature: 7cus47jcw7usd9wfme4i8xrw7uskkaqu Authentication-Results: imf11.hostedemail.com; dkim=pass header.d=redhat.com header.s=mimecast20190719 header.b=e+Rrl8W2; dmarc=pass (policy=none) header.from=redhat.com; spf=none (imf11.hostedemail.com: domain of peterx@redhat.com has no SPF policy when checking 170.10.133.124) smtp.mailfrom=peterx@redhat.com X-HE-Tag: 1626301298-268777 X-Bogosity: Ham, tests=bogofilter, spamicity=0.000000, version=1.2.4 Sender: owner-linux-mm@kvack.org Precedence: bulk X-Loop: owner-majordomo@kvack.org List-ID: Instead of trying to introduce one variable for every new zap_details fields, let's introduce a flag so that it can start to encode true/false informations. Let's start to use this flag first to clean up the only check_mapping variable. Firstly, the name "check_mapping" implies this is a "boolean", but actually it stores the mapping inside, just in a way that it won't be set if we don't want to check the mapping. To make things clearer, introduce the 1st zap flag ZAP_FLAG_CHECK_MAPPING, so that we only check against the mapping if this bit set. At the same time, we can rename check_mapping into zap_mapping and set it always. Since at it, introduce another helper zap_check_mapping_skip() and use it in zap_pte_range() properly. Some old comments have been removed in zap_pte_range() because they're duplicated, and since now we're with ZAP_FLAG_CHECK_MAPPING flag, it'll be very easy to grep this information by simply grepping the flag. It'll also make life easier when we want to e.g. pass in zap_flags into the callers like unmap_mapping_pages() (instead of adding new booleans besides the even_cows parameter). Signed-off-by: Peter Xu --- include/linux/mm.h | 19 ++++++++++++++++++- mm/memory.c | 34 ++++++++++------------------------ 2 files changed, 28 insertions(+), 25 deletions(-) diff --git a/include/linux/mm.h b/include/linux/mm.h index 795b3cd643ca..26088174daab 100644 --- a/include/linux/mm.h +++ b/include/linux/mm.h @@ -1715,14 +1715,31 @@ static inline bool can_do_mlock(void) { return false; } extern int user_shm_lock(size_t, struct ucounts *); extern void user_shm_unlock(size_t, struct ucounts *); +/* Whether to check page->mapping when zapping */ +#define ZAP_FLAG_CHECK_MAPPING BIT(0) + /* * Parameter block passed down to zap_pte_range in exceptional cases. */ struct zap_details { - struct address_space *check_mapping; /* Check page->mapping if set */ + struct address_space *zap_mapping; struct page *single_page; /* Locked page to be unmapped */ + unsigned long zap_flags; }; +/* Return true if skip zapping this page, false otherwise */ +static inline bool +zap_skip_check_mapping(struct zap_details *details, struct page *page) +{ + if (!details || !page) + return false; + + if (!(details->zap_flags & ZAP_FLAG_CHECK_MAPPING)) + return false; + + return details->zap_mapping != page_rmapping(page); +} + struct page *vm_normal_page(struct vm_area_struct *vma, unsigned long addr, pte_t pte); struct page *vm_normal_page_pmd(struct vm_area_struct *vma, unsigned long addr, diff --git a/mm/memory.c b/mm/memory.c index 4c269d7b3d83..2a5a6650f069 100644 --- a/mm/memory.c +++ b/mm/memory.c @@ -1333,16 +1333,8 @@ static unsigned long zap_pte_range(struct mmu_gather *tlb, struct page *page; page = vm_normal_page(vma, addr, ptent); - if (unlikely(details) && page) { - /* - * unmap_shared_mapping_pages() wants to - * invalidate cache without truncating: - * unmap shared but keep private pages. - */ - if (details->check_mapping && - details->check_mapping != page_rmapping(page)) - continue; - } + if (unlikely(zap_skip_check_mapping(details, page))) + continue; ptent = ptep_get_and_clear_full(mm, addr, pte, tlb->fullmm); tlb_remove_tlb_entry(tlb, pte, addr); @@ -1375,17 +1367,8 @@ static unsigned long zap_pte_range(struct mmu_gather *tlb, is_device_exclusive_entry(entry)) { struct page *page = pfn_swap_entry_to_page(entry); - if (unlikely(details && details->check_mapping)) { - /* - * unmap_shared_mapping_pages() wants to - * invalidate cache without truncating: - * unmap shared but keep private pages. - */ - if (details->check_mapping != - page_rmapping(page)) - continue; - } - + if (unlikely(zap_skip_check_mapping(details, page))) + continue; pte_clear_not_present_full(mm, addr, pte, tlb->fullmm); rss[mm_counter(page)]--; @@ -3369,8 +3352,9 @@ void unmap_mapping_page(struct page *page) first_index = page->index; last_index = page->index + thp_nr_pages(page) - 1; - details.check_mapping = mapping; + details.zap_mapping = mapping; details.single_page = page; + details.zap_flags = ZAP_FLAG_CHECK_MAPPING; i_mmap_lock_write(mapping); if (unlikely(!RB_EMPTY_ROOT(&mapping->i_mmap.rb_root))) @@ -3395,9 +3379,11 @@ void unmap_mapping_pages(struct address_space *mapping, pgoff_t start, pgoff_t nr, bool even_cows) { pgoff_t first_index = start, last_index = start + nr - 1; - struct zap_details details = { }; + struct zap_details details = { .zap_mapping = mapping }; + + if (!even_cows) + details.zap_flags |= ZAP_FLAG_CHECK_MAPPING; - details.check_mapping = even_cows ? NULL : mapping; if (last_index < first_index) last_index = ULONG_MAX; From patchwork Wed Jul 14 22:21:00 2021 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Peter Xu X-Patchwork-Id: 12377955 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org X-Spam-Level: X-Spam-Status: No, score=-16.6 required=3.0 tests=BAYES_00,DKIM_INVALID, DKIM_SIGNED,HEADER_FROM_DIFFERENT_DOMAINS,INCLUDES_CR_TRAILER,INCLUDES_PATCH, MAILING_LIST_MULTI,SPF_HELO_NONE,SPF_PASS,URIBL_BLOCKED,USER_AGENT_GIT autolearn=ham autolearn_force=no version=3.4.0 Received: from mail.kernel.org (mail.kernel.org [198.145.29.99]) by smtp.lore.kernel.org (Postfix) with ESMTP id D5BCDC47E4B for ; Wed, 14 Jul 2021 22:21:43 +0000 (UTC) Received: from kanga.kvack.org (kanga.kvack.org [205.233.56.17]) by mail.kernel.org (Postfix) with ESMTP id 88DB3613C5 for ; Wed, 14 Jul 2021 22:21:43 +0000 (UTC) DMARC-Filter: OpenDMARC Filter v1.3.2 mail.kernel.org 88DB3613C5 Authentication-Results: mail.kernel.org; dmarc=fail (p=none dis=none) header.from=redhat.com Authentication-Results: mail.kernel.org; spf=pass smtp.mailfrom=owner-linux-mm@kvack.org Received: by kanga.kvack.org (Postfix) id 4CEDB6B00B6; Wed, 14 Jul 2021 18:21:41 -0400 (EDT) Received: by kanga.kvack.org (Postfix, from userid 40) id 425326B00B7; Wed, 14 Jul 2021 18:21:41 -0400 (EDT) X-Delivered-To: int-list-linux-mm@kvack.org Received: by kanga.kvack.org (Postfix, from userid 63042) id 057CF8D0003; Wed, 14 Jul 2021 18:21:40 -0400 (EDT) X-Delivered-To: linux-mm@kvack.org Received: from forelay.hostedemail.com (smtprelay0067.hostedemail.com [216.40.44.67]) by kanga.kvack.org (Postfix) with ESMTP id A454F6B00B6 for ; Wed, 14 Jul 2021 18:21:40 -0400 (EDT) Received: from smtpin30.hostedemail.com (10.5.19.251.rfc1918.com [10.5.19.251]) by forelay04.hostedemail.com (Postfix) with ESMTP id 979C4160AF for ; Wed, 14 Jul 2021 22:21:39 +0000 (UTC) X-FDA: 78362616318.30.7E0182F Received: from us-smtp-delivery-124.mimecast.com (us-smtp-delivery-124.mimecast.com [216.205.24.124]) by imf03.hostedemail.com (Postfix) with ESMTP id 4864930001A1 for ; Wed, 14 Jul 2021 22:21:39 +0000 (UTC) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=redhat.com; s=mimecast20190719; t=1626301298; h=from:from:reply-to:subject:subject:date:date:message-id:message-id: to:to:cc:cc:mime-version:mime-version:content-type:content-type: content-transfer-encoding:content-transfer-encoding: in-reply-to:in-reply-to:references:references; bh=DnKyw7yDWXQofeLoBv2gn06czfrlb3UiiX/R37cvmsE=; b=Yb3mKJqumFFmNQvUL253S9DMdmhKMn14rUGB83ooosN/meVWiidXJryrOgeq19UzUi+mCf h+LCImI5ETWbpLIY0cjnzbc6GRF1aEjj99P49gyKoVy7kjvV9KmXQkH9RoF2DhC4qZ0sXP tszdyWt356L0w4JNlwwwTiRymDL2g68= Received: from mail-qk1-f200.google.com (mail-qk1-f200.google.com [209.85.222.200]) (Using TLS) by relay.mimecast.com with ESMTP id us-mta-49-gzy_dZTRMjK4leikx6cj3Q-1; Wed, 14 Jul 2021 18:21:37 -0400 X-MC-Unique: gzy_dZTRMjK4leikx6cj3Q-1 Received: by mail-qk1-f200.google.com with SMTP id t144-20020a3746960000b02903ad9c5e94baso2290260qka.16 for ; Wed, 14 Jul 2021 15:21:37 -0700 (PDT) X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20161025; h=x-gm-message-state:from:to:cc:subject:date:message-id:in-reply-to :references:mime-version:content-transfer-encoding; bh=DnKyw7yDWXQofeLoBv2gn06czfrlb3UiiX/R37cvmsE=; b=Z8goZrdaWujO99oTjKaGSbbJoyzt8LbHxgJTJ/Lq0weacV+QwF3f0jJE8SLrxsF5gR yqGhHvGqzom5R2+ALIQ7EZfsf4L+4xt3mTX89Af0AGmVJoHYWZMrDpW379XhKehQJlym jWpDdLc19oM1sWx1W25lDSVe8Nv4voVySj9ZQJjF1EyeUj03K5xmMxlIe3cGnVf/OA/+ NKx8hhDV4FAZg2HUti/S64myr0coiWJWtytwvTJ1zjeK81zEt4Qhds4UOvKQv06WtRQb vxVewk669FqYCMHd6LTAeijn4ulCiypPkZhlSZU6qhVNgX/s6j6SYpvbelKIrauyExUE rq5w== X-Gm-Message-State: AOAM533zZugYnr6J709WBH9UWhSIJIEPUEyLY7xO4gDMzo0gRREwn75I DyaKiRrg6o27hDdoJrcsS3doLp0qRZyIWLVZX1FW1Xkz/d0yZsClmvNJ14rZLfMPuynuFw11Fc7 7hhKlw/bAF2w= X-Received: by 2002:ae9:e911:: with SMTP id x17mr189375qkf.371.1626301296700; Wed, 14 Jul 2021 15:21:36 -0700 (PDT) X-Google-Smtp-Source: ABdhPJy3A2Wvg6inICIbbeHBjb+ttmIrgQi5IfcXgb8Oye1uuRPIVOvrfqGI3ZbgjTSbv8g+Jmw2JA== X-Received: by 2002:ae9:e911:: with SMTP id x17mr189360qkf.371.1626301296471; Wed, 14 Jul 2021 15:21:36 -0700 (PDT) Received: from localhost.localdomain (bras-base-toroon474qw-grc-65-184-144-111-238.dsl.bell.ca. [184.144.111.238]) by smtp.gmail.com with ESMTPSA id b25sm1625854qka.123.2021.07.14.15.21.34 (version=TLS1_3 cipher=TLS_AES_256_GCM_SHA384 bits=256/256); Wed, 14 Jul 2021 15:21:35 -0700 (PDT) From: Peter Xu To: linux-kernel@vger.kernel.org, linux-mm@kvack.org Cc: Jason Gunthorpe , peterx@redhat.com, Matthew Wilcox , Andrew Morton , Axel Rasmussen , Nadav Amit , Jerome Glisse , Mike Rapoport , Miaohe Lin , Hugh Dickins , Alistair Popple , Andrea Arcangeli , Mike Kravetz , "Kirill A . Shutemov" , David Hildenbrand Subject: [PATCH v4 09/26] mm: Introduce ZAP_FLAG_SKIP_SWAP Date: Wed, 14 Jul 2021 18:21:00 -0400 Message-Id: <20210714222117.47648-10-peterx@redhat.com> X-Mailer: git-send-email 2.31.1 In-Reply-To: <20210714222117.47648-1-peterx@redhat.com> References: <20210714222117.47648-1-peterx@redhat.com> MIME-Version: 1.0 X-Mimecast-Spam-Score: 0 X-Mimecast-Originator: redhat.com X-Rspamd-Server: rspam04 X-Rspamd-Queue-Id: 4864930001A1 X-Stat-Signature: 94gczef5awrrjkdccqmbak95c8wdmx3a Authentication-Results: imf03.hostedemail.com; dkim=pass header.d=redhat.com header.s=mimecast20190719 header.b=Yb3mKJqu; dmarc=pass (policy=none) header.from=redhat.com; spf=none (imf03.hostedemail.com: domain of peterx@redhat.com has no SPF policy when checking 216.205.24.124) smtp.mailfrom=peterx@redhat.com X-HE-Tag: 1626301299-650341 X-Bogosity: Ham, tests=bogofilter, spamicity=0.000000, version=1.2.4 Sender: owner-linux-mm@kvack.org Precedence: bulk X-Loop: owner-majordomo@kvack.org List-ID: Firstly, the comment in zap_pte_range() is misleading because it checks against details rather than check_mappings, so it's against what the code did. Meanwhile, it's confusing too on not explaining why passing in the details pointer would mean to skip all swap entries. New user of zap_details could very possibly miss this fact if they don't read deep until zap_pte_range() because there's no comment at zap_details talking about it at all, so swap entries could be errornously skipped without being noticed. This partly reverts 3e8715fdc03e ("mm: drop zap_details::check_swap_entries"), but introduce ZAP_FLAG_SKIP_SWAP flag, which means the opposite of previous "details" parameter: the caller should explicitly set this to skip swap entries, otherwise swap entries will always be considered (which is still the major case here). Cc: Kirill A. Shutemov Reviewed-by: Alistair Popple Signed-off-by: Peter Xu --- include/linux/mm.h | 12 ++++++++++++ mm/memory.c | 8 +++++--- 2 files changed, 17 insertions(+), 3 deletions(-) diff --git a/include/linux/mm.h b/include/linux/mm.h index 26088174daab..62a75e4414e3 100644 --- a/include/linux/mm.h +++ b/include/linux/mm.h @@ -1717,6 +1717,8 @@ extern void user_shm_unlock(size_t, struct ucounts *); /* Whether to check page->mapping when zapping */ #define ZAP_FLAG_CHECK_MAPPING BIT(0) +/* Whether to skip zapping swap entries */ +#define ZAP_FLAG_SKIP_SWAP BIT(1) /* * Parameter block passed down to zap_pte_range in exceptional cases. @@ -1740,6 +1742,16 @@ zap_skip_check_mapping(struct zap_details *details, struct page *page) return details->zap_mapping != page_rmapping(page); } +/* Return true if skip swap entries, false otherwise */ +static inline bool +zap_skip_swap(struct zap_details *details) +{ + if (!details) + return false; + + return details->zap_flags & ZAP_FLAG_SKIP_SWAP; +} + struct page *vm_normal_page(struct vm_area_struct *vma, unsigned long addr, pte_t pte); struct page *vm_normal_page_pmd(struct vm_area_struct *vma, unsigned long addr, diff --git a/mm/memory.c b/mm/memory.c index 2a5a6650f069..d6b1adbf29e4 100644 --- a/mm/memory.c +++ b/mm/memory.c @@ -1379,8 +1379,7 @@ static unsigned long zap_pte_range(struct mmu_gather *tlb, continue; } - /* If details->check_mapping, we leave swap entries. */ - if (unlikely(details)) + if (unlikely(zap_skip_swap(details))) continue; if (!non_swap_entry(entry)) @@ -3379,7 +3378,10 @@ void unmap_mapping_pages(struct address_space *mapping, pgoff_t start, pgoff_t nr, bool even_cows) { pgoff_t first_index = start, last_index = start + nr - 1; - struct zap_details details = { .zap_mapping = mapping }; + struct zap_details details = { + .zap_mapping = mapping, + .zap_flags = ZAP_FLAG_SKIP_SWAP, + }; if (!even_cows) details.zap_flags |= ZAP_FLAG_CHECK_MAPPING; From patchwork Wed Jul 14 22:21:01 2021 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Peter Xu X-Patchwork-Id: 12377957 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org X-Spam-Level: X-Spam-Status: No, score=-16.6 required=3.0 tests=BAYES_00,DKIM_INVALID, DKIM_SIGNED,HEADER_FROM_DIFFERENT_DOMAINS,INCLUDES_CR_TRAILER,INCLUDES_PATCH, MAILING_LIST_MULTI,SPF_HELO_NONE,SPF_PASS,USER_AGENT_GIT autolearn=ham autolearn_force=no version=3.4.0 Received: from mail.kernel.org (mail.kernel.org [198.145.29.99]) by smtp.lore.kernel.org (Postfix) with ESMTP id D580BC1B08C for ; Wed, 14 Jul 2021 22:21:45 +0000 (UTC) Received: from kanga.kvack.org (kanga.kvack.org [205.233.56.17]) by mail.kernel.org (Postfix) with ESMTP id 87668613C5 for ; Wed, 14 Jul 2021 22:21:45 +0000 (UTC) DMARC-Filter: OpenDMARC Filter v1.3.2 mail.kernel.org 87668613C5 Authentication-Results: mail.kernel.org; dmarc=fail (p=none dis=none) header.from=redhat.com Authentication-Results: mail.kernel.org; spf=pass smtp.mailfrom=owner-linux-mm@kvack.org Received: by kanga.kvack.org (Postfix) id 2A15C6B00B9; Wed, 14 Jul 2021 18:21:44 -0400 (EDT) Received: by kanga.kvack.org (Postfix, from userid 40) id 251166B00BA; Wed, 14 Jul 2021 18:21:44 -0400 (EDT) X-Delivered-To: int-list-linux-mm@kvack.org Received: by kanga.kvack.org (Postfix, from userid 63042) id F21AD6B00BB; Wed, 14 Jul 2021 18:21:43 -0400 (EDT) X-Delivered-To: linux-mm@kvack.org Received: from forelay.hostedemail.com (smtprelay0006.hostedemail.com [216.40.44.6]) by kanga.kvack.org (Postfix) with ESMTP id BE6A66B00B9 for ; Wed, 14 Jul 2021 18:21:43 -0400 (EDT) Received: from smtpin27.hostedemail.com (10.5.19.251.rfc1918.com [10.5.19.251]) by forelay05.hostedemail.com (Postfix) with ESMTP id AA1FD184799E0 for ; Wed, 14 Jul 2021 22:21:42 +0000 (UTC) X-FDA: 78362616444.27.8A87658 Received: from us-smtp-delivery-124.mimecast.com (us-smtp-delivery-124.mimecast.com [216.205.24.124]) by imf24.hostedemail.com (Postfix) with ESMTP id 59826B000099 for ; Wed, 14 Jul 2021 22:21:42 +0000 (UTC) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=redhat.com; s=mimecast20190719; t=1626301301; h=from:from:reply-to:subject:subject:date:date:message-id:message-id: to:to:cc:cc:mime-version:mime-version:content-type:content-type: content-transfer-encoding:content-transfer-encoding: in-reply-to:in-reply-to:references:references; bh=H812mv/wTrZmXHNWLysXz5OhepU17RDOgYfB/9PUivE=; b=Q3N6v9pOpQBkb7fRmA6EJjCgZaCepUpTBxy0T2bd3/4qCRYtmCd7gHP13yy4GXypND8zrw VhOKaLlTpk6m1B+f8RWjWWMTiCKCYhP1t98hSnz68c7pbnWd8ffYO4CVL45WyiRGGo/+OS Q6DeWbvz0cTSmMxyUcgWFxuq/iBN1kk= Received: from mail-qt1-f198.google.com (mail-qt1-f198.google.com [209.85.160.198]) (Using TLS) by relay.mimecast.com with ESMTP id us-mta-491-zXkDZxgkMQm7HVT3BAJJeA-1; Wed, 14 Jul 2021 18:21:41 -0400 X-MC-Unique: zXkDZxgkMQm7HVT3BAJJeA-1 Received: by mail-qt1-f198.google.com with SMTP id 44-20020aed30af0000b029024e8ccfcd07so2820713qtf.11 for ; Wed, 14 Jul 2021 15:21:40 -0700 (PDT) X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20161025; h=x-gm-message-state:from:to:cc:subject:date:message-id:in-reply-to :references:mime-version:content-transfer-encoding; bh=H812mv/wTrZmXHNWLysXz5OhepU17RDOgYfB/9PUivE=; b=ech9kpSekYA0hxjS/U+qvTnzuLxDDGcJ6HUay7ckqE+My+kqYELSp9FlhcxDUvZAIc wPYg8d4AqS3iN6il4bBFI70OZGt+OL+oXVPkin+u9Bi57F0JZGIOMAtHca4zU9eELg3/ pNf/+gOdK4XqjX8ioDw01N/ZJstnVfI7Jg0oz8FiB17LIgDVFoPWqSUR/KJfZ/hzio2e c6M/2P/KH4d5lEluoA3EOLnmzcISH+k8Q5do9BOS20kpyVbmRKnVXtip0ucsu+PhrCD3 azkiZ2bV6lNgX5iN8gwM5m94ENt8G52K1FHRshPYT11pIv4oEtwnI64RnlU3tuaYyeRM 7ogQ== X-Gm-Message-State: AOAM532QtgzWiDTi59t6csoHoDX9NHkt8c7zZc1GV124iH0IMhi7KmxZ 8JQeyFZkBBIMAw/f6tF02k/UF9VhI0WRSWpKdc95IKmmo03IaNiSLqBldx3cjdmHGInRAT9iTfv 7fl8dkYpODCo= X-Received: by 2002:a05:620a:749:: with SMTP id i9mr197422qki.307.1626301298395; Wed, 14 Jul 2021 15:21:38 -0700 (PDT) X-Google-Smtp-Source: ABdhPJzHYlutPxUiBbHqP+vSZyJAcuxKco53LdGPPWF3poLQbmRvjHjziOODXUM7Cndyj1DnANybjA== X-Received: by 2002:a05:620a:749:: with SMTP id i9mr197388qki.307.1626301298138; Wed, 14 Jul 2021 15:21:38 -0700 (PDT) Received: from localhost.localdomain (bras-base-toroon474qw-grc-65-184-144-111-238.dsl.bell.ca. [184.144.111.238]) by smtp.gmail.com with ESMTPSA id b25sm1625854qka.123.2021.07.14.15.21.36 (version=TLS1_3 cipher=TLS_AES_256_GCM_SHA384 bits=256/256); Wed, 14 Jul 2021 15:21:37 -0700 (PDT) From: Peter Xu To: linux-kernel@vger.kernel.org, linux-mm@kvack.org Cc: Jason Gunthorpe , peterx@redhat.com, Matthew Wilcox , Andrew Morton , Axel Rasmussen , Nadav Amit , Jerome Glisse , Mike Rapoport , Miaohe Lin , Hugh Dickins , Alistair Popple , Andrea Arcangeli , Mike Kravetz , "Kirill A . Shutemov" , David Hildenbrand Subject: [PATCH v4 10/26] shmem/userfaultfd: Persist uffd-wp bit across zapping for file-backed Date: Wed, 14 Jul 2021 18:21:01 -0400 Message-Id: <20210714222117.47648-11-peterx@redhat.com> X-Mailer: git-send-email 2.31.1 In-Reply-To: <20210714222117.47648-1-peterx@redhat.com> References: <20210714222117.47648-1-peterx@redhat.com> MIME-Version: 1.0 X-Mimecast-Spam-Score: 0 X-Mimecast-Originator: redhat.com X-Rspamd-Server: rspam06 X-Rspamd-Queue-Id: 59826B000099 X-Stat-Signature: jm9obxq7kif9hecdw6nroatc8opbqniy Authentication-Results: imf24.hostedemail.com; dkim=pass header.d=redhat.com header.s=mimecast20190719 header.b=Q3N6v9pO; spf=none (imf24.hostedemail.com: domain of peterx@redhat.com has no SPF policy when checking 216.205.24.124) smtp.mailfrom=peterx@redhat.com; dmarc=pass (policy=none) header.from=redhat.com X-HE-Tag: 1626301302-918973 X-Bogosity: Ham, tests=bogofilter, spamicity=0.000000, version=1.2.4 Sender: owner-linux-mm@kvack.org Precedence: bulk X-Loop: owner-majordomo@kvack.org List-ID: File-backed memory is prone to being unmapped at any time. It means all information in the pte will be dropped, including the uffd-wp flag. Since the uffd-wp info cannot be stored in page cache or swap cache, persist this wr-protect information by installing the special uffd-wp marker pte when we're going to unmap a uffd wr-protected pte. When the pte is accessed again, we will know it's previously wr-protected by recognizing the special pte. Meanwhile add a new flag ZAP_FLAG_DROP_FILE_UFFD_WP when we don't want to persist such an information. For example, when destroying the whole vma, or punching a hole in a shmem file. For the latter, we can only drop the uffd-wp bit when holding the page lock. It means the unmap_mapping_range() in shmem_fallocate() still reuqires to zap without ZAP_FLAG_DROP_FILE_UFFD_WP because that's still racy with the page faults. Signed-off-by: Peter Xu --- include/linux/mm.h | 11 +++++++++ include/linux/mm_inline.h | 42 +++++++++++++++++++++++++++++++++ mm/memory.c | 49 +++++++++++++++++++++++++++++++++++++-- mm/rmap.c | 8 +++++++ 4 files changed, 108 insertions(+), 2 deletions(-) diff --git a/include/linux/mm.h b/include/linux/mm.h index 62a75e4414e3..8de230fc7b84 100644 --- a/include/linux/mm.h +++ b/include/linux/mm.h @@ -1719,6 +1719,8 @@ extern void user_shm_unlock(size_t, struct ucounts *); #define ZAP_FLAG_CHECK_MAPPING BIT(0) /* Whether to skip zapping swap entries */ #define ZAP_FLAG_SKIP_SWAP BIT(1) +/* Whether to completely drop uffd-wp entries for file-backed memory */ +#define ZAP_FLAG_DROP_FILE_UFFD_WP BIT(2) /* * Parameter block passed down to zap_pte_range in exceptional cases. @@ -1752,6 +1754,15 @@ zap_skip_swap(struct zap_details *details) return details->zap_flags & ZAP_FLAG_SKIP_SWAP; } +static inline bool +zap_drop_file_uffd_wp(struct zap_details *details) +{ + if (!details) + return false; + + return details->zap_flags & ZAP_FLAG_DROP_FILE_UFFD_WP; +} + struct page *vm_normal_page(struct vm_area_struct *vma, unsigned long addr, pte_t pte); struct page *vm_normal_page_pmd(struct vm_area_struct *vma, unsigned long addr, diff --git a/include/linux/mm_inline.h b/include/linux/mm_inline.h index 355ea1ee32bd..95556b4bfe7a 100644 --- a/include/linux/mm_inline.h +++ b/include/linux/mm_inline.h @@ -4,6 +4,8 @@ #include #include +#include +#include /** * page_is_file_lru - should the page be on a file LRU or anon LRU? @@ -104,4 +106,44 @@ static __always_inline void del_page_from_lru_list(struct page *page, update_lru_size(lruvec, page_lru(page), page_zonenum(page), -thp_nr_pages(page)); } + +/* + * If this pte is wr-protected by uffd-wp in any form, arm the special pte to + * replace a none pte. NOTE! This should only be called when *pte is already + * cleared so we will never accidentally replace something valuable. Meanwhile + * none pte also means we are not demoting the pte so tlb flushed is not needed. + * E.g., when pte cleared the caller should have taken care of the tlb flush. + * + * Must be called with pgtable lock held. + */ +static inline void +pte_install_uffd_wp_if_needed(struct vm_area_struct *vma, unsigned long addr, + pte_t *pte, pte_t pteval) +{ +#ifdef CONFIG_USERFAULTFD + bool arm_uffd_pte = false; + + /* The current status of the pte should be "cleared" before calling */ + WARN_ON_ONCE(!pte_none(*pte)); + + if (vma_is_anonymous(vma)) + return; + + /* A uffd-wp wr-protected normal pte */ + if (unlikely(pte_present(pteval) && pte_uffd_wp(pteval))) + arm_uffd_pte = true; + + /* + * A uffd-wp wr-protected swap pte. Note: this should even work for + * pte_swp_uffd_wp_special() too. + */ + if (unlikely(is_swap_pte(pteval) && pte_swp_uffd_wp(pteval))) + arm_uffd_pte = true; + + if (unlikely(arm_uffd_pte)) + set_pte_at(vma->vm_mm, addr, pte, + pte_swp_mkuffd_wp_special(vma)); +#endif +} + #endif diff --git a/mm/memory.c b/mm/memory.c index d6b1adbf29e4..223781f115e9 100644 --- a/mm/memory.c +++ b/mm/memory.c @@ -73,6 +73,7 @@ #include #include #include +#include #include @@ -1301,6 +1302,21 @@ copy_page_range(struct vm_area_struct *dst_vma, struct vm_area_struct *src_vma) return ret; } +/* + * This function makes sure that we'll replace the none pte with an uffd-wp + * swap special pte marker when necessary. Must be with the pgtable lock held. + */ +static inline void +zap_install_uffd_wp_if_needed(struct vm_area_struct *vma, + unsigned long addr, pte_t *pte, + struct zap_details *details, pte_t pteval) +{ + if (zap_drop_file_uffd_wp(details)) + return; + + pte_install_uffd_wp_if_needed(vma, addr, pte, pteval); +} + static unsigned long zap_pte_range(struct mmu_gather *tlb, struct vm_area_struct *vma, pmd_t *pmd, unsigned long addr, unsigned long end, @@ -1338,6 +1354,8 @@ static unsigned long zap_pte_range(struct mmu_gather *tlb, ptent = ptep_get_and_clear_full(mm, addr, pte, tlb->fullmm); tlb_remove_tlb_entry(tlb, pte, addr); + zap_install_uffd_wp_if_needed(vma, addr, pte, details, + ptent); if (unlikely(!page)) continue; @@ -1362,6 +1380,22 @@ static unsigned long zap_pte_range(struct mmu_gather *tlb, continue; } + /* + * If this is a special uffd-wp marker pte... Drop it only if + * enforced to do so. + */ + if (unlikely(is_swap_special_pte(ptent))) { + WARN_ON_ONCE(!pte_swp_uffd_wp_special(ptent)); + /* + * If this is a common unmap of ptes, keep this as is. + * Drop it only if this is a whole-vma destruction. + */ + if (zap_drop_file_uffd_wp(details)) + ptep_get_and_clear_full(mm, addr, pte, + tlb->fullmm); + continue; + } + entry = pte_to_swp_entry(ptent); if (is_device_private_entry(entry) || is_device_exclusive_entry(entry)) { @@ -1369,6 +1403,13 @@ static unsigned long zap_pte_range(struct mmu_gather *tlb, if (unlikely(zap_skip_check_mapping(details, page))) continue; + /* + * Both device private/exclusive mappings should only + * work with anonymous page so far, so we don't need to + * consider uffd-wp bit when zap. For more information, + * see zap_install_uffd_wp_if_needed(). + */ + WARN_ON_ONCE(!vma_is_anonymous(vma)); pte_clear_not_present_full(mm, addr, pte, tlb->fullmm); rss[mm_counter(page)]--; @@ -1393,6 +1434,7 @@ static unsigned long zap_pte_range(struct mmu_gather *tlb, if (unlikely(!free_swap_and_cache(entry))) print_bad_pte(vma, addr, ptent, NULL); pte_clear_not_present_full(mm, addr, pte, tlb->fullmm); + zap_install_uffd_wp_if_needed(vma, addr, pte, details, ptent); } while (pte++, addr += PAGE_SIZE, addr != end); add_mm_rss_vec(mm, rss); @@ -1603,12 +1645,15 @@ void unmap_vmas(struct mmu_gather *tlb, unsigned long end_addr) { struct mmu_notifier_range range; + struct zap_details details = { + .zap_flags = ZAP_FLAG_DROP_FILE_UFFD_WP, + }; mmu_notifier_range_init(&range, MMU_NOTIFY_UNMAP, 0, vma, vma->vm_mm, start_addr, end_addr); mmu_notifier_invalidate_range_start(&range); for ( ; vma && vma->vm_start < end_addr; vma = vma->vm_next) - unmap_single_vma(tlb, vma, start_addr, end_addr, NULL); + unmap_single_vma(tlb, vma, start_addr, end_addr, &details); mmu_notifier_invalidate_range_end(&range); } @@ -3353,7 +3398,7 @@ void unmap_mapping_page(struct page *page) details.zap_mapping = mapping; details.single_page = page; - details.zap_flags = ZAP_FLAG_CHECK_MAPPING; + details.zap_flags = ZAP_FLAG_CHECK_MAPPING | ZAP_FLAG_DROP_FILE_UFFD_WP; i_mmap_lock_write(mapping); if (unlikely(!RB_EMPTY_ROOT(&mapping->i_mmap.rb_root))) diff --git a/mm/rmap.c b/mm/rmap.c index 795f9d5f8386..92ba81567089 100644 --- a/mm/rmap.c +++ b/mm/rmap.c @@ -72,6 +72,7 @@ #include #include #include +#include #include @@ -1515,6 +1516,13 @@ static bool try_to_unmap_one(struct page *page, struct vm_area_struct *vma, pteval = ptep_clear_flush(vma, address, pvmw.pte); } + /* + * Now the pte is cleared. If this is uffd-wp armed pte, we + * may want to replace a none pte with a marker pte if it's + * file-backed, so we don't lose the tracking information. + */ + pte_install_uffd_wp_if_needed(vma, address, pvmw.pte, pteval); + /* Move the dirty bit to the page. Now the pte is gone. */ if (pte_dirty(pteval)) set_page_dirty(page); From patchwork Wed Jul 14 22:24:33 2021 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Peter Xu X-Patchwork-Id: 12377961 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org X-Spam-Level: X-Spam-Status: No, score=-16.6 required=3.0 tests=BAYES_00,DKIM_INVALID, DKIM_SIGNED,HEADER_FROM_DIFFERENT_DOMAINS,INCLUDES_CR_TRAILER,INCLUDES_PATCH, MAILING_LIST_MULTI,SPF_HELO_NONE,SPF_PASS,USER_AGENT_GIT autolearn=ham autolearn_force=no version=3.4.0 Received: from mail.kernel.org (mail.kernel.org [198.145.29.99]) by smtp.lore.kernel.org (Postfix) with ESMTP id 0DA77C12002 for ; Wed, 14 Jul 2021 22:24:42 +0000 (UTC) Received: from kanga.kvack.org (kanga.kvack.org [205.233.56.17]) by mail.kernel.org (Postfix) with ESMTP id A914B600D1 for ; Wed, 14 Jul 2021 22:24:41 +0000 (UTC) DMARC-Filter: OpenDMARC Filter v1.3.2 mail.kernel.org A914B600D1 Authentication-Results: mail.kernel.org; dmarc=fail (p=none dis=none) header.from=redhat.com Authentication-Results: mail.kernel.org; spf=pass smtp.mailfrom=owner-linux-mm@kvack.org Received: by kanga.kvack.org (Postfix) id EA8B76B00BD; Wed, 14 Jul 2021 18:24:41 -0400 (EDT) Received: by kanga.kvack.org (Postfix, from userid 40) id E59576B00BE; Wed, 14 Jul 2021 18:24:41 -0400 (EDT) X-Delivered-To: int-list-linux-mm@kvack.org Received: by kanga.kvack.org (Postfix, from userid 63042) id CAB688D0003; Wed, 14 Jul 2021 18:24:41 -0400 (EDT) X-Delivered-To: linux-mm@kvack.org Received: from forelay.hostedemail.com (smtprelay0121.hostedemail.com [216.40.44.121]) by kanga.kvack.org (Postfix) with ESMTP id A03E46B00BD for ; Wed, 14 Jul 2021 18:24:41 -0400 (EDT) Received: from smtpin18.hostedemail.com (10.5.19.251.rfc1918.com [10.5.19.251]) by forelay04.hostedemail.com (Postfix) with ESMTP id 9236B231B1 for ; Wed, 14 Jul 2021 22:24:40 +0000 (UTC) X-FDA: 78362623920.18.C64747E Received: from us-smtp-delivery-124.mimecast.com (us-smtp-delivery-124.mimecast.com [170.10.133.124]) by imf27.hostedemail.com (Postfix) with ESMTP id 288DC70000AF for ; Wed, 14 Jul 2021 22:24:40 +0000 (UTC) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=redhat.com; s=mimecast20190719; t=1626301479; h=from:from:reply-to:subject:subject:date:date:message-id:message-id: to:to:cc:cc:mime-version:mime-version:content-type:content-type: content-transfer-encoding:content-transfer-encoding: in-reply-to:in-reply-to:references:references; bh=8MXCY9EtHu8l00OeGEX9KSWIeSxJZ91wjW9WQpZLBho=; b=DW5639ykFP64RzWxxFTFvCAl84yuS6u+WHUV51ipWC1Pa2gQuPEnSduVyGf38D3aPhLoUk Mt6LXXAcjuzW/ZasBFtuMxRQV897qf1w2nil/Lw1NAXTqUG3fz8EXzTerTeUpvh+FMSRcl rmuwX9KLJH5xbfVAm5hsYXc1Ecg60qA= Received: from mail-qk1-f200.google.com (mail-qk1-f200.google.com [209.85.222.200]) (Using TLS) by relay.mimecast.com with ESMTP id us-mta-592-U0nImRbjPk-hV4-HG9n_Ow-1; Wed, 14 Jul 2021 18:24:38 -0400 X-MC-Unique: U0nImRbjPk-hV4-HG9n_Ow-1 Received: by mail-qk1-f200.google.com with SMTP id ay38-20020a05620a17a6b02903b7bed4e05eso2319258qkb.6 for ; Wed, 14 Jul 2021 15:24:38 -0700 (PDT) X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20161025; h=x-gm-message-state:from:to:cc:subject:date:message-id:in-reply-to :references:mime-version:content-transfer-encoding; bh=8MXCY9EtHu8l00OeGEX9KSWIeSxJZ91wjW9WQpZLBho=; b=EUiNbN2Wac8lZcfGhYokHWwqcriYmmiy78/8hIAFD63XrB4LSeGVahagjkT2kQXWUv AWHUQEgKWsut7vmAKMug6J3Et9LRJET+S9fb5nEojRbiLGcT0o6zt42bvmYbNy0k8ImA ZLVfxxM/NDULAs0idJC2VUHfEvOWYwCZBS5q2CK/9nFO6sjXFmBS4tDH7zUOIaCS1zCr 03AUIisdrOUpaVx9eVTAodkcgAvG+wheKjurb7Rjbz8hIQ7PVgkx0liFFXh0GuPhgF+1 MfvCiiTkwiOW1Co11X8X7WPR2WlAQtHuvVcUjyf2YD63Z9cvhTWInMIHaDvTIO0DlUML HtCA== X-Gm-Message-State: AOAM530oILjTTcpEtqeoBd1GqWIb1zXkL2chfghl7awLf86kJt9Ktsl+ Rikz8WcLtZm/jQqd8sXWGNxY/DyONo4PbPVYOFTJu4KiBA1sBrQ4QE96OhMH9RuTdctnSYib78m 2Fw1BbmiWm1V4S2jVeJLUrDz/UJr8FKN20Pg3MhVodkxbPJZ0Rc6j2+vB6ATO X-Received: by 2002:a0c:f244:: with SMTP id z4mr351840qvl.27.1626301477499; Wed, 14 Jul 2021 15:24:37 -0700 (PDT) X-Google-Smtp-Source: ABdhPJxXznl2Ndf5naG4U62173v8gGkYkHnaeWte5gx6dL3tnVR+2chcnWckUY/pRUVCWX6tOzr7Lw== X-Received: by 2002:a0c:f244:: with SMTP id z4mr351808qvl.27.1626301477140; Wed, 14 Jul 2021 15:24:37 -0700 (PDT) Received: from localhost.localdomain (bras-base-toroon474qw-grc-65-184-144-111-238.dsl.bell.ca. [184.144.111.238]) by smtp.gmail.com with ESMTPSA id n184sm1661512qkb.22.2021.07.14.15.24.35 (version=TLS1_3 cipher=TLS_AES_256_GCM_SHA384 bits=256/256); Wed, 14 Jul 2021 15:24:36 -0700 (PDT) From: Peter Xu To: linux-mm@kvack.org, linux-kernel@vger.kernel.org Cc: Andrew Morton , Mike Kravetz , Axel Rasmussen , Miaohe Lin , "Kirill A . Shutemov" , Hugh Dickins , Jason Gunthorpe , Alistair Popple , Matthew Wilcox , peterx@redhat.com, Jerome Glisse , Andrea Arcangeli , Mike Rapoport , Nadav Amit , David Hildenbrand Subject: [PATCH v4 11/26] shmem/userfaultfd: Allow wr-protect none pte for file-backed mem Date: Wed, 14 Jul 2021 18:24:33 -0400 Message-Id: <20210714222433.48637-1-peterx@redhat.com> X-Mailer: git-send-email 2.31.1 In-Reply-To: <20210714222117.47648-1-peterx@redhat.com> References: <20210714222117.47648-1-peterx@redhat.com> MIME-Version: 1.0 X-Mimecast-Spam-Score: 0 X-Mimecast-Originator: redhat.com Authentication-Results: imf27.hostedemail.com; dkim=pass header.d=redhat.com header.s=mimecast20190719 header.b=DW5639yk; spf=none (imf27.hostedemail.com: domain of peterx@redhat.com has no SPF policy when checking 170.10.133.124) smtp.mailfrom=peterx@redhat.com; dmarc=pass (policy=none) header.from=redhat.com X-Rspamd-Server: rspam05 X-Stat-Signature: pq8tu8o5jrcjed4u7w5askas769hpm99 X-Rspamd-Queue-Id: 288DC70000AF X-HE-Tag: 1626301480-355729 X-Bogosity: Ham, tests=bogofilter, spamicity=0.000000, version=1.2.4 Sender: owner-linux-mm@kvack.org Precedence: bulk X-Loop: owner-majordomo@kvack.org List-ID: File-backed memory differs from anonymous memory in that even if the pte is missing, the data could still resides either in the file or in page/swap cache. So when wr-protect a pte, we need to consider none ptes too. We do that by installing the uffd-wp special swap pte as a marker. So when there's a future write to the pte, the fault handler will go the special path to first fault-in the page as read-only, then report to userfaultfd server with the wr-protect message. On the other hand, when unprotecting a page, it's also possible that the pte got unmapped but replaced by the special uffd-wp marker. Then we'll need to be able to recover from a uffd-wp special swap pte into a none pte, so that the next access to the page will fault in correctly as usual when trigger the fault handler next time, rather than sending a uffd-wp message. Special care needs to be taken throughout the change_protection_range() process. Since now we allow user to wr-protect a none pte, we need to be able to pre-populate the page table entries if we see !anonymous && MM_CP_UFFD_WP requests, otherwise change_protection_range() will always skip when the pgtable entry does not exist. Note that this patch only covers the small pages (pte level) but not covering any of the transparent huge pages yet. But this will be a base for thps too. Signed-off-by: Peter Xu --- mm/mprotect.c | 48 ++++++++++++++++++++++++++++++++++++++++++++++++ 1 file changed, 48 insertions(+) diff --git a/mm/mprotect.c b/mm/mprotect.c index 4b743394afbe..8ec85b276975 100644 --- a/mm/mprotect.c +++ b/mm/mprotect.c @@ -29,6 +29,7 @@ #include #include #include +#include #include #include #include @@ -186,6 +187,32 @@ static unsigned long change_pte_range(struct vm_area_struct *vma, pmd_t *pmd, set_pte_at(vma->vm_mm, addr, pte, newpte); pages++; } + } else if (unlikely(is_swap_special_pte(oldpte))) { + if (uffd_wp_resolve && !vma_is_anonymous(vma) && + pte_swp_uffd_wp_special(oldpte)) { + /* + * This is uffd-wp special pte and we'd like to + * unprotect it. What we need to do is simply + * recover the pte into a none pte; the next + * page fault will fault in the page. + */ + pte_clear(vma->vm_mm, addr, pte); + pages++; + } + } else { + /* It must be an none page, or what else?.. */ + WARN_ON_ONCE(!pte_none(oldpte)); + if (unlikely(uffd_wp && !vma_is_anonymous(vma))) { + /* + * For file-backed mem, we need to be able to + * wr-protect even for a none pte! Because + * even if the pte is null, the page/swap cache + * could exist. + */ + set_pte_at(vma->vm_mm, addr, pte, + pte_swp_mkuffd_wp_special(vma)); + pages++; + } } } while (pte++, addr += PAGE_SIZE, addr != end); arch_leave_lazy_mmu_mode(); @@ -219,6 +246,25 @@ static inline int pmd_none_or_clear_bad_unless_trans_huge(pmd_t *pmd) return 0; } +/* + * File-backed vma allows uffd wr-protect upon none ptes, because even if pte + * is missing, page/swap cache could exist. When that happens, the wr-protect + * information will be stored in the page table entries with the marker (e.g., + * PTE_SWP_UFFD_WP_SPECIAL). Prepare for that by always populating the page + * tables to pte level, so that we'll install the markers in change_pte_range() + * where necessary. + * + * Note that we only need to do this in pmd level, because if pmd does not + * exist, it means the whole range covered by the pmd entry (of a pud) does not + * contain any valid data but all zeros. Then nothing to wr-protect. + */ +#define change_protection_prepare(vma, pmd, addr, cp_flags) \ + do { \ + if (unlikely((cp_flags & MM_CP_UFFD_WP) && pmd_none(*pmd) && \ + !vma_is_anonymous(vma))) \ + WARN_ON_ONCE(pte_alloc(vma->vm_mm, pmd)); \ + } while (0) + static inline unsigned long change_pmd_range(struct vm_area_struct *vma, pud_t *pud, unsigned long addr, unsigned long end, pgprot_t newprot, unsigned long cp_flags) @@ -237,6 +283,8 @@ static inline unsigned long change_pmd_range(struct vm_area_struct *vma, next = pmd_addr_end(addr, end); + change_protection_prepare(vma, pmd, addr, cp_flags); + /* * Automatic NUMA balancing walks the tables with mmap_lock * held for read. It's possible a parallel update to occur From patchwork Wed Jul 14 22:24:37 2021 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Peter Xu X-Patchwork-Id: 12377963 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org X-Spam-Level: X-Spam-Status: No, score=-16.6 required=3.0 tests=BAYES_00,DKIM_INVALID, DKIM_SIGNED,HEADER_FROM_DIFFERENT_DOMAINS,INCLUDES_CR_TRAILER,INCLUDES_PATCH, MAILING_LIST_MULTI,SPF_HELO_NONE,SPF_PASS,USER_AGENT_GIT autolearn=ham autolearn_force=no version=3.4.0 Received: from mail.kernel.org (mail.kernel.org [198.145.29.99]) by smtp.lore.kernel.org (Postfix) with ESMTP id 92B9BC12002 for ; Wed, 14 Jul 2021 22:24:45 +0000 (UTC) Received: from kanga.kvack.org (kanga.kvack.org [205.233.56.17]) by mail.kernel.org (Postfix) with ESMTP id 3CB9E613C9 for ; Wed, 14 Jul 2021 22:24:45 +0000 (UTC) DMARC-Filter: OpenDMARC Filter v1.3.2 mail.kernel.org 3CB9E613C9 Authentication-Results: mail.kernel.org; dmarc=fail (p=none dis=none) header.from=redhat.com Authentication-Results: mail.kernel.org; spf=pass smtp.mailfrom=owner-linux-mm@kvack.org Received: by kanga.kvack.org (Postfix) id 903EE6B00BF; Wed, 14 Jul 2021 18:24:45 -0400 (EDT) Received: by kanga.kvack.org (Postfix, from userid 40) id 8DA696B00C0; Wed, 14 Jul 2021 18:24:45 -0400 (EDT) X-Delivered-To: int-list-linux-mm@kvack.org Received: by kanga.kvack.org (Postfix, from userid 63042) id 77AFC6B00C1; Wed, 14 Jul 2021 18:24:45 -0400 (EDT) X-Delivered-To: linux-mm@kvack.org Received: from forelay.hostedemail.com (smtprelay0039.hostedemail.com [216.40.44.39]) by kanga.kvack.org (Postfix) with ESMTP id 4DCB86B00BF for ; Wed, 14 Jul 2021 18:24:45 -0400 (EDT) Received: from smtpin05.hostedemail.com (10.5.19.251.rfc1918.com [10.5.19.251]) by forelay05.hostedemail.com (Postfix) with ESMTP id 3A94E1837DCAE for ; Wed, 14 Jul 2021 22:24:44 +0000 (UTC) X-FDA: 78362624088.05.189866D Received: from us-smtp-delivery-124.mimecast.com (us-smtp-delivery-124.mimecast.com [170.10.133.124]) by imf14.hostedemail.com (Postfix) with ESMTP id D1AEC6001A90 for ; Wed, 14 Jul 2021 22:24:43 +0000 (UTC) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=redhat.com; s=mimecast20190719; t=1626301483; h=from:from:reply-to:subject:subject:date:date:message-id:message-id: to:to:cc:cc:mime-version:mime-version:content-type:content-type: content-transfer-encoding:content-transfer-encoding: in-reply-to:in-reply-to:references:references; bh=ltWxXgdsERQEx3E6ySA6em4pDoRaSyrvcOj0w7z20l8=; b=RwHKUGCGdjuNICP9+Tf+23LWsdsfZtref41n6psOCiEI9KaEb8Ke9KZK4cz6y08IYx/sXU cx2mFs58uBR3YzImA3Z1Kzb8qaFmDj2hv7JUUJJYj9YNL3xnr9DOgssDJfKZO/WEZAR0y9 tsAtJFvCHPlrs518gbkGaqvRWhQ9Hbw= Received: from mail-qk1-f198.google.com (mail-qk1-f198.google.com [209.85.222.198]) (Using TLS) by relay.mimecast.com with ESMTP id us-mta-172-I_xluNMkMsqURc7jm9Le-Q-1; Wed, 14 Jul 2021 18:24:42 -0400 X-MC-Unique: I_xluNMkMsqURc7jm9Le-Q-1 Received: by mail-qk1-f198.google.com with SMTP id o14-20020a05620a0d4eb02903a5eee61155so2315122qkl.9 for ; Wed, 14 Jul 2021 15:24:42 -0700 (PDT) X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20161025; h=x-gm-message-state:from:to:cc:subject:date:message-id:in-reply-to :references:mime-version:content-transfer-encoding; bh=ltWxXgdsERQEx3E6ySA6em4pDoRaSyrvcOj0w7z20l8=; b=n4f1a/CyhxHsHEg0TVcCNEcJMVvy7STTHFV33l89g779rYhhylbKI5WiAFPa9iCDgz +stKrly7sAP3kDY4drPLHqpmhuxMgrO7Od2OlnOpXOgXABLMatm5oAvmEyltaC8nEsew m4WaG7bF6S2VFHmRJ9uyvvKKusR8DO3gsHEfT1p1gbASF4OBeW6z50HHqdFC7MPuPQ23 mS6D8qkiBqP0B3wPIr/oTfcxwU+jNT2C5GdiMWrY9FVvxjQwSmQi7BoWouNuJ+0JHER9 D0NmbeyZQr04N4roytj2NFh+nDx4acalxphQIvp+1o/3xtb8oBYGeY+e0/w5qKOxOfp9 ICSQ== X-Gm-Message-State: AOAM532R155iJzQGYw22lQG6LJkcFxppmNgY/4HsIJxaOTbdUP6e6Sbi T94SjgNWg0JZxFzX3tDs39SKTRnnEqoZVqAeDjTjTtiBNTpkAGjKRsAws/HCgFRyGPdTeYJNUy3 R78yRuX+C7WuE9vocVs9Zkw8Ip1SUH7EAZ+G6iT8T5MgnOZX6ozHdJ45K7YhS X-Received: by 2002:ac8:44c2:: with SMTP id b2mr348930qto.216.1626301481624; Wed, 14 Jul 2021 15:24:41 -0700 (PDT) X-Google-Smtp-Source: ABdhPJxltkXhESK8mTiLGdx//VedRH0qopTtyIpRgcuMJZv4e/oNyE6Nw/k7eAZhcln38G+Gz1hD3g== X-Received: by 2002:ac8:44c2:: with SMTP id b2mr348896qto.216.1626301481364; Wed, 14 Jul 2021 15:24:41 -0700 (PDT) Received: from localhost.localdomain (bras-base-toroon474qw-grc-65-184-144-111-238.dsl.bell.ca. [184.144.111.238]) by smtp.gmail.com with ESMTPSA id t125sm1645040qkf.41.2021.07.14.15.24.39 (version=TLS1_3 cipher=TLS_AES_256_GCM_SHA384 bits=256/256); Wed, 14 Jul 2021 15:24:40 -0700 (PDT) From: Peter Xu To: linux-mm@kvack.org, linux-kernel@vger.kernel.org Cc: Andrew Morton , Mike Kravetz , Axel Rasmussen , Miaohe Lin , "Kirill A . Shutemov" , Hugh Dickins , Jason Gunthorpe , Alistair Popple , Matthew Wilcox , peterx@redhat.com, Jerome Glisse , Andrea Arcangeli , Mike Rapoport , Nadav Amit , David Hildenbrand Subject: [PATCH v4 12/26] shmem/userfaultfd: Allows file-back mem to be uffd wr-protected on thps Date: Wed, 14 Jul 2021 18:24:37 -0400 Message-Id: <20210714222437.48687-1-peterx@redhat.com> X-Mailer: git-send-email 2.31.1 In-Reply-To: <20210714222117.47648-1-peterx@redhat.com> References: <20210714222117.47648-1-peterx@redhat.com> MIME-Version: 1.0 X-Mimecast-Spam-Score: 0 X-Mimecast-Originator: redhat.com Authentication-Results: imf14.hostedemail.com; dkim=pass header.d=redhat.com header.s=mimecast20190719 header.b=RwHKUGCG; spf=none (imf14.hostedemail.com: domain of peterx@redhat.com has no SPF policy when checking 170.10.133.124) smtp.mailfrom=peterx@redhat.com; dmarc=pass (policy=none) header.from=redhat.com X-Rspamd-Server: rspam02 X-Stat-Signature: e53hfakxpyf8k354fdexn8ecu5t3fj7j X-Rspamd-Queue-Id: D1AEC6001A90 X-HE-Tag: 1626301483-160719 X-Bogosity: Ham, tests=bogofilter, spamicity=0.000000, version=1.2.4 Sender: owner-linux-mm@kvack.org Precedence: bulk X-Loop: owner-majordomo@kvack.org List-ID: We don't have "huge" version of PTE_SWP_UFFD_WP_SPECIAL, instead when necessary we split the thp if the huge page is uffd wr-protected previously. However split the thp is not enough, because file-backed thp is handled totally differently comparing to anonymous thps - rather than doing a real split, the thp pmd will simply got dropped in __split_huge_pmd_locked(). That is definitely not enough if e.g. when there is a thp covers range [0, 2M) but we want to wr-protect small page resides in [4K, 8K) range, because after __split_huge_pmd() returns, there will be a none pmd. Here we leverage the previously introduced change_protection_prepare() macro so that we'll populate the pmd with a pgtable page. Then change_pte_range() will do all the rest for us, e.g., install the uffd-wp swap special pte marker at any pte that we'd like to wr-protect, under the protection of pgtable lock. Signed-off-by: Peter Xu --- mm/mprotect.c | 10 +++++++++- 1 file changed, 9 insertions(+), 1 deletion(-) diff --git a/mm/mprotect.c b/mm/mprotect.c index 8ec85b276975..3fcb87b59696 100644 --- a/mm/mprotect.c +++ b/mm/mprotect.c @@ -306,8 +306,16 @@ static inline unsigned long change_pmd_range(struct vm_area_struct *vma, } if (is_swap_pmd(*pmd) || pmd_trans_huge(*pmd) || pmd_devmap(*pmd)) { - if (next - addr != HPAGE_PMD_SIZE) { + if (next - addr != HPAGE_PMD_SIZE || + /* Uffd wr-protecting a file-backed memory range */ + unlikely(!vma_is_anonymous(vma) && + (cp_flags & MM_CP_UFFD_WP))) { __split_huge_pmd(vma, pmd, addr, false, NULL); + /* + * For file-backed, the pmd could have been + * gone; still provide a pte pgtable if needed. + */ + change_protection_prepare(vma, pmd, addr, cp_flags); } else { int nr_ptes = change_huge_pmd(vma, pmd, addr, newprot, cp_flags); From patchwork Wed Jul 14 22:24:41 2021 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Peter Xu X-Patchwork-Id: 12377965 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org X-Spam-Level: X-Spam-Status: No, score=-16.6 required=3.0 tests=BAYES_00,DKIM_INVALID, DKIM_SIGNED,HEADER_FROM_DIFFERENT_DOMAINS,INCLUDES_CR_TRAILER,INCLUDES_PATCH, MAILING_LIST_MULTI,SPF_HELO_NONE,SPF_PASS,USER_AGENT_GIT autolearn=ham autolearn_force=no version=3.4.0 Received: from mail.kernel.org (mail.kernel.org [198.145.29.99]) by smtp.lore.kernel.org (Postfix) with ESMTP id F28CEC12002 for ; Wed, 14 Jul 2021 22:24:49 +0000 (UTC) Received: from kanga.kvack.org (kanga.kvack.org [205.233.56.17]) by mail.kernel.org (Postfix) with ESMTP id A89AB600D1 for ; Wed, 14 Jul 2021 22:24:49 +0000 (UTC) DMARC-Filter: OpenDMARC Filter v1.3.2 mail.kernel.org A89AB600D1 Authentication-Results: mail.kernel.org; dmarc=fail (p=none dis=none) header.from=redhat.com Authentication-Results: mail.kernel.org; spf=pass smtp.mailfrom=owner-linux-mm@kvack.org Received: by kanga.kvack.org (Postfix) id 052936B00C1; Wed, 14 Jul 2021 18:24:50 -0400 (EDT) Received: by kanga.kvack.org (Postfix, from userid 40) id 029236B00C2; Wed, 14 Jul 2021 18:24:49 -0400 (EDT) X-Delivered-To: int-list-linux-mm@kvack.org Received: by kanga.kvack.org (Postfix, from userid 63042) id DE5176B00C3; Wed, 14 Jul 2021 18:24:49 -0400 (EDT) X-Delivered-To: linux-mm@kvack.org Received: from forelay.hostedemail.com (smtprelay0175.hostedemail.com [216.40.44.175]) by kanga.kvack.org (Postfix) with ESMTP id BC7D66B00C1 for ; Wed, 14 Jul 2021 18:24:49 -0400 (EDT) Received: from smtpin31.hostedemail.com (10.5.19.251.rfc1918.com [10.5.19.251]) by forelay05.hostedemail.com (Postfix) with ESMTP id 99CAF1838DC16 for ; Wed, 14 Jul 2021 22:24:48 +0000 (UTC) X-FDA: 78362624256.31.98C347D Received: from us-smtp-delivery-124.mimecast.com (us-smtp-delivery-124.mimecast.com [216.205.24.124]) by imf28.hostedemail.com (Postfix) with ESMTP id 473CC90000A4 for ; Wed, 14 Jul 2021 22:24:48 +0000 (UTC) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=redhat.com; s=mimecast20190719; t=1626301487; h=from:from:reply-to:subject:subject:date:date:message-id:message-id: to:to:cc:cc:mime-version:mime-version:content-type:content-type: content-transfer-encoding:content-transfer-encoding: in-reply-to:in-reply-to:references:references; bh=LFcYHlyDKbLlgbIUnPtt3LhncHUQTe/w3qvUe3g6dgM=; b=I+2dbi5Q9ZGTcPdU4QQ0CS5fRtSOjWg8752IMvbHrR02jPVJQ30ZNt5UnL7e3xrNiqtp0u HQvXrTlLg1t+CMn9TGrbL2qWSeafG6bvEoVqtwFh5Q53VWvkbqMyawkEClfcCvTDVZa5q4 GoUD0ScHuupxXnZGo6WBTAMqoT56qps= Received: from mail-qv1-f69.google.com (mail-qv1-f69.google.com [209.85.219.69]) (Using TLS) by relay.mimecast.com with ESMTP id us-mta-227-cBXvo-70NlGgA0IP1PkVOA-1; Wed, 14 Jul 2021 18:24:46 -0400 X-MC-Unique: cBXvo-70NlGgA0IP1PkVOA-1 Received: by mail-qv1-f69.google.com with SMTP id t15-20020a056214036fb02902dc72b5283dso2665751qvu.23 for ; Wed, 14 Jul 2021 15:24:46 -0700 (PDT) X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20161025; h=x-gm-message-state:from:to:cc:subject:date:message-id:in-reply-to :references:mime-version:content-transfer-encoding; bh=LFcYHlyDKbLlgbIUnPtt3LhncHUQTe/w3qvUe3g6dgM=; b=CaLxOetDEh8CSYjl1JDO2lxBzvZZdOxdzlaasQk4RKAZc4oviw8enpiPWR5isfE8Df dzZODE0bQIrMNgYvYVfTrhIcp2mTpr/Ebkop71hr6+/HcjOeCHHZnqHxknOCXvBASxm5 h2UfrG2I0dVM7aOsqKRwiyPJIWBsDFBg4LPYmCYTxWWUoWFpgBf1tO/JUHggrR41qGI/ EWfsFPa6+f2/GEV8h+FlNmn9ZTYqMXhxUevxziJ04jCqSuF9nqrSNazpP9r73etVMev4 L0Tp1u8Jw4KjHdGC5B98DW2qBVkOyfRxX0EEY5Z0LYj8ApQDAi/MGJcXkJvHJ6JOLRHz YcFg== X-Gm-Message-State: AOAM531xJnxC78nXsIOORO4O8nKBsL7jPjAzWrpRXfDcXiA/coJ8fHF4 jfkNGj2AyHkN1taf2IL/m1NCr8BN7sifO7WQeyE1JbkhyilJZ0fAjBZ0O49k9301uFSt/osQw+Y MPb9lINsEiVU6MEujTEmNO2jujEpG7uNJLm9edVKpXJpAZ+vItxk8cVKlcVii X-Received: by 2002:ac8:5853:: with SMTP id h19mr363312qth.66.1626301486009; Wed, 14 Jul 2021 15:24:46 -0700 (PDT) X-Google-Smtp-Source: ABdhPJzBfKAWFYo/mOF5P/V0APrbl+7RO7yIBtyeG/gjJiUvpZs7s3HU04/TQ0OhCIH6sjSAsXdp8Q== X-Received: by 2002:ac8:5853:: with SMTP id h19mr363274qth.66.1626301485652; Wed, 14 Jul 2021 15:24:45 -0700 (PDT) Received: from localhost.localdomain (bras-base-toroon474qw-grc-65-184-144-111-238.dsl.bell.ca. [184.144.111.238]) by smtp.gmail.com with ESMTPSA id t125sm1645141qkf.41.2021.07.14.15.24.44 (version=TLS1_3 cipher=TLS_AES_256_GCM_SHA384 bits=256/256); Wed, 14 Jul 2021 15:24:45 -0700 (PDT) From: Peter Xu To: linux-mm@kvack.org, linux-kernel@vger.kernel.org Cc: Andrew Morton , Mike Kravetz , Axel Rasmussen , Miaohe Lin , "Kirill A . Shutemov" , Hugh Dickins , Jason Gunthorpe , Alistair Popple , Matthew Wilcox , peterx@redhat.com, Jerome Glisse , Andrea Arcangeli , Mike Rapoport , Nadav Amit , David Hildenbrand Subject: [PATCH v4 13/26] shmem/userfaultfd: Handle the left-overed special swap ptes Date: Wed, 14 Jul 2021 18:24:41 -0400 Message-Id: <20210714222441.48737-1-peterx@redhat.com> X-Mailer: git-send-email 2.31.1 In-Reply-To: <20210714222117.47648-1-peterx@redhat.com> References: <20210714222117.47648-1-peterx@redhat.com> MIME-Version: 1.0 X-Mimecast-Spam-Score: 0 X-Mimecast-Originator: redhat.com X-Rspamd-Server: rspam06 X-Rspamd-Queue-Id: 473CC90000A4 X-Stat-Signature: tsetkj75rftkrot63easr58n5bm7um63 Authentication-Results: imf28.hostedemail.com; dkim=pass header.d=redhat.com header.s=mimecast20190719 header.b=I+2dbi5Q; spf=none (imf28.hostedemail.com: domain of peterx@redhat.com has no SPF policy when checking 216.205.24.124) smtp.mailfrom=peterx@redhat.com; dmarc=pass (policy=none) header.from=redhat.com X-HE-Tag: 1626301488-939314 X-Bogosity: Ham, tests=bogofilter, spamicity=0.000000, version=1.2.4 Sender: owner-linux-mm@kvack.org Precedence: bulk X-Loop: owner-majordomo@kvack.org List-ID: Note that the special uffd-wp swap pte can be left over even if the page under the pte got evicted. Normally when evict a page, we will unmap the ptes by walking through the reverse mapping. However we never tracked such information for the special swap ptes because they're not real mappings but just markers. So we need to take care of that when we see a marker but when it's actually meaningless (the page behind it got evicted). We have already taken care of that in e.g. alloc_set_pte() where we'll treat the special swap pte as pte_none() when necessary. However we need to also teach userfaultfd itself on either UFFDIO_COPY or handling page faults, so that everything will still work as expected. Signed-off-by: Peter Xu --- fs/userfaultfd.c | 15 +++++++++++++++ mm/userfaultfd.c | 13 ++++++++++++- 2 files changed, 27 insertions(+), 1 deletion(-) diff --git a/fs/userfaultfd.c b/fs/userfaultfd.c index f6e0f0c0d0e5..e1c1cbc7bcc8 100644 --- a/fs/userfaultfd.c +++ b/fs/userfaultfd.c @@ -329,6 +329,21 @@ static inline bool userfaultfd_must_wait(struct userfaultfd_ctx *ctx, */ if (pte_none(*pte)) ret = true; + /* + * We also treat the swap special uffd-wp pte as the pte_none() here. + * This should in most cases be a missing event, as we never handle + * wr-protect upon a special uffd-wp swap pte - it should first be + * converted into a normal read request before handling wp. It just + * means the page/swap cache that backing this pte is gone, so this + * special pte is leftover. + * + * We can't simply replace it with a none pte because we're not with + * the pgtable lock here. Instead of taking it and clearing the pte, + * the easy way is to let UFFDIO_COPY understand this pte too when + * trying to install a new page onto it. + */ + if (pte_swp_uffd_wp_special(*pte)) + ret = true; if (!pte_write(*pte) && (reason & VM_UFFD_WP)) ret = true; pte_unmap(pte); diff --git a/mm/userfaultfd.c b/mm/userfaultfd.c index 2a9c9e6eb876..0c7212dfb95d 100644 --- a/mm/userfaultfd.c +++ b/mm/userfaultfd.c @@ -100,7 +100,18 @@ int mfill_atomic_install_pte(struct mm_struct *dst_mm, pmd_t *dst_pmd, } ret = -EEXIST; - if (!pte_none(*dst_pte)) + /* + * Besides the none pte, we also allow UFFDIO_COPY to install a pte + * onto the uffd-wp swap special pte, because that pte should be the + * same as a pte_none() just in that it contains wr-protect information + * (which could only be dropped when unmap the memory). + * + * It's safe to drop that marker because we know this is part of a + * MISSING fault, and the caller is very clear about this page missing + * rather than wr-protected. Then we're sure the wr-protect bit is + * just a leftover so it's useless already and is the same as none pte. + */ + if (!pte_none(*dst_pte) && !pte_swp_uffd_wp_special(*dst_pte)) goto out_unlock; if (page_in_cache) From patchwork Wed Jul 14 22:24:46 2021 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Peter Xu X-Patchwork-Id: 12377967 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org X-Spam-Level: X-Spam-Status: No, score=-16.6 required=3.0 tests=BAYES_00,DKIM_INVALID, DKIM_SIGNED,HEADER_FROM_DIFFERENT_DOMAINS,INCLUDES_CR_TRAILER,INCLUDES_PATCH, MAILING_LIST_MULTI,SPF_HELO_NONE,SPF_PASS,USER_AGENT_GIT autolearn=ham autolearn_force=no version=3.4.0 Received: from mail.kernel.org (mail.kernel.org [198.145.29.99]) by smtp.lore.kernel.org (Postfix) with ESMTP id D6B68C1B08C for ; Wed, 14 Jul 2021 22:24:55 +0000 (UTC) Received: from kanga.kvack.org (kanga.kvack.org [205.233.56.17]) by mail.kernel.org (Postfix) with ESMTP id 8D87B613C9 for ; Wed, 14 Jul 2021 22:24:55 +0000 (UTC) DMARC-Filter: OpenDMARC Filter v1.3.2 mail.kernel.org 8D87B613C9 Authentication-Results: mail.kernel.org; dmarc=fail (p=none dis=none) header.from=redhat.com Authentication-Results: mail.kernel.org; spf=pass smtp.mailfrom=owner-linux-mm@kvack.org Received: by kanga.kvack.org (Postfix) id DE5E96B00C3; Wed, 14 Jul 2021 18:24:55 -0400 (EDT) Received: by kanga.kvack.org (Postfix, from userid 40) id DBCFA6B00C4; Wed, 14 Jul 2021 18:24:55 -0400 (EDT) X-Delivered-To: int-list-linux-mm@kvack.org Received: by kanga.kvack.org (Postfix, from userid 63042) id C36E06B00C5; Wed, 14 Jul 2021 18:24:55 -0400 (EDT) X-Delivered-To: linux-mm@kvack.org Received: from forelay.hostedemail.com (smtprelay0043.hostedemail.com [216.40.44.43]) by kanga.kvack.org (Postfix) with ESMTP id A132E6B00C3 for ; Wed, 14 Jul 2021 18:24:55 -0400 (EDT) Received: from smtpin36.hostedemail.com (10.5.19.251.rfc1918.com [10.5.19.251]) by forelay01.hostedemail.com (Postfix) with ESMTP id 8ABB2184799F3 for ; Wed, 14 Jul 2021 22:24:54 +0000 (UTC) X-FDA: 78362624508.36.F3A0994 Received: from us-smtp-delivery-124.mimecast.com (us-smtp-delivery-124.mimecast.com [216.205.24.124]) by imf20.hostedemail.com (Postfix) with ESMTP id 36605D0000A4 for ; Wed, 14 Jul 2021 22:24:54 +0000 (UTC) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=redhat.com; s=mimecast20190719; t=1626301493; h=from:from:reply-to:subject:subject:date:date:message-id:message-id: to:to:cc:cc:mime-version:mime-version:content-type:content-type: content-transfer-encoding:content-transfer-encoding: in-reply-to:in-reply-to:references:references; bh=l/r2tEksA7iAG8svGDApMzfpTe7AXzjcSqF0fovWgO0=; b=RJbrB6Bgp7xAH3ltkdtkyxjq4e2mVkPetCuJ+kPAqcNv/939JYkDTEWbzezsdTyd4bb++E vcWiNI5RAH4M7n3E+fjv8rgn6iuYaPJ0zMn1jCOzREm9ulQn0ddGvH36I6ddf3B3teZMc2 yjf+JGAeIx3d+uOhdK1DwDahwyxuzXk= Received: from mail-qt1-f200.google.com (mail-qt1-f200.google.com [209.85.160.200]) (Using TLS) by relay.mimecast.com with ESMTP id us-mta-587-mAuTaMZVOYqoBpG6VfAV9g-1; Wed, 14 Jul 2021 18:24:50 -0400 X-MC-Unique: mAuTaMZVOYqoBpG6VfAV9g-1 Received: by mail-qt1-f200.google.com with SMTP id b6-20020a05622a0206b029025eb726dd9bso389344qtx.8 for ; Wed, 14 Jul 2021 15:24:50 -0700 (PDT) X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20161025; h=x-gm-message-state:from:to:cc:subject:date:message-id:in-reply-to :references:mime-version:content-transfer-encoding; bh=l/r2tEksA7iAG8svGDApMzfpTe7AXzjcSqF0fovWgO0=; b=WaD2vGPju4kJR/nnL9QBhH7XeKhWEPkImCqjzGq4ovsoYF62Hi3LBhpP/aHuQInLsF zqZs/7RF9LHn3vp3l769SkJp+ijyRAohm8lsAfGpNmt79R2Yzk+TkvijbctD06qiO6IT Fb5OtmrqB7b6IpR4s1rWtZmAlzA6D7Rn9BKPOUl3wu/BhL5R6Q3qZtwAtT0bkF0fysMb Oqfp2cNrUog9kx4Cb03DsS6BvxHATDmeyoV9p0OI6lL0i+ychU4xHmhbSG4WUR7h+B1/ /0y3Cq5El/+0mKORC7kN/A2BDFmVqlOwtjUIZEpuP5hqs00CE4VPxaIA40PIHS9d9Cr0 5CTw== X-Gm-Message-State: AOAM532+5UX9DLIDliNch7cCPF35y7HpW0fOFqPF8KCydO8eeBo+MPG5 zoY+72iOfmzH63G0FoJPUbp7AvQCr/TpG2wxPiGnRYvpAZsl7s67DCaY7hLruYdt8YVwKEdNZpD ApkQmXFcVHYtnasz4BmAlRLOWI0u1X8As4LIAcNpp9lg7HJorS/fTH7zyIkUd X-Received: by 2002:ac8:764e:: with SMTP id i14mr320346qtr.247.1626301490117; Wed, 14 Jul 2021 15:24:50 -0700 (PDT) X-Google-Smtp-Source: ABdhPJxke+1R9jvCzeFeO4Xhy524xOIxKblC2IZ31epDh9hl1os1ZvwizYxkPAeMEaZbjJZGzGEjNQ== X-Received: by 2002:ac8:764e:: with SMTP id i14mr320312qtr.247.1626301489853; Wed, 14 Jul 2021 15:24:49 -0700 (PDT) Received: from localhost.localdomain (bras-base-toroon474qw-grc-65-184-144-111-238.dsl.bell.ca. [184.144.111.238]) by smtp.gmail.com with ESMTPSA id u184sm1655517qkd.53.2021.07.14.15.24.48 (version=TLS1_3 cipher=TLS_AES_256_GCM_SHA384 bits=256/256); Wed, 14 Jul 2021 15:24:49 -0700 (PDT) From: Peter Xu To: linux-mm@kvack.org, linux-kernel@vger.kernel.org Cc: Andrew Morton , Mike Kravetz , Axel Rasmussen , Miaohe Lin , "Kirill A . Shutemov" , Hugh Dickins , Jason Gunthorpe , Alistair Popple , Matthew Wilcox , peterx@redhat.com, Jerome Glisse , Andrea Arcangeli , Mike Rapoport , Nadav Amit , David Hildenbrand Subject: [PATCH v4 14/26] shmem/userfaultfd: Pass over uffd-wp special swap pte when fork() Date: Wed, 14 Jul 2021 18:24:46 -0400 Message-Id: <20210714222446.48788-1-peterx@redhat.com> X-Mailer: git-send-email 2.31.1 In-Reply-To: <20210714222117.47648-1-peterx@redhat.com> References: <20210714222117.47648-1-peterx@redhat.com> MIME-Version: 1.0 X-Mimecast-Spam-Score: 0 X-Mimecast-Originator: redhat.com Authentication-Results: imf20.hostedemail.com; dkim=pass header.d=redhat.com header.s=mimecast20190719 header.b=RJbrB6Bg; spf=none (imf20.hostedemail.com: domain of peterx@redhat.com has no SPF policy when checking 216.205.24.124) smtp.mailfrom=peterx@redhat.com; dmarc=pass (policy=none) header.from=redhat.com X-Rspamd-Server: rspam05 X-Stat-Signature: j3oeaedonf6pm94ihyfjoke5b4m5dk9b X-Rspamd-Queue-Id: 36605D0000A4 X-HE-Tag: 1626301494-257948 X-Bogosity: Ham, tests=bogofilter, spamicity=0.000000, version=1.2.4 Sender: owner-linux-mm@kvack.org Precedence: bulk X-Loop: owner-majordomo@kvack.org List-ID: It should be handled similarly like other uffd-wp wr-protected ptes: we should pass it over when the dst_vma has VM_UFFD_WP armed, otherwise drop it. Signed-off-by: Peter Xu --- mm/memory.c | 15 ++++++++++++++- 1 file changed, 14 insertions(+), 1 deletion(-) diff --git a/mm/memory.c b/mm/memory.c index 223781f115e9..af91bee934c7 100644 --- a/mm/memory.c +++ b/mm/memory.c @@ -776,8 +776,21 @@ copy_nonpresent_pte(struct mm_struct *dst_mm, struct mm_struct *src_mm, unsigned long vm_flags = dst_vma->vm_flags; pte_t pte = *src_pte; struct page *page; - swp_entry_t entry = pte_to_swp_entry(pte); + swp_entry_t entry; + + if (unlikely(is_swap_special_pte(pte))) { + /* + * uffd-wp special swap pte is the only possibility for now. + * If dst vma is registered with uffd-wp, copy it over. + * Otherwise, ignore this pte as if it's a none pte would work. + */ + WARN_ON_ONCE(!pte_swp_uffd_wp_special(pte)); + if (userfaultfd_wp(dst_vma)) + set_pte_at(dst_mm, addr, dst_pte, pte); + return 0; + } + entry = pte_to_swp_entry(pte); if (likely(!non_swap_entry(entry))) { if (swap_duplicate(entry) < 0) return -EIO; From patchwork Wed Jul 14 22:24:50 2021 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Peter Xu X-Patchwork-Id: 12377969 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org X-Spam-Level: X-Spam-Status: No, score=-16.6 required=3.0 tests=BAYES_00,DKIM_INVALID, DKIM_SIGNED,HEADER_FROM_DIFFERENT_DOMAINS,INCLUDES_CR_TRAILER,INCLUDES_PATCH, MAILING_LIST_MULTI,SPF_HELO_NONE,SPF_PASS,USER_AGENT_GIT autolearn=ham autolearn_force=no version=3.4.0 Received: from mail.kernel.org (mail.kernel.org [198.145.29.99]) by smtp.lore.kernel.org (Postfix) with ESMTP id 73FBDC12002 for ; Wed, 14 Jul 2021 22:24:58 +0000 (UTC) Received: from kanga.kvack.org (kanga.kvack.org [205.233.56.17]) by mail.kernel.org (Postfix) with ESMTP id 20C20600D1 for ; Wed, 14 Jul 2021 22:24:58 +0000 (UTC) DMARC-Filter: OpenDMARC Filter v1.3.2 mail.kernel.org 20C20600D1 Authentication-Results: mail.kernel.org; dmarc=fail (p=none dis=none) header.from=redhat.com Authentication-Results: mail.kernel.org; spf=pass smtp.mailfrom=owner-linux-mm@kvack.org Received: by kanga.kvack.org (Postfix) id 7567F6B00C5; Wed, 14 Jul 2021 18:24:58 -0400 (EDT) Received: by kanga.kvack.org (Postfix, from userid 40) id 72EB76B00C6; Wed, 14 Jul 2021 18:24:58 -0400 (EDT) X-Delivered-To: int-list-linux-mm@kvack.org Received: by kanga.kvack.org (Postfix, from userid 63042) id 57F7E8D0003; Wed, 14 Jul 2021 18:24:58 -0400 (EDT) X-Delivered-To: linux-mm@kvack.org Received: from forelay.hostedemail.com (smtprelay0205.hostedemail.com [216.40.44.205]) by kanga.kvack.org (Postfix) with ESMTP id 34A0F6B00C5 for ; Wed, 14 Jul 2021 18:24:58 -0400 (EDT) Received: from smtpin23.hostedemail.com (10.5.19.251.rfc1918.com [10.5.19.251]) by forelay01.hostedemail.com (Postfix) with ESMTP id 1F2E818479699 for ; Wed, 14 Jul 2021 22:24:57 +0000 (UTC) X-FDA: 78362624634.23.64D3195 Received: from us-smtp-delivery-124.mimecast.com (us-smtp-delivery-124.mimecast.com [170.10.133.124]) by imf29.hostedemail.com (Postfix) with ESMTP id CA785900025A for ; Wed, 14 Jul 2021 22:24:56 +0000 (UTC) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=redhat.com; s=mimecast20190719; t=1626301496; h=from:from:reply-to:subject:subject:date:date:message-id:message-id: to:to:cc:cc:mime-version:mime-version:content-type:content-type: content-transfer-encoding:content-transfer-encoding: in-reply-to:in-reply-to:references:references; bh=HoJrf5XHMT1foDa7QiH7gGocyne+rxUVzp3LYIPfb1I=; b=JxQ+wQa2VE3rgN3et7fv9q5iTKkItghwjzXB0R4nm/1dTgyQ7c/3O5ZiwYy6PHwYKFGLMi XZWRSW5MsglUrXJgiQ+SIC/XugCnj5IDeP3N0/anblvHqWznV7XjJZKT+GkQDo6kGI4AyX SO+i500LzmW5BtUX5I5YPHQywe5XpX4= Received: from mail-qv1-f72.google.com (mail-qv1-f72.google.com [209.85.219.72]) (Using TLS) by relay.mimecast.com with ESMTP id us-mta-570-GbsGxMcmN_6IfO5dKno5ig-1; Wed, 14 Jul 2021 18:24:55 -0400 X-MC-Unique: GbsGxMcmN_6IfO5dKno5ig-1 Received: by mail-qv1-f72.google.com with SMTP id bk10-20020a05621406eab02902d1aac9c421so2722513qvb.1 for ; Wed, 14 Jul 2021 15:24:55 -0700 (PDT) X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20161025; h=x-gm-message-state:from:to:cc:subject:date:message-id:in-reply-to :references:mime-version:content-transfer-encoding; bh=HoJrf5XHMT1foDa7QiH7gGocyne+rxUVzp3LYIPfb1I=; b=JUqulJHUCc3HMPD/dUjkBfBHzcm1IhnJ+Axfv0sX/ikSOd8q5QPcOP8ufbnTvn3Hn/ LDslGLEJfEQujXchcHoQrBFxYegUr+IwpyWiS7XNpgWwJKEcC3d2eS13fqiUHbygHQke MJato2PBwzZ6VrrvM4H8Uy74bivUWGaBs7J4nIzKnCWas6iR0EsES+Mfw3whQgO9eAuj OSJNU0SbFD9Cp17O3Tds2YJYaQG9oq3jpUPVRmY8R5mLS4WO/u5cyifZmOmtfP8maxUj EAjXCyme1Xlv+ltrLqoxVjV7fI0kYca+K3TZ9bKOg8HN/2DVKn2IPi1rlApTIJXHpCLY nbjA== X-Gm-Message-State: AOAM531rV0chk6La7+VbeOcnr5q5vkhLGqQQ/IFddpBGt4cLni9JX+8t WK8mDmKEtIz+cbSZ6tqLIbyA+Gji9FMbGgyXV70ehawNRoHFjtIoUYuOqdWPtC+Sr3MbQ9xlu6/ sM2ASWJ8SRPRnqOvVlZ60KVRZ/FsSoi1WlFLOLY/YNhd8Dadjdpy9fnuzbraB X-Received: by 2002:a05:620a:99b:: with SMTP id x27mr241783qkx.226.1626301494414; Wed, 14 Jul 2021 15:24:54 -0700 (PDT) X-Google-Smtp-Source: ABdhPJzOW00e58jO9AZD7fw0CnPfeLPUWJrjUqdtrilL2qz6WVvPrJUbhPG7ghrnW4FLQUsYSIj/ew== X-Received: by 2002:a05:620a:99b:: with SMTP id x27mr241746qkx.226.1626301494149; Wed, 14 Jul 2021 15:24:54 -0700 (PDT) Received: from localhost.localdomain (bras-base-toroon474qw-grc-65-184-144-111-238.dsl.bell.ca. [184.144.111.238]) by smtp.gmail.com with ESMTPSA id f36sm1288702qtb.40.2021.07.14.15.24.52 (version=TLS1_3 cipher=TLS_AES_256_GCM_SHA384 bits=256/256); Wed, 14 Jul 2021 15:24:53 -0700 (PDT) From: Peter Xu To: linux-mm@kvack.org, linux-kernel@vger.kernel.org Cc: Andrew Morton , Mike Kravetz , Axel Rasmussen , Miaohe Lin , "Kirill A . Shutemov" , Hugh Dickins , Jason Gunthorpe , Alistair Popple , Matthew Wilcox , peterx@redhat.com, Jerome Glisse , Andrea Arcangeli , Mike Rapoport , Nadav Amit , David Hildenbrand Subject: [PATCH v4 15/26] mm/hugetlb: Drop __unmap_hugepage_range definition from hugetlb.h Date: Wed, 14 Jul 2021 18:24:50 -0400 Message-Id: <20210714222450.48840-1-peterx@redhat.com> X-Mailer: git-send-email 2.31.1 In-Reply-To: <20210714222117.47648-1-peterx@redhat.com> References: <20210714222117.47648-1-peterx@redhat.com> MIME-Version: 1.0 X-Mimecast-Spam-Score: 0 X-Mimecast-Originator: redhat.com Authentication-Results: imf29.hostedemail.com; dkim=pass header.d=redhat.com header.s=mimecast20190719 header.b=JxQ+wQa2; spf=none (imf29.hostedemail.com: domain of peterx@redhat.com has no SPF policy when checking 170.10.133.124) smtp.mailfrom=peterx@redhat.com; dmarc=pass (policy=none) header.from=redhat.com X-Rspamd-Server: rspam03 X-Rspamd-Queue-Id: CA785900025A X-Stat-Signature: w6ostrs9gf4qf6y3znad5tg135uz4buc X-HE-Tag: 1626301496-562389 X-Bogosity: Ham, tests=bogofilter, spamicity=0.000000, version=1.2.4 Sender: owner-linux-mm@kvack.org Precedence: bulk X-Loop: owner-majordomo@kvack.org List-ID: Drop it in the header since it's only used in hugetlb.c. Suggested-by: Mike Kravetz Signed-off-by: Peter Xu Reported-by: kernel test robot Reported-by: kernel test robot Reported-by: kernel test robot --- include/linux/hugetlb.h | 10 ---------- 1 file changed, 10 deletions(-) diff --git a/include/linux/hugetlb.h b/include/linux/hugetlb.h index f7ca1a3870ea..c30f39815e13 100644 --- a/include/linux/hugetlb.h +++ b/include/linux/hugetlb.h @@ -143,9 +143,6 @@ void __unmap_hugepage_range_final(struct mmu_gather *tlb, struct vm_area_struct *vma, unsigned long start, unsigned long end, struct page *ref_page); -void __unmap_hugepage_range(struct mmu_gather *tlb, struct vm_area_struct *vma, - unsigned long start, unsigned long end, - struct page *ref_page); void hugetlb_report_meminfo(struct seq_file *); int hugetlb_report_node_meminfo(char *buf, int len, int nid); void hugetlb_show_meminfo(void); @@ -385,13 +382,6 @@ static inline void __unmap_hugepage_range_final(struct mmu_gather *tlb, BUG(); } -static inline void __unmap_hugepage_range(struct mmu_gather *tlb, - struct vm_area_struct *vma, unsigned long start, - unsigned long end, struct page *ref_page) -{ - BUG(); -} - static inline vm_fault_t hugetlb_fault(struct mm_struct *mm, struct vm_area_struct *vma, unsigned long address, unsigned int flags) From patchwork Wed Jul 14 22:24:54 2021 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Peter Xu X-Patchwork-Id: 12377971 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org X-Spam-Level: X-Spam-Status: No, score=-16.6 required=3.0 tests=BAYES_00,DKIM_INVALID, DKIM_SIGNED,HEADER_FROM_DIFFERENT_DOMAINS,INCLUDES_CR_TRAILER,INCLUDES_PATCH, MAILING_LIST_MULTI,SPF_HELO_NONE,SPF_PASS,USER_AGENT_GIT autolearn=ham autolearn_force=no version=3.4.0 Received: from mail.kernel.org (mail.kernel.org [198.145.29.99]) by smtp.lore.kernel.org (Postfix) with ESMTP id 05B8EC12002 for ; Wed, 14 Jul 2021 22:25:02 +0000 (UTC) Received: from kanga.kvack.org (kanga.kvack.org [205.233.56.17]) by mail.kernel.org (Postfix) with ESMTP id AFC8D600D1 for ; Wed, 14 Jul 2021 22:25:01 +0000 (UTC) DMARC-Filter: OpenDMARC Filter v1.3.2 mail.kernel.org AFC8D600D1 Authentication-Results: mail.kernel.org; dmarc=fail (p=none dis=none) header.from=redhat.com Authentication-Results: mail.kernel.org; spf=pass smtp.mailfrom=owner-linux-mm@kvack.org Received: by kanga.kvack.org (Postfix) id 0C6606B00C7; Wed, 14 Jul 2021 18:25:02 -0400 (EDT) Received: by kanga.kvack.org (Postfix, from userid 40) id 076E76B00C8; Wed, 14 Jul 2021 18:25:02 -0400 (EDT) X-Delivered-To: int-list-linux-mm@kvack.org Received: by kanga.kvack.org (Postfix, from userid 63042) id E32698D0003; Wed, 14 Jul 2021 18:25:01 -0400 (EDT) X-Delivered-To: linux-mm@kvack.org Received: from forelay.hostedemail.com (smtprelay0231.hostedemail.com [216.40.44.231]) by kanga.kvack.org (Postfix) with ESMTP id BF0126B00C7 for ; Wed, 14 Jul 2021 18:25:01 -0400 (EDT) Received: from smtpin28.hostedemail.com (10.5.19.251.rfc1918.com [10.5.19.251]) by forelay04.hostedemail.com (Postfix) with ESMTP id B1FF11CB18 for ; Wed, 14 Jul 2021 22:25:00 +0000 (UTC) X-FDA: 78362624760.28.CC98FF0 Received: from us-smtp-delivery-124.mimecast.com (us-smtp-delivery-124.mimecast.com [170.10.133.124]) by imf01.hostedemail.com (Postfix) with ESMTP id 641F7500C711 for ; Wed, 14 Jul 2021 22:25:00 +0000 (UTC) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=redhat.com; s=mimecast20190719; t=1626301499; h=from:from:reply-to:subject:subject:date:date:message-id:message-id: to:to:cc:cc:mime-version:mime-version:content-type:content-type: content-transfer-encoding:content-transfer-encoding: in-reply-to:in-reply-to:references:references; bh=SAmjiglASlkj6C13Cutnvz5k0d0w8IXdZ7Qyh6+DwYY=; b=h5kYj4PgQM4cGkmzj/kvNZhQylSoYTnUxWRCwPDutaAngKfg1/2EPlHej7ggPuNYYImY+E 0pcCsW+EoMw5TiD6fTKLeWDob8PaF8x9ZHY/TeoGIpTTIwMK3Eay7MbJRWThHXbKrpW9TK zzRwvGZzOBDR7dChNZP5aVr4Nfq8VO0= Received: from mail-qk1-f198.google.com (mail-qk1-f198.google.com [209.85.222.198]) (Using TLS) by relay.mimecast.com with ESMTP id us-mta-350-u8AHv9z0PQG1bAx1aq_gVQ-1; Wed, 14 Jul 2021 18:24:59 -0400 X-MC-Unique: u8AHv9z0PQG1bAx1aq_gVQ-1 Received: by mail-qk1-f198.google.com with SMTP id f203-20020a379cd40000b02903b861bec838so2312509qke.7 for ; Wed, 14 Jul 2021 15:24:59 -0700 (PDT) X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20161025; h=x-gm-message-state:from:to:cc:subject:date:message-id:in-reply-to :references:mime-version:content-transfer-encoding; bh=SAmjiglASlkj6C13Cutnvz5k0d0w8IXdZ7Qyh6+DwYY=; b=kV4jUX9dmtUIk2ehol6JEPBQ84jgJU+Du8lxRhf9TgwXGI7Fr0FEGhjxXdMMV2Uizc dLMWfN8h8r6DJJs7iwIhqEmWWenoDeitmNuk2LmaGLIIqsRvMUSDDdiCYS6Z56kNyAfA o9OZky0Ipds5M9cfetLxS+7IqujAwkMHiqkf5Lu+gsnW/MLSjcmGlQTdLqS0ab8shYSs Aoc2IUj8vLmtqEN2PujNZVSo4qml3zptoLFsHlrz/KmZL6BGJRXKPX149jh3oGPIxzkc GOD8V0l7rVLr0siYoFnFfveY4w+p1FCiozX/RjTbcPZdjAaCyKTb32BhXPJ9xs+E2dEq fFdw== X-Gm-Message-State: AOAM530IZCfaO895YnYCQQo8MaQcsom4munerLhj7Yq1+jIm+pqjjeDl UrRQtbBbmfiOvHmPaCQe3tFrVMQ0Rdn2gZNMLzYqBUm2PXjf73PhJu5SpqTsv1hfNoJfZcyTjdV C0ViRXXs+/wbcBgyUOvjcILiXjd7wtnfMh46X8Q6g8qIToRcK5jhiaOQUHceY X-Received: by 2002:a0c:ff01:: with SMTP id w1mr251311qvt.28.1626301498218; Wed, 14 Jul 2021 15:24:58 -0700 (PDT) X-Google-Smtp-Source: ABdhPJzsaEytrl+N0UaFyQiUBIwG7Khd7frW2mnu6eY1ibxH9wfPndm7ab048Rnu8cItyAyW05WjVg== X-Received: by 2002:a0c:ff01:: with SMTP id w1mr251275qvt.28.1626301497935; Wed, 14 Jul 2021 15:24:57 -0700 (PDT) Received: from localhost.localdomain (bras-base-toroon474qw-grc-65-184-144-111-238.dsl.bell.ca. [184.144.111.238]) by smtp.gmail.com with ESMTPSA id r5sm1274775qtm.75.2021.07.14.15.24.56 (version=TLS1_3 cipher=TLS_AES_256_GCM_SHA384 bits=256/256); Wed, 14 Jul 2021 15:24:57 -0700 (PDT) From: Peter Xu To: linux-mm@kvack.org, linux-kernel@vger.kernel.org Cc: Andrew Morton , Mike Kravetz , Axel Rasmussen , Miaohe Lin , "Kirill A . Shutemov" , Hugh Dickins , Jason Gunthorpe , Alistair Popple , Matthew Wilcox , peterx@redhat.com, Jerome Glisse , Andrea Arcangeli , Mike Rapoport , Nadav Amit , David Hildenbrand Subject: [PATCH v4 16/26] mm/hugetlb: Introduce huge pte version of uffd-wp helpers Date: Wed, 14 Jul 2021 18:24:54 -0400 Message-Id: <20210714222454.48891-1-peterx@redhat.com> X-Mailer: git-send-email 2.31.1 In-Reply-To: <20210714222117.47648-1-peterx@redhat.com> References: <20210714222117.47648-1-peterx@redhat.com> MIME-Version: 1.0 X-Mimecast-Spam-Score: 0 X-Mimecast-Originator: redhat.com Authentication-Results: imf01.hostedemail.com; dkim=pass header.d=redhat.com header.s=mimecast20190719 header.b=h5kYj4Pg; spf=none (imf01.hostedemail.com: domain of peterx@redhat.com has no SPF policy when checking 170.10.133.124) smtp.mailfrom=peterx@redhat.com; dmarc=pass (policy=none) header.from=redhat.com X-Stat-Signature: o89c7dqnmhkw549bu1p8akk3ofigpp6b X-Rspamd-Queue-Id: 641F7500C711 X-Rspamd-Server: rspam01 X-HE-Tag: 1626301500-148862 X-Bogosity: Ham, tests=bogofilter, spamicity=0.000000, version=1.2.4 Sender: owner-linux-mm@kvack.org Precedence: bulk X-Loop: owner-majordomo@kvack.org List-ID: They will be used in the follow up patches to either check/set/clear uffd-wp bit of a huge pte. So far it reuses all the small pte helpers. Archs can overwrite these versions when necessary (with __HAVE_ARCH_HUGE_PTE_UFFD_WP* macros) in the future. Signed-off-by: Peter Xu --- include/asm-generic/hugetlb.h | 15 +++++++++++++++ 1 file changed, 15 insertions(+) diff --git a/include/asm-generic/hugetlb.h b/include/asm-generic/hugetlb.h index 8e1e6244a89d..c45b9deb41ff 100644 --- a/include/asm-generic/hugetlb.h +++ b/include/asm-generic/hugetlb.h @@ -32,6 +32,21 @@ static inline pte_t huge_pte_modify(pte_t pte, pgprot_t newprot) return pte_modify(pte, newprot); } +static inline pte_t huge_pte_mkuffd_wp(pte_t pte) +{ + return pte_mkuffd_wp(pte); +} + +static inline pte_t huge_pte_clear_uffd_wp(pte_t pte) +{ + return pte_clear_uffd_wp(pte); +} + +static inline int huge_pte_uffd_wp(pte_t pte) +{ + return pte_uffd_wp(pte); +} + #ifndef __HAVE_ARCH_HUGE_PTE_CLEAR static inline void huge_pte_clear(struct mm_struct *mm, unsigned long addr, pte_t *ptep, unsigned long sz) From patchwork Wed Jul 14 22:24:58 2021 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Peter Xu X-Patchwork-Id: 12377973 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org X-Spam-Level: X-Spam-Status: No, score=-16.6 required=3.0 tests=BAYES_00,DKIM_INVALID, DKIM_SIGNED,HEADER_FROM_DIFFERENT_DOMAINS,INCLUDES_CR_TRAILER,INCLUDES_PATCH, MAILING_LIST_MULTI,SPF_HELO_NONE,SPF_PASS,USER_AGENT_GIT autolearn=ham autolearn_force=no version=3.4.0 Received: from mail.kernel.org (mail.kernel.org [198.145.29.99]) by smtp.lore.kernel.org (Postfix) with ESMTP id 4F9F2C12002 for ; Wed, 14 Jul 2021 22:25:06 +0000 (UTC) Received: from kanga.kvack.org (kanga.kvack.org [205.233.56.17]) by mail.kernel.org (Postfix) with ESMTP id E0015613C9 for ; Wed, 14 Jul 2021 22:25:05 +0000 (UTC) DMARC-Filter: OpenDMARC Filter v1.3.2 mail.kernel.org E0015613C9 Authentication-Results: mail.kernel.org; dmarc=fail (p=none dis=none) header.from=redhat.com Authentication-Results: mail.kernel.org; spf=pass smtp.mailfrom=owner-linux-mm@kvack.org Received: by kanga.kvack.org (Postfix) id 3ABB76B00C8; Wed, 14 Jul 2021 18:25:06 -0400 (EDT) Received: by kanga.kvack.org (Postfix, from userid 40) id 35D178D0003; Wed, 14 Jul 2021 18:25:06 -0400 (EDT) X-Delivered-To: int-list-linux-mm@kvack.org Received: by kanga.kvack.org (Postfix, from userid 63042) id 1D6D56B00CA; Wed, 14 Jul 2021 18:25:06 -0400 (EDT) X-Delivered-To: linux-mm@kvack.org Received: from forelay.hostedemail.com (smtprelay0014.hostedemail.com [216.40.44.14]) by kanga.kvack.org (Postfix) with ESMTP id EE0B36B00C8 for ; Wed, 14 Jul 2021 18:25:05 -0400 (EDT) Received: from smtpin06.hostedemail.com (10.5.19.251.rfc1918.com [10.5.19.251]) by forelay02.hostedemail.com (Postfix) with ESMTP id DE2A9231D8 for ; Wed, 14 Jul 2021 22:25:04 +0000 (UTC) X-FDA: 78362624928.06.268265F Received: from us-smtp-delivery-124.mimecast.com (us-smtp-delivery-124.mimecast.com [170.10.133.124]) by imf02.hostedemail.com (Postfix) with ESMTP id 9AB89700177D for ; Wed, 14 Jul 2021 22:25:04 +0000 (UTC) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=redhat.com; s=mimecast20190719; t=1626301504; h=from:from:reply-to:subject:subject:date:date:message-id:message-id: to:to:cc:cc:mime-version:mime-version:content-type:content-type: content-transfer-encoding:content-transfer-encoding: in-reply-to:in-reply-to:references:references; bh=pgOZ+9IjwVfKb5tKhAxRusG1z8GoeGAv34cmBph/B3s=; b=W8baDqN/jefgG6jJBd53gMdPIp354ntaXo/f8eTCjhydJMqFF5d/IrCD7iP0bZEnCKR36m o6VX7j2Bhv/g2ArTHN+u9xXocrNGGiWoCxmx1w9jy6K71rIeF3DkhiN3ZmIq9ADoSW3jfI nkhB5CARv7zFis+XraQELk/r8nnRnuc= Received: from mail-qk1-f199.google.com (mail-qk1-f199.google.com [209.85.222.199]) (Using TLS) by relay.mimecast.com with ESMTP id us-mta-406-Ki_ouCYJMte_E-xk4Wf9cg-1; Wed, 14 Jul 2021 18:25:03 -0400 X-MC-Unique: Ki_ouCYJMte_E-xk4Wf9cg-1 Received: by mail-qk1-f199.google.com with SMTP id i190-20020a3786c70000b02903b54f40b442so2360771qkd.0 for ; Wed, 14 Jul 2021 15:25:03 -0700 (PDT) X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20161025; h=x-gm-message-state:from:to:cc:subject:date:message-id:in-reply-to :references:mime-version:content-transfer-encoding; bh=pgOZ+9IjwVfKb5tKhAxRusG1z8GoeGAv34cmBph/B3s=; b=Sq3ukUZlr9YqMM41JEiM0QOCKkaSpe18mMyrNWmpprm8r64UbfkG7CACDLTspI1mxy pXmJTZcFWfLyZ2lLM9u2eoV/0cmZjYVjyKID6eydQ0ndIhkV5LSyguO6fQAZ8pnUqePf bW6+KTYsEDcpb445PkWRBzNx6FWfevceS84kumFc/7E4F5t/GBmkNKcmp9LevGQ/JYx0 i+GN0JOoI8RMa0hqM5gsM/kBnpG1/lSUnBRw4gSGa+b005DATJbX9AttaiABGDUh78+l /Mtq6Tp3O11sssTLMrjbJQmbkW/qREZe5pioxsFtbQ5D8BeDw9KrLgjVEPPomE46ObPx gVhg== X-Gm-Message-State: AOAM531JfV9x7nRVvyMR4ShVTtgwhL6n2h7r9XadBsz20LthyqdaV5mc dbJpMvRrh1abnLAXgmH/yq8hbUXqcWm2RXpNW3nkZnhbcKnWbFa65B2UU+qsK0xDKwDCF2wF0qP 1Sf3xGcLDfYkFkcX231TFLuSIoeTSzeJ7trfbQj7XqUnZNt5hP7bLLgiQDu2X X-Received: by 2002:ac8:5ac3:: with SMTP id d3mr318904qtd.257.1626301502438; Wed, 14 Jul 2021 15:25:02 -0700 (PDT) X-Google-Smtp-Source: ABdhPJwmUP5TjPEpwzMNS3GOKkC/oUMalgKy2Zc3e2y7pAB8MiTJNFhofDrLUpO66BNt/9fBL+Yu3A== X-Received: by 2002:ac8:5ac3:: with SMTP id d3mr318869qtd.257.1626301502188; Wed, 14 Jul 2021 15:25:02 -0700 (PDT) Received: from localhost.localdomain (bras-base-toroon474qw-grc-65-184-144-111-238.dsl.bell.ca. [184.144.111.238]) by smtp.gmail.com with ESMTPSA id o126sm1606702qka.74.2021.07.14.15.25.00 (version=TLS1_3 cipher=TLS_AES_256_GCM_SHA384 bits=256/256); Wed, 14 Jul 2021 15:25:01 -0700 (PDT) From: Peter Xu To: linux-mm@kvack.org, linux-kernel@vger.kernel.org Cc: Andrew Morton , Mike Kravetz , Axel Rasmussen , Miaohe Lin , "Kirill A . Shutemov" , Hugh Dickins , Jason Gunthorpe , Alistair Popple , Matthew Wilcox , peterx@redhat.com, Jerome Glisse , Andrea Arcangeli , Mike Rapoport , Nadav Amit , David Hildenbrand Subject: [PATCH v4 17/26] hugetlb/userfaultfd: Hook page faults for uffd write protection Date: Wed, 14 Jul 2021 18:24:58 -0400 Message-Id: <20210714222458.48942-1-peterx@redhat.com> X-Mailer: git-send-email 2.31.1 In-Reply-To: <20210714222117.47648-1-peterx@redhat.com> References: <20210714222117.47648-1-peterx@redhat.com> MIME-Version: 1.0 X-Mimecast-Spam-Score: 0 X-Mimecast-Originator: redhat.com Authentication-Results: imf02.hostedemail.com; dkim=pass header.d=redhat.com header.s=mimecast20190719 header.b="W8baDqN/"; spf=none (imf02.hostedemail.com: domain of peterx@redhat.com has no SPF policy when checking 170.10.133.124) smtp.mailfrom=peterx@redhat.com; dmarc=pass (policy=none) header.from=redhat.com X-Stat-Signature: tk4bfzq6sahp6pgrkn5mw68ugqwzabm4 X-Rspamd-Queue-Id: 9AB89700177D X-Rspamd-Server: rspam01 X-HE-Tag: 1626301504-224691 X-Bogosity: Ham, tests=bogofilter, spamicity=0.000000, version=1.2.4 Sender: owner-linux-mm@kvack.org Precedence: bulk X-Loop: owner-majordomo@kvack.org List-ID: Hook up hugetlbfs_fault() with the capability to handle userfaultfd-wp faults. We do this slightly earlier than hugetlb_cow() so that we can avoid taking some extra locks that we definitely don't need. Reviewed-by: Mike Kravetz Signed-off-by: Peter Xu --- mm/hugetlb.c | 19 +++++++++++++++++++ 1 file changed, 19 insertions(+) diff --git a/mm/hugetlb.c b/mm/hugetlb.c index 924553aa8f78..8559b8bb7fa5 100644 --- a/mm/hugetlb.c +++ b/mm/hugetlb.c @@ -5059,6 +5059,25 @@ vm_fault_t hugetlb_fault(struct mm_struct *mm, struct vm_area_struct *vma, if (unlikely(!pte_same(entry, huge_ptep_get(ptep)))) goto out_ptl; + /* Handle userfault-wp first, before trying to lock more pages */ + if (userfaultfd_wp(vma) && huge_pte_uffd_wp(huge_ptep_get(ptep)) && + (flags & FAULT_FLAG_WRITE) && !huge_pte_write(entry)) { + struct vm_fault vmf = { + .vma = vma, + .address = haddr, + .flags = flags, + }; + + spin_unlock(ptl); + if (pagecache_page) { + unlock_page(pagecache_page); + put_page(pagecache_page); + } + mutex_unlock(&hugetlb_fault_mutex_table[hash]); + i_mmap_unlock_read(mapping); + return handle_userfault(&vmf, VM_UFFD_WP); + } + /* * hugetlb_cow() requires page locks of pte_page(entry) and * pagecache_page, so here we need take the former one From patchwork Wed Jul 14 22:25:02 2021 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Peter Xu X-Patchwork-Id: 12377975 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org X-Spam-Level: X-Spam-Status: No, score=-16.6 required=3.0 tests=BAYES_00,DKIM_INVALID, DKIM_SIGNED,HEADER_FROM_DIFFERENT_DOMAINS,INCLUDES_CR_TRAILER,INCLUDES_PATCH, MAILING_LIST_MULTI,SPF_HELO_NONE,USER_AGENT_GIT autolearn=ham autolearn_force=no version=3.4.0 Received: from mail.kernel.org (mail.kernel.org [198.145.29.99]) by smtp.lore.kernel.org (Postfix) with ESMTP id 0C867C47E4E for ; Wed, 14 Jul 2021 22:25:12 +0000 (UTC) Received: from kanga.kvack.org (kanga.kvack.org [205.233.56.17]) by mail.kernel.org (Postfix) with ESMTP id BBB3C600D1 for ; Wed, 14 Jul 2021 22:25:11 +0000 (UTC) DMARC-Filter: OpenDMARC Filter v1.3.2 mail.kernel.org BBB3C600D1 Authentication-Results: mail.kernel.org; dmarc=fail (p=none dis=none) header.from=redhat.com Authentication-Results: mail.kernel.org; spf=pass smtp.mailfrom=owner-linux-mm@kvack.org Received: by kanga.kvack.org (Postfix) id 7CC4B6B00CB; Wed, 14 Jul 2021 18:25:11 -0400 (EDT) Received: by kanga.kvack.org (Postfix, from userid 40) id 77BBA6B00CC; Wed, 14 Jul 2021 18:25:11 -0400 (EDT) X-Delivered-To: int-list-linux-mm@kvack.org Received: by kanga.kvack.org (Postfix, from userid 63042) id 5F48F6B00CD; Wed, 14 Jul 2021 18:25:11 -0400 (EDT) X-Delivered-To: linux-mm@kvack.org Received: from forelay.hostedemail.com (smtprelay0135.hostedemail.com [216.40.44.135]) by kanga.kvack.org (Postfix) with ESMTP id 3AA6A6B00CB for ; Wed, 14 Jul 2021 18:25:11 -0400 (EDT) Received: from smtpin10.hostedemail.com (10.5.19.251.rfc1918.com [10.5.19.251]) by forelay03.hostedemail.com (Postfix) with ESMTP id 27D40824556B for ; Wed, 14 Jul 2021 22:25:10 +0000 (UTC) X-FDA: 78362625180.10.EBEBC6D Received: from us-smtp-delivery-124.mimecast.com (us-smtp-delivery-124.mimecast.com [216.205.24.124]) by imf12.hostedemail.com (Postfix) with ESMTP id C08BE10000B2 for ; Wed, 14 Jul 2021 22:25:09 +0000 (UTC) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=redhat.com; s=mimecast20190719; t=1626301509; h=from:from:reply-to:subject:subject:date:date:message-id:message-id: to:to:cc:cc:mime-version:mime-version:content-type:content-type: content-transfer-encoding:content-transfer-encoding: in-reply-to:in-reply-to:references:references; bh=KxMjY2pIac+F3Pu7DKFzRfyzJMN42tR8v58EMRPGSu0=; b=V+evOwuvF+y/uXed+Az9pAGw56jwAJH2OrlEg9JZWXXSCYDTbaPmUl60p+DqGGuYKQsMrc /HyP3Yo8hrFrsRsAMmrPy9Pc24LUQ53ZK06T9yQIcuVWxxDvQ2EoktgnkyrLR2eoV6DukO nzssa2g6t4i04rTeejGMI8rOI1bSAlQ= Received: from mail-qk1-f199.google.com (mail-qk1-f199.google.com [209.85.222.199]) (Using TLS) by relay.mimecast.com with ESMTP id us-mta-213-_GDzmRh6OeCQjHJ7U5uGXg-1; Wed, 14 Jul 2021 18:25:08 -0400 X-MC-Unique: _GDzmRh6OeCQjHJ7U5uGXg-1 Received: by mail-qk1-f199.google.com with SMTP id x11-20020ae9f80b0000b02903b8d1e253a9so90071qkh.11 for ; Wed, 14 Jul 2021 15:25:08 -0700 (PDT) X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20161025; h=x-gm-message-state:from:to:cc:subject:date:message-id:in-reply-to :references:mime-version:content-transfer-encoding; bh=KxMjY2pIac+F3Pu7DKFzRfyzJMN42tR8v58EMRPGSu0=; b=LDXdqBQKpSbAoJOF+2fHyEXwS8EfwTM1piSmdrVqCw22arQhW0oKUJFhuMlCEKXtRV 1HSSc1NhxBT+ezGePRevMhJhZi/Gvj//vnIf3ARI69JnD5SnfMFbqUUKD8mDr7xUyXJT E8XnAf08OK36YDwlFRzJIkq6Lnfyqy2GuUV3Pg4qD/d9yHSEKdoyalzT+uV1T2BBPlGv M3Q/HTDX3arZP72mqjZEcstsOsfcbMnadfzUwtfQRrbguR/0iFQgUr09CMjzxTf1SESP cvnbPo5w7u+UsaLReWE6UBx9NSzAyVDUz8DT6BHWzNZmI5sDTJASgqrLAD9hru5aDe74 ftAw== X-Gm-Message-State: AOAM532yWWxSW3krl4sD5o78j1OuKQhjT2UwmRYTX90jHeJms9fZtT+8 S5bI9VA2agqVWzrkQX2eWI77V3GAqJ6n6m0kgY/WATTodgWCALUSdvn72h9xWOUTqSVVgjNDqoq vtg64zOjmBKyTjXp/PyDfRLROa+k44pP2buiOD26RID4OiJzmeCoRNQGHeF9v X-Received: by 2002:ac8:58cb:: with SMTP id u11mr335299qta.342.1626301507162; Wed, 14 Jul 2021 15:25:07 -0700 (PDT) X-Google-Smtp-Source: ABdhPJzIZlX8JAribpNrq0wkAFbGM0ZiKXb3wEfDbm8EyndS32qRWndHNP3FXvvKZoMsl3TvmITdTw== X-Received: by 2002:ac8:58cb:: with SMTP id u11mr335254qta.342.1626301506807; Wed, 14 Jul 2021 15:25:06 -0700 (PDT) Received: from localhost.localdomain (bras-base-toroon474qw-grc-65-184-144-111-238.dsl.bell.ca. [184.144.111.238]) by smtp.gmail.com with ESMTPSA id 7sm1300668qtx.33.2021.07.14.15.25.05 (version=TLS1_3 cipher=TLS_AES_256_GCM_SHA384 bits=256/256); Wed, 14 Jul 2021 15:25:06 -0700 (PDT) From: Peter Xu To: linux-mm@kvack.org, linux-kernel@vger.kernel.org Cc: Andrew Morton , Mike Kravetz , Axel Rasmussen , Miaohe Lin , "Kirill A . Shutemov" , Hugh Dickins , Jason Gunthorpe , Alistair Popple , Matthew Wilcox , peterx@redhat.com, Jerome Glisse , Andrea Arcangeli , Mike Rapoport , Nadav Amit , David Hildenbrand Subject: [PATCH v4 18/26] hugetlb/userfaultfd: Take care of UFFDIO_COPY_MODE_WP Date: Wed, 14 Jul 2021 18:25:02 -0400 Message-Id: <20210714222502.48992-1-peterx@redhat.com> X-Mailer: git-send-email 2.31.1 In-Reply-To: <20210714222117.47648-1-peterx@redhat.com> References: <20210714222117.47648-1-peterx@redhat.com> MIME-Version: 1.0 X-Mimecast-Spam-Score: 0 X-Mimecast-Originator: redhat.com X-Rspamd-Server: rspam06 X-Rspamd-Queue-Id: C08BE10000B2 X-Stat-Signature: wy1kfkxirak67tybkqfj1u4y6xewiwqq Authentication-Results: imf12.hostedemail.com; dkim=pass header.d=redhat.com header.s=mimecast20190719 header.b=V+evOwuv; spf=none (imf12.hostedemail.com: domain of peterx@redhat.com has no SPF policy when checking 216.205.24.124) smtp.mailfrom=peterx@redhat.com; dmarc=pass (policy=none) header.from=redhat.com X-HE-Tag: 1626301509-925136 X-Bogosity: Ham, tests=bogofilter, spamicity=0.000000, version=1.2.4 Sender: owner-linux-mm@kvack.org Precedence: bulk X-Loop: owner-majordomo@kvack.org List-ID: Firstly, pass the wp_copy variable into hugetlb_mcopy_atomic_pte() thoughout the stack. Then, apply the UFFD_WP bit if UFFDIO_COPY_MODE_WP is with UFFDIO_COPY. Introduce huge_pte_mkuffd_wp() for it. Hugetlb pages are only managed by hugetlbfs, so we're safe even without setting dirty bit in the huge pte if the page is installed as read-only. However we'd better still keep the dirty bit set for a read-only UFFDIO_COPY pte (when UFFDIO_COPY_MODE_WP bit is set), not only to match what we do with shmem, but also because the page does contain dirty data that the kernel just copied from the userspace. Signed-off-by: Peter Xu --- include/linux/hugetlb.h | 6 ++++-- mm/hugetlb.c | 22 +++++++++++++++++----- mm/userfaultfd.c | 12 ++++++++---- 3 files changed, 29 insertions(+), 11 deletions(-) diff --git a/include/linux/hugetlb.h b/include/linux/hugetlb.h index c30f39815e13..fcdbf9f46d85 100644 --- a/include/linux/hugetlb.h +++ b/include/linux/hugetlb.h @@ -155,7 +155,8 @@ int hugetlb_mcopy_atomic_pte(struct mm_struct *dst_mm, pte_t *dst_pte, unsigned long dst_addr, unsigned long src_addr, enum mcopy_atomic_mode mode, - struct page **pagep); + struct page **pagep, + bool wp_copy); #endif /* CONFIG_USERFAULTFD */ bool hugetlb_reserve_pages(struct inode *inode, long from, long to, struct vm_area_struct *vma, @@ -336,7 +337,8 @@ static inline int hugetlb_mcopy_atomic_pte(struct mm_struct *dst_mm, unsigned long dst_addr, unsigned long src_addr, enum mcopy_atomic_mode mode, - struct page **pagep) + struct page **pagep, + bool wp_copy) { BUG(); return 0; diff --git a/mm/hugetlb.c b/mm/hugetlb.c index 8559b8bb7fa5..f4efcb8c6214 100644 --- a/mm/hugetlb.c +++ b/mm/hugetlb.c @@ -5141,7 +5141,8 @@ int hugetlb_mcopy_atomic_pte(struct mm_struct *dst_mm, unsigned long dst_addr, unsigned long src_addr, enum mcopy_atomic_mode mode, - struct page **pagep) + struct page **pagep, + bool wp_copy) { bool is_continue = (mode == MCOPY_ATOMIC_CONTINUE); struct hstate *h = hstate_vma(dst_vma); @@ -5277,17 +5278,28 @@ int hugetlb_mcopy_atomic_pte(struct mm_struct *dst_mm, hugepage_add_new_anon_rmap(page, dst_vma, dst_addr); } - /* For CONTINUE on a non-shared VMA, don't set VM_WRITE for CoW. */ - if (is_continue && !vm_shared) + /* + * For either: (1) CONTINUE on a non-shared VMA, or (2) UFFDIO_COPY + * with wp flag set, don't set pte write bit. + */ + if (wp_copy || (is_continue && !vm_shared)) writable = 0; else writable = dst_vma->vm_flags & VM_WRITE; _dst_pte = make_huge_pte(dst_vma, page, writable); - if (writable) - _dst_pte = huge_pte_mkdirty(_dst_pte); + /* + * Always mark UFFDIO_COPY page dirty; note that this may not be + * extremely important for hugetlbfs for now since swapping is not + * supported, but we should still be clear in that this page cannot be + * thrown away at will, even if write bit not set. + */ + _dst_pte = huge_pte_mkdirty(_dst_pte); _dst_pte = pte_mkyoung(_dst_pte); + if (wp_copy) + _dst_pte = huge_pte_mkuffd_wp(_dst_pte); + set_huge_pte_at(dst_mm, dst_addr, dst_pte, _dst_pte); (void)huge_ptep_set_access_flags(dst_vma, dst_addr, dst_pte, _dst_pte, diff --git a/mm/userfaultfd.c b/mm/userfaultfd.c index 0c7212dfb95d..501d6b9f7a5a 100644 --- a/mm/userfaultfd.c +++ b/mm/userfaultfd.c @@ -297,7 +297,8 @@ static __always_inline ssize_t __mcopy_atomic_hugetlb(struct mm_struct *dst_mm, unsigned long dst_start, unsigned long src_start, unsigned long len, - enum mcopy_atomic_mode mode) + enum mcopy_atomic_mode mode, + bool wp_copy) { int vm_shared = dst_vma->vm_flags & VM_SHARED; ssize_t err; @@ -393,7 +394,8 @@ static __always_inline ssize_t __mcopy_atomic_hugetlb(struct mm_struct *dst_mm, } err = hugetlb_mcopy_atomic_pte(dst_mm, dst_pte, dst_vma, - dst_addr, src_addr, mode, &page); + dst_addr, src_addr, mode, &page, + wp_copy); mutex_unlock(&hugetlb_fault_mutex_table[hash]); i_mmap_unlock_read(mapping); @@ -448,7 +450,8 @@ extern ssize_t __mcopy_atomic_hugetlb(struct mm_struct *dst_mm, unsigned long dst_start, unsigned long src_start, unsigned long len, - enum mcopy_atomic_mode mode); + enum mcopy_atomic_mode mode, + bool wp_copy); #endif /* CONFIG_HUGETLB_PAGE */ static __always_inline ssize_t mfill_atomic_pte(struct mm_struct *dst_mm, @@ -568,7 +571,8 @@ static __always_inline ssize_t __mcopy_atomic(struct mm_struct *dst_mm, */ if (is_vm_hugetlb_page(dst_vma)) return __mcopy_atomic_hugetlb(dst_mm, dst_vma, dst_start, - src_start, len, mcopy_mode); + src_start, len, mcopy_mode, + wp_copy); if (!vma_is_anonymous(dst_vma) && !vma_is_shmem(dst_vma)) goto out_unlock; From patchwork Wed Jul 14 22:25:07 2021 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Peter Xu X-Patchwork-Id: 12377977 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org X-Spam-Level: X-Spam-Status: No, score=-16.6 required=3.0 tests=BAYES_00,DKIM_INVALID, DKIM_SIGNED,HEADER_FROM_DIFFERENT_DOMAINS,INCLUDES_CR_TRAILER,INCLUDES_PATCH, MAILING_LIST_MULTI,SPF_HELO_NONE,USER_AGENT_GIT autolearn=ham autolearn_force=no version=3.4.0 Received: from mail.kernel.org (mail.kernel.org [198.145.29.99]) by smtp.lore.kernel.org (Postfix) with ESMTP id 4E88CC1B08C for ; Wed, 14 Jul 2021 22:25:15 +0000 (UTC) Received: from kanga.kvack.org (kanga.kvack.org [205.233.56.17]) by mail.kernel.org (Postfix) with ESMTP id 0E1F7613C9 for ; Wed, 14 Jul 2021 22:25:15 +0000 (UTC) DMARC-Filter: OpenDMARC Filter v1.3.2 mail.kernel.org 0E1F7613C9 Authentication-Results: mail.kernel.org; dmarc=fail (p=none dis=none) header.from=redhat.com Authentication-Results: mail.kernel.org; spf=pass smtp.mailfrom=owner-linux-mm@kvack.org Received: by kanga.kvack.org (Postfix) id 5A91B6B00CE; Wed, 14 Jul 2021 18:25:15 -0400 (EDT) Received: by kanga.kvack.org (Postfix, from userid 40) id 57E686B00CF; Wed, 14 Jul 2021 18:25:15 -0400 (EDT) X-Delivered-To: int-list-linux-mm@kvack.org Received: by kanga.kvack.org (Postfix, from userid 63042) id 3D19B6B00D0; Wed, 14 Jul 2021 18:25:15 -0400 (EDT) X-Delivered-To: linux-mm@kvack.org Received: from forelay.hostedemail.com (smtprelay0152.hostedemail.com [216.40.44.152]) by kanga.kvack.org (Postfix) with ESMTP id 197FE6B00CE for ; Wed, 14 Jul 2021 18:25:15 -0400 (EDT) Received: from smtpin01.hostedemail.com (10.5.19.251.rfc1918.com [10.5.19.251]) by forelay04.hostedemail.com (Postfix) with ESMTP id 0D9431CB18 for ; Wed, 14 Jul 2021 22:25:14 +0000 (UTC) X-FDA: 78362625348.01.2B710CC Received: from us-smtp-delivery-124.mimecast.com (us-smtp-delivery-124.mimecast.com [216.205.24.124]) by imf17.hostedemail.com (Postfix) with ESMTP id A9EB2F00039A for ; Wed, 14 Jul 2021 22:25:13 +0000 (UTC) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=redhat.com; s=mimecast20190719; t=1626301513; h=from:from:reply-to:subject:subject:date:date:message-id:message-id: to:to:cc:cc:mime-version:mime-version:content-type:content-type: content-transfer-encoding:content-transfer-encoding: in-reply-to:in-reply-to:references:references; bh=6xF0M7X1DsWIfOb+PL4bMVWgDi6eXN7r5W8R38MPpTs=; b=QSks2AbXV/+DK9uSKgRg+jt3vHfy5xRjJLIbjyhvUlFNvZ1grgUgbxH0+XSqCAO6iy3WAj 3GXoRPXAn3G4Z039VR50r8AZ7SV6WDt+K4aBGijRT3eHrQxzYuHqX/Rso6vBP+oPeQqvGr fgA+citw7CtK5oE3oahkAZd8b1474g4= Received: from mail-qk1-f199.google.com (mail-qk1-f199.google.com [209.85.222.199]) (Using TLS) by relay.mimecast.com with ESMTP id us-mta-231-UHqc8VYoOua97aK4bl5Oog-1; Wed, 14 Jul 2021 18:25:12 -0400 X-MC-Unique: UHqc8VYoOua97aK4bl5Oog-1 Received: by mail-qk1-f199.google.com with SMTP id r190-20020a375dc70000b02903acea04c19fso2317066qkb.8 for ; Wed, 14 Jul 2021 15:25:12 -0700 (PDT) X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20161025; h=x-gm-message-state:from:to:cc:subject:date:message-id:in-reply-to :references:mime-version:content-transfer-encoding; bh=6xF0M7X1DsWIfOb+PL4bMVWgDi6eXN7r5W8R38MPpTs=; b=sNeKpv+TI0jcJfsuNVvL1OHWMuw+kbiVRESmHbOeJwz+Jek8fGWqbLyoHRLuaYP6Vz Pcf85TuqaWyR6JEkHIcZnlmd/gmTPUc1meYTxlZi5aG9o89q3w4461H7kObj6yPj8djv LZU4GQ219d1YNCIgJYeCZm81DYls4m8nQk/XWR5ORpe6OzKJ3BNk4dF6NhvdKCs983O8 d5oPoJlPOhnFAdt50feSlZDNjx9mBVU1h2+PK153Z3JoGSgYd2Jnr/leLpTXUTSZEbod cjDPC3OSirG0tILM5XRjY6BeGFFr7UfEikCjxCOxyT5kwtrnbJoWfn8wtvOQPlEtCkLo 5blQ== X-Gm-Message-State: AOAM530hGUVJxGfsfQzWgwd8WFS5k1V1yuwzrFZVge3UjNuj/sRCt5cC G0rOwd0wn1mUBkEEl7KZSPuSht/eyqL8LAnoMefFqZcBMKHNJo3tzHhYwvDk57XF+wIlRNZLDtj rFd4EyClxFR6iOkpANCN8kg+rw3X6OK9yM9FEgrDV8/Z+qPJp7WzH92M1mRSj X-Received: by 2002:a05:620a:21da:: with SMTP id h26mr234719qka.364.1626301511163; Wed, 14 Jul 2021 15:25:11 -0700 (PDT) X-Google-Smtp-Source: ABdhPJyAzfpIJGP3b/Bp6JWursq5waaz3pJsfn/nHxprQEqq4SvYcLJVducew1EhJJTmsYUN8IEusQ== X-Received: by 2002:a05:620a:21da:: with SMTP id h26mr234684qka.364.1626301510880; Wed, 14 Jul 2021 15:25:10 -0700 (PDT) Received: from localhost.localdomain (bras-base-toroon474qw-grc-65-184-144-111-238.dsl.bell.ca. [184.144.111.238]) by smtp.gmail.com with ESMTPSA id i123sm16830qkf.60.2021.07.14.15.25.09 (version=TLS1_3 cipher=TLS_AES_256_GCM_SHA384 bits=256/256); Wed, 14 Jul 2021 15:25:10 -0700 (PDT) From: Peter Xu To: linux-mm@kvack.org, linux-kernel@vger.kernel.org Cc: Andrew Morton , Mike Kravetz , Axel Rasmussen , Miaohe Lin , "Kirill A . Shutemov" , Hugh Dickins , Jason Gunthorpe , Alistair Popple , Matthew Wilcox , peterx@redhat.com, Jerome Glisse , Andrea Arcangeli , Mike Rapoport , Nadav Amit , David Hildenbrand Subject: [PATCH v4 19/26] hugetlb/userfaultfd: Handle UFFDIO_WRITEPROTECT Date: Wed, 14 Jul 2021 18:25:07 -0400 Message-Id: <20210714222507.49042-1-peterx@redhat.com> X-Mailer: git-send-email 2.31.1 In-Reply-To: <20210714222117.47648-1-peterx@redhat.com> References: <20210714222117.47648-1-peterx@redhat.com> MIME-Version: 1.0 X-Mimecast-Spam-Score: 0 X-Mimecast-Originator: redhat.com X-Rspamd-Server: rspam06 X-Rspamd-Queue-Id: A9EB2F00039A X-Stat-Signature: 3z7qeqiiitznwrqjh8i5q8keth77yd7f Authentication-Results: imf17.hostedemail.com; dkim=pass header.d=redhat.com header.s=mimecast20190719 header.b=QSks2AbX; spf=none (imf17.hostedemail.com: domain of peterx@redhat.com has no SPF policy when checking 216.205.24.124) smtp.mailfrom=peterx@redhat.com; dmarc=pass (policy=none) header.from=redhat.com X-HE-Tag: 1626301513-815718 X-Bogosity: Ham, tests=bogofilter, spamicity=0.000000, version=1.2.4 Sender: owner-linux-mm@kvack.org Precedence: bulk X-Loop: owner-majordomo@kvack.org List-ID: This starts from passing cp_flags into hugetlb_change_protection() so hugetlb will be able to handle MM_CP_UFFD_WP[_RESOLVE] requests. huge_pte_clear_uffd_wp() is introduced to handle the case where the UFFDIO_WRITEPROTECT is requested upon migrating huge page entries. Reviewed-by: Mike Kravetz Signed-off-by: Peter Xu --- include/linux/hugetlb.h | 6 ++++-- mm/hugetlb.c | 13 ++++++++++++- mm/mprotect.c | 3 ++- mm/userfaultfd.c | 8 ++++++++ 4 files changed, 26 insertions(+), 4 deletions(-) diff --git a/include/linux/hugetlb.h b/include/linux/hugetlb.h index fcdbf9f46d85..e19ca363803d 100644 --- a/include/linux/hugetlb.h +++ b/include/linux/hugetlb.h @@ -205,7 +205,8 @@ struct page *follow_huge_pgd(struct mm_struct *mm, unsigned long address, int pmd_huge(pmd_t pmd); int pud_huge(pud_t pud); unsigned long hugetlb_change_protection(struct vm_area_struct *vma, - unsigned long address, unsigned long end, pgprot_t newprot); + unsigned long address, unsigned long end, pgprot_t newprot, + unsigned long cp_flags); bool is_hugetlb_entry_migration(pte_t pte); void hugetlb_unshare_all_pmds(struct vm_area_struct *vma); @@ -372,7 +373,8 @@ static inline void move_hugetlb_state(struct page *oldpage, static inline unsigned long hugetlb_change_protection( struct vm_area_struct *vma, unsigned long address, - unsigned long end, pgprot_t newprot) + unsigned long end, pgprot_t newprot, + unsigned long cp_flags) { return 0; } diff --git a/mm/hugetlb.c b/mm/hugetlb.c index f4efcb8c6214..7d3558265b6f 100644 --- a/mm/hugetlb.c +++ b/mm/hugetlb.c @@ -5519,7 +5519,8 @@ long follow_hugetlb_page(struct mm_struct *mm, struct vm_area_struct *vma, } unsigned long hugetlb_change_protection(struct vm_area_struct *vma, - unsigned long address, unsigned long end, pgprot_t newprot) + unsigned long address, unsigned long end, + pgprot_t newprot, unsigned long cp_flags) { struct mm_struct *mm = vma->vm_mm; unsigned long start = address; @@ -5529,6 +5530,8 @@ unsigned long hugetlb_change_protection(struct vm_area_struct *vma, unsigned long pages = 0; bool shared_pmd = false; struct mmu_notifier_range range; + bool uffd_wp = cp_flags & MM_CP_UFFD_WP; + bool uffd_wp_resolve = cp_flags & MM_CP_UFFD_WP_RESOLVE; /* * In the case of shared PMDs, the area to flush could be beyond @@ -5570,6 +5573,10 @@ unsigned long hugetlb_change_protection(struct vm_area_struct *vma, entry = make_readable_migration_entry( swp_offset(entry)); newpte = swp_entry_to_pte(entry); + if (uffd_wp) + newpte = pte_swp_mkuffd_wp(newpte); + else if (uffd_wp_resolve) + newpte = pte_swp_clear_uffd_wp(newpte); set_huge_swap_pte_at(mm, address, ptep, newpte, huge_page_size(h)); pages++; @@ -5584,6 +5591,10 @@ unsigned long hugetlb_change_protection(struct vm_area_struct *vma, old_pte = huge_ptep_modify_prot_start(vma, address, ptep); pte = pte_mkhuge(huge_pte_modify(old_pte, newprot)); pte = arch_make_huge_pte(pte, shift, vma->vm_flags); + if (uffd_wp) + pte = huge_pte_mkuffd_wp(huge_pte_wrprotect(pte)); + else if (uffd_wp_resolve) + pte = huge_pte_clear_uffd_wp(pte); huge_ptep_modify_prot_commit(vma, address, ptep, old_pte, pte); pages++; } diff --git a/mm/mprotect.c b/mm/mprotect.c index 3fcb87b59696..96f4df023439 100644 --- a/mm/mprotect.c +++ b/mm/mprotect.c @@ -426,7 +426,8 @@ unsigned long change_protection(struct vm_area_struct *vma, unsigned long start, BUG_ON((cp_flags & MM_CP_UFFD_WP_ALL) == MM_CP_UFFD_WP_ALL); if (is_vm_hugetlb_page(vma)) - pages = hugetlb_change_protection(vma, start, end, newprot); + pages = hugetlb_change_protection(vma, start, end, newprot, + cp_flags); else pages = change_protection_range(vma, start, end, newprot, cp_flags); diff --git a/mm/userfaultfd.c b/mm/userfaultfd.c index 501d6b9f7a5a..7ba721aca1c5 100644 --- a/mm/userfaultfd.c +++ b/mm/userfaultfd.c @@ -695,6 +695,7 @@ int mwriteprotect_range(struct mm_struct *dst_mm, unsigned long start, unsigned long len, bool enable_wp, bool *mmap_changing) { struct vm_area_struct *dst_vma; + unsigned long page_mask; pgprot_t newprot; int err; @@ -731,6 +732,13 @@ int mwriteprotect_range(struct mm_struct *dst_mm, unsigned long start, if (!vma_is_anonymous(dst_vma)) goto out_unlock; + if (is_vm_hugetlb_page(dst_vma)) { + err = -EINVAL; + page_mask = vma_kernel_pagesize(dst_vma) - 1; + if ((start & page_mask) || (len & page_mask)) + goto out_unlock; + } + if (enable_wp) newprot = vm_get_page_prot(dst_vma->vm_flags & ~(VM_WRITE)); else From patchwork Wed Jul 14 22:25:11 2021 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Peter Xu X-Patchwork-Id: 12377979 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org X-Spam-Level: X-Spam-Status: No, score=-16.6 required=3.0 tests=BAYES_00,DKIM_INVALID, DKIM_SIGNED,HEADER_FROM_DIFFERENT_DOMAINS,INCLUDES_CR_TRAILER,INCLUDES_PATCH, MAILING_LIST_MULTI,SPF_HELO_NONE,USER_AGENT_GIT autolearn=ham autolearn_force=no version=3.4.0 Received: from mail.kernel.org (mail.kernel.org [198.145.29.99]) by smtp.lore.kernel.org (Postfix) with ESMTP id 4A971C47E4E for ; Wed, 14 Jul 2021 22:25:19 +0000 (UTC) Received: from kanga.kvack.org (kanga.kvack.org [205.233.56.17]) by mail.kernel.org (Postfix) with ESMTP id F29B2613C9 for ; Wed, 14 Jul 2021 22:25:18 +0000 (UTC) DMARC-Filter: OpenDMARC Filter v1.3.2 mail.kernel.org F29B2613C9 Authentication-Results: mail.kernel.org; dmarc=fail (p=none dis=none) header.from=redhat.com Authentication-Results: mail.kernel.org; spf=pass smtp.mailfrom=owner-linux-mm@kvack.org Received: by kanga.kvack.org (Postfix) id 50B976B00D0; Wed, 14 Jul 2021 18:25:19 -0400 (EDT) Received: by kanga.kvack.org (Postfix, from userid 40) id 4E36B6B00D1; Wed, 14 Jul 2021 18:25:19 -0400 (EDT) X-Delivered-To: int-list-linux-mm@kvack.org Received: by kanga.kvack.org (Postfix, from userid 63042) id 35C416B00D2; Wed, 14 Jul 2021 18:25:19 -0400 (EDT) X-Delivered-To: linux-mm@kvack.org Received: from forelay.hostedemail.com (smtprelay0024.hostedemail.com [216.40.44.24]) by kanga.kvack.org (Postfix) with ESMTP id 138FB6B00D0 for ; Wed, 14 Jul 2021 18:25:19 -0400 (EDT) Received: from smtpin22.hostedemail.com (10.5.19.251.rfc1918.com [10.5.19.251]) by forelay04.hostedemail.com (Postfix) with ESMTP id 0DC651C943 for ; Wed, 14 Jul 2021 22:25:18 +0000 (UTC) X-FDA: 78362625516.22.0B7EFFF Received: from us-smtp-delivery-124.mimecast.com (us-smtp-delivery-124.mimecast.com [170.10.133.124]) by imf29.hostedemail.com (Postfix) with ESMTP id A31A19000253 for ; Wed, 14 Jul 2021 22:25:17 +0000 (UTC) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=redhat.com; s=mimecast20190719; t=1626301517; h=from:from:reply-to:subject:subject:date:date:message-id:message-id: to:to:cc:cc:mime-version:mime-version:content-type:content-type: content-transfer-encoding:content-transfer-encoding: in-reply-to:in-reply-to:references:references; bh=FG94X/udzrS/4H8a9ROacZylO2qMwKMDk77nG92mz5U=; b=Kpyj3P1g1XHu52rO2ZEgbk6oAov94oPE1JsekJ3m9Lb1oSXkpSIgJ9mec6WNqkEErto2kY fH+GpgQV1pBRdutcfphQv/3g31OBWQrWNljaBbyWZx+wxNpsUydOSM9raHq70HBwimiBPH UuGuu+NV3l1bPIS5j7T0KwaW3Hxfhc0= Received: from mail-qk1-f197.google.com (mail-qk1-f197.google.com [209.85.222.197]) (Using TLS) by relay.mimecast.com with ESMTP id us-mta-516-40SMe_o_N0WjjftH6w5EYg-1; Wed, 14 Jul 2021 18:25:16 -0400 X-MC-Unique: 40SMe_o_N0WjjftH6w5EYg-1 Received: by mail-qk1-f197.google.com with SMTP id j9-20020a05620a0009b02903b770762a3cso2280434qki.17 for ; Wed, 14 Jul 2021 15:25:16 -0700 (PDT) X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20161025; h=x-gm-message-state:from:to:cc:subject:date:message-id:in-reply-to :references:mime-version:content-transfer-encoding; bh=FG94X/udzrS/4H8a9ROacZylO2qMwKMDk77nG92mz5U=; b=ra+gV6qPCb3G44LE6QuBvCcB0Czk3lhpwT/BYCGBYeWuG+NHimq4Xo3EYlC0gXIIi7 vtmgqteg267X2d3KKvqsy7sNULql89CKZnU6hEEkKFJUU493fA7GLIlFJeGZsnJNQYqh pIaJEbNPN0WzJAat7sV22BclkNl6mH9W7ueh8ZmWSv9xBK2n2Qlb94mdGveN0DlIPC+L w8zfBABTN9x4mEo7BUOqE9CHHFSeeNciLt4b81rYnMjVSXGanFoK/lLB0FrUT91eMBe6 o49RyE6whIKycdsSybS0EWrdsmDgH0MMmEYr9cOk501erwyErUvRDScb6fW53o/c1u2s IQ4g== X-Gm-Message-State: AOAM532EnRCZzO7eNhAd2yj0YGsUWM1ieqNTtR6g4I1zRQ3XV2icrdWh Fl4dHD96gFMzJIYDL+tO00rDr5fxn8MOUg3zuh3YL0loHaSC/ntXj8G5cw5vEBdY8ydjzKWa8Rd cA4gzYsgwoB2X3B8hS/rMR1tfhEt/J1i7aDBAQWx0NWYvVe1I8J+D0O30lMdF X-Received: by 2002:ac8:4e95:: with SMTP id 21mr342292qtp.234.1626301515557; Wed, 14 Jul 2021 15:25:15 -0700 (PDT) X-Google-Smtp-Source: ABdhPJyXR+Vu/+o0F5Ebn+71mCKuvUYN3/uYB7J35r5bAygeXC05i/PyQzKZDPriZMgwiauQ0WseFw== X-Received: by 2002:ac8:4e95:: with SMTP id 21mr342258qtp.234.1626301515288; Wed, 14 Jul 2021 15:25:15 -0700 (PDT) Received: from localhost.localdomain (bras-base-toroon474qw-grc-65-184-144-111-238.dsl.bell.ca. [184.144.111.238]) by smtp.gmail.com with ESMTPSA id f62sm1662862qke.135.2021.07.14.15.25.13 (version=TLS1_3 cipher=TLS_AES_256_GCM_SHA384 bits=256/256); Wed, 14 Jul 2021 15:25:14 -0700 (PDT) From: Peter Xu To: linux-mm@kvack.org, linux-kernel@vger.kernel.org Cc: Andrew Morton , Mike Kravetz , Axel Rasmussen , Miaohe Lin , "Kirill A . Shutemov" , Hugh Dickins , Jason Gunthorpe , Alistair Popple , Matthew Wilcox , peterx@redhat.com, Jerome Glisse , Andrea Arcangeli , Mike Rapoport , Nadav Amit , David Hildenbrand Subject: [PATCH v4 20/26] mm/hugetlb: Introduce huge version of special swap pte helpers Date: Wed, 14 Jul 2021 18:25:11 -0400 Message-Id: <20210714222511.49094-1-peterx@redhat.com> X-Mailer: git-send-email 2.31.1 In-Reply-To: <20210714222117.47648-1-peterx@redhat.com> References: <20210714222117.47648-1-peterx@redhat.com> MIME-Version: 1.0 X-Mimecast-Spam-Score: 0 X-Mimecast-Originator: redhat.com Authentication-Results: imf29.hostedemail.com; dkim=pass header.d=redhat.com header.s=mimecast20190719 header.b=Kpyj3P1g; spf=none (imf29.hostedemail.com: domain of peterx@redhat.com has no SPF policy when checking 170.10.133.124) smtp.mailfrom=peterx@redhat.com; dmarc=pass (policy=none) header.from=redhat.com X-Rspamd-Server: rspam03 X-Rspamd-Queue-Id: A31A19000253 X-Stat-Signature: whijrydh1bfp9krrnd1nxi3jasnzj3gd X-HE-Tag: 1626301517-32978 X-Bogosity: Ham, tests=bogofilter, spamicity=0.000000, version=1.2.4 Sender: owner-linux-mm@kvack.org Precedence: bulk X-Loop: owner-majordomo@kvack.org List-ID: This is to let hugetlbfs be prepared to also recognize swap special ptes just like uffd-wp special swap ptes. Reviewed-by: Mike Kravetz Signed-off-by: Peter Xu --- mm/hugetlb.c | 24 ++++++++++++++++++++++-- 1 file changed, 22 insertions(+), 2 deletions(-) diff --git a/mm/hugetlb.c b/mm/hugetlb.c index 7d3558265b6f..6a66b598245e 100644 --- a/mm/hugetlb.c +++ b/mm/hugetlb.c @@ -95,6 +95,26 @@ static inline bool subpool_is_free(struct hugepage_subpool *spool) return true; } +/* + * These are sister versions of is_swap_pte() and pte_has_swap_entry(). We + * need standalone ones because huge_pte_none() is handled differently from + * pte_none(). For more information, please refer to comments above + * is_swap_pte() and pte_has_swap_entry(). + * + * Here we directly reuse the pte level of swap special ptes, for example, the + * pte_swp_uffd_wp_special(). It just stands for a huge page rather than a + * small page for hugetlbfs pages. + */ +static inline bool is_huge_swap_pte(pte_t pte) +{ + return !huge_pte_none(pte) && !pte_present(pte); +} + +static inline bool huge_pte_has_swap_entry(pte_t pte) +{ + return is_huge_swap_pte(pte) && !is_swap_special_pte(pte); +} + static inline void unlock_or_release_subpool(struct hugepage_subpool *spool, unsigned long irq_flags) { @@ -4138,7 +4158,7 @@ bool is_hugetlb_entry_migration(pte_t pte) { swp_entry_t swp; - if (huge_pte_none(pte) || pte_present(pte)) + if (!huge_pte_has_swap_entry(pte)) return false; swp = pte_to_swp_entry(pte); if (is_migration_entry(swp)) @@ -4151,7 +4171,7 @@ static bool is_hugetlb_entry_hwpoisoned(pte_t pte) { swp_entry_t swp; - if (huge_pte_none(pte) || pte_present(pte)) + if (!huge_pte_has_swap_entry(pte)) return false; swp = pte_to_swp_entry(pte); if (is_hwpoison_entry(swp)) From patchwork Wed Jul 14 22:25:15 2021 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Peter Xu X-Patchwork-Id: 12377981 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org X-Spam-Level: X-Spam-Status: No, score=-16.6 required=3.0 tests=BAYES_00,DKIM_INVALID, DKIM_SIGNED,HEADER_FROM_DIFFERENT_DOMAINS,INCLUDES_CR_TRAILER,INCLUDES_PATCH, MAILING_LIST_MULTI,SPF_HELO_NONE,USER_AGENT_GIT autolearn=ham autolearn_force=no version=3.4.0 Received: from mail.kernel.org (mail.kernel.org [198.145.29.99]) by smtp.lore.kernel.org (Postfix) with ESMTP id 9D377C47E48 for ; Wed, 14 Jul 2021 22:25:23 +0000 (UTC) Received: from kanga.kvack.org (kanga.kvack.org [205.233.56.17]) by mail.kernel.org (Postfix) with ESMTP id 4DE70613D1 for ; Wed, 14 Jul 2021 22:25:23 +0000 (UTC) DMARC-Filter: OpenDMARC Filter v1.3.2 mail.kernel.org 4DE70613D1 Authentication-Results: mail.kernel.org; dmarc=fail (p=none dis=none) header.from=redhat.com Authentication-Results: mail.kernel.org; spf=pass smtp.mailfrom=owner-linux-mm@kvack.org Received: by kanga.kvack.org (Postfix) id 9C6566B00D2; Wed, 14 Jul 2021 18:25:23 -0400 (EDT) Received: by kanga.kvack.org (Postfix, from userid 40) id 99D1C6B00D3; Wed, 14 Jul 2021 18:25:23 -0400 (EDT) X-Delivered-To: int-list-linux-mm@kvack.org Received: by kanga.kvack.org (Postfix, from userid 63042) id 7F0C86B00D4; Wed, 14 Jul 2021 18:25:23 -0400 (EDT) X-Delivered-To: linux-mm@kvack.org Received: from forelay.hostedemail.com (smtprelay0178.hostedemail.com [216.40.44.178]) by kanga.kvack.org (Postfix) with ESMTP id 5A9D66B00D2 for ; Wed, 14 Jul 2021 18:25:23 -0400 (EDT) Received: from smtpin06.hostedemail.com (10.5.19.251.rfc1918.com [10.5.19.251]) by forelay01.hostedemail.com (Postfix) with ESMTP id 4651918479C79 for ; Wed, 14 Jul 2021 22:25:22 +0000 (UTC) X-FDA: 78362625684.06.9F3C62A Received: from us-smtp-delivery-124.mimecast.com (us-smtp-delivery-124.mimecast.com [216.205.24.124]) by imf15.hostedemail.com (Postfix) with ESMTP id DB673D0000B8 for ; Wed, 14 Jul 2021 22:25:21 +0000 (UTC) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=redhat.com; s=mimecast20190719; t=1626301521; h=from:from:reply-to:subject:subject:date:date:message-id:message-id: to:to:cc:cc:mime-version:mime-version:content-type:content-type: content-transfer-encoding:content-transfer-encoding: in-reply-to:in-reply-to:references:references; bh=5H2OJLKid4CPQSAYIKsaabG8iASKh4992R4GxsoqMUU=; b=LeqpfBzTX8/jBzFX7dXKOmQ3ApS8qa9f5P6cL7nFfGiMDFuTrRcvJR53DVtBilWASn7K+c 9zR3+dlkLrPwwGZ1sXkl4WBgvEYjob8KxKiXRhSEcCZvXWKPXmt6KfW4QjOU/clvQr48vM jcOcw7ly8KY2tOj9N6wHny8fo8HBQDo= Received: from mail-qv1-f71.google.com (mail-qv1-f71.google.com [209.85.219.71]) (Using TLS) by relay.mimecast.com with ESMTP id us-mta-595-7M86jNZEPh6qrSPZpLmtTQ-1; Wed, 14 Jul 2021 18:25:20 -0400 X-MC-Unique: 7M86jNZEPh6qrSPZpLmtTQ-1 Received: by mail-qv1-f71.google.com with SMTP id k12-20020a0cfd6c0000b029020df9543019so2686016qvs.14 for ; Wed, 14 Jul 2021 15:25:20 -0700 (PDT) X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20161025; h=x-gm-message-state:from:to:cc:subject:date:message-id:in-reply-to :references:mime-version:content-transfer-encoding; bh=5H2OJLKid4CPQSAYIKsaabG8iASKh4992R4GxsoqMUU=; b=hvkKacfONi2RrkmuytJR97byPrXmvGa4qzKUtp5KE1MkhhGG7HTvV/UtoNFlKRDOIs y0IT7Yum5N/LXIIiMUAAFC4t3PifM+2PzIabcZ0eYQLdlK+zaS1Oy/KysBGoyslyJa0S tYoC8dGQL0XN/VgSUDOqKOmJjjOly66n9TDUGvluXUb6ddqq2bWsBDyDL8wbkm/Jbhvh lQn0SSMtVSpdogGQUK00tuU9TyNtMFn7+nEMrUZx9CLPlxk7ykpVZt7hi/hTM9J/ADFp 8jz33qByEr7f6snT6QlJ3e05qDjEzGrr3RaLmmMJ53VXTzIHz6hFiTKwYVMEo+adfGnO szWg== X-Gm-Message-State: AOAM533+bqtdHVTQSZD1jvUzRAhQGuWo6t69tCi7U37k8+IsHnlYnxd5 GJlIjHdVvrGTZRws5tAWi3YCQwQbgO3A7Id2EFAgWTTOPX8qphKBbVS+4jvfX0uQwtHJ9k9LCPf GMdxNrGc69prMLA4G//uVv5Fo0MOoAl/6MAUPD5IGJgZdH1LhOANL6Popi/YP X-Received: by 2002:ad4:45c6:: with SMTP id v6mr417661qvt.38.1626301519630; Wed, 14 Jul 2021 15:25:19 -0700 (PDT) X-Google-Smtp-Source: ABdhPJxDsfUj8eHJSSZFsQqs5KUYC+Cg/+WCkNyKzSsO7dLgYBKhrd4DqJafrAOU4DH6gx9z1hUVsA== X-Received: by 2002:ad4:45c6:: with SMTP id v6mr417628qvt.38.1626301519369; Wed, 14 Jul 2021 15:25:19 -0700 (PDT) Received: from localhost.localdomain (bras-base-toroon474qw-grc-65-184-144-111-238.dsl.bell.ca. [184.144.111.238]) by smtp.gmail.com with ESMTPSA id s202sm1671235qke.80.2021.07.14.15.25.18 (version=TLS1_3 cipher=TLS_AES_256_GCM_SHA384 bits=256/256); Wed, 14 Jul 2021 15:25:18 -0700 (PDT) From: Peter Xu To: linux-mm@kvack.org, linux-kernel@vger.kernel.org Cc: Andrew Morton , Mike Kravetz , Axel Rasmussen , Miaohe Lin , "Kirill A . Shutemov" , Hugh Dickins , Jason Gunthorpe , Alistair Popple , Matthew Wilcox , peterx@redhat.com, Jerome Glisse , Andrea Arcangeli , Mike Rapoport , Nadav Amit , David Hildenbrand Subject: [PATCH v4 21/26] hugetlb/userfaultfd: Handle uffd-wp special pte in hugetlb pf handler Date: Wed, 14 Jul 2021 18:25:15 -0400 Message-Id: <20210714222515.49144-1-peterx@redhat.com> X-Mailer: git-send-email 2.31.1 In-Reply-To: <20210714222117.47648-1-peterx@redhat.com> References: <20210714222117.47648-1-peterx@redhat.com> MIME-Version: 1.0 X-Mimecast-Spam-Score: 0 X-Mimecast-Originator: redhat.com Authentication-Results: imf15.hostedemail.com; dkim=pass header.d=redhat.com header.s=mimecast20190719 header.b=LeqpfBzT; spf=none (imf15.hostedemail.com: domain of peterx@redhat.com has no SPF policy when checking 216.205.24.124) smtp.mailfrom=peterx@redhat.com; dmarc=pass (policy=none) header.from=redhat.com X-Rspamd-Server: rspam03 X-Rspamd-Queue-Id: DB673D0000B8 X-Stat-Signature: mk33haups6rfpekbsiwd1ay7dh9rjg63 X-HE-Tag: 1626301521-310581 X-Bogosity: Ham, tests=bogofilter, spamicity=0.000000, version=1.2.4 Sender: owner-linux-mm@kvack.org Precedence: bulk X-Loop: owner-majordomo@kvack.org List-ID: Teach the hugetlb page fault code to understand uffd-wp special pte. For example, when seeing such a pte we need to convert any write fault into a read one (which is fake - we'll retry the write later if so). Meanwhile, for handle_userfault() we'll need to make sure we must wait for the special swap pte too just like a none pte. Note that we also need to teach UFFDIO_COPY about this special pte across the code path so that we can safely install a new page at this special pte as long as we know it's a stall entry. Signed-off-by: Peter Xu --- fs/userfaultfd.c | 5 ++++- mm/hugetlb.c | 26 ++++++++++++++++++++------ mm/userfaultfd.c | 5 ++++- 3 files changed, 28 insertions(+), 8 deletions(-) diff --git a/fs/userfaultfd.c b/fs/userfaultfd.c index e1c1cbc7bcc8..644df737fbb2 100644 --- a/fs/userfaultfd.c +++ b/fs/userfaultfd.c @@ -245,8 +245,11 @@ static inline bool userfaultfd_huge_must_wait(struct userfaultfd_ctx *ctx, /* * Lockless access: we're in a wait_event so it's ok if it * changes under us. + * + * Regarding uffd-wp special case, please refer to comments in + * userfaultfd_must_wait(). */ - if (huge_pte_none(pte)) + if (huge_pte_none(pte) || pte_swp_uffd_wp_special(pte)) ret = true; if (!huge_pte_write(pte) && (reason & VM_UFFD_WP)) ret = true; diff --git a/mm/hugetlb.c b/mm/hugetlb.c index 6a66b598245e..15e5de480cf0 100644 --- a/mm/hugetlb.c +++ b/mm/hugetlb.c @@ -4786,7 +4786,8 @@ static inline vm_fault_t hugetlb_handle_userfault(struct vm_area_struct *vma, static vm_fault_t hugetlb_no_page(struct mm_struct *mm, struct vm_area_struct *vma, struct address_space *mapping, pgoff_t idx, - unsigned long address, pte_t *ptep, unsigned int flags) + unsigned long address, pte_t *ptep, + pte_t old_pte, unsigned int flags) { struct hstate *h = hstate_vma(vma); vm_fault_t ret = VM_FAULT_SIGBUS; @@ -4910,7 +4911,7 @@ static vm_fault_t hugetlb_no_page(struct mm_struct *mm, ptl = huge_pte_lock(h, mm, ptep); ret = 0; - if (!huge_pte_none(huge_ptep_get(ptep))) + if (!pte_same(huge_ptep_get(ptep), old_pte)) goto backout; if (anon_rmap) { @@ -4920,6 +4921,12 @@ static vm_fault_t hugetlb_no_page(struct mm_struct *mm, page_dup_rmap(page, true); new_pte = make_huge_pte(vma, page, ((vma->vm_flags & VM_WRITE) && (vma->vm_flags & VM_SHARED))); + /* + * If this pte was previously wr-protected, keep it wr-protected even + * if populated. + */ + if (unlikely(pte_swp_uffd_wp_special(old_pte))) + new_pte = huge_pte_wrprotect(huge_pte_mkuffd_wp(new_pte)); set_huge_pte_at(mm, haddr, ptep, new_pte); hugetlb_count_add(pages_per_huge_page(h), mm); @@ -5035,8 +5042,13 @@ vm_fault_t hugetlb_fault(struct mm_struct *mm, struct vm_area_struct *vma, mutex_lock(&hugetlb_fault_mutex_table[hash]); entry = huge_ptep_get(ptep); - if (huge_pte_none(entry)) { - ret = hugetlb_no_page(mm, vma, mapping, idx, address, ptep, flags); + /* + * uffd-wp-special should be handled merely the same as pte none + * because it's basically a none pte with a special marker + */ + if (huge_pte_none(entry) || pte_swp_uffd_wp_special(entry)) { + ret = hugetlb_no_page(mm, vma, mapping, idx, address, ptep, + entry, flags); goto out_mutex; } @@ -5170,7 +5182,7 @@ int hugetlb_mcopy_atomic_pte(struct mm_struct *dst_mm, pgoff_t idx = vma_hugecache_offset(h, dst_vma, dst_addr); unsigned long size; int vm_shared = dst_vma->vm_flags & VM_SHARED; - pte_t _dst_pte; + pte_t _dst_pte, cur_pte; spinlock_t *ptl; int ret = -ENOMEM; struct page *page; @@ -5287,8 +5299,10 @@ int hugetlb_mcopy_atomic_pte(struct mm_struct *dst_mm, if (idx >= size) goto out_release_unlock; + cur_pte = huge_ptep_get(dst_pte); ret = -EEXIST; - if (!huge_pte_none(huge_ptep_get(dst_pte))) + /* Please refer to shmem_mfill_atomic_pte() for uffd-wp special case */ + if (!huge_pte_none(cur_pte) && !pte_swp_uffd_wp_special(cur_pte)) goto out_release_unlock; if (vm_shared) { diff --git a/mm/userfaultfd.c b/mm/userfaultfd.c index 7ba721aca1c5..a8038903effd 100644 --- a/mm/userfaultfd.c +++ b/mm/userfaultfd.c @@ -363,6 +363,8 @@ static __always_inline ssize_t __mcopy_atomic_hugetlb(struct mm_struct *dst_mm, } while (src_addr < src_start + len) { + pte_t pteval; + BUG_ON(dst_addr >= dst_start + len); /* @@ -385,8 +387,9 @@ static __always_inline ssize_t __mcopy_atomic_hugetlb(struct mm_struct *dst_mm, goto out_unlock; } + pteval = huge_ptep_get(dst_pte); if (mode != MCOPY_ATOMIC_CONTINUE && - !huge_pte_none(huge_ptep_get(dst_pte))) { + !huge_pte_none(pteval) && !pte_swp_uffd_wp_special(pteval)) { err = -EEXIST; mutex_unlock(&hugetlb_fault_mutex_table[hash]); i_mmap_unlock_read(mapping); From patchwork Wed Jul 14 22:25:19 2021 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Peter Xu X-Patchwork-Id: 12377983 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org X-Spam-Level: X-Spam-Status: No, score=-16.6 required=3.0 tests=BAYES_00,DKIM_INVALID, DKIM_SIGNED,HEADER_FROM_DIFFERENT_DOMAINS,INCLUDES_CR_TRAILER,INCLUDES_PATCH, MAILING_LIST_MULTI,SPF_HELO_NONE,USER_AGENT_GIT autolearn=ham autolearn_force=no version=3.4.0 Received: from mail.kernel.org (mail.kernel.org [198.145.29.99]) by smtp.lore.kernel.org (Postfix) with ESMTP id 88F79C12002 for ; Wed, 14 Jul 2021 22:25:27 +0000 (UTC) Received: from kanga.kvack.org (kanga.kvack.org [205.233.56.17]) by mail.kernel.org (Postfix) with ESMTP id 3EDA9600D1 for ; Wed, 14 Jul 2021 22:25:27 +0000 (UTC) DMARC-Filter: OpenDMARC Filter v1.3.2 mail.kernel.org 3EDA9600D1 Authentication-Results: mail.kernel.org; dmarc=fail (p=none dis=none) header.from=redhat.com Authentication-Results: mail.kernel.org; spf=pass smtp.mailfrom=owner-linux-mm@kvack.org Received: by kanga.kvack.org (Postfix) id 903526B00D4; Wed, 14 Jul 2021 18:25:27 -0400 (EDT) Received: by kanga.kvack.org (Postfix, from userid 40) id 8B2606B00D5; Wed, 14 Jul 2021 18:25:27 -0400 (EDT) X-Delivered-To: int-list-linux-mm@kvack.org Received: by kanga.kvack.org (Postfix, from userid 63042) id 6B7F48D0003; Wed, 14 Jul 2021 18:25:27 -0400 (EDT) X-Delivered-To: linux-mm@kvack.org Received: from forelay.hostedemail.com (smtprelay0022.hostedemail.com [216.40.44.22]) by kanga.kvack.org (Postfix) with ESMTP id 49DE06B00D4 for ; Wed, 14 Jul 2021 18:25:27 -0400 (EDT) Received: from smtpin09.hostedemail.com (10.5.19.251.rfc1918.com [10.5.19.251]) by forelay01.hostedemail.com (Postfix) with ESMTP id 3F3F518484CAE for ; Wed, 14 Jul 2021 22:25:26 +0000 (UTC) X-FDA: 78362625852.09.ECDAF47 Received: from us-smtp-delivery-124.mimecast.com (us-smtp-delivery-124.mimecast.com [216.205.24.124]) by imf11.hostedemail.com (Postfix) with ESMTP id D7090F0000A8 for ; Wed, 14 Jul 2021 22:25:25 +0000 (UTC) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=redhat.com; s=mimecast20190719; t=1626301525; h=from:from:reply-to:subject:subject:date:date:message-id:message-id: to:to:cc:cc:mime-version:mime-version:content-type:content-type: content-transfer-encoding:content-transfer-encoding: in-reply-to:in-reply-to:references:references; bh=4EzkVzkNiBeZzaTPMlb/0FnWgb1POP033DBuy6Id530=; b=b1JSRGqlZjPlcgLPS72gkEvg75CXizZBM8DzP+k8xlOCmGEl+2aQn2kYRnXEjxhkA3t1i1 rUtPMml9Nmv/UVjTNpqQJLFpn6sIOCdJj77DgxqPmKtClCUvzMfs39wGC5P0GIMhclut+g RWBjDUYrhmN4YrenyrzG3wrzfyimeKo= DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=redhat.com; s=mimecast20190719; t=1626301525; h=from:from:reply-to:subject:subject:date:date:message-id:message-id: to:to:cc:cc:mime-version:mime-version:content-type:content-type: content-transfer-encoding:content-transfer-encoding: in-reply-to:in-reply-to:references:references; bh=4EzkVzkNiBeZzaTPMlb/0FnWgb1POP033DBuy6Id530=; b=b1JSRGqlZjPlcgLPS72gkEvg75CXizZBM8DzP+k8xlOCmGEl+2aQn2kYRnXEjxhkA3t1i1 rUtPMml9Nmv/UVjTNpqQJLFpn6sIOCdJj77DgxqPmKtClCUvzMfs39wGC5P0GIMhclut+g RWBjDUYrhmN4YrenyrzG3wrzfyimeKo= Received: from mail-qt1-f198.google.com (mail-qt1-f198.google.com [209.85.160.198]) (Using TLS) by relay.mimecast.com with ESMTP id us-mta-425-ILzpduGLPq6A89uxmHtRVQ-1; Wed, 14 Jul 2021 18:25:24 -0400 X-MC-Unique: ILzpduGLPq6A89uxmHtRVQ-1 Received: by mail-qt1-f198.google.com with SMTP id h17-20020ac858510000b0290251fc608f73so2826241qth.10 for ; Wed, 14 Jul 2021 15:25:24 -0700 (PDT) X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20161025; h=x-gm-message-state:from:to:cc:subject:date:message-id:in-reply-to :references:mime-version:content-transfer-encoding; bh=4EzkVzkNiBeZzaTPMlb/0FnWgb1POP033DBuy6Id530=; b=hdkrzhavLHJzf/4e8RgboTlwKaAUVyGnHiFn0mT1B5Nn51O+Yj0ClFR8nU/2PnVfNP 9zYqaUB+OcDVBeo1o7qzeWIPnIAtDl94yLAx+OQY9QJOs/p1P7bEVe60OvhVmWlwoTnT LTXRxCM6sfey5L6GmYtPtdolG2roFTc7WvbB9jVh6sGuzgGbji0D/Ska1MAEmPooH3uA n7jrp8HGzs3iMd8RXiDau+v2iasMhQT1VTyXtHRv/sst1rSAYaMvSVVZ6dVYsOnCixe9 OosUaX8DMmq96HTAp982lq3s9yOPQknUOQDbB7+iZ+BXfbG1BrRclnp+J4YHN1OALMg2 pS3Q== X-Gm-Message-State: AOAM531Lfm6hmuMUxyXqHxrz1lhwuEjwRAmxI2oe2ll1XqcranhLMpcU l1Jww6vzguqBsPs8L1qyu818wQyfzS/6nKiWBzU66/qe/p6C0oYaffsS4TJNCeBz6sBDaYRhn92 qLCHRw3ExH8tZqNVTry7sAt1cr1U2BiMeLEkQatD2SsLMWjj+EkFDv7hzErCo X-Received: by 2002:ac8:7d90:: with SMTP id c16mr305203qtd.337.1626301523592; Wed, 14 Jul 2021 15:25:23 -0700 (PDT) X-Google-Smtp-Source: ABdhPJzhP5fg1wkNrQQCAKpXp/mImlAGeXr4rNaibuFCbezexu1TgG1nGIouuRpG9MYniJZSSS7zdQ== X-Received: by 2002:ac8:7d90:: with SMTP id c16mr305170qtd.337.1626301523315; Wed, 14 Jul 2021 15:25:23 -0700 (PDT) Received: from localhost.localdomain (bras-base-toroon474qw-grc-65-184-144-111-238.dsl.bell.ca. [184.144.111.238]) by smtp.gmail.com with ESMTPSA id b3sm1299441qto.49.2021.07.14.15.25.22 (version=TLS1_3 cipher=TLS_AES_256_GCM_SHA384 bits=256/256); Wed, 14 Jul 2021 15:25:22 -0700 (PDT) From: Peter Xu To: linux-mm@kvack.org, linux-kernel@vger.kernel.org Cc: Andrew Morton , Mike Kravetz , Axel Rasmussen , Miaohe Lin , "Kirill A . Shutemov" , Hugh Dickins , Jason Gunthorpe , Alistair Popple , Matthew Wilcox , peterx@redhat.com, Jerome Glisse , Andrea Arcangeli , Mike Rapoport , Nadav Amit , David Hildenbrand Subject: [PATCH v4 22/26] hugetlb/userfaultfd: Allow wr-protect none ptes Date: Wed, 14 Jul 2021 18:25:19 -0400 Message-Id: <20210714222519.49194-1-peterx@redhat.com> X-Mailer: git-send-email 2.31.1 In-Reply-To: <20210714222117.47648-1-peterx@redhat.com> References: <20210714222117.47648-1-peterx@redhat.com> MIME-Version: 1.0 X-Mimecast-Spam-Score: 0 X-Mimecast-Originator: redhat.com Authentication-Results: imf11.hostedemail.com; dkim=pass header.d=redhat.com header.s=mimecast20190719 header.b=b1JSRGql; dkim=pass header.d=redhat.com header.s=mimecast20190719 header.b=b1JSRGql; spf=none (imf11.hostedemail.com: domain of peterx@redhat.com has no SPF policy when checking 216.205.24.124) smtp.mailfrom=peterx@redhat.com; dmarc=pass (policy=none) header.from=redhat.com X-Stat-Signature: exjoytcaaac9tnz13zrgsn3u4bqgpred X-Rspamd-Queue-Id: D7090F0000A8 X-Rspamd-Server: rspam01 X-HE-Tag: 1626301525-479020 X-Bogosity: Ham, tests=bogofilter, spamicity=0.000000, version=1.2.4 Sender: owner-linux-mm@kvack.org Precedence: bulk X-Loop: owner-majordomo@kvack.org List-ID: Teach hugetlbfs code to wr-protect none ptes just in case the page cache existed for that pte. Meanwhile we also need to be able to recognize a uffd-wp marker pte and remove it for uffd_wp_resolve. Since at it, introduce a variable "psize" to replace all references to the huge page size fetcher. Reviewed-by: Mike Kravetz Signed-off-by: Peter Xu --- mm/hugetlb.c | 29 +++++++++++++++++++++++++---- 1 file changed, 25 insertions(+), 4 deletions(-) diff --git a/mm/hugetlb.c b/mm/hugetlb.c index 15e5de480cf0..6ae911185554 100644 --- a/mm/hugetlb.c +++ b/mm/hugetlb.c @@ -5561,7 +5561,7 @@ unsigned long hugetlb_change_protection(struct vm_area_struct *vma, pte_t *ptep; pte_t pte; struct hstate *h = hstate_vma(vma); - unsigned long pages = 0; + unsigned long pages = 0, psize = huge_page_size(h); bool shared_pmd = false; struct mmu_notifier_range range; bool uffd_wp = cp_flags & MM_CP_UFFD_WP; @@ -5581,13 +5581,19 @@ unsigned long hugetlb_change_protection(struct vm_area_struct *vma, mmu_notifier_invalidate_range_start(&range); i_mmap_lock_write(vma->vm_file->f_mapping); - for (; address < end; address += huge_page_size(h)) { + for (; address < end; address += psize) { spinlock_t *ptl; - ptep = huge_pte_offset(mm, address, huge_page_size(h)); + ptep = huge_pte_offset(mm, address, psize); if (!ptep) continue; ptl = huge_pte_lock(h, mm, ptep); if (huge_pmd_unshare(mm, vma, &address, ptep)) { + /* + * When uffd-wp is enabled on the vma, unshare + * shouldn't happen at all. Warn about it if it + * happened due to some reason. + */ + WARN_ON_ONCE(uffd_wp || uffd_wp_resolve); pages++; spin_unlock(ptl); shared_pmd = true; @@ -5612,12 +5618,21 @@ unsigned long hugetlb_change_protection(struct vm_area_struct *vma, else if (uffd_wp_resolve) newpte = pte_swp_clear_uffd_wp(newpte); set_huge_swap_pte_at(mm, address, ptep, - newpte, huge_page_size(h)); + newpte, psize); pages++; } spin_unlock(ptl); continue; } + if (unlikely(is_swap_special_pte(pte))) { + WARN_ON_ONCE(!pte_swp_uffd_wp_special(pte)); + /* + * This is changing a non-present pte into a none pte, + * no need for huge_ptep_modify_prot_start/commit(). + */ + if (uffd_wp_resolve) + huge_pte_clear(mm, address, ptep, psize); + } if (!huge_pte_none(pte)) { pte_t old_pte; unsigned int shift = huge_page_shift(hstate_vma(vma)); @@ -5631,6 +5646,12 @@ unsigned long hugetlb_change_protection(struct vm_area_struct *vma, pte = huge_pte_clear_uffd_wp(pte); huge_ptep_modify_prot_commit(vma, address, ptep, old_pte, pte); pages++; + } else { + /* None pte */ + if (unlikely(uffd_wp)) + /* Safe to modify directly (none->non-present). */ + set_huge_pte_at(mm, address, ptep, + pte_swp_mkuffd_wp_special(vma)); } spin_unlock(ptl); } From patchwork Wed Jul 14 22:25:23 2021 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Peter Xu X-Patchwork-Id: 12377985 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org X-Spam-Level: X-Spam-Status: No, score=-16.6 required=3.0 tests=BAYES_00,DKIM_INVALID, DKIM_SIGNED,HEADER_FROM_DIFFERENT_DOMAINS,INCLUDES_CR_TRAILER,INCLUDES_PATCH, MAILING_LIST_MULTI,SPF_HELO_NONE,USER_AGENT_GIT autolearn=ham autolearn_force=no version=3.4.0 Received: from mail.kernel.org (mail.kernel.org [198.145.29.99]) by smtp.lore.kernel.org (Postfix) with ESMTP id DE53FC47E4B for ; Wed, 14 Jul 2021 22:25:31 +0000 (UTC) Received: from kanga.kvack.org (kanga.kvack.org [205.233.56.17]) by mail.kernel.org (Postfix) with ESMTP id 91471613CB for ; Wed, 14 Jul 2021 22:25:31 +0000 (UTC) DMARC-Filter: OpenDMARC Filter v1.3.2 mail.kernel.org 91471613CB Authentication-Results: mail.kernel.org; dmarc=fail (p=none dis=none) header.from=redhat.com Authentication-Results: mail.kernel.org; spf=pass smtp.mailfrom=owner-linux-mm@kvack.org Received: by kanga.kvack.org (Postfix) id E0C1D6B00D6; Wed, 14 Jul 2021 18:25:31 -0400 (EDT) Received: by kanga.kvack.org (Postfix, from userid 40) id DE29F6B00D7; Wed, 14 Jul 2021 18:25:31 -0400 (EDT) X-Delivered-To: int-list-linux-mm@kvack.org Received: by kanga.kvack.org (Postfix, from userid 63042) id C5C708D0003; Wed, 14 Jul 2021 18:25:31 -0400 (EDT) X-Delivered-To: linux-mm@kvack.org Received: from forelay.hostedemail.com (smtprelay0226.hostedemail.com [216.40.44.226]) by kanga.kvack.org (Postfix) with ESMTP id 9C46E6B00D6 for ; Wed, 14 Jul 2021 18:25:31 -0400 (EDT) Received: from smtpin01.hostedemail.com (10.5.19.251.rfc1918.com [10.5.19.251]) by forelay02.hostedemail.com (Postfix) with ESMTP id 92EBC1CB18 for ; Wed, 14 Jul 2021 22:25:30 +0000 (UTC) X-FDA: 78362626020.01.233931F Received: from us-smtp-delivery-124.mimecast.com (us-smtp-delivery-124.mimecast.com [170.10.133.124]) by imf21.hostedemail.com (Postfix) with ESMTP id 4CCFCD008A78 for ; Wed, 14 Jul 2021 22:25:30 +0000 (UTC) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=redhat.com; s=mimecast20190719; t=1626301529; h=from:from:reply-to:subject:subject:date:date:message-id:message-id: to:to:cc:cc:mime-version:mime-version:content-type:content-type: content-transfer-encoding:content-transfer-encoding: in-reply-to:in-reply-to:references:references; bh=BeGxf/WN0sIL59LjqfsOTv+uY8L0fxb8LgKXxdZ3d9k=; b=NVxGpkiw2+swW3ILgGE18nFrJTq0SvPxsQYtM771BHHSCi5jKzcvOvglFeQXjYRlWgRMw4 aHqkT8eHGjsMeoSIZf50DMELJqtopJLRjVU9BTo9SmDC4KWe3G2owXyT0qCWSgPz31Iqmd KLtup0juJOxOaTnRm7FOl1uJLOmnh/E= Received: from mail-qt1-f199.google.com (mail-qt1-f199.google.com [209.85.160.199]) (Using TLS) by relay.mimecast.com with ESMTP id us-mta-2-_gFlS5qCNTuk6xoOlDuiVw-1; Wed, 14 Jul 2021 18:25:28 -0400 X-MC-Unique: _gFlS5qCNTuk6xoOlDuiVw-1 Received: by mail-qt1-f199.google.com with SMTP id c18-20020a05622a0592b0290251fb198592so2856400qtb.1 for ; Wed, 14 Jul 2021 15:25:28 -0700 (PDT) X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20161025; h=x-gm-message-state:from:to:cc:subject:date:message-id:in-reply-to :references:mime-version:content-transfer-encoding; bh=BeGxf/WN0sIL59LjqfsOTv+uY8L0fxb8LgKXxdZ3d9k=; b=VzNY2bw6xqZR0HhZ3ze1HtfkSJRab6Ver6xGciDYVAEyxOpO4kpCrOpMkrEUvKh9ah ZlT9xRuv9H+XsJ0j9emM9rZ4xyXROXsTFapT9vV4eoymr11jSUWsNVl6p2BhIaGELiK4 bM0B7hZFXiWJokUtGJde/oM04QNuhGzeYksoosv29r6Sm5Kj5LaAS2w+fHmBVsE+4S5h Du7nVf9W+L4uC263sbhU5oUr4Rlxfn/xu4YpzJFiqMUAFcbcB6y5y0HoXDNZJCrnoClg xWrDvcumd5y48DT9dlxGzmYTgBcd1WIH84tGGJV66Hcz9wGMYETESqkuv881XhTNMFgx F0+Q== X-Gm-Message-State: AOAM533qWYTDL3OY27ny5VlkRhJBCKo/EblVgrp1LVbdupIA7HOJYW1N AMlhZVpP6IYcRzI8AuKSTq96253hxF2V0yY8ZEyMhLW6X2zCxU3AbBXl8Ooq2O4PXNwwRh5P1i/ xiOSn36KPFj4wGe5Jlxxxiqau686PdOmU+4S/bdlAtnbqVFgjgMJifjLHlg6L X-Received: by 2002:a05:622a:108:: with SMTP id u8mr346934qtw.326.1626301527831; Wed, 14 Jul 2021 15:25:27 -0700 (PDT) X-Google-Smtp-Source: ABdhPJxhH2E/mNL4S4lcKN8iqUttULcoCsiKBylftVG/y/YX+EjEY6AHZocFdk+5PeHpSVpurzLf0w== X-Received: by 2002:a05:622a:108:: with SMTP id u8mr346890qtw.326.1626301527532; Wed, 14 Jul 2021 15:25:27 -0700 (PDT) Received: from localhost.localdomain (bras-base-toroon474qw-grc-65-184-144-111-238.dsl.bell.ca. [184.144.111.238]) by smtp.gmail.com with ESMTPSA id k186sm1657101qkb.29.2021.07.14.15.25.26 (version=TLS1_3 cipher=TLS_AES_256_GCM_SHA384 bits=256/256); Wed, 14 Jul 2021 15:25:27 -0700 (PDT) From: Peter Xu To: linux-mm@kvack.org, linux-kernel@vger.kernel.org Cc: Andrew Morton , Mike Kravetz , Axel Rasmussen , Miaohe Lin , "Kirill A . Shutemov" , Hugh Dickins , Jason Gunthorpe , Alistair Popple , Matthew Wilcox , peterx@redhat.com, Jerome Glisse , Andrea Arcangeli , Mike Rapoport , Nadav Amit , David Hildenbrand Subject: [PATCH v4 23/26] hugetlb/userfaultfd: Only drop uffd-wp special pte if required Date: Wed, 14 Jul 2021 18:25:23 -0400 Message-Id: <20210714222523.49245-1-peterx@redhat.com> X-Mailer: git-send-email 2.31.1 In-Reply-To: <20210714222117.47648-1-peterx@redhat.com> References: <20210714222117.47648-1-peterx@redhat.com> MIME-Version: 1.0 X-Mimecast-Spam-Score: 0 X-Mimecast-Originator: redhat.com Authentication-Results: imf21.hostedemail.com; dkim=pass header.d=redhat.com header.s=mimecast20190719 header.b=NVxGpkiw; spf=none (imf21.hostedemail.com: domain of peterx@redhat.com has no SPF policy when checking 170.10.133.124) smtp.mailfrom=peterx@redhat.com; dmarc=pass (policy=none) header.from=redhat.com X-Rspamd-Server: rspam05 X-Stat-Signature: xtandbsitdz4ycrrwd89ofyu836dzae4 X-Rspamd-Queue-Id: 4CCFCD008A78 X-HE-Tag: 1626301530-373664 X-Bogosity: Ham, tests=bogofilter, spamicity=0.000000, version=1.2.4 Sender: owner-linux-mm@kvack.org Precedence: bulk X-Loop: owner-majordomo@kvack.org List-ID: As with shmem uffd-wp special ptes, only drop the uffd-wp special swap pte if unmapping an entire vma or synchronized such that faults can not race with the unmap operation. This requires passing zap_flags all the way to the lowest level hugetlb unmap routine: __unmap_hugepage_range. In general, unmap calls originated in hugetlbfs code will pass the ZAP_FLAG_DROP_FILE_UFFD_WP flag as synchronization is in place to prevent faults. The exception is hole punch which will first unmap without any synchronization. Later when hole punch actually removes the page from the file, it will check to see if there was a subsequent fault and if so take the hugetlb fault mutex while unmapping again. This second unmap will pass in ZAP_FLAG_DROP_FILE_UFFD_WP. The core justification of "whether to apply ZAP_FLAG_DROP_FILE_UFFD_WP flag when unmap a hugetlb range" is (IMHO): we should never reach a state when a page fault could errornously fault in a page-cache page that was wr-protected to be writable, even in an extremely short period. That could happen if e.g. we pass ZAP_FLAG_DROP_FILE_UFFD_WP in hugetlbfs_punch_hole() when calling hugetlb_vmdelete_list(), because if a page fault triggers after that call and before the remove_inode_hugepages() right after it, the page cache can be mapped writable again in the small window, which can cause data corruption. Reviewed-by: Mike Kravetz Signed-off-by: Peter Xu --- fs/hugetlbfs/inode.c | 15 +++++++++------ include/linux/hugetlb.h | 8 +++++--- mm/hugetlb.c | 27 +++++++++++++++++++++------ mm/memory.c | 5 ++++- 4 files changed, 39 insertions(+), 16 deletions(-) diff --git a/fs/hugetlbfs/inode.c b/fs/hugetlbfs/inode.c index 926eeb9bf4eb..fdbb972b781b 100644 --- a/fs/hugetlbfs/inode.c +++ b/fs/hugetlbfs/inode.c @@ -404,7 +404,8 @@ static void remove_huge_page(struct page *page) } static void -hugetlb_vmdelete_list(struct rb_root_cached *root, pgoff_t start, pgoff_t end) +hugetlb_vmdelete_list(struct rb_root_cached *root, pgoff_t start, pgoff_t end, + unsigned long zap_flags) { struct vm_area_struct *vma; @@ -437,7 +438,7 @@ hugetlb_vmdelete_list(struct rb_root_cached *root, pgoff_t start, pgoff_t end) } unmap_hugepage_range(vma, vma->vm_start + v_offset, v_end, - NULL); + NULL, zap_flags); } } @@ -515,7 +516,8 @@ static void remove_inode_hugepages(struct inode *inode, loff_t lstart, mutex_lock(&hugetlb_fault_mutex_table[hash]); hugetlb_vmdelete_list(&mapping->i_mmap, index * pages_per_huge_page(h), - (index + 1) * pages_per_huge_page(h)); + (index + 1) * pages_per_huge_page(h), + ZAP_FLAG_DROP_FILE_UFFD_WP); i_mmap_unlock_write(mapping); } @@ -581,7 +583,8 @@ static void hugetlb_vmtruncate(struct inode *inode, loff_t offset) i_mmap_lock_write(mapping); i_size_write(inode, offset); if (!RB_EMPTY_ROOT(&mapping->i_mmap.rb_root)) - hugetlb_vmdelete_list(&mapping->i_mmap, pgoff, 0); + hugetlb_vmdelete_list(&mapping->i_mmap, pgoff, 0, + ZAP_FLAG_DROP_FILE_UFFD_WP); i_mmap_unlock_write(mapping); remove_inode_hugepages(inode, offset, LLONG_MAX); } @@ -614,8 +617,8 @@ static long hugetlbfs_punch_hole(struct inode *inode, loff_t offset, loff_t len) i_mmap_lock_write(mapping); if (!RB_EMPTY_ROOT(&mapping->i_mmap.rb_root)) hugetlb_vmdelete_list(&mapping->i_mmap, - hole_start >> PAGE_SHIFT, - hole_end >> PAGE_SHIFT); + hole_start >> PAGE_SHIFT, + hole_end >> PAGE_SHIFT, 0); i_mmap_unlock_write(mapping); remove_inode_hugepages(inode, hole_start, hole_end); inode_unlock(inode); diff --git a/include/linux/hugetlb.h b/include/linux/hugetlb.h index e19ca363803d..809bb63ecf9e 100644 --- a/include/linux/hugetlb.h +++ b/include/linux/hugetlb.h @@ -138,11 +138,12 @@ long follow_hugetlb_page(struct mm_struct *, struct vm_area_struct *, unsigned long *, unsigned long *, long, unsigned int, int *); void unmap_hugepage_range(struct vm_area_struct *, - unsigned long, unsigned long, struct page *); + unsigned long, unsigned long, struct page *, + unsigned long); void __unmap_hugepage_range_final(struct mmu_gather *tlb, struct vm_area_struct *vma, unsigned long start, unsigned long end, - struct page *ref_page); + struct page *ref_page, unsigned long zap_flags); void hugetlb_report_meminfo(struct seq_file *); int hugetlb_report_node_meminfo(char *buf, int len, int nid); void hugetlb_show_meminfo(void); @@ -381,7 +382,8 @@ static inline unsigned long hugetlb_change_protection( static inline void __unmap_hugepage_range_final(struct mmu_gather *tlb, struct vm_area_struct *vma, unsigned long start, - unsigned long end, struct page *ref_page) + unsigned long end, struct page *ref_page, + unsigned long zap_flags) { BUG(); } diff --git a/mm/hugetlb.c b/mm/hugetlb.c index 6ae911185554..cc5616d78f35 100644 --- a/mm/hugetlb.c +++ b/mm/hugetlb.c @@ -4353,7 +4353,7 @@ int copy_hugetlb_page_range(struct mm_struct *dst, struct mm_struct *src, void __unmap_hugepage_range(struct mmu_gather *tlb, struct vm_area_struct *vma, unsigned long start, unsigned long end, - struct page *ref_page) + struct page *ref_page, unsigned long zap_flags) { struct mm_struct *mm = vma->vm_mm; unsigned long address; @@ -4405,6 +4405,19 @@ void __unmap_hugepage_range(struct mmu_gather *tlb, struct vm_area_struct *vma, continue; } + if (unlikely(is_swap_special_pte(pte))) { + WARN_ON_ONCE(!pte_swp_uffd_wp_special(pte)); + /* + * Only drop the special swap uffd-wp pte if + * e.g. unmapping a vma or punching a hole (with proper + * lock held so that concurrent page fault won't happen). + */ + if (zap_flags & ZAP_FLAG_DROP_FILE_UFFD_WP) + huge_pte_clear(mm, address, ptep, sz); + spin_unlock(ptl); + continue; + } + /* * Migrating hugepage or HWPoisoned hugepage is already * unmapped and its refcount is dropped, so just clear pte here. @@ -4456,9 +4469,10 @@ void __unmap_hugepage_range(struct mmu_gather *tlb, struct vm_area_struct *vma, void __unmap_hugepage_range_final(struct mmu_gather *tlb, struct vm_area_struct *vma, unsigned long start, - unsigned long end, struct page *ref_page) + unsigned long end, struct page *ref_page, + unsigned long zap_flags) { - __unmap_hugepage_range(tlb, vma, start, end, ref_page); + __unmap_hugepage_range(tlb, vma, start, end, ref_page, zap_flags); /* * Clear this flag so that x86's huge_pmd_share page_table_shareable @@ -4474,12 +4488,13 @@ void __unmap_hugepage_range_final(struct mmu_gather *tlb, } void unmap_hugepage_range(struct vm_area_struct *vma, unsigned long start, - unsigned long end, struct page *ref_page) + unsigned long end, struct page *ref_page, + unsigned long zap_flags) { struct mmu_gather tlb; tlb_gather_mmu(&tlb, vma->vm_mm); - __unmap_hugepage_range(&tlb, vma, start, end, ref_page); + __unmap_hugepage_range(&tlb, vma, start, end, ref_page, zap_flags); tlb_finish_mmu(&tlb); } @@ -4534,7 +4549,7 @@ static void unmap_ref_private(struct mm_struct *mm, struct vm_area_struct *vma, */ if (!is_vma_resv_set(iter_vma, HPAGE_RESV_OWNER)) unmap_hugepage_range(iter_vma, address, - address + huge_page_size(h), page); + address + huge_page_size(h), page, 0); } i_mmap_unlock_write(mapping); } diff --git a/mm/memory.c b/mm/memory.c index af91bee934c7..c4a80f45e48f 100644 --- a/mm/memory.c +++ b/mm/memory.c @@ -1626,8 +1626,11 @@ static void unmap_single_vma(struct mmu_gather *tlb, * safe to do nothing in this case. */ if (vma->vm_file) { + unsigned long zap_flags = details ? + details->zap_flags : 0; i_mmap_lock_write(vma->vm_file->f_mapping); - __unmap_hugepage_range_final(tlb, vma, start, end, NULL); + __unmap_hugepage_range_final(tlb, vma, start, end, + NULL, zap_flags); i_mmap_unlock_write(vma->vm_file->f_mapping); } } else From patchwork Wed Jul 14 22:25:28 2021 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Peter Xu X-Patchwork-Id: 12377987 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org X-Spam-Level: X-Spam-Status: No, score=-16.6 required=3.0 tests=BAYES_00,DKIM_INVALID, DKIM_SIGNED,HEADER_FROM_DIFFERENT_DOMAINS,INCLUDES_CR_TRAILER,INCLUDES_PATCH, MAILING_LIST_MULTI,SPF_HELO_NONE,USER_AGENT_GIT autolearn=ham autolearn_force=no version=3.4.0 Received: from mail.kernel.org (mail.kernel.org [198.145.29.99]) by smtp.lore.kernel.org (Postfix) with ESMTP id BFF9BC47E4D for ; Wed, 14 Jul 2021 22:25:35 +0000 (UTC) Received: from kanga.kvack.org (kanga.kvack.org [205.233.56.17]) by mail.kernel.org (Postfix) with ESMTP id 6E8B3613C9 for ; Wed, 14 Jul 2021 22:25:35 +0000 (UTC) DMARC-Filter: OpenDMARC Filter v1.3.2 mail.kernel.org 6E8B3613C9 Authentication-Results: mail.kernel.org; dmarc=fail (p=none dis=none) header.from=redhat.com Authentication-Results: mail.kernel.org; spf=pass smtp.mailfrom=owner-linux-mm@kvack.org Received: by kanga.kvack.org (Postfix) id BE9486B00D8; Wed, 14 Jul 2021 18:25:35 -0400 (EDT) Received: by kanga.kvack.org (Postfix, from userid 40) id BBF9A8D0003; Wed, 14 Jul 2021 18:25:35 -0400 (EDT) X-Delivered-To: int-list-linux-mm@kvack.org Received: by kanga.kvack.org (Postfix, from userid 63042) id A39BE6B00DA; Wed, 14 Jul 2021 18:25:35 -0400 (EDT) X-Delivered-To: linux-mm@kvack.org Received: from forelay.hostedemail.com (smtprelay0146.hostedemail.com [216.40.44.146]) by kanga.kvack.org (Postfix) with ESMTP id 7F8AB6B00D8 for ; Wed, 14 Jul 2021 18:25:35 -0400 (EDT) Received: from smtpin22.hostedemail.com (10.5.19.251.rfc1918.com [10.5.19.251]) by forelay04.hostedemail.com (Postfix) with ESMTP id 6BADA1E4DE for ; Wed, 14 Jul 2021 22:25:34 +0000 (UTC) X-FDA: 78362626188.22.535B45E Received: from us-smtp-delivery-124.mimecast.com (us-smtp-delivery-124.mimecast.com [170.10.133.124]) by imf21.hostedemail.com (Postfix) with ESMTP id 1B1CDD008A73 for ; Wed, 14 Jul 2021 22:25:33 +0000 (UTC) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=redhat.com; s=mimecast20190719; t=1626301533; h=from:from:reply-to:subject:subject:date:date:message-id:message-id: to:to:cc:cc:mime-version:mime-version:content-type:content-type: content-transfer-encoding:content-transfer-encoding: in-reply-to:in-reply-to:references:references; bh=d+qr2gkGCf8UGCRyFBArhsHy6tP5+E2Jys4dwvbHGGg=; b=OOL302q5CRWEHDYqPbP47jWi/wgopq7PEtMf/Y5BMDwbhKgQYudqsbaEfaxMDJ6Hzp1QqD bLVYQWUz4dGup9KInI2JMVxtPcz6oQvLvD03ZpcjlkvQK7Rt0YOmpi7dv8pITaear14dYR owoyUDkp4d7dtAODZB8/w1M/FYR+kuA= Received: from mail-qk1-f198.google.com (mail-qk1-f198.google.com [209.85.222.198]) (Using TLS) by relay.mimecast.com with ESMTP id us-mta-278-Uk75DobDMymV4aAuTX-lAA-1; Wed, 14 Jul 2021 18:25:32 -0400 X-MC-Unique: Uk75DobDMymV4aAuTX-lAA-1 Received: by mail-qk1-f198.google.com with SMTP id y5-20020a37af050000b02903a9c3f8b89fso2332203qke.2 for ; Wed, 14 Jul 2021 15:25:32 -0700 (PDT) X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20161025; h=x-gm-message-state:from:to:cc:subject:date:message-id:in-reply-to :references:mime-version:content-transfer-encoding; bh=d+qr2gkGCf8UGCRyFBArhsHy6tP5+E2Jys4dwvbHGGg=; b=FeXU1wnmykw34DqJG/uFf/vc/G2qqpCE966j8fYOXQVcOV0Qeg5ASoiDqiFX1jXC9X qPO0L3udtfj+82i++iPCDzkUXa4ldA30DMPff3XkZ8ndQtmrdEykWck6UnaSW9deTCFw hVs6Rj6mNwIFwx3EXrHMaAKkPrrLmSHpYWCX1cSxZfJEfP2IoUO7H8CsIyVmGIvZYdlc 1wT/VVg0BzTzzp+AMdIovIXEVQ0k2Sgt4Fdh/1gKTxjA/Ot21w/7jFJqzLF4ISzWyyIB 6tShiLEGpA5b+vAUzJR+CeqOEP3w4QWxr9cB4zQ8XsdQPdOYQmKt8zpV/zjc1Yt8m430 NtPA== X-Gm-Message-State: AOAM531RHX+yC2/W8MZUymzwylib/SOEgSs0rRbKMsP1xYWBcot8resq Aq6gtm9UMUZwy4bldRexOPn1NnHfu0oG21n/W4evEd/awjvNnuO9P/0wiRlXW+6F1E5STrlFcLN fa/NzvJwE/EKFUTZxZR4qkAZrn4qgLHbSemlHob99ko2ZL30TKj/D/czEiBMM X-Received: by 2002:a0c:f850:: with SMTP id g16mr231027qvo.5.1626301532015; Wed, 14 Jul 2021 15:25:32 -0700 (PDT) X-Google-Smtp-Source: ABdhPJyMAg8OaJDdVqZ7CGPKyUVwq/ziIAcVf370Zekn0zY8Li1Qe9olZ37NU65OmsJDzDU/U84feA== X-Received: by 2002:a0c:f850:: with SMTP id g16mr230995qvo.5.1626301531801; Wed, 14 Jul 2021 15:25:31 -0700 (PDT) Received: from localhost.localdomain (bras-base-toroon474qw-grc-65-184-144-111-238.dsl.bell.ca. [184.144.111.238]) by smtp.gmail.com with ESMTPSA id e123sm1658706qkf.103.2021.07.14.15.25.30 (version=TLS1_3 cipher=TLS_AES_256_GCM_SHA384 bits=256/256); Wed, 14 Jul 2021 15:25:31 -0700 (PDT) From: Peter Xu To: linux-mm@kvack.org, linux-kernel@vger.kernel.org Cc: Andrew Morton , Mike Kravetz , Axel Rasmussen , Miaohe Lin , "Kirill A . Shutemov" , Hugh Dickins , Jason Gunthorpe , Alistair Popple , Matthew Wilcox , peterx@redhat.com, Jerome Glisse , Andrea Arcangeli , Mike Rapoport , Nadav Amit , David Hildenbrand Subject: [PATCH v4 24/26] mm/pagemap: Recognize uffd-wp bit for shmem/hugetlbfs Date: Wed, 14 Jul 2021 18:25:28 -0400 Message-Id: <20210714222528.49296-1-peterx@redhat.com> X-Mailer: git-send-email 2.31.1 In-Reply-To: <20210714222117.47648-1-peterx@redhat.com> References: <20210714222117.47648-1-peterx@redhat.com> MIME-Version: 1.0 X-Mimecast-Spam-Score: 0 X-Mimecast-Originator: redhat.com Authentication-Results: imf21.hostedemail.com; dkim=pass header.d=redhat.com header.s=mimecast20190719 header.b=OOL302q5; spf=none (imf21.hostedemail.com: domain of peterx@redhat.com has no SPF policy when checking 170.10.133.124) smtp.mailfrom=peterx@redhat.com; dmarc=pass (policy=none) header.from=redhat.com X-Rspamd-Server: rspam05 X-Stat-Signature: suguheb8r51gixtrpf8zjxpy3t1wbewt X-Rspamd-Queue-Id: 1B1CDD008A73 X-HE-Tag: 1626301533-241227 X-Bogosity: Ham, tests=bogofilter, spamicity=0.000000, version=1.2.4 Sender: owner-linux-mm@kvack.org Precedence: bulk X-Loop: owner-majordomo@kvack.org List-ID: This requires the pagemap code to be able to recognize the newly introduced swap special pte for uffd-wp, meanwhile the general case for hugetlb that we recently start to support. It should make pagemap uffd-wp support complete. Signed-off-by: Peter Xu --- fs/proc/task_mmu.c | 7 +++++++ 1 file changed, 7 insertions(+) diff --git a/fs/proc/task_mmu.c b/fs/proc/task_mmu.c index 9c5af77b5290..988e29fa1f00 100644 --- a/fs/proc/task_mmu.c +++ b/fs/proc/task_mmu.c @@ -1389,6 +1389,8 @@ static pagemap_entry_t pte_to_pagemap_entry(struct pagemapread *pm, flags |= PM_SWAP; if (is_pfn_swap_entry(entry)) page = pfn_swap_entry_to_page(entry); + } else if (pte_swp_uffd_wp_special(pte)) { + flags |= PM_UFFD_WP; } if (page && !PageAnon(page)) @@ -1522,10 +1524,15 @@ static int pagemap_hugetlb_range(pte_t *ptep, unsigned long hmask, if (page_mapcount(page) == 1) flags |= PM_MMAP_EXCLUSIVE; + if (huge_pte_uffd_wp(pte)) + flags |= PM_UFFD_WP; + flags |= PM_PRESENT; if (pm->show_pfn) frame = pte_pfn(pte) + ((addr & ~hmask) >> PAGE_SHIFT); + } else if (pte_swp_uffd_wp_special(pte)) { + flags |= PM_UFFD_WP; } for (; addr != end; addr += PAGE_SIZE) { From patchwork Wed Jul 14 22:25:32 2021 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Peter Xu X-Patchwork-Id: 12377989 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org X-Spam-Level: X-Spam-Status: No, score=-16.6 required=3.0 tests=BAYES_00,DKIM_INVALID, DKIM_SIGNED,HEADER_FROM_DIFFERENT_DOMAINS,INCLUDES_CR_TRAILER,INCLUDES_PATCH, MAILING_LIST_MULTI,SPF_HELO_NONE,USER_AGENT_GIT autolearn=ham autolearn_force=no version=3.4.0 Received: from mail.kernel.org (mail.kernel.org [198.145.29.99]) by smtp.lore.kernel.org (Postfix) with ESMTP id 342F5C47E48 for ; Wed, 14 Jul 2021 22:25:40 +0000 (UTC) Received: from kanga.kvack.org (kanga.kvack.org [205.233.56.17]) by mail.kernel.org (Postfix) with ESMTP id D411B613C9 for ; Wed, 14 Jul 2021 22:25:39 +0000 (UTC) DMARC-Filter: OpenDMARC Filter v1.3.2 mail.kernel.org D411B613C9 Authentication-Results: mail.kernel.org; dmarc=fail (p=none dis=none) header.from=redhat.com Authentication-Results: mail.kernel.org; spf=pass smtp.mailfrom=owner-linux-mm@kvack.org Received: by kanga.kvack.org (Postfix) id 3372C6B00DB; Wed, 14 Jul 2021 18:25:40 -0400 (EDT) Received: by kanga.kvack.org (Postfix, from userid 40) id 30D366B00DC; Wed, 14 Jul 2021 18:25:40 -0400 (EDT) X-Delivered-To: int-list-linux-mm@kvack.org Received: by kanga.kvack.org (Postfix, from userid 63042) id 15F9B6B00DD; Wed, 14 Jul 2021 18:25:40 -0400 (EDT) X-Delivered-To: linux-mm@kvack.org Received: from forelay.hostedemail.com (smtprelay0228.hostedemail.com [216.40.44.228]) by kanga.kvack.org (Postfix) with ESMTP id E68806B00DB for ; Wed, 14 Jul 2021 18:25:39 -0400 (EDT) Received: from smtpin36.hostedemail.com (10.5.19.251.rfc1918.com [10.5.19.251]) by forelay04.hostedemail.com (Postfix) with ESMTP id E17FE117EE for ; Wed, 14 Jul 2021 22:25:38 +0000 (UTC) X-FDA: 78362626356.36.80B307E Received: from us-smtp-delivery-124.mimecast.com (us-smtp-delivery-124.mimecast.com [170.10.133.124]) by imf20.hostedemail.com (Postfix) with ESMTP id 86C49D000099 for ; Wed, 14 Jul 2021 22:25:38 +0000 (UTC) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=redhat.com; s=mimecast20190719; t=1626301538; h=from:from:reply-to:subject:subject:date:date:message-id:message-id: to:to:cc:cc:mime-version:mime-version:content-type:content-type: content-transfer-encoding:content-transfer-encoding: in-reply-to:in-reply-to:references:references; bh=dFbEtd/mIByyEP+/voadEMN/U7jbPWDYUbizxptrAqI=; b=CJPAPdoHahkKQJRHgpHPz6fV7K9aViQxLiAAnnrQv0X3hx4g0DtCYPLXW/qdbtjP9/7kyW JpVTUsWJUnP8bOrsu0SFgJnfG+pNk1ItcbjrwgMIZJUR0rXv9wQhLwsdKWs4DXt+ipTgMp 2ZVMMvfrnHUw883lNpBbxqg+/jw9XLA= Received: from mail-qk1-f199.google.com (mail-qk1-f199.google.com [209.85.222.199]) (Using TLS) by relay.mimecast.com with ESMTP id us-mta-394-W8pavLM0PPWKS7frO3_MUw-1; Wed, 14 Jul 2021 18:25:37 -0400 X-MC-Unique: W8pavLM0PPWKS7frO3_MUw-1 Received: by mail-qk1-f199.google.com with SMTP id n77-20020a3740500000b02903b496f6f425so2331823qka.1 for ; Wed, 14 Jul 2021 15:25:37 -0700 (PDT) X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20161025; h=x-gm-message-state:from:to:cc:subject:date:message-id:in-reply-to :references:mime-version:content-transfer-encoding; bh=dFbEtd/mIByyEP+/voadEMN/U7jbPWDYUbizxptrAqI=; b=Zs2Lp1PIqt3dEIw/MFUU346t0MXMAIXxt3Ujktab5ZSgIpO+bugy3STWXPm7czX8Ut TfRDfkGeUWqTt+2Zv53XqJey+++CEILsDiD11VBYFQWUSURzS9zteUTxhIeoDmPDmyNp APE2xIYWNN8GZ/g86sUkwoc77NstcNF12Ovyd81HAI8W0JitrVsdPJLHDG84nrisuYh5 rq3Rkejfl8qv/EqU9pcyDnnoR5VDXHlKwjZj2K7lcrPCj8++UZ9rzpLeoq4/nWvPZqa1 aMKh/8KZDXmz6LC3MM05iZ0spXzEhP/88J/k0M1cQzmIfY3Ufx+OXOl2kDzANPqL+C/j oqFA== X-Gm-Message-State: AOAM531hfzEfoEANJryd5fDbpoinpGaDlebfX5/ln4hIwLTGiPk7hIW9 okfH7J1r/FDzYZHRt3GRdl6AzjMKv3O4Mu5VjEGUB9qzBk8r9KO6B8lIygDYU0kxjtJEIx8Wi9B /tQhOz2arxe73KLiD+IwU3eqNk3xeuNarkIEYrecFeDWKDc+3rGfXrSwL3tv4 X-Received: by 2002:a0c:ea31:: with SMTP id t17mr395994qvp.17.1626301536175; Wed, 14 Jul 2021 15:25:36 -0700 (PDT) X-Google-Smtp-Source: ABdhPJw53ht68lD3bKbnn3cnQFQTWnQJ+Eh2wLk/PWKIQCFK05/1q7xBdAqiFmgIZ2PbZMKSfP/wvA== X-Received: by 2002:a0c:ea31:: with SMTP id t17mr395961qvp.17.1626301535921; Wed, 14 Jul 2021 15:25:35 -0700 (PDT) Received: from localhost.localdomain (bras-base-toroon474qw-grc-65-184-144-111-238.dsl.bell.ca. [184.144.111.238]) by smtp.gmail.com with ESMTPSA id f1sm1626336qkh.75.2021.07.14.15.25.34 (version=TLS1_3 cipher=TLS_AES_256_GCM_SHA384 bits=256/256); Wed, 14 Jul 2021 15:25:35 -0700 (PDT) From: Peter Xu To: linux-mm@kvack.org, linux-kernel@vger.kernel.org Cc: Andrew Morton , Mike Kravetz , Axel Rasmussen , Miaohe Lin , "Kirill A . Shutemov" , Hugh Dickins , Jason Gunthorpe , Alistair Popple , Matthew Wilcox , peterx@redhat.com, Jerome Glisse , Andrea Arcangeli , Mike Rapoport , Nadav Amit , David Hildenbrand Subject: [PATCH v4 25/26] mm/userfaultfd: Enable write protection for shmem & hugetlbfs Date: Wed, 14 Jul 2021 18:25:32 -0400 Message-Id: <20210714222532.49346-1-peterx@redhat.com> X-Mailer: git-send-email 2.31.1 In-Reply-To: <20210714222117.47648-1-peterx@redhat.com> References: <20210714222117.47648-1-peterx@redhat.com> MIME-Version: 1.0 X-Mimecast-Spam-Score: 0 X-Mimecast-Originator: redhat.com Authentication-Results: imf20.hostedemail.com; dkim=pass header.d=redhat.com header.s=mimecast20190719 header.b=CJPAPdoH; spf=none (imf20.hostedemail.com: domain of peterx@redhat.com has no SPF policy when checking 170.10.133.124) smtp.mailfrom=peterx@redhat.com; dmarc=pass (policy=none) header.from=redhat.com X-Rspamd-Server: rspam02 X-Stat-Signature: tb64okpqpbcnay8yx3b3qh3r5wdqo7bq X-Rspamd-Queue-Id: 86C49D000099 X-HE-Tag: 1626301538-182542 X-Bogosity: Ham, tests=bogofilter, spamicity=0.000000, version=1.2.4 Sender: owner-linux-mm@kvack.org Precedence: bulk X-Loop: owner-majordomo@kvack.org List-ID: We've had all the necessary changes ready for both shmem and hugetlbfs. Turn on all the shmem/hugetlbfs switches for userfaultfd-wp. We can expand UFFD_API_RANGE_IOCTLS_BASIC with _UFFDIO_WRITEPROTECT too because all existing types now support write protection mode. Since vma_can_userfault() will be used elsewhere, move into userfaultfd_k.h. Signed-off-by: Peter Xu --- fs/userfaultfd.c | 21 ++------------------- include/linux/userfaultfd_k.h | 12 ++++++++++++ include/uapi/linux/userfaultfd.h | 10 ++++++++-- mm/userfaultfd.c | 9 +++------ 4 files changed, 25 insertions(+), 27 deletions(-) diff --git a/fs/userfaultfd.c b/fs/userfaultfd.c index 644df737fbb2..980879117110 100644 --- a/fs/userfaultfd.c +++ b/fs/userfaultfd.c @@ -1275,24 +1275,6 @@ static __always_inline int validate_range(struct mm_struct *mm, return 0; } -static inline bool vma_can_userfault(struct vm_area_struct *vma, - unsigned long vm_flags) -{ - /* FIXME: add WP support to hugetlbfs and shmem */ - if (vm_flags & VM_UFFD_WP) { - if (is_vm_hugetlb_page(vma) || vma_is_shmem(vma)) - return false; - } - - if (vm_flags & VM_UFFD_MINOR) { - if (!(is_vm_hugetlb_page(vma) || vma_is_shmem(vma))) - return false; - } - - return vma_is_anonymous(vma) || is_vm_hugetlb_page(vma) || - vma_is_shmem(vma); -} - static int userfaultfd_register(struct userfaultfd_ctx *ctx, unsigned long arg) { @@ -1966,7 +1948,8 @@ static int userfaultfd_api(struct userfaultfd_ctx *ctx, ~(UFFD_FEATURE_MINOR_HUGETLBFS | UFFD_FEATURE_MINOR_SHMEM); #endif #ifndef CONFIG_HAVE_ARCH_USERFAULTFD_WP - uffdio_api.features &= ~UFFD_FEATURE_PAGEFAULT_FLAG_WP; + uffdio_api.features &= + ~(UFFD_FEATURE_PAGEFAULT_FLAG_WP | UFFD_FEATURE_WP_HUGETLBFS_SHMEM); #endif uffdio_api.ioctls = UFFD_API_IOCTLS; ret = -EFAULT; diff --git a/include/linux/userfaultfd_k.h b/include/linux/userfaultfd_k.h index 47c74ef635ab..c4228acd1d80 100644 --- a/include/linux/userfaultfd_k.h +++ b/include/linux/userfaultfd_k.h @@ -16,6 +16,7 @@ #include #include #include +#include /* The set of all possible UFFD-related VM flags. */ #define __VM_UFFD_FLAGS (VM_UFFD_MISSING | VM_UFFD_WP | VM_UFFD_MINOR) @@ -138,6 +139,17 @@ static inline bool userfaultfd_armed(struct vm_area_struct *vma) return vma->vm_flags & __VM_UFFD_FLAGS; } +static inline bool vma_can_userfault(struct vm_area_struct *vma, + unsigned long vm_flags) +{ + if (vm_flags & VM_UFFD_MINOR) + return is_vm_hugetlb_page(vma) || vma_is_shmem(vma); + + return vma_is_anonymous(vma) || is_vm_hugetlb_page(vma) || + vma_is_shmem(vma); +} + + extern int dup_userfaultfd(struct vm_area_struct *, struct list_head *); extern void dup_userfaultfd_complete(struct list_head *); diff --git a/include/uapi/linux/userfaultfd.h b/include/uapi/linux/userfaultfd.h index 05b31d60acf6..a67b5185a7a9 100644 --- a/include/uapi/linux/userfaultfd.h +++ b/include/uapi/linux/userfaultfd.h @@ -32,7 +32,8 @@ UFFD_FEATURE_SIGBUS | \ UFFD_FEATURE_THREAD_ID | \ UFFD_FEATURE_MINOR_HUGETLBFS | \ - UFFD_FEATURE_MINOR_SHMEM) + UFFD_FEATURE_MINOR_SHMEM | \ + UFFD_FEATURE_WP_HUGETLBFS_SHMEM) #define UFFD_API_IOCTLS \ ((__u64)1 << _UFFDIO_REGISTER | \ (__u64)1 << _UFFDIO_UNREGISTER | \ @@ -46,7 +47,8 @@ #define UFFD_API_RANGE_IOCTLS_BASIC \ ((__u64)1 << _UFFDIO_WAKE | \ (__u64)1 << _UFFDIO_COPY | \ - (__u64)1 << _UFFDIO_CONTINUE) + (__u64)1 << _UFFDIO_CONTINUE | \ + (__u64)1 << _UFFDIO_WRITEPROTECT) /* * Valid ioctl command number range with this API is from 0x00 to @@ -189,6 +191,9 @@ struct uffdio_api { * * UFFD_FEATURE_MINOR_SHMEM indicates the same support as * UFFD_FEATURE_MINOR_HUGETLBFS, but for shmem-backed pages instead. + * + * UFFD_FEATURE_WP_HUGETLBFS_SHMEM indicates that userfaultfd + * write-protection mode is supported on both shmem and hugetlbfs. */ #define UFFD_FEATURE_PAGEFAULT_FLAG_WP (1<<0) #define UFFD_FEATURE_EVENT_FORK (1<<1) @@ -201,6 +206,7 @@ struct uffdio_api { #define UFFD_FEATURE_THREAD_ID (1<<8) #define UFFD_FEATURE_MINOR_HUGETLBFS (1<<9) #define UFFD_FEATURE_MINOR_SHMEM (1<<10) +#define UFFD_FEATURE_WP_HUGETLBFS_SHMEM (1<<11) __u64 features; __u64 ioctls; diff --git a/mm/userfaultfd.c b/mm/userfaultfd.c index a8038903effd..6587ed787707 100644 --- a/mm/userfaultfd.c +++ b/mm/userfaultfd.c @@ -724,15 +724,12 @@ int mwriteprotect_range(struct mm_struct *dst_mm, unsigned long start, err = -ENOENT; dst_vma = find_dst_vma(dst_mm, start, len); - /* - * Make sure the vma is not shared, that the dst range is - * both valid and fully within a single existing vma. - */ - if (!dst_vma || (dst_vma->vm_flags & VM_SHARED)) + + if (!dst_vma) goto out_unlock; if (!userfaultfd_wp(dst_vma)) goto out_unlock; - if (!vma_is_anonymous(dst_vma)) + if (!vma_can_userfault(dst_vma, dst_vma->vm_flags)) goto out_unlock; if (is_vm_hugetlb_page(dst_vma)) { From patchwork Wed Jul 14 22:25:36 2021 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Peter Xu X-Patchwork-Id: 12377991 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org X-Spam-Level: X-Spam-Status: No, score=-16.6 required=3.0 tests=BAYES_00,DKIM_INVALID, DKIM_SIGNED,HEADER_FROM_DIFFERENT_DOMAINS,INCLUDES_CR_TRAILER,INCLUDES_PATCH, MAILING_LIST_MULTI,SPF_HELO_NONE,USER_AGENT_GIT autolearn=ham autolearn_force=no version=3.4.0 Received: from mail.kernel.org (mail.kernel.org [198.145.29.99]) by smtp.lore.kernel.org (Postfix) with ESMTP id 5EAECC12002 for ; Wed, 14 Jul 2021 22:25:44 +0000 (UTC) Received: from kanga.kvack.org (kanga.kvack.org [205.233.56.17]) by mail.kernel.org (Postfix) with ESMTP id 12BA9613CA for ; Wed, 14 Jul 2021 22:25:44 +0000 (UTC) DMARC-Filter: OpenDMARC Filter v1.3.2 mail.kernel.org 12BA9613CA Authentication-Results: mail.kernel.org; dmarc=fail (p=none dis=none) header.from=redhat.com Authentication-Results: mail.kernel.org; spf=pass smtp.mailfrom=owner-linux-mm@kvack.org Received: by kanga.kvack.org (Postfix) id 669EF6B00DD; Wed, 14 Jul 2021 18:25:44 -0400 (EDT) Received: by kanga.kvack.org (Postfix, from userid 40) id 61C9D6B00DE; Wed, 14 Jul 2021 18:25:44 -0400 (EDT) X-Delivered-To: int-list-linux-mm@kvack.org Received: by kanga.kvack.org (Postfix, from userid 63042) id 4941A6B00DF; Wed, 14 Jul 2021 18:25:44 -0400 (EDT) X-Delivered-To: linux-mm@kvack.org Received: from forelay.hostedemail.com (smtprelay0148.hostedemail.com [216.40.44.148]) by kanga.kvack.org (Postfix) with ESMTP id 25BD76B00DD for ; Wed, 14 Jul 2021 18:25:44 -0400 (EDT) Received: from smtpin17.hostedemail.com (10.5.19.251.rfc1918.com [10.5.19.251]) by forelay05.hostedemail.com (Postfix) with ESMTP id 0A87518484CAE for ; Wed, 14 Jul 2021 22:25:43 +0000 (UTC) X-FDA: 78362626566.17.ED2C735 Received: from us-smtp-delivery-124.mimecast.com (us-smtp-delivery-124.mimecast.com [216.205.24.124]) by imf11.hostedemail.com (Postfix) with ESMTP id ABF2AF00020B for ; Wed, 14 Jul 2021 22:25:42 +0000 (UTC) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=redhat.com; s=mimecast20190719; t=1626301542; h=from:from:reply-to:subject:subject:date:date:message-id:message-id: to:to:cc:cc:mime-version:mime-version:content-type:content-type: content-transfer-encoding:content-transfer-encoding: in-reply-to:in-reply-to:references:references; bh=Uq4QT2ZrcVHt/byJv1ujYeldpYEbg+Q7RaFfi3UPyeE=; b=PYGkOnSSRvOBpoR5w8sTJD1Nf89dAkin6AEZmgrBH5ORnv4pSjxSVUpKIbR6uzW6woWkbt X8rMGhEjCK9uUgvsEQgSv/xhPUwHkR1+8mmRH45IemlNKODTZAvyNUuNyhwKWbZSJfoCS0 UdCTlVKwhVAaBi+oDTHTDy0n0rJGbwA= Received: from mail-qk1-f197.google.com (mail-qk1-f197.google.com [209.85.222.197]) (Using TLS) by relay.mimecast.com with ESMTP id us-mta-29-KciQxx8gO82hKR2nODpScg-1; Wed, 14 Jul 2021 18:25:41 -0400 X-MC-Unique: KciQxx8gO82hKR2nODpScg-1 Received: by mail-qk1-f197.google.com with SMTP id bi3-20020a05620a3183b02903b55bbe1ef9so2297353qkb.13 for ; Wed, 14 Jul 2021 15:25:41 -0700 (PDT) X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20161025; h=x-gm-message-state:from:to:cc:subject:date:message-id:in-reply-to :references:mime-version:content-transfer-encoding; bh=Uq4QT2ZrcVHt/byJv1ujYeldpYEbg+Q7RaFfi3UPyeE=; b=mjlV4O1euuFTkcVSrfOEV79hWw4NHWPNaOZtIesSXsfZluB4j1vs1QuHh3TK4m/gLC vZrEz7Z4CR8G9aWKtpiCD6YFi2FLBRLjPwOGvgaCbpYSZxk66Gisxi32WdzYzNnAdT/8 9EbqGa86dmZHLhXSRLxJ1bgcEhVj2nMv5gpLP77rJT+QCa3acVvj/w4CBDE4vRFJnGx9 wmG3PeprhHwUA/aujGC9afBKGq6w+LvXV88RKOj+HxhC6v7Q6BCWxjIjPt3SXWnRCd+8 cIRmSc8QmVI6W+pywKRP+gTjqMPv807Zk++xjtaYjIwmx4/6R2DxBQnMFupv7OVeZ3Nx i5pw== X-Gm-Message-State: AOAM533fsqZScgnQxkJUHANi6dGIyIVNH7Wyd/LW3i1YpCn54qaSXJZS AOdozqgnM6Ysar0cb43rl5ekaGpeebTTqa+nePw+A1dCTpTEERK7n5Tf4B1m7B3LlqACU4sOti9 stE/AhLutgV6m/BDSoVk+ZPH0TXybuXlLB485cDm74Z2hf36rcEijDK3zyz+Y X-Received: by 2002:ae9:ea18:: with SMTP id f24mr221982qkg.177.1626301540481; Wed, 14 Jul 2021 15:25:40 -0700 (PDT) X-Google-Smtp-Source: ABdhPJwKWio2waCxeqg1e164RijCKEKXk6rILIQsIZLa2XpqKIZLo+ivNYlFqZ2GZR+4gVoFHCbOFQ== X-Received: by 2002:ae9:ea18:: with SMTP id f24mr221946qkg.177.1626301540222; Wed, 14 Jul 2021 15:25:40 -0700 (PDT) Received: from localhost.localdomain (bras-base-toroon474qw-grc-65-184-144-111-238.dsl.bell.ca. [184.144.111.238]) by smtp.gmail.com with ESMTPSA id j22sm1213396qtj.60.2021.07.14.15.25.38 (version=TLS1_3 cipher=TLS_AES_256_GCM_SHA384 bits=256/256); Wed, 14 Jul 2021 15:25:39 -0700 (PDT) From: Peter Xu To: linux-mm@kvack.org, linux-kernel@vger.kernel.org Cc: Andrew Morton , Mike Kravetz , Axel Rasmussen , Miaohe Lin , "Kirill A . Shutemov" , Hugh Dickins , Jason Gunthorpe , Alistair Popple , Matthew Wilcox , peterx@redhat.com, Jerome Glisse , Andrea Arcangeli , Mike Rapoport , Nadav Amit , David Hildenbrand Subject: [PATCH v4 26/26] userfaultfd/selftests: Enable uffd-wp for shmem/hugetlbfs Date: Wed, 14 Jul 2021 18:25:36 -0400 Message-Id: <20210714222536.49397-1-peterx@redhat.com> X-Mailer: git-send-email 2.31.1 In-Reply-To: <20210714222117.47648-1-peterx@redhat.com> References: <20210714222117.47648-1-peterx@redhat.com> MIME-Version: 1.0 X-Mimecast-Spam-Score: 0 X-Mimecast-Originator: redhat.com Authentication-Results: imf11.hostedemail.com; dkim=pass header.d=redhat.com header.s=mimecast20190719 header.b=PYGkOnSS; spf=none (imf11.hostedemail.com: domain of peterx@redhat.com has no SPF policy when checking 216.205.24.124) smtp.mailfrom=peterx@redhat.com; dmarc=pass (policy=none) header.from=redhat.com X-Rspamd-Server: rspam03 X-Rspamd-Queue-Id: ABF2AF00020B X-Stat-Signature: rmo1kr6tikedsiuht7oz3x6ebuhk85ur X-HE-Tag: 1626301542-366164 X-Bogosity: Ham, tests=bogofilter, spamicity=0.000000, version=1.2.4 Sender: owner-linux-mm@kvack.org Precedence: bulk X-Loop: owner-majordomo@kvack.org List-ID: After we added support for shmem and hugetlbfs, we can turn uffd-wp test on always now. Define HUGETLB_EXPECTED_IOCTLS to avoid using UFFD_API_RANGE_IOCTLS_BASIC, because UFFD_API_RANGE_IOCTLS_BASIC is normally a superset of capabilities, while the test may not satisfy them all. E.g., when hugetlb registered without minor mode, then we need to explicitly remove _UFFDIO_CONTINUE. Same thing to uffd-wp, as we'll need to explicitly remove _UFFDIO_WRITEPROTECT if not registered with uffd-wp. For the long term, we may consider dropping UFFD_API_* macros completely from uapi/linux/userfaultfd.h header files, because it may cause kernel header update to easily break userspace. Signed-off-by: Peter Xu --- tools/testing/selftests/vm/userfaultfd.c | 9 +++++---- 1 file changed, 5 insertions(+), 4 deletions(-) diff --git a/tools/testing/selftests/vm/userfaultfd.c b/tools/testing/selftests/vm/userfaultfd.c index e363bdaff59d..015f2df8ece4 100644 --- a/tools/testing/selftests/vm/userfaultfd.c +++ b/tools/testing/selftests/vm/userfaultfd.c @@ -80,7 +80,7 @@ static int test_type; static volatile bool test_uffdio_copy_eexist = true; static volatile bool test_uffdio_zeropage_eexist = true; /* Whether to test uffd write-protection */ -static bool test_uffdio_wp = false; +static bool test_uffdio_wp = true; /* Whether to test uffd minor faults */ static bool test_uffdio_minor = false; @@ -320,6 +320,9 @@ struct uffd_test_ops { (1 << _UFFDIO_ZEROPAGE) | \ (1 << _UFFDIO_WRITEPROTECT)) +#define HUGETLB_EXPECTED_IOCTLS ((1 << _UFFDIO_WAKE) | \ + (1 << _UFFDIO_COPY)) + static struct uffd_test_ops anon_uffd_test_ops = { .expected_ioctls = ANON_EXPECTED_IOCTLS, .allocate_area = anon_allocate_area, @@ -335,7 +338,7 @@ static struct uffd_test_ops shmem_uffd_test_ops = { }; static struct uffd_test_ops hugetlb_uffd_test_ops = { - .expected_ioctls = UFFD_API_RANGE_IOCTLS_BASIC & ~(1 << _UFFDIO_CONTINUE), + .expected_ioctls = HUGETLB_EXPECTED_IOCTLS, .allocate_area = hugetlb_allocate_area, .release_pages = hugetlb_release_pages, .alias_mapping = hugetlb_alias_mapping, @@ -1580,8 +1583,6 @@ static void set_test_type(const char *type) if (!strcmp(type, "anon")) { test_type = TEST_ANON; uffd_test_ops = &anon_uffd_test_ops; - /* Only enable write-protect test for anonymous test */ - test_uffdio_wp = true; } else if (!strcmp(type, "hugetlb")) { test_type = TEST_HUGETLB; uffd_test_ops = &hugetlb_uffd_test_ops;