From patchwork Mon Jul 18 12:01:59 2022 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Nadav Amit X-Patchwork-Id: 12921680 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from kanga.kvack.org (kanga.kvack.org [205.233.56.17]) by smtp.lore.kernel.org (Postfix) with ESMTP id 87889C433EF for ; Mon, 18 Jul 2022 19:37:07 +0000 (UTC) Received: by kanga.kvack.org (Postfix) id 140526B0072; Mon, 18 Jul 2022 15:37:07 -0400 (EDT) Received: by kanga.kvack.org (Postfix, from userid 40) id 0F0C16B0073; Mon, 18 Jul 2022 15:37:07 -0400 (EDT) X-Delivered-To: int-list-linux-mm@kvack.org Received: by kanga.kvack.org (Postfix, from userid 63042) id EFA508E0001; Mon, 18 Jul 2022 15:37:06 -0400 (EDT) X-Delivered-To: linux-mm@kvack.org Received: from relay.hostedemail.com (smtprelay0016.hostedemail.com [216.40.44.16]) by kanga.kvack.org (Postfix) with ESMTP id E12386B0072 for ; Mon, 18 Jul 2022 15:37:06 -0400 (EDT) Received: from smtpin13.hostedemail.com (a10.router.float.18 [10.200.18.1]) by unirelay06.hostedemail.com (Postfix) with ESMTP id ACB3C34D9C for ; Mon, 18 Jul 2022 19:37:06 +0000 (UTC) X-FDA: 79701228852.13.ED8ABCE Received: from relay3.hostedemail.com (smtprelay0013.hostedemail.com [216.40.44.13]) by imf31.hostedemail.com (Postfix) with ESMTP id 4921D20042 for ; Mon, 18 Jul 2022 19:37:06 +0000 (UTC) Received: from smtpin15.hostedemail.com (a10.router.float.18 [10.200.18.1]) by unirelay10.hostedemail.com (Postfix) with ESMTP id E3C01E5C for ; Mon, 18 Jul 2022 19:37:05 +0000 (UTC) X-FDA: 79701228810.15.27B2127 Received: from mail-pf1-f171.google.com (mail-pf1-f171.google.com [209.85.210.171]) by imf30.hostedemail.com (Postfix) with ESMTP id 682948008E for ; Mon, 18 Jul 2022 19:37:05 +0000 (UTC) Received: by mail-pf1-f171.google.com with SMTP id w185so11569319pfb.4 for ; Mon, 18 Jul 2022 12:37:05 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=20210112; h=from:to:cc:subject:date:message-id:in-reply-to:references :mime-version:content-transfer-encoding; bh=lqN6lUA6CKPQ6pXduKwXrnAjAVaaU8p/HZgGKWiS1UU=; b=anEaeCdDUpWprBhzF25fJpAdtW+LgFlwZ2ebfZ8wM0kpnCfn2B96Ny11a5FiZPFqb+ 8d217vnOhZgu2PfDrycJhxqOQiXud4xvWn2c80DYc99oVi3Ir9FehDJAGUmSeOUhksfr QR+ahMTY316YC8qzqWOkrwI9aF+uUCzKo6Q+X7SXWicVhjcsJrIFPjQgPYBlrrFSkyza nrpLpAgplwp8N1Y7IPDrjQick91Rj9cFFO7tm3RrwD9n6LXQmJHs9J6KNpKvgDrlFFhE LWoW2wCSjvgnI+efvRVSp0tdhvWKQfmNMDQryOAu3HMUlJExJUMY0OFO8aRltUkZiSpA eLoQ== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20210112; h=x-gm-message-state:from:to:cc:subject:date:message-id:in-reply-to :references:mime-version:content-transfer-encoding; bh=lqN6lUA6CKPQ6pXduKwXrnAjAVaaU8p/HZgGKWiS1UU=; b=aUdW0oXkWVR2Rq4/A75wnkCl9vNGwLIGk8SJi4FiPtcAIpdsv+7WDKWD18xRM3OMIT /i1O67oQF0Lqe6fsFx20+gr4/L4NH+jsQMeMwYrgnNH+wuSPp/WdTApBGmJ0QFjNZouC jyVF7138oQsP6JB4q2NwkGhKUMSuU8fycndOr5kU7KCfT9z8Kz4rfubf+Rm33QU3zOhx k1jEABr5l75hCxn6gP7FmQ/aeXErVvKWaqxljx6CxOfX6feMXWy27gHqW9nIsHQExcZ/ TAR7Z3seXIN+VrWBipUhZ7sdugy+ZvdHVpmlZ2JKCDxSm/elaDKyZKDNnlOjy6JnJOQM bePA== X-Gm-Message-State: AJIora9GdzbXo8ITf25LAqtrX4Yo6mfoJR7DZ6qeVyHbDQWAciXqm2CW 0cK7LvQx++YWM1i1v3PDGaIA0sUbr1gbYA== X-Google-Smtp-Source: AGRyM1t4cIpL6IVhzKiQES1MTghU8BQMTlI5pi0eeK/ksqsiBBWEFmy9twLEG4Iqd+Ne+Q/gUcyJmQ== X-Received: by 2002:a05:6a00:b92:b0:52a:e60d:dfbb with SMTP id g18-20020a056a000b9200b0052ae60ddfbbmr29561053pfj.72.1658173024046; Mon, 18 Jul 2022 12:37:04 -0700 (PDT) Received: from sc2-haas01-esx0118.eng.vmware.com ([66.170.99.1]) by smtp.gmail.com with ESMTPSA id q6-20020a170902a3c600b0016bc4a6ce28sm9907887plb.98.2022.07.18.12.37.02 (version=TLS1_3 cipher=TLS_AES_256_GCM_SHA384 bits=256/256); Mon, 18 Jul 2022 12:37:03 -0700 (PDT) From: Nadav Amit X-Google-Original-From: Nadav Amit To: linux-mm@kvack.org Cc: linux-kernel@vger.kernel.org, Andrew Morton , Mike Rapoport , Axel Rasmussen , Nadav Amit , Andrea Arcangeli , Andrew Cooper , Andy Lutomirski , Dave Hansen , David Hildenbrand , Peter Xu , Peter Zijlstra , Thomas Gleixner , Will Deacon , Yu Zhao , Nick Piggin Subject: [RFC PATCH 01/14] userfaultfd: set dirty and young on writeprotect Date: Mon, 18 Jul 2022 05:01:59 -0700 Message-Id: <20220718120212.3180-2-namit@vmware.com> X-Mailer: git-send-email 2.25.1 In-Reply-To: <20220718120212.3180-1-namit@vmware.com> References: <20220718120212.3180-1-namit@vmware.com> MIME-Version: 1.0 ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=hostedemail.com; s=arc-20220608; t=1658173026; h=from:from:sender:reply-to:subject:subject:date:date: message-id:message-id:to:to:cc:cc:mime-version:mime-version: content-type:content-transfer-encoding:content-transfer-encoding: in-reply-to:in-reply-to:references:references:dkim-signature; bh=lqN6lUA6CKPQ6pXduKwXrnAjAVaaU8p/HZgGKWiS1UU=; b=iZ0k9TcdGpyunnPRtkwpyBGEyD1hN81HMzZDOXM7oTbPgqnGizssfSeG/w8icX12dlCUAA kNql6qIMffcT+sa9frsD9KuHP/yKKJgYkNMzh6SCvPxeiu93ToXk49MZo4V42Ij8aVrw5G tosUxmtTgx4rNZC6jHcmZFShP1H2yBs= ARC-Seal: i=1; s=arc-20220608; d=hostedemail.com; t=1658173026; a=rsa-sha256; cv=none; b=0ZsDqb7y7MLSKWeO3brU7AKqoyvyf81OKXji7i/NVkdc9BlBLPPgaRLqjrf8eWodlYVgOu x3PDMACH+lV2DkHnnRuAoqceYiE2uCXcpSKjxFr1eJ7iHDB3Hz1ozzcpl6PU1cdm0rvo4F 5CgLlm7Cl3kGjeLMjaeNrtGREdL+TaI= ARC-Authentication-Results: i=1; imf31.hostedemail.com; dkim=pass header.d=gmail.com header.s=20210112 header.b=anEaeCdD; dmarc=pass (policy=none) header.from=gmail.com; spf=none (imf31.hostedemail.com: domain of MAILER-DAEMON@hostedemail.com has no SPF policy when checking 216.40.44.13) smtp.mailfrom=MAILER-DAEMON@hostedemail.com X-Rspamd-Server: rspam10 X-Stat-Signature: ygyf64m7bkj6mi9j8dqc8omw5ehgms6r X-Rspam-User: X-Rspamd-Queue-Id: 4921D20042 X-HE-Tag-Orig: 1658173025-990891 Authentication-Results: imf31.hostedemail.com; dkim=pass header.d=gmail.com header.s=20210112 header.b=anEaeCdD; dmarc=pass (policy=none) header.from=gmail.com; spf=none (imf31.hostedemail.com: domain of MAILER-DAEMON@hostedemail.com has no SPF policy when checking 216.40.44.13) smtp.mailfrom=MAILER-DAEMON@hostedemail.com X-HE-Tag: 1658173026-469861 X-Bogosity: Ham, tests=bogofilter, spamicity=0.000000, version=1.2.4 Sender: owner-linux-mm@kvack.org Precedence: bulk X-Loop: owner-majordomo@kvack.org List-ID: From: Nadav Amit When userfaultfd makes a PTE writable, it can now change the PTE directly, in some cases, without going triggering a page-fault first. Yet, doing so might leave the PTE that was write-unprotected as old and clean. At least on x86, this would cause a >500 cycles overhead when the PTE is first accessed. Use MM_CP_WILL_NEED to set the PTE as young and dirty when userfaultfd gets a hint that the page is likely to be used. Avoid changing the PTE to young and dirty in other cases to avoid excessive writeback and messing with the page reclamation logic. Cc: Andrea Arcangeli Cc: Andrew Cooper Cc: Andrew Morton Cc: Andy Lutomirski Cc: Dave Hansen Cc: David Hildenbrand Cc: Peter Xu Cc: Peter Zijlstra Cc: Thomas Gleixner Cc: Will Deacon Cc: Yu Zhao Cc: Nick Piggin --- include/linux/mm.h | 2 ++ mm/mprotect.c | 9 ++++++++- mm/userfaultfd.c | 8 ++++++-- 3 files changed, 16 insertions(+), 3 deletions(-) diff --git a/include/linux/mm.h b/include/linux/mm.h index 9cc02a7e503b..4afd75ce5875 100644 --- a/include/linux/mm.h +++ b/include/linux/mm.h @@ -1988,6 +1988,8 @@ extern unsigned long move_page_tables(struct vm_area_struct *vma, /* Whether this change is for write protecting */ #define MM_CP_UFFD_WP (1UL << 2) /* do wp */ #define MM_CP_UFFD_WP_RESOLVE (1UL << 3) /* Resolve wp */ +/* Whether to try to mark entries as dirty as they are to be written */ +#define MM_CP_WILL_NEED (1UL << 4) #define MM_CP_UFFD_WP_ALL (MM_CP_UFFD_WP | \ MM_CP_UFFD_WP_RESOLVE) diff --git a/mm/mprotect.c b/mm/mprotect.c index 996a97e213ad..34c2dfb68c42 100644 --- a/mm/mprotect.c +++ b/mm/mprotect.c @@ -82,6 +82,7 @@ static unsigned long change_pte_range(struct mmu_gather *tlb, bool prot_numa = cp_flags & MM_CP_PROT_NUMA; bool uffd_wp = cp_flags & MM_CP_UFFD_WP; bool uffd_wp_resolve = cp_flags & MM_CP_UFFD_WP_RESOLVE; + bool will_need = cp_flags & MM_CP_WILL_NEED; tlb_change_page_size(tlb, PAGE_SIZE); @@ -172,6 +173,9 @@ static unsigned long change_pte_range(struct mmu_gather *tlb, ptent = pte_clear_uffd_wp(ptent); } + if (will_need) + ptent = pte_mkyoung(ptent); + /* * In some writable, shared mappings, we might want * to catch actual write access -- see @@ -187,8 +191,11 @@ static unsigned long change_pte_range(struct mmu_gather *tlb, */ if ((cp_flags & MM_CP_TRY_CHANGE_WRITABLE) && !pte_write(ptent) && - can_change_pte_writable(vma, addr, ptent)) + can_change_pte_writable(vma, addr, ptent)) { ptent = pte_mkwrite(ptent); + if (will_need) + ptent = pte_mkdirty(ptent); + } ptep_modify_prot_commit(vma, addr, pte, oldpte, ptent); if (pte_needs_flush(oldpte, ptent)) diff --git a/mm/userfaultfd.c b/mm/userfaultfd.c index 954c6980b29f..e0492f5f06a0 100644 --- a/mm/userfaultfd.c +++ b/mm/userfaultfd.c @@ -749,6 +749,7 @@ int mwriteprotect_range(struct mm_struct *dst_mm, unsigned long start, bool enable_wp = uffd_flags & UFFD_FLAGS_WP; struct vm_area_struct *dst_vma; unsigned long page_mask; + unsigned long cp_flags; struct mmu_gather tlb; pgprot_t newprot; int err; @@ -795,9 +796,12 @@ int mwriteprotect_range(struct mm_struct *dst_mm, unsigned long start, else newprot = vm_get_page_prot(dst_vma->vm_flags); + cp_flags = enable_wp ? MM_CP_UFFD_WP : MM_CP_UFFD_WP_RESOLVE; + if (uffd_flags & (UFFD_FLAGS_ACCESS_LIKELY|UFFD_FLAGS_WRITE_LIKELY)) + cp_flags |= MM_CP_WILL_NEED; + tlb_gather_mmu(&tlb, dst_mm); - change_protection(&tlb, dst_vma, start, start + len, newprot, - enable_wp ? MM_CP_UFFD_WP : MM_CP_UFFD_WP_RESOLVE); + change_protection(&tlb, dst_vma, start, start + len, newprot, cp_flags); tlb_finish_mmu(&tlb); err = 0; From patchwork Mon Jul 18 12:02:00 2022 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Nadav Amit X-Patchwork-Id: 12921681 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from kanga.kvack.org (kanga.kvack.org [205.233.56.17]) by smtp.lore.kernel.org (Postfix) with ESMTP id 01874C43334 for ; Mon, 18 Jul 2022 19:37:10 +0000 (UTC) Received: by kanga.kvack.org (Postfix) id 935CF6B0073; Mon, 18 Jul 2022 15:37:10 -0400 (EDT) Received: by kanga.kvack.org (Postfix, from userid 40) id 8DF966B0074; Mon, 18 Jul 2022 15:37:10 -0400 (EDT) X-Delivered-To: int-list-linux-mm@kvack.org Received: by kanga.kvack.org (Postfix, from userid 63042) id 780CD6B0075; Mon, 18 Jul 2022 15:37:10 -0400 (EDT) X-Delivered-To: linux-mm@kvack.org Received: from relay.hostedemail.com (smtprelay0013.hostedemail.com [216.40.44.13]) by kanga.kvack.org (Postfix) with ESMTP id 6C11A6B0073 for ; Mon, 18 Jul 2022 15:37:10 -0400 (EDT) Received: from smtpin17.hostedemail.com (a10.router.float.18 [10.200.18.1]) by unirelay01.hostedemail.com (Postfix) with ESMTP id 416E360A2F for ; Mon, 18 Jul 2022 19:37:10 +0000 (UTC) X-FDA: 79701229020.17.3F39350 Received: from relay3.hostedemail.com (smtprelay0014.hostedemail.com [216.40.44.14]) by imf19.hostedemail.com (Postfix) with ESMTP id E091C1A0082 for ; Mon, 18 Jul 2022 19:37:09 +0000 (UTC) Received: from smtpin29.hostedemail.com (a10.router.float.18 [10.200.18.1]) by unirelay09.hostedemail.com (Postfix) with ESMTP id 6416D34E5C for ; Mon, 18 Jul 2022 19:37:07 +0000 (UTC) X-FDA: 79701228894.29.995F1F2 Received: from mail-pg1-f181.google.com (mail-pg1-f181.google.com [209.85.215.181]) by imf05.hostedemail.com (Postfix) with ESMTP id 0498110006C for ; Mon, 18 Jul 2022 19:37:06 +0000 (UTC) Received: by mail-pg1-f181.google.com with SMTP id s206so11532880pgs.3 for ; Mon, 18 Jul 2022 12:37:06 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=20210112; h=from:to:cc:subject:date:message-id:in-reply-to:references :mime-version:content-transfer-encoding; bh=IufP9Zf+LxI7/ja1tm+EfHoZ34yAqbNH4YNdr2fLKzg=; b=qG3NzQadQYCy4rNcC/00/m1k00Rdnq8ihunEXn9ko/dtWH2HiC/JLZcB0XE3Fmrnme 4uHW7JOvxcluNfpQxgbQVqeUgNSwGxPrf47yeAbr1vPKr2M8T3ojhSwKcIotbVOpuGpC 03vIgXyGqL1xqLnoRk6HZYGbEsfTuBQb3UTilxepLrrh6di54zWWHrvnUwafOcNixDvI mAiMqZJOgnaD3MYRYn1evGphPH12+idguFStTv1WGAkRhN95WV3+4haBwVkVtpdhUTwQ dAycGnGYVMeeHc5jfDHQcsMcyMaz2Yv5VlzLzbDlj5bMe2rXOZ12FUP7+Pdwyzf05Qjg mXxg== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20210112; h=x-gm-message-state:from:to:cc:subject:date:message-id:in-reply-to :references:mime-version:content-transfer-encoding; bh=IufP9Zf+LxI7/ja1tm+EfHoZ34yAqbNH4YNdr2fLKzg=; b=7FJX2ruEifV+H9qgXbmsK5fSIba+10B2XXyDZDbFgsRYxDQUuk+gUUKMKKuntzS/Hv /D6OT53u2ecf4fvaPlIea0cN2Njzr1+i4LSVHM62YdC1j3zEDgi+BLarABPJISwpGc1P dIf3GIsLJREyrqbReakFxAozhKi8vcRWp/dO3x0If/T3P4ghFicf/apGH5LoVSB3vUng Kej3ztZGXiXy0Z5KUqhRD0/HjLQI8y9sn55Nd6lQz3IoF9FYBxyGHmJAJG9DJ4EFZBpm E498afKAqcUPYCeUNLjHQEp0GS099uufcYUlwkgqEUWEIuQPMHHE4JbTfd52rtOEo6M8 pWkA== X-Gm-Message-State: AJIora9lhOCEvD+eq82f4zjdu6QZlbNmiNZMBIdeFq5V/NXxCh9OsK9g S+dvnmcorOQYAOci/Xzw8ziPQxuamrnvWQ== X-Google-Smtp-Source: AGRyM1s4dUG7omvru0Q+c8qc7tyPUfIKb/kzYGzarbIemngP+tEUPjt2ISjbd7jZhK5SI6J3kDZqpg== X-Received: by 2002:a63:504c:0:b0:419:d02e:f42a with SMTP id q12-20020a63504c000000b00419d02ef42amr17915418pgl.566.1658173025597; Mon, 18 Jul 2022 12:37:05 -0700 (PDT) Received: from sc2-haas01-esx0118.eng.vmware.com ([66.170.99.1]) by smtp.gmail.com with ESMTPSA id q6-20020a170902a3c600b0016bc4a6ce28sm9907887plb.98.2022.07.18.12.37.04 (version=TLS1_3 cipher=TLS_AES_256_GCM_SHA384 bits=256/256); Mon, 18 Jul 2022 12:37:05 -0700 (PDT) From: Nadav Amit X-Google-Original-From: Nadav Amit To: linux-mm@kvack.org Cc: linux-kernel@vger.kernel.org, Andrew Morton , Mike Rapoport , Axel Rasmussen , Nadav Amit , Andrea Arcangeli , Andrew Cooper , Andy Lutomirski , Dave Hansen , David Hildenbrand , Peter Xu , Peter Zijlstra , Thomas Gleixner , Will Deacon , Yu Zhao , Nick Piggin Subject: [RFC PATCH 02/14] userfaultfd: try to map write-unprotected pages Date: Mon, 18 Jul 2022 05:02:00 -0700 Message-Id: <20220718120212.3180-3-namit@vmware.com> X-Mailer: git-send-email 2.25.1 In-Reply-To: <20220718120212.3180-1-namit@vmware.com> References: <20220718120212.3180-1-namit@vmware.com> MIME-Version: 1.0 ARC-Authentication-Results: i=1; imf19.hostedemail.com; dkim=pass header.d=gmail.com header.s=20210112 header.b=qG3NzQad; dmarc=pass (policy=none) header.from=gmail.com; spf=none (imf19.hostedemail.com: domain of MAILER-DAEMON@hostedemail.com has no SPF policy when checking 216.40.44.14) smtp.mailfrom=MAILER-DAEMON@hostedemail.com ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=hostedemail.com; s=arc-20220608; t=1658173029; h=from:from:sender:reply-to:subject:subject:date:date: message-id:message-id:to:to:cc:cc:mime-version:mime-version: content-type:content-transfer-encoding:content-transfer-encoding: in-reply-to:in-reply-to:references:references:dkim-signature; bh=IufP9Zf+LxI7/ja1tm+EfHoZ34yAqbNH4YNdr2fLKzg=; b=gg4QBsSols5Rgq/RWL3bRH0mWubMRceLN5L6/9iwR4Y9Q67CZ7ToIlsqTc4q3RYFptfEXL vrT331I6U6UH4pxWGRN0OrpMBLJlO1xiYAiKHkRhCcKXznsm61F9L2uIJjZj59kB3XdmXC BC+MNK8ZfFtjUIGamqkzjO6ZJvzxkvE= ARC-Seal: i=1; s=arc-20220608; d=hostedemail.com; t=1658173029; a=rsa-sha256; cv=none; b=0t12F7UrxKkCgaFHqQfCPCGEx6/WE0qluaE0nGRBLJYpzkFeIHqI6ZAKzOjldqD64/A/zn c9wjSIjAVNlyJUJ6V5Otb8x1f6OUjm4QRhelKZ3i6RT95mVxFVvgIDtx5Od/jZF8wMDnse 5JVe4C1FgTCAXanJ+ICTpnI0P8/A/W0= X-Rspamd-Queue-Id: E091C1A0082 X-Rspam-User: X-HE-Tag-Orig: 1658173026-465922 Authentication-Results: imf19.hostedemail.com; dkim=pass header.d=gmail.com header.s=20210112 header.b=qG3NzQad; dmarc=pass (policy=none) header.from=gmail.com; spf=none (imf19.hostedemail.com: domain of MAILER-DAEMON@hostedemail.com has no SPF policy when checking 216.40.44.14) smtp.mailfrom=MAILER-DAEMON@hostedemail.com X-Rspamd-Server: rspam11 X-Stat-Signature: fm1issdfb8qpdq56etyq5xsu8ia91znn X-HE-Tag: 1658173029-854845 X-Bogosity: Ham, tests=bogofilter, spamicity=0.000000, version=1.2.4 Sender: owner-linux-mm@kvack.org Precedence: bulk X-Loop: owner-majordomo@kvack.org List-ID: From: Nadav Amit When using userfaultfd write-(un)protect ioctl, try to change the PTE to be writable. This would save a page-fault afterwards. Cc: Andrea Arcangeli Cc: Andrew Cooper Cc: Andrew Morton Cc: Andy Lutomirski Cc: Dave Hansen Cc: David Hildenbrand Cc: Peter Xu Cc: Peter Zijlstra Cc: Thomas Gleixner Cc: Will Deacon Cc: Yu Zhao Cc: Nick Piggin Signed-off-by: Nadav Amit Acked-by: Peter Xu --- mm/userfaultfd.c | 2 ++ 1 file changed, 2 insertions(+) diff --git a/mm/userfaultfd.c b/mm/userfaultfd.c index e0492f5f06a0..6013b217e9f3 100644 --- a/mm/userfaultfd.c +++ b/mm/userfaultfd.c @@ -799,6 +799,8 @@ int mwriteprotect_range(struct mm_struct *dst_mm, unsigned long start, cp_flags = enable_wp ? MM_CP_UFFD_WP : MM_CP_UFFD_WP_RESOLVE; if (uffd_flags & (UFFD_FLAGS_ACCESS_LIKELY|UFFD_FLAGS_WRITE_LIKELY)) cp_flags |= MM_CP_WILL_NEED; + if (!enable_wp && (uffd_flags & UFFD_FLAGS_WRITE_LIKELY)) + cp_flags |= MM_CP_TRY_CHANGE_WRITABLE; tlb_gather_mmu(&tlb, dst_mm); change_protection(&tlb, dst_vma, start, start + len, newprot, cp_flags); From patchwork Mon Jul 18 12:02:01 2022 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Nadav Amit X-Patchwork-Id: 12921682 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from kanga.kvack.org (kanga.kvack.org [205.233.56.17]) by smtp.lore.kernel.org (Postfix) with ESMTP id 9EDC1CCA481 for ; Mon, 18 Jul 2022 19:37:11 +0000 (UTC) Received: by kanga.kvack.org (Postfix) id D76BA6B0074; Mon, 18 Jul 2022 15:37:10 -0400 (EDT) Received: by kanga.kvack.org (Postfix, from userid 40) id CFE986B0078; Mon, 18 Jul 2022 15:37:10 -0400 (EDT) X-Delivered-To: int-list-linux-mm@kvack.org Received: by kanga.kvack.org (Postfix, from userid 63042) id BA77C6B0075; Mon, 18 Jul 2022 15:37:10 -0400 (EDT) X-Delivered-To: linux-mm@kvack.org Received: from relay.hostedemail.com (smtprelay0015.hostedemail.com [216.40.44.15]) by kanga.kvack.org (Postfix) with ESMTP id 95C1F6B0075 for ; Mon, 18 Jul 2022 15:37:10 -0400 (EDT) Received: from smtpin27.hostedemail.com (a10.router.float.18 [10.200.18.1]) by unirelay08.hostedemail.com (Postfix) with ESMTP id 6EFB6208B0 for ; Mon, 18 Jul 2022 19:37:10 +0000 (UTC) X-FDA: 79701229020.27.6B78DF0 Received: from relay3.hostedemail.com (smtprelay0013.hostedemail.com [216.40.44.13]) by imf08.hostedemail.com (Postfix) with ESMTP id 0E14616005D for ; Mon, 18 Jul 2022 19:37:09 +0000 (UTC) Received: from smtpin17.hostedemail.com (a10.router.float.18 [10.200.18.1]) by unirelay09.hostedemail.com (Postfix) with ESMTP id 121C234EC3 for ; Mon, 18 Jul 2022 19:37:08 +0000 (UTC) X-FDA: 79701228978.17.7B59532 Received: from mail-pl1-f180.google.com (mail-pl1-f180.google.com [209.85.214.180]) by imf07.hostedemail.com (Postfix) with ESMTP id 976354005E for ; Mon, 18 Jul 2022 19:37:08 +0000 (UTC) Received: by mail-pl1-f180.google.com with SMTP id x21so9962412plb.3 for ; Mon, 18 Jul 2022 12:37:08 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=20210112; h=from:to:cc:subject:date:message-id:in-reply-to:references :mime-version:content-transfer-encoding; bh=2Q49j7ZnjpJbe7tnNHIQw9SAVjFrSuB+lmoi9+DDawg=; b=bmUPVvdzZ44bZXvMXaJebmRi8Em4oQjsNDUw0UeAiKmw4D56qcZbcTGB0JlqUCsblt OEV1P1bs52wmG0gm8jY3wqSSUBITQIxayVYMmqk0JsmL+4pXoNVZ2FH1OsJW8kiZmEr5 o7V/37GKM2yOHpp0l6kahMySGW2vJQoZxf4VUu7MLfRdvtcC2k5ObYDHuP/XNyCSsW5v F/svHC9579Af/qJy3HoEVjxdNA4i/N87zD/iGJjAednCkdcRy5LqlQH1N6K6bzUuXkKo O+nHVRq37bCF+vMHywbE4q/ySw3g4zg984mjk3Zd66z+yPz9kQc8fmRVgs/2ZAhpyHge Nzfg== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20210112; h=x-gm-message-state:from:to:cc:subject:date:message-id:in-reply-to :references:mime-version:content-transfer-encoding; bh=2Q49j7ZnjpJbe7tnNHIQw9SAVjFrSuB+lmoi9+DDawg=; b=XQ6D7KBEccTRZG0A5cPQmLVEeY095wUTdEHKoxrCcg5Z9KpzDfnAKDOm09w9Sm1pOP bPN1NzWxms2G17C8BGGJdb1Hfx9pGmyXejfsbkN8IksQAZJM4WdeJ5wt+Om0gqde7ZDB b5z2BAHARzRwQ8st1/e/+7k7PWTExeeVm73oY72xdTjJNJwnvmMRuamW1aWUkR6sL0g8 zxN89eizYS4vGGsQMWA4TPK8F045fjF+h3w3GUtjf6jaRBiJ7J/38ZLYorgX48om+HG4 thw65vfaOrVWM5wg/q7amYUdO8gFf8KezZUkB75hYZ2xgza5FaA7EA9aUue1sL6mMkJw c00Q== X-Gm-Message-State: AJIora/KIuJMg8uv6jfRGcVTV34aZMpU6L/16SbvyjO03TqhJmx9wxVH rQfylhJXtp/UOie7XQ2JL1tlFyoQPdDugg== X-Google-Smtp-Source: AGRyM1tTWf0SQi53/iAko8YuS1u16C/ViaTRcBXow+cR8M+MN0AQg8swqeSra7RTo6Z8gZakiBjToQ== X-Received: by 2002:a17:90a:d195:b0:1ef:8eb2:4f4d with SMTP id fu21-20020a17090ad19500b001ef8eb24f4dmr39367516pjb.104.1658173027111; Mon, 18 Jul 2022 12:37:07 -0700 (PDT) Received: from sc2-haas01-esx0118.eng.vmware.com ([66.170.99.1]) by smtp.gmail.com with ESMTPSA id q6-20020a170902a3c600b0016bc4a6ce28sm9907887plb.98.2022.07.18.12.37.05 (version=TLS1_3 cipher=TLS_AES_256_GCM_SHA384 bits=256/256); Mon, 18 Jul 2022 12:37:06 -0700 (PDT) From: Nadav Amit X-Google-Original-From: Nadav Amit To: linux-mm@kvack.org Cc: linux-kernel@vger.kernel.org, Andrew Morton , Mike Rapoport , Axel Rasmussen , Nadav Amit , Andrea Arcangeli , Andrew Cooper , Andy Lutomirski , Dave Hansen , David Hildenbrand , Peter Xu , Peter Zijlstra , Thomas Gleixner , Will Deacon , Yu Zhao , Nick Piggin Subject: [RFC PATCH 03/14] mm/mprotect: allow exclusive anon pages to be writable Date: Mon, 18 Jul 2022 05:02:01 -0700 Message-Id: <20220718120212.3180-4-namit@vmware.com> X-Mailer: git-send-email 2.25.1 In-Reply-To: <20220718120212.3180-1-namit@vmware.com> References: <20220718120212.3180-1-namit@vmware.com> MIME-Version: 1.0 ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=hostedemail.com; s=arc-20220608; t=1658173030; h=from:from:sender:reply-to:subject:subject:date:date: message-id:message-id:to:to:cc:cc:mime-version:mime-version: content-type:content-transfer-encoding:content-transfer-encoding: in-reply-to:in-reply-to:references:references:dkim-signature; bh=2Q49j7ZnjpJbe7tnNHIQw9SAVjFrSuB+lmoi9+DDawg=; b=TIWAfZtIdhTOciZJHvYUmG9PYg85RtNxSzt5NZ73gSHRjERbZDor5kRQyOAVo3FcJHZVg4 yDQcJfpDTDCaO0PQF48vUnTp8OSgq9HcA2Vd1rocSRh5FVOFsALbZgLH62F7gPfew87ifO LfZIQXZbx8s4Xz4kpv5QtZOSGNErEJc= ARC-Seal: i=1; s=arc-20220608; d=hostedemail.com; t=1658173030; a=rsa-sha256; cv=none; b=JsywnCUKaKGOalR8D0vWsvzhXk0TvBOgsDRv2cQACyMCZ/vNj0Gtl8mOOcBf2UEiKK/Klv 4p9aKXDeOE3ecluDjZ8yAxFBFU5Yl07lmRRPscLJo7Af5gQtiD7iTYPMIVnzCnnQHlWES8 pXlugIhm/XtuzFtp7yF8b8HZtRGSCZA= ARC-Authentication-Results: i=1; imf08.hostedemail.com; dkim=pass header.d=gmail.com header.s=20210112 header.b=bmUPVvdz; dmarc=pass (policy=none) header.from=gmail.com; spf=none (imf08.hostedemail.com: domain of MAILER-DAEMON@hostedemail.com has no SPF policy when checking 216.40.44.13) smtp.mailfrom=MAILER-DAEMON@hostedemail.com X-Rspamd-Server: rspam10 X-Stat-Signature: o65rifqfj14paw46bwfpt5jgy4qsuht7 X-Rspam-User: X-Rspamd-Queue-Id: 0E14616005D X-HE-Tag-Orig: 1658173028-844299 Authentication-Results: imf08.hostedemail.com; dkim=pass header.d=gmail.com header.s=20210112 header.b=bmUPVvdz; dmarc=pass (policy=none) header.from=gmail.com; spf=none (imf08.hostedemail.com: domain of MAILER-DAEMON@hostedemail.com has no SPF policy when checking 216.40.44.13) smtp.mailfrom=MAILER-DAEMON@hostedemail.com X-HE-Tag: 1658173029-858071 X-Bogosity: Ham, tests=bogofilter, spamicity=0.000000, version=1.2.4 Sender: owner-linux-mm@kvack.org Precedence: bulk X-Loop: owner-majordomo@kvack.org List-ID: From: Nadav Amit Anonymous pages might have the dirty bit clear, but this should not prevent mprotect from making them writable if they are exclusive. Therefore, skip the test whether the page is dirty in this case. Cc: Andrea Arcangeli Cc: Andrew Cooper Cc: Andrew Morton Cc: Andy Lutomirski Cc: Dave Hansen Cc: David Hildenbrand Cc: Peter Xu Cc: Peter Zijlstra Cc: Thomas Gleixner Cc: Will Deacon Cc: Yu Zhao Cc: Nick Piggin Signed-off-by: Nadav Amit --- mm/mprotect.c | 5 +++-- 1 file changed, 3 insertions(+), 2 deletions(-) diff --git a/mm/mprotect.c b/mm/mprotect.c index 34c2dfb68c42..da5b9bf8204f 100644 --- a/mm/mprotect.c +++ b/mm/mprotect.c @@ -45,7 +45,7 @@ static inline bool can_change_pte_writable(struct vm_area_struct *vma, VM_BUG_ON(!(vma->vm_flags & VM_WRITE) || pte_write(pte)); - if (pte_protnone(pte) || !pte_dirty(pte)) + if (pte_protnone(pte)) return false; /* Do we need write faults for softdirty tracking? */ @@ -66,7 +66,8 @@ static inline bool can_change_pte_writable(struct vm_area_struct *vma, page = vm_normal_page(vma, addr, pte); if (!page || !PageAnon(page) || !PageAnonExclusive(page)) return false; - } + } else if (!pte_dirty(pte)) + return false; return true; } From patchwork Mon Jul 18 12:02:02 2022 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Nadav Amit X-Patchwork-Id: 12921683 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from kanga.kvack.org (kanga.kvack.org [205.233.56.17]) by smtp.lore.kernel.org (Postfix) with ESMTP id 1170FC433EF for ; Mon, 18 Jul 2022 19:37:13 +0000 (UTC) Received: by kanga.kvack.org (Postfix) id 980856B0075; Mon, 18 Jul 2022 15:37:11 -0400 (EDT) Received: by kanga.kvack.org (Postfix, from userid 40) id 909216B0078; Mon, 18 Jul 2022 15:37:11 -0400 (EDT) X-Delivered-To: int-list-linux-mm@kvack.org Received: by kanga.kvack.org (Postfix, from userid 63042) id 7804F8E0001; Mon, 18 Jul 2022 15:37:11 -0400 (EDT) X-Delivered-To: linux-mm@kvack.org Received: from relay.hostedemail.com (smtprelay0014.hostedemail.com [216.40.44.14]) by kanga.kvack.org (Postfix) with ESMTP id 6719B6B0075 for ; Mon, 18 Jul 2022 15:37:11 -0400 (EDT) Received: from smtpin02.hostedemail.com (a10.router.float.18 [10.200.18.1]) by unirelay08.hostedemail.com (Postfix) with ESMTP id 3099020C44 for ; Mon, 18 Jul 2022 19:37:11 +0000 (UTC) X-FDA: 79701229062.02.4AA2D9E Received: from relay3.hostedemail.com (smtprelay0011.hostedemail.com [216.40.44.11]) by imf09.hostedemail.com (Postfix) with ESMTP id C825B140004 for ; Mon, 18 Jul 2022 19:37:10 +0000 (UTC) Received: from smtpin25.hostedemail.com (a10.router.float.18 [10.200.18.1]) by unirelay06.hostedemail.com (Postfix) with ESMTP id 7A5E734D9B for ; Mon, 18 Jul 2022 19:37:10 +0000 (UTC) X-FDA: 79701229020.25.ACDC87B Received: from mail-pl1-f182.google.com (mail-pl1-f182.google.com [209.85.214.182]) by imf03.hostedemail.com (Postfix) with ESMTP id EBF0920051 for ; Mon, 18 Jul 2022 19:37:09 +0000 (UTC) Received: by mail-pl1-f182.google.com with SMTP id p8so191830plq.13 for ; Mon, 18 Jul 2022 12:37:09 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=20210112; h=from:to:cc:subject:date:message-id:in-reply-to:references :mime-version:content-transfer-encoding; bh=ZFfNYrVG625GOSA+id1AogN2HiBF2G/dud76HkUGe+4=; b=DuOhKFmRzGfZbq1nE2NE6Kc5a1/zJpTjh2RvqjQrCWls/7Qhm59F3Q5qnZAqPiZzON HTPLLExPZor3+WUD0GCeFJpaZ9HYJzT2HyRhOUdXKPSxPoB6NcCCp6BaC0PUnMozc1Yt 7yR/B8zLzmfdp4wBCIwoubwwq+IB152Z6aXNEBRtKOMqAeRb0Yg89EaxtZKRLj96Zhk2 zoX+JoTOX8z/qePogfLcdbultYnUpTJzA0NIOrLiUF3n+b/B8dbqeFNxtjocuKqlEg9I gVwTOuaGbUDx39ekLwLxOUStg09K/QpEk4bRVNpBu1NY3eLXRtkHF8N04DDoJYpcUW4c qdtw== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20210112; h=x-gm-message-state:from:to:cc:subject:date:message-id:in-reply-to :references:mime-version:content-transfer-encoding; bh=ZFfNYrVG625GOSA+id1AogN2HiBF2G/dud76HkUGe+4=; b=BFOQgqURyXt7NH63vtV6Chm2k0I5LsNjpaoY3MgZyQ8XGAive2xtwvFdN5s77lUxYm eX3Qiv6CI8iMIesK2o85AAjlZTyEemkelBEMwpzXuKTA74RcRZGbbNf2gYGRVzhTprea fFBcszt3fzz1NFjlFDjtFahC1Qmy2Q8sY5N+nWWFoD69xsmllOtcQGtPNFnvyfKWr90/ ux4wR/ZjQG2bEkuPRoDs/a39ooDfAfVFriIZeWNEtgDk/i4+0NPKdzutFDD/YBUKJ19+ /d26QirToQy1jSZKxC4wcDSyeZHiFIQ/Yjieut2/NBzaO664MzvcErAm8fifGqOhHAEN zQpQ== X-Gm-Message-State: AJIora/eArfS9GaltOQKgsBSXHdKGoD5KGpeZr1MF8efZtZJYIXVfvES 1Pm3Tibhuhs5nuA92OOSk73j9h/zgRq9Og== X-Google-Smtp-Source: AGRyM1vEHCEomLrXWGc5eN5sitXXmIDRxEROcYfVCHE3CQXw0PbDpM+yh+ClwMd2ENoAaXz/XYUQHw== X-Received: by 2002:a17:90a:9f8d:b0:1f0:253e:3ecf with SMTP id o13-20020a17090a9f8d00b001f0253e3ecfmr41557002pjp.33.1658173028683; Mon, 18 Jul 2022 12:37:08 -0700 (PDT) Received: from sc2-haas01-esx0118.eng.vmware.com ([66.170.99.1]) by smtp.gmail.com with ESMTPSA id q6-20020a170902a3c600b0016bc4a6ce28sm9907887plb.98.2022.07.18.12.37.07 (version=TLS1_3 cipher=TLS_AES_256_GCM_SHA384 bits=256/256); Mon, 18 Jul 2022 12:37:08 -0700 (PDT) From: Nadav Amit X-Google-Original-From: Nadav Amit To: linux-mm@kvack.org Cc: linux-kernel@vger.kernel.org, Andrew Morton , Mike Rapoport , Axel Rasmussen , Nadav Amit , Andrea Arcangeli , Andrew Cooper , Andy Lutomirski , Dave Hansen , David Hildenbrand , Peter Xu , Peter Zijlstra , Thomas Gleixner , Will Deacon , Yu Zhao , Nick Piggin Subject: [RFC PATCH 04/14] mm/mprotect: preserve write with MM_CP_TRY_CHANGE_WRITABLE Date: Mon, 18 Jul 2022 05:02:02 -0700 Message-Id: <20220718120212.3180-5-namit@vmware.com> X-Mailer: git-send-email 2.25.1 In-Reply-To: <20220718120212.3180-1-namit@vmware.com> References: <20220718120212.3180-1-namit@vmware.com> MIME-Version: 1.0 ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=hostedemail.com; s=arc-20220608; t=1658173030; h=from:from:sender:reply-to:subject:subject:date:date: message-id:message-id:to:to:cc:cc:mime-version:mime-version: content-type:content-transfer-encoding:content-transfer-encoding: in-reply-to:in-reply-to:references:references:dkim-signature; bh=ZFfNYrVG625GOSA+id1AogN2HiBF2G/dud76HkUGe+4=; b=R7MJfMAovCUv4S/toifyPKIPEXU6ET7ft+a6ck04eSGQDVg7zvCg3wSTCFVOxm7KnEoTuw UAy8yVdl2+B5LTjbWUIPB7OX6pjOvdulp5NkbjQzmpk1oM54RNPQUdEFbZM7v12KFlhFmM UvTXqrXRIf4rFjHwUe4wKOlgBVo7x2A= ARC-Authentication-Results: i=1; imf09.hostedemail.com; dkim=pass header.d=gmail.com header.s=20210112 header.b=DuOhKFmR; spf=none (imf09.hostedemail.com: domain of MAILER-DAEMON@hostedemail.com has no SPF policy when checking 216.40.44.11) smtp.mailfrom=MAILER-DAEMON@hostedemail.com; dmarc=pass (policy=none) header.from=gmail.com ARC-Seal: i=1; s=arc-20220608; d=hostedemail.com; t=1658173030; a=rsa-sha256; cv=none; b=beFqH6zN2xQT/Y1eiwDBvwAlAVAgV+bWzJNGJ3NzrVGZyXDjw/6QiDaFWaEYwVtcMTu2aN aUl0gzTCU/5bGUw8PK99YJFSTLHvvEay1jAeXvEB2QXS7ICcs1ChD6P8XsuCp2AOFZ/KYc IxuoumwzGl4N/Lh2Hu4a0fBYrvrogr0= X-HE-Tag-Orig: 1658173029-138361 Authentication-Results: imf09.hostedemail.com; dkim=pass header.d=gmail.com header.s=20210112 header.b=DuOhKFmR; spf=none (imf09.hostedemail.com: domain of MAILER-DAEMON@hostedemail.com has no SPF policy when checking 216.40.44.11) smtp.mailfrom=MAILER-DAEMON@hostedemail.com; dmarc=pass (policy=none) header.from=gmail.com X-Rspamd-Server: rspam01 X-Rspamd-Queue-Id: C825B140004 X-Stat-Signature: y9txmwpa73d7kpk8nzjo4n8kzwtkrfpy X-Rspam-User: X-HE-Tag: 1658173030-704531 X-Bogosity: Ham, tests=bogofilter, spamicity=0.000000, version=1.2.4 Sender: owner-linux-mm@kvack.org Precedence: bulk X-Loop: owner-majordomo@kvack.org List-ID: From: Nadav Amit When MM_CP_TRY_CHANGE_WRITABLE is used, change_pte_range() tries to set PTEs as writable. Yet, writable PTEs might still become read-only, due to various limitations of the logic that determines whether a PTE can become writable (see can_change_pte_writable()). Anyhow, it is much easier to keep the writable bit set when MM_CP_TRY_CHANGE_WRITABLE is used than to first clear it and then figure out whether it can be set again. Preserve the write-bit when MM_CP_TRY_CHANGE_WRITABLE is used, similarly to the way it is done with NUMA. Cc: Andrea Arcangeli Cc: Andrew Cooper Cc: Andrew Morton Cc: Andy Lutomirski Cc: Dave Hansen Cc: David Hildenbrand Cc: Peter Xu Cc: Peter Zijlstra Cc: Thomas Gleixner Cc: Will Deacon Cc: Yu Zhao Cc: Nick Piggin Signed-off-by: Nadav Amit --- mm/mprotect.c | 7 ++++--- 1 file changed, 4 insertions(+), 3 deletions(-) diff --git a/mm/mprotect.c b/mm/mprotect.c index da5b9bf8204f..92bfb17dcb8a 100644 --- a/mm/mprotect.c +++ b/mm/mprotect.c @@ -84,6 +84,7 @@ static unsigned long change_pte_range(struct mmu_gather *tlb, bool uffd_wp = cp_flags & MM_CP_UFFD_WP; bool uffd_wp_resolve = cp_flags & MM_CP_UFFD_WP_RESOLVE; bool will_need = cp_flags & MM_CP_WILL_NEED; + bool try_change_writable = cp_flags & MM_CP_TRY_CHANGE_WRITABLE; tlb_change_page_size(tlb, PAGE_SIZE); @@ -114,7 +115,8 @@ static unsigned long change_pte_range(struct mmu_gather *tlb, oldpte = *pte; if (pte_present(oldpte)) { pte_t ptent; - bool preserve_write = prot_numa && pte_write(oldpte); + bool preserve_write = (prot_numa || try_change_writable) && + pte_write(oldpte); /* * Avoid trapping faults against the zero or KSM @@ -190,8 +192,7 @@ static unsigned long change_pte_range(struct mmu_gather *tlb, * example, if a PTE is already dirty and no other * COW or special handling is required. */ - if ((cp_flags & MM_CP_TRY_CHANGE_WRITABLE) && - !pte_write(ptent) && + if (try_change_writable && !pte_write(ptent) && can_change_pte_writable(vma, addr, ptent)) { ptent = pte_mkwrite(ptent); if (will_need) From patchwork Mon Jul 18 12:02:04 2022 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Nadav Amit X-Patchwork-Id: 12921684 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from kanga.kvack.org (kanga.kvack.org [205.233.56.17]) by smtp.lore.kernel.org (Postfix) with ESMTP id 3954CC43334 for ; Mon, 18 Jul 2022 19:37:15 +0000 (UTC) Received: by kanga.kvack.org (Postfix) id D52026B0078; Mon, 18 Jul 2022 15:37:14 -0400 (EDT) Received: by kanga.kvack.org (Postfix, from userid 40) id CD89E6B007B; Mon, 18 Jul 2022 15:37:14 -0400 (EDT) X-Delivered-To: int-list-linux-mm@kvack.org Received: by kanga.kvack.org (Postfix, from userid 63042) id B50FD8E0001; Mon, 18 Jul 2022 15:37:14 -0400 (EDT) X-Delivered-To: linux-mm@kvack.org Received: from relay.hostedemail.com (smtprelay0014.hostedemail.com [216.40.44.14]) by kanga.kvack.org (Postfix) with ESMTP id A2EEE6B0078 for ; Mon, 18 Jul 2022 15:37:14 -0400 (EDT) Received: from smtpin30.hostedemail.com (a10.router.float.18 [10.200.18.1]) by unirelay10.hostedemail.com (Postfix) with ESMTP id 67AEC9E8 for ; Mon, 18 Jul 2022 19:37:14 +0000 (UTC) X-FDA: 79701229188.30.39FCCBD Received: from relay4.hostedemail.com (smtprelay0017.hostedemail.com [216.40.44.17]) by imf30.hostedemail.com (Postfix) with ESMTP id 0135280088 for ; Mon, 18 Jul 2022 19:37:13 +0000 (UTC) Received: from smtpin30.hostedemail.com (a10.router.float.18 [10.200.18.1]) by unirelay11.hostedemail.com (Postfix) with ESMTP id B05C68015F for ; Mon, 18 Jul 2022 19:37:13 +0000 (UTC) X-FDA: 79701229146.30.6F670F0 Received: from mail-pg1-f178.google.com (mail-pg1-f178.google.com [209.85.215.178]) by imf02.hostedemail.com (Postfix) with ESMTP id 3F2F480048 for ; Mon, 18 Jul 2022 19:37:13 +0000 (UTC) Received: by mail-pg1-f178.google.com with SMTP id 72so11548812pge.0 for ; Mon, 18 Jul 2022 12:37:12 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=20210112; h=from:to:cc:subject:date:message-id:in-reply-to:references :mime-version:content-transfer-encoding; bh=VCDxDoUSCrnTZdtdcyuvlf+NNtVJJIcoOQ0MDquBzAY=; b=WpzOa1eCI6gKdwOYRlP/7uAcBkP9V49tEy6TRy1l2Khiu+GZnm10Q5jyVbyYM1tYJ0 r1e+kLptD6I/s6ViDo6K3ZWQxtQqTivN9316p9KbowhoCjz6+CLLmK18EEI8JLQoLMTw 6hJP+pqwUio5gv5yYp+Q5f9oq+Nlk4vL7U3Nr2uzyliGndyYJu/aty4nLWwUOlzgE7f3 XKi59Yz5xUZS3IH7VwQS7RSyx2mkvafbnQ3f/hs9aloww2hNMJj8gDsL4ymdSB9RE3Xb MzJM2PoJh3hUc1G7Ka+jmv39doe16fxqrhkTV1xozTVG+hkSSQ0CDqVkr23SSAAypVrA ptew== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20210112; h=x-gm-message-state:from:to:cc:subject:date:message-id:in-reply-to :references:mime-version:content-transfer-encoding; bh=VCDxDoUSCrnTZdtdcyuvlf+NNtVJJIcoOQ0MDquBzAY=; b=JJX8wuNuVvEFQEzJWK24LBosMYqf5eYzSTOooNnh1byb/0WoTk3PASShZE3qYuAndo S+/fQe+svwmG1kLoyKV6vmWsvhcHXpbAz431v8STM03EvICyUuNTLKpHqYEPLMRHqBKK Nbg5AB0HcJuVXIaAIpYRd6jxhJm/2ez0oWtqZkZrJLEDvuhzVcuYtKNeJC/pcJdDSyNU 9i9vRWw/nGEVE38gtuH7zvZEr8AdeJVSflkYUfVjyLw1iS6rbnOv8zlCXgDiCisHj0eL PTh1pqHOs5k665SLOm1p/kL4t0pEuYUK6Z1DdwiGdY4npKpicyEbKV3ZyqeptzCaJj7k m3gA== X-Gm-Message-State: AJIora9SG1qsCGKAbyzrv7aVtoUa0jnGdkXItg/rDB+mfcmNRgVG9g9x N3uJ/4K/VC9gLC1NxtFmpD/T3ibdfGv1vg== X-Google-Smtp-Source: AGRyM1vGJivctvX5ZkdLIJ4O9MXoDMilDP3XL2pp+bNJOG2Zy6lN3VLUtfwzhoCDKo97VJCK/R1iTA== X-Received: by 2002:a63:fc48:0:b0:40d:ad0a:a868 with SMTP id r8-20020a63fc48000000b0040dad0aa868mr25533373pgk.204.1658173031839; Mon, 18 Jul 2022 12:37:11 -0700 (PDT) Received: from sc2-haas01-esx0118.eng.vmware.com ([66.170.99.1]) by smtp.gmail.com with ESMTPSA id q6-20020a170902a3c600b0016bc4a6ce28sm9907887plb.98.2022.07.18.12.37.10 (version=TLS1_3 cipher=TLS_AES_256_GCM_SHA384 bits=256/256); Mon, 18 Jul 2022 12:37:11 -0700 (PDT) From: Nadav Amit X-Google-Original-From: Nadav Amit To: linux-mm@kvack.org Cc: linux-kernel@vger.kernel.org, Andrew Morton , Mike Rapoport , Axel Rasmussen , Nadav Amit , Andrea Arcangeli , Andrew Cooper , Andy Lutomirski , Dave Hansen , David Hildenbrand , Peter Xu , Peter Zijlstra , Thomas Gleixner , Will Deacon , Yu Zhao , Nick Piggin Subject: [RFC PATCH 06/14] mm/rmap: avoid flushing on page_vma_mkclean_one() when possible Date: Mon, 18 Jul 2022 05:02:04 -0700 Message-Id: <20220718120212.3180-7-namit@vmware.com> X-Mailer: git-send-email 2.25.1 In-Reply-To: <20220718120212.3180-1-namit@vmware.com> References: <20220718120212.3180-1-namit@vmware.com> MIME-Version: 1.0 ARC-Seal: i=1; s=arc-20220608; d=hostedemail.com; t=1658173034; a=rsa-sha256; cv=none; b=42joCuQNU6DadOs1uQy0TWio77iMse9jrMEcIrx2f0B5b/09XDJn3kl4x5KhSlafw4qXHG 4ZgRPLhervFqm7bg2D5FO23Z/Kq8mKu3U50FaIi+w7Itrei6/AgTeVXEz1qUR3na0ZgwhC hdPEVWVaEm7c5bcrft7vFn31fm4P2v0= ARC-Authentication-Results: i=1; imf30.hostedemail.com; dkim=pass header.d=gmail.com header.s=20210112 header.b=WpzOa1eC; dmarc=pass (policy=none) header.from=gmail.com; spf=none (imf30.hostedemail.com: domain of MAILER-DAEMON@hostedemail.com has no SPF policy when checking 216.40.44.17) smtp.mailfrom=MAILER-DAEMON@hostedemail.com ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=hostedemail.com; s=arc-20220608; t=1658173034; h=from:from:sender:reply-to:subject:subject:date:date: message-id:message-id:to:to:cc:cc:mime-version:mime-version: content-type:content-transfer-encoding:content-transfer-encoding: in-reply-to:in-reply-to:references:references:dkim-signature; bh=VCDxDoUSCrnTZdtdcyuvlf+NNtVJJIcoOQ0MDquBzAY=; b=5oMKLUMq4HkmhXyF3rpxOj52u6E1LHb7DB0f9MX0dLHq8IoGYfhL8jCBDWG8jCIvY/Ydc8 4pS0nShvOmpc27mfNGH5Fvl0UrOYFY2y5FMoSpLRXG3sN6GEfxm6Y1xnQ5y05/yqnZhQbS So9cxJ8xSYSjb6qtU0vRuOD8WDUpeKc= X-Rspamd-Queue-Id: 0135280088 X-Rspam-User: X-HE-Tag-Orig: 1658173033-655444 Authentication-Results: imf30.hostedemail.com; dkim=pass header.d=gmail.com header.s=20210112 header.b=WpzOa1eC; dmarc=pass (policy=none) header.from=gmail.com; spf=none (imf30.hostedemail.com: domain of MAILER-DAEMON@hostedemail.com has no SPF policy when checking 216.40.44.17) smtp.mailfrom=MAILER-DAEMON@hostedemail.com X-Rspamd-Server: rspam08 X-Stat-Signature: 1zhztpqsuretrmaen4g65tj6imd9r1p4 X-HE-Tag: 1658173033-875377 X-Bogosity: Ham, tests=bogofilter, spamicity=0.000000, version=1.2.4 Sender: owner-linux-mm@kvack.org Precedence: bulk X-Loop: owner-majordomo@kvack.org List-ID: From: Nadav Amit x86 is capable to avoid TLB flush on clean writable entries. page_vma_mkclean_one() does not take advantage of this behavior. Adapt it. Cc: Andrea Arcangeli Cc: Andrew Cooper Cc: Andrew Morton Cc: Andy Lutomirski Cc: Dave Hansen Cc: David Hildenbrand Cc: Peter Xu Cc: Peter Zijlstra Cc: Thomas Gleixner Cc: Will Deacon Cc: Yu Zhao Cc: Nick Piggin Signed-off-by: Nadav Amit --- mm/rmap.c | 16 ++++++++++++---- 1 file changed, 12 insertions(+), 4 deletions(-) diff --git a/mm/rmap.c b/mm/rmap.c index 83172ee0ea35..23997c387858 100644 --- a/mm/rmap.c +++ b/mm/rmap.c @@ -961,17 +961,25 @@ static int page_vma_mkclean_one(struct page_vma_mapped_walk *pvmw) address = pvmw->address; if (pvmw->pte) { - pte_t entry; + pte_t entry, oldpte; pte_t *pte = pvmw->pte; if (!pte_dirty(*pte) && !pte_write(*pte)) continue; flush_cache_page(vma, address, pte_pfn(*pte)); - entry = ptep_clear_flush(vma, address, pte); - entry = pte_wrprotect(entry); + oldpte = ptep_modify_prot_start(pvmw->vma, address, + pte); + + entry = pte_wrprotect(oldpte); entry = pte_mkclean(entry); - set_pte_at(vma->vm_mm, address, pte, entry); + + if (pte_needs_flush(oldpte, entry) || + mm_tlb_flush_pending(vma->vm_mm)) + flush_tlb_page(vma, address); + + ptep_modify_prot_commit(vma, address, pte, oldpte, + entry); ret = 1; } else { #ifdef CONFIG_TRANSPARENT_HUGEPAGE From patchwork Mon Jul 18 12:02:05 2022 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Nadav Amit X-Patchwork-Id: 12921685 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from kanga.kvack.org (kanga.kvack.org [205.233.56.17]) by smtp.lore.kernel.org (Postfix) with ESMTP id 8AAAEC433EF for ; Mon, 18 Jul 2022 19:37:16 +0000 (UTC) Received: by kanga.kvack.org (Postfix) id 22D626B007B; Mon, 18 Jul 2022 15:37:16 -0400 (EDT) Received: by kanga.kvack.org (Postfix, from userid 40) id 1667F6B007D; Mon, 18 Jul 2022 15:37:16 -0400 (EDT) X-Delivered-To: int-list-linux-mm@kvack.org Received: by kanga.kvack.org (Postfix, from userid 63042) id EFA9B6B007E; Mon, 18 Jul 2022 15:37:15 -0400 (EDT) X-Delivered-To: linux-mm@kvack.org Received: from relay.hostedemail.com (smtprelay0013.hostedemail.com [216.40.44.13]) by kanga.kvack.org (Postfix) with ESMTP id DC7BE6B007B for ; Mon, 18 Jul 2022 15:37:15 -0400 (EDT) Received: from smtpin08.hostedemail.com (a10.router.float.18 [10.200.18.1]) by unirelay09.hostedemail.com (Postfix) with ESMTP id B785E34E8A for ; Mon, 18 Jul 2022 19:37:15 +0000 (UTC) X-FDA: 79701229230.08.3EA0D29 Received: from relay4.hostedemail.com (smtprelay0015.hostedemail.com [216.40.44.15]) by imf18.hostedemail.com (Postfix) with ESMTP id 51C171C0010 for ; Mon, 18 Jul 2022 19:37:15 +0000 (UTC) Received: from smtpin01.hostedemail.com (a10.router.float.18 [10.200.18.1]) by unirelay09.hostedemail.com (Postfix) with ESMTP id 0783D34E8A for ; Mon, 18 Jul 2022 19:37:15 +0000 (UTC) X-FDA: 79701229230.01.54B86B5 Received: from mail-pf1-f175.google.com (mail-pf1-f175.google.com [209.85.210.175]) by imf01.hostedemail.com (Postfix) with ESMTP id A2E6D40067 for ; Mon, 18 Jul 2022 19:37:14 +0000 (UTC) Received: by mail-pf1-f175.google.com with SMTP id b9so11547282pfp.10 for ; Mon, 18 Jul 2022 12:37:14 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=20210112; h=from:to:cc:subject:date:message-id:in-reply-to:references :mime-version:content-transfer-encoding; bh=ii9tnumcH6f4uVFiJaNvbNbCHj/eOFwFpwV99AWDIWY=; b=gF1O4Ap8c6u3Wl7EJAnk5cpari+SSbCrCwIHnkKi2iGutzFYOra9oMYtovijltSliX f9n+3+zOazFlTlGbgoP7cASWhUOA3yRUCNf36n4asDLBRZ2n5F6V77g8Ih0ng07spWdH HCMUUeGMN6x1yYhcSmRTYhGUaieuitiLXWFPOh9CH2CMEqYcyByqPimLVW+wP36dQPZe 1RJE9T2eaZawBkz5E8JuXZAG/5xKtgo8+wNL/iuOrcsSXLKNEDQJjqpz/5Yuv3v3GVFQ DvCcYoxTnP9tWp+lBd1uyqN2PAo5ihDVfdOn4Uj1GiSnYTj0YlvWIZ5h14BKB8GCHPjA 0gFA== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20210112; h=x-gm-message-state:from:to:cc:subject:date:message-id:in-reply-to :references:mime-version:content-transfer-encoding; bh=ii9tnumcH6f4uVFiJaNvbNbCHj/eOFwFpwV99AWDIWY=; b=iv8q0B49VRrQBKVK33Zi+a97CVTSQ/vXQurrRMamXtyL2GANUdiE02IImXOZ4zWgsL 4Cfz/MdwC73of8Ag1AjLmgmXGoKqoPgCS7u+ryRhwL28M3gHTcBqwpIlInI/5WbN0Aij e7Zct2kRLXvf7OmIrgTyzYOL5A/S821hppeeZqqZfseass0weJKRLu4WJHz3juESWw67 PhOStWNxdPyReJaRV390LyGUZAocZUwtnApvRktn6IZ+juLH+lJxvjijuvV8ozz1I8U9 FK7uxAOAca+8l48w14eBNRIl6GfxbzY9Bf7ZXBYl+XWMx2a8HMSgTL5Lxbxg/2q4KNZ4 KMfQ== X-Gm-Message-State: AJIora/P6+wDUnl7kY41usssUA8kAS8+bFNxDy3U0fxq2+f/OKMtYVqw P1f2AzleAZUKn38cNuMZKiloybiCmJOb7Q== X-Google-Smtp-Source: AGRyM1tECueM0uV0ckJl2HFWgTNipKEQZcrDzMSchFkRcqY0hBmCA7xorYY92YcZXwgaVycH49hQXg== X-Received: by 2002:a05:6a00:3388:b0:52a:c018:6cdf with SMTP id cm8-20020a056a00338800b0052ac0186cdfmr29910792pfb.55.1658173033302; Mon, 18 Jul 2022 12:37:13 -0700 (PDT) Received: from sc2-haas01-esx0118.eng.vmware.com ([66.170.99.1]) by smtp.gmail.com with ESMTPSA id q6-20020a170902a3c600b0016bc4a6ce28sm9907887plb.98.2022.07.18.12.37.11 (version=TLS1_3 cipher=TLS_AES_256_GCM_SHA384 bits=256/256); Mon, 18 Jul 2022 12:37:12 -0700 (PDT) From: Nadav Amit X-Google-Original-From: Nadav Amit To: linux-mm@kvack.org Cc: linux-kernel@vger.kernel.org, Andrew Morton , Mike Rapoport , Axel Rasmussen , Nadav Amit , Andrea Arcangeli , Andrew Cooper , Andy Lutomirski , Dave Hansen , David Hildenbrand , Peter Xu , Peter Zijlstra , Thomas Gleixner , Will Deacon , Yu Zhao , Nick Piggin Subject: [RFC PATCH 07/14] mm: do fix spurious page-faults for instruction faults Date: Mon, 18 Jul 2022 05:02:05 -0700 Message-Id: <20220718120212.3180-8-namit@vmware.com> X-Mailer: git-send-email 2.25.1 In-Reply-To: <20220718120212.3180-1-namit@vmware.com> References: <20220718120212.3180-1-namit@vmware.com> MIME-Version: 1.0 ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=hostedemail.com; s=arc-20220608; t=1658173035; h=from:from:sender:reply-to:subject:subject:date:date: message-id:message-id:to:to:cc:cc:mime-version:mime-version: content-type:content-transfer-encoding:content-transfer-encoding: in-reply-to:in-reply-to:references:references:dkim-signature; bh=ii9tnumcH6f4uVFiJaNvbNbCHj/eOFwFpwV99AWDIWY=; b=Qc/vmOjLrqE8B1NeJT/OJhXug1kVjAwEAK3K7xFX1w3D5vv08AgVBLkbjglK4aH+G6vjlD +GU0sF4IAza5vts8ym1OtcFfj3rBrhpKowNFQHjtrIB1HQibneKr+DRVMAXLHVaOyn7iHm alhdmz/8l6bFaaCr+q7LEyCOrM/a+gM= ARC-Authentication-Results: i=1; imf18.hostedemail.com; dkim=pass header.d=gmail.com header.s=20210112 header.b=gF1O4Ap8; spf=none (imf18.hostedemail.com: domain of MAILER-DAEMON@hostedemail.com has no SPF policy when checking 216.40.44.15) smtp.mailfrom=MAILER-DAEMON@hostedemail.com; dmarc=pass (policy=none) header.from=gmail.com ARC-Seal: i=1; s=arc-20220608; d=hostedemail.com; t=1658173035; a=rsa-sha256; cv=none; b=XQJ2TkGfDEyJpmuE5qsYyedskpeVN2YHzCx8rdMTmL/RRbjY6ctBdrd2jMhQhVLRJwNWqC AKuf5U6PDRB5MLJqnF9jKRqGrrSRZ+pz7OHwVZw7LG9p1/FslpTe+mMZx0YHXR9MQ25bWy LZFZ3kLypPra0LgXuZnmtSX3KbmL+P0= X-HE-Tag-Orig: 1658173034-732500 Authentication-Results: imf18.hostedemail.com; dkim=pass header.d=gmail.com header.s=20210112 header.b=gF1O4Ap8; spf=none (imf18.hostedemail.com: domain of MAILER-DAEMON@hostedemail.com has no SPF policy when checking 216.40.44.15) smtp.mailfrom=MAILER-DAEMON@hostedemail.com; dmarc=pass (policy=none) header.from=gmail.com X-Rspamd-Server: rspam01 X-Rspamd-Queue-Id: 51C171C0010 X-Stat-Signature: msy4mhmouzp4jm65ii7wgu59spt1g99x X-Rspam-User: X-HE-Tag: 1658173035-205881 X-Bogosity: Ham, tests=bogofilter, spamicity=0.000000, version=1.2.4 Sender: owner-linux-mm@kvack.org Precedence: bulk X-Loop: owner-majordomo@kvack.org List-ID: From: Nadav Amit The next patches might cause spurious instruction faults on x86. To prevent them from occurring too much, call flush_tlb_fix_spurious_fault() for page-faults on code fetching as well. The callee is expected to do a full flush, or whatever is necessary to avoid further TLB flushes. Cc: Andrea Arcangeli Cc: Andrew Cooper Cc: Andrew Morton Cc: Andy Lutomirski Cc: Dave Hansen Cc: David Hildenbrand Cc: Peter Xu Cc: Peter Zijlstra Cc: Thomas Gleixner Cc: Will Deacon Cc: Yu Zhao Cc: Nick Piggin Signed-off-by: Nadav Amit --- mm/memory.c | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/mm/memory.c b/mm/memory.c index 31ec3f0071a2..152a47876c36 100644 --- a/mm/memory.c +++ b/mm/memory.c @@ -4924,7 +4924,7 @@ static vm_fault_t handle_pte_fault(struct vm_fault *vmf) * This still avoids useless tlb flushes for .text page faults * with threads. */ - if (vmf->flags & FAULT_FLAG_WRITE) + if (vmf->flags & (FAULT_FLAG_WRITE|FAULT_FLAG_INSTRUCTION)) flush_tlb_fix_spurious_fault(vmf->vma, vmf->address); } unlock: From patchwork Mon Jul 18 12:02:06 2022 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Nadav Amit X-Patchwork-Id: 12921686 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from kanga.kvack.org (kanga.kvack.org [205.233.56.17]) by smtp.lore.kernel.org (Postfix) with ESMTP id 4D5E4C43334 for ; Mon, 18 Jul 2022 19:37:18 +0000 (UTC) Received: by kanga.kvack.org (Postfix) id DC69D6B007D; Mon, 18 Jul 2022 15:37:17 -0400 (EDT) Received: by kanga.kvack.org (Postfix, from userid 40) id D26BF6B007E; Mon, 18 Jul 2022 15:37:17 -0400 (EDT) X-Delivered-To: int-list-linux-mm@kvack.org Received: by kanga.kvack.org (Postfix, from userid 63042) id B52196B0080; Mon, 18 Jul 2022 15:37:17 -0400 (EDT) X-Delivered-To: linux-mm@kvack.org Received: from relay.hostedemail.com (smtprelay0015.hostedemail.com [216.40.44.15]) by kanga.kvack.org (Postfix) with ESMTP id A1E196B007D for ; Mon, 18 Jul 2022 15:37:17 -0400 (EDT) Received: from smtpin30.hostedemail.com (a10.router.float.18 [10.200.18.1]) by unirelay12.hostedemail.com (Postfix) with ESMTP id 759AB120496 for ; Mon, 18 Jul 2022 19:37:17 +0000 (UTC) X-FDA: 79701229314.30.44CDC89 Received: from relay3.hostedemail.com (smtprelay0011.hostedemail.com [216.40.44.11]) by imf03.hostedemail.com (Postfix) with ESMTP id 14B922003B for ; Mon, 18 Jul 2022 19:37:16 +0000 (UTC) Received: from smtpin04.hostedemail.com (a10.router.float.18 [10.200.18.1]) by unirelay12.hostedemail.com (Postfix) with ESMTP id A8A2112092C for ; Mon, 18 Jul 2022 19:37:16 +0000 (UTC) X-FDA: 79701229272.04.5DCB66E Received: from mail-pf1-f176.google.com (mail-pf1-f176.google.com [209.85.210.176]) by imf12.hostedemail.com (Postfix) with ESMTP id 3FF1C40083 for ; Mon, 18 Jul 2022 19:37:16 +0000 (UTC) Received: by mail-pf1-f176.google.com with SMTP id o12so11562043pfp.5 for ; Mon, 18 Jul 2022 12:37:15 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=20210112; h=from:to:cc:subject:date:message-id:in-reply-to:references :mime-version:content-transfer-encoding; bh=pKws8yGRo63vLoo5PvEhs398wa0pzhCZmm6r0b5wNuc=; b=qMpMBqoEJowM7hHEVUp4urmfW4BmRk2K5WRu6rjNNtcGI/q+u2qcbzoTj0dymzU/YN GICq1D6c+IwKTAtMkM/ialUwiAbHd2qyav/oi4Lb2/tQir2MH5L1/9FN/OSYwOL11ogc J95A/oNh44qpCzubW1bL4lW6pv2SZPldzPpGkHfXoFe/mJfk3/K11+ZWWai+ZuaWUxuj ZktYEG+NaXZRrAZnM4dfOCYWFErlRI/RQXv4NrUjDJOmLWof5hAYkt6ED39XDfIu11Ol QDBHOQm0nz2tV3l6dVnFXwCweU/mVRE1/C2vxelapt4M8n9mqIfwuSDyhKsokwR/n6pa 5lRA== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20210112; h=x-gm-message-state:from:to:cc:subject:date:message-id:in-reply-to :references:mime-version:content-transfer-encoding; bh=pKws8yGRo63vLoo5PvEhs398wa0pzhCZmm6r0b5wNuc=; b=PQy3lRxFlTEKaONKKmm4rXArEN5H51fbGtxl2pFXKSCnMzW9eLE4ArPJZ/eard0loC e96ml50LySoceuAPmIBSyKSGABQfqNLHi8tBFTLy7SVHXlvjyqsbK4n6nTlmfqeGXTQd BKStk0yX49IMLwnrmXwTlLY6YCTZZDgsIei6nTCGgsva8sccVvi093NhJJYeETGEzcyf saYA/wYImsMQyLfmRJEm5ZIvBVLd765pdqtLKcRHWCx+OaW7TQ/mArCXcJYDsqR7721V qSc+gb236tkJg9P8NH7fkYFEh3X+u/MQdHyBRI2J+Zu0+L7p/cxWQKvNX3BkQ96J8pBD lQTw== X-Gm-Message-State: AJIora+BKsdedRmHnLXdELTj/brhhdC5YfHDB6QH60WBsMl7Mne2xw7Q 2ofU2c3OMnKSYj71yONZFtgvoQZGaqTNpg== X-Google-Smtp-Source: AGRyM1v2EDFEp/irIA5oLmEjHJRkKj6neMr06kHIqrOuqyOom6q/rRWy9YaOFocxFmxkQmdYOSbCBw== X-Received: by 2002:aa7:9256:0:b0:52a:cbf7:43ea with SMTP id 22-20020aa79256000000b0052acbf743eamr29902665pfp.7.1658173034758; Mon, 18 Jul 2022 12:37:14 -0700 (PDT) Received: from sc2-haas01-esx0118.eng.vmware.com ([66.170.99.1]) by smtp.gmail.com with ESMTPSA id q6-20020a170902a3c600b0016bc4a6ce28sm9907887plb.98.2022.07.18.12.37.13 (version=TLS1_3 cipher=TLS_AES_256_GCM_SHA384 bits=256/256); Mon, 18 Jul 2022 12:37:14 -0700 (PDT) From: Nadav Amit X-Google-Original-From: Nadav Amit To: linux-mm@kvack.org Cc: linux-kernel@vger.kernel.org, Andrew Morton , Mike Rapoport , Axel Rasmussen , Nadav Amit , Andrea Arcangeli , Andrew Cooper , Andy Lutomirski , Dave Hansen , David Hildenbrand , Peter Xu , Peter Zijlstra , Thomas Gleixner , Will Deacon , Yu Zhao , Nick Piggin Subject: [RFC PATCH 08/14] x86/mm: introduce flush_tlb_fix_spurious_fault Date: Mon, 18 Jul 2022 05:02:06 -0700 Message-Id: <20220718120212.3180-9-namit@vmware.com> X-Mailer: git-send-email 2.25.1 In-Reply-To: <20220718120212.3180-1-namit@vmware.com> References: <20220718120212.3180-1-namit@vmware.com> MIME-Version: 1.0 ARC-Seal: i=1; s=arc-20220608; d=hostedemail.com; t=1658173037; a=rsa-sha256; cv=none; b=PCg+9E2eR19+/srxxEsTw9hQHSclFcOs857dgI/MVhg+FQHTlUbb4zfhR1YD3fUHuOpiJx r17rrBLj2wyEM1bAJKzDhRBla87FqJcUJZOYhJJ1lTQLaVM/PXzI/hbmpQYsZg3lgmsFZm uHcPtGNd2iQPaHfSM3VFCV9upO4GxwY= ARC-Authentication-Results: i=1; imf03.hostedemail.com; dkim=pass header.d=gmail.com header.s=20210112 header.b=qMpMBqoE; spf=none (imf03.hostedemail.com: domain of MAILER-DAEMON@hostedemail.com has no SPF policy when checking 216.40.44.11) smtp.mailfrom=MAILER-DAEMON@hostedemail.com; dmarc=pass (policy=none) header.from=gmail.com ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=hostedemail.com; s=arc-20220608; t=1658173037; h=from:from:sender:reply-to:subject:subject:date:date: message-id:message-id:to:to:cc:cc:mime-version:mime-version: content-type:content-transfer-encoding:content-transfer-encoding: in-reply-to:in-reply-to:references:references:dkim-signature; bh=pKws8yGRo63vLoo5PvEhs398wa0pzhCZmm6r0b5wNuc=; b=sk4xmPf1gO83MsCkFsLrBLxkDR+Rr2HaaLS0iuwY+MhyZZTwD/cZ+RMYMi5w8QdPX90Kyx hsm7c0G165R9mBaTXgskOU9N3au1oTzUE+NsvsDCCM7mxEeBRFqadaGnxOkR4sW4zPn6kf lzp3KqsXTegZtWDuYz0WB5EDJdlObW0= X-Rspamd-Queue-Id: 14B922003B X-HE-Tag-Orig: 1658173036-305858 Authentication-Results: imf03.hostedemail.com; dkim=pass header.d=gmail.com header.s=20210112 header.b=qMpMBqoE; spf=none (imf03.hostedemail.com: domain of MAILER-DAEMON@hostedemail.com has no SPF policy when checking 216.40.44.11) smtp.mailfrom=MAILER-DAEMON@hostedemail.com; dmarc=pass (policy=none) header.from=gmail.com X-Rspamd-Server: rspam05 X-Rspam-User: X-Stat-Signature: mx1wbyh5ubzxmbukizweyjojn3ejjsgu X-HE-Tag: 1658173036-936677 X-Bogosity: Ham, tests=bogofilter, spamicity=0.000000, version=1.2.4 Sender: owner-linux-mm@kvack.org Precedence: bulk X-Loop: owner-majordomo@kvack.org List-ID: From: Nadav Amit The next patches introduce relaxed TLB flushes for x86, which would require a full TLB flush upon spurious page-fault. If a spurious page-fault occurs on x86, check if the local TLB generation is out of sync and perform a TLB flush if needed. Cc: Andrea Arcangeli Cc: Andrew Cooper Cc: Andrew Morton Cc: Andy Lutomirski Cc: Dave Hansen Cc: David Hildenbrand Cc: Peter Xu Cc: Peter Zijlstra Cc: Thomas Gleixner Cc: Will Deacon Cc: Yu Zhao Cc: Nick Piggin Signed-off-by: Nadav Amit --- arch/x86/include/asm/pgtable.h | 4 +++- arch/x86/mm/tlb.c | 17 +++++++++++++++++ 2 files changed, 20 insertions(+), 1 deletion(-) diff --git a/arch/x86/include/asm/pgtable.h b/arch/x86/include/asm/pgtable.h index 44e2d6f1dbaa..1fbdaff1bb7a 100644 --- a/arch/x86/include/asm/pgtable.h +++ b/arch/x86/include/asm/pgtable.h @@ -1079,7 +1079,9 @@ static inline void ptep_set_wrprotect(struct mm_struct *mm, clear_bit(_PAGE_BIT_RW, (unsigned long *)&ptep->pte); } -#define flush_tlb_fix_spurious_fault(vma, address) do { } while (0) +extern void flush_tlb_fix_spurious_fault(struct vm_area_struct *vma, + unsigned long address); +#define flush_tlb_fix_spurious_fault flush_tlb_fix_spurious_fault #define mk_pmd(page, pgprot) pfn_pmd(page_to_pfn(page), (pgprot)) diff --git a/arch/x86/mm/tlb.c b/arch/x86/mm/tlb.c index d400b6d9d246..ff3bcc55435e 100644 --- a/arch/x86/mm/tlb.c +++ b/arch/x86/mm/tlb.c @@ -955,6 +955,23 @@ static void put_flush_tlb_info(void) #endif } +void flush_tlb_fix_spurious_fault(struct vm_area_struct *vma, + unsigned long address) +{ + u32 loaded_mm_asid = this_cpu_read(cpu_tlbstate.loaded_mm_asid); + u64 mm_tlb_gen = atomic64_read(&vma->vm_mm->context.tlb_gen); + u64 local_tlb_gen = this_cpu_read(cpu_tlbstate.ctxs[loaded_mm_asid].tlb_gen); + struct flush_tlb_info *info; + + if (local_tlb_gen == mm_tlb_gen) + return; + + preempt_disable(); + info = get_flush_tlb_info(NULL, 0, TLB_FLUSH_ALL, 0, false, 0); + flush_tlb_func(info); + preempt_enable(); +} + void flush_tlb_mm_range(struct mm_struct *mm, unsigned long start, unsigned long end, unsigned int stride_shift, bool freed_tables) From patchwork Mon Jul 18 12:02:08 2022 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Nadav Amit X-Patchwork-Id: 12921687 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from kanga.kvack.org (kanga.kvack.org [205.233.56.17]) by smtp.lore.kernel.org (Postfix) with ESMTP id 6BE63C43334 for ; Mon, 18 Jul 2022 19:37:21 +0000 (UTC) Received: by kanga.kvack.org (Postfix) id 0A3D86B007E; Mon, 18 Jul 2022 15:37:21 -0400 (EDT) Received: by kanga.kvack.org (Postfix, from userid 40) id 0545A6B0080; Mon, 18 Jul 2022 15:37:20 -0400 (EDT) X-Delivered-To: int-list-linux-mm@kvack.org Received: by kanga.kvack.org (Postfix, from userid 63042) id E36E36B0081; Mon, 18 Jul 2022 15:37:20 -0400 (EDT) X-Delivered-To: linux-mm@kvack.org Received: from relay.hostedemail.com (smtprelay0012.hostedemail.com [216.40.44.12]) by kanga.kvack.org (Postfix) with ESMTP id CFA196B007E for ; Mon, 18 Jul 2022 15:37:20 -0400 (EDT) Received: from smtpin16.hostedemail.com (a10.router.float.18 [10.200.18.1]) by unirelay10.hostedemail.com (Postfix) with ESMTP id A42FF809 for ; Mon, 18 Jul 2022 19:37:20 +0000 (UTC) X-FDA: 79701229440.16.82ACFF7 Received: from relay5.hostedemail.com (smtprelay0016.hostedemail.com [216.40.44.16]) by imf05.hostedemail.com (Postfix) with ESMTP id 0276310006D for ; Mon, 18 Jul 2022 19:37:19 +0000 (UTC) Received: from smtpin01.hostedemail.com (a10.router.float.18 [10.200.18.1]) by unirelay07.hostedemail.com (Postfix) with ESMTP id A8B0F20B4C for ; Mon, 18 Jul 2022 19:37:19 +0000 (UTC) X-FDA: 79701229398.01.0A437E0 Received: from mail-pj1-f43.google.com (mail-pj1-f43.google.com [209.85.216.43]) by imf02.hostedemail.com (Postfix) with ESMTP id 388BA80043 for ; Mon, 18 Jul 2022 19:37:19 +0000 (UTC) Received: by mail-pj1-f43.google.com with SMTP id n4-20020a17090a73c400b001f1e87432c2so924615pjk.3 for ; Mon, 18 Jul 2022 12:37:18 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=20210112; h=from:to:cc:subject:date:message-id:in-reply-to:references :mime-version:content-transfer-encoding; bh=Oo4Pz1r5XHW7nwxpGgJZANPDnH94I0NGToeI6/zKa1I=; b=Yta4aSxqKdJJamMX5Ka9KLD4dubuyf5nvRkNeJsovDB9Ytj1BQ5qWdRgLzIHHHERB6 5sXf3niGo/72wXP0jlBMi/vVkFJX8hLRvbMlrw7tatNFrNNgmNMB9HbzNICkzbNcXs3w 6cu7pwr4AsBJZ3fn8/KYDPZk8nX+6Rg3Ljpt2/TGFqbUZ7cKL97bfzfYsiBq4vhT+Kw9 MVxwsvKLWzOIIRIJm9ZWmpzyPYTX/gBlEmxcuQU3TzAh1tBYia4QkzmOzqCE6j5LO15n QaXqO0JTJLp//5mmCRV7FullleEzXsADDRB6uSgI3/pl3o0omrcymnmvGc3DAicOGAQh YnEg== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20210112; h=x-gm-message-state:from:to:cc:subject:date:message-id:in-reply-to :references:mime-version:content-transfer-encoding; bh=Oo4Pz1r5XHW7nwxpGgJZANPDnH94I0NGToeI6/zKa1I=; b=ju1ztVNcc1qktWOOyZ1zP7vywxNHqaYmb3gdut6bfYr9lzejTETrJRMG5bnQ+5MDHD VxmpES83I7VNvFLy7D1wLGDYnk/tAuEN+lBUxlkZE0oEHP5l6P+7kQH+gDrplU2F+hjk +mav3b6LiIU9epKRzDA0wPJVaLpUv+wsMhBZ/ys4kToPJlmLCySNvl/hiwLdUhhayUY3 bMour5W+fm+o5SCGfltch4vYkjjOK0wgt6+q6ZB14/UdT9/U1RUN+W2Is/WNcMJQDZ7U Fag0o4K5MaqGkYL6FNBVVLiy5dt13949LQEQm5NqDLPDLud8hRr93f9Xst6m7AKzoyFU 1zOg== X-Gm-Message-State: AJIora+zflD28i7dAS51IQwFiQWF3xDNu/PKDH4/FnBX3hSdwNzIgrye cIgpjcpxE36QoYPam2V674GybQ/kQxgFzg== X-Google-Smtp-Source: AGRyM1v14qCZwBb3XJU5MS/ZzWDs+xXqoaSfao3rsct5ND+6jaOGoGucVBBiblDjUX+cd9G34nsdiw== X-Received: by 2002:a17:902:7106:b0:16c:6c95:6153 with SMTP id a6-20020a170902710600b0016c6c956153mr29356100pll.166.1658173037752; Mon, 18 Jul 2022 12:37:17 -0700 (PDT) Received: from sc2-haas01-esx0118.eng.vmware.com ([66.170.99.1]) by smtp.gmail.com with ESMTPSA id q6-20020a170902a3c600b0016bc4a6ce28sm9907887plb.98.2022.07.18.12.37.16 (version=TLS1_3 cipher=TLS_AES_256_GCM_SHA384 bits=256/256); Mon, 18 Jul 2022 12:37:17 -0700 (PDT) From: Nadav Amit X-Google-Original-From: Nadav Amit To: linux-mm@kvack.org Cc: linux-kernel@vger.kernel.org, Andrew Morton , Mike Rapoport , Axel Rasmussen , Nadav Amit , Andrea Arcangeli , Andrew Cooper , Andy Lutomirski , Dave Hansen , David Hildenbrand , Peter Xu , Peter Zijlstra , Thomas Gleixner , Will Deacon , Yu Zhao , Nick Piggin Subject: [RFC PATCH 10/14] x86/mm: introduce relaxed TLB flushes Date: Mon, 18 Jul 2022 05:02:08 -0700 Message-Id: <20220718120212.3180-11-namit@vmware.com> X-Mailer: git-send-email 2.25.1 In-Reply-To: <20220718120212.3180-1-namit@vmware.com> References: <20220718120212.3180-1-namit@vmware.com> MIME-Version: 1.0 ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=hostedemail.com; s=arc-20220608; t=1658173040; h=from:from:sender:reply-to:subject:subject:date:date: message-id:message-id:to:to:cc:cc:mime-version:mime-version: content-type:content-transfer-encoding:content-transfer-encoding: in-reply-to:in-reply-to:references:references:dkim-signature; bh=Oo4Pz1r5XHW7nwxpGgJZANPDnH94I0NGToeI6/zKa1I=; b=db3vXujvG+LZIL9BIHCqEP0YevMse7Bfaa/WSI8mi3IfnPX5+5xT8KfKUayvMXgXaVlLQT s0B0oVJBBgjai6FcOx4IT7/+/UDKfH84ga7PG59i0nq4P0xWqMF4RlXR8H11aCeomSV05d ftLvYB8AWyLueY4pvqzt9PJ0LwbYgRM= ARC-Seal: i=1; s=arc-20220608; d=hostedemail.com; t=1658173040; a=rsa-sha256; cv=none; b=Caz15EcojeI9EOzdVj/zxSq8ov5f5rT6nRrQi7/QTP9ZFrPMfMe6gtGZWY9X2laSDPSpFf rG2hnvfIVEJV4a8gMpeUBUwqMMalEX3KnObIbivHvARcHw5T6HyxNFhDCQvmtX1ow4I6hm 7rfn9qCTZI07FpwvtxSHSTO3mVa+rMc= ARC-Authentication-Results: i=1; imf05.hostedemail.com; dkim=pass header.d=gmail.com header.s=20210112 header.b=Yta4aSxq; dmarc=pass (policy=none) header.from=gmail.com; spf=none (imf05.hostedemail.com: domain of MAILER-DAEMON@hostedemail.com has no SPF policy when checking 216.40.44.16) smtp.mailfrom=MAILER-DAEMON@hostedemail.com X-Stat-Signature: mis3z9t4nuxa5tdxxtrsbgwx5b7bdwj4 X-Rspamd-Queue-Id: 0276310006D X-HE-Tag-Orig: 1658173038-930647 X-Rspam-User: Authentication-Results: imf05.hostedemail.com; dkim=pass header.d=gmail.com header.s=20210112 header.b=Yta4aSxq; dmarc=pass (policy=none) header.from=gmail.com; spf=none (imf05.hostedemail.com: domain of MAILER-DAEMON@hostedemail.com has no SPF policy when checking 216.40.44.16) smtp.mailfrom=MAILER-DAEMON@hostedemail.com X-Rspamd-Server: rspam06 X-HE-Tag: 1658173039-338064 X-Bogosity: Ham, tests=bogofilter, spamicity=0.000000, version=1.2.4 Sender: owner-linux-mm@kvack.org Precedence: bulk X-Loop: owner-majordomo@kvack.org List-ID: From: Nadav Amit Introduce relaxed TLB flushes in x86. When protection is removed from PTEs (i.e., PTEs become writeable or executable), relaxed TLB flushes would be used. Relaxed TLB flushes do flush the local TLB, but do not flush remote TLBs. If later a spurious page-fault is encountered, and the local TLB generation is found to be out of sync with the mm's TLB generation, a full TLB flush takes place to prevent further spurious page-faults from occurring. Cc: Andrea Arcangeli Cc: Andrew Cooper Cc: Andrew Morton Cc: Andy Lutomirski Cc: Dave Hansen Cc: David Hildenbrand Cc: Peter Xu Cc: Peter Zijlstra Cc: Thomas Gleixner Cc: Will Deacon Cc: Yu Zhao Cc: Nick Piggin Signed-off-by: Nadav Amit --- arch/x86/include/asm/tlb.h | 3 ++- arch/x86/include/asm/tlbflush.h | 9 +++++---- arch/x86/kernel/alternative.c | 2 +- arch/x86/kernel/ldt.c | 3 ++- arch/x86/mm/tlb.c | 4 ++-- 5 files changed, 12 insertions(+), 9 deletions(-) diff --git a/arch/x86/include/asm/tlb.h b/arch/x86/include/asm/tlb.h index 1bfe979bb9bc..51c85136f9a8 100644 --- a/arch/x86/include/asm/tlb.h +++ b/arch/x86/include/asm/tlb.h @@ -20,7 +20,8 @@ static inline void tlb_flush(struct mmu_gather *tlb) end = tlb->end; } - flush_tlb_mm_range(tlb->mm, start, end, stride_shift, tlb->freed_tables); + flush_tlb_mm_range(tlb->mm, start, end, stride_shift, tlb->freed_tables, + tlb->strict); } /* diff --git a/arch/x86/include/asm/tlbflush.h b/arch/x86/include/asm/tlbflush.h index 77d4810e5a5d..230cd1d24fe6 100644 --- a/arch/x86/include/asm/tlbflush.h +++ b/arch/x86/include/asm/tlbflush.h @@ -220,23 +220,24 @@ void flush_tlb_multi(const struct cpumask *cpumask, #endif #define flush_tlb_mm(mm) \ - flush_tlb_mm_range(mm, 0UL, TLB_FLUSH_ALL, 0UL, true) + flush_tlb_mm_range(mm, 0UL, TLB_FLUSH_ALL, 0UL, true, true) #define flush_tlb_range(vma, start, end) \ flush_tlb_mm_range((vma)->vm_mm, start, end, \ ((vma)->vm_flags & VM_HUGETLB) \ ? huge_page_shift(hstate_vma(vma)) \ - : PAGE_SHIFT, false) + : PAGE_SHIFT, false, true) extern void flush_tlb_all(void); extern void flush_tlb_mm_range(struct mm_struct *mm, unsigned long start, unsigned long end, unsigned int stride_shift, - bool freed_tables); + bool freed_tables, bool strict); extern void flush_tlb_kernel_range(unsigned long start, unsigned long end); static inline void flush_tlb_page(struct vm_area_struct *vma, unsigned long a) { - flush_tlb_mm_range(vma->vm_mm, a, a + PAGE_SIZE, PAGE_SHIFT, false); + flush_tlb_mm_range(vma->vm_mm, a, a + PAGE_SIZE, PAGE_SHIFT, false, + true); } static inline u64 inc_mm_tlb_gen(struct mm_struct *mm) diff --git a/arch/x86/kernel/alternative.c b/arch/x86/kernel/alternative.c index e257f6c80372..48945a47fd76 100644 --- a/arch/x86/kernel/alternative.c +++ b/arch/x86/kernel/alternative.c @@ -1099,7 +1099,7 @@ static void *__text_poke(text_poke_f func, void *addr, const void *src, size_t l */ flush_tlb_mm_range(poking_mm, poking_addr, poking_addr + (cross_page_boundary ? 2 : 1) * PAGE_SIZE, - PAGE_SHIFT, false); + PAGE_SHIFT, false, true); if (func == text_poke_memcpy) { /* diff --git a/arch/x86/kernel/ldt.c b/arch/x86/kernel/ldt.c index 525876e7b9f4..7c7bc97324bc 100644 --- a/arch/x86/kernel/ldt.c +++ b/arch/x86/kernel/ldt.c @@ -372,7 +372,8 @@ static void unmap_ldt_struct(struct mm_struct *mm, struct ldt_struct *ldt) } va = (unsigned long)ldt_slot_va(ldt->slot); - flush_tlb_mm_range(mm, va, va + nr_pages * PAGE_SIZE, PAGE_SHIFT, false); + flush_tlb_mm_range(mm, va, va + nr_pages * PAGE_SIZE, PAGE_SHIFT, false, + true); } #else /* !CONFIG_PAGE_TABLE_ISOLATION */ diff --git a/arch/x86/mm/tlb.c b/arch/x86/mm/tlb.c index ff3bcc55435e..ec5033d28a97 100644 --- a/arch/x86/mm/tlb.c +++ b/arch/x86/mm/tlb.c @@ -974,7 +974,7 @@ void flush_tlb_fix_spurious_fault(struct vm_area_struct *vma, void flush_tlb_mm_range(struct mm_struct *mm, unsigned long start, unsigned long end, unsigned int stride_shift, - bool freed_tables) + bool freed_tables, bool strict) { struct flush_tlb_info *info; u64 new_tlb_gen; @@ -1000,7 +1000,7 @@ void flush_tlb_mm_range(struct mm_struct *mm, unsigned long start, * a local TLB flush is needed. Optimize this use-case by calling * flush_tlb_func_local() directly in this case. */ - if (cpumask_any_but(mm_cpumask(mm), cpu) < nr_cpu_ids) { + if (strict && cpumask_any_but(mm_cpumask(mm), cpu) < nr_cpu_ids) { flush_tlb_multi(mm_cpumask(mm), info); } else if (mm == this_cpu_read(cpu_tlbstate.loaded_mm)) { lockdep_assert_irqs_enabled(); From patchwork Mon Jul 18 12:02:09 2022 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Nadav Amit X-Patchwork-Id: 12921688 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from kanga.kvack.org (kanga.kvack.org [205.233.56.17]) by smtp.lore.kernel.org (Postfix) with ESMTP id A6C85CCA481 for ; Mon, 18 Jul 2022 19:37:22 +0000 (UTC) Received: by kanga.kvack.org (Postfix) id 428B26B0080; Mon, 18 Jul 2022 15:37:22 -0400 (EDT) Received: by kanga.kvack.org (Postfix, from userid 40) id 363186B0081; Mon, 18 Jul 2022 15:37:22 -0400 (EDT) X-Delivered-To: int-list-linux-mm@kvack.org Received: by kanga.kvack.org (Postfix, from userid 63042) id 166206B0082; Mon, 18 Jul 2022 15:37:22 -0400 (EDT) X-Delivered-To: linux-mm@kvack.org Received: from relay.hostedemail.com (smtprelay0016.hostedemail.com [216.40.44.16]) by kanga.kvack.org (Postfix) with ESMTP id F36866B0080 for ; Mon, 18 Jul 2022 15:37:21 -0400 (EDT) Received: from smtpin11.hostedemail.com (a10.router.float.18 [10.200.18.1]) by unirelay08.hostedemail.com (Postfix) with ESMTP id D2759208B0 for ; Mon, 18 Jul 2022 19:37:21 +0000 (UTC) X-FDA: 79701229482.11.EBD5885 Received: from relay5.hostedemail.com (smtprelay0012.hostedemail.com [216.40.44.12]) by imf02.hostedemail.com (Postfix) with ESMTP id 875FD80043 for ; Mon, 18 Jul 2022 19:37:21 +0000 (UTC) Received: from smtpin29.hostedemail.com (a10.router.float.18 [10.200.18.1]) by unirelay08.hostedemail.com (Postfix) with ESMTP id 3C1F0208B0 for ; Mon, 18 Jul 2022 19:37:21 +0000 (UTC) X-FDA: 79701229482.29.2F78B5B Received: from mail-pf1-f171.google.com (mail-pf1-f171.google.com [209.85.210.171]) by imf07.hostedemail.com (Postfix) with ESMTP id ABCD94005B for ; Mon, 18 Jul 2022 19:37:20 +0000 (UTC) Received: by mail-pf1-f171.google.com with SMTP id 70so11573785pfx.1 for ; Mon, 18 Jul 2022 12:37:20 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=20210112; h=from:to:cc:subject:date:message-id:in-reply-to:references :mime-version:content-transfer-encoding; bh=sYv+0w7UbdJfCd/9qiOzPSdo3DPlp98QBnk6mmrNOzw=; b=LfD4eNC9SIqr9cYWB25Xbr4WiNNia1nr+3GwzKavBUssVyFOxkqXDhxuEsCKotgHGX IWop0WCMjmuoVKLDBbwwaKBQDtdmqkvwVj7sFVhuA7x/GT8cqq96HiJcsEWIREVlkZkC XkrXrhlO0AHfhi7Ka0pK/Bk6TGPHIcLO4v6aQWeY6gTl5lS1/EBV6g/dpM3ldLkVpRs4 P/DRFwB289y/gdLxuvwxWRrdWSS/ANky/BYuh5pHXytyibxUmY1neqRo3cZIvwKFVK6F YK0uUoEexTXqrgimYNcJK1bFS70JivrQyx9FnwmH/rpHZPac+0k2ux4Obajgr/ZrEpGI ztog== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20210112; h=x-gm-message-state:from:to:cc:subject:date:message-id:in-reply-to :references:mime-version:content-transfer-encoding; bh=sYv+0w7UbdJfCd/9qiOzPSdo3DPlp98QBnk6mmrNOzw=; b=ZA+RbUAx3hUg+3Suqs2XmzGf3lYQuKpUkrmZ3YHFyGX8bezA2A8L88YeTGdZJfZq7N dixbd8F/7Pm4snlp0JMpbaNCln9Y4oI+ahZhZUbcf1ZGZWA1pq0yUHqid/+K3QGZRVOR zF+ZCHbFqoTTVCe08A2USb/OxK+TeBNoniBthI1ZpN5Rv3KuTXePaTUDTHn8Y4mNNBdi kYW4wQWtDvGZesAOVkx8owaDwPLP4lDfI+256D5ft+8U4r1dkqsvMSRaz+SO3Hrh0exd AfU7EK+0rrJ7Ym0YvPq4+0GhU51fP3KBDoJVs/WKX3h/8PlvC4cgwM0pmLy6E/6uITQB zwug== X-Gm-Message-State: AJIora9A+hGsa8XN6lj+q7NmFZA0KHIeSPfPTR7XOoivLE6076AwjZ3x Lxg1Jba9nzErs/c7IPO1W2Dr6NXKxhj96A== X-Google-Smtp-Source: AGRyM1vupv7tu2tOFZ1Su8p2ph2+On2wlRIi/LDIoBmyKSXGXyJd3hN8YQju5I/uoHMU9RGeKryGig== X-Received: by 2002:a63:2684:0:b0:415:18d8:78dd with SMTP id m126-20020a632684000000b0041518d878ddmr26454387pgm.33.1658173039290; Mon, 18 Jul 2022 12:37:19 -0700 (PDT) Received: from sc2-haas01-esx0118.eng.vmware.com ([66.170.99.1]) by smtp.gmail.com with ESMTPSA id q6-20020a170902a3c600b0016bc4a6ce28sm9907887plb.98.2022.07.18.12.37.17 (version=TLS1_3 cipher=TLS_AES_256_GCM_SHA384 bits=256/256); Mon, 18 Jul 2022 12:37:18 -0700 (PDT) From: Nadav Amit X-Google-Original-From: Nadav Amit To: linux-mm@kvack.org Cc: linux-kernel@vger.kernel.org, Andrew Morton , Mike Rapoport , Axel Rasmussen , Nadav Amit , Andrea Arcangeli , Andrew Cooper , Andy Lutomirski , Dave Hansen , David Hildenbrand , Peter Xu , Peter Zijlstra , Thomas Gleixner , Will Deacon , Yu Zhao , Nick Piggin Subject: [RFC PATCH 11/14] x86/mm: use relaxed TLB flushes when protection is removed Date: Mon, 18 Jul 2022 05:02:09 -0700 Message-Id: <20220718120212.3180-12-namit@vmware.com> X-Mailer: git-send-email 2.25.1 In-Reply-To: <20220718120212.3180-1-namit@vmware.com> References: <20220718120212.3180-1-namit@vmware.com> MIME-Version: 1.0 ARC-Seal: i=1; s=arc-20220608; d=hostedemail.com; t=1658173041; a=rsa-sha256; cv=none; b=IP0jhPYthAKoolBIK1G2Qm/d1hp8D6gG0Wi9khQxeSvgLmU9ErrPmc1PS2vyoaQ1+2a5re DuDON6vAQKucpkHyeKFXemX1secpQvV8p20rYjCx9U7TSTOqf177Tk4jzTntQ1MSjMWj08 HGY5tFhvfPkYIFltFMN4n7ucnlUA1UA= ARC-Authentication-Results: i=1; imf02.hostedemail.com; dkim=pass header.d=gmail.com header.s=20210112 header.b=LfD4eNC9; dmarc=pass (policy=none) header.from=gmail.com; spf=none (imf02.hostedemail.com: domain of MAILER-DAEMON@hostedemail.com has no SPF policy when checking 216.40.44.12) smtp.mailfrom=MAILER-DAEMON@hostedemail.com ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=hostedemail.com; s=arc-20220608; t=1658173041; h=from:from:sender:reply-to:subject:subject:date:date: message-id:message-id:to:to:cc:cc:mime-version:mime-version: content-type:content-transfer-encoding:content-transfer-encoding: in-reply-to:in-reply-to:references:references:dkim-signature; bh=sYv+0w7UbdJfCd/9qiOzPSdo3DPlp98QBnk6mmrNOzw=; b=wbJDoPWRIJR+NkF7T3kE3OcQVCxTKqNBPGYhBFo7yJ6/PWyvSv1gb4I+hZExZxC2qErQff PWF0kMrYYd8RMg35Rc1gYCKgDfHa5lN/fuT10Lvvi+b3T1HfFYPz7kUY3OlDkK9ttbVPtp sEssSipx0SdPR2BLFxGNskGNkQJVJyM= X-HE-Tag-Orig: 1658173040-117575 X-Stat-Signature: dnbdxze65imc9519jjzmas6yydijcuzq X-Rspamd-Queue-Id: 875FD80043 Authentication-Results: imf02.hostedemail.com; dkim=pass header.d=gmail.com header.s=20210112 header.b=LfD4eNC9; dmarc=pass (policy=none) header.from=gmail.com; spf=none (imf02.hostedemail.com: domain of MAILER-DAEMON@hostedemail.com has no SPF policy when checking 216.40.44.12) smtp.mailfrom=MAILER-DAEMON@hostedemail.com X-Rspam-User: X-Rspamd-Server: rspam09 X-HE-Tag: 1658173041-170570 X-Bogosity: Ham, tests=bogofilter, spamicity=0.000000, version=1.2.4 Sender: owner-linux-mm@kvack.org Precedence: bulk X-Loop: owner-majordomo@kvack.org List-ID: From: Nadav Amit When checking x86 PTE flags to determine whether a TLB flush is needed, determine whether a relaxed TLB flush is sufficient. If protection is added (NX removed or W added), indicate that a relaxed TLB flush would suffice. Cc: Andrea Arcangeli Cc: Andrew Cooper Cc: Andrew Morton Cc: Andy Lutomirski Cc: Dave Hansen Cc: David Hildenbrand Cc: Peter Xu Cc: Peter Zijlstra Cc: Thomas Gleixner Cc: Will Deacon Cc: Yu Zhao Cc: Nick Piggin Signed-off-by: Nadav Amit --- arch/x86/include/asm/tlbflush.h | 20 ++++++++++++++++---- 1 file changed, 16 insertions(+), 4 deletions(-) diff --git a/arch/x86/include/asm/tlbflush.h b/arch/x86/include/asm/tlbflush.h index 230cd1d24fe6..4f98735ab07a 100644 --- a/arch/x86/include/asm/tlbflush.h +++ b/arch/x86/include/asm/tlbflush.h @@ -271,18 +271,23 @@ static inline enum pte_flush_type pte_flags_flush_type(unsigned long oldflags, * dirty/access bit if needed without a fault. */ const pteval_t flush_on_clear = _PAGE_DIRTY | _PAGE_PRESENT | - _PAGE_ACCESSED; + _PAGE_ACCESSED | _PAGE_RW; + const pteval_t flush_on_set = _PAGE_NX; + const pteval_t flush_on_set_relaxed = _PAGE_RW; + const pteval_t flush_on_clear_relaxed = _PAGE_NX; const pteval_t software_flags = _PAGE_SOFTW1 | _PAGE_SOFTW2 | _PAGE_SOFTW3 | _PAGE_SOFTW4; - const pteval_t flush_on_change = _PAGE_RW | _PAGE_USER | _PAGE_PWT | + const pteval_t flush_on_change = _PAGE_USER | _PAGE_PWT | _PAGE_PCD | _PAGE_PSE | _PAGE_GLOBAL | _PAGE_PAT | _PAGE_PAT_LARGE | _PAGE_PKEY_BIT0 | _PAGE_PKEY_BIT1 | - _PAGE_PKEY_BIT2 | _PAGE_PKEY_BIT3 | _PAGE_NX; + _PAGE_PKEY_BIT2 | _PAGE_PKEY_BIT3; unsigned long diff = oldflags ^ newflags; BUILD_BUG_ON(flush_on_clear & software_flags); BUILD_BUG_ON(flush_on_clear & flush_on_change); BUILD_BUG_ON(flush_on_change & software_flags); + BUILD_BUG_ON(flush_on_change & flush_on_clear_relaxed); + BUILD_BUG_ON(flush_on_change & flush_on_set_relaxed); /* Ignore software flags */ diff &= ~software_flags; @@ -301,9 +306,16 @@ static inline enum pte_flush_type pte_flags_flush_type(unsigned long oldflags, if (diff & flush_on_change) return PTE_FLUSH_STRICT; + if (diff & oldflags & flush_on_clear_relaxed) + return PTE_FLUSH_RELAXED; + + if (diff & newflags & flush_on_set_relaxed) + return PTE_FLUSH_RELAXED; + /* Ensure there are no flags that were left behind */ if (IS_ENABLED(CONFIG_DEBUG_VM) && - (diff & ~(flush_on_clear | software_flags | flush_on_change))) { + (diff & ~(flush_on_clear | flush_on_set | + software_flags | flush_on_change))) { VM_WARN_ON_ONCE(1); return PTE_FLUSH_STRICT; } From patchwork Mon Jul 18 12:02:10 2022 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Nadav Amit X-Patchwork-Id: 12921689 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from kanga.kvack.org (kanga.kvack.org [205.233.56.17]) by smtp.lore.kernel.org (Postfix) with ESMTP id A9228C43334 for ; Mon, 18 Jul 2022 19:37:24 +0000 (UTC) Received: by kanga.kvack.org (Postfix) id 3DA386B0073; Mon, 18 Jul 2022 15:37:24 -0400 (EDT) Received: by kanga.kvack.org (Postfix, from userid 40) id 3B03E6B0074; Mon, 18 Jul 2022 15:37:24 -0400 (EDT) X-Delivered-To: int-list-linux-mm@kvack.org Received: by kanga.kvack.org (Postfix, from userid 63042) id 1DD3B8E0001; Mon, 18 Jul 2022 15:37:24 -0400 (EDT) X-Delivered-To: linux-mm@kvack.org Received: from relay.hostedemail.com (smtprelay0015.hostedemail.com [216.40.44.15]) by kanga.kvack.org (Postfix) with ESMTP id 096B76B0073 for ; Mon, 18 Jul 2022 15:37:24 -0400 (EDT) Received: from smtpin03.hostedemail.com (a10.router.float.18 [10.200.18.1]) by unirelay13.hostedemail.com (Postfix) with ESMTP id DAF5D60A04 for ; Mon, 18 Jul 2022 19:37:23 +0000 (UTC) X-FDA: 79701229566.03.0058B24 Received: from relay5.hostedemail.com (smtprelay0013.hostedemail.com [216.40.44.13]) by imf22.hostedemail.com (Postfix) with ESMTP id 2A20EC0087 for ; Mon, 18 Jul 2022 19:37:22 +0000 (UTC) Received: from smtpin28.hostedemail.com (a10.router.float.18 [10.200.18.1]) by unirelay12.hostedemail.com (Postfix) with ESMTP id CC642120D26 for ; Mon, 18 Jul 2022 19:37:22 +0000 (UTC) X-FDA: 79701229524.28.72552C0 Received: from mail-pl1-f175.google.com (mail-pl1-f175.google.com [209.85.214.175]) by imf14.hostedemail.com (Postfix) with ESMTP id 8208310006F for ; Mon, 18 Jul 2022 19:37:22 +0000 (UTC) Received: by mail-pl1-f175.google.com with SMTP id g17so9967770plh.2 for ; Mon, 18 Jul 2022 12:37:22 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=20210112; h=from:to:cc:subject:date:message-id:in-reply-to:references :mime-version:content-transfer-encoding; bh=U3bHH+VCApNk0EgSMkQpPIu/hogjL02kkGQOh371JqE=; b=cgVgCcOubWJlYQh87ciup80RfrTt14z9y9XheMGRDWpUOcN7tz83N1/LeVdqY4FC8x NjZ299ZmT5ayN+y3qeWtB4AiGGlLv0bTeoa41zMsLcKw7Yvbr8j3tdxbVbn/BCQj5UnM N+DKf8yCPT06zTRIR4vx6LrfvTTsQMm6mDcaVmmqoslMBPkuYDGQM8LAXE49wLZerlr4 O2XHyJrKy3OzAIEITCEW9D6Zdl1GZeNoRYmVFo5+eof3svAuQ9oW9pdkFfi1jwDp+2Kd cmWr6cYwHWT0ai9wmSNVyZ4nkkJV4zU269t4/tA7BL3pdN3OppGjcXUqwL5ga4yr3+rK lK/g== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20210112; h=x-gm-message-state:from:to:cc:subject:date:message-id:in-reply-to :references:mime-version:content-transfer-encoding; bh=U3bHH+VCApNk0EgSMkQpPIu/hogjL02kkGQOh371JqE=; b=HfHEmDoQcJNWv/VeJedjmBkF/tMTJ+VRQxNFr8CdPzLEk0dxz6trfPkPZvu+Pd7fwN p0JN6BCxYFtfsnboeoI8aaKMFu+Ua+rxxuMPLI8YkV37qfqgc/QH2nKHtS6K0Ga8HjuV 3NcGaf2WlOmRljyi7p5J8TT/ZlJc0rCP6pK7MRYvvvTBYJDyj/fHUcel9tR1pPyEaGjY tgRbtihLVy2HWHTpzDngB3qMFEH9ndSq0Wa+z1w2bzA/Rv1C1bYoWQWXbRavTO504f0Y 9Al6F+4vsi9rKs2ze3leW37HeGey4ofMhNCPNcDV9HQ3VHYzS3ounhFSkktLxHBu0pvw cJEw== X-Gm-Message-State: AJIora/8SkcWVy2wASeGoHV6LsllxwjyD6xk8SyE3PuDIXUmKHaRjjj+ wGiv/kyL8U5p7URA2fbs7aAcakS59V41+g== X-Google-Smtp-Source: AGRyM1t9fp3tA/qUiJLR5rxiQilje7wUJ0XNs2hdR5hLaXg/+Pz/WHXJKpxZoMJbK3OuMmRL/9fyZw== X-Received: by 2002:a17:90a:f114:b0:1ef:991f:12e7 with SMTP id cc20-20020a17090af11400b001ef991f12e7mr41584874pjb.199.1658173040943; Mon, 18 Jul 2022 12:37:20 -0700 (PDT) Received: from sc2-haas01-esx0118.eng.vmware.com ([66.170.99.1]) by smtp.gmail.com with ESMTPSA id q6-20020a170902a3c600b0016bc4a6ce28sm9907887plb.98.2022.07.18.12.37.19 (version=TLS1_3 cipher=TLS_AES_256_GCM_SHA384 bits=256/256); Mon, 18 Jul 2022 12:37:20 -0700 (PDT) From: Nadav Amit X-Google-Original-From: Nadav Amit To: linux-mm@kvack.org Cc: linux-kernel@vger.kernel.org, Andrew Morton , Mike Rapoport , Axel Rasmussen , Nadav Amit , Andrea Arcangeli , Andrew Cooper , Andy Lutomirski , Dave Hansen , David Hildenbrand , Peter Xu , Peter Zijlstra , Thomas Gleixner , Will Deacon , Yu Zhao , Nick Piggin Subject: [RFC PATCH 12/14] x86/tlb: no flush on PTE change from RW->RO when PTE is clean Date: Mon, 18 Jul 2022 05:02:10 -0700 Message-Id: <20220718120212.3180-13-namit@vmware.com> X-Mailer: git-send-email 2.25.1 In-Reply-To: <20220718120212.3180-1-namit@vmware.com> References: <20220718120212.3180-1-namit@vmware.com> MIME-Version: 1.0 ARC-Seal: i=1; s=arc-20220608; d=hostedemail.com; t=1658173043; a=rsa-sha256; cv=none; b=iovMz3F11hN/RqYVvNB/KSoEFbTc7TbDYr6j2qFyZtNQxDMmwU4WoZYLDV4pW8a7HzWQLs SE55CQ34SvO/tRBudRXJ06sm9d6g7d6T0irF0Kp5JDnSn7EPu9yc4tz5rJbxlse7UNih5e +UlqhY+6LNcz0sRfqLhQtK0qCqPACAM= ARC-Authentication-Results: i=1; imf22.hostedemail.com; dkim=pass header.d=gmail.com header.s=20210112 header.b=cgVgCcOu; dmarc=pass (policy=none) header.from=gmail.com; spf=none (imf22.hostedemail.com: domain of MAILER-DAEMON@hostedemail.com has no SPF policy when checking 216.40.44.13) smtp.mailfrom=MAILER-DAEMON@hostedemail.com ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=hostedemail.com; s=arc-20220608; t=1658173043; h=from:from:sender:reply-to:subject:subject:date:date: message-id:message-id:to:to:cc:cc:mime-version:mime-version: content-type:content-transfer-encoding:content-transfer-encoding: in-reply-to:in-reply-to:references:references:dkim-signature; bh=U3bHH+VCApNk0EgSMkQpPIu/hogjL02kkGQOh371JqE=; b=zYzFsn+FLVn770j4TymcNB2SAjh+33MRAJwIgzfoGrUOJ6aE3iy3jBOU1zyTHF+zQuCvWy EvSI07IhW4HJrfmUYdsXO8jjMwbjwV4NekOwrJsIu4zl5GQKeXi6uTDN4HQpJ+Mo5MbtJP 5nvD+8O0KA95bw1p3owsVBljYPWwQPY= X-HE-Tag-Orig: 1658173042-468649 X-Rspam-User: Authentication-Results: imf22.hostedemail.com; dkim=pass header.d=gmail.com header.s=20210112 header.b=cgVgCcOu; dmarc=pass (policy=none) header.from=gmail.com; spf=none (imf22.hostedemail.com: domain of MAILER-DAEMON@hostedemail.com has no SPF policy when checking 216.40.44.13) smtp.mailfrom=MAILER-DAEMON@hostedemail.com X-Rspamd-Queue-Id: 2A20EC0087 X-Rspamd-Server: rspam12 X-Stat-Signature: qt84gm64q7i4ewdggs63tpwsotqan9hi X-HE-Tag: 1658173042-229999 X-Bogosity: Ham, tests=bogofilter, spamicity=0.000000, version=1.2.4 Sender: owner-linux-mm@kvack.org Precedence: bulk X-Loop: owner-majordomo@kvack.org List-ID: From: Nadav Amit On x86 it is possible to skip a TLB flush when a RW entry become RO and the PTE is clean. Add logic to detect this case and skip the flush. Cc: Andrea Arcangeli Cc: Andrew Cooper Cc: Andrew Morton Cc: Andy Lutomirski Cc: Dave Hansen Cc: David Hildenbrand Cc: Peter Xu Cc: Peter Zijlstra Cc: Thomas Gleixner Cc: Will Deacon Cc: Yu Zhao Cc: Nick Piggin Signed-off-by: Nadav Amit --- arch/x86/include/asm/tlbflush.h | 16 ++++++++++++++-- 1 file changed, 14 insertions(+), 2 deletions(-) diff --git a/arch/x86/include/asm/tlbflush.h b/arch/x86/include/asm/tlbflush.h index 4f98735ab07a..58c95e36b098 100644 --- a/arch/x86/include/asm/tlbflush.h +++ b/arch/x86/include/asm/tlbflush.h @@ -271,8 +271,9 @@ static inline enum pte_flush_type pte_flags_flush_type(unsigned long oldflags, * dirty/access bit if needed without a fault. */ const pteval_t flush_on_clear = _PAGE_DIRTY | _PAGE_PRESENT | - _PAGE_ACCESSED | _PAGE_RW; + _PAGE_ACCESSED; const pteval_t flush_on_set = _PAGE_NX; + const pteval_t flush_on_special = _PAGE_RW; const pteval_t flush_on_set_relaxed = _PAGE_RW; const pteval_t flush_on_clear_relaxed = _PAGE_NX; const pteval_t software_flags = _PAGE_SOFTW1 | _PAGE_SOFTW2 | @@ -302,6 +303,17 @@ static inline enum pte_flush_type pte_flags_flush_type(unsigned long oldflags, if (diff & oldflags & flush_on_clear) return PTE_FLUSH_STRICT; + /* + * Did any of the 'flush_on_set' flags was set between 'oldflags' and + * 'newflags'? + */ + if (diff & newflags & flush_on_set) + return PTE_FLUSH_STRICT; + + /* On RW->RO, a flush is needed if the old entry is dirty */ + if ((diff & oldflags & _PAGE_RW) && (oldflags & _PAGE_DIRTY)) + return PTE_FLUSH_STRICT; + /* Flush on modified flags. */ if (diff & flush_on_change) return PTE_FLUSH_STRICT; @@ -314,7 +326,7 @@ static inline enum pte_flush_type pte_flags_flush_type(unsigned long oldflags, /* Ensure there are no flags that were left behind */ if (IS_ENABLED(CONFIG_DEBUG_VM) && - (diff & ~(flush_on_clear | flush_on_set | + (diff & ~(flush_on_clear | flush_on_set | flush_on_special | software_flags | flush_on_change))) { VM_WARN_ON_ONCE(1); return PTE_FLUSH_STRICT; From patchwork Mon Jul 18 12:02:12 2022 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Nadav Amit X-Patchwork-Id: 12921690 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from kanga.kvack.org (kanga.kvack.org [205.233.56.17]) by smtp.lore.kernel.org (Postfix) with ESMTP id C01BCC433EF for ; Mon, 18 Jul 2022 19:37:27 +0000 (UTC) Received: by kanga.kvack.org (Postfix) id 5AD5C6B0074; Mon, 18 Jul 2022 15:37:27 -0400 (EDT) Received: by kanga.kvack.org (Postfix, from userid 40) id 55D636B0081; Mon, 18 Jul 2022 15:37:27 -0400 (EDT) X-Delivered-To: int-list-linux-mm@kvack.org Received: by kanga.kvack.org (Postfix, from userid 63042) id 3FE4A8E0001; Mon, 18 Jul 2022 15:37:27 -0400 (EDT) X-Delivered-To: linux-mm@kvack.org Received: from relay.hostedemail.com (smtprelay0011.hostedemail.com [216.40.44.11]) by kanga.kvack.org (Postfix) with ESMTP id 2DF426B0074 for ; Mon, 18 Jul 2022 15:37:27 -0400 (EDT) Received: from smtpin05.hostedemail.com (a10.router.float.18 [10.200.18.1]) by unirelay08.hostedemail.com (Postfix) with ESMTP id 0F06C20DB5 for ; Mon, 18 Jul 2022 19:37:27 +0000 (UTC) X-FDA: 79701229734.05.8E2B6F6 Received: from relay5.hostedemail.com (smtprelay0010.hostedemail.com [216.40.44.10]) by imf06.hostedemail.com (Postfix) with ESMTP id 7267C18004E for ; Mon, 18 Jul 2022 19:37:26 +0000 (UTC) Received: from smtpin24.hostedemail.com (a10.router.float.18 [10.200.18.1]) by unirelay11.hostedemail.com (Postfix) with ESMTP id 2DA298015F for ; Mon, 18 Jul 2022 19:37:26 +0000 (UTC) X-FDA: 79701229692.24.E2E4418 Received: from mail-pj1-f41.google.com (mail-pj1-f41.google.com [209.85.216.41]) by imf02.hostedemail.com (Postfix) with ESMTP id D53388004B for ; Mon, 18 Jul 2022 19:37:25 +0000 (UTC) Received: by mail-pj1-f41.google.com with SMTP id t5-20020a17090a6a0500b001ef965b262eso13643444pjj.5 for ; Mon, 18 Jul 2022 12:37:25 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=20210112; h=from:to:cc:subject:date:message-id:in-reply-to:references :mime-version:content-transfer-encoding; bh=vrKPjJvEMBTDr1mCiDbgd+KWQ8nvzpXNXEsnVSmuvO0=; b=eYUZpQJpHryXrfyw7d48dRMqIWNv2+baxNAcAVwwnO0fjHTTRD50J1bwh/PG8hjhyo RFIxBYYQfj5C9aZGvM+1u1GlB88xW+kndwLcOf6g2YEW+59i/GnfGPV2S9WxiHUwpSKX x08be9IZJ+YBx/qdBlDRbTMoOz/KXj85/cPSJxz55YfQzzX3Jpf+kiiIG4NAlj1yOMjc BjlyIgdaoYZ3WQ5j2mm/bj4G39v948AmRF31dtKwb52uX06Lm82D+vYO7r8Ancx0fHXp /xQCRujNj3hHmv8itzoQcsbS0vCIUj1i2sOEK0ErYyuRJHrObKtzScHJD8DsT83a0u+y G2WQ== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20210112; h=x-gm-message-state:from:to:cc:subject:date:message-id:in-reply-to :references:mime-version:content-transfer-encoding; bh=vrKPjJvEMBTDr1mCiDbgd+KWQ8nvzpXNXEsnVSmuvO0=; b=p6mAS2C0WkBj3lAF/7PVJHvykEXhsE6Gisxcq0ukvmGX8nn8yR0r2gFqOrI4EUJgYC QN/fDiue2+zth7EJseJnuWjwWGqeOxbMR7i5WF8J/ClqLBgFNpwkDIbY7Yf6yrKEntzu 5OAdoQXYJ1GcBgHfa7yXJeS0rpyBMNL1o+LGuePB0QGdYxXIHS0NL2OUkL7Qh9BMVDs0 rcD0u1X3PdI0DIx9jaYhOhjwXB9GoueNJkNPjupB9OsQTZYhCL9ooPjZBmIQC2qDlA2g /Kx1qvIPZ5wjAo1Dcg0dJFQ11CdkrHYzBiZ2/Y/936Lj1YNXPLXT1BR4osi48GnLBNpu EaPg== X-Gm-Message-State: AJIora+6aNG3toGzOdnTT1OgrGwawISkWUGwprDPFJXcKpAPg/0G+I0M x9+EGv+eOrEEf9K1nMMc84dx1KU8Mf2ZaA== X-Google-Smtp-Source: AGRyM1uYMSGTx4GyYdBLjJpMepu7xtMk4UhhFdK1EgIxLwbQTenVoVfEyUI9bNANKUd9XUVG9W4PUA== X-Received: by 2002:a17:903:1111:b0:16a:acf4:e951 with SMTP id n17-20020a170903111100b0016aacf4e951mr29606165plh.72.1658173044426; Mon, 18 Jul 2022 12:37:24 -0700 (PDT) Received: from sc2-haas01-esx0118.eng.vmware.com ([66.170.99.1]) by smtp.gmail.com with ESMTPSA id q6-20020a170902a3c600b0016bc4a6ce28sm9907887plb.98.2022.07.18.12.37.22 (version=TLS1_3 cipher=TLS_AES_256_GCM_SHA384 bits=256/256); Mon, 18 Jul 2022 12:37:24 -0700 (PDT) From: Nadav Amit X-Google-Original-From: Nadav Amit To: linux-mm@kvack.org Cc: linux-kernel@vger.kernel.org, Andrew Morton , Mike Rapoport , Axel Rasmussen , Nadav Amit , Andrea Arcangeli , Andrew Cooper , Andy Lutomirski , Dave Hansen , David Hildenbrand , Peter Xu , Peter Zijlstra , Thomas Gleixner , Will Deacon , Yu Zhao , Nick Piggin Subject: [RFC PATCH 14/14] mm: conditional check of pfn in pte_flush_type Date: Mon, 18 Jul 2022 05:02:12 -0700 Message-Id: <20220718120212.3180-15-namit@vmware.com> X-Mailer: git-send-email 2.25.1 In-Reply-To: <20220718120212.3180-1-namit@vmware.com> References: <20220718120212.3180-1-namit@vmware.com> MIME-Version: 1.0 ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=hostedemail.com; s=arc-20220608; t=1658173046; h=from:from:sender:reply-to:subject:subject:date:date: message-id:message-id:to:to:cc:cc:mime-version:mime-version: content-type:content-transfer-encoding:content-transfer-encoding: in-reply-to:in-reply-to:references:references:dkim-signature; bh=vrKPjJvEMBTDr1mCiDbgd+KWQ8nvzpXNXEsnVSmuvO0=; b=B0XCiCg5TURkSKSd+2sy52UekafPkcXQ5zwNTvsC2+T63kCUczoENNXh92BJUuFLcUs+It VddiUgvdT+Wk/emWNQLr8hPgeeSInHPKKzrEznm0RcDOtH+d7U3gAPzkCyJjZTQTqz1Z76 Za0gDbt73Yb6L14wddF+RqWFVfthBug= ARC-Authentication-Results: i=1; imf06.hostedemail.com; dkim=pass header.d=gmail.com header.s=20210112 header.b=eYUZpQJp; spf=none (imf06.hostedemail.com: domain of MAILER-DAEMON@hostedemail.com has no SPF policy when checking 216.40.44.10) smtp.mailfrom=MAILER-DAEMON@hostedemail.com; dmarc=pass (policy=none) header.from=gmail.com ARC-Seal: i=1; s=arc-20220608; d=hostedemail.com; t=1658173046; a=rsa-sha256; cv=none; b=oThEBHArchJ1+poYfl1RFEU6wle0+lS0oJfAJg3sOR+HHjoDHJfBjrkcBoOefTAXDio4Ld orUXWESaGp36L1DH36DSIiCI2gKV34S/fTbOLv7rOO8tys1s7J36PLYETi7KdQnO4+92pu Fz7MzLuyzvmrAKQNffu4d93Ribwy0PU= X-Stat-Signature: j39ryna9ubxiu6x9bkwea361fyau7q8y X-Rspamd-Queue-Id: 7267C18004E X-HE-Tag-Orig: 1658173045-356782 X-Rspamd-Server: rspam02 X-Rspam-User: Authentication-Results: imf06.hostedemail.com; dkim=pass header.d=gmail.com header.s=20210112 header.b=eYUZpQJp; spf=none (imf06.hostedemail.com: domain of MAILER-DAEMON@hostedemail.com has no SPF policy when checking 216.40.44.10) smtp.mailfrom=MAILER-DAEMON@hostedemail.com; dmarc=pass (policy=none) header.from=gmail.com X-HE-Tag: 1658173046-740852 X-Bogosity: Ham, tests=bogofilter, spamicity=0.000000, version=1.2.4 Sender: owner-linux-mm@kvack.org Precedence: bulk X-Loop: owner-majordomo@kvack.org List-ID: From: Nadav Amit Checking whether PFNs in two PTEs are the same takes surprisingly large number of instructions. Yet in fact, in most cases the caller to pte_flush_type() already knows if the PFN was changed. For instance, mprotect() does not change the PFN, but only modifies the protection flags. Add argument to pte_flush_type() to indicate whether the PFN should be checked. Keep checking it in mm-debug to see if some caller was wrong to assume the PFN is the same. Cc: Andrea Arcangeli Cc: Andrew Cooper Cc: Andrew Morton Cc: Andy Lutomirski Cc: Dave Hansen Cc: David Hildenbrand Cc: Peter Xu Cc: Peter Zijlstra Cc: Thomas Gleixner Cc: Will Deacon Cc: Yu Zhao Cc: Nick Piggin Signed-off-by: Nadav Amit --- arch/x86/include/asm/tlbflush.h | 14 ++++++++++---- include/asm-generic/tlb.h | 6 ++++-- mm/huge_memory.c | 2 +- mm/mprotect.c | 2 +- mm/rmap.c | 2 +- 5 files changed, 17 insertions(+), 9 deletions(-) diff --git a/arch/x86/include/asm/tlbflush.h b/arch/x86/include/asm/tlbflush.h index 58c95e36b098..50349861fdc9 100644 --- a/arch/x86/include/asm/tlbflush.h +++ b/arch/x86/include/asm/tlbflush.h @@ -340,14 +340,17 @@ static inline enum pte_flush_type pte_flags_flush_type(unsigned long oldflags, * whether a strict or relaxed TLB flush is need. It should only be used on * userspace PTEs. */ -static inline enum pte_flush_type pte_flush_type(pte_t oldpte, pte_t newpte) +static inline enum pte_flush_type pte_flush_type(pte_t oldpte, pte_t newpte, + bool check_pfn) { /* !PRESENT -> * ; no need for flush */ if (!(pte_flags(oldpte) & _PAGE_PRESENT)) return PTE_FLUSH_NONE; /* PFN changed ; needs flush */ - if (pte_pfn(oldpte) != pte_pfn(newpte)) + if (!check_pfn) + VM_BUG_ON(pte_pfn(oldpte) != pte_pfn(newpte)); + else if (pte_pfn(oldpte) != pte_pfn(newpte)) return PTE_FLUSH_STRICT; /* @@ -363,14 +366,17 @@ static inline enum pte_flush_type pte_flush_type(pte_t oldpte, pte_t newpte) * huge_pmd_flush_type() checks whether permissions were demoted and require a * flush. It should only be used for userspace huge PMDs. */ -static inline enum pte_flush_type huge_pmd_flush_type(pmd_t oldpmd, pmd_t newpmd) +static inline enum pte_flush_type huge_pmd_flush_type(pmd_t oldpmd, pmd_t newpmd, + bool check_pfn) { /* !PRESENT -> * ; no need for flush */ if (!(pmd_flags(oldpmd) & _PAGE_PRESENT)) return PTE_FLUSH_NONE; /* PFN changed ; needs flush */ - if (pmd_pfn(oldpmd) != pmd_pfn(newpmd)) + if (!check_pfn) + VM_BUG_ON(pmd_pfn(oldpmd) != pmd_pfn(newpmd)); + else if (pmd_pfn(oldpmd) != pmd_pfn(newpmd)) return PTE_FLUSH_STRICT; /* diff --git a/include/asm-generic/tlb.h b/include/asm-generic/tlb.h index 07b3eb8caf63..aee9da6cc5d5 100644 --- a/include/asm-generic/tlb.h +++ b/include/asm-generic/tlb.h @@ -677,14 +677,16 @@ static inline void tlb_flush_p4d_range(struct mmu_gather *tlb, #endif #ifndef pte_flush_type -static inline struct pte_flush_type pte_flush_type(pte_t oldpte, pte_t newpte) +static inline struct pte_flush_type pte_flush_type(pte_t oldpte, pte_t newpte, + bool check_pfn) { return PTE_FLUSH_STRICT; } #endif #ifndef huge_pmd_flush_type -static inline bool huge_pmd_flush_type(pmd_t oldpmd, pmd_t newpmd) +static inline bool huge_pmd_flush_type(pmd_t oldpmd, pmd_t newpmd, + bool check_pfn) { return PTE_FLUSH_STRICT; } diff --git a/mm/huge_memory.c b/mm/huge_memory.c index b32b7da0f6f7..92a7b3ca317f 100644 --- a/mm/huge_memory.c +++ b/mm/huge_memory.c @@ -1818,7 +1818,7 @@ int change_huge_pmd(struct mmu_gather *tlb, struct vm_area_struct *vma, flush_type = PTE_FLUSH_STRICT; if (!tlb->strict) - flush_type = huge_pmd_flush_type(oldpmd, entry); + flush_type = huge_pmd_flush_type(oldpmd, entry, false); if (flush_type != PTE_FLUSH_NONE) tlb_flush_pmd_range(tlb, addr, HPAGE_PMD_SIZE, flush_type == PTE_FLUSH_STRICT); diff --git a/mm/mprotect.c b/mm/mprotect.c index cf775f6c8c08..78081d7f4edf 100644 --- a/mm/mprotect.c +++ b/mm/mprotect.c @@ -204,7 +204,7 @@ static unsigned long change_pte_range(struct mmu_gather *tlb, flush_type = PTE_FLUSH_STRICT; if (!tlb->strict) - flush_type = pte_flush_type(oldpte, ptent); + flush_type = pte_flush_type(oldpte, ptent, false); if (flush_type != PTE_FLUSH_NONE) tlb_flush_pte_range(tlb, addr, PAGE_SIZE, flush_type == PTE_FLUSH_STRICT); diff --git a/mm/rmap.c b/mm/rmap.c index 62f4b2a4f067..63261619b607 100644 --- a/mm/rmap.c +++ b/mm/rmap.c @@ -974,7 +974,7 @@ static int page_vma_mkclean_one(struct page_vma_mapped_walk *pvmw) entry = pte_wrprotect(oldpte); entry = pte_mkclean(entry); - if (pte_flush_type(oldpte, entry) != PTE_FLUSH_NONE || + if (pte_flush_type(oldpte, entry, false) != PTE_FLUSH_NONE || mm_tlb_flush_pending(vma->vm_mm)) flush_tlb_page(vma, address);