From patchwork Mon Jul 18 12:02:10 2022 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Nadav Amit X-Patchwork-Id: 12921689 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from kanga.kvack.org (kanga.kvack.org [205.233.56.17]) by smtp.lore.kernel.org (Postfix) with ESMTP id A9228C43334 for ; Mon, 18 Jul 2022 19:37:24 +0000 (UTC) Received: by kanga.kvack.org (Postfix) id 3DA386B0073; Mon, 18 Jul 2022 15:37:24 -0400 (EDT) Received: by kanga.kvack.org (Postfix, from userid 40) id 3B03E6B0074; Mon, 18 Jul 2022 15:37:24 -0400 (EDT) X-Delivered-To: int-list-linux-mm@kvack.org Received: by kanga.kvack.org (Postfix, from userid 63042) id 1DD3B8E0001; Mon, 18 Jul 2022 15:37:24 -0400 (EDT) X-Delivered-To: linux-mm@kvack.org Received: from relay.hostedemail.com (smtprelay0015.hostedemail.com [216.40.44.15]) by kanga.kvack.org (Postfix) with ESMTP id 096B76B0073 for ; Mon, 18 Jul 2022 15:37:24 -0400 (EDT) Received: from smtpin03.hostedemail.com (a10.router.float.18 [10.200.18.1]) by unirelay13.hostedemail.com (Postfix) with ESMTP id DAF5D60A04 for ; Mon, 18 Jul 2022 19:37:23 +0000 (UTC) X-FDA: 79701229566.03.0058B24 Received: from relay5.hostedemail.com (smtprelay0013.hostedemail.com [216.40.44.13]) by imf22.hostedemail.com (Postfix) with ESMTP id 2A20EC0087 for ; Mon, 18 Jul 2022 19:37:22 +0000 (UTC) Received: from smtpin28.hostedemail.com (a10.router.float.18 [10.200.18.1]) by unirelay12.hostedemail.com (Postfix) with ESMTP id CC642120D26 for ; Mon, 18 Jul 2022 19:37:22 +0000 (UTC) X-FDA: 79701229524.28.72552C0 Received: from mail-pl1-f175.google.com (mail-pl1-f175.google.com [209.85.214.175]) by imf14.hostedemail.com (Postfix) with ESMTP id 8208310006F for ; Mon, 18 Jul 2022 19:37:22 +0000 (UTC) Received: by mail-pl1-f175.google.com with SMTP id g17so9967770plh.2 for ; Mon, 18 Jul 2022 12:37:22 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=20210112; h=from:to:cc:subject:date:message-id:in-reply-to:references :mime-version:content-transfer-encoding; bh=U3bHH+VCApNk0EgSMkQpPIu/hogjL02kkGQOh371JqE=; b=cgVgCcOubWJlYQh87ciup80RfrTt14z9y9XheMGRDWpUOcN7tz83N1/LeVdqY4FC8x NjZ299ZmT5ayN+y3qeWtB4AiGGlLv0bTeoa41zMsLcKw7Yvbr8j3tdxbVbn/BCQj5UnM N+DKf8yCPT06zTRIR4vx6LrfvTTsQMm6mDcaVmmqoslMBPkuYDGQM8LAXE49wLZerlr4 O2XHyJrKy3OzAIEITCEW9D6Zdl1GZeNoRYmVFo5+eof3svAuQ9oW9pdkFfi1jwDp+2Kd cmWr6cYwHWT0ai9wmSNVyZ4nkkJV4zU269t4/tA7BL3pdN3OppGjcXUqwL5ga4yr3+rK lK/g== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20210112; h=x-gm-message-state:from:to:cc:subject:date:message-id:in-reply-to :references:mime-version:content-transfer-encoding; bh=U3bHH+VCApNk0EgSMkQpPIu/hogjL02kkGQOh371JqE=; b=HfHEmDoQcJNWv/VeJedjmBkF/tMTJ+VRQxNFr8CdPzLEk0dxz6trfPkPZvu+Pd7fwN p0JN6BCxYFtfsnboeoI8aaKMFu+Ua+rxxuMPLI8YkV37qfqgc/QH2nKHtS6K0Ga8HjuV 3NcGaf2WlOmRljyi7p5J8TT/ZlJc0rCP6pK7MRYvvvTBYJDyj/fHUcel9tR1pPyEaGjY tgRbtihLVy2HWHTpzDngB3qMFEH9ndSq0Wa+z1w2bzA/Rv1C1bYoWQWXbRavTO504f0Y 9Al6F+4vsi9rKs2ze3leW37HeGey4ofMhNCPNcDV9HQ3VHYzS3ounhFSkktLxHBu0pvw cJEw== X-Gm-Message-State: AJIora/8SkcWVy2wASeGoHV6LsllxwjyD6xk8SyE3PuDIXUmKHaRjjj+ wGiv/kyL8U5p7URA2fbs7aAcakS59V41+g== X-Google-Smtp-Source: AGRyM1t9fp3tA/qUiJLR5rxiQilje7wUJ0XNs2hdR5hLaXg/+Pz/WHXJKpxZoMJbK3OuMmRL/9fyZw== X-Received: by 2002:a17:90a:f114:b0:1ef:991f:12e7 with SMTP id cc20-20020a17090af11400b001ef991f12e7mr41584874pjb.199.1658173040943; Mon, 18 Jul 2022 12:37:20 -0700 (PDT) Received: from sc2-haas01-esx0118.eng.vmware.com ([66.170.99.1]) by smtp.gmail.com with ESMTPSA id q6-20020a170902a3c600b0016bc4a6ce28sm9907887plb.98.2022.07.18.12.37.19 (version=TLS1_3 cipher=TLS_AES_256_GCM_SHA384 bits=256/256); Mon, 18 Jul 2022 12:37:20 -0700 (PDT) From: Nadav Amit X-Google-Original-From: Nadav Amit To: linux-mm@kvack.org Cc: linux-kernel@vger.kernel.org, Andrew Morton , Mike Rapoport , Axel Rasmussen , Nadav Amit , Andrea Arcangeli , Andrew Cooper , Andy Lutomirski , Dave Hansen , David Hildenbrand , Peter Xu , Peter Zijlstra , Thomas Gleixner , Will Deacon , Yu Zhao , Nick Piggin Subject: [RFC PATCH 12/14] x86/tlb: no flush on PTE change from RW->RO when PTE is clean Date: Mon, 18 Jul 2022 05:02:10 -0700 Message-Id: <20220718120212.3180-13-namit@vmware.com> X-Mailer: git-send-email 2.25.1 In-Reply-To: <20220718120212.3180-1-namit@vmware.com> References: <20220718120212.3180-1-namit@vmware.com> MIME-Version: 1.0 ARC-Seal: i=1; s=arc-20220608; d=hostedemail.com; t=1658173043; a=rsa-sha256; cv=none; b=iovMz3F11hN/RqYVvNB/KSoEFbTc7TbDYr6j2qFyZtNQxDMmwU4WoZYLDV4pW8a7HzWQLs SE55CQ34SvO/tRBudRXJ06sm9d6g7d6T0irF0Kp5JDnSn7EPu9yc4tz5rJbxlse7UNih5e +UlqhY+6LNcz0sRfqLhQtK0qCqPACAM= ARC-Authentication-Results: i=1; imf22.hostedemail.com; dkim=pass header.d=gmail.com header.s=20210112 header.b=cgVgCcOu; dmarc=pass (policy=none) header.from=gmail.com; spf=none (imf22.hostedemail.com: domain of MAILER-DAEMON@hostedemail.com has no SPF policy when checking 216.40.44.13) smtp.mailfrom=MAILER-DAEMON@hostedemail.com ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=hostedemail.com; s=arc-20220608; t=1658173043; h=from:from:sender:reply-to:subject:subject:date:date: message-id:message-id:to:to:cc:cc:mime-version:mime-version: content-type:content-transfer-encoding:content-transfer-encoding: in-reply-to:in-reply-to:references:references:dkim-signature; bh=U3bHH+VCApNk0EgSMkQpPIu/hogjL02kkGQOh371JqE=; b=zYzFsn+FLVn770j4TymcNB2SAjh+33MRAJwIgzfoGrUOJ6aE3iy3jBOU1zyTHF+zQuCvWy EvSI07IhW4HJrfmUYdsXO8jjMwbjwV4NekOwrJsIu4zl5GQKeXi6uTDN4HQpJ+Mo5MbtJP 5nvD+8O0KA95bw1p3owsVBljYPWwQPY= X-HE-Tag-Orig: 1658173042-468649 X-Rspam-User: Authentication-Results: imf22.hostedemail.com; dkim=pass header.d=gmail.com header.s=20210112 header.b=cgVgCcOu; dmarc=pass (policy=none) header.from=gmail.com; spf=none (imf22.hostedemail.com: domain of MAILER-DAEMON@hostedemail.com has no SPF policy when checking 216.40.44.13) smtp.mailfrom=MAILER-DAEMON@hostedemail.com X-Rspamd-Queue-Id: 2A20EC0087 X-Rspamd-Server: rspam12 X-Stat-Signature: qt84gm64q7i4ewdggs63tpwsotqan9hi X-HE-Tag: 1658173042-229999 X-Bogosity: Ham, tests=bogofilter, spamicity=0.000000, version=1.2.4 Sender: owner-linux-mm@kvack.org Precedence: bulk X-Loop: owner-majordomo@kvack.org List-ID: From: Nadav Amit On x86 it is possible to skip a TLB flush when a RW entry become RO and the PTE is clean. Add logic to detect this case and skip the flush. Cc: Andrea Arcangeli Cc: Andrew Cooper Cc: Andrew Morton Cc: Andy Lutomirski Cc: Dave Hansen Cc: David Hildenbrand Cc: Peter Xu Cc: Peter Zijlstra Cc: Thomas Gleixner Cc: Will Deacon Cc: Yu Zhao Cc: Nick Piggin Signed-off-by: Nadav Amit --- arch/x86/include/asm/tlbflush.h | 16 ++++++++++++++-- 1 file changed, 14 insertions(+), 2 deletions(-) diff --git a/arch/x86/include/asm/tlbflush.h b/arch/x86/include/asm/tlbflush.h index 4f98735ab07a..58c95e36b098 100644 --- a/arch/x86/include/asm/tlbflush.h +++ b/arch/x86/include/asm/tlbflush.h @@ -271,8 +271,9 @@ static inline enum pte_flush_type pte_flags_flush_type(unsigned long oldflags, * dirty/access bit if needed without a fault. */ const pteval_t flush_on_clear = _PAGE_DIRTY | _PAGE_PRESENT | - _PAGE_ACCESSED | _PAGE_RW; + _PAGE_ACCESSED; const pteval_t flush_on_set = _PAGE_NX; + const pteval_t flush_on_special = _PAGE_RW; const pteval_t flush_on_set_relaxed = _PAGE_RW; const pteval_t flush_on_clear_relaxed = _PAGE_NX; const pteval_t software_flags = _PAGE_SOFTW1 | _PAGE_SOFTW2 | @@ -302,6 +303,17 @@ static inline enum pte_flush_type pte_flags_flush_type(unsigned long oldflags, if (diff & oldflags & flush_on_clear) return PTE_FLUSH_STRICT; + /* + * Did any of the 'flush_on_set' flags was set between 'oldflags' and + * 'newflags'? + */ + if (diff & newflags & flush_on_set) + return PTE_FLUSH_STRICT; + + /* On RW->RO, a flush is needed if the old entry is dirty */ + if ((diff & oldflags & _PAGE_RW) && (oldflags & _PAGE_DIRTY)) + return PTE_FLUSH_STRICT; + /* Flush on modified flags. */ if (diff & flush_on_change) return PTE_FLUSH_STRICT; @@ -314,7 +326,7 @@ static inline enum pte_flush_type pte_flags_flush_type(unsigned long oldflags, /* Ensure there are no flags that were left behind */ if (IS_ENABLED(CONFIG_DEBUG_VM) && - (diff & ~(flush_on_clear | flush_on_set | + (diff & ~(flush_on_clear | flush_on_set | flush_on_special | software_flags | flush_on_change))) { VM_WARN_ON_ONCE(1); return PTE_FLUSH_STRICT;