From patchwork Tue Oct 9 04:32:50 2018 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Ashish Mhetre X-Patchwork-Id: 10631917 Return-Path: Received: from mail.wl.linuxfoundation.org (pdx-wl-mail.web.codeaurora.org [172.30.200.125]) by pdx-korg-patchwork-2.web.codeaurora.org (Postfix) with ESMTP id D4FCC16B1 for ; Tue, 9 Oct 2018 04:33:01 +0000 (UTC) Received: from mail.wl.linuxfoundation.org (localhost [127.0.0.1]) by mail.wl.linuxfoundation.org (Postfix) with ESMTP id C02D729A9C for ; Tue, 9 Oct 2018 04:33:01 +0000 (UTC) Received: by mail.wl.linuxfoundation.org (Postfix, from userid 486) id B2B7329AB3; Tue, 9 Oct 2018 04:33:01 +0000 (UTC) X-Spam-Checker-Version: SpamAssassin 3.3.1 (2010-03-16) on pdx-wl-mail.web.codeaurora.org X-Spam-Level: X-Spam-Status: No, score=-3.0 required=2.0 tests=BAYES_00,DKIM_SIGNED, DKIM_VALID,DKIM_VALID_AU,MAILING_LIST_MULTI,RCVD_IN_DNSWL_NONE autolearn=ham version=3.3.1 Received: from kanga.kvack.org (kanga.kvack.org [205.233.56.17]) by mail.wl.linuxfoundation.org (Postfix) with ESMTP id 7CB1D29A9C for ; Tue, 9 Oct 2018 04:33:00 +0000 (UTC) Received: by kanga.kvack.org (Postfix) id B71766B0003; Tue, 9 Oct 2018 00:32:58 -0400 (EDT) Delivered-To: linux-mm-outgoing@kvack.org Received: by kanga.kvack.org (Postfix, from userid 40) id B1F056B0005; Tue, 9 Oct 2018 00:32:58 -0400 (EDT) X-Original-To: int-list-linux-mm@kvack.org X-Delivered-To: int-list-linux-mm@kvack.org Received: by kanga.kvack.org (Postfix, from userid 63042) id A0C876B0006; Tue, 9 Oct 2018 00:32:58 -0400 (EDT) X-Original-To: linux-mm@kvack.org X-Delivered-To: linux-mm@kvack.org Received: from mail-yb1-f199.google.com (mail-yb1-f199.google.com [209.85.219.199]) by kanga.kvack.org (Postfix) with ESMTP id 6EB5D6B0003 for ; Tue, 9 Oct 2018 00:32:58 -0400 (EDT) Received: by mail-yb1-f199.google.com with SMTP id z14-v6so101758ybp.6 for ; Mon, 08 Oct 2018 21:32:58 -0700 (PDT) X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20161025; h=x-gm-message-state:from:to:cc:subject:date:message-id:mime-version :dkim-signature; bh=XnjA21G31lCCBYquM0x83DD/c7kKanEJCZvEYVKBo0Y=; b=lLsBRCk7zi56VtsQSHaHXZDmIIDl89fPx9iNfEl9j0x0naxEhEJ+n/EsdEyeF/Ip/b VrpgBqmVE1wqo6V74fyKTWdEVs6T/MdBlKMnjk376V6u+duuel79mNk/4kKISMxWpkM6 m83Y7HkYuiVtivq2gafYM7jsafTF//a/qx1wqHsiPuHPFjxRWTKuxH4JpS1V+29Rvyct AG9Xg25zSA1iorG9TxbzDKJ3BuuDWv1scqWXP5SECznp3ixzUowSSxVyjumR15MwrgUE BtpFaar+THTnHSdwgJ27f0xKTp4L3EdHgJWrMctJaTx0+b8XB4QoG8n3JwQ+HSOqHpqM kFUA== X-Gm-Message-State: ABuFfogdxgze2Kical8AvdudgfXH+8zmgYxmqmMpvyXErZ3+819158Nt cfOyCW61n1rRqs3hrQQg82tvQAuZpQc90W0+VT1NSW4S4JVzeBgvxLkdsWu5vCKuq/8k7cGAUdg +aC02eF542kj84FKlLNTNuHYLz53SfOqzm+/DWHDPDpzf9bXXB4eMy/vzUCmyNbImOQ== X-Received: by 2002:a0d:fc07:: with SMTP id m7-v6mr15334180ywf.220.1539059578100; Mon, 08 Oct 2018 21:32:58 -0700 (PDT) X-Google-Smtp-Source: ACcGV61lLhvWpVQ+FExjYyhDH0l7iF/Cse/Yn8ppW0W7HUAEVc2YTWS7vEoHDbHdwVAfb/9iHl6B X-Received: by 2002:a0d:fc07:: with SMTP id m7-v6mr15334165ywf.220.1539059577236; Mon, 08 Oct 2018 21:32:57 -0700 (PDT) ARC-Seal: i=1; a=rsa-sha256; t=1539059577; cv=none; d=google.com; s=arc-20160816; b=ekCDueRddwoE9qW0pm0HVX3wZ91L/iT+nmyCpwVllo6VPJ1HESxps8TwwzlOiTJTIL nFun1f8CCbtER+DuSMT29VFSLTL0iQLQGRgN2Ydge97ZU1rJUAIbJwa0r8jFw6XkdGBj 5X1pQkFMKCiwlxiErLujWWD4vluiam7J6Q/KsNKcg//vvunMzv5/KH0I53qOjc5PAkLL Kw8nOndiIjzjkq0GAs+macRiw0wQT5456d74Z9/GYoXiAFjE7TYhEvzn+HEu1CSaOgJ0 RenOa98rQNItjMQcYeqQPAI1KMnROUtuGuFCMQQeRN+vhjcw6IUnNXdDdGg5dxSCbHez b3fg== ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=google.com; s=arc-20160816; h=dkim-signature:mime-version:message-id:date:subject:cc:to:from; bh=XnjA21G31lCCBYquM0x83DD/c7kKanEJCZvEYVKBo0Y=; b=cO8sM9bkpuxSPriFVcY9cvE1y/3pp28DQnPYC2l1+YU6QXyrxJwR5GkNceUiIDvmC+ /VDQQ77A2BtEHKL2YsZkB6BPnDVOU51fDYywktJzjr+wErC6fnvIOfwqTMkaoW8yZFiM q41E2pFTvGt6hvbUlQyUFlnOrp8sWuXmXVZFFfzSyUnyzHT3Sc/9BPQXlgZRZLCmgbcl qD+Cb5EnOJSSo7MBO9evp9qVcLWC/9/5f4S6mNROZ9z1QMNGuIFYpsDQwL3PI548yqQu DaxBUnph/2CipvN5ekLbCFgkdJuUwUIRqDH+hcenHMPwpqtAmHvox/nsOcvpL+npIeYN ELyQ== ARC-Authentication-Results: i=1; mx.google.com; dkim=pass header.i=@nvidia.com header.s=n1 header.b=eLBgd77b; spf=pass (google.com: domain of amhetre@nvidia.com designates 216.228.121.64 as permitted sender) smtp.mailfrom=amhetre@nvidia.com; dmarc=pass (p=NONE sp=NONE dis=NONE) header.from=nvidia.com Received: from hqemgate15.nvidia.com (hqemgate15.nvidia.com. [216.228.121.64]) by mx.google.com with ESMTPS id q188-v6si4785364ywc.354.2018.10.08.21.32.56 for (version=TLS1_2 cipher=ECDHE-RSA-AES128-GCM-SHA256 bits=128/128); Mon, 08 Oct 2018 21:32:57 -0700 (PDT) Received-SPF: pass (google.com: domain of amhetre@nvidia.com designates 216.228.121.64 as permitted sender) client-ip=216.228.121.64; Authentication-Results: mx.google.com; dkim=pass header.i=@nvidia.com header.s=n1 header.b=eLBgd77b; spf=pass (google.com: domain of amhetre@nvidia.com designates 216.228.121.64 as permitted sender) smtp.mailfrom=amhetre@nvidia.com; dmarc=pass (p=NONE sp=NONE dis=NONE) header.from=nvidia.com Received: from hqpgpgate101.nvidia.com (Not Verified[216.228.121.13]) by hqemgate15.nvidia.com (using TLS: TLSv1.2, DES-CBC3-SHA) id ; Mon, 08 Oct 2018 21:32:02 -0700 Received: from HQMAIL106.nvidia.com ([172.20.161.6]) by hqpgpgate101.nvidia.com (PGP Universal service); Mon, 08 Oct 2018 21:32:56 -0700 X-PGP-Universal: processed; by hqpgpgate101.nvidia.com on Mon, 08 Oct 2018 21:32:56 -0700 Received: from HQMAIL103.nvidia.com (172.20.187.11) by HQMAIL106.nvidia.com (172.18.146.12) with Microsoft SMTP Server (TLS) id 15.0.1395.4; Tue, 9 Oct 2018 04:32:55 +0000 Received: from hqnvemgw01.nvidia.com (172.20.150.20) by HQMAIL103.nvidia.com (172.20.187.11) with Microsoft SMTP Server (TLS) id 15.0.1395.4 via Frontend Transport; Tue, 9 Oct 2018 04:32:55 +0000 Received: from amhetre.nvidia.com (Not Verified[10.24.229.42]) by hqnvemgw01.nvidia.com with Trustwave SEG (v7,5,8,10121) id ; Mon, 08 Oct 2018 21:32:55 -0700 From: Ashish Mhetre To: , CC: , , Shaohua Li , Shaohua Li , , Peter Zijlstra , Ingo Molnar Subject: [PATCH] x86/mm: In the PTE swapout page reclaim case clear the accessed bit instead of flushing the TLB Date: Tue, 9 Oct 2018 10:02:50 +0530 Message-ID: <1539059570-9043-1-git-send-email-amhetre@nvidia.com> X-Mailer: git-send-email 2.1.4 X-NVConfidentiality: public MIME-Version: 1.0 DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=nvidia.com; s=n1; t=1539059522; bh=XnjA21G31lCCBYquM0x83DD/c7kKanEJCZvEYVKBo0Y=; h=X-PGP-Universal:From:To:CC:Subject:Date:Message-ID:X-Mailer: X-NVConfidentiality:MIME-Version:Content-Type; b=eLBgd77b4s0bKUCStQJblzVIaZ2LdpyOqMKw4mcp95K08bjwS48Q/eqGUjRbpQ2T3 hDXhdFFGPpwKG60sk/t8SLOdgzcsmeWXd4HDtoA4B7LbKDpNb5BOGEP1cIIb3pOIn6 C9xRhOC2S2yyR1DXped8DV3zGQVaawYGDwG+eASV+wtWQAuvKQqmbdcJKnUmM+DHLp QKkNbPT/ftXkkaPeVrmUcz6Tv69SXjQT+F/udqPvrGVq/zq6209hvx4iqgyE8Z6t2O bI0v7s6M3kOGy9ENZVIssN9M8rtRh4s9AeMfjp6cFRf+kQ5pOctWru9aEEu2/X3AzZ ytxaGrh0zaR2g== X-Bogosity: Ham, tests=bogofilter, spamicity=0.000000, version=1.2.4 Sender: owner-linux-mm@kvack.org Precedence: bulk X-Loop: owner-majordomo@kvack.org List-ID: X-Virus-Scanned: ClamAV using ClamSMTP From: Shaohua Li We use the accessed bit to age a page at page reclaim time, and currently we also flush the TLB when doing so. But in some workloads TLB flush overhead is very heavy. In my simple multithreaded app with a lot of swap to several pcie SSDs, removing the tlb flush gives about 20% ~ 30% swapout speedup. Fortunately just removing the TLB flush is a valid optimization: on x86 CPUs, clearing the accessed bit without a TLB flush doesn't cause data corruption. It could cause incorrect page aging and the (mistaken) reclaim of hot pages, but the chance of that should be relatively low. So as a performance optimization don't flush the TLB when clearing the accessed bit, it will eventually be flushed by a context switch or a VM operation anyway. [ In the rare event of it not getting flushed for a long time the delay shouldn't really matter because there's no real memory pressure for swapout to react to. ] Suggested-by: Linus Torvalds Signed-off-by: Shaohua Li Acked-by: Rik van Riel Acked-by: Mel Gorman Acked-by: Hugh Dickins Acked-by: Johannes Weiner Cc: linux-mm@kvack.org Cc: Peter Zijlstra Link: http://lkml.kernel.org/r/20140408075809.GA1764@kernel.org [ Rewrote the changelog and the code comments. ] Signed-off-by: Ingo Molnar --- arch/x86/mm/pgtable.c | 21 ++++++++++++++------- 1 file changed, 14 insertions(+), 7 deletions(-) diff --git a/arch/x86/mm/pgtable.c b/arch/x86/mm/pgtable.c index c96314a..0004ac7 100644 --- a/arch/x86/mm/pgtable.c +++ b/arch/x86/mm/pgtable.c @@ -399,13 +399,20 @@ int pmdp_test_and_clear_young(struct vm_area_struct *vma, int ptep_clear_flush_young(struct vm_area_struct *vma, unsigned long address, pte_t *ptep) { - int young; - - young = ptep_test_and_clear_young(vma, address, ptep); - if (young) - flush_tlb_page(vma, address); - - return young; + /* + * On x86 CPUs, clearing the accessed bit without a TLB flush + * doesn't cause data corruption. [ It could cause incorrect + * page aging and the (mistaken) reclaim of hot pages, but the + * chance of that should be relatively low. ] + * + * So as a performance optimization don't flush the TLB when + * clearing the accessed bit, it will eventually be flushed by + * a context switch or a VM operation anyway. [ In the rare + * event of it not getting flushed for a long time the delay + * shouldn't really matter because there's no real memory + * pressure for swapout to react to. ] + */ + return ptep_test_and_clear_young(vma, address, ptep); } #ifdef CONFIG_TRANSPARENT_HUGEPAGE