From patchwork Wed Jul 25 15:52:43 2018 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Nicholas Piggin X-Patchwork-Id: 10544241 Return-Path: Received: from mail.wl.linuxfoundation.org (pdx-wl-mail.web.codeaurora.org [172.30.200.125]) by pdx-korg-patchwork-2.web.codeaurora.org (Postfix) with ESMTP id A1C401822 for ; Wed, 25 Jul 2018 15:53:01 +0000 (UTC) Received: from mail.wl.linuxfoundation.org (localhost [127.0.0.1]) by mail.wl.linuxfoundation.org (Postfix) with ESMTP id 917222A5C1 for ; Wed, 25 Jul 2018 15:53:01 +0000 (UTC) Received: by mail.wl.linuxfoundation.org (Postfix, from userid 486) id 8FE702A65B; Wed, 25 Jul 2018 15:53:01 +0000 (UTC) X-Spam-Checker-Version: SpamAssassin 3.3.1 (2010-03-16) on pdx-wl-mail.web.codeaurora.org X-Spam-Level: X-Spam-Status: No, score=-3.0 required=2.0 tests=BAYES_00,DKIM_SIGNED, DKIM_VALID,DKIM_VALID_AU,FREEMAIL_FROM,MAILING_LIST_MULTI,RCVD_IN_DNSWL_NONE autolearn=ham version=3.3.1 Received: from kanga.kvack.org (kanga.kvack.org [205.233.56.17]) by mail.wl.linuxfoundation.org (Postfix) with ESMTP id 401BD2A5C1 for ; Wed, 25 Jul 2018 15:53:01 +0000 (UTC) Received: by kanga.kvack.org (Postfix) id 190FB6B0287; Wed, 25 Jul 2018 11:53:00 -0400 (EDT) Delivered-To: linux-mm-outgoing@kvack.org Received: by kanga.kvack.org (Postfix, from userid 40) id 141996B028B; Wed, 25 Jul 2018 11:53:00 -0400 (EDT) X-Original-To: int-list-linux-mm@kvack.org X-Delivered-To: int-list-linux-mm@kvack.org Received: by kanga.kvack.org (Postfix, from userid 63042) id 057806B028C; Wed, 25 Jul 2018 11:53:00 -0400 (EDT) X-Original-To: linux-mm@kvack.org X-Delivered-To: linux-mm@kvack.org Received: from mail-pf1-f199.google.com (mail-pf1-f199.google.com [209.85.210.199]) by kanga.kvack.org (Postfix) with ESMTP id B4CD66B0287 for ; Wed, 25 Jul 2018 11:52:59 -0400 (EDT) Received: by mail-pf1-f199.google.com with SMTP id u16-v6so1000674pfm.15 for ; Wed, 25 Jul 2018 08:52:59 -0700 (PDT) X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20161025; h=x-gm-message-state:dkim-signature:from:to:cc:subject:date :message-id:in-reply-to:references; bh=K/wRWZpM/AGbkrzMDg7aU9hs152pni5PDFBMxhN+AjM=; b=X7nFxPakV9eJMmFwhW8CyZ6V43ajJdua8+kKB37DM0pRU8k8xg5AAxflCTGLWkHG60 vUpdql3UhaHz7F6CSKcHhtaUd+FWbsZiNG0xKklNauVNbqY5BrP2zJj+GiHZzOQeftef h9cU4LXVbPVqg426k72rLHqweCPXf8tACD7S4wMR7hJ7pIJ5368lIQuHdeRcXmkKgIaU i4yAwtC/zvD2rxUF2fjEiYEgy6rwNxM+dIH4+LKsluyNO8Soaqp5Ud0qYz8L6eYc7And /OYRqsL5J2nNodEdQFYqXWCXYixXTVLP7EWmTic5xLVo5fDwkEq36GdvksCC4Ken9ueO 9RwQ== X-Gm-Message-State: AOUpUlFKy3aBqA/6n1uQIOpcaYj3Wg/7jv8EPrL5+sERK74m8zK5eha2 7r1eO1JdvETPfBGqcsRpwq8LIlM0Zy5JpjIjuYgaUlts0dEKdhjYiw6sy5edixdA3vNMItqFvdl OFHunAaFx3hzmCkAZappKLLbuZhOpElUGIuR2eZBeso4uLtTWxpac6bG0MuItFZ8C2eEgv9KVxf crgcvzW/ViEU0d8UpqOs3tOYeY/qPsWmwXy6dQQrrWK9rxBD43vwjWa06NVpQ1n7K7jg7yviCud /f0cOaEWDrteUnJR/JsHrukUy9EmbId4CVEK9traC+CRa/29a06kcPwZ+nnvf7xnBmb/t9S+yfh Ltn/v1cdc1YzWXE9vn7BH8oCufbjXyWaRsewBGRB6rEtCi7OSupIe7CdBLQVsXI+l++3XAnVc6C X X-Received: by 2002:a17:902:b40c:: with SMTP id x12-v6mr21911272plr.163.1532533979435; Wed, 25 Jul 2018 08:52:59 -0700 (PDT) X-Received: by 2002:a17:902:b40c:: with SMTP id x12-v6mr21911242plr.163.1532533978792; Wed, 25 Jul 2018 08:52:58 -0700 (PDT) ARC-Seal: i=1; a=rsa-sha256; t=1532533978; cv=none; d=google.com; s=arc-20160816; b=mYEthgNJYj60lMU5ERDiYO+//t2jFilukiG92i/RiBHoQ9kuBiEeR8QGRe9+6H5KdD 2EggyIuusTTu4U1u4Xs9glEWZ/a0TtxwIzO6hOlfgtH2ejdkx1JimTd7LDUIj/hKxb3Q FlXRBo+WYiUjbFOHoAWlQ0qoV1R0rDHdO6onUrhTa0eyEjQU+iEheRoohXt1iF7tVOFb wRmVeotTMiqVLcAp4MwqiQ7UXHyoLgqjBOcpgYb8B81bR6/2Ly1fAnDoXwj6anwaousQ ln/qBj7AxNQFLrAmSnzQNGzgE3y9xTXqTQ54rsFYGRBL7T1609wxOMn3wsJIOpJ0a4Rg 1hVA== ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=google.com; s=arc-20160816; h=references:in-reply-to:message-id:date:subject:cc:to:from :dkim-signature:arc-authentication-results; bh=K/wRWZpM/AGbkrzMDg7aU9hs152pni5PDFBMxhN+AjM=; b=hGFf0cQWPCd2dogSceQktx18QM3UQ4bjRKCtp4jtUkZzPTMuKfCD2QsLRj1FMdaYtY P/DRCTSgxXSp8rOE3rALgdyB8h9+g5E7kqs0YE5Phr04yoGlWX+A5YHA16IUn1ju11vv 6RLYpaNmm3/BvsQBl+VNaR6G61aWvtSqBcb433VWI/6YbQ4L7ToByCDStxVCWbkRSe4W ETFqKulxsNx6R1PR1RyBUA0qdt167b/vsMUoqkWOegfg9KsYfeFbmrYf98cluSYhDsw4 jBM+vstiXLhhunDrp6aEBFyG+VZN5fwpRhH6WiRsVLznmNzu1BjOf/c30JffrQKPV60+ xbxA== ARC-Authentication-Results: i=1; mx.google.com; dkim=pass header.i=@gmail.com header.s=20161025 header.b="ma8z2A/j"; spf=pass (google.com: domain of npiggin@gmail.com designates 209.85.220.65 as permitted sender) smtp.mailfrom=npiggin@gmail.com; dmarc=pass (p=NONE sp=QUARANTINE dis=NONE) header.from=gmail.com Received: from mail-sor-f65.google.com (mail-sor-f65.google.com. [209.85.220.65]) by mx.google.com with SMTPS id m125-v6sor4033155pgm.6.2018.07.25.08.52.58 for (Google Transport Security); Wed, 25 Jul 2018 08:52:58 -0700 (PDT) Received-SPF: pass (google.com: domain of npiggin@gmail.com designates 209.85.220.65 as permitted sender) client-ip=209.85.220.65; Authentication-Results: mx.google.com; dkim=pass header.i=@gmail.com header.s=20161025 header.b="ma8z2A/j"; spf=pass (google.com: domain of npiggin@gmail.com designates 209.85.220.65 as permitted sender) smtp.mailfrom=npiggin@gmail.com; dmarc=pass (p=NONE sp=QUARANTINE dis=NONE) header.from=gmail.com DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=20161025; h=from:to:cc:subject:date:message-id:in-reply-to:references; bh=K/wRWZpM/AGbkrzMDg7aU9hs152pni5PDFBMxhN+AjM=; b=ma8z2A/jxCaq6DwkQd+T6yi1HYjCEMWw04jtBYac+r+9P+XWO3u0CcZH6C/HHdNfdL EXfbmyrrmjcCC5dX2uTLmNXwo5NgzqVs+mgr2kbUamSCp+uj+bdE+Kbi3sRdDobhk2zz 9qEIUbu1Ga5Vou4YrU6yoQ3g8LZHqbuJMtiSsXrsw2LfBDuHnWLebxxwjz75wpRXY6VI l00aTzzGomupAFS9R6//RWoThsOfPEJ7GC1xJ6imm2NMev6EVhJFABHGbiWB3ER9DftY LEtDiwyD7NTy3V1CSlxlA5aeMDFcy2Kio0wihJMXi4QDYConhfCl4Ga+fHOSTgGmhUuU WJdg== X-Google-Smtp-Source: AAOMgpdWBIwJmQ1fXACJaTG3kCLAXCXzsiKVtGTzpi2ABhQFW42EOk+OCDOD9R0JjB81aE7aQ2/mYg== X-Received: by 2002:a63:555:: with SMTP id 82-v6mr21394995pgf.25.1532533978421; Wed, 25 Jul 2018 08:52:58 -0700 (PDT) Received: from roar.au.ibm.com ([61.69.188.107]) by smtp.gmail.com with ESMTPSA id j72-v6sm23438315pge.19.2018.07.25.08.52.55 (version=TLS1_2 cipher=ECDHE-RSA-AES128-GCM-SHA256 bits=128/128); Wed, 25 Jul 2018 08:52:57 -0700 (PDT) From: Nicholas Piggin To: linux-mm@kvack.org Cc: Nicholas Piggin , linux-arch@vger.kernel.org Subject: [RFC PATCH 1/4] mm: munmap optimise single threaded page freeing Date: Thu, 26 Jul 2018 01:52:43 +1000 Message-Id: <20180725155246.1085-2-npiggin@gmail.com> X-Mailer: git-send-email 2.17.0 In-Reply-To: <20180725155246.1085-1-npiggin@gmail.com> References: <20180725155246.1085-1-npiggin@gmail.com> X-Bogosity: Ham, tests=bogofilter, spamicity=0.000000, version=1.2.4 Sender: owner-linux-mm@kvack.org Precedence: bulk X-Loop: owner-majordomo@kvack.org List-ID: X-Virus-Scanned: ClamAV using ClamSMTP In case a single threaded process is zapping its own mappings, there should be no concurrent memory accesses through the TLBs, and so it is safe to free pages immediately rather than batch them up. --- mm/memory.c | 9 +++++++++ 1 file changed, 9 insertions(+) diff --git a/mm/memory.c b/mm/memory.c index 135d18b31e44..773d588b371d 100644 --- a/mm/memory.c +++ b/mm/memory.c @@ -296,6 +296,15 @@ bool __tlb_remove_page_size(struct mmu_gather *tlb, struct page *page, int page_ VM_BUG_ON(!tlb->end); VM_WARN_ON(tlb->page_size != page_size); + /* + * When this is our mm and there are no other users, there can not be + * a concurrent memory access. + */ + if (current->mm == tlb->mm && atomic_read(&tlb->mm->mm_users) < 2) { + free_page_and_swap_cache(page); + return false; + } + batch = tlb->active; /* * Add the page and check if we are full. If so From patchwork Wed Jul 25 15:52:44 2018 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Nicholas Piggin X-Patchwork-Id: 10544243 Return-Path: Received: from mail.wl.linuxfoundation.org (pdx-wl-mail.web.codeaurora.org [172.30.200.125]) by pdx-korg-patchwork-2.web.codeaurora.org (Postfix) with ESMTP id AF964139A for ; Wed, 25 Jul 2018 15:53:04 +0000 (UTC) Received: from mail.wl.linuxfoundation.org (localhost [127.0.0.1]) by mail.wl.linuxfoundation.org (Postfix) with ESMTP id 9E5202A5C1 for ; Wed, 25 Jul 2018 15:53:04 +0000 (UTC) Received: by mail.wl.linuxfoundation.org (Postfix, from userid 486) id 91D5D2A5FF; Wed, 25 Jul 2018 15:53:04 +0000 (UTC) X-Spam-Checker-Version: SpamAssassin 3.3.1 (2010-03-16) on pdx-wl-mail.web.codeaurora.org X-Spam-Level: X-Spam-Status: No, score=-3.0 required=2.0 tests=BAYES_00,DKIM_SIGNED, DKIM_VALID,DKIM_VALID_AU,FREEMAIL_FROM,MAILING_LIST_MULTI,RCVD_IN_DNSWL_NONE autolearn=ham version=3.3.1 Received: from kanga.kvack.org (kanga.kvack.org [205.233.56.17]) by mail.wl.linuxfoundation.org (Postfix) with ESMTP id 30D3E2A5C1 for ; Wed, 25 Jul 2018 15:53:04 +0000 (UTC) Received: by kanga.kvack.org (Postfix) id C1BFB6B028C; Wed, 25 Jul 2018 11:53:02 -0400 (EDT) Delivered-To: linux-mm-outgoing@kvack.org Received: by kanga.kvack.org (Postfix, from userid 40) id BA3C76B028D; Wed, 25 Jul 2018 11:53:02 -0400 (EDT) X-Original-To: int-list-linux-mm@kvack.org X-Delivered-To: int-list-linux-mm@kvack.org Received: by kanga.kvack.org (Postfix, from userid 63042) id 95C406B0291; Wed, 25 Jul 2018 11:53:02 -0400 (EDT) X-Original-To: linux-mm@kvack.org X-Delivered-To: linux-mm@kvack.org Received: from mail-pf1-f198.google.com (mail-pf1-f198.google.com [209.85.210.198]) by kanga.kvack.org (Postfix) with ESMTP id 4F2866B028C for ; Wed, 25 Jul 2018 11:53:02 -0400 (EDT) Received: by mail-pf1-f198.google.com with SMTP id j15-v6so989717pff.12 for ; Wed, 25 Jul 2018 08:53:02 -0700 (PDT) X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20161025; h=x-gm-message-state:dkim-signature:from:to:cc:subject:date :message-id:in-reply-to:references; bh=gLcXLHSvMjLl1a5SqU+HvKEkz+b8ngxKOrFFxTPno9c=; b=L9n/ATa0af6oTRmCeEJtIDXk00hHhyogCRssPFFvxOe1o79kSw83kV+HpNLgbibDuR 2P4t/k6ijtip8Qi3ypZDuWG1djYUulZXkgpYBlzxCyFOcADtyT9+YJat5+mHRmLk8ISN BwLE9Jga2WNp2ezZlGzg1GOKL119saYrCGJMH2SdU4A+x8SrCWBA3YwRNwk2kGMq5SpF d8p1y/TEx220eAPkJZQAHZ7WDfT0gGtGdUllWf/SClN5Rvyx7PImdxB2g8bKoytvKdJQ 3/0KJD36xXFto4aQoU2UwjAuX8O9l3KWPCvcydFXVYvN7io/FaexyNHVnmQOQ36PfmRY w7rg== X-Gm-Message-State: AOUpUlGXGKgtHEltS2oc9mnu2IDFYC5CCqaV9Ow72U0RStlaW+NBQ8AR oU+/u16vZ0GHnJjog3JnYSECtid1BOKMP8Yeu814EAQYqD+6covvBqHLAADcD8H+UcYkhslnq5H 52yJ3nGPoXsGe+Ip6+pIZ3WiJcMZHxwvMFt9zCuv9hNtqBG3zfvDGkJfdd1S+VU/9CXAEVTf/x9 TdJzvrdBoNf/Y9cJ3PubbVrjg/T5NYypQE+eMMGCqm/8DXc5qNBNJ7grxoC90TqaEkZ4CvWybqA SLtG6p0v/dNmvs7TcFQ7GUT8Kqv9djbVmMA0HziZ68O0VpHruefHNGL/8WeIpOt+jU6ZgcuPBM/ BdxpSOfEzokaxxwK+ZagH6R71Dw6xWc2fGcvOJkcPwxfOFuoee1rHJLjCITlGGLyzphFwcMdv8o N X-Received: by 2002:a63:c902:: with SMTP id o2-v6mr21100152pgg.118.1532533982030; Wed, 25 Jul 2018 08:53:02 -0700 (PDT) X-Received: by 2002:a63:c902:: with SMTP id o2-v6mr21100122pgg.118.1532533981421; Wed, 25 Jul 2018 08:53:01 -0700 (PDT) ARC-Seal: i=1; a=rsa-sha256; t=1532533981; cv=none; d=google.com; s=arc-20160816; b=l6keGQO7oJiU7P4OJlHnYlvZZ7EtOOUYyVAccH5goJ590B/O8zvzNfkK/pwK0J675j l8vmA1taQOInY2RSceh4uj3rlsp2nCeRDqqc4W/mJxefCgKNWKr0IQgQ0ve+J7chSCuj q7tXaYxRN91IVCY/PJmz4fnVH1bAtK6GWhStqvDU54Y6sJphS2DxjneoYTcL5Zn5Rw1t fSDtbUXhrJXHIIRn1W+e1necLTcT2qTpTrNFaf3hdWT7PEcLSsdeaVTNwprmAi4CNOhn l77eo9ajsimLEAixCJkInm0oRfIu9tY/imRxds/ft9I0byInUA10cfUt2BdrTPv/Lk5U xfiw== ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=google.com; s=arc-20160816; h=references:in-reply-to:message-id:date:subject:cc:to:from :dkim-signature:arc-authentication-results; bh=gLcXLHSvMjLl1a5SqU+HvKEkz+b8ngxKOrFFxTPno9c=; b=xJAVTZ0xIhp9qWgDqZYbsiYZr6xnBVWxlXdCBp3R7uK3geti/cExU0yhuBdkInSbdE DevJqVfB3sAbNUuyMskpF4teJ3TZFgXjciGMahFTwyH0LYRof9n/JopD6s4G5+5zIJMW Grs+r9cRCmb2NQuzfZiF0XukFMtc1NvxdtB26Za/bKwszJuPvNPrbY8/auu8Cvf9XSk4 VPHq3GsL8juMuKeeekhBSXZyuD0qyPh9WK945mZhrMoqeyaqFdncMegVqfRaRJiI+HR/ +cGMHS5DgB7anWNC1s927ojFBUoayx20Qq+ovb4Zqr22JpHlPT9vVLLtgP4+zQgkCQfu 7m8Q== ARC-Authentication-Results: i=1; mx.google.com; dkim=pass header.i=@gmail.com header.s=20161025 header.b=dp250E5Z; spf=pass (google.com: domain of npiggin@gmail.com designates 209.85.220.65 as permitted sender) smtp.mailfrom=npiggin@gmail.com; dmarc=pass (p=NONE sp=QUARANTINE dis=NONE) header.from=gmail.com Received: from mail-sor-f65.google.com (mail-sor-f65.google.com. [209.85.220.65]) by mx.google.com with SMTPS id h3-v6sor4121712pld.86.2018.07.25.08.53.01 for (Google Transport Security); Wed, 25 Jul 2018 08:53:01 -0700 (PDT) Received-SPF: pass (google.com: domain of npiggin@gmail.com designates 209.85.220.65 as permitted sender) client-ip=209.85.220.65; Authentication-Results: mx.google.com; dkim=pass header.i=@gmail.com header.s=20161025 header.b=dp250E5Z; spf=pass (google.com: domain of npiggin@gmail.com designates 209.85.220.65 as permitted sender) smtp.mailfrom=npiggin@gmail.com; dmarc=pass (p=NONE sp=QUARANTINE dis=NONE) header.from=gmail.com DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=20161025; h=from:to:cc:subject:date:message-id:in-reply-to:references; bh=gLcXLHSvMjLl1a5SqU+HvKEkz+b8ngxKOrFFxTPno9c=; b=dp250E5Z4av6TXwK2FiGOeTPADAjItKut5H7zFcF/d/fRqS0s7hknOVTU5+VMXfkaX 7g6kTdbtmk/9F1vjL0RBc0x4ii5PRtKgiqpLol4KYSCCMf34MNfVuHnVoRCmwW4BcCwI S7jqxfOjmvOEbfKDt700G6izbgGXRzHoMRvCws4TQ3Ep8jLrrpEggfkwcGJ5D/Vswtok QWz/ISpB5YCqt6DgFlm+wNse/+Uafm2BuIh5Fw5zSjVYfWo0zGjtcdmAQ2snYolKEEVt 4Uh7ww5zAgPc8impdYT5CsrefOveyK50E+Sfg2bQKE47FLZo1lmOOkKZMpqFAmzJSS17 nWRw== X-Google-Smtp-Source: AAOMgpd85CNFa7uhnNYwyQIBOy3EvOvHYch1sIdeIaHCb3oR5VPQYm8BX8WmRkAFxDf6mENjqT64gw== X-Received: by 2002:a17:902:758c:: with SMTP id j12-v6mr21455822pll.195.1532533981052; Wed, 25 Jul 2018 08:53:01 -0700 (PDT) Received: from roar.au.ibm.com ([61.69.188.107]) by smtp.gmail.com with ESMTPSA id j72-v6sm23438315pge.19.2018.07.25.08.52.58 (version=TLS1_2 cipher=ECDHE-RSA-AES128-GCM-SHA256 bits=128/128); Wed, 25 Jul 2018 08:53:00 -0700 (PDT) From: Nicholas Piggin To: linux-mm@kvack.org Cc: Nicholas Piggin , linux-arch@vger.kernel.org Subject: [RFC PATCH 2/4] mm: zap_pte_range only flush under ptl if a dirty shared page was unmapped Date: Thu, 26 Jul 2018 01:52:44 +1000 Message-Id: <20180725155246.1085-3-npiggin@gmail.com> X-Mailer: git-send-email 2.17.0 In-Reply-To: <20180725155246.1085-1-npiggin@gmail.com> References: <20180725155246.1085-1-npiggin@gmail.com> X-Bogosity: Ham, tests=bogofilter, spamicity=0.000000, version=1.2.4 Sender: owner-linux-mm@kvack.org Precedence: bulk X-Loop: owner-majordomo@kvack.org List-ID: X-Virus-Scanned: ClamAV using ClamSMTP The force_flush is used for two cases, a tlb batch full, and a shared dirty page unmapped. Only the latter is required to flush the TLB under the page table lock, because the problem is page_mkclean returning when there are still writable TLB entries the page can be modified with. We are encountering cases of soft lockups due to high TLB flush latency with very large guests. There is probably some contetion in hypervisor and interconnect tuning to be done, and it's actually a hash MMU guest which has a whole other set of issues, but I'm looking for general ways to reduce TLB fushing under locks. --- mm/memory.c | 10 ++++++++-- 1 file changed, 8 insertions(+), 2 deletions(-) diff --git a/mm/memory.c b/mm/memory.c index 773d588b371d..1161ed3f1d0b 100644 --- a/mm/memory.c +++ b/mm/memory.c @@ -1281,6 +1281,7 @@ static unsigned long zap_pte_range(struct mmu_gather *tlb, { struct mm_struct *mm = tlb->mm; int force_flush = 0; + int locked_flush = 0; int rss[NR_MM_COUNTERS]; spinlock_t *ptl; pte_t *start_pte; @@ -1322,6 +1323,7 @@ static unsigned long zap_pte_range(struct mmu_gather *tlb, if (!PageAnon(page)) { if (pte_dirty(ptent)) { force_flush = 1; + locked_flush = 1; set_page_dirty(page); } if (pte_young(ptent) && @@ -1384,7 +1386,7 @@ static unsigned long zap_pte_range(struct mmu_gather *tlb, arch_leave_lazy_mmu_mode(); /* Do the actual TLB flush before dropping ptl */ - if (force_flush) + if (locked_flush) tlb_flush_mmu_tlbonly(tlb); pte_unmap_unlock(start_pte, ptl); @@ -1395,8 +1397,12 @@ static unsigned long zap_pte_range(struct mmu_gather *tlb, * memory too. Restart if we didn't do everything. */ if (force_flush) { - force_flush = 0; + if (!locked_flush) + tlb_flush_mmu_tlbonly(tlb); tlb_flush_mmu_free(tlb); + + force_flush = 0; + locked_flush = 0; if (addr != end) goto again; } From patchwork Wed Jul 25 15:52:45 2018 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Nicholas Piggin X-Patchwork-Id: 10544247 Return-Path: Received: from mail.wl.linuxfoundation.org (pdx-wl-mail.web.codeaurora.org [172.30.200.125]) by pdx-korg-patchwork-2.web.codeaurora.org (Postfix) with ESMTP id 579889093 for ; Wed, 25 Jul 2018 15:53:07 +0000 (UTC) Received: from mail.wl.linuxfoundation.org (localhost [127.0.0.1]) by mail.wl.linuxfoundation.org (Postfix) with ESMTP id 455A52930B for ; Wed, 25 Jul 2018 15:53:07 +0000 (UTC) Received: by mail.wl.linuxfoundation.org (Postfix, from userid 486) id 397F92A5AF; Wed, 25 Jul 2018 15:53:07 +0000 (UTC) X-Spam-Checker-Version: SpamAssassin 3.3.1 (2010-03-16) on pdx-wl-mail.web.codeaurora.org X-Spam-Level: X-Spam-Status: No, score=-3.0 required=2.0 tests=BAYES_00,DKIM_SIGNED, DKIM_VALID,DKIM_VALID_AU,FREEMAIL_FROM,MAILING_LIST_MULTI,RCVD_IN_DNSWL_NONE autolearn=ham version=3.3.1 Received: from kanga.kvack.org (kanga.kvack.org [205.233.56.17]) by mail.wl.linuxfoundation.org (Postfix) with ESMTP id D33882930B for ; Wed, 25 Jul 2018 15:53:06 +0000 (UTC) Received: by kanga.kvack.org (Postfix) id 64E666B0291; Wed, 25 Jul 2018 11:53:05 -0400 (EDT) Delivered-To: linux-mm-outgoing@kvack.org Received: by kanga.kvack.org (Postfix, from userid 40) id 5D95D6B0292; Wed, 25 Jul 2018 11:53:05 -0400 (EDT) X-Original-To: int-list-linux-mm@kvack.org X-Delivered-To: int-list-linux-mm@kvack.org Received: by kanga.kvack.org (Postfix, from userid 63042) id 458A66B0295; Wed, 25 Jul 2018 11:53:05 -0400 (EDT) X-Original-To: linux-mm@kvack.org X-Delivered-To: linux-mm@kvack.org Received: from mail-pl0-f69.google.com (mail-pl0-f69.google.com [209.85.160.69]) by kanga.kvack.org (Postfix) with ESMTP id E5B4B6B0291 for ; Wed, 25 Jul 2018 11:53:04 -0400 (EDT) Received: by mail-pl0-f69.google.com with SMTP id w18-v6so5675412plp.3 for ; Wed, 25 Jul 2018 08:53:04 -0700 (PDT) X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20161025; h=x-gm-message-state:dkim-signature:from:to:cc:subject:date :message-id:in-reply-to:references; bh=mHYcANdv1ziuRcHNi26L8UfTXr0LxJ2t3caZEIGgtE0=; b=uhXRpNIa31sIEQ98fk4/OggLLCqD5RYWnAdT+jJOiKJ//54UPsX5jBsgcHT9der1Rt 7hVrCH1+QOjSsDGKMWYgnP5tCEtg+GxakYoVUWRA5RPVoJ/Smtg0OtxdbsBPAAC9q0hx EspcK75/Tp7qgBvJA1GpEC8gU1NFo3csHP1rKetd++EUN3TMFaOOaCWqFGlBZBbsbVQj YPMOZwhDz3DhcXy6ttef3mZB6NI/cVU44leYYtRhPdlmgn0rhdMue1ijvNWV0BvnG4j6 namUiVqIehJtcWdrXFPowU63KbcTVSgym9LIUExx9yAp8sLoRe5/TUDG+i7hxqcTwoUI IJSw== X-Gm-Message-State: AOUpUlEl5K3m5+W7gNvJ/DPLJ41KuXpqmNAd+5fEtS13sY1uZ7/zMDpj 8GOrCq8F7+H/so0JVmcEZr1i7KQL4tQV7mDzs14UWohRYv+HVjGi7Hgc1fU6i9FMn9XXUMue89U y3c1l6PGvnE0cizdncijdLeXvOtqjrYQleYzYV8tP6NW7+uCnbs0fGq01NZb2UwaUhOuhED49K0 91Y7KkbfN/S6hvIRuOF8y7JgTFe5KtKVgu2RvQ0dP8wAQq5jVfEqnbrWkTRobLkeUEdAicuLuCj GHdWPS30VIGTQFR4GyvOEICFZhhBqWmgmSxvM9K4dDJqrT3a4ioSuwohJ6dJ5jwcS8A7wHO/+LF MDHKZyMpnWaTLvzAtDHUuYMKMPFvr7wAgvdi4eelwYtKdgE1XQADbcSta9E9LZaQ/gF8mZPbQAU r X-Received: by 2002:a17:902:8e81:: with SMTP id bg1-v6mr21490401plb.129.1532533984629; Wed, 25 Jul 2018 08:53:04 -0700 (PDT) X-Received: by 2002:a17:902:8e81:: with SMTP id bg1-v6mr21490371plb.129.1532533984007; Wed, 25 Jul 2018 08:53:04 -0700 (PDT) ARC-Seal: i=1; a=rsa-sha256; t=1532533983; cv=none; d=google.com; s=arc-20160816; b=IhrVKOOHjRpJ13J8WMj7Lg2+1ZM4PfDPDE0Jn5+x4iHicwvc7rUt1B4GJ1ZlYgkTn8 /udYdmGmwg+POMemNMw8XuMmOiLbspeGO/uYnDf3Auro09iJSCM0he3kSI96LCjnjtMM DZSxe7Cw9Y38vECfc9oIPVivRR5NC4Pf/G1XFRXD8fcY9ExzK07U0TpbKRzlzJF8gR9r Q1xjBzyWr91RJZ8O5gSIBOIV+IGgCujaglSIaHKR615ydi5UFbUbFc7pIsl4Q9YWibU8 6kqA3sFu4KAky10rPe/SGIlcHia5Hr0eGWZT3AJiRZs7FL15m/2cf4GEXgQ9lduPoEpr nYhg== ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=google.com; s=arc-20160816; h=references:in-reply-to:message-id:date:subject:cc:to:from :dkim-signature:arc-authentication-results; bh=mHYcANdv1ziuRcHNi26L8UfTXr0LxJ2t3caZEIGgtE0=; b=H4tEGLb7L6bmicGAj/p8KL9de22k/bk4Ozy9t9/T9kgvvQuGojVooJyvqFeJ1rRY+Q qhBH0GjrARoSDdahGwd7mfQoCvd8dYpSSKHTr2SqCiMamz1R46NjrhjLlRftZW5u8oG3 +T+fWGq/H5NczmE03g76V1ACBYbzZN1IiM66J+T/ONDKSCa8dFpblFi8tAUx1Pd27MzW KI+yRr2RWztgF4PPEhzqdTDO2kPa6ekMULdTp5BazZUciqp4h7/MlZxSxnT/Wu8MXkgW a9B1H4nrS7GP3Ri62DoDeWYy1x7GXJma7/XJ8aLGrFQ8h6vvMkpzcw3d/0qIwTauDBVd 44lw== ARC-Authentication-Results: i=1; mx.google.com; dkim=pass header.i=@gmail.com header.s=20161025 header.b=nok2v7d0; spf=pass (google.com: domain of npiggin@gmail.com designates 209.85.220.65 as permitted sender) smtp.mailfrom=npiggin@gmail.com; dmarc=pass (p=NONE sp=QUARANTINE dis=NONE) header.from=gmail.com Received: from mail-sor-f65.google.com (mail-sor-f65.google.com. [209.85.220.65]) by mx.google.com with SMTPS id t5-v6sor4879475plr.67.2018.07.25.08.53.03 for (Google Transport Security); Wed, 25 Jul 2018 08:53:03 -0700 (PDT) Received-SPF: pass (google.com: domain of npiggin@gmail.com designates 209.85.220.65 as permitted sender) client-ip=209.85.220.65; Authentication-Results: mx.google.com; dkim=pass header.i=@gmail.com header.s=20161025 header.b=nok2v7d0; spf=pass (google.com: domain of npiggin@gmail.com designates 209.85.220.65 as permitted sender) smtp.mailfrom=npiggin@gmail.com; dmarc=pass (p=NONE sp=QUARANTINE dis=NONE) header.from=gmail.com DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=20161025; h=from:to:cc:subject:date:message-id:in-reply-to:references; bh=mHYcANdv1ziuRcHNi26L8UfTXr0LxJ2t3caZEIGgtE0=; b=nok2v7d0fAP74uh787q3CuiuLXrypeSVLpy/NfSnhOySsVeS2hWOyX/qYUPlx9m5mZ 7kEW3feqvbiHQ1nXKfCapViJ3EiSDWj6i+nhNS0qp7EVVnGR9I+f2H6OzzKD5ACWHc9r v62Ddpz7Yb5B1pVjf37Rp9O3fRxeQJLqN0ri4KAWhll9XhpxC1IVW9JRCitR+987wi1P To2FkAL2BcD0jadrRR05srhoM6mJ6ASQhc3RJ+4LwpMfIO/9ssEdbt9wO2DoJ9u0fDrI SYTVOLAhGjUq4Kysgf7k9DnAYrk+/paw5tb0jET/ImyEgBUtP7iZKEfI3XlZByQ576jH +tLw== X-Google-Smtp-Source: AAOMgpcXSEuiq3wvBnnZlwnrdeXmCaHsgXgmpAUdxpDRwsg5Bh40RkW7jUK6WQZyxdwGWogvTJVoDQ== X-Received: by 2002:a17:902:a5cb:: with SMTP id t11-v6mr21491322plq.71.1532533983615; Wed, 25 Jul 2018 08:53:03 -0700 (PDT) Received: from roar.au.ibm.com ([61.69.188.107]) by smtp.gmail.com with ESMTPSA id j72-v6sm23438315pge.19.2018.07.25.08.53.01 (version=TLS1_2 cipher=ECDHE-RSA-AES128-GCM-SHA256 bits=128/128); Wed, 25 Jul 2018 08:53:03 -0700 (PDT) From: Nicholas Piggin To: linux-mm@kvack.org Cc: Nicholas Piggin , linux-arch@vger.kernel.org Subject: [RFC PATCH 3/4] mm: zap_pte_range optimise fullmm handling for dirty shared pages Date: Thu, 26 Jul 2018 01:52:45 +1000 Message-Id: <20180725155246.1085-4-npiggin@gmail.com> X-Mailer: git-send-email 2.17.0 In-Reply-To: <20180725155246.1085-1-npiggin@gmail.com> References: <20180725155246.1085-1-npiggin@gmail.com> X-Bogosity: Ham, tests=bogofilter, spamicity=0.000000, version=1.2.4 Sender: owner-linux-mm@kvack.org Precedence: bulk X-Loop: owner-majordomo@kvack.org List-ID: X-Virus-Scanned: ClamAV using ClamSMTP Shared dirty pages do not need to be flushed under page table lock for the fullmm case, because there will be no subsequent access through the TLBs. --- mm/memory.c | 14 ++++++++++++-- 1 file changed, 12 insertions(+), 2 deletions(-) diff --git a/mm/memory.c b/mm/memory.c index 1161ed3f1d0b..490689909186 100644 --- a/mm/memory.c +++ b/mm/memory.c @@ -1322,8 +1322,18 @@ static unsigned long zap_pte_range(struct mmu_gather *tlb, if (!PageAnon(page)) { if (pte_dirty(ptent)) { - force_flush = 1; - locked_flush = 1; + /* + * Page must be flushed from TLBs + * before releasing PTL to synchronize + * with page_mkclean and avoid another + * thread writing to the page through + * the old TLB after it was marked + * clean. + */ + if (!tlb->fullmm) { + force_flush = 1; + locked_flush = 1; + } set_page_dirty(page); } if (pte_young(ptent) && From patchwork Wed Jul 25 15:52:46 2018 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Nicholas Piggin X-Patchwork-Id: 10544249 Return-Path: Received: from mail.wl.linuxfoundation.org (pdx-wl-mail.web.codeaurora.org [172.30.200.125]) by pdx-korg-patchwork-2.web.codeaurora.org (Postfix) with ESMTP id 2C706139A for ; Wed, 25 Jul 2018 15:53:11 +0000 (UTC) Received: from mail.wl.linuxfoundation.org (localhost [127.0.0.1]) by mail.wl.linuxfoundation.org (Postfix) with ESMTP id 1B2B22A62C for ; Wed, 25 Jul 2018 15:53:11 +0000 (UTC) Received: by mail.wl.linuxfoundation.org (Postfix, from userid 486) id 0F7482A5D9; Wed, 25 Jul 2018 15:53:11 +0000 (UTC) X-Spam-Checker-Version: SpamAssassin 3.3.1 (2010-03-16) on pdx-wl-mail.web.codeaurora.org X-Spam-Level: X-Spam-Status: No, score=-3.0 required=2.0 tests=BAYES_00,DKIM_SIGNED, DKIM_VALID,DKIM_VALID_AU,FREEMAIL_FROM,MAILING_LIST_MULTI,RCVD_IN_DNSWL_NONE autolearn=ham version=3.3.1 Received: from kanga.kvack.org (kanga.kvack.org [205.233.56.17]) by mail.wl.linuxfoundation.org (Postfix) with ESMTP id 4644C28C1C for ; Wed, 25 Jul 2018 15:53:10 +0000 (UTC) Received: by kanga.kvack.org (Postfix) id 7F9446B0295; Wed, 25 Jul 2018 11:53:08 -0400 (EDT) Delivered-To: linux-mm-outgoing@kvack.org Received: by kanga.kvack.org (Postfix, from userid 40) id 784286B0296; Wed, 25 Jul 2018 11:53:08 -0400 (EDT) X-Original-To: int-list-linux-mm@kvack.org X-Delivered-To: int-list-linux-mm@kvack.org Received: by kanga.kvack.org (Postfix, from userid 63042) id 5B1CC6B029A; Wed, 25 Jul 2018 11:53:08 -0400 (EDT) X-Original-To: linux-mm@kvack.org X-Delivered-To: linux-mm@kvack.org Received: from mail-pg1-f198.google.com (mail-pg1-f198.google.com [209.85.215.198]) by kanga.kvack.org (Postfix) with ESMTP id 129FE6B0295 for ; Wed, 25 Jul 2018 11:53:08 -0400 (EDT) Received: by mail-pg1-f198.google.com with SMTP id e19-v6so5014599pgv.11 for ; Wed, 25 Jul 2018 08:53:08 -0700 (PDT) X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20161025; h=x-gm-message-state:dkim-signature:from:to:cc:subject:date :message-id:in-reply-to:references; bh=bnIhgs4ewuvZ6PC2wzBBOY45XHEVcX8KLV28U6C8Xdw=; b=DzNdytEyVB93tSq7DT5ugIOgtAaDw+fz5HQSQ5GQfit86usmdbTdUrDnomg1NZ0SZJ E7+RL4XkM48uB2hbVx0evuEVwrecvxop6JavB7Nfq3FojFKwCkl2D7lfbKjXRRPn70fR qVn47/NhyY/2lMzhVtAng3n50s4P4Hu1JckdblrHb2iI0UE81YKx2Ozf69eFSzGnSYSr 71Op5UFnRnDxed9oQh7QiR7+xAjE1uvVg6msvOIx6DsvcaRVv9wYsbm2CeabmNb+uKCi kaMCAVM79gv1Li1KW3Cq91jjKnq5UOLaQ33G1z5xtRCffGp037+WiXTKAzzTaJRFZLr1 oTAQ== X-Gm-Message-State: AOUpUlFdnR02G9utYf7UEp+OdtCkAJBj8f+EtJlN8oFl27/bmq596SqK SRBSXSbAQPkXfap0j6GpvFeavYs3UekKZTus2NS9q7tCvDTyw1subR0dU6TtF25AvzMMznnnfpy 0imxyz4wv8kgVRmsV6iQ4hpZgpUxDqKW5PT/o51YFHt63/nqi92HfIRAaXMFDKxnQXst8nI5BIY XI3Cds1mu+OdbXfoTw5sr3oZvE/eQZkzV9TiMasLrqoCNhqSKKzHOWmsL7z7R5fp9Zni6E/nBnl QixZbSrVJfFQLLFMSOI4WZX86VwXdrLa69qsbAeA5EqQWodKx07GcA0y1Y8QUxcu93IPaoEuga1 /QWiWmJ6kgs5uM6NBGU4k9q9b8/iwpV3Kh1HyksKlA65XH/oX462JG1LSlyfvy7AH6XIH53FOXu 0 X-Received: by 2002:a63:8449:: with SMTP id k70-v6mr21232993pgd.309.1532533987744; Wed, 25 Jul 2018 08:53:07 -0700 (PDT) X-Received: by 2002:a63:8449:: with SMTP id k70-v6mr21232955pgd.309.1532533986818; Wed, 25 Jul 2018 08:53:06 -0700 (PDT) ARC-Seal: i=1; a=rsa-sha256; t=1532533986; cv=none; d=google.com; s=arc-20160816; b=Y7SQoHluL1Cyc1MdPFLnEWrqsmHvgeZFp7ed3cYYLdOTMVIuEg6Eqx4K6uwPvgtrlC yRIqFSEjbxAFjY+Gc4Ce2pDQDG3E4WocPWQmJyVby/H3tmnXythbsWnIZBDMm57aUl4I tpC8IiRXSiqOlBozPh66OO8EW7xNNu/tMj4PWMV0YozleiuDurX37FJhpsdusB5osden cyv/L0uFeadTiEFazaanie9geae2ZuBhWpA29FM1Oeh6q8MtALYBV6Ajf4H2pn7PzRdS UPhr21hKoPIUshmgy52NstO58QvwJ+H0zEvWGRK4qAGIyfbcy6IF9MeqyM/K7v7bAqv4 Y6gw== ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=google.com; s=arc-20160816; h=references:in-reply-to:message-id:date:subject:cc:to:from :dkim-signature:arc-authentication-results; bh=bnIhgs4ewuvZ6PC2wzBBOY45XHEVcX8KLV28U6C8Xdw=; b=SZxvhlZ9/1nU/3zNITHnaoGxC/ZKgJziSftroDhFlLUQ4CjKgTIiOTNRB4Aok1nZZ9 nI6Cd/nNq909xP8c7lVjAEdJXXDiNBGc8Zl5+t+np+BBt65QGL1ocvL1maffwogYC9qz CPue+lDFF6hSSA9FIBccht1Qd4Z1iF5KwRS5wDQZu4WANmi7MD30kB8gonvYR6jM91rt vSIXfCWMoFus9V7zp49VGAKNDG0dlR7nFTygK1emV7VNsciE3DARLr5HD51R9H99WPmU 4pVhKyA1YihCLYb/BCtXzvzVe9wyjXj/KHcB1CyaxYaLqloBnVMzD6g18ENl8mAfdBKW iw6Q== ARC-Authentication-Results: i=1; mx.google.com; dkim=pass header.i=@gmail.com header.s=20161025 header.b=fIbI3ZkQ; spf=pass (google.com: domain of npiggin@gmail.com designates 209.85.220.65 as permitted sender) smtp.mailfrom=npiggin@gmail.com; dmarc=pass (p=NONE sp=QUARANTINE dis=NONE) header.from=gmail.com Received: from mail-sor-f65.google.com (mail-sor-f65.google.com. [209.85.220.65]) by mx.google.com with SMTPS id i13-v6sor4057961pgi.127.2018.07.25.08.53.06 for (Google Transport Security); Wed, 25 Jul 2018 08:53:06 -0700 (PDT) Received-SPF: pass (google.com: domain of npiggin@gmail.com designates 209.85.220.65 as permitted sender) client-ip=209.85.220.65; Authentication-Results: mx.google.com; dkim=pass header.i=@gmail.com header.s=20161025 header.b=fIbI3ZkQ; spf=pass (google.com: domain of npiggin@gmail.com designates 209.85.220.65 as permitted sender) smtp.mailfrom=npiggin@gmail.com; dmarc=pass (p=NONE sp=QUARANTINE dis=NONE) header.from=gmail.com DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=20161025; h=from:to:cc:subject:date:message-id:in-reply-to:references; bh=bnIhgs4ewuvZ6PC2wzBBOY45XHEVcX8KLV28U6C8Xdw=; b=fIbI3ZkQvC7teSReRGUJngCCNZLNRAJX+WKnES7UFEmGRqWvemxnctWyAvuPAbFdnq yHzup+UDM24mP94mz5Q3oD4vKXCduPJh+kT4chgBDubrSp9JA/uIn44miuGJfgypFlJT kBIHgW7AvVyS1o11IiUEtOLvgao9ZWASOjAcdwT6t3uaExkqEfLMbU4aApP+g00nHZaC fT2AdJBpmHa1MQp6OQJxwNtJU2BJMd/hZrcuDgqqL1/wXgSdQ7uoTJAwwNUb2k5fFBOT JCFrEHyXIdr3tIZUD/OASgG96j1XNhCJHJEODkN4XQNbZRPcuftcL+04VCdflCWddBL2 s72w== X-Google-Smtp-Source: AAOMgpdMcLV/nJw0jrLWzUg9SiqBKpLLKIhWFnhAzvz55fvNAPf+pYIQ6Fdcxm7YtQfQ9SxWN3ml6Q== X-Received: by 2002:a65:40cd:: with SMTP id u13-v6mr21456233pgp.334.1532533986356; Wed, 25 Jul 2018 08:53:06 -0700 (PDT) Received: from roar.au.ibm.com ([61.69.188.107]) by smtp.gmail.com with ESMTPSA id j72-v6sm23438315pge.19.2018.07.25.08.53.03 (version=TLS1_2 cipher=ECDHE-RSA-AES128-GCM-SHA256 bits=128/128); Wed, 25 Jul 2018 08:53:05 -0700 (PDT) From: Nicholas Piggin To: linux-mm@kvack.org Cc: Nicholas Piggin , linux-arch@vger.kernel.org Subject: [RFC PATCH 4/4] mm: optimise flushing and pte manipulation for single threaded access Date: Thu, 26 Jul 2018 01:52:46 +1000 Message-Id: <20180725155246.1085-5-npiggin@gmail.com> X-Mailer: git-send-email 2.17.0 In-Reply-To: <20180725155246.1085-1-npiggin@gmail.com> References: <20180725155246.1085-1-npiggin@gmail.com> X-Bogosity: Ham, tests=bogofilter, spamicity=0.000000, version=1.2.4 Sender: owner-linux-mm@kvack.org Precedence: bulk X-Loop: owner-majordomo@kvack.org List-ID: X-Virus-Scanned: ClamAV using ClamSMTP I think many tlb->fullmm users actually want to know if there could be concurrent memory accesses via the mappings being invalidated. If there is no risk of the page being modified after the pte is changed and if the pte dirty/young bits won't be modified by the hardware concurrently, there is no reason to defer page freeing or atomically update ptes. I haven't gone carefully through all archs, maybe there is some exception. But for core mm code I haven't been able to see why the single threaded case should be different than the fullmm case in terms of the pte updates and tlb flushing. --- include/asm-generic/tlb.h | 3 +++ mm/huge_memory.c | 4 ++-- mm/madvise.c | 4 ++-- mm/memory.c | 13 +++++++------ 4 files changed, 14 insertions(+), 10 deletions(-) diff --git a/include/asm-generic/tlb.h b/include/asm-generic/tlb.h index a55ef1425f0d..601c9fefda8e 100644 --- a/include/asm-generic/tlb.h +++ b/include/asm-generic/tlb.h @@ -100,6 +100,9 @@ struct mmu_gather { /* we are in the middle of an operation to clear * a full mm and can make some optimizations */ unsigned int fullmm : 1, + /* we have a single thread, current, which is active in the user + * address space */ + singlethread: 1, /* we have performed an operation which * requires a complete flush of the tlb */ need_flush_all : 1; diff --git a/mm/huge_memory.c b/mm/huge_memory.c index 1cd7c1a57a14..8e3958a811f0 100644 --- a/mm/huge_memory.c +++ b/mm/huge_memory.c @@ -1703,7 +1703,7 @@ int zap_huge_pmd(struct mmu_gather *tlb, struct vm_area_struct *vma, * operations. */ orig_pmd = pmdp_huge_get_and_clear_full(tlb->mm, addr, pmd, - tlb->fullmm); + tlb->singlethread); tlb_remove_pmd_tlb_entry(tlb, pmd, addr); if (vma_is_dax(vma)) { if (arch_needs_pgtable_deposit()) @@ -1971,7 +1971,7 @@ int zap_huge_pud(struct mmu_gather *tlb, struct vm_area_struct *vma, * operations. */ orig_pud = pudp_huge_get_and_clear_full(tlb->mm, addr, pud, - tlb->fullmm); + tlb->singlethread); tlb_remove_pud_tlb_entry(tlb, pud, addr); if (vma_is_dax(vma)) { spin_unlock(ptl); diff --git a/mm/madvise.c b/mm/madvise.c index 4d3c922ea1a1..d9e1f3ac8067 100644 --- a/mm/madvise.c +++ b/mm/madvise.c @@ -350,7 +350,7 @@ static int madvise_free_pte_range(pmd_t *pmd, unsigned long addr, continue; nr_swap--; free_swap_and_cache(entry); - pte_clear_not_present_full(mm, addr, pte, tlb->fullmm); + pte_clear_not_present_full(mm, addr, pte, tlb->singlethread); continue; } @@ -417,7 +417,7 @@ static int madvise_free_pte_range(pmd_t *pmd, unsigned long addr, * after pte clearing. */ ptent = ptep_get_and_clear_full(mm, addr, pte, - tlb->fullmm); + tlb->singlethread); ptent = pte_mkold(ptent); ptent = pte_mkclean(ptent); diff --git a/mm/memory.c b/mm/memory.c index 490689909186..471ba07bc7e3 100644 --- a/mm/memory.c +++ b/mm/memory.c @@ -221,8 +221,9 @@ void arch_tlb_gather_mmu(struct mmu_gather *tlb, struct mm_struct *mm, { tlb->mm = mm; - /* Is it from 0 to ~0? */ tlb->fullmm = !(start | (end+1)); + tlb->singlethread = (mm == current->mm) && + (atomic_read(&mm->mm_users) <= 1); tlb->need_flush_all = 0; tlb->local.next = NULL; tlb->local.nr = 0; @@ -300,7 +301,7 @@ bool __tlb_remove_page_size(struct mmu_gather *tlb, struct page *page, int page_ * When this is our mm and there are no other users, there can not be * a concurrent memory access. */ - if (current->mm == tlb->mm && atomic_read(&tlb->mm->mm_users) < 2) { + if (tlb->singlethread) { free_page_and_swap_cache(page); return false; } @@ -1315,7 +1316,7 @@ static unsigned long zap_pte_range(struct mmu_gather *tlb, continue; } ptent = ptep_get_and_clear_full(mm, addr, pte, - tlb->fullmm); + tlb->singlethread); tlb_remove_tlb_entry(tlb, pte, addr); if (unlikely(!page)) continue; @@ -1330,7 +1331,7 @@ static unsigned long zap_pte_range(struct mmu_gather *tlb, * the old TLB after it was marked * clean. */ - if (!tlb->fullmm) { + if (!tlb->singlethread) { force_flush = 1; locked_flush = 1; } @@ -1367,7 +1368,7 @@ static unsigned long zap_pte_range(struct mmu_gather *tlb, continue; } - pte_clear_not_present_full(mm, addr, pte, tlb->fullmm); + pte_clear_not_present_full(mm, addr, pte, tlb->singlethread); rss[mm_counter(page)]--; page_remove_rmap(page, false); put_page(page); @@ -1389,7 +1390,7 @@ static unsigned long zap_pte_range(struct mmu_gather *tlb, } if (unlikely(!free_swap_and_cache(entry))) print_bad_pte(vma, addr, ptent, NULL); - pte_clear_not_present_full(mm, addr, pte, tlb->fullmm); + pte_clear_not_present_full(mm, addr, pte, tlb->singlethread); } while (pte++, addr += PAGE_SIZE, addr != end); add_mm_rss_vec(mm, rss);