From patchwork Wed Apr 24 11:12:05 2019 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Sebastian Andrzej Siewior X-Patchwork-Id: 10914563 Return-Path: Received: from mail.wl.linuxfoundation.org (pdx-wl-mail.web.codeaurora.org [172.30.200.125]) by pdx-korg-patchwork-2.web.codeaurora.org (Postfix) with ESMTP id C78FB1515 for ; Wed, 24 Apr 2019 11:12:31 +0000 (UTC) Received: from mail.wl.linuxfoundation.org (localhost [127.0.0.1]) by mail.wl.linuxfoundation.org (Postfix) with ESMTP id B608028A97 for ; Wed, 24 Apr 2019 11:12:31 +0000 (UTC) Received: by mail.wl.linuxfoundation.org (Postfix, from userid 486) id B415628A9C; Wed, 24 Apr 2019 11:12:31 +0000 (UTC) X-Spam-Checker-Version: SpamAssassin 3.3.1 (2010-03-16) on pdx-wl-mail.web.codeaurora.org X-Spam-Level: X-Spam-Status: No, score=-2.9 required=2.0 tests=BAYES_00,MAILING_LIST_MULTI, RCVD_IN_DNSWL_NONE autolearn=ham version=3.3.1 Received: from kanga.kvack.org (kanga.kvack.org [205.233.56.17]) by mail.wl.linuxfoundation.org (Postfix) with ESMTP id 54B8428A50 for ; Wed, 24 Apr 2019 11:12:31 +0000 (UTC) Received: by kanga.kvack.org (Postfix) id 76B9A6B0006; Wed, 24 Apr 2019 07:12:30 -0400 (EDT) Delivered-To: linux-mm-outgoing@kvack.org Received: by kanga.kvack.org (Postfix, from userid 40) id 6F4CA6B0007; Wed, 24 Apr 2019 07:12:30 -0400 (EDT) X-Original-To: int-list-linux-mm@kvack.org X-Delivered-To: int-list-linux-mm@kvack.org Received: by kanga.kvack.org (Postfix, from userid 63042) id 597296B0008; Wed, 24 Apr 2019 07:12:30 -0400 (EDT) X-Original-To: linux-mm@kvack.org X-Delivered-To: linux-mm@kvack.org Received: from mail-wm1-f69.google.com (mail-wm1-f69.google.com [209.85.128.69]) by kanga.kvack.org (Postfix) with ESMTP id 0ABC36B0006 for ; Wed, 24 Apr 2019 07:12:30 -0400 (EDT) Received: by mail-wm1-f69.google.com with SMTP id q3so2651467wmc.0 for ; Wed, 24 Apr 2019 04:12:29 -0700 (PDT) X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20161025; h=x-original-authentication-results:x-gm-message-state:from:to:cc :subject:date:message-id:in-reply-to:references:mime-version :content-transfer-encoding; bh=g7i4uaKwh7TnZU3pyi6vSF6/UGnHMRj8eJfZ5arzTEI=; b=MSBI8a1c5WEhpTuNtIpAu2r/ZyMWa2p8agtVior7yY1KkHQlpzS7TUia52+fcWSmOT RjW++2xkKDVUOESUjgIACXLlkh2XJ+3pHzoN/Sv0t3W9RSCblYkkzKh3i3+664AG8O4V xzMtWPZAVOP8UceFB95TI/pCFZNaYcyu/gXIcVrERE3b8g1ZjGs4R6T8MpELlO9CGA5s ozi/6LUZMCP55KRy9rdmIadYvOMyIUeBVkttVtvJ4vXTz7pPp1lgtgk49VUcOBYQX90A QWr7eD8R5GsLZEMBE9yPAwr44WuniqcfleuhrDD7X0T7xS/DOSD2pVMOO+y3WKPWbzEU jguQ== X-Original-Authentication-Results: mx.google.com; spf=pass (google.com: best guess record for domain of bigeasy@linutronix.de designates 2a01:7a0:2:106d:700::1 as permitted sender) smtp.mailfrom=bigeasy@linutronix.de X-Gm-Message-State: APjAAAWKU87ZGp23z27duCDYV9FM3SqcyTZOQnNQG1gCDNqURPvFUKqS Ldt8l9kAvPiNK/GnfFldDuNon09ybnF/ZRqJJ7bQ+y5Wg431fMa2dZftdniCcMNa5j/9wJtj2Ol diVw5uKMRWrn+UmkYNi5yuZca9OSHsoBQ4tFJbPrShBet3lfD/4GbmgKoGEQnB7qDQA== X-Received: by 2002:a1c:2109:: with SMTP id h9mr6261485wmh.68.1556104349525; Wed, 24 Apr 2019 04:12:29 -0700 (PDT) X-Google-Smtp-Source: APXvYqz64p22WvWpH8sXCGiH1lplwOGvTtx9IG0B6E6vs/gPXeWFryeyx/0IyWTma1Dz1RBYg6Xw X-Received: by 2002:a1c:2109:: with SMTP id h9mr6261421wmh.68.1556104348360; Wed, 24 Apr 2019 04:12:28 -0700 (PDT) ARC-Seal: i=1; a=rsa-sha256; t=1556104348; cv=none; d=google.com; s=arc-20160816; b=MmZXtrtS4a6nXP7LOe89AESoaGy7Lmz8xCdzRwuPBX2ZKOImqjqiqpC9UvJ6uT4bdr 3pSqkfJ2IxccR/qzw78rbvTycVNQdK7Wr5saLx9Z6AsP3KGD8Mya8u7Ws6uGMvNJtRH5 DZO6QVNx5yELtS6+K0GBaa7L9hHyubILlJ4Znuvfr7R1NkAp5WvpgtnTnQ0KtiJ9LCvy OwCQVBwYwZkKHRAggGxwCY4qe8rKfu4N7aFeApM9BtD/lH3xmK9QpIP0CsBkWA9WGCUg RaBNlCud6xKZQ4xtCnVTcTJLZzHS0c+VCIKZXs3oUqkh8lsOSqcmFE9Ioq8DtKCuKkfA NbbA== ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=google.com; s=arc-20160816; h=content-transfer-encoding:mime-version:references:in-reply-to :message-id:date:subject:cc:to:from; bh=g7i4uaKwh7TnZU3pyi6vSF6/UGnHMRj8eJfZ5arzTEI=; b=i46jBDoVakDJ1ZO+LfpIzsum3gZXayacX0RCKTus7XRwy1aXNOSwImVF6JZP9LtDyh oh3wKyKFZ1py8R9iI4mwmem4Z0oyealQgWXNQYzOx+xLba3FJerej8goZ2PHtdHep1Pv ff7O1GPTuqGXdP6MikKEUQGLfrrPbJvygmeJ0IdPGSFY2MtXW07FtkY8fMUucYHUncBA o3E/Y1gTHqNo+v98iv+Ai1bIqgFAlZd6xFTdUW7I+tf5Msb4oVbW1rmdk0U6PwdRm/oT GE5JRerNE+M2je8UDj92LnCKxHPDd2f9h0eJiS9PpPWMmG7+3f24nfRt17/p6ULG+o9o vq2w== ARC-Authentication-Results: i=1; mx.google.com; spf=pass (google.com: best guess record for domain of bigeasy@linutronix.de designates 2a01:7a0:2:106d:700::1 as permitted sender) smtp.mailfrom=bigeasy@linutronix.de Received: from Galois.linutronix.de (Galois.linutronix.de. [2a01:7a0:2:106d:700::1]) by mx.google.com with ESMTPS id r12si3037758wrx.173.2019.04.24.04.12.28 for (version=TLS1_2 cipher=AES128-SHA bits=128/128); Wed, 24 Apr 2019 04:12:28 -0700 (PDT) Received-SPF: pass (google.com: best guess record for domain of bigeasy@linutronix.de designates 2a01:7a0:2:106d:700::1 as permitted sender) client-ip=2a01:7a0:2:106d:700::1; Authentication-Results: mx.google.com; spf=pass (google.com: best guess record for domain of bigeasy@linutronix.de designates 2a01:7a0:2:106d:700::1 as permitted sender) smtp.mailfrom=bigeasy@linutronix.de Received: from localhost ([127.0.0.1] helo=flow.W.breakpoint.cc) by Galois.linutronix.de with esmtp (Exim 4.80) (envelope-from ) id 1hJFpC-0006KY-E8; Wed, 24 Apr 2019 13:12:26 +0200 From: Sebastian Andrzej Siewior To: linux-mm@kvack.org Cc: tglx@linutronix.de, frederic@kernel.org, Christoph Lameter , anna-maria@linutronix.de, Sebastian Andrzej Siewior Subject: [PATCH 1/4] mm/page_alloc: Split drain_local_pages() Date: Wed, 24 Apr 2019 13:12:05 +0200 Message-Id: <20190424111208.24459-2-bigeasy@linutronix.de> X-Mailer: git-send-email 2.20.1 In-Reply-To: <20190424111208.24459-1-bigeasy@linutronix.de> References: <20190424111208.24459-1-bigeasy@linutronix.de> MIME-Version: 1.0 X-Bogosity: Ham, tests=bogofilter, spamicity=0.000000, version=1.2.4 Sender: owner-linux-mm@kvack.org Precedence: bulk X-Loop: owner-majordomo@kvack.org List-ID: X-Virus-Scanned: ClamAV using ClamSMTP From: Anna-Maria Gleixner Splitting the functionality of drain_local_pages() into a separate function. This is a preparatory work for introducing the static key dependend locking mechanism. No functional change. Signed-off-by: Anna-Maria Gleixner Signed-off-by: Sebastian Andrzej Siewior --- include/linux/gfp.h | 1 + mm/page_alloc.c | 13 +++++++++---- 2 files changed, 10 insertions(+), 4 deletions(-) diff --git a/include/linux/gfp.h b/include/linux/gfp.h index fdab7de7490df..fcad3a07c9b04 100644 --- a/include/linux/gfp.h +++ b/include/linux/gfp.h @@ -557,6 +557,7 @@ extern void page_frag_free(void *addr); void page_alloc_init(void); void drain_zone_pages(struct zone *zone, struct per_cpu_pages *pcp); void drain_all_pages(struct zone *zone); +void drain_cpu_pages(unsigned int cpu, struct zone *zone); void drain_local_pages(struct zone *zone); void page_alloc_init_late(void); diff --git a/mm/page_alloc.c b/mm/page_alloc.c index c6ce20aaf80bb..cf120e0700035 100644 --- a/mm/page_alloc.c +++ b/mm/page_alloc.c @@ -2713,6 +2713,14 @@ static void drain_pages(unsigned int cpu) } } +void drain_cpu_pages(unsigned int cpu, struct zone *zone) +{ + if (zone) + drain_pages_zone(cpu, zone); + else + drain_pages(cpu); +} + /* * Spill all of this CPU's per-cpu pages back into the buddy allocator. * @@ -2723,10 +2731,7 @@ void drain_local_pages(struct zone *zone) { int cpu = smp_processor_id(); - if (zone) - drain_pages_zone(cpu, zone); - else - drain_pages(cpu); + drain_cpu_pages(cpu, zone); } static void drain_local_pages_wq(struct work_struct *work) From patchwork Wed Apr 24 11:12:06 2019 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Sebastian Andrzej Siewior X-Patchwork-Id: 10914565 Return-Path: Received: from mail.wl.linuxfoundation.org (pdx-wl-mail.web.codeaurora.org [172.30.200.125]) by pdx-korg-patchwork-2.web.codeaurora.org (Postfix) with ESMTP id 61E4F1708 for ; Wed, 24 Apr 2019 11:12:38 +0000 (UTC) Received: from mail.wl.linuxfoundation.org (localhost [127.0.0.1]) by mail.wl.linuxfoundation.org (Postfix) with ESMTP id 5011828869 for ; Wed, 24 Apr 2019 11:12:38 +0000 (UTC) Received: by mail.wl.linuxfoundation.org (Postfix, from userid 486) id 440BE289B1; Wed, 24 Apr 2019 11:12:38 +0000 (UTC) X-Spam-Checker-Version: SpamAssassin 3.3.1 (2010-03-16) on pdx-wl-mail.web.codeaurora.org X-Spam-Level: X-Spam-Status: No, score=-2.9 required=2.0 tests=BAYES_00,MAILING_LIST_MULTI, RCVD_IN_DNSWL_NONE autolearn=ham version=3.3.1 Received: from kanga.kvack.org (kanga.kvack.org [205.233.56.17]) by mail.wl.linuxfoundation.org (Postfix) with ESMTP id 50A4E28869 for ; Wed, 24 Apr 2019 11:12:37 +0000 (UTC) Received: by kanga.kvack.org (Postfix) id 4D4506B0007; Wed, 24 Apr 2019 07:12:36 -0400 (EDT) Delivered-To: linux-mm-outgoing@kvack.org Received: by kanga.kvack.org (Postfix, from userid 40) id 45E666B0008; Wed, 24 Apr 2019 07:12:36 -0400 (EDT) X-Original-To: int-list-linux-mm@kvack.org X-Delivered-To: int-list-linux-mm@kvack.org Received: by kanga.kvack.org (Postfix, from userid 63042) id 28B366B000A; Wed, 24 Apr 2019 07:12:36 -0400 (EDT) X-Original-To: linux-mm@kvack.org X-Delivered-To: linux-mm@kvack.org Received: from mail-wm1-f70.google.com (mail-wm1-f70.google.com [209.85.128.70]) by kanga.kvack.org (Postfix) with ESMTP id C9B2B6B0007 for ; Wed, 24 Apr 2019 07:12:35 -0400 (EDT) Received: by mail-wm1-f70.google.com with SMTP id n11so2640776wmh.2 for ; Wed, 24 Apr 2019 04:12:35 -0700 (PDT) X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20161025; h=x-original-authentication-results:x-gm-message-state:from:to:cc :subject:date:message-id:in-reply-to:references:mime-version :content-transfer-encoding; bh=WMpspZdiTfo3o0Ccu0Zw52cfcU4CJMh6yc6VE/EvfkE=; b=YmH4/UeRbZdpGrRkYmoJjW8K1aKGzts5xWacJVHb9rdFwjjoE4OSluk448oSBO7OG3 kXQOa99NXl72gxe3vEoab0Y1pdmenjvXPvpMv23OgJ4d5Zy/35t4KMZC6cEsKo4ZDusj 6pMef0tVxTMO7ERxS0+QyfJ4YVdtuHVa0p0fqsWMwEWO4yJm+XEKffvXF3K69RNiZVwe zt/6g2vZV8EYhI9QCPaIt5ANclet/ANKwpbqHFn5diFZUpe3NKmTEnTz09EIWo9x2g/c 0Raqs5+l/W/CuqBrWCrxW5fy9pkC+k0m3hs9GRoq2nQGh6OA63/8TGQ70FUUEYYJm3ki /1bg== X-Original-Authentication-Results: mx.google.com; spf=pass (google.com: best guess record for domain of bigeasy@linutronix.de designates 2a01:7a0:2:106d:700::1 as permitted sender) smtp.mailfrom=bigeasy@linutronix.de X-Gm-Message-State: APjAAAWflboGGt71RgSTS3rXuCJvP8yjBdnhppcpRFwy5U3agro9h5yB iNGQmdU2FQEBJHEexYEV8y1VpRgaaW105qPG3qvHMuU/LmPjMcg679Yqqc3K85snYc5Vvitxf5H DLmuxKQTr92FMyZVFqXvYIpybxaDGjLMqKjBMoord274OzhdEFnvrf1y3+RIjHPCcQQ== X-Received: by 2002:adf:f68c:: with SMTP id v12mr4568985wrp.40.1556104355331; Wed, 24 Apr 2019 04:12:35 -0700 (PDT) X-Google-Smtp-Source: APXvYqxXnS5Ioke8EcZpblCbCG+YutpZWNd+CLct/h+Zeaiv8xDmK2xLhuElIswPeygjtoD7PAcf X-Received: by 2002:adf:f68c:: with SMTP id v12mr4568915wrp.40.1556104354145; Wed, 24 Apr 2019 04:12:34 -0700 (PDT) ARC-Seal: i=1; a=rsa-sha256; t=1556104354; cv=none; d=google.com; s=arc-20160816; b=SI3wBn+U0XESqtnyxaocR036hfHGJsxT+bWm4pkfY1LT0LA4Se89je69A4Gonei2GH E1NSBFRUKPewI9t5fWuywfVxT945TZQiwQ0KUHV/QTbimg228+/7cjTFIaQn+rIfXOin CwpBzxHMgIEZzbIMDM23oFu5iP5COK95JaJDkkyLMNWadwzmFeyZbTBimR1H41+9A5IM tKc9hV6/1L49PtqDN79b8eBbNQ58rqF1S8RI5t0VusfOlw6H+7LgP7dzef1VGpjkTxep sXJLWdA6CcfMWhLNtlQ0NnoCCMEGKopuiAw79aKaF7tX2zLJY/Tp5bE7j+eO+2cKED6O mz0A== ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=google.com; s=arc-20160816; h=content-transfer-encoding:mime-version:references:in-reply-to :message-id:date:subject:cc:to:from; bh=WMpspZdiTfo3o0Ccu0Zw52cfcU4CJMh6yc6VE/EvfkE=; b=YlljcXZZlquTbTzUweQOqmX378lTxc3ULeHWGqvqp2fZAqMXo0iK7eDIRkGm5avbAU FdZB/BzsaPV79UfewH6LwGNB0AjuTy9igcqa8UFXIh5ROm3wt+YIXEJyDAarhpyRVAPR wc2g6C75V5ordeaDljOUu3ljAQMd957eBLJVB2ElHovLjOX8PEnU5yqa1Tvne7+kbbJ+ ui1/or6hAIIt+VFCRcvVcCs2CU//28QXchlcUV6xZPelBo6l1ze/iQEQ8Vdwo1WNaWmI N2ZlIXEiFAEeE/kz/6lDCZn2pEgtv0vaO17KAc45k+lz2x+QGl3Sj1tWsRIyHeMoKEkf BUNA== ARC-Authentication-Results: i=1; mx.google.com; spf=pass (google.com: best guess record for domain of bigeasy@linutronix.de designates 2a01:7a0:2:106d:700::1 as permitted sender) smtp.mailfrom=bigeasy@linutronix.de Received: from Galois.linutronix.de (Galois.linutronix.de. [2a01:7a0:2:106d:700::1]) by mx.google.com with ESMTPS id e9si13497081wrr.339.2019.04.24.04.12.33 for (version=TLS1_2 cipher=AES128-SHA bits=128/128); Wed, 24 Apr 2019 04:12:34 -0700 (PDT) Received-SPF: pass (google.com: best guess record for domain of bigeasy@linutronix.de designates 2a01:7a0:2:106d:700::1 as permitted sender) client-ip=2a01:7a0:2:106d:700::1; Authentication-Results: mx.google.com; spf=pass (google.com: best guess record for domain of bigeasy@linutronix.de designates 2a01:7a0:2:106d:700::1 as permitted sender) smtp.mailfrom=bigeasy@linutronix.de Received: from localhost ([127.0.0.1] helo=flow.W.breakpoint.cc) by Galois.linutronix.de with esmtp (Exim 4.80) (envelope-from ) id 1hJFpI-0006KY-BW; Wed, 24 Apr 2019 13:12:32 +0200 From: Sebastian Andrzej Siewior To: linux-mm@kvack.org Cc: tglx@linutronix.de, frederic@kernel.org, Christoph Lameter , anna-maria@linutronix.de, Sebastian Andrzej Siewior Subject: [PATCH 2/4] mm/swap: Add static key dependent pagevec locking Date: Wed, 24 Apr 2019 13:12:06 +0200 Message-Id: <20190424111208.24459-3-bigeasy@linutronix.de> X-Mailer: git-send-email 2.20.1 In-Reply-To: <20190424111208.24459-1-bigeasy@linutronix.de> References: <20190424111208.24459-1-bigeasy@linutronix.de> MIME-Version: 1.0 X-Bogosity: Ham, tests=bogofilter, spamicity=0.000000, version=1.2.4 Sender: owner-linux-mm@kvack.org Precedence: bulk X-Loop: owner-majordomo@kvack.org List-ID: X-Virus-Scanned: ClamAV using ClamSMTP From: Thomas Gleixner The locking of struct pagevec is done by disabling preemption. In case the struct has be accessed form interrupt context then interrupts are disabled. This means the struct can only be accessed locally from the CPU. There is also no lockdep coverage which would scream during if it accessed from wrong context. Create struct swap_pagevec which contains of a pagevec member and a spin_lock_t. Introduce a static key, which changes the locking behavior only if the key is set in the following way: Before the struct is accessed the spin_lock has to be acquired instead of using preempt_disable(). Since the struct is used CPU-locally there is no spinning on the lock but the lock is acquired immediately. If the struct is accessed from interrupt context, spin_lock_irqsave() is used. No functional change yet because static key is not enabled. [anna-maria: introduce static key] Signed-off-by: Thomas Gleixner Signed-off-by: Anna-Maria Gleixner Signed-off-by: Sebastian Andrzej Siewior --- mm/compaction.c | 14 ++-- mm/internal.h | 2 + mm/swap.c | 186 +++++++++++++++++++++++++++++++++++++++--------- 3 files changed, 165 insertions(+), 37 deletions(-) diff --git a/mm/compaction.c b/mm/compaction.c index 3319e0872d014..ec47c96186771 100644 --- a/mm/compaction.c +++ b/mm/compaction.c @@ -2224,10 +2224,16 @@ compact_zone(struct compact_control *cc, struct capture_control *capc) block_start_pfn(cc->migrate_pfn, cc->order); if (last_migrated_pfn < current_block_start) { - cpu = get_cpu(); - lru_add_drain_cpu(cpu); - drain_local_pages(cc->zone); - put_cpu(); + if (static_branch_unlikely(&use_pvec_lock)) { + cpu = get_cpu(); + lru_add_drain_cpu(cpu); + drain_local_pages(cc->zone); + put_cpu(); + } else { + cpu = raw_smp_processor_id(); + lru_add_drain_cpu(cpu); + drain_cpu_pages(cpu, cc->zone); + } /* No more flushing until we migrate again */ last_migrated_pfn = 0; } diff --git a/mm/internal.h b/mm/internal.h index 9eeaf2b95166f..ddfa760e61652 100644 --- a/mm/internal.h +++ b/mm/internal.h @@ -36,6 +36,8 @@ /* Do not use these with a slab allocator */ #define GFP_SLAB_BUG_MASK (__GFP_DMA32|__GFP_HIGHMEM|~__GFP_BITS_MASK) +extern struct static_key_false use_pvec_lock; + void page_writeback_init(void); vm_fault_t do_swap_page(struct vm_fault *vmf); diff --git a/mm/swap.c b/mm/swap.c index 301ed4e043205..136c80480dbde 100644 --- a/mm/swap.c +++ b/mm/swap.c @@ -43,14 +43,107 @@ /* How many pages do we try to swap or page in/out together? */ int page_cluster; -static DEFINE_PER_CPU(struct pagevec, lru_add_pvec); -static DEFINE_PER_CPU(struct pagevec, lru_rotate_pvecs); -static DEFINE_PER_CPU(struct pagevec, lru_deactivate_file_pvecs); -static DEFINE_PER_CPU(struct pagevec, lru_lazyfree_pvecs); +DEFINE_STATIC_KEY_FALSE(use_pvec_lock); + +struct swap_pagevec { + spinlock_t lock; + struct pagevec pvec; +}; + +#define DEFINE_PER_CPU_PAGEVEC(lvar) \ + DEFINE_PER_CPU(struct swap_pagevec, lvar) = { \ + .lock = __SPIN_LOCK_UNLOCKED((lvar).lock) } + +static DEFINE_PER_CPU_PAGEVEC(lru_add_pvec); +static DEFINE_PER_CPU_PAGEVEC(lru_rotate_pvecs); +static DEFINE_PER_CPU_PAGEVEC(lru_deactivate_file_pvecs); +static DEFINE_PER_CPU_PAGEVEC(lru_lazyfree_pvecs); #ifdef CONFIG_SMP -static DEFINE_PER_CPU(struct pagevec, activate_page_pvecs); +static DEFINE_PER_CPU_PAGEVEC(activate_page_pvecs); #endif +static inline +struct swap_pagevec *lock_swap_pvec(struct swap_pagevec __percpu *p) +{ + struct swap_pagevec *swpvec; + + if (static_branch_likely(&use_pvec_lock)) { + swpvec = raw_cpu_ptr(p); + + spin_lock(&swpvec->lock); + } else { + swpvec = &get_cpu_var(*p); + } + return swpvec; +} + +static inline struct swap_pagevec * +lock_swap_pvec_cpu(struct swap_pagevec __percpu *p, int cpu) +{ + struct swap_pagevec *swpvec = per_cpu_ptr(p, cpu); + + if (static_branch_likely(&use_pvec_lock)) + spin_lock(&swpvec->lock); + + return swpvec; +} + +static inline struct swap_pagevec * +lock_swap_pvec_irqsave(struct swap_pagevec __percpu *p, unsigned long *flags) +{ + struct swap_pagevec *swpvec; + + if (static_branch_likely(&use_pvec_lock)) { + swpvec = raw_cpu_ptr(p); + + spin_lock_irqsave(&swpvec->lock, (*flags)); + } else { + local_irq_save(*flags); + + swpvec = this_cpu_ptr(p); + } + return swpvec; +} + +static inline struct swap_pagevec * +lock_swap_pvec_cpu_irqsave(struct swap_pagevec __percpu *p, int cpu, + unsigned long *flags) +{ + struct swap_pagevec *swpvec = per_cpu_ptr(p, cpu); + + if (static_branch_likely(&use_pvec_lock)) + spin_lock_irqsave(&swpvec->lock, *flags); + else + local_irq_save(*flags); + + return swpvec; +} + +static inline void unlock_swap_pvec(struct swap_pagevec *swpvec, + struct swap_pagevec __percpu *p) +{ + if (static_branch_likely(&use_pvec_lock)) + spin_unlock(&swpvec->lock); + else + put_cpu_var(*p); + +} + +static inline void unlock_swap_pvec_cpu(struct swap_pagevec *swpvec) +{ + if (static_branch_likely(&use_pvec_lock)) + spin_unlock(&swpvec->lock); +} + +static inline void +unlock_swap_pvec_irqrestore(struct swap_pagevec *swpvec, unsigned long flags) +{ + if (static_branch_likely(&use_pvec_lock)) + spin_unlock_irqrestore(&swpvec->lock, flags); + else + local_irq_restore(flags); +} + /* * This path almost never happens for VM activity - pages are normally * freed via pagevecs. But it gets used by networking. @@ -248,15 +341,17 @@ void rotate_reclaimable_page(struct page *page) { if (!PageLocked(page) && !PageDirty(page) && !PageUnevictable(page) && PageLRU(page)) { + struct swap_pagevec *swpvec; struct pagevec *pvec; unsigned long flags; get_page(page); - local_irq_save(flags); - pvec = this_cpu_ptr(&lru_rotate_pvecs); + + swpvec = lock_swap_pvec_irqsave(&lru_rotate_pvecs, &flags); + pvec = &swpvec->pvec; if (!pagevec_add(pvec, page) || PageCompound(page)) pagevec_move_tail(pvec); - local_irq_restore(flags); + unlock_swap_pvec_irqrestore(swpvec, flags); } } @@ -291,27 +386,32 @@ static void __activate_page(struct page *page, struct lruvec *lruvec, #ifdef CONFIG_SMP static void activate_page_drain(int cpu) { - struct pagevec *pvec = &per_cpu(activate_page_pvecs, cpu); + struct swap_pagevec *swpvec = lock_swap_pvec_cpu(&activate_page_pvecs, cpu); + struct pagevec *pvec = &swpvec->pvec; if (pagevec_count(pvec)) pagevec_lru_move_fn(pvec, __activate_page, NULL); + unlock_swap_pvec_cpu(swpvec); } static bool need_activate_page_drain(int cpu) { - return pagevec_count(&per_cpu(activate_page_pvecs, cpu)) != 0; + return pagevec_count(per_cpu_ptr(&activate_page_pvecs.pvec, cpu)) != 0; } void activate_page(struct page *page) { page = compound_head(page); if (PageLRU(page) && !PageActive(page) && !PageUnevictable(page)) { - struct pagevec *pvec = &get_cpu_var(activate_page_pvecs); + struct swap_pagevec *swpvec; + struct pagevec *pvec; get_page(page); + swpvec = lock_swap_pvec(&activate_page_pvecs); + pvec = &swpvec->pvec; if (!pagevec_add(pvec, page) || PageCompound(page)) pagevec_lru_move_fn(pvec, __activate_page, NULL); - put_cpu_var(activate_page_pvecs); + unlock_swap_pvec(swpvec, &activate_page_pvecs); } } @@ -333,7 +433,8 @@ void activate_page(struct page *page) static void __lru_cache_activate_page(struct page *page) { - struct pagevec *pvec = &get_cpu_var(lru_add_pvec); + struct swap_pagevec *swpvec = lock_swap_pvec(&lru_add_pvec); + struct pagevec *pvec = &swpvec->pvec; int i; /* @@ -355,7 +456,7 @@ static void __lru_cache_activate_page(struct page *page) } } - put_cpu_var(lru_add_pvec); + unlock_swap_pvec(swpvec, &lru_add_pvec); } /* @@ -397,12 +498,13 @@ EXPORT_SYMBOL(mark_page_accessed); static void __lru_cache_add(struct page *page) { - struct pagevec *pvec = &get_cpu_var(lru_add_pvec); + struct swap_pagevec *swpvec = lock_swap_pvec(&lru_add_pvec); + struct pagevec *pvec = &swpvec->pvec; get_page(page); if (!pagevec_add(pvec, page) || PageCompound(page)) __pagevec_lru_add(pvec); - put_cpu_var(lru_add_pvec); + unlock_swap_pvec(swpvec, &lru_add_pvec); } /** @@ -570,28 +672,34 @@ static void lru_lazyfree_fn(struct page *page, struct lruvec *lruvec, */ void lru_add_drain_cpu(int cpu) { - struct pagevec *pvec = &per_cpu(lru_add_pvec, cpu); + struct swap_pagevec *swpvec = lock_swap_pvec_cpu(&lru_add_pvec, cpu); + struct pagevec *pvec = &swpvec->pvec; + unsigned long flags; if (pagevec_count(pvec)) __pagevec_lru_add(pvec); + unlock_swap_pvec_cpu(swpvec); - pvec = &per_cpu(lru_rotate_pvecs, cpu); + swpvec = lock_swap_pvec_cpu_irqsave(&lru_rotate_pvecs, cpu, &flags); + pvec = &swpvec->pvec; if (pagevec_count(pvec)) { - unsigned long flags; /* No harm done if a racing interrupt already did this */ - local_irq_save(flags); pagevec_move_tail(pvec); - local_irq_restore(flags); } + unlock_swap_pvec_irqrestore(swpvec, flags); - pvec = &per_cpu(lru_deactivate_file_pvecs, cpu); + swpvec = lock_swap_pvec_cpu(&lru_deactivate_file_pvecs, cpu); + pvec = &swpvec->pvec; if (pagevec_count(pvec)) pagevec_lru_move_fn(pvec, lru_deactivate_file_fn, NULL); + unlock_swap_pvec_cpu(swpvec); - pvec = &per_cpu(lru_lazyfree_pvecs, cpu); + swpvec = lock_swap_pvec_cpu(&lru_lazyfree_pvecs, cpu); + pvec = &swpvec->pvec; if (pagevec_count(pvec)) pagevec_lru_move_fn(pvec, lru_lazyfree_fn, NULL); + unlock_swap_pvec_cpu(swpvec); activate_page_drain(cpu); } @@ -606,6 +714,9 @@ void lru_add_drain_cpu(int cpu) */ void deactivate_file_page(struct page *page) { + struct swap_pagevec *swpvec; + struct pagevec *pvec; + /* * In a workload with many unevictable page such as mprotect, * unevictable page deactivation for accelerating reclaim is pointless. @@ -614,11 +725,12 @@ void deactivate_file_page(struct page *page) return; if (likely(get_page_unless_zero(page))) { - struct pagevec *pvec = &get_cpu_var(lru_deactivate_file_pvecs); + swpvec = lock_swap_pvec(&lru_deactivate_file_pvecs); + pvec = &swpvec->pvec; if (!pagevec_add(pvec, page) || PageCompound(page)) pagevec_lru_move_fn(pvec, lru_deactivate_file_fn, NULL); - put_cpu_var(lru_deactivate_file_pvecs); + unlock_swap_pvec(swpvec, &lru_deactivate_file_pvecs); } } @@ -631,21 +743,29 @@ void deactivate_file_page(struct page *page) */ void mark_page_lazyfree(struct page *page) { + struct swap_pagevec *swpvec; + struct pagevec *pvec; + if (PageLRU(page) && PageAnon(page) && PageSwapBacked(page) && !PageSwapCache(page) && !PageUnevictable(page)) { - struct pagevec *pvec = &get_cpu_var(lru_lazyfree_pvecs); + swpvec = lock_swap_pvec(&lru_lazyfree_pvecs); + pvec = &swpvec->pvec; get_page(page); if (!pagevec_add(pvec, page) || PageCompound(page)) pagevec_lru_move_fn(pvec, lru_lazyfree_fn, NULL); - put_cpu_var(lru_lazyfree_pvecs); + unlock_swap_pvec(swpvec, &lru_lazyfree_pvecs); } } void lru_add_drain(void) { - lru_add_drain_cpu(get_cpu()); - put_cpu(); + if (static_branch_likely(&use_pvec_lock)) { + lru_add_drain_cpu(raw_smp_processor_id()); + } else { + lru_add_drain_cpu(get_cpu()); + put_cpu(); + } } #ifdef CONFIG_SMP @@ -683,10 +803,10 @@ void lru_add_drain_all(void) for_each_online_cpu(cpu) { struct work_struct *work = &per_cpu(lru_add_drain_work, cpu); - if (pagevec_count(&per_cpu(lru_add_pvec, cpu)) || - pagevec_count(&per_cpu(lru_rotate_pvecs, cpu)) || - pagevec_count(&per_cpu(lru_deactivate_file_pvecs, cpu)) || - pagevec_count(&per_cpu(lru_lazyfree_pvecs, cpu)) || + if (pagevec_count(&per_cpu(lru_add_pvec.pvec, cpu)) || + pagevec_count(&per_cpu(lru_rotate_pvecs.pvec, cpu)) || + pagevec_count(&per_cpu(lru_deactivate_file_pvecs.pvec, cpu)) || + pagevec_count(&per_cpu(lru_lazyfree_pvecs.pvec, cpu)) || need_activate_page_drain(cpu)) { INIT_WORK(work, lru_add_drain_per_cpu); queue_work_on(cpu, mm_percpu_wq, work); From patchwork Wed Apr 24 11:12:07 2019 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Sebastian Andrzej Siewior X-Patchwork-Id: 10914567 Return-Path: Received: from mail.wl.linuxfoundation.org (pdx-wl-mail.web.codeaurora.org [172.30.200.125]) by pdx-korg-patchwork-2.web.codeaurora.org (Postfix) with ESMTP id DC6EB1515 for ; Wed, 24 Apr 2019 11:12:42 +0000 (UTC) Received: from mail.wl.linuxfoundation.org (localhost [127.0.0.1]) by mail.wl.linuxfoundation.org (Postfix) with ESMTP id C9F4428869 for ; Wed, 24 Apr 2019 11:12:42 +0000 (UTC) Received: by mail.wl.linuxfoundation.org (Postfix, from userid 486) id BE12A289B1; Wed, 24 Apr 2019 11:12:42 +0000 (UTC) X-Spam-Checker-Version: SpamAssassin 3.3.1 (2010-03-16) on pdx-wl-mail.web.codeaurora.org X-Spam-Level: X-Spam-Status: No, score=-2.9 required=2.0 tests=BAYES_00,MAILING_LIST_MULTI, RCVD_IN_DNSWL_NONE autolearn=ham version=3.3.1 Received: from kanga.kvack.org (kanga.kvack.org [205.233.56.17]) by mail.wl.linuxfoundation.org (Postfix) with ESMTP id 4182D28869 for ; Wed, 24 Apr 2019 11:12:42 +0000 (UTC) Received: by kanga.kvack.org (Postfix) id 5988A6B0008; Wed, 24 Apr 2019 07:12:41 -0400 (EDT) Delivered-To: linux-mm-outgoing@kvack.org Received: by kanga.kvack.org (Postfix, from userid 40) id 520326B000A; Wed, 24 Apr 2019 07:12:41 -0400 (EDT) X-Original-To: int-list-linux-mm@kvack.org X-Delivered-To: int-list-linux-mm@kvack.org Received: by kanga.kvack.org (Postfix, from userid 63042) id 3E97D6B000C; Wed, 24 Apr 2019 07:12:41 -0400 (EDT) X-Original-To: linux-mm@kvack.org X-Delivered-To: linux-mm@kvack.org Received: from mail-wr1-f69.google.com (mail-wr1-f69.google.com [209.85.221.69]) by kanga.kvack.org (Postfix) with ESMTP id E5F6C6B0008 for ; Wed, 24 Apr 2019 07:12:40 -0400 (EDT) Received: by mail-wr1-f69.google.com with SMTP id t9so17413130wrs.16 for ; Wed, 24 Apr 2019 04:12:40 -0700 (PDT) X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20161025; h=x-original-authentication-results:x-gm-message-state:from:to:cc :subject:date:message-id:in-reply-to:references:mime-version :content-transfer-encoding; bh=DIn5FEbRXYQzdZBIvpyZtr7sGYOlqQ3CKLeCoHcIFJI=; b=EHPOroHkBXyRuQqEMfpS6rrnLhstH5BfGTNNF38WXWw+l8HqCNilX4S09GNuhslJWm hJV2XxgP2xtwKldJRCNHgzr8D/gptNCJ+Pbv/6V2y/sECELzPwTki8DpytpVycdCt/s9 pdSUuqkNz6ZgKNjMvq1YeBI73b2pV7aIUSXgAUwn1pSQ8dQmBL44SmfSYX4FdyAlDF2Q 85RWqlcgUqsRihV2MI/Pwp+ehbVw6NO2sjOAK3Thr8xx6JNZLYKkpR/WG/jRvGReZg1+ cVP0IC7X9y4A0P9N1p2pIrfSF1NdcrP3xDIaezt158fPwRUCqErOhCsEBvdb5Of16LVQ 1BlA== X-Original-Authentication-Results: mx.google.com; spf=pass (google.com: best guess record for domain of bigeasy@linutronix.de designates 2a01:7a0:2:106d:700::1 as permitted sender) smtp.mailfrom=bigeasy@linutronix.de X-Gm-Message-State: APjAAAUOCLHpDkpOQR5E/wSb58LdKUIPdnQ3c7AKkEeR2KrrLAsvNQMr JOs642MVEwdb4QN8yPGuyC3dOtzDZNUn2wslOBcVIQyOpaEiQP4n4JM9k+e0aDfym+G9EOEgEnE T6VAzyx7o/e9xOk83OtQqrGquBe2359ob3ukDCaRBfpAMTavhhHy2SqYlZW+b+ffqrg== X-Received: by 2002:a5d:6848:: with SMTP id o8mr6426032wrw.204.1556104360404; Wed, 24 Apr 2019 04:12:40 -0700 (PDT) X-Google-Smtp-Source: APXvYqzX97uhTFodJ1QsuwvLCXJnBdenyVyLJvrd+F1coAKQth47A2i9bBKxzqjpGrRJxhQUidYh X-Received: by 2002:a5d:6848:: with SMTP id o8mr6425984wrw.204.1556104359561; Wed, 24 Apr 2019 04:12:39 -0700 (PDT) ARC-Seal: i=1; a=rsa-sha256; t=1556104359; cv=none; d=google.com; s=arc-20160816; b=qUheegLreovQh0i91Y1Fi7SjBCTRc5DisxG9DGmFHyjE3q89xeMlqNs6JA7i/Iq2S3 q9zDvYhY+Rgy9MatfYm6BnqhYDt4CBadm3wJF/e/y8I3qXgGncIMyynjmcqRaYDJpzZg SiR7hSX160FL20mHR2cKd8BypyiouqvO2tnb8t9TRnYtZlJ2JRdcX1qUPZluqe13Axml z4YfVKnY0R0FIpljMLjPpTKvn49R4PVMM+CnQj7GHOZMgVU33TGdqMk4vD4vRowVO1vC n/GT9LAuw26p9yDw4bezLMW+V6YhQ5r3mPIOWPEqco0CKJh9/KQWjn/n6iO8h5ypoD9W TRiQ== ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=google.com; s=arc-20160816; h=content-transfer-encoding:mime-version:references:in-reply-to :message-id:date:subject:cc:to:from; bh=DIn5FEbRXYQzdZBIvpyZtr7sGYOlqQ3CKLeCoHcIFJI=; b=bsEcpXXxf0NRwYI5B2UTHRd+NmpWRuCkZ8CIAjXWJ14mcplXpLxHESHrI0wx9691yu PPrj/eAx2aoyr02/bt63o5WDM9FyDPtKNxZb+bZbXiQzNV4bjM0a0fhT5eioPJQOmU94 /C7QLbKr2cqMtGBigWBZfkZurCANdpP3fJedA1+aXAWmWgiXsqoDTRLd+pCva+4N/qDQ Vf0g6q/qwRMZJsTaWTRpPIdg0XO4xrHp/QdHXMpdkJJXBHBjur4h9+9/tEIU98jxdwdi MeicunYNWBlEU1FR8BZdygcIpqGTJa4hqlPd1UEohU2o8W9BDohN/d7vJKvHrm9tucAu 1iRQ== ARC-Authentication-Results: i=1; mx.google.com; spf=pass (google.com: best guess record for domain of bigeasy@linutronix.de designates 2a01:7a0:2:106d:700::1 as permitted sender) smtp.mailfrom=bigeasy@linutronix.de Received: from Galois.linutronix.de (Galois.linutronix.de. [2a01:7a0:2:106d:700::1]) by mx.google.com with ESMTPS id z9si3588666wmi.28.2019.04.24.04.12.39 for (version=TLS1_2 cipher=AES128-SHA bits=128/128); Wed, 24 Apr 2019 04:12:39 -0700 (PDT) Received-SPF: pass (google.com: best guess record for domain of bigeasy@linutronix.de designates 2a01:7a0:2:106d:700::1 as permitted sender) client-ip=2a01:7a0:2:106d:700::1; Authentication-Results: mx.google.com; spf=pass (google.com: best guess record for domain of bigeasy@linutronix.de designates 2a01:7a0:2:106d:700::1 as permitted sender) smtp.mailfrom=bigeasy@linutronix.de Received: from localhost ([127.0.0.1] helo=flow.W.breakpoint.cc) by Galois.linutronix.de with esmtp (Exim 4.80) (envelope-from ) id 1hJFpN-0006KY-Rs; Wed, 24 Apr 2019 13:12:38 +0200 From: Sebastian Andrzej Siewior To: linux-mm@kvack.org Cc: tglx@linutronix.de, frederic@kernel.org, Christoph Lameter , anna-maria@linutronix.de, Sebastian Andrzej Siewior Subject: [PATCH 3/4] mm/swap: Access struct pagevec remotely Date: Wed, 24 Apr 2019 13:12:07 +0200 Message-Id: <20190424111208.24459-4-bigeasy@linutronix.de> X-Mailer: git-send-email 2.20.1 In-Reply-To: <20190424111208.24459-1-bigeasy@linutronix.de> References: <20190424111208.24459-1-bigeasy@linutronix.de> MIME-Version: 1.0 X-Bogosity: Ham, tests=bogofilter, spamicity=0.000000, version=1.2.4 Sender: owner-linux-mm@kvack.org Precedence: bulk X-Loop: owner-majordomo@kvack.org List-ID: X-Virus-Scanned: ClamAV using ClamSMTP From: Thomas Gleixner When the newly introduced static key would be enabled, struct pagevec is locked during access. So it is possible to access it from a remote CPU. The advantage is that the work can be done from the "requesting" CPU without firing a worker on a remote CPU and waiting for it to complete the work. No functional change because static key is not enabled. Signed-off-by: Thomas Gleixner Signed-off-by: Anna-Maria Gleixner Signed-off-by: Sebastian Andrzej Siewior --- mm/swap.c | 75 +++++++++++++++++++++++++++++++++---------------------- 1 file changed, 45 insertions(+), 30 deletions(-) diff --git a/mm/swap.c b/mm/swap.c index 136c80480dbde..ea623255cd305 100644 --- a/mm/swap.c +++ b/mm/swap.c @@ -774,7 +774,8 @@ static DEFINE_PER_CPU(struct work_struct, lru_add_drain_work); static void lru_add_drain_per_cpu(struct work_struct *dummy) { - lru_add_drain(); + if (static_branch_unlikely(&use_pvec_lock)) + lru_add_drain(); } /* @@ -786,38 +787,52 @@ static void lru_add_drain_per_cpu(struct work_struct *dummy) */ void lru_add_drain_all(void) { - static DEFINE_MUTEX(lock); - static struct cpumask has_work; - int cpu; + if (static_branch_likely(&use_pvec_lock)) { + int cpu; - /* - * Make sure nobody triggers this path before mm_percpu_wq is fully - * initialized. - */ - if (WARN_ON(!mm_percpu_wq)) - return; - - mutex_lock(&lock); - cpumask_clear(&has_work); - - for_each_online_cpu(cpu) { - struct work_struct *work = &per_cpu(lru_add_drain_work, cpu); - - if (pagevec_count(&per_cpu(lru_add_pvec.pvec, cpu)) || - pagevec_count(&per_cpu(lru_rotate_pvecs.pvec, cpu)) || - pagevec_count(&per_cpu(lru_deactivate_file_pvecs.pvec, cpu)) || - pagevec_count(&per_cpu(lru_lazyfree_pvecs.pvec, cpu)) || - need_activate_page_drain(cpu)) { - INIT_WORK(work, lru_add_drain_per_cpu); - queue_work_on(cpu, mm_percpu_wq, work); - cpumask_set_cpu(cpu, &has_work); + for_each_online_cpu(cpu) { + if (pagevec_count(&per_cpu(lru_add_pvec.pvec, cpu)) || + pagevec_count(&per_cpu(lru_rotate_pvecs.pvec, cpu)) || + pagevec_count(&per_cpu(lru_deactivate_file_pvecs.pvec, cpu)) || + pagevec_count(&per_cpu(lru_lazyfree_pvecs.pvec, cpu)) || + need_activate_page_drain(cpu)) { + lru_add_drain_cpu(cpu); + } } + } else { + static DEFINE_MUTEX(lock); + static struct cpumask has_work; + int cpu; + + /* + * Make sure nobody triggers this path before mm_percpu_wq + * is fully initialized. + */ + if (WARN_ON(!mm_percpu_wq)) + return; + + mutex_lock(&lock); + cpumask_clear(&has_work); + + for_each_online_cpu(cpu) { + struct work_struct *work = &per_cpu(lru_add_drain_work, cpu); + + if (pagevec_count(&per_cpu(lru_add_pvec.pvec, cpu)) || + pagevec_count(&per_cpu(lru_rotate_pvecs.pvec, cpu)) || + pagevec_count(&per_cpu(lru_deactivate_file_pvecs.pvec, cpu)) || + pagevec_count(&per_cpu(lru_lazyfree_pvecs.pvec, cpu)) || + need_activate_page_drain(cpu)) { + INIT_WORK(work, lru_add_drain_per_cpu); + queue_work_on(cpu, mm_percpu_wq, work); + cpumask_set_cpu(cpu, &has_work); + } + } + + for_each_cpu(cpu, &has_work) + flush_work(&per_cpu(lru_add_drain_work, cpu)); + + mutex_unlock(&lock); } - - for_each_cpu(cpu, &has_work) - flush_work(&per_cpu(lru_add_drain_work, cpu)); - - mutex_unlock(&lock); } #else void lru_add_drain_all(void) From patchwork Wed Apr 24 11:12:08 2019 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Sebastian Andrzej Siewior X-Patchwork-Id: 10914569 Return-Path: Received: from mail.wl.linuxfoundation.org (pdx-wl-mail.web.codeaurora.org [172.30.200.125]) by pdx-korg-patchwork-2.web.codeaurora.org (Postfix) with ESMTP id D32EB922 for ; Wed, 24 Apr 2019 11:12:47 +0000 (UTC) Received: from mail.wl.linuxfoundation.org (localhost [127.0.0.1]) by mail.wl.linuxfoundation.org (Postfix) with ESMTP id C3520289B1 for ; Wed, 24 Apr 2019 11:12:47 +0000 (UTC) Received: by mail.wl.linuxfoundation.org (Postfix, from userid 486) id B79C6289F1; Wed, 24 Apr 2019 11:12:47 +0000 (UTC) X-Spam-Checker-Version: SpamAssassin 3.3.1 (2010-03-16) on pdx-wl-mail.web.codeaurora.org X-Spam-Level: X-Spam-Status: No, score=-2.9 required=2.0 tests=BAYES_00,MAILING_LIST_MULTI, RCVD_IN_DNSWL_NONE autolearn=ham version=3.3.1 Received: from kanga.kvack.org (kanga.kvack.org [205.233.56.17]) by mail.wl.linuxfoundation.org (Postfix) with ESMTP id 5752C289B1 for ; Wed, 24 Apr 2019 11:12:47 +0000 (UTC) Received: by kanga.kvack.org (Postfix) id 71FDA6B000A; Wed, 24 Apr 2019 07:12:46 -0400 (EDT) Delivered-To: linux-mm-outgoing@kvack.org Received: by kanga.kvack.org (Postfix, from userid 40) id 6A7456B000C; Wed, 24 Apr 2019 07:12:46 -0400 (EDT) X-Original-To: int-list-linux-mm@kvack.org X-Delivered-To: int-list-linux-mm@kvack.org Received: by kanga.kvack.org (Postfix, from userid 63042) id 549086B000D; Wed, 24 Apr 2019 07:12:46 -0400 (EDT) X-Original-To: linux-mm@kvack.org X-Delivered-To: linux-mm@kvack.org Received: from mail-wr1-f71.google.com (mail-wr1-f71.google.com [209.85.221.71]) by kanga.kvack.org (Postfix) with ESMTP id 0483D6B000A for ; Wed, 24 Apr 2019 07:12:46 -0400 (EDT) Received: by mail-wr1-f71.google.com with SMTP id r7so7886319wrc.14 for ; Wed, 24 Apr 2019 04:12:45 -0700 (PDT) X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20161025; h=x-original-authentication-results:x-gm-message-state:from:to:cc :subject:date:message-id:in-reply-to:references:mime-version :content-transfer-encoding; bh=Nrdy8KLOBuiWA1DS+gjdUN5k8KpcliPOrG7O1lyxcrA=; b=sSVjntas4HIVnMmKgIVm3kG7B2zvt3hQD5ojpV3B7XYLdhzMxfY36wk5Bam8DPmNmE 8UwWGawdX10RMYjxu2o1t3BpO4EwaWROPefhnsahNDrHxXRnlGwkw4O13TJ7IznWvt/X hv78+gxUHWm64QaVAObTGcs50FnxFjAwXZfd0q9UfFS5aSbWExk4ZgxKh2cBlWtHYVmi KJlVN03taPabObW2W6zm9U75Zgq9QSxc/7BRZtUl8E+e2egKRCNQEJRRa0siWhhy4hiI LOS/jmDZIGRUvBRupq//m0zbEUwYhBxWpd8vW0TO2VdxS5YI7nBI5Hy54E4HNOv9OLpE VR5g== X-Original-Authentication-Results: mx.google.com; spf=pass (google.com: best guess record for domain of bigeasy@linutronix.de designates 2a01:7a0:2:106d:700::1 as permitted sender) smtp.mailfrom=bigeasy@linutronix.de X-Gm-Message-State: APjAAAXONogCGNp/cdclw/g62AxdaIp8ixfB1hA4VMqhoeDZ3OdG3sfD G1ECiaHbSfWlbpecTgbhLcAeDqTpvrhZ0MEwz/8onUzJr8wp4ysEGuVdH5sHxe12QREs+nGQORY dNOYN1kzb2Rb6m/IJ41pf1vPWOrv/TA/jmr9Rhcg56le+TdRT6HMG1hzASqdRssZ8TQ== X-Received: by 2002:adf:ec09:: with SMTP id x9mr21931000wrn.187.1556104365546; Wed, 24 Apr 2019 04:12:45 -0700 (PDT) X-Google-Smtp-Source: APXvYqzo9d9GgidaV+51RMED+QpB0TS2eh8Ko2iewy06X/MohlM1/r7KJX2e/5a9/YSMd3udq/cM X-Received: by 2002:adf:ec09:: with SMTP id x9mr21930945wrn.187.1556104364683; Wed, 24 Apr 2019 04:12:44 -0700 (PDT) ARC-Seal: i=1; a=rsa-sha256; t=1556104364; cv=none; d=google.com; s=arc-20160816; b=HrPWftXCf0W4KdBKPz6plG45txC9mC5IG0LgpjGAdYt4w7Ozwtm2eEigskmj6NA+5/ XHI9Op9hQKmwEzrK2C5bvO3g33H3yH5+Vh07O1tODEKauTZaT3+JNvE+wJU/SfJSI5OR r4lj5lNHjkvZ2du/2XDGoX0t40wqCTglzTI8CZRXyk1wxjniZpxg+Y/sUcYz6982TJXh r4f3VMZ9kuFub7Q78XaAVL1rtQ0a+idxkYPPCrgRU7IWwVovX2FpeC5zm+T0vodYFTJI qrWA2iAFniKgBXTJkkFVJpWs37M9NS3YAvyjYgpA5dOGLA/KTnPhIN3/BBNWs8fDsD+V Fn6w== ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=google.com; s=arc-20160816; h=content-transfer-encoding:mime-version:references:in-reply-to :message-id:date:subject:cc:to:from; bh=Nrdy8KLOBuiWA1DS+gjdUN5k8KpcliPOrG7O1lyxcrA=; b=NVPyGOzF8+uYRnqj7q1KKob1MOvjjkAgnMj9GS64oaTM+Z7qLL2HKEU2l0TrT3MrqN 1ErgEMI6sjSfBYKC9Z0jEqoe7DT74ewZVuR5qOmCnlE/yGLK+wBUoGSfZukxGg5UXV/s Tpu4J4kz7ZdALm7nDyZKr84A3vLEsj/BAVn8Fb8C4fdPufDzM1fnCEUbVMxBTy/taZSY nC0vLSNcWoa/clyAazl7T4Zxe5A9pbL5e0SDatB7quYHtgR4AFkdSqm+fMU8ZeEMFLCG GuZMZDuMBP4P2YoDWdVSECQxWeA+nTmMCms4OkmjN0H/F07gfp/4XumWaUoTwWVVr6ko mYWg== ARC-Authentication-Results: i=1; mx.google.com; spf=pass (google.com: best guess record for domain of bigeasy@linutronix.de designates 2a01:7a0:2:106d:700::1 as permitted sender) smtp.mailfrom=bigeasy@linutronix.de Received: from Galois.linutronix.de (Galois.linutronix.de. [2a01:7a0:2:106d:700::1]) by mx.google.com with ESMTPS id g130si13144045wmf.140.2019.04.24.04.12.44 for (version=TLS1_2 cipher=AES128-SHA bits=128/128); Wed, 24 Apr 2019 04:12:44 -0700 (PDT) Received-SPF: pass (google.com: best guess record for domain of bigeasy@linutronix.de designates 2a01:7a0:2:106d:700::1 as permitted sender) client-ip=2a01:7a0:2:106d:700::1; Authentication-Results: mx.google.com; spf=pass (google.com: best guess record for domain of bigeasy@linutronix.de designates 2a01:7a0:2:106d:700::1 as permitted sender) smtp.mailfrom=bigeasy@linutronix.de Received: from localhost ([127.0.0.1] helo=flow.W.breakpoint.cc) by Galois.linutronix.de with esmtp (Exim 4.80) (envelope-from ) id 1hJFpS-0006KY-OT; Wed, 24 Apr 2019 13:12:42 +0200 From: Sebastian Andrzej Siewior To: linux-mm@kvack.org Cc: tglx@linutronix.de, frederic@kernel.org, Christoph Lameter , anna-maria@linutronix.de, Sebastian Andrzej Siewior Subject: [PATCH 4/4] mm/swap: Enable "use_pvec_lock" nohz_full dependent Date: Wed, 24 Apr 2019 13:12:08 +0200 Message-Id: <20190424111208.24459-5-bigeasy@linutronix.de> X-Mailer: git-send-email 2.20.1 In-Reply-To: <20190424111208.24459-1-bigeasy@linutronix.de> References: <20190424111208.24459-1-bigeasy@linutronix.de> MIME-Version: 1.0 X-Bogosity: Ham, tests=bogofilter, spamicity=0.000000, version=1.2.4 Sender: owner-linux-mm@kvack.org Precedence: bulk X-Loop: owner-majordomo@kvack.org List-ID: X-Virus-Scanned: ClamAV using ClamSMTP From: Anna-Maria Gleixner When a system runs with CONFIG_NO_HZ_FULL enabled, the tick of CPUs listed in 'nohz_full=' kernel command line parameter should be stopped whenever possible. The tick stays longer stopped, when work for this CPU is handled by another CPU. With the already introduced static key 'use_pvec_lock' there is the possibility to prevent firing a worker for mm/swap work on a remote CPU with a stopped tick. Therefore enabling the static key in case kernel command line parameter 'nohz_full=' setup was successful, which implies that CONFIG_NO_HZ_FULL is set. Signed-off-by: Anna-Maria Gleixner Signed-off-by: Sebastian Andrzej Siewior --- kernel/sched/isolation.c | 14 +++++++++++++- 1 file changed, 13 insertions(+), 1 deletion(-) diff --git a/kernel/sched/isolation.c b/kernel/sched/isolation.c index b02d148e76727..b532f448cab42 100644 --- a/kernel/sched/isolation.c +++ b/kernel/sched/isolation.c @@ -7,6 +7,7 @@ * */ #include "sched.h" +#include "../../mm/internal.h" DEFINE_STATIC_KEY_FALSE(housekeeping_overridden); EXPORT_SYMBOL_GPL(housekeeping_overridden); @@ -116,10 +117,21 @@ static int __init housekeeping_setup(char *str, enum hk_flags flags) static int __init housekeeping_nohz_full_setup(char *str) { unsigned int flags; + int ret; flags = HK_FLAG_TICK | HK_FLAG_WQ | HK_FLAG_TIMER | HK_FLAG_RCU | HK_FLAG_MISC; - return housekeeping_setup(str, flags); + ret = housekeeping_setup(str, flags); + + /* + * Protect struct pagevec with a lock instead using preemption disable; + * with lock protection, remote handling of events instead of queue + * work on remote cpu is default behavior. + */ + if (ret) + static_branch_enable(&use_pvec_lock); + + return ret; } __setup("nohz_full=", housekeeping_nohz_full_setup);