From patchwork Tue Sep 11 00:59:47 2018 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Daniel Jordan X-Patchwork-Id: 10594957 Return-Path: Received: from mail.wl.linuxfoundation.org (pdx-wl-mail.web.codeaurora.org [172.30.200.125]) by pdx-korg-patchwork-2.web.codeaurora.org (Postfix) with ESMTP id B1C686CB for ; Tue, 11 Sep 2018 01:00:12 +0000 (UTC) Received: from mail.wl.linuxfoundation.org (localhost [127.0.0.1]) by mail.wl.linuxfoundation.org (Postfix) with ESMTP id A02F329010 for ; Tue, 11 Sep 2018 01:00:12 +0000 (UTC) Received: by mail.wl.linuxfoundation.org (Postfix, from userid 486) id 939CF2901E; Tue, 11 Sep 2018 01:00:12 +0000 (UTC) X-Spam-Checker-Version: SpamAssassin 3.3.1 (2010-03-16) on pdx-wl-mail.web.codeaurora.org X-Spam-Level: X-Spam-Status: No, score=-3.0 required=2.0 tests=BAYES_00,DKIM_SIGNED, DKIM_VALID,DKIM_VALID_AU,MAILING_LIST_MULTI,RCVD_IN_DNSWL_NONE, UNPARSEABLE_RELAY autolearn=ham version=3.3.1 Received: from kanga.kvack.org (kanga.kvack.org [205.233.56.17]) by mail.wl.linuxfoundation.org (Postfix) with ESMTP id 028A629010 for ; Tue, 11 Sep 2018 01:00:12 +0000 (UTC) Received: by kanga.kvack.org (Postfix) id BED758E0007; Mon, 10 Sep 2018 21:00:09 -0400 (EDT) Delivered-To: linux-mm-outgoing@kvack.org Received: by kanga.kvack.org (Postfix, from userid 40) id B29FD8E0001; Mon, 10 Sep 2018 21:00:09 -0400 (EDT) X-Original-To: int-list-linux-mm@kvack.org X-Delivered-To: int-list-linux-mm@kvack.org Received: by kanga.kvack.org (Postfix, from userid 63042) id 97A218E0007; Mon, 10 Sep 2018 21:00:09 -0400 (EDT) X-Original-To: linux-mm@kvack.org X-Delivered-To: linux-mm@kvack.org Received: from mail-yw1-f69.google.com (mail-yw1-f69.google.com [209.85.161.69]) by kanga.kvack.org (Postfix) with ESMTP id 65B488E0001 for ; Mon, 10 Sep 2018 21:00:09 -0400 (EDT) Received: by mail-yw1-f69.google.com with SMTP id d20-v6so14510853ywa.16 for ; Mon, 10 Sep 2018 18:00:09 -0700 (PDT) X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20161025; h=x-gm-message-state:dkim-signature:from:to:cc:subject:date :message-id:in-reply-to:references; bh=Unlfp95VRYdh0zaKiyB62CoToWdwSBQEhoPttxdHTH0=; b=mi3JJZyd53O/B9nDw4WoPK17HQOb4+h1M5Dx99huqMxHzkqtnDEeJ2WjlfzxpP7zFq s+KUvSyZt9ygPJ7AjDgkdAcyLeOTgOPDhGJJkBQDIXLXubQghKIB+wmP93pR22scaSgi PfSTr/4kp8H3sp3Wz+vk9yORcu6F4On9Jrm4qLoOUbI7zS3qNoPIwcUc+izPAHfRZ6EJ /rjFF0nH5NIRXs4457kwB7B1gItoo+FRHrzX9QzrTXz5NFemiQ1HN9HdqKEcaYYFv5mg o/Zhf4e+oY1ZMWtj2yhVmQRIsLeP5YsQhMscpNMSWytjOc1L852yrvR8oeBkt8aTaA4D 4olA== X-Gm-Message-State: APzg51C1RYLx87+TH3t8aEtMq0KXSpea8EQoE2j0hfVvvwGYXE+MmG1Y 3NI561NWHCLc7TBeFNVrh5fSqldxie2rW71roROBfQtDjum5LRTRblTAycdDSdMrv3ISvvDpHp9 DFIhdxODJmTFoeHaioOyXU2WxfOa93L8ZJ4NB9Hv8PszH52eZ7QyTHP/Ev7N31mB86A== X-Received: by 2002:a25:b7d0:: with SMTP id u16-v6mr4168258ybj.52.1536627609105; Mon, 10 Sep 2018 18:00:09 -0700 (PDT) X-Google-Smtp-Source: ANB0VdYAxDu3E7/XDYfosFgNlRJ6Tbb9/ksDntAvPuXnpGpxspRkd54hEacaQa3a/zbIjm7epPe1 X-Received: by 2002:a25:b7d0:: with SMTP id u16-v6mr4168246ybj.52.1536627608384; Mon, 10 Sep 2018 18:00:08 -0700 (PDT) ARC-Seal: i=1; a=rsa-sha256; t=1536627608; cv=none; d=google.com; s=arc-20160816; b=spJs7Ouco5CBhtf7EhC6I1lPoZ4m/yNInsu7+ASPDTMA4ssqVeZP0txGMaNMUmJFNV mvkQe1Zriv2vvxy8oi2bC77rfvbGu+VdU4eJtKUKb+l+OtXGCCzNjI8C6xrwqy9GOdL/ k9TNzBOBoLkHSE/t1Qf9Ucmp8GEP0q0jrwjvaK2uvGs6DMnZ/7lC7yqhytC0ztOusWz2 f6Knlivk8sqoJM1R3iJ7yiqnI5Swf1H8XyFxVXgzA55aitpceTL89KhkyG9WlvGuL9tt 5rhBnw/FmorIJI5waybuVORBKrFJEa5CIyfVTUyudFsg+btmwnTfICCZeCseldZ9m5nL qNwQ== ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=google.com; s=arc-20160816; h=references:in-reply-to:message-id:date:subject:cc:to:from :dkim-signature; bh=Unlfp95VRYdh0zaKiyB62CoToWdwSBQEhoPttxdHTH0=; b=OjXgY7vDJyTaaee+bn9Qs5YB4Hdzd7XdEAquQpG8N+N9OVMLth2IoerrHfdPeOanHX z1u4ItfNfBdjxdkDprzlRCdnnbN7G/TyVWRWQTlfufYIQTRieDTXNyRRmZyuUEvJ90s7 M47Cw/IpntlIPQXlzDmUt5Y3qy16EwBlAFl0euPSDcDsHhWFkcjo44bAyMq1GHJNk6mY SdNC4CYqVnGVFG/xSD2haw9bgLvOgBHWrj538F+VYv4kFBv9087QsSCPSjClQVzU8ixa Os7MZHyFtOsaLfW5B2O3KTpJPS4yxC4YLkaM1EaxWoZqxYbXKeIZLfwnkQjpqQr52OTR vcQA== ARC-Authentication-Results: i=1; mx.google.com; dkim=pass header.i=@oracle.com header.s=corp-2018-07-02 header.b=u02JgrKp; spf=pass (google.com: domain of daniel.m.jordan@oracle.com designates 156.151.31.86 as permitted sender) smtp.mailfrom=daniel.m.jordan@oracle.com; dmarc=pass (p=NONE sp=NONE dis=NONE) header.from=oracle.com Received: from userp2130.oracle.com (userp2130.oracle.com. [156.151.31.86]) by mx.google.com with ESMTPS id f186-v6si4335184ywc.56.2018.09.10.18.00.08 for (version=TLS1_2 cipher=ECDHE-RSA-AES128-GCM-SHA256 bits=128/128); Mon, 10 Sep 2018 18:00:08 -0700 (PDT) Received-SPF: pass (google.com: domain of daniel.m.jordan@oracle.com designates 156.151.31.86 as permitted sender) client-ip=156.151.31.86; Authentication-Results: mx.google.com; dkim=pass header.i=@oracle.com header.s=corp-2018-07-02 header.b=u02JgrKp; spf=pass (google.com: domain of daniel.m.jordan@oracle.com designates 156.151.31.86 as permitted sender) smtp.mailfrom=daniel.m.jordan@oracle.com; dmarc=pass (p=NONE sp=NONE dis=NONE) header.from=oracle.com Received: from pps.filterd (userp2130.oracle.com [127.0.0.1]) by userp2130.oracle.com (8.16.0.22/8.16.0.22) with SMTP id w8B0xj2Y077213; Tue, 11 Sep 2018 01:00:02 GMT DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=oracle.com; h=from : to : cc : subject : date : message-id : in-reply-to : references; s=corp-2018-07-02; bh=Unlfp95VRYdh0zaKiyB62CoToWdwSBQEhoPttxdHTH0=; b=u02JgrKpOKciZh4jINL64Aaced2qCB6CUVl7FrYZy0RUveF/au1rTGvHhRN2hYrSsqOX mXpPg69Smv9wvL83jQoBnTjZJj4hZz253cFDZPKyfZCOtuCRENLSBnkuT+xeN5YHz2vQ B/N0a4cFKrl2nldUy5awBHcgytQ4KygmvGdPSD0Jz06RAKEYki474Cy771pcDccz4iWH MSB6ULzQLLW9EDbpw+ifT2JW/lD/3JeMGOYUME4df8C9o2RMIWfmihKrs9kdMArxIAh6 tcuiuuEPeSJOoNNvDbDwBSGKg8C8sQmtXFESclhP+bjKeedhWMGcts5RHE+aW6I8mV3p cA== Received: from aserv0022.oracle.com (aserv0022.oracle.com [141.146.126.234]) by userp2130.oracle.com with ESMTP id 2mc5ut94ed-1 (version=TLSv1.2 cipher=ECDHE-RSA-AES256-GCM-SHA384 bits=256 verify=OK); Tue, 11 Sep 2018 01:00:02 +0000 Received: from userv0122.oracle.com (userv0122.oracle.com [156.151.31.75]) by aserv0022.oracle.com (8.14.4/8.14.4) with ESMTP id w8B101Na005446 (version=TLSv1/SSLv3 cipher=DHE-RSA-AES256-GCM-SHA384 bits=256 verify=OK); Tue, 11 Sep 2018 01:00:01 GMT Received: from abhmp0009.oracle.com (abhmp0009.oracle.com [141.146.116.15]) by userv0122.oracle.com (8.14.4/8.14.4) with ESMTP id w8B0xxqM020122; Tue, 11 Sep 2018 01:00:00 GMT Received: from localhost.localdomain (/73.143.71.164) by default (Oracle Beehive Gateway v4.0) with ESMTP ; Mon, 10 Sep 2018 17:59:59 -0700 From: Daniel Jordan To: linux-mm@kvack.org, linux-kernel@vger.kernel.org, cgroups@vger.kernel.org Cc: aaron.lu@intel.com, ak@linux.intel.com, akpm@linux-foundation.org, dave.dice@oracle.com, dave.hansen@linux.intel.com, hannes@cmpxchg.org, levyossi@icloud.com, ldufour@linux.vnet.ibm.com, mgorman@techsingularity.net, mhocko@kernel.org, Pavel.Tatashin@microsoft.com, steven.sistare@oracle.com, tim.c.chen@intel.com, vdavydov.dev@gmail.com, ying.huang@intel.com Subject: [RFC PATCH v2 6/8] mm: splice local lists onto the front of the LRU Date: Mon, 10 Sep 2018 20:59:47 -0400 Message-Id: <20180911005949.5635-3-daniel.m.jordan@oracle.com> X-Mailer: git-send-email 2.18.0 In-Reply-To: <20180911004240.4758-1-daniel.m.jordan@oracle.com> References: <20180911004240.4758-1-daniel.m.jordan@oracle.com> X-Proofpoint-Virus-Version: vendor=nai engine=5900 definitions=9012 signatures=668708 X-Proofpoint-Spam-Details: rule=notspam policy=default score=0 suspectscore=0 malwarescore=0 phishscore=0 bulkscore=0 spamscore=0 mlxscore=0 mlxlogscore=999 adultscore=0 classifier=spam adjust=0 reason=mlx scancount=1 engine=8.0.1-1807170000 definitions=main-1809110009 X-Bogosity: Ham, tests=bogofilter, spamicity=0.000000, version=1.2.4 Sender: owner-linux-mm@kvack.org Precedence: bulk X-Loop: owner-majordomo@kvack.org List-ID: X-Virus-Scanned: ClamAV using ClamSMTP The add-to-front LRU path currently adds one page at a time to the front of an LRU. This is slow when using the concurrent algorithm described in the next patch because the LRU head node will be locked for every page that's added. Instead, prepare local lists of pages, grouped by LRU, to be added to a given LRU in a single splice operation. The batching effect will reduce the amount of time that the LRU head is locked per page added. Signed-off-by: Daniel Jordan --- mm/swap.c | 123 ++++++++++++++++++++++++++++++++++++++++++++++++++++-- 1 file changed, 119 insertions(+), 4 deletions(-) diff --git a/mm/swap.c b/mm/swap.c index b1030eb7f459..07b951727a11 100644 --- a/mm/swap.c +++ b/mm/swap.c @@ -865,8 +865,52 @@ void lru_add_page_tail(struct page *page, struct page *page_tail, } #endif /* CONFIG_TRANSPARENT_HUGEPAGE */ -static void __pagevec_lru_add_fn(struct page *page, struct lruvec *lruvec, - void *arg) +#define MAX_LRU_SPLICES 4 + +struct lru_splice { + struct list_head list; + struct list_head *lru; + struct pglist_data *pgdat; +}; + +/* + * Adds a page to a local list for splicing, or else to the singletons + * list for individual processing. + * + * Returns the new number of splices in the splices list. + */ +static size_t add_page_to_splice(struct page *page, struct pglist_data *pgdat, + struct lru_splice *splices, size_t nr_splices, + struct list_head *singletons, + struct list_head *lru) +{ + int i; + + for (i = 0; i < nr_splices; ++i) { + if (splices[i].lru == lru) { + list_add(&page->lru, &splices[i].list); + return nr_splices; + } + } + + if (nr_splices < MAX_LRU_SPLICES) { + INIT_LIST_HEAD(&splices[nr_splices].list); + splices[nr_splices].lru = lru; + splices[nr_splices].pgdat = pgdat; + list_add(&page->lru, &splices[nr_splices].list); + ++nr_splices; + } else { + list_add(&page->lru, singletons); + } + + return nr_splices; +} + +static size_t pagevec_lru_add_splice(struct page *page, struct lruvec *lruvec, + struct pglist_data *pgdat, + struct lru_splice *splices, + size_t nr_splices, + struct list_head *singletons) { enum lru_list lru; int was_unevictable = TestClearPageUnevictable(page); @@ -916,8 +960,12 @@ static void __pagevec_lru_add_fn(struct page *page, struct lruvec *lruvec, count_vm_event(UNEVICTABLE_PGCULLED); } - add_page_to_lru_list(page, lruvec, lru); + nr_splices = add_page_to_splice(page, pgdat, splices, nr_splices, + singletons, &lruvec->lists[lru]); + update_lru_size(lruvec, lru, page_zonenum(page), hpage_nr_pages(page)); trace_mm_lru_insertion(page, lru); + + return nr_splices; } /* @@ -926,7 +974,74 @@ static void __pagevec_lru_add_fn(struct page *page, struct lruvec *lruvec, */ void __pagevec_lru_add(struct pagevec *pvec) { - pagevec_lru_move_fn(pvec, __pagevec_lru_add_fn, NULL); + int i; + struct pglist_data *pagepgdat, *pgdat = NULL; + unsigned long flags = 0; + struct lru_splice splices[MAX_LRU_SPLICES]; + size_t nr_splices = 0; + LIST_HEAD(singletons); + struct page *page; + struct lruvec *lruvec; + enum lru_list lru; + + /* + * Sort the pages into local lists to splice onto the LRU. In the + * common case there should be few of these local lists. + */ + for (i = 0; i < pagevec_count(pvec); ++i) { + page = pvec->pages[i]; + pagepgdat = page_pgdat(page); + + /* + * Take lru_lock now so that setting PageLRU and setting the + * local list's links appear to happen atomically. + */ + if (pagepgdat != pgdat) { + if (pgdat) + write_unlock_irqrestore(&pgdat->lru_lock, flags); + pgdat = pagepgdat; + write_lock_irqsave(&pgdat->lru_lock, flags); + } + + lruvec = mem_cgroup_page_lruvec(page, pagepgdat); + + nr_splices = pagevec_lru_add_splice(page, lruvec, pagepgdat, + splices, nr_splices, + &singletons); + } + + for (i = 0; i < nr_splices; ++i) { + struct lru_splice *splice = &splices[i]; + + if (splice->pgdat != pgdat) { + if (pgdat) + write_unlock_irqrestore(&pgdat->lru_lock, flags); + pgdat = splice->pgdat; + write_lock_irqsave(&pgdat->lru_lock, flags); + } + list_splice(&splice->list, splice->lru); + } + + while (!list_empty(&singletons)) { + page = list_first_entry(&singletons, struct page, lru); + list_del(singletons.next); + pagepgdat = page_pgdat(page); + + if (pagepgdat != pgdat) { + if (pgdat) + write_unlock_irqrestore(&pgdat->lru_lock, flags); + pgdat = pagepgdat; + write_lock_irqsave(&pgdat->lru_lock, flags); + } + + lruvec = mem_cgroup_page_lruvec(page, pgdat); + lru = page_lru(page); + list_add(&page->lru, &lruvec->lists[lru]); + } + if (pgdat) + write_unlock_irqrestore(&pgdat->lru_lock, flags); + release_pages(pvec->pages, pvec->nr); + pagevec_reinit(pvec); } EXPORT_SYMBOL(__pagevec_lru_add);