From patchwork Mon Nov 5 16:55:57 2018 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Daniel Jordan X-Patchwork-Id: 10668671 Return-Path: Received: from mail.wl.linuxfoundation.org (pdx-wl-mail.web.codeaurora.org [172.30.200.125]) by pdx-korg-patchwork-2.web.codeaurora.org (Postfix) with ESMTP id 3FF9614BD for ; Mon, 5 Nov 2018 16:57:08 +0000 (UTC) Received: from mail.wl.linuxfoundation.org (localhost [127.0.0.1]) by mail.wl.linuxfoundation.org (Postfix) with ESMTP id 2CEC8293D9 for ; Mon, 5 Nov 2018 16:57:08 +0000 (UTC) Received: by mail.wl.linuxfoundation.org (Postfix, from userid 486) id 20CB42946D; Mon, 5 Nov 2018 16:57:08 +0000 (UTC) X-Spam-Checker-Version: SpamAssassin 3.3.1 (2010-03-16) on pdx-wl-mail.web.codeaurora.org X-Spam-Level: X-Spam-Status: No, score=-3.0 required=2.0 tests=BAYES_00,DKIM_SIGNED, DKIM_VALID,DKIM_VALID_AU,MAILING_LIST_MULTI,RCVD_IN_DNSWL_NONE, UNPARSEABLE_RELAY autolearn=ham version=3.3.1 Received: from kanga.kvack.org (kanga.kvack.org [205.233.56.17]) by mail.wl.linuxfoundation.org (Postfix) with ESMTP id 8587B293D9 for ; Mon, 5 Nov 2018 16:57:07 +0000 (UTC) Received: by kanga.kvack.org (Postfix) id 42ADB6B0275; Mon, 5 Nov 2018 11:56:44 -0500 (EST) Delivered-To: linux-mm-outgoing@kvack.org Received: by kanga.kvack.org (Postfix, from userid 40) id 3D9616B0276; Mon, 5 Nov 2018 11:56:44 -0500 (EST) X-Original-To: int-list-linux-mm@kvack.org X-Delivered-To: int-list-linux-mm@kvack.org Received: by kanga.kvack.org (Postfix, from userid 63042) id 2A5286B0277; Mon, 5 Nov 2018 11:56:44 -0500 (EST) X-Original-To: linux-mm@kvack.org X-Delivered-To: linux-mm@kvack.org Received: from mail-yw1-f71.google.com (mail-yw1-f71.google.com [209.85.161.71]) by kanga.kvack.org (Postfix) with ESMTP id E84406B0275 for ; Mon, 5 Nov 2018 11:56:43 -0500 (EST) Received: by mail-yw1-f71.google.com with SMTP id i1-v6so7901757ywd.4 for ; Mon, 05 Nov 2018 08:56:43 -0800 (PST) X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20161025; h=x-gm-message-state:dkim-signature:from:to:cc:subject:date :message-id:in-reply-to:references:mime-version :content-transfer-encoding; bh=UTB2Sr3dL/IqP1KEApaT4aDh/kChWZlvDCupI65rZ48=; b=iXaRg2VbHbDEEXGZiJWh+lThIkYXS1iXuh6u4p/alzL+gvqdiIb1bWa/qQAhHwQBSC d5d9/CAqv/znB3G/fqG9WZaMptS6XNvnNNSQlUUUIYdzeGVRRlX3pifkfws0fzIizsNP jSDGbmg/tsWgkMkICGMjfOmZf3FUj2qFV+1CQZ2JxkEuLP+TY5kdPtZ+olarV9lYtml6 7bRw4V/De3PLNyvIHaqQ60o3zj/xtTRKv3+34apFomu3FYvqGAmnpTbg064Dm8wrksQ1 V8eAOENnp8lLovoL8g/5gAEWgAo3tne+dJMxDfdN9GnxJZahiJb0SUP01XsfyxtYPc1i Oa1A== X-Gm-Message-State: AGRZ1gKXKcZE+JkMehyRO6Np8y1I610/2KNmhMADfVKyjK1nYIyqp1mJ 5FeSNZ8sFxPKiPF182lfjTxQEfCBniJSCaVG2L1Gz+wSbiKoMbkBc2Hb3PcfHWUXBgeHvkfhjYq GwnzSgGGm9tjCLb9yLqKH3JYUIO91F3Z+VM22N5rSA80KJfIkGfxsHIGWKCE1rzzTVw== X-Received: by 2002:a25:4486:: with SMTP id r128-v6mr8237247yba.441.1541437003620; Mon, 05 Nov 2018 08:56:43 -0800 (PST) X-Google-Smtp-Source: AJdET5dNv041fUQ89VXgD9QOjGATIf4ratq7fxr95b1A1XpccW5stSqAmrklLnPcx8uBH0fYZp7c X-Received: by 2002:a25:4486:: with SMTP id r128-v6mr8237200yba.441.1541437002865; Mon, 05 Nov 2018 08:56:42 -0800 (PST) ARC-Seal: i=1; a=rsa-sha256; t=1541437002; cv=none; d=google.com; s=arc-20160816; b=VOGTfKISxJHWN5ruJs6f7YqfXUtePI5NP+dXtWWmPeJfSnSQvMn91kGEKDrvD8pz4H zlBhO5MYYtF7tye8wGk+hdnIH1zIrOMOsvlKPDF3wqBnymm0mlEoyltHgYd1lvysnSJH pkurSlLwX5y6naq2Q8PHMYLCBol6xe/k5Kxr4s9IG6fU5xkKbRzM6zZsIDa7DHSo1eUT AB6R9KlFXIudh6u5Xzk6PhnObknshesqF0pXi8CxuXy2rgDnis8JSUfQBJTZd2lV4eow 81/+SZzLHlQcvBX/TACifoYI1SCLRCwKBfctBTo3mUgCBEUVn2ixb+DTMwjlZE1JSeKZ lK5g== ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=google.com; s=arc-20160816; h=content-transfer-encoding:mime-version:references:in-reply-to :message-id:date:subject:cc:to:from:dkim-signature; bh=UTB2Sr3dL/IqP1KEApaT4aDh/kChWZlvDCupI65rZ48=; b=Pz7ZL4zIfOvfDvPpbqutSPkCaM3C85ylvZAMfZQb4vwHDozf3XLKRJrKRPwAvbdkHL 7Z1L/8vSgSNDgWZKRXBcPoA7eOgBJgxuy/GSDDAHJZ0YYsMrjYnNVPMrTrrf5IMWpvvH dP8yuf4jBUXOHGt93TNqpxGklfqte8LzUE5N9pFmgfuIbjSSrnEgF+cmUpXpRfHUJsYW ODpzHR+BxT2iSgu3KA/n5VbuqezIqNx7RsvoJa1JePabg+klZg9niZHhgSIqwYl7IS9a ni4zlLpRkWYfcloe/D4Y9wHjQ/it+9n9M4qp0Db7R2iVa3/dBHeYG3uE1wcTJVR1pPkH 4OOg== ARC-Authentication-Results: i=1; mx.google.com; dkim=pass header.i=@oracle.com header.s=corp-2018-07-02 header.b=0QHz6k1R; spf=pass (google.com: domain of daniel.m.jordan@oracle.com designates 156.151.31.85 as permitted sender) smtp.mailfrom=daniel.m.jordan@oracle.com; dmarc=pass (p=NONE sp=NONE dis=NONE) header.from=oracle.com Received: from userp2120.oracle.com (userp2120.oracle.com. [156.151.31.85]) by mx.google.com with ESMTPS id r132-v6si26801412ybc.409.2018.11.05.08.56.42 for (version=TLS1_2 cipher=ECDHE-RSA-AES128-GCM-SHA256 bits=128/128); Mon, 05 Nov 2018 08:56:42 -0800 (PST) Received-SPF: pass (google.com: domain of daniel.m.jordan@oracle.com designates 156.151.31.85 as permitted sender) client-ip=156.151.31.85; Authentication-Results: mx.google.com; dkim=pass header.i=@oracle.com header.s=corp-2018-07-02 header.b=0QHz6k1R; spf=pass (google.com: domain of daniel.m.jordan@oracle.com designates 156.151.31.85 as permitted sender) smtp.mailfrom=daniel.m.jordan@oracle.com; dmarc=pass (p=NONE sp=NONE dis=NONE) header.from=oracle.com Received: from pps.filterd (userp2120.oracle.com [127.0.0.1]) by userp2120.oracle.com (8.16.0.22/8.16.0.22) with SMTP id wA5Gs5sA052327; Mon, 5 Nov 2018 16:56:28 GMT DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=oracle.com; h=from : to : cc : subject : date : message-id : in-reply-to : references : mime-version : content-transfer-encoding; s=corp-2018-07-02; bh=UTB2Sr3dL/IqP1KEApaT4aDh/kChWZlvDCupI65rZ48=; b=0QHz6k1R+f97qO32+2IlnJ35k/qUdjYjcuv1TKJ4wUIXfkE6+JNJj0qrM0nghND3+w+O cAjLW1fqp98CLgSQLbB6wBLogI18brwjKrrfhq11qDMjTejr785PdYVoH1VxZ2Z/Z0Ze IydMcrlOPWbdmdONU9w42O4UZXjZbXyGe439Fz1ate0YvQ/+SB7ln/O45xJFihP1AfNJ gpG1cyhtQippKedqJxaNV46cYx78VZ6QmEvxrTeeunagc78IXJs7JRQgsruiD7iRwojV 3sUN/yabo7coxLf76IbIeC66BuznidvengcvZ1/T2h08L3xA3Af1c93X59WstgN/g6Ru bg== Received: from aserv0022.oracle.com (aserv0022.oracle.com [141.146.126.234]) by userp2120.oracle.com with ESMTP id 2nh4aqg2e6-1 (version=TLSv1.2 cipher=ECDHE-RSA-AES256-GCM-SHA384 bits=256 verify=OK); Mon, 05 Nov 2018 16:56:28 +0000 Received: from aserv0121.oracle.com (aserv0121.oracle.com [141.146.126.235]) by aserv0022.oracle.com (8.14.4/8.14.4) with ESMTP id wA5GuRaO022744 (version=TLSv1/SSLv3 cipher=DHE-RSA-AES256-GCM-SHA384 bits=256 verify=OK); Mon, 5 Nov 2018 16:56:27 GMT Received: from abhmp0006.oracle.com (abhmp0006.oracle.com [141.146.116.12]) by aserv0121.oracle.com (8.14.4/8.13.8) with ESMTP id wA5GuRg4018838; Mon, 5 Nov 2018 16:56:27 GMT Received: from localhost.localdomain (/73.60.114.248) by default (Oracle Beehive Gateway v4.0) with ESMTP ; Mon, 05 Nov 2018 08:56:26 -0800 From: Daniel Jordan To: linux-mm@kvack.org, kvm@vger.kernel.org, linux-kernel@vger.kernel.org Cc: aarcange@redhat.com, aaron.lu@intel.com, akpm@linux-foundation.org, alex.williamson@redhat.com, bsd@redhat.com, daniel.m.jordan@oracle.com, darrick.wong@oracle.com, dave.hansen@linux.intel.com, jgg@mellanox.com, jwadams@google.com, jiangshanlai@gmail.com, mhocko@kernel.org, mike.kravetz@oracle.com, Pavel.Tatashin@microsoft.com, prasad.singamsetty@oracle.com, rdunlap@infradead.org, steven.sistare@oracle.com, tim.c.chen@intel.com, tj@kernel.org, vbabka@suse.cz Subject: [RFC PATCH v4 12/13] mm: parallelize clear_gigantic_page Date: Mon, 5 Nov 2018 11:55:57 -0500 Message-Id: <20181105165558.11698-13-daniel.m.jordan@oracle.com> X-Mailer: git-send-email 2.19.1 In-Reply-To: <20181105165558.11698-1-daniel.m.jordan@oracle.com> References: <20181105165558.11698-1-daniel.m.jordan@oracle.com> MIME-Version: 1.0 X-Proofpoint-Virus-Version: vendor=nai engine=5900 definitions=9068 signatures=668683 X-Proofpoint-Spam-Details: rule=notspam policy=default score=0 suspectscore=0 malwarescore=0 phishscore=0 bulkscore=0 spamscore=0 mlxscore=0 mlxlogscore=999 adultscore=0 classifier=spam adjust=0 reason=mlx scancount=1 engine=8.0.1-1807170000 definitions=main-1811050153 X-Bogosity: Ham, tests=bogofilter, spamicity=0.000000, version=1.2.4 Sender: owner-linux-mm@kvack.org Precedence: bulk X-Loop: owner-majordomo@kvack.org List-ID: X-Virus-Scanned: ClamAV using ClamSMTP Parallelize clear_gigantic_page, which zeroes any page size larger than 8M (e.g. 1G on x86). Performance results (the default number of threads is 4; higher thread counts shown for context only): Machine: Intel(R) Xeon(R) CPU E7-8895 v3 @ 2.60GHz, 288 CPUs, 1T memory Test: Clear a range of gigantic pages (triggered via fallocate) nthread speedup size (GiB) min time (s) stdev 1 100 41.13 0.03 2 2.03x 100 20.26 0.14 4 4.28x 100 9.62 0.09 8 8.39x 100 4.90 0.05 16 10.44x 100 3.94 0.03 1 200 89.68 0.35 2 2.21x 200 40.64 0.18 4 4.64x 200 19.33 0.32 8 8.99x 200 9.98 0.04 16 11.27x 200 7.96 0.04 1 400 188.20 1.57 2 2.30x 400 81.84 0.09 4 4.63x 400 40.62 0.26 8 8.92x 400 21.09 0.50 16 11.78x 400 15.97 0.25 1 800 434.91 1.81 2 2.54x 800 170.97 1.46 4 4.98x 800 87.38 1.91 8 10.15x 800 42.86 2.59 16 12.99x 800 33.48 0.83 The speedups are mostly due to the fact that more threads can use more memory bandwidth. The loop we're stressing on the x86 chip in this test is clear_page_erms, which tops out at a bandwidth of 2550 MiB/s with one thread. We get the same bandwidth per thread for 2, 4, or 8 threads, but at 16 threads the per-thread bandwidth drops to 1420 MiB/s. However, the performance also improves over a single thread because of the ktask threads' NUMA awareness (ktask migrates worker threads to the node local to the work being done). This becomes a bigger factor as the amount of pages to zero grows to include memory from multiple nodes, so that speedups increase as the size increases. Signed-off-by: Daniel Jordan --- mm/memory.c | 32 ++++++++++++++++++++++++-------- 1 file changed, 24 insertions(+), 8 deletions(-) diff --git a/mm/memory.c b/mm/memory.c index 15c417e8e31d..445d06537905 100644 --- a/mm/memory.c +++ b/mm/memory.c @@ -69,6 +69,7 @@ #include #include #include +#include #include #include @@ -4415,19 +4416,28 @@ static inline void process_huge_page( } } -static void clear_gigantic_page(struct page *page, - unsigned long addr, - unsigned int pages_per_huge_page) +struct cgp_args { + struct page *base_page; + unsigned long addr; +}; + +static int clear_gigantic_page_chunk(unsigned long start, unsigned long end, + struct cgp_args *args) { - int i; - struct page *p = page; + struct page *base_page = args->base_page; + struct page *p = base_page; + unsigned long addr = args->addr; + unsigned long i; might_sleep(); - for (i = 0; i < pages_per_huge_page; - i++, p = mem_map_next(p, page, i)) { + for (i = start; i < end; ++i) { cond_resched(); clear_user_highpage(p, addr + i * PAGE_SIZE); + + p = mem_map_next(p, base_page, i); } + + return KTASK_RETURN_SUCCESS; } static void clear_subpage(unsigned long addr, int idx, void *arg) @@ -4444,7 +4454,13 @@ void clear_huge_page(struct page *page, ~(((unsigned long)pages_per_huge_page << PAGE_SHIFT) - 1); if (unlikely(pages_per_huge_page > MAX_ORDER_NR_PAGES)) { - clear_gigantic_page(page, addr, pages_per_huge_page); + struct cgp_args args = {page, addr}; + struct ktask_node node = {0, pages_per_huge_page, + page_to_nid(page)}; + DEFINE_KTASK_CTL(ctl, clear_gigantic_page_chunk, &args, + KTASK_MEM_CHUNK); + + ktask_run_numa(&node, 1, &ctl); return; }