From patchwork Tue Nov 20 03:31:19 2018 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Wei Yang X-Patchwork-Id: 10689843 Return-Path: Received: from mail.wl.linuxfoundation.org (pdx-wl-mail.web.codeaurora.org [172.30.200.125]) by pdx-korg-patchwork-2.web.codeaurora.org (Postfix) with ESMTP id C2D085A4 for ; Tue, 20 Nov 2018 03:31:35 +0000 (UTC) Received: from mail.wl.linuxfoundation.org (localhost [127.0.0.1]) by mail.wl.linuxfoundation.org (Postfix) with ESMTP id B01472A4DE for ; Tue, 20 Nov 2018 03:31:35 +0000 (UTC) Received: by mail.wl.linuxfoundation.org (Postfix, from userid 486) id A2BF52A4E9; Tue, 20 Nov 2018 03:31:35 +0000 (UTC) X-Spam-Checker-Version: SpamAssassin 3.3.1 (2010-03-16) on pdx-wl-mail.web.codeaurora.org X-Spam-Level: X-Spam-Status: No, score=-3.0 required=2.0 tests=BAYES_00,DKIM_SIGNED, DKIM_VALID,DKIM_VALID_AU,FREEMAIL_FROM,MAILING_LIST_MULTI,RCVD_IN_DNSWL_NONE autolearn=ham version=3.3.1 Received: from kanga.kvack.org (kanga.kvack.org [205.233.56.17]) by mail.wl.linuxfoundation.org (Postfix) with ESMTP id 5E61C2A4DE for ; Tue, 20 Nov 2018 03:31:34 +0000 (UTC) Received: by kanga.kvack.org (Postfix) id 2C0FB6B1DF3; Mon, 19 Nov 2018 22:31:33 -0500 (EST) Delivered-To: linux-mm-outgoing@kvack.org Received: by kanga.kvack.org (Postfix, from userid 40) id 270B26B1DF4; Mon, 19 Nov 2018 22:31:33 -0500 (EST) X-Original-To: int-list-linux-mm@kvack.org X-Delivered-To: int-list-linux-mm@kvack.org Received: by kanga.kvack.org (Postfix, from userid 63042) id 139606B1DF5; Mon, 19 Nov 2018 22:31:33 -0500 (EST) X-Original-To: linux-mm@kvack.org X-Delivered-To: linux-mm@kvack.org Received: from mail-pf1-f200.google.com (mail-pf1-f200.google.com [209.85.210.200]) by kanga.kvack.org (Postfix) with ESMTP id C5FBF6B1DF3 for ; Mon, 19 Nov 2018 22:31:32 -0500 (EST) Received: by mail-pf1-f200.google.com with SMTP id 190-v6so527340pfd.7 for ; Mon, 19 Nov 2018 19:31:32 -0800 (PST) X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20161025; h=x-gm-message-state:dkim-signature:from:to:cc:subject:date :message-id:in-reply-to:references; bh=oACUhRhBMm9GvHof1u4eRHQSk5RVEsksLt+Uy1z3OU8=; b=Gb7uqlfvQQc5SDlEOCFdJd8zYM9cb7TfXn9Utr66rHjSwPt4obWfP+dib3DLHfJcSG KMJtYP9tALfNoD+OBUVvnsciKOBrvv/WBmcbFs4yQRDqbgA/xzQoTys0+BXsiRrKP/bY 2qpRGx9n4Q8R/f/9SKnTF8IyAtkRQNx9kTDJmMEYn5LnRESpDVsuiuL1GG3pAA3wPNjD B6afZme8NfGrSnx6dqUDFbQLaCeHQLlv46OPdcc7eAdCwqFLXMWhoxH5w847K8+IvuoF WBuKbu+miv3Wv+NmxABwHXuux7pi8bsPZjcPGLMPI4MINSaV37MicOEizD4pXqz1IPnu dy0Q== X-Gm-Message-State: AGRZ1gI00Vm4A1vbvoU2z+lLHVq+8GdLtlizStD8w5wtxHEf8y1/9hoR 5Qh+7H33Oc8kfIM+SJjDFoQZVAmw2cWK9rdp9anP+vX/TYRdgmYmHidnaHI1+6DojtYO85rMoAS Fk+ZZ5mSg7qVvM2Pqug9Pzr2lA4m8GY2g/l+DEcYnxEI3qbSIHvAYtkz7ZR28RLjoO37WKOXhzr xTx+BMCxm8SpaYCFu+BAxqHs4XhiOp6oOMKKxsWHwxSSdravQ3eJFFzD0XHPT9WcX5oqDaABfGD GyXbaJlm6238jWpbKag/0AIrvn+yVse+QrV85AcRBB+J5Hc2JIBMpPoyLUj/KXkerAcEfQnXSjz lGK/C/UBhFeErNOi+KatHWTJ0rG9FL4T5WQ5ZCaLQ07EnyAc7Bt7fHgwFEoAOxqTau35cm4PB6y 9 X-Received: by 2002:a62:6085:: with SMTP id u127-v6mr424912pfb.147.1542684692272; Mon, 19 Nov 2018 19:31:32 -0800 (PST) X-Received: by 2002:a62:6085:: with SMTP id u127-v6mr424841pfb.147.1542684691279; Mon, 19 Nov 2018 19:31:31 -0800 (PST) ARC-Seal: i=1; a=rsa-sha256; t=1542684691; cv=none; d=google.com; s=arc-20160816; b=yycecdQ45cymnmi29L1x4EskSbP39HkURZTuZPFRZso/g6DzvKOjlmfMgmr51MUFOP z0XRcbpd4Sh35K/f2TszIxri0BZ8vpLC51Fdx+R+loVQQtce+OTNrg0pKQ67EJN6CQMw VVfQGDiBZF8M3kDUR+rLzWwpmeQPyvVex6GLUrMTA/UNa4qUpc8/V2O3GNcCjHcwpnwD U+jKlshUmqYnYRNRKL44wseg900c6DBHvoSCtbKz+8ehmDrlTFfTlB9V8ZFjEiOXbmTh 5chnILu7sBkVjjpSFd9h9SGBZ4cd76x0hgJ3wdK1alu4f357NwK2gOmZPMQnxrPMd6vH CrHQ== ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=google.com; s=arc-20160816; h=references:in-reply-to:message-id:date:subject:cc:to:from :dkim-signature; bh=oACUhRhBMm9GvHof1u4eRHQSk5RVEsksLt+Uy1z3OU8=; b=h6AgUqtbQ+bXfajxpG4djuZwzSoMKs3wYocF/aZofrvmF9o4PZkR5fhH65oJLNbD5u +mQ5TJSAEhD1eogE/cZ9XueDb91ujGKOZ04VwbH0x8xtzNxYBEVjMH2oZxwdoppFsTL9 UKtSdB+Da4OtRLKYEcerPaXgsYbKtcf3hBudEVcKfqqoRJeqJnlMg1XRaDfofR6ANGnv 3sualaQn4rR0JUoWDF1KlxhTixHiXQQhRQXqgo2nqtNbOBh2KKZn2y2xt4eg1fcXB6KC z9ubLWueIqVUQGr0cLkvqhrQoO0RbZgAUn1r0YC26cIf4P3rbAJyzy6nakgOPAjenLIH jgpw== ARC-Authentication-Results: i=1; mx.google.com; dkim=pass header.i=@gmail.com header.s=20161025 header.b=SNSJHPyb; spf=pass (google.com: domain of richard.weiyang@gmail.com designates 209.85.220.41 as permitted sender) smtp.mailfrom=richard.weiyang@gmail.com; dmarc=pass (p=NONE sp=QUARANTINE dis=NONE) header.from=gmail.com Received: from mail-sor-f41.google.com (mail-sor-f41.google.com. [209.85.220.41]) by mx.google.com with SMTPS id f21sor46540866pgm.40.2018.11.19.19.31.31 for (Google Transport Security); Mon, 19 Nov 2018 19:31:31 -0800 (PST) Received-SPF: pass (google.com: domain of richard.weiyang@gmail.com designates 209.85.220.41 as permitted sender) client-ip=209.85.220.41; Authentication-Results: mx.google.com; dkim=pass header.i=@gmail.com header.s=20161025 header.b=SNSJHPyb; spf=pass (google.com: domain of richard.weiyang@gmail.com designates 209.85.220.41 as permitted sender) smtp.mailfrom=richard.weiyang@gmail.com; dmarc=pass (p=NONE sp=QUARANTINE dis=NONE) header.from=gmail.com DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=20161025; h=from:to:cc:subject:date:message-id:in-reply-to:references; bh=oACUhRhBMm9GvHof1u4eRHQSk5RVEsksLt+Uy1z3OU8=; b=SNSJHPybailxE2Wz5YfNkJ4ugXwq/aecGUl4d3j+VVxL7Fgo52+x5j9JYhKX0z64R2 jJ0Cy3p83eg+7U3CbVN7KEGOmWAEYJWvZ4kvhi0SU1K/zZMX5vdzXshWPWrh3+EEIjO6 C/jXUYq7QCencz1SWA99AzDTD0NO0IQVtrT6YpJTVsTV013UJffmw4oNvleLh04V4HSk PvVh2um5Q2diD3IZx7DGh5rKUyE29/Um87m5V24lpmVNP5shJuAMLH+fwdb+xHrHas5F RIEQjX5aXpjvfxEC2llzWVSrctUyEbwp7Udnir0uKhXFE3/G8HVi8PONYRw+NJEAjii+ 0J2A== X-Google-Smtp-Source: AJdET5csDSzEzpzu0xE+GY9xJzDngn0Aul6X26NOnOp66Ebyh1jZa64tyX+t7a2jexL+me4vuuSZ+g== X-Received: by 2002:a63:101d:: with SMTP id f29mr390600pgl.38.1542684690755; Mon, 19 Nov 2018 19:31:30 -0800 (PST) Received: from localhost ([185.92.221.13]) by smtp.gmail.com with ESMTPSA id u78sm56773215pfi.2.2018.11.19.19.31.29 (version=TLS1_2 cipher=ECDHE-RSA-AES128-GCM-SHA256 bits=128/128); Mon, 19 Nov 2018 19:31:30 -0800 (PST) From: Wei Yang To: cl@linux.com, penberg@kernel.org, mhocko@kernel.org, akpm@linux-foundation.org Cc: linux-mm@kvack.org, Wei Yang Subject: [PATCH v2] mm/slub: improve performance by skipping checked node in get_any_partial() Date: Tue, 20 Nov 2018 11:31:19 +0800 Message-Id: <20181120033119.30013-1-richard.weiyang@gmail.com> X-Mailer: git-send-email 2.15.1 In-Reply-To: <20181108011204.9491-1-richard.weiyang@gmail.com> References: <20181108011204.9491-1-richard.weiyang@gmail.com> X-Bogosity: Ham, tests=bogofilter, spamicity=0.000000, version=1.2.4 Sender: owner-linux-mm@kvack.org Precedence: bulk X-Loop: owner-majordomo@kvack.org List-ID: X-Virus-Scanned: ClamAV using ClamSMTP 1. Background Current slub has three layers: * cpu_slab * percpu_partial * per node partial list Slub allocator tries to get an object from top to bottom. When it can't get an object from the upper two layers, it will search the per node partial list. The is done in get_partial(). The abstraction of get_partial() may looks like this: get_partial() get_partial_node() get_any_partial() for_each_zone_zonelist() The idea behind this is: it first try a local node, then try other nodes if caller doesn't specify a node. 2. Room for Improvement When we look one step deeper in get_any_partial(), it tries to get a proper node by for_each_zone_zonelist(), which iterates on the node_zonelists. This behavior would introduce some redundant check on the same node. Because: * the local node is already checked in get_partial_node() * one node may have several zones on node_zonelists 3. Solution Proposed in Patch We could reduce these redundant check by record the last unsuccessful node and then skip it. 4. Tests & Result After some tests, the result shows this may improve the system a little, especially on a machine with only one node. 4.1 Test Description There are two cases for two system configurations. Test Cases: 1. counter comparison 2. kernel build test System Configuration: 1. One node machine with 4G 2. Four node machine with 8G 4.2 Result for Test 1 Test 1: counter comparison This is a test with hacked kernel to record times function get_any_partial() is invoked and times the inner loop iterates. By comparing the ratio of two counters, we get to know how many inner loops we skipped. Here is a snip of the test patch. --- static void *get_any_partial() { get_partial_count++; do { for_each_zone_zonelist() { get_partial_try_count++; } } while(); return NULL; } --- The result of (get_partial_count / get_partial_try_count): +----------+----------------+------------+-------------+ | | Base | Patched | Improvement| +----------+----------------+------------+-------------+ |One Node | 1:3 | 1:0 | - 100% | +----------+----------------+------------+-------------+ |Four Nodes| 1:5.8 | 1:2.5 | - 56% | +----------+----------------+------------+-------------+ 4.3 Result for Test 2 Test 2: kernel build Command used: > time make -j8 bzImage Each version/system configuration combination has four round kernel build tests. Take the average result of real to compare. +----------+----------------+------------+-------------+ | | Base | Patched | Improvement| +----------+----------------+------------+-------------+ |One Node | 4m41s | 4m32s | - 4.47% | +----------+----------------+------------+-------------+ |Four Nodes| 4m45s | 4m39s | - 2.92% | +----------+----------------+------------+-------------+ Signed-off-by: Wei Yang Reviewed-by: Wei Yang Signed-off-by: Wei Yang Signed-off-by: Andrew Morton --- v3: * replace nmask with except to reduce potential stack overflow and copy overhead * test this in two cases and two system configurations and list the result v2: * rewrite the changelog and add a comment based on Andrew's comment --- mm/slub.c | 14 ++++++++++++-- 1 file changed, 12 insertions(+), 2 deletions(-) diff --git a/mm/slub.c b/mm/slub.c index e3629cd7aff1..3d93a07d86d9 100644 --- a/mm/slub.c +++ b/mm/slub.c @@ -1873,7 +1873,7 @@ static void *get_partial_node(struct kmem_cache *s, struct kmem_cache_node *n, * Get a page from somewhere. Search in increasing NUMA distances. */ static void *get_any_partial(struct kmem_cache *s, gfp_t flags, - struct kmem_cache_cpu *c) + struct kmem_cache_cpu *c, int except) { #ifdef CONFIG_NUMA struct zonelist *zonelist; @@ -1911,6 +1911,9 @@ static void *get_any_partial(struct kmem_cache *s, gfp_t flags, for_each_zone_zonelist(zone, z, zonelist, high_zoneidx) { struct kmem_cache_node *n; + if (except == zone_to_nid(zone)) + continue; + n = get_node(s, zone_to_nid(zone)); if (n && cpuset_zone_allowed(zone, flags) && @@ -1927,6 +1930,13 @@ static void *get_any_partial(struct kmem_cache *s, gfp_t flags, return object; } } + /* + * Fail to get object from this node, either because + * 1. Fails in if check + * 2. NULl object returns from get_partial_node() + * Skip it next time. + */ + except = zone_to_nid(zone); } } while (read_mems_allowed_retry(cpuset_mems_cookie)); #endif @@ -1951,7 +1961,7 @@ static void *get_partial(struct kmem_cache *s, gfp_t flags, int node, if (object || node != NUMA_NO_NODE) return object; - return get_any_partial(s, flags, c); + return get_any_partial(s, flags, c, searchnode); } #ifdef CONFIG_PREEMPT