From patchwork Tue Jul 31 09:06:44 2018 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Vlastimil Babka X-Patchwork-Id: 10550493 Return-Path: Received: from mail.wl.linuxfoundation.org (pdx-wl-mail.web.codeaurora.org [172.30.200.125]) by pdx-korg-patchwork-2.web.codeaurora.org (Postfix) with ESMTP id 65EBF13BF for ; Tue, 31 Jul 2018 09:07:17 +0000 (UTC) Received: from mail.wl.linuxfoundation.org (localhost [127.0.0.1]) by mail.wl.linuxfoundation.org (Postfix) with ESMTP id 534A829FB4 for ; Tue, 31 Jul 2018 09:07:17 +0000 (UTC) Received: by mail.wl.linuxfoundation.org (Postfix, from userid 486) id 472C22A2EC; Tue, 31 Jul 2018 09:07:17 +0000 (UTC) X-Spam-Checker-Version: SpamAssassin 3.3.1 (2010-03-16) on pdx-wl-mail.web.codeaurora.org X-Spam-Level: X-Spam-Status: No, score=-2.9 required=2.0 tests=BAYES_00,MAILING_LIST_MULTI, RCVD_IN_DNSWL_NONE autolearn=ham version=3.3.1 Received: from kanga.kvack.org (kanga.kvack.org [205.233.56.17]) by mail.wl.linuxfoundation.org (Postfix) with ESMTP id D4AFA29FB4 for ; Tue, 31 Jul 2018 09:07:15 +0000 (UTC) Received: by kanga.kvack.org (Postfix) id 2A2766B000C; Tue, 31 Jul 2018 05:07:04 -0400 (EDT) Delivered-To: linux-mm-outgoing@kvack.org Received: by kanga.kvack.org (Postfix, from userid 40) id 1DBC06B026A; Tue, 31 Jul 2018 05:07:04 -0400 (EDT) X-Original-To: int-list-linux-mm@kvack.org X-Delivered-To: int-list-linux-mm@kvack.org Received: by kanga.kvack.org (Postfix, from userid 63042) id DE6906B026E; Tue, 31 Jul 2018 05:07:03 -0400 (EDT) X-Original-To: linux-mm@kvack.org X-Delivered-To: linux-mm@kvack.org Received: from mail-pl0-f72.google.com (mail-pl0-f72.google.com [209.85.160.72]) by kanga.kvack.org (Postfix) with ESMTP id 77B336B026C for ; Tue, 31 Jul 2018 05:07:03 -0400 (EDT) Received: by mail-pl0-f72.google.com with SMTP id d10-v6so10880306pll.22 for ; Tue, 31 Jul 2018 02:07:03 -0700 (PDT) X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20161025; h=x-original-authentication-results:x-gm-message-state:from:to:cc :subject:date:message-id:in-reply-to:references; bh=oiOFvoby39JPIqFNnjyYrLC6KUl+yJOIMXOwq30jedA=; b=CUdiLQUF+1Q4VWxe5i0322UAjTqBZoe4VunS0DPWqmWfC0uxtqa+NgH2Ie9TfoTSZB tf4muwKsEE/IMRQl/F+vp5ql1StzycVGET77u9TXdMr9sdhj68tDiToUpGvEw7KNe1mW VMnhC4XuzYp/lVGA19vUQ//rYhd2rOE+RXdPfG1lCf37pp8iz1K5CzP3MLq9AY58JYnF wbAzeGdDa3Jsw3QKRbQ0V5GA3u6mGSN86DTMMo1L9I8hfCMKzy3nlRzt3pX0jx3yNM+z JZ42jjbVQFgSFCnnWJ8x4HVUlkBtTXjjHg/dhTsrhCM775kk2CQCJPA+YycDVygeT1lg eh2Q== X-Original-Authentication-Results: mx.google.com; spf=pass (google.com: domain of vbabka@suse.cz designates 195.135.220.15 as permitted sender) smtp.mailfrom=vbabka@suse.cz X-Gm-Message-State: AOUpUlGAElEfMt/azQs6lA9LZmnkIxGqyIzMowcEfsoPj5qEmbMR9RbY njY5gt0wZhdcwhqJ49UIWcysJlHdVWU7CEqxQKEQnuE9fV8H0Vp1ciUL1m4oaS61BI5n4leOLnR fjIB7nbSNtqiDhod7bDDtf/cDVrjJud+fxqf8eqmJo5cMBjUCBGPWubU7ZT5n4+Ioiw== X-Received: by 2002:a17:902:bc8b:: with SMTP id bb11-v6mr19293794plb.112.1533028023078; Tue, 31 Jul 2018 02:07:03 -0700 (PDT) X-Google-Smtp-Source: AAOMgpeZNF51bffKOV0QiUF/mIFgLDClAR7pyMM+OAEz923ArDGbBucDFzwa0buqOOgx3EHHGpJD X-Received: by 2002:a17:902:bc8b:: with SMTP id bb11-v6mr19293736plb.112.1533028021781; Tue, 31 Jul 2018 02:07:01 -0700 (PDT) ARC-Seal: i=1; a=rsa-sha256; t=1533028021; cv=none; d=google.com; s=arc-20160816; b=XEz8AuuQmdrYXbbdYQcnUoS+wVsWW6rLVeLOrq5kccR2E1aCkBPhj6McyFQroNBLNg 28+gSXcdrvjQB8TyEMcuLYLXVrfr3MswqCfZuys5iBPkcg0XelAmI3uSXQaY3W45JUjJ SXQpY5mXeJ1hPE7lvF/rewNjL4QZBZrMqo4/evJ9Vp/SSW6f0XyBcqBDZDu3QfEOY2uK VSPGa3IEIcNkdlFB3+Mlq7mZIJdCYCmBJZQwe8309LamEA2foXG1psM8LW2NJrLgdPOi E/Y8nVDPFUijfHfgJYTi1L26y943IA2Rqfe+9/twTjaIm2caeBSmy16OxuFrFtBqbA7Y X5ww== ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=google.com; s=arc-20160816; h=references:in-reply-to:message-id:date:subject:cc:to:from :arc-authentication-results; bh=oiOFvoby39JPIqFNnjyYrLC6KUl+yJOIMXOwq30jedA=; b=tz9mWdMp0i1ZN9TM6PDTnDm43fogJWaBI+O0z9jbVoZeoNHeg6Ik3yJgtwkiU//Kl7 ZBUrHapxZLKpVzLYeQxBBNwuw6mxmd9WOS9DC+qi1KiC6Q0hoNu8tPyA1wES4AaySx5x 6DoE5kY2BMAroxOm6D6+XkRR7j4W5PNyLYnbpbuf1M0JhXirtiI+H+HdIKXnirPUPi3/ JGVUYRl53LmhADNzL+IPqmefkQA3iEuok4x5e02Opa6qn75yj5JuxsWVmvxyQ/HrnJR2 lWzZ/dNUiHwRpH+Tz7tenz3mEKa9Xbok5k2XpPfWDzHN1AwRxxDUufEbnF8KxywYybDq i17A== ARC-Authentication-Results: i=1; mx.google.com; spf=pass (google.com: domain of vbabka@suse.cz designates 195.135.220.15 as permitted sender) smtp.mailfrom=vbabka@suse.cz Received: from mx1.suse.de (mx2.suse.de. [195.135.220.15]) by mx.google.com with ESMTPS id l63-v6si11594650plb.106.2018.07.31.02.07.01 for (version=TLS1_2 cipher=ECDHE-RSA-AES128-GCM-SHA256 bits=128/128); Tue, 31 Jul 2018 02:07:01 -0700 (PDT) Received-SPF: pass (google.com: domain of vbabka@suse.cz designates 195.135.220.15 as permitted sender) client-ip=195.135.220.15; Authentication-Results: mx.google.com; spf=pass (google.com: domain of vbabka@suse.cz designates 195.135.220.15 as permitted sender) smtp.mailfrom=vbabka@suse.cz X-Virus-Scanned: by amavisd-new at test-mx.suse.de Received: from relay2.suse.de (unknown [195.135.220.254]) by mx1.suse.de (Postfix) with ESMTP id 8C153ACD6; Tue, 31 Jul 2018 09:06:58 +0000 (UTC) From: Vlastimil Babka To: Andrew Morton Cc: linux-mm@kvack.org, linux-kernel@vger.kernel.org, linux-api@vger.kernel.org, Roman Gushchin , Michal Hocko , Johannes Weiner , Christoph Lameter , David Rientjes , Joonsoo Kim , Mel Gorman , Matthew Wilcox , Vlastimil Babka Subject: [PATCH v4 1/6] mm, slab: combine kmalloc_caches and kmalloc_dma_caches Date: Tue, 31 Jul 2018 11:06:44 +0200 Message-Id: <20180731090649.16028-2-vbabka@suse.cz> X-Mailer: git-send-email 2.18.0 In-Reply-To: <20180731090649.16028-1-vbabka@suse.cz> References: <20180731090649.16028-1-vbabka@suse.cz> X-Bogosity: Ham, tests=bogofilter, spamicity=0.000000, version=1.2.4 Sender: owner-linux-mm@kvack.org Precedence: bulk X-Loop: owner-majordomo@kvack.org List-ID: X-Virus-Scanned: ClamAV using ClamSMTP The kmalloc caches currently mainain separate (optional) array kmalloc_dma_caches for __GFP_DMA allocations. There are tests for __GFP_DMA in the allocation hotpaths. We can avoid the branches by combining kmalloc_caches and kmalloc_dma_caches into a single two-dimensional array where the outer dimension is cache "type". This will also allow to add kmalloc-reclaimable caches as a third type. Signed-off-by: Vlastimil Babka Acked-by: Mel Gorman Acked-by: Christoph Lameter Acked-by: Roman Gushchin --- include/linux/slab.h | 42 +++++++++++++++++++++++++++++++----------- mm/slab.c | 4 ++-- mm/slab_common.c | 31 ++++++++++++------------------- mm/slub.c | 13 +++++++------ 4 files changed, 52 insertions(+), 38 deletions(-) diff --git a/include/linux/slab.h b/include/linux/slab.h index 14e3fe4bd6a1..f35f2c1f37b9 100644 --- a/include/linux/slab.h +++ b/include/linux/slab.h @@ -295,12 +295,29 @@ static inline void __check_heap_object(const void *ptr, unsigned long n, #define SLAB_OBJ_MIN_SIZE (KMALLOC_MIN_SIZE < 16 ? \ (KMALLOC_MIN_SIZE) : 16) +enum kmalloc_cache_type { + KMALLOC_NORMAL = 0, +#ifdef CONFIG_ZONE_DMA + KMALLOC_DMA, +#endif + NR_KMALLOC_TYPES +}; + #ifndef CONFIG_SLOB -extern struct kmem_cache *kmalloc_caches[KMALLOC_SHIFT_HIGH + 1]; +extern struct kmem_cache * +kmalloc_caches[NR_KMALLOC_TYPES][KMALLOC_SHIFT_HIGH + 1]; + +static __always_inline enum kmalloc_cache_type kmalloc_type(gfp_t flags) +{ + int is_dma = 0; + #ifdef CONFIG_ZONE_DMA -extern struct kmem_cache *kmalloc_dma_caches[KMALLOC_SHIFT_HIGH + 1]; + is_dma = !!(flags & __GFP_DMA); #endif + return is_dma; +} + /* * Figure out which kmalloc slab an allocation of a certain size * belongs to. @@ -501,18 +518,20 @@ static __always_inline void *kmalloc_large(size_t size, gfp_t flags) static __always_inline void *kmalloc(size_t size, gfp_t flags) { if (__builtin_constant_p(size)) { +#ifndef CONFIG_SLOB + unsigned int index; +#endif if (size > KMALLOC_MAX_CACHE_SIZE) return kmalloc_large(size, flags); #ifndef CONFIG_SLOB - if (!(flags & GFP_DMA)) { - unsigned int index = kmalloc_index(size); + index = kmalloc_index(size); - if (!index) - return ZERO_SIZE_PTR; + if (!index) + return ZERO_SIZE_PTR; - return kmem_cache_alloc_trace(kmalloc_caches[index], - flags, size); - } + return kmem_cache_alloc_trace( + kmalloc_caches[kmalloc_type(flags)][index], + flags, size); #endif } return __kmalloc(size, flags); @@ -542,13 +561,14 @@ static __always_inline void *kmalloc_node(size_t size, gfp_t flags, int node) { #ifndef CONFIG_SLOB if (__builtin_constant_p(size) && - size <= KMALLOC_MAX_CACHE_SIZE && !(flags & GFP_DMA)) { + size <= KMALLOC_MAX_CACHE_SIZE) { unsigned int i = kmalloc_index(size); if (!i) return ZERO_SIZE_PTR; - return kmem_cache_alloc_node_trace(kmalloc_caches[i], + return kmem_cache_alloc_node_trace( + kmalloc_caches[kmalloc_type(flags)][i], flags, node, size); } #endif diff --git a/mm/slab.c b/mm/slab.c index aa76a70e087e..9515798f37b2 100644 --- a/mm/slab.c +++ b/mm/slab.c @@ -1288,7 +1288,7 @@ void __init kmem_cache_init(void) * Initialize the caches that provide memory for the kmem_cache_node * structures first. Without this, further allocations will bug. */ - kmalloc_caches[INDEX_NODE] = create_kmalloc_cache( + kmalloc_caches[KMALLOC_NORMAL][INDEX_NODE] = create_kmalloc_cache( kmalloc_info[INDEX_NODE].name, kmalloc_size(INDEX_NODE), ARCH_KMALLOC_FLAGS, 0, kmalloc_size(INDEX_NODE)); @@ -1304,7 +1304,7 @@ void __init kmem_cache_init(void) for_each_online_node(nid) { init_list(kmem_cache, &init_kmem_cache_node[CACHE_CACHE + nid], nid); - init_list(kmalloc_caches[INDEX_NODE], + init_list(kmalloc_caches[KMALLOC_NORMAL][INDEX_NODE], &init_kmem_cache_node[SIZE_NODE + nid], nid); } } diff --git a/mm/slab_common.c b/mm/slab_common.c index 2296caf87bfb..4572941440f3 100644 --- a/mm/slab_common.c +++ b/mm/slab_common.c @@ -973,14 +973,10 @@ struct kmem_cache *__init create_kmalloc_cache(const char *name, return s; } -struct kmem_cache *kmalloc_caches[KMALLOC_SHIFT_HIGH + 1] __ro_after_init; +struct kmem_cache * +kmalloc_caches[NR_KMALLOC_TYPES][KMALLOC_SHIFT_HIGH + 1] __ro_after_init; EXPORT_SYMBOL(kmalloc_caches); -#ifdef CONFIG_ZONE_DMA -struct kmem_cache *kmalloc_dma_caches[KMALLOC_SHIFT_HIGH + 1] __ro_after_init; -EXPORT_SYMBOL(kmalloc_dma_caches); -#endif - /* * Conversion table for small slabs sizes / 8 to the index in the * kmalloc array. This is necessary for slabs < 192 since we have non power @@ -1040,12 +1036,7 @@ struct kmem_cache *kmalloc_slab(size_t size, gfp_t flags) } else index = fls(size - 1); -#ifdef CONFIG_ZONE_DMA - if (unlikely((flags & GFP_DMA))) - return kmalloc_dma_caches[index]; - -#endif - return kmalloc_caches[index]; + return kmalloc_caches[kmalloc_type(flags)][index]; } /* @@ -1119,7 +1110,8 @@ void __init setup_kmalloc_cache_index_table(void) static void __init new_kmalloc_cache(int idx, slab_flags_t flags) { - kmalloc_caches[idx] = create_kmalloc_cache(kmalloc_info[idx].name, + kmalloc_caches[KMALLOC_NORMAL][idx] = create_kmalloc_cache( + kmalloc_info[idx].name, kmalloc_info[idx].size, flags, 0, kmalloc_info[idx].size); } @@ -1132,9 +1124,10 @@ static void __init new_kmalloc_cache(int idx, slab_flags_t flags) void __init create_kmalloc_caches(slab_flags_t flags) { int i; + int type = KMALLOC_NORMAL; for (i = KMALLOC_SHIFT_LOW; i <= KMALLOC_SHIFT_HIGH; i++) { - if (!kmalloc_caches[i]) + if (!kmalloc_caches[type][i]) new_kmalloc_cache(i, flags); /* @@ -1142,9 +1135,9 @@ void __init create_kmalloc_caches(slab_flags_t flags) * These have to be created immediately after the * earlier power of two caches */ - if (KMALLOC_MIN_SIZE <= 32 && !kmalloc_caches[1] && i == 6) + if (KMALLOC_MIN_SIZE <= 32 && !kmalloc_caches[type][1] && i == 6) new_kmalloc_cache(1, flags); - if (KMALLOC_MIN_SIZE <= 64 && !kmalloc_caches[2] && i == 7) + if (KMALLOC_MIN_SIZE <= 64 && !kmalloc_caches[type][2] && i == 7) new_kmalloc_cache(2, flags); } @@ -1153,7 +1146,7 @@ void __init create_kmalloc_caches(slab_flags_t flags) #ifdef CONFIG_ZONE_DMA for (i = 0; i <= KMALLOC_SHIFT_HIGH; i++) { - struct kmem_cache *s = kmalloc_caches[i]; + struct kmem_cache *s = kmalloc_caches[KMALLOC_NORMAL][i]; if (s) { unsigned int size = kmalloc_size(i); @@ -1161,8 +1154,8 @@ void __init create_kmalloc_caches(slab_flags_t flags) "dma-kmalloc-%u", size); BUG_ON(!n); - kmalloc_dma_caches[i] = create_kmalloc_cache(n, - size, SLAB_CACHE_DMA | flags, 0, 0); + kmalloc_caches[KMALLOC_DMA][i] = create_kmalloc_cache( + n, size, SLAB_CACHE_DMA | flags, 0, 0); } } #endif diff --git a/mm/slub.c b/mm/slub.c index 51258eff4178..a7b4657ea8e0 100644 --- a/mm/slub.c +++ b/mm/slub.c @@ -4659,6 +4659,7 @@ static int list_locations(struct kmem_cache *s, char *buf, static void __init resiliency_test(void) { u8 *p; + int type = KMALLOC_NORMAL; BUILD_BUG_ON(KMALLOC_MIN_SIZE > 16 || KMALLOC_SHIFT_HIGH < 10); @@ -4671,7 +4672,7 @@ static void __init resiliency_test(void) pr_err("\n1. kmalloc-16: Clobber Redzone/next pointer 0x12->0x%p\n\n", p + 16); - validate_slab_cache(kmalloc_caches[4]); + validate_slab_cache(kmalloc_caches[type][4]); /* Hmmm... The next two are dangerous */ p = kzalloc(32, GFP_KERNEL); @@ -4680,33 +4681,33 @@ static void __init resiliency_test(void) p); pr_err("If allocated object is overwritten then not detectable\n\n"); - validate_slab_cache(kmalloc_caches[5]); + validate_slab_cache(kmalloc_caches[type][5]); p = kzalloc(64, GFP_KERNEL); p += 64 + (get_cycles() & 0xff) * sizeof(void *); *p = 0x56; pr_err("\n3. kmalloc-64: corrupting random byte 0x56->0x%p\n", p); pr_err("If allocated object is overwritten then not detectable\n\n"); - validate_slab_cache(kmalloc_caches[6]); + validate_slab_cache(kmalloc_caches[type][6]); pr_err("\nB. Corruption after free\n"); p = kzalloc(128, GFP_KERNEL); kfree(p); *p = 0x78; pr_err("1. kmalloc-128: Clobber first word 0x78->0x%p\n\n", p); - validate_slab_cache(kmalloc_caches[7]); + validate_slab_cache(kmalloc_caches[type][7]); p = kzalloc(256, GFP_KERNEL); kfree(p); p[50] = 0x9a; pr_err("\n2. kmalloc-256: Clobber 50th byte 0x9a->0x%p\n\n", p); - validate_slab_cache(kmalloc_caches[8]); + validate_slab_cache(kmalloc_caches[type][8]); p = kzalloc(512, GFP_KERNEL); kfree(p); p[512] = 0xab; pr_err("\n3. kmalloc-512: Clobber redzone 0xab->0x%p\n\n", p); - validate_slab_cache(kmalloc_caches[9]); + validate_slab_cache(kmalloc_caches[type][9]); } #else #ifdef CONFIG_SYSFS From patchwork Tue Jul 31 09:06:45 2018 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Vlastimil Babka X-Patchwork-Id: 10550489 Return-Path: Received: from mail.wl.linuxfoundation.org (pdx-wl-mail.web.codeaurora.org [172.30.200.125]) by pdx-korg-patchwork-2.web.codeaurora.org (Postfix) with ESMTP id 68CF8139A for ; Tue, 31 Jul 2018 09:07:10 +0000 (UTC) Received: from mail.wl.linuxfoundation.org (localhost [127.0.0.1]) by mail.wl.linuxfoundation.org (Postfix) with ESMTP id 5674F29FB4 for ; Tue, 31 Jul 2018 09:07:10 +0000 (UTC) Received: by mail.wl.linuxfoundation.org (Postfix, from userid 486) id 4A8682A206; Tue, 31 Jul 2018 09:07:10 +0000 (UTC) X-Spam-Checker-Version: SpamAssassin 3.3.1 (2010-03-16) on pdx-wl-mail.web.codeaurora.org X-Spam-Level: X-Spam-Status: No, score=-2.9 required=2.0 tests=BAYES_00,MAILING_LIST_MULTI, RCVD_IN_DNSWL_NONE autolearn=ham version=3.3.1 Received: from kanga.kvack.org (kanga.kvack.org [205.233.56.17]) by mail.wl.linuxfoundation.org (Postfix) with ESMTP id A82FB29FB4 for ; Tue, 31 Jul 2018 09:07:09 +0000 (UTC) Received: by kanga.kvack.org (Postfix) id BC4196B0266; Tue, 31 Jul 2018 05:07:03 -0400 (EDT) Delivered-To: linux-mm-outgoing@kvack.org Received: by kanga.kvack.org (Postfix, from userid 40) id 92CC36B000C; Tue, 31 Jul 2018 05:07:03 -0400 (EDT) X-Original-To: int-list-linux-mm@kvack.org X-Delivered-To: int-list-linux-mm@kvack.org Received: by kanga.kvack.org (Postfix, from userid 63042) id 77EB66B026A; Tue, 31 Jul 2018 05:07:03 -0400 (EDT) X-Original-To: linux-mm@kvack.org X-Delivered-To: linux-mm@kvack.org Received: from mail-pl0-f72.google.com (mail-pl0-f72.google.com [209.85.160.72]) by kanga.kvack.org (Postfix) with ESMTP id 239796B0010 for ; Tue, 31 Jul 2018 05:07:03 -0400 (EDT) Received: by mail-pl0-f72.google.com with SMTP id az8-v6so10944511plb.15 for ; Tue, 31 Jul 2018 02:07:03 -0700 (PDT) X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20161025; h=x-original-authentication-results:x-gm-message-state:from:to:cc :subject:date:message-id:in-reply-to:references; bh=VZYOiAD3wrtrXYcfZKeldGvXRLaSYRD8JPqMQHNBua4=; b=iJOVbx+2fWqXTLx2Q2P6ZYsipCTko2yHPFPEFeW0a8lyS6c1dYX/UOWOk8JmdV31mn EPA2cyE7eb632c/x9DDUssH2m8Ge3Ga+JxF6sh2S/TZcwnm93rhdsO5ozThFPfYRjY3P 2ASeODXduCIDvJiSrDwWulQ12gM0bdCz1SEQl/9Jk4gMCQKudFDDustnTFqzc74RryQ3 PVSDFUjUohfhf9t/TdjGGmrlynfPFmaFb+MCqbJFCLH1sWq89Bj+Ccip+QahyAAHsso+ noLyyALFh3nyBA2Vq/AVAXfUV0nBq9o+qmpCT9zx9gz8G7syScp5Nm1Wz2BicLBQs65X lvrQ== X-Original-Authentication-Results: mx.google.com; spf=pass (google.com: domain of vbabka@suse.cz designates 195.135.220.15 as permitted sender) smtp.mailfrom=vbabka@suse.cz X-Gm-Message-State: AOUpUlFMz5xBwBqPJZny8DamP7K0t61ARV0GvIbADimOdVq/FfLj43wQ tfve7vYoNdyaT8rwGOcqjDr2vRfFL79gMm5rQYKk4BP2lTxRmRuLh0HDLTbHD2Y677QzOyPScJi NfvXTr4zU7VXkFK6Lkl4+j+kkzc2XhIRktF0il1FhJD8YNVeVq3Ut2Hewvu5SP4oJcQ== X-Received: by 2002:a62:2459:: with SMTP id r86-v6mr21242963pfj.31.1533028022807; Tue, 31 Jul 2018 02:07:02 -0700 (PDT) X-Google-Smtp-Source: AAOMgpcOn9aG4t4kaBnKBM9RbfPP7Q0tGA0wFMGXRQdRhIsDftSEH67uALHFZS1OgsbO2qDgIvLy X-Received: by 2002:a62:2459:: with SMTP id r86-v6mr21242914pfj.31.1533028021758; Tue, 31 Jul 2018 02:07:01 -0700 (PDT) ARC-Seal: i=1; a=rsa-sha256; t=1533028021; cv=none; d=google.com; s=arc-20160816; b=s5YCaaWfVlHJnvfZLG1OBBHg1cHOI+bkQlOUHkAe7iYvDlanJUFfU0QFDBe7xW7zWo ce5NhiDI6KjprSurT2WPXAs4z5172NQgRyg2sWQvls9EidLhFBC0+strzUU1VKmWZSVk zjYaQkox2P9wcxkYO+UCxtbo75gFoqXCktd5GKW5h3TUnckZTHHDWcdwFNaa/Z3ZWz69 UCtY8xAt8XrxR6ZtqoAX9DXUL3JhrR43dZAoB3D1kt3g+KBNGy+LLAA+OVM/G3as++4Z fZgscmfLsC+oxvOGi/d6897r/1+663P1Gi/nRZt2nYZ+LkSagbOtA3g+YyFNBopcGDfS 4eSw== ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=google.com; s=arc-20160816; h=references:in-reply-to:message-id:date:subject:cc:to:from :arc-authentication-results; bh=VZYOiAD3wrtrXYcfZKeldGvXRLaSYRD8JPqMQHNBua4=; b=WsJBq2M4c4TPsRBI1c4tK6gBP3ou8/OB2RNWnpw8pckPj0RdvidYt7WhrO35DWfok+ QEDO5cnIms1oqN8mCzOKm6r1a/6UOAhwiW9dqLch55YcwvTfn/BoQMT2K8ICIPxyxSH1 mn2ZBVmwQsJYPNvp787IxCuN8qWgeAQixJAcUIFNL5nw0Rve1Pjy53TyTwGsAWIQ0FK8 LUubBgRQ6CpssLNg4VuBUJpE4BxIoITCwkEADE/BpEeHo2VZ4lsH6uwZP2RKPdpkvbvJ mSOhu0dEc/ZZX8f+Gz4b/p3Kyb8aT8g5N7n5HOrLPNNVY/lQKbSmf66XDA3GDVTQrkUy /oww== ARC-Authentication-Results: i=1; mx.google.com; spf=pass (google.com: domain of vbabka@suse.cz designates 195.135.220.15 as permitted sender) smtp.mailfrom=vbabka@suse.cz Received: from mx1.suse.de (mx2.suse.de. [195.135.220.15]) by mx.google.com with ESMTPS id i21-v6si13452255pgg.513.2018.07.31.02.07.01 for (version=TLS1_2 cipher=ECDHE-RSA-AES128-GCM-SHA256 bits=128/128); Tue, 31 Jul 2018 02:07:01 -0700 (PDT) Received-SPF: pass (google.com: domain of vbabka@suse.cz designates 195.135.220.15 as permitted sender) client-ip=195.135.220.15; Authentication-Results: mx.google.com; spf=pass (google.com: domain of vbabka@suse.cz designates 195.135.220.15 as permitted sender) smtp.mailfrom=vbabka@suse.cz X-Virus-Scanned: by amavisd-new at test-mx.suse.de Received: from relay1.suse.de (unknown [195.135.220.254]) by mx1.suse.de (Postfix) with ESMTP id 8CF10AE1E; Tue, 31 Jul 2018 09:06:58 +0000 (UTC) From: Vlastimil Babka To: Andrew Morton Cc: linux-mm@kvack.org, linux-kernel@vger.kernel.org, linux-api@vger.kernel.org, Roman Gushchin , Michal Hocko , Johannes Weiner , Christoph Lameter , David Rientjes , Joonsoo Kim , Mel Gorman , Matthew Wilcox , Vlastimil Babka Subject: [PATCH v4 2/6] mm, slab/slub: introduce kmalloc-reclaimable caches Date: Tue, 31 Jul 2018 11:06:45 +0200 Message-Id: <20180731090649.16028-3-vbabka@suse.cz> X-Mailer: git-send-email 2.18.0 In-Reply-To: <20180731090649.16028-1-vbabka@suse.cz> References: <20180731090649.16028-1-vbabka@suse.cz> X-Bogosity: Ham, tests=bogofilter, spamicity=0.000000, version=1.2.4 Sender: owner-linux-mm@kvack.org Precedence: bulk X-Loop: owner-majordomo@kvack.org List-ID: X-Virus-Scanned: ClamAV using ClamSMTP Kmem caches can be created with a SLAB_RECLAIM_ACCOUNT flag, which indicates they contain objects which can be reclaimed under memory pressure (typically through a shrinker). This makes the slab pages accounted as NR_SLAB_RECLAIMABLE in vmstat, which is reflected also the MemAvailable meminfo counter and in overcommit decisions. The slab pages are also allocated with __GFP_RECLAIMABLE, which is good for anti-fragmentation through grouping pages by mobility. The generic kmalloc-X caches are created without this flag, but sometimes are used also for objects that can be reclaimed, which due to varying size cannot have a dedicated kmem cache with SLAB_RECLAIM_ACCOUNT flag. A prominent example are dcache external names, which prompted the creation of a new, manually managed vmstat counter NR_INDIRECTLY_RECLAIMABLE_BYTES in commit f1782c9bc547 ("dcache: account external names as indirectly reclaimable memory"). To better handle this and any other similar cases, this patch introduces SLAB_RECLAIM_ACCOUNT variants of kmalloc caches, named kmalloc-rcl-X. They are used whenever the kmalloc() call passes __GFP_RECLAIMABLE among gfp flags. They are added to the kmalloc_caches array as a new type. Allocations with both __GFP_DMA and __GFP_RECLAIMABLE will use a dma type cache. This change only applies to SLAB and SLUB, not SLOB. This is fine, since SLOB's target are tiny system and this patch does add some overhead of kmem management objects. Signed-off-by: Vlastimil Babka Acked-by: Mel Gorman Acked-by: Christoph Lameter Acked-by: Roman Gushchin --- include/linux/slab.h | 16 ++++++++++++++- mm/slab_common.c | 48 ++++++++++++++++++++++++++++---------------- 2 files changed, 46 insertions(+), 18 deletions(-) diff --git a/include/linux/slab.h b/include/linux/slab.h index f35f2c1f37b9..b79a7cd59875 100644 --- a/include/linux/slab.h +++ b/include/linux/slab.h @@ -295,8 +295,13 @@ static inline void __check_heap_object(const void *ptr, unsigned long n, #define SLAB_OBJ_MIN_SIZE (KMALLOC_MIN_SIZE < 16 ? \ (KMALLOC_MIN_SIZE) : 16) +/* + * Whenever changing this, take care of that kmalloc_type() and + * create_kmalloc_caches() still work as intended. + */ enum kmalloc_cache_type { KMALLOC_NORMAL = 0, + KMALLOC_RECLAIM, #ifdef CONFIG_ZONE_DMA KMALLOC_DMA, #endif @@ -310,12 +315,21 @@ kmalloc_caches[NR_KMALLOC_TYPES][KMALLOC_SHIFT_HIGH + 1]; static __always_inline enum kmalloc_cache_type kmalloc_type(gfp_t flags) { int is_dma = 0; + int type_dma = 0; + int is_reclaimable; #ifdef CONFIG_ZONE_DMA is_dma = !!(flags & __GFP_DMA); + type_dma = is_dma * KMALLOC_DMA; #endif - return is_dma; + is_reclaimable = !!(flags & __GFP_RECLAIMABLE); + + /* + * If an allocation is both __GFP_DMA and __GFP_RECLAIMABLE, return + * KMALLOC_DMA and effectively ignore __GFP_RECLAIMABLE + */ + return type_dma + (is_reclaimable & !is_dma) * KMALLOC_RECLAIM; } /* diff --git a/mm/slab_common.c b/mm/slab_common.c index 4572941440f3..03f40b273ea3 100644 --- a/mm/slab_common.c +++ b/mm/slab_common.c @@ -1108,10 +1108,21 @@ void __init setup_kmalloc_cache_index_table(void) } } -static void __init new_kmalloc_cache(int idx, slab_flags_t flags) +static void __init +new_kmalloc_cache(int idx, int type, slab_flags_t flags) { - kmalloc_caches[KMALLOC_NORMAL][idx] = create_kmalloc_cache( - kmalloc_info[idx].name, + const char *name; + + if (type == KMALLOC_RECLAIM) { + flags |= SLAB_RECLAIM_ACCOUNT; + name = kasprintf(GFP_NOWAIT, "kmalloc-rcl-%u", + kmalloc_info[idx].size); + BUG_ON(!name); + } else { + name = kmalloc_info[idx].name; + } + + kmalloc_caches[type][idx] = create_kmalloc_cache(name, kmalloc_info[idx].size, flags, 0, kmalloc_info[idx].size); } @@ -1123,22 +1134,25 @@ static void __init new_kmalloc_cache(int idx, slab_flags_t flags) */ void __init create_kmalloc_caches(slab_flags_t flags) { - int i; - int type = KMALLOC_NORMAL; + int i, type; - for (i = KMALLOC_SHIFT_LOW; i <= KMALLOC_SHIFT_HIGH; i++) { - if (!kmalloc_caches[type][i]) - new_kmalloc_cache(i, flags); + for (type = KMALLOC_NORMAL; type <= KMALLOC_RECLAIM; type++) { + for (i = KMALLOC_SHIFT_LOW; i <= KMALLOC_SHIFT_HIGH; i++) { + if (!kmalloc_caches[type][i]) + new_kmalloc_cache(i, type, flags); - /* - * Caches that are not of the two-to-the-power-of size. - * These have to be created immediately after the - * earlier power of two caches - */ - if (KMALLOC_MIN_SIZE <= 32 && !kmalloc_caches[type][1] && i == 6) - new_kmalloc_cache(1, flags); - if (KMALLOC_MIN_SIZE <= 64 && !kmalloc_caches[type][2] && i == 7) - new_kmalloc_cache(2, flags); + /* + * Caches that are not of the two-to-the-power-of size. + * These have to be created immediately after the + * earlier power of two caches + */ + if (KMALLOC_MIN_SIZE <= 32 && i == 6 && + !kmalloc_caches[type][1]) + new_kmalloc_cache(1, type, flags); + if (KMALLOC_MIN_SIZE <= 64 && i == 7 && + !kmalloc_caches[type][2]) + new_kmalloc_cache(2, type, flags); + } } /* Kmalloc array is now usable */ From patchwork Tue Jul 31 09:06:46 2018 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Vlastimil Babka X-Patchwork-Id: 10550487 Return-Path: Received: from mail.wl.linuxfoundation.org (pdx-wl-mail.web.codeaurora.org [172.30.200.125]) by pdx-korg-patchwork-2.web.codeaurora.org (Postfix) with ESMTP id 94B9E13BF for ; Tue, 31 Jul 2018 09:07:07 +0000 (UTC) Received: from mail.wl.linuxfoundation.org (localhost [127.0.0.1]) by mail.wl.linuxfoundation.org (Postfix) with ESMTP id 82E7629FB4 for ; Tue, 31 Jul 2018 09:07:07 +0000 (UTC) Received: by mail.wl.linuxfoundation.org (Postfix, from userid 486) id 76F362A206; Tue, 31 Jul 2018 09:07:07 +0000 (UTC) X-Spam-Checker-Version: SpamAssassin 3.3.1 (2010-03-16) on pdx-wl-mail.web.codeaurora.org X-Spam-Level: X-Spam-Status: No, score=-2.9 required=2.0 tests=BAYES_00,MAILING_LIST_MULTI, RCVD_IN_DNSWL_NONE autolearn=ham version=3.3.1 Received: from kanga.kvack.org (kanga.kvack.org [205.233.56.17]) by mail.wl.linuxfoundation.org (Postfix) with ESMTP id 0504C29FB4 for ; Tue, 31 Jul 2018 09:07:06 +0000 (UTC) Received: by kanga.kvack.org (Postfix) id 93C3D6B026D; Tue, 31 Jul 2018 05:07:03 -0400 (EDT) Delivered-To: linux-mm-outgoing@kvack.org Received: by kanga.kvack.org (Postfix, from userid 40) id 83E186B000E; Tue, 31 Jul 2018 05:07:03 -0400 (EDT) X-Original-To: int-list-linux-mm@kvack.org X-Delivered-To: int-list-linux-mm@kvack.org Received: by kanga.kvack.org (Postfix, from userid 63042) id 6E0546B026B; Tue, 31 Jul 2018 05:07:03 -0400 (EDT) X-Original-To: linux-mm@kvack.org X-Delivered-To: linux-mm@kvack.org Received: from mail-pf1-f199.google.com (mail-pf1-f199.google.com [209.85.210.199]) by kanga.kvack.org (Postfix) with ESMTP id 153A56B000E for ; Tue, 31 Jul 2018 05:07:03 -0400 (EDT) Received: by mail-pf1-f199.google.com with SMTP id a23-v6so4442203pfo.23 for ; Tue, 31 Jul 2018 02:07:03 -0700 (PDT) X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20161025; h=x-original-authentication-results:x-gm-message-state:from:to:cc :subject:date:message-id:in-reply-to:references; bh=mago/OtTdc+yTn/7CKAknhNyQz4OFjN/yAQZ5QW50oM=; b=TDRd8L3KuKuZ+bAWnJ1tWjJoJZYtUO+bvNmcsqMiYYJVuTkRfXILsOjGNPFSNdfp/g w1J8sMm1I5dIsQaHOkipFAKR4MDMt052gs8m5JJ6fM5GiIwDyFeMmzp3uXaDCWRPp0P+ D7lGqRXi6KGdIsbrwEHKdn80KH8I9yJT+DyUA7/2NAVqb2JCR0osoPmItxtAuSKh2jXO cnJ8i/yC8LOrKCPMlpVAlNZhrLrcwK4q15S4NRw0nvCXpS9ZNLKlcv6rk5zNB71LZmu3 EKaOMzmNqR4R50dxyA0HF76d6XgByFpKmbROcjRG7cjB6jk0tbaXlia4D45BjGaWpK99 Btlg== X-Original-Authentication-Results: mx.google.com; spf=pass (google.com: domain of vbabka@suse.cz designates 195.135.220.15 as permitted sender) smtp.mailfrom=vbabka@suse.cz X-Gm-Message-State: AOUpUlEe0AxtszuxfqigJxo1gvo1y6ARINtRg58M0BkwHq7vK1QRLDAE FlEvYVPnIsptGiA1iW6AAh/hB0FGY4o7Jognip+fpRPvXu0/3PfKbgDhQrOVcP4ZOA0KD90AMXi T0IEBJhOJ6I/JFXl5hUoSWTFhD3i6uePXMLzhsURiHLLJoEzauJqcGnBRdXB/DyYWEw== X-Received: by 2002:a17:902:da4:: with SMTP id 33-v6mr19406661plv.193.1533028022743; Tue, 31 Jul 2018 02:07:02 -0700 (PDT) X-Google-Smtp-Source: AAOMgpfGBiLixYRXm2P0YFXV0ybNIOJLvmD3fATPNAeBoXzxHj8roCDnxKbo8iHzSI3Qs6/U0tT/ X-Received: by 2002:a17:902:da4:: with SMTP id 33-v6mr19406622plv.193.1533028021799; Tue, 31 Jul 2018 02:07:01 -0700 (PDT) ARC-Seal: i=1; a=rsa-sha256; t=1533028021; cv=none; d=google.com; s=arc-20160816; b=vJsVwRwRtWQ6lXAZt3sURwjco25Sb89MhnYAg16QmaNYpOd792NbAuP96aFAGHx5vw m0EVih/uBTEknO6OEpsRvnT8TUXn9aLbNTct+N48HgEDXFOzbseAuknz7u1cXaQNhJpi ZvEvKm6HopO0X7j/kMPh0Rmctq2z49m4unfQl4TQbu06WbawKRF9QfiP1GtZHzHPY4jE KPN1lWak6PSQzFwIRzuumB4WPVYV7j44E11WJ7RBvDDzOtAj11OjEhxgbRE4Too6a21V UdwEP7jBcsjo3cWjzAjwtQZNkE8SI6fnoAdDkVDkcUDC6rBbY8VscDjhUFtkHXI7PeN8 4EIQ== ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=google.com; s=arc-20160816; h=references:in-reply-to:message-id:date:subject:cc:to:from :arc-authentication-results; bh=mago/OtTdc+yTn/7CKAknhNyQz4OFjN/yAQZ5QW50oM=; b=TCysoIFpIUxn5B5oLM9A6QhC3/4xU7QjmgOI5CuWisqFGKORYwzL0CFnAAYettkz4s O1GoBNZpsNcxX+ABQXj4pO0s/tjf7+56TZUihmrsPPuwvdaiNMSinb9GQBNcBweqsakZ drINJ6Tff3TJDLYjQ2ICdsYLLw0EpNVdROoHJc0igA1AnoY7Da8IyRKmLSdu8PBeuTeh xo10uICqy9+xVQyNPYg9UK2kLzPWLF/pKPi6M020RLCxNYPHkgOoNbwE6Eabbdhboh3T f97Jj8M8lf9ymmrVyct4N5EjZSG+BXxOoevJZdoleuWHcuTpexZwPlDZqCOGUaCM8L8R k+1Q== ARC-Authentication-Results: i=1; mx.google.com; spf=pass (google.com: domain of vbabka@suse.cz designates 195.135.220.15 as permitted sender) smtp.mailfrom=vbabka@suse.cz Received: from mx1.suse.de (mx2.suse.de. [195.135.220.15]) by mx.google.com with ESMTPS id q3-v6si13459834pgf.40.2018.07.31.02.07.01 for (version=TLS1_2 cipher=ECDHE-RSA-AES128-GCM-SHA256 bits=128/128); Tue, 31 Jul 2018 02:07:01 -0700 (PDT) Received-SPF: pass (google.com: domain of vbabka@suse.cz designates 195.135.220.15 as permitted sender) client-ip=195.135.220.15; Authentication-Results: mx.google.com; spf=pass (google.com: domain of vbabka@suse.cz designates 195.135.220.15 as permitted sender) smtp.mailfrom=vbabka@suse.cz X-Virus-Scanned: by amavisd-new at test-mx.suse.de Received: from relay2.suse.de (unknown [195.135.220.254]) by mx1.suse.de (Postfix) with ESMTP id 8C4ECADBB; Tue, 31 Jul 2018 09:06:58 +0000 (UTC) From: Vlastimil Babka To: Andrew Morton Cc: linux-mm@kvack.org, linux-kernel@vger.kernel.org, linux-api@vger.kernel.org, Roman Gushchin , Michal Hocko , Johannes Weiner , Christoph Lameter , David Rientjes , Joonsoo Kim , Mel Gorman , Matthew Wilcox , Vlastimil Babka Subject: [PATCH v4 3/6] dcache: allocate external names from reclaimable kmalloc caches Date: Tue, 31 Jul 2018 11:06:46 +0200 Message-Id: <20180731090649.16028-4-vbabka@suse.cz> X-Mailer: git-send-email 2.18.0 In-Reply-To: <20180731090649.16028-1-vbabka@suse.cz> References: <20180731090649.16028-1-vbabka@suse.cz> X-Bogosity: Ham, tests=bogofilter, spamicity=0.000000, version=1.2.4 Sender: owner-linux-mm@kvack.org Precedence: bulk X-Loop: owner-majordomo@kvack.org List-ID: X-Virus-Scanned: ClamAV using ClamSMTP We can use the newly introduced kmalloc-reclaimable-X caches, to allocate external names in dcache, which will take care of the proper accounting automatically, and also improve anti-fragmentation page grouping. This effectively reverts commit f1782c9bc547 ("dcache: account external names as indirectly reclaimable memory") and instead passes __GFP_RECLAIMABLE to kmalloc(). The accounting thus moves from NR_INDIRECTLY_RECLAIMABLE_BYTES to NR_SLAB_RECLAIMABLE, which is also considered in MemAvailable calculation and overcommit decisions. Signed-off-by: Vlastimil Babka Acked-by: Mel Gorman Acked-by: Roman Gushchin --- fs/dcache.c | 38 +++++++++----------------------------- 1 file changed, 9 insertions(+), 29 deletions(-) diff --git a/fs/dcache.c b/fs/dcache.c index 0e8e5de3c48a..518c9ed8db8c 100644 --- a/fs/dcache.c +++ b/fs/dcache.c @@ -257,24 +257,10 @@ static void __d_free(struct rcu_head *head) kmem_cache_free(dentry_cache, dentry); } -static void __d_free_external_name(struct rcu_head *head) -{ - struct external_name *name = container_of(head, struct external_name, - u.head); - - mod_node_page_state(page_pgdat(virt_to_page(name)), - NR_INDIRECTLY_RECLAIMABLE_BYTES, - -ksize(name)); - - kfree(name); -} - static void __d_free_external(struct rcu_head *head) { struct dentry *dentry = container_of(head, struct dentry, d_u.d_rcu); - - __d_free_external_name(&external_name(dentry)->u.head); - + kfree(external_name(dentry)); kmem_cache_free(dentry_cache, dentry); } @@ -305,7 +291,7 @@ void release_dentry_name_snapshot(struct name_snapshot *name) struct external_name *p; p = container_of(name->name, struct external_name, name[0]); if (unlikely(atomic_dec_and_test(&p->u.count))) - call_rcu(&p->u.head, __d_free_external_name); + kfree_rcu(p, u.head); } } EXPORT_SYMBOL(release_dentry_name_snapshot); @@ -1608,7 +1594,6 @@ EXPORT_SYMBOL(d_invalidate); struct dentry *__d_alloc(struct super_block *sb, const struct qstr *name) { - struct external_name *ext = NULL; struct dentry *dentry; char *dname; int err; @@ -1629,14 +1614,15 @@ struct dentry *__d_alloc(struct super_block *sb, const struct qstr *name) dname = dentry->d_iname; } else if (name->len > DNAME_INLINE_LEN-1) { size_t size = offsetof(struct external_name, name[1]); - - ext = kmalloc(size + name->len, GFP_KERNEL_ACCOUNT); - if (!ext) { + struct external_name *p = kmalloc(size + name->len, + GFP_KERNEL_ACCOUNT | + __GFP_RECLAIMABLE); + if (!p) { kmem_cache_free(dentry_cache, dentry); return NULL; } - atomic_set(&ext->u.count, 1); - dname = ext->name; + atomic_set(&p->u.count, 1); + dname = p->name; } else { dname = dentry->d_iname; } @@ -1675,12 +1661,6 @@ struct dentry *__d_alloc(struct super_block *sb, const struct qstr *name) } } - if (unlikely(ext)) { - pg_data_t *pgdat = page_pgdat(virt_to_page(ext)); - mod_node_page_state(pgdat, NR_INDIRECTLY_RECLAIMABLE_BYTES, - ksize(ext)); - } - this_cpu_inc(nr_dentry); return dentry; @@ -2761,7 +2741,7 @@ static void copy_name(struct dentry *dentry, struct dentry *target) dentry->d_name.hash_len = target->d_name.hash_len; } if (old_name && likely(atomic_dec_and_test(&old_name->u.count))) - call_rcu(&old_name->u.head, __d_free_external_name); + kfree_rcu(old_name, u.head); } /* From patchwork Tue Jul 31 09:06:47 2018 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Vlastimil Babka X-Patchwork-Id: 10550495 Return-Path: Received: from mail.wl.linuxfoundation.org (pdx-wl-mail.web.codeaurora.org [172.30.200.125]) by pdx-korg-patchwork-2.web.codeaurora.org (Postfix) with ESMTP id E6015139A for ; Tue, 31 Jul 2018 09:07:22 +0000 (UTC) Received: from mail.wl.linuxfoundation.org (localhost [127.0.0.1]) by mail.wl.linuxfoundation.org (Postfix) with ESMTP id D3B1C29FB4 for ; Tue, 31 Jul 2018 09:07:22 +0000 (UTC) Received: by mail.wl.linuxfoundation.org (Postfix, from userid 486) id C7E032A2EC; Tue, 31 Jul 2018 09:07:22 +0000 (UTC) X-Spam-Checker-Version: SpamAssassin 3.3.1 (2010-03-16) on pdx-wl-mail.web.codeaurora.org X-Spam-Level: X-Spam-Status: No, score=-2.9 required=2.0 tests=BAYES_00,MAILING_LIST_MULTI, RCVD_IN_DNSWL_NONE autolearn=ham version=3.3.1 Received: from kanga.kvack.org (kanga.kvack.org [205.233.56.17]) by mail.wl.linuxfoundation.org (Postfix) with ESMTP id 27B1529FB4 for ; Tue, 31 Jul 2018 09:07:22 +0000 (UTC) Received: by kanga.kvack.org (Postfix) id DF4EA6B026C; Tue, 31 Jul 2018 05:07:04 -0400 (EDT) Delivered-To: linux-mm-outgoing@kvack.org Received: by kanga.kvack.org (Postfix, from userid 40) id D49AD6B026B; Tue, 31 Jul 2018 05:07:04 -0400 (EDT) X-Original-To: int-list-linux-mm@kvack.org X-Delivered-To: int-list-linux-mm@kvack.org Received: by kanga.kvack.org (Postfix, from userid 63042) id BEAAD6B026E; Tue, 31 Jul 2018 05:07:04 -0400 (EDT) X-Original-To: linux-mm@kvack.org X-Delivered-To: linux-mm@kvack.org Received: from mail-pg1-f199.google.com (mail-pg1-f199.google.com [209.85.215.199]) by kanga.kvack.org (Postfix) with ESMTP id 77A426B026A for ; Tue, 31 Jul 2018 05:07:04 -0400 (EDT) Received: by mail-pg1-f199.google.com with SMTP id a26-v6so9003968pgw.7 for ; Tue, 31 Jul 2018 02:07:04 -0700 (PDT) X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20161025; h=x-original-authentication-results:x-gm-message-state:from:to:cc :subject:date:message-id:in-reply-to:references; bh=Zmu2pax0++6UTwp6O2vWwFt4kT16ClgiEPaq5JXs28w=; b=Yt4A4Vj0KrF12oB5JcYawut405r/gVV2/OF72miPj0dJsHDiZVZHPlSyxMetyFXFuI 8SoqjFMMPx/J6/0WhAduKnv+0g5Pxegcw5DKbs3rjxsedaxfIXK6nQhNR/hAncGADfrn +LS63pX7SvIeUEefpvnf89NKtSoNfwP9d7Sy7LU9FcGU0pMHN9GUT5tKF4sUkDXIY75O QCHvkRGwhbCXyDezbT7HAlmKWWqpMapmZxoGz6XHus/kmGcXB88IKQ7CVO96K5kcmLEV KTTuqdW1HaIi7tmk5EVolvbMaiGD5aUSfFX0B6iTz5NKMunqHWtrMDSCDhm5HApyHLrD u8bg== X-Original-Authentication-Results: mx.google.com; spf=pass (google.com: domain of vbabka@suse.cz designates 195.135.220.15 as permitted sender) smtp.mailfrom=vbabka@suse.cz X-Gm-Message-State: AOUpUlH5OaS7xL247VQilWTG+B1pZKsX+Hu8Rj0BOR0dv+hUtzl4S2o0 Sj/1p8Bnh6fpzBRq4tjElIiPY/62IJy4cqf7COa1PSDO3f+CUzp6jd4MNMQ/bCuOp0WjcH14z5O 2dNZ+255CB8SBD//WgHCjykE5eQQShoterFwd0LL3tSZdvwXBqKgNhV4FCVy6XZjMcA== X-Received: by 2002:a62:a649:: with SMTP id t70-v6mr21434535pfe.149.1533028024163; Tue, 31 Jul 2018 02:07:04 -0700 (PDT) X-Google-Smtp-Source: AAOMgpc00rAL3Mu3hxWfJ6/wZy2/BhkCJTWMmjtIPFt89GXbqlqNL5KoP45zuAY8PXW2dBsIKiJ2 X-Received: by 2002:a62:a649:: with SMTP id t70-v6mr21434495pfe.149.1533028023355; Tue, 31 Jul 2018 02:07:03 -0700 (PDT) ARC-Seal: i=1; a=rsa-sha256; t=1533028023; cv=none; d=google.com; s=arc-20160816; b=tEIvduERRYlblBxCPAhXc43LwoqUbleuo2GR2GeqG2mgqtqwbcYQ+otOLKtubphd6x NkuhvkgsZMuT2O1XZICZBm1k/7uA72gzAB+XzZTlCXzOKPrv/xw5mhhL2cGdF2DIfLWv 4wdHTg9HnbtUjkIoy1Ij/dRVtgGy4ZyrMM5jjxmUXXxkENbYc1hixuKoS/pNbgFdocp4 qE+OnRfprqYis16RKJowge7kdizRRd2jXIV/XMLbahaHyaPLF4k/rD5IpiNlAIbj7m8S kEhcbPwqNMu1Mc1O6PrhFMWPRwKbcQrzgYwLS7zLJoaxhG1LSyvYiflTPsuXFD6bxRa+ 8NEQ== ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=google.com; s=arc-20160816; h=references:in-reply-to:message-id:date:subject:cc:to:from :arc-authentication-results; bh=Zmu2pax0++6UTwp6O2vWwFt4kT16ClgiEPaq5JXs28w=; b=fP5sHfCyy+SakcSEpGFKDHVmbTmBnThoELPv45p3c7Hvti0JYS9jkb4n8R7sx1CZri iefZFhYAe/bLGYywnQ0YJ/4Xhzn7k3BQe5xiI61pM+90zhePtPUKwXMjmuzC2xhSCOfJ oCy6wXgR6KSKf0yktPA1HvdtYpkbG8/McjKlE3dQuRadWNvNO2AzJFnGCxZEpzWHiDUB KYTroaOji/Ns9FD+ToEl6+YAdCfUwOa60X8u6IUNL2bzwusNHIn2lNUDn1jdrp8Opc/e ARu74wuliN68NqlzRPKjYZRJ75kaw7cd24LI5qDNhDKtA35ECID7kdhuGV5cx3qC/drj u4Mg== ARC-Authentication-Results: i=1; mx.google.com; spf=pass (google.com: domain of vbabka@suse.cz designates 195.135.220.15 as permitted sender) smtp.mailfrom=vbabka@suse.cz Received: from mx1.suse.de (mx2.suse.de. [195.135.220.15]) by mx.google.com with ESMTPS id m193-v6si13532380pfc.312.2018.07.31.02.07.02 for (version=TLS1_2 cipher=ECDHE-RSA-AES128-GCM-SHA256 bits=128/128); Tue, 31 Jul 2018 02:07:03 -0700 (PDT) Received-SPF: pass (google.com: domain of vbabka@suse.cz designates 195.135.220.15 as permitted sender) client-ip=195.135.220.15; Authentication-Results: mx.google.com; spf=pass (google.com: domain of vbabka@suse.cz designates 195.135.220.15 as permitted sender) smtp.mailfrom=vbabka@suse.cz X-Virus-Scanned: by amavisd-new at test-mx.suse.de Received: from relay1.suse.de (unknown [195.135.220.254]) by mx1.suse.de (Postfix) with ESMTP id 90331AE6D; Tue, 31 Jul 2018 09:06:58 +0000 (UTC) From: Vlastimil Babka To: Andrew Morton Cc: linux-mm@kvack.org, linux-kernel@vger.kernel.org, linux-api@vger.kernel.org, Roman Gushchin , Michal Hocko , Johannes Weiner , Christoph Lameter , David Rientjes , Joonsoo Kim , Mel Gorman , Matthew Wilcox , Vlastimil Babka , Vijayanand Jitta , Laura Abbott , Sumit Semwal Subject: [PATCH v4 4/6] mm: rename and change semantics of nr_indirectly_reclaimable_bytes Date: Tue, 31 Jul 2018 11:06:47 +0200 Message-Id: <20180731090649.16028-5-vbabka@suse.cz> X-Mailer: git-send-email 2.18.0 In-Reply-To: <20180731090649.16028-1-vbabka@suse.cz> References: <20180731090649.16028-1-vbabka@suse.cz> X-Bogosity: Ham, tests=bogofilter, spamicity=0.000000, version=1.2.4 Sender: owner-linux-mm@kvack.org Precedence: bulk X-Loop: owner-majordomo@kvack.org List-ID: X-Virus-Scanned: ClamAV using ClamSMTP The vmstat counter NR_INDIRECTLY_RECLAIMABLE_BYTES was introduced by commit eb59254608bc ("mm: introduce NR_INDIRECTLY_RECLAIMABLE_BYTES") with the goal of accounting objects that can be reclaimed, but cannot be allocated via a SLAB_RECLAIM_ACCOUNT cache. This is now possible via kmalloc() with __GFP_RECLAIMABLE flag, and the dcache external names user is converted. The counter is however still useful for accounting direct page allocations (i.e. not slab) with a shrinker, such as the ION page pool. So keep it, and: - change granularity to pages to be more like other counters; sub-page allocations should be able to use kmalloc - rename the counter to NR_KERNEL_MISC_RECLAIMABLE - expose the counter again in vmstat as "nr_kernel_misc_reclaimable"; we can again remove the check for not printing "hidden" counters Signed-off-by: Vlastimil Babka Cc: Vijayanand Jitta Cc: Laura Abbott Cc: Sumit Semwal Acked-by: Christoph Lameter Acked-by: Roman Gushchin --- drivers/staging/android/ion/ion_page_pool.c | 8 ++++---- include/linux/mmzone.h | 2 +- mm/page_alloc.c | 19 +++++++------------ mm/util.c | 3 +-- mm/vmstat.c | 6 +----- 5 files changed, 14 insertions(+), 24 deletions(-) diff --git a/drivers/staging/android/ion/ion_page_pool.c b/drivers/staging/android/ion/ion_page_pool.c index 9bc56eb48d2a..0d2a95957ee8 100644 --- a/drivers/staging/android/ion/ion_page_pool.c +++ b/drivers/staging/android/ion/ion_page_pool.c @@ -33,8 +33,8 @@ static void ion_page_pool_add(struct ion_page_pool *pool, struct page *page) pool->low_count++; } - mod_node_page_state(page_pgdat(page), NR_INDIRECTLY_RECLAIMABLE_BYTES, - (1 << (PAGE_SHIFT + pool->order))); + mod_node_page_state(page_pgdat(page), NR_KERNEL_MISC_RECLAIMABLE, + 1 << pool->order); mutex_unlock(&pool->mutex); } @@ -53,8 +53,8 @@ static struct page *ion_page_pool_remove(struct ion_page_pool *pool, bool high) } list_del(&page->lru); - mod_node_page_state(page_pgdat(page), NR_INDIRECTLY_RECLAIMABLE_BYTES, - -(1 << (PAGE_SHIFT + pool->order))); + mod_node_page_state(page_pgdat(page), NR_KERNEL_MISC_RECLAIMABLE, + -(1 << pool->order)); return page; } diff --git a/include/linux/mmzone.h b/include/linux/mmzone.h index 32699b2dc52a..c2f6bc4c9e8a 100644 --- a/include/linux/mmzone.h +++ b/include/linux/mmzone.h @@ -180,7 +180,7 @@ enum node_stat_item { NR_VMSCAN_IMMEDIATE, /* Prioritise for reclaim when writeback ends */ NR_DIRTIED, /* page dirtyings since bootup */ NR_WRITTEN, /* page writings since bootup */ - NR_INDIRECTLY_RECLAIMABLE_BYTES, /* measured in bytes */ + NR_KERNEL_MISC_RECLAIMABLE, /* reclaimable non-slab kernel pages */ NR_VM_NODE_STAT_ITEMS }; diff --git a/mm/page_alloc.c b/mm/page_alloc.c index 5d800d61ddb7..91f75bf4404d 100644 --- a/mm/page_alloc.c +++ b/mm/page_alloc.c @@ -4704,6 +4704,7 @@ long si_mem_available(void) unsigned long pagecache; unsigned long wmark_low = 0; unsigned long pages[NR_LRU_LISTS]; + unsigned long reclaimable; struct zone *zone; int lru; @@ -4729,19 +4730,13 @@ long si_mem_available(void) available += pagecache; /* - * Part of the reclaimable slab consists of items that are in use, - * and cannot be freed. Cap this estimate at the low watermark. + * Part of the reclaimable slab and other kernel memory consists of + * items that are in use, and cannot be freed. Cap this estimate at the + * low watermark. */ - available += global_node_page_state(NR_SLAB_RECLAIMABLE) - - min(global_node_page_state(NR_SLAB_RECLAIMABLE) / 2, - wmark_low); - - /* - * Part of the kernel memory, which can be released under memory - * pressure. - */ - available += global_node_page_state(NR_INDIRECTLY_RECLAIMABLE_BYTES) >> - PAGE_SHIFT; + reclaimable = global_node_page_state(NR_SLAB_RECLAIMABLE) + + global_node_page_state(NR_KERNEL_MISC_RECLAIMABLE); + available += reclaimable - min(reclaimable / 2, wmark_low); if (available < 0) available = 0; diff --git a/mm/util.c b/mm/util.c index 3351659200e6..891f0654e7b5 100644 --- a/mm/util.c +++ b/mm/util.c @@ -675,8 +675,7 @@ int __vm_enough_memory(struct mm_struct *mm, long pages, int cap_sys_admin) * Part of the kernel memory, which can be released * under memory pressure. */ - free += global_node_page_state( - NR_INDIRECTLY_RECLAIMABLE_BYTES) >> PAGE_SHIFT; + free += global_node_page_state(NR_KERNEL_MISC_RECLAIMABLE); /* * Leave reserved pages. The pages are not for anonymous pages. diff --git a/mm/vmstat.c b/mm/vmstat.c index 8ba0870ecddd..c5e52f94ba5f 100644 --- a/mm/vmstat.c +++ b/mm/vmstat.c @@ -1161,7 +1161,7 @@ const char * const vmstat_text[] = { "nr_vmscan_immediate_reclaim", "nr_dirtied", "nr_written", - "", /* nr_indirectly_reclaimable */ + "nr_kernel_misc_reclaimable", /* enum writeback_stat_item counters */ "nr_dirty_threshold", @@ -1704,10 +1704,6 @@ static int vmstat_show(struct seq_file *m, void *arg) unsigned long *l = arg; unsigned long off = l - (unsigned long *)m->private; - /* Skip hidden vmstat items. */ - if (*vmstat_text[off] == '\0') - return 0; - seq_puts(m, vmstat_text[off]); seq_put_decimal_ull(m, " ", *l); seq_putc(m, '\n'); From patchwork Tue Jul 31 09:06:48 2018 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Vlastimil Babka X-Patchwork-Id: 10550491 Return-Path: Received: from mail.wl.linuxfoundation.org (pdx-wl-mail.web.codeaurora.org [172.30.200.125]) by pdx-korg-patchwork-2.web.codeaurora.org (Postfix) with ESMTP id 8692613BF for ; Tue, 31 Jul 2018 09:07:13 +0000 (UTC) Received: from mail.wl.linuxfoundation.org (localhost [127.0.0.1]) by mail.wl.linuxfoundation.org (Postfix) with ESMTP id 73C4229FB4 for ; Tue, 31 Jul 2018 09:07:13 +0000 (UTC) Received: by mail.wl.linuxfoundation.org (Postfix, from userid 486) id 67E222A206; Tue, 31 Jul 2018 09:07:13 +0000 (UTC) X-Spam-Checker-Version: SpamAssassin 3.3.1 (2010-03-16) on pdx-wl-mail.web.codeaurora.org X-Spam-Level: X-Spam-Status: No, score=-2.9 required=2.0 tests=BAYES_00,MAILING_LIST_MULTI, RCVD_IN_DNSWL_NONE autolearn=ham version=3.3.1 Received: from kanga.kvack.org (kanga.kvack.org [205.233.56.17]) by mail.wl.linuxfoundation.org (Postfix) with ESMTP id C2BD529FB4 for ; Tue, 31 Jul 2018 09:07:12 +0000 (UTC) Received: by kanga.kvack.org (Postfix) id ED2576B0010; Tue, 31 Jul 2018 05:07:03 -0400 (EDT) Delivered-To: linux-mm-outgoing@kvack.org Received: by kanga.kvack.org (Postfix, from userid 40) id EA8846B000C; Tue, 31 Jul 2018 05:07:03 -0400 (EDT) X-Original-To: int-list-linux-mm@kvack.org X-Delivered-To: int-list-linux-mm@kvack.org Received: by kanga.kvack.org (Postfix, from userid 63042) id A3A4C6B0010; Tue, 31 Jul 2018 05:07:03 -0400 (EDT) X-Original-To: linux-mm@kvack.org X-Delivered-To: linux-mm@kvack.org Received: from mail-pf1-f200.google.com (mail-pf1-f200.google.com [209.85.210.200]) by kanga.kvack.org (Postfix) with ESMTP id 470436B0266 for ; Tue, 31 Jul 2018 05:07:03 -0400 (EDT) Received: by mail-pf1-f200.google.com with SMTP id u13-v6so2916985pfm.8 for ; Tue, 31 Jul 2018 02:07:03 -0700 (PDT) X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20161025; h=x-original-authentication-results:x-gm-message-state:from:to:cc :subject:date:message-id:in-reply-to:references; bh=mW6+9YTAWwathwufRY0fsrPlMQhXnCW1ekmWuTUyq7w=; b=fIkdImGbwnKI/1CJABK2bmHJfJjnxcLFVMBhpkifrbvRVdngEgxCyOLzQLjsVIkM1W hJ0zQr0UJ9+8QotcA7QLzxK2kSILFKpQMrkm0nW5sMcL7fwoNQkHX/QNi0ruXuiP4iZs GR4vovfBNEINsnZZFGU1J+F8YnH4yshdywKs0chehZNm1tzTu+16Xrf6HbllpblI4y+J qgmIhhkE/dzk0yao+UsJvN+oCQE8XkuWRhNadZPhf9QQKAUyl/mi+37A+CBd3qS/1LkT mtNiTKS4XrnScIY396XFXGSHLBUk6FEeiw3Fa+OGVRGLJmDK7E7unnoC7vHgQJ6+tC+2 Ag9A== X-Original-Authentication-Results: mx.google.com; spf=pass (google.com: domain of vbabka@suse.cz designates 195.135.220.15 as permitted sender) smtp.mailfrom=vbabka@suse.cz X-Gm-Message-State: AOUpUlE4ko04NX/i2QbXULVdx6WWzHOgI61AH7CbKZutPN1i5re7W5m5 s06O5FWo7ugZP6KD0RW4QV2WbdK//9+oR88UfT3qoPpdqh+jGU9F08Hmjb6Gt7vDjgclDSEYybr 2mK6465X5uXHJdRiYfSdBbZe6u5eBlZSC+GFkVrWAqlN6R8Nzu16oCiuy6O5p9x8yyQ== X-Received: by 2002:a17:902:9348:: with SMTP id g8-v6mr19575317plp.302.1533028022848; Tue, 31 Jul 2018 02:07:02 -0700 (PDT) X-Google-Smtp-Source: AAOMgpch1JnJxhMVhhTp28lR86tFWH2L1JfX+AyQOJnpYWrL4kRwWub+gMIfny1nmMcwR4QCh4Pl X-Received: by 2002:a17:902:9348:: with SMTP id g8-v6mr19575262plp.302.1533028021779; Tue, 31 Jul 2018 02:07:01 -0700 (PDT) ARC-Seal: i=1; a=rsa-sha256; t=1533028021; cv=none; d=google.com; s=arc-20160816; b=Z1EPILSAqn0aFV32dyxqD9j32uRsPCAUomE2fgQgjw7FgkcYPtwsm91BlafEs2pC7/ x2SRCYrvpslpqds8RE1mv0xi0t74i0Dvs4b5JX8V5575HJCZz3XCNb5TvVenobuHADe1 l+m5pDqhTUEvCmkVEDQXZYdmIM15GPXmcgD0WBtDKB52gn8RQeuIeuiaGg8KNt1e63ba HztuwGMfe9GKvhdUgPx6FPsIhYBuyBnFJk1ETOztZA+bbtZyE8aSfDoqCaCgZXdEonYm WuajMK0qZs1aDM0mLHIqZwqBFdvxnYOtR/JUuFoJS/5jkvxzRp2HTc9heJd0+G+J8GRw M0sg== ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=google.com; s=arc-20160816; h=references:in-reply-to:message-id:date:subject:cc:to:from :arc-authentication-results; bh=mW6+9YTAWwathwufRY0fsrPlMQhXnCW1ekmWuTUyq7w=; b=FdAY4z4/MVrZtQK4ToOuo8B2nV9W4vQLkciZAa6g1dWcufys8Al1pxEIC5zkt7Jg7g GfKiCSNIxDWdiyRCvj7ZIEuMQbb1YQNzPgwIHB6dEVynlNchQ+Qcb/md1DPHKf980AM0 uwLpfInF+LVVw9r4AbjbeCxFyZIW9CWUqQrMbGuXGQ9uPHOjZqIgDIVn0CwS5WrevKwN ri+Z4qTUPPwuXgZ+Sf7UTLu4+po+80ZZUyAsuxoSzqLWczwXAJ9ow8a8SCw6+dShLT22 xEytV+MAJeMHkyP7JEUuDiIchNwg1E25ZW3zEjznsnXYz4Cv+3U03UH1t4Lu7RcRVS1B mupQ== ARC-Authentication-Results: i=1; mx.google.com; spf=pass (google.com: domain of vbabka@suse.cz designates 195.135.220.15 as permitted sender) smtp.mailfrom=vbabka@suse.cz Received: from mx1.suse.de (mx2.suse.de. [195.135.220.15]) by mx.google.com with ESMTPS id y92-v6si8601870plb.378.2018.07.31.02.07.01 for (version=TLS1_2 cipher=ECDHE-RSA-AES128-GCM-SHA256 bits=128/128); Tue, 31 Jul 2018 02:07:01 -0700 (PDT) Received-SPF: pass (google.com: domain of vbabka@suse.cz designates 195.135.220.15 as permitted sender) client-ip=195.135.220.15; Authentication-Results: mx.google.com; spf=pass (google.com: domain of vbabka@suse.cz designates 195.135.220.15 as permitted sender) smtp.mailfrom=vbabka@suse.cz X-Virus-Scanned: by amavisd-new at test-mx.suse.de Received: from relay2.suse.de (unknown [195.135.220.254]) by mx1.suse.de (Postfix) with ESMTP id 957A4AEC0; Tue, 31 Jul 2018 09:06:58 +0000 (UTC) From: Vlastimil Babka To: Andrew Morton Cc: linux-mm@kvack.org, linux-kernel@vger.kernel.org, linux-api@vger.kernel.org, Roman Gushchin , Michal Hocko , Johannes Weiner , Christoph Lameter , David Rientjes , Joonsoo Kim , Mel Gorman , Matthew Wilcox , Vlastimil Babka Subject: [PATCH v4 5/6] mm, proc: add KReclaimable to /proc/meminfo Date: Tue, 31 Jul 2018 11:06:48 +0200 Message-Id: <20180731090649.16028-6-vbabka@suse.cz> X-Mailer: git-send-email 2.18.0 In-Reply-To: <20180731090649.16028-1-vbabka@suse.cz> References: <20180731090649.16028-1-vbabka@suse.cz> X-Bogosity: Ham, tests=bogofilter, spamicity=0.000000, version=1.2.4 Sender: owner-linux-mm@kvack.org Precedence: bulk X-Loop: owner-majordomo@kvack.org List-ID: X-Virus-Scanned: ClamAV using ClamSMTP The vmstat NR_KERNEL_MISC_RECLAIMABLE counter is for kernel non-slab allocations that can be reclaimed via shrinker. In /proc/meminfo, we can show the sum of all reclaimable kernel allocations (including slab) as "KReclaimable". Add the same counter also to per-node meminfo under /sys With this counter, users will have more complete information about kernel memory usage. Non-slab reclaimable pages (currently just the ION allocator) will not be missing from /proc/meminfo, making users wonder where part of their memory went. More precisely, they already appear in MemAvailable, but without the new counter, it's not obvious why the value in MemAvailable doesn't fully correspond with the sum of other counters participating in it. Signed-off-by: Vlastimil Babka Acked-by: Roman Gushchin --- Documentation/filesystems/proc.txt | 4 ++++ drivers/base/node.c | 19 ++++++++++++------- fs/proc/meminfo.c | 16 ++++++++-------- 3 files changed, 24 insertions(+), 15 deletions(-) diff --git a/Documentation/filesystems/proc.txt b/Documentation/filesystems/proc.txt index 520f6a84cf50..6a255f960ab5 100644 --- a/Documentation/filesystems/proc.txt +++ b/Documentation/filesystems/proc.txt @@ -858,6 +858,7 @@ Writeback: 0 kB AnonPages: 861800 kB Mapped: 280372 kB Shmem: 644 kB +KReclaimable: 168048 kB Slab: 284364 kB SReclaimable: 159856 kB SUnreclaim: 124508 kB @@ -921,6 +922,9 @@ AnonHugePages: Non-file backed huge pages mapped into userspace page tables ShmemHugePages: Memory used by shared memory (shmem) and tmpfs allocated with huge pages ShmemPmdMapped: Shared memory mapped into userspace with huge pages +KReclaimable: Kernel allocations that the kernel will attempt to reclaim + under memory pressure. Includes SReclaimable (below), and other + direct allocations with a shrinker. Slab: in-kernel data structures cache SReclaimable: Part of Slab, that might be reclaimed, such as caches SUnreclaim: Part of Slab, that cannot be reclaimed on memory pressure diff --git a/drivers/base/node.c b/drivers/base/node.c index a5e821d09656..81cef8031eae 100644 --- a/drivers/base/node.c +++ b/drivers/base/node.c @@ -67,8 +67,11 @@ static ssize_t node_read_meminfo(struct device *dev, int nid = dev->id; struct pglist_data *pgdat = NODE_DATA(nid); struct sysinfo i; + unsigned long sreclaimable, sunreclaimable; si_meminfo_node(&i, nid); + sreclaimable = node_page_state(pgdat, NR_SLAB_RECLAIMABLE); + sunreclaimable = node_page_state(pgdat, NR_SLAB_UNRECLAIMABLE); n = sprintf(buf, "Node %d MemTotal: %8lu kB\n" "Node %d MemFree: %8lu kB\n" @@ -118,6 +121,7 @@ static ssize_t node_read_meminfo(struct device *dev, "Node %d NFS_Unstable: %8lu kB\n" "Node %d Bounce: %8lu kB\n" "Node %d WritebackTmp: %8lu kB\n" + "Node %d KReclaimable: %8lu kB\n" "Node %d Slab: %8lu kB\n" "Node %d SReclaimable: %8lu kB\n" "Node %d SUnreclaim: %8lu kB\n" @@ -138,20 +142,21 @@ static ssize_t node_read_meminfo(struct device *dev, nid, K(node_page_state(pgdat, NR_UNSTABLE_NFS)), nid, K(sum_zone_node_page_state(nid, NR_BOUNCE)), nid, K(node_page_state(pgdat, NR_WRITEBACK_TEMP)), - nid, K(node_page_state(pgdat, NR_SLAB_RECLAIMABLE) + - node_page_state(pgdat, NR_SLAB_UNRECLAIMABLE)), - nid, K(node_page_state(pgdat, NR_SLAB_RECLAIMABLE)), + nid, K(sreclaimable + + node_page_state(pgdat, NR_KERNEL_MISC_RECLAIMABLE)), + nid, K(sreclaimable + sunreclaimable), + nid, K(sreclaimable), + nid, K(sunreclaimable) #ifdef CONFIG_TRANSPARENT_HUGEPAGE - nid, K(node_page_state(pgdat, NR_SLAB_UNRECLAIMABLE)), + , nid, K(node_page_state(pgdat, NR_ANON_THPS) * HPAGE_PMD_NR), nid, K(node_page_state(pgdat, NR_SHMEM_THPS) * HPAGE_PMD_NR), nid, K(node_page_state(pgdat, NR_SHMEM_PMDMAPPED) * - HPAGE_PMD_NR)); -#else - nid, K(node_page_state(pgdat, NR_SLAB_UNRECLAIMABLE))); + HPAGE_PMD_NR) #endif + ); n += hugetlb_report_node_meminfo(nid, buf + n); return n; } diff --git a/fs/proc/meminfo.c b/fs/proc/meminfo.c index 2fb04846ed11..61a18477bc07 100644 --- a/fs/proc/meminfo.c +++ b/fs/proc/meminfo.c @@ -37,6 +37,7 @@ static int meminfo_proc_show(struct seq_file *m, void *v) long cached; long available; unsigned long pages[NR_LRU_LISTS]; + unsigned long sreclaimable, sunreclaim; int lru; si_meminfo(&i); @@ -52,6 +53,8 @@ static int meminfo_proc_show(struct seq_file *m, void *v) pages[lru] = global_node_page_state(NR_LRU_BASE + lru); available = si_mem_available(); + sreclaimable = global_node_page_state(NR_SLAB_RECLAIMABLE); + sunreclaim = global_node_page_state(NR_SLAB_UNRECLAIMABLE); show_val_kb(m, "MemTotal: ", i.totalram); show_val_kb(m, "MemFree: ", i.freeram); @@ -93,14 +96,11 @@ static int meminfo_proc_show(struct seq_file *m, void *v) show_val_kb(m, "Mapped: ", global_node_page_state(NR_FILE_MAPPED)); show_val_kb(m, "Shmem: ", i.sharedram); - show_val_kb(m, "Slab: ", - global_node_page_state(NR_SLAB_RECLAIMABLE) + - global_node_page_state(NR_SLAB_UNRECLAIMABLE)); - - show_val_kb(m, "SReclaimable: ", - global_node_page_state(NR_SLAB_RECLAIMABLE)); - show_val_kb(m, "SUnreclaim: ", - global_node_page_state(NR_SLAB_UNRECLAIMABLE)); + show_val_kb(m, "KReclaimable: ", sreclaimable + + global_node_page_state(NR_KERNEL_MISC_RECLAIMABLE)); + show_val_kb(m, "Slab: ", sreclaimable + sunreclaim); + show_val_kb(m, "SReclaimable: ", sreclaimable); + show_val_kb(m, "SUnreclaim: ", sunreclaim); seq_printf(m, "KernelStack: %8lu kB\n", global_zone_page_state(NR_KERNEL_STACK_KB)); show_val_kb(m, "PageTables: ", From patchwork Tue Jul 31 09:06:49 2018 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Vlastimil Babka X-Patchwork-Id: 10550497 Return-Path: Received: from mail.wl.linuxfoundation.org (pdx-wl-mail.web.codeaurora.org [172.30.200.125]) by pdx-korg-patchwork-2.web.codeaurora.org (Postfix) with ESMTP id DD6B6139A for ; Tue, 31 Jul 2018 09:07:26 +0000 (UTC) Received: from mail.wl.linuxfoundation.org (localhost [127.0.0.1]) by mail.wl.linuxfoundation.org (Postfix) with ESMTP id CAE9129FB4 for ; Tue, 31 Jul 2018 09:07:26 +0000 (UTC) Received: by mail.wl.linuxfoundation.org (Postfix, from userid 486) id BEBB12A206; Tue, 31 Jul 2018 09:07:26 +0000 (UTC) X-Spam-Checker-Version: SpamAssassin 3.3.1 (2010-03-16) on pdx-wl-mail.web.codeaurora.org X-Spam-Level: X-Spam-Status: No, score=-2.9 required=2.0 tests=BAYES_00,MAILING_LIST_MULTI, RCVD_IN_DNSWL_NONE autolearn=ham version=3.3.1 Received: from kanga.kvack.org (kanga.kvack.org [205.233.56.17]) by mail.wl.linuxfoundation.org (Postfix) with ESMTP id 965AE29FB4 for ; Tue, 31 Jul 2018 09:07:25 +0000 (UTC) Received: by kanga.kvack.org (Postfix) id 146BA6B026A; Tue, 31 Jul 2018 05:07:05 -0400 (EDT) Delivered-To: linux-mm-outgoing@kvack.org Received: by kanga.kvack.org (Postfix, from userid 40) id 09F6E6B026E; Tue, 31 Jul 2018 05:07:04 -0400 (EDT) X-Original-To: int-list-linux-mm@kvack.org X-Delivered-To: int-list-linux-mm@kvack.org Received: by kanga.kvack.org (Postfix, from userid 63042) id CFABD6B026F; Tue, 31 Jul 2018 05:07:04 -0400 (EDT) X-Original-To: linux-mm@kvack.org X-Delivered-To: linux-mm@kvack.org Received: from mail-pg1-f200.google.com (mail-pg1-f200.google.com [209.85.215.200]) by kanga.kvack.org (Postfix) with ESMTP id 7D45D6B026B for ; Tue, 31 Jul 2018 05:07:04 -0400 (EDT) Received: by mail-pg1-f200.google.com with SMTP id u4-v6so9054741pgr.2 for ; Tue, 31 Jul 2018 02:07:04 -0700 (PDT) X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20161025; h=x-original-authentication-results:x-gm-message-state:from:to:cc :subject:date:message-id:in-reply-to:references; bh=LzPHsjWwFDpL1zC34y1Rxpr3uNE34eqWyprZiH7q9xI=; b=A9fvlGmCfEcC4pml8Rtx8t9rNsHCoVxBJLlbj0DDiagY6bO58ixqcw/KR9/Jg7uL5K AxRDnbiZEaVZg8ayfD+LAF6nYYEEcR5IeNUyCZ0FX+AohuD55qEdp+0Vt0RT1p9VoMf/ HLKkCSVQ0+qdUm7yNMPw4NQtO2GHNWHiccAi0MSdIEQTsQKEFveggQR1i5yTFZ1vuoxH 1FBeEFBrkpj4NHO4PwJUwuXWDKJ75qZcoLdkIdObUXhneOeFGN0v9IJ0AOTb2ammzhKz FsVLKWbj0QE1sj34cYhwcpxbHD+VQJ+o/W85ay+S67ZQDaPbBymkLaZefjNqmST+J8Rw 4BLA== X-Original-Authentication-Results: mx.google.com; spf=pass (google.com: domain of vbabka@suse.cz designates 195.135.220.15 as permitted sender) smtp.mailfrom=vbabka@suse.cz X-Gm-Message-State: AOUpUlEqhSggPx0Rj3p5hdFIttC3028jctj2XY+GesIz5ahcgbcdI6RS /gz78QQnb+yckL277Czb/VzxN7Uo2oshJ18LfyZKCdAOYy6mG/wGwOM67LqX/Yj/jUbIzFeKdeX 5YbpyNYq/+O5Vmb64vZP0sfbOivC+rfYgkiAIX4ezwagcLqay/jON7tKfwcKd20P4BQ== X-Received: by 2002:a62:98d6:: with SMTP id d83-v6mr21612196pfk.186.1533028024200; Tue, 31 Jul 2018 02:07:04 -0700 (PDT) X-Google-Smtp-Source: AAOMgpd7XDJyDIdKVoA+sAG3eUWonIjhOAdHf+4a3bBlv9sLuIgsgkoEDzXMTeNxtim+O0TyOV5D X-Received: by 2002:a62:98d6:: with SMTP id d83-v6mr21612139pfk.186.1533028023390; Tue, 31 Jul 2018 02:07:03 -0700 (PDT) ARC-Seal: i=1; a=rsa-sha256; t=1533028023; cv=none; d=google.com; s=arc-20160816; b=i4RFZX4TI+pY0EkA5zl7X6GDpgo4w/0+x2hRmuxRIU5Wtboewk8f2Mkg1/MS+huAhg z+O7dK7c7p315O+QiSZPtOYsnxgat6/ov1r0Xc86h5UQG4E/LlYYhU0Y0Huapgc1O9NC QUDi9T2KcMIm3ABSW/OqfAZcP1aQuY1z2qU0j5G74kfLjAXlKF7tI0I6LhbskcAfx33N ssQ1H/gfPPpYKX9Rqfm35EW+euD30Q+cd7V6Ow7PFC6GhbBL4dI9CMmbY3bSUU0IZF5w XbJEk9vSr7oDJY7zAaDUaxTNVI0cx4taBNZ7Mzovxl4E3hZx3rqaTEzwyZJK+oOAOKa7 eX4w== ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=google.com; s=arc-20160816; h=references:in-reply-to:message-id:date:subject:cc:to:from :arc-authentication-results; bh=LzPHsjWwFDpL1zC34y1Rxpr3uNE34eqWyprZiH7q9xI=; b=ET7Z/25jzSMG5paAvSuijezfvR4+3R6N1PGJO9EroaC2+yqEmrEBTI2J9n/eRyYQrn /66IhNBWIS0KZVRHca/5R2OcJsZ872nbOlFwrML9SDgfvggup3O9AL4I3VCVTwhgA/uU cRnaWN8JbYjmG/FWPoOZxSEF/B0qLmiReO0LqN8/LqEbu+HNi6nrlqYjZvHZBpyRORtf DrNpQpa1fYzM8L0VMSsz0+mRUEpux0BXuKvcI55daWPBCDuqe7rYMIpkwpLUJRZbn95R 3MS7U6NAgfBgwLvqtDa6e8qbMtvCQVZkC3813Z7HYn38ysVKZuT+4XewcaNbSRfS6VaL DNhg== ARC-Authentication-Results: i=1; mx.google.com; spf=pass (google.com: domain of vbabka@suse.cz designates 195.135.220.15 as permitted sender) smtp.mailfrom=vbabka@suse.cz Received: from mx1.suse.de (mx2.suse.de. [195.135.220.15]) by mx.google.com with ESMTPS id m11-v6si6161325pgk.468.2018.07.31.02.07.02 for (version=TLS1_2 cipher=ECDHE-RSA-AES128-GCM-SHA256 bits=128/128); Tue, 31 Jul 2018 02:07:03 -0700 (PDT) Received-SPF: pass (google.com: domain of vbabka@suse.cz designates 195.135.220.15 as permitted sender) client-ip=195.135.220.15; Authentication-Results: mx.google.com; spf=pass (google.com: domain of vbabka@suse.cz designates 195.135.220.15 as permitted sender) smtp.mailfrom=vbabka@suse.cz X-Virus-Scanned: by amavisd-new at test-mx.suse.de Received: from relay1.suse.de (unknown [195.135.220.254]) by mx1.suse.de (Postfix) with ESMTP id 99E36AECB; Tue, 31 Jul 2018 09:06:58 +0000 (UTC) From: Vlastimil Babka To: Andrew Morton Cc: linux-mm@kvack.org, linux-kernel@vger.kernel.org, linux-api@vger.kernel.org, Roman Gushchin , Michal Hocko , Johannes Weiner , Christoph Lameter , David Rientjes , Joonsoo Kim , Mel Gorman , Matthew Wilcox , Vlastimil Babka Subject: [PATCH v4 6/6] mm, slab: shorten kmalloc cache names for large sizes Date: Tue, 31 Jul 2018 11:06:49 +0200 Message-Id: <20180731090649.16028-7-vbabka@suse.cz> X-Mailer: git-send-email 2.18.0 In-Reply-To: <20180731090649.16028-1-vbabka@suse.cz> References: <20180731090649.16028-1-vbabka@suse.cz> X-Bogosity: Ham, tests=bogofilter, spamicity=0.000000, version=1.2.4 Sender: owner-linux-mm@kvack.org Precedence: bulk X-Loop: owner-majordomo@kvack.org List-ID: X-Virus-Scanned: ClamAV using ClamSMTP Kmalloc cache names can get quite long for large object sizes, when the sizes are expressed in bytes. Use 'k' and 'M' prefixes to make the names as short as possible e.g. in /proc/slabinfo. This works, as we mostly use power-of-two sizes, with exceptions only below 1k. Example: 'kmalloc-4194304' becomes 'kmalloc-4M' Suggested-by: Matthew Wilcox Signed-off-by: Vlastimil Babka Acked-by: Mel Gorman Acked-by: Christoph Lameter Acked-by: Roman Gushchin --- mm/slab_common.c | 38 ++++++++++++++++++++++++++------------ 1 file changed, 26 insertions(+), 12 deletions(-) diff --git a/mm/slab_common.c b/mm/slab_common.c index 03f40b273ea3..a07fcb2551f6 100644 --- a/mm/slab_common.c +++ b/mm/slab_common.c @@ -1050,15 +1050,15 @@ const struct kmalloc_info_struct kmalloc_info[] __initconst = { {"kmalloc-16", 16}, {"kmalloc-32", 32}, {"kmalloc-64", 64}, {"kmalloc-128", 128}, {"kmalloc-256", 256}, {"kmalloc-512", 512}, - {"kmalloc-1024", 1024}, {"kmalloc-2048", 2048}, - {"kmalloc-4096", 4096}, {"kmalloc-8192", 8192}, - {"kmalloc-16384", 16384}, {"kmalloc-32768", 32768}, - {"kmalloc-65536", 65536}, {"kmalloc-131072", 131072}, - {"kmalloc-262144", 262144}, {"kmalloc-524288", 524288}, - {"kmalloc-1048576", 1048576}, {"kmalloc-2097152", 2097152}, - {"kmalloc-4194304", 4194304}, {"kmalloc-8388608", 8388608}, - {"kmalloc-16777216", 16777216}, {"kmalloc-33554432", 33554432}, - {"kmalloc-67108864", 67108864} + {"kmalloc-1k", 1024}, {"kmalloc-2k", 2048}, + {"kmalloc-4k", 4096}, {"kmalloc-8k", 8192}, + {"kmalloc-16k", 16384}, {"kmalloc-32k", 32768}, + {"kmalloc-64k", 65536}, {"kmalloc-128k", 131072}, + {"kmalloc-256k", 262144}, {"kmalloc-512k", 524288}, + {"kmalloc-1M", 1048576}, {"kmalloc-2M", 2097152}, + {"kmalloc-4M", 4194304}, {"kmalloc-8M", 8388608}, + {"kmalloc-16M", 16777216}, {"kmalloc-32M", 33554432}, + {"kmalloc-64M", 67108864} }; /* @@ -1108,6 +1108,21 @@ void __init setup_kmalloc_cache_index_table(void) } } +static const char * +kmalloc_cache_name(const char *prefix, unsigned int size) +{ + + static const char units[3] = "\0kM"; + int idx = 0; + + while (size >= 1024 && (size % 1024 == 0)) { + size /= 1024; + idx++; + } + + return kasprintf(GFP_NOWAIT, "%s-%u%c", prefix, size, units[idx]); +} + static void __init new_kmalloc_cache(int idx, int type, slab_flags_t flags) { @@ -1115,7 +1130,7 @@ new_kmalloc_cache(int idx, int type, slab_flags_t flags) if (type == KMALLOC_RECLAIM) { flags |= SLAB_RECLAIM_ACCOUNT; - name = kasprintf(GFP_NOWAIT, "kmalloc-rcl-%u", + name = kmalloc_cache_name("kmalloc-rcl", kmalloc_info[idx].size); BUG_ON(!name); } else { @@ -1164,8 +1179,7 @@ void __init create_kmalloc_caches(slab_flags_t flags) if (s) { unsigned int size = kmalloc_size(i); - char *n = kasprintf(GFP_NOWAIT, - "dma-kmalloc-%u", size); + const char *n = kmalloc_cache_name("dma-kmalloc", size); BUG_ON(!n); kmalloc_caches[KMALLOC_DMA][i] = create_kmalloc_cache(