From patchwork Wed Jul 18 13:36:14 2018 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Vlastimil Babka X-Patchwork-Id: 10532483 Return-Path: Received: from mail.wl.linuxfoundation.org (pdx-wl-mail.web.codeaurora.org [172.30.200.125]) by pdx-korg-patchwork.web.codeaurora.org (Postfix) with ESMTP id 95810601D2 for ; Wed, 18 Jul 2018 13:36:44 +0000 (UTC) Received: from mail.wl.linuxfoundation.org (localhost [127.0.0.1]) by mail.wl.linuxfoundation.org (Postfix) with ESMTP id 85CDF292E3 for ; Wed, 18 Jul 2018 13:36:44 +0000 (UTC) Received: by mail.wl.linuxfoundation.org (Postfix, from userid 486) id 7A1E7293BB; Wed, 18 Jul 2018 13:36:44 +0000 (UTC) X-Spam-Checker-Version: SpamAssassin 3.3.1 (2010-03-16) on pdx-wl-mail.web.codeaurora.org X-Spam-Level: X-Spam-Status: No, score=-2.9 required=2.0 tests=BAYES_00, MAILING_LIST_MULTI, RCVD_IN_DNSWL_NONE autolearn=ham version=3.3.1 Received: from kanga.kvack.org (kanga.kvack.org [205.233.56.17]) by mail.wl.linuxfoundation.org (Postfix) with ESMTP id BB6F0292E3 for ; Wed, 18 Jul 2018 13:36:43 +0000 (UTC) Received: by kanga.kvack.org (Postfix) id B06CE6B000E; Wed, 18 Jul 2018 09:36:32 -0400 (EDT) Delivered-To: linux-mm-outgoing@kvack.org Received: by kanga.kvack.org (Postfix, from userid 40) id A03356B0269; Wed, 18 Jul 2018 09:36:32 -0400 (EDT) X-Original-To: int-list-linux-mm@kvack.org X-Delivered-To: int-list-linux-mm@kvack.org Received: by kanga.kvack.org (Postfix, from userid 63042) id 738996B0266; Wed, 18 Jul 2018 09:36:32 -0400 (EDT) X-Original-To: linux-mm@kvack.org X-Delivered-To: linux-mm@kvack.org Received: from mail-ed1-f72.google.com (mail-ed1-f72.google.com [209.85.208.72]) by kanga.kvack.org (Postfix) with ESMTP id 018096B000D for ; Wed, 18 Jul 2018 09:36:32 -0400 (EDT) Received: by mail-ed1-f72.google.com with SMTP id p5-v6so1949341edh.16 for ; Wed, 18 Jul 2018 06:36:31 -0700 (PDT) X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20161025; h=x-original-authentication-results:x-gm-message-state:from:to:cc :subject:date:message-id:in-reply-to:references; bh=rZ3KSSdkGYFmOrEmsWhTepBg4MH9CyfYFOvnO7bPNjU=; b=CYdTBJrOmeH6gAi2d4B0uvNBHmu4PVzYNKPi4uvVeir/z5vbhKq8fjK0S7B+WfrmRd KsOhb2J8mF16TD3lH37mCatqZkXqc9N2vQz/1aNYvTgrMMiaKkhgNS0+uACRo0N7fBsv E4ZWgEYU4znN1vTRxab4G15WGj+krbJlJP4RU+xfh/jUBBwraqxnZQYbMjB8eA8jgu16 JpW4v2jcnaTCzO6bAn67W0PPbDCMVn3zeLO1tcPByVzufuQ8fk+yY1ELWsk8imVo/y5+ CqBDY3Y0gE8xGXEAP4wPTUDfrAcw/ErFr+4yf6Czsie1F4frVBKLK+BV2kblzUK7roI7 72mQ== X-Original-Authentication-Results: mx.google.com; spf=pass (google.com: domain of vbabka@suse.cz designates 195.135.220.15 as permitted sender) smtp.mailfrom=vbabka@suse.cz X-Gm-Message-State: AOUpUlERRdo0aAsRRkbUwnj762fWGrBMUmqo362+R7NWsn7HxJkDwmsP laHDoyJ7BKEjeLD3W8eF/SF9pDe+47s7tjex6G7BPygDpuhkmkrh3zAm2uYVg112YpwW0JrwVPa txZSXVRLjUetY3U3iVe+sEKLsuARJGYudqf0ClBq1dqWqODy1FfJ3DKRtiYQS726Ssg== X-Received: by 2002:a50:c2d1:: with SMTP id u17-v6mr7235938edf.119.1531920991455; Wed, 18 Jul 2018 06:36:31 -0700 (PDT) X-Google-Smtp-Source: AAOMgpf0cCZQGzRpAh2CF3w/lTz0z15zduHhkD/w8wNHK9NC6vuyEqRIMBnNs25WXYszeXe0zoPn X-Received: by 2002:a50:c2d1:: with SMTP id u17-v6mr7235856edf.119.1531920990207; Wed, 18 Jul 2018 06:36:30 -0700 (PDT) ARC-Seal: i=1; a=rsa-sha256; t=1531920990; cv=none; d=google.com; s=arc-20160816; b=EQdh8IABEui1vAPp0oWzLXNNyE0+qRrFqMpMBl/acGhntAORRpt9rCSK8Gr9EPDdxh 60Ow8gavlJIDETcYpUFW62ZABWb20G6rrsaJAJtcx3f5fOH9bo3SUmewLvUxcba9cVGe pj8n7Tda2Xi/ZL51mbLYucX7MHgI7KxaUgc42/92nV/+HoZk1Fi48PYjsLI/Vd+TUKYL 9Ou7Cknl5hLYZGP4vKeN/s43tgZxNaNNCMsNdNxtU+kQPiL+X6FsNvIseuYpPpECwefi fucSZc7bhAt6AoTdoez0hlnSUxR2dfu+boMnJoih6xaGkWx98kHoD3Am2bpWfW7FpHHm hzjg== ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=google.com; s=arc-20160816; h=references:in-reply-to:message-id:date:subject:cc:to:from :arc-authentication-results; bh=rZ3KSSdkGYFmOrEmsWhTepBg4MH9CyfYFOvnO7bPNjU=; b=lxz/NThQA9Uv08X80L92ybVTOsramIErj+nzRk4fFgkVNKFtTfoHTbs45en9SCqlYK Mkq9ZEYcrOI5HW56zkfcd/a7UFnz7XHs0T8+/Y1ESaK6G/obVPpa0/VlPmRbILDlutSs 2tMHp+qMNzgUAIUqP8NzgE0oUpCcnuBHElemovHVcYVzQQG9hrabqdu5IoRofUGBsFbW bU80+sZNCYru/5v0fJpSuQfrJpHLhMNUK45PNXYKGDKbMnabuROXGyguLTV+iNnUERAy P8BXfo1SMSCq5Y2EuMTCHK8Sch9jmWt2OR3Qorur++WUiqBgNUgqElMqKGa5h9w6Amms 1eKg== ARC-Authentication-Results: i=1; mx.google.com; spf=pass (google.com: domain of vbabka@suse.cz designates 195.135.220.15 as permitted sender) smtp.mailfrom=vbabka@suse.cz Received: from mx1.suse.de (mx2.suse.de. [195.135.220.15]) by mx.google.com with ESMTPS id m18-v6si3022716edf.0.2018.07.18.06.36.30 for (version=TLS1_2 cipher=ECDHE-RSA-AES128-GCM-SHA256 bits=128/128); Wed, 18 Jul 2018 06:36:30 -0700 (PDT) Received-SPF: pass (google.com: domain of vbabka@suse.cz designates 195.135.220.15 as permitted sender) client-ip=195.135.220.15; Authentication-Results: mx.google.com; spf=pass (google.com: domain of vbabka@suse.cz designates 195.135.220.15 as permitted sender) smtp.mailfrom=vbabka@suse.cz X-Virus-Scanned: by amavisd-new at test-mx.suse.de Received: from relay1.suse.de (unknown [195.135.220.254]) by mx1.suse.de (Postfix) with ESMTP id DA74CADBC; Wed, 18 Jul 2018 13:36:28 +0000 (UTC) From: Vlastimil Babka To: Andrew Morton Cc: linux-mm@kvack.org, linux-kernel@vger.kernel.org, linux-api@vger.kernel.org, Roman Gushchin , Michal Hocko , Johannes Weiner , Christoph Lameter , David Rientjes , Joonsoo Kim , Mel Gorman , Matthew Wilcox , Vlastimil Babka Subject: [PATCH v3 1/7] mm, slab: combine kmalloc_caches and kmalloc_dma_caches Date: Wed, 18 Jul 2018 15:36:14 +0200 Message-Id: <20180718133620.6205-2-vbabka@suse.cz> X-Mailer: git-send-email 2.18.0 In-Reply-To: <20180718133620.6205-1-vbabka@suse.cz> References: <20180718133620.6205-1-vbabka@suse.cz> X-Bogosity: Ham, tests=bogofilter, spamicity=0.000000, version=1.2.4 Sender: owner-linux-mm@kvack.org Precedence: bulk X-Loop: owner-majordomo@kvack.org List-ID: X-Virus-Scanned: ClamAV using ClamSMTP The kmalloc caches currently mainain separate (optional) array kmalloc_dma_caches for __GFP_DMA allocations. There are tests for __GFP_DMA in the allocation hotpaths. We can avoid the branches by combining kmalloc_caches and kmalloc_dma_caches into a single two-dimensional array where the outer dimension is cache "type". This will also allow to add kmalloc-reclaimable caches as a third type. Signed-off-by: Vlastimil Babka Acked-by: Mel Gorman Acked-by: Christoph Lameter --- include/linux/slab.h | 41 ++++++++++++++++++++++++++++++----------- mm/slab.c | 4 ++-- mm/slab_common.c | 30 +++++++++++------------------- mm/slub.c | 13 +++++++------ 4 files changed, 50 insertions(+), 38 deletions(-) diff --git a/include/linux/slab.h b/include/linux/slab.h index 14e3fe4bd6a1..4299c59353a1 100644 --- a/include/linux/slab.h +++ b/include/linux/slab.h @@ -295,12 +295,28 @@ static inline void __check_heap_object(const void *ptr, unsigned long n, #define SLAB_OBJ_MIN_SIZE (KMALLOC_MIN_SIZE < 16 ? \ (KMALLOC_MIN_SIZE) : 16) +#define KMALLOC_NORMAL 0 +#ifdef CONFIG_ZONE_DMA +#define KMALLOC_DMA 1 +#define KMALLOC_TYPES 2 +#else +#define KMALLOC_TYPES 1 +#endif + #ifndef CONFIG_SLOB -extern struct kmem_cache *kmalloc_caches[KMALLOC_SHIFT_HIGH + 1]; +extern struct kmem_cache *kmalloc_caches[KMALLOC_TYPES][KMALLOC_SHIFT_HIGH + 1]; + +static __always_inline unsigned int kmalloc_type(gfp_t flags) +{ + int is_dma = 0; + #ifdef CONFIG_ZONE_DMA -extern struct kmem_cache *kmalloc_dma_caches[KMALLOC_SHIFT_HIGH + 1]; + is_dma = !!(flags & __GFP_DMA); #endif + return is_dma; +} + /* * Figure out which kmalloc slab an allocation of a certain size * belongs to. @@ -501,18 +517,20 @@ static __always_inline void *kmalloc_large(size_t size, gfp_t flags) static __always_inline void *kmalloc(size_t size, gfp_t flags) { if (__builtin_constant_p(size)) { +#ifndef CONFIG_SLOB + unsigned int index; +#endif if (size > KMALLOC_MAX_CACHE_SIZE) return kmalloc_large(size, flags); #ifndef CONFIG_SLOB - if (!(flags & GFP_DMA)) { - unsigned int index = kmalloc_index(size); + index = kmalloc_index(size); - if (!index) - return ZERO_SIZE_PTR; + if (!index) + return ZERO_SIZE_PTR; - return kmem_cache_alloc_trace(kmalloc_caches[index], - flags, size); - } + return kmem_cache_alloc_trace( + kmalloc_caches[kmalloc_type(flags)][index], + flags, size); #endif } return __kmalloc(size, flags); @@ -542,13 +560,14 @@ static __always_inline void *kmalloc_node(size_t size, gfp_t flags, int node) { #ifndef CONFIG_SLOB if (__builtin_constant_p(size) && - size <= KMALLOC_MAX_CACHE_SIZE && !(flags & GFP_DMA)) { + size <= KMALLOC_MAX_CACHE_SIZE) { unsigned int i = kmalloc_index(size); if (!i) return ZERO_SIZE_PTR; - return kmem_cache_alloc_node_trace(kmalloc_caches[i], + return kmem_cache_alloc_node_trace( + kmalloc_caches[kmalloc_type(flags)][i], flags, node, size); } #endif diff --git a/mm/slab.c b/mm/slab.c index aa76a70e087e..9515798f37b2 100644 --- a/mm/slab.c +++ b/mm/slab.c @@ -1288,7 +1288,7 @@ void __init kmem_cache_init(void) * Initialize the caches that provide memory for the kmem_cache_node * structures first. Without this, further allocations will bug. */ - kmalloc_caches[INDEX_NODE] = create_kmalloc_cache( + kmalloc_caches[KMALLOC_NORMAL][INDEX_NODE] = create_kmalloc_cache( kmalloc_info[INDEX_NODE].name, kmalloc_size(INDEX_NODE), ARCH_KMALLOC_FLAGS, 0, kmalloc_size(INDEX_NODE)); @@ -1304,7 +1304,7 @@ void __init kmem_cache_init(void) for_each_online_node(nid) { init_list(kmem_cache, &init_kmem_cache_node[CACHE_CACHE + nid], nid); - init_list(kmalloc_caches[INDEX_NODE], + init_list(kmalloc_caches[KMALLOC_NORMAL][INDEX_NODE], &init_kmem_cache_node[SIZE_NODE + nid], nid); } } diff --git a/mm/slab_common.c b/mm/slab_common.c index 2296caf87bfb..4614248ca381 100644 --- a/mm/slab_common.c +++ b/mm/slab_common.c @@ -973,14 +973,9 @@ struct kmem_cache *__init create_kmalloc_cache(const char *name, return s; } -struct kmem_cache *kmalloc_caches[KMALLOC_SHIFT_HIGH + 1] __ro_after_init; +struct kmem_cache *kmalloc_caches[KMALLOC_TYPES][KMALLOC_SHIFT_HIGH + 1] __ro_after_init; EXPORT_SYMBOL(kmalloc_caches); -#ifdef CONFIG_ZONE_DMA -struct kmem_cache *kmalloc_dma_caches[KMALLOC_SHIFT_HIGH + 1] __ro_after_init; -EXPORT_SYMBOL(kmalloc_dma_caches); -#endif - /* * Conversion table for small slabs sizes / 8 to the index in the * kmalloc array. This is necessary for slabs < 192 since we have non power @@ -1040,12 +1035,7 @@ struct kmem_cache *kmalloc_slab(size_t size, gfp_t flags) } else index = fls(size - 1); -#ifdef CONFIG_ZONE_DMA - if (unlikely((flags & GFP_DMA))) - return kmalloc_dma_caches[index]; - -#endif - return kmalloc_caches[index]; + return kmalloc_caches[kmalloc_type(flags)][index]; } /* @@ -1119,7 +1109,8 @@ void __init setup_kmalloc_cache_index_table(void) static void __init new_kmalloc_cache(int idx, slab_flags_t flags) { - kmalloc_caches[idx] = create_kmalloc_cache(kmalloc_info[idx].name, + kmalloc_caches[KMALLOC_NORMAL][idx] = create_kmalloc_cache( + kmalloc_info[idx].name, kmalloc_info[idx].size, flags, 0, kmalloc_info[idx].size); } @@ -1132,9 +1123,10 @@ static void __init new_kmalloc_cache(int idx, slab_flags_t flags) void __init create_kmalloc_caches(slab_flags_t flags) { int i; + int type = KMALLOC_NORMAL; for (i = KMALLOC_SHIFT_LOW; i <= KMALLOC_SHIFT_HIGH; i++) { - if (!kmalloc_caches[i]) + if (!kmalloc_caches[type][i]) new_kmalloc_cache(i, flags); /* @@ -1142,9 +1134,9 @@ void __init create_kmalloc_caches(slab_flags_t flags) * These have to be created immediately after the * earlier power of two caches */ - if (KMALLOC_MIN_SIZE <= 32 && !kmalloc_caches[1] && i == 6) + if (KMALLOC_MIN_SIZE <= 32 && !kmalloc_caches[type][1] && i == 6) new_kmalloc_cache(1, flags); - if (KMALLOC_MIN_SIZE <= 64 && !kmalloc_caches[2] && i == 7) + if (KMALLOC_MIN_SIZE <= 64 && !kmalloc_caches[type][2] && i == 7) new_kmalloc_cache(2, flags); } @@ -1153,7 +1145,7 @@ void __init create_kmalloc_caches(slab_flags_t flags) #ifdef CONFIG_ZONE_DMA for (i = 0; i <= KMALLOC_SHIFT_HIGH; i++) { - struct kmem_cache *s = kmalloc_caches[i]; + struct kmem_cache *s = kmalloc_caches[KMALLOC_NORMAL][i]; if (s) { unsigned int size = kmalloc_size(i); @@ -1161,8 +1153,8 @@ void __init create_kmalloc_caches(slab_flags_t flags) "dma-kmalloc-%u", size); BUG_ON(!n); - kmalloc_dma_caches[i] = create_kmalloc_cache(n, - size, SLAB_CACHE_DMA | flags, 0, 0); + kmalloc_caches[KMALLOC_DMA][i] = create_kmalloc_cache( + n, size, SLAB_CACHE_DMA | flags, 0, 0); } } #endif diff --git a/mm/slub.c b/mm/slub.c index 51258eff4178..a7b4657ea8e0 100644 --- a/mm/slub.c +++ b/mm/slub.c @@ -4659,6 +4659,7 @@ static int list_locations(struct kmem_cache *s, char *buf, static void __init resiliency_test(void) { u8 *p; + int type = KMALLOC_NORMAL; BUILD_BUG_ON(KMALLOC_MIN_SIZE > 16 || KMALLOC_SHIFT_HIGH < 10); @@ -4671,7 +4672,7 @@ static void __init resiliency_test(void) pr_err("\n1. kmalloc-16: Clobber Redzone/next pointer 0x12->0x%p\n\n", p + 16); - validate_slab_cache(kmalloc_caches[4]); + validate_slab_cache(kmalloc_caches[type][4]); /* Hmmm... The next two are dangerous */ p = kzalloc(32, GFP_KERNEL); @@ -4680,33 +4681,33 @@ static void __init resiliency_test(void) p); pr_err("If allocated object is overwritten then not detectable\n\n"); - validate_slab_cache(kmalloc_caches[5]); + validate_slab_cache(kmalloc_caches[type][5]); p = kzalloc(64, GFP_KERNEL); p += 64 + (get_cycles() & 0xff) * sizeof(void *); *p = 0x56; pr_err("\n3. kmalloc-64: corrupting random byte 0x56->0x%p\n", p); pr_err("If allocated object is overwritten then not detectable\n\n"); - validate_slab_cache(kmalloc_caches[6]); + validate_slab_cache(kmalloc_caches[type][6]); pr_err("\nB. Corruption after free\n"); p = kzalloc(128, GFP_KERNEL); kfree(p); *p = 0x78; pr_err("1. kmalloc-128: Clobber first word 0x78->0x%p\n\n", p); - validate_slab_cache(kmalloc_caches[7]); + validate_slab_cache(kmalloc_caches[type][7]); p = kzalloc(256, GFP_KERNEL); kfree(p); p[50] = 0x9a; pr_err("\n2. kmalloc-256: Clobber 50th byte 0x9a->0x%p\n\n", p); - validate_slab_cache(kmalloc_caches[8]); + validate_slab_cache(kmalloc_caches[type][8]); p = kzalloc(512, GFP_KERNEL); kfree(p); p[512] = 0xab; pr_err("\n3. kmalloc-512: Clobber redzone 0xab->0x%p\n\n", p); - validate_slab_cache(kmalloc_caches[9]); + validate_slab_cache(kmalloc_caches[type][9]); } #else #ifdef CONFIG_SYSFS