From patchwork Mon Sep 16 14:45:52 2019 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Pengfei Li X-Patchwork-Id: 11147187 Return-Path: Received: from mail.kernel.org (pdx-korg-mail-1.web.codeaurora.org [172.30.200.123]) by pdx-korg-patchwork-2.web.codeaurora.org (Postfix) with ESMTP id A4C1914ED for ; Mon, 16 Sep 2019 14:46:36 +0000 (UTC) Received: from kanga.kvack.org (kanga.kvack.org [205.233.56.17]) by mail.kernel.org (Postfix) with ESMTP id 5D67021670 for ; Mon, 16 Sep 2019 14:46:36 +0000 (UTC) Authentication-Results: mail.kernel.org; dkim=fail reason="signature verification failed" (2048-bit key) header.d=gmail.com header.i=@gmail.com header.b="lxp0jGsW" DMARC-Filter: OpenDMARC Filter v1.3.2 mail.kernel.org 5D67021670 Authentication-Results: mail.kernel.org; dmarc=fail (p=none dis=none) header.from=gmail.com Authentication-Results: mail.kernel.org; spf=pass smtp.mailfrom=owner-linux-mm@kvack.org Received: by kanga.kvack.org (Postfix) id 915F06B0008; Mon, 16 Sep 2019 10:46:35 -0400 (EDT) Delivered-To: linux-mm-outgoing@kvack.org Received: by kanga.kvack.org (Postfix, from userid 40) id 8ED1A6B000A; Mon, 16 Sep 2019 10:46:35 -0400 (EDT) X-Original-To: int-list-linux-mm@kvack.org X-Delivered-To: int-list-linux-mm@kvack.org Received: by kanga.kvack.org (Postfix, from userid 63042) id 80E0A6B000C; Mon, 16 Sep 2019 10:46:35 -0400 (EDT) X-Original-To: linux-mm@kvack.org X-Delivered-To: linux-mm@kvack.org Received: from forelay.hostedemail.com (smtprelay0154.hostedemail.com [216.40.44.154]) by kanga.kvack.org (Postfix) with ESMTP id 613F66B0008 for ; Mon, 16 Sep 2019 10:46:35 -0400 (EDT) Received: from smtpin08.hostedemail.com (10.5.19.251.rfc1918.com [10.5.19.251]) by forelay04.hostedemail.com (Postfix) with SMTP id 041EB52D8 for ; Mon, 16 Sep 2019 14:46:35 +0000 (UTC) X-FDA: 75941059950.08.kiss36_1c4e4a44a901e X-Spam-Summary: 2,0,0,b19cf853fa396757,d41d8cd98f00b204,lpf.vector@gmail.com,:akpm@linux-foundation.org:vbabka@suse.cz:cl@linux.com:penberg@kernel.org:rientjes@google.com:iamjoonsoo.kim@lge.com::linux-kernel@vger.kernel.org:guro@fb.com:lpf.vector@gmail.com,RULES_HIT:2:41:69:355:379:421:541:800:960:968:973:981:988:989:1260:1311:1314:1345:1359:1437:1500:1515:1535:1605:1730:1747:1777:1792:1801:2198:2199:2393:2559:2562:2693:3138:3139:3140:3141:3142:3865:3866:3867:3868:3870:3871:3874:4049:4120:4321:4605:5007:6238:6261:6653:7514:7903:8603:9413:9592:10004:11026:11232:11473:11658:11914:12043:12291:12296:12297:12438:12517:12519:12555:12683:12895:12986:13161:13229:13894:14394:14687:21080:21444:21450:21451:21627:21666:21740:30029:30030:30054:30064,0,RBL:209.85.214.194:@gmail.com:.lbl8.mailshell.net-62.50.0.100 66.100.201.100,CacheIP:none,Bayesian:0.5,0.5,0.5,Netcheck:none,DomainCache:0,MSF:not bulk,SPF:fp,MSBL:0,DNSBL:none,Custom_rules:0:0:0,LFtime:25,LUA_SUMMARY:none X-HE-Tag: kiss36_1c4e4a44a901e X-Filterd-Recvd-Size: 9086 Received: from mail-pl1-f194.google.com (mail-pl1-f194.google.com [209.85.214.194]) by imf34.hostedemail.com (Postfix) with ESMTP for ; Mon, 16 Sep 2019 14:46:34 +0000 (UTC) Received: by mail-pl1-f194.google.com with SMTP id x3so17000036plr.12 for ; Mon, 16 Sep 2019 07:46:34 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=20161025; h=from:to:cc:subject:date:message-id:in-reply-to:references :mime-version:content-transfer-encoding; bh=dZvPUhHT8+0/JcqR3/I68iZikorHw/l25qBcUtuCr9U=; b=lxp0jGsW5SKgtGeD68P/E+1P1Z1JXtgMeabFqU7FkL/ETQnxsbVwwCLgEXexmLnkRH iMxuitWshQqwoFx1ADGIlpCMnVO2qwzbSYWeFvnCbyeJft+ha2ZXy1rJ94ZfandF/PlX K+lUHKabM+lE6sWb+nvz4kRYjETieJk/rRMKwgHxRucGCmsPSkLpFby0dVRaDPJQIafa L+X/vFeLPxj2Xl1/zCgQOFt3JEGknlKdZYBm1jUTfWqa77ef7OEMXNi6FYzG9KHbosv0 O0OQokmUSsksjgMWUI/vpkfNj/XHxNxp54vPeNUB6o1JhQdtCZ850WNvnVyv4Y7qo6+3 fzjw== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20161025; h=x-gm-message-state:from:to:cc:subject:date:message-id:in-reply-to :references:mime-version:content-transfer-encoding; bh=dZvPUhHT8+0/JcqR3/I68iZikorHw/l25qBcUtuCr9U=; b=SFyH6Zm75D0mWnMIDHlIe6dWkF98OLm0w1BsQEI0PXY0ByFSxWXgYpGRYA9J2bmn2i hl7krfz+hXSg1s1VZy+49HuYUUur7zBROQ8tUd0LI6o5lPgzzDdHlkUmGa3URoMuQdJG Jbf5g9KUAQ3M1fdT8rFxf7pPsjv3C9UBF9kQBm3qFb04aXVbti+1A8EnwKrl1hg50b4T pPAqkhPvXl329Lwq83KgU3t4EfEMADqSWkIRxRl1b9rnvOYSqaZRragJdirTl0PwkdBS tOVDTEtw2YM/AFHV5x0p1yItxIpFds6XwpP24XoC94rWhS/LW4eSCf5RMtQnVgO5bTXh ryFA== X-Gm-Message-State: APjAAAXOQMVb2f7i+NIvjiE8Lqk7GhVAoOU+KlyznLmCpX6RNUKHUB8i /IZa4veudAcPTBuSLKJkiFY= X-Google-Smtp-Source: APXvYqw5GgPczEn9LOIYLaCdLz+m++WYbNMNkyqhk4eD3gdEPa40dsAcO+vIL7GTieCB76CZxxJxvA== X-Received: by 2002:a17:902:6b06:: with SMTP id o6mr141532plk.129.1568645193523; Mon, 16 Sep 2019 07:46:33 -0700 (PDT) Received: from localhost.localdomain.localdomain ([2408:823c:c11:160:b8c3:8577:bf2f:3]) by smtp.gmail.com with ESMTPSA id d190sm15036004pgc.25.2019.09.16.07.46.23 (version=TLS1_3 cipher=TLS_AES_256_GCM_SHA384 bits=256/256); Mon, 16 Sep 2019 07:46:32 -0700 (PDT) From: Pengfei Li To: akpm@linux-foundation.org Cc: vbabka@suse.cz, cl@linux.com, penberg@kernel.org, rientjes@google.com, iamjoonsoo.kim@lge.com, linux-mm@kvack.org, linux-kernel@vger.kernel.org, guro@fb.com, Pengfei Li Subject: [PATCH v5 1/7] mm, slab: Make kmalloc_info[] contain all types of names Date: Mon, 16 Sep 2019 22:45:52 +0800 Message-Id: <20190916144558.27282-2-lpf.vector@gmail.com> X-Mailer: git-send-email 2.21.0 In-Reply-To: <20190916144558.27282-1-lpf.vector@gmail.com> References: <20190916144558.27282-1-lpf.vector@gmail.com> MIME-Version: 1.0 X-Bogosity: Ham, tests=bogofilter, spamicity=0.000000, version=1.2.4 Sender: owner-linux-mm@kvack.org Precedence: bulk X-Loop: owner-majordomo@kvack.org List-ID: There are three types of kmalloc, KMALLOC_NORMAL, KMALLOC_RECLAIM and KMALLOC_DMA. The name of KMALLOC_NORMAL is contained in kmalloc_info[].name, but the names of KMALLOC_RECLAIM and KMALLOC_DMA are dynamically generated by kmalloc_cache_name(). This patch predefines the names of all types of kmalloc to save the time spent dynamically generating names. Besides, remove the kmalloc_cache_name() that is no longer used. Signed-off-by: Pengfei Li Acked-by: Vlastimil Babka Acked-by: Roman Gushchin Acked-by: David Rientjes --- mm/slab.c | 2 +- mm/slab.h | 2 +- mm/slab_common.c | 91 ++++++++++++++++++++++++++---------------------- 3 files changed, 51 insertions(+), 44 deletions(-) diff --git a/mm/slab.c b/mm/slab.c index 9df370558e5d..c42b6211f42e 100644 --- a/mm/slab.c +++ b/mm/slab.c @@ -1247,7 +1247,7 @@ void __init kmem_cache_init(void) * structures first. Without this, further allocations will bug. */ kmalloc_caches[KMALLOC_NORMAL][INDEX_NODE] = create_kmalloc_cache( - kmalloc_info[INDEX_NODE].name, + kmalloc_info[INDEX_NODE].name[KMALLOC_NORMAL], kmalloc_size(INDEX_NODE), ARCH_KMALLOC_FLAGS, 0, kmalloc_size(INDEX_NODE)); slab_state = PARTIAL_NODE; diff --git a/mm/slab.h b/mm/slab.h index 9057b8056b07..2fc8f956906a 100644 --- a/mm/slab.h +++ b/mm/slab.h @@ -76,7 +76,7 @@ extern struct kmem_cache *kmem_cache; /* A table of kmalloc cache names and sizes */ extern const struct kmalloc_info_struct { - const char *name; + const char *name[NR_KMALLOC_TYPES]; unsigned int size; } kmalloc_info[]; diff --git a/mm/slab_common.c b/mm/slab_common.c index 807490fe217a..4e78490292df 100644 --- a/mm/slab_common.c +++ b/mm/slab_common.c @@ -1092,26 +1092,56 @@ struct kmem_cache *kmalloc_slab(size_t size, gfp_t flags) return kmalloc_caches[kmalloc_type(flags)][index]; } +#ifdef CONFIG_ZONE_DMA +#define INIT_KMALLOC_INFO(__size, __short_size) \ +{ \ + .name[KMALLOC_NORMAL] = "kmalloc-" #__short_size, \ + .name[KMALLOC_RECLAIM] = "kmalloc-rcl-" #__short_size, \ + .name[KMALLOC_DMA] = "dma-kmalloc-" #__short_size, \ + .size = __size, \ +} +#else +#define INIT_KMALLOC_INFO(__size, __short_size) \ +{ \ + .name[KMALLOC_NORMAL] = "kmalloc-" #__short_size, \ + .name[KMALLOC_RECLAIM] = "kmalloc-rcl-" #__short_size, \ + .size = __size, \ +} +#endif + /* * kmalloc_info[] is to make slub_debug=,kmalloc-xx option work at boot time. * kmalloc_index() supports up to 2^26=64MB, so the final entry of the table is * kmalloc-67108864. */ const struct kmalloc_info_struct kmalloc_info[] __initconst = { - {NULL, 0}, {"kmalloc-96", 96}, - {"kmalloc-192", 192}, {"kmalloc-8", 8}, - {"kmalloc-16", 16}, {"kmalloc-32", 32}, - {"kmalloc-64", 64}, {"kmalloc-128", 128}, - {"kmalloc-256", 256}, {"kmalloc-512", 512}, - {"kmalloc-1k", 1024}, {"kmalloc-2k", 2048}, - {"kmalloc-4k", 4096}, {"kmalloc-8k", 8192}, - {"kmalloc-16k", 16384}, {"kmalloc-32k", 32768}, - {"kmalloc-64k", 65536}, {"kmalloc-128k", 131072}, - {"kmalloc-256k", 262144}, {"kmalloc-512k", 524288}, - {"kmalloc-1M", 1048576}, {"kmalloc-2M", 2097152}, - {"kmalloc-4M", 4194304}, {"kmalloc-8M", 8388608}, - {"kmalloc-16M", 16777216}, {"kmalloc-32M", 33554432}, - {"kmalloc-64M", 67108864} + INIT_KMALLOC_INFO(0, 0), + INIT_KMALLOC_INFO(96, 96), + INIT_KMALLOC_INFO(192, 192), + INIT_KMALLOC_INFO(8, 8), + INIT_KMALLOC_INFO(16, 16), + INIT_KMALLOC_INFO(32, 32), + INIT_KMALLOC_INFO(64, 64), + INIT_KMALLOC_INFO(128, 128), + INIT_KMALLOC_INFO(256, 256), + INIT_KMALLOC_INFO(512, 512), + INIT_KMALLOC_INFO(1024, 1k), + INIT_KMALLOC_INFO(2048, 2k), + INIT_KMALLOC_INFO(4096, 4k), + INIT_KMALLOC_INFO(8192, 8k), + INIT_KMALLOC_INFO(16384, 16k), + INIT_KMALLOC_INFO(32768, 32k), + INIT_KMALLOC_INFO(65536, 64k), + INIT_KMALLOC_INFO(131072, 128k), + INIT_KMALLOC_INFO(262144, 256k), + INIT_KMALLOC_INFO(524288, 512k), + INIT_KMALLOC_INFO(1048576, 1M), + INIT_KMALLOC_INFO(2097152, 2M), + INIT_KMALLOC_INFO(4194304, 4M), + INIT_KMALLOC_INFO(8388608, 8M), + INIT_KMALLOC_INFO(16777216, 16M), + INIT_KMALLOC_INFO(33554432, 32M), + INIT_KMALLOC_INFO(67108864, 64M) }; /* @@ -1161,36 +1191,14 @@ void __init setup_kmalloc_cache_index_table(void) } } -static const char * -kmalloc_cache_name(const char *prefix, unsigned int size) -{ - - static const char units[3] = "\0kM"; - int idx = 0; - - while (size >= 1024 && (size % 1024 == 0)) { - size /= 1024; - idx++; - } - - return kasprintf(GFP_NOWAIT, "%s-%u%c", prefix, size, units[idx]); -} - static void __init new_kmalloc_cache(int idx, int type, slab_flags_t flags) { - const char *name; - - if (type == KMALLOC_RECLAIM) { + if (type == KMALLOC_RECLAIM) flags |= SLAB_RECLAIM_ACCOUNT; - name = kmalloc_cache_name("kmalloc-rcl", - kmalloc_info[idx].size); - BUG_ON(!name); - } else { - name = kmalloc_info[idx].name; - } - kmalloc_caches[type][idx] = create_kmalloc_cache(name, + kmalloc_caches[type][idx] = create_kmalloc_cache( + kmalloc_info[idx].name[type], kmalloc_info[idx].size, flags, 0, kmalloc_info[idx].size); } @@ -1232,11 +1240,10 @@ void __init create_kmalloc_caches(slab_flags_t flags) if (s) { unsigned int size = kmalloc_size(i); - const char *n = kmalloc_cache_name("dma-kmalloc", size); - BUG_ON(!n); kmalloc_caches[KMALLOC_DMA][i] = create_kmalloc_cache( - n, size, SLAB_CACHE_DMA | flags, 0, 0); + kmalloc_info[i].name[KMALLOC_DMA], + size, SLAB_CACHE_DMA | flags, 0, 0); } } #endif From patchwork Mon Sep 16 14:45:53 2019 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Pengfei Li X-Patchwork-Id: 11147189 Return-Path: Received: from mail.kernel.org (pdx-korg-mail-1.web.codeaurora.org [172.30.200.123]) by pdx-korg-patchwork-2.web.codeaurora.org (Postfix) with ESMTP id DD35414ED for ; Mon, 16 Sep 2019 14:46:46 +0000 (UTC) Received: from kanga.kvack.org (kanga.kvack.org [205.233.56.17]) by mail.kernel.org (Postfix) with ESMTP id AA35621670 for ; Mon, 16 Sep 2019 14:46:46 +0000 (UTC) Authentication-Results: mail.kernel.org; dkim=fail reason="signature verification failed" (2048-bit key) header.d=gmail.com header.i=@gmail.com header.b="BmJ5hx5b" DMARC-Filter: OpenDMARC Filter v1.3.2 mail.kernel.org AA35621670 Authentication-Results: mail.kernel.org; dmarc=fail (p=none dis=none) header.from=gmail.com Authentication-Results: mail.kernel.org; spf=pass smtp.mailfrom=owner-linux-mm@kvack.org Received: by kanga.kvack.org (Postfix) id DD0946B000A; Mon, 16 Sep 2019 10:46:45 -0400 (EDT) Delivered-To: linux-mm-outgoing@kvack.org Received: by kanga.kvack.org (Postfix, from userid 40) id DA71F6B000C; Mon, 16 Sep 2019 10:46:45 -0400 (EDT) X-Original-To: int-list-linux-mm@kvack.org X-Delivered-To: int-list-linux-mm@kvack.org Received: by kanga.kvack.org (Postfix, from userid 63042) id CBC946B000D; Mon, 16 Sep 2019 10:46:45 -0400 (EDT) X-Original-To: linux-mm@kvack.org X-Delivered-To: linux-mm@kvack.org Received: from forelay.hostedemail.com (smtprelay0101.hostedemail.com [216.40.44.101]) by kanga.kvack.org (Postfix) with ESMTP id AB3116B000A for ; Mon, 16 Sep 2019 10:46:45 -0400 (EDT) Received: from smtpin12.hostedemail.com (10.5.19.251.rfc1918.com [10.5.19.251]) by forelay05.hostedemail.com (Postfix) with SMTP id 5412F181AC9AE for ; Mon, 16 Sep 2019 14:46:45 +0000 (UTC) X-FDA: 75941060370.12.art13_1dca717ae2c2d X-Spam-Summary: 2,0,0,235e114e56ea37a2,d41d8cd98f00b204,lpf.vector@gmail.com,:akpm@linux-foundation.org:vbabka@suse.cz:cl@linux.com:penberg@kernel.org:rientjes@google.com:iamjoonsoo.kim@lge.com::linux-kernel@vger.kernel.org:guro@fb.com:lpf.vector@gmail.com,RULES_HIT:41:69:355:379:541:800:960:968:973:988:989:1260:1311:1314:1345:1359:1437:1515:1535:1542:1711:1730:1747:1777:1792:2198:2199:2393:2559:2562:2693:3138:3139:3140:3141:3142:3353:3865:3866:3874:4321:4605:5007:6119:6261:6653:7514:7903:8603:9413:9592:10004:11026:11473:11658:11914:12043:12296:12297:12438:12517:12519:12555:12895:12986:13161:13229:13894:14181:14394:14687:14721:21080:21444:21451:21627:21666:30029:30054:30064,0,RBL:209.85.210.196:@gmail.com:.lbl8.mailshell.net-62.50.0.100 66.100.201.100,CacheIP:none,Bayesian:0.5,0.5,0.5,Netcheck:none,DomainCache:0,MSF:not bulk,SPF:fp,MSBL:0,DNSBL:none,Custom_rules:0:0:0,LFtime:27,LUA_SUMMARY:none X-HE-Tag: art13_1dca717ae2c2d X-Filterd-Recvd-Size: 5462 Received: from mail-pf1-f196.google.com (mail-pf1-f196.google.com [209.85.210.196]) by imf26.hostedemail.com (Postfix) with ESMTP for ; Mon, 16 Sep 2019 14:46:44 +0000 (UTC) Received: by mail-pf1-f196.google.com with SMTP id 205so44520pfw.2 for ; Mon, 16 Sep 2019 07:46:44 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=20161025; h=from:to:cc:subject:date:message-id:in-reply-to:references :mime-version:content-transfer-encoding; bh=d7Q0z2EasZ8/1ldA12KLzNbebPjqFp3ShlnBdaO1amY=; b=BmJ5hx5bZeYzBaMASNSBw7TKgI8T7IVDzo+AXUN2oSUIlaTP7YJxGtSd0pJWV1Vs9r 6ZAn7Ja08Yrd3PP4tJ/j03GSLhLfjub1yp/1hMNM4ZACT89L7mS3gWZFe5Y4AS6WuCxv Uq1eECAtwGOmRZoCKIImWTnydyaGZJxL77AvGPnUo2bJKH3h9uZZVN9OJ69WldsdCLVb BzX/FLRAMZL75nm5ozBNb1RWKrf15wKqnqlVPi60uhPqI5xLdNwDf2yy9htLeVZ5wiNz 4PqAS9SV/5Huxe9n45jQMKxeLK4Bi9thYUvQiw+3v22lZevqW7D78kaCUJVhKh7bHfsc /c6g== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20161025; h=x-gm-message-state:from:to:cc:subject:date:message-id:in-reply-to :references:mime-version:content-transfer-encoding; bh=d7Q0z2EasZ8/1ldA12KLzNbebPjqFp3ShlnBdaO1amY=; b=QQZ/YDZErXxJlZNFUMOH3LV3IMGurQFeHiaUuKtCWXwlBvT1gh+xteqO8tIUJYU9+b 9pgbUaAZR11Gq+LHm92WFhzI5SzzQ1gt0KiHtliAif8SBGF6RZjO7TE6txFQCcTi1PBT /tyCZtGfUKmL4/rcxTUt0UgWRZ4vMFhshs5CbnxabzQixdhJxSYc2FeXNppq3dgtW8um x+EdKHrCAzQJOXX62nddmGCKKGEKtg2AG1/CY3VAyNg0XHmmtvaqShoEZtzmEgX2pypw vDnRgZugtAdM+aqFcTX72wLx3y6HjLSKAMxbsAqJmtdbsr5uMq/zZy8nMFkAVJu+OEP1 LeTA== X-Gm-Message-State: APjAAAWMzJTwmjKWWU7hE3qg7erFrUtD/41Njl2Zfm/5z64ooIeRt7Sg KL9RiyvKBilwwwlX3mJ0SeA= X-Google-Smtp-Source: APXvYqyZZupA7waxaBg53lTnH6e5LUJ+eqZAgQ/hO1ZOP8yq6SUAJruqY8t8GbSlhBFk9FUwd8J4Vw== X-Received: by 2002:a62:75d2:: with SMTP id q201mr71101005pfc.43.1568645203635; Mon, 16 Sep 2019 07:46:43 -0700 (PDT) Received: from localhost.localdomain.localdomain ([2408:823c:c11:160:b8c3:8577:bf2f:3]) by smtp.gmail.com with ESMTPSA id d190sm15036004pgc.25.2019.09.16.07.46.34 (version=TLS1_3 cipher=TLS_AES_256_GCM_SHA384 bits=256/256); Mon, 16 Sep 2019 07:46:42 -0700 (PDT) From: Pengfei Li To: akpm@linux-foundation.org Cc: vbabka@suse.cz, cl@linux.com, penberg@kernel.org, rientjes@google.com, iamjoonsoo.kim@lge.com, linux-mm@kvack.org, linux-kernel@vger.kernel.org, guro@fb.com, Pengfei Li Subject: [PATCH v5 2/7] mm, slab: Remove unused kmalloc_size() Date: Mon, 16 Sep 2019 22:45:53 +0800 Message-Id: <20190916144558.27282-3-lpf.vector@gmail.com> X-Mailer: git-send-email 2.21.0 In-Reply-To: <20190916144558.27282-1-lpf.vector@gmail.com> References: <20190916144558.27282-1-lpf.vector@gmail.com> MIME-Version: 1.0 X-Bogosity: Ham, tests=bogofilter, spamicity=0.000000, version=1.2.4 Sender: owner-linux-mm@kvack.org Precedence: bulk X-Loop: owner-majordomo@kvack.org List-ID: The size of kmalloc can be obtained from kmalloc_info[], so remove kmalloc_size() that will not be used anymore. Signed-off-by: Pengfei Li Acked-by: Vlastimil Babka Acked-by: Roman Gushchin Acked-by: David Rientjes --- include/linux/slab.h | 20 -------------------- mm/slab.c | 5 +++-- mm/slab_common.c | 5 ++--- 3 files changed, 5 insertions(+), 25 deletions(-) diff --git a/include/linux/slab.h b/include/linux/slab.h index 56c9c7eed34e..e773e5764b7b 100644 --- a/include/linux/slab.h +++ b/include/linux/slab.h @@ -557,26 +557,6 @@ static __always_inline void *kmalloc(size_t size, gfp_t flags) return __kmalloc(size, flags); } -/* - * Determine size used for the nth kmalloc cache. - * return size or 0 if a kmalloc cache for that - * size does not exist - */ -static __always_inline unsigned int kmalloc_size(unsigned int n) -{ -#ifndef CONFIG_SLOB - if (n > 2) - return 1U << n; - - if (n == 1 && KMALLOC_MIN_SIZE <= 32) - return 96; - - if (n == 2 && KMALLOC_MIN_SIZE <= 64) - return 192; -#endif - return 0; -} - static __always_inline void *kmalloc_node(size_t size, gfp_t flags, int node) { #ifndef CONFIG_SLOB diff --git a/mm/slab.c b/mm/slab.c index c42b6211f42e..7bc4e90e1147 100644 --- a/mm/slab.c +++ b/mm/slab.c @@ -1248,8 +1248,9 @@ void __init kmem_cache_init(void) */ kmalloc_caches[KMALLOC_NORMAL][INDEX_NODE] = create_kmalloc_cache( kmalloc_info[INDEX_NODE].name[KMALLOC_NORMAL], - kmalloc_size(INDEX_NODE), ARCH_KMALLOC_FLAGS, - 0, kmalloc_size(INDEX_NODE)); + kmalloc_info[INDEX_NODE].size, + ARCH_KMALLOC_FLAGS, 0, + kmalloc_info[INDEX_NODE].size); slab_state = PARTIAL_NODE; setup_kmalloc_cache_index_table(); diff --git a/mm/slab_common.c b/mm/slab_common.c index 4e78490292df..df030cf9f44f 100644 --- a/mm/slab_common.c +++ b/mm/slab_common.c @@ -1239,11 +1239,10 @@ void __init create_kmalloc_caches(slab_flags_t flags) struct kmem_cache *s = kmalloc_caches[KMALLOC_NORMAL][i]; if (s) { - unsigned int size = kmalloc_size(i); - kmalloc_caches[KMALLOC_DMA][i] = create_kmalloc_cache( kmalloc_info[i].name[KMALLOC_DMA], - size, SLAB_CACHE_DMA | flags, 0, 0); + kmalloc_info[i].size, + SLAB_CACHE_DMA | flags, 0, 0); } } #endif From patchwork Mon Sep 16 14:45:54 2019 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Pengfei Li X-Patchwork-Id: 11147191 Return-Path: Received: from mail.kernel.org (pdx-korg-mail-1.web.codeaurora.org [172.30.200.123]) by pdx-korg-patchwork-2.web.codeaurora.org (Postfix) with ESMTP id 302E11708 for ; Mon, 16 Sep 2019 14:46:57 +0000 (UTC) Received: from kanga.kvack.org (kanga.kvack.org [205.233.56.17]) by mail.kernel.org (Postfix) with ESMTP id F092321848 for ; Mon, 16 Sep 2019 14:46:56 +0000 (UTC) Authentication-Results: mail.kernel.org; dkim=fail reason="signature verification failed" (2048-bit key) header.d=gmail.com header.i=@gmail.com header.b="l0IZ2qlW" DMARC-Filter: OpenDMARC Filter v1.3.2 mail.kernel.org F092321848 Authentication-Results: mail.kernel.org; dmarc=fail (p=none dis=none) header.from=gmail.com Authentication-Results: mail.kernel.org; spf=pass smtp.mailfrom=owner-linux-mm@kvack.org Received: by kanga.kvack.org (Postfix) id 370026B000C; Mon, 16 Sep 2019 10:46:56 -0400 (EDT) Delivered-To: linux-mm-outgoing@kvack.org Received: by kanga.kvack.org (Postfix, from userid 40) id 347DD6B000D; Mon, 16 Sep 2019 10:46:56 -0400 (EDT) X-Original-To: int-list-linux-mm@kvack.org X-Delivered-To: int-list-linux-mm@kvack.org Received: by kanga.kvack.org (Postfix, from userid 63042) id 283FC6B000E; Mon, 16 Sep 2019 10:46:56 -0400 (EDT) X-Original-To: linux-mm@kvack.org X-Delivered-To: linux-mm@kvack.org Received: from forelay.hostedemail.com (smtprelay0103.hostedemail.com [216.40.44.103]) by kanga.kvack.org (Postfix) with ESMTP id 072286B000C for ; Mon, 16 Sep 2019 10:46:55 -0400 (EDT) Received: from smtpin23.hostedemail.com (10.5.19.251.rfc1918.com [10.5.19.251]) by forelay04.hostedemail.com (Postfix) with SMTP id B3FCE4425 for ; Mon, 16 Sep 2019 14:46:55 +0000 (UTC) X-FDA: 75941060790.23.bait11_1f56322c87255 X-Spam-Summary: 2,0,0,da7517507e86ac33,d41d8cd98f00b204,lpf.vector@gmail.com,:akpm@linux-foundation.org:vbabka@suse.cz:cl@linux.com:penberg@kernel.org:rientjes@google.com:iamjoonsoo.kim@lge.com::linux-kernel@vger.kernel.org:guro@fb.com:lpf.vector@gmail.com,RULES_HIT:41:355:379:541:800:960:973:988:989:1260:1311:1314:1345:1359:1437:1515:1534:1541:1711:1730:1747:1777:1792:2198:2199:2393:2559:2562:3138:3139:3140:3141:3142:3352:3865:3866:3872:4321:5007:6261:6653:7514:9413:10004:11026:11473:11658:11914:12043:12297:12438:12517:12519:12555:12895:13069:13161:13229:13255:13311:13357:13894:14096:14181:14384:14394:14687:14721:21080:21444:21450:21451:21627:21666:30054:30064,0,RBL:209.85.210.194:@gmail.com:.lbl8.mailshell.net-62.50.0.100 66.100.201.100,CacheIP:none,Bayesian:0.5,0.5,0.5,Netcheck:none,DomainCache:0,MSF:not bulk,SPF:fp,MSBL:0,DNSBL:none,Custom_rules:0:0:0,LFtime:25,LUA_SUMMARY:none X-HE-Tag: bait11_1f56322c87255 X-Filterd-Recvd-Size: 4276 Received: from mail-pf1-f194.google.com (mail-pf1-f194.google.com [209.85.210.194]) by imf02.hostedemail.com (Postfix) with ESMTP for ; Mon, 16 Sep 2019 14:46:55 +0000 (UTC) Received: by mail-pf1-f194.google.com with SMTP id x127so27479pfb.7 for ; Mon, 16 Sep 2019 07:46:55 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=20161025; h=from:to:cc:subject:date:message-id:in-reply-to:references :mime-version:content-transfer-encoding; bh=jCVsoiANbO0A0rAsPONqC8CmNnKvwITi8BzVSSbb6XQ=; b=l0IZ2qlWNiFhcK1qlGvKwC1wuMV95H42NFba70/jxZptHT9Rbemzw5VlpIuap1kS+d BX18/2l1ApbcJoKmv7zOkvQWoHljxRBunQDUfccRw5l0+ZqMcxOpjERJlc9byEy5VUuq MWOBQ0M7hMpOYmGcR2h8EG7yBUrQq0nQg6fHWj+QbiUV5M0fE6m3/t+dKCJuaMg/9k2r bqJTJSajXVA6+5QTIUssK8713gmQXTyFyEr7269syveyZwN1RGjO9pj6xW/eYT+/OpxV hGVhx+/ZSgzb+S+Qm6Gvj8RJ/X5TPS6XSU4KeA2bI8MZOvJw62IsUUx1+lSiKCDVfpTG 5F8A== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20161025; h=x-gm-message-state:from:to:cc:subject:date:message-id:in-reply-to :references:mime-version:content-transfer-encoding; bh=jCVsoiANbO0A0rAsPONqC8CmNnKvwITi8BzVSSbb6XQ=; b=ihtrwgTQBf3HlqQwSoTV2geZzFiLchQQj7ewyq0yPKdqAbIPv/EaD3nR46h8QNYC4Y bxBcCz6tAFxTN/+TzTgLWtlzXCFOqc3KjRXUpJE6+yth9QEDkOAsOTTypjPpbkR7D8MJ Y9F4xDRL/Y5h3JrHMQSJiqhjuLIDT8VDcHRzYnjG32yFaLcE7FHyhvGq9jdqijaB4DsH xkXOoQNI81brX47/Q2HcLClButLYFFw2ADTWzCELJyqd0+UegWUwp81yCA7vqizgHLiz B5riK7ENsxnaLvyYOTaQAWkdMruVmcZrPn6TXV8iFJzNjYp53JZ0G/muFh9NVg6EtOod 34cA== X-Gm-Message-State: APjAAAVKUnY03Uxvmv6dToT0bo2/oCa+phxoZx9eh36vOtiqtl1rIona GXVYoDyQ95AUDUgLERUfjBU= X-Google-Smtp-Source: APXvYqyyA+wz35DCND9zfqzqpU24YpmTLwayeIV3v5e/r0L7Cl3JOR/qARquRdKD/5+zmkGiT5juiw== X-Received: by 2002:a17:90a:ca0e:: with SMTP id x14mr108909pjt.70.1568645214399; Mon, 16 Sep 2019 07:46:54 -0700 (PDT) Received: from localhost.localdomain.localdomain ([2408:823c:c11:160:b8c3:8577:bf2f:3]) by smtp.gmail.com with ESMTPSA id d190sm15036004pgc.25.2019.09.16.07.46.44 (version=TLS1_3 cipher=TLS_AES_256_GCM_SHA384 bits=256/256); Mon, 16 Sep 2019 07:46:53 -0700 (PDT) From: Pengfei Li To: akpm@linux-foundation.org Cc: vbabka@suse.cz, cl@linux.com, penberg@kernel.org, rientjes@google.com, iamjoonsoo.kim@lge.com, linux-mm@kvack.org, linux-kernel@vger.kernel.org, guro@fb.com, Pengfei Li Subject: [PATCH v5 3/7] mm, slab_common: Use enum kmalloc_cache_type to iterate over kmalloc caches Date: Mon, 16 Sep 2019 22:45:54 +0800 Message-Id: <20190916144558.27282-4-lpf.vector@gmail.com> X-Mailer: git-send-email 2.21.0 In-Reply-To: <20190916144558.27282-1-lpf.vector@gmail.com> References: <20190916144558.27282-1-lpf.vector@gmail.com> MIME-Version: 1.0 X-Bogosity: Ham, tests=bogofilter, spamicity=0.000000, version=1.2.4 Sender: owner-linux-mm@kvack.org Precedence: bulk X-Loop: owner-majordomo@kvack.org List-ID: The type of local variable *type* of new_kmalloc_cache() should be enum kmalloc_cache_type instead of int, so correct it. Signed-off-by: Pengfei Li Acked-by: Vlastimil Babka Acked-by: Roman Gushchin Acked-by: David Rientjes --- mm/slab_common.c | 5 +++-- 1 file changed, 3 insertions(+), 2 deletions(-) diff --git a/mm/slab_common.c b/mm/slab_common.c index df030cf9f44f..23054b8b75b6 100644 --- a/mm/slab_common.c +++ b/mm/slab_common.c @@ -1192,7 +1192,7 @@ void __init setup_kmalloc_cache_index_table(void) } static void __init -new_kmalloc_cache(int idx, int type, slab_flags_t flags) +new_kmalloc_cache(int idx, enum kmalloc_cache_type type, slab_flags_t flags) { if (type == KMALLOC_RECLAIM) flags |= SLAB_RECLAIM_ACCOUNT; @@ -1210,7 +1210,8 @@ new_kmalloc_cache(int idx, int type, slab_flags_t flags) */ void __init create_kmalloc_caches(slab_flags_t flags) { - int i, type; + int i; + enum kmalloc_cache_type type; for (type = KMALLOC_NORMAL; type <= KMALLOC_RECLAIM; type++) { for (i = KMALLOC_SHIFT_LOW; i <= KMALLOC_SHIFT_HIGH; i++) { From patchwork Mon Sep 16 14:45:55 2019 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Pengfei Li X-Patchwork-Id: 11147193 Return-Path: Received: from mail.kernel.org (pdx-korg-mail-1.web.codeaurora.org [172.30.200.123]) by pdx-korg-patchwork-2.web.codeaurora.org (Postfix) with ESMTP id F223A14ED for ; Mon, 16 Sep 2019 14:47:08 +0000 (UTC) Received: from kanga.kvack.org (kanga.kvack.org [205.233.56.17]) by mail.kernel.org (Postfix) with ESMTP id BEE9F21670 for ; Mon, 16 Sep 2019 14:47:08 +0000 (UTC) Authentication-Results: mail.kernel.org; dkim=fail reason="signature verification failed" (2048-bit key) header.d=gmail.com header.i=@gmail.com header.b="YR9U5b2A" DMARC-Filter: OpenDMARC Filter v1.3.2 mail.kernel.org BEE9F21670 Authentication-Results: mail.kernel.org; dmarc=fail (p=none dis=none) header.from=gmail.com Authentication-Results: mail.kernel.org; spf=pass smtp.mailfrom=owner-linux-mm@kvack.org Received: by kanga.kvack.org (Postfix) id 0133B6B000D; Mon, 16 Sep 2019 10:47:08 -0400 (EDT) Delivered-To: linux-mm-outgoing@kvack.org Received: by kanga.kvack.org (Postfix, from userid 40) id F07606B000E; Mon, 16 Sep 2019 10:47:07 -0400 (EDT) X-Original-To: int-list-linux-mm@kvack.org X-Delivered-To: int-list-linux-mm@kvack.org Received: by kanga.kvack.org (Postfix, from userid 63042) id E1C566B0010; Mon, 16 Sep 2019 10:47:07 -0400 (EDT) X-Original-To: linux-mm@kvack.org X-Delivered-To: linux-mm@kvack.org Received: from forelay.hostedemail.com (smtprelay0112.hostedemail.com [216.40.44.112]) by kanga.kvack.org (Postfix) with ESMTP id BF8246B000D for ; Mon, 16 Sep 2019 10:47:07 -0400 (EDT) Received: from smtpin29.hostedemail.com (10.5.19.251.rfc1918.com [10.5.19.251]) by forelay05.hostedemail.com (Postfix) with SMTP id 6B535181AC9AE for ; Mon, 16 Sep 2019 14:47:06 +0000 (UTC) X-FDA: 75941061252.29.snake60_20e48c57c5100 X-Spam-Summary: 2,0,0,19812857bf735524,d41d8cd98f00b204,lpf.vector@gmail.com,:akpm@linux-foundation.org:vbabka@suse.cz:cl@linux.com:penberg@kernel.org:rientjes@google.com:iamjoonsoo.kim@lge.com::linux-kernel@vger.kernel.org:guro@fb.com:lpf.vector@gmail.com,RULES_HIT:41:355:379:541:800:960:966:973:988:989:1260:1311:1314:1345:1359:1437:1515:1535:1542:1711:1730:1747:1777:1792:2196:2199:2393:2559:2562:2693:2901:3138:3139:3140:3141:3142:3353:3369:3865:3866:3867:3870:3871:3872:3874:4250:4321:4385:5007:6261:6653:7514:7903:8603:9413:10004:10241:11026:11473:11658:11914:12296:12297:12438:12517:12519:12555:12895:12986:13894:14096:14181:14394:14687:14721:21080:21212:21444:21451:21627:21666:30054,0,RBL:209.85.214.196:@gmail.com:.lbl8.mailshell.net-62.50.0.100 66.100.201.100,CacheIP:none,Bayesian:0.5,0.5,0.5,Netcheck:none,DomainCache:0,MSF:not bulk,SPF:fp,MSBL:0,DNSBL:neutral,Custom_rules:0:0:0,LFtime:25,LUA_SUMMARY:none X-HE-Tag: snake60_20e48c57c5100 X-Filterd-Recvd-Size: 5155 Received: from mail-pl1-f196.google.com (mail-pl1-f196.google.com [209.85.214.196]) by imf01.hostedemail.com (Postfix) with ESMTP for ; Mon, 16 Sep 2019 14:47:05 +0000 (UTC) Received: by mail-pl1-f196.google.com with SMTP id q24so557243plr.13 for ; Mon, 16 Sep 2019 07:47:05 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=20161025; h=from:to:cc:subject:date:message-id:in-reply-to:references :mime-version:content-transfer-encoding; bh=QbVF1PSqvP2xnQhufiGK8AJvI4iylUVt8uZ1yhy2wwc=; b=YR9U5b2Al2uZzdBiRyBtWDbXpGBtWrT4PMKL+x1AZt3LYMS+m3xzN98uaYTyO+CFUF 7IhKvR0RKbfMl53lB1Ph16DPrOSZmSqW+QKgSAysJpNWZ2zntkD2HbA5vaUegHGgsgAZ oJ+A1++3S4ceNlCIng74lZeCWfICtAWZCEdhaZQwsZKVTrdPCKh0WuGIjZhjI4aE+o5m +BxDxp98Qw4Z9Bnmonbf1saR7UI/vHeAMNjYTplEWhlTdWKZdRVyuv56bI+tizq7Sit9 GdxJfk7bGuackN22jFCiFIs8FrfIzZC0RtxbQiBgucqMNmxnwNsiCOCCLWLgf4vxn6CR 44lQ== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20161025; h=x-gm-message-state:from:to:cc:subject:date:message-id:in-reply-to :references:mime-version:content-transfer-encoding; bh=QbVF1PSqvP2xnQhufiGK8AJvI4iylUVt8uZ1yhy2wwc=; b=Bcqf7phVWDn9N/2NoxLXYQFTBdPxZpZcCtvLCMHohgs1WpymwRVSpaOvsl6x/SZ/JJ wy/8AdsGGcWEtUu7E4OaJu0UumGmwbnpNe6nZSkUiaEjvK+ja63qazdMZbvGjnLfa1Ud LXsoLpxM82b+FviIz46KOi4Xw8DfkrqrAw7UMzW15Jzw5TG7WBx7EjKgXRmR0EALeYyE TZMAWsmqIy3lw79bHAy+9FtiyUiZH2URWZSLghk2QA+xDD/SKYWs4Ez4fjZCKK8OHdUi yCZFzXCFtA7fQ/X6+CPokqOdYg5BAh15S/4xUGvF4eZcYWZRFx1+Z9oynv1pNiUeyrX0 z8Bg== X-Gm-Message-State: APjAAAVLuZ/XccEUMbW3AFXhkdosG0xkBOVRVNnu8G/9uCSzWX53B6LL O6xQWQlh7LXca+sMUq9Z9BQ= X-Google-Smtp-Source: APXvYqx+StnT131uYw7DO0XD9HWiydJc/EU947IkydJV5DEyZ1C4JCqpo96WEbZQgnCh54f/is3EFg== X-Received: by 2002:a17:902:a418:: with SMTP id p24mr161368plq.312.1568645225140; Mon, 16 Sep 2019 07:47:05 -0700 (PDT) Received: from localhost.localdomain.localdomain ([2408:823c:c11:160:b8c3:8577:bf2f:3]) by smtp.gmail.com with ESMTPSA id d190sm15036004pgc.25.2019.09.16.07.46.55 (version=TLS1_3 cipher=TLS_AES_256_GCM_SHA384 bits=256/256); Mon, 16 Sep 2019 07:47:04 -0700 (PDT) From: Pengfei Li To: akpm@linux-foundation.org Cc: vbabka@suse.cz, cl@linux.com, penberg@kernel.org, rientjes@google.com, iamjoonsoo.kim@lge.com, linux-mm@kvack.org, linux-kernel@vger.kernel.org, guro@fb.com, Pengfei Li Subject: [PATCH v5 4/7] mm, slab: Return ZERO_SIZE_ALLOC for zero sized kmalloc requests Date: Mon, 16 Sep 2019 22:45:55 +0800 Message-Id: <20190916144558.27282-5-lpf.vector@gmail.com> X-Mailer: git-send-email 2.21.0 In-Reply-To: <20190916144558.27282-1-lpf.vector@gmail.com> References: <20190916144558.27282-1-lpf.vector@gmail.com> MIME-Version: 1.0 X-Bogosity: Ham, tests=bogofilter, spamicity=0.000000, version=1.2.4 Sender: owner-linux-mm@kvack.org Precedence: bulk X-Loop: owner-majordomo@kvack.org List-ID: This is a preparation patch, just replace 0 with ZERO_SIZE_ALLOC as the return value of zero sized requests. Signed-off-by: Pengfei Li Acked-by: David Rientjes --- include/linux/slab.h | 16 +++++++++++----- 1 file changed, 11 insertions(+), 5 deletions(-) diff --git a/include/linux/slab.h b/include/linux/slab.h index e773e5764b7b..1f05f68f2c3e 100644 --- a/include/linux/slab.h +++ b/include/linux/slab.h @@ -121,14 +121,20 @@ #define SLAB_DEACTIVATED ((slab_flags_t __force)0x10000000U) /* - * ZERO_SIZE_PTR will be returned for zero sized kmalloc requests. + * ZERO_SIZE_ALLOC will be returned by kmalloc_index() if it was zero sized + * requests. * + * After that, ZERO_SIZE_PTR will be returned by the function that called + * kmalloc_index(). + * Dereferencing ZERO_SIZE_PTR will lead to a distinct access fault. * * ZERO_SIZE_PTR can be passed to kfree though in the same way that NULL can. * Both make kfree a no-op. */ -#define ZERO_SIZE_PTR ((void *)16) +#define ZERO_SIZE_ALLOC (UINT_MAX) + +#define ZERO_SIZE_PTR ((void *)16) #define ZERO_OR_NULL_PTR(x) ((unsigned long)(x) <= \ (unsigned long)ZERO_SIZE_PTR) @@ -350,7 +356,7 @@ static __always_inline enum kmalloc_cache_type kmalloc_type(gfp_t flags) static __always_inline unsigned int kmalloc_index(size_t size) { if (!size) - return 0; + return ZERO_SIZE_ALLOC; if (size <= KMALLOC_MIN_SIZE) return KMALLOC_SHIFT_LOW; @@ -546,7 +552,7 @@ static __always_inline void *kmalloc(size_t size, gfp_t flags) #ifndef CONFIG_SLOB index = kmalloc_index(size); - if (!index) + if (index == ZERO_SIZE_ALLOC) return ZERO_SIZE_PTR; return kmem_cache_alloc_trace( @@ -564,7 +570,7 @@ static __always_inline void *kmalloc_node(size_t size, gfp_t flags, int node) size <= KMALLOC_MAX_CACHE_SIZE) { unsigned int i = kmalloc_index(size); - if (!i) + if (i == ZERO_SIZE_ALLOC) return ZERO_SIZE_PTR; return kmem_cache_alloc_node_trace( From patchwork Mon Sep 16 14:45:56 2019 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Pengfei Li X-Patchwork-Id: 11147195 Return-Path: Received: from mail.kernel.org (pdx-korg-mail-1.web.codeaurora.org [172.30.200.123]) by pdx-korg-patchwork-2.web.codeaurora.org (Postfix) with ESMTP id AC86B1708 for ; Mon, 16 Sep 2019 14:47:21 +0000 (UTC) Received: from kanga.kvack.org (kanga.kvack.org [205.233.56.17]) by mail.kernel.org (Postfix) with ESMTP id 5D761206C2 for ; Mon, 16 Sep 2019 14:47:21 +0000 (UTC) Authentication-Results: mail.kernel.org; dkim=fail reason="signature verification failed" (2048-bit key) header.d=gmail.com header.i=@gmail.com header.b="FmwT2jkk" DMARC-Filter: OpenDMARC Filter v1.3.2 mail.kernel.org 5D761206C2 Authentication-Results: mail.kernel.org; dmarc=fail (p=none dis=none) header.from=gmail.com Authentication-Results: mail.kernel.org; spf=pass smtp.mailfrom=owner-linux-mm@kvack.org Received: by kanga.kvack.org (Postfix) id 8299B6B000E; Mon, 16 Sep 2019 10:47:20 -0400 (EDT) Delivered-To: linux-mm-outgoing@kvack.org Received: by kanga.kvack.org (Postfix, from userid 40) id 7D9626B0010; Mon, 16 Sep 2019 10:47:20 -0400 (EDT) X-Original-To: int-list-linux-mm@kvack.org X-Delivered-To: int-list-linux-mm@kvack.org Received: by kanga.kvack.org (Postfix, from userid 63042) id 6C9D96B0266; Mon, 16 Sep 2019 10:47:20 -0400 (EDT) X-Original-To: linux-mm@kvack.org X-Delivered-To: linux-mm@kvack.org Received: from forelay.hostedemail.com (smtprelay0151.hostedemail.com [216.40.44.151]) by kanga.kvack.org (Postfix) with ESMTP id 44B0E6B000E for ; Mon, 16 Sep 2019 10:47:20 -0400 (EDT) Received: from smtpin01.hostedemail.com (10.5.19.251.rfc1918.com [10.5.19.251]) by forelay01.hostedemail.com (Postfix) with SMTP id E0555180AD802 for ; Mon, 16 Sep 2019 14:47:19 +0000 (UTC) X-FDA: 75941061798.01.shelf24_22c36c1176122 X-Spam-Summary: 2,0,0,cbb1cc9df5b4b15e,d41d8cd98f00b204,lpf.vector@gmail.com,:akpm@linux-foundation.org:vbabka@suse.cz:cl@linux.com:penberg@kernel.org:rientjes@google.com:iamjoonsoo.kim@lge.com::linux-kernel@vger.kernel.org:guro@fb.com:lpf.vector@gmail.com,RULES_HIT:41:69:327:355:379:421:541:800:960:966:967:968:973:982:988:989:1260:1311:1314:1345:1359:1437:1515:1605:1730:1747:1777:1792:1801:2106:2196:2198:2199:2200:2393:2525:2553:2559:2563:2682:2685:2693:2859:2912:2933:2937:2939:2942:2945:2947:2951:2954:3022:3138:3139:3140:3141:3142:3622:3865:3866:3867:3868:3870:3871:3872:3873:3874:3934:3936:3938:3941:3944:3947:3950:3953:3956:3959:4321:4385:4605:5007:6119:6238:6261:6653:6671:7208:7514:7875:7903:7904:8603:9025:9121:9413:9592:10004:11026:11232:11233:11473:11657:11658:11914:12043:12294:12296:12297:12438:12517:12519:12555:12679:12683:12698:12737:12895:12986:13007:13156:13228:13255:13894:14394:14687:21080:21324:21433:21444:21451:21524:21525:21611:21627:21666:21740:30012:30029:3003 0:30054: X-HE-Tag: shelf24_22c36c1176122 X-Filterd-Recvd-Size: 24936 Received: from mail-pf1-f195.google.com (mail-pf1-f195.google.com [209.85.210.195]) by imf01.hostedemail.com (Postfix) with ESMTP for ; Mon, 16 Sep 2019 14:47:18 +0000 (UTC) Received: by mail-pf1-f195.google.com with SMTP id y22so41590pfr.3 for ; Mon, 16 Sep 2019 07:47:18 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=20161025; h=from:to:cc:subject:date:message-id:in-reply-to:references :mime-version:content-transfer-encoding; bh=E6nBG1QtGqK/l4L6rXp8ovm9EADM5hWw+d+gucBhX0I=; b=FmwT2jkkDn4+mMDZI1w74oVEgWpjg8wGt24H3Tvm23NRHKCknY/4fINH26CmMiszwl Bw7saZ/b9Z5AZYy8iunCleuHzq7QUzn2rpmTpRjiLr4KCoJ9IbO+4K8Jipq9wHR7G33d lkEmvTHVOWDScnh6rP4/rUtHtd+JulKhj3vIkJXdNw08dk7pAJJvOCB7WeJyEUW8Rr0C S9cZOLov/zVCnba7gyNDZJ8dE9tv5XyUffDPf7V0tz2Vvbyr1ZPwwr3pkrX6hc7AI67k X3pgh09y7r0dwi1IcOXBktucGuNFN/wMyOqoTCMXqwPntP6g17R5x2ib24SeqIRr57At es/Q== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20161025; h=x-gm-message-state:from:to:cc:subject:date:message-id:in-reply-to :references:mime-version:content-transfer-encoding; bh=E6nBG1QtGqK/l4L6rXp8ovm9EADM5hWw+d+gucBhX0I=; b=ZQZfmILEEf7OhzbcjtTGKKuNk0jHuuqsAtQ885fZDZmq1YdnVBQIraVpMKQ/9jZqJT 8b5IivGUOBo9IcfOqsiHNpjwu7lMfuJdLwiBfl9eA2j3q/lpZFpqNC5Iif6J3bZEJJAZ u4EW2tynJMxjr+/TwZtVfGh2faBL+93Y/9TMMIq5S8hmY9zDb/CKd0swtfoW49HALDP+ WA4Y+rm80W2wvPugrfHwywX8xsuX54zhkP3SEnWMnHONKNmq/UW1/cSkeuJq+jXJHXkc Y/dtVollJWkCNHVRk+0Zt5RxBEs2GmNdiuro3adv33IaiQKu64EhjGV5lkFk031w52N2 7jhg== X-Gm-Message-State: APjAAAX8Su7iYEVKB4+lyaKi++wvKDj3pElp8TsmQot7B4tDI26+BMbh KlgFECGMRufawJl7p76U5Go= X-Google-Smtp-Source: APXvYqxRaqi59hHRAuKqGCohBtAwHkzTPiFw0/WD1X3WubGjDWtRHUYO922WrHZDNvu+QzVwhA3few== X-Received: by 2002:a63:4181:: with SMTP id o123mr22333874pga.164.1568645237468; Mon, 16 Sep 2019 07:47:17 -0700 (PDT) Received: from localhost.localdomain.localdomain ([2408:823c:c11:160:b8c3:8577:bf2f:3]) by smtp.gmail.com with ESMTPSA id d190sm15036004pgc.25.2019.09.16.07.47.05 (version=TLS1_3 cipher=TLS_AES_256_GCM_SHA384 bits=256/256); Mon, 16 Sep 2019 07:47:16 -0700 (PDT) From: Pengfei Li To: akpm@linux-foundation.org Cc: vbabka@suse.cz, cl@linux.com, penberg@kernel.org, rientjes@google.com, iamjoonsoo.kim@lge.com, linux-mm@kvack.org, linux-kernel@vger.kernel.org, guro@fb.com, Pengfei Li Subject: [PATCH v5 5/7] mm, slab_common: Make kmalloc_caches[] start at size KMALLOC_MIN_SIZE Date: Mon, 16 Sep 2019 22:45:56 +0800 Message-Id: <20190916144558.27282-6-lpf.vector@gmail.com> X-Mailer: git-send-email 2.21.0 In-Reply-To: <20190916144558.27282-1-lpf.vector@gmail.com> References: <20190916144558.27282-1-lpf.vector@gmail.com> MIME-Version: 1.0 X-Bogosity: Ham, tests=bogofilter, spamicity=0.000000, version=1.2.4 Sender: owner-linux-mm@kvack.org Precedence: bulk X-Loop: owner-majordomo@kvack.org List-ID: Currently, kmalloc_cache[] is not sorted by size, kmalloc_cache[0] is kmalloc-96, kmalloc_cache[1] is kmalloc-192 (when ARCH_DMA_MINALIGN is not defined). As suggested by Vlastimil Babka, "Since you're doing these cleanups, have you considered reordering kmalloc_info, size_index, kmalloc_index() etc so that sizes 96 and 192 are ordered naturally between 64, 128 and 256? That should remove various special casing such as in create_kmalloc_caches(). I can't guarantee it will be possible without breaking e.g. constant folding optimizations etc., but seems to me it should be feasible. (There are definitely more places to change than those I listed.)" So this patch reordered kmalloc_info[], kmalloc_caches[], and modified kmalloc_index() and kmalloc_slab() accordingly. As a result, there is no subtle judgment about size in create_kmalloc_caches(). And initialize kmalloc_cache[] from 0 instead of KMALLOC_SHIFT_LOW. I used ./scripts/bloat-o-meter to measure the impact of this patch on performance. The results show that it brings some benefits. Considering the size change of kmalloc_info[], the size of the code is actually about 641 bytes less. (Note: The original kmalloc_info[] was renamed to all_kmalloc_info[]) $ ./scripts/bloat-o-meter vmlinux.old vmlinux.patch_1-5 add/remove: 1/2 grow/shrink: 6/64 up/down: 872/-1113 (-241) Function old new delta all_kmalloc_info - 832 +832 crypto_create_tfm 211 225 +14 ieee80211_key_alloc 1159 1169 +10 nl80211_parse_sched_scan 2787 2795 +8 ida_alloc_range 951 955 +4 find_get_context.isra 634 637 +3 sd_probe 947 948 +1 nla_strdup 143 142 -1 trace_parser_get_init 71 69 -2 pkcs7_verify.cold 318 316 -2 xhci_alloc_tt_info 349 346 -3 units 323 320 -3 nl80211_set_reg 642 639 -3 i915_sw_fence_await_dma_fence 445 441 -4 nf_queue 671 666 -5 kmalloc_slab 102 97 -5 xhci_segment_alloc 303 297 -6 xhci_alloc_container_ctx 221 215 -6 xfrm_policy_alloc 277 271 -6 selinux_sk_alloc_security 119 113 -6 sdev_evt_send_simple 124 118 -6 sdev_evt_alloc 85 79 -6 sbitmap_queue_init_node 424 418 -6 regulatory_hint_found_beacon 400 394 -6 nf_ct_tmpl_alloc 91 85 -6 gss_create_cred 146 140 -6 drm_flip_work_allocate_task 76 70 -6 cfg80211_stop_iface 266 260 -6 cfg80211_sinfo_alloc_tid_stats 83 77 -6 cfg80211_port_authorized 218 212 -6 cfg80211_ibss_joined 341 335 -6 call_usermodehelper_setup 155 149 -6 bpf_prog_alloc_no_stats 188 182 -6 blk_alloc_flush_queue 197 191 -6 bdi_alloc_node 201 195 -6 _netlbl_catmap_getnode 253 247 -6 ____ip_mc_inc_group 481 475 -6 pkcs7_verify 1503 1495 -8 audit_log_d_path 204 196 -8 xprt_switch_alloc 145 136 -9 xhci_ring_alloc 378 369 -9 xhci_mem_init 3673 3664 -9 xhci_alloc_virt_device 505 496 -9 xhci_alloc_stream_info 727 718 -9 xhci_alloc_command 221 212 -9 tcp_sendmsg_locked 3129 3120 -9 tcp_md5_do_add 783 774 -9 tcp_fastopen_defer_connect 279 270 -9 sr_read_tochdr.isra 260 251 -9 sr_read_tocentry.isra 337 328 -9 sr_is_xa 385 376 -9 sr_get_mcn 269 260 -9 scsi_probe_and_add_lun 2947 2938 -9 ring_buffer_read_prepare 103 94 -9 request_firmware_nowait 405 396 -9 ohci_urb_enqueue 3185 3176 -9 nfs_alloc_seqid 96 87 -9 nfs4_get_state_owner 1049 1040 -9 nfs4_do_close 587 578 -9 mempool_create_node 173 164 -9 ip6_setup_cork 1030 1021 -9 dma_pool_alloc 419 410 -9 devres_open_group 223 214 -9 cfg80211_parse_mbssid_data 2406 2397 -9 __igmp_group_dropped 629 619 -10 gss_import_sec_context 187 176 -11 ip_setup_cork 374 362 -12 __i915_sw_fence_await_sw_fence 429 417 -12 kmalloc_caches 336 312 -24 create_kmalloc_caches 270 214 -56 kmalloc_cache_name 57 - -57 new_kmalloc_cache 112 - -112 kmalloc_info 432 8 -424 Total: Before=14789209, After=14788968, chg -0.00% Signed-off-by: Pengfei Li --- include/linux/slab.h | 96 ++++++++++++++++---------- mm/slab.h | 10 +-- mm/slab_common.c | 161 +++++++++++++++++-------------------------- mm/slub.c | 12 ++-- 4 files changed, 136 insertions(+), 143 deletions(-) diff --git a/include/linux/slab.h b/include/linux/slab.h index 1f05f68f2c3e..f53bb6980110 100644 --- a/include/linux/slab.h +++ b/include/linux/slab.h @@ -297,6 +297,23 @@ static inline void __check_heap_object(const void *ptr, unsigned long n, #define KMALLOC_MIN_SIZE (1 << KMALLOC_SHIFT_LOW) #endif +#define KMALLOC_CACHE_MIN_NUM (KMALLOC_SHIFT_HIGH - KMALLOC_SHIFT_LOW + 1) + +#if KMALLOC_MIN_SIZE > 64 + #define KMALLOC_SIZE_96_EXIST (0) + #define KMALLOC_SIZE_192_EXIST (0) +#elif KMALLOC_MIN_SIZE > 32 + #define KMALLOC_SIZE_96_EXIST (0) + #define KMALLOC_SIZE_192_EXIST (1) +#else + #define KMALLOC_SIZE_96_EXIST (1) + #define KMALLOC_SIZE_192_EXIST (1) +#endif + +#define KMALLOC_CACHE_NUM (KMALLOC_CACHE_MIN_NUM \ + + KMALLOC_SIZE_96_EXIST \ + + KMALLOC_SIZE_192_EXIST) + /* * This restriction comes from byte sized index implementation. * Page size is normally 2^12 bytes and, in this case, if we want to use @@ -323,7 +340,7 @@ enum kmalloc_cache_type { #ifndef CONFIG_SLOB extern struct kmem_cache * -kmalloc_caches[NR_KMALLOC_TYPES][KMALLOC_SHIFT_HIGH + 1]; +kmalloc_caches[NR_KMALLOC_TYPES][KMALLOC_CACHE_NUM]; static __always_inline enum kmalloc_cache_type kmalloc_type(gfp_t flags) { @@ -345,13 +362,18 @@ static __always_inline enum kmalloc_cache_type kmalloc_type(gfp_t flags) #endif } +/* kmalloc_index adjust size: (0, 96] */ +#define KMALLOC_IDX_ADJ_0 (KMALLOC_SHIFT_LOW) + +/* kmalloc_index adjust size: (96, 192] */ +#define KMALLOC_IDX_ADJ_1 (KMALLOC_IDX_ADJ_0 - KMALLOC_SIZE_96_EXIST) + +/* kmalloc_index adjust size: (192, N] */ +#define KMALLOC_IDX_ADJ_2 (KMALLOC_IDX_ADJ_1 - KMALLOC_SIZE_192_EXIST) + /* * Figure out which kmalloc slab an allocation of a certain size * belongs to. - * 0 = zero alloc - * 1 = 65 .. 96 bytes - * 2 = 129 .. 192 bytes - * n = 2^(n-1)+1 .. 2^n */ static __always_inline unsigned int kmalloc_index(size_t size) { @@ -359,36 +381,40 @@ static __always_inline unsigned int kmalloc_index(size_t size) return ZERO_SIZE_ALLOC; if (size <= KMALLOC_MIN_SIZE) - return KMALLOC_SHIFT_LOW; - - if (KMALLOC_MIN_SIZE <= 32 && size > 64 && size <= 96) - return 1; - if (KMALLOC_MIN_SIZE <= 64 && size > 128 && size <= 192) - return 2; - if (size <= 8) return 3; - if (size <= 16) return 4; - if (size <= 32) return 5; - if (size <= 64) return 6; - if (size <= 128) return 7; - if (size <= 256) return 8; - if (size <= 512) return 9; - if (size <= 1024) return 10; - if (size <= 2 * 1024) return 11; - if (size <= 4 * 1024) return 12; - if (size <= 8 * 1024) return 13; - if (size <= 16 * 1024) return 14; - if (size <= 32 * 1024) return 15; - if (size <= 64 * 1024) return 16; - if (size <= 128 * 1024) return 17; - if (size <= 256 * 1024) return 18; - if (size <= 512 * 1024) return 19; - if (size <= 1024 * 1024) return 20; - if (size <= 2 * 1024 * 1024) return 21; - if (size <= 4 * 1024 * 1024) return 22; - if (size <= 8 * 1024 * 1024) return 23; - if (size <= 16 * 1024 * 1024) return 24; - if (size <= 32 * 1024 * 1024) return 25; - if (size <= 64 * 1024 * 1024) return 26; + return 0; + +#if KMALLOC_SIZE_96_EXIST == 1 + if (size > 64 && size <= 96) return (7 - KMALLOC_IDX_ADJ_0); +#endif + +#if KMALLOC_SIZE_192_EXIST == 1 + if (size > 128 && size <= 192) return (8 - KMALLOC_IDX_ADJ_1); +#endif + + if (size <= 8) return ( 3 - KMALLOC_IDX_ADJ_0); + if (size <= 16) return ( 4 - KMALLOC_IDX_ADJ_0); + if (size <= 32) return ( 5 - KMALLOC_IDX_ADJ_0); + if (size <= 64) return ( 6 - KMALLOC_IDX_ADJ_0); + if (size <= 128) return ( 7 - KMALLOC_IDX_ADJ_1); + if (size <= 256) return ( 8 - KMALLOC_IDX_ADJ_2); + if (size <= 512) return ( 9 - KMALLOC_IDX_ADJ_2); + if (size <= 1024) return (10 - KMALLOC_IDX_ADJ_2); + if (size <= 2 * 1024) return (11 - KMALLOC_IDX_ADJ_2); + if (size <= 4 * 1024) return (12 - KMALLOC_IDX_ADJ_2); + if (size <= 8 * 1024) return (13 - KMALLOC_IDX_ADJ_2); + if (size <= 16 * 1024) return (14 - KMALLOC_IDX_ADJ_2); + if (size <= 32 * 1024) return (15 - KMALLOC_IDX_ADJ_2); + if (size <= 64 * 1024) return (16 - KMALLOC_IDX_ADJ_2); + if (size <= 128 * 1024) return (17 - KMALLOC_IDX_ADJ_2); + if (size <= 256 * 1024) return (18 - KMALLOC_IDX_ADJ_2); + if (size <= 512 * 1024) return (19 - KMALLOC_IDX_ADJ_2); + if (size <= 1024 * 1024) return (20 - KMALLOC_IDX_ADJ_2); + if (size <= 2 * 1024 * 1024) return (21 - KMALLOC_IDX_ADJ_2); + if (size <= 4 * 1024 * 1024) return (22 - KMALLOC_IDX_ADJ_2); + if (size <= 8 * 1024 * 1024) return (23 - KMALLOC_IDX_ADJ_2); + if (size <= 16 * 1024 * 1024) return (24 - KMALLOC_IDX_ADJ_2); + if (size <= 32 * 1024 * 1024) return (25 - KMALLOC_IDX_ADJ_2); + if (size <= 64 * 1024 * 1024) return (26 - KMALLOC_IDX_ADJ_2); BUG(); /* Will never be reached. Needed because the compiler may complain */ diff --git a/mm/slab.h b/mm/slab.h index 2fc8f956906a..3ada65ef1118 100644 --- a/mm/slab.h +++ b/mm/slab.h @@ -63,6 +63,11 @@ enum slab_state { FULL /* Everything is working */ }; +struct kmalloc_info_struct { + const char *name[NR_KMALLOC_TYPES]; + unsigned int size; +}; + extern enum slab_state slab_state; /* The slab cache mutex protects the management structures during changes */ @@ -75,10 +80,7 @@ extern struct list_head slab_caches; extern struct kmem_cache *kmem_cache; /* A table of kmalloc cache names and sizes */ -extern const struct kmalloc_info_struct { - const char *name[NR_KMALLOC_TYPES]; - unsigned int size; -} kmalloc_info[]; +extern const struct kmalloc_info_struct * const kmalloc_info; #ifndef CONFIG_SLOB /* Kmalloc array related functions */ diff --git a/mm/slab_common.c b/mm/slab_common.c index 23054b8b75b6..eeef5ac8d04d 100644 --- a/mm/slab_common.c +++ b/mm/slab_common.c @@ -1028,7 +1028,7 @@ struct kmem_cache *__init create_kmalloc_cache(const char *name, } struct kmem_cache * -kmalloc_caches[NR_KMALLOC_TYPES][KMALLOC_SHIFT_HIGH + 1] __ro_after_init = +kmalloc_caches[NR_KMALLOC_TYPES][KMALLOC_CACHE_NUM] __ro_after_init = { /* initialization for https://bugs.llvm.org/show_bug.cgi?id=42570 */ }; EXPORT_SYMBOL(kmalloc_caches); @@ -1039,30 +1039,30 @@ EXPORT_SYMBOL(kmalloc_caches); * fls. */ static u8 size_index[24] __ro_after_init = { - 3, /* 8 */ - 4, /* 16 */ - 5, /* 24 */ - 5, /* 32 */ - 6, /* 40 */ - 6, /* 48 */ - 6, /* 56 */ - 6, /* 64 */ - 1, /* 72 */ - 1, /* 80 */ - 1, /* 88 */ - 1, /* 96 */ - 7, /* 104 */ - 7, /* 112 */ - 7, /* 120 */ - 7, /* 128 */ - 2, /* 136 */ - 2, /* 144 */ - 2, /* 152 */ - 2, /* 160 */ - 2, /* 168 */ - 2, /* 176 */ - 2, /* 184 */ - 2 /* 192 */ + (3 - KMALLOC_IDX_ADJ_0), /* 8 */ + (4 - KMALLOC_IDX_ADJ_0), /* 16 */ + (5 - KMALLOC_IDX_ADJ_0), /* 24 */ + (5 - KMALLOC_IDX_ADJ_0), /* 32 */ + (6 - KMALLOC_IDX_ADJ_0), /* 40 */ + (6 - KMALLOC_IDX_ADJ_0), /* 48 */ + (6 - KMALLOC_IDX_ADJ_0), /* 56 */ + (6 - KMALLOC_IDX_ADJ_0), /* 64 */ + (7 - KMALLOC_IDX_ADJ_0), /* 72 */ + (7 - KMALLOC_IDX_ADJ_0), /* 80 */ + (7 - KMALLOC_IDX_ADJ_0), /* 88 */ + (7 - KMALLOC_IDX_ADJ_0), /* 96 */ + (7 - KMALLOC_IDX_ADJ_1), /* 104 */ + (7 - KMALLOC_IDX_ADJ_1), /* 112 */ + (7 - KMALLOC_IDX_ADJ_1), /* 120 */ + (7 - KMALLOC_IDX_ADJ_1), /* 128 */ + (8 - KMALLOC_IDX_ADJ_1), /* 136 */ + (8 - KMALLOC_IDX_ADJ_1), /* 144 */ + (8 - KMALLOC_IDX_ADJ_1), /* 152 */ + (8 - KMALLOC_IDX_ADJ_1), /* 160 */ + (8 - KMALLOC_IDX_ADJ_1), /* 168 */ + (8 - KMALLOC_IDX_ADJ_1), /* 176 */ + (8 - KMALLOC_IDX_ADJ_1), /* 184 */ + (8 - KMALLOC_IDX_ADJ_1), /* 192 */ }; static inline unsigned int size_index_elem(unsigned int bytes) @@ -1086,12 +1086,19 @@ struct kmem_cache *kmalloc_slab(size_t size, gfp_t flags) } else { if (WARN_ON_ONCE(size > KMALLOC_MAX_CACHE_SIZE)) return NULL; - index = fls(size - 1); + + index = fls(size - 1) - KMALLOC_IDX_ADJ_2; } return kmalloc_caches[kmalloc_type(flags)][index]; } + +#define KMALLOC_INFO_SHIFT_LOW (3) + +#define KMALLOC_INFO_START_IDX (KMALLOC_SHIFT_LOW - KMALLOC_INFO_SHIFT_LOW) + + #ifdef CONFIG_ZONE_DMA #define INIT_KMALLOC_INFO(__size, __short_size) \ { \ @@ -1110,40 +1117,35 @@ struct kmem_cache *kmalloc_slab(size_t size, gfp_t flags) #endif /* - * kmalloc_info[] is to make slub_debug=,kmalloc-xx option work at boot time. - * kmalloc_index() supports up to 2^26=64MB, so the final entry of the table is - * kmalloc-67108864. + * all_kmalloc_info[] is to make slub_debug=, kmalloc-xx option work at boot + * time. kmalloc_index() supports up to 2^26=64MB, so the final entry of the + * table is kmalloc-67108864. */ -const struct kmalloc_info_struct kmalloc_info[] __initconst = { - INIT_KMALLOC_INFO(0, 0), - INIT_KMALLOC_INFO(96, 96), - INIT_KMALLOC_INFO(192, 192), - INIT_KMALLOC_INFO(8, 8), - INIT_KMALLOC_INFO(16, 16), - INIT_KMALLOC_INFO(32, 32), - INIT_KMALLOC_INFO(64, 64), - INIT_KMALLOC_INFO(128, 128), - INIT_KMALLOC_INFO(256, 256), - INIT_KMALLOC_INFO(512, 512), - INIT_KMALLOC_INFO(1024, 1k), - INIT_KMALLOC_INFO(2048, 2k), - INIT_KMALLOC_INFO(4096, 4k), - INIT_KMALLOC_INFO(8192, 8k), - INIT_KMALLOC_INFO(16384, 16k), - INIT_KMALLOC_INFO(32768, 32k), - INIT_KMALLOC_INFO(65536, 64k), - INIT_KMALLOC_INFO(131072, 128k), - INIT_KMALLOC_INFO(262144, 256k), - INIT_KMALLOC_INFO(524288, 512k), - INIT_KMALLOC_INFO(1048576, 1M), - INIT_KMALLOC_INFO(2097152, 2M), - INIT_KMALLOC_INFO(4194304, 4M), - INIT_KMALLOC_INFO(8388608, 8M), - INIT_KMALLOC_INFO(16777216, 16M), - INIT_KMALLOC_INFO(33554432, 32M), - INIT_KMALLOC_INFO(67108864, 64M) +static const struct kmalloc_info_struct all_kmalloc_info[] __initconst = { + INIT_KMALLOC_INFO( 8, 8), INIT_KMALLOC_INFO( 16, 16), + INIT_KMALLOC_INFO( 32, 32), INIT_KMALLOC_INFO( 64, 64), +#if KMALLOC_SIZE_96_EXIST == 1 + INIT_KMALLOC_INFO( 96, 96), +#endif + INIT_KMALLOC_INFO( 128, 128), +#if KMALLOC_SIZE_192_EXIST == 1 + INIT_KMALLOC_INFO( 192, 192), +#endif + INIT_KMALLOC_INFO( 256, 256), INIT_KMALLOC_INFO( 512, 512), + INIT_KMALLOC_INFO( 1024, 1k), INIT_KMALLOC_INFO( 2048, 2k), + INIT_KMALLOC_INFO( 4096, 4k), INIT_KMALLOC_INFO( 8192, 8k), + INIT_KMALLOC_INFO( 16384, 16k), INIT_KMALLOC_INFO( 32768, 32k), + INIT_KMALLOC_INFO( 65536, 64k), INIT_KMALLOC_INFO( 131072, 128k), + INIT_KMALLOC_INFO( 262144, 256k), INIT_KMALLOC_INFO( 524288, 512k), + INIT_KMALLOC_INFO( 1048576, 1M), INIT_KMALLOC_INFO( 2097152, 2M), + INIT_KMALLOC_INFO( 4194304, 4M), INIT_KMALLOC_INFO( 8388608, 8M), + INIT_KMALLOC_INFO(16777216, 16M), INIT_KMALLOC_INFO(33554432, 32M), + INIT_KMALLOC_INFO(67108864, 64M) }; +const struct kmalloc_info_struct * const __initconst +kmalloc_info = &all_kmalloc_info[KMALLOC_INFO_START_IDX]; + /* * Patch up the size_index table if we have strange large alignment * requirements for the kmalloc array. This is only the case for @@ -1162,33 +1164,8 @@ void __init setup_kmalloc_cache_index_table(void) BUILD_BUG_ON(KMALLOC_MIN_SIZE > 256 || (KMALLOC_MIN_SIZE & (KMALLOC_MIN_SIZE - 1))); - for (i = 8; i < KMALLOC_MIN_SIZE; i += 8) { - unsigned int elem = size_index_elem(i); - - if (elem >= ARRAY_SIZE(size_index)) - break; - size_index[elem] = KMALLOC_SHIFT_LOW; - } - - if (KMALLOC_MIN_SIZE >= 64) { - /* - * The 96 byte size cache is not used if the alignment - * is 64 byte. - */ - for (i = 64 + 8; i <= 96; i += 8) - size_index[size_index_elem(i)] = 7; - - } - - if (KMALLOC_MIN_SIZE >= 128) { - /* - * The 192 byte sized cache is not used if the alignment - * is 128 byte. Redirect kmalloc to use the 256 byte cache - * instead. - */ - for (i = 128 + 8; i <= 192; i += 8) - size_index[size_index_elem(i)] = 8; - } + for (i = 8; i < KMALLOC_MIN_SIZE && i <= 192; i += 8) + size_index[size_index_elem(i)] = 0; } static void __init @@ -1214,21 +1191,9 @@ void __init create_kmalloc_caches(slab_flags_t flags) enum kmalloc_cache_type type; for (type = KMALLOC_NORMAL; type <= KMALLOC_RECLAIM; type++) { - for (i = KMALLOC_SHIFT_LOW; i <= KMALLOC_SHIFT_HIGH; i++) { + for (i = 0; i < KMALLOC_CACHE_NUM; i++) { if (!kmalloc_caches[type][i]) new_kmalloc_cache(i, type, flags); - - /* - * Caches that are not of the two-to-the-power-of size. - * These have to be created immediately after the - * earlier power of two caches - */ - if (KMALLOC_MIN_SIZE <= 32 && i == 6 && - !kmalloc_caches[type][1]) - new_kmalloc_cache(1, type, flags); - if (KMALLOC_MIN_SIZE <= 64 && i == 7 && - !kmalloc_caches[type][2]) - new_kmalloc_cache(2, type, flags); } } @@ -1236,7 +1201,7 @@ void __init create_kmalloc_caches(slab_flags_t flags) slab_state = UP; #ifdef CONFIG_ZONE_DMA - for (i = 0; i <= KMALLOC_SHIFT_HIGH; i++) { + for (i = 0; i < KMALLOC_CACHE_NUM; i++) { struct kmem_cache *s = kmalloc_caches[KMALLOC_NORMAL][i]; if (s) { diff --git a/mm/slub.c b/mm/slub.c index 8834563cdb4b..0e92ebdcacc9 100644 --- a/mm/slub.c +++ b/mm/slub.c @@ -4711,7 +4711,7 @@ static void __init resiliency_test(void) pr_err("\n1. kmalloc-16: Clobber Redzone/next pointer 0x12->0x%p\n\n", p + 16); - validate_slab_cache(kmalloc_caches[type][4]); + validate_slab_cache(kmalloc_caches[type][1]); /* Hmmm... The next two are dangerous */ p = kzalloc(32, GFP_KERNEL); @@ -4720,33 +4720,33 @@ static void __init resiliency_test(void) p); pr_err("If allocated object is overwritten then not detectable\n\n"); - validate_slab_cache(kmalloc_caches[type][5]); + validate_slab_cache(kmalloc_caches[type][2]); p = kzalloc(64, GFP_KERNEL); p += 64 + (get_cycles() & 0xff) * sizeof(void *); *p = 0x56; pr_err("\n3. kmalloc-64: corrupting random byte 0x56->0x%p\n", p); pr_err("If allocated object is overwritten then not detectable\n\n"); - validate_slab_cache(kmalloc_caches[type][6]); + validate_slab_cache(kmalloc_caches[type][3]); pr_err("\nB. Corruption after free\n"); p = kzalloc(128, GFP_KERNEL); kfree(p); *p = 0x78; pr_err("1. kmalloc-128: Clobber first word 0x78->0x%p\n\n", p); - validate_slab_cache(kmalloc_caches[type][7]); + validate_slab_cache(kmalloc_caches[type][5]); p = kzalloc(256, GFP_KERNEL); kfree(p); p[50] = 0x9a; pr_err("\n2. kmalloc-256: Clobber 50th byte 0x9a->0x%p\n\n", p); - validate_slab_cache(kmalloc_caches[type][8]); + validate_slab_cache(kmalloc_caches[type][7]); p = kzalloc(512, GFP_KERNEL); kfree(p); p[512] = 0xab; pr_err("\n3. kmalloc-512: Clobber redzone 0xab->0x%p\n\n", p); - validate_slab_cache(kmalloc_caches[type][9]); + validate_slab_cache(kmalloc_caches[type][8]); } #else #ifdef CONFIG_SYSFS From patchwork Mon Sep 16 14:45:57 2019 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Pengfei Li X-Patchwork-Id: 11147197 Return-Path: Received: from mail.kernel.org (pdx-korg-mail-1.web.codeaurora.org [172.30.200.123]) by pdx-korg-patchwork-2.web.codeaurora.org (Postfix) with ESMTP id 2ABC91708 for ; Mon, 16 Sep 2019 14:47:30 +0000 (UTC) Received: from kanga.kvack.org (kanga.kvack.org [205.233.56.17]) by mail.kernel.org (Postfix) with ESMTP id DE703206C2 for ; Mon, 16 Sep 2019 14:47:29 +0000 (UTC) Authentication-Results: mail.kernel.org; dkim=fail reason="signature verification failed" (2048-bit key) header.d=gmail.com header.i=@gmail.com header.b="IOlY8Iiy" DMARC-Filter: OpenDMARC Filter v1.3.2 mail.kernel.org DE703206C2 Authentication-Results: mail.kernel.org; dmarc=fail (p=none dis=none) header.from=gmail.com Authentication-Results: mail.kernel.org; spf=pass smtp.mailfrom=owner-linux-mm@kvack.org Received: by kanga.kvack.org (Postfix) id 125186B0010; Mon, 16 Sep 2019 10:47:29 -0400 (EDT) Delivered-To: linux-mm-outgoing@kvack.org Received: by kanga.kvack.org (Postfix, from userid 40) id 0D6756B0266; Mon, 16 Sep 2019 10:47:29 -0400 (EDT) X-Original-To: int-list-linux-mm@kvack.org X-Delivered-To: int-list-linux-mm@kvack.org Received: by kanga.kvack.org (Postfix, from userid 63042) id F2EA56B0269; Mon, 16 Sep 2019 10:47:28 -0400 (EDT) X-Original-To: linux-mm@kvack.org X-Delivered-To: linux-mm@kvack.org Received: from forelay.hostedemail.com (smtprelay0116.hostedemail.com [216.40.44.116]) by kanga.kvack.org (Postfix) with ESMTP id D4E986B0010 for ; Mon, 16 Sep 2019 10:47:28 -0400 (EDT) Received: from smtpin15.hostedemail.com (10.5.19.251.rfc1918.com [10.5.19.251]) by forelay02.hostedemail.com (Postfix) with SMTP id 8C18862D0 for ; Mon, 16 Sep 2019 14:47:28 +0000 (UTC) X-FDA: 75941062176.15.taste29_24181e529e613 X-Spam-Summary: 2,0,0,4761e1714c69a137,d41d8cd98f00b204,lpf.vector@gmail.com,:akpm@linux-foundation.org:vbabka@suse.cz:cl@linux.com:penberg@kernel.org:rientjes@google.com:iamjoonsoo.kim@lge.com::linux-kernel@vger.kernel.org:guro@fb.com:lpf.vector@gmail.com,RULES_HIT:41:69:355:379:541:800:960:968:973:982:988:989:1260:1311:1314:1345:1359:1437:1515:1535:1543:1711:1730:1747:1777:1792:1801:2198:2199:2393:2559:2562:2693:2731:3138:3139:3140:3141:3142:3354:3865:3866:3867:3868:3870:3871:3872:3874:4117:4321:4605:5007:6119:6261:6653:6671:7208:7514:7903:8603:9113:9121:9413:9592:10004:11026:11232:11473:11658:11914:12043:12296:12297:12438:12517:12519:12555:12679:12683:12895:12986:13007:13156:13228:13894:14040:14093:14181:14394:14687:14721:21080:21444:21450:21451:21627:21666:30029:30054,0,RBL:209.85.215.194:@gmail.com:.lbl8.mailshell.net-62.18.175.100 66.100.201.100,CacheIP:none,Bayesian:0.5,0.5,0.5,Netcheck:none,DomainCache:0,MSF:not bulk,SPF:fp,MSBL:0,DNSBL:neutral,Custom_rules:0:0:0,LFti me:25,LU X-HE-Tag: taste29_24181e529e613 X-Filterd-Recvd-Size: 6149 Received: from mail-pg1-f194.google.com (mail-pg1-f194.google.com [209.85.215.194]) by imf07.hostedemail.com (Postfix) with ESMTP for ; Mon, 16 Sep 2019 14:47:27 +0000 (UTC) Received: by mail-pg1-f194.google.com with SMTP id m29so143575pgc.3 for ; Mon, 16 Sep 2019 07:47:27 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=20161025; h=from:to:cc:subject:date:message-id:in-reply-to:references :mime-version:content-transfer-encoding; bh=MNNib8gt8D5ASBLKcMdRsaxiVFvW1NuIs5T2u607D5s=; b=IOlY8IiyhiINdaw4WXA78dm8UGXZyyZz9UFC2pIgNRNJVM280Qj8ycGPH7YGw7kH0r RvACjnRmqdnV5s2daW6y8GbvWAg0zD1AcgoChTbMBrCnKcXEimaQ7tn9UmoqsG9Nd0Lf aVNwYp6T5hkad6RQ5nyqEdr3M4r334LD2JBL2rniFr3EerOPXGBsrulMerGkXtovTSeT Du6uCtTZYHq1XivTzOJy+SNJVqklRUzF7kdgJLWGhkN+vXKGXY77/2UEHr/6iPyw5515 ApZMAh/hFptfbH5UZQI21J+Ps1NU6EzlidLP8IY0dZNvP/atFQtEJEJpG7a7pMr7RMIn ViHQ== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20161025; h=x-gm-message-state:from:to:cc:subject:date:message-id:in-reply-to :references:mime-version:content-transfer-encoding; bh=MNNib8gt8D5ASBLKcMdRsaxiVFvW1NuIs5T2u607D5s=; b=bJKtl04eF4GKHQWkn7MDhHdDKuxXQNO3+wnqXLNF2tvetZnA6+4fxvnSzJcckt3rpU z2apx7a2PivGMNl6ADclDBmJYo7LuIy4pG+zPjZRb0p+ZeeLSkCk7x8COqKWRE1jGYsM ykEziQbdqHFJgVc5/EOVbe6g/ZyQvPs4EQ3DiiCB/qZtDGLGsgE+Vvi5GAxZiQYxmetS JvUDKjOrBQMh0ac96928fsd09jibpn65WQZseYY9GBtZJfVCq3jADxiQhE065tRwZzk8 Pg0U4h1XfJAHwDsaCmA7ydAdOKAIFsNGGKMFZ40T3MWsIsj4O731yBU1stn8OsMS9NTb AfHA== X-Gm-Message-State: APjAAAVZy/jgzF2xIB1BzEczzvzz52NNanm82xew5YIwigqU8IYgljq0 8oJVuGxLTeDwXDHr89WOqNc= X-Google-Smtp-Source: APXvYqwe0QaroPtORfVgYBuaV2agHHnU6wFwMZPYA4cBOqnuAGgRLiJYZymup0lCl846co8IdyPR1g== X-Received: by 2002:a17:90a:b286:: with SMTP id c6mr84290pjr.1.1568645247003; Mon, 16 Sep 2019 07:47:27 -0700 (PDT) Received: from localhost.localdomain.localdomain ([2408:823c:c11:160:b8c3:8577:bf2f:3]) by smtp.gmail.com with ESMTPSA id d190sm15036004pgc.25.2019.09.16.07.47.18 (version=TLS1_3 cipher=TLS_AES_256_GCM_SHA384 bits=256/256); Mon, 16 Sep 2019 07:47:26 -0700 (PDT) From: Pengfei Li To: akpm@linux-foundation.org Cc: vbabka@suse.cz, cl@linux.com, penberg@kernel.org, rientjes@google.com, iamjoonsoo.kim@lge.com, linux-mm@kvack.org, linux-kernel@vger.kernel.org, guro@fb.com, Pengfei Li Subject: [PATCH v5 6/7] mm, slab_common: Initialize the same size of kmalloc_caches[] Date: Mon, 16 Sep 2019 22:45:57 +0800 Message-Id: <20190916144558.27282-7-lpf.vector@gmail.com> X-Mailer: git-send-email 2.21.0 In-Reply-To: <20190916144558.27282-1-lpf.vector@gmail.com> References: <20190916144558.27282-1-lpf.vector@gmail.com> MIME-Version: 1.0 X-Bogosity: Ham, tests=bogofilter, spamicity=0.000000, version=1.2.4 Sender: owner-linux-mm@kvack.org Precedence: bulk X-Loop: owner-majordomo@kvack.org List-ID: In the current implementation, KMALLOC_RECLAIM is not initialized until all the KMALLOC_NORMAL sizes have been initialized. But for a particular size, create_kmalloc_caches() can be executed faster by initializing different types of kmalloc in order. $ ./scripts/bloat-o-meter vmlinux.patch_1-5 vmlinux.patch_1-6 add/remove: 0/0 grow/shrink: 0/1 up/down: 0/-11 (-11) Function old new delta create_kmalloc_caches 214 203 -11 Total: Before=14788968, After=14788957, chg -0.00% Although the benefits are small (more judgment is made for robustness), create_kmalloc_caches() is much simpler. Besides, KMALLOC_DMA will be initialized after "slab_state = UP", this does not seem to be necessary. Commit f97d5f634d3b ("slab: Common function to create the kmalloc array") introduces create_kmalloc_caches(). And I found that for SLAB, KMALLOC_DMA is initialized before "slab_state = UP". But for SLUB, KMALLOC_DMA is initialized after "slab_state = UP". Based on this fact, I think it is okay to initialize KMALLOC_DMA before "slab_state = UP". Signed-off-by: Pengfei Li --- mm/slab_common.c | 35 +++++++++++++---------------------- 1 file changed, 13 insertions(+), 22 deletions(-) diff --git a/mm/slab_common.c b/mm/slab_common.c index eeef5ac8d04d..00f2cfc66dbd 100644 --- a/mm/slab_common.c +++ b/mm/slab_common.c @@ -1168,11 +1168,11 @@ void __init setup_kmalloc_cache_index_table(void) size_index[size_index_elem(i)] = 0; } -static void __init +static __always_inline void __init new_kmalloc_cache(int idx, enum kmalloc_cache_type type, slab_flags_t flags) { - if (type == KMALLOC_RECLAIM) - flags |= SLAB_RECLAIM_ACCOUNT; + if (kmalloc_caches[type][idx]) + return; kmalloc_caches[type][idx] = create_kmalloc_cache( kmalloc_info[idx].name[type], @@ -1188,30 +1188,21 @@ new_kmalloc_cache(int idx, enum kmalloc_cache_type type, slab_flags_t flags) void __init create_kmalloc_caches(slab_flags_t flags) { int i; - enum kmalloc_cache_type type; - for (type = KMALLOC_NORMAL; type <= KMALLOC_RECLAIM; type++) { - for (i = 0; i < KMALLOC_CACHE_NUM; i++) { - if (!kmalloc_caches[type][i]) - new_kmalloc_cache(i, type, flags); - } - } + for (i = 0; i < KMALLOC_CACHE_NUM; i++) { + new_kmalloc_cache(i, KMALLOC_NORMAL, flags); - /* Kmalloc array is now usable */ - slab_state = UP; + new_kmalloc_cache(i, KMALLOC_RECLAIM, + flags | SLAB_RECLAIM_ACCOUNT); #ifdef CONFIG_ZONE_DMA - for (i = 0; i < KMALLOC_CACHE_NUM; i++) { - struct kmem_cache *s = kmalloc_caches[KMALLOC_NORMAL][i]; - - if (s) { - kmalloc_caches[KMALLOC_DMA][i] = create_kmalloc_cache( - kmalloc_info[i].name[KMALLOC_DMA], - kmalloc_info[i].size, - SLAB_CACHE_DMA | flags, 0, 0); - } - } + new_kmalloc_cache(i, KMALLOC_DMA, + flags | SLAB_CACHE_DMA); #endif + } + + /* Kmalloc array is now usable */ + slab_state = UP; } #endif /* !CONFIG_SLOB */ From patchwork Mon Sep 16 14:45:58 2019 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Pengfei Li X-Patchwork-Id: 11147199 Return-Path: Received: from mail.kernel.org (pdx-korg-mail-1.web.codeaurora.org [172.30.200.123]) by pdx-korg-patchwork-2.web.codeaurora.org (Postfix) with ESMTP id 4C83214ED for ; Mon, 16 Sep 2019 14:47:41 +0000 (UTC) Received: from kanga.kvack.org (kanga.kvack.org [205.233.56.17]) by mail.kernel.org (Postfix) with ESMTP id F2F83206C2 for ; Mon, 16 Sep 2019 14:47:40 +0000 (UTC) Authentication-Results: mail.kernel.org; dkim=fail reason="signature verification failed" (2048-bit key) header.d=gmail.com header.i=@gmail.com header.b="AchZsL8y" DMARC-Filter: OpenDMARC Filter v1.3.2 mail.kernel.org F2F83206C2 Authentication-Results: mail.kernel.org; dmarc=fail (p=none dis=none) header.from=gmail.com Authentication-Results: mail.kernel.org; spf=pass smtp.mailfrom=owner-linux-mm@kvack.org Received: by kanga.kvack.org (Postfix) id 0B03B6B0266; Mon, 16 Sep 2019 10:47:40 -0400 (EDT) Delivered-To: linux-mm-outgoing@kvack.org Received: by kanga.kvack.org (Postfix, from userid 40) id 061766B0269; Mon, 16 Sep 2019 10:47:40 -0400 (EDT) X-Original-To: int-list-linux-mm@kvack.org X-Delivered-To: int-list-linux-mm@kvack.org Received: by kanga.kvack.org (Postfix, from userid 63042) id E69496B026A; Mon, 16 Sep 2019 10:47:39 -0400 (EDT) X-Original-To: linux-mm@kvack.org X-Delivered-To: linux-mm@kvack.org Received: from forelay.hostedemail.com (smtprelay0029.hostedemail.com [216.40.44.29]) by kanga.kvack.org (Postfix) with ESMTP id C42176B0266 for ; Mon, 16 Sep 2019 10:47:39 -0400 (EDT) Received: from smtpin25.hostedemail.com (10.5.19.251.rfc1918.com [10.5.19.251]) by forelay05.hostedemail.com (Postfix) with SMTP id 744BE181AC9AE for ; Mon, 16 Sep 2019 14:47:39 +0000 (UTC) X-FDA: 75941062638.25.eye33_25a857d616531 X-Spam-Summary: 2,0,0,dcf05e4279bb725f,d41d8cd98f00b204,lpf.vector@gmail.com,:akpm@linux-foundation.org:vbabka@suse.cz:cl@linux.com:penberg@kernel.org:rientjes@google.com:iamjoonsoo.kim@lge.com::linux-kernel@vger.kernel.org:guro@fb.com:lpf.vector@gmail.com,RULES_HIT:1:2:41:355:379:541:559:800:960:966:967:968:973:982:988:989:1260:1311:1314:1345:1359:1437:1515:1605:1730:1747:1777:1792:1801:2194:2196:2199:2200:2393:2525:2559:2563:2682:2685:2693:2859:2918:2933:2937:2939:2942:2945:2947:2951:2954:3022:3138:3139:3140:3141:3142:3865:3866:3867:3868:3871:3874:3934:3936:3938:3941:3944:3947:3950:3953:3956:3959:4052:4321:4385:4605:5007:6119:6261:6653:6671:7208:7514:7875:7903:7904:8603:9025:9121:9413:10004:11026:11473:11658:11914:12043:12294:12296:12297:12438:12517:12519:12555:12679:12698:12737:12895:12986:13007:13255:13894:14394:14687:21080:21433:21444:21451:21524:21611:21627:21666:21740:30029:30054,0,RBL:209.85.210.193:@gmail.com:.lbl8.mailshell.net-62.50.0.100 66.100.201.100,CacheIP:non e,Bayesi X-HE-Tag: eye33_25a857d616531 X-Filterd-Recvd-Size: 12799 Received: from mail-pf1-f193.google.com (mail-pf1-f193.google.com [209.85.210.193]) by imf09.hostedemail.com (Postfix) with ESMTP for ; Mon, 16 Sep 2019 14:47:38 +0000 (UTC) Received: by mail-pf1-f193.google.com with SMTP id q12so21941pff.9 for ; Mon, 16 Sep 2019 07:47:38 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=20161025; h=from:to:cc:subject:date:message-id:in-reply-to:references :mime-version:content-transfer-encoding; bh=ap/iQx5BaMbDDbZ/PwDKM/VLAAqHmAWjY/d3zJTaoAw=; b=AchZsL8y6BmUqWecrjwrDZRs3JX50sZmVyMFL+4oB7VoOrNc/t5iJWTxnY8NsH5YEj qMeD1dS0cG6M3gPTCzZKsHJJIjoi2VPObz53IzelsjfKryKMzAEjfryn045BdGi1sNOW IYhGY+eHzjT7iNX/N6uIJH9o4N0QyJFFmAU6kSPSEG3NYjNJ3mOZB9IYU40rKBG33RHT tmdu7z4cBD5synBIxIYxd5l+lnr9tILaWiJQNGxROu290GRwmH/0P/S+VDJMjmO8fn3V bql3ZTfBmNUpKJRv4ToX/dIt+N0R6p0hYu7ID9V4j+7+JeoIRc+PghJ2WsGol6Rkmigs Ftdw== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20161025; h=x-gm-message-state:from:to:cc:subject:date:message-id:in-reply-to :references:mime-version:content-transfer-encoding; bh=ap/iQx5BaMbDDbZ/PwDKM/VLAAqHmAWjY/d3zJTaoAw=; b=gw5ZdapwVh2STUFx8n6WA6P0PIgzlkW3W4R9kVhZ5HpPqL8302dSPnNZalm6ijkMBe iA0hVHQ4t2ssZkrUpd9J554rN/P1bCMBQ1hBW0eZhWcp0a56BMQoMNlQv0OLAi0qtZsE qjoaTN79hQKuggFgNsVrRH2FdDARLexfmf/dGF/d9I4oG+nuTjLnw4eta+XTNU3eL+Fa TjtAD2SWQyfv6YdbGh/IQj/0KyboSR+tKZb276FKI+RygeVZcuKtcCG/NVF0SiD/J4ex uF4u8DWzL6Za2lkPqML898ZwrZk7k/t0j1WYn1MBCZ6j/13A/cX2FROzXvgR0wPdkGEs wmRA== X-Gm-Message-State: APjAAAUe94XrygppQ6UL+pceWFJW3Eg5JywejsYg0ZUAa5H9eL/Cr/aE AaCRGowkcBi7FDNPraNIUP0= X-Google-Smtp-Source: APXvYqxbiDWKbgfGTpb2sMSxrCcFiHNB9BEpZjV/ztznrOFPIyCbL79PaQnFlxxN67IWT8Towcojxw== X-Received: by 2002:a62:76d1:: with SMTP id r200mr73535722pfc.27.1568645257576; Mon, 16 Sep 2019 07:47:37 -0700 (PDT) Received: from localhost.localdomain.localdomain ([2408:823c:c11:160:b8c3:8577:bf2f:3]) by smtp.gmail.com with ESMTPSA id d190sm15036004pgc.25.2019.09.16.07.47.27 (version=TLS1_3 cipher=TLS_AES_256_GCM_SHA384 bits=256/256); Mon, 16 Sep 2019 07:47:37 -0700 (PDT) From: Pengfei Li To: akpm@linux-foundation.org Cc: vbabka@suse.cz, cl@linux.com, penberg@kernel.org, rientjes@google.com, iamjoonsoo.kim@lge.com, linux-mm@kvack.org, linux-kernel@vger.kernel.org, guro@fb.com, Pengfei Li Subject: [PATCH v5 7/7] mm, slab_common: Modify kmalloc_caches[type][idx] to kmalloc_caches[idx][type] Date: Mon, 16 Sep 2019 22:45:58 +0800 Message-Id: <20190916144558.27282-8-lpf.vector@gmail.com> X-Mailer: git-send-email 2.21.0 In-Reply-To: <20190916144558.27282-1-lpf.vector@gmail.com> References: <20190916144558.27282-1-lpf.vector@gmail.com> MIME-Version: 1.0 X-Bogosity: Ham, tests=bogofilter, spamicity=0.000000, version=1.2.4 Sender: owner-linux-mm@kvack.org Precedence: bulk X-Loop: owner-majordomo@kvack.org List-ID: KMALLOC_NORMAL is the most frequently accessed, and kmalloc_caches[] is initialized by different types of the same size. So modifying kmalloc_caches[type][idx] to kmalloc_caches[idx][type] will benefit performance. $ ./scripts/bloat-o-meter vmlinux.patch_1-6 vmlinux.patch_1-7 add/remove: 0/0 grow/shrink: 2/57 up/down: 8/-458 (-450) Function old new delta tg3_self_test 4255 4259 +4 nf_queue 666 670 +4 kmalloc_slab 97 93 -4 i915_sw_fence_await_dma_fence 441 437 -4 __igmp_group_dropped 619 615 -4 gss_import_sec_context 176 170 -6 xhci_alloc_command 212 205 -7 xprt_switch_alloc 136 128 -8 xhci_segment_alloc 297 289 -8 xhci_ring_alloc 369 361 -8 xhci_mem_init 3664 3656 -8 xhci_alloc_virt_device 496 488 -8 xhci_alloc_tt_info 346 338 -8 xhci_alloc_stream_info 718 710 -8 xhci_alloc_container_ctx 215 207 -8 xfrm_policy_alloc 271 263 -8 tcp_sendmsg_locked 3120 3112 -8 tcp_md5_do_add 774 766 -8 tcp_fastopen_defer_connect 270 262 -8 sr_read_tochdr.isra 251 243 -8 sr_read_tocentry.isra 328 320 -8 sr_is_xa 376 368 -8 sr_get_mcn 260 252 -8 selinux_sk_alloc_security 113 105 -8 sdev_evt_send_simple 118 110 -8 sdev_evt_alloc 79 71 -8 scsi_probe_and_add_lun 2938 2930 -8 sbitmap_queue_init_node 418 410 -8 ring_buffer_read_prepare 94 86 -8 request_firmware_nowait 396 388 -8 regulatory_hint_found_beacon 394 386 -8 ohci_urb_enqueue 3176 3168 -8 nla_strdup 142 134 -8 nfs_alloc_seqid 87 79 -8 nfs4_get_state_owner 1040 1032 -8 nfs4_do_close 578 570 -8 nf_ct_tmpl_alloc 85 77 -8 mempool_create_node 164 156 -8 ip_setup_cork 362 354 -8 ip6_setup_cork 1021 1013 -8 gss_create_cred 140 132 -8 drm_flip_work_allocate_task 70 62 -8 dma_pool_alloc 410 402 -8 devres_open_group 214 206 -8 create_kmalloc_caches 203 195 -8 cfg80211_stop_iface 260 252 -8 cfg80211_sinfo_alloc_tid_stats 77 69 -8 cfg80211_port_authorized 212 204 -8 cfg80211_parse_mbssid_data 2397 2389 -8 cfg80211_ibss_joined 335 327 -8 call_usermodehelper_setup 149 141 -8 bpf_prog_alloc_no_stats 182 174 -8 blk_alloc_flush_queue 191 183 -8 bdi_alloc_node 195 187 -8 audit_log_d_path 196 188 -8 _netlbl_catmap_getnode 247 239 -8 ____ip_mc_inc_group 475 467 -8 __i915_sw_fence_await_sw_fence 417 405 -12 ida_alloc_range 955 934 -21 Total: Before=14788957, After=14788507, chg -0.00% Signed-off-by: Pengfei Li Acked-by: David Rientjes Reported-by: kernel test robot --- include/linux/slab.h | 6 +++--- mm/slab.c | 4 ++-- mm/slab_common.c | 8 ++++---- mm/slub.c | 12 ++++++------ 4 files changed, 15 insertions(+), 15 deletions(-) diff --git a/include/linux/slab.h b/include/linux/slab.h index f53bb6980110..0842db5f7053 100644 --- a/include/linux/slab.h +++ b/include/linux/slab.h @@ -340,7 +340,7 @@ enum kmalloc_cache_type { #ifndef CONFIG_SLOB extern struct kmem_cache * -kmalloc_caches[NR_KMALLOC_TYPES][KMALLOC_CACHE_NUM]; +kmalloc_caches[KMALLOC_CACHE_NUM][NR_KMALLOC_TYPES]; static __always_inline enum kmalloc_cache_type kmalloc_type(gfp_t flags) { @@ -582,7 +582,7 @@ static __always_inline void *kmalloc(size_t size, gfp_t flags) return ZERO_SIZE_PTR; return kmem_cache_alloc_trace( - kmalloc_caches[kmalloc_type(flags)][index], + kmalloc_caches[index][kmalloc_type(flags)], flags, size); #endif } @@ -600,7 +600,7 @@ static __always_inline void *kmalloc_node(size_t size, gfp_t flags, int node) return ZERO_SIZE_PTR; return kmem_cache_alloc_node_trace( - kmalloc_caches[kmalloc_type(flags)][i], + kmalloc_caches[i][kmalloc_type(flags)], flags, node, size); } #endif diff --git a/mm/slab.c b/mm/slab.c index 7bc4e90e1147..079c3e6ced1f 100644 --- a/mm/slab.c +++ b/mm/slab.c @@ -1246,7 +1246,7 @@ void __init kmem_cache_init(void) * Initialize the caches that provide memory for the kmem_cache_node * structures first. Without this, further allocations will bug. */ - kmalloc_caches[KMALLOC_NORMAL][INDEX_NODE] = create_kmalloc_cache( + kmalloc_caches[INDEX_NODE][KMALLOC_NORMAL] = create_kmalloc_cache( kmalloc_info[INDEX_NODE].name[KMALLOC_NORMAL], kmalloc_info[INDEX_NODE].size, ARCH_KMALLOC_FLAGS, 0, @@ -1263,7 +1263,7 @@ void __init kmem_cache_init(void) for_each_online_node(nid) { init_list(kmem_cache, &init_kmem_cache_node[CACHE_CACHE + nid], nid); - init_list(kmalloc_caches[KMALLOC_NORMAL][INDEX_NODE], + init_list(kmalloc_caches[INDEX_NODE][KMALLOC_NORMAL], &init_kmem_cache_node[SIZE_NODE + nid], nid); } } diff --git a/mm/slab_common.c b/mm/slab_common.c index 00f2cfc66dbd..03d4c1521d21 100644 --- a/mm/slab_common.c +++ b/mm/slab_common.c @@ -1028,7 +1028,7 @@ struct kmem_cache *__init create_kmalloc_cache(const char *name, } struct kmem_cache * -kmalloc_caches[NR_KMALLOC_TYPES][KMALLOC_CACHE_NUM] __ro_after_init = +kmalloc_caches[KMALLOC_CACHE_NUM][NR_KMALLOC_TYPES] __ro_after_init = { /* initialization for https://bugs.llvm.org/show_bug.cgi?id=42570 */ }; EXPORT_SYMBOL(kmalloc_caches); @@ -1090,7 +1090,7 @@ struct kmem_cache *kmalloc_slab(size_t size, gfp_t flags) index = fls(size - 1) - KMALLOC_IDX_ADJ_2; } - return kmalloc_caches[kmalloc_type(flags)][index]; + return kmalloc_caches[index][kmalloc_type(flags)]; } @@ -1171,10 +1171,10 @@ void __init setup_kmalloc_cache_index_table(void) static __always_inline void __init new_kmalloc_cache(int idx, enum kmalloc_cache_type type, slab_flags_t flags) { - if (kmalloc_caches[type][idx]) + if (kmalloc_caches[idx][type]) return; - kmalloc_caches[type][idx] = create_kmalloc_cache( + kmalloc_caches[idx][type] = create_kmalloc_cache( kmalloc_info[idx].name[type], kmalloc_info[idx].size, flags, 0, kmalloc_info[idx].size); diff --git a/mm/slub.c b/mm/slub.c index 0e92ebdcacc9..e87243a16768 100644 --- a/mm/slub.c +++ b/mm/slub.c @@ -4711,7 +4711,7 @@ static void __init resiliency_test(void) pr_err("\n1. kmalloc-16: Clobber Redzone/next pointer 0x12->0x%p\n\n", p + 16); - validate_slab_cache(kmalloc_caches[type][1]); + validate_slab_cache(kmalloc_caches[1][type]); /* Hmmm... The next two are dangerous */ p = kzalloc(32, GFP_KERNEL); @@ -4720,33 +4720,33 @@ static void __init resiliency_test(void) p); pr_err("If allocated object is overwritten then not detectable\n\n"); - validate_slab_cache(kmalloc_caches[type][2]); + validate_slab_cache(kmalloc_caches[2][type]); p = kzalloc(64, GFP_KERNEL); p += 64 + (get_cycles() & 0xff) * sizeof(void *); *p = 0x56; pr_err("\n3. kmalloc-64: corrupting random byte 0x56->0x%p\n", p); pr_err("If allocated object is overwritten then not detectable\n\n"); - validate_slab_cache(kmalloc_caches[type][3]); + validate_slab_cache(kmalloc_caches[3][type]); pr_err("\nB. Corruption after free\n"); p = kzalloc(128, GFP_KERNEL); kfree(p); *p = 0x78; pr_err("1. kmalloc-128: Clobber first word 0x78->0x%p\n\n", p); - validate_slab_cache(kmalloc_caches[type][5]); + validate_slab_cache(kmalloc_caches[5][type]); p = kzalloc(256, GFP_KERNEL); kfree(p); p[50] = 0x9a; pr_err("\n2. kmalloc-256: Clobber 50th byte 0x9a->0x%p\n\n", p); - validate_slab_cache(kmalloc_caches[type][7]); + validate_slab_cache(kmalloc_caches[7][type]); p = kzalloc(512, GFP_KERNEL); kfree(p); p[512] = 0xab; pr_err("\n3. kmalloc-512: Clobber redzone 0xab->0x%p\n\n", p); - validate_slab_cache(kmalloc_caches[type][8]); + validate_slab_cache(kmalloc_caches[8][type]); } #else #ifdef CONFIG_SYSFS