From patchwork Sun Sep 15 16:51:10 2019 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Pengfei Li X-Patchwork-Id: 11146001 Return-Path: Received: from mail.kernel.org (pdx-korg-mail-1.web.codeaurora.org [172.30.200.123]) by pdx-korg-patchwork-2.web.codeaurora.org (Postfix) with ESMTP id 0AC271599 for ; Sun, 15 Sep 2019 16:52:07 +0000 (UTC) Received: from kanga.kvack.org (kanga.kvack.org [205.233.56.17]) by mail.kernel.org (Postfix) with ESMTP id BF28B214DA for ; Sun, 15 Sep 2019 16:52:06 +0000 (UTC) Authentication-Results: mail.kernel.org; dkim=fail reason="signature verification failed" (2048-bit key) header.d=gmail.com header.i=@gmail.com header.b="HxyRRhku" DMARC-Filter: OpenDMARC Filter v1.3.2 mail.kernel.org BF28B214DA Authentication-Results: mail.kernel.org; dmarc=fail (p=none dis=none) header.from=gmail.com Authentication-Results: mail.kernel.org; spf=pass smtp.mailfrom=owner-linux-mm@kvack.org Received: by kanga.kvack.org (Postfix) id 770C56B0006; Sun, 15 Sep 2019 12:52:05 -0400 (EDT) Delivered-To: linux-mm-outgoing@kvack.org Received: by kanga.kvack.org (Postfix, from userid 40) id 6F9FE6B0007; Sun, 15 Sep 2019 12:52:05 -0400 (EDT) X-Original-To: int-list-linux-mm@kvack.org X-Delivered-To: int-list-linux-mm@kvack.org Received: by kanga.kvack.org (Postfix, from userid 63042) id 5E8576B0008; Sun, 15 Sep 2019 12:52:05 -0400 (EDT) X-Original-To: linux-mm@kvack.org X-Delivered-To: linux-mm@kvack.org Received: from forelay.hostedemail.com (smtprelay0047.hostedemail.com [216.40.44.47]) by kanga.kvack.org (Postfix) with ESMTP id 380FE6B0006 for ; Sun, 15 Sep 2019 12:52:05 -0400 (EDT) Received: from smtpin18.hostedemail.com (10.5.19.251.rfc1918.com [10.5.19.251]) by forelay04.hostedemail.com (Postfix) with SMTP id D40843AB7 for ; Sun, 15 Sep 2019 16:52:04 +0000 (UTC) X-FDA: 75937747368.18.fly07_2f30f51d5562f X-Spam-Summary: 2,0,0,7895d62772edf353,d41d8cd98f00b204,lpf.vector@gmail.com,:akpm@linux-foundation.org:vbabka@suse.cz:cl@linux.com:penberg@kernel.org:rientjes@google.com:iamjoonsoo.kim@lge.com::linux-kernel@vger.kernel.org:guro@fb.com:lpf.vector@gmail.com,RULES_HIT:2:41:69:355:379:421:541:800:960:968:973:981:988:989:1260:1311:1314:1345:1359:1437:1500:1515:1535:1605:1606:1730:1747:1777:1792:1801:2198:2199:2393:2559:2562:2693:3138:3139:3140:3141:3142:3865:3866:3867:3868:3870:3871:3874:4120:4321:4605:5007:6238:6261:6653:7514:7903:8603:9413:9592:10004:11026:11232:11473:11658:11914:12043:12291:12296:12297:12438:12517:12519:12555:12683:12895:12986:13161:13229:13894:14394:14687:21080:21444:21450:21451:21627:21666:21740:30029:30030:30054:30064,0,RBL:209.85.214.193:@gmail.com:.lbl8.mailshell.net-62.50.0.100 66.100.201.100,CacheIP:none,Bayesian:0.5,0.5,0.5,Netcheck:none,DomainCache:0,MSF:not bulk,SPF:fp,MSBL:0,DNSBL:neutral,Custom_rules:0:0:0,LFtime:26,LUA_SUMMARY:none X-HE-Tag: fly07_2f30f51d5562f X-Filterd-Recvd-Size: 9003 Received: from mail-pl1-f193.google.com (mail-pl1-f193.google.com [209.85.214.193]) by imf50.hostedemail.com (Postfix) with ESMTP for ; Sun, 15 Sep 2019 16:52:04 +0000 (UTC) Received: by mail-pl1-f193.google.com with SMTP id e5so4487658pls.9 for ; Sun, 15 Sep 2019 09:52:04 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=20161025; h=from:to:cc:subject:date:message-id:in-reply-to:references :mime-version:content-transfer-encoding; bh=7ScgMFAOOXn7zrz2ktcww61aqak7m9GXt4/M+m23yyA=; b=HxyRRhku+r+TWCBAeSOnIQ/EV7x2QfkHNrITn82kiYpKsbeLxWalOoqLcGZiMO2z/6 6HXZ+bPn45R+gDk3qwX/jsrW3TFWLiqh28peqMPYfQqYt3Y1rL+WGb4dDVFLKRm358yd dnm7Wu+6UbpO3K+7d09az1olEAkvVB1KE+beZAfVQ9zfYBxOKgcuV86HrnomhJJydfpG 0pFUeiFklihtM5JyCa/q8JzMCKUpADTqp6yaZNjdEb88bnnQGPRcv64AQkLE1uVAvJ07 3fLlMKjJlsIb04/W+G+ZVMFGPPoNHp/waHsijysHPzgd7XGcAvJ8HZmm+BLwup2nVgpd l1sA== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20161025; h=x-gm-message-state:from:to:cc:subject:date:message-id:in-reply-to :references:mime-version:content-transfer-encoding; bh=7ScgMFAOOXn7zrz2ktcww61aqak7m9GXt4/M+m23yyA=; b=RAGU8UMOo5XKloCst5FQrh+5eu2FlUU7LUsaU/ss9xnsEef0o+x2ctYLYrXMfs4Q1b Hccr/QuWLQnxnAqwBd0c6qi0Q5Z826WWYdupeoDiwzVhINUaXWYxD51r0aVf8D0cdXgt l/Q9IgbtelEdzhVy4jLHitLaGFkEHd8bNBfGXG/4BIg7Sk7eZDsEEziXiJiw0lBHp4x1 ErzEKfgVEptc0tIQoXDp+oY/u9R3Pnb/eU+uGl6FT5g8DD7QRVE4qAXSWT7s+qRkHMTL hf0ZS5BAQatnNJnHc0rSy59O6cM2es9wNJZosoMcpYTzIadT4IgIbhK3xPvtfRcR1e2e VBSA== X-Gm-Message-State: APjAAAVRYGMUb20DZYpjY8YW8PHZRIQhlg1qBC7Ta7JDwlBkVOz9pwoP d5klIZcPdgHdzy8lbde6MM0= X-Google-Smtp-Source: APXvYqwLsuJ8IOV6Qqs3dL5Nh1ny0IcSVM+V3+VVQm14YEenWTXg4JGweX73Nhexh782NmXKFVb8iQ== X-Received: by 2002:a17:902:780c:: with SMTP id p12mr58796959pll.290.1568566323340; Sun, 15 Sep 2019 09:52:03 -0700 (PDT) Received: from localhost.localdomain.localdomain ([2408:823c:c11:160:b8c3:8577:bf2f:3]) by smtp.gmail.com with ESMTPSA id a4sm4383259pgq.6.2019.09.15.09.51.54 (version=TLS1_3 cipher=TLS_AES_256_GCM_SHA384 bits=256/256); Sun, 15 Sep 2019 09:52:02 -0700 (PDT) From: Pengfei Li To: akpm@linux-foundation.org Cc: vbabka@suse.cz, cl@linux.com, penberg@kernel.org, rientjes@google.com, iamjoonsoo.kim@lge.com, linux-mm@kvack.org, linux-kernel@vger.kernel.org, guro@fb.com, Pengfei Li Subject: [PATCH v4 1/7] mm, slab: Make kmalloc_info[] contain all types of names Date: Mon, 16 Sep 2019 00:51:10 +0800 Message-Id: <20190915165121.7237-2-lpf.vector@gmail.com> X-Mailer: git-send-email 2.21.0 In-Reply-To: <20190915165121.7237-1-lpf.vector@gmail.com> References: <20190915165121.7237-1-lpf.vector@gmail.com> MIME-Version: 1.0 X-Bogosity: Ham, tests=bogofilter, spamicity=0.000000, version=1.2.4 Sender: owner-linux-mm@kvack.org Precedence: bulk X-Loop: owner-majordomo@kvack.org List-ID: There are three types of kmalloc, KMALLOC_NORMAL, KMALLOC_RECLAIM and KMALLOC_DMA. The name of KMALLOC_NORMAL is contained in kmalloc_info[].name, but the names of KMALLOC_RECLAIM and KMALLOC_DMA are dynamically generated by kmalloc_cache_name(). This patch predefines the names of all types of kmalloc to save the time spent dynamically generating names. Besides, remove the kmalloc_cache_name() that is no longer used. Signed-off-by: Pengfei Li Acked-by: Vlastimil Babka Acked-by: Roman Gushchin --- mm/slab.c | 2 +- mm/slab.h | 2 +- mm/slab_common.c | 91 ++++++++++++++++++++++++++---------------------- 3 files changed, 51 insertions(+), 44 deletions(-) diff --git a/mm/slab.c b/mm/slab.c index 9df370558e5d..c42b6211f42e 100644 --- a/mm/slab.c +++ b/mm/slab.c @@ -1247,7 +1247,7 @@ void __init kmem_cache_init(void) * structures first. Without this, further allocations will bug. */ kmalloc_caches[KMALLOC_NORMAL][INDEX_NODE] = create_kmalloc_cache( - kmalloc_info[INDEX_NODE].name, + kmalloc_info[INDEX_NODE].name[KMALLOC_NORMAL], kmalloc_size(INDEX_NODE), ARCH_KMALLOC_FLAGS, 0, kmalloc_size(INDEX_NODE)); slab_state = PARTIAL_NODE; diff --git a/mm/slab.h b/mm/slab.h index 9057b8056b07..2fc8f956906a 100644 --- a/mm/slab.h +++ b/mm/slab.h @@ -76,7 +76,7 @@ extern struct kmem_cache *kmem_cache; /* A table of kmalloc cache names and sizes */ extern const struct kmalloc_info_struct { - const char *name; + const char *name[NR_KMALLOC_TYPES]; unsigned int size; } kmalloc_info[]; diff --git a/mm/slab_common.c b/mm/slab_common.c index 807490fe217a..002e16673581 100644 --- a/mm/slab_common.c +++ b/mm/slab_common.c @@ -1092,26 +1092,56 @@ struct kmem_cache *kmalloc_slab(size_t size, gfp_t flags) return kmalloc_caches[kmalloc_type(flags)][index]; } +#ifdef CONFIG_ZONE_DMA +#define SET_KMALLOC_SIZE(__size, __short_size) \ +{ \ + .name[KMALLOC_NORMAL] = "kmalloc-" #__short_size, \ + .name[KMALLOC_RECLAIM] = "kmalloc-rcl-" #__short_size, \ + .name[KMALLOC_DMA] = "dma-kmalloc-" #__short_size, \ + .size = __size, \ +} +#else +#define SET_KMALLOC_SIZE(__size, __short_size) \ +{ \ + .name[KMALLOC_NORMAL] = "kmalloc-" #__short_size, \ + .name[KMALLOC_RECLAIM] = "kmalloc-rcl-" #__short_size, \ + .size = __size, \ +} +#endif + /* * kmalloc_info[] is to make slub_debug=,kmalloc-xx option work at boot time. * kmalloc_index() supports up to 2^26=64MB, so the final entry of the table is * kmalloc-67108864. */ const struct kmalloc_info_struct kmalloc_info[] __initconst = { - {NULL, 0}, {"kmalloc-96", 96}, - {"kmalloc-192", 192}, {"kmalloc-8", 8}, - {"kmalloc-16", 16}, {"kmalloc-32", 32}, - {"kmalloc-64", 64}, {"kmalloc-128", 128}, - {"kmalloc-256", 256}, {"kmalloc-512", 512}, - {"kmalloc-1k", 1024}, {"kmalloc-2k", 2048}, - {"kmalloc-4k", 4096}, {"kmalloc-8k", 8192}, - {"kmalloc-16k", 16384}, {"kmalloc-32k", 32768}, - {"kmalloc-64k", 65536}, {"kmalloc-128k", 131072}, - {"kmalloc-256k", 262144}, {"kmalloc-512k", 524288}, - {"kmalloc-1M", 1048576}, {"kmalloc-2M", 2097152}, - {"kmalloc-4M", 4194304}, {"kmalloc-8M", 8388608}, - {"kmalloc-16M", 16777216}, {"kmalloc-32M", 33554432}, - {"kmalloc-64M", 67108864} + SET_KMALLOC_SIZE(0, 0), + SET_KMALLOC_SIZE(96, 96), + SET_KMALLOC_SIZE(192, 192), + SET_KMALLOC_SIZE(8, 8), + SET_KMALLOC_SIZE(16, 16), + SET_KMALLOC_SIZE(32, 32), + SET_KMALLOC_SIZE(64, 64), + SET_KMALLOC_SIZE(128, 128), + SET_KMALLOC_SIZE(256, 256), + SET_KMALLOC_SIZE(512, 512), + SET_KMALLOC_SIZE(1024, 1k), + SET_KMALLOC_SIZE(2048, 2k), + SET_KMALLOC_SIZE(4096, 4k), + SET_KMALLOC_SIZE(8192, 8k), + SET_KMALLOC_SIZE(16384, 16k), + SET_KMALLOC_SIZE(32768, 32k), + SET_KMALLOC_SIZE(65536, 64k), + SET_KMALLOC_SIZE(131072, 128k), + SET_KMALLOC_SIZE(262144, 256k), + SET_KMALLOC_SIZE(524288, 512k), + SET_KMALLOC_SIZE(1048576, 1M), + SET_KMALLOC_SIZE(2097152, 2M), + SET_KMALLOC_SIZE(4194304, 4M), + SET_KMALLOC_SIZE(8388608, 8M), + SET_KMALLOC_SIZE(16777216, 16M), + SET_KMALLOC_SIZE(33554432, 32M), + SET_KMALLOC_SIZE(67108864, 64M) }; /* @@ -1161,36 +1191,14 @@ void __init setup_kmalloc_cache_index_table(void) } } -static const char * -kmalloc_cache_name(const char *prefix, unsigned int size) -{ - - static const char units[3] = "\0kM"; - int idx = 0; - - while (size >= 1024 && (size % 1024 == 0)) { - size /= 1024; - idx++; - } - - return kasprintf(GFP_NOWAIT, "%s-%u%c", prefix, size, units[idx]); -} - static void __init new_kmalloc_cache(int idx, int type, slab_flags_t flags) { - const char *name; - - if (type == KMALLOC_RECLAIM) { + if (type == KMALLOC_RECLAIM) flags |= SLAB_RECLAIM_ACCOUNT; - name = kmalloc_cache_name("kmalloc-rcl", - kmalloc_info[idx].size); - BUG_ON(!name); - } else { - name = kmalloc_info[idx].name; - } - kmalloc_caches[type][idx] = create_kmalloc_cache(name, + kmalloc_caches[type][idx] = create_kmalloc_cache( + kmalloc_info[idx].name[type], kmalloc_info[idx].size, flags, 0, kmalloc_info[idx].size); } @@ -1232,11 +1240,10 @@ void __init create_kmalloc_caches(slab_flags_t flags) if (s) { unsigned int size = kmalloc_size(i); - const char *n = kmalloc_cache_name("dma-kmalloc", size); - BUG_ON(!n); kmalloc_caches[KMALLOC_DMA][i] = create_kmalloc_cache( - n, size, SLAB_CACHE_DMA | flags, 0, 0); + kmalloc_info[i].name[KMALLOC_DMA], + size, SLAB_CACHE_DMA | flags, 0, 0); } } #endif From patchwork Sun Sep 15 16:51:11 2019 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Pengfei Li X-Patchwork-Id: 11146003 Return-Path: Received: from mail.kernel.org (pdx-korg-mail-1.web.codeaurora.org [172.30.200.123]) by pdx-korg-patchwork-2.web.codeaurora.org (Postfix) with ESMTP id 345051599 for ; Sun, 15 Sep 2019 16:52:14 +0000 (UTC) Received: from kanga.kvack.org (kanga.kvack.org [205.233.56.17]) by mail.kernel.org (Postfix) with ESMTP id 00B95214D9 for ; Sun, 15 Sep 2019 16:52:13 +0000 (UTC) Authentication-Results: mail.kernel.org; dkim=fail reason="signature verification failed" (2048-bit key) header.d=gmail.com header.i=@gmail.com header.b="mddTC6Sg" DMARC-Filter: OpenDMARC Filter v1.3.2 mail.kernel.org 00B95214D9 Authentication-Results: mail.kernel.org; dmarc=fail (p=none dis=none) header.from=gmail.com Authentication-Results: mail.kernel.org; spf=pass smtp.mailfrom=owner-linux-mm@kvack.org Received: by kanga.kvack.org (Postfix) id 378F66B0007; Sun, 15 Sep 2019 12:52:13 -0400 (EDT) Delivered-To: linux-mm-outgoing@kvack.org Received: by kanga.kvack.org (Postfix, from userid 40) id 302406B0008; Sun, 15 Sep 2019 12:52:13 -0400 (EDT) X-Original-To: int-list-linux-mm@kvack.org X-Delivered-To: int-list-linux-mm@kvack.org Received: by kanga.kvack.org (Postfix, from userid 63042) id 1F0E76B000A; Sun, 15 Sep 2019 12:52:13 -0400 (EDT) X-Original-To: linux-mm@kvack.org X-Delivered-To: linux-mm@kvack.org Received: from forelay.hostedemail.com (smtprelay0148.hostedemail.com [216.40.44.148]) by kanga.kvack.org (Postfix) with ESMTP id EFC2E6B0007 for ; Sun, 15 Sep 2019 12:52:12 -0400 (EDT) Received: from smtpin08.hostedemail.com (10.5.19.251.rfc1918.com [10.5.19.251]) by forelay03.hostedemail.com (Postfix) with SMTP id 98962824CA2E for ; Sun, 15 Sep 2019 16:52:12 +0000 (UTC) X-FDA: 75937747704.08.bun79_30563d1227812 X-Spam-Summary: 2,0,0,11382fa99d803706,d41d8cd98f00b204,lpf.vector@gmail.com,:akpm@linux-foundation.org:vbabka@suse.cz:cl@linux.com:penberg@kernel.org:rientjes@google.com:iamjoonsoo.kim@lge.com::linux-kernel@vger.kernel.org:guro@fb.com:lpf.vector@gmail.com,RULES_HIT:41:69:355:379:541:800:960:968:973:988:989:1260:1311:1314:1345:1359:1437:1515:1535:1542:1711:1730:1747:1777:1792:2198:2199:2393:2559:2562:2693:3138:3139:3140:3141:3142:3353:3865:3866:3874:4321:4605:5007:6119:6261:6653:7514:7903:8603:9413:9592:10004:11026:11473:11658:11914:12043:12296:12297:12438:12517:12519:12555:12895:12986:13161:13229:13894:14181:14394:14687:14721:21080:21444:21451:21627:21666:30029:30054:30064,0,RBL:209.85.215.193:@gmail.com:.lbl8.mailshell.net-62.50.175.100 66.100.201.100,CacheIP:none,Bayesian:0.5,0.5,0.5,Netcheck:none,DomainCache:0,MSF:not bulk,SPF:fp,MSBL:0,DNSBL:neutral,Custom_rules:0:0:0,LFtime:27,LUA_SUMMARY:none X-HE-Tag: bun79_30563d1227812 X-Filterd-Recvd-Size: 5410 Received: from mail-pg1-f193.google.com (mail-pg1-f193.google.com [209.85.215.193]) by imf40.hostedemail.com (Postfix) with ESMTP for ; Sun, 15 Sep 2019 16:52:12 +0000 (UTC) Received: by mail-pg1-f193.google.com with SMTP id n190so18048418pgn.0 for ; Sun, 15 Sep 2019 09:52:12 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=20161025; h=from:to:cc:subject:date:message-id:in-reply-to:references :mime-version:content-transfer-encoding; bh=zr0c07NjDmSH5Cw3kJRhGheNFiNUbd4oIPE+u9OeCKU=; b=mddTC6SgYO2Ab78FK7UUhuQd1kwETldDpr5BXxpYb+fc8mge2xlIGeYMxpVaUY7nvR uOTQyCbJmz0EEvUKAWI/cQrJjSTlpzkpPRdJKMKc7X1W7v2JE4nGXDqTFvy/O/ovzm97 lXhRGG1iv1ixHyBj1NEDWSxrw+enMNU4S8QHT60tz59ACuf4xK2fN9cwytHTUk9m9Trc A9N9+gwU962IUT+ELhMevHDxVxIR6Fwy8cEdahUQyI1PY8mFjHUEEp2gnuqyC/RkuiCQ z5PuGS651NaGM7xzH6WRd5AYpNubAX4kYNWeYgqGN7SAVL1BR0L1gDaUq+6gmu3kwFN4 1qfQ== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20161025; h=x-gm-message-state:from:to:cc:subject:date:message-id:in-reply-to :references:mime-version:content-transfer-encoding; bh=zr0c07NjDmSH5Cw3kJRhGheNFiNUbd4oIPE+u9OeCKU=; b=BRglEYTcAp+CF6kQq63CKrix/jdwDXLkOf7gdNZZAsPhsIZaEQtm1HF7Wivr5SpS9W 7tg8ENfAeOXpVG+fa/p+m/ZTG8qRbJtCloWpeAs/QydIOkAY5FdDyHBt2LWpnBFxdRot QhsiaZdW8d3VcP9VamlVvwWsn2YrR2xlns6lbDUmlUKZdFkLRtZ0CD/J+s7sDS+YSaWc A4Ztgn0a9hRzH1SaqN3ledel0sELbvbfLGOKEXXABp2vq4xRnb4M2RVC5eoh1MQM0d0z 1YLgZPxPbGG4aRHC+ZGNFv1z8KJ93QMhiLZhdS6sbShgRwmLV5Vj7xQh0/880wtE+a7y p3Pw== X-Gm-Message-State: APjAAAXi51NkKuRa79fDPXGYx96C6Asgup6IPjfdDek3pMSoDLkVuLr+ UrZk1x9QWxgnilsj2wQeLKQ= X-Google-Smtp-Source: APXvYqxRn4ewzSZQkn/FFbl0qa7yiPu7uWCtjRCH+R5ndNnAWNrY7LsBcmNmfGEgVnhOaGKimOoOLg== X-Received: by 2002:a62:2d3:: with SMTP id 202mr69461597pfc.141.1568566331261; Sun, 15 Sep 2019 09:52:11 -0700 (PDT) Received: from localhost.localdomain.localdomain ([2408:823c:c11:160:b8c3:8577:bf2f:3]) by smtp.gmail.com with ESMTPSA id a4sm4383259pgq.6.2019.09.15.09.52.03 (version=TLS1_3 cipher=TLS_AES_256_GCM_SHA384 bits=256/256); Sun, 15 Sep 2019 09:52:10 -0700 (PDT) From: Pengfei Li To: akpm@linux-foundation.org Cc: vbabka@suse.cz, cl@linux.com, penberg@kernel.org, rientjes@google.com, iamjoonsoo.kim@lge.com, linux-mm@kvack.org, linux-kernel@vger.kernel.org, guro@fb.com, Pengfei Li Subject: [PATCH v4 2/7] mm, slab: Remove unused kmalloc_size() Date: Mon, 16 Sep 2019 00:51:11 +0800 Message-Id: <20190915165121.7237-3-lpf.vector@gmail.com> X-Mailer: git-send-email 2.21.0 In-Reply-To: <20190915165121.7237-1-lpf.vector@gmail.com> References: <20190915165121.7237-1-lpf.vector@gmail.com> MIME-Version: 1.0 X-Bogosity: Ham, tests=bogofilter, spamicity=0.000000, version=1.2.4 Sender: owner-linux-mm@kvack.org Precedence: bulk X-Loop: owner-majordomo@kvack.org List-ID: The size of kmalloc can be obtained from kmalloc_info[], so remove kmalloc_size() that will not be used anymore. Signed-off-by: Pengfei Li Acked-by: Vlastimil Babka Acked-by: Roman Gushchin --- include/linux/slab.h | 20 -------------------- mm/slab.c | 5 +++-- mm/slab_common.c | 5 ++--- 3 files changed, 5 insertions(+), 25 deletions(-) diff --git a/include/linux/slab.h b/include/linux/slab.h index 56c9c7eed34e..e773e5764b7b 100644 --- a/include/linux/slab.h +++ b/include/linux/slab.h @@ -557,26 +557,6 @@ static __always_inline void *kmalloc(size_t size, gfp_t flags) return __kmalloc(size, flags); } -/* - * Determine size used for the nth kmalloc cache. - * return size or 0 if a kmalloc cache for that - * size does not exist - */ -static __always_inline unsigned int kmalloc_size(unsigned int n) -{ -#ifndef CONFIG_SLOB - if (n > 2) - return 1U << n; - - if (n == 1 && KMALLOC_MIN_SIZE <= 32) - return 96; - - if (n == 2 && KMALLOC_MIN_SIZE <= 64) - return 192; -#endif - return 0; -} - static __always_inline void *kmalloc_node(size_t size, gfp_t flags, int node) { #ifndef CONFIG_SLOB diff --git a/mm/slab.c b/mm/slab.c index c42b6211f42e..7bc4e90e1147 100644 --- a/mm/slab.c +++ b/mm/slab.c @@ -1248,8 +1248,9 @@ void __init kmem_cache_init(void) */ kmalloc_caches[KMALLOC_NORMAL][INDEX_NODE] = create_kmalloc_cache( kmalloc_info[INDEX_NODE].name[KMALLOC_NORMAL], - kmalloc_size(INDEX_NODE), ARCH_KMALLOC_FLAGS, - 0, kmalloc_size(INDEX_NODE)); + kmalloc_info[INDEX_NODE].size, + ARCH_KMALLOC_FLAGS, 0, + kmalloc_info[INDEX_NODE].size); slab_state = PARTIAL_NODE; setup_kmalloc_cache_index_table(); diff --git a/mm/slab_common.c b/mm/slab_common.c index 002e16673581..8b542cfcc4f2 100644 --- a/mm/slab_common.c +++ b/mm/slab_common.c @@ -1239,11 +1239,10 @@ void __init create_kmalloc_caches(slab_flags_t flags) struct kmem_cache *s = kmalloc_caches[KMALLOC_NORMAL][i]; if (s) { - unsigned int size = kmalloc_size(i); - kmalloc_caches[KMALLOC_DMA][i] = create_kmalloc_cache( kmalloc_info[i].name[KMALLOC_DMA], - size, SLAB_CACHE_DMA | flags, 0, 0); + kmalloc_info[i].size, + SLAB_CACHE_DMA | flags, 0, 0); } } #endif From patchwork Sun Sep 15 16:51:12 2019 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Pengfei Li X-Patchwork-Id: 11146007 Return-Path: Received: from mail.kernel.org (pdx-korg-mail-1.web.codeaurora.org [172.30.200.123]) by pdx-korg-patchwork-2.web.codeaurora.org (Postfix) with ESMTP id B606F1599 for ; Sun, 15 Sep 2019 16:52:22 +0000 (UTC) Received: from kanga.kvack.org (kanga.kvack.org [205.233.56.17]) by mail.kernel.org (Postfix) with ESMTP id 82B30214D8 for ; Sun, 15 Sep 2019 16:52:22 +0000 (UTC) Authentication-Results: mail.kernel.org; dkim=fail reason="signature verification failed" (2048-bit key) header.d=gmail.com header.i=@gmail.com header.b="G+yzSSKX" DMARC-Filter: OpenDMARC Filter v1.3.2 mail.kernel.org 82B30214D8 Authentication-Results: mail.kernel.org; dmarc=fail (p=none dis=none) header.from=gmail.com Authentication-Results: mail.kernel.org; spf=pass smtp.mailfrom=owner-linux-mm@kvack.org Received: by kanga.kvack.org (Postfix) id C039D6B0008; Sun, 15 Sep 2019 12:52:21 -0400 (EDT) Delivered-To: linux-mm-outgoing@kvack.org Received: by kanga.kvack.org (Postfix, from userid 40) id B8DB16B000A; Sun, 15 Sep 2019 12:52:21 -0400 (EDT) X-Original-To: int-list-linux-mm@kvack.org X-Delivered-To: int-list-linux-mm@kvack.org Received: by kanga.kvack.org (Postfix, from userid 63042) id A546C6B000C; Sun, 15 Sep 2019 12:52:21 -0400 (EDT) X-Original-To: linux-mm@kvack.org X-Delivered-To: linux-mm@kvack.org Received: from forelay.hostedemail.com (smtprelay0011.hostedemail.com [216.40.44.11]) by kanga.kvack.org (Postfix) with ESMTP id 81A0B6B0008 for ; Sun, 15 Sep 2019 12:52:21 -0400 (EDT) Received: from smtpin27.hostedemail.com (10.5.19.251.rfc1918.com [10.5.19.251]) by forelay01.hostedemail.com (Postfix) with SMTP id 3436A180AD7C3 for ; Sun, 15 Sep 2019 16:52:21 +0000 (UTC) X-FDA: 75937748082.27.run57_31985446d4040 X-Spam-Summary: 2,0,0,fb6ba42b3c1699ee,d41d8cd98f00b204,lpf.vector@gmail.com,:akpm@linux-foundation.org:vbabka@suse.cz:cl@linux.com:penberg@kernel.org:rientjes@google.com:iamjoonsoo.kim@lge.com::linux-kernel@vger.kernel.org:guro@fb.com:lpf.vector@gmail.com,RULES_HIT:41:355:379:541:800:960:973:988:989:1260:1311:1314:1345:1359:1437:1515:1534:1541:1711:1730:1747:1777:1792:2198:2199:2393:2559:2562:3138:3139:3140:3141:3142:3352:3865:3866:3872:5007:6261:6653:7514:9413:10004:11026:11473:11658:11914:12043:12297:12438:12517:12519:12555:12895:13069:13161:13229:13255:13311:13357:13894:14096:14181:14384:14394:14687:14721:21080:21444:21450:21451:21627:21666:30054:30064,0,RBL:209.85.215.194:@gmail.com:.lbl8.mailshell.net-62.18.175.100 66.100.201.100,CacheIP:none,Bayesian:0.5,0.5,0.5,Netcheck:none,DomainCache:0,MSF:not bulk,SPF:fp,MSBL:0,DNSBL:neutral,Custom_rules:0:0:0,LFtime:28,LUA_SUMMARY:none X-HE-Tag: run57_31985446d4040 X-Filterd-Recvd-Size: 4227 Received: from mail-pg1-f194.google.com (mail-pg1-f194.google.com [209.85.215.194]) by imf24.hostedemail.com (Postfix) with ESMTP for ; Sun, 15 Sep 2019 16:52:20 +0000 (UTC) Received: by mail-pg1-f194.google.com with SMTP id u17so18021143pgi.6 for ; Sun, 15 Sep 2019 09:52:20 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=20161025; h=from:to:cc:subject:date:message-id:in-reply-to:references :mime-version:content-transfer-encoding; bh=X86RdWoaDKqJbFDIoXti2bda7+kkyeUkiLhSN5JzlKU=; b=G+yzSSKXmJhEM++iRzQ5m03oqsOK0gURMOljXVO2EqysSm4iIuA8NRRNAzOgjgp9o0 sO+oIkPpkU/Sp5z9Xvvt05SjxNgybmP59Iqv0pMgpE1rryezPSIjCAWwBApg9FKuh12a A9zRNleYomC430D/DcAJPZTKpZ0d7KXzbh05CtiInyVKSAKWWQbzoJcl+X4HS9XCKvz5 CZ5W8hKFIdkXo1bHos/qh8RTXnc2JyknMtLKzFSt9JPwYMVbCx62Mlx5XfstNU8ZtmvB GNQbyJOz56HKKnNBUbzNEybDic3axGq12Qi4DRLAQBWgVdNA84WHfhVpry3M1Yo33xLu idjQ== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20161025; h=x-gm-message-state:from:to:cc:subject:date:message-id:in-reply-to :references:mime-version:content-transfer-encoding; bh=X86RdWoaDKqJbFDIoXti2bda7+kkyeUkiLhSN5JzlKU=; b=HWTuIfC8+VtjifGM3buV1Vhzy/xv1+/coimBPtm4vVEl6sXcMty2m/P+hzyRArVS2a LbvF81YZVPNnoyCKIzOdX7I4iJYSf1ikFtootmvcg8iAdIfq42DYfnbqkKTdLEeVm2st /Wee9mXkA/fL2JbkMRkmvzFFu17Bglattff4YcmNMFiybaOcHnfLJoan3OaeLAWLNzE/ /Yi3n38sIusJsIIsMnDIFe7IFGG1i6wya5YNK6QNzMjRZJ1zClYc4MFmuRqpxHO6wvvU 7DCsUec2bZnG/x3Jef7TVORzpgotTRclLWQH+nssA4ZXXBWdqaVRrWaI90j3Ey96+BXU eK9g== X-Gm-Message-State: APjAAAXo7fyyHn06YIJm4UEUerJ3RkiwYfphV7Y/ScQr2ioeuYct+JAC RirXZQfLAKR2Sbs4rGGDytzjVkmLo2E= X-Google-Smtp-Source: APXvYqyipnT98ResURsw038tfVzIDoxKbCXlrtw3kuy4NXyjm2pDCAcW6gupvqKfpody7f93AEDgXg== X-Received: by 2002:aa7:9307:: with SMTP id 7mr8117547pfj.224.1568566340018; Sun, 15 Sep 2019 09:52:20 -0700 (PDT) Received: from localhost.localdomain.localdomain ([2408:823c:c11:160:b8c3:8577:bf2f:3]) by smtp.gmail.com with ESMTPSA id a4sm4383259pgq.6.2019.09.15.09.52.11 (version=TLS1_3 cipher=TLS_AES_256_GCM_SHA384 bits=256/256); Sun, 15 Sep 2019 09:52:19 -0700 (PDT) From: Pengfei Li To: akpm@linux-foundation.org Cc: vbabka@suse.cz, cl@linux.com, penberg@kernel.org, rientjes@google.com, iamjoonsoo.kim@lge.com, linux-mm@kvack.org, linux-kernel@vger.kernel.org, guro@fb.com, Pengfei Li Subject: [PATCH v4 3/7] mm, slab_common: use enum kmalloc_cache_type to iterate over kmalloc caches Date: Mon, 16 Sep 2019 00:51:12 +0800 Message-Id: <20190915165121.7237-4-lpf.vector@gmail.com> X-Mailer: git-send-email 2.21.0 In-Reply-To: <20190915165121.7237-1-lpf.vector@gmail.com> References: <20190915165121.7237-1-lpf.vector@gmail.com> MIME-Version: 1.0 X-Bogosity: Ham, tests=bogofilter, spamicity=0.000000, version=1.2.4 Sender: owner-linux-mm@kvack.org Precedence: bulk X-Loop: owner-majordomo@kvack.org List-ID: The type of local variable *type* of new_kmalloc_cache() should be enum kmalloc_cache_type instead of int, so correct it. Signed-off-by: Pengfei Li Acked-by: Vlastimil Babka Acked-by: Roman Gushchin --- mm/slab_common.c | 5 +++-- 1 file changed, 3 insertions(+), 2 deletions(-) diff --git a/mm/slab_common.c b/mm/slab_common.c index 8b542cfcc4f2..af45b5278fdc 100644 --- a/mm/slab_common.c +++ b/mm/slab_common.c @@ -1192,7 +1192,7 @@ void __init setup_kmalloc_cache_index_table(void) } static void __init -new_kmalloc_cache(int idx, int type, slab_flags_t flags) +new_kmalloc_cache(int idx, enum kmalloc_cache_type type, slab_flags_t flags) { if (type == KMALLOC_RECLAIM) flags |= SLAB_RECLAIM_ACCOUNT; @@ -1210,7 +1210,8 @@ new_kmalloc_cache(int idx, int type, slab_flags_t flags) */ void __init create_kmalloc_caches(slab_flags_t flags) { - int i, type; + int i; + enum kmalloc_cache_type type; for (type = KMALLOC_NORMAL; type <= KMALLOC_RECLAIM; type++) { for (i = KMALLOC_SHIFT_LOW; i <= KMALLOC_SHIFT_HIGH; i++) { From patchwork Sun Sep 15 16:51:14 2019 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Pengfei Li X-Patchwork-Id: 11146011 Return-Path: Received: from mail.kernel.org (pdx-korg-mail-1.web.codeaurora.org [172.30.200.123]) by pdx-korg-patchwork-2.web.codeaurora.org (Postfix) with ESMTP id 161AC14DB for ; Sun, 15 Sep 2019 16:52:41 +0000 (UTC) Received: from kanga.kvack.org (kanga.kvack.org [205.233.56.17]) by mail.kernel.org (Postfix) with ESMTP id CDA682171F for ; Sun, 15 Sep 2019 16:52:40 +0000 (UTC) Authentication-Results: mail.kernel.org; dkim=fail reason="signature verification failed" (2048-bit key) header.d=gmail.com header.i=@gmail.com header.b="Wv2SR5sO" DMARC-Filter: OpenDMARC Filter v1.3.2 mail.kernel.org CDA682171F Authentication-Results: mail.kernel.org; dmarc=fail (p=none dis=none) header.from=gmail.com Authentication-Results: mail.kernel.org; spf=pass smtp.mailfrom=owner-linux-mm@kvack.org Received: by kanga.kvack.org (Postfix) id 1CBFB6B0006; Sun, 15 Sep 2019 12:52:40 -0400 (EDT) Delivered-To: linux-mm-outgoing@kvack.org Received: by kanga.kvack.org (Postfix, from userid 40) id 153C76B000A; Sun, 15 Sep 2019 12:52:40 -0400 (EDT) X-Original-To: int-list-linux-mm@kvack.org X-Delivered-To: int-list-linux-mm@kvack.org Received: by kanga.kvack.org (Postfix, from userid 63042) id 0423B6B000C; Sun, 15 Sep 2019 12:52:39 -0400 (EDT) X-Original-To: linux-mm@kvack.org X-Delivered-To: linux-mm@kvack.org Received: from forelay.hostedemail.com (smtprelay0251.hostedemail.com [216.40.44.251]) by kanga.kvack.org (Postfix) with ESMTP id D4B9D6B0006 for ; Sun, 15 Sep 2019 12:52:39 -0400 (EDT) Received: from smtpin11.hostedemail.com (10.5.19.251.rfc1918.com [10.5.19.251]) by forelay02.hostedemail.com (Postfix) with SMTP id 8B0382C07 for ; Sun, 15 Sep 2019 16:52:39 +0000 (UTC) X-FDA: 75937748838.11.hole63_3441fb1e10563 X-Spam-Summary: 2,0,0,1173c88bb82d53e9,d41d8cd98f00b204,lpf.vector@gmail.com,:akpm@linux-foundation.org:vbabka@suse.cz:cl@linux.com:penberg@kernel.org:rientjes@google.com:iamjoonsoo.kim@lge.com::linux-kernel@vger.kernel.org:guro@fb.com:lpf.vector@gmail.com,RULES_HIT:41:355:379:541:800:960:966:973:988:989:1260:1311:1314:1345:1359:1437:1515:1535:1542:1711:1730:1747:1777:1792:2196:2199:2393:2559:2562:2693:2901:3138:3139:3140:3141:3142:3353:3369:3865:3866:3867:3870:3871:3872:3874:4250:4321:4385:5007:6261:6653:7514:7903:8603:9413:10004:10241:11026:11473:11658:11914:12296:12297:12438:12517:12519:12555:12895:12986:13894:14096:14181:14394:14687:14721:21080:21212:21444:21451:21627:21666:30054,0,RBL:209.85.215.195:@gmail.com:.lbl8.mailshell.net-62.18.175.100 66.100.201.100,CacheIP:none,Bayesian:0.5,0.5,0.5,Netcheck:none,DomainCache:0,MSF:not bulk,SPF:fp,MSBL:0,DNSBL:neutral,Custom_rules:0:0:0,LFtime:24,LUA_SUMMARY:none X-HE-Tag: hole63_3441fb1e10563 X-Filterd-Recvd-Size: 5098 Received: from mail-pg1-f195.google.com (mail-pg1-f195.google.com [209.85.215.195]) by imf47.hostedemail.com (Postfix) with ESMTP for ; Sun, 15 Sep 2019 16:52:38 +0000 (UTC) Received: by mail-pg1-f195.google.com with SMTP id n190so18048833pgn.0 for ; Sun, 15 Sep 2019 09:52:38 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=20161025; h=from:to:cc:subject:date:message-id:in-reply-to:references :mime-version:content-transfer-encoding; bh=nii43oMTaFtktKbnijjugyrLFfb5ZKvbiN8kMj3vNPE=; b=Wv2SR5sO7CBMS93mcNuzKaC33L131eKLxc2sWGOw4gZsA2mpeA6eOWAaQ/RIHpcvzE UYJ4XZ0cp7TvEj4HUH5RXcDkYGpi4w7RgV9JDzthjil/VX+OCBWBLiGeksGPio5nQ4W/ n+P7xO6d5cPH1Y2J1pefYxNd2a+8t/tYy1U5iv+qth1jnHJ5CU82f/XrNno5sG7w6YdU k8CQzOk8cSFIhkYRzTxSKtBvSH2vA3zRKAyWAPB3rtnYqJJ23EwyaBpsmIiHUeFdrX++ aRpLJJDVoVUYXcJKu5QRRIglkt97eQgVnqG/6gNYv3YgbZY8dd7gDsjlzpAuZyCQ/9hB GHpQ== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20161025; h=x-gm-message-state:from:to:cc:subject:date:message-id:in-reply-to :references:mime-version:content-transfer-encoding; bh=nii43oMTaFtktKbnijjugyrLFfb5ZKvbiN8kMj3vNPE=; b=jVbSCbrSZ5yJ+shiHCocJUQ7gsS1egp33wCCc7wrEVmGbKBJdnylMmv9oZIbvJ9iJN XejujOKlg/hx0kwd0HNJitDMqvkm+3NS8vgHrdPNJ7GXI0A6tyZ9yGv0UnSm3hm97tAo Wvea1Opm+6+6SNqLbg5jfZz3ffzNz0oi8gQkAhI7QyMyTeVq0kQYHa+XvCihRrWnQkCn MykRjk0rFJp3iB+NDfmkshsWwZEsdSaO9LjZClGtn7rMfrWW+DWS2+xeZeO7Py0HfiG/ q0jLr4+V+oQZJa6HbpSYs17qmvFqSHIzqxzBYkBSMWfV6vdrtLbmEMZBYQdTXqIZhWkn AV2w== X-Gm-Message-State: APjAAAUtZ+I+A0DdWAY01G8Aea8T5BHNNz9ytdAfW3x3qtfeZg7ofhzQ xC0hKkToIo8JFXXfcmpEeVQ= X-Google-Smtp-Source: APXvYqxYf54cYNQWLvZP6JklEbGw44XbGka9yQDf7cpKtKvKEjKS+BbJ7rhYqAqRIGCfavv9PckRFQ== X-Received: by 2002:aa7:8f03:: with SMTP id x3mr66745495pfr.91.1568566358307; Sun, 15 Sep 2019 09:52:38 -0700 (PDT) Received: from localhost.localdomain.localdomain ([2408:823c:c11:160:b8c3:8577:bf2f:3]) by smtp.gmail.com with ESMTPSA id a4sm4383259pgq.6.2019.09.15.09.52.29 (version=TLS1_3 cipher=TLS_AES_256_GCM_SHA384 bits=256/256); Sun, 15 Sep 2019 09:52:37 -0700 (PDT) From: Pengfei Li To: akpm@linux-foundation.org Cc: vbabka@suse.cz, cl@linux.com, penberg@kernel.org, rientjes@google.com, iamjoonsoo.kim@lge.com, linux-mm@kvack.org, linux-kernel@vger.kernel.org, guro@fb.com, Pengfei Li Subject: [PATCH v4 4/7] mm, slab: return ZERO_SIZE_ALLOC for zero sized kmalloc requests Date: Mon, 16 Sep 2019 00:51:14 +0800 Message-Id: <20190915165121.7237-6-lpf.vector@gmail.com> X-Mailer: git-send-email 2.21.0 In-Reply-To: <20190915165121.7237-1-lpf.vector@gmail.com> References: <20190915165121.7237-1-lpf.vector@gmail.com> MIME-Version: 1.0 X-Bogosity: Ham, tests=bogofilter, spamicity=0.000000, version=1.2.4 Sender: owner-linux-mm@kvack.org Precedence: bulk X-Loop: owner-majordomo@kvack.org List-ID: This is a preparation patch, just replace 0 with ZERO_SIZE_ALLOC as the return value of zero sized requests. Signed-off-by: Pengfei Li --- include/linux/slab.h | 16 +++++++++++----- 1 file changed, 11 insertions(+), 5 deletions(-) diff --git a/include/linux/slab.h b/include/linux/slab.h index e773e5764b7b..1f05f68f2c3e 100644 --- a/include/linux/slab.h +++ b/include/linux/slab.h @@ -121,14 +121,20 @@ #define SLAB_DEACTIVATED ((slab_flags_t __force)0x10000000U) /* - * ZERO_SIZE_PTR will be returned for zero sized kmalloc requests. + * ZERO_SIZE_ALLOC will be returned by kmalloc_index() if it was zero sized + * requests. * + * After that, ZERO_SIZE_PTR will be returned by the function that called + * kmalloc_index(). + * Dereferencing ZERO_SIZE_PTR will lead to a distinct access fault. * * ZERO_SIZE_PTR can be passed to kfree though in the same way that NULL can. * Both make kfree a no-op. */ -#define ZERO_SIZE_PTR ((void *)16) +#define ZERO_SIZE_ALLOC (UINT_MAX) + +#define ZERO_SIZE_PTR ((void *)16) #define ZERO_OR_NULL_PTR(x) ((unsigned long)(x) <= \ (unsigned long)ZERO_SIZE_PTR) @@ -350,7 +356,7 @@ static __always_inline enum kmalloc_cache_type kmalloc_type(gfp_t flags) static __always_inline unsigned int kmalloc_index(size_t size) { if (!size) - return 0; + return ZERO_SIZE_ALLOC; if (size <= KMALLOC_MIN_SIZE) return KMALLOC_SHIFT_LOW; @@ -546,7 +552,7 @@ static __always_inline void *kmalloc(size_t size, gfp_t flags) #ifndef CONFIG_SLOB index = kmalloc_index(size); - if (!index) + if (index == ZERO_SIZE_ALLOC) return ZERO_SIZE_PTR; return kmem_cache_alloc_trace( @@ -564,7 +570,7 @@ static __always_inline void *kmalloc_node(size_t size, gfp_t flags, int node) size <= KMALLOC_MAX_CACHE_SIZE) { unsigned int i = kmalloc_index(size); - if (!i) + if (i == ZERO_SIZE_ALLOC) return ZERO_SIZE_PTR; return kmem_cache_alloc_node_trace( From patchwork Sun Sep 15 16:51:16 2019 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Pengfei Li X-Patchwork-Id: 11146015 Return-Path: Received: from mail.kernel.org (pdx-korg-mail-1.web.codeaurora.org [172.30.200.123]) by pdx-korg-patchwork-2.web.codeaurora.org (Postfix) with ESMTP id 229991599 for ; Sun, 15 Sep 2019 16:53:06 +0000 (UTC) Received: from kanga.kvack.org (kanga.kvack.org [205.233.56.17]) by mail.kernel.org (Postfix) with ESMTP id B9D6D214DA for ; Sun, 15 Sep 2019 16:53:05 +0000 (UTC) Authentication-Results: mail.kernel.org; dkim=fail reason="signature verification failed" (2048-bit key) header.d=gmail.com header.i=@gmail.com header.b="WtgFNtXI" DMARC-Filter: OpenDMARC Filter v1.3.2 mail.kernel.org B9D6D214DA Authentication-Results: mail.kernel.org; dmarc=fail (p=none dis=none) header.from=gmail.com Authentication-Results: mail.kernel.org; spf=pass smtp.mailfrom=owner-linux-mm@kvack.org Received: by kanga.kvack.org (Postfix) id F10D26B0006; Sun, 15 Sep 2019 12:53:04 -0400 (EDT) Delivered-To: linux-mm-outgoing@kvack.org Received: by kanga.kvack.org (Postfix, from userid 40) id EC01A6B0008; Sun, 15 Sep 2019 12:53:04 -0400 (EDT) X-Original-To: int-list-linux-mm@kvack.org X-Delivered-To: int-list-linux-mm@kvack.org Received: by kanga.kvack.org (Postfix, from userid 63042) id D88C46B000C; Sun, 15 Sep 2019 12:53:04 -0400 (EDT) X-Original-To: linux-mm@kvack.org X-Delivered-To: linux-mm@kvack.org Received: from forelay.hostedemail.com (smtprelay0030.hostedemail.com [216.40.44.30]) by kanga.kvack.org (Postfix) with ESMTP id A9CC36B0006 for ; Sun, 15 Sep 2019 12:53:04 -0400 (EDT) Received: from smtpin14.hostedemail.com (10.5.19.251.rfc1918.com [10.5.19.251]) by forelay03.hostedemail.com (Postfix) with SMTP id 4EBB282437CF for ; Sun, 15 Sep 2019 16:53:04 +0000 (UTC) X-FDA: 75937749888.14.snail26_37cd5accb0e3a X-Spam-Summary: 2,0,0,09d39e00e0e6dbab,d41d8cd98f00b204,lpf.vector@gmail.com,:akpm@linux-foundation.org:vbabka@suse.cz:cl@linux.com:penberg@kernel.org:rientjes@google.com:iamjoonsoo.kim@lge.com::linux-kernel@vger.kernel.org:guro@fb.com:lpf.vector@gmail.com,RULES_HIT:41:69:327:355:379:421:541:800:960:966:967:968:973:982:988:989:1260:1311:1314:1345:1359:1437:1500:1515:1605:1730:1747:1777:1792:1801:2106:2196:2198:2199:2200:2393:2525:2553:2559:2563:2682:2685:2693:2859:2912:2933:2937:2939:2942:2945:2947:2951:2954:3022:3138:3139:3140:3141:3142:3622:3865:3866:3867:3868:3870:3871:3872:3873:3874:3934:3936:3938:3941:3944:3947:3950:3953:3956:3959:4321:4385:4605:5007:6119:6238:6261:6653:6671:7208:7514:7875:7903:7904:8603:9025:9121:9413:9592:10004:11026:11232:11233:11473:11657:11658:11914:12043:12294:12296:12297:12438:12517:12519:12555:12679:12683:12698:12737:12895:12986:13007:13156:13228:13255:13894:14394:14687:21080:21324:21433:21444:21451:21524:21525:21611:21627:21666:21740:30012:30029 :30030:3 X-HE-Tag: snail26_37cd5accb0e3a X-Filterd-Recvd-Size: 24918 Received: from mail-pf1-f196.google.com (mail-pf1-f196.google.com [209.85.210.196]) by imf11.hostedemail.com (Postfix) with ESMTP for ; Sun, 15 Sep 2019 16:53:03 +0000 (UTC) Received: by mail-pf1-f196.google.com with SMTP id q10so21122800pfl.0 for ; Sun, 15 Sep 2019 09:53:03 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=20161025; h=from:to:cc:subject:date:message-id:in-reply-to:references :mime-version:content-transfer-encoding; bh=rGRlPiR3gI4yFwzk+whmCLzMmdtttoXjltpehi5xOx0=; b=WtgFNtXIqXL0KHnJR3v61GOTDBdgHcYOW1rKGuKMqm95XQoNGEUmfZ1K7d4jzGoF0D VLt9LfZlXPrS0tGw1J25ASnbCdBMQ5XIpenTn/BEF/MDipGxGc4ISwXyMLzuxv/lJsPv oO85y9wboQv+XaWSElQShN9N9ENu159c2xMHu+kVCFMqKDgwBxoca2J0INXS4uszCo8P UvWh5DR2pZ9R1O+indnxPIGGRVk4jzlLprn7CbpPem9VzxGsEj34oXoe5wvSD4dMWnrV rubEGJCJEPxpR+gFMud5PDnh5hpkOw3FnQFAOtez92qFgrwgi7IBt+Y0Ly/xqqW/gCtD 33wA== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20161025; h=x-gm-message-state:from:to:cc:subject:date:message-id:in-reply-to :references:mime-version:content-transfer-encoding; bh=rGRlPiR3gI4yFwzk+whmCLzMmdtttoXjltpehi5xOx0=; b=krn8agNcex1Gu8g0aVAMg9h214ob160kJTpDovuVYFxs3ZxPuIaGahXo3u1wQxmGki qMC80yeBcnCmxERpQiB8AzeUysIFF9zk3JR8Q21a7N+ORLeBqtF5DD1uQj3MvfrEmax3 hBBpJw4jTJa0TXf0ksnQCGfq8rkQmGa0jsZRUb5YLrX46yZ3xAiAYVjUloDd4VlVjjRj MeGBqtjviDClu8muJlYWZ27CQexJh3vww5A0OyDe8fzwbJ2v9t10ygle+QX84hal8DxL MB3s/oVN9ubavD95WXQSQbsW9N7MZI+1Zt2XLa0wmfcgpozCIaoeCjARYyft5gNhNF7A wciw== X-Gm-Message-State: APjAAAXNKtheK9VKU+ZK59mIk5249/XSjq0+MLZC2rjnJprtlBrYqc+6 LGtT0NvzKnUXhTh61+zt7iQ= X-Google-Smtp-Source: APXvYqwgvN+IA+UBfPTIgA/7CcnaBOfIsN8v7rtd0mF0ynWMlepsCdDJhkF1zH75mNSP3wJHk6vMVQ== X-Received: by 2002:a63:29c7:: with SMTP id p190mr16632491pgp.124.1568566382184; Sun, 15 Sep 2019 09:53:02 -0700 (PDT) Received: from localhost.localdomain.localdomain ([2408:823c:c11:160:b8c3:8577:bf2f:3]) by smtp.gmail.com with ESMTPSA id a4sm4383259pgq.6.2019.09.15.09.52.49 (version=TLS1_3 cipher=TLS_AES_256_GCM_SHA384 bits=256/256); Sun, 15 Sep 2019 09:53:01 -0700 (PDT) From: Pengfei Li To: akpm@linux-foundation.org Cc: vbabka@suse.cz, cl@linux.com, penberg@kernel.org, rientjes@google.com, iamjoonsoo.kim@lge.com, linux-mm@kvack.org, linux-kernel@vger.kernel.org, guro@fb.com, Pengfei Li Subject: [PATCH v4 5/7] mm, slab_common: make kmalloc_caches[] start at size KMALLOC_MIN_SIZE Date: Mon, 16 Sep 2019 00:51:16 +0800 Message-Id: <20190915165121.7237-8-lpf.vector@gmail.com> X-Mailer: git-send-email 2.21.0 In-Reply-To: <20190915165121.7237-1-lpf.vector@gmail.com> References: <20190915165121.7237-1-lpf.vector@gmail.com> MIME-Version: 1.0 X-Bogosity: Ham, tests=bogofilter, spamicity=0.000000, version=1.2.4 Sender: owner-linux-mm@kvack.org Precedence: bulk X-Loop: owner-majordomo@kvack.org List-ID: Currently, kmalloc_cache[] is not sorted by size, kmalloc_cache[0] is kmalloc-96, kmalloc_cache[1] is kmalloc-192 (when ARCH_DMA_MINALIGN is not defined). As suggested by Vlastimil Babka, "Since you're doing these cleanups, have you considered reordering kmalloc_info, size_index, kmalloc_index() etc so that sizes 96 and 192 are ordered naturally between 64, 128 and 256? That should remove various special casing such as in create_kmalloc_caches(). I can't guarantee it will be possible without breaking e.g. constant folding optimizations etc., but seems to me it should be feasible. (There are definitely more places to change than those I listed.)" So this patch reordered kmalloc_info[], kmalloc_caches[], and modified kmalloc_index() and kmalloc_slab() accordingly. As a result, there is no subtle judgment about size in create_kmalloc_caches(). And initialize kmalloc_cache[] from 0 instead of KMALLOC_SHIFT_LOW. I used ./scripts/bloat-o-meter to measure the impact of this patch on performance. The results show that it brings some benefits. Considering the size change of kmalloc_info[], the size of the code is actually about 641 bytes less. (Note: The original kmalloc_info[] was renamed to all_kmalloc_info[]) $ ./scripts/bloat-o-meter vmlinux.old vmlinux.patch_1-5 add/remove: 1/2 grow/shrink: 6/64 up/down: 872/-1113 (-241) Function old new delta all_kmalloc_info - 832 +832 crypto_create_tfm 211 225 +14 ieee80211_key_alloc 1159 1169 +10 nl80211_parse_sched_scan 2787 2795 +8 ida_alloc_range 951 955 +4 find_get_context.isra 634 637 +3 sd_probe 947 948 +1 nla_strdup 143 142 -1 trace_parser_get_init 71 69 -2 pkcs7_verify.cold 318 316 -2 xhci_alloc_tt_info 349 346 -3 units 323 320 -3 nl80211_set_reg 642 639 -3 i915_sw_fence_await_dma_fence 445 441 -4 nf_queue 671 666 -5 kmalloc_slab 102 97 -5 xhci_segment_alloc 303 297 -6 xhci_alloc_container_ctx 221 215 -6 xfrm_policy_alloc 277 271 -6 selinux_sk_alloc_security 119 113 -6 sdev_evt_send_simple 124 118 -6 sdev_evt_alloc 85 79 -6 sbitmap_queue_init_node 424 418 -6 regulatory_hint_found_beacon 400 394 -6 nf_ct_tmpl_alloc 91 85 -6 gss_create_cred 146 140 -6 drm_flip_work_allocate_task 76 70 -6 cfg80211_stop_iface 266 260 -6 cfg80211_sinfo_alloc_tid_stats 83 77 -6 cfg80211_port_authorized 218 212 -6 cfg80211_ibss_joined 341 335 -6 call_usermodehelper_setup 155 149 -6 bpf_prog_alloc_no_stats 188 182 -6 blk_alloc_flush_queue 197 191 -6 bdi_alloc_node 201 195 -6 _netlbl_catmap_getnode 253 247 -6 ____ip_mc_inc_group 481 475 -6 pkcs7_verify 1503 1495 -8 audit_log_d_path 204 196 -8 xprt_switch_alloc 145 136 -9 xhci_ring_alloc 378 369 -9 xhci_mem_init 3673 3664 -9 xhci_alloc_virt_device 505 496 -9 xhci_alloc_stream_info 727 718 -9 xhci_alloc_command 221 212 -9 tcp_sendmsg_locked 3129 3120 -9 tcp_md5_do_add 783 774 -9 tcp_fastopen_defer_connect 279 270 -9 sr_read_tochdr.isra 260 251 -9 sr_read_tocentry.isra 337 328 -9 sr_is_xa 385 376 -9 sr_get_mcn 269 260 -9 scsi_probe_and_add_lun 2947 2938 -9 ring_buffer_read_prepare 103 94 -9 request_firmware_nowait 405 396 -9 ohci_urb_enqueue 3185 3176 -9 nfs_alloc_seqid 96 87 -9 nfs4_get_state_owner 1049 1040 -9 nfs4_do_close 587 578 -9 mempool_create_node 173 164 -9 ip6_setup_cork 1030 1021 -9 dma_pool_alloc 419 410 -9 devres_open_group 223 214 -9 cfg80211_parse_mbssid_data 2406 2397 -9 __igmp_group_dropped 629 619 -10 gss_import_sec_context 187 176 -11 ip_setup_cork 374 362 -12 __i915_sw_fence_await_sw_fence 429 417 -12 kmalloc_caches 336 312 -24 create_kmalloc_caches 270 214 -56 kmalloc_cache_name 57 - -57 new_kmalloc_cache 112 - -112 kmalloc_info 432 8 -424 Total: Before=14874616, After=14874375, chg -0.00% Signed-off-by: Pengfei Li --- include/linux/slab.h | 96 ++++++++++++++++---------- mm/slab.h | 10 +-- mm/slab_common.c | 158 ++++++++++++++++--------------------------- mm/slub.c | 12 ++-- 4 files changed, 133 insertions(+), 143 deletions(-) diff --git a/include/linux/slab.h b/include/linux/slab.h index 1f05f68f2c3e..f53bb6980110 100644 --- a/include/linux/slab.h +++ b/include/linux/slab.h @@ -297,6 +297,23 @@ static inline void __check_heap_object(const void *ptr, unsigned long n, #define KMALLOC_MIN_SIZE (1 << KMALLOC_SHIFT_LOW) #endif +#define KMALLOC_CACHE_MIN_NUM (KMALLOC_SHIFT_HIGH - KMALLOC_SHIFT_LOW + 1) + +#if KMALLOC_MIN_SIZE > 64 + #define KMALLOC_SIZE_96_EXIST (0) + #define KMALLOC_SIZE_192_EXIST (0) +#elif KMALLOC_MIN_SIZE > 32 + #define KMALLOC_SIZE_96_EXIST (0) + #define KMALLOC_SIZE_192_EXIST (1) +#else + #define KMALLOC_SIZE_96_EXIST (1) + #define KMALLOC_SIZE_192_EXIST (1) +#endif + +#define KMALLOC_CACHE_NUM (KMALLOC_CACHE_MIN_NUM \ + + KMALLOC_SIZE_96_EXIST \ + + KMALLOC_SIZE_192_EXIST) + /* * This restriction comes from byte sized index implementation. * Page size is normally 2^12 bytes and, in this case, if we want to use @@ -323,7 +340,7 @@ enum kmalloc_cache_type { #ifndef CONFIG_SLOB extern struct kmem_cache * -kmalloc_caches[NR_KMALLOC_TYPES][KMALLOC_SHIFT_HIGH + 1]; +kmalloc_caches[NR_KMALLOC_TYPES][KMALLOC_CACHE_NUM]; static __always_inline enum kmalloc_cache_type kmalloc_type(gfp_t flags) { @@ -345,13 +362,18 @@ static __always_inline enum kmalloc_cache_type kmalloc_type(gfp_t flags) #endif } +/* kmalloc_index adjust size: (0, 96] */ +#define KMALLOC_IDX_ADJ_0 (KMALLOC_SHIFT_LOW) + +/* kmalloc_index adjust size: (96, 192] */ +#define KMALLOC_IDX_ADJ_1 (KMALLOC_IDX_ADJ_0 - KMALLOC_SIZE_96_EXIST) + +/* kmalloc_index adjust size: (192, N] */ +#define KMALLOC_IDX_ADJ_2 (KMALLOC_IDX_ADJ_1 - KMALLOC_SIZE_192_EXIST) + /* * Figure out which kmalloc slab an allocation of a certain size * belongs to. - * 0 = zero alloc - * 1 = 65 .. 96 bytes - * 2 = 129 .. 192 bytes - * n = 2^(n-1)+1 .. 2^n */ static __always_inline unsigned int kmalloc_index(size_t size) { @@ -359,36 +381,40 @@ static __always_inline unsigned int kmalloc_index(size_t size) return ZERO_SIZE_ALLOC; if (size <= KMALLOC_MIN_SIZE) - return KMALLOC_SHIFT_LOW; - - if (KMALLOC_MIN_SIZE <= 32 && size > 64 && size <= 96) - return 1; - if (KMALLOC_MIN_SIZE <= 64 && size > 128 && size <= 192) - return 2; - if (size <= 8) return 3; - if (size <= 16) return 4; - if (size <= 32) return 5; - if (size <= 64) return 6; - if (size <= 128) return 7; - if (size <= 256) return 8; - if (size <= 512) return 9; - if (size <= 1024) return 10; - if (size <= 2 * 1024) return 11; - if (size <= 4 * 1024) return 12; - if (size <= 8 * 1024) return 13; - if (size <= 16 * 1024) return 14; - if (size <= 32 * 1024) return 15; - if (size <= 64 * 1024) return 16; - if (size <= 128 * 1024) return 17; - if (size <= 256 * 1024) return 18; - if (size <= 512 * 1024) return 19; - if (size <= 1024 * 1024) return 20; - if (size <= 2 * 1024 * 1024) return 21; - if (size <= 4 * 1024 * 1024) return 22; - if (size <= 8 * 1024 * 1024) return 23; - if (size <= 16 * 1024 * 1024) return 24; - if (size <= 32 * 1024 * 1024) return 25; - if (size <= 64 * 1024 * 1024) return 26; + return 0; + +#if KMALLOC_SIZE_96_EXIST == 1 + if (size > 64 && size <= 96) return (7 - KMALLOC_IDX_ADJ_0); +#endif + +#if KMALLOC_SIZE_192_EXIST == 1 + if (size > 128 && size <= 192) return (8 - KMALLOC_IDX_ADJ_1); +#endif + + if (size <= 8) return ( 3 - KMALLOC_IDX_ADJ_0); + if (size <= 16) return ( 4 - KMALLOC_IDX_ADJ_0); + if (size <= 32) return ( 5 - KMALLOC_IDX_ADJ_0); + if (size <= 64) return ( 6 - KMALLOC_IDX_ADJ_0); + if (size <= 128) return ( 7 - KMALLOC_IDX_ADJ_1); + if (size <= 256) return ( 8 - KMALLOC_IDX_ADJ_2); + if (size <= 512) return ( 9 - KMALLOC_IDX_ADJ_2); + if (size <= 1024) return (10 - KMALLOC_IDX_ADJ_2); + if (size <= 2 * 1024) return (11 - KMALLOC_IDX_ADJ_2); + if (size <= 4 * 1024) return (12 - KMALLOC_IDX_ADJ_2); + if (size <= 8 * 1024) return (13 - KMALLOC_IDX_ADJ_2); + if (size <= 16 * 1024) return (14 - KMALLOC_IDX_ADJ_2); + if (size <= 32 * 1024) return (15 - KMALLOC_IDX_ADJ_2); + if (size <= 64 * 1024) return (16 - KMALLOC_IDX_ADJ_2); + if (size <= 128 * 1024) return (17 - KMALLOC_IDX_ADJ_2); + if (size <= 256 * 1024) return (18 - KMALLOC_IDX_ADJ_2); + if (size <= 512 * 1024) return (19 - KMALLOC_IDX_ADJ_2); + if (size <= 1024 * 1024) return (20 - KMALLOC_IDX_ADJ_2); + if (size <= 2 * 1024 * 1024) return (21 - KMALLOC_IDX_ADJ_2); + if (size <= 4 * 1024 * 1024) return (22 - KMALLOC_IDX_ADJ_2); + if (size <= 8 * 1024 * 1024) return (23 - KMALLOC_IDX_ADJ_2); + if (size <= 16 * 1024 * 1024) return (24 - KMALLOC_IDX_ADJ_2); + if (size <= 32 * 1024 * 1024) return (25 - KMALLOC_IDX_ADJ_2); + if (size <= 64 * 1024 * 1024) return (26 - KMALLOC_IDX_ADJ_2); BUG(); /* Will never be reached. Needed because the compiler may complain */ diff --git a/mm/slab.h b/mm/slab.h index 2fc8f956906a..3ada65ef1118 100644 --- a/mm/slab.h +++ b/mm/slab.h @@ -63,6 +63,11 @@ enum slab_state { FULL /* Everything is working */ }; +struct kmalloc_info_struct { + const char *name[NR_KMALLOC_TYPES]; + unsigned int size; +}; + extern enum slab_state slab_state; /* The slab cache mutex protects the management structures during changes */ @@ -75,10 +80,7 @@ extern struct list_head slab_caches; extern struct kmem_cache *kmem_cache; /* A table of kmalloc cache names and sizes */ -extern const struct kmalloc_info_struct { - const char *name[NR_KMALLOC_TYPES]; - unsigned int size; -} kmalloc_info[]; +extern const struct kmalloc_info_struct * const kmalloc_info; #ifndef CONFIG_SLOB /* Kmalloc array related functions */ diff --git a/mm/slab_common.c b/mm/slab_common.c index af45b5278fdc..2aed30deb071 100644 --- a/mm/slab_common.c +++ b/mm/slab_common.c @@ -1028,7 +1028,7 @@ struct kmem_cache *__init create_kmalloc_cache(const char *name, } struct kmem_cache * -kmalloc_caches[NR_KMALLOC_TYPES][KMALLOC_SHIFT_HIGH + 1] __ro_after_init = +kmalloc_caches[NR_KMALLOC_TYPES][KMALLOC_CACHE_NUM] __ro_after_init = { /* initialization for https://bugs.llvm.org/show_bug.cgi?id=42570 */ }; EXPORT_SYMBOL(kmalloc_caches); @@ -1039,30 +1039,30 @@ EXPORT_SYMBOL(kmalloc_caches); * fls. */ static u8 size_index[24] __ro_after_init = { - 3, /* 8 */ - 4, /* 16 */ - 5, /* 24 */ - 5, /* 32 */ - 6, /* 40 */ - 6, /* 48 */ - 6, /* 56 */ - 6, /* 64 */ - 1, /* 72 */ - 1, /* 80 */ - 1, /* 88 */ - 1, /* 96 */ - 7, /* 104 */ - 7, /* 112 */ - 7, /* 120 */ - 7, /* 128 */ - 2, /* 136 */ - 2, /* 144 */ - 2, /* 152 */ - 2, /* 160 */ - 2, /* 168 */ - 2, /* 176 */ - 2, /* 184 */ - 2 /* 192 */ + (3 - KMALLOC_IDX_ADJ_0), /* 8 */ + (4 - KMALLOC_IDX_ADJ_0), /* 16 */ + (5 - KMALLOC_IDX_ADJ_0), /* 24 */ + (5 - KMALLOC_IDX_ADJ_0), /* 32 */ + (6 - KMALLOC_IDX_ADJ_0), /* 40 */ + (6 - KMALLOC_IDX_ADJ_0), /* 48 */ + (6 - KMALLOC_IDX_ADJ_0), /* 56 */ + (6 - KMALLOC_IDX_ADJ_0), /* 64 */ + (7 - KMALLOC_IDX_ADJ_0), /* 72 */ + (7 - KMALLOC_IDX_ADJ_0), /* 80 */ + (7 - KMALLOC_IDX_ADJ_0), /* 88 */ + (7 - KMALLOC_IDX_ADJ_0), /* 96 */ + (7 - KMALLOC_IDX_ADJ_1), /* 104 */ + (7 - KMALLOC_IDX_ADJ_1), /* 112 */ + (7 - KMALLOC_IDX_ADJ_1), /* 120 */ + (7 - KMALLOC_IDX_ADJ_1), /* 128 */ + (8 - KMALLOC_IDX_ADJ_1), /* 136 */ + (8 - KMALLOC_IDX_ADJ_1), /* 144 */ + (8 - KMALLOC_IDX_ADJ_1), /* 152 */ + (8 - KMALLOC_IDX_ADJ_1), /* 160 */ + (8 - KMALLOC_IDX_ADJ_1), /* 168 */ + (8 - KMALLOC_IDX_ADJ_1), /* 176 */ + (8 - KMALLOC_IDX_ADJ_1), /* 184 */ + (8 - KMALLOC_IDX_ADJ_1), /* 192 */ }; static inline unsigned int size_index_elem(unsigned int bytes) @@ -1086,13 +1086,17 @@ struct kmem_cache *kmalloc_slab(size_t size, gfp_t flags) } else { if (WARN_ON_ONCE(size > KMALLOC_MAX_CACHE_SIZE)) return NULL; - index = fls(size - 1); + + index = fls(size - 1) - KMALLOC_IDX_ADJ_2; } return kmalloc_caches[kmalloc_type(flags)][index]; } #ifdef CONFIG_ZONE_DMA + +#define KMALLOC_INFO_SHIFT_LOW (3) +#define KMALLOC_INFO_START_IDX (KMALLOC_SHIFT_LOW - KMALLOC_INFO_SHIFT_LOW) #define SET_KMALLOC_SIZE(__size, __short_size) \ { \ .name[KMALLOC_NORMAL] = "kmalloc-" #__short_size, \ @@ -1110,40 +1114,35 @@ struct kmem_cache *kmalloc_slab(size_t size, gfp_t flags) #endif /* - * kmalloc_info[] is to make slub_debug=,kmalloc-xx option work at boot time. - * kmalloc_index() supports up to 2^26=64MB, so the final entry of the table is - * kmalloc-67108864. + * all_kmalloc_info[] is to make slub_debug=, kmalloc-xx option work at boot + * time. kmalloc_index() supports up to 2^26=64MB, so the final entry of the + * table is kmalloc-67108864. */ -const struct kmalloc_info_struct kmalloc_info[] __initconst = { - SET_KMALLOC_SIZE(0, 0), - SET_KMALLOC_SIZE(96, 96), - SET_KMALLOC_SIZE(192, 192), - SET_KMALLOC_SIZE(8, 8), - SET_KMALLOC_SIZE(16, 16), - SET_KMALLOC_SIZE(32, 32), - SET_KMALLOC_SIZE(64, 64), - SET_KMALLOC_SIZE(128, 128), - SET_KMALLOC_SIZE(256, 256), - SET_KMALLOC_SIZE(512, 512), - SET_KMALLOC_SIZE(1024, 1k), - SET_KMALLOC_SIZE(2048, 2k), - SET_KMALLOC_SIZE(4096, 4k), - SET_KMALLOC_SIZE(8192, 8k), - SET_KMALLOC_SIZE(16384, 16k), - SET_KMALLOC_SIZE(32768, 32k), - SET_KMALLOC_SIZE(65536, 64k), - SET_KMALLOC_SIZE(131072, 128k), - SET_KMALLOC_SIZE(262144, 256k), - SET_KMALLOC_SIZE(524288, 512k), - SET_KMALLOC_SIZE(1048576, 1M), - SET_KMALLOC_SIZE(2097152, 2M), - SET_KMALLOC_SIZE(4194304, 4M), - SET_KMALLOC_SIZE(8388608, 8M), - SET_KMALLOC_SIZE(16777216, 16M), - SET_KMALLOC_SIZE(33554432, 32M), - SET_KMALLOC_SIZE(67108864, 64M) +const struct kmalloc_info_struct all_kmalloc_info[] __initconst = { + SET_KMALLOC_SIZE( 8, 8), SET_KMALLOC_SIZE( 16, 16), + SET_KMALLOC_SIZE( 32, 32), SET_KMALLOC_SIZE( 64, 64), +#if KMALLOC_SIZE_96_EXIST == 1 + SET_KMALLOC_SIZE( 96, 96), +#endif + SET_KMALLOC_SIZE( 128, 128), +#if KMALLOC_SIZE_192_EXIST == 1 + SET_KMALLOC_SIZE( 192, 192), +#endif + SET_KMALLOC_SIZE( 256, 256), SET_KMALLOC_SIZE( 512, 512), + SET_KMALLOC_SIZE( 1024, 1k), SET_KMALLOC_SIZE( 2048, 2k), + SET_KMALLOC_SIZE( 4096, 4k), SET_KMALLOC_SIZE( 8192, 8k), + SET_KMALLOC_SIZE( 16384, 16k), SET_KMALLOC_SIZE( 32768, 32k), + SET_KMALLOC_SIZE( 65536, 64k), SET_KMALLOC_SIZE( 131072, 128k), + SET_KMALLOC_SIZE( 262144, 256k), SET_KMALLOC_SIZE( 524288, 512k), + SET_KMALLOC_SIZE( 1048576, 1M), SET_KMALLOC_SIZE( 2097152, 2M), + SET_KMALLOC_SIZE( 4194304, 4M), SET_KMALLOC_SIZE( 8388608, 8M), + SET_KMALLOC_SIZE(16777216, 16M), SET_KMALLOC_SIZE(33554432, 32M), + SET_KMALLOC_SIZE(67108864, 64M) }; +const struct kmalloc_info_struct * const __initconst +kmalloc_info = &all_kmalloc_info[KMALLOC_INFO_START_IDX]; + /* * Patch up the size_index table if we have strange large alignment * requirements for the kmalloc array. This is only the case for @@ -1162,33 +1161,8 @@ void __init setup_kmalloc_cache_index_table(void) BUILD_BUG_ON(KMALLOC_MIN_SIZE > 256 || (KMALLOC_MIN_SIZE & (KMALLOC_MIN_SIZE - 1))); - for (i = 8; i < KMALLOC_MIN_SIZE; i += 8) { - unsigned int elem = size_index_elem(i); - - if (elem >= ARRAY_SIZE(size_index)) - break; - size_index[elem] = KMALLOC_SHIFT_LOW; - } - - if (KMALLOC_MIN_SIZE >= 64) { - /* - * The 96 byte size cache is not used if the alignment - * is 64 byte. - */ - for (i = 64 + 8; i <= 96; i += 8) - size_index[size_index_elem(i)] = 7; - - } - - if (KMALLOC_MIN_SIZE >= 128) { - /* - * The 192 byte sized cache is not used if the alignment - * is 128 byte. Redirect kmalloc to use the 256 byte cache - * instead. - */ - for (i = 128 + 8; i <= 192; i += 8) - size_index[size_index_elem(i)] = 8; - } + for (i = 8; i < KMALLOC_MIN_SIZE && i <= 192; i += 8) + size_index[size_index_elem(i)] = 0; } static void __init @@ -1214,21 +1188,9 @@ void __init create_kmalloc_caches(slab_flags_t flags) enum kmalloc_cache_type type; for (type = KMALLOC_NORMAL; type <= KMALLOC_RECLAIM; type++) { - for (i = KMALLOC_SHIFT_LOW; i <= KMALLOC_SHIFT_HIGH; i++) { + for (i = 0; i < KMALLOC_CACHE_NUM; i++) { if (!kmalloc_caches[type][i]) new_kmalloc_cache(i, type, flags); - - /* - * Caches that are not of the two-to-the-power-of size. - * These have to be created immediately after the - * earlier power of two caches - */ - if (KMALLOC_MIN_SIZE <= 32 && i == 6 && - !kmalloc_caches[type][1]) - new_kmalloc_cache(1, type, flags); - if (KMALLOC_MIN_SIZE <= 64 && i == 7 && - !kmalloc_caches[type][2]) - new_kmalloc_cache(2, type, flags); } } @@ -1236,7 +1198,7 @@ void __init create_kmalloc_caches(slab_flags_t flags) slab_state = UP; #ifdef CONFIG_ZONE_DMA - for (i = 0; i <= KMALLOC_SHIFT_HIGH; i++) { + for (i = 0; i < KMALLOC_CACHE_NUM; i++) { struct kmem_cache *s = kmalloc_caches[KMALLOC_NORMAL][i]; if (s) { diff --git a/mm/slub.c b/mm/slub.c index 8834563cdb4b..0e92ebdcacc9 100644 --- a/mm/slub.c +++ b/mm/slub.c @@ -4711,7 +4711,7 @@ static void __init resiliency_test(void) pr_err("\n1. kmalloc-16: Clobber Redzone/next pointer 0x12->0x%p\n\n", p + 16); - validate_slab_cache(kmalloc_caches[type][4]); + validate_slab_cache(kmalloc_caches[type][1]); /* Hmmm... The next two are dangerous */ p = kzalloc(32, GFP_KERNEL); @@ -4720,33 +4720,33 @@ static void __init resiliency_test(void) p); pr_err("If allocated object is overwritten then not detectable\n\n"); - validate_slab_cache(kmalloc_caches[type][5]); + validate_slab_cache(kmalloc_caches[type][2]); p = kzalloc(64, GFP_KERNEL); p += 64 + (get_cycles() & 0xff) * sizeof(void *); *p = 0x56; pr_err("\n3. kmalloc-64: corrupting random byte 0x56->0x%p\n", p); pr_err("If allocated object is overwritten then not detectable\n\n"); - validate_slab_cache(kmalloc_caches[type][6]); + validate_slab_cache(kmalloc_caches[type][3]); pr_err("\nB. Corruption after free\n"); p = kzalloc(128, GFP_KERNEL); kfree(p); *p = 0x78; pr_err("1. kmalloc-128: Clobber first word 0x78->0x%p\n\n", p); - validate_slab_cache(kmalloc_caches[type][7]); + validate_slab_cache(kmalloc_caches[type][5]); p = kzalloc(256, GFP_KERNEL); kfree(p); p[50] = 0x9a; pr_err("\n2. kmalloc-256: Clobber 50th byte 0x9a->0x%p\n\n", p); - validate_slab_cache(kmalloc_caches[type][8]); + validate_slab_cache(kmalloc_caches[type][7]); p = kzalloc(512, GFP_KERNEL); kfree(p); p[512] = 0xab; pr_err("\n3. kmalloc-512: Clobber redzone 0xab->0x%p\n\n", p); - validate_slab_cache(kmalloc_caches[type][9]); + validate_slab_cache(kmalloc_caches[type][8]); } #else #ifdef CONFIG_SYSFS From patchwork Sun Sep 15 16:51:18 2019 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Pengfei Li X-Patchwork-Id: 11146019 Return-Path: Received: from mail.kernel.org (pdx-korg-mail-1.web.codeaurora.org [172.30.200.123]) by pdx-korg-patchwork-2.web.codeaurora.org (Postfix) with ESMTP id 26F2D14DB for ; Sun, 15 Sep 2019 16:53:25 +0000 (UTC) Received: from kanga.kvack.org (kanga.kvack.org [205.233.56.17]) by mail.kernel.org (Postfix) with ESMTP id DA8DB214D9 for ; Sun, 15 Sep 2019 16:53:24 +0000 (UTC) Authentication-Results: mail.kernel.org; dkim=fail reason="signature verification failed" (2048-bit key) header.d=gmail.com header.i=@gmail.com header.b="jVgOxzsa" DMARC-Filter: OpenDMARC Filter v1.3.2 mail.kernel.org DA8DB214D9 Authentication-Results: mail.kernel.org; dmarc=fail (p=none dis=none) header.from=gmail.com Authentication-Results: mail.kernel.org; spf=pass smtp.mailfrom=owner-linux-mm@kvack.org Received: by kanga.kvack.org (Postfix) id 23F506B0003; Sun, 15 Sep 2019 12:53:24 -0400 (EDT) Delivered-To: linux-mm-outgoing@kvack.org Received: by kanga.kvack.org (Postfix, from userid 40) id 1EFB56B000A; Sun, 15 Sep 2019 12:53:24 -0400 (EDT) X-Original-To: int-list-linux-mm@kvack.org X-Delivered-To: int-list-linux-mm@kvack.org Received: by kanga.kvack.org (Postfix, from userid 63042) id 1057A6B000C; Sun, 15 Sep 2019 12:53:24 -0400 (EDT) X-Original-To: linux-mm@kvack.org X-Delivered-To: linux-mm@kvack.org Received: from forelay.hostedemail.com (smtprelay0106.hostedemail.com [216.40.44.106]) by kanga.kvack.org (Postfix) with ESMTP id DEB126B0003 for ; Sun, 15 Sep 2019 12:53:23 -0400 (EDT) Received: from smtpin19.hostedemail.com (10.5.19.251.rfc1918.com [10.5.19.251]) by forelay04.hostedemail.com (Postfix) with SMTP id 798102C06 for ; Sun, 15 Sep 2019 16:53:23 +0000 (UTC) X-FDA: 75937750686.19.pet28_3aa764645b314 X-Spam-Summary: 50,3,0,b130e0f7db268a18,d41d8cd98f00b204,lpf.vector@gmail.com,:akpm@linux-foundation.org:vbabka@suse.cz:cl@linux.com:penberg@kernel.org:rientjes@google.com:iamjoonsoo.kim@lge.com::linux-kernel@vger.kernel.org:guro@fb.com:lpf.vector@gmail.com,RULES_HIT:41:69:355:379:541:800:960:968:973:982:988:989:1260:1311:1314:1345:1359:1437:1515:1535:1543:1711:1730:1747:1777:1792:1801:2198:2199:2393:2504:2559:2562:2693:2731:3138:3139:3140:3141:3142:3354:3865:3866:3867:3868:3870:3871:3872:3874:4117:4321:4605:5007:6261:6653:7514:7903:8603:9121:9413:9592:10004:11026:11232:11473:11658:11914:12043:12296:12297:12438:12517:12519:12555:12683:12895:13156:13228:13894:14040:14093:14181:14394:14687:14721:21080:21444:21450:21451:21627:21666:30029:30054,0,RBL:209.85.215.193:@gmail.com:.lbl8.mailshell.net-62.50.175.100 66.100.201.100,CacheIP:none,Bayesian:0.5,0.5,0.5,Netcheck:none,DomainCache:0,MSF:not bulk,SPF:fp,MSBL:0,DNSBL:neutral,Custom_rules:0:0:0,LFtime:24,LUA_SUMMARY:none X-HE-Tag: pet28_3aa764645b314 X-Filterd-Recvd-Size: 6318 Received: from mail-pg1-f193.google.com (mail-pg1-f193.google.com [209.85.215.193]) by imf38.hostedemail.com (Postfix) with ESMTP for ; Sun, 15 Sep 2019 16:53:22 +0000 (UTC) Received: by mail-pg1-f193.google.com with SMTP id u72so18013830pgb.10 for ; Sun, 15 Sep 2019 09:53:22 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=20161025; h=from:to:cc:subject:date:message-id:in-reply-to:references :mime-version:content-transfer-encoding; bh=3ghBrDy6wMKk43GDL/rFQB3qO4WlHynWL3G2LPomcOY=; b=jVgOxzsaBL7d6+d3FL3j79i1Fvbge77Zkqt6VNcmsJHg+TdLOdCa9HZ7rqMVW+EkvX M7/xxz9g3zkcLp59hc8nHrNfKiXw2o7u5iWTX1afNpqNDKJYUgE1MHiyaiZ/xV7XCwgy SRPhmSR5oe9NAe4ppvY6n/YpvWcv+j01Zop8QJilVoYpSccyHjw0sVGVFTv6HYF7rpwv LdB7gTH75HT9MzWEd9Vsj+/jtLt62KVzWvqoOGXm6tUnyQKfXuV6gmeFBt0Y6C3lV+vV psoqcjB9SSXA4H7rY2x5sTdSp/UqS61F/PdfPB/t3wNoqrb6gNs0hmuW1m3Mohn/ZtKH Nzyg== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20161025; h=x-gm-message-state:from:to:cc:subject:date:message-id:in-reply-to :references:mime-version:content-transfer-encoding; bh=3ghBrDy6wMKk43GDL/rFQB3qO4WlHynWL3G2LPomcOY=; b=Hjzn64hvD6XsXxmfxfThP6vwCuzyWVCDd8w/rgBZqmk0gCICajEvM+p/KKcku/iC4h 54/oyzO8lqqxklLIZqrf1nOWgMORiMR72DBATdrZFC9An7p+gaK+6y3pKSoKub1UPegM AthgdzmRPzJ6uUArF9I3MOJSUc0zi7lFq08KRQ2AuQdYsVRdY+MdJdDOyc4Vv3OG97Md hUURyK1Q7fTfiDOv0O+XWI/FLRNwXbVqH+JgZlqUrgWPVW+zuY9d2nyo9PDlwW2aHcgn h3KO2mMqkhrbi/pyaV2g18rKQIGFldql5lO+owAbQ3bFVSiQWMl0yNGIeqeP7C8H7Tvj Pb9w== X-Gm-Message-State: APjAAAWy2Ch7by8FAsvPtlrsOcvxZ82Vztv+9RMLCiDVLSwTsxa+kGTn ojr7+f0Ai58o4t829yq2SZM= X-Google-Smtp-Source: APXvYqyT0EudM329i6xxR9PsWR8BAYr560iqmnyO6W8R7cfNb0ye4ziGG50q8+ogMrkk+tXgw9rOWw== X-Received: by 2002:aa7:8a8a:: with SMTP id a10mr52476468pfc.131.1568566402236; Sun, 15 Sep 2019 09:53:22 -0700 (PDT) Received: from localhost.localdomain.localdomain ([2408:823c:c11:160:b8c3:8577:bf2f:3]) by smtp.gmail.com with ESMTPSA id a4sm4383259pgq.6.2019.09.15.09.53.12 (version=TLS1_3 cipher=TLS_AES_256_GCM_SHA384 bits=256/256); Sun, 15 Sep 2019 09:53:21 -0700 (PDT) From: Pengfei Li To: akpm@linux-foundation.org Cc: vbabka@suse.cz, cl@linux.com, penberg@kernel.org, rientjes@google.com, iamjoonsoo.kim@lge.com, linux-mm@kvack.org, linux-kernel@vger.kernel.org, guro@fb.com, Pengfei Li Subject: [PATCH v4 6/7] mm, slab_common: initialize the same size of kmalloc_caches[] Date: Mon, 16 Sep 2019 00:51:18 +0800 Message-Id: <20190915165121.7237-10-lpf.vector@gmail.com> X-Mailer: git-send-email 2.21.0 In-Reply-To: <20190915165121.7237-1-lpf.vector@gmail.com> References: <20190915165121.7237-1-lpf.vector@gmail.com> MIME-Version: 1.0 X-Bogosity: Ham, tests=bogofilter, spamicity=0.000000, version=1.2.4 Sender: owner-linux-mm@kvack.org Precedence: bulk X-Loop: owner-majordomo@kvack.org List-ID: In the current implementation, KMALLOC_RECLAIM is not initialized until all the KMALLOC_NORMAL sizes have been initialized. But for a particular size, create_kmalloc_caches() can be executed faster by initializing different types of kmalloc in order. $ ./scripts/bloat-o-meter vmlinux.old vmlinux.patch_1-5 add/remove: 1/2 grow/shrink: 6/64 up/down: 872/-1113 (-241) Function old new delta create_kmalloc_caches 270 214 -56 $ ./scripts/bloat-o-meter vmlinux.old vmlinux.patch_1-6 add/remove: 1/2 grow/shrink: 6/64 up/down: 872/-1172 (-300) Function old new delta create_kmalloc_caches 270 155 -115 We can see that it really gets the benefits. Besides, KMALLOC_DMA will be initialized after "slab_state = UP", this does not seem to be necessary. Commit f97d5f634d3b ("slab: Common function to create the kmalloc array") introduces create_kmalloc_caches(). And I found that for SLAB, KMALLOC_DMA is initialized before "slab_state = UP". But for SLUB, KMALLOC_DMA is initialized after "slab_state = UP". Based on this fact, I think it is okay to initialize KMALLOC_DMA before "slab_state = UP". Signed-off-by: Pengfei Li --- mm/slab_common.c | 35 ++++++++++++----------------------- 1 file changed, 12 insertions(+), 23 deletions(-) diff --git a/mm/slab_common.c b/mm/slab_common.c index 2aed30deb071..e7903bd28b1f 100644 --- a/mm/slab_common.c +++ b/mm/slab_common.c @@ -1165,12 +1165,9 @@ void __init setup_kmalloc_cache_index_table(void) size_index[size_index_elem(i)] = 0; } -static void __init +static __always_inline void __init new_kmalloc_cache(int idx, enum kmalloc_cache_type type, slab_flags_t flags) { - if (type == KMALLOC_RECLAIM) - flags |= SLAB_RECLAIM_ACCOUNT; - kmalloc_caches[type][idx] = create_kmalloc_cache( kmalloc_info[idx].name[type], kmalloc_info[idx].size, flags, 0, @@ -1185,30 +1182,22 @@ new_kmalloc_cache(int idx, enum kmalloc_cache_type type, slab_flags_t flags) void __init create_kmalloc_caches(slab_flags_t flags) { int i; - enum kmalloc_cache_type type; - for (type = KMALLOC_NORMAL; type <= KMALLOC_RECLAIM; type++) { - for (i = 0; i < KMALLOC_CACHE_NUM; i++) { - if (!kmalloc_caches[type][i]) - new_kmalloc_cache(i, type, flags); - } - } + for (i = 0; i < KMALLOC_CACHE_NUM; i++) { + if (!kmalloc_caches[KMALLOC_NORMAL][i]) + new_kmalloc_cache(i, KMALLOC_NORMAL, flags); - /* Kmalloc array is now usable */ - slab_state = UP; + new_kmalloc_cache(i, KMALLOC_RECLAIM, + flags | SLAB_RECLAIM_ACCOUNT); #ifdef CONFIG_ZONE_DMA - for (i = 0; i < KMALLOC_CACHE_NUM; i++) { - struct kmem_cache *s = kmalloc_caches[KMALLOC_NORMAL][i]; - - if (s) { - kmalloc_caches[KMALLOC_DMA][i] = create_kmalloc_cache( - kmalloc_info[i].name[KMALLOC_DMA], - kmalloc_info[i].size, - SLAB_CACHE_DMA | flags, 0, 0); - } - } + new_kmalloc_cache(i, KMALLOC_DMA, + flags | SLAB_CACHE_DMA); #endif + } + + /* Kmalloc array is now usable */ + slab_state = UP; } #endif /* !CONFIG_SLOB */ From patchwork Sun Sep 15 16:51:20 2019 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Pengfei Li X-Patchwork-Id: 11146023 Return-Path: Received: from mail.kernel.org (pdx-korg-mail-1.web.codeaurora.org [172.30.200.123]) by pdx-korg-patchwork-2.web.codeaurora.org (Postfix) with ESMTP id B797214DB for ; Sun, 15 Sep 2019 16:53:43 +0000 (UTC) Received: from kanga.kvack.org (kanga.kvack.org [205.233.56.17]) by mail.kernel.org (Postfix) with ESMTP id 69EC12168B for ; Sun, 15 Sep 2019 16:53:43 +0000 (UTC) Authentication-Results: mail.kernel.org; dkim=fail reason="signature verification failed" (2048-bit key) header.d=gmail.com header.i=@gmail.com header.b="HUwCU5p/" DMARC-Filter: OpenDMARC Filter v1.3.2 mail.kernel.org 69EC12168B Authentication-Results: mail.kernel.org; dmarc=fail (p=none dis=none) header.from=gmail.com Authentication-Results: mail.kernel.org; spf=pass smtp.mailfrom=owner-linux-mm@kvack.org Received: by kanga.kvack.org (Postfix) id 9B9876B000C; Sun, 15 Sep 2019 12:53:42 -0400 (EDT) Delivered-To: linux-mm-outgoing@kvack.org Received: by kanga.kvack.org (Postfix, from userid 40) id 969C86B000D; Sun, 15 Sep 2019 12:53:42 -0400 (EDT) X-Original-To: int-list-linux-mm@kvack.org X-Delivered-To: int-list-linux-mm@kvack.org Received: by kanga.kvack.org (Postfix, from userid 63042) id 8809D6B000E; Sun, 15 Sep 2019 12:53:42 -0400 (EDT) X-Original-To: linux-mm@kvack.org X-Delivered-To: linux-mm@kvack.org Received: from forelay.hostedemail.com (smtprelay0254.hostedemail.com [216.40.44.254]) by kanga.kvack.org (Postfix) with ESMTP id 66EDB6B000C for ; Sun, 15 Sep 2019 12:53:42 -0400 (EDT) Received: from smtpin23.hostedemail.com (10.5.19.251.rfc1918.com [10.5.19.251]) by forelay01.hostedemail.com (Postfix) with SMTP id 10509180AD7C3 for ; Sun, 15 Sep 2019 16:53:42 +0000 (UTC) X-FDA: 75937751484.23.flag97_3d5715abf0232 X-Spam-Summary: 2,0,0,7d91c419b40eb208,d41d8cd98f00b204,lpf.vector@gmail.com,:akpm@linux-foundation.org:vbabka@suse.cz:cl@linux.com:penberg@kernel.org:rientjes@google.com:iamjoonsoo.kim@lge.com::linux-kernel@vger.kernel.org:guro@fb.com:lpf.vector@gmail.com,RULES_HIT:1:2:41:355:379:541:559:800:960:966:967:968:973:982:988:989:1260:1311:1314:1345:1359:1437:1515:1605:1730:1747:1777:1792:1801:2194:2196:2199:2200:2393:2525:2559:2563:2682:2685:2693:2859:2918:2933:2937:2939:2942:2945:2947:2951:2954:3022:3138:3139:3140:3141:3142:3865:3866:3867:3868:3871:3874:3934:3936:3938:3941:3944:3947:3950:3953:3956:3959:4052:4321:4385:4605:5007:6119:6261:6653:6671:7208:7514:7875:7903:7904:8603:9025:9121:9413:10004:11026:11473:11658:11914:12043:12294:12296:12297:12438:12517:12519:12555:12679:12698:12737:12895:12986:13007:13255:13894:14394:14687:21080:21433:21444:21451:21524:21611:21627:21666:21740:30029:30054,0,RBL:209.85.215.195:@gmail.com:.lbl8.mailshell.net-62.18.175.100 66.100.201.100,CacheIP:n one,Baye X-HE-Tag: flag97_3d5715abf0232 X-Filterd-Recvd-Size: 13010 Received: from mail-pg1-f195.google.com (mail-pg1-f195.google.com [209.85.215.195]) by imf38.hostedemail.com (Postfix) with ESMTP for ; Sun, 15 Sep 2019 16:53:41 +0000 (UTC) Received: by mail-pg1-f195.google.com with SMTP id 4so18003731pgm.12 for ; Sun, 15 Sep 2019 09:53:41 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=20161025; h=from:to:cc:subject:date:message-id:in-reply-to:references :mime-version:content-transfer-encoding; bh=wnjVX8y3jjuiJNFHe5uOkqxCWTM5n6lVUDEKEUO860s=; b=HUwCU5p/rPyk8m4saz8aml32uHKw4BTMSh5Y6TIkG4OvTsm0Rb5cgIEbeNfbLpguxi E8etezLznc+kI7KhBiJbN0JttJDe2Se/Tp+9Um+EiS3i/nYQrrJwat3cUlEK/n9Vekt+ TP3TuFIFnFPxRU0+gI1k8D4m8pZUNP5LDDlyp96EdjCiOnwj5lJE2gsAyH3fBJ1y+x7/ bCl73hBcfDBIwPUnsruU9XuzRHKhWdN9d1Oo6zm5msXXFrVEsvLqWNLO8BJ7KDVpU+lK Gzq51dpDFYNhD++DK/3HOYYttS8g/0TodkRqDjoGFlj/I0Zt1xOu4DqCCKcanYoSmzdm YYfg== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20161025; h=x-gm-message-state:from:to:cc:subject:date:message-id:in-reply-to :references:mime-version:content-transfer-encoding; bh=wnjVX8y3jjuiJNFHe5uOkqxCWTM5n6lVUDEKEUO860s=; b=GW8BuRx25UpjM8BMZe+bexjSxIaYypVH/71QB+w8qQyYSEbR76KaQZGD6ipIuBbuwY bJtJ7Lq45jkq8w2yulrlgwYWAyI/iUWnaB/fTH3bAaKvcZwAu0INhUkoPVXob5+s0IuD ZQyfDONqbS10EBIOSSkiCT/xzjh22IviGxLJnwPyWugMDHmIou0RRY5FZ3FeNjswxtgY 3ScHgljpQS+WxFkvNKSaL181r59xBfS9u+gGE0k1RCSOQIoHkkyitPRsoejOET/Wm/mm gGzXJunkdC04BYywUZBZE0XAO6NVzpJvCFFp7y0UcCpM1N+tEPW97GA06TL+2K0Wpm90 eWEw== X-Gm-Message-State: APjAAAUmyS/p68tcsFfc0xj+XUFBUXHUWThbi6e5AEfk3VmdQgy0c8vc PbMF4LS7hvCvXTafXpbQXVgDY4WjAhU= X-Google-Smtp-Source: APXvYqxWna99YStw5YRBrkDQemXY26rnCJB3XGThI8zr10DbF+qP+8f8JFyPMP+4smD1HBg72y8Ikw== X-Received: by 2002:aa7:97aa:: with SMTP id d10mr10482490pfq.176.1568566420540; Sun, 15 Sep 2019 09:53:40 -0700 (PDT) Received: from localhost.localdomain.localdomain ([2408:823c:c11:160:b8c3:8577:bf2f:3]) by smtp.gmail.com with ESMTPSA id a4sm4383259pgq.6.2019.09.15.09.53.31 (version=TLS1_3 cipher=TLS_AES_256_GCM_SHA384 bits=256/256); Sun, 15 Sep 2019 09:53:40 -0700 (PDT) From: Pengfei Li To: akpm@linux-foundation.org Cc: vbabka@suse.cz, cl@linux.com, penberg@kernel.org, rientjes@google.com, iamjoonsoo.kim@lge.com, linux-mm@kvack.org, linux-kernel@vger.kernel.org, guro@fb.com, Pengfei Li Subject: [PATCH v4 7/7] mm, slab_common: modify kmalloc_caches[type][idx] to kmalloc_caches[idx][type] Date: Mon, 16 Sep 2019 00:51:20 +0800 Message-Id: <20190915165121.7237-12-lpf.vector@gmail.com> X-Mailer: git-send-email 2.21.0 In-Reply-To: <20190915165121.7237-1-lpf.vector@gmail.com> References: <20190915165121.7237-1-lpf.vector@gmail.com> MIME-Version: 1.0 X-Bogosity: Ham, tests=bogofilter, spamicity=0.000000, version=1.2.4 Sender: owner-linux-mm@kvack.org Precedence: bulk X-Loop: owner-majordomo@kvack.org List-ID: KMALLOC_NORMAL is the most frequently accessed, and kmalloc_caches[] is initialized by different types of the same size. So modifying kmalloc_caches[type][idx] to kmalloc_caches[idx][type] will benefit performance. $ ./scripts/bloat-o-meter vmlinux.patch_1-6 vmlinux.patch_1-7 add/remove: 0/0 grow/shrink: 2/57 up/down: 8/-457 (-449) Function old new delta tg3_self_test 4255 4259 +4 nf_queue 666 670 +4 kmalloc_slab 97 93 -4 i915_sw_fence_await_dma_fence 441 437 -4 __igmp_group_dropped 619 615 -4 gss_import_sec_context 176 170 -6 xhci_alloc_command 212 205 -7 create_kmalloc_caches 155 148 -7 xprt_switch_alloc 136 128 -8 xhci_segment_alloc 297 289 -8 xhci_ring_alloc 369 361 -8 xhci_mem_init 3664 3656 -8 xhci_alloc_virt_device 496 488 -8 xhci_alloc_tt_info 346 338 -8 xhci_alloc_stream_info 718 710 -8 xhci_alloc_container_ctx 215 207 -8 xfrm_policy_alloc 271 263 -8 tcp_sendmsg_locked 3120 3112 -8 tcp_md5_do_add 774 766 -8 tcp_fastopen_defer_connect 270 262 -8 sr_read_tochdr.isra 251 243 -8 sr_read_tocentry.isra 328 320 -8 sr_is_xa 376 368 -8 sr_get_mcn 260 252 -8 selinux_sk_alloc_security 113 105 -8 sdev_evt_send_simple 118 110 -8 sdev_evt_alloc 79 71 -8 scsi_probe_and_add_lun 2938 2930 -8 sbitmap_queue_init_node 418 410 -8 ring_buffer_read_prepare 94 86 -8 request_firmware_nowait 396 388 -8 regulatory_hint_found_beacon 394 386 -8 ohci_urb_enqueue 3176 3168 -8 nla_strdup 142 134 -8 nfs_alloc_seqid 87 79 -8 nfs4_get_state_owner 1040 1032 -8 nfs4_do_close 578 570 -8 nf_ct_tmpl_alloc 85 77 -8 mempool_create_node 164 156 -8 ip_setup_cork 362 354 -8 ip6_setup_cork 1021 1013 -8 gss_create_cred 140 132 -8 drm_flip_work_allocate_task 70 62 -8 dma_pool_alloc 410 402 -8 devres_open_group 214 206 -8 cfg80211_stop_iface 260 252 -8 cfg80211_sinfo_alloc_tid_stats 77 69 -8 cfg80211_port_authorized 212 204 -8 cfg80211_parse_mbssid_data 2397 2389 -8 cfg80211_ibss_joined 335 327 -8 call_usermodehelper_setup 149 141 -8 bpf_prog_alloc_no_stats 182 174 -8 blk_alloc_flush_queue 191 183 -8 bdi_alloc_node 195 187 -8 audit_log_d_path 196 188 -8 _netlbl_catmap_getnode 247 239 -8 ____ip_mc_inc_group 475 467 -8 __i915_sw_fence_await_sw_fence 417 405 -12 ida_alloc_range 955 934 -21 Total: Before=14874316, After=14873867, chg -0.00% Signed-off-by: Pengfei Li --- include/linux/slab.h | 6 +++--- mm/slab.c | 4 ++-- mm/slab_common.c | 8 ++++---- mm/slub.c | 12 ++++++------ 4 files changed, 15 insertions(+), 15 deletions(-) diff --git a/include/linux/slab.h b/include/linux/slab.h index f53bb6980110..0842db5f7053 100644 --- a/include/linux/slab.h +++ b/include/linux/slab.h @@ -340,7 +340,7 @@ enum kmalloc_cache_type { #ifndef CONFIG_SLOB extern struct kmem_cache * -kmalloc_caches[NR_KMALLOC_TYPES][KMALLOC_CACHE_NUM]; +kmalloc_caches[KMALLOC_CACHE_NUM][NR_KMALLOC_TYPES]; static __always_inline enum kmalloc_cache_type kmalloc_type(gfp_t flags) { @@ -582,7 +582,7 @@ static __always_inline void *kmalloc(size_t size, gfp_t flags) return ZERO_SIZE_PTR; return kmem_cache_alloc_trace( - kmalloc_caches[kmalloc_type(flags)][index], + kmalloc_caches[index][kmalloc_type(flags)], flags, size); #endif } @@ -600,7 +600,7 @@ static __always_inline void *kmalloc_node(size_t size, gfp_t flags, int node) return ZERO_SIZE_PTR; return kmem_cache_alloc_node_trace( - kmalloc_caches[kmalloc_type(flags)][i], + kmalloc_caches[i][kmalloc_type(flags)], flags, node, size); } #endif diff --git a/mm/slab.c b/mm/slab.c index 7bc4e90e1147..079c3e6ced1f 100644 --- a/mm/slab.c +++ b/mm/slab.c @@ -1246,7 +1246,7 @@ void __init kmem_cache_init(void) * Initialize the caches that provide memory for the kmem_cache_node * structures first. Without this, further allocations will bug. */ - kmalloc_caches[KMALLOC_NORMAL][INDEX_NODE] = create_kmalloc_cache( + kmalloc_caches[INDEX_NODE][KMALLOC_NORMAL] = create_kmalloc_cache( kmalloc_info[INDEX_NODE].name[KMALLOC_NORMAL], kmalloc_info[INDEX_NODE].size, ARCH_KMALLOC_FLAGS, 0, @@ -1263,7 +1263,7 @@ void __init kmem_cache_init(void) for_each_online_node(nid) { init_list(kmem_cache, &init_kmem_cache_node[CACHE_CACHE + nid], nid); - init_list(kmalloc_caches[KMALLOC_NORMAL][INDEX_NODE], + init_list(kmalloc_caches[INDEX_NODE][KMALLOC_NORMAL], &init_kmem_cache_node[SIZE_NODE + nid], nid); } } diff --git a/mm/slab_common.c b/mm/slab_common.c index e7903bd28b1f..0f465eae32f6 100644 --- a/mm/slab_common.c +++ b/mm/slab_common.c @@ -1028,7 +1028,7 @@ struct kmem_cache *__init create_kmalloc_cache(const char *name, } struct kmem_cache * -kmalloc_caches[NR_KMALLOC_TYPES][KMALLOC_CACHE_NUM] __ro_after_init = +kmalloc_caches[KMALLOC_CACHE_NUM][NR_KMALLOC_TYPES] __ro_after_init = { /* initialization for https://bugs.llvm.org/show_bug.cgi?id=42570 */ }; EXPORT_SYMBOL(kmalloc_caches); @@ -1090,7 +1090,7 @@ struct kmem_cache *kmalloc_slab(size_t size, gfp_t flags) index = fls(size - 1) - KMALLOC_IDX_ADJ_2; } - return kmalloc_caches[kmalloc_type(flags)][index]; + return kmalloc_caches[index][kmalloc_type(flags)]; } #ifdef CONFIG_ZONE_DMA @@ -1168,7 +1168,7 @@ void __init setup_kmalloc_cache_index_table(void) static __always_inline void __init new_kmalloc_cache(int idx, enum kmalloc_cache_type type, slab_flags_t flags) { - kmalloc_caches[type][idx] = create_kmalloc_cache( + kmalloc_caches[idx][type] = create_kmalloc_cache( kmalloc_info[idx].name[type], kmalloc_info[idx].size, flags, 0, kmalloc_info[idx].size); @@ -1184,7 +1184,7 @@ void __init create_kmalloc_caches(slab_flags_t flags) int i; for (i = 0; i < KMALLOC_CACHE_NUM; i++) { - if (!kmalloc_caches[KMALLOC_NORMAL][i]) + if (!kmalloc_caches[i][KMALLOC_NORMAL]) new_kmalloc_cache(i, KMALLOC_NORMAL, flags); new_kmalloc_cache(i, KMALLOC_RECLAIM, diff --git a/mm/slub.c b/mm/slub.c index 0e92ebdcacc9..e87243a16768 100644 --- a/mm/slub.c +++ b/mm/slub.c @@ -4711,7 +4711,7 @@ static void __init resiliency_test(void) pr_err("\n1. kmalloc-16: Clobber Redzone/next pointer 0x12->0x%p\n\n", p + 16); - validate_slab_cache(kmalloc_caches[type][1]); + validate_slab_cache(kmalloc_caches[1][type]); /* Hmmm... The next two are dangerous */ p = kzalloc(32, GFP_KERNEL); @@ -4720,33 +4720,33 @@ static void __init resiliency_test(void) p); pr_err("If allocated object is overwritten then not detectable\n\n"); - validate_slab_cache(kmalloc_caches[type][2]); + validate_slab_cache(kmalloc_caches[2][type]); p = kzalloc(64, GFP_KERNEL); p += 64 + (get_cycles() & 0xff) * sizeof(void *); *p = 0x56; pr_err("\n3. kmalloc-64: corrupting random byte 0x56->0x%p\n", p); pr_err("If allocated object is overwritten then not detectable\n\n"); - validate_slab_cache(kmalloc_caches[type][3]); + validate_slab_cache(kmalloc_caches[3][type]); pr_err("\nB. Corruption after free\n"); p = kzalloc(128, GFP_KERNEL); kfree(p); *p = 0x78; pr_err("1. kmalloc-128: Clobber first word 0x78->0x%p\n\n", p); - validate_slab_cache(kmalloc_caches[type][5]); + validate_slab_cache(kmalloc_caches[5][type]); p = kzalloc(256, GFP_KERNEL); kfree(p); p[50] = 0x9a; pr_err("\n2. kmalloc-256: Clobber 50th byte 0x9a->0x%p\n\n", p); - validate_slab_cache(kmalloc_caches[type][7]); + validate_slab_cache(kmalloc_caches[7][type]); p = kzalloc(512, GFP_KERNEL); kfree(p); p[512] = 0xab; pr_err("\n3. kmalloc-512: Clobber redzone 0xab->0x%p\n\n", p); - validate_slab_cache(kmalloc_caches[type][8]); + validate_slab_cache(kmalloc_caches[8][type]); } #else #ifdef CONFIG_SYSFS