From patchwork Wed Aug 17 10:18:10 2022 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Hyeonggon Yoo <42.hyeyoo@gmail.com> X-Patchwork-Id: 12945680 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from kanga.kvack.org (kanga.kvack.org [205.233.56.17]) by smtp.lore.kernel.org (Postfix) with ESMTP id 02A9CC3F6B0 for ; Wed, 17 Aug 2022 10:18:51 +0000 (UTC) Received: by kanga.kvack.org (Postfix) id 681CC6B0074; Wed, 17 Aug 2022 06:18:51 -0400 (EDT) Received: by kanga.kvack.org (Postfix, from userid 40) id 631156B0075; Wed, 17 Aug 2022 06:18:51 -0400 (EDT) X-Delivered-To: int-list-linux-mm@kvack.org Received: by kanga.kvack.org (Postfix, from userid 63042) id 4AB008D0001; Wed, 17 Aug 2022 06:18:51 -0400 (EDT) X-Delivered-To: linux-mm@kvack.org Received: from relay.hostedemail.com (smtprelay0015.hostedemail.com [216.40.44.15]) by kanga.kvack.org (Postfix) with ESMTP id 396BB6B0074 for ; Wed, 17 Aug 2022 06:18:51 -0400 (EDT) Received: from smtpin24.hostedemail.com (a10.router.float.18 [10.200.18.1]) by unirelay05.hostedemail.com (Postfix) with ESMTP id E8AA1408CD for ; Wed, 17 Aug 2022 10:18:50 +0000 (UTC) X-FDA: 79808686020.24.9415925 Received: from mail-pl1-f171.google.com (mail-pl1-f171.google.com [209.85.214.171]) by imf20.hostedemail.com (Postfix) with ESMTP id 6D6391C01B9 for ; Wed, 17 Aug 2022 10:18:50 +0000 (UTC) Received: by mail-pl1-f171.google.com with SMTP id d10so11627543plr.6 for ; Wed, 17 Aug 2022 03:18:50 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=20210112; h=content-transfer-encoding:mime-version:references:in-reply-to :message-id:date:subject:cc:to:from:from:to:cc; bh=YF+ekAMEny9NvsTn9a4lw7RA5UzOw69huq16Y0TTu/E=; b=UfKWZydc6cWoE65QiAxGGqIjSiAtD0FTW2ouaiH24BCMLqRxnKci1GLgS3gQ0c38ek h0RdQwvcWm8nSdGolt0rJF9eWWHjgf55b7AQqnT8MrSz7KCgCBMK18+jtkbyltSBiYT7 GeQFS07ovAMivSBIlqQUviSC17HCZxz/u98IdnOkiuGW0dsKh6bqUINMIoYBWrTEU/FL E8jRoYD9dG4GB74/3ZIBuyUptfi6z7J/NpbxwzIrwwqUmrRO4wLe8XtVhJ48Sxvv92XM aYiFRioz4iekqq1KWjYcFnBXtIVDWLY53Ua9PQmCFRHAhGcsl5HcgVlOCk64KtGLNm1j XqTQ== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20210112; h=content-transfer-encoding:mime-version:references:in-reply-to :message-id:date:subject:cc:to:from:x-gm-message-state:from:to:cc; bh=YF+ekAMEny9NvsTn9a4lw7RA5UzOw69huq16Y0TTu/E=; b=AINo1HoNtMppJdcPkc4WiRKKlLF28qqS4H95lDYkzZIoWROVcqEgLvaW+M8boy083R 8x/gTIuuW5OKb0kjZV9A1gdUNy0PUIB0O76KwoSzjmHdgFwyienbKA8RSeWOpvEGlmRA R02NOgw4sgOSQabaY9xv9hJmhpV10s0XV7EhKzZO6EF1stFoEulf+R+ImKQfOw6K4e8S 4x9DOqGhZK86bx/IvOEBnGqoduJnHdFf9ZPdi9e6n7mADNAZZGnOIzCB+8LnCFFkn7Fm VXkopgoazNy2FnRh/8LsNgEXn+ls1lcTbgfOfog9mzPbt9+wLFdFE0TAjraz/L27gNuz XKxA== X-Gm-Message-State: ACgBeo2SSUc8yKUkqMP7KFCkNcz6Ide2Pmn0xeaiMB93mISxQX/6uWy1 PBmzElCFYLfZT/jmmmzNhXc= X-Google-Smtp-Source: AA6agR54kAAuhdp73DFDVedKK36Zsc3Hl4XZr1ca5vnebvrjU6WllrJ9ITzm7kSiuXYJqzEV6/nhpQ== X-Received: by 2002:a17:902:bd41:b0:172:74c9:2a08 with SMTP id b1-20020a170902bd4100b0017274c92a08mr10779640plx.9.1660731529461; Wed, 17 Aug 2022 03:18:49 -0700 (PDT) Received: from hyeyoo.. ([114.29.91.56]) by smtp.gmail.com with ESMTPSA id d8-20020a170903230800b00172633fc236sm1071318plh.174.2022.08.17.03.18.45 (version=TLS1_3 cipher=TLS_AES_256_GCM_SHA384 bits=256/256); Wed, 17 Aug 2022 03:18:48 -0700 (PDT) From: Hyeonggon Yoo <42.hyeyoo@gmail.com> To: Christoph Lameter , Pekka Enberg , David Rientjes , Joonsoo Kim , Andrew Morton , Vlastimil Babka , Roman Gushchin Cc: Hyeonggon Yoo <42.hyeyoo@gmail.com>, linux-mm@kvack.org, linux-kernel@vger.kernel.org Subject: [PATCH v4 01/17] mm/slab: move NUMA-related code to __do_cache_alloc() Date: Wed, 17 Aug 2022 19:18:10 +0900 Message-Id: <20220817101826.236819-2-42.hyeyoo@gmail.com> X-Mailer: git-send-email 2.32.0 In-Reply-To: <20220817101826.236819-1-42.hyeyoo@gmail.com> References: <20220817101826.236819-1-42.hyeyoo@gmail.com> MIME-Version: 1.0 ARC-Seal: i=1; s=arc-20220608; d=hostedemail.com; t=1660731530; a=rsa-sha256; cv=none; b=jHmOvvo0iL486IsSTM/kcFsiMf00hvVFBqknc4BV4szIX709R1JcQkOxmpj7q45HGlP0wk kB7/XjC+RbU7o5eJtOtKEgtVv+IVYF0oQ/CM1596fZmelYJOKfcZLJNPn8oIe1y6X3BDti BeRkQtQOwHdnsz7fdUnn54zI6ffaKZE= ARC-Authentication-Results: i=1; imf20.hostedemail.com; dkim=pass header.d=gmail.com header.s=20210112 header.b=UfKWZydc; spf=pass (imf20.hostedemail.com: domain of 42.hyeyoo@gmail.com designates 209.85.214.171 as permitted sender) smtp.mailfrom=42.hyeyoo@gmail.com; dmarc=pass (policy=none) header.from=gmail.com ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=hostedemail.com; s=arc-20220608; t=1660731530; h=from:from:sender:reply-to:subject:subject:date:date: message-id:message-id:to:to:cc:cc:mime-version:mime-version: content-type:content-transfer-encoding:content-transfer-encoding: in-reply-to:in-reply-to:references:references:dkim-signature; bh=YF+ekAMEny9NvsTn9a4lw7RA5UzOw69huq16Y0TTu/E=; b=ti28T3cuhomkpra8AjhngEtCu4sQaJ+zKzYV9oZsT+SeVYcYgcluE/x2TKqFtHs/fiklw+ 0AtD8vTcuNphkedxSRcBRpVcu6/6FZ9+nu5HgTkbRSXzHj4FzfTECgMwNU5OnPm4vNvdT9 g9oSgPYl0TSk3pjlalugqaALd3YAtl4= Authentication-Results: imf20.hostedemail.com; dkim=pass header.d=gmail.com header.s=20210112 header.b=UfKWZydc; spf=pass (imf20.hostedemail.com: domain of 42.hyeyoo@gmail.com designates 209.85.214.171 as permitted sender) smtp.mailfrom=42.hyeyoo@gmail.com; dmarc=pass (policy=none) header.from=gmail.com X-Rspamd-Server: rspam02 X-Rspamd-Queue-Id: 6D6391C01B9 X-Stat-Signature: wrhfwyk8dt8kcbaa4ijjz7utwkncwop4 X-Rspam-User: X-HE-Tag: 1660731530-137874 X-Bogosity: Ham, tests=bogofilter, spamicity=0.000000, version=1.2.4 Sender: owner-linux-mm@kvack.org Precedence: bulk X-Loop: owner-majordomo@kvack.org List-ID: To implement slab_alloc_node() independent of NUMA configuration, move NUMA fallback/alternate allocation code into __do_cache_alloc(). One functional change here is not to check availability of node when allocating from local node. Signed-off-by: Hyeonggon Yoo <42.hyeyoo@gmail.com> Reviewed-by: Vlastimil Babka --- mm/slab.c | 68 +++++++++++++++++++++++++------------------------------ 1 file changed, 31 insertions(+), 37 deletions(-) diff --git a/mm/slab.c b/mm/slab.c index 10e96137b44f..1656393f55cb 100644 --- a/mm/slab.c +++ b/mm/slab.c @@ -3180,13 +3180,14 @@ static void *____cache_alloc_node(struct kmem_cache *cachep, gfp_t flags, return obj ? obj : fallback_alloc(cachep, flags); } +static void *__do_cache_alloc(struct kmem_cache *cachep, gfp_t flags, int nodeid); + static __always_inline void * slab_alloc_node(struct kmem_cache *cachep, gfp_t flags, int nodeid, size_t orig_size, unsigned long caller) { unsigned long save_flags; void *ptr; - int slab_node = numa_mem_id(); struct obj_cgroup *objcg = NULL; bool init = false; @@ -3200,30 +3201,7 @@ slab_alloc_node(struct kmem_cache *cachep, gfp_t flags, int nodeid, size_t orig_ goto out_hooks; local_irq_save(save_flags); - - if (nodeid == NUMA_NO_NODE) - nodeid = slab_node; - - if (unlikely(!get_node(cachep, nodeid))) { - /* Node not bootstrapped yet */ - ptr = fallback_alloc(cachep, flags); - goto out; - } - - if (nodeid == slab_node) { - /* - * Use the locally cached objects if possible. - * However ____cache_alloc does not allow fallback - * to other nodes. It may fail while we still have - * objects on other nodes available. - */ - ptr = ____cache_alloc(cachep, flags); - if (ptr) - goto out; - } - /* ___cache_alloc_node can fall back to other nodes */ - ptr = ____cache_alloc_node(cachep, flags, nodeid); -out: + ptr = __do_cache_alloc(cachep, flags, nodeid); local_irq_restore(save_flags); ptr = cache_alloc_debugcheck_after(cachep, flags, ptr, caller); init = slab_want_init_on_alloc(flags, cachep); @@ -3234,31 +3212,46 @@ slab_alloc_node(struct kmem_cache *cachep, gfp_t flags, int nodeid, size_t orig_ } static __always_inline void * -__do_cache_alloc(struct kmem_cache *cache, gfp_t flags) +__do_cache_alloc(struct kmem_cache *cachep, gfp_t flags, int nodeid) { - void *objp; + void *objp = NULL; + int slab_node = numa_mem_id(); - if (current->mempolicy || cpuset_do_slab_mem_spread()) { - objp = alternate_node_alloc(cache, flags); - if (objp) - goto out; + if (nodeid == NUMA_NO_NODE) { + if (current->mempolicy || cpuset_do_slab_mem_spread()) { + objp = alternate_node_alloc(cachep, flags); + if (objp) + goto out; + } + /* + * Use the locally cached objects if possible. + * However ____cache_alloc does not allow fallback + * to other nodes. It may fail while we still have + * objects on other nodes available. + */ + objp = ____cache_alloc(cachep, flags); + nodeid = slab_node; + } else if (nodeid == slab_node) { + objp = ____cache_alloc(cachep, flags); + } else if (!get_node(cachep, nodeid)) { + /* Node not bootstrapped yet */ + objp = fallback_alloc(cachep, flags); + goto out; } - objp = ____cache_alloc(cache, flags); /* * We may just have run out of memory on the local node. * ____cache_alloc_node() knows how to locate memory on other nodes */ if (!objp) - objp = ____cache_alloc_node(cache, flags, numa_mem_id()); - + objp = ____cache_alloc_node(cachep, flags, nodeid); out: return objp; } #else static __always_inline void * -__do_cache_alloc(struct kmem_cache *cachep, gfp_t flags) +__do_cache_alloc(struct kmem_cache *cachep, gfp_t flags, int nodeid __maybe_unused) { return ____cache_alloc(cachep, flags); } @@ -3284,7 +3277,7 @@ slab_alloc(struct kmem_cache *cachep, struct list_lru *lru, gfp_t flags, goto out; local_irq_save(save_flags); - objp = __do_cache_alloc(cachep, flags); + objp = __do_cache_alloc(cachep, flags, NUMA_NO_NODE); local_irq_restore(save_flags); objp = cache_alloc_debugcheck_after(cachep, flags, objp, caller); prefetchw(objp); @@ -3521,7 +3514,8 @@ int kmem_cache_alloc_bulk(struct kmem_cache *s, gfp_t flags, size_t size, local_irq_disable(); for (i = 0; i < size; i++) { - void *objp = kfence_alloc(s, s->object_size, flags) ?: __do_cache_alloc(s, flags); + void *objp = kfence_alloc(s, s->object_size, flags) ?: + __do_cache_alloc(s, flags, NUMA_NO_NODE); if (unlikely(!objp)) goto error; From patchwork Wed Aug 17 10:18:11 2022 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Hyeonggon Yoo <42.hyeyoo@gmail.com> X-Patchwork-Id: 12945681 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from kanga.kvack.org (kanga.kvack.org [205.233.56.17]) by smtp.lore.kernel.org (Postfix) with ESMTP id 4BA87C25B08 for ; Wed, 17 Aug 2022 10:18:55 +0000 (UTC) Received: by kanga.kvack.org (Postfix) id BB1B96B0075; Wed, 17 Aug 2022 06:18:54 -0400 (EDT) Received: by kanga.kvack.org (Postfix, from userid 40) id B61256B0078; Wed, 17 Aug 2022 06:18:54 -0400 (EDT) X-Delivered-To: int-list-linux-mm@kvack.org Received: by kanga.kvack.org (Postfix, from userid 63042) id 9DB598D0001; Wed, 17 Aug 2022 06:18:54 -0400 (EDT) X-Delivered-To: linux-mm@kvack.org Received: from relay.hostedemail.com (smtprelay0012.hostedemail.com [216.40.44.12]) by kanga.kvack.org (Postfix) with ESMTP id 903F56B0075 for ; Wed, 17 Aug 2022 06:18:54 -0400 (EDT) Received: from smtpin01.hostedemail.com (a10.router.float.18 [10.200.18.1]) by unirelay05.hostedemail.com (Postfix) with ESMTP id 76D324032F for ; Wed, 17 Aug 2022 10:18:54 +0000 (UTC) X-FDA: 79808686188.01.2C67953 Received: from mail-pl1-f171.google.com (mail-pl1-f171.google.com [209.85.214.171]) by imf17.hostedemail.com (Postfix) with ESMTP id 1664E401D5 for ; Wed, 17 Aug 2022 10:18:53 +0000 (UTC) Received: by mail-pl1-f171.google.com with SMTP id p18so11625428plr.8 for ; Wed, 17 Aug 2022 03:18:53 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=20210112; h=content-transfer-encoding:mime-version:references:in-reply-to :message-id:date:subject:cc:to:from:from:to:cc; bh=3U+8LTY1y1RiFgIZBWF6u26rllF04yVkQZAKSsbfV88=; b=o+eLIVEt1kKstQYEPuqhNvCnf19Ci8D4sguN4nwyRDg+FIafrV/2wNuYCQ2hFw3zGm S6N7ygUZ8Y3rRkuN6woRRJ+Quhna5GnJ2uHgH0Q9rBOQfQqNAV7WITAIQeyeoNrsvKml Jpb+Vrz1P+0ZDV94y7bs9t7AvX0CKI1ML3eaeNBRxGzAeTOEKamIM5xHaBoRzs5/Ezfr VwBcXDv+0j2RdEMV+SS/24NpVShba96jr/Gh84kADiL2bA2natEjWxx0V3htEPQJU204 MYfnMt26N2c4+oh7JQl/ePMkQ/TZMX9uSZxy7tqjfgkNazF5/K7U2pMO1gQc9rOg7+Yg uK8g== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20210112; h=content-transfer-encoding:mime-version:references:in-reply-to :message-id:date:subject:cc:to:from:x-gm-message-state:from:to:cc; bh=3U+8LTY1y1RiFgIZBWF6u26rllF04yVkQZAKSsbfV88=; b=lS3t2o3gHUsejayqjMeU3BiGltKluE/KgZqMO83Qa1B74O0YiwoTazXVqmqUbBVShd 6wOCBqQl6dDvZOwLlj5cZtTj84j7enRT+hTcveYAVDRrWVJtWuA2ht4xdWA7zMMAOK6d kV+WE0ihRHwiW09Eci53S2w28LNgEWLiQ4f4OBFs18208WCc3ERo9saPAA2ClpxCYpio xYPKjvgGlzeiJAdtRTbXR+zoJrCQx897AwTYWxJWxYO/Pm/vpqd286K4jyqk9j9rGYhk P58u2+/JKlvY9JN8zAdHp/fKkFTppqjuZFk20dnEqR9+tgQmydacb/gtjJKX1IepvVMb wwng== X-Gm-Message-State: ACgBeo3EHs70k9f4tglcmoL/foow9fJw/TQ9K+Z5oYUmCUAbV7toWW1J o0zRqCHFH+GTdeb4K2WDdTQ= X-Google-Smtp-Source: AA6agR5y8J8WikcfzF1i0kGroiRKnzVfSzPJ1UB2rSumOIO5ossoFYJ0lYdU+nkXvjiUEBedYMvovg== X-Received: by 2002:a17:902:d2c6:b0:16e:d285:c602 with SMTP id n6-20020a170902d2c600b0016ed285c602mr25604783plc.81.1660731533224; Wed, 17 Aug 2022 03:18:53 -0700 (PDT) Received: from hyeyoo.. ([114.29.91.56]) by smtp.gmail.com with ESMTPSA id d8-20020a170903230800b00172633fc236sm1071318plh.174.2022.08.17.03.18.49 (version=TLS1_3 cipher=TLS_AES_256_GCM_SHA384 bits=256/256); Wed, 17 Aug 2022 03:18:52 -0700 (PDT) From: Hyeonggon Yoo <42.hyeyoo@gmail.com> To: Christoph Lameter , Pekka Enberg , David Rientjes , Joonsoo Kim , Andrew Morton , Vlastimil Babka , Roman Gushchin Cc: Hyeonggon Yoo <42.hyeyoo@gmail.com>, linux-mm@kvack.org, linux-kernel@vger.kernel.org Subject: [PATCH v4 02/17] mm/slab: cleanup slab_alloc() and slab_alloc_node() Date: Wed, 17 Aug 2022 19:18:11 +0900 Message-Id: <20220817101826.236819-3-42.hyeyoo@gmail.com> X-Mailer: git-send-email 2.32.0 In-Reply-To: <20220817101826.236819-1-42.hyeyoo@gmail.com> References: <20220817101826.236819-1-42.hyeyoo@gmail.com> MIME-Version: 1.0 ARC-Seal: i=1; s=arc-20220608; d=hostedemail.com; t=1660731534; a=rsa-sha256; cv=none; b=O8iLi5T1nGxm7oXeVK6xNP4BkpEU9OZMyQWfjDjKCTPBIO8aMpK84KE7w53D4jV+9kdIop VEw7P9cyeZH/IsSGQcH77IEXy1limykM3sMHKXZ/1uYp3IC5xa3vwg84dtybr1RKyQ245Y B5bghujpw7JxjY2pgsWpYP2j2vyX/jU= ARC-Authentication-Results: i=1; imf17.hostedemail.com; dkim=pass header.d=gmail.com header.s=20210112 header.b=o+eLIVEt; spf=pass (imf17.hostedemail.com: domain of 42.hyeyoo@gmail.com designates 209.85.214.171 as permitted sender) smtp.mailfrom=42.hyeyoo@gmail.com; dmarc=pass (policy=none) header.from=gmail.com ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=hostedemail.com; s=arc-20220608; t=1660731534; h=from:from:sender:reply-to:subject:subject:date:date: message-id:message-id:to:to:cc:cc:mime-version:mime-version: content-type:content-transfer-encoding:content-transfer-encoding: in-reply-to:in-reply-to:references:references:dkim-signature; bh=3U+8LTY1y1RiFgIZBWF6u26rllF04yVkQZAKSsbfV88=; b=ZsOKcaVK3W8DRPk0DUugYkswY14GAyzWfhuvbYu2H2sUwJbdHHo2OMkfnxQ2ti/7jOBvF7 WPwD9NmgH1mqSnWGbu4XX797/aR3VQuGLxOAEl8spKb5X/UhHjznPIvFQzT6v5+Bv8MSVa p8VZnVh1VO95R5gjKG+ch2+CJRdP22w= Authentication-Results: imf17.hostedemail.com; dkim=pass header.d=gmail.com header.s=20210112 header.b=o+eLIVEt; spf=pass (imf17.hostedemail.com: domain of 42.hyeyoo@gmail.com designates 209.85.214.171 as permitted sender) smtp.mailfrom=42.hyeyoo@gmail.com; dmarc=pass (policy=none) header.from=gmail.com X-Rspamd-Server: rspam02 X-Rspamd-Queue-Id: 1664E401D5 X-Stat-Signature: f7xno8zq6cwm3auroooh8wrfhytbqbm3 X-Rspam-User: X-HE-Tag: 1660731533-529799 X-Bogosity: Ham, tests=bogofilter, spamicity=0.000000, version=1.2.4 Sender: owner-linux-mm@kvack.org Precedence: bulk X-Loop: owner-majordomo@kvack.org List-ID: Make slab_alloc_node() available even when CONFIG_NUMA=n and make slab_alloc() wrapper of slab_alloc_node(). This is necessary for further cleanup. Signed-off-by: Hyeonggon Yoo <42.hyeyoo@gmail.com> Reviewed-by: Vlastimil Babka --- mm/slab.c | 49 +++++++++++++------------------------------------ 1 file changed, 13 insertions(+), 36 deletions(-) diff --git a/mm/slab.c b/mm/slab.c index 1656393f55cb..748dd085f38e 100644 --- a/mm/slab.c +++ b/mm/slab.c @@ -3180,37 +3180,6 @@ static void *____cache_alloc_node(struct kmem_cache *cachep, gfp_t flags, return obj ? obj : fallback_alloc(cachep, flags); } -static void *__do_cache_alloc(struct kmem_cache *cachep, gfp_t flags, int nodeid); - -static __always_inline void * -slab_alloc_node(struct kmem_cache *cachep, gfp_t flags, int nodeid, size_t orig_size, - unsigned long caller) -{ - unsigned long save_flags; - void *ptr; - struct obj_cgroup *objcg = NULL; - bool init = false; - - flags &= gfp_allowed_mask; - cachep = slab_pre_alloc_hook(cachep, NULL, &objcg, 1, flags); - if (unlikely(!cachep)) - return NULL; - - ptr = kfence_alloc(cachep, orig_size, flags); - if (unlikely(ptr)) - goto out_hooks; - - local_irq_save(save_flags); - ptr = __do_cache_alloc(cachep, flags, nodeid); - local_irq_restore(save_flags); - ptr = cache_alloc_debugcheck_after(cachep, flags, ptr, caller); - init = slab_want_init_on_alloc(flags, cachep); - -out_hooks: - slab_post_alloc_hook(cachep, objcg, flags, 1, &ptr, init); - return ptr; -} - static __always_inline void * __do_cache_alloc(struct kmem_cache *cachep, gfp_t flags, int nodeid) { @@ -3259,8 +3228,8 @@ __do_cache_alloc(struct kmem_cache *cachep, gfp_t flags, int nodeid __maybe_unus #endif /* CONFIG_NUMA */ static __always_inline void * -slab_alloc(struct kmem_cache *cachep, struct list_lru *lru, gfp_t flags, - size_t orig_size, unsigned long caller) +slab_alloc_node(struct kmem_cache *cachep, struct list_lru *lru, gfp_t flags, + int nodeid, size_t orig_size, unsigned long caller) { unsigned long save_flags; void *objp; @@ -3277,7 +3246,7 @@ slab_alloc(struct kmem_cache *cachep, struct list_lru *lru, gfp_t flags, goto out; local_irq_save(save_flags); - objp = __do_cache_alloc(cachep, flags, NUMA_NO_NODE); + objp = __do_cache_alloc(cachep, flags, nodeid); local_irq_restore(save_flags); objp = cache_alloc_debugcheck_after(cachep, flags, objp, caller); prefetchw(objp); @@ -3288,6 +3257,14 @@ slab_alloc(struct kmem_cache *cachep, struct list_lru *lru, gfp_t flags, return objp; } +static __always_inline void * +slab_alloc(struct kmem_cache *cachep, struct list_lru *lru, gfp_t flags, + size_t orig_size, unsigned long caller) +{ + return slab_alloc_node(cachep, lru, flags, NUMA_NO_NODE, orig_size, + caller); +} + /* * Caller needs to acquire correct kmem_cache_node's list_lock * @list: List of detached free slabs should be freed by caller @@ -3574,7 +3551,7 @@ EXPORT_SYMBOL(kmem_cache_alloc_trace); */ void *kmem_cache_alloc_node(struct kmem_cache *cachep, gfp_t flags, int nodeid) { - void *ret = slab_alloc_node(cachep, flags, nodeid, cachep->object_size, _RET_IP_); + void *ret = slab_alloc_node(cachep, NULL, flags, nodeid, cachep->object_size, _RET_IP_); trace_kmem_cache_alloc_node(_RET_IP_, ret, cachep, cachep->object_size, cachep->size, @@ -3592,7 +3569,7 @@ void *kmem_cache_alloc_node_trace(struct kmem_cache *cachep, { void *ret; - ret = slab_alloc_node(cachep, flags, nodeid, size, _RET_IP_); + ret = slab_alloc_node(cachep, NULL, flags, nodeid, size, _RET_IP_); ret = kasan_kmalloc(cachep, ret, size, flags); trace_kmalloc_node(_RET_IP_, ret, cachep, From patchwork Wed Aug 17 10:18:12 2022 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Hyeonggon Yoo <42.hyeyoo@gmail.com> X-Patchwork-Id: 12945682 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from kanga.kvack.org (kanga.kvack.org [205.233.56.17]) by smtp.lore.kernel.org (Postfix) with ESMTP id 191CFC25B08 for ; Wed, 17 Aug 2022 10:18:59 +0000 (UTC) Received: by kanga.kvack.org (Postfix) id A51F66B0078; Wed, 17 Aug 2022 06:18:58 -0400 (EDT) Received: by kanga.kvack.org (Postfix, from userid 40) id A016C8D0001; Wed, 17 Aug 2022 06:18:58 -0400 (EDT) X-Delivered-To: int-list-linux-mm@kvack.org Received: by kanga.kvack.org (Postfix, from userid 63042) id 8542E6B007D; Wed, 17 Aug 2022 06:18:58 -0400 (EDT) X-Delivered-To: linux-mm@kvack.org Received: from relay.hostedemail.com (smtprelay0010.hostedemail.com [216.40.44.10]) by kanga.kvack.org (Postfix) with ESMTP id 757B06B0078 for ; Wed, 17 Aug 2022 06:18:58 -0400 (EDT) Received: from smtpin12.hostedemail.com (a10.router.float.18 [10.200.18.1]) by unirelay02.hostedemail.com (Postfix) with ESMTP id 501B61205CA for ; Wed, 17 Aug 2022 10:18:58 +0000 (UTC) X-FDA: 79808686356.12.5AD809E Received: from mail-pf1-f171.google.com (mail-pf1-f171.google.com [209.85.210.171]) by imf07.hostedemail.com (Postfix) with ESMTP id E32F6401CE for ; Wed, 17 Aug 2022 10:18:57 +0000 (UTC) Received: by mail-pf1-f171.google.com with SMTP id f30so11683921pfq.4 for ; Wed, 17 Aug 2022 03:18:57 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=20210112; h=content-transfer-encoding:mime-version:references:in-reply-to :message-id:date:subject:cc:to:from:from:to:cc; bh=dErSdygiPBNsBAP9y5F3S7ekn7p7kWL1SSm4SRF+jts=; b=KIqkM8kQCeSA6lOVWRvpaj88kdiJxGl/Ysg2mgJZh2bmi8Z7ykNum+Sat5cbEwzI4o tJ9CQgyQ2lQHwfH1rn/NIri+J4RdqujaoQE4celkgwmqnSgxb6MM9n8Ugfl0YgQFVCCF qbEJkWrfYGeokUQ2kxtj6TpA/TmzBVhMBFIYW4H9R2DnFRc9UgPJp8ELDkGVqgXWUlSa 7qDrKZStXLbr9GKkt0ktkM2F1xg/QoZIRR2LRd6Tqjdmq9bkvc6nUDi99zje8wiX0TWW DTJx/nZe3iCU60gnNHxi+W3B5sZTCWYzMNPsAJAV2wB1I7IouxSJYPE29lRRuHDGYAVK kYNw== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20210112; h=content-transfer-encoding:mime-version:references:in-reply-to :message-id:date:subject:cc:to:from:x-gm-message-state:from:to:cc; bh=dErSdygiPBNsBAP9y5F3S7ekn7p7kWL1SSm4SRF+jts=; b=xCsicdYmHeSd37+D+ahxkGWZoRH4Oy1VInUJWLzJcu4EZpQzgST2wc6Xp1KbE+M6zn QhtjCd7Qe11vIMADYuPBhIOkX9p+z/I/gJQR/DHfW8wJ2BMCZEvPj/MCdzsgzsWG+CL6 0+L6cBKMTSD0bPfZv8FfCSecpbJkhrqAbDQebm6VxBjCucI0lF/BsO0Nz9aKfTde9PNw 5PzcJHmyA6z8HGQe1pHMlKyBevi9Aadtg2nbueohQ3UAtvibQOSQGqB4KCCzVPJC8R5Z vWZjD54V8YaCYhlSPwbSEnTfZj7FUXobI77F7dyPTnK8yJXfPPFiOiTiQ6qeDd9a1aQd M4Uw== X-Gm-Message-State: ACgBeo3St4YEhR16gSnelwCKmars2z6cq2H+kIJBtQ+YGoCjlWd7HHyW VND6Uz8jgkMr3NjC7Dlq7vY= X-Google-Smtp-Source: AA6agR50UeU3HEpeP9GVa1wNnr14Ztlk2ksOg095VJMlH8rYkzaa0EC00mWPps4A86HdPSen/TmygQ== X-Received: by 2002:a05:6a00:1910:b0:52f:13d7:44c4 with SMTP id y16-20020a056a00191000b0052f13d744c4mr24266690pfi.32.1660731536946; Wed, 17 Aug 2022 03:18:56 -0700 (PDT) Received: from hyeyoo.. ([114.29.91.56]) by smtp.gmail.com with ESMTPSA id d8-20020a170903230800b00172633fc236sm1071318plh.174.2022.08.17.03.18.53 (version=TLS1_3 cipher=TLS_AES_256_GCM_SHA384 bits=256/256); Wed, 17 Aug 2022 03:18:56 -0700 (PDT) From: Hyeonggon Yoo <42.hyeyoo@gmail.com> To: Christoph Lameter , Pekka Enberg , David Rientjes , Joonsoo Kim , Andrew Morton , Vlastimil Babka , Roman Gushchin Cc: Hyeonggon Yoo <42.hyeyoo@gmail.com>, linux-mm@kvack.org, linux-kernel@vger.kernel.org Subject: [PATCH v4 03/17] mm/slab_common: remove CONFIG_NUMA ifdefs for common kmalloc functions Date: Wed, 17 Aug 2022 19:18:12 +0900 Message-Id: <20220817101826.236819-4-42.hyeyoo@gmail.com> X-Mailer: git-send-email 2.32.0 In-Reply-To: <20220817101826.236819-1-42.hyeyoo@gmail.com> References: <20220817101826.236819-1-42.hyeyoo@gmail.com> MIME-Version: 1.0 ARC-Seal: i=1; s=arc-20220608; d=hostedemail.com; t=1660731538; a=rsa-sha256; cv=none; b=0alHWwkIy/MKqpiN7CnJnz5U+aWOhy8me5J55v08sR+565CcQo3DVo0ibBa3ZL1lfjPxTL xQMQ3Rj3riByyHyDyyLNDWqEWx5IzJ0iOM2yDbBisQIvDqUZN/YkIHOS67fbTm6dnxaD/n 4jc1TADoXyXiW98/5jM0wt4aB0bjpsE= ARC-Authentication-Results: i=1; imf07.hostedemail.com; dkim=pass header.d=gmail.com header.s=20210112 header.b=KIqkM8kQ; dmarc=pass (policy=none) header.from=gmail.com; spf=pass (imf07.hostedemail.com: domain of 42.hyeyoo@gmail.com designates 209.85.210.171 as permitted sender) smtp.mailfrom=42.hyeyoo@gmail.com ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=hostedemail.com; s=arc-20220608; t=1660731538; h=from:from:sender:reply-to:subject:subject:date:date: message-id:message-id:to:to:cc:cc:mime-version:mime-version: content-type:content-transfer-encoding:content-transfer-encoding: in-reply-to:in-reply-to:references:references:dkim-signature; bh=dErSdygiPBNsBAP9y5F3S7ekn7p7kWL1SSm4SRF+jts=; b=nLfAQhA0BoCXexkgweLdsFvb42sv8KS5sLNzBPTigzux00WCa/fvECNWrKZWMjdVrp7xAE Gm1qldGF8s3/eLBsbLl8uGYX1YHPQ//n+r16k9/DPkJvwjQ5XuQmYmnPcXG2Sn78Tod83I 6BP6IO8/I2lNI48Z8KIW7C4bWtBQ020= X-Stat-Signature: pe4e94wxiw4y5gfbwwi9a1uxhoy65cxr X-Rspamd-Queue-Id: E32F6401CE Authentication-Results: imf07.hostedemail.com; dkim=pass header.d=gmail.com header.s=20210112 header.b=KIqkM8kQ; dmarc=pass (policy=none) header.from=gmail.com; spf=pass (imf07.hostedemail.com: domain of 42.hyeyoo@gmail.com designates 209.85.210.171 as permitted sender) smtp.mailfrom=42.hyeyoo@gmail.com X-Rspam-User: X-Rspamd-Server: rspam10 X-HE-Tag: 1660731537-936899 X-Bogosity: Ham, tests=bogofilter, spamicity=0.000000, version=1.2.4 Sender: owner-linux-mm@kvack.org Precedence: bulk X-Loop: owner-majordomo@kvack.org List-ID: Now that slab_alloc_node() is available for SLAB when CONFIG_NUMA=n, remove CONFIG_NUMA ifdefs for common kmalloc functions. Signed-off-by: Hyeonggon Yoo <42.hyeyoo@gmail.com> Reviewed-by: Vlastimil Babka --- include/linux/slab.h | 28 ---------------------------- mm/slab.c | 2 -- mm/slob.c | 5 +---- mm/slub.c | 6 ------ 4 files changed, 1 insertion(+), 40 deletions(-) diff --git a/include/linux/slab.h b/include/linux/slab.h index 0fefdf528e0d..4754c834b0e3 100644 --- a/include/linux/slab.h +++ b/include/linux/slab.h @@ -456,38 +456,18 @@ static __always_inline void kfree_bulk(size_t size, void **p) kmem_cache_free_bulk(NULL, size, p); } -#ifdef CONFIG_NUMA void *__kmalloc_node(size_t size, gfp_t flags, int node) __assume_kmalloc_alignment __alloc_size(1); void *kmem_cache_alloc_node(struct kmem_cache *s, gfp_t flags, int node) __assume_slab_alignment __malloc; -#else -static __always_inline __alloc_size(1) void *__kmalloc_node(size_t size, gfp_t flags, int node) -{ - return __kmalloc(size, flags); -} - -static __always_inline void *kmem_cache_alloc_node(struct kmem_cache *s, gfp_t flags, int node) -{ - return kmem_cache_alloc(s, flags); -} -#endif #ifdef CONFIG_TRACING extern void *kmem_cache_alloc_trace(struct kmem_cache *s, gfp_t flags, size_t size) __assume_slab_alignment __alloc_size(3); -#ifdef CONFIG_NUMA extern void *kmem_cache_alloc_node_trace(struct kmem_cache *s, gfp_t gfpflags, int node, size_t size) __assume_slab_alignment __alloc_size(4); -#else -static __always_inline __alloc_size(4) void *kmem_cache_alloc_node_trace(struct kmem_cache *s, - gfp_t gfpflags, int node, size_t size) -{ - return kmem_cache_alloc_trace(s, gfpflags, size); -} -#endif /* CONFIG_NUMA */ #else /* CONFIG_TRACING */ static __always_inline __alloc_size(3) void *kmem_cache_alloc_trace(struct kmem_cache *s, @@ -701,20 +681,12 @@ static inline __alloc_size(1, 2) void *kcalloc_node(size_t n, size_t size, gfp_t } -#ifdef CONFIG_NUMA extern void *__kmalloc_node_track_caller(size_t size, gfp_t flags, int node, unsigned long caller) __alloc_size(1); #define kmalloc_node_track_caller(size, flags, node) \ __kmalloc_node_track_caller(size, flags, node, \ _RET_IP_) -#else /* CONFIG_NUMA */ - -#define kmalloc_node_track_caller(size, flags, node) \ - kmalloc_track_caller(size, flags) - -#endif /* CONFIG_NUMA */ - /* * Shortcuts */ diff --git a/mm/slab.c b/mm/slab.c index 748dd085f38e..0acd65358c83 100644 --- a/mm/slab.c +++ b/mm/slab.c @@ -3535,7 +3535,6 @@ kmem_cache_alloc_trace(struct kmem_cache *cachep, gfp_t flags, size_t size) EXPORT_SYMBOL(kmem_cache_alloc_trace); #endif -#ifdef CONFIG_NUMA /** * kmem_cache_alloc_node - Allocate an object on the specified node * @cachep: The cache to allocate from. @@ -3609,7 +3608,6 @@ void *__kmalloc_node_track_caller(size_t size, gfp_t flags, return __do_kmalloc_node(size, flags, node, caller); } EXPORT_SYMBOL(__kmalloc_node_track_caller); -#endif /* CONFIG_NUMA */ #ifdef CONFIG_PRINTK void __kmem_obj_info(struct kmem_obj_info *kpp, void *object, struct slab *slab) diff --git a/mm/slob.c b/mm/slob.c index 2bd4f476c340..74d850967213 100644 --- a/mm/slob.c +++ b/mm/slob.c @@ -536,14 +536,12 @@ void *__kmalloc_track_caller(size_t size, gfp_t gfp, unsigned long caller) } EXPORT_SYMBOL(__kmalloc_track_caller); -#ifdef CONFIG_NUMA void *__kmalloc_node_track_caller(size_t size, gfp_t gfp, int node, unsigned long caller) { return __do_kmalloc_node(size, gfp, node, caller); } EXPORT_SYMBOL(__kmalloc_node_track_caller); -#endif void kfree(const void *block) { @@ -647,7 +645,7 @@ void *kmem_cache_alloc_lru(struct kmem_cache *cachep, struct list_lru *lru, gfp_ return slob_alloc_node(cachep, flags, NUMA_NO_NODE); } EXPORT_SYMBOL(kmem_cache_alloc_lru); -#ifdef CONFIG_NUMA + void *__kmalloc_node(size_t size, gfp_t gfp, int node) { return __do_kmalloc_node(size, gfp, node, _RET_IP_); @@ -659,7 +657,6 @@ void *kmem_cache_alloc_node(struct kmem_cache *cachep, gfp_t gfp, int node) return slob_alloc_node(cachep, gfp, node); } EXPORT_SYMBOL(kmem_cache_alloc_node); -#endif static void __kmem_cache_free(void *b, int size) { diff --git a/mm/slub.c b/mm/slub.c index 862dbd9af4f5..b29b3c9d3175 100644 --- a/mm/slub.c +++ b/mm/slub.c @@ -3287,7 +3287,6 @@ void *kmem_cache_alloc_trace(struct kmem_cache *s, gfp_t gfpflags, size_t size) EXPORT_SYMBOL(kmem_cache_alloc_trace); #endif -#ifdef CONFIG_NUMA void *kmem_cache_alloc_node(struct kmem_cache *s, gfp_t gfpflags, int node) { void *ret = slab_alloc_node(s, NULL, gfpflags, node, _RET_IP_, s->object_size); @@ -3314,7 +3313,6 @@ void *kmem_cache_alloc_node_trace(struct kmem_cache *s, } EXPORT_SYMBOL(kmem_cache_alloc_node_trace); #endif -#endif /* CONFIG_NUMA */ /* * Slow path handling. This may still be called frequently since objects @@ -4427,7 +4425,6 @@ void *__kmalloc(size_t size, gfp_t flags) } EXPORT_SYMBOL(__kmalloc); -#ifdef CONFIG_NUMA static void *kmalloc_large_node(size_t size, gfp_t flags, int node) { struct page *page; @@ -4474,7 +4471,6 @@ void *__kmalloc_node(size_t size, gfp_t flags, int node) return ret; } EXPORT_SYMBOL(__kmalloc_node); -#endif /* CONFIG_NUMA */ #ifdef CONFIG_HARDENED_USERCOPY /* @@ -4930,7 +4926,6 @@ void *__kmalloc_track_caller(size_t size, gfp_t gfpflags, unsigned long caller) } EXPORT_SYMBOL(__kmalloc_track_caller); -#ifdef CONFIG_NUMA void *__kmalloc_node_track_caller(size_t size, gfp_t gfpflags, int node, unsigned long caller) { @@ -4960,7 +4955,6 @@ void *__kmalloc_node_track_caller(size_t size, gfp_t gfpflags, return ret; } EXPORT_SYMBOL(__kmalloc_node_track_caller); -#endif #ifdef CONFIG_SYSFS static int count_inuse(struct slab *slab) From patchwork Wed Aug 17 10:18:13 2022 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Hyeonggon Yoo <42.hyeyoo@gmail.com> X-Patchwork-Id: 12945683 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from kanga.kvack.org (kanga.kvack.org [205.233.56.17]) by smtp.lore.kernel.org (Postfix) with ESMTP id D1F88C25B08 for ; Wed, 17 Aug 2022 10:19:02 +0000 (UTC) Received: by kanga.kvack.org (Postfix) id 6CF2B6B007B; Wed, 17 Aug 2022 06:19:02 -0400 (EDT) Received: by kanga.kvack.org (Postfix, from userid 40) id 67F386B007D; Wed, 17 Aug 2022 06:19:02 -0400 (EDT) X-Delivered-To: int-list-linux-mm@kvack.org Received: by kanga.kvack.org (Postfix, from userid 63042) id 4F9186B007E; Wed, 17 Aug 2022 06:19:02 -0400 (EDT) X-Delivered-To: linux-mm@kvack.org Received: from relay.hostedemail.com (smtprelay0017.hostedemail.com [216.40.44.17]) by kanga.kvack.org (Postfix) with ESMTP id 404146B007B for ; Wed, 17 Aug 2022 06:19:02 -0400 (EDT) Received: from smtpin01.hostedemail.com (a10.router.float.18 [10.200.18.1]) by unirelay01.hostedemail.com (Postfix) with ESMTP id 1B73D1C3EED for ; Wed, 17 Aug 2022 10:19:02 +0000 (UTC) X-FDA: 79808686524.01.F78A782 Received: from mail-pj1-f53.google.com (mail-pj1-f53.google.com [209.85.216.53]) by imf10.hostedemail.com (Postfix) with ESMTP id BC4ECC01D2 for ; Wed, 17 Aug 2022 10:19:01 +0000 (UTC) Received: by mail-pj1-f53.google.com with SMTP id m10-20020a17090a730a00b001fa986fd8eeso1437221pjk.0 for ; Wed, 17 Aug 2022 03:19:01 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=20210112; h=content-transfer-encoding:mime-version:references:in-reply-to :message-id:date:subject:cc:to:from:from:to:cc; bh=Ga89EBm/M0pSIGR8VO7j68eF/ZO7QJRQA6L4btLKSFg=; b=QvUbGhywEPeBhwJmjbmSzIFybywy0vyCr2SJpbyRtQyIAz/VHy0E9scAmI4tj9TDt1 7CCk4lbN/5d6jWw7cwcGb8PBl3ye5q15nhrzA5sqxvDUGMSFqVg7NYHiX3XBRAni81eB auRwaRmIuJf7ov23ZgTvUnKbzv3DtzTYDWOO4ADxnf805wypIVs5TPMc3HWNxAfQge4u Tg6TyCAaDMi3EWtSGO6vJy/T3OsHv9bFnT5fSSzBnJsGjmYouddxdZ11fsxvo7fUAqG1 9FcIao+/tb8ciwjz9SJKcMtwT9GpZ1eH3PJdTUfkmGedsuJqSgTl3kHnOfOaXBbqmEkk cGOg== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20210112; h=content-transfer-encoding:mime-version:references:in-reply-to :message-id:date:subject:cc:to:from:x-gm-message-state:from:to:cc; bh=Ga89EBm/M0pSIGR8VO7j68eF/ZO7QJRQA6L4btLKSFg=; b=TTQTrE0voTC5Q2jPww98fFf0RtYB/f/o3o0O9RakBlCNs9DJhe/w2MI5JSE3KEWN8S z+xP6M2rVMVTutSeWSLKXZupIUx7x8xoA+NKdemare/gG4rCONsDlNmEDi4bodEzPOA8 XkZAu0us59l0MI7041Bcg7Gg1nXmYfa5TUVWKqYkiUfDgimzQzaRrKSnFmloaitdfMlP D9j8X+lCGHCPYGbQd0ZyPD3IgFti5aFO1j6dBgKHrDqB84FnUOL1A5hsrKU1ej+D3d6E WV5YpOyfsgRtuwbscCZ2QhF1VF5Qx3ocOwmT+8uyP04O3A8LlwfPqEgILOMb0hGiLyDD SRSA== X-Gm-Message-State: ACgBeo2DRCC6P3CVtcP+WbYVamw/sl5qtfshPYpbRzzm4IsY7hRb7a1f tb5FKMxnecxdtnWSNHLHxRw= X-Google-Smtp-Source: AA6agR7bPV9zEhLcxPDQaHMgtadEN07BrNJzVtA2QjN0NjCwWz3hrowIypljfqszymfPovsGjThBlg== X-Received: by 2002:a17:902:9307:b0:170:d739:a3cf with SMTP id bc7-20020a170902930700b00170d739a3cfmr26025458plb.127.1660731540819; Wed, 17 Aug 2022 03:19:00 -0700 (PDT) Received: from hyeyoo.. ([114.29.91.56]) by smtp.gmail.com with ESMTPSA id d8-20020a170903230800b00172633fc236sm1071318plh.174.2022.08.17.03.18.57 (version=TLS1_3 cipher=TLS_AES_256_GCM_SHA384 bits=256/256); Wed, 17 Aug 2022 03:18:59 -0700 (PDT) From: Hyeonggon Yoo <42.hyeyoo@gmail.com> To: Christoph Lameter , Pekka Enberg , David Rientjes , Joonsoo Kim , Andrew Morton , Vlastimil Babka , Roman Gushchin Cc: Hyeonggon Yoo <42.hyeyoo@gmail.com>, linux-mm@kvack.org, linux-kernel@vger.kernel.org Subject: [PATCH v4 04/17] mm/slab_common: cleanup kmalloc_track_caller() Date: Wed, 17 Aug 2022 19:18:13 +0900 Message-Id: <20220817101826.236819-5-42.hyeyoo@gmail.com> X-Mailer: git-send-email 2.32.0 In-Reply-To: <20220817101826.236819-1-42.hyeyoo@gmail.com> References: <20220817101826.236819-1-42.hyeyoo@gmail.com> MIME-Version: 1.0 ARC-Seal: i=1; s=arc-20220608; d=hostedemail.com; t=1660731541; a=rsa-sha256; cv=none; b=YiGUnY11k02fZ+FPEJjaLtez84dGOv83thCwC3/O1M+OUBIBXKFJglxUA8VWxqybm0hj9/ c9ZRo4nL+1dj8TyXccDMEMse40ATYmP1dmhMI+rRChhzMqdmToky0/IElil4Aaogg9C32L UE+/IfvLTYcGsA11P6E77Xc2vlJGmFI= ARC-Authentication-Results: i=1; imf10.hostedemail.com; dkim=pass header.d=gmail.com header.s=20210112 header.b=QvUbGhyw; spf=pass (imf10.hostedemail.com: domain of 42.hyeyoo@gmail.com designates 209.85.216.53 as permitted sender) smtp.mailfrom=42.hyeyoo@gmail.com; dmarc=pass (policy=none) header.from=gmail.com ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=hostedemail.com; s=arc-20220608; t=1660731541; h=from:from:sender:reply-to:subject:subject:date:date: message-id:message-id:to:to:cc:cc:mime-version:mime-version: content-type:content-transfer-encoding:content-transfer-encoding: in-reply-to:in-reply-to:references:references:dkim-signature; bh=Ga89EBm/M0pSIGR8VO7j68eF/ZO7QJRQA6L4btLKSFg=; b=mowV7PpYQayRwBmHIcKV7wvm/s0N1Q1yS166Y4a+kvN2VZKC4MfF3CRdow8N8sGTEI0SXN F8E2phGZhWXwcQb/wuY1WfMwUlBl+4j6/K0c1Zz3JvDg4fuj/sKRg/Rt3U89yrHR1t7O7K H184kSlrc3ab953EAXyQdJXo0tODQsI= Authentication-Results: imf10.hostedemail.com; dkim=pass header.d=gmail.com header.s=20210112 header.b=QvUbGhyw; spf=pass (imf10.hostedemail.com: domain of 42.hyeyoo@gmail.com designates 209.85.216.53 as permitted sender) smtp.mailfrom=42.hyeyoo@gmail.com; dmarc=pass (policy=none) header.from=gmail.com X-Rspam-User: X-Stat-Signature: 648jt1twd183st86xymwfgfx1ikoex6b X-Rspamd-Queue-Id: BC4ECC01D2 X-Rspamd-Server: rspam06 X-HE-Tag: 1660731541-161989 X-Bogosity: Ham, tests=bogofilter, spamicity=0.000000, version=1.2.4 Sender: owner-linux-mm@kvack.org Precedence: bulk X-Loop: owner-majordomo@kvack.org List-ID: Make kmalloc_track_caller() wrapper of kmalloc_node_track_caller(). Signed-off-by: Hyeonggon Yoo <42.hyeyoo@gmail.com> Reviewed-by: Vlastimil Babka --- include/linux/slab.h | 17 ++++++++--------- mm/slab.c | 6 ------ mm/slob.c | 6 ------ mm/slub.c | 22 ---------------------- 4 files changed, 8 insertions(+), 43 deletions(-) diff --git a/include/linux/slab.h b/include/linux/slab.h index 4754c834b0e3..a0e57df3d5a4 100644 --- a/include/linux/slab.h +++ b/include/linux/slab.h @@ -651,6 +651,12 @@ static inline __alloc_size(1, 2) void *kcalloc(size_t n, size_t size, gfp_t flag return kmalloc_array(n, size, flags | __GFP_ZERO); } +void *__kmalloc_node_track_caller(size_t size, gfp_t flags, int node, + unsigned long caller) __alloc_size(1); +#define kmalloc_node_track_caller(size, flags, node) \ + __kmalloc_node_track_caller(size, flags, node, \ + _RET_IP_) + /* * kmalloc_track_caller is a special version of kmalloc that records the * calling function of the routine calling it for slab leak tracking instead @@ -659,9 +665,9 @@ static inline __alloc_size(1, 2) void *kcalloc(size_t n, size_t size, gfp_t flag * allocator where we care about the real place the memory allocation * request comes from. */ -extern void *__kmalloc_track_caller(size_t size, gfp_t flags, unsigned long caller); #define kmalloc_track_caller(size, flags) \ - __kmalloc_track_caller(size, flags, _RET_IP_) + __kmalloc_node_track_caller(size, flags, \ + NUMA_NO_NODE, _RET_IP_) static inline __alloc_size(1, 2) void *kmalloc_array_node(size_t n, size_t size, gfp_t flags, int node) @@ -680,13 +686,6 @@ static inline __alloc_size(1, 2) void *kcalloc_node(size_t n, size_t size, gfp_t return kmalloc_array_node(n, size, flags | __GFP_ZERO, node); } - -extern void *__kmalloc_node_track_caller(size_t size, gfp_t flags, int node, - unsigned long caller) __alloc_size(1); -#define kmalloc_node_track_caller(size, flags, node) \ - __kmalloc_node_track_caller(size, flags, node, \ - _RET_IP_) - /* * Shortcuts */ diff --git a/mm/slab.c b/mm/slab.c index 0acd65358c83..611e630ff860 100644 --- a/mm/slab.c +++ b/mm/slab.c @@ -3665,12 +3665,6 @@ void *__kmalloc(size_t size, gfp_t flags) } EXPORT_SYMBOL(__kmalloc); -void *__kmalloc_track_caller(size_t size, gfp_t flags, unsigned long caller) -{ - return __do_kmalloc(size, flags, caller); -} -EXPORT_SYMBOL(__kmalloc_track_caller); - /** * kmem_cache_free - Deallocate an object * @cachep: The cache the allocation was from. diff --git a/mm/slob.c b/mm/slob.c index 74d850967213..96b08acd72ce 100644 --- a/mm/slob.c +++ b/mm/slob.c @@ -530,12 +530,6 @@ void *__kmalloc(size_t size, gfp_t gfp) } EXPORT_SYMBOL(__kmalloc); -void *__kmalloc_track_caller(size_t size, gfp_t gfp, unsigned long caller) -{ - return __do_kmalloc_node(size, gfp, NUMA_NO_NODE, caller); -} -EXPORT_SYMBOL(__kmalloc_track_caller); - void *__kmalloc_node_track_caller(size_t size, gfp_t gfp, int node, unsigned long caller) { diff --git a/mm/slub.c b/mm/slub.c index b29b3c9d3175..c82a4062f730 100644 --- a/mm/slub.c +++ b/mm/slub.c @@ -4904,28 +4904,6 @@ int __kmem_cache_create(struct kmem_cache *s, slab_flags_t flags) return 0; } -void *__kmalloc_track_caller(size_t size, gfp_t gfpflags, unsigned long caller) -{ - struct kmem_cache *s; - void *ret; - - if (unlikely(size > KMALLOC_MAX_CACHE_SIZE)) - return kmalloc_large(size, gfpflags); - - s = kmalloc_slab(size, gfpflags); - - if (unlikely(ZERO_OR_NULL_PTR(s))) - return s; - - ret = slab_alloc(s, NULL, gfpflags, caller, size); - - /* Honor the call site pointer we received. */ - trace_kmalloc(caller, ret, s, size, s->size, gfpflags); - - return ret; -} -EXPORT_SYMBOL(__kmalloc_track_caller); - void *__kmalloc_node_track_caller(size_t size, gfp_t gfpflags, int node, unsigned long caller) { From patchwork Wed Aug 17 10:18:14 2022 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Hyeonggon Yoo <42.hyeyoo@gmail.com> X-Patchwork-Id: 12945684 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from kanga.kvack.org (kanga.kvack.org [205.233.56.17]) by smtp.lore.kernel.org (Postfix) with ESMTP id C5178C25B08 for ; Wed, 17 Aug 2022 10:19:06 +0000 (UTC) Received: by kanga.kvack.org (Postfix) id 5E4B16B007D; Wed, 17 Aug 2022 06:19:06 -0400 (EDT) Received: by kanga.kvack.org (Postfix, from userid 40) id 5947D6B007E; Wed, 17 Aug 2022 06:19:06 -0400 (EDT) X-Delivered-To: int-list-linux-mm@kvack.org Received: by kanga.kvack.org (Postfix, from userid 63042) id 435E36B0080; Wed, 17 Aug 2022 06:19:06 -0400 (EDT) X-Delivered-To: linux-mm@kvack.org Received: from relay.hostedemail.com (smtprelay0017.hostedemail.com [216.40.44.17]) by kanga.kvack.org (Postfix) with ESMTP id 3530D6B007D for ; Wed, 17 Aug 2022 06:19:06 -0400 (EDT) Received: from smtpin05.hostedemail.com (a10.router.float.18 [10.200.18.1]) by unirelay05.hostedemail.com (Postfix) with ESMTP id 1871A408B2 for ; Wed, 17 Aug 2022 10:19:06 +0000 (UTC) X-FDA: 79808686692.05.E625691 Received: from mail-pl1-f169.google.com (mail-pl1-f169.google.com [209.85.214.169]) by imf20.hostedemail.com (Postfix) with ESMTP id 722891C01B0 for ; Wed, 17 Aug 2022 10:19:05 +0000 (UTC) Received: by mail-pl1-f169.google.com with SMTP id jm11so9402732plb.13 for ; Wed, 17 Aug 2022 03:19:05 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=20210112; h=content-transfer-encoding:mime-version:references:in-reply-to :message-id:date:subject:cc:to:from:from:to:cc; bh=Mkyu1siQR2SpurMp6n+bbzTxyYWe6qfMz+gCu1SCQOw=; b=X2p/rtQnRcbuMgMv2/g/yUiUfAVdUulEF3feqNKVrhUZdVHitdHgnz7ivcfoIx3igS 9Z7pzrer2u1uWO9TpTToV0JMowwrebRH/hn+zxBes/WTZLmiFHlUGg7OV0x5mQF4pjhB iS/yUR/5G8ydpytcdtAmTFV+6le+Id6p15CCQLrcCZWXGI3QhrohoOLenisPn6QjWiGy JzoqVv0xOpVNZBCBu0TEVHqg1wrUi1Oik2yLS5IoO0sHdi9QJXDhyqQtgZRsySjh/8K9 9Q4DS66jOw0tidDJfbl5N9ohtrw9R9K/zkSMZ1hFq3Br1VS0pgECuG97fzQC//gJ1w4s qyAg== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20210112; h=content-transfer-encoding:mime-version:references:in-reply-to :message-id:date:subject:cc:to:from:x-gm-message-state:from:to:cc; bh=Mkyu1siQR2SpurMp6n+bbzTxyYWe6qfMz+gCu1SCQOw=; b=HGnFWMx8J0A4ZzPsdzLgUMUP6GZsxrTpVj4P83f0OPTGeHeOr2kxuppFHMhAOev21I gBG9jcPMPqMmUBly/FxZy9ATDRxfZSQwYPDMlsDo8RqNMHz8y57K7Hakx11NNhyfj766 U0HG73xviR9Xf/vWAvVSxFYljnGlNzkwuf9OdKKJ1VXjtFJu6MmN+js18c+Z7Um5cMt4 HPCai0dkwpPb80lgl8sO0fJft1rrWCqi3dLj6dFQ7I85uxNd5UOJewAEvioxL6yCXHrn IzI9gs+428dsbP/m4htgzzYf+pHImFy+UxJmG8DnfVOoR9NOcSUh7bilHY0l52uLPOcL BWKA== X-Gm-Message-State: ACgBeo1nrlkbUJY+15ao8US0eJ3z1ejXt4SI+jtVy/WkbB40W4l7SzEz xZVGqG7Xf5+ij53T6ZsAWyu1oM3UDCw= X-Google-Smtp-Source: AA6agR6pvX9vtBet0UY+NHp3e4BoO2UXGfR8fh/YGIkfc65JKCmnJtMLBxUCK0a3mVeWGiCp0plydA== X-Received: by 2002:a17:902:7883:b0:170:d646:7f00 with SMTP id q3-20020a170902788300b00170d6467f00mr25919187pll.164.1660731544517; Wed, 17 Aug 2022 03:19:04 -0700 (PDT) Received: from hyeyoo.. ([114.29.91.56]) by smtp.gmail.com with ESMTPSA id d8-20020a170903230800b00172633fc236sm1071318plh.174.2022.08.17.03.19.01 (version=TLS1_3 cipher=TLS_AES_256_GCM_SHA384 bits=256/256); Wed, 17 Aug 2022 03:19:03 -0700 (PDT) From: Hyeonggon Yoo <42.hyeyoo@gmail.com> To: Christoph Lameter , Pekka Enberg , David Rientjes , Joonsoo Kim , Andrew Morton , Vlastimil Babka , Roman Gushchin Cc: Hyeonggon Yoo <42.hyeyoo@gmail.com>, linux-mm@kvack.org, linux-kernel@vger.kernel.org Subject: [PATCH v4 05/17] mm/sl[au]b: factor out __do_kmalloc_node() Date: Wed, 17 Aug 2022 19:18:14 +0900 Message-Id: <20220817101826.236819-6-42.hyeyoo@gmail.com> X-Mailer: git-send-email 2.32.0 In-Reply-To: <20220817101826.236819-1-42.hyeyoo@gmail.com> References: <20220817101826.236819-1-42.hyeyoo@gmail.com> MIME-Version: 1.0 ARC-Seal: i=1; s=arc-20220608; d=hostedemail.com; t=1660731545; a=rsa-sha256; cv=none; b=SY3bvSACMv1zUq2EcfkL220QsttVhTFhqpu50chCHZq6d83rvzhkZk4OU8k4Ypl9tTQc1A VeKteqn8X4cTPdsvcWsc+6UzQ0wkfQRbEA4B3mWHXwTkxmzG9MjNBqYar2wH1f8HhUflMu wZVIJcLwbQkd8gNFOBzQcTSnmPB80sU= ARC-Authentication-Results: i=1; imf20.hostedemail.com; dkim=pass header.d=gmail.com header.s=20210112 header.b="X2p/rtQn"; spf=pass (imf20.hostedemail.com: domain of 42.hyeyoo@gmail.com designates 209.85.214.169 as permitted sender) smtp.mailfrom=42.hyeyoo@gmail.com; dmarc=pass (policy=none) header.from=gmail.com ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=hostedemail.com; s=arc-20220608; t=1660731545; h=from:from:sender:reply-to:subject:subject:date:date: message-id:message-id:to:to:cc:cc:mime-version:mime-version: content-type:content-transfer-encoding:content-transfer-encoding: in-reply-to:in-reply-to:references:references:dkim-signature; bh=Mkyu1siQR2SpurMp6n+bbzTxyYWe6qfMz+gCu1SCQOw=; b=PnL+lHFlNCaz4ZigaPSw3cSF9kMolIaNwBr6ebqEiyYx0OL5h22Zh9HOJyh6kh+fZFzPc+ 0E7QkkwElvtin4J5z/Tiu2imcdNxnQdqHDFOW4YxGWPr+B4NwlqLXUB+Fx7v1qFINs3efH OwrJGV1ooekt6mWjjw7pzMSXR3almeA= Authentication-Results: imf20.hostedemail.com; dkim=pass header.d=gmail.com header.s=20210112 header.b="X2p/rtQn"; spf=pass (imf20.hostedemail.com: domain of 42.hyeyoo@gmail.com designates 209.85.214.169 as permitted sender) smtp.mailfrom=42.hyeyoo@gmail.com; dmarc=pass (policy=none) header.from=gmail.com X-Rspamd-Server: rspam05 X-Rspam-User: X-Stat-Signature: d69f7a44xxztaeg1qabyuqroesqrixmf X-Rspamd-Queue-Id: 722891C01B0 X-HE-Tag: 1660731545-421777 X-Bogosity: Ham, tests=bogofilter, spamicity=0.000000, version=1.2.4 Sender: owner-linux-mm@kvack.org Precedence: bulk X-Loop: owner-majordomo@kvack.org List-ID: __kmalloc(), __kmalloc_node(), __kmalloc_node_track_caller() mostly do same job. Factor out common code into __do_kmalloc_node(). Note that this patch also fixes missing kasan_kmalloc() in SLUB's __kmalloc_node_track_caller(). Signed-off-by: Hyeonggon Yoo <42.hyeyoo@gmail.com> Reviewed-by: Vlastimil Babka --- mm/slab.c | 30 +---------------------- mm/slub.c | 71 +++++++++++++++---------------------------------------- 2 files changed, 20 insertions(+), 81 deletions(-) diff --git a/mm/slab.c b/mm/slab.c index 611e630ff860..8c08d7f3dead 100644 --- a/mm/slab.c +++ b/mm/slab.c @@ -3631,37 +3631,9 @@ void __kmem_obj_info(struct kmem_obj_info *kpp, void *object, struct slab *slab) } #endif -/** - * __do_kmalloc - allocate memory - * @size: how many bytes of memory are required. - * @flags: the type of memory to allocate (see kmalloc). - * @caller: function caller for debug tracking of the caller - * - * Return: pointer to the allocated memory or %NULL in case of error - */ -static __always_inline void *__do_kmalloc(size_t size, gfp_t flags, - unsigned long caller) -{ - struct kmem_cache *cachep; - void *ret; - - if (unlikely(size > KMALLOC_MAX_CACHE_SIZE)) - return NULL; - cachep = kmalloc_slab(size, flags); - if (unlikely(ZERO_OR_NULL_PTR(cachep))) - return cachep; - ret = slab_alloc(cachep, NULL, flags, size, caller); - - ret = kasan_kmalloc(cachep, ret, size, flags); - trace_kmalloc(caller, ret, cachep, - size, cachep->size, flags); - - return ret; -} - void *__kmalloc(size_t size, gfp_t flags) { - return __do_kmalloc(size, flags, _RET_IP_); + return __do_kmalloc_node(size, flags, NUMA_NO_NODE, _RET_IP_); } EXPORT_SYMBOL(__kmalloc); diff --git a/mm/slub.c b/mm/slub.c index c82a4062f730..f9929ba858ec 100644 --- a/mm/slub.c +++ b/mm/slub.c @@ -4402,29 +4402,6 @@ static int __init setup_slub_min_objects(char *str) __setup("slub_min_objects=", setup_slub_min_objects); -void *__kmalloc(size_t size, gfp_t flags) -{ - struct kmem_cache *s; - void *ret; - - if (unlikely(size > KMALLOC_MAX_CACHE_SIZE)) - return kmalloc_large(size, flags); - - s = kmalloc_slab(size, flags); - - if (unlikely(ZERO_OR_NULL_PTR(s))) - return s; - - ret = slab_alloc(s, NULL, flags, _RET_IP_, size); - - trace_kmalloc(_RET_IP_, ret, s, size, s->size, flags); - - ret = kasan_kmalloc(s, ret, size, flags); - - return ret; -} -EXPORT_SYMBOL(__kmalloc); - static void *kmalloc_large_node(size_t size, gfp_t flags, int node) { struct page *page; @@ -4442,7 +4419,8 @@ static void *kmalloc_large_node(size_t size, gfp_t flags, int node) return kmalloc_large_node_hook(ptr, size, flags); } -void *__kmalloc_node(size_t size, gfp_t flags, int node) +static __always_inline +void *__do_kmalloc_node(size_t size, gfp_t flags, int node, unsigned long caller) { struct kmem_cache *s; void *ret; @@ -4450,7 +4428,7 @@ void *__kmalloc_node(size_t size, gfp_t flags, int node) if (unlikely(size > KMALLOC_MAX_CACHE_SIZE)) { ret = kmalloc_large_node(size, flags, node); - trace_kmalloc_node(_RET_IP_, ret, NULL, + trace_kmalloc_node(caller, ret, NULL, size, PAGE_SIZE << get_order(size), flags, node); @@ -4462,16 +4440,28 @@ void *__kmalloc_node(size_t size, gfp_t flags, int node) if (unlikely(ZERO_OR_NULL_PTR(s))) return s; - ret = slab_alloc_node(s, NULL, flags, node, _RET_IP_, size); + ret = slab_alloc_node(s, NULL, flags, node, caller, size); - trace_kmalloc_node(_RET_IP_, ret, s, size, s->size, flags, node); + trace_kmalloc_node(caller, ret, s, size, s->size, flags, node); ret = kasan_kmalloc(s, ret, size, flags); return ret; } + +void *__kmalloc_node(size_t size, gfp_t flags, int node) +{ + return __do_kmalloc_node(size, flags, node, _RET_IP_); +} EXPORT_SYMBOL(__kmalloc_node); +void *__kmalloc(size_t size, gfp_t flags) +{ + return __do_kmalloc_node(size, flags, NUMA_NO_NODE, _RET_IP_); +} +EXPORT_SYMBOL(__kmalloc); + + #ifdef CONFIG_HARDENED_USERCOPY /* * Rejects incorrectly sized objects and objects that are to be copied @@ -4905,32 +4895,9 @@ int __kmem_cache_create(struct kmem_cache *s, slab_flags_t flags) } void *__kmalloc_node_track_caller(size_t size, gfp_t gfpflags, - int node, unsigned long caller) + int node, unsigned long caller) { - struct kmem_cache *s; - void *ret; - - if (unlikely(size > KMALLOC_MAX_CACHE_SIZE)) { - ret = kmalloc_large_node(size, gfpflags, node); - - trace_kmalloc_node(caller, ret, NULL, - size, PAGE_SIZE << get_order(size), - gfpflags, node); - - return ret; - } - - s = kmalloc_slab(size, gfpflags); - - if (unlikely(ZERO_OR_NULL_PTR(s))) - return s; - - ret = slab_alloc_node(s, NULL, gfpflags, node, caller, size); - - /* Honor the call site pointer we received. */ - trace_kmalloc_node(caller, ret, s, size, s->size, gfpflags, node); - - return ret; + return __do_kmalloc_node(size, gfpflags, node, caller); } EXPORT_SYMBOL(__kmalloc_node_track_caller); From patchwork Wed Aug 17 10:18:15 2022 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Hyeonggon Yoo <42.hyeyoo@gmail.com> X-Patchwork-Id: 12945685 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from kanga.kvack.org (kanga.kvack.org [205.233.56.17]) by smtp.lore.kernel.org (Postfix) with ESMTP id 3F5C5C25B08 for ; Wed, 17 Aug 2022 10:19:10 +0000 (UTC) Received: by kanga.kvack.org (Postfix) id C9C556B0075; Wed, 17 Aug 2022 06:19:09 -0400 (EDT) Received: by kanga.kvack.org (Postfix, from userid 40) id C4C1D6B007E; Wed, 17 Aug 2022 06:19:09 -0400 (EDT) X-Delivered-To: int-list-linux-mm@kvack.org Received: by kanga.kvack.org (Postfix, from userid 63042) id AECA18D0001; Wed, 17 Aug 2022 06:19:09 -0400 (EDT) X-Delivered-To: linux-mm@kvack.org Received: from relay.hostedemail.com (smtprelay0015.hostedemail.com [216.40.44.15]) by kanga.kvack.org (Postfix) with ESMTP id A08E26B0075 for ; Wed, 17 Aug 2022 06:19:09 -0400 (EDT) Received: from smtpin08.hostedemail.com (a10.router.float.18 [10.200.18.1]) by unirelay02.hostedemail.com (Postfix) with ESMTP id 7240B1208CE for ; Wed, 17 Aug 2022 10:19:09 +0000 (UTC) X-FDA: 79808686818.08.DE6904F Received: from mail-pg1-f178.google.com (mail-pg1-f178.google.com [209.85.215.178]) by imf06.hostedemail.com (Postfix) with ESMTP id 0EBA01801AF for ; Wed, 17 Aug 2022 10:19:08 +0000 (UTC) Received: by mail-pg1-f178.google.com with SMTP id q16so11605226pgq.6 for ; Wed, 17 Aug 2022 03:19:08 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=20210112; h=content-transfer-encoding:mime-version:references:in-reply-to :message-id:date:subject:cc:to:from:from:to:cc; bh=mA3ow7hUqSccT7bNwtvWTEd/QzJvQLSRyOCnJvDyp6U=; b=KSORlivSY1gDZvvmSEL4PxisN5IhBGQ1AeMvUGp8veSEXXByVMSzBjP4EJs747a5pq EofZUC/QfmHd03v73+6exjCC4vKOklIzSLxg0nltRXkCXFS4Sc0OwHmR9KIl6TyAEJrc fHJBGiEwu0cAFe0dzqsjwmqN5xt8ff032WOB4BvTPxYzEfZz7b1pP7lJm08KccLkCQHJ jt569vdr1+djQJL67jwzVOqUDjxqN7XyixPqfbCIXDwsI35VKNdGOl8hek/QXE1J67DP RNLPeV6A5AyUr6bx88K6xfAy8EGoftktHop8hdTCXciIpaf7F43jIYcW+0Qxx8yVYu8U qlTg== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20210112; h=content-transfer-encoding:mime-version:references:in-reply-to :message-id:date:subject:cc:to:from:x-gm-message-state:from:to:cc; bh=mA3ow7hUqSccT7bNwtvWTEd/QzJvQLSRyOCnJvDyp6U=; b=JCULsbI11OaFg1JzESroQbcfiJWwoKYMcAhW+5FSJ7ei2wGWwtd2d3HuzgwOb+cRPZ au6YfMu+Ep0wNZUB9ygSkTQJUo9AYxU95Y3N7z57/TDzB52qTpN2yiYvpHDSd3VOjLDn kFVLXKIJHQx2ChYZRStzsczrTKZSEWBrHM8LAvADmNFLsFolbBdofSyJLURjUp7ZElke 8Fnlv+whjxsnK2/LcZ1bDo34NeVywic3LSKqL3nZA/n/VWncge2uFCO967pUVtJogXD7 h3WqebQom4tou10oYvJQNaN+dIFwmIrdx11pEUnTzWkLXdouQSlbmKYo9Kll1NNynf+W vwEA== X-Gm-Message-State: ACgBeo3FVXyASBIpNBD9HAZnkR9WAfiuvNMyilvA6YIqtHLI7ORb1YML alOblBx0gWcAq4Yex3GSsTHZ19sz9Bw= X-Google-Smtp-Source: AA6agR7yr6u/vBTtWdBbY4Pd43tYRkW3wO6YJeVXJ+XtjSNhD6MsvXydP/mxQVwJjej0KHAHrH/4tQ== X-Received: by 2002:a63:494a:0:b0:41c:f29e:2a2e with SMTP id y10-20020a63494a000000b0041cf29e2a2emr21247258pgk.477.1660731548139; Wed, 17 Aug 2022 03:19:08 -0700 (PDT) Received: from hyeyoo.. ([114.29.91.56]) by smtp.gmail.com with ESMTPSA id d8-20020a170903230800b00172633fc236sm1071318plh.174.2022.08.17.03.19.04 (version=TLS1_3 cipher=TLS_AES_256_GCM_SHA384 bits=256/256); Wed, 17 Aug 2022 03:19:07 -0700 (PDT) From: Hyeonggon Yoo <42.hyeyoo@gmail.com> To: Christoph Lameter , Pekka Enberg , David Rientjes , Joonsoo Kim , Andrew Morton , Vlastimil Babka , Roman Gushchin Cc: Hyeonggon Yoo <42.hyeyoo@gmail.com>, linux-mm@kvack.org, linux-kernel@vger.kernel.org Subject: [PATCH v4 06/17] mm/slab_common: fold kmalloc_order_trace() into kmalloc_large() Date: Wed, 17 Aug 2022 19:18:15 +0900 Message-Id: <20220817101826.236819-7-42.hyeyoo@gmail.com> X-Mailer: git-send-email 2.32.0 In-Reply-To: <20220817101826.236819-1-42.hyeyoo@gmail.com> References: <20220817101826.236819-1-42.hyeyoo@gmail.com> MIME-Version: 1.0 ARC-Authentication-Results: i=1; imf06.hostedemail.com; dkim=pass header.d=gmail.com header.s=20210112 header.b=KSORlivS; spf=pass (imf06.hostedemail.com: domain of 42.hyeyoo@gmail.com designates 209.85.215.178 as permitted sender) smtp.mailfrom=42.hyeyoo@gmail.com; dmarc=pass (policy=none) header.from=gmail.com ARC-Seal: i=1; s=arc-20220608; d=hostedemail.com; t=1660731549; a=rsa-sha256; cv=none; b=Nw86vYCyhm/gOBRytGnVhknAluY1DTbYkRhbp1sCHBGbHaoWrRcSuacJHibQOBkHI6NFjx rBVck3b/VwmTGsMzxqirXbUq5uWYXM+ctKLoMJW2bN6HOvN15QSzSkVzSTchpb5yq5taKR MkPiLp6MqJdTxm6OdqzNYavKfybjTNc= ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=hostedemail.com; s=arc-20220608; t=1660731549; h=from:from:sender:reply-to:subject:subject:date:date: message-id:message-id:to:to:cc:cc:mime-version:mime-version: content-type:content-transfer-encoding:content-transfer-encoding: in-reply-to:in-reply-to:references:references:dkim-signature; bh=mA3ow7hUqSccT7bNwtvWTEd/QzJvQLSRyOCnJvDyp6U=; b=z5QVLoqZfnm7r0FwBgcUycM6D5Lwp3ediTM4rZ7isjr5GzIM3BnwkAjxcJfPr2yyFYGwrY RgUvrY4GAKbGWX33vyYYThOcVUjDt0dpxufeYBXKrrdYGY6E3Fh7wCOUTJeBYYZjk3iOe2 fofl3bhrK5wf+U0TdrLbzaahdANnY7Q= Authentication-Results: imf06.hostedemail.com; dkim=pass header.d=gmail.com header.s=20210112 header.b=KSORlivS; spf=pass (imf06.hostedemail.com: domain of 42.hyeyoo@gmail.com designates 209.85.215.178 as permitted sender) smtp.mailfrom=42.hyeyoo@gmail.com; dmarc=pass (policy=none) header.from=gmail.com X-Stat-Signature: 14y9omp9yf8bhbghbpmt8ua4ttzzuyyt X-Rspamd-Server: rspam09 X-Rspam-User: X-Rspamd-Queue-Id: 0EBA01801AF X-HE-Tag: 1660731548-361636 X-Bogosity: Ham, tests=bogofilter, spamicity=0.000000, version=1.2.4 Sender: owner-linux-mm@kvack.org Precedence: bulk X-Loop: owner-majordomo@kvack.org List-ID: There is no caller of kmalloc_order_trace() except kmalloc_large(). Fold it into kmalloc_large() and remove kmalloc_order{,_trace}(). Also add tracepoint in kmalloc_large() that was previously in kmalloc_order_trace(). Signed-off-by: Hyeonggon Yoo <42.hyeyoo@gmail.com> Reviewed-by: Vlastimil Babka --- include/linux/slab.h | 22 ++-------------------- mm/slab_common.c | 17 ++++------------- 2 files changed, 6 insertions(+), 33 deletions(-) diff --git a/include/linux/slab.h b/include/linux/slab.h index a0e57df3d5a4..15a4c59da59e 100644 --- a/include/linux/slab.h +++ b/include/linux/slab.h @@ -489,26 +489,8 @@ static __always_inline void *kmem_cache_alloc_node_trace(struct kmem_cache *s, g } #endif /* CONFIG_TRACING */ -extern void *kmalloc_order(size_t size, gfp_t flags, unsigned int order) __assume_page_alignment - __alloc_size(1); - -#ifdef CONFIG_TRACING -extern void *kmalloc_order_trace(size_t size, gfp_t flags, unsigned int order) - __assume_page_alignment __alloc_size(1); -#else -static __always_inline __alloc_size(1) void *kmalloc_order_trace(size_t size, gfp_t flags, - unsigned int order) -{ - return kmalloc_order(size, flags, order); -} -#endif - -static __always_inline __alloc_size(1) void *kmalloc_large(size_t size, gfp_t flags) -{ - unsigned int order = get_order(size); - return kmalloc_order_trace(size, flags, order); -} - +void *kmalloc_large(size_t size, gfp_t flags) __assume_page_alignment + __alloc_size(1); /** * kmalloc - allocate memory * @size: how many bytes of memory are required. diff --git a/mm/slab_common.c b/mm/slab_common.c index 17996649cfe3..8b1988544b89 100644 --- a/mm/slab_common.c +++ b/mm/slab_common.c @@ -905,16 +905,16 @@ gfp_t kmalloc_fix_flags(gfp_t flags) * directly to the page allocator. We use __GFP_COMP, because we will need to * know the allocation order to free the pages properly in kfree. */ -void *kmalloc_order(size_t size, gfp_t flags, unsigned int order) +void *kmalloc_large(size_t size, gfp_t flags) { void *ret = NULL; struct page *page; + unsigned int order = get_order(size); if (unlikely(flags & GFP_SLAB_BUG_MASK)) flags = kmalloc_fix_flags(flags); - flags |= __GFP_COMP; - page = alloc_pages(flags, order); + page = alloc_pages(flags | __GFP_COMP, order); if (likely(page)) { ret = page_address(page); mod_lruvec_page_state(page, NR_SLAB_UNRECLAIMABLE_B, @@ -923,19 +923,10 @@ void *kmalloc_order(size_t size, gfp_t flags, unsigned int order) ret = kasan_kmalloc_large(ret, size, flags); /* As ret might get tagged, call kmemleak hook after KASAN. */ kmemleak_alloc(ret, size, 1, flags); - return ret; -} -EXPORT_SYMBOL(kmalloc_order); - -#ifdef CONFIG_TRACING -void *kmalloc_order_trace(size_t size, gfp_t flags, unsigned int order) -{ - void *ret = kmalloc_order(size, flags, order); trace_kmalloc(_RET_IP_, ret, NULL, size, PAGE_SIZE << order, flags); return ret; } -EXPORT_SYMBOL(kmalloc_order_trace); -#endif +EXPORT_SYMBOL(kmalloc_large); #ifdef CONFIG_SLAB_FREELIST_RANDOM /* Randomize a generic freelist */ From patchwork Wed Aug 17 10:18:16 2022 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Hyeonggon Yoo <42.hyeyoo@gmail.com> X-Patchwork-Id: 12945686 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from kanga.kvack.org (kanga.kvack.org [205.233.56.17]) by smtp.lore.kernel.org (Postfix) with ESMTP id 0C300C25B08 for ; Wed, 17 Aug 2022 10:19:14 +0000 (UTC) Received: by kanga.kvack.org (Postfix) id 967BA6B007E; Wed, 17 Aug 2022 06:19:13 -0400 (EDT) Received: by kanga.kvack.org (Postfix, from userid 40) id 918296B0080; Wed, 17 Aug 2022 06:19:13 -0400 (EDT) X-Delivered-To: int-list-linux-mm@kvack.org Received: by kanga.kvack.org (Postfix, from userid 63042) id 791558D0001; Wed, 17 Aug 2022 06:19:13 -0400 (EDT) X-Delivered-To: linux-mm@kvack.org Received: from relay.hostedemail.com (smtprelay0010.hostedemail.com [216.40.44.10]) by kanga.kvack.org (Postfix) with ESMTP id 6C1746B007E for ; Wed, 17 Aug 2022 06:19:13 -0400 (EDT) Received: from smtpin14.hostedemail.com (a10.router.float.18 [10.200.18.1]) by unirelay03.hostedemail.com (Postfix) with ESMTP id 39EFFA0871 for ; Wed, 17 Aug 2022 10:19:13 +0000 (UTC) X-FDA: 79808686986.14.02B84D6 Received: from mail-pg1-f173.google.com (mail-pg1-f173.google.com [209.85.215.173]) by imf06.hostedemail.com (Postfix) with ESMTP id D05EC1801AF for ; Wed, 17 Aug 2022 10:19:12 +0000 (UTC) Received: by mail-pg1-f173.google.com with SMTP id 73so11601430pgb.9 for ; Wed, 17 Aug 2022 03:19:12 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=20210112; h=content-transfer-encoding:mime-version:references:in-reply-to :message-id:date:subject:cc:to:from:from:to:cc; bh=V13eBLL+I9iGduc+JnUgsCOIMqteurACvuOFaESMVck=; b=npKhTo3ktjcY8GrNZPcQJecpqhpmyjY86O+1b2c9YapnvSiuEYHiYe5hnCn/sUP0QT qNkoC7kX+tJi8lPw28MEQm733wFAvgSGKZHtoE1lhSMoCOdt3XXcAE+Mv7OAcQdFAvX+ HhH7RPa6voDdqpdjNP0u52d2Qik1vWrHYJBtcDSAmM+IC8xz9TstXJaxtT7fK2MNa8tY a+XUbmdJlWpmSC7QsgIfWYyubzXveHLy0AUl6kjCLyHahj3m/nD1qZPHeIe6fNarqGw7 QA4ZQNVBFjw1Ozp+l5pBhj89GteMBMwljxJ7dQawuvf0VZQXj9hzJhJRJy8QSr8Qix4u YK8A== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20210112; h=content-transfer-encoding:mime-version:references:in-reply-to :message-id:date:subject:cc:to:from:x-gm-message-state:from:to:cc; bh=V13eBLL+I9iGduc+JnUgsCOIMqteurACvuOFaESMVck=; b=Zvl2ne0eAHvnWWGrRG3Ta3rPaF0W0OguCsiliH07FEzRmOhcOoZElCeQyyS5GcyD/t qIxMcI/OvrkTfBLEECrnPHUNKuHbSRJ7ceIrDRgOdDrDjME8vPO30LTjRVaUKrFsabsg Py7FrySLeoIgCbj2i1W7LNYdbT6DDGUeiQrdaOg7cntkwmgk6TtWd36hgVMuqrsSgbA0 7+Pe2PYyz0VNp+qbsmKv/uL/tpSXydlY5qTME+QIEU6V4whK6ItBPw0kaDCcRGz63cBo w8YYzRf605gE0XC9E1HTlxXfk/5/TGNGcjd2Ipq8qNsUlab+y5I7gnvH/X1nT7R64r4q X5EA== X-Gm-Message-State: ACgBeo05587Yjr405AnLy1b1lhMHCQ0J8WKwqGyHCGBSQ/hE7dQ368xM 8hvqadmBm6P9yoI2vji4MxE= X-Google-Smtp-Source: AA6agR5Ow2fAgG6PDOU6KesG5RJ+Tea/+u3A+H4XdF3ir3eqFdhtzZcFsxZkt3rDAsMg+9NGYF8LDg== X-Received: by 2002:a65:4605:0:b0:41c:3d73:9385 with SMTP id v5-20020a654605000000b0041c3d739385mr20546331pgq.168.1660731551918; Wed, 17 Aug 2022 03:19:11 -0700 (PDT) Received: from hyeyoo.. ([114.29.91.56]) by smtp.gmail.com with ESMTPSA id d8-20020a170903230800b00172633fc236sm1071318plh.174.2022.08.17.03.19.08 (version=TLS1_3 cipher=TLS_AES_256_GCM_SHA384 bits=256/256); Wed, 17 Aug 2022 03:19:11 -0700 (PDT) From: Hyeonggon Yoo <42.hyeyoo@gmail.com> To: Christoph Lameter , Pekka Enberg , David Rientjes , Joonsoo Kim , Andrew Morton , Vlastimil Babka , Roman Gushchin Cc: Hyeonggon Yoo <42.hyeyoo@gmail.com>, linux-mm@kvack.org, linux-kernel@vger.kernel.org Subject: [PATCH v4 07/17] mm/slub: move kmalloc_large_node() to slab_common.c Date: Wed, 17 Aug 2022 19:18:16 +0900 Message-Id: <20220817101826.236819-8-42.hyeyoo@gmail.com> X-Mailer: git-send-email 2.32.0 In-Reply-To: <20220817101826.236819-1-42.hyeyoo@gmail.com> References: <20220817101826.236819-1-42.hyeyoo@gmail.com> MIME-Version: 1.0 ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=hostedemail.com; s=arc-20220608; t=1660731552; h=from:from:sender:reply-to:subject:subject:date:date: message-id:message-id:to:to:cc:cc:mime-version:mime-version: content-type:content-transfer-encoding:content-transfer-encoding: in-reply-to:in-reply-to:references:references:dkim-signature; bh=V13eBLL+I9iGduc+JnUgsCOIMqteurACvuOFaESMVck=; b=oUYn3c4Rv2T6bfERLCki9p+90va57lHK4yL4VsD4zHCU7DoAwh0ErQs+0kMd6S1nc98nzd BJza5yu+wRrJ6fRrViF4ZeNp2FUS/SRsiY28S+AUKUXpyBYBRlSkOiRtdmVeLKL4GvDoF+ kzt6+U/Cdu1jQX7fJhq1ptMHLbn0OKA= ARC-Authentication-Results: i=1; imf06.hostedemail.com; dkim=pass header.d=gmail.com header.s=20210112 header.b=npKhTo3k; spf=pass (imf06.hostedemail.com: domain of 42.hyeyoo@gmail.com designates 209.85.215.173 as permitted sender) smtp.mailfrom=42.hyeyoo@gmail.com; dmarc=pass (policy=none) header.from=gmail.com ARC-Seal: i=1; s=arc-20220608; d=hostedemail.com; t=1660731552; a=rsa-sha256; cv=none; b=4ls7c49wjls0/GQRtyoJuXDl0dvdTWx3gl4pCWasxkkl5RvOEsIpLu9Au4QtENdtI4zPIK hTBN6esuF0UXUGr2BOIiOuInQwcot/C1lwqAVsI2RNL30RAcFQ1g7XTngnh7c/BJaTJbMz wq4yz9+yWsIxYz/Md8foPViZEvVijsU= X-Rspamd-Server: rspam04 X-Rspamd-Queue-Id: D05EC1801AF X-Rspam-User: Authentication-Results: imf06.hostedemail.com; dkim=pass header.d=gmail.com header.s=20210112 header.b=npKhTo3k; spf=pass (imf06.hostedemail.com: domain of 42.hyeyoo@gmail.com designates 209.85.215.173 as permitted sender) smtp.mailfrom=42.hyeyoo@gmail.com; dmarc=pass (policy=none) header.from=gmail.com X-Stat-Signature: qdbg4yohhd57ci9iaecohich1gjxwgpi X-HE-Tag: 1660731552-305834 X-Bogosity: Ham, tests=bogofilter, spamicity=0.000000, version=1.2.4 Sender: owner-linux-mm@kvack.org Precedence: bulk X-Loop: owner-majordomo@kvack.org List-ID: In later patch SLAB will also pass requests larger than order-1 page to page allocator. Move kmalloc_large_node() to slab_common.c. Fold kmalloc_large_node_hook() into kmalloc_large_node() as there is no other caller. Signed-off-by: Hyeonggon Yoo <42.hyeyoo@gmail.com> Reviewed-by: Vlastimil Babka --- include/linux/slab.h | 4 ++++ mm/slab_common.c | 22 ++++++++++++++++++++++ mm/slub.c | 25 ------------------------- 3 files changed, 26 insertions(+), 25 deletions(-) diff --git a/include/linux/slab.h b/include/linux/slab.h index 15a4c59da59e..082499306098 100644 --- a/include/linux/slab.h +++ b/include/linux/slab.h @@ -491,6 +491,10 @@ static __always_inline void *kmem_cache_alloc_node_trace(struct kmem_cache *s, g void *kmalloc_large(size_t size, gfp_t flags) __assume_page_alignment __alloc_size(1); + +void *kmalloc_large_node(size_t size, gfp_t flags, int node) __assume_page_alignment + __alloc_size(1); + /** * kmalloc - allocate memory * @size: how many bytes of memory are required. diff --git a/mm/slab_common.c b/mm/slab_common.c index 8b1988544b89..1b9101f9cb21 100644 --- a/mm/slab_common.c +++ b/mm/slab_common.c @@ -928,6 +928,28 @@ void *kmalloc_large(size_t size, gfp_t flags) } EXPORT_SYMBOL(kmalloc_large); +void *kmalloc_large_node(size_t size, gfp_t flags, int node) +{ + struct page *page; + void *ptr = NULL; + unsigned int order = get_order(size); + + flags |= __GFP_COMP; + page = alloc_pages_node(node, flags, order); + if (page) { + ptr = page_address(page); + mod_lruvec_page_state(page, NR_SLAB_UNRECLAIMABLE_B, + PAGE_SIZE << order); + } + + ptr = kasan_kmalloc_large(ptr, size, flags); + /* As ptr might get tagged, call kmemleak hook after KASAN. */ + kmemleak_alloc(ptr, size, 1, flags); + + return ptr; +} +EXPORT_SYMBOL(kmalloc_large_node); + #ifdef CONFIG_SLAB_FREELIST_RANDOM /* Randomize a generic freelist */ static void freelist_randomize(struct rnd_state *state, unsigned int *list, diff --git a/mm/slub.c b/mm/slub.c index f9929ba858ec..5e7819ade2c4 100644 --- a/mm/slub.c +++ b/mm/slub.c @@ -1704,14 +1704,6 @@ static bool freelist_corrupted(struct kmem_cache *s, struct slab *slab, * Hooks for other subsystems that check memory allocations. In a typical * production configuration these hooks all should produce no code at all. */ -static inline void *kmalloc_large_node_hook(void *ptr, size_t size, gfp_t flags) -{ - ptr = kasan_kmalloc_large(ptr, size, flags); - /* As ptr might get tagged, call kmemleak hook after KASAN. */ - kmemleak_alloc(ptr, size, 1, flags); - return ptr; -} - static __always_inline void kfree_hook(void *x) { kmemleak_free(x); @@ -4402,23 +4394,6 @@ static int __init setup_slub_min_objects(char *str) __setup("slub_min_objects=", setup_slub_min_objects); -static void *kmalloc_large_node(size_t size, gfp_t flags, int node) -{ - struct page *page; - void *ptr = NULL; - unsigned int order = get_order(size); - - flags |= __GFP_COMP; - page = alloc_pages_node(node, flags, order); - if (page) { - ptr = page_address(page); - mod_lruvec_page_state(page, NR_SLAB_UNRECLAIMABLE_B, - PAGE_SIZE << order); - } - - return kmalloc_large_node_hook(ptr, size, flags); -} - static __always_inline void *__do_kmalloc_node(size_t size, gfp_t flags, int node, unsigned long caller) { From patchwork Wed Aug 17 10:18:17 2022 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Hyeonggon Yoo <42.hyeyoo@gmail.com> X-Patchwork-Id: 12945687 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from kanga.kvack.org (kanga.kvack.org [205.233.56.17]) by smtp.lore.kernel.org (Postfix) with ESMTP id 129B1C25B08 for ; Wed, 17 Aug 2022 10:19:18 +0000 (UTC) Received: by kanga.kvack.org (Postfix) id AC7706B0080; Wed, 17 Aug 2022 06:19:17 -0400 (EDT) Received: by kanga.kvack.org (Postfix, from userid 40) id A77658D0001; Wed, 17 Aug 2022 06:19:17 -0400 (EDT) X-Delivered-To: int-list-linux-mm@kvack.org Received: by kanga.kvack.org (Postfix, from userid 63042) id 8F0F26B0082; Wed, 17 Aug 2022 06:19:17 -0400 (EDT) X-Delivered-To: linux-mm@kvack.org Received: from relay.hostedemail.com (smtprelay0014.hostedemail.com [216.40.44.14]) by kanga.kvack.org (Postfix) with ESMTP id 7FC736B0080 for ; Wed, 17 Aug 2022 06:19:17 -0400 (EDT) Received: from smtpin15.hostedemail.com (a10.router.float.18 [10.200.18.1]) by unirelay04.hostedemail.com (Postfix) with ESMTP id 5DC7D1A14F7 for ; Wed, 17 Aug 2022 10:19:17 +0000 (UTC) X-FDA: 79808687154.15.BF6DD12 Received: from mail-pg1-f182.google.com (mail-pg1-f182.google.com [209.85.215.182]) by imf20.hostedemail.com (Postfix) with ESMTP id D39CC1C01BA for ; Wed, 17 Aug 2022 10:19:16 +0000 (UTC) Received: by mail-pg1-f182.google.com with SMTP id d71so11580716pgc.13 for ; Wed, 17 Aug 2022 03:19:16 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=20210112; h=content-transfer-encoding:mime-version:references:in-reply-to :message-id:date:subject:cc:to:from:from:to:cc; bh=tUizZNwzs6wrGKQx11/4tG0R+Fhf4KT2C7xsd5j/OPo=; b=bY7PoNXTNun67H8bazC4lB6u5PTF5Ax0sm7RPZ5xOp0oXn6dZLFedj/4unFw/BlDDL mqTrR5AxcbSlSpRA45N8LaKISXNbmmfAewTh4D/t5lrPW/t+REUE7SsxahxsKg0Srcp6 dbyfTNerch1YbEmDg9hmcmdCMQm4UMFLr5EzYYzC3g/dLOYmxFQBLsyr6pG8YE4an+oZ WndXjF7Fgvn7VRcUho4F+SN/MWvoYbwWZJRjMjiuRkbB/PM7TYriAIlcffJhkczVh74l Osfzb/vU+aEncSSEVKo4JlhOnKOHr2lS5Th50FdOQLTxUyRRZQTVfbg2+BChABSBVcSI IbDg== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20210112; h=content-transfer-encoding:mime-version:references:in-reply-to :message-id:date:subject:cc:to:from:x-gm-message-state:from:to:cc; bh=tUizZNwzs6wrGKQx11/4tG0R+Fhf4KT2C7xsd5j/OPo=; b=KZR68TNm/d7XsqOhfhtGaewq83uszT3N23HhZsQlyWu4RNmVlfgsg4I99mLVyc5cJi F2xRV770AANaSVci43HZJyJw5fVreokYGJX7u15yca7VHgag2OG3oMrxYjbtEBr4V14A 7QrZsc3jdm0Jmgn7dUnInbGR2nBOMVQyG5Ni31PqjJbpQQ6JnM0BpgB/ApzkDmo2mK4x yN2Q8EzhzJUj7cDbNlL2ND7tkC9jXoE77djK5pkr+ctTVdJzlzey7Vth0fyJYWOSM2Lg byjdBg32ngne6LcNrxZ/b3R7GUveCRq/JyaeyqYZTsntQz5LAMbMVWSNwHFzH4qXkSBk MeLw== X-Gm-Message-State: ACgBeo1uS4GT/uMdANVQgt3iFpPWEXbqn+cldlEXVxeIU2CTHvsi/5JA q+eF5/BiEIVJOJO8sq0sicc= X-Google-Smtp-Source: AA6agR4P3TPHzp2fITNYUXY7GsNidyO6PDhQ+K8ehcHtYRDvNqx9oqwhJcudXfK2S6wxGaR7sSRMSA== X-Received: by 2002:a63:5916:0:b0:41d:2c8c:7492 with SMTP id n22-20020a635916000000b0041d2c8c7492mr20956775pgb.81.1660731555697; Wed, 17 Aug 2022 03:19:15 -0700 (PDT) Received: from hyeyoo.. ([114.29.91.56]) by smtp.gmail.com with ESMTPSA id d8-20020a170903230800b00172633fc236sm1071318plh.174.2022.08.17.03.19.12 (version=TLS1_3 cipher=TLS_AES_256_GCM_SHA384 bits=256/256); Wed, 17 Aug 2022 03:19:14 -0700 (PDT) From: Hyeonggon Yoo <42.hyeyoo@gmail.com> To: Christoph Lameter , Pekka Enberg , David Rientjes , Joonsoo Kim , Andrew Morton , Vlastimil Babka , Roman Gushchin Cc: Hyeonggon Yoo <42.hyeyoo@gmail.com>, linux-mm@kvack.org, linux-kernel@vger.kernel.org Subject: [PATCH v4 08/17] mm/slab_common: kmalloc_node: pass large requests to page allocator Date: Wed, 17 Aug 2022 19:18:17 +0900 Message-Id: <20220817101826.236819-9-42.hyeyoo@gmail.com> X-Mailer: git-send-email 2.32.0 In-Reply-To: <20220817101826.236819-1-42.hyeyoo@gmail.com> References: <20220817101826.236819-1-42.hyeyoo@gmail.com> MIME-Version: 1.0 ARC-Authentication-Results: i=1; imf20.hostedemail.com; dkim=pass header.d=gmail.com header.s=20210112 header.b=bY7PoNXT; spf=pass (imf20.hostedemail.com: domain of 42.hyeyoo@gmail.com designates 209.85.215.182 as permitted sender) smtp.mailfrom=42.hyeyoo@gmail.com; dmarc=pass (policy=none) header.from=gmail.com ARC-Seal: i=1; s=arc-20220608; d=hostedemail.com; t=1660731556; a=rsa-sha256; cv=none; b=ZPuOGaBz/twphkyIpr3PDF5sE1acOW+hVyig//DFUBy5qoAVmOIdlLAWHSU08QpGJGeqKS hqFFNCHI5tRXGDWcHu7YX3Ut6M3KZiSZqAtXhqI9RPzr2YwuCe6Afqd/V++9mAB/k4imLm EpBseaeMUFyq+ISCFVvx0unLSNNKuuQ= ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=hostedemail.com; s=arc-20220608; t=1660731556; h=from:from:sender:reply-to:subject:subject:date:date: message-id:message-id:to:to:cc:cc:mime-version:mime-version: content-type:content-transfer-encoding:content-transfer-encoding: in-reply-to:in-reply-to:references:references:dkim-signature; bh=tUizZNwzs6wrGKQx11/4tG0R+Fhf4KT2C7xsd5j/OPo=; b=wBE7zPwEnwHPfV7yvWqGcMFbpr3vl0rsULA1kl7jrV1zl0GL3FYxoFEP1d/8avSRRNyOyy JchBj2BOWya3Wltb1oBMNyM3I9PMRfy60mtdQkyiRxE871oWfYymV6Ek87h+9KbRVMnGpb InFQFjWF2DJSSCZlclUbQR9GsslFdgM= X-Stat-Signature: z7xpm6n7gm7qd864rzxzpemjzymt6ad7 X-Rspamd-Queue-Id: D39CC1C01BA X-Rspamd-Server: rspam08 Authentication-Results: imf20.hostedemail.com; dkim=pass header.d=gmail.com header.s=20210112 header.b=bY7PoNXT; spf=pass (imf20.hostedemail.com: domain of 42.hyeyoo@gmail.com designates 209.85.215.182 as permitted sender) smtp.mailfrom=42.hyeyoo@gmail.com; dmarc=pass (policy=none) header.from=gmail.com X-Rspam-User: X-HE-Tag: 1660731556-533901 X-Bogosity: Ham, tests=bogofilter, spamicity=0.000000, version=1.2.4 Sender: owner-linux-mm@kvack.org Precedence: bulk X-Loop: owner-majordomo@kvack.org List-ID: Now that kmalloc_large_node() is in common code, pass large requests to page allocator in kmalloc_node() using kmalloc_large_node(). One problem is that currently there is no tracepoint in kmalloc_large_node(). Instead of simply putting tracepoint in it, use kmalloc_large_node{,_notrace} depending on its caller to show useful address for both inlined kmalloc_node() and __kmalloc_node_track_caller() when large objects are allocated. Signed-off-by: Hyeonggon Yoo <42.hyeyoo@gmail.com> Reviewed-by: Vlastimil Babka --- include/linux/slab.h | 26 +++++++++++++++++++------- mm/slab.h | 2 ++ mm/slab_common.c | 11 ++++++++++- mm/slub.c | 2 +- 4 files changed, 32 insertions(+), 9 deletions(-) diff --git a/include/linux/slab.h b/include/linux/slab.h index 082499306098..fd2e129fc813 100644 --- a/include/linux/slab.h +++ b/include/linux/slab.h @@ -571,23 +571,35 @@ static __always_inline __alloc_size(1) void *kmalloc(size_t size, gfp_t flags) return __kmalloc(size, flags); } +#ifndef CONFIG_SLOB static __always_inline __alloc_size(1) void *kmalloc_node(size_t size, gfp_t flags, int node) { -#ifndef CONFIG_SLOB - if (__builtin_constant_p(size) && - size <= KMALLOC_MAX_CACHE_SIZE) { - unsigned int i = kmalloc_index(size); + if (__builtin_constant_p(size)) { + unsigned int index; - if (!i) + if (size > KMALLOC_MAX_CACHE_SIZE) + return kmalloc_large_node(size, flags, node); + + index = kmalloc_index(size); + + if (!index) return ZERO_SIZE_PTR; return kmem_cache_alloc_node_trace( - kmalloc_caches[kmalloc_type(flags)][i], + kmalloc_caches[kmalloc_type(flags)][index], flags, node, size); } -#endif return __kmalloc_node(size, flags, node); } +#else +static __always_inline __alloc_size(1) void *kmalloc_node(size_t size, gfp_t flags, int node) +{ + if (__builtin_constant_p(size) && size > KMALLOC_MAX_CACHE_SIZE) + return kmalloc_large_node(size, flags, node); + + return __kmalloc_node(size, flags, node); +} +#endif /** * kmalloc_array - allocate memory for an array. diff --git a/mm/slab.h b/mm/slab.h index 4ec82bec15ec..40322bcf07be 100644 --- a/mm/slab.h +++ b/mm/slab.h @@ -273,6 +273,8 @@ void create_kmalloc_caches(slab_flags_t); /* Find the kmalloc slab corresponding for a certain size */ struct kmem_cache *kmalloc_slab(size_t, gfp_t); + +void *kmalloc_large_node_notrace(size_t size, gfp_t flags, int node); #endif gfp_t kmalloc_fix_flags(gfp_t flags); diff --git a/mm/slab_common.c b/mm/slab_common.c index 1b9101f9cb21..7a0942d54424 100644 --- a/mm/slab_common.c +++ b/mm/slab_common.c @@ -928,7 +928,7 @@ void *kmalloc_large(size_t size, gfp_t flags) } EXPORT_SYMBOL(kmalloc_large); -void *kmalloc_large_node(size_t size, gfp_t flags, int node) +void *kmalloc_large_node_notrace(size_t size, gfp_t flags, int node) { struct page *page; void *ptr = NULL; @@ -948,6 +948,15 @@ void *kmalloc_large_node(size_t size, gfp_t flags, int node) return ptr; } + +void *kmalloc_large_node(size_t size, gfp_t flags, int node) +{ + void *ret = kmalloc_large_node_notrace(size, flags, node); + + trace_kmalloc_node(_RET_IP_, ret, NULL, size, + PAGE_SIZE << get_order(size), flags, node); + return ret; +} EXPORT_SYMBOL(kmalloc_large_node); #ifdef CONFIG_SLAB_FREELIST_RANDOM diff --git a/mm/slub.c b/mm/slub.c index 5e7819ade2c4..165fe87af204 100644 --- a/mm/slub.c +++ b/mm/slub.c @@ -4401,7 +4401,7 @@ void *__do_kmalloc_node(size_t size, gfp_t flags, int node, unsigned long caller void *ret; if (unlikely(size > KMALLOC_MAX_CACHE_SIZE)) { - ret = kmalloc_large_node(size, flags, node); + ret = kmalloc_large_node_notrace(size, flags, node); trace_kmalloc_node(caller, ret, NULL, size, PAGE_SIZE << get_order(size), From patchwork Wed Aug 17 10:18:18 2022 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Hyeonggon Yoo <42.hyeyoo@gmail.com> X-Patchwork-Id: 12945688 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from kanga.kvack.org (kanga.kvack.org [205.233.56.17]) by smtp.lore.kernel.org (Postfix) with ESMTP id 83187C2BB41 for ; Wed, 17 Aug 2022 10:19:21 +0000 (UTC) Received: by kanga.kvack.org (Postfix) id 23B916B0081; Wed, 17 Aug 2022 06:19:21 -0400 (EDT) Received: by kanga.kvack.org (Postfix, from userid 40) id 1EB928D0001; Wed, 17 Aug 2022 06:19:21 -0400 (EDT) X-Delivered-To: int-list-linux-mm@kvack.org Received: by kanga.kvack.org (Postfix, from userid 63042) id 08D036B0083; Wed, 17 Aug 2022 06:19:21 -0400 (EDT) X-Delivered-To: linux-mm@kvack.org Received: from relay.hostedemail.com (smtprelay0015.hostedemail.com [216.40.44.15]) by kanga.kvack.org (Postfix) with ESMTP id EDD9E6B0081 for ; Wed, 17 Aug 2022 06:19:20 -0400 (EDT) Received: from smtpin17.hostedemail.com (a10.router.float.18 [10.200.18.1]) by unirelay07.hostedemail.com (Postfix) with ESMTP id CFDB91605A6 for ; Wed, 17 Aug 2022 10:19:20 +0000 (UTC) X-FDA: 79808687280.17.043E991 Received: from mail-pj1-f53.google.com (mail-pj1-f53.google.com [209.85.216.53]) by imf02.hostedemail.com (Postfix) with ESMTP id 7A63D80049 for ; Wed, 17 Aug 2022 10:19:20 +0000 (UTC) Received: by mail-pj1-f53.google.com with SMTP id s36-20020a17090a69a700b001faad0a7a34so1388344pjj.4 for ; Wed, 17 Aug 2022 03:19:20 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=20210112; h=content-transfer-encoding:mime-version:references:in-reply-to :message-id:date:subject:cc:to:from:from:to:cc; bh=yob3/+IwNeWgOZVqPDUGNXwyBAO5sJAn+qw3hk8rQbE=; b=Tna2JkyXzqn5qE47zwCxnPqMJBvOs2MucxtFWhJJ0c/lzCw9K5Z32HPbeuTEFcnaXm euh69XaJIiwNQsXoFXICFSU+/Oca1A0aBr+VKjaQwRsqZn0ImCwMvZM0N8OAtELq5gWO 8FFxXGzMld8SE/bklMsCfmz8zmuRhUc1KMMSce/NCBQ/LcnIuDs1AfKmri8IhYcblm+V rrNNS4fF5tsR5jmnvk57xPPvLXbTFJrN6HyWLycKFPInDB+wHEqRePYu7yjFgNU3vP5E BG+0gLROd9cISMZo+XFOB9NN0yZR8ajSr+tMCE72foa4xqmEUVSggTGPNNWEhj7k62Sa xqUQ== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20210112; h=content-transfer-encoding:mime-version:references:in-reply-to :message-id:date:subject:cc:to:from:x-gm-message-state:from:to:cc; bh=yob3/+IwNeWgOZVqPDUGNXwyBAO5sJAn+qw3hk8rQbE=; b=BUN9EFedNdSThqilWxCzf42mUbxokvaXLKXNYP79c5qOwwcANNEWyeKPRTnEHdo9f9 YTZiCInc/alJzVBVks4dFI2XJkj/M1MxDzPCRu8nRTVGicLefsyFlRcl5bjELyKh5WmM Qvc64zB1BCfO/wfOoAVIkMflNGFONarDPp2v8UR8FKUja6ItXflbYPuEvtVo9uqq09Bf zRWSoZNm0G1G1eo8fe6jyYb1dBKna/R4tXWZL5aH5eEE+jEFdJ2iuqc0JlAt8u1tGyuX cVKb+vlfQCQlzzKj4ZfZdOi0P8JC/NOO21oC6i73uoAnJUI0xJTH77HyHZOv63Siz0cy Ch3Q== X-Gm-Message-State: ACgBeo1ONQXTJVC/TSt4NknQNid+olPxhWHuC5hlCZYD7+hdRfaX2EaY CufhWaGWssBYeotUEx7SQkI= X-Google-Smtp-Source: AA6agR4US9kp+7qbu6CM869/bm31zI3vqMNat3pYfqBfuj2fagq4H+iDwarnNS8c/JYltDYP+XKAMw== X-Received: by 2002:a17:90b:48c3:b0:1fa:7ade:384a with SMTP id li3-20020a17090b48c300b001fa7ade384amr3021105pjb.106.1660731559586; Wed, 17 Aug 2022 03:19:19 -0700 (PDT) Received: from hyeyoo.. ([114.29.91.56]) by smtp.gmail.com with ESMTPSA id d8-20020a170903230800b00172633fc236sm1071318plh.174.2022.08.17.03.19.16 (version=TLS1_3 cipher=TLS_AES_256_GCM_SHA384 bits=256/256); Wed, 17 Aug 2022 03:19:18 -0700 (PDT) From: Hyeonggon Yoo <42.hyeyoo@gmail.com> To: Christoph Lameter , Pekka Enberg , David Rientjes , Joonsoo Kim , Andrew Morton , Vlastimil Babka , Roman Gushchin Cc: Hyeonggon Yoo <42.hyeyoo@gmail.com>, linux-mm@kvack.org, linux-kernel@vger.kernel.org Subject: [PATCH v4 09/17] mm/slab_common: cleanup kmalloc_large() Date: Wed, 17 Aug 2022 19:18:18 +0900 Message-Id: <20220817101826.236819-10-42.hyeyoo@gmail.com> X-Mailer: git-send-email 2.32.0 In-Reply-To: <20220817101826.236819-1-42.hyeyoo@gmail.com> References: <20220817101826.236819-1-42.hyeyoo@gmail.com> MIME-Version: 1.0 ARC-Seal: i=1; s=arc-20220608; d=hostedemail.com; t=1660731560; a=rsa-sha256; cv=none; b=JiQPxZNn2tFSC4ZcEaZuCSQh72T0oR/QwizcCQXC/OhsNKNqfLHa+XSsafNwvk5glh/EZb KeRQYwK2mL1hfHApQttZ7uTfQLzf3nvGlfXXvXnV1AL7J9V6Qix3a8N0X29hwxC1C34CDS Cqg7B4jpqXeA19DLXt70fr98JCdBrjM= ARC-Authentication-Results: i=1; imf02.hostedemail.com; dkim=pass header.d=gmail.com header.s=20210112 header.b=Tna2JkyX; dmarc=pass (policy=none) header.from=gmail.com; spf=pass (imf02.hostedemail.com: domain of 42.hyeyoo@gmail.com designates 209.85.216.53 as permitted sender) smtp.mailfrom=42.hyeyoo@gmail.com ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=hostedemail.com; s=arc-20220608; t=1660731560; h=from:from:sender:reply-to:subject:subject:date:date: message-id:message-id:to:to:cc:cc:mime-version:mime-version: content-type:content-transfer-encoding:content-transfer-encoding: in-reply-to:in-reply-to:references:references:dkim-signature; bh=yob3/+IwNeWgOZVqPDUGNXwyBAO5sJAn+qw3hk8rQbE=; b=WyHmaUiWx2ajdi+8dgIf6bbj9SOYkyzebzH6ffiuKIe3L/gHxiUr2yxQc9ruI8jmzyVNS5 h8KBsbQBw4NtJeZZ3MXclLXdKtCk4gCYXhQdCGanXyH+bRgH8RJ21QBMj8rnYRMK+igKms jHPS2OBm463liqPdl6s1AIcQwOKk5FI= X-Stat-Signature: hynefuukitg17xbtm4i7rwe8t4q7z95x X-Rspamd-Queue-Id: 7A63D80049 Authentication-Results: imf02.hostedemail.com; dkim=pass header.d=gmail.com header.s=20210112 header.b=Tna2JkyX; dmarc=pass (policy=none) header.from=gmail.com; spf=pass (imf02.hostedemail.com: domain of 42.hyeyoo@gmail.com designates 209.85.216.53 as permitted sender) smtp.mailfrom=42.hyeyoo@gmail.com X-Rspam-User: X-Rspamd-Server: rspam10 X-HE-Tag: 1660731560-901787 X-Bogosity: Ham, tests=bogofilter, spamicity=0.000000, version=1.2.4 Sender: owner-linux-mm@kvack.org Precedence: bulk X-Loop: owner-majordomo@kvack.org List-ID: Now that kmalloc_large() and kmalloc_large_node() do mostly same job, make kmalloc_large() wrapper of kmalloc_large_node_notrace(). In the meantime, add missing flag fix code in kmalloc_large_node_notrace(). Signed-off-by: Hyeonggon Yoo <42.hyeyoo@gmail.com> Reviewed-by: Vlastimil Babka --- mm/slab_common.c | 35 +++++++++++++---------------------- 1 file changed, 13 insertions(+), 22 deletions(-) diff --git a/mm/slab_common.c b/mm/slab_common.c index 7a0942d54424..51ccd0545816 100644 --- a/mm/slab_common.c +++ b/mm/slab_common.c @@ -905,28 +905,6 @@ gfp_t kmalloc_fix_flags(gfp_t flags) * directly to the page allocator. We use __GFP_COMP, because we will need to * know the allocation order to free the pages properly in kfree. */ -void *kmalloc_large(size_t size, gfp_t flags) -{ - void *ret = NULL; - struct page *page; - unsigned int order = get_order(size); - - if (unlikely(flags & GFP_SLAB_BUG_MASK)) - flags = kmalloc_fix_flags(flags); - - page = alloc_pages(flags | __GFP_COMP, order); - if (likely(page)) { - ret = page_address(page); - mod_lruvec_page_state(page, NR_SLAB_UNRECLAIMABLE_B, - PAGE_SIZE << order); - } - ret = kasan_kmalloc_large(ret, size, flags); - /* As ret might get tagged, call kmemleak hook after KASAN. */ - kmemleak_alloc(ret, size, 1, flags); - trace_kmalloc(_RET_IP_, ret, NULL, size, PAGE_SIZE << order, flags); - return ret; -} -EXPORT_SYMBOL(kmalloc_large); void *kmalloc_large_node_notrace(size_t size, gfp_t flags, int node) { @@ -934,6 +912,9 @@ void *kmalloc_large_node_notrace(size_t size, gfp_t flags, int node) void *ptr = NULL; unsigned int order = get_order(size); + if (unlikely(flags & GFP_SLAB_BUG_MASK)) + flags = kmalloc_fix_flags(flags); + flags |= __GFP_COMP; page = alloc_pages_node(node, flags, order); if (page) { @@ -949,6 +930,16 @@ void *kmalloc_large_node_notrace(size_t size, gfp_t flags, int node) return ptr; } +void *kmalloc_large(size_t size, gfp_t flags) +{ + void *ret = kmalloc_large_node_notrace(size, flags, NUMA_NO_NODE); + + trace_kmalloc(_RET_IP_, ret, NULL, size, + PAGE_SIZE << get_order(size), flags); + return ret; +} +EXPORT_SYMBOL(kmalloc_large); + void *kmalloc_large_node(size_t size, gfp_t flags, int node) { void *ret = kmalloc_large_node_notrace(size, flags, node); From patchwork Wed Aug 17 10:18:19 2022 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Hyeonggon Yoo <42.hyeyoo@gmail.com> X-Patchwork-Id: 12945689 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from kanga.kvack.org (kanga.kvack.org [205.233.56.17]) by smtp.lore.kernel.org (Postfix) with ESMTP id 75442C25B08 for ; Wed, 17 Aug 2022 10:19:25 +0000 (UTC) Received: by kanga.kvack.org (Postfix) id 151516B0074; Wed, 17 Aug 2022 06:19:25 -0400 (EDT) Received: by kanga.kvack.org (Postfix, from userid 40) id 101E56B007D; Wed, 17 Aug 2022 06:19:25 -0400 (EDT) X-Delivered-To: int-list-linux-mm@kvack.org Received: by kanga.kvack.org (Postfix, from userid 63042) id EBDF76B0082; Wed, 17 Aug 2022 06:19:24 -0400 (EDT) X-Delivered-To: linux-mm@kvack.org Received: from relay.hostedemail.com (smtprelay0011.hostedemail.com [216.40.44.11]) by kanga.kvack.org (Postfix) with ESMTP id D8FF76B0074 for ; Wed, 17 Aug 2022 06:19:24 -0400 (EDT) Received: from smtpin31.hostedemail.com (a10.router.float.18 [10.200.18.1]) by unirelay07.hostedemail.com (Postfix) with ESMTP id B81D216087A for ; Wed, 17 Aug 2022 10:19:24 +0000 (UTC) X-FDA: 79808687448.31.EDA400C Received: from mail-pl1-f171.google.com (mail-pl1-f171.google.com [209.85.214.171]) by imf05.hostedemail.com (Postfix) with ESMTP id 4FD851001C7 for ; Wed, 17 Aug 2022 10:19:24 +0000 (UTC) Received: by mail-pl1-f171.google.com with SMTP id t3so3760389ply.2 for ; Wed, 17 Aug 2022 03:19:24 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=20210112; h=content-transfer-encoding:mime-version:references:in-reply-to :message-id:date:subject:cc:to:from:from:to:cc; bh=paupgt1G6cH3RqeKhiTalrXFR+ytTeKPfHkaPAB0YdA=; b=bK1aYHR1zgO31MEYMp0z27Z1q5eFMJ+tzZkZPU/xUi7QSEqc9gJgYHZt9kOlcxyZiS l7ibFrX6CclG1r7qw3jgmSgOkKp/IdIrlIVH9ewekQmF9WODMD0nStvwZw7chdKQEsRo uY/mDnL/2e/Ac/izgf03aEtmPtRgcUeBJFudRS3LJlSPla97U7S29W0QH4L/lqMEpGpZ veX1hxijOV7f9qs7xnEiiXbnoobxzdA5MMvAKhA6uiRDKnGKAXdSpRhL+hzJGLvv4CDp rgA2OfPYtOnydhMw3vtTwO1eo/x8yi4gzdShPaxj1bvxTxnz7N7RRKksHrVlKaFyeEdq FdCw== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20210112; h=content-transfer-encoding:mime-version:references:in-reply-to :message-id:date:subject:cc:to:from:x-gm-message-state:from:to:cc; bh=paupgt1G6cH3RqeKhiTalrXFR+ytTeKPfHkaPAB0YdA=; b=VPqIgHuZqTCxk0BFQb0b35yXYbhYVzslr2t6fjo3adx1/OR0LNCsfGe9GiJVO4OZ+D Y7IXtZjq1t9UXjGEqVcNgkVHhvDSPf2ykOtMopEI8U1xmTOZGn7tya8J5/f8kbR5orbg kunnt4lbyfn/YRpqTOEhtcmtVvxh3eIDnrd/4i8Y5suWX2usuJtxSGOyh5O1iYZWInM1 oxisYq48O2J5S71aFoI3owfNnIGE3QyOfNg726gsCZLZhSA1SAOt4hZbBq4JX59Vq/O/ x3PNv10Bm2LSzDk3ZYwOnLZvjFO3eJQcIB7erQsNdNej4J8rJZYR1hnNlpa4Agm4g6Uh 6Xnw== X-Gm-Message-State: ACgBeo0MuQit6BHFuWMdKXoslEwq5p0tudaGSRKV6BjS/UTLTVGlub3u WKzqAGIJAkUoCGWYGkb3CqA= X-Google-Smtp-Source: AA6agR4LT57t3RNaykBg4K7+bvNe5zXr5bsxJoXJ3gs6cRtgISyVA95RK4nraz+NxdIVhKsWrJ/jeg== X-Received: by 2002:a17:903:11c7:b0:171:2818:4cd7 with SMTP id q7-20020a17090311c700b0017128184cd7mr25306274plh.136.1660731563319; Wed, 17 Aug 2022 03:19:23 -0700 (PDT) Received: from hyeyoo.. ([114.29.91.56]) by smtp.gmail.com with ESMTPSA id d8-20020a170903230800b00172633fc236sm1071318plh.174.2022.08.17.03.19.19 (version=TLS1_3 cipher=TLS_AES_256_GCM_SHA384 bits=256/256); Wed, 17 Aug 2022 03:19:22 -0700 (PDT) From: Hyeonggon Yoo <42.hyeyoo@gmail.com> To: Christoph Lameter , Pekka Enberg , David Rientjes , Joonsoo Kim , Andrew Morton , Vlastimil Babka , Roman Gushchin Cc: Hyeonggon Yoo <42.hyeyoo@gmail.com>, linux-mm@kvack.org, linux-kernel@vger.kernel.org Subject: [PATCH v4 10/17] mm/slab: kmalloc: pass requests larger than order-1 page to page allocator Date: Wed, 17 Aug 2022 19:18:19 +0900 Message-Id: <20220817101826.236819-11-42.hyeyoo@gmail.com> X-Mailer: git-send-email 2.32.0 In-Reply-To: <20220817101826.236819-1-42.hyeyoo@gmail.com> References: <20220817101826.236819-1-42.hyeyoo@gmail.com> MIME-Version: 1.0 ARC-Seal: i=1; s=arc-20220608; d=hostedemail.com; t=1660731564; a=rsa-sha256; cv=none; b=zDuAVL3S888aiV8ZuE4gTG7rJWF0Fatk1UgBY5KVI7RApEw7ueVxq7BBLcVrMOMhyqFOSK zPkQA1DE7plC6C+Rvrw+oRn64LJtwDyvyp6KyqOZ4GM2bvv2ch/ZZQMxcwMkWGN8pxhUyc 1LG06HcswK9/OeJjnOWb4kfhUV03rFs= ARC-Authentication-Results: i=1; imf05.hostedemail.com; dkim=pass header.d=gmail.com header.s=20210112 header.b=bK1aYHR1; spf=pass (imf05.hostedemail.com: domain of 42.hyeyoo@gmail.com designates 209.85.214.171 as permitted sender) smtp.mailfrom=42.hyeyoo@gmail.com; dmarc=pass (policy=none) header.from=gmail.com ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=hostedemail.com; s=arc-20220608; t=1660731564; h=from:from:sender:reply-to:subject:subject:date:date: message-id:message-id:to:to:cc:cc:mime-version:mime-version: content-type:content-transfer-encoding:content-transfer-encoding: in-reply-to:in-reply-to:references:references:dkim-signature; bh=paupgt1G6cH3RqeKhiTalrXFR+ytTeKPfHkaPAB0YdA=; b=tBgKBHxr5OyGbFvsLIXsANmRWi+sWl6ukOplw3Ag5wjvHBzfG5a3TarU/WKmmxIZvACLJ9 Sgq2A1R9lKgFF++e2ExPcL8Id7QqZ3elumNp4yBwxBDZlwY/uahWDoxn/5NyZExS+AFA4X CrG4dJOJ9fa673jhnYUU+wh+HaD5ziU= X-Stat-Signature: stc69z5z4i8nx8pw19sa1ddzxi4kkttt X-Rspamd-Queue-Id: 4FD851001C7 Authentication-Results: imf05.hostedemail.com; dkim=pass header.d=gmail.com header.s=20210112 header.b=bK1aYHR1; spf=pass (imf05.hostedemail.com: domain of 42.hyeyoo@gmail.com designates 209.85.214.171 as permitted sender) smtp.mailfrom=42.hyeyoo@gmail.com; dmarc=pass (policy=none) header.from=gmail.com X-Rspam-User: X-Rspamd-Server: rspam03 X-HE-Tag: 1660731564-201516 X-Bogosity: Ham, tests=bogofilter, spamicity=0.000000, version=1.2.4 Sender: owner-linux-mm@kvack.org Precedence: bulk X-Loop: owner-majordomo@kvack.org List-ID: There is not much benefit for serving large objects in kmalloc(). Let's pass large requests to page allocator like SLUB for better maintenance of common code. Signed-off-by: Hyeonggon Yoo <42.hyeyoo@gmail.com> Reviewed-by: Vlastimil Babka --- include/linux/slab.h | 23 ++++------------- mm/slab.c | 60 +++++++++++++++++++++++++++++++------------- mm/slab.h | 3 +++ mm/slab_common.c | 25 ++++++++++++------ mm/slub.c | 19 -------------- 5 files changed, 68 insertions(+), 62 deletions(-) diff --git a/include/linux/slab.h b/include/linux/slab.h index fd2e129fc813..4ee5b2fed164 100644 --- a/include/linux/slab.h +++ b/include/linux/slab.h @@ -243,27 +243,17 @@ static inline unsigned int arch_slab_minalign(void) #ifdef CONFIG_SLAB /* - * The largest kmalloc size supported by the SLAB allocators is - * 32 megabyte (2^25) or the maximum allocatable page order if that is - * less than 32 MB. - * - * WARNING: Its not easy to increase this value since the allocators have - * to do various tricks to work around compiler limitations in order to - * ensure proper constant folding. + * SLAB and SLUB directly allocates requests fitting in to an order-1 page + * (PAGE_SIZE*2). Larger requests are passed to the page allocator. */ -#define KMALLOC_SHIFT_HIGH ((MAX_ORDER + PAGE_SHIFT - 1) <= 25 ? \ - (MAX_ORDER + PAGE_SHIFT - 1) : 25) -#define KMALLOC_SHIFT_MAX KMALLOC_SHIFT_HIGH +#define KMALLOC_SHIFT_HIGH (PAGE_SHIFT + 1) +#define KMALLOC_SHIFT_MAX (MAX_ORDER + PAGE_SHIFT - 1) #ifndef KMALLOC_SHIFT_LOW #define KMALLOC_SHIFT_LOW 5 #endif #endif #ifdef CONFIG_SLUB -/* - * SLUB directly allocates requests fitting in to an order-1 page - * (PAGE_SIZE*2). Larger requests are passed to the page allocator. - */ #define KMALLOC_SHIFT_HIGH (PAGE_SHIFT + 1) #define KMALLOC_SHIFT_MAX (MAX_ORDER + PAGE_SHIFT - 1) #ifndef KMALLOC_SHIFT_LOW @@ -415,10 +405,6 @@ static __always_inline unsigned int __kmalloc_index(size_t size, if (size <= 512 * 1024) return 19; if (size <= 1024 * 1024) return 20; if (size <= 2 * 1024 * 1024) return 21; - if (size <= 4 * 1024 * 1024) return 22; - if (size <= 8 * 1024 * 1024) return 23; - if (size <= 16 * 1024 * 1024) return 24; - if (size <= 32 * 1024 * 1024) return 25; if (!IS_ENABLED(CONFIG_PROFILE_ALL_BRANCHES) && size_is_constant) BUILD_BUG_ON_MSG(1, "unexpected size in kmalloc_index()"); @@ -428,6 +414,7 @@ static __always_inline unsigned int __kmalloc_index(size_t size, /* Will never be reached. Needed because the compiler may complain */ return -1; } +static_assert(PAGE_SHIFT <= 20); #define kmalloc_index(s) __kmalloc_index(s, true) #endif /* !CONFIG_SLOB */ diff --git a/mm/slab.c b/mm/slab.c index 8c08d7f3dead..10c9af904410 100644 --- a/mm/slab.c +++ b/mm/slab.c @@ -3585,11 +3585,19 @@ __do_kmalloc_node(size_t size, gfp_t flags, int node, unsigned long caller) struct kmem_cache *cachep; void *ret; - if (unlikely(size > KMALLOC_MAX_CACHE_SIZE)) - return NULL; + if (unlikely(size > KMALLOC_MAX_CACHE_SIZE)) { + ret = kmalloc_large_node_notrace(size, flags, node); + + trace_kmalloc_node(caller, ret, NULL, size, + PAGE_SIZE << get_order(size), + flags, node); + return ret; + } + cachep = kmalloc_slab(size, flags); if (unlikely(ZERO_OR_NULL_PTR(cachep))) return cachep; + ret = kmem_cache_alloc_node_trace(cachep, flags, node, size); ret = kasan_kmalloc(cachep, ret, size, flags); @@ -3664,17 +3672,27 @@ EXPORT_SYMBOL(kmem_cache_free); void kmem_cache_free_bulk(struct kmem_cache *orig_s, size_t size, void **p) { - struct kmem_cache *s; - size_t i; local_irq_disable(); - for (i = 0; i < size; i++) { + for (int i = 0; i < size; i++) { void *objp = p[i]; + struct kmem_cache *s; - if (!orig_s) /* called via kfree_bulk */ - s = virt_to_cache(objp); - else + if (!orig_s) { + struct folio *folio = virt_to_folio(objp); + + /* called via kfree_bulk */ + if (!folio_test_slab(folio)) { + local_irq_enable(); + free_large_kmalloc(folio, objp); + local_irq_disable(); + continue; + } + s = folio_slab(folio)->slab_cache; + } else { s = cache_from_obj(orig_s, objp); + } + if (!s) continue; @@ -3703,20 +3721,24 @@ void kfree(const void *objp) { struct kmem_cache *c; unsigned long flags; + struct folio *folio; trace_kfree(_RET_IP_, objp); if (unlikely(ZERO_OR_NULL_PTR(objp))) return; - local_irq_save(flags); - kfree_debugcheck(objp); - c = virt_to_cache(objp); - if (!c) { - local_irq_restore(flags); + + folio = virt_to_folio(objp); + if (!folio_test_slab(folio)) { + free_large_kmalloc(folio, (void *)objp); return; } - debug_check_no_locks_freed(objp, c->object_size); + c = folio_slab(folio)->slab_cache; + + local_irq_save(flags); + kfree_debugcheck(objp); + debug_check_no_locks_freed(objp, c->object_size); debug_check_no_obj_freed(objp, c->object_size); __cache_free(c, (void *)objp, _RET_IP_); local_irq_restore(flags); @@ -4138,15 +4160,17 @@ void __check_heap_object(const void *ptr, unsigned long n, size_t __ksize(const void *objp) { struct kmem_cache *c; - size_t size; + struct folio *folio; BUG_ON(!objp); if (unlikely(objp == ZERO_SIZE_PTR)) return 0; - c = virt_to_cache(objp); - size = c ? c->object_size : 0; + folio = virt_to_folio(objp); + if (!folio_test_slab(folio)) + return folio_size(folio); - return size; + c = folio_slab(folio)->slab_cache; + return c->object_size; } EXPORT_SYMBOL(__ksize); diff --git a/mm/slab.h b/mm/slab.h index 40322bcf07be..381ba3e6b2a1 100644 --- a/mm/slab.h +++ b/mm/slab.h @@ -660,6 +660,9 @@ static inline struct kmem_cache *cache_from_obj(struct kmem_cache *s, void *x) print_tracking(cachep, x); return cachep; } + +void free_large_kmalloc(struct folio *folio, void *object); + #endif /* CONFIG_SLOB */ static inline size_t slab_ksize(const struct kmem_cache *s) diff --git a/mm/slab_common.c b/mm/slab_common.c index 51ccd0545816..5a2e81f42ee9 100644 --- a/mm/slab_common.c +++ b/mm/slab_common.c @@ -744,8 +744,8 @@ struct kmem_cache *kmalloc_slab(size_t size, gfp_t flags) /* * kmalloc_info[] is to make slub_debug=,kmalloc-xx option work at boot time. - * kmalloc_index() supports up to 2^25=32MB, so the final entry of the table is - * kmalloc-32M. + * kmalloc_index() supports up to 2^21=2MB, so the final entry of the table is + * kmalloc-2M. */ const struct kmalloc_info_struct kmalloc_info[] __initconst = { INIT_KMALLOC_INFO(0, 0), @@ -769,11 +769,7 @@ const struct kmalloc_info_struct kmalloc_info[] __initconst = { INIT_KMALLOC_INFO(262144, 256k), INIT_KMALLOC_INFO(524288, 512k), INIT_KMALLOC_INFO(1048576, 1M), - INIT_KMALLOC_INFO(2097152, 2M), - INIT_KMALLOC_INFO(4194304, 4M), - INIT_KMALLOC_INFO(8388608, 8M), - INIT_KMALLOC_INFO(16777216, 16M), - INIT_KMALLOC_INFO(33554432, 32M) + INIT_KMALLOC_INFO(2097152, 2M) }; /* @@ -886,6 +882,21 @@ void __init create_kmalloc_caches(slab_flags_t flags) /* Kmalloc array is now usable */ slab_state = UP; } + +void free_large_kmalloc(struct folio *folio, void *object) +{ + unsigned int order = folio_order(folio); + + if (WARN_ON_ONCE(order == 0)) + pr_warn_once("object pointer: 0x%p\n", object); + + kmemleak_free(object); + kasan_kfree_large(object); + + mod_lruvec_page_state(folio_page(folio, 0), NR_SLAB_UNRECLAIMABLE_B, + -(PAGE_SIZE << order)); + __free_pages(folio_page(folio, 0), order); +} #endif /* !CONFIG_SLOB */ gfp_t kmalloc_fix_flags(gfp_t flags) diff --git a/mm/slub.c b/mm/slub.c index 165fe87af204..a659874c5d44 100644 --- a/mm/slub.c +++ b/mm/slub.c @@ -1704,12 +1704,6 @@ static bool freelist_corrupted(struct kmem_cache *s, struct slab *slab, * Hooks for other subsystems that check memory allocations. In a typical * production configuration these hooks all should produce no code at all. */ -static __always_inline void kfree_hook(void *x) -{ - kmemleak_free(x); - kasan_kfree_large(x); -} - static __always_inline bool slab_free_hook(struct kmem_cache *s, void *x, bool init) { @@ -3550,19 +3544,6 @@ struct detached_freelist { struct kmem_cache *s; }; -static inline void free_large_kmalloc(struct folio *folio, void *object) -{ - unsigned int order = folio_order(folio); - - if (WARN_ON_ONCE(order == 0)) - pr_warn_once("object pointer: 0x%p\n", object); - - kfree_hook(object); - mod_lruvec_page_state(folio_page(folio, 0), NR_SLAB_UNRECLAIMABLE_B, - -(PAGE_SIZE << order)); - __free_pages(folio_page(folio, 0), order); -} - /* * This function progressively scans the array with free objects (with * a limited look ahead) and extract objects belonging to the same From patchwork Wed Aug 17 10:18:20 2022 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Hyeonggon Yoo <42.hyeyoo@gmail.com> X-Patchwork-Id: 12945690 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from kanga.kvack.org (kanga.kvack.org [205.233.56.17]) by smtp.lore.kernel.org (Postfix) with ESMTP id E7A62C25B08 for ; Wed, 17 Aug 2022 10:19:28 +0000 (UTC) Received: by kanga.kvack.org (Postfix) id 8A4276B007D; Wed, 17 Aug 2022 06:19:28 -0400 (EDT) Received: by kanga.kvack.org (Postfix, from userid 40) id 855B86B007E; Wed, 17 Aug 2022 06:19:28 -0400 (EDT) X-Delivered-To: int-list-linux-mm@kvack.org Received: by kanga.kvack.org (Postfix, from userid 63042) id 6F4D26B0082; Wed, 17 Aug 2022 06:19:28 -0400 (EDT) X-Delivered-To: linux-mm@kvack.org Received: from relay.hostedemail.com (smtprelay0011.hostedemail.com [216.40.44.11]) by kanga.kvack.org (Postfix) with ESMTP id 6147B6B007D for ; Wed, 17 Aug 2022 06:19:28 -0400 (EDT) Received: from smtpin22.hostedemail.com (a10.router.float.18 [10.200.18.1]) by unirelay10.hostedemail.com (Postfix) with ESMTP id 3F1E5C082A for ; Wed, 17 Aug 2022 10:19:28 +0000 (UTC) X-FDA: 79808687616.22.B760F41 Received: from mail-pj1-f42.google.com (mail-pj1-f42.google.com [209.85.216.42]) by imf29.hostedemail.com (Postfix) with ESMTP id E35C31201D9 for ; Wed, 17 Aug 2022 10:19:27 +0000 (UTC) Received: by mail-pj1-f42.google.com with SMTP id o14-20020a17090a0a0e00b001fabfd3369cso46274pjo.5 for ; Wed, 17 Aug 2022 03:19:27 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=20210112; h=content-transfer-encoding:mime-version:references:in-reply-to :message-id:date:subject:cc:to:from:from:to:cc; bh=Daa1VQEO90b1RAILID/2eNYhl4WJ1tJNJRwlSK6+APs=; b=W6PsCrjgbKrvxCBQCQHMSo2WhEZ8f/XuU5argCzxfQbBHYDA1W+hhpS31AosnPk6c3 TElwq517lBeG71MGD7nDV2ZvRWe9isPvVlbvTEil+r7q/kjZ0aNeUrN1juLGdNQyuFJh uJj5MGga4wTp0NxUlS7V7NDJ9eL/w6uQjug0SXLFsnMRn8NhKiFmcu9NFlmAx4G0ntZj dYnxkRsZN1UIAQbOtXGrJpVZF7XG6XG72fY/9/4yvSnFY7NPplZJX6vHi2yNc2ekS7bw qDfKyvCT+s9isO3UuKeGikXmR15NMOykh6JOFOPYQKQS5imqPTpNszSyBQHc1X20QdVh eONA== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20210112; h=content-transfer-encoding:mime-version:references:in-reply-to :message-id:date:subject:cc:to:from:x-gm-message-state:from:to:cc; bh=Daa1VQEO90b1RAILID/2eNYhl4WJ1tJNJRwlSK6+APs=; b=NePjTyPpnhSXashkpwLqrkNJ9U56/Ha0uo/GkhtmWD2S7SulKYVc+TQSgWgHtE+6vm SlvVo9s8bK0b5dnpAXltlIrDHQzwfQUEOkEIjsyiCOH3KZlVaXmBYhZULvncp4BSguHN RcMsf0fBsaTwD4lUY4pGru7Ay1tFroCwm4TzDY3vojoZ1PFD5ARflOVtyOhDqGKTourq P7jPNfScSyLO6zHuFFJ73WtR6NAHvDZNT05h0A0GuJXoPRcr/EgPTllaan2Bo3CE9oEs yhbg4SctfpNlbe0ekrTf0PWo2RfKomUzhHyfS18Mt150zuX6hyVk6fa5MTx2qrIpFmma G4lw== X-Gm-Message-State: ACgBeo0PtqiiLdUliFGsl+xrHbUBcaQ1yAkTpNcScZgiZN7e2su2Re/X hPtfwMTwKx9AyL8NcByHwIE= X-Google-Smtp-Source: AA6agR5NX3zrqI+s7B2wouE/rq8MdzR3qps4dt4XbqfqKpvrdEuq5/VqipgEphbdDE+OOpPjq7+FaQ== X-Received: by 2002:a17:90b:3ecd:b0:1f5:6330:8295 with SMTP id rm13-20020a17090b3ecd00b001f563308295mr2966134pjb.174.1660731567045; Wed, 17 Aug 2022 03:19:27 -0700 (PDT) Received: from hyeyoo.. ([114.29.91.56]) by smtp.gmail.com with ESMTPSA id d8-20020a170903230800b00172633fc236sm1071318plh.174.2022.08.17.03.19.23 (version=TLS1_3 cipher=TLS_AES_256_GCM_SHA384 bits=256/256); Wed, 17 Aug 2022 03:19:26 -0700 (PDT) From: Hyeonggon Yoo <42.hyeyoo@gmail.com> To: Christoph Lameter , Pekka Enberg , David Rientjes , Joonsoo Kim , Andrew Morton , Vlastimil Babka , Roman Gushchin Cc: Hyeonggon Yoo <42.hyeyoo@gmail.com>, linux-mm@kvack.org, linux-kernel@vger.kernel.org Subject: [PATCH v4 11/17] mm/sl[au]b: introduce common alloc/free functions without tracepoint Date: Wed, 17 Aug 2022 19:18:20 +0900 Message-Id: <20220817101826.236819-12-42.hyeyoo@gmail.com> X-Mailer: git-send-email 2.32.0 In-Reply-To: <20220817101826.236819-1-42.hyeyoo@gmail.com> References: <20220817101826.236819-1-42.hyeyoo@gmail.com> MIME-Version: 1.0 ARC-Seal: i=1; s=arc-20220608; d=hostedemail.com; t=1660731567; a=rsa-sha256; cv=none; b=tNJ5OoHkl68I6mSXPX+uy8D7bOC4OOD+gxmeUtF3Zwz73IE24Ot3dS6YqyW1wFNFx9C0xL W96n89VZVrbZFNleuKqOa1oUpeoh5GzNhMiMOZzntHIiIpwd/O/O3A3EXZlKiyrQCunk94 /sXOmvZQ0bbwbluK0JThozhEabYgWCE= ARC-Authentication-Results: i=1; imf29.hostedemail.com; dkim=pass header.d=gmail.com header.s=20210112 header.b=W6PsCrjg; spf=pass (imf29.hostedemail.com: domain of 42.hyeyoo@gmail.com designates 209.85.216.42 as permitted sender) smtp.mailfrom=42.hyeyoo@gmail.com; dmarc=pass (policy=none) header.from=gmail.com ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=hostedemail.com; s=arc-20220608; t=1660731567; h=from:from:sender:reply-to:subject:subject:date:date: message-id:message-id:to:to:cc:cc:mime-version:mime-version: content-type:content-transfer-encoding:content-transfer-encoding: in-reply-to:in-reply-to:references:references:dkim-signature; bh=Daa1VQEO90b1RAILID/2eNYhl4WJ1tJNJRwlSK6+APs=; b=nDBqKUMYdKroV5UgnZfRZuq1AEQWuJjaqmwA500JCs2yeTJB5W697Es0cxi4pq9C03OJZx 1ygtIHf0eK7Ge2xKOEacwECtPLHVEW7HyM9B6YDi/+XXfXSK3Hnv/0jnPNI+pSCsKgx5l5 AU15uWjSZGBiH30qOJ8XRG92eSRJ4g8= X-Stat-Signature: mbhzi6nt5z83mgpm4c9bm919t5t4yynr X-Rspamd-Queue-Id: E35C31201D9 Authentication-Results: imf29.hostedemail.com; dkim=pass header.d=gmail.com header.s=20210112 header.b=W6PsCrjg; spf=pass (imf29.hostedemail.com: domain of 42.hyeyoo@gmail.com designates 209.85.216.42 as permitted sender) smtp.mailfrom=42.hyeyoo@gmail.com; dmarc=pass (policy=none) header.from=gmail.com X-Rspam-User: X-Rspamd-Server: rspam03 X-HE-Tag: 1660731567-676411 X-Bogosity: Ham, tests=bogofilter, spamicity=0.000000, version=1.2.4 Sender: owner-linux-mm@kvack.org Precedence: bulk X-Loop: owner-majordomo@kvack.org List-ID: To unify kmalloc functions in later patch, introduce common alloc/free functions that does not have tracepoint. Signed-off-by: Hyeonggon Yoo <42.hyeyoo@gmail.com> Reviewed-by: Vlastimil Babka --- mm/slab.c | 36 +++++++++++++++++++++++++++++------- mm/slab.h | 4 ++++ mm/slub.c | 13 +++++++++++++ 3 files changed, 46 insertions(+), 7 deletions(-) diff --git a/mm/slab.c b/mm/slab.c index 10c9af904410..aa61851b0a07 100644 --- a/mm/slab.c +++ b/mm/slab.c @@ -3560,6 +3560,14 @@ void *kmem_cache_alloc_node(struct kmem_cache *cachep, gfp_t flags, int nodeid) } EXPORT_SYMBOL(kmem_cache_alloc_node); +void *__kmem_cache_alloc_node(struct kmem_cache *cachep, gfp_t flags, + int nodeid, size_t orig_size, + unsigned long caller) +{ + return slab_alloc_node(cachep, NULL, flags, nodeid, + orig_size, caller); +} + #ifdef CONFIG_TRACING void *kmem_cache_alloc_node_trace(struct kmem_cache *cachep, gfp_t flags, @@ -3645,6 +3653,26 @@ void *__kmalloc(size_t size, gfp_t flags) } EXPORT_SYMBOL(__kmalloc); +static __always_inline +void __do_kmem_cache_free(struct kmem_cache *cachep, void *objp, + unsigned long caller) +{ + unsigned long flags; + + local_irq_save(flags); + debug_check_no_locks_freed(objp, cachep->object_size); + if (!(cachep->flags & SLAB_DEBUG_OBJECTS)) + debug_check_no_obj_freed(objp, cachep->object_size); + __cache_free(cachep, objp, caller); + local_irq_restore(flags); +} + +void __kmem_cache_free(struct kmem_cache *cachep, void *objp, + unsigned long caller) +{ + __do_kmem_cache_free(cachep, objp, caller); +} + /** * kmem_cache_free - Deallocate an object * @cachep: The cache the allocation was from. @@ -3655,18 +3683,12 @@ EXPORT_SYMBOL(__kmalloc); */ void kmem_cache_free(struct kmem_cache *cachep, void *objp) { - unsigned long flags; cachep = cache_from_obj(cachep, objp); if (!cachep) return; trace_kmem_cache_free(_RET_IP_, objp, cachep->name); - local_irq_save(flags); - debug_check_no_locks_freed(objp, cachep->object_size); - if (!(cachep->flags & SLAB_DEBUG_OBJECTS)) - debug_check_no_obj_freed(objp, cachep->object_size); - __cache_free(cachep, objp, _RET_IP_); - local_irq_restore(flags); + __do_kmem_cache_free(cachep, objp, _RET_IP_); } EXPORT_SYMBOL(kmem_cache_free); diff --git a/mm/slab.h b/mm/slab.h index 381ba3e6b2a1..4e90ed0ab635 100644 --- a/mm/slab.h +++ b/mm/slab.h @@ -275,6 +275,10 @@ void create_kmalloc_caches(slab_flags_t); struct kmem_cache *kmalloc_slab(size_t, gfp_t); void *kmalloc_large_node_notrace(size_t size, gfp_t flags, int node); +void *__kmem_cache_alloc_node(struct kmem_cache *s, gfp_t gfpflags, + int node, size_t orig_size, + unsigned long caller); +void __kmem_cache_free(struct kmem_cache *s, void *x, unsigned long caller); #endif gfp_t kmalloc_fix_flags(gfp_t flags); diff --git a/mm/slub.c b/mm/slub.c index a659874c5d44..a11f78c2647c 100644 --- a/mm/slub.c +++ b/mm/slub.c @@ -3262,6 +3262,14 @@ void *kmem_cache_alloc_lru(struct kmem_cache *s, struct list_lru *lru, } EXPORT_SYMBOL(kmem_cache_alloc_lru); +void *__kmem_cache_alloc_node(struct kmem_cache *s, gfp_t gfpflags, + int node, size_t orig_size, + unsigned long caller) +{ + return slab_alloc_node(s, NULL, gfpflags, node, + caller, orig_size); +} + #ifdef CONFIG_TRACING void *kmem_cache_alloc_trace(struct kmem_cache *s, gfp_t gfpflags, size_t size) { @@ -3526,6 +3534,11 @@ void ___cache_free(struct kmem_cache *cache, void *x, unsigned long addr) } #endif +void __kmem_cache_free(struct kmem_cache *s, void *x, unsigned long caller) +{ + slab_free(s, virt_to_slab(x), x, NULL, &x, 1, caller); +} + void kmem_cache_free(struct kmem_cache *s, void *x) { s = cache_from_obj(s, x); From patchwork Wed Aug 17 10:18:21 2022 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Hyeonggon Yoo <42.hyeyoo@gmail.com> X-Patchwork-Id: 12945691 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from kanga.kvack.org (kanga.kvack.org [205.233.56.17]) by smtp.lore.kernel.org (Postfix) with ESMTP id 5C5EEC2BB41 for ; Wed, 17 Aug 2022 10:19:33 +0000 (UTC) Received: by kanga.kvack.org (Postfix) id C9BE16B007E; Wed, 17 Aug 2022 06:19:32 -0400 (EDT) Received: by kanga.kvack.org (Postfix, from userid 40) id C4D5C6B0073; Wed, 17 Aug 2022 06:19:32 -0400 (EDT) X-Delivered-To: int-list-linux-mm@kvack.org Received: by kanga.kvack.org (Postfix, from userid 63042) id A77768D0001; Wed, 17 Aug 2022 06:19:32 -0400 (EDT) X-Delivered-To: linux-mm@kvack.org Received: from relay.hostedemail.com (smtprelay0013.hostedemail.com [216.40.44.13]) by kanga.kvack.org (Postfix) with ESMTP id 987626B007E for ; Wed, 17 Aug 2022 06:19:32 -0400 (EDT) Received: from smtpin21.hostedemail.com (a10.router.float.18 [10.200.18.1]) by unirelay05.hostedemail.com (Postfix) with ESMTP id 6B327407AE for ; Wed, 17 Aug 2022 10:19:32 +0000 (UTC) X-FDA: 79808687784.21.C5D7316 Received: from mail-pj1-f47.google.com (mail-pj1-f47.google.com [209.85.216.47]) by imf04.hostedemail.com (Postfix) with ESMTP id 09D49401B2 for ; Wed, 17 Aug 2022 10:19:31 +0000 (UTC) Received: by mail-pj1-f47.google.com with SMTP id gp7so12074590pjb.4 for ; Wed, 17 Aug 2022 03:19:31 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=20210112; h=content-transfer-encoding:mime-version:references:in-reply-to :message-id:date:subject:cc:to:from:from:to:cc; bh=dz+RrLrwLk+xXgnSRFPm4loOb+4mn5KEX029pojpz/g=; b=EdO5lWBDyEtukF9A+7HE/7DKzG4hdASyYOEZ0XAIe1KW64g4FGZU0PT5I44da9T14r IfuRLAlAV1vA2pc/+jQLC+Uv3+XB+6ENROPFDzWCZItxwMAZW3aKEcignejNQHZnS0w3 VNeiIXv5pqy7syFwbox3b58swjVbRQ/DM0zAQmAmO6b5FbZhZkG4GjSr4DfIK9n//X72 79GBOmBfEyoeThZ2Y+E+smekrYZ2hNpLD1+YdJcM/8ILL1YoeOguDTMgnhsfT2Ey9TfY iwowPTAIsytcKB5vefJo2FvI5tXF+ZS/7TPm1i73WqbCDQ3PuaF/wE/u1ru8Vt0oZjvx jQ6g== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20210112; h=content-transfer-encoding:mime-version:references:in-reply-to :message-id:date:subject:cc:to:from:x-gm-message-state:from:to:cc; bh=dz+RrLrwLk+xXgnSRFPm4loOb+4mn5KEX029pojpz/g=; b=wTjUxbzPx1SlF3PaXxEsgTs4w1rjlNc3aobN0o2CYDErhsqwM7TU9+wu1vRQbgGinY S6pkSCK2pe8o+p8HS67R2NbE/9d4jtxU0g2GPna0hLj9s7fWhj1Xr/N2+NHtcIfGN6yD vRnHWic7OCPyocAGe0S2w7BCcGJSYA6K7RuMwIRpDSCcV5u9aT8010uUZ4DbfNoUOCWm YAQM6NIuG6ykFKIlGGxy5wfN6+m3sBiTGjGSuh7zx+r2gVHCEtBBf+I+Nf4AP5WxBXwE VpEwWxnBEhrypTqBX2nIE5uUfCDQqtrIIPWUEzDP8sC2g31x87Cyt9mmX+i8BHxAxkc4 KTVg== X-Gm-Message-State: ACgBeo3cSCs/NfRh6ZpDx1oO3D0bCTXp0DSwuF3mtakZpEZJYqPGpjDS g7kRQDwXOvg2idVDmVW6oY+zHnE/F0U= X-Google-Smtp-Source: AA6agR6s+Wo7cDyxpJkmcNzbA+ig037E6IsZ/0zdxuBesVy8M5k0aYhgK97VFdOKH8n82GyYTPWJyg== X-Received: by 2002:a17:90b:4b47:b0:1f7:2e06:5752 with SMTP id mi7-20020a17090b4b4700b001f72e065752mr2997195pjb.187.1660731570997; Wed, 17 Aug 2022 03:19:30 -0700 (PDT) Received: from hyeyoo.. ([114.29.91.56]) by smtp.gmail.com with ESMTPSA id d8-20020a170903230800b00172633fc236sm1071318plh.174.2022.08.17.03.19.27 (version=TLS1_3 cipher=TLS_AES_256_GCM_SHA384 bits=256/256); Wed, 17 Aug 2022 03:19:29 -0700 (PDT) From: Hyeonggon Yoo <42.hyeyoo@gmail.com> To: Christoph Lameter , Pekka Enberg , David Rientjes , Joonsoo Kim , Andrew Morton , Vlastimil Babka , Roman Gushchin Cc: Hyeonggon Yoo <42.hyeyoo@gmail.com>, linux-mm@kvack.org, linux-kernel@vger.kernel.org Subject: [PATCH v4 12/17] mm/sl[au]b: generalize kmalloc subsystem Date: Wed, 17 Aug 2022 19:18:21 +0900 Message-Id: <20220817101826.236819-13-42.hyeyoo@gmail.com> X-Mailer: git-send-email 2.32.0 In-Reply-To: <20220817101826.236819-1-42.hyeyoo@gmail.com> References: <20220817101826.236819-1-42.hyeyoo@gmail.com> MIME-Version: 1.0 ARC-Seal: i=1; s=arc-20220608; d=hostedemail.com; t=1660731572; a=rsa-sha256; cv=none; b=4rT3+xEw/D52R6PZ0w6DZCVszql2Hwy8xDaS9wfMrlRHpxS8mu6nM09mlHK7yEZYq6al5t C0L+JU0p6allPJI6veoLJe78dM46RcgdU7Pj0VWrPfEY8/iYDxP5SoBsa3oZhWaNf1LCT0 Qy0/RBOxMAaZk6y6wpQwdJZulvYGKR0= ARC-Authentication-Results: i=1; imf04.hostedemail.com; dkim=pass header.d=gmail.com header.s=20210112 header.b=EdO5lWBD; spf=pass (imf04.hostedemail.com: domain of 42.hyeyoo@gmail.com designates 209.85.216.47 as permitted sender) smtp.mailfrom=42.hyeyoo@gmail.com; dmarc=pass (policy=none) header.from=gmail.com ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=hostedemail.com; s=arc-20220608; t=1660731572; h=from:from:sender:reply-to:subject:subject:date:date: message-id:message-id:to:to:cc:cc:mime-version:mime-version: content-type:content-transfer-encoding:content-transfer-encoding: in-reply-to:in-reply-to:references:references:dkim-signature; bh=dz+RrLrwLk+xXgnSRFPm4loOb+4mn5KEX029pojpz/g=; b=qJw2Ulxw36MLcdVuJo72oA3SKVFeFRWJtUYV4p/QdZP8qwbD8aIUdutBBjBidDjTkk+UQJ T3lEvmViYV9Cjq2vCzBVPFnjwdBn3Puuuiqzvy5ObOnbqEs+h57Km5y4To/olZkgMtoJ+C +fVkkxZ9+Maot28urOJIq4M8lWpzDXk= Authentication-Results: imf04.hostedemail.com; dkim=pass header.d=gmail.com header.s=20210112 header.b=EdO5lWBD; spf=pass (imf04.hostedemail.com: domain of 42.hyeyoo@gmail.com designates 209.85.216.47 as permitted sender) smtp.mailfrom=42.hyeyoo@gmail.com; dmarc=pass (policy=none) header.from=gmail.com X-Rspam-User: X-Stat-Signature: zgxbaf8ud9b4yggbznjqirjoic813g5o X-Rspamd-Queue-Id: 09D49401B2 X-Rspamd-Server: rspam06 X-HE-Tag: 1660731571-191310 X-Bogosity: Ham, tests=bogofilter, spamicity=0.000000, version=1.2.4 Sender: owner-linux-mm@kvack.org Precedence: bulk X-Loop: owner-majordomo@kvack.org List-ID: Now everything in kmalloc subsystem can be generalized. Let's do it! Generalize __do_kmalloc_node(), __kmalloc_node_track_caller(), kfree(), __ksize(), __kmalloc(), __kmalloc_node() and move them to slab_common.c. In the meantime, rename kmalloc_large_node_notrace() to __kmalloc_large_node() and make it static as it's now only called in slab_common.c. Signed-off-by: Hyeonggon Yoo <42.hyeyoo@gmail.com> Reviewed-by: Vlastimil Babka --- mm/slab.c | 108 ---------------------------------------------- mm/slab.h | 1 - mm/slab_common.c | 109 +++++++++++++++++++++++++++++++++++++++++++++-- mm/slub.c | 87 ------------------------------------- 4 files changed, 106 insertions(+), 199 deletions(-) diff --git a/mm/slab.c b/mm/slab.c index aa61851b0a07..5b234e3ab165 100644 --- a/mm/slab.c +++ b/mm/slab.c @@ -3587,44 +3587,6 @@ void *kmem_cache_alloc_node_trace(struct kmem_cache *cachep, EXPORT_SYMBOL(kmem_cache_alloc_node_trace); #endif -static __always_inline void * -__do_kmalloc_node(size_t size, gfp_t flags, int node, unsigned long caller) -{ - struct kmem_cache *cachep; - void *ret; - - if (unlikely(size > KMALLOC_MAX_CACHE_SIZE)) { - ret = kmalloc_large_node_notrace(size, flags, node); - - trace_kmalloc_node(caller, ret, NULL, size, - PAGE_SIZE << get_order(size), - flags, node); - return ret; - } - - cachep = kmalloc_slab(size, flags); - if (unlikely(ZERO_OR_NULL_PTR(cachep))) - return cachep; - - ret = kmem_cache_alloc_node_trace(cachep, flags, node, size); - ret = kasan_kmalloc(cachep, ret, size, flags); - - return ret; -} - -void *__kmalloc_node(size_t size, gfp_t flags, int node) -{ - return __do_kmalloc_node(size, flags, node, _RET_IP_); -} -EXPORT_SYMBOL(__kmalloc_node); - -void *__kmalloc_node_track_caller(size_t size, gfp_t flags, - int node, unsigned long caller) -{ - return __do_kmalloc_node(size, flags, node, caller); -} -EXPORT_SYMBOL(__kmalloc_node_track_caller); - #ifdef CONFIG_PRINTK void __kmem_obj_info(struct kmem_obj_info *kpp, void *object, struct slab *slab) { @@ -3647,12 +3609,6 @@ void __kmem_obj_info(struct kmem_obj_info *kpp, void *object, struct slab *slab) } #endif -void *__kmalloc(size_t size, gfp_t flags) -{ - return __do_kmalloc_node(size, flags, NUMA_NO_NODE, _RET_IP_); -} -EXPORT_SYMBOL(__kmalloc); - static __always_inline void __do_kmem_cache_free(struct kmem_cache *cachep, void *objp, unsigned long caller) @@ -3730,43 +3686,6 @@ void kmem_cache_free_bulk(struct kmem_cache *orig_s, size_t size, void **p) } EXPORT_SYMBOL(kmem_cache_free_bulk); -/** - * kfree - free previously allocated memory - * @objp: pointer returned by kmalloc. - * - * If @objp is NULL, no operation is performed. - * - * Don't free memory not originally allocated by kmalloc() - * or you will run into trouble. - */ -void kfree(const void *objp) -{ - struct kmem_cache *c; - unsigned long flags; - struct folio *folio; - - trace_kfree(_RET_IP_, objp); - - if (unlikely(ZERO_OR_NULL_PTR(objp))) - return; - - folio = virt_to_folio(objp); - if (!folio_test_slab(folio)) { - free_large_kmalloc(folio, (void *)objp); - return; - } - - c = folio_slab(folio)->slab_cache; - - local_irq_save(flags); - kfree_debugcheck(objp); - debug_check_no_locks_freed(objp, c->object_size); - debug_check_no_obj_freed(objp, c->object_size); - __cache_free(c, (void *)objp, _RET_IP_); - local_irq_restore(flags); -} -EXPORT_SYMBOL(kfree); - /* * This initializes kmem_cache_node or resizes various caches for all nodes. */ @@ -4169,30 +4088,3 @@ void __check_heap_object(const void *ptr, unsigned long n, usercopy_abort("SLAB object", cachep->name, to_user, offset, n); } #endif /* CONFIG_HARDENED_USERCOPY */ - -/** - * __ksize -- Uninstrumented ksize. - * @objp: pointer to the object - * - * Unlike ksize(), __ksize() is uninstrumented, and does not provide the same - * safety checks as ksize() with KASAN instrumentation enabled. - * - * Return: size of the actual memory used by @objp in bytes - */ -size_t __ksize(const void *objp) -{ - struct kmem_cache *c; - struct folio *folio; - - BUG_ON(!objp); - if (unlikely(objp == ZERO_SIZE_PTR)) - return 0; - - folio = virt_to_folio(objp); - if (!folio_test_slab(folio)) - return folio_size(folio); - - c = folio_slab(folio)->slab_cache; - return c->object_size; -} -EXPORT_SYMBOL(__ksize); diff --git a/mm/slab.h b/mm/slab.h index 4e90ed0ab635..4d8330d57573 100644 --- a/mm/slab.h +++ b/mm/slab.h @@ -274,7 +274,6 @@ void create_kmalloc_caches(slab_flags_t); /* Find the kmalloc slab corresponding for a certain size */ struct kmem_cache *kmalloc_slab(size_t, gfp_t); -void *kmalloc_large_node_notrace(size_t size, gfp_t flags, int node); void *__kmem_cache_alloc_node(struct kmem_cache *s, gfp_t gfpflags, int node, size_t orig_size, unsigned long caller); diff --git a/mm/slab_common.c b/mm/slab_common.c index 5a2e81f42ee9..c8242b4e2223 100644 --- a/mm/slab_common.c +++ b/mm/slab_common.c @@ -897,6 +897,109 @@ void free_large_kmalloc(struct folio *folio, void *object) -(PAGE_SIZE << order)); __free_pages(folio_page(folio, 0), order); } + +static void *__kmalloc_large_node(size_t size, gfp_t flags, int node); +static __always_inline +void *__do_kmalloc_node(size_t size, gfp_t flags, int node, unsigned long caller) +{ + struct kmem_cache *s; + void *ret; + + if (unlikely(size > KMALLOC_MAX_CACHE_SIZE)) { + ret = __kmalloc_large_node(size, flags, node); + trace_kmalloc_node(caller, ret, NULL, + size, PAGE_SIZE << get_order(size), + flags, node); + return ret; + } + + s = kmalloc_slab(size, flags); + + if (unlikely(ZERO_OR_NULL_PTR(s))) + return s; + + ret = __kmem_cache_alloc_node(s, flags, node, size, caller); + ret = kasan_kmalloc(s, ret, size, flags); + trace_kmalloc_node(caller, ret, s, size, + s->size, flags, node); + return ret; +} + +void *__kmalloc_node(size_t size, gfp_t flags, int node) +{ + return __do_kmalloc_node(size, flags, node, _RET_IP_); +} +EXPORT_SYMBOL(__kmalloc_node); + +void *__kmalloc(size_t size, gfp_t flags) +{ + return __do_kmalloc_node(size, flags, NUMA_NO_NODE, _RET_IP_); +} +EXPORT_SYMBOL(__kmalloc); + +void *__kmalloc_node_track_caller(size_t size, gfp_t flags, + int node, unsigned long caller) +{ + return __do_kmalloc_node(size, flags, node, caller); +} +EXPORT_SYMBOL(__kmalloc_node_track_caller); + +/** + * kfree - free previously allocated memory + * @objp: pointer returned by kmalloc. + * + * If @objp is NULL, no operation is performed. + * + * Don't free memory not originally allocated by kmalloc() + * or you will run into trouble. + */ +void kfree(const void *object) +{ + struct folio *folio; + struct slab *slab; + struct kmem_cache *s; + + trace_kfree(_RET_IP_, object); + + if (unlikely(ZERO_OR_NULL_PTR(object))) + return; + + folio = virt_to_folio(object); + if (unlikely(!folio_test_slab(folio))) { + free_large_kmalloc(folio, (void *)object); + return; + } + + slab = folio_slab(folio); + s = slab->slab_cache; + __kmem_cache_free(s, (void *)object, _RET_IP_); +} +EXPORT_SYMBOL(kfree); + +/** + * __ksize -- Uninstrumented ksize. + * @objp: pointer to the object + * + * Unlike ksize(), __ksize() is uninstrumented, and does not provide the same + * safety checks as ksize() with KASAN instrumentation enabled. + * + * Return: size of the actual memory used by @objp in bytes + */ +size_t __ksize(const void *object) +{ + struct folio *folio; + + if (unlikely(object == ZERO_SIZE_PTR)) + return 0; + + folio = virt_to_folio(object); + + if (unlikely(!folio_test_slab(folio))) + return folio_size(folio); + + return slab_ksize(folio_slab(folio)->slab_cache); +} +EXPORT_SYMBOL(__ksize); #endif /* !CONFIG_SLOB */ gfp_t kmalloc_fix_flags(gfp_t flags) @@ -917,7 +1020,7 @@ gfp_t kmalloc_fix_flags(gfp_t flags) * know the allocation order to free the pages properly in kfree. */ -void *kmalloc_large_node_notrace(size_t size, gfp_t flags, int node) +void *__kmalloc_large_node(size_t size, gfp_t flags, int node) { struct page *page; void *ptr = NULL; @@ -943,7 +1046,7 @@ void *kmalloc_large_node_notrace(size_t size, gfp_t flags, int node) void *kmalloc_large(size_t size, gfp_t flags) { - void *ret = kmalloc_large_node_notrace(size, flags, NUMA_NO_NODE); + void *ret = __kmalloc_large_node(size, flags, NUMA_NO_NODE); trace_kmalloc(_RET_IP_, ret, NULL, size, PAGE_SIZE << get_order(size), flags); @@ -953,7 +1056,7 @@ EXPORT_SYMBOL(kmalloc_large); void *kmalloc_large_node(size_t size, gfp_t flags, int node) { - void *ret = kmalloc_large_node_notrace(size, flags, node); + void *ret = __kmalloc_large_node(size, flags, node); trace_kmalloc_node(_RET_IP_, ret, NULL, size, PAGE_SIZE << get_order(size), flags, node); diff --git a/mm/slub.c b/mm/slub.c index a11f78c2647c..cd49785d59e1 100644 --- a/mm/slub.c +++ b/mm/slub.c @@ -4388,49 +4388,6 @@ static int __init setup_slub_min_objects(char *str) __setup("slub_min_objects=", setup_slub_min_objects); -static __always_inline -void *__do_kmalloc_node(size_t size, gfp_t flags, int node, unsigned long caller) -{ - struct kmem_cache *s; - void *ret; - - if (unlikely(size > KMALLOC_MAX_CACHE_SIZE)) { - ret = kmalloc_large_node_notrace(size, flags, node); - - trace_kmalloc_node(caller, ret, NULL, - size, PAGE_SIZE << get_order(size), - flags, node); - - return ret; - } - - s = kmalloc_slab(size, flags); - - if (unlikely(ZERO_OR_NULL_PTR(s))) - return s; - - ret = slab_alloc_node(s, NULL, flags, node, caller, size); - - trace_kmalloc_node(caller, ret, s, size, s->size, flags, node); - - ret = kasan_kmalloc(s, ret, size, flags); - - return ret; -} - -void *__kmalloc_node(size_t size, gfp_t flags, int node) -{ - return __do_kmalloc_node(size, flags, node, _RET_IP_); -} -EXPORT_SYMBOL(__kmalloc_node); - -void *__kmalloc(size_t size, gfp_t flags) -{ - return __do_kmalloc_node(size, flags, NUMA_NO_NODE, _RET_IP_); -} -EXPORT_SYMBOL(__kmalloc); - - #ifdef CONFIG_HARDENED_USERCOPY /* * Rejects incorrectly sized objects and objects that are to be copied @@ -4481,43 +4438,6 @@ void __check_heap_object(const void *ptr, unsigned long n, } #endif /* CONFIG_HARDENED_USERCOPY */ -size_t __ksize(const void *object) -{ - struct folio *folio; - - if (unlikely(object == ZERO_SIZE_PTR)) - return 0; - - folio = virt_to_folio(object); - - if (unlikely(!folio_test_slab(folio))) - return folio_size(folio); - - return slab_ksize(folio_slab(folio)->slab_cache); -} -EXPORT_SYMBOL(__ksize); - -void kfree(const void *x) -{ - struct folio *folio; - struct slab *slab; - void *object = (void *)x; - - trace_kfree(_RET_IP_, x); - - if (unlikely(ZERO_OR_NULL_PTR(x))) - return; - - folio = virt_to_folio(x); - if (unlikely(!folio_test_slab(folio))) { - free_large_kmalloc(folio, object); - return; - } - slab = folio_slab(folio); - slab_free(slab->slab_cache, slab, object, NULL, &object, 1, _RET_IP_); -} -EXPORT_SYMBOL(kfree); - #define SHRINK_PROMOTE_MAX 32 /* @@ -4863,13 +4783,6 @@ int __kmem_cache_create(struct kmem_cache *s, slab_flags_t flags) return 0; } -void *__kmalloc_node_track_caller(size_t size, gfp_t gfpflags, - int node, unsigned long caller) -{ - return __do_kmalloc_node(size, gfpflags, node, caller); -} -EXPORT_SYMBOL(__kmalloc_node_track_caller); - #ifdef CONFIG_SYSFS static int count_inuse(struct slab *slab) { From patchwork Wed Aug 17 10:18:22 2022 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Hyeonggon Yoo <42.hyeyoo@gmail.com> X-Patchwork-Id: 12945692 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from kanga.kvack.org (kanga.kvack.org [205.233.56.17]) by smtp.lore.kernel.org (Postfix) with ESMTP id B0846C25B08 for ; Wed, 17 Aug 2022 10:19:36 +0000 (UTC) Received: by kanga.kvack.org (Postfix) id 5224E6B0073; Wed, 17 Aug 2022 06:19:36 -0400 (EDT) Received: by kanga.kvack.org (Postfix, from userid 40) id 4D1C06B0081; Wed, 17 Aug 2022 06:19:36 -0400 (EDT) X-Delivered-To: int-list-linux-mm@kvack.org Received: by kanga.kvack.org (Postfix, from userid 63042) id 39A078D0001; Wed, 17 Aug 2022 06:19:36 -0400 (EDT) X-Delivered-To: linux-mm@kvack.org Received: from relay.hostedemail.com (smtprelay0011.hostedemail.com [216.40.44.11]) by kanga.kvack.org (Postfix) with ESMTP id 2B4806B0073 for ; Wed, 17 Aug 2022 06:19:36 -0400 (EDT) Received: from smtpin05.hostedemail.com (a10.router.float.18 [10.200.18.1]) by unirelay09.hostedemail.com (Postfix) with ESMTP id 0BF02807B4 for ; Wed, 17 Aug 2022 10:19:36 +0000 (UTC) X-FDA: 79808687952.05.42AAC9E Received: from mail-pl1-f182.google.com (mail-pl1-f182.google.com [209.85.214.182]) by imf13.hostedemail.com (Postfix) with ESMTP id A2B37201CE for ; Wed, 17 Aug 2022 10:19:35 +0000 (UTC) Received: by mail-pl1-f182.google.com with SMTP id m2so11643562pls.4 for ; Wed, 17 Aug 2022 03:19:35 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=20210112; h=content-transfer-encoding:mime-version:references:in-reply-to :message-id:date:subject:cc:to:from:from:to:cc; bh=ec/2ehXn71Tv+L2UuMNScuaYO4eN7Mil3n5xQnD/usM=; b=NE0QMyh9rN6jdYaOFqchLIWssHGMXYg5VraHfgM7OQRM+zrFJ0sZt1cK5vRqGOJIjs 2sumSFJ9OzlmGQJknoSvF79MKY7HzOVwM+FMBsmM/NtUaYtVtJah0x/V08MPUBJu9G0r USMvaDN/IxXCi+icvzS/yp1yXqo25FI3uXLQuapSGVQAS9dq/PLRMO7builXKV3vcD6U Bfs1iFowef9AILGtXjTVAd9///7WkTRoBf4C0iDUl8UJmAbbf3gKs+1JPrsmgbSng7lH uHZys+ZJYWdTwKwL7D74AL4ZVVEa1WlVDGTTUFtctN72UkextK+408oGudwHSofmY0Ly mnmA== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20210112; h=content-transfer-encoding:mime-version:references:in-reply-to :message-id:date:subject:cc:to:from:x-gm-message-state:from:to:cc; bh=ec/2ehXn71Tv+L2UuMNScuaYO4eN7Mil3n5xQnD/usM=; b=l821OadkEQ4f2t0DuHQEvGficTc7DM0un0koxoBagH07LcUmrCXsTPvhG2YKEDSRY7 IqHbURnRH+k7oQf4WlOg2iI1vrZNVY2WWa6/b+po7VH6LJbPE8/g8Ml8apXKGfs8aJK5 LDELSbwoE3xXMv9wpIajNgtaOas2iBJpc1ehM+U9iipkxbQjcbRSNjf6GnUpKbTf9u8s IWUoqXFT8ifDy16zlbXKpi3DTKwyF+3+dYyZIOaATKjuBPzlK7zMcS85S2al4dIk551b d1NiidUfsYiZHAUG/jf4ZKhRYVBqKOJLyga6AZJw2zH9uz3sTDjIZXg1RycjyrmWLyLV KWNQ== X-Gm-Message-State: ACgBeo3KrGdFocC/eQAsLmTd0uxe9PfpppkSQ6YmjWiFaK2uQxAHzSAS PMwrh591iwuo9oeOC6Lfj9BLCLJvDhQ= X-Google-Smtp-Source: AA6agR42HTFa0hi7Z2W2UpfZGw0BiJjZK6mjVfVq8nVA2E4qeRRVezYO2uWKdvXuyzgFXyIJ29hflA== X-Received: by 2002:a17:90b:17c9:b0:1f3:3a7c:a3a7 with SMTP id me9-20020a17090b17c900b001f33a7ca3a7mr3137221pjb.76.1660731574780; Wed, 17 Aug 2022 03:19:34 -0700 (PDT) Received: from hyeyoo.. ([114.29.91.56]) by smtp.gmail.com with ESMTPSA id d8-20020a170903230800b00172633fc236sm1071318plh.174.2022.08.17.03.19.31 (version=TLS1_3 cipher=TLS_AES_256_GCM_SHA384 bits=256/256); Wed, 17 Aug 2022 03:19:33 -0700 (PDT) From: Hyeonggon Yoo <42.hyeyoo@gmail.com> To: Christoph Lameter , Pekka Enberg , David Rientjes , Joonsoo Kim , Andrew Morton , Vlastimil Babka , Roman Gushchin Cc: Hyeonggon Yoo <42.hyeyoo@gmail.com>, linux-mm@kvack.org, linux-kernel@vger.kernel.org Subject: [PATCH v4 13/17] mm/sl[au]b: cleanup kmem_cache_alloc[_node]_trace() Date: Wed, 17 Aug 2022 19:18:22 +0900 Message-Id: <20220817101826.236819-14-42.hyeyoo@gmail.com> X-Mailer: git-send-email 2.32.0 In-Reply-To: <20220817101826.236819-1-42.hyeyoo@gmail.com> References: <20220817101826.236819-1-42.hyeyoo@gmail.com> MIME-Version: 1.0 ARC-Seal: i=1; s=arc-20220608; d=hostedemail.com; t=1660731575; a=rsa-sha256; cv=none; b=2Utz3x2juHS8Zz8w8NAftw30S+mnEvZVlY1ifgKLV97GkkQMqtwVY/KwKO0ElEAY9dwrXy RpR3Vxu+J6PAkBdzR90+493W02jivQUweQs5+vK2oM/5XTmsy4Sazz2aorm5oe9B2wXCIP Ihy60QV33yDzA45sXFV18f8b6biOEXQ= ARC-Authentication-Results: i=1; imf13.hostedemail.com; dkim=pass header.d=gmail.com header.s=20210112 header.b=NE0QMyh9; spf=pass (imf13.hostedemail.com: domain of 42.hyeyoo@gmail.com designates 209.85.214.182 as permitted sender) smtp.mailfrom=42.hyeyoo@gmail.com; dmarc=pass (policy=none) header.from=gmail.com ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=hostedemail.com; s=arc-20220608; t=1660731575; h=from:from:sender:reply-to:subject:subject:date:date: message-id:message-id:to:to:cc:cc:mime-version:mime-version: content-type:content-transfer-encoding:content-transfer-encoding: in-reply-to:in-reply-to:references:references:dkim-signature; bh=ec/2ehXn71Tv+L2UuMNScuaYO4eN7Mil3n5xQnD/usM=; b=oN7TBdA3myBP81de2UX2e8+xvH5rEM5+LnfP3A665pHxkIELgZvfvhmDVp/rJ9dOKY0wpV X5bosqBXKVyaIx2wfYkPoNJDi/S7TW/XJvE/vuYSUT+jcfPh6TMhld/KULmCfQiVnkg8Pg 0vD2kaxdUJXGf1Zm0GHP+cgKPuZsTY0= Authentication-Results: imf13.hostedemail.com; dkim=pass header.d=gmail.com header.s=20210112 header.b=NE0QMyh9; spf=pass (imf13.hostedemail.com: domain of 42.hyeyoo@gmail.com designates 209.85.214.182 as permitted sender) smtp.mailfrom=42.hyeyoo@gmail.com; dmarc=pass (policy=none) header.from=gmail.com X-Rspamd-Server: rspam02 X-Rspamd-Queue-Id: A2B37201CE X-Stat-Signature: 7dmmrjbr7c1k45jacnb8duutiyh5i6dn X-Rspam-User: X-HE-Tag: 1660731575-619800 X-Bogosity: Ham, tests=bogofilter, spamicity=0.000000, version=1.2.4 Sender: owner-linux-mm@kvack.org Precedence: bulk X-Loop: owner-majordomo@kvack.org List-ID: This patch does: - Despite its name, kmem_cache_alloc[_node]_trace() is hook for inlined kmalloc. So rename it to kmalloc[_node]_trace(). - Move its implementation to slab_common.c by using __kmem_cache_alloc_node(), but keep CONFIG_TRACING=n varients to save a function call when CONFIG_TRACING=n. - Use __assume_kmalloc_alignment for kmalloc[_node]_trace instead of __assume_slab_alignement. Generally kmalloc has larger alignment requirements. Suggested-by: Vlastimil Babka Signed-off-by: Hyeonggon Yoo <42.hyeyoo@gmail.com> Reviewed-by: Vlastimil Babka --- include/linux/slab.h | 27 ++++++++++++++------------- mm/slab.c | 35 ----------------------------------- mm/slab_common.c | 27 +++++++++++++++++++++++++++ mm/slub.c | 27 --------------------------- 4 files changed, 41 insertions(+), 75 deletions(-) diff --git a/include/linux/slab.h b/include/linux/slab.h index 4ee5b2fed164..c8e485ce8815 100644 --- a/include/linux/slab.h +++ b/include/linux/slab.h @@ -449,16 +449,16 @@ void *kmem_cache_alloc_node(struct kmem_cache *s, gfp_t flags, int node) __assum __malloc; #ifdef CONFIG_TRACING -extern void *kmem_cache_alloc_trace(struct kmem_cache *s, gfp_t flags, size_t size) - __assume_slab_alignment __alloc_size(3); - -extern void *kmem_cache_alloc_node_trace(struct kmem_cache *s, gfp_t gfpflags, - int node, size_t size) __assume_slab_alignment - __alloc_size(4); +void *kmalloc_trace(struct kmem_cache *s, gfp_t flags, size_t size) + __assume_kmalloc_alignment __alloc_size(3); +void *kmalloc_node_trace(struct kmem_cache *s, gfp_t gfpflags, + int node, size_t size) __assume_kmalloc_alignment + __alloc_size(4); #else /* CONFIG_TRACING */ -static __always_inline __alloc_size(3) void *kmem_cache_alloc_trace(struct kmem_cache *s, - gfp_t flags, size_t size) +/* Save a function call when CONFIG_TRACING=n */ +static __always_inline __alloc_size(3) +void *kmalloc_trace(struct kmem_cache *s, gfp_t flags, size_t size) { void *ret = kmem_cache_alloc(s, flags); @@ -466,8 +466,9 @@ static __always_inline __alloc_size(3) void *kmem_cache_alloc_trace(struct kmem_ return ret; } -static __always_inline void *kmem_cache_alloc_node_trace(struct kmem_cache *s, gfp_t gfpflags, - int node, size_t size) +static __always_inline __alloc_size(4) +void *kmalloc_node_trace(struct kmem_cache *s, gfp_t gfpflags, + int node, size_t size) { void *ret = kmem_cache_alloc_node(s, gfpflags, node); @@ -550,7 +551,7 @@ static __always_inline __alloc_size(1) void *kmalloc(size_t size, gfp_t flags) if (!index) return ZERO_SIZE_PTR; - return kmem_cache_alloc_trace( + return kmalloc_trace( kmalloc_caches[kmalloc_type(flags)][index], flags, size); #endif @@ -572,9 +573,9 @@ static __always_inline __alloc_size(1) void *kmalloc_node(size_t size, gfp_t fla if (!index) return ZERO_SIZE_PTR; - return kmem_cache_alloc_node_trace( + return kmalloc_node_trace( kmalloc_caches[kmalloc_type(flags)][index], - flags, node, size); + flags, node, size); } return __kmalloc_node(size, flags, node); } diff --git a/mm/slab.c b/mm/slab.c index 5b234e3ab165..8d9d0fbf9792 100644 --- a/mm/slab.c +++ b/mm/slab.c @@ -3519,22 +3519,6 @@ int kmem_cache_alloc_bulk(struct kmem_cache *s, gfp_t flags, size_t size, } EXPORT_SYMBOL(kmem_cache_alloc_bulk); -#ifdef CONFIG_TRACING -void * -kmem_cache_alloc_trace(struct kmem_cache *cachep, gfp_t flags, size_t size) -{ - void *ret; - - ret = slab_alloc(cachep, NULL, flags, size, _RET_IP_); - - ret = kasan_kmalloc(cachep, ret, size, flags); - trace_kmalloc(_RET_IP_, ret, cachep, - size, cachep->size, flags); - return ret; -} -EXPORT_SYMBOL(kmem_cache_alloc_trace); -#endif - /** * kmem_cache_alloc_node - Allocate an object on the specified node * @cachep: The cache to allocate from. @@ -3568,25 +3552,6 @@ void *__kmem_cache_alloc_node(struct kmem_cache *cachep, gfp_t flags, orig_size, caller); } -#ifdef CONFIG_TRACING -void *kmem_cache_alloc_node_trace(struct kmem_cache *cachep, - gfp_t flags, - int nodeid, - size_t size) -{ - void *ret; - - ret = slab_alloc_node(cachep, NULL, flags, nodeid, size, _RET_IP_); - - ret = kasan_kmalloc(cachep, ret, size, flags); - trace_kmalloc_node(_RET_IP_, ret, cachep, - size, cachep->size, - flags, nodeid); - return ret; -} -EXPORT_SYMBOL(kmem_cache_alloc_node_trace); -#endif - #ifdef CONFIG_PRINTK void __kmem_obj_info(struct kmem_obj_info *kpp, void *object, struct slab *slab) { diff --git a/mm/slab_common.c b/mm/slab_common.c index c8242b4e2223..d8e8c41c12f1 100644 --- a/mm/slab_common.c +++ b/mm/slab_common.c @@ -1000,6 +1000,33 @@ size_t __ksize(const void *object) return slab_ksize(folio_slab(folio)->slab_cache); } EXPORT_SYMBOL(__ksize); + +#ifdef CONFIG_TRACING +void *kmalloc_trace(struct kmem_cache *s, gfp_t gfpflags, size_t size) +{ + void *ret = __kmem_cache_alloc_node(s, gfpflags, NUMA_NO_NODE, + size, _RET_IP_); + + trace_kmalloc_node(_RET_IP_, ret, s, size, s->size, + gfpflags, NUMA_NO_NODE); + + ret = kasan_kmalloc(s, ret, size, gfpflags); + return ret; +} +EXPORT_SYMBOL(kmalloc_trace); + +void *kmalloc_node_trace(struct kmem_cache *s, gfp_t gfpflags, + int node, size_t size) +{ + void *ret = __kmem_cache_alloc_node(s, gfpflags, node, size, _RET_IP_); + + trace_kmalloc_node(_RET_IP_, ret, s, size, s->size, gfpflags, node); + + ret = kasan_kmalloc(s, ret, size, gfpflags); + return ret; +} +EXPORT_SYMBOL(kmalloc_node_trace); +#endif /* !CONFIG_TRACING */ #endif /* !CONFIG_SLOB */ gfp_t kmalloc_fix_flags(gfp_t flags) diff --git a/mm/slub.c b/mm/slub.c index cd49785d59e1..7d7fd9d4e8fa 100644 --- a/mm/slub.c +++ b/mm/slub.c @@ -3270,17 +3270,6 @@ void *__kmem_cache_alloc_node(struct kmem_cache *s, gfp_t gfpflags, caller, orig_size); } -#ifdef CONFIG_TRACING -void *kmem_cache_alloc_trace(struct kmem_cache *s, gfp_t gfpflags, size_t size) -{ - void *ret = slab_alloc(s, NULL, gfpflags, _RET_IP_, size); - trace_kmalloc(_RET_IP_, ret, s, size, s->size, gfpflags); - ret = kasan_kmalloc(s, ret, size, gfpflags); - return ret; -} -EXPORT_SYMBOL(kmem_cache_alloc_trace); -#endif - void *kmem_cache_alloc_node(struct kmem_cache *s, gfp_t gfpflags, int node) { void *ret = slab_alloc_node(s, NULL, gfpflags, node, _RET_IP_, s->object_size); @@ -3292,22 +3281,6 @@ void *kmem_cache_alloc_node(struct kmem_cache *s, gfp_t gfpflags, int node) } EXPORT_SYMBOL(kmem_cache_alloc_node); -#ifdef CONFIG_TRACING -void *kmem_cache_alloc_node_trace(struct kmem_cache *s, - gfp_t gfpflags, - int node, size_t size) -{ - void *ret = slab_alloc_node(s, NULL, gfpflags, node, _RET_IP_, size); - - trace_kmalloc_node(_RET_IP_, ret, s, - size, s->size, gfpflags, node); - - ret = kasan_kmalloc(s, ret, size, gfpflags); - return ret; -} -EXPORT_SYMBOL(kmem_cache_alloc_node_trace); -#endif - /* * Slow path handling. This may still be called frequently since objects * have a longer lifetime than the cpu slabs in most processing loads. From patchwork Wed Aug 17 10:18:23 2022 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Hyeonggon Yoo <42.hyeyoo@gmail.com> X-Patchwork-Id: 12945693 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from kanga.kvack.org (kanga.kvack.org [205.233.56.17]) by smtp.lore.kernel.org (Postfix) with ESMTP id 903D8C25B08 for ; Wed, 17 Aug 2022 10:19:40 +0000 (UTC) Received: by kanga.kvack.org (Postfix) id 2FEF96B0078; Wed, 17 Aug 2022 06:19:40 -0400 (EDT) Received: by kanga.kvack.org (Postfix, from userid 40) id 288FB6B0081; Wed, 17 Aug 2022 06:19:40 -0400 (EDT) X-Delivered-To: int-list-linux-mm@kvack.org Received: by kanga.kvack.org (Postfix, from userid 63042) id 1017A8D0001; Wed, 17 Aug 2022 06:19:40 -0400 (EDT) X-Delivered-To: linux-mm@kvack.org Received: from relay.hostedemail.com (smtprelay0016.hostedemail.com [216.40.44.16]) by kanga.kvack.org (Postfix) with ESMTP id 027AE6B0078 for ; Wed, 17 Aug 2022 06:19:40 -0400 (EDT) Received: from smtpin23.hostedemail.com (a10.router.float.18 [10.200.18.1]) by unirelay09.hostedemail.com (Postfix) with ESMTP id C680C807B4 for ; Wed, 17 Aug 2022 10:19:39 +0000 (UTC) X-FDA: 79808688078.23.8BB7841 Received: from mail-pl1-f182.google.com (mail-pl1-f182.google.com [209.85.214.182]) by imf03.hostedemail.com (Postfix) with ESMTP id 62D4F201D4 for ; Wed, 17 Aug 2022 10:19:39 +0000 (UTC) Received: by mail-pl1-f182.google.com with SMTP id 20so3029859plo.10 for ; Wed, 17 Aug 2022 03:19:39 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=20210112; h=content-transfer-encoding:mime-version:references:in-reply-to :message-id:date:subject:cc:to:from:from:to:cc; bh=CyypfYVW40Zxy99wL/B3ZBuo0BqIs8lbgANQnRCbVC4=; b=VUPnz/D4yt281vtciG3zIkCG8wzFEAcCtpanFMktSXNw3J+m8nk9VhKsXldatbEuVk 7veCFdOCyM5L9juZ7+zaNNDNbCjir0zuhAChAv4ze6fHgRbYNrEp3oGaQe/U+Sf+d5pt Yy77QuS5zhR6dRA2l8LZNj5iKE52sp0+6qq2dWEJQDCtO3kbgU9yuiKEqLRhQ36QXUvm cULrJbg6FjbXLKN6MOoYW7eIV6hEb5Cl7DuKhA/L7MymZs1HFm2qm8JBfheXLZsYkKbI 3wuFAByfXclzWmoQct96dE06Q21O52/t2mBW2qED00il5VbqLfUSj0ljRzMRYFcLB7DR P65Q== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20210112; h=content-transfer-encoding:mime-version:references:in-reply-to :message-id:date:subject:cc:to:from:x-gm-message-state:from:to:cc; bh=CyypfYVW40Zxy99wL/B3ZBuo0BqIs8lbgANQnRCbVC4=; b=we+pVS1FQfOSUzwOQG5UbJKZqg3azUuwExR+nxk4q/DRdCNcug6CACgxZKXLhT0S+4 J9ctqmt2SpEAcyzTwdho0++TXThvOXGs1BpBHOK1wmdoGJbVnj7cILerq/zwq8pCZMxp KeP2r9AIkJu/HFVPYGuxtP4+ee0AsO4zawqE1oc0DAx4z5jTp7MTtHMTq60U/2pzOy9T odIpHB2lVtaN2EZksjyeCBGSE5fTiJ3ESQgcqsQSElbcHYnv7m6RaWSVWWQjAJHqYyvv 8ce9FvKHPaO7NikgC/nIf0a6lHlcqR9QMokhV/JlO4fGpdJdC2a/Ao6qQu+k0LKL1PPb BMQA== X-Gm-Message-State: ACgBeo27dFbCwzB09T5w8zE8s/y7n+WwB3pupQmMtAaPii7apOglthV2 7pm5SwIOu4EwrCP1iHvZx4c= X-Google-Smtp-Source: AA6agR4ltHmPKihrwg6pI9XNaqqOJ7qEKaijv1U195SEBQTrbkQ2dpNmYsxi6ciIdPMPJQ90fxJE7g== X-Received: by 2002:a17:90b:390c:b0:1f5:5bbc:2e8 with SMTP id ob12-20020a17090b390c00b001f55bbc02e8mr3047899pjb.233.1660731578454; Wed, 17 Aug 2022 03:19:38 -0700 (PDT) Received: from hyeyoo.. ([114.29.91.56]) by smtp.gmail.com with ESMTPSA id d8-20020a170903230800b00172633fc236sm1071318plh.174.2022.08.17.03.19.35 (version=TLS1_3 cipher=TLS_AES_256_GCM_SHA384 bits=256/256); Wed, 17 Aug 2022 03:19:37 -0700 (PDT) From: Hyeonggon Yoo <42.hyeyoo@gmail.com> To: Christoph Lameter , Pekka Enberg , David Rientjes , Joonsoo Kim , Andrew Morton , Vlastimil Babka , Roman Gushchin Cc: Hyeonggon Yoo <42.hyeyoo@gmail.com>, linux-mm@kvack.org, linux-kernel@vger.kernel.org Subject: [PATCH v4 14/17] mm/slab_common: unify NUMA and UMA version of tracepoints Date: Wed, 17 Aug 2022 19:18:23 +0900 Message-Id: <20220817101826.236819-15-42.hyeyoo@gmail.com> X-Mailer: git-send-email 2.32.0 In-Reply-To: <20220817101826.236819-1-42.hyeyoo@gmail.com> References: <20220817101826.236819-1-42.hyeyoo@gmail.com> MIME-Version: 1.0 ARC-Seal: i=1; s=arc-20220608; d=hostedemail.com; t=1660731579; a=rsa-sha256; cv=none; b=AeKVopoiq0C0epz3by1UJ6EymZtPD0sbXn3Ce1SmfdWP2x/VLIUR6AovXWZs8XH/cXsTm0 SHo04ETiazEslV0XAVDmFfR7tmMCsBSVu0wAPCdPBrRW6623jk55YtbRUvQN9D3tpvijcE 44STLWPCyr3nU5yIaAPJVkcknE1Hcfc= ARC-Authentication-Results: i=1; imf03.hostedemail.com; dkim=pass header.d=gmail.com header.s=20210112 header.b="VUPnz/D4"; spf=pass (imf03.hostedemail.com: domain of 42.hyeyoo@gmail.com designates 209.85.214.182 as permitted sender) smtp.mailfrom=42.hyeyoo@gmail.com; dmarc=pass (policy=none) header.from=gmail.com ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=hostedemail.com; s=arc-20220608; t=1660731579; h=from:from:sender:reply-to:subject:subject:date:date: message-id:message-id:to:to:cc:cc:mime-version:mime-version: content-type:content-transfer-encoding:content-transfer-encoding: in-reply-to:in-reply-to:references:references:dkim-signature; bh=CyypfYVW40Zxy99wL/B3ZBuo0BqIs8lbgANQnRCbVC4=; b=DNMRaP0iWRXt7eFDyq8W5PmmaTh1Iu/ogCDf5luqzy2fqRuUe6KNk9m+F1wcEjif6+XHIY 5EZeWaMAV3S08yAgvr5a21/mSKEUmojQMgMqPGS2ME28ymhURaArjXk1g9Hi8aVjBSkmoh HDieInG9Oo6aVfYPL9H/03oWbbK2i2k= X-Stat-Signature: 5htw9ed88ptfmtrsitsnjjmeafe9ee3i X-Rspamd-Queue-Id: 62D4F201D4 Authentication-Results: imf03.hostedemail.com; dkim=pass header.d=gmail.com header.s=20210112 header.b="VUPnz/D4"; spf=pass (imf03.hostedemail.com: domain of 42.hyeyoo@gmail.com designates 209.85.214.182 as permitted sender) smtp.mailfrom=42.hyeyoo@gmail.com; dmarc=pass (policy=none) header.from=gmail.com X-Rspam-User: X-Rspamd-Server: rspam03 X-HE-Tag: 1660731579-685257 X-Bogosity: Ham, tests=bogofilter, spamicity=0.000000, version=1.2.4 Sender: owner-linux-mm@kvack.org Precedence: bulk X-Loop: owner-majordomo@kvack.org List-ID: Drop kmem_alloc event class, rename kmem_alloc_node to kmem_alloc, and remove _node postfix for NUMA version of tracepoints. This will break some tools that depend on {kmem_cache_alloc,kmalloc}_node, but at this point maintaining both kmem_alloc and kmem_alloc_node event classes does not makes sense at all. Signed-off-by: Hyeonggon Yoo <42.hyeyoo@gmail.com> Reviewed-by: Vlastimil Babka --- include/trace/events/kmem.h | 60 ++----------------------------------- mm/slab.c | 9 +++--- mm/slab_common.c | 21 +++++-------- mm/slob.c | 20 ++++++------- mm/slub.c | 6 ++-- 5 files changed, 27 insertions(+), 89 deletions(-) diff --git a/include/trace/events/kmem.h b/include/trace/events/kmem.h index 4cb51ace600d..e078ebcdc4b1 100644 --- a/include/trace/events/kmem.h +++ b/include/trace/events/kmem.h @@ -11,62 +11,6 @@ DECLARE_EVENT_CLASS(kmem_alloc, - TP_PROTO(unsigned long call_site, - const void *ptr, - struct kmem_cache *s, - size_t bytes_req, - size_t bytes_alloc, - gfp_t gfp_flags), - - TP_ARGS(call_site, ptr, s, bytes_req, bytes_alloc, gfp_flags), - - TP_STRUCT__entry( - __field( unsigned long, call_site ) - __field( const void *, ptr ) - __field( size_t, bytes_req ) - __field( size_t, bytes_alloc ) - __field( unsigned long, gfp_flags ) - __field( bool, accounted ) - ), - - TP_fast_assign( - __entry->call_site = call_site; - __entry->ptr = ptr; - __entry->bytes_req = bytes_req; - __entry->bytes_alloc = bytes_alloc; - __entry->gfp_flags = (__force unsigned long)gfp_flags; - __entry->accounted = IS_ENABLED(CONFIG_MEMCG_KMEM) ? - ((gfp_flags & __GFP_ACCOUNT) || - (s && s->flags & SLAB_ACCOUNT)) : false; - ), - - TP_printk("call_site=%pS ptr=%p bytes_req=%zu bytes_alloc=%zu gfp_flags=%s accounted=%s", - (void *)__entry->call_site, - __entry->ptr, - __entry->bytes_req, - __entry->bytes_alloc, - show_gfp_flags(__entry->gfp_flags), - __entry->accounted ? "true" : "false") -); - -DEFINE_EVENT(kmem_alloc, kmalloc, - - TP_PROTO(unsigned long call_site, const void *ptr, struct kmem_cache *s, - size_t bytes_req, size_t bytes_alloc, gfp_t gfp_flags), - - TP_ARGS(call_site, ptr, s, bytes_req, bytes_alloc, gfp_flags) -); - -DEFINE_EVENT(kmem_alloc, kmem_cache_alloc, - - TP_PROTO(unsigned long call_site, const void *ptr, struct kmem_cache *s, - size_t bytes_req, size_t bytes_alloc, gfp_t gfp_flags), - - TP_ARGS(call_site, ptr, s, bytes_req, bytes_alloc, gfp_flags) -); - -DECLARE_EVENT_CLASS(kmem_alloc_node, - TP_PROTO(unsigned long call_site, const void *ptr, struct kmem_cache *s, @@ -109,7 +53,7 @@ DECLARE_EVENT_CLASS(kmem_alloc_node, __entry->accounted ? "true" : "false") ); -DEFINE_EVENT(kmem_alloc_node, kmalloc_node, +DEFINE_EVENT(kmem_alloc, kmalloc, TP_PROTO(unsigned long call_site, const void *ptr, struct kmem_cache *s, size_t bytes_req, size_t bytes_alloc, @@ -118,7 +62,7 @@ DEFINE_EVENT(kmem_alloc_node, kmalloc_node, TP_ARGS(call_site, ptr, s, bytes_req, bytes_alloc, gfp_flags, node) ); -DEFINE_EVENT(kmem_alloc_node, kmem_cache_alloc_node, +DEFINE_EVENT(kmem_alloc, kmem_cache_alloc, TP_PROTO(unsigned long call_site, const void *ptr, struct kmem_cache *s, size_t bytes_req, size_t bytes_alloc, diff --git a/mm/slab.c b/mm/slab.c index 8d9d0fbf9792..2fd400203ac2 100644 --- a/mm/slab.c +++ b/mm/slab.c @@ -3440,8 +3440,8 @@ void *__kmem_cache_alloc_lru(struct kmem_cache *cachep, struct list_lru *lru, { void *ret = slab_alloc(cachep, lru, flags, cachep->object_size, _RET_IP_); - trace_kmem_cache_alloc(_RET_IP_, ret, cachep, - cachep->object_size, cachep->size, flags); + trace_kmem_cache_alloc(_RET_IP_, ret, cachep, cachep->object_size, + cachep->size, flags, NUMA_NO_NODE); return ret; } @@ -3536,9 +3536,8 @@ void *kmem_cache_alloc_node(struct kmem_cache *cachep, gfp_t flags, int nodeid) { void *ret = slab_alloc_node(cachep, NULL, flags, nodeid, cachep->object_size, _RET_IP_); - trace_kmem_cache_alloc_node(_RET_IP_, ret, cachep, - cachep->object_size, cachep->size, - flags, nodeid); + trace_kmem_cache_alloc(_RET_IP_, ret, cachep, cachep->object_size, + cachep->size, flags, nodeid); return ret; } diff --git a/mm/slab_common.c b/mm/slab_common.c index d8e8c41c12f1..f34be57b00c8 100644 --- a/mm/slab_common.c +++ b/mm/slab_common.c @@ -907,9 +907,8 @@ void *__do_kmalloc_node(size_t size, gfp_t flags, int node, unsigned long caller if (unlikely(size > KMALLOC_MAX_CACHE_SIZE)) { ret = __kmalloc_large_node(size, flags, node); - trace_kmalloc_node(caller, ret, NULL, - size, PAGE_SIZE << get_order(size), - flags, node); + trace_kmalloc(_RET_IP_, ret, NULL, size, + PAGE_SIZE << get_order(size), flags, node); return ret; } @@ -920,8 +919,7 @@ void *__do_kmalloc_node(size_t size, gfp_t flags, int node, unsigned long caller ret = __kmem_cache_alloc_node(s, flags, node, size, caller); ret = kasan_kmalloc(s, ret, size, flags); - trace_kmalloc_node(caller, ret, s, size, - s->size, flags, node); + trace_kmalloc(_RET_IP_, ret, s, size, s->size, flags, node); return ret; } @@ -1007,8 +1005,7 @@ void *kmalloc_trace(struct kmem_cache *s, gfp_t gfpflags, size_t size) void *ret = __kmem_cache_alloc_node(s, gfpflags, NUMA_NO_NODE, size, _RET_IP_); - trace_kmalloc_node(_RET_IP_, ret, s, size, s->size, - gfpflags, NUMA_NO_NODE); + trace_kmalloc(_RET_IP_, ret, s, size, s->size, gfpflags, NUMA_NO_NODE); ret = kasan_kmalloc(s, ret, size, gfpflags); return ret; @@ -1020,7 +1017,7 @@ void *kmalloc_node_trace(struct kmem_cache *s, gfp_t gfpflags, { void *ret = __kmem_cache_alloc_node(s, gfpflags, node, size, _RET_IP_); - trace_kmalloc_node(_RET_IP_, ret, s, size, s->size, gfpflags, node); + trace_kmalloc(_RET_IP_, ret, s, size, s->size, gfpflags, node); ret = kasan_kmalloc(s, ret, size, gfpflags); return ret; @@ -1076,7 +1073,7 @@ void *kmalloc_large(size_t size, gfp_t flags) void *ret = __kmalloc_large_node(size, flags, NUMA_NO_NODE); trace_kmalloc(_RET_IP_, ret, NULL, size, - PAGE_SIZE << get_order(size), flags); + PAGE_SIZE << get_order(size), flags, NUMA_NO_NODE); return ret; } EXPORT_SYMBOL(kmalloc_large); @@ -1085,8 +1082,8 @@ void *kmalloc_large_node(size_t size, gfp_t flags, int node) { void *ret = __kmalloc_large_node(size, flags, node); - trace_kmalloc_node(_RET_IP_, ret, NULL, size, - PAGE_SIZE << get_order(size), flags, node); + trace_kmalloc(_RET_IP_, ret, NULL, size, + PAGE_SIZE << get_order(size), flags, node); return ret; } EXPORT_SYMBOL(kmalloc_large_node); @@ -1421,8 +1418,6 @@ EXPORT_SYMBOL(ksize); /* Tracepoints definitions. */ EXPORT_TRACEPOINT_SYMBOL(kmalloc); EXPORT_TRACEPOINT_SYMBOL(kmem_cache_alloc); -EXPORT_TRACEPOINT_SYMBOL(kmalloc_node); -EXPORT_TRACEPOINT_SYMBOL(kmem_cache_alloc_node); EXPORT_TRACEPOINT_SYMBOL(kfree); EXPORT_TRACEPOINT_SYMBOL(kmem_cache_free); diff --git a/mm/slob.c b/mm/slob.c index 96b08acd72ce..3208c56d8f82 100644 --- a/mm/slob.c +++ b/mm/slob.c @@ -507,8 +507,8 @@ __do_kmalloc_node(size_t size, gfp_t gfp, int node, unsigned long caller) *m = size; ret = (void *)m + minalign; - trace_kmalloc_node(caller, ret, NULL, - size, size + minalign, gfp, node); + trace_kmalloc(caller, ret, NULL, size, + size + minalign, gfp, node); } else { unsigned int order = get_order(size); @@ -516,8 +516,8 @@ __do_kmalloc_node(size_t size, gfp_t gfp, int node, unsigned long caller) gfp |= __GFP_COMP; ret = slob_new_pages(gfp, order, node); - trace_kmalloc_node(caller, ret, NULL, - size, PAGE_SIZE << order, gfp, node); + trace_kmalloc(caller, ret, NULL, size, + PAGE_SIZE << order, gfp, node); } kmemleak_alloc(ret, size, 1, gfp); @@ -608,14 +608,14 @@ static void *slob_alloc_node(struct kmem_cache *c, gfp_t flags, int node) if (c->size < PAGE_SIZE) { b = slob_alloc(c->size, flags, c->align, node, 0); - trace_kmem_cache_alloc_node(_RET_IP_, b, NULL, c->object_size, - SLOB_UNITS(c->size) * SLOB_UNIT, - flags, node); + trace_kmem_cache_alloc(_RET_IP_, b, NULL, c->object_size, + SLOB_UNITS(c->size) * SLOB_UNIT, + flags, node); } else { b = slob_new_pages(flags, get_order(c->size), node); - trace_kmem_cache_alloc_node(_RET_IP_, b, NULL, c->object_size, - PAGE_SIZE << get_order(c->size), - flags, node); + trace_kmem_cache_alloc(_RET_IP_, b, NULL, c->object_size, + PAGE_SIZE << get_order(c->size), + flags, node); } if (b && c->ctor) { diff --git a/mm/slub.c b/mm/slub.c index 7d7fd9d4e8fa..22e4ccf06638 100644 --- a/mm/slub.c +++ b/mm/slub.c @@ -3244,7 +3244,7 @@ void *__kmem_cache_alloc_lru(struct kmem_cache *s, struct list_lru *lru, void *ret = slab_alloc(s, lru, gfpflags, _RET_IP_, s->object_size); trace_kmem_cache_alloc(_RET_IP_, ret, s, s->object_size, - s->size, gfpflags); + s->size, gfpflags, NUMA_NO_NODE); return ret; } @@ -3274,8 +3274,8 @@ void *kmem_cache_alloc_node(struct kmem_cache *s, gfp_t gfpflags, int node) { void *ret = slab_alloc_node(s, NULL, gfpflags, node, _RET_IP_, s->object_size); - trace_kmem_cache_alloc_node(_RET_IP_, ret, s, - s->object_size, s->size, gfpflags, node); + trace_kmem_cache_alloc(_RET_IP_, ret, s, s->object_size, + s->size, gfpflags, node); return ret; } From patchwork Wed Aug 17 10:18:24 2022 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Hyeonggon Yoo <42.hyeyoo@gmail.com> X-Patchwork-Id: 12945694 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from kanga.kvack.org (kanga.kvack.org [205.233.56.17]) by smtp.lore.kernel.org (Postfix) with ESMTP id 8813DC25B08 for ; Wed, 17 Aug 2022 10:19:44 +0000 (UTC) Received: by kanga.kvack.org (Postfix) id 289D66B007B; Wed, 17 Aug 2022 06:19:44 -0400 (EDT) Received: by kanga.kvack.org (Postfix, from userid 40) id 2399C8D0001; Wed, 17 Aug 2022 06:19:44 -0400 (EDT) X-Delivered-To: int-list-linux-mm@kvack.org Received: by kanga.kvack.org (Postfix, from userid 63042) id 0DB466B0081; Wed, 17 Aug 2022 06:19:44 -0400 (EDT) X-Delivered-To: linux-mm@kvack.org Received: from relay.hostedemail.com (smtprelay0012.hostedemail.com [216.40.44.12]) by kanga.kvack.org (Postfix) with ESMTP id F3A416B007B for ; Wed, 17 Aug 2022 06:19:43 -0400 (EDT) Received: from smtpin23.hostedemail.com (a10.router.float.18 [10.200.18.1]) by unirelay06.hostedemail.com (Postfix) with ESMTP id C5FFFAB61C for ; Wed, 17 Aug 2022 10:19:43 +0000 (UTC) X-FDA: 79808688246.23.0927DD9 Received: from mail-pf1-f171.google.com (mail-pf1-f171.google.com [209.85.210.171]) by imf06.hostedemail.com (Postfix) with ESMTP id 6904B180052 for ; Wed, 17 Aug 2022 10:19:43 +0000 (UTC) Received: by mail-pf1-f171.google.com with SMTP id k14so11694984pfh.0 for ; Wed, 17 Aug 2022 03:19:43 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=20210112; h=content-transfer-encoding:mime-version:references:in-reply-to :message-id:date:subject:cc:to:from:from:to:cc; bh=NUhisoUUDw6CQz+v9YGeettBV60tR4WaAKuvhlxdtBs=; b=lhCz68I4EgFx+wWyEe5AMNw5KeaZgl8syXATH2qGGWEIgxjlpooCBpohcHeUeiDDsE jE8b0xGnKbCNY75aS6eBlqIEMXv82HseRtahkUw/WZmYNK8Z0Bgd/loYGY6xdUlksdXB ejx/2wQqBg1/WgEWz7oJjb404IHTLZVdjcTqTdLd+wEoTDTG4nCVWq82n37zykZW865K Cg4LGy5uNJUROcc0ZUoLdjKM4mi4+x1DtxpbT6uTD6JP7qiP5WVhOWmgsDvCYx54yIjd ZyzQDLn7MI3DGfQlRiRQA4A1roAGqH/PC4LR/WxOSi/4R6FkzBpxeipD6+7yv162wSUP 83+g== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20210112; h=content-transfer-encoding:mime-version:references:in-reply-to :message-id:date:subject:cc:to:from:x-gm-message-state:from:to:cc; bh=NUhisoUUDw6CQz+v9YGeettBV60tR4WaAKuvhlxdtBs=; b=WYOtu9J2XEiipwOdwi2+6uuPv4e7zq0HveC5N1bYIYb7gtTI/8JIrZPz4RCoG6Mgfk qM34BwB8UQ1H+5OgJ//jE42KHHocv5eEOC6po4ticGeRgFgNUW15WlkrVE30skv0LTdf a5XolB/X1Kt17HoDlvhqGqlRyzqzzGKczPCKSGN7RUsfKCoqth+tFr/q3ZIVYHXXYLi0 IazU2umBFfzmZaUvXtc6/hvtHvtG+nD2xvAQDaxJV1Rt1PnzQpz50d2f6Exbf29O7ASd ciO6xgPPj5CGiQ6FFwxCRsRFeoEP/PSnTphgHlmIU9um9htB56R+hTwDmrrKDoUki5Do UThg== X-Gm-Message-State: ACgBeo2Gqfa+B0WyB3H5xEHUuLFAzMNjEIMdD+8ZRSq1zc/omSQZh1mG 92u2qqVSBIrw3Y1PohQ4gSA= X-Google-Smtp-Source: AA6agR5UJmaMtf04/yHbQgrf6h8TvXMUUhoRNtxO3WVLNbTHMdy2nAQOOKD3bg2S2eKBtrYQEnSLbg== X-Received: by 2002:a62:4ed3:0:b0:52d:9b4a:d91a with SMTP id c202-20020a624ed3000000b0052d9b4ad91amr24901250pfb.8.1660731582428; Wed, 17 Aug 2022 03:19:42 -0700 (PDT) Received: from hyeyoo.. ([114.29.91.56]) by smtp.gmail.com with ESMTPSA id d8-20020a170903230800b00172633fc236sm1071318plh.174.2022.08.17.03.19.38 (version=TLS1_3 cipher=TLS_AES_256_GCM_SHA384 bits=256/256); Wed, 17 Aug 2022 03:19:41 -0700 (PDT) From: Hyeonggon Yoo <42.hyeyoo@gmail.com> To: Christoph Lameter , Pekka Enberg , David Rientjes , Joonsoo Kim , Andrew Morton , Vlastimil Babka , Roman Gushchin Cc: Hyeonggon Yoo <42.hyeyoo@gmail.com>, linux-mm@kvack.org, linux-kernel@vger.kernel.org, Vasily Averin Subject: [PATCH v4 15/17] mm/slab_common: drop kmem_alloc & avoid dereferencing fields when not using Date: Wed, 17 Aug 2022 19:18:24 +0900 Message-Id: <20220817101826.236819-16-42.hyeyoo@gmail.com> X-Mailer: git-send-email 2.32.0 In-Reply-To: <20220817101826.236819-1-42.hyeyoo@gmail.com> References: <20220817101826.236819-1-42.hyeyoo@gmail.com> MIME-Version: 1.0 ARC-Seal: i=1; s=arc-20220608; d=hostedemail.com; t=1660731583; a=rsa-sha256; cv=none; b=wpdJxWh58ek5ib6l4zk2hhuV+HyJWRYzFGRGdme+Z40JeTZML/ZM8eJe2x4iTmqnvNVN2S OysRV+OZWSpLP7z1U6wuEfCmhcOA0zhTnFIHKOdjeFT1hq3zISUbLTkW3/gd6karDuEmd8 GRVvJaMq2PypyCgqZzfD9rxWklMaJHI= ARC-Authentication-Results: i=1; imf06.hostedemail.com; dkim=pass header.d=gmail.com header.s=20210112 header.b=lhCz68I4; spf=pass (imf06.hostedemail.com: domain of 42.hyeyoo@gmail.com designates 209.85.210.171 as permitted sender) smtp.mailfrom=42.hyeyoo@gmail.com; dmarc=pass (policy=none) header.from=gmail.com ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=hostedemail.com; s=arc-20220608; t=1660731583; h=from:from:sender:reply-to:subject:subject:date:date: message-id:message-id:to:to:cc:cc:mime-version:mime-version: content-type:content-transfer-encoding:content-transfer-encoding: in-reply-to:in-reply-to:references:references:dkim-signature; bh=NUhisoUUDw6CQz+v9YGeettBV60tR4WaAKuvhlxdtBs=; b=W+h2F5hWbHqM6YF9I5hQcdF5zXggu5pCHwcTmpkmDf0wJ5PFT9ZALlozVlabPzYnnSO9Gy ocFwbbvNeYvpXyGjHXxlV4EG8rTXWQHJQGxwByHBrY/RATUwcdpuqIpR2umNW8oMNuMgMJ /0QF0bRd3H8vj30dlRROPbO5X8JftbU= Authentication-Results: imf06.hostedemail.com; dkim=pass header.d=gmail.com header.s=20210112 header.b=lhCz68I4; spf=pass (imf06.hostedemail.com: domain of 42.hyeyoo@gmail.com designates 209.85.210.171 as permitted sender) smtp.mailfrom=42.hyeyoo@gmail.com; dmarc=pass (policy=none) header.from=gmail.com X-Rspam-User: X-Stat-Signature: 3m6uzq5fgcoyjfcb5yz95gwdf8ie8zp7 X-Rspamd-Queue-Id: 6904B180052 X-Rspamd-Server: rspam06 X-HE-Tag: 1660731583-563831 X-Bogosity: Ham, tests=bogofilter, spamicity=0.000000, version=1.2.4 Sender: owner-linux-mm@kvack.org Precedence: bulk X-Loop: owner-majordomo@kvack.org List-ID: Drop kmem_alloc event class, and define kmalloc and kmem_cache_alloc using TRACE_EVENT() macro. And then this patch does: - Do not pass pointer to struct kmem_cache to trace_kmalloc. gfp flag is enough to know if it's accounted or not. - Avoid dereferencing s->object_size and s->size when not using kmem_cache_alloc event. - Avoid dereferencing s->name in when not using kmem_cache_free event. - Adjust s->size to SLOB_UNITS(s->size) * SLOB_UNIT in SLOB Cc: Vasily Averin Suggested-by: Vlastimil Babka Signed-off-by: Hyeonggon Yoo <42.hyeyoo@gmail.com> Reviewed-by: Vlastimil Babka --- include/trace/events/kmem.h | 64 ++++++++++++++++++++++++------------- mm/slab.c | 8 ++--- mm/slab_common.c | 16 +++++----- mm/slob.c | 19 +++++------ mm/slub.c | 8 ++--- 5 files changed, 64 insertions(+), 51 deletions(-) diff --git a/include/trace/events/kmem.h b/include/trace/events/kmem.h index e078ebcdc4b1..8c6f96604244 100644 --- a/include/trace/events/kmem.h +++ b/include/trace/events/kmem.h @@ -9,17 +9,15 @@ #include #include -DECLARE_EVENT_CLASS(kmem_alloc, +TRACE_EVENT(kmem_cache_alloc, TP_PROTO(unsigned long call_site, const void *ptr, struct kmem_cache *s, - size_t bytes_req, - size_t bytes_alloc, gfp_t gfp_flags, int node), - TP_ARGS(call_site, ptr, s, bytes_req, bytes_alloc, gfp_flags, node), + TP_ARGS(call_site, ptr, s, gfp_flags, node), TP_STRUCT__entry( __field( unsigned long, call_site ) @@ -34,13 +32,13 @@ DECLARE_EVENT_CLASS(kmem_alloc, TP_fast_assign( __entry->call_site = call_site; __entry->ptr = ptr; - __entry->bytes_req = bytes_req; - __entry->bytes_alloc = bytes_alloc; + __entry->bytes_req = s->object_size; + __entry->bytes_alloc = s->size; __entry->gfp_flags = (__force unsigned long)gfp_flags; __entry->node = node; __entry->accounted = IS_ENABLED(CONFIG_MEMCG_KMEM) ? ((gfp_flags & __GFP_ACCOUNT) || - (s && s->flags & SLAB_ACCOUNT)) : false; + (s->flags & SLAB_ACCOUNT)) : false; ), TP_printk("call_site=%pS ptr=%p bytes_req=%zu bytes_alloc=%zu gfp_flags=%s node=%d accounted=%s", @@ -53,22 +51,44 @@ DECLARE_EVENT_CLASS(kmem_alloc, __entry->accounted ? "true" : "false") ); -DEFINE_EVENT(kmem_alloc, kmalloc, +TRACE_EVENT(kmalloc, - TP_PROTO(unsigned long call_site, const void *ptr, - struct kmem_cache *s, size_t bytes_req, size_t bytes_alloc, - gfp_t gfp_flags, int node), + TP_PROTO(unsigned long call_site, + const void *ptr, + size_t bytes_req, + size_t bytes_alloc, + gfp_t gfp_flags, + int node), - TP_ARGS(call_site, ptr, s, bytes_req, bytes_alloc, gfp_flags, node) -); + TP_ARGS(call_site, ptr, bytes_req, bytes_alloc, gfp_flags, node), -DEFINE_EVENT(kmem_alloc, kmem_cache_alloc, + TP_STRUCT__entry( + __field( unsigned long, call_site ) + __field( const void *, ptr ) + __field( size_t, bytes_req ) + __field( size_t, bytes_alloc ) + __field( unsigned long, gfp_flags ) + __field( int, node ) + ), - TP_PROTO(unsigned long call_site, const void *ptr, - struct kmem_cache *s, size_t bytes_req, size_t bytes_alloc, - gfp_t gfp_flags, int node), + TP_fast_assign( + __entry->call_site = call_site; + __entry->ptr = ptr; + __entry->bytes_req = bytes_req; + __entry->bytes_alloc = bytes_alloc; + __entry->gfp_flags = (__force unsigned long)gfp_flags; + __entry->node = node; + ), - TP_ARGS(call_site, ptr, s, bytes_req, bytes_alloc, gfp_flags, node) + TP_printk("call_site=%pS ptr=%p bytes_req=%zu bytes_alloc=%zu gfp_flags=%s node=%d accounted=%s", + (void *)__entry->call_site, + __entry->ptr, + __entry->bytes_req, + __entry->bytes_alloc, + show_gfp_flags(__entry->gfp_flags), + __entry->node, + (IS_ENABLED(CONFIG_MEMCG_KMEM) && + (__entry->gfp_flags & __GFP_ACCOUNT)) ? "true" : "false") ); TRACE_EVENT(kfree, @@ -93,20 +113,20 @@ TRACE_EVENT(kfree, TRACE_EVENT(kmem_cache_free, - TP_PROTO(unsigned long call_site, const void *ptr, const char *name), + TP_PROTO(unsigned long call_site, const void *ptr, const struct kmem_cache *s), - TP_ARGS(call_site, ptr, name), + TP_ARGS(call_site, ptr, s), TP_STRUCT__entry( __field( unsigned long, call_site ) __field( const void *, ptr ) - __string( name, name ) + __string( name, s->name ) ), TP_fast_assign( __entry->call_site = call_site; __entry->ptr = ptr; - __assign_str(name, name); + __assign_str(name, s->name); ), TP_printk("call_site=%pS ptr=%p name=%s", diff --git a/mm/slab.c b/mm/slab.c index 2fd400203ac2..a5486ff8362a 100644 --- a/mm/slab.c +++ b/mm/slab.c @@ -3440,8 +3440,7 @@ void *__kmem_cache_alloc_lru(struct kmem_cache *cachep, struct list_lru *lru, { void *ret = slab_alloc(cachep, lru, flags, cachep->object_size, _RET_IP_); - trace_kmem_cache_alloc(_RET_IP_, ret, cachep, cachep->object_size, - cachep->size, flags, NUMA_NO_NODE); + trace_kmem_cache_alloc(_RET_IP_, ret, cachep, flags, NUMA_NO_NODE); return ret; } @@ -3536,8 +3535,7 @@ void *kmem_cache_alloc_node(struct kmem_cache *cachep, gfp_t flags, int nodeid) { void *ret = slab_alloc_node(cachep, NULL, flags, nodeid, cachep->object_size, _RET_IP_); - trace_kmem_cache_alloc(_RET_IP_, ret, cachep, cachep->object_size, - cachep->size, flags, nodeid); + trace_kmem_cache_alloc(_RET_IP_, ret, cachep, flags, nodeid); return ret; } @@ -3607,7 +3605,7 @@ void kmem_cache_free(struct kmem_cache *cachep, void *objp) if (!cachep) return; - trace_kmem_cache_free(_RET_IP_, objp, cachep->name); + trace_kmem_cache_free(_RET_IP_, objp, cachep); __do_kmem_cache_free(cachep, objp, _RET_IP_); } EXPORT_SYMBOL(kmem_cache_free); diff --git a/mm/slab_common.c b/mm/slab_common.c index f34be57b00c8..e53016c9a6e9 100644 --- a/mm/slab_common.c +++ b/mm/slab_common.c @@ -907,7 +907,7 @@ void *__do_kmalloc_node(size_t size, gfp_t flags, int node, unsigned long caller if (unlikely(size > KMALLOC_MAX_CACHE_SIZE)) { ret = __kmalloc_large_node(size, flags, node); - trace_kmalloc(_RET_IP_, ret, NULL, size, + trace_kmalloc(_RET_IP_, ret, size, PAGE_SIZE << get_order(size), flags, node); return ret; } @@ -919,7 +919,7 @@ void *__do_kmalloc_node(size_t size, gfp_t flags, int node, unsigned long caller ret = __kmem_cache_alloc_node(s, flags, node, size, caller); ret = kasan_kmalloc(s, ret, size, flags); - trace_kmalloc(_RET_IP_, ret, s, size, s->size, flags, node); + trace_kmalloc(_RET_IP_, ret, size, s->size, flags, node); return ret; } @@ -1005,7 +1005,7 @@ void *kmalloc_trace(struct kmem_cache *s, gfp_t gfpflags, size_t size) void *ret = __kmem_cache_alloc_node(s, gfpflags, NUMA_NO_NODE, size, _RET_IP_); - trace_kmalloc(_RET_IP_, ret, s, size, s->size, gfpflags, NUMA_NO_NODE); + trace_kmalloc(_RET_IP_, ret, size, s->size, gfpflags, NUMA_NO_NODE); ret = kasan_kmalloc(s, ret, size, gfpflags); return ret; @@ -1017,7 +1017,7 @@ void *kmalloc_node_trace(struct kmem_cache *s, gfp_t gfpflags, { void *ret = __kmem_cache_alloc_node(s, gfpflags, node, size, _RET_IP_); - trace_kmalloc(_RET_IP_, ret, s, size, s->size, gfpflags, node); + trace_kmalloc(_RET_IP_, ret, size, s->size, gfpflags, node); ret = kasan_kmalloc(s, ret, size, gfpflags); return ret; @@ -1072,8 +1072,8 @@ void *kmalloc_large(size_t size, gfp_t flags) { void *ret = __kmalloc_large_node(size, flags, NUMA_NO_NODE); - trace_kmalloc(_RET_IP_, ret, NULL, size, - PAGE_SIZE << get_order(size), flags, NUMA_NO_NODE); + trace_kmalloc(_RET_IP_, ret, size, PAGE_SIZE << get_order(size), + flags, NUMA_NO_NODE); return ret; } EXPORT_SYMBOL(kmalloc_large); @@ -1082,8 +1082,8 @@ void *kmalloc_large_node(size_t size, gfp_t flags, int node) { void *ret = __kmalloc_large_node(size, flags, node); - trace_kmalloc(_RET_IP_, ret, NULL, size, - PAGE_SIZE << get_order(size), flags, node); + trace_kmalloc(_RET_IP_, ret, size, PAGE_SIZE << get_order(size), + flags, node); return ret; } EXPORT_SYMBOL(kmalloc_large_node); diff --git a/mm/slob.c b/mm/slob.c index 3208c56d8f82..771af84576bf 100644 --- a/mm/slob.c +++ b/mm/slob.c @@ -507,8 +507,7 @@ __do_kmalloc_node(size_t size, gfp_t gfp, int node, unsigned long caller) *m = size; ret = (void *)m + minalign; - trace_kmalloc(caller, ret, NULL, size, - size + minalign, gfp, node); + trace_kmalloc(caller, ret, size, size + minalign, gfp, node); } else { unsigned int order = get_order(size); @@ -516,8 +515,7 @@ __do_kmalloc_node(size_t size, gfp_t gfp, int node, unsigned long caller) gfp |= __GFP_COMP; ret = slob_new_pages(gfp, order, node); - trace_kmalloc(caller, ret, NULL, size, - PAGE_SIZE << order, gfp, node); + trace_kmalloc(caller, ret, size, PAGE_SIZE << order, gfp, node); } kmemleak_alloc(ret, size, 1, gfp); @@ -594,6 +592,9 @@ int __kmem_cache_create(struct kmem_cache *c, slab_flags_t flags) /* leave room for rcu footer at the end of object */ c->size += sizeof(struct slob_rcu); } + + /* Actual size allocated */ + c->size = SLOB_UNITS(c->size) * SLOB_UNIT; c->flags = flags; return 0; } @@ -608,14 +609,10 @@ static void *slob_alloc_node(struct kmem_cache *c, gfp_t flags, int node) if (c->size < PAGE_SIZE) { b = slob_alloc(c->size, flags, c->align, node, 0); - trace_kmem_cache_alloc(_RET_IP_, b, NULL, c->object_size, - SLOB_UNITS(c->size) * SLOB_UNIT, - flags, node); + trace_kmem_cache_alloc(_RET_IP_, b, c, flags, node); } else { b = slob_new_pages(flags, get_order(c->size), node); - trace_kmem_cache_alloc(_RET_IP_, b, NULL, c->object_size, - PAGE_SIZE << get_order(c->size), - flags, node); + trace_kmem_cache_alloc(_RET_IP_, b, c, flags, node); } if (b && c->ctor) { @@ -671,7 +668,7 @@ static void kmem_rcu_free(struct rcu_head *head) void kmem_cache_free(struct kmem_cache *c, void *b) { kmemleak_free_recursive(b, c->flags); - trace_kmem_cache_free(_RET_IP_, b, c->name); + trace_kmem_cache_free(_RET_IP_, b, c); if (unlikely(c->flags & SLAB_TYPESAFE_BY_RCU)) { struct slob_rcu *slob_rcu; slob_rcu = b + (c->size - sizeof(struct slob_rcu)); diff --git a/mm/slub.c b/mm/slub.c index 22e4ccf06638..8083a6ee5f15 100644 --- a/mm/slub.c +++ b/mm/slub.c @@ -3243,8 +3243,7 @@ void *__kmem_cache_alloc_lru(struct kmem_cache *s, struct list_lru *lru, { void *ret = slab_alloc(s, lru, gfpflags, _RET_IP_, s->object_size); - trace_kmem_cache_alloc(_RET_IP_, ret, s, s->object_size, - s->size, gfpflags, NUMA_NO_NODE); + trace_kmem_cache_alloc(_RET_IP_, ret, s, gfpflags, NUMA_NO_NODE); return ret; } @@ -3274,8 +3273,7 @@ void *kmem_cache_alloc_node(struct kmem_cache *s, gfp_t gfpflags, int node) { void *ret = slab_alloc_node(s, NULL, gfpflags, node, _RET_IP_, s->object_size); - trace_kmem_cache_alloc(_RET_IP_, ret, s, s->object_size, - s->size, gfpflags, node); + trace_kmem_cache_alloc(_RET_IP_, ret, s, gfpflags, node); return ret; } @@ -3517,7 +3515,7 @@ void kmem_cache_free(struct kmem_cache *s, void *x) s = cache_from_obj(s, x); if (!s) return; - trace_kmem_cache_free(_RET_IP_, x, s->name); + trace_kmem_cache_free(_RET_IP_, x, s); slab_free(s, virt_to_slab(x), x, NULL, &x, 1, _RET_IP_); } EXPORT_SYMBOL(kmem_cache_free); From patchwork Wed Aug 17 10:18:25 2022 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Hyeonggon Yoo <42.hyeyoo@gmail.com> X-Patchwork-Id: 12945696 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from kanga.kvack.org (kanga.kvack.org [205.233.56.17]) by smtp.lore.kernel.org (Postfix) with ESMTP id 47DB6C32772 for ; Wed, 17 Aug 2022 10:20:17 +0000 (UTC) Received: by kanga.kvack.org (Postfix) id D60416B0074; Wed, 17 Aug 2022 06:20:16 -0400 (EDT) Received: by kanga.kvack.org (Postfix, from userid 40) id BFDD86B0075; Wed, 17 Aug 2022 06:20:16 -0400 (EDT) X-Delivered-To: int-list-linux-mm@kvack.org Received: by kanga.kvack.org (Postfix, from userid 63042) id AC5066B007B; Wed, 17 Aug 2022 06:20:16 -0400 (EDT) X-Delivered-To: linux-mm@kvack.org Received: from relay.hostedemail.com (smtprelay0013.hostedemail.com [216.40.44.13]) by kanga.kvack.org (Postfix) with ESMTP id 9C8116B0074 for ; Wed, 17 Aug 2022 06:20:16 -0400 (EDT) Received: from smtpin20.hostedemail.com (a10.router.float.18 [10.200.18.1]) by unirelay04.hostedemail.com (Postfix) with ESMTP id BFBF81A0848 for ; Wed, 17 Aug 2022 10:19:47 +0000 (UTC) X-FDA: 79808688414.20.094F0E6 Received: from mail-pg1-f175.google.com (mail-pg1-f175.google.com [209.85.215.175]) by imf19.hostedemail.com (Postfix) with ESMTP id 615E81A01A9 for ; Wed, 17 Aug 2022 10:19:46 +0000 (UTC) Received: by mail-pg1-f175.google.com with SMTP id d71so11581697pgc.13 for ; Wed, 17 Aug 2022 03:19:46 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=20210112; h=content-transfer-encoding:mime-version:references:in-reply-to :message-id:date:subject:cc:to:from:from:to:cc; bh=Ru5NzUFWFY1mp//X6stEiyuQPjnMcLZ7ult0h/+qL8E=; b=YqBZ+mIsukwjNFm5uj1S+x8Wp0d/9vKvRmeFd9hIw3ZdJzV0rLL/Zmbnggn5OJciun 9UgY/TPWTSYXcxq3xtTI7HAeHmaNStsepckVV8MkSvRWAfWqJIP4XYhCXx+jwx0i4yiN bDOIH+nLhf+U8CWc08sNAyxHmLQSsy5Cx5fVlJI/WKog3wjMJKqopoCY1lL1nV2rKqEP MJvm2+Ki7N/FCE2q6MsxhJDfWyPKkgM9Zd0qGEGxxKsD+tht0wnGOoErPE1d9WFBRlfm WZ8xpj3b/K6MAqYWvqfxR58XK7+JIc77qIu1Q+pijxc5z+kSq8bTsIh5WE9CfHIG0SYI oreQ== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20210112; h=content-transfer-encoding:mime-version:references:in-reply-to :message-id:date:subject:cc:to:from:x-gm-message-state:from:to:cc; bh=Ru5NzUFWFY1mp//X6stEiyuQPjnMcLZ7ult0h/+qL8E=; b=vr7IxqL/QoRUSG17ko0Gh8hVSYmV5J0eEbK8jMuMg3u/wsPSMh1a8z+TKsKvoYsFYo MuyfMQn/JML2SkZXaScV4wMX3Xs4B6kH/GEq5Xjk0MgUwUvR4CO7PYytzKAD81Df+XC2 UwuzPzoq/CjTWsvvpuOwO8kl6BFofIOEbQjcyp8QFFpOAFJXZ2PyXbCr2VU6ecRdVf97 A34XcMRTxBkj2Miu3AF5C/0ZbMaZmVDGlPrXgQD4oaGPo1qK8Jhla8TZAmbkphGs9ZS9 7Oj+vQbtJ4PkTa3Ha4GiNkYmH4H8hAXz4BCbaMvKSwSB65x5wV5czBu7Q5i5xbGrCZq2 6Opg== X-Gm-Message-State: ACgBeo33NP3AhcTLWqB9LA9nqAIecLz8hdtG176GxvUUgSZabhUmT6Du /eW65CdwJWyMoZsQ3AE4jto= X-Google-Smtp-Source: AA6agR6ifSZS508FUcNv8AUeG2x8oARm+JTo00bgXTAg+e0iyMO0y0O7vjS+8iYkahz8ahDH0g7Mmg== X-Received: by 2002:a63:fd14:0:b0:41a:20e8:c1e2 with SMTP id d20-20020a63fd14000000b0041a20e8c1e2mr21017032pgh.286.1660731586184; Wed, 17 Aug 2022 03:19:46 -0700 (PDT) Received: from hyeyoo.. ([114.29.91.56]) by smtp.gmail.com with ESMTPSA id d8-20020a170903230800b00172633fc236sm1071318plh.174.2022.08.17.03.19.42 (version=TLS1_3 cipher=TLS_AES_256_GCM_SHA384 bits=256/256); Wed, 17 Aug 2022 03:19:45 -0700 (PDT) From: Hyeonggon Yoo <42.hyeyoo@gmail.com> To: Christoph Lameter , Pekka Enberg , David Rientjes , Joonsoo Kim , Andrew Morton , Vlastimil Babka , Roman Gushchin Cc: Hyeonggon Yoo <42.hyeyoo@gmail.com>, linux-mm@kvack.org, linux-kernel@vger.kernel.org Subject: [PATCH v4 16/17] mm/slab_common: move declaration of __ksize() to mm/slab.h Date: Wed, 17 Aug 2022 19:18:25 +0900 Message-Id: <20220817101826.236819-17-42.hyeyoo@gmail.com> X-Mailer: git-send-email 2.32.0 In-Reply-To: <20220817101826.236819-1-42.hyeyoo@gmail.com> References: <20220817101826.236819-1-42.hyeyoo@gmail.com> MIME-Version: 1.0 ARC-Authentication-Results: i=1; imf19.hostedemail.com; dkim=pass header.d=gmail.com header.s=20210112 header.b=YqBZ+mIs; spf=pass (imf19.hostedemail.com: domain of 42.hyeyoo@gmail.com designates 209.85.215.175 as permitted sender) smtp.mailfrom=42.hyeyoo@gmail.com; dmarc=pass (policy=none) header.from=gmail.com ARC-Seal: i=1; s=arc-20220608; d=hostedemail.com; t=1660731587; a=rsa-sha256; cv=none; b=ZbBx+nfRBrNfeUoSGN/qViC2cLR1+yyaZIPRKDJfB1YOym2iQeP7HHG+l0/oRypZ/UHCAw Qs6F/hu2jrbezYBmR1uxPEI6FzOR02nRR9ZPpSB2LLoEJUHM+C+4+3o2LmLLK//xRZ1flv PXJlYPqwjpuZOpK3mpN3jBQETwPXWus= ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=hostedemail.com; s=arc-20220608; t=1660731587; h=from:from:sender:reply-to:subject:subject:date:date: message-id:message-id:to:to:cc:cc:mime-version:mime-version: content-type:content-transfer-encoding:content-transfer-encoding: in-reply-to:in-reply-to:references:references:dkim-signature; bh=Ru5NzUFWFY1mp//X6stEiyuQPjnMcLZ7ult0h/+qL8E=; b=WIwpSR3cKEDc7sf8zp1YyvblqNNXjQcPx8VLv2xktf5uCgy24n89OVy6UzOIOPOWuOZMM2 6SX4eVkwQ3we2mUSFSPkK6I0cwUv1nuwtIstUSodOOUu1Io9M83HaFepaNkUa19ecHIfVK UHDGxF0ZPg0znLRD7tqSK1xMS1cw7rI= Authentication-Results: imf19.hostedemail.com; dkim=pass header.d=gmail.com header.s=20210112 header.b=YqBZ+mIs; spf=pass (imf19.hostedemail.com: domain of 42.hyeyoo@gmail.com designates 209.85.215.175 as permitted sender) smtp.mailfrom=42.hyeyoo@gmail.com; dmarc=pass (policy=none) header.from=gmail.com X-Rspam-User: X-Rspamd-Server: rspam12 X-Stat-Signature: 9j6fj3phu9f3x667yq6tknfq1m649odw X-Rspamd-Queue-Id: 615E81A01A9 X-HE-Tag: 1660731586-459100 X-Bogosity: Ham, tests=bogofilter, spamicity=0.000000, version=1.2.4 Sender: owner-linux-mm@kvack.org Precedence: bulk X-Loop: owner-majordomo@kvack.org List-ID: __ksize() is only called by KASAN. Remove export symbol and move declaration to mm/slab.h as we don't want to grow its callers. Signed-off-by: Hyeonggon Yoo <42.hyeyoo@gmail.com> Reviewed-by: Vlastimil Babka --- include/linux/slab.h | 1 - mm/slab.h | 2 ++ mm/slab_common.c | 11 +---------- mm/slob.c | 1 - 4 files changed, 3 insertions(+), 12 deletions(-) diff --git a/include/linux/slab.h b/include/linux/slab.h index c8e485ce8815..9b592e611cb1 100644 --- a/include/linux/slab.h +++ b/include/linux/slab.h @@ -187,7 +187,6 @@ int kmem_cache_shrink(struct kmem_cache *s); void * __must_check krealloc(const void *objp, size_t new_size, gfp_t flags) __alloc_size(2); void kfree(const void *objp); void kfree_sensitive(const void *objp); -size_t __ksize(const void *objp); size_t ksize(const void *objp); #ifdef CONFIG_PRINTK bool kmem_valid_obj(void *object); diff --git a/mm/slab.h b/mm/slab.h index 4d8330d57573..65023f000d42 100644 --- a/mm/slab.h +++ b/mm/slab.h @@ -668,6 +668,8 @@ void free_large_kmalloc(struct folio *folio, void *object); #endif /* CONFIG_SLOB */ +size_t __ksize(const void *objp); + static inline size_t slab_ksize(const struct kmem_cache *s) { #ifndef CONFIG_SLUB diff --git a/mm/slab_common.c b/mm/slab_common.c index e53016c9a6e9..9c273a5fb0d7 100644 --- a/mm/slab_common.c +++ b/mm/slab_common.c @@ -974,15 +974,7 @@ void kfree(const void *object) } EXPORT_SYMBOL(kfree); -/** - * __ksize -- Uninstrumented ksize. - * @objp: pointer to the object - * - * Unlike ksize(), __ksize() is uninstrumented, and does not provide the same - * safety checks as ksize() with KASAN instrumentation enabled. - * - * Return: size of the actual memory used by @objp in bytes - */ +/* Uninstrumented ksize. Only called by KASAN. */ size_t __ksize(const void *object) { struct folio *folio; @@ -997,7 +989,6 @@ size_t __ksize(const void *object) return slab_ksize(folio_slab(folio)->slab_cache); } -EXPORT_SYMBOL(__ksize); #ifdef CONFIG_TRACING void *kmalloc_trace(struct kmem_cache *s, gfp_t gfpflags, size_t size) diff --git a/mm/slob.c b/mm/slob.c index 771af84576bf..45a061b8ba38 100644 --- a/mm/slob.c +++ b/mm/slob.c @@ -584,7 +584,6 @@ size_t __ksize(const void *block) m = (unsigned int *)(block - align); return SLOB_UNITS(*m) * SLOB_UNIT; } -EXPORT_SYMBOL(__ksize); int __kmem_cache_create(struct kmem_cache *c, slab_flags_t flags) { From patchwork Wed Aug 17 10:18:26 2022 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Hyeonggon Yoo <42.hyeyoo@gmail.com> X-Patchwork-Id: 12945695 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from kanga.kvack.org (kanga.kvack.org [205.233.56.17]) by smtp.lore.kernel.org (Postfix) with ESMTP id B79B2C25B08 for ; Wed, 17 Aug 2022 10:20:16 +0000 (UTC) Received: by kanga.kvack.org (Postfix) id 5492E6B0073; Wed, 17 Aug 2022 06:20:16 -0400 (EDT) Received: by kanga.kvack.org (Postfix, from userid 40) id 4F9616B0074; Wed, 17 Aug 2022 06:20:16 -0400 (EDT) X-Delivered-To: int-list-linux-mm@kvack.org Received: by kanga.kvack.org (Postfix, from userid 63042) id 3E8246B0075; Wed, 17 Aug 2022 06:20:16 -0400 (EDT) X-Delivered-To: linux-mm@kvack.org Received: from relay.hostedemail.com (smtprelay0017.hostedemail.com [216.40.44.17]) by kanga.kvack.org (Postfix) with ESMTP id 312C96B0073 for ; Wed, 17 Aug 2022 06:20:16 -0400 (EDT) Received: from smtpin26.hostedemail.com (a10.router.float.18 [10.200.18.1]) by unirelay06.hostedemail.com (Postfix) with ESMTP id 154E1AAE99 for ; Wed, 17 Aug 2022 10:20:16 +0000 (UTC) X-FDA: 79808689632.26.96FD3D9 Received: from mail-pg1-f169.google.com (mail-pg1-f169.google.com [209.85.215.169]) by imf24.hostedemail.com (Postfix) with ESMTP id 1A7931801F2 for ; Wed, 17 Aug 2022 10:19:50 +0000 (UTC) Received: by mail-pg1-f169.google.com with SMTP id 24so11605059pgr.7 for ; Wed, 17 Aug 2022 03:19:50 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=20210112; h=content-transfer-encoding:mime-version:references:in-reply-to :message-id:date:subject:cc:to:from:from:to:cc; bh=QvbBlM64XGXvCYpSrw2ZhXubC7osA9SaKrvHemruWUk=; b=OH1i67TRQUnuhYunmmI/MYZkQu3kqwPcoWFksU+md4pWKAZnkbL8LMUnQFZcQ2ihXg AXvgKmdFw8N9HiH4T1CGwUKqhKiLDSXrAw1D6savzF8DUwAN1q0NeZNH18ESnyEIQGu6 nugZqGOPIbMKIQDiyvg9qiWA1W+EysZuIoPc58j95+BudKMg4WPJiSVfrxmz36ASlbSh Xd07i+oFEYQ7f2iWznifckbG5x6HlGnLRSUwyz0uFBT2C0TNca2HBDwbWCXuxm3Pel3N uxY6CHinDJ/cnbjrWvKy3JqwZ9dHWsFlPLb2AlhKDtCzlwsdULxFoAWm6UV0hO7fTKtB +wnQ== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20210112; h=content-transfer-encoding:mime-version:references:in-reply-to :message-id:date:subject:cc:to:from:x-gm-message-state:from:to:cc; bh=QvbBlM64XGXvCYpSrw2ZhXubC7osA9SaKrvHemruWUk=; b=aUPb7D5gfu4uXopOjVwlobBa4ztw0GuUv3Xfd/GdIA+57WerxnRcOOCBD6Vb8mmLKt +9bZVwDz73R6kJ2OrpmmLTukQdwuA/z/CRe5vLU06H2vHQ41cn7b/a7NXyzRAH0I+Iph tyyhuHtN27c+95IKtppjN65zdc0PKvq2q9F6tyAhAfUR6nr7urjYooDpuDiRf0ygREFK 3TBifi08cn/fJcvHjutYDTxb4+P784zQ8Lasg6LPjOZNX0loNEAp9IPXefHDm0uA/ggp htgZqRGndmUGMaknG/OkTcIOfeBvGgAG5to13YFZ+AH5JkZSruWAnFTUdc3NRfQD2LLH cicg== X-Gm-Message-State: ACgBeo2V/2MkzGtC2lQxk/6eB3i6RhAS5/mLUfVyHEmm39/JmlKIz/a7 UibHzgVB6kLiyuVjIgR81FU= X-Google-Smtp-Source: AA6agR5BivBlhg3XB+mK1tMfAb8bN7Obm740H6tldqe9nQKKdLaKub/ufv6AC+sNXwisZ+Ihenb5GA== X-Received: by 2002:a05:6a00:1c53:b0:52d:d673:2241 with SMTP id s19-20020a056a001c5300b0052dd6732241mr24646290pfw.71.1660731590183; Wed, 17 Aug 2022 03:19:50 -0700 (PDT) Received: from hyeyoo.. ([114.29.91.56]) by smtp.gmail.com with ESMTPSA id d8-20020a170903230800b00172633fc236sm1071318plh.174.2022.08.17.03.19.46 (version=TLS1_3 cipher=TLS_AES_256_GCM_SHA384 bits=256/256); Wed, 17 Aug 2022 03:19:49 -0700 (PDT) From: Hyeonggon Yoo <42.hyeyoo@gmail.com> To: Christoph Lameter , Pekka Enberg , David Rientjes , Joonsoo Kim , Andrew Morton , Vlastimil Babka , Roman Gushchin Cc: Hyeonggon Yoo <42.hyeyoo@gmail.com>, linux-mm@kvack.org, linux-kernel@vger.kernel.org, Marco Elver Subject: [PATCH v4 17/17] mm/sl[au]b: check if large object is valid in __ksize() Date: Wed, 17 Aug 2022 19:18:26 +0900 Message-Id: <20220817101826.236819-18-42.hyeyoo@gmail.com> X-Mailer: git-send-email 2.32.0 In-Reply-To: <20220817101826.236819-1-42.hyeyoo@gmail.com> References: <20220817101826.236819-1-42.hyeyoo@gmail.com> MIME-Version: 1.0 ARC-Seal: i=1; s=arc-20220608; d=hostedemail.com; t=1660731591; a=rsa-sha256; cv=none; b=xpzJSMfMckAi/lN01OvVmkHOqfHiZQxkVvdxrCXoC9MBOMq4dcFv9KtGiqogSc4TqvN6Yu qXwV8H9RoTv67ryUKZZsHs6r/LgFomDjmp6gDLDgXTOPumJGWuB/hd3h4v4rJJ+As4UAbF biS+GJnpNRrBtIMwi+57E1EvQMYI8mM= ARC-Authentication-Results: i=1; imf24.hostedemail.com; dkim=pass header.d=gmail.com header.s=20210112 header.b=OH1i67TR; spf=pass (imf24.hostedemail.com: domain of 42.hyeyoo@gmail.com designates 209.85.215.169 as permitted sender) smtp.mailfrom=42.hyeyoo@gmail.com; dmarc=pass (policy=none) header.from=gmail.com ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=hostedemail.com; s=arc-20220608; t=1660731591; h=from:from:sender:reply-to:subject:subject:date:date: message-id:message-id:to:to:cc:cc:mime-version:mime-version: content-type:content-transfer-encoding:content-transfer-encoding: in-reply-to:in-reply-to:references:references:dkim-signature; bh=QvbBlM64XGXvCYpSrw2ZhXubC7osA9SaKrvHemruWUk=; b=wC9SD3OVyV/mPhShxJBJyB8Io9qdXXp71ampARHwSnTfneqpHbY+LAcIVSQ1/NjwpevLHf RmOWUA0b/CP9qFu9W0kgm37tRNlaLhGxosg2o+pl6op16pR5Vnct9Z6nrfGwSv6qZuURoH JYiwTxV5/nr52mUsBYlmEwZv4xBqWaU= X-Rspamd-Server: rspam01 X-Rspamd-Queue-Id: 1A7931801F2 X-Rspam-User: Authentication-Results: imf24.hostedemail.com; dkim=pass header.d=gmail.com header.s=20210112 header.b=OH1i67TR; spf=pass (imf24.hostedemail.com: domain of 42.hyeyoo@gmail.com designates 209.85.215.169 as permitted sender) smtp.mailfrom=42.hyeyoo@gmail.com; dmarc=pass (policy=none) header.from=gmail.com X-Stat-Signature: nod57adw6q54kiapn1o79z3i6go8h1ud X-HE-Tag: 1660731590-294158 X-Bogosity: Ham, tests=bogofilter, spamicity=0.000000, version=1.2.4 Sender: owner-linux-mm@kvack.org Precedence: bulk X-Loop: owner-majordomo@kvack.org List-ID: If address of large object is not beginning of folio or size of the folio is too small, it must be invalid. BUG() in such cases. Cc: Marco Elver Suggested-by: Vlastimil Babka Signed-off-by: Hyeonggon Yoo <42.hyeyoo@gmail.com> Reviewed-by: Vlastimil Babka --- mm/slab_common.c | 5 ++++- 1 file changed, 4 insertions(+), 1 deletion(-) diff --git a/mm/slab_common.c b/mm/slab_common.c index 9c273a5fb0d7..98d029212682 100644 --- a/mm/slab_common.c +++ b/mm/slab_common.c @@ -984,8 +984,11 @@ size_t __ksize(const void *object) folio = virt_to_folio(object); - if (unlikely(!folio_test_slab(folio))) + if (unlikely(!folio_test_slab(folio))) { + BUG_ON(folio_size(folio) <= KMALLOC_MAX_CACHE_SIZE); + BUG_ON(object != folio_address(folio)); return folio_size(folio); + } return slab_ksize(folio_slab(folio)->slab_cache); }