From patchwork Fri Apr 29 12:30:44 2022 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Muchun Song X-Patchwork-Id: 12831944 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from kanga.kvack.org (kanga.kvack.org [205.233.56.17]) by smtp.lore.kernel.org (Postfix) with ESMTP id 4218BC433FE for ; Fri, 29 Apr 2022 12:31:48 +0000 (UTC) Received: by kanga.kvack.org (Postfix) id A40896B0071; Fri, 29 Apr 2022 08:31:47 -0400 (EDT) Received: by kanga.kvack.org (Postfix, from userid 40) id 9F0006B0073; Fri, 29 Apr 2022 08:31:47 -0400 (EDT) X-Delivered-To: int-list-linux-mm@kvack.org Received: by kanga.kvack.org (Postfix, from userid 63042) id 890936B0074; Fri, 29 Apr 2022 08:31:47 -0400 (EDT) X-Delivered-To: linux-mm@kvack.org Received: from relay.hostedemail.com (relay.hostedemail.com [64.99.140.27]) by kanga.kvack.org (Postfix) with ESMTP id 79F5A6B0071 for ; Fri, 29 Apr 2022 08:31:47 -0400 (EDT) Received: from smtpin13.hostedemail.com (a10.router.float.18 [10.200.18.1]) by unirelay13.hostedemail.com (Postfix) with ESMTP id 522FD60382 for ; Fri, 29 Apr 2022 12:31:47 +0000 (UTC) X-FDA: 79409853054.13.AD1D0D9 Received: from mail-pj1-f46.google.com (mail-pj1-f46.google.com [209.85.216.46]) by imf25.hostedemail.com (Postfix) with ESMTP id DB408A0073 for ; Fri, 29 Apr 2022 12:31:36 +0000 (UTC) Received: by mail-pj1-f46.google.com with SMTP id e24so7008999pjt.2 for ; Fri, 29 Apr 2022 05:31:45 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=bytedance-com.20210112.gappssmtp.com; s=20210112; h=from:to:cc:subject:date:message-id:mime-version :content-transfer-encoding; bh=Eg2Tfir9pHru+0y52IDTTuwENJ/XlBb6d6nsn8kHlL4=; b=JuupZ9lE2kGKdecH2KpkpiMHWO/wtQ7tX4+41Qums/5G/pWbUqQPJZlIuahxN5xluJ T7IrgGT1P0qfwTWTiwB3b7s1CPIPLY8xyNVZ2hZ7HH4I3P8j4x0E4DvnyoXKXfDavfHi q4/xtM/T/GcP9rUnt/SgqCx+mye/LcrqSNP0iuMtH5p67Dl6oG4bX/6OtH3TCmKZwYIL cHDQibv3XgelOVc1pDq9kNI4ALUbJ1LE3OeBFI+wfXYgLlUcT45Qk7AcpFzujZ4L0Kfn MzhaNBQMTEGxx7I7iRcz2a2p6Uc/89w6b9G2ZHT2EifaYdsBATND7ZPCnP7LrG9zvuJg 1DwA== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20210112; h=x-gm-message-state:from:to:cc:subject:date:message-id:mime-version :content-transfer-encoding; bh=Eg2Tfir9pHru+0y52IDTTuwENJ/XlBb6d6nsn8kHlL4=; b=ye4diydFOgpU9B4+FW/7tqWrsFT6AaIi0hdg7MuPlCkxrc0C70JPMtyBjFsZYwhaFo B3F1+XbarXI6hmyyrIFh5W1UV/uVGcF0e+tRNF5bZb/KOzUkH9zMnZApZIGOre54Sneh gA5xPVbSQGMdqO8fthuIsE2gCZsdMFKecH0IVeEgb5nrafx8yd6s2G7K8sRhYF61LcFq DQ92yxlA40YK1N4wrIazAwNTM9eUb1iWH8W+LuhL0Sm47kjvGuI6E4MocV4CLZKuUt/l BMs8YmcoFhpkZgXx3JH0smA/o0x25ZBxFCWAJMPYZG1M8wq/o7Kv3F6quLLyR5lk3JQ0 L1Tg== X-Gm-Message-State: AOAM53125/syQWZt/vUDyOJxrVkbgCUaWFbidsp6tKSSO0P9D68JWpAo HrnGrEk4oJI/2Aixls38yUJ5kQ== X-Google-Smtp-Source: ABdhPJyfNbj68vTqtTt4TeAGXgknLMQzbc+FFu6vcDfuPJJMbbJx7qkp2oU8JLINGNSf2tFQrvct1w== X-Received: by 2002:a17:902:7c13:b0:156:ca91:877f with SMTP id x19-20020a1709027c1300b00156ca91877fmr38219769pll.15.1651235504904; Fri, 29 Apr 2022 05:31:44 -0700 (PDT) Received: from FVFYT0MHHV2J.bytedance.net ([139.177.225.239]) by smtp.gmail.com with ESMTPSA id g15-20020a056a000b8f00b004fa9dbf27desm3294511pfj.55.2022.04.29.05.31.40 (version=TLS1_3 cipher=TLS_AES_256_GCM_SHA384 bits=256/256); Fri, 29 Apr 2022 05:31:44 -0700 (PDT) From: Muchun Song To: cl@linux.com, penberg@kernel.org, rientjes@google.com, iamjoonsoo.kim@lge.com, akpm@linux-foundation.org, vbabka@suse.cz, roman.gushchin@linux.dev Cc: linux-mm@kvack.org, linux-kernel@vger.kernel.org, smuchun@gmail.com, Muchun Song Subject: [PATCH v2] mm: slab: optimize memcg_slab_free_hook() Date: Fri, 29 Apr 2022 20:30:44 +0800 Message-Id: <20220429123044.37885-1-songmuchun@bytedance.com> X-Mailer: git-send-email 2.32.0 (Apple Git-132) MIME-Version: 1.0 Authentication-Results: imf25.hostedemail.com; dkim=pass header.d=bytedance-com.20210112.gappssmtp.com header.s=20210112 header.b=JuupZ9lE; dmarc=pass (policy=none) header.from=bytedance.com; spf=pass (imf25.hostedemail.com: domain of songmuchun@bytedance.com designates 209.85.216.46 as permitted sender) smtp.mailfrom=songmuchun@bytedance.com X-Rspamd-Server: rspam06 X-Rspamd-Queue-Id: DB408A0073 X-Rspam-User: X-Stat-Signature: 63tg43xxjue4cssymksdgwcc9d45wx6x X-HE-Tag: 1651235496-190413 X-Bogosity: Ham, tests=bogofilter, spamicity=0.000000, version=1.2.4 Sender: owner-linux-mm@kvack.org Precedence: bulk X-Loop: owner-majordomo@kvack.org List-ID: Most callers of memcg_slab_free_hook() already know the slab, which could be passed to memcg_slab_free_hook() directly to reduce the overhead of an another call of virt_to_slab(). For bulk freeing of objects, the call of slab_objcgs() in the loop in memcg_slab_free_hook() is redundant as well. Rework memcg_slab_free_hook() and build_detached_freelist() to reduce those unnecessary overhead and make memcg_slab_free_hook() can handle bulk freeing in slab_free(). Move the calling site of memcg_slab_free_hook() from do_slab_free() to slab_free() for slub to make the code clearer since the logic is weird (e.g. the caller need to judge whether it needs to call memcg_slab_free_hook()). It is easy to make mistakes like missing calling of memcg_slab_free_hook() like fixes of: commit d1b2cf6cb84a ("mm: memcg/slab: uncharge during kmem_cache_free_bulk()") commit ae085d7f9365 ("mm: kfence: fix missing objcg housekeeping for SLAB") This optimization is mainly for bulk objects freeing. The following numbers is shown for 16-object freeing. before after kmem_cache_free_bulk: ~430 ns ~400 ns The overhead is reduced by about 7% for 16-object freeing. Signed-off-by: Muchun Song Reviewed-by: Hyeonggon Yoo <42.hyeyoo@gmail.com> --- v2: - Add numbers to commit log. mm/slab.c | 4 ++-- mm/slab.h | 30 ++++++++--------------------- mm/slub.c | 66 +++++++++++++++++++++------------------------------------------ 3 files changed, 32 insertions(+), 68 deletions(-) diff --git a/mm/slab.c b/mm/slab.c index f8cd00f4ba13..2174962055ae 100644 --- a/mm/slab.c +++ b/mm/slab.c @@ -3406,9 +3406,10 @@ static __always_inline void __cache_free(struct kmem_cache *cachep, void *objp, { bool init; + memcg_slab_free_hook(cachep, virt_to_slab(objp), &objp, 1); + if (is_kfence_address(objp)) { kmemleak_free_recursive(objp, cachep->flags); - memcg_slab_free_hook(cachep, &objp, 1); __kfence_free(objp); return; } @@ -3441,7 +3442,6 @@ void ___cache_free(struct kmem_cache *cachep, void *objp, check_irq_off(); kmemleak_free_recursive(objp, cachep->flags); objp = cache_free_debugcheck(cachep, objp, caller); - memcg_slab_free_hook(cachep, &objp, 1); /* * Skip calling cache_free_alien() when the platform is not numa. diff --git a/mm/slab.h b/mm/slab.h index db9fb5c8dae7..a8d5eb1c323f 100644 --- a/mm/slab.h +++ b/mm/slab.h @@ -547,36 +547,22 @@ static inline void memcg_slab_post_alloc_hook(struct kmem_cache *s, obj_cgroup_put(objcg); } -static inline void memcg_slab_free_hook(struct kmem_cache *s_orig, +static inline void memcg_slab_free_hook(struct kmem_cache *s, struct slab *slab, void **p, int objects) { - struct kmem_cache *s; struct obj_cgroup **objcgs; - struct obj_cgroup *objcg; - struct slab *slab; - unsigned int off; int i; if (!memcg_kmem_enabled()) return; - for (i = 0; i < objects; i++) { - if (unlikely(!p[i])) - continue; - - slab = virt_to_slab(p[i]); - /* we could be given a kmalloc_large() object, skip those */ - if (!slab) - continue; - - objcgs = slab_objcgs(slab); - if (!objcgs) - continue; + objcgs = slab_objcgs(slab); + if (!objcgs) + return; - if (!s_orig) - s = slab->slab_cache; - else - s = s_orig; + for (i = 0; i < objects; i++) { + struct obj_cgroup *objcg; + unsigned int off; off = obj_to_index(s, slab, p[i]); objcg = objcgs[off]; @@ -628,7 +614,7 @@ static inline void memcg_slab_post_alloc_hook(struct kmem_cache *s, { } -static inline void memcg_slab_free_hook(struct kmem_cache *s, +static inline void memcg_slab_free_hook(struct kmem_cache *s, struct slab *slab, void **p, int objects) { } diff --git a/mm/slub.c b/mm/slub.c index 1f699ddfff7f..3794afe32b5f 100644 --- a/mm/slub.c +++ b/mm/slub.c @@ -3435,9 +3435,6 @@ static __always_inline void do_slab_free(struct kmem_cache *s, struct kmem_cache_cpu *c; unsigned long tid; - /* memcg_slab_free_hook() is already called for bulk free. */ - if (!tail) - memcg_slab_free_hook(s, &head, 1); redo: /* * Determine the currently cpus per cpu slab. @@ -3497,9 +3494,10 @@ static __always_inline void do_slab_free(struct kmem_cache *s, } static __always_inline void slab_free(struct kmem_cache *s, struct slab *slab, - void *head, void *tail, int cnt, + void *head, void *tail, void **p, int cnt, unsigned long addr) { + memcg_slab_free_hook(s, slab, p, cnt); /* * With KASAN enabled slab_free_freelist_hook modifies the freelist * to remove objects, whose reuse must be delayed. @@ -3521,7 +3519,7 @@ void kmem_cache_free(struct kmem_cache *s, void *x) if (!s) return; trace_kmem_cache_free(_RET_IP_, x, s->name); - slab_free(s, virt_to_slab(x), x, NULL, 1, _RET_IP_); + slab_free(s, virt_to_slab(x), x, NULL, &x, 1, _RET_IP_); } EXPORT_SYMBOL(kmem_cache_free); @@ -3562,79 +3560,59 @@ static inline int build_detached_freelist(struct kmem_cache *s, size_t size, void **p, struct detached_freelist *df) { - size_t first_skipped_index = 0; int lookahead = 3; void *object; struct folio *folio; - struct slab *slab; - - /* Always re-init detached_freelist */ - df->slab = NULL; - - do { - object = p[--size]; - /* Do we need !ZERO_OR_NULL_PTR(object) here? (for kfree) */ - } while (!object && size); - - if (!object) - return 0; + size_t same; + object = p[--size]; folio = virt_to_folio(object); if (!s) { /* Handle kalloc'ed objects */ if (unlikely(!folio_test_slab(folio))) { free_large_kmalloc(folio, object); - p[size] = NULL; /* mark object processed */ + df->slab = NULL; return size; } /* Derive kmem_cache from object */ - slab = folio_slab(folio); - df->s = slab->slab_cache; + df->slab = folio_slab(folio); + df->s = df->slab->slab_cache; } else { - slab = folio_slab(folio); + df->slab = folio_slab(folio); df->s = cache_from_obj(s, object); /* Support for memcg */ } - if (is_kfence_address(object)) { - slab_free_hook(df->s, object, false); - __kfence_free(object); - p[size] = NULL; /* mark object processed */ - return size; - } - /* Start new detached freelist */ - df->slab = slab; - set_freepointer(df->s, object, NULL); df->tail = object; df->freelist = object; - p[size] = NULL; /* mark object processed */ df->cnt = 1; + if (is_kfence_address(object)) + return size; + + set_freepointer(df->s, object, NULL); + + same = size; while (size) { object = p[--size]; - if (!object) - continue; /* Skip processed objects */ - /* df->slab is always set at this point */ if (df->slab == virt_to_slab(object)) { /* Opportunity build freelist */ set_freepointer(df->s, object, df->freelist); df->freelist = object; df->cnt++; - p[size] = NULL; /* mark object processed */ - + same--; + if (size != same) + swap(p[size], p[same]); continue; } /* Limit look ahead search */ if (!--lookahead) break; - - if (!first_skipped_index) - first_skipped_index = size + 1; } - return first_skipped_index; + return same; } /* Note that interrupts must be enabled when calling this function. */ @@ -3643,7 +3621,6 @@ void kmem_cache_free_bulk(struct kmem_cache *s, size_t size, void **p) if (WARN_ON(!size)) return; - memcg_slab_free_hook(s, p, size); do { struct detached_freelist df; @@ -3651,7 +3628,8 @@ void kmem_cache_free_bulk(struct kmem_cache *s, size_t size, void **p) if (!df.slab) continue; - slab_free(df.s, df.slab, df.freelist, df.tail, df.cnt, _RET_IP_); + slab_free(df.s, df.slab, df.freelist, df.tail, &p[size], df.cnt, + _RET_IP_); } while (likely(size)); } EXPORT_SYMBOL(kmem_cache_free_bulk); @@ -4554,7 +4532,7 @@ void kfree(const void *x) return; } slab = folio_slab(folio); - slab_free(slab->slab_cache, slab, object, NULL, 1, _RET_IP_); + slab_free(slab->slab_cache, slab, object, NULL, &object, 1, _RET_IP_); } EXPORT_SYMBOL(kfree);