From patchwork Mon Nov 25 17:16:17 2024 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Shakeel Butt X-Patchwork-Id: 13885153 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from kanga.kvack.org (kanga.kvack.org [205.233.56.17]) by smtp.lore.kernel.org (Postfix) with ESMTP id 860DAD58D69 for ; Mon, 25 Nov 2024 17:16:38 +0000 (UTC) Received: by kanga.kvack.org (Postfix) id 068726B0085; Mon, 25 Nov 2024 12:16:38 -0500 (EST) Received: by kanga.kvack.org (Postfix, from userid 40) id 018CC6B0088; Mon, 25 Nov 2024 12:16:37 -0500 (EST) X-Delivered-To: int-list-linux-mm@kvack.org Received: by kanga.kvack.org (Postfix, from userid 63042) id E221C6B0089; Mon, 25 Nov 2024 12:16:37 -0500 (EST) X-Delivered-To: linux-mm@kvack.org Received: from relay.hostedemail.com (smtprelay0016.hostedemail.com [216.40.44.16]) by kanga.kvack.org (Postfix) with ESMTP id C37BC6B0085 for ; Mon, 25 Nov 2024 12:16:37 -0500 (EST) Received: from smtpin02.hostedemail.com (a10.router.float.18 [10.200.18.1]) by unirelay09.hostedemail.com (Postfix) with ESMTP id 40DA980BFE for ; Mon, 25 Nov 2024 17:16:37 +0000 (UTC) X-FDA: 82825271592.02.B4ECCB1 Received: from out-184.mta0.migadu.com (out-184.mta0.migadu.com [91.218.175.184]) by imf24.hostedemail.com (Postfix) with ESMTP id 02324180025 for ; Mon, 25 Nov 2024 17:16:34 +0000 (UTC) Authentication-Results: imf24.hostedemail.com; dkim=pass header.d=linux.dev header.s=key1 header.b=RUX7TPDY; spf=pass (imf24.hostedemail.com: domain of shakeel.butt@linux.dev designates 91.218.175.184 as permitted sender) smtp.mailfrom=shakeel.butt@linux.dev; dmarc=pass (policy=none) header.from=linux.dev ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=hostedemail.com; s=arc-20220608; t=1732554991; h=from:from:sender:reply-to:subject:subject:date:date: message-id:message-id:to:to:cc:cc:mime-version:mime-version: content-type:content-transfer-encoding:content-transfer-encoding: in-reply-to:references:dkim-signature; bh=OOy1ir+JYbbaetd7l4qfetb+lCDRU+An6aa01S+lbiE=; b=S4CxtLlvAY+MVNB45R0on7SKpRChLEsY5Q69SC9vz/R208NhhkUZDZaLLx/MaWDCIs7N88 YomBgs/IUdC4NAYTOMqcQ7WxrUytka6kaoLoQqbVKxr5r+Yivgn+0Cg701iFSuLCEHcPSG DJklIKNDdbmnTib3r/9Lm5eb6pGfD0o= ARC-Authentication-Results: i=1; imf24.hostedemail.com; dkim=pass header.d=linux.dev header.s=key1 header.b=RUX7TPDY; spf=pass (imf24.hostedemail.com: domain of shakeel.butt@linux.dev designates 91.218.175.184 as permitted sender) smtp.mailfrom=shakeel.butt@linux.dev; dmarc=pass (policy=none) header.from=linux.dev ARC-Seal: i=1; s=arc-20220608; d=hostedemail.com; t=1732554991; a=rsa-sha256; cv=none; b=euOruuRqfSHHayLiqRwd4oLLnI+queJq6HXS5lTfWNwhpYGqKJKP0Pu5uEd/Lx2xLA82vv XZYdZsmhHKsoOAodeSO78W1MOEIBAbQh1P/vk+6OH9R3/oODa8mdptCvk8+/hIlphvixHB WnBuCTfqVrxKimcApN3Oy+4+USqdBB4= X-Report-Abuse: Please report any abuse attempt to abuse@migadu.com and include these headers. DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=linux.dev; s=key1; t=1732554993; h=from:from:reply-to:subject:subject:date:date:message-id:message-id: to:to:cc:cc:mime-version:mime-version: content-transfer-encoding:content-transfer-encoding; bh=OOy1ir+JYbbaetd7l4qfetb+lCDRU+An6aa01S+lbiE=; b=RUX7TPDYANJNZphrQpLzYR0ZU1pPx5mxru1GIAiB1s6o47RUdqUZW0vBiwki+WuKzwcVAc jT8UTbQnmWmuYb87XIm4onSYXKRFnKoWrEaLNkZHbtoTpOetY7BrlzNpXlpRP5QYhRkA0j Cl6Vg7Qup19w65c3TeKp7twfZsobJPg= From: Shakeel Butt To: Andrew Morton Cc: Johannes Weiner , Michal Hocko , Roman Gushchin , Muchun Song , Vlastimil Babka , Axel Rasmussen , Steven Rostedt , Suren Baghdasaryan , Matthew Wilcox , linux-mm@kvack.org, cgroups@vger.kernel.org, linux-kernel@vger.kernel.org, Meta kernel team Subject: [PATCH v2] mm: mmap_lock: optimize mmap_lock tracepoints Date: Mon, 25 Nov 2024 09:16:17 -0800 Message-ID: <20241125171617.113892-1-shakeel.butt@linux.dev> MIME-Version: 1.0 X-Migadu-Flow: FLOW_OUT X-Rspamd-Server: rspam06 X-Rspamd-Queue-Id: 02324180025 X-Stat-Signature: 8z8khnkus5wh81qcmg5yeokoj1kddz53 X-Rspam-User: X-HE-Tag: 1732554994-841531 X-HE-Meta: U2FsdGVkX1+MRrugy8GuNkjS3qUlmnFWIFxmowUsOjliS0O46R08tTdxBdWheVHm10jlCmFi/+3Nt9KFhIKwl+phQNVrxEi5KlWdfd2MgLrceLVYAa1d9mipYnKjba/thU1Q79zIgMsuY4REl6AvvOS7w2cidXnXWqu4mxhEElwkJCAyuav28U56BZoBZxPcU5PYxUByc4YurL7bjz08CfdZRUOVD3QPrPGnIHnOb52/55XE3ylv09xDdszRqWyCa42ayUez5rNYARQdxAjGg1NMJ3xH535mKOHfcV5Ape2J8b6PEYDNwBGqXDn5gvSZ7iopAi1Z5M+IIrGGR+/5pUuAP8SktTQcCS0u2X4T1Duv/lmh/r+omdKT8fvOv75kV77RotXCRE3B0wOcgHXnPtG6UaVVty64/eX72dsOgYvRh41nE39iWQbkZmnoP78y8hI+1tbKrmomk71qFdlqFmLxjULbMyg/eowfwebmLq6Hw+vhXqyF9qYmlMmtYR/ThWT26Nnp6PwpIfqdMG52OZMoW63GfnrwCiO16TvBjiPkw1OqmwWWkJYYsdU3aOTqGM3drhmAJW7fkK+eztcOkJmA1RbhZQPzcOTkQePVF1tGBrjM6sr51OhBfOjG+b/yAcqZ0A9AtOWJE6M5we1K9ccS8TUWapNLAWtA7shOrD7LqEauqWAdd2XxohpAHhM3N1aJoSt5KEFdaoKS10lmfXBJsH0vWYeoZCejmeWJYr6zWR2r1+kyH8NNgFzhR+mz8+SGecPZfVQkZAANuvGgCHZeT/K7TUnKQWRpY8bUvMILskchA+BFJfQhlGwtwL3UfZcoojBA7Q8zwmW1hwKJ5V7/meIPKgJzr1g0n+mtT9+IYsOqrEM0rAr89mZkUeNzHN1caTW1zhqz9a2C4rMh+MIeQtypIqdBvetdEXLIJ+8UXyhwnEU+K+ks4rkzfSzMcmMggYw9Te6Gh/CuY3I CTVkr+Ke nnRa9c60rDft1kkzpjdBPszEb9FEcX8yvJ/Innhic4cBwPewjVj6DEtBICyWlKTcCKjAiMij7eDre4ogrOOIwvS9IKeZbufcq559Iz0iDZs5YyqJiryKcDoTs0Hzkg/X6mH769ESAGM+wO4JPkjPwpPoz5v4T9sExdiS3QudzykoJHO2aPYfyuaUR2AmX59HOSA/4 X-Bogosity: Ham, tests=bogofilter, spamicity=0.000000, version=1.2.4 Sender: owner-linux-mm@kvack.org Precedence: bulk X-Loop: owner-majordomo@kvack.org List-ID: List-Subscribe: List-Unsubscribe: We are starting to deploy mmap_lock tracepoint monitoring across our fleet and the early results showed that these tracepoints are consuming significant amount of CPUs in kernfs_path_from_node when enabled. It seems like the kernel is trying to resolve the cgroup path in the fast path of the locking code path when the tracepoints are enabled. In addition for some application their metrics are regressing when monitoring is enabled. The cgroup path resolution can be slow and should not be done in the fast path. Most userspace tools, like bpftrace, provides functionality to get the cgroup path from cgroup id, so let's just trace the cgroup id and the users can use better tools to get the path in the slow path. Signed-off-by: Shakeel Butt Reviewed-by: Yosry Ahmed --- Changes since v1: - Fixed commit message (Yosry). - Renamed memcg_id_from_mm to cgroup_id_from_mm (Yosry). - Fixed handling of root memcg (Yosry). - Fixed mem_cgroup_disabled() handling. - Kept mm pointer printing based on Steven's feedback. include/linux/memcontrol.h | 22 ++++++++++++++ include/trace/events/mmap_lock.h | 32 ++++++++++---------- mm/mmap_lock.c | 50 ++------------------------------ 3 files changed, 40 insertions(+), 64 deletions(-) diff --git a/include/linux/memcontrol.h b/include/linux/memcontrol.h index 5502aa8e138e..b28180269e75 100644 --- a/include/linux/memcontrol.h +++ b/include/linux/memcontrol.h @@ -1046,6 +1046,23 @@ static inline void memcg_memory_event_mm(struct mm_struct *mm, void split_page_memcg(struct page *head, int old_order, int new_order); +static inline u64 cgroup_id_from_mm(struct mm_struct *mm) +{ + struct mem_cgroup *memcg; + u64 id; + + if (mem_cgroup_disabled()) + return 0; + + rcu_read_lock(); + memcg = mem_cgroup_from_task(rcu_dereference(mm->owner)); + if (!memcg) + memcg = root_mem_cgroup; + id = cgroup_id(memcg->css.cgroup); + rcu_read_unlock(); + return id; +} + #else /* CONFIG_MEMCG */ #define MEM_CGROUP_ID_SHIFT 0 @@ -1466,6 +1483,11 @@ void count_memcg_event_mm(struct mm_struct *mm, enum vm_event_item idx) static inline void split_page_memcg(struct page *head, int old_order, int new_order) { } + +static inline u64 cgroup_id_from_mm(struct mm_struct *mm) +{ + return 0; +} #endif /* CONFIG_MEMCG */ /* diff --git a/include/trace/events/mmap_lock.h b/include/trace/events/mmap_lock.h index bc2e3ad787b3..cf9f9faf8914 100644 --- a/include/trace/events/mmap_lock.h +++ b/include/trace/events/mmap_lock.h @@ -5,6 +5,7 @@ #if !defined(_TRACE_MMAP_LOCK_H) || defined(TRACE_HEADER_MULTI_READ) #define _TRACE_MMAP_LOCK_H +#include #include #include @@ -12,64 +13,61 @@ struct mm_struct; DECLARE_EVENT_CLASS(mmap_lock, - TP_PROTO(struct mm_struct *mm, const char *memcg_path, bool write), + TP_PROTO(struct mm_struct *mm, bool write), - TP_ARGS(mm, memcg_path, write), + TP_ARGS(mm, write), TP_STRUCT__entry( __field(struct mm_struct *, mm) - __string(memcg_path, memcg_path) + __field(u64, memcg_id) __field(bool, write) ), TP_fast_assign( __entry->mm = mm; - __assign_str(memcg_path); + __entry->memcg_id = cgroup_id_from_mm(mm); __entry->write = write; ), TP_printk( - "mm=%p memcg_path=%s write=%s", - __entry->mm, - __get_str(memcg_path), + "mm=%p memcg_id=%llu write=%s", + __entry->mm, __entry->memcg_id, __entry->write ? "true" : "false" ) ); #define DEFINE_MMAP_LOCK_EVENT(name) \ DEFINE_EVENT(mmap_lock, name, \ - TP_PROTO(struct mm_struct *mm, const char *memcg_path, \ - bool write), \ - TP_ARGS(mm, memcg_path, write)) + TP_PROTO(struct mm_struct *mm, bool write), \ + TP_ARGS(mm, write)) DEFINE_MMAP_LOCK_EVENT(mmap_lock_start_locking); DEFINE_MMAP_LOCK_EVENT(mmap_lock_released); TRACE_EVENT(mmap_lock_acquire_returned, - TP_PROTO(struct mm_struct *mm, const char *memcg_path, bool write, - bool success), + TP_PROTO(struct mm_struct *mm, bool write, bool success), - TP_ARGS(mm, memcg_path, write, success), + TP_ARGS(mm, write, success), TP_STRUCT__entry( __field(struct mm_struct *, mm) - __string(memcg_path, memcg_path) + __field(u64, memcg_id) __field(bool, write) __field(bool, success) ), TP_fast_assign( __entry->mm = mm; - __assign_str(memcg_path); + __entry->memcg_id = cgroup_id_from_mm(mm); __entry->write = write; __entry->success = success; ), TP_printk( - "mm=%p memcg_path=%s write=%s success=%s", + "mm=%p memcg_id=%llu write=%s success=%s", __entry->mm, - __get_str(memcg_path), + __entry->memcg_id, __entry->write ? "true" : "false", __entry->success ? "true" : "false" ) diff --git a/mm/mmap_lock.c b/mm/mmap_lock.c index f186d57df2c6..e7dbaf96aa17 100644 --- a/mm/mmap_lock.c +++ b/mm/mmap_lock.c @@ -17,51 +17,7 @@ EXPORT_TRACEPOINT_SYMBOL(mmap_lock_start_locking); EXPORT_TRACEPOINT_SYMBOL(mmap_lock_acquire_returned); EXPORT_TRACEPOINT_SYMBOL(mmap_lock_released); -#ifdef CONFIG_MEMCG - -/* - * Size of the buffer for memcg path names. Ignoring stack trace support, - * trace_events_hist.c uses MAX_FILTER_STR_VAL for this, so we also use it. - */ -#define MEMCG_PATH_BUF_SIZE MAX_FILTER_STR_VAL - -#define TRACE_MMAP_LOCK_EVENT(type, mm, ...) \ - do { \ - if (trace_mmap_lock_##type##_enabled()) { \ - char buf[MEMCG_PATH_BUF_SIZE]; \ - get_mm_memcg_path(mm, buf, sizeof(buf)); \ - trace_mmap_lock_##type(mm, buf, ##__VA_ARGS__); \ - } \ - } while (0) - -#else /* !CONFIG_MEMCG */ - -#define TRACE_MMAP_LOCK_EVENT(type, mm, ...) \ - trace_mmap_lock_##type(mm, "", ##__VA_ARGS__) - -#endif /* CONFIG_MEMCG */ - #ifdef CONFIG_TRACING -#ifdef CONFIG_MEMCG -/* - * Write the given mm_struct's memcg path to a buffer. If the path cannot be - * determined, empty string is written. - */ -static void get_mm_memcg_path(struct mm_struct *mm, char *buf, size_t buflen) -{ - struct mem_cgroup *memcg; - - buf[0] = '\0'; - memcg = get_mem_cgroup_from_mm(mm); - if (memcg == NULL) - return; - if (memcg->css.cgroup) - cgroup_path(memcg->css.cgroup, buf, buflen); - css_put(&memcg->css); -} - -#endif /* CONFIG_MEMCG */ - /* * Trace calls must be in a separate file, as otherwise there's a circular * dependency between linux/mmap_lock.h and trace/events/mmap_lock.h. @@ -69,20 +25,20 @@ static void get_mm_memcg_path(struct mm_struct *mm, char *buf, size_t buflen) void __mmap_lock_do_trace_start_locking(struct mm_struct *mm, bool write) { - TRACE_MMAP_LOCK_EVENT(start_locking, mm, write); + trace_mmap_lock_start_locking(mm, write); } EXPORT_SYMBOL(__mmap_lock_do_trace_start_locking); void __mmap_lock_do_trace_acquire_returned(struct mm_struct *mm, bool write, bool success) { - TRACE_MMAP_LOCK_EVENT(acquire_returned, mm, write, success); + trace_mmap_lock_acquire_returned(mm, write, success); } EXPORT_SYMBOL(__mmap_lock_do_trace_acquire_returned); void __mmap_lock_do_trace_released(struct mm_struct *mm, bool write) { - TRACE_MMAP_LOCK_EVENT(released, mm, write); + trace_mmap_lock_released(mm, write); } EXPORT_SYMBOL(__mmap_lock_do_trace_released); #endif /* CONFIG_TRACING */