From patchwork Mon Jan 13 05:20:14 2025 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Chun-Tse Shao X-Patchwork-Id: 13936743 Received: from mail-pl1-f201.google.com (mail-pl1-f201.google.com [209.85.214.201]) (using TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 (128/128 bits)) (No client certificate requested) by smtp.subspace.kernel.org (Postfix) with ESMTPS id 944AA1BD014 for ; Mon, 13 Jan 2025 05:22:48 +0000 (UTC) Authentication-Results: smtp.subspace.kernel.org; arc=none smtp.client-ip=209.85.214.201 ARC-Seal: i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1736745770; cv=none; b=gwGCruzl7G/ZhP1gR+nfmbn8v1CWKekz6pVEot5SfTOr4D5gBBgkVuaG+7TNj6b970nRba4FTK9tQMCiQ1lqnxESaEjDrJ/ooZSvwq2j4/lhuwybXf+MUk2xaYnxP2IwNNeIjIA80xHMKXjraAqJv4S/wfQCYo9/J6MWvw5wWUc= ARC-Message-Signature: i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1736745770; c=relaxed/simple; bh=PdTe3jFnoCLWzm3kmTMuOUCvtR+GmRocsG2mc9gfwJE=; h=Date:In-Reply-To:Mime-Version:References:Message-ID:Subject:From: To:Cc:Content-Type; b=tCVgOOUeHLNivJle8GoBds1VwY4xYagUwY+PfO/s3op9Rk+6EtN8jTAIYuZtvwuqOcTb0UzM3nStrlD0FiJqmp+ULVfMyJWnp6iIplpmndSOjlVGdtVXDp7ZhzxRB6eA/U6Ui8bndQg+Y7Gi3vNBnK1Ne9HI0wpNnQDUENeeRsM= ARC-Authentication-Results: i=1; smtp.subspace.kernel.org; dmarc=pass (p=reject dis=none) header.from=google.com; spf=pass smtp.mailfrom=flex--ctshao.bounces.google.com; dkim=pass (2048-bit key) header.d=google.com header.i=@google.com header.b=XTy6+AD5; arc=none smtp.client-ip=209.85.214.201 Authentication-Results: smtp.subspace.kernel.org; dmarc=pass (p=reject dis=none) header.from=google.com Authentication-Results: smtp.subspace.kernel.org; spf=pass smtp.mailfrom=flex--ctshao.bounces.google.com Authentication-Results: smtp.subspace.kernel.org; dkim=pass (2048-bit key) header.d=google.com header.i=@google.com header.b="XTy6+AD5" Received: by mail-pl1-f201.google.com with SMTP id d9443c01a7336-21681a2c0d5so70088665ad.2 for ; Sun, 12 Jan 2025 21:22:48 -0800 (PST) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=google.com; s=20230601; t=1736745768; x=1737350568; darn=vger.kernel.org; h=cc:to:from:subject:message-id:references:mime-version:in-reply-to :date:from:to:cc:subject:date:message-id:reply-to; bh=QOc+Zq7RKw8PAGfDRJz3cORnG4jrHFHNhA/YUYdk0Pc=; b=XTy6+AD5WymuFRHb+EP/IS3Tj3Cttqx3mYARK8FWDBP6jImcIRXZ7ZWCV44Sp/1Xzp eX6XgE7Pk57tf3dVuXzVPCAgzlQSYqVDZhRZgeYFxG5GYWELkEvg8A89n/0keRrxjjsU DkOa2p2yBKcH6A9LuUGX3ZRyMc9eeaTaYw28egdkwr2keqfPiLIB2sY+f7Iz++wB3FOr CwxJHAwFxnAsErw9tUpZFR5u295jC0KWBSOfGW2NDzG9zkqAWitEyjuQap/q+GkiQqpW t37xAgwLmrcntI7sacNtgEAaOKTQcJaF694WoNEilalnIVLP0GJaBQ6xSY4cQZOP0IMg hwcA== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20230601; t=1736745768; x=1737350568; h=cc:to:from:subject:message-id:references:mime-version:in-reply-to :date:x-gm-message-state:from:to:cc:subject:date:message-id:reply-to; bh=QOc+Zq7RKw8PAGfDRJz3cORnG4jrHFHNhA/YUYdk0Pc=; b=xBQ6+fT54kjCXdEnCz63e5HBOs/buYFIEHNlVh6nkeSHg/vbUZA00oo9NKuWmnVdHq NKTJuxkzHgVrp2RIR+CMIZP6DZ5rj0p7/M947futo6d+zRXX57vZqUHC8EgVJmTP9O5P 1KtniMD/YZF1zXARMul1czPkSwlNr8PtX/0uevgzGWC4Kh8s9CyZNj6i+HGjXpE/fn9s 4CGaxVAk0bfhK+Z0QRjTRAq/wMZ8Wp72Nm7PJNng/qso5L8JkkUKPBAYkFR+J6WayWrV Tmi/hRskhDpL1lxvLnMMyg1wOu7EF08psLD+A3wElRO54uT7a8goU7T24W7cT4aoiq/0 tNpA== X-Forwarded-Encrypted: i=1; AJvYcCXMEqLYp71w5oGmznC7cwROFI8oO1PEVrOV8WsGmI1KMFcyPHfiR6LeBfoU5hzk/aRROf8=@vger.kernel.org X-Gm-Message-State: AOJu0YyVWaI+ZRC/n/rGBwoEhPLhWoYn6VGPFFBfo9b/DtHmbaI3/YmY CiEfyf1ltFQ8jThewnqmP6qZaLFmjAFZ4xr+pZoZ5O8jhvBbS/NlMDz4CC6liddvYwMP75D8//2 Zbw== X-Google-Smtp-Source: AGHT+IHH/Mxk3EqK90R+Z9E4lysnhPshUJ3SCSQXzHeG1/4fEs9Zwt+JxdWBKH1V7xSGCQ4evVrdtYgZbdw= X-Received: from pfay7.prod.google.com ([2002:a05:6a00:1807:b0:72a:89d4:9641]) (user=ctshao job=prod-delivery.src-stubby-dispatcher) by 2002:a05:6a20:d045:b0:1e1:afa9:d38a with SMTP id adf61e73a8af0-1e88d0dfa70mr29764707637.1.1736745768047; Sun, 12 Jan 2025 21:22:48 -0800 (PST) Date: Sun, 12 Jan 2025 21:20:14 -0800 In-Reply-To: <20250113052220.2105645-1-ctshao@google.com> Precedence: bulk X-Mailing-List: bpf@vger.kernel.org List-Id: List-Subscribe: List-Unsubscribe: Mime-Version: 1.0 References: <20250113052220.2105645-1-ctshao@google.com> X-Mailer: git-send-email 2.47.1.688.g23fc6f90ad-goog Message-ID: <20250113052220.2105645-2-ctshao@google.com> Subject: [PATCH v2 1/4] perf lock: Add bpf maps for owner stack tracing From: Chun-Tse Shao To: linux-kernel@vger.kernel.org Cc: Chun-Tse Shao , peterz@infradead.org, mingo@redhat.com, acme@kernel.org, namhyung@kernel.org, mark.rutland@arm.com, alexander.shishkin@linux.intel.com, jolsa@kernel.org, irogers@google.com, adrian.hunter@intel.com, kan.liang@linux.intel.com, linux-perf-users@vger.kernel.org, bpf@vger.kernel.org Add few bpf maps in order to tracing owner stack. Signed-off-by: Chun-Tse Shao --- tools/perf/util/bpf_lock_contention.c | 17 ++++++-- .../perf/util/bpf_skel/lock_contention.bpf.c | 40 +++++++++++++++++-- tools/perf/util/bpf_skel/lock_data.h | 6 +++ 3 files changed, 56 insertions(+), 7 deletions(-) diff --git a/tools/perf/util/bpf_lock_contention.c b/tools/perf/util/bpf_lock_contention.c index 41a1ad087895..c9c58f243ceb 100644 --- a/tools/perf/util/bpf_lock_contention.c +++ b/tools/perf/util/bpf_lock_contention.c @@ -41,9 +41,20 @@ int lock_contention_prepare(struct lock_contention *con) else bpf_map__set_max_entries(skel->maps.task_data, 1); - if (con->save_callstack) - bpf_map__set_max_entries(skel->maps.stacks, con->map_nr_entries); - else + if (con->save_callstack) { + bpf_map__set_max_entries(skel->maps.stacks, + con->map_nr_entries); + if (con->owner) { + bpf_map__set_value_size(skel->maps.owner_stacks_entries, + con->max_stack * sizeof(u64)); + bpf_map__set_value_size( + skel->maps.contention_owner_stacks, + con->max_stack * sizeof(u64)); + bpf_map__set_key_size(skel->maps.owner_lock_stat, + con->max_stack * sizeof(u64)); + skel->rodata->max_stack = con->max_stack; + } + } else bpf_map__set_max_entries(skel->maps.stacks, 1); if (target__has_cpu(target)) { diff --git a/tools/perf/util/bpf_skel/lock_contention.bpf.c b/tools/perf/util/bpf_skel/lock_contention.bpf.c index 1069bda5d733..05da19fdab23 100644 --- a/tools/perf/util/bpf_skel/lock_contention.bpf.c +++ b/tools/perf/util/bpf_skel/lock_contention.bpf.c @@ -19,13 +19,37 @@ #define LCB_F_PERCPU (1U << 4) #define LCB_F_MUTEX (1U << 5) -/* callstack storage */ + /* tmp buffer for owner callstack */ struct { - __uint(type, BPF_MAP_TYPE_STACK_TRACE); + __uint(type, BPF_MAP_TYPE_PERCPU_ARRAY); __uint(key_size, sizeof(__u32)); __uint(value_size, sizeof(__u64)); + __uint(max_entries, 1); +} owner_stacks_entries SEC(".maps"); + +/* a map for tracing lock address to owner data */ +struct { + __uint(type, BPF_MAP_TYPE_HASH); + __uint(key_size, sizeof(__u64)); // lock address + __uint(value_size, sizeof(cotd)); __uint(max_entries, MAX_ENTRIES); -} stacks SEC(".maps"); +} contention_owner_tracing SEC(".maps"); + +/* a map for tracing lock address to owner stacktrace */ +struct { + __uint(type, BPF_MAP_TYPE_HASH); + __uint(key_size, sizeof(__u64)); // lock address + __uint(value_size, sizeof(__u64)); // straktrace + __uint(max_entries, MAX_ENTRIES); +} contention_owner_stacks SEC(".maps"); + +/* owner callstack to contention data storage */ +struct { + __uint(type, BPF_MAP_TYPE_HASH); + __uint(key_size, sizeof(__u64)); + __uint(value_size, sizeof(struct contention_data)); + __uint(max_entries, MAX_ENTRIES); +} owner_lock_stat SEC(".maps"); /* maintain timestamp at the beginning of contention */ struct { @@ -43,6 +67,14 @@ struct { __uint(max_entries, 1); } tstamp_cpu SEC(".maps"); +/* callstack storage */ +struct { + __uint(type, BPF_MAP_TYPE_STACK_TRACE); + __uint(key_size, sizeof(__u32)); + __uint(value_size, sizeof(__u64)); + __uint(max_entries, MAX_ENTRIES); +} stacks SEC(".maps"); + /* actual lock contention statistics */ struct { __uint(type, BPF_MAP_TYPE_HASH); @@ -126,6 +158,7 @@ const volatile int needs_callstack; const volatile int stack_skip; const volatile int lock_owner; const volatile int use_cgroup_v2; +const volatile int max_stack; /* determine the key of lock stat */ const volatile int aggr_mode; @@ -436,7 +469,6 @@ int contention_end(u64 *ctx) return 0; need_delete = true; } - duration = bpf_ktime_get_ns() - pelem->timestamp; if ((__s64)duration < 0) { __sync_fetch_and_add(&time_fail, 1); diff --git a/tools/perf/util/bpf_skel/lock_data.h b/tools/perf/util/bpf_skel/lock_data.h index de12892f992f..1ef0bca9860e 100644 --- a/tools/perf/util/bpf_skel/lock_data.h +++ b/tools/perf/util/bpf_skel/lock_data.h @@ -3,6 +3,12 @@ #ifndef UTIL_BPF_SKEL_LOCK_DATA_H #define UTIL_BPF_SKEL_LOCK_DATA_H +typedef struct contention_owner_tracing_data { + u32 pid; // Who has the lock. + u64 timestamp; // The time while the owner acquires lock and contention is going on. + u32 count; // How many waiters for this lock. +} cotd; + struct tstamp_data { u64 timestamp; u64 lock; From patchwork Mon Jan 13 05:20:15 2025 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Chun-Tse Shao X-Patchwork-Id: 13936744 Received: from mail-pl1-f201.google.com (mail-pl1-f201.google.com [209.85.214.201]) (using TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 (128/128 bits)) (No client certificate requested) by smtp.subspace.kernel.org (Postfix) with ESMTPS id D6ACF8248C for ; Mon, 13 Jan 2025 05:23:10 +0000 (UTC) Authentication-Results: smtp.subspace.kernel.org; arc=none smtp.client-ip=209.85.214.201 ARC-Seal: i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1736745792; cv=none; b=XNOzcuShAjWbMEfkAt+MZU/N06u6/gD2pmiN7HXxJJQ/BFbTFnuCiDFPcpoW/r3IWxwlW6MrP2EEedsKys5rEhWddRT1PEaMMUMNN6APwAt+gFv1BzI4tiTV3Z6sZKafL/9wUYtOFjU6RV1PyXasmcCgUGhxvLZILzdWNXwRWqc= ARC-Message-Signature: i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1736745792; c=relaxed/simple; bh=pQOb1PSWvkn5i2/mDD/OCElWZVdAqdPQiu1a5zPnVkc=; h=Date:In-Reply-To:Mime-Version:References:Message-ID:Subject:From: To:Cc:Content-Type; b=m4vzpgmk0/wsRpE+rHw9HXPldKqwxE3M0uYX6ogVP27195qxNZMJvBdK4XhNYL7Fn9GICcVkHUIwNWYjfmcDSTDVoiLkADS3wBQstLX2DUg/xYYbaFaNr776rlCUarixo9gQMYPqI1aTTj7wQ569rCdi21bJbhCiYi5mxreAUEg= ARC-Authentication-Results: i=1; smtp.subspace.kernel.org; dmarc=pass (p=reject dis=none) header.from=google.com; spf=pass smtp.mailfrom=flex--ctshao.bounces.google.com; dkim=pass (2048-bit key) header.d=google.com header.i=@google.com header.b=qGDb84OK; arc=none smtp.client-ip=209.85.214.201 Authentication-Results: smtp.subspace.kernel.org; dmarc=pass (p=reject dis=none) header.from=google.com Authentication-Results: smtp.subspace.kernel.org; spf=pass smtp.mailfrom=flex--ctshao.bounces.google.com Authentication-Results: smtp.subspace.kernel.org; dkim=pass (2048-bit key) header.d=google.com header.i=@google.com header.b="qGDb84OK" Received: by mail-pl1-f201.google.com with SMTP id d9443c01a7336-21655569152so73771285ad.2 for ; Sun, 12 Jan 2025 21:23:10 -0800 (PST) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=google.com; s=20230601; t=1736745790; x=1737350590; darn=vger.kernel.org; h=cc:to:from:subject:message-id:references:mime-version:in-reply-to :date:from:to:cc:subject:date:message-id:reply-to; bh=VVfSqsB1425hgeh1CCvgyoytNRCKm7B7/XnFcKeLkPw=; b=qGDb84OKMaH8E/XPWhfu6YqFC4903e1J0YeLAVZufEHbCQITZdpsmTI3z1RspsomWK pAZI5NMU0b8fY7WS60l9w2ub9l3+Q7OEU9wAJnn5bQh3Vz68IC53pFP4SYmppOBMF7Be tzqvqJtmS5XI6/cNy1G7kf+GXpJCnJtyHNTuUBXagolnTl4cVgg9qhx+hMzeJdr7AjGI f2ZUS51bNbl3sOW13M0xTbFdWEsmhxJoK0OIH6bZHvXRutudk66oyVYOJ5zdy39HFlI8 2xFR26QVT9fk24FClofRzWQi5bHhNpDTu8w1V3rglK/IYLU+we79EPn5VDpJ2xGWxfTz 502w== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20230601; t=1736745790; x=1737350590; h=cc:to:from:subject:message-id:references:mime-version:in-reply-to :date:x-gm-message-state:from:to:cc:subject:date:message-id:reply-to; bh=VVfSqsB1425hgeh1CCvgyoytNRCKm7B7/XnFcKeLkPw=; b=ssjVDiTLN4owgSDntP48WUG8WOBXNbVV0mo5oYq1wCeTa0RxFFq1QRFlV7Dy2XdRCF xURNS5RzqDlAs3husbtILskGHIniCtEweWCZNxDO/ty+aWCKcIeIPxapbNk2rQetRj6J Ukn2/x33YBT4mNs1WsHIEurf7MkTRvbKZ/WaFJzsMQHAGj42sPMh7RBKlwnyyZtjQNl/ OMgmxT6v2RSnhyJl3J3AyoxMl4fK+k6d14YY+gOK9Y+om2O19iq4HDBB1ITiKxkI0I4u 4QWRoEG0uwK+riYX9SCkdcqo0gYW1b+yvTV/2Dao2HSXpIL2ElI8u8qI9wYvnSz0Fk9M ZBvA== X-Forwarded-Encrypted: i=1; AJvYcCXs6y/zJLFJRQobZe7zy5a/yXtxSnSCoOFuDHXYGQAGT2i844+3h97OVqSuPfwXGwUgQIg=@vger.kernel.org X-Gm-Message-State: AOJu0YzfhvzuKQTx5AjriCFaU353XzyitTclwS3t8gJCjZTzhaYY6hIJ aJBegDLI7Hp5kLNBjpwHCdYmfUVpAGiKFG0XL4GgazVglX9mFsnArai81/CzozaJOVv1TtbUjPl JjQ== X-Google-Smtp-Source: AGHT+IEZWkENHpl86IzSgwf0OFY8eoSHS59AUCniUII1KvxAtH7cndK43/Nilx8npxLdrV5enN/fkbSZh4M= X-Received: from pgbgb2.prod.google.com ([2002:a05:6a02:4b42:b0:86d:55d8:7944]) (user=ctshao job=prod-delivery.src-stubby-dispatcher) by 2002:a17:902:ecc7:b0:216:3b31:56c2 with SMTP id d9443c01a7336-21a83fded24mr314809615ad.53.1736745790291; Sun, 12 Jan 2025 21:23:10 -0800 (PST) Date: Sun, 12 Jan 2025 21:20:15 -0800 In-Reply-To: <20250113052220.2105645-1-ctshao@google.com> Precedence: bulk X-Mailing-List: bpf@vger.kernel.org List-Id: List-Subscribe: List-Unsubscribe: Mime-Version: 1.0 References: <20250113052220.2105645-1-ctshao@google.com> X-Mailer: git-send-email 2.47.1.688.g23fc6f90ad-goog Message-ID: <20250113052220.2105645-3-ctshao@google.com> Subject: [PATCH v2 2/4] perf lock: Retrieve owner callstack in bpf program From: Chun-Tse Shao To: linux-kernel@vger.kernel.org Cc: Chun-Tse Shao , peterz@infradead.org, mingo@redhat.com, acme@kernel.org, namhyung@kernel.org, mark.rutland@arm.com, alexander.shishkin@linux.intel.com, jolsa@kernel.org, irogers@google.com, adrian.hunter@intel.com, kan.liang@linux.intel.com, linux-perf-users@vger.kernel.org, bpf@vger.kernel.org Tracing owner callstack in `contention_begin()` and `contention_end()`, and storing in `owner_lock_stat` bpf map. Signed-off-by: Chun-Tse Shao --- .../perf/util/bpf_skel/lock_contention.bpf.c | 152 +++++++++++++++++- 1 file changed, 151 insertions(+), 1 deletion(-) diff --git a/tools/perf/util/bpf_skel/lock_contention.bpf.c b/tools/perf/util/bpf_skel/lock_contention.bpf.c index 05da19fdab23..3f47fbfa237c 100644 --- a/tools/perf/util/bpf_skel/lock_contention.bpf.c +++ b/tools/perf/util/bpf_skel/lock_contention.bpf.c @@ -7,6 +7,7 @@ #include #include "lock_data.h" +#include /* for collect_lock_syms(). 4096 was rejected by the verifier */ #define MAX_CPUS 1024 @@ -178,6 +179,9 @@ int data_fail; int task_map_full; int data_map_full; +struct task_struct *bpf_task_from_pid(s32 pid) __ksym; +void bpf_task_release(struct task_struct *p) __ksym; + static inline __u64 get_current_cgroup_id(void) { struct task_struct *task; @@ -407,6 +411,60 @@ int contention_begin(u64 *ctx) pelem->flags = (__u32)ctx[1]; if (needs_callstack) { + u32 i = 0; + int owner_pid; + unsigned long *entries; + struct task_struct *task; + cotd *data; + + if (!lock_owner) + goto contention_begin_skip_owner_callstack; + + task = get_lock_owner(pelem->lock, pelem->flags); + if (!task) + goto contention_begin_skip_owner_callstack; + + owner_pid = BPF_CORE_READ(task, pid); + + entries = bpf_map_lookup_elem(&owner_stacks_entries, &i); + if (!entries) + goto contention_begin_skip_owner_callstack; + for (i = 0; i < max_stack; i++) + entries[i] = 0x0; + + task = bpf_task_from_pid(owner_pid); + if (task) { + bpf_get_task_stack(task, entries, + max_stack * sizeof(unsigned long), + 0); + bpf_task_release(task); + } + + data = bpf_map_lookup_elem(&contention_owner_tracing, + &(pelem->lock)); + + // Contention just happens, or corner case `lock` is owned by + // process not `owner_pid`. + if (!data || data->pid != owner_pid) { + cotd first = { + .pid = owner_pid, + .timestamp = pelem->timestamp, + .count = 1, + }; + bpf_map_update_elem(&contention_owner_tracing, + &(pelem->lock), &first, BPF_ANY); + bpf_map_update_elem(&contention_owner_stacks, + &(pelem->lock), entries, BPF_ANY); + } + // Contention is going on and new waiter joins. + else { + __sync_fetch_and_add(&data->count, 1); + // TODO: Since for owner the callstack would change at + // different time, We should check and report if the + // callstack is different with the recorded one in + // `contention_owner_stacks`. + } +contention_begin_skip_owner_callstack: pelem->stack_id = bpf_get_stackid(ctx, &stacks, BPF_F_FAST_STACK_CMP | stack_skip); if (pelem->stack_id < 0) @@ -443,6 +501,7 @@ int contention_end(u64 *ctx) struct tstamp_data *pelem; struct contention_key key = {}; struct contention_data *data; + __u64 timestamp; __u64 duration; bool need_delete = false; @@ -469,12 +528,103 @@ int contention_end(u64 *ctx) return 0; need_delete = true; } - duration = bpf_ktime_get_ns() - pelem->timestamp; + timestamp = bpf_ktime_get_ns(); + duration = timestamp - pelem->timestamp; if ((__s64)duration < 0) { __sync_fetch_and_add(&time_fail, 1); goto out; } + if (needs_callstack && lock_owner) { + u64 owner_contention_time; + unsigned long *owner_stack; + struct contention_data *cdata; + cotd *otdata; + + otdata = bpf_map_lookup_elem(&contention_owner_tracing, + &(pelem->lock)); + owner_stack = bpf_map_lookup_elem(&contention_owner_stacks, + &(pelem->lock)); + if (!otdata || !owner_stack) + goto contention_end_skip_owner_callstack; + + owner_contention_time = timestamp - otdata->timestamp; + + // Update `owner_lock_stat` if `owner_stack` is + // available. + if (owner_stack[0] != 0x0) { + cdata = bpf_map_lookup_elem(&owner_lock_stat, + owner_stack); + if (!cdata) { + struct contention_data first = { + .total_time = owner_contention_time, + .max_time = owner_contention_time, + .min_time = owner_contention_time, + .count = 1, + .flags = pelem->flags, + }; + bpf_map_update_elem(&owner_lock_stat, + owner_stack, &first, + BPF_ANY); + } else { + __sync_fetch_and_add(&cdata->total_time, + owner_contention_time); + __sync_fetch_and_add(&cdata->count, 1); + + /* FIXME: need atomic operations */ + if (cdata->max_time < owner_contention_time) + cdata->max_time = owner_contention_time; + if (cdata->min_time > owner_contention_time) + cdata->min_time = owner_contention_time; + } + } + + // No contention is going on, delete `lock` in + // `contention_owner_tracing` and + // `contention_owner_stacks` + if (otdata->count <= 1) { + bpf_map_delete_elem(&contention_owner_tracing, + &(pelem->lock)); + bpf_map_delete_elem(&contention_owner_stacks, + &(pelem->lock)); + } + // Contention is still going on, with a new owner + // (current task). `otdata` should be updated accordingly. + else { + (otdata->count)--; + + // If ctx[1] is not 0, the current task terminates lock + // waiting without acquiring it. Owner is not changed. + if (ctx[1] == 0) { + u32 i = 0; + unsigned long *entries = bpf_map_lookup_elem( + &owner_stacks_entries, &i); + if (entries) { + for (i = 0; i < (u32)max_stack; i++) + entries[i] = 0x0; + + bpf_get_task_stack( + bpf_get_current_task_btf(), + entries, + max_stack * + sizeof(unsigned long), + 0); + bpf_map_update_elem( + &contention_owner_stacks, + &(pelem->lock), entries, + BPF_ANY); + } + + otdata->pid = pid; + otdata->timestamp = timestamp; + } + + bpf_map_update_elem(&contention_owner_tracing, + &(pelem->lock), otdata, BPF_ANY); + } + } +contention_end_skip_owner_callstack: + switch (aggr_mode) { case LOCK_AGGR_CALLER: key.stack_id = pelem->stack_id; From patchwork Mon Jan 13 05:20:16 2025 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Chun-Tse Shao X-Patchwork-Id: 13936745 Received: from mail-pj1-f74.google.com (mail-pj1-f74.google.com [209.85.216.74]) (using TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 (128/128 bits)) (No client certificate requested) by smtp.subspace.kernel.org (Postfix) with ESMTPS id 5FAC78248C for ; Mon, 13 Jan 2025 05:23:33 +0000 (UTC) Authentication-Results: smtp.subspace.kernel.org; arc=none smtp.client-ip=209.85.216.74 ARC-Seal: i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1736745814; cv=none; b=HRH0kOctFINJ8WRZSFyVMQmfgu9noDdZmvQCfxoAr3tqFfFlusq2Kx3zfCqgnkg0lvU98lz8q3Nbi+J54fGo9TFd7vavOu3TJcTVzfIlL6GeO6HvpbXtzQHl5Z1nLElkOnBgu7dFzlY/0sdqWvDZ4yhPMQism8FC0MGsWkffBRA= ARC-Message-Signature: i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1736745814; c=relaxed/simple; bh=7r3spX1VTM6b7Ljhi+bgUnGyr89CaYIzvQ4zgRIgmY8=; h=Date:In-Reply-To:Mime-Version:References:Message-ID:Subject:From: To:Cc:Content-Type; b=YoPz2neK4VcJ/+F4pX+Rd1ieiDKCmDsYuWxQX6LJ3E0KTa55TjSeXEnstgA/yVbDiqN+A5HbtC4ndxzDAsPFVZB4yMKO+LRfCMnmUaz1ybriQaoSaIkopvY8kqXdHMzGKSm0PsLTmEQPyryfB0yS8o8VdbT1kRm9EK+FktdFxe4= ARC-Authentication-Results: i=1; smtp.subspace.kernel.org; dmarc=pass (p=reject dis=none) header.from=google.com; spf=pass smtp.mailfrom=flex--ctshao.bounces.google.com; dkim=pass (2048-bit key) header.d=google.com header.i=@google.com header.b=l9gWE0Mn; arc=none smtp.client-ip=209.85.216.74 Authentication-Results: smtp.subspace.kernel.org; dmarc=pass (p=reject dis=none) header.from=google.com Authentication-Results: smtp.subspace.kernel.org; spf=pass smtp.mailfrom=flex--ctshao.bounces.google.com Authentication-Results: smtp.subspace.kernel.org; dkim=pass (2048-bit key) header.d=google.com header.i=@google.com header.b="l9gWE0Mn" Received: by mail-pj1-f74.google.com with SMTP id 98e67ed59e1d1-2ef79403c5eso11400487a91.0 for ; Sun, 12 Jan 2025 21:23:33 -0800 (PST) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=google.com; s=20230601; t=1736745813; x=1737350613; darn=vger.kernel.org; h=cc:to:from:subject:message-id:references:mime-version:in-reply-to :date:from:to:cc:subject:date:message-id:reply-to; bh=nK90VQgfJdgs3XY2vlzkWlR2hkLCvAiNELZrM3w+SYE=; b=l9gWE0MnlB+/Ap7+uXWp5IZlT8D00cC1G1wWbXa/wxRT8pmFlC8gbbzwKGGTGLbrlo tOohqX5fYEJsUDFXzeGmO1z0YC1XAfHbXdTuixGP0fLidnBXFj3NjZJFuckuUC7rPSO1 zhFsw93ndcCTEn4qLpi7ryM/L+O+B1q2aAt7PcnIwQXoYRW4JhYUdIEMXKBbe49Ffpl+ VSftBygeSO8hdgblj93H8jQ6rhbC84ukl8qm11rN7G6ERTPsli6RtP85DSvIGRNpvUra LOiP5aak2gib4/SqN8YHYnWs53IAEiO8EGvfqgjktn96VTT/aPxgpyplzWHU96J4Mrks Iqlg== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20230601; t=1736745813; x=1737350613; h=cc:to:from:subject:message-id:references:mime-version:in-reply-to :date:x-gm-message-state:from:to:cc:subject:date:message-id:reply-to; bh=nK90VQgfJdgs3XY2vlzkWlR2hkLCvAiNELZrM3w+SYE=; b=cgmdI45tv0pQtr9uOgVumk7N5ExKRT+qZfVlbgsDYYsFLm4Y0z53GnDwBAIv1MzBVw tAzkNL3G1HZmH/pXSAdUQrLHBSO5tCHyZlA3NNoGCyiGQEQy1ZRdQhX7+L1/b9co3XF8 N8fdKVRBW+w2IAIO2H2bytyQZKVMj+gmE+4nuoIszGGXevq3uluLieAKz7VMRzxX3EGh gST6Z84A8xX7DdwFk5gwMs9Z7ITMAmY5zeopSBZWaM05xto4d1o8L2y2HvcV5dV6bvbD RB/rQu6UwpM/SiV4qo10URtrPvq1XFFlv587xkP8cIRpHi+b8dLV1tsdftqqPbcOS1fH V5+Q== X-Forwarded-Encrypted: i=1; AJvYcCW9FTXnvfAbBrO6/bt8JfhuQUP7qsLeZVFgiS6XVVzOg6rGduDMi8hBYXmEwq0ln1EqtNI=@vger.kernel.org X-Gm-Message-State: AOJu0YwHj8DzPDgFLVJ3GGwlfuhsDQtUK60YqTioXqQ1csn7uo6kdq+T ETAuOmcqn98ONFWkAy1gFdtmK/6xIrZ8R1HEWmNRRfiAAbj5WUV4d3umWtIPlhOzAsufEHwTLXP iTw== X-Google-Smtp-Source: AGHT+IE2T0lvVUiEK+HXuBqpPOONhCmR9IKp8db4I+ItSYAUaSSsYzmbt6l7f7ysmcwdqshEHrllE2BAF+4= X-Received: from pfiv28.prod.google.com ([2002:aa7:99dc:0:b0:725:dec7:dd47]) (user=ctshao job=prod-delivery.src-stubby-dispatcher) by 2002:a05:6a00:3cc2:b0:726:f7c9:7b36 with SMTP id d2e1a72fcca58-72d21f325b8mr31890687b3a.8.1736745812736; Sun, 12 Jan 2025 21:23:32 -0800 (PST) Date: Sun, 12 Jan 2025 21:20:16 -0800 In-Reply-To: <20250113052220.2105645-1-ctshao@google.com> Precedence: bulk X-Mailing-List: bpf@vger.kernel.org List-Id: List-Subscribe: List-Unsubscribe: Mime-Version: 1.0 References: <20250113052220.2105645-1-ctshao@google.com> X-Mailer: git-send-email 2.47.1.688.g23fc6f90ad-goog Message-ID: <20250113052220.2105645-4-ctshao@google.com> Subject: [PATCH v2 3/4] perf lock: Make rb_tree helper functions generic From: Chun-Tse Shao To: linux-kernel@vger.kernel.org Cc: Chun-Tse Shao , peterz@infradead.org, mingo@redhat.com, acme@kernel.org, namhyung@kernel.org, mark.rutland@arm.com, alexander.shishkin@linux.intel.com, jolsa@kernel.org, irogers@google.com, adrian.hunter@intel.com, kan.liang@linux.intel.com, linux-perf-users@vger.kernel.org, bpf@vger.kernel.org The rb_tree helper functions can be reused for parsing `owner_lock_stat` into rb tree for sorting. Signed-off-by: Chun-Tse Shao --- tools/perf/builtin-lock.c | 34 +++++++++++++++++++++++----------- 1 file changed, 23 insertions(+), 11 deletions(-) diff --git a/tools/perf/builtin-lock.c b/tools/perf/builtin-lock.c index 062e2b56a2ab..f9b7620444c0 100644 --- a/tools/perf/builtin-lock.c +++ b/tools/perf/builtin-lock.c @@ -438,16 +438,13 @@ static void combine_lock_stats(struct lock_stat *st) rb_insert_color(&st->rb, &sorted); } -static void insert_to_result(struct lock_stat *st, - int (*bigger)(struct lock_stat *, struct lock_stat *)) +static void insert_to(struct rb_root *rr, struct lock_stat *st, + int (*bigger)(struct lock_stat *, struct lock_stat *)) { - struct rb_node **rb = &result.rb_node; + struct rb_node **rb = &rr->rb_node; struct rb_node *parent = NULL; struct lock_stat *p; - if (combine_locks && st->combined) - return; - while (*rb) { p = container_of(*rb, struct lock_stat, rb); parent = *rb; @@ -459,13 +456,21 @@ static void insert_to_result(struct lock_stat *st, } rb_link_node(&st->rb, parent, rb); - rb_insert_color(&st->rb, &result); + rb_insert_color(&st->rb, rr); } -/* returns left most element of result, and erase it */ -static struct lock_stat *pop_from_result(void) +static inline void insert_to_result(struct lock_stat *st, + int (*bigger)(struct lock_stat *, + struct lock_stat *)) +{ + if (combine_locks && st->combined) + return; + insert_to(&result, st, bigger); +} + +static inline struct lock_stat *pop_from(struct rb_root *rr) { - struct rb_node *node = result.rb_node; + struct rb_node *node = rr->rb_node; if (!node) return NULL; @@ -473,8 +478,15 @@ static struct lock_stat *pop_from_result(void) while (node->rb_left) node = node->rb_left; - rb_erase(node, &result); + rb_erase(node, rr); return container_of(node, struct lock_stat, rb); + +} + +/* returns left most element of result, and erase it */ +static struct lock_stat *pop_from_result(void) +{ + return pop_from(&result); } struct lock_stat *lock_stat_find(u64 addr) From patchwork Mon Jan 13 05:20:17 2025 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Chun-Tse Shao X-Patchwork-Id: 13936746 Received: from mail-pl1-f201.google.com (mail-pl1-f201.google.com [209.85.214.201]) (using TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 (128/128 bits)) (No client certificate requested) by smtp.subspace.kernel.org (Postfix) with ESMTPS id A1700230D1E for ; Mon, 13 Jan 2025 05:23:55 +0000 (UTC) Authentication-Results: smtp.subspace.kernel.org; arc=none smtp.client-ip=209.85.214.201 ARC-Seal: i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1736745837; cv=none; b=rwfVwwPoqJE/sZDdgnrGnoq7h9e3chNClXayUZj/2VsE9Ngaoxi5FwbBVsOdFniQPqT2RJrtcccsm1zzrEB5rWluk1J6NV/ac8VEoO1ZfD1/7VhULCtdU5EmC2+QAnQtZPa6itW3eDJ1FfN+bynLLzns0f8Lnz7VGpGUXfjOLQc= ARC-Message-Signature: i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1736745837; c=relaxed/simple; bh=03zst4yp3vVxlteaRKGEcsn5oX+laHuxdfF/IJtBYcM=; h=Date:In-Reply-To:Mime-Version:References:Message-ID:Subject:From: To:Cc:Content-Type; b=Ph+21NnQO4WLesO7cIg3HqovoMsy647SLu60BYqkvlRo//lNIRBiKxW2nvZgkSgsQascxLQ+4vJcLBWfpkUv/YRaXh8BVR7H0zkNNch/RE4tujeckygvMyO6YZsrUIyqp0RQry1zTxlKRcXBMP4JaareD4VeviH49wzqenkJRUc= ARC-Authentication-Results: i=1; smtp.subspace.kernel.org; dmarc=pass (p=reject dis=none) header.from=google.com; spf=pass smtp.mailfrom=flex--ctshao.bounces.google.com; dkim=pass (2048-bit key) header.d=google.com header.i=@google.com header.b=gcb10A2b; arc=none smtp.client-ip=209.85.214.201 Authentication-Results: smtp.subspace.kernel.org; dmarc=pass (p=reject dis=none) header.from=google.com Authentication-Results: smtp.subspace.kernel.org; spf=pass smtp.mailfrom=flex--ctshao.bounces.google.com Authentication-Results: smtp.subspace.kernel.org; dkim=pass (2048-bit key) header.d=google.com header.i=@google.com header.b="gcb10A2b" Received: by mail-pl1-f201.google.com with SMTP id d9443c01a7336-2163d9a730aso72954945ad.1 for ; Sun, 12 Jan 2025 21:23:55 -0800 (PST) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=google.com; s=20230601; t=1736745835; x=1737350635; darn=vger.kernel.org; h=cc:to:from:subject:message-id:references:mime-version:in-reply-to :date:from:to:cc:subject:date:message-id:reply-to; bh=RChDgtCHJsZ+9exXiY5ah/G2LrjJgqqretYf70z53cE=; b=gcb10A2bFkaWPWANnQV1ERcPu6/6JvLsNc/y3tPIFazNasM85X2qeXZfocvxMyVVBD qG58CTygZYfw8Cwyy2wh79pX7m0k+kXAGwFn3VrKoCptFC1iWfrw2fG6WLgablII09NH WeNAyfKk66aw429W3LVb1ubU1MMQgnHPTF3ulinpe86DokMVwjSXMbKP45WrGYy6wT7X Y38meIZ0bDMpo2zQKj1Eqe7rKBa0k4QX1byfB6QsKr7KFJf850lmUp/EfSrCxt3WPdYr 4rVjNOWJo8kgxNBIHcNeK3n8MRKyMEp5A4pKfVfkm5tpTugKX4dMLP1qao5jmOE/JbBg 1k6g== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20230601; t=1736745835; x=1737350635; h=cc:to:from:subject:message-id:references:mime-version:in-reply-to :date:x-gm-message-state:from:to:cc:subject:date:message-id:reply-to; bh=RChDgtCHJsZ+9exXiY5ah/G2LrjJgqqretYf70z53cE=; b=rEsX/i6MTPYmlOlakK19++19Iw2C3Oa4RJKLapFsnvNgmjKlUEftJ2Mz1ZMkkSFZu6 RPlPckzF960y+rpQpMANdCo7BvAZv97dnHYB+mLmf0XNcb662XEcQB0+MaGqzZX7LlyS q+nXEhMMT4/fB/7NIqG/gpQjehkpJ5g1wHF8zwujpf/kLj/WqYtY3CsXJjgVCmwQycrv xTQiIWEXJoQ+ccKlor2mUSKJyUx5bS+hPBmnPYaUcZNdnMKqH52sSkz91djl/uo8Kbtc BOTeD9MsxUSc6L7XVUEohZuTYfTcIfzpYUpz4I4yZW8x5w6p7dXdoM6XIcjR0X8xBt95 Ojig== X-Forwarded-Encrypted: i=1; AJvYcCVFqqhCjN9/iEIYzPP+drv076loJ8bCMu1jEVsmS7OqDjPKEcfb1hQ1mf4ggZU4LChDZbc=@vger.kernel.org X-Gm-Message-State: AOJu0YxFG4Ce6Dionk7W2la7bS3eLQrf9u7hT78CsKvwZNLcBhEUQgYQ NwYpwAvyGz7la0c7Sh2m9ZfqPXgIl1FzvLYxQqBbgapwmdIIVV7IrEfJo+Xc2Lq5xJi1wjdIdi+ B7g== X-Google-Smtp-Source: AGHT+IHe87MO8RdBUIZgJ1nejREfqTCdjCsLkwV3QgX98VRh5u0vSOR1/pY2hZ34l1o3zuzz9VxVXKLGz5E= X-Received: from pgsb3.prod.google.com ([2002:a65:67c3:0:b0:7fd:460b:daa3]) (user=ctshao job=prod-delivery.src-stubby-dispatcher) by 2002:a17:902:d4c6:b0:216:1a59:5bbf with SMTP id d9443c01a7336-21a8d52fbd2mr235352605ad.0.1736745835018; Sun, 12 Jan 2025 21:23:55 -0800 (PST) Date: Sun, 12 Jan 2025 21:20:17 -0800 In-Reply-To: <20250113052220.2105645-1-ctshao@google.com> Precedence: bulk X-Mailing-List: bpf@vger.kernel.org List-Id: List-Subscribe: List-Unsubscribe: Mime-Version: 1.0 References: <20250113052220.2105645-1-ctshao@google.com> X-Mailer: git-send-email 2.47.1.688.g23fc6f90ad-goog Message-ID: <20250113052220.2105645-5-ctshao@google.com> Subject: [PATCH v2 4/4] perf lock: Report owner stack in usermode From: Chun-Tse Shao To: linux-kernel@vger.kernel.org Cc: Chun-Tse Shao , peterz@infradead.org, mingo@redhat.com, acme@kernel.org, namhyung@kernel.org, mark.rutland@arm.com, alexander.shishkin@linux.intel.com, jolsa@kernel.org, irogers@google.com, adrian.hunter@intel.com, kan.liang@linux.intel.com, linux-perf-users@vger.kernel.org, bpf@vger.kernel.org Parse `owner_lock_stat` into a rb tree, and report owner lock stats with stack trace in order. Example output: $ sudo ~/linux/tools/perf/perf lock con -abvo -Y mutex-spin -E3 perf bench sched pipe ... contended total wait max wait avg wait type caller 171 1.55 ms 20.26 us 9.06 us mutex pipe_read+0x57 0xffffffffac6318e7 pipe_read+0x57 0xffffffffac623862 vfs_read+0x332 0xffffffffac62434b ksys_read+0xbb 0xfffffffface604b2 do_syscall_64+0x82 0xffffffffad00012f entry_SYSCALL_64_after_hwframe+0x76 36 193.71 us 15.27 us 5.38 us mutex pipe_write+0x50 0xffffffffac631ee0 pipe_write+0x50 0xffffffffac6241db vfs_write+0x3bb 0xffffffffac6244ab ksys_write+0xbb 0xfffffffface604b2 do_syscall_64+0x82 0xffffffffad00012f entry_SYSCALL_64_after_hwframe+0x76 4 51.22 us 16.47 us 12.80 us mutex do_epoll_wait+0x24d 0xffffffffac691f0d do_epoll_wait+0x24d 0xffffffffac69249b do_epoll_pwait.part.0+0xb 0xffffffffac693ba5 __x64_sys_epoll_pwait+0x95 0xfffffffface604b2 do_syscall_64+0x82 0xffffffffad00012f entry_SYSCALL_64_after_hwframe+0x76 === owner stack trace === 3 31.24 us 15.27 us 10.41 us mutex pipe_read+0x348 0xffffffffac631bd8 pipe_read+0x348 0xffffffffac623862 vfs_read+0x332 0xffffffffac62434b ksys_read+0xbb 0xfffffffface604b2 do_syscall_64+0x82 0xffffffffad00012f entry_SYSCALL_64_after_hwframe+0x76 ... Signed-off-by: Chun-Tse Shao --- tools/perf/builtin-lock.c | 20 ++++++++++++- tools/perf/util/bpf_lock_contention.c | 41 +++++++++++++++++++++++++++ tools/perf/util/lock-contention.h | 2 ++ 3 files changed, 62 insertions(+), 1 deletion(-) diff --git a/tools/perf/builtin-lock.c b/tools/perf/builtin-lock.c index f9b7620444c0..0dfec175b25b 100644 --- a/tools/perf/builtin-lock.c +++ b/tools/perf/builtin-lock.c @@ -42,6 +42,7 @@ #include #include #include +#include static struct perf_session *session; static struct target target; @@ -1926,6 +1927,23 @@ static void print_contention_result(struct lock_contention *con) break; } + if (con->owner && con->save_callstack) { + struct rb_root root = RB_ROOT; + + + if (symbol_conf.field_sep) + fprintf(lock_output, "# owner stack trace:\n"); + else + fprintf(lock_output, "\n=== owner stack trace ===\n\n"); + while ((st = pop_owner_stack_trace(con))) + insert_to(&root, st, compare); + + while ((st = pop_from(&root))) { + print_lock_stat(con, st); + zfree(st); + } + } + if (print_nr_entries) { /* update the total/bad stats */ while ((st = pop_from_result())) { @@ -2071,7 +2089,7 @@ static int check_lock_contention_options(const struct option *options, } } - if (show_lock_owner) + if (show_lock_owner && !verbose) show_thread_stats = true; return 0; diff --git a/tools/perf/util/bpf_lock_contention.c b/tools/perf/util/bpf_lock_contention.c index c9c58f243ceb..a63d5ffac386 100644 --- a/tools/perf/util/bpf_lock_contention.c +++ b/tools/perf/util/bpf_lock_contention.c @@ -414,6 +414,47 @@ static const char *lock_contention_get_name(struct lock_contention *con, return name_buf; } +struct lock_stat *pop_owner_stack_trace(struct lock_contention *con) +{ + int fd; + u64 *stack_trace; + struct contention_data data = {}; + size_t stack_size = con->max_stack * sizeof(*stack_trace); + struct lock_stat *st; + char name[KSYM_NAME_LEN]; + + fd = bpf_map__fd(skel->maps.owner_lock_stat); + + stack_trace = zalloc(stack_size); + if (stack_trace == NULL) + return NULL; + + if (bpf_map_get_next_key(fd, NULL, stack_trace)) + return NULL; + + bpf_map_lookup_elem(fd, stack_trace, &data); + st = zalloc(sizeof(struct lock_stat)); + + strcpy(name, stack_trace[0] ? lock_contention_get_name(con, NULL, + stack_trace, 0) : + "unknown"); + + st->name = strdup(name); + st->flags = data.flags; + st->nr_contended = data.count; + st->wait_time_total = data.total_time; + st->wait_time_max = data.max_time; + st->wait_time_min = data.min_time; + st->callstack = memdup(stack_trace, stack_size); + + if (data.count) + st->avg_wait_time = data.total_time / data.count; + + bpf_map_delete_elem(fd, stack_trace); + free(stack_trace); + + return st; +} int lock_contention_read(struct lock_contention *con) { int fd, stack, err = 0; diff --git a/tools/perf/util/lock-contention.h b/tools/perf/util/lock-contention.h index 1a7248ff3889..83b400a36137 100644 --- a/tools/perf/util/lock-contention.h +++ b/tools/perf/util/lock-contention.h @@ -156,6 +156,8 @@ int lock_contention_stop(void); int lock_contention_read(struct lock_contention *con); int lock_contention_finish(struct lock_contention *con); +struct lock_stat *pop_owner_stack_trace(struct lock_contention *con); + #else /* !HAVE_BPF_SKEL */ static inline int lock_contention_prepare(struct lock_contention *con __maybe_unused)