From patchwork Thu Sep 8 06:37:51 2022 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Namhyung Kim X-Patchwork-Id: 12969702 X-Patchwork-Delegate: bpf@iogearbox.net Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from vger.kernel.org (vger.kernel.org [23.128.96.18]) by smtp.lore.kernel.org (Postfix) with ESMTP id 04422C6FA83 for ; Thu, 8 Sep 2022 06:38:07 +0000 (UTC) Received: (majordomo@vger.kernel.org) by vger.kernel.org via listexpand id S230384AbiIHGiF (ORCPT ); Thu, 8 Sep 2022 02:38:05 -0400 Received: from lindbergh.monkeyblade.net ([23.128.96.19]:33796 "EHLO lindbergh.monkeyblade.net" rhost-flags-OK-OK-OK-OK) by vger.kernel.org with ESMTP id S230338AbiIHGiB (ORCPT ); Thu, 8 Sep 2022 02:38:01 -0400 Received: from mail-pj1-x102c.google.com (mail-pj1-x102c.google.com [IPv6:2607:f8b0:4864:20::102c]) by lindbergh.monkeyblade.net (Postfix) with ESMTPS id DFE15C59C9; Wed, 7 Sep 2022 23:38:00 -0700 (PDT) Received: by mail-pj1-x102c.google.com with SMTP id i15-20020a17090a4b8f00b0020073b4ac27so1374268pjh.3; Wed, 07 Sep 2022 23:38:00 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=20210112; h=content-transfer-encoding:mime-version:references:in-reply-to :message-id:date:subject:cc:to:from:sender:from:to:cc:subject:date; bh=ejkeCXq6weG9XOFKJtIPiuPMVIfy3kaHlllbgNPcMHo=; b=jvk02Sd6LWDcLrSGxtyVyRXDiX308iJXu9xgUdMBWKFRZcYndUnbffhXpMd/UCB74z UtO+dqkA4NMchvn2LKNbXLncZu1rSc0M814Yk1F0V8UWj8pfCNNApRl44tNCqx2amjSP 0ovToD/EElFWb6Rcg+n6NSG9PDR8NSUDvK0EbJy00QQoyjxqWVIXq/LNcJJtdGaz+gXm hrfUnWN+e4rZjGjDH7tLuCWEWB1wVctsBTsgZmYw+KEGOlCljJr1Bx3OXGLoPxE3xWVw Trzib5uPpGFgBCXMxRbfp7yg6SuzqOqA/j0qb9AHfEQ/VrQo8kYKFRPovQ7mdf3DkVGb vcFA== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20210112; h=content-transfer-encoding:mime-version:references:in-reply-to :message-id:date:subject:cc:to:from:sender:x-gm-message-state:from :to:cc:subject:date; bh=ejkeCXq6weG9XOFKJtIPiuPMVIfy3kaHlllbgNPcMHo=; b=iUS2ZgXaC1dk6o8R0arcXGNyaCJCP888+Rk9m9ge4QYJ/eSEdGuf+TTZe+UqJ7QxAf SCmPNYdBC2ETS1HNnWek/FniG1G5uHa1pxA4myIwmVIfpjvf7UptMY+lznrohQRpMUQE WS99tV3J+d7+/J3RpMHX4ashR7x97kVEtdjEYMOtE9I6rBhpQ6R/+mV7dZmFc08FJIpa NuxEF2mV97FxCEb3BtFeqNwAlgzx1M4/xjVCq7NudvXaE6fe7F6UeMFqpPpCB1pCVjit vQIuVJTvYW42XJ+L9VMGe4ojkjPfhbIv28ie7eYO3AWWSBzeomve3jlr4WYBNpa3ZLuw 3SFg== X-Gm-Message-State: ACgBeo14UTYyyESvez2PckibE2iYR0+aT3ou1+gmnftTjMQIAJvJPWp0 TrY8mREyJvU/4uJtRzm9BMY= X-Google-Smtp-Source: AA6agR7+CSf6WyxxfU25l/GEhRlB2ckH2Ox6ikq0kZlWxLx8j5Quzfq6cJ4vZShbWtsgSkT6XZzTaw== X-Received: by 2002:a17:903:110f:b0:171:3afa:e688 with SMTP id n15-20020a170903110f00b001713afae688mr7767234plh.162.1662619080242; Wed, 07 Sep 2022 23:38:00 -0700 (PDT) Received: from balhae.hsd1.ca.comcast.net ([2601:647:6780:1040:7f21:d032:3f14:a4e6]) by smtp.gmail.com with ESMTPSA id o33-20020a17090a0a2400b001fb0fc33d72sm890716pjo.47.2022.09.07.23.37.59 (version=TLS1_3 cipher=TLS_AES_256_GCM_SHA384 bits=256/256); Wed, 07 Sep 2022 23:37:59 -0700 (PDT) Sender: Namhyung Kim From: Namhyung Kim To: Arnaldo Carvalho de Melo , Jiri Olsa Cc: Ingo Molnar , Peter Zijlstra , LKML , Ian Rogers , linux-perf-users@vger.kernel.org, Song Liu , bpf@vger.kernel.org Subject: [PATCH 1/4] perf lock contention: Factor out get_symbol_name_offset() Date: Wed, 7 Sep 2022 23:37:51 -0700 Message-Id: <20220908063754.1369709-2-namhyung@kernel.org> X-Mailer: git-send-email 2.37.2.789.g6183377224-goog In-Reply-To: <20220908063754.1369709-1-namhyung@kernel.org> References: <20220908063754.1369709-1-namhyung@kernel.org> MIME-Version: 1.0 Precedence: bulk List-ID: X-Mailing-List: bpf@vger.kernel.org It's to convert addr to symbol+offset. Signed-off-by: Namhyung Kim --- tools/perf/builtin-lock.c | 28 +++++++++++++++++++--------- 1 file changed, 19 insertions(+), 9 deletions(-) diff --git a/tools/perf/builtin-lock.c b/tools/perf/builtin-lock.c index 70197c0593b1..2a5672f8d22e 100644 --- a/tools/perf/builtin-lock.c +++ b/tools/perf/builtin-lock.c @@ -902,6 +902,23 @@ bool is_lock_function(struct machine *machine, u64 addr) return false; } +static int get_symbol_name_offset(struct map *map, struct symbol *sym, u64 ip, + char *buf, int size) +{ + u64 offset; + + if (map == NULL || sym == NULL) { + buf[0] = '\0'; + return 0; + } + + offset = map->map_ip(map, ip) - sym->start; + + if (offset) + return scnprintf(buf, size, "%s+%#lx", sym->name, offset); + else + return strlcpy(buf, sym->name, size); +} static int lock_contention_caller(struct evsel *evsel, struct perf_sample *sample, char *buf, int size) { @@ -944,15 +961,8 @@ static int lock_contention_caller(struct evsel *evsel, struct perf_sample *sampl sym = node->ms.sym; if (sym && !is_lock_function(machine, node->ip)) { - struct map *map = node->ms.map; - u64 offset; - - offset = map->map_ip(map, node->ip) - sym->start; - - if (offset) - scnprintf(buf, size, "%s+%#lx", sym->name, offset); - else - strlcpy(buf, sym->name, size); + get_symbol_name_offset(node->ms.map, sym, node->ip, + buf, size); return 0; } From patchwork Thu Sep 8 06:37:52 2022 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Namhyung Kim X-Patchwork-Id: 12969703 X-Patchwork-Delegate: bpf@iogearbox.net Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from vger.kernel.org (vger.kernel.org [23.128.96.18]) by smtp.lore.kernel.org (Postfix) with ESMTP id 48AACC6FA89 for ; Thu, 8 Sep 2022 06:38:08 +0000 (UTC) Received: (majordomo@vger.kernel.org) by vger.kernel.org via listexpand id S230379AbiIHGiG (ORCPT ); Thu, 8 Sep 2022 02:38:06 -0400 Received: from lindbergh.monkeyblade.net ([23.128.96.19]:33792 "EHLO lindbergh.monkeyblade.net" rhost-flags-OK-OK-OK-OK) by vger.kernel.org with ESMTP id S230374AbiIHGiD (ORCPT ); Thu, 8 Sep 2022 02:38:03 -0400 Received: from mail-pj1-x102e.google.com (mail-pj1-x102e.google.com [IPv6:2607:f8b0:4864:20::102e]) by lindbergh.monkeyblade.net (Postfix) with ESMTPS id 48D92C696E; Wed, 7 Sep 2022 23:38:02 -0700 (PDT) Received: by mail-pj1-x102e.google.com with SMTP id s14-20020a17090a6e4e00b0020057c70943so1362042pjm.1; Wed, 07 Sep 2022 23:38:02 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=20210112; h=content-transfer-encoding:mime-version:references:in-reply-to :message-id:date:subject:cc:to:from:sender:from:to:cc:subject:date; bh=AtYzzReklNQuupYTXgf6CIVkK5TRUPCV7GIBcB2m37I=; b=DV796Ll6T89j3sjpAJ/2sM4bIKE+8QfCLmIkAZm9uITBJy8vKgSEhkcwuVqHproNks VbSkMAdJbrAy9gA+q9WWa/M9PyxFzrOJwazjIVbk/p0ipWXIUItQD8z4ND12QN9AIAEg 94ZFpnGDH+DKXz5pM8mzv7vymvj0JNwHPETW9YigRo8JhHvFN/a37hbD/ssXexVgtSeF Jj8XphFTtl7wES5KwnXB3w0iMUQASN1i5hvPKXYFMCjanuPr8sreIvnxdCZXKHarAcMK oq2JIXpugcKd09tl0lQcGwXjtygIucubOCX3hmy0Rwmw1qdMqylE5DNjPqQPCA7sy6W3 mT/w== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20210112; h=content-transfer-encoding:mime-version:references:in-reply-to :message-id:date:subject:cc:to:from:sender:x-gm-message-state:from :to:cc:subject:date; bh=AtYzzReklNQuupYTXgf6CIVkK5TRUPCV7GIBcB2m37I=; b=fOKHx9s0GdA1C/kPJiEfhtYi43vmQOOE3JfEzzO5fW4td+e3U8TpcDPZTSdwir87Gq Im6xDjtHuiRm9HB0UJh9gM66LMxDlDzZTf47KupfvtwuGcmkdqUXe3xFnGVMR6mrzudv g6gBIJFNmQJ+V/UNgFB9RCnTfszPg5nOkt7FA1pIJD9E8RyAvVwlEQrFsAS57Tu4oYd5 pbFNcn7vDNacLAQ/ZvJOtY+A/FX6I5TJROlTWx4GNy2AqOazZHh4PP3xdNFnoTgQ0snX sgCSwi+AmMoGLK21GPyRXo8rQ/o/ObHc7niyZWJjRncbFJOfLoVNDsEXu0K6OPCzpQgV kxqw== X-Gm-Message-State: ACgBeo2vGvsAXuKChbhNQO3nnmq9SwfAepQJQOvHCOlYYLZR+2uE3oKm ZqU+yy6giTAljMqHS31ftWjZR8PSUrY= X-Google-Smtp-Source: AA6agR7aKc93cU9fhSJjoIEDyivdEXSav31T+fdl+ov8F/8MAec91bRtv4PcW6l5BNZNSAsOnnTC3A== X-Received: by 2002:a17:903:24e:b0:172:6c9d:14e0 with SMTP id j14-20020a170903024e00b001726c9d14e0mr7741072plh.84.1662619081689; Wed, 07 Sep 2022 23:38:01 -0700 (PDT) Received: from balhae.hsd1.ca.comcast.net ([2601:647:6780:1040:7f21:d032:3f14:a4e6]) by smtp.gmail.com with ESMTPSA id o33-20020a17090a0a2400b001fb0fc33d72sm890716pjo.47.2022.09.07.23.38.00 (version=TLS1_3 cipher=TLS_AES_256_GCM_SHA384 bits=256/256); Wed, 07 Sep 2022 23:38:01 -0700 (PDT) Sender: Namhyung Kim From: Namhyung Kim To: Arnaldo Carvalho de Melo , Jiri Olsa Cc: Ingo Molnar , Peter Zijlstra , LKML , Ian Rogers , linux-perf-users@vger.kernel.org, Song Liu , bpf@vger.kernel.org Subject: [PATCH 2/4] perf lock contention: Show full callstack with -v option Date: Wed, 7 Sep 2022 23:37:52 -0700 Message-Id: <20220908063754.1369709-3-namhyung@kernel.org> X-Mailer: git-send-email 2.37.2.789.g6183377224-goog In-Reply-To: <20220908063754.1369709-1-namhyung@kernel.org> References: <20220908063754.1369709-1-namhyung@kernel.org> MIME-Version: 1.0 Precedence: bulk List-ID: X-Mailing-List: bpf@vger.kernel.org Currently it shows a caller function for each entry, but users need to see the full call stacks sometimes. Use -v/--verbose option to do that. # perf lock con -a -b -v sleep 3 Looking at the vmlinux_path (8 entries long) symsrc__init: cannot get elf header. Using /proc/kcore for kernel data Using /proc/kallsyms for symbols contended total wait max wait avg wait type caller 1 10.74 us 10.74 us 10.74 us spinlock __bpf_trace_contention_begin+0xb 0xffffffffc03b5c47 bpf_prog_bf07ae9e2cbd02c5_contention_begin+0x117 0xffffffffc03b5c47 bpf_prog_bf07ae9e2cbd02c5_contention_begin+0x117 0xffffffffbb8b8e75 bpf_trace_run2+0x35 0xffffffffbb7eab9b __bpf_trace_contention_begin+0xb 0xffffffffbb7ebe75 queued_spin_lock_slowpath+0x1f5 0xffffffffbc1c26ff _raw_spin_lock+0x1f 0xffffffffbb841015 tick_do_update_jiffies64+0x25 0xffffffffbb8409ee tick_irq_enter+0x9e 1 7.70 us 7.70 us 7.70 us spinlock __bpf_trace_contention_begin+0xb 0xffffffffc03b5c47 bpf_prog_bf07ae9e2cbd02c5_contention_begin+0x117 0xffffffffc03b5c47 bpf_prog_bf07ae9e2cbd02c5_contention_begin+0x117 0xffffffffbb8b8e75 bpf_trace_run2+0x35 0xffffffffbb7eab9b __bpf_trace_contention_begin+0xb 0xffffffffbb7ebe75 queued_spin_lock_slowpath+0x1f5 0xffffffffbc1c26ff _raw_spin_lock+0x1f 0xffffffffbb7bc27e raw_spin_rq_lock_nested+0xe 0xffffffffbb7cef9c load_balance+0x66c Signed-off-by: Namhyung Kim --- tools/perf/builtin-lock.c | 43 ++++++++++++++++++++++----- tools/perf/util/bpf_lock_contention.c | 9 ++++++ tools/perf/util/lock-contention.h | 1 + 3 files changed, 46 insertions(+), 7 deletions(-) diff --git a/tools/perf/builtin-lock.c b/tools/perf/builtin-lock.c index 2a5672f8d22e..e66fbb38d8df 100644 --- a/tools/perf/builtin-lock.c +++ b/tools/perf/builtin-lock.c @@ -972,13 +972,14 @@ static int lock_contention_caller(struct evsel *evsel, struct perf_sample *sampl return -1; } -static u64 callchain_id(struct evsel *evsel, struct perf_sample *sample) +static u64 callchain_id(struct evsel *evsel, struct perf_sample *sample, u64 *callchains) { struct callchain_cursor *cursor = &callchain_cursor; struct machine *machine = &session->machines.host; struct thread *thread; u64 hash = 0; int skip = 0; + int i = 0; int ret; thread = machine__findnew_thread(machine, -1, sample->pid); @@ -1002,6 +1003,9 @@ static u64 callchain_id(struct evsel *evsel, struct perf_sample *sample) if (node == NULL) break; + if (callchains) + callchains[i++] = node->ip; + /* skip first few entries - for lock functions */ if (++skip <= CONTENTION_STACK_SKIP) goto next; @@ -1025,6 +1029,7 @@ static int report_lock_contention_begin_event(struct evsel *evsel, struct lock_seq_stat *seq; u64 addr = evsel__intval(evsel, sample, "lock_addr"); u64 key; + u64 callchains[CONTENTION_STACK_DEPTH]; switch (aggr_mode) { case LOCK_AGGR_ADDR: @@ -1034,7 +1039,9 @@ static int report_lock_contention_begin_event(struct evsel *evsel, key = sample->tid; break; case LOCK_AGGR_CALLER: - key = callchain_id(evsel, sample); + if (verbose) + memset(callchains, 0, sizeof(callchains)); + key = callchain_id(evsel, sample, verbose ? callchains : NULL); break; default: pr_err("Invalid aggregation mode: %d\n", aggr_mode); @@ -1053,6 +1060,12 @@ static int report_lock_contention_begin_event(struct evsel *evsel, ls = lock_stat_findnew(key, caller, flags); if (!ls) return -ENOMEM; + + if (aggr_mode == LOCK_AGGR_CALLER && verbose) { + ls->callstack = memdup(callchains, sizeof(callchains)); + if (ls->callstack == NULL) + return -ENOMEM; + } } ts = thread_stat_findnew(sample->tid); @@ -1117,7 +1130,7 @@ static int report_lock_contention_end_event(struct evsel *evsel, key = sample->tid; break; case LOCK_AGGR_CALLER: - key = callchain_id(evsel, sample); + key = callchain_id(evsel, sample, NULL); break; default: pr_err("Invalid aggregation mode: %d\n", aggr_mode); @@ -1466,7 +1479,7 @@ static void sort_contention_result(void) sort_result(); } -static void print_contention_result(void) +static void print_contention_result(struct lock_contention *con) { struct lock_stat *st; struct lock_key *key; @@ -1505,6 +1518,22 @@ static void print_contention_result(void) } pr_info(" %10s %s\n", get_type_str(st), st->name); + if (verbose) { + struct map *kmap; + struct symbol *sym; + char buf[128]; + u64 ip; + + for (int i = 0; i < CONTENTION_STACK_DEPTH; i++) { + if (!st->callstack || !st->callstack[i]) + break; + + ip = st->callstack[i]; + sym = machine__find_kernel_symbol(con->machine, ip, &kmap); + get_symbol_name_offset(kmap, sym, ip, buf, sizeof(buf)); + pr_info("\t\t\t%#lx %s\n", (unsigned long)ip, buf); + } + } } print_bad_events(bad, total); @@ -1620,6 +1649,8 @@ static int __cmd_contention(int argc, const char **argv) return PTR_ERR(session); } + con.machine = &session->machines.host; + /* for lock function check */ symbol_conf.sort_by_name = true; symbol__init(&session->header.env); @@ -1638,8 +1669,6 @@ static int __cmd_contention(int argc, const char **argv) signal(SIGCHLD, sighandler); signal(SIGTERM, sighandler); - con.machine = &session->machines.host; - con.evlist = evlist__new(); if (con.evlist == NULL) { err = -ENOMEM; @@ -1711,7 +1740,7 @@ static int __cmd_contention(int argc, const char **argv) setup_pager(); sort_contention_result(); - print_contention_result(); + print_contention_result(&con); out_delete: evlist__delete(con.evlist); diff --git a/tools/perf/util/bpf_lock_contention.c b/tools/perf/util/bpf_lock_contention.c index c591a66733ef..6545bee65347 100644 --- a/tools/perf/util/bpf_lock_contention.c +++ b/tools/perf/util/bpf_lock_contention.c @@ -8,6 +8,7 @@ #include "util/thread_map.h" #include "util/lock-contention.h" #include +#include #include #include "bpf_skel/lock_contention.skel.h" @@ -171,6 +172,14 @@ int lock_contention_read(struct lock_contention *con) return -1; } + if (verbose) { + st->callstack = memdup(stack_trace, sizeof(stack_trace)); + if (st->callstack == NULL) { + free(st); + return -1; + } + } + hlist_add_head(&st->hash_entry, con->result); prev_key = key; } diff --git a/tools/perf/util/lock-contention.h b/tools/perf/util/lock-contention.h index 2146efc33396..bdb6e2a61e5b 100644 --- a/tools/perf/util/lock-contention.h +++ b/tools/perf/util/lock-contention.h @@ -11,6 +11,7 @@ struct lock_stat { u64 addr; /* address of lockdep_map, used as ID */ char *name; /* for strcpy(), we cannot use const */ + u64 *callstack; unsigned int nr_acquire; unsigned int nr_acquired; From patchwork Thu Sep 8 06:37:53 2022 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Namhyung Kim X-Patchwork-Id: 12969704 X-Patchwork-Delegate: bpf@iogearbox.net Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from vger.kernel.org (vger.kernel.org [23.128.96.18]) by smtp.lore.kernel.org (Postfix) with ESMTP id 14F7CC6FA83 for ; Thu, 8 Sep 2022 06:38:09 +0000 (UTC) Received: (majordomo@vger.kernel.org) by vger.kernel.org via listexpand id S230399AbiIHGiH (ORCPT ); Thu, 8 Sep 2022 02:38:07 -0400 Received: from lindbergh.monkeyblade.net ([23.128.96.19]:33872 "EHLO lindbergh.monkeyblade.net" rhost-flags-OK-OK-OK-OK) by vger.kernel.org with ESMTP id S230391AbiIHGiF (ORCPT ); Thu, 8 Sep 2022 02:38:05 -0400 Received: from mail-pj1-x102a.google.com (mail-pj1-x102a.google.com [IPv6:2607:f8b0:4864:20::102a]) by lindbergh.monkeyblade.net (Postfix) with ESMTPS id E93BFC59D3; Wed, 7 Sep 2022 23:38:03 -0700 (PDT) Received: by mail-pj1-x102a.google.com with SMTP id fv3so10684353pjb.0; Wed, 07 Sep 2022 23:38:03 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=20210112; h=content-transfer-encoding:mime-version:references:in-reply-to :message-id:date:subject:cc:to:from:sender:from:to:cc:subject:date; bh=DkRY6cqz3FlVC+lgkVgVD+Fncc7hcTBy/c6wwEkWsko=; b=XG0KJtYiLaeVeduW4SOmWJ3DtmrKj7To4hEvZKbKFxkhlnQOxktHJG3XOQtVFbQBRw dUMpM+5ia5vbtD8pqXL9F1fmsw3E8zlWW63IWHXeMIzmdbrmbCl1hnWYOuRpSrx4amAH 6zQ4vtmOp1i+8yhKc4sSYpHoYfE0RtvNz7d+jolY6vZNbmY7XfdlXxc0h628rPgxu1n6 djZzJuq2n50YmthzW/CEs0n6SrvK9jsEI2W6AlL9NYWjZmk3wpyXxHt0xVY8rUr5+1nj +3ec0Ivxyh3OQmO6/51UeWGaHmDodlD2ohfB7HdWRmbf+jCgdNdty1uIXmwgwpYEx/1E WFeQ== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20210112; h=content-transfer-encoding:mime-version:references:in-reply-to :message-id:date:subject:cc:to:from:sender:x-gm-message-state:from :to:cc:subject:date; bh=DkRY6cqz3FlVC+lgkVgVD+Fncc7hcTBy/c6wwEkWsko=; b=7c4C9hp6s/+XmuNUN0X8Y7BR3OU1xn9FdUinvufouEOqnCVDsNjVMBxRH9I6qDutaL +QGSjwlomBcWHB3xTbwuZVdTH5mC790QBficp7Tje+D546zLU6YqW7ca2pYJIjTM2s40 kELXIskST8twuBJw8NInJTaSHUjOVa0+zLecc0dgF7c64ZajwReYuLnFSGtzkGy5/msI vqo/OybLUF04xqV6GdHllG8ruCfZk7qf/+u+hQ5L9FZ0FpiOj8SccfWt9D1vzq6rguid mRZguyxtkhh8EPTOd3HfSf9M/QyMOowL3MybscrSqZYDgDF3yikq79dEhVekoZj7BwbX zpuA== X-Gm-Message-State: ACgBeo1NelYepNOC8Afcl4vQo6FvQ6lhHGC4IAEJ0sj9E4+WXlcX3eDH 0Dt2f0hWFybqh3HQsbfJ6Z8= X-Google-Smtp-Source: AA6agR4VCqnyQmP686jiPAgUunom+WROf7zeas1U0hL7tdnDx1hKEFZx+cGMv376SR9h92kUDWtKyA== X-Received: by 2002:a17:90a:4a0e:b0:1fe:1c89:a6b7 with SMTP id e14-20020a17090a4a0e00b001fe1c89a6b7mr2549600pjh.239.1662619083292; Wed, 07 Sep 2022 23:38:03 -0700 (PDT) Received: from balhae.hsd1.ca.comcast.net ([2601:647:6780:1040:7f21:d032:3f14:a4e6]) by smtp.gmail.com with ESMTPSA id o33-20020a17090a0a2400b001fb0fc33d72sm890716pjo.47.2022.09.07.23.38.01 (version=TLS1_3 cipher=TLS_AES_256_GCM_SHA384 bits=256/256); Wed, 07 Sep 2022 23:38:02 -0700 (PDT) Sender: Namhyung Kim From: Namhyung Kim To: Arnaldo Carvalho de Melo , Jiri Olsa Cc: Ingo Molnar , Peter Zijlstra , LKML , Ian Rogers , linux-perf-users@vger.kernel.org, Song Liu , bpf@vger.kernel.org Subject: [PATCH 3/4] perf lock contention: Allow to change stack depth and skip Date: Wed, 7 Sep 2022 23:37:53 -0700 Message-Id: <20220908063754.1369709-4-namhyung@kernel.org> X-Mailer: git-send-email 2.37.2.789.g6183377224-goog In-Reply-To: <20220908063754.1369709-1-namhyung@kernel.org> References: <20220908063754.1369709-1-namhyung@kernel.org> MIME-Version: 1.0 Precedence: bulk List-ID: X-Mailing-List: bpf@vger.kernel.org It needs stack traces to find callers of locks. To minimize the performance overhead it only collects up to 8 entries for each stack trace. And it skips first 3 entries as they came from BPF, tracepoint and lock functions which are not interested for most users. But it turned out that those numbers are different in some configuration. Using fixed number can result in non meaningful caller names. Let's make them adjustable with --stack-depth and --skip-stack options. On my setup, the default output is like below: # perf lock con -ab -F contended,wait_total sleep 3 contended total wait type caller 28 4.55 ms rwlock:W __bpf_trace_contention_begin+0xb 33 1.67 ms rwlock:W __bpf_trace_contention_begin+0xb 12 580.28 us spinlock __bpf_trace_contention_begin+0xb 60 240.54 us rwsem:R __bpf_trace_contention_begin+0xb 27 64.45 us spinlock __bpf_trace_contention_begin+0xb If I change the stack skip to 5, the result will be like: # perf lock con -ab -F contended,wait_total --stack-skip 5 sleep 3 contended total wait type caller 32 715.45 us spinlock folio_lruvec_lock_irqsave+0x61 26 550.22 us spinlock folio_lruvec_lock_irqsave+0x61 15 486.93 us rwsem:R mmap_read_lock+0x13 12 139.66 us rwsem:W vm_mmap_pgoff+0x93 1 7.04 us spinlock tick_do_update_jiffies64+0x25 Signed-off-by: Namhyung Kim --- tools/perf/Documentation/perf-lock.txt | 6 ++++++ tools/perf/builtin-lock.c | 20 +++++++++++++++----- tools/perf/util/bpf_lock_contention.c | 7 ++++--- tools/perf/util/lock-contention.h | 2 ++ 4 files changed, 27 insertions(+), 8 deletions(-) diff --git a/tools/perf/Documentation/perf-lock.txt b/tools/perf/Documentation/perf-lock.txt index 193c5d8b8db9..5f2dc634258e 100644 --- a/tools/perf/Documentation/perf-lock.txt +++ b/tools/perf/Documentation/perf-lock.txt @@ -148,6 +148,12 @@ CONTENTION OPTIONS --map-nr-entries:: Maximum number of BPF map entries (default: 10240). +--max-stack:: + Maximum stack depth when collecting lock contention (default: 8). + +--stack-skip + Number of stack depth to skip when finding a lock caller (default: 3). + SEE ALSO -------- diff --git a/tools/perf/builtin-lock.c b/tools/perf/builtin-lock.c index e66fbb38d8df..13900ac1d186 100644 --- a/tools/perf/builtin-lock.c +++ b/tools/perf/builtin-lock.c @@ -56,6 +56,8 @@ static bool combine_locks; static bool show_thread_stats; static bool use_bpf; static unsigned long bpf_map_entries = 10240; +static int max_stack_depth = CONTENTION_STACK_DEPTH; +static int stack_skip = CONTENTION_STACK_SKIP; static enum { LOCK_AGGR_ADDR, @@ -939,7 +941,7 @@ static int lock_contention_caller(struct evsel *evsel, struct perf_sample *sampl /* use caller function name from the callchain */ ret = thread__resolve_callchain(thread, cursor, evsel, sample, - NULL, NULL, CONTENTION_STACK_DEPTH); + NULL, NULL, max_stack_depth); if (ret != 0) { thread__put(thread); return -1; @@ -956,7 +958,7 @@ static int lock_contention_caller(struct evsel *evsel, struct perf_sample *sampl break; /* skip first few entries - for lock functions */ - if (++skip <= CONTENTION_STACK_SKIP) + if (++skip <= stack_skip) goto next; sym = node->ms.sym; @@ -988,7 +990,7 @@ static u64 callchain_id(struct evsel *evsel, struct perf_sample *sample, u64 *ca /* use caller function name from the callchain */ ret = thread__resolve_callchain(thread, cursor, evsel, sample, - NULL, NULL, CONTENTION_STACK_DEPTH); + NULL, NULL, max_stack_depth); thread__put(thread); if (ret != 0) @@ -1007,7 +1009,7 @@ static u64 callchain_id(struct evsel *evsel, struct perf_sample *sample, u64 *ca callchains[i++] = node->ip; /* skip first few entries - for lock functions */ - if (++skip <= CONTENTION_STACK_SKIP) + if (++skip <= stack_skip) goto next; if (node->ms.sym && is_lock_function(machine, node->ip)) @@ -1524,7 +1526,7 @@ static void print_contention_result(struct lock_contention *con) char buf[128]; u64 ip; - for (int i = 0; i < CONTENTION_STACK_DEPTH; i++) { + for (int i = 0; i < max_stack_depth; i++) { if (!st->callstack || !st->callstack[i]) break; @@ -1641,6 +1643,8 @@ static int __cmd_contention(int argc, const char **argv) .target = &target, .result = &lockhash_table[0], .map_nr_entries = bpf_map_entries, + .max_stack = max_stack_depth, + .stack_skip = stack_skip, }; session = perf_session__new(use_bpf ? NULL : &data, &eops); @@ -1904,6 +1908,12 @@ int cmd_lock(int argc, const char **argv) "Trace on existing thread id (exclusive to --pid)"), OPT_CALLBACK(0, "map-nr-entries", &bpf_map_entries, "num", "Max number of BPF map entries", parse_map_entry), + OPT_INTEGER(0, "max-stack", &max_stack_depth, + "Set the maximum stack depth when collecting lock contention, " + "Default: " __stringify(CONTENTION_STACK_DEPTH)), + OPT_INTEGER(0, "stack-skip", &stack_skip, + "Set the number of stack depth to skip when finding a lock caller, " + "Default: " __stringify(CONTENTION_STACK_SKIP)), OPT_PARENT(lock_options) }; diff --git a/tools/perf/util/bpf_lock_contention.c b/tools/perf/util/bpf_lock_contention.c index 6545bee65347..ef5323c78ffc 100644 --- a/tools/perf/util/bpf_lock_contention.c +++ b/tools/perf/util/bpf_lock_contention.c @@ -41,6 +41,7 @@ int lock_contention_prepare(struct lock_contention *con) return -1; } + bpf_map__set_value_size(skel->maps.stacks, con->max_stack * sizeof(u64)); bpf_map__set_max_entries(skel->maps.stacks, con->map_nr_entries); bpf_map__set_max_entries(skel->maps.lock_stat, con->map_nr_entries); @@ -115,7 +116,7 @@ int lock_contention_read(struct lock_contention *con) struct lock_contention_data data; struct lock_stat *st; struct machine *machine = con->machine; - u64 stack_trace[CONTENTION_STACK_DEPTH]; + u64 stack_trace[con->max_stack]; fd = bpf_map__fd(skel->maps.lock_stat); stack = bpf_map__fd(skel->maps.stacks); @@ -146,9 +147,9 @@ int lock_contention_read(struct lock_contention *con) bpf_map_lookup_elem(stack, &key, stack_trace); /* skip BPF + lock internal functions */ - idx = CONTENTION_STACK_SKIP; + idx = con->stack_skip; while (is_lock_function(machine, stack_trace[idx]) && - idx < CONTENTION_STACK_DEPTH - 1) + idx < con->max_stack - 1) idx++; st->addr = stack_trace[idx]; diff --git a/tools/perf/util/lock-contention.h b/tools/perf/util/lock-contention.h index bdb6e2a61e5b..67db311fc9df 100644 --- a/tools/perf/util/lock-contention.h +++ b/tools/perf/util/lock-contention.h @@ -115,6 +115,8 @@ struct lock_contention { struct hlist_head *result; unsigned long map_nr_entries; unsigned long lost; + int max_stack; + int stack_skip; }; #ifdef HAVE_BPF_SKEL From patchwork Thu Sep 8 06:37:54 2022 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Namhyung Kim X-Patchwork-Id: 12969705 X-Patchwork-Delegate: bpf@iogearbox.net Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from vger.kernel.org (vger.kernel.org [23.128.96.18]) by smtp.lore.kernel.org (Postfix) with ESMTP id A38FDC6FA8A for ; Thu, 8 Sep 2022 06:38:10 +0000 (UTC) Received: (majordomo@vger.kernel.org) by vger.kernel.org via listexpand id S230397AbiIHGiI (ORCPT ); Thu, 8 Sep 2022 02:38:08 -0400 Received: from lindbergh.monkeyblade.net ([23.128.96.19]:33772 "EHLO lindbergh.monkeyblade.net" rhost-flags-OK-OK-OK-OK) by vger.kernel.org with ESMTP id S229563AbiIHGiF (ORCPT ); Thu, 8 Sep 2022 02:38:05 -0400 Received: from mail-pf1-x431.google.com (mail-pf1-x431.google.com [IPv6:2607:f8b0:4864:20::431]) by lindbergh.monkeyblade.net (Postfix) with ESMTPS id 530FAC58EB; Wed, 7 Sep 2022 23:38:05 -0700 (PDT) Received: by mail-pf1-x431.google.com with SMTP id c198so7069066pfc.13; Wed, 07 Sep 2022 23:38:05 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=20210112; h=content-transfer-encoding:mime-version:references:in-reply-to :message-id:date:subject:cc:to:from:sender:from:to:cc:subject:date; bh=XrKerZgfr8OsAjyFj/pfcukHC6xfGt6r5ZUbVw1UMjA=; b=oOdlYalln0v3/1vyUGIKPtIcrNI+M5Q3tPPtiaFjUtIQwIdFPvQHolv1ZZff2MKD0K 5Z7+1eocGHNmvFLqu1gK9bD5qKRVVfxwD6MDAmngWQQtWzArLLCTMbOWSGVyDmlqkmau Neaifc+kN8U2q2v7EPjbq34WPdHXtumPbLkDNcjdllxbyUUeLdKBcZxg+3c3PEZBqJqV cbDoGoLHAeKbgN60n3JMHsN7dySDoP8UPsQqO8smJni3T6uTtNStCeJAlRQd3bBKhhcm 8NyRla0UmgTwdq9+zrHRXk2sgxJ8cu7KyqKCOQ9m1+D7S9/yLh8J0oE8NyP3Tomp/XC1 ZRyQ== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20210112; h=content-transfer-encoding:mime-version:references:in-reply-to :message-id:date:subject:cc:to:from:sender:x-gm-message-state:from :to:cc:subject:date; bh=XrKerZgfr8OsAjyFj/pfcukHC6xfGt6r5ZUbVw1UMjA=; b=BMvFBeZqksgIaBHAYGZBX7jnyDNfA/4KU0izfm+kNeysT8Z4y77v0y9LRtfrIoE8+d D/xpMhW8xDe6mk9S7BPEznWIo9om7AyocAyHH6YI7LpYcnvpfkcyAVFcethObFNwHCdj JB/QW1mxGYGHi2WPrylxCiCrUSTaVxId1shmadt3SOYPEwZl6Msdk30akGdXYeMBevoO tAxvwsBMyrnUbUFT+tLQAvtBJYxgUtDY9J5INSHCm1fyFf3M6Tq/I4L6LfDudgw8LpOr TZLsuJoSSrnotrHe8meZlKuOgGow45YQGR6AmD8s2ycMw9RKXS1BJkdHPEasRTzIVPQB WV6A== X-Gm-Message-State: ACgBeo3mq0JsTpp6+vEiRdZfuspoIkCLcw9yO3YE85QQS9+/+jXqaTMQ g2EvYpGsySVVjlpmEU4/640= X-Google-Smtp-Source: AA6agR6Wl3fV/Z9sVDie6QCrAqMZ49pcxWl4rCafzAzHBFQFVXu+/mGB+oPJ0jddzwvRiMsZ1KnAXQ== X-Received: by 2002:a63:e317:0:b0:432:38c0:bde3 with SMTP id f23-20020a63e317000000b0043238c0bde3mr6352573pgh.567.1662619084779; Wed, 07 Sep 2022 23:38:04 -0700 (PDT) Received: from balhae.hsd1.ca.comcast.net ([2601:647:6780:1040:7f21:d032:3f14:a4e6]) by smtp.gmail.com with ESMTPSA id o33-20020a17090a0a2400b001fb0fc33d72sm890716pjo.47.2022.09.07.23.38.03 (version=TLS1_3 cipher=TLS_AES_256_GCM_SHA384 bits=256/256); Wed, 07 Sep 2022 23:38:04 -0700 (PDT) Sender: Namhyung Kim From: Namhyung Kim To: Arnaldo Carvalho de Melo , Jiri Olsa Cc: Ingo Molnar , Peter Zijlstra , LKML , Ian Rogers , linux-perf-users@vger.kernel.org, Song Liu , bpf@vger.kernel.org Subject: [PATCH 4/4] perf lock contention: Skip stack trace from BPF Date: Wed, 7 Sep 2022 23:37:54 -0700 Message-Id: <20220908063754.1369709-5-namhyung@kernel.org> X-Mailer: git-send-email 2.37.2.789.g6183377224-goog In-Reply-To: <20220908063754.1369709-1-namhyung@kernel.org> References: <20220908063754.1369709-1-namhyung@kernel.org> MIME-Version: 1.0 Precedence: bulk List-ID: X-Mailing-List: bpf@vger.kernel.org Currently it collects stack traces to max size then skip entries. Because we don't have control how to skip perf callchains. But BPF can do it with bpf_get_stackid() with a flag. Say we have max-stack=4 and stack-skip=2, we get these stack traces. Before: After: ---> +---+ <-- ---> +---+ <-- / | | \ / | | \ | +---+ usable | +---+ | max | | / max | | | stack +---+ <-- stack +---+ usable | | X | | | | | | +---+ skip | +---+ | \ | X | \ | | / ---> +---+ ---> +---+ <-- <=== collection | X | +---+ skip | X | +---+ Signed-off-by: Namhyung Kim --- tools/perf/util/bpf_lock_contention.c | 7 ++++--- tools/perf/util/bpf_skel/lock_contention.bpf.c | 3 ++- 2 files changed, 6 insertions(+), 4 deletions(-) diff --git a/tools/perf/util/bpf_lock_contention.c b/tools/perf/util/bpf_lock_contention.c index ef5323c78ffc..efe5b9968e77 100644 --- a/tools/perf/util/bpf_lock_contention.c +++ b/tools/perf/util/bpf_lock_contention.c @@ -93,6 +93,8 @@ int lock_contention_prepare(struct lock_contention *con) bpf_map_update_elem(fd, &pid, &val, BPF_ANY); } + skel->bss->stack_skip = con->stack_skip; + lock_contention_bpf__attach(skel); return 0; } @@ -127,7 +129,7 @@ int lock_contention_read(struct lock_contention *con) while (!bpf_map_get_next_key(fd, &prev_key, &key)) { struct map *kmap; struct symbol *sym; - int idx; + int idx = 0; bpf_map_lookup_elem(fd, &key, &data); st = zalloc(sizeof(*st)); @@ -146,8 +148,7 @@ int lock_contention_read(struct lock_contention *con) bpf_map_lookup_elem(stack, &key, stack_trace); - /* skip BPF + lock internal functions */ - idx = con->stack_skip; + /* skip lock internal functions */ while (is_lock_function(machine, stack_trace[idx]) && idx < con->max_stack - 1) idx++; diff --git a/tools/perf/util/bpf_skel/lock_contention.bpf.c b/tools/perf/util/bpf_skel/lock_contention.bpf.c index 9e8b94eb6320..e107d71f0f1a 100644 --- a/tools/perf/util/bpf_skel/lock_contention.bpf.c +++ b/tools/perf/util/bpf_skel/lock_contention.bpf.c @@ -72,6 +72,7 @@ struct { int enabled; int has_cpu; int has_task; +int stack_skip; /* error stat */ unsigned long lost; @@ -117,7 +118,7 @@ int contention_begin(u64 *ctx) pelem->timestamp = bpf_ktime_get_ns(); pelem->lock = (__u64)ctx[0]; pelem->flags = (__u32)ctx[1]; - pelem->stack_id = bpf_get_stackid(ctx, &stacks, BPF_F_FAST_STACK_CMP); + pelem->stack_id = bpf_get_stackid(ctx, &stacks, BPF_F_FAST_STACK_CMP | stack_skip); if (pelem->stack_id < 0) lost++;