From patchwork Fri Mar 22 00:00:36 2024 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Andrii Nakryiko X-Patchwork-Id: 13599471 X-Patchwork-Delegate: bpf@iogearbox.net Received: from smtp.kernel.org (aws-us-west-2-korg-mail-1.web.codeaurora.org [10.30.226.201]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (No client certificate requested) by smtp.subspace.kernel.org (Postfix) with ESMTPS id BABC25CB8 for ; Fri, 22 Mar 2024 00:00:53 +0000 (UTC) Authentication-Results: smtp.subspace.kernel.org; arc=none smtp.client-ip=10.30.226.201 ARC-Seal: i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1711065653; cv=none; b=OHu6qgT6tg00MmwxaOx5QkJZcuBVovnxzCzS4M5MudFt1VvSJbRkkxz6cNWuUL6FwjJ1hCOgwJkObMRtSzK45ihZ8ezrrGHlRqAW2ip9/EuZjqpor8HNikac1v3bNhQyo7cioam9uxfCYA68H8F8VY8vo/1yILI8sz9pWyww70g= ARC-Message-Signature: i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1711065653; c=relaxed/simple; bh=3Ho0kc21D/CUkprAbOk1NKR5OimHhuqMiecj4/zbJQg=; h=From:To:Cc:Subject:Date:Message-ID:In-Reply-To:References: MIME-Version; b=QUxTTip+wjy0iGZiZAubnCA+61PYwYJAWABfO+XodOz10if2UcQU0avkxwPE3Zccbjlvki6pGYRK6KMHfz46zHmIItNucOuN4g9v25HkCa6/dHQKoCKAti5wRWdXR9j1VMX4zaEWQULk0d+1niVyPQfmGKDwhFU/gk0BN7Mo8B0= ARC-Authentication-Results: i=1; smtp.subspace.kernel.org; dkim=pass (2048-bit key) header.d=kernel.org header.i=@kernel.org header.b=Y5U0R6V1; arc=none smtp.client-ip=10.30.226.201 Authentication-Results: smtp.subspace.kernel.org; dkim=pass (2048-bit key) header.d=kernel.org header.i=@kernel.org header.b="Y5U0R6V1" Received: by smtp.kernel.org (Postfix) with ESMTPSA id 26B7BC433F1; Fri, 22 Mar 2024 00:00:53 +0000 (UTC) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/simple; d=kernel.org; s=k20201202; t=1711065653; bh=3Ho0kc21D/CUkprAbOk1NKR5OimHhuqMiecj4/zbJQg=; h=From:To:Cc:Subject:Date:In-Reply-To:References:From; b=Y5U0R6V1WYfD48S4ewSfgf7wp/0nl2s/6ochiUn1Ey1QdcFQy/FA7tkua3aiQJ8+z oa08mhFFQS5hIYQuRmuVPCl4loL5Bd9CM1A4fTPUkSiljennZs4Ib6OAGM7W7HYHoU xXiCL3fF3BzIpAcNm8Ks7RxTSFNSsmHIVOcziwle/ZbQegJo3NwkP51Mjfiey9ZYJw M5VW5mQwJqTkbbYH+HsbHC0HFXoVDC6AF0p84uSSI3iQeJMm0j8rjHZ1L71SfKD1vY hciRBpPI/RhsskSV8hVDxgraCE6sf0O4mtVm8gIvEHyhHk2lxFMda1rmFRAGDDqHD0 MGm974Tk5XXnA== From: Andrii Nakryiko To: bpf@vger.kernel.org, ast@kernel.org, daniel@iogearbox.net, martin.lau@kernel.org Cc: andrii@kernel.org, kernel-team@meta.com Subject: [PATCH bpf-next 1/6] selftests/bpf: rename and clean up userspace-triggered benchmarks Date: Thu, 21 Mar 2024 17:00:36 -0700 Message-ID: <20240322000041.2919948-2-andrii@kernel.org> X-Mailer: git-send-email 2.43.0 In-Reply-To: <20240322000041.2919948-1-andrii@kernel.org> References: <20240322000041.2919948-1-andrii@kernel.org> Precedence: bulk X-Mailing-List: bpf@vger.kernel.org List-Id: List-Subscribe: List-Unsubscribe: MIME-Version: 1.0 X-Patchwork-Delegate: bpf@iogearbox.net Rename uprobe-base to more precise usermode-count (it will match other baseline-like benchmarks, kernel-count and syscall-count). Also use BENCH_TRIG_USERMODE() macro to define all usermode-based triggering benchmarks, which include usermode-count and uprobe/uretprobe benchmarks. Signed-off-by: Andrii Nakryiko --- tools/testing/selftests/bpf/bench.c | 14 ++- .../selftests/bpf/benchs/bench_trigger.c | 106 ++++++------------ .../selftests/bpf/benchs/run_bench_uprobes.sh | 2 +- 3 files changed, 49 insertions(+), 73 deletions(-) diff --git a/tools/testing/selftests/bpf/bench.c b/tools/testing/selftests/bpf/bench.c index b2b4c391eb0a..7ca1e1eb5c30 100644 --- a/tools/testing/selftests/bpf/bench.c +++ b/tools/testing/selftests/bpf/bench.c @@ -491,7 +491,12 @@ extern const struct bench bench_rename_kretprobe; extern const struct bench bench_rename_rawtp; extern const struct bench bench_rename_fentry; extern const struct bench bench_rename_fexit; + +/* pure counting benchmarks to establish theoretical lmits */ +extern const struct bench bench_trig_usermode_count; extern const struct bench bench_trig_base; + +/* kernel-side syscall-triggered benchmarks */ extern const struct bench bench_trig_tp; extern const struct bench bench_trig_rawtp; extern const struct bench bench_trig_kprobe; @@ -502,13 +507,15 @@ extern const struct bench bench_trig_fentry; extern const struct bench bench_trig_fexit; extern const struct bench bench_trig_fentry_sleep; extern const struct bench bench_trig_fmodret; -extern const struct bench bench_trig_uprobe_base; + +/* uprobe/uretprobe benchmarks */ extern const struct bench bench_trig_uprobe_nop; extern const struct bench bench_trig_uretprobe_nop; extern const struct bench bench_trig_uprobe_push; extern const struct bench bench_trig_uretprobe_push; extern const struct bench bench_trig_uprobe_ret; extern const struct bench bench_trig_uretprobe_ret; + extern const struct bench bench_rb_libbpf; extern const struct bench bench_rb_custom; extern const struct bench bench_pb_libbpf; @@ -539,7 +546,10 @@ static const struct bench *benchs[] = { &bench_rename_rawtp, &bench_rename_fentry, &bench_rename_fexit, + /* pure counting benchmarks for establishing theoretical limits */ + &bench_trig_usermode_count, &bench_trig_base, + /* syscall-driven triggering benchmarks */ &bench_trig_tp, &bench_trig_rawtp, &bench_trig_kprobe, @@ -550,7 +560,7 @@ static const struct bench *benchs[] = { &bench_trig_fexit, &bench_trig_fentry_sleep, &bench_trig_fmodret, - &bench_trig_uprobe_base, + /* uprobes */ &bench_trig_uprobe_nop, &bench_trig_uretprobe_nop, &bench_trig_uprobe_push, diff --git a/tools/testing/selftests/bpf/benchs/bench_trigger.c b/tools/testing/selftests/bpf/benchs/bench_trigger.c index b7aea79495ba..68e703b46d9b 100644 --- a/tools/testing/selftests/bpf/benchs/bench_trigger.c +++ b/tools/testing/selftests/bpf/benchs/bench_trigger.c @@ -12,6 +12,7 @@ /* BPF triggering benchmarks */ static struct trigger_ctx { struct trigger_bench *skel; + bool usermode_counters; } ctx; static struct counter base_hits[MAX_BUCKETS]; @@ -72,7 +73,10 @@ static void *trigger_producer(void *input) static void trigger_measure(struct bench_res *res) { - res->hits = sum_and_reset_counters(ctx.skel->bss->hits); + if (ctx.usermode_counters) + res->hits = sum_and_reset_counters(base_hits); + else + res->hits = sum_and_reset_counters(ctx.skel->bss->hits); } static void setup_ctx(void) @@ -190,7 +194,7 @@ __weak void uprobe_target_ret(void) asm volatile (""); } -static void *uprobe_base_producer(void *input) +static void *uprobe_producer_count(void *input) { while (true) { uprobe_target_nop(); @@ -246,32 +250,37 @@ static void usetup(bool use_retprobe, void *target_addr) ctx.skel->links.bench_trigger_uprobe = link; } -static void uprobe_setup_nop(void) +static void usermode_count_setup(void) +{ + ctx.usermode_counters = true; +} + +static void uprobe_nop_setup(void) { usetup(false, &uprobe_target_nop); } -static void uretprobe_setup_nop(void) +static void uretprobe_nop_setup(void) { usetup(true, &uprobe_target_nop); } -static void uprobe_setup_push(void) +static void uprobe_push_setup(void) { usetup(false, &uprobe_target_push); } -static void uretprobe_setup_push(void) +static void uretprobe_push_setup(void) { usetup(true, &uprobe_target_push); } -static void uprobe_setup_ret(void) +static void uprobe_ret_setup(void) { usetup(false, &uprobe_target_ret); } -static void uretprobe_setup_ret(void) +static void uretprobe_ret_setup(void) { usetup(true, &uprobe_target_ret); } @@ -385,65 +394,22 @@ const struct bench bench_trig_fmodret = { .report_final = hits_drops_report_final, }; -const struct bench bench_trig_uprobe_base = { - .name = "trig-uprobe-base", - .setup = NULL, /* no uprobe/uretprobe is attached */ - .producer_thread = uprobe_base_producer, - .measure = trigger_base_measure, - .report_progress = hits_drops_report_progress, - .report_final = hits_drops_report_final, -}; - -const struct bench bench_trig_uprobe_nop = { - .name = "trig-uprobe-nop", - .setup = uprobe_setup_nop, - .producer_thread = uprobe_producer_nop, - .measure = trigger_measure, - .report_progress = hits_drops_report_progress, - .report_final = hits_drops_report_final, -}; - -const struct bench bench_trig_uretprobe_nop = { - .name = "trig-uretprobe-nop", - .setup = uretprobe_setup_nop, - .producer_thread = uprobe_producer_nop, - .measure = trigger_measure, - .report_progress = hits_drops_report_progress, - .report_final = hits_drops_report_final, -}; - -const struct bench bench_trig_uprobe_push = { - .name = "trig-uprobe-push", - .setup = uprobe_setup_push, - .producer_thread = uprobe_producer_push, - .measure = trigger_measure, - .report_progress = hits_drops_report_progress, - .report_final = hits_drops_report_final, -}; - -const struct bench bench_trig_uretprobe_push = { - .name = "trig-uretprobe-push", - .setup = uretprobe_setup_push, - .producer_thread = uprobe_producer_push, - .measure = trigger_measure, - .report_progress = hits_drops_report_progress, - .report_final = hits_drops_report_final, -}; - -const struct bench bench_trig_uprobe_ret = { - .name = "trig-uprobe-ret", - .setup = uprobe_setup_ret, - .producer_thread = uprobe_producer_ret, - .measure = trigger_measure, - .report_progress = hits_drops_report_progress, - .report_final = hits_drops_report_final, -}; - -const struct bench bench_trig_uretprobe_ret = { - .name = "trig-uretprobe-ret", - .setup = uretprobe_setup_ret, - .producer_thread = uprobe_producer_ret, - .measure = trigger_measure, - .report_progress = hits_drops_report_progress, - .report_final = hits_drops_report_final, -}; +/* uprobe benchmarks */ +#define BENCH_TRIG_USERMODE(KIND, PRODUCER, NAME) \ +const struct bench bench_trig_##KIND = { \ + .name = "trig-" NAME, \ + .validate = trigger_validate, \ + .setup = KIND##_setup, \ + .producer_thread = uprobe_producer_##PRODUCER, \ + .measure = trigger_measure, \ + .report_progress = hits_drops_report_progress, \ + .report_final = hits_drops_report_final, \ +} + +BENCH_TRIG_USERMODE(usermode_count, count, "usermode-count"); +BENCH_TRIG_USERMODE(uprobe_nop, nop, "uprobe-nop"); +BENCH_TRIG_USERMODE(uprobe_push, push, "uprobe-push"); +BENCH_TRIG_USERMODE(uprobe_ret, ret, "uprobe-ret"); +BENCH_TRIG_USERMODE(uretprobe_nop, nop, "uretprobe-nop"); +BENCH_TRIG_USERMODE(uretprobe_push, push, "uretprobe-push"); +BENCH_TRIG_USERMODE(uretprobe_ret, ret, "uretprobe-ret"); diff --git a/tools/testing/selftests/bpf/benchs/run_bench_uprobes.sh b/tools/testing/selftests/bpf/benchs/run_bench_uprobes.sh index 9bdcc74e03a4..36021d243397 100755 --- a/tools/testing/selftests/bpf/benchs/run_bench_uprobes.sh +++ b/tools/testing/selftests/bpf/benchs/run_bench_uprobes.sh @@ -2,7 +2,7 @@ set -eufo pipefail -for i in base {uprobe,uretprobe}-{nop,push,ret} +for i in usermode-count base {uprobe,uretprobe}-{nop,push,ret} do summary=$(sudo ./bench -w2 -d5 -a trig-$i | tail -n1 | cut -d'(' -f1 | cut -d' ' -f3-) printf "%-15s: %s\n" $i "$summary" From patchwork Fri Mar 22 00:00:37 2024 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 8bit X-Patchwork-Submitter: Andrii Nakryiko X-Patchwork-Id: 13599472 X-Patchwork-Delegate: bpf@iogearbox.net Received: from smtp.kernel.org (aws-us-west-2-korg-mail-1.web.codeaurora.org [10.30.226.201]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (No client certificate requested) by smtp.subspace.kernel.org (Postfix) with ESMTPS id 9FA58BE65 for ; Fri, 22 Mar 2024 00:00:56 +0000 (UTC) Authentication-Results: smtp.subspace.kernel.org; arc=none smtp.client-ip=10.30.226.201 ARC-Seal: i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1711065656; cv=none; b=gsgS/iBdCzdZM5jnkuVovQbIL2wMvO3S3zZf/BFVH06OzVNhE2C4XCTLeVFY1EpvF2E2ThV1VTxzyB88d1LKVTDSXGe1KF6at0xANKywLKD6Ji02k9c2eefGXlLCh23HDi1wcbNlTPJzgH9CWMmYfMm5Bp46iZGAHXmorYyk4x0= ARC-Message-Signature: i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1711065656; c=relaxed/simple; bh=QBix4nOMRi0CG6TSGD0Dr9v+cIJ8puRms/VCh13VqjA=; h=From:To:Cc:Subject:Date:Message-ID:In-Reply-To:References: MIME-Version:Content-Type; b=j1ElRdhluGL0mqpPP6GTYFfuAPNZlltTLy20/k2DJxxynSvdg8wsheD2J0cWW18jHu4gdUUT6JVePoo0BV7QlRGuQipIFBU3T1ELoe5GTz9YrV0d0NazFZ7tR5KCLP4Z05YHGoLb8kC9uXF2YAoAbWQYQRv2eT9Riuhml+VFZQI= ARC-Authentication-Results: i=1; smtp.subspace.kernel.org; dkim=pass (2048-bit key) header.d=kernel.org header.i=@kernel.org header.b=MBP6ixB1; arc=none smtp.client-ip=10.30.226.201 Authentication-Results: smtp.subspace.kernel.org; dkim=pass (2048-bit key) header.d=kernel.org header.i=@kernel.org header.b="MBP6ixB1" Received: by smtp.kernel.org (Postfix) with ESMTPSA id 52F30C433C7; Fri, 22 Mar 2024 00:00:56 +0000 (UTC) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/simple; d=kernel.org; s=k20201202; t=1711065656; bh=QBix4nOMRi0CG6TSGD0Dr9v+cIJ8puRms/VCh13VqjA=; h=From:To:Cc:Subject:Date:In-Reply-To:References:From; b=MBP6ixB1/qzUYS7LIym1s3N2nnAqLJqSWqnp6ppCYyNoVp1F3THKttGbhhQeCISUs TjhjgTh0y7+pbXsNoEfGYPoKXseK86/a/mBHhoUpK5EMCEqJl3DAbcNMtYV0hTxNDG Nzq5q52qpEU7ycHa51pspLnqaRcMtXkygtGvY2dAzpcnoQfWi0I00I1n2UZUMF/08s b9XnGhxW3pjEoSlTrqjb8ZaALqcm3+SHMvoumq1INSl5rPoggvvrd3KUUtEVXqxbRs XP6OqotHDPuSjlvC803N0IhDs8vlWn2Eck/DZWJ9F3Bwf9YWstk5kfHTO+EXmplpOs YqwrTQ78RXXCQ== From: Andrii Nakryiko To: bpf@vger.kernel.org, ast@kernel.org, daniel@iogearbox.net, martin.lau@kernel.org Cc: andrii@kernel.org, kernel-team@meta.com, Jiri Olsa Subject: [PATCH bpf-next 2/6] selftests/bpf: add batched, mostly in-kernel BPF triggering benchmarks Date: Thu, 21 Mar 2024 17:00:37 -0700 Message-ID: <20240322000041.2919948-3-andrii@kernel.org> X-Mailer: git-send-email 2.43.0 In-Reply-To: <20240322000041.2919948-1-andrii@kernel.org> References: <20240322000041.2919948-1-andrii@kernel.org> Precedence: bulk X-Mailing-List: bpf@vger.kernel.org List-Id: List-Subscribe: List-Unsubscribe: MIME-Version: 1.0 X-Patchwork-Delegate: bpf@iogearbox.net Existing kprobe/fentry triggering benchmarks have 1-to-1 mapping between one syscall execution and BPF program run. While we use a fast get_pgid() syscall, syscall overhead can still be non-trivial. This patch adds kprobe/fentry set of benchmarks significantly amortizing the cost of syscall vs actual BPF triggering overhead. We do this by employing BPF_PROG_TEST_RUN command to trigger "driver" raw_tp program which does a tight parameterized loop calling cheap BPF helper (bpf_get_numa_node_id()), to which kprobe/fentry programs are attached for benchmarking. This way 1 bpf() syscall causes N executions of BPF program being benchmarked. N defaults to 100, but can be adjusted with --trig-batch-iters CLI argument. For comparison we also implement a new baseline program that instead of triggering another BPF program just does N atomic per-CPU counter increments, establishing the limit for all other types of program within this batched benchmarking setup. Taking the final set of benchmarks added in this patch set (including tp/raw_tp/fmodret, added in later patch), and keeping for now "legacy" syscall-driven benchmarks, we can capture all triggering benchmarks in one place for comparison, before we remove the legacy ones (and rename xxx-batched into just xxx). $ benchs/run_bench_trigger.sh usermode-count : 79.500 ± 0.024M/s kernel-count : 49.949 ± 0.081M/s syscall-count : 9.009 ± 0.007M/s fentry-batch : 31.002 ± 0.015M/s fexit-batch : 20.372 ± 0.028M/s fmodret-batch : 21.651 ± 0.659M/s rawtp-batch : 36.775 ± 0.264M/s tp-batch : 19.411 ± 0.248M/s kprobe-batch : 12.949 ± 0.220M/s kprobe-multi-batch : 15.400 ± 0.007M/s kretprobe-batch : 5.559 ± 0.011M/s kretprobe-multi-batch: 5.861 ± 0.003M/s fentry-legacy : 8.329 ± 0.004M/s fexit-legacy : 6.239 ± 0.003M/s fmodret-legacy : 6.595 ± 0.001M/s rawtp-legacy : 8.305 ± 0.004M/s tp-legacy : 6.382 ± 0.001M/s kprobe-legacy : 5.528 ± 0.003M/s kprobe-multi-legacy : 5.864 ± 0.022M/s kretprobe-legacy : 3.081 ± 0.001M/s kretprobe-multi-legacy: 3.193 ± 0.001M/s Note how xxx-batch variants are measured with significantly higher throughput, even though it's exactly the same in-kernel overhead. As such, results can be compared only between benchmarks of the same kind (syscall vs batched): fentry-legacy : 8.329 ± 0.004M/s fentry-batch : 31.002 ± 0.015M/s kprobe-multi-legacy : 5.864 ± 0.022M/s kprobe-multi-batch : 15.400 ± 0.007M/s Note also that syscall-count is setting a theoretical limit for syscall-triggered benchmarks, while kernel-count is setting similar limits for batch variants. usermode-count is a happy and unachievable case of user space counting without doing any syscalls, and is mostly the measure of CPU speed for such a trivial benchmark. As was mentioned, tp/raw_tp/fmodret require kernel-side kfunc to produce similar benchmark, which we address in a separate patch. Note that run_bench_trigger.sh allows to override a list of benchmarks to run, which is very useful for performance work. Cc: Jiri Olsa Signed-off-by: Andrii Nakryiko --- tools/testing/selftests/bpf/bench.c | 21 ++- .../selftests/bpf/benchs/bench_trigger.c | 130 +++++++++++++++++- .../selftests/bpf/benchs/run_bench_trigger.sh | 24 +++- .../selftests/bpf/progs/trigger_bench.c | 67 ++++++++- 4 files changed, 235 insertions(+), 7 deletions(-) diff --git a/tools/testing/selftests/bpf/bench.c b/tools/testing/selftests/bpf/bench.c index 7ca1e1eb5c30..484bcbeaa819 100644 --- a/tools/testing/selftests/bpf/bench.c +++ b/tools/testing/selftests/bpf/bench.c @@ -280,6 +280,7 @@ extern struct argp bench_strncmp_argp; extern struct argp bench_hashmap_lookup_argp; extern struct argp bench_local_storage_create_argp; extern struct argp bench_htab_mem_argp; +extern struct argp bench_trigger_batch_argp; static const struct argp_child bench_parsers[] = { { &bench_ringbufs_argp, 0, "Ring buffers benchmark", 0 }, @@ -292,6 +293,7 @@ static const struct argp_child bench_parsers[] = { { &bench_hashmap_lookup_argp, 0, "Hashmap lookup benchmark", 0 }, { &bench_local_storage_create_argp, 0, "local-storage-create benchmark", 0 }, { &bench_htab_mem_argp, 0, "hash map memory benchmark", 0 }, + { &bench_trigger_batch_argp, 0, "BPF triggering benchmark", 0 }, {}, }; @@ -508,6 +510,15 @@ extern const struct bench bench_trig_fexit; extern const struct bench bench_trig_fentry_sleep; extern const struct bench bench_trig_fmodret; +/* batched, staying mostly in-kernel benchmarks */ +extern const struct bench bench_trig_kernel_count; +extern const struct bench bench_trig_kprobe_batch; +extern const struct bench bench_trig_kretprobe_batch; +extern const struct bench bench_trig_kprobe_multi_batch; +extern const struct bench bench_trig_kretprobe_multi_batch; +extern const struct bench bench_trig_fentry_batch; +extern const struct bench bench_trig_fexit_batch; + /* uprobe/uretprobe benchmarks */ extern const struct bench bench_trig_uprobe_nop; extern const struct bench bench_trig_uretprobe_nop; @@ -548,7 +559,7 @@ static const struct bench *benchs[] = { &bench_rename_fexit, /* pure counting benchmarks for establishing theoretical limits */ &bench_trig_usermode_count, - &bench_trig_base, + &bench_trig_kernel_count, /* syscall-driven triggering benchmarks */ &bench_trig_tp, &bench_trig_rawtp, @@ -560,6 +571,13 @@ static const struct bench *benchs[] = { &bench_trig_fexit, &bench_trig_fentry_sleep, &bench_trig_fmodret, + /* batched, staying mostly in-kernel triggers */ + &bench_trig_kprobe_batch, + &bench_trig_kretprobe_batch, + &bench_trig_kprobe_multi_batch, + &bench_trig_kretprobe_multi_batch, + &bench_trig_fentry_batch, + &bench_trig_fexit_batch, /* uprobes */ &bench_trig_uprobe_nop, &bench_trig_uretprobe_nop, @@ -567,6 +585,7 @@ static const struct bench *benchs[] = { &bench_trig_uretprobe_push, &bench_trig_uprobe_ret, &bench_trig_uretprobe_ret, + /* ringbuf/perfbuf benchmarks */ &bench_rb_libbpf, &bench_rb_custom, &bench_pb_libbpf, diff --git a/tools/testing/selftests/bpf/benchs/bench_trigger.c b/tools/testing/selftests/bpf/benchs/bench_trigger.c index 68e703b46d9b..21fa1a84ecd2 100644 --- a/tools/testing/selftests/bpf/benchs/bench_trigger.c +++ b/tools/testing/selftests/bpf/benchs/bench_trigger.c @@ -1,11 +1,54 @@ // SPDX-License-Identifier: GPL-2.0 /* Copyright (c) 2020 Facebook */ #define _GNU_SOURCE +#include #include +#include #include "bench.h" #include "trigger_bench.skel.h" #include "trace_helpers.h" +static struct { + __u32 batch_iters; +} args = { + .batch_iters = 100, +}; + +enum { + ARG_TRIG_BATCH_ITERS = 7000, +}; + +static const struct argp_option opts[] = { + { "trig-batch-iters", ARG_TRIG_BATCH_ITERS, "BATCH_ITER_CNT", 0, + "Number of in-kernel iterations per one driver test run"}, + {}, +}; + +static error_t parse_arg(int key, char *arg, struct argp_state *state) +{ + long ret; + + switch (key) { + case ARG_TRIG_BATCH_ITERS: + ret = strtol(arg, NULL, 10); + if (ret < 1 || ret > UINT_MAX) { + fprintf(stderr, "invalid --trig-batch-iters value"); + argp_usage(state); + } + args.batch_iters = ret; + break; + default: + return ARGP_ERR_UNKNOWN; + } + + return 0; +} + +const struct argp bench_trigger_batch_argp = { + .options = opts, + .parser = parse_arg, +}; + /* adjust slot shift in inc_hits() if changing */ #define MAX_BUCKETS 256 @@ -13,6 +56,7 @@ static struct trigger_ctx { struct trigger_bench *skel; bool usermode_counters; + int driver_prog_fd; } ctx; static struct counter base_hits[MAX_BUCKETS]; @@ -71,6 +115,16 @@ static void *trigger_producer(void *input) return NULL; } +static void *trigger_producer_batch(void *input) +{ + int fd = ctx.driver_prog_fd ?: bpf_program__fd(ctx.skel->progs.trigger_driver); + + while (true) + bpf_prog_test_run_opts(fd, NULL); + + return NULL; +} + static void trigger_measure(struct bench_res *res) { if (ctx.usermode_counters) @@ -81,13 +135,23 @@ static void trigger_measure(struct bench_res *res) static void setup_ctx(void) { + int err; + setup_libbpf(); - ctx.skel = trigger_bench__open_and_load(); + ctx.skel = trigger_bench__open(); if (!ctx.skel) { fprintf(stderr, "failed to open skeleton\n"); exit(1); } + + ctx.skel->rodata->batch_iters = args.batch_iters; + + err = trigger_bench__load(ctx.skel); + if (err) { + fprintf(stderr, "failed to open skeleton\n"); + exit(1); + } } static void attach_bpf(struct bpf_program *prog) @@ -161,6 +225,50 @@ static void trigger_fmodret_setup(void) attach_bpf(ctx.skel->progs.bench_trigger_fmodret); } +/* Batched, staying mostly in-kernel triggering setups */ +static void trigger_kernel_count_setup(void) +{ + setup_ctx(); + /* override driver program */ + ctx.driver_prog_fd = bpf_program__fd(ctx.skel->progs.trigger_count); +} + +static void trigger_kprobe_batch_setup(void) +{ + setup_ctx(); + attach_bpf(ctx.skel->progs.bench_trigger_kprobe_batch); +} + +static void trigger_kretprobe_batch_setup(void) +{ + setup_ctx(); + attach_bpf(ctx.skel->progs.bench_trigger_kretprobe_batch); +} + +static void trigger_kprobe_multi_batch_setup(void) +{ + setup_ctx(); + attach_bpf(ctx.skel->progs.bench_trigger_kprobe_multi_batch); +} + +static void trigger_kretprobe_multi_batch_setup(void) +{ + setup_ctx(); + attach_bpf(ctx.skel->progs.bench_trigger_kretprobe_multi_batch); +} + +static void trigger_fentry_batch_setup(void) +{ + setup_ctx(); + attach_bpf(ctx.skel->progs.bench_trigger_fentry_batch); +} + +static void trigger_fexit_batch_setup(void) +{ + setup_ctx(); + attach_bpf(ctx.skel->progs.bench_trigger_fexit_batch); +} + /* make sure call is not inlined and not avoided by compiler, so __weak and * inline asm volatile in the body of the function * @@ -394,6 +502,26 @@ const struct bench bench_trig_fmodret = { .report_final = hits_drops_report_final, }; +/* batched (staying mostly in kernel) kprobe/fentry benchmarks */ +#define BENCH_TRIG_BATCH(KIND, NAME) \ +const struct bench bench_trig_##KIND = { \ + .name = "trig-" NAME, \ + .setup = trigger_##KIND##_setup, \ + .producer_thread = trigger_producer_batch, \ + .measure = trigger_measure, \ + .report_progress = hits_drops_report_progress, \ + .report_final = hits_drops_report_final, \ + .argp = &bench_trigger_batch_argp, \ +} + +BENCH_TRIG_BATCH(kernel_count, "kernel-count"); +BENCH_TRIG_BATCH(kprobe_batch, "kprobe-batch"); +BENCH_TRIG_BATCH(kretprobe_batch, "kretprobe-batch"); +BENCH_TRIG_BATCH(kprobe_multi_batch, "kprobe-multi-batch"); +BENCH_TRIG_BATCH(kretprobe_multi_batch, "kretprobe-multi-batch"); +BENCH_TRIG_BATCH(fentry_batch, "fentry-batch"); +BENCH_TRIG_BATCH(fexit_batch, "fexit-batch"); + /* uprobe benchmarks */ #define BENCH_TRIG_USERMODE(KIND, PRODUCER, NAME) \ const struct bench bench_trig_##KIND = { \ diff --git a/tools/testing/selftests/bpf/benchs/run_bench_trigger.sh b/tools/testing/selftests/bpf/benchs/run_bench_trigger.sh index 78e83f243294..b58ec33ea18c 100755 --- a/tools/testing/selftests/bpf/benchs/run_bench_trigger.sh +++ b/tools/testing/selftests/bpf/benchs/run_bench_trigger.sh @@ -2,8 +2,24 @@ set -eufo pipefail -for i in base tp rawtp kprobe fentry fmodret -do - summary=$(sudo ./bench -w2 -d5 -a trig-$i | tail -n1 | cut -d'(' -f1 | cut -d' ' -f3-) - printf "%-10s: %s\n" $i "$summary" +def_tests=( \ + usermode-count kernel-count syscall-count \ + fentry-batch fexit-batch \ + kprobe-batch kprobe-multi-batch \ + kretprobe-batch kretprobe-multi-batch \ + fentry fexit fmodret \ + rawtp tp \ + kprobe kprobe-multi kretprobe kretprobe-multi \ +) + +tests=("$@") +if [ ${#tests[@]} -eq 0 ]; then + tests=("${def_tests[@]}") +fi + +p=${PROD_CNT:-1} + +for t in "${tests[@]}"; do + summary=$(sudo ./bench -w2 -d5 -a -p$p trig-$t | tail -n1 | cut -d'(' -f1 | cut -d' ' -f3-) + printf "%-21s: %s\n" $t "$summary" done diff --git a/tools/testing/selftests/bpf/progs/trigger_bench.c b/tools/testing/selftests/bpf/progs/trigger_bench.c index 42ec202015ed..f0b76afa5017 100644 --- a/tools/testing/selftests/bpf/progs/trigger_bench.c +++ b/tools/testing/selftests/bpf/progs/trigger_bench.c @@ -1,6 +1,5 @@ // SPDX-License-Identifier: GPL-2.0 // Copyright (c) 2020 Facebook - #include #include #include @@ -103,3 +102,69 @@ int bench_trigger_uprobe(void *ctx) inc_counter(); return 0; } + +const volatile int batch_iters = 0; + +SEC("raw_tp") +int trigger_count(void *ctx) +{ + int i; + + for (i = 0; i < batch_iters; i++) + inc_counter(); + + return 0; +} + +SEC("raw_tp") +int trigger_driver(void *ctx) +{ + int i; + + for (i = 0; i < batch_iters; i++) + (void)bpf_get_numa_node_id(); /* attach point for benchmarking */ + + return 0; +} + +SEC("kprobe/bpf_get_numa_node_id") +int bench_trigger_kprobe_batch(void *ctx) +{ + inc_counter(); + return 0; +} + +SEC("kretprobe/bpf_get_numa_node_id") +int bench_trigger_kretprobe_batch(void *ctx) +{ + inc_counter(); + return 0; +} + +SEC("kprobe.multi/bpf_get_numa_node_id") +int bench_trigger_kprobe_multi_batch(void *ctx) +{ + inc_counter(); + return 0; +} + +SEC("kretprobe.multi/bpf_get_numa_node_id") +int bench_trigger_kretprobe_multi_batch(void *ctx) +{ + inc_counter(); + return 0; +} + +SEC("fentry/bpf_get_numa_node_id") +int bench_trigger_fentry_batch(void *ctx) +{ + inc_counter(); + return 0; +} + +SEC("fexit/bpf_get_numa_node_id") +int bench_trigger_fexit_batch(void *ctx) +{ + inc_counter(); + return 0; +} From patchwork Fri Mar 22 00:00:38 2024 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Andrii Nakryiko X-Patchwork-Id: 13599473 X-Patchwork-Delegate: bpf@iogearbox.net Received: from smtp.kernel.org (aws-us-west-2-korg-mail-1.web.codeaurora.org [10.30.226.201]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (No client certificate requested) by smtp.subspace.kernel.org (Postfix) with ESMTPS id 5ECCD5CB8 for ; Fri, 22 Mar 2024 00:01:00 +0000 (UTC) Authentication-Results: smtp.subspace.kernel.org; arc=none smtp.client-ip=10.30.226.201 ARC-Seal: i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1711065660; cv=none; b=eRgM2ujOBLZd1ODgEYP1DfjFtXlrFrmSe5h3mHfyhvRDHVyU3mbUzT7/2KJP0dfDI0JCzLFinXfT8eZ/oMnvGcPJu+L6GTxZBxRjQ9VWDZreM3o3QQ/9OiNuOzriI/SWJuZpxooU8F/OXVjmlDbY9k6FCShQXA8M0uEKUt3tXKY= ARC-Message-Signature: i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1711065660; c=relaxed/simple; bh=jn7vAcmlMFhaNxKL1nG0LFqBI0+15mT10O5VSx+AQuM=; h=From:To:Cc:Subject:Date:Message-ID:In-Reply-To:References: MIME-Version; b=aBX2sVw9n/SCpG2th0biZ1Ac/lFAxAIrKKocoQH+XFYgFUXDUYyNaEXtcRMKgnboMIo0h/wVvPnTSossutTxabRrLVzkznM/ij8hSOfZVQYe6AbuS/Fg/+YnzcpeLi/jPx8LAM1I7TZ+AuFUxER8V8lwOHq04lZ3/s2ldyGRWpA= ARC-Authentication-Results: i=1; smtp.subspace.kernel.org; dkim=pass (2048-bit key) header.d=kernel.org header.i=@kernel.org header.b=mB+/H5Ra; arc=none smtp.client-ip=10.30.226.201 Authentication-Results: smtp.subspace.kernel.org; dkim=pass (2048-bit key) header.d=kernel.org header.i=@kernel.org header.b="mB+/H5Ra" Received: by smtp.kernel.org (Postfix) with ESMTPSA id A26C5C433F1; Fri, 22 Mar 2024 00:00:59 +0000 (UTC) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/simple; d=kernel.org; s=k20201202; t=1711065659; bh=jn7vAcmlMFhaNxKL1nG0LFqBI0+15mT10O5VSx+AQuM=; h=From:To:Cc:Subject:Date:In-Reply-To:References:From; b=mB+/H5RacoNpiwfOm1ultTQQiXkVN4iUyxBuYs373X1OnxjYtbCSGaRZEoz/06UtZ Rl8LaadIBsV8GXdA/YGoOx16UWHG09IvImOIuRsZJglH1wu0Gc6Zcac0yqXHHtRMwy ipkBEBie72lxepmDPLHf83Apou0tLQGge4zaUcMQ7cREGWT2KSPJQRBGCJCOLcgcEF KP774v9Z4SMgwxgmQRU9abEQNbO6sZlS8g+YeAngT8tJGJhvrtYeHbTIigzBs+wCle hTMBIW/HQ8EGELRwDZ+ynhFiESPNaYLaaXl6N8Rp49qrrzdVX5LoRuWaI0WcY8QTGp x9esiMcYKoVmw== From: Andrii Nakryiko To: bpf@vger.kernel.org, ast@kernel.org, daniel@iogearbox.net, martin.lau@kernel.org Cc: andrii@kernel.org, kernel-team@meta.com Subject: [PATCH bpf-next 3/6] selftests/bpf: remove syscall-driven benchs, keep syscall-count only Date: Thu, 21 Mar 2024 17:00:38 -0700 Message-ID: <20240322000041.2919948-4-andrii@kernel.org> X-Mailer: git-send-email 2.43.0 In-Reply-To: <20240322000041.2919948-1-andrii@kernel.org> References: <20240322000041.2919948-1-andrii@kernel.org> Precedence: bulk X-Mailing-List: bpf@vger.kernel.org List-Id: List-Subscribe: List-Unsubscribe: MIME-Version: 1.0 X-Patchwork-Delegate: bpf@iogearbox.net Remove "legacy" benchmarks triggered by syscalls in favor of newly added in-kernel/batched benchmarks. Drop -batched suffix now as well. Next patch will restore "feature parity" by adding back tp/raw_tp/fmodret benchmarks based on in-kernel kfunc approach. Signed-off-by: Andrii Nakryiko --- tools/testing/selftests/bpf/bench.c | 32 +-- .../selftests/bpf/benchs/bench_trigger.c | 213 +++--------------- .../selftests/bpf/benchs/run_bench_trigger.sh | 11 +- .../selftests/bpf/benchs/run_bench_uprobes.sh | 2 +- .../selftests/bpf/progs/trigger_bench.c | 83 +------ 5 files changed, 42 insertions(+), 299 deletions(-) diff --git a/tools/testing/selftests/bpf/bench.c b/tools/testing/selftests/bpf/bench.c index 484bcbeaa819..8b16841a3dec 100644 --- a/tools/testing/selftests/bpf/bench.c +++ b/tools/testing/selftests/bpf/bench.c @@ -496,28 +496,16 @@ extern const struct bench bench_rename_fexit; /* pure counting benchmarks to establish theoretical lmits */ extern const struct bench bench_trig_usermode_count; -extern const struct bench bench_trig_base; +extern const struct bench bench_trig_syscall_count; +extern const struct bench bench_trig_kernel_count; -/* kernel-side syscall-triggered benchmarks */ -extern const struct bench bench_trig_tp; -extern const struct bench bench_trig_rawtp; +/* batched, staying mostly in-kernel benchmarks */ extern const struct bench bench_trig_kprobe; extern const struct bench bench_trig_kretprobe; extern const struct bench bench_trig_kprobe_multi; extern const struct bench bench_trig_kretprobe_multi; extern const struct bench bench_trig_fentry; extern const struct bench bench_trig_fexit; -extern const struct bench bench_trig_fentry_sleep; -extern const struct bench bench_trig_fmodret; - -/* batched, staying mostly in-kernel benchmarks */ -extern const struct bench bench_trig_kernel_count; -extern const struct bench bench_trig_kprobe_batch; -extern const struct bench bench_trig_kretprobe_batch; -extern const struct bench bench_trig_kprobe_multi_batch; -extern const struct bench bench_trig_kretprobe_multi_batch; -extern const struct bench bench_trig_fentry_batch; -extern const struct bench bench_trig_fexit_batch; /* uprobe/uretprobe benchmarks */ extern const struct bench bench_trig_uprobe_nop; @@ -560,24 +548,14 @@ static const struct bench *benchs[] = { /* pure counting benchmarks for establishing theoretical limits */ &bench_trig_usermode_count, &bench_trig_kernel_count, - /* syscall-driven triggering benchmarks */ - &bench_trig_tp, - &bench_trig_rawtp, + &bench_trig_syscall_count, + /* batched, staying mostly in-kernel triggers */ &bench_trig_kprobe, &bench_trig_kretprobe, &bench_trig_kprobe_multi, &bench_trig_kretprobe_multi, &bench_trig_fentry, &bench_trig_fexit, - &bench_trig_fentry_sleep, - &bench_trig_fmodret, - /* batched, staying mostly in-kernel triggers */ - &bench_trig_kprobe_batch, - &bench_trig_kretprobe_batch, - &bench_trig_kprobe_multi_batch, - &bench_trig_kretprobe_multi_batch, - &bench_trig_fentry_batch, - &bench_trig_fexit_batch, /* uprobes */ &bench_trig_uprobe_nop, &bench_trig_uretprobe_nop, diff --git a/tools/testing/selftests/bpf/benchs/bench_trigger.c b/tools/testing/selftests/bpf/benchs/bench_trigger.c index 21fa1a84ecd2..567aad2d6f09 100644 --- a/tools/testing/selftests/bpf/benchs/bench_trigger.c +++ b/tools/testing/selftests/bpf/benchs/bench_trigger.c @@ -94,24 +94,17 @@ static void trigger_validate(void) } } -static void *trigger_base_producer(void *input) -{ - while (true) { - (void)syscall(__NR_getpgid); - inc_counter(base_hits); - } - return NULL; -} - -static void trigger_base_measure(struct bench_res *res) -{ - res->hits = sum_and_reset_counters(base_hits); -} - static void *trigger_producer(void *input) { - while (true) - (void)syscall(__NR_getpgid); + if (ctx.usermode_counters) { + while (true) { + (void)syscall(__NR_getpgid); + inc_counter(base_hits); + } + } else { + while (true) + (void)syscall(__NR_getpgid); + } return NULL; } @@ -165,16 +158,17 @@ static void attach_bpf(struct bpf_program *prog) } } -static void trigger_tp_setup(void) +static void trigger_syscall_count_setup(void) { - setup_ctx(); - attach_bpf(ctx.skel->progs.bench_trigger_tp); + ctx.usermode_counters = true; } -static void trigger_rawtp_setup(void) +/* Batched, staying mostly in-kernel triggering setups */ +static void trigger_kernel_count_setup(void) { setup_ctx(); - attach_bpf(ctx.skel->progs.bench_trigger_raw_tp); + /* override driver program */ + ctx.driver_prog_fd = bpf_program__fd(ctx.skel->progs.trigger_count); } static void trigger_kprobe_setup(void) @@ -213,62 +207,6 @@ static void trigger_fexit_setup(void) attach_bpf(ctx.skel->progs.bench_trigger_fexit); } -static void trigger_fentry_sleep_setup(void) -{ - setup_ctx(); - attach_bpf(ctx.skel->progs.bench_trigger_fentry_sleep); -} - -static void trigger_fmodret_setup(void) -{ - setup_ctx(); - attach_bpf(ctx.skel->progs.bench_trigger_fmodret); -} - -/* Batched, staying mostly in-kernel triggering setups */ -static void trigger_kernel_count_setup(void) -{ - setup_ctx(); - /* override driver program */ - ctx.driver_prog_fd = bpf_program__fd(ctx.skel->progs.trigger_count); -} - -static void trigger_kprobe_batch_setup(void) -{ - setup_ctx(); - attach_bpf(ctx.skel->progs.bench_trigger_kprobe_batch); -} - -static void trigger_kretprobe_batch_setup(void) -{ - setup_ctx(); - attach_bpf(ctx.skel->progs.bench_trigger_kretprobe_batch); -} - -static void trigger_kprobe_multi_batch_setup(void) -{ - setup_ctx(); - attach_bpf(ctx.skel->progs.bench_trigger_kprobe_multi_batch); -} - -static void trigger_kretprobe_multi_batch_setup(void) -{ - setup_ctx(); - attach_bpf(ctx.skel->progs.bench_trigger_kretprobe_multi_batch); -} - -static void trigger_fentry_batch_setup(void) -{ - setup_ctx(); - attach_bpf(ctx.skel->progs.bench_trigger_fentry_batch); -} - -static void trigger_fexit_batch_setup(void) -{ - setup_ctx(); - attach_bpf(ctx.skel->progs.bench_trigger_fexit_batch); -} - /* make sure call is not inlined and not avoided by compiler, so __weak and * inline asm volatile in the body of the function * @@ -393,109 +331,10 @@ static void uretprobe_ret_setup(void) usetup(true, &uprobe_target_ret); } -const struct bench bench_trig_base = { - .name = "trig-base", - .validate = trigger_validate, - .producer_thread = trigger_base_producer, - .measure = trigger_base_measure, - .report_progress = hits_drops_report_progress, - .report_final = hits_drops_report_final, -}; - -const struct bench bench_trig_tp = { - .name = "trig-tp", - .validate = trigger_validate, - .setup = trigger_tp_setup, - .producer_thread = trigger_producer, - .measure = trigger_measure, - .report_progress = hits_drops_report_progress, - .report_final = hits_drops_report_final, -}; - -const struct bench bench_trig_rawtp = { - .name = "trig-rawtp", - .validate = trigger_validate, - .setup = trigger_rawtp_setup, - .producer_thread = trigger_producer, - .measure = trigger_measure, - .report_progress = hits_drops_report_progress, - .report_final = hits_drops_report_final, -}; - -const struct bench bench_trig_kprobe = { - .name = "trig-kprobe", - .validate = trigger_validate, - .setup = trigger_kprobe_setup, - .producer_thread = trigger_producer, - .measure = trigger_measure, - .report_progress = hits_drops_report_progress, - .report_final = hits_drops_report_final, -}; - -const struct bench bench_trig_kretprobe = { - .name = "trig-kretprobe", - .validate = trigger_validate, - .setup = trigger_kretprobe_setup, - .producer_thread = trigger_producer, - .measure = trigger_measure, - .report_progress = hits_drops_report_progress, - .report_final = hits_drops_report_final, -}; - -const struct bench bench_trig_kprobe_multi = { - .name = "trig-kprobe-multi", - .validate = trigger_validate, - .setup = trigger_kprobe_multi_setup, - .producer_thread = trigger_producer, - .measure = trigger_measure, - .report_progress = hits_drops_report_progress, - .report_final = hits_drops_report_final, -}; - -const struct bench bench_trig_kretprobe_multi = { - .name = "trig-kretprobe-multi", - .validate = trigger_validate, - .setup = trigger_kretprobe_multi_setup, - .producer_thread = trigger_producer, - .measure = trigger_measure, - .report_progress = hits_drops_report_progress, - .report_final = hits_drops_report_final, -}; - -const struct bench bench_trig_fentry = { - .name = "trig-fentry", - .validate = trigger_validate, - .setup = trigger_fentry_setup, - .producer_thread = trigger_producer, - .measure = trigger_measure, - .report_progress = hits_drops_report_progress, - .report_final = hits_drops_report_final, -}; - -const struct bench bench_trig_fexit = { - .name = "trig-fexit", - .validate = trigger_validate, - .setup = trigger_fexit_setup, - .producer_thread = trigger_producer, - .measure = trigger_measure, - .report_progress = hits_drops_report_progress, - .report_final = hits_drops_report_final, -}; - -const struct bench bench_trig_fentry_sleep = { - .name = "trig-fentry-sleep", - .validate = trigger_validate, - .setup = trigger_fentry_sleep_setup, - .producer_thread = trigger_producer, - .measure = trigger_measure, - .report_progress = hits_drops_report_progress, - .report_final = hits_drops_report_final, -}; - -const struct bench bench_trig_fmodret = { - .name = "trig-fmodret", +const struct bench bench_trig_syscall_count = { + .name = "trig-syscall-count", .validate = trigger_validate, - .setup = trigger_fmodret_setup, + .setup = trigger_syscall_count_setup, .producer_thread = trigger_producer, .measure = trigger_measure, .report_progress = hits_drops_report_progress, @@ -503,7 +342,7 @@ const struct bench bench_trig_fmodret = { }; /* batched (staying mostly in kernel) kprobe/fentry benchmarks */ -#define BENCH_TRIG_BATCH(KIND, NAME) \ +#define BENCH_TRIG_KERNEL(KIND, NAME) \ const struct bench bench_trig_##KIND = { \ .name = "trig-" NAME, \ .setup = trigger_##KIND##_setup, \ @@ -514,13 +353,13 @@ const struct bench bench_trig_##KIND = { \ .argp = &bench_trigger_batch_argp, \ } -BENCH_TRIG_BATCH(kernel_count, "kernel-count"); -BENCH_TRIG_BATCH(kprobe_batch, "kprobe-batch"); -BENCH_TRIG_BATCH(kretprobe_batch, "kretprobe-batch"); -BENCH_TRIG_BATCH(kprobe_multi_batch, "kprobe-multi-batch"); -BENCH_TRIG_BATCH(kretprobe_multi_batch, "kretprobe-multi-batch"); -BENCH_TRIG_BATCH(fentry_batch, "fentry-batch"); -BENCH_TRIG_BATCH(fexit_batch, "fexit-batch"); +BENCH_TRIG_KERNEL(kernel_count, "kernel-count"); +BENCH_TRIG_KERNEL(kprobe, "kprobe"); +BENCH_TRIG_KERNEL(kretprobe, "kretprobe"); +BENCH_TRIG_KERNEL(kprobe_multi, "kprobe-multi"); +BENCH_TRIG_KERNEL(kretprobe_multi, "kretprobe-multi"); +BENCH_TRIG_KERNEL(fentry, "fentry"); +BENCH_TRIG_KERNEL(fexit, "fexit"); /* uprobe benchmarks */ #define BENCH_TRIG_USERMODE(KIND, PRODUCER, NAME) \ diff --git a/tools/testing/selftests/bpf/benchs/run_bench_trigger.sh b/tools/testing/selftests/bpf/benchs/run_bench_trigger.sh index b58ec33ea18c..6e790e294260 100755 --- a/tools/testing/selftests/bpf/benchs/run_bench_trigger.sh +++ b/tools/testing/selftests/bpf/benchs/run_bench_trigger.sh @@ -4,12 +4,9 @@ set -eufo pipefail def_tests=( \ usermode-count kernel-count syscall-count \ - fentry-batch fexit-batch \ - kprobe-batch kprobe-multi-batch \ - kretprobe-batch kretprobe-multi-batch \ - fentry fexit fmodret \ - rawtp tp \ - kprobe kprobe-multi kretprobe kretprobe-multi \ + fentry fexit \ + kprobe kprobe-multi \ + kretprobe kretprobe-multi \ ) tests=("$@") @@ -21,5 +18,5 @@ p=${PROD_CNT:-1} for t in "${tests[@]}"; do summary=$(sudo ./bench -w2 -d5 -a -p$p trig-$t | tail -n1 | cut -d'(' -f1 | cut -d' ' -f3-) - printf "%-21s: %s\n" $t "$summary" + printf "%-15s: %s\n" $t "$summary" done diff --git a/tools/testing/selftests/bpf/benchs/run_bench_uprobes.sh b/tools/testing/selftests/bpf/benchs/run_bench_uprobes.sh index 36021d243397..af169f831f2f 100755 --- a/tools/testing/selftests/bpf/benchs/run_bench_uprobes.sh +++ b/tools/testing/selftests/bpf/benchs/run_bench_uprobes.sh @@ -2,7 +2,7 @@ set -eufo pipefail -for i in usermode-count base {uprobe,uretprobe}-{nop,push,ret} +for i in usermode-count syscall-count {uprobe,uretprobe}-{nop,push,ret} do summary=$(sudo ./bench -w2 -d5 -a trig-$i | tail -n1 | cut -d'(' -f1 | cut -d' ' -f3-) printf "%-15s: %s\n" $i "$summary" diff --git a/tools/testing/selftests/bpf/progs/trigger_bench.c b/tools/testing/selftests/bpf/progs/trigger_bench.c index f0b76afa5017..81990e45b547 100644 --- a/tools/testing/selftests/bpf/progs/trigger_bench.c +++ b/tools/testing/selftests/bpf/progs/trigger_bench.c @@ -25,77 +25,6 @@ static __always_inline void inc_counter(void) __sync_add_and_fetch(&hits[cpu & CPU_MASK].value, 1); } -SEC("tp/syscalls/sys_enter_getpgid") -int bench_trigger_tp(void *ctx) -{ - inc_counter(); - return 0; -} - -SEC("raw_tp/sys_enter") -int BPF_PROG(bench_trigger_raw_tp, struct pt_regs *regs, long id) -{ - if (id == __NR_getpgid) - inc_counter(); - return 0; -} - -SEC("kprobe/" SYS_PREFIX "sys_getpgid") -int bench_trigger_kprobe(void *ctx) -{ - inc_counter(); - return 0; -} - -SEC("kretprobe/" SYS_PREFIX "sys_getpgid") -int bench_trigger_kretprobe(void *ctx) -{ - inc_counter(); - return 0; -} - -SEC("kprobe.multi/" SYS_PREFIX "sys_getpgid") -int bench_trigger_kprobe_multi(void *ctx) -{ - inc_counter(); - return 0; -} - -SEC("kretprobe.multi/" SYS_PREFIX "sys_getpgid") -int bench_trigger_kretprobe_multi(void *ctx) -{ - inc_counter(); - return 0; -} - -SEC("fentry/" SYS_PREFIX "sys_getpgid") -int bench_trigger_fentry(void *ctx) -{ - inc_counter(); - return 0; -} - -SEC("fexit/" SYS_PREFIX "sys_getpgid") -int bench_trigger_fexit(void *ctx) -{ - inc_counter(); - return 0; -} - -SEC("fentry.s/" SYS_PREFIX "sys_getpgid") -int bench_trigger_fentry_sleep(void *ctx) -{ - inc_counter(); - return 0; -} - -SEC("fmod_ret/" SYS_PREFIX "sys_getpgid") -int bench_trigger_fmodret(void *ctx) -{ - inc_counter(); - return -22; -} - SEC("uprobe") int bench_trigger_uprobe(void *ctx) { @@ -128,42 +57,42 @@ int trigger_driver(void *ctx) } SEC("kprobe/bpf_get_numa_node_id") -int bench_trigger_kprobe_batch(void *ctx) +int bench_trigger_kprobe(void *ctx) { inc_counter(); return 0; } SEC("kretprobe/bpf_get_numa_node_id") -int bench_trigger_kretprobe_batch(void *ctx) +int bench_trigger_kretprobe(void *ctx) { inc_counter(); return 0; } SEC("kprobe.multi/bpf_get_numa_node_id") -int bench_trigger_kprobe_multi_batch(void *ctx) +int bench_trigger_kprobe_multi(void *ctx) { inc_counter(); return 0; } SEC("kretprobe.multi/bpf_get_numa_node_id") -int bench_trigger_kretprobe_multi_batch(void *ctx) +int bench_trigger_kretprobe_multi(void *ctx) { inc_counter(); return 0; } SEC("fentry/bpf_get_numa_node_id") -int bench_trigger_fentry_batch(void *ctx) +int bench_trigger_fentry(void *ctx) { inc_counter(); return 0; } SEC("fexit/bpf_get_numa_node_id") -int bench_trigger_fexit_batch(void *ctx) +int bench_trigger_fexit(void *ctx) { inc_counter(); return 0; From patchwork Fri Mar 22 00:00:39 2024 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Andrii Nakryiko X-Patchwork-Id: 13599474 X-Patchwork-Delegate: bpf@iogearbox.net Received: from smtp.kernel.org (aws-us-west-2-korg-mail-1.web.codeaurora.org [10.30.226.201]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (No client certificate requested) by smtp.subspace.kernel.org (Postfix) with ESMTPS id 328162F30 for ; Fri, 22 Mar 2024 00:01:03 +0000 (UTC) Authentication-Results: smtp.subspace.kernel.org; arc=none smtp.client-ip=10.30.226.201 ARC-Seal: i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1711065663; cv=none; b=jUlAb0+df0ykViqqK1RMgPlMr/VV/R9U6CqWrumuVkt1lXmgShKXoXBbOJPdCxNtzIGkETCfjvinmRaVP+hZYhd41Jl/ZJDwtmv7XysUkB4UzUqzyrC9ekOLbeixqoCwdMjQ32tq4DKF6b8DcySE7gfWY/dFBcoSa1raDpov7HY= ARC-Message-Signature: i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1711065663; c=relaxed/simple; bh=FfoAZ0Wl0rvhW1Qg/x8M4XrMSTRiLcIbAoP//uN5TUE=; h=From:To:Cc:Subject:Date:Message-ID:In-Reply-To:References: MIME-Version; b=N+VI2w0V4S9mxt0Ct79WN83CxPIQbfb/XfeB3ESOYORTPSE9sinbAvi7EmZWrGOAncwxjQTQFV5RD8/XqBR/D8AtY5MzmgzqC7QVOiiaQ0lQgDTwZkrG1jRuhvOZDAn4+R94zEavHtpnwF/2KShhkjV2yidTsmHyG6D7S7mTfwo= ARC-Authentication-Results: i=1; smtp.subspace.kernel.org; dkim=pass (2048-bit key) header.d=kernel.org header.i=@kernel.org header.b=LBjB+l/f; arc=none smtp.client-ip=10.30.226.201 Authentication-Results: smtp.subspace.kernel.org; dkim=pass (2048-bit key) header.d=kernel.org header.i=@kernel.org header.b="LBjB+l/f" Received: by smtp.kernel.org (Postfix) with ESMTPSA id DD3D4C433F1; Fri, 22 Mar 2024 00:01:02 +0000 (UTC) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/simple; d=kernel.org; s=k20201202; t=1711065663; bh=FfoAZ0Wl0rvhW1Qg/x8M4XrMSTRiLcIbAoP//uN5TUE=; h=From:To:Cc:Subject:Date:In-Reply-To:References:From; b=LBjB+l/fM36z5pD2zuIpVqCArLKY9FxaQIQaz9o7LdnhJIfei5O2E9RfTNG2XbGOU s/hwK5rf8CxRs1ZawM2grE9NT7CnnxNw7NKHs3TK6JJuotYajyF3aaB3S8jRWiSoJe eAbmeURpcGYRKJrcevrNvitLA5Dv89P6amjPNmnTtn2iwBp94RLgg2uRO+68mWs9UO dRF8Yt2DnozClYRetSvGb9GoINMLs0c706KcCLm9FOClmmKlUqK7Cy9lCMKq5di7O3 7UW698KnMcApynxmwPkH7CpVfxmoSHUqMO6CAGLbVc/IZu8fWa11SUllfOToe7/HGa MgX8FjLzPCbig== From: Andrii Nakryiko To: bpf@vger.kernel.org, ast@kernel.org, daniel@iogearbox.net, martin.lau@kernel.org Cc: andrii@kernel.org, kernel-team@meta.com Subject: [PATCH bpf-next 4/6] selftests/bpf: lazy-load trigger bench BPF programs Date: Thu, 21 Mar 2024 17:00:39 -0700 Message-ID: <20240322000041.2919948-5-andrii@kernel.org> X-Mailer: git-send-email 2.43.0 In-Reply-To: <20240322000041.2919948-1-andrii@kernel.org> References: <20240322000041.2919948-1-andrii@kernel.org> Precedence: bulk X-Mailing-List: bpf@vger.kernel.org List-Id: List-Subscribe: List-Unsubscribe: MIME-Version: 1.0 X-Patchwork-Delegate: bpf@iogearbox.net Instead of front-loading all possible benchmarking BPF programs for trigger benchmarks, explicitly specify which BPF programs are used by specific benchmark and load only it. This allows to be more flexible in supporting older kernels, where some program types might not be possible to load (e.g., those that rely on newly added kfunc). Signed-off-by: Andrii Nakryiko --- .../selftests/bpf/benchs/bench_trigger.c | 36 +++++++++++++++++-- .../selftests/bpf/progs/trigger_bench.c | 18 +++++----- 2 files changed, 42 insertions(+), 12 deletions(-) diff --git a/tools/testing/selftests/bpf/benchs/bench_trigger.c b/tools/testing/selftests/bpf/benchs/bench_trigger.c index 567aad2d6f09..e454d7de2252 100644 --- a/tools/testing/selftests/bpf/benchs/bench_trigger.c +++ b/tools/testing/selftests/bpf/benchs/bench_trigger.c @@ -128,8 +128,6 @@ static void trigger_measure(struct bench_res *res) static void setup_ctx(void) { - int err; - setup_libbpf(); ctx.skel = trigger_bench__open(); @@ -138,7 +136,15 @@ static void setup_ctx(void) exit(1); } + /* default "driver" BPF program */ + bpf_program__set_autoload(ctx.skel->progs.trigger_driver, true); + ctx.skel->rodata->batch_iters = args.batch_iters; +} + +static void load_ctx(void) +{ + int err; err = trigger_bench__load(ctx.skel); if (err) { @@ -167,6 +173,9 @@ static void trigger_syscall_count_setup(void) static void trigger_kernel_count_setup(void) { setup_ctx(); + bpf_program__set_autoload(ctx.skel->progs.trigger_driver, false); + bpf_program__set_autoload(ctx.skel->progs.trigger_count, true); + load_ctx(); /* override driver program */ ctx.driver_prog_fd = bpf_program__fd(ctx.skel->progs.trigger_count); } @@ -174,36 +183,48 @@ static void trigger_kernel_count_setup(void) static void trigger_kprobe_setup(void) { setup_ctx(); + bpf_program__set_autoload(ctx.skel->progs.bench_trigger_kprobe, true); + load_ctx(); attach_bpf(ctx.skel->progs.bench_trigger_kprobe); } static void trigger_kretprobe_setup(void) { setup_ctx(); + bpf_program__set_autoload(ctx.skel->progs.bench_trigger_kretprobe, true); + load_ctx(); attach_bpf(ctx.skel->progs.bench_trigger_kretprobe); } static void trigger_kprobe_multi_setup(void) { setup_ctx(); + bpf_program__set_autoload(ctx.skel->progs.bench_trigger_kprobe_multi, true); + load_ctx(); attach_bpf(ctx.skel->progs.bench_trigger_kprobe_multi); } static void trigger_kretprobe_multi_setup(void) { setup_ctx(); + bpf_program__set_autoload(ctx.skel->progs.bench_trigger_kretprobe_multi, true); + load_ctx(); attach_bpf(ctx.skel->progs.bench_trigger_kretprobe_multi); } static void trigger_fentry_setup(void) { setup_ctx(); + bpf_program__set_autoload(ctx.skel->progs.bench_trigger_fentry, true); + load_ctx(); attach_bpf(ctx.skel->progs.bench_trigger_fentry); } static void trigger_fexit_setup(void) { setup_ctx(); + bpf_program__set_autoload(ctx.skel->progs.bench_trigger_fexit, true); + load_ctx(); attach_bpf(ctx.skel->progs.bench_trigger_fexit); } @@ -274,15 +295,24 @@ static void usetup(bool use_retprobe, void *target_addr) { size_t uprobe_offset; struct bpf_link *link; + int err; setup_libbpf(); - ctx.skel = trigger_bench__open_and_load(); + ctx.skel = trigger_bench__open(); if (!ctx.skel) { fprintf(stderr, "failed to open skeleton\n"); exit(1); } + bpf_program__set_autoload(ctx.skel->progs.bench_trigger_uprobe, true); + + err = trigger_bench__load(ctx.skel); + if (err) { + fprintf(stderr, "failed to load skeleton\n"); + exit(1); + } + uprobe_offset = get_uprobe_offset(target_addr); link = bpf_program__attach_uprobe(ctx.skel->progs.bench_trigger_uprobe, use_retprobe, diff --git a/tools/testing/selftests/bpf/progs/trigger_bench.c b/tools/testing/selftests/bpf/progs/trigger_bench.c index 81990e45b547..07587cb3c9f5 100644 --- a/tools/testing/selftests/bpf/progs/trigger_bench.c +++ b/tools/testing/selftests/bpf/progs/trigger_bench.c @@ -25,7 +25,7 @@ static __always_inline void inc_counter(void) __sync_add_and_fetch(&hits[cpu & CPU_MASK].value, 1); } -SEC("uprobe") +SEC("?uprobe") int bench_trigger_uprobe(void *ctx) { inc_counter(); @@ -34,7 +34,7 @@ int bench_trigger_uprobe(void *ctx) const volatile int batch_iters = 0; -SEC("raw_tp") +SEC("?raw_tp") int trigger_count(void *ctx) { int i; @@ -45,7 +45,7 @@ int trigger_count(void *ctx) return 0; } -SEC("raw_tp") +SEC("?raw_tp") int trigger_driver(void *ctx) { int i; @@ -56,42 +56,42 @@ int trigger_driver(void *ctx) return 0; } -SEC("kprobe/bpf_get_numa_node_id") +SEC("?kprobe/bpf_get_numa_node_id") int bench_trigger_kprobe(void *ctx) { inc_counter(); return 0; } -SEC("kretprobe/bpf_get_numa_node_id") +SEC("?kretprobe/bpf_get_numa_node_id") int bench_trigger_kretprobe(void *ctx) { inc_counter(); return 0; } -SEC("kprobe.multi/bpf_get_numa_node_id") +SEC("?kprobe.multi/bpf_get_numa_node_id") int bench_trigger_kprobe_multi(void *ctx) { inc_counter(); return 0; } -SEC("kretprobe.multi/bpf_get_numa_node_id") +SEC("?kretprobe.multi/bpf_get_numa_node_id") int bench_trigger_kretprobe_multi(void *ctx) { inc_counter(); return 0; } -SEC("fentry/bpf_get_numa_node_id") +SEC("?fentry/bpf_get_numa_node_id") int bench_trigger_fentry(void *ctx) { inc_counter(); return 0; } -SEC("fexit/bpf_get_numa_node_id") +SEC("?fexit/bpf_get_numa_node_id") int bench_trigger_fexit(void *ctx) { inc_counter(); From patchwork Fri Mar 22 00:00:40 2024 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Andrii Nakryiko X-Patchwork-Id: 13599475 X-Patchwork-Delegate: bpf@iogearbox.net Received: from smtp.kernel.org (aws-us-west-2-korg-mail-1.web.codeaurora.org [10.30.226.201]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (No client certificate requested) by smtp.subspace.kernel.org (Postfix) with ESMTPS id BAB9D818 for ; Fri, 22 Mar 2024 00:01:06 +0000 (UTC) Authentication-Results: smtp.subspace.kernel.org; arc=none smtp.client-ip=10.30.226.201 ARC-Seal: i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1711065666; cv=none; b=PAwhFOo/zAefKEE3DnqFO1INgdRfLNu2YuvM9GvTSXtMtSz/TQdeU+2OLObUQ//HA+w9zo510kvME5JIDrd0JeERm73SfuOxFSEEKbMgIu392BNq97ndGgDCj1733S4Gw6Ub0K42pBGfdo3CqrwTipQFvMwtrS+HjNoIvtj6zhw= ARC-Message-Signature: i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1711065666; c=relaxed/simple; bh=I0Fr6K0RHrxGOfhvSpmKiz3mE+CFTNXIv68ut2OVi3M=; h=From:To:Cc:Subject:Date:Message-ID:In-Reply-To:References: MIME-Version; b=ki+H9ykXBVndWJxk4IIWoX5ZOfsyfoa8rPE8tqwYfNtWnIXavD5OcRdGzydKbfMXhC18SsUUCmEvKQQhMIKKNnhfKMJAkE4d7W2+LXIRopnRwJ8vlQgrySFdcypeEg/YqJxv9m1UBDV/u1ZG23lvczNqBq3q3JTkQPQIM1PctoU= ARC-Authentication-Results: i=1; smtp.subspace.kernel.org; dkim=pass (2048-bit key) header.d=kernel.org header.i=@kernel.org header.b=JXw66Xku; arc=none smtp.client-ip=10.30.226.201 Authentication-Results: smtp.subspace.kernel.org; dkim=pass (2048-bit key) header.d=kernel.org header.i=@kernel.org header.b="JXw66Xku" Received: by smtp.kernel.org (Postfix) with ESMTPSA id 1EAD3C433F1; Fri, 22 Mar 2024 00:01:06 +0000 (UTC) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/simple; d=kernel.org; s=k20201202; t=1711065666; bh=I0Fr6K0RHrxGOfhvSpmKiz3mE+CFTNXIv68ut2OVi3M=; h=From:To:Cc:Subject:Date:In-Reply-To:References:From; b=JXw66Xku8ZzH/ZS9AELei+n2PUSSp62LIs8gCxdHRfkm0TumqreCtRVnwFgavKCh1 MhBbOnDCatNyoJCuKUoKkrBtjhKoMehSC50h9LH6HMGNP4lgEf4n4kcUa5yBS2mgiI 8oex1W7R/jIp+FvNR+8obAMlIXI/29kCK1hMB016H/5/6mQDJyLbUPw+ihQRdlCP7o t1+E8o7dDUQTjOtDyS890jSIdjHqA3+0E8fI32HBamfWX2j1xMyvf/r4fIw+u1Lmo7 ywN3g0TmmKPPmYWfbH2928IjBdjUYQ2i9Q67daX+MXneihpcIdOpyDv+K0Hg7TupE2 5v/A8121hNibw== From: Andrii Nakryiko To: bpf@vger.kernel.org, ast@kernel.org, daniel@iogearbox.net, martin.lau@kernel.org Cc: andrii@kernel.org, kernel-team@meta.com Subject: [PATCH bpf-next 5/6] bpf: add bpf_test_tp() kfunc triggering tp and allowing error injection Date: Thu, 21 Mar 2024 17:00:40 -0700 Message-ID: <20240322000041.2919948-6-andrii@kernel.org> X-Mailer: git-send-email 2.43.0 In-Reply-To: <20240322000041.2919948-1-andrii@kernel.org> References: <20240322000041.2919948-1-andrii@kernel.org> Precedence: bulk X-Mailing-List: bpf@vger.kernel.org List-Id: List-Subscribe: List-Unsubscribe: MIME-Version: 1.0 X-Patchwork-Delegate: bpf@iogearbox.net Add a simple bpf_test_tp() kfunc, available to all program types, that is useful for various testing and benchmarking scenarios, as it allows to trigger most tracing BPF program types from BPF side, allowing to do complex testing and benchmarking scenarios. Signed-off-by: Andrii Nakryiko --- kernel/bpf/bpf_test.h | 34 ++++++++++++++++++++++++++++++++++ kernel/bpf/helpers.c | 13 +++++++++++++ 2 files changed, 47 insertions(+) create mode 100644 kernel/bpf/bpf_test.h diff --git a/kernel/bpf/bpf_test.h b/kernel/bpf/bpf_test.h new file mode 100644 index 000000000000..c779927d3574 --- /dev/null +++ b/kernel/bpf/bpf_test.h @@ -0,0 +1,34 @@ +/* SPDX-License-Identifier: GPL-2.0 */ +#undef TRACE_SYSTEM +#define TRACE_SYSTEM bpf_test + +#if !defined(_TRACE_BPF_TEST_H) || defined(TRACE_HEADER_MULTI_READ) + +#define _TRACE_BPF_TEST_H + +#include + +TRACE_EVENT(bpf_test, + + TP_PROTO(int nonce), + + TP_ARGS(nonce), + + TP_STRUCT__entry( + __field(int, nonce) + ), + + TP_fast_assign( + __entry->nonce = nonce; + ), + + TP_printk("nonce %d", __entry->nonce) +); + +#endif /* _TRACE_BPF_TEST_H */ + +#undef TRACE_INCLUDE_PATH +#define TRACE_INCLUDE_PATH . +#define TRACE_INCLUDE_FILE bpf_test + +#include diff --git a/kernel/bpf/helpers.c b/kernel/bpf/helpers.c index 9234174ccb21..67bf9659c447 100644 --- a/kernel/bpf/helpers.c +++ b/kernel/bpf/helpers.c @@ -26,6 +26,10 @@ #include "../../lib/kstrtox.h" +#define CREATE_TRACE_POINTS +#include "bpf_test.h" + + /* If kernel subsystem is allowing eBPF programs to call this function, * inside its own verifier_ops->get_func_proto() callback it should return * bpf_map_lookup_elem_proto, so that verifier can properly check the arguments @@ -2549,6 +2553,14 @@ __bpf_kfunc void bpf_throw(u64 cookie) WARN(1, "A call to BPF exception callback should never return\n"); } +__bpf_kfunc int bpf_test_tp(int nonce) +{ + trace_bpf_test(nonce); + + return nonce; +} +ALLOW_ERROR_INJECTION(bpf_test_tp, ERRNO); + __bpf_kfunc_end_defs(); BTF_KFUNCS_START(generic_btf_ids) @@ -2625,6 +2637,7 @@ BTF_ID_FLAGS(func, bpf_dynptr_is_null) BTF_ID_FLAGS(func, bpf_dynptr_is_rdonly) BTF_ID_FLAGS(func, bpf_dynptr_size) BTF_ID_FLAGS(func, bpf_dynptr_clone) +BTF_ID_FLAGS(func, bpf_test_tp) BTF_KFUNCS_END(common_btf_ids) static const struct btf_kfunc_id_set common_kfunc_set = { From patchwork Fri Mar 22 00:00:41 2024 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Andrii Nakryiko X-Patchwork-Id: 13599476 X-Patchwork-Delegate: bpf@iogearbox.net Received: from smtp.kernel.org (aws-us-west-2-korg-mail-1.web.codeaurora.org [10.30.226.201]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (No client certificate requested) by smtp.subspace.kernel.org (Postfix) with ESMTPS id ED2C83C38 for ; Fri, 22 Mar 2024 00:01:09 +0000 (UTC) Authentication-Results: smtp.subspace.kernel.org; arc=none smtp.client-ip=10.30.226.201 ARC-Seal: i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1711065671; cv=none; b=iNVepfMj9Xdrofx0mckeLmhiW+qHYnl+08SUtKhRCsMxQvJoEkD6AAIjnYZXeVUdBCiOTFIa8fzIEHTRpAc862acTrMw08IZwzvKo5XCXUfh2Ztk/DlsBhJQ/ZEFU/7oBezsKfsFw3GHilM5J84dITtLP6IiM0kNdH0ZNG8T/+U= ARC-Message-Signature: i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1711065671; c=relaxed/simple; bh=wdvQRy4wOgdYyfXL64ZdYS6o5z9qVPOWHI30GDYsYQc=; h=From:To:Cc:Subject:Date:Message-ID:In-Reply-To:References: MIME-Version; b=N537YNIteCZbegmvitvuxkaxapiJcPDaipDVGeYzWBKnrmxmJzcaPwGdk15/sQmeEee1D2u0Bsh4aeUnMIk0U83szDjWb2Nt3Z78O6NjP6femv0iEzu8qtbQNGlu3MueLnFhI01UXUlQXYpO1jFvNGsRB7aXjVhyy0voIcMaLTA= ARC-Authentication-Results: i=1; smtp.subspace.kernel.org; dkim=pass (2048-bit key) header.d=kernel.org header.i=@kernel.org header.b=sQ9EdDZo; arc=none smtp.client-ip=10.30.226.201 Authentication-Results: smtp.subspace.kernel.org; dkim=pass (2048-bit key) header.d=kernel.org header.i=@kernel.org header.b="sQ9EdDZo" Received: by smtp.kernel.org (Postfix) with ESMTPSA id 48A66C433C7; Fri, 22 Mar 2024 00:01:09 +0000 (UTC) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/simple; d=kernel.org; s=k20201202; t=1711065669; bh=wdvQRy4wOgdYyfXL64ZdYS6o5z9qVPOWHI30GDYsYQc=; h=From:To:Cc:Subject:Date:In-Reply-To:References:From; b=sQ9EdDZoORxYgJ3zegG1gU4SxHC+Snd/gAkpTg8nZ4n0YL8sa0btYZiWvl69St1Fh tlw1ZEL7IlaabU9Y2e/U6JxAKwHtz6pktROAkUvbgt/86w4aXW3lw+8RZ/TAcjntFL 3awDl/95Irg1MjRKxQSyqM60PiZrxLAFfwHabupqZbDxLBJokN+AsKCcNejJLsPhpY kZeirNJaW+kXW4ScLd118z6tSC4RCSWY0ZYwo7w4i58RLCm/TFP4TkOEMdD8KyLMxe UQ/Ou96DWtdd4gzuGBJDWKOxE81qDv7fy1c5EIz8lofQoH6nVfMW0ysBArN4QzIOWA OfXqVYtXYjFjA== From: Andrii Nakryiko To: bpf@vger.kernel.org, ast@kernel.org, daniel@iogearbox.net, martin.lau@kernel.org Cc: andrii@kernel.org, kernel-team@meta.com Subject: [PATCH bpf-next 6/6] selftests/bpf: add batched tp/raw_tp/fmodret tests Date: Thu, 21 Mar 2024 17:00:41 -0700 Message-ID: <20240322000041.2919948-7-andrii@kernel.org> X-Mailer: git-send-email 2.43.0 In-Reply-To: <20240322000041.2919948-1-andrii@kernel.org> References: <20240322000041.2919948-1-andrii@kernel.org> Precedence: bulk X-Mailing-List: bpf@vger.kernel.org List-Id: List-Subscribe: List-Unsubscribe: MIME-Version: 1.0 X-Patchwork-Delegate: bpf@iogearbox.net Utilize bpf_test_tp() kfunc to have a fast way to trigger tp/raw_tp/fmodret programs from another BPF program, which gives us comparable batched benchmarks to (batched) kprobe/fentry benchmarks. We don't switch kprobe/fentry batched benchmarks to this kfunc to make bench tool usable on older kernels as well. Signed-off-by: Andrii Nakryiko --- tools/testing/selftests/bpf/bench.c | 6 +++ .../selftests/bpf/benchs/bench_trigger.c | 39 +++++++++++++++++++ .../selftests/bpf/benchs/run_bench_trigger.sh | 3 +- .../selftests/bpf/progs/trigger_bench.c | 34 ++++++++++++++++ 4 files changed, 81 insertions(+), 1 deletion(-) diff --git a/tools/testing/selftests/bpf/bench.c b/tools/testing/selftests/bpf/bench.c index 8b16841a3dec..82de56c8162e 100644 --- a/tools/testing/selftests/bpf/bench.c +++ b/tools/testing/selftests/bpf/bench.c @@ -506,6 +506,9 @@ extern const struct bench bench_trig_kprobe_multi; extern const struct bench bench_trig_kretprobe_multi; extern const struct bench bench_trig_fentry; extern const struct bench bench_trig_fexit; +extern const struct bench bench_trig_fmodret; +extern const struct bench bench_trig_tp; +extern const struct bench bench_trig_rawtp; /* uprobe/uretprobe benchmarks */ extern const struct bench bench_trig_uprobe_nop; @@ -556,6 +559,9 @@ static const struct bench *benchs[] = { &bench_trig_kretprobe_multi, &bench_trig_fentry, &bench_trig_fexit, + &bench_trig_fmodret, + &bench_trig_tp, + &bench_trig_rawtp, /* uprobes */ &bench_trig_uprobe_nop, &bench_trig_uretprobe_nop, diff --git a/tools/testing/selftests/bpf/benchs/bench_trigger.c b/tools/testing/selftests/bpf/benchs/bench_trigger.c index e454d7de2252..5c4b8987ccbd 100644 --- a/tools/testing/selftests/bpf/benchs/bench_trigger.c +++ b/tools/testing/selftests/bpf/benchs/bench_trigger.c @@ -228,6 +228,42 @@ static void trigger_fexit_setup(void) attach_bpf(ctx.skel->progs.bench_trigger_fexit); } +static void trigger_fmodret_setup(void) +{ + setup_ctx(); + bpf_program__set_autoload(ctx.skel->progs.trigger_driver, false); + bpf_program__set_autoload(ctx.skel->progs.trigger_driver_kfunc, true); + bpf_program__set_autoload(ctx.skel->progs.bench_trigger_fmodret, true); + load_ctx(); + /* override driver program */ + ctx.driver_prog_fd = bpf_program__fd(ctx.skel->progs.trigger_driver_kfunc); + attach_bpf(ctx.skel->progs.bench_trigger_fmodret); +} + +static void trigger_tp_setup(void) +{ + setup_ctx(); + bpf_program__set_autoload(ctx.skel->progs.trigger_driver, false); + bpf_program__set_autoload(ctx.skel->progs.trigger_driver_kfunc, true); + bpf_program__set_autoload(ctx.skel->progs.bench_trigger_tp, true); + load_ctx(); + /* override driver program */ + ctx.driver_prog_fd = bpf_program__fd(ctx.skel->progs.trigger_driver_kfunc); + attach_bpf(ctx.skel->progs.bench_trigger_tp); +} + +static void trigger_rawtp_setup(void) +{ + setup_ctx(); + bpf_program__set_autoload(ctx.skel->progs.trigger_driver, false); + bpf_program__set_autoload(ctx.skel->progs.trigger_driver_kfunc, true); + bpf_program__set_autoload(ctx.skel->progs.bench_trigger_rawtp, true); + load_ctx(); + /* override driver program */ + ctx.driver_prog_fd = bpf_program__fd(ctx.skel->progs.trigger_driver_kfunc); + attach_bpf(ctx.skel->progs.bench_trigger_rawtp); +} + /* make sure call is not inlined and not avoided by compiler, so __weak and * inline asm volatile in the body of the function * @@ -390,6 +426,9 @@ BENCH_TRIG_KERNEL(kprobe_multi, "kprobe-multi"); BENCH_TRIG_KERNEL(kretprobe_multi, "kretprobe-multi"); BENCH_TRIG_KERNEL(fentry, "fentry"); BENCH_TRIG_KERNEL(fexit, "fexit"); +BENCH_TRIG_KERNEL(fmodret, "fmodret"); +BENCH_TRIG_KERNEL(tp, "tp"); +BENCH_TRIG_KERNEL(rawtp, "rawtp"); /* uprobe benchmarks */ #define BENCH_TRIG_USERMODE(KIND, PRODUCER, NAME) \ diff --git a/tools/testing/selftests/bpf/benchs/run_bench_trigger.sh b/tools/testing/selftests/bpf/benchs/run_bench_trigger.sh index 6e790e294260..a690f5a68b6b 100755 --- a/tools/testing/selftests/bpf/benchs/run_bench_trigger.sh +++ b/tools/testing/selftests/bpf/benchs/run_bench_trigger.sh @@ -4,7 +4,8 @@ set -eufo pipefail def_tests=( \ usermode-count kernel-count syscall-count \ - fentry fexit \ + fentry fexit fmodret \ + rawtp tp \ kprobe kprobe-multi \ kretprobe kretprobe-multi \ ) diff --git a/tools/testing/selftests/bpf/progs/trigger_bench.c b/tools/testing/selftests/bpf/progs/trigger_bench.c index 07587cb3c9f5..914ed625441a 100644 --- a/tools/testing/selftests/bpf/progs/trigger_bench.c +++ b/tools/testing/selftests/bpf/progs/trigger_bench.c @@ -56,6 +56,19 @@ int trigger_driver(void *ctx) return 0; } +extern int bpf_test_tp(int nonce) __ksym __weak; + +SEC("?raw_tp") +int trigger_driver_kfunc(void *ctx) +{ + int i; + + for (i = 0; i < batch_iters; i++) + (void)bpf_test_tp(0); /* attach point for benchmarking */ + + return 0; +} + SEC("?kprobe/bpf_get_numa_node_id") int bench_trigger_kprobe(void *ctx) { @@ -97,3 +110,24 @@ int bench_trigger_fexit(void *ctx) inc_counter(); return 0; } + +SEC("?fmod_ret/bpf_test_tp") +int bench_trigger_fmodret(void *ctx) +{ + inc_counter(); + return -22; +} + +SEC("?tp/bpf_test/bpf_test") +int bench_trigger_tp(void *ctx) +{ + inc_counter(); + return 0; +} + +SEC("?raw_tp/bpf_test") +int bench_trigger_rawtp(void *ctx) +{ + inc_counter(); + return 0; +}