From patchwork Mon Mar 11 09:35:18 2024 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: =?utf-8?b?5qKm6b6Z6JGj?= X-Patchwork-Id: 13588447 X-Patchwork-Delegate: bjorn@kernel.org Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from bombadil.infradead.org (bombadil.infradead.org [198.137.202.133]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (No client certificate requested) by smtp.lore.kernel.org (Postfix) with ESMTPS id BC394C5475B for ; Mon, 11 Mar 2024 09:34:49 +0000 (UTC) DKIM-Signature: v=1; a=rsa-sha256; q=dns/txt; c=relaxed/relaxed; d=lists.infradead.org; s=bombadil.20210309; h=Sender: Content-Transfer-Encoding:Content-Type:List-Subscribe:List-Help:List-Post: List-Archive:List-Unsubscribe:List-Id:MIME-Version:References:In-Reply-To: Message-Id:Date:Subject:Cc:To:From:Reply-To:Content-ID:Content-Description: Resent-Date:Resent-From:Resent-Sender:Resent-To:Resent-Cc:Resent-Message-ID: List-Owner; bh=EZoEY20ctcdeaJasvoQ+e9p2/9DSC6W7wByRZ9EU9so=; b=QnWs4GQHCnh61U p8mfVTG5EVUhKEJiW4O5bnWg4WVdu1UcPcalIfkkGbYoGOe4Em577bN1lbxFCv99+S4j0XcpDtbld bdviElJxWt3RRrj0xrnohtpbEwSwHW211zFETssnI/FOip0KNGvwXeCi6m8gexWtUG7X1qrNEfJ/r D1jSzpEuErTqD1r/wpJvC6JFRKd3fE/AXrJ4s+nAJhVlebeNZCg4rZt+NalMH422GnxQ2NWzCtKWz FyoXaIe8XuBUCyrlZmDSnRowSWEZuQjmo2H8ky5U1ydW5IKWts6Jw3az2P9koRxTYygPYeK2Y81QF DsDdrC5Z8EBdndtzqC1w==; Received: from localhost ([::1] helo=bombadil.infradead.org) by bombadil.infradead.org with esmtp (Exim 4.97.1 #2 (Red Hat Linux)) id 1rjc3Q-00000000rua-0rvS; Mon, 11 Mar 2024 09:34:44 +0000 Received: from mail-pf1-x429.google.com ([2607:f8b0:4864:20::429]) by bombadil.infradead.org with esmtps (Exim 4.97.1 #2 (Red Hat Linux)) id 1rjc3J-00000000rpW-45IM for linux-riscv@lists.infradead.org; Mon, 11 Mar 2024 09:34:40 +0000 Received: by mail-pf1-x429.google.com with SMTP id d2e1a72fcca58-6e55731af5cso1863068b3a.0 for ; Mon, 11 Mar 2024 02:34:36 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=bytedance.com; s=google; t=1710149676; x=1710754476; darn=lists.infradead.org; h=content-transfer-encoding:mime-version:references:in-reply-to :message-id:date:subject:cc:to:from:from:to:cc:subject:date :message-id:reply-to; bh=1DNwS3vzNHKnmJ9gtI+mTH5qdekBkGxgiUpqVAYV/P4=; b=K/fRkXaVS102iVClmbFSeKtA59LPy0+tu9Tdv4OUSi+jj9iWdVmdt9wYhcfIYlK+3q hlx6a7qqRPDKLmNRtW/SlVePJlcis8Aia0yf8yzY7DFfyD22zzktupL66YW+pfFjkhxP vorz21fn4I+89brenHW3uEe98nIqAKHsjXYcf6pEAlqcuqIt4EaQLVRzRU++oJs5Pwl+ dH0O/wA8o1WzNaxNIjmNgoVMZAhdYkmMIFV++i/7DyvNVDXzxRVk8Pk8o7u6+IAMz4Ci aw03CR/MVVAxYCyF9chtV43LW21MrOPcm9NJbYlBwMn+fThb0VuOdN9YkJA6STn3oNrp pDYw== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20230601; t=1710149676; x=1710754476; h=content-transfer-encoding:mime-version:references:in-reply-to :message-id:date:subject:cc:to:from:x-gm-message-state:from:to:cc :subject:date:message-id:reply-to; bh=1DNwS3vzNHKnmJ9gtI+mTH5qdekBkGxgiUpqVAYV/P4=; b=iPcqpofEeRYhiwSZaIHgwRALoRD9KOGD5mjh9WDeTEy9amOUGxohMcO/6p3GYI9EvM D2JOQrFALsCsRjqPwLIgIa8XhvZYVmonbgm6Gu04iizojcKUwsxba/1YohdaYrnqPO76 Ac8AlaKmiJdC0EcwZ1ef2DDY0XWe4kIeXBk3VAyuYJw9zEbLufwRuez4jTuZlm6BIXLI gUTmrGQV4cbsUzuWKoi4phSYQf2HnvtsOTUghCUSrKJRRhWQI72vfKmyshwJuStIDKn2 /ACsOtD9qI9IS9P777mILs5f4ISkW1ahXmqYnXbAaLE527wZFX+Vh0A8faIPJGDBfNgm dVlw== X-Forwarded-Encrypted: i=1; AJvYcCX+RVihW2So7zYMAECoPD/j7GqxlVYn6xvcZo0Q68dhTZFCwL/PmdCHgfjkTSkMf0ann5uzyiKrUqefpLD+NuQnHAkX++0pqqbnnT620w4M X-Gm-Message-State: AOJu0YzWzxzoPab6Nwb/etuIzSaXFWa1qhXj5jXgKVjAsFHK63TYeNYc CnxMpMgbUXvRylomzcLpNs/Ln9XwFEjYyj0SYKFHZLYbwqr6ZTdIa9b5hGb5WIoIWDgUbMaPUYr U X-Google-Smtp-Source: AGHT+IHp6WfuUeDSjQx2cauv1mQ/eMbX3s4chl5kJVOY3tZJvKu5JYf89lzfYO6yWTa1+IHjqNIpUg== X-Received: by 2002:a05:6a21:7890:b0:1a1:6cf5:5bf1 with SMTP id bf16-20020a056a21789000b001a16cf55bf1mr3944233pzc.12.1710149676017; Mon, 11 Mar 2024 02:34:36 -0700 (PDT) Received: from localhost.localdomain ([43.129.25.208]) by smtp.gmail.com with ESMTPSA id h9-20020a170902f7c900b001dcad9cbf8bsm4253365plw.239.2024.03.11.02.34.28 (version=TLS1_3 cipher=TLS_AES_256_GCM_SHA384 bits=256/256); Mon, 11 Mar 2024 02:34:35 -0700 (PDT) From: Menglong Dong To: andrii@kernel.org Cc: ast@kernel.org, daniel@iogearbox.net, martin.lau@linux.dev, eddyz87@gmail.com, song@kernel.org, yonghong.song@linux.dev, john.fastabend@gmail.com, kpsingh@kernel.org, sdf@google.com, haoluo@google.com, jolsa@kernel.org, agordeev@linux.ibm.com, borntraeger@linux.ibm.com, svens@linux.ibm.com, davem@davemloft.net, dsahern@kernel.org, dave.hansen@linux.intel.com, x86@kernel.org, rostedt@goodmis.org, mathieu.desnoyers@efficios.com, quentin@isovalent.com, bpf@vger.kernel.org, linux-arm-kernel@lists.infradead.org, linux-kernel@vger.kernel.org, linux-riscv@lists.infradead.org, linux-s390@vger.kernel.org, netdev@vger.kernel.org, linux-trace-kernel@vger.kernel.org, linux-kselftest@vger.kernel.org, linux-stm32@st-md-mailman.stormreply.com, Menglong Dong Subject: [PATCH bpf-next v2 1/9] bpf: tracing: add support to record and check the accessed args Date: Mon, 11 Mar 2024 17:35:18 +0800 Message-Id: <20240311093526.1010158-2-dongmenglong.8@bytedance.com> X-Mailer: git-send-email 2.39.2 In-Reply-To: <20240311093526.1010158-1-dongmenglong.8@bytedance.com> References: <20240311093526.1010158-1-dongmenglong.8@bytedance.com> MIME-Version: 1.0 X-CRM114-Version: 20100106-BlameMichelson ( TRE 0.8.0 (BSD) ) MR-646709E3 X-CRM114-CacheID: sfid-20240311_023438_210412_7C417FE5 X-CRM114-Status: GOOD ( 23.31 ) X-BeenThere: linux-riscv@lists.infradead.org X-Mailman-Version: 2.1.34 Precedence: list List-Id: List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Sender: "linux-riscv" Errors-To: linux-riscv-bounces+linux-riscv=archiver.kernel.org@lists.infradead.org In this commit, we add the 'accessed_args' field to struct bpf_prog_aux, which is used to record the accessed index of the function args in btf_ctx_access(). Meanwhile, we add the function btf_check_func_part_match() to compare the accessed function args of two function prototype. This function will be used in the following commit. Signed-off-by: Menglong Dong --- include/linux/bpf.h | 4 ++ kernel/bpf/btf.c | 108 +++++++++++++++++++++++++++++++++++++++++++- 2 files changed, 110 insertions(+), 2 deletions(-) diff --git a/include/linux/bpf.h b/include/linux/bpf.h index 95e07673cdc1..0f677fdcfcc7 100644 --- a/include/linux/bpf.h +++ b/include/linux/bpf.h @@ -1461,6 +1461,7 @@ struct bpf_prog_aux { const struct btf_type *attach_func_proto; /* function name for valid attach_btf_id */ const char *attach_func_name; + u64 accessed_args; struct bpf_prog **func; void *jit_data; /* JIT specific data. arch dependent */ struct bpf_jit_poke_descriptor *poke_tab; @@ -2565,6 +2566,9 @@ struct bpf_reg_state; int btf_prepare_func_args(struct bpf_verifier_env *env, int subprog); int btf_check_type_match(struct bpf_verifier_log *log, const struct bpf_prog *prog, struct btf *btf, const struct btf_type *t); +int btf_check_func_part_match(struct btf *btf1, const struct btf_type *t1, + struct btf *btf2, const struct btf_type *t2, + u64 func_args); const char *btf_find_decl_tag_value(const struct btf *btf, const struct btf_type *pt, int comp_idx, const char *tag_key); int btf_find_next_decl_tag(const struct btf *btf, const struct btf_type *pt, diff --git a/kernel/bpf/btf.c b/kernel/bpf/btf.c index 170d017e8e4a..c2a0299d4358 100644 --- a/kernel/bpf/btf.c +++ b/kernel/bpf/btf.c @@ -6125,19 +6125,24 @@ static bool is_int_ptr(struct btf *btf, const struct btf_type *t) } static u32 get_ctx_arg_idx(struct btf *btf, const struct btf_type *func_proto, - int off) + int off, int *aligned_idx) { const struct btf_param *args; const struct btf_type *t; u32 offset = 0, nr_args; int i; + if (aligned_idx) + *aligned_idx = -ENOENT; + if (!func_proto) return off / 8; nr_args = btf_type_vlen(func_proto); args = (const struct btf_param *)(func_proto + 1); for (i = 0; i < nr_args; i++) { + if (aligned_idx && offset == off) + *aligned_idx = i; t = btf_type_skip_modifiers(btf, args[i].type, NULL); offset += btf_type_is_ptr(t) ? 8 : roundup(t->size, 8); if (off < offset) @@ -6207,7 +6212,7 @@ bool btf_ctx_access(int off, int size, enum bpf_access_type type, tname, off); return false; } - arg = get_ctx_arg_idx(btf, t, off); + arg = get_ctx_arg_idx(btf, t, off, NULL); args = (const struct btf_param *)(t + 1); /* if (t == NULL) Fall back to default BPF prog with * MAX_BPF_FUNC_REG_ARGS u64 arguments. @@ -6217,6 +6222,9 @@ bool btf_ctx_access(int off, int size, enum bpf_access_type type, /* skip first 'void *__data' argument in btf_trace_##name typedef */ args++; nr_args--; + prog->aux->accessed_args |= (1 << (arg + 1)); + } else { + prog->aux->accessed_args |= (1 << arg); } if (arg > nr_args) { @@ -7024,6 +7032,102 @@ int btf_check_type_match(struct bpf_verifier_log *log, const struct bpf_prog *pr return btf_check_func_type_match(log, btf1, t1, btf2, t2); } +static u32 get_ctx_arg_total_size(struct btf *btf, const struct btf_type *t) +{ + const struct btf_param *args; + u32 size = 0, nr_args; + int i; + + nr_args = btf_type_vlen(t); + args = (const struct btf_param *)(t + 1); + for (i = 0; i < nr_args; i++) { + t = btf_type_skip_modifiers(btf, args[i].type, NULL); + size += btf_type_is_ptr(t) ? 8 : roundup(t->size, 8); + } + + return size; +} + +/* This function is similar to btf_check_func_type_match(), except that it + * only compare some function args of the function prototype t1 and t2. + */ +int btf_check_func_part_match(struct btf *btf1, const struct btf_type *func1, + struct btf *btf2, const struct btf_type *func2, + u64 func_args) +{ + const struct btf_param *args1, *args2; + u32 nargs1, i, offset = 0; + const char *s1, *s2; + + if (!btf_type_is_func_proto(func1) || !btf_type_is_func_proto(func2)) + return -EINVAL; + + args1 = (const struct btf_param *)(func1 + 1); + args2 = (const struct btf_param *)(func2 + 1); + nargs1 = btf_type_vlen(func1); + + for (i = 0; i <= nargs1; i++) { + const struct btf_type *t1, *t2; + + if (!(func_args & (1 << i))) + goto next; + + if (i < nargs1) { + int t2_index; + + /* get the index of the arg corresponding to args1[i] + * by the offset. + */ + get_ctx_arg_idx(btf2, func2, offset, &t2_index); + if (t2_index < 0) + return -EINVAL; + + t1 = btf_type_skip_modifiers(btf1, args1[i].type, NULL); + t2 = btf_type_skip_modifiers(btf2, args2[t2_index].type, + NULL); + } else { + /* i == nargs1, this is the index of return value of t1 */ + if (get_ctx_arg_total_size(btf1, func1) != + get_ctx_arg_total_size(btf2, func2)) + return -EINVAL; + + /* check the return type of t1 and t2 */ + t1 = btf_type_skip_modifiers(btf1, func1->type, NULL); + t2 = btf_type_skip_modifiers(btf2, func2->type, NULL); + } + + if (t1->info != t2->info || + (btf_type_has_size(t1) && t1->size != t2->size)) + return -EINVAL; + if (btf_type_is_int(t1) || btf_is_any_enum(t1)) + goto next; + + if (btf_type_is_struct(t1)) + goto on_struct; + + if (!btf_type_is_ptr(t1)) + return -EINVAL; + + t1 = btf_type_skip_modifiers(btf1, t1->type, NULL); + t2 = btf_type_skip_modifiers(btf2, t2->type, NULL); + if (!btf_type_is_struct(t1) || !btf_type_is_struct(t2)) + return -EINVAL; + +on_struct: + s1 = btf_name_by_offset(btf1, t1->name_off); + s2 = btf_name_by_offset(btf2, t2->name_off); + if (strcmp(s1, s2)) + return -EINVAL; +next: + if (i < nargs1) { + t1 = btf_type_skip_modifiers(btf1, args1[i].type, NULL); + offset += btf_type_is_ptr(t1) ? 8 : roundup(t1->size, 8); + } + } + + return 0; +} + static bool btf_is_dynptr_ptr(const struct btf *btf, const struct btf_type *t) { const char *name; From patchwork Mon Mar 11 09:35:19 2024 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: =?utf-8?b?5qKm6b6Z6JGj?= X-Patchwork-Id: 13588448 X-Patchwork-Delegate: bjorn@kernel.org Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from bombadil.infradead.org (bombadil.infradead.org [198.137.202.133]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (No client certificate requested) by smtp.lore.kernel.org (Postfix) with ESMTPS id 813CBC5475B for ; Mon, 11 Mar 2024 09:34:58 +0000 (UTC) DKIM-Signature: v=1; a=rsa-sha256; q=dns/txt; c=relaxed/relaxed; d=lists.infradead.org; s=bombadil.20210309; h=Sender: Content-Transfer-Encoding:Content-Type:List-Subscribe:List-Help:List-Post: List-Archive:List-Unsubscribe:List-Id:MIME-Version:References:In-Reply-To: Message-Id:Date:Subject:Cc:To:From:Reply-To:Content-ID:Content-Description: Resent-Date:Resent-From:Resent-Sender:Resent-To:Resent-Cc:Resent-Message-ID: List-Owner; bh=u0dWdLXFpU9TSCGxUtBRpP/0sdXhRYEpWLiuhBKUy+E=; b=S0JYgwF1aRlyVc 5pEt/HrTeMeoDBL8iB7cbMvAO4k8fqdfPLN1FCJQ5es2fUTiZi2a+q2n4SdKtj6S4Ng5AaPzCvAAh S5Z51czLsM/h5LUmXZor1NCcvLnXDpTZRU9A0Ptv9ZqDJNbbYXhqqlCkJKeMKTfjDdA5Hf2SiogdF Nl2v3Nm+IwpRkp2TquxOfhyOzHRrp2RRsEkFqn8Ufxu1C6oyAwwrhZ95Upjgxsc3D8xiJS0nh8iFq R3PfCfo1n011iRj/SEwW39sRP3OyhNRVUD3zDHG3u9rX/pq9lq6qRQrovV1oEfDHV2nrFF5YgWFH8 lMZfZph0msvkpcjTcB8g==; Received: from localhost ([::1] helo=bombadil.infradead.org) by bombadil.infradead.org with esmtp (Exim 4.97.1 #2 (Red Hat Linux)) id 1rjc3Z-00000000rzF-18RC; Mon, 11 Mar 2024 09:34:53 +0000 Received: from mail-pf1-x42f.google.com ([2607:f8b0:4864:20::42f]) by bombadil.infradead.org with esmtps (Exim 4.97.1 #2 (Red Hat Linux)) id 1rjc3S-00000000ruV-2dHn for linux-riscv@lists.infradead.org; Mon, 11 Mar 2024 09:34:50 +0000 Received: by mail-pf1-x42f.google.com with SMTP id d2e1a72fcca58-6e676ea4e36so1807267b3a.3 for ; Mon, 11 Mar 2024 02:34:44 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=bytedance.com; s=google; t=1710149683; x=1710754483; darn=lists.infradead.org; h=content-transfer-encoding:mime-version:references:in-reply-to :message-id:date:subject:cc:to:from:from:to:cc:subject:date :message-id:reply-to; bh=QjSWV41JlB3dq7HwIpKEF4e0TQa/o2d+dOhHoSuTF7U=; b=Oi9/9cE0k09cbyYVZlQ8tcPoQgG9mk9x2qcGfO4AU5Mrg75W9HWJg9r7vpXfyaJ5rs 2nlpRZlfh1UxV5vl5WS+FYGxHrTbDJ5SGJb0zCUQC1uzw/MCwT/IsRFljS3tyVFU2Meu 0btCJGqoIEXfcKa2wAIVrVDNq3uI4hR19wFMrFc0RObtv7IhjM/TtQunol6utTUvWoEX eUVLCue1l0FecC44krRxwHRy4nVr8PYETmnni1xJwWgABF9uNcL52oWWGj/YXMQ+WGuk lGsGsYJfWqeVQx73lJmODuAwoYSdjMdOEdYvsNx9UzOgzqovxBeLOcGtk0eZX1ID3Piv y7BA== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20230601; t=1710149683; x=1710754483; h=content-transfer-encoding:mime-version:references:in-reply-to :message-id:date:subject:cc:to:from:x-gm-message-state:from:to:cc :subject:date:message-id:reply-to; bh=QjSWV41JlB3dq7HwIpKEF4e0TQa/o2d+dOhHoSuTF7U=; b=XDMBaUfxTTtvrj75+mbOvuWfH9P7qH/OIr/uwJ5lzNTXoEp8LF1TEGLB3+wRsZt2bq GvLHSacWm3v3Ls8QKYbDMR+MsrVQU3q0Gu2nRSVQZe5wxWPymIlkNH/IgJglnUoqdA2Y FTHxErLLyPCYNUQwuxjwesHimQ+Ohoyax44Ksq5sWkg+PvFPMrrVgSRyGW+b47HUZN6f HRZwjbSoxk2Pcvrf9qhF241lOZolg1ZRzUUh65MDvK+HwhY9KhKsMEyn/MY/8G11tSgf 3ZuDZ6B/i3gP1+ZwGw8P/z0qvvXo9GiRS5i3CUDhFMiw3eIRxVJNe1AjfYwkfZY5BnB5 8WAA== X-Forwarded-Encrypted: i=1; AJvYcCUC+VOQEbBZWHKV9N/Kb3DD2Ys2p1wPac+YaGAaxwZguNKp9BYM8ttlyOj+UwTXuGmlU5vvhnqN+8wW8XWUquiz7K4tO9Rsj5ynT1jFQCBt X-Gm-Message-State: AOJu0YzQo8PYbSyoxtkl4vp7AKQo2VSPIUp3g20MwAzV+JYFqiKx5puO M0YMZL70nxNgmCI6fDUWmeVsX2Fu6cWFgCSisUPOlWIIBftM01j7s3luSPxc3W8= X-Google-Smtp-Source: AGHT+IFbj6YfcHBYvXMM2NWizkuYpniqg3bImjqE4WhHnt6u4m7Hz7unmoAwRceekSZcHEkG6VDAnw== X-Received: by 2002:a05:6a20:438a:b0:1a1:431c:6696 with SMTP id i10-20020a056a20438a00b001a1431c6696mr4574882pzl.49.1710149683554; Mon, 11 Mar 2024 02:34:43 -0700 (PDT) Received: from localhost.localdomain ([43.129.25.208]) by smtp.gmail.com with ESMTPSA id h9-20020a170902f7c900b001dcad9cbf8bsm4253365plw.239.2024.03.11.02.34.36 (version=TLS1_3 cipher=TLS_AES_256_GCM_SHA384 bits=256/256); Mon, 11 Mar 2024 02:34:43 -0700 (PDT) From: Menglong Dong To: andrii@kernel.org Cc: ast@kernel.org, daniel@iogearbox.net, martin.lau@linux.dev, eddyz87@gmail.com, song@kernel.org, yonghong.song@linux.dev, john.fastabend@gmail.com, kpsingh@kernel.org, sdf@google.com, haoluo@google.com, jolsa@kernel.org, agordeev@linux.ibm.com, borntraeger@linux.ibm.com, svens@linux.ibm.com, davem@davemloft.net, dsahern@kernel.org, dave.hansen@linux.intel.com, x86@kernel.org, rostedt@goodmis.org, mathieu.desnoyers@efficios.com, quentin@isovalent.com, bpf@vger.kernel.org, linux-arm-kernel@lists.infradead.org, linux-kernel@vger.kernel.org, linux-riscv@lists.infradead.org, linux-s390@vger.kernel.org, netdev@vger.kernel.org, linux-trace-kernel@vger.kernel.org, linux-kselftest@vger.kernel.org, linux-stm32@st-md-mailman.stormreply.com, Menglong Dong Subject: [PATCH bpf-next v2 2/9] bpf: refactor the modules_array to ptr_array Date: Mon, 11 Mar 2024 17:35:19 +0800 Message-Id: <20240311093526.1010158-3-dongmenglong.8@bytedance.com> X-Mailer: git-send-email 2.39.2 In-Reply-To: <20240311093526.1010158-1-dongmenglong.8@bytedance.com> References: <20240311093526.1010158-1-dongmenglong.8@bytedance.com> MIME-Version: 1.0 X-CRM114-Version: 20100106-BlameMichelson ( TRE 0.8.0 (BSD) ) MR-646709E3 X-CRM114-CacheID: sfid-20240311_023446_793110_482550D0 X-CRM114-Status: GOOD ( 17.57 ) X-BeenThere: linux-riscv@lists.infradead.org X-Mailman-Version: 2.1.34 Precedence: list List-Id: List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Sender: "linux-riscv" Errors-To: linux-riscv-bounces+linux-riscv=archiver.kernel.org@lists.infradead.org Refactor the struct modules_array to more general struct ptr_array, which is used to store the pointers. Meanwhiles, introduce the bpf_try_add_ptr(), which checks the existing of the ptr before adding it to the array. Seems it should be moved to another files in "lib", and I'm not sure where to add it now, and let's move it to kernel/bpf/syscall.c for now. Signed-off-by: Menglong Dong --- include/linux/bpf.h | 10 +++++++++ kernel/bpf/syscall.c | 37 +++++++++++++++++++++++++++++++ kernel/trace/bpf_trace.c | 48 ++++++---------------------------------- 3 files changed, 54 insertions(+), 41 deletions(-) diff --git a/include/linux/bpf.h b/include/linux/bpf.h index 0f677fdcfcc7..997765cdf474 100644 --- a/include/linux/bpf.h +++ b/include/linux/bpf.h @@ -304,6 +304,16 @@ struct bpf_map { s64 __percpu *elem_count; }; +struct ptr_array { + void **ptrs; + int cnt; + int cap; +}; + +int bpf_add_ptr(struct ptr_array *arr, void *ptr); +bool bpf_has_ptr(struct ptr_array *arr, struct module *mod); +int bpf_try_add_ptr(struct ptr_array *arr, void *ptr); + static inline const char *btf_field_type_name(enum btf_field_type type) { switch (type) { diff --git a/kernel/bpf/syscall.c b/kernel/bpf/syscall.c index f63f4da4db5e..4f230fd1f8e4 100644 --- a/kernel/bpf/syscall.c +++ b/kernel/bpf/syscall.c @@ -479,6 +479,43 @@ static void bpf_map_release_memcg(struct bpf_map *map) } #endif +int bpf_add_ptr(struct ptr_array *arr, void *ptr) +{ + void **ptrs; + + if (arr->cnt == arr->cap) { + arr->cap = max(16, arr->cap * 3 / 2); + ptrs = krealloc_array(arr->ptrs, arr->cap, sizeof(*ptrs), GFP_KERNEL); + if (!ptrs) + return -ENOMEM; + arr->ptrs = ptrs; + } + + arr->ptrs[arr->cnt] = ptr; + arr->cnt++; + return 0; +} + +bool bpf_has_ptr(struct ptr_array *arr, struct module *mod) +{ + int i; + + for (i = arr->cnt - 1; i >= 0; i--) { + if (arr->ptrs[i] == mod) + return true; + } + return false; +} + +int bpf_try_add_ptr(struct ptr_array *arr, void *ptr) +{ + if (bpf_has_ptr(arr, ptr)) + return -EEXIST; + if (bpf_add_ptr(arr, ptr)) + return -ENOMEM; + return 0; +} + static int btf_field_cmp(const void *a, const void *b) { const struct btf_field *f1 = a, *f2 = b; diff --git a/kernel/trace/bpf_trace.c b/kernel/trace/bpf_trace.c index 241ddf5e3895..791e97a3f8e3 100644 --- a/kernel/trace/bpf_trace.c +++ b/kernel/trace/bpf_trace.c @@ -2873,43 +2873,9 @@ static void symbols_swap_r(void *a, void *b, int size, const void *priv) } } -struct modules_array { - struct module **mods; - int mods_cnt; - int mods_cap; -}; - -static int add_module(struct modules_array *arr, struct module *mod) -{ - struct module **mods; - - if (arr->mods_cnt == arr->mods_cap) { - arr->mods_cap = max(16, arr->mods_cap * 3 / 2); - mods = krealloc_array(arr->mods, arr->mods_cap, sizeof(*mods), GFP_KERNEL); - if (!mods) - return -ENOMEM; - arr->mods = mods; - } - - arr->mods[arr->mods_cnt] = mod; - arr->mods_cnt++; - return 0; -} - -static bool has_module(struct modules_array *arr, struct module *mod) -{ - int i; - - for (i = arr->mods_cnt - 1; i >= 0; i--) { - if (arr->mods[i] == mod) - return true; - } - return false; -} - static int get_modules_for_addrs(struct module ***mods, unsigned long *addrs, u32 addrs_cnt) { - struct modules_array arr = {}; + struct ptr_array arr = {}; u32 i, err = 0; for (i = 0; i < addrs_cnt; i++) { @@ -2918,7 +2884,7 @@ static int get_modules_for_addrs(struct module ***mods, unsigned long *addrs, u3 preempt_disable(); mod = __module_address(addrs[i]); /* Either no module or we it's already stored */ - if (!mod || has_module(&arr, mod)) { + if (!mod || bpf_has_ptr(&arr, mod)) { preempt_enable(); continue; } @@ -2927,7 +2893,7 @@ static int get_modules_for_addrs(struct module ***mods, unsigned long *addrs, u3 preempt_enable(); if (err) break; - err = add_module(&arr, mod); + err = bpf_add_ptr(&arr, mod); if (err) { module_put(mod); break; @@ -2936,14 +2902,14 @@ static int get_modules_for_addrs(struct module ***mods, unsigned long *addrs, u3 /* We return either err < 0 in case of error, ... */ if (err) { - kprobe_multi_put_modules(arr.mods, arr.mods_cnt); - kfree(arr.mods); + kprobe_multi_put_modules((struct module **)arr.ptrs, arr.cnt); + kfree(arr.ptrs); return err; } /* or number of modules found if everything is ok. */ - *mods = arr.mods; - return arr.mods_cnt; + *mods = (struct module **)arr.ptrs; + return arr.cnt; } static int addrs_check_error_injection_list(unsigned long *addrs, u32 cnt) From patchwork Mon Mar 11 09:35:20 2024 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: =?utf-8?b?5qKm6b6Z6JGj?= X-Patchwork-Id: 13588449 X-Patchwork-Delegate: bjorn@kernel.org Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from bombadil.infradead.org (bombadil.infradead.org [198.137.202.133]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (No client certificate requested) by smtp.lore.kernel.org (Postfix) with ESMTPS id 1F8F9C54E58 for ; Mon, 11 Mar 2024 09:35:23 +0000 (UTC) DKIM-Signature: v=1; a=rsa-sha256; q=dns/txt; c=relaxed/relaxed; d=lists.infradead.org; s=bombadil.20210309; h=Sender: Content-Transfer-Encoding:Content-Type:List-Subscribe:List-Help:List-Post: List-Archive:List-Unsubscribe:List-Id:MIME-Version:References:In-Reply-To: Message-Id:Date:Subject:Cc:To:From:Reply-To:Content-ID:Content-Description: Resent-Date:Resent-From:Resent-Sender:Resent-To:Resent-Cc:Resent-Message-ID: List-Owner; bh=auXzvFTZkJOHJkCH5SD+/hE8Hc0GtJ6dov+n5fmY3O0=; b=wxUas/w4J5Blic 1eAQjFW1P8gYkCPV6HeUTeA7hSp28qAxF8Pl4KqVYx0cYxOzKJLtpuOaEzx0Hf/o6A8jBq/taFMPO /aQjDdmEmThjhRrZIiBq4eBeEzLUi5g50kz8JIJh0CwintNk7tu1T7ydAwiPMKG9CZIDGJwICWBaF uECHREZ71oepH6RcLt3LdayhFXFOSkvi0ifokGEoTd+ZX8zTU69BwVWxXrR2YxGizlP+2B9DFMky2 QYHAcT+7iB1BCBy3n+DKYQqvargOmYlmwXvgWtItUPEBMQL80WTF3GTRPaWsy+UvAYT97X99Sy3t9 0sSVTYxpA4H0eLAV4GlA==; Received: from localhost ([::1] helo=bombadil.infradead.org) by bombadil.infradead.org with esmtp (Exim 4.97.1 #2 (Red Hat Linux)) id 1rjc3w-00000000sL4-2OIE; Mon, 11 Mar 2024 09:35:16 +0000 Received: from mail-pf1-x436.google.com ([2607:f8b0:4864:20::436]) by bombadil.infradead.org with esmtps (Exim 4.97.1 #2 (Red Hat Linux)) id 1rjc3Y-00000000ryQ-4C76 for linux-riscv@lists.infradead.org; Mon, 11 Mar 2024 09:35:02 +0000 Received: by mail-pf1-x436.google.com with SMTP id d2e1a72fcca58-6e69a15824fso265121b3a.2 for ; Mon, 11 Mar 2024 02:34:51 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=bytedance.com; s=google; t=1710149691; x=1710754491; darn=lists.infradead.org; h=content-transfer-encoding:mime-version:references:in-reply-to :message-id:date:subject:cc:to:from:from:to:cc:subject:date :message-id:reply-to; bh=vy46p1zwEtiQYPk8lFao3bF3SRbHkbUPWpJ3i7/28CA=; b=lKmVXi+ehwKCDerfFFEi1MJoYCj/e7TKWXcMy7seD9IN0i3gnVX+itw0ZiukQhqjFV G2ROCppleL2mqqSnbMhjR06r6aCfgGYdfLLGcI0wDdYXSOkNTnsi8IzU8r7txViFbavK rJwDgdyhQcOEZKAADIWXxpCAuRn+zpO7APaPJg0oPI90G7eAPjE+VZURfLtbq3dbkhbQ NqQlKiG5JlzUToSum/rJJomdxYYu1Aq3UJcTQsbPWpgoWSxVrmpeHa+GZYckyn1W+zoZ IcDClllJTyPZnhEcg4CRYQW8XzYq7I7BpeYwtjQ+xJ/uwQrQdzuM92/ZJzEsYQxpNqxs fYOA== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20230601; t=1710149691; x=1710754491; h=content-transfer-encoding:mime-version:references:in-reply-to :message-id:date:subject:cc:to:from:x-gm-message-state:from:to:cc :subject:date:message-id:reply-to; bh=vy46p1zwEtiQYPk8lFao3bF3SRbHkbUPWpJ3i7/28CA=; b=LRbZmJjsibYIdbiEij/qt6QIKTVg+kJyFME5BQhPoHDHdL+8z36eiLnxI5Ogj4/ito kCrvE8MEu05WO7F0cUv9WZCcInbc0tZczsvoPPxlmXXLcPfISkfbCLuy1ihgMtMBuCpm Aw/lHJOPMz29NCWo3GqTOeF8tz0GDlyo092YHAnYv6GgXdo32vxDYPEoht1pDRuACiNe HEkSJB72YAihkbmC1WrqPj1xdzghp6eQ2eBJqPosQC9S6HmEGdMTekpSWA7Zg9Je4iN1 dXhELjEGHO+SA6+cHNrnNiW4oO0TKAK5VCXHeDB3XhM0Dar8paqP5iXvCVgDlFbJN6Fu ze9A== X-Forwarded-Encrypted: i=1; AJvYcCWo52pCYmsm4hDps8chojGPIQdN93QHXwNfjBtTuiv6L1qcJ8TnahfLlrUiD7i0VvZkSxF04YLFloGbMiOeTDpA7t6jU+1YuuXwWZeV+JuN X-Gm-Message-State: AOJu0YxQ59QtCOJwGW575Y/jA9QB58twJZPDPJU7US3XMiJTG3Zzpmkw wEIDTxXf2w8vFz/SIS9Z1FOcK8MVum16YE/u+UFxS+hI8BanyKOcRVIlcoGp4YU= X-Google-Smtp-Source: AGHT+IFjBcGV00k1T/PRAurGJKimsKfvd3nWSQWSksUKxJJH0GZ/AViWHcHFMZ3KmKpuqEbt5SPRAA== X-Received: by 2002:a05:6a20:96d1:b0:1a1:86e9:ef9b with SMTP id hq17-20020a056a2096d100b001a186e9ef9bmr3095077pzc.18.1710149691293; Mon, 11 Mar 2024 02:34:51 -0700 (PDT) Received: from localhost.localdomain ([43.129.25.208]) by smtp.gmail.com with ESMTPSA id h9-20020a170902f7c900b001dcad9cbf8bsm4253365plw.239.2024.03.11.02.34.43 (version=TLS1_3 cipher=TLS_AES_256_GCM_SHA384 bits=256/256); Mon, 11 Mar 2024 02:34:51 -0700 (PDT) From: Menglong Dong To: andrii@kernel.org Cc: ast@kernel.org, daniel@iogearbox.net, martin.lau@linux.dev, eddyz87@gmail.com, song@kernel.org, yonghong.song@linux.dev, john.fastabend@gmail.com, kpsingh@kernel.org, sdf@google.com, haoluo@google.com, jolsa@kernel.org, agordeev@linux.ibm.com, borntraeger@linux.ibm.com, svens@linux.ibm.com, davem@davemloft.net, dsahern@kernel.org, dave.hansen@linux.intel.com, x86@kernel.org, rostedt@goodmis.org, mathieu.desnoyers@efficios.com, quentin@isovalent.com, bpf@vger.kernel.org, linux-arm-kernel@lists.infradead.org, linux-kernel@vger.kernel.org, linux-riscv@lists.infradead.org, linux-s390@vger.kernel.org, netdev@vger.kernel.org, linux-trace-kernel@vger.kernel.org, linux-kselftest@vger.kernel.org, linux-stm32@st-md-mailman.stormreply.com, Menglong Dong Subject: [PATCH bpf-next v2 3/9] bpf: trampoline: introduce struct bpf_tramp_link_conn Date: Mon, 11 Mar 2024 17:35:20 +0800 Message-Id: <20240311093526.1010158-4-dongmenglong.8@bytedance.com> X-Mailer: git-send-email 2.39.2 In-Reply-To: <20240311093526.1010158-1-dongmenglong.8@bytedance.com> References: <20240311093526.1010158-1-dongmenglong.8@bytedance.com> MIME-Version: 1.0 X-CRM114-Version: 20100106-BlameMichelson ( TRE 0.8.0 (BSD) ) MR-646709E3 X-CRM114-CacheID: sfid-20240311_023453_229862_826F90DC X-CRM114-Status: GOOD ( 18.79 ) X-BeenThere: linux-riscv@lists.infradead.org X-Mailman-Version: 2.1.34 Precedence: list List-Id: List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Sender: "linux-riscv" Errors-To: linux-riscv-bounces+linux-riscv=archiver.kernel.org@lists.infradead.org For now, bpf_tramp_link is added to the hash list of tr->progs_hlist when attaching. This means that bpf_link and trampoline is one-to-one, and is not friendly to the multi-link trampoline that we commit in the following patches. Therefore, now we introduce the struct bpf_tramp_link_conn to be the bridge between bpf_tramp_link and trampoline. And we also chang the type of links in struct bpf_tramp_links to struct bpf_tramp_link_conn. Signed-off-by: Menglong Dong --- arch/arm64/net/bpf_jit_comp.c | 4 ++-- arch/riscv/net/bpf_jit_comp64.c | 4 ++-- arch/s390/net/bpf_jit_comp.c | 4 ++-- arch/x86/net/bpf_jit_comp.c | 4 ++-- include/linux/bpf.h | 12 +++++++--- kernel/bpf/bpf_struct_ops.c | 3 ++- kernel/bpf/syscall.c | 3 ++- kernel/bpf/trampoline.c | 42 +++++++++++++++++---------------- net/bpf/bpf_dummy_struct_ops.c | 1 + 9 files changed, 44 insertions(+), 33 deletions(-) diff --git a/arch/arm64/net/bpf_jit_comp.c b/arch/arm64/net/bpf_jit_comp.c index c5b461dda438..b6f7d8a6d372 100644 --- a/arch/arm64/net/bpf_jit_comp.c +++ b/arch/arm64/net/bpf_jit_comp.c @@ -1810,14 +1810,14 @@ bool bpf_jit_supports_subprog_tailcalls(void) return true; } -static void invoke_bpf_prog(struct jit_ctx *ctx, struct bpf_tramp_link *l, +static void invoke_bpf_prog(struct jit_ctx *ctx, struct bpf_tramp_link_conn *l, int args_off, int retval_off, int run_ctx_off, bool save_ret) { __le32 *branch; u64 enter_prog; u64 exit_prog; - struct bpf_prog *p = l->link.prog; + struct bpf_prog *p = l->link->prog; int cookie_off = offsetof(struct bpf_tramp_run_ctx, bpf_cookie); enter_prog = (u64)bpf_trampoline_enter(p); diff --git a/arch/riscv/net/bpf_jit_comp64.c b/arch/riscv/net/bpf_jit_comp64.c index aac190085472..c147053001db 100644 --- a/arch/riscv/net/bpf_jit_comp64.c +++ b/arch/riscv/net/bpf_jit_comp64.c @@ -700,11 +700,11 @@ static void restore_args(int nregs, int args_off, struct rv_jit_context *ctx) } } -static int invoke_bpf_prog(struct bpf_tramp_link *l, int args_off, int retval_off, +static int invoke_bpf_prog(struct bpf_tramp_link_conn *l, int args_off, int retval_off, int run_ctx_off, bool save_ret, struct rv_jit_context *ctx) { int ret, branch_off; - struct bpf_prog *p = l->link.prog; + struct bpf_prog *p = l->link->prog; int cookie_off = offsetof(struct bpf_tramp_run_ctx, bpf_cookie); if (l->cookie) { diff --git a/arch/s390/net/bpf_jit_comp.c b/arch/s390/net/bpf_jit_comp.c index b418333bb086..177efbc1b5ec 100644 --- a/arch/s390/net/bpf_jit_comp.c +++ b/arch/s390/net/bpf_jit_comp.c @@ -2243,12 +2243,12 @@ static void load_imm64(struct bpf_jit *jit, int dst_reg, u64 val) static int invoke_bpf_prog(struct bpf_tramp_jit *tjit, const struct btf_func_model *m, - struct bpf_tramp_link *tlink, bool save_ret) + struct bpf_tramp_link_conn *tlink, bool save_ret) { struct bpf_jit *jit = &tjit->common; int cookie_off = tjit->run_ctx_off + offsetof(struct bpf_tramp_run_ctx, bpf_cookie); - struct bpf_prog *p = tlink->link.prog; + struct bpf_prog *p = tlink->link->prog; int patch; /* diff --git a/arch/x86/net/bpf_jit_comp.c b/arch/x86/net/bpf_jit_comp.c index e1390d1e331b..e7f9f987770d 100644 --- a/arch/x86/net/bpf_jit_comp.c +++ b/arch/x86/net/bpf_jit_comp.c @@ -2261,14 +2261,14 @@ static void restore_regs(const struct btf_func_model *m, u8 **prog, } static int invoke_bpf_prog(const struct btf_func_model *m, u8 **pprog, - struct bpf_tramp_link *l, int stack_size, + struct bpf_tramp_link_conn *l, int stack_size, int run_ctx_off, bool save_ret, void *image, void *rw_image) { u8 *prog = *pprog; u8 *jmp_insn; int ctx_cookie_off = offsetof(struct bpf_tramp_run_ctx, bpf_cookie); - struct bpf_prog *p = l->link.prog; + struct bpf_prog *p = l->link->prog; u64 cookie = l->cookie; /* mov rdi, cookie */ diff --git a/include/linux/bpf.h b/include/linux/bpf.h index 997765cdf474..2b5cd6100fc4 100644 --- a/include/linux/bpf.h +++ b/include/linux/bpf.h @@ -56,6 +56,7 @@ struct bpf_token; struct user_namespace; struct super_block; struct inode; +struct bpf_tramp_link; extern struct idr btf_idr; extern spinlock_t btf_idr_lock; @@ -1090,7 +1091,7 @@ enum { }; struct bpf_tramp_links { - struct bpf_tramp_link *links[BPF_MAX_TRAMP_LINKS]; + struct bpf_tramp_link_conn *links[BPF_MAX_TRAMP_LINKS]; int nr_links; }; @@ -1597,12 +1598,17 @@ struct bpf_link_ops { struct bpf_map *old_map); }; -struct bpf_tramp_link { - struct bpf_link link; +struct bpf_tramp_link_conn { + struct bpf_link *link; struct hlist_node tramp_hlist; u64 cookie; }; +struct bpf_tramp_link { + struct bpf_link link; + struct bpf_tramp_link_conn conn; +}; + struct bpf_shim_tramp_link { struct bpf_tramp_link link; struct bpf_trampoline *trampoline; diff --git a/kernel/bpf/bpf_struct_ops.c b/kernel/bpf/bpf_struct_ops.c index 43356faaa057..4fbe2faa80a8 100644 --- a/kernel/bpf/bpf_struct_ops.c +++ b/kernel/bpf/bpf_struct_ops.c @@ -549,7 +549,7 @@ int bpf_struct_ops_prepare_trampoline(struct bpf_tramp_links *tlinks, void *image = *_image; int size; - tlinks[BPF_TRAMP_FENTRY].links[0] = link; + tlinks[BPF_TRAMP_FENTRY].links[0] = &link->conn; tlinks[BPF_TRAMP_FENTRY].nr_links = 1; if (model->ret_size > 0) @@ -710,6 +710,7 @@ static long bpf_struct_ops_map_update_elem(struct bpf_map *map, void *key, err = -ENOMEM; goto reset_unlock; } + link->conn.link = &link->link; bpf_link_init(&link->link, BPF_LINK_TYPE_STRUCT_OPS, &bpf_struct_ops_link_lops, prog); st_map->links[i] = &link->link; diff --git a/kernel/bpf/syscall.c b/kernel/bpf/syscall.c index 4f230fd1f8e4..d1cd645ef9ac 100644 --- a/kernel/bpf/syscall.c +++ b/kernel/bpf/syscall.c @@ -3339,6 +3339,7 @@ static int bpf_tracing_prog_attach(struct bpf_prog *prog, } link = kzalloc(sizeof(*link), GFP_USER); + link->link.conn.link = &link->link.link; if (!link) { err = -ENOMEM; goto out_put_prog; @@ -3346,7 +3347,7 @@ static int bpf_tracing_prog_attach(struct bpf_prog *prog, bpf_link_init(&link->link.link, BPF_LINK_TYPE_TRACING, &bpf_tracing_link_lops, prog); link->attach_type = prog->expected_attach_type; - link->link.cookie = bpf_cookie; + link->link.conn.cookie = bpf_cookie; mutex_lock(&prog->aux->dst_mutex); diff --git a/kernel/bpf/trampoline.c b/kernel/bpf/trampoline.c index d382f5ebe06c..cf9b84f785f3 100644 --- a/kernel/bpf/trampoline.c +++ b/kernel/bpf/trampoline.c @@ -228,9 +228,9 @@ static int register_fentry(struct bpf_trampoline *tr, void *new_addr) static struct bpf_tramp_links * bpf_trampoline_get_progs(const struct bpf_trampoline *tr, int *total, bool *ip_arg) { - struct bpf_tramp_link *link; + struct bpf_tramp_link_conn *link_conn; + struct bpf_tramp_link_conn **links; struct bpf_tramp_links *tlinks; - struct bpf_tramp_link **links; int kind; *total = 0; @@ -243,9 +243,9 @@ bpf_trampoline_get_progs(const struct bpf_trampoline *tr, int *total, bool *ip_a *total += tr->progs_cnt[kind]; links = tlinks[kind].links; - hlist_for_each_entry(link, &tr->progs_hlist[kind], tramp_hlist) { - *ip_arg |= link->link.prog->call_get_func_ip; - *links++ = link; + hlist_for_each_entry(link_conn, &tr->progs_hlist[kind], tramp_hlist) { + *ip_arg |= link_conn->link->prog->call_get_func_ip; + *links++ = link_conn; } } return tlinks; @@ -521,14 +521,14 @@ static enum bpf_tramp_prog_type bpf_attach_type_to_tramp(struct bpf_prog *prog) } } -static int __bpf_trampoline_link_prog(struct bpf_tramp_link *link, struct bpf_trampoline *tr) +static int __bpf_trampoline_link_prog(struct bpf_tramp_link_conn *link, struct bpf_trampoline *tr) { enum bpf_tramp_prog_type kind; - struct bpf_tramp_link *link_exiting; + struct bpf_tramp_link_conn *link_exiting; int err = 0; int cnt = 0, i; - kind = bpf_attach_type_to_tramp(link->link.prog); + kind = bpf_attach_type_to_tramp(link->link->prog); if (tr->extension_prog) /* cannot attach fentry/fexit if extension prog is attached. * cannot overwrite extension prog either. @@ -542,9 +542,9 @@ static int __bpf_trampoline_link_prog(struct bpf_tramp_link *link, struct bpf_tr /* Cannot attach extension if fentry/fexit are in use. */ if (cnt) return -EBUSY; - tr->extension_prog = link->link.prog; + tr->extension_prog = link->link->prog; return bpf_arch_text_poke(tr->func.addr, BPF_MOD_JUMP, NULL, - link->link.prog->bpf_func); + link->link->prog->bpf_func); } if (cnt >= BPF_MAX_TRAMP_LINKS) return -E2BIG; @@ -552,7 +552,7 @@ static int __bpf_trampoline_link_prog(struct bpf_tramp_link *link, struct bpf_tr /* prog already linked */ return -EBUSY; hlist_for_each_entry(link_exiting, &tr->progs_hlist[kind], tramp_hlist) { - if (link_exiting->link.prog != link->link.prog) + if (link_exiting->link->prog != link->link->prog) continue; /* prog already linked */ return -EBUSY; @@ -573,17 +573,17 @@ int bpf_trampoline_link_prog(struct bpf_tramp_link *link, struct bpf_trampoline int err; mutex_lock(&tr->mutex); - err = __bpf_trampoline_link_prog(link, tr); + err = __bpf_trampoline_link_prog(&link->conn, tr); mutex_unlock(&tr->mutex); return err; } -static int __bpf_trampoline_unlink_prog(struct bpf_tramp_link *link, struct bpf_trampoline *tr) +static int __bpf_trampoline_unlink_prog(struct bpf_tramp_link_conn *link, struct bpf_trampoline *tr) { enum bpf_tramp_prog_type kind; int err; - kind = bpf_attach_type_to_tramp(link->link.prog); + kind = bpf_attach_type_to_tramp(link->link->prog); if (kind == BPF_TRAMP_REPLACE) { WARN_ON_ONCE(!tr->extension_prog); err = bpf_arch_text_poke(tr->func.addr, BPF_MOD_JUMP, @@ -602,7 +602,7 @@ int bpf_trampoline_unlink_prog(struct bpf_tramp_link *link, struct bpf_trampolin int err; mutex_lock(&tr->mutex); - err = __bpf_trampoline_unlink_prog(link, tr); + err = __bpf_trampoline_unlink_prog(&link->conn, tr); mutex_unlock(&tr->mutex); return err; } @@ -645,6 +645,7 @@ static struct bpf_shim_tramp_link *cgroup_shim_alloc(const struct bpf_prog *prog if (!shim_link) return NULL; + shim_link->link.conn.link = &shim_link->link.link; p = bpf_prog_alloc(1, 0); if (!p) { kfree(shim_link); @@ -672,15 +673,16 @@ static struct bpf_shim_tramp_link *cgroup_shim_alloc(const struct bpf_prog *prog static struct bpf_shim_tramp_link *cgroup_shim_find(struct bpf_trampoline *tr, bpf_func_t bpf_func) { - struct bpf_tramp_link *link; + struct bpf_tramp_link_conn *link_conn; int kind; for (kind = 0; kind < BPF_TRAMP_MAX; kind++) { - hlist_for_each_entry(link, &tr->progs_hlist[kind], tramp_hlist) { - struct bpf_prog *p = link->link.prog; + hlist_for_each_entry(link_conn, &tr->progs_hlist[kind], tramp_hlist) { + struct bpf_prog *p = link_conn->link->prog; if (p->bpf_func == bpf_func) - return container_of(link, struct bpf_shim_tramp_link, link); + return container_of((struct bpf_tramp_link *)link_conn->link, + struct bpf_shim_tramp_link, link); } } @@ -731,7 +733,7 @@ int bpf_trampoline_link_cgroup_shim(struct bpf_prog *prog, goto err; } - err = __bpf_trampoline_link_prog(&shim_link->link, tr); + err = __bpf_trampoline_link_prog(&shim_link->link.conn, tr); if (err) goto err; diff --git a/net/bpf/bpf_dummy_struct_ops.c b/net/bpf/bpf_dummy_struct_ops.c index 1b5f812e6972..35a2cf60eef6 100644 --- a/net/bpf/bpf_dummy_struct_ops.c +++ b/net/bpf/bpf_dummy_struct_ops.c @@ -120,6 +120,7 @@ int bpf_struct_ops_test_run(struct bpf_prog *prog, const union bpf_attr *kattr, err = -ENOMEM; goto out; } + link->conn.link = &link->link; /* prog doesn't take the ownership of the reference from caller */ bpf_prog_inc(prog); bpf_link_init(&link->link, BPF_LINK_TYPE_STRUCT_OPS, &bpf_struct_ops_link_lops, prog); From patchwork Mon Mar 11 09:35:21 2024 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: =?utf-8?b?5qKm6b6Z6JGj?= X-Patchwork-Id: 13588450 X-Patchwork-Delegate: bjorn@kernel.org Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from bombadil.infradead.org (bombadil.infradead.org [198.137.202.133]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (No client certificate requested) by smtp.lore.kernel.org (Postfix) with ESMTPS id E69A5C54E58 for ; Mon, 11 Mar 2024 09:35:28 +0000 (UTC) DKIM-Signature: v=1; a=rsa-sha256; q=dns/txt; c=relaxed/relaxed; d=lists.infradead.org; s=bombadil.20210309; h=Sender: Content-Transfer-Encoding:Content-Type:List-Subscribe:List-Help:List-Post: List-Archive:List-Unsubscribe:List-Id:MIME-Version:References:In-Reply-To: Message-Id:Date:Subject:Cc:To:From:Reply-To:Content-ID:Content-Description: Resent-Date:Resent-From:Resent-Sender:Resent-To:Resent-Cc:Resent-Message-ID: List-Owner; bh=UOtcksJNeZ7CxwDY36HRbbAMWv5pT+zbYC67aQ0T2VQ=; b=C3ry6I2nx94TwH y+EeUgTd8QKdQo2ZXXqkE47le6KwBUfwSdYrPKr54ZMTl9Y/QRZImQmkvbhpvYPWnhNm17ERkk0FF ijhoe1V2zp0Faycw4v96plQ2IhK7mAyo6EcW5nax5chjMWBOa8an9GI1bThOrxzf/ZXeO4lNCANr9 DO+x9w1c59BxF+jbFUiOtWv0RXi6KDspq2Nx8JieCmJhP4r3yN2aic0DdxA4cWXwEJtaQXjjH8lJk glF1qT2e0pYkJmXlwjUp1Uhn5zsVamHlYzEmCEAuA/b4FT51p2TxHm0Ev2E7c+vBWXABkshRqvZQR O3YdvKgzFdCzL9CwcUjg==; Received: from localhost ([::1] helo=bombadil.infradead.org) by bombadil.infradead.org with esmtp (Exim 4.97.1 #2 (Red Hat Linux)) id 1rjc43-00000000sR8-3b3A; Mon, 11 Mar 2024 09:35:23 +0000 Received: from mail-pg1-x52e.google.com ([2607:f8b0:4864:20::52e]) by bombadil.infradead.org with esmtps (Exim 4.97.1 #2 (Red Hat Linux)) id 1rjc3l-00000000s66-2h0L for linux-riscv@lists.infradead.org; Mon, 11 Mar 2024 09:35:12 +0000 Received: by mail-pg1-x52e.google.com with SMTP id 41be03b00d2f7-5e42b4bbfa4so1925016a12.1 for ; Mon, 11 Mar 2024 02:34:59 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=bytedance.com; s=google; t=1710149699; x=1710754499; darn=lists.infradead.org; h=content-transfer-encoding:mime-version:references:in-reply-to :message-id:date:subject:cc:to:from:from:to:cc:subject:date :message-id:reply-to; bh=mQHWXqakG53bwp/DHGcig31BilBtYFxqqjfUWdtPR6g=; b=NvQBc+QqknuJzih+/np89Ob81zaYI2X8JVNypKjg4c1iuH3j9sH0FkWq0VR64tqFRY wZ8xtQNY6TBLn9hbothAm6rB8mpfIZL7sqbx4yK4VRFI8qMgk1vX9QMzsSxMxgdj5ziw KzTRIroyK3tJzPqKY5LyVSBxVM6xOslrTY3ay2b7069qzu4UrjtUnFxEDZCugzhGgBZ5 4eOInzu8Y36Ubr2RVXDLNtQCvl1De61qn7GBPp3eff+5lTMGhYxPEXQIzvaoxr5hLuro ZiMJonQGvw6NkLS+OFZKJMOrc/MkfYTka1/0Fc9b9GjfsJNhAGlq2eLSCwnu0BcsEPdA EyKg== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20230601; t=1710149699; x=1710754499; h=content-transfer-encoding:mime-version:references:in-reply-to :message-id:date:subject:cc:to:from:x-gm-message-state:from:to:cc :subject:date:message-id:reply-to; bh=mQHWXqakG53bwp/DHGcig31BilBtYFxqqjfUWdtPR6g=; b=g8WB+DYmCpUHx4JY/8d9Vwts2zE8e5foTzBCZpt264ru/8RhzsTEt1nMpu137A+rgm ipu1f/DTzFDqzSzzO25/PfOmsFRUTAhk98BJ58p6r6eu4XTPKzOcKDly8k70RhbMqikK GHydNxW8yYzpVlIEpfdF18lm3F/JTT1mhy2HlnlPR33PA+pi8ht3izzWShH3Yfm7v879 /eyZmgYBIWLOK8ZrmBdrVqtR7FhLulLAlykeev7bSNQ9w6TrTZtr9eCMbQOmSbNeZSHk Txy5SUXH9an0/RUUcIW+lqIKbQajg7SxEkL5/SBSbx3ztLgaukFBPHAAJ7xpZ/sPy6D7 L5lA== X-Forwarded-Encrypted: i=1; AJvYcCWxjl6xQth2Bvu9HpqVe9n+AngqyCMG13hi5l+xuaNI3rDtLsYNHgBHczIgk7IlqMLHJ0pISelm6uwxBSvoGP0caxsxF7xzsJCAHdIJovdQ X-Gm-Message-State: AOJu0Yy3tmB/OqV/WLnjEOYoDw7LvMcaZMSx1g9DKQDSJs+4A35rAEFg RCj5gbsqYLfHCOBRJ9ziq5m1uxIL28f7cirI2WiMqwN2jGQBOithkCc/OOmeXzg= X-Google-Smtp-Source: AGHT+IHwrlcPzEPWwJhB9wbDVFgwEUaTXK8PIDDimv1XZruHWD6zTTqAqef2QwBY7Qc+7bmyEVvx+w== X-Received: by 2002:a05:6a20:6f8f:b0:1a3:1d7e:342b with SMTP id gv15-20020a056a206f8f00b001a31d7e342bmr631985pzb.48.1710149698990; Mon, 11 Mar 2024 02:34:58 -0700 (PDT) Received: from localhost.localdomain ([43.129.25.208]) by smtp.gmail.com with ESMTPSA id h9-20020a170902f7c900b001dcad9cbf8bsm4253365plw.239.2024.03.11.02.34.51 (version=TLS1_3 cipher=TLS_AES_256_GCM_SHA384 bits=256/256); Mon, 11 Mar 2024 02:34:58 -0700 (PDT) From: Menglong Dong To: andrii@kernel.org Cc: ast@kernel.org, daniel@iogearbox.net, martin.lau@linux.dev, eddyz87@gmail.com, song@kernel.org, yonghong.song@linux.dev, john.fastabend@gmail.com, kpsingh@kernel.org, sdf@google.com, haoluo@google.com, jolsa@kernel.org, agordeev@linux.ibm.com, borntraeger@linux.ibm.com, svens@linux.ibm.com, davem@davemloft.net, dsahern@kernel.org, dave.hansen@linux.intel.com, x86@kernel.org, rostedt@goodmis.org, mathieu.desnoyers@efficios.com, quentin@isovalent.com, bpf@vger.kernel.org, linux-arm-kernel@lists.infradead.org, linux-kernel@vger.kernel.org, linux-riscv@lists.infradead.org, linux-s390@vger.kernel.org, netdev@vger.kernel.org, linux-trace-kernel@vger.kernel.org, linux-kselftest@vger.kernel.org, linux-stm32@st-md-mailman.stormreply.com, Menglong Dong Subject: [PATCH bpf-next v2 4/9] bpf: trampoline: introduce bpf_tramp_multi_link Date: Mon, 11 Mar 2024 17:35:21 +0800 Message-Id: <20240311093526.1010158-5-dongmenglong.8@bytedance.com> X-Mailer: git-send-email 2.39.2 In-Reply-To: <20240311093526.1010158-1-dongmenglong.8@bytedance.com> References: <20240311093526.1010158-1-dongmenglong.8@bytedance.com> MIME-Version: 1.0 X-CRM114-Version: 20100106-BlameMichelson ( TRE 0.8.0 (BSD) ) MR-646709E3 X-CRM114-CacheID: sfid-20240311_023506_309708_E4E51E58 X-CRM114-Status: GOOD ( 12.20 ) X-BeenThere: linux-riscv@lists.infradead.org X-Mailman-Version: 2.1.34 Precedence: list List-Id: List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Sender: "linux-riscv" Errors-To: linux-riscv-bounces+linux-riscv=archiver.kernel.org@lists.infradead.org Introduce the struct bpf_tramp_multi_link, which is used to attach a bpf_link to multi trampoline. Meanwhile, introduce corresponding function bpf_trampoline_multi_{link,unlink}_prog. Signed-off-by: Menglong Dong --- include/linux/bpf.h | 14 ++++++++++++ kernel/bpf/trampoline.c | 47 +++++++++++++++++++++++++++++++++++++++++ 2 files changed, 61 insertions(+) diff --git a/include/linux/bpf.h b/include/linux/bpf.h index 2b5cd6100fc4..4e8f17d9f022 100644 --- a/include/linux/bpf.h +++ b/include/linux/bpf.h @@ -57,6 +57,7 @@ struct user_namespace; struct super_block; struct inode; struct bpf_tramp_link; +struct bpf_tramp_multi_link; extern struct idr btf_idr; extern spinlock_t btf_idr_lock; @@ -1282,6 +1283,8 @@ struct bpf_trampoline *bpf_trampoline_get(u64 key, struct bpf_attach_target_info *tgt_info); void bpf_trampoline_put(struct bpf_trampoline *tr); int arch_prepare_bpf_dispatcher(void *image, void *buf, s64 *funcs, int num_funcs); +int bpf_trampoline_multi_link_prog(struct bpf_tramp_multi_link *link); +int bpf_trampoline_multi_unlink_prog(struct bpf_tramp_multi_link *link); /* * When the architecture supports STATIC_CALL replace the bpf_dispatcher_fn @@ -1614,6 +1617,17 @@ struct bpf_shim_tramp_link { struct bpf_trampoline *trampoline; }; +struct bpf_tramp_multi_link_entry { + struct bpf_trampoline *trampoline; + struct bpf_tramp_link_conn conn; +}; + +struct bpf_tramp_multi_link { + struct bpf_link link; + u32 cnt; + struct bpf_tramp_multi_link_entry *entries; +}; + struct bpf_tracing_link { struct bpf_tramp_link link; enum bpf_attach_type attach_type; diff --git a/kernel/bpf/trampoline.c b/kernel/bpf/trampoline.c index cf9b84f785f3..2167aa3fe583 100644 --- a/kernel/bpf/trampoline.c +++ b/kernel/bpf/trampoline.c @@ -607,6 +607,53 @@ int bpf_trampoline_unlink_prog(struct bpf_tramp_link *link, struct bpf_trampolin return err; } +static int __bpf_trampoline_multi_unlink_prog(struct bpf_tramp_multi_link *link, + u32 cnt) +{ + struct bpf_tramp_multi_link_entry *entry; + struct bpf_trampoline *tr; + int err = 0, i; + + for (i = 0; i < cnt; i++) { + entry = &link->entries[i]; + tr = entry->trampoline; + mutex_lock(&tr->mutex); + err = __bpf_trampoline_unlink_prog(&entry->conn, + entry->trampoline); + mutex_unlock(&tr->mutex); + if (err) + break; + } + return err; +} + +int bpf_trampoline_multi_unlink_prog(struct bpf_tramp_multi_link *link) +{ + return __bpf_trampoline_multi_unlink_prog(link, link->cnt); +} + +int bpf_trampoline_multi_link_prog(struct bpf_tramp_multi_link *link) +{ + struct bpf_tramp_multi_link_entry *entry; + struct bpf_trampoline *tr; + int err = 0, i; + + for (i = 0; i < link->cnt; i++) { + entry = &link->entries[i]; + tr = entry->trampoline; + mutex_lock(&tr->mutex); + err = __bpf_trampoline_link_prog(&entry->conn, tr); + mutex_unlock(&tr->mutex); + if (err) + goto unlink; + } + + return 0; +unlink: + __bpf_trampoline_multi_unlink_prog(link, i); + return err; +} + #if defined(CONFIG_CGROUP_BPF) && defined(CONFIG_BPF_LSM) static void bpf_shim_tramp_link_release(struct bpf_link *link) { From patchwork Mon Mar 11 09:35:22 2024 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: =?utf-8?b?5qKm6b6Z6JGj?= X-Patchwork-Id: 13588451 X-Patchwork-Delegate: bjorn@kernel.org Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from bombadil.infradead.org (bombadil.infradead.org [198.137.202.133]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (No client certificate requested) by smtp.lore.kernel.org (Postfix) with ESMTPS id 76651C54E58 for ; Mon, 11 Mar 2024 09:35:35 +0000 (UTC) DKIM-Signature: v=1; a=rsa-sha256; q=dns/txt; c=relaxed/relaxed; d=lists.infradead.org; s=bombadil.20210309; h=Sender: Content-Transfer-Encoding:Content-Type:List-Subscribe:List-Help:List-Post: List-Archive:List-Unsubscribe:List-Id:MIME-Version:References:In-Reply-To: Message-Id:Date:Subject:Cc:To:From:Reply-To:Content-ID:Content-Description: Resent-Date:Resent-From:Resent-Sender:Resent-To:Resent-Cc:Resent-Message-ID: List-Owner; bh=Oj87StymKxXMVkI1s1+xrihK1x40J8EzpTpksLFYNwY=; b=HHwaZgvSIkdt4e jCSJ8b8JSLr5LQsiPlo4f5JDT8k7+zhHZ9JL4U5Txbc2jTgmsj6YMMDAasw+2tBFY+vhS99LRp2N/ peM+ixF+5n7GCpKRFg/oGATFwT0ZTgVXyXl1JQM/o/LUZt5+h/E0AMElbh8pqhnIvHkjptML4mqaV WW8jxa8fYdr/2a++H0cPmtzerf3+sv2Ynu8SHob8TzHwln74cCLP4zmtakv1h9mp9pIFXl2WhzbXG PwlHyWegZwUdkhAjfITu40/S940/LTofOBLfEXSy4sgw6HSHkmNa9soYBm28A2CDgsUqjrOlAaYGS QfPIajU3YUbwyoXLsJGg==; Received: from localhost ([::1] helo=bombadil.infradead.org) by bombadil.infradead.org with esmtp (Exim 4.97.1 #2 (Red Hat Linux)) id 1rjc47-00000000sUZ-3n7B; Mon, 11 Mar 2024 09:35:28 +0000 Received: from mail-pl1-x62e.google.com ([2607:f8b0:4864:20::62e]) by bombadil.infradead.org with esmtps (Exim 4.97.1 #2 (Red Hat Linux)) id 1rjc3q-00000000sCR-0RG6 for linux-riscv@lists.infradead.org; Mon, 11 Mar 2024 09:35:13 +0000 Received: by mail-pl1-x62e.google.com with SMTP id d9443c01a7336-1dd9568fc51so7333495ad.2 for ; Mon, 11 Mar 2024 02:35:07 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=bytedance.com; s=google; t=1710149706; x=1710754506; darn=lists.infradead.org; h=content-transfer-encoding:mime-version:references:in-reply-to :message-id:date:subject:cc:to:from:from:to:cc:subject:date :message-id:reply-to; bh=nrO/qg8uwgz9YPMWghckYtP0xPd/PM4sF9e4mk1PfXk=; b=Rdw312IYdAtKnnVTSrEZYgvc041uoEsP3bainYonpj2lJnIwImtNdcnbngOrUBgAhN okcChRHaDn7ybKS80quB3w68Q2888Zgafp+zPKhelyYUrEv1o6fAuPLbGdqUIaHioQbQ 2OjQTl17KsmGGNbfTBOqSm3zoxilv61mkZXWcjtgnvXrNqp/8aOggKlsySYJCU0o3naD 4wVmppKv/anrtWHWnZiN6UZZOjE1GBQzsMIVZ5wnm+UinckHyc+mR6AXezYbpdrOeFgS L1pUHvKCiA0eQ3vZZLQWSuPsbIlvJoiCL02oXPG2vi5DMkP0UqriKjIwqiiUkE+C0X3x PHIg== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20230601; t=1710149706; x=1710754506; h=content-transfer-encoding:mime-version:references:in-reply-to :message-id:date:subject:cc:to:from:x-gm-message-state:from:to:cc :subject:date:message-id:reply-to; bh=nrO/qg8uwgz9YPMWghckYtP0xPd/PM4sF9e4mk1PfXk=; b=xQXkIxFTfy1AU9asR/O5vxcBeubT4c5WJOJ9nLK8PMRgibMASuMl6TMZLcRf5t0+FE oJrisGK/V4LE5K02Hg28//TIbAemzfPUN3vhr8KvMazbexoHrbN1go5yl9Cdsgj7Gi2s pGxh7Kyo/LY3inEI2BwDeyIGvsOmgBc8/Yf6P30yuK0eVGj9OZrPcBUVw6XQwXBDejls B1xDIJKyFBUky7cJl2ef4ZXOWBD5aZOqrIg9MkUjQ4VhHUi0B3U31+k8q0uYloN8MxyU aCRFfGGtdksvHxRjm7cT2i4014Uwy7KnZInBN1ffvlhmjY2d3GCU2PkKhygg8CihiNK9 UzHw== X-Forwarded-Encrypted: i=1; AJvYcCUwdLVxMzipSX5qa5+cb10gNif3ZPyNkqIniaAbGOogrLJAeKKSQn9XtA/knQvwoYZ2y964721GSbZQLJW8lgKmZzClpAtSQ5mTmR2540N6 X-Gm-Message-State: AOJu0YyJJgKsrXnylwV2aGDTTCjBm/1JMxUBzXsaBJTnoxj97HIOpQLc 1qBeq2t+UrmigX7l/HP1gzVo5Wg9NULXyPNy1le9lPgz29JCGFpl5W0eJuR0wWA= X-Google-Smtp-Source: AGHT+IGdSAmTRkYfDKtyWphJJzXlBtH2IoijdRXhZZCIJP7eVn99sHsdiqYvOsYl4WPd5euBmexj4g== X-Received: by 2002:a17:902:bcc6:b0:1dc:f803:85b3 with SMTP id o6-20020a170902bcc600b001dcf80385b3mr4689221pls.43.1710149706575; Mon, 11 Mar 2024 02:35:06 -0700 (PDT) Received: from localhost.localdomain ([43.129.25.208]) by smtp.gmail.com with ESMTPSA id h9-20020a170902f7c900b001dcad9cbf8bsm4253365plw.239.2024.03.11.02.34.59 (version=TLS1_3 cipher=TLS_AES_256_GCM_SHA384 bits=256/256); Mon, 11 Mar 2024 02:35:06 -0700 (PDT) From: Menglong Dong To: andrii@kernel.org Cc: ast@kernel.org, daniel@iogearbox.net, martin.lau@linux.dev, eddyz87@gmail.com, song@kernel.org, yonghong.song@linux.dev, john.fastabend@gmail.com, kpsingh@kernel.org, sdf@google.com, haoluo@google.com, jolsa@kernel.org, agordeev@linux.ibm.com, borntraeger@linux.ibm.com, svens@linux.ibm.com, davem@davemloft.net, dsahern@kernel.org, dave.hansen@linux.intel.com, x86@kernel.org, rostedt@goodmis.org, mathieu.desnoyers@efficios.com, quentin@isovalent.com, bpf@vger.kernel.org, linux-arm-kernel@lists.infradead.org, linux-kernel@vger.kernel.org, linux-riscv@lists.infradead.org, linux-s390@vger.kernel.org, netdev@vger.kernel.org, linux-trace-kernel@vger.kernel.org, linux-kselftest@vger.kernel.org, linux-stm32@st-md-mailman.stormreply.com, Menglong Dong Subject: [PATCH bpf-next v2 5/9] bpf: verifier: add btf to the function args of bpf_check_attach_target Date: Mon, 11 Mar 2024 17:35:22 +0800 Message-Id: <20240311093526.1010158-6-dongmenglong.8@bytedance.com> X-Mailer: git-send-email 2.39.2 In-Reply-To: <20240311093526.1010158-1-dongmenglong.8@bytedance.com> References: <20240311093526.1010158-1-dongmenglong.8@bytedance.com> MIME-Version: 1.0 X-CRM114-Version: 20100106-BlameMichelson ( TRE 0.8.0 (BSD) ) MR-646709E3 X-CRM114-CacheID: sfid-20240311_023510_893612_89C41711 X-CRM114-Status: GOOD ( 13.23 ) X-BeenThere: linux-riscv@lists.infradead.org X-Mailman-Version: 2.1.34 Precedence: list List-Id: List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Sender: "linux-riscv" Errors-To: linux-riscv-bounces+linux-riscv=archiver.kernel.org@lists.infradead.org Add target btf to the function args of bpf_check_attach_target(), then the caller can specify the btf to check. Signed-off-by: Menglong Dong --- include/linux/bpf_verifier.h | 1 + kernel/bpf/syscall.c | 6 ++++-- kernel/bpf/trampoline.c | 1 + kernel/bpf/verifier.c | 8 +++++--- 4 files changed, 11 insertions(+), 5 deletions(-) diff --git a/include/linux/bpf_verifier.h b/include/linux/bpf_verifier.h index 4b0f6600e499..6cb20efcfac3 100644 --- a/include/linux/bpf_verifier.h +++ b/include/linux/bpf_verifier.h @@ -811,6 +811,7 @@ static inline void bpf_trampoline_unpack_key(u64 key, u32 *obj_id, u32 *btf_id) int bpf_check_attach_target(struct bpf_verifier_log *log, const struct bpf_prog *prog, const struct bpf_prog *tgt_prog, + struct btf *btf, u32 btf_id, struct bpf_attach_target_info *tgt_info); void bpf_free_kfunc_btf_tab(struct bpf_kfunc_btf_tab *tab); diff --git a/kernel/bpf/syscall.c b/kernel/bpf/syscall.c index d1cd645ef9ac..6128c3131141 100644 --- a/kernel/bpf/syscall.c +++ b/kernel/bpf/syscall.c @@ -3401,9 +3401,11 @@ static int bpf_tracing_prog_attach(struct bpf_prog *prog, * need a new trampoline and a check for compatibility */ struct bpf_attach_target_info tgt_info = {}; + struct btf *btf; - err = bpf_check_attach_target(NULL, prog, tgt_prog, btf_id, - &tgt_info); + btf = tgt_prog ? tgt_prog->aux->btf : prog->aux->attach_btf; + err = bpf_check_attach_target(NULL, prog, tgt_prog, btf, + btf_id, &tgt_info); if (err) goto out_unlock; diff --git a/kernel/bpf/trampoline.c b/kernel/bpf/trampoline.c index 2167aa3fe583..b00d53af8fcb 100644 --- a/kernel/bpf/trampoline.c +++ b/kernel/bpf/trampoline.c @@ -747,6 +747,7 @@ int bpf_trampoline_link_cgroup_shim(struct bpf_prog *prog, int err; err = bpf_check_attach_target(NULL, prog, NULL, + prog->aux->attach_btf, prog->aux->attach_btf_id, &tgt_info); if (err) diff --git a/kernel/bpf/verifier.c b/kernel/bpf/verifier.c index bf084c693507..4493ecc23597 100644 --- a/kernel/bpf/verifier.c +++ b/kernel/bpf/verifier.c @@ -20613,6 +20613,7 @@ static int check_non_sleepable_error_inject(u32 btf_id) int bpf_check_attach_target(struct bpf_verifier_log *log, const struct bpf_prog *prog, const struct bpf_prog *tgt_prog, + struct btf *btf, u32 btf_id, struct bpf_attach_target_info *tgt_info) { @@ -20623,7 +20624,6 @@ int bpf_check_attach_target(struct bpf_verifier_log *log, const struct btf_type *t; bool conservative = true; const char *tname; - struct btf *btf; long addr = 0; struct module *mod = NULL; @@ -20631,7 +20631,6 @@ int bpf_check_attach_target(struct bpf_verifier_log *log, bpf_log(log, "Tracing programs must provide btf_id\n"); return -EINVAL; } - btf = tgt_prog ? tgt_prog->aux->btf : prog->aux->attach_btf; if (!btf) { bpf_log(log, "FENTRY/FEXIT program can only be attached to another program annotated with BTF\n"); @@ -20940,6 +20939,7 @@ static int check_attach_btf_id(struct bpf_verifier_env *env) struct bpf_attach_target_info tgt_info = {}; u32 btf_id = prog->aux->attach_btf_id; struct bpf_trampoline *tr; + struct btf *btf; int ret; u64 key; @@ -20964,7 +20964,9 @@ static int check_attach_btf_id(struct bpf_verifier_env *env) prog->type != BPF_PROG_TYPE_EXT) return 0; - ret = bpf_check_attach_target(&env->log, prog, tgt_prog, btf_id, &tgt_info); + btf = tgt_prog ? tgt_prog->aux->btf : prog->aux->attach_btf; + ret = bpf_check_attach_target(&env->log, prog, tgt_prog, btf, + btf_id, &tgt_info); if (ret) return ret; From patchwork Mon Mar 11 09:35:23 2024 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: =?utf-8?b?5qKm6b6Z6JGj?= X-Patchwork-Id: 13588452 X-Patchwork-Delegate: bjorn@kernel.org Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from bombadil.infradead.org (bombadil.infradead.org [198.137.202.133]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (No client certificate requested) by smtp.lore.kernel.org (Postfix) with ESMTPS id 6AE39C5475B for ; Mon, 11 Mar 2024 09:36:00 +0000 (UTC) DKIM-Signature: v=1; a=rsa-sha256; q=dns/txt; c=relaxed/relaxed; d=lists.infradead.org; s=bombadil.20210309; h=Sender: Content-Transfer-Encoding:Content-Type:List-Subscribe:List-Help:List-Post: List-Archive:List-Unsubscribe:List-Id:MIME-Version:References:In-Reply-To: Message-Id:Date:Subject:Cc:To:From:Reply-To:Content-ID:Content-Description: Resent-Date:Resent-From:Resent-Sender:Resent-To:Resent-Cc:Resent-Message-ID: List-Owner; bh=Ax9OfyEyMFdWmLd+i1icNdksDf5Xp4neIK8fYQRVztY=; b=x1vP9CYwryYFCz +DJFo/ZG3MCBbcvLzthJlW5pBVB4rmN1e1k9spMKuDIJNfcAwEkwjqZiNMgZ6Ob0KojdwoD0YrPCc bD16AqcVuIZOzSiM5aHhY8RMQseqcC4cywYOHKiEbG18XtiMlwQUpTysITyJCznDGHJU7BMzXLKT2 2xDjLQg5vzRJDuqiJEU6DeOZ+Q3lng1LyFEKq1nirVE6ZR369d0nlsu2wlXeJFoH85CaJvoIktVnT W1rbHMiXCQgcYUzytQeRnWpCZ6mYHJsba8PWtcgQWXXcCl9tQa9Fy67L3ibeTCpV1mLfkl+qEVzK2 lPDIWTdUObeuf0nA1BKg==; Received: from localhost ([::1] helo=bombadil.infradead.org) by bombadil.infradead.org with esmtp (Exim 4.97.1 #2 (Red Hat Linux)) id 1rjc4T-00000000sju-3zIz; Mon, 11 Mar 2024 09:35:49 +0000 Received: from mail-pl1-x62c.google.com ([2607:f8b0:4864:20::62c]) by bombadil.infradead.org with esmtps (Exim 4.97.1 #2 (Red Hat Linux)) id 1rjc3w-00000000sJm-0ueR for linux-riscv@lists.infradead.org; Mon, 11 Mar 2024 09:35:30 +0000 Received: by mail-pl1-x62c.google.com with SMTP id d9443c01a7336-1dc49b00bdbso19974435ad.3 for ; Mon, 11 Mar 2024 02:35:15 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=bytedance.com; s=google; t=1710149714; x=1710754514; darn=lists.infradead.org; h=content-transfer-encoding:mime-version:references:in-reply-to :message-id:date:subject:cc:to:from:from:to:cc:subject:date :message-id:reply-to; bh=mXY72K13/QWs+XYqq+1yre2I5i9spjZlqZiX5c3uWc8=; b=CRlDDoWsmsDZgcn7Or8mpAuDxEOPXVkaKVma8ZLp0ypauaorAaWTigWravQbVDxQU+ lKrTrTnp7SL/SFgkoGS9YuS40NXBjYziLQ+BT99WFp9FeOpuKp5rSEFqhBck+U4sGYky Maf2b8FQYBmItb3X4foame08+xBkq+tCMEmVqW1oMIkREnjRntVXzT4O0uo6e5oPzUAH qxuUO3P45is3k1MVx+k8GbJp31Cy1PjZQB3N031Oy3JBIsrWyMxvPOVoCoY9fTF4JvM7 7yxqcvue/DTzqFIWag0E/0+jRZ87Njych71oppXJ7OnWjxVI1N0IjyyckxZ6cWIWIavI BMxw== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20230601; t=1710149714; x=1710754514; h=content-transfer-encoding:mime-version:references:in-reply-to :message-id:date:subject:cc:to:from:x-gm-message-state:from:to:cc :subject:date:message-id:reply-to; bh=mXY72K13/QWs+XYqq+1yre2I5i9spjZlqZiX5c3uWc8=; b=Tce03k803Gx6mXZGcoD9qj9Au4MP4lMu3KrePURjYXsU+EcxOKQE5UcA2x968u5Tqw 3MwKesC3sVeeTTqruDPLyTyUkT1UeI4nbMSpk7AReNssLhWqTnOGuTwYB42UlijjZZhe /g2aZeXASVaccLwHWfrJ8A9vhKalZs6HlkgJh9rxYv3XsEkQ6ase2nFgaxtuVjw+nioD a89ZJK2aTVxRidRXUCHiArwF953YPWXvZd66JjH4wnSdsSe6ZAMOZdVNli5V/8A9oGRv yx30P+8FokkFpSCbdCYmrVRXjVUMhU7pDo1np8+slRqky1Knufyh6ksj2c7HvmaWl703 DxdA== X-Forwarded-Encrypted: i=1; AJvYcCWcOAYe+pQoMwCUr1HaxBabxZS6w08jtsXk3gZSdN3V+gkb4to304Kpb19RVGogltZlFDstFJ0qdEt+dMBR+n1LTLBzUlNA32PdtLhoTR4d X-Gm-Message-State: AOJu0YzrkEoZcari2f6V/enA4P8AU4cKA1dkmTra/cKCNtyPlz69jTJl bIn7zrK3/djS4UwXpRxorD4XC1Nch8a3j8aSC35unoP2t5obCTtHVIc3dCJAXE0= X-Google-Smtp-Source: AGHT+IG1tHZkGA6ilF0C9eA74u4gWwk/Df5+HU+geMG3CnFVxYVwwj+wjdOVgVKG4WncqAPf+3Dvvw== X-Received: by 2002:a17:902:e806:b0:1dd:88cf:205c with SMTP id u6-20020a170902e80600b001dd88cf205cmr4529480plg.28.1710149714274; Mon, 11 Mar 2024 02:35:14 -0700 (PDT) Received: from localhost.localdomain ([43.129.25.208]) by smtp.gmail.com with ESMTPSA id h9-20020a170902f7c900b001dcad9cbf8bsm4253365plw.239.2024.03.11.02.35.06 (version=TLS1_3 cipher=TLS_AES_256_GCM_SHA384 bits=256/256); Mon, 11 Mar 2024 02:35:14 -0700 (PDT) From: Menglong Dong To: andrii@kernel.org Cc: ast@kernel.org, daniel@iogearbox.net, martin.lau@linux.dev, eddyz87@gmail.com, song@kernel.org, yonghong.song@linux.dev, john.fastabend@gmail.com, kpsingh@kernel.org, sdf@google.com, haoluo@google.com, jolsa@kernel.org, agordeev@linux.ibm.com, borntraeger@linux.ibm.com, svens@linux.ibm.com, davem@davemloft.net, dsahern@kernel.org, dave.hansen@linux.intel.com, x86@kernel.org, rostedt@goodmis.org, mathieu.desnoyers@efficios.com, quentin@isovalent.com, bpf@vger.kernel.org, linux-arm-kernel@lists.infradead.org, linux-kernel@vger.kernel.org, linux-riscv@lists.infradead.org, linux-s390@vger.kernel.org, netdev@vger.kernel.org, linux-trace-kernel@vger.kernel.org, linux-kselftest@vger.kernel.org, linux-stm32@st-md-mailman.stormreply.com, Menglong Dong Subject: [PATCH bpf-next v2 6/9] bpf: tracing: add multi-link support Date: Mon, 11 Mar 2024 17:35:23 +0800 Message-Id: <20240311093526.1010158-7-dongmenglong.8@bytedance.com> X-Mailer: git-send-email 2.39.2 In-Reply-To: <20240311093526.1010158-1-dongmenglong.8@bytedance.com> References: <20240311093526.1010158-1-dongmenglong.8@bytedance.com> MIME-Version: 1.0 X-CRM114-Version: 20100106-BlameMichelson ( TRE 0.8.0 (BSD) ) MR-646709E3 X-CRM114-CacheID: sfid-20240311_023516_536937_D3BFF807 X-CRM114-Status: GOOD ( 29.01 ) X-BeenThere: linux-riscv@lists.infradead.org X-Mailman-Version: 2.1.34 Precedence: list List-Id: List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Sender: "linux-riscv" Errors-To: linux-riscv-bounces+linux-riscv=archiver.kernel.org@lists.infradead.org In this commit, we add the support to allow attaching a tracing BPF program to multi hooks, which is similar to BPF_TRACE_KPROBE_MULTI. The use case is obvious. For now, we have to create a BPF program for each kernel function, for which we want to trace, even through all the program have the same (or similar logic). This can consume extra memory, and make the program loading slow if we have plenty of kernel function to trace. The KPROBE_MULTI maybe a alternative, but it can't do what TRACING do. For example, the kretprobe can't obtain the function args, but the FEXIT can. For now, we support to create multi-link for fentry/fexit/modify_return with the following new attach types that we introduce: BPF_TRACE_FENTRY_MULTI BPF_TRACE_FEXIT_MULTI BPF_MODIFY_RETURN_MULTI We introduce the struct bpf_tracing_multi_link for this purpose, which can hold all the kernel modules, target bpf program (for attaching to bpf program) or target btf (for attaching to kernel function) that we referenced. Meanwhiles, every trampoline for the function that we attach to is also stored in "struct bpf_tramp_multi_link_entry *entries" in struct bpf_tramp_multi_link. During loading, the first target is used for verification by the verifer. And during attaching, we check the consistency of all the targets with the target that we loaded, which is the first target. Signed-off-by: Menglong Dong --- include/linux/bpf.h | 11 + include/linux/bpf_types.h | 1 + include/uapi/linux/bpf.h | 10 + kernel/bpf/btf.c | 5 + kernel/bpf/syscall.c | 379 +++++++++++++++++++++++++++++++++ kernel/bpf/trampoline.c | 7 +- kernel/bpf/verifier.c | 16 +- net/core/bpf_sk_storage.c | 2 + tools/include/uapi/linux/bpf.h | 10 + 9 files changed, 438 insertions(+), 3 deletions(-) diff --git a/include/linux/bpf.h b/include/linux/bpf.h index 4e8f17d9f022..28fac2d0964a 100644 --- a/include/linux/bpf.h +++ b/include/linux/bpf.h @@ -1635,6 +1635,17 @@ struct bpf_tracing_link { struct bpf_prog *tgt_prog; }; +struct bpf_tracing_multi_link { + struct bpf_tramp_multi_link link; + enum bpf_attach_type attach_type; + u32 prog_cnt; + u32 btf_cnt; + struct bpf_prog **tgt_progs; + struct btf **tgt_btfs; + u32 mods_cnt; + struct module **mods; +}; + struct bpf_link_primer { struct bpf_link *link; struct file *file; diff --git a/include/linux/bpf_types.h b/include/linux/bpf_types.h index 94baced5a1ad..a93195bd825a 100644 --- a/include/linux/bpf_types.h +++ b/include/linux/bpf_types.h @@ -152,3 +152,4 @@ BPF_LINK_TYPE(BPF_LINK_TYPE_PERF_EVENT, perf) BPF_LINK_TYPE(BPF_LINK_TYPE_KPROBE_MULTI, kprobe_multi) BPF_LINK_TYPE(BPF_LINK_TYPE_STRUCT_OPS, struct_ops) BPF_LINK_TYPE(BPF_LINK_TYPE_UPROBE_MULTI, uprobe_multi) +BPF_LINK_TYPE(BPF_LINK_TYPE_TRACING_MULTI, tracing_multi) diff --git a/include/uapi/linux/bpf.h b/include/uapi/linux/bpf.h index 85ec7fc799d7..f01c4f463c0d 100644 --- a/include/uapi/linux/bpf.h +++ b/include/uapi/linux/bpf.h @@ -1114,6 +1114,9 @@ enum bpf_attach_type { BPF_CGROUP_UNIX_GETSOCKNAME, BPF_NETKIT_PRIMARY, BPF_NETKIT_PEER, + BPF_TRACE_FENTRY_MULTI, + BPF_TRACE_FEXIT_MULTI, + BPF_MODIFY_RETURN_MULTI, __MAX_BPF_ATTACH_TYPE }; @@ -1134,6 +1137,7 @@ enum bpf_link_type { BPF_LINK_TYPE_TCX = 11, BPF_LINK_TYPE_UPROBE_MULTI = 12, BPF_LINK_TYPE_NETKIT = 13, + BPF_LINK_TYPE_TRACING_MULTI = 14, __MAX_BPF_LINK_TYPE, }; @@ -1726,6 +1730,12 @@ union bpf_attr { */ __u64 cookie; } tracing; + struct { + __u32 cnt; + __aligned_u64 tgt_fds; + __aligned_u64 btf_ids; + __aligned_u64 cookies; + } tracing_multi; struct { __u32 pf; __u32 hooknum; diff --git a/kernel/bpf/btf.c b/kernel/bpf/btf.c index c2a0299d4358..2d6e9e680091 100644 --- a/kernel/bpf/btf.c +++ b/kernel/bpf/btf.c @@ -5879,6 +5879,9 @@ static int btf_validate_prog_ctx_type(struct bpf_verifier_log *log, const struct case BPF_TRACE_FENTRY: case BPF_TRACE_FEXIT: case BPF_MODIFY_RETURN: + case BPF_TRACE_FENTRY_MULTI: + case BPF_TRACE_FEXIT_MULTI: + case BPF_MODIFY_RETURN_MULTI: /* allow u64* as ctx */ if (btf_is_int(t) && t->size == 8) return 0; @@ -6238,6 +6241,7 @@ bool btf_ctx_access(int off, int size, enum bpf_access_type type, case BPF_LSM_CGROUP: case BPF_LSM_MAC: case BPF_TRACE_FEXIT: + case BPF_TRACE_FEXIT_MULTI: /* When LSM programs are attached to void LSM hooks * they use FEXIT trampolines and when attached to * int LSM hooks, they use MODIFY_RETURN trampolines. @@ -6256,6 +6260,7 @@ bool btf_ctx_access(int off, int size, enum bpf_access_type type, t = btf_type_by_id(btf, t->type); break; case BPF_MODIFY_RETURN: + case BPF_MODIFY_RETURN_MULTI: /* For now the BPF_MODIFY_RETURN can only be attached to * functions that return an int. */ diff --git a/kernel/bpf/syscall.c b/kernel/bpf/syscall.c index 6128c3131141..3e45584e4898 100644 --- a/kernel/bpf/syscall.c +++ b/kernel/bpf/syscall.c @@ -3273,6 +3273,34 @@ static const struct bpf_link_ops bpf_tracing_link_lops = { .fill_link_info = bpf_tracing_link_fill_link_info, }; +static int bpf_tracing_check_multi(struct bpf_prog *prog, + struct bpf_prog *tgt_prog, + struct btf *btf2, + const struct btf_type *t2) +{ + const struct btf_type *t1; + struct btf *btf1; + + /* this case is already valided in bpf_check_attach_target() */ + if (prog->type == BPF_PROG_TYPE_EXT) + return 0; + + btf1 = prog->aux->dst_prog ? prog->aux->dst_prog->aux->btf : + prog->aux->attach_btf; + if (!btf1) + return -EOPNOTSUPP; + + btf2 = btf2 ?: tgt_prog->aux->btf; + t1 = prog->aux->attach_func_proto; + + /* the target is the same as the origin one, this is a re-attach */ + if (t1 == t2) + return 0; + + return btf_check_func_part_match(btf1, t1, btf2, t2, + prog->aux->accessed_args); +} + static int bpf_tracing_prog_attach(struct bpf_prog *prog, int tgt_prog_fd, u32 btf_id, @@ -3473,6 +3501,350 @@ static int bpf_tracing_prog_attach(struct bpf_prog *prog, return err; } +static void __bpf_tracing_multi_link_release(struct bpf_tracing_multi_link *link) +{ + int i; + + if (link->mods_cnt) { + for (i = 0; i < link->mods_cnt; i++) + module_put(link->mods[i]); + kfree(link->mods); + } + + if (link->prog_cnt) { + for (i = 0; i < link->prog_cnt; i++) + bpf_prog_put(link->tgt_progs[i]); + kfree(link->tgt_progs); + } + + if (link->btf_cnt) { + for (i = 0; i < link->btf_cnt; i++) + btf_put(link->tgt_btfs[i]); + kfree(link->tgt_btfs); + } + + if (link->link.cnt) { + for (i = 0; i < link->link.cnt; i++) + bpf_trampoline_put(link->link.entries[i].trampoline); + kfree(link->link.entries); + } +} + +static void bpf_tracing_multi_link_release(struct bpf_link *link) +{ + struct bpf_tracing_multi_link *multi_link = + container_of(link, struct bpf_tracing_multi_link, link.link); + + bpf_trampoline_multi_unlink_prog(&multi_link->link); + __bpf_tracing_multi_link_release(multi_link); +} + +static void bpf_tracing_multi_link_dealloc(struct bpf_link *link) +{ + struct bpf_tracing_multi_link *tr_link = + container_of(link, struct bpf_tracing_multi_link, link.link); + + kfree(tr_link); +} + +static void bpf_tracing_multi_link_show_fdinfo(const struct bpf_link *link, + struct seq_file *seq) +{ + struct bpf_tracing_multi_link *tr_link = + container_of(link, struct bpf_tracing_multi_link, link.link); + u32 target_btf_id, target_obj_id; + int i; + + for (i = 0; i < tr_link->link.cnt; i++) { + bpf_trampoline_unpack_key(tr_link->link.entries[i].trampoline->key, + &target_obj_id, &target_btf_id); + seq_printf(seq, + "attach_type:\t%d\n" + "target_obj_id:\t%u\n" + "target_btf_id:\t%u\n", + tr_link->attach_type, + target_obj_id, + target_btf_id); + } +} + +static const struct bpf_link_ops bpf_tracing_multi_link_lops = { + .release = bpf_tracing_multi_link_release, + .dealloc = bpf_tracing_multi_link_dealloc, + .show_fdinfo = bpf_tracing_multi_link_show_fdinfo, +}; + +#define MAX_TRACING_MULTI_CNT 1024 + +static int bpf_tracing_get_target(u32 fd, struct bpf_prog **tgt_prog, + struct btf **tgt_btf) +{ + struct bpf_prog *prog = NULL; + struct btf *btf = NULL; + int err = 0; + + if (fd) { + prog = bpf_prog_get(fd); + if (!IS_ERR(prog)) + goto found; + + prog = NULL; + /* "fd" is the fd of the kernel module BTF */ + btf = btf_get_by_fd(fd); + if (IS_ERR(btf)) { + err = PTR_ERR(btf); + goto err; + } + if (!btf_is_kernel(btf)) { + btf_put(btf); + err = -EOPNOTSUPP; + goto err; + } + } else { + btf = bpf_get_btf_vmlinux(); + if (IS_ERR(btf)) { + err = PTR_ERR(btf); + goto err; + } + if (!btf) { + err = -EINVAL; + goto err; + } + btf_get(btf); + } +found: + *tgt_prog = prog; + *tgt_btf = btf; + return 0; +err: + *tgt_prog = NULL; + *tgt_btf = NULL; + return err; +} + +static int bpf_tracing_multi_link_check(const union bpf_attr *attr, u32 **btf_ids, + u32 **tgt_fds, u64 **cookies, + u32 cnt) +{ + void __user *ubtf_ids; + void __user *utgt_fds; + void __user *ucookies; + void *tmp; + int i; + + if (!cnt) + return -EINVAL; + + if (cnt > MAX_TRACING_MULTI_CNT) + return -E2BIG; + + ucookies = u64_to_user_ptr(attr->link_create.tracing_multi.cookies); + if (ucookies) { + tmp = kvmalloc_array(cnt, sizeof(**cookies), GFP_KERNEL); + if (!tmp) + return -ENOMEM; + + *cookies = tmp; + if (copy_from_user(tmp, ucookies, cnt * sizeof(**cookies))) + return -EFAULT; + } + + utgt_fds = u64_to_user_ptr(attr->link_create.tracing_multi.tgt_fds); + if (utgt_fds) { + tmp = kvmalloc_array(cnt, sizeof(**tgt_fds), GFP_KERNEL); + if (!tmp) + return -ENOMEM; + + *tgt_fds = tmp; + if (copy_from_user(tmp, utgt_fds, cnt * sizeof(**tgt_fds))) + return -EFAULT; + } + + ubtf_ids = u64_to_user_ptr(attr->link_create.tracing_multi.btf_ids); + if (!ubtf_ids) + return -EINVAL; + + tmp = kvmalloc_array(cnt, sizeof(**btf_ids), GFP_KERNEL); + if (!tmp) + return -ENOMEM; + + *btf_ids = tmp; + if (copy_from_user(tmp, ubtf_ids, cnt * sizeof(**btf_ids))) + return -EFAULT; + + for (i = 0; i < cnt; i++) { + if (!(*btf_ids)[i]) + return -EINVAL; + } + + return 0; +} + +static void bpf_tracing_multi_link_ptr_fill(struct bpf_tracing_multi_link *link, + struct ptr_array *progs, + struct ptr_array *mods, + struct ptr_array *btfs) +{ + link->mods = (struct module **) mods->ptrs; + link->mods_cnt = mods->cnt; + link->tgt_btfs = (struct btf **) btfs->ptrs; + link->btf_cnt = btfs->cnt; + link->tgt_progs = (struct bpf_prog **) progs->ptrs; + link->prog_cnt = progs->cnt; +} + +static int bpf_tracing_prog_attach_multi(const union bpf_attr *attr, + struct bpf_prog *prog) +{ + struct bpf_tracing_multi_link *link = NULL; + u32 cnt, *btf_ids = NULL, *tgt_fds = NULL; + struct bpf_link_primer link_primer; + struct ptr_array prog_array = { }; + struct ptr_array btf_array = { }; + struct ptr_array mod_array = { }; + u64 *cookies = NULL; + int err = 0, i; + + if ((prog->expected_attach_type != BPF_TRACE_FENTRY_MULTI && + prog->expected_attach_type != BPF_TRACE_FEXIT_MULTI && + prog->expected_attach_type != BPF_MODIFY_RETURN_MULTI) || + prog->type != BPF_PROG_TYPE_TRACING) + return -EINVAL; + + cnt = attr->link_create.tracing_multi.cnt; + err = bpf_tracing_multi_link_check(attr, &btf_ids, &tgt_fds, &cookies, + cnt); + if (err) + goto err_out; + + link = kzalloc(sizeof(*link), GFP_USER); + if (!link) { + err = -ENOMEM; + goto err_out; + } + link->link.entries = kzalloc(sizeof(*link->link.entries) * cnt, + GFP_USER); + if (!link->link.entries) { + err = -ENOMEM; + goto err_out; + } + + bpf_link_init(&link->link.link, BPF_LINK_TYPE_TRACING_MULTI, + &bpf_tracing_multi_link_lops, prog); + link->attach_type = prog->expected_attach_type; + + mutex_lock(&prog->aux->dst_mutex); + + /* program is already attached, re-attach is not supported here yet */ + if (!prog->aux->dst_trampoline) { + err = -EEXIST; + goto err_out_unlock; + } + + for (i = 0; i < cnt; i++) { + struct bpf_attach_target_info tgt_info = {}; + struct bpf_tramp_multi_link_entry *entry; + struct bpf_prog *tgt_prog = NULL; + struct bpf_trampoline *tr = NULL; + u32 tgt_fd, btf_id = btf_ids[i]; + struct btf *tgt_btf = NULL; + struct module *mod = NULL; + u64 key = 0; + + entry = &link->link.entries[i]; + tgt_fd = tgt_fds ? tgt_fds[i] : 0; + err = bpf_tracing_get_target(tgt_fd, &tgt_prog, &tgt_btf); + if (err) + goto err_out_unlock; + + if (tgt_prog) { + err = bpf_try_add_ptr(&prog_array, tgt_prog); + if (err) { + bpf_prog_put(tgt_prog); + if (err != -EEXIST) + goto err_out_unlock; + } + } + + if (tgt_btf) { + err = bpf_try_add_ptr(&btf_array, tgt_btf); + if (err) { + btf_put(tgt_btf); + if (err != -EEXIST) + goto err_out_unlock; + } + } + + prog->aux->attach_tracing_prog = tgt_prog && + tgt_prog->type == BPF_PROG_TYPE_TRACING && + prog->type == BPF_PROG_TYPE_TRACING; + + err = bpf_check_attach_target(NULL, prog, tgt_prog, tgt_btf, + btf_id, &tgt_info); + if (err) + goto err_out_unlock; + + mod = tgt_info.tgt_mod; + if (mod) { + err = bpf_try_add_ptr(&mod_array, mod); + if (err) { + module_put(mod); + if (err != -EEXIST) + goto err_out_unlock; + } + } + + err = bpf_tracing_check_multi(prog, tgt_prog, tgt_btf, + tgt_info.tgt_type); + if (err) + goto err_out_unlock; + + key = bpf_trampoline_compute_key(tgt_prog, tgt_btf, btf_id); + tr = bpf_trampoline_get(key, &tgt_info); + if (!tr) { + err = -ENOMEM; + goto err_out_unlock; + } + + entry->conn.cookie = cookies ? cookies[i] : 0; + entry->conn.link = &link->link.link; + entry->trampoline = tr; + link->link.cnt++; + } + + err = bpf_trampoline_multi_link_prog(&link->link); + if (err) + goto err_out_unlock; + + err = bpf_link_prime(&link->link.link, &link_primer); + if (err) { + bpf_trampoline_multi_unlink_prog(&link->link); + goto err_out_unlock; + } + + bpf_tracing_multi_link_ptr_fill(link, &prog_array, &mod_array, + &btf_array); + bpf_trampoline_put(prog->aux->dst_trampoline); + prog->aux->dst_trampoline = NULL; + mutex_unlock(&prog->aux->dst_mutex); + + kfree(btf_ids); + kfree(tgt_fds); + kfree(cookies); + return bpf_link_settle(&link_primer); +err_out_unlock: + bpf_tracing_multi_link_ptr_fill(link, &prog_array, &mod_array, + &btf_array); + __bpf_tracing_multi_link_release(link); + mutex_unlock(&prog->aux->dst_mutex); +err_out: + kfree(btf_ids); + kfree(tgt_fds); + kfree(cookies); + kfree(link); + return err; +} + struct bpf_raw_tp_link { struct bpf_link link; struct bpf_raw_event_map *btp; @@ -3924,6 +4296,9 @@ attach_type_to_prog_type(enum bpf_attach_type attach_type) case BPF_TRACE_FENTRY: case BPF_TRACE_FEXIT: case BPF_MODIFY_RETURN: + case BPF_TRACE_FENTRY_MULTI: + case BPF_TRACE_FEXIT_MULTI: + case BPF_MODIFY_RETURN_MULTI: return BPF_PROG_TYPE_TRACING; case BPF_LSM_MAC: return BPF_PROG_TYPE_LSM; @@ -5201,6 +5576,10 @@ static int link_create(union bpf_attr *attr, bpfptr_t uattr) ret = bpf_iter_link_attach(attr, uattr, prog); else if (prog->expected_attach_type == BPF_LSM_CGROUP) ret = cgroup_bpf_link_attach(attr, prog); + else if (prog->expected_attach_type == BPF_TRACE_FENTRY_MULTI || + prog->expected_attach_type == BPF_TRACE_FEXIT_MULTI || + prog->expected_attach_type == BPF_MODIFY_RETURN_MULTI) + ret = bpf_tracing_prog_attach_multi(attr, prog); else ret = bpf_tracing_prog_attach(prog, attr->link_create.target_fd, diff --git a/kernel/bpf/trampoline.c b/kernel/bpf/trampoline.c index b00d53af8fcb..6d249303a96e 100644 --- a/kernel/bpf/trampoline.c +++ b/kernel/bpf/trampoline.c @@ -111,7 +111,9 @@ bool bpf_prog_has_trampoline(const struct bpf_prog *prog) return (ptype == BPF_PROG_TYPE_TRACING && (eatype == BPF_TRACE_FENTRY || eatype == BPF_TRACE_FEXIT || - eatype == BPF_MODIFY_RETURN)) || + eatype == BPF_MODIFY_RETURN || + eatype == BPF_TRACE_FENTRY_MULTI || eatype == BPF_TRACE_FEXIT_MULTI || + eatype == BPF_MODIFY_RETURN_MULTI)) || (ptype == BPF_PROG_TYPE_LSM && eatype == BPF_LSM_MAC); } @@ -503,10 +505,13 @@ static enum bpf_tramp_prog_type bpf_attach_type_to_tramp(struct bpf_prog *prog) { switch (prog->expected_attach_type) { case BPF_TRACE_FENTRY: + case BPF_TRACE_FENTRY_MULTI: return BPF_TRAMP_FENTRY; case BPF_MODIFY_RETURN: + case BPF_MODIFY_RETURN_MULTI: return BPF_TRAMP_MODIFY_RETURN; case BPF_TRACE_FEXIT: + case BPF_TRACE_FEXIT_MULTI: return BPF_TRAMP_FEXIT; case BPF_LSM_MAC: if (!prog->aux->attach_func_proto->type) diff --git a/kernel/bpf/verifier.c b/kernel/bpf/verifier.c index 4493ecc23597..f878edfcf987 100644 --- a/kernel/bpf/verifier.c +++ b/kernel/bpf/verifier.c @@ -15405,10 +15405,13 @@ static int check_return_code(struct bpf_verifier_env *env, int regno, const char switch (env->prog->expected_attach_type) { case BPF_TRACE_FENTRY: case BPF_TRACE_FEXIT: + case BPF_TRACE_FENTRY_MULTI: + case BPF_TRACE_FEXIT_MULTI: range = retval_range(0, 0); break; case BPF_TRACE_RAW_TP: case BPF_MODIFY_RETURN: + case BPF_MODIFY_RETURN_MULTI: return 0; case BPF_TRACE_ITER: break; @@ -20709,7 +20712,9 @@ int bpf_check_attach_target(struct bpf_verifier_log *log, if (tgt_prog->type == BPF_PROG_TYPE_TRACING && prog_extension && (tgt_prog->expected_attach_type == BPF_TRACE_FENTRY || - tgt_prog->expected_attach_type == BPF_TRACE_FEXIT)) { + tgt_prog->expected_attach_type == BPF_TRACE_FEXIT || + tgt_prog->expected_attach_type == BPF_TRACE_FENTRY_MULTI || + tgt_prog->expected_attach_type == BPF_TRACE_FEXIT_MULTI)) { /* Program extensions can extend all program types * except fentry/fexit. The reason is the following. * The fentry/fexit programs are used for performance @@ -20784,6 +20789,9 @@ int bpf_check_attach_target(struct bpf_verifier_log *log, case BPF_LSM_CGROUP: case BPF_TRACE_FENTRY: case BPF_TRACE_FEXIT: + case BPF_MODIFY_RETURN_MULTI: + case BPF_TRACE_FENTRY_MULTI: + case BPF_TRACE_FEXIT_MULTI: if (!btf_type_is_func(t)) { bpf_log(log, "attach_btf_id %u is not a function\n", btf_id); @@ -20869,7 +20877,8 @@ int bpf_check_attach_target(struct bpf_verifier_log *log, bpf_log(log, "%s is not sleepable\n", tname); return ret; } - } else if (prog->expected_attach_type == BPF_MODIFY_RETURN) { + } else if (prog->expected_attach_type == BPF_MODIFY_RETURN || + prog->expected_attach_type == BPF_MODIFY_RETURN_MULTI) { if (tgt_prog) { module_put(mod); bpf_log(log, "can't modify return codes of BPF programs\n"); @@ -20922,6 +20931,9 @@ static bool can_be_sleepable(struct bpf_prog *prog) case BPF_TRACE_FEXIT: case BPF_MODIFY_RETURN: case BPF_TRACE_ITER: + case BPF_TRACE_FENTRY_MULTI: + case BPF_TRACE_FEXIT_MULTI: + case BPF_MODIFY_RETURN_MULTI: return true; default: return false; diff --git a/net/core/bpf_sk_storage.c b/net/core/bpf_sk_storage.c index 6c4d90b24d46..712ae31593e5 100644 --- a/net/core/bpf_sk_storage.c +++ b/net/core/bpf_sk_storage.c @@ -371,6 +371,8 @@ static bool bpf_sk_storage_tracing_allowed(const struct bpf_prog *prog) return true; case BPF_TRACE_FENTRY: case BPF_TRACE_FEXIT: + case BPF_TRACE_FENTRY_MULTI: + case BPF_TRACE_FEXIT_MULTI: btf_vmlinux = bpf_get_btf_vmlinux(); if (IS_ERR_OR_NULL(btf_vmlinux)) return false; diff --git a/tools/include/uapi/linux/bpf.h b/tools/include/uapi/linux/bpf.h index 85ec7fc799d7..f01c4f463c0d 100644 --- a/tools/include/uapi/linux/bpf.h +++ b/tools/include/uapi/linux/bpf.h @@ -1114,6 +1114,9 @@ enum bpf_attach_type { BPF_CGROUP_UNIX_GETSOCKNAME, BPF_NETKIT_PRIMARY, BPF_NETKIT_PEER, + BPF_TRACE_FENTRY_MULTI, + BPF_TRACE_FEXIT_MULTI, + BPF_MODIFY_RETURN_MULTI, __MAX_BPF_ATTACH_TYPE }; @@ -1134,6 +1137,7 @@ enum bpf_link_type { BPF_LINK_TYPE_TCX = 11, BPF_LINK_TYPE_UPROBE_MULTI = 12, BPF_LINK_TYPE_NETKIT = 13, + BPF_LINK_TYPE_TRACING_MULTI = 14, __MAX_BPF_LINK_TYPE, }; @@ -1726,6 +1730,12 @@ union bpf_attr { */ __u64 cookie; } tracing; + struct { + __u32 cnt; + __aligned_u64 tgt_fds; + __aligned_u64 btf_ids; + __aligned_u64 cookies; + } tracing_multi; struct { __u32 pf; __u32 hooknum; From patchwork Mon Mar 11 09:35:24 2024 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: =?utf-8?b?5qKm6b6Z6JGj?= X-Patchwork-Id: 13588453 X-Patchwork-Delegate: bjorn@kernel.org Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from bombadil.infradead.org (bombadil.infradead.org [198.137.202.133]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (No client certificate requested) by smtp.lore.kernel.org (Postfix) with ESMTPS id 80624C5475B for ; Mon, 11 Mar 2024 09:36:24 +0000 (UTC) DKIM-Signature: v=1; a=rsa-sha256; q=dns/txt; c=relaxed/relaxed; d=lists.infradead.org; s=bombadil.20210309; h=Sender: Content-Transfer-Encoding:Content-Type:List-Subscribe:List-Help:List-Post: List-Archive:List-Unsubscribe:List-Id:MIME-Version:References:In-Reply-To: Message-Id:Date:Subject:Cc:To:From:Reply-To:Content-ID:Content-Description: Resent-Date:Resent-From:Resent-Sender:Resent-To:Resent-Cc:Resent-Message-ID: List-Owner; bh=oeisL0ADcz5yGo+r8CdYPz+dg4VHCajvI0Fc/N3z6OA=; b=r84tbQbcSKDntI aTN9vx4T5DhZewvBC8sDRZd/t0q0G3/6fxEiQCKDMdyqge/zXoWmuHCPnY1B1KZYsk+NDv6Cjfa+0 +cgEiHnHRMxgI2IHPA+0lEWB4EzMee5rRLaJG6txdzPmD+JIJfyHBTXtXaE/6p+49D1Juse/hLrBi VpOGQkt6VwuvfrJZSUY8HCnDK3+CoqH9JlSmwE/zVRqTn0yGI11Ess/0hProho3mhE9iJPiZIgJin OFeNcFwYbfPj3PSdOfyAHljmy/V/eIsy+zsp2+5i777k4jfJ7shxKq6RH0eIkid526pxXn+L7cZLP 2cA4C4HqFhZyEIyTt1ww==; Received: from localhost ([::1] helo=bombadil.infradead.org) by bombadil.infradead.org with esmtp (Exim 4.97.1 #2 (Red Hat Linux)) id 1rjc4w-00000000t4G-19tU; Mon, 11 Mar 2024 09:36:18 +0000 Received: from desiato.infradead.org ([2001:8b0:10b:1:d65d:64ff:fe57:4e05]) by bombadil.infradead.org with esmtps (Exim 4.97.1 #2 (Red Hat Linux)) id 1rjc4I-00000000sav-1RoR for linux-riscv@bombadil.infradead.org; Mon, 11 Mar 2024 09:35:39 +0000 DKIM-Signature: v=1; a=rsa-sha256; q=dns/txt; c=relaxed/relaxed; d=infradead.org; s=desiato.20200630; h=Content-Transfer-Encoding:MIME-Version :References:In-Reply-To:Message-Id:Date:Subject:Cc:To:From:Sender:Reply-To: Content-Type:Content-ID:Content-Description; bh=XgF85BcnRi6YfuBECKn9LdNVtsda3wFJUMkfKzYPaTg=; b=CTU7HRUs67m7rIHV7ym4pm16xp 4RWRRtqSWhC6dm+TlNzHm8UBazbh3xd42zhNaUnd1SQDys98K6UHGRLM2v6hZ6ol5EilUAgBXlf5L scrvhwsZfsbQyPIC7h4tNAhTBkGNZ/23fAsAmAHLbN4LcEA/1wmyHR7wFnOnX4HNh78RXqJjcAA7n 2HOmaF2a5iN97R37HRrmc0JZf/XqocveNvIdAwFR7DGWR6EKZF4WV10WN1m7195qV3v8kmtziON34 F1m1bMlF0sM/tDrjRkX3zLrJKWdsV9mUmxejfqx57DcFfZ/IEd+Y2E4O2CxI9DJgfVEljYOSwfHlE RMhmIQ9w==; Received: from mail-pf1-x42d.google.com ([2607:f8b0:4864:20::42d]) by desiato.infradead.org with esmtps (Exim 4.97.1 #2 (Red Hat Linux)) id 1rjc4C-00000008stc-2qFp for linux-riscv@lists.infradead.org; Mon, 11 Mar 2024 09:35:37 +0000 Received: by mail-pf1-x42d.google.com with SMTP id d2e1a72fcca58-6e64647af39so3960166b3a.1 for ; Mon, 11 Mar 2024 02:35:23 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=bytedance.com; s=google; t=1710149722; x=1710754522; darn=lists.infradead.org; h=content-transfer-encoding:mime-version:references:in-reply-to :message-id:date:subject:cc:to:from:from:to:cc:subject:date :message-id:reply-to; bh=XgF85BcnRi6YfuBECKn9LdNVtsda3wFJUMkfKzYPaTg=; b=Cl0aCj4X9vNnuNEIdwdfG9Gv0+pNqPMs+FgK/8yMXMnrftFi7JLksHLuiXMVG1C0Fd qCpC+kvHrd7tUbmEvoD3CpW6tVxONAue5t//CdUdUM7gio7wLZPc7gZq33xatp7niInc 9heP6hLmCmTmBcBmz/9K1AGsXAUTg8Gq7eW63Zoq0DagmAsR9tOy4zu8ok+ShcXOU30k 3JtEQK6X8/6vXbn0uxSmHisUOP69/PCvERwwKHc5PL2VTXfJE/3aPptJj0A3H1pwlg7T RlBGhz0s5L46SgKfMg4UTY9/sdTPLP9KUI4UMTKBPp9DlqWPJ7zj7bKEV7VuzNCyBO+s 0sGg== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20230601; t=1710149722; x=1710754522; h=content-transfer-encoding:mime-version:references:in-reply-to :message-id:date:subject:cc:to:from:x-gm-message-state:from:to:cc :subject:date:message-id:reply-to; bh=XgF85BcnRi6YfuBECKn9LdNVtsda3wFJUMkfKzYPaTg=; b=gs059JaY9o0QfG3jUrCkO1RXQ5tZQW3D925MgmjyPWcXjaCZYali3aC1hDO+S9PrOI 9jHqf3z13F4VRTLZdwwPfygoB+mSsBX8eeQmQG9CgB6QFlOGh4QjESSBe1CNkSuZZmxJ vlcElsl/ii+BahjnmLk2uXA6gYB1FVOCrzs0uSU9fZzatCbXX3SJ3jfi9/2OIAvqhN47 SF7a6aENeqeEgj4fxYHNxBvwEebHlNje6/VUYkmJfy29Cz6hfOhMaYXJbyAds+9e2Ux6 Syv4cu6jofRtYSzfKcs4xxrQoLLdh0yWoKfAljCrZDmzsabCDryFqsSIjQRF2mBqUJYC 8E+w== X-Forwarded-Encrypted: i=1; AJvYcCUKSiCyQTxmRZoW+UKeLRso0EKc7LBU9afIqYlAp++Ip2PM1/eqDejaMJC3WrzAlCm+ngnIuHCVnmrpOEKwg676fXGHuJxVGZDHVSSFTiX2 X-Gm-Message-State: AOJu0YyppPrOv9xBjhr52iCgQV+Q1FNBTnl4CcS2M33/+E/Hme9U71UT G1oKjS1DbGdoEnIFer1cLi6aYLVhTOuj6IXUT/0dwMsN5o1uW6CL7eT56ciAgdw= X-Google-Smtp-Source: AGHT+IFGOb1BTeFqqztsZqYiXrANUPvsqRtIr4BKGzv55kaAP5codanubKYDi1f1h+yfBwPdORvCXA== X-Received: by 2002:a05:6a20:3944:b0:1a1:6c19:a175 with SMTP id r4-20020a056a20394400b001a16c19a175mr8143881pzg.8.1710149721824; Mon, 11 Mar 2024 02:35:21 -0700 (PDT) Received: from localhost.localdomain ([43.129.25.208]) by smtp.gmail.com with ESMTPSA id h9-20020a170902f7c900b001dcad9cbf8bsm4253365plw.239.2024.03.11.02.35.14 (version=TLS1_3 cipher=TLS_AES_256_GCM_SHA384 bits=256/256); Mon, 11 Mar 2024 02:35:21 -0700 (PDT) From: Menglong Dong To: andrii@kernel.org Cc: ast@kernel.org, daniel@iogearbox.net, martin.lau@linux.dev, eddyz87@gmail.com, song@kernel.org, yonghong.song@linux.dev, john.fastabend@gmail.com, kpsingh@kernel.org, sdf@google.com, haoluo@google.com, jolsa@kernel.org, agordeev@linux.ibm.com, borntraeger@linux.ibm.com, svens@linux.ibm.com, davem@davemloft.net, dsahern@kernel.org, dave.hansen@linux.intel.com, x86@kernel.org, rostedt@goodmis.org, mathieu.desnoyers@efficios.com, quentin@isovalent.com, bpf@vger.kernel.org, linux-arm-kernel@lists.infradead.org, linux-kernel@vger.kernel.org, linux-riscv@lists.infradead.org, linux-s390@vger.kernel.org, netdev@vger.kernel.org, linux-trace-kernel@vger.kernel.org, linux-kselftest@vger.kernel.org, linux-stm32@st-md-mailman.stormreply.com, Menglong Dong Subject: [PATCH bpf-next v2 7/9] libbpf: don't free btf if program of multi-link tracing existing Date: Mon, 11 Mar 2024 17:35:24 +0800 Message-Id: <20240311093526.1010158-8-dongmenglong.8@bytedance.com> X-Mailer: git-send-email 2.39.2 In-Reply-To: <20240311093526.1010158-1-dongmenglong.8@bytedance.com> References: <20240311093526.1010158-1-dongmenglong.8@bytedance.com> MIME-Version: 1.0 X-CRM114-Version: 20100106-BlameMichelson ( TRE 0.8.0 (BSD) ) MR-646709E3 X-CRM114-CacheID: sfid-20240311_093532_888839_15CAB7F3 X-CRM114-Status: GOOD ( 15.45 ) X-BeenThere: linux-riscv@lists.infradead.org X-Mailman-Version: 2.1.34 Precedence: list List-Id: List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Sender: "linux-riscv" Errors-To: linux-riscv-bounces+linux-riscv=archiver.kernel.org@lists.infradead.org By default, the kernel btf that we load during loading program will be freed after the programs are loaded in bpf_object_load(). However, we still need to use these btf for tracing of multi-link during attaching. Therefore, we don't free the btfs until the bpf object is closed if any bpf programs of the type multi-link tracing exist. Meanwhile, introduce the new api bpf_object__free_btf() to manually free the btfs after attaching. Signed-off-by: Menglong Dong --- tools/lib/bpf/libbpf.c | 47 ++++++++++++++++++++++++++++++---------- tools/lib/bpf/libbpf.h | 2 ++ tools/lib/bpf/libbpf.map | 1 + 3 files changed, 38 insertions(+), 12 deletions(-) diff --git a/tools/lib/bpf/libbpf.c b/tools/lib/bpf/libbpf.c index 567ad367e7aa..fd5428494a7e 100644 --- a/tools/lib/bpf/libbpf.c +++ b/tools/lib/bpf/libbpf.c @@ -8267,6 +8267,39 @@ static int bpf_object_prepare_struct_ops(struct bpf_object *obj) return 0; } +void bpf_object__free_btfs(struct bpf_object *obj) +{ + int i; + + /* clean up module BTFs */ + for (i = 0; i < obj->btf_module_cnt; i++) { + close(obj->btf_modules[i].fd); + btf__free(obj->btf_modules[i].btf); + free(obj->btf_modules[i].name); + } + free(obj->btf_modules); + obj->btf_modules = NULL; + obj->btf_module_cnt = 0; + + /* clean up vmlinux BTF */ + btf__free(obj->btf_vmlinux); + obj->btf_vmlinux = NULL; +} + +static void bpf_object_early_free_btf(struct bpf_object *obj) +{ + struct bpf_program *prog; + + bpf_object__for_each_program(prog, obj) { + if (prog->expected_attach_type == BPF_TRACE_FENTRY_MULTI || + prog->expected_attach_type == BPF_TRACE_FEXIT_MULTI || + prog->expected_attach_type == BPF_MODIFY_RETURN_MULTI) + return; + } + + bpf_object__free_btfs(obj); +} + static int bpf_object_load(struct bpf_object *obj, int extra_log_level, const char *target_btf_path) { int err, i; @@ -8307,18 +8340,7 @@ static int bpf_object_load(struct bpf_object *obj, int extra_log_level, const ch /* clean up fd_array */ zfree(&obj->fd_array); - /* clean up module BTFs */ - for (i = 0; i < obj->btf_module_cnt; i++) { - close(obj->btf_modules[i].fd); - btf__free(obj->btf_modules[i].btf); - free(obj->btf_modules[i].name); - } - free(obj->btf_modules); - - /* clean up vmlinux BTF */ - btf__free(obj->btf_vmlinux); - obj->btf_vmlinux = NULL; - + bpf_object_early_free_btf(obj); obj->loaded = true; /* doesn't matter if successfully or not */ if (err) @@ -8791,6 +8813,7 @@ void bpf_object__close(struct bpf_object *obj) usdt_manager_free(obj->usdt_man); obj->usdt_man = NULL; + bpf_object__free_btfs(obj); bpf_gen__free(obj->gen_loader); bpf_object__elf_finish(obj); bpf_object_unload(obj); diff --git a/tools/lib/bpf/libbpf.h b/tools/lib/bpf/libbpf.h index 5723cbbfcc41..c41a909ea4c1 100644 --- a/tools/lib/bpf/libbpf.h +++ b/tools/lib/bpf/libbpf.h @@ -299,6 +299,8 @@ LIBBPF_API struct bpf_program * bpf_object__find_program_by_name(const struct bpf_object *obj, const char *name); +LIBBPF_API void bpf_object__free_btfs(struct bpf_object *obj); + LIBBPF_API int libbpf_prog_type_by_name(const char *name, enum bpf_prog_type *prog_type, enum bpf_attach_type *expected_attach_type); diff --git a/tools/lib/bpf/libbpf.map b/tools/lib/bpf/libbpf.map index 86804fd90dd1..57642b78917f 100644 --- a/tools/lib/bpf/libbpf.map +++ b/tools/lib/bpf/libbpf.map @@ -413,4 +413,5 @@ LIBBPF_1.4.0 { bpf_token_create; btf__new_split; btf_ext__raw_data; + bpf_object__free_btfs; } LIBBPF_1.3.0; From patchwork Mon Mar 11 09:35:25 2024 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: =?utf-8?b?5qKm6b6Z6JGj?= X-Patchwork-Id: 13588454 X-Patchwork-Delegate: bjorn@kernel.org Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from bombadil.infradead.org (bombadil.infradead.org [198.137.202.133]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (No client certificate requested) by smtp.lore.kernel.org (Postfix) with ESMTPS id 980D0C5475B for ; Mon, 11 Mar 2024 09:36:27 +0000 (UTC) DKIM-Signature: v=1; a=rsa-sha256; q=dns/txt; c=relaxed/relaxed; d=lists.infradead.org; s=bombadil.20210309; h=Sender: Content-Transfer-Encoding:Content-Type:List-Subscribe:List-Help:List-Post: List-Archive:List-Unsubscribe:List-Id:MIME-Version:References:In-Reply-To: Message-Id:Date:Subject:Cc:To:From:Reply-To:Content-ID:Content-Description: Resent-Date:Resent-From:Resent-Sender:Resent-To:Resent-Cc:Resent-Message-ID: List-Owner; bh=Z6hBnAOs4H86Mho27/07x4O/NhX+egdGhpWKE0Ijayo=; b=qbn5XozEHYMFq2 2tz4ZdWPaBeRoBW/0wyP5owJ6kodWGr7dW0M2L5bKw65KphJ5surCUMjyi/ob8ue6MIGvcTbzSkra v+Nzz96C7zu2UgHhta1GzGf21AsYLoYm+Jq8sPIaMt/6IOYCWmRLsMcqhs8s30lvbfUrP2rsIvSDv e0pfpVmYFQTvvlrOLwrL9An9oN0Ko0Fo+HmSRGCrDK56+zkB5HAb95/bpNSISVxrzB6k4sAF3jZa/ Mvb5Z6zzpH0HFd4UF0ATAhiZFd7fsl7kret++QPepnXsEpz65jWHiS982yzkgGXaqXY2Cld1fXq4Y w3jfS7MAvCciTQIPqoRA==; Received: from localhost ([::1] helo=bombadil.infradead.org) by bombadil.infradead.org with esmtp (Exim 4.97.1 #2 (Red Hat Linux)) id 1rjc50-00000000t7j-0GT8; Mon, 11 Mar 2024 09:36:22 +0000 Received: from desiato.infradead.org ([2001:8b0:10b:1:d65d:64ff:fe57:4e05]) by bombadil.infradead.org with esmtps (Exim 4.97.1 #2 (Red Hat Linux)) id 1rjc4L-00000000scz-0hAe for linux-riscv@bombadil.infradead.org; Mon, 11 Mar 2024 09:35:42 +0000 DKIM-Signature: v=1; a=rsa-sha256; q=dns/txt; c=relaxed/relaxed; d=infradead.org; s=desiato.20200630; h=Content-Transfer-Encoding:MIME-Version :References:In-Reply-To:Message-Id:Date:Subject:Cc:To:From:Sender:Reply-To: Content-Type:Content-ID:Content-Description; bh=JFf08lEs3m84/dbvV4ZlLUOMjMFpJvm5xegIfbyJ9HU=; b=awkcXaxvgmf3jaEDdlCQgM6D+2 KP+tG+GhyLkf6KIDDrUf9sKHjgsaehBaVO6Jl7OOQ1jbxHKiv3cWE2FEH8QV6lIaBtvrvo2d/mZ1E 0qD1vxsvKzQ9jjxoXW68ASZsoho9fQFxZXRhi7WvO8oOHMxtX5lBb+Ni4Z8MSC9yCYBHmntpTYkRf xpeQ4Pu87bTtndnR/1fMd1QgRL3VgBELfY46u8P85Z/QYPJ3/AE30b5wv0anHe0uxgvs1GEq16CBL tiK1gbYR+4o2XMP4hX39TR7sCK1snFNUGeejFTF1cFy52DR6sEN43GNB4w2RJI5jKp/6/kSNp6YaE HxXEuXRg==; Received: from mail-pl1-x631.google.com ([2607:f8b0:4864:20::631]) by desiato.infradead.org with esmtps (Exim 4.97.1 #2 (Red Hat Linux)) id 1rjc4E-00000008suJ-3T9p for linux-riscv@lists.infradead.org; Mon, 11 Mar 2024 09:35:39 +0000 Received: by mail-pl1-x631.google.com with SMTP id d9443c01a7336-1dc96f64c10so31670685ad.1 for ; Mon, 11 Mar 2024 02:35:34 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=bytedance.com; s=google; t=1710149731; x=1710754531; darn=lists.infradead.org; h=content-transfer-encoding:mime-version:references:in-reply-to :message-id:date:subject:cc:to:from:from:to:cc:subject:date :message-id:reply-to; bh=JFf08lEs3m84/dbvV4ZlLUOMjMFpJvm5xegIfbyJ9HU=; b=AmaI1sKSOgsRIZzqumRE094xJMPUFQ5oCTmcs5O7tk9HVrZvcbfggG3AcoqSUCBM/p d8Fh7HtZkDovUNUDGbBgrEOS0L2rowzuvIk3Bze6IwzUVXC+Jqsr3dZo3Hbyr+y6kh4y 3xs03hMQGRniPUZRQeblOEvYx6gX42Wy9b5NK7OF1Yqn65Ej1ejwWB9z0LvVc4GnV+nl sE5ml+hx6QVDXhm6gcSGF1L7tZzgixLL5w0yDmXk54RtidrZH3kNjZVzOfI+zhjcYm1x sr2j+cTESd+2tjeTr+VK0yEKcOgSuEXBowJnX/cohSvbZY6Fse0JOlBqHHXKfZSZboJC Zx+g== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20230601; t=1710149731; x=1710754531; h=content-transfer-encoding:mime-version:references:in-reply-to :message-id:date:subject:cc:to:from:x-gm-message-state:from:to:cc :subject:date:message-id:reply-to; bh=JFf08lEs3m84/dbvV4ZlLUOMjMFpJvm5xegIfbyJ9HU=; b=Ds2NR8SXuIBp9sfNqTElgtbVnF85EQv0bhgx33iDPrg6s7F8RIQd0cT5417Spu+pYX kCZvK2YqYXC6CZErICu8UMmpHqKNIZ0M5BNwzOqJblFHlUWa52DsF7phc22PtmhR5PUw aD4Bm6SfJXHZfCx6veBudCFl7PQN6uomP8JelSR1XSzpKbw2mQ9RnqmqUEZgzoeg3397 NN7qO924501vz2Cec+BGkPhDQ2gh7y0/IOEYVDd3TUAFQm6mDrsJTCcv3iAlsP4aAu3m QcDZvfl1yDkBliUadVH2eUoKTYHB2d1OMdIkM7kLAVVA76apLApoCYgvkDwg+tSAh5IC Perw== X-Forwarded-Encrypted: i=1; AJvYcCX/wuvX0MApZSJyro2sMylsMNHz+cIHKuS3F9BixdMFaRvJG7K1gKRfiQDk0GJJNKsqy8lzg6Oh0YppuD7mvlsEp3ZiNfXsSqELZksr2rS8 X-Gm-Message-State: AOJu0YwTuuBGwELtmC9jfsIhg7I8ufb+c4HRJG0OXeSNpS6JP2tx271U BK5WQ+8ZpB86mzOXQxleSXB+R27zeiOnFvYBaXV1pawrt96x5dNR25yZ+xy2Hbo= X-Google-Smtp-Source: AGHT+IFQIVN/SFlbgP5cEEuZ7moW9KP5l9jB6qy51la06bqNfTxjZ7YG5WDDYrE2U6yDVhQ3L+9Yxw== X-Received: by 2002:a17:902:988c:b0:1dd:772e:d504 with SMTP id s12-20020a170902988c00b001dd772ed504mr4091891plp.66.1710149729346; Mon, 11 Mar 2024 02:35:29 -0700 (PDT) Received: from localhost.localdomain ([43.129.25.208]) by smtp.gmail.com with ESMTPSA id h9-20020a170902f7c900b001dcad9cbf8bsm4253365plw.239.2024.03.11.02.35.22 (version=TLS1_3 cipher=TLS_AES_256_GCM_SHA384 bits=256/256); Mon, 11 Mar 2024 02:35:29 -0700 (PDT) From: Menglong Dong To: andrii@kernel.org Cc: ast@kernel.org, daniel@iogearbox.net, martin.lau@linux.dev, eddyz87@gmail.com, song@kernel.org, yonghong.song@linux.dev, john.fastabend@gmail.com, kpsingh@kernel.org, sdf@google.com, haoluo@google.com, jolsa@kernel.org, agordeev@linux.ibm.com, borntraeger@linux.ibm.com, svens@linux.ibm.com, davem@davemloft.net, dsahern@kernel.org, dave.hansen@linux.intel.com, x86@kernel.org, rostedt@goodmis.org, mathieu.desnoyers@efficios.com, quentin@isovalent.com, bpf@vger.kernel.org, linux-arm-kernel@lists.infradead.org, linux-kernel@vger.kernel.org, linux-riscv@lists.infradead.org, linux-s390@vger.kernel.org, netdev@vger.kernel.org, linux-trace-kernel@vger.kernel.org, linux-kselftest@vger.kernel.org, linux-stm32@st-md-mailman.stormreply.com, Menglong Dong Subject: [PATCH bpf-next v2 8/9] libbpf: add support for the multi-link of tracing Date: Mon, 11 Mar 2024 17:35:25 +0800 Message-Id: <20240311093526.1010158-9-dongmenglong.8@bytedance.com> X-Mailer: git-send-email 2.39.2 In-Reply-To: <20240311093526.1010158-1-dongmenglong.8@bytedance.com> References: <20240311093526.1010158-1-dongmenglong.8@bytedance.com> MIME-Version: 1.0 X-CRM114-Version: 20100106-BlameMichelson ( TRE 0.8.0 (BSD) ) MR-646709E3 X-CRM114-CacheID: sfid-20240311_093535_003312_4D0F22E3 X-CRM114-Status: GOOD ( 24.06 ) X-BeenThere: linux-riscv@lists.infradead.org X-Mailman-Version: 2.1.34 Precedence: list List-Id: List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Sender: "linux-riscv" Errors-To: linux-riscv-bounces+linux-riscv=archiver.kernel.org@lists.infradead.org Add support for the attach types of: BPF_TRACE_FENTRY_MULTI BPF_TRACE_FEXIT_MULTI BPF_MODIFY_RETURN_MULTI Signed-off-by: Menglong Dong --- tools/bpf/bpftool/common.c | 3 + tools/lib/bpf/bpf.c | 10 +++ tools/lib/bpf/bpf.h | 6 ++ tools/lib/bpf/libbpf.c | 168 ++++++++++++++++++++++++++++++++++++- tools/lib/bpf/libbpf.h | 14 ++++ tools/lib/bpf/libbpf.map | 1 + 6 files changed, 199 insertions(+), 3 deletions(-) diff --git a/tools/bpf/bpftool/common.c b/tools/bpf/bpftool/common.c index cc6e6aae2447..ffc85256671d 100644 --- a/tools/bpf/bpftool/common.c +++ b/tools/bpf/bpftool/common.c @@ -1089,6 +1089,9 @@ const char *bpf_attach_type_input_str(enum bpf_attach_type t) case BPF_TRACE_FENTRY: return "fentry"; case BPF_TRACE_FEXIT: return "fexit"; case BPF_MODIFY_RETURN: return "mod_ret"; + case BPF_TRACE_FENTRY_MULTI: return "fentry_multi"; + case BPF_TRACE_FEXIT_MULTI: return "fexit_multi"; + case BPF_MODIFY_RETURN_MULTI: return "mod_ret_multi"; case BPF_SK_REUSEPORT_SELECT: return "sk_skb_reuseport_select"; case BPF_SK_REUSEPORT_SELECT_OR_MIGRATE: return "sk_skb_reuseport_select_or_migrate"; default: return libbpf_bpf_attach_type_str(t); diff --git a/tools/lib/bpf/bpf.c b/tools/lib/bpf/bpf.c index 97ec005c3c47..63d4734dbae4 100644 --- a/tools/lib/bpf/bpf.c +++ b/tools/lib/bpf/bpf.c @@ -793,6 +793,16 @@ int bpf_link_create(int prog_fd, int target_fd, if (!OPTS_ZEROED(opts, tracing)) return libbpf_err(-EINVAL); break; + case BPF_TRACE_FENTRY_MULTI: + case BPF_TRACE_FEXIT_MULTI: + case BPF_MODIFY_RETURN_MULTI: + attr.link_create.tracing_multi.btf_ids = ptr_to_u64(OPTS_GET(opts, tracing_multi.btf_ids, 0)); + attr.link_create.tracing_multi.tgt_fds = ptr_to_u64(OPTS_GET(opts, tracing_multi.tgt_fds, 0)); + attr.link_create.tracing_multi.cookies = ptr_to_u64(OPTS_GET(opts, tracing_multi.cookies, 0)); + attr.link_create.tracing_multi.cnt = OPTS_GET(opts, tracing_multi.cnt, 0); + if (!OPTS_ZEROED(opts, tracing_multi)) + return libbpf_err(-EINVAL); + break; case BPF_NETFILTER: attr.link_create.netfilter.pf = OPTS_GET(opts, netfilter.pf, 0); attr.link_create.netfilter.hooknum = OPTS_GET(opts, netfilter.hooknum, 0); diff --git a/tools/lib/bpf/bpf.h b/tools/lib/bpf/bpf.h index df0db2f0cdb7..e28c88d6cfa4 100644 --- a/tools/lib/bpf/bpf.h +++ b/tools/lib/bpf/bpf.h @@ -419,6 +419,12 @@ struct bpf_link_create_opts { struct { __u64 cookie; } tracing; + struct { + __u32 cnt; + const __u32 *btf_ids; + const __u32 *tgt_fds; + const __u64 *cookies; + } tracing_multi; struct { __u32 pf; __u32 hooknum; diff --git a/tools/lib/bpf/libbpf.c b/tools/lib/bpf/libbpf.c index fd5428494a7e..821214774941 100644 --- a/tools/lib/bpf/libbpf.c +++ b/tools/lib/bpf/libbpf.c @@ -132,6 +132,9 @@ static const char * const attach_type_name[] = { [BPF_TRACE_UPROBE_MULTI] = "trace_uprobe_multi", [BPF_NETKIT_PRIMARY] = "netkit_primary", [BPF_NETKIT_PEER] = "netkit_peer", + [BPF_TRACE_FENTRY_MULTI] = "trace_fentry_multi", + [BPF_TRACE_FEXIT_MULTI] = "trace_fexit_multi", + [BPF_MODIFY_RETURN_MULTI] = "modify_return_multi", }; static const char * const link_type_name[] = { @@ -381,6 +384,8 @@ enum sec_def_flags { SEC_XDP_FRAGS = 16, /* Setup proper attach type for usdt probes. */ SEC_USDT = 32, + /* attachment target is multi-link */ + SEC_ATTACH_BTF_MULTI = 64, }; struct bpf_sec_def { @@ -7160,9 +7165,9 @@ static int libbpf_prepare_prog_load(struct bpf_program *prog, if ((def & SEC_USDT) && kernel_supports(prog->obj, FEAT_UPROBE_MULTI_LINK)) prog->expected_attach_type = BPF_TRACE_UPROBE_MULTI; - if ((def & SEC_ATTACH_BTF) && !prog->attach_btf_id) { + if ((def & (SEC_ATTACH_BTF | SEC_ATTACH_BTF_MULTI)) && !prog->attach_btf_id) { int btf_obj_fd = 0, btf_type_id = 0, err; - const char *attach_name; + const char *attach_name, *name_end; attach_name = strchr(prog->sec_name, '/'); if (!attach_name) { @@ -7181,7 +7186,27 @@ static int libbpf_prepare_prog_load(struct bpf_program *prog, } attach_name++; /* skip over / */ - err = libbpf_find_attach_btf_id(prog, attach_name, &btf_obj_fd, &btf_type_id); + name_end = strchr(attach_name, ','); + /* for multi-link tracing, use the first target symbol during + * loading. + */ + if ((def & SEC_ATTACH_BTF_MULTI) && name_end) { + int len = name_end - attach_name + 1; + char *first_tgt; + + first_tgt = malloc(len); + if (!first_tgt) + return -ENOMEM; + strncpy(first_tgt, attach_name, len); + first_tgt[len - 1] = '\0'; + err = libbpf_find_attach_btf_id(prog, first_tgt, &btf_obj_fd, + &btf_type_id); + free(first_tgt); + } else { + err = libbpf_find_attach_btf_id(prog, attach_name, &btf_obj_fd, + &btf_type_id); + } + if (err) return err; @@ -9149,6 +9174,7 @@ static int attach_kprobe_multi(const struct bpf_program *prog, long cookie, stru static int attach_uprobe_multi(const struct bpf_program *prog, long cookie, struct bpf_link **link); static int attach_lsm(const struct bpf_program *prog, long cookie, struct bpf_link **link); static int attach_iter(const struct bpf_program *prog, long cookie, struct bpf_link **link); +static int attach_trace_multi(const struct bpf_program *prog, long cookie, struct bpf_link **link); static const struct bpf_sec_def section_defs[] = { SEC_DEF("socket", SOCKET_FILTER, 0, SEC_NONE), @@ -9192,6 +9218,13 @@ static const struct bpf_sec_def section_defs[] = { SEC_DEF("fentry.s+", TRACING, BPF_TRACE_FENTRY, SEC_ATTACH_BTF | SEC_SLEEPABLE, attach_trace), SEC_DEF("fmod_ret.s+", TRACING, BPF_MODIFY_RETURN, SEC_ATTACH_BTF | SEC_SLEEPABLE, attach_trace), SEC_DEF("fexit.s+", TRACING, BPF_TRACE_FEXIT, SEC_ATTACH_BTF | SEC_SLEEPABLE, attach_trace), + SEC_DEF("tp_btf+", TRACING, BPF_TRACE_RAW_TP, SEC_ATTACH_BTF, attach_trace), + SEC_DEF("fentry.multi+", TRACING, BPF_TRACE_FENTRY_MULTI, SEC_ATTACH_BTF_MULTI, attach_trace_multi), + SEC_DEF("fmod_ret.multi+", TRACING, BPF_MODIFY_RETURN_MULTI, SEC_ATTACH_BTF_MULTI, attach_trace_multi), + SEC_DEF("fexit.multi+", TRACING, BPF_TRACE_FEXIT_MULTI, SEC_ATTACH_BTF_MULTI, attach_trace_multi), + SEC_DEF("fentry.multi.s+", TRACING, BPF_TRACE_FENTRY_MULTI, SEC_ATTACH_BTF_MULTI | SEC_SLEEPABLE, attach_trace_multi), + SEC_DEF("fmod_ret.multi.s+", TRACING, BPF_MODIFY_RETURN_MULTI, SEC_ATTACH_BTF_MULTI | SEC_SLEEPABLE, attach_trace_multi), + SEC_DEF("fexit.multi.s+", TRACING, BPF_TRACE_FEXIT_MULTI, SEC_ATTACH_BTF_MULTI | SEC_SLEEPABLE, attach_trace_multi), SEC_DEF("freplace+", EXT, 0, SEC_ATTACH_BTF, attach_trace), SEC_DEF("lsm+", LSM, BPF_LSM_MAC, SEC_ATTACH_BTF, attach_lsm), SEC_DEF("lsm.s+", LSM, BPF_LSM_MAC, SEC_ATTACH_BTF | SEC_SLEEPABLE, attach_lsm), @@ -12300,6 +12333,135 @@ static int attach_trace(const struct bpf_program *prog, long cookie, struct bpf_ return libbpf_get_error(*link); } +struct bpf_link *bpf_program__attach_trace_multi_opts(const struct bpf_program *prog, + const struct bpf_trace_multi_opts *opts) +{ + LIBBPF_OPTS(bpf_link_create_opts, link_opts); + __u32 *btf_ids = NULL, *tgt_fds = NULL; + struct bpf_link *link = NULL; + char errmsg[STRERR_BUFSIZE]; + int prog_fd, pfd, cnt, err; + + if (!OPTS_VALID(opts, bpf_trace_multi_opts)) + return libbpf_err_ptr(-EINVAL); + + prog_fd = bpf_program__fd(prog); + if (prog_fd < 0) { + pr_warn("prog '%s': can't attach before loaded\n", prog->name); + return libbpf_err_ptr(-EINVAL); + } + + cnt = OPTS_GET(opts, cnt, 0); + if (opts->syms) { + int btf_obj_fd, btf_type_id, i; + + if (opts->btf_ids || opts->tgt_fds) { + pr_warn("can set both opts->syms and opts->btf_ids\n"); + return libbpf_err_ptr(-EINVAL); + } + + btf_ids = malloc(sizeof(*btf_ids) * cnt); + tgt_fds = malloc(sizeof(*tgt_fds) * cnt); + if (!btf_ids || !tgt_fds) { + err = -ENOMEM; + goto err_free; + } + for (i = 0; i < cnt; i++) { + btf_obj_fd = btf_type_id = 0; + + err = find_kernel_btf_id(prog->obj, opts->syms[i], + prog->expected_attach_type, &btf_obj_fd, + &btf_type_id); + if (err) + goto err_free; + btf_ids[i] = btf_type_id; + tgt_fds[i] = btf_obj_fd; + } + link_opts.tracing_multi.btf_ids = btf_ids; + link_opts.tracing_multi.tgt_fds = tgt_fds; + } else { + link_opts.tracing_multi.btf_ids = OPTS_GET(opts, btf_ids, 0); + link_opts.tracing_multi.tgt_fds = OPTS_GET(opts, tgt_fds, 0); + } + + link = calloc(1, sizeof(*link)); + if (!link) { + err = -ENOMEM; + goto err_free; + } + link->detach = &bpf_link__detach_fd; + + link_opts.tracing_multi.cookies = OPTS_GET(opts, cookies, 0); + link_opts.tracing_multi.cnt = cnt; + + pfd = bpf_link_create(prog_fd, 0, bpf_program__expected_attach_type(prog), &link_opts); + if (pfd < 0) { + err = -errno; + pr_warn("prog '%s': failed to attach: %s\n", + prog->name, libbpf_strerror_r(pfd, errmsg, sizeof(errmsg))); + goto err_free; + } + link->fd = pfd; + + free(btf_ids); + free(tgt_fds); + return link; +err_free: + free(btf_ids); + free(tgt_fds); + free(link); + return libbpf_err_ptr(err); +} + +static int attach_trace_multi(const struct bpf_program *prog, long cookie, struct bpf_link **link) +{ + LIBBPF_OPTS(bpf_trace_multi_opts, opts); + int i, err, len, cnt = 1; + char **syms, *buf, *name; + const char *spec; + + spec = strchr(prog->sec_name, '/'); + if (!spec || !*(++spec)) + return -EINVAL; + + len = strlen(spec); + buf = malloc(len + 1); + if (!buf) + return -ENOMEM; + + strcpy(buf, spec); + for (i = 0; i < len; i++) { + if (buf[i] == ',') + cnt++; + } + + syms = malloc(sizeof(*syms) * cnt); + if (!syms) { + err = -ENOMEM; + goto out_free; + } + + opts.syms = (const char **)syms; + opts.cnt = cnt; + name = buf; + err = -EINVAL; + while (name) { + if (*name == '\0') + goto out_free; + *(syms++) = name; + name = strchr(name, ','); + if (name) + *(name++) = '\0'; + } + + *link = bpf_program__attach_trace_multi_opts(prog, &opts); + err = libbpf_get_error(*link); +out_free: + free(buf); + free(opts.syms); + return err; +} + static int attach_lsm(const struct bpf_program *prog, long cookie, struct bpf_link **link) { *link = bpf_program__attach_lsm(prog); diff --git a/tools/lib/bpf/libbpf.h b/tools/lib/bpf/libbpf.h index c41a909ea4c1..9bca44d5adfa 100644 --- a/tools/lib/bpf/libbpf.h +++ b/tools/lib/bpf/libbpf.h @@ -790,6 +790,20 @@ bpf_program__attach_xdp(const struct bpf_program *prog, int ifindex); LIBBPF_API struct bpf_link * bpf_program__attach_freplace(const struct bpf_program *prog, int target_fd, const char *attach_func_name); +struct bpf_trace_multi_opts { + /* size of this struct, for forward/backward compatibility */ + size_t sz; + const char **syms; + __u32 *btf_ids; + __u32 *tgt_fds; + __u64 *cookies; + size_t cnt; +}; +#define bpf_trace_multi_opts__last_field cnt + +LIBBPF_API struct bpf_link * +bpf_program__attach_trace_multi_opts(const struct bpf_program *prog, + const struct bpf_trace_multi_opts *opts); struct bpf_netfilter_opts { /* size of this struct, for forward/backward compatibility */ diff --git a/tools/lib/bpf/libbpf.map b/tools/lib/bpf/libbpf.map index 57642b78917f..94933898df44 100644 --- a/tools/lib/bpf/libbpf.map +++ b/tools/lib/bpf/libbpf.map @@ -414,4 +414,5 @@ LIBBPF_1.4.0 { btf__new_split; btf_ext__raw_data; bpf_object__free_btfs; + bpf_program__attach_trace_multi_opts; } LIBBPF_1.3.0; From patchwork Mon Mar 11 09:35:26 2024 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: =?utf-8?b?5qKm6b6Z6JGj?= X-Patchwork-Id: 13588455 X-Patchwork-Delegate: bjorn@kernel.org Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from bombadil.infradead.org (bombadil.infradead.org [198.137.202.133]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (No client certificate requested) by smtp.lore.kernel.org (Postfix) with ESMTPS id 7BFD0C54E58 for ; Mon, 11 Mar 2024 09:37:19 +0000 (UTC) DKIM-Signature: v=1; a=rsa-sha256; q=dns/txt; c=relaxed/relaxed; d=lists.infradead.org; s=bombadil.20210309; h=Sender: Content-Transfer-Encoding:Content-Type:List-Subscribe:List-Help:List-Post: List-Archive:List-Unsubscribe:List-Id:MIME-Version:References:In-Reply-To: Message-Id:Date:Subject:Cc:To:From:Reply-To:Content-ID:Content-Description: Resent-Date:Resent-From:Resent-Sender:Resent-To:Resent-Cc:Resent-Message-ID: List-Owner; bh=O67lSYyaONaX6Vt7w3G+/AS+uW6BdCqzi2KD87bUvrw=; b=Vh5vqw1D5Obczj wwrAjXBtSwVB1FU5Ibh4t/DcHtlzxQPq7dl65gpb+D8zg68USRjxAwSjJXyUzRvDvrhZ6driWpte5 pLS+HgEP1ycit7CGuM+TJX9e5441HXe5pdggB7ECQpRfZuLdvwNLXq3v55joQI/3aTLOtaFsoDP9R vmxXXeLfkejUIZCG8c3FL2O2SSddLJ6UTX1jMRrCA0N0R/NOs3zxsJLemO6kdpjIB+5mQrBfVZrK4 ZqNMI/Dh/WbfbHRlCJ94KCSThqx6MT4l8Onbstc1iJJMsxt8b+AypPQNIGgZ294PcAE/t7Kh/JBBs 9j5OwVGZ+R43RXJjN4VA==; Received: from localhost ([::1] helo=bombadil.infradead.org) by bombadil.infradead.org with esmtp (Exim 4.97.1 #2 (Red Hat Linux)) id 1rjc5b-00000000tWR-2OgV; Mon, 11 Mar 2024 09:36:59 +0000 Received: from desiato.infradead.org ([2001:8b0:10b:1:d65d:64ff:fe57:4e05]) by bombadil.infradead.org with esmtps (Exim 4.97.1 #2 (Red Hat Linux)) id 1rjc4O-00000000sg7-1YBl for linux-riscv@bombadil.infradead.org; Mon, 11 Mar 2024 09:35:47 +0000 DKIM-Signature: v=1; a=rsa-sha256; q=dns/txt; c=relaxed/relaxed; d=infradead.org; s=desiato.20200630; h=Content-Transfer-Encoding:MIME-Version :References:In-Reply-To:Message-Id:Date:Subject:Cc:To:From:Sender:Reply-To: Content-Type:Content-ID:Content-Description; bh=D0H4fA44zT5N27s+Kh4SgtJTB2b28yPECpS+di519/k=; b=pEtrAJpwHr8ymJQ0mopCCLcTxZ 5sVBdL9lltNyiQh2/A3CWeuOr0PIUbiBN74thLHeTvqW7Qm9dixIxzSYwQH+i8NcOHWFyiGBeRD9+ H+aADiEYQWqhREQagei+OjpN88k6TcD6DNu9JkCZX/TKfdxs+WFpwDD+0/fxg5AKeUmKhpCK6kIbJ 2DP/RLP7r2IuD116eVsRC/9yrcqvHugDAMgkatTFh5jEridCGwpH7jOLZWF3sv7SueMVgjqgNI5Fc LxxWsWRA44aRw7uhzyvxRYpnprV9UN6rO8rbO0yfaEP7zEb+BocuNppQX8OT/Ynll5r2/y2HVfdlO F7V9sODA==; Received: from mail-pl1-x636.google.com ([2607:f8b0:4864:20::636]) by desiato.infradead.org with esmtps (Exim 4.97.1 #2 (Red Hat Linux)) id 1rjc4L-00000008svE-0nLQ for linux-riscv@lists.infradead.org; Mon, 11 Mar 2024 09:35:43 +0000 Received: by mail-pl1-x636.google.com with SMTP id d9443c01a7336-1dd9066b7c3so5001855ad.2 for ; Mon, 11 Mar 2024 02:35:40 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=bytedance.com; s=google; t=1710149739; x=1710754539; darn=lists.infradead.org; h=content-transfer-encoding:mime-version:references:in-reply-to :message-id:date:subject:cc:to:from:from:to:cc:subject:date :message-id:reply-to; bh=D0H4fA44zT5N27s+Kh4SgtJTB2b28yPECpS+di519/k=; b=VueC3GrwC2tCXM4eY8FBrNqEvS0/YAxHkn2todl4EfIBTAyXQKENL4Yfo67g6vQ7mY Fd7dGEWdkCeLkPgqH2TYX+zHG+CBOTIdNpaI8OVD2x+C5+fGwbjG2JNwTxyqc9QlmJi6 +KVRAr7p36Ro26roIT5J/NojNSsXz0JNVKYljwdBII/+U3rDZkng8CX2SMDeDLsFyc+m TsTtDUvM/w8IbWL/29gOlk7p7Zm12S5F25F360MSuEkomjxHF7MBu1C+LcBy3XLO2H+Y n8Njb10N8y1hIJbhIapuU2N5feiZ6r1hFFTMTzwbKPC2gtCcn2D3OorOiXRELCWBV4E+ mNDQ== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20230601; t=1710149739; x=1710754539; h=content-transfer-encoding:mime-version:references:in-reply-to :message-id:date:subject:cc:to:from:x-gm-message-state:from:to:cc :subject:date:message-id:reply-to; bh=D0H4fA44zT5N27s+Kh4SgtJTB2b28yPECpS+di519/k=; b=dJhoZlRxbsBYw9wV517UDnvCXs2gGkw7sbCsLdl4T/QPzjLsng75wZXTRi3OuPeOgW 62osBjLPICflqPjrMRLLu2A1QeLaXbOcH3YErSdHIa3QW5mpPvrznsBJBZyOnL0h74xL CobX5EIaG80DeNsyV7udJ6jJ3yVSKoUVtFxRQPS8BG83W6HlV0cAs36MGCW/Bno3dDwh uF08rsF4LgI6koS1mzZdz0rcD6vdr7od9BH+gWb/ClsrwBcUdWvO7BQHue+MiFCGgG3s 6DWMc769+9RBLAPbLxg2/ZchAuHWo3nAMGRxQx6IFVvx6p9pBamrLrjzqMxUMGiUvfF6 z+lg== X-Forwarded-Encrypted: i=1; AJvYcCV4C7WnIy/lVh67mqkVx7h61esNgKQCwI/IRNLvuPSgpTegHqPhtnpkUxOJ+J3Sjbp7M9vzII4XsTAsvV8mKnvu7nx5ZxeCJZUjlyke0gRl X-Gm-Message-State: AOJu0YyO3nQhiwtRkv1OkJ8V2QUAckJePzwpPRPHCcTo8aU6TTaMqqP1 gAOqCPdqx+tEoszdjcQCW/dd2/6YeA+Ylk7Akf5g6tEs6t2dfs7yv64eXRXXk1g= X-Google-Smtp-Source: AGHT+IF9o9Zp/qBA6Xo09/VLeiyzi/BAQ9e7SOdKG4PQAZHA8aQWeEpKs4fq0ske4qR64Qj0bN0ISw== X-Received: by 2002:a17:902:c947:b0:1dd:a179:5ff3 with SMTP id i7-20020a170902c94700b001dda1795ff3mr1317685pla.9.1710149739100; Mon, 11 Mar 2024 02:35:39 -0700 (PDT) Received: from localhost.localdomain ([43.129.25.208]) by smtp.gmail.com with ESMTPSA id h9-20020a170902f7c900b001dcad9cbf8bsm4253365plw.239.2024.03.11.02.35.31 (version=TLS1_3 cipher=TLS_AES_256_GCM_SHA384 bits=256/256); Mon, 11 Mar 2024 02:35:38 -0700 (PDT) From: Menglong Dong To: andrii@kernel.org Cc: ast@kernel.org, daniel@iogearbox.net, martin.lau@linux.dev, eddyz87@gmail.com, song@kernel.org, yonghong.song@linux.dev, john.fastabend@gmail.com, kpsingh@kernel.org, sdf@google.com, haoluo@google.com, jolsa@kernel.org, agordeev@linux.ibm.com, borntraeger@linux.ibm.com, svens@linux.ibm.com, davem@davemloft.net, dsahern@kernel.org, dave.hansen@linux.intel.com, x86@kernel.org, rostedt@goodmis.org, mathieu.desnoyers@efficios.com, quentin@isovalent.com, bpf@vger.kernel.org, linux-arm-kernel@lists.infradead.org, linux-kernel@vger.kernel.org, linux-riscv@lists.infradead.org, linux-s390@vger.kernel.org, netdev@vger.kernel.org, linux-trace-kernel@vger.kernel.org, linux-kselftest@vger.kernel.org, linux-stm32@st-md-mailman.stormreply.com, Menglong Dong Subject: [PATCH bpf-next v2 9/9] selftests/bpf: add testcases for multi-link of tracing Date: Mon, 11 Mar 2024 17:35:26 +0800 Message-Id: <20240311093526.1010158-10-dongmenglong.8@bytedance.com> X-Mailer: git-send-email 2.39.2 In-Reply-To: <20240311093526.1010158-1-dongmenglong.8@bytedance.com> References: <20240311093526.1010158-1-dongmenglong.8@bytedance.com> MIME-Version: 1.0 X-CRM114-Version: 20100106-BlameMichelson ( TRE 0.8.0 (BSD) ) MR-646709E3 X-CRM114-CacheID: sfid-20240311_093541_414793_68E371AE X-CRM114-Status: GOOD ( 18.07 ) X-BeenThere: linux-riscv@lists.infradead.org X-Mailman-Version: 2.1.34 Precedence: list List-Id: List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Sender: "linux-riscv" Errors-To: linux-riscv-bounces+linux-riscv=archiver.kernel.org@lists.infradead.org In this commit, we add some testcases for the following attach types: BPF_TRACE_FENTRY_MULTI BPF_TRACE_FEXIT_MULTI BPF_MODIFY_RETURN_MULTI Signed-off-by: Menglong Dong --- net/bpf/test_run.c | 3 + .../selftests/bpf/bpf_testmod/bpf_testmod.c | 49 ++++ .../bpf/prog_tests/tracing_multi_link.c | 153 +++++++++++++ .../selftests/bpf/progs/tracing_multi_test.c | 209 ++++++++++++++++++ 4 files changed, 414 insertions(+) create mode 100644 tools/testing/selftests/bpf/prog_tests/tracing_multi_link.c create mode 100644 tools/testing/selftests/bpf/progs/tracing_multi_test.c diff --git a/net/bpf/test_run.c b/net/bpf/test_run.c index 5535f9adc658..126218297984 100644 --- a/net/bpf/test_run.c +++ b/net/bpf/test_run.c @@ -673,6 +673,8 @@ int bpf_prog_test_run_tracing(struct bpf_prog *prog, switch (prog->expected_attach_type) { case BPF_TRACE_FENTRY: case BPF_TRACE_FEXIT: + case BPF_TRACE_FENTRY_MULTI: + case BPF_TRACE_FEXIT_MULTI: if (bpf_fentry_test1(1) != 2 || bpf_fentry_test2(2, 3) != 5 || bpf_fentry_test3(4, 5, 6) != 15 || @@ -685,6 +687,7 @@ int bpf_prog_test_run_tracing(struct bpf_prog *prog, goto out; break; case BPF_MODIFY_RETURN: + case BPF_MODIFY_RETURN_MULTI: ret = bpf_modify_return_test(1, &b); if (b != 2) side_effect++; diff --git a/tools/testing/selftests/bpf/bpf_testmod/bpf_testmod.c b/tools/testing/selftests/bpf/bpf_testmod/bpf_testmod.c index 39ad96a18123..99a941b26cff 100644 --- a/tools/testing/selftests/bpf/bpf_testmod/bpf_testmod.c +++ b/tools/testing/selftests/bpf/bpf_testmod/bpf_testmod.c @@ -98,12 +98,61 @@ bpf_testmod_test_struct_arg_8(u64 a, void *b, short c, int d, void *e, return bpf_testmod_test_struct_arg_result; } +noinline int +bpf_testmod_test_struct_arg_9(struct bpf_testmod_struct_arg_2 a, + struct bpf_testmod_struct_arg_1 b) { + bpf_testmod_test_struct_arg_result = a.a + a.b + b.a; + return bpf_testmod_test_struct_arg_result; +} + +noinline int +bpf_testmod_test_struct_arg_10(int a, struct bpf_testmod_struct_arg_2 b) { + bpf_testmod_test_struct_arg_result = a + b.a + b.b; + return bpf_testmod_test_struct_arg_result; +} + +noinline struct bpf_testmod_struct_arg_2 * +bpf_testmod_test_struct_arg_11(int a, struct bpf_testmod_struct_arg_2 b, int c) { + bpf_testmod_test_struct_arg_result = a + b.a + b.b + c; + return (void *)bpf_testmod_test_struct_arg_result; +} + +noinline int +bpf_testmod_test_struct_arg_12(int a, struct bpf_testmod_struct_arg_2 b, int *c) { + bpf_testmod_test_struct_arg_result = a + b.a + b.b + *c; + return bpf_testmod_test_struct_arg_result; +} + noinline int bpf_testmod_test_arg_ptr_to_struct(struct bpf_testmod_struct_arg_1 *a) { bpf_testmod_test_struct_arg_result = a->a; return bpf_testmod_test_struct_arg_result; } +noinline int +bpf_testmod_test_arg_ptr_1(struct bpf_testmod_struct_arg_1 *a) { + bpf_testmod_test_struct_arg_result = a->a; + return bpf_testmod_test_struct_arg_result; +} + +noinline int +bpf_testmod_test_arg_ptr_2(struct bpf_testmod_struct_arg_2 *a) { + bpf_testmod_test_struct_arg_result = a->a + a->b; + return bpf_testmod_test_struct_arg_result; +} + +noinline int +bpf_testmod_test_arg_ptr_3(int a, struct bpf_testmod_struct_arg_2 *b) { + bpf_testmod_test_struct_arg_result = a + b->a + b->b; + return bpf_testmod_test_struct_arg_result; +} + +noinline int +bpf_testmod_test_arg_ptr_4(struct bpf_testmod_struct_arg_2 *a, int b) { + bpf_testmod_test_struct_arg_result = a->a + a->b + b; + return bpf_testmod_test_struct_arg_result; +} + __bpf_kfunc void bpf_testmod_test_mod_kfunc(int i) { diff --git a/tools/testing/selftests/bpf/prog_tests/tracing_multi_link.c b/tools/testing/selftests/bpf/prog_tests/tracing_multi_link.c new file mode 100644 index 000000000000..61701a5b3494 --- /dev/null +++ b/tools/testing/selftests/bpf/prog_tests/tracing_multi_link.c @@ -0,0 +1,153 @@ +// SPDX-License-Identifier: GPL-2.0 +/* Copyright (c) 2024 Bytedance. */ + +#include +#include "tracing_multi_test.skel.h" + +static void test_skel_auto_api(void) +{ + struct tracing_multi_test *skel; + int err; + + skel = tracing_multi_test__open_and_load(); + if (!ASSERT_OK_PTR(skel, "tracing_multi_test__open_and_load")) + return; + + /* disable all programs that should fail */ + bpf_program__set_autoattach(skel->progs.fentry_fail_test1, false); + bpf_program__set_autoattach(skel->progs.fentry_fail_test2, false); + bpf_program__set_autoattach(skel->progs.fentry_fail_test3, false); + bpf_program__set_autoattach(skel->progs.fentry_fail_test4, false); + bpf_program__set_autoattach(skel->progs.fentry_fail_test5, false); + bpf_program__set_autoattach(skel->progs.fentry_fail_test6, false); + bpf_program__set_autoattach(skel->progs.fentry_fail_test7, false); + bpf_program__set_autoattach(skel->progs.fentry_fail_test8, false); + + bpf_program__set_autoattach(skel->progs.fexit_fail_test1, false); + bpf_program__set_autoattach(skel->progs.fexit_fail_test2, false); + bpf_program__set_autoattach(skel->progs.fexit_fail_test3, false); + + err = tracing_multi_test__attach(skel); + bpf_object__free_btfs(skel->obj); + if (!ASSERT_OK(err, "tracing_multi_test__attach")) + goto cleanup; + +cleanup: + tracing_multi_test__destroy(skel); +} + +static void test_skel_manual_api(void) +{ + struct tracing_multi_test *skel; + struct bpf_link *link; + int err; + + skel = tracing_multi_test__open_and_load(); + if (!ASSERT_OK_PTR(skel, "tracing_multi_test__open_and_load")) + return; + +#define RUN_TEST(name, success) \ +do { \ + link = bpf_program__attach(skel->progs.name); \ + err = libbpf_get_error(link); \ + if (!ASSERT_OK(success ? err : !err, \ + "bpf_program__attach: " #name)) \ + goto cleanup; \ + skel->links.name = err ? NULL : link; \ +} while (0) + + RUN_TEST(fentry_success_test1, true); + RUN_TEST(fentry_success_test2, true); + RUN_TEST(fentry_success_test3, true); + RUN_TEST(fentry_success_test4, true); + RUN_TEST(fentry_success_test5, true); + + RUN_TEST(fexit_success_test1, true); + RUN_TEST(fexit_success_test2, true); + + RUN_TEST(fmod_ret_success_test1, true); + + RUN_TEST(fentry_fail_test1, false); + RUN_TEST(fentry_fail_test2, false); + RUN_TEST(fentry_fail_test3, false); + RUN_TEST(fentry_fail_test4, false); + RUN_TEST(fentry_fail_test5, false); + RUN_TEST(fentry_fail_test6, false); + RUN_TEST(fentry_fail_test7, false); + RUN_TEST(fentry_fail_test8, false); + + RUN_TEST(fexit_fail_test1, false); + RUN_TEST(fexit_fail_test2, false); + RUN_TEST(fexit_fail_test3, false); + +cleanup: + tracing_multi_test__destroy(skel); +} + +static void tracing_multi_test_run(struct tracing_multi_test *skel) +{ + LIBBPF_OPTS(bpf_test_run_opts, topts); + int err, prog_fd; + + prog_fd = bpf_program__fd(skel->progs.fentry_manual_test1); + err = bpf_prog_test_run_opts(prog_fd, &topts); + ASSERT_OK(err, "test_run"); + ASSERT_EQ(topts.retval, 0, "test_run"); + + ASSERT_EQ(skel->bss->fentry_test1_result, 1, "fentry_test1_result"); + ASSERT_EQ(skel->bss->fentry_test2_result, 1, "fentry_test2_result"); + ASSERT_EQ(skel->bss->fentry_test3_result, 1, "fentry_test3_result"); + ASSERT_EQ(skel->bss->fentry_test4_result, 1, "fentry_test4_result"); + ASSERT_EQ(skel->bss->fentry_test5_result, 1, "fentry_test5_result"); + ASSERT_EQ(skel->bss->fentry_test6_result, 1, "fentry_test6_result"); + ASSERT_EQ(skel->bss->fentry_test7_result, 1, "fentry_test7_result"); + ASSERT_EQ(skel->bss->fentry_test8_result, 1, "fentry_test8_result"); +} + +static void test_attach_api(void) +{ + LIBBPF_OPTS(bpf_trace_multi_opts, opts); + struct tracing_multi_test *skel; + struct bpf_link *link; + const char *syms[8] = { + "bpf_fentry_test1", + "bpf_fentry_test2", + "bpf_fentry_test3", + "bpf_fentry_test4", + "bpf_fentry_test5", + "bpf_fentry_test6", + "bpf_fentry_test7", + "bpf_fentry_test8", + }; + __u64 cookies[] = {1, 7, 2, 3, 4, 5, 6, 8}; + + skel = tracing_multi_test__open_and_load(); + if (!ASSERT_OK_PTR(skel, "tracing_multi_test__open_and_load")) + return; + + opts.syms = syms; + opts.cookies = cookies; + opts.cnt = ARRAY_SIZE(syms); + link = bpf_program__attach_trace_multi_opts(skel->progs.fentry_manual_test1, + &opts); + bpf_object__free_btfs(skel->obj); + if (!ASSERT_OK_PTR(link, "bpf_program__attach_trace_multi_opts")) + goto cleanup; + skel->links.fentry_manual_test1 = link; + + skel->bss->pid = getpid(); + skel->bss->test_cookie = true; + tracing_multi_test_run(skel); +cleanup: + tracing_multi_test__destroy(skel); +} + +void test_tracing_multi_attach(void) +{ + if (test__start_subtest("skel_auto_api")) + test_skel_auto_api(); + if (test__start_subtest("skel_manual_api")) + test_skel_manual_api(); + if (test__start_subtest("attach_api")) + test_attach_api(); +} diff --git a/tools/testing/selftests/bpf/progs/tracing_multi_test.c b/tools/testing/selftests/bpf/progs/tracing_multi_test.c new file mode 100644 index 000000000000..adfa4c2f6ee3 --- /dev/null +++ b/tools/testing/selftests/bpf/progs/tracing_multi_test.c @@ -0,0 +1,209 @@ +// SPDX-License-Identifier: GPL-2.0 +/* Copyright (c) 2024 ByteDance */ +#include +#include +#include +#include "bpf_misc.h" + +char _license[] SEC("license") = "GPL"; + +struct bpf_testmod_struct_arg_1 { + int a; +}; +struct bpf_testmod_struct_arg_2 { + long a; + long b; +}; + +__u64 test_result = 0; + +int pid = 0; +int test_cookie = 0; + +__u64 fentry_test1_result = 0; +__u64 fentry_test2_result = 0; +__u64 fentry_test3_result = 0; +__u64 fentry_test4_result = 0; +__u64 fentry_test5_result = 0; +__u64 fentry_test6_result = 0; +__u64 fentry_test7_result = 0; +__u64 fentry_test8_result = 0; + +extern const void bpf_fentry_test1 __ksym; +extern const void bpf_fentry_test2 __ksym; +extern const void bpf_fentry_test3 __ksym; +extern const void bpf_fentry_test4 __ksym; +extern const void bpf_fentry_test5 __ksym; +extern const void bpf_fentry_test6 __ksym; +extern const void bpf_fentry_test7 __ksym; +extern const void bpf_fentry_test8 __ksym; + +SEC("fentry.multi/bpf_testmod_test_struct_arg_1,bpf_testmod_test_struct_arg_9") +int BPF_PROG2(fentry_success_test1, struct bpf_testmod_struct_arg_2, a) +{ + test_result = a.a + a.b; + return 0; +} + +SEC("fentry.multi/bpf_testmod_test_struct_arg_2,bpf_testmod_test_struct_arg_10") +int BPF_PROG2(fentry_success_test2, int, a, struct bpf_testmod_struct_arg_2, b) +{ + test_result = a + b.a + b.b; + return 0; +} + +SEC("fentry.multi/bpf_testmod_test_arg_ptr_2,bpf_testmod_test_arg_ptr_4") +int BPF_PROG(fentry_success_test3, struct bpf_testmod_struct_arg_2 *a) +{ + test_result = a->a + a->b; + return 0; +} + +SEC("fentry.multi/bpf_testmod_test_struct_arg_1,bpf_testmod_test_struct_arg_4") +int BPF_PROG2(fentry_success_test4, struct bpf_testmod_struct_arg_2, a, int, b, + int, c) +{ + test_result = c; + return 0; +} + +SEC("fentry.multi/bpf_testmod_test_struct_arg_1,bpf_testmod_test_struct_arg_2") +int BPF_PROG2(fentry_success_test5, struct bpf_testmod_struct_arg_2, a, int, b, + int, c) +{ + test_result = c; + return 0; +} + +SEC("fentry.multi/bpf_testmod_test_struct_arg_1,bpf_testmod_test_struct_arg_1") +int BPF_PROG2(fentry_fail_test1, struct bpf_testmod_struct_arg_2, a) +{ + test_result = a.a + a.b; + return 0; +} + +SEC("fentry.multi/bpf_testmod_test_struct_arg_1,bpf_testmod_test_struct_arg_2") +int BPF_PROG2(fentry_fail_test2, struct bpf_testmod_struct_arg_2, a) +{ + test_result = a.a + a.b; + return 0; +} + +SEC("fentry.multi/bpf_testmod_test_struct_arg_1,bpf_testmod_test_arg_ptr_2") +int BPF_PROG2(fentry_fail_test3, struct bpf_testmod_struct_arg_2, a) +{ + test_result = a.a + a.b; + return 0; +} + +SEC("fentry.multi/bpf_testmod_test_struct_arg_2,bpf_testmod_test_struct_arg_2") +int BPF_PROG2(fentry_fail_test4, int, a, struct bpf_testmod_struct_arg_2, b) +{ + test_result = a + b.a + b.b; + return 0; +} + +SEC("fentry.multi/bpf_testmod_test_struct_arg_2,bpf_testmod_test_struct_arg_9") +int BPF_PROG2(fentry_fail_test5, int, a, struct bpf_testmod_struct_arg_2, b) +{ + test_result = a + b.a + b.b; + return 0; +} + +SEC("fentry.multi/bpf_testmod_test_struct_arg_2,bpf_testmod_test_arg_ptr_3") +int BPF_PROG2(fentry_fail_test6, int, a, struct bpf_testmod_struct_arg_2, b) +{ + test_result = a + b.a + b.b; + return 0; +} + +SEC("fentry.multi/bpf_testmod_test_arg_ptr_2,bpf_testmod_test_arg_ptr_3") +int BPF_PROG(fentry_fail_test7, struct bpf_testmod_struct_arg_2 *a) +{ + test_result = a->a + a->b; + return 0; +} + +SEC("fentry.multi/bpf_testmod_test_struct_arg_1,bpf_testmod_test_struct_arg_12") +int BPF_PROG2(fentry_fail_test8, struct bpf_testmod_struct_arg_2, a, int, b, + int, c) +{ + test_result = c; + return 0; +} + +SEC("fexit.multi/bpf_testmod_test_struct_arg_1,bpf_testmod_test_struct_arg_2,bpf_testmod_test_struct_arg_3") +int BPF_PROG2(fexit_success_test1, struct bpf_testmod_struct_arg_2, a, int, b, + int, c, int, retval) +{ + test_result = retval; + return 0; +} + +SEC("fexit.multi/bpf_testmod_test_struct_arg_2,bpf_testmod_test_struct_arg_12") +int BPF_PROG2(fexit_success_test2, int, a, struct bpf_testmod_struct_arg_2, b, + int, c, int, retval) +{ + test_result = a + b.a + b.b + retval; + return 0; +} + +SEC("fexit.multi/bpf_testmod_test_struct_arg_1,bpf_testmod_test_struct_arg_4") +int BPF_PROG2(fexit_fail_test1, struct bpf_testmod_struct_arg_2, a, int, b, + int, c, int, retval) +{ + test_result = retval; + return 0; +} + +SEC("fexit.multi/bpf_testmod_test_struct_arg_2,bpf_testmod_test_struct_arg_10") +int BPF_PROG2(fexit_fail_test2, int, a, struct bpf_testmod_struct_arg_2, b, + int, c, int, retval) +{ + test_result = a + b.a + b.b + retval; + return 0; +} + +SEC("fexit.multi/bpf_testmod_test_struct_arg_2,bpf_testmod_test_struct_arg_11") +int BPF_PROG2(fexit_fail_test3, int, a, struct bpf_testmod_struct_arg_2, b, + int, c, int, retval) +{ + test_result = a + b.a + b.b + retval; + return 0; +} + +SEC("fmod_ret.multi/bpf_modify_return_test,bpf_modify_return_test2") +int BPF_PROG(fmod_ret_success_test1, int a, int *b) +{ + return 0; +} + +static void tracing_multi_check(unsigned long long *ctx) +{ + if (bpf_get_current_pid_tgid() >> 32 != pid) + return; + + __u64 cookie = test_cookie ? bpf_get_attach_cookie(ctx) : 0; + __u64 addr = bpf_get_func_ip(ctx); + +#define SET(__var, __addr, __cookie) ({ \ + if (((const void *) addr == __addr) && \ + (!test_cookie || (cookie == __cookie))) \ + __var = 1; \ +}) + SET(fentry_test1_result, &bpf_fentry_test1, 1); + SET(fentry_test2_result, &bpf_fentry_test2, 7); + SET(fentry_test3_result, &bpf_fentry_test3, 2); + SET(fentry_test4_result, &bpf_fentry_test4, 3); + SET(fentry_test5_result, &bpf_fentry_test5, 4); + SET(fentry_test6_result, &bpf_fentry_test6, 5); + SET(fentry_test7_result, &bpf_fentry_test7, 6); + SET(fentry_test8_result, &bpf_fentry_test8, 8); +} + +SEC("fentry.multi/bpf_fentry_test1") +int BPF_PROG(fentry_manual_test1) +{ + tracing_multi_check(ctx); + return 0; +}