From patchwork Tue Jun 13 02:52:24 2023 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Menglong Dong X-Patchwork-Id: 13277830 X-Patchwork-Delegate: bpf@iogearbox.net Received: from lindbergh.monkeyblade.net (lindbergh.monkeyblade.net [23.128.96.19]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (No client certificate requested) by smtp.subspace.kernel.org (Postfix) with ESMTPS id D1D857F for ; Tue, 13 Jun 2023 02:53:11 +0000 (UTC) Received: from mail-pj1-x1042.google.com (mail-pj1-x1042.google.com [IPv6:2607:f8b0:4864:20::1042]) by lindbergh.monkeyblade.net (Postfix) with ESMTPS id 8BA62EE; Mon, 12 Jun 2023 19:53:10 -0700 (PDT) Received: by mail-pj1-x1042.google.com with SMTP id 98e67ed59e1d1-25bf9dadfe8so1010598a91.2; Mon, 12 Jun 2023 19:53:10 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=20221208; t=1686624790; x=1689216790; h=content-transfer-encoding:mime-version:references:in-reply-to :message-id:date:subject:cc:to:from:from:to:cc:subject:date :message-id:reply-to; bh=yhfGhyxz9mxcYU3gu8XgR3Vlbsf30YJwKDPoiQbXWnM=; b=sL1tTQz4lw+fvdYDYr1RQKRj7JRGn5jb4frHHtc2pkSggplruhRo8YEK+qA/XMsPPI MRFbBf2ef9fZKHk8OuFR6CkvtO+2bbCIx0Qm4kQjIKfoabyrhvGkKKU0lXlBiiQhPnjw huwd7Xp2gzmw1RmcTWyoh1NQIvoW7jnpOJGEq+MS0J9FI3VzXZ89Nd2iu8NGikDsIhDC RNT9aH774C57KoYrvPNZUFW67Jcdr0/9Od+3uycBFDumFrUFq+uMVz58Ij8zWhUxkoTf iWg4df8Jkm2zvm4CCOA9jswDcrmzoEJDNkziKuWmmlCdnggYJMgjB51TSKIAzF0p6jxt pUBg== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20221208; t=1686624790; x=1689216790; h=content-transfer-encoding:mime-version:references:in-reply-to :message-id:date:subject:cc:to:from:x-gm-message-state:from:to:cc :subject:date:message-id:reply-to; bh=yhfGhyxz9mxcYU3gu8XgR3Vlbsf30YJwKDPoiQbXWnM=; b=ecBEIcy8pNj0HUcqjFMuefZdo0BQSWtWJM6X3Vkk/qghCHDyXafsJqDy7yGJd1paf4 gVuOXHxKYli9AsNaLG5sdC94OwXDygqJGlN2W2EH9BjIXNLBxt8Gx2nsT6W62RdSX47E Acl70qzxEFJsHvwRFH/hFwADtLQgiDd4zqjJc0DVfc0V9cagMz0fClFKgb1Sl54qZSv1 ZiXEbjFXMMdzpQpkiJAr6RiGWGyY4dUNG1qpM09vxk90ioyTbcIfxT+Wq4u940KuJC6J HbMe2h82shUc0LpV0WkVKbzIXUMpH0NpeC7Yefea/a2dHEoJruizaJ2HmgxsNG+ldkO6 7z7A== X-Gm-Message-State: AC+VfDxhpQJLlx+m8o/R/68T0vcRmbzg0Wh56HULqdrYCbbZTRlrwknT sKueXW7VHyUQ23gTuDJWmvs= X-Google-Smtp-Source: ACHHUZ4Q49fgtFu0mLg2pIRWhivEkTAPXLUVuTZ/wzxXWIUnmlQe2OfYhNAy9c9Bns//94X8IsjuEQ== X-Received: by 2002:a17:90b:4d83:b0:253:30e1:7d68 with SMTP id oj3-20020a17090b4d8300b0025330e17d68mr9890409pjb.0.1686624789918; Mon, 12 Jun 2023 19:53:09 -0700 (PDT) Received: from localhost.localdomain ([203.205.141.85]) by smtp.gmail.com with ESMTPSA id v65-20020a632f44000000b00543e9e17207sm8240207pgv.30.2023.06.12.19.53.05 (version=TLS1_3 cipher=TLS_AES_256_GCM_SHA384 bits=256/256); Mon, 12 Jun 2023 19:53:09 -0700 (PDT) From: menglong8.dong@gmail.com X-Google-Original-From: imagedong@tencent.com To: yhs@meta.com, alexei.starovoitov@gmail.com Cc: ast@kernel.org, daniel@iogearbox.net, andrii@kernel.org, martin.lau@linux.dev, song@kernel.org, yhs@fb.com, john.fastabend@gmail.com, kpsingh@kernel.org, sdf@google.com, haoluo@google.com, jolsa@kernel.org, benbjiang@tencent.com, bpf@vger.kernel.org, linux-kernel@vger.kernel.org, Menglong Dong Subject: [PATCH bpf-next v5 1/3] bpf, x86: clean garbage values when store args from regs into stack Date: Tue, 13 Jun 2023 10:52:24 +0800 Message-Id: <20230613025226.3167956-2-imagedong@tencent.com> X-Mailer: git-send-email 2.40.1 In-Reply-To: <20230613025226.3167956-1-imagedong@tencent.com> References: <20230613025226.3167956-1-imagedong@tencent.com> Precedence: bulk X-Mailing-List: bpf@vger.kernel.org List-Id: List-Subscribe: List-Unsubscribe: MIME-Version: 1.0 X-Spam-Status: No, score=-2.1 required=5.0 tests=BAYES_00,DKIM_SIGNED, DKIM_VALID,DKIM_VALID_AU,DKIM_VALID_EF,FREEMAIL_FROM, RCVD_IN_DNSWL_NONE,SPF_HELO_NONE,SPF_PASS,T_SCC_BODY_TEXT_LINE autolearn=ham autolearn_force=no version=3.4.6 X-Spam-Checker-Version: SpamAssassin 3.4.6 (2021-04-09) on lindbergh.monkeyblade.net X-Patchwork-Delegate: bpf@iogearbox.net From: Menglong Dong There are garbage values in upper bytes when we store the arguments into stack in save_regs() if the size of the argument less then 8. As we already reserve 8 byte for the arguments in regs and stack, it is ok to store/restore the regs in BPF_DW size. Then, the garbage values in upper bytes will be cleaned. Signed-off-by: Menglong Dong Acked-by: Yonghong Song --- arch/x86/net/bpf_jit_comp.c | 35 ++++++----------------------------- 1 file changed, 6 insertions(+), 29 deletions(-) diff --git a/arch/x86/net/bpf_jit_comp.c b/arch/x86/net/bpf_jit_comp.c index 1056bbf55b17..a407fbbffecd 100644 --- a/arch/x86/net/bpf_jit_comp.c +++ b/arch/x86/net/bpf_jit_comp.c @@ -1860,57 +1860,34 @@ st: if (is_imm8(insn->off)) static void save_regs(const struct btf_func_model *m, u8 **prog, int nr_regs, int stack_size) { - int i, j, arg_size; - bool next_same_struct = false; + int i; /* Store function arguments to stack. * For a function that accepts two pointers the sequence will be: * mov QWORD PTR [rbp-0x10],rdi * mov QWORD PTR [rbp-0x8],rsi */ - for (i = 0, j = 0; i < min(nr_regs, 6); i++) { - /* The arg_size is at most 16 bytes, enforced by the verifier. */ - arg_size = m->arg_size[j]; - if (arg_size > 8) { - arg_size = 8; - next_same_struct = !next_same_struct; - } - - emit_stx(prog, bytes_to_bpf_size(arg_size), - BPF_REG_FP, + for (i = 0; i < min(nr_regs, 6); i++) + emit_stx(prog, BPF_DW, BPF_REG_FP, i == 5 ? X86_REG_R9 : BPF_REG_1 + i, -(stack_size - i * 8)); - - j = next_same_struct ? j : j + 1; - } } static void restore_regs(const struct btf_func_model *m, u8 **prog, int nr_regs, int stack_size) { - int i, j, arg_size; - bool next_same_struct = false; + int i; /* Restore function arguments from stack. * For a function that accepts two pointers the sequence will be: * EMIT4(0x48, 0x8B, 0x7D, 0xF0); mov rdi,QWORD PTR [rbp-0x10] * EMIT4(0x48, 0x8B, 0x75, 0xF8); mov rsi,QWORD PTR [rbp-0x8] */ - for (i = 0, j = 0; i < min(nr_regs, 6); i++) { - /* The arg_size is at most 16 bytes, enforced by the verifier. */ - arg_size = m->arg_size[j]; - if (arg_size > 8) { - arg_size = 8; - next_same_struct = !next_same_struct; - } - - emit_ldx(prog, bytes_to_bpf_size(arg_size), + for (i = 0; i < min(nr_regs, 6); i++) + emit_ldx(prog, BPF_DW, i == 5 ? X86_REG_R9 : BPF_REG_1 + i, BPF_REG_FP, -(stack_size - i * 8)); - - j = next_same_struct ? j : j + 1; - } } static int invoke_bpf_prog(const struct btf_func_model *m, u8 **pprog, From patchwork Tue Jun 13 02:52:25 2023 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Menglong Dong X-Patchwork-Id: 13277831 X-Patchwork-Delegate: bpf@iogearbox.net Received: from lindbergh.monkeyblade.net (lindbergh.monkeyblade.net [23.128.96.19]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (No client certificate requested) by smtp.subspace.kernel.org (Postfix) with ESMTPS id 04DC97F for ; Tue, 13 Jun 2023 02:53:18 +0000 (UTC) Received: from mail-pf1-x444.google.com (mail-pf1-x444.google.com [IPv6:2607:f8b0:4864:20::444]) by lindbergh.monkeyblade.net (Postfix) with ESMTPS id 1B9811734; Mon, 12 Jun 2023 19:53:15 -0700 (PDT) Received: by mail-pf1-x444.google.com with SMTP id d2e1a72fcca58-652d76be8c2so5492384b3a.3; Mon, 12 Jun 2023 19:53:15 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=20221208; t=1686624794; x=1689216794; h=content-transfer-encoding:mime-version:references:in-reply-to :message-id:date:subject:cc:to:from:from:to:cc:subject:date :message-id:reply-to; bh=h3R3RZvz5t7u2FrRguPhM7I6OHwZlHgebZcb5fE7XJc=; b=Jrmhly/VsdcsiEtSClZcmP0QeCJIBhwUKHTVU3hqABM/2sgYIv0WSH8OIPl/Lrz+9H xSxLMGF5XH+m0RAdViHnGuTLFluBBdInvo6FdBWwLJKqec9tN52qYEoMoa8HP7ezpOgU l+LqiQ5rXBpiiEh3VV47QO4R7Teis278Z1a6oFlYsjIzSZGOrSbctH+o2yvw8EpJb0iv EYhYrMyiqRDa9XTm41Ng6I12Za2vSFArvTrRJ9KpvL7oclhqdTVeJOMlPguftJkf0U2Q HMvykz/DlxBUUVXGAGrXQPfkSEdg9HA1M+kX23mhDbh+tYieHaKIRktR0Xn6PKilnCTV FtAQ== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20221208; t=1686624794; x=1689216794; h=content-transfer-encoding:mime-version:references:in-reply-to :message-id:date:subject:cc:to:from:x-gm-message-state:from:to:cc :subject:date:message-id:reply-to; bh=h3R3RZvz5t7u2FrRguPhM7I6OHwZlHgebZcb5fE7XJc=; b=XwNiC43N66eukpvHF/SNO3WX7+tCvGB9hxNTRisqSE4hn5NPx9nDw42cW5jwJ4p5aL pVTj2gJVDnT7PsuH7Z04U4t9pE8XShTOHmYNJM7s5lv5bDUUJydoKzMzbgGUk6vGJOA5 sqmZP8ed1gSdsIgYJx9VWunV5D7VJXu6PNTXvx+6taolMfVvyNm7WtvOea2Q1Qxe1FpD 8DjSWKUc4IA3Hx27q06yphSz4pOE4KwxGAXGGoqG7dHUCR+skP/jqtnZja1DpvB0vSrl 5nLn75b2/7Z/0KDb1JaPfQx4pG3VQZDwmLFj2n7tORHUVDYKvrDkJ1NVDD8foOsCDJcN eaAw== X-Gm-Message-State: AC+VfDwHkyyog8fF3t8Ly8HDXmUl4C7hobmA3dYRrwY2PVcIviIsoDvV ty94pa5FI0sS0AD675b3nLXWLXZF2nyhbDgl X-Google-Smtp-Source: ACHHUZ7Ae8rgG3HTB+RJK0Rj+PIIUgHf2clxuOrhMiezp1XjGMD1BUNgWk1R0tEA+3PO576JBZhGOg== X-Received: by 2002:a05:6a20:9382:b0:101:4e04:cef1 with SMTP id x2-20020a056a20938200b001014e04cef1mr13666481pzh.27.1686624794260; Mon, 12 Jun 2023 19:53:14 -0700 (PDT) Received: from localhost.localdomain ([203.205.141.85]) by smtp.gmail.com with ESMTPSA id v65-20020a632f44000000b00543e9e17207sm8240207pgv.30.2023.06.12.19.53.10 (version=TLS1_3 cipher=TLS_AES_256_GCM_SHA384 bits=256/256); Mon, 12 Jun 2023 19:53:13 -0700 (PDT) From: menglong8.dong@gmail.com X-Google-Original-From: imagedong@tencent.com To: yhs@meta.com, alexei.starovoitov@gmail.com Cc: ast@kernel.org, daniel@iogearbox.net, andrii@kernel.org, martin.lau@linux.dev, song@kernel.org, yhs@fb.com, john.fastabend@gmail.com, kpsingh@kernel.org, sdf@google.com, haoluo@google.com, jolsa@kernel.org, benbjiang@tencent.com, bpf@vger.kernel.org, linux-kernel@vger.kernel.org, Menglong Dong Subject: [PATCH bpf-next v5 2/3] bpf, x86: allow function arguments up to 12 for TRACING Date: Tue, 13 Jun 2023 10:52:25 +0800 Message-Id: <20230613025226.3167956-3-imagedong@tencent.com> X-Mailer: git-send-email 2.40.1 In-Reply-To: <20230613025226.3167956-1-imagedong@tencent.com> References: <20230613025226.3167956-1-imagedong@tencent.com> Precedence: bulk X-Mailing-List: bpf@vger.kernel.org List-Id: List-Subscribe: List-Unsubscribe: MIME-Version: 1.0 X-Spam-Status: No, score=-2.1 required=5.0 tests=BAYES_00,DKIM_SIGNED, DKIM_VALID,DKIM_VALID_AU,DKIM_VALID_EF,FREEMAIL_FROM, RCVD_IN_DNSWL_NONE,SPF_HELO_NONE,SPF_PASS,T_SCC_BODY_TEXT_LINE autolearn=ham autolearn_force=no version=3.4.6 X-Spam-Checker-Version: SpamAssassin 3.4.6 (2021-04-09) on lindbergh.monkeyblade.net X-Patchwork-Delegate: bpf@iogearbox.net From: Menglong Dong For now, the BPF program of type BPF_PROG_TYPE_TRACING can only be used on the kernel functions whose arguments count less than 6. This is not friendly at all, as too many functions have arguments count more than 6. According to the current kernel version, below is a statistics of the function arguments count: argument count | function count 7 | 704 8 | 270 9 | 84 10 | 47 11 | 47 12 | 27 13 | 22 14 | 5 15 | 0 16 | 1 Therefore, let's enhance it by increasing the function arguments count allowed in arch_prepare_bpf_trampoline(), for now, only x86_64. For the case that we don't need to call origin function, which means without BPF_TRAMP_F_CALL_ORIG, we need only copy the function arguments that stored in the frame of the caller to current frame. The arguments of arg6-argN are stored in "$rbp + 0x18", we need copy them to "$rbp - regs_off + (6 * 8)". For the case with BPF_TRAMP_F_CALL_ORIG, we need prepare the arguments in stack before call origin function, which means we need alloc extra "8 * (arg_count - 6)" memory in the top of the stack. Note, there should not be any data be pushed to the stack before call the origin function. Then, we have to store rbx with 'mov' instead of 'push'. We use EMIT3_off32() or EMIT4() for "lea" and "sub". The range of the imm in "lea" and "sub" is [-128, 127] if EMIT4() is used. Therefore, we use EMIT3_off32() instead if the imm out of the range. It works well for the FENTRY/FEXIT/MODIFY_RETURN. Signed-off-by: Menglong Dong --- v5: - consider the case of the struct in arguments can't be hold by regs v4: - make the stack 16-byte aligned if passing args on-stack is needed - add the function arguments statistics to the commit log v3: - use EMIT3_off32() for "lea" and "sub" only on necessary - make 12 as the maximum arguments count v2: - instead EMIT4 with EMIT3_off32 for "lea" to prevent overflow - make MAX_BPF_FUNC_ARGS as the maximum argument count --- arch/x86/net/bpf_jit_comp.c | 221 +++++++++++++++++++++++++++++++----- 1 file changed, 195 insertions(+), 26 deletions(-) diff --git a/arch/x86/net/bpf_jit_comp.c b/arch/x86/net/bpf_jit_comp.c index a407fbbffecd..47c699594dd8 100644 --- a/arch/x86/net/bpf_jit_comp.c +++ b/arch/x86/net/bpf_jit_comp.c @@ -1857,37 +1857,165 @@ st: if (is_imm8(insn->off)) return proglen; } -static void save_regs(const struct btf_func_model *m, u8 **prog, int nr_regs, - int stack_size) +static inline void clean_stack_garbage(const struct btf_func_model *m, + u8 **pprog, int nr_args_on_stack, + int stack_size) { - int i; + int arg_size, off; + u8 *prog; + + if (nr_args_on_stack != 1) + return; + + /* the size of the last argument */ + arg_size = m->arg_size[m->nr_args - 1]; + + /* Generally speaking, the compiler will pass the arguments + * on-stack with "push" instruction, which will take 8-byte + * on the stack. On this case, there won't be garbage values + * while we copy the arguments from origin stack frame to current + * in BPF_DW. + * + * However, sometimes the compiler will only allocate 4-byte on + * the stack for the arguments. For now, this case will only + * happen if there is only one argument on-stack and its size + * not more than 4 byte. On this case, there will be garbage + * values on the upper 4-byte where we store the argument on + * current stack frame. + * + * arguments on origin stack: + * + * stack_arg_1(4-byte) xxx(4-byte) + * + * what we copy: + * + * stack_arg_1(8-byte): stack_arg_1(origin) xxx + * + * and the xxx is the garbage values which we should clean here. + */ + if (arg_size <= 4) { + off = -(stack_size - 4); + prog = *pprog; + /* mov DWORD PTR [rbp + off], 0 */ + if (!is_imm8(off)) + EMIT2_off32(0xC7, 0x85, off); + else + EMIT3(0xC7, 0x45, off); + EMIT(0, 4); + *pprog = prog; + } +} + +static void save_args(const struct btf_func_model *m, u8 **prog, + int stack_size, bool on_stack) +{ + int arg_regs, first_off, nr_regs = 0, nr_stack = 0; + int i, j; /* Store function arguments to stack. * For a function that accepts two pointers the sequence will be: * mov QWORD PTR [rbp-0x10],rdi * mov QWORD PTR [rbp-0x8],rsi */ - for (i = 0; i < min(nr_regs, 6); i++) - emit_stx(prog, BPF_DW, BPF_REG_FP, - i == 5 ? X86_REG_R9 : BPF_REG_1 + i, - -(stack_size - i * 8)); + for (i = 0; i < min_t(int, m->nr_args, MAX_BPF_FUNC_ARGS); i++) { + arg_regs = (m->arg_size[i] + 7) / 8; + + /* According to the research of Yonghong, struct members + * should be all in register or all on the stack. + * Meanwhile, the compiler will pass the argument on regs + * if the remained regs can hold the argument. + * + * Disorder of the args can happen. For example: + * + * struct foo_struct { + * long a; + * int b; + * }; + * int foo(char, char, char, char, char, struct foo_struct, + * char); + * + * the arg1-5,arg7 will be passed by regs, and arg6 will + * by stack. + * + * Therefore, we should keep the same logic as here when + * we restore the regs in restore_regs. + */ + if (nr_regs + arg_regs > 6) { + /* copy function arguments from origin stack frame + * into current stack frame. + * + * The starting address of the arguments on-stack + * is: + * rbp + 8(push rbp) + + * 8(return addr of origin call) + + * 8(return addr of the caller) + * which means: rbp + 24 + */ + for (j = 0; j < arg_regs; j++) { + emit_ldx(prog, BPF_DW, BPF_REG_0, BPF_REG_FP, + nr_stack * 8 + 0x18); + emit_stx(prog, BPF_DW, BPF_REG_FP, BPF_REG_0, + -stack_size); + + if (!nr_stack) + first_off = stack_size; + stack_size -= 8; + nr_stack++; + } + } else { + /* Only copy the arguments on-stack to current + * 'stack_size' and ignore the regs, used to + * prepare the arguments on-stack for orign call. + */ + if (on_stack) { + nr_regs += arg_regs; + continue; + } + + /* copy the arguments from regs into stack */ + for (j = 0; j < arg_regs; j++) { + emit_stx(prog, BPF_DW, BPF_REG_FP, + nr_regs == 5 ? X86_REG_R9 : BPF_REG_1 + i, + -stack_size); + stack_size -= 8; + nr_regs++; + } + } + } + + clean_stack_garbage(m, prog, nr_stack, first_off); } -static void restore_regs(const struct btf_func_model *m, u8 **prog, int nr_regs, +static void restore_regs(const struct btf_func_model *m, u8 **prog, int stack_size) { - int i; + int i, j, arg_regs, nr_regs = 0; /* Restore function arguments from stack. * For a function that accepts two pointers the sequence will be: * EMIT4(0x48, 0x8B, 0x7D, 0xF0); mov rdi,QWORD PTR [rbp-0x10] * EMIT4(0x48, 0x8B, 0x75, 0xF8); mov rsi,QWORD PTR [rbp-0x8] + * + * The logic here is similar to what we do in save_args() */ - for (i = 0; i < min(nr_regs, 6); i++) - emit_ldx(prog, BPF_DW, - i == 5 ? X86_REG_R9 : BPF_REG_1 + i, - BPF_REG_FP, - -(stack_size - i * 8)); + for (i = 0; i < min_t(int, m->nr_args, MAX_BPF_FUNC_ARGS); i++) { + arg_regs = (m->arg_size[i] + 7) / 8; + if (nr_regs + arg_regs <= 6) { + for (j = 0; j < arg_regs; j++) { + emit_ldx(prog, BPF_DW, + nr_regs == 5 ? X86_REG_R9 : BPF_REG_1 + i, + BPF_REG_FP, + -stack_size); + stack_size -= 8; + nr_regs++; + } + } else { + stack_size -= 8 * arg_regs; + } + + if (nr_regs >= 6) + break; + } } static int invoke_bpf_prog(const struct btf_func_model *m, u8 **pprog, @@ -1915,7 +2043,10 @@ static int invoke_bpf_prog(const struct btf_func_model *m, u8 **pprog, /* arg1: mov rdi, progs[i] */ emit_mov_imm64(&prog, BPF_REG_1, (long) p >> 32, (u32) (long) p); /* arg2: lea rsi, [rbp - ctx_cookie_off] */ - EMIT4(0x48, 0x8D, 0x75, -run_ctx_off); + if (!is_imm8(-run_ctx_off)) + EMIT3_off32(0x48, 0x8D, 0xB5, -run_ctx_off); + else + EMIT4(0x48, 0x8D, 0x75, -run_ctx_off); if (emit_rsb_call(&prog, bpf_trampoline_enter(p), prog)) return -EINVAL; @@ -1931,7 +2062,10 @@ static int invoke_bpf_prog(const struct btf_func_model *m, u8 **pprog, emit_nops(&prog, 2); /* arg1: lea rdi, [rbp - stack_size] */ - EMIT4(0x48, 0x8D, 0x7D, -stack_size); + if (!is_imm8(-stack_size)) + EMIT3_off32(0x48, 0x8D, 0xBD, -stack_size); + else + EMIT4(0x48, 0x8D, 0x7D, -stack_size); /* arg2: progs[i]->insnsi for interpreter */ if (!p->jited) emit_mov_imm64(&prog, BPF_REG_2, @@ -1961,7 +2095,10 @@ static int invoke_bpf_prog(const struct btf_func_model *m, u8 **pprog, /* arg2: mov rsi, rbx <- start time in nsec */ emit_mov_reg(&prog, true, BPF_REG_2, BPF_REG_6); /* arg3: lea rdx, [rbp - run_ctx_off] */ - EMIT4(0x48, 0x8D, 0x55, -run_ctx_off); + if (!is_imm8(-run_ctx_off)) + EMIT3_off32(0x48, 0x8D, 0x95, -run_ctx_off); + else + EMIT4(0x48, 0x8D, 0x55, -run_ctx_off); if (emit_rsb_call(&prog, bpf_trampoline_exit(p), prog)) return -EINVAL; @@ -2113,7 +2250,7 @@ int arch_prepare_bpf_trampoline(struct bpf_tramp_image *im, void *image, void *i void *func_addr) { int i, ret, nr_regs = m->nr_args, stack_size = 0; - int regs_off, nregs_off, ip_off, run_ctx_off; + int regs_off, nregs_off, ip_off, run_ctx_off, arg_stack_off, rbx_off; struct bpf_tramp_links *fentry = &tlinks[BPF_TRAMP_FENTRY]; struct bpf_tramp_links *fexit = &tlinks[BPF_TRAMP_FEXIT]; struct bpf_tramp_links *fmod_ret = &tlinks[BPF_TRAMP_MODIFY_RETURN]; @@ -2127,8 +2264,10 @@ int arch_prepare_bpf_trampoline(struct bpf_tramp_image *im, void *image, void *i if (m->arg_flags[i] & BTF_FMODEL_STRUCT_ARG) nr_regs += (m->arg_size[i] + 7) / 8 - 1; - /* x86-64 supports up to 6 arguments. 7+ can be added in the future */ - if (nr_regs > 6) + /* x86-64 supports up to MAX_BPF_FUNC_ARGS arguments. 1-6 + * are passed through regs, the remains are through stack. + */ + if (nr_regs > MAX_BPF_FUNC_ARGS) return -ENOTSUPP; /* Generated trampoline stack layout: @@ -2147,7 +2286,14 @@ int arch_prepare_bpf_trampoline(struct bpf_tramp_image *im, void *image, void *i * * RBP - ip_off [ traced function ] BPF_TRAMP_F_IP_ARG flag * + * RBP - rbx_off [ rbx value ] always + * * RBP - run_ctx_off [ bpf_tramp_run_ctx ] + * + * [ stack_argN ] BPF_TRAMP_F_CALL_ORIG + * [ ... ] + * [ stack_arg2 ] + * RBP - arg_stack_off [ stack_arg1 ] */ /* room for return value of orig_call or fentry prog */ @@ -2167,9 +2313,25 @@ int arch_prepare_bpf_trampoline(struct bpf_tramp_image *im, void *image, void *i ip_off = stack_size; + stack_size += 8; + rbx_off = stack_size; + stack_size += (sizeof(struct bpf_tramp_run_ctx) + 7) & ~0x7; run_ctx_off = stack_size; + if (nr_regs > 6 && (flags & BPF_TRAMP_F_CALL_ORIG)) { + stack_size += (nr_regs - 6) * 8; + /* make sure the stack pointer is 16-byte aligned if we + * need pass arguments on stack, which means + * [stack_size + 8(rbp) + 8(rip) + 8(origin rip)] + * should be 16-byte aligned. Following code depend on + * that stack_size is already 8-byte aligned. + */ + stack_size += (stack_size % 16) ? 0 : 8; + } + + arg_stack_off = stack_size; + if (flags & BPF_TRAMP_F_SKIP_FRAME) { /* skip patched call instruction and point orig_call to actual * body of the kernel function. @@ -2189,8 +2351,14 @@ int arch_prepare_bpf_trampoline(struct bpf_tramp_image *im, void *image, void *i x86_call_depth_emit_accounting(&prog, NULL); EMIT1(0x55); /* push rbp */ EMIT3(0x48, 0x89, 0xE5); /* mov rbp, rsp */ - EMIT4(0x48, 0x83, 0xEC, stack_size); /* sub rsp, stack_size */ - EMIT1(0x53); /* push rbx */ + if (!is_imm8(stack_size)) + /* sub rsp, stack_size */ + EMIT3_off32(0x48, 0x81, 0xEC, stack_size); + else + /* sub rsp, stack_size */ + EMIT4(0x48, 0x83, 0xEC, stack_size); + /* mov QWORD PTR [rbp - rbx_off], rbx */ + emit_stx(&prog, BPF_DW, BPF_REG_FP, BPF_REG_6, -rbx_off); /* Store number of argument registers of the traced function: * mov rax, nr_regs @@ -2208,7 +2376,7 @@ int arch_prepare_bpf_trampoline(struct bpf_tramp_image *im, void *image, void *i emit_stx(&prog, BPF_DW, BPF_REG_FP, BPF_REG_0, -ip_off); } - save_regs(m, &prog, nr_regs, regs_off); + save_args(m, &prog, regs_off, false); if (flags & BPF_TRAMP_F_CALL_ORIG) { /* arg1: mov rdi, im */ @@ -2238,7 +2406,8 @@ int arch_prepare_bpf_trampoline(struct bpf_tramp_image *im, void *image, void *i } if (flags & BPF_TRAMP_F_CALL_ORIG) { - restore_regs(m, &prog, nr_regs, regs_off); + restore_regs(m, &prog, regs_off); + save_args(m, &prog, arg_stack_off, true); if (flags & BPF_TRAMP_F_ORIG_STACK) { emit_ldx(&prog, BPF_DW, BPF_REG_0, BPF_REG_FP, 8); @@ -2279,7 +2448,7 @@ int arch_prepare_bpf_trampoline(struct bpf_tramp_image *im, void *image, void *i } if (flags & BPF_TRAMP_F_RESTORE_REGS) - restore_regs(m, &prog, nr_regs, regs_off); + restore_regs(m, &prog, regs_off); /* This needs to be done regardless. If there were fmod_ret programs, * the return value is only updated on the stack and still needs to be @@ -2298,7 +2467,7 @@ int arch_prepare_bpf_trampoline(struct bpf_tramp_image *im, void *image, void *i if (save_ret) emit_ldx(&prog, BPF_DW, BPF_REG_0, BPF_REG_FP, -8); - EMIT1(0x5B); /* pop rbx */ + emit_ldx(&prog, BPF_DW, BPF_REG_6, BPF_REG_FP, -rbx_off); EMIT1(0xC9); /* leave */ if (flags & BPF_TRAMP_F_SKIP_FRAME) /* skip our return address and return to parent */ From patchwork Tue Jun 13 02:52:26 2023 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Menglong Dong X-Patchwork-Id: 13277832 X-Patchwork-Delegate: bpf@iogearbox.net Received: from lindbergh.monkeyblade.net (lindbergh.monkeyblade.net [23.128.96.19]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (No client certificate requested) by smtp.subspace.kernel.org (Postfix) with ESMTPS id 41ABB7F for ; Tue, 13 Jun 2023 02:53:23 +0000 (UTC) Received: from mail-ot1-x342.google.com (mail-ot1-x342.google.com [IPv6:2607:f8b0:4864:20::342]) by lindbergh.monkeyblade.net (Postfix) with ESMTPS id B81651991; Mon, 12 Jun 2023 19:53:19 -0700 (PDT) Received: by mail-ot1-x342.google.com with SMTP id 46e09a7af769-6b280319df5so3334089a34.3; Mon, 12 Jun 2023 19:53:19 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=20221208; t=1686624799; x=1689216799; h=content-transfer-encoding:mime-version:references:in-reply-to :message-id:date:subject:cc:to:from:from:to:cc:subject:date :message-id:reply-to; bh=5v/B2wo+H7MgbJawsb1mh4k2q1GRnaIUu6dzRZ7Tos8=; b=pnbxpEraEGNwAGxwjgFylY/wJw3Y+82r/O8AaT1NJxWJCAKdeg/AFJfeQ8rNzIrbGP H89A3wZAfdIyJRrcPJ3uGuzq51fduRqGX/GgQH7eANzorjGt5lphvk0dMRHlnDqdgPKi snK242eNePtHU2HBvbWHx/tnUIbgBAQYqZ1eN4Bjcq/2cpHcMREjhBNXSse9i5KwO0w4 bPo2tVKzoMeDyU2ECIdq4NJ9LOBed0Vc9kLCHGMj7mRd6SSuoR19oAn8LCzk7vBvtLY7 s/VNBtq1rNqrF80fIkD2oAh4/GZftA2r23v88xUn+i0A6vwYeIUGTKjK1BgRJdto0lPS kwWg== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20221208; t=1686624799; x=1689216799; h=content-transfer-encoding:mime-version:references:in-reply-to :message-id:date:subject:cc:to:from:x-gm-message-state:from:to:cc :subject:date:message-id:reply-to; bh=5v/B2wo+H7MgbJawsb1mh4k2q1GRnaIUu6dzRZ7Tos8=; b=V3k9dP9o7M55NrUfhC3MtzrMd8QJi5X5mrdbz2r753fhjMN98OhjnA3stiw1RyXEdx EqvZnQx7kDJ9RLje8/obg8VRV9n7QLROtREusUhjiw8c8znga3UYY2Ndk8sqqcjcqgmd rcksPfanShtAA7z5aSIBegNtvLZmo0taTwVzc/x6EbJGzyXRkU6EMD0GTaZWYzjV2zzx fTn+7UwN72ThFNygQblEykRlmSG6ohSQWlExYTT0ZgO1hUVv+hX0dbLs6CXoNUl9PILW v3zO6xO3xqQNUWCONnGUMdp8JahPL7v+TFEpWDC9JrIPPv4xZXccBxLYhutZYe2bGZm3 QSYg== X-Gm-Message-State: AC+VfDyw5d3Y4vTx3V7AIPkyhtONqt9g18DiX/vM9nDS6OAvCh9Bbjoz EXLM+bdZgLC8Q3VSZLVpwME= X-Google-Smtp-Source: ACHHUZ7/LjTYqreCxkD/iajL2aXNjjt6bB23h7ynycSQnHiPxHLqoGdpTOYv2uRVJ/PmVss18MJxlg== X-Received: by 2002:a05:6830:1007:b0:6b0:cfce:6d1f with SMTP id a7-20020a056830100700b006b0cfce6d1fmr8613930otp.24.1686624798783; Mon, 12 Jun 2023 19:53:18 -0700 (PDT) Received: from localhost.localdomain ([203.205.141.85]) by smtp.gmail.com with ESMTPSA id v65-20020a632f44000000b00543e9e17207sm8240207pgv.30.2023.06.12.19.53.14 (version=TLS1_3 cipher=TLS_AES_256_GCM_SHA384 bits=256/256); Mon, 12 Jun 2023 19:53:18 -0700 (PDT) From: menglong8.dong@gmail.com X-Google-Original-From: imagedong@tencent.com To: yhs@meta.com, alexei.starovoitov@gmail.com Cc: ast@kernel.org, daniel@iogearbox.net, andrii@kernel.org, martin.lau@linux.dev, song@kernel.org, yhs@fb.com, john.fastabend@gmail.com, kpsingh@kernel.org, sdf@google.com, haoluo@google.com, jolsa@kernel.org, benbjiang@tencent.com, bpf@vger.kernel.org, linux-kernel@vger.kernel.org, Menglong Dong Subject: [PATCH bpf-next v5 3/3] selftests/bpf: add testcase for TRACING with 6+ arguments Date: Tue, 13 Jun 2023 10:52:26 +0800 Message-Id: <20230613025226.3167956-4-imagedong@tencent.com> X-Mailer: git-send-email 2.40.1 In-Reply-To: <20230613025226.3167956-1-imagedong@tencent.com> References: <20230613025226.3167956-1-imagedong@tencent.com> Precedence: bulk X-Mailing-List: bpf@vger.kernel.org List-Id: List-Subscribe: List-Unsubscribe: MIME-Version: 1.0 X-Spam-Status: No, score=-2.1 required=5.0 tests=BAYES_00,DKIM_SIGNED, DKIM_VALID,DKIM_VALID_AU,DKIM_VALID_EF,FREEMAIL_FROM, RCVD_IN_DNSWL_NONE,SPF_HELO_NONE,SPF_PASS,T_SCC_BODY_TEXT_LINE autolearn=ham autolearn_force=no version=3.4.6 X-Spam-Checker-Version: SpamAssassin 3.4.6 (2021-04-09) on lindbergh.monkeyblade.net X-Patchwork-Delegate: bpf@iogearbox.net From: Menglong Dong Add test9/test10 in fexit_test.c and fentry_test.c to test the fentry and fexit whose target function have 7/11 arguments. Correspondingly, add bpf_testmod_fentry_test7() and bpf_testmod_fentry_test11() to bpf_testmod.c Meanwhile, add bpf_modify_return_test2() to test_run.c to test the MODIFY_RETURN with 7 arguments. And the testcases passed: ./test_progs -t fexit Summary: 5/12 PASSED, 0 SKIPPED, 0 FAILED ./test_progs -t fentry Summary: 3/0 PASSED, 0 SKIPPED, 0 FAILED ./test_progs -t modify_return Summary: 1/0 PASSED, 0 SKIPPED, 0 FAILED Signed-off-by: Menglong Dong --- v5: - add testcases for MODIFY_RETURN v4: - use different type for args in bpf_testmod_fentry_test{7,12} - add testcase for grabage values in ctx v3: - move bpf_fentry_test{7,12} to bpf_testmod.c and rename them to bpf_testmod_fentry_test{7,12} meanwhile - get return value by bpf_get_func_ret() in "fexit/bpf_testmod_fentry_test12", as we don't change ___bpf_ctx_cast() in this version --- net/bpf/test_run.c | 23 ++++++-- .../selftests/bpf/bpf_testmod/bpf_testmod.c | 45 +++++++++++++++- .../selftests/bpf/prog_tests/fentry_fexit.c | 4 +- .../selftests/bpf/prog_tests/fentry_test.c | 2 + .../selftests/bpf/prog_tests/fexit_test.c | 2 + .../selftests/bpf/prog_tests/modify_return.c | 20 ++++++- .../testing/selftests/bpf/progs/fentry_test.c | 52 ++++++++++++++++++ .../testing/selftests/bpf/progs/fexit_test.c | 54 +++++++++++++++++++ .../selftests/bpf/progs/modify_return.c | 40 ++++++++++++++ 9 files changed, 235 insertions(+), 7 deletions(-) diff --git a/net/bpf/test_run.c b/net/bpf/test_run.c index 2321bd2f9964..df58e8bf5e07 100644 --- a/net/bpf/test_run.c +++ b/net/bpf/test_run.c @@ -561,6 +561,13 @@ __bpf_kfunc int bpf_modify_return_test(int a, int *b) return a + *b; } +__bpf_kfunc int bpf_modify_return_test2(int a, int *b, short c, int d, + void *e, char f, int g) +{ + *b += 1; + return a + *b + c + d + (long)e + f + g; +} + int noinline bpf_fentry_shadow_test(int a) { return a + 1; @@ -596,9 +603,13 @@ __diag_pop(); BTF_SET8_START(bpf_test_modify_return_ids) BTF_ID_FLAGS(func, bpf_modify_return_test) +BTF_ID_FLAGS(func, bpf_modify_return_test2) BTF_ID_FLAGS(func, bpf_fentry_test1, KF_SLEEPABLE) BTF_SET8_END(bpf_test_modify_return_ids) +BTF_ID_LIST(bpf_modify_return_test_id) +BTF_ID(func, bpf_modify_return_test) + static const struct btf_kfunc_id_set bpf_test_modify_return_set = { .owner = THIS_MODULE, .set = &bpf_test_modify_return_ids, @@ -661,9 +672,15 @@ int bpf_prog_test_run_tracing(struct bpf_prog *prog, goto out; break; case BPF_MODIFY_RETURN: - ret = bpf_modify_return_test(1, &b); - if (b != 2) - side_effect = 1; + if (prog->aux->attach_btf_id == *bpf_modify_return_test_id) { + ret = bpf_modify_return_test(1, &b); + if (b != 2) + side_effect = 1; + } else { + ret = bpf_modify_return_test2(1, &b, 3, 4, (void *)5, 6, 7); + if (b != 2) + side_effect = 1; + } break; default: goto out; diff --git a/tools/testing/selftests/bpf/bpf_testmod/bpf_testmod.c b/tools/testing/selftests/bpf/bpf_testmod/bpf_testmod.c index aaf6ef1201c7..737ef0a5dca4 100644 --- a/tools/testing/selftests/bpf/bpf_testmod/bpf_testmod.c +++ b/tools/testing/selftests/bpf/bpf_testmod/bpf_testmod.c @@ -34,6 +34,11 @@ struct bpf_testmod_struct_arg_3 { int b[]; }; +struct bpf_testmod_struct_arg_4 { + u64 a; + int b; +}; + __diag_push(); __diag_ignore_all("-Wmissing-prototypes", "Global functions as their definitions will be in bpf_testmod.ko BTF"); @@ -191,6 +196,35 @@ noinline int bpf_testmod_fentry_test3(char a, int b, u64 c) return a + b + c; } +noinline int bpf_testmod_fentry_test7(u64 a, void *b, short c, int d, + void *e, char f, int g) +{ + return a + (long)b + c + d + (long)e + f + g; +} + +noinline int bpf_testmod_fentry_test11(u64 a, void *b, short c, int d, + void *e, char f, int g, + unsigned int h, long i, __u64 j, + unsigned long k) +{ + return a + (long)b + c + d + (long)e + f + g + h + i + j + k; +} + +noinline int bpf_testmod_fentry_test_struct1(u64 a, void *b, short c, + int d, void *e, + struct bpf_testmod_struct_arg_4 f) +{ + return a + (long)b + c + d + (long)e + f.a + f.b; +} + +noinline int bpf_testmod_fentry_test_struct2(u64 a, void *b, short c, + int d, void *e, + struct bpf_testmod_struct_arg_4 f, + int g) +{ + return a + (long)b + c + d + (long)e + f.a + f.b + g; +} + int bpf_testmod_fentry_ok; noinline ssize_t @@ -206,6 +240,7 @@ bpf_testmod_test_read(struct file *file, struct kobject *kobj, struct bpf_testmod_struct_arg_1 struct_arg1 = {10}; struct bpf_testmod_struct_arg_2 struct_arg2 = {2, 3}; struct bpf_testmod_struct_arg_3 *struct_arg3; + struct bpf_testmod_struct_arg_4 struct_arg4 = {21, 22}; int i = 1; while (bpf_testmod_return_ptr(i)) @@ -243,7 +278,15 @@ bpf_testmod_test_read(struct file *file, struct kobject *kobj, if (bpf_testmod_fentry_test1(1) != 2 || bpf_testmod_fentry_test2(2, 3) != 5 || - bpf_testmod_fentry_test3(4, 5, 6) != 15) + bpf_testmod_fentry_test3(4, 5, 6) != 15 || + bpf_testmod_fentry_test7(16, (void *)17, 18, 19, (void *)20, + 21, 22) != 133 || + bpf_testmod_fentry_test11(16, (void *)17, 18, 19, (void *)20, + 21, 22, 23, 24, 25, 26) != 231 || + bpf_testmod_fentry_test_struct1(16, (void *)17, 18, 19, + (void *)20, struct_arg4) != 133 || + bpf_testmod_fentry_test_struct2(16, (void *)17, 18, 19, + (void *)20, struct_arg4, 23) != 156) goto out; bpf_testmod_fentry_ok = 1; diff --git a/tools/testing/selftests/bpf/prog_tests/fentry_fexit.c b/tools/testing/selftests/bpf/prog_tests/fentry_fexit.c index 130f5b82d2e6..7decf57e7591 100644 --- a/tools/testing/selftests/bpf/prog_tests/fentry_fexit.c +++ b/tools/testing/selftests/bpf/prog_tests/fentry_fexit.c @@ -31,10 +31,12 @@ void test_fentry_fexit(void) ASSERT_OK(err, "ipv6 test_run"); ASSERT_OK(topts.retval, "ipv6 test retval"); + ASSERT_OK(trigger_module_test_read(1), "trigger_read"); + fentry_res = (__u64 *)fentry_skel->bss; fexit_res = (__u64 *)fexit_skel->bss; printf("%lld\n", fentry_skel->bss->test1_result); - for (i = 0; i < 8; i++) { + for (i = 0; i < 13; i++) { ASSERT_EQ(fentry_res[i], 1, "fentry result"); ASSERT_EQ(fexit_res[i], 1, "fexit result"); } diff --git a/tools/testing/selftests/bpf/prog_tests/fentry_test.c b/tools/testing/selftests/bpf/prog_tests/fentry_test.c index c0d1d61d5f66..e1c0ce40febf 100644 --- a/tools/testing/selftests/bpf/prog_tests/fentry_test.c +++ b/tools/testing/selftests/bpf/prog_tests/fentry_test.c @@ -24,6 +24,8 @@ static int fentry_test(struct fentry_test_lskel *fentry_skel) ASSERT_OK(err, "test_run"); ASSERT_EQ(topts.retval, 0, "test_run"); + ASSERT_OK(trigger_module_test_read(1), "trigger_read"); + result = (__u64 *)fentry_skel->bss; for (i = 0; i < sizeof(*fentry_skel->bss) / sizeof(__u64); i++) { if (!ASSERT_EQ(result[i], 1, "fentry_result")) diff --git a/tools/testing/selftests/bpf/prog_tests/fexit_test.c b/tools/testing/selftests/bpf/prog_tests/fexit_test.c index 101b7343036b..ea81fa913ec6 100644 --- a/tools/testing/selftests/bpf/prog_tests/fexit_test.c +++ b/tools/testing/selftests/bpf/prog_tests/fexit_test.c @@ -24,6 +24,8 @@ static int fexit_test(struct fexit_test_lskel *fexit_skel) ASSERT_OK(err, "test_run"); ASSERT_EQ(topts.retval, 0, "test_run"); + ASSERT_OK(trigger_module_test_read(1), "trigger_read"); + result = (__u64 *)fexit_skel->bss; for (i = 0; i < sizeof(*fexit_skel->bss) / sizeof(__u64); i++) { if (!ASSERT_EQ(result[i], 1, "fexit_result")) diff --git a/tools/testing/selftests/bpf/prog_tests/modify_return.c b/tools/testing/selftests/bpf/prog_tests/modify_return.c index 5d9955af6247..93febb6d81ef 100644 --- a/tools/testing/selftests/bpf/prog_tests/modify_return.c +++ b/tools/testing/selftests/bpf/prog_tests/modify_return.c @@ -11,7 +11,8 @@ #define UPPER(x) ((x) >> 16) -static void run_test(__u32 input_retval, __u16 want_side_effect, __s16 want_ret) +static void run_test(__u32 input_retval, __u16 want_side_effect, + __s16 want_ret, __s16 want_ret2) { struct modify_return *skel = NULL; int err, prog_fd; @@ -41,6 +42,19 @@ static void run_test(__u32 input_retval, __u16 want_side_effect, __s16 want_ret) ASSERT_EQ(skel->bss->fexit_result, 1, "modify_return fexit_result"); ASSERT_EQ(skel->bss->fmod_ret_result, 1, "modify_return fmod_ret_result"); + prog_fd = bpf_program__fd(skel->progs.fmod_ret_test2); + err = bpf_prog_test_run_opts(prog_fd, &topts); + ASSERT_OK(err, "test_run"); + + side_effect = UPPER(topts.retval); + ret = LOWER(topts.retval); + + ASSERT_EQ(ret, want_ret2, "test_run ret2"); + ASSERT_EQ(side_effect, want_side_effect, "modify_return side_effect2"); + ASSERT_EQ(skel->bss->fentry_result2, 1, "modify_return fentry_result2"); + ASSERT_EQ(skel->bss->fexit_result2, 1, "modify_return fexit_result2"); + ASSERT_EQ(skel->bss->fmod_ret_result2, 1, "modify_return fmod_ret_result2"); + cleanup: modify_return__destroy(skel); } @@ -50,8 +64,10 @@ void serial_test_modify_return(void) { run_test(0 /* input_retval */, 1 /* want_side_effect */, - 4 /* want_ret */); + 4 /* want_ret */, + 29 /* want_ret */); run_test(-EINVAL /* input_retval */, 0 /* want_side_effect */, + -EINVAL /* want_ret */, -EINVAL /* want_ret */); } diff --git a/tools/testing/selftests/bpf/progs/fentry_test.c b/tools/testing/selftests/bpf/progs/fentry_test.c index 52a550d281d9..3614c85c4ead 100644 --- a/tools/testing/selftests/bpf/progs/fentry_test.c +++ b/tools/testing/selftests/bpf/progs/fentry_test.c @@ -77,3 +77,55 @@ int BPF_PROG(test8, struct bpf_fentry_test_t *arg) test8_result = 1; return 0; } + +__u64 test9_result = 0; +SEC("fentry/bpf_testmod_fentry_test7") +int BPF_PROG(test9, __u64 a, void *b, short c, int d, void *e, char f, + int g) +{ + test9_result = a == 16 && b == (void *)17 && c == 18 && d == 19 && + e == (void *)20 && f == 21 && g == 22; + return 0; +} + +__u64 test10_result = 0; +SEC("fentry/bpf_testmod_fentry_test11") +int BPF_PROG(test10, __u64 a, void *b, short c, int d, void *e, char f, + int g, unsigned int h, long i, __u64 j, unsigned long k) +{ + test10_result = a == 16 && b == (void *)17 && c == 18 && d == 19 && + e == (void *)20 && f == 21 && g == 22 && h == 23 && + i == 24 && j == 25 && k == 26; + return 0; +} + +__u64 test11_result = 0; +SEC("fentry/bpf_testmod_fentry_test11") +int BPF_PROG(test11, __u64 a, __u64 b, __u64 c, __u64 d, __u64 e, __u64 f, + __u64 g, __u64 h, __u64 i, __u64 j, __u64 k) +{ + test11_result = a == 16 && b == 17 && c == 18 && d == 19 && + e == 20 && f == 21 && g == 22 && h == 23 && + i == 24 && j == 25 && k == 26; + return 0; +} + +__u64 test12_result = 0; +SEC("fentry/bpf_testmod_fentry_test_struct1") +int BPF_PROG(test12, __u64 a, void *b, short c, int d, void *e, __u64 s_a, + int s_b) +{ + test12_result = a == 16 && b == (void *)17 && c == 18 && d == 19 && + e == (void *)20 && s_a == 21 && s_b == 22; + return 0; +} + +__u64 test13_result = 0; +SEC("fentry/bpf_testmod_fentry_test_struct2") +int BPF_PROG(test13, __u64 a, void *b, short c, int d, void *e, __u64 s_a, + int s_b, int f) +{ + test13_result = a == 16 && b == (void *)17 && c == 18 && d == 19 && + e == (void *)20 && s_a == 21 && s_b == 22 && f == 23; + return 0; +} diff --git a/tools/testing/selftests/bpf/progs/fexit_test.c b/tools/testing/selftests/bpf/progs/fexit_test.c index 8f1ccb7302e1..f6c50e4663db 100644 --- a/tools/testing/selftests/bpf/progs/fexit_test.c +++ b/tools/testing/selftests/bpf/progs/fexit_test.c @@ -78,3 +78,57 @@ int BPF_PROG(test8, struct bpf_fentry_test_t *arg) test8_result = 1; return 0; } + +__u64 test9_result = 0; +SEC("fexit/bpf_testmod_fentry_test7") +int BPF_PROG(test9, __u64 a, void *b, short c, int d, void *e, char f, + int g, int ret) +{ + test9_result = a == 16 && b == (void *)17 && c == 18 && d == 19 && + e == (void *)20 && f == 21 && g == 22 && ret == 133; + return 0; +} + +__u64 test10_result = 0; +SEC("fexit/bpf_testmod_fentry_test11") +int BPF_PROG(test10, __u64 a, void *b, short c, int d, void *e, char f, + int g, unsigned int h, long i, __u64 j, unsigned long k, + int ret) +{ + test10_result = a == 16 && b == (void *)17 && c == 18 && d == 19 && + e == (void *)20 && f == 21 && g == 22 && h == 23 && + i == 24 && j == 25 && k == 26 && ret == 231; + return 0; +} + +__u64 test11_result = 0; +SEC("fexit/bpf_testmod_fentry_test11") +int BPF_PROG(test11, __u64 a, __u64 b, __u64 c, __u64 d, __u64 e, __u64 f, + __u64 g, __u64 h, __u64 i, __u64 j, __u64 k, __u64 ret) +{ + test11_result = a == 16 && b == 17 && c == 18 && d == 19 && + e == 20 && f == 21 && g == 22 && h == 23 && + i == 24 && j == 25 && k == 26 && ret == 231; + return 0; +} + +__u64 test12_result = 0; +SEC("fexit/bpf_testmod_fentry_test_struct1") +int BPF_PROG(test12, __u64 a, void *b, short c, int d, void *e, __u64 s_a, + int s_b, __u64 ret) +{ + test12_result = a == 16 && b == (void *)17 && c == 18 && d == 19 && + e == (void *)20 && s_a == 21 && s_b == 22 && ret == 133; + return 0; +} + +__u64 test13_result = 0; +SEC("fexit/bpf_testmod_fentry_test_struct2") +int BPF_PROG(test13, __u64 a, void *b, short c, int d, void *e, __u64 s_a, + int s_b, int f, __u64 ret) +{ + test13_result = a == 16 && b == (void *)17 && c == 18 && d == 19 && + e == (void *)20 && s_a == 21 && s_b == 22 && f == 23 && + ret == 156; + return 0; +} diff --git a/tools/testing/selftests/bpf/progs/modify_return.c b/tools/testing/selftests/bpf/progs/modify_return.c index 8b7466a15c6b..3376d4849f58 100644 --- a/tools/testing/selftests/bpf/progs/modify_return.c +++ b/tools/testing/selftests/bpf/progs/modify_return.c @@ -47,3 +47,43 @@ int BPF_PROG(fexit_test, int a, __u64 b, int ret) return 0; } + +static int sequence2; + +__u64 fentry_result2 = 0; +SEC("fentry/bpf_modify_return_test2") +int BPF_PROG(fentry_test2, int a, int *b, short c, int d, void *e, char f, + int g) +{ + sequence2++; + fentry_result2 = (sequence2 == 1); + return 0; +} + +__u64 fmod_ret_result2 = 0; +SEC("fmod_ret/bpf_modify_return_test2") +int BPF_PROG(fmod_ret_test2, int a, int *b, short c, int d, void *e, char f, + int g, int ret) +{ + sequence2++; + /* This is the first fmod_ret program, the ret passed should be 0 */ + fmod_ret_result2 = (sequence2 == 2 && ret == 0); + return input_retval; +} + +__u64 fexit_result2 = 0; +SEC("fexit/bpf_modify_return_test2") +int BPF_PROG(fexit_test2, int a, int *b, short c, int d, void *e, char f, + int g, int ret) +{ + sequence2++; + /* If the input_reval is non-zero a successful modification should have + * occurred. + */ + if (input_retval) + fexit_result2 = (sequence2 == 3 && ret == input_retval); + else + fexit_result2 = (sequence2 == 3 && ret == 29); + + return 0; +}