From patchwork Fri Jun 9 09:56:51 2023 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Menglong Dong X-Patchwork-Id: 13273608 X-Patchwork-Delegate: bpf@iogearbox.net Received: from lindbergh.monkeyblade.net (lindbergh.monkeyblade.net [23.128.96.19]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (No client certificate requested) by smtp.subspace.kernel.org (Postfix) with ESMTPS id 5497313AE9 for ; Fri, 9 Jun 2023 09:57:24 +0000 (UTC) Received: from mail-pf1-x443.google.com (mail-pf1-x443.google.com [IPv6:2607:f8b0:4864:20::443]) by lindbergh.monkeyblade.net (Postfix) with ESMTPS id ECEE17AB5; Fri, 9 Jun 2023 02:57:06 -0700 (PDT) Received: by mail-pf1-x443.google.com with SMTP id d2e1a72fcca58-653bed78635so1172530b3a.0; Fri, 09 Jun 2023 02:57:06 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=20221208; t=1686304626; x=1688896626; h=content-transfer-encoding:mime-version:references:in-reply-to :message-id:date:subject:cc:to:from:from:to:cc:subject:date :message-id:reply-to; bh=Crp08FOLgFvZnpAjUWNQfC7YZ3uDv9qVkOafGIbXOmc=; b=Smb8m1uIPGvu+9/rI1RV7KsRjkbz3obLfvopVGmch4HO34+2L5cVmeqUvwT14H9XJ0 ppMoPMoGIgUMmErqqQVf/X4hV3gXp7w5ZazODxV3j/71+LI1JhIRM6VQtqDNAB9HQocr mK6woCwCJqUykOdrHqVgTg0X31rCzfZjQ0IOxOWarwNpReZZarn7sG3E3yTe+TntYSLu K2Lkk1QisQh9bTbma2hc4QzLi5Wf9+AgOTtiuml1GXpzsrmpsimuF00QSs45YNtJRpWp SvSsXFxVAIyiCyDEzZK5ZGyL4YZPuXyBBDhE+8zti3grnsImeIhyt8wKJf+1mj+c5SEk yibQ== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20221208; t=1686304626; x=1688896626; h=content-transfer-encoding:mime-version:references:in-reply-to :message-id:date:subject:cc:to:from:x-gm-message-state:from:to:cc :subject:date:message-id:reply-to; bh=Crp08FOLgFvZnpAjUWNQfC7YZ3uDv9qVkOafGIbXOmc=; b=I75IBPRVJrxP9c2RK4V68VQwj9Tvm1iLSUWww9AllrvGIvykR2tMe+tUSYnJKNBIna OpDhpyzpnsiaCw87L8sMiU7p1vn8pLL9B2NvQeb15hmJfTQr7gZibDYUEXj4H4tUQ0p6 eyiczDODkpNJJCMTS6zkX465i7Yw2Fv8TDMxMLv8f91/0c7BbWUOV4Ndke7j1xLBENPg MpxtBGARAgXwF37yrnG+3A7lw7rf2vlQx5iFVlMOM7cNXPGl/iBfRAToNZz6RwlmfBb7 gmqkR119ByiEsypqPaX+m1mVd6X1EEZt2gH5AvDaSrgNLbUYVOOr+nVWnuDOZhtpN3MN r87w== X-Gm-Message-State: AC+VfDwYvBzVEU+pG8kSV6N+fo4sl2ZbegPZh51iimIWvSJxh/xZSRra Mzcw1KyEaorE8yMOLiKJeao= X-Google-Smtp-Source: ACHHUZ5klU42zeWXEXFhJ/lUf4IP7IYmKkj0UbDbcezZANyrqtfrx9FeHnNSut6VACtE9beXzChGBA== X-Received: by 2002:a05:6a21:99a0:b0:117:4d57:d3ad with SMTP id ve32-20020a056a2199a000b001174d57d3admr5350154pzb.3.1686304626199; Fri, 09 Jun 2023 02:57:06 -0700 (PDT) Received: from localhost.localdomain ([43.132.98.114]) by smtp.gmail.com with ESMTPSA id 26-20020a63175a000000b0053f3797fc4asm2603369pgx.0.2023.06.09.02.57.02 (version=TLS1_3 cipher=TLS_AES_256_GCM_SHA384 bits=256/256); Fri, 09 Jun 2023 02:57:05 -0700 (PDT) From: menglong8.dong@gmail.com X-Google-Original-From: imagedong@tencent.com To: andrii.nakryiko@gmail.com, alan.maguire@oracle.com Cc: ast@kernel.org, daniel@iogearbox.net, andrii@kernel.org, martin.lau@linux.dev, song@kernel.org, yhs@fb.com, john.fastabend@gmail.com, kpsingh@kernel.org, sdf@google.com, haoluo@google.com, jolsa@kernel.org, bpf@vger.kernel.org, linux-kernel@vger.kernel.org, Menglong Dong Subject: [PATCH bpf-next v4 1/3] bpf, x86: allow function arguments up to 12 for TRACING Date: Fri, 9 Jun 2023 17:56:51 +0800 Message-Id: <20230609095653.1406173-2-imagedong@tencent.com> X-Mailer: git-send-email 2.40.1 In-Reply-To: <20230609095653.1406173-1-imagedong@tencent.com> References: <20230609095653.1406173-1-imagedong@tencent.com> Precedence: bulk X-Mailing-List: bpf@vger.kernel.org List-Id: List-Subscribe: List-Unsubscribe: MIME-Version: 1.0 X-Spam-Status: No, score=-2.1 required=5.0 tests=BAYES_00,DKIM_SIGNED, DKIM_VALID,DKIM_VALID_AU,DKIM_VALID_EF,FREEMAIL_FROM, RCVD_IN_DNSWL_NONE,SPF_HELO_NONE,SPF_PASS,T_SCC_BODY_TEXT_LINE autolearn=ham autolearn_force=no version=3.4.6 X-Spam-Checker-Version: SpamAssassin 3.4.6 (2021-04-09) on lindbergh.monkeyblade.net X-Patchwork-Delegate: bpf@iogearbox.net From: Menglong Dong For now, the BPF program of type BPF_PROG_TYPE_TRACING can only be used on the kernel functions whose arguments count less than 6. This is not friendly at all, as too many functions have arguments count more than 6. According to the current kernel version, below is a statistics of the function arguments count: argument count | function count 7 | 704 8 | 270 9 | 84 10 | 47 11 | 47 12 | 27 13 | 22 14 | 5 15 | 0 16 | 1 Therefore, let's enhance it by increasing the function arguments count allowed in arch_prepare_bpf_trampoline(), for now, only x86_64. For the case that we don't need to call origin function, which means without BPF_TRAMP_F_CALL_ORIG, we need only copy the function arguments that stored in the frame of the caller to current frame. The arguments of arg6-argN are stored in "$rbp + 0x18", we need copy them to "$rbp - regs_off + (6 * 8)". For the case with BPF_TRAMP_F_CALL_ORIG, we need prepare the arguments in stack before call origin function, which means we need alloc extra "8 * (arg_count - 6)" memory in the top of the stack. Note, there should not be any data be pushed to the stack before call the origin function. Then, we have to store rbx with 'mov' instead of 'push'. We use EMIT3_off32() or EMIT4() for "lea" and "sub". The range of the imm in "lea" and "sub" is [-128, 127] if EMIT4() is used. Therefore, we use EMIT3_off32() instead if the imm out of the range. It works well for the FENTRY/FEXIT/MODIFY_RETURN, I'm not sure if there are other complicated cases. Signed-off-by: Menglong Dong --- v4: - make the stack 16-byte aligned if passing args on-stack is needed - add the function arguments statistics to the commit log v3: - use EMIT3_off32() for "lea" and "sub" only on necessary - make 12 as the maximum arguments count v2: - instead EMIT4 with EMIT3_off32 for "lea" to prevent overflow - make MAX_BPF_FUNC_ARGS as the maximum argument count --- arch/x86/net/bpf_jit_comp.c | 125 ++++++++++++++++++++++++++++++++---- 1 file changed, 111 insertions(+), 14 deletions(-) diff --git a/arch/x86/net/bpf_jit_comp.c b/arch/x86/net/bpf_jit_comp.c index 1056bbf55b17..a767e13c8c85 100644 --- a/arch/x86/net/bpf_jit_comp.c +++ b/arch/x86/net/bpf_jit_comp.c @@ -1868,7 +1868,7 @@ static void save_regs(const struct btf_func_model *m, u8 **prog, int nr_regs, * mov QWORD PTR [rbp-0x10],rdi * mov QWORD PTR [rbp-0x8],rsi */ - for (i = 0, j = 0; i < min(nr_regs, 6); i++) { + for (i = 0, j = 0; i < min(nr_regs, MAX_BPF_FUNC_ARGS); i++) { /* The arg_size is at most 16 bytes, enforced by the verifier. */ arg_size = m->arg_size[j]; if (arg_size > 8) { @@ -1876,10 +1876,31 @@ static void save_regs(const struct btf_func_model *m, u8 **prog, int nr_regs, next_same_struct = !next_same_struct; } - emit_stx(prog, bytes_to_bpf_size(arg_size), - BPF_REG_FP, - i == 5 ? X86_REG_R9 : BPF_REG_1 + i, - -(stack_size - i * 8)); + if (i <= 5) { + /* copy function arguments from regs into stack */ + emit_stx(prog, bytes_to_bpf_size(arg_size), + BPF_REG_FP, + i == 5 ? X86_REG_R9 : BPF_REG_1 + i, + -(stack_size - i * 8)); + } else { + /* copy function arguments from origin stack frame + * into current stack frame. + * + * The starting address of the arguments on-stack + * is: + * rbp + 8(push rbp) + + * 8(return addr of origin call) + + * 8(return addr of the caller) + * which means: rbp + 24 + */ + emit_ldx(prog, bytes_to_bpf_size(arg_size), + BPF_REG_0, BPF_REG_FP, + (i - 6) * 8 + 0x18); + emit_stx(prog, bytes_to_bpf_size(arg_size), + BPF_REG_FP, + BPF_REG_0, + -(stack_size - i * 8)); + } j = next_same_struct ? j : j + 1; } @@ -1913,6 +1934,41 @@ static void restore_regs(const struct btf_func_model *m, u8 **prog, int nr_regs, } } +static void prepare_origin_stack(const struct btf_func_model *m, u8 **prog, + int nr_regs, int stack_size) +{ + int i, j, arg_size; + bool next_same_struct = false; + + if (nr_regs <= 6) + return; + + /* Prepare the function arguments in stack before call origin + * function. These arguments must be stored in the top of the + * stack. + */ + for (i = 0, j = 0; i < min(nr_regs, MAX_BPF_FUNC_ARGS); i++) { + /* The arg_size is at most 16 bytes, enforced by the verifier. */ + arg_size = m->arg_size[j]; + if (arg_size > 8) { + arg_size = 8; + next_same_struct = !next_same_struct; + } + + if (i > 5) { + emit_ldx(prog, bytes_to_bpf_size(arg_size), + BPF_REG_0, BPF_REG_FP, + (i - 6) * 8 + 0x18); + emit_stx(prog, bytes_to_bpf_size(arg_size), + BPF_REG_FP, + BPF_REG_0, + -(stack_size - (i - 6) * 8)); + } + + j = next_same_struct ? j : j + 1; + } +} + static int invoke_bpf_prog(const struct btf_func_model *m, u8 **pprog, struct bpf_tramp_link *l, int stack_size, int run_ctx_off, bool save_ret) @@ -1938,7 +1994,10 @@ static int invoke_bpf_prog(const struct btf_func_model *m, u8 **pprog, /* arg1: mov rdi, progs[i] */ emit_mov_imm64(&prog, BPF_REG_1, (long) p >> 32, (u32) (long) p); /* arg2: lea rsi, [rbp - ctx_cookie_off] */ - EMIT4(0x48, 0x8D, 0x75, -run_ctx_off); + if (!is_imm8(-run_ctx_off)) + EMIT3_off32(0x48, 0x8D, 0xB5, -run_ctx_off); + else + EMIT4(0x48, 0x8D, 0x75, -run_ctx_off); if (emit_rsb_call(&prog, bpf_trampoline_enter(p), prog)) return -EINVAL; @@ -1954,7 +2013,10 @@ static int invoke_bpf_prog(const struct btf_func_model *m, u8 **pprog, emit_nops(&prog, 2); /* arg1: lea rdi, [rbp - stack_size] */ - EMIT4(0x48, 0x8D, 0x7D, -stack_size); + if (!is_imm8(-stack_size)) + EMIT3_off32(0x48, 0x8D, 0xBD, -stack_size); + else + EMIT4(0x48, 0x8D, 0x7D, -stack_size); /* arg2: progs[i]->insnsi for interpreter */ if (!p->jited) emit_mov_imm64(&prog, BPF_REG_2, @@ -1984,7 +2046,10 @@ static int invoke_bpf_prog(const struct btf_func_model *m, u8 **pprog, /* arg2: mov rsi, rbx <- start time in nsec */ emit_mov_reg(&prog, true, BPF_REG_2, BPF_REG_6); /* arg3: lea rdx, [rbp - run_ctx_off] */ - EMIT4(0x48, 0x8D, 0x55, -run_ctx_off); + if (!is_imm8(-run_ctx_off)) + EMIT3_off32(0x48, 0x8D, 0x95, -run_ctx_off); + else + EMIT4(0x48, 0x8D, 0x55, -run_ctx_off); if (emit_rsb_call(&prog, bpf_trampoline_exit(p), prog)) return -EINVAL; @@ -2136,7 +2201,7 @@ int arch_prepare_bpf_trampoline(struct bpf_tramp_image *im, void *image, void *i void *func_addr) { int i, ret, nr_regs = m->nr_args, stack_size = 0; - int regs_off, nregs_off, ip_off, run_ctx_off; + int regs_off, nregs_off, ip_off, run_ctx_off, arg_stack_off, rbx_off; struct bpf_tramp_links *fentry = &tlinks[BPF_TRAMP_FENTRY]; struct bpf_tramp_links *fexit = &tlinks[BPF_TRAMP_FEXIT]; struct bpf_tramp_links *fmod_ret = &tlinks[BPF_TRAMP_MODIFY_RETURN]; @@ -2150,8 +2215,10 @@ int arch_prepare_bpf_trampoline(struct bpf_tramp_image *im, void *image, void *i if (m->arg_flags[i] & BTF_FMODEL_STRUCT_ARG) nr_regs += (m->arg_size[i] + 7) / 8 - 1; - /* x86-64 supports up to 6 arguments. 7+ can be added in the future */ - if (nr_regs > 6) + /* x86-64 supports up to MAX_BPF_FUNC_ARGS arguments. 1-6 + * are passed through regs, the remains are through stack. + */ + if (nr_regs > MAX_BPF_FUNC_ARGS) return -ENOTSUPP; /* Generated trampoline stack layout: @@ -2170,7 +2237,14 @@ int arch_prepare_bpf_trampoline(struct bpf_tramp_image *im, void *image, void *i * * RBP - ip_off [ traced function ] BPF_TRAMP_F_IP_ARG flag * + * RBP - rbx_off [ rbx value ] always + * * RBP - run_ctx_off [ bpf_tramp_run_ctx ] + * + * [ stack_argN ] BPF_TRAMP_F_CALL_ORIG + * [ ... ] + * [ stack_arg2 ] + * RBP - arg_stack_off [ stack_arg1 ] */ /* room for return value of orig_call or fentry prog */ @@ -2190,9 +2264,25 @@ int arch_prepare_bpf_trampoline(struct bpf_tramp_image *im, void *image, void *i ip_off = stack_size; + stack_size += 8; + rbx_off = stack_size; + stack_size += (sizeof(struct bpf_tramp_run_ctx) + 7) & ~0x7; run_ctx_off = stack_size; + if (nr_regs > 6 && (flags & BPF_TRAMP_F_CALL_ORIG)) { + stack_size += (nr_regs - 6) * 8; + /* make sure the stack pointer is 16-byte aligned if we + * need pass arguments on stack, which means + * [stack_size + 8(rbp) + 8(rip) + 8(origin rip)] + * should be 16-byte aligned. Following code depend on + * that stack_size is already 8-byte aligned. + */ + stack_size += (stack_size % 16) ? 0 : 8; + } + + arg_stack_off = stack_size; + if (flags & BPF_TRAMP_F_SKIP_FRAME) { /* skip patched call instruction and point orig_call to actual * body of the kernel function. @@ -2212,8 +2302,14 @@ int arch_prepare_bpf_trampoline(struct bpf_tramp_image *im, void *image, void *i x86_call_depth_emit_accounting(&prog, NULL); EMIT1(0x55); /* push rbp */ EMIT3(0x48, 0x89, 0xE5); /* mov rbp, rsp */ - EMIT4(0x48, 0x83, 0xEC, stack_size); /* sub rsp, stack_size */ - EMIT1(0x53); /* push rbx */ + if (!is_imm8(stack_size)) + /* sub rsp, stack_size */ + EMIT3_off32(0x48, 0x81, 0xEC, stack_size); + else + /* sub rsp, stack_size */ + EMIT4(0x48, 0x83, 0xEC, stack_size); + /* mov QWORD PTR [rbp - rbx_off], rbx */ + emit_stx(&prog, BPF_DW, BPF_REG_FP, BPF_REG_6, -rbx_off); /* Store number of argument registers of the traced function: * mov rax, nr_regs @@ -2262,6 +2358,7 @@ int arch_prepare_bpf_trampoline(struct bpf_tramp_image *im, void *image, void *i if (flags & BPF_TRAMP_F_CALL_ORIG) { restore_regs(m, &prog, nr_regs, regs_off); + prepare_origin_stack(m, &prog, nr_regs, arg_stack_off); if (flags & BPF_TRAMP_F_ORIG_STACK) { emit_ldx(&prog, BPF_DW, BPF_REG_0, BPF_REG_FP, 8); @@ -2321,7 +2418,7 @@ int arch_prepare_bpf_trampoline(struct bpf_tramp_image *im, void *image, void *i if (save_ret) emit_ldx(&prog, BPF_DW, BPF_REG_0, BPF_REG_FP, -8); - EMIT1(0x5B); /* pop rbx */ + emit_ldx(&prog, BPF_DW, BPF_REG_6, BPF_REG_FP, -rbx_off); EMIT1(0xC9); /* leave */ if (flags & BPF_TRAMP_F_SKIP_FRAME) /* skip our return address and return to parent */ From patchwork Fri Jun 9 09:56:52 2023 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Menglong Dong X-Patchwork-Id: 13273609 X-Patchwork-Delegate: bpf@iogearbox.net Received: from lindbergh.monkeyblade.net (lindbergh.monkeyblade.net [23.128.96.19]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (No client certificate requested) by smtp.subspace.kernel.org (Postfix) with ESMTPS id 6BE4C13AE9 for ; Fri, 9 Jun 2023 09:57:25 +0000 (UTC) Received: from mail-oo1-xc44.google.com (mail-oo1-xc44.google.com [IPv6:2607:f8b0:4864:20::c44]) by lindbergh.monkeyblade.net (Postfix) with ESMTPS id 1D2CB7AB6; Fri, 9 Jun 2023 02:57:11 -0700 (PDT) Received: by mail-oo1-xc44.google.com with SMTP id 006d021491bc7-559409cc490so1019905eaf.3; Fri, 09 Jun 2023 02:57:11 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=20221208; t=1686304630; x=1688896630; h=content-transfer-encoding:mime-version:references:in-reply-to :message-id:date:subject:cc:to:from:from:to:cc:subject:date :message-id:reply-to; bh=odT6+rgQqp77JJ8LV+JNNotDPvzRm1aU2ce2tTNslKg=; b=nGtuEwHEkJJ/K9Cxr9t+VLarD+KhlwiVqpwzlWVzTAtiotn2+8vMZSaYKUGQ3Bwyx7 Y9HXa/bGqMSFbWAznCM18RPQNpuYASmEx+jUjWwQOLRDY2SiUMQnWMhTHSPghgjzO2s/ ftrAIH2TEv84ZfMI+/6QdFdbyT4X5USxzOtilJnJxQ3AOVGLjqLSddP4Gf2PY71jR6Br FDc8jCjbW97PHyUWGX2WO4TDf8H9QK0yiT5a8sshAjMn7X6rn8ek0wH/huLwDCfGi7Aj SSk8iFI/YMrWibMVjD/0ke9RRhkt/V9oVk1PwOtARSNR85znARMLxESZtYJCXt/JUEuX slRw== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20221208; t=1686304630; x=1688896630; h=content-transfer-encoding:mime-version:references:in-reply-to :message-id:date:subject:cc:to:from:x-gm-message-state:from:to:cc :subject:date:message-id:reply-to; bh=odT6+rgQqp77JJ8LV+JNNotDPvzRm1aU2ce2tTNslKg=; b=PdLK/2asvBp30kivTrnNsr7qJa84uL9mUEg9W5yV6kIGObZAVC88eHOFXTRJy/bNTY QyI/WYPra+56NIcy6aG1MRpj0aUxZwLet5ig9VgdaJ7kO2yv8vxFgyS4PD/jMBK/xfSN nv2bYnWtV/9C1ch4zJVqp9TylS4N3DIlS9gZTfXM7oBhCN4MDLZs54WFdMMXgdAxi7bg ua8x3PgFHWSMBxszLGULysjneOWF/xAyT5pbJNnDEpbYv12l3PyydfJGgZBUvv99YAto bNfh0UOG6zLZG8Pk9j1w2IEVhz4LA8QNOrbljmpl81xd1ScZDr4VBvheKIWJ0tsf78od pFNg== X-Gm-Message-State: AC+VfDyV+E6KqrCfiR3lWbJVe0WXzMJ1HDpIieOWNcx1W0B1oHYg90hz CGoA3XVkNGhGg8/azO7wf/k= X-Google-Smtp-Source: ACHHUZ7L7Afe4yMt/dqD3g6NNiT/1f3jVS/4vSDqVpfulu8Hq0p1NopPUsk7YHP0OtYHPqmsuU1edw== X-Received: by 2002:a05:6358:4f82:b0:129:f7ad:20f1 with SMTP id cg2-20020a0563584f8200b00129f7ad20f1mr794812rwb.23.1686304630163; Fri, 09 Jun 2023 02:57:10 -0700 (PDT) Received: from localhost.localdomain ([43.132.98.114]) by smtp.gmail.com with ESMTPSA id 26-20020a63175a000000b0053f3797fc4asm2603369pgx.0.2023.06.09.02.57.06 (version=TLS1_3 cipher=TLS_AES_256_GCM_SHA384 bits=256/256); Fri, 09 Jun 2023 02:57:09 -0700 (PDT) From: menglong8.dong@gmail.com X-Google-Original-From: imagedong@tencent.com To: andrii.nakryiko@gmail.com, alan.maguire@oracle.com Cc: ast@kernel.org, daniel@iogearbox.net, andrii@kernel.org, martin.lau@linux.dev, song@kernel.org, yhs@fb.com, john.fastabend@gmail.com, kpsingh@kernel.org, sdf@google.com, haoluo@google.com, jolsa@kernel.org, bpf@vger.kernel.org, linux-kernel@vger.kernel.org, Menglong Dong Subject: [PATCH bpf-next v4 2/3] bpf, x86: clean garbage value in the stack of trampoline Date: Fri, 9 Jun 2023 17:56:52 +0800 Message-Id: <20230609095653.1406173-3-imagedong@tencent.com> X-Mailer: git-send-email 2.40.1 In-Reply-To: <20230609095653.1406173-1-imagedong@tencent.com> References: <20230609095653.1406173-1-imagedong@tencent.com> Precedence: bulk X-Mailing-List: bpf@vger.kernel.org List-Id: List-Subscribe: List-Unsubscribe: MIME-Version: 1.0 X-Spam-Status: No, score=-2.1 required=5.0 tests=BAYES_00,DKIM_SIGNED, DKIM_VALID,DKIM_VALID_AU,DKIM_VALID_EF,FREEMAIL_FROM, RCVD_IN_DNSWL_NONE,SPF_HELO_NONE,SPF_PASS,T_SCC_BODY_TEXT_LINE autolearn=ham autolearn_force=no version=3.4.6 X-Spam-Checker-Version: SpamAssassin 3.4.6 (2021-04-09) on lindbergh.monkeyblade.net X-Patchwork-Delegate: bpf@iogearbox.net From: Menglong Dong There are garbage values in upper bytes when we store the arguments into stack in save_regs() if the size of the argument less then 8. As we already reserve 8 byte for the arguments in regs and stack, it is ok to store/restore the regs in BPF_DW size. Then, the garbage values in upper bytes will be cleaned. Signed-off-by: Menglong Dong --- v4: - clean grabage value when argument count is 7 --- arch/x86/net/bpf_jit_comp.c | 45 ++++++++++++++++++++++++++----------- 1 file changed, 32 insertions(+), 13 deletions(-) diff --git a/arch/x86/net/bpf_jit_comp.c b/arch/x86/net/bpf_jit_comp.c index a767e13c8c85..f6f51a5d14db 100644 --- a/arch/x86/net/bpf_jit_comp.c +++ b/arch/x86/net/bpf_jit_comp.c @@ -1857,6 +1857,28 @@ st: if (is_imm8(insn->off)) return proglen; } +static inline void clean_garbage(u8 **pprog, int nr_regs, int stack_size, + int arg_size) +{ + u8 *prog; + + /* clean potential garbage values in upper 32-bit. 'stack_size' + * here is the offset of the 7th argument on-stack. + */ + if (nr_regs == 7 && arg_size <= 4) { + int off = -(stack_size - 4); + + prog = *pprog; + /* mov DWORD PTR [rbp + off], 0 */ + if (!is_imm8(off)) + EMIT2_off32(0xC7, 0x85, off); + else + EMIT3(0xC7, 0x45, off); + EMIT(0, 4); + *pprog = prog; + } +} + static void save_regs(const struct btf_func_model *m, u8 **prog, int nr_regs, int stack_size) { @@ -1878,8 +1900,7 @@ static void save_regs(const struct btf_func_model *m, u8 **prog, int nr_regs, if (i <= 5) { /* copy function arguments from regs into stack */ - emit_stx(prog, bytes_to_bpf_size(arg_size), - BPF_REG_FP, + emit_stx(prog, BPF_DW, BPF_REG_FP, i == 5 ? X86_REG_R9 : BPF_REG_1 + i, -(stack_size - i * 8)); } else { @@ -1893,17 +1914,16 @@ static void save_regs(const struct btf_func_model *m, u8 **prog, int nr_regs, * 8(return addr of the caller) * which means: rbp + 24 */ - emit_ldx(prog, bytes_to_bpf_size(arg_size), - BPF_REG_0, BPF_REG_FP, + emit_ldx(prog, BPF_DW, BPF_REG_0, BPF_REG_FP, (i - 6) * 8 + 0x18); - emit_stx(prog, bytes_to_bpf_size(arg_size), - BPF_REG_FP, - BPF_REG_0, + emit_stx(prog, BPF_DW, BPF_REG_FP, BPF_REG_0, -(stack_size - i * 8)); } j = next_same_struct ? j : j + 1; } + + clean_garbage(prog, nr_regs, stack_size - 6 * 8, arg_size); } static void restore_regs(const struct btf_func_model *m, u8 **prog, int nr_regs, @@ -1925,7 +1945,7 @@ static void restore_regs(const struct btf_func_model *m, u8 **prog, int nr_regs, next_same_struct = !next_same_struct; } - emit_ldx(prog, bytes_to_bpf_size(arg_size), + emit_ldx(prog, BPF_DW, i == 5 ? X86_REG_R9 : BPF_REG_1 + i, BPF_REG_FP, -(stack_size - i * 8)); @@ -1956,17 +1976,16 @@ static void prepare_origin_stack(const struct btf_func_model *m, u8 **prog, } if (i > 5) { - emit_ldx(prog, bytes_to_bpf_size(arg_size), - BPF_REG_0, BPF_REG_FP, + emit_ldx(prog, BPF_DW, BPF_REG_0, BPF_REG_FP, (i - 6) * 8 + 0x18); - emit_stx(prog, bytes_to_bpf_size(arg_size), - BPF_REG_FP, - BPF_REG_0, + emit_stx(prog, BPF_DW, BPF_REG_FP, BPF_REG_0, -(stack_size - (i - 6) * 8)); } j = next_same_struct ? j : j + 1; } + + clean_garbage(prog, nr_regs, stack_size, arg_size); } static int invoke_bpf_prog(const struct btf_func_model *m, u8 **pprog, From patchwork Fri Jun 9 09:56:53 2023 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Menglong Dong X-Patchwork-Id: 13273610 X-Patchwork-Delegate: bpf@iogearbox.net Received: from lindbergh.monkeyblade.net (lindbergh.monkeyblade.net [23.128.96.19]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (No client certificate requested) by smtp.subspace.kernel.org (Postfix) with ESMTPS id 004DB13AE9 for ; Fri, 9 Jun 2023 09:57:26 +0000 (UTC) Received: from mail-ot1-x344.google.com (mail-ot1-x344.google.com [IPv6:2607:f8b0:4864:20::344]) by lindbergh.monkeyblade.net (Postfix) with ESMTPS id 35A046A41; Fri, 9 Jun 2023 02:57:15 -0700 (PDT) Received: by mail-ot1-x344.google.com with SMTP id 46e09a7af769-6af74ca9f4aso512144a34.3; Fri, 09 Jun 2023 02:57:15 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=20221208; t=1686304634; x=1688896634; h=content-transfer-encoding:mime-version:references:in-reply-to :message-id:date:subject:cc:to:from:from:to:cc:subject:date :message-id:reply-to; bh=JxLT1h6F8xwdKl2Puy3yKVy/htvueJ5tWaXe00nH7Yg=; b=sp36f1+U7UIE/GzY+Yor6cPco4VrwtLSsueBy42O7/RbqdM7sh1E7DAJoVGRB4D08C AVRyAps37X77X+p7geVsUgTcJPOLERyGPG0wfVRqDCKbOQgaE+O4sX3bu715jsMnlfhJ Hms9ZrQfwjIrV4eTIPIHzePPlVGLCb3yure6JJUIger0zzxnrOPh3fwJAzMmXIBGZ1Im yLwSWFK2oDysLS/u4I+YEviZELWHuYYiF2j+4MmpD5ry9+5LTr9bH9Ho31qkciaUFuRr fa8/fl3uRk+r57KbVT+0Y/2vt5kqncvRj9S+TPpKXYZAADCsVsroW3diglG7UHMYr/lH yNRg== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20221208; t=1686304634; x=1688896634; h=content-transfer-encoding:mime-version:references:in-reply-to :message-id:date:subject:cc:to:from:x-gm-message-state:from:to:cc :subject:date:message-id:reply-to; bh=JxLT1h6F8xwdKl2Puy3yKVy/htvueJ5tWaXe00nH7Yg=; b=a05Inib7kikKzMjIfu/LTjuiTBW3L1Lt5PRQ6FpBYc9TlDmauOalAR23fYSPfEE0od eiqMSqzCFRxe6sHnMMKnVFzL9bkAjx1nnuPwfsIJ0jGJw58rjSsDMuT5adUBZG6Cz6ZS xONLWoCBHtjn2GlR4JZ3qDsXhvTADAbpJ5WLafGRMcB0l9l8LWHfFP2Jaj07qbSxgQn9 A1X0uq6d/lYa1nbE6nngVnIU5ofiYaRYmR90UkOr4WTGBbvxhwAKRPFMJUNEGaccljtb jxpELTKm5cuAe6TItybygiOS1LyC731moqKk3+tkhyIy096a+FXcZqd2rPK0ksxmgA2d D5Aw== X-Gm-Message-State: AC+VfDwHYQJg6BEMg58CLGjdpgvkoHnNeiKbE0vvOkWRL6DDXeLip2ZB zJJYE95XCxU7/RzEtqH1DmM= X-Google-Smtp-Source: ACHHUZ563rm/yEIvDaWJviav8CScXDT+LaSwJKY6T56YgIF+XN8X9xf8uXDh+JUNuIPpHR+FYs2R1A== X-Received: by 2002:a05:6359:3a8:b0:127:c478:641c with SMTP id eg40-20020a05635903a800b00127c478641cmr783805rwb.28.1686304634260; Fri, 09 Jun 2023 02:57:14 -0700 (PDT) Received: from localhost.localdomain ([43.132.98.114]) by smtp.gmail.com with ESMTPSA id 26-20020a63175a000000b0053f3797fc4asm2603369pgx.0.2023.06.09.02.57.10 (version=TLS1_3 cipher=TLS_AES_256_GCM_SHA384 bits=256/256); Fri, 09 Jun 2023 02:57:13 -0700 (PDT) From: menglong8.dong@gmail.com X-Google-Original-From: imagedong@tencent.com To: andrii.nakryiko@gmail.com, alan.maguire@oracle.com Cc: ast@kernel.org, daniel@iogearbox.net, andrii@kernel.org, martin.lau@linux.dev, song@kernel.org, yhs@fb.com, john.fastabend@gmail.com, kpsingh@kernel.org, sdf@google.com, haoluo@google.com, jolsa@kernel.org, bpf@vger.kernel.org, linux-kernel@vger.kernel.org, Menglong Dong Subject: [PATCH bpf-next v4 3/3] selftests/bpf: add testcase for FENTRY/FEXIT with 6+ arguments Date: Fri, 9 Jun 2023 17:56:53 +0800 Message-Id: <20230609095653.1406173-4-imagedong@tencent.com> X-Mailer: git-send-email 2.40.1 In-Reply-To: <20230609095653.1406173-1-imagedong@tencent.com> References: <20230609095653.1406173-1-imagedong@tencent.com> Precedence: bulk X-Mailing-List: bpf@vger.kernel.org List-Id: List-Subscribe: List-Unsubscribe: MIME-Version: 1.0 X-Spam-Status: No, score=-2.1 required=5.0 tests=BAYES_00,DKIM_SIGNED, DKIM_VALID,DKIM_VALID_AU,DKIM_VALID_EF,FREEMAIL_FROM, RCVD_IN_DNSWL_NONE,SPF_HELO_NONE,SPF_PASS,T_SCC_BODY_TEXT_LINE autolearn=ham autolearn_force=no version=3.4.6 X-Spam-Checker-Version: SpamAssassin 3.4.6 (2021-04-09) on lindbergh.monkeyblade.net X-Patchwork-Delegate: bpf@iogearbox.net From: Menglong Dong Add test9/test10 in fexit_test.c and fentry_test.c to test the fentry and fexit whose target function have 7/12 arguments. Correspondingly, add bpf_testmod_fentry_test7() and bpf_testmod_fentry_test12() to bpf_testmod.c And the testcases passed: ./test_progs -t fexit Summary: 5/12 PASSED, 0 SKIPPED, 0 FAILED ./test_progs -t fentry Summary: 3/0 PASSED, 0 SKIPPED, 0 FAILED Signed-off-by: Menglong Dong --- v4: - use different type for args in bpf_testmod_fentry_test{7,12} - add testcase for grabage values in ctx v3: - move bpf_fentry_test{7,12} to bpf_testmod.c and rename them to bpf_testmod_fentry_test{7,12} meanwhile - get return value by bpf_get_func_ret() in "fexit/bpf_testmod_fentry_test12", as we don't change ___bpf_ctx_cast() in this version --- .../selftests/bpf/bpf_testmod/bpf_testmod.c | 19 ++++++- .../selftests/bpf/prog_tests/fentry_fexit.c | 4 +- .../selftests/bpf/prog_tests/fentry_test.c | 2 + .../selftests/bpf/prog_tests/fexit_test.c | 2 + .../testing/selftests/bpf/progs/fentry_test.c | 33 +++++++++++ .../testing/selftests/bpf/progs/fexit_test.c | 57 +++++++++++++++++++ 6 files changed, 115 insertions(+), 2 deletions(-) diff --git a/tools/testing/selftests/bpf/bpf_testmod/bpf_testmod.c b/tools/testing/selftests/bpf/bpf_testmod/bpf_testmod.c index cf216041876c..66615fdbe3df 100644 --- a/tools/testing/selftests/bpf/bpf_testmod/bpf_testmod.c +++ b/tools/testing/selftests/bpf/bpf_testmod/bpf_testmod.c @@ -191,6 +191,19 @@ noinline int bpf_testmod_fentry_test3(char a, int b, u64 c) return a + b + c; } +noinline int bpf_testmod_fentry_test7(u64 a, void *b, short c, int d, + void *e, u64 f, u64 g) +{ + return a + (long)b + c + d + (long)e + f + g; +} + +noinline int bpf_testmod_fentry_test12(u64 a, void *b, short c, int d, + void *e, u64 f, u64 g, u64 h, + u64 i, u64 j, u64 k, u64 l) +{ + return a + (long)b + c + d + (long)e + f + g + h + i + j + k + l; +} + __diag_pop(); int bpf_testmod_fentry_ok; @@ -245,7 +258,11 @@ bpf_testmod_test_read(struct file *file, struct kobject *kobj, if (bpf_testmod_fentry_test1(1) != 2 || bpf_testmod_fentry_test2(2, 3) != 5 || - bpf_testmod_fentry_test3(4, 5, 6) != 15) + bpf_testmod_fentry_test3(4, 5, 6) != 15 || + bpf_testmod_fentry_test7(16, (void *)17, 18, 19, (void *)20, + 21, 22) != 133 || + bpf_testmod_fentry_test12(16, (void *)17, 18, 19, (void *)20, + 21, 22, 23, 24, 25, 26, 27) != 258) goto out; bpf_testmod_fentry_ok = 1; diff --git a/tools/testing/selftests/bpf/prog_tests/fentry_fexit.c b/tools/testing/selftests/bpf/prog_tests/fentry_fexit.c index 130f5b82d2e6..0078acee0ede 100644 --- a/tools/testing/selftests/bpf/prog_tests/fentry_fexit.c +++ b/tools/testing/selftests/bpf/prog_tests/fentry_fexit.c @@ -31,10 +31,12 @@ void test_fentry_fexit(void) ASSERT_OK(err, "ipv6 test_run"); ASSERT_OK(topts.retval, "ipv6 test retval"); + ASSERT_OK(trigger_module_test_read(1), "trigger_read"); + fentry_res = (__u64 *)fentry_skel->bss; fexit_res = (__u64 *)fexit_skel->bss; printf("%lld\n", fentry_skel->bss->test1_result); - for (i = 0; i < 8; i++) { + for (i = 0; i < 11; i++) { ASSERT_EQ(fentry_res[i], 1, "fentry result"); ASSERT_EQ(fexit_res[i], 1, "fexit result"); } diff --git a/tools/testing/selftests/bpf/prog_tests/fentry_test.c b/tools/testing/selftests/bpf/prog_tests/fentry_test.c index c0d1d61d5f66..e1c0ce40febf 100644 --- a/tools/testing/selftests/bpf/prog_tests/fentry_test.c +++ b/tools/testing/selftests/bpf/prog_tests/fentry_test.c @@ -24,6 +24,8 @@ static int fentry_test(struct fentry_test_lskel *fentry_skel) ASSERT_OK(err, "test_run"); ASSERT_EQ(topts.retval, 0, "test_run"); + ASSERT_OK(trigger_module_test_read(1), "trigger_read"); + result = (__u64 *)fentry_skel->bss; for (i = 0; i < sizeof(*fentry_skel->bss) / sizeof(__u64); i++) { if (!ASSERT_EQ(result[i], 1, "fentry_result")) diff --git a/tools/testing/selftests/bpf/prog_tests/fexit_test.c b/tools/testing/selftests/bpf/prog_tests/fexit_test.c index 101b7343036b..ea81fa913ec6 100644 --- a/tools/testing/selftests/bpf/prog_tests/fexit_test.c +++ b/tools/testing/selftests/bpf/prog_tests/fexit_test.c @@ -24,6 +24,8 @@ static int fexit_test(struct fexit_test_lskel *fexit_skel) ASSERT_OK(err, "test_run"); ASSERT_EQ(topts.retval, 0, "test_run"); + ASSERT_OK(trigger_module_test_read(1), "trigger_read"); + result = (__u64 *)fexit_skel->bss; for (i = 0; i < sizeof(*fexit_skel->bss) / sizeof(__u64); i++) { if (!ASSERT_EQ(result[i], 1, "fexit_result")) diff --git a/tools/testing/selftests/bpf/progs/fentry_test.c b/tools/testing/selftests/bpf/progs/fentry_test.c index 52a550d281d9..91dbf63b3ba1 100644 --- a/tools/testing/selftests/bpf/progs/fentry_test.c +++ b/tools/testing/selftests/bpf/progs/fentry_test.c @@ -77,3 +77,36 @@ int BPF_PROG(test8, struct bpf_fentry_test_t *arg) test8_result = 1; return 0; } + +__u64 test9_result = 0; +SEC("fentry/bpf_testmod_fentry_test7") +int BPF_PROG(test9, __u64 a, void *b, short c, int d, void *e, char f, + int g) +{ + test9_result = a == 16 && b == (void *)17 && c == 18 && d == 19 && + e == (void *)20 && f == 21 && g == 22; + return 0; +} + +__u64 test10_result = 0; +SEC("fentry/bpf_testmod_fentry_test12") +int BPF_PROG(test10, __u64 a, void *b, short c, int d, void *e, char f, + int g, unsigned int h, long i, __u64 j, unsigned long k, + unsigned char l) +{ + test10_result = a == 16 && b == (void *)17 && c == 18 && d == 19 && + e == (void *)20 && f == 21 && g == 22 && h == 23 && + i == 24 && j == 25 && k == 26 && l == 27; + return 0; +} + +__u64 test11_result = 0; +SEC("fentry/bpf_testmod_fentry_test12") +int BPF_PROG(test11, __u64 a, __u64 b, __u64 c, __u64 d, __u64 e, __u64 f, + __u64 g, __u64 h, __u64 i, __u64 j, __u64 k, __u64 l) +{ + test11_result = a == 16 && b == 17 && c == 18 && d == 19 && + e == 20 && f == 21 && g == 22 && h == 23 && + i == 24 && j == 25 && k == 26 && l == 27; + return 0; +} diff --git a/tools/testing/selftests/bpf/progs/fexit_test.c b/tools/testing/selftests/bpf/progs/fexit_test.c index 8f1ccb7302e1..a6d8e03ff5b7 100644 --- a/tools/testing/selftests/bpf/progs/fexit_test.c +++ b/tools/testing/selftests/bpf/progs/fexit_test.c @@ -78,3 +78,60 @@ int BPF_PROG(test8, struct bpf_fentry_test_t *arg) test8_result = 1; return 0; } + +__u64 test9_result = 0; +SEC("fexit/bpf_testmod_fentry_test7") +int BPF_PROG(test9, __u64 a, void *b, short c, int d, void *e, char f, + int g, int ret) +{ + test9_result = a == 16 && b == (void *)17 && c == 18 && d == 19 && + e == (void *)20 && f == 21 && g == 22 && ret == 133; + return 0; +} + +__u64 test10_result = 0; +SEC("fexit/bpf_testmod_fentry_test12") +int BPF_PROG(test10, __u64 a, void *b, short c, int d, void *e, char f, + int g, unsigned int h, long i, __u64 j, unsigned long k, + unsigned char l) +{ + __u64 ret; + int err; + + /* BPF_PROG() don't support 14 arguments, and ctx[12] can't be + * accessed yet. So we get the return value by bpf_get_func_ret() + * for now. + */ + err = bpf_get_func_ret(ctx, &ret); + if (err) + return 0; + + test10_result = a == 16 && b == (void *)17 && c == 18 && d == 19 && + e == (void *)20 && f == 21 && g == 22 && h == 23 && + i == 24 && j == 25 && k == 26 && l == 27 && + (int)ret == 258; + return 0; +} + +__u64 test11_result = 0; +SEC("fexit/bpf_testmod_fentry_test12") +int BPF_PROG(test11, __u64 a, __u64 b, __u64 c, __u64 d, __u64 e, __u64 f, + __u64 g, __u64 h, __u64 i, __u64 j, __u64 k, __u64 l) +{ + __u64 ret; + int err; + + /* BPF_PROG() don't support 14 arguments, and ctx[12] can't be + * accessed yet. So we get the return value by bpf_get_func_ret() + * for now. + */ + err = bpf_get_func_ret(ctx, &ret); + if (err) + return 0; + + test11_result = a == 16 && b == 17 && c == 18 && d == 19 && + e == 20 && f == 21 && g == 22 && h == 23 && + i == 24 && j == 25 && k == 26 && l == 27 && + ret == 258; + return 0; +}