From patchwork Mon Mar 4 20:28:03 2024 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Puranjay Mohan X-Patchwork-Id: 13581200 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from bombadil.infradead.org (bombadil.infradead.org [198.137.202.133]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (No client certificate requested) by smtp.lore.kernel.org (Postfix) with ESMTPS id 84244C48BF6 for ; Mon, 4 Mar 2024 20:28:41 +0000 (UTC) DKIM-Signature: v=1; a=rsa-sha256; q=dns/txt; c=relaxed/relaxed; d=lists.infradead.org; s=bombadil.20210309; h=Sender: Content-Transfer-Encoding:Content-Type:List-Subscribe:List-Help:List-Post: List-Archive:List-Unsubscribe:List-Id:MIME-Version:Message-Id:Date:Subject:Cc :To:From:Reply-To:Content-ID:Content-Description:Resent-Date:Resent-From: Resent-Sender:Resent-To:Resent-Cc:Resent-Message-ID:In-Reply-To:References: List-Owner; bh=kKr8Ggw0w3FTqRepUFUHoek/EFS42k+9Y3s9lb7yg7E=; b=EDRuuAIorbb+Gr Os1CuNqUFQbkY1K1BvYc0+hLolEEb2WvRdwbsW3HspK8Kd7WoynIgg5Y38r3WbsWtf6z1awG/LTzi CAfDCtYJm1pnEhpGgGvFubBfovI8SwhC50QysVu3rzWH9UV+zkCDQUjxI0K5UH3pvyUY0zh2Eh/zo pizvmBD7Le8n2CxNibYqGsr9KERVFDrl/Rt6SgAH0MM7rfnQFHLxCIIb+M8ALwNV5TYR2U8C3kvFs f1oT9PTYX4lLrPu4c946J9+dj2bXziQRVj873QXPwlDLngaI0hPYk1fIO+Ky4MAVWKB+F69NnEW3J JYrEb6rPvxj+e6w+6rhw==; Received: from localhost ([::1] helo=bombadil.infradead.org) by bombadil.infradead.org with esmtp (Exim 4.97.1 #2 (Red Hat Linux)) id 1rhEvI-0000000Ac7F-0hOt; Mon, 04 Mar 2024 20:28:32 +0000 Received: from mail-wm1-x334.google.com ([2a00:1450:4864:20::334]) by bombadil.infradead.org with esmtps (Exim 4.97.1 #2 (Red Hat Linux)) id 1rhEvE-0000000Ac52-3glV for linux-arm-kernel@lists.infradead.org; Mon, 04 Mar 2024 20:28:30 +0000 Received: by mail-wm1-x334.google.com with SMTP id 5b1f17b1804b1-412e795f458so6913835e9.3 for ; Mon, 04 Mar 2024 12:28:26 -0800 (PST) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=20230601; t=1709584105; x=1710188905; darn=lists.infradead.org; h=content-transfer-encoding:mime-version:message-id:date:subject:cc :to:from:from:to:cc:subject:date:message-id:reply-to; bh=C8374zjIQzoAP6NawexwSb/Q1q4lgTdoXyDk5N8EOwc=; b=VJYqqxBOD87vfTB/gkIYRZJDX31XNwQIhqmk8t4kUaw8Q4AqFIIfJ777RtowKgbI0X notRYuzRxxtWSxqd1LjxImje+pkw1Ki+glVYSf6/HY9T2+o3ZS8dp9VYe+VJcgmXtLiI hF+b82q/NxsI0V+VoCkJjqxLyieAQ5HPdIm4LyxTQ+qAQr7h5cyKEa/qB4TBc3Frq7kI 0XJb9+EQNnNkXEIEoLbTukByAxwlKkoITfWEqz/BoMyDiGbcaXeIeTn476eev2+2ND3C 3Pst2R8bUpGx0gleODDlLqBqtmr+bxbY1LBa5PrOnhCdixnSWjMDApPZbzCNc4ASMlCc 1RJA== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20230601; t=1709584105; x=1710188905; h=content-transfer-encoding:mime-version:message-id:date:subject:cc :to:from:x-gm-message-state:from:to:cc:subject:date:message-id :reply-to; bh=C8374zjIQzoAP6NawexwSb/Q1q4lgTdoXyDk5N8EOwc=; b=rCQRXGLmDuccjKFm31dOoGuiNUwU9FKhv116vAurXP5Ru3fDOP5OrTZsenBIWPxTFZ PH6Kq8in29hGMxOIILduHSyOugYaCtxMKKlS7cvEt0i0rae09yBLOLdtdg5bcJ5nfeKP ZPYzOBx04PCsUOZM+KLj1nLfveO3UNnOcc4w6l/iybFNLBZPdKHCnInk3/0sN4lqpkbq SGun+lBkG0RHYRHyEDUz8HLfN6Np4BmCouePd80fzt13xR16qeZCfN78hjK1CucXp8hS TEzKD/OFjYNXM/l3ff2uAXXqWMGjX2wwCI8ivwMKaG2/QSEoQ1bqwgY1y4qswI3TAq7w HF2Q== X-Forwarded-Encrypted: i=1; AJvYcCVMuBh6DZ5d20qznAzzJbomFgEuPi4oFh64RxP2nEdEjqVP9yiBC/pRABIjcusKcVdgsdpLx00iEfTcqzy/3an+JliCdXMRe+f47l/to3JuK4VO840= X-Gm-Message-State: AOJu0YwgYF3cWsVnyxAVgZsU65eGfnCH0th5zqAWtf2qs/MoQF2crwM9 SyAh9zw9yE/BV+amETvO5opsUSN7BA7p8iZhxetYHN77TRPNHRV1 X-Google-Smtp-Source: AGHT+IHbTbMtHaq9bspX/vZCCO1+aoX9vsybER44sO+00buq4wgGQxNpZxIz6qKi6v4tq9zIdZpxYQ== X-Received: by 2002:a05:600c:5191:b0:412:e902:87a7 with SMTP id fa17-20020a05600c519100b00412e90287a7mr1117353wmb.20.1709584105222; Mon, 04 Mar 2024 12:28:25 -0800 (PST) Received: from localhost (54-240-197-231.amazon.com. [54.240.197.231]) by smtp.gmail.com with ESMTPSA id o39-20020a05600c512700b00412d60cb914sm8808428wms.5.2024.03.04.12.28.24 (version=TLS1_2 cipher=ECDHE-ECDSA-AES128-GCM-SHA256 bits=128/128); Mon, 04 Mar 2024 12:28:24 -0800 (PST) From: Puranjay Mohan To: ast@kernel.org, daniel@iogearbox.net, andrii@kernel.org, martin.lau@linux.dev, song@kernel.org, catalin.marinas@arm.com, mark.rutland@arm.com, bpf@vger.kernel.org, kpsingh@kernel.org, linux-arm-kernel@lists.infradead.org, linux-kernel@vger.kernel.org, xukuohai@huaweicloud.com Cc: puranjay12@gmail.com Subject: [PATCH bpf-next] arm64, bpf: Use bpf_prog_pack for arm64 bpf trampoline Date: Mon, 4 Mar 2024 20:28:03 +0000 Message-Id: <20240304202803.31400-1-puranjay12@gmail.com> X-Mailer: git-send-email 2.40.1 MIME-Version: 1.0 X-CRM114-Version: 20100106-BlameMichelson ( TRE 0.8.0 (BSD) ) MR-646709E3 X-CRM114-CacheID: sfid-20240304_122828_947729_E49FC579 X-CRM114-Status: GOOD ( 17.06 ) X-BeenThere: linux-arm-kernel@lists.infradead.org X-Mailman-Version: 2.1.34 Precedence: list List-Id: List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Sender: "linux-arm-kernel" Errors-To: linux-arm-kernel-bounces+linux-arm-kernel=archiver.kernel.org@lists.infradead.org We used bpf_prog_pack to aggregate bpf programs into huge page to relieve the iTLB pressure on the system. This was merged for ARM64[1] We can apply it to bpf trampoline as well. This would increase the preformance of fentry and struct_ops programs. [1] https://lore.kernel.org/bpf/20240228141824.119877-1-puranjay12@gmail.com/ Signed-off-by: Puranjay Mohan Reviewed-by: Pu Lehui --- arch/arm64/net/bpf_jit_comp.c | 55 +++++++++++++++++++++++++++++------ 1 file changed, 46 insertions(+), 9 deletions(-) diff --git a/arch/arm64/net/bpf_jit_comp.c b/arch/arm64/net/bpf_jit_comp.c index 5afc7a525eca..c5b461dda438 100644 --- a/arch/arm64/net/bpf_jit_comp.c +++ b/arch/arm64/net/bpf_jit_comp.c @@ -2076,7 +2076,7 @@ static int prepare_trampoline(struct jit_ctx *ctx, struct bpf_tramp_image *im, /* store return value */ emit(A64_STR64I(A64_R(0), A64_SP, retval_off), ctx); /* reserve a nop for bpf_tramp_image_put */ - im->ip_after_call = ctx->image + ctx->idx; + im->ip_after_call = ctx->ro_image + ctx->idx; emit(A64_NOP, ctx); } @@ -2091,7 +2091,7 @@ static int prepare_trampoline(struct jit_ctx *ctx, struct bpf_tramp_image *im, run_ctx_off, false); if (flags & BPF_TRAMP_F_CALL_ORIG) { - im->ip_epilogue = ctx->image + ctx->idx; + im->ip_epilogue = ctx->ro_image + ctx->idx; emit_addr_mov_i64(A64_R(0), (const u64)im, ctx); emit_call((const u64)__bpf_tramp_exit, ctx); } @@ -2124,9 +2124,6 @@ static int prepare_trampoline(struct jit_ctx *ctx, struct bpf_tramp_image *im, emit(A64_RET(A64_R(10)), ctx); } - if (ctx->image) - bpf_flush_icache(ctx->image, ctx->image + ctx->idx); - kfree(branches); return ctx->idx; @@ -2169,14 +2166,43 @@ int arch_bpf_trampoline_size(const struct btf_func_model *m, u32 flags, return ret < 0 ? ret : ret * AARCH64_INSN_SIZE; } -int arch_prepare_bpf_trampoline(struct bpf_tramp_image *im, void *image, - void *image_end, const struct btf_func_model *m, +void *arch_alloc_bpf_trampoline(unsigned int size) +{ + return bpf_prog_pack_alloc(size, jit_fill_hole); +} + +void arch_free_bpf_trampoline(void *image, unsigned int size) +{ + bpf_prog_pack_free(image, size); +} + +void arch_protect_bpf_trampoline(void *image, unsigned int size) +{ +} + +void arch_unprotect_bpf_trampoline(void *image, unsigned int size) +{ +} + +int arch_prepare_bpf_trampoline(struct bpf_tramp_image *im, void *ro_image, + void *ro_image_end, const struct btf_func_model *m, u32 flags, struct bpf_tramp_links *tlinks, void *func_addr) { int ret, nregs; + void *image, *tmp; + u32 size = ro_image_end - ro_image; + + /* image doesn't need to be in module memory range, so we can + * use kvmalloc. + */ + image = kvmalloc(size, GFP_KERNEL); + if (!image) + return -ENOMEM; + struct jit_ctx ctx = { .image = image, + .ro_image = ro_image, .idx = 0, }; @@ -2185,15 +2211,26 @@ int arch_prepare_bpf_trampoline(struct bpf_tramp_image *im, void *image, if (nregs > 8) return -ENOTSUPP; - jit_fill_hole(image, (unsigned int)(image_end - image)); + jit_fill_hole(image, (unsigned int)(ro_image_end - ro_image)); ret = prepare_trampoline(&ctx, im, tlinks, func_addr, nregs, flags); - if (ret > 0 && validate_code(&ctx) < 0) + if (ret > 0 && validate_code(&ctx) < 0) { ret = -EINVAL; + goto out; + } if (ret > 0) ret *= AARCH64_INSN_SIZE; + tmp = bpf_arch_text_copy(ro_image, image, size); + if (IS_ERR(tmp)) { + ret = PTR_ERR(tmp); + goto out; + } + + bpf_flush_icache(ro_image, ro_image + size); +out: + kvfree(image); return ret; }