From patchwork Thu Jul 20 15:49:40 2023 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 8bit X-Patchwork-Submitter: Puranjay Mohan X-Patchwork-Id: 13320719 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from bombadil.infradead.org (bombadil.infradead.org [198.137.202.133]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (No client certificate requested) by smtp.lore.kernel.org (Postfix) with ESMTPS id BFF40C001DC for ; Thu, 20 Jul 2023 15:49:54 +0000 (UTC) DKIM-Signature: v=1; a=rsa-sha256; q=dns/txt; c=relaxed/relaxed; d=lists.infradead.org; s=bombadil.20210309; h=Sender: Content-Transfer-Encoding:Content-Type:List-Subscribe:List-Help:List-Post: List-Archive:List-Unsubscribe:List-Id:MIME-Version:References:In-Reply-To: Message-Id:Date:Subject:Cc:To:From:Reply-To:Content-ID:Content-Description: Resent-Date:Resent-From:Resent-Sender:Resent-To:Resent-Cc:Resent-Message-ID: List-Owner; bh=akqHSIMbI3KFApecjxaMYwsfPGo/fjUlpvOsue1oxNo=; b=dlUO3y5qD2NR6U kBg1jKgiYZZNcxEukcX4JPkwHGb7UzL4Z6TP782ocQT6qglcLK6lxh9UthCyvfUXGXcyqFrIchkc7 PnEErAk9/Fkuef4k/57fiFOfVFtlU2lC+DLNupLiVpcKvsVdscQoToqJ9ifFrzfuCw+sH6QCGf/p3 jC0R5kQp4SCtRS/tW66mjYBU102QfcWXCF397wUgFWkRFexfs4nWc7m0DfQW21HWyr5O9Q6ksZ0LW 9zBOM6cDzEhpbeJ6i508Hc0sKgAe8Z1kdDFBilq/X3V5LA4dZq3LOae/hII/ESF3AWNfGNq0aN/GJ me70/8A6Z1gS2U7tGXZA==; Received: from localhost ([::1] helo=bombadil.infradead.org) by bombadil.infradead.org with esmtp (Exim 4.96 #2 (Red Hat Linux)) id 1qMVuZ-00BaPv-18; Thu, 20 Jul 2023 15:49:51 +0000 Received: from mail-wm1-x332.google.com ([2a00:1450:4864:20::332]) by bombadil.infradead.org with esmtps (Exim 4.96 #2 (Red Hat Linux)) id 1qMVuV-00BaNp-2y for linux-riscv@lists.infradead.org; Thu, 20 Jul 2023 15:49:49 +0000 Received: by mail-wm1-x332.google.com with SMTP id 5b1f17b1804b1-3fbc63c2e84so8156595e9.3 for ; Thu, 20 Jul 2023 08:49:44 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=20221208; t=1689868183; x=1690472983; h=content-transfer-encoding:mime-version:references:in-reply-to :message-id:date:subject:cc:to:from:from:to:cc:subject:date :message-id:reply-to; bh=+TdRmj0UkAm5gPWjI6OLYQN5hjGdtS3ZVtVx8EMYM4Y=; b=F7yWBNeaU2+/3BBqgAuECm5xIe9Z49EL2iGWNoPjMiiSOa6u+saOFkPGbnxYdlUe8J Hu7XI/oX+2x/xn/pyzs/tvziKsRDPzA1cWMqNj+jvEMxUec5OXy72zcN6UNx893UAPKb fdNU/3kjaN/aa401cgeZxLYyUfa1QkkTSOL6j7JnazSTAYDjlAsR4e9Bx0ubQ6t5zU2F kIAOeLUBfuoVOJaAWI5hkY8H90OjfbjdRi9NwqDkFrxHYe6dQR3Jj607PDh4Og/o2LbE lp1q3TIbOpdzUHzO4e3DphgNIGi4F4CXjEO3oZAvRDLDNlStHvyCijkV9MVIj6e4ilv0 Q8iQ== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20221208; t=1689868183; x=1690472983; h=content-transfer-encoding:mime-version:references:in-reply-to :message-id:date:subject:cc:to:from:x-gm-message-state:from:to:cc :subject:date:message-id:reply-to; bh=+TdRmj0UkAm5gPWjI6OLYQN5hjGdtS3ZVtVx8EMYM4Y=; b=LMahAh7a9f6uz3m1y8c6EEbq/+kibeahRlpfP2Tx2MeVUI8kYVtoONGoVl8oF3mvw6 zNALl86uFH1JfcvL61c9ldbKTahgN0KTPYSCwNS3hgwKBcfZ5qolUjpy6QRgDggpupRK NfOvVk0on8t5wx8Jjiqz5MPk0PHVaeLtDZ0PYWpazscOL55jZmFM7yQ4TfPYGqbFHu3T hq5avC4xcV7DMtDF7W/K5FO0Hh6TheBl9CwQj2UtbH2Twugc3/jKQYedSE+tX30G3aj8 b68Z7UEcXc/q3LbPf63UWfjpzNXXN5QEf9t5VJcBIYuSaA2kKOB13a5sq6prObK1igMA 80aA== X-Gm-Message-State: ABy/qLb6kHpcD5GXNd4PqPHLrBw8CSlt+szkJjWny34rLihA2eK6WleJ rR3zrEoXLJEr88WYFyFn/2I= X-Google-Smtp-Source: APBJJlEuXMI5sektNSXxZZSfzFEf8zfF8nebOdPeEXqLNDX8t0fvGqMk9j5hKGKR3pU29ru33nGvrw== X-Received: by 2002:a05:600c:3647:b0:3f9:8c3:6805 with SMTP id y7-20020a05600c364700b003f908c36805mr4563818wmq.7.1689868183273; Thu, 20 Jul 2023 08:49:43 -0700 (PDT) Received: from ip-172-31-22-112.eu-west-1.compute.internal (ec2-34-244-51-157.eu-west-1.compute.amazonaws.com. [34.244.51.157]) by smtp.gmail.com with ESMTPSA id u6-20020a05600c00c600b003fbdd9c72aasm1471021wmm.21.2023.07.20.08.49.42 (version=TLS1_3 cipher=TLS_AES_256_GCM_SHA384 bits=256/256); Thu, 20 Jul 2023 08:49:43 -0700 (PDT) From: Puranjay Mohan To: paul.walmsley@sifive.com, palmer@dabbelt.com, aou@eecs.berkeley.edu, pulehui@huawei.com, conor.dooley@microchip.com, ast@kernel.org, daniel@iogearbox.net, andrii@kernel.org, martin.lau@linux.dev, song@kernel.org, yhs@fb.com, kpsingh@kernel.org, bjorn@kernel.org, bpf@vger.kernel.org, linux-riscv@lists.infradead.org, linux-kernel@vger.kernel.org Cc: puranjay12@gmail.com Subject: [PATCH bpf-next 1/2] riscv: Extend patch_text_nosync() for multiple pages Date: Thu, 20 Jul 2023 15:49:40 +0000 Message-Id: <20230720154941.1504-2-puranjay12@gmail.com> X-Mailer: git-send-email 2.40.1 In-Reply-To: <20230720154941.1504-1-puranjay12@gmail.com> References: <20230720154941.1504-1-puranjay12@gmail.com> MIME-Version: 1.0 X-CRM114-Version: 20100106-BlameMichelson ( TRE 0.8.0 (BSD) ) MR-646709E3 X-CRM114-CacheID: sfid-20230720_084947_961925_352AC7FE X-CRM114-Status: GOOD ( 12.95 ) X-BeenThere: linux-riscv@lists.infradead.org X-Mailman-Version: 2.1.34 Precedence: list List-Id: List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Sender: "linux-riscv" Errors-To: linux-riscv-bounces+linux-riscv=archiver.kernel.org@lists.infradead.org The patch_insn_write() function currently doesn't work for multiple pages of instructions, therefore patch_text_nosync() will fail with a page fault if called with lengths spanning multiple pages. This commit extends the patch_insn_write() function to support multiple pages by copying at max 2 pages at a time in a loop. This implementation is similar to text_poke_copy() function of x86. Signed-off-by: Puranjay Mohan Reviewed-by: Björn Töpel --- arch/riscv/kernel/patch.c | 29 ++++++++++++++++++++++++----- 1 file changed, 24 insertions(+), 5 deletions(-) diff --git a/arch/riscv/kernel/patch.c b/arch/riscv/kernel/patch.c index 575e71d6c8ae..b2dbfcfdef85 100644 --- a/arch/riscv/kernel/patch.c +++ b/arch/riscv/kernel/patch.c @@ -53,7 +53,7 @@ static void patch_unmap(int fixmap) } NOKPROBE_SYMBOL(patch_unmap); -static int patch_insn_write(void *addr, const void *insn, size_t len) +static int __patch_insn_write(void *addr, const void *insn, size_t len) { void *waddr = addr; bool across_pages = (((uintptr_t) addr & ~PAGE_MASK) + len) > PAGE_SIZE; @@ -74,7 +74,7 @@ static int patch_insn_write(void *addr, const void *insn, size_t len) lockdep_assert_held(&text_mutex); if (across_pages) - patch_map(addr + len, FIX_TEXT_POKE1); + patch_map(addr + PAGE_SIZE, FIX_TEXT_POKE1); waddr = patch_map(addr, FIX_TEXT_POKE0); @@ -87,15 +87,34 @@ static int patch_insn_write(void *addr, const void *insn, size_t len) return ret; } -NOKPROBE_SYMBOL(patch_insn_write); +NOKPROBE_SYMBOL(__patch_insn_write); #else -static int patch_insn_write(void *addr, const void *insn, size_t len) +static int __patch_insn_write(void *addr, const void *insn, size_t len) { return copy_to_kernel_nofault(addr, insn, len); } -NOKPROBE_SYMBOL(patch_insn_write); +NOKPROBE_SYMBOL(__patch_insn_write); #endif /* CONFIG_MMU */ +static int patch_insn_write(void *addr, const void *insn, size_t len) +{ + size_t patched = 0; + size_t size; + int ret = 0; + + while (patched < len && !ret) { + size = min_t(size_t, + PAGE_SIZE * 2 - offset_in_page(addr + patched), + len - patched); + ret = __patch_insn_write(addr + patched, insn + patched, size); + + patched += size; + } + + return ret; +} +NOKPROBE_SYMBOL(patch_insn_write); + int patch_text_nosync(void *addr, const void *insns, size_t len) { u32 *tp = addr; From patchwork Thu Jul 20 15:49:41 2023 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Puranjay Mohan X-Patchwork-Id: 13320720 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from bombadil.infradead.org (bombadil.infradead.org [198.137.202.133]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (No client certificate requested) by smtp.lore.kernel.org (Postfix) with ESMTPS id 7C510EB64DA for ; Thu, 20 Jul 2023 15:49:55 +0000 (UTC) DKIM-Signature: v=1; a=rsa-sha256; q=dns/txt; c=relaxed/relaxed; d=lists.infradead.org; s=bombadil.20210309; h=Sender: Content-Transfer-Encoding:Content-Type:List-Subscribe:List-Help:List-Post: List-Archive:List-Unsubscribe:List-Id:MIME-Version:References:In-Reply-To: Message-Id:Date:Subject:Cc:To:From:Reply-To:Content-ID:Content-Description: Resent-Date:Resent-From:Resent-Sender:Resent-To:Resent-Cc:Resent-Message-ID: List-Owner; bh=jccHh64ZFfZ61oTFFSXi18cKLYjUosetE0NRf2MZwzo=; b=3+37w25rtN1byd jjDbubosOu2ZzImeqYqNDjglDV72rh/2kfTZsI295HSnMazBIanCwYtoRQqyMRyGVp++fysGvCWUZ K7AtV5CpVvBDNEPn0jleN7dnGdvIe/dqm/zyzYBYOdo7U/KnDCa//32r5vfb3T3dGpbW+veb+fZMu On+Ag7MXaZLYi/KhgQJUvvVz9HXHpEa1KDhZ+dKzETK59z0XB0Q1e0yO1gUVpi+OSrycfYfypnKFa h5e1jEexycmpTy5X0QjQshy8hIS5OMT3mGTKwvS8hoR4j2JBWgnyQWkz2bUWOdP0qzAqsAc29zqHB LIrfXrOHA+arrPuwxVMw==; Received: from localhost ([::1] helo=bombadil.infradead.org) by bombadil.infradead.org with esmtp (Exim 4.96 #2 (Red Hat Linux)) id 1qMVuZ-00BaQS-31; Thu, 20 Jul 2023 15:49:51 +0000 Received: from mail-lj1-x22f.google.com ([2a00:1450:4864:20::22f]) by bombadil.infradead.org with esmtps (Exim 4.96 #2 (Red Hat Linux)) id 1qMVuW-00BaO2-26 for linux-riscv@lists.infradead.org; Thu, 20 Jul 2023 15:49:50 +0000 Received: by mail-lj1-x22f.google.com with SMTP id 38308e7fff4ca-2b74310566cso14606341fa.2 for ; Thu, 20 Jul 2023 08:49:45 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=20221208; t=1689868184; x=1690472984; h=content-transfer-encoding:mime-version:references:in-reply-to :message-id:date:subject:cc:to:from:from:to:cc:subject:date :message-id:reply-to; bh=Tljgv3DtPBaijlderTUsocnZ3FS2Qirhc9/9iIaydKs=; b=ZU2x2o1rMgFGMr8mMVk6zGghYR6w7eoIHkKwSq3ON29PON+cRVMHnVFl6821unV/mO PxYXgW6nJtQJY9SXWlcN0r5O8JGioMwhykODrnXrpdTFA2SdFKZz9JzRlCikzsOlD8re Kh9j82zvryW37+UUsxGUOEdu7mdbKly0I2VBLSztlhT1g5be9+b4FUC7+FLVr+gt1gP+ QhSMTY+aN0exmyvd0U+9co6wmqy4Xwq9OzTA8ysaDBygybD5cUP7DsbwVeqEEXvSsjUl n03Hs2ccd44WtQXOQZNV5WIYuzDJ4VRDBKrwQkcXIwHRL2Z7OQKt1pnda3GA/rllQIWy 2OPQ== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20221208; t=1689868184; x=1690472984; h=content-transfer-encoding:mime-version:references:in-reply-to :message-id:date:subject:cc:to:from:x-gm-message-state:from:to:cc :subject:date:message-id:reply-to; bh=Tljgv3DtPBaijlderTUsocnZ3FS2Qirhc9/9iIaydKs=; b=QuGn5ihEVY53LpWT/ve4cqHVe92G4BzGGXZn8Zr78YRsaI/OOoNv/gRaDceSNSVijZ Nd/Iks6TxgAETdFBPSNATJ1C4rxc/FRUjtWOLrx3RPsT9ZOkEuCN7CKFICGCxJbBUZKo cEA2dFIX3i9bldbOkuEz+iNIbrreSXsygKM3ZyV4W+SMtWoheg8gr5ZSqgSA4AlMm7pE ztca/5SYiZ+hgv7Lb61WM0xpJkeez+pWzMF7pw5rlk6FD4FF3ow/lBjr2P0CNu5cUPdl f8Mxa5OP2tNkI6U3Empi/LSAw1GI8oC0oDjXhx60+lrY2MHX830GKVVCVNOGh9wGhZKY f/sQ== X-Gm-Message-State: ABy/qLYO5IhDvv/hUq7iNzQilY3/hJ2JYjET6SYjpE4G7y8FR7WS1u6q Kmm6pfXsI0eezcVvu2SfiOA= X-Google-Smtp-Source: APBJJlEL6GlbcfCTPUDbfQ9jRnM/KFl0/76YNmQ+0eEdndU1KtOqzyJGQKfALdiYSzqZ9grORZOwuQ== X-Received: by 2002:a2e:9c4e:0:b0:2b9:4805:a039 with SMTP id t14-20020a2e9c4e000000b002b94805a039mr2201554ljj.51.1689868184061; Thu, 20 Jul 2023 08:49:44 -0700 (PDT) Received: from ip-172-31-22-112.eu-west-1.compute.internal (ec2-34-244-51-157.eu-west-1.compute.amazonaws.com. [34.244.51.157]) by smtp.gmail.com with ESMTPSA id u6-20020a05600c00c600b003fbdd9c72aasm1471021wmm.21.2023.07.20.08.49.43 (version=TLS1_3 cipher=TLS_AES_256_GCM_SHA384 bits=256/256); Thu, 20 Jul 2023 08:49:43 -0700 (PDT) From: Puranjay Mohan To: paul.walmsley@sifive.com, palmer@dabbelt.com, aou@eecs.berkeley.edu, pulehui@huawei.com, conor.dooley@microchip.com, ast@kernel.org, daniel@iogearbox.net, andrii@kernel.org, martin.lau@linux.dev, song@kernel.org, yhs@fb.com, kpsingh@kernel.org, bjorn@kernel.org, bpf@vger.kernel.org, linux-riscv@lists.infradead.org, linux-kernel@vger.kernel.org Cc: puranjay12@gmail.com Subject: [PATCH bpf-next 2/2] bpf, riscv: use prog pack allocator in the BPF JIT Date: Thu, 20 Jul 2023 15:49:41 +0000 Message-Id: <20230720154941.1504-3-puranjay12@gmail.com> X-Mailer: git-send-email 2.40.1 In-Reply-To: <20230720154941.1504-1-puranjay12@gmail.com> References: <20230720154941.1504-1-puranjay12@gmail.com> MIME-Version: 1.0 X-CRM114-Version: 20100106-BlameMichelson ( TRE 0.8.0 (BSD) ) MR-646709E3 X-CRM114-CacheID: sfid-20230720_084948_697681_394149F1 X-CRM114-Status: GOOD ( 29.68 ) X-BeenThere: linux-riscv@lists.infradead.org X-Mailman-Version: 2.1.34 Precedence: list List-Id: List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Sender: "linux-riscv" Errors-To: linux-riscv-bounces+linux-riscv=archiver.kernel.org@lists.infradead.org Use bpf_jit_binary_pack_alloc() for memory management of JIT binaries in RISCV BPF JIT. The bpf_jit_binary_pack_alloc creates a pair of RW and RX buffers. The JIT writes the program into the RW buffer. When the JIT is done, the program is copied to the final RX buffer with bpf_jit_binary_pack_finalize. Implement bpf_arch_text_copy() and bpf_arch_text_invalidate() for RISCV JIT as these functions are required by bpf_jit_binary_pack allocator. Signed-off-by: Puranjay Mohan --- arch/riscv/net/bpf_jit.h | 3 + arch/riscv/net/bpf_jit_comp64.c | 56 ++++++++++++--- arch/riscv/net/bpf_jit_core.c | 117 +++++++++++++++++++++++++++----- 3 files changed, 150 insertions(+), 26 deletions(-) diff --git a/arch/riscv/net/bpf_jit.h b/arch/riscv/net/bpf_jit.h index 2717f5490428..ad69319c8ea7 100644 --- a/arch/riscv/net/bpf_jit.h +++ b/arch/riscv/net/bpf_jit.h @@ -68,6 +68,7 @@ static inline bool is_creg(u8 reg) struct rv_jit_context { struct bpf_prog *prog; u16 *insns; /* RV insns */ + u16 *ro_insns; int ninsns; int prologue_len; int epilogue_offset; @@ -85,7 +86,9 @@ static inline int ninsns_rvoff(int ninsns) struct rv_jit_data { struct bpf_binary_header *header; + struct bpf_binary_header *ro_header; u8 *image; + u8 *ro_image; struct rv_jit_context ctx; }; diff --git a/arch/riscv/net/bpf_jit_comp64.c b/arch/riscv/net/bpf_jit_comp64.c index c648864c8cd1..3d6165c6608b 100644 --- a/arch/riscv/net/bpf_jit_comp64.c +++ b/arch/riscv/net/bpf_jit_comp64.c @@ -142,7 +142,11 @@ static bool in_auipc_jalr_range(s64 val) /* Emit fixed-length instructions for address */ static int emit_addr(u8 rd, u64 addr, bool extra_pass, struct rv_jit_context *ctx) { - u64 ip = (u64)(ctx->insns + ctx->ninsns); + /* + * Use the ro_insns(RX) to calculate the offset as the BPF program will + * finally run from this memory region. + */ + u64 ip = (u64)(ctx->ro_insns + ctx->ninsns); s64 off = addr - ip; s64 upper = (off + (1 << 11)) >> 12; s64 lower = off & 0xfff; @@ -463,7 +467,11 @@ static int emit_call(u64 addr, bool fixed_addr, struct rv_jit_context *ctx) u64 ip; if (addr && ctx->insns) { - ip = (u64)(long)(ctx->insns + ctx->ninsns); + /* + * Use the ro_insns(RX) to calculate the offset as the BPF + * program will finally run from this memory region. + */ + ip = (u64)(long)(ctx->ro_insns + ctx->ninsns); off = addr - ip; } @@ -576,7 +584,8 @@ static int add_exception_handler(const struct bpf_insn *insn, { struct exception_table_entry *ex; unsigned long pc; - off_t offset; + off_t ins_offset; + off_t fixup_offset; if (!ctx->insns || !ctx->prog->aux->extable || BPF_MODE(insn->code) != BPF_PROBE_MEM) return 0; @@ -591,12 +600,17 @@ static int add_exception_handler(const struct bpf_insn *insn, return -EINVAL; ex = &ctx->prog->aux->extable[ctx->nexentries]; - pc = (unsigned long)&ctx->insns[ctx->ninsns - insn_len]; + pc = (unsigned long)&ctx->ro_insns[ctx->ninsns - insn_len]; - offset = pc - (long)&ex->insn; - if (WARN_ON_ONCE(offset >= 0 || offset < INT_MIN)) + /* + * This is the relative offset of the instruction that may fault from + * the exception table itself. This will be written to the exception + * table and if this instruction faults, the destination register will + * be set to '0' and the execution will jump to the next instruction. + */ + ins_offset = pc - (long)&ex->insn; + if (WARN_ON_ONCE(ins_offset >= 0 || ins_offset < INT_MIN)) return -ERANGE; - ex->insn = offset; /* * Since the extable follows the program, the fixup offset is always @@ -605,12 +619,25 @@ static int add_exception_handler(const struct bpf_insn *insn, * bits. We don't need to worry about buildtime or runtime sort * modifying the upper bits because the table is already sorted, and * isn't part of the main exception table. + * + * The fixup_offset is set to the next instruction from the instruction + * that may fault. The execution will jump to this after handling the + * fault. */ - offset = (long)&ex->fixup - (pc + insn_len * sizeof(u16)); - if (!FIELD_FIT(BPF_FIXUP_OFFSET_MASK, offset)) + fixup_offset = (long)&ex->fixup - (pc + insn_len * sizeof(u16)); + if (!FIELD_FIT(BPF_FIXUP_OFFSET_MASK, fixup_offset)) return -ERANGE; - ex->fixup = FIELD_PREP(BPF_FIXUP_OFFSET_MASK, offset) | + /* + * The offsets above have been calculated using the RO buffer but we + * need to use the R/W buffer for writes. + * switch ex to rw buffer for writing. + */ + ex = (void *)ctx->insns + ((void *)ex - (void *)ctx->ro_insns); + + ex->insn = ins_offset; + + ex->fixup = FIELD_PREP(BPF_FIXUP_OFFSET_MASK, fixup_offset) | FIELD_PREP(BPF_FIXUP_REG_MASK, dst_reg); ex->type = EX_TYPE_BPF; @@ -995,6 +1022,7 @@ int arch_prepare_bpf_trampoline(struct bpf_tramp_image *im, void *image, ctx.ninsns = 0; ctx.insns = NULL; + ctx.ro_insns = NULL; ret = __arch_prepare_bpf_trampoline(im, m, tlinks, func_addr, flags, &ctx); if (ret < 0) return ret; @@ -1003,7 +1031,15 @@ int arch_prepare_bpf_trampoline(struct bpf_tramp_image *im, void *image, return -EFBIG; ctx.ninsns = 0; + /* + * The bpf_int_jit_compile() uses a RW buffer (ctx.insns) to write the + * JITed instructions and later copies it to a RX region (ctx.ro_insns). + * It also uses ctx.ro_insns to calculate offsets for jumps etc. As the + * trampoline image uses the same memory area for writing and execution, + * both ctx.insns and ctx.ro_insns can be set to image. + */ ctx.insns = image; + ctx.ro_insns = image; ret = __arch_prepare_bpf_trampoline(im, m, tlinks, func_addr, flags, &ctx); if (ret < 0) return ret; diff --git a/arch/riscv/net/bpf_jit_core.c b/arch/riscv/net/bpf_jit_core.c index 7a26a3e1c73c..b0fa0084f493 100644 --- a/arch/riscv/net/bpf_jit_core.c +++ b/arch/riscv/net/bpf_jit_core.c @@ -8,6 +8,8 @@ #include #include +#include +#include #include "bpf_jit.h" /* Number of iterations to try until offsets converge. */ @@ -117,16 +119,27 @@ struct bpf_prog *bpf_int_jit_compile(struct bpf_prog *prog) sizeof(struct exception_table_entry); prog_size = sizeof(*ctx->insns) * ctx->ninsns; - jit_data->header = - bpf_jit_binary_alloc(prog_size + extable_size, - &jit_data->image, - sizeof(u32), - bpf_fill_ill_insns); - if (!jit_data->header) { + jit_data->ro_header = + bpf_jit_binary_pack_alloc(prog_size + + extable_size, + &jit_data->ro_image, + sizeof(u32), + &jit_data->header, + &jit_data->image, + bpf_fill_ill_insns); + if (!jit_data->ro_header) { prog = orig_prog; goto out_offset; } + /* + * Use the image(RW) for writing the JITed instructions. But also save + * the ro_image(RX) for calculating the offsets in the image. The RW + * image will be later copied to the RX image from where the program + * will run. The bpf_jit_binary_pack_finalize() will do this copy in the + * final step. + */ + ctx->ro_insns = (u16 *)jit_data->ro_image; ctx->insns = (u16 *)jit_data->image; /* * Now, when the image is allocated, the image can @@ -138,14 +151,12 @@ struct bpf_prog *bpf_int_jit_compile(struct bpf_prog *prog) if (i == NR_JIT_ITERATIONS) { pr_err("bpf-jit: image did not converge in <%d passes!\n", i); - if (jit_data->header) - bpf_jit_binary_free(jit_data->header); prog = orig_prog; - goto out_offset; + goto out_free_hdr; } if (extable_size) - prog->aux->extable = (void *)ctx->insns + prog_size; + prog->aux->extable = (void *)ctx->ro_insns + prog_size; skip_init_ctx: pass++; @@ -154,23 +165,35 @@ struct bpf_prog *bpf_int_jit_compile(struct bpf_prog *prog) bpf_jit_build_prologue(ctx); if (build_body(ctx, extra_pass, NULL)) { - bpf_jit_binary_free(jit_data->header); prog = orig_prog; - goto out_offset; + goto out_free_hdr; } bpf_jit_build_epilogue(ctx); if (bpf_jit_enable > 1) bpf_jit_dump(prog->len, prog_size, pass, ctx->insns); - prog->bpf_func = (void *)ctx->insns; + prog->bpf_func = (void *)ctx->ro_insns; prog->jited = 1; prog->jited_len = prog_size; - bpf_flush_icache(jit_data->header, ctx->insns + ctx->ninsns); - if (!prog->is_func || extra_pass) { - bpf_jit_binary_lock_ro(jit_data->header); + if (WARN_ON(bpf_jit_binary_pack_finalize(prog, + jit_data->ro_header, + jit_data->header))) { + /* ro_header has been freed */ + jit_data->ro_header = NULL; + prog = orig_prog; + goto out_offset; + } + /* + * The instructions have now been copied to the ROX region from + * where they will execute. + * Write any modified data cache blocks out to memory and + * invalidate the corresponding blocks in the instruction cache. + */ + bpf_flush_icache(jit_data->ro_header, + ctx->ro_insns + ctx->ninsns); for (i = 0; i < prog->len; i++) ctx->offset[i] = ninsns_rvoff(ctx->offset[i]); bpf_prog_fill_jited_linfo(prog, ctx->offset); @@ -185,6 +208,15 @@ struct bpf_prog *bpf_int_jit_compile(struct bpf_prog *prog) bpf_jit_prog_release_other(prog, prog == orig_prog ? tmp : orig_prog); return prog; + +out_free_hdr: + if (jit_data->header) { + bpf_arch_text_copy(&jit_data->ro_header->size, + &jit_data->header->size, + sizeof(jit_data->header->size)); + bpf_jit_binary_pack_free(jit_data->ro_header, jit_data->header); + } + goto out_offset; } u64 bpf_jit_alloc_exec_limit(void) @@ -204,3 +236,56 @@ void bpf_jit_free_exec(void *addr) { return vfree(addr); } + +void *bpf_arch_text_copy(void *dst, void *src, size_t len) +{ + int ret; + + mutex_lock(&text_mutex); + ret = patch_text_nosync(dst, src, len); + mutex_unlock(&text_mutex); + + if (ret) + return ERR_PTR(-EINVAL); + + return dst; +} + +int bpf_arch_text_invalidate(void *dst, size_t len) +{ + __le32 *ptr; + int ret = 0; + u32 inval = 0; + + for (ptr = dst; ret == 0 && len >= sizeof(u32); len -= sizeof(u32)) { + mutex_lock(&text_mutex); + ret = patch_text_nosync(ptr++, &inval, sizeof(u32)); + mutex_unlock(&text_mutex); + } + + return ret; +} + +void bpf_jit_free(struct bpf_prog *prog) +{ + if (prog->jited) { + struct rv_jit_data *jit_data = prog->aux->jit_data; + struct bpf_binary_header *hdr; + + /* + * If we fail the final pass of JIT (from jit_subprogs), + * the program may not be finalized yet. Call finalize here + * before freeing it. + */ + if (jit_data) { + bpf_jit_binary_pack_finalize(prog, jit_data->ro_header, + jit_data->header); + kfree(jit_data); + } + hdr = bpf_jit_binary_pack_hdr(prog); + bpf_jit_binary_pack_free(hdr, NULL); + WARN_ON_ONCE(!bpf_prog_kallsyms_verify_off(prog)); + } + + bpf_prog_unlock_free(prog); +}