From patchwork Thu Aug 31 13:12:26 2023 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 8bit X-Patchwork-Submitter: Puranjay Mohan X-Patchwork-Id: 13371558 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from bombadil.infradead.org (bombadil.infradead.org [198.137.202.133]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (No client certificate requested) by smtp.lore.kernel.org (Postfix) with ESMTPS id BEC3BC83F33 for ; Thu, 31 Aug 2023 13:12:48 +0000 (UTC) DKIM-Signature: v=1; a=rsa-sha256; q=dns/txt; c=relaxed/relaxed; d=lists.infradead.org; s=bombadil.20210309; h=Sender: Content-Transfer-Encoding:Content-Type:List-Subscribe:List-Help:List-Post: List-Archive:List-Unsubscribe:List-Id:MIME-Version:References:In-Reply-To: Message-Id:Date:Subject:Cc:To:From:Reply-To:Content-ID:Content-Description: Resent-Date:Resent-From:Resent-Sender:Resent-To:Resent-Cc:Resent-Message-ID: List-Owner; bh=QIRsOt8NfAZ9Ct/b816kOksusJleiuw9R9Ds7OLFAlI=; b=HoMRypeRALlAnc mt9i1udpP3vEsnQ732AUG6+Hn58ELnzpz7791uSfkNYjSPKEkz1O98qxAuhS9Xu7iijTlTEa6Dt5m OzQnXC4m7gRpKnD2pd4J+lR8Xo59xwuSWKYUZ5eWrDpI618Dyv3NsKP+5H12dczW3bLNyP8MRgHmz +yyioVX+4DCi78Zc12mAYXzfE7RLBqm6UYr7RuB5TIxkeRcQD107FvzIGwtfHALusI7o2aCjn4EGc uzHABKFniLlNsjLhSgzuHsjXvVh7WFT57ycdZgxStTucgP0FzTs/0UITWSJtkB4EbwCTLVJ+pysQ4 CoF30S6gBN3s8rZghftg==; Received: from localhost ([::1] helo=bombadil.infradead.org) by bombadil.infradead.org with esmtp (Exim 4.96 #2 (Red Hat Linux)) id 1qbhTY-00FLDW-0Q; Thu, 31 Aug 2023 13:12:44 +0000 Received: from mail-wr1-x433.google.com ([2a00:1450:4864:20::433]) by bombadil.infradead.org with esmtps (Exim 4.96 #2 (Red Hat Linux)) id 1qbhTQ-00FLA1-1q for linux-riscv@lists.infradead.org; Thu, 31 Aug 2023 13:12:37 +0000 Received: by mail-wr1-x433.google.com with SMTP id ffacd0b85a97d-31aeee69de0so604712f8f.2 for ; Thu, 31 Aug 2023 06:12:33 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=20221208; t=1693487552; x=1694092352; darn=lists.infradead.org; h=content-transfer-encoding:mime-version:references:in-reply-to :message-id:date:subject:cc:to:from:from:to:cc:subject:date :message-id:reply-to; bh=Vw9aCO6QF7ehSGg3eSofLWFhJ8EHSCsmR6IG4YjeGA0=; b=U697WiVWH2LdzgL3eVe5M+9/IimkGdZC/G2X8uY4SNFguL+NYYd6sNLsiuvdHmsYw8 8jCHbU3pK+UVKMBmlClNCcVC8S1+tLpAsCOnRk1J8Srts68P4toP8qvRaRCx/02sXMX8 M0dFoNPv1dtR1am+vTnbFTsQNOl+cR1E7i+4Bu5tWViUonYqOUDk2OPWftC7Hv3jtCvc VcPDquB7dd1ljNUThmb6qOhRWrpWaJ7l4PqemKB2JSAkCRueK7s4gXibN55G5XtrLGS8 tehH4XO4pz+vaxJrGAC/1JrojgXDgGYmlbaQsSojec0YetbOp00AEef/9TfPIIpryr3Z lYmw== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20221208; t=1693487552; x=1694092352; h=content-transfer-encoding:mime-version:references:in-reply-to :message-id:date:subject:cc:to:from:x-gm-message-state:from:to:cc :subject:date:message-id:reply-to; bh=Vw9aCO6QF7ehSGg3eSofLWFhJ8EHSCsmR6IG4YjeGA0=; b=Im4Sq4MfDOa/LMue9SOEN00NzP37rNGwWyq3NLjqnuLrNz+5DNUwf0BVu7j19N5h00 1zUERFSh8+tlujWQyec7hRxVDdVQZZZKJvexLyv6N+l7f2inVV7RrHFfwRNy2WqIGU0o Q0sff8KcyCIWSjfhhUyPuvUJVCZDST1GjCSAbS/y0mRs7PArpPn0+kCwQURsTyv1Nm9+ MbkRBsmRhCRIDFMjBnkgS9msgUfKYaOtPU7jwjvsDvigFMco2Z45btfeU2h3grLgR8nQ tJv1noIpIGPjbRsAFzzeTN8/lk8frBNHWRiVZ0/3zVdRV9lLyW9MM1BCi7fk/I3BGHG4 WW6Q== X-Gm-Message-State: AOJu0YwevEYNmnQGtg4oWK2XMr8/UHNNgFv3U05QpoF8sSt1eV1bIlfI qY56tU1Dnod82oZ1v6Aq+lEJJBjWjrfbu6nEtVk= X-Google-Smtp-Source: AGHT+IEV1nZTWVMPYeNOWwrZQScu0+JsXp36Y/oH9nCnr88bGLZYxfwLclYlSX63KCt8UHKIEmHBIA== X-Received: by 2002:adf:e54f:0:b0:319:82c9:8e7d with SMTP id z15-20020adfe54f000000b0031982c98e7dmr3698994wrm.31.1693487551537; Thu, 31 Aug 2023 06:12:31 -0700 (PDT) Received: from ip-172-31-30-46.eu-west-1.compute.internal (ec2-54-170-241-106.eu-west-1.compute.amazonaws.com. [54.170.241.106]) by smtp.gmail.com with ESMTPSA id a28-20020a5d457c000000b00317f70240afsm2206607wrc.27.2023.08.31.06.12.30 (version=TLS1_3 cipher=TLS_AES_256_GCM_SHA384 bits=256/256); Thu, 31 Aug 2023 06:12:31 -0700 (PDT) From: Puranjay Mohan To: paul.walmsley@sifive.com, palmer@dabbelt.com, aou@eecs.berkeley.edu, pulehui@huawei.com, conor.dooley@microchip.com, ast@kernel.org, daniel@iogearbox.net, andrii@kernel.org, martin.lau@linux.dev, song@kernel.org, yhs@fb.com, kpsingh@kernel.org, bjorn@kernel.org, bpf@vger.kernel.org, linux-riscv@lists.infradead.org, linux-kernel@vger.kernel.org Cc: puranjay12@gmail.com Subject: [PATCH bpf-next v4 1/4] bpf: make bpf_prog_pack allocator portable Date: Thu, 31 Aug 2023 13:12:26 +0000 Message-Id: <20230831131229.497941-2-puranjay12@gmail.com> X-Mailer: git-send-email 2.40.1 In-Reply-To: <20230831131229.497941-1-puranjay12@gmail.com> References: <20230831131229.497941-1-puranjay12@gmail.com> MIME-Version: 1.0 X-CRM114-Version: 20100106-BlameMichelson ( TRE 0.8.0 (BSD) ) MR-646709E3 X-CRM114-CacheID: sfid-20230831_061236_614231_C1D56E75 X-CRM114-Status: GOOD ( 13.39 ) X-BeenThere: linux-riscv@lists.infradead.org X-Mailman-Version: 2.1.34 Precedence: list List-Id: List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Sender: "linux-riscv" Errors-To: linux-riscv-bounces+linux-riscv=archiver.kernel.org@lists.infradead.org The bpf_prog_pack allocator currently uses module_alloc() and module_memfree() to allocate and free memory. This is not portable because different architectures use different methods for allocating memory for BPF programs. Like ARM64 and riscv use vmalloc()/vfree(). Use bpf_jit_alloc_exec() and bpf_jit_free_exec() for memory management in bpf_prog_pack allocator. Other architectures can override these with their implementation and will be able to use bpf_prog_pack directly. On architectures that don't override bpf_jit_alloc/free_exec() this is basically a NOP. Signed-off-by: Puranjay Mohan Acked-by: Song Liu Acked-by: Björn Töpel Tested-by: Björn Töpel --- kernel/bpf/core.c | 8 ++++---- 1 file changed, 4 insertions(+), 4 deletions(-) diff --git a/kernel/bpf/core.c b/kernel/bpf/core.c index 0f8f036d8bd1..4e3ce0542e31 100644 --- a/kernel/bpf/core.c +++ b/kernel/bpf/core.c @@ -870,7 +870,7 @@ static struct bpf_prog_pack *alloc_new_pack(bpf_jit_fill_hole_t bpf_fill_ill_ins GFP_KERNEL); if (!pack) return NULL; - pack->ptr = module_alloc(BPF_PROG_PACK_SIZE); + pack->ptr = bpf_jit_alloc_exec(BPF_PROG_PACK_SIZE); if (!pack->ptr) { kfree(pack); return NULL; @@ -894,7 +894,7 @@ void *bpf_prog_pack_alloc(u32 size, bpf_jit_fill_hole_t bpf_fill_ill_insns) mutex_lock(&pack_mutex); if (size > BPF_PROG_PACK_SIZE) { size = round_up(size, PAGE_SIZE); - ptr = module_alloc(size); + ptr = bpf_jit_alloc_exec(size); if (ptr) { bpf_fill_ill_insns(ptr, size); set_vm_flush_reset_perms(ptr); @@ -932,7 +932,7 @@ void bpf_prog_pack_free(struct bpf_binary_header *hdr) mutex_lock(&pack_mutex); if (hdr->size > BPF_PROG_PACK_SIZE) { - module_memfree(hdr); + bpf_jit_free_exec(hdr); goto out; } @@ -956,7 +956,7 @@ void bpf_prog_pack_free(struct bpf_binary_header *hdr) if (bitmap_find_next_zero_area(pack->bitmap, BPF_PROG_CHUNK_COUNT, 0, BPF_PROG_CHUNK_COUNT, 0) == 0) { list_del(&pack->list); - module_memfree(pack->ptr); + bpf_jit_free_exec(pack->ptr); kfree(pack); } out: From patchwork Thu Aug 31 13:12:27 2023 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 8bit X-Patchwork-Submitter: Puranjay Mohan X-Patchwork-Id: 13371557 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from bombadil.infradead.org (bombadil.infradead.org [198.137.202.133]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (No client certificate requested) by smtp.lore.kernel.org (Postfix) with ESMTPS id E054AC83F01 for ; Thu, 31 Aug 2023 13:12:48 +0000 (UTC) DKIM-Signature: v=1; a=rsa-sha256; q=dns/txt; c=relaxed/relaxed; d=lists.infradead.org; s=bombadil.20210309; h=Sender: Content-Transfer-Encoding:Content-Type:List-Subscribe:List-Help:List-Post: List-Archive:List-Unsubscribe:List-Id:MIME-Version:References:In-Reply-To: Message-Id:Date:Subject:Cc:To:From:Reply-To:Content-ID:Content-Description: Resent-Date:Resent-From:Resent-Sender:Resent-To:Resent-Cc:Resent-Message-ID: List-Owner; bh=hybb2qcqI/eM+/Yb6xUTcRP7FqPYaFIWSJmJE7cwFa8=; b=tNdIAuNpf5/a+A nqA3Gu0nSBI7gXU5XoXTVriBlpbUHu0x9G/Ce7PdChUh8idx+P+22Sa10dBSJk9xJWFN8nnHwF4H8 /9HLvF5OYzG5sSFrGUpRhnQA3Nw9+C7WeQAKj05fEUF6EWVRiZy6THvalVS1cD1Wiu311z248QFkm KL58DZknYEWF157mIgvOvq38TDTGfH90h9C9jGjrDunF4t64kuJnF9Ew7eLxFBjT+ly2f64t7yOYr qpIG7sFD5/FH8psWp3DYnpUMvJX3gOkvrmoCYg17vbwtq6EI3e1owWBDYKjxFT77QVIQzqZ3CiCob kz6+byPQdxanuOj5xb0g==; Received: from localhost ([::1] helo=bombadil.infradead.org) by bombadil.infradead.org with esmtp (Exim 4.96 #2 (Red Hat Linux)) id 1qbhTX-00FLDG-1p; Thu, 31 Aug 2023 13:12:43 +0000 Received: from mail-wr1-x42d.google.com ([2a00:1450:4864:20::42d]) by bombadil.infradead.org with esmtps (Exim 4.96 #2 (Red Hat Linux)) id 1qbhTQ-00FLA3-0e for linux-riscv@lists.infradead.org; Thu, 31 Aug 2023 13:12:37 +0000 Received: by mail-wr1-x42d.google.com with SMTP id ffacd0b85a97d-31781e15a0cso601968f8f.3 for ; Thu, 31 Aug 2023 06:12:34 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=20221208; t=1693487553; x=1694092353; darn=lists.infradead.org; h=content-transfer-encoding:mime-version:references:in-reply-to :message-id:date:subject:cc:to:from:from:to:cc:subject:date :message-id:reply-to; bh=CUYKChQYfJZZ8DPO0whW/zno6UWIarXDmyfOm92F8UA=; b=gT+vFZ42RdJ6gnLLr8WT+mDJz64+f7Up7VWNrhNgfizBKAMOHBUtg2/0KEs3Nz3tKr VgTNnOdQA22sPTdGqOxkhC9aJVzgeqoZCdtH/wRJdJhHdj8RkNMWfR4iO407l5pFFfu1 /QrpMPkrX+pVqLvBLwskotFCNjAiJh36J6xQrf24vrcN4qceL68UQMA31+uzvlLxAAbl Dsq/zR0MSu3gMPvAKqbbAlG4YPAJdvvqFQkzEounuaAcK2N7/pEl8v5QUpfgSph3Y1tD LBZs7jiFAomnGmP/w3zJYnokB2OUChAY4/rIiquIgjjcDCFPs6l3jOGBXrgtSFln3k5h s0qA== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20221208; t=1693487553; x=1694092353; h=content-transfer-encoding:mime-version:references:in-reply-to :message-id:date:subject:cc:to:from:x-gm-message-state:from:to:cc :subject:date:message-id:reply-to; bh=CUYKChQYfJZZ8DPO0whW/zno6UWIarXDmyfOm92F8UA=; b=WJmHugycDzuVXxkVDP8yJ2G4DS6zEgIj5TsqyCXoLnw0+P5K21uduOzIgomj5NBnRd yUBOGzc8BhlXUbeT+KFiMnvsLTfvBnZKAeWEyCDn5Yss4pDWpDw63MbSfB0Gep2HjVMa mXYhoHcdgtothLADekMAEMYhJzJkIkExs3EmtaXH/Ssm6KSUFVsiG7qyNyY/XGwrut2b V+cmh44W2qQNWfccZDJrmHwiT+fiMPztY6c5NTXV3lg9FwUGw57FNZwiseD2TjBWHBA9 xXdDF0fd5+WJwnI5VLDctvE0TqOP3MipVxsvZLM9r50DRoRDDSrG85RUczKTRqpUDm9f VJoQ== X-Gm-Message-State: AOJu0YzXVuL6+NsZGgHssRt/xqbQohQe4EHHEFBYSCx9jZe8qS7Ysx6h Dd7UYZyU/CN5ZeJahaX9PT0= X-Google-Smtp-Source: AGHT+IEmRLcQA4Zh4qOEOVWO+OJdUOgzn5p2uNiV+/qiisIBP67acZzJXuCuSC/kJMqcbfCWT7qEVg== X-Received: by 2002:a5d:668f:0:b0:317:6a7c:6e07 with SMTP id l15-20020a5d668f000000b003176a7c6e07mr3756142wru.32.1693487553087; Thu, 31 Aug 2023 06:12:33 -0700 (PDT) Received: from ip-172-31-30-46.eu-west-1.compute.internal (ec2-54-170-241-106.eu-west-1.compute.amazonaws.com. [54.170.241.106]) by smtp.gmail.com with ESMTPSA id a28-20020a5d457c000000b00317f70240afsm2206607wrc.27.2023.08.31.06.12.32 (version=TLS1_3 cipher=TLS_AES_256_GCM_SHA384 bits=256/256); Thu, 31 Aug 2023 06:12:32 -0700 (PDT) From: Puranjay Mohan To: paul.walmsley@sifive.com, palmer@dabbelt.com, aou@eecs.berkeley.edu, pulehui@huawei.com, conor.dooley@microchip.com, ast@kernel.org, daniel@iogearbox.net, andrii@kernel.org, martin.lau@linux.dev, song@kernel.org, yhs@fb.com, kpsingh@kernel.org, bjorn@kernel.org, bpf@vger.kernel.org, linux-riscv@lists.infradead.org, linux-kernel@vger.kernel.org Cc: puranjay12@gmail.com Subject: [PATCH bpf-next v4 2/4] riscv: extend patch_text_nosync() for multiple pages Date: Thu, 31 Aug 2023 13:12:27 +0000 Message-Id: <20230831131229.497941-3-puranjay12@gmail.com> X-Mailer: git-send-email 2.40.1 In-Reply-To: <20230831131229.497941-1-puranjay12@gmail.com> References: <20230831131229.497941-1-puranjay12@gmail.com> MIME-Version: 1.0 X-CRM114-Version: 20100106-BlameMichelson ( TRE 0.8.0 (BSD) ) MR-646709E3 X-CRM114-CacheID: sfid-20230831_061236_239858_748B6AB3 X-CRM114-Status: GOOD ( 15.74 ) X-BeenThere: linux-riscv@lists.infradead.org X-Mailman-Version: 2.1.34 Precedence: list List-Id: List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Sender: "linux-riscv" Errors-To: linux-riscv-bounces+linux-riscv=archiver.kernel.org@lists.infradead.org The patch_insn_write() function currently doesn't work for multiple pages of instructions, therefore patch_text_nosync() will fail with a page fault if called with lengths spanning multiple pages. This commit extends the patch_insn_write() function to support multiple pages by copying at max 2 pages at a time in a loop. This implementation is similar to text_poke_copy() function of x86. Signed-off-by: Puranjay Mohan Reviewed-by: Pu Lehui Reviewed-by: Björn Töpel Tested-by: Björn Töpel --- arch/riscv/kernel/patch.c | 37 ++++++++++++++++++++++++++++++++----- 1 file changed, 32 insertions(+), 5 deletions(-) diff --git a/arch/riscv/kernel/patch.c b/arch/riscv/kernel/patch.c index 575e71d6c8ae..2c97e246f4dc 100644 --- a/arch/riscv/kernel/patch.c +++ b/arch/riscv/kernel/patch.c @@ -53,12 +53,18 @@ static void patch_unmap(int fixmap) } NOKPROBE_SYMBOL(patch_unmap); -static int patch_insn_write(void *addr, const void *insn, size_t len) +static int __patch_insn_write(void *addr, const void *insn, size_t len) { void *waddr = addr; bool across_pages = (((uintptr_t) addr & ~PAGE_MASK) + len) > PAGE_SIZE; int ret; + /* + * Only two pages can be mapped at a time for writing. + */ + if (len + offset_in_page(addr) > 2 * PAGE_SIZE) + return -EINVAL; + /* * Before reaching here, it was expected to lock the text_mutex * already, so we don't need to give another lock here and could @@ -74,7 +80,7 @@ static int patch_insn_write(void *addr, const void *insn, size_t len) lockdep_assert_held(&text_mutex); if (across_pages) - patch_map(addr + len, FIX_TEXT_POKE1); + patch_map(addr + PAGE_SIZE, FIX_TEXT_POKE1); waddr = patch_map(addr, FIX_TEXT_POKE0); @@ -87,15 +93,36 @@ static int patch_insn_write(void *addr, const void *insn, size_t len) return ret; } -NOKPROBE_SYMBOL(patch_insn_write); +NOKPROBE_SYMBOL(__patch_insn_write); #else -static int patch_insn_write(void *addr, const void *insn, size_t len) +static int __patch_insn_write(void *addr, const void *insn, size_t len) { return copy_to_kernel_nofault(addr, insn, len); } -NOKPROBE_SYMBOL(patch_insn_write); +NOKPROBE_SYMBOL(__patch_insn_write); #endif /* CONFIG_MMU */ +static int patch_insn_write(void *addr, const void *insn, size_t len) +{ + size_t patched = 0; + size_t size; + int ret = 0; + + /* + * Copy the instructions to the destination address, two pages at a time + * because __patch_insn_write() can only handle len <= 2 * PAGE_SIZE. + */ + while (patched < len && !ret) { + size = min_t(size_t, PAGE_SIZE * 2 - offset_in_page(addr + patched), len - patched); + ret = __patch_insn_write(addr + patched, insn + patched, size); + + patched += size; + } + + return ret; +} +NOKPROBE_SYMBOL(patch_insn_write); + int patch_text_nosync(void *addr, const void *insns, size_t len) { u32 *tp = addr; From patchwork Thu Aug 31 13:12:28 2023 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 8bit X-Patchwork-Submitter: Puranjay Mohan X-Patchwork-Id: 13371556 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from bombadil.infradead.org (bombadil.infradead.org [198.137.202.133]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (No client certificate requested) by smtp.lore.kernel.org (Postfix) with ESMTPS id 1E6AEC83F12 for ; Thu, 31 Aug 2023 13:12:47 +0000 (UTC) DKIM-Signature: v=1; a=rsa-sha256; q=dns/txt; c=relaxed/relaxed; d=lists.infradead.org; s=bombadil.20210309; h=Sender: Content-Transfer-Encoding:Content-Type:List-Subscribe:List-Help:List-Post: List-Archive:List-Unsubscribe:List-Id:MIME-Version:References:In-Reply-To: Message-Id:Date:Subject:Cc:To:From:Reply-To:Content-ID:Content-Description: Resent-Date:Resent-From:Resent-Sender:Resent-To:Resent-Cc:Resent-Message-ID: List-Owner; bh=sev0ABGf7SBJO+qKtrpWu4el2TASeh6YsomMqxVYawU=; b=1IkAQ5bNYwDhgz i57p89e6ZU+fqpdrOeNu8yJ/QNxjkpCD0c5QWlYc7ndg5IsuP2vWyfqmg4MP0TZt1wuU0vbQYUIu3 RTfTEC7J01ZbyPBZ6yKD550Tx3XkTQNEZX3DS7+zC6/ItqRsWdkZB5bVHT2JnQ3QsZrJRxhOvOlIv c49v4vcwty3M+o7iP5kjAL0mej1SkS/OrjWP1hK8U5gP/SHLXcHSLWVVbYARFE8dgIfGRk1XVxws+ RG1kPZU9JZyUZYAJ9S0BO21ZY8ZvKO3syuc2lLFMTNcG6hMpzraFICnaE3PS6HavvwUxHEgZOSHaK xqfkFwMxYOhiwZnjSOaQ==; Received: from localhost ([::1] helo=bombadil.infradead.org) by bombadil.infradead.org with esmtp (Exim 4.96 #2 (Red Hat Linux)) id 1qbhTS-00FLC1-2o; Thu, 31 Aug 2023 13:12:38 +0000 Received: from mail-wm1-x330.google.com ([2a00:1450:4864:20::330]) by bombadil.infradead.org with esmtps (Exim 4.96 #2 (Red Hat Linux)) id 1qbhTP-00FLA6-0w for linux-riscv@lists.infradead.org; Thu, 31 Aug 2023 13:12:36 +0000 Received: by mail-wm1-x330.google.com with SMTP id 5b1f17b1804b1-401d10e3e54so7372965e9.2 for ; Thu, 31 Aug 2023 06:12:35 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=20221208; t=1693487554; x=1694092354; darn=lists.infradead.org; h=content-transfer-encoding:mime-version:references:in-reply-to :message-id:date:subject:cc:to:from:from:to:cc:subject:date :message-id:reply-to; bh=egv6cBK34EgIbjBrWp9fTvnprfq8zkhKE0zbSvUdTM8=; b=LX9MUNcpgvh80Tg86kv0xD+W8VCxUuxfghY5Oe7kQS7a5TjeXoFULqiX1JixJSrsGd l22zbVXjp+vGTqMZasZ4SI/L4hVdDX9ZPKH31aiqdljcgcYg2uIlHB6J8Wm2RxZnZag5 s9W+er4cUGTAny4ITq+IvJIKWIFoP6F6h6e5L0x4Sl20sP0paq5S0y9hC49r5rFNnJJP nRHVmb60Erh8o7kCNH9KiNZzor1JtBmyI4g9yDyu8pjb+0ii9eduJCK5aQ/jUhrEWLPy 72FCjv6ylEw8k8rkXDLZAS1tIcqAd6nHoLOog7rLxFaEpYELA9K1EmovEfjllrlkC5/j b0yQ== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20221208; t=1693487554; x=1694092354; h=content-transfer-encoding:mime-version:references:in-reply-to :message-id:date:subject:cc:to:from:x-gm-message-state:from:to:cc :subject:date:message-id:reply-to; bh=egv6cBK34EgIbjBrWp9fTvnprfq8zkhKE0zbSvUdTM8=; b=ZpMTEA1jsekFomou92u1M48i4Qy2gXNbsuPXI/uUBW9xH2wLuh67iVCVpjuWLmSKe9 IwcxlndLtcOGWZEg6ynIJQnQ1ecs4+/bCdZkhjxVBYbALXgfuGWXqHqEYgkYR9hlpoB8 MmXzsegQRrt3KvqJF6TJcMndLrqWzkoEkhLdZTvmvWDSKeJe5f7ddNZG4VHlJnetmeMS Axdu3JpOSyxTA4Xj2L1paArHXj2h9covh4MTyTniDKadn4oJ1aPevDHDLze/xj2kwQJk CTe6xaikgqY+/J3P/VRwqHqC9USIVVhElyj1NiYR2xlgH3gQN2Wfo2ou0UjeiVZW/92J Uuog== X-Gm-Message-State: AOJu0YxpKFdboh6L1CaifzHTf//3sC/9J9eraZsea5bV3bP6fgnpBFqG LkTY6eVcr/k2ou2EUmMfVc0= X-Google-Smtp-Source: AGHT+IFNloiz3CzOTUOmKRegrJpuf48gN1sNQVaqZVlmbqhoFqXkrky0uDTwEvywMEW/CZlXcMQGhQ== X-Received: by 2002:a5d:5689:0:b0:31a:dbd1:bf6 with SMTP id f9-20020a5d5689000000b0031adbd10bf6mr4195294wrv.68.1693487553990; Thu, 31 Aug 2023 06:12:33 -0700 (PDT) Received: from ip-172-31-30-46.eu-west-1.compute.internal (ec2-54-170-241-106.eu-west-1.compute.amazonaws.com. [54.170.241.106]) by smtp.gmail.com with ESMTPSA id a28-20020a5d457c000000b00317f70240afsm2206607wrc.27.2023.08.31.06.12.33 (version=TLS1_3 cipher=TLS_AES_256_GCM_SHA384 bits=256/256); Thu, 31 Aug 2023 06:12:33 -0700 (PDT) From: Puranjay Mohan To: paul.walmsley@sifive.com, palmer@dabbelt.com, aou@eecs.berkeley.edu, pulehui@huawei.com, conor.dooley@microchip.com, ast@kernel.org, daniel@iogearbox.net, andrii@kernel.org, martin.lau@linux.dev, song@kernel.org, yhs@fb.com, kpsingh@kernel.org, bjorn@kernel.org, bpf@vger.kernel.org, linux-riscv@lists.infradead.org, linux-kernel@vger.kernel.org Cc: puranjay12@gmail.com Subject: [PATCH bpf-next v4 3/4] riscv: implement a memset like function for text Date: Thu, 31 Aug 2023 13:12:28 +0000 Message-Id: <20230831131229.497941-4-puranjay12@gmail.com> X-Mailer: git-send-email 2.40.1 In-Reply-To: <20230831131229.497941-1-puranjay12@gmail.com> References: <20230831131229.497941-1-puranjay12@gmail.com> MIME-Version: 1.0 X-CRM114-Version: 20100106-BlameMichelson ( TRE 0.8.0 (BSD) ) MR-646709E3 X-CRM114-CacheID: sfid-20230831_061235_329730_8BB76462 X-CRM114-Status: GOOD ( 15.79 ) X-BeenThere: linux-riscv@lists.infradead.org X-Mailman-Version: 2.1.34 Precedence: list List-Id: List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Sender: "linux-riscv" Errors-To: linux-riscv-bounces+linux-riscv=archiver.kernel.org@lists.infradead.org The BPF JIT needs to write invalid instructions to RX regions of memory to invalidate removed BPF programs. This needs a function like memset() that can work with RX memory. Implement patch_text_set_nosync() which is similar to text_poke_set() of x86. Signed-off-by: Puranjay Mohan Reviewed-by: Pu Lehui Acked-by: Björn Töpel Tested-by: Björn Töpel --- arch/riscv/include/asm/patch.h | 1 + arch/riscv/kernel/patch.c | 77 ++++++++++++++++++++++++++++++++++ 2 files changed, 78 insertions(+) diff --git a/arch/riscv/include/asm/patch.h b/arch/riscv/include/asm/patch.h index 63c98833d510..e88b52d39eac 100644 --- a/arch/riscv/include/asm/patch.h +++ b/arch/riscv/include/asm/patch.h @@ -7,6 +7,7 @@ #define _ASM_RISCV_PATCH_H int patch_text_nosync(void *addr, const void *insns, size_t len); +int patch_text_set_nosync(void *addr, u8 c, size_t len); int patch_text(void *addr, u32 *insns, int ninsns); extern int riscv_patch_in_stop_machine; diff --git a/arch/riscv/kernel/patch.c b/arch/riscv/kernel/patch.c index 2c97e246f4dc..13ee7bf589a1 100644 --- a/arch/riscv/kernel/patch.c +++ b/arch/riscv/kernel/patch.c @@ -6,6 +6,7 @@ #include #include #include +#include #include #include #include @@ -53,6 +54,39 @@ static void patch_unmap(int fixmap) } NOKPROBE_SYMBOL(patch_unmap); +static int __patch_insn_set(void *addr, u8 c, size_t len) +{ + void *waddr = addr; + bool across_pages = (((uintptr_t)addr & ~PAGE_MASK) + len) > PAGE_SIZE; + + /* + * Only two pages can be mapped at a time for writing. + */ + if (len + offset_in_page(addr) > 2 * PAGE_SIZE) + return -EINVAL; + /* + * Before reaching here, it was expected to lock the text_mutex + * already, so we don't need to give another lock here and could + * ensure that it was safe between each cores. + */ + lockdep_assert_held(&text_mutex); + + if (across_pages) + patch_map(addr + PAGE_SIZE, FIX_TEXT_POKE1); + + waddr = patch_map(addr, FIX_TEXT_POKE0); + + memset(waddr, c, len); + + patch_unmap(FIX_TEXT_POKE0); + + if (across_pages) + patch_unmap(FIX_TEXT_POKE1); + + return 0; +} +NOKPROBE_SYMBOL(__patch_insn_set); + static int __patch_insn_write(void *addr, const void *insn, size_t len) { void *waddr = addr; @@ -95,6 +129,14 @@ static int __patch_insn_write(void *addr, const void *insn, size_t len) } NOKPROBE_SYMBOL(__patch_insn_write); #else +static int __patch_insn_set(void *addr, u8 c, size_t len) +{ + memset(addr, c, len); + + return 0; +} +NOKPROBE_SYMBOL(__patch_insn_set); + static int __patch_insn_write(void *addr, const void *insn, size_t len) { return copy_to_kernel_nofault(addr, insn, len); @@ -102,6 +144,41 @@ static int __patch_insn_write(void *addr, const void *insn, size_t len) NOKPROBE_SYMBOL(__patch_insn_write); #endif /* CONFIG_MMU */ +static int patch_insn_set(void *addr, u8 c, size_t len) +{ + size_t patched = 0; + size_t size; + int ret = 0; + + /* + * __patch_insn_set() can only work on 2 pages at a time so call it in a + * loop with len <= 2 * PAGE_SIZE. + */ + while (patched < len && !ret) { + size = min_t(size_t, PAGE_SIZE * 2 - offset_in_page(addr + patched), len - patched); + ret = __patch_insn_set(addr + patched, c, size); + + patched += size; + } + + return ret; +} +NOKPROBE_SYMBOL(patch_insn_set); + +int patch_text_set_nosync(void *addr, u8 c, size_t len) +{ + u32 *tp = addr; + int ret; + + ret = patch_insn_set(tp, c, len); + + if (!ret) + flush_icache_range((uintptr_t)tp, (uintptr_t)tp + len); + + return ret; +} +NOKPROBE_SYMBOL(patch_text_set_nosync); + static int patch_insn_write(void *addr, const void *insn, size_t len) { size_t patched = 0; From patchwork Thu Aug 31 13:12:29 2023 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 8bit X-Patchwork-Submitter: Puranjay Mohan X-Patchwork-Id: 13371560 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from bombadil.infradead.org (bombadil.infradead.org [198.137.202.133]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (No client certificate requested) by smtp.lore.kernel.org (Postfix) with ESMTPS id 458D1C83F32 for ; Thu, 31 Aug 2023 13:12:57 +0000 (UTC) DKIM-Signature: v=1; a=rsa-sha256; q=dns/txt; c=relaxed/relaxed; d=lists.infradead.org; s=bombadil.20210309; h=Sender: Content-Transfer-Encoding:Content-Type:List-Subscribe:List-Help:List-Post: List-Archive:List-Unsubscribe:List-Id:MIME-Version:References:In-Reply-To: Message-Id:Date:Subject:Cc:To:From:Reply-To:Content-ID:Content-Description: Resent-Date:Resent-From:Resent-Sender:Resent-To:Resent-Cc:Resent-Message-ID: List-Owner; bh=heq+p1bfqP+W/gfackNSkpXxhUXQYUEZ3uk8KIb5Hpc=; b=qgFbz3bcYPtuuy OFNUnXQy9Wb59LIxbur5EqadOnp0NEyRcwfcn7EY+xabHVstx9atbPkDWNK2p/0vNYbcz/iQMu+yg JeHEuzpudMFpoeBkOmVkh6sPAaXEYi/fq/yuOaAPFc4cJuheYPoESEJ4WUrnEclsGbfpIu418M0Ls gfO6mjP+2y/CWzE6ZV0Zl+R+XSpR2gg+9gi0LX23WoqyKEGhyfABj6zyE+Fu2njGtxP1reMv+dVJv yJfUW6xCF8/2zvQWRZnfIKxmjqYf4xbuwbcQHBBDBrfF56thhepxOB4w4qIGAgwd7ctMId+JKaBaO /J+slFlVqfM9SD3DymsA==; Received: from localhost ([::1] helo=bombadil.infradead.org) by bombadil.infradead.org with esmtp (Exim 4.96 #2 (Red Hat Linux)) id 1qbhTg-00FLGV-3A; Thu, 31 Aug 2023 13:12:52 +0000 Received: from mail-wr1-x431.google.com ([2a00:1450:4864:20::431]) by bombadil.infradead.org with esmtps (Exim 4.96 #2 (Red Hat Linux)) id 1qbhTe-00FLAa-0h for linux-riscv@lists.infradead.org; Thu, 31 Aug 2023 13:12:52 +0000 Received: by mail-wr1-x431.google.com with SMTP id ffacd0b85a97d-31ad9155414so580877f8f.3 for ; Thu, 31 Aug 2023 06:12:36 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=20221208; t=1693487555; x=1694092355; darn=lists.infradead.org; h=content-transfer-encoding:mime-version:references:in-reply-to :message-id:date:subject:cc:to:from:from:to:cc:subject:date :message-id:reply-to; bh=gIMU3R/784Kep9+E6+IYdpBEBlTVFvtgxjE0UdQB+9Y=; b=GVHi8WKEr4tCRARCuNzAQbauWP/hOHeUC/6g2Ife9WNG/enTXqDhqCpQJ0tOOoJKFy nO4j6v9l+Kb9EsG+zWnnzGuqlfOZMPdkKG9igTI33FIIfHcYwF5FalqSQk7NliSsMqtN HfD6nC+u6KvwlrU/Bu1UV2YO3NNw48VRfqWdZ0xtgfTpCqK01vQzjUOd2+ghTRfVXxNH OlOmOlVmALbFG+PhaJhq+bnKv57Bys3eQ1v7JPOTofO3Hk5Ff56hWLGsjOYi3XEwEMaV 7KSDSLAglNsorqTXg09pOY7+Px+Aia2K4CKQ6jSWcqtCLilxh5q5+S9Y+3svMce0ArAN +z9g== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20221208; t=1693487555; x=1694092355; h=content-transfer-encoding:mime-version:references:in-reply-to :message-id:date:subject:cc:to:from:x-gm-message-state:from:to:cc :subject:date:message-id:reply-to; bh=gIMU3R/784Kep9+E6+IYdpBEBlTVFvtgxjE0UdQB+9Y=; b=eApiRMbXcTyuKtNgtZP67Tn9QX1+4SLoUQNjwCh7uYdgn6BXM51amN9R1Vc/P5/tqM XanSeBlXU3yJGsrIgv9HuRjxtGQ5KycQtua4vw4MB1zlWkLXjeT9LnL9jPKcHWqtwJae UrsaMCMei44SBk/4WArBscWrz9Z3K4uTMgk0I/i81DLBAmL6fa1eVHWB8bt5yo1TGIVn Ym9LFznLuqXA57hGgnrZ4S5vvAX9HTkRKY7qJuiWa2zCgAUpwFH4kbzfgQQVsMZqhq7d PGjF/XuA7KcFwvPhR3JgIFpy92vAEUz2WFH3ETvj3KKn2QfVklRTtylpK8uU3s0fDsNG rh4w== X-Gm-Message-State: AOJu0YzHr/HC13Kr5RMQmPix96HcUHcgohBkN48cCoDRffqH0MCZpDJ/ tAAid1fgygcAaOq+MuUR1BA= X-Google-Smtp-Source: AGHT+IEX01CcB90H6M//tMAksTdrRU3oXZS6EIgEO41p+gPRhcFcWd8/J9G4u5vWze8/EQmsFvgJcw== X-Received: by 2002:a5d:630e:0:b0:318:7bd:349e with SMTP id i14-20020a5d630e000000b0031807bd349emr4075319wru.29.1693487554859; Thu, 31 Aug 2023 06:12:34 -0700 (PDT) Received: from ip-172-31-30-46.eu-west-1.compute.internal (ec2-54-170-241-106.eu-west-1.compute.amazonaws.com. [54.170.241.106]) by smtp.gmail.com with ESMTPSA id a28-20020a5d457c000000b00317f70240afsm2206607wrc.27.2023.08.31.06.12.34 (version=TLS1_3 cipher=TLS_AES_256_GCM_SHA384 bits=256/256); Thu, 31 Aug 2023 06:12:34 -0700 (PDT) From: Puranjay Mohan To: paul.walmsley@sifive.com, palmer@dabbelt.com, aou@eecs.berkeley.edu, pulehui@huawei.com, conor.dooley@microchip.com, ast@kernel.org, daniel@iogearbox.net, andrii@kernel.org, martin.lau@linux.dev, song@kernel.org, yhs@fb.com, kpsingh@kernel.org, bjorn@kernel.org, bpf@vger.kernel.org, linux-riscv@lists.infradead.org, linux-kernel@vger.kernel.org Cc: puranjay12@gmail.com Subject: [PATCH bpf-next v4 4/4] bpf, riscv: use prog pack allocator in the BPF JIT Date: Thu, 31 Aug 2023 13:12:29 +0000 Message-Id: <20230831131229.497941-5-puranjay12@gmail.com> X-Mailer: git-send-email 2.40.1 In-Reply-To: <20230831131229.497941-1-puranjay12@gmail.com> References: <20230831131229.497941-1-puranjay12@gmail.com> MIME-Version: 1.0 X-CRM114-Version: 20100106-BlameMichelson ( TRE 0.8.0 (BSD) ) MR-646709E3 X-CRM114-CacheID: sfid-20230831_061250_256385_7E35693D X-CRM114-Status: GOOD ( 29.90 ) X-BeenThere: linux-riscv@lists.infradead.org X-Mailman-Version: 2.1.34 Precedence: list List-Id: List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Sender: "linux-riscv" Errors-To: linux-riscv-bounces+linux-riscv=archiver.kernel.org@lists.infradead.org Use bpf_jit_binary_pack_alloc() for memory management of JIT binaries in RISCV BPF JIT. The bpf_jit_binary_pack_alloc creates a pair of RW and RX buffers. The JIT writes the program into the RW buffer. When the JIT is done, the program is copied to the final RX buffer with bpf_jit_binary_pack_finalize. Implement bpf_arch_text_copy() and bpf_arch_text_invalidate() for RISCV JIT as these functions are required by bpf_jit_binary_pack allocator. Signed-off-by: Puranjay Mohan Reviewed-by: Song Liu Reviewed-by: Pu Lehui Acked-by: Björn Töpel Tested-by: Björn Töpel --- arch/riscv/net/bpf_jit.h | 3 + arch/riscv/net/bpf_jit_comp64.c | 60 ++++++++++++++---- arch/riscv/net/bpf_jit_core.c | 106 +++++++++++++++++++++++++++----- 3 files changed, 141 insertions(+), 28 deletions(-) diff --git a/arch/riscv/net/bpf_jit.h b/arch/riscv/net/bpf_jit.h index d21c6c92a683..a5ce1ab76ece 100644 --- a/arch/riscv/net/bpf_jit.h +++ b/arch/riscv/net/bpf_jit.h @@ -68,6 +68,7 @@ static inline bool is_creg(u8 reg) struct rv_jit_context { struct bpf_prog *prog; u16 *insns; /* RV insns */ + u16 *ro_insns; int ninsns; int prologue_len; int epilogue_offset; @@ -85,7 +86,9 @@ static inline int ninsns_rvoff(int ninsns) struct rv_jit_data { struct bpf_binary_header *header; + struct bpf_binary_header *ro_header; u8 *image; + u8 *ro_image; struct rv_jit_context ctx; }; diff --git a/arch/riscv/net/bpf_jit_comp64.c b/arch/riscv/net/bpf_jit_comp64.c index 8423f4ddf8f5..ecd3ae6f4116 100644 --- a/arch/riscv/net/bpf_jit_comp64.c +++ b/arch/riscv/net/bpf_jit_comp64.c @@ -144,7 +144,11 @@ static bool in_auipc_jalr_range(s64 val) /* Emit fixed-length instructions for address */ static int emit_addr(u8 rd, u64 addr, bool extra_pass, struct rv_jit_context *ctx) { - u64 ip = (u64)(ctx->insns + ctx->ninsns); + /* + * Use the ro_insns(RX) to calculate the offset as the BPF program will + * finally run from this memory region. + */ + u64 ip = (u64)(ctx->ro_insns + ctx->ninsns); s64 off = addr - ip; s64 upper = (off + (1 << 11)) >> 12; s64 lower = off & 0xfff; @@ -464,8 +468,12 @@ static int emit_call(u64 addr, bool fixed_addr, struct rv_jit_context *ctx) s64 off = 0; u64 ip; - if (addr && ctx->insns) { - ip = (u64)(long)(ctx->insns + ctx->ninsns); + if (addr && ctx->insns && ctx->ro_insns) { + /* + * Use the ro_insns(RX) to calculate the offset as the BPF + * program will finally run from this memory region. + */ + ip = (u64)(long)(ctx->ro_insns + ctx->ninsns); off = addr - ip; } @@ -578,9 +586,10 @@ static int add_exception_handler(const struct bpf_insn *insn, { struct exception_table_entry *ex; unsigned long pc; - off_t offset; + off_t ins_offset; + off_t fixup_offset; - if (!ctx->insns || !ctx->prog->aux->extable || + if (!ctx->insns || !ctx->ro_insns || !ctx->prog->aux->extable || (BPF_MODE(insn->code) != BPF_PROBE_MEM && BPF_MODE(insn->code) != BPF_PROBE_MEMSX)) return 0; @@ -594,12 +603,17 @@ static int add_exception_handler(const struct bpf_insn *insn, return -EINVAL; ex = &ctx->prog->aux->extable[ctx->nexentries]; - pc = (unsigned long)&ctx->insns[ctx->ninsns - insn_len]; + pc = (unsigned long)&ctx->ro_insns[ctx->ninsns - insn_len]; - offset = pc - (long)&ex->insn; - if (WARN_ON_ONCE(offset >= 0 || offset < INT_MIN)) + /* + * This is the relative offset of the instruction that may fault from + * the exception table itself. This will be written to the exception + * table and if this instruction faults, the destination register will + * be set to '0' and the execution will jump to the next instruction. + */ + ins_offset = pc - (long)&ex->insn; + if (WARN_ON_ONCE(ins_offset >= 0 || ins_offset < INT_MIN)) return -ERANGE; - ex->insn = offset; /* * Since the extable follows the program, the fixup offset is always @@ -608,12 +622,25 @@ static int add_exception_handler(const struct bpf_insn *insn, * bits. We don't need to worry about buildtime or runtime sort * modifying the upper bits because the table is already sorted, and * isn't part of the main exception table. + * + * The fixup_offset is set to the next instruction from the instruction + * that may fault. The execution will jump to this after handling the + * fault. */ - offset = (long)&ex->fixup - (pc + insn_len * sizeof(u16)); - if (!FIELD_FIT(BPF_FIXUP_OFFSET_MASK, offset)) + fixup_offset = (long)&ex->fixup - (pc + insn_len * sizeof(u16)); + if (!FIELD_FIT(BPF_FIXUP_OFFSET_MASK, fixup_offset)) return -ERANGE; - ex->fixup = FIELD_PREP(BPF_FIXUP_OFFSET_MASK, offset) | + /* + * The offsets above have been calculated using the RO buffer but we + * need to use the R/W buffer for writes. + * switch ex to rw buffer for writing. + */ + ex = (void *)ctx->insns + ((void *)ex - (void *)ctx->ro_insns); + + ex->insn = ins_offset; + + ex->fixup = FIELD_PREP(BPF_FIXUP_OFFSET_MASK, fixup_offset) | FIELD_PREP(BPF_FIXUP_REG_MASK, dst_reg); ex->type = EX_TYPE_BPF; @@ -1007,6 +1034,7 @@ int arch_prepare_bpf_trampoline(struct bpf_tramp_image *im, void *image, ctx.ninsns = 0; ctx.insns = NULL; + ctx.ro_insns = NULL; ret = __arch_prepare_bpf_trampoline(im, m, tlinks, func_addr, flags, &ctx); if (ret < 0) return ret; @@ -1015,7 +1043,15 @@ int arch_prepare_bpf_trampoline(struct bpf_tramp_image *im, void *image, return -EFBIG; ctx.ninsns = 0; + /* + * The bpf_int_jit_compile() uses a RW buffer (ctx.insns) to write the + * JITed instructions and later copies it to a RX region (ctx.ro_insns). + * It also uses ctx.ro_insns to calculate offsets for jumps etc. As the + * trampoline image uses the same memory area for writing and execution, + * both ctx.insns and ctx.ro_insns can be set to image. + */ ctx.insns = image; + ctx.ro_insns = image; ret = __arch_prepare_bpf_trampoline(im, m, tlinks, func_addr, flags, &ctx); if (ret < 0) return ret; diff --git a/arch/riscv/net/bpf_jit_core.c b/arch/riscv/net/bpf_jit_core.c index 7a26a3e1c73c..7b70ccb7fec3 100644 --- a/arch/riscv/net/bpf_jit_core.c +++ b/arch/riscv/net/bpf_jit_core.c @@ -8,6 +8,8 @@ #include #include +#include +#include #include "bpf_jit.h" /* Number of iterations to try until offsets converge. */ @@ -117,16 +119,24 @@ struct bpf_prog *bpf_int_jit_compile(struct bpf_prog *prog) sizeof(struct exception_table_entry); prog_size = sizeof(*ctx->insns) * ctx->ninsns; - jit_data->header = - bpf_jit_binary_alloc(prog_size + extable_size, - &jit_data->image, - sizeof(u32), - bpf_fill_ill_insns); - if (!jit_data->header) { + jit_data->ro_header = + bpf_jit_binary_pack_alloc(prog_size + extable_size, + &jit_data->ro_image, sizeof(u32), + &jit_data->header, &jit_data->image, + bpf_fill_ill_insns); + if (!jit_data->ro_header) { prog = orig_prog; goto out_offset; } + /* + * Use the image(RW) for writing the JITed instructions. But also save + * the ro_image(RX) for calculating the offsets in the image. The RW + * image will be later copied to the RX image from where the program + * will run. The bpf_jit_binary_pack_finalize() will do this copy in the + * final step. + */ + ctx->ro_insns = (u16 *)jit_data->ro_image; ctx->insns = (u16 *)jit_data->image; /* * Now, when the image is allocated, the image can @@ -138,14 +148,12 @@ struct bpf_prog *bpf_int_jit_compile(struct bpf_prog *prog) if (i == NR_JIT_ITERATIONS) { pr_err("bpf-jit: image did not converge in <%d passes!\n", i); - if (jit_data->header) - bpf_jit_binary_free(jit_data->header); prog = orig_prog; - goto out_offset; + goto out_free_hdr; } if (extable_size) - prog->aux->extable = (void *)ctx->insns + prog_size; + prog->aux->extable = (void *)ctx->ro_insns + prog_size; skip_init_ctx: pass++; @@ -154,23 +162,33 @@ struct bpf_prog *bpf_int_jit_compile(struct bpf_prog *prog) bpf_jit_build_prologue(ctx); if (build_body(ctx, extra_pass, NULL)) { - bpf_jit_binary_free(jit_data->header); prog = orig_prog; - goto out_offset; + goto out_free_hdr; } bpf_jit_build_epilogue(ctx); if (bpf_jit_enable > 1) bpf_jit_dump(prog->len, prog_size, pass, ctx->insns); - prog->bpf_func = (void *)ctx->insns; + prog->bpf_func = (void *)ctx->ro_insns; prog->jited = 1; prog->jited_len = prog_size; - bpf_flush_icache(jit_data->header, ctx->insns + ctx->ninsns); - if (!prog->is_func || extra_pass) { - bpf_jit_binary_lock_ro(jit_data->header); + if (WARN_ON(bpf_jit_binary_pack_finalize(prog, jit_data->ro_header, + jit_data->header))) { + /* ro_header has been freed */ + jit_data->ro_header = NULL; + prog = orig_prog; + goto out_offset; + } + /* + * The instructions have now been copied to the ROX region from + * where they will execute. + * Write any modified data cache blocks out to memory and + * invalidate the corresponding blocks in the instruction cache. + */ + bpf_flush_icache(jit_data->ro_header, ctx->ro_insns + ctx->ninsns); for (i = 0; i < prog->len; i++) ctx->offset[i] = ninsns_rvoff(ctx->offset[i]); bpf_prog_fill_jited_linfo(prog, ctx->offset); @@ -185,6 +203,14 @@ struct bpf_prog *bpf_int_jit_compile(struct bpf_prog *prog) bpf_jit_prog_release_other(prog, prog == orig_prog ? tmp : orig_prog); return prog; + +out_free_hdr: + if (jit_data->header) { + bpf_arch_text_copy(&jit_data->ro_header->size, &jit_data->header->size, + sizeof(jit_data->header->size)); + bpf_jit_binary_pack_free(jit_data->ro_header, jit_data->header); + } + goto out_offset; } u64 bpf_jit_alloc_exec_limit(void) @@ -204,3 +230,51 @@ void bpf_jit_free_exec(void *addr) { return vfree(addr); } + +void *bpf_arch_text_copy(void *dst, void *src, size_t len) +{ + int ret; + + mutex_lock(&text_mutex); + ret = patch_text_nosync(dst, src, len); + mutex_unlock(&text_mutex); + + if (ret) + return ERR_PTR(-EINVAL); + + return dst; +} + +int bpf_arch_text_invalidate(void *dst, size_t len) +{ + int ret; + + mutex_lock(&text_mutex); + ret = patch_text_set_nosync(dst, 0, len); + mutex_unlock(&text_mutex); + + return ret; +} + +void bpf_jit_free(struct bpf_prog *prog) +{ + if (prog->jited) { + struct rv_jit_data *jit_data = prog->aux->jit_data; + struct bpf_binary_header *hdr; + + /* + * If we fail the final pass of JIT (from jit_subprogs), + * the program may not be finalized yet. Call finalize here + * before freeing it. + */ + if (jit_data) { + bpf_jit_binary_pack_finalize(prog, jit_data->ro_header, jit_data->header); + kfree(jit_data); + } + hdr = bpf_jit_binary_pack_hdr(prog); + bpf_jit_binary_pack_free(hdr, NULL); + WARN_ON_ONCE(!bpf_prog_kallsyms_verify_off(prog)); + } + + bpf_prog_unlock_free(prog); +}