From patchwork Thu Aug 24 13:31:33 2023 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 8bit X-Patchwork-Submitter: Puranjay Mohan X-Patchwork-Id: 13364146 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from bombadil.infradead.org (bombadil.infradead.org [198.137.202.133]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (No client certificate requested) by smtp.lore.kernel.org (Postfix) with ESMTPS id 3C3CFC27C40 for ; Thu, 24 Aug 2023 13:31:49 +0000 (UTC) DKIM-Signature: v=1; a=rsa-sha256; q=dns/txt; c=relaxed/relaxed; d=lists.infradead.org; s=bombadil.20210309; h=Sender: Content-Transfer-Encoding:Content-Type:List-Subscribe:List-Help:List-Post: List-Archive:List-Unsubscribe:List-Id:MIME-Version:References:In-Reply-To: Message-Id:Date:Subject:Cc:To:From:Reply-To:Content-ID:Content-Description: Resent-Date:Resent-From:Resent-Sender:Resent-To:Resent-Cc:Resent-Message-ID: List-Owner; bh=C6D0jb3nOBr0URqx+rj1tGEJ485/Q0JOluUnhyKZwlM=; b=HOd3V9+AKufDUF ZHwm/sbZg+Ld8O2QhWEfVao5KbITIM0PKxvU2ithdj8bQeYoXaVEHGwOm1BnpMDLxSl5oPey2yZmh mFE5yCLQKcTQk8LtRqpfUaZo9BuMDDA+pww9kdQXTTd/QivB90zFFnIG7fLxNAnFeHSjR0Z72lJDN CD8l8eyD3J1Aftv0Sz7NrNbSDJZ2BrSn1h42BUn/E152Gwduq1PenjEaIxT8y0L8iJ5KdUkR5PbkX 4bmHiuuD5KMbMdWIDQzytwjUMFkVBrDQ7mCpZb4H8zqNhpxmk/3Oe4qy1rMs/eLeAHwun1sTOY3Po 5AphWTA4XpfrYGyUuNlA==; Received: from localhost ([::1] helo=bombadil.infradead.org) by bombadil.infradead.org with esmtp (Exim 4.96 #2 (Red Hat Linux)) id 1qZAR6-00392V-2L; Thu, 24 Aug 2023 13:31:44 +0000 Received: from mail-wr1-x434.google.com ([2a00:1450:4864:20::434]) by bombadil.infradead.org with esmtps (Exim 4.96 #2 (Red Hat Linux)) id 1qZAR3-00390p-2A for linux-riscv@lists.infradead.org; Thu, 24 Aug 2023 13:31:43 +0000 Received: by mail-wr1-x434.google.com with SMTP id ffacd0b85a97d-307d20548adso5568749f8f.0 for ; Thu, 24 Aug 2023 06:31:39 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=20221208; t=1692883898; x=1693488698; h=content-transfer-encoding:mime-version:references:in-reply-to :message-id:date:subject:cc:to:from:from:to:cc:subject:date :message-id:reply-to; bh=HOjnMoCdk8ero0eIWnTRV8IDjWokCkOdT233dw8f8cM=; b=iMnUSfI2+Q9SIs1K6hZhMo+Arr01ZqvJxwD3uxjGKTxs2C3mOEX8zTkSa/iw1/ea+A 1obGFCB1YEh4uU1B9tgX84lYcb0Efl5eHke1ZttpwSWnkzqrkRoI8QXmaqnlazK0TD5+ 9tsnxUjQJNkcnAO60l48rHELcsLqQKRjrtwVMkLHZFhD3Fsa/mrCPT4+o+K9IrXPhiaG sYLcigRuUCUk0huxgn6LcK5vsuf1FNxZqb/70GapWKjid2DGaXJagHrHYax7v/q3dpy1 5uX+gS81yYp8mQKkfYi+eMqRjrxMSLuHa0Vdk/52tySindL0WVTzzrxO7oz8N0y0DGq0 OKcw== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20221208; t=1692883898; x=1693488698; h=content-transfer-encoding:mime-version:references:in-reply-to :message-id:date:subject:cc:to:from:x-gm-message-state:from:to:cc :subject:date:message-id:reply-to; bh=HOjnMoCdk8ero0eIWnTRV8IDjWokCkOdT233dw8f8cM=; b=b0wvE98VYKvLjTVDR1jqegHPP9XvIZaOWj6UfPAtLJMcM+PCPiUoZgdsrZv7WVICPt x9cVlEfkzAkKUt+9EJ8RKPDNYDPHy/QLiZmACa+Q+WFdVjZ/k5LP8mqG1VoNWfK1TRAj EmozmtQmKT/TsKiziYbpgtuRBLwt9FToosHtYedpJL3UaJjcThedc9DL+S/BdV5EdGTe flH1wapfdXr3JJ8CqRxmd4t6KM5Vh6Tza/RSUtfIWCmRmhkDC1V92iwAcEDzUYiCfF2p spqB23d14EWBonbrb0PNvlEhliFhBhuny9zoFh075QMjECjQ1ji1THpl3/FQI/+yJgCq YSvw== X-Gm-Message-State: AOJu0YxWjzeY5WdLDaog/XDbR7O64H00N35f8Pt9lI3fMvHGh3sfCr4a SD3zlzQHxzkRn/evZWsE8BM= X-Google-Smtp-Source: AGHT+IGYrNSHQo0VfFl5CRfj3Gtb5eiirSq1QGjvtMJU8Gr4wK3Ak9RlHQPqQpRwwQlFkga9ww4Lcg== X-Received: by 2002:a5d:62cf:0:b0:319:79a9:4d9e with SMTP id o15-20020a5d62cf000000b0031979a94d9emr11901798wrv.44.1692883898481; Thu, 24 Aug 2023 06:31:38 -0700 (PDT) Received: from ip-172-31-30-46.eu-west-1.compute.internal (ec2-54-170-241-106.eu-west-1.compute.amazonaws.com. [54.170.241.106]) by smtp.gmail.com with ESMTPSA id h11-20020a5d548b000000b00317e77106dbsm22396112wrv.48.2023.08.24.06.31.37 (version=TLS1_3 cipher=TLS_AES_256_GCM_SHA384 bits=256/256); Thu, 24 Aug 2023 06:31:38 -0700 (PDT) From: Puranjay Mohan To: paul.walmsley@sifive.com, palmer@dabbelt.com, aou@eecs.berkeley.edu, pulehui@huawei.com, conor.dooley@microchip.com, ast@kernel.org, daniel@iogearbox.net, andrii@kernel.org, martin.lau@linux.dev, song@kernel.org, yhs@fb.com, kpsingh@kernel.org, bjorn@kernel.org, bpf@vger.kernel.org, linux-riscv@lists.infradead.org, linux-kernel@vger.kernel.org Cc: puranjay12@gmail.com Subject: [PATCH bpf-next v2 1/3] riscv: extend patch_text_nosync() for multiple pages Date: Thu, 24 Aug 2023 13:31:33 +0000 Message-Id: <20230824133135.1176709-2-puranjay12@gmail.com> X-Mailer: git-send-email 2.40.1 In-Reply-To: <20230824133135.1176709-1-puranjay12@gmail.com> References: <20230824133135.1176709-1-puranjay12@gmail.com> MIME-Version: 1.0 X-CRM114-Version: 20100106-BlameMichelson ( TRE 0.8.0 (BSD) ) MR-646709E3 X-CRM114-CacheID: sfid-20230824_063141_711539_D926DB77 X-CRM114-Status: GOOD ( 15.45 ) X-BeenThere: linux-riscv@lists.infradead.org X-Mailman-Version: 2.1.34 Precedence: list List-Id: List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Sender: "linux-riscv" Errors-To: linux-riscv-bounces+linux-riscv=archiver.kernel.org@lists.infradead.org The patch_insn_write() function currently doesn't work for multiple pages of instructions, therefore patch_text_nosync() will fail with a page fault if called with lengths spanning multiple pages. This commit extends the patch_insn_write() function to support multiple pages by copying at max 2 pages at a time in a loop. This implementation is similar to text_poke_copy() function of x86. Signed-off-by: Puranjay Mohan Reviewed-by: Björn Töpel --- arch/riscv/kernel/patch.c | 39 ++++++++++++++++++++++++++++++++++----- 1 file changed, 34 insertions(+), 5 deletions(-) diff --git a/arch/riscv/kernel/patch.c b/arch/riscv/kernel/patch.c index 575e71d6c8ae..465b2eebbc37 100644 --- a/arch/riscv/kernel/patch.c +++ b/arch/riscv/kernel/patch.c @@ -53,12 +53,18 @@ static void patch_unmap(int fixmap) } NOKPROBE_SYMBOL(patch_unmap); -static int patch_insn_write(void *addr, const void *insn, size_t len) +static int __patch_insn_write(void *addr, const void *insn, size_t len) { void *waddr = addr; bool across_pages = (((uintptr_t) addr & ~PAGE_MASK) + len) > PAGE_SIZE; int ret; + /* + * Only two pages can be mapped at a time for writing. + */ + if (len > 2 * PAGE_SIZE) + return -EINVAL; + /* * Before reaching here, it was expected to lock the text_mutex * already, so we don't need to give another lock here and could @@ -74,7 +80,7 @@ static int patch_insn_write(void *addr, const void *insn, size_t len) lockdep_assert_held(&text_mutex); if (across_pages) - patch_map(addr + len, FIX_TEXT_POKE1); + patch_map(addr + PAGE_SIZE, FIX_TEXT_POKE1); waddr = patch_map(addr, FIX_TEXT_POKE0); @@ -87,15 +93,38 @@ static int patch_insn_write(void *addr, const void *insn, size_t len) return ret; } -NOKPROBE_SYMBOL(patch_insn_write); +NOKPROBE_SYMBOL(__patch_insn_write); #else -static int patch_insn_write(void *addr, const void *insn, size_t len) +static int __patch_insn_write(void *addr, const void *insn, size_t len) { return copy_to_kernel_nofault(addr, insn, len); } -NOKPROBE_SYMBOL(patch_insn_write); +NOKPROBE_SYMBOL(__patch_insn_write); #endif /* CONFIG_MMU */ +static int patch_insn_write(void *addr, const void *insn, size_t len) +{ + size_t patched = 0; + size_t size; + int ret = 0; + + /* + * Copy the instructions to the destination address, two pages at a time + * because __patch_insn_write() can only handle len <= 2 * PAGE_SIZE. + */ + while (patched < len && !ret) { + size = min_t(size_t, + PAGE_SIZE * 2 - offset_in_page(addr + patched), + len - patched); + ret = __patch_insn_write(addr + patched, insn + patched, size); + + patched += size; + } + + return ret; +} +NOKPROBE_SYMBOL(patch_insn_write); + int patch_text_nosync(void *addr, const void *insns, size_t len) { u32 *tp = addr; From patchwork Thu Aug 24 13:31:34 2023 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Puranjay Mohan X-Patchwork-Id: 13364147 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from bombadil.infradead.org (bombadil.infradead.org [198.137.202.133]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (No client certificate requested) by smtp.lore.kernel.org (Postfix) with ESMTPS id B8A9CC27C40 for ; Thu, 24 Aug 2023 13:31:52 +0000 (UTC) DKIM-Signature: v=1; a=rsa-sha256; q=dns/txt; c=relaxed/relaxed; d=lists.infradead.org; s=bombadil.20210309; h=Sender: Content-Transfer-Encoding:Content-Type:List-Subscribe:List-Help:List-Post: List-Archive:List-Unsubscribe:List-Id:MIME-Version:References:In-Reply-To: Message-Id:Date:Subject:Cc:To:From:Reply-To:Content-ID:Content-Description: Resent-Date:Resent-From:Resent-Sender:Resent-To:Resent-Cc:Resent-Message-ID: List-Owner; bh=xkqYZY+Hh69kZxbc/ZzgmG8PdfiZn08pOTYkhh9Scvw=; b=J0A1PBAjPz7Z9a N2cn146J8DNKgT/1/0u5U0Uu32ds0hPnLNDXoOemqBzf5X6OA3OlzNKOT/pEPHHeGM7W+wG4gcX5A /wEwS5XwmMlijMTco3XWo1NcAjMpTd9GBHqpuFpVac5cjb3pkmiMaJdqKXx5lL0UlBH5resZvPJCh EldLkFDKSNDCRBuFD2UB9ssQFlrKleNnbvgCG2N21s+VWNoe+PUJS7TilRJpeEdw/CYI86QHMvFMc FhF+8EuL4pdwnPKxWGw726gItuQApSvVcIevEyZkSzVE7U0XAyXglmUmHcI6BJuFNCU9HCsgHQjZE ZwNaj5Dg7gzD3/DQDmbA==; Received: from localhost ([::1] helo=bombadil.infradead.org) by bombadil.infradead.org with esmtp (Exim 4.96 #2 (Red Hat Linux)) id 1qZARA-00393U-1Y; Thu, 24 Aug 2023 13:31:48 +0000 Received: from mail-wr1-x430.google.com ([2a00:1450:4864:20::430]) by bombadil.infradead.org with esmtps (Exim 4.96 #2 (Red Hat Linux)) id 1qZAR7-00391a-2Z for linux-riscv@lists.infradead.org; Thu, 24 Aug 2023 13:31:47 +0000 Received: by mail-wr1-x430.google.com with SMTP id ffacd0b85a97d-31c5a2e8501so3467314f8f.0 for ; Thu, 24 Aug 2023 06:31:42 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=20221208; t=1692883901; x=1693488701; h=content-transfer-encoding:mime-version:references:in-reply-to :message-id:date:subject:cc:to:from:from:to:cc:subject:date :message-id:reply-to; bh=lRbZ/8a2wFTgwFE+Czu2PpLLLzMWLxHTHSPnuA9Gm+o=; b=oYWYvKePLZGjS+ZoEl8nqGnfKWJ/5ZYZqjbab0+kcN0rDQKfC+uPSsnIgWkMRmchaP kmB/bloU4msL5s6GFjkSzVQf8js0C978vVWo4JbuGczsYJ/hiJcB914hx8ipwG2pLE4r 5xV7/f4Hn73bYoAhdKka0EWmDvBuFjYAdFIL+QTaLTz04odlXiv4A1ZxyzykLLfmtDYe ZKWDxm+ZsoNc7WWZC2lUfFzhqCg8jcrKz+Y3S42S1EZz3NHqw0cr+G45BZdYegXPzh12 3c6OYT7W+bmFMjwwWfodGWLxbHSaXz5xU5SccK3v8bNMhYtWEHJSa0F0Rpaz0D6PcFD3 HFig== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20221208; t=1692883901; x=1693488701; h=content-transfer-encoding:mime-version:references:in-reply-to :message-id:date:subject:cc:to:from:x-gm-message-state:from:to:cc :subject:date:message-id:reply-to; bh=lRbZ/8a2wFTgwFE+Czu2PpLLLzMWLxHTHSPnuA9Gm+o=; b=fSOR7cSSLGJU4gX1AGIVDcu7fd+YiLvnjcpMO6RggS9KuCmOgRVt7v2RDsy9sSZw9Z FGdagkrvHQHQneZtaOEGXQ3GmKEUUMAokZufB+Oio/WTLUtkVPjZOj1OFFrf/efWYkRz 7oHcY2jSRqmRcc7jVEBRRtAsmsKm8812YCOD7mErASRY48hcAiBw2dSoFSV9fuMDM71b UgkAMC2GXvskboeFWEm/SKWY72+yJRHfsZGkS/gGkmvxDhqjDUhUxq6G7rXHBFt+9MLt BRu1Nl5LS/voGNCDoETI6guB1RiIqeprPa1Yq5sq13iJ+78nZRD3u1H7IL8VU7SyB/+u 1dwQ== X-Gm-Message-State: AOJu0YyQK3NoTfDSSvEGzcdXlNOMYJciubzsuJwGOiGwKveyIub6eeWE lKJGSLLsXMEcNwM9lSnuXx8= X-Google-Smtp-Source: AGHT+IHSeNPDuiAo4mvJL6SeFZqm5Y/OINirWF+gSjBoFG2Ur8fcxnPmq3uvy/x+VkYH3pkNmTQiNA== X-Received: by 2002:a5d:6084:0:b0:31a:d9bc:47a2 with SMTP id w4-20020a5d6084000000b0031ad9bc47a2mr13370813wrt.53.1692883901036; Thu, 24 Aug 2023 06:31:41 -0700 (PDT) Received: from ip-172-31-30-46.eu-west-1.compute.internal (ec2-54-170-241-106.eu-west-1.compute.amazonaws.com. [54.170.241.106]) by smtp.gmail.com with ESMTPSA id h11-20020a5d548b000000b00317e77106dbsm22396112wrv.48.2023.08.24.06.31.38 (version=TLS1_3 cipher=TLS_AES_256_GCM_SHA384 bits=256/256); Thu, 24 Aug 2023 06:31:39 -0700 (PDT) From: Puranjay Mohan To: paul.walmsley@sifive.com, palmer@dabbelt.com, aou@eecs.berkeley.edu, pulehui@huawei.com, conor.dooley@microchip.com, ast@kernel.org, daniel@iogearbox.net, andrii@kernel.org, martin.lau@linux.dev, song@kernel.org, yhs@fb.com, kpsingh@kernel.org, bjorn@kernel.org, bpf@vger.kernel.org, linux-riscv@lists.infradead.org, linux-kernel@vger.kernel.org Cc: puranjay12@gmail.com Subject: [PATCH bpf-next v2 2/3] riscv: implement a memset like function for text Date: Thu, 24 Aug 2023 13:31:34 +0000 Message-Id: <20230824133135.1176709-3-puranjay12@gmail.com> X-Mailer: git-send-email 2.40.1 In-Reply-To: <20230824133135.1176709-1-puranjay12@gmail.com> References: <20230824133135.1176709-1-puranjay12@gmail.com> MIME-Version: 1.0 X-CRM114-Version: 20100106-BlameMichelson ( TRE 0.8.0 (BSD) ) MR-646709E3 X-CRM114-CacheID: sfid-20230824_063145_845880_F51B6DA8 X-CRM114-Status: GOOD ( 14.35 ) X-BeenThere: linux-riscv@lists.infradead.org X-Mailman-Version: 2.1.34 Precedence: list List-Id: List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Sender: "linux-riscv" Errors-To: linux-riscv-bounces+linux-riscv=archiver.kernel.org@lists.infradead.org The BPF JIT needs to write invalid instructions to RX regions of memory to invalidate removed BPF programs. This needs a function like memset() that can work with RX memory. Implement patch_text_set_nosync() which is similar to text_poke_set() of x86. Signed-off-by: Puranjay Mohan --- arch/riscv/include/asm/patch.h | 1 + arch/riscv/kernel/patch.c | 74 ++++++++++++++++++++++++++++++++++ 2 files changed, 75 insertions(+) diff --git a/arch/riscv/include/asm/patch.h b/arch/riscv/include/asm/patch.h index 63c98833d510..aa5c1830ea43 100644 --- a/arch/riscv/include/asm/patch.h +++ b/arch/riscv/include/asm/patch.h @@ -7,6 +7,7 @@ #define _ASM_RISCV_PATCH_H int patch_text_nosync(void *addr, const void *insns, size_t len); +int patch_text_set_nosync(void *addr, const int c, size_t len); int patch_text(void *addr, u32 *insns, int ninsns); extern int riscv_patch_in_stop_machine; diff --git a/arch/riscv/kernel/patch.c b/arch/riscv/kernel/patch.c index 465b2eebbc37..24d49999ac1a 100644 --- a/arch/riscv/kernel/patch.c +++ b/arch/riscv/kernel/patch.c @@ -13,6 +13,7 @@ #include #include #include +#include struct patch_insn { void *addr; @@ -53,6 +54,34 @@ static void patch_unmap(int fixmap) } NOKPROBE_SYMBOL(patch_unmap); +static int __patch_insn_set(void *addr, const int c, size_t len) +{ + void *waddr = addr; + bool across_pages = (((uintptr_t) addr & ~PAGE_MASK) + len) > PAGE_SIZE; + int ret; + + /* + * Only two pages can be mapped at a time for writing. + */ + if (len > 2 * PAGE_SIZE) + return -EINVAL; + + if (across_pages) + patch_map(addr + PAGE_SIZE, FIX_TEXT_POKE1); + + waddr = patch_map(addr, FIX_TEXT_POKE0); + + memset(waddr, c, len); + + patch_unmap(FIX_TEXT_POKE0); + + if (across_pages) + patch_unmap(FIX_TEXT_POKE1); + + return 0; +} +NOKPROBE_SYMBOL(__patch_insn_set); + static int __patch_insn_write(void *addr, const void *insn, size_t len) { void *waddr = addr; @@ -95,6 +124,14 @@ static int __patch_insn_write(void *addr, const void *insn, size_t len) } NOKPROBE_SYMBOL(__patch_insn_write); #else +static int __patch_insn_set (void *addr, const int c, size_t len) +{ + memset(addr, c, len); + + return 0; +} +NOKPROBE_SYMBOL(__patch_insn_set); + static int __patch_insn_write(void *addr, const void *insn, size_t len) { return copy_to_kernel_nofault(addr, insn, len); @@ -102,6 +139,43 @@ static int __patch_insn_write(void *addr, const void *insn, size_t len) NOKPROBE_SYMBOL(__patch_insn_write); #endif /* CONFIG_MMU */ +static int patch_insn_set(void *addr, const int c, size_t len) +{ + size_t patched = 0; + size_t size; + int ret = 0; + + /* + * __patch_insn_set() can only work on 2 pages at a time so call it in a + * loop with len <= 2 * PAGE_SIZE. + */ + while (patched < len && !ret) { + size = min_t(size_t, + PAGE_SIZE * 2 - offset_in_page(addr + patched), + len - patched); + ret = __patch_insn_set(addr + patched, c, size); + + patched += size; + } + + return ret; +} +NOKPROBE_SYMBOL(patch_insn_set); + +int patch_text_set_nosync(void *addr, const int c, size_t len) +{ + u32 *tp = addr; + int ret; + + ret = patch_insn_set(tp, c, len); + + if (!ret) + flush_icache_range((uintptr_t) tp, (uintptr_t) tp + len); + + return ret; +} +NOKPROBE_SYMBOL(patch_text_set_nosync); + static int patch_insn_write(void *addr, const void *insn, size_t len) { size_t patched = 0; From patchwork Thu Aug 24 13:31:35 2023 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Puranjay Mohan X-Patchwork-Id: 13364148 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from bombadil.infradead.org (bombadil.infradead.org [198.137.202.133]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (No client certificate requested) by smtp.lore.kernel.org (Postfix) with ESMTPS id 0FFDDC6FA8F for ; Thu, 24 Aug 2023 13:31:57 +0000 (UTC) DKIM-Signature: v=1; a=rsa-sha256; q=dns/txt; c=relaxed/relaxed; d=lists.infradead.org; s=bombadil.20210309; h=Sender: Content-Transfer-Encoding:Content-Type:List-Subscribe:List-Help:List-Post: List-Archive:List-Unsubscribe:List-Id:MIME-Version:References:In-Reply-To: Message-Id:Date:Subject:Cc:To:From:Reply-To:Content-ID:Content-Description: Resent-Date:Resent-From:Resent-Sender:Resent-To:Resent-Cc:Resent-Message-ID: List-Owner; bh=lIsuS0ycwo14YGhTyUKnhPmy9Fr2uQKlB1sQxDM8KIM=; b=OozzaFxB5LH6s1 msZFy2YkqWlZNzU7MDKbODkjDku2qZG68yn6pZKzOQBBT+7cHJhZcZrhcyR3ymEdTviBbj0s3mUkf 59fjy0MtojhDq2W/d5PHsnCOgfWJZiSyPRznUsZWxIUgxwdlNnr3aNl6Jvp4dldihwfUZf331i7CU mi5XimJJEfHgRO+qsYHExDnfda+auLm2q7iTsLN3PG1A/ES3aZxxTwVf/lobdZJPMCD7iwW8c0CTc FZn1cZgSE17j7Y5WkodeBj+6bbJF5D5+ka1TukCjtD+arkuMzZUPdvlNs3DIJVv72xN4EAlBRx5fJ 0Sa8Haj6hi51wnexVoOA==; Received: from localhost ([::1] helo=bombadil.infradead.org) by bombadil.infradead.org with esmtp (Exim 4.96 #2 (Red Hat Linux)) id 1qZARE-00394v-0a; Thu, 24 Aug 2023 13:31:52 +0000 Received: from mail-wr1-x429.google.com ([2a00:1450:4864:20::429]) by bombadil.infradead.org with esmtps (Exim 4.96 #2 (Red Hat Linux)) id 1qZAR9-00392G-1B for linux-riscv@lists.infradead.org; Thu, 24 Aug 2023 13:31:49 +0000 Received: by mail-wr1-x429.google.com with SMTP id ffacd0b85a97d-31c73c21113so739142f8f.1 for ; Thu, 24 Aug 2023 06:31:44 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=20221208; t=1692883903; x=1693488703; h=content-transfer-encoding:mime-version:references:in-reply-to :message-id:date:subject:cc:to:from:from:to:cc:subject:date :message-id:reply-to; bh=ZE4gDzK4BarFOpFDUm7HsASsZoYlPGB/PtNYZtQuQCs=; b=kUDT9RjwDHv+JZXpX798v6QF8WCmiH9+ChraAqrdEgCcxpCTrkdSXC0HSqVuKGjMoe 4TZ8TMsgeLbCUFChDXAMCx6GW8BPtylnx6TxgmN4afs+pjIBNFfdfZRMMXCg+w1vCtJQ QVz/uXJ5LUmDSA8901Uj6fGVc1hqiBYqIfWuYJXgvMS6bBU6xifqsCXcLQUGB2WF/+l/ h+AhNeV5kA1gZ/qCEYrW5jhJfQe+YSE9g5+qL1Uo3Fukd/qZCisCNVUW6QJ+LdkiFr9N 3otdSppiB1rW3Gw2F1X2OCk3aNWlspGnyXPpEmCKaIABB28qu6r4/gT6duzmsg+48xQe +rEg== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20221208; t=1692883903; x=1693488703; h=content-transfer-encoding:mime-version:references:in-reply-to :message-id:date:subject:cc:to:from:x-gm-message-state:from:to:cc :subject:date:message-id:reply-to; bh=ZE4gDzK4BarFOpFDUm7HsASsZoYlPGB/PtNYZtQuQCs=; b=FAXFlKhWPb214FEWUXWT1wOCzd+Xpo9gLAVgxgapXBIv4STqn/oXKpfQ6N4A+oCAkq aAcTfQzhk0h2e+ZpTvVeL2IMKqG1hp9Vk+2EJ5EuNiqy1m4OFoXP7wT0VCpbnvl1kR8A lO7X08yizHGL9S0q/xkYrkZxNg00Jd3Ay43xJZ6j5qn423wcLnH0J0p3LpEd6KbWWI9o VdrnWN2vXzrIMzP5RHvkEcO1O5YVOzwVgDD4pFBM/mXiFl4Zerttr7XTPL+fD5BRv8zt EkpAMnN+XKB9oEZ1u289lABAiSjqwUgLsRnDvS9GODjSaYmLEe8Zg1aFJk4sCfFMhXuW 4bMw== X-Gm-Message-State: AOJu0YzurCBa9FYM06o3s48aCCMw7nHvAWpXLLnRDxsBmJmjvzDHpwxj ae/mlKiBuf4dBlW8Wt12Q2s= X-Google-Smtp-Source: AGHT+IE1DHrzD3RMD7X4DeWFzUxZYCQraZB2mYQ/IPMdz7u1EG1Wbjt6XtsI/mZNX+FYlVXWg+AhbQ== X-Received: by 2002:adf:f602:0:b0:31c:5e4d:4e75 with SMTP id t2-20020adff602000000b0031c5e4d4e75mr7471812wrp.44.1692883902923; Thu, 24 Aug 2023 06:31:42 -0700 (PDT) Received: from ip-172-31-30-46.eu-west-1.compute.internal (ec2-54-170-241-106.eu-west-1.compute.amazonaws.com. [54.170.241.106]) by smtp.gmail.com with ESMTPSA id h11-20020a5d548b000000b00317e77106dbsm22396112wrv.48.2023.08.24.06.31.41 (version=TLS1_3 cipher=TLS_AES_256_GCM_SHA384 bits=256/256); Thu, 24 Aug 2023 06:31:41 -0700 (PDT) From: Puranjay Mohan To: paul.walmsley@sifive.com, palmer@dabbelt.com, aou@eecs.berkeley.edu, pulehui@huawei.com, conor.dooley@microchip.com, ast@kernel.org, daniel@iogearbox.net, andrii@kernel.org, martin.lau@linux.dev, song@kernel.org, yhs@fb.com, kpsingh@kernel.org, bjorn@kernel.org, bpf@vger.kernel.org, linux-riscv@lists.infradead.org, linux-kernel@vger.kernel.org Cc: puranjay12@gmail.com Subject: [PATCH bpf-next v2 3/3] bpf, riscv: use prog pack allocator in the BPF JIT Date: Thu, 24 Aug 2023 13:31:35 +0000 Message-Id: <20230824133135.1176709-4-puranjay12@gmail.com> X-Mailer: git-send-email 2.40.1 In-Reply-To: <20230824133135.1176709-1-puranjay12@gmail.com> References: <20230824133135.1176709-1-puranjay12@gmail.com> MIME-Version: 1.0 X-CRM114-Version: 20100106-BlameMichelson ( TRE 0.8.0 (BSD) ) MR-646709E3 X-CRM114-CacheID: sfid-20230824_063147_408590_8C4CC204 X-CRM114-Status: GOOD ( 29.46 ) X-BeenThere: linux-riscv@lists.infradead.org X-Mailman-Version: 2.1.34 Precedence: list List-Id: List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Sender: "linux-riscv" Errors-To: linux-riscv-bounces+linux-riscv=archiver.kernel.org@lists.infradead.org Use bpf_jit_binary_pack_alloc() for memory management of JIT binaries in RISCV BPF JIT. The bpf_jit_binary_pack_alloc creates a pair of RW and RX buffers. The JIT writes the program into the RW buffer. When the JIT is done, the program is copied to the final RX buffer with bpf_jit_binary_pack_finalize. Implement bpf_arch_text_copy() and bpf_arch_text_invalidate() for RISCV JIT as these functions are required by bpf_jit_binary_pack allocator. Signed-off-by: Puranjay Mohan Reviewed-by: Song Liu --- arch/riscv/net/bpf_jit.h | 3 + arch/riscv/net/bpf_jit_comp64.c | 56 +++++++++++++--- arch/riscv/net/bpf_jit_core.c | 113 +++++++++++++++++++++++++++----- 3 files changed, 146 insertions(+), 26 deletions(-) diff --git a/arch/riscv/net/bpf_jit.h b/arch/riscv/net/bpf_jit.h index 2717f5490428..ad69319c8ea7 100644 --- a/arch/riscv/net/bpf_jit.h +++ b/arch/riscv/net/bpf_jit.h @@ -68,6 +68,7 @@ static inline bool is_creg(u8 reg) struct rv_jit_context { struct bpf_prog *prog; u16 *insns; /* RV insns */ + u16 *ro_insns; int ninsns; int prologue_len; int epilogue_offset; @@ -85,7 +86,9 @@ static inline int ninsns_rvoff(int ninsns) struct rv_jit_data { struct bpf_binary_header *header; + struct bpf_binary_header *ro_header; u8 *image; + u8 *ro_image; struct rv_jit_context ctx; }; diff --git a/arch/riscv/net/bpf_jit_comp64.c b/arch/riscv/net/bpf_jit_comp64.c index 0ca4f5c0097c..d77b16338ba2 100644 --- a/arch/riscv/net/bpf_jit_comp64.c +++ b/arch/riscv/net/bpf_jit_comp64.c @@ -144,7 +144,11 @@ static bool in_auipc_jalr_range(s64 val) /* Emit fixed-length instructions for address */ static int emit_addr(u8 rd, u64 addr, bool extra_pass, struct rv_jit_context *ctx) { - u64 ip = (u64)(ctx->insns + ctx->ninsns); + /* + * Use the ro_insns(RX) to calculate the offset as the BPF program will + * finally run from this memory region. + */ + u64 ip = (u64)(ctx->ro_insns + ctx->ninsns); s64 off = addr - ip; s64 upper = (off + (1 << 11)) >> 12; s64 lower = off & 0xfff; @@ -465,7 +469,11 @@ static int emit_call(u64 addr, bool fixed_addr, struct rv_jit_context *ctx) u64 ip; if (addr && ctx->insns) { - ip = (u64)(long)(ctx->insns + ctx->ninsns); + /* + * Use the ro_insns(RX) to calculate the offset as the BPF + * program will finally run from this memory region. + */ + ip = (u64)(long)(ctx->ro_insns + ctx->ninsns); off = addr - ip; } @@ -578,7 +586,8 @@ static int add_exception_handler(const struct bpf_insn *insn, { struct exception_table_entry *ex; unsigned long pc; - off_t offset; + off_t ins_offset; + off_t fixup_offset; if (!ctx->insns || !ctx->prog->aux->extable || BPF_MODE(insn->code) != BPF_PROBE_MEM) return 0; @@ -593,12 +602,17 @@ static int add_exception_handler(const struct bpf_insn *insn, return -EINVAL; ex = &ctx->prog->aux->extable[ctx->nexentries]; - pc = (unsigned long)&ctx->insns[ctx->ninsns - insn_len]; + pc = (unsigned long)&ctx->ro_insns[ctx->ninsns - insn_len]; - offset = pc - (long)&ex->insn; - if (WARN_ON_ONCE(offset >= 0 || offset < INT_MIN)) + /* + * This is the relative offset of the instruction that may fault from + * the exception table itself. This will be written to the exception + * table and if this instruction faults, the destination register will + * be set to '0' and the execution will jump to the next instruction. + */ + ins_offset = pc - (long)&ex->insn; + if (WARN_ON_ONCE(ins_offset >= 0 || ins_offset < INT_MIN)) return -ERANGE; - ex->insn = offset; /* * Since the extable follows the program, the fixup offset is always @@ -607,12 +621,25 @@ static int add_exception_handler(const struct bpf_insn *insn, * bits. We don't need to worry about buildtime or runtime sort * modifying the upper bits because the table is already sorted, and * isn't part of the main exception table. + * + * The fixup_offset is set to the next instruction from the instruction + * that may fault. The execution will jump to this after handling the + * fault. */ - offset = (long)&ex->fixup - (pc + insn_len * sizeof(u16)); - if (!FIELD_FIT(BPF_FIXUP_OFFSET_MASK, offset)) + fixup_offset = (long)&ex->fixup - (pc + insn_len * sizeof(u16)); + if (!FIELD_FIT(BPF_FIXUP_OFFSET_MASK, fixup_offset)) return -ERANGE; - ex->fixup = FIELD_PREP(BPF_FIXUP_OFFSET_MASK, offset) | + /* + * The offsets above have been calculated using the RO buffer but we + * need to use the R/W buffer for writes. + * switch ex to rw buffer for writing. + */ + ex = (void *)ctx->insns + ((void *)ex - (void *)ctx->ro_insns); + + ex->insn = ins_offset; + + ex->fixup = FIELD_PREP(BPF_FIXUP_OFFSET_MASK, fixup_offset) | FIELD_PREP(BPF_FIXUP_REG_MASK, dst_reg); ex->type = EX_TYPE_BPF; @@ -1006,6 +1033,7 @@ int arch_prepare_bpf_trampoline(struct bpf_tramp_image *im, void *image, ctx.ninsns = 0; ctx.insns = NULL; + ctx.ro_insns = NULL; ret = __arch_prepare_bpf_trampoline(im, m, tlinks, func_addr, flags, &ctx); if (ret < 0) return ret; @@ -1014,7 +1042,15 @@ int arch_prepare_bpf_trampoline(struct bpf_tramp_image *im, void *image, return -EFBIG; ctx.ninsns = 0; + /* + * The bpf_int_jit_compile() uses a RW buffer (ctx.insns) to write the + * JITed instructions and later copies it to a RX region (ctx.ro_insns). + * It also uses ctx.ro_insns to calculate offsets for jumps etc. As the + * trampoline image uses the same memory area for writing and execution, + * both ctx.insns and ctx.ro_insns can be set to image. + */ ctx.insns = image; + ctx.ro_insns = image; ret = __arch_prepare_bpf_trampoline(im, m, tlinks, func_addr, flags, &ctx); if (ret < 0) return ret; diff --git a/arch/riscv/net/bpf_jit_core.c b/arch/riscv/net/bpf_jit_core.c index 7a26a3e1c73c..4c8dffc09368 100644 --- a/arch/riscv/net/bpf_jit_core.c +++ b/arch/riscv/net/bpf_jit_core.c @@ -8,6 +8,8 @@ #include #include +#include +#include #include "bpf_jit.h" /* Number of iterations to try until offsets converge. */ @@ -117,16 +119,27 @@ struct bpf_prog *bpf_int_jit_compile(struct bpf_prog *prog) sizeof(struct exception_table_entry); prog_size = sizeof(*ctx->insns) * ctx->ninsns; - jit_data->header = - bpf_jit_binary_alloc(prog_size + extable_size, - &jit_data->image, - sizeof(u32), - bpf_fill_ill_insns); - if (!jit_data->header) { + jit_data->ro_header = + bpf_jit_binary_pack_alloc(prog_size + + extable_size, + &jit_data->ro_image, + sizeof(u32), + &jit_data->header, + &jit_data->image, + bpf_fill_ill_insns); + if (!jit_data->ro_header) { prog = orig_prog; goto out_offset; } + /* + * Use the image(RW) for writing the JITed instructions. But also save + * the ro_image(RX) for calculating the offsets in the image. The RW + * image will be later copied to the RX image from where the program + * will run. The bpf_jit_binary_pack_finalize() will do this copy in the + * final step. + */ + ctx->ro_insns = (u16 *)jit_data->ro_image; ctx->insns = (u16 *)jit_data->image; /* * Now, when the image is allocated, the image can @@ -138,14 +151,12 @@ struct bpf_prog *bpf_int_jit_compile(struct bpf_prog *prog) if (i == NR_JIT_ITERATIONS) { pr_err("bpf-jit: image did not converge in <%d passes!\n", i); - if (jit_data->header) - bpf_jit_binary_free(jit_data->header); prog = orig_prog; - goto out_offset; + goto out_free_hdr; } if (extable_size) - prog->aux->extable = (void *)ctx->insns + prog_size; + prog->aux->extable = (void *)ctx->ro_insns + prog_size; skip_init_ctx: pass++; @@ -154,23 +165,35 @@ struct bpf_prog *bpf_int_jit_compile(struct bpf_prog *prog) bpf_jit_build_prologue(ctx); if (build_body(ctx, extra_pass, NULL)) { - bpf_jit_binary_free(jit_data->header); prog = orig_prog; - goto out_offset; + goto out_free_hdr; } bpf_jit_build_epilogue(ctx); if (bpf_jit_enable > 1) bpf_jit_dump(prog->len, prog_size, pass, ctx->insns); - prog->bpf_func = (void *)ctx->insns; + prog->bpf_func = (void *)ctx->ro_insns; prog->jited = 1; prog->jited_len = prog_size; - bpf_flush_icache(jit_data->header, ctx->insns + ctx->ninsns); - if (!prog->is_func || extra_pass) { - bpf_jit_binary_lock_ro(jit_data->header); + if (WARN_ON(bpf_jit_binary_pack_finalize(prog, + jit_data->ro_header, + jit_data->header))) { + /* ro_header has been freed */ + jit_data->ro_header = NULL; + prog = orig_prog; + goto out_offset; + } + /* + * The instructions have now been copied to the ROX region from + * where they will execute. + * Write any modified data cache blocks out to memory and + * invalidate the corresponding blocks in the instruction cache. + */ + bpf_flush_icache(jit_data->ro_header, + ctx->ro_insns + ctx->ninsns); for (i = 0; i < prog->len; i++) ctx->offset[i] = ninsns_rvoff(ctx->offset[i]); bpf_prog_fill_jited_linfo(prog, ctx->offset); @@ -185,6 +208,15 @@ struct bpf_prog *bpf_int_jit_compile(struct bpf_prog *prog) bpf_jit_prog_release_other(prog, prog == orig_prog ? tmp : orig_prog); return prog; + +out_free_hdr: + if (jit_data->header) { + bpf_arch_text_copy(&jit_data->ro_header->size, + &jit_data->header->size, + sizeof(jit_data->header->size)); + bpf_jit_binary_pack_free(jit_data->ro_header, jit_data->header); + } + goto out_offset; } u64 bpf_jit_alloc_exec_limit(void) @@ -204,3 +236,52 @@ void bpf_jit_free_exec(void *addr) { return vfree(addr); } + +void *bpf_arch_text_copy(void *dst, void *src, size_t len) +{ + int ret; + + mutex_lock(&text_mutex); + ret = patch_text_nosync(dst, src, len); + mutex_unlock(&text_mutex); + + if (ret) + return ERR_PTR(-EINVAL); + + return dst; +} + +int bpf_arch_text_invalidate(void *dst, size_t len) +{ + int ret = 0; + + mutex_lock(&text_mutex); + ret = patch_text_set_nosync(dst, 0, len); + mutex_unlock(&text_mutex); + + return ret; +} + +void bpf_jit_free(struct bpf_prog *prog) +{ + if (prog->jited) { + struct rv_jit_data *jit_data = prog->aux->jit_data; + struct bpf_binary_header *hdr; + + /* + * If we fail the final pass of JIT (from jit_subprogs), + * the program may not be finalized yet. Call finalize here + * before freeing it. + */ + if (jit_data) { + bpf_jit_binary_pack_finalize(prog, jit_data->ro_header, + jit_data->header); + kfree(jit_data); + } + hdr = bpf_jit_binary_pack_hdr(prog); + bpf_jit_binary_pack_free(hdr, NULL); + WARN_ON_ONCE(!bpf_prog_kallsyms_verify_off(prog)); + } + + bpf_prog_unlock_free(prog); +}