From patchwork Wed Apr 24 17:35:48 2024 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 8bit X-Patchwork-Submitter: Puranjay Mohan X-Patchwork-Id: 13642266 Received: from smtp.kernel.org (aws-us-west-2-korg-mail-1.web.codeaurora.org [10.30.226.201]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (No client certificate requested) by smtp.subspace.kernel.org (Postfix) with ESMTPS id DFD7D13777A; Wed, 24 Apr 2024 17:35:55 +0000 (UTC) Authentication-Results: smtp.subspace.kernel.org; arc=none smtp.client-ip=10.30.226.201 ARC-Seal: i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1713980156; cv=none; b=d8t+Hu+exXksoXn0q8+EYVstR/adm37kOq2LzdcLov2UPgwbL67YXo6e6tAtOSXsUgUH94x/z6o2upeS7x0K8DqpnupM4bfqUepr83wczX1jtnhPGdkq46izfvLHj+1mMveEk5NX6XnP+36inygWRWaU/E/LAqv6XSZcPYGQ/vI= ARC-Message-Signature: i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1713980156; c=relaxed/simple; bh=7HEn49881ZVhW2IJr2wiBxqLiYSA2beD/1tV1j+igLM=; h=From:To:Cc:Subject:Date:Message-Id:MIME-Version:Content-Type; b=UolbFMKAkMN6+MrqHfKYf+iESMrz16WsufUKlg3U+ZYAdDYTqV19U5lAZbfMu71h22pi8ssnWYK/qse2PRgQrB3e5zF8waq+I+ZnaKZAzzH/bPa+U6D673jKeKwJAXTr4UeGF/lw9UFkumyPb1Wdp2cvDHQIR862gsb7nact5sQ= ARC-Authentication-Results: i=1; smtp.subspace.kernel.org; dkim=pass (2048-bit key) header.d=kernel.org header.i=@kernel.org header.b=Hz/hkevx; arc=none smtp.client-ip=10.30.226.201 Authentication-Results: smtp.subspace.kernel.org; dkim=pass (2048-bit key) header.d=kernel.org header.i=@kernel.org header.b="Hz/hkevx" Received: by smtp.kernel.org (Postfix) with ESMTPSA id 5D231C113CD; Wed, 24 Apr 2024 17:35:55 +0000 (UTC) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/simple; d=kernel.org; s=k20201202; t=1713980155; bh=7HEn49881ZVhW2IJr2wiBxqLiYSA2beD/1tV1j+igLM=; h=From:To:Cc:Subject:Date:From; b=Hz/hkevxIPDYBVk5UVgdGfxxXhTcboKGa6o4tmIkB2+Fzbrp6LMr+/AJEL9yWzpzf tXJJw/JkzGPBddUjD5AXEH4feaQOX7jt8Wjxc3VyWoxmV425hBRXuHQdYj+kt/XKST I9NlUK3PsN0PsQXmndK8yJh2C0tZZbsxk42vUtGz/dQIgy3cd83u724X+N81PBCCAp utWPWhFCZFFfHCsiL37vglqZN78LbkwrLwKcdqfiD6U6gWtcWftOdiflwRqueQDPz4 QU6Yd2IPqoWqA3a7vY3BAi8qCndF3H7GbmOZ6X/nBI7uK2y46l0B6crNo/O6+9B09q noAATq+6+OTKA== From: Puranjay Mohan To: Catalin Marinas , Will Deacon , Alexei Starovoitov , Daniel Borkmann , Andrii Nakryiko , Martin KaFai Lau , Eduard Zingerman , Song Liu , Yonghong Song , John Fastabend , KP Singh , Stanislav Fomichev , Hao Luo , Jiri Olsa , Zi Shen Lim , Xu Kuohai , Florent Revest , linux-arm-kernel@lists.infradead.org, linux-kernel@vger.kernel.org, bpf@vger.kernel.org Cc: puranjay12@gmail.com Subject: [PATCH bpf-next v2 0/2] bpf, arm64: Support per-cpu instructions Date: Wed, 24 Apr 2024 17:35:48 +0000 Message-Id: <20240424173550.16359-1-puranjay@kernel.org> X-Mailer: git-send-email 2.40.1 Precedence: bulk X-Mailing-List: bpf@vger.kernel.org List-Id: List-Subscribe: List-Unsubscribe: MIME-Version: 1.0 X-Patchwork-Delegate: bpf@iogearbox.net Changes in v1 -> v2: v1: https://lore.kernel.org/all/20240405091707.66675-1-puranjay12@gmail.com/ - Add a patch to inline bpf_get_smp_processor_id() - Fix an issue in MRS instruction encoding as pointed out by Will - Remove CONFIG_SMP check This series adds the support of internal only per-CPU instructions and inlines the bpf_get_smp_processor_id() helper for ARM64 BPF JIT. Here is an example of bpf_get_smp_processor_id() and percpu_array_map_lookup_elem() before and after this series. BPF ===== BEFORE AFTER -------- ------- int cpu = bpf_get_smp_processor_id(); int cpu = bpf_get_smp_processor_id(); (85) call bpf_get_smp_processor_id#229032 (18) r0 = 0xffff800082072008 (bf) r0 = r0 (61) r0 = *(u32 *)(r0 +0) p = bpf_map_lookup_elem(map, &zero); p = bpf_map_lookup_elem(map, &zero); (18) r1 = map[id:78] (18) r1 = map[id:153] (18) r2 = map[id:82][0]+65536 (18) r2 = map[id:157][0]+65536 (85) call percpu_array_map_lookup_elem#313512 (07) r1 += 496 (61) r0 = *(u32 *)(r2 +0) (35) if r0 >= 0x1 goto pc+5 (67) r0 <<= 3 (0f) r0 += r1 (79) r0 = *(u64 *)(r0 +0) (bf) r0 = r0 (05) goto pc+1 (b7) r0 = 0 ARM64 JIT =========== BEFORE AFTER -------- ------- int cpu = bpf_get_smp_processor_id(); int cpu = bpf_get_smp_processor_id(); mov x10, #0xfffffffffffff4d0 mov x7, #0xffff8000ffffffff movk x10, #0x802b, lsl #16 movk x7, #0x8207, lsl #16 movk x10, #0x8000, lsl #32 movk x7, #0x2008 blr x10 mrs x10, tpidr_el1 add x7, x0, #0x0 add x7, x7, x10 ldr w7, [x7] p = bpf_map_lookup_elem(map, &zero); p = bpf_map_lookup_elem(map, &zero); mov x0, #0xffff0003ffffffff mov x0, #0xffff0003ffffffff movk x0, #0xce5c, lsl #16 movk x0, #0xe0f3, lsl #16 movk x0, #0xca00 movk x0, #0x7c00 mov x1, #0xffff8000ffffffff mov x1, #0xffff8000ffffffff movk x1, #0x8bdb, lsl #16 movk x1, #0xb0c7, lsl #16 movk x1, #0x6000 movk x1, #0xe000 mov x10, #0xffffffffffff3ed0 add x0, x0, #0x1f0 movk x10, #0x802d, lsl #16 ldr w7, [x1] movk x10, #0x8000, lsl #32 cmp x7, #0x1 blr x10 b.cs 0x0000000000000090 add x7, x0, #0x0 lsl x7, x7, #3 add x7, x7, x0 ldr x7, [x7] mrs x10, tpidr_el1 add x7, x7, x10 b 0x0000000000000094 mov x7, #0x0 Performance improvement found using benchmark[1] BEFORE AFTER -------- ------- glob-arr-inc : 23.817 ± 0.019M/s glob-arr-inc : 24.631 ± 0.027M/s arr-inc : 23.253 ± 0.019M/s arr-inc : 23.742 ± 0.023M/s hash-inc : 12.258 ± 0.010M/s hash-inc : 12.625 ± 0.004M/s [1] https://github.com/anakryiko/linux/commit/8dec900975ef Puranjay Mohan (2): arm64, bpf: add internal-only MOV instruction to resolve per-CPU addrs bpf, arm64: inline bpf_get_smp_processor_id() helper arch/arm64/include/asm/insn.h | 7 +++++++ arch/arm64/lib/insn.c | 11 +++++++++++ arch/arm64/net/bpf_jit.h | 6 ++++++ arch/arm64/net/bpf_jit_comp.c | 14 ++++++++++++++ kernel/bpf/verifier.c | 11 ++++++++++- 5 files changed, 48 insertions(+), 1 deletion(-)