From patchwork Sat Jan 25 02:17:50 2025 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Peilin Ye X-Patchwork-Id: 13950114 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from bombadil.infradead.org (bombadil.infradead.org [198.137.202.133]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (No client certificate requested) by smtp.lore.kernel.org (Postfix) with ESMTPS id A3D0AC0218B for ; Sat, 25 Jan 2025 02:20:16 +0000 (UTC) DKIM-Signature: v=1; a=rsa-sha256; q=dns/txt; c=relaxed/relaxed; d=lists.infradead.org; s=bombadil.20210309; h=Sender:List-Subscribe:List-Help :List-Post:List-Archive:List-Unsubscribe:List-Id:Content-Type:Cc:To:From: Subject:Message-ID:References:Mime-Version:In-Reply-To:Date:Reply-To: Content-Transfer-Encoding:Content-ID:Content-Description:Resent-Date: Resent-From:Resent-Sender:Resent-To:Resent-Cc:Resent-Message-ID:List-Owner; bh=SAhOVvwp95DURlQLkWceP8oOZonYxW56RQWdJ+tqdYo=; b=LrNOJ3YVQ9mZFaZTAsadVLLRan ZfcRW7JzWvuxssJYwEnlzUlUBWGk8jgJMeWHuQ3KySPC136hv4FzOyf5tKeaXEyqjuU+qQGfiKBiE ngKoLQDcIP1xROmzPeuIGKRknQZOdv5CgsNZvvPbSXcsG+Ko3NduZcJK5Hn1YzVpufhUSQG9+YZRV il8Ovv3WotJITHHeAwFy1JoSZ8feuojfUOUhsTC6XrjnNhzEZsJKLTQxEjOR/ldSLvw7lAK6wkQ9f YCh9yCczJqqolzU7IXoZyUmJOJF1+bS6fts9CTM82zc+US5LKW8KSLMiTFUhd8Wa2zICGAJiw0n8F OyJQH1HA==; Received: from localhost ([::1] helo=bombadil.infradead.org) by bombadil.infradead.org with esmtp (Exim 4.98 #2 (Red Hat Linux)) id 1tbVmH-0000000FuF8-1h0s; Sat, 25 Jan 2025 02:20:05 +0000 Received: from mail-pj1-x104a.google.com ([2607:f8b0:4864:20::104a]) by bombadil.infradead.org with esmtps (Exim 4.98 #2 (Red Hat Linux)) id 1tbVkP-0000000Ftrd-0UM0 for linux-arm-kernel@lists.infradead.org; Sat, 25 Jan 2025 02:18:10 +0000 Received: by mail-pj1-x104a.google.com with SMTP id 98e67ed59e1d1-2ef80d30df1so5030593a91.1 for ; Fri, 24 Jan 2025 18:18:08 -0800 (PST) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=google.com; s=20230601; t=1737771487; x=1738376287; darn=lists.infradead.org; h=cc:to:from:subject:message-id:references:mime-version:in-reply-to :date:from:to:cc:subject:date:message-id:reply-to; bh=SAhOVvwp95DURlQLkWceP8oOZonYxW56RQWdJ+tqdYo=; b=BoW85YMS1nBjYbjUlB3JR4hkVBD/gdZKmNJlPoeB6KlSD7ftoVWIXEbDYw55jwfCB0 F1/IMxwIsLCeFV4wMCy6AshV46w9ud/sb04ed7+HAgyOsy+M6HfNjG1Sg4XbOJxHfwmP 0KrHkm/uJuAAWLpL2opPJVCn8KdIiTjbJIMo5Sy5lYVo2uEw6ZbjFoGcmved9N0N7rTk v90fXl52MzVtlPWdfvLXQjuWhqsXmBWAUa2ow9rIV7FDYrRSikgQ1FP1AAsta31pLtpa tJaH6Qi6Hnm2un9SC5rUSgYwM/+7ZBFuoDPhmTbI+nzfGidUGFySZ5Mv+wMhL3t6AWH9 rZEw== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20230601; t=1737771487; x=1738376287; h=cc:to:from:subject:message-id:references:mime-version:in-reply-to :date:x-gm-message-state:from:to:cc:subject:date:message-id:reply-to; bh=SAhOVvwp95DURlQLkWceP8oOZonYxW56RQWdJ+tqdYo=; b=PORxVIujZTNKsTBAPMBnibYtcBuNh3mVzAnKzzUZJqsOx7vwmtif6zE3nsmstnDn5D dBPwZLN8hf4u9p9e81h1dJbhDcge4BCCjvGawGoHKbKAemBVp7EdDCKQQYQpFH0VDfA8 8Ze1x4cTzTIrHtpJv1JCLzdBe6u3LddqRDKXI/OpR6O5jmqPkD7Ekze4WKkNkVahjDX9 7x8mob5AQIZYXsrYy/w3eg1pVRmWvJw/8L6cC/oZP2jsapbZaG2tw3rW1AOtuhkQiZrP uGE3l1VhheFOvdWYHTjyr8rITZaDksPPftYUXOA1eNiJ5xQESM3tyrJ+jixMTcQxn8Wq MwQA== X-Forwarded-Encrypted: i=1; AJvYcCV6m4claEzqaMwcrCZzXLjag95U6TCQsVO9fKkNB4ssLLvyM1FRvhNDUwpJiz46Da3zVpGXp+FzaoxyxTYFw2as@lists.infradead.org X-Gm-Message-State: AOJu0Yz+RG2QKMbp9+O2Z3fCudo1r6aOnn9hi778ZWx7whPnUu8GjdgC n4TJmWWYFWyLqxQaD5oifShvJqARt2NQCXWTcP8Lmmm/4J7XiKT1EWuS6vv43GvhZJXC/+kUvwy BpQnEmZXYAA== X-Google-Smtp-Source: AGHT+IH+1cbrPKWyykDgl/vF7cElMEjsc9SSUcFXj1Np6Z8pLDvoFHDe47yUm55K1xK5bhBJYeqphCuyAJpw0w== X-Received: from pfbbx20.prod.google.com ([2002:a05:6a00:4294:b0:729:14f9:2f50]) (user=yepeilin job=prod-delivery.src-stubby-dispatcher) by 2002:a05:6a00:2e24:b0:728:8c17:127d with SMTP id d2e1a72fcca58-72daf948568mr52769436b3a.8.1737771487369; Fri, 24 Jan 2025 18:18:07 -0800 (PST) Date: Sat, 25 Jan 2025 02:17:50 +0000 In-Reply-To: Mime-Version: 1.0 References: X-Mailer: git-send-email 2.48.1.262.g85cc9f2d1e-goog Message-ID: Subject: [PATCH bpf-next v1 1/8] bpf/verifier: Factor out atomic_ptr_type_ok() From: Peilin Ye To: bpf@vger.kernel.org, linux-arm-kernel@lists.infradead.org Cc: Peilin Ye , bpf@ietf.org, Xu Kuohai , Eduard Zingerman , David Vernet , Alexei Starovoitov , Daniel Borkmann , Andrii Nakryiko , Martin KaFai Lau , Song Liu , Yonghong Song , John Fastabend , KP Singh , Stanislav Fomichev , Hao Luo , Jiri Olsa , Jonathan Corbet , "Paul E. McKenney" , Puranjay Mohan , Catalin Marinas , Will Deacon , Quentin Monnet , Mykola Lysenko , Shuah Khan , Josh Don , Barret Rhoden , Neel Natu , Benjamin Segall , linux-kernel@vger.kernel.org X-CRM114-Version: 20100106-BlameMichelson ( TRE 0.8.0 (BSD) ) MR-646709E3 X-CRM114-CacheID: sfid-20250124_181809_155411_322B30BC X-CRM114-Status: GOOD ( 12.75 ) X-BeenThere: linux-arm-kernel@lists.infradead.org X-Mailman-Version: 2.1.34 Precedence: list List-Id: List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Sender: "linux-arm-kernel" Errors-To: linux-arm-kernel-bounces+linux-arm-kernel=archiver.kernel.org@lists.infradead.org Factor out atomic_ptr_type_ok() as a helper function to be used later. Signed-off-by: Peilin Ye --- kernel/bpf/verifier.c | 26 +++++++++++++++++++++----- 1 file changed, 21 insertions(+), 5 deletions(-) diff --git a/kernel/bpf/verifier.c b/kernel/bpf/verifier.c index 9971c03adfd5..0935b72fe716 100644 --- a/kernel/bpf/verifier.c +++ b/kernel/bpf/verifier.c @@ -6116,6 +6116,26 @@ static bool is_arena_reg(struct bpf_verifier_env *env, int regno) return reg->type == PTR_TO_ARENA; } +/* Return false if @regno contains a pointer whose type isn't supported for + * atomic instruction @insn. + */ +static bool atomic_ptr_type_ok(struct bpf_verifier_env *env, int regno, + struct bpf_insn *insn) +{ + if (is_ctx_reg(env, regno)) + return false; + if (is_pkt_reg(env, regno)) + return false; + if (is_flow_key_reg(env, regno)) + return false; + if (is_sk_reg(env, regno)) + return false; + if (is_arena_reg(env, regno)) + return bpf_jit_supports_insn(insn, true); + + return true; +} + static u32 *reg2btf_ids[__BPF_REG_TYPE_MAX] = { #ifdef CONFIG_NET [PTR_TO_SOCKET] = &btf_sock_ids[BTF_SOCK_TYPE_SOCK], @@ -7572,11 +7592,7 @@ static int check_atomic(struct bpf_verifier_env *env, int insn_idx, struct bpf_i return -EACCES; } - if (is_ctx_reg(env, insn->dst_reg) || - is_pkt_reg(env, insn->dst_reg) || - is_flow_key_reg(env, insn->dst_reg) || - is_sk_reg(env, insn->dst_reg) || - (is_arena_reg(env, insn->dst_reg) && !bpf_jit_supports_insn(insn, true))) { + if (!atomic_ptr_type_ok(env, insn->dst_reg, insn)) { verbose(env, "BPF_ATOMIC stores into R%d %s is not allowed\n", insn->dst_reg, reg_type_str(env, reg_state(env, insn->dst_reg)->type)); From patchwork Sat Jan 25 02:18:17 2025 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Peilin Ye X-Patchwork-Id: 13950115 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from bombadil.infradead.org (bombadil.infradead.org [198.137.202.133]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (No client certificate requested) by smtp.lore.kernel.org (Postfix) with ESMTPS id 72889C02181 for ; Sat, 25 Jan 2025 02:21:36 +0000 (UTC) DKIM-Signature: v=1; a=rsa-sha256; q=dns/txt; c=relaxed/relaxed; d=lists.infradead.org; s=bombadil.20210309; h=Sender:List-Subscribe:List-Help :List-Post:List-Archive:List-Unsubscribe:List-Id:Content-Type:Cc:To:From: Subject:Message-ID:References:Mime-Version:In-Reply-To:Date:Reply-To: Content-Transfer-Encoding:Content-ID:Content-Description:Resent-Date: Resent-From:Resent-Sender:Resent-To:Resent-Cc:Resent-Message-ID:List-Owner; bh=GIRUjVUPGvvoC/G+tLqvrHi6mwo3DeIvXXHsctD0qOI=; b=dFvYZkXl88QhSc6mfTJHx/5HsV PtwKOlGVVzMjsdOdtgdGY9AnsQFuoPNFm3+i9u1QT/q4z6hZw152OVYvgn9IMp8ezCAcisxIX5srj aqqbk7eUoKbMHKKC2BQP+2XK8nB9tIHDcVrD93B0+fGmE5noDIb/MMVxTj6h79iGvtP9tt6NCIxmn T2jVyeY6Eg7gsP+A2MdKKxdNF4WkEEajuMXSfcMVk7fZPZgSLzRCzJtcjG2xv/9WVYz33+qgw8TG9 7tck/PENU4+Gp0qVpQG4JvaSiyeAS741Wrj9mbuUCP3+ESnMT6LmzlMM1TQlvJT+jgVQZl1mWkcPl 5W3ojvZg==; Received: from localhost ([::1] helo=bombadil.infradead.org) by bombadil.infradead.org with esmtp (Exim 4.98 #2 (Red Hat Linux)) id 1tbVnX-0000000FuNV-0Hf4; Sat, 25 Jan 2025 02:21:23 +0000 Received: from mail-pl1-x64a.google.com ([2607:f8b0:4864:20::64a]) by bombadil.infradead.org with esmtps (Exim 4.98 #2 (Red Hat Linux)) id 1tbVke-0000000FttX-13xD for linux-arm-kernel@lists.infradead.org; Sat, 25 Jan 2025 02:18:26 +0000 Received: by mail-pl1-x64a.google.com with SMTP id d9443c01a7336-2162f80040aso47775365ad.1 for ; Fri, 24 Jan 2025 18:18:23 -0800 (PST) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=google.com; s=20230601; t=1737771502; x=1738376302; darn=lists.infradead.org; h=cc:to:from:subject:message-id:references:mime-version:in-reply-to :date:from:to:cc:subject:date:message-id:reply-to; bh=GIRUjVUPGvvoC/G+tLqvrHi6mwo3DeIvXXHsctD0qOI=; b=I9uahkWoam6CLaTtjXCqpvmEiV4DTKBfqFPk7ME/aOV+6wElkj5aT/GFYhK+j4XJo3 ZNw21cLJ6XB1Og1auS4rerrynyMOeSJR640QC/XdBHSrwHPJxJsoJbLCDgC1ofgL1Jul xfZlDLCDAZy07wQw+LfS/2+9nFkTdrLwzPWF0+aMnHU0KYnD4qYNSBcsW8246dXfEMtR p/jB3bBkfvacGPCVUz16Nsahy3wXJ5jpZ/G0mnT3SYy7mxflrsIVDW8zbDeqF73sFMJ9 zAKB0w1lZPz12+f/sWYO/jWHiYjBmAdwDLaXCPgZY4GDPaGQU6UQH2lyNyHbAXVnyh+s Usew== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20230601; t=1737771502; x=1738376302; h=cc:to:from:subject:message-id:references:mime-version:in-reply-to :date:x-gm-message-state:from:to:cc:subject:date:message-id:reply-to; bh=GIRUjVUPGvvoC/G+tLqvrHi6mwo3DeIvXXHsctD0qOI=; b=pn1psA7NsPxROvtcOjNGnleNFfFwhIei+jZXsmZXwW5pyAU4YopVkWYg8/0U68oNku jQFxyKukV4cpVKJRxohBCUGkPhGh5m+nmeTFYsOFG5C3ysbUXu4ZhNmcDeEDIou4cgMq NhUyCcPnbSnWkAvyjDSbVlG2LCnu0p6hj5bsnXOm407LNOPfDFjWls5bL5Nu5TDmjqxO jPoH+ADw3rXBHeoFGPS12IKa4ut/cy50gbI+7YTUj0eBE2aEtCxN4pe6VZa/XqNdjjUc eRisYoPSZtBJZsviMmEX+7R+esYCGRlhHvaPSTH8m9sHLInWWdvf4lvW31OXv81i/DLO c5pw== X-Forwarded-Encrypted: i=1; AJvYcCVhKRr+fcdw2/pwDDovdA4VlThATMLZtOGt703fGqT8Gc0fpb2h0lAGOtpfyJKhmgYiTZlnntcUQvBvCof0Fccv@lists.infradead.org X-Gm-Message-State: AOJu0Yww0b/tHy9SPJ9HgAzRqlkkjTzuVVZCBbLFa5dxhxkrKkbM8Vwd sR7UaqptiYC0CjW79xxMs0b/p4u7/nx6yfDe04jmwbsOPZ+yZgKaVjWMjEfWi6WHOJgN00guhtb F2huEi1LYjw== X-Google-Smtp-Source: AGHT+IHg0VQQH90pQuEfyGGIfiVpVl5OiI/iOGIaKUaWYcp8ziT8hehv5mLQYN4xTkvy2RPykSchBVyc1gr7cw== X-Received: from plxd18.prod.google.com ([2002:a17:902:ef12:b0:216:5566:3c11]) (user=yepeilin job=prod-delivery.src-stubby-dispatcher) by 2002:a17:902:e751:b0:215:9d29:9724 with SMTP id d9443c01a7336-21c355c6146mr550619555ad.38.1737771502432; Fri, 24 Jan 2025 18:18:22 -0800 (PST) Date: Sat, 25 Jan 2025 02:18:17 +0000 In-Reply-To: Mime-Version: 1.0 References: X-Mailer: git-send-email 2.48.1.262.g85cc9f2d1e-goog Message-ID: <4831a11659532873c20f0c693c248e5cefaddcbf.1737763916.git.yepeilin@google.com> Subject: [PATCH bpf-next v1 2/8] bpf/verifier: Factor out check_atomic_rmw() From: Peilin Ye To: bpf@vger.kernel.org, linux-arm-kernel@lists.infradead.org Cc: Peilin Ye , bpf@ietf.org, Xu Kuohai , Eduard Zingerman , David Vernet , Alexei Starovoitov , Daniel Borkmann , Andrii Nakryiko , Martin KaFai Lau , Song Liu , Yonghong Song , John Fastabend , KP Singh , Stanislav Fomichev , Hao Luo , Jiri Olsa , Jonathan Corbet , "Paul E. McKenney" , Puranjay Mohan , Catalin Marinas , Will Deacon , Quentin Monnet , Mykola Lysenko , Shuah Khan , Josh Don , Barret Rhoden , Neel Natu , Benjamin Segall , linux-kernel@vger.kernel.org X-CRM114-Version: 20100106-BlameMichelson ( TRE 0.8.0 (BSD) ) MR-646709E3 X-CRM114-CacheID: sfid-20250124_181824_293927_7310C04D X-CRM114-Status: GOOD ( 12.67 ) X-BeenThere: linux-arm-kernel@lists.infradead.org X-Mailman-Version: 2.1.34 Precedence: list List-Id: List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Sender: "linux-arm-kernel" Errors-To: linux-arm-kernel-bounces+linux-arm-kernel=archiver.kernel.org@lists.infradead.org Currently, check_atomic() only handles atomic read-modify-write (RMW) instructions. Since we are planning to introduce other types of atomic instructions (i.e., atomic load/store), extract the existing RMW handling logic into its own function named check_atomic_rmw(). Signed-off-by: Peilin Ye --- kernel/bpf/verifier.c | 40 ++++++++++++++++++++++------------------ 1 file changed, 22 insertions(+), 18 deletions(-) diff --git a/kernel/bpf/verifier.c b/kernel/bpf/verifier.c index 0935b72fe716..2b3caa7549af 100644 --- a/kernel/bpf/verifier.c +++ b/kernel/bpf/verifier.c @@ -7536,28 +7536,12 @@ static int check_mem_access(struct bpf_verifier_env *env, int insn_idx, u32 regn static int save_aux_ptr_type(struct bpf_verifier_env *env, enum bpf_reg_type type, bool allow_trust_mismatch); -static int check_atomic(struct bpf_verifier_env *env, int insn_idx, struct bpf_insn *insn) +static int check_atomic_rmw(struct bpf_verifier_env *env, int insn_idx, + struct bpf_insn *insn) { int load_reg; int err; - switch (insn->imm) { - case BPF_ADD: - case BPF_ADD | BPF_FETCH: - case BPF_AND: - case BPF_AND | BPF_FETCH: - case BPF_OR: - case BPF_OR | BPF_FETCH: - case BPF_XOR: - case BPF_XOR | BPF_FETCH: - case BPF_XCHG: - case BPF_CMPXCHG: - break; - default: - verbose(env, "BPF_ATOMIC uses invalid atomic opcode %02x\n", insn->imm); - return -EINVAL; - } - if (BPF_SIZE(insn->code) != BPF_W && BPF_SIZE(insn->code) != BPF_DW) { verbose(env, "invalid atomic operand size\n"); return -EINVAL; @@ -7641,6 +7625,26 @@ static int check_atomic(struct bpf_verifier_env *env, int insn_idx, struct bpf_i return 0; } +static int check_atomic(struct bpf_verifier_env *env, int insn_idx, struct bpf_insn *insn) +{ + switch (insn->imm) { + case BPF_ADD: + case BPF_ADD | BPF_FETCH: + case BPF_AND: + case BPF_AND | BPF_FETCH: + case BPF_OR: + case BPF_OR | BPF_FETCH: + case BPF_XOR: + case BPF_XOR | BPF_FETCH: + case BPF_XCHG: + case BPF_CMPXCHG: + return check_atomic_rmw(env, insn_idx, insn); + default: + verbose(env, "BPF_ATOMIC uses invalid atomic opcode %02x\n", insn->imm); + return -EINVAL; + } +} + /* When register 'regno' is used to read the stack (either directly or through * a helper function) make sure that it's within stack boundary and, depending * on the access type and privileges, that all elements of the stack are From patchwork Sat Jan 25 02:18:26 2025 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Peilin Ye X-Patchwork-Id: 13950116 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from bombadil.infradead.org (bombadil.infradead.org [198.137.202.133]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (No client certificate requested) by smtp.lore.kernel.org (Postfix) with ESMTPS id 8ECE7C02181 for ; Sat, 25 Jan 2025 02:22:52 +0000 (UTC) DKIM-Signature: v=1; a=rsa-sha256; q=dns/txt; c=relaxed/relaxed; d=lists.infradead.org; s=bombadil.20210309; h=Sender:List-Subscribe:List-Help :List-Post:List-Archive:List-Unsubscribe:List-Id:Content-Type:Cc:To:From: Subject:Message-ID:References:Mime-Version:In-Reply-To:Date:Reply-To: Content-Transfer-Encoding:Content-ID:Content-Description:Resent-Date: Resent-From:Resent-Sender:Resent-To:Resent-Cc:Resent-Message-ID:List-Owner; bh=QfXz68+jbykT93+Op9h9AP72g+l2XAcrP0ZoIG9EpCQ=; b=3jutRPBOplYyWP+cfVw5J3xy2u Kec+gFsfwOPylXHLTzJzKRbWTPpEX9G9r8VG3yx2dAZeDn67xEkq4/MDlrOBIp8YDleSFwd0akWFG YxYmfBnf7WcCkXOF3h3KfJzQ4j66jDn0kkSvamQSSS6KyzdzXOwLHDp1EJr7dk59ZP1zn9Xb3GljV DoLTJlF26gCipuEcX3pWCBJHQZHJVSnMLiOJPm1wP3qEUgEr2Zem82dM03qJxLFgMH9vRfsoMY0t/ /pZJi7p/9YNrKuwpgSa3R1/IO0X4WYE7Nmkx/NFHLBJvFYc/3o+GT2TaGzFxPTmpuWIV2A7Wt9amS /n5LfVZw==; Received: from localhost ([::1] helo=bombadil.infradead.org) by bombadil.infradead.org with esmtp (Exim 4.98 #2 (Red Hat Linux)) id 1tbVon-0000000FuUA-2nW8; Sat, 25 Jan 2025 02:22:41 +0000 Received: from mail-pl1-x64a.google.com ([2607:f8b0:4864:20::64a]) by bombadil.infradead.org with esmtps (Exim 4.98 #2 (Red Hat Linux)) id 1tbVkn-0000000FtvJ-1aUK for linux-arm-kernel@lists.infradead.org; Sat, 25 Jan 2025 02:18:35 +0000 Received: by mail-pl1-x64a.google.com with SMTP id d9443c01a7336-21661949f23so77647635ad.3 for ; Fri, 24 Jan 2025 18:18:32 -0800 (PST) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=google.com; s=20230601; t=1737771512; x=1738376312; darn=lists.infradead.org; h=cc:to:from:subject:message-id:references:mime-version:in-reply-to :date:from:to:cc:subject:date:message-id:reply-to; bh=QfXz68+jbykT93+Op9h9AP72g+l2XAcrP0ZoIG9EpCQ=; b=ZK44USf0+kgax+rfv8MGC+NpgNl68aG9CRTS1jQCXxjCxhu/9/kfPDRXKwRxeVidd3 AjXwVPc1CwSTtAsf+O+Hvk1C8alRVgRrhJnbTgDjyZ7/6PQ1HIzMwBfYURtzy1AEtRlN IVb4KyjYiCWOOQ6lnMZpi5vHjIx8bwuwfa3E/8VZFHYo+fyIjvx2nivH0DQAAuHc0NnG pq0lkLKkC2r7cIWWw1MCyJ7c8hP9S0gvVj9ryrnYY+Pl6gSrWKTahSFH26AVksuwqIIj xGqVH6U3VQrv/8Cp6fMgaoci9DBotO/ZAeVonOh/xiX5jxowM85adGz0NVO0gQ9Ba0P8 OUUA== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20230601; t=1737771512; x=1738376312; h=cc:to:from:subject:message-id:references:mime-version:in-reply-to :date:x-gm-message-state:from:to:cc:subject:date:message-id:reply-to; bh=QfXz68+jbykT93+Op9h9AP72g+l2XAcrP0ZoIG9EpCQ=; b=JhnE1gZZx7yvalKieOj98dArsj2UboAnqlviwKBlXBN8GfZYn2AQSWcunXZ+T0p6O4 TCsBBRnOYoq4vn1cHEyxWunuJwzHY9kOc8qqhVG3xlaTJLvw+s0cW7/mZhRAwC4atm61 F0c1uGh74rcDAbWXzThn/Xu1uGlvCdasa+EFO7nZVzrOa/Hw/gZdLnit9zGgA/zRvJQc tMkywCupPnIQshANMvNP0ykcCWxaQSKdji4dBYlsl/a4HzBX2Vb2MwtOpBJCC8LjIoaB albZpMOrlntDgG1yKIFHVEZc4vJE4DJKXZK9HzPTk8qeid5HKkVZ3/mGJVsPjD2usMLZ SVRw== X-Forwarded-Encrypted: i=1; AJvYcCWZibbnf569eRpOOpr6CdoTSJpE1Cd01lMSbP1KvSOwt8iMapwCrxchTgqAtPKMU91PCQmDSjM/Ndm0QPBHALIp@lists.infradead.org X-Gm-Message-State: AOJu0Yxl65uH8b178plgWEnpdVT+DHe0wqKpQqpX5PleTicX8LEpzaot tm73Qteiq2PNzgXCv95LafQygunOY5CqIJITp3TBIXZjeBXBysP/qJbPvr6azgkew3GeluO62zY FlpKXRrrH0w== X-Google-Smtp-Source: AGHT+IGZQK7/pNts6mhkSIhk0pPUdISf5PSC4pmQWwsrIxHaXQTvoanEm5EYdjLznU9XH91CE3rnm6GOlT11cA== X-Received: from pldt14.prod.google.com ([2002:a17:903:40ce:b0:20c:5d5a:9d64]) (user=yepeilin job=prod-delivery.src-stubby-dispatcher) by 2002:a17:902:d4d2:b0:215:94eb:adb6 with SMTP id d9443c01a7336-21c3563c51amr523357365ad.40.1737771512513; Fri, 24 Jan 2025 18:18:32 -0800 (PST) Date: Sat, 25 Jan 2025 02:18:26 +0000 In-Reply-To: Mime-Version: 1.0 References: X-Mailer: git-send-email 2.48.1.262.g85cc9f2d1e-goog Message-ID: Subject: [PATCH bpf-next v1 3/8] bpf: Introduce load-acquire and store-release instructions From: Peilin Ye To: bpf@vger.kernel.org, linux-arm-kernel@lists.infradead.org Cc: Peilin Ye , bpf@ietf.org, Xu Kuohai , Eduard Zingerman , David Vernet , Alexei Starovoitov , Daniel Borkmann , Andrii Nakryiko , Martin KaFai Lau , Song Liu , Yonghong Song , John Fastabend , KP Singh , Stanislav Fomichev , Hao Luo , Jiri Olsa , Jonathan Corbet , "Paul E. McKenney" , Puranjay Mohan , Catalin Marinas , Will Deacon , Quentin Monnet , Mykola Lysenko , Shuah Khan , Josh Don , Barret Rhoden , Neel Natu , Benjamin Segall , linux-kernel@vger.kernel.org X-CRM114-Version: 20100106-BlameMichelson ( TRE 0.8.0 (BSD) ) MR-646709E3 X-CRM114-CacheID: sfid-20250124_181833_420354_A19D76E9 X-CRM114-Status: GOOD ( 21.71 ) X-BeenThere: linux-arm-kernel@lists.infradead.org X-Mailman-Version: 2.1.34 Precedence: list List-Id: List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Sender: "linux-arm-kernel" Errors-To: linux-arm-kernel-bounces+linux-arm-kernel=archiver.kernel.org@lists.infradead.org Introduce BPF instructions with load-acquire and store-release semantics, as discussed in [1]. The following new flags are defined: BPF_ATOMIC_LOAD 0x10 BPF_ATOMIC_STORE 0x20 BPF_ATOMIC_TYPE(imm) ((imm) & 0xf0) BPF_RELAXED 0x0 BPF_ACQUIRE 0x1 BPF_RELEASE 0x2 BPF_ACQ_REL 0x3 BPF_SEQ_CST 0x4 BPF_LOAD_ACQ (BPF_ATOMIC_LOAD | BPF_ACQUIRE) BPF_STORE_REL (BPF_ATOMIC_STORE | BPF_RELEASE) A "load-acquire" is a BPF_STX | BPF_ATOMIC instruction with the 'imm' field set to BPF_LOAD_ACQ (0x11). Similarly, a "store-release" is a BPF_STX | BPF_ATOMIC instruction with the 'imm' field set to BPF_STORE_REL (0x22). Unlike existing atomic operations that only support BPF_W (32-bit) and BPF_DW (64-bit) size modifiers, load-acquires and store-releases also support BPF_B (8-bit) and BPF_H (16-bit). An 8- or 16-bit load-acquire zero-extends the value before writing it to a 32-bit register, just like ARM64 instruction LDARH and friends. As an example, consider the following 64-bit load-acquire BPF instruction: db 10 00 00 11 00 00 00 r0 = load_acquire((u64 *)(r1 + 0x0)) opcode (0xdb): BPF_ATOMIC | BPF_DW | BPF_STX imm (0x00000011): BPF_LOAD_ACQ Similarly, a 16-bit BPF store-release: cb 21 00 00 22 00 00 00 store_release((u16 *)(r1 + 0x0), w2) opcode (0xcb): BPF_ATOMIC | BPF_H | BPF_STX imm (0x00000022): BPF_STORE_REL [1] https://lore.kernel.org/all/20240729183246.4110549-1-yepeilin@google.com/ Signed-off-by: Peilin Ye Acked-by: Eduard Zingerman --- include/linux/filter.h | 2 + include/uapi/linux/bpf.h | 13 +++++ kernel/bpf/core.c | 41 +++++++++++++- kernel/bpf/disasm.c | 12 +++++ kernel/bpf/verifier.c | 99 ++++++++++++++++++++++++++++++++-- tools/include/uapi/linux/bpf.h | 13 +++++ 6 files changed, 175 insertions(+), 5 deletions(-) diff --git a/include/linux/filter.h b/include/linux/filter.h index a3ea46281595..e36812a5b01f 100644 --- a/include/linux/filter.h +++ b/include/linux/filter.h @@ -364,6 +364,8 @@ static inline bool insn_is_cast_user(const struct bpf_insn *insn) * BPF_XOR | BPF_FETCH src_reg = atomic_fetch_xor(dst_reg + off16, src_reg); * BPF_XCHG src_reg = atomic_xchg(dst_reg + off16, src_reg) * BPF_CMPXCHG r0 = atomic_cmpxchg(dst_reg + off16, r0, src_reg) + * BPF_LOAD_ACQ dst_reg = smp_load_acquire(src_reg + off16) + * BPF_STORE_REL smp_store_release(dst_reg + off16, src_reg) */ #define BPF_ATOMIC_OP(SIZE, OP, DST, SRC, OFF) \ diff --git a/include/uapi/linux/bpf.h b/include/uapi/linux/bpf.h index 2acf9b336371..4a20a125eb46 100644 --- a/include/uapi/linux/bpf.h +++ b/include/uapi/linux/bpf.h @@ -51,6 +51,19 @@ #define BPF_XCHG (0xe0 | BPF_FETCH) /* atomic exchange */ #define BPF_CMPXCHG (0xf0 | BPF_FETCH) /* atomic compare-and-write */ +#define BPF_ATOMIC_LOAD 0x10 +#define BPF_ATOMIC_STORE 0x20 +#define BPF_ATOMIC_TYPE(imm) ((imm) & 0xf0) + +#define BPF_RELAXED 0x00 +#define BPF_ACQUIRE 0x01 +#define BPF_RELEASE 0x02 +#define BPF_ACQ_REL 0x03 +#define BPF_SEQ_CST 0x04 + +#define BPF_LOAD_ACQ (BPF_ATOMIC_LOAD | BPF_ACQUIRE) /* load-acquire */ +#define BPF_STORE_REL (BPF_ATOMIC_STORE | BPF_RELEASE) /* store-release */ + enum bpf_cond_pseudo_jmp { BPF_MAY_GOTO = 0, }; diff --git a/kernel/bpf/core.c b/kernel/bpf/core.c index da729cbbaeb9..ab082ab9d535 100644 --- a/kernel/bpf/core.c +++ b/kernel/bpf/core.c @@ -1663,14 +1663,17 @@ EXPORT_SYMBOL_GPL(__bpf_call_base); INSN_3(JMP, JSET, K), \ INSN_2(JMP, JA), \ INSN_2(JMP32, JA), \ + /* Atomic operations. */ \ + INSN_3(STX, ATOMIC, B), \ + INSN_3(STX, ATOMIC, H), \ + INSN_3(STX, ATOMIC, W), \ + INSN_3(STX, ATOMIC, DW), \ /* Store instructions. */ \ /* Register based. */ \ INSN_3(STX, MEM, B), \ INSN_3(STX, MEM, H), \ INSN_3(STX, MEM, W), \ INSN_3(STX, MEM, DW), \ - INSN_3(STX, ATOMIC, W), \ - INSN_3(STX, ATOMIC, DW), \ /* Immediate based. */ \ INSN_3(ST, MEM, B), \ INSN_3(ST, MEM, H), \ @@ -2169,6 +2172,8 @@ static u64 ___bpf_prog_run(u64 *regs, const struct bpf_insn *insn) STX_ATOMIC_DW: STX_ATOMIC_W: + STX_ATOMIC_H: + STX_ATOMIC_B: switch (IMM) { ATOMIC_ALU_OP(BPF_ADD, add) ATOMIC_ALU_OP(BPF_AND, and) @@ -2196,6 +2201,38 @@ static u64 ___bpf_prog_run(u64 *regs, const struct bpf_insn *insn) (atomic64_t *)(unsigned long) (DST + insn->off), (u64) BPF_R0, (u64) SRC); break; + case BPF_LOAD_ACQ: + switch (BPF_SIZE(insn->code)) { +#define LOAD_ACQUIRE(SIZEOP, SIZE) \ + case BPF_##SIZEOP: \ + DST = (SIZE)smp_load_acquire( \ + (SIZE *)(unsigned long)(SRC + insn->off)); \ + break; + LOAD_ACQUIRE(B, u8) + LOAD_ACQUIRE(H, u16) + LOAD_ACQUIRE(W, u32) + LOAD_ACQUIRE(DW, u64) +#undef LOAD_ACQUIRE + default: + goto default_label; + } + break; + case BPF_STORE_REL: + switch (BPF_SIZE(insn->code)) { +#define STORE_RELEASE(SIZEOP, SIZE) \ + case BPF_##SIZEOP: \ + smp_store_release( \ + (SIZE *)(unsigned long)(DST + insn->off), (SIZE)SRC); \ + break; + STORE_RELEASE(B, u8) + STORE_RELEASE(H, u16) + STORE_RELEASE(W, u32) + STORE_RELEASE(DW, u64) +#undef STORE_RELEASE + default: + goto default_label; + } + break; default: goto default_label; diff --git a/kernel/bpf/disasm.c b/kernel/bpf/disasm.c index 309c4aa1b026..974d172d6735 100644 --- a/kernel/bpf/disasm.c +++ b/kernel/bpf/disasm.c @@ -267,6 +267,18 @@ void print_bpf_insn(const struct bpf_insn_cbs *cbs, BPF_SIZE(insn->code) == BPF_DW ? "64" : "", bpf_ldst_string[BPF_SIZE(insn->code) >> 3], insn->dst_reg, insn->off, insn->src_reg); + } else if (BPF_MODE(insn->code) == BPF_ATOMIC && + insn->imm == BPF_LOAD_ACQ) { + verbose(cbs->private_data, "(%02x) r%d = load_acquire((%s *)(r%d %+d))\n", + insn->code, insn->dst_reg, + bpf_ldst_string[BPF_SIZE(insn->code) >> 3], + insn->src_reg, insn->off); + } else if (BPF_MODE(insn->code) == BPF_ATOMIC && + insn->imm == BPF_STORE_REL) { + verbose(cbs->private_data, "(%02x) store_release((%s *)(r%d %+d), r%d)\n", + insn->code, + bpf_ldst_string[BPF_SIZE(insn->code) >> 3], + insn->dst_reg, insn->off, insn->src_reg); } else { verbose(cbs->private_data, "BUG_%02x\n", insn->code); } diff --git a/kernel/bpf/verifier.c b/kernel/bpf/verifier.c index 2b3caa7549af..e44abe22aac9 100644 --- a/kernel/bpf/verifier.c +++ b/kernel/bpf/verifier.c @@ -579,6 +579,13 @@ static bool is_cmpxchg_insn(const struct bpf_insn *insn) insn->imm == BPF_CMPXCHG; } +static bool is_atomic_load_insn(const struct bpf_insn *insn) +{ + return BPF_CLASS(insn->code) == BPF_STX && + BPF_MODE(insn->code) == BPF_ATOMIC && + BPF_ATOMIC_TYPE(insn->imm) == BPF_ATOMIC_LOAD; +} + static int __get_spi(s32 off) { return (-off - 1) / BPF_REG_SIZE; @@ -3481,7 +3488,7 @@ static bool is_reg64(struct bpf_verifier_env *env, struct bpf_insn *insn, } if (class == BPF_STX) { - /* BPF_STX (including atomic variants) has multiple source + /* BPF_STX (including atomic variants) has one or more source * operands, one of which is a ptr. Check whether the caller is * asking about it. */ @@ -4095,7 +4102,7 @@ static int backtrack_insn(struct bpf_verifier_env *env, int idx, int subseq_idx, * dreg still needs precision before this insn */ } - } else if (class == BPF_LDX) { + } else if (class == BPF_LDX || is_atomic_load_insn(insn)) { if (!bt_is_reg_set(bt, dreg)) return 0; bt_clear_reg(bt, dreg); @@ -7625,6 +7632,86 @@ static int check_atomic_rmw(struct bpf_verifier_env *env, int insn_idx, return 0; } +static int check_atomic_load(struct bpf_verifier_env *env, int insn_idx, + struct bpf_insn *insn) +{ + struct bpf_reg_state *regs = cur_regs(env); + int err; + + err = check_reg_arg(env, insn->src_reg, SRC_OP); + if (err) + return err; + + err = check_reg_arg(env, insn->dst_reg, DST_OP_NO_MARK); + if (err) + return err; + + if (!atomic_ptr_type_ok(env, insn->src_reg, insn)) { + verbose(env, "BPF_ATOMIC loads from R%d %s is not allowed\n", + insn->src_reg, + reg_type_str(env, reg_state(env, insn->src_reg)->type)); + return -EACCES; + } + + if (is_arena_reg(env, insn->src_reg)) { + err = save_aux_ptr_type(env, PTR_TO_ARENA, false); + if (err) + return err; + } + + /* Check whether we can read the memory. */ + err = check_mem_access(env, insn_idx, insn->src_reg, insn->off, + BPF_SIZE(insn->code), BPF_READ, insn->dst_reg, + true, false); + if (err) + return err; + + err = reg_bounds_sanity_check(env, ®s[insn->dst_reg], "atomic_load"); + if (err) + return err; + return 0; +} + +static int check_atomic_store(struct bpf_verifier_env *env, int insn_idx, + struct bpf_insn *insn) +{ + int err; + + err = check_reg_arg(env, insn->src_reg, SRC_OP); + if (err) + return err; + + err = check_reg_arg(env, insn->dst_reg, SRC_OP); + if (err) + return err; + + if (is_pointer_value(env, insn->src_reg)) { + verbose(env, "R%d leaks addr into mem\n", insn->src_reg); + return -EACCES; + } + + if (!atomic_ptr_type_ok(env, insn->dst_reg, insn)) { + verbose(env, "BPF_ATOMIC stores into R%d %s is not allowed\n", + insn->dst_reg, + reg_type_str(env, reg_state(env, insn->dst_reg)->type)); + return -EACCES; + } + + if (is_arena_reg(env, insn->dst_reg)) { + err = save_aux_ptr_type(env, PTR_TO_ARENA, false); + if (err) + return err; + } + + /* Check whether we can write into the memory. */ + err = check_mem_access(env, insn_idx, insn->dst_reg, insn->off, + BPF_SIZE(insn->code), BPF_WRITE, insn->src_reg, + true, false); + if (err) + return err; + return 0; +} + static int check_atomic(struct bpf_verifier_env *env, int insn_idx, struct bpf_insn *insn) { switch (insn->imm) { @@ -7639,6 +7726,10 @@ static int check_atomic(struct bpf_verifier_env *env, int insn_idx, struct bpf_i case BPF_XCHG: case BPF_CMPXCHG: return check_atomic_rmw(env, insn_idx, insn); + case BPF_LOAD_ACQ: + return check_atomic_load(env, insn_idx, insn); + case BPF_STORE_REL: + return check_atomic_store(env, insn_idx, insn); default: verbose(env, "BPF_ATOMIC uses invalid atomic opcode %02x\n", insn->imm); return -EINVAL; @@ -20420,7 +20511,9 @@ static int convert_ctx_accesses(struct bpf_verifier_env *env) insn->code == (BPF_ST | BPF_MEM | BPF_W) || insn->code == (BPF_ST | BPF_MEM | BPF_DW)) { type = BPF_WRITE; - } else if ((insn->code == (BPF_STX | BPF_ATOMIC | BPF_W) || + } else if ((insn->code == (BPF_STX | BPF_ATOMIC | BPF_B) || + insn->code == (BPF_STX | BPF_ATOMIC | BPF_H) || + insn->code == (BPF_STX | BPF_ATOMIC | BPF_W) || insn->code == (BPF_STX | BPF_ATOMIC | BPF_DW)) && env->insn_aux_data[i + delta].ptr_type == PTR_TO_ARENA) { insn->code = BPF_STX | BPF_PROBE_ATOMIC | BPF_SIZE(insn->code); diff --git a/tools/include/uapi/linux/bpf.h b/tools/include/uapi/linux/bpf.h index 2acf9b336371..4a20a125eb46 100644 --- a/tools/include/uapi/linux/bpf.h +++ b/tools/include/uapi/linux/bpf.h @@ -51,6 +51,19 @@ #define BPF_XCHG (0xe0 | BPF_FETCH) /* atomic exchange */ #define BPF_CMPXCHG (0xf0 | BPF_FETCH) /* atomic compare-and-write */ +#define BPF_ATOMIC_LOAD 0x10 +#define BPF_ATOMIC_STORE 0x20 +#define BPF_ATOMIC_TYPE(imm) ((imm) & 0xf0) + +#define BPF_RELAXED 0x00 +#define BPF_ACQUIRE 0x01 +#define BPF_RELEASE 0x02 +#define BPF_ACQ_REL 0x03 +#define BPF_SEQ_CST 0x04 + +#define BPF_LOAD_ACQ (BPF_ATOMIC_LOAD | BPF_ACQUIRE) /* load-acquire */ +#define BPF_STORE_REL (BPF_ATOMIC_STORE | BPF_RELEASE) /* store-release */ + enum bpf_cond_pseudo_jmp { BPF_MAY_GOTO = 0, }; From patchwork Sat Jan 25 02:18:40 2025 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Peilin Ye X-Patchwork-Id: 13950117 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from bombadil.infradead.org (bombadil.infradead.org [198.137.202.133]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (No client certificate requested) by smtp.lore.kernel.org (Postfix) with ESMTPS id 298D3C02181 for ; Sat, 25 Jan 2025 02:24:13 +0000 (UTC) DKIM-Signature: v=1; a=rsa-sha256; q=dns/txt; c=relaxed/relaxed; d=lists.infradead.org; s=bombadil.20210309; h=Sender:List-Subscribe:List-Help :List-Post:List-Archive:List-Unsubscribe:List-Id:Content-Type:Cc:To:From: Subject:Message-ID:References:Mime-Version:In-Reply-To:Date:Reply-To: Content-Transfer-Encoding:Content-ID:Content-Description:Resent-Date: Resent-From:Resent-Sender:Resent-To:Resent-Cc:Resent-Message-ID:List-Owner; bh=0jhO7BAME7t86oa1kVif63p2DnH3NcNv11+/BWAtyDY=; b=EKx4Gj9e681A9Tz1JGycq4QF3T L6KnPiZfaYeIWpouTftxFpNvndgzpMcLMVnrin0GgAIeWFTpAOV+Q0kyzeyGjlsV+WHvxQ4gyMyCr yHiBtAhQ1XGkr1U9zgKZFTSDV+LBl5JXUT/Z9Ol2UjFxk1/46P8dmordEQhdEQ10t2kP3lxA2agyy L/t7a2O0TVReEymsr+MdVHmQ8sF2Og/3NlZarTtWlS26UvjcVCRhCNUJfJI31xH+kl32Wc2Z1bAcY 3nGEt1CUlDW2mmD84Mr6wgJQFobaLg+W2IwlMNFqoY9RyisSeFRJS2uaOZf3mS2ISqFk9+1om6Tth KiceHTAg==; Received: from localhost ([::1] helo=bombadil.infradead.org) by bombadil.infradead.org with esmtp (Exim 4.98 #2 (Red Hat Linux)) id 1tbVq4-0000000FuZW-1cis; Sat, 25 Jan 2025 02:24:00 +0000 Received: from mail-pl1-x64a.google.com ([2607:f8b0:4864:20::64a]) by bombadil.infradead.org with esmtps (Exim 4.98 #2 (Red Hat Linux)) id 1tbVl0-0000000Ftxk-3isb for linux-arm-kernel@lists.infradead.org; Sat, 25 Jan 2025 02:18:47 +0000 Received: by mail-pl1-x64a.google.com with SMTP id d9443c01a7336-21648c8601cso46531165ad.2 for ; Fri, 24 Jan 2025 18:18:46 -0800 (PST) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=google.com; s=20230601; t=1737771526; x=1738376326; darn=lists.infradead.org; h=cc:to:from:subject:message-id:references:mime-version:in-reply-to :date:from:to:cc:subject:date:message-id:reply-to; bh=0jhO7BAME7t86oa1kVif63p2DnH3NcNv11+/BWAtyDY=; b=MmXYew9njyfxQa8pComPmP75vwPoN+bxd5GNnYJP0z1NyoHjaYSvF2uDRloJoa/m6k H2lTklKlGNqPzkdmJwYFQKCuVG8FYMaSp/jDuV5DFxyvJckwiivmzPnpJ1z9zdYQf6Rv WgLQVMUH+C7ja4AcVj30P/oGac31fSND1leUUizjInriuFU288jL5JPNbVJmXW4o0xv6 x5v6ce5xSUrlaGtr9EOzzaLuLRrUy0yzyflYrDrUk+ZXEYUjh0RA9GwOaFaWIvxu1YjW 3Rox5bG1EDBBGQ0g/1tSaDg0lTS/IxVQe9Gqb719QtWETcHIzzg4I7EoOesHUAPnKSdn 07zA== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20230601; t=1737771526; x=1738376326; h=cc:to:from:subject:message-id:references:mime-version:in-reply-to :date:x-gm-message-state:from:to:cc:subject:date:message-id:reply-to; bh=0jhO7BAME7t86oa1kVif63p2DnH3NcNv11+/BWAtyDY=; b=sjEFaA9fwO0sYV+qK5h2twcxiyRiaFW2yeSDbpyUFir4bxihN1ISfms81Diy+9c5a3 ue2vOeIrhYfK3NEVucfdJ62ORlVpei+/buTW/zdGNSgazHQ4LaInLpBTbqpuiQW5BVqB HpDH37lp9UGArGTs8A5cedvPftaqbB++hDQ3Uqjr6JD/m88O9u5fhsQN7AiCQ48KLpGc sjj0iGo3R+sBPVe5vukD0tmO2WxGRqPI8s5KL+Ec81w0lLCT+qbnf3C/72TusLzncjeE DO8l3dRZ+Ct7A0TeVP78EuQlJQQv6sGbR78nqv81ByCkRFgzq+GjvSZZ0G+QFbw+KkBX 6IXA== X-Forwarded-Encrypted: i=1; AJvYcCXIa0PhY+4cpSEWojetk4ojif8VKA5EpfbuLtRleRjYGzIX69hKpX7WPKYaIwU/dYCerzIaRHHueX8pwHC3QnNY@lists.infradead.org X-Gm-Message-State: AOJu0YxEdYqCCmkY05jlPhyXYi6wu8mBUY0o7DHOVqyNEp6O+TwQ8Vv9 +A/fiG3oC2FY86vdySCL2SxOPJXDA17cdvWxJ62Uq2yKMDGLDocpKtMaosWNDkW2J/IIMmQMbnp uBcaYGIv+PA== X-Google-Smtp-Source: AGHT+IGOA2myrdjHdShHtNzy3sHhO4gQRnqhkbbsoC8Yq6DOPMUbkkwaNv2wF6MPelQcaIQVD67axVPVSm5LaA== X-Received: from plnq11.prod.google.com ([2002:a17:902:f78b:b0:216:1543:196d]) (user=yepeilin job=prod-delivery.src-stubby-dispatcher) by 2002:a17:902:ea10:b0:21a:8716:faab with SMTP id d9443c01a7336-21c352ec1f0mr490310075ad.16.1737771525995; Fri, 24 Jan 2025 18:18:45 -0800 (PST) Date: Sat, 25 Jan 2025 02:18:40 +0000 In-Reply-To: Mime-Version: 1.0 References: X-Mailer: git-send-email 2.48.1.262.g85cc9f2d1e-goog Message-ID: Subject: [PATCH bpf-next v1 4/8] arm64: insn: Add BIT(23) to {load,store}_ex's mask From: Peilin Ye To: bpf@vger.kernel.org, linux-arm-kernel@lists.infradead.org Cc: Peilin Ye , bpf@ietf.org, Xu Kuohai , Eduard Zingerman , David Vernet , Alexei Starovoitov , Daniel Borkmann , Andrii Nakryiko , Martin KaFai Lau , Song Liu , Yonghong Song , John Fastabend , KP Singh , Stanislav Fomichev , Hao Luo , Jiri Olsa , Jonathan Corbet , "Paul E. McKenney" , Puranjay Mohan , Catalin Marinas , Will Deacon , Quentin Monnet , Mykola Lysenko , Shuah Khan , Josh Don , Barret Rhoden , Neel Natu , Benjamin Segall , linux-kernel@vger.kernel.org X-CRM114-Version: 20100106-BlameMichelson ( TRE 0.8.0 (BSD) ) MR-646709E3 X-CRM114-CacheID: sfid-20250124_181846_928497_3A68158B X-CRM114-Status: GOOD ( 10.82 ) X-BeenThere: linux-arm-kernel@lists.infradead.org X-Mailman-Version: 2.1.34 Precedence: list List-Id: List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Sender: "linux-arm-kernel" Errors-To: linux-arm-kernel-bounces+linux-arm-kernel=archiver.kernel.org@lists.infradead.org We are planning to add load-acquire (LDAR{,B,H}) and store-release (STLR{,B,H}) instructions to insn.{c,h}; add BIT(23) to mask of load_ex and store_ex to prevent aarch64_insn_is_{load,store}_ex() from returning false-positives for load-acquire and store-release instructions. Reference: Arm Architecture Reference Manual (ARM DDI 0487K.a, ID032224), * C6.2.228 LDXR * C6.2.165 LDAXR * C6.2.161 LDAR * C6.2.393 STXR * C6.2.360 STLXR * C6.2.353 STLR Signed-off-by: Peilin Ye --- arch/arm64/include/asm/insn.h | 4 ++-- 1 file changed, 2 insertions(+), 2 deletions(-) diff --git a/arch/arm64/include/asm/insn.h b/arch/arm64/include/asm/insn.h index e390c432f546..2d8316b3abaf 100644 --- a/arch/arm64/include/asm/insn.h +++ b/arch/arm64/include/asm/insn.h @@ -351,8 +351,8 @@ __AARCH64_INSN_FUNCS(ldr_imm, 0x3FC00000, 0x39400000) __AARCH64_INSN_FUNCS(ldr_lit, 0xBF000000, 0x18000000) __AARCH64_INSN_FUNCS(ldrsw_lit, 0xFF000000, 0x98000000) __AARCH64_INSN_FUNCS(exclusive, 0x3F800000, 0x08000000) -__AARCH64_INSN_FUNCS(load_ex, 0x3F400000, 0x08400000) -__AARCH64_INSN_FUNCS(store_ex, 0x3F400000, 0x08000000) +__AARCH64_INSN_FUNCS(load_ex, 0x3FC00000, 0x08400000) +__AARCH64_INSN_FUNCS(store_ex, 0x3FC00000, 0x08000000) __AARCH64_INSN_FUNCS(mops, 0x3B200C00, 0x19000400) __AARCH64_INSN_FUNCS(stp, 0x7FC00000, 0x29000000) __AARCH64_INSN_FUNCS(ldp, 0x7FC00000, 0x29400000) From patchwork Sat Jan 25 02:19:00 2025 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Peilin Ye X-Patchwork-Id: 13950118 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from bombadil.infradead.org (bombadil.infradead.org [198.137.202.133]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (No client certificate requested) by smtp.lore.kernel.org (Postfix) with ESMTPS id 380E1C02181 for ; Sat, 25 Jan 2025 02:25:28 +0000 (UTC) DKIM-Signature: v=1; a=rsa-sha256; q=dns/txt; c=relaxed/relaxed; d=lists.infradead.org; s=bombadil.20210309; h=Sender:List-Subscribe:List-Help :List-Post:List-Archive:List-Unsubscribe:List-Id:Content-Type:Cc:To:From: Subject:Message-ID:References:Mime-Version:In-Reply-To:Date:Reply-To: Content-Transfer-Encoding:Content-ID:Content-Description:Resent-Date: Resent-From:Resent-Sender:Resent-To:Resent-Cc:Resent-Message-ID:List-Owner; bh=ff8OoM3RwJ/YS9q+sO/KvAZcdKn8t6Rkpge4WfBIGvQ=; b=k3cNef1Pp4wkQYs0ck/XHpfU1p 464OGMoSspeboY/iBg8jp4+0TFHcp/wPyCZXKSejv6LIf1dioFHB506hE+3WM71eNwJG3t1SGYdy8 QcYCgyOcb5TcVKDrLHnm5R//c1ZVKLICD2BvvJY4u4nS1D5+zZQ5T1RYzFAVe+SztemWnjItEaeMh RSNlkSlFSmAxDou5kZDGo6zVPdK37H85ZuJuKC/jvpvQ3ZjGyZ0RPqzOfQy88rATq1clHQ1U6ZTgJ EQmxe1Sm3g/Qh0Xngv3Qj6f6g7SHRwKlGU9z1AlYv5U/FHyyo6kK1oYspEPbidB7YPnDSxJZTc/kx 2YIGV5vQ==; Received: from localhost ([::1] helo=bombadil.infradead.org) by bombadil.infradead.org with esmtp (Exim 4.98 #2 (Red Hat Linux)) id 1tbVrJ-0000000Fuex-4BlG; Sat, 25 Jan 2025 02:25:18 +0000 Received: from mail-pl1-x649.google.com ([2607:f8b0:4864:20::649]) by bombadil.infradead.org with esmtps (Exim 4.98 #2 (Red Hat Linux)) id 1tbVlL-0000000Fu30-1KFZ for linux-arm-kernel@lists.infradead.org; Sat, 25 Jan 2025 02:19:08 +0000 Received: by mail-pl1-x649.google.com with SMTP id d9443c01a7336-21638389f63so40929045ad.1 for ; Fri, 24 Jan 2025 18:19:06 -0800 (PST) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=google.com; s=20230601; t=1737771545; x=1738376345; darn=lists.infradead.org; h=cc:to:from:subject:message-id:references:mime-version:in-reply-to :date:from:to:cc:subject:date:message-id:reply-to; bh=ff8OoM3RwJ/YS9q+sO/KvAZcdKn8t6Rkpge4WfBIGvQ=; b=bz/eWi16XDFU52dA2k7aX5VDzPrYtgAhdcwrsIYm1z7FxmzA1hC8dZUuhY/hQc/ln9 kXAzosy0WABEnNuWjnMpEWZImU46n1q4YPWn0myDd0H5bKScITm31vzWswGEqG3AJWhY G/gJpa4gZhqEd7JIkgwoCxR8Qlh73TzQJYyFXMsN2QHApSDlpUgcWWhoYAu6UDEvflyY AJ50DVVgy4NVCYKGmhjPfrrMBnER7IjoraEHRnZmmnd5nsyZ1a8y0Pj2gfPs4ao7Qhgr WndYaJexu/G6A2jl8uEIpPExo1VP0ip1aRafLmRIQyNsn5SK2QQ7xV6+fry0ZWy/u0FD eU9w== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20230601; t=1737771545; x=1738376345; h=cc:to:from:subject:message-id:references:mime-version:in-reply-to :date:x-gm-message-state:from:to:cc:subject:date:message-id:reply-to; bh=ff8OoM3RwJ/YS9q+sO/KvAZcdKn8t6Rkpge4WfBIGvQ=; b=Km2jfZCCOTX1O75/VfuOwfsOg3Mjo4z/FfgT2lBJsdtP+9h/isz5GJpkXv7DZEbcCV KwX5D9QnC+36jVjVvoh9pIanuRBxHHjumqQtACcOoXRJaesgVUXQvmKNPjbmActP013K vDZkCUAT29oiSHIUyItu1/RKqt5LoR7ysocv1NaQ6IzPWEeBeLVN1VOXtJobxU9Dn7Rq H29EYezTExZtws7Yd2UUaQOPFTar+cTBwZifvJncAw4cDhXkP6LSigAXDg8JFLGo/iVf xto1lvX74DLP3q3DcpAlPL6loUJFXsKnLX2X8fpmuTm/UThzr2ZCNE8Tpq/1Alw9RPAh xVUw== X-Forwarded-Encrypted: i=1; AJvYcCUTA9BolVezW2GgwvpvIPZ90byxtfvV0W3SOzhtRQEJtAA03HOQPvjk5K7VN0q+tlobsScP4B+TCoW+6j9SVd2T@lists.infradead.org X-Gm-Message-State: AOJu0YwiUQK43tsH1WCqe9tqsmgKqPVII3ecL+mDBmSmtkvCcZBinvSP 8v/UcxRDTV/my0Y0YRqoNY3rRyjN8h9T6pLLAgz4j9rbvqaMRwLt/wYywjGIIwEczy4zED99SGJ CxfnsHDtNIQ== X-Google-Smtp-Source: AGHT+IF5yitEZ2ynO7/1xRDH6f+mEY3tHHd6tHuHsyfWb71s1fY6pLTuNoYRwXYGNFFpe0KHLpuGhOpUmxzkiQ== X-Received: from plgn4.prod.google.com ([2002:a17:902:f604:b0:212:4d11:70f5]) (user=yepeilin job=prod-delivery.src-stubby-dispatcher) by 2002:a17:902:ce01:b0:211:e812:3948 with SMTP id d9443c01a7336-21c34cc010emr536071555ad.0.1737771545532; Fri, 24 Jan 2025 18:19:05 -0800 (PST) Date: Sat, 25 Jan 2025 02:19:00 +0000 In-Reply-To: Mime-Version: 1.0 References: X-Mailer: git-send-email 2.48.1.262.g85cc9f2d1e-goog Message-ID: <7544131164e5a3ab1aa192e895c883106d8dd324.1737763916.git.yepeilin@google.com> Subject: [PATCH bpf-next v1 5/8] arm64: insn: Add load-acquire and store-release instructions From: Peilin Ye To: bpf@vger.kernel.org, linux-arm-kernel@lists.infradead.org Cc: Peilin Ye , bpf@ietf.org, Xu Kuohai , Eduard Zingerman , David Vernet , Alexei Starovoitov , Daniel Borkmann , Andrii Nakryiko , Martin KaFai Lau , Song Liu , Yonghong Song , John Fastabend , KP Singh , Stanislav Fomichev , Hao Luo , Jiri Olsa , Jonathan Corbet , "Paul E. McKenney" , Puranjay Mohan , Catalin Marinas , Will Deacon , Quentin Monnet , Mykola Lysenko , Shuah Khan , Josh Don , Barret Rhoden , Neel Natu , Benjamin Segall , linux-kernel@vger.kernel.org X-CRM114-Version: 20100106-BlameMichelson ( TRE 0.8.0 (BSD) ) MR-646709E3 X-CRM114-CacheID: sfid-20250124_181907_353736_8FC304C6 X-CRM114-Status: GOOD ( 11.13 ) X-BeenThere: linux-arm-kernel@lists.infradead.org X-Mailman-Version: 2.1.34 Precedence: list List-Id: List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Sender: "linux-arm-kernel" Errors-To: linux-arm-kernel-bounces+linux-arm-kernel=archiver.kernel.org@lists.infradead.org Add load-acquire ("load_acq", LDAR{,B,H}) and store-release ("store_rel", STLR{,B,H}) instructions. Breakdown of encoding: size L (Rs) o0 (Rt2) Rn Rt mask (0x3fdffc00): 00 111111 1 1 0 11111 1 11111 00000 00000 value, load_acq (0x08dffc00): 00 001000 1 1 0 11111 1 11111 00000 00000 value, store_rel (0x089ffc00): 00 001000 1 0 0 11111 1 11111 00000 00000 As suggested by Xu [1], include all Should-Be-One (SBO) bits ("Rs" and "Rt2" fields) in the "mask" and "value" numbers. It is worth noting that we are adding the "no offset" variant of STLR instead of the "pre-index" variant, which has a different encoding. Reference: Arm Architecture Reference Manual (ARM DDI 0487K.a, ID032224), * C6.2.161 LDAR * C6.2.353 STLR [1] https://lore.kernel.org/bpf/4e6641ce-3f1e-4251-8daf-4dd4b77d08c4@huaweicloud.com/ Signed-off-by: Peilin Ye --- arch/arm64/include/asm/insn.h | 8 ++++++++ arch/arm64/lib/insn.c | 28 ++++++++++++++++++++++++++++ 2 files changed, 36 insertions(+) diff --git a/arch/arm64/include/asm/insn.h b/arch/arm64/include/asm/insn.h index 2d8316b3abaf..39577f1d079a 100644 --- a/arch/arm64/include/asm/insn.h +++ b/arch/arm64/include/asm/insn.h @@ -188,8 +188,10 @@ enum aarch64_insn_ldst_type { AARCH64_INSN_LDST_STORE_PAIR_PRE_INDEX, AARCH64_INSN_LDST_LOAD_PAIR_POST_INDEX, AARCH64_INSN_LDST_STORE_PAIR_POST_INDEX, + AARCH64_INSN_LDST_LOAD_ACQ, AARCH64_INSN_LDST_LOAD_EX, AARCH64_INSN_LDST_LOAD_ACQ_EX, + AARCH64_INSN_LDST_STORE_REL, AARCH64_INSN_LDST_STORE_EX, AARCH64_INSN_LDST_STORE_REL_EX, AARCH64_INSN_LDST_SIGNED_LOAD_IMM_OFFSET, @@ -351,6 +353,8 @@ __AARCH64_INSN_FUNCS(ldr_imm, 0x3FC00000, 0x39400000) __AARCH64_INSN_FUNCS(ldr_lit, 0xBF000000, 0x18000000) __AARCH64_INSN_FUNCS(ldrsw_lit, 0xFF000000, 0x98000000) __AARCH64_INSN_FUNCS(exclusive, 0x3F800000, 0x08000000) +__AARCH64_INSN_FUNCS(load_acq, 0x3FDFFC00, 0x08DFFC00) +__AARCH64_INSN_FUNCS(store_rel, 0x3FDFFC00, 0x089FFC00) __AARCH64_INSN_FUNCS(load_ex, 0x3FC00000, 0x08400000) __AARCH64_INSN_FUNCS(store_ex, 0x3FC00000, 0x08000000) __AARCH64_INSN_FUNCS(mops, 0x3B200C00, 0x19000400) @@ -602,6 +606,10 @@ u32 aarch64_insn_gen_load_store_pair(enum aarch64_insn_register reg1, int offset, enum aarch64_insn_variant variant, enum aarch64_insn_ldst_type type); +u32 aarch64_insn_gen_load_acq_store_rel(enum aarch64_insn_register reg, + enum aarch64_insn_register base, + enum aarch64_insn_size_type size, + enum aarch64_insn_ldst_type type); u32 aarch64_insn_gen_load_store_ex(enum aarch64_insn_register reg, enum aarch64_insn_register base, enum aarch64_insn_register state, diff --git a/arch/arm64/lib/insn.c b/arch/arm64/lib/insn.c index b008a9b46a7f..f8b83f4d9171 100644 --- a/arch/arm64/lib/insn.c +++ b/arch/arm64/lib/insn.c @@ -540,6 +540,34 @@ u32 aarch64_insn_gen_load_store_pair(enum aarch64_insn_register reg1, offset >> shift); } +u32 aarch64_insn_gen_load_acq_store_rel(enum aarch64_insn_register reg, + enum aarch64_insn_register base, + enum aarch64_insn_size_type size, + enum aarch64_insn_ldst_type type) +{ + u32 insn; + + switch (type) { + case AARCH64_INSN_LDST_LOAD_ACQ: + insn = aarch64_insn_get_load_acq_value(); + break; + case AARCH64_INSN_LDST_STORE_REL: + insn = aarch64_insn_get_store_rel_value(); + break; + default: + pr_err("%s: unknown load-acquire/store-release encoding %d\n", __func__, type); + return AARCH64_BREAK_FAULT; + } + + insn = aarch64_insn_encode_ldst_size(size, insn); + + insn = aarch64_insn_encode_register(AARCH64_INSN_REGTYPE_RT, insn, + reg); + + return aarch64_insn_encode_register(AARCH64_INSN_REGTYPE_RN, insn, + base); +} + u32 aarch64_insn_gen_load_store_ex(enum aarch64_insn_register reg, enum aarch64_insn_register base, enum aarch64_insn_register state, From patchwork Sat Jan 25 02:19:09 2025 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Peilin Ye X-Patchwork-Id: 13950119 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from bombadil.infradead.org (bombadil.infradead.org [198.137.202.133]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (No client certificate requested) by smtp.lore.kernel.org (Postfix) with ESMTPS id 0DD4DC02181 for ; Sat, 25 Jan 2025 02:26:50 +0000 (UTC) DKIM-Signature: v=1; a=rsa-sha256; q=dns/txt; c=relaxed/relaxed; d=lists.infradead.org; s=bombadil.20210309; h=Sender:List-Subscribe:List-Help :List-Post:List-Archive:List-Unsubscribe:List-Id:Content-Type:Cc:To:From: Subject:Message-ID:References:Mime-Version:In-Reply-To:Date:Reply-To: Content-Transfer-Encoding:Content-ID:Content-Description:Resent-Date: Resent-From:Resent-Sender:Resent-To:Resent-Cc:Resent-Message-ID:List-Owner; bh=3/pYcf0sANYo9TEsaEY4ONRLzSTmKybwrzwLYycpJYE=; b=TRDXKpM+FXrfgqpGjBcpQEkrmM Yo8HF940+tYOnOZ6Uu2BoeEKzldFqyV7V1xkzlCAmLb8JfF9XlNCQzsfc6GRAeYj3rqO5LVXnX6uh 8ocrL3KWtjL7As8O7AtMEusUIe52L16ajoKiAe8CI/Rdbn5+gNYlsjg0+3tyRKuyr72WJFwOgJNJK 545aRjiDPFuTrsNgnQRZMFocYHclJzdASqrjjVVUTl3eCAJBdTS/iSAINvv0jJMKy2tWIopxZNyNW agyUjdRatJZsRe5o1VX1UtqEMgIqvjCc/dVrRDdv2lB+EeHVEmX2HNGValjr5x2vLls+6chTfyOhk mSn7TbeA==; Received: from localhost ([::1] helo=bombadil.infradead.org) by bombadil.infradead.org with esmtp (Exim 4.98 #2 (Red Hat Linux)) id 1tbVsb-0000000FulR-2fwn; Sat, 25 Jan 2025 02:26:37 +0000 Received: from mail-pj1-x104a.google.com ([2607:f8b0:4864:20::104a]) by bombadil.infradead.org with esmtps (Exim 4.98 #2 (Red Hat Linux)) id 1tbVlV-0000000Fu5K-1AKI for linux-arm-kernel@lists.infradead.org; Sat, 25 Jan 2025 02:19:18 +0000 Received: by mail-pj1-x104a.google.com with SMTP id 98e67ed59e1d1-2f2a9743093so5481199a91.3 for ; Fri, 24 Jan 2025 18:19:16 -0800 (PST) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=google.com; s=20230601; t=1737771556; x=1738376356; darn=lists.infradead.org; h=cc:to:from:subject:message-id:references:mime-version:in-reply-to :date:from:to:cc:subject:date:message-id:reply-to; bh=3/pYcf0sANYo9TEsaEY4ONRLzSTmKybwrzwLYycpJYE=; b=QEN86rXQhORYdE10SIsBaAeJnZX+6W33rfZ63iUY6Q53dpnG803Xry9OrDSr8Ok5fq FeYVaHt3Fed2b6ud8at4CBdjMdYlkQevNUlbfDzSLN95vohi0t9W7WEqU0lvZllvi9IG fO/q6VE9FinKrCygVOiCkjTtgsHTAQhqble/8DBIqKwKA+QRgP61rpJndo74cw++U8tq runOeGBpGmCk4ObOv2x1PN+WtAnaSAiRJfeXXlm3cF2YfkvidxaaC7kqm3jFZAsIx7ph +1a568QSHXCLxBEgnoiZqRO0vPvj/FQxsDscCqZdc15xXDqi8fOJIL2yVJln183oU9s5 xYCA== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20230601; t=1737771556; x=1738376356; h=cc:to:from:subject:message-id:references:mime-version:in-reply-to :date:x-gm-message-state:from:to:cc:subject:date:message-id:reply-to; bh=3/pYcf0sANYo9TEsaEY4ONRLzSTmKybwrzwLYycpJYE=; b=Z7iOFDeljlHK+zjAIdYoJzcWFAiEa7z29qJCXpaVE08DUAr5f1lm6QluwOd3Jd5gT6 m8PkmGgnyK/1ZxIKaON0Di4WxVwLc8Ne6/pD9bMEcL3d9Kazi5uPL6Q3VTAe/3XTMg0H Nef0yZNqSugtuUOM2eDNb5VgaMJZw10pUdum9/Rvamg0DtHRxDY1tW7NLu0jz+EUf8mw HGBv4zjynNf/BxFn2INFM+KX2SdrHEFq/OPtLAzcqNM6uPLyTmHNSeJi8kW0Quq217iE VvE5vLBwI1kd+l9WLWdQ4kmTQnMflk6zwQdTB+OPbgJkWq8tvBlz7hxY2j92Id8Hgi+O U1hg== X-Forwarded-Encrypted: i=1; AJvYcCXIBje9cQmA1NQkR/0uD73LEFZidGVe7EQunYopSSdvvyV5MVoDw92bGEKsRSuCWdFSpkmwVAgFt3DSZ+pMGbhX@lists.infradead.org X-Gm-Message-State: AOJu0YwcsckfFq8CdjgmHboYrit0DKg9XmyPcnmxPUT1CfdCRmw8pAQt 45UKUpj0xWeSqv/bVXX67ebY82lyr3Mida3axYzYsyEF0/Ov57K0IHOgp9oPCOrrNUwTJxbf6Ko +si8yU2Tphw== X-Google-Smtp-Source: AGHT+IHLeumw4Cen/udsKyGtvAVMvHHnzYuG7MNiJRP+RmXCfv67AF/APEi6auE44B6c9jmN91D0FezxRYqdMA== X-Received: from pjd4.prod.google.com ([2002:a17:90b:54c4:b0:2d3:d4ca:5fb0]) (user=yepeilin job=prod-delivery.src-stubby-dispatcher) by 2002:a17:90b:2e4e:b0:2ee:fd53:2b17 with SMTP id 98e67ed59e1d1-2f782d97276mr45715232a91.29.1737771555858; Fri, 24 Jan 2025 18:19:15 -0800 (PST) Date: Sat, 25 Jan 2025 02:19:09 +0000 In-Reply-To: Mime-Version: 1.0 References: X-Mailer: git-send-email 2.48.1.262.g85cc9f2d1e-goog Message-ID: <1733f889a4b46d13844d1083b5cfba5005b22e86.1737763916.git.yepeilin@google.com> Subject: [PATCH bpf-next v1 6/8] bpf, arm64: Support load-acquire and store-release instructions From: Peilin Ye To: bpf@vger.kernel.org, linux-arm-kernel@lists.infradead.org Cc: Peilin Ye , bpf@ietf.org, Xu Kuohai , Eduard Zingerman , David Vernet , Alexei Starovoitov , Daniel Borkmann , Andrii Nakryiko , Martin KaFai Lau , Song Liu , Yonghong Song , John Fastabend , KP Singh , Stanislav Fomichev , Hao Luo , Jiri Olsa , Jonathan Corbet , "Paul E. McKenney" , Puranjay Mohan , Catalin Marinas , Will Deacon , Quentin Monnet , Mykola Lysenko , Shuah Khan , Josh Don , Barret Rhoden , Neel Natu , Benjamin Segall , linux-kernel@vger.kernel.org X-CRM114-Version: 20100106-BlameMichelson ( TRE 0.8.0 (BSD) ) MR-646709E3 X-CRM114-CacheID: sfid-20250124_181917_314686_C467EFC2 X-CRM114-Status: GOOD ( 15.72 ) X-BeenThere: linux-arm-kernel@lists.infradead.org X-Mailman-Version: 2.1.34 Precedence: list List-Id: List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Sender: "linux-arm-kernel" Errors-To: linux-arm-kernel-bounces+linux-arm-kernel=archiver.kernel.org@lists.infradead.org Support BPF load-acquire (BPF_LOAD_ACQ) and store-release (BPF_STORE_REL) instructions in the arm64 JIT compiler. For example: db 10 00 00 11 00 00 00 r0 = load_acquire((u64 *)(r1 + 0x0)) 95 00 00 00 00 00 00 00 exit opcode (0xdb): BPF_ATOMIC | BPF_DW | BPF_STX imm (0x00000011): BPF_LOAD_ACQ The JIT compiler would emit an LDAR instruction for the above, e.g.: ldar x7, [x0] Similarly, consider the following 16-bit store-release: cb 21 00 00 22 00 00 00 store_release((u16 *)(r1 + 0x0), w2) 95 00 00 00 00 00 00 00 exit opcode (0xcb): BPF_ATOMIC | BPF_H | BPF_STX imm (0x00000022): BPF_ATOMIC_STORE | BPF_RELEASE An STLRH instruction would be emitted, e.g.: stlrh w1, [x0] For a complete mapping: load-acquire 8-bit LDARB (BPF_LOAD_ACQ) 16-bit LDARH 32-bit LDAR (32-bit) 64-bit LDAR (64-bit) store-release 8-bit STLRB (BPF_STORE_REL) 16-bit STLRH 32-bit STLR (32-bit) 64-bit STLR (64-bit) Arena accesses are supported. bpf_jit_supports_insn(..., /*in_arena=*/true) always returns true for BPF_LOAD_ACQ and BPF_STORE_REL instructions, as they don't depend on ARM64_HAS_LSE_ATOMICS. Signed-off-by: Peilin Ye --- arch/arm64/net/bpf_jit.h | 20 ++++++++ arch/arm64/net/bpf_jit_comp.c | 92 ++++++++++++++++++++++++++++++++++- 2 files changed, 110 insertions(+), 2 deletions(-) diff --git a/arch/arm64/net/bpf_jit.h b/arch/arm64/net/bpf_jit.h index b22ab2f97a30..a3b0e693a125 100644 --- a/arch/arm64/net/bpf_jit.h +++ b/arch/arm64/net/bpf_jit.h @@ -119,6 +119,26 @@ aarch64_insn_gen_load_store_ex(Rt, Rn, Rs, A64_SIZE(sf), \ AARCH64_INSN_LDST_STORE_REL_EX) +/* Load-acquire & store-release */ +#define A64_LDAR(Rt, Rn, size) \ + aarch64_insn_gen_load_acq_store_rel(Rt, Rn, AARCH64_INSN_SIZE_##size, \ + AARCH64_INSN_LDST_LOAD_ACQ) +#define A64_STLR(Rt, Rn, size) \ + aarch64_insn_gen_load_acq_store_rel(Rt, Rn, AARCH64_INSN_SIZE_##size, \ + AARCH64_INSN_LDST_STORE_REL) + +/* Rt = [Rn] (load acquire) */ +#define A64_LDARB(Wt, Xn) A64_LDAR(Wt, Xn, 8) +#define A64_LDARH(Wt, Xn) A64_LDAR(Wt, Xn, 16) +#define A64_LDAR32(Wt, Xn) A64_LDAR(Wt, Xn, 32) +#define A64_LDAR64(Xt, Xn) A64_LDAR(Xt, Xn, 64) + +/* [Rn] = Rt (store release) */ +#define A64_STLRB(Wt, Xn) A64_STLR(Wt, Xn, 8) +#define A64_STLRH(Wt, Xn) A64_STLR(Wt, Xn, 16) +#define A64_STLR32(Wt, Xn) A64_STLR(Wt, Xn, 32) +#define A64_STLR64(Xt, Xn) A64_STLR(Xt, Xn, 64) + /* * LSE atomics * diff --git a/arch/arm64/net/bpf_jit_comp.c b/arch/arm64/net/bpf_jit_comp.c index 8446848edddb..488cbe094551 100644 --- a/arch/arm64/net/bpf_jit_comp.c +++ b/arch/arm64/net/bpf_jit_comp.c @@ -647,6 +647,87 @@ static int emit_bpf_tail_call(struct jit_ctx *ctx) return 0; } +static inline bool is_atomic_load_store(const s32 imm) +{ + const s32 type = BPF_ATOMIC_TYPE(imm); + + return type == BPF_ATOMIC_LOAD || type == BPF_ATOMIC_STORE; +} + +static int emit_atomic_load_store(const struct bpf_insn *insn, struct jit_ctx *ctx) +{ + const s32 type = BPF_ATOMIC_TYPE(insn->imm); + const s16 off = insn->off; + const u8 code = insn->code; + const bool arena = BPF_MODE(code) == BPF_PROBE_ATOMIC; + const u8 arena_vm_base = bpf2a64[ARENA_VM_START]; + const u8 dst = bpf2a64[insn->dst_reg]; + const u8 src = bpf2a64[insn->src_reg]; + const u8 tmp = bpf2a64[TMP_REG_1]; + u8 reg; + + switch (type) { + case BPF_ATOMIC_LOAD: + reg = src; + break; + case BPF_ATOMIC_STORE: + reg = dst; + break; + default: + pr_err_once("unknown atomic load/store op type %02x\n", type); + return -EINVAL; + } + + if (off) { + emit_a64_add_i(1, tmp, reg, tmp, off, ctx); + reg = tmp; + } + if (arena) { + emit(A64_ADD(1, tmp, reg, arena_vm_base), ctx); + reg = tmp; + } + + switch (insn->imm) { + case BPF_LOAD_ACQ: + switch (BPF_SIZE(code)) { + case BPF_B: + emit(A64_LDARB(dst, reg), ctx); + break; + case BPF_H: + emit(A64_LDARH(dst, reg), ctx); + break; + case BPF_W: + emit(A64_LDAR32(dst, reg), ctx); + break; + case BPF_DW: + emit(A64_LDAR64(dst, reg), ctx); + break; + } + break; + case BPF_STORE_REL: + switch (BPF_SIZE(code)) { + case BPF_B: + emit(A64_STLRB(src, reg), ctx); + break; + case BPF_H: + emit(A64_STLRH(src, reg), ctx); + break; + case BPF_W: + emit(A64_STLR32(src, reg), ctx); + break; + case BPF_DW: + emit(A64_STLR64(src, reg), ctx); + break; + } + break; + default: + pr_err_once("unknown atomic load/store op code %02x\n", insn->imm); + return -EINVAL; + } + + return 0; +} + #ifdef CONFIG_ARM64_LSE_ATOMICS static int emit_lse_atomic(const struct bpf_insn *insn, struct jit_ctx *ctx) { @@ -1641,11 +1722,17 @@ static int build_insn(const struct bpf_insn *insn, struct jit_ctx *ctx, return ret; break; + case BPF_STX | BPF_ATOMIC | BPF_B: + case BPF_STX | BPF_ATOMIC | BPF_H: case BPF_STX | BPF_ATOMIC | BPF_W: case BPF_STX | BPF_ATOMIC | BPF_DW: + case BPF_STX | BPF_PROBE_ATOMIC | BPF_B: + case BPF_STX | BPF_PROBE_ATOMIC | BPF_H: case BPF_STX | BPF_PROBE_ATOMIC | BPF_W: case BPF_STX | BPF_PROBE_ATOMIC | BPF_DW: - if (cpus_have_cap(ARM64_HAS_LSE_ATOMICS)) + if (is_atomic_load_store(insn->imm)) + ret = emit_atomic_load_store(insn, ctx); + else if (cpus_have_cap(ARM64_HAS_LSE_ATOMICS)) ret = emit_lse_atomic(insn, ctx); else ret = emit_ll_sc_atomic(insn, ctx); @@ -2669,7 +2756,8 @@ bool bpf_jit_supports_insn(struct bpf_insn *insn, bool in_arena) switch (insn->code) { case BPF_STX | BPF_ATOMIC | BPF_W: case BPF_STX | BPF_ATOMIC | BPF_DW: - if (!cpus_have_cap(ARM64_HAS_LSE_ATOMICS)) + if (!is_atomic_load_store(insn->imm) && + !cpus_have_cap(ARM64_HAS_LSE_ATOMICS)) return false; } return true; From patchwork Sat Jan 25 02:19:30 2025 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Peilin Ye X-Patchwork-Id: 13950120 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from bombadil.infradead.org (bombadil.infradead.org [198.137.202.133]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (No client certificate requested) by smtp.lore.kernel.org (Postfix) with ESMTPS id E7C28C02181 for ; Sat, 25 Jan 2025 02:28:06 +0000 (UTC) DKIM-Signature: v=1; a=rsa-sha256; q=dns/txt; c=relaxed/relaxed; d=lists.infradead.org; s=bombadil.20210309; h=Sender:List-Subscribe:List-Help :List-Post:List-Archive:List-Unsubscribe:List-Id:Content-Type:Cc:To:From: Subject:Message-ID:References:Mime-Version:In-Reply-To:Date:Reply-To: Content-Transfer-Encoding:Content-ID:Content-Description:Resent-Date: Resent-From:Resent-Sender:Resent-To:Resent-Cc:Resent-Message-ID:List-Owner; bh=TUVtAh3jk+1Z9FYL8oOTJSugJD98SnZKa+Q8sEqs79Q=; b=KsJqJ+0X+8Vlo/MWUAqDfyMIm1 fFCm0/YhosZcTdVe9uUJXA8ZYV+oAnefmMH2VnWK8qegaPZM9QyFgCukZ83QbsK2byI2OLOaqCB6X FR4LTJFEfPew+ECO2BNoaTPaKFzT5Un+MEuvurxboZa6twDfXb1Zp7IjEc9xNquDJAoz9yI2ByTD7 KdF9n8LKci1LhIzF+l8YfrnsnbMRLf4LuXNQZ86RrNejy2fPpYnLiVxPY9OjCz2DVJ8/YnAuK4YiX 3XNdnyh9Kym/0CqZCOLNBhRrngTspY4MeMbNlBZcvllEczxr5eq/0f4nv+4n+33kXeu06EKn+l42+ 8jZUB2Iw==; Received: from localhost ([::1] helo=bombadil.infradead.org) by bombadil.infradead.org with esmtp (Exim 4.98 #2 (Red Hat Linux)) id 1tbVtr-0000000FurO-2B1E; Sat, 25 Jan 2025 02:27:55 +0000 Received: from mail-pj1-x1049.google.com ([2607:f8b0:4864:20::1049]) by bombadil.infradead.org with esmtps (Exim 4.98 #2 (Red Hat Linux)) id 1tbVlo-0000000Fu9f-2guY for linux-arm-kernel@lists.infradead.org; Sat, 25 Jan 2025 02:19:38 +0000 Received: by mail-pj1-x1049.google.com with SMTP id 98e67ed59e1d1-2ee9f66cb12so5540858a91.1 for ; Fri, 24 Jan 2025 18:19:35 -0800 (PST) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=google.com; s=20230601; t=1737771575; x=1738376375; darn=lists.infradead.org; h=cc:to:from:subject:message-id:references:mime-version:in-reply-to :date:from:to:cc:subject:date:message-id:reply-to; bh=TUVtAh3jk+1Z9FYL8oOTJSugJD98SnZKa+Q8sEqs79Q=; b=bZWrljxkkINMvI8qPAkqfFsRvky35Xr7NhNp7zOXgxF1cGPRYbvAFZ0ry/8sABr/1U XRT0Chg4anJ+BWioyTRSGqQq7Coqx6q939ef/lrZRtiB4vORHWkGQpyst7ApzVvbi9q/ de+U71lEtAlQOR3fDcSyvg6X+qCKCibqywwMujwG6u/fZlrwAR9DMehxbm1jFau5mXCt mQkAn7Cl2BdK0qHsdsQs3T9smwisQL1LkYoWB9k+Okc7dyitoliZYp/YBVUnYJMAo19S STXfKHtL2/cZg8m171PjuCQlldCf3N2187e3BO12DYFAZnNwgJDoAgoPAvWF5raz22GZ kRMQ== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20230601; t=1737771575; x=1738376375; h=cc:to:from:subject:message-id:references:mime-version:in-reply-to :date:x-gm-message-state:from:to:cc:subject:date:message-id:reply-to; bh=TUVtAh3jk+1Z9FYL8oOTJSugJD98SnZKa+Q8sEqs79Q=; b=XIDdUabYFeM/+yin0bp00FehjlKkDQ8Wyf/wzWnDzLBSVgXcdmVIJGeesri4h5oOjU AzfEeGB5EMvwE/O25jjNMWBuYfviOsu/ZFfj7lCAID9eKJudYT7oTbTpIXR4nhZZymt6 pAFvq0FN5gG0rW0bs2zygc3+jga8GW/E2idggBzqmculQHz9GsYbCRxs5Qq4RXrBBNap gW/O2hiGMttf5ZUR7tUZlDUWpaZMN1OkB/UYej72tb/4G9Pgiu6qKkmkrPLa1YpNAbIB TaCH2eOGcnk+iLuTdd9VBttlq/5nskkNTSdOeyewP94szvaHCxihrSYVKNQR3+TxxDOJ zNvg== X-Forwarded-Encrypted: i=1; AJvYcCVFfw0YUg2klCLTgayaTDm6HBRNXR9GCsqmz+pquDUKfYqGOuNHVrmI5C4Qn3Rpa8M0bacmYevwsJdH0kTT19Yo@lists.infradead.org X-Gm-Message-State: AOJu0YxjDUswGcrqVOoutau27hs7zhN+TPT60bKAh7o9WsBF0y3ep01t mMDqulq6vysi7sN9z1bNE2iI8KhoxpxFp/yYfTqta0p0AYtmL/TxEYiar58CcDOC0gNwcdtSQbF kr7ixB19rTA== X-Google-Smtp-Source: AGHT+IF5/7KDUnK2hWYlKFk6TCuy51i/bgTYYOR6pwVpRa8Y8ICyfmquFNa7v3yriLE2ixCD8VdqiMqRbmEGkg== X-Received: from pfbcg8.prod.google.com ([2002:a05:6a00:2908:b0:725:d8bc:33e1]) (user=yepeilin job=prod-delivery.src-stubby-dispatcher) by 2002:a05:6a00:6812:b0:72d:b36a:4497 with SMTP id d2e1a72fcca58-72db36a45eemr40542229b3a.3.1737771575436; Fri, 24 Jan 2025 18:19:35 -0800 (PST) Date: Sat, 25 Jan 2025 02:19:30 +0000 In-Reply-To: Mime-Version: 1.0 References: X-Mailer: git-send-email 2.48.1.262.g85cc9f2d1e-goog Message-ID: <3f2de7c6e5d2def7bdfb091347c1dacea0915974.1737763916.git.yepeilin@google.com> Subject: [PATCH bpf-next v1 7/8] selftests/bpf: Add selftests for load-acquire and store-release instructions From: Peilin Ye To: bpf@vger.kernel.org, linux-arm-kernel@lists.infradead.org Cc: Peilin Ye , bpf@ietf.org, Xu Kuohai , Eduard Zingerman , David Vernet , Alexei Starovoitov , Daniel Borkmann , Andrii Nakryiko , Martin KaFai Lau , Song Liu , Yonghong Song , John Fastabend , KP Singh , Stanislav Fomichev , Hao Luo , Jiri Olsa , Jonathan Corbet , "Paul E. McKenney" , Puranjay Mohan , Catalin Marinas , Will Deacon , Quentin Monnet , Mykola Lysenko , Shuah Khan , Josh Don , Barret Rhoden , Neel Natu , Benjamin Segall , linux-kernel@vger.kernel.org X-CRM114-Version: 20100106-BlameMichelson ( TRE 0.8.0 (BSD) ) MR-646709E3 X-CRM114-CacheID: sfid-20250124_181936_689173_0C26A9BD X-CRM114-Status: GOOD ( 21.28 ) X-BeenThere: linux-arm-kernel@lists.infradead.org X-Mailman-Version: 2.1.34 Precedence: list List-Id: List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Sender: "linux-arm-kernel" Errors-To: linux-arm-kernel-bounces+linux-arm-kernel=archiver.kernel.org@lists.infradead.org Add several ./test_progs tests: - atomics/load_acquire - atomics/store_release - arena_atomics/load_acquire - arena_atomics/store_release - verifier_load_acquire/* - verifier_store_release/* - verifier_precision/bpf_load_acquire - verifier_precision/bpf_store_release The last two tests are added to check if backtrack_insn() handles the new instructions correctly. Additionally, the last test also makes sure that the verifier "remembers" the value (in src_reg) we store-release into e.g. a stack slot. For example, if we take a look at the test program: #0: "r1 = 8;" #1: "store_release((u64 *)(r10 - 8), r1);" #2: "r1 = *(u64 *)(r10 - 8);" #3: "r2 = r10;" #4: "r2 += r1;" /* mark_precise */ #5: "r0 = 0;" #6: "exit;" At #1, if the verifier doesn't remember that we wrote 8 to the stack, then later at #4 we would be adding an unbounded scalar value to the stack pointer, which would cause the program to be rejected: VERIFIER LOG: ============= ... math between fp pointer and register with unbounded min value is not allowed All new tests depend on the pre-defined __BPF_FEATURE_LOAD_ACQ_STORE_REL feature macro, which implies -mcpu>=v4. Signed-off-by: Peilin Ye --- .../selftests/bpf/prog_tests/arena_atomics.c | 61 ++++++- .../selftests/bpf/prog_tests/atomics.c | 57 ++++++- .../selftests/bpf/prog_tests/verifier.c | 4 + .../selftests/bpf/progs/arena_atomics.c | 62 ++++++- tools/testing/selftests/bpf/progs/atomics.c | 62 ++++++- .../bpf/progs/verifier_load_acquire.c | 92 +++++++++++ .../selftests/bpf/progs/verifier_precision.c | 39 +++++ .../bpf/progs/verifier_store_release.c | 153 ++++++++++++++++++ 8 files changed, 524 insertions(+), 6 deletions(-) create mode 100644 tools/testing/selftests/bpf/progs/verifier_load_acquire.c create mode 100644 tools/testing/selftests/bpf/progs/verifier_store_release.c diff --git a/tools/testing/selftests/bpf/prog_tests/arena_atomics.c b/tools/testing/selftests/bpf/prog_tests/arena_atomics.c index 26e7c06c6cb4..81d3575d7652 100644 --- a/tools/testing/selftests/bpf/prog_tests/arena_atomics.c +++ b/tools/testing/selftests/bpf/prog_tests/arena_atomics.c @@ -162,6 +162,60 @@ static void test_uaf(struct arena_atomics *skel) ASSERT_EQ(skel->arena->uaf_recovery_fails, 0, "uaf_recovery_fails"); } +static void test_load_acquire(struct arena_atomics *skel) +{ + LIBBPF_OPTS(bpf_test_run_opts, topts); + int err, prog_fd; + + if (skel->data->skip_lacq_srel_tests) { + printf("%s:SKIP:Clang does not support BPF load-acquire or addr_space_cast\n", + __func__); + test__skip(); + return; + } + + /* No need to attach it, just run it directly */ + prog_fd = bpf_program__fd(skel->progs.load_acquire); + err = bpf_prog_test_run_opts(prog_fd, &topts); + if (!ASSERT_OK(err, "test_run_opts err")) + return; + if (!ASSERT_OK(topts.retval, "test_run_opts retval")) + return; + + ASSERT_EQ(skel->arena->load_acquire8_result, 0x12, "load_acquire8_result"); + ASSERT_EQ(skel->arena->load_acquire16_result, 0x1234, "load_acquire16_result"); + ASSERT_EQ(skel->arena->load_acquire32_result, 0x12345678, "load_acquire32_result"); + ASSERT_EQ(skel->arena->load_acquire64_result, 0x1234567890abcdef, + "load_acquire64_result"); +} + +static void test_store_release(struct arena_atomics *skel) +{ + LIBBPF_OPTS(bpf_test_run_opts, topts); + int err, prog_fd; + + if (skel->data->skip_lacq_srel_tests) { + printf("%s:SKIP:Clang does not support BPF store-release or addr_space_cast\n", + __func__); + test__skip(); + return; + } + + /* No need to attach it, just run it directly */ + prog_fd = bpf_program__fd(skel->progs.store_release); + err = bpf_prog_test_run_opts(prog_fd, &topts); + if (!ASSERT_OK(err, "test_run_opts err")) + return; + if (!ASSERT_OK(topts.retval, "test_run_opts retval")) + return; + + ASSERT_EQ(skel->arena->store_release8_result, 0x12, "store_release8_result"); + ASSERT_EQ(skel->arena->store_release16_result, 0x1234, "store_release16_result"); + ASSERT_EQ(skel->arena->store_release32_result, 0x12345678, "store_release32_result"); + ASSERT_EQ(skel->arena->store_release64_result, 0x1234567890abcdef, + "store_release64_result"); +} + void test_arena_atomics(void) { struct arena_atomics *skel; @@ -171,7 +225,7 @@ void test_arena_atomics(void) if (!ASSERT_OK_PTR(skel, "arena atomics skeleton open")) return; - if (skel->data->skip_tests) { + if (skel->data->skip_all_tests) { printf("%s:SKIP:no ENABLE_ATOMICS_TESTS or no addr_space_cast support in clang", __func__); test__skip(); @@ -199,6 +253,11 @@ void test_arena_atomics(void) if (test__start_subtest("uaf")) test_uaf(skel); + if (test__start_subtest("load_acquire")) + test_load_acquire(skel); + if (test__start_subtest("store_release")) + test_store_release(skel); + cleanup: arena_atomics__destroy(skel); } diff --git a/tools/testing/selftests/bpf/prog_tests/atomics.c b/tools/testing/selftests/bpf/prog_tests/atomics.c index 13e101f370a1..5d7cff3eed2b 100644 --- a/tools/testing/selftests/bpf/prog_tests/atomics.c +++ b/tools/testing/selftests/bpf/prog_tests/atomics.c @@ -162,6 +162,56 @@ static void test_xchg(struct atomics_lskel *skel) ASSERT_EQ(skel->bss->xchg32_result, 1, "xchg32_result"); } +static void test_load_acquire(struct atomics_lskel *skel) +{ + LIBBPF_OPTS(bpf_test_run_opts, topts); + int err, prog_fd; + + if (skel->data->skip_lacq_srel_tests) { + printf("%s:SKIP:Clang does not support BPF load-acquire\n", __func__); + test__skip(); + return; + } + + /* No need to attach it, just run it directly */ + prog_fd = skel->progs.load_acquire.prog_fd; + err = bpf_prog_test_run_opts(prog_fd, &topts); + if (!ASSERT_OK(err, "test_run_opts err")) + return; + if (!ASSERT_OK(topts.retval, "test_run_opts retval")) + return; + + ASSERT_EQ(skel->bss->load_acquire8_result, 0x12, "load_acquire8_result"); + ASSERT_EQ(skel->bss->load_acquire16_result, 0x1234, "load_acquire16_result"); + ASSERT_EQ(skel->bss->load_acquire32_result, 0x12345678, "load_acquire32_result"); + ASSERT_EQ(skel->bss->load_acquire64_result, 0x1234567890abcdef, "load_acquire64_result"); +} + +static void test_store_release(struct atomics_lskel *skel) +{ + LIBBPF_OPTS(bpf_test_run_opts, topts); + int err, prog_fd; + + if (skel->data->skip_lacq_srel_tests) { + printf("%s:SKIP:Clang does not support BPF store-release\n", __func__); + test__skip(); + return; + } + + /* No need to attach it, just run it directly */ + prog_fd = skel->progs.store_release.prog_fd; + err = bpf_prog_test_run_opts(prog_fd, &topts); + if (!ASSERT_OK(err, "test_run_opts err")) + return; + if (!ASSERT_OK(topts.retval, "test_run_opts retval")) + return; + + ASSERT_EQ(skel->bss->store_release8_result, 0x12, "store_release8_result"); + ASSERT_EQ(skel->bss->store_release16_result, 0x1234, "store_release16_result"); + ASSERT_EQ(skel->bss->store_release32_result, 0x12345678, "store_release32_result"); + ASSERT_EQ(skel->bss->store_release64_result, 0x1234567890abcdef, "store_release64_result"); +} + void test_atomics(void) { struct atomics_lskel *skel; @@ -170,7 +220,7 @@ void test_atomics(void) if (!ASSERT_OK_PTR(skel, "atomics skeleton load")) return; - if (skel->data->skip_tests) { + if (skel->data->skip_all_tests) { printf("%s:SKIP:no ENABLE_ATOMICS_TESTS (missing Clang BPF atomics support)", __func__); test__skip(); @@ -193,6 +243,11 @@ void test_atomics(void) if (test__start_subtest("xchg")) test_xchg(skel); + if (test__start_subtest("load_acquire")) + test_load_acquire(skel); + if (test__start_subtest("store_release")) + test_store_release(skel); + cleanup: atomics_lskel__destroy(skel); } diff --git a/tools/testing/selftests/bpf/prog_tests/verifier.c b/tools/testing/selftests/bpf/prog_tests/verifier.c index 8a0e1ff8a2dc..8bdad4167cf5 100644 --- a/tools/testing/selftests/bpf/prog_tests/verifier.c +++ b/tools/testing/selftests/bpf/prog_tests/verifier.c @@ -45,6 +45,7 @@ #include "verifier_ldsx.skel.h" #include "verifier_leak_ptr.skel.h" #include "verifier_linked_scalars.skel.h" +#include "verifier_load_acquire.skel.h" #include "verifier_loops1.skel.h" #include "verifier_lwt.skel.h" #include "verifier_map_in_map.skel.h" @@ -80,6 +81,7 @@ #include "verifier_spill_fill.skel.h" #include "verifier_spin_lock.skel.h" #include "verifier_stack_ptr.skel.h" +#include "verifier_store_release.skel.h" #include "verifier_subprog_precision.skel.h" #include "verifier_subreg.skel.h" #include "verifier_tailcall_jit.skel.h" @@ -173,6 +175,7 @@ void test_verifier_int_ptr(void) { RUN(verifier_int_ptr); } void test_verifier_iterating_callbacks(void) { RUN(verifier_iterating_callbacks); } void test_verifier_jeq_infer_not_null(void) { RUN(verifier_jeq_infer_not_null); } void test_verifier_jit_convergence(void) { RUN(verifier_jit_convergence); } +void test_verifier_load_acquire(void) { RUN(verifier_load_acquire); } void test_verifier_ld_ind(void) { RUN(verifier_ld_ind); } void test_verifier_ldsx(void) { RUN(verifier_ldsx); } void test_verifier_leak_ptr(void) { RUN(verifier_leak_ptr); } @@ -211,6 +214,7 @@ void test_verifier_sockmap_mutate(void) { RUN(verifier_sockmap_mutate); } void test_verifier_spill_fill(void) { RUN(verifier_spill_fill); } void test_verifier_spin_lock(void) { RUN(verifier_spin_lock); } void test_verifier_stack_ptr(void) { RUN(verifier_stack_ptr); } +void test_verifier_store_release(void) { RUN(verifier_store_release); } void test_verifier_subprog_precision(void) { RUN(verifier_subprog_precision); } void test_verifier_subreg(void) { RUN(verifier_subreg); } void test_verifier_tailcall_jit(void) { RUN(verifier_tailcall_jit); } diff --git a/tools/testing/selftests/bpf/progs/arena_atomics.c b/tools/testing/selftests/bpf/progs/arena_atomics.c index 40dd57fca5cc..fe8b67d9c87b 100644 --- a/tools/testing/selftests/bpf/progs/arena_atomics.c +++ b/tools/testing/selftests/bpf/progs/arena_atomics.c @@ -19,9 +19,15 @@ struct { } arena SEC(".maps"); #if defined(ENABLE_ATOMICS_TESTS) && defined(__BPF_FEATURE_ADDR_SPACE_CAST) -bool skip_tests __attribute((__section__(".data"))) = false; +bool skip_all_tests __attribute((__section__(".data"))) = false; #else -bool skip_tests = true; +bool skip_all_tests = true; +#endif + +#if defined(__BPF_FEATURE_LOAD_ACQ_STORE_REL) && defined(__BPF_FEATURE_ADDR_SPACE_CAST) +bool skip_lacq_srel_tests __attribute((__section__(".data"))) = false; +#else +bool skip_lacq_srel_tests = true; #endif __u32 pid = 0; @@ -274,4 +280,56 @@ int uaf(const void *ctx) return 0; } +__u8 __arena_global load_acquire8_value = 0x12; +__u16 __arena_global load_acquire16_value = 0x1234; +__u32 __arena_global load_acquire32_value = 0x12345678; +__u64 __arena_global load_acquire64_value = 0x1234567890abcdef; + +__u8 __arena_global load_acquire8_result = 0; +__u16 __arena_global load_acquire16_result = 0; +__u32 __arena_global load_acquire32_result = 0; +__u64 __arena_global load_acquire64_result = 0; + +SEC("raw_tp/sys_enter") +int load_acquire(const void *ctx) +{ + if (pid != (bpf_get_current_pid_tgid() >> 32)) + return 0; + +#ifdef __BPF_FEATURE_LOAD_ACQ_STORE_REL + load_acquire8_result = __atomic_load_n(&load_acquire8_value, __ATOMIC_ACQUIRE); + load_acquire16_result = __atomic_load_n(&load_acquire16_value, __ATOMIC_ACQUIRE); + load_acquire32_result = __atomic_load_n(&load_acquire32_value, __ATOMIC_ACQUIRE); + load_acquire64_result = __atomic_load_n(&load_acquire64_value, __ATOMIC_ACQUIRE); +#endif + + return 0; +} + +__u8 __arena_global store_release8_result = 0; +__u16 __arena_global store_release16_result = 0; +__u32 __arena_global store_release32_result = 0; +__u64 __arena_global store_release64_result = 0; + +SEC("raw_tp/sys_enter") +int store_release(const void *ctx) +{ + if (pid != (bpf_get_current_pid_tgid() >> 32)) + return 0; + +#ifdef __BPF_FEATURE_LOAD_ACQ_STORE_REL + __u8 val8 = 0x12; + __u16 val16 = 0x1234; + __u32 val32 = 0x12345678; + __u64 val64 = 0x1234567890abcdef; + + __atomic_store_n(&store_release8_result, val8, __ATOMIC_RELEASE); + __atomic_store_n(&store_release16_result, val16, __ATOMIC_RELEASE); + __atomic_store_n(&store_release32_result, val32, __ATOMIC_RELEASE); + __atomic_store_n(&store_release64_result, val64, __ATOMIC_RELEASE); +#endif + + return 0; +} + char _license[] SEC("license") = "GPL"; diff --git a/tools/testing/selftests/bpf/progs/atomics.c b/tools/testing/selftests/bpf/progs/atomics.c index f89c7f0cc53b..4c23d7d0d37d 100644 --- a/tools/testing/selftests/bpf/progs/atomics.c +++ b/tools/testing/selftests/bpf/progs/atomics.c @@ -5,9 +5,15 @@ #include #ifdef ENABLE_ATOMICS_TESTS -bool skip_tests __attribute((__section__(".data"))) = false; +bool skip_all_tests __attribute((__section__(".data"))) = false; #else -bool skip_tests = true; +bool skip_all_tests = true; +#endif + +#ifdef __BPF_FEATURE_LOAD_ACQ_STORE_REL +bool skip_lacq_srel_tests __attribute((__section__(".data"))) = false; +#else +bool skip_lacq_srel_tests = true; #endif __u32 pid = 0; @@ -168,3 +174,55 @@ int xchg(const void *ctx) return 0; } + +__u8 load_acquire8_value = 0x12; +__u16 load_acquire16_value = 0x1234; +__u32 load_acquire32_value = 0x12345678; +__u64 load_acquire64_value = 0x1234567890abcdef; + +__u8 load_acquire8_result = 0; +__u16 load_acquire16_result = 0; +__u32 load_acquire32_result = 0; +__u64 load_acquire64_result = 0; + +SEC("raw_tp/sys_enter") +int load_acquire(const void *ctx) +{ + if (pid != (bpf_get_current_pid_tgid() >> 32)) + return 0; + +#ifdef __BPF_FEATURE_LOAD_ACQ_STORE_REL + load_acquire8_result = __atomic_load_n(&load_acquire8_value, __ATOMIC_ACQUIRE); + load_acquire16_result = __atomic_load_n(&load_acquire16_value, __ATOMIC_ACQUIRE); + load_acquire32_result = __atomic_load_n(&load_acquire32_value, __ATOMIC_ACQUIRE); + load_acquire64_result = __atomic_load_n(&load_acquire64_value, __ATOMIC_ACQUIRE); +#endif + + return 0; +} + +__u8 store_release8_result = 0; +__u16 store_release16_result = 0; +__u32 store_release32_result = 0; +__u64 store_release64_result = 0; + +SEC("raw_tp/sys_enter") +int store_release(const void *ctx) +{ + if (pid != (bpf_get_current_pid_tgid() >> 32)) + return 0; + +#ifdef __BPF_FEATURE_LOAD_ACQ_STORE_REL + __u8 val8 = 0x12; + __u16 val16 = 0x1234; + __u32 val32 = 0x12345678; + __u64 val64 = 0x1234567890abcdef; + + __atomic_store_n(&store_release8_result, val8, __ATOMIC_RELEASE); + __atomic_store_n(&store_release16_result, val16, __ATOMIC_RELEASE); + __atomic_store_n(&store_release32_result, val32, __ATOMIC_RELEASE); + __atomic_store_n(&store_release64_result, val64, __ATOMIC_RELEASE); +#endif + + return 0; +} diff --git a/tools/testing/selftests/bpf/progs/verifier_load_acquire.c b/tools/testing/selftests/bpf/progs/verifier_load_acquire.c new file mode 100644 index 000000000000..506df4b8231b --- /dev/null +++ b/tools/testing/selftests/bpf/progs/verifier_load_acquire.c @@ -0,0 +1,92 @@ +// SPDX-License-Identifier: GPL-2.0 + +#include +#include +#include "bpf_misc.h" + +#if defined(__TARGET_ARCH_arm64) && defined(__BPF_FEATURE_LOAD_ACQ_STORE_REL) + +SEC("socket") +__description("load-acquire, 8-bit") +__success __success_unpriv __retval(0x12) +__naked void load_acquire_8(void) +{ + asm volatile ( + "*(u8 *)(r10 - 1) = 0x12;" + "w0 = load_acquire((u8 *)(r10 - 1));" + "exit;" + ::: __clobber_all); +} + +SEC("socket") +__description("load-acquire, 16-bit") +__success __success_unpriv __retval(0x1234) +__naked void load_acquire_16(void) +{ + asm volatile ( + "*(u16 *)(r10 - 2) = 0x1234;" + "w0 = load_acquire((u16 *)(r10 - 2));" + "exit;" + ::: __clobber_all); +} + +SEC("socket") +__description("load-acquire, 32-bit") +__success __success_unpriv __retval(0x12345678) +__naked void load_acquire_32(void) +{ + asm volatile ( + "*(u32 *)(r10 - 4) = 0x12345678;" + "w0 = load_acquire((u32 *)(r10 - 4));" + "exit;" + ::: __clobber_all); +} + +SEC("socket") +__description("load-acquire, 64-bit") +__success __success_unpriv __retval(0x1234567890abcdef) +__naked void load_acquire_64(void) +{ + asm volatile ( + "*(u64 *)(r10 - 8) = 0x1234567890abcdef;" + "r0 = load_acquire((u64 *)(r10 - 8));" + "exit;" + ::: __clobber_all); +} + +SEC("socket") +__description("load-acquire with uninitialized src_reg") +__failure __failure_unpriv __msg("R2 !read_ok") +__naked void load_acquire_with_uninitialized_src_reg(void) +{ + asm volatile ( + "r0 = load_acquire((u64 *)(r2 + 0));" + "exit;" + ::: __clobber_all); +} + +SEC("socket") +__description("load-acquire with non-pointer src_reg") +__failure __failure_unpriv __msg("R1 invalid mem access 'scalar'") +__naked void load_acquire_with_non_pointer_src_reg(void) +{ + asm volatile ( + "r1 = 0;" + "r0 = load_acquire((u64 *)(r1 + 0));" + "exit;" + ::: __clobber_all); +} + +#else + +SEC("socket") +__description("load-acquire is not supported by compiler or jit, use a dummy test") +__success +int dummy_test(void) +{ + return 0; +} + +#endif + +char _license[] SEC("license") = "GPL"; diff --git a/tools/testing/selftests/bpf/progs/verifier_precision.c b/tools/testing/selftests/bpf/progs/verifier_precision.c index 6b564d4c0986..7d5b9e95e3cf 100644 --- a/tools/testing/selftests/bpf/progs/verifier_precision.c +++ b/tools/testing/selftests/bpf/progs/verifier_precision.c @@ -90,6 +90,45 @@ __naked int bpf_end_bswap(void) ::: __clobber_all); } +#if defined(__TARGET_ARCH_arm64) && defined(__BPF_FEATURE_LOAD_ACQ_STORE_REL) + +SEC("?raw_tp") +__success __log_level(2) +__msg("mark_precise: frame0: regs=r1 stack= before 2: (bf) r2 = r10") +__msg("mark_precise: frame0: regs=r1 stack= before 1: (db) r1 = load_acquire((u64 *)(r10 -8))") +__msg("mark_precise: frame0: regs= stack=-8 before 0: (7a) *(u64 *)(r10 -8) = 8") +__naked int bpf_load_acquire(void) +{ + asm volatile ( + "*(u64 *)(r10 - 8) = 8;" + "r1 = load_acquire((u64 *)(r10 - 8));" + "r2 = r10;" + "r2 += r1;" /* mark_precise */ + "r0 = 0;" + "exit;" + ::: __clobber_all); +} + +SEC("?raw_tp") +__success __log_level(2) +__msg("mark_precise: frame0: regs=r1 stack= before 3: (bf) r2 = r10") +__msg("mark_precise: frame0: regs=r1 stack= before 2: (79) r1 = *(u64 *)(r10 -8)") +__msg("mark_precise: frame0: regs= stack=-8 before 1: (db) store_release((u64 *)(r10 -8), r1)") +__msg("mark_precise: frame0: regs=r1 stack= before 0: (b7) r1 = 8") +__naked int bpf_store_release(void) +{ + asm volatile ( + "r1 = 8;" + "store_release((u64 *)(r10 - 8), r1);" + "r1 = *(u64 *)(r10 - 8);" + "r2 = r10;" + "r2 += r1;" /* mark_precise */ + "r0 = 0;" + "exit;" + ::: __clobber_all); +} + +#endif /* load-acquire, store-release */ #endif /* v4 instruction */ SEC("?raw_tp") diff --git a/tools/testing/selftests/bpf/progs/verifier_store_release.c b/tools/testing/selftests/bpf/progs/verifier_store_release.c new file mode 100644 index 000000000000..d8c3b73388cb --- /dev/null +++ b/tools/testing/selftests/bpf/progs/verifier_store_release.c @@ -0,0 +1,153 @@ +// SPDX-License-Identifier: GPL-2.0 + +#include +#include +#include "bpf_misc.h" + +#if defined(__TARGET_ARCH_arm64) && defined(__BPF_FEATURE_LOAD_ACQ_STORE_REL) + +SEC("socket") +__description("store-release, 8-bit") +__success __success_unpriv __retval(0x12) +__naked void store_release_8(void) +{ + asm volatile ( + "w1 = 0x12;" + "store_release((u8 *)(r10 - 1), w1);" + "w0 = *(u8 *)(r10 - 1);" + "exit;" + ::: __clobber_all); +} + +SEC("socket") +__description("store-release, 16-bit") +__success __success_unpriv __retval(0x1234) +__naked void store_release_16(void) +{ + asm volatile ( + "w1 = 0x1234;" + "store_release((u16 *)(r10 - 2), w1);" + "w0 = *(u16 *)(r10 - 2);" + "exit;" + ::: __clobber_all); +} + +SEC("socket") +__description("store-release, 32-bit") +__success __success_unpriv __retval(0x12345678) +__naked void store_release_32(void) +{ + asm volatile ( + "w1 = 0x12345678;" + "store_release((u32 *)(r10 - 4), w1);" + "w0 = *(u32 *)(r10 - 4);" + "exit;" + ::: __clobber_all); +} + +SEC("socket") +__description("store-release, 64-bit") +__success __success_unpriv __retval(0x1234567890abcdef) +__naked void store_release_64(void) +{ + asm volatile ( + "r1 = 0x1234567890abcdef;" + "store_release((u64 *)(r10 - 8), r1);" + "r0 = *(u64 *)(r10 - 8);" + "exit;" + ::: __clobber_all); +} + +SEC("socket") +__description("store-release with uninitialized src_reg") +__failure __failure_unpriv __msg("R2 !read_ok") +__naked void store_release_with_uninitialized_src_reg(void) +{ + asm volatile ( + "store_release((u64 *)(r10 - 8), r2);" + "exit;" + ::: __clobber_all); +} + +SEC("socket") +__description("store-release with uninitialized dst_reg") +__failure __failure_unpriv __msg("R2 !read_ok") +__naked void store_release_with_uninitialized_dst_reg(void) +{ + asm volatile ( + "r1 = 0x1234567890abcdef;" + "store_release((u64 *)(r2 - 8), r1);" + "exit;" + ::: __clobber_all); +} + +SEC("socket") +__description("store-release with non-pointer dst_reg") +__failure __failure_unpriv __msg("R1 invalid mem access 'scalar'") +__naked void store_release_with_non_pointer_dst_reg(void) +{ + asm volatile ( + "r1 = 0;" + "store_release((u64 *)(r1 + 0), r1);" + "exit;" + ::: __clobber_all); +} + +SEC("socket") +__description("store-release, leak pointer to stack") +__success __retval(0) +__failure_unpriv __msg_unpriv("R1 leaks addr into mem") +__naked void store_release_leak_pointer_to_stack(void) +{ + asm volatile ( + "store_release((u64 *)(r10 - 8), r1);" + "r0 = 0;" + "exit;" + ::: __clobber_all); +} + +struct { + __uint(type, BPF_MAP_TYPE_HASH); + __uint(max_entries, 1); + __type(key, long long); + __type(value, long long); +} map_hash_8b SEC(".maps"); + +SEC("socket") +__description("store-release, leak pointer to map") +__success __retval(0) +__failure_unpriv __msg_unpriv("R6 leaks addr into mem") +__naked void store_release_leak_pointer_to_map(void) +{ + asm volatile ( + "r6 = r1;" + "r1 = 0;" + "*(u64 *)(r10 - 8) = r1;" + "r2 = r10;" + "r2 += -8;" + "r1 = %[map_hash_8b] ll;" + "call %[bpf_map_lookup_elem];" + "if r0 == 0 goto l0_%=;" + "store_release((u64 *)(r0 + 0), r6);" +"l0_%=:" + "r0 = 0;" + "exit;" + : + : __imm(bpf_map_lookup_elem), + __imm_addr(map_hash_8b) + : __clobber_all); +} + +#else + +SEC("socket") +__description("store-release is not supported by compiler or jit, use a dummy test") +__success +int dummy_test(void) +{ + return 0; +} + +#endif + +char _license[] SEC("license") = "GPL"; From patchwork Sat Jan 25 02:19:39 2025 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Peilin Ye X-Patchwork-Id: 13950121 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from bombadil.infradead.org (bombadil.infradead.org [198.137.202.133]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (No client certificate requested) by smtp.lore.kernel.org (Postfix) with ESMTPS id 9A19EC02181 for ; Sat, 25 Jan 2025 02:29:27 +0000 (UTC) DKIM-Signature: v=1; a=rsa-sha256; q=dns/txt; c=relaxed/relaxed; d=lists.infradead.org; s=bombadil.20210309; h=Sender:List-Subscribe:List-Help :List-Post:List-Archive:List-Unsubscribe:List-Id:Content-Type:Cc:To:From: Subject:Message-ID:References:Mime-Version:In-Reply-To:Date:Reply-To: Content-Transfer-Encoding:Content-ID:Content-Description:Resent-Date: Resent-From:Resent-Sender:Resent-To:Resent-Cc:Resent-Message-ID:List-Owner; bh=LoHUEUhOV0x0mPYHihiLdSKtz41TIEtVbSj3jmRsaOk=; b=zB/SOd2/aarrLHi1vZpsRuxkQr /r0pjQRJ9avinAsWNHwrdEbmNqDjOAy9i86qMuHfsj43orG1KqLxWyAqmnaFouk7jr7+g6udmzoZq /b7Tp3c1YldG6fH8Erw4x8WZSbNAQJQ4/aZM8VK7Sb7W4O6Bn9S6O+cGotc+w+ULoLmqSWOmsaNt6 aT1hZb6OYYUW0DqVf3lBaARUYL4eI0Fozw7o4FyYGVOdMvoRAekrbkcGvsN4DHaRcuy6vvEkOipyv 1vcr/Z2NuIoBsE9FpgHrSSLszidnvIkyJvrEA1bDa00Q/3GxtKyjdb/q/o1mfLo2Uu6jovqUsZL+O iATd6iTA==; Received: from localhost ([::1] helo=bombadil.infradead.org) by bombadil.infradead.org with esmtp (Exim 4.98 #2 (Red Hat Linux)) id 1tbVv8-0000000Fuwl-13gZ; Sat, 25 Jan 2025 02:29:14 +0000 Received: from mail-pl1-x649.google.com ([2607:f8b0:4864:20::649]) by bombadil.infradead.org with esmtps (Exim 4.98 #2 (Red Hat Linux)) id 1tbVm1-0000000FuC6-0PpC for linux-arm-kernel@lists.infradead.org; Sat, 25 Jan 2025 02:19:50 +0000 Received: by mail-pl1-x649.google.com with SMTP id d9443c01a7336-2166f9f52fbso80459685ad.2 for ; Fri, 24 Jan 2025 18:19:48 -0800 (PST) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=google.com; s=20230601; t=1737771588; x=1738376388; darn=lists.infradead.org; h=cc:to:from:subject:message-id:references:mime-version:in-reply-to :date:from:to:cc:subject:date:message-id:reply-to; bh=LoHUEUhOV0x0mPYHihiLdSKtz41TIEtVbSj3jmRsaOk=; b=Em7pLlRC00SpJgYVCPpUkflLHkl7mL5M3OMmKOf4eBS1CQgGxJTuJ6W+QB8RElxchf v4CN71Jw4rIyxhdDvyofjX4urxONlXXEw2rUToiHcytYHhnYwdYdEltvLrh55CzpqAyX 1GxfFP9MlD4HLj7dD3M4L6UVnXO6aAjgy8T0dZJhk+DjypthsK/dBJDiU2rKGiWVtKP9 ufPJh+DgC2XsAcnSnXcRwrB/uXoNEt4eUudE85GqE0oW5O+DmvZmUFyg9fswH73uzaW8 QiHDyMBZwJFz0xPQiI2aCBzY8Cg/qM7szPAWSZwxKtB1cU+Nr9XoqnGW7jMvecQrbfW+ JH4w== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20230601; t=1737771588; x=1738376388; h=cc:to:from:subject:message-id:references:mime-version:in-reply-to :date:x-gm-message-state:from:to:cc:subject:date:message-id:reply-to; bh=LoHUEUhOV0x0mPYHihiLdSKtz41TIEtVbSj3jmRsaOk=; b=GN3azL0574OK2Q5s/Bm7D5rLG6fKFuR7KJIiZZLsLNfAomm1SWdtJnUoPB0WOhzx0A iOXWYzssCFNHRClenWlsvU3gMe0dSkOL48ZQjOgvAmVjefXsML5Jetapl/nc7foOg60Y z7DXE2f2/p57jh6tP+pDAIWA3UzI1ayUlYnEYqQnNsjBjaU54nGBRpTUNTuE9OLZtd3J pSXvuDs5vAfkU5V/J1ZWYonWTux6bJDMVekiJ2Trhcgwb7KEZGrHtA1aMC2/zP4C+MLE 66Vx+XiVHFPRTOjevFtcyMf+P7orRSMrJ2n7+TmpBhyYij/hy05IHdU/yHlJltbLGPYL HNmA== X-Forwarded-Encrypted: i=1; AJvYcCVeKo4MHN+lIMW0OzaDZuL1fCtsxcHUDdFf+K9/F7QPCqfISLBgG1BoKbeS8oDFjN26OPlsi4yEys7HY0YGClV6@lists.infradead.org X-Gm-Message-State: AOJu0Yw458LrovV8nbyr1FprY4mnlFNXK+yn6lR/Lpg9pqynH//RFKv7 8njZwoO+nsK7Jq4HeTM14Q/60Qppsa0KziSYP9ZKJNNhlKOWHIv1GpxpCaRf02UybequfykCXN1 2Z82ly2/8NQ== X-Google-Smtp-Source: AGHT+IHGh/EcJQlJc3OUFLXhqDr5SV+ajo77ZpLxRt1CFaw1WVG3Fg0XoLCrcpjLJvkD2sLS2HovJh1AnpwYjQ== X-Received: from plez12.prod.google.com ([2002:a17:902:cccc:b0:215:dbd8:a6e7]) (user=yepeilin job=prod-delivery.src-stubby-dispatcher) by 2002:a17:903:1381:b0:216:361a:783d with SMTP id d9443c01a7336-21c3562097amr541898135ad.28.1737771587999; Fri, 24 Jan 2025 18:19:47 -0800 (PST) Date: Sat, 25 Jan 2025 02:19:39 +0000 In-Reply-To: Mime-Version: 1.0 References: X-Mailer: git-send-email 2.48.1.262.g85cc9f2d1e-goog Message-ID: Subject: [PATCH bpf-next v1 8/8] bpf, docs: Update instruction-set.rst for load-acquire and store-release instructions From: Peilin Ye To: bpf@vger.kernel.org, linux-arm-kernel@lists.infradead.org Cc: Peilin Ye , bpf@ietf.org, Xu Kuohai , Eduard Zingerman , David Vernet , Alexei Starovoitov , Daniel Borkmann , Andrii Nakryiko , Martin KaFai Lau , Song Liu , Yonghong Song , John Fastabend , KP Singh , Stanislav Fomichev , Hao Luo , Jiri Olsa , Jonathan Corbet , "Paul E. McKenney" , Puranjay Mohan , Catalin Marinas , Will Deacon , Quentin Monnet , Mykola Lysenko , Shuah Khan , Josh Don , Barret Rhoden , Neel Natu , Benjamin Segall , linux-kernel@vger.kernel.org X-CRM114-Version: 20100106-BlameMichelson ( TRE 0.8.0 (BSD) ) MR-646709E3 X-CRM114-CacheID: sfid-20250124_181949_138232_B0DB7F3D X-CRM114-Status: GOOD ( 14.66 ) X-BeenThere: linux-arm-kernel@lists.infradead.org X-Mailman-Version: 2.1.34 Precedence: list List-Id: List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Sender: "linux-arm-kernel" Errors-To: linux-arm-kernel-bounces+linux-arm-kernel=archiver.kernel.org@lists.infradead.org Update documentation for the new load-acquire and store-release instructions. Rename existing atomic operations as "atomic read-modify-write (RMW) operations". Following RFC 9669, section 7.3. "Adding Instructions", create new conformance groups "atomic32v2" and "atomic64v2", where: * atomic32v2: includes all instructions in "atomic32", plus the new 8-bit, 16-bit and 32-bit atomic load-acquire and store-release instructions * atomic64v2: includes all instructions in "atomic64" and "atomic32v2", plus the new 64-bit atomic load-acquire and store-release instructions Cc: bpf@ietf.org Signed-off-by: Peilin Ye --- .../bpf/standardization/instruction-set.rst | 114 +++++++++++++++--- 1 file changed, 98 insertions(+), 16 deletions(-) diff --git a/Documentation/bpf/standardization/instruction-set.rst b/Documentation/bpf/standardization/instruction-set.rst index ab820d565052..86917932e9ef 100644 --- a/Documentation/bpf/standardization/instruction-set.rst +++ b/Documentation/bpf/standardization/instruction-set.rst @@ -139,8 +139,14 @@ This document defines the following conformance groups: specification unless otherwise noted. * base64: includes base32, plus instructions explicitly noted as being in the base64 conformance group. -* atomic32: includes 32-bit atomic operation instructions (see `Atomic operations`_). -* atomic64: includes atomic32, plus 64-bit atomic operation instructions. +* atomic32: includes 32-bit atomic read-modify-write instructions (see + `Atomic operations`_). +* atomic32v2: includes atomic32, plus 8-bit, 16-bit and 32-bit atomic + load-acquire and store-release instructions. +* atomic64: includes atomic32, plus 64-bit atomic read-modify-write + instructions. +* atomic64v2: unifies atomic32v2 and atomic64, plus 64-bit atomic load-acquire + and store-release instructions. * divmul32: includes 32-bit division, multiplication, and modulo instructions. * divmul64: includes divmul32, plus 64-bit division, multiplication, and modulo instructions. @@ -653,20 +659,31 @@ Atomic operations are operations that operate on memory and can not be interrupted or corrupted by other access to the same memory region by other BPF programs or means outside of this specification. -All atomic operations supported by BPF are encoded as store operations -that use the ``ATOMIC`` mode modifier as follows: +All atomic operations supported by BPF are encoded as ``STX`` instructions +that use the ``ATOMIC`` mode modifier, with the 'imm' field encoding the +actual atomic operation. These operations are categorized based on the second +lowest nibble (bits 4-7) of the 'imm' field: -* ``{ATOMIC, W, STX}`` for 32-bit operations, which are +* ``ATOMIC_LOAD`` and ``ATOMIC_STORE`` indicate atomic load and store + operations, respectively (see `Atomic load and store operations`_). +* All other defined values indicate an atomic read-modify-write operation, as + described in the following section. + +Atomic read-modify-write operations +~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~ + +The atomic read-modify-write (RMW) operations are encoded as follows: + +* ``{ATOMIC, W, STX}`` for 32-bit RMW operations, which are part of the "atomic32" conformance group. -* ``{ATOMIC, DW, STX}`` for 64-bit operations, which are +* ``{ATOMIC, DW, STX}`` for 64-bit RMW operations, which are part of the "atomic64" conformance group. -* 8-bit and 16-bit wide atomic operations are not supported. +* 8-bit and 16-bit wide atomic RMW operations are not supported. -The 'imm' field is used to encode the actual atomic operation. -Simple atomic operation use a subset of the values defined to encode -arithmetic operations in the 'imm' field to encode the atomic operation: +Simple atomic RMW operation use a subset of the values defined to encode +arithmetic operations in the 'imm' field to encode the atomic RMW operation: -.. table:: Simple atomic operations +.. table:: Simple atomic read-modify-write operations ======== ===== =========== imm value description @@ -686,10 +703,10 @@ arithmetic operations in the 'imm' field to encode the atomic operation: *(u64 *)(dst + offset) += src -In addition to the simple atomic operations, there also is a modifier and -two complex atomic operations: +In addition to the simple atomic RMW operations, there also is a modifier and +two complex atomic RMW operations: -.. table:: Complex atomic operations +.. table:: Complex atomic read-modify-write operations =========== ================ =========================== imm value description @@ -699,8 +716,8 @@ two complex atomic operations: CMPXCHG 0xf0 | FETCH atomic compare and exchange =========== ================ =========================== -The ``FETCH`` modifier is optional for simple atomic operations, and -always set for the complex atomic operations. If the ``FETCH`` flag +The ``FETCH`` modifier is optional for simple atomic RMW operations, and +always set for the complex atomic RMW operations. If the ``FETCH`` flag is set, then the operation also overwrites ``src`` with the value that was in memory before it was modified. @@ -713,6 +730,71 @@ The ``CMPXCHG`` operation atomically compares the value addressed by value that was at ``dst + offset`` before the operation is zero-extended and loaded back to ``R0``. +Atomic load and store operations +~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~ + +To encode an atomic load or store operation, the lowest 8 bits of the 'imm' +field are divided as follows:: + + +-+-+-+-+-+-+-+-+ + | type | order | + +-+-+-+-+-+-+-+-+ + +**type** + The operation type is one of: + +.. table:: Atomic load and store operation types + + ============ ===== ============ + type value description + ============ ===== ============ + ATOMIC_LOAD 0x1 atomic load + ATOMIC_STORE 0x2 atomic store + ============ ===== ============ + +**order** + The memory order is one of: + +.. table:: Memory orders + + ======= ===== ======================= + order value description + ======= ===== ======================= + RELAXED 0x0 relaxed + ACQUIRE 0x1 acquire + RELEASE 0x2 release + ACQ_REL 0x3 acquire and release + SEQ_CST 0x4 sequentially consistent + ======= ===== ======================= + +Currently the following combinations of ``type`` and ``order`` are allowed: + +.. table:: Atomic load and store operations + + ========= ===== ==================== + imm value description + ========= ===== ==================== + LOAD_ACQ 0x11 atomic load-acquire + STORE_REL 0x22 atomic store-release + ========= ===== ==================== + +``{ATOMIC, , STX}`` with 'imm' = LOAD_ACQ means:: + + dst = load_acquire((unsigned size *)(src + offset)) + +``{ATOMIC, , STX}`` with 'imm' = STORE_REL means:: + + store_release((unsigned size *)(dst + offset), src) + +Where '' is one of: ``B``, ``H``, ``W``, or ``DW``, and 'unsigned size' +is one of: u8, u16, u32, or u64. + +8-bit, 16-bit and 32-bit atomic load-acquire and store-release instructions +are part of the "atomic32v2" conformance group. + +64-bit atomic load-acquire and store-release instructions are part of the +"atomic64v2" conformance group. + 64-bit immediate instructions -----------------------------