[bpf-next,v2] s390/bpf: Emit a barrier for BPF_FETCH instructions

Commit Message

Ilya Leoshkevich May 7, 2024, 12:02 a.m. UTC

BPF_ATOMIC_OP() macro documentation states that "BPF_ADD | BPF_FETCH"
should be the same as atomic_fetch_add(), which is currently not the
case on s390x: the serialization instruction "bcr 14,0" is missing.
This applies to "and", "or" and "xor" variants too.

s390x is allowed to reorder stores with subsequent fetches from
different addresses, so code relying on BPF_FETCH acting as a barrier,
for example:

  stw [%r0], 1
  afadd [%r1], %r2
  ldxw %r3, [%r4]

may be broken. Fix it by emitting "bcr 14,0".

Note that a separate serialization instruction is not needed for
BPF_XCHG and BPF_CMPXCHG, because COMPARE AND SWAP performs
serialization itself.

Fixes: ba3b86b9cef0 ("s390/bpf: Implement new atomic ops")
Reported-by: Puranjay Mohan <[e-mail]>
Closes: https://lore.kernel.org/bpf/[e-mail]/
Signed-off-by: Ilya Leoshkevich <[e-mail]>
---

v1: https://lore.kernel.org/bpf/[e-mail]/
v1 -> v2: Emit a barrier only for BPF_FETCH variants;
          Add an example of the code that may break to the commit
          message (Puranjay).

 arch/s390/net/bpf_jit_comp.c | 8 ++++++--
 1 file changed, 6 insertions(+), 2 deletions(-)

Comments

Puranjay Mohan May 7, 2024, 11:10 a.m. UTC | #1

Ilya Leoshkevich <[e-mail]> writes:

> BPF_ATOMIC_OP() macro documentation states that "BPF_ADD | BPF_FETCH"
> should be the same as atomic_fetch_add(), which is currently not the
> case on s390x: the serialization instruction "bcr 14,0" is missing.
> This applies to "and", "or" and "xor" variants too.
>
> s390x is allowed to reorder stores with subsequent fetches from
> different addresses, so code relying on BPF_FETCH acting as a barrier,
> for example:
>
>   stw [%r0], 1
>   afadd [%r1], %r2
>   ldxw %r3, [%r4]
>
> may be broken. Fix it by emitting "bcr 14,0".
>
> Note that a separate serialization instruction is not needed for
> BPF_XCHG and BPF_CMPXCHG, because COMPARE AND SWAP performs
> serialization itself.
>
> Fixes: ba3b86b9cef0 ("s390/bpf: Implement new atomic ops")
> Reported-by: Puranjay Mohan <[e-mail]>
> Closes: https://lore.kernel.org/bpf/[e-mail]/
> Signed-off-by: Ilya Leoshkevich <[e-mail]>

Reviewed-by: Puranjay Mohan <[e-mail]>

Thanks,
Puranjay

[e-mail] May 13, 2024, midnight UTC | #2

Hello:

This patch was applied to bpf/bpf-next.git (master)
by Alexei Starovoitov <[e-mail]>:

On Tue,  7 May 2024 02:02:49 +0200 you wrote:
> BPF_ATOMIC_OP() macro documentation states that "BPF_ADD | BPF_FETCH"
> should be the same as atomic_fetch_add(), which is currently not the
> case on s390x: the serialization instruction "bcr 14,0" is missing.
> This applies to "and", "or" and "xor" variants too.
> 
> s390x is allowed to reorder stores with subsequent fetches from
> different addresses, so code relying on BPF_FETCH acting as a barrier,
> for example:
> 
> [...]

Here is the summary with links:
  - [bpf-next,v2] s390/bpf: Emit a barrier for BPF_FETCH instructions
    https://git.kernel.org/bpf/bpf-next/c/68378982f0b2

You are awesome, thank you!

Patch

diff --git a/arch/s390/net/bpf_jit_comp.c b/arch/s390/net/bpf_jit_comp.c
index fa2f824e3b06..4be8f5cadd02 100644
--- a/arch/s390/net/bpf_jit_comp.c
+++ b/arch/s390/net/bpf_jit_comp.c
@@ -1427,8 +1427,12 @@  static noinline int bpf_jit_insn(struct bpf_jit *jit, struct bpf_prog *fp,
 	EMIT6_DISP_LH(0xeb000000, is32 ? (op32) : (op64),		\
 		      (insn->imm & BPF_FETCH) ? src_reg : REG_W0,	\
 		      src_reg, dst_reg, off);				\
-	if (is32 && (insn->imm & BPF_FETCH))				\
-		EMIT_ZERO(src_reg);					\
+	if (insn->imm & BPF_FETCH) {					\
+		/* bcr 14,0 - see atomic_fetch_{add,and,or,xor}() */	\
+		_EMIT2(0x07e0);						\
+		if (is32)                                               \
+			EMIT_ZERO(src_reg);				\
+	}								\
 } while (0)
 		case BPF_ADD:
 		case BPF_ADD | BPF_FETCH: