[bpf] riscv, bpf: Emit fixed-length imm64 for BPF_PSEUDO_FUNC

Commit Message

Pu Lehui Nov. 30, 2022, 3:38 a.m. UTC

From: Pu Lehui <[e-mail]>

For BPF_PSEUDO_FUNC instruction, verifier will refill imm with
correct addresses of bpf_calls and then run last pass of JIT.
Since the emit_imm of RV64 is variable-length, which will emit
appropriate length instructions accorroding to the imm, it may
broke ctx->offset, and lead to unpredictable problem, such as
inaccurate jump. So let's fix it with fixed-length imm64 insns.

Fixes: 69c087ba6225 ("bpf: Add bpf_for_each_map_elem() helper")
Signed-off-by: Pu Lehui <[e-mail]>
---
 arch/riscv/net/bpf_jit_comp64.c | 31 ++++++++++++++++++++++++++++++-
 1 file changed, 30 insertions(+), 1 deletion(-)

Comments

Björn Töpel Nov. 30, 2022, 11:38 a.m. UTC | #1

Pu Lehui <[e-mail]> writes:

> From: Pu Lehui <[e-mail]>
>
> For BPF_PSEUDO_FUNC instruction, verifier will refill imm with
> correct addresses of bpf_calls and then run last pass of JIT.
> Since the emit_imm of RV64 is variable-length, which will emit
> appropriate length instructions accorroding to the imm, it may
> broke ctx->offset, and lead to unpredictable problem, such as
> inaccurate jump. So let's fix it with fixed-length imm64 insns.

Ah, nice one! So, the the invariant doesn't hold (the image grow in the
last pass).

> Fixes: 69c087ba6225 ("bpf: Add bpf_for_each_map_elem() helper")

This is odd? This can't be the right Fixes-tag...

> Signed-off-by: Pu Lehui <[e-mail]>
> ---
>  arch/riscv/net/bpf_jit_comp64.c | 31 ++++++++++++++++++++++++++++++-
>  1 file changed, 30 insertions(+), 1 deletion(-)
>
> diff --git a/arch/riscv/net/bpf_jit_comp64.c b/arch/riscv/net/bpf_jit_comp64.c
> index eb99df41fa33..f984d5fa014b 100644
> --- a/arch/riscv/net/bpf_jit_comp64.c
> +++ b/arch/riscv/net/bpf_jit_comp64.c
> @@ -139,6 +139,30 @@ static bool in_auipc_jalr_range(s64 val)
>  		val < ((1L << 31) - (1L << 11));
>  }
>  
> +/* Emit fixed-length instructions for 32-bit imm */
> +static void emit_fixed_imm32(u8 rd, s32 val, struct rv_jit_context *ctx)
> +{
> +	s32 upper = (val + (1U << 11)) >> 12;
> +	s32 lower = ((val & 0xfff) << 20) >> 20;
> +
> +	emit(rv_lui(rd, upper), ctx);
> +	emit(rv_addi(rd, rd, lower), ctx);
> +}
> +
> +/* Emit fixed-length instructions for 64-bit imm */
> +static void emit_fixed_imm64(u8 rd, s64 val, struct rv_jit_context *ctx)
> +{
> +	/* Compensation for sign-extension of rv_addi */
> +	s32 imm_hi = (val + (1U << 31)) >> 32;
> +	s32 imm_lo = val;
> +
> +	emit_fixed_imm32(rd, imm_hi, ctx);
> +	emit_fixed_imm32(RV_REG_T1, imm_lo, ctx);
> +	emit(rv_slli(rd, rd, 32), ctx);
> +	emit(rv_add(rd, rd, RV_REG_T1), ctx);
> +}

Hmm, will this really be fixed? We can end up with compressed
instructions, which can then be a non-compressed in the last pass, and
we have the same problem?

The range of valid address for RV64 (sv39 to sv57) are
0xffffffff00000000 to 0xffffffffffffffff, so I think we can do better
than 6 insn, no? My gut feeling (I need to tinker a bit) is that 4
should be sufficient.

Note that worst case for a imm64 load are 8 instructions, but this is
not the general case.


Björn

Björn Töpel Nov. 30, 2022, 4:04 p.m. UTC | #2

Björn Töpel <[e-mail]> writes:

> The range of valid address for RV64 (sv39 to sv57) are
> 0xffffffff00000000 to 0xffffffffffffffff, so I think we can do better
> than 6 insn, no? My gut feeling (I need to tinker a bit) is that 4
> should be sufficient.

Ok, thinking a bit more about it; A proper address will at have at least
2B alignment, so that means that we can construct an address with lui,
addi, and slli (31 bits).

u64 addr = 0xffffffff12345678;

u32 (imm |= 0xffffffffUL) >> 1;
u32 upper = (imm + (1 << 11)) >> 12;
u32 lower = imm & 0xfff;
u32 rupper = upper | 0x80000; // for sign extend

NB! Make sure it's !C insn, and emit:

lui xx, rupper
addi xx, xx, lower
slli xx, xx, 1

Now we'll have fixed three insns. WDYT?


Björn

Pu Lehui Dec. 2, 2022, 9:58 a.m. UTC | #3

On 2022/11/30 19:38, Björn Töpel wrote:
> Pu Lehui <[e-mail]> writes:
> 
>> From: Pu Lehui <[e-mail]>
>>
>> For BPF_PSEUDO_FUNC instruction, verifier will refill imm with
>> correct addresses of bpf_calls and then run last pass of JIT.
>> Since the emit_imm of RV64 is variable-length, which will emit
>> appropriate length instructions accorroding to the imm, it may
>> broke ctx->offset, and lead to unpredictable problem, such as
>> inaccurate jump. So let's fix it with fixed-length imm64 insns.
> 
> Ah, nice one! So, the the invariant doesn't hold (the image grow in the
> last pass).
> 
>> Fixes: 69c087ba6225 ("bpf: Add bpf_for_each_map_elem() helper")
> 
> This is odd? This can't be the right Fixes-tag...
> 

Only BPF_PSEUDO_FUNC instruction need extra jit pass after refill imm in 
jit_subprogs. Others, like bpf helper call, will update ctx->offset in 
jit iterations. So the fixes-tag is 69c087ba6225.

>> Signed-off-by: Pu Lehui <[e-mail]>
>> ---
>>   arch/riscv/net/bpf_jit_comp64.c | 31 ++++++++++++++++++++++++++++++-
>>   1 file changed, 30 insertions(+), 1 deletion(-)
>>
>> diff --git a/arch/riscv/net/bpf_jit_comp64.c b/arch/riscv/net/bpf_jit_comp64.c
>> index eb99df41fa33..f984d5fa014b 100644
>> --- a/arch/riscv/net/bpf_jit_comp64.c
>> +++ b/arch/riscv/net/bpf_jit_comp64.c
>> @@ -139,6 +139,30 @@ static bool in_auipc_jalr_range(s64 val)
>>   		val < ((1L << 31) - (1L << 11));
>>   }
>>   
>> +/* Emit fixed-length instructions for 32-bit imm */
>> +static void emit_fixed_imm32(u8 rd, s32 val, struct rv_jit_context *ctx)
>> +{
>> +	s32 upper = (val + (1U << 11)) >> 12;
>> +	s32 lower = ((val & 0xfff) << 20) >> 20;
>> +
>> +	emit(rv_lui(rd, upper), ctx);
>> +	emit(rv_addi(rd, rd, lower), ctx);
>> +}
>> +
>> +/* Emit fixed-length instructions for 64-bit imm */
>> +static void emit_fixed_imm64(u8 rd, s64 val, struct rv_jit_context *ctx)
>> +{
>> +	/* Compensation for sign-extension of rv_addi */
>> +	s32 imm_hi = (val + (1U << 31)) >> 32;
>> +	s32 imm_lo = val;
>> +
>> +	emit_fixed_imm32(rd, imm_hi, ctx);
>> +	emit_fixed_imm32(RV_REG_T1, imm_lo, ctx);
>> +	emit(rv_slli(rd, rd, 32), ctx);
>> +	emit(rv_add(rd, rd, RV_REG_T1), ctx);
>> +}
> 
> Hmm, will this really be fixed? We can end up with compressed
> instructions, which can then be a non-compressed in the last pass, and
> we have the same problem?
> 
> The range of valid address for RV64 (sv39 to sv57) are
> 0xffffffff00000000 to 0xffffffffffffffff, so I think we can do better
> than 6 insn, no? My gut feeling (I need to tinker a bit) is that 4
> should be sufficient.
> 
> Note that worst case for a imm64 load are 8 instructions, but this is
> not the general case.
> 
> 
> Björn

Patch

diff --git a/arch/riscv/net/bpf_jit_comp64.c b/arch/riscv/net/bpf_jit_comp64.c
index eb99df41fa33..f984d5fa014b 100644
--- a/arch/riscv/net/bpf_jit_comp64.c
+++ b/arch/riscv/net/bpf_jit_comp64.c
@@ -139,6 +139,30 @@  static bool in_auipc_jalr_range(s64 val)
 		val < ((1L << 31) - (1L << 11));
 }
 
+/* Emit fixed-length instructions for 32-bit imm */
+static void emit_fixed_imm32(u8 rd, s32 val, struct rv_jit_context *ctx)
+{
+	s32 upper = (val + (1U << 11)) >> 12;
+	s32 lower = ((val & 0xfff) << 20) >> 20;
+
+	emit(rv_lui(rd, upper), ctx);
+	emit(rv_addi(rd, rd, lower), ctx);
+}
+
+/* Emit fixed-length instructions for 64-bit imm */
+static void emit_fixed_imm64(u8 rd, s64 val, struct rv_jit_context *ctx)
+{
+	/* Compensation for sign-extension of rv_addi */
+	s32 imm_hi = (val + (1U << 31)) >> 32;
+	s32 imm_lo = val;
+
+	emit_fixed_imm32(rd, imm_hi, ctx);
+	emit_fixed_imm32(RV_REG_T1, imm_lo, ctx);
+	emit(rv_slli(rd, rd, 32), ctx);
+	emit(rv_add(rd, rd, RV_REG_T1), ctx);
+}
+
+/* Emit variable-length instructions for 32-bit and 64-bit imm */
 static void emit_imm(u8 rd, s64 val, struct rv_jit_context *ctx)
 {
 	/* Note that the immediate from the add is sign-extended,
@@ -1053,7 +1077,12 @@  int bpf_jit_emit_insn(const struct bpf_insn *insn, struct rv_jit_context *ctx,
 		u64 imm64;
 
 		imm64 = (u64)insn1.imm << 32 | (u32)imm;
-		emit_imm(rd, imm64, ctx);
+		if (bpf_pseudo_func(insn))
+			/* fixed-length insns for extra jit pass */
+			emit_fixed_imm64(rd, imm64, ctx);
+		else
+			emit_imm(rd, imm64, ctx);
+
 		return 1;
 	}