[bpf-next,2/2] bpf,riscv: Implement bpf_addr_space_cast instruction

Commit Message

Puranjay Mohan March 23, 2024, 3:46 p.m. UTC

LLVM generates bpf_addr_space_cast instruction while translating
pointers between native (zero) address space and
__attribute__((address_space(N))). The addr_space=0 is reserved as
bpf_arena address space.

rY = addr_space_cast(rX, 0, 1) is processed by the verifier and
converted to normal 32-bit move: wX = wY

rY = addr_space_cast(rX, 1, 0) has to be converted by JIT:

Here I explain using symbolic language what the JIT is supposed to do:
We have:
	src = [src_upper32][src_lower32] // 64 bit src kernel pointer
	uvm = [uvm_upper32][uvm_lower32] // 64 bit user_vm_start

The JIT has to make the dst reg like following
	dst = [uvm_upper32][src_lower32] // if src_lower32 != 0
	dst = [00000000000][00000000000] // if src_lower32 == 0

Signed-off-by: Puranjay Mohan <[e-mail]>
---
 arch/riscv/net/bpf_jit.h        |  1 +
 arch/riscv/net/bpf_jit_comp64.c | 15 +++++++++++++++
 arch/riscv/net/bpf_jit_core.c   |  1 +
 3 files changed, 17 insertions(+)

Comments

Pu Lehui March 23, 2024, 4:43 p.m. UTC | #1

On 2024/3/23 23:46, Puranjay Mohan wrote:
> LLVM generates bpf_addr_space_cast instruction while translating
[snip]
>   
>   /* Convert from ninsns to bytes. */
> diff --git a/arch/riscv/net/bpf_jit_comp64.c b/arch/riscv/net/bpf_jit_comp64.c
> index f51b832eafb6..3c389e75cb96 100644
> --- a/arch/riscv/net/bpf_jit_comp64.c
> +++ b/arch/riscv/net/bpf_jit_comp64.c
> @@ -1083,6 +1083,16 @@ int bpf_jit_emit_insn(const struct bpf_insn *insn, struct rv_jit_context *ctx,
>   	/* dst = src */
>   	case BPF_ALU | BPF_MOV | BPF_X:
>   	case BPF_ALU64 | BPF_MOV | BPF_X:
> +		if (BPF_CLASS(insn->code) == BPF_ALU64 && insn->off == BPF_ADDR_SPACE_CAST &&
> +		    insn->imm == 1U << 16) {
> +			emit_mv(RV_REG_T1, rs, ctx); > +			emit_zextw(RV_REG_T1, RV_REG_T1, ctx);
combine mv and zextw will be better
> +			emit_imm(rd, (ctx->user_vm_start >> 32) << 32, ctx);
> +			emit(rv_beq(RV_REG_T1, RV_REG_ZERO, 4), ctx);
> +			emit_or(RV_REG_T1, rd, RV_REG_T1, ctx);
> +			emit_mv(rd, RV_REG_T1, ctx);
ditto, but for or and mv

Puranjay Mohan March 23, 2024, 4:49 p.m. UTC | #2

Pu Lehui <[e-mail]> writes:

> On 2024/3/23 23:46, Puranjay Mohan wrote:
>> LLVM generates bpf_addr_space_cast instruction while translating
> [snip]
>>   
>>   /* Convert from ninsns to bytes. */
>> diff --git a/arch/riscv/net/bpf_jit_comp64.c b/arch/riscv/net/bpf_jit_comp64.c
>> index f51b832eafb6..3c389e75cb96 100644
>> --- a/arch/riscv/net/bpf_jit_comp64.c
>> +++ b/arch/riscv/net/bpf_jit_comp64.c
>> @@ -1083,6 +1083,16 @@ int bpf_jit_emit_insn(const struct bpf_insn *insn, struct rv_jit_context *ctx,
>>   	/* dst = src */
>>   	case BPF_ALU | BPF_MOV | BPF_X:
>>   	case BPF_ALU64 | BPF_MOV | BPF_X:
>> +		if (BPF_CLASS(insn->code) == BPF_ALU64 && insn->off == BPF_ADDR_SPACE_CAST &&
>> +		    insn->imm == 1U << 16) {
>> +			emit_mv(RV_REG_T1, rs, ctx); > +			emit_zextw(RV_REG_T1, RV_REG_T1, ctx);
> combine mv and zextw will be better

Do you suggest doing:

emit_zextw(RV_REG_T1, rs, ctx);

Will do it in next version.

>> +			emit_imm(rd, (ctx->user_vm_start >> 32) << 32, ctx);
>> +			emit(rv_beq(RV_REG_T1, RV_REG_ZERO, 4), ctx);
>> +			emit_or(RV_REG_T1, rd, RV_REG_T1, ctx);
>> +			emit_mv(rd, RV_REG_T1, ctx);
> ditto, but for or and mv

How would we combine or and mv? 
also, we have a beq above and in one case both or and mv should happen,
but in other case only mv should happen.

Thanks,
Puranjay

Alexei Starovoitov March 23, 2024, 6:37 p.m. UTC | #3

On Sat, Mar 23, 2024 at 8:47 AM Puranjay Mohan <[e-mail]> wrote:
>
> LLVM generates bpf_addr_space_cast instruction while translating
> pointers between native (zero) address space and
> __attribute__((address_space(N))). The addr_space=0 is reserved as
> bpf_arena address space.
>
> rY = addr_space_cast(rX, 0, 1) is processed by the verifier and
> converted to normal 32-bit move: wX = wY
>
> rY = addr_space_cast(rX, 1, 0) has to be converted by JIT:
>
> Here I explain using symbolic language what the JIT is supposed to do:
> We have:
>         src = [src_upper32][src_lower32] // 64 bit src kernel pointer
>         uvm = [uvm_upper32][uvm_lower32] // 64 bit user_vm_start

This is a bit misleading.
src_lower32 are always equal to uvm_lower32
and src_upper32 are either zero or uvm_upper32.

Hence most of the time llvm doesn't generate this insn,
since it knows that upper 32 bit are uvm_upper32.

> The JIT has to make the dst reg like following
>         dst = [uvm_upper32][src_lower32] // if src_lower32 != 0
>         dst = [00000000000][00000000000] // if src_lower32 == 0
>
> Signed-off-by: Puranjay Mohan <[e-mail]>
> ---
>  arch/riscv/net/bpf_jit.h        |  1 +
>  arch/riscv/net/bpf_jit_comp64.c | 15 +++++++++++++++
>  arch/riscv/net/bpf_jit_core.c   |  1 +
>  3 files changed, 17 insertions(+)
>
> diff --git a/arch/riscv/net/bpf_jit.h b/arch/riscv/net/bpf_jit.h
> index 8a47da08dd9c..5fc374ed98ea 100644
> --- a/arch/riscv/net/bpf_jit.h
> +++ b/arch/riscv/net/bpf_jit.h
> @@ -82,6 +82,7 @@ struct rv_jit_context {
>         unsigned long flags;
>         int stack_size;
>         u64 arena_vm_start;
> +       u64 user_vm_start;
>  };
>
>  /* Convert from ninsns to bytes. */
> diff --git a/arch/riscv/net/bpf_jit_comp64.c b/arch/riscv/net/bpf_jit_comp64.c
> index f51b832eafb6..3c389e75cb96 100644
> --- a/arch/riscv/net/bpf_jit_comp64.c
> +++ b/arch/riscv/net/bpf_jit_comp64.c
> @@ -1083,6 +1083,16 @@ int bpf_jit_emit_insn(const struct bpf_insn *insn, struct rv_jit_context *ctx,
>         /* dst = src */
>         case BPF_ALU | BPF_MOV | BPF_X:
>         case BPF_ALU64 | BPF_MOV | BPF_X:
> +               if (BPF_CLASS(insn->code) == BPF_ALU64 && insn->off == BPF_ADDR_SPACE_CAST &&
> +                   insn->imm == 1U << 16) {

Let's add a generic helper like insn_is_zext(),
call it insn_is_cast_user() ?
and use it in all JIT-s ?
I should have added it right away when I did x86 part. Sorry.

And a comment next to the helper that addr space cast 0->1
is for converting bpf arena pointers to user vma.
Hence the name.

Same comments for arm64 JIT arena support.

pw-bot: cr

Pu Lehui March 24, 2024, 2:39 a.m. UTC | #4

On 2024/3/24 0:49, Puranjay Mohan wrote:
> Pu Lehui <[e-mail]> writes:
> 
>> On 2024/3/23 23:46, Puranjay Mohan wrote:
>>> LLVM generates bpf_addr_space_cast instruction while translating
>> [snip]
>>>    
>>>    /* Convert from ninsns to bytes. */
>>> diff --git a/arch/riscv/net/bpf_jit_comp64.c b/arch/riscv/net/bpf_jit_comp64.c
>>> index f51b832eafb6..3c389e75cb96 100644
>>> --- a/arch/riscv/net/bpf_jit_comp64.c
>>> +++ b/arch/riscv/net/bpf_jit_comp64.c
>>> @@ -1083,6 +1083,16 @@ int bpf_jit_emit_insn(const struct bpf_insn *insn, struct rv_jit_context *ctx,
>>>    	/* dst = src */
>>>    	case BPF_ALU | BPF_MOV | BPF_X:
>>>    	case BPF_ALU64 | BPF_MOV | BPF_X:
>>> +		if (BPF_CLASS(insn->code) == BPF_ALU64 && insn->off == BPF_ADDR_SPACE_CAST &&
>>> +		    insn->imm == 1U << 16) {
>>> +			emit_mv(RV_REG_T1, rs, ctx); > +			emit_zextw(RV_REG_T1, RV_REG_T1, ctx);
>> combine mv and zextw will be better
> 
> Do you suggest doing:
> 
> emit_zextw(RV_REG_T1, rs, ctx);
> 
> Will do it in next version.
> 
>>> +			emit_imm(rd, (ctx->user_vm_start >> 32) << 32, ctx);
>>> +			emit(rv_beq(RV_REG_T1, RV_REG_ZERO, 4), ctx);
>>> +			emit_or(RV_REG_T1, rd, RV_REG_T1, ctx);
>>> +			emit_mv(rd, RV_REG_T1, ctx);
>> ditto, but for or and mv
> 
> How would we combine or and mv?
> also, we have a beq above and in one case both or and mv should happen,
> but in other case only mv should happen.
> 
Okay, another branch is that t1 is zero, but not rd.

> Thanks,
> Puranjay

Patch

diff --git a/arch/riscv/net/bpf_jit.h b/arch/riscv/net/bpf_jit.h
index 8a47da08dd9c..5fc374ed98ea 100644
--- a/arch/riscv/net/bpf_jit.h
+++ b/arch/riscv/net/bpf_jit.h
@@ -82,6 +82,7 @@  struct rv_jit_context {
 	unsigned long flags;
 	int stack_size;
 	u64 arena_vm_start;
+	u64 user_vm_start;
 };
 
 /* Convert from ninsns to bytes. */
diff --git a/arch/riscv/net/bpf_jit_comp64.c b/arch/riscv/net/bpf_jit_comp64.c
index f51b832eafb6..3c389e75cb96 100644
--- a/arch/riscv/net/bpf_jit_comp64.c
+++ b/arch/riscv/net/bpf_jit_comp64.c
@@ -1083,6 +1083,16 @@  int bpf_jit_emit_insn(const struct bpf_insn *insn, struct rv_jit_context *ctx,
 	/* dst = src */
 	case BPF_ALU | BPF_MOV | BPF_X:
 	case BPF_ALU64 | BPF_MOV | BPF_X:
+		if (BPF_CLASS(insn->code) == BPF_ALU64 && insn->off == BPF_ADDR_SPACE_CAST &&
+		    insn->imm == 1U << 16) {
+			emit_mv(RV_REG_T1, rs, ctx);
+			emit_zextw(RV_REG_T1, RV_REG_T1, ctx);
+			emit_imm(rd, (ctx->user_vm_start >> 32) << 32, ctx);
+			emit(rv_beq(RV_REG_T1, RV_REG_ZERO, 4), ctx);
+			emit_or(RV_REG_T1, rd, RV_REG_T1, ctx);
+			emit_mv(rd, RV_REG_T1, ctx);
+			break;
+		}
 		if (imm == 1) {
 			/* Special mov32 for zext */
 			emit_zextw(rd, rd, ctx);
@@ -2010,3 +2020,8 @@  bool bpf_jit_supports_ptr_xchg(void)
 {
 	return true;
 }
+
+bool bpf_jit_supports_arena(void)
+{
+	return true;
+}
diff --git a/arch/riscv/net/bpf_jit_core.c b/arch/riscv/net/bpf_jit_core.c
index 9b6696b1290a..aaef1d0c7c46 100644
--- a/arch/riscv/net/bpf_jit_core.c
+++ b/arch/riscv/net/bpf_jit_core.c
@@ -82,6 +82,7 @@  struct bpf_prog *bpf_int_jit_compile(struct bpf_prog *prog)
 	}
 
 	ctx->arena_vm_start = bpf_arena_get_kern_vm_start(prog->aux->arena);
+	ctx->user_vm_start = bpf_arena_get_user_vm_start(prog->aux->arena);
 	ctx->prog = prog;
 	ctx->offset = kcalloc(prog->len, sizeof(int), GFP_KERNEL);
 	if (!ctx->offset) {