| Message ID | 20220906133613.54928-7-quentin@isovalent.com (mailing list archive) |
|---|---|
| State | Superseded |
| Delegated to: | BPF |
| Headers | show |
| Series | bpftool: Add LLVM as default library for disassembling JIT-ed programs | expand |
On Tue, Sep 6, 2022 at 6:36 AM Quentin Monnet <quentin@isovalent.com> wrote: > > Naturally, the display of disassembled instructions comes with a few > minor differences. Here is a sample output with libbfd (already > supported before this patch): > > # bpftool prog dump jited id 56 > bpf_prog_6deef7357e7b4530: > 0: nopl 0x0(%rax,%rax,1) > 5: xchg %ax,%ax > 7: push %rbp > 8: mov %rsp,%rbp > b: push %rbx > c: push %r13 > e: push %r14 > 10: mov %rdi,%rbx > 13: movzwq 0xb0(%rbx),%r13 > 1b: xor %r14d,%r14d > 1e: or $0x2,%r14d > 22: mov $0x1,%eax > 27: cmp $0x2,%r14 > 2b: jne 0x000000000000002f > 2d: xor %eax,%eax > 2f: pop %r14 > 31: pop %r13 > 33: pop %rbx > 34: leave > 35: ret > 36: int3 > > LLVM supports several variants that we could set when initialising the > disassembler, for example with: > > LLVMSetDisasmOptions(*ctx, > LLVMDisassembler_Option_AsmPrinterVariant); > > but the default printer is kept for now. Here is the output with LLVM: > > # bpftool prog dump jited id 56 > bpf_prog_6deef7357e7b4530: > 0: nopl (%rax,%rax) > 5: nop > 7: pushq %rbp > 8: movq %rsp, %rbp > b: pushq %rbx > c: pushq %r13 > e: pushq %r14 > 10: movq %rdi, %rbx > 13: movzwq 176(%rbx), %r13 > 1b: xorl %r14d, %r14d > 1e: orl $2, %r14d > 22: movl $1, %eax > 27: cmpq $2, %r14 > 2b: jne 2 > 2d: xorl %eax, %eax > 2f: popq %r14 If I'm reading the asm correctly the difference is significant. jne 0x2f was an absolute address and jmps were easy to follow. While in llvm disasm it's 'jne 2' ?! What is 2 ? 2 bytes from the next insn of 0x2d ? That is super hard to read. Is there a way to tune/configure llvm disasm?
On Tue, Sep 6, 2022 at 6:46 AM Quentin Monnet <quentin@isovalent.com> wrote: > [...] > + > +static int > +init_context(disasm_ctx_t *ctx, const char *arch, > + __maybe_unused const char *disassembler_options, > + __maybe_unused unsigned char *image, __maybe_unused ssize_t len) > +{ > + char *triple; > + > + if (arch) { > + p_err("Architecture %s not supported", arch); > + return -1; > + } Does this mean we stop supporting arch by default (prefer llvm over bfd)? Thanks, Song
On 07/09/2022 00:46, Alexei Starovoitov wrote: > On Tue, Sep 6, 2022 at 6:36 AM Quentin Monnet <quentin@isovalent.com> wrote: >> >> Naturally, the display of disassembled instructions comes with a few >> minor differences. Here is a sample output with libbfd (already >> supported before this patch): >> >> # bpftool prog dump jited id 56 >> bpf_prog_6deef7357e7b4530: >> 0: nopl 0x0(%rax,%rax,1) >> 5: xchg %ax,%ax >> 7: push %rbp >> 8: mov %rsp,%rbp >> b: push %rbx >> c: push %r13 >> e: push %r14 >> 10: mov %rdi,%rbx >> 13: movzwq 0xb0(%rbx),%r13 >> 1b: xor %r14d,%r14d >> 1e: or $0x2,%r14d >> 22: mov $0x1,%eax >> 27: cmp $0x2,%r14 >> 2b: jne 0x000000000000002f >> 2d: xor %eax,%eax >> 2f: pop %r14 >> 31: pop %r13 >> 33: pop %rbx >> 34: leave >> 35: ret >> 36: int3 >> >> LLVM supports several variants that we could set when initialising the >> disassembler, for example with: >> >> LLVMSetDisasmOptions(*ctx, >> LLVMDisassembler_Option_AsmPrinterVariant); >> >> but the default printer is kept for now. Here is the output with LLVM: >> >> # bpftool prog dump jited id 56 >> bpf_prog_6deef7357e7b4530: >> 0: nopl (%rax,%rax) >> 5: nop >> 7: pushq %rbp >> 8: movq %rsp, %rbp >> b: pushq %rbx >> c: pushq %r13 >> e: pushq %r14 >> 10: movq %rdi, %rbx >> 13: movzwq 176(%rbx), %r13 >> 1b: xorl %r14d, %r14d >> 1e: orl $2, %r14d >> 22: movl $1, %eax >> 27: cmpq $2, %r14 >> 2b: jne 2 >> 2d: xorl %eax, %eax >> 2f: popq %r14 > > If I'm reading the asm correctly the difference is significant. > jne 0x2f was an absolute address and jmps were easy > to follow. > While in llvm disasm it's 'jne 2' ?! What is 2 ? > 2 bytes from the next insn of 0x2d ? Yes, that's it. Apparently, this is how the operand is encoded, and libbfd does the translation to the absolute address: # bpftool prog dump jited id 7868 opcodes [...] 2b: jne 0x000000000000002f 75 02 [...] The same difference is observable between objdump and llvm-objdump on an x86-64 binary for example, although they usually have labels to refer to ("jne -22 <_obstack_memory_used+0x7d0>"), making the navigation easier. The only mention I could find of that difference is a report from 2013 [0]. [0] https://discourse.llvm.org/t/llvm-objdump-disassembling-jmp/29584/2 > That is super hard to read. > Is there a way to tune/configure llvm disasm? There's a function and some options to tune it, but I tried them and none applies to converting the jump operands. int LLVMSetDisasmOptions(LLVMDisasmContextRef DC, uint64_t Options); /* The option to produce marked up assembly. */ #define LLVMDisassembler_Option_UseMarkup 1 /* The option to print immediates as hex. */ #define LLVMDisassembler_Option_PrintImmHex 2 /* The option use the other assembler printer variant */ #define LLVMDisassembler_Option_AsmPrinterVariant 4 /* The option to set comment on instructions */ #define LLVMDisassembler_Option_SetInstrComments 8 /* The option to print latency information alongside instructions */ #define LLVMDisassembler_Option_PrintLatency 16 I found that LLVMDisassembler_Option_AsmPrinterVariant read better, although in my patch I kept the default output which looked closer to the existing from libbfd. Here's what the option produces: bpf_prog_6deef7357e7b4530: 0: nop dword ptr [rax + rax] 5: nop 7: push rbp 8: mov rbp, rsp b: push rbx c: push r13 e: push r14 10: mov rbx, rdi 13: movzx r13, word ptr [rbx + 180] 1b: xor r14d, r14d 1e: or r14d, 2 22: mov eax, 1 27: cmp r14, 2 2b: jne 2 2d: xor eax, eax 2f: pop r14 31: pop r13 33: pop rbx 34: leave 35: re But the jne operand remains a '2'. I'm not aware of any option to change it in LLVM's disassembler :(.
On 07/09/2022 01:06, Song Liu wrote: > On Tue, Sep 6, 2022 at 6:46 AM Quentin Monnet <quentin@isovalent.com> wrote: >> > [...] >> + >> +static int >> +init_context(disasm_ctx_t *ctx, const char *arch, >> + __maybe_unused const char *disassembler_options, >> + __maybe_unused unsigned char *image, __maybe_unused ssize_t len) >> +{ >> + char *triple; >> + >> + if (arch) { >> + p_err("Architecture %s not supported", arch); >> + return -1; >> + } > > Does this mean we stop supporting arch by default (prefer llvm > over bfd)? We do drop support in practice, because the "arch" is only used for nfp (we only use this when the program is not using the host architecture, so when it's offloaded - see ifindex_to_bfd_params() in common.c), and LLVM has no support for nfp. Although on second thought, it would probably be cleaner to set the arch anyway in the snippet above, and to let LLVM return an error if it doesn't know about it, so that we don't have to update bpftool in the future if a new arch is used for BPF offload. I can update for the next iteration.
On Wed, Sep 7, 2022 at 7:20 AM Quentin Monnet <quentin@isovalent.com> wrote: > > On 07/09/2022 00:46, Alexei Starovoitov wrote: > > On Tue, Sep 6, 2022 at 6:36 AM Quentin Monnet <quentin@isovalent.com> wrote: > >> > >> Naturally, the display of disassembled instructions comes with a few > >> minor differences. Here is a sample output with libbfd (already > >> supported before this patch): > >> > >> # bpftool prog dump jited id 56 > >> bpf_prog_6deef7357e7b4530: > >> 0: nopl 0x0(%rax,%rax,1) > >> 5: xchg %ax,%ax > >> 7: push %rbp > >> 8: mov %rsp,%rbp > >> b: push %rbx > >> c: push %r13 > >> e: push %r14 > >> 10: mov %rdi,%rbx > >> 13: movzwq 0xb0(%rbx),%r13 > >> 1b: xor %r14d,%r14d > >> 1e: or $0x2,%r14d > >> 22: mov $0x1,%eax > >> 27: cmp $0x2,%r14 > >> 2b: jne 0x000000000000002f > >> 2d: xor %eax,%eax > >> 2f: pop %r14 > >> 31: pop %r13 > >> 33: pop %rbx > >> 34: leave > >> 35: ret > >> 36: int3 > >> > >> LLVM supports several variants that we could set when initialising the > >> disassembler, for example with: > >> > >> LLVMSetDisasmOptions(*ctx, > >> LLVMDisassembler_Option_AsmPrinterVariant); > >> > >> but the default printer is kept for now. Here is the output with LLVM: > >> > >> # bpftool prog dump jited id 56 > >> bpf_prog_6deef7357e7b4530: > >> 0: nopl (%rax,%rax) > >> 5: nop > >> 7: pushq %rbp > >> 8: movq %rsp, %rbp > >> b: pushq %rbx > >> c: pushq %r13 > >> e: pushq %r14 > >> 10: movq %rdi, %rbx > >> 13: movzwq 176(%rbx), %r13 > >> 1b: xorl %r14d, %r14d > >> 1e: orl $2, %r14d > >> 22: movl $1, %eax > >> 27: cmpq $2, %r14 > >> 2b: jne 2 > >> 2d: xorl %eax, %eax > >> 2f: popq %r14 > > > > If I'm reading the asm correctly the difference is significant. > > jne 0x2f was an absolute address and jmps were easy > > to follow. > > While in llvm disasm it's 'jne 2' ?! What is 2 ? > > 2 bytes from the next insn of 0x2d ? > > Yes, that's it. Apparently, this is how the operand is encoded, and > libbfd does the translation to the absolute address: > > # bpftool prog dump jited id 7868 opcodes > [...] > 2b: jne 0x000000000000002f > 75 02 > [...] > > The same difference is observable between objdump and llvm-objdump on an > x86-64 binary for example, although they usually have labels to refer to > ("jne -22 <_obstack_memory_used+0x7d0>"), making the navigation > easier. The only mention I could find of that difference is a report > from 2013 [0]. > > [0] https://discourse.llvm.org/t/llvm-objdump-disassembling-jmp/29584/2 > > > That is super hard to read. > > Is there a way to tune/configure llvm disasm? > > There's a function and some options to tune it, but I tried them and > none applies to converting the jump operands. > > int LLVMSetDisasmOptions(LLVMDisasmContextRef DC, uint64_t Options); > > /* The option to produce marked up assembly. */ > #define LLVMDisassembler_Option_UseMarkup 1 > /* The option to print immediates as hex. */ > #define LLVMDisassembler_Option_PrintImmHex 2 > /* The option use the other assembler printer variant */ > #define LLVMDisassembler_Option_AsmPrinterVariant 4 > /* The option to set comment on instructions */ > #define LLVMDisassembler_Option_SetInstrComments 8 > /* The option to print latency information alongside instructions */ > #define LLVMDisassembler_Option_PrintLatency 16 > > I found that LLVMDisassembler_Option_AsmPrinterVariant read better, > although in my patch I kept the default output which looked closer to > the existing from libbfd. Here's what the option produces: > > bpf_prog_6deef7357e7b4530: > 0: nop dword ptr [rax + rax] > 5: nop > 7: push rbp > 8: mov rbp, rsp > b: push rbx > c: push r13 > e: push r14 > 10: mov rbx, rdi > 13: movzx r13, word ptr [rbx + 180] > 1b: xor r14d, r14d > 1e: or r14d, 2 > 22: mov eax, 1 > 27: cmp r14, 2 > 2b: jne 2 > 2d: xor eax, eax > 2f: pop r14 > 31: pop r13 > 33: pop rbx > 34: leave > 35: re > > But the jne operand remains a '2'. I'm not aware of any option to change > it in LLVM's disassembler :(. Hmm. llvm-objdump -d test_maps looks fine: 41bfcb: e8 6f f7 ff ff callq 0x41b73f <find_extern_btf_id> the must be something llvm disasm is missing when you feed raw bytes into it. Please keep investigating. In this form I'm afraid it's no go.
On 07/09/2022 17:10, Alexei Starovoitov wrote: > On Wed, Sep 7, 2022 at 7:20 AM Quentin Monnet <quentin@isovalent.com> wrote: >> >> On 07/09/2022 00:46, Alexei Starovoitov wrote: >>> On Tue, Sep 6, 2022 at 6:36 AM Quentin Monnet <quentin@isovalent.com> wrote: >>>> >>>> Naturally, the display of disassembled instructions comes with a few >>>> minor differences. Here is a sample output with libbfd (already >>>> supported before this patch): >>>> >>>> # bpftool prog dump jited id 56 >>>> bpf_prog_6deef7357e7b4530: >>>> 0: nopl 0x0(%rax,%rax,1) >>>> 5: xchg %ax,%ax >>>> 7: push %rbp >>>> 8: mov %rsp,%rbp >>>> b: push %rbx >>>> c: push %r13 >>>> e: push %r14 >>>> 10: mov %rdi,%rbx >>>> 13: movzwq 0xb0(%rbx),%r13 >>>> 1b: xor %r14d,%r14d >>>> 1e: or $0x2,%r14d >>>> 22: mov $0x1,%eax >>>> 27: cmp $0x2,%r14 >>>> 2b: jne 0x000000000000002f >>>> 2d: xor %eax,%eax >>>> 2f: pop %r14 >>>> 31: pop %r13 >>>> 33: pop %rbx >>>> 34: leave >>>> 35: ret >>>> 36: int3 >>>> >>>> LLVM supports several variants that we could set when initialising the >>>> disassembler, for example with: >>>> >>>> LLVMSetDisasmOptions(*ctx, >>>> LLVMDisassembler_Option_AsmPrinterVariant); >>>> >>>> but the default printer is kept for now. Here is the output with LLVM: >>>> >>>> # bpftool prog dump jited id 56 >>>> bpf_prog_6deef7357e7b4530: >>>> 0: nopl (%rax,%rax) >>>> 5: nop >>>> 7: pushq %rbp >>>> 8: movq %rsp, %rbp >>>> b: pushq %rbx >>>> c: pushq %r13 >>>> e: pushq %r14 >>>> 10: movq %rdi, %rbx >>>> 13: movzwq 176(%rbx), %r13 >>>> 1b: xorl %r14d, %r14d >>>> 1e: orl $2, %r14d >>>> 22: movl $1, %eax >>>> 27: cmpq $2, %r14 >>>> 2b: jne 2 >>>> 2d: xorl %eax, %eax >>>> 2f: popq %r14 >>> >>> If I'm reading the asm correctly the difference is significant. >>> jne 0x2f was an absolute address and jmps were easy >>> to follow. >>> While in llvm disasm it's 'jne 2' ?! What is 2 ? >>> 2 bytes from the next insn of 0x2d ? >> >> Yes, that's it. Apparently, this is how the operand is encoded, and >> libbfd does the translation to the absolute address: >> >> # bpftool prog dump jited id 7868 opcodes >> [...] >> 2b: jne 0x000000000000002f >> 75 02 >> [...] >> >> The same difference is observable between objdump and llvm-objdump on an >> x86-64 binary for example, although they usually have labels to refer to >> ("jne -22 <_obstack_memory_used+0x7d0>"), making the navigation >> easier. The only mention I could find of that difference is a report >> from 2013 [0]. >> >> [0] https://discourse.llvm.org/t/llvm-objdump-disassembling-jmp/29584/2 >> >>> That is super hard to read. >>> Is there a way to tune/configure llvm disasm? >> >> There's a function and some options to tune it, but I tried them and >> none applies to converting the jump operands. >> >> int LLVMSetDisasmOptions(LLVMDisasmContextRef DC, uint64_t Options); >> >> /* The option to produce marked up assembly. */ >> #define LLVMDisassembler_Option_UseMarkup 1 >> /* The option to print immediates as hex. */ >> #define LLVMDisassembler_Option_PrintImmHex 2 >> /* The option use the other assembler printer variant */ >> #define LLVMDisassembler_Option_AsmPrinterVariant 4 >> /* The option to set comment on instructions */ >> #define LLVMDisassembler_Option_SetInstrComments 8 >> /* The option to print latency information alongside instructions */ >> #define LLVMDisassembler_Option_PrintLatency 16 >> >> I found that LLVMDisassembler_Option_AsmPrinterVariant read better, >> although in my patch I kept the default output which looked closer to >> the existing from libbfd. Here's what the option produces: >> >> bpf_prog_6deef7357e7b4530: >> 0: nop dword ptr [rax + rax] >> 5: nop >> 7: push rbp >> 8: mov rbp, rsp >> b: push rbx >> c: push r13 >> e: push r14 >> 10: mov rbx, rdi >> 13: movzx r13, word ptr [rbx + 180] >> 1b: xor r14d, r14d >> 1e: or r14d, 2 >> 22: mov eax, 1 >> 27: cmp r14, 2 >> 2b: jne 2 >> 2d: xor eax, eax >> 2f: pop r14 >> 31: pop r13 >> 33: pop rbx >> 34: leave >> 35: re >> >> But the jne operand remains a '2'. I'm not aware of any option to change >> it in LLVM's disassembler :(. > > Hmm. llvm-objdump -d test_maps > looks fine: > 41bfcb: e8 6f f7 ff ff callq 0x41b73f > <find_extern_btf_id> > > the must be something llvm disasm is missing when you feed raw bytes > into it. > Please keep investigating. In this form I'm afraid it's no go. OK, I'll keep looking
> On Sep 7, 2022, at 7:20 AM, Quentin Monnet <quentin@isovalent.com> wrote: > > On 07/09/2022 01:06, Song Liu wrote: >> On Tue, Sep 6, 2022 at 6:46 AM Quentin Monnet <quentin@isovalent.com> wrote: >>> >> [...] >>> + >>> +static int >>> +init_context(disasm_ctx_t *ctx, const char *arch, >>> + __maybe_unused const char *disassembler_options, >>> + __maybe_unused unsigned char *image, __maybe_unused ssize_t len) >>> +{ >>> + char *triple; >>> + >>> + if (arch) { >>> + p_err("Architecture %s not supported", arch); >>> + return -1; >>> + } >> >> Does this mean we stop supporting arch by default (prefer llvm >> over bfd)? > > We do drop support in practice, because the "arch" is only used for nfp > (we only use this when the program is not using the host architecture, > so when it's offloaded - see ifindex_to_bfd_params() in common.c), and > LLVM has no support for nfp. > > Although on second thought, it would probably be cleaner to set the arch > anyway in the snippet above, and to let LLVM return an error if it > doesn't know about it, so that we don't have to update bpftool in the > future if a new arch is used for BPF offload. I can update for the next > iteration. Sounds good! Thanks for looking into different options. Song
On 9/7/22 9:33 AM, Quentin Monnet wrote: > On 07/09/2022 17:10, Alexei Starovoitov wrote: >> On Wed, Sep 7, 2022 at 7:20 AM Quentin Monnet <quentin@isovalent.com> wrote: >>> >>> On 07/09/2022 00:46, Alexei Starovoitov wrote: >>>> On Tue, Sep 6, 2022 at 6:36 AM Quentin Monnet <quentin@isovalent.com> wrote: >>>>> >>>>> Naturally, the display of disassembled instructions comes with a few >>>>> minor differences. Here is a sample output with libbfd (already >>>>> supported before this patch): >>>>> >>>>> # bpftool prog dump jited id 56 >>>>> bpf_prog_6deef7357e7b4530: >>>>> 0: nopl 0x0(%rax,%rax,1) >>>>> 5: xchg %ax,%ax >>>>> 7: push %rbp >>>>> 8: mov %rsp,%rbp >>>>> b: push %rbx >>>>> c: push %r13 >>>>> e: push %r14 >>>>> 10: mov %rdi,%rbx >>>>> 13: movzwq 0xb0(%rbx),%r13 >>>>> 1b: xor %r14d,%r14d >>>>> 1e: or $0x2,%r14d >>>>> 22: mov $0x1,%eax >>>>> 27: cmp $0x2,%r14 >>>>> 2b: jne 0x000000000000002f >>>>> 2d: xor %eax,%eax >>>>> 2f: pop %r14 >>>>> 31: pop %r13 >>>>> 33: pop %rbx >>>>> 34: leave >>>>> 35: ret >>>>> 36: int3 >>>>> >>>>> LLVM supports several variants that we could set when initialising the >>>>> disassembler, for example with: >>>>> >>>>> LLVMSetDisasmOptions(*ctx, >>>>> LLVMDisassembler_Option_AsmPrinterVariant); >>>>> >>>>> but the default printer is kept for now. Here is the output with LLVM: >>>>> >>>>> # bpftool prog dump jited id 56 >>>>> bpf_prog_6deef7357e7b4530: >>>>> 0: nopl (%rax,%rax) >>>>> 5: nop >>>>> 7: pushq %rbp >>>>> 8: movq %rsp, %rbp >>>>> b: pushq %rbx >>>>> c: pushq %r13 >>>>> e: pushq %r14 >>>>> 10: movq %rdi, %rbx >>>>> 13: movzwq 176(%rbx), %r13 >>>>> 1b: xorl %r14d, %r14d >>>>> 1e: orl $2, %r14d >>>>> 22: movl $1, %eax >>>>> 27: cmpq $2, %r14 >>>>> 2b: jne 2 >>>>> 2d: xorl %eax, %eax >>>>> 2f: popq %r14 >>>> >>>> If I'm reading the asm correctly the difference is significant. >>>> jne 0x2f was an absolute address and jmps were easy >>>> to follow. >>>> While in llvm disasm it's 'jne 2' ?! What is 2 ? >>>> 2 bytes from the next insn of 0x2d ? >>> >>> Yes, that's it. Apparently, this is how the operand is encoded, and >>> libbfd does the translation to the absolute address: >>> >>> # bpftool prog dump jited id 7868 opcodes >>> [...] >>> 2b: jne 0x000000000000002f >>> 75 02 >>> [...] >>> >>> The same difference is observable between objdump and llvm-objdump on an >>> x86-64 binary for example, although they usually have labels to refer to >>> ("jne -22 <_obstack_memory_used+0x7d0>"), making the navigation >>> easier. The only mention I could find of that difference is a report >>> from 2013 [0]. >>> >>> [0] https://discourse.llvm.org/t/llvm-objdump-disassembling-jmp/29584/2 >>> >>>> That is super hard to read. >>>> Is there a way to tune/configure llvm disasm? >>> >>> There's a function and some options to tune it, but I tried them and >>> none applies to converting the jump operands. >>> >>> int LLVMSetDisasmOptions(LLVMDisasmContextRef DC, uint64_t Options); >>> >>> /* The option to produce marked up assembly. */ >>> #define LLVMDisassembler_Option_UseMarkup 1 >>> /* The option to print immediates as hex. */ >>> #define LLVMDisassembler_Option_PrintImmHex 2 >>> /* The option use the other assembler printer variant */ >>> #define LLVMDisassembler_Option_AsmPrinterVariant 4 >>> /* The option to set comment on instructions */ >>> #define LLVMDisassembler_Option_SetInstrComments 8 >>> /* The option to print latency information alongside instructions */ >>> #define LLVMDisassembler_Option_PrintLatency 16 >>> >>> I found that LLVMDisassembler_Option_AsmPrinterVariant read better, >>> although in my patch I kept the default output which looked closer to >>> the existing from libbfd. Here's what the option produces: >>> >>> bpf_prog_6deef7357e7b4530: >>> 0: nop dword ptr [rax + rax] >>> 5: nop >>> 7: push rbp >>> 8: mov rbp, rsp >>> b: push rbx >>> c: push r13 >>> e: push r14 >>> 10: mov rbx, rdi >>> 13: movzx r13, word ptr [rbx + 180] >>> 1b: xor r14d, r14d >>> 1e: or r14d, 2 >>> 22: mov eax, 1 >>> 27: cmp r14, 2 >>> 2b: jne 2 >>> 2d: xor eax, eax >>> 2f: pop r14 >>> 31: pop r13 >>> 33: pop rbx >>> 34: leave >>> 35: re >>> >>> But the jne operand remains a '2'. I'm not aware of any option to change >>> it in LLVM's disassembler :(. >> >> Hmm. llvm-objdump -d test_maps >> looks fine: >> 41bfcb: e8 6f f7 ff ff callq 0x41b73f >> <find_extern_btf_id> >> >> the must be something llvm disasm is missing when you feed raw bytes >> into it. >> Please keep investigating. In this form I'm afraid it's no go. > > OK, I'll keep looking Quentin, if eventually there is no existing solution for this problem, we could improve llvm API to encode branch target in more easy-to-understand form.
On 07/09/2022 19:02, Yonghong Song wrote: > > > On 9/7/22 9:33 AM, Quentin Monnet wrote: >> On 07/09/2022 17:10, Alexei Starovoitov wrote: >>> On Wed, Sep 7, 2022 at 7:20 AM Quentin Monnet <quentin@isovalent.com> >>> wrote: >>>> >>>> On 07/09/2022 00:46, Alexei Starovoitov wrote: >>>>> On Tue, Sep 6, 2022 at 6:36 AM Quentin Monnet >>>>> <quentin@isovalent.com> wrote: >>>>>> # bpftool prog dump jited id 56 >>>>>> bpf_prog_6deef7357e7b4530: >>>>>> 0: nopl (%rax,%rax) >>>>>> 5: nop >>>>>> 7: pushq %rbp >>>>>> 8: movq %rsp, %rbp >>>>>> b: pushq %rbx >>>>>> c: pushq %r13 >>>>>> e: pushq %r14 >>>>>> 10: movq %rdi, %rbx >>>>>> 13: movzwq 176(%rbx), %r13 >>>>>> 1b: xorl %r14d, %r14d >>>>>> 1e: orl $2, %r14d >>>>>> 22: movl $1, %eax >>>>>> 27: cmpq $2, %r14 >>>>>> 2b: jne 2 >>>>>> 2d: xorl %eax, %eax >>>>>> 2f: popq %r14 >>>>> >>>>> If I'm reading the asm correctly the difference is significant. >>>>> jne 0x2f was an absolute address and jmps were easy >>>>> to follow. >>>>> While in llvm disasm it's 'jne 2' ?! What is 2 ? >>>>> 2 bytes from the next insn of 0x2d ? >>>> >>>> Yes, that's it. Apparently, this is how the operand is encoded, and >>>> libbfd does the translation to the absolute address: >>>> >>>> # bpftool prog dump jited id 7868 opcodes >>>> [...] >>>> 2b: jne 0x000000000000002f >>>> 75 02 >>>> [...] >>>> >>>> The same difference is observable between objdump and llvm-objdump >>>> on an >>>> x86-64 binary for example, although they usually have labels to >>>> refer to >>>> ("jne -22 <_obstack_memory_used+0x7d0>"), making the navigation >>>> easier. The only mention I could find of that difference is a report >>>> from 2013 [0]. >>>> >>>> [0] https://discourse.llvm.org/t/llvm-objdump-disassembling-jmp/29584/2 >>>> >>>>> That is super hard to read. >>>>> Is there a way to tune/configure llvm disasm? >>>> >>>> There's a function and some options to tune it, but I tried them and >>>> none applies to converting the jump operands. >>>> >>>> int LLVMSetDisasmOptions(LLVMDisasmContextRef DC, uint64_t >>>> Options); >>>> >>>> /* The option to produce marked up assembly. */ >>>> #define LLVMDisassembler_Option_UseMarkup 1 >>>> /* The option to print immediates as hex. */ >>>> #define LLVMDisassembler_Option_PrintImmHex 2 >>>> /* The option use the other assembler printer variant */ >>>> #define LLVMDisassembler_Option_AsmPrinterVariant 4 >>>> /* The option to set comment on instructions */ >>>> #define LLVMDisassembler_Option_SetInstrComments 8 >>>> /* The option to print latency information alongside >>>> instructions */ >>>> #define LLVMDisassembler_Option_PrintLatency 16 >>>> >>>> I found that LLVMDisassembler_Option_AsmPrinterVariant read better, >>>> although in my patch I kept the default output which looked closer to >>>> the existing from libbfd. Here's what the option produces: >>>> >>>> bpf_prog_6deef7357e7b4530: >>>> 0: nop dword ptr [rax + rax] >>>> 5: nop >>>> 7: push rbp >>>> 8: mov rbp, rsp >>>> b: push rbx >>>> c: push r13 >>>> e: push r14 >>>> 10: mov rbx, rdi >>>> 13: movzx r13, word ptr [rbx + 180] >>>> 1b: xor r14d, r14d >>>> 1e: or r14d, 2 >>>> 22: mov eax, 1 >>>> 27: cmp r14, 2 >>>> 2b: jne 2 >>>> 2d: xor eax, eax >>>> 2f: pop r14 >>>> 31: pop r13 >>>> 33: pop rbx >>>> 34: leave >>>> 35: re >>>> >>>> But the jne operand remains a '2'. I'm not aware of any option to >>>> change >>>> it in LLVM's disassembler :(. >>> >>> Hmm. llvm-objdump -d test_maps >>> looks fine: >>> 41bfcb: e8 6f f7 ff ff callq 0x41b73f >>> <find_extern_btf_id> >>> >>> the must be something llvm disasm is missing when you feed raw bytes >>> into it. >>> Please keep investigating. In this form I'm afraid it's no go. >> >> OK, I'll keep looking > > Quentin, if eventually there is no existing solution for this problem, > we could improve llvm API to encode branch target in more > easy-to-understand form. > TL;DR: I figured it out, with no LLVM fix required. I'll send a v2. --- The details: In my previous example, I ran on Ubuntu 20.04 with llvm-objdump v10. Looking at just the v11 with the same command, I get the relative addresses like in Alexei's output: $ llvm-objdump-10 -d /usr/bin/grep | grep -m1 jne 4f20: 0f 85 e7 0b 00 00 jne 3047 <.text+0xddd> $ llvm-objdump-11 -d /usr/bin/grep | grep -m1 jne 4f20: 0f 85 e7 0b 00 00 jne 0x5b0d <.text+0xddd> $ printf '%#x\n' $((0x4f20 + 3047 + 6)) 0x5b0d The change was introduced between the two versions by the following commits: 5fad05e80dd0 ("[MCInstPrinter] Pass `Address` parameter to MCOI::OPERAND_PCREL typed operands. NFC") https://reviews.llvm.org/D76574 87de9a0786d9 ("[X86InstPrinter] Change printPCRelImm to print the target address in hexadecimal form") https://reviews.llvm.org/D76580 The first one in particular introduces a PrintBranchImmAsAddress variable in the MCInstPrinter, and makes llvm-obdump call "setPrintBranchImmAsAddress(true);". Now, this function is not exposed to the C interface, llvm-c, that bpftool would use. Instead, it's available through the more comprehensive C++ interface that llvm-objdump relies on. I've tried to replicate these two commits in the MCDisassembler that llvm-c interfaces with, and succeeded. But while looking at unit tests to extend it with the relevant checks, I realised that the existing test would already expect an address instead of an operand [0]. Turns out that the symbolLookupCallback() callback defined in the test and passed when creating the disassembler ends up changing the display so we get the operands as addresses, as we want. [0] https://github.com/llvm/llvm-project/blob/llvmorg-15.0.0/llvm/unittests/MC/Disassembler.cpp#L64 --- I validated that this works in bpftool too, and will send v2 with that change. P.S. Thank you Niklas for testing!
diff --git a/tools/bpf/bpftool/Makefile b/tools/bpf/bpftool/Makefile index 8060c7013d4f..d96584e2dae7 100644 --- a/tools/bpf/bpftool/Makefile +++ b/tools/bpf/bpftool/Makefile @@ -95,12 +95,14 @@ RM ?= rm -f FEATURE_USER = .bpftool FEATURE_TESTS := clang-bpf-co-re +FEATURE_TESTS += llvm FEATURE_TESTS += libcap FEATURE_TESTS += libbfd FEATURE_TESTS += disassembler-four-args FEATURE_TESTS += disassembler-init-styled FEATURE_DISPLAY := clang-bpf-co-re +FEATURE_DISPLAY += llvm FEATURE_DISPLAY += libcap FEATURE_DISPLAY += libbfd @@ -133,27 +135,36 @@ all: $(OUTPUT)bpftool SRCS := $(wildcard *.c) -ifeq ($(feature-libbfd),1) - LIBS += -lbfd -ldl -lopcodes -else ifeq ($(feature-libbfd-liberty),1) - LIBS += -lbfd -ldl -lopcodes -liberty -else ifeq ($(feature-libbfd-liberty-z),1) - LIBS += -lbfd -ldl -lopcodes -liberty -lz -endif +ifeq ($(feature-llvm),1) + # If LLVM is available, use it for JIT disassembly + CFLAGS += -DHAVE_LLVM_SUPPORT + CFLAGS += $(shell llvm-config --cflags --libs) + LIBS += $(shell llvm-config --libs) + LDFLAGS += $(shell llvm-config --ldflags) +else + # Fall back on libbfd + ifeq ($(feature-libbfd),1) + LIBS += -lbfd -ldl -lopcodes + else ifeq ($(feature-libbfd-liberty),1) + LIBS += -lbfd -ldl -lopcodes -liberty + else ifeq ($(feature-libbfd-liberty-z),1) + LIBS += -lbfd -ldl -lopcodes -liberty -lz + endif -# If one of the above feature combinations is set, we support libbfd -ifneq ($(filter -lbfd,$(LIBS)),) - CFLAGS += -DHAVE_LIBBFD_SUPPORT + # If one of the above feature combinations is set, we support libbfd + ifneq ($(filter -lbfd,$(LIBS)),) + CFLAGS += -DHAVE_LIBBFD_SUPPORT - # Libbfd interface changed over time, figure out what we need - ifeq ($(feature-disassembler-four-args), 1) - CFLAGS += -DDISASM_FOUR_ARGS_SIGNATURE - endif - ifeq ($(feature-disassembler-init-styled), 1) - CFLAGS += -DDISASM_INIT_STYLED + # Libbfd interface changed over time, figure out what we need + ifeq ($(feature-disassembler-four-args), 1) + CFLAGS += -DDISASM_FOUR_ARGS_SIGNATURE + endif + ifeq ($(feature-disassembler-init-styled), 1) + CFLAGS += -DDISASM_INIT_STYLED + endif endif endif -ifeq ($(filter -DHAVE_LIBBFD_SUPPORT,$(CFLAGS)),) +ifeq ($(filter -DHAVE_LLVM_SUPPORT -DHAVE_LIBBFD_SUPPORT,$(CFLAGS)),) # No support for JIT disassembly SRCS := $(filter-out jit_disasm.c,$(SRCS)) endif diff --git a/tools/bpf/bpftool/jit_disasm.c b/tools/bpf/bpftool/jit_disasm.c index e31ad3950fd6..8128b0cf2ad3 100644 --- a/tools/bpf/bpftool/jit_disasm.c +++ b/tools/bpf/bpftool/jit_disasm.c @@ -20,18 +20,105 @@ #include <stdlib.h> #include <unistd.h> #include <string.h> -#include <bfd.h> -#include <dis-asm.h> #include <sys/stat.h> #include <limits.h> #include <bpf/libbpf.h> + +#ifdef HAVE_LLVM_SUPPORT +#include <llvm-c/Core.h> +#include <llvm-c/Disassembler.h> +#include <llvm-c/Target.h> +#include <llvm-c/TargetMachine.h> +#endif + +#ifdef HAVE_LIBBFD_SUPPORT +#include <bfd.h> +#include <dis-asm.h> #include <tools/dis-asm-compat.h> +#endif #include "json_writer.h" #include "main.h" static int oper_count; +#ifdef HAVE_LLVM_SUPPORT +#define DISASM_SPACER + +typedef LLVMDisasmContextRef disasm_ctx_t; + +static int printf_json(char *s) +{ + s = strtok(s, " \t"); + jsonw_string_field(json_wtr, "operation", s); + + jsonw_name(json_wtr, "operands"); + jsonw_start_array(json_wtr); + oper_count = 1; + + while ((s = strtok(NULL, " \t,()")) != 0) { + jsonw_string(json_wtr, s); + oper_count++; + } + return 0; +} + +static int +init_context(disasm_ctx_t *ctx, const char *arch, + __maybe_unused const char *disassembler_options, + __maybe_unused unsigned char *image, __maybe_unused ssize_t len) +{ + char *triple; + + if (arch) { + p_err("Architecture %s not supported", arch); + return -1; + } + + triple = LLVMGetDefaultTargetTriple(); + *ctx = LLVMCreateDisasm(triple, NULL, 0, NULL, NULL); + LLVMDisposeMessage(triple); + + if (!*ctx) { + p_err("Failed to create disassembler"); + return -1; + } + + return 0; +} + +static void destroy_context(disasm_ctx_t *ctx) +{ + LLVMDisposeMessage(*ctx); +} + +static int +disassemble_insn(disasm_ctx_t *ctx, unsigned char *image, ssize_t len, int pc) +{ + char buf[256]; + int count; + + count = LLVMDisasmInstruction(*ctx, image + pc, len - pc, pc, + buf, sizeof(buf)); + if (json_output) + printf_json(buf); + else + printf("%s", buf); + + return count; +} + +int disasm_init(void) +{ + LLVMInitializeNativeTarget(); + LLVMInitializeNativeDisassembler(); + return 0; +} +#endif /* HAVE_LLVM_SUPPORT */ + +#ifdef HAVE_LIBBFD_SUPPORT +#define DISASM_SPACER "\t" + typedef struct { struct disassemble_info *info; disassembler_ftype disassemble; @@ -210,6 +297,7 @@ int disasm_init(void) bfd_init(); return 0; } +#endif /* HAVE_LIBBPFD_SUPPORT */ int disasm_print_insn(unsigned char *image, ssize_t len, int opcodes, const char *arch, const char *disassembler_options, @@ -252,7 +340,7 @@ int disasm_print_insn(unsigned char *image, ssize_t len, int opcodes, if (linfo) btf_dump_linfo_plain(btf, linfo, "; ", linum); - printf("%4x:\t", pc); + printf("%4x:" DISASM_SPACER, pc); } count = disassemble_insn(&ctx, image, len, pc); diff --git a/tools/bpf/bpftool/main.h b/tools/bpf/bpftool/main.h index c9e171082cf6..9a149c67aa5d 100644 --- a/tools/bpf/bpftool/main.h +++ b/tools/bpf/bpftool/main.h @@ -172,7 +172,7 @@ int map_parse_fds(int *argc, char ***argv, int **fds); int map_parse_fd_and_info(int *argc, char ***argv, void *info, __u32 *info_len); struct bpf_prog_linfo; -#ifdef HAVE_LIBBFD_SUPPORT +#if defined(HAVE_LLVM_SUPPORT) || defined(HAVE_LIBBFD_SUPPORT) int disasm_print_insn(unsigned char *image, ssize_t len, int opcodes, const char *arch, const char *disassembler_options, const struct btf *btf, @@ -193,7 +193,7 @@ int disasm_print_insn(unsigned char *image, ssize_t len, int opcodes, } static inline int disasm_init(void) { - p_err("No libbfd support"); + p_err("No JIT disassembly support"); return -1; } #endif
To disassemble instructions for JIT-ed programs, bpftool has relied on the libbfd library. This has been problematic in the past: libbfd's interface is not meant to be stable and has changed several times. For building bpftool, we have to detect how the libbfd version on the system behaves, which is why we have to handle features disassembler-four-args and disassembler-init-styled in the Makefile. When it comes to shipping bpftool, this has also caused issues with several distribution maintainers unwilling to support the feature (see for example Debian's page for binutils-dev, which ships libbfd: "Note that building Debian packages which depend on the shared libbfd is Not Allowed." [0]). For these reasons, we add support for LLVM as an alternative to libbfd for disassembling instructions of JIT-ed programs. Thanks to the preparation work in the previous commits, it's easy to add the library by passing the relevant compilation options in the Makefile, and by adding the functions for setting up the LLVM disassembler in file jit_disasm.c. Naturally, the display of disassembled instructions comes with a few minor differences. Here is a sample output with libbfd (already supported before this patch): # bpftool prog dump jited id 56 bpf_prog_6deef7357e7b4530: 0: nopl 0x0(%rax,%rax,1) 5: xchg %ax,%ax 7: push %rbp 8: mov %rsp,%rbp b: push %rbx c: push %r13 e: push %r14 10: mov %rdi,%rbx 13: movzwq 0xb0(%rbx),%r13 1b: xor %r14d,%r14d 1e: or $0x2,%r14d 22: mov $0x1,%eax 27: cmp $0x2,%r14 2b: jne 0x000000000000002f 2d: xor %eax,%eax 2f: pop %r14 31: pop %r13 33: pop %rbx 34: leave 35: ret 36: int3 LLVM supports several variants that we could set when initialising the disassembler, for example with: LLVMSetDisasmOptions(*ctx, LLVMDisassembler_Option_AsmPrinterVariant); but the default printer is kept for now. Here is the output with LLVM: # bpftool prog dump jited id 56 bpf_prog_6deef7357e7b4530: 0: nopl (%rax,%rax) 5: nop 7: pushq %rbp 8: movq %rsp, %rbp b: pushq %rbx c: pushq %r13 e: pushq %r14 10: movq %rdi, %rbx 13: movzwq 176(%rbx), %r13 1b: xorl %r14d, %r14d 1e: orl $2, %r14d 22: movl $1, %eax 27: cmpq $2, %r14 2b: jne 2 2d: xorl %eax, %eax 2f: popq %r14 31: popq %r13 33: popq %rbx 34: leave 35: retq 36: int3 The LLVM disassembler comes as the default choice, with libbfd as a fall-back. Of course, we could replace libbfd entirely and avoid supporting two different libraries. One reason for keeping libbfd is that, right now, it works well, we have all we need in terms of features detection in the Makefile, so it provides a fallback for disassembling JIT-ed programs if libbfd is installed but LLVM is not. The other motivation is that libbfd supports nfp instruction for Netronome's SmartNICs and can be used to disassemble offloaded programs, something that LLVM cannot do. If libbfd's interface breaks again in the future, we might reconsider keeping support for it. [0] https://packages.debian.org/buster/binutils-dev Signed-off-by: Quentin Monnet <quentin@isovalent.com> --- tools/bpf/bpftool/Makefile | 45 ++++++++++------ tools/bpf/bpftool/jit_disasm.c | 94 ++++++++++++++++++++++++++++++++-- tools/bpf/bpftool/main.h | 4 +- 3 files changed, 121 insertions(+), 22 deletions(-)