| Message ID | 20210829183608.2297877-1-me@ubique.spb.ru (mailing list archive) |
|---|---|
| Headers | show |
| Series | bpfilter | expand |
On 8/29/21 12:35 PM, Dmitrii Banshchikov wrote: > The patchset is based on the patches from David S. Miller [1] and > Daniel Borkmann [2]. > > The main goal of the patchset is to prepare bpfilter for > iptables' configuration blob parsing and code generation. The referenced patches are from 2018. Since then, and since this is bpf-next, places like [1] indicate that we are moving on from iptables towards nftables. Any thoughts? [1] https://wiki.archlinux.org/title/Iptables > > The patchset introduces data structures and code for matches, > targets, rules and tables. Beside that the code generation > is introduced. > > The first version of the code generation supports only "inline" > mode - all chains and their rules emit instructions in linear > approach. The plan for the code generation is to introduce a > bpf_map_for_each subprogram that will handle all rules that > aren't generated in inline mode due to verifier's limit. This > shall allow to handle arbitrary large rule sets. > > Things that are not implemented yet: > 1) The process of switching from the previous BPF programs to the > new set isn't atomic. > 2) The code generation for FORWARD chain isn't supported > 3) Counters setsockopts() are not handled > 4) No support of device ifindex - it's hardcoded > 5) No helper subprog for counters update > > Another problem is using iptables' blobs for tests and filter > table initialization. While it saves lines something more > maintainable should be done here. > > The plan for the next iteration: > 1) Handle large rule sets via bpf_map_for_each > 2) Add a helper program for counters update > 3) Handle iptables' counters setsockopts() > 4) Handle ifindex > 5) Add TCP match > > Patch 1 adds definitions of the used types. > Patch 2 adds logging to bpfilter. > Patch 3 adds bpfilter header to tools > Patch 4 adds an associative map. > Patch 5 adds code generation basis > Patches 6/7/8/9 add code for matches, targets, rules and table. > Patch 10 adds code generation for table > Patch 11 handles hooked setsockopt(2) calls. > Patch 12 adds filter table > Patch 13 uses prepared code in main(). > ... > > 1. https://lore.kernel.org/patchwork/patch/902785/ > 2. https://lore.kernel.org/patchwork/patch/902783/ > 3. https://kernel.ubuntu.com/~cking/stress-ng/stress-ng.pdf
On Sun, Aug 29, 2021 at 01:13:53PM -0600, Raymond Burkholder wrote: > On 8/29/21 12:35 PM, Dmitrii Banshchikov wrote: > > The patchset is based on the patches from David S. Miller [1] and > > Daniel Borkmann [2]. > > > > The main goal of the patchset is to prepare bpfilter for > > iptables' configuration blob parsing and code generation. > > The referenced patches are from 2018. Since then, and since this is > bpf-next, places like [1] indicate that we are moving on from iptables > towards nftables. > > Any thoughts? I'm not sure what kind of thoughts you expect. If your question is why to use an outdated interface the answer is - this is just a starting point and nothing prevents us from having our own interface in future . > > [1] https://wiki.archlinux.org/title/Iptables >
On 2021-08-29 2:35 p.m., Dmitrii Banshchikov wrote: [..] > And here are some performance tests. > > The environment consists of two machines(sender and receiver) > connected with 10Gbps link via switch. The sender uses DPDK to > simulate QUIC packets(89 bytes long) from random IP. The switch > measures the generated traffic to be about 7066377568 bits/sec, > 9706553 packets/sec. > > The receiver is a 2 socket 2680v3 + HT and uses either iptables, > nft or bpfilter to filter out UDP traffic. > > Two tests were made. Two rulesets(default policy was to ACCEPT) > were used in each test: > > ``` > iptables -A INPUT -p udp -m udp --dport 1500 -j DROP > ``` > and > ``` > iptables -A INPUT -s 1.1.1.1/32 -p udp -m udp --dport 1000 -j DROP > iptables -A INPUT -s 2.2.2.2/32 -p udp -m udp --dport 2000 -j DROP > ... > iptables -A INPUT -s 31.31.31.31/32 -p udp -m udp --dport 31000 -j DROP > iptables -A INPUT -p udp -m udp --dport 1500 -j DROP > ``` > > The first test measures performance of the receiver via stress-ng > [3] in bogo-ops. The upper-bound(there are no firewall and no > traffic) value for bogo-ops is 8148-8210. The lower bound value > (there is traffic but no firewall) is 6567-6643. > The stress-ng command used: stress-ng -t60 -c48 --metrics-brief. > > The second test measures the number the of dropped packets. The > receiver keeps only 1 CPU online and disables all > others(maxcpus=1 and set number of cores per socket to 1 in > BIOS). The number of the dropped packets is collected via > iptables-legacy -nvL, iptables -nvL and bpftool map dump id. > > Test 1: bogo-ops(the more the better) > iptables nft bpfilter > 1 rule: 6474-6554 6483-6515 7996-8008 > 32 rules: 6374-6433 5761-5804 7997-8042 > > > Test 2: number of dropped packets(the more the better) > iptables nft bpfilter > 1 rule: 234M-241M 220M 900M+ > 32 rules: 186M-196M 97M-98M 900M+ > > > Please let me know if you see a gap in the testing environment. General perf testing will depend on the nature of the use case you are trying to target. What is the nature of the app? Is it just receiving packets and counting? Does it exemplify something something real in your network or is just purely benchmarking? Both are valid. What else can it do (eg are you interested in latency accounting etc)? What i have seen in practise for iptables deployments is a default drop and in general an accept list. Per ruleset IP address aggregation is typically achieved via ipset. So your mileage may vary... Having said that: Our testing[1] approach is typically for a worst case scenario. i.e we make sure you structure the rulesets such that all of the linear rulesets will be iterated and we eventually hit the default ruleset. We also try to reduce variability in the results. A lot of small things could affect your reproducibility, so we try to avoid them. For example, from what you described: You are sending from a random IP - that means each packet will hit a random ruleset (for the case of 32 rulesets). And some rules will likely be hit more often than others. The likelihood of reproducing the same results for multiple runs gets lower as you increase the number of rulesets. From a collection perspective: Looking at the nature of the CPU utilization is important Softirq vs system calls vs user app. Your test workload seems to be very specific to ingress host. So in reality you are more constrained by kernel->user syscalls (which will be hidden if you are mostly dropping in the kernel as opposed to letting packets go to user space). Something is not clear from your email: You seem to indicate that no traffic was running in test 1. If so, why would 32 rulesets give different results than 1? cheers, jamal [1] https://netdevconf.info/0x15/session.html?Linux-ACL-Performance-Analysis
On Mon, Aug 30, 2021 at 09:56:18PM -0400, Jamal Hadi Salim wrote: > On 2021-08-29 2:35 p.m., Dmitrii Banshchikov wrote: > > [..] > > > And here are some performance tests. > > > > The environment consists of two machines(sender and receiver) > > connected with 10Gbps link via switch. The sender uses DPDK to > > simulate QUIC packets(89 bytes long) from random IP. The switch > > measures the generated traffic to be about 7066377568 bits/sec, > > 9706553 packets/sec. > > > > The receiver is a 2 socket 2680v3 + HT and uses either iptables, > > nft or bpfilter to filter out UDP traffic. > > > > Two tests were made. Two rulesets(default policy was to ACCEPT) > > were used in each test: > > > > ``` > > iptables -A INPUT -p udp -m udp --dport 1500 -j DROP > > ``` > > and > > ``` > > iptables -A INPUT -s 1.1.1.1/32 -p udp -m udp --dport 1000 -j DROP > > iptables -A INPUT -s 2.2.2.2/32 -p udp -m udp --dport 2000 -j DROP > > ... > > iptables -A INPUT -s 31.31.31.31/32 -p udp -m udp --dport 31000 -j DROP > > iptables -A INPUT -p udp -m udp --dport 1500 -j DROP > > ``` > > > > The first test measures performance of the receiver via stress-ng > > [3] in bogo-ops. The upper-bound(there are no firewall and no > > traffic) value for bogo-ops is 8148-8210. The lower bound value > > (there is traffic but no firewall) is 6567-6643. > > The stress-ng command used: stress-ng -t60 -c48 --metrics-brief. > > > > The second test measures the number the of dropped packets. The > > receiver keeps only 1 CPU online and disables all > > others(maxcpus=1 and set number of cores per socket to 1 in > > BIOS). The number of the dropped packets is collected via > > iptables-legacy -nvL, iptables -nvL and bpftool map dump id. > > > > Test 1: bogo-ops(the more the better) > > iptables nft bpfilter > > 1 rule: 6474-6554 6483-6515 7996-8008 > > 32 rules: 6374-6433 5761-5804 7997-8042 > > > > > > Test 2: number of dropped packets(the more the better) > > iptables nft bpfilter > > 1 rule: 234M-241M 220M 900M+ > > 32 rules: 186M-196M 97M-98M 900M+ > > > > > > Please let me know if you see a gap in the testing environment. > > General perf testing will depend on the nature of the use case > you are trying to target. > What is the nature of the app? Is it just receiving packets and > counting? Does it exemplify something something real in your > network or is just purely benchmarking? Both are valid. > What else can it do (eg are you interested in latency accounting etc)? > What i have seen in practise for iptables deployments is a default > drop and in general an accept list. Per ruleset IP address aggregation > is typically achieved via ipset. So your mileage may vary... This was a pure benchmarking with the single goal - show that there might exist scenarios when using bpfilter may provide some performance benefits. > > Having said that: > Our testing[1] approach is typically for a worst case scenario. > i.e we make sure you structure the rulesets such that all of the > linear rulesets will be iterated and we eventually hit the default > ruleset. > We also try to reduce variability in the results. A lot of small > things could affect your reproducibility, so we try to avoid them. > For example, from what you described: > You are sending from a random IP - that means each packet will hit > a random ruleset (for the case of 32 rulesets). And some rules will > likely be hit more often than others. The likelihood of reproducing the > same results for multiple runs gets lower as you increase the number > of rulesets. > From a collection perspective: > Looking at the nature of the CPU utilization is important > Softirq vs system calls vs user app. > Your test workload seems to be very specific to ingress host. > So in reality you are more constrained by kernel->user syscalls > (which will be hidden if you are mostly dropping in the kernel > as opposed to letting packets go to user space). > > Something is not clear from your email: > You seem to indicate that no traffic was running in test 1. > If so, why would 32 rulesets give different results than 1? I mentioned the lower and upper bound values for bogo-ops on the machine. The lower bound is when there is traffic and no firewall at all. The upper bound is when there is no firewall and no traffic. Then the first test measures bogo-ops for two rule sets when there is traffic for either iptables, nft or bpfilter. > > cheers, > jamal > > [1] https://netdevconf.info/0x15/session.html?Linux-ACL-Performance-Analysis
On 2021-08-31 8:48 a.m., Dmitrii Banshchikov wrote: > On Mon, Aug 30, 2021 at 09:56:18PM -0400, Jamal Hadi Salim wrote: >> On 2021-08-29 2:35 p.m., Dmitrii Banshchikov wrote: >> >> Something is not clear from your email: >> You seem to indicate that no traffic was running in test 1. >> If so, why would 32 rulesets give different results than 1? > > I mentioned the lower and upper bound values for bogo-ops on the > machine. The lower bound is when there is traffic and no firewall > at all. The upper bound is when there is no firewall and no > traffic. Then the first test measures bogo-ops for two rule sets > when there is traffic for either iptables, nft or bpfilter. > Thanks, I totally misread that. I did look at stress-ng initially to use it to stress the system and emulate some real world setup (polluting cache etc) while testing but the variability of the results was bad - so dropped it earlier. Maybe we should look at it again. cheers, jamal
The patchset is based on the patches from David S. Miller [1] and Daniel Borkmann [2]. The main goal of the patchset is to prepare bpfilter for iptables' configuration blob parsing and code generation. The patchset introduces data structures and code for matches, targets, rules and tables. Beside that the code generation is introduced. The first version of the code generation supports only "inline" mode - all chains and their rules emit instructions in linear approach. The plan for the code generation is to introduce a bpf_map_for_each subprogram that will handle all rules that aren't generated in inline mode due to verifier's limit. This shall allow to handle arbitrary large rule sets. Things that are not implemented yet: 1) The process of switching from the previous BPF programs to the new set isn't atomic. 2) The code generation for FORWARD chain isn't supported 3) Counters setsockopts() are not handled 4) No support of device ifindex - it's hardcoded 5) No helper subprog for counters update Another problem is using iptables' blobs for tests and filter table initialization. While it saves lines something more maintainable should be done here. The plan for the next iteration: 1) Handle large rule sets via bpf_map_for_each 2) Add a helper program for counters update 3) Handle iptables' counters setsockopts() 4) Handle ifindex 5) Add TCP match Patch 1 adds definitions of the used types. Patch 2 adds logging to bpfilter. Patch 3 adds bpfilter header to tools Patch 4 adds an associative map. Patch 5 adds code generation basis Patches 6/7/8/9 add code for matches, targets, rules and table. Patch 10 adds code generation for table Patch 11 handles hooked setsockopt(2) calls. Patch 12 adds filter table Patch 13 uses prepared code in main(). And here are some performance tests. The environment consists of two machines(sender and receiver) connected with 10Gbps link via switch. The sender uses DPDK to simulate QUIC packets(89 bytes long) from random IP. The switch measures the generated traffic to be about 7066377568 bits/sec, 9706553 packets/sec. The receiver is a 2 socket 2680v3 + HT and uses either iptables, nft or bpfilter to filter out UDP traffic. Two tests were made. Two rulesets(default policy was to ACCEPT) were used in each test: ``` iptables -A INPUT -p udp -m udp --dport 1500 -j DROP ``` and ``` iptables -A INPUT -s 1.1.1.1/32 -p udp -m udp --dport 1000 -j DROP iptables -A INPUT -s 2.2.2.2/32 -p udp -m udp --dport 2000 -j DROP ... iptables -A INPUT -s 31.31.31.31/32 -p udp -m udp --dport 31000 -j DROP iptables -A INPUT -p udp -m udp --dport 1500 -j DROP ``` The first test measures performance of the receiver via stress-ng [3] in bogo-ops. The upper-bound(there are no firewall and no traffic) value for bogo-ops is 8148-8210. The lower bound value (there is traffic but no firewall) is 6567-6643. The stress-ng command used: stress-ng -t60 -c48 --metrics-brief. The second test measures the number the of dropped packets. The receiver keeps only 1 CPU online and disables all others(maxcpus=1 and set number of cores per socket to 1 in BIOS). The number of the dropped packets is collected via iptables-legacy -nvL, iptables -nvL and bpftool map dump id. Test 1: bogo-ops(the more the better) iptables nft bpfilter 1 rule: 6474-6554 6483-6515 7996-8008 32 rules: 6374-6433 5761-5804 7997-8042 Test 2: number of dropped packets(the more the better) iptables nft bpfilter 1 rule: 234M-241M 220M 900M+ 32 rules: 186M-196M 97M-98M 900M+ Please let me know if you see a gap in the testing environment. v1 -> v2 Maps: * Use map_upsert instead of separate map_insert and map_update Matches: * Add a new virtual call - gen_inline. The call is used for * inline generating of a rule's match. Targets: * Add a new virtual call - gen_inline. The call is used for inline generating of a rule's target. Rules: * Add code generation for rules Table: * Add struct table_ops * Add map for table_ops * Add filter table * Reorganize the way filter table is initialized Sockopts: * Install/uninstall BPF programs while handling IPT_SO_SET_REPLACE Code generation: * Add first version of the code generation Dependencies: * Add libbpf v0 -> v1 IO: * Use ssize_t in pvm_read, pvm_write for total_bytes * Move IO functions into sockopt.c and main.c Logging: * Use LOGLEVEL_EMERG, LOGLEVEL_NOTICE, LOGLEVE_DEBUG while logging to /dev/kmsg * Prepend log message with <n> where n is log level * Conditionally enable BFLOG_DEBUG messages * Merge bflog.{h,c} into context.h Matches: * Reorder fields in struct match_ops for tight packing * Get rid of struct match_ops_map * Rename udp_match_ops to xt_udp * Use XT_ALIGN macro * Store payload size in match size * Move udp match routines into a separate file Targets: * Reorder fields in struct target_ops for tight packing * Get rid of struct target_ops_map * Add comments for convert_verdict function Rules: * Add validation Tables: * Combine table_map and table_list into table_index * Add validation Sockopts: * Handle IPT_SO_GET_REVISION_TARGET 1. https://lore.kernel.org/patchwork/patch/902785/ 2. https://lore.kernel.org/patchwork/patch/902783/ 3. https://kernel.ubuntu.com/~cking/stress-ng/stress-ng.pdf Dmitrii Banshchikov (13): bpfilter: Add types for usermode helper bpfilter: Add logging facility tools: Add bpfilter usermode helper header bpfilter: Add map container bpfilter: Add codegen infrastructure bpfilter: Add struct match bpfilter: Add struct target bpfilter: Add struct rule bpfilter: Add struct table bpfilter: Add table codegen bpfilter: Add handling of setsockopt() calls bpfilter: Add filter table bpfilter: Handle setsockopts include/uapi/linux/bpfilter.h | 154 +++ net/bpfilter/Makefile | 16 +- net/bpfilter/codegen.c | 903 ++++++++++++++++++ net/bpfilter/codegen.h | 189 ++++ net/bpfilter/context.c | 138 +++ net/bpfilter/context.h | 47 + net/bpfilter/filter-table.c | 246 +++++ net/bpfilter/filter-table.h | 17 + net/bpfilter/main.c | 126 ++- net/bpfilter/map-common.c | 50 + net/bpfilter/map-common.h | 18 + net/bpfilter/match.c | 49 + net/bpfilter/match.h | 36 + net/bpfilter/rule.c | 239 +++++ net/bpfilter/rule.h | 34 + net/bpfilter/sockopt.c | 441 +++++++++ net/bpfilter/sockopt.h | 14 + net/bpfilter/table.c | 346 +++++++ net/bpfilter/table.h | 54 ++ net/bpfilter/target.c | 184 ++++ net/bpfilter/target.h | 52 + net/bpfilter/xt_udp.c | 96 ++ tools/include/uapi/linux/bpfilter.h | 178 ++++ .../testing/selftests/bpf/bpfilter/.gitignore | 8 + tools/testing/selftests/bpf/bpfilter/Makefile | 59 ++ .../selftests/bpf/bpfilter/bpfilter_util.h | 79 ++ .../selftests/bpf/bpfilter/test_codegen.c | 293 ++++++ .../testing/selftests/bpf/bpfilter/test_map.c | 63 ++ .../selftests/bpf/bpfilter/test_match.c | 61 ++ .../selftests/bpf/bpfilter/test_rule.c | 55 ++ .../selftests/bpf/bpfilter/test_target.c | 85 ++ .../selftests/bpf/bpfilter/test_xt_udp.c | 41 + 32 files changed, 4327 insertions(+), 44 deletions(-) create mode 100644 net/bpfilter/codegen.c create mode 100644 net/bpfilter/codegen.h create mode 100644 net/bpfilter/context.c create mode 100644 net/bpfilter/context.h create mode 100644 net/bpfilter/filter-table.c create mode 100644 net/bpfilter/filter-table.h create mode 100644 net/bpfilter/map-common.c create mode 100644 net/bpfilter/map-common.h create mode 100644 net/bpfilter/match.c create mode 100644 net/bpfilter/match.h create mode 100644 net/bpfilter/rule.c create mode 100644 net/bpfilter/rule.h create mode 100644 net/bpfilter/sockopt.c create mode 100644 net/bpfilter/sockopt.h create mode 100644 net/bpfilter/table.c create mode 100644 net/bpfilter/table.h create mode 100644 net/bpfilter/target.c create mode 100644 net/bpfilter/target.h create mode 100644 net/bpfilter/xt_udp.c create mode 100644 tools/include/uapi/linux/bpfilter.h create mode 100644 tools/testing/selftests/bpf/bpfilter/.gitignore create mode 100644 tools/testing/selftests/bpf/bpfilter/Makefile create mode 100644 tools/testing/selftests/bpf/bpfilter/bpfilter_util.h create mode 100644 tools/testing/selftests/bpf/bpfilter/test_codegen.c create mode 100644 tools/testing/selftests/bpf/bpfilter/test_map.c create mode 100644 tools/testing/selftests/bpf/bpfilter/test_match.c create mode 100644 tools/testing/selftests/bpf/bpfilter/test_rule.c create mode 100644 tools/testing/selftests/bpf/bpfilter/test_target.c create mode 100644 tools/testing/selftests/bpf/bpfilter/test_xt_udp.c