| Message ID | cover.1695679700.git.falcon@tinylab.org (mailing list archive) |
|---|---|
| Headers | show |
| Series | DCE/DSE: Add Dead Syscalls Elimination support, part1 | expand |
On Tue, Sep 26, 2023, at 00:33, Zhangjin Wu wrote: > > This series aims to add DCE based DSE support, here is the first > revision of the RFC patchset [1], the whole series includes three parts, > here is the Part1. > > This Part1 adds basic DCE based DSE support. > > Part2 will further eliminate the unused syscalls forcely kept by the > exception tables. > > Part3 will add DSE test support with nolibc-test.c. I missed the RFC version, but I think this is a useful thing to have overall, though it will probably need to go through a couple of revisions and rewrites, mostly to ensure we are not adding complexity that gets in the way of other improvements I would like to see to the syscall entry handling. It would be nice to include some size numbers here for at least one practical use case. If you have a defconfig for a shipping product with a small kernel, what is the 'size -B' output you see comparing with and without DCE and, and with DCE+DSE? There is generally not much work going into micro-optimizing the size of the kernel image any more, for a number of reasons, but if you are able to show that this is a noticeable improvement, we should be able to find a way to do it. Geert is doing statistics about size bloat over time, and anything that undoes a couple of years worth of bloat would clearly be significant here. Another alternative would be to resume the work done by Nicolas Pitre, who added Kconfig symbols for controlling groups of system calls. Since we already have a number of those compile time options, adding more of them should generally be less controversial and more consistent, while bringing most of the same benefits. Arnd
On Tue, Sep 26, 2023, at 09:14, Arnd Bergmann wrote: > On Tue, Sep 26, 2023, at 00:33, Zhangjin Wu wrote: > > It would be nice to include some size numbers here for at least > one practical use case. If you have a defconfig for a shipping > product with a small kernel, what is the 'size -B' output you > see comparing with and without DCE and, and with DCE+DSE? To follow up on this myself, for a very rough baseline, I tried a riscv tinyconfig build with and without CONFIG_LD_DEAD_CODE_DATA_ELIMINATION (this is currently not supported on arm, so I did not try it there), and then another build with simply *all* system calls stubbed out by hacking asm/syscall-wrapper.h: $ size build/tmp/vmlinux-* text data bss dec hex filename 754772 220016 71841 1046629 ff865 vmlinux-tinyconfig 717500 223368 71841 1012709 f73e5 vmlinux-tiny+nosyscalls 567310 176200 71473 814983 c6f87 vmlinux-tiny+gc-sections 493278 170752 71433 735463 b38e7 vmlinux-tiny+gc-sections+nosyscalls 10120058 3572756 493701 14186515 d87813 vmlinux-defconfig 9953934 3529004 491525 13974463 d53bbf vmlinux-defconfig+gc 9709856 3500600 489221 13699677 d10a5d vmlinux-defconfig+gc+nosyscalls This would put us at an upper bound of 10% size savings (80kb) for tinyconfig, which is clearly significant. For defconfig, it's still 2.0% or 275kb size reduction when all syscalls are dropped. Arnd
On Tue, Sep 26, 2023, at 13:24, Arnd Bergmann wrote: > $ size build/tmp/vmlinux-* > text data bss dec hex filename > 754772 220016 71841 1046629 ff865 vmlinux-tinyconfig > 717500 223368 71841 1012709 f73e5 vmlinux-tiny+nosyscalls > 567310 176200 71473 814983 c6f87 vmlinux-tiny+gc-sections > 493278 170752 71433 735463 b38e7 vmlinux-tiny+gc-sections+nosyscalls > 10120058 3572756 493701 14186515 d87813 vmlinux-defconfig > 9953934 3529004 491525 13974463 d53bbf vmlinux-defconfig+gc > 9709856 3500600 489221 13699677 d10a5d vmlinux-defconfig+gc+nosyscalls > > This would put us at an upper bound of 10% size savings (80kb) for > tinyconfig, which is clearly significant. For defconfig, it's > still 2.0% or 275kb size reduction when all syscalls are dropped. I did one more test to see which syscalls actually cause bloat in when CONFIG_LD_DEAD_CODE_DATA_ELIMINATION is set in order to drop them all. I build the above riscv tinyconfig with CONFIG_LD_DEAD_CODE_DATA_ELIMINATION and truncated the syscall table before and after each syscall to see the size difference. A lot of syscalls are already conditional, so those show up as 0, 4 or 8 bytes (not sure why they are not always 0). Others could probably be made to fit within some category that can be made optional (e.g. xattr or adjtimex). Having a Kconfig option for those would also let users remove even more code that is not useful without the syscalls but might be called from somewhere else in the kernel. Arnd syscall size name ------------------------- 0 8 io_setup 1 4 io_destroy 2 8 io_submit 3 4 io_cancel 4 8 io_getevents 5 1496 setxattr 6 28 lsetxattr 7 148 fsetxattr 8 1404 getxattr 9 16 lgetxattr 10 80 fgetxattr 11 276 listxattr 12 16 llistxattr 13 68 flistxattr 14 460 removexattr 15 20 lremovexattr 16 92 fremovexattr 17 240 getcwd 18 4 lookup_dcookie 19 8 eventfd2 20 4 epoll_create1 21 8 epoll_ctl 22 4 epoll_pwait 23 64 dup 24 300 dup3 25 1684 fcntl 26 4 inotify_init1 27 8 inotify_add_watch 29 0 ioctl 28 4 inotify_rm_watch 30 8 ioprio_set 31 4 ioprio_get 32 8 flock 33 456 mknodat 34 192 mkdirat 35 64 unlinkat 36 208 symlinkat 38 0 renameat 37 324 linkat 40 0 mount 39 64 umount2 42 0 nfsservctl 41 708 pivot_root 43 424 statfs 44 132 fstatfs 45 272 truncate 46 216 ftruncate 47 88 fallocate 48 420 faccessat 49 120 chdir 50 112 fchdir 51 120 chroot 52 68 fchmod 53 164 fchmodat 54 184 fchownat 55 136 fchown 56 184 openat 57 204 close 58 4 vhangup 59 648 pipe2 61 0 getdents64 60 4 quotactl 62 148 lseek 63 328 read 64 356 write 65 952 readv 66 252 writev 67 92 pread64 68 92 pwrite64 69 100 preadv 71 0 sendfile 72 0 pselect6 70 100 pwritev 73 132 ppoll 74 4 signalfd4 75 2808 vmsplice 76 1388 splice 77 536 tee 78 424 readlinkat 79 244 fstatat 80 64 fstat 81 296 sync 82 100 fsync 83 20 fdatasync 84 448 sync_file_range 85 8 timerfd_create 86 4 timerfd_settime 87 8 timerfd_gettime 88 300 utimensat 89 4 acct 90 8 capget 91 4 capset 92 24 personality 93 24 exit 94 24 exit_group 95 16 waitid 96 28 set_tid_address 97 608 unshare 98 4 futex 99 8 set_robust_list 100 4 get_robust_list 101 276 nanosleep 103 0 setitimer 102 8 getitimer 104 4 kexec_load 105 8 init_module 107 0 timer_create 108 0 timer_gettime 109 0 timer_getoverrun 110 0 timer_settime 111 0 timer_delete 106 4 delete_module 112 44 clock_settime 113 88 clock_gettime 114 64 clock_getres 115 160 clock_nanosleep 116 8 syslog 117 740 ptrace 118 140 sched_setparam 119 36 sched_setscheduler 120 64 sched_getscheduler 121 88 sched_getparam 122 196 sched_setaffinity 123 180 sched_getaffinity 124 24 sched_yield 125 60 sched_get_priority_max 126 60 sched_get_priority_min 127 164 sched_rr_get_interval 128 12 restart_syscall 129 304 kill 130 212 tkill 131 40 tgkill 132 100 sigaltstack 133 104 rt_sigsuspend 134 396 rt_sigaction 135 180 rt_sigprocmask 136 76 rt_sigpending 137 336 rt_sigtimedwait 139 0 rt_sigreturn 138 120 rt_sigqueueinfo 140 396 setpriority 141 276 getpriority 142 1256 reboot 143 4 setregid 144 8 setgid 145 4 setreuid 146 8 setuid 147 4 setresuid 148 8 getresuid 149 4 setresgid 150 8 getresgid 151 4 setfsuid 152 8 setfsgid 153 152 times 154 252 setpgid 155 48 getpgid 156 48 getsid 157 140 setsid 158 8 getgroups 159 4 setgroups 160 172 uname 161 132 sethostname 162 136 setdomainname 163 156 getrlimit 164 52 setrlimit 165 88 getrusage 167 0 prctl 168 0 getcpu 169 0 gettimeofday 170 0 settimeofday 166 24 umask 171 1514 adjtimex 172 20 getpid 173 20 getppid 174 4 getuid 175 4 geteuid 176 4 getgid 177 4 getegid 178 20 gettid 179 276 sysinfo 180 4 mq_open 181 8 mq_unlink 182 4 mq_timedsend 183 8 mq_timedreceive 184 4 mq_notify 185 8 mq_getsetattr 186 4 msgget 187 8 msgctl 188 4 msgrcv 189 8 msgsnd 190 4 semget 191 8 semctl 192 4 semtimedop 193 8 semop 194 4 shmget 195 8 shmctl 196 4 shmat 197 8 shmdt 198 4 socket 199 8 socketpair 200 4 bind 201 8 listen 202 4 accept 203 8 connect 204 4 getsockname 205 8 getpeername 206 4 sendto 207 8 recvfrom 208 4 setsockopt 209 8 getsockopt 210 4 shutdown 211 8 sendmsg 212 4 recvmsg 213 460 readahead 214 2872 brk 215 288 munmap 216 4268 mremap 217 4 add_key 218 8 request_key 219 4 keyctl 220 100 clone 221 724 execve 222 2504 mmap 223 8 fadvise64 224 4 swapon 225 8 swapoff 226 2180 mprotect 227 320 msync 228 1140 mlock 229 84 munlock 230 304 mlockall 231 52 munlockall 232 828 mincore 233 4 madvise 234 324 remap_file_pages 235 4 mbind 236 8 get_mempolicy 237 4 set_mempolicy 238 8 migrate_pages 239 4 move_pages 240 132 rt_tgsigqueueinfo 241 8 perf_event_open 242 4 accept4 244 0 arch_specific_syscall 243 8 recvmmsg 260 100 wait4 261 252 prlimit64 262 8 fanotify_init 263 4 fanotify_mark 264 8 name_to_handle_at 266 0 clock_adjtime 265 4 open_by_handle_at 267 120 syncfs 268 624 setns 269 4 sendmmsg 270 8 process_vm_readv 271 4 process_vm_writev 272 8 kcmp 274 0 sched_setattr 273 4 finit_module 275 208 sched_getattr 276 2364 renameat2 277 4 seccomp 278 124 getrandom 279 4 memfd_create 280 8 bpf 281 52 execveat 282 4 userfaultfd 283 8 membarrier 284 40 mlock2 285 708 copy_file_range 286 32 preadv2 287 32 pwritev2 288 8 pkey_mprotect 289 4 pkey_alloc 290 8 pkey_free 291 356 statx 292 4 io_pgetevents 424 244 pidfd_send_signal 425 8 io_uring_setup 426 4 io_uring_enter 427 8 io_uring_register 428 368 open_tree 429 404 move_mount 430 556 fsopen 431 1056 fsconfig 432 484 fsmount 433 220 fspick 434 124 pidfd_open 435 516 clone3 436 240 close_range 437 120 openat2 438 304 pidfd_getfd 439 12 faccessat2 440 8 process_madvise 441 4 epoll_pwait2 442 1088 mount_setattr 443 8 quotactl_fd 444 4 landlock_create_ruleset 445 8 landlock_add_rule 446 4 landlock_restrict_self 447 8 memfd_secret 448 240 process_mrelease 449 4 futex_waitv 450 8 set_mempolicy_home_node 451 4 cachestat 452 28 fchmodat2 454 4 futex_wake
On Tue, 26 Sep 2023, Arnd Bergmann wrote: > On Tue, Sep 26, 2023, at 09:14, Arnd Bergmann wrote: > > On Tue, Sep 26, 2023, at 00:33, Zhangjin Wu wrote: > > > > It would be nice to include some size numbers here for at least > > one practical use case. If you have a defconfig for a shipping > > product with a small kernel, what is the 'size -B' output you > > see comparing with and without DCE and, and with DCE+DSE? > > To follow up on this myself, for a very rough baseline, > I tried a riscv tinyconfig build with and without > CONFIG_LD_DEAD_CODE_DATA_ELIMINATION (this is currently > not supported on arm, so I did not try it there), and > then another build with simply *all* system calls stubbed > out by hacking asm/syscall-wrapper.h: > > $ size build/tmp/vmlinux-* > text data bss dec hex filename > 754772 220016 71841 1046629 ff865 vmlinux-tinyconfig > 717500 223368 71841 1012709 f73e5 vmlinux-tiny+nosyscalls > 567310 176200 71473 814983 c6f87 vmlinux-tiny+gc-sections > 493278 170752 71433 735463 b38e7 vmlinux-tiny+gc-sections+nosyscalls > 10120058 3572756 493701 14186515 d87813 vmlinux-defconfig > 9953934 3529004 491525 13974463 d53bbf vmlinux-defconfig+gc > 9709856 3500600 489221 13699677 d10a5d vmlinux-defconfig+gc+nosyscalls > > This would put us at an upper bound of 10% size savings (80kb) for > tinyconfig, which is clearly significant. For defconfig, it's > still 2.0% or 275kb size reduction when all syscalls are dropped. I did something similar a while ago. Results included here: https://lwn.net/Articles/746780/ In my case, stubbing out all syscalls produced a 7.8% reduction which was somewhat disappointing compared to other techniques. Of course it all depends on what is your actual goal. Nicolas
On Tue, Sep 26, 2023, at 22:49, Nicolas Pitre wrote: > On Tue, 26 Sep 2023, Arnd Bergmann wrote: > >> $ size build/tmp/vmlinux-* >> text data bss dec hex filename >> 754772 220016 71841 1046629 ff865 vmlinux-tinyconfig >> 717500 223368 71841 1012709 f73e5 vmlinux-tiny+nosyscalls >> 567310 176200 71473 814983 c6f87 vmlinux-tiny+gc-sections >> 493278 170752 71433 735463 b38e7 vmlinux-tiny+gc-sections+nosyscalls >> 10120058 3572756 493701 14186515 d87813 vmlinux-defconfig >> 9953934 3529004 491525 13974463 d53bbf vmlinux-defconfig+gc >> 9709856 3500600 489221 13699677 d10a5d vmlinux-defconfig+gc+nosyscalls >> >> This would put us at an upper bound of 10% size savings (80kb) for >> tinyconfig, which is clearly significant. For defconfig, it's >> still 2.0% or 275kb size reduction when all syscalls are dropped. > > I did something similar a while ago. Results included here: > > https://lwn.net/Articles/746780/ > > In my case, stubbing out all syscalls produced a 7.8% reduction which > was somewhat disappointing compared to other techniques. Of course it > all depends on what is your actual goal. Thanks for the link, I had forgotten about your article. With all the findings combined, I guess the filtering at the syscall table level is not all that promising any more. Going through the list of saved space, I ended up with 5.7% (47kb) in the best case after I left the 40 syscalls from the example in this thread. Removing entire groups of features using normal Kconfig symbols based on the remaining syscalls that have the largest size probably gives better results. I can see possible groups of syscalls that could be disabled under CONFIG_EXPERT, along with making their underlying infrastructure optional: - xattr - ptrace - adjtimex - splice/vmsplice/tee - unshare/setns - sched_* After those, one would quickly hit diminishing returns. Arnd
Here is a test result about DEAD_CODE_DATA_ELIMINATION (DCE) and dead syscalls elimination (DSE). It's based on config[1] and a simple hello.c initramfs. In the DSE test, we set CONFIG_SYSCALLS_USED="sys_write sys_exit sys_reboot," which is used by hello.c to simply print "Hello" then exit and shut down qemu. | | syscall remain | vmlinux size | vmlinux after trip | | ---------------------------------- | -------------- | ---------------- | ------------------ | | disable DCE | 236 | 2559632 | 1963400 | | enable DCE | 208 | 2037384 (-20.4%) | 1485776 (-24.3%) | | enable DCE and DSE(SHE_GROUP) | 3 | 1856640 (-27.6%) | 1354424 (-31.0%) | | enable DCE and DSE(SHE_LINK_ORDER) | 3 | 1856664 (-27.6%) | 1354424 (-31.0%) | It shows that dead syscalls elimination can save 7% of space based on DCE. [1]: https://pastebin.com/KG4fd7aT
I don't know why linux-kernel@vger.kernel.org reject my email send out by thunderbird. So here I am resending this mail with git send-email. Here is a test result about DEAD_CODE_DATA_ELIMINATION (DCE) and dead syscalls elimination (DSE). It's based on config[1] and a simple hello.c initramfs. In the DSE test, we set CONFIG_SYSCALLS_USED="sys_write sys_exit sys_reboot," which is used by hello.c to simply print "Hello" then exit and shut down qemu. | | syscall remain | vmlinux size | vmlinux after strip | | ---------------------------------- | -------------- | ---------------- | ------------------- | | disable DCE | 236 | 2559632 | 1963400 | | enable DCE | 208 | 2037384 (-20.4%) | 1485776 (-24.3%) | | enable DCE and DSE(SHE_GROUP) | 3 | 1856640 (-27.6%) | 1354424 (-31.0%) | | enable DCE and DSE(SHE_LINK_ORDER) | 3 | 1856664 (-27.6%) | 1354424 (-31.0%) | It shows that dead syscalls elimination can save 7% of space based on DCE. [1]: https://pastebin.com/KG4fd7aT
I didn't test DSE with explicit KEEP() in the previous mail. So, I will make up for it now. This test result is about DEAD_CODE_DATA_ELIMINATION (DCE) and dead syscalls elimination (DSE). It's based on config[1] and a simple hello.c initramfs. We set CONFIG_SYSCALLS_USED="sys_write sys_exit sys_reboot", which is used by hello.c to simply print "Hello" then exit and shut down qemu. | | syscall remain | vmlinux size | vmlinux after strip | | ------------------------------------------------------------ | -------------- | ---------------- | ------------------- | | disable DCE | 236 | 2559632 | 1963400 | | enable DCE | 208 | 2037384 (-20.4%) | 1485776 (-24.3%) | | enable DCE and DSE with explicit KEEP() of except table | 17 | 1899208 (-25.8%) | 1387272 (-29.3%) | | enable DCE and DSE without KEEP() (By SHF_GROUP method) | 3 | 1856640 (-27.6%) | 1354424 (-31.0%) | | enable DCE and DSE without KEEP() (By SHE_LINK_ORDER method) | 3 | 1856664 (-27.6%) | 1354424 (-31.0%) | It shows that dead syscalls elimination can save 7% of space based on DCE. Although no KEEP() can only save up 2% space, it can reduce the attack surface and eliminate the misuse of KEEP(). It ensures that every orphan section is not orphaned anymore. [1]: https://pastebin.com/KG4fd7aT
Hi, all This series aims to add DCE based DSE support, here is the first revision of the RFC patchset [1], the whole series includes three parts, here is the Part1. This Part1 adds basic DCE based DSE support. Part2 will further eliminate the unused syscalls forcely kept by the exception tables. Part3 will add DSE test support with nolibc-test.c. Changes from RFC patchset [1]: - The DCE support [2] for RISC-V has been merged [3] - The "nolibc: Record used syscalls in their own sections" [4] will be delayed to Part3 - Add debug support for DCE - Further allows CONFIG_USED_SYSCALLS accept a file stores used syscalls - Now, only accepts symbolic syscalls, not support integral number again - Works with newly added riscv syscalls suffix: __riscv_ - Further trims the syscall tables by removing the tailing invalid parts The nolibc-test based initrd run well on riscv64 kernel image with dead syscalls eliminated: $ nm build/riscv64/virt/linux/v6.6-rc2/vmlinux | grep "T __riscv_sys" | grep -v sys_ni_syscall | wc -l 48 These options should be enabled: CONFIG_LD_DEAD_CODE_DATA_ELIMINATION=y CONFIG_LD_DEAD_CODE_DATA_ELIMINATION_DEBUG=y CONFIG_TRIM_UNUSED_SYSCALLS=y CONFIG_USED_SYSCALLS="sys_dup sys_dup3 sys_ioctl sys_mknodat sys_mkdirat sys_unlinkat sys_symlinkat sys_linkat sys_mount sys_chdir sys_chroot sys_fchmodat sys_fchownat sys_openat sys_close sys_pipe2 sys_getdents64 sys_lseek sys_read sys_write sys_pselect6 sys_ppoll sys_exit sys_sched_yield sys_kill sys_reboot sys_getpgid sys_prctl sys_gettimeofday sys_getpid sys_getppid sys_getuid sys_geteuid sys_brk sys_munmap sys_clone sys_execve sys_mmap sys_wait4 sys_statx" The really used syscalls: $ echo "sys_dup sys_dup3 sys_ioctl sys_mknodat sys_mkdirat sys_unlinkat sys_symlinkat sys_linkat sys_mount sys_chdir sys_chroot sys_fchmodat sys_fchownat sys_openat sys_close sys_pipe2 sys_getdents64 sys_lseek sys_read sys_write sys_pselect6 sys_ppoll sys_exit sys_sched_yield sys_kill sys_reboot sys_getpgid sys_prctl sys_gettimeofday sys_getpid sys_getppid sys_getuid sys_geteuid sys_brk sys_munmap sys_clone sys_execve sys_mmap sys_wait4 sys_statx" | tr ' ' '\n' | wc -l 40 Thanks to Yuan Tan, he has researched and verified the elimination of the unused syscalls forcely kept by the exception tables, both section group and section link order attributes of ld work. part2 will be sent out soon to further remove another 8 unused syscalls and eventually we are able to run a dead loop application on a kernel image without syscalls. Best Regards, Zhangjin Wu --- [1]: https://lore.kernel.org/lkml/cover.1676594211.git.falcon@tinylab.org/ [2]: https://lore.kernel.org/lkml/234017be6d06ef84844583230542e31068fa3685.1676594211.git.falcon@tinylab.org/ [3]: https://lore.kernel.org/lkml/CAFP8O3+41QFVyNTVJ2iZYkB0tqnvdLTAoGShgGy-qPP1PHjBEw@mail.gmail.com/ [4]: https://lore.kernel.org/lkml/cbcbfbb37cabfd9aed6088c75515e4ea86006cff.1676594211.git.falcon@tinylab.org/ Zhangjin Wu (7): DCE: add debug support DCE/DSE: add unused syscalls elimination configure support DCE/DSE: Add a new scripts/Makefile.syscalls DCE/DSE: mips: add HAVE_TRIM_UNUSED_SYSCALLS support DCE/DSE: riscv: move syscall tables to syscalls/ DCE/DSE: riscv: add HAVE_TRIM_UNUSED_SYSCALLS support DCE/DSE: riscv: trim syscall tables Makefile | 3 + arch/mips/Kconfig | 1 + arch/mips/kernel/syscalls/Makefile | 23 ++++++- arch/riscv/Kconfig | 1 + arch/riscv/include/asm/unistd.h | 2 + arch/riscv/kernel/Makefile | 7 +- arch/riscv/kernel/syscalls/Makefile | 69 +++++++++++++++++++ .../{ => syscalls}/compat_syscall_table.c | 4 +- .../kernel/{ => syscalls}/syscall_table.c | 4 +- init/Kconfig | 49 +++++++++++++ scripts/Makefile.syscalls | 29 ++++++++ 11 files changed, 182 insertions(+), 10 deletions(-) create mode 100644 arch/riscv/kernel/syscalls/Makefile rename arch/riscv/kernel/{ => syscalls}/compat_syscall_table.c (82%) rename arch/riscv/kernel/{ => syscalls}/syscall_table.c (83%) create mode 100644 scripts/Makefile.syscalls