| Message ID | 20230414052840.1994456-1-mcgrof@kernel.org (mailing list archive) |
|---|---|
| Headers | show |
| Series | module: fix virtual memory wasted on finit_module() | expand |
On Thu, Apr 13, 2023 at 10:28:38PM -0700, Luis Chamberlain wrote: > At first I wondered if > we could use file descriptor hints to just exlude users early on boot > before SYSTEM_RUNNING. I couldn't find much, but if there are some ways > to do that -- then the last patch can be simplified to do just that. > The second patch proves essentially that we can just send -EBUSY to > duplicate requests, at least for duplicate module loads and the world > doesn't fall apart. It *would* solve the issue. The patch however > borrows tons of the code from the first, and if we're realy going to > rely on something like that we may as well share. But I'm hopeful that > perhaps there are some jucier file descriptor tricks we can use to > just make a file mutually exlusivive and introduce a new kread which > lets finit_module() use that. The saving grace is that at least all > finit_module() calls *wait*, contray to request_module() calls and so > the solution can be much simpler. > > The end result is 0 wasted virtual memory bytes. > > Any ideas how not to make patch 2 suck as-is ? The more I think about it, a file descriptor based approach would have to loop over all tasks as we're not sure if userspace would use the same thread and just fork requests or what. One would then have to loop over all tasks and look for files that match the same name. This approach is domain specific to internal kernel reads and I am thinking now it is more appropriate. Since this is a bootup issue for races with module loading what we could do is just have say kernel_read_file_from_fd_excl() which would just call a shared __kernel_read_file_from_fd(..., bool excl, ...) and the kernel_read_file_from_fd() could just set that excl to false while the new one sets it to true. Then finit_module() could just use that. Luis
The graph from my v3 patch series [0] which tries to resolve the virtual memory lost bytes due to duplicates says it all: +----------------------------------------------------------------------------+ 14GB |-+ + + + + *+ +-| | **** | | *** | | ** | 12GB |-+ ** +-| | ** | | ** | | ** | | ** | 10GB |-+ ** +-| | ** | | ** | | ** | 8GB |-+ ** +-| waste | ** ### | | ** #### | | ** ####### | 6GB |-+ **** #### +-| | * #### | | * #### | | ***** #### | 4GB |-+ ** #### +-| | ** #### | | ** #### | | ** #### | 2GB |-+ ** ##### +-| | * #### | | * #### Before ******* | | **## + + + + After ####### | +----------------------------------------------------------------------------+ 0 50 100 150 200 250 300 CPUs count So we really need to debug to see WTF, because really, WTF. The first patch tries to answer the question if the issue is module auto-loading being abused and that causing the issues. The patch proves that the answer is no, but it does also help us find *a few* requests which can get a bit of love to avoid duplicates. My system at least found one. So it adds a debugging facility to let you do that. As I was writing the commit log for my first patch series [0] I was noting that this is it... and the obvious conclusion is that the culprit is udev issuing requests per CPU for tons of modules. I didn't feel comfortable in writing that this is it and we can't really do anything before really trying hard. So I gave it a good 'ol college try. At first I wondered if we could use file descriptor hints to just exlude users early on boot before SYSTEM_RUNNING. I couldn't find much, but if there are some ways to do that -- then the last patch can be simplified to do just that. The second patch proves essentially that we can just send -EBUSY to duplicate requests, at least for duplicate module loads and the world doesn't fall apart. It *would* solve the issue. The patch however borrows tons of the code from the first, and if we're realy going to rely on something like that we may as well share. But I'm hopeful that perhaps there are some jucier file descriptor tricks we can use to just make a file mutually exlusivive and introduce a new kread which lets finit_module() use that. The saving grace is that at least all finit_module() calls *wait*, contray to request_module() calls and so the solution can be much simpler. The end result is 0 wasted virtual memory bytes. Any ideas how not to make patch 2 suck as-is ? Yes -- we can also go fix udev, or libkmod, and that's what should be done. However, it seems silly to not fix if the fix is as trivial as patch 2 demonstrates. If you want to test / muck with all this you can use my branch 20230413-module-alloc-opts [1]: [0] https://lkml.kernel.org/r/20230414050836.1984746-1-mcgrof@kernel.org [1] https://git.kernel.org/pub/scm/linux/kernel/git/mcgrof/linux.git/log/?h=20230413-module-alloc-opts Luis Chamberlain (2): module: add debugging auto-load duplicate module support kread: avoid duplicates fs/kernel_read_file.c | 150 +++++++++++++++++++++++++ kernel/module/Kconfig | 40 +++++++ kernel/module/Makefile | 1 + kernel/module/dups.c | 234 +++++++++++++++++++++++++++++++++++++++ kernel/module/internal.h | 15 +++ kernel/module/kmod.c | 23 +++- kernel/module/main.c | 6 +- 7 files changed, 463 insertions(+), 6 deletions(-) create mode 100644 kernel/module/dups.c