| Message ID | 20220127085608.306306-1-rppt@kernel.org (mailing list archive) |
|---|---|
| Headers | show |
| Series | Prototype for direct map awareness in page allocator | expand |
On Thu, Jan 27, 2022 at 10:56:05AM +0200, Mike Rapoport wrote: > From: Mike Rapoport <rppt@linux.ibm.com> > > Hi, > > This is a second attempt to make page allocator aware of the direct map > layout and allow grouping of the pages that must be mapped at PTE level in > the direct map. > Hello mike, It may be a silly question... Looking at implementation of set_memory*(), they only split PMD/PUD-sized entries. But why not _merge_ them when all entries have same permissions after changing permission of an entry? I think grouping __GFP_UNMAPPED allocations would help reducing direct map fragmentation, but IMHO merging split entries seems better to be done in those helpers than in page allocator. For example: 1) set_memory_ro() splits 1 RW PMD entry into 511 RW PTE entries and 1 RO PTE entry. 2) before freeing the pages, we call set_memory_rw() and we have 512 RW PTE entries. Then we can merge it to 1 RW PMD entry. 3) after 2) we can do same thing about PMD-sized entries and merge them into 1 PUD entry if 512 PMD entries have same permissions. [...] > Mike Rapoport (3): > mm/page_alloc: introduce __GFP_UNMAPPED and MIGRATE_UNMAPPED > mm/secretmem: use __GFP_UNMAPPED to allocate pages > EXPERIMENTAL: x86/module: use __GFP_UNMAPPED in module_alloc > > arch/Kconfig | 7 ++ > arch/x86/Kconfig | 1 + > arch/x86/kernel/module.c | 2 +- > include/linux/gfp.h | 13 +++- > include/linux/mmzone.h | 11 +++ > include/trace/events/mmflags.h | 3 +- > mm/internal.h | 2 +- > mm/page_alloc.c | 129 ++++++++++++++++++++++++++++++++- > mm/secretmem.c | 8 +- > 9 files changed, 162 insertions(+), 14 deletions(-) > > > base-commit: e783362eb54cd99b2cac8b3a9aeac942e6f6ac07 > -- > 2.34.1 >
Hello Hyeonggon, On Tue, Apr 26, 2022 at 05:54:49PM +0900, Hyeonggon Yoo wrote: > On Thu, Jan 27, 2022 at 10:56:05AM +0200, Mike Rapoport wrote: > > From: Mike Rapoport <rppt@linux.ibm.com> > > > > Hi, > > > > This is a second attempt to make page allocator aware of the direct map > > layout and allow grouping of the pages that must be mapped at PTE level in > > the direct map. > > > > Hello mike, It may be a silly question... > > Looking at implementation of set_memory*(), they only split > PMD/PUD-sized entries. But why not _merge_ them when all entries > have same permissions after changing permission of an entry? > > I think grouping __GFP_UNMAPPED allocations would help reducing > direct map fragmentation, but IMHO merging split entries seems better > to be done in those helpers than in page allocator. Maybe, I didn't got as far as to try merging split entries in the direct map. IIRC, Kirill sent a patch for collapsing huge pages in the direct map some time ago, but there still was something that had to initiate the collapse. > For example: > 1) set_memory_ro() splits 1 RW PMD entry into 511 RW PTE > entries and 1 RO PTE entry. > > 2) before freeing the pages, we call set_memory_rw() and we have > 512 RW PTE entries. Then we can merge it to 1 RW PMD entry. For this we need to check permissions of all 512 pages to make sure we can use a PMD entry to map them. Not sure that doing the scan in each set_memory call won't cause an overall slowdown. > 3) after 2) we can do same thing about PMD-sized entries > and merge them into 1 PUD entry if 512 PMD entries have > same permissions. > > [...] > > > Mike Rapoport (3): > > mm/page_alloc: introduce __GFP_UNMAPPED and MIGRATE_UNMAPPED > > mm/secretmem: use __GFP_UNMAPPED to allocate pages > > EXPERIMENTAL: x86/module: use __GFP_UNMAPPED in module_alloc > > > > arch/Kconfig | 7 ++ > > arch/x86/Kconfig | 1 + > > arch/x86/kernel/module.c | 2 +- > > include/linux/gfp.h | 13 +++- > > include/linux/mmzone.h | 11 +++ > > include/trace/events/mmflags.h | 3 +- > > mm/internal.h | 2 +- > > mm/page_alloc.c | 129 ++++++++++++++++++++++++++++++++- > > mm/secretmem.c | 8 +- > > 9 files changed, 162 insertions(+), 14 deletions(-) > > > > > > base-commit: e783362eb54cd99b2cac8b3a9aeac942e6f6ac07 > > -- > > 2.34.1 > > > > -- > Thanks, > Hyeonggon
On Tue, Apr 26, 2022 at 06:21:57PM +0300, Mike Rapoport wrote: > Hello Hyeonggon, > > On Tue, Apr 26, 2022 at 05:54:49PM +0900, Hyeonggon Yoo wrote: > > On Thu, Jan 27, 2022 at 10:56:05AM +0200, Mike Rapoport wrote: > > > From: Mike Rapoport <rppt@linux.ibm.com> > > > > > > Hi, > > > > > > This is a second attempt to make page allocator aware of the direct map > > > layout and allow grouping of the pages that must be mapped at PTE level in > > > the direct map. > > > > > > > Hello mike, It may be a silly question... > > > > Looking at implementation of set_memory*(), they only split > > PMD/PUD-sized entries. But why not _merge_ them when all entries > > have same permissions after changing permission of an entry? > > > > I think grouping __GFP_UNMAPPED allocations would help reducing > > direct map fragmentation, but IMHO merging split entries seems better > > to be done in those helpers than in page allocator. > > Maybe, I didn't got as far as to try merging split entries in the direct > map. IIRC, Kirill sent a patch for collapsing huge pages in the direct map > some time ago, but there still was something that had to initiate the > collapse. But in this case buddy allocator's view of direct map is quite limited. It cannot merge 2M entries to 1G entry as it does not support big allocations. Also it cannot merge entries of pages freed in boot process as they weren't allocated from page allocator. And it will become harder when pages in MIGRATE_UNMAPPED is borrowed from another migrate type.... So it would be nice if we can efficiently merge mappings in change_page_attr_set(). this approach can handle cases above. I think in this case grouping allocations and merging mappings should be done separately. > > For example: > > 1) set_memory_ro() splits 1 RW PMD entry into 511 RW PTE > > entries and 1 RO PTE entry. > > > > 2) before freeing the pages, we call set_memory_rw() and we have > > 512 RW PTE entries. Then we can merge it to 1 RW PMD entry. > > For this we need to check permissions of all 512 pages to make sure we can > use a PMD entry to map them. Of course that may be slow. Maybe one way to optimize this is using some bits in struct page, something like: each bit of page->direct_map_split (unsigned long) is set when at least one entry in (PTRS_PER_PTE = 512)/(BITS_PER_LONG = 64) = 8 entries has special permissions. Then we just need to set the corresponding bit when splitting mappings and iterate 8 entries when changing permission back again. (and then unset the bit when 8 entries has usual permissions). we can decide to merge by checking if page->direct_map_split is zero. When scanning, 8 entries would fit into one cacheline. Any other ideas? > Not sure that doing the scan in each set_memory call won't cause an overall > slowdown. I think we can evaluate it by measuring boot time and bpf/module load/unload time. Is there any other workload that is directly affected by performance of set_memory*()? > > 3) after 2) we can do same thing about PMD-sized entries > > and merge them into 1 PUD entry if 512 PMD entries have > > same permissions. > > [...] > > > Mike Rapoport (3): > > > mm/page_alloc: introduce __GFP_UNMAPPED and MIGRATE_UNMAPPED > > > mm/secretmem: use __GFP_UNMAPPED to allocate pages > > > EXPERIMENTAL: x86/module: use __GFP_UNMAPPED in module_alloc > > -- > > Thanks, > > Hyeonggon > > -- > Sincerely yours, > Mike.
On Sat, Apr 30, 2022 at 01:44:16PM +0000, Hyeonggon Yoo wrote: > On Tue, Apr 26, 2022 at 06:21:57PM +0300, Mike Rapoport wrote: > > Hello Hyeonggon, > > > > On Tue, Apr 26, 2022 at 05:54:49PM +0900, Hyeonggon Yoo wrote: > > > On Thu, Jan 27, 2022 at 10:56:05AM +0200, Mike Rapoport wrote: > > > > From: Mike Rapoport <rppt@linux.ibm.com> > > > > > > > > Hi, > > > > > > > > This is a second attempt to make page allocator aware of the direct map > > > > layout and allow grouping of the pages that must be mapped at PTE level in > > > > the direct map. > > > > > > > > > > Hello mike, It may be a silly question... > > > > > > Looking at implementation of set_memory*(), they only split > > > PMD/PUD-sized entries. But why not _merge_ them when all entries > > > have same permissions after changing permission of an entry? > > > > > > I think grouping __GFP_UNMAPPED allocations would help reducing > > > direct map fragmentation, but IMHO merging split entries seems better > > > to be done in those helpers than in page allocator. > > > > Maybe, I didn't got as far as to try merging split entries in the direct > > map. IIRC, Kirill sent a patch for collapsing huge pages in the direct map > > some time ago, but there still was something that had to initiate the > > collapse. > > But in this case buddy allocator's view of direct map is quite limited. > It cannot merge 2M entries to 1G entry as it does not support > big allocations. Also it cannot merge entries of pages freed in boot process > as they weren't allocated from page allocator. > > And it will become harder when pages in MIGRATE_UNMAPPED is borrowed > from another migrate type.... > > So it would be nice if we can efficiently merge mappings in > change_page_attr_set(). this approach can handle cases above. > > I think in this case grouping allocations and merging mappings > should be done separately. I've added the provision to merge the mappings in __free_one_page() because at that spot we know for sure we can replace multiple PTEs with a single PMD. I'm not saying there should be no additional mechanism for collapsing direct map pages, but I don't know when and how it should be invoked. > > > For example: > > > 1) set_memory_ro() splits 1 RW PMD entry into 511 RW PTE > > > entries and 1 RO PTE entry. > > > > > > 2) before freeing the pages, we call set_memory_rw() and we have > > > 512 RW PTE entries. Then we can merge it to 1 RW PMD entry. > > > > For this we need to check permissions of all 512 pages to make sure we can > > use a PMD entry to map them. > > Of course that may be slow. Maybe one way to optimize this is using some bits > in struct page, something like: each bit of page->direct_map_split (unsigned long) > is set when at least one entry in (PTRS_PER_PTE = 512)/(BITS_PER_LONG = 64) = 8 entries > has special permissions. > > Then we just need to set the corresponding bit when splitting mappings and > iterate 8 entries when changing permission back again. (and then unset the bit when 8 entries has > usual permissions). we can decide to merge by checking if page->direct_map_split is zero. > > When scanning, 8 entries would fit into one cacheline. > > Any other ideas? > > > Not sure that doing the scan in each set_memory call won't cause an overall > > slowdown. > > I think we can evaluate it by measuring boot time and bpf/module > load/unload time. > > Is there any other workload that is directly affected > by performance of set_memory*()? > > > > 3) after 2) we can do same thing about PMD-sized entries > > > and merge them into 1 PUD entry if 512 PMD entries have > > > same permissions. > > > [...] > > > > Mike Rapoport (3): > > > > mm/page_alloc: introduce __GFP_UNMAPPED and MIGRATE_UNMAPPED > > > > mm/secretmem: use __GFP_UNMAPPED to allocate pages > > > > EXPERIMENTAL: x86/module: use __GFP_UNMAPPED in module_alloc > > > -- > > > Thanks, > > > Hyeonggon > > > > -- > > Sincerely yours, > > Mike.
On Mon, May 02, 2022 at 09:44:48PM -0700, Mike Rapoport wrote: > On Sat, Apr 30, 2022 at 01:44:16PM +0000, Hyeonggon Yoo wrote: > > On Tue, Apr 26, 2022 at 06:21:57PM +0300, Mike Rapoport wrote: > > > Hello Hyeonggon, > > > > > > On Tue, Apr 26, 2022 at 05:54:49PM +0900, Hyeonggon Yoo wrote: > > > > On Thu, Jan 27, 2022 at 10:56:05AM +0200, Mike Rapoport wrote: > > > > > From: Mike Rapoport <rppt@linux.ibm.com> > > > > > > > > > > Hi, > > > > > > > > > > This is a second attempt to make page allocator aware of the direct map > > > > > layout and allow grouping of the pages that must be mapped at PTE level in > > > > > the direct map. > > > > > > > > > > > > > Hello mike, It may be a silly question... > > > > > > > > Looking at implementation of set_memory*(), they only split > > > > PMD/PUD-sized entries. But why not _merge_ them when all entries > > > > have same permissions after changing permission of an entry? > > > > > > > > I think grouping __GFP_UNMAPPED allocations would help reducing > > > > direct map fragmentation, but IMHO merging split entries seems better > > > > to be done in those helpers than in page allocator. > > > > > > Maybe, I didn't got as far as to try merging split entries in the direct > > > map. IIRC, Kirill sent a patch for collapsing huge pages in the direct map > > > some time ago, but there still was something that had to initiate the > > > collapse. > > > > But in this case buddy allocator's view of direct map is quite limited. > > It cannot merge 2M entries to 1G entry as it does not support > > big allocations. Also it cannot merge entries of pages freed in boot process > > as they weren't allocated from page allocator. > > > > And it will become harder when pages in MIGRATE_UNMAPPED is borrowed > > from another migrate type.... > > > > So it would be nice if we can efficiently merge mappings in > > change_page_attr_set(). this approach can handle cases above. > > > > I think in this case grouping allocations and merging mappings > > should be done separately. > > I've added the provision to merge the mappings in __free_one_page() because > at that spot we know for sure we can replace multiple PTEs with a single > PMD. Actually no external merging mechanism is needed if CPA supports merging mappings. Recently I started to implement similar idea I described above. The approach is slightly different as it does not scan the page table but updates count of number of mappings that has non-standard protection bits. (being "non-standard" means pgprot is not equal to PAGE_KERNEL.) It increases split_count when standard mapping becomes non-standard and decreases split_count in the opposite case. It merges mappings when the count become zero. Updating counts and merging is invoked in __change_page_attr(), which is called by set_memory_{rw,ro}(), set_direct_map_{default,invalid}_noflush(), ... etc. The implementation looks like revert_page() function that existed in arch/i386/mm/pageattr.c decades ago... There are some issues like 1) set_memory_4k()-ed memory should not be merged and 2) we need to be extremely sure that the count is always valid. But I think this approach is definitely worth trying. I'll send a RFC versionin to list after a bit of more work. And still, I think grouping allocations using migrate type would work well with adding merging feature in CPA. Thanks! Hyeonggon > I'm not saying there should be no additional mechanism for collapsing > direct map pages, but I don't know when and how it should be invoked. > > > > > For example: > > > > 1) set_memory_ro() splits 1 RW PMD entry into 511 RW PTE > > > > entries and 1 RO PTE entry. > > > > > > > > 2) before freeing the pages, we call set_memory_rw() and we have > > > > 512 RW PTE entries. Then we can merge it to 1 RW PMD entry. > > > > > > For this we need to check permissions of all 512 pages to make sure we can > > > use a PMD entry to map them. > > > > Of course that may be slow. Maybe one way to optimize this is using some bits > > in struct page, something like: each bit of page->direct_map_split (unsigned long) > > is set when at least one entry in (PTRS_PER_PTE = 512)/(BITS_PER_LONG = 64) = 8 entries > > has special permissions. > > > > Then we just need to set the corresponding bit when splitting mappings and > > iterate 8 entries when changing permission back again. (and then unset the bit when 8 entries has > > usual permissions). we can decide to merge by checking if page->direct_map_split is zero. > > > > When scanning, 8 entries would fit into one cacheline. > > > > Any other ideas? > > > > > Not sure that doing the scan in each set_memory call won't cause an overall > > > slowdown. > > > > I think we can evaluate it by measuring boot time and bpf/module > > load/unload time. > > > > Is there any other workload that is directly affected > > by performance of set_memory*()? > > > > > > 3) after 2) we can do same thing about PMD-sized entries > > > > and merge them into 1 PUD entry if 512 PMD entries have > > > > same permissions. > > > > [...] > > > > > Mike Rapoport (3): > > > > > mm/page_alloc: introduce __GFP_UNMAPPED and MIGRATE_UNMAPPED > > > > > mm/secretmem: use __GFP_UNMAPPED to allocate pages > > > > > EXPERIMENTAL: x86/module: use __GFP_UNMAPPED in module_alloc > > > > -- > > > > Thanks, > > > > Hyeonggon > > > > > > -- > > > Sincerely yours, > > > Mike. > > -- > Sincerely yours, > Mike.
On Mon, May 02, 2022 at 09:44:48PM -0700, Mike Rapoport wrote: > On Sat, Apr 30, 2022 at 01:44:16PM +0000, Hyeonggon Yoo wrote: > > On Tue, Apr 26, 2022 at 06:21:57PM +0300, Mike Rapoport wrote: > > > Hello Hyeonggon, > > > > > > On Tue, Apr 26, 2022 at 05:54:49PM +0900, Hyeonggon Yoo wrote: > > > > On Thu, Jan 27, 2022 at 10:56:05AM +0200, Mike Rapoport wrote: > > > > > From: Mike Rapoport <rppt@linux.ibm.com> > > > > > > > > > > Hi, > > > > > > > > > > This is a second attempt to make page allocator aware of the direct map > > > > > layout and allow grouping of the pages that must be mapped at PTE level in > > > > > the direct map. > > > > > > > > > > > > > Hello mike, It may be a silly question... > > > > > > > > Looking at implementation of set_memory*(), they only split > > > > PMD/PUD-sized entries. But why not _merge_ them when all entries > > > > have same permissions after changing permission of an entry? > > > > > > > > I think grouping __GFP_UNMAPPED allocations would help reducing > > > > direct map fragmentation, but IMHO merging split entries seems better > > > > to be done in those helpers than in page allocator. > > > > > > Maybe, I didn't got as far as to try merging split entries in the direct > > > map. IIRC, Kirill sent a patch for collapsing huge pages in the direct map > > > some time ago, but there still was something that had to initiate the > > > collapse. > > > > But in this case buddy allocator's view of direct map is quite limited. > > It cannot merge 2M entries to 1G entry as it does not support > > big allocations. Also it cannot merge entries of pages freed in boot process > > as they weren't allocated from page allocator. > > > > And it will become harder when pages in MIGRATE_UNMAPPED is borrowed > > from another migrate type.... > > > > So it would be nice if we can efficiently merge mappings in > > change_page_attr_set(). this approach can handle cases above. > > > > I think in this case grouping allocations and merging mappings > > should be done separately. > > I've added the provision to merge the mappings in __free_one_page() because > at that spot we know for sure we can replace multiple PTEs with a single > PMD. > > I'm not saying there should be no additional mechanism for collapsing > direct map pages, but I don't know when and how it should be invoked. > I'm still thinking about a way to accurately track number of split pages - because tracking number of split pages only in CPA code may be inaccurate when kernel page table is changed outside CPA. In case you wonder, my code is available at: https://github.com/hygoni/linux/tree/merge-mapping-v1r3 it also adds vmstat items: # cat /proc/vmstat | grep direct_map direct_map_level2_splits 1079 direct_map_level3_splits 6 direct_map_level1_merges 1079 direct_map_level2_merges 6 Thanks, Hyeonggon > > > > For example: > > > > 1) set_memory_ro() splits 1 RW PMD entry into 511 RW PTE > > > > entries and 1 RO PTE entry. > > > > > > > > 2) before freeing the pages, we call set_memory_rw() and we have > > > > 512 RW PTE entries. Then we can merge it to 1 RW PMD entry. > > > > > > For this we need to check permissions of all 512 pages to make sure we can > > > use a PMD entry to map them. > > > > Of course that may be slow. Maybe one way to optimize this is using some bits > > in struct page, something like: each bit of page->direct_map_split (unsigned long) > > is set when at least one entry in (PTRS_PER_PTE = 512)/(BITS_PER_LONG = 64) = 8 entries > > has special permissions. > > > > Then we just need to set the corresponding bit when splitting mappings and > > iterate 8 entries when changing permission back again. (and then unset the bit when 8 entries has > > usual permissions). we can decide to merge by checking if page->direct_map_split is zero. > > > > When scanning, 8 entries would fit into one cacheline. > > > > Any other ideas? > > > > > Not sure that doing the scan in each set_memory call won't cause an overall > > > slowdown. > > > > I think we can evaluate it by measuring boot time and bpf/module > > load/unload time. > > > > Is there any other workload that is directly affected > > by performance of set_memory*()? > > > > > > 3) after 2) we can do same thing about PMD-sized entries > > > > and merge them into 1 PUD entry if 512 PMD entries have > > > > same permissions. > > > > [...] > > > > > Mike Rapoport (3): > > > > > mm/page_alloc: introduce __GFP_UNMAPPED and MIGRATE_UNMAPPED > > > > > mm/secretmem: use __GFP_UNMAPPED to allocate pages > > > > > EXPERIMENTAL: x86/module: use __GFP_UNMAPPED in module_alloc > > > > -- > > > > Thanks, > > > > Hyeonggon > > > > > > -- > > > Sincerely yours, > > > Mike. > > -- > Sincerely yours, > Mike.
From: Mike Rapoport <rppt@linux.ibm.com> Hi, This is a second attempt to make page allocator aware of the direct map layout and allow grouping of the pages that must be mapped at PTE level in the direct map. In a way this a v2 of "mm/page_alloc: cache pte-mapped allocations" (https://lore.kernel.org/all/20210823132513.15836-1-rppt@kernel.org) This time the grouping is implemented as a new migrate type - MIGRATE_UNMAPPED, and a GFP flag - __GFP_UNMAPPED to request such pages (more details are below). I've abandoned the example use-case of page table protection because people seemed to concentrate on it too much; instead, this set has two other use-cases as the examples of how __GFP_UNMAPPED can be used. First one is to switch secretmem to use the new mechanism, which is straight forward optimization. The second use-case is to enable __GFP_UNMAPPED in x86::module_alloc() that is essentially used as a method to allocate code pages and thus requires permission changes for basic pages in the direct map. This change reduced the amount of large page splits in the direct map from several tens to a single digit number for boot + several bpftrace runs. This set is x86 specific at the moment because other architectures either do not support set_memory APIs that split the direct^w linear map (e.g. PowerPC) or only enable set_memory APIs when the linear map uses basic page size (like arm64). == Motivation == There are use-cases that need to remove pages from the direct map or at least map them with 4K granularity. Whenever this is done e.g. with set_memory/set_direct_map APIs, the PUD and PMD sized mappings in the direct map are split into smaller pages. To reduce the performance hit caused by the fragmentation of the direct map it make sense to group and/or cache the 4K pages removed from the direct map so that the split large pages won't be all over the place. There were RFCs for grouped page allocations for vmalloc permissions [1] and for using PKS to protect page tables [2] as well as an attempt to use a pool of large pages in secretmtm [3]. == Implementation overview == The pages that need to be removed from the direct map are grouped in the free lists in a dedicated migrate type called MIGRATE_UNMAPPED. When there a page allocation request with __GFP_UNMAPPED set and MIGRATE_UNMAPPED list is empty, a page of the requested order is stolen from another migrate type, just like it would happen for the existing fallback processing. The difference is that the pageblock containing that page is remapped with PTEs in the direct map, and all the free pages in that pageblock are moved to MIGRATE_UNMAPPED unconditionally. The pages are actually unmapped only in the end of post_alloc_hook() and they are mapped back in free_pages_prepare() to keep the existing page initialization, clearing and poisoning logic intact. (this makes MIGRATE_UNMAPPED name imprecise, but I don't really like MIGRATE_PTE_MAPPED and have no better ideas) In this prototype __alloc_pages_bulk() does not deal with __GFP_UNMAPPED and this flag cannot be used with SL*B allocators. [1] https://lore.kernel.org/lkml/20210405203711.1095940-1-rick.p.edgecombe@intel.com [2] https://lore.kernel.org/lkml/20210505003032.489164-1-rick.p.edgecombe@intel.com [3] https://lore.kernel.org/lkml/20210121122723.3446-8-rppt@kernel.org Mike Rapoport (3): mm/page_alloc: introduce __GFP_UNMAPPED and MIGRATE_UNMAPPED mm/secretmem: use __GFP_UNMAPPED to allocate pages EXPERIMENTAL: x86/module: use __GFP_UNMAPPED in module_alloc arch/Kconfig | 7 ++ arch/x86/Kconfig | 1 + arch/x86/kernel/module.c | 2 +- include/linux/gfp.h | 13 +++- include/linux/mmzone.h | 11 +++ include/trace/events/mmflags.h | 3 +- mm/internal.h | 2 +- mm/page_alloc.c | 129 ++++++++++++++++++++++++++++++++- mm/secretmem.c | 8 +- 9 files changed, 162 insertions(+), 14 deletions(-) base-commit: e783362eb54cd99b2cac8b3a9aeac942e6f6ac07