| Message ID | 20220826064535.1941190-1-usama.anjum@collabora.com (mailing list archive) |
|---|---|
| Headers | show |
| Series | Implement IOCTL to get and clear soft dirty PTE | expand |
On 8/26/22 11:45 AM, Muhammad Usama Anjum wrote: > > Hello, > > This patch series implements a new ioctl on the pagemap proc fs file to > get, clear and perform both get and clear at the same time atomically on > the specified range of the memory. > > Soft-dirty PTE bit of the memory pages can be viewed by using pagemap > procfs file. The soft-dirty PTE bit for the whole memory range of the > process can be cleared by writing to the clear_refs file. This series > adds features that weren't present earlier. > - There is no atomic get soft-dirty PTE bit status and clear operation > present. > - The soft-dirty PTE bit of only a part of memory cannot be cleared. > > Historically, soft-dirty PTE bit tracking has been used in the CRIU > project. The proc fs interface is enough for that as I think the process > is frozen. We have the use case where we need to track the soft-dirty > PTE bit for the running processes. We need this tracking and clear > mechanism of a region of memory while the process is running to emulate > the getWriteWatch() syscall of Windows. This syscall is used by games to > keep track of dirty pages and keep processing only the dirty pages. This > new ioctl can be used by the CRIU project and other applications which > require soft-dirty PTE bit information. > > As in the current kernel there is no way to clear a part of memory (instead > of clearing the Soft-Dirty bits for the entire process) and get+clear > operation cannot be performed atomically, there are other methods to mimic > this information entirely in userspace with poor performance: > - The mprotect syscall and SIGSEGV handler for bookkeeping > - The userfaultfd syscall with the handler for bookkeeping > Some benchmarks can be seen [1]. > > This ioctl can be used by the CRIU project and other applications which > require soft-dirty PTE bit information. The following operations are > supported in this ioctl: > - Get the pages that are soft-dirty. > - Clear the pages which are soft-dirty. > - The optional flag to ignore the VM_SOFTDIRTY and only track per page > soft-dirty PTE bit Thoughts? > > There are two decisions which have been taken about how to get the output > from the syscall. > - Return offsets of the pages from the start in the vec > - Stop execution when vec is filled with dirty pages > These two arguments doesn't follow the mincore() philosophy where the > output array corresponds to the address range in one to one fashion, hence > the output buffer length isn't passed and only a flag is set if the page > is present. This makes mincore() easy to use with less control. We are > passing the size of the output array and putting return data consecutively > which is offset of dirty pages from the start. The user can convert these > offsets back into the dirty page addresses easily. Suppose, the user want > to get first 10 dirty pages from a total memory of 100 pages. He'll > allocate output buffer of size 10 and the ioctl will abort after finding the > 10 pages. This behaviour is needed to support Windows' getWriteWatch(). The > behaviour like mincore() can be achieved by passing output buffer of 100 > size. This interface can be used for any desired behaviour. > > [1] https://lore.kernel.org/lkml/54d4c322-cd6e-eefd-b161-2af2b56aae24@collabora.com/ > > Regards, > Muhammad Usama Anjum > > Muhammad Usama Anjum (4): > fs/proc/task_mmu: update functions to clear the soft-dirty PTE bit > fs/proc/task_mmu: Implement IOCTL to get and clear soft dirty PTE bit > selftests: vm: add pagemap ioctl tests > mm: add documentation of the new ioctl on pagemap > > Documentation/admin-guide/mm/soft-dirty.rst | 42 +- > fs/proc/task_mmu.c | 342 ++++++++++- > include/uapi/linux/fs.h | 23 + > tools/include/uapi/linux/fs.h | 23 + > tools/testing/selftests/vm/.gitignore | 1 + > tools/testing/selftests/vm/Makefile | 2 + > tools/testing/selftests/vm/pagemap_ioctl.c | 649 ++++++++++++++++++++ > 7 files changed, 1050 insertions(+), 32 deletions(-) > create mode 100644 tools/testing/selftests/vm/pagemap_ioctl.c >
On Fri, Aug 26, 2022 at 11:45:31AM +0500, Muhammad Usama Anjum wrote: > > Hello, > > This patch series implements a new ioctl on the pagemap proc fs file to > get, clear and perform both get and clear at the same time atomically on > the specified range of the memory. > > Soft-dirty PTE bit of the memory pages can be viewed by using pagemap > procfs file. The soft-dirty PTE bit for the whole memory range of the > process can be cleared by writing to the clear_refs file. This series > adds features that weren't present earlier. > - There is no atomic get soft-dirty PTE bit status and clear operation > present. > - The soft-dirty PTE bit of only a part of memory cannot be cleared. > > Historically, soft-dirty PTE bit tracking has been used in the CRIU > project. The proc fs interface is enough for that as I think the process > is frozen. We have the use case where we need to track the soft-dirty > PTE bit for the running processes. We need this tracking and clear > mechanism of a region of memory while the process is running to emulate > the getWriteWatch() syscall of Windows. This syscall is used by games to > keep track of dirty pages and keep processing only the dirty pages. This > new ioctl can be used by the CRIU project and other applications which > require soft-dirty PTE bit information. > > As in the current kernel there is no way to clear a part of memory (instead > of clearing the Soft-Dirty bits for the entire process) and get+clear > operation cannot be performed atomically, there are other methods to mimic > this information entirely in userspace with poor performance: > - The mprotect syscall and SIGSEGV handler for bookkeeping > - The userfaultfd syscall with the handler for bookkeeping > Some benchmarks can be seen [1]. > > This ioctl can be used by the CRIU project and other applications which > require soft-dirty PTE bit information. The following operations are > supported in this ioctl: > - Get the pages that are soft-dirty. I think this interface doesn't have to be limited by the soft-dirty bits only. For example, CRIU needs to know whether file, present and swap bits are set or not. I mean we should be able to specify for what pages we need to get info for. An ioctl argument can have these four fields: * required bits (rmask & mask == mask) - all bits from this mask have to be set. * any of these bits (amask & mask != 0) - any of these bits is set. * exclude masks (emask & mask == 0) = none of these bits are set. * return mask - bits that have to be reported to user. > - Clear the pages which are soft-dirty. > - The optional flag to ignore the VM_SOFTDIRTY and only track per page > soft-dirty PTE bit > > There are two decisions which have been taken about how to get the output > from the syscall. > - Return offsets of the pages from the start in the vec We can conside to return regions that contains pages with the same set of bits. struct page_region { void *start; long size; u64 bitmap; } And ioctl returns arrays of page_region-s. I believe it will be more compact form for many cases. > - Stop execution when vec is filled with dirty pages > These two arguments doesn't follow the mincore() philosophy where the > output array corresponds to the address range in one to one fashion, hence > the output buffer length isn't passed and only a flag is set if the page > is present. This makes mincore() easy to use with less control. We are > passing the size of the output array and putting return data consecutively > which is offset of dirty pages from the start. The user can convert these > offsets back into the dirty page addresses easily. Suppose, the user want > to get first 10 dirty pages from a total memory of 100 pages. He'll > allocate output buffer of size 10 and the ioctl will abort after finding the > 10 pages. This behaviour is needed to support Windows' getWriteWatch(). The > behaviour like mincore() can be achieved by passing output buffer of 100 > size. This interface can be used for any desired behaviour. > > [1] https://lore.kernel.org/lkml/54d4c322-cd6e-eefd-b161-2af2b56aae24@collabora.com/ > > Regards, > Muhammad Usama Anjum > > Muhammad Usama Anjum (4): > fs/proc/task_mmu: update functions to clear the soft-dirty PTE bit > fs/proc/task_mmu: Implement IOCTL to get and clear soft dirty PTE bit > selftests: vm: add pagemap ioctl tests > mm: add documentation of the new ioctl on pagemap > > Documentation/admin-guide/mm/soft-dirty.rst | 42 +- > fs/proc/task_mmu.c | 342 ++++++++++- > include/uapi/linux/fs.h | 23 + > tools/include/uapi/linux/fs.h | 23 + > tools/testing/selftests/vm/.gitignore | 1 + > tools/testing/selftests/vm/Makefile | 2 + > tools/testing/selftests/vm/pagemap_ioctl.c | 649 ++++++++++++++++++++ > 7 files changed, 1050 insertions(+), 32 deletions(-) > create mode 100644 tools/testing/selftests/vm/pagemap_ioctl.c > > -- > 2.30.2 >
Hi, Thank you for reviewing. On 9/19/22 7:58 PM, Andrei Vagin wrote: >> This ioctl can be used by the CRIU project and other applications which >> require soft-dirty PTE bit information. The following operations are >> supported in this ioctl: >> - Get the pages that are soft-dirty. > > I think this interface doesn't have to be limited by the soft-dirty > bits only. For example, CRIU needs to know whether file, present and swap bits > are set or not. These operations can be performed by pagemap procfs file. Definitely performing them through IOCTL will be faster. But I'm trying to add a simple IOCTL by which some specific PTE bit can be read and cleared atomically. This IOCTL can be extended to include other bits like file, present and swap bits by keeping the interface simple. The following mask advice is nice. But if we add that kind of masking, it'll start to look like a filter on top of pagemap. My intention is to not duplicate the functionality already provided by the pagemap. One may ask, then why am I adding "get the soft-dirty pages" functionality? I'm adding it to complement the get and clear operation. The "get" and "get and clear" operations with special flag (PAGEMAP_SD_NO_REUSED_REGIONS) can give results quicker by not splitting the VMAs. > > I mean we should be able to specify for what pages we need to get info > for. An ioctl argument can have these four fields: > * required bits (rmask & mask == mask) - all bits from this mask have to be set. > * any of these bits (amask & mask != 0) - any of these bits is set. > * exclude masks (emask & mask == 0) = none of these bits are set. > * return mask - bits that have to be reported to user. > >> - Clear the pages which are soft-dirty. >> - The optional flag to ignore the VM_SOFTDIRTY and only track per page >> soft-dirty PTE bit >> >> There are two decisions which have been taken about how to get the output >> from the syscall. >> - Return offsets of the pages from the start in the vec > > We can conside to return regions that contains pages with the same set > of bits. > > struct page_region { > void *start; > long size; > u64 bitmap; > } > > And ioctl returns arrays of page_region-s. I believe it will be more > compact form for many cases. Thank you for mentioning this. I'd considered this while development. But I gave up and used the simple array to return the offsets of the pages as in the problem I'm trying to solve, the dirty pages may be present amid non-dirty pages. The range may not be useful in that case. Also we want to return only a specific number of pages of interest. The following paragraph explains it. > >> - Stop execution when vec is filled with dirty pages >> These two arguments doesn't follow the mincore() philosophy where the >> output array corresponds to the address range in one to one fashion, hence >> the output buffer length isn't passed and only a flag is set if the page >> is present. This makes mincore() easy to use with less control. We are >> passing the size of the output array and putting return data consecutively >> which is offset of dirty pages from the start. The user can convert these >> offsets back into the dirty page addresses easily. Suppose, the user want >> to get first 10 dirty pages from a total memory of 100 pages. He'll >> allocate output buffer of size 10 and the ioctl will abort after finding the >> 10 pages. This behaviour is needed to support Windows' getWriteWatch(). The >> behaviour like mincore() can be achieved by passing output buffer of 100 >> size. This interface can be used for any desired behaviour. >> >> [1] https://lore.kernel.org/lkml/54d4c322-cd6e-eefd-b161-2af2b56aae24@collabora.com/ >> >> Regards, >> Muhammad Usama Anjum >> >> Muhammad Usama Anjum (4): >> fs/proc/task_mmu: update functions to clear the soft-dirty PTE bit >> fs/proc/task_mmu: Implement IOCTL to get and clear soft dirty PTE bit >> selftests: vm: add pagemap ioctl tests >> mm: add documentation of the new ioctl on pagemap >> >> Documentation/admin-guide/mm/soft-dirty.rst | 42 +- >> fs/proc/task_mmu.c | 342 ++++++++++- >> include/uapi/linux/fs.h | 23 + >> tools/include/uapi/linux/fs.h | 23 + >> tools/testing/selftests/vm/.gitignore | 1 + >> tools/testing/selftests/vm/Makefile | 2 + >> tools/testing/selftests/vm/pagemap_ioctl.c | 649 ++++++++++++++++++++ >> 7 files changed, 1050 insertions(+), 32 deletions(-) >> create mode 100644 tools/testing/selftests/vm/pagemap_ioctl.c >> >> -- >> 2.30.2 >>
Hi Suren, I'd shared this problem with you at Plumbers. Please review and share your thoughts. Thanks, Usama On 8/26/22 11:45 AM, Muhammad Usama Anjum wrote: > > Hello, > > This patch series implements a new ioctl on the pagemap proc fs file to > get, clear and perform both get and clear at the same time atomically on > the specified range of the memory. > > Soft-dirty PTE bit of the memory pages can be viewed by using pagemap > procfs file. The soft-dirty PTE bit for the whole memory range of the > process can be cleared by writing to the clear_refs file. This series > adds features that weren't present earlier. > - There is no atomic get soft-dirty PTE bit status and clear operation > present. > - The soft-dirty PTE bit of only a part of memory cannot be cleared. > > Historically, soft-dirty PTE bit tracking has been used in the CRIU > project. The proc fs interface is enough for that as I think the process > is frozen. We have the use case where we need to track the soft-dirty > PTE bit for the running processes. We need this tracking and clear > mechanism of a region of memory while the process is running to emulate > the getWriteWatch() syscall of Windows. This syscall is used by games to > keep track of dirty pages and keep processing only the dirty pages. This > new ioctl can be used by the CRIU project and other applications which > require soft-dirty PTE bit information. > > As in the current kernel there is no way to clear a part of memory (instead > of clearing the Soft-Dirty bits for the entire process) and get+clear > operation cannot be performed atomically, there are other methods to mimic > this information entirely in userspace with poor performance: > - The mprotect syscall and SIGSEGV handler for bookkeeping > - The userfaultfd syscall with the handler for bookkeeping > Some benchmarks can be seen [1]. > > This ioctl can be used by the CRIU project and other applications which > require soft-dirty PTE bit information. The following operations are > supported in this ioctl: > - Get the pages that are soft-dirty. > - Clear the pages which are soft-dirty. > - The optional flag to ignore the VM_SOFTDIRTY and only track per page > soft-dirty PTE bit > > There are two decisions which have been taken about how to get the output > from the syscall. > - Return offsets of the pages from the start in the vec > - Stop execution when vec is filled with dirty pages > These two arguments doesn't follow the mincore() philosophy where the > output array corresponds to the address range in one to one fashion, hence > the output buffer length isn't passed and only a flag is set if the page > is present. This makes mincore() easy to use with less control. We are > passing the size of the output array and putting return data consecutively > which is offset of dirty pages from the start. The user can convert these > offsets back into the dirty page addresses easily. Suppose, the user want > to get first 10 dirty pages from a total memory of 100 pages. He'll > allocate output buffer of size 10 and the ioctl will abort after finding the > 10 pages. This behaviour is needed to support Windows' getWriteWatch(). The > behaviour like mincore() can be achieved by passing output buffer of 100 > size. This interface can be used for any desired behaviour. > > [1] https://lore.kernel.org/lkml/54d4c322-cd6e-eefd-b161-2af2b56aae24@collabora.com/ > > Regards, > Muhammad Usama Anjum > > Muhammad Usama Anjum (4): > fs/proc/task_mmu: update functions to clear the soft-dirty PTE bit > fs/proc/task_mmu: Implement IOCTL to get and clear soft dirty PTE bit > selftests: vm: add pagemap ioctl tests > mm: add documentation of the new ioctl on pagemap > > Documentation/admin-guide/mm/soft-dirty.rst | 42 +- > fs/proc/task_mmu.c | 342 ++++++++++- > include/uapi/linux/fs.h | 23 + > tools/include/uapi/linux/fs.h | 23 + > tools/testing/selftests/vm/.gitignore | 1 + > tools/testing/selftests/vm/Makefile | 2 + > tools/testing/selftests/vm/pagemap_ioctl.c | 649 ++++++++++++++++++++ > 7 files changed, 1050 insertions(+), 32 deletions(-) > create mode 100644 tools/testing/selftests/vm/pagemap_ioctl.c >
Any thoughts about it? On 8/26/22 11:45 AM, Muhammad Usama Anjum wrote: > > Hello, > > This patch series implements a new ioctl on the pagemap proc fs file to > get, clear and perform both get and clear at the same time atomically on > the specified range of the memory. > > Soft-dirty PTE bit of the memory pages can be viewed by using pagemap > procfs file. The soft-dirty PTE bit for the whole memory range of the > process can be cleared by writing to the clear_refs file. This series > adds features that weren't present earlier. > - There is no atomic get soft-dirty PTE bit status and clear operation > present. > - The soft-dirty PTE bit of only a part of memory cannot be cleared. > > Historically, soft-dirty PTE bit tracking has been used in the CRIU > project. The proc fs interface is enough for that as I think the process > is frozen. We have the use case where we need to track the soft-dirty > PTE bit for the running processes. We need this tracking and clear > mechanism of a region of memory while the process is running to emulate > the getWriteWatch() syscall of Windows. This syscall is used by games to > keep track of dirty pages and keep processing only the dirty pages. This > new ioctl can be used by the CRIU project and other applications which > require soft-dirty PTE bit information. > > As in the current kernel there is no way to clear a part of memory (instead > of clearing the Soft-Dirty bits for the entire process) and get+clear > operation cannot be performed atomically, there are other methods to mimic > this information entirely in userspace with poor performance: > - The mprotect syscall and SIGSEGV handler for bookkeeping > - The userfaultfd syscall with the handler for bookkeeping > Some benchmarks can be seen [1]. > > This ioctl can be used by the CRIU project and other applications which > require soft-dirty PTE bit information. The following operations are > supported in this ioctl: > - Get the pages that are soft-dirty. > - Clear the pages which are soft-dirty. > - The optional flag to ignore the VM_SOFTDIRTY and only track per page > soft-dirty PTE bit > > There are two decisions which have been taken about how to get the output > from the syscall. > - Return offsets of the pages from the start in the vec > - Stop execution when vec is filled with dirty pages > These two arguments doesn't follow the mincore() philosophy where the > output array corresponds to the address range in one to one fashion, hence > the output buffer length isn't passed and only a flag is set if the page > is present. This makes mincore() easy to use with less control. We are > passing the size of the output array and putting return data consecutively > which is offset of dirty pages from the start. The user can convert these > offsets back into the dirty page addresses easily. Suppose, the user want > to get first 10 dirty pages from a total memory of 100 pages. He'll > allocate output buffer of size 10 and the ioctl will abort after finding the > 10 pages. This behaviour is needed to support Windows' getWriteWatch(). The > behaviour like mincore() can be achieved by passing output buffer of 100 > size. This interface can be used for any desired behaviour. > > [1] https://lore.kernel.org/lkml/54d4c322-cd6e-eefd-b161-2af2b56aae24@collabora.com/ > > Regards, > Muhammad Usama Anjum > > Muhammad Usama Anjum (4): > fs/proc/task_mmu: update functions to clear the soft-dirty PTE bit > fs/proc/task_mmu: Implement IOCTL to get and clear soft dirty PTE bit > selftests: vm: add pagemap ioctl tests > mm: add documentation of the new ioctl on pagemap > > Documentation/admin-guide/mm/soft-dirty.rst | 42 +- > fs/proc/task_mmu.c | 342 ++++++++++- > include/uapi/linux/fs.h | 23 + > tools/include/uapi/linux/fs.h | 23 + > tools/testing/selftests/vm/.gitignore | 1 + > tools/testing/selftests/vm/Makefile | 2 + > tools/testing/selftests/vm/pagemap_ioctl.c | 649 ++++++++++++++++++++ > 7 files changed, 1050 insertions(+), 32 deletions(-) > create mode 100644 tools/testing/selftests/vm/pagemap_ioctl.c >
On Wed, Sep 21, 2022 at 11:26 AM Muhammad Usama Anjum <usama.anjum@collabora.com> wrote: > > Hi, > > Thank you for reviewing. > > On 9/19/22 7:58 PM, Andrei Vagin wrote: > >> This ioctl can be used by the CRIU project and other applications which > >> require soft-dirty PTE bit information. The following operations are > >> supported in this ioctl: > >> - Get the pages that are soft-dirty. > > > > I think this interface doesn't have to be limited by the soft-dirty > > bits only. For example, CRIU needs to know whether file, present and swap bits > > are set or not. > These operations can be performed by pagemap procfs file. Definitely > performing them through IOCTL will be faster. But I'm trying to add a > simple IOCTL by which some specific PTE bit can be read and cleared > atomically. This IOCTL can be extended to include other bits like file, > present and swap bits by keeping the interface simple. The following > mask advice is nice. But if we add that kind of masking, it'll start to > look like a filter on top of pagemap. My intention is to not duplicate > the functionality already provided by the pagemap. One may ask, then why > am I adding "get the soft-dirty pages" functionality? I'm adding it to > complement the get and clear operation. The "get" and "get and clear" > operations with special flag (PAGEMAP_SD_NO_REUSED_REGIONS) can give > results quicker by not splitting the VMAs. This simple interface is good only for a limited number of use-cases. The interface that I suggest doesn't duplicate more code than this one, but it is much more universal. It will be a big mess if you add a separate API for each specific use-case. > > > > > I mean we should be able to specify for what pages we need to get info > > for. An ioctl argument can have these four fields: > > * required bits (rmask & mask == mask) - all bits from this mask have to be set. > > * any of these bits (amask & mask != 0) - any of these bits is set. > > * exclude masks (emask & mask == 0) = none of these bits are set. > > * return mask - bits that have to be reported to user. > > > >> - Clear the pages which are soft-dirty. > >> - The optional flag to ignore the VM_SOFTDIRTY and only track per page > >> soft-dirty PTE bit > >> > >> There are two decisions which have been taken about how to get the output > >> from the syscall. > >> - Return offsets of the pages from the start in the vec > > > > We can conside to return regions that contains pages with the same set > > of bits. > > > > struct page_region { > > void *start; > > long size; > > u64 bitmap; > > } > > > > And ioctl returns arrays of page_region-s. I believe it will be more > > compact form for many cases. > Thank you for mentioning this. I'd considered this while development. > But I gave up and used the simple array to return the offsets of the > pages as in the problem I'm trying to solve, the dirty pages may be > present amid non-dirty pages. The range may not be useful in that case. This is a good example. If we expect more than two consequent pages on average, the "region" interface looks more prefered. I don't know your use-case, but in the case of CRIU, this assumption looks reasonable. > Also we want to return only a specific number of pages of interest. The > following paragraph explains it. > > > > >> - Stop execution when vec is filled with dirty pages > >> These two arguments doesn't follow the mincore() philosophy where the > >> output array corresponds to the address range in one to one fashion, hence > >> the output buffer length isn't passed and only a flag is set if the page > >> is present. This makes mincore() easy to use with less control. We are > >> passing the size of the output array and putting return data consecutively > >> which is offset of dirty pages from the start. The user can convert these > >> offsets back into the dirty page addresses easily. Suppose, the user want > >> to get first 10 dirty pages from a total memory of 100 pages. He'll > >> allocate output buffer of size 10 and the ioctl will abort after finding the > >> 10 pages. This behaviour is needed to support Windows' getWriteWatch(). The > >> behaviour like mincore() can be achieved by passing output buffer of 100 > >> size. This interface can be used for any desired behaviour. Now, it is more clear where this interface came from. It repeats the interface of Windows' getWriteWatch. I think we have to look wider. The interface that reports regions will be more efficient for many use-cases. As for getWriteWatch, it will require a bit more code in user-space, but this code is trivial. Thanks, Andrei
On 9/28/22 10:24 PM, Andrei Vagin wrote: > On Wed, Sep 21, 2022 at 11:26 AM Muhammad Usama Anjum > <usama.anjum@collabora.com> wrote: >> >> Hi, >> >> Thank you for reviewing. >> >> On 9/19/22 7:58 PM, Andrei Vagin wrote: >>>> This ioctl can be used by the CRIU project and other applications which >>>> require soft-dirty PTE bit information. The following operations are >>>> supported in this ioctl: >>>> - Get the pages that are soft-dirty. >>> >>> I think this interface doesn't have to be limited by the soft-dirty >>> bits only. For example, CRIU needs to know whether file, present and swap bits >>> are set or not. >> These operations can be performed by pagemap procfs file. Definitely >> performing them through IOCTL will be faster. But I'm trying to add a >> simple IOCTL by which some specific PTE bit can be read and cleared >> atomically. This IOCTL can be extended to include other bits like file, >> present and swap bits by keeping the interface simple. The following >> mask advice is nice. But if we add that kind of masking, it'll start to >> look like a filter on top of pagemap. My intention is to not duplicate >> the functionality already provided by the pagemap. One may ask, then why >> am I adding "get the soft-dirty pages" functionality? I'm adding it to >> complement the get and clear operation. The "get" and "get and clear" >> operations with special flag (PAGEMAP_SD_NO_REUSED_REGIONS) can give >> results quicker by not splitting the VMAs. > > This simple interface is good only for a limited number of use-cases. > The interface > that I suggest doesn't duplicate more code than this one, but it is much more > universal. It will be a big mess if you add a separate API for each > specific use-case. > > >>> I mean we should be able to specify for what pages we need to get info >>> for. An ioctl argument can have these four fields: >>> * required bits (rmask & mask == mask) - all bits from this mask have to be set. >>> * any of these bits (amask & mask != 0) - any of these bits is set. >>> * exclude masks (emask & mask == 0) = none of these bits are set. >>> * return mask - bits that have to be reported to user. The required mask (rmask) makes sense to me. At the moment, I only know about the practical use case for the required mask. Can you share how can any and exclude masks help for the CRIU? >>> >>>> - Clear the pages which are soft-dirty. >>>> - The optional flag to ignore the VM_SOFTDIRTY and only track per page >>>> soft-dirty PTE bit >>>> >>>> There are two decisions which have been taken about how to get the output >>>> from the syscall. >>>> - Return offsets of the pages from the start in the vec >>> >>> We can conside to return regions that contains pages with the same set >>> of bits. >>> >>> struct page_region { >>> void *start; >>> long size; >>> u64 bitmap; >>> } >>> >>> And ioctl returns arrays of page_region-s. I believe it will be more >>> compact form for many cases. >> Thank you for mentioning this. I'd considered this while development. >> But I gave up and used the simple array to return the offsets of the >> pages as in the problem I'm trying to solve, the dirty pages may be >> present amid non-dirty pages. The range may not be useful in that case. > > This is a good example. If we expect more than two consequent pages > on average, the "region" interface looks more prefered. I don't know your > use-case, but in the case of CRIU, this assumption looks reasonable. > >> Also we want to return only a specific number of pages of interest. The >> following paragraph explains it. >> >>> >>>> - Stop execution when vec is filled with dirty pages >>>> These two arguments doesn't follow the mincore() philosophy where the >>>> output array corresponds to the address range in one to one fashion, hence >>>> the output buffer length isn't passed and only a flag is set if the page >>>> is present. This makes mincore() easy to use with less control. We are >>>> passing the size of the output array and putting return data consecutively >>>> which is offset of dirty pages from the start. The user can convert these >>>> offsets back into the dirty page addresses easily. Suppose, the user want >>>> to get first 10 dirty pages from a total memory of 100 pages. He'll >>>> allocate output buffer of size 10 and the ioctl will abort after finding the >>>> 10 pages. This behaviour is needed to support Windows' getWriteWatch(). The >>>> behaviour like mincore() can be achieved by passing output buffer of 100 >>>> size. This interface can be used for any desired behaviour. > > Now, it is more clear where this interface came from. It repeats the interface > of Windows' getWriteWatch. I think we have to look wider. The > interface that reports > regions will be more efficient for many use-cases. As for > getWriteWatch, it will require > a bit more code in user-space, but this code is trivial. > > Thanks, > Andrei
On Mon, Oct 03, 2022 at 04:21:22PM +0500, Muhammad Usama Anjum wrote: > On 9/28/22 10:24 PM, Andrei Vagin wrote: > > On Wed, Sep 21, 2022 at 11:26 AM Muhammad Usama Anjum > > <usama.anjum@collabora.com> wrote: > >> > >> Hi, > >> > >> Thank you for reviewing. > >> > >> On 9/19/22 7:58 PM, Andrei Vagin wrote: > >>>> This ioctl can be used by the CRIU project and other applications which > >>>> require soft-dirty PTE bit information. The following operations are > >>>> supported in this ioctl: > >>>> - Get the pages that are soft-dirty. > >>> > >>> I think this interface doesn't have to be limited by the soft-dirty > >>> bits only. For example, CRIU needs to know whether file, present and swap bits > >>> are set or not. > >> These operations can be performed by pagemap procfs file. Definitely > >> performing them through IOCTL will be faster. But I'm trying to add a > >> simple IOCTL by which some specific PTE bit can be read and cleared > >> atomically. This IOCTL can be extended to include other bits like file, > >> present and swap bits by keeping the interface simple. The following > >> mask advice is nice. But if we add that kind of masking, it'll start to > >> look like a filter on top of pagemap. My intention is to not duplicate > >> the functionality already provided by the pagemap. One may ask, then why > >> am I adding "get the soft-dirty pages" functionality? I'm adding it to > >> complement the get and clear operation. The "get" and "get and clear" > >> operations with special flag (PAGEMAP_SD_NO_REUSED_REGIONS) can give > >> results quicker by not splitting the VMAs. > > > > This simple interface is good only for a limited number of use-cases. > > The interface > > that I suggest doesn't duplicate more code than this one, but it is much more > > universal. It will be a big mess if you add a separate API for each > > specific use-case. > > > > > >>> I mean we should be able to specify for what pages we need to get info > >>> for. An ioctl argument can have these four fields: > >>> * required bits (rmask & mask == mask) - all bits from this mask have to be set. > >>> * any of these bits (amask & mask != 0) - any of these bits is set. > >>> * exclude masks (emask & mask == 0) = none of these bits are set. > >>> * return mask - bits that have to be reported to user. > The required mask (rmask) makes sense to me. At the moment, I only know > about the practical use case for the required mask. Can you share how > can any and exclude masks help for the CRIU? > I looked at should_dump_page in the CRIU code: https://github.com/checkpoint-restore/criu/blob/45641ab26d7bb78706a6215fdef8f9133abf8d10/criu/mem.c#L102 When CRIU dumps file private mappings, it needs to get pages that have PME_PRESENT or PME_SWAP but don't have PME_FILE. > >>>> - Clear the pages which are soft-dirty. > >>>> - The optional flag to ignore the VM_SOFTDIRTY and only track per page > >>>> soft-dirty PTE bit > >>>> > >>>> There are two decisions which have been taken about how to get the output > >>>> from the syscall. > >>>> - Return offsets of the pages from the start in the vec > >>> > >>> We can conside to return regions that contains pages with the same set > >>> of bits. > >>> > >>> struct page_region { > >>> void *start; > >>> long size; > >>> u64 bitmap; > >>> } > >>> > >>> And ioctl returns arrays of page_region-s. I believe it will be more > >>> compact form for many cases. > >> Thank you for mentioning this. I'd considered this while development. > >> But I gave up and used the simple array to return the offsets of the > >> pages as in the problem I'm trying to solve, the dirty pages may be > >> present amid non-dirty pages. The range may not be useful in that case. > > > > This is a good example. If we expect more than two consequent pages > > on average, the "region" interface looks more prefered. I don't know your > > use-case, but in the case of CRIU, this assumption looks reasonable. > > > >> Also we want to return only a specific number of pages of interest. The > >> following paragraph explains it. > >> > >>> > >>>> - Stop execution when vec is filled with dirty pages > >>>> These two arguments doesn't follow the mincore() philosophy where the > >>>> output array corresponds to the address range in one to one fashion, hence > >>>> the output buffer length isn't passed and only a flag is set if the page > >>>> is present. This makes mincore() easy to use with less control. We are > >>>> passing the size of the output array and putting return data consecutively > >>>> which is offset of dirty pages from the start. The user can convert these > >>>> offsets back into the dirty page addresses easily. Suppose, the user want > >>>> to get first 10 dirty pages from a total memory of 100 pages. He'll > >>>> allocate output buffer of size 10 and the ioctl will abort after finding the > >>>> 10 pages. This behaviour is needed to support Windows' getWriteWatch(). The > >>>> behaviour like mincore() can be achieved by passing output buffer of 100 > >>>> size. This interface can be used for any desired behaviour. > > > > Now, it is more clear where this interface came from. It repeats the > > interface of Windows' getWriteWatch. I think we have to look wider. > > The interface that reports regions will be more efficient for many > > use-cases. As for getWriteWatch, it will require a bit more code in > > user-space, but this code is trivial. I added Danylo to CC. I think he has a good use-case for the new interface. Danylo, could you describe it here. > > > > Thanks, > > Andrei > > -- > Muhammad Usama Anjum
On Tue, Oct 11, 2022 at 6:52 AM Andrei Vagin <avagin@gmail.com> wrote: > > On Mon, Oct 03, 2022 at 04:21:22PM +0500, Muhammad Usama Anjum wrote: > > On 9/28/22 10:24 PM, Andrei Vagin wrote: > > > On Wed, Sep 21, 2022 at 11:26 AM Muhammad Usama Anjum > > > <usama.anjum@collabora.com> wrote: > > >> > > >> Hi, > > >> > > >> Thank you for reviewing. > > >> > > >> On 9/19/22 7:58 PM, Andrei Vagin wrote: > > >>>> This ioctl can be used by the CRIU project and other applications which > > >>>> require soft-dirty PTE bit information. The following operations are > > >>>> supported in this ioctl: > > >>>> - Get the pages that are soft-dirty. > > >>> > > >>> I think this interface doesn't have to be limited by the soft-dirty > > >>> bits only. For example, CRIU needs to know whether file, present and swap bits > > >>> are set or not. > > >> These operations can be performed by pagemap procfs file. Definitely > > >> performing them through IOCTL will be faster. But I'm trying to add a > > >> simple IOCTL by which some specific PTE bit can be read and cleared > > >> atomically. This IOCTL can be extended to include other bits like file, > > >> present and swap bits by keeping the interface simple. The following > > >> mask advice is nice. But if we add that kind of masking, it'll start to > > >> look like a filter on top of pagemap. My intention is to not duplicate > > >> the functionality already provided by the pagemap. One may ask, then why > > >> am I adding "get the soft-dirty pages" functionality? I'm adding it to > > >> complement the get and clear operation. The "get" and "get and clear" > > >> operations with special flag (PAGEMAP_SD_NO_REUSED_REGIONS) can give > > >> results quicker by not splitting the VMAs. > > > > > > This simple interface is good only for a limited number of use-cases. > > > The interface > > > that I suggest doesn't duplicate more code than this one, but it is much more > > > universal. It will be a big mess if you add a separate API for each > > > specific use-case. > > > > > > > > >>> I mean we should be able to specify for what pages we need to get info > > >>> for. An ioctl argument can have these four fields: > > >>> * required bits (rmask & mask == mask) - all bits from this mask have to be set. > > >>> * any of these bits (amask & mask != 0) - any of these bits is set. > > >>> * exclude masks (emask & mask == 0) = none of these bits are set. > > >>> * return mask - bits that have to be reported to user. > > The required mask (rmask) makes sense to me. At the moment, I only know > > about the practical use case for the required mask. Can you share how > > can any and exclude masks help for the CRIU? > > > > I looked at should_dump_page in the CRIU code: > https://github.com/checkpoint-restore/criu/blob/45641ab26d7bb78706a6215fdef8f9133abf8d10/criu/mem.c#L102 > > When CRIU dumps file private mappings, it needs to get pages that have > PME_PRESENT or PME_SWAP but don't have PME_FILE. I would really like to see the mask discussed will be adopted. With it CRIU will be able to migrate huge sparse VMAs assuming that a single hole is processed in O(1) time. Use cases for migrating sparse VMAs are binaries sanitized with ASAN, MSAN or TSAN [1]. All of these sanitizers produce sparse mappings of shadow memory [2]. Being able to migrate such binaries allows to highly reduce the amount of work needed to identify and fix post-migration crashes, which happen constantly. > > > >>>> - Clear the pages which are soft-dirty. > > >>>> - The optional flag to ignore the VM_SOFTDIRTY and only track per page > > >>>> soft-dirty PTE bit > > >>>> > > >>>> There are two decisions which have been taken about how to get the output > > >>>> from the syscall. > > >>>> - Return offsets of the pages from the start in the vec > > >>> > > >>> We can conside to return regions that contains pages with the same set > > >>> of bits. > > >>> > > >>> struct page_region { > > >>> void *start; > > >>> long size; > > >>> u64 bitmap; > > >>> } > > >>> > > >>> And ioctl returns arrays of page_region-s. I believe it will be more > > >>> compact form for many cases. > > >> Thank you for mentioning this. I'd considered this while development. > > >> But I gave up and used the simple array to return the offsets of the > > >> pages as in the problem I'm trying to solve, the dirty pages may be > > >> present amid non-dirty pages. The range may not be useful in that case. > > > > > > This is a good example. If we expect more than two consequent pages > > > on average, the "region" interface looks more prefered. I don't know your > > > use-case, but in the case of CRIU, this assumption looks reasonable. Plus one for page_region data structure. It will make ASAN shadow memory representation much more compact as well as any other classical use-case. [1] https://github.com/google/sanitizers [2] https://github.com/google/sanitizers/wiki/AddressSanitizerAlgorithm#64-bit Best, Danylo
>>>>>> I mean we should be able to specify for what pages we need to get info >>>>>> for. An ioctl argument can have these four fields: >>>>>> * required bits (rmask & mask == mask) - all bits from this mask have to be set. >>>>>> * any of these bits (amask & mask != 0) - any of these bits is set. >>>>>> * exclude masks (emask & mask == 0) = none of these bits are set. >>>>>> * return mask - bits that have to be reported to user. >>> The required mask (rmask) makes sense to me. At the moment, I only know >>> about the practical use case for the required mask. Can you share how >>> can any and exclude masks help for the CRIU? >>> >> >> I looked at should_dump_page in the CRIU code: >> https://github.com/checkpoint-restore/criu/blob/45641ab26d7bb78706a6215fdef8f9133abf8d10/criu/mem.c#L102 >> >> When CRIU dumps file private mappings, it needs to get pages that have >> PME_PRESENT or PME_SWAP but don't have PME_FILE. > > I would really like to see the mask discussed will be adopted. With it CRIU will > be able to migrate huge sparse VMAs assuming that a single hole is processed in > O(1) time. > > Use cases for migrating sparse VMAs are binaries sanitized with ASAN, MSAN or > TSAN [1]. All of these sanitizers produce sparse mappings of shadow memory [2]. > Being able to migrate such binaries allows to highly reduce the amount of work > needed to identify and fix post-migration crashes, which happen constantly. > Hello all, I've included the masks which the CRIU developers have specified. max_out_page is another new optional variable which is needed to terminate the operation without visiting all the pages after finding the max_out_page number of desired pages. There is no way to terminate the operation without this variable. How does the interface looks now? Please comment. /* PAGEMAP IOCTL */ #define PAGEMAP_GET _IOWR('f', 16, struct pagemap_sd_args) #define PAGEMAP_CLEAR _IOWR('f', 17, struct pagemap_sd_args) #define PAGEMAP_GET_AND_CLEAR _IOWR('f', 18, struct pagemap_sd_args) /* Bits are set in the bitmap of the page_region and masks in pagemap_sd_args */ #define PAGE_IS_SD 1 << 0 #define PAGE_IS_FILE 1 << 1 #define PAGE_IS_PRESENT 1 << 2 #define PAGE_IS_SWAPED 1 << 3 /** * struct page_region - Page region with bitmap flags * @start: Start of the region * @len: Length of the region * bitmap: Bits sets for the region */ struct page_region { __u64 start; __u64 len; __u64 bitmap; }; /** * struct pagemap_sd_args - Soft-dirty IOCTL argument * @start: Starting address * @len: Length of the region * @vec: Output page_region struct array * @vec_len: Length of the page_region struct array * @max_out_page: Optional max output pages (It must be less than vec_len if specified) * @flags: Special flags for the IOCTL * @rmask: Special flags for the IOCTL * @amask: Special flags for the IOCTL * @emask: Special flags for the IOCTL * @__reserved: Reserved member to preserve data alignment. Must be 0. */ struct pagemap_sd_args { __u64 __user start; __u64 len; __u64 __user vec; // page_region __u64 vec_len; // sizeof(page_region) __u32 flags; // special flags __u32 rmask; __u32 amask; __u32 emask; __u32 max_out_page; __u32 __reserved; }; /* Special flags */ #define PAGEMAP_NO_REUSED_REGIONS 0x1 >> >>>>>>> - Clear the pages which are soft-dirty. >>>>>>> - The optional flag to ignore the VM_SOFTDIRTY and only track per page >>>>>>> soft-dirty PTE bit >>>>>>> >>>>>>> There are two decisions which have been taken about how to get the output >>>>>>> from the syscall. >>>>>>> - Return offsets of the pages from the start in the vec >>>>>> >>>>>> We can conside to return regions that contains pages with the same set >>>>>> of bits. >>>>>> >>>>>> struct page_region { >>>>>> void *start; >>>>>> long size; >>>>>> u64 bitmap; >>>>>> } >>>>>> >>>>>> And ioctl returns arrays of page_region-s. I believe it will be more >>>>>> compact form for many cases. >>>>> Thank you for mentioning this. I'd considered this while development. >>>>> But I gave up and used the simple array to return the offsets of the >>>>> pages as in the problem I'm trying to solve, the dirty pages may be >>>>> present amid non-dirty pages. The range may not be useful in that case. >>>> >>>> This is a good example. If we expect more than two consequent pages >>>> on average, the "region" interface looks more prefered. I don't know your >>>> use-case, but in the case of CRIU, this assumption looks reasonable. > > Plus one for page_region data structure. It will make ASAN shadow memory > representation much more compact as well as any other classical use-case. > > [1] https://github.com/google/sanitizers > [2] https://github.com/google/sanitizers/wiki/AddressSanitizerAlgorithm#64-bit > > Best, > Danylo >
On Tue, Oct 18, 2022 at 03:36:24PM +0500, Muhammad Usama Anjum wrote: > /** > * struct pagemap_sd_args - Soft-dirty IOCTL argument > * @start: Starting address > * @len: Length of the region > * @vec: Output page_region struct array > * @vec_len: Length of the page_region struct array > * @max_out_page: Optional max output pages (It must be less than vec_len if > specified) > * @flags: Special flags for the IOCTL > * @rmask: Special flags for the IOCTL > * @amask: Special flags for the IOCTL > * @emask: Special flags for the IOCTL What do you mean exactly by "special flags"? > * @__reserved: Reserved member to preserve data alignment. Must be 0. > */ > struct pagemap_sd_args { > __u64 __user start; > __u64 len; > __u64 __user vec; // page_region __user is a marking for a pointer, not a u64, right? Now the fact that you treat it like a pointer later in the kernel is different, but that shouldn't be on the uapi header file. You can put it in the kerneldoc, which you did not do. thanks, greg k-h
On Tue, 18 Oct 2022 at 12:36, Muhammad Usama Anjum <usama.anjum@collabora.com> wrote: [...] > I've included the masks which the CRIU developers have specified. > max_out_page is another new optional variable which is needed to > terminate the operation without visiting all the pages after finding the > max_out_page number of desired pages. There is no way to terminate the > operation without this variable. > > How does the interface looks now? Please comment. > > /* PAGEMAP IOCTL */ > #define PAGEMAP_GET _IOWR('f', 16, struct pagemap_sd_args) > #define PAGEMAP_CLEAR _IOWR('f', 17, struct pagemap_sd_args) > #define PAGEMAP_GET_AND_CLEAR _IOWR('f', 18, struct pagemap_sd_args) Why are three IOCTLs needed? Could CLEAR be a flag (like the PAGEMAP_NO_REUSED_REGIONS) or 'cmask' and GET be implied when vec != NULL? > /* Bits are set in the bitmap of the page_region and masks in > pagemap_sd_args */ > #define PAGE_IS_SD 1 << 0 > #define PAGE_IS_FILE 1 << 1 > #define PAGE_IS_PRESENT 1 << 2 > #define PAGE_IS_SWAPED 1 << 3 > > /** > * struct page_region - Page region with bitmap flags > * @start: Start of the region > * @len: Length of the region > * bitmap: Bits sets for the region > */ > struct page_region { > __u64 start; > __u64 len; > __u64 bitmap; > }; Could you explain what units start and len are using? Are they bytes or pages (what size)? > /** > * struct pagemap_sd_args - Soft-dirty IOCTL argument Nit: it's not soft-dirty-specific anymore. Maybe "pagemap_scan_args"? > * @start: Starting address > * @len: Length of the region > * @vec: Output page_region struct array > * @vec_len: Length of the page_region struct array > * @max_out_page: Optional max output pages (It must be less than > vec_len if specified) Why is it required to be less than vec_len? vec_len effectively specifies max number of ranges to find, and this new additional field counts pages, I suppose? BTW, if we count pages, then what size of them? Maybe using bytes (matching start/len fields) would be more consistent? > * @flags: Special flags for the IOCTL > * @rmask: Special flags for the IOCTL > * @amask: Special flags for the IOCTL > * @emask: Special flags for the IOCTL > * @__reserved: Reserved member to preserve data alignment. Must be 0. > */ > struct pagemap_sd_args { > __u64 __user start; > __u64 len; > __u64 __user vec; // page_region > __u64 vec_len; // sizeof(page_region) > __u32 flags; // special flags > __u32 rmask; > __u32 amask; > __u32 emask; > __u32 max_out_page; > __u32 __reserved; > }; > > /* Special flags */ > #define PAGEMAP_NO_REUSED_REGIONS 0x1 What does this flag do? Best Regards Michał Mirosław
On 10/18/22 3:48 PM, Greg KH wrote: > On Tue, Oct 18, 2022 at 03:36:24PM +0500, Muhammad Usama Anjum wrote: >> /** >> * struct pagemap_sd_args - Soft-dirty IOCTL argument >> * @start: Starting address >> * @len: Length of the region >> * @vec: Output page_region struct array >> * @vec_len: Length of the page_region struct array >> * @max_out_page: Optional max output pages (It must be less than vec_len if >> specified) >> * @flags: Special flags for the IOCTL >> * @rmask: Special flags for the IOCTL >> * @amask: Special flags for the IOCTL >> * @emask: Special flags for the IOCTL > > What do you mean exactly by "special flags"? Sorry typo in the comments above. Optional flag can be specified in the flag. At the moment, there is only one flag(PAGEMAP_NO_REUSED_REGIONS). /** * struct pagemap_sd_args - Soft-dirty IOCTL argument * @start: Starting address * @len: Length of the region * @vec: Output page_region struct array * @vec_len: Length of the page_region struct array * @max_out_page: Optional max output pages (It must be less than vec_len if specified) * @flags: Special flags for the IOCTL * @rmask: Required mask - All of these bits have to be set * @amask: Any mask - Any of these bits are set * @emask: Exclude mask - None of these bits are set * @rmask: Bits that have to be reported to the user in page_region */ struct pagemap_scan_args { __u64 __user start; __u64 len; __u64 __user vec; __u64 vec_len; __u32 max_out_page; __u32 flags; __u32 rmask; __u32 amask; __u32 emask; __u32 rmask; }; > >> * @__reserved: Reserved member to preserve data alignment. Must be 0. >> */ >> struct pagemap_sd_args { >> __u64 __user start; >> __u64 len; >> __u64 __user vec; // page_region > > __user is a marking for a pointer, not a u64, right? Now the fact that > you treat it like a pointer later in the kernel is different, but that > shouldn't be on the uapi header file. You can put it in the kerneldoc, > which you did not do. I'll update. > > thanks, > > greg k-h
On 10/18/22 4:11 PM, Michał Mirosław wrote: > On Tue, 18 Oct 2022 at 12:36, Muhammad Usama Anjum > <usama.anjum@collabora.com> wrote: > [...] >> I've included the masks which the CRIU developers have specified. >> max_out_page is another new optional variable which is needed to >> terminate the operation without visiting all the pages after finding the >> max_out_page number of desired pages. There is no way to terminate the >> operation without this variable. >> >> How does the interface looks now? Please comment. >> >> /* PAGEMAP IOCTL */ >> #define PAGEMAP_GET _IOWR('f', 16, struct pagemap_sd_args) >> #define PAGEMAP_CLEAR _IOWR('f', 17, struct pagemap_sd_args) >> #define PAGEMAP_GET_AND_CLEAR _IOWR('f', 18, struct pagemap_sd_args) > > Why are three IOCTLs needed? Could CLEAR be a flag (like the > PAGEMAP_NO_REUSED_REGIONS) or 'cmask' and GET be implied when vec != > NULL? Makes sense. Yes, this can be done. > >> /* Bits are set in the bitmap of the page_region and masks in >> pagemap_sd_args */ >> #define PAGE_IS_SD 1 << 0 >> #define PAGE_IS_FILE 1 << 1 >> #define PAGE_IS_PRESENT 1 << 2 >> #define PAGE_IS_SWAPED 1 << 3 >> >> /** >> * struct page_region - Page region with bitmap flags >> * @start: Start of the region >> * @len: Length of the region >> * bitmap: Bits sets for the region >> */ >> struct page_region { >> __u64 start; >> __u64 len; >> __u64 bitmap; >> }; > > Could you explain what units start and len are using? Are they bytes > or pages (what size)? These are page sizes. Start must be the starting address of the page. Length don't need to be the exact multiple of the page size. All the pages in the length are included just like mincore() syscall. > >> /** >> * struct pagemap_sd_args - Soft-dirty IOCTL argument > > Nit: it's not soft-dirty-specific anymore. Maybe "pagemap_scan_args"? > >> * @start: Starting address >> * @len: Length of the region >> * @vec: Output page_region struct array >> * @vec_len: Length of the page_region struct array >> * @max_out_page: Optional max output pages (It must be less than >> vec_len if specified) > > Why is it required to be less than vec_len? vec_len effectively > specifies max number of ranges to find, and this new additional field > counts pages, I suppose? > BTW, if we count pages, then what size of them? Maybe using bytes > (matching start/len fields) would be more consistent? Yes, it if for counting pages. As the regions can have multiple pages, user cannot specify through the number of regions that how many pages does he need. Page size is used here as well like the start and len. This is optional argument as this is only needed to emulate the Windows syscall getWriteWatch. > >> * @flags: Special flags for the IOCTL >> * @rmask: Special flags for the IOCTL >> * @amask: Special flags for the IOCTL >> * @emask: Special flags for the IOCTL >> * @__reserved: Reserved member to preserve data alignment. Must be 0. >> */ >> struct pagemap_sd_args { >> __u64 __user start; >> __u64 len; >> __u64 __user vec; // page_region >> __u64 vec_len; // sizeof(page_region) >> __u32 flags; // special flags >> __u32 rmask; >> __u32 amask; >> __u32 emask; >> __u32 max_out_page; >> __u32 __reserved; >> }; >> >> /* Special flags */ >> #define PAGEMAP_NO_REUSED_REGIONS 0x1 > > What does this flag do? Some non-dirty pages get marked as dirty because of the kernel's internal activity. The dirty bit of the pages is stored in the VMA flags and in the per page flags. If any of these two bits are set, the page is considered to be dirty. Suppose you have cleared the dirty bit of half of VMA which will be done by splitting the VMA and clearing dirty flag in the half VMA and the pages in it. Now kernel may decide to merge the VMAs again as dirty bit of VMAs isn't considered if the VMAs should be merged. So the half VMA becomes dirty again. This splitting/merging costs performance. The application receives a lot of pages which aren't dirty in reality but marked as dirty. Performance is lost again here. This PAGEMAP_NO_REUSED_REGIONS flag is used to don't depend on the dirty flag in the VMA flags. It only depends on the individual page dirty bit. With doing this, the new memory regions which are just created, doesn't look like dirty when seen with the IOCTL, but look dirty when seen from pagemap. This seems okay as the user of this flag know the implication of using it. > > Best Regards > Michał Mirosław
On 10/18/22 3:36 PM, Muhammad Usama Anjum wrote: >>>>>>> I mean we should be able to specify for what pages we need to get >>>>>>> info >>>>>>> for. An ioctl argument can have these four fields: >>>>>>> * required bits (rmask & mask == mask) - all bits from this mask >>>>>>> have to be set. >>>>>>> * any of these bits (amask & mask != 0) - any of these bits is set. >>>>>>> * exclude masks (emask & mask == 0) = none of these bits are set. >>>>>>> * return mask - bits that have to be reported to user. >>>> The required mask (rmask) makes sense to me. At the moment, I only know >>>> about the practical use case for the required mask. Can you share how >>>> can any and exclude masks help for the CRIU? >>>> >>> >>> I looked at should_dump_page in the CRIU code: >>> https://github.com/checkpoint-restore/criu/blob/45641ab26d7bb78706a6215fdef8f9133abf8d10/criu/mem.c#L102 >>> >>> When CRIU dumps file private mappings, it needs to get pages that have >>> PME_PRESENT or PME_SWAP but don't have PME_FILE. >> >> I would really like to see the mask discussed will be adopted. With it >> CRIU will >> be able to migrate huge sparse VMAs assuming that a single hole is >> processed in >> O(1) time. >> >> Use cases for migrating sparse VMAs are binaries sanitized with ASAN, >> MSAN or >> TSAN [1]. All of these sanitizers produce sparse mappings of shadow >> memory [2]. >> Being able to migrate such binaries allows to highly reduce the amount >> of work >> needed to identify and fix post-migration crashes, which happen >> constantly. >> > > Hello all, > > I've included the masks which the CRIU developers have specified. > max_out_page is another new optional variable which is needed to > terminate the operation without visiting all the pages after finding the > max_out_page number of desired pages. There is no way to terminate the > operation without this variable. > > How does the interface looks now? Please comment. > Updated interface with only one IOCTL. If vec is defined, get operation will be performed. If PAGEMAP_SD_CLEAR flag is specified, soft dirty bit will be cleared as well. CLEAR flag can only be specified for clearing soft dirty bit. /* PAGEMAP IOCTL */ #define PAGEMAP_SCAN _IOWR('f', 16, struct pagemap_sd_args) /* Bits are set in the bitmap of the page_region and masks in pagemap_sd_args */ #define PAGE_IS_SD 1 << 0 #define PAGE_IS_FILE 1 << 1 #define PAGE_IS_PRESENT 1 << 2 #define PAGE_IS_SWAPED 1 << 3 /** * struct page_region - Page region with bitmap flags * @start: Start of the region * @len: Length of the region * bitmap: Bits sets for the region */ struct page_region { __u64 start; __u64 len; __u64 bitmap; }; /** * struct pagemap_sd_args - Soft-dirty IOCTL argument * @start: Starting address of the page * @len: Length of the region (All the pages in this length are included) * @vec: Output page_region struct array * @vec_len: Length of the page_region struct array * @max_out_page: Optional max output pages (It must be less than vec_len if specified) * @flags: Special flags for the IOCTL * @rmask: Required mask - All of these bits have to be set * @amask: Any mask - Any of these bits are set * @emask: Exclude mask - None of these bits are set * @rmask: Bits that have to be reported to the user in page_region */ struct pagemap_scan_args { __u64 __user start; __u64 len; __u64 __user vec; __u64 vec_len; __u32 max_out_page; __u32 flags; __u32 rmask; __u32 amask; __u32 emask; __u32 rmask; }; /* Special flags */ #define PAGEMAP_SD_CLEAR 1 << 0 #define PAGEMAP_NO_REUSED_REGIONS 1 << 1 > /* PAGEMAP IOCTL */ > #define PAGEMAP_GET _IOWR('f', 16, struct pagemap_sd_args) > #define PAGEMAP_CLEAR _IOWR('f', 17, struct pagemap_sd_args) > #define PAGEMAP_GET_AND_CLEAR _IOWR('f', 18, struct pagemap_sd_args) > > /* Bits are set in the bitmap of the page_region and masks in > pagemap_sd_args */ > #define PAGE_IS_SD 1 << 0 > #define PAGE_IS_FILE 1 << 1 > #define PAGE_IS_PRESENT 1 << 2 > #define PAGE_IS_SWAPED 1 << 3 > > /** > * struct page_region - Page region with bitmap flags > * @start: Start of the region > * @len: Length of the region > * bitmap: Bits sets for the region > */ > struct page_region { > __u64 start; > __u64 len; > __u64 bitmap; > }; > > /** > * struct pagemap_sd_args - Soft-dirty IOCTL argument > * @start: Starting address > * @len: Length of the region > * @vec: Output page_region struct array > * @vec_len: Length of the page_region struct array > * @max_out_page: Optional max output pages (It must be less than > vec_len if specified) > * @flags: Special flags for the IOCTL > * @rmask: Special flags for the IOCTL > * @amask: Special flags for the IOCTL > * @emask: Special flags for the IOCTL > * @__reserved: Reserved member to preserve data alignment. Must > be 0. > */ > struct pagemap_sd_args { > __u64 __user start; > __u64 len; > __u64 __user vec; // page_region > __u64 vec_len; // sizeof(page_region) > __u32 flags; // special flags > __u32 rmask; > __u32 amask; > __u32 emask; > __u32 max_out_page; > __u32 __reserved; > }; > > /* Special flags */ > #define PAGEMAP_NO_REUSED_REGIONS 0x1 > > >>> >>>>>>>> - Clear the pages which are soft-dirty. >>>>>>>> - The optional flag to ignore the VM_SOFTDIRTY and only track >>>>>>>> per page >>>>>>>> soft-dirty PTE bit >>>>>>>> >>>>>>>> There are two decisions which have been taken about how to get >>>>>>>> the output >>>>>>>> from the syscall. >>>>>>>> - Return offsets of the pages from the start in the vec >>>>>>> >>>>>>> We can conside to return regions that contains pages with the >>>>>>> same set >>>>>>> of bits. >>>>>>> >>>>>>> struct page_region { >>>>>>> void *start; >>>>>>> long size; >>>>>>> u64 bitmap; >>>>>>> } >>>>>>> >>>>>>> And ioctl returns arrays of page_region-s. I believe it will be more >>>>>>> compact form for many cases. >>>>>> Thank you for mentioning this. I'd considered this while development. >>>>>> But I gave up and used the simple array to return the offsets of the >>>>>> pages as in the problem I'm trying to solve, the dirty pages may be >>>>>> present amid non-dirty pages. The range may not be useful in that >>>>>> case. >>>>> >>>>> This is a good example. If we expect more than two consequent pages >>>>> on average, the "region" interface looks more prefered. I don't >>>>> know your >>>>> use-case, but in the case of CRIU, this assumption looks reasonable. >> >> Plus one for page_region data structure. It will make ASAN shadow memory >> representation much more compact as well as any other classical use-case. >> >> [1] https://github.com/google/sanitizers >> [2] >> https://github.com/google/sanitizers/wiki/AddressSanitizerAlgorithm#64-bit >> >> Best, >> Danylo >>
On Tue, 18 Oct 2022 at 15:23, Muhammad Usama Anjum <usama.anjum@collabora.com> wrote: > > On 10/18/22 4:11 PM, Michał Mirosław wrote: > > On Tue, 18 Oct 2022 at 12:36, Muhammad Usama Anjum > > <usama.anjum@collabora.com> wrote: [...] > >> * @start: Starting address > >> * @len: Length of the region > >> * @vec: Output page_region struct array > >> * @vec_len: Length of the page_region struct array > >> * @max_out_page: Optional max output pages (It must be less than > >> vec_len if specified) > > > > Why is it required to be less than vec_len? vec_len effectively > > specifies max number of ranges to find, and this new additional field > > counts pages, I suppose? > > BTW, if we count pages, then what size of them? Maybe using bytes > > (matching start/len fields) would be more consistent? > Yes, it if for counting pages. As the regions can have multiple pages, > user cannot specify through the number of regions that how many pages > does he need. Page size is used here as well like the start and len. > This is optional argument as this is only needed to emulate the Windows > syscall getWriteWatch. I'm wondering about the condition that max_out_page < vec_len. Since both count different things (pages vs ranges) I would expect there is no strict relation between them and information returned is as much as fits both (IOW: at most vec_len ranges spanning not more than max_out_page pages). The field's name and description I'd suggest improving: maybe 'max_pages' with a comment that 0 = unlimited? [...] > >> /* Special flags */ > >> #define PAGEMAP_NO_REUSED_REGIONS 0x1 > > > > What does this flag do? > Some non-dirty pages get marked as dirty because of the kernel's > internal activity. The dirty bit of the pages is stored in the VMA flags > and in the per page flags. If any of these two bits are set, the page is > considered to be dirty. Suppose you have cleared the dirty bit of half > of VMA which will be done by splitting the VMA and clearing dirty flag > in the half VMA and the pages in it. Now kernel may decide to merge the > VMAs again as dirty bit of VMAs isn't considered if the VMAs should be > merged. So the half VMA becomes dirty again. This splitting/merging > costs performance. The application receives a lot of pages which aren't > dirty in reality but marked as dirty. Performance is lost again here. > > This PAGEMAP_NO_REUSED_REGIONS flag is used to don't depend on the dirty > flag in the VMA flags. It only depends on the individual page dirty bit. > With doing this, the new memory regions which are just created, doesn't > look like dirty when seen with the IOCTL, but look dirty when seen from > pagemap. This seems okay as the user of this flag know the implication > of using it. Thanks for explaining! Could you include this as a comment in the patch? Best Regards Michał Mirosław
On Tue, Oct 18, 2022 at 06:32:46PM +0500, Muhammad Usama Anjum wrote: > On 10/18/22 3:36 PM, Muhammad Usama Anjum wrote: > > > > > > > > I mean we should be able to specify for what > > > > > > > > pages we need to get info > > > > > > > > for. An ioctl argument can have these four fields: > > > > > > > > * required bits (rmask & mask == mask) - all > > > > > > > > bits from this mask have to be set. > > > > > > > > * any of these bits (amask & mask != 0) - any of these bits is set. > > > > > > > > * exclude masks (emask & mask == 0) = none of these bits are set. > > > > > > > > * return mask - bits that have to be reported to user. > > > > > The required mask (rmask) makes sense to me. At the moment, I only know > > > > > about the practical use case for the required mask. Can you share how > > > > > can any and exclude masks help for the CRIU? > > > > > > > > > > > > > I looked at should_dump_page in the CRIU code: > > > > https://github.com/checkpoint-restore/criu/blob/45641ab26d7bb78706a6215fdef8f9133abf8d10/criu/mem.c#L102 > > > > > > > > When CRIU dumps file private mappings, it needs to get pages that have > > > > PME_PRESENT or PME_SWAP but don't have PME_FILE. > > > > > > I would really like to see the mask discussed will be adopted. With > > > it CRIU will > > > be able to migrate huge sparse VMAs assuming that a single hole is > > > processed in > > > O(1) time. > > > > > > Use cases for migrating sparse VMAs are binaries sanitized with > > > ASAN, MSAN or > > > TSAN [1]. All of these sanitizers produce sparse mappings of shadow > > > memory [2]. > > > Being able to migrate such binaries allows to highly reduce the > > > amount of work > > > needed to identify and fix post-migration crashes, which happen > > > constantly. > > > > > > > Hello all, > > > > I've included the masks which the CRIU developers have specified. > > max_out_page is another new optional variable which is needed to > > terminate the operation without visiting all the pages after finding the > > max_out_page number of desired pages. There is no way to terminate the > > operation without this variable. > > > > How does the interface looks now? Please comment. > > > Updated interface with only one IOCTL. If vec is defined, get operation will > be performed. If PAGEMAP_SD_CLEAR flag is specified, soft dirty bit will be > cleared as well. CLEAR flag can only be specified for clearing soft dirty > bit. > > /* PAGEMAP IOCTL */ > #define PAGEMAP_SCAN _IOWR('f', 16, struct pagemap_sd_args) > > /* Bits are set in the bitmap of the page_region and masks in > pagemap_sd_args */ > #define PAGE_IS_SD 1 << 0 > #define PAGE_IS_FILE 1 << 1 > #define PAGE_IS_PRESENT 1 << 2 > #define PAGE_IS_SWAPED 1 << 3 > > /** > * struct page_region - Page region with bitmap flags > * @start: Start of the region > * @len: Length of the region > * bitmap: Bits sets for the region > */ > struct page_region { > __u64 start; > __u64 len; > __u64 bitmap; > }; > > /** > * struct pagemap_sd_args - Soft-dirty IOCTL argument > * @start: Starting address of the page > * @len: Length of the region (All the pages in this length are included) > * @vec: Output page_region struct array > * @vec_len: Length of the page_region struct array > * @max_out_page: Optional max output pages (It must be less than vec_len if > specified) > * @flags: Special flags for the IOCTL > * @rmask: Required mask - All of these bits have to be set Why have it at all if it always has to be all 1s? > * @amask: Any mask - Any of these bits are set which ones? > * @emask: Exclude mask - None of these bits are set Why have it, if none are ever set? > * @rmask: Bits that have to be reported to the user in page_region I feel like I have no idea what these bits are... Anyway, please send a real patch, with real code, so we have a better idea of what is happening. AFTER you have tested and made it all work properly. thanks, greg k-h
On 10/18/22 10:17 PM, Michał Mirosław wrote: > On Tue, 18 Oct 2022 at 15:23, Muhammad Usama Anjum > <usama.anjum@collabora.com> wrote: >> >> On 10/18/22 4:11 PM, Michał Mirosław wrote: >>> On Tue, 18 Oct 2022 at 12:36, Muhammad Usama Anjum >>> <usama.anjum@collabora.com> wrote: > [...] >>>> * @start: Starting address >>>> * @len: Length of the region >>>> * @vec: Output page_region struct array >>>> * @vec_len: Length of the page_region struct array >>>> * @max_out_page: Optional max output pages (It must be less than >>>> vec_len if specified) >>> >>> Why is it required to be less than vec_len? vec_len effectively >>> specifies max number of ranges to find, and this new additional field >>> counts pages, I suppose? >>> BTW, if we count pages, then what size of them? Maybe using bytes >>> (matching start/len fields) would be more consistent? >> Yes, it if for counting pages. As the regions can have multiple pages, >> user cannot specify through the number of regions that how many pages >> does he need. Page size is used here as well like the start and len. >> This is optional argument as this is only needed to emulate the Windows >> syscall getWriteWatch. > > I'm wondering about the condition that max_out_page < vec_len. Since > both count different things (pages vs ranges) I would expect there is > no strict relation between them and information returned is as much as > fits both (IOW: at most vec_len ranges spanning not more than > max_out_page pages). The field's name and description I'd suggest > improving: maybe 'max_pages' with a comment that 0 = unlimited? Correct, max_pages with this comment is what I want. I'll update. vec_len represents the total number of the page_range array elements. If the pages which we want to return are sparse or the consective pages have different flags, we'll only return one page in one page_range struct. In this case if someone has specified max_pages to be 10, vec_len must be at least 10 to keep store the 10 pages. So max_pages <= vec_len. > > [...] >>>> /* Special flags */ >>>> #define PAGEMAP_NO_REUSED_REGIONS 0x1 >>> >>> What does this flag do? >> Some non-dirty pages get marked as dirty because of the kernel's >> internal activity. The dirty bit of the pages is stored in the VMA flags >> and in the per page flags. If any of these two bits are set, the page is >> considered to be dirty. Suppose you have cleared the dirty bit of half >> of VMA which will be done by splitting the VMA and clearing dirty flag >> in the half VMA and the pages in it. Now kernel may decide to merge the >> VMAs again as dirty bit of VMAs isn't considered if the VMAs should be >> merged. So the half VMA becomes dirty again. This splitting/merging >> costs performance. The application receives a lot of pages which aren't >> dirty in reality but marked as dirty. Performance is lost again here. >> >> This PAGEMAP_NO_REUSED_REGIONS flag is used to don't depend on the dirty >> flag in the VMA flags. It only depends on the individual page dirty bit. >> With doing this, the new memory regions which are just created, doesn't >> look like dirty when seen with the IOCTL, but look dirty when seen from >> pagemap. This seems okay as the user of this flag know the implication >> of using it. > > Thanks for explaining! Could you include this as a comment in the patch? Will do. > > Best Regards > Michał Mirosław