[RFC,0/3] PCI/P2PMEM: introduce pci_p2pdma_align_size API

Message

Max Gurtovoy June 2, 2021, 11:10 a.m. UTC

Hi all,
In hotplugged memory (from check_pfn_span function):
"
Disallow all operations smaller than a sub-section and only
allow operations smaller than a section for
SPARSEMEM_VMEMMAP. Note that check_hotplug_memory_range()
enforces a larger memory_block_size_bytes() granularity for
memory that will be marked online, so this check should only
fire for direct arch_{add,remove}_memory() users outside of
add_memory_resource()
"

This restriction will disqualify, for example, large NVMe CMBs that might have
non power of 2 number of pages (e.g. 32767 pages of 4KB). For these
devices, the CMB size will be rounded down from 0x7fff000 to 0x7e00000
but it's better than having un-mapped CMB.

If we all agree on the approach, this RFC can assist as-is to these NVMe
devices and other P2PMEM devices in the future and can considered for
the next merge window.

Max Gurtovoy (3):
  mm,memory_hotplug: export mhp min alignment
  PCI/P2PMEM: introduce pci_p2pdma_align_size API
  nvme-pci: align CMB size according to P2PMEM alignment

 drivers/nvme/host/pci.c        |  7 +++++++
 drivers/pci/p2pdma.c           | 23 +++++++++++++++++++++++
 include/linux/memory_hotplug.h |  5 +++++
 include/linux/pci-p2pdma.h     |  5 +++++
 mm/memory_hotplug.c            | 33 +++++++++++++++++++--------------
 5 files changed, 59 insertions(+), 14 deletions(-)

Comments

David Hildenbrand June 2, 2021, 12:08 p.m. UTC | #1

On 02.06.21 13:10, Max Gurtovoy wrote:
> Hi all,
> In hotplugged memory (from check_pfn_span function):
> "
> Disallow all operations smaller than a sub-section and only
> allow operations smaller than a section for
> SPARSEMEM_VMEMMAP. Note that check_hotplug_memory_range()
> enforces a larger memory_block_size_bytes() granularity for
> memory that will be marked online, so this check should only
> fire for direct arch_{add,remove}_memory() users outside of
> add_memory_resource()
> "
> 
> This restriction will disqualify, for example, large NVMe CMBs that might have
> non power of 2 number of pages (e.g. 32767 pages of 4KB). For these
> devices, the CMB size will be rounded down from 0x7fff000 to 0x7e00000
> but it's better than having un-mapped CMB.

Just some high-level questions:

A CMB is just a PCI BAR used for communicating with the device, to be 
mapped into physical address space, right? I assume the relevant hotplug 
code is:

drivers/pci/p2pdma.c:   addr = devm_memremap_pages(&pdev->dev, pgmap);

correct?


Having a BAR span such weird sizes will most probably never be fully 
supported. But if sub-sections work for you, great.