| Message ID | e74aee54657de6f8d643a92ba21f467db149c2c8.1528714312.git.yi.z.zhang@linux.intel.com (mailing list archive) |
|---|---|
| State | New, archived |
| Headers | show |
On Mon, Jun 11, 2018 at 9:26 AM, Stefan Hajnoczi <stefanha@redhat.com> wrote: > On Mon, Jun 11, 2018 at 06:54:25PM +0800, Zhang Yi wrote: >> Nvdimm driver use Memory hot-plug APIs to map it's pmem resource, >> which at a section granularity. >> >> When QEMU emulated the vNVDIMM device, decrease the label-storage, >> QEMU will put the vNVDIMMs directly next to one another in physical >> address space, which means that the boundary between them won't >> align to the 128 MB memory section size. > > I'm having a hard time parsing this. > > Where does the "128 MB memory section size" come from? ACPI? > A chipset-specific value? > The devm_memremap_pages() implementation use the memory hotplug core to allocate the 'struct page' array/map for persistent memory. Memory hotplug can only be performed in terms of sections, 128MB on x86_64. There is some limited support for allowing devm_memremap_pages() to overlap 'System RAM' within a given section, but it does not currently support multiple devm_memremap_pages() calls overlapping within the same section. There is currently a kernel bug where we do not handle this unsupported configuration gracefully. The fix will cause configurations configurations that try to overlap 2 persistent memory ranges in the same section to fail. The proposed fix is trying to make sure that QEMU does not run afoul of this constraint. There is currently no line of sight to reduce the minimum memory hotplug alignment size to less than 128M. Also, as other architectures outside of x86_64 add devm_memremap_pages() support, the minimum section alignment constraint might change and is a property of a guest OS. My understanding is that some guest OSes might expect an even larger persistent memory minimum alignment.
On 一, 2018-06-11 at 19:55 -0700, Dan Williams wrote: > On Mon, Jun 11, 2018 at 9:26 AM, Stefan Hajnoczi <stefanha@redhat.com > > wrote: > > > > On Mon, Jun 11, 2018 at 06:54:25PM +0800, Zhang Yi wrote: > > > > > > Nvdimm driver use Memory hot-plug APIs to map it's pmem resource, > > > which at a section granularity. > > > > > > When QEMU emulated the vNVDIMM device, decrease the label- > > > storage, > > > QEMU will put the vNVDIMMs directly next to one another in > > > physical > > > address space, which means that the boundary between them won't > > > align to the 128 MB memory section size. > > I'm having a hard time parsing this. > > > > Where does the "128 MB memory section size" come from? ACPI? > > A chipset-specific value? > > > The devm_memremap_pages() implementation use the memory hotplug core > to allocate the 'struct page' array/map for persistent memory. Memory > hotplug can only be performed in terms of sections, 128MB on x86_64. > There is some limited support for allowing devm_memremap_pages() to > overlap 'System RAM' within a given section, but it does not > currently > support multiple devm_memremap_pages() calls overlapping within the > same section. There is currently a kernel bug where we do not handle > this unsupported configuration gracefully. The fix will cause > configurations configurations that try to overlap 2 persistent memory > ranges in the same section to fail. > > The proposed fix is trying to make sure that QEMU does not run afoul > of this constraint. > > There is currently no line of sight to reduce the minimum memory > hotplug alignment size to less than 128M. Also, as other > architectures > outside of x86_64 add devm_memremap_pages() support, the minimum > section alignment constraint might change and is a property of a > guest > OS. My understanding is that some guest OSes might expect an even > larger persistent memory minimum alignment. > Thanks Dan's explanation, I still have a question that why we overlapping the un-align area instead of drop it? and let it align to the next section.
On Tue, 12 Jun 2018 23:04:25 +0800 Haozhong Zhang <hzzhan9@gmail.com> wrote: > On 06/11/18 19:55, Dan Williams wrote: > > On Mon, Jun 11, 2018 at 9:26 AM, Stefan Hajnoczi <stefanha@redhat.com> wrote: > > > On Mon, Jun 11, 2018 at 06:54:25PM +0800, Zhang Yi wrote: > > >> Nvdimm driver use Memory hot-plug APIs to map it's pmem resource, > > >> which at a section granularity. > > >> > > >> When QEMU emulated the vNVDIMM device, decrease the label-storage, > > >> QEMU will put the vNVDIMMs directly next to one another in physical > > >> address space, which means that the boundary between them won't > > >> align to the 128 MB memory section size. > > > > > > I'm having a hard time parsing this. > > > > > > Where does the "128 MB memory section size" come from? ACPI? > > > A chipset-specific value? > > > > > > > The devm_memremap_pages() implementation use the memory hotplug core > > to allocate the 'struct page' array/map for persistent memory. Memory > > hotplug can only be performed in terms of sections, 128MB on x86_64. > > IIUC, it also affects the normal RAM hotplug to a Linux VM on QEMU. If > that is the case, it will be helpful to lift this option to pc-dimm. Default alignment on page size boundary is implemented for the reason that QEMU has no idea about guest os alignments req. and these requirements might vary greatly depending on guest os running. With some guests it works just fine even with 2M alignments/dimm sizes. So it's up to upper layers which know what guest os is running to pick plugged dimm sizes. So if a particular linux version minimum block size is 128, then mgmt needs to plug dimm with size which is multiple of that. That should satisfy whatever alignment req guest os has. In case of nvdimm we need to fix address allocation in QEMU to account for label size which broke above rule leading to "overlap" over label area of nvdimm which isn't mapped into guest address space, but that's probably it. PS: not related to patch question. Intel guys contributed most of the code to nvdimm and continue actively to develop it. Can we have a designated maintainer for nvdimm part from Intel in addition to authors who just code/merge feature and disappear (not reachable) shortly after that? > Thanks, > Haozhong > > > There is some limited support for allowing devm_memremap_pages() to > > overlap 'System RAM' within a given section, but it does not currently > > support multiple devm_memremap_pages() calls overlapping within the > > same section. There is currently a kernel bug where we do not handle > > this unsupported configuration gracefully. The fix will cause > > configurations configurations that try to overlap 2 persistent memory > > ranges in the same section to fail. > > > > The proposed fix is trying to make sure that QEMU does not run afoul > > of this constraint. > > > > There is currently no line of sight to reduce the minimum memory > > hotplug alignment size to less than 128M. Also, as other architectures > > outside of x86_64 add devm_memremap_pages() support, the minimum > > section alignment constraint might change and is a property of a guest > > OS. My understanding is that some guest OSes might expect an even > > larger persistent memory minimum alignment. > >
diff --git a/hw/mem/nvdimm.c b/hw/mem/nvdimm.c index 4087aca..ff6e171 100644 --- a/hw/mem/nvdimm.c +++ b/hw/mem/nvdimm.c @@ -109,7 +109,7 @@ static void nvdimm_realize(PCDIMMDevice *dimm, Error **errp) NVDIMMDevice *nvdimm = NVDIMM(dimm); uint64_t align, pmem_size, size = memory_region_size(mr); - align = memory_region_get_alignment(mr); + align = MAX(memory_region_get_alignment(mr), NVDIMM_ALIGN_SIZE); pmem_size = size - nvdimm->label_size; nvdimm->label_data = memory_region_get_ram_ptr(mr) + pmem_size; diff --git a/include/hw/mem/nvdimm.h b/include/hw/mem/nvdimm.h index 3c82751..1d384e4 100644 --- a/include/hw/mem/nvdimm.h +++ b/include/hw/mem/nvdimm.h @@ -41,6 +41,7 @@ * at least 128KB in size, which holds around 1000 labels." */ #define MIN_NAMESPACE_LABEL_SIZE (128UL << 10) +#define NVDIMM_ALIGN_SIZE (128UL << 20) #define TYPE_NVDIMM "nvdimm" #define NVDIMM(obj) OBJECT_CHECK(NVDIMMDevice, (obj), TYPE_NVDIMM)
Nvdimm driver use Memory hot-plug APIs to map it's pmem resource, which at a section granularity. When QEMU emulated the vNVDIMM device, decrease the label-storage, QEMU will put the vNVDIMMs directly next to one another in physical address space, which means that the boundary between them won't align to the 128 MB memory section size. Signed-off-by: Zhang Yi <yi.z.zhang@linux.intel.com> --- hw/mem/nvdimm.c | 2 +- include/hw/mem/nvdimm.h | 1 + 2 files changed, 2 insertions(+), 1 deletion(-)