| Message ID | 20241016192445.3118-1-gourry@gourry.net (mailing list archive) |
|---|---|
| Headers | show |
| Series | mm/memblock,x86,acpi: hotplug memory alignment advisement | expand |
Am 16.10.24 um 21:24 schrieb Gregory Price: > When physical address regions are not aligned to memory block size, > the misaligned portion is lost (stranded capacity). > > Block size (min/max/selected) is architecture defined. Most architectures > tend to use the minimum block size or some simplistic heurist. On x86, > memory block size increases up to 2GB, and is otherwise fitted to the > alignment of non-hotplug (special purpose memory). > > CXL exposes its memory for management through the ACPI CEDT (CXL Early > Detection Table) in a field called the CXL Fixed Memory Window. Per > the CXL specification, this memory must be aligned to at least 256MB. > > When a CFMW aligns on a size less than the block size, this causes a > loss of up to 2GB per CFMW on x86. It is not uncommon for CFMW to be > allocated per-device - though this behavior is BIOS defined. > > This patch set provides 3 things: > 1) implement advise/probe functions in mm/memblock.c to report/probe > architecture agnostic hotplug memory alignment advice. > 2) update x86 memblock size logic to consider the hotplug advice > 3) add code in acpi/numa/srat.c to report CFMW alignment advice > > The advisement interfaces are design to be called during arch_init > code prior to allocator and smp_init. start_kernel will call these > through setup_arch() (via acpi and mm/init_64.c on x86), which occurs > prior to mm_core_init and smp_init - so no need for atomics. > > There's an attempt to signal callers to advise() that probe has already > occurred, but this is predicated on the notion that probe() actually > occurs (which presently only happens on x86). This is to assist debugging > future users who may mistakenly call this after allocator or smp init. > > Likewise, if probe() occurs more than once, we return -EBUSY to prevent > inconsistent values from being reported - i.e. this interaction should > happen exactly once, and all other behavior is an error / the probed > value should be acquired via memory_block_size_bytes() instead. > > Suggested-by: Ira Weiny <ira.weiny@intel.com> > Suggested-by: David Hildenbrand <david@redhat.com> > Suggested-by: Dan Williams <dan.j.williams@intel.com> > Signed-off-by: Gregory Price <gourry@gourry.net> Just as a side note, a while ago there was a discussion about variable-sized memory blocks -- essentially removing memory_block_size_bytes(). The main issue is that this would change /sys/devices/system/memory/ in ways it could break existing user space. I believe there are other corner cases that are a bit nasty to handle (e.g., removing parts of a larger memory block), but likely it could be handled.
On Mon, Oct 21, 2024 at 11:51:38AM +0200, David Hildenbrand wrote: > > > Am 16.10.24 um 21:24 schrieb Gregory Price: > > When physical address regions are not aligned to memory block size, > > the misaligned portion is lost (stranded capacity). > > > > Block size (min/max/selected) is architecture defined. Most architectures > > tend to use the minimum block size or some simplistic heurist. On x86, > > memory block size increases up to 2GB, and is otherwise fitted to the > > alignment of non-hotplug (special purpose memory). > > > > CXL exposes its memory for management through the ACPI CEDT (CXL Early > > Detection Table) in a field called the CXL Fixed Memory Window. Per > > the CXL specification, this memory must be aligned to at least 256MB. > > > > When a CFMW aligns on a size less than the block size, this causes a > > loss of up to 2GB per CFMW on x86. It is not uncommon for CFMW to be > > allocated per-device - though this behavior is BIOS defined. > > > > This patch set provides 3 things: > > 1) implement advise/probe functions in mm/memblock.c to report/probe > > architecture agnostic hotplug memory alignment advice. > > 2) update x86 memblock size logic to consider the hotplug advice > > 3) add code in acpi/numa/srat.c to report CFMW alignment advice > > > > The advisement interfaces are design to be called during arch_init > > code prior to allocator and smp_init. start_kernel will call these > > through setup_arch() (via acpi and mm/init_64.c on x86), which occurs > > prior to mm_core_init and smp_init - so no need for atomics. > > > > There's an attempt to signal callers to advise() that probe has already > > occurred, but this is predicated on the notion that probe() actually > > occurs (which presently only happens on x86). This is to assist debugging > > future users who may mistakenly call this after allocator or smp init. > > > > Likewise, if probe() occurs more than once, we return -EBUSY to prevent > > inconsistent values from being reported - i.e. this interaction should > > happen exactly once, and all other behavior is an error / the probed > > value should be acquired via memory_block_size_bytes() instead. > > > > Suggested-by: Ira Weiny <ira.weiny@intel.com> > > Suggested-by: David Hildenbrand <david@redhat.com> > > Suggested-by: Dan Williams <dan.j.williams@intel.com> > > Signed-off-by: Gregory Price <gourry@gourry.net> > > Just as a side note, a while ago there was a discussion about variable-sized > memory blocks -- essentially removing memory_block_size_bytes(). > If you have any links, happy to do some reading up on it. Was going to look into some more memblock behavior in the future so it's worth looking at. > > The main issue is that this would change /sys/devices/system/memory/ in ways > it could break existing user space. I believe there are other corner cases > that are a bit nasty to handle (e.g., removing parts of a larger memory > block), but likely it could be handled. > This is why I wanted to avoid a new interface in the first place and just piggyback on set_memory_block_size_order - now there are two interfaces to do the same thing and more hurdles. But I suppose the suggestive-nature of this one makes it far less offensive since it can be completely ignored. ~Gregory
When physical address regions are not aligned to memory block size, the misaligned portion is lost (stranded capacity). Block size (min/max/selected) is architecture defined. Most architectures tend to use the minimum block size or some simplistic heurist. On x86, memory block size increases up to 2GB, and is otherwise fitted to the alignment of non-hotplug (special purpose memory). CXL exposes its memory for management through the ACPI CEDT (CXL Early Detection Table) in a field called the CXL Fixed Memory Window. Per the CXL specification, this memory must be aligned to at least 256MB. When a CFMW aligns on a size less than the block size, this causes a loss of up to 2GB per CFMW on x86. It is not uncommon for CFMW to be allocated per-device - though this behavior is BIOS defined. This patch set provides 3 things: 1) implement advise/probe functions in mm/memblock.c to report/probe architecture agnostic hotplug memory alignment advice. 2) update x86 memblock size logic to consider the hotplug advice 3) add code in acpi/numa/srat.c to report CFMW alignment advice The advisement interfaces are design to be called during arch_init code prior to allocator and smp_init. start_kernel will call these through setup_arch() (via acpi and mm/init_64.c on x86), which occurs prior to mm_core_init and smp_init - so no need for atomics. There's an attempt to signal callers to advise() that probe has already occurred, but this is predicated on the notion that probe() actually occurs (which presently only happens on x86). This is to assist debugging future users who may mistakenly call this after allocator or smp init. Likewise, if probe() occurs more than once, we return -EBUSY to prevent inconsistent values from being reported - i.e. this interaction should happen exactly once, and all other behavior is an error / the probed value should be acquired via memory_block_size_bytes() instead. Suggested-by: Ira Weiny <ira.weiny@intel.com> Suggested-by: David Hildenbrand <david@redhat.com> Suggested-by: Dan Williams <dan.j.williams@intel.com> Signed-off-by: Gregory Price <gourry@gourry.net> Gregory Price (3): mm/memblock: implement memblock_advise_size_order and probe functions x86: probe memblock size advisement value during mm init acpi,srat: reduce memory block size if CFMWS has a smaller alignment arch/x86/mm/init_64.c | 16 +++++++++++++ drivers/acpi/numa/srat.c | 42 ++++++++++++++++++++++++++++++++++ include/linux/memblock.h | 2 ++ mm/memblock.c | 49 ++++++++++++++++++++++++++++++++++++++++ 4 files changed, 109 insertions(+)