| Message ID | 20231123092343.1703707-1-sumanthk@linux.ibm.com (mailing list archive) |
|---|---|
| Headers | show |
| Series | implement "memmap on memory" feature on s390 | expand |
On 23.11.23 10:23, Sumanth Korikkar wrote: > Introduce MHP_OFFLINE_INACCESSIBLE mhp_flag to mark the hotplugged > memory block as inaccessible during the memory hotplug addition phase. > With support for "memmap on memory", the altmap is prepared at this > stage. Architectures like s390 anticipate that memmap should not be > accessed until memory is physically accessible and is accessible only > when it enters the memory hotplug onlining phase using the memory > notifier. Introduce the flag to inform the memory hotplug > infrastructure that the memory remains inaccessible until the memory > hotplug onlining phase begins. > > Implementation considerations: > mhp inaccessible flag is initially set in altmap. This is useful in > arch_add_memory(). When the memory block device is added, the mhp > inaccessible information is passed to memory_block. The flag is used in > subsequent patch to avoid accessing memmap during memory hotplug > addition phase. > > Signed-off-by: Sumanth Korikkar <sumanthk@linux.ibm.com> > --- > drivers/base/memory.c | 2 ++ > include/linux/memory.h | 1 + > include/linux/memory_hotplug.h | 10 ++++++++++ > include/linux/memremap.h | 1 + > mm/memory_hotplug.c | 3 ++- > 5 files changed, 16 insertions(+), 1 deletion(-) > > diff --git a/drivers/base/memory.c b/drivers/base/memory.c > index 8a13babd826c..51915d5c3f88 100644 > --- a/drivers/base/memory.c > +++ b/drivers/base/memory.c > @@ -774,6 +774,8 @@ static int add_memory_block(unsigned long block_id, unsigned long state, > mem->state = state; > mem->nid = NUMA_NO_NODE; > mem->altmap = altmap; > + if (altmap) > + mem->inaccessible = altmap->inaccessible; > INIT_LIST_HEAD(&mem->group_next); > > #ifndef CONFIG_NUMA > diff --git a/include/linux/memory.h b/include/linux/memory.h > index f53cfdaaaa41..655714d4e65a 100644 > --- a/include/linux/memory.h > +++ b/include/linux/memory.h > @@ -67,6 +67,7 @@ struct memory_group { > struct memory_block { > unsigned long start_section_nr; > unsigned long state; /* serialized by the dev->lock */ > + bool inaccessible; /* during memory addition phase */ Is that really required? After all, the altmap is stored in the memory block and accessible there. > int online_type; /* for passing data to online routine */ > int nid; /* NID for this memory block */ > /* > diff --git a/include/linux/memory_hotplug.h b/include/linux/memory_hotplug.h > index 7d2076583494..8988cd5ad55d 100644 > --- a/include/linux/memory_hotplug.h > +++ b/include/linux/memory_hotplug.h > @@ -106,6 +106,16 @@ typedef int __bitwise mhp_t; > * implies the node id (nid). > */ > #define MHP_NID_IS_MGID ((__force mhp_t)BIT(2)) > +/* > + * Mark the hotplugged memory block as inaccessible during the memory hotplug > + * addition phase. With support for "memmap on memory," the altmap is prepared > + * at this stage. Architectures like s390 anticipate that memmap should not be > + * accessed until memory is physically accessible and is accessible only when > + * it enters the memory hotplug onlining phase using the memory notifier. > + * Utilize this flag to inform the memory hotplug infrastructure that the > + * memory remains inaccessible until the memory hotplug onlining phase begins. > + */ > +#define MHP_OFFLINE_INACCESSIBLE ((__force mhp_t)BIT(3)) I'd suggest to squash all 3 patches. Then we can properly document here: /* * The hotplugged memory is completely inaccessible while the memory is * offline. The memory provider will handle MEM_PREPARE_ONLINE / * MEM_FINISH_OFFLINE notifications and make the memory accessible. * * This flag is only relevant when used along with MHP_MEMMAP_ON_MEMORY, * because the altmap cannot be written (e.g., poisoned) when adding * memory -- before it is set online. * * This allows for adding memory with an altmap that is not currently * made available by a hypervisor. When onlining that memory, the * hypervisor can be instructed to make that memory available, and * the onlining phase will not require any memory allocations, which is * helpful in low-memory situations. */ Cheers, David / dhildenb
Hi All, The patch series implements "memmap on memory" feature on s390. Patch 1 introduces new mhp_flag MHP_OFFLINE_INACCESSIBLE to mark memory as not accessible until memory hotplug online phase begins. Patch 2 avoids page_init_poison() on memmap during mhp addition phase, when mhp_flag MHP_OFFLINE_INACCESSIBLE is passed over from add_memory() Patch 3 introduces MEM_PREPARE_ONLINE/MEM_FINISH_OFFLINE memory notifiers to prepare the transition of memory to and from a physically accessible state. This enhancement is crucial for implementing the "memmap on memory" feature for s390 in a subsequent patch. Patches 4 allocates vmemmap pages from self-contained memory range for s390. It allocates memory map (struct pages array) from the hotplugged memory range, rather than using system memory by passing altmap to vmemmap functions. Patch 5 removes unhandled memory notifier types on s390. Patch 6 implements MEM_PREPARE_ONLINE/MEM_FINISH_OFFLINE memory notifiers on s390. MEM_PREPARE_ONLINE memory notifier makes memory block physical accessible via sclp assign command. The notifier ensures self-contained memory maps are accessible and hence enabling the "memmap on memory" on s390. MEM_FINISH_OFFLINE memory notifier shifts the memory block to an inaccessible state via sclp unassign command Patch 7 finally enables MHP_MEMMAP_ON_MEMORY on s390 These patches are rebased on top of three fixes: mm: use vmem_altmap code without CONFIG_ZONE_DEVICE mm/memory_hotplug: fix error handling in add_memory_resource() mm/memory_hotplug: add missing mem_hotplug_lock v2: * Fixes are integrated and hence removed from this patch series Suggestions from David: * Add new flag MHP_OFFLINE_INACCESSIBLE to avoid accessing memory during memory hotplug addition phase. * Avoid page_init_poison() on memmap during mhp addition phase, when MHP_OFFLINE_INACCESSIBLE mhp_flag is passed in add_memory(). * Do not skip add_pages() in arch_add_memory(). Similarly, remove similar hacks in arch_remove_memory(). * Use MHP_PREPARE_ONLINE/MHP_FINISH_OFFLINE naming convention for new memory notifiers. * Rearrange removal of unused s390 memory notifier. * Necessary commit messages changes. Thank you Sumanth Korikkar (7): mm/memory_hotplug: introduce mhp_flag MHP_OFFLINE_INACCESSIBLE mm/memory_hotplug: avoid poisoning memmap during mhp addition phase mm/memory_hotplug: introduce MEM_PREPARE_ONLINE/MEM_FINISH_OFFLINE notifiers s390/mm: allocate vmemmap pages from self-contained memory range s390/sclp: remove unhandled memory notifier type s390/mm: implement MEM_PREPARE_ONLINE/MEM_FINISH_OFFLINE notifiers s390: enable MHP_MEMMAP_ON_MEMORY arch/s390/Kconfig | 1 + arch/s390/mm/init.c | 3 -- arch/s390/mm/vmem.c | 62 +++++++++++++++++++--------------- drivers/base/memory.c | 23 +++++++++++-- drivers/s390/char/sclp_cmd.c | 31 ++++++++++++----- include/linux/memory.h | 3 ++ include/linux/memory_hotplug.h | 12 ++++++- include/linux/memremap.h | 1 + mm/memory_hotplug.c | 30 ++++++++++++++-- mm/sparse.c | 3 +- 10 files changed, 124 insertions(+), 45 deletions(-)