| Message ID | 158489357825.1457606.17352509511987748598.stgit@dwillia2-desk3.amr.corp.intel.com (mailing list archive) |
|---|---|
| State | New, archived |
| Headers | show |
| Series | Manual definition of Soft Reserved memory devices | expand |
On 3/22/20 4:12 PM, Dan Williams wrote: > The hmem enabling in commit 'cf8741ac57ed ("ACPI: NUMA: HMAT: Register > "soft reserved" memory as an "hmem" device")' only registered ranges to > the hmem driver for each soft-reservation that also appeared in the > HMAT. While this is meant to encourage platform firmware to "do the > right thing" and publish an HMAT, the corollary is that platforms that > fail to publish an accurate HMAT will strand memory from Linux usage. > Additionally, the "efi_fake_mem" kernel command line option enabling > will strand memory by default without an HMAT. > > Arrange for "soft reserved" memory that goes unclaimed by HMAT entries > to be published as raw resource ranges for the hmem driver to consume. > > Include a module parameter to disable either this fallback behavior, or > the hmat enabling from creating hmem devices. The module parameter > requires the hmem device enabling to have unique name in the module > namespace: "device_hmem". > > Rather than mark this x86-only, include an interim phys_to_target_node() > implementation for arm64. > > Cc: Jonathan Cameron <Jonathan.Cameron@huawei.com> > Cc: Brice Goglin <Brice.Goglin@inria.fr> > Cc: Ard Biesheuvel <ard.biesheuvel@linaro.org> > Cc: "Rafael J. Wysocki" <rjw@rjwysocki.net> > Cc: Jeff Moyer <jmoyer@redhat.com> > Cc: Catalin Marinas <catalin.marinas@arm.com> > Cc: Will Deacon <will@kernel.org> > Signed-off-by: Dan Williams <dan.j.williams@intel.com> > --- > arch/arm64/mm/numa.c | 13 +++++++++++++ > drivers/dax/Kconfig | 1 + > drivers/dax/hmem/Makefile | 3 ++- > drivers/dax/hmem/device.c | 33 +++++++++++++++++++++++++++++++++ > 4 files changed, 49 insertions(+), 1 deletion(-) > [...] > diff --git a/drivers/dax/hmem/device.c b/drivers/dax/hmem/device.c > index 99bc15a8b031..f9c5fa8b1880 100644 > --- a/drivers/dax/hmem/device.c > +++ b/drivers/dax/hmem/device.c > @@ -4,6 +4,9 @@ > #include <linux/module.h> > #include <linux/mm.h> > > +static bool nohmem; > +module_param_named(disable, nohmem, bool, 0444); > + > void hmem_register_device(int target_nid, struct resource *r) > { > /* define a clean / non-busy resource for the platform device */ > @@ -16,6 +19,9 @@ void hmem_register_device(int target_nid, struct resource *r) > struct memregion_info info; > int rc, id; > > + if (nohmem) > + return; > + > rc = region_intersects(res.start, resource_size(&res), IORESOURCE_MEM, > IORES_DESC_SOFT_RESERVED); > if (rc != REGION_INTERSECTS) > @@ -62,3 +68,30 @@ void hmem_register_device(int target_nid, struct resource *r) > out_pdev: > memregion_free(id); > } > + > +static __init int hmem_register_one(struct resource *res, void *data) > +{ > + /* > + * If the resource is not a top-level resource it was already > + * assigned to a device by the HMAT parsing. > + */ > + if (res->parent != &iomem_resource) > + return 0; > + > + hmem_register_device(phys_to_target_node(res->start), res); > + > + return 0; Should we add an error returning value to hmem_register_device() perhaps this ought to be reflected in hmem_register_one(). > +} > + > +static __init int hmem_init(void) > +{ > + walk_iomem_res_desc(IORES_DESC_SOFT_RESERVED, > + IORESOURCE_MEM, 0, -1, NULL, hmem_register_one); > + return 0; > +} > + (...) and then perhaps here returning in the initcall if any of the resources failed hmem registration? > +/* > + * As this is a fallback for address ranges unclaimed by the ACPI HMAT > + * parsing it must be at an initcall level greater than hmat_init(). > + */ > +late_initcall(hmem_init); Regardless of the nit (which ties in to patch 4), looks good. So, FWIW: Reviewed-by: Joao Martins <joao.m.martins@oracle.com> For the hmem changes. Joao
On Tue, Mar 24, 2020 at 12:41 PM Joao Martins <joao.m.martins@oracle.com> wrote: > > On 3/22/20 4:12 PM, Dan Williams wrote: > > The hmem enabling in commit 'cf8741ac57ed ("ACPI: NUMA: HMAT: Register > > "soft reserved" memory as an "hmem" device")' only registered ranges to > > the hmem driver for each soft-reservation that also appeared in the > > HMAT. While this is meant to encourage platform firmware to "do the > > right thing" and publish an HMAT, the corollary is that platforms that > > fail to publish an accurate HMAT will strand memory from Linux usage. > > Additionally, the "efi_fake_mem" kernel command line option enabling > > will strand memory by default without an HMAT. > > > > Arrange for "soft reserved" memory that goes unclaimed by HMAT entries > > to be published as raw resource ranges for the hmem driver to consume. > > > > Include a module parameter to disable either this fallback behavior, or > > the hmat enabling from creating hmem devices. The module parameter > > requires the hmem device enabling to have unique name in the module > > namespace: "device_hmem". > > > > Rather than mark this x86-only, include an interim phys_to_target_node() > > implementation for arm64. > > > > Cc: Jonathan Cameron <Jonathan.Cameron@huawei.com> > > Cc: Brice Goglin <Brice.Goglin@inria.fr> > > Cc: Ard Biesheuvel <ard.biesheuvel@linaro.org> > > Cc: "Rafael J. Wysocki" <rjw@rjwysocki.net> > > Cc: Jeff Moyer <jmoyer@redhat.com> > > Cc: Catalin Marinas <catalin.marinas@arm.com> > > Cc: Will Deacon <will@kernel.org> > > Signed-off-by: Dan Williams <dan.j.williams@intel.com> > > --- > > arch/arm64/mm/numa.c | 13 +++++++++++++ > > drivers/dax/Kconfig | 1 + > > drivers/dax/hmem/Makefile | 3 ++- > > drivers/dax/hmem/device.c | 33 +++++++++++++++++++++++++++++++++ > > 4 files changed, 49 insertions(+), 1 deletion(-) > > > > [...] > > > diff --git a/drivers/dax/hmem/device.c b/drivers/dax/hmem/device.c > > index 99bc15a8b031..f9c5fa8b1880 100644 > > --- a/drivers/dax/hmem/device.c > > +++ b/drivers/dax/hmem/device.c > > @@ -4,6 +4,9 @@ > > #include <linux/module.h> > > #include <linux/mm.h> > > > > +static bool nohmem; > > +module_param_named(disable, nohmem, bool, 0444); > > + > > void hmem_register_device(int target_nid, struct resource *r) > > { > > /* define a clean / non-busy resource for the platform device */ > > @@ -16,6 +19,9 @@ void hmem_register_device(int target_nid, struct resource *r) > > struct memregion_info info; > > int rc, id; > > > > + if (nohmem) > > + return; > > + > > rc = region_intersects(res.start, resource_size(&res), IORESOURCE_MEM, > > IORES_DESC_SOFT_RESERVED); > > if (rc != REGION_INTERSECTS) > > @@ -62,3 +68,30 @@ void hmem_register_device(int target_nid, struct resource *r) > > out_pdev: > > memregion_free(id); > > } > > + > > +static __init int hmem_register_one(struct resource *res, void *data) > > +{ > > + /* > > + * If the resource is not a top-level resource it was already > > + * assigned to a device by the HMAT parsing. > > + */ > > + if (res->parent != &iomem_resource) > > + return 0; > > + > > + hmem_register_device(phys_to_target_node(res->start), res); > > + > > + return 0; > > Should we add an error returning value to hmem_register_device() perhaps this > ought to be reflected in hmem_register_one(). > > > +} > > + > > +static __init int hmem_init(void) > > +{ > > + walk_iomem_res_desc(IORES_DESC_SOFT_RESERVED, > > + IORESOURCE_MEM, 0, -1, NULL, hmem_register_one); > > + return 0; > > +} > > + > > (...) and then perhaps here returning in the initcall if any of the resources > failed hmem registration? Except that hmem_register_one() is a stop-gap to collect soft-reserved ranges that were not already registered, and it's not an error to find already registered devices. However, I do think it's a good idea to log registrations that failed for other reasons.
On 3/24/20 9:06 PM, Dan Williams wrote: > On Tue, Mar 24, 2020 at 12:41 PM Joao Martins <joao.m.martins@oracle.com> wrote: >> >> On 3/22/20 4:12 PM, Dan Williams wrote: >>> The hmem enabling in commit 'cf8741ac57ed ("ACPI: NUMA: HMAT: Register >>> "soft reserved" memory as an "hmem" device")' only registered ranges to >>> the hmem driver for each soft-reservation that also appeared in the >>> HMAT. While this is meant to encourage platform firmware to "do the >>> right thing" and publish an HMAT, the corollary is that platforms that >>> fail to publish an accurate HMAT will strand memory from Linux usage. >>> Additionally, the "efi_fake_mem" kernel command line option enabling >>> will strand memory by default without an HMAT. >>> >>> Arrange for "soft reserved" memory that goes unclaimed by HMAT entries >>> to be published as raw resource ranges for the hmem driver to consume. >>> >>> Include a module parameter to disable either this fallback behavior, or >>> the hmat enabling from creating hmem devices. The module parameter >>> requires the hmem device enabling to have unique name in the module >>> namespace: "device_hmem". >>> >>> Rather than mark this x86-only, include an interim phys_to_target_node() >>> implementation for arm64. >>> >>> Cc: Jonathan Cameron <Jonathan.Cameron@huawei.com> >>> Cc: Brice Goglin <Brice.Goglin@inria.fr> >>> Cc: Ard Biesheuvel <ard.biesheuvel@linaro.org> >>> Cc: "Rafael J. Wysocki" <rjw@rjwysocki.net> >>> Cc: Jeff Moyer <jmoyer@redhat.com> >>> Cc: Catalin Marinas <catalin.marinas@arm.com> >>> Cc: Will Deacon <will@kernel.org> >>> Signed-off-by: Dan Williams <dan.j.williams@intel.com> >>> --- >>> arch/arm64/mm/numa.c | 13 +++++++++++++ >>> drivers/dax/Kconfig | 1 + >>> drivers/dax/hmem/Makefile | 3 ++- >>> drivers/dax/hmem/device.c | 33 +++++++++++++++++++++++++++++++++ >>> 4 files changed, 49 insertions(+), 1 deletion(-) >>> >> >> [...] >> >>> diff --git a/drivers/dax/hmem/device.c b/drivers/dax/hmem/device.c >>> index 99bc15a8b031..f9c5fa8b1880 100644 >>> --- a/drivers/dax/hmem/device.c >>> +++ b/drivers/dax/hmem/device.c >>> @@ -4,6 +4,9 @@ >>> #include <linux/module.h> >>> #include <linux/mm.h> >>> >>> +static bool nohmem; >>> +module_param_named(disable, nohmem, bool, 0444); >>> + >>> void hmem_register_device(int target_nid, struct resource *r) >>> { >>> /* define a clean / non-busy resource for the platform device */ >>> @@ -16,6 +19,9 @@ void hmem_register_device(int target_nid, struct resource *r) >>> struct memregion_info info; >>> int rc, id; >>> >>> + if (nohmem) >>> + return; >>> + >>> rc = region_intersects(res.start, resource_size(&res), IORESOURCE_MEM, >>> IORES_DESC_SOFT_RESERVED); >>> if (rc != REGION_INTERSECTS) >>> @@ -62,3 +68,30 @@ void hmem_register_device(int target_nid, struct resource *r) >>> out_pdev: >>> memregion_free(id); >>> } >>> + >>> +static __init int hmem_register_one(struct resource *res, void *data) >>> +{ >>> + /* >>> + * If the resource is not a top-level resource it was already >>> + * assigned to a device by the HMAT parsing. >>> + */ >>> + if (res->parent != &iomem_resource) >>> + return 0; >>> + >>> + hmem_register_device(phys_to_target_node(res->start), res); >>> + >>> + return 0; >> >> Should we add an error returning value to hmem_register_device() perhaps this >> ought to be reflected in hmem_register_one(). >> >>> +} >>> + >>> +static __init int hmem_init(void) >>> +{ >>> + walk_iomem_res_desc(IORES_DESC_SOFT_RESERVED, >>> + IORESOURCE_MEM, 0, -1, NULL, hmem_register_one); >>> + return 0; >>> +} >>> + >> >> (...) and then perhaps here returning in the initcall if any of the resources >> failed hmem registration? > > Except that hmem_register_one() is a stop-gap to collect soft-reserved > ranges that were not already registered, and it's not an error to find > already registered devices. > /nods And if we were to return an error (say for hmem0 out of 4 hmem ones) before walking through all soft-reserved found resources, if would skip registration for the remaining ones. Joao
On Sun, Mar 22, 2020 at 09:12:58AM -0700, Dan Williams wrote: > The hmem enabling in commit 'cf8741ac57ed ("ACPI: NUMA: HMAT: Register > "soft reserved" memory as an "hmem" device")' only registered ranges to > the hmem driver for each soft-reservation that also appeared in the > HMAT. While this is meant to encourage platform firmware to "do the > right thing" and publish an HMAT, the corollary is that platforms that > fail to publish an accurate HMAT will strand memory from Linux usage. > Additionally, the "efi_fake_mem" kernel command line option enabling > will strand memory by default without an HMAT. > > Arrange for "soft reserved" memory that goes unclaimed by HMAT entries > to be published as raw resource ranges for the hmem driver to consume. > > Include a module parameter to disable either this fallback behavior, or > the hmat enabling from creating hmem devices. The module parameter > requires the hmem device enabling to have unique name in the module > namespace: "device_hmem". > > Rather than mark this x86-only, include an interim phys_to_target_node() > implementation for arm64. > > Cc: Jonathan Cameron <Jonathan.Cameron@huawei.com> > Cc: Brice Goglin <Brice.Goglin@inria.fr> > Cc: Ard Biesheuvel <ard.biesheuvel@linaro.org> > Cc: "Rafael J. Wysocki" <rjw@rjwysocki.net> > Cc: Jeff Moyer <jmoyer@redhat.com> > Cc: Catalin Marinas <catalin.marinas@arm.com> > Cc: Will Deacon <will@kernel.org> > Signed-off-by: Dan Williams <dan.j.williams@intel.com> > --- > arch/arm64/mm/numa.c | 13 +++++++++++++ > drivers/dax/Kconfig | 1 + > drivers/dax/hmem/Makefile | 3 ++- > drivers/dax/hmem/device.c | 33 +++++++++++++++++++++++++++++++++ > 4 files changed, 49 insertions(+), 1 deletion(-) > > diff --git a/arch/arm64/mm/numa.c b/arch/arm64/mm/numa.c > index 4decf1659700..00fba21eaec0 100644 > --- a/arch/arm64/mm/numa.c > +++ b/arch/arm64/mm/numa.c > @@ -468,3 +468,16 @@ int memory_add_physaddr_to_nid(u64 addr) > pr_warn("Unknown node for memory at 0x%llx, assuming node 0\n", addr); > return 0; > } > + > +/* > + * device-dax instance registrations want a valid target-node in case > + * they are ever onlined as memory (see hmem_register_device()). > + * > + * TODO: consult cached numa info > + */ > +int phys_to_target_node(phys_addr_t addr) > +{ > + pr_warn_once("Unknown target node for memory at 0x%llx, assuming node 0\n", > + addr); > + return 0; > +} Could you implement a generic version of this by iterating over the nodes with for_each_{,online_}node() and checking for intersection with node_{start,end}_pfn()? Will
On Wed, Mar 25, 2020 at 4:10 AM Will Deacon <will@kernel.org> wrote: > > On Sun, Mar 22, 2020 at 09:12:58AM -0700, Dan Williams wrote: > > The hmem enabling in commit 'cf8741ac57ed ("ACPI: NUMA: HMAT: Register > > "soft reserved" memory as an "hmem" device")' only registered ranges to > > the hmem driver for each soft-reservation that also appeared in the > > HMAT. While this is meant to encourage platform firmware to "do the > > right thing" and publish an HMAT, the corollary is that platforms that > > fail to publish an accurate HMAT will strand memory from Linux usage. > > Additionally, the "efi_fake_mem" kernel command line option enabling > > will strand memory by default without an HMAT. > > > > Arrange for "soft reserved" memory that goes unclaimed by HMAT entries > > to be published as raw resource ranges for the hmem driver to consume. > > > > Include a module parameter to disable either this fallback behavior, or > > the hmat enabling from creating hmem devices. The module parameter > > requires the hmem device enabling to have unique name in the module > > namespace: "device_hmem". > > > > Rather than mark this x86-only, include an interim phys_to_target_node() > > implementation for arm64. > > > > Cc: Jonathan Cameron <Jonathan.Cameron@huawei.com> > > Cc: Brice Goglin <Brice.Goglin@inria.fr> > > Cc: Ard Biesheuvel <ard.biesheuvel@linaro.org> > > Cc: "Rafael J. Wysocki" <rjw@rjwysocki.net> > > Cc: Jeff Moyer <jmoyer@redhat.com> > > Cc: Catalin Marinas <catalin.marinas@arm.com> > > Cc: Will Deacon <will@kernel.org> > > Signed-off-by: Dan Williams <dan.j.williams@intel.com> > > --- > > arch/arm64/mm/numa.c | 13 +++++++++++++ > > drivers/dax/Kconfig | 1 + > > drivers/dax/hmem/Makefile | 3 ++- > > drivers/dax/hmem/device.c | 33 +++++++++++++++++++++++++++++++++ > > 4 files changed, 49 insertions(+), 1 deletion(-) > > > > diff --git a/arch/arm64/mm/numa.c b/arch/arm64/mm/numa.c > > index 4decf1659700..00fba21eaec0 100644 > > --- a/arch/arm64/mm/numa.c > > +++ b/arch/arm64/mm/numa.c > > @@ -468,3 +468,16 @@ int memory_add_physaddr_to_nid(u64 addr) > > pr_warn("Unknown node for memory at 0x%llx, assuming node 0\n", addr); > > return 0; > > } > > + > > +/* > > + * device-dax instance registrations want a valid target-node in case > > + * they are ever onlined as memory (see hmem_register_device()). > > + * > > + * TODO: consult cached numa info > > + */ > > +int phys_to_target_node(phys_addr_t addr) > > +{ > > + pr_warn_once("Unknown target node for memory at 0x%llx, assuming node 0\n", > > + addr); > > + return 0; > > +} > > Could you implement a generic version of this by iterating over the nodes > with for_each_{,online_}node() and checking for intersection with > node_{start,end}_pfn()? Interesting. The gap is that node_{start,end}_pfn() requires node_data[] which to date architectures have only setup for online nodes. Recall a target node is an offline node that could come online later. However, reworking offline data into node_data could be the local solution for arm64, I'd just ask that each of the 6 memory-hotplug capable archs go make that opt-in decision themselves.
diff --git a/arch/arm64/mm/numa.c b/arch/arm64/mm/numa.c index 4decf1659700..00fba21eaec0 100644 --- a/arch/arm64/mm/numa.c +++ b/arch/arm64/mm/numa.c @@ -468,3 +468,16 @@ int memory_add_physaddr_to_nid(u64 addr) pr_warn("Unknown node for memory at 0x%llx, assuming node 0\n", addr); return 0; } + +/* + * device-dax instance registrations want a valid target-node in case + * they are ever onlined as memory (see hmem_register_device()). + * + * TODO: consult cached numa info + */ +int phys_to_target_node(phys_addr_t addr) +{ + pr_warn_once("Unknown target node for memory at 0x%llx, assuming node 0\n", + addr); + return 0; +} diff --git a/drivers/dax/Kconfig b/drivers/dax/Kconfig index a229f45d34aa..163edde6ba41 100644 --- a/drivers/dax/Kconfig +++ b/drivers/dax/Kconfig @@ -50,6 +50,7 @@ config DEV_DAX_HMEM config DEV_DAX_HMEM_DEVICES depends on DEV_DAX_HMEM + select NUMA_KEEP_MEMINFO if NUMA def_bool y config DEV_DAX_KMEM diff --git a/drivers/dax/hmem/Makefile b/drivers/dax/hmem/Makefile index a9d353d0c9ed..57377b4c3d47 100644 --- a/drivers/dax/hmem/Makefile +++ b/drivers/dax/hmem/Makefile @@ -1,5 +1,6 @@ # SPDX-License-Identifier: GPL-2.0 obj-$(CONFIG_DEV_DAX_HMEM) += dax_hmem.o -obj-$(CONFIG_DEV_DAX_HMEM_DEVICES) += device.o +obj-$(CONFIG_DEV_DAX_HMEM_DEVICES) += device_hmem.o +device_hmem-y := device.o dax_hmem-y := hmem.o diff --git a/drivers/dax/hmem/device.c b/drivers/dax/hmem/device.c index 99bc15a8b031..f9c5fa8b1880 100644 --- a/drivers/dax/hmem/device.c +++ b/drivers/dax/hmem/device.c @@ -4,6 +4,9 @@ #include <linux/module.h> #include <linux/mm.h> +static bool nohmem; +module_param_named(disable, nohmem, bool, 0444); + void hmem_register_device(int target_nid, struct resource *r) { /* define a clean / non-busy resource for the platform device */ @@ -16,6 +19,9 @@ void hmem_register_device(int target_nid, struct resource *r) struct memregion_info info; int rc, id; + if (nohmem) + return; + rc = region_intersects(res.start, resource_size(&res), IORESOURCE_MEM, IORES_DESC_SOFT_RESERVED); if (rc != REGION_INTERSECTS) @@ -62,3 +68,30 @@ void hmem_register_device(int target_nid, struct resource *r) out_pdev: memregion_free(id); } + +static __init int hmem_register_one(struct resource *res, void *data) +{ + /* + * If the resource is not a top-level resource it was already + * assigned to a device by the HMAT parsing. + */ + if (res->parent != &iomem_resource) + return 0; + + hmem_register_device(phys_to_target_node(res->start), res); + + return 0; +} + +static __init int hmem_init(void) +{ + walk_iomem_res_desc(IORES_DESC_SOFT_RESERVED, + IORESOURCE_MEM, 0, -1, NULL, hmem_register_one); + return 0; +} + +/* + * As this is a fallback for address ranges unclaimed by the ACPI HMAT + * parsing it must be at an initcall level greater than hmat_init(). + */ +late_initcall(hmem_init);
The hmem enabling in commit 'cf8741ac57ed ("ACPI: NUMA: HMAT: Register "soft reserved" memory as an "hmem" device")' only registered ranges to the hmem driver for each soft-reservation that also appeared in the HMAT. While this is meant to encourage platform firmware to "do the right thing" and publish an HMAT, the corollary is that platforms that fail to publish an accurate HMAT will strand memory from Linux usage. Additionally, the "efi_fake_mem" kernel command line option enabling will strand memory by default without an HMAT. Arrange for "soft reserved" memory that goes unclaimed by HMAT entries to be published as raw resource ranges for the hmem driver to consume. Include a module parameter to disable either this fallback behavior, or the hmat enabling from creating hmem devices. The module parameter requires the hmem device enabling to have unique name in the module namespace: "device_hmem". Rather than mark this x86-only, include an interim phys_to_target_node() implementation for arm64. Cc: Jonathan Cameron <Jonathan.Cameron@huawei.com> Cc: Brice Goglin <Brice.Goglin@inria.fr> Cc: Ard Biesheuvel <ard.biesheuvel@linaro.org> Cc: "Rafael J. Wysocki" <rjw@rjwysocki.net> Cc: Jeff Moyer <jmoyer@redhat.com> Cc: Catalin Marinas <catalin.marinas@arm.com> Cc: Will Deacon <will@kernel.org> Signed-off-by: Dan Williams <dan.j.williams@intel.com> --- arch/arm64/mm/numa.c | 13 +++++++++++++ drivers/dax/Kconfig | 1 + drivers/dax/hmem/Makefile | 3 ++- drivers/dax/hmem/device.c | 33 +++++++++++++++++++++++++++++++++ 4 files changed, 49 insertions(+), 1 deletion(-)