| Message ID | 20200604080120.2628-1-miles.chen@mediatek.com (mailing list archive) |
|---|---|
| State | New, archived |
| Headers | show |
| Series | iommu/mediatek: Use totalram_pages to setup enable_4GB | expand |
On 04.06.20 10:01, Miles Chen wrote: > To build this driver as a kernel module, we cannot use > the unexported symbol "max_pfn" to setup enable_4GB. > > Use totalram_pages() instead to setup enable_4GB. > > Suggested-by: Mike Rapoport <rppt@linux.ibm.com> > Signed-off-by: Miles Chen <miles.chen@mediatek.com> > Cc: David Hildenbrand <david@redhat.com> > Cc: Yong Wu <yong.wu@mediatek.com> > Cc: Chao Hao <chao.hao@mediatek.com> > --- > drivers/iommu/mtk_iommu.c | 5 ++--- > 1 file changed, 2 insertions(+), 3 deletions(-) > > diff --git a/drivers/iommu/mtk_iommu.c b/drivers/iommu/mtk_iommu.c > index 5f4d6df59cf6..c2798a6e0e38 100644 > --- a/drivers/iommu/mtk_iommu.c > +++ b/drivers/iommu/mtk_iommu.c > @@ -3,7 +3,6 @@ > * Copyright (c) 2015-2016 MediaTek Inc. > * Author: Yong Wu <yong.wu@mediatek.com> > */ > -#include <linux/memblock.h> > #include <linux/bug.h> > #include <linux/clk.h> > #include <linux/component.h> > @@ -626,8 +625,8 @@ static int mtk_iommu_probe(struct platform_device *pdev) > return -ENOMEM; > data->protect_base = ALIGN(virt_to_phys(protect), MTK_PROTECT_PA_ALIGN); > > - /* Whether the current dram is over 4GB */ > - data->enable_4GB = !!(max_pfn > (BIT_ULL(32) >> PAGE_SHIFT)); > + /* Whether the current dram is over 4GB, note: DRAM start at 1GB */ > + data->enable_4GB = !!(totalram_pages() > ((SZ_2G + SZ_1G) >> PAGE_SHIFT)); A similar thing seems to be done by drivers/media/platform/mtk-vpu/mtk_vpu.c: vpu->enable_4GB = !!(totalram_pages() > (SZ_2G >> PAGE_SHIFT)); I do wonder if some weird memory hotplug setups might give you false negatives. E.g., start a VM with 1GB and hotplug 1GB - it will be hotplugged on x86-64 above 4GB, turning max_pfn into 5GB. totalram_pages() should return something < 2GB. Same can happen when you have a VM and use ballooning to fake-unplug memory, making totalram_pages() return something < 4GB, but leaving usable pfns >= 4GB. but ... I don't know if I understood what "enable_4GB" needs/implies ... I don't know if this is applicable to VMs at all (on real HW such memory hotplug setups should not exist) ... I don't know how this code would react to memory hotplug, so if the condition changes after the driver loaded and enable_4GB would suddenly apply. Again, most probably not relevant on real HW, only for VMs.
On Thu, 2020-06-04 at 10:25 +0200, David Hildenbrand wrote: > On 04.06.20 10:01, Miles Chen wrote: > > To build this driver as a kernel module, we cannot use > > the unexported symbol "max_pfn" to setup enable_4GB. > > > > Use totalram_pages() instead to setup enable_4GB. > > > > Suggested-by: Mike Rapoport <rppt@linux.ibm.com> > > Signed-off-by: Miles Chen <miles.chen@mediatek.com> > > Cc: David Hildenbrand <david@redhat.com> > > Cc: Yong Wu <yong.wu@mediatek.com> > > Cc: Chao Hao <chao.hao@mediatek.com> > > --- > > drivers/iommu/mtk_iommu.c | 5 ++--- > > 1 file changed, 2 insertions(+), 3 deletions(-) > > > > diff --git a/drivers/iommu/mtk_iommu.c b/drivers/iommu/mtk_iommu.c > > index 5f4d6df59cf6..c2798a6e0e38 100644 > > --- a/drivers/iommu/mtk_iommu.c > > +++ b/drivers/iommu/mtk_iommu.c > > @@ -3,7 +3,6 @@ > > * Copyright (c) 2015-2016 MediaTek Inc. > > * Author: Yong Wu <yong.wu@mediatek.com> > > */ > > -#include <linux/memblock.h> > > #include <linux/bug.h> > > #include <linux/clk.h> > > #include <linux/component.h> > > @@ -626,8 +625,8 @@ static int mtk_iommu_probe(struct platform_device *pdev) > > return -ENOMEM; > > data->protect_base = ALIGN(virt_to_phys(protect), MTK_PROTECT_PA_ALIGN); > > > > - /* Whether the current dram is over 4GB */ > > - data->enable_4GB = !!(max_pfn > (BIT_ULL(32) >> PAGE_SHIFT)); > > + /* Whether the current dram is over 4GB, note: DRAM start at 1GB */ > > + data->enable_4GB = !!(totalram_pages() > ((SZ_2G + SZ_1G) >> PAGE_SHIFT)); > > A similar thing seems to be done by > drivers/media/platform/mtk-vpu/mtk_vpu.c: > vpu->enable_4GB = !!(totalram_pages() > (SZ_2G >> PAGE_SHIFT)); > > I do wonder if some weird memory hotplug setups might give you false > negatives. > > E.g., start a VM with 1GB and hotplug 1GB - it will be hotplugged on > x86-64 above 4GB, turning max_pfn into 5GB. totalram_pages() should > return something < 2GB. > > Same can happen when you have a VM and use ballooning to fake-unplug > memory, making totalram_pages() return something < 4GB, but leaving > usable pfns >= 4GB Yes. Yingjoe also told me that this patch is not correct. Thanks for pointing this out. totalram_pages() does not work for some cases: e.g., DRAM start @0x4000_0000 and DRAM size is 0x1_0000_0000 but we reserve large amount of memory, which makes totalram_pages() < 3GB but it is possible to allocate a pfn >= 4GB. I will discuss this internally. Miles > . > > but > ... I don't know if I understood what "enable_4GB" needs/implies > ... I don't know if this is applicable to VMs > at all (on real HW such > memory hotplug setups should not exist) > ... I don't know how this code would react to memory hotplug, so if the > condition changes after the driver loaded and enable_4GB would > suddenly apply. Again, most probably not relevant on real HW, only > for VMs. >
On 04.06.20 11:49, Miles Chen wrote: > On Thu, 2020-06-04 at 10:25 +0200, David Hildenbrand wrote: >> On 04.06.20 10:01, Miles Chen wrote: >>> To build this driver as a kernel module, we cannot use >>> the unexported symbol "max_pfn" to setup enable_4GB. >>> >>> Use totalram_pages() instead to setup enable_4GB. >>> >>> Suggested-by: Mike Rapoport <rppt@linux.ibm.com> >>> Signed-off-by: Miles Chen <miles.chen@mediatek.com> >>> Cc: David Hildenbrand <david@redhat.com> >>> Cc: Yong Wu <yong.wu@mediatek.com> >>> Cc: Chao Hao <chao.hao@mediatek.com> >>> --- >>> drivers/iommu/mtk_iommu.c | 5 ++--- >>> 1 file changed, 2 insertions(+), 3 deletions(-) >>> >>> diff --git a/drivers/iommu/mtk_iommu.c b/drivers/iommu/mtk_iommu.c >>> index 5f4d6df59cf6..c2798a6e0e38 100644 >>> --- a/drivers/iommu/mtk_iommu.c >>> +++ b/drivers/iommu/mtk_iommu.c >>> @@ -3,7 +3,6 @@ >>> * Copyright (c) 2015-2016 MediaTek Inc. >>> * Author: Yong Wu <yong.wu@mediatek.com> >>> */ >>> -#include <linux/memblock.h> >>> #include <linux/bug.h> >>> #include <linux/clk.h> >>> #include <linux/component.h> >>> @@ -626,8 +625,8 @@ static int mtk_iommu_probe(struct platform_device *pdev) >>> return -ENOMEM; >>> data->protect_base = ALIGN(virt_to_phys(protect), MTK_PROTECT_PA_ALIGN); >>> >>> - /* Whether the current dram is over 4GB */ >>> - data->enable_4GB = !!(max_pfn > (BIT_ULL(32) >> PAGE_SHIFT)); >>> + /* Whether the current dram is over 4GB, note: DRAM start at 1GB */ >>> + data->enable_4GB = !!(totalram_pages() > ((SZ_2G + SZ_1G) >> PAGE_SHIFT)); >> >> A similar thing seems to be done by >> drivers/media/platform/mtk-vpu/mtk_vpu.c: >> vpu->enable_4GB = !!(totalram_pages() > (SZ_2G >> PAGE_SHIFT)); >> >> I do wonder if some weird memory hotplug setups might give you false >> negatives. >> >> E.g., start a VM with 1GB and hotplug 1GB - it will be hotplugged on >> x86-64 above 4GB, turning max_pfn into 5GB. totalram_pages() should >> return something < 2GB. >> >> Same can happen when you have a VM and use ballooning to fake-unplug >> memory, making totalram_pages() return something < 4GB, but leaving >> usable pfns >= 4GB > > Yes. Yingjoe also told me that this patch is not correct. > > Thanks for pointing this out. totalram_pages() does not work > for some cases: > Just a thought: If memory hotplug is applicable as well, you might either want to always assume data->enable_4GB, or handle memory hotplug events from the memory notifier, when new memory gets onlined (not sure how tricky that is).
On Thu, Jun 04, 2020 at 01:32:40PM +0200, David Hildenbrand wrote: > Just a thought: If memory hotplug is applicable as well, you might > either want to always assume data->enable_4GB, or handle memory hotplug > events from the memory notifier, when new memory gets onlined (not sure > how tricky that is). We probably want a highest_pfn_possible() or similar API instead of having drivers poking into random VM internals.
On 04.06.20 17:06, Christoph Hellwig wrote: > On Thu, Jun 04, 2020 at 01:32:40PM +0200, David Hildenbrand wrote: >> Just a thought: If memory hotplug is applicable as well, you might >> either want to always assume data->enable_4GB, or handle memory hotplug >> events from the memory notifier, when new memory gets onlined (not sure >> how tricky that is). > > We probably want a highest_pfn_possible() or similar API instead of > having drivers poking into random VM internals. Well, memory notifiers are a reasonable api used accross the kernel to get notified when new memory is onlined to the buddy that could be used for allocations. highest_pfn_possible() would have to default to something linked to MAX_PHYSMEM_BITS whenever memory hotplug is configured, I am not sure how helpful that is (IOW, you can just default to enable_4GB=true in that case instead in most cases).
On 04.06.20 17:27, David Hildenbrand wrote: > On 04.06.20 17:06, Christoph Hellwig wrote: >> On Thu, Jun 04, 2020 at 01:32:40PM +0200, David Hildenbrand wrote: >>> Just a thought: If memory hotplug is applicable as well, you might >>> either want to always assume data->enable_4GB, or handle memory hotplug >>> events from the memory notifier, when new memory gets onlined (not sure >>> how tricky that is). >> >> We probably want a highest_pfn_possible() or similar API instead of >> having drivers poking into random VM internals. > > Well, memory notifiers are a reasonable api used accross the kernel to > get notified when new memory is onlined to the buddy that could be used > for allocations. > > highest_pfn_possible() would have to default to something linked to > MAX_PHYSMEM_BITS whenever memory hotplug is configured, I am not sure > how helpful that is (IOW, you can just default to enable_4GB=true in > that case instead in most cases). Correction: At least on x86-64 we have max_possible_pfn, which will consult the ACPI SRAT table to figure out the maximum possible PFN. (Without SRAT, max_possible_pfn will point at the end of initial boot memory and not consider hotplug memory - something that e.g., newer QEMU versions work around by creating SRAT tables if memory hotplug might be possible, even if there is no actual NUMA configuration). pci-swiotlb.c similarly relies on that to figure out if there are any !DMA addresses to handle.
diff --git a/drivers/iommu/mtk_iommu.c b/drivers/iommu/mtk_iommu.c index 5f4d6df59cf6..c2798a6e0e38 100644 --- a/drivers/iommu/mtk_iommu.c +++ b/drivers/iommu/mtk_iommu.c @@ -3,7 +3,6 @@ * Copyright (c) 2015-2016 MediaTek Inc. * Author: Yong Wu <yong.wu@mediatek.com> */ -#include <linux/memblock.h> #include <linux/bug.h> #include <linux/clk.h> #include <linux/component.h> @@ -626,8 +625,8 @@ static int mtk_iommu_probe(struct platform_device *pdev) return -ENOMEM; data->protect_base = ALIGN(virt_to_phys(protect), MTK_PROTECT_PA_ALIGN); - /* Whether the current dram is over 4GB */ - data->enable_4GB = !!(max_pfn > (BIT_ULL(32) >> PAGE_SHIFT)); + /* Whether the current dram is over 4GB, note: DRAM start at 1GB */ + data->enable_4GB = !!(totalram_pages() > ((SZ_2G + SZ_1G) >> PAGE_SHIFT)); if (!data->plat_data->has_4gb_mode) data->enable_4GB = false;
To build this driver as a kernel module, we cannot use the unexported symbol "max_pfn" to setup enable_4GB. Use totalram_pages() instead to setup enable_4GB. Suggested-by: Mike Rapoport <rppt@linux.ibm.com> Signed-off-by: Miles Chen <miles.chen@mediatek.com> Cc: David Hildenbrand <david@redhat.com> Cc: Yong Wu <yong.wu@mediatek.com> Cc: Chao Hao <chao.hao@mediatek.com> --- drivers/iommu/mtk_iommu.c | 5 ++--- 1 file changed, 2 insertions(+), 3 deletions(-)