| Message ID | 20221013025722.48802-2-Henry.Wang@arm.com (mailing list archive) |
|---|---|
| State | New, archived |
| Headers | show |
| Series | Static heap follow-up doc changes | expand |
On Thu, 13 Oct 2022, Henry Wang wrote: > The static heap feature requires user to know the minimal size of > heap to make sure the system can work. Since the heap controlled > by Xen is intended to provide memory for the whole system, not only > the boot time memory allocation should be covered by the static > heap region, but also the runtime allocation should be covered. > > The main source of runtime allocation is the memory for the P2M. > Currently, from XSA-409, the P2M memory is bounded by the P2M pool. > So make this part as the minimal requirement of static heap. The > amount of memory allocated after all the guests have been created > should be quite limited and mostly predictable. > > This commit adds documentation that explains how a user can size the > static heap region. > > Signed-off-by: Henry Wang <Henry.Wang@arm.com> Acked-by: Stefano Stabellini <sstabellini@kernel.org> > --- > docs/misc/arm/device-tree/booting.txt | 7 +++++++ > 1 file changed, 7 insertions(+) > > diff --git a/docs/misc/arm/device-tree/booting.txt b/docs/misc/arm/device-tree/booting.txt > index 87eaa3e254..046f28ce31 100644 > --- a/docs/misc/arm/device-tree/booting.txt > +++ b/docs/misc/arm/device-tree/booting.txt > @@ -531,6 +531,13 @@ Below is an example on how to specify the static heap in device tree: > RAM starting from the host physical address 0x30000000 of 1GB size will > be reserved as static heap. > > +Users should be mindful that the static heap should at least satisfy the > +allocation of the P2M maps for all guests. Currently, the minimal requirement > +of per-domain P2M pages pool is in-sync with function > +libxl__get_required_paging_memory() (for xl-created domUs) and > +domain_p2m_pages() (for dom0less domUs), that is, 1MB per vCPU, plus 4KiB per > +MiB of RAM for the P2M map, and plus 512KiB to cover extended regions. > + > Static Shared Memory > ==================== > > -- > 2.17.1 >
Hi Henry, On 13/10/2022 03:57, Henry Wang wrote: > The static heap feature requires user to know the minimal size of > heap to make sure the system can work. Since the heap controlled > by Xen is intended to provide memory for the whole system, not only > the boot time memory allocation should be covered by the static > heap region, but also the runtime allocation should be covered. > > The main source of runtime allocation is the memory for the P2M. > Currently, from XSA-409, the P2M memory is bounded by the P2M pool. > So make this part as the minimal requirement of static heap. The > amount of memory allocated after all the guests have been created > should be quite limited and mostly predictable. > > This commit adds documentation that explains how a user can size the > static heap region. > > Signed-off-by: Henry Wang <Henry.Wang@arm.com> > --- > docs/misc/arm/device-tree/booting.txt | 7 +++++++ > 1 file changed, 7 insertions(+) > > diff --git a/docs/misc/arm/device-tree/booting.txt b/docs/misc/arm/device-tree/booting.txt > index 87eaa3e254..046f28ce31 100644 > --- a/docs/misc/arm/device-tree/booting.txt > +++ b/docs/misc/arm/device-tree/booting.txt > @@ -531,6 +531,13 @@ Below is an example on how to specify the static heap in device tree: > RAM starting from the host physical address 0x30000000 of 1GB size will > be reserved as static heap. > > +Users should be mindful that the static heap should at least satisfy the > +allocation of the P2M maps for all guests. Currently, the minimal requirement > +of per-domain P2M pages pool is in-sync with function > +libxl__get_required_paging_memory() (for xl-created domUs) and > +domain_p2m_pages() (for dom0less domUs), that is, 1MB per vCPU, plus 4KiB per > +MiB of RAM for the P2M map, and plus 512KiB to cover extended regions. I think this wording is OK if the feature is a tech preview. However, if this is security supported, we need to provide some more details about the size. In particular, this doesn't tell a user how they can find the size that would fit them. Can this be decided with a formula? If not, could a user configure his system without the reserved heap and find out the memory usage from dom0 (or a debug keys)? Cheers,
Hi Julien, > -----Original Message----- > From: Julien Grall <julien@xen.org> > Subject: Re: [PATCH for-4.17 1/2] docs: Document the minimal requirement > of static heap > > Hi Henry, > > > +Users should be mindful that the static heap should at least satisfy the > > +allocation of the P2M maps for all guests. Currently, the minimal > requirement > > +of per-domain P2M pages pool is in-sync with function > > +libxl__get_required_paging_memory() (for xl-created domUs) and > > +domain_p2m_pages() (for dom0less domUs), that is, 1MB per vCPU, plus > 4KiB per > > +MiB of RAM for the P2M map, and plus 512KiB to cover extended regions. > > I think this wording is OK if the feature is a tech preview. However, if > this is security supported, we need to provide some more details about > the size. > > In particular, this doesn't tell a user how they can find the size that > would fit them. Can this be decided with a formula? My feeling of the formula would be: My feeling of the formula would be: Total heap size needed per guest = 1MB * num_guest_vcpu + 4KB * guest_ram_size_in_mb + 512KB + the memory allocated from heap by xzalloc/xzalloc_array for various uses for example alloc_domain_struct(), d->shared_info, evtchn_bucket, etc. Is this formula somehow make sense to you? I think we need to have a rough estimation of the last part (boot time allocation) though. Kind regards, Henry > > If not, could a user configure his system without the reserved heap and > find out the memory usage from dom0 (or a debug keys)? > > Cheers, > > > -- > Julien Grall
On 14/10/2022 09:58, Henry Wang wrote: > Hi Julien, > >> -----Original Message----- >> From: Julien Grall <julien@xen.org> >> Subject: Re: [PATCH for-4.17 1/2] docs: Document the minimal requirement >> of static heap >> >> Hi Henry, >> >>> +Users should be mindful that the static heap should at least satisfy the >>> +allocation of the P2M maps for all guests. Currently, the minimal >> requirement >>> +of per-domain P2M pages pool is in-sync with function >>> +libxl__get_required_paging_memory() (for xl-created domUs) and >>> +domain_p2m_pages() (for dom0less domUs), that is, 1MB per vCPU, plus >> 4KiB per >>> +MiB of RAM for the P2M map, and plus 512KiB to cover extended regions. >> >> I think this wording is OK if the feature is a tech preview. However, if >> this is security supported, we need to provide some more details about >> the size. >> >> In particular, this doesn't tell a user how they can find the size that >> would fit them. Can this be decided with a formula? > > My feeling of the formula would be: > > My feeling of the formula would be: > > Total heap size needed per guest = 1MB * num_guest_vcpu + > 4KB * guest_ram_size_in_mb + 512KB + > the memory allocated from heap by xzalloc/xzalloc_array for various uses > for example alloc_domain_struct(), d->shared_info, evtchn_bucket, etc. There are also some pages allocated using alloc_{xen,dom}heap_pages(). We also need to take into account runtime allocation done by some hypercalls (I can't remember which one) or subsystem like OPTee. In addition to that, you also have memory for the system. E.g frametables, Xen page-tables, various driver allocations... > > Is this formula somehow make sense to you? I think we need to have a > rough estimation of the last part (boot time allocation) though. That's going to be hard. It will vary depending on your system and this could change in the future as we add more features. For instance, I expect the PCI passthrough will need some memory to keep track of all the devices. I am worry the formula will become complex. Ideally we need to have a very simple formula. If that's not possible, then we need to provide a way for the user to estimate it at runtime (like what I suggested before). Cheers,
Hi Julien, > -----Original Message----- > From: Julien Grall <julien@xen.org> > >>> +Users should be mindful that the static heap should at least satisfy the > >>> +allocation of the P2M maps for all guests. Currently, the minimal > >> requirement > >>> +of per-domain P2M pages pool is in-sync with function > >>> +libxl__get_required_paging_memory() (for xl-created domUs) and > >>> +domain_p2m_pages() (for dom0less domUs), that is, 1MB per vCPU, > plus > >> 4KiB per > >>> +MiB of RAM for the P2M map, and plus 512KiB to cover extended > regions. > >> > >> I think this wording is OK if the feature is a tech preview. However, if > >> this is security supported, we need to provide some more details about > >> the size. > >> > >> In particular, this doesn't tell a user how they can find the size that > >> would fit them. Can this be decided with a formula? > > My feeling of the formula would be: > > > > Total heap size needed per guest = 1MB * num_guest_vcpu + > > 4KB * guest_ram_size_in_mb + 512KB + > > the memory allocated from heap by xzalloc/xzalloc_array for > various uses > > for example alloc_domain_struct(), d->shared_info, evtchn_bucket, etc. > > There are also some pages allocated using alloc_{xen,dom}heap_pages(). > We also need to take into account runtime allocation done by some > hypercalls (I can't remember which one) or subsystem like OPTee. > > In addition to that, you also have memory for the system. E.g > frametables, Xen page-tables, various driver allocations... > > > > > Is this formula somehow make sense to you? I think we need to have a > > rough estimation of the last part (boot time allocation) though. > > That's going to be hard. It will vary depending on your system and this > could change in the future as we add more features. For instance, I > expect the PCI passthrough will need some memory to keep track of all > the devices. > > I am worry the formula will become complex. Ideally we need to have a > very simple formula. If that's not possible, then we need to provide a > way for the user to estimate it at runtime (like what I suggested before). I agree, I think the simple formula can only be achieved is we have an estimation of the worst case scenario of those scattered memory usages. I remember I once had a try so let me try to find the results back that time... I am also very interested in the method that you proposed to provide a mechanism for users to get the system memory allocation at runtime. But IIUC this needs some work in another series. Could you please confirm if I am understanding correctly? Or probably Xen has some mechanisms that I am likely unaware? Thanks! Kind regards, Henry > > Cheers, > > -- > Julien Grall
On 14/10/2022 10:31, Henry Wang wrote: > Hi Julien, Hi, > >> -----Original Message----- >> From: Julien Grall <julien@xen.org> >>>>> +Users should be mindful that the static heap should at least satisfy the >>>>> +allocation of the P2M maps for all guests. Currently, the minimal >>>> requirement >>>>> +of per-domain P2M pages pool is in-sync with function >>>>> +libxl__get_required_paging_memory() (for xl-created domUs) and >>>>> +domain_p2m_pages() (for dom0less domUs), that is, 1MB per vCPU, >> plus >>>> 4KiB per >>>>> +MiB of RAM for the P2M map, and plus 512KiB to cover extended >> regions. >>>> >>>> I think this wording is OK if the feature is a tech preview. However, if >>>> this is security supported, we need to provide some more details about >>>> the size. >>>> >>>> In particular, this doesn't tell a user how they can find the size that >>>> would fit them. Can this be decided with a formula? >>> My feeling of the formula would be: >>> >>> Total heap size needed per guest = 1MB * num_guest_vcpu + >>> 4KB * guest_ram_size_in_mb + 512KB + >>> the memory allocated from heap by xzalloc/xzalloc_array for >> various uses >>> for example alloc_domain_struct(), d->shared_info, evtchn_bucket, etc. >> >> There are also some pages allocated using alloc_{xen,dom}heap_pages(). >> We also need to take into account runtime allocation done by some >> hypercalls (I can't remember which one) or subsystem like OPTee. >> >> In addition to that, you also have memory for the system. E.g >> frametables, Xen page-tables, various driver allocations... >> >>> >>> Is this formula somehow make sense to you? I think we need to have a >>> rough estimation of the last part (boot time allocation) though. >> >> That's going to be hard. It will vary depending on your system and this >> could change in the future as we add more features. For instance, I >> expect the PCI passthrough will need some memory to keep track of all >> the devices. >> >> I am worry the formula will become complex. Ideally we need to have a >> very simple formula. If that's not possible, then we need to provide a >> way for the user to estimate it at runtime (like what I suggested before). > > I agree, I think the simple formula can only be achieved is we have an > estimation of the worst case scenario of those scattered memory usages. > I remember I once had a try so let me try to find the results back that time... > > I am also very interested in the method that you proposed to provide a > mechanism for users to get the system memory allocation at runtime. But > IIUC this needs some work in another series. Could you please confirm if I > am understanding correctly? Or probably Xen has some mechanisms that > I am likely unaware? Thanks! It will depend the way you account memory statically allocated to domains in Xen. We already provide the total amount of memory in the system and how much is free. The values can be retrieved using ``xl info``. * When not allocated, is this considered free or used? * Are they included in the total memory? If the answer is no for both (possibly just one), then we will need to provide extra hypercalls to expose the size of the xenheap and how much is free. Cheers,
Hi Julien, > -----Original Message----- > From: Julien Grall <julien@xen.org> > Subject: Re: [PATCH for-4.17 1/2] docs: Document the minimal requirement > of static heap > >>> Is this formula somehow make sense to you? I think we need to have a > >>> rough estimation of the last part (boot time allocation) though. > >> > >> That's going to be hard. It will vary depending on your system and this > >> could change in the future as we add more features. For instance, I > >> expect the PCI passthrough will need some memory to keep track of all > >> the devices. > >> > >> I am worry the formula will become complex. Ideally we need to have a > >> very simple formula. If that's not possible, then we need to provide a > >> way for the user to estimate it at runtime (like what I suggested before). > > > > I agree, I think the simple formula can only be achieved is we have an > > estimation of the worst case scenario of those scattered memory usages. > > I remember I once had a try so let me try to find the results back that time... > > > > I am also very interested in the method that you proposed to provide a > > mechanism for users to get the system memory allocation at runtime. But > > IIUC this needs some work in another series. Could you please confirm if I > > am understanding correctly? Or probably Xen has some mechanisms that > > I am likely unaware? Thanks! > > It will depend the way you account memory statically allocated to > domains in Xen. > > We already provide the total amount of memory in the system and how > much > is free. The values can be retrieved using ``xl info``. > > * When not allocated, is this considered free or used? > * Are they included in the total memory? > > If the answer is no for both (possibly just one), then we will need to > provide extra hypercalls to expose the size of the xenheap and how much > is free. Thanks for sharing your thoughts. I will add a follow-up series for this work. Before we have this mechanism, I think the patch 2/2 should be dropped. Kind regards, Henry > > Cheers, > > -- > Julien Grall
diff --git a/docs/misc/arm/device-tree/booting.txt b/docs/misc/arm/device-tree/booting.txt index 87eaa3e254..046f28ce31 100644 --- a/docs/misc/arm/device-tree/booting.txt +++ b/docs/misc/arm/device-tree/booting.txt @@ -531,6 +531,13 @@ Below is an example on how to specify the static heap in device tree: RAM starting from the host physical address 0x30000000 of 1GB size will be reserved as static heap. +Users should be mindful that the static heap should at least satisfy the +allocation of the P2M maps for all guests. Currently, the minimal requirement +of per-domain P2M pages pool is in-sync with function +libxl__get_required_paging_memory() (for xl-created domUs) and +domain_p2m_pages() (for dom0less domUs), that is, 1MB per vCPU, plus 4KiB per +MiB of RAM for the P2M map, and plus 512KiB to cover extended regions. + Static Shared Memory ====================
The static heap feature requires user to know the minimal size of heap to make sure the system can work. Since the heap controlled by Xen is intended to provide memory for the whole system, not only the boot time memory allocation should be covered by the static heap region, but also the runtime allocation should be covered. The main source of runtime allocation is the memory for the P2M. Currently, from XSA-409, the P2M memory is bounded by the P2M pool. So make this part as the minimal requirement of static heap. The amount of memory allocated after all the guests have been created should be quite limited and mostly predictable. This commit adds documentation that explains how a user can size the static heap region. Signed-off-by: Henry Wang <Henry.Wang@arm.com> --- docs/misc/arm/device-tree/booting.txt | 7 +++++++ 1 file changed, 7 insertions(+)