| Message ID | 20210518052113.725808-9-penny.zheng@arm.com (mailing list archive) |
|---|---|
| State | Superseded |
| Headers | show |
| Series | Domain on Static Allocation | expand |
On 18.05.2021 07:21, Penny Zheng wrote: > Since page_list under struct domain refers to linked pages as gueast RAM > allocated from heap, it should not include reserved pages of static memory. > > The number of PGC_reserved pages assigned to a domain is tracked in > a new 'reserved_pages' counter. Also introduce a new reserved_page_list > to link pages of static memory. Let page_to_list return reserved_page_list, > when flag is PGC_reserved. > > Later, when domain get destroyed or restarted, those new values will help > relinquish memory to proper place, not been given back to heap. > > Signed-off-by: Penny Zheng <penny.zheng@arm.com> > --- > xen/common/domain.c | 1 + > xen/common/page_alloc.c | 7 +++++-- > xen/include/xen/sched.h | 5 +++++ > 3 files changed, 11 insertions(+), 2 deletions(-) This contradicts the title's prefix: There's no Arm-specific change here at all. But imo the title is correct, and the changes should be Arm-specific. There's no point having struct domain fields on e.g. x86 which aren't used there at all. > --- a/xen/common/page_alloc.c > +++ b/xen/common/page_alloc.c > @@ -2410,7 +2410,7 @@ int assign_pages( > > for ( i = 0; i < nr_pfns; i++ ) > { > - ASSERT(!(pg[i].count_info & ~PGC_extra)); > + ASSERT(!(pg[i].count_info & ~(PGC_extra | PGC_reserved))); > if ( pg[i].count_info & PGC_extra ) > extra_pages++; > } > @@ -2439,6 +2439,9 @@ int assign_pages( > } > } > > + if ( pg[0].count_info & PGC_reserved ) > + d->reserved_pages += nr_pfns; I guess this again will fail to build on x86. > @@ -588,6 +590,9 @@ static inline struct page_list_head *page_to_list( > if ( pg->count_info & PGC_extra ) > return &d->extra_page_list; > > + if ( pg->count_info & PGC_reserved ) > + return &d->reserved_page_list; Same here. Jan
Hi Jan > -----Original Message----- > From: Jan Beulich <jbeulich@suse.com> > Sent: Tuesday, May 18, 2021 3:39 PM > To: Penny Zheng <Penny.Zheng@arm.com> > Cc: Bertrand Marquis <Bertrand.Marquis@arm.com>; Wei Chen > <Wei.Chen@arm.com>; nd <nd@arm.com>; xen-devel@lists.xenproject.org; > sstabellini@kernel.org; julien@xen.org > Subject: Re: [PATCH 08/10] xen/arm: introduce reserved_page_list > > On 18.05.2021 07:21, Penny Zheng wrote: > > Since page_list under struct domain refers to linked pages as gueast > > RAM allocated from heap, it should not include reserved pages of static > memory. > > > > The number of PGC_reserved pages assigned to a domain is tracked in a > > new 'reserved_pages' counter. Also introduce a new reserved_page_list > > to link pages of static memory. Let page_to_list return > > reserved_page_list, when flag is PGC_reserved. > > > > Later, when domain get destroyed or restarted, those new values will > > help relinquish memory to proper place, not been given back to heap. > > > > Signed-off-by: Penny Zheng <penny.zheng@arm.com> > > --- > > xen/common/domain.c | 1 + > > xen/common/page_alloc.c | 7 +++++-- > > xen/include/xen/sched.h | 5 +++++ > > 3 files changed, 11 insertions(+), 2 deletions(-) > > This contradicts the title's prefix: There's no Arm-specific change here at all. > But imo the title is correct, and the changes should be Arm-specific. There's > no point having struct domain fields on e.g. x86 which aren't used there at all. > Yep, you're right. I'll add ifdefs in the following changes. > > --- a/xen/common/page_alloc.c > > +++ b/xen/common/page_alloc.c > > @@ -2410,7 +2410,7 @@ int assign_pages( > > > > for ( i = 0; i < nr_pfns; i++ ) > > { > > - ASSERT(!(pg[i].count_info & ~PGC_extra)); > > + ASSERT(!(pg[i].count_info & ~(PGC_extra | > > + PGC_reserved))); > > if ( pg[i].count_info & PGC_extra ) > > extra_pages++; > > } > > @@ -2439,6 +2439,9 @@ int assign_pages( > > } > > } > > > > + if ( pg[0].count_info & PGC_reserved ) > > + d->reserved_pages += nr_pfns; > > I guess this again will fail to build on x86. > > > @@ -588,6 +590,9 @@ static inline struct page_list_head *page_to_list( > > if ( pg->count_info & PGC_extra ) > > return &d->extra_page_list; > > > > + if ( pg->count_info & PGC_reserved ) > > + return &d->reserved_page_list; > > Same here. > > Jan Thanks Penny
Hi Penny, On 18/05/2021 06:21, Penny Zheng wrote: > Since page_list under struct domain refers to linked pages as gueast RAM s/gueast/guest/ > allocated from heap, it should not include reserved pages of static memory. You already have PGC_reserved to indicate they are "static memory". So why do you need yet another list? > > The number of PGC_reserved pages assigned to a domain is tracked in > a new 'reserved_pages' counter. Also introduce a new reserved_page_list > to link pages of static memory. Let page_to_list return reserved_page_list, > when flag is PGC_reserved. > > Later, when domain get destroyed or restarted, those new values will help > relinquish memory to proper place, not been given back to heap. > > Signed-off-by: Penny Zheng <penny.zheng@arm.com> > --- > xen/common/domain.c | 1 + > xen/common/page_alloc.c | 7 +++++-- > xen/include/xen/sched.h | 5 +++++ > 3 files changed, 11 insertions(+), 2 deletions(-) > > diff --git a/xen/common/domain.c b/xen/common/domain.c > index 6b71c6d6a9..c38afd2969 100644 > --- a/xen/common/domain.c > +++ b/xen/common/domain.c > @@ -578,6 +578,7 @@ struct domain *domain_create(domid_t domid, > INIT_PAGE_LIST_HEAD(&d->page_list); > INIT_PAGE_LIST_HEAD(&d->extra_page_list); > INIT_PAGE_LIST_HEAD(&d->xenpage_list); > + INIT_PAGE_LIST_HEAD(&d->reserved_page_list); > > spin_lock_init(&d->node_affinity_lock); > d->node_affinity = NODE_MASK_ALL; > diff --git a/xen/common/page_alloc.c b/xen/common/page_alloc.c > index f1f1296a61..e3f07ec6c5 100644 > --- a/xen/common/page_alloc.c > +++ b/xen/common/page_alloc.c > @@ -2410,7 +2410,7 @@ int assign_pages( > > for ( i = 0; i < nr_pfns; i++ ) > { > - ASSERT(!(pg[i].count_info & ~PGC_extra)); > + ASSERT(!(pg[i].count_info & ~(PGC_extra | PGC_reserved))); I think this change belongs to the previous patch. > if ( pg[i].count_info & PGC_extra ) > extra_pages++; > } > @@ -2439,6 +2439,9 @@ int assign_pages( > } > } > > + if ( pg[0].count_info & PGC_reserved ) > + d->reserved_pages += nr_pfns; reserved_pages doesn't seem to be ever read or decremented. So why do you need it? > + > if ( !(memflags & MEMF_no_refcount) && > unlikely(domain_adjust_tot_pages(d, nr_pfns) == nr_pfns) ) > get_knownalive_domain(d); > @@ -2452,7 +2455,7 @@ int assign_pages( > page_set_reserved_owner(&pg[i], d); > smp_wmb(); /* Domain pointer must be visible before updating refcnt. */ > pg[i].count_info = > - (pg[i].count_info & PGC_extra) | PGC_allocated | 1; > + (pg[i].count_info & (PGC_extra | PGC_reserved)) | PGC_allocated | 1; Same here. > page_list_add_tail(&pg[i], page_to_list(d, &pg[i])); > } > > diff --git a/xen/include/xen/sched.h b/xen/include/xen/sched.h > index 3982167144..b6333ed8bb 100644 > --- a/xen/include/xen/sched.h > +++ b/xen/include/xen/sched.h > @@ -368,6 +368,7 @@ struct domain > struct page_list_head page_list; /* linked list */ > struct page_list_head extra_page_list; /* linked list (size extra_pages) */ > struct page_list_head xenpage_list; /* linked list (size xenheap_pages) */ > + struct page_list_head reserved_page_list; /* linked list (size reserved pages) */ > > /* > * This field should only be directly accessed by domain_adjust_tot_pages() > @@ -379,6 +380,7 @@ struct domain > unsigned int outstanding_pages; /* pages claimed but not possessed */ > unsigned int max_pages; /* maximum value for domain_tot_pages() */ > unsigned int extra_pages; /* pages not included in domain_tot_pages() */ > + unsigned int reserved_pages; /* pages of static memory */ > atomic_t shr_pages; /* shared pages */ > atomic_t paged_pages; /* paged-out pages */ > > @@ -588,6 +590,9 @@ static inline struct page_list_head *page_to_list( > if ( pg->count_info & PGC_extra ) > return &d->extra_page_list; > > + if ( pg->count_info & PGC_reserved ) > + return &d->reserved_page_list; > + > return &d->page_list; > } > > Cheers,
On 18.05.2021 10:38, Penny Zheng wrote: > Hi Jan > >> -----Original Message----- >> From: Jan Beulich <jbeulich@suse.com> >> Sent: Tuesday, May 18, 2021 3:39 PM >> To: Penny Zheng <Penny.Zheng@arm.com> >> Cc: Bertrand Marquis <Bertrand.Marquis@arm.com>; Wei Chen >> <Wei.Chen@arm.com>; nd <nd@arm.com>; xen-devel@lists.xenproject.org; >> sstabellini@kernel.org; julien@xen.org >> Subject: Re: [PATCH 08/10] xen/arm: introduce reserved_page_list >> >> On 18.05.2021 07:21, Penny Zheng wrote: >>> Since page_list under struct domain refers to linked pages as gueast >>> RAM allocated from heap, it should not include reserved pages of static >> memory. >>> >>> The number of PGC_reserved pages assigned to a domain is tracked in a >>> new 'reserved_pages' counter. Also introduce a new reserved_page_list >>> to link pages of static memory. Let page_to_list return >>> reserved_page_list, when flag is PGC_reserved. >>> >>> Later, when domain get destroyed or restarted, those new values will >>> help relinquish memory to proper place, not been given back to heap. >>> >>> Signed-off-by: Penny Zheng <penny.zheng@arm.com> >>> --- >>> xen/common/domain.c | 1 + >>> xen/common/page_alloc.c | 7 +++++-- >>> xen/include/xen/sched.h | 5 +++++ >>> 3 files changed, 11 insertions(+), 2 deletions(-) >> >> This contradicts the title's prefix: There's no Arm-specific change here at all. >> But imo the title is correct, and the changes should be Arm-specific. There's >> no point having struct domain fields on e.g. x86 which aren't used there at all. >> > > Yep, you're right. > I'll add ifdefs in the following changes. As necessary, please. Moving stuff to Arm-specific files would be preferable. Jan
Hi Julien > -----Original Message----- > From: Julien Grall <julien@xen.org> > Sent: Tuesday, May 18, 2021 7:02 PM > To: Penny Zheng <Penny.Zheng@arm.com>; xen-devel@lists.xenproject.org; > sstabellini@kernel.org > Cc: Bertrand Marquis <Bertrand.Marquis@arm.com>; Wei Chen > <Wei.Chen@arm.com>; nd <nd@arm.com> > Subject: Re: [PATCH 08/10] xen/arm: introduce reserved_page_list > > Hi Penny, > > On 18/05/2021 06:21, Penny Zheng wrote: > > Since page_list under struct domain refers to linked pages as gueast > > RAM > > s/gueast/guest/ > Thx~ > > allocated from heap, it should not include reserved pages of static memory. > > You already have PGC_reserved to indicate they are "static memory". So why > do you need yet another list? > > > > > The number of PGC_reserved pages assigned to a domain is tracked in a > > new 'reserved_pages' counter. Also introduce a new reserved_page_list > > to link pages of static memory. Let page_to_list return > > reserved_page_list, when flag is PGC_reserved. > > > > Later, when domain get destroyed or restarted, those new values will > > help relinquish memory to proper place, not been given back to heap. > > > > Signed-off-by: Penny Zheng <penny.zheng@arm.com> > > --- > > xen/common/domain.c | 1 + > > xen/common/page_alloc.c | 7 +++++-- > > xen/include/xen/sched.h | 5 +++++ > > 3 files changed, 11 insertions(+), 2 deletions(-) > > > > diff --git a/xen/common/domain.c b/xen/common/domain.c index > > 6b71c6d6a9..c38afd2969 100644 > > --- a/xen/common/domain.c > > +++ b/xen/common/domain.c > > @@ -578,6 +578,7 @@ struct domain *domain_create(domid_t domid, > > INIT_PAGE_LIST_HEAD(&d->page_list); > > INIT_PAGE_LIST_HEAD(&d->extra_page_list); > > INIT_PAGE_LIST_HEAD(&d->xenpage_list); > > + INIT_PAGE_LIST_HEAD(&d->reserved_page_list); > > > > spin_lock_init(&d->node_affinity_lock); > > d->node_affinity = NODE_MASK_ALL; diff --git > > a/xen/common/page_alloc.c b/xen/common/page_alloc.c index > > f1f1296a61..e3f07ec6c5 100644 > > --- a/xen/common/page_alloc.c > > +++ b/xen/common/page_alloc.c > > @@ -2410,7 +2410,7 @@ int assign_pages( > > > > for ( i = 0; i < nr_pfns; i++ ) > > { > > - ASSERT(!(pg[i].count_info & ~PGC_extra)); > > + ASSERT(!(pg[i].count_info & ~(PGC_extra | > > + PGC_reserved))); > I think this change belongs to the previous patch. > Ok. It will be re-organized into previous commit of "xen/arm: intruduce alloc_domstatic_pages" > > if ( pg[i].count_info & PGC_extra ) > > extra_pages++; > > } > > @@ -2439,6 +2439,9 @@ int assign_pages( > > } > > } > > > > + if ( pg[0].count_info & PGC_reserved ) > > + d->reserved_pages += nr_pfns; > > reserved_pages doesn't seem to be ever read or decremented. So why do > you need it? > Yeah, I may delete it from this Patch Serie. Like I addressed in before commits: "when implementing rebooting domain on static allocation, memory will be relinquished and right now, all shall be freed back to heap, which is not suitable for static memory here. ` relinquish_memory(d, &d->page_list) --> put_page --> free_domheap_page` For pages in PGC_reserved, now, I am considering that, other than giving it back to heap, maybe creating a new global `struct page_info*[DOMID]` value to hold. So it is better to have a new field in struct page_info, as follows, to hold such info. /* Page is reserved. */ struct { /* * Reserved Owner of this page, * if this page is reserved to a specific domain. */ domid_t reserved_owner; } reserved; " So I will delete here, and re-import when implementing rebooting domain on static allocation. > > + > > if ( !(memflags & MEMF_no_refcount) && > > unlikely(domain_adjust_tot_pages(d, nr_pfns) == nr_pfns) ) > > get_knownalive_domain(d); > > @@ -2452,7 +2455,7 @@ int assign_pages( > > page_set_reserved_owner(&pg[i], d); > > smp_wmb(); /* Domain pointer must be visible before updating > refcnt. */ > > pg[i].count_info = > > - (pg[i].count_info & PGC_extra) | PGC_allocated | 1; > > + (pg[i].count_info & (PGC_extra | PGC_reserved)) | > > + PGC_allocated | 1; > > Same here. I'll re-organize it to the previous commit. > > > page_list_add_tail(&pg[i], page_to_list(d, &pg[i])); > > } > > > > diff --git a/xen/include/xen/sched.h b/xen/include/xen/sched.h index > > 3982167144..b6333ed8bb 100644 > > --- a/xen/include/xen/sched.h > > +++ b/xen/include/xen/sched.h > > @@ -368,6 +368,7 @@ struct domain > > struct page_list_head page_list; /* linked list */ > > struct page_list_head extra_page_list; /* linked list (size extra_pages) */ > > struct page_list_head xenpage_list; /* linked list (size > > xenheap_pages) */ > > + struct page_list_head reserved_page_list; /* linked list (size > > + reserved pages) */ > > > > /* > > * This field should only be directly accessed by > > domain_adjust_tot_pages() @@ -379,6 +380,7 @@ struct domain > > unsigned int outstanding_pages; /* pages claimed but not possessed > */ > > unsigned int max_pages; /* maximum value for > domain_tot_pages() */ > > unsigned int extra_pages; /* pages not included in > domain_tot_pages() */ > > + unsigned int reserved_pages; /* pages of static memory */ > > atomic_t shr_pages; /* shared pages */ > > atomic_t paged_pages; /* paged-out pages */ > > > > @@ -588,6 +590,9 @@ static inline struct page_list_head *page_to_list( > > if ( pg->count_info & PGC_extra ) > > return &d->extra_page_list; > > > > + if ( pg->count_info & PGC_reserved ) > > + return &d->reserved_page_list; > > + > > return &d->page_list; > > } > > > > > > Cheers, > > -- > Julien Grall Cheers Penny Zheng
Hi Jan > -----Original Message----- > From: Jan Beulich <jbeulich@suse.com> > Sent: Tuesday, May 18, 2021 7:25 PM > To: Penny Zheng <Penny.Zheng@arm.com> > Cc: Bertrand Marquis <Bertrand.Marquis@arm.com>; Wei Chen > <Wei.Chen@arm.com>; nd <nd@arm.com>; xen-devel@lists.xenproject.org; > sstabellini@kernel.org; julien@xen.org > Subject: Re: [PATCH 08/10] xen/arm: introduce reserved_page_list > > On 18.05.2021 10:38, Penny Zheng wrote: > > Hi Jan > > > >> -----Original Message----- > >> From: Jan Beulich <jbeulich@suse.com> > >> Sent: Tuesday, May 18, 2021 3:39 PM > >> To: Penny Zheng <Penny.Zheng@arm.com> > >> Cc: Bertrand Marquis <Bertrand.Marquis@arm.com>; Wei Chen > >> <Wei.Chen@arm.com>; nd <nd@arm.com>; xen- > devel@lists.xenproject.org; > >> sstabellini@kernel.org; julien@xen.org > >> Subject: Re: [PATCH 08/10] xen/arm: introduce reserved_page_list > >> > >> On 18.05.2021 07:21, Penny Zheng wrote: > >>> Since page_list under struct domain refers to linked pages as gueast > >>> RAM allocated from heap, it should not include reserved pages of > >>> static > >> memory. > >>> > >>> The number of PGC_reserved pages assigned to a domain is tracked in > >>> a new 'reserved_pages' counter. Also introduce a new > >>> reserved_page_list to link pages of static memory. Let page_to_list > >>> return reserved_page_list, when flag is PGC_reserved. > >>> > >>> Later, when domain get destroyed or restarted, those new values will > >>> help relinquish memory to proper place, not been given back to heap. > >>> > >>> Signed-off-by: Penny Zheng <penny.zheng@arm.com> > >>> --- > >>> xen/common/domain.c | 1 + > >>> xen/common/page_alloc.c | 7 +++++-- xen/include/xen/sched.h | 5 > >>> +++++ > >>> 3 files changed, 11 insertions(+), 2 deletions(-) > >> > >> This contradicts the title's prefix: There's no Arm-specific change here at > all. > >> But imo the title is correct, and the changes should be Arm-specific. > >> There's no point having struct domain fields on e.g. x86 which aren't used > there at all. > >> > > > > Yep, you're right. > > I'll add ifdefs in the following changes. > > As necessary, please. Moving stuff to Arm-specific files would be preferable. > Sure, I'll add a new CONFIG_STATICMEM to include all related functions and variables. Thx > Jan Cheers Penny Zheng
diff --git a/xen/common/domain.c b/xen/common/domain.c index 6b71c6d6a9..c38afd2969 100644 --- a/xen/common/domain.c +++ b/xen/common/domain.c @@ -578,6 +578,7 @@ struct domain *domain_create(domid_t domid, INIT_PAGE_LIST_HEAD(&d->page_list); INIT_PAGE_LIST_HEAD(&d->extra_page_list); INIT_PAGE_LIST_HEAD(&d->xenpage_list); + INIT_PAGE_LIST_HEAD(&d->reserved_page_list); spin_lock_init(&d->node_affinity_lock); d->node_affinity = NODE_MASK_ALL; diff --git a/xen/common/page_alloc.c b/xen/common/page_alloc.c index f1f1296a61..e3f07ec6c5 100644 --- a/xen/common/page_alloc.c +++ b/xen/common/page_alloc.c @@ -2410,7 +2410,7 @@ int assign_pages( for ( i = 0; i < nr_pfns; i++ ) { - ASSERT(!(pg[i].count_info & ~PGC_extra)); + ASSERT(!(pg[i].count_info & ~(PGC_extra | PGC_reserved))); if ( pg[i].count_info & PGC_extra ) extra_pages++; } @@ -2439,6 +2439,9 @@ int assign_pages( } } + if ( pg[0].count_info & PGC_reserved ) + d->reserved_pages += nr_pfns; + if ( !(memflags & MEMF_no_refcount) && unlikely(domain_adjust_tot_pages(d, nr_pfns) == nr_pfns) ) get_knownalive_domain(d); @@ -2452,7 +2455,7 @@ int assign_pages( page_set_reserved_owner(&pg[i], d); smp_wmb(); /* Domain pointer must be visible before updating refcnt. */ pg[i].count_info = - (pg[i].count_info & PGC_extra) | PGC_allocated | 1; + (pg[i].count_info & (PGC_extra | PGC_reserved)) | PGC_allocated | 1; page_list_add_tail(&pg[i], page_to_list(d, &pg[i])); } diff --git a/xen/include/xen/sched.h b/xen/include/xen/sched.h index 3982167144..b6333ed8bb 100644 --- a/xen/include/xen/sched.h +++ b/xen/include/xen/sched.h @@ -368,6 +368,7 @@ struct domain struct page_list_head page_list; /* linked list */ struct page_list_head extra_page_list; /* linked list (size extra_pages) */ struct page_list_head xenpage_list; /* linked list (size xenheap_pages) */ + struct page_list_head reserved_page_list; /* linked list (size reserved pages) */ /* * This field should only be directly accessed by domain_adjust_tot_pages() @@ -379,6 +380,7 @@ struct domain unsigned int outstanding_pages; /* pages claimed but not possessed */ unsigned int max_pages; /* maximum value for domain_tot_pages() */ unsigned int extra_pages; /* pages not included in domain_tot_pages() */ + unsigned int reserved_pages; /* pages of static memory */ atomic_t shr_pages; /* shared pages */ atomic_t paged_pages; /* paged-out pages */ @@ -588,6 +590,9 @@ static inline struct page_list_head *page_to_list( if ( pg->count_info & PGC_extra ) return &d->extra_page_list; + if ( pg->count_info & PGC_reserved ) + return &d->reserved_page_list; + return &d->page_list; }
Since page_list under struct domain refers to linked pages as gueast RAM allocated from heap, it should not include reserved pages of static memory. The number of PGC_reserved pages assigned to a domain is tracked in a new 'reserved_pages' counter. Also introduce a new reserved_page_list to link pages of static memory. Let page_to_list return reserved_page_list, when flag is PGC_reserved. Later, when domain get destroyed or restarted, those new values will help relinquish memory to proper place, not been given back to heap. Signed-off-by: Penny Zheng <penny.zheng@arm.com> --- xen/common/domain.c | 1 + xen/common/page_alloc.c | 7 +++++-- xen/include/xen/sched.h | 5 +++++ 3 files changed, 11 insertions(+), 2 deletions(-)