diff mbox series

[v10,8/9] xen: retrieve reserved pages on populate_physmap

Message ID 20220816023658.3551936-9-Penny.Zheng@arm.com (mailing list archive)
State Superseded
Headers show
Series populate/unpopulate memory when domain on static allocation | expand

Commit Message

Penny Zheng Aug. 16, 2022, 2:36 a.m. UTC
When a static domain populates memory through populate_physmap at runtime,
it shall retrieve reserved pages from resv_page_list to make sure that
guest RAM is still restricted in statically configured memory regions.
This commit also introduces a new helper acquire_reserved_page to make it work.

Signed-off-by: Penny Zheng <penny.zheng@arm.com>
---
v10 changes:
- add lock on the fail path
---
v9 changes:
- Use ASSERT_ALLOC_CONTEXT() in acquire_reserved_page
- Add free_staticmem_pages to undo prepare_staticmem_pages when
assign_domstatic_pages
- Remove redundant static in error message
---
v8 changes:
- As concurrent free/allocate could modify the resv_page_list, we still
need the lock
---
v7 changes:
- remove the lock, since we add the page to rsv_page_list after it has
been totally freed.
---
v6 changes:
- drop the lock before returning
---
v5 changes:
- extract common codes for assigning pages into a helper assign_domstatic_pages
- refine commit message
- remove stub function acquire_reserved_page
- Alloc/free of memory can happen concurrently. So access to rsv_page_list
needs to be protected with a spinlock
---
v4 changes:
- miss dropping __init in acquire_domstatic_pages
- add the page back to the reserved list in case of error
- remove redundant printk
- refine log message and make it warn level
---
v3 changes:
- move is_domain_using_staticmem to the common header file
- remove #ifdef CONFIG_STATIC_MEMORY-ary
- remove meaningless page_to_mfn(page) in error log
---
v2 changes:
- introduce acquire_reserved_page to retrieve reserved pages from
resv_page_list
- forbid non-zero-order requests in populate_physmap
- let is_domain_static return ((void)(d), false) on x86
---
 xen/common/memory.c     | 23 +++++++++++++
 xen/common/page_alloc.c | 76 ++++++++++++++++++++++++++++++++---------
 xen/include/xen/mm.h    |  1 +
 3 files changed, 83 insertions(+), 17 deletions(-)

Comments

Jan Beulich Aug. 17, 2022, 10:04 a.m. UTC | #1
On 16.08.2022 04:36, Penny Zheng wrote:
> @@ -2867,6 +2854,61 @@ int __init acquire_domstatic_pages(struct domain *d, mfn_t smfn,
>  
>      return 0;
>  }
> +
> +/*
> + * Acquire nr_mfns contiguous pages, starting at #smfn, of static memory,
> + * then assign them to one specific domain #d.
> + */
> +int __init acquire_domstatic_pages(struct domain *d, mfn_t smfn,
> +                                   unsigned int nr_mfns, unsigned int memflags)
> +{
> +    struct page_info *pg;
> +
> +    ASSERT_ALLOC_CONTEXT();
> +
> +    pg = acquire_staticmem_pages(smfn, nr_mfns, memflags);
> +    if ( !pg )
> +        return -ENOENT;
> +
> +    if ( assign_domstatic_pages(d, pg, nr_mfns, memflags) )
> +        return -EINVAL;
> +
> +    return 0;
> +}
> +
> +/*
> + * Acquire a page from reserved page list(resv_page_list), when populating
> + * memory for static domain on runtime.
> + */
> +mfn_t acquire_reserved_page(struct domain *d, unsigned int memflags)
> +{
> +    struct page_info *page;
> +
> +    ASSERT_ALLOC_CONTEXT();
> +
> +    /* Acquire a page from reserved page list(resv_page_list). */
> +    spin_lock(&d->page_alloc_lock);
> +    page = page_list_remove_head(&d->resv_page_list);
> +    spin_unlock(&d->page_alloc_lock);
> +    if ( unlikely(!page) )
> +        return INVALID_MFN;
> +
> +    if ( !prepare_staticmem_pages(page, 1, memflags) )
> +        goto fail;
> +
> +    if ( assign_domstatic_pages(d, page, 1, memflags) )
> +        goto fail_assign;
> +
> +    return page_to_mfn(page);
> +
> + fail_assign:
> +    free_staticmem_pages(page, 1, memflags & MEMF_no_scrub);

Doesn't this need to be !(memflags & MEMF_no_scrub)? And then -
with assignment having failed and with it being just a single page
we're talking about, the page was not exposed to the guest at any
point afaict. So I don't see the need for scrubbing in the first
place.

Also I think the rename of the function would better be done first,
since then you wouldn't need to touch this line again right in the
next patch, and the prepare/unprepare pairing would also be visible
right here. This would then also better fit with the introduction
of prepare_*() in the previous patch (which, afaic, the name
change could also be merged into; FTAOD I don't mind it to be
separate).

Jan
Penny Zheng Sept. 5, 2022, 7:08 a.m. UTC | #2
Hi jan 

> -----Original Message-----
> From: Jan Beulich <jbeulich@suse.com>
> Sent: Wednesday, August 17, 2022 6:05 PM
> To: Penny Zheng <Penny.Zheng@arm.com>
> Cc: Wei Chen <Wei.Chen@arm.com>; Andrew Cooper
> <andrew.cooper3@citrix.com>; George Dunlap <george.dunlap@citrix.com>;
> Julien Grall <julien@xen.org>; Stefano Stabellini <sstabellini@kernel.org>;
> Wei Liu <wl@xen.org>; xen-devel@lists.xenproject.org
> Subject: Re: [PATCH v10 8/9] xen: retrieve reserved pages on
> populate_physmap
> 
> On 16.08.2022 04:36, Penny Zheng wrote:
> > @@ -2867,6 +2854,61 @@ int __init acquire_domstatic_pages(struct
> > domain *d, mfn_t smfn,
> >
> >      return 0;
> >  }
> > +
> > +/*
> > + * Acquire nr_mfns contiguous pages, starting at #smfn, of static
> > +memory,
> > + * then assign them to one specific domain #d.
> > + */
> > +int __init acquire_domstatic_pages(struct domain *d, mfn_t smfn,
> > +                                   unsigned int nr_mfns, unsigned int
> > +memflags) {
> > +    struct page_info *pg;
> > +
> > +    ASSERT_ALLOC_CONTEXT();
> > +
> > +    pg = acquire_staticmem_pages(smfn, nr_mfns, memflags);
> > +    if ( !pg )
> > +        return -ENOENT;
> > +
> > +    if ( assign_domstatic_pages(d, pg, nr_mfns, memflags) )
> > +        return -EINVAL;
> > +
> > +    return 0;
> > +}
> > +
> > +/*
> > + * Acquire a page from reserved page list(resv_page_list), when
> > +populating
> > + * memory for static domain on runtime.
> > + */
> > +mfn_t acquire_reserved_page(struct domain *d, unsigned int memflags)
> > +{
> > +    struct page_info *page;
> > +
> > +    ASSERT_ALLOC_CONTEXT();
> > +
> > +    /* Acquire a page from reserved page list(resv_page_list). */
> > +    spin_lock(&d->page_alloc_lock);
> > +    page = page_list_remove_head(&d->resv_page_list);
> > +    spin_unlock(&d->page_alloc_lock);
> > +    if ( unlikely(!page) )
> > +        return INVALID_MFN;
> > +
> > +    if ( !prepare_staticmem_pages(page, 1, memflags) )
> > +        goto fail;
> > +
> > +    if ( assign_domstatic_pages(d, page, 1, memflags) )
> > +        goto fail_assign;
> > +
> > +    return page_to_mfn(page);
> > +
> > + fail_assign:
> > +    free_staticmem_pages(page, 1, memflags & MEMF_no_scrub);
> 
> Doesn't this need to be !(memflags & MEMF_no_scrub)? And then - with

I got a bit confused about this flag MEMF_no_scrub, does it mean no need
to scrub? 
Since I saw that in alloc_domheap_pages(...)
    if ( assign_page(pg, order, d, memflags) )
    {
        free_heap_pages(pg, order, memflags & MEMF_no_scrub);
        return NULL;
    }
It doesn't contain exclamation mark too...

> assignment having failed and with it being just a single page we're talking
> about, the page was not exposed to the guest at any point afaict. So I don't
> see the need for scrubbing in the first place.
> 
> Also I think the rename of the function would better be done first, since then
> you wouldn't need to touch this line again right in the next patch, and the
> prepare/unprepare pairing would also be visible right here. This would then
> also better fit with the introduction of prepare_*() in the previous patch
> (which, afaic, the name change could also be merged into; FTAOD I don't
> mind it to be separate).
> 
> Jan
Jan Beulich Sept. 6, 2022, 6:33 a.m. UTC | #3
On 05.09.2022 09:08, Penny Zheng wrote:
> Hi jan 
> 
>> -----Original Message-----
>> From: Jan Beulich <jbeulich@suse.com>
>> Sent: Wednesday, August 17, 2022 6:05 PM
>> To: Penny Zheng <Penny.Zheng@arm.com>
>> Cc: Wei Chen <Wei.Chen@arm.com>; Andrew Cooper
>> <andrew.cooper3@citrix.com>; George Dunlap <george.dunlap@citrix.com>;
>> Julien Grall <julien@xen.org>; Stefano Stabellini <sstabellini@kernel.org>;
>> Wei Liu <wl@xen.org>; xen-devel@lists.xenproject.org
>> Subject: Re: [PATCH v10 8/9] xen: retrieve reserved pages on
>> populate_physmap
>>
>> On 16.08.2022 04:36, Penny Zheng wrote:
>>> @@ -2867,6 +2854,61 @@ int __init acquire_domstatic_pages(struct
>>> domain *d, mfn_t smfn,
>>>
>>>      return 0;
>>>  }
>>> +
>>> +/*
>>> + * Acquire nr_mfns contiguous pages, starting at #smfn, of static
>>> +memory,
>>> + * then assign them to one specific domain #d.
>>> + */
>>> +int __init acquire_domstatic_pages(struct domain *d, mfn_t smfn,
>>> +                                   unsigned int nr_mfns, unsigned int
>>> +memflags) {
>>> +    struct page_info *pg;
>>> +
>>> +    ASSERT_ALLOC_CONTEXT();
>>> +
>>> +    pg = acquire_staticmem_pages(smfn, nr_mfns, memflags);
>>> +    if ( !pg )
>>> +        return -ENOENT;
>>> +
>>> +    if ( assign_domstatic_pages(d, pg, nr_mfns, memflags) )
>>> +        return -EINVAL;
>>> +
>>> +    return 0;
>>> +}
>>> +
>>> +/*
>>> + * Acquire a page from reserved page list(resv_page_list), when
>>> +populating
>>> + * memory for static domain on runtime.
>>> + */
>>> +mfn_t acquire_reserved_page(struct domain *d, unsigned int memflags)
>>> +{
>>> +    struct page_info *page;
>>> +
>>> +    ASSERT_ALLOC_CONTEXT();
>>> +
>>> +    /* Acquire a page from reserved page list(resv_page_list). */
>>> +    spin_lock(&d->page_alloc_lock);
>>> +    page = page_list_remove_head(&d->resv_page_list);
>>> +    spin_unlock(&d->page_alloc_lock);
>>> +    if ( unlikely(!page) )
>>> +        return INVALID_MFN;
>>> +
>>> +    if ( !prepare_staticmem_pages(page, 1, memflags) )
>>> +        goto fail;
>>> +
>>> +    if ( assign_domstatic_pages(d, page, 1, memflags) )
>>> +        goto fail_assign;
>>> +
>>> +    return page_to_mfn(page);
>>> +
>>> + fail_assign:
>>> +    free_staticmem_pages(page, 1, memflags & MEMF_no_scrub);
>>
>> Doesn't this need to be !(memflags & MEMF_no_scrub)? And then - with
> 
> I got a bit confused about this flag MEMF_no_scrub, does it mean no need
> to scrub? 

Yes, as its name says.

> Since I saw that in alloc_domheap_pages(...)
>     if ( assign_page(pg, order, d, memflags) )
>     {
>         free_heap_pages(pg, order, memflags & MEMF_no_scrub);
>         return NULL;
>     }
> It doesn't contain exclamation mark too...

Hmm, you're right - on these error paths the scrubbing is needed if
the page wasn't previously scrubbed, as part of the set of pages may
have been transiently exposed to the guest (and by guessing it may
have been able to actually access the pages; I'm inclined to say it's
its own fault though if that way information is being leaked).

But ...

>> assignment having failed and with it being just a single page we're talking
>> about, the page was not exposed to the guest at any point afaict. So I don't
>> see the need for scrubbing in the first place.

while my comment wasn't really correct, as said - you don't need any
scrubbing here at all, I think.

Jan
Penny Zheng Sept. 6, 2022, 7:14 a.m. UTC | #4
Hi Jan

> -----Original Message-----
> From: Jan Beulich <jbeulich@suse.com>
> Sent: Tuesday, September 6, 2022 2:34 PM
> To: Penny Zheng <Penny.Zheng@arm.com>
> Cc: Wei Chen <Wei.Chen@arm.com>; Andrew Cooper
> <andrew.cooper3@citrix.com>; George Dunlap <george.dunlap@citrix.com>;
> Julien Grall <julien@xen.org>; Stefano Stabellini <sstabellini@kernel.org>;
> Wei Liu <wl@xen.org>; xen-devel@lists.xenproject.org
> Subject: Re: [PATCH v10 8/9] xen: retrieve reserved pages on
> populate_physmap
> 
> On 05.09.2022 09:08, Penny Zheng wrote:
> > Hi jan
> >
> >> -----Original Message-----
> >> From: Jan Beulich <jbeulich@suse.com>
> >> Sent: Wednesday, August 17, 2022 6:05 PM
> >> To: Penny Zheng <Penny.Zheng@arm.com>
> >> Cc: Wei Chen <Wei.Chen@arm.com>; Andrew Cooper
> >> <andrew.cooper3@citrix.com>; George Dunlap
> >> <george.dunlap@citrix.com>; Julien Grall <julien@xen.org>; Stefano
> >> Stabellini <sstabellini@kernel.org>; Wei Liu <wl@xen.org>;
> >> xen-devel@lists.xenproject.org
> >> Subject: Re: [PATCH v10 8/9] xen: retrieve reserved pages on
> >> populate_physmap
> >>
> >> On 16.08.2022 04:36, Penny Zheng wrote:
> >>> @@ -2867,6 +2854,61 @@ int __init acquire_domstatic_pages(struct
> >>> domain *d, mfn_t smfn,
> >>>
> >>>      return 0;
> >>>  }
> >>> +
> >>> +/*
> >>> + * Acquire nr_mfns contiguous pages, starting at #smfn, of static
> >>> +memory,
> >>> + * then assign them to one specific domain #d.
> >>> + */
> >>> +int __init acquire_domstatic_pages(struct domain *d, mfn_t smfn,
> >>> +                                   unsigned int nr_mfns, unsigned
> >>> +int
> >>> +memflags) {
> >>> +    struct page_info *pg;
> >>> +
> >>> +    ASSERT_ALLOC_CONTEXT();
> >>> +
> >>> +    pg = acquire_staticmem_pages(smfn, nr_mfns, memflags);
> >>> +    if ( !pg )
> >>> +        return -ENOENT;
> >>> +
> >>> +    if ( assign_domstatic_pages(d, pg, nr_mfns, memflags) )
> >>> +        return -EINVAL;
> >>> +
> >>> +    return 0;
> >>> +}
> >>> +
> >>> +/*
> >>> + * Acquire a page from reserved page list(resv_page_list), when
> >>> +populating
> >>> + * memory for static domain on runtime.
> >>> + */
> >>> +mfn_t acquire_reserved_page(struct domain *d, unsigned int
> >>> +memflags) {
> >>> +    struct page_info *page;
> >>> +
> >>> +    ASSERT_ALLOC_CONTEXT();
> >>> +
> >>> +    /* Acquire a page from reserved page list(resv_page_list). */
> >>> +    spin_lock(&d->page_alloc_lock);
> >>> +    page = page_list_remove_head(&d->resv_page_list);
> >>> +    spin_unlock(&d->page_alloc_lock);
> >>> +    if ( unlikely(!page) )
> >>> +        return INVALID_MFN;
> >>> +
> >>> +    if ( !prepare_staticmem_pages(page, 1, memflags) )
> >>> +        goto fail;
> >>> +
> >>> +    if ( assign_domstatic_pages(d, page, 1, memflags) )
> >>> +        goto fail_assign;
> >>> +
> >>> +    return page_to_mfn(page);
> >>> +
> >>> + fail_assign:
> >>> +    free_staticmem_pages(page, 1, memflags & MEMF_no_scrub);
> >>
> >> Doesn't this need to be !(memflags & MEMF_no_scrub)? And then - with
> >
> > I got a bit confused about this flag MEMF_no_scrub, does it mean no
> > need to scrub?
> 
> Yes, as its name says.
> 
> > Since I saw that in alloc_domheap_pages(...)
> >     if ( assign_page(pg, order, d, memflags) )
> >     {
> >         free_heap_pages(pg, order, memflags & MEMF_no_scrub);
> >         return NULL;
> >     }
> > It doesn't contain exclamation mark too...
> 
> Hmm, you're right - on these error paths the scrubbing is needed if the page
> wasn't previously scrubbed, as part of the set of pages may have been
> transiently exposed to the guest (and by guessing it may have been able to
> actually access the pages; I'm inclined to say it's its own fault though if that
> way information is being leaked).
> 

Then, the same for the acquire_domstatic_pages(...)

    if ( assign_pages(pg, nr_mfns, d, memflags) )
    {
        free_staticmem_pages(pg, nr_mfns, memflags & MEMF_no_scrub);
        return -EINVAL;
    }
On this error path, it has misused the MEMF_no_scrub too.
But IMO, as we are talking about these pages will always be reserved to the guest,
maybe here it also doesn't need scrubbing at all?
 
> But ...
> 
> >> assignment having failed and with it being just a single page we're
> >> talking about, the page was not exposed to the guest at any point
> >> afaict. So I don't see the need for scrubbing in the first place.
> 
> while my comment wasn't really correct, as said - you don't need any
> scrubbing here at all, I think.
> 
> Jan
Jan Beulich Sept. 6, 2022, 7:19 a.m. UTC | #5
On 06.09.2022 09:14, Penny Zheng wrote:
> Hi Jan
> 
>> -----Original Message-----
>> From: Jan Beulich <jbeulich@suse.com>
>> Sent: Tuesday, September 6, 2022 2:34 PM
>> To: Penny Zheng <Penny.Zheng@arm.com>
>> Cc: Wei Chen <Wei.Chen@arm.com>; Andrew Cooper
>> <andrew.cooper3@citrix.com>; George Dunlap <george.dunlap@citrix.com>;
>> Julien Grall <julien@xen.org>; Stefano Stabellini <sstabellini@kernel.org>;
>> Wei Liu <wl@xen.org>; xen-devel@lists.xenproject.org
>> Subject: Re: [PATCH v10 8/9] xen: retrieve reserved pages on
>> populate_physmap
>>
>> On 05.09.2022 09:08, Penny Zheng wrote:
>>> Hi jan
>>>
>>>> -----Original Message-----
>>>> From: Jan Beulich <jbeulich@suse.com>
>>>> Sent: Wednesday, August 17, 2022 6:05 PM
>>>> To: Penny Zheng <Penny.Zheng@arm.com>
>>>> Cc: Wei Chen <Wei.Chen@arm.com>; Andrew Cooper
>>>> <andrew.cooper3@citrix.com>; George Dunlap
>>>> <george.dunlap@citrix.com>; Julien Grall <julien@xen.org>; Stefano
>>>> Stabellini <sstabellini@kernel.org>; Wei Liu <wl@xen.org>;
>>>> xen-devel@lists.xenproject.org
>>>> Subject: Re: [PATCH v10 8/9] xen: retrieve reserved pages on
>>>> populate_physmap
>>>>
>>>> On 16.08.2022 04:36, Penny Zheng wrote:
>>>>> @@ -2867,6 +2854,61 @@ int __init acquire_domstatic_pages(struct
>>>>> domain *d, mfn_t smfn,
>>>>>
>>>>>      return 0;
>>>>>  }
>>>>> +
>>>>> +/*
>>>>> + * Acquire nr_mfns contiguous pages, starting at #smfn, of static
>>>>> +memory,
>>>>> + * then assign them to one specific domain #d.
>>>>> + */
>>>>> +int __init acquire_domstatic_pages(struct domain *d, mfn_t smfn,
>>>>> +                                   unsigned int nr_mfns, unsigned
>>>>> +int
>>>>> +memflags) {
>>>>> +    struct page_info *pg;
>>>>> +
>>>>> +    ASSERT_ALLOC_CONTEXT();
>>>>> +
>>>>> +    pg = acquire_staticmem_pages(smfn, nr_mfns, memflags);
>>>>> +    if ( !pg )
>>>>> +        return -ENOENT;
>>>>> +
>>>>> +    if ( assign_domstatic_pages(d, pg, nr_mfns, memflags) )
>>>>> +        return -EINVAL;
>>>>> +
>>>>> +    return 0;
>>>>> +}
>>>>> +
>>>>> +/*
>>>>> + * Acquire a page from reserved page list(resv_page_list), when
>>>>> +populating
>>>>> + * memory for static domain on runtime.
>>>>> + */
>>>>> +mfn_t acquire_reserved_page(struct domain *d, unsigned int
>>>>> +memflags) {
>>>>> +    struct page_info *page;
>>>>> +
>>>>> +    ASSERT_ALLOC_CONTEXT();
>>>>> +
>>>>> +    /* Acquire a page from reserved page list(resv_page_list). */
>>>>> +    spin_lock(&d->page_alloc_lock);
>>>>> +    page = page_list_remove_head(&d->resv_page_list);
>>>>> +    spin_unlock(&d->page_alloc_lock);
>>>>> +    if ( unlikely(!page) )
>>>>> +        return INVALID_MFN;
>>>>> +
>>>>> +    if ( !prepare_staticmem_pages(page, 1, memflags) )
>>>>> +        goto fail;
>>>>> +
>>>>> +    if ( assign_domstatic_pages(d, page, 1, memflags) )
>>>>> +        goto fail_assign;
>>>>> +
>>>>> +    return page_to_mfn(page);
>>>>> +
>>>>> + fail_assign:
>>>>> +    free_staticmem_pages(page, 1, memflags & MEMF_no_scrub);
>>>>
>>>> Doesn't this need to be !(memflags & MEMF_no_scrub)? And then - with
>>>
>>> I got a bit confused about this flag MEMF_no_scrub, does it mean no
>>> need to scrub?
>>
>> Yes, as its name says.
>>
>>> Since I saw that in alloc_domheap_pages(...)
>>>     if ( assign_page(pg, order, d, memflags) )
>>>     {
>>>         free_heap_pages(pg, order, memflags & MEMF_no_scrub);
>>>         return NULL;
>>>     }
>>> It doesn't contain exclamation mark too...
>>
>> Hmm, you're right - on these error paths the scrubbing is needed if the page
>> wasn't previously scrubbed, as part of the set of pages may have been
>> transiently exposed to the guest (and by guessing it may have been able to
>> actually access the pages; I'm inclined to say it's its own fault though if that
>> way information is being leaked).
>>
> 
> Then, the same for the acquire_domstatic_pages(...)
> 
>     if ( assign_pages(pg, nr_mfns, d, memflags) )
>     {
>         free_staticmem_pages(pg, nr_mfns, memflags & MEMF_no_scrub);
>         return -EINVAL;
>     }
> On this error path, it has misused the MEMF_no_scrub too.

Why do you say "misused"?

> But IMO, as we are talking about these pages will always be reserved to the guest,
> maybe here it also doesn't need scrubbing at all?

Perhaps. It feels as if we had been there before, quite some time ago.

Jan
diff mbox series

Patch

diff --git a/xen/common/memory.c b/xen/common/memory.c
index bc89442ba5..ae8163a738 100644
--- a/xen/common/memory.c
+++ b/xen/common/memory.c
@@ -245,6 +245,29 @@  static void populate_physmap(struct memop_args *a)
 
                 mfn = _mfn(gpfn);
             }
+            else if ( is_domain_using_staticmem(d) )
+            {
+                /*
+                 * No easy way to guarantee the retrieved pages are contiguous,
+                 * so forbid non-zero-order requests here.
+                 */
+                if ( a->extent_order != 0 )
+                {
+                    gdprintk(XENLOG_WARNING,
+                             "Cannot allocate static order-%u pages for %pd\n",
+                             a->extent_order, d);
+                    goto out;
+                }
+
+                mfn = acquire_reserved_page(d, a->memflags);
+                if ( mfn_eq(mfn, INVALID_MFN) )
+                {
+                    gdprintk(XENLOG_WARNING,
+                             "%pd: failed to retrieve a reserved page\n",
+                             d);
+                    goto out;
+                }
+            }
             else
             {
                 page = alloc_domheap_pages(d, a->extent_order, a->memflags);
diff --git a/xen/common/page_alloc.c b/xen/common/page_alloc.c
index 0ee697705c..9c6d369d10 100644
--- a/xen/common/page_alloc.c
+++ b/xen/common/page_alloc.c
@@ -2751,9 +2751,8 @@  void free_domstatic_page(struct page_info *page)
         put_domain(d);
 }
 
-static bool __init prepare_staticmem_pages(struct page_info *pg,
-                                           unsigned long nr_mfns,
-                                           unsigned int memflags)
+static bool prepare_staticmem_pages(struct page_info *pg, unsigned long nr_mfns,
+                                    unsigned int memflags)
 {
     bool need_tlbflush = false;
     uint32_t tlbflush_timestamp = 0;
@@ -2834,21 +2833,9 @@  static struct page_info * __init acquire_staticmem_pages(mfn_t smfn,
     return pg;
 }
 
-/*
- * Acquire nr_mfns contiguous pages, starting at #smfn, of static memory,
- * then assign them to one specific domain #d.
- */
-int __init acquire_domstatic_pages(struct domain *d, mfn_t smfn,
-                                   unsigned int nr_mfns, unsigned int memflags)
+static int assign_domstatic_pages(struct domain *d, struct page_info *pg,
+                                  unsigned int nr_mfns, unsigned int memflags)
 {
-    struct page_info *pg;
-
-    ASSERT_ALLOC_CONTEXT();
-
-    pg = acquire_staticmem_pages(smfn, nr_mfns, memflags);
-    if ( !pg )
-        return -ENOENT;
-
     if ( !d || (memflags & (MEMF_no_owner | MEMF_no_refcount)) )
     {
         /*
@@ -2867,6 +2854,61 @@  int __init acquire_domstatic_pages(struct domain *d, mfn_t smfn,
 
     return 0;
 }
+
+/*
+ * Acquire nr_mfns contiguous pages, starting at #smfn, of static memory,
+ * then assign them to one specific domain #d.
+ */
+int __init acquire_domstatic_pages(struct domain *d, mfn_t smfn,
+                                   unsigned int nr_mfns, unsigned int memflags)
+{
+    struct page_info *pg;
+
+    ASSERT_ALLOC_CONTEXT();
+
+    pg = acquire_staticmem_pages(smfn, nr_mfns, memflags);
+    if ( !pg )
+        return -ENOENT;
+
+    if ( assign_domstatic_pages(d, pg, nr_mfns, memflags) )
+        return -EINVAL;
+
+    return 0;
+}
+
+/*
+ * Acquire a page from reserved page list(resv_page_list), when populating
+ * memory for static domain on runtime.
+ */
+mfn_t acquire_reserved_page(struct domain *d, unsigned int memflags)
+{
+    struct page_info *page;
+
+    ASSERT_ALLOC_CONTEXT();
+
+    /* Acquire a page from reserved page list(resv_page_list). */
+    spin_lock(&d->page_alloc_lock);
+    page = page_list_remove_head(&d->resv_page_list);
+    spin_unlock(&d->page_alloc_lock);
+    if ( unlikely(!page) )
+        return INVALID_MFN;
+
+    if ( !prepare_staticmem_pages(page, 1, memflags) )
+        goto fail;
+
+    if ( assign_domstatic_pages(d, page, 1, memflags) )
+        goto fail_assign;
+
+    return page_to_mfn(page);
+
+ fail_assign:
+    free_staticmem_pages(page, 1, memflags & MEMF_no_scrub);
+ fail:
+    spin_lock(&d->page_alloc_lock);
+    page_list_add_tail(page, &d->resv_page_list);
+    spin_unlock(&d->page_alloc_lock);
+    return INVALID_MFN;
+}
 #endif
 
 /*
diff --git a/xen/include/xen/mm.h b/xen/include/xen/mm.h
index deadf4b2a1..5d29aea7ad 100644
--- a/xen/include/xen/mm.h
+++ b/xen/include/xen/mm.h
@@ -198,6 +198,7 @@  struct npfec {
 #else
 #define MAX_ORDER 20 /* 2^20 contiguous pages */
 #endif
+mfn_t acquire_reserved_page(struct domain *d, unsigned int memflags);
 
 /* Private domain structs for DOMID_XEN, DOMID_IO, etc. */
 extern struct domain *dom_xen, *dom_io;