| Message ID | 20201213154534.54826-4-songmuchun@bytedance.com (mailing list archive) |
|---|---|
| State | New, archived |
| Headers | show |
| Series | Free some vmemmap pages of HugeTLB page | expand |
On Sun, Dec 13, 2020 at 11:45:26PM +0800, Muchun Song wrote: > + > +/* > + * vmemmap_rmap_walk - walk vmemmap page table > + * > + * @rmap_pte: called for each non-empty PTE (lowest-level) entry. > + * @reuse: the page which is reused for the tail vmemmap pages. > + * @vmemmap_pages: the list head of the vmemmap pages that can be freed. > + */ > +struct vmemmap_rmap_walk { > + void (*rmap_pte)(pte_t *pte, unsigned long addr, > + struct vmemmap_rmap_walk *walk); > + struct page *reuse; > + struct list_head *vmemmap_pages; > +}; Why did you chose this approach in this version? Earlier versions of this patchset had a single vmemmap_to_pmd() function which returned the PMD, and now we have serveral vmemmap_{levels}_range and a vmemmap_rmap_walk. A brief explanation about why this change was introduced would have been nice. I guess it is because ealier versions were too oriented for the usecase this patchset presents, while the new versions tries to be more broad about future re-uses of the interface?
On Wed, Dec 16, 2020 at 9:06 PM Oscar Salvador <osalvador@suse.de> wrote: > > On Sun, Dec 13, 2020 at 11:45:26PM +0800, Muchun Song wrote: > > + > > +/* > > + * vmemmap_rmap_walk - walk vmemmap page table > > + * > > + * @rmap_pte: called for each non-empty PTE (lowest-level) entry. > > + * @reuse: the page which is reused for the tail vmemmap pages. > > + * @vmemmap_pages: the list head of the vmemmap pages that can be freed. > > + */ > > +struct vmemmap_rmap_walk { > > + void (*rmap_pte)(pte_t *pte, unsigned long addr, > > + struct vmemmap_rmap_walk *walk); > > + struct page *reuse; > > + struct list_head *vmemmap_pages; > > +}; > > Why did you chose this approach in this version? > Earlier versions of this patchset had a single vmemmap_to_pmd() function > which returned the PMD, and now we have serveral vmemmap_{levels}_range > and a vmemmap_rmap_walk. This approach will be more universal. :-) > A brief explanation about why this change was introduced would have been nice. > > I guess it is because ealier versions were too oriented for the usecase > this patchset presents, while the new versions tries to be more broad > about future re-uses of the interface? Yeah, you are right. I plan to reuse those interfaces in the feature. Thanks. > > > -- > Oscar Salvador > SUSE L3
On 12/13/20 7:45 AM, Muchun Song wrote: > Every HugeTLB has more than one struct page structure. We __know__ that > we only use the first 4(HUGETLB_CGROUP_MIN_ORDER) struct page structures > to store metadata associated with each HugeTLB. > > There are a lot of struct page structures associated with each HugeTLB > page. For tail pages, the value of compound_head is the same. So we can > reuse first page of tail page structures. We map the virtual addresses > of the remaining pages of tail page structures to the first tail page > struct, and then free these page frames. Therefore, we need to reserve > two pages as vmemmap areas. > > When we allocate a HugeTLB page from the buddy, we can free some vmemmap > pages associated with each HugeTLB page. It is more appropriate to do it > in the prep_new_huge_page(). > > The free_vmemmap_pages_per_hpage(), which indicates how many vmemmap > pages associated with a HugeTLB page can be freed, returns zero for > now, which means the feature is disabled. We will enable it once all > the infrastructure is there. > > Signed-off-by: Muchun Song <songmuchun@bytedance.com> > --- > include/linux/bootmem_info.h | 27 +++++- > include/linux/mm.h | 2 + > mm/Makefile | 1 + > mm/hugetlb.c | 3 + > mm/hugetlb_vmemmap.c | 209 +++++++++++++++++++++++++++++++++++++++++++ > mm/hugetlb_vmemmap.h | 20 +++++ > mm/sparse-vmemmap.c | 170 +++++++++++++++++++++++++++++++++++ > 7 files changed, 431 insertions(+), 1 deletion(-) > create mode 100644 mm/hugetlb_vmemmap.c > create mode 100644 mm/hugetlb_vmemmap.h > diff --git a/mm/sparse-vmemmap.c b/mm/sparse-vmemmap.c > index 16183d85a7d5..78c527617e8d 100644 > --- a/mm/sparse-vmemmap.c > +++ b/mm/sparse-vmemmap.c > @@ -27,8 +27,178 @@ > #include <linux/spinlock.h> > #include <linux/vmalloc.h> > #include <linux/sched.h> > +#include <linux/pgtable.h> > +#include <linux/bootmem_info.h> > + > #include <asm/dma.h> > #include <asm/pgalloc.h> > +#include <asm/tlbflush.h> > + > +/* > + * vmemmap_rmap_walk - walk vmemmap page table I am not sure if 'rmap' should be part of these names. rmap today is mostly about reverse mapping lookup. Did you use rmap for 'remap', or because this code is patterned after the page table walking rmap code? Just think the naming could cause some confusion. > + * > + * @rmap_pte: called for each non-empty PTE (lowest-level) entry. > + * @reuse: the page which is reused for the tail vmemmap pages. > + * @vmemmap_pages: the list head of the vmemmap pages that can be freed. > + */ > +struct vmemmap_rmap_walk { > + void (*rmap_pte)(pte_t *pte, unsigned long addr, > + struct vmemmap_rmap_walk *walk); > + struct page *reuse; > + struct list_head *vmemmap_pages; > +}; > + > +/* > + * The index of the pte page table which is mapped to the tail of the > + * vmemmap page. > + */ > +#define VMEMMAP_TAIL_PAGE_REUSE -1 That is the index/offset from the range to be remapped. See comments below. > + > +static void vmemmap_pte_range(pmd_t *pmd, unsigned long addr, > + unsigned long end, struct vmemmap_rmap_walk *walk) > +{ > + pte_t *pte; > + > + pte = pte_offset_kernel(pmd, addr); > + do { > + BUG_ON(pte_none(*pte)); > + > + if (!walk->reuse) > + walk->reuse = pte_page(pte[VMEMMAP_TAIL_PAGE_REUSE]); It may be just me, but I don't like the pte[-1] here. It certainly does work as designed because we want to remap all pages in the range to the page before the range (at offset -1). But, we do not really validate this 'reuse' page. There is the BUG_ON(pte_none(*pte)) as a sanity check, but we do nothing similar for pte[-1]. Based on the usage for HugeTLB pages, we can be confident that pte[-1] is actually a pte. In discussions with Oscar, you mentioned another possible use for these routines. Don't change anything based on my opinion only. I would like to see what others think as well. > + > + if (walk->rmap_pte) > + walk->rmap_pte(pte, addr, walk); > + } while (pte++, addr += PAGE_SIZE, addr != end); > +} > + > +static void vmemmap_pmd_range(pud_t *pud, unsigned long addr, > + unsigned long end, struct vmemmap_rmap_walk *walk) > +{ > + pmd_t *pmd; > + unsigned long next; > + > + pmd = pmd_offset(pud, addr); > + do { > + BUG_ON(pmd_none(*pmd)); > + > + next = pmd_addr_end(addr, end); > + vmemmap_pte_range(pmd, addr, next, walk); > + } while (pmd++, addr = next, addr != end); > +} > + > +static void vmemmap_pud_range(p4d_t *p4d, unsigned long addr, > + unsigned long end, struct vmemmap_rmap_walk *walk) > +{ > + pud_t *pud; > + unsigned long next; > + > + pud = pud_offset(p4d, addr); > + do { > + BUG_ON(pud_none(*pud)); > + > + next = pud_addr_end(addr, end); > + vmemmap_pmd_range(pud, addr, next, walk); > + } while (pud++, addr = next, addr != end); > +} > + > +static void vmemmap_p4d_range(pgd_t *pgd, unsigned long addr, > + unsigned long end, struct vmemmap_rmap_walk *walk) > +{ > + p4d_t *p4d; > + unsigned long next; > + > + p4d = p4d_offset(pgd, addr); > + do { > + BUG_ON(p4d_none(*p4d)); > + > + next = p4d_addr_end(addr, end); > + vmemmap_pud_range(p4d, addr, next, walk); > + } while (p4d++, addr = next, addr != end); > +} > + > +static void vmemmap_remap_range(unsigned long start, unsigned long end, > + struct vmemmap_rmap_walk *walk) > +{ > + unsigned long addr = start; > + unsigned long next; > + pgd_t *pgd; > + > + VM_BUG_ON(!IS_ALIGNED(start, PAGE_SIZE)); > + VM_BUG_ON(!IS_ALIGNED(end, PAGE_SIZE)); > + > + pgd = pgd_offset_k(addr); > + do { > + BUG_ON(pgd_none(*pgd)); > + > + next = pgd_addr_end(addr, end); > + vmemmap_p4d_range(pgd, addr, next, walk); > + } while (pgd++, addr = next, addr != end); > + > + flush_tlb_kernel_range(start, end); > +} > + > +/* > + * Free a vmemmap page. A vmemmap page can be allocated from the memblock > + * allocator or buddy allocator. If the PG_reserved flag is set, it means > + * that it allocated from the memblock allocator, just free it via the > + * free_bootmem_page(). Otherwise, use __free_page(). > + */ > +static inline void free_vmemmap_page(struct page *page) > +{ > + if (PageReserved(page)) > + free_bootmem_page(page); > + else > + __free_page(page); > +} > + > +/* Free a list of the vmemmap pages */ > +static void free_vmemmap_page_list(struct list_head *list) > +{ > + struct page *page, *next; > + > + list_for_each_entry_safe(page, next, list, lru) { > + list_del(&page->lru); > + free_vmemmap_page(page); > + } > +} > + > +static void vmemmap_remap_reuse_pte(pte_t *pte, unsigned long addr, > + struct vmemmap_rmap_walk *walk) See vmemmap_remap_reuse rename suggestion below. I would suggest reuse be dropped from the name here and just be called 'vmemmap_remap_pte'. > +{ > + /* > + * Make the tail pages are mapped with read-only to catch > + * illegal write operation to the tail pages. > + */ > + pgprot_t pgprot = PAGE_KERNEL_RO; > + pte_t entry = mk_pte(walk->reuse, pgprot); > + struct page *page; > + > + page = pte_page(*pte); > + list_add(&page->lru, walk->vmemmap_pages); > + > + set_pte_at(&init_mm, addr, pte, entry); > +} > + > +/** > + * vmemmap_remap_reuse - remap the vmemmap virtual address range My original commnet here was: Not sure if the word '_reuse' is best in this function name. To me, the name implies this routine will reuse vmemmap pages. Perhaps, it makes more sense to rename as 'vmemmap_remap_free'? It will first remap, then free vmemmap. But, then I looked at the code above and perhaps you are using the word '_reuse' because the page before the range will be reused? The vmemmap page at offset VMEMMAP_TAIL_PAGE_REUSE (-1). > + * [start, start + size) to the page which > + * [start - PAGE_SIZE, start) is mapped. > + * @start: start address of the vmemmap virtual address range > + * @end: size of the vmemmap virtual address range ^^^^ should be @size:
On Wed, Dec 16, 2020 at 02:08:30PM -0800, Mike Kravetz wrote: > > + * vmemmap_rmap_walk - walk vmemmap page table > > I am not sure if 'rmap' should be part of these names. rmap today is mostly > about reverse mapping lookup. Did you use rmap for 'remap', or because this > code is patterned after the page table walking rmap code? Just think the > naming could cause some confusion. I also had the same feeling about the 'rmap' usage. > > + > > +static void vmemmap_pte_range(pmd_t *pmd, unsigned long addr, > > + unsigned long end, struct vmemmap_rmap_walk *walk) > > +{ > > + pte_t *pte; > > + > > + pte = pte_offset_kernel(pmd, addr); > > + do { > > + BUG_ON(pte_none(*pte)); > > + > > + if (!walk->reuse) > > + walk->reuse = pte_page(pte[VMEMMAP_TAIL_PAGE_REUSE]); > > It may be just me, but I don't like the pte[-1] here. It certainly does work > as designed because we want to remap all pages in the range to the page before > the range (at offset -1). But, we do not really validate this 'reuse' page. > There is the BUG_ON(pte_none(*pte)) as a sanity check, but we do nothing similar > for pte[-1]. Based on the usage for HugeTLB pages, we can be confident that > pte[-1] is actually a pte. In discussions with Oscar, you mentioned another > possible use for these routines. Without giving it much of a thought, I guess we could duplicate the BUG_ON for the pte outside the loop, and add a new one for pte[-1]. Also, since walk->reuse seems to not change once it is set, we can take it outside the loop? e.g: pte *pte; pte = pte_offset_kernel(pmd, addr); BUG_ON(pte_none(*pte)); BUG_ON(pte_none(pte[VMEMMAP_TAIL_PAGE_REUSE])); walk->reuse = pte_page(pte[VMEMMAP_TAIL_PAGE_REUSE]); do { .... } while... Or I am not sure whether we want to keep it inside the loop in case future cases change walk->reuse during the operation. But to be honest, I do not think it is realistic of all future possible uses of this, so I would rather keep it simple for now.
On 12/16/20 2:25 PM, Oscar Salvador wrote: > On Wed, Dec 16, 2020 at 02:08:30PM -0800, Mike Kravetz wrote: >>> + * vmemmap_rmap_walk - walk vmemmap page table >>> + >>> +static void vmemmap_pte_range(pmd_t *pmd, unsigned long addr, >>> + unsigned long end, struct vmemmap_rmap_walk *walk) >>> +{ >>> + pte_t *pte; >>> + >>> + pte = pte_offset_kernel(pmd, addr); >>> + do { >>> + BUG_ON(pte_none(*pte)); >>> + >>> + if (!walk->reuse) >>> + walk->reuse = pte_page(pte[VMEMMAP_TAIL_PAGE_REUSE]); >> >> It may be just me, but I don't like the pte[-1] here. It certainly does work >> as designed because we want to remap all pages in the range to the page before >> the range (at offset -1). But, we do not really validate this 'reuse' page. >> There is the BUG_ON(pte_none(*pte)) as a sanity check, but we do nothing similar >> for pte[-1]. Based on the usage for HugeTLB pages, we can be confident that >> pte[-1] is actually a pte. In discussions with Oscar, you mentioned another >> possible use for these routines. > > Without giving it much of a thought, I guess we could duplicate the > BUG_ON for the pte outside the loop, and add a new one for pte[-1]. > Also, since walk->reuse seems to not change once it is set, we can take > it outside the loop? e.g: > > pte *pte; > > pte = pte_offset_kernel(pmd, addr); > BUG_ON(pte_none(*pte)); > BUG_ON(pte_none(pte[VMEMMAP_TAIL_PAGE_REUSE])); > walk->reuse = pte_page(pte[VMEMMAP_TAIL_PAGE_REUSE]); > do { > .... > } while... > > Or I am not sure whether we want to keep it inside the loop in case > future cases change walk->reuse during the operation. > But to be honest, I do not think it is realistic of all future possible > uses of this, so I would rather keep it simple for now. I was thinking about possibly passing the 'reuse' address as another parameter to vmemmap_remap_reuse(). We could add this addr to the vmemmap_rmap_walk struct and set walk->reuse when we get to the pte for that address. Of course this would imply that the addr would need to be part of the range. Ideally, we would walk the page table to get to the reuse page. My concern was not explicitly about adding the BUG_ON. In more general use, *pte could be the first entry on a pte page. And, then pte[-1] may not even be a pte. Again, I don't think this matters for the current HugeTLB use case. Just a little concerned if code is put to use for other purposes.
On Thu, Dec 17, 2020 at 6:08 AM Mike Kravetz <mike.kravetz@oracle.com> wrote: > > On 12/13/20 7:45 AM, Muchun Song wrote: > > Every HugeTLB has more than one struct page structure. We __know__ that > > we only use the first 4(HUGETLB_CGROUP_MIN_ORDER) struct page structures > > to store metadata associated with each HugeTLB. > > > > There are a lot of struct page structures associated with each HugeTLB > > page. For tail pages, the value of compound_head is the same. So we can > > reuse first page of tail page structures. We map the virtual addresses > > of the remaining pages of tail page structures to the first tail page > > struct, and then free these page frames. Therefore, we need to reserve > > two pages as vmemmap areas. > > > > When we allocate a HugeTLB page from the buddy, we can free some vmemmap > > pages associated with each HugeTLB page. It is more appropriate to do it > > in the prep_new_huge_page(). > > > > The free_vmemmap_pages_per_hpage(), which indicates how many vmemmap > > pages associated with a HugeTLB page can be freed, returns zero for > > now, which means the feature is disabled. We will enable it once all > > the infrastructure is there. > > > > Signed-off-by: Muchun Song <songmuchun@bytedance.com> > > --- > > include/linux/bootmem_info.h | 27 +++++- > > include/linux/mm.h | 2 + > > mm/Makefile | 1 + > > mm/hugetlb.c | 3 + > > mm/hugetlb_vmemmap.c | 209 +++++++++++++++++++++++++++++++++++++++++++ > > mm/hugetlb_vmemmap.h | 20 +++++ > > mm/sparse-vmemmap.c | 170 +++++++++++++++++++++++++++++++++++ > > 7 files changed, 431 insertions(+), 1 deletion(-) > > create mode 100644 mm/hugetlb_vmemmap.c > > create mode 100644 mm/hugetlb_vmemmap.h > > > diff --git a/mm/sparse-vmemmap.c b/mm/sparse-vmemmap.c > > index 16183d85a7d5..78c527617e8d 100644 > > --- a/mm/sparse-vmemmap.c > > +++ b/mm/sparse-vmemmap.c > > @@ -27,8 +27,178 @@ > > #include <linux/spinlock.h> > > #include <linux/vmalloc.h> > > #include <linux/sched.h> > > +#include <linux/pgtable.h> > > +#include <linux/bootmem_info.h> > > + > > #include <asm/dma.h> > > #include <asm/pgalloc.h> > > +#include <asm/tlbflush.h> > > + > > +/* > > + * vmemmap_rmap_walk - walk vmemmap page table > > I am not sure if 'rmap' should be part of these names. rmap today is mostly > about reverse mapping lookup. Did you use rmap for 'remap', or because this > code is patterned after the page table walking rmap code? Just think the > naming could cause some confusion. Yeah. I should use "remap" to avoid confusion. > > > + * > > + * @rmap_pte: called for each non-empty PTE (lowest-level) entry. > > + * @reuse: the page which is reused for the tail vmemmap pages. > > + * @vmemmap_pages: the list head of the vmemmap pages that can be freed. > > + */ > > +struct vmemmap_rmap_walk { > > + void (*rmap_pte)(pte_t *pte, unsigned long addr, > > + struct vmemmap_rmap_walk *walk); > > + struct page *reuse; > > + struct list_head *vmemmap_pages; > > +}; > > + > > +/* > > + * The index of the pte page table which is mapped to the tail of the > > + * vmemmap page. > > + */ > > +#define VMEMMAP_TAIL_PAGE_REUSE -1 > > That is the index/offset from the range to be remapped. See comments below. You are right. I need to update the comment. > > > + > > +static void vmemmap_pte_range(pmd_t *pmd, unsigned long addr, > > + unsigned long end, struct vmemmap_rmap_walk *walk) > > +{ > > + pte_t *pte; > > + > > + pte = pte_offset_kernel(pmd, addr); > > + do { > > + BUG_ON(pte_none(*pte)); > > + > > + if (!walk->reuse) > > + walk->reuse = pte_page(pte[VMEMMAP_TAIL_PAGE_REUSE]); > > It may be just me, but I don't like the pte[-1] here. It certainly does work > as designed because we want to remap all pages in the range to the page before > the range (at offset -1). But, we do not really validate this 'reuse' page. > There is the BUG_ON(pte_none(*pte)) as a sanity check, but we do nothing similar > for pte[-1]. Based on the usage for HugeTLB pages, we can be confident that > pte[-1] is actually a pte. In discussions with Oscar, you mentioned another > possible use for these routines. Yeah, we should add a BUG_ON for pte[-1]. > > Don't change anything based on my opinion only. I would like to see what > others think as well. > > > + > > + if (walk->rmap_pte) > > + walk->rmap_pte(pte, addr, walk); > > + } while (pte++, addr += PAGE_SIZE, addr != end); > > +} > > + > > +static void vmemmap_pmd_range(pud_t *pud, unsigned long addr, > > + unsigned long end, struct vmemmap_rmap_walk *walk) > > +{ > > + pmd_t *pmd; > > + unsigned long next; > > + > > + pmd = pmd_offset(pud, addr); > > + do { > > + BUG_ON(pmd_none(*pmd)); > > + > > + next = pmd_addr_end(addr, end); > > + vmemmap_pte_range(pmd, addr, next, walk); > > + } while (pmd++, addr = next, addr != end); > > +} > > + > > +static void vmemmap_pud_range(p4d_t *p4d, unsigned long addr, > > + unsigned long end, struct vmemmap_rmap_walk *walk) > > +{ > > + pud_t *pud; > > + unsigned long next; > > + > > + pud = pud_offset(p4d, addr); > > + do { > > + BUG_ON(pud_none(*pud)); > > + > > + next = pud_addr_end(addr, end); > > + vmemmap_pmd_range(pud, addr, next, walk); > > + } while (pud++, addr = next, addr != end); > > +} > > + > > +static void vmemmap_p4d_range(pgd_t *pgd, unsigned long addr, > > + unsigned long end, struct vmemmap_rmap_walk *walk) > > +{ > > + p4d_t *p4d; > > + unsigned long next; > > + > > + p4d = p4d_offset(pgd, addr); > > + do { > > + BUG_ON(p4d_none(*p4d)); > > + > > + next = p4d_addr_end(addr, end); > > + vmemmap_pud_range(p4d, addr, next, walk); > > + } while (p4d++, addr = next, addr != end); > > +} > > + > > +static void vmemmap_remap_range(unsigned long start, unsigned long end, > > + struct vmemmap_rmap_walk *walk) > > +{ > > + unsigned long addr = start; > > + unsigned long next; > > + pgd_t *pgd; > > + > > + VM_BUG_ON(!IS_ALIGNED(start, PAGE_SIZE)); > > + VM_BUG_ON(!IS_ALIGNED(end, PAGE_SIZE)); > > + > > + pgd = pgd_offset_k(addr); > > + do { > > + BUG_ON(pgd_none(*pgd)); > > + > > + next = pgd_addr_end(addr, end); > > + vmemmap_p4d_range(pgd, addr, next, walk); > > + } while (pgd++, addr = next, addr != end); > > + > > + flush_tlb_kernel_range(start, end); > > +} > > + > > +/* > > + * Free a vmemmap page. A vmemmap page can be allocated from the memblock > > + * allocator or buddy allocator. If the PG_reserved flag is set, it means > > + * that it allocated from the memblock allocator, just free it via the > > + * free_bootmem_page(). Otherwise, use __free_page(). > > + */ > > +static inline void free_vmemmap_page(struct page *page) > > +{ > > + if (PageReserved(page)) > > + free_bootmem_page(page); > > + else > > + __free_page(page); > > +} > > + > > +/* Free a list of the vmemmap pages */ > > +static void free_vmemmap_page_list(struct list_head *list) > > +{ > > + struct page *page, *next; > > + > > + list_for_each_entry_safe(page, next, list, lru) { > > + list_del(&page->lru); > > + free_vmemmap_page(page); > > + } > > +} > > + > > +static void vmemmap_remap_reuse_pte(pte_t *pte, unsigned long addr, > > + struct vmemmap_rmap_walk *walk) > > See vmemmap_remap_reuse rename suggestion below. I would suggest reuse > be dropped from the name here and just be called 'vmemmap_remap_pte'. OK. Will do that. > > > +{ > > + /* > > + * Make the tail pages are mapped with read-only to catch > > + * illegal write operation to the tail pages. > > + */ > > + pgprot_t pgprot = PAGE_KERNEL_RO; > > + pte_t entry = mk_pte(walk->reuse, pgprot); > > + struct page *page; > > + > > + page = pte_page(*pte); > > + list_add(&page->lru, walk->vmemmap_pages); > > + > > + set_pte_at(&init_mm, addr, pte, entry); > > +} > > + > > +/** > > + * vmemmap_remap_reuse - remap the vmemmap virtual address range > > My original commnet here was: > > Not sure if the word '_reuse' is best in this function name. To me, the name > implies this routine will reuse vmemmap pages. Perhaps, it makes more sense > to rename as 'vmemmap_remap_free'? It will first remap, then free vmemmap. The vmemmap_remap_free is also a good name to me. In the next patch, we can use vmemmap_remap_alloc for allocating vmemmap pages. Looks very symmetrical. :-) Thanks Mike. > > But, then I looked at the code above and perhaps you are using the word > '_reuse' because the page before the range will be reused? The vmemmap Yeah. You are right. > page at offset VMEMMAP_TAIL_PAGE_REUSE (-1). > > > + * [start, start + size) to the page which > > + * [start - PAGE_SIZE, start) is mapped. > > + * @start: start address of the vmemmap virtual address range > > + * @end: size of the vmemmap virtual address range > > ^^^^ should be @size: Oh, Yeah. Forgot to update it. Thanks. > > -- > Mike Kravetz > > > + */ > > +void vmemmap_remap_reuse(unsigned long start, unsigned long size) > > +{ > > + unsigned long end = start + size; > > + LIST_HEAD(vmemmap_pages); > > + > > + struct vmemmap_rmap_walk walk = { > > + .rmap_pte = vmemmap_remap_reuse_pte, > > + .vmemmap_pages = &vmemmap_pages, > > + }; > > + > > + vmemmap_remap_range(start, end, &walk); > > + free_vmemmap_page_list(&vmemmap_pages); > > +} > > > > /* > > * Allocate a block of memory to be used to back the virtual memory map > >
On Thu, Dec 17, 2020 at 6:52 AM Mike Kravetz <mike.kravetz@oracle.com> wrote: > > On 12/16/20 2:25 PM, Oscar Salvador wrote: > > On Wed, Dec 16, 2020 at 02:08:30PM -0800, Mike Kravetz wrote: > >>> + * vmemmap_rmap_walk - walk vmemmap page table > >>> + > >>> +static void vmemmap_pte_range(pmd_t *pmd, unsigned long addr, > >>> + unsigned long end, struct vmemmap_rmap_walk *walk) > >>> +{ > >>> + pte_t *pte; > >>> + > >>> + pte = pte_offset_kernel(pmd, addr); > >>> + do { > >>> + BUG_ON(pte_none(*pte)); > >>> + > >>> + if (!walk->reuse) > >>> + walk->reuse = pte_page(pte[VMEMMAP_TAIL_PAGE_REUSE]); > >> > >> It may be just me, but I don't like the pte[-1] here. It certainly does work > >> as designed because we want to remap all pages in the range to the page before > >> the range (at offset -1). But, we do not really validate this 'reuse' page. > >> There is the BUG_ON(pte_none(*pte)) as a sanity check, but we do nothing similar > >> for pte[-1]. Based on the usage for HugeTLB pages, we can be confident that > >> pte[-1] is actually a pte. In discussions with Oscar, you mentioned another > >> possible use for these routines. > > > > Without giving it much of a thought, I guess we could duplicate the > > BUG_ON for the pte outside the loop, and add a new one for pte[-1]. > > Also, since walk->reuse seems to not change once it is set, we can take > > it outside the loop? e.g: > > > > pte *pte; > > > > pte = pte_offset_kernel(pmd, addr); > > BUG_ON(pte_none(*pte)); > > BUG_ON(pte_none(pte[VMEMMAP_TAIL_PAGE_REUSE])); > > walk->reuse = pte_page(pte[VMEMMAP_TAIL_PAGE_REUSE]); > > do { > > .... > > } while... > > > > Or I am not sure whether we want to keep it inside the loop in case > > future cases change walk->reuse during the operation. > > But to be honest, I do not think it is realistic of all future possible > > uses of this, so I would rather keep it simple for now. > > I was thinking about possibly passing the 'reuse' address as another parameter > to vmemmap_remap_reuse(). We could add this addr to the vmemmap_rmap_walk > struct and set walk->reuse when we get to the pte for that address. Of > course this would imply that the addr would need to be part of the range. Maybe adding another one parameter is unnecessary. How about doing this in the vmemmap_remap_reuse? The 'reuse' address just is start + PAGE_SIZE. void vmemmap_remap_free(unsigned long start, unsigned long size) { unsigned long end = start + size; unsigned long reuse_addr = start + PAGE_SIZE; LIST_HEAD(vmemmap_pages); struct vmemmap_remap_walk walk = { .remap_pte = vmemmap_remap_pte, .vmemmap_pages = &vmemmap_pages, .reuse_addr = reuse_addr. }; } > > Ideally, we would walk the page table to get to the reuse page. My concern > was not explicitly about adding the BUG_ON. In more general use, *pte could > be the first entry on a pte page. And, then pte[-1] may not even be a pte. > > Again, I don't think this matters for the current HugeTLB use case. Just a > little concerned if code is put to use for other purposes. > -- > Mike Kravetz
On Thu, Dec 17, 2020 at 2:54 PM Muchun Song <songmuchun@bytedance.com> wrote: > > On Thu, Dec 17, 2020 at 6:52 AM Mike Kravetz <mike.kravetz@oracle.com> wrote: > > > > On 12/16/20 2:25 PM, Oscar Salvador wrote: > > > On Wed, Dec 16, 2020 at 02:08:30PM -0800, Mike Kravetz wrote: > > >>> + * vmemmap_rmap_walk - walk vmemmap page table > > >>> + > > >>> +static void vmemmap_pte_range(pmd_t *pmd, unsigned long addr, > > >>> + unsigned long end, struct vmemmap_rmap_walk *walk) > > >>> +{ > > >>> + pte_t *pte; > > >>> + > > >>> + pte = pte_offset_kernel(pmd, addr); > > >>> + do { > > >>> + BUG_ON(pte_none(*pte)); > > >>> + > > >>> + if (!walk->reuse) > > >>> + walk->reuse = pte_page(pte[VMEMMAP_TAIL_PAGE_REUSE]); > > >> > > >> It may be just me, but I don't like the pte[-1] here. It certainly does work > > >> as designed because we want to remap all pages in the range to the page before > > >> the range (at offset -1). But, we do not really validate this 'reuse' page. > > >> There is the BUG_ON(pte_none(*pte)) as a sanity check, but we do nothing similar > > >> for pte[-1]. Based on the usage for HugeTLB pages, we can be confident that > > >> pte[-1] is actually a pte. In discussions with Oscar, you mentioned another > > >> possible use for these routines. > > > > > > Without giving it much of a thought, I guess we could duplicate the > > > BUG_ON for the pte outside the loop, and add a new one for pte[-1]. > > > Also, since walk->reuse seems to not change once it is set, we can take > > > it outside the loop? e.g: > > > > > > pte *pte; > > > > > > pte = pte_offset_kernel(pmd, addr); > > > BUG_ON(pte_none(*pte)); > > > BUG_ON(pte_none(pte[VMEMMAP_TAIL_PAGE_REUSE])); > > > walk->reuse = pte_page(pte[VMEMMAP_TAIL_PAGE_REUSE]); > > > do { > > > .... > > > } while... > > > > > > Or I am not sure whether we want to keep it inside the loop in case > > > future cases change walk->reuse during the operation. > > > But to be honest, I do not think it is realistic of all future possible > > > uses of this, so I would rather keep it simple for now. > > > > I was thinking about possibly passing the 'reuse' address as another parameter > > to vmemmap_remap_reuse(). We could add this addr to the vmemmap_rmap_walk > > struct and set walk->reuse when we get to the pte for that address. Of > > course this would imply that the addr would need to be part of the range. > > Maybe adding another one parameter is unnecessary. How about doing > this in the vmemmap_remap_reuse? > > The 'reuse' address just is start + PAGE_SIZE. > > void vmemmap_remap_free(unsigned long start, unsigned long size) > { > unsigned long end = start + size; > unsigned long reuse_addr = start + PAGE_SIZE; ^^^ Here is "-" Sorry. > LIST_HEAD(vmemmap_pages); > > struct vmemmap_remap_walk walk = { > .remap_pte = vmemmap_remap_pte, > .vmemmap_pages = &vmemmap_pages, > .reuse_addr = reuse_addr. > }; > > } > > > > > Ideally, we would walk the page table to get to the reuse page. My concern > > was not explicitly about adding the BUG_ON. In more general use, *pte could > > be the first entry on a pte page. And, then pte[-1] may not even be a pte. > > > > Again, I don't think this matters for the current HugeTLB use case. Just a > > little concerned if code is put to use for other purposes. > > -- > > Mike Kravetz > > > > -- > Yours, > Muchun
diff --git a/include/linux/bootmem_info.h b/include/linux/bootmem_info.h index 4ed6dee1adc9..4c80b7be1771 100644 --- a/include/linux/bootmem_info.h +++ b/include/linux/bootmem_info.h @@ -2,7 +2,7 @@ #ifndef __LINUX_BOOTMEM_INFO_H #define __LINUX_BOOTMEM_INFO_H -#include <linux/mmzone.h> +#include <linux/mm.h> /* * Types for free bootmem stored in page->lru.next. These have to be in @@ -22,6 +22,27 @@ void __init register_page_bootmem_info_node(struct pglist_data *pgdat); void get_page_bootmem(unsigned long info, struct page *page, unsigned long type); void put_page_bootmem(struct page *page); + +/* + * Any memory allocated via the memblock allocator and not via the + * buddy will be marked reserved already in the memmap. For those + * pages, we can call this function to free it to buddy allocator. + */ +static inline void free_bootmem_page(struct page *page) +{ + unsigned long magic = (unsigned long)page->freelist; + + /* + * The reserve_bootmem_region sets the reserved flag on bootmem + * pages. + */ + VM_WARN_ON(page_ref_count(page) != 2); + + if (magic == SECTION_INFO || magic == MIX_SECTION_INFO) + put_page_bootmem(page); + else + VM_WARN_ON(1); +} #else static inline void register_page_bootmem_info_node(struct pglist_data *pgdat) { @@ -35,6 +56,10 @@ static inline void get_page_bootmem(unsigned long info, struct page *page, unsigned long type) { } + +static inline void free_bootmem_page(struct page *page) +{ +} #endif #endif /* __LINUX_BOOTMEM_INFO_H */ diff --git a/include/linux/mm.h b/include/linux/mm.h index eabe7d9f80d8..ab02e405a979 100644 --- a/include/linux/mm.h +++ b/include/linux/mm.h @@ -3005,6 +3005,8 @@ static inline void print_vma_addr(char *prefix, unsigned long rip) } #endif +void vmemmap_remap_reuse(unsigned long start, unsigned long size); + void *sparse_buffer_alloc(unsigned long size); struct page * __populate_section_memmap(unsigned long pfn, unsigned long nr_pages, int nid, struct vmem_altmap *altmap); diff --git a/mm/Makefile b/mm/Makefile index ed4b88fa0f5e..056801d8daae 100644 --- a/mm/Makefile +++ b/mm/Makefile @@ -71,6 +71,7 @@ obj-$(CONFIG_FRONTSWAP) += frontswap.o obj-$(CONFIG_ZSWAP) += zswap.o obj-$(CONFIG_HAS_DMA) += dmapool.o obj-$(CONFIG_HUGETLBFS) += hugetlb.o +obj-$(CONFIG_HUGETLB_PAGE_FREE_VMEMMAP) += hugetlb_vmemmap.o obj-$(CONFIG_NUMA) += mempolicy.o obj-$(CONFIG_SPARSEMEM) += sparse.o obj-$(CONFIG_SPARSEMEM_VMEMMAP) += sparse-vmemmap.o diff --git a/mm/hugetlb.c b/mm/hugetlb.c index 1f3bf1710b66..140135fc8113 100644 --- a/mm/hugetlb.c +++ b/mm/hugetlb.c @@ -42,6 +42,7 @@ #include <linux/userfaultfd_k.h> #include <linux/page_owner.h> #include "internal.h" +#include "hugetlb_vmemmap.h" int hugetlb_max_hstate __read_mostly; unsigned int default_hstate_idx; @@ -1497,6 +1498,8 @@ void free_huge_page(struct page *page) static void prep_new_huge_page(struct hstate *h, struct page *page, int nid) { + free_huge_page_vmemmap(h, page); + INIT_LIST_HEAD(&page->lru); set_compound_page_dtor(page, HUGETLB_PAGE_DTOR); set_hugetlb_cgroup(page, NULL); diff --git a/mm/hugetlb_vmemmap.c b/mm/hugetlb_vmemmap.c new file mode 100644 index 000000000000..5a714bd60d6b --- /dev/null +++ b/mm/hugetlb_vmemmap.c @@ -0,0 +1,209 @@ +// SPDX-License-Identifier: GPL-2.0 +/* + * Free some vmemmap pages of HugeTLB + * + * Copyright (c) 2020, Bytedance. All rights reserved. + * + * Author: Muchun Song <songmuchun@bytedance.com> + * + * The struct page structures (page structs) are used to describe a physical + * page frame. By default, there is a one-to-one mapping from a page frame to + * it's corresponding page struct. + * + * The HugeTLB pages consist of multiple base page size pages and is supported + * by many architectures. See hugetlbpage.rst in the Documentation directory + * for more details. On the x86-64 architecture, HugeTLB pages of size 2MB and + * 1GB are currently supported. Since the base page size on x86 is 4KB, a 2MB + * HugeTLB page consists of 512 base pages and a 1GB HugeTLB page consists of + * 4096 base pages. For each base page, there is a corresponding page struct. + * + * Within the HugeTLB subsystem, only the first 4 page structs are used to + * contain unique information about a HugeTLB page. HUGETLB_CGROUP_MIN_ORDER + * provides this upper limit. The only 'useful' information in the remaining + * page structs is the compound_head field, and this field is the same for all + * tail pages. + * + * By removing redundant page structs for HugeTLB pages, memory can returned to + * the buddy allocator for other uses. + * + * Different architectures support different HugeTLB pages. For example, the + * following table is the HugeTLB page size supported by x86 and arm64 + * architectures. Becasue arm64 supports 4k, 16k, and 64k base pages and + * supports contiguous entries, so it supports many kinds of sizes of HugeTLB + * page. + * + * +--------------+-----------+-----------------------------------------------+ + * | Architecture | Page Size | HugeTLB Page Size | + * +--------------+-----------+-----------+-----------+-----------+-----------+ + * | x86-64 | 4KB | 2MB | 1GB | | | + * +--------------+-----------+-----------+-----------+-----------+-----------+ + * | | 4KB | 64KB | 2MB | 32MB | 1GB | + * | +-----------+-----------+-----------+-----------+-----------+ + * | arm64 | 16KB | 2MB | 32MB | 1GB | | + * | +-----------+-----------+-----------+-----------+-----------+ + * | | 64KB | 2MB | 512MB | 16GB | | + * +--------------+-----------+-----------+-----------+-----------+-----------+ + * + * When the system boot up, every HugeTLB page has more than one struct page + * structs whose size is (unit: pages): + * + * struct_size = HugeTLB_Size / PAGE_SIZE * sizeof(struct page) / PAGE_SIZE + * + * Where HugeTLB_Size is the size of the HugeTLB page. We know that the size + * of the HugeTLB page is always n times PAGE_SIZE. So we can get the following + * relationship. + * + * HugeTLB_Size = n * PAGE_SIZE + * + * Then, + * + * struct_size = n * PAGE_SIZE / PAGE_SIZE * sizeof(struct page) / PAGE_SIZE + * = n * sizeof(struct page) / PAGE_SIZE + * + * We can use huge mapping at the pud/pmd level for the HugeTLB page. + * + * For the HugeTLB page of the pmd level mapping, then + * + * struct_size = n * sizeof(struct page) / PAGE_SIZE + * = PAGE_SIZE / sizeof(pte_t) * sizeof(struct page) / PAGE_SIZE + * = sizeof(struct page) / sizeof(pte_t) + * = 64 / 8 + * = 8 (pages) + * + * Where n is how many pte entries which one page can contains. So the value of + * n is (PAGE_SIZE / sizeof(pte_t)). + * + * This optimization only supports 64-bit system, so the value of sizeof(pte_t) + * is 8. And this optimization also applicable only when the size of struct page + * is a power of two. In most cases, the size of struct page is 64 (e.g. x86-64 + * and arm64). So if we use pmd level mapping for a HugeTLB page, the size of + * struct page structs of it is 8 pages whose size depends on the size of the + * base page. + * + * For the HugeTLB page of the pud level mapping, then + * + * struct_size = PAGE_SIZE / sizeof(pmd_t) * struct_size(pmd) + * = PAGE_SIZE / 8 * 8 (pages) + * = PAGE_SIZE (pages) + * + * Where the struct_size(pmd) is the size of the struct page structs of a + * HugeTLB page of the pmd level mapping. + * + * Next, we take the pmd level mapping of the HugeTLB page as an example to + * show the internal implementation of this optimization. There are 8 pages + * struct page structs associated with a HugeTLB page which is pmd mapped. + * + * Here is how things look before optimization. + * + * HugeTLB struct pages(8 pages) page frame(8 pages) + * +-----------+ ---virt_to_page---> +-----------+ mapping to +-----------+ + * | | | 0 | -------------> | 0 | + * | | +-----------+ +-----------+ + * | | | 1 | -------------> | 1 | + * | | +-----------+ +-----------+ + * | | | 2 | -------------> | 2 | + * | | +-----------+ +-----------+ + * | | | 3 | -------------> | 3 | + * | | +-----------+ +-----------+ + * | | | 4 | -------------> | 4 | + * | PMD | +-----------+ +-----------+ + * | level | | 5 | -------------> | 5 | + * | mapping | +-----------+ +-----------+ + * | | | 6 | -------------> | 6 | + * | | +-----------+ +-----------+ + * | | | 7 | -------------> | 7 | + * | | +-----------+ +-----------+ + * | | + * | | + * | | + * +-----------+ + * + * The value of page->compound_head is the same for all tail pages. The first + * page of page structs (page 0) associated with the HugeTLB page contains the 4 + * page structs necessary to describe the HugeTLB. The only use of the remaining + * pages of page structs (page 1 to page 7) is to point to page->compound_head. + * Therefore, we can remap pages 2 to 7 to page 1. Only 2 pages of page structs + * will be used for each HugeTLB page. This will allow us to free the remaining + * 6 pages to the buddy allocator. + * + * Here is how things look after remapping. + * + * HugeTLB struct pages(8 pages) page frame(8 pages) + * +-----------+ ---virt_to_page---> +-----------+ mapping to +-----------+ + * | | | 0 | -------------> | 0 | + * | | +-----------+ +-----------+ + * | | | 1 | -------------> | 1 | + * | | +-----------+ +-----------+ + * | | | 2 | ----------------^ ^ ^ ^ ^ ^ + * | | +-----------+ | | | | | + * | | | 3 | ------------------+ | | | | + * | | +-----------+ | | | | + * | | | 4 | --------------------+ | | | + * | PMD | +-----------+ | | | + * | level | | 5 | ----------------------+ | | + * | mapping | +-----------+ | | + * | | | 6 | ------------------------+ | + * | | +-----------+ | + * | | | 7 | --------------------------+ + * | | +-----------+ + * | | + * | | + * | | + * +-----------+ + * + * When a HugeTLB is freed to the buddy system, we should allocate 6 pages for + * vmemmap pages and restore the previous mapping relationship. + * + * For the HugeTLB page of the pud level mapping. It is similar to the former. + * We also can use this approach to free (PAGE_SIZE - 2) vmemmap pages. + * + * Apart from the HugeTLB page of the pmd/pud level mapping, some architectures + * (e.g. aarch64) provides a contiguous bit in the translation table entries + * that hints to the MMU to indicate that it is one of a contiguous set of + * entries that can be cached in a single TLB entry. + * + * The contiguous bit is used to increase the mapping size at the pmd and pte + * (last) level. So this type of HugeTLB page can be optimized only when its + * size of the struct page structs is greater than 2 pages. + */ +#define pr_fmt(fmt) "HugeTLB vmemmap: " fmt + +#include "hugetlb_vmemmap.h" + +/* + * There are a lot of struct page structures associated with each HugeTLB page. + * For tail pages, the value of compound_head is the same. So we can reuse first + * page of tail page structures. We map the virtual addresses of the remaining + * pages of tail page structures to the first tail page struct, and then free + * these page frames. Therefore, we need to reserve two pages as vmemmap areas. + */ +#define RESERVE_VMEMMAP_NR 2U +#define RESERVE_VMEMMAP_SIZE (RESERVE_VMEMMAP_NR << PAGE_SHIFT) + +/* + * How many vmemmap pages associated with a HugeTLB page that can be freed + * to the buddy allocator. + * + * Todo: Returns zero for now, which means the feature is disabled. We will + * enable it once all the infrastructure is there. + */ +static inline unsigned int free_vmemmap_pages_per_hpage(struct hstate *h) +{ + return 0; +} + +static inline unsigned long free_vmemmap_pages_size_per_hpage(struct hstate *h) +{ + return (unsigned long)free_vmemmap_pages_per_hpage(h) << PAGE_SHIFT; +} + +void free_huge_page_vmemmap(struct hstate *h, struct page *head) +{ + unsigned long vmemmap_addr = (unsigned long)head; + + if (!free_vmemmap_pages_per_hpage(h)) + return; + + vmemmap_remap_reuse(vmemmap_addr + RESERVE_VMEMMAP_SIZE, + free_vmemmap_pages_size_per_hpage(h)); +} diff --git a/mm/hugetlb_vmemmap.h b/mm/hugetlb_vmemmap.h new file mode 100644 index 000000000000..6923f03534d5 --- /dev/null +++ b/mm/hugetlb_vmemmap.h @@ -0,0 +1,20 @@ +// SPDX-License-Identifier: GPL-2.0 +/* + * Free some vmemmap pages of HugeTLB + * + * Copyright (c) 2020, Bytedance. All rights reserved. + * + * Author: Muchun Song <songmuchun@bytedance.com> + */ +#ifndef _LINUX_HUGETLB_VMEMMAP_H +#define _LINUX_HUGETLB_VMEMMAP_H +#include <linux/hugetlb.h> + +#ifdef CONFIG_HUGETLB_PAGE_FREE_VMEMMAP +void free_huge_page_vmemmap(struct hstate *h, struct page *head); +#else +static inline void free_huge_page_vmemmap(struct hstate *h, struct page *head) +{ +} +#endif /* CONFIG_HUGETLB_PAGE_FREE_VMEMMAP */ +#endif /* _LINUX_HUGETLB_VMEMMAP_H */ diff --git a/mm/sparse-vmemmap.c b/mm/sparse-vmemmap.c index 16183d85a7d5..78c527617e8d 100644 --- a/mm/sparse-vmemmap.c +++ b/mm/sparse-vmemmap.c @@ -27,8 +27,178 @@ #include <linux/spinlock.h> #include <linux/vmalloc.h> #include <linux/sched.h> +#include <linux/pgtable.h> +#include <linux/bootmem_info.h> + #include <asm/dma.h> #include <asm/pgalloc.h> +#include <asm/tlbflush.h> + +/* + * vmemmap_rmap_walk - walk vmemmap page table + * + * @rmap_pte: called for each non-empty PTE (lowest-level) entry. + * @reuse: the page which is reused for the tail vmemmap pages. + * @vmemmap_pages: the list head of the vmemmap pages that can be freed. + */ +struct vmemmap_rmap_walk { + void (*rmap_pte)(pte_t *pte, unsigned long addr, + struct vmemmap_rmap_walk *walk); + struct page *reuse; + struct list_head *vmemmap_pages; +}; + +/* + * The index of the pte page table which is mapped to the tail of the + * vmemmap page. + */ +#define VMEMMAP_TAIL_PAGE_REUSE -1 + +static void vmemmap_pte_range(pmd_t *pmd, unsigned long addr, + unsigned long end, struct vmemmap_rmap_walk *walk) +{ + pte_t *pte; + + pte = pte_offset_kernel(pmd, addr); + do { + BUG_ON(pte_none(*pte)); + + if (!walk->reuse) + walk->reuse = pte_page(pte[VMEMMAP_TAIL_PAGE_REUSE]); + + if (walk->rmap_pte) + walk->rmap_pte(pte, addr, walk); + } while (pte++, addr += PAGE_SIZE, addr != end); +} + +static void vmemmap_pmd_range(pud_t *pud, unsigned long addr, + unsigned long end, struct vmemmap_rmap_walk *walk) +{ + pmd_t *pmd; + unsigned long next; + + pmd = pmd_offset(pud, addr); + do { + BUG_ON(pmd_none(*pmd)); + + next = pmd_addr_end(addr, end); + vmemmap_pte_range(pmd, addr, next, walk); + } while (pmd++, addr = next, addr != end); +} + +static void vmemmap_pud_range(p4d_t *p4d, unsigned long addr, + unsigned long end, struct vmemmap_rmap_walk *walk) +{ + pud_t *pud; + unsigned long next; + + pud = pud_offset(p4d, addr); + do { + BUG_ON(pud_none(*pud)); + + next = pud_addr_end(addr, end); + vmemmap_pmd_range(pud, addr, next, walk); + } while (pud++, addr = next, addr != end); +} + +static void vmemmap_p4d_range(pgd_t *pgd, unsigned long addr, + unsigned long end, struct vmemmap_rmap_walk *walk) +{ + p4d_t *p4d; + unsigned long next; + + p4d = p4d_offset(pgd, addr); + do { + BUG_ON(p4d_none(*p4d)); + + next = p4d_addr_end(addr, end); + vmemmap_pud_range(p4d, addr, next, walk); + } while (p4d++, addr = next, addr != end); +} + +static void vmemmap_remap_range(unsigned long start, unsigned long end, + struct vmemmap_rmap_walk *walk) +{ + unsigned long addr = start; + unsigned long next; + pgd_t *pgd; + + VM_BUG_ON(!IS_ALIGNED(start, PAGE_SIZE)); + VM_BUG_ON(!IS_ALIGNED(end, PAGE_SIZE)); + + pgd = pgd_offset_k(addr); + do { + BUG_ON(pgd_none(*pgd)); + + next = pgd_addr_end(addr, end); + vmemmap_p4d_range(pgd, addr, next, walk); + } while (pgd++, addr = next, addr != end); + + flush_tlb_kernel_range(start, end); +} + +/* + * Free a vmemmap page. A vmemmap page can be allocated from the memblock + * allocator or buddy allocator. If the PG_reserved flag is set, it means + * that it allocated from the memblock allocator, just free it via the + * free_bootmem_page(). Otherwise, use __free_page(). + */ +static inline void free_vmemmap_page(struct page *page) +{ + if (PageReserved(page)) + free_bootmem_page(page); + else + __free_page(page); +} + +/* Free a list of the vmemmap pages */ +static void free_vmemmap_page_list(struct list_head *list) +{ + struct page *page, *next; + + list_for_each_entry_safe(page, next, list, lru) { + list_del(&page->lru); + free_vmemmap_page(page); + } +} + +static void vmemmap_remap_reuse_pte(pte_t *pte, unsigned long addr, + struct vmemmap_rmap_walk *walk) +{ + /* + * Make the tail pages are mapped with read-only to catch + * illegal write operation to the tail pages. + */ + pgprot_t pgprot = PAGE_KERNEL_RO; + pte_t entry = mk_pte(walk->reuse, pgprot); + struct page *page; + + page = pte_page(*pte); + list_add(&page->lru, walk->vmemmap_pages); + + set_pte_at(&init_mm, addr, pte, entry); +} + +/** + * vmemmap_remap_reuse - remap the vmemmap virtual address range + * [start, start + size) to the page which + * [start - PAGE_SIZE, start) is mapped. + * @start: start address of the vmemmap virtual address range + * @end: size of the vmemmap virtual address range + */ +void vmemmap_remap_reuse(unsigned long start, unsigned long size) +{ + unsigned long end = start + size; + LIST_HEAD(vmemmap_pages); + + struct vmemmap_rmap_walk walk = { + .rmap_pte = vmemmap_remap_reuse_pte, + .vmemmap_pages = &vmemmap_pages, + }; + + vmemmap_remap_range(start, end, &walk); + free_vmemmap_page_list(&vmemmap_pages); +} /* * Allocate a block of memory to be used to back the virtual memory map
Every HugeTLB has more than one struct page structure. We __know__ that we only use the first 4(HUGETLB_CGROUP_MIN_ORDER) struct page structures to store metadata associated with each HugeTLB. There are a lot of struct page structures associated with each HugeTLB page. For tail pages, the value of compound_head is the same. So we can reuse first page of tail page structures. We map the virtual addresses of the remaining pages of tail page structures to the first tail page struct, and then free these page frames. Therefore, we need to reserve two pages as vmemmap areas. When we allocate a HugeTLB page from the buddy, we can free some vmemmap pages associated with each HugeTLB page. It is more appropriate to do it in the prep_new_huge_page(). The free_vmemmap_pages_per_hpage(), which indicates how many vmemmap pages associated with a HugeTLB page can be freed, returns zero for now, which means the feature is disabled. We will enable it once all the infrastructure is there. Signed-off-by: Muchun Song <songmuchun@bytedance.com> --- include/linux/bootmem_info.h | 27 +++++- include/linux/mm.h | 2 + mm/Makefile | 1 + mm/hugetlb.c | 3 + mm/hugetlb_vmemmap.c | 209 +++++++++++++++++++++++++++++++++++++++++++ mm/hugetlb_vmemmap.h | 20 +++++ mm/sparse-vmemmap.c | 170 +++++++++++++++++++++++++++++++++++ 7 files changed, 431 insertions(+), 1 deletion(-) create mode 100644 mm/hugetlb_vmemmap.c create mode 100644 mm/hugetlb_vmemmap.h