[v2,07/14] mm/hugetlb_vmemmap: move comment block to Documentation/vm

Commit Message

Joao Martins June 17, 2021, 6:45 p.m. UTC

In preparation for device-dax for using hugetlbfs compound page tail
deduplication technique, move the comment block explanation into a
common place in Documentation/vm.

Cc: Muchun Song <[e-mail]>
Cc: Mike Kravetz <[e-mail]>
Suggested-by: Dan Williams <[e-mail]>
Signed-off-by: Joao Martins <[e-mail]>
---
 Documentation/vm/compound_pagemaps.rst | 170 +++++++++++++++++++++++++
 Documentation/vm/index.rst             |   1 +
 mm/hugetlb_vmemmap.c                   | 162 +----------------------
 3 files changed, 172 insertions(+), 161 deletions(-)
 create mode 100644 Documentation/vm/compound_pagemaps.rst

Comments

Muchun Song June 21, 2021, 1:12 p.m. UTC | #1

On Fri, Jun 18, 2021 at 2:46 AM Joao Martins <[e-mail]> wrote:
>
> In preparation for device-dax for using hugetlbfs compound page tail
> deduplication technique, move the comment block explanation into a
> common place in Documentation/vm.
>
> Cc: Muchun Song <[e-mail]>
> Cc: Mike Kravetz <[e-mail]>
> Suggested-by: Dan Williams <[e-mail]>
> Signed-off-by: Joao Martins <[e-mail]>
> ---
>  Documentation/vm/compound_pagemaps.rst | 170 +++++++++++++++++++++++++
>  Documentation/vm/index.rst             |   1 +
>  mm/hugetlb_vmemmap.c                   | 162 +----------------------
>  3 files changed, 172 insertions(+), 161 deletions(-)
>  create mode 100644 Documentation/vm/compound_pagemaps.rst

IMHO, how about the name of vmemmap_remap.rst? page_frags.rst seems
to tell people it's about the page mapping not its vmemmap mapping.

Thanks.

>
> diff --git a/Documentation/vm/compound_pagemaps.rst b/Documentation/vm/compound_pagemaps.rst
> new file mode 100644
> index 000000000000..6b1af50e8201
> --- /dev/null
> +++ b/Documentation/vm/compound_pagemaps.rst
> @@ -0,0 +1,170 @@
> +.. SPDX-License-Identifier: GPL-2.0
> +
> +.. _commpound_pagemaps:
> +
> +==================================
> +Free some vmemmap pages of HugeTLB
> +==================================
> +
> +The struct page structures (page structs) are used to describe a physical
> +page frame. By default, there is a one-to-one mapping from a page frame to
> +it's corresponding page struct.
> +
> +HugeTLB pages consist of multiple base page size pages and is supported by
> +many architectures. See hugetlbpage.rst in the Documentation directory for
> +more details. On the x86-64 architecture, HugeTLB pages of size 2MB and 1GB
> +are currently supported. Since the base page size on x86 is 4KB, a 2MB
> +HugeTLB page consists of 512 base pages and a 1GB HugeTLB page consists of
> +4096 base pages. For each base page, there is a corresponding page struct.
> +
> +Within the HugeTLB subsystem, only the first 4 page structs are used to
> +contain unique information about a HugeTLB page. __NR_USED_SUBPAGE provides
> +this upper limit. The only 'useful' information in the remaining page structs
> +is the compound_head field, and this field is the same for all tail pages.
> +
> +By removing redundant page structs for HugeTLB pages, memory can be returned
> +to the buddy allocator for other uses.
> +
> +Different architectures support different HugeTLB pages. For example, the
> +following table is the HugeTLB page size supported by x86 and arm64
> +architectures. Because arm64 supports 4k, 16k, and 64k base pages and
> +supports contiguous entries, so it supports many kinds of sizes of HugeTLB
> +page.
> +
> ++--------------+-----------+-----------------------------------------------+
> +| Architecture | Page Size |                HugeTLB Page Size              |
> ++--------------+-----------+-----------+-----------+-----------+-----------+
> +|    x86-64    |    4KB    |    2MB    |    1GB    |           |           |
> ++--------------+-----------+-----------+-----------+-----------+-----------+
> +|              |    4KB    |   64KB    |    2MB    |    32MB   |    1GB    |
> +|              +-----------+-----------+-----------+-----------+-----------+
> +|    arm64     |   16KB    |    2MB    |   32MB    |     1GB   |           |
> +|              +-----------+-----------+-----------+-----------+-----------+
> +|              |   64KB    |    2MB    |  512MB    |    16GB   |           |
> ++--------------+-----------+-----------+-----------+-----------+-----------+
> +
> +When the system boot up, every HugeTLB page has more than one struct page
> +structs which size is (unit: pages):
> +
> +   struct_size = HugeTLB_Size / PAGE_SIZE * sizeof(struct page) / PAGE_SIZE
> +
> +Where HugeTLB_Size is the size of the HugeTLB page. We know that the size
> +of the HugeTLB page is always n times PAGE_SIZE. So we can get the following
> +relationship.
> +
> +   HugeTLB_Size = n * PAGE_SIZE
> +
> +Then,
> +
> +   struct_size = n * PAGE_SIZE / PAGE_SIZE * sizeof(struct page) / PAGE_SIZE
> +               = n * sizeof(struct page) / PAGE_SIZE
> +
> +We can use huge mapping at the pud/pmd level for the HugeTLB page.
> +
> +For the HugeTLB page of the pmd level mapping, then
> +
> +   struct_size = n * sizeof(struct page) / PAGE_SIZE
> +               = PAGE_SIZE / sizeof(pte_t) * sizeof(struct page) / PAGE_SIZE
> +               = sizeof(struct page) / sizeof(pte_t)
> +               = 64 / 8
> +               = 8 (pages)
> +
> +Where n is how many pte entries which one page can contains. So the value of
> +n is (PAGE_SIZE / sizeof(pte_t)).
> +
> +This optimization only supports 64-bit system, so the value of sizeof(pte_t)
> +is 8. And this optimization also applicable only when the size of struct page
> +is a power of two. In most cases, the size of struct page is 64 bytes (e.g.
> +x86-64 and arm64). So if we use pmd level mapping for a HugeTLB page, the
> +size of struct page structs of it is 8 page frames which size depends on the
> +size of the base page.
> +
> +For the HugeTLB page of the pud level mapping, then
> +
> +   struct_size = PAGE_SIZE / sizeof(pmd_t) * struct_size(pmd)
> +               = PAGE_SIZE / 8 * 8 (pages)
> +               = PAGE_SIZE (pages)
> +
> +Where the struct_size(pmd) is the size of the struct page structs of a
> +HugeTLB page of the pmd level mapping.
> +
> +E.g.: A 2MB HugeTLB page on x86_64 consists in 8 page frames while 1GB
> +HugeTLB page consists in 4096.
> +
> +Next, we take the pmd level mapping of the HugeTLB page as an example to
> +show the internal implementation of this optimization. There are 8 pages
> +struct page structs associated with a HugeTLB page which is pmd mapped.
> +
> +Here is how things look before optimization.
> +
> +    HugeTLB                  struct pages(8 pages)         page frame(8 pages)
> + +-----------+ ---virt_to_page---> +-----------+   mapping to   +-----------+
> + |           |                     |     0     | -------------> |     0     |
> + |           |                     +-----------+                +-----------+
> + |           |                     |     1     | -------------> |     1     |
> + |           |                     +-----------+                +-----------+
> + |           |                     |     2     | -------------> |     2     |
> + |           |                     +-----------+                +-----------+
> + |           |                     |     3     | -------------> |     3     |
> + |           |                     +-----------+                +-----------+
> + |           |                     |     4     | -------------> |     4     |
> + |    PMD    |                     +-----------+                +-----------+
> + |   level   |                     |     5     | -------------> |     5     |
> + |  mapping  |                     +-----------+                +-----------+
> + |           |                     |     6     | -------------> |     6     |
> + |           |                     +-----------+                +-----------+
> + |           |                     |     7     | -------------> |     7     |
> + |           |                     +-----------+                +-----------+
> + |           |
> + |           |
> + |           |
> + +-----------+
> +
> +The value of page->compound_head is the same for all tail pages. The first
> +page of page structs (page 0) associated with the HugeTLB page contains the 4
> +page structs necessary to describe the HugeTLB. The only use of the remaining
> +pages of page structs (page 1 to page 7) is to point to page->compound_head.
> +Therefore, we can remap pages 2 to 7 to page 1. Only 2 pages of page structs
> +will be used for each HugeTLB page. This will allow us to free the remaining
> +6 pages to the buddy allocator.
> +
> +Here is how things look after remapping.
> +
> +    HugeTLB                  struct pages(8 pages)         page frame(8 pages)
> + +-----------+ ---virt_to_page---> +-----------+   mapping to   +-----------+
> + |           |                     |     0     | -------------> |     0     |
> + |           |                     +-----------+                +-----------+
> + |           |                     |     1     | -------------> |     1     |
> + |           |                     +-----------+                +-----------+
> + |           |                     |     2     | ----------------^ ^ ^ ^ ^ ^
> + |           |                     +-----------+                   | | | | |
> + |           |                     |     3     | ------------------+ | | | |
> + |           |                     +-----------+                     | | | |
> + |           |                     |     4     | --------------------+ | | |
> + |    PMD    |                     +-----------+                       | | |
> + |   level   |                     |     5     | ----------------------+ | |
> + |  mapping  |                     +-----------+                         | |
> + |           |                     |     6     | ------------------------+ |
> + |           |                     +-----------+                           |
> + |           |                     |     7     | --------------------------+
> + |           |                     +-----------+
> + |           |
> + |           |
> + |           |
> + +-----------+
> +
> +When a HugeTLB is freed to the buddy system, we should allocate 6 pages for
> +vmemmap pages and restore the previous mapping relationship.
> +
> +For the HugeTLB page of the pud level mapping. It is similar to the former.
> +We also can use this approach to free (PAGE_SIZE - 2) vmemmap pages.
> +
> +Apart from the HugeTLB page of the pmd/pud level mapping, some architectures
> +(e.g. aarch64) provides a contiguous bit in the translation table entries
> +that hints to the MMU to indicate that it is one of a contiguous set of
> +entries that can be cached in a single TLB entry.
> +
> +The contiguous bit is used to increase the mapping size at the pmd and pte
> +(last) level. So this type of HugeTLB page can be optimized only when its
> +size of the struct page structs is greater than 2 pages.
> +
> diff --git a/Documentation/vm/index.rst b/Documentation/vm/index.rst
> index eff5fbd492d0..19f981a73a54 100644
> --- a/Documentation/vm/index.rst
> +++ b/Documentation/vm/index.rst
> @@ -31,6 +31,7 @@ descriptions of data structures and algorithms.
>     active_mm
>     arch_pgtable_helpers
>     balance
> +   commpound_pagemaps
>     cleancache
>     free_page_reporting
>     frontswap
> diff --git a/mm/hugetlb_vmemmap.c b/mm/hugetlb_vmemmap.c
> index c540c21e26f5..69d1f0a90e02 100644
> --- a/mm/hugetlb_vmemmap.c
> +++ b/mm/hugetlb_vmemmap.c
> @@ -6,167 +6,7 @@
>   *
>   *     Author: Muchun Song <[e-mail]>
>   *
> - * The struct page structures (page structs) are used to describe a physical
> - * page frame. By default, there is a one-to-one mapping from a page frame to
> - * it's corresponding page struct.
> - *
> - * HugeTLB pages consist of multiple base page size pages and is supported by
> - * many architectures. See hugetlbpage.rst in the Documentation directory for
> - * more details. On the x86-64 architecture, HugeTLB pages of size 2MB and 1GB
> - * are currently supported. Since the base page size on x86 is 4KB, a 2MB
> - * HugeTLB page consists of 512 base pages and a 1GB HugeTLB page consists of
> - * 4096 base pages. For each base page, there is a corresponding page struct.
> - *
> - * Within the HugeTLB subsystem, only the first 4 page structs are used to
> - * contain unique information about a HugeTLB page. __NR_USED_SUBPAGE provides
> - * this upper limit. The only 'useful' information in the remaining page structs
> - * is the compound_head field, and this field is the same for all tail pages.
> - *
> - * By removing redundant page structs for HugeTLB pages, memory can be returned
> - * to the buddy allocator for other uses.
> - *
> - * Different architectures support different HugeTLB pages. For example, the
> - * following table is the HugeTLB page size supported by x86 and arm64
> - * architectures. Because arm64 supports 4k, 16k, and 64k base pages and
> - * supports contiguous entries, so it supports many kinds of sizes of HugeTLB
> - * page.
> - *
> - * +--------------+-----------+-----------------------------------------------+
> - * | Architecture | Page Size |                HugeTLB Page Size              |
> - * +--------------+-----------+-----------+-----------+-----------+-----------+
> - * |    x86-64    |    4KB    |    2MB    |    1GB    |           |           |
> - * +--------------+-----------+-----------+-----------+-----------+-----------+
> - * |              |    4KB    |   64KB    |    2MB    |    32MB   |    1GB    |
> - * |              +-----------+-----------+-----------+-----------+-----------+
> - * |    arm64     |   16KB    |    2MB    |   32MB    |     1GB   |           |
> - * |              +-----------+-----------+-----------+-----------+-----------+
> - * |              |   64KB    |    2MB    |  512MB    |    16GB   |           |
> - * +--------------+-----------+-----------+-----------+-----------+-----------+
> - *
> - * When the system boot up, every HugeTLB page has more than one struct page
> - * structs which size is (unit: pages):
> - *
> - *    struct_size = HugeTLB_Size / PAGE_SIZE * sizeof(struct page) / PAGE_SIZE
> - *
> - * Where HugeTLB_Size is the size of the HugeTLB page. We know that the size
> - * of the HugeTLB page is always n times PAGE_SIZE. So we can get the following
> - * relationship.
> - *
> - *    HugeTLB_Size = n * PAGE_SIZE
> - *
> - * Then,
> - *
> - *    struct_size = n * PAGE_SIZE / PAGE_SIZE * sizeof(struct page) / PAGE_SIZE
> - *                = n * sizeof(struct page) / PAGE_SIZE
> - *
> - * We can use huge mapping at the pud/pmd level for the HugeTLB page.
> - *
> - * For the HugeTLB page of the pmd level mapping, then
> - *
> - *    struct_size = n * sizeof(struct page) / PAGE_SIZE
> - *                = PAGE_SIZE / sizeof(pte_t) * sizeof(struct page) / PAGE_SIZE
> - *                = sizeof(struct page) / sizeof(pte_t)
> - *                = 64 / 8
> - *                = 8 (pages)
> - *
> - * Where n is how many pte entries which one page can contains. So the value of
> - * n is (PAGE_SIZE / sizeof(pte_t)).
> - *
> - * This optimization only supports 64-bit system, so the value of sizeof(pte_t)
> - * is 8. And this optimization also applicable only when the size of struct page
> - * is a power of two. In most cases, the size of struct page is 64 bytes (e.g.
> - * x86-64 and arm64). So if we use pmd level mapping for a HugeTLB page, the
> - * size of struct page structs of it is 8 page frames which size depends on the
> - * size of the base page.
> - *
> - * For the HugeTLB page of the pud level mapping, then
> - *
> - *    struct_size = PAGE_SIZE / sizeof(pmd_t) * struct_size(pmd)
> - *                = PAGE_SIZE / 8 * 8 (pages)
> - *                = PAGE_SIZE (pages)
> - *
> - * Where the struct_size(pmd) is the size of the struct page structs of a
> - * HugeTLB page of the pmd level mapping.
> - *
> - * E.g.: A 2MB HugeTLB page on x86_64 consists in 8 page frames while 1GB
> - * HugeTLB page consists in 4096.
> - *
> - * Next, we take the pmd level mapping of the HugeTLB page as an example to
> - * show the internal implementation of this optimization. There are 8 pages
> - * struct page structs associated with a HugeTLB page which is pmd mapped.
> - *
> - * Here is how things look before optimization.
> - *
> - *    HugeTLB                  struct pages(8 pages)         page frame(8 pages)
> - * +-----------+ ---virt_to_page---> +-----------+   mapping to   +-----------+
> - * |           |                     |     0     | -------------> |     0     |
> - * |           |                     +-----------+                +-----------+
> - * |           |                     |     1     | -------------> |     1     |
> - * |           |                     +-----------+                +-----------+
> - * |           |                     |     2     | -------------> |     2     |
> - * |           |                     +-----------+                +-----------+
> - * |           |                     |     3     | -------------> |     3     |
> - * |           |                     +-----------+                +-----------+
> - * |           |                     |     4     | -------------> |     4     |
> - * |    PMD    |                     +-----------+                +-----------+
> - * |   level   |                     |     5     | -------------> |     5     |
> - * |  mapping  |                     +-----------+                +-----------+
> - * |           |                     |     6     | -------------> |     6     |
> - * |           |                     +-----------+                +-----------+
> - * |           |                     |     7     | -------------> |     7     |
> - * |           |                     +-----------+                +-----------+
> - * |           |
> - * |           |
> - * |           |
> - * +-----------+
> - *
> - * The value of page->compound_head is the same for all tail pages. The first
> - * page of page structs (page 0) associated with the HugeTLB page contains the 4
> - * page structs necessary to describe the HugeTLB. The only use of the remaining
> - * pages of page structs (page 1 to page 7) is to point to page->compound_head.
> - * Therefore, we can remap pages 2 to 7 to page 1. Only 2 pages of page structs
> - * will be used for each HugeTLB page. This will allow us to free the remaining
> - * 6 pages to the buddy allocator.
> - *
> - * Here is how things look after remapping.
> - *
> - *    HugeTLB                  struct pages(8 pages)         page frame(8 pages)
> - * +-----------+ ---virt_to_page---> +-----------+   mapping to   +-----------+
> - * |           |                     |     0     | -------------> |     0     |
> - * |           |                     +-----------+                +-----------+
> - * |           |                     |     1     | -------------> |     1     |
> - * |           |                     +-----------+                +-----------+
> - * |           |                     |     2     | ----------------^ ^ ^ ^ ^ ^
> - * |           |                     +-----------+                   | | | | |
> - * |           |                     |     3     | ------------------+ | | | |
> - * |           |                     +-----------+                     | | | |
> - * |           |                     |     4     | --------------------+ | | |
> - * |    PMD    |                     +-----------+                       | | |
> - * |   level   |                     |     5     | ----------------------+ | |
> - * |  mapping  |                     +-----------+                         | |
> - * |           |                     |     6     | ------------------------+ |
> - * |           |                     +-----------+                           |
> - * |           |                     |     7     | --------------------------+
> - * |           |                     +-----------+
> - * |           |
> - * |           |
> - * |           |
> - * +-----------+
> - *
> - * When a HugeTLB is freed to the buddy system, we should allocate 6 pages for
> - * vmemmap pages and restore the previous mapping relationship.
> - *
> - * For the HugeTLB page of the pud level mapping. It is similar to the former.
> - * We also can use this approach to free (PAGE_SIZE - 2) vmemmap pages.
> - *
> - * Apart from the HugeTLB page of the pmd/pud level mapping, some architectures
> - * (e.g. aarch64) provides a contiguous bit in the translation table entries
> - * that hints to the MMU to indicate that it is one of a contiguous set of
> - * entries that can be cached in a single TLB entry.
> - *
> - * The contiguous bit is used to increase the mapping size at the pmd and pte
> - * (last) level. So this type of HugeTLB page can be optimized only when its
> - * size of the struct page structs is greater than 2 pages.
> + * See Documentation/vm/compound_pagemaps.rst
>   */
>  #define pr_fmt(fmt)    "HugeTLB: " fmt
>
> --
> 2.17.1
>

Joao Martins June 21, 2021, 1:42 p.m. UTC | #2

On 6/21/21 2:12 PM, Muchun Song wrote:
> On Fri, Jun 18, 2021 at 2:46 AM Joao Martins <[e-mail]> wrote:
>>
>> In preparation for device-dax for using hugetlbfs compound page tail
>> deduplication technique, move the comment block explanation into a
>> common place in Documentation/vm.
>>
>> Cc: Muchun Song <[e-mail]>
>> Cc: Mike Kravetz <[e-mail]>
>> Suggested-by: Dan Williams <[e-mail]>
>> Signed-off-by: Joao Martins <[e-mail]>
>> ---
>>  Documentation/vm/compound_pagemaps.rst | 170 +++++++++++++++++++++++++
>>  Documentation/vm/index.rst             |   1 +
>>  mm/hugetlb_vmemmap.c                   | 162 +----------------------
>>  3 files changed, 172 insertions(+), 161 deletions(-)
>>  create mode 100644 Documentation/vm/compound_pagemaps.rst
> 
> IMHO, how about the name of vmemmap_remap.rst? page_frags.rst seems
> to tell people it's about the page mapping not its vmemmap mapping.
> 

Good point.

FWIW, I wanted to avoid the use of the word 'remap' solely because that might be
implementation specific e.g. hugetlbfs remaps struct pages, whereas device-dax will
populate struct pages already with the tail dedup.

Me using 'compound_pagemaps' was short of 'compound struct page map' or 'compound vmemmap'.

Maybe one other alternative is 'tail_dedup.rst' or 'metadata_dedup.rst' ? That's probably
more generic to what really is being done.

Regardless, I am also good with 'vmemmap_remap.rst' if that's what folks prefer.


>>
>> diff --git a/Documentation/vm/compound_pagemaps.rst b/Documentation/vm/compound_pagemaps.rst
>> new file mode 100644
>> index 000000000000..6b1af50e8201
>> --- /dev/null
>> +++ b/Documentation/vm/compound_pagemaps.rst
>> @@ -0,0 +1,170 @@
>> +.. SPDX-License-Identifier: GPL-2.0
>> +
>> +.. _commpound_pagemaps:
>> +
>> +==================================
>> +Free some vmemmap pages of HugeTLB
>> +==================================
>> +
>> +The struct page structures (page structs) are used to describe a physical
>> +page frame. By default, there is a one-to-one mapping from a page frame to
>> +it's corresponding page struct.
>> +
>> +HugeTLB pages consist of multiple base page size pages and is supported by
>> +many architectures. See hugetlbpage.rst in the Documentation directory for
>> +more details. On the x86-64 architecture, HugeTLB pages of size 2MB and 1GB
>> +are currently supported. Since the base page size on x86 is 4KB, a 2MB
>> +HugeTLB page consists of 512 base pages and a 1GB HugeTLB page consists of
>> +4096 base pages. For each base page, there is a corresponding page struct.
>> +
>> +Within the HugeTLB subsystem, only the first 4 page structs are used to
>> +contain unique information about a HugeTLB page. __NR_USED_SUBPAGE provides
>> +this upper limit. The only 'useful' information in the remaining page structs
>> +is the compound_head field, and this field is the same for all tail pages.
>> +
>> +By removing redundant page structs for HugeTLB pages, memory can be returned
>> +to the buddy allocator for other uses.
>> +
>> +Different architectures support different HugeTLB pages. For example, the
>> +following table is the HugeTLB page size supported by x86 and arm64
>> +architectures. Because arm64 supports 4k, 16k, and 64k base pages and
>> +supports contiguous entries, so it supports many kinds of sizes of HugeTLB
>> +page.
>> +
>> ++--------------+-----------+-----------------------------------------------+
>> +| Architecture | Page Size |                HugeTLB Page Size              |
>> ++--------------+-----------+-----------+-----------+-----------+-----------+
>> +|    x86-64    |    4KB    |    2MB    |    1GB    |           |           |
>> ++--------------+-----------+-----------+-----------+-----------+-----------+
>> +|              |    4KB    |   64KB    |    2MB    |    32MB   |    1GB    |
>> +|              +-----------+-----------+-----------+-----------+-----------+
>> +|    arm64     |   16KB    |    2MB    |   32MB    |     1GB   |           |
>> +|              +-----------+-----------+-----------+-----------+-----------+
>> +|              |   64KB    |    2MB    |  512MB    |    16GB   |           |
>> ++--------------+-----------+-----------+-----------+-----------+-----------+
>> +
>> +When the system boot up, every HugeTLB page has more than one struct page
>> +structs which size is (unit: pages):
>> +
>> +   struct_size = HugeTLB_Size / PAGE_SIZE * sizeof(struct page) / PAGE_SIZE
>> +
>> +Where HugeTLB_Size is the size of the HugeTLB page. We know that the size
>> +of the HugeTLB page is always n times PAGE_SIZE. So we can get the following
>> +relationship.
>> +
>> +   HugeTLB_Size = n * PAGE_SIZE
>> +
>> +Then,
>> +
>> +   struct_size = n * PAGE_SIZE / PAGE_SIZE * sizeof(struct page) / PAGE_SIZE
>> +               = n * sizeof(struct page) / PAGE_SIZE
>> +
>> +We can use huge mapping at the pud/pmd level for the HugeTLB page.
>> +
>> +For the HugeTLB page of the pmd level mapping, then
>> +
>> +   struct_size = n * sizeof(struct page) / PAGE_SIZE
>> +               = PAGE_SIZE / sizeof(pte_t) * sizeof(struct page) / PAGE_SIZE
>> +               = sizeof(struct page) / sizeof(pte_t)
>> +               = 64 / 8
>> +               = 8 (pages)
>> +
>> +Where n is how many pte entries which one page can contains. So the value of
>> +n is (PAGE_SIZE / sizeof(pte_t)).
>> +
>> +This optimization only supports 64-bit system, so the value of sizeof(pte_t)
>> +is 8. And this optimization also applicable only when the size of struct page
>> +is a power of two. In most cases, the size of struct page is 64 bytes (e.g.
>> +x86-64 and arm64). So if we use pmd level mapping for a HugeTLB page, the
>> +size of struct page structs of it is 8 page frames which size depends on the
>> +size of the base page.
>> +
>> +For the HugeTLB page of the pud level mapping, then
>> +
>> +   struct_size = PAGE_SIZE / sizeof(pmd_t) * struct_size(pmd)
>> +               = PAGE_SIZE / 8 * 8 (pages)
>> +               = PAGE_SIZE (pages)
>> +
>> +Where the struct_size(pmd) is the size of the struct page structs of a
>> +HugeTLB page of the pmd level mapping.
>> +
>> +E.g.: A 2MB HugeTLB page on x86_64 consists in 8 page frames while 1GB
>> +HugeTLB page consists in 4096.
>> +
>> +Next, we take the pmd level mapping of the HugeTLB page as an example to
>> +show the internal implementation of this optimization. There are 8 pages
>> +struct page structs associated with a HugeTLB page which is pmd mapped.
>> +
>> +Here is how things look before optimization.
>> +
>> +    HugeTLB                  struct pages(8 pages)         page frame(8 pages)
>> + +-----------+ ---virt_to_page---> +-----------+   mapping to   +-----------+
>> + |           |                     |     0     | -------------> |     0     |
>> + |           |                     +-----------+                +-----------+
>> + |           |                     |     1     | -------------> |     1     |
>> + |           |                     +-----------+                +-----------+
>> + |           |                     |     2     | -------------> |     2     |
>> + |           |                     +-----------+                +-----------+
>> + |           |                     |     3     | -------------> |     3     |
>> + |           |                     +-----------+                +-----------+
>> + |           |                     |     4     | -------------> |     4     |
>> + |    PMD    |                     +-----------+                +-----------+
>> + |   level   |                     |     5     | -------------> |     5     |
>> + |  mapping  |                     +-----------+                +-----------+
>> + |           |                     |     6     | -------------> |     6     |
>> + |           |                     +-----------+                +-----------+
>> + |           |                     |     7     | -------------> |     7     |
>> + |           |                     +-----------+                +-----------+
>> + |           |
>> + |           |
>> + |           |
>> + +-----------+
>> +
>> +The value of page->compound_head is the same for all tail pages. The first
>> +page of page structs (page 0) associated with the HugeTLB page contains the 4
>> +page structs necessary to describe the HugeTLB. The only use of the remaining
>> +pages of page structs (page 1 to page 7) is to point to page->compound_head.
>> +Therefore, we can remap pages 2 to 7 to page 1. Only 2 pages of page structs
>> +will be used for each HugeTLB page. This will allow us to free the remaining
>> +6 pages to the buddy allocator.
>> +
>> +Here is how things look after remapping.
>> +
>> +    HugeTLB                  struct pages(8 pages)         page frame(8 pages)
>> + +-----------+ ---virt_to_page---> +-----------+   mapping to   +-----------+
>> + |           |                     |     0     | -------------> |     0     |
>> + |           |                     +-----------+                +-----------+
>> + |           |                     |     1     | -------------> |     1     |
>> + |           |                     +-----------+                +-----------+
>> + |           |                     |     2     | ----------------^ ^ ^ ^ ^ ^
>> + |           |                     +-----------+                   | | | | |
>> + |           |                     |     3     | ------------------+ | | | |
>> + |           |                     +-----------+                     | | | |
>> + |           |                     |     4     | --------------------+ | | |
>> + |    PMD    |                     +-----------+                       | | |
>> + |   level   |                     |     5     | ----------------------+ | |
>> + |  mapping  |                     +-----------+                         | |
>> + |           |                     |     6     | ------------------------+ |
>> + |           |                     +-----------+                           |
>> + |           |                     |     7     | --------------------------+
>> + |           |                     +-----------+
>> + |           |
>> + |           |
>> + |           |
>> + +-----------+
>> +
>> +When a HugeTLB is freed to the buddy system, we should allocate 6 pages for
>> +vmemmap pages and restore the previous mapping relationship.
>> +
>> +For the HugeTLB page of the pud level mapping. It is similar to the former.
>> +We also can use this approach to free (PAGE_SIZE - 2) vmemmap pages.
>> +
>> +Apart from the HugeTLB page of the pmd/pud level mapping, some architectures
>> +(e.g. aarch64) provides a contiguous bit in the translation table entries
>> +that hints to the MMU to indicate that it is one of a contiguous set of
>> +entries that can be cached in a single TLB entry.
>> +
>> +The contiguous bit is used to increase the mapping size at the pmd and pte
>> +(last) level. So this type of HugeTLB page can be optimized only when its
>> +size of the struct page structs is greater than 2 pages.
>> +
>> diff --git a/Documentation/vm/index.rst b/Documentation/vm/index.rst
>> index eff5fbd492d0..19f981a73a54 100644
>> --- a/Documentation/vm/index.rst
>> +++ b/Documentation/vm/index.rst
>> @@ -31,6 +31,7 @@ descriptions of data structures and algorithms.
>>     active_mm
>>     arch_pgtable_helpers
>>     balance
>> +   commpound_pagemaps
>>     cleancache
>>     free_page_reporting
>>     frontswap
>> diff --git a/mm/hugetlb_vmemmap.c b/mm/hugetlb_vmemmap.c
>> index c540c21e26f5..69d1f0a90e02 100644
>> --- a/mm/hugetlb_vmemmap.c
>> +++ b/mm/hugetlb_vmemmap.c
>> @@ -6,167 +6,7 @@
>>   *
>>   *     Author: Muchun Song <[e-mail]>
>>   *
>> - * The struct page structures (page structs) are used to describe a physical
>> - * page frame. By default, there is a one-to-one mapping from a page frame to
>> - * it's corresponding page struct.
>> - *
>> - * HugeTLB pages consist of multiple base page size pages and is supported by
>> - * many architectures. See hugetlbpage.rst in the Documentation directory for
>> - * more details. On the x86-64 architecture, HugeTLB pages of size 2MB and 1GB
>> - * are currently supported. Since the base page size on x86 is 4KB, a 2MB
>> - * HugeTLB page consists of 512 base pages and a 1GB HugeTLB page consists of
>> - * 4096 base pages. For each base page, there is a corresponding page struct.
>> - *
>> - * Within the HugeTLB subsystem, only the first 4 page structs are used to
>> - * contain unique information about a HugeTLB page. __NR_USED_SUBPAGE provides
>> - * this upper limit. The only 'useful' information in the remaining page structs
>> - * is the compound_head field, and this field is the same for all tail pages.
>> - *
>> - * By removing redundant page structs for HugeTLB pages, memory can be returned
>> - * to the buddy allocator for other uses.
>> - *
>> - * Different architectures support different HugeTLB pages. For example, the
>> - * following table is the HugeTLB page size supported by x86 and arm64
>> - * architectures. Because arm64 supports 4k, 16k, and 64k base pages and
>> - * supports contiguous entries, so it supports many kinds of sizes of HugeTLB
>> - * page.
>> - *
>> - * +--------------+-----------+-----------------------------------------------+
>> - * | Architecture | Page Size |                HugeTLB Page Size              |
>> - * +--------------+-----------+-----------+-----------+-----------+-----------+
>> - * |    x86-64    |    4KB    |    2MB    |    1GB    |           |           |
>> - * +--------------+-----------+-----------+-----------+-----------+-----------+
>> - * |              |    4KB    |   64KB    |    2MB    |    32MB   |    1GB    |
>> - * |              +-----------+-----------+-----------+-----------+-----------+
>> - * |    arm64     |   16KB    |    2MB    |   32MB    |     1GB   |           |
>> - * |              +-----------+-----------+-----------+-----------+-----------+
>> - * |              |   64KB    |    2MB    |  512MB    |    16GB   |           |
>> - * +--------------+-----------+-----------+-----------+-----------+-----------+
>> - *
>> - * When the system boot up, every HugeTLB page has more than one struct page
>> - * structs which size is (unit: pages):
>> - *
>> - *    struct_size = HugeTLB_Size / PAGE_SIZE * sizeof(struct page) / PAGE_SIZE
>> - *
>> - * Where HugeTLB_Size is the size of the HugeTLB page. We know that the size
>> - * of the HugeTLB page is always n times PAGE_SIZE. So we can get the following
>> - * relationship.
>> - *
>> - *    HugeTLB_Size = n * PAGE_SIZE
>> - *
>> - * Then,
>> - *
>> - *    struct_size = n * PAGE_SIZE / PAGE_SIZE * sizeof(struct page) / PAGE_SIZE
>> - *                = n * sizeof(struct page) / PAGE_SIZE
>> - *
>> - * We can use huge mapping at the pud/pmd level for the HugeTLB page.
>> - *
>> - * For the HugeTLB page of the pmd level mapping, then
>> - *
>> - *    struct_size = n * sizeof(struct page) / PAGE_SIZE
>> - *                = PAGE_SIZE / sizeof(pte_t) * sizeof(struct page) / PAGE_SIZE
>> - *                = sizeof(struct page) / sizeof(pte_t)
>> - *                = 64 / 8
>> - *                = 8 (pages)
>> - *
>> - * Where n is how many pte entries which one page can contains. So the value of
>> - * n is (PAGE_SIZE / sizeof(pte_t)).
>> - *
>> - * This optimization only supports 64-bit system, so the value of sizeof(pte_t)
>> - * is 8. And this optimization also applicable only when the size of struct page
>> - * is a power of two. In most cases, the size of struct page is 64 bytes (e.g.
>> - * x86-64 and arm64). So if we use pmd level mapping for a HugeTLB page, the
>> - * size of struct page structs of it is 8 page frames which size depends on the
>> - * size of the base page.
>> - *
>> - * For the HugeTLB page of the pud level mapping, then
>> - *
>> - *    struct_size = PAGE_SIZE / sizeof(pmd_t) * struct_size(pmd)
>> - *                = PAGE_SIZE / 8 * 8 (pages)
>> - *                = PAGE_SIZE (pages)
>> - *
>> - * Where the struct_size(pmd) is the size of the struct page structs of a
>> - * HugeTLB page of the pmd level mapping.
>> - *
>> - * E.g.: A 2MB HugeTLB page on x86_64 consists in 8 page frames while 1GB
>> - * HugeTLB page consists in 4096.
>> - *
>> - * Next, we take the pmd level mapping of the HugeTLB page as an example to
>> - * show the internal implementation of this optimization. There are 8 pages
>> - * struct page structs associated with a HugeTLB page which is pmd mapped.
>> - *
>> - * Here is how things look before optimization.
>> - *
>> - *    HugeTLB                  struct pages(8 pages)         page frame(8 pages)
>> - * +-----------+ ---virt_to_page---> +-----------+   mapping to   +-----------+
>> - * |           |                     |     0     | -------------> |     0     |
>> - * |           |                     +-----------+                +-----------+
>> - * |           |                     |     1     | -------------> |     1     |
>> - * |           |                     +-----------+                +-----------+
>> - * |           |                     |     2     | -------------> |     2     |
>> - * |           |                     +-----------+                +-----------+
>> - * |           |                     |     3     | -------------> |     3     |
>> - * |           |                     +-----------+                +-----------+
>> - * |           |                     |     4     | -------------> |     4     |
>> - * |    PMD    |                     +-----------+                +-----------+
>> - * |   level   |                     |     5     | -------------> |     5     |
>> - * |  mapping  |                     +-----------+                +-----------+
>> - * |           |                     |     6     | -------------> |     6     |
>> - * |           |                     +-----------+                +-----------+
>> - * |           |                     |     7     | -------------> |     7     |
>> - * |           |                     +-----------+                +-----------+
>> - * |           |
>> - * |           |
>> - * |           |
>> - * +-----------+
>> - *
>> - * The value of page->compound_head is the same for all tail pages. The first
>> - * page of page structs (page 0) associated with the HugeTLB page contains the 4
>> - * page structs necessary to describe the HugeTLB. The only use of the remaining
>> - * pages of page structs (page 1 to page 7) is to point to page->compound_head.
>> - * Therefore, we can remap pages 2 to 7 to page 1. Only 2 pages of page structs
>> - * will be used for each HugeTLB page. This will allow us to free the remaining
>> - * 6 pages to the buddy allocator.
>> - *
>> - * Here is how things look after remapping.
>> - *
>> - *    HugeTLB                  struct pages(8 pages)         page frame(8 pages)
>> - * +-----------+ ---virt_to_page---> +-----------+   mapping to   +-----------+
>> - * |           |                     |     0     | -------------> |     0     |
>> - * |           |                     +-----------+                +-----------+
>> - * |           |                     |     1     | -------------> |     1     |
>> - * |           |                     +-----------+                +-----------+
>> - * |           |                     |     2     | ----------------^ ^ ^ ^ ^ ^
>> - * |           |                     +-----------+                   | | | | |
>> - * |           |                     |     3     | ------------------+ | | | |
>> - * |           |                     +-----------+                     | | | |
>> - * |           |                     |     4     | --------------------+ | | |
>> - * |    PMD    |                     +-----------+                       | | |
>> - * |   level   |                     |     5     | ----------------------+ | |
>> - * |  mapping  |                     +-----------+                         | |
>> - * |           |                     |     6     | ------------------------+ |
>> - * |           |                     +-----------+                           |
>> - * |           |                     |     7     | --------------------------+
>> - * |           |                     +-----------+
>> - * |           |
>> - * |           |
>> - * |           |
>> - * +-----------+
>> - *
>> - * When a HugeTLB is freed to the buddy system, we should allocate 6 pages for
>> - * vmemmap pages and restore the previous mapping relationship.
>> - *
>> - * For the HugeTLB page of the pud level mapping. It is similar to the former.
>> - * We also can use this approach to free (PAGE_SIZE - 2) vmemmap pages.
>> - *
>> - * Apart from the HugeTLB page of the pmd/pud level mapping, some architectures
>> - * (e.g. aarch64) provides a contiguous bit in the translation table entries
>> - * that hints to the MMU to indicate that it is one of a contiguous set of
>> - * entries that can be cached in a single TLB entry.
>> - *
>> - * The contiguous bit is used to increase the mapping size at the pmd and pte
>> - * (last) level. So this type of HugeTLB page can be optimized only when its
>> - * size of the struct page structs is greater than 2 pages.
>> + * See Documentation/vm/compound_pagemaps.rst
>>   */
>>  #define pr_fmt(fmt)    "HugeTLB: " fmt
>>
>> --
>> 2.17.1
>>
>

Mike Kravetz July 13, 2021, 12:14 a.m. UTC | #3

On 6/21/21 6:42 AM, Joao Martins wrote:
> On 6/21/21 2:12 PM, Muchun Song wrote:
>> On Fri, Jun 18, 2021 at 2:46 AM Joao Martins <[e-mail]> wrote:
>>>
>>> In preparation for device-dax for using hugetlbfs compound page tail
>>> deduplication technique, move the comment block explanation into a
>>> common place in Documentation/vm.
>>>
>>> Cc: Muchun Song <[e-mail]>
>>> Cc: Mike Kravetz <[e-mail]>
>>> Suggested-by: Dan Williams <[e-mail]>
>>> Signed-off-by: Joao Martins <[e-mail]>
>>> ---
>>>  Documentation/vm/compound_pagemaps.rst | 170 +++++++++++++++++++++++++
>>>  Documentation/vm/index.rst             |   1 +
>>>  mm/hugetlb_vmemmap.c                   | 162 +----------------------
>>>  3 files changed, 172 insertions(+), 161 deletions(-)
>>>  create mode 100644 Documentation/vm/compound_pagemaps.rst
>>
>> IMHO, how about the name of vmemmap_remap.rst? page_frags.rst seems
>> to tell people it's about the page mapping not its vmemmap mapping.
>>
> 
> Good point.
> 
> FWIW, I wanted to avoid the use of the word 'remap' solely because that might be
> implementation specific e.g. hugetlbfs remaps struct pages, whereas device-dax will
> populate struct pages already with the tail dedup.
> 
> Me using 'compound_pagemaps' was short of 'compound struct page map' or 'compound vmemmap'.
> 
> Maybe one other alternative is 'tail_dedup.rst' or 'metadata_dedup.rst' ? That's probably
> more generic to what really is being done.
> 
> Regardless, I am also good with 'vmemmap_remap.rst' if that's what folks prefer.
> 

How about vmemmap_dedup?

I do think it is a good idea to move this to a common documentation file
if Device DAX is going to use the same technique.

Joao Martins July 13, 2021, 1:11 a.m. UTC | #4

On 7/13/21 1:14 AM, Mike Kravetz wrote:
> On 6/21/21 6:42 AM, Joao Martins wrote:
>> On 6/21/21 2:12 PM, Muchun Song wrote:
>>> On Fri, Jun 18, 2021 at 2:46 AM Joao Martins <[e-mail]> wrote:
>>>>
>>>> In preparation for device-dax for using hugetlbfs compound page tail
>>>> deduplication technique, move the comment block explanation into a
>>>> common place in Documentation/vm.
>>>>
>>>> Cc: Muchun Song <[e-mail]>
>>>> Cc: Mike Kravetz <[e-mail]>
>>>> Suggested-by: Dan Williams <[e-mail]>
>>>> Signed-off-by: Joao Martins <[e-mail]>
>>>> ---
>>>>  Documentation/vm/compound_pagemaps.rst | 170 +++++++++++++++++++++++++
>>>>  Documentation/vm/index.rst             |   1 +
>>>>  mm/hugetlb_vmemmap.c                   | 162 +----------------------
>>>>  3 files changed, 172 insertions(+), 161 deletions(-)
>>>>  create mode 100644 Documentation/vm/compound_pagemaps.rst
>>>
>>> IMHO, how about the name of vmemmap_remap.rst? page_frags.rst seems
>>> to tell people it's about the page mapping not its vmemmap mapping.
>>>
>>
>> Good point.
>>
>> FWIW, I wanted to avoid the use of the word 'remap' solely because that might be
>> implementation specific e.g. hugetlbfs remaps struct pages, whereas device-dax will
>> populate struct pages already with the tail dedup.
>>
>> Me using 'compound_pagemaps' was short of 'compound struct page map' or 'compound vmemmap'.
>>
>> Maybe one other alternative is 'tail_dedup.rst' or 'metadata_dedup.rst' ? That's probably
>> more generic to what really is being done.
>>
>> Regardless, I am also good with 'vmemmap_remap.rst' if that's what folks prefer.
>>
> 
> How about vmemmap_dedup?
> 
Sounds good to me, I'll rename it.

> I do think it is a good idea to move this to a common documentation file
> if Device DAX is going to use the same technique.
>

Patch

diff --git a/Documentation/vm/compound_pagemaps.rst b/Documentation/vm/compound_pagemaps.rst
new file mode 100644
index 000000000000..6b1af50e8201
--- /dev/null
+++ b/Documentation/vm/compound_pagemaps.rst
@@ -0,0 +1,170 @@ 
+.. SPDX-License-Identifier: GPL-2.0
+
+.. _commpound_pagemaps:
+
+==================================
+Free some vmemmap pages of HugeTLB
+==================================
+
+The struct page structures (page structs) are used to describe a physical
+page frame. By default, there is a one-to-one mapping from a page frame to
+it's corresponding page struct.
+
+HugeTLB pages consist of multiple base page size pages and is supported by
+many architectures. See hugetlbpage.rst in the Documentation directory for
+more details. On the x86-64 architecture, HugeTLB pages of size 2MB and 1GB
+are currently supported. Since the base page size on x86 is 4KB, a 2MB
+HugeTLB page consists of 512 base pages and a 1GB HugeTLB page consists of
+4096 base pages. For each base page, there is a corresponding page struct.
+
+Within the HugeTLB subsystem, only the first 4 page structs are used to
+contain unique information about a HugeTLB page. __NR_USED_SUBPAGE provides
+this upper limit. The only 'useful' information in the remaining page structs
+is the compound_head field, and this field is the same for all tail pages.
+
+By removing redundant page structs for HugeTLB pages, memory can be returned
+to the buddy allocator for other uses.
+
+Different architectures support different HugeTLB pages. For example, the
+following table is the HugeTLB page size supported by x86 and arm64
+architectures. Because arm64 supports 4k, 16k, and 64k base pages and
+supports contiguous entries, so it supports many kinds of sizes of HugeTLB
+page.
+
++--------------+-----------+-----------------------------------------------+
+| Architecture | Page Size |                HugeTLB Page Size              |
++--------------+-----------+-----------+-----------+-----------+-----------+
+|    x86-64    |    4KB    |    2MB    |    1GB    |           |           |
++--------------+-----------+-----------+-----------+-----------+-----------+
+|              |    4KB    |   64KB    |    2MB    |    32MB   |    1GB    |
+|              +-----------+-----------+-----------+-----------+-----------+
+|    arm64     |   16KB    |    2MB    |   32MB    |     1GB   |           |
+|              +-----------+-----------+-----------+-----------+-----------+
+|              |   64KB    |    2MB    |  512MB    |    16GB   |           |
++--------------+-----------+-----------+-----------+-----------+-----------+
+
+When the system boot up, every HugeTLB page has more than one struct page
+structs which size is (unit: pages):
+
+   struct_size = HugeTLB_Size / PAGE_SIZE * sizeof(struct page) / PAGE_SIZE
+
+Where HugeTLB_Size is the size of the HugeTLB page. We know that the size
+of the HugeTLB page is always n times PAGE_SIZE. So we can get the following
+relationship.
+
+   HugeTLB_Size = n * PAGE_SIZE
+
+Then,
+
+   struct_size = n * PAGE_SIZE / PAGE_SIZE * sizeof(struct page) / PAGE_SIZE
+               = n * sizeof(struct page) / PAGE_SIZE
+
+We can use huge mapping at the pud/pmd level for the HugeTLB page.
+
+For the HugeTLB page of the pmd level mapping, then
+
+   struct_size = n * sizeof(struct page) / PAGE_SIZE
+               = PAGE_SIZE / sizeof(pte_t) * sizeof(struct page) / PAGE_SIZE
+               = sizeof(struct page) / sizeof(pte_t)
+               = 64 / 8
+               = 8 (pages)
+
+Where n is how many pte entries which one page can contains. So the value of
+n is (PAGE_SIZE / sizeof(pte_t)).
+
+This optimization only supports 64-bit system, so the value of sizeof(pte_t)
+is 8. And this optimization also applicable only when the size of struct page
+is a power of two. In most cases, the size of struct page is 64 bytes (e.g.
+x86-64 and arm64). So if we use pmd level mapping for a HugeTLB page, the
+size of struct page structs of it is 8 page frames which size depends on the
+size of the base page.
+
+For the HugeTLB page of the pud level mapping, then
+
+   struct_size = PAGE_SIZE / sizeof(pmd_t) * struct_size(pmd)
+               = PAGE_SIZE / 8 * 8 (pages)
+               = PAGE_SIZE (pages)
+
+Where the struct_size(pmd) is the size of the struct page structs of a
+HugeTLB page of the pmd level mapping.
+
+E.g.: A 2MB HugeTLB page on x86_64 consists in 8 page frames while 1GB
+HugeTLB page consists in 4096.
+
+Next, we take the pmd level mapping of the HugeTLB page as an example to
+show the internal implementation of this optimization. There are 8 pages
+struct page structs associated with a HugeTLB page which is pmd mapped.
+
+Here is how things look before optimization.
+
+    HugeTLB                  struct pages(8 pages)         page frame(8 pages)
+ +-----------+ ---virt_to_page---> +-----------+   mapping to   +-----------+
+ |           |                     |     0     | -------------> |     0     |
+ |           |                     +-----------+                +-----------+
+ |           |                     |     1     | -------------> |     1     |
+ |           |                     +-----------+                +-----------+
+ |           |                     |     2     | -------------> |     2     |
+ |           |                     +-----------+                +-----------+
+ |           |                     |     3     | -------------> |     3     |
+ |           |                     +-----------+                +-----------+
+ |           |                     |     4     | -------------> |     4     |
+ |    PMD    |                     +-----------+                +-----------+
+ |   level   |                     |     5     | -------------> |     5     |
+ |  mapping  |                     +-----------+                +-----------+
+ |           |                     |     6     | -------------> |     6     |
+ |           |                     +-----------+                +-----------+
+ |           |                     |     7     | -------------> |     7     |
+ |           |                     +-----------+                +-----------+
+ |           |
+ |           |
+ |           |
+ +-----------+
+
+The value of page->compound_head is the same for all tail pages. The first
+page of page structs (page 0) associated with the HugeTLB page contains the 4
+page structs necessary to describe the HugeTLB. The only use of the remaining
+pages of page structs (page 1 to page 7) is to point to page->compound_head.
+Therefore, we can remap pages 2 to 7 to page 1. Only 2 pages of page structs
+will be used for each HugeTLB page. This will allow us to free the remaining
+6 pages to the buddy allocator.
+
+Here is how things look after remapping.
+
+    HugeTLB                  struct pages(8 pages)         page frame(8 pages)
+ +-----------+ ---virt_to_page---> +-----------+   mapping to   +-----------+
+ |           |                     |     0     | -------------> |     0     |
+ |           |                     +-----------+                +-----------+
+ |           |                     |     1     | -------------> |     1     |
+ |           |                     +-----------+                +-----------+
+ |           |                     |     2     | ----------------^ ^ ^ ^ ^ ^
+ |           |                     +-----------+                   | | | | |
+ |           |                     |     3     | ------------------+ | | | |
+ |           |                     +-----------+                     | | | |
+ |           |                     |     4     | --------------------+ | | |
+ |    PMD    |                     +-----------+                       | | |
+ |   level   |                     |     5     | ----------------------+ | |
+ |  mapping  |                     +-----------+                         | |
+ |           |                     |     6     | ------------------------+ |
+ |           |                     +-----------+                           |
+ |           |                     |     7     | --------------------------+
+ |           |                     +-----------+
+ |           |
+ |           |
+ |           |
+ +-----------+
+
+When a HugeTLB is freed to the buddy system, we should allocate 6 pages for
+vmemmap pages and restore the previous mapping relationship.
+
+For the HugeTLB page of the pud level mapping. It is similar to the former.
+We also can use this approach to free (PAGE_SIZE - 2) vmemmap pages.
+
+Apart from the HugeTLB page of the pmd/pud level mapping, some architectures
+(e.g. aarch64) provides a contiguous bit in the translation table entries
+that hints to the MMU to indicate that it is one of a contiguous set of
+entries that can be cached in a single TLB entry.
+
+The contiguous bit is used to increase the mapping size at the pmd and pte
+(last) level. So this type of HugeTLB page can be optimized only when its
+size of the struct page structs is greater than 2 pages.
+
diff --git a/Documentation/vm/index.rst b/Documentation/vm/index.rst
index eff5fbd492d0..19f981a73a54 100644
--- a/Documentation/vm/index.rst
+++ b/Documentation/vm/index.rst
@@ -31,6 +31,7 @@  descriptions of data structures and algorithms.
    active_mm
    arch_pgtable_helpers
    balance
+   commpound_pagemaps
    cleancache
    free_page_reporting
    frontswap
diff --git a/mm/hugetlb_vmemmap.c b/mm/hugetlb_vmemmap.c
index c540c21e26f5..69d1f0a90e02 100644
--- a/mm/hugetlb_vmemmap.c
+++ b/mm/hugetlb_vmemmap.c
@@ -6,167 +6,7 @@ 
  *
  *     Author: Muchun Song <[e-mail]>
  *
- * The struct page structures (page structs) are used to describe a physical
- * page frame. By default, there is a one-to-one mapping from a page frame to
- * it's corresponding page struct.
- *
- * HugeTLB pages consist of multiple base page size pages and is supported by
- * many architectures. See hugetlbpage.rst in the Documentation directory for
- * more details. On the x86-64 architecture, HugeTLB pages of size 2MB and 1GB
- * are currently supported. Since the base page size on x86 is 4KB, a 2MB
- * HugeTLB page consists of 512 base pages and a 1GB HugeTLB page consists of
- * 4096 base pages. For each base page, there is a corresponding page struct.
- *
- * Within the HugeTLB subsystem, only the first 4 page structs are used to
- * contain unique information about a HugeTLB page. __NR_USED_SUBPAGE provides
- * this upper limit. The only 'useful' information in the remaining page structs
- * is the compound_head field, and this field is the same for all tail pages.
- *
- * By removing redundant page structs for HugeTLB pages, memory can be returned
- * to the buddy allocator for other uses.
- *
- * Different architectures support different HugeTLB pages. For example, the
- * following table is the HugeTLB page size supported by x86 and arm64
- * architectures. Because arm64 supports 4k, 16k, and 64k base pages and
- * supports contiguous entries, so it supports many kinds of sizes of HugeTLB
- * page.
- *
- * +--------------+-----------+-----------------------------------------------+
- * | Architecture | Page Size |                HugeTLB Page Size              |
- * +--------------+-----------+-----------+-----------+-----------+-----------+
- * |    x86-64    |    4KB    |    2MB    |    1GB    |           |           |
- * +--------------+-----------+-----------+-----------+-----------+-----------+
- * |              |    4KB    |   64KB    |    2MB    |    32MB   |    1GB    |
- * |              +-----------+-----------+-----------+-----------+-----------+
- * |    arm64     |   16KB    |    2MB    |   32MB    |     1GB   |           |
- * |              +-----------+-----------+-----------+-----------+-----------+
- * |              |   64KB    |    2MB    |  512MB    |    16GB   |           |
- * +--------------+-----------+-----------+-----------+-----------+-----------+
- *
- * When the system boot up, every HugeTLB page has more than one struct page
- * structs which size is (unit: pages):
- *
- *    struct_size = HugeTLB_Size / PAGE_SIZE * sizeof(struct page) / PAGE_SIZE
- *
- * Where HugeTLB_Size is the size of the HugeTLB page. We know that the size
- * of the HugeTLB page is always n times PAGE_SIZE. So we can get the following
- * relationship.
- *
- *    HugeTLB_Size = n * PAGE_SIZE
- *
- * Then,
- *
- *    struct_size = n * PAGE_SIZE / PAGE_SIZE * sizeof(struct page) / PAGE_SIZE
- *                = n * sizeof(struct page) / PAGE_SIZE
- *
- * We can use huge mapping at the pud/pmd level for the HugeTLB page.
- *
- * For the HugeTLB page of the pmd level mapping, then
- *
- *    struct_size = n * sizeof(struct page) / PAGE_SIZE
- *                = PAGE_SIZE / sizeof(pte_t) * sizeof(struct page) / PAGE_SIZE
- *                = sizeof(struct page) / sizeof(pte_t)
- *                = 64 / 8
- *                = 8 (pages)
- *
- * Where n is how many pte entries which one page can contains. So the value of
- * n is (PAGE_SIZE / sizeof(pte_t)).
- *
- * This optimization only supports 64-bit system, so the value of sizeof(pte_t)
- * is 8. And this optimization also applicable only when the size of struct page
- * is a power of two. In most cases, the size of struct page is 64 bytes (e.g.
- * x86-64 and arm64). So if we use pmd level mapping for a HugeTLB page, the
- * size of struct page structs of it is 8 page frames which size depends on the
- * size of the base page.
- *
- * For the HugeTLB page of the pud level mapping, then
- *
- *    struct_size = PAGE_SIZE / sizeof(pmd_t) * struct_size(pmd)
- *                = PAGE_SIZE / 8 * 8 (pages)
- *                = PAGE_SIZE (pages)
- *
- * Where the struct_size(pmd) is the size of the struct page structs of a
- * HugeTLB page of the pmd level mapping.
- *
- * E.g.: A 2MB HugeTLB page on x86_64 consists in 8 page frames while 1GB
- * HugeTLB page consists in 4096.
- *
- * Next, we take the pmd level mapping of the HugeTLB page as an example to
- * show the internal implementation of this optimization. There are 8 pages
- * struct page structs associated with a HugeTLB page which is pmd mapped.
- *
- * Here is how things look before optimization.
- *
- *    HugeTLB                  struct pages(8 pages)         page frame(8 pages)
- * +-----------+ ---virt_to_page---> +-----------+   mapping to   +-----------+
- * |           |                     |     0     | -------------> |     0     |
- * |           |                     +-----------+                +-----------+
- * |           |                     |     1     | -------------> |     1     |
- * |           |                     +-----------+                +-----------+
- * |           |                     |     2     | -------------> |     2     |
- * |           |                     +-----------+                +-----------+
- * |           |                     |     3     | -------------> |     3     |
- * |           |                     +-----------+                +-----------+
- * |           |                     |     4     | -------------> |     4     |
- * |    PMD    |                     +-----------+                +-----------+
- * |   level   |                     |     5     | -------------> |     5     |
- * |  mapping  |                     +-----------+                +-----------+
- * |           |                     |     6     | -------------> |     6     |
- * |           |                     +-----------+                +-----------+
- * |           |                     |     7     | -------------> |     7     |
- * |           |                     +-----------+                +-----------+
- * |           |
- * |           |
- * |           |
- * +-----------+
- *
- * The value of page->compound_head is the same for all tail pages. The first
- * page of page structs (page 0) associated with the HugeTLB page contains the 4
- * page structs necessary to describe the HugeTLB. The only use of the remaining
- * pages of page structs (page 1 to page 7) is to point to page->compound_head.
- * Therefore, we can remap pages 2 to 7 to page 1. Only 2 pages of page structs
- * will be used for each HugeTLB page. This will allow us to free the remaining
- * 6 pages to the buddy allocator.
- *
- * Here is how things look after remapping.
- *
- *    HugeTLB                  struct pages(8 pages)         page frame(8 pages)
- * +-----------+ ---virt_to_page---> +-----------+   mapping to   +-----------+
- * |           |                     |     0     | -------------> |     0     |
- * |           |                     +-----------+                +-----------+
- * |           |                     |     1     | -------------> |     1     |
- * |           |                     +-----------+                +-----------+
- * |           |                     |     2     | ----------------^ ^ ^ ^ ^ ^
- * |           |                     +-----------+                   | | | | |
- * |           |                     |     3     | ------------------+ | | | |
- * |           |                     +-----------+                     | | | |
- * |           |                     |     4     | --------------------+ | | |
- * |    PMD    |                     +-----------+                       | | |
- * |   level   |                     |     5     | ----------------------+ | |
- * |  mapping  |                     +-----------+                         | |
- * |           |                     |     6     | ------------------------+ |
- * |           |                     +-----------+                           |
- * |           |                     |     7     | --------------------------+
- * |           |                     +-----------+
- * |           |
- * |           |
- * |           |
- * +-----------+
- *
- * When a HugeTLB is freed to the buddy system, we should allocate 6 pages for
- * vmemmap pages and restore the previous mapping relationship.
- *
- * For the HugeTLB page of the pud level mapping. It is similar to the former.
- * We also can use this approach to free (PAGE_SIZE - 2) vmemmap pages.
- *
- * Apart from the HugeTLB page of the pmd/pud level mapping, some architectures
- * (e.g. aarch64) provides a contiguous bit in the translation table entries
- * that hints to the MMU to indicate that it is one of a contiguous set of
- * entries that can be cached in a single TLB entry.
- *
- * The contiguous bit is used to increase the mapping size at the pmd and pte
- * (last) level. So this type of HugeTLB page can be optimized only when its
- * size of the struct page structs is greater than 2 pages.
+ * See Documentation/vm/compound_pagemaps.rst
  */
 #define pr_fmt(fmt)	"HugeTLB: " fmt