From patchwork Mon May 6 08:46:25 2024 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Baolin Wang X-Patchwork-Id: 13655142 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from kanga.kvack.org (kanga.kvack.org [205.233.56.17]) by smtp.lore.kernel.org (Postfix) with ESMTP id CA609C4345F for ; Mon, 6 May 2024 08:46:52 +0000 (UTC) Received: by kanga.kvack.org (Postfix) id 321676B0083; Mon, 6 May 2024 04:46:52 -0400 (EDT) Received: by kanga.kvack.org (Postfix, from userid 40) id 2CB616B0087; Mon, 6 May 2024 04:46:52 -0400 (EDT) X-Delivered-To: int-list-linux-mm@kvack.org Received: by kanga.kvack.org (Postfix, from userid 63042) id 192F96B0085; Mon, 6 May 2024 04:46:52 -0400 (EDT) X-Delivered-To: linux-mm@kvack.org Received: from relay.hostedemail.com (smtprelay0017.hostedemail.com [216.40.44.17]) by kanga.kvack.org (Postfix) with ESMTP id F0A1D6B007B for ; Mon, 6 May 2024 04:46:51 -0400 (EDT) Received: from smtpin03.hostedemail.com (a10.router.float.18 [10.200.18.1]) by unirelay02.hostedemail.com (Postfix) with ESMTP id 9396E1208AE for ; Mon, 6 May 2024 08:46:51 +0000 (UTC) X-FDA: 82087340622.03.85BD006 Received: from out30-119.freemail.mail.aliyun.com (out30-119.freemail.mail.aliyun.com [115.124.30.119]) by imf06.hostedemail.com (Postfix) with ESMTP id A810E18001E for ; Mon, 6 May 2024 08:46:48 +0000 (UTC) Authentication-Results: imf06.hostedemail.com; dkim=pass header.d=linux.alibaba.com header.s=default header.b="M4G/dyZ5"; spf=pass (imf06.hostedemail.com: domain of baolin.wang@linux.alibaba.com designates 115.124.30.119 as permitted sender) smtp.mailfrom=baolin.wang@linux.alibaba.com; dmarc=pass (policy=none) header.from=linux.alibaba.com ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=hostedemail.com; s=arc-20220608; t=1714985209; h=from:from:sender:reply-to:subject:subject:date:date: message-id:message-id:to:to:cc:cc:mime-version:mime-version: content-type:content-transfer-encoding:content-transfer-encoding: in-reply-to:in-reply-to:references:references:dkim-signature; bh=p4+ojbN485lnDhEdHj7u8+ldhF5xWmY9Y6UtO0qnCFI=; b=uEJvk4MtiUqsse/yYDgYc//VZxmOlxCOUMu+gZRPvdjqsMN4cr5MRRRRnlbxTtp2TDmOJk gqONU8GoK5YfrjMsT8AOREue178AgUS9rz/8X6rqSdZuBjSzzV9V6FuSgwnh4wMZ5SMWiB giZnR/slxeEvvpTuLmWp0ZFtPWMruDs= ARC-Authentication-Results: i=1; imf06.hostedemail.com; dkim=pass header.d=linux.alibaba.com header.s=default header.b="M4G/dyZ5"; spf=pass (imf06.hostedemail.com: domain of baolin.wang@linux.alibaba.com designates 115.124.30.119 as permitted sender) smtp.mailfrom=baolin.wang@linux.alibaba.com; dmarc=pass (policy=none) header.from=linux.alibaba.com ARC-Seal: i=1; s=arc-20220608; d=hostedemail.com; t=1714985209; a=rsa-sha256; cv=none; b=0v9pcrCg8telyMGKinxK6pVC9KlHSW0xOmJ60/TdXarl29vvE+fS2uibYAQ3sf2c9ddPDN hQwBSVH9DmfsFugiuruARbyYM8umV2Gk6KKWN1GW8/NZ8f2cLjA/NT1Ud9EHszllWwLRXk 4jCTqK1WrX/QnnFvjQ7DRhDTlblrkdQ= DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=linux.alibaba.com; s=default; t=1714985205; h=From:To:Subject:Date:Message-Id:MIME-Version; bh=p4+ojbN485lnDhEdHj7u8+ldhF5xWmY9Y6UtO0qnCFI=; b=M4G/dyZ5V2GLWpno88uZhxml1FzQhmZmjzn7sY0ruOavuCP5oPev+hAzC2+Fg6K26gu1yaaEb0Mx+ArbuNpkW4Tuyk6ui+sTn9VRLxfu4nGJiyNGAhZvWDwqGpzD+gP7mClCkTmO4SPKoihph6k2k8/+5Rael0lQWAhWK/a1a1Y= X-Alimail-AntiSpam: AC=PASS;BC=-1|-1;BR=01201311R571e4;CH=green;DM=||false|;DS=||;FP=0|-1|-1|-1|0|-1|-1|-1;HT=maildocker-contentspam033032014031;MF=baolin.wang@linux.alibaba.com;NM=1;PH=DS;RN=14;SR=0;TI=SMTPD_---0W5weiJg_1714985203; Received: from localhost(mailfrom:baolin.wang@linux.alibaba.com fp:SMTPD_---0W5weiJg_1714985203) by smtp.aliyun-inc.com; Mon, 06 May 2024 16:46:44 +0800 From: Baolin Wang To: akpm@linux-foundation.org, hughd@google.com Cc: willy@infradead.org, david@redhat.com, ioworker0@gmail.com, wangkefeng.wang@huawei.com, ying.huang@intel.com, 21cnbao@gmail.com, ryan.roberts@arm.com, shy828301@gmail.com, ziy@nvidia.com, baolin.wang@linux.alibaba.com, linux-mm@kvack.org, linux-kernel@vger.kernel.org Subject: [PATCH 1/8] mm: move highest_order() and next_order() out of the THP config Date: Mon, 6 May 2024 16:46:25 +0800 Message-Id: X-Mailer: git-send-email 2.39.3 In-Reply-To: References: MIME-Version: 1.0 X-Stat-Signature: qqw8h97fcthfuftbc8jy9kq8ed3wohte X-Rspam-User: X-Rspamd-Queue-Id: A810E18001E X-Rspamd-Server: rspam05 X-HE-Tag: 1714985208-495486 X-HE-Meta: U2FsdGVkX1/nCOdOOAAyHj97kT15gkTW/Lvm7KZ2kGqACWj36f30z5FzGRnMzrTk8pfgJCrQF1lA7GJ5Oq7wTLrP1ehWTapZpQWuDp61knLkbUl5XlYmrAuUeAxnpRLfeviRwasfJ03nEpZtEtopTpuCW1V4qoSrYKvMnWOf0NKr4zOC29QyU4WXEP6sbdvDh0LS9N2tA1kSEMnBbELBjNygc/9kG1cAXylvZeWt1LSkRRkc1g2B5ix4OIj86sjPVtX39ef6yItCuN295yV9QfCfwjMDe0cDlsVMSKY301knDWrndDDuOdNVJ+k/eDV2K6d0t64M0gyAwf7loowR2ugan/pNBGI5JZWXGWSuOr5wJLnNHW/7ihBs5ikihwJtuk6OROkZZdSmSPqOlAi1ypXZH4hpTNX0EeuV+7CSq6kR35y9ZHg8Z9MAhHxLYRfrIjyM0RHUZD09lir8dK/ZGsM7M/6+3WqlLwCFj/IvWDluF/CChum9RreZPRZfvduIxML36ZEJ43zz1wZLwosp9blRppyWjOY6+qEO+F4Y4GNSAf6noPQUoy115iY5yK+xycGdiTw9/41my8bqpZ/ULNnNQtcdVluzWQZI1ZdntiVrr5lpCoquq5Q4WcJVt1+R7JYS0xwyZKCYD7WjQq17lNAde0Z4E8XCWtq5iMn7ZnQWDnNc5+bW/qrykrCjyWnN1Fh70meXSsO+keKoFoiwr1/zy2wf4g56TiuFXXPYv4LDz31qbHvvIGM3d8SuDCNYi28C7CqQajQW8WrBtqYrApvO4fJ6sMAfIEz+TpE2lU7NqXDVZAkGhxEXC4CFQDrq89aNAYqoYQtxM7lFRoR9EMEeHAB2y7vHRw8Jr7Gtr+V+omVEdtNejsYAshNzpZmVHW57503XkVE3YMViPAR3bZppv+xCRXglUd8qE1vR0DNkBbRv7H3sesQsI9VpY7a1+YSjWmSCaUPGS+A+Wac qk7naAh3 zOWwycjU5d6pDdwMhatLsUSW56GOV+bSQInWMTLeGCo1xtNMaVOBldr6oU3S/Q0ojpX3+ORt+4B5BNo02YMD9OQqNnR1lVUpbF3lQ9dSSDo3V5G/7Z/m2YvpmUocsJ+ka3OP56I3EA47yBVRtSiiF85kK4WZBV81S5MZuQ6WOUM0krN1xAK7K6JBym9CMObyxkfDfvcm2A4h/q2QtZIk0BYdS6Ul9ryyGFcoNvx+mRwkjqPw= X-Bogosity: Ham, tests=bogofilter, spamicity=0.000000, version=1.2.4 Sender: owner-linux-mm@kvack.org Precedence: bulk X-Loop: owner-majordomo@kvack.org List-ID: List-Subscribe: List-Unsubscribe: Move highest_order() and next_order() out of the CONFIG_TRANSPARENT_HUGEPAGE macro, which can be common functions to be used. Signed-off-by: Baolin Wang --- include/linux/huge_mm.h | 22 +++++++++++----------- 1 file changed, 11 insertions(+), 11 deletions(-) diff --git a/include/linux/huge_mm.h b/include/linux/huge_mm.h index 017cee864080..e49b56c40a11 100644 --- a/include/linux/huge_mm.h +++ b/include/linux/huge_mm.h @@ -106,6 +106,17 @@ extern struct kobj_attribute shmem_enabled_attr; #define HPAGE_PUD_MASK (~(HPAGE_PUD_SIZE - 1)) #define HPAGE_PUD_SIZE ((1UL) << HPAGE_PUD_SHIFT) +static inline int highest_order(unsigned long orders) +{ + return fls_long(orders) - 1; +} + +static inline int next_order(unsigned long *orders, int prev) +{ + *orders &= ~BIT(prev); + return highest_order(*orders); +} + #ifdef CONFIG_TRANSPARENT_HUGEPAGE extern unsigned long transparent_hugepage_flags; @@ -138,17 +149,6 @@ static inline bool hugepage_flags_enabled(void) huge_anon_orders_madvise; } -static inline int highest_order(unsigned long orders) -{ - return fls_long(orders) - 1; -} - -static inline int next_order(unsigned long *orders, int prev) -{ - *orders &= ~BIT(prev); - return highest_order(*orders); -} - /* * Do the below checks: * - For file vma, check if the linear page offset of vma is From patchwork Mon May 6 08:46:26 2024 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Baolin Wang X-Patchwork-Id: 13655144 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from kanga.kvack.org (kanga.kvack.org [205.233.56.17]) by smtp.lore.kernel.org (Postfix) with ESMTP id 0A986C4345F for ; Mon, 6 May 2024 08:46:56 +0000 (UTC) Received: by kanga.kvack.org (Postfix) id 05DC76B0082; Mon, 6 May 2024 04:46:53 -0400 (EDT) Received: by kanga.kvack.org (Postfix, from userid 40) id F28156B0085; Mon, 6 May 2024 04:46:52 -0400 (EDT) X-Delivered-To: int-list-linux-mm@kvack.org Received: by kanga.kvack.org (Postfix, from userid 63042) id DEEF46B0087; Mon, 6 May 2024 04:46:52 -0400 (EDT) X-Delivered-To: linux-mm@kvack.org Received: from relay.hostedemail.com (smtprelay0014.hostedemail.com [216.40.44.14]) by kanga.kvack.org (Postfix) with ESMTP id BF71C6B0082 for ; Mon, 6 May 2024 04:46:52 -0400 (EDT) Received: from smtpin01.hostedemail.com (a10.router.float.18 [10.200.18.1]) by unirelay06.hostedemail.com (Postfix) with ESMTP id 84613A1606 for ; Mon, 6 May 2024 08:46:52 +0000 (UTC) X-FDA: 82087340664.01.205020E Received: from out30-110.freemail.mail.aliyun.com (out30-110.freemail.mail.aliyun.com [115.124.30.110]) by imf07.hostedemail.com (Postfix) with ESMTP id 584464000A for ; Mon, 6 May 2024 08:46:49 +0000 (UTC) Authentication-Results: imf07.hostedemail.com; dkim=pass header.d=linux.alibaba.com header.s=default header.b="VaC/kRe1"; spf=pass (imf07.hostedemail.com: domain of baolin.wang@linux.alibaba.com designates 115.124.30.110 as permitted sender) smtp.mailfrom=baolin.wang@linux.alibaba.com; dmarc=pass (policy=none) header.from=linux.alibaba.com ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=hostedemail.com; s=arc-20220608; t=1714985210; h=from:from:sender:reply-to:subject:subject:date:date: message-id:message-id:to:to:cc:cc:mime-version:mime-version: content-type:content-transfer-encoding:content-transfer-encoding: in-reply-to:in-reply-to:references:references:dkim-signature; bh=zEr/Eu5mgqpEki8XSyTh3V+LovJcAoeI3KW1/F4hkgg=; b=6034HVEC4nUAMupUvE/wVWOSw9LyyKfUsctOZs+dDW+tqY1/h7ok0IoxHcAU3/CZjZ4mu0 tLIsFMNwyCmDHLSuFw6F5JgJWstIVwdYio9XitaxiUro1M1dzEJgNJXs7IC1m/Yk+j/ory jIqKAB5uAfUwvZIvjNzjUFk3uah+M80= ARC-Authentication-Results: i=1; imf07.hostedemail.com; dkim=pass header.d=linux.alibaba.com header.s=default header.b="VaC/kRe1"; spf=pass (imf07.hostedemail.com: domain of baolin.wang@linux.alibaba.com designates 115.124.30.110 as permitted sender) smtp.mailfrom=baolin.wang@linux.alibaba.com; dmarc=pass (policy=none) header.from=linux.alibaba.com ARC-Seal: i=1; s=arc-20220608; d=hostedemail.com; t=1714985210; a=rsa-sha256; cv=none; b=EuoUVUaiEq5CdaqAqU3+bKhoX/wirY6z02ttAg+XsC/Jp2r5M7xMRJ/vUyISBClH+cSjz9 5Ty6iQmVdymYdY02JadgCJKIZ0kAqC02ELeMBDYr/OAPbjUvkh8IDCTd+LgKQkN43gML5d i5/eZEcZ/SZlHE85+g2/SLuQlPKd7Mo= DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=linux.alibaba.com; s=default; t=1714985207; h=From:To:Subject:Date:Message-Id:MIME-Version; bh=zEr/Eu5mgqpEki8XSyTh3V+LovJcAoeI3KW1/F4hkgg=; b=VaC/kRe1uXXZZhETUNZOyPdATJLv9wdN9iawFkKUMls4e7lUzINH4uM0PxZENld2U6PYzjxI7QUSAKP0Az7SdKF6WA6ea1z1iMbU4a+R0Od02k2s4ZvGl6Pkke/GkaIFIoCl7ij1a0NoZcbMhyA1FELXiH5ihqdVzv3+OtK4feY= X-Alimail-AntiSpam: AC=PASS;BC=-1|-1;BR=01201311R101e4;CH=green;DM=||false|;DS=||;FP=0|-1|-1|-1|0|-1|-1|-1;HT=maildocker-contentspam033022160150;MF=baolin.wang@linux.alibaba.com;NM=1;PH=DS;RN=14;SR=0;TI=SMTPD_---0W5weiKG_1714985205; Received: from localhost(mailfrom:baolin.wang@linux.alibaba.com fp:SMTPD_---0W5weiKG_1714985205) by smtp.aliyun-inc.com; Mon, 06 May 2024 16:46:46 +0800 From: Baolin Wang To: akpm@linux-foundation.org, hughd@google.com Cc: willy@infradead.org, david@redhat.com, ioworker0@gmail.com, wangkefeng.wang@huawei.com, ying.huang@intel.com, 21cnbao@gmail.com, ryan.roberts@arm.com, shy828301@gmail.com, ziy@nvidia.com, baolin.wang@linux.alibaba.com, linux-mm@kvack.org, linux-kernel@vger.kernel.org Subject: [PATCH 2/8] mm: memory: extend finish_fault() to support large folio Date: Mon, 6 May 2024 16:46:26 +0800 Message-Id: X-Mailer: git-send-email 2.39.3 In-Reply-To: References: MIME-Version: 1.0 X-Rspamd-Queue-Id: 584464000A X-Rspam-User: X-Rspamd-Server: rspam04 X-Stat-Signature: 1d7ysk6wm9ouu71oq8anbc6fhyy5bd1q X-HE-Tag: 1714985209-3402 X-HE-Meta: U2FsdGVkX19I7TloFPKUMudz4wFycH+EPmSX9BKy8Gn9nFQT3hCWD4eOHOKp/Qzj7ydug+QC/Id7Lemi3qeWKmgTpUUsDyU37AIFAPyReiLS7px5LDHemS2CLxLx196P0W8KKjGkKMnosEpHed3NW427cG6Sx9PQuiCZqWSuSmr9Uhf8yteNCu6ceR1i4e1ing2wDvhS8mkM6iV5BFMANHrKygJiP+KPLSQJyq3tYyWvCkjokTSxvXZT5aXUoEZI4EyYnVBnHob9F75NtJSDAyTKrWdkzxG8nw+1tZayQ65FHnIB49rmCOPWnci/K/luzhN1c0LavxXf5HR5MvmHMKFmgXS+pOFuLk3bANZAC8/jUNQgWAIDRbe6s0VKvxW/8XMV/mWKOSvUOyTaTULh8NaKVre+DUf3Qp55SxHn8eWnCkSeXmT7hFW5V+TWJPzqPmSs/ldElWm1vsGtwdBFWEcOcPx6msMtC/+w+UH6GSHhxuCeWEZU28T9X20GCVJ85c1aZO3JhkeaXGd7t7s/WRLA5Mp/MjBCjE23PVHnaGWAc6YpP8oLdzgYxwPgqkVQRpqpW9mT4atAeb2Ez3CAbwxJExCi5PqPLrT0u7ccu+ZhiS3S9rqDYozk7bqazJSCTFfVHy0o5M3tsjPc8zsJ2pDGJLhQQZTNTNnl+RsuLvHuXd7sbAlDM7JJuLE53hMQkU6mpbMMOpUdQpqN8hXIRkbEW5QDDtPqtJLdH2CeOfhrg6QevZJSNi7uFhdfX5ua/0H1+/impSRU6LMAZbudn+nryvaqt1Fl5OluMc1rMFhtQNcdCxkMkxLVz+Q3hM+CTCuZ8r0f84ocvWUV8tBmEsgHb8waeeBSGbpfypM3l9Ke3Wp625102xhkLiiFCtcj/zmz+p7d+XeXwYT12aVzC4ZcdoR5hek2GuJ5bPFPbb3Jdlg/njwM+e/LJvvBxgOQraIk4TvzD8SIXeXoNOr msHSE3xw i84f0qwn/1BqQKncc/npo/YSq2IMKEwa0Lq43I3KZxBK4JoMp/eypQUAobo7qgwcgoXN7KS+R3sT0dLhzO+Ji7aisYeeR8S0UpuB6YY3N7YOFdGwjQdvJ6eMRWt6Ad7uUv/jSoGrz1Q8E5184rszNEOfdlW/WWnivq6HPfPk95FYAbwhB/pR9gGsJ5C/woGs3V65FdliVrx7oeVJjS+4LW5BwaJUfKxFv+Iee24zu2eC5BVcdIcE4DJXF5e20yeyWE7U+dWOV6aypedCaLSRilnhOnrAnj2Hiqol/bWFB3KkfC3t9MMbQLhdOjOBBmIPC7icI X-Bogosity: Ham, tests=bogofilter, spamicity=0.000000, version=1.2.4 Sender: owner-linux-mm@kvack.org Precedence: bulk X-Loop: owner-majordomo@kvack.org List-ID: List-Subscribe: List-Unsubscribe: Add large folio mapping establishment support for finish_fault() as a preparation, to support multi-size THP allocation of anonymous shmem pages in the following patches. Signed-off-by: Baolin Wang --- mm/memory.c | 43 +++++++++++++++++++++++++++++++++---------- 1 file changed, 33 insertions(+), 10 deletions(-) diff --git a/mm/memory.c b/mm/memory.c index eea6e4984eae..936377220b77 100644 --- a/mm/memory.c +++ b/mm/memory.c @@ -4747,9 +4747,12 @@ vm_fault_t finish_fault(struct vm_fault *vmf) { struct vm_area_struct *vma = vmf->vma; struct page *page; + struct folio *folio; vm_fault_t ret; bool is_cow = (vmf->flags & FAULT_FLAG_WRITE) && !(vma->vm_flags & VM_SHARED); + int type, nr_pages, i; + unsigned long addr = vmf->address; /* Did we COW the page? */ if (is_cow) @@ -4780,24 +4783,44 @@ vm_fault_t finish_fault(struct vm_fault *vmf) return VM_FAULT_OOM; } + folio = page_folio(page); + nr_pages = folio_nr_pages(folio); + + if (unlikely(userfaultfd_armed(vma))) { + nr_pages = 1; + } else if (nr_pages > 1) { + unsigned long start = ALIGN_DOWN(vmf->address, nr_pages * PAGE_SIZE); + unsigned long end = start + nr_pages * PAGE_SIZE; + + /* In case the folio size in page cache beyond the VMA limits. */ + addr = max(start, vma->vm_start); + nr_pages = (min(end, vma->vm_end) - addr) >> PAGE_SHIFT; + + page = folio_page(folio, (addr - start) >> PAGE_SHIFT); + } vmf->pte = pte_offset_map_lock(vma->vm_mm, vmf->pmd, - vmf->address, &vmf->ptl); + addr, &vmf->ptl); if (!vmf->pte) return VM_FAULT_NOPAGE; /* Re-check under ptl */ - if (likely(!vmf_pte_changed(vmf))) { - struct folio *folio = page_folio(page); - int type = is_cow ? MM_ANONPAGES : mm_counter_file(folio); - - set_pte_range(vmf, folio, page, 1, vmf->address); - add_mm_counter(vma->vm_mm, type, 1); - ret = 0; - } else { - update_mmu_tlb(vma, vmf->address, vmf->pte); + if (nr_pages == 1 && unlikely(vmf_pte_changed(vmf))) { + update_mmu_tlb(vma, addr, vmf->pte); + ret = VM_FAULT_NOPAGE; + goto unlock; + } else if (nr_pages > 1 && !pte_range_none(vmf->pte, nr_pages)) { + for (i = 0; i < nr_pages; i++) + update_mmu_tlb(vma, addr + PAGE_SIZE * i, vmf->pte + i); ret = VM_FAULT_NOPAGE; + goto unlock; } + set_pte_range(vmf, folio, page, nr_pages, addr); + type = is_cow ? MM_ANONPAGES : mm_counter_file(folio); + add_mm_counter(vma->vm_mm, type, nr_pages); + ret = 0; + +unlock: pte_unmap_unlock(vmf->pte, vmf->ptl); return ret; } From patchwork Mon May 6 08:46:27 2024 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Baolin Wang X-Patchwork-Id: 13655145 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from kanga.kvack.org (kanga.kvack.org [205.233.56.17]) by smtp.lore.kernel.org (Postfix) with ESMTP id 84E6FC10F16 for ; Mon, 6 May 2024 08:46:58 +0000 (UTC) Received: by kanga.kvack.org (Postfix) id B33626B0085; Mon, 6 May 2024 04:46:53 -0400 (EDT) Received: by kanga.kvack.org (Postfix, from userid 40) id A1F576B0087; Mon, 6 May 2024 04:46:53 -0400 (EDT) X-Delivered-To: int-list-linux-mm@kvack.org Received: by kanga.kvack.org (Postfix, from userid 63042) id 8E6B76B0088; Mon, 6 May 2024 04:46:53 -0400 (EDT) X-Delivered-To: linux-mm@kvack.org Received: from relay.hostedemail.com (smtprelay0015.hostedemail.com [216.40.44.15]) by kanga.kvack.org (Postfix) with ESMTP id 6C9A46B0085 for ; Mon, 6 May 2024 04:46:53 -0400 (EDT) Received: from smtpin21.hostedemail.com (a10.router.float.18 [10.200.18.1]) by unirelay06.hostedemail.com (Postfix) with ESMTP id 2F85CA1606 for ; Mon, 6 May 2024 08:46:53 +0000 (UTC) X-FDA: 82087340706.21.0CBA338 Received: from out30-113.freemail.mail.aliyun.com (out30-113.freemail.mail.aliyun.com [115.124.30.113]) by imf06.hostedemail.com (Postfix) with ESMTP id 256E4180012 for ; Mon, 6 May 2024 08:46:50 +0000 (UTC) Authentication-Results: imf06.hostedemail.com; dkim=pass header.d=linux.alibaba.com header.s=default header.b=T4eV50PJ; spf=pass (imf06.hostedemail.com: domain of baolin.wang@linux.alibaba.com designates 115.124.30.113 as permitted sender) smtp.mailfrom=baolin.wang@linux.alibaba.com; dmarc=pass (policy=none) header.from=linux.alibaba.com ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=hostedemail.com; s=arc-20220608; t=1714985211; h=from:from:sender:reply-to:subject:subject:date:date: message-id:message-id:to:to:cc:cc:mime-version:mime-version: content-type:content-transfer-encoding:content-transfer-encoding: in-reply-to:in-reply-to:references:references:dkim-signature; bh=e+vn44n5PSOyoo2A5G9/u00K/6MLWmZWHwZJ5wJKuyA=; b=wkrjcDnLDDYcbOgGRnD6JZUG4Vqvk7T6R4adXexzE6a7hzarYShucSx+xq5FID+FM0Aaz1 nQ/wX2TgtpieQRBNvHYubBxZPZzvLTvkCp3oLVi4neB4gwowXW0ZS0bCBlJ7Gfg8iuAW75 mS66QL/5Eekgs47RJYt7x7myxGQAvM8= ARC-Authentication-Results: i=1; imf06.hostedemail.com; dkim=pass header.d=linux.alibaba.com header.s=default header.b=T4eV50PJ; spf=pass (imf06.hostedemail.com: domain of baolin.wang@linux.alibaba.com designates 115.124.30.113 as permitted sender) smtp.mailfrom=baolin.wang@linux.alibaba.com; dmarc=pass (policy=none) header.from=linux.alibaba.com ARC-Seal: i=1; s=arc-20220608; d=hostedemail.com; t=1714985211; a=rsa-sha256; cv=none; b=SrYcOIKq+BzdHtNYMY+HRyc2L442BbUEiLjhbmz2QKk4txcXNU4IARrEq21IXcEUFq3pgX QY2AFmw2nacurZscGUjOF10NT6iNdg4kpZgcY/4RSHVYM3KrVLI3KzTEGHzs2Mewu/b7Ut IUTZHbvNS9JWv2GJm0Q13L0/NfxEtNQ= DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=linux.alibaba.com; s=default; t=1714985208; h=From:To:Subject:Date:Message-Id:MIME-Version; bh=e+vn44n5PSOyoo2A5G9/u00K/6MLWmZWHwZJ5wJKuyA=; b=T4eV50PJV3ZiYLN+BHbrXRlmnMSpFm8kK/ej/VeLnj22cuIvJMBGyHkvj4qpTsCJ8OjEObD2CWubAzKBXE9E52xkVNjJyyBYkKNq4Z7WHje4H1s+R8lBXlfKK3UMltf1aKY+poGxPWWW9S8gmuqmvUXf+Y5unjT+U6SKgtJG3gc= X-Alimail-AntiSpam: AC=PASS;BC=-1|-1;BR=01201311R721e4;CH=green;DM=||false|;DS=||;FP=0|-1|-1|-1|0|-1|-1|-1;HT=maildocker-contentspam033037067111;MF=baolin.wang@linux.alibaba.com;NM=1;PH=DS;RN=14;SR=0;TI=SMTPD_---0W5wQIAj_1714985206; Received: from localhost(mailfrom:baolin.wang@linux.alibaba.com fp:SMTPD_---0W5wQIAj_1714985206) by smtp.aliyun-inc.com; Mon, 06 May 2024 16:46:47 +0800 From: Baolin Wang To: akpm@linux-foundation.org, hughd@google.com Cc: willy@infradead.org, david@redhat.com, ioworker0@gmail.com, wangkefeng.wang@huawei.com, ying.huang@intel.com, 21cnbao@gmail.com, ryan.roberts@arm.com, shy828301@gmail.com, ziy@nvidia.com, baolin.wang@linux.alibaba.com, linux-mm@kvack.org, linux-kernel@vger.kernel.org Subject: [PATCH 3/8] mm: shmem: add an 'order' parameter for shmem_alloc_hugefolio() Date: Mon, 6 May 2024 16:46:27 +0800 Message-Id: <1484509ad5a9751b449e01fb695c4e7634be14bf.1714978902.git.baolin.wang@linux.alibaba.com> X-Mailer: git-send-email 2.39.3 In-Reply-To: References: MIME-Version: 1.0 X-Rspamd-Queue-Id: 256E4180012 X-Rspam-User: X-Rspamd-Server: rspam04 X-Stat-Signature: 3195jsphcgfjh5k5pkg6jocqu3ce4bnf X-HE-Tag: 1714985210-191679 X-HE-Meta: U2FsdGVkX181VBxB/HurniHfCWfVGjc63bjo0V0mZZRFpRYQaRJXG/A3d3W3sJtKmwz2PvtqGG9HtGqw+bvgWSfRwTcBPc50XauIDrMsf6xwW887QAeWFPuLgnQ6n3eh0dWqCE8WsfRSexsSHfCuTr2yZz1uTpjGPUKrzhaDV7+YxRrPPR6cf1gMRfjk6OgDckDRYYxxmhUC2/xVrskGkqAguMi8nJslls9zNURQBWSwfj9FOtrUzr+gFd3e+mXllmAT4LqINV9CL2NwU+bi2pwR1JAaDOaLZkvh5zZ/N1HFuRyh1xeAZ/IzllrGwgscqd+tSDfjALmjY2fYzsKL22BQ0kCkSAODAxPXqIvjF3kOjmbA1adF+Nq3CM3Kmg4GeG++C3t5f8ja9HnzNu1gjIwAaZ1ptKou2Em0OFqYdu62hKrEVYmR0rGz5B+QE35CNpzhhyAu69bHaoAoascQKzpBdc8COjkzJ2/jmwqj/70vMC+xFU2kvHEGIHIQ20BixUqxYj1w8GRSuF87ijj70cMWZm/RtyqpKCmhNSGODUheP/zImLos/XNJoPTF9J/85HtOL42ELNwnVY5EZIWhQXtoptqzy/9yNM4yP8Z7uLQlmEVqnlrmnbm1RzRTNgH8V7iBgj6Hi1kBVYK9FEdil8XjS5PrG4/M8BcEFgc5udknS/KMhsypNqHSgdNSOGXwuhNma88w7/FqigMSuB8t0PQ20TDS7t71mYJ6GN6T8dYfA+rJNR9dKVa4Hun39FkbqO3AKVsnKqkBHgMbJDqXB1QNT5mmCqXoN8QHo38MWzv8tVUtO2GBbG/mCXW2GSJDKMZZuAxJjG0+3rMl1LqSkm61DI7hM+V9rdbBwu3IoNOtDz+OJ42HF7ES/e/lC+H0TBEWX/y0EWO08WMRuAO532g1q0boef19uDseVxQN+W3iadhwRxmgclzTSE6jhfV386nzdfa4l2J5p8Lcsaz f3cxyDDX XwO/Eo/VcWx2wzmzu0oYq5b89gi/xHqrYFRo9sIShJJZEljMd4Y1cbmKaBhwl7oTJd+yZ+vNdLXGpIyhVddvrNgt/y0rP6UhXLixU+Ryw1AOYqhyKOY290LPRy8BBRxf98LaBvTAEgEwXFx2vBUtEA1NjkwngWFTa0PTRzZDmGb+B2PtO6Xrlr7RP7iWTokll147JP+ymaf+SGT5XEo6rC9AO82z3D5zgiBN0BX+rC25SzIyQr7MFAVCeJarX7dJTgMUw1iBd0sgigq6ju+zBdFVQmLjlvubkNnnL X-Bogosity: Ham, tests=bogofilter, spamicity=0.000000, version=1.2.4 Sender: owner-linux-mm@kvack.org Precedence: bulk X-Loop: owner-majordomo@kvack.org List-ID: List-Subscribe: List-Unsubscribe: Add a new parameter to specify the huge page order for shmem_alloc_hugefolio(), as a preparation to supoort mTHP. Signed-off-by: Baolin Wang --- mm/shmem.c | 8 ++++---- 1 file changed, 4 insertions(+), 4 deletions(-) diff --git a/mm/shmem.c b/mm/shmem.c index fa2a0ed97507..e4483c4596a8 100644 --- a/mm/shmem.c +++ b/mm/shmem.c @@ -1604,14 +1604,14 @@ static gfp_t limit_gfp_mask(gfp_t huge_gfp, gfp_t limit_gfp) } static struct folio *shmem_alloc_hugefolio(gfp_t gfp, - struct shmem_inode_info *info, pgoff_t index) + struct shmem_inode_info *info, pgoff_t index, int order) { struct mempolicy *mpol; pgoff_t ilx; struct page *page; - mpol = shmem_get_pgoff_policy(info, index, HPAGE_PMD_ORDER, &ilx); - page = alloc_pages_mpol(gfp, HPAGE_PMD_ORDER, mpol, ilx, numa_node_id()); + mpol = shmem_get_pgoff_policy(info, index, order, &ilx); + page = alloc_pages_mpol(gfp, order, mpol, ilx, numa_node_id()); mpol_cond_put(mpol); return page_rmappable_folio(page); @@ -1660,7 +1660,7 @@ static struct folio *shmem_alloc_and_add_folio(gfp_t gfp, index + HPAGE_PMD_NR - 1, XA_PRESENT)) return ERR_PTR(-E2BIG); - folio = shmem_alloc_hugefolio(gfp, info, index); + folio = shmem_alloc_hugefolio(gfp, info, index, HPAGE_PMD_ORDER); if (!folio) count_vm_event(THP_FILE_FALLBACK); } else { From patchwork Mon May 6 08:46:28 2024 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Baolin Wang X-Patchwork-Id: 13655146 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from kanga.kvack.org (kanga.kvack.org [205.233.56.17]) by smtp.lore.kernel.org (Postfix) with ESMTP id D9D1AC25B5C for ; Mon, 6 May 2024 08:47:00 +0000 (UTC) Received: by kanga.kvack.org (Postfix) id ABD7B6B0088; Mon, 6 May 2024 04:46:55 -0400 (EDT) Received: by kanga.kvack.org (Postfix, from userid 40) id A48486B0089; Mon, 6 May 2024 04:46:55 -0400 (EDT) X-Delivered-To: int-list-linux-mm@kvack.org Received: by kanga.kvack.org (Postfix, from userid 63042) id 89A1A6B008A; Mon, 6 May 2024 04:46:55 -0400 (EDT) X-Delivered-To: linux-mm@kvack.org Received: from relay.hostedemail.com (smtprelay0011.hostedemail.com [216.40.44.11]) by kanga.kvack.org (Postfix) with ESMTP id 69FD36B0088 for ; Mon, 6 May 2024 04:46:55 -0400 (EDT) Received: from smtpin18.hostedemail.com (a10.router.float.18 [10.200.18.1]) by unirelay02.hostedemail.com (Postfix) with ESMTP id E7A5E1206EE for ; Mon, 6 May 2024 08:46:54 +0000 (UTC) X-FDA: 82087340748.18.670AD2F Received: from out30-110.freemail.mail.aliyun.com (out30-110.freemail.mail.aliyun.com [115.124.30.110]) by imf18.hostedemail.com (Postfix) with ESMTP id BB1041C0003 for ; Mon, 6 May 2024 08:46:52 +0000 (UTC) Authentication-Results: imf18.hostedemail.com; dkim=pass header.d=linux.alibaba.com header.s=default header.b=IY9VGJ6D; dmarc=pass (policy=none) header.from=linux.alibaba.com; spf=pass (imf18.hostedemail.com: domain of baolin.wang@linux.alibaba.com designates 115.124.30.110 as permitted sender) smtp.mailfrom=baolin.wang@linux.alibaba.com ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=hostedemail.com; s=arc-20220608; t=1714985213; h=from:from:sender:reply-to:subject:subject:date:date: message-id:message-id:to:to:cc:cc:mime-version:mime-version: content-type:content-transfer-encoding:content-transfer-encoding: in-reply-to:in-reply-to:references:references:dkim-signature; bh=aUksnnssWSvRHZu6SVpR2z8swC3sgIQdPUjmp/ogrf0=; b=xjOU8ZMivlMi4n5eOiRb2ysPpkEWlFApqBJdw56Vx5khH7/p7yg/eNYuOndr5X5AIAFjC6 2B7QbV4kY9Bqe+M07ddWfTBqLTXLiA8WCZQb57IX1gZ5y3ncr0QsU83vfP48Ob8xrQwz8F K0w1Sc4kH/VgYKoW8h9OUI1GcB54C3k= ARC-Authentication-Results: i=1; imf18.hostedemail.com; dkim=pass header.d=linux.alibaba.com header.s=default header.b=IY9VGJ6D; dmarc=pass (policy=none) header.from=linux.alibaba.com; spf=pass (imf18.hostedemail.com: domain of baolin.wang@linux.alibaba.com designates 115.124.30.110 as permitted sender) smtp.mailfrom=baolin.wang@linux.alibaba.com ARC-Seal: i=1; s=arc-20220608; d=hostedemail.com; t=1714985213; a=rsa-sha256; cv=none; b=EVAIjdfkReJ75Q6IcRzJQWSEF3dI629K8wdWIQUKbOd4zlFanJHOsOeuRxhIbi7gp47cmX reH45NcfA6eGyKLy9VluOAX5InBCXf0HsOdhfq8PWV5oavCpEabpxxyiZj778wjzuvZzS6 P0GRpT15uMiWApcDxmnBCl/xIET+QK0= DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=linux.alibaba.com; s=default; t=1714985210; h=From:To:Subject:Date:Message-Id:MIME-Version; bh=aUksnnssWSvRHZu6SVpR2z8swC3sgIQdPUjmp/ogrf0=; b=IY9VGJ6DM3ecPUsrIW5sU7/DsX6UCulXdLbmY8KhuODns0Ub09PNSB9lgwqSOxnE+UR/VIC4DogAkOp+BrT1VoWPMC9XvNTMqYqD+KK3q+cThQalWqqaeymPJKcMPBuazejZ/N6eJseKjIes6n4n4fI7WRHYfkKWTbxtrF+2HX8= X-Alimail-AntiSpam: AC=PASS;BC=-1|-1;BR=01201311R101e4;CH=green;DM=||false|;DS=||;FP=0|-1|-1|-1|0|-1|-1|-1;HT=maildocker-contentspam033068173054;MF=baolin.wang@linux.alibaba.com;NM=1;PH=DS;RN=14;SR=0;TI=SMTPD_---0W5w3v7-_1714985207; Received: from localhost(mailfrom:baolin.wang@linux.alibaba.com fp:SMTPD_---0W5w3v7-_1714985207) by smtp.aliyun-inc.com; Mon, 06 May 2024 16:46:48 +0800 From: Baolin Wang To: akpm@linux-foundation.org, hughd@google.com Cc: willy@infradead.org, david@redhat.com, ioworker0@gmail.com, wangkefeng.wang@huawei.com, ying.huang@intel.com, 21cnbao@gmail.com, ryan.roberts@arm.com, shy828301@gmail.com, ziy@nvidia.com, baolin.wang@linux.alibaba.com, linux-mm@kvack.org, linux-kernel@vger.kernel.org Subject: [PATCH 4/8] mm: shmem: add THP validation for PMD-mapped THP related statistics Date: Mon, 6 May 2024 16:46:28 +0800 Message-Id: X-Mailer: git-send-email 2.39.3 In-Reply-To: References: MIME-Version: 1.0 X-Rspamd-Server: rspam07 X-Rspamd-Queue-Id: BB1041C0003 X-Stat-Signature: 7d5hkjkbs53cnueqitiy7yqnyxbx3giq X-Rspam-User: X-HE-Tag: 1714985212-391604 X-HE-Meta: U2FsdGVkX1/Yh3vRs8Wuvk8Dai5I32P72pH30SFaqfAv/GpS6HPP4E/hY4HVYf5B1ODKyFVU/5SeTs3dBhTDPhYdiGBFFLA+bfzClHrcnL0K1nLSCIWsjCtf23J5aE8p6QdgCuy1hOPc63c3XEVLdd0Db83jDPvyUmiHJhkKuFY+F3yHWEvIKgQkLuU2C8zGp2PsyqmQaQ/R1HepMvqOOp8MzwbvlJ2/iIINuVfbz5k5u12UOLnyJCzcZRZxDSROnV4IuW7VA+C1At8JIvRweRGksIou2UdO+A/fwud4w58o+j4Ch0wPm49tF5ugkHzYatssdeqgxEY68WMJ9znhHzswPSLLW+/ayEy88JHbSvlvnaz16pnu5H9aXqzq2fC+vUPRcMI/iKeSFDXoUSC8qU7ihE3Nyecngu+SwTSfmHMQcRV+Eh33ZFHqvxUged23IrAU52Sfs7dxX1V94StQyW4F+GnJwbmKPK+0FsPcSiG+fZwDsVTY1rVaYgszk9d+48Dog/n7+9GyoJgWhx1Wfn/CDuPyJ5c20JdTqv/XANPjWnegB0Rm/SveJDnsc6qHPiS7BbJR63MuSdkpyYUcJdAxsnceohgawPFGZTBnAIyA7P/3E1jji7MlmBAfpuOWALUKsUdNGNSFZb8c4zbzVu/mUMlk0KWgk9Tivq6g3eDyn4sDtUZwWCjJfdYYRlVWIcCN++GeNRZPj5iV/a1zit9eVrxWvr4JB/qDNEiIrYoq+41GPP5K3k+EJAPCp0eVAfU2wBBVWk8NfW+lrDLs1paLwKO6SA47cxwP6bmmrorofGkYigNnVPUtj/b4oRO24kOdoP1WlhztEzDYeDSPzH1QAB4rRAj0LZ/Zs4R5pmS2dTfCSMP18DWn5/xJfB9ZATtv6Otq12g+GKF69zHyqFHUOyLj+AATqoqcudjjdBZ5atr+NbIAqfIaIB/8ZbYqlDnPGeI0v1h0wYzUl/O FvU5kLpx lvAwfdjZw83FQf/+0rvYOtQrZyyjvz2yvVZGgViP1u9dDEwB7YdWO6rbS/6LRexvsGDzjkQTo3WCSleO3guf4my1SYv7lqKraD8QOLQdz4rw7koD+g0E2vgXf26fEp7CfyoKHANl6w9QlvHnOFYro8dBhO2yO3+4jJlpgvjJ/61Ali8d+MD7efQLiNRv/iCyLr5FTEXZNqJJcMQSHA7AZZRPcuPA0rPABBmm437WqoLHImYXMib4jOnj5Ww== X-Bogosity: Ham, tests=bogofilter, spamicity=0.000000, version=1.2.4 Sender: owner-linux-mm@kvack.org Precedence: bulk X-Loop: owner-majordomo@kvack.org List-ID: List-Subscribe: List-Unsubscribe: In order to extend support for mTHP, add THP validation for PMD-mapped THP related statistics to avoid statistical confusion. Signed-off-by: Baolin Wang Reviewed-by: Barry Song --- mm/shmem.c | 7 ++++--- 1 file changed, 4 insertions(+), 3 deletions(-) diff --git a/mm/shmem.c b/mm/shmem.c index e4483c4596a8..a383ea9a89a5 100644 --- a/mm/shmem.c +++ b/mm/shmem.c @@ -1661,7 +1661,7 @@ static struct folio *shmem_alloc_and_add_folio(gfp_t gfp, return ERR_PTR(-E2BIG); folio = shmem_alloc_hugefolio(gfp, info, index, HPAGE_PMD_ORDER); - if (!folio) + if (!folio && pages == HPAGE_PMD_NR) count_vm_event(THP_FILE_FALLBACK); } else { pages = 1; @@ -1679,7 +1679,7 @@ static struct folio *shmem_alloc_and_add_folio(gfp_t gfp, if (xa_find(&mapping->i_pages, &index, index + pages - 1, XA_PRESENT)) { error = -EEXIST; - } else if (huge) { + } else if (pages == HPAGE_PMD_NR) { count_vm_event(THP_FILE_FALLBACK); count_vm_event(THP_FILE_FALLBACK_CHARGE); } @@ -2045,7 +2045,8 @@ static int shmem_get_folio_gfp(struct inode *inode, pgoff_t index, folio = shmem_alloc_and_add_folio(huge_gfp, inode, index, fault_mm, true); if (!IS_ERR(folio)) { - count_vm_event(THP_FILE_ALLOC); + if (folio_test_pmd_mappable(folio)) + count_vm_event(THP_FILE_ALLOC); goto alloced; } if (PTR_ERR(folio) == -EEXIST) From patchwork Mon May 6 08:46:29 2024 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Baolin Wang X-Patchwork-Id: 13655147 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from kanga.kvack.org (kanga.kvack.org [205.233.56.17]) by smtp.lore.kernel.org (Postfix) with ESMTP id 817EBC10F16 for ; Mon, 6 May 2024 08:47:03 +0000 (UTC) Received: by kanga.kvack.org (Postfix) id CBF5C6B0089; Mon, 6 May 2024 04:46:56 -0400 (EDT) Received: by kanga.kvack.org (Postfix, from userid 40) id C42106B008A; Mon, 6 May 2024 04:46:56 -0400 (EDT) X-Delivered-To: int-list-linux-mm@kvack.org Received: by kanga.kvack.org (Postfix, from userid 63042) id A97096B008C; Mon, 6 May 2024 04:46:56 -0400 (EDT) X-Delivered-To: linux-mm@kvack.org Received: from relay.hostedemail.com (smtprelay0014.hostedemail.com [216.40.44.14]) by kanga.kvack.org (Postfix) with ESMTP id 7B57B6B0089 for ; Mon, 6 May 2024 04:46:56 -0400 (EDT) Received: from smtpin08.hostedemail.com (a10.router.float.18 [10.200.18.1]) by unirelay06.hostedemail.com (Postfix) with ESMTP id 38658A1589 for ; Mon, 6 May 2024 08:46:56 +0000 (UTC) X-FDA: 82087340832.08.6530EBC Received: from out30-99.freemail.mail.aliyun.com (out30-99.freemail.mail.aliyun.com [115.124.30.99]) by imf17.hostedemail.com (Postfix) with ESMTP id 303F040011 for ; Mon, 6 May 2024 08:46:53 +0000 (UTC) Authentication-Results: imf17.hostedemail.com; dkim=pass header.d=linux.alibaba.com header.s=default header.b=Ub8G9q8o; spf=pass (imf17.hostedemail.com: domain of baolin.wang@linux.alibaba.com designates 115.124.30.99 as permitted sender) smtp.mailfrom=baolin.wang@linux.alibaba.com; dmarc=pass (policy=none) header.from=linux.alibaba.com ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=hostedemail.com; s=arc-20220608; t=1714985214; h=from:from:sender:reply-to:subject:subject:date:date: message-id:message-id:to:to:cc:cc:mime-version:mime-version: content-type:content-transfer-encoding:content-transfer-encoding: in-reply-to:in-reply-to:references:references:dkim-signature; bh=vbAZ9jfAMkqVV3Kg32TDOediCKK1trUe5zVBa6e0pWM=; b=y0c+MDK14YVFKPbSkMKOnmBS+Jmqc8Dxpt7eM3G/ca5hAiZfkmrdQOMBPn4M7SMcIS+Vkf 8lr9/ilsr8M8B4loRfDieLNb0utZWdHe+EJNNOLXJZqYaDUtMg6B//RdbyPgppKhEKcnC4 u91CCOI/ACDWhb+qSzBFDas5DydrnbI= ARC-Seal: i=1; s=arc-20220608; d=hostedemail.com; t=1714985214; a=rsa-sha256; cv=none; b=P2GAwUh14ZgfQpk+oEzoTPJDUSKs0SozeUx6JnBuQKW/fM7hd/KhBcisUz3tvgvqBctbAH kuvRoWNZ9vhdX9vzMOcAQKm8OJan9q0shkEyHXG0QRNiqLC5+q3mK/OP1q/UwTJ/uEqb7c eGfuc/XtGVsZGPl7jeF+W2wthSmGRk8= ARC-Authentication-Results: i=1; imf17.hostedemail.com; dkim=pass header.d=linux.alibaba.com header.s=default header.b=Ub8G9q8o; spf=pass (imf17.hostedemail.com: domain of baolin.wang@linux.alibaba.com designates 115.124.30.99 as permitted sender) smtp.mailfrom=baolin.wang@linux.alibaba.com; dmarc=pass (policy=none) header.from=linux.alibaba.com DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=linux.alibaba.com; s=default; t=1714985211; h=From:To:Subject:Date:Message-Id:MIME-Version; bh=vbAZ9jfAMkqVV3Kg32TDOediCKK1trUe5zVBa6e0pWM=; b=Ub8G9q8ogrbtZbAejVHsJaQ5hy3TmGBXla5qxvJxHTjlKWqFJdq+vSRTfo8cL7BU0MWwCfpiXQHF006MHYP4w7YgSWqBSnEkvT2MWgrKE2BOQ0sJPqXEX1oeWIIBpPK1Ln50/4/hTiHQXUUFp3/7id1wSawPSxzWNcGS/Bs9gzc= X-Alimail-AntiSpam: AC=PASS;BC=-1|-1;BR=01201311R181e4;CH=green;DM=||false|;DS=||;FP=0|-1|-1|-1|0|-1|-1|-1;HT=maildocker-contentspam033045046011;MF=baolin.wang@linux.alibaba.com;NM=1;PH=DS;RN=14;SR=0;TI=SMTPD_---0W5w3v7S_1714985209; Received: from localhost(mailfrom:baolin.wang@linux.alibaba.com fp:SMTPD_---0W5w3v7S_1714985209) by smtp.aliyun-inc.com; Mon, 06 May 2024 16:46:50 +0800 From: Baolin Wang To: akpm@linux-foundation.org, hughd@google.com Cc: willy@infradead.org, david@redhat.com, ioworker0@gmail.com, wangkefeng.wang@huawei.com, ying.huang@intel.com, 21cnbao@gmail.com, ryan.roberts@arm.com, shy828301@gmail.com, ziy@nvidia.com, baolin.wang@linux.alibaba.com, linux-mm@kvack.org, linux-kernel@vger.kernel.org Subject: [PATCH 5/8] mm: shmem: add multi-size THP sysfs interface for anonymous shmem Date: Mon, 6 May 2024 16:46:29 +0800 Message-Id: <6b4afed1ef26dbd08ae9ec58449b329564dcef3e.1714978902.git.baolin.wang@linux.alibaba.com> X-Mailer: git-send-email 2.39.3 In-Reply-To: References: MIME-Version: 1.0 X-Rspam-User: X-Stat-Signature: 9yqnufi3wz4xg4x6a9byn8ybhkrhgdpn X-Rspamd-Server: rspam11 X-Rspamd-Queue-Id: 303F040011 X-HE-Tag: 1714985213-146569 X-HE-Meta: U2FsdGVkX1+X8moZw7YP3BOoF3ggPDVK/qNqpvh/YLZtVlV7Y2nrr6ur+OdWGUV2LpZzIE7lMG8mRHb4w/N+es4gnh5bLbWaG3rQV0icwbmxZzN5qppSiDhYiqaoa8hiG4mDnVgr2XD3zp+ey0fF6uVe136iGAwHFIsruWpxkMjZXs4mwgWSAKR2/WwVjBxhJ73F63jx+0VAsevNLgm9jag357d+LMH4Rb5lLdYHzbOpp1UXMC/DiaWw1fH7IwYvIgmSojyZThWMmfp3Iru74DWDoa8Oh+2tNHRJf2v6isWV9sMqT8VOKLksW9MzBfBpFOckLgHnOGIqBZq2l33p0AsWc3v2U0Fyddo/GO6s5vpnbTcAWfyBu3AztSm8wy6vDzo9mq70T/cTP6lZr1ONolulLLdkmg2zWyHXFrYawQRULnTsWFxEsDxrJ7hjFYA33ZjCPmyZ/jHilG4HA+4XhJcvaVbWBzbZInio5rjGl5cm8RttKOAlsml6eucrqaogV1J8LQm15rT/4Orwx8owFsojRZ/xxMn8Bnzd6BWrSyskBolVIU3KbGE+o1ID6ECE82XvIYHXxqfenpkN9zb/IXyyh0ZhJzjYJU60zGoK122+nqxN2PpP9AFRvFt1mafKSoj1yJC9Ef1dZGHAQ1MLg53P/0aeHaNH7lX6s2uTidDGzi0w/RkzG0ViKNtQ45M23lj38GymsUXat40NIU8t//hk1qmUmd6opxhPR1jiVZIBUr+4IS89mtFGpDiq1O6e1ohFasJuhcjr9aSWcoWlWjFCyG4MBmKmrKvxYKET+cfqpt6mqKShBtyes0d7YTYZK8CXAVKBSuCmUFPHS5HCemmjc3VMyFbnPH6b3zWVq/Ho36EenwIyQE43A9BIx+E0wPafnmgIT4zvTyHReYOSodQPfZnl9/cG25siSYAfYJBRmuJ8md7wzJzADd9j0451T0WMwWMdBLZDciBE6Z8 FVska/ew bu2Yt9zPIq93z0qSBXkR1eOglCMy0NGoPBX65O/KFs/AR2xcOYt2gC0vn1qJQrshQdFnp8cgcDJE7Jv9Wgot3p9HFfGE115LZU2eNlzIjKaM7Wrd1hVDBfErWPIpr49b73dRIl+zrTbraD91ste2gUb6WyukZ1++SW67BKFw5aCQF44/X4xyV01UCYo9p5Yzt+kAeNIoP9sbz8sUb80XXUa0y/SezP7Oy7Ej95wKh8SCSb+g= X-Bogosity: Ham, tests=bogofilter, spamicity=0.000000, version=1.2.4 Sender: owner-linux-mm@kvack.org Precedence: bulk X-Loop: owner-majordomo@kvack.org List-ID: List-Subscribe: List-Unsubscribe: To support the use of mTHP with anonymous shmem, add a new sysfs interface 'shmem_enabled' in the '/sys/kernel/mm/transparent_hugepage/hugepages-kB/' directory for each mTHP to control whether shmem is enabled for that mTHP, with a value similar to the top level 'shmem_enabled', which can be set to: "always", "inherit (to inherit the top level setting)", "within_size", "advise", "never", "deny", "force". These values follow the same semantics as the top level, except the 'deny' is equivalent to 'never', and 'force' is equivalent to 'always' to keep compatibility. By default, PMD-sized hugepages have enabled="inherit" and all other hugepage sizes have enabled="never" for '/sys/kernel/mm/transparent_hugepage/hugepages-xxkB/shmem_enabled'. In addition, if top level value is 'force', then only PMD-sized hugepages have enabled="inherit", otherwise configuration will be failed and vice versa. That means now we will avoid using non-PMD sized THP to override the global huge allocation. Signed-off-by: Baolin Wang --- Documentation/admin-guide/mm/transhuge.rst | 29 +++++++ include/linux/huge_mm.h | 10 +++ mm/huge_memory.c | 11 +-- mm/shmem.c | 96 ++++++++++++++++++++++ 4 files changed, 138 insertions(+), 8 deletions(-) diff --git a/Documentation/admin-guide/mm/transhuge.rst b/Documentation/admin-guide/mm/transhuge.rst index 076443cc10a6..a28496e15bdb 100644 --- a/Documentation/admin-guide/mm/transhuge.rst +++ b/Documentation/admin-guide/mm/transhuge.rst @@ -332,6 +332,35 @@ deny force Force the huge option on for all - very useful for testing; +Anonymous shmem can also use "multi-size THP" (mTHP) by adding a new sysfs knob +to control mTHP allocation: /sys/kernel/mm/transparent_hugepage/hugepages-kB/shmem_enabled. +Its value for each mTHP is essentially consistent with the global setting, except +for the addition of 'inherit' to ensure compatibility with the global settings. +always + Attempt to allocate huge pages every time we need a new page; + +inherit + Inherit the top-level "shmem_enabled" value. By default, PMD-sized hugepages + have enabled="inherit" and all other hugepage sizes have enabled="never"; + +never + Do not allocate huge pages; + +within_size + Only allocate huge page if it will be fully within i_size. + Also respect fadvise()/madvise() hints; + +advise + Only allocate huge pages if requested with fadvise()/madvise(); + +deny + Has the same semantics as 'never', now mTHP allocation policy is only + used for anonymous shmem and no not override tmpfs. + +force + Has the same semantics as 'always', now mTHP allocation policy is only + used for anonymous shmem and no not override tmpfs. + Need of application restart =========================== diff --git a/include/linux/huge_mm.h b/include/linux/huge_mm.h index e49b56c40a11..dbd6b3f56210 100644 --- a/include/linux/huge_mm.h +++ b/include/linux/huge_mm.h @@ -6,6 +6,7 @@ #include #include /* only for vma_is_dax() */ +#include vm_fault_t do_huge_pmd_anonymous_page(struct vm_fault *vmf); int copy_huge_pmd(struct mm_struct *dst_mm, struct mm_struct *src_mm, @@ -63,6 +64,7 @@ ssize_t single_hugepage_flag_show(struct kobject *kobj, struct kobj_attribute *attr, char *buf, enum transparent_hugepage_flag flag); extern struct kobj_attribute shmem_enabled_attr; +extern struct kobj_attribute thpsize_shmem_enabled_attr; /* * Mask of all large folio orders supported for anonymous THP; all orders up to @@ -265,6 +267,14 @@ unsigned long thp_vma_allowable_orders(struct vm_area_struct *vma, return __thp_vma_allowable_orders(vma, vm_flags, tva_flags, orders); } +struct thpsize { + struct kobject kobj; + struct list_head node; + int order; +}; + +#define to_thpsize(kobj) container_of(kobj, struct thpsize, kobj) + enum mthp_stat_item { MTHP_STAT_ANON_FAULT_ALLOC, MTHP_STAT_ANON_FAULT_FALLBACK, diff --git a/mm/huge_memory.c b/mm/huge_memory.c index 9efb6fefc391..d3080a8843f2 100644 --- a/mm/huge_memory.c +++ b/mm/huge_memory.c @@ -449,14 +449,6 @@ static void thpsize_release(struct kobject *kobj); static DEFINE_SPINLOCK(huge_anon_orders_lock); static LIST_HEAD(thpsize_list); -struct thpsize { - struct kobject kobj; - struct list_head node; - int order; -}; - -#define to_thpsize(kobj) container_of(kobj, struct thpsize, kobj) - static ssize_t thpsize_enabled_show(struct kobject *kobj, struct kobj_attribute *attr, char *buf) { @@ -517,6 +509,9 @@ static struct kobj_attribute thpsize_enabled_attr = static struct attribute *thpsize_attrs[] = { &thpsize_enabled_attr.attr, +#ifdef CONFIG_SHMEM + &thpsize_shmem_enabled_attr.attr, +#endif NULL, }; diff --git a/mm/shmem.c b/mm/shmem.c index a383ea9a89a5..59cc26d44344 100644 --- a/mm/shmem.c +++ b/mm/shmem.c @@ -131,6 +131,14 @@ struct shmem_options { #define SHMEM_SEEN_QUOTA 32 }; +#ifdef CONFIG_TRANSPARENT_HUGEPAGE +static unsigned long huge_anon_shmem_orders_always __read_mostly; +static unsigned long huge_anon_shmem_orders_madvise __read_mostly; +static unsigned long huge_anon_shmem_orders_inherit __read_mostly; +static unsigned long huge_anon_shmem_orders_within_size __read_mostly; +static DEFINE_SPINLOCK(huge_anon_shmem_orders_lock); +#endif + #ifdef CONFIG_TMPFS static unsigned long shmem_default_max_blocks(void) { @@ -4687,6 +4695,12 @@ void __init shmem_init(void) SHMEM_SB(shm_mnt->mnt_sb)->huge = shmem_huge; else shmem_huge = SHMEM_HUGE_NEVER; /* just in case it was patched */ + + /* + * Default to setting PMD-sized THP to inherit the global setting and + * disable all other multi-size THPs, when anonymous shmem uses mTHP. + */ + huge_anon_shmem_orders_inherit = BIT(HPAGE_PMD_ORDER); #endif return; @@ -4746,6 +4760,11 @@ static ssize_t shmem_enabled_store(struct kobject *kobj, huge != SHMEM_HUGE_NEVER && huge != SHMEM_HUGE_DENY) return -EINVAL; + /* Do not override huge allocation policy with non-PMD sized mTHP */ + if (huge == SHMEM_HUGE_FORCE && + huge_anon_shmem_orders_inherit != BIT(HPAGE_PMD_ORDER)) + return -EINVAL; + shmem_huge = huge; if (shmem_huge > SHMEM_HUGE_DENY) SHMEM_SB(shm_mnt->mnt_sb)->huge = shmem_huge; @@ -4753,6 +4772,83 @@ static ssize_t shmem_enabled_store(struct kobject *kobj, } struct kobj_attribute shmem_enabled_attr = __ATTR_RW(shmem_enabled); + +static ssize_t thpsize_shmem_enabled_show(struct kobject *kobj, + struct kobj_attribute *attr, char *buf) +{ + int order = to_thpsize(kobj)->order; + const char *output; + + if (test_bit(order, &huge_anon_shmem_orders_always)) + output = "[always] inherit within_size advise never deny [force]"; + else if (test_bit(order, &huge_anon_shmem_orders_inherit)) + output = "always [inherit] within_size advise never deny force"; + else if (test_bit(order, &huge_anon_shmem_orders_within_size)) + output = "always inherit [within_size] advise never deny force"; + else if (test_bit(order, &huge_anon_shmem_orders_madvise)) + output = "always inherit within_size [advise] never deny force"; + else + output = "always inherit within_size advise [never] [deny] force"; + + return sysfs_emit(buf, "%s\n", output); +} + +static ssize_t thpsize_shmem_enabled_store(struct kobject *kobj, + struct kobj_attribute *attr, + const char *buf, size_t count) +{ + int order = to_thpsize(kobj)->order; + ssize_t ret = count; + + if (sysfs_streq(buf, "always") || sysfs_streq(buf, "force")) { + spin_lock(&huge_anon_shmem_orders_lock); + clear_bit(order, &huge_anon_shmem_orders_inherit); + clear_bit(order, &huge_anon_shmem_orders_madvise); + clear_bit(order, &huge_anon_shmem_orders_within_size); + set_bit(order, &huge_anon_shmem_orders_always); + spin_unlock(&huge_anon_shmem_orders_lock); + } else if (sysfs_streq(buf, "inherit")) { + /* Do not override huge allocation policy with non-PMD sized mTHP */ + if (shmem_huge == SHMEM_HUGE_FORCE && + order != HPAGE_PMD_ORDER) + return -EINVAL; + + spin_lock(&huge_anon_shmem_orders_lock); + clear_bit(order, &huge_anon_shmem_orders_always); + clear_bit(order, &huge_anon_shmem_orders_madvise); + clear_bit(order, &huge_anon_shmem_orders_within_size); + set_bit(order, &huge_anon_shmem_orders_inherit); + spin_unlock(&huge_anon_shmem_orders_lock); + } else if (sysfs_streq(buf, "within_size")) { + spin_lock(&huge_anon_shmem_orders_lock); + clear_bit(order, &huge_anon_shmem_orders_always); + clear_bit(order, &huge_anon_shmem_orders_inherit); + clear_bit(order, &huge_anon_shmem_orders_madvise); + set_bit(order, &huge_anon_shmem_orders_within_size); + spin_unlock(&huge_anon_shmem_orders_lock); + } else if (sysfs_streq(buf, "madvise")) { + spin_lock(&huge_anon_shmem_orders_lock); + clear_bit(order, &huge_anon_shmem_orders_always); + clear_bit(order, &huge_anon_shmem_orders_inherit); + clear_bit(order, &huge_anon_shmem_orders_within_size); + set_bit(order, &huge_anon_shmem_orders_madvise); + spin_unlock(&huge_anon_shmem_orders_lock); + } else if (sysfs_streq(buf, "never") || sysfs_streq(buf, "deny")) { + spin_lock(&huge_anon_shmem_orders_lock); + clear_bit(order, &huge_anon_shmem_orders_always); + clear_bit(order, &huge_anon_shmem_orders_inherit); + clear_bit(order, &huge_anon_shmem_orders_within_size); + clear_bit(order, &huge_anon_shmem_orders_madvise); + spin_unlock(&huge_anon_shmem_orders_lock); + } else { + ret = -EINVAL; + } + + return ret; +} + +struct kobj_attribute thpsize_shmem_enabled_attr = + __ATTR(shmem_enabled, 0644, thpsize_shmem_enabled_show, thpsize_shmem_enabled_store); #endif /* CONFIG_TRANSPARENT_HUGEPAGE && CONFIG_SYSFS */ #else /* !CONFIG_SHMEM */ From patchwork Mon May 6 08:46:30 2024 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Baolin Wang X-Patchwork-Id: 13655148 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from kanga.kvack.org (kanga.kvack.org [205.233.56.17]) by smtp.lore.kernel.org (Postfix) with ESMTP id E8BE6C4345F for ; Mon, 6 May 2024 08:47:05 +0000 (UTC) Received: by kanga.kvack.org (Postfix) id CBD576B008C; Mon, 6 May 2024 04:46:58 -0400 (EDT) Received: by kanga.kvack.org (Postfix, from userid 40) id C1CE06B0092; Mon, 6 May 2024 04:46:58 -0400 (EDT) X-Delivered-To: int-list-linux-mm@kvack.org Received: by kanga.kvack.org (Postfix, from userid 63042) id A7A306B0093; Mon, 6 May 2024 04:46:58 -0400 (EDT) X-Delivered-To: linux-mm@kvack.org Received: from relay.hostedemail.com (smtprelay0015.hostedemail.com [216.40.44.15]) by kanga.kvack.org (Postfix) with ESMTP id 858DF6B008C for ; Mon, 6 May 2024 04:46:58 -0400 (EDT) Received: from smtpin30.hostedemail.com (a10.router.float.18 [10.200.18.1]) by unirelay09.hostedemail.com (Postfix) with ESMTP id 33EF380903 for ; Mon, 6 May 2024 08:46:58 +0000 (UTC) X-FDA: 82087340916.30.8ADCDE7 Received: from out30-112.freemail.mail.aliyun.com (out30-112.freemail.mail.aliyun.com [115.124.30.112]) by imf06.hostedemail.com (Postfix) with ESMTP id C69A318000F for ; Mon, 6 May 2024 08:46:55 +0000 (UTC) Authentication-Results: imf06.hostedemail.com; dkim=pass header.d=linux.alibaba.com header.s=default header.b=nIi1ioel; dmarc=pass (policy=none) header.from=linux.alibaba.com; spf=pass (imf06.hostedemail.com: domain of baolin.wang@linux.alibaba.com designates 115.124.30.112 as permitted sender) smtp.mailfrom=baolin.wang@linux.alibaba.com ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=hostedemail.com; s=arc-20220608; t=1714985216; h=from:from:sender:reply-to:subject:subject:date:date: message-id:message-id:to:to:cc:cc:mime-version:mime-version: content-type:content-transfer-encoding:content-transfer-encoding: in-reply-to:in-reply-to:references:references:dkim-signature; bh=JkwuLgB40/tFCSMT5YJyWsVrrngk9w7x0Ah/Vo6lKcs=; b=nnTgxjR8wBpUZzPmc+Kh4hu+W+nF/j2oqL5ErdlGPFJJ6NIFyQZnTFTqu93+mm1iFrgWBg No2G+C43nMDLoFZKEu81DonlF6njypxvN8B7B8doM1tc8AAWCrEh5U6fq7D6IUmT7nweee HI1jAyJnoLZmZp7jIyFl6k6VbizPfNI= ARC-Authentication-Results: i=1; imf06.hostedemail.com; dkim=pass header.d=linux.alibaba.com header.s=default header.b=nIi1ioel; dmarc=pass (policy=none) header.from=linux.alibaba.com; spf=pass (imf06.hostedemail.com: domain of baolin.wang@linux.alibaba.com designates 115.124.30.112 as permitted sender) smtp.mailfrom=baolin.wang@linux.alibaba.com ARC-Seal: i=1; s=arc-20220608; d=hostedemail.com; t=1714985216; a=rsa-sha256; cv=none; b=hRyjjwnyXY1JvhBDw3HeXYjVkEFcdlSNSvVv0b7XoQkcZVra9PAnUZUh0GxcVvcOuRtgEK PS5G05S72S2Z3XYqGrjnrly8ySnIHs1tM057mGJyv1Wq22Ho+Wx2V0VeRWuIV64a8o7Q+t Bv7egc8ZAkkkHKXRrrrtLRKIgRzV2+4= DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=linux.alibaba.com; s=default; t=1714985213; h=From:To:Subject:Date:Message-Id:MIME-Version; bh=JkwuLgB40/tFCSMT5YJyWsVrrngk9w7x0Ah/Vo6lKcs=; b=nIi1ioelu5JoHLDNYOk6CvONM4ZsnGK/ByI0NKRsUgDurk29F4qptpQRIfUmufbEI6txS71B0wNpweUMCnqVigJ1Brjk0/+JwrkozdsCezf0RXUiUgTFeI0UUD1/SXpfQx5BanLme2qmpK4Khd93N1YSRuoQq7YOqT1/DF0cx8c= X-Alimail-AntiSpam: AC=PASS;BC=-1|-1;BR=01201311R141e4;CH=green;DM=||false|;DS=||;FP=0|-1|-1|-1|0|-1|-1|-1;HT=maildocker-contentspam033068173054;MF=baolin.wang@linux.alibaba.com;NM=1;PH=DS;RN=14;SR=0;TI=SMTPD_---0W5wQICN_1714985210; Received: from localhost(mailfrom:baolin.wang@linux.alibaba.com fp:SMTPD_---0W5wQICN_1714985210) by smtp.aliyun-inc.com; Mon, 06 May 2024 16:46:51 +0800 From: Baolin Wang To: akpm@linux-foundation.org, hughd@google.com Cc: willy@infradead.org, david@redhat.com, ioworker0@gmail.com, wangkefeng.wang@huawei.com, ying.huang@intel.com, 21cnbao@gmail.com, ryan.roberts@arm.com, shy828301@gmail.com, ziy@nvidia.com, baolin.wang@linux.alibaba.com, linux-mm@kvack.org, linux-kernel@vger.kernel.org Subject: [PATCH 6/8] mm: shmem: add mTHP support for anonymous shmem Date: Mon, 6 May 2024 16:46:30 +0800 Message-Id: X-Mailer: git-send-email 2.39.3 In-Reply-To: References: MIME-Version: 1.0 X-Stat-Signature: 8q4fdcjzunmnhwyeb1rpko4z63rsokpu X-Rspamd-Queue-Id: C69A318000F X-Rspam-User: X-Rspamd-Server: rspam12 X-HE-Tag: 1714985215-408752 X-HE-Meta: U2FsdGVkX19SlV3DhpHFZbUGQXGfn1rhB4RLpjidoXA6jFfoZ9Jqj3BMfcTv42RkixmkExb9KelQ0vhc8XXdVaG3fgU3MYweE4b+IC4LRM7WoMwjyh80KZl2DGzZdz1Mq1oZXbP+GzD66TWmPYfiVZF11i1kAark8uBTUe2BWZ+0rPTN0R3KsOr82e9vxRdv4Z3wF40FG+cMetLX5D592B7U1qr6X17ZDl3lOiNocNIK27D9TS+8iY34x1yEbvl0MQdVrYb9rVyPw26uzzbf9m8WFjRhMItlvD5Ha+vxAslxg/4GYPpN64lvCbs2PSbkKGiBjgwPN5g16sMO+4+YiP/GxgwyUSSpxfAXIn9c74dfwERQOPjeLHXOYQi55m1y+xzy0LE0I0FcJR8IyjHyeVfBtopSSfKnT80Mxn2ynr8kW1hyokS3o3l8l1x0RRRVrcgvOezkTEc7kVGKQTSw8NUdT2KRiD6QUaNwH6Hdd6dX3E7Mlqket7X+CaXeaZ7sc6bMpyHH+SDyik/od4w5sqJyv8dcNi7ZO0TPxGRGtxB0K1ZTdi47Qr/M4d+MOJvSuz868ApEoXFQ18mXxA3LPrE+qy8e+MrZRclTP35oOOlk9fnB8IB0a9yXO6xZU3/l8Zy4Yhi+M8RyTKbe5ILOYWef7qaOG0wKCIFOTlXm3ILnYvZptsCgvFkbw8MYQLQ3JoBBtXtI4uzE/0dqDLDq/jMd6g6RtqKW5FdEJqQACCTQ2exIh3HxVOCJQYSvhULo/djfDa+zE2JDM9JLBpGBVbM6K0/NKBa+PTFKRbhQEQ5eL3AwA5euWi0AijZoTUDkLRuBGqoWTnj8lB/qVMmkoDCPh8Tqg3dB5V2F7deiRQvy+38AeClYnNZ10QYTX/2xocxBc0ggK4C445UX+UaEMwE8GtUWJphSpWeleF4wcqdC87eMmo3XCL/Dl+0WbXPx9g5Cq0UtnI7OI9AMUAg bQrDadLe HvYNCvQuvfMKioQJgsvybCOxXPdTC0YVw2OlMIkANDZY90O6Z9+Hi5HClMT1toJBGM9T3oDUa7Uz69mw7FmsSxc0S8S9IktNXyn5Zrkj9ZuUPXkI2wGwL1vAZo9D7MezzPPjTGN6qus2eGa40+Mwh5qyGQN75tHVFCTadvHECcae0V+BU1ePh3mLDAAFtUlk6pPKQQVij9kO0OmJO7/48+FmVDdg4xGFBdv5++mqcWY8OVrE= X-Bogosity: Ham, tests=bogofilter, spamicity=0.000000, version=1.2.4 Sender: owner-linux-mm@kvack.org Precedence: bulk X-Loop: owner-majordomo@kvack.org List-ID: List-Subscribe: List-Unsubscribe: Commit 19eaf44954df adds multi-size THP (mTHP) for anonymous pages, that can allow THP to be configured through the sysfs interface located at '/sys/kernel/mm/transparent_hugepage/hugepage-XXkb/enabled'. However, the anonymous share pages will ignore the anonymous mTHP rule configured through the sysfs interface, and can only use the PMD-mapped THP, that is not reasonable. Users expect to apply the mTHP rule for all anonymous pages, including the anonymous share pages, in order to enjoy the benefits of mTHP. For example, lower latency than PMD-mapped THP, smaller memory bloat than PMD-mapped THP, contiguous PTEs on ARM architecture to reduce TLB miss etc. The primary strategy is similar to supporting anonymous mTHP. Introduce a new interface '/mm/transparent_hugepage/hugepage-XXkb/shmem_enabled', which can have all the same values as the top-level '/sys/kernel/mm/transparent_hugepage/shmem_enabled', with adding a new additional "inherit" option. By default all sizes will be set to "never" except PMD size, which is set to "inherit". This ensures backward compatibility with the shmem enabled of the top level, meanwhile also allows independent control of shmem enabled for each mTHP. Signed-off-by: Baolin Wang --- mm/shmem.c | 177 +++++++++++++++++++++++++++++++++++++++++++++-------- 1 file changed, 150 insertions(+), 27 deletions(-) diff --git a/mm/shmem.c b/mm/shmem.c index 59cc26d44344..08ccea5170a1 100644 --- a/mm/shmem.c +++ b/mm/shmem.c @@ -1611,6 +1611,106 @@ static gfp_t limit_gfp_mask(gfp_t huge_gfp, gfp_t limit_gfp) return result; } +#ifdef CONFIG_TRANSPARENT_HUGEPAGE +static unsigned long anon_shmem_allowable_huge_orders(struct inode *inode, + struct vm_area_struct *vma, pgoff_t index, + bool global_huge) +{ + unsigned long mask = READ_ONCE(huge_anon_shmem_orders_always); + unsigned long within_size_orders = READ_ONCE(huge_anon_shmem_orders_within_size); + unsigned long vm_flags = vma->vm_flags; + /* + * Check all the (large) orders below HPAGE_PMD_ORDER + 1 that + * are enabled for this vma. + */ + unsigned long orders = BIT(PMD_ORDER + 1) - 1; + loff_t i_size; + int order; + + if ((vm_flags & VM_NOHUGEPAGE) || + test_bit(MMF_DISABLE_THP, &vma->vm_mm->flags)) + return 0; + + /* If the hardware/firmware marked hugepage support disabled. */ + if (transparent_hugepage_flags & (1 << TRANSPARENT_HUGEPAGE_UNSUPPORTED)) + return 0; + + /* + * Following the 'deny' semantics of the top level, force the huge + * option off from all mounts. + */ + if (shmem_huge == SHMEM_HUGE_DENY) + return 0; + /* + * Only allow inherit orders if the top-level value is 'force', which + * means non-PMD sized THP can not override 'huge' mount option now. + */ + if (shmem_huge == SHMEM_HUGE_FORCE) + return READ_ONCE(huge_anon_shmem_orders_inherit); + + /* Allow mTHP that will be fully within i_size. */ + order = highest_order(within_size_orders); + while (within_size_orders) { + index = round_up(index + 1, order); + i_size = round_up(i_size_read(inode), PAGE_SIZE); + if (i_size >> PAGE_SHIFT >= index) { + mask |= within_size_orders; + break; + } + + order = next_order(&within_size_orders, order); + } + + if (vm_flags & VM_HUGEPAGE) + mask |= READ_ONCE(huge_anon_shmem_orders_madvise); + + if (global_huge) + mask |= READ_ONCE(huge_anon_shmem_orders_inherit); + + return orders & mask; +} + +static unsigned long anon_shmem_suitable_orders(struct inode *inode, struct vm_fault *vmf, + struct address_space *mapping, pgoff_t index, + unsigned long orders) +{ + struct vm_area_struct *vma = vmf->vma; + unsigned long pages; + int order; + + orders = thp_vma_suitable_orders(vma, vmf->address, orders); + if (!orders) + return 0; + + /* Find the highest order that can add into the page cache */ + order = highest_order(orders); + while (orders) { + pages = 1UL << order; + index = round_down(index, pages); + if (!xa_find(&mapping->i_pages, &index, + index + pages - 1, XA_PRESENT)) + break; + order = next_order(&orders, order); + } + + return orders; +} +#else +static unsigned long anon_shmem_allowable_huge_orders(struct inode *inode, + struct vm_area_struct *vma, pgoff_t index, + bool global_huge) +{ + return 0; +} + +static unsigned long anon_shmem_suitable_orders(struct inode *inode, struct vm_fault *vmf, + struct address_space *mapping, pgoff_t index, + unsigned long orders) +{ + return 0; +} +#endif /* CONFIG_TRANSPARENT_HUGEPAGE */ + static struct folio *shmem_alloc_hugefolio(gfp_t gfp, struct shmem_inode_info *info, pgoff_t index, int order) { @@ -1639,38 +1739,55 @@ static struct folio *shmem_alloc_folio(gfp_t gfp, return (struct folio *)page; } -static struct folio *shmem_alloc_and_add_folio(gfp_t gfp, - struct inode *inode, pgoff_t index, - struct mm_struct *fault_mm, bool huge) +static struct folio *shmem_alloc_and_add_folio(struct vm_fault *vmf, + gfp_t gfp, struct inode *inode, pgoff_t index, + struct mm_struct *fault_mm, bool huge, unsigned long orders) { struct address_space *mapping = inode->i_mapping; struct shmem_inode_info *info = SHMEM_I(inode); + struct vm_area_struct *vma = vmf ? vmf->vma : NULL; + unsigned long suitable_orders; struct folio *folio; long pages; - int error; + int error, order; if (!IS_ENABLED(CONFIG_TRANSPARENT_HUGEPAGE)) huge = false; - if (huge) { - pages = HPAGE_PMD_NR; - index = round_down(index, HPAGE_PMD_NR); + if (huge || orders > 0) { + if (vma && vma_is_anon_shmem(vma) && orders) { + suitable_orders = anon_shmem_suitable_orders(inode, vmf, + mapping, index, orders); + } else { + pages = HPAGE_PMD_NR; + suitable_orders = BIT(HPAGE_PMD_ORDER); + index = round_down(index, HPAGE_PMD_NR); - /* - * Check for conflict before waiting on a huge allocation. - * Conflict might be that a huge page has just been allocated - * and added to page cache by a racing thread, or that there - * is already at least one small page in the huge extent. - * Be careful to retry when appropriate, but not forever! - * Elsewhere -EEXIST would be the right code, but not here. - */ - if (xa_find(&mapping->i_pages, &index, + /* + * Check for conflict before waiting on a huge allocation. + * Conflict might be that a huge page has just been allocated + * and added to page cache by a racing thread, or that there + * is already at least one small page in the huge extent. + * Be careful to retry when appropriate, but not forever! + * Elsewhere -EEXIST would be the right code, but not here. + */ + if (xa_find(&mapping->i_pages, &index, index + HPAGE_PMD_NR - 1, XA_PRESENT)) - return ERR_PTR(-E2BIG); + return ERR_PTR(-E2BIG); + } - folio = shmem_alloc_hugefolio(gfp, info, index, HPAGE_PMD_ORDER); - if (!folio && pages == HPAGE_PMD_NR) - count_vm_event(THP_FILE_FALLBACK); + order = highest_order(suitable_orders); + while (suitable_orders) { + pages = 1 << order; + index = round_down(index, pages); + folio = shmem_alloc_hugefolio(gfp, info, index, order); + if (folio) + goto allocated; + + if (pages == HPAGE_PMD_NR) + count_vm_event(THP_FILE_FALLBACK); + order = next_order(&suitable_orders, order); + } } else { pages = 1; folio = shmem_alloc_folio(gfp, info, index); @@ -1678,6 +1795,7 @@ static struct folio *shmem_alloc_and_add_folio(gfp_t gfp, if (!folio) return ERR_PTR(-ENOMEM); +allocated: __folio_set_locked(folio); __folio_set_swapbacked(folio); @@ -1972,7 +2090,8 @@ static int shmem_get_folio_gfp(struct inode *inode, pgoff_t index, struct mm_struct *fault_mm; struct folio *folio; int error; - bool alloced; + bool alloced, huge; + unsigned long orders = 0; if (WARN_ON_ONCE(!shmem_mapping(inode->i_mapping))) return -EINVAL; @@ -2044,14 +2163,18 @@ static int shmem_get_folio_gfp(struct inode *inode, pgoff_t index, return 0; } - if (shmem_is_huge(inode, index, false, fault_mm, - vma ? vma->vm_flags : 0)) { + huge = shmem_is_huge(inode, index, false, fault_mm, + vma ? vma->vm_flags : 0); + /* Find hugepage orders that are allowed for anonymous shmem. */ + if (vma && vma_is_anon_shmem(vma)) + orders = anon_shmem_allowable_huge_orders(inode, vma, index, huge); + if (huge || orders > 0) { gfp_t huge_gfp; huge_gfp = vma_thp_gfp_mask(vma); huge_gfp = limit_gfp_mask(huge_gfp, gfp); - folio = shmem_alloc_and_add_folio(huge_gfp, - inode, index, fault_mm, true); + folio = shmem_alloc_and_add_folio(vmf, huge_gfp, + inode, index, fault_mm, true, orders); if (!IS_ERR(folio)) { if (folio_test_pmd_mappable(folio)) count_vm_event(THP_FILE_ALLOC); @@ -2061,7 +2184,7 @@ static int shmem_get_folio_gfp(struct inode *inode, pgoff_t index, goto repeat; } - folio = shmem_alloc_and_add_folio(gfp, inode, index, fault_mm, false); + folio = shmem_alloc_and_add_folio(vmf, gfp, inode, index, fault_mm, false, 0); if (IS_ERR(folio)) { error = PTR_ERR(folio); if (error == -EEXIST) @@ -2072,7 +2195,7 @@ static int shmem_get_folio_gfp(struct inode *inode, pgoff_t index, alloced: alloced = true; - if (folio_test_pmd_mappable(folio) && + if (folio_test_large(folio) && DIV_ROUND_UP(i_size_read(inode), PAGE_SIZE) < folio_next_index(folio) - 1) { struct shmem_sb_info *sbinfo = SHMEM_SB(inode->i_sb); From patchwork Mon May 6 08:46:31 2024 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Baolin Wang X-Patchwork-Id: 13655149 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from kanga.kvack.org (kanga.kvack.org [205.233.56.17]) by smtp.lore.kernel.org (Postfix) with ESMTP id C9CF7C10F16 for ; Mon, 6 May 2024 08:47:08 +0000 (UTC) Received: by kanga.kvack.org (Postfix) id 42E4D6B0092; Mon, 6 May 2024 04:46:59 -0400 (EDT) Received: by kanga.kvack.org (Postfix, from userid 40) id 3B7656B0093; Mon, 6 May 2024 04:46:59 -0400 (EDT) X-Delivered-To: int-list-linux-mm@kvack.org Received: by kanga.kvack.org (Postfix, from userid 63042) id 259A56B0095; Mon, 6 May 2024 04:46:59 -0400 (EDT) X-Delivered-To: linux-mm@kvack.org Received: from relay.hostedemail.com (smtprelay0011.hostedemail.com [216.40.44.11]) by kanga.kvack.org (Postfix) with ESMTP id F3B286B0092 for ; Mon, 6 May 2024 04:46:58 -0400 (EDT) Received: from smtpin13.hostedemail.com (a10.router.float.18 [10.200.18.1]) by unirelay06.hostedemail.com (Postfix) with ESMTP id BE570A166E for ; Mon, 6 May 2024 08:46:58 +0000 (UTC) X-FDA: 82087340916.13.C11AEB6 Received: from out30-101.freemail.mail.aliyun.com (out30-101.freemail.mail.aliyun.com [115.124.30.101]) by imf02.hostedemail.com (Postfix) with ESMTP id B3CD78000E for ; Mon, 6 May 2024 08:46:56 +0000 (UTC) Authentication-Results: imf02.hostedemail.com; dkim=pass header.d=linux.alibaba.com header.s=default header.b=hTI8TkZN; spf=pass (imf02.hostedemail.com: domain of baolin.wang@linux.alibaba.com designates 115.124.30.101 as permitted sender) smtp.mailfrom=baolin.wang@linux.alibaba.com; dmarc=pass (policy=none) header.from=linux.alibaba.com ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=hostedemail.com; s=arc-20220608; t=1714985217; h=from:from:sender:reply-to:subject:subject:date:date: message-id:message-id:to:to:cc:cc:mime-version:mime-version: content-type:content-transfer-encoding:content-transfer-encoding: in-reply-to:in-reply-to:references:references:dkim-signature; bh=JwVqRNDg6ovq+W40+OWE62ViZ4KVyR46bNtfHFl1yVM=; b=Jn13WqqP0Q9ilaZ3xDNy2G6hn4gBxSMfA6S/CWAk/51I/c89WInhnTeF8ZkKYqS3ycpivc 5t3jKKbUNkFDObcbA2rJNsC6hHl9hGmIWCc5LRJQ7po1FUZbCW8A6aZvhPmG2B7dbJjPdg nDJi862nlR6Yh07t73EADR4aqKlOhV0= ARC-Seal: i=1; s=arc-20220608; d=hostedemail.com; t=1714985217; a=rsa-sha256; cv=none; b=R1uzZKwXnK7l8MztNIRQd0405ZX/BupLs+mOuoBxzqEfEx2m55aBS7HUyw5xbviwbokwn2 xteqhYqk+0e0fBL1Y7Ev01aJ8b2h4Q82+46Qaf0wbG+OXVfmkLUNYzFzutUn90qjT3YF6W Q6nj3XxLz3+AvuTnq6jMQTXVUJzZfyY= ARC-Authentication-Results: i=1; imf02.hostedemail.com; dkim=pass header.d=linux.alibaba.com header.s=default header.b=hTI8TkZN; spf=pass (imf02.hostedemail.com: domain of baolin.wang@linux.alibaba.com designates 115.124.30.101 as permitted sender) smtp.mailfrom=baolin.wang@linux.alibaba.com; dmarc=pass (policy=none) header.from=linux.alibaba.com DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=linux.alibaba.com; s=default; t=1714985214; h=From:To:Subject:Date:Message-Id:MIME-Version; bh=JwVqRNDg6ovq+W40+OWE62ViZ4KVyR46bNtfHFl1yVM=; b=hTI8TkZNM6lKGKawrV3DkYoFzOJc8c7v2EIZKKcMaJgQQg4Y1cgdQ68VHEVw0FFjLZ16/ZJ7v1iS63EOLnpgFzcBb3WVjEkoiGniXL4Pbm+M40L2xpKtj7DcJXZB025Z7rlvmTy63nmifCH/jgoKJldV/fyyc87p/SZzTKil4/E= X-Alimail-AntiSpam: AC=PASS;BC=-1|-1;BR=01201311R111e4;CH=green;DM=||false|;DS=||;FP=0|-1|-1|-1|0|-1|-1|-1;HT=maildocker-contentspam033045046011;MF=baolin.wang@linux.alibaba.com;NM=1;PH=DS;RN=14;SR=0;TI=SMTPD_---0W5wQICo_1714985212; Received: from localhost(mailfrom:baolin.wang@linux.alibaba.com fp:SMTPD_---0W5wQICo_1714985212) by smtp.aliyun-inc.com; Mon, 06 May 2024 16:46:53 +0800 From: Baolin Wang To: akpm@linux-foundation.org, hughd@google.com Cc: willy@infradead.org, david@redhat.com, ioworker0@gmail.com, wangkefeng.wang@huawei.com, ying.huang@intel.com, 21cnbao@gmail.com, ryan.roberts@arm.com, shy828301@gmail.com, ziy@nvidia.com, baolin.wang@linux.alibaba.com, linux-mm@kvack.org, linux-kernel@vger.kernel.org Subject: [PATCH 7/8] mm: shmem: add mTHP size alignment in shmem_get_unmapped_area Date: Mon, 6 May 2024 16:46:31 +0800 Message-Id: <6e7528bd80309e887427a6bade4b1233a92214d0.1714978902.git.baolin.wang@linux.alibaba.com> X-Mailer: git-send-email 2.39.3 In-Reply-To: References: MIME-Version: 1.0 X-Rspam-User: X-Stat-Signature: 5incii6zdyna5771gj48mstbgnobk4wj X-Rspamd-Server: rspam11 X-Rspamd-Queue-Id: B3CD78000E X-HE-Tag: 1714985216-23824 X-HE-Meta: U2FsdGVkX1/i6idDJoax1nM/OBpZtxC5G7TSdmnwY2i8dskmJ4Z5te6b/izmhlGo+ntLixq5KkJYgca08oUOwM7soXDBM+Sl0SShLPKvjPtOBNM405a6umWNi96zpPiD6kNy4Xa3qz1uhxIVHHRjBv3v/43LArKJU3QVSiOIRgErYdYUT5DN3Dx4qWHutEd285RsTVLOzb9LMn5djIv/N0J/Hc2BPpyQsjAaCwYTds+UUF36rXZUHlGlUPTc5Y9GnPVXDC4WJVhyRLVIjQB9mYT9/TagkmxwAokMpv4REv49yl1rfYAEAAIRC80tYRU8+TyQC6G/kKWv7J0CDyxPLNI932DUri0jNgjMjfeT+W6DUGb0DxZ/VSIXPFh3WEl0J7UAp/3xxa8toUsX8AJuFTRLJDpUqj27FPu2PDx50XHYTe3BCLfmwc16Ogygk+ie9aJv8irwzYauc+IAChZXfYX7L3FhjPnSVxoLWzXKcLRbR107Ta999732TBA/j0gLScq8zEb97W3UrEcQqQWj5qy04Bl1sH+dpJ79/TYhwCwFvAQxSfaKkdRSAa+1QepgKzM/ObLQXsJw6IVJLg8YYaYz6vTTAKtdkhP2iBtzM1x31DpTHPkrKmFFl+SWI6biqXXRhxO5FDSGzMwRO7dDRIR37vn1GOl2I6pozdn/m/aDCv+LUzbtI9b9mvBzXcwQaMy6sWjR2bLPxBNY3ras0OhW1f8MNwKd3NKT4VDO4Sji0oelwMIQcqFkERygKjOnc65GsDvxOz3BiCe2jeEHHrPCU0VHNnRNBMXH99SUvWfi9pGwRsUJ8kPVP1BbfP761nY9/aOoeosmvaLEqmNN5faOH/j/DeClxHkxbilD2ERram5mw50rJ1Wk0qerIXIt+EoG6WzJ++LJBJyDs2STcV0hwCtdCqG2296QQldyaSEGA99XY6QpV36BiSzd6F5HObn4+i8ssy8KE4P6sPR +hjJuO4l DhFRFrajfLCf36CU8FjFg/EYtY9PZEhOwu0/iWZfLJqt5stUlALKAHknyzdGb3Iy13t5eoaVXZ/vIdAXLs01Hj+OEUNlAQ3KcrvlMZ+keOOwAJ/wPNcuEiU2kYEStDR4RyfeAIgSmZCKm45zJksSut94kYgBFhbw0Ypd/P9CEH9TB+mnWn3COUBBviUGBYwAdg2wFSClPNOCEYFtL2RlJpzPH+b7P50ScObN6TD3wcOSi/IjZI5QMpxKhwqoZ7mNc1VY6czg5Ve/Q3ZMWGrOVHGT5sa2CF+V9QaeB X-Bogosity: Ham, tests=bogofilter, spamicity=0.000000, version=1.2.4 Sender: owner-linux-mm@kvack.org Precedence: bulk X-Loop: owner-majordomo@kvack.org List-ID: List-Subscribe: List-Unsubscribe: Although the top-level hugepage allocation can be turned off, anonymous shmem can still use mTHP by configuring the sysfs interface located at '/sys/kernel/mm/transparent_hugepage/hugepage-XXkb/shmem_enabled'. Therefore, add alignment for mTHP size to provide a suitable alignment address in shmem_get_unmapped_area(). Signed-off-by: Baolin Wang --- mm/shmem.c | 35 ++++++++++++++++++++++++++--------- 1 file changed, 26 insertions(+), 9 deletions(-) diff --git a/mm/shmem.c b/mm/shmem.c index 08ccea5170a1..27107afdff9e 100644 --- a/mm/shmem.c +++ b/mm/shmem.c @@ -2404,6 +2404,7 @@ unsigned long shmem_get_unmapped_area(struct file *file, unsigned long inflated_len; unsigned long inflated_addr; unsigned long inflated_offset; + unsigned long hpage_size = HPAGE_PMD_SIZE; if (len > TASK_SIZE) return -ENOMEM; @@ -2422,8 +2423,6 @@ unsigned long shmem_get_unmapped_area(struct file *file, if (shmem_huge == SHMEM_HUGE_DENY) return addr; - if (len < HPAGE_PMD_SIZE) - return addr; if (flags & MAP_FIXED) return addr; /* @@ -2437,6 +2436,8 @@ unsigned long shmem_get_unmapped_area(struct file *file, if (shmem_huge != SHMEM_HUGE_FORCE) { struct super_block *sb; + unsigned long __maybe_unused hpage_orders; + int order = 0; if (file) { VM_BUG_ON(file->f_op != &shmem_file_operations); @@ -2449,18 +2450,34 @@ unsigned long shmem_get_unmapped_area(struct file *file, if (IS_ERR(shm_mnt)) return addr; sb = shm_mnt->mnt_sb; + + /* + * Find the highest mTHP order used for anonymous shmem to + * provide a suitable alignment address. + */ +#ifdef CONFIG_TRANSPARENT_HUGEPAGE + hpage_orders = READ_ONCE(huge_anon_shmem_orders_always); + hpage_orders |= READ_ONCE(huge_anon_shmem_orders_within_size); + hpage_orders |= READ_ONCE(huge_anon_shmem_orders_madvise); + hpage_orders |= READ_ONCE(huge_anon_shmem_orders_inherit); + order = highest_order(hpage_orders); + hpage_size = PAGE_SIZE << order; +#endif } - if (SHMEM_SB(sb)->huge == SHMEM_HUGE_NEVER) + if (SHMEM_SB(sb)->huge == SHMEM_HUGE_NEVER && !order) return addr; } - offset = (pgoff << PAGE_SHIFT) & (HPAGE_PMD_SIZE-1); - if (offset && offset + len < 2 * HPAGE_PMD_SIZE) + if (len < hpage_size) + return addr; + + offset = (pgoff << PAGE_SHIFT) & (hpage_size - 1); + if (offset && offset + len < 2 * hpage_size) return addr; - if ((addr & (HPAGE_PMD_SIZE-1)) == offset) + if ((addr & (hpage_size - 1)) == offset) return addr; - inflated_len = len + HPAGE_PMD_SIZE - PAGE_SIZE; + inflated_len = len + hpage_size - PAGE_SIZE; if (inflated_len > TASK_SIZE) return addr; if (inflated_len < len) @@ -2473,10 +2490,10 @@ unsigned long shmem_get_unmapped_area(struct file *file, if (inflated_addr & ~PAGE_MASK) return addr; - inflated_offset = inflated_addr & (HPAGE_PMD_SIZE-1); + inflated_offset = inflated_addr & (hpage_size - 1); inflated_addr += offset - inflated_offset; if (inflated_offset > offset) - inflated_addr += HPAGE_PMD_SIZE; + inflated_addr += hpage_size; if (inflated_addr > TASK_SIZE - len) return addr; From patchwork Mon May 6 08:46:32 2024 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Baolin Wang X-Patchwork-Id: 13655150 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from kanga.kvack.org (kanga.kvack.org [205.233.56.17]) by smtp.lore.kernel.org (Postfix) with ESMTP id 89396C4345F for ; Mon, 6 May 2024 08:47:11 +0000 (UTC) Received: by kanga.kvack.org (Postfix) id 36C876B0095; Mon, 6 May 2024 04:47:01 -0400 (EDT) Received: by kanga.kvack.org (Postfix, from userid 40) id 258266B0096; Mon, 6 May 2024 04:47:01 -0400 (EDT) X-Delivered-To: int-list-linux-mm@kvack.org Received: by kanga.kvack.org (Postfix, from userid 63042) id 0359A6B0098; Mon, 6 May 2024 04:47:00 -0400 (EDT) X-Delivered-To: linux-mm@kvack.org Received: from relay.hostedemail.com (smtprelay0016.hostedemail.com [216.40.44.16]) by kanga.kvack.org (Postfix) with ESMTP id D46E46B0095 for ; Mon, 6 May 2024 04:47:00 -0400 (EDT) Received: from smtpin15.hostedemail.com (a10.router.float.18 [10.200.18.1]) by unirelay04.hostedemail.com (Postfix) with ESMTP id 82B3F1A085C for ; Mon, 6 May 2024 08:47:00 +0000 (UTC) X-FDA: 82087341000.15.4839BDD Received: from out30-132.freemail.mail.aliyun.com (out30-132.freemail.mail.aliyun.com [115.124.30.132]) by imf15.hostedemail.com (Postfix) with ESMTP id 52C0BA0016 for ; Mon, 6 May 2024 08:46:57 +0000 (UTC) Authentication-Results: imf15.hostedemail.com; dkim=pass header.d=linux.alibaba.com header.s=default header.b=pusY9aSU; dmarc=pass (policy=none) header.from=linux.alibaba.com; spf=pass (imf15.hostedemail.com: domain of baolin.wang@linux.alibaba.com designates 115.124.30.132 as permitted sender) smtp.mailfrom=baolin.wang@linux.alibaba.com ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=hostedemail.com; s=arc-20220608; t=1714985218; h=from:from:sender:reply-to:subject:subject:date:date: message-id:message-id:to:to:cc:cc:mime-version:mime-version: content-type:content-transfer-encoding:content-transfer-encoding: in-reply-to:in-reply-to:references:references:dkim-signature; bh=NqOIe5Oq6MCM0KXK59PH5ZOetWOLxng8pqZWeky8+zw=; b=1G+fks81V7B6/mTKI6XGdaf6IpOq26YEVWH2iY9OMNKnYv2tjB39jjf1n+h9mKAHu5Wpmo FRIOmvvuQzIBVoNDtI6bYQO7aqUAZkx4Ae+deO/iYa3qdX7FMEeg3yoMqxUPrC6nhFzURR LccvkIjmF4W7P5TOV9416SA/OGjrLlU= ARC-Seal: i=1; s=arc-20220608; d=hostedemail.com; t=1714985218; a=rsa-sha256; cv=none; b=7jIYj3YRwsn4lt9TjIvY+T7ji1hmMdMgVWXftzhfKC22nyGytV4c1RADuwdlHscRJ9M1NA 5XJx7u6JA0AcstEUmIgwtF5O7KCcUsc3ys4J5U9P8FWC6FAKQfrkC8wd/JCMCvVR3XblAH OHtjXXXt2DlJ+7qpc/0RwYNTULr1uWc= ARC-Authentication-Results: i=1; imf15.hostedemail.com; dkim=pass header.d=linux.alibaba.com header.s=default header.b=pusY9aSU; dmarc=pass (policy=none) header.from=linux.alibaba.com; spf=pass (imf15.hostedemail.com: domain of baolin.wang@linux.alibaba.com designates 115.124.30.132 as permitted sender) smtp.mailfrom=baolin.wang@linux.alibaba.com DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=linux.alibaba.com; s=default; t=1714985215; h=From:To:Subject:Date:Message-Id:MIME-Version; bh=NqOIe5Oq6MCM0KXK59PH5ZOetWOLxng8pqZWeky8+zw=; b=pusY9aSUFQ1Msf+lw85BS2IRAzWgqVbe6qKkjtmqkG/gSZKs9ogcYvRCoovzUIbXDz2YrWdBPOLZTFGxe7EEAbUFAOzbhWGWkGEleeD6N6rPQJMBCs/OBObhe7/WMJy1Iqw6jyDyFDoczP/VxGSzHqYE7wEmB07aPH3xPXA6qus= X-Alimail-AntiSpam: AC=PASS;BC=-1|-1;BR=01201311R111e4;CH=green;DM=||false|;DS=||;FP=0|-1|-1|-1|0|-1|-1|-1;HT=maildocker-contentspam033022160150;MF=baolin.wang@linux.alibaba.com;NM=1;PH=DS;RN=14;SR=0;TI=SMTPD_---0W5weiNH_1714985213; Received: from localhost(mailfrom:baolin.wang@linux.alibaba.com fp:SMTPD_---0W5weiNH_1714985213) by smtp.aliyun-inc.com; Mon, 06 May 2024 16:46:54 +0800 From: Baolin Wang To: akpm@linux-foundation.org, hughd@google.com Cc: willy@infradead.org, david@redhat.com, ioworker0@gmail.com, wangkefeng.wang@huawei.com, ying.huang@intel.com, 21cnbao@gmail.com, ryan.roberts@arm.com, shy828301@gmail.com, ziy@nvidia.com, baolin.wang@linux.alibaba.com, linux-mm@kvack.org, linux-kernel@vger.kernel.org Subject: [PATCH 8/8] mm: shmem: add mTHP counters for anonymous shmem Date: Mon, 6 May 2024 16:46:32 +0800 Message-Id: <34f0b24f7c061f796d91ade9766e20a9da0c7651.1714978902.git.baolin.wang@linux.alibaba.com> X-Mailer: git-send-email 2.39.3 In-Reply-To: References: MIME-Version: 1.0 X-Rspamd-Server: rspam08 X-Rspamd-Queue-Id: 52C0BA0016 X-Stat-Signature: kr83848u55n7jc9tikonif5htxjypmem X-Rspam-User: X-HE-Tag: 1714985217-860587 X-HE-Meta: U2FsdGVkX19HkUXjRMu1Ung9EcCPxTeU8UACVDcxpDTpAuFaFztzLeqor2jk2Pakgml1+6CT0bhoRlte6E8nSWD4NULbDFSGnvYmC8IT6njuLmdlyEWB/1eC0ymRK5trljSAohe8gt5RkaMpCjlenlg54akevDfB76eD7eahHjJJLWbZT9JJ0Yt0P3d8lLoQRl5RT91SKADai+pHGI24VGykuDtHgRaKD3QZv75PcWy+rkvATgCP3Kg/W4J7Cmk6aRTKzMzBDMwCMWh1ZQN3AmTcV6j6/lb15y9g8caAS7kfW+kpO4ySvMyQPdvdBg0wCe4rd7dKj3sFj8Rx7UrdBCQOiz3tieUfcy1yDGSpjVw0XfMESGcSqiHZ7OdbUuv5wpxF9I3IC5pejq0I7ltSadFGGgpj+Shn0+/SNdswo3zd2jFUeyjiGnLSjR3McsYFuLKDyf89nN5dNsvj4zcfUQJ10PAASf5S/0ppf3yPsOrMOxp1hWgW3CwRUs7azi5CF+g5Ikol7k/SyNEQVGY8XoNAfz/0xFHHXc4HJt2J2ScD3C6NiTl30udYevVhOmyOlyDJN5281K2kooSCZfENErvxNkLORHMVuf8b9Y9IR2x+36nOKOHcNMziOjeCW+1ricPb+eCvN6k31fOJ03j8yl5HGMW9Sc7lZculjNuTkaqqh0/u+ZrE5tLsWWhXSW26mvaM4wvfob1lXlVpspdE043Dv8t40zTLDzaQB2mxuvT4Ug5aU4ln5pBWcACWmspQthOyMbeWepae5/SKAvyojH2yvYT/tcIi/XVY0D0PYCJ72/t5tj4B2Yb/tBJMHT0H+jrZ36RE1vbHonTNbu0GXWmlpCgiYrz6mZTIv5dj3kG9zr+QPFJ2/PdCJxY1hACn7CA4dcFJoHMjGenqu9OZHkSmOI5rLlEdo9Xnkgm4vEB6bUsPQN2vi70jIEv0tFMV/GlFfpA9vhZJMSE8Umi qlLkQlhG 11uh8eEJ6zA68JHTLpZ/uCBqzNixtn2qJsu8wW7FsyeaXnDo1shmhETR50UCmyTNZTuCRCSc3PG8a6dYGU9agruUXIH0tp4ikWinJEUd8grNBmLPqzXPUsI/upjOEu4v7dn8mJ4vpUe4jaK/fiSVF74ZKhkDELR39KU6TZn0p5jNXg951BMOVQP1+XoB5E3OUVkZXefAb5RKZpoxnsS0RPvEQIB4QJIqultvzGq6Rjd1V7ZFCHJ7UQHxZSw== X-Bogosity: Ham, tests=bogofilter, spamicity=0.000000, version=1.2.4 Sender: owner-linux-mm@kvack.org Precedence: bulk X-Loop: owner-majordomo@kvack.org List-ID: List-Subscribe: List-Unsubscribe: Add mTHP counters for anonymous shmem. Signed-off-by: Baolin Wang --- include/linux/huge_mm.h | 3 +++ mm/huge_memory.c | 6 ++++++ mm/shmem.c | 18 +++++++++++++++--- 3 files changed, 24 insertions(+), 3 deletions(-) diff --git a/include/linux/huge_mm.h b/include/linux/huge_mm.h index dbd6b3f56210..c15bebb2cf53 100644 --- a/include/linux/huge_mm.h +++ b/include/linux/huge_mm.h @@ -281,6 +281,9 @@ enum mthp_stat_item { MTHP_STAT_ANON_FAULT_FALLBACK_CHARGE, MTHP_STAT_ANON_SWPOUT, MTHP_STAT_ANON_SWPOUT_FALLBACK, + MTHP_STAT_FILE_ALLOC, + MTHP_STAT_FILE_FALLBACK, + MTHP_STAT_FILE_FALLBACK_CHARGE, __MTHP_STAT_COUNT }; diff --git a/mm/huge_memory.c b/mm/huge_memory.c index d3080a8843f2..fcda6ae604f6 100644 --- a/mm/huge_memory.c +++ b/mm/huge_memory.c @@ -555,6 +555,9 @@ DEFINE_MTHP_STAT_ATTR(anon_fault_fallback, MTHP_STAT_ANON_FAULT_FALLBACK); DEFINE_MTHP_STAT_ATTR(anon_fault_fallback_charge, MTHP_STAT_ANON_FAULT_FALLBACK_CHARGE); DEFINE_MTHP_STAT_ATTR(anon_swpout, MTHP_STAT_ANON_SWPOUT); DEFINE_MTHP_STAT_ATTR(anon_swpout_fallback, MTHP_STAT_ANON_SWPOUT_FALLBACK); +DEFINE_MTHP_STAT_ATTR(file_alloc, MTHP_STAT_FILE_ALLOC); +DEFINE_MTHP_STAT_ATTR(file_fallback, MTHP_STAT_FILE_FALLBACK); +DEFINE_MTHP_STAT_ATTR(file_fallback_charge, MTHP_STAT_FILE_FALLBACK_CHARGE); static struct attribute *stats_attrs[] = { &anon_fault_alloc_attr.attr, @@ -562,6 +565,9 @@ static struct attribute *stats_attrs[] = { &anon_fault_fallback_charge_attr.attr, &anon_swpout_attr.attr, &anon_swpout_fallback_attr.attr, + &file_alloc_attr.attr, + &file_fallback_attr.attr, + &file_fallback_charge_attr.attr, NULL, }; diff --git a/mm/shmem.c b/mm/shmem.c index 27107afdff9e..1af2f0aa384d 100644 --- a/mm/shmem.c +++ b/mm/shmem.c @@ -1786,6 +1786,9 @@ static struct folio *shmem_alloc_and_add_folio(struct vm_fault *vmf, if (pages == HPAGE_PMD_NR) count_vm_event(THP_FILE_FALLBACK); +#ifdef CONFIG_TRANSPARENT_HUGEPAGE + count_mthp_stat(order, MTHP_STAT_FILE_FALLBACK); +#endif order = next_order(&suitable_orders, order); } } else { @@ -1805,9 +1808,15 @@ static struct folio *shmem_alloc_and_add_folio(struct vm_fault *vmf, if (xa_find(&mapping->i_pages, &index, index + pages - 1, XA_PRESENT)) { error = -EEXIST; - } else if (pages == HPAGE_PMD_NR) { - count_vm_event(THP_FILE_FALLBACK); - count_vm_event(THP_FILE_FALLBACK_CHARGE); + } else if (pages > 1) { + if (pages == HPAGE_PMD_NR) { + count_vm_event(THP_FILE_FALLBACK); + count_vm_event(THP_FILE_FALLBACK_CHARGE); + } +#ifdef CONFIG_TRANSPARENT_HUGEPAGE + count_mthp_stat(folio_order(folio), MTHP_STAT_FILE_FALLBACK); + count_mthp_stat(folio_order(folio), MTHP_STAT_FILE_FALLBACK_CHARGE); +#endif } goto unlock; } @@ -2178,6 +2187,9 @@ static int shmem_get_folio_gfp(struct inode *inode, pgoff_t index, if (!IS_ERR(folio)) { if (folio_test_pmd_mappable(folio)) count_vm_event(THP_FILE_ALLOC); +#ifdef CONFIG_TRANSPARENT_HUGEPAGE + count_mthp_stat(folio_order(folio), MTHP_STAT_FILE_ALLOC); +#endif goto alloced; } if (PTR_ERR(folio) == -EEXIST)