From patchwork Mon Apr 22 07:02:39 2024 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Baolin Wang X-Patchwork-Id: 13637715 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from kanga.kvack.org (kanga.kvack.org [205.233.56.17]) by smtp.lore.kernel.org (Postfix) with ESMTP id 1A2A1C4345F for ; Mon, 22 Apr 2024 07:03:05 +0000 (UTC) Received: by kanga.kvack.org (Postfix) id 569846B0088; Mon, 22 Apr 2024 03:03:05 -0400 (EDT) Received: by kanga.kvack.org (Postfix, from userid 40) id 5190A6B008A; Mon, 22 Apr 2024 03:03:05 -0400 (EDT) X-Delivered-To: int-list-linux-mm@kvack.org Received: by kanga.kvack.org (Postfix, from userid 63042) id 412A16B008C; Mon, 22 Apr 2024 03:03:05 -0400 (EDT) X-Delivered-To: linux-mm@kvack.org Received: from relay.hostedemail.com (smtprelay0013.hostedemail.com [216.40.44.13]) by kanga.kvack.org (Postfix) with ESMTP id 23F726B0088 for ; Mon, 22 Apr 2024 03:03:05 -0400 (EDT) Received: from smtpin06.hostedemail.com (a10.router.float.18 [10.200.18.1]) by unirelay07.hostedemail.com (Postfix) with ESMTP id 8707F160B10 for ; Mon, 22 Apr 2024 07:03:04 +0000 (UTC) X-FDA: 82036275888.06.60C9C4E Received: from out30-119.freemail.mail.aliyun.com (out30-119.freemail.mail.aliyun.com [115.124.30.119]) by imf23.hostedemail.com (Postfix) with ESMTP id AFAC5140011 for ; Mon, 22 Apr 2024 07:03:01 +0000 (UTC) Authentication-Results: imf23.hostedemail.com; dkim=pass header.d=linux.alibaba.com header.s=default header.b=d9H7dXA3; spf=pass (imf23.hostedemail.com: domain of baolin.wang@linux.alibaba.com designates 115.124.30.119 as permitted sender) smtp.mailfrom=baolin.wang@linux.alibaba.com; dmarc=pass (policy=none) header.from=linux.alibaba.com ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=hostedemail.com; s=arc-20220608; t=1713769382; h=from:from:sender:reply-to:subject:subject:date:date: message-id:message-id:to:to:cc:cc:mime-version:mime-version: content-type:content-transfer-encoding:content-transfer-encoding: in-reply-to:in-reply-to:references:references:dkim-signature; bh=iA51Bkqzy9zcXZ9S3Nkg6q/BBso18kv2IFiMWpEhGCs=; b=VnD08C9cwwnaK+0KAcV9n4slEyBoUY5uPloMBbdENmEdLDIRvfBUSH8wmFIGxvhMragxLd dr/02CKYi8oVLQpKSlVssoyhuKwgjmlzA3kkWiuGgrsWJFMLZwygqcf72w5s7YAvdgFtej y/4H0auplB52rpvqb/KD0Ov25Nq9FQQ= ARC-Seal: i=1; s=arc-20220608; d=hostedemail.com; t=1713769382; a=rsa-sha256; cv=none; b=IUVIlHMxnWZkCbgFCTZifVF/vDcWtPmQ2q+euCVNynpHFdbFVv2s7pimUJa0G5qBpu0ojy 9G9Xn0eDm4jqXcHQgSTI32BVgi7X5zi72Y/E451nvXhhd8rHmGxEsO5sFX3mDE12xhJ/Xe +Bhocli6hpWvugotVhBYcvWVYRAQiAs= ARC-Authentication-Results: i=1; imf23.hostedemail.com; dkim=pass header.d=linux.alibaba.com header.s=default header.b=d9H7dXA3; spf=pass (imf23.hostedemail.com: domain of baolin.wang@linux.alibaba.com designates 115.124.30.119 as permitted sender) smtp.mailfrom=baolin.wang@linux.alibaba.com; dmarc=pass (policy=none) header.from=linux.alibaba.com DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=linux.alibaba.com; s=default; t=1713769378; h=From:To:Subject:Date:Message-Id:MIME-Version; bh=iA51Bkqzy9zcXZ9S3Nkg6q/BBso18kv2IFiMWpEhGCs=; b=d9H7dXA3OkN5DMCvZyHjmmKv7zMP+IdJ9zqkOuoZ0p+W8tBIIEhqlgSceJJq1t1yms1JHJ2VSlMYTqaz2Au6/rREcdgMlWllo4gONZI28TaeFugKCrksHHLgAiAl2M0XWaegw8q2PG7XtPQTDGlwjijQnre9x+n77yV/drgkE6s= X-Alimail-AntiSpam: AC=PASS;BC=-1|-1;BR=01201311R191e4;CH=green;DM=||false|;DS=||;FP=0|-1|-1|-1|0|-1|-1|-1;HT=ay29a033018045168;MF=baolin.wang@linux.alibaba.com;NM=1;PH=DS;RN=13;SR=0;TI=SMTPD_---0W5.T-Y-_1713769375; Received: from localhost(mailfrom:baolin.wang@linux.alibaba.com fp:SMTPD_---0W5.T-Y-_1713769375) by smtp.aliyun-inc.com; Mon, 22 Apr 2024 15:02:57 +0800 From: Baolin Wang To: akpm@linux-foundation.org, hughd@google.com Cc: willy@infradead.org, david@redhat.com, wangkefeng.wang@huawei.com, 21cnbao@gmail.com, ryan.roberts@arm.com, ying.huang@intel.com, shy828301@gmail.com, ziy@nvidia.com, baolin.wang@linux.alibaba.com, linux-mm@kvack.org, linux-kernel@vger.kernel.org Subject: [RFC PATCH 1/5] mm: memory: extend finish_fault() to support large folio Date: Mon, 22 Apr 2024 15:02:39 +0800 Message-Id: <358aefb1858b63164894d7d8504f3dae0b495366.1713755580.git.baolin.wang@linux.alibaba.com> X-Mailer: git-send-email 2.39.3 In-Reply-To: References: MIME-Version: 1.0 X-Stat-Signature: mxo5gb9n8zs1muqgudzjoet8z4numpjz X-Rspamd-Queue-Id: AFAC5140011 X-Rspamd-Server: rspam06 X-Rspam-User: X-HE-Tag: 1713769381-714974 X-HE-Meta: U2FsdGVkX188HN7dYdHOI6cJiDVEO3Df0pADI/V1PK9JeLfM7R+nllY2BLak8K/juRTyXakxM8B4b8AsmAIi/yEDRHzfPpd5yK4ORa39ZPV1JmQ3dlrfTLGoluXyuCpuAcjlK8Dx0M8xB1eOwjU2APDMBPIF2oBvHE+SBVhFfE0dCTLGV5D+wT3qivkAlWU4uzmEnG01WWT7esIR1AV+rLegn6HDUFw/UbaSdLCMyYpvXmblWhOGwL8j5XgY9dn1Jhp2TwTmDjB4zwlB+QMs0UipAd4IMRgP3GP3rUvyIIcPdpEfYg7v/QxddLZKNDnCSsUxzIBiUYvnqYM2E6JXUHKvhylgS7MadD2sDGWbW8ktb0y0TSklHx2wzgN/aizilkzp3fAfifqkuMxy3BP8TxkVv4TqPahQOfCJ5lS3XMwLW2pykCnRuotyUZWadw2Nty0PLUbSTzApQDwWGutV3FI9rzQbplXPaflzcnzVXYi8Vcv1aSvAwjU/x/S64dBFgeIC8o8Tm8/KAkygs3Hm6wv8n8NJSSvKUoeQKR9UKttjSg2ZqLvtYoxLso10k3ZDnImaonebX3h1aQq6vkep+0wamSIWJB8S/QyVIqKIqK8LchdjM/fWG/4YCQ03FkwL1eDTTV4Kg6us4Ufr6kMYAjxbVOy86zQXlfs5TtpY96eClNmWKQoa2//6crrZxKolQLn0DqT8sE2b5KbymuxdPChjD5uliMX01zcaClj2iDXJ7+dM0G2KXikMWrb3U+gWgDpkkO0Q3wiEZKAJWq7fz5fvJxNzfkrvtL2tR7GYjTEfclSugS3rnn7LPprQ8YwAY6ZaKsdsD82ExmOKM4WLLfoblYv10hOAAHiwiEqhY3t/0ZzbWat5XEX6b/u+Hao6Ms9zFaUfNabvwiZs0eBKU61KyPuCztnYk+KrpbSOyyvAbkHg+L7lQPjFPfsiB58K7SrG56n7QYW9LVUmbZq Q2vJE9sW fgD/uSHbgbZxAuJdbyxMBxZ6ttuLg2a2kPrn7puWc3cbjU27ebdBCzGOiyqbO0iANACwiWI7LiwhrLRRFGEtGedbswteDVY74PIRr56zy4VfDfpvEt1uqswE5ngONPzY9eLtJPRAkT9o9WefLIQ17pVsYheaHBbfCE85+tjFDLHuKoOPk4OtamGaQIq1ShqVsyfdHqHy16OskYjxjSeO6Pgt85bKnXEmn87I6ZWm5kUKB5KUjydK3cgHHtMcbI1gBfIFzvdFZQo7if+zznijRTTUnG34U8QZ1cm1/nUZIif1zuiBqSOv5Uo3O34kWoomD7p5d X-Bogosity: Ham, tests=bogofilter, spamicity=0.000000, version=1.2.4 Sender: owner-linux-mm@kvack.org Precedence: bulk X-Loop: owner-majordomo@kvack.org List-ID: List-Subscribe: List-Unsubscribe: Add large folio mapping establishment support for finish_fault() as a preparation, to support multi-size THP allocation of anonymous shared pages in the following patches. Signed-off-by: Baolin Wang --- mm/memory.c | 25 ++++++++++++++++++------- 1 file changed, 18 insertions(+), 7 deletions(-) diff --git a/mm/memory.c b/mm/memory.c index b6fa5146b260..094a76730776 100644 --- a/mm/memory.c +++ b/mm/memory.c @@ -4766,7 +4766,10 @@ vm_fault_t finish_fault(struct vm_fault *vmf) { struct vm_area_struct *vma = vmf->vma; struct page *page; + struct folio *folio; vm_fault_t ret; + int nr_pages, i; + unsigned long addr; /* Did we COW the page? */ if ((vmf->flags & FAULT_FLAG_WRITE) && !(vma->vm_flags & VM_SHARED)) @@ -4797,22 +4800,30 @@ vm_fault_t finish_fault(struct vm_fault *vmf) return VM_FAULT_OOM; } + folio = page_folio(page); + nr_pages = folio_nr_pages(folio); + addr = ALIGN_DOWN(vmf->address, nr_pages * PAGE_SIZE); vmf->pte = pte_offset_map_lock(vma->vm_mm, vmf->pmd, - vmf->address, &vmf->ptl); + addr, &vmf->ptl); if (!vmf->pte) return VM_FAULT_NOPAGE; /* Re-check under ptl */ - if (likely(!vmf_pte_changed(vmf))) { - struct folio *folio = page_folio(page); - - set_pte_range(vmf, folio, page, 1, vmf->address); - ret = 0; - } else { + if (nr_pages == 1 && vmf_pte_changed(vmf)) { update_mmu_tlb(vma, vmf->address, vmf->pte); ret = VM_FAULT_NOPAGE; + goto unlock; + } else if (nr_pages > 1 && !pte_range_none(vmf->pte, nr_pages)) { + for (i = 0; i < nr_pages; i++) + update_mmu_tlb(vma, addr + PAGE_SIZE * i, vmf->pte + i); + ret = VM_FAULT_NOPAGE; + goto unlock; } + set_pte_range(vmf, folio, &folio->page, nr_pages, addr); + ret = 0; + +unlock: pte_unmap_unlock(vmf->pte, vmf->ptl); return ret; } From patchwork Mon Apr 22 07:02:40 2024 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Baolin Wang X-Patchwork-Id: 13637717 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from kanga.kvack.org (kanga.kvack.org [205.233.56.17]) by smtp.lore.kernel.org (Postfix) with ESMTP id AAE7AC4345F for ; Mon, 22 Apr 2024 07:03:09 +0000 (UTC) Received: by kanga.kvack.org (Postfix) id D7F8B6B008A; Mon, 22 Apr 2024 03:03:05 -0400 (EDT) Received: by kanga.kvack.org (Postfix, from userid 40) id D2F656B008C; Mon, 22 Apr 2024 03:03:05 -0400 (EDT) X-Delivered-To: int-list-linux-mm@kvack.org Received: by kanga.kvack.org (Postfix, from userid 63042) id BD0176B0093; Mon, 22 Apr 2024 03:03:05 -0400 (EDT) X-Delivered-To: linux-mm@kvack.org Received: from relay.hostedemail.com (smtprelay0011.hostedemail.com [216.40.44.11]) by kanga.kvack.org (Postfix) with ESMTP id 9D3216B008C for ; Mon, 22 Apr 2024 03:03:05 -0400 (EDT) Received: from smtpin07.hostedemail.com (a10.router.float.18 [10.200.18.1]) by unirelay05.hostedemail.com (Postfix) with ESMTP id 5139840ABB for ; Mon, 22 Apr 2024 07:03:05 +0000 (UTC) X-FDA: 82036275930.07.AEB1845 Received: from out30-133.freemail.mail.aliyun.com (out30-133.freemail.mail.aliyun.com [115.124.30.133]) by imf26.hostedemail.com (Postfix) with ESMTP id 4C168140023 for ; Mon, 22 Apr 2024 07:03:02 +0000 (UTC) Authentication-Results: imf26.hostedemail.com; dkim=pass header.d=linux.alibaba.com header.s=default header.b=tDCV8JxE; spf=pass (imf26.hostedemail.com: domain of baolin.wang@linux.alibaba.com designates 115.124.30.133 as permitted sender) smtp.mailfrom=baolin.wang@linux.alibaba.com; dmarc=pass (policy=none) header.from=linux.alibaba.com ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=hostedemail.com; s=arc-20220608; t=1713769383; h=from:from:sender:reply-to:subject:subject:date:date: message-id:message-id:to:to:cc:cc:mime-version:mime-version: content-type:content-transfer-encoding:content-transfer-encoding: in-reply-to:in-reply-to:references:references:dkim-signature; bh=BV2U/IeVyqp2XSH/3wx9gpceXxijONOM8cRleKyQmTE=; b=X4tm+FYSpZq0NsjVPfBcgod5vW/Ojoza5a+AYbUGVoZOJlJyOR2OzgtCIqAQrctUl3CQjG ItrlyIMB73M26WG9cQl1/Ir5b5C5VSjVHA3TgQL2Rg17NCj4dnsNzLEVBQdXEkV/t9Ki0K 1nfIorjAeASJ7DkdkfHxO8vVCDyOxs0= ARC-Authentication-Results: i=1; imf26.hostedemail.com; dkim=pass header.d=linux.alibaba.com header.s=default header.b=tDCV8JxE; spf=pass (imf26.hostedemail.com: domain of baolin.wang@linux.alibaba.com designates 115.124.30.133 as permitted sender) smtp.mailfrom=baolin.wang@linux.alibaba.com; dmarc=pass (policy=none) header.from=linux.alibaba.com ARC-Seal: i=1; s=arc-20220608; d=hostedemail.com; t=1713769383; a=rsa-sha256; cv=none; b=c4wicyA0nRzKnd/8wLdPxIx38uYnuyE9dVG8568Y+2bXG5OXJNe/xh7iwtDZcGrrfYNN2s WsDeDDBmsvzORw4tCvfJhTNHmI2b6UpRi1xfjKiNMuPD18MD+J1uOtQXtbbZCYOK1SVDFw Kxkspwn0vpYl/mpggcbFEhCBD8Nlmw0= DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=linux.alibaba.com; s=default; t=1713769380; h=From:To:Subject:Date:Message-Id:MIME-Version; bh=BV2U/IeVyqp2XSH/3wx9gpceXxijONOM8cRleKyQmTE=; b=tDCV8JxEBCbqsQDyR0vP1FoyKkyzBNksQ1vUo9tSdXVQLC+8aj9bSUOj1fJ8OyL1HC1zu90J4qQ21uq0mOdfF2zWdEUhtdb7Q5z29x2j3J+9g9y9v/iUG9NQL5MtAuuh6jL4rlYLap2RbNR5G7N5jlTe9dTMuZ3xMRCuNqwqEys= X-Alimail-AntiSpam: AC=PASS;BC=-1|-1;BR=01201311R891e4;CH=green;DM=||false|;DS=||;FP=0|-1|-1|-1|0|-1|-1|-1;HT=ay29a033018045176;MF=baolin.wang@linux.alibaba.com;NM=1;PH=DS;RN=13;SR=0;TI=SMTPD_---0W5.T-YL_1713769377; Received: from localhost(mailfrom:baolin.wang@linux.alibaba.com fp:SMTPD_---0W5.T-YL_1713769377) by smtp.aliyun-inc.com; Mon, 22 Apr 2024 15:02:58 +0800 From: Baolin Wang To: akpm@linux-foundation.org, hughd@google.com Cc: willy@infradead.org, david@redhat.com, wangkefeng.wang@huawei.com, 21cnbao@gmail.com, ryan.roberts@arm.com, ying.huang@intel.com, shy828301@gmail.com, ziy@nvidia.com, baolin.wang@linux.alibaba.com, linux-mm@kvack.org, linux-kernel@vger.kernel.org Subject: [RFC PATCH 2/5] mm: shmem: add an 'order' parameter for shmem_alloc_hugefolio() Date: Mon, 22 Apr 2024 15:02:40 +0800 Message-Id: <2014bf7370d78bc1f5600731af5bf8f569e5868b.1713755580.git.baolin.wang@linux.alibaba.com> X-Mailer: git-send-email 2.39.3 In-Reply-To: References: MIME-Version: 1.0 X-Rspamd-Queue-Id: 4C168140023 X-Stat-Signature: jgoce81w1d1q15hw66uais4hndm1inj1 X-Rspam-User: X-Rspamd-Server: rspam07 X-HE-Tag: 1713769382-727366 X-HE-Meta: U2FsdGVkX18wPj53CdkZf0bZxlp6c5WD0LQNXL+8XUnmSDa8riN8r/TchVyKDClZCJRZo8Wp7K9kgwwxteTLSPHDlAarpCSNkCDess7LT/c6Q+X5gO1F5SOtc50y5dg+62DmhK6U2VtpZSg+IZ8smbJS5svPYt6FeyAhnxs4FcEQ6Q84x+OGQhCrH7Q5da2Qggz9bPQOYXXo6EhAh1Mw0EP++MAcEYKjL/60f0M0ZLZGbFkS/Q7iEfpOVaoViXXQzYUJjJWWEb2ea9LyGSQSmcWAAxEDHmDOY2wJm2m0D9vb6yW1M1B+XhoXLNEOb+ZAjLcJnWOVkBB5k9RJxmNrtJEw+hJ7h+fkzlkTfHJ1SyQqYP8GBch4G+WOJKE2nQ7CYn6WbdCgZxh42mN8IAPoQjCoFoQEclH8T7cDk5f3s5mJ10dT++TMK3ngaDobHOokYng5YMHL0aSsj1ZFrs2OYAZUAYq5PBfDHM++csOv3NHmMTEaM3DrnrpY0vh/VtATU0Lx904712mV42jY8BhU9WU0RKKDnwta+s+sOwnFz4fmPNmcFkEBGr/5+sHfYge2duO51bHDZvvQcB/1K/mjf4ovGxY4p25sESIa5PPJ4RPognwxdWnrONHdMftMekDfhDxtoWfm29ba4ZYwFrT/OltoNl+7yulamXT86eNwehQLEx6LjwSbRe0onGTMcx3Vb9vCnBWVFx+9aEB0B1LLzr1OI+jdzyBEYjtIL5ruWluR3mKMSZsLHoUkat96EdVGhe8nCJylXJrMDCZxlg4HFrrd2dlTHTMvr2XjUesZsmBpp+kUpekfR+Eoh1irMzO3OEvLyLSgHcLGVL1j4WTON0cgJ0ZFih49qC0+2wpmnI+9I+foQNaotRiD7mk/1kx0H7rdEoIrHsWR0t5Kvmf+oFQnHEWAQ9Taw3OcZtVh4gfA+/lxzFojva4yWcZE34aNhQUfs96/5nRslQa2PvF eUNXyaWL kJq8UCwBgC/oSEc302u6d4pTtY4Ao+0gGynflMrQ7F1zTBCGfRLlE5o8wi2fCtdkOblh65NHKzz8a3j4nVWINXYcf8rLSSjgOBU+p/JtwXlrYlWVlWX4ox/o5FB1zSkiv5rgD+/xpSlr5T6Qb+LL2ysYhhnYqdVhcoijPXFtgrelwjk9QCRH6guXkNZvRAoyc+96toZv2c/JQf7RKF97Jgw6XT4iqY5Fb9lGTeFXLLLHREJho53Fl5lvFb2SZn+3AKiHuq57X+8/FP3hqNeUO5ItqVvgh4KMvsCKQ X-Bogosity: Ham, tests=bogofilter, spamicity=0.000000, version=1.2.4 Sender: owner-linux-mm@kvack.org Precedence: bulk X-Loop: owner-majordomo@kvack.org List-ID: List-Subscribe: List-Unsubscribe: Add a new parameter to specify the huge page order for shmem_alloc_hugefolio(), as a preparation to supoort mTHP. Signed-off-by: Baolin Wang --- mm/shmem.c | 11 ++++++----- 1 file changed, 6 insertions(+), 5 deletions(-) diff --git a/mm/shmem.c b/mm/shmem.c index fa2a0ed97507..893c88efc45f 100644 --- a/mm/shmem.c +++ b/mm/shmem.c @@ -1604,14 +1604,14 @@ static gfp_t limit_gfp_mask(gfp_t huge_gfp, gfp_t limit_gfp) } static struct folio *shmem_alloc_hugefolio(gfp_t gfp, - struct shmem_inode_info *info, pgoff_t index) + struct shmem_inode_info *info, pgoff_t index, int order) { struct mempolicy *mpol; pgoff_t ilx; struct page *page; - mpol = shmem_get_pgoff_policy(info, index, HPAGE_PMD_ORDER, &ilx); - page = alloc_pages_mpol(gfp, HPAGE_PMD_ORDER, mpol, ilx, numa_node_id()); + mpol = shmem_get_pgoff_policy(info, index, order, &ilx); + page = alloc_pages_mpol(gfp, order, mpol, ilx, numa_node_id()); mpol_cond_put(mpol); return page_rmappable_folio(page); @@ -1639,13 +1639,14 @@ static struct folio *shmem_alloc_and_add_folio(gfp_t gfp, struct shmem_inode_info *info = SHMEM_I(inode); struct folio *folio; long pages; - int error; + int error, order; if (!IS_ENABLED(CONFIG_TRANSPARENT_HUGEPAGE)) huge = false; if (huge) { pages = HPAGE_PMD_NR; + order = HPAGE_PMD_ORDER; index = round_down(index, HPAGE_PMD_NR); /* @@ -1660,7 +1661,7 @@ static struct folio *shmem_alloc_and_add_folio(gfp_t gfp, index + HPAGE_PMD_NR - 1, XA_PRESENT)) return ERR_PTR(-E2BIG); - folio = shmem_alloc_hugefolio(gfp, info, index); + folio = shmem_alloc_hugefolio(gfp, info, index, order); if (!folio) count_vm_event(THP_FILE_FALLBACK); } else { From patchwork Mon Apr 22 07:02:41 2024 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Baolin Wang X-Patchwork-Id: 13637718 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from kanga.kvack.org (kanga.kvack.org [205.233.56.17]) by smtp.lore.kernel.org (Postfix) with ESMTP id 71BC7C4345F for ; Mon, 22 Apr 2024 07:03:13 +0000 (UTC) Received: by kanga.kvack.org (Postfix) id D5BEC6B008C; Mon, 22 Apr 2024 03:03:06 -0400 (EDT) Received: by kanga.kvack.org (Postfix, from userid 40) id CE1AB6B0092; Mon, 22 Apr 2024 03:03:06 -0400 (EDT) X-Delivered-To: int-list-linux-mm@kvack.org Received: by kanga.kvack.org (Postfix, from userid 63042) id B83D46B0093; Mon, 22 Apr 2024 03:03:06 -0400 (EDT) X-Delivered-To: linux-mm@kvack.org Received: from relay.hostedemail.com (smtprelay0010.hostedemail.com [216.40.44.10]) by kanga.kvack.org (Postfix) with ESMTP id 95FA16B008C for ; Mon, 22 Apr 2024 03:03:06 -0400 (EDT) Received: from smtpin07.hostedemail.com (a10.router.float.18 [10.200.18.1]) by unirelay02.hostedemail.com (Postfix) with ESMTP id 443CF120A4E for ; Mon, 22 Apr 2024 07:03:06 +0000 (UTC) X-FDA: 82036275972.07.DDE9084 Received: from out30-118.freemail.mail.aliyun.com (out30-118.freemail.mail.aliyun.com [115.124.30.118]) by imf23.hostedemail.com (Postfix) with ESMTP id D3AA8140018 for ; Mon, 22 Apr 2024 07:03:03 +0000 (UTC) Authentication-Results: imf23.hostedemail.com; dkim=pass header.d=linux.alibaba.com header.s=default header.b=goBv34hw; spf=pass (imf23.hostedemail.com: domain of baolin.wang@linux.alibaba.com designates 115.124.30.118 as permitted sender) smtp.mailfrom=baolin.wang@linux.alibaba.com; dmarc=pass (policy=none) header.from=linux.alibaba.com ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=hostedemail.com; s=arc-20220608; t=1713769384; h=from:from:sender:reply-to:subject:subject:date:date: message-id:message-id:to:to:cc:cc:mime-version:mime-version: content-type:content-transfer-encoding:content-transfer-encoding: in-reply-to:in-reply-to:references:references:dkim-signature; bh=0SsdcViQ2lxDH4gZYEPPhvXdWD/avqV+Env2Q3pLz7o=; b=vBzhKrxG5yido2dpEZuX+cP61DhCfytKqh5NN6YNBDDQeqpdWbe0qyaDEZjqpbsWYzBNb1 y2LqozkJba4O/dttsS1x9CjGQuP77h9BLzkqIUIN9nCAyudz3YTcbtisoQG3WQCmq4zcem jgbPTeeNpMwdVP7HaJwHp3tEFttgmRI= ARC-Seal: i=1; s=arc-20220608; d=hostedemail.com; t=1713769384; a=rsa-sha256; cv=none; b=Nu/dYOObz7ix33LIiICOtyxfq6jggWEm4XRaTpFq9+E6i6msWhwA15BRSIP4x04y/kea9e MH+KA+KqpfebxBaCgVmatZbw3lgUEcZ3OzQAUqlR2J9f4PV8+qBecdt7lt4x5FTT1WsI2q ygdqW3J2aA2oG+c8GamnTJbZSEnbfAc= ARC-Authentication-Results: i=1; imf23.hostedemail.com; dkim=pass header.d=linux.alibaba.com header.s=default header.b=goBv34hw; spf=pass (imf23.hostedemail.com: domain of baolin.wang@linux.alibaba.com designates 115.124.30.118 as permitted sender) smtp.mailfrom=baolin.wang@linux.alibaba.com; dmarc=pass (policy=none) header.from=linux.alibaba.com DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=linux.alibaba.com; s=default; t=1713769381; h=From:To:Subject:Date:Message-Id:MIME-Version; bh=0SsdcViQ2lxDH4gZYEPPhvXdWD/avqV+Env2Q3pLz7o=; b=goBv34hww93Jw8e3t4O4G8/ufkQgeZVJPNDOb3rqoX9XQhrlJAT2O4fZRs9/hlu+FvA1+/mfSnm9tSQr/+tOEHwv1iSKQnBjA66Az/Hv1YLNMWKnQJ1nm6mp9L/LikHGKe32xORmnNg/TCsFSFgoV3UYW3hAY0yMJGjNLb/yA0c= X-Alimail-AntiSpam: AC=PASS;BC=-1|-1;BR=01201311R121e4;CH=green;DM=||false|;DS=||;FP=0|-1|-1|-1|0|-1|-1|-1;HT=ay29a033018046051;MF=baolin.wang@linux.alibaba.com;NM=1;PH=DS;RN=13;SR=0;TI=SMTPD_---0W5-amKe_1713769378; Received: from localhost(mailfrom:baolin.wang@linux.alibaba.com fp:SMTPD_---0W5-amKe_1713769378) by smtp.aliyun-inc.com; Mon, 22 Apr 2024 15:02:59 +0800 From: Baolin Wang To: akpm@linux-foundation.org, hughd@google.com Cc: willy@infradead.org, david@redhat.com, wangkefeng.wang@huawei.com, 21cnbao@gmail.com, ryan.roberts@arm.com, ying.huang@intel.com, shy828301@gmail.com, ziy@nvidia.com, baolin.wang@linux.alibaba.com, linux-mm@kvack.org, linux-kernel@vger.kernel.org Subject: [RFC PATCH 3/5] mm: shmem: add THP validation for PMD-mapped THP related statistics Date: Mon, 22 Apr 2024 15:02:41 +0800 Message-Id: <4107dcc957f3b62a37c83e30ca22305c373ef149.1713755580.git.baolin.wang@linux.alibaba.com> X-Mailer: git-send-email 2.39.3 In-Reply-To: References: MIME-Version: 1.0 X-Rspamd-Server: rspam12 X-Rspamd-Queue-Id: D3AA8140018 X-Rspam-User: X-Stat-Signature: mebjxygoe686b8w9dp7h81x43irsy4ya X-HE-Tag: 1713769383-416963 X-HE-Meta: U2FsdGVkX1/4OO/TaZbafvgraospcfnhbEWtHa5RVv0AeSZ8a1Xd3aBQRto0DlF3u+HAKLmbXofH9rE1nYZGh5ODUx21ro5q87x5b+m9aA9EmcLUYt60u9mz8kNaVIkZ4JfvpBVlqHEJ/m+IecdFoytneNlIttWv1d6EioAouhOc+roUdS7jXhZWRTpSy90GtDHPFa9LldIvOi/JQck8ia31skTGO7cVt7E6FGmJPeDawIoHXvqy51Jvku0DfFegtPxZ7e5OSUvxtwUSBfnla+Epn4cY6cUElqS1zcn19y+feEbyVL3LY1sJNIbyAQfcW9KIOYIqT05r8ydZKLxwujqgV3sa50gFxgle5zWPL/LjUYpJIhHDFey+em6svrke0iHu+q/YR6k9WpUCoObFaYZbdJIixm9IMT+rVVhb0DmOVTWsJQ4zUEjytks5qtrxL+ft6uhCjjJ3WZRntwUYaB0x2oGQ4z4F6VDw8vyeYVFirLOiS/2P0wwZiGDBuQWNaZiI1ktoncxB5UpKi9VOR5cStMmhbB7e5Ur35GLbbr1FM5RCZzhm3dqGe3BeWdTfv6/Y31XIrfbs9eRnDjFy3RfhOHklBQZLbW3rqYZ+1gtDpwUk8dNVqMep/W1mKh3AA/2lo/jl/1c+98faNkQ5L4l4gG77nS4c+2pTWtZlcJ0Ps1/wlBqc9tsf6dTLOKe4GqbWROnsZBvQSPScBVTKQKhUdpV32CyLsA+Cf6TgHz9iVQVShDJKfyJ3veVaBmICmljai9G62KyPIjIBAkQlMJDxkNnG7H1QG5qMkgDnXZK5GbiW45qJwKteQ6+eZK8aVEPl3QpduFj5sMSLdZymBGIfKrBSUm0YKAT/3AIuKQtoVmL0lj/Tq30xnqY75bH14scF/YC+pFU9arG31moC2knORg3ese8Fvmrmab+rEJT6akruc1FNIGUlHTjQopp9pYaxTBF7m1oo/gvR8jb F3v8LP7j LwBwOR7R4eUNCTMZzt2s0U1xvC/xHu4zdMG3mMphh2qWa0hTiQc+kC415htKiZjrWpXbDwDRYkq3UJ7VAq/OP87TmG8pgUhpwEy97EXXfWZ+d0E/70He99tqqRx5Vq+keBoEsQ1GrknkfT4uP3eEbZ4N8gVM58PVF3BImwXYT5vBEZEKPp4Lwin3d4Jk6RKh1P1hyaI0S4RtIwawOwlt5MekFgGoXoqRWfbViIPw10xkOnR0= X-Bogosity: Ham, tests=bogofilter, spamicity=0.000000, version=1.2.4 Sender: owner-linux-mm@kvack.org Precedence: bulk X-Loop: owner-majordomo@kvack.org List-ID: List-Subscribe: List-Unsubscribe: In order to extend support for mTHP, add THP validation for PMD-mapped THP related statistics to avoid statistical confusion. Signed-off-by: Baolin Wang Reviewed-by: Barry Song --- mm/shmem.c | 7 ++++--- 1 file changed, 4 insertions(+), 3 deletions(-) diff --git a/mm/shmem.c b/mm/shmem.c index 893c88efc45f..b4afda71a3f0 100644 --- a/mm/shmem.c +++ b/mm/shmem.c @@ -1662,7 +1662,7 @@ static struct folio *shmem_alloc_and_add_folio(gfp_t gfp, return ERR_PTR(-E2BIG); folio = shmem_alloc_hugefolio(gfp, info, index, order); - if (!folio) + if (!folio && pages == HPAGE_PMD_NR) count_vm_event(THP_FILE_FALLBACK); } else { pages = 1; @@ -1680,7 +1680,7 @@ static struct folio *shmem_alloc_and_add_folio(gfp_t gfp, if (xa_find(&mapping->i_pages, &index, index + pages - 1, XA_PRESENT)) { error = -EEXIST; - } else if (huge) { + } else if (pages == HPAGE_PMD_NR) { count_vm_event(THP_FILE_FALLBACK); count_vm_event(THP_FILE_FALLBACK_CHARGE); } @@ -2046,7 +2046,8 @@ static int shmem_get_folio_gfp(struct inode *inode, pgoff_t index, folio = shmem_alloc_and_add_folio(huge_gfp, inode, index, fault_mm, true); if (!IS_ERR(folio)) { - count_vm_event(THP_FILE_ALLOC); + if (folio_test_pmd_mappable(folio)) + count_vm_event(THP_FILE_ALLOC); goto alloced; } if (PTR_ERR(folio) == -EEXIST) From patchwork Mon Apr 22 07:02:42 2024 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Baolin Wang X-Patchwork-Id: 13637719 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from kanga.kvack.org (kanga.kvack.org [205.233.56.17]) by smtp.lore.kernel.org (Postfix) with ESMTP id 0C15BC4345F for ; Mon, 22 Apr 2024 07:03:17 +0000 (UTC) Received: by kanga.kvack.org (Postfix) id CE4DC6B0092; Mon, 22 Apr 2024 03:03:07 -0400 (EDT) Received: by kanga.kvack.org (Postfix, from userid 40) id C6D396B0095; Mon, 22 Apr 2024 03:03:07 -0400 (EDT) X-Delivered-To: int-list-linux-mm@kvack.org Received: by kanga.kvack.org (Postfix, from userid 63042) id ABFD46B0096; Mon, 22 Apr 2024 03:03:07 -0400 (EDT) X-Delivered-To: linux-mm@kvack.org Received: from relay.hostedemail.com (smtprelay0011.hostedemail.com [216.40.44.11]) by kanga.kvack.org (Postfix) with ESMTP id 8BF1B6B0092 for ; Mon, 22 Apr 2024 03:03:07 -0400 (EDT) Received: from smtpin14.hostedemail.com (a10.router.float.18 [10.200.18.1]) by unirelay06.hostedemail.com (Postfix) with ESMTP id 410DBA17F0 for ; Mon, 22 Apr 2024 07:03:07 +0000 (UTC) X-FDA: 82036276014.14.F01079B Received: from out30-112.freemail.mail.aliyun.com (out30-112.freemail.mail.aliyun.com [115.124.30.112]) by imf05.hostedemail.com (Postfix) with ESMTP id 15066100007 for ; Mon, 22 Apr 2024 07:03:04 +0000 (UTC) Authentication-Results: imf05.hostedemail.com; dkim=pass header.d=linux.alibaba.com header.s=default header.b=sIJvjLOp; spf=pass (imf05.hostedemail.com: domain of baolin.wang@linux.alibaba.com designates 115.124.30.112 as permitted sender) smtp.mailfrom=baolin.wang@linux.alibaba.com; dmarc=pass (policy=none) header.from=linux.alibaba.com ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=hostedemail.com; s=arc-20220608; t=1713769385; h=from:from:sender:reply-to:subject:subject:date:date: message-id:message-id:to:to:cc:cc:mime-version:mime-version: content-type:content-transfer-encoding:content-transfer-encoding: in-reply-to:in-reply-to:references:references:dkim-signature; bh=7TGllQsMjthzKZ5x9FWnodK5JAUiOYpAC2Vw3I8Fy2s=; b=n1+uMDtkTCsF4/3x+AeSNy9Ud4Il1SnJcbCxFZ4sGvvi8VAipBpJxHe4cXha7hCkLk+CPb pFXSvJgWl0af4+Y1FJLMd54SFGwriFgRStCF8vE9WaH3zsSruNBt5EZjacHYWuDQQj1cs8 nEAHFGzx+8gD01n+oeoOk1T0K63p2L4= ARC-Seal: i=1; s=arc-20220608; d=hostedemail.com; t=1713769385; a=rsa-sha256; cv=none; b=LwzZOY+JdNuTiLoDJKTVWoK0mZI95Iee9mXtlMOHD7OgxmK9SV1RG/gb+5wTwwsy0D7OJ+ DHY8uaU4yjZIe4KKEPZ13LC8BWLuTbo5fqKPo38Owip1oXJoGQH6fHma8nQfV/AfQfA2xE eBkx/Wm0yaHd8cI7ei3CwfTkBGR/urk= ARC-Authentication-Results: i=1; imf05.hostedemail.com; dkim=pass header.d=linux.alibaba.com header.s=default header.b=sIJvjLOp; spf=pass (imf05.hostedemail.com: domain of baolin.wang@linux.alibaba.com designates 115.124.30.112 as permitted sender) smtp.mailfrom=baolin.wang@linux.alibaba.com; dmarc=pass (policy=none) header.from=linux.alibaba.com DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=linux.alibaba.com; s=default; t=1713769382; h=From:To:Subject:Date:Message-Id:MIME-Version; bh=7TGllQsMjthzKZ5x9FWnodK5JAUiOYpAC2Vw3I8Fy2s=; b=sIJvjLOpoGBVpSh6ATOaD4yiBb1zUyVgTz+1eP0lm88bEMO2lnfTy+BvSqFKR6TF7MrYCMDtavG1ZK4n+EAAD7FwQb/B3zzp5hE6DPIXEbRe22rTVSwcGMTYOLXXi0htrr6ls9EFcXFdyiS3Q2VL9Q3sa0yn1q1vUkSutQMPTS8= X-Alimail-AntiSpam: AC=PASS;BC=-1|-1;BR=01201311R171e4;CH=green;DM=||false|;DS=||;FP=0|-1|-1|-1|0|-1|-1|-1;HT=ay29a033018046059;MF=baolin.wang@linux.alibaba.com;NM=1;PH=DS;RN=13;SR=0;TI=SMTPD_---0W5.WnjE_1713769380; Received: from localhost(mailfrom:baolin.wang@linux.alibaba.com fp:SMTPD_---0W5.WnjE_1713769380) by smtp.aliyun-inc.com; Mon, 22 Apr 2024 15:03:01 +0800 From: Baolin Wang To: akpm@linux-foundation.org, hughd@google.com Cc: willy@infradead.org, david@redhat.com, wangkefeng.wang@huawei.com, 21cnbao@gmail.com, ryan.roberts@arm.com, ying.huang@intel.com, shy828301@gmail.com, ziy@nvidia.com, baolin.wang@linux.alibaba.com, linux-mm@kvack.org, linux-kernel@vger.kernel.org Subject: [RFC PATCH 4/5] mm: shmem: add mTHP support for anonymous share pages Date: Mon, 22 Apr 2024 15:02:42 +0800 Message-Id: <8f2725a856dc02c7c89b4139cc3628686c9de524.1713755580.git.baolin.wang@linux.alibaba.com> X-Mailer: git-send-email 2.39.3 In-Reply-To: References: MIME-Version: 1.0 X-Stat-Signature: hdkeet579bkcbd469b7n4ioxchwaw4fr X-Rspamd-Queue-Id: 15066100007 X-Rspamd-Server: rspam06 X-Rspam-User: X-HE-Tag: 1713769384-967460 X-HE-Meta: U2FsdGVkX19TdnocLpxf8nAJs3ypR5EUlXjV1v3u8Vl/29GnDG2hMJINAB3oAxEeXRh6Nan5milQyKJ8ss49yu2Ht7WRXL9y+o/rUe6zZCm3dXlyv13N37WGB5nHZgsozyajNK8cXHgWVtuuHk04ByeE9isowmYKB7cYaShRJZP/EUwCWnnRPt79UnUGCWB5i0baGk8Jw3Zgh8Yy3NyP0/c7EJFH7mqBEcWzBRELKEoOdKhoEo1pXaQJgUufQDXUrvwSBIsijDl8pwb7Zr03kHmd2RFoD3lqvJOYKUOTAPN7OgSSTPvtqCKztQNKuplsZVaCqIWuXE6UnwPxMpY8FTJF8zReTLw+M4Oc209NvT/qZIFAVYPkpPB2bWb4tKEHrwYDLszqfGFlXqrq+vO9BdXspQTH2bKVcrftUbfT8pgFmWW97gfOFImfajkTmdTue6QNfnIC9YVTD7q12JApZvbzOTt3AUZJP/LkyISh8aM1Q8oPJTDWg4/4Sm04plvbmMYkxTWQxwdt9r2ik7+4KON7af8Qj4Er2lrzco0e6QIZB0gmwf+d44P39Fz+TWPW8bgu8xO2Yzjqa3DiYoGHy6Rb1pEzoe2X3omfSPiXhwb3GXML0mRwAIF4T87pnxh5UA6ngIoFjSyaqHJ+6yhx2RLjkdd1fdG1wR03B5qU1Nus6or2uoHDm/6Tl/XQu70Z9zp5kInPSH+K6VIDn9pnieXwT8NCvdMKgLzgwnUXawqNILDzLfIRMk6RUTH8GlWslQeZgzYxEGOzD8UEsXfnqumi6AVYLXTIfgi+uNzmrwN8wCUivfIsPPvQo+JyKR7PW8VkJg2cZi14w5Qb2HaiIcwKLtgPIjsPZSbnx3sYjuz9cHSAAqZ5iZP1NH4CdYlcGdHYkhLl6pv0m5GDXpVWDs821z2pW+8ELzW2PTEYr+KI3ycDfcMAPfiqMfUPrAAFZUG47NLq7IFQqsRceZA 2uGVPTZx CN4lYDEFypZZEW3bOwsyTbH9yLFKWRZ/KOtxHcHqhqW1xhYx/UxCVoj0ChMLfjXllKZZtwXi/ZFvzR9x5juXfZ3+UIzrJ1m8cGa8RWmfyvqOAmXbDS0114b3nWCtiZZdiSbvchYWBJVJjtvc5cKKI2kR7edqGL9aDL+mgBYWNONLIo+L1ivnpOste9g== X-Bogosity: Ham, tests=bogofilter, spamicity=0.000000, version=1.2.4 Sender: owner-linux-mm@kvack.org Precedence: bulk X-Loop: owner-majordomo@kvack.org List-ID: List-Subscribe: List-Unsubscribe: Commit 19eaf44954df adds multi-size THP (mTHP) for anonymous pages, that can allow THP to be configured through the sysfs interface located at '/sys/kernel/mm/transparent_hugepage/hugepage-XXkb/enabled'. However, the anonymous share pages will ignore the anonymous mTHP rule configured through the sysfs interface, and can only use the PMD-mapped THP, that is not reasonable. Users expect to apply the mTHP rule for all anonymous pages, including the anonymous share pages, in order to enjoy the benefits of mTHP. For example, lower latency than PMD-mapped THP, smaller memory bloat than PMD-mapped THP, contiguous PTEs on ARM architecture to reduce TLB miss etc. The primary strategy is that, the use of huge pages for anonymous shared pages still follows the global control determined by the mount option "huge=" parameter or the sysfs interface at '/sys/kernel/mm/transparent_hugepage/shmem_enabled'. The utilization of mTHP is allowed only when the global 'huge' switch is enabled. Subsequently, the mTHP sysfs interface (/sys/kernel/mm/transparent_hugepage/hugepage-XXkb/enabled) is checked to determine the mTHP size that can be used for large folio allocation for these anonymous shared pages. Signed-off-by: Baolin Wang --- include/linux/huge_mm.h | 2 +- mm/huge_memory.c | 4 +- mm/shmem.c | 92 +++++++++++++++++++++++++++++++---------- 3 files changed, 74 insertions(+), 24 deletions(-) diff --git a/include/linux/huge_mm.h b/include/linux/huge_mm.h index b67294d5814f..26b6fa98d8ac 100644 --- a/include/linux/huge_mm.h +++ b/include/linux/huge_mm.h @@ -246,7 +246,7 @@ unsigned long thp_vma_allowable_orders(struct vm_area_struct *vma, unsigned long orders) { /* Optimization to check if required orders are enabled early. */ - if (enforce_sysfs && vma_is_anonymous(vma)) { + if (enforce_sysfs && (vma_is_anonymous(vma) || vma_is_anon_shmem(vma))) { unsigned long mask = READ_ONCE(huge_anon_orders_always); if (vm_flags & VM_HUGEPAGE) diff --git a/mm/huge_memory.c b/mm/huge_memory.c index 9a1b57ef9c60..9e52c0db7580 100644 --- a/mm/huge_memory.c +++ b/mm/huge_memory.c @@ -86,7 +86,7 @@ unsigned long __thp_vma_allowable_orders(struct vm_area_struct *vma, unsigned long orders) { /* Check the intersection of requested and supported orders. */ - orders &= vma_is_anonymous(vma) ? + orders &= (vma_is_anonymous(vma) || vma_is_anon_shmem(vma)) ? THP_ORDERS_ALL_ANON : THP_ORDERS_ALL_FILE; if (!orders) return 0; @@ -152,7 +152,7 @@ unsigned long __thp_vma_allowable_orders(struct vm_area_struct *vma, !enforce_sysfs, vma->vm_mm, vm_flags) ? orders : 0; - if (!vma_is_anonymous(vma)) { + if (!vma_is_anonymous(vma) && !vma_is_anon_shmem(vma)) { /* * Enforce sysfs THP requirements as necessary. Anonymous vmas * were already handled in thp_vma_allowable_orders(). diff --git a/mm/shmem.c b/mm/shmem.c index b4afda71a3f0..8b009e7040b2 100644 --- a/mm/shmem.c +++ b/mm/shmem.c @@ -1603,6 +1603,39 @@ static gfp_t limit_gfp_mask(gfp_t huge_gfp, gfp_t limit_gfp) return result; } +static unsigned long anon_shmem_suitable_orders(struct vm_fault *vmf, + struct address_space *mapping, pgoff_t index) +{ + struct vm_area_struct *vma = vmf ? vmf->vma : NULL; + unsigned long orders, pages; + int order; + + /* + * Get a list of all the (large) orders below PMD_ORDER + 1 that are enabled + * for this vma. Then filter out the orders that can't be allocated over + * the faulting address and still be fully contained in the vma. + */ + orders = thp_vma_allowable_orders(vma, vma->vm_flags, false, true, true, + BIT(PMD_ORDER + 1) - 1); + orders = thp_vma_suitable_orders(vma, vmf->address, orders); + + if (!orders) + return orders; + + /* Find the highest order that can add into the page cache */ + order = highest_order(orders); + while (orders) { + pages = 1UL << order; + index = round_down(index, pages); + if (!xa_find(&mapping->i_pages, &index, + index + pages - 1, XA_PRESENT)) + break; + order = next_order(&orders, order); + } + + return orders; +} + static struct folio *shmem_alloc_hugefolio(gfp_t gfp, struct shmem_inode_info *info, pgoff_t index, int order) { @@ -1631,39 +1664,55 @@ static struct folio *shmem_alloc_folio(gfp_t gfp, return (struct folio *)page; } -static struct folio *shmem_alloc_and_add_folio(gfp_t gfp, - struct inode *inode, pgoff_t index, +static struct folio *shmem_alloc_and_add_folio(struct vm_fault *vmf, + gfp_t gfp, struct inode *inode, pgoff_t index, struct mm_struct *fault_mm, bool huge) { struct address_space *mapping = inode->i_mapping; struct shmem_inode_info *info = SHMEM_I(inode); + struct vm_area_struct *vma = vmf ? vmf->vma : NULL; struct folio *folio; long pages; int error, order; + unsigned long orders; if (!IS_ENABLED(CONFIG_TRANSPARENT_HUGEPAGE)) huge = false; if (huge) { - pages = HPAGE_PMD_NR; - order = HPAGE_PMD_ORDER; - index = round_down(index, HPAGE_PMD_NR); + if (vma && vma_is_anon_shmem(vma)) { + orders = anon_shmem_suitable_orders(vmf, mapping, index); + WARN_ON_ONCE(!orders); + } else { + pages = HPAGE_PMD_NR; + orders = BIT(HPAGE_PMD_ORDER); + index = round_down(index, HPAGE_PMD_NR); - /* - * Check for conflict before waiting on a huge allocation. - * Conflict might be that a huge page has just been allocated - * and added to page cache by a racing thread, or that there - * is already at least one small page in the huge extent. - * Be careful to retry when appropriate, but not forever! - * Elsewhere -EEXIST would be the right code, but not here. - */ - if (xa_find(&mapping->i_pages, &index, + /* + * Check for conflict before waiting on a huge allocation. + * Conflict might be that a huge page has just been allocated + * and added to page cache by a racing thread, or that there + * is already at least one small page in the huge extent. + * Be careful to retry when appropriate, but not forever! + * Elsewhere -EEXIST would be the right code, but not here. + */ + if (xa_find(&mapping->i_pages, &index, index + HPAGE_PMD_NR - 1, XA_PRESENT)) - return ERR_PTR(-E2BIG); + return ERR_PTR(-E2BIG); + } - folio = shmem_alloc_hugefolio(gfp, info, index, order); - if (!folio && pages == HPAGE_PMD_NR) - count_vm_event(THP_FILE_FALLBACK); + order = highest_order(orders); + while (orders) { + pages = 1 << order; + index = round_down(index, pages); + folio = shmem_alloc_hugefolio(gfp, info, index, order); + if (folio) + goto allocated; + + if (pages == HPAGE_PMD_NR) + count_vm_event(THP_FILE_FALLBACK); + order = next_order(&orders, order); + } } else { pages = 1; folio = shmem_alloc_folio(gfp, info, index); @@ -1671,6 +1720,7 @@ static struct folio *shmem_alloc_and_add_folio(gfp_t gfp, if (!folio) return ERR_PTR(-ENOMEM); +allocated: __folio_set_locked(folio); __folio_set_swapbacked(folio); @@ -2043,7 +2093,7 @@ static int shmem_get_folio_gfp(struct inode *inode, pgoff_t index, huge_gfp = vma_thp_gfp_mask(vma); huge_gfp = limit_gfp_mask(huge_gfp, gfp); - folio = shmem_alloc_and_add_folio(huge_gfp, + folio = shmem_alloc_and_add_folio(vmf, huge_gfp, inode, index, fault_mm, true); if (!IS_ERR(folio)) { if (folio_test_pmd_mappable(folio)) @@ -2054,7 +2104,7 @@ static int shmem_get_folio_gfp(struct inode *inode, pgoff_t index, goto repeat; } - folio = shmem_alloc_and_add_folio(gfp, inode, index, fault_mm, false); + folio = shmem_alloc_and_add_folio(vmf, gfp, inode, index, fault_mm, false); if (IS_ERR(folio)) { error = PTR_ERR(folio); if (error == -EEXIST) @@ -2065,7 +2115,7 @@ static int shmem_get_folio_gfp(struct inode *inode, pgoff_t index, alloced: alloced = true; - if (folio_test_pmd_mappable(folio) && + if (folio_test_large(folio) && DIV_ROUND_UP(i_size_read(inode), PAGE_SIZE) < folio_next_index(folio) - 1) { struct shmem_sb_info *sbinfo = SHMEM_SB(inode->i_sb); From patchwork Mon Apr 22 07:02:43 2024 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Baolin Wang X-Patchwork-Id: 13637720 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from kanga.kvack.org (kanga.kvack.org [205.233.56.17]) by smtp.lore.kernel.org (Postfix) with ESMTP id DD7D0C4345F for ; Mon, 22 Apr 2024 07:03:20 +0000 (UTC) Received: by kanga.kvack.org (Postfix) id 850896B0095; Mon, 22 Apr 2024 03:03:08 -0400 (EDT) Received: by kanga.kvack.org (Postfix, from userid 40) id 7D7BA6B0096; Mon, 22 Apr 2024 03:03:08 -0400 (EDT) X-Delivered-To: int-list-linux-mm@kvack.org Received: by kanga.kvack.org (Postfix, from userid 63042) id 684AB6B0098; Mon, 22 Apr 2024 03:03:08 -0400 (EDT) X-Delivered-To: linux-mm@kvack.org Received: from relay.hostedemail.com (smtprelay0014.hostedemail.com [216.40.44.14]) by kanga.kvack.org (Postfix) with ESMTP id 485926B0095 for ; Mon, 22 Apr 2024 03:03:08 -0400 (EDT) Received: from smtpin12.hostedemail.com (a10.router.float.18 [10.200.18.1]) by unirelay10.hostedemail.com (Postfix) with ESMTP id C6C1EC09A1 for ; Mon, 22 Apr 2024 07:03:07 +0000 (UTC) X-FDA: 82036276014.12.D26CCFF Received: from out30-118.freemail.mail.aliyun.com (out30-118.freemail.mail.aliyun.com [115.124.30.118]) by imf03.hostedemail.com (Postfix) with ESMTP id E119420021 for ; Mon, 22 Apr 2024 07:03:05 +0000 (UTC) Authentication-Results: imf03.hostedemail.com; dkim=pass header.d=linux.alibaba.com header.s=default header.b=FaxMVXv9; spf=pass (imf03.hostedemail.com: domain of baolin.wang@linux.alibaba.com designates 115.124.30.118 as permitted sender) smtp.mailfrom=baolin.wang@linux.alibaba.com; dmarc=pass (policy=none) header.from=linux.alibaba.com ARC-Seal: i=1; s=arc-20220608; d=hostedemail.com; t=1713769386; a=rsa-sha256; cv=none; b=iGhiNo0o4pHdCHFUjFzWBYRT1HxgHUVpka9ppj+8gtw0ugVAVC2chosGxy6N1WBJjGO6Ks yra5LJjpEQGDO5DrGw7eKfPwZKgzT+YGCEju3EPPlGzViyP2J/8h5+XaiKgR6bsypg8m3k ckkvLAXr3bAPZ/LNuKMCsa+NQQ2GmPo= ARC-Authentication-Results: i=1; imf03.hostedemail.com; dkim=pass header.d=linux.alibaba.com header.s=default header.b=FaxMVXv9; spf=pass (imf03.hostedemail.com: domain of baolin.wang@linux.alibaba.com designates 115.124.30.118 as permitted sender) smtp.mailfrom=baolin.wang@linux.alibaba.com; dmarc=pass (policy=none) header.from=linux.alibaba.com ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=hostedemail.com; s=arc-20220608; t=1713769386; h=from:from:sender:reply-to:subject:subject:date:date: message-id:message-id:to:to:cc:cc:mime-version:mime-version: content-type:content-transfer-encoding:content-transfer-encoding: in-reply-to:in-reply-to:references:references:dkim-signature; bh=wvGbdG/PyCf+R85lbGysoGrudwoqTXGK9KleQ+1X+Xo=; b=qxsr6pm6fbShsPJXaMV5X2O/FwXpPvxA7ip8nlYjdV9ilKAd6XGudlkg7UqJErYB/au2fF 5Fb64IndoG3e8VnfD8YrTH+tm1qkUoZhv66hniaJjrnA2Z7rvf7b2l1tQYribDZfco96uO R3FQRD60Znh2b2SHMQKXD4L4S1A/oeg= DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=linux.alibaba.com; s=default; t=1713769383; h=From:To:Subject:Date:Message-Id:MIME-Version; bh=wvGbdG/PyCf+R85lbGysoGrudwoqTXGK9KleQ+1X+Xo=; b=FaxMVXv99zb2IRvGSjadM3i1YeJTdWrY8tELEhoEGzy7MIYUzM1bfJYkw2ZObXp+KBxtwnC/o9t52yihjyVwNANo8YtArlXehxDkKn+yxBmQ7LA4F23vUPINCuAblvZCNiqaDBxPj2S0XqgWxiQa7V03dw4ym40E+4q/JmkmkYc= X-Alimail-AntiSpam: AC=PASS;BC=-1|-1;BR=01201311R421e4;CH=green;DM=||false|;DS=||;FP=0|-1|-1|-1|0|-1|-1|-1;HT=ay29a033018046056;MF=baolin.wang@linux.alibaba.com;NM=1;PH=DS;RN=13;SR=0;TI=SMTPD_---0W5-amM8_1713769381; Received: from localhost(mailfrom:baolin.wang@linux.alibaba.com fp:SMTPD_---0W5-amM8_1713769381) by smtp.aliyun-inc.com; Mon, 22 Apr 2024 15:03:02 +0800 From: Baolin Wang To: akpm@linux-foundation.org, hughd@google.com Cc: willy@infradead.org, david@redhat.com, wangkefeng.wang@huawei.com, 21cnbao@gmail.com, ryan.roberts@arm.com, ying.huang@intel.com, shy828301@gmail.com, ziy@nvidia.com, baolin.wang@linux.alibaba.com, linux-mm@kvack.org, linux-kernel@vger.kernel.org Subject: [RFC PATCH 5/5] mm: shmem: add anonymous share mTHP counters Date: Mon, 22 Apr 2024 15:02:43 +0800 Message-Id: <05d0096e4ec3e572d1d52d33a31a661321ac1551.1713755580.git.baolin.wang@linux.alibaba.com> X-Mailer: git-send-email 2.39.3 In-Reply-To: References: MIME-Version: 1.0 X-Rspamd-Server: rspam03 X-Rspamd-Queue-Id: E119420021 X-Stat-Signature: pci8jtp9duiasa9i7usr45u9xodedbtg X-Rspam-User: X-HE-Tag: 1713769385-246326 X-HE-Meta: U2FsdGVkX1/4wcPYIuv6AGu8Ch0LVY9q4xv63u1tnSKJDzaQMP8rcMsQ4fKn5Dzkrh94c6f0lsImYHaiOm0tLaGrqkY06cz0Kc3M92HUJKuMHDjYJ1cCccJ8ZG3TGBgni2g299lMY2G6nZYv+EdtN/klRTUabb5CawMZYZr7qOIYD4b6vR2mdleABMtwDXr7eP6SklLyyhrsiPi34qkgLvZeQQ9h67L07YRuHF86e/ZoQ1EW/qlYVadxPkMMJWyVvutQerwLMREEygOoqvQ/q7EVlCVRZRnPGZsF8KUSUxok1BD8P61tNDWAPvJ5Bt+SDCLP8UHmPIeT5+DoMstdAIAjpVCFlZBIDoTHw3165+LQUEYr7J+0HKmIIVNTCpn28L9ZwjsF+GT18OrRHo1TX/2lwIE8dVlqDwlT3/G3fxPH4A7OZeisbhDAeGti60PR/4uiBztSe2glvY1VJ8C5gizYSdYUXthFnx+6+e9p0re7GgkO6jpaNp8b7Qk4mrBX5wCpPvsRZ1VdpgWzSUijFyKQOlTzcjAx6BdbrG+dV9p8YQkPHXY3TTrbv5DXUZw4/uK1xduVmGX7+WR7gu4ZMYBMdu5lUquTNDLyp7gCe4g+esnN4WYPJ7pepsC6sgHiE0x67hXtQBkmvm3W9JU8NOfJynxdD5jVP7WHrEJFKaZb57ZKLxl/lj3oH37YQFRvnwr8fmQ48bru9ee92XkYdh++JFl7D1cV4wdAY/Mj4b9tA54ItBn48noH68fLZRNJ6hKSf3+j5AakVxFmIeeMBeZ99d9bq/4eCvNHO4xE1CI+hs4Bm2qqveLVrrWgPB1Mu7QbOzdWmocJ2KXOpMuyWaWZuokVj41i/FHCmdqvbM2lrBttrd8pyFs6LzRyhDrOa9tVOdQRFiNycMBIGB3Jj5DUYyvppJi+ X-Bogosity: Ham, tests=bogofilter, spamicity=0.000000, version=1.2.4 Sender: owner-linux-mm@kvack.org Precedence: bulk X-Loop: owner-majordomo@kvack.org List-ID: List-Subscribe: List-Unsubscribe: Signed-off-by: Baolin Wang --- include/linux/huge_mm.h | 2 ++ mm/huge_memory.c | 4 ++++ mm/shmem.c | 5 ++++- 3 files changed, 10 insertions(+), 1 deletion(-) diff --git a/include/linux/huge_mm.h b/include/linux/huge_mm.h index 26b6fa98d8ac..67b9c1acad31 100644 --- a/include/linux/huge_mm.h +++ b/include/linux/huge_mm.h @@ -270,6 +270,8 @@ enum mthp_stat_item { MTHP_STAT_ANON_SWPOUT, MTHP_STAT_ANON_SWPOUT_FALLBACK, MTHP_STAT_ANON_SWPIN_REFAULT, + MTHP_STAT_SHMEM_ANON_ALLOC, + MTHP_STAT_SHMEM_ANON_ALLOC_FALLBACK, __MTHP_STAT_COUNT }; diff --git a/mm/huge_memory.c b/mm/huge_memory.c index 9e52c0db7580..dc15240c1ab3 100644 --- a/mm/huge_memory.c +++ b/mm/huge_memory.c @@ -557,6 +557,8 @@ DEFINE_MTHP_STAT_ATTR(anon_alloc_fallback, MTHP_STAT_ANON_ALLOC_FALLBACK); DEFINE_MTHP_STAT_ATTR(anon_swpout, MTHP_STAT_ANON_SWPOUT); DEFINE_MTHP_STAT_ATTR(anon_swpout_fallback, MTHP_STAT_ANON_SWPOUT_FALLBACK); DEFINE_MTHP_STAT_ATTR(anon_swpin_refault, MTHP_STAT_ANON_SWPIN_REFAULT); +DEFINE_MTHP_STAT_ATTR(shmem_anon_alloc, MTHP_STAT_SHMEM_ANON_ALLOC); +DEFINE_MTHP_STAT_ATTR(shmem_anon_alloc_fallback, MTHP_STAT_SHMEM_ANON_ALLOC_FALLBACK); static struct attribute *stats_attrs[] = { &anon_alloc_attr.attr, @@ -564,6 +566,8 @@ static struct attribute *stats_attrs[] = { &anon_swpout_attr.attr, &anon_swpout_fallback_attr.attr, &anon_swpin_refault_attr.attr, + &shmem_anon_alloc_attr.attr, + &shmem_anon_alloc_fallback_attr.attr, NULL, }; diff --git a/mm/shmem.c b/mm/shmem.c index 8b009e7040b2..4a0aa75ab29c 100644 --- a/mm/shmem.c +++ b/mm/shmem.c @@ -1706,11 +1706,14 @@ static struct folio *shmem_alloc_and_add_folio(struct vm_fault *vmf, pages = 1 << order; index = round_down(index, pages); folio = shmem_alloc_hugefolio(gfp, info, index, order); - if (folio) + if (folio) { + count_mthp_stat(order, MTHP_STAT_SHMEM_ANON_ALLOC); goto allocated; + } if (pages == HPAGE_PMD_NR) count_vm_event(THP_FILE_FALLBACK); + count_mthp_stat(order, MTHP_STAT_SHMEM_ANON_ALLOC_FALLBACK); order = next_order(&orders, order); } } else {