From patchwork Tue Sep 19 13:55:54 2023 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Daniel Gomez X-Patchwork-Id: 13391495 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from kanga.kvack.org (kanga.kvack.org [205.233.56.17]) by smtp.lore.kernel.org (Postfix) with ESMTP id E7CAACD5BC2 for ; Tue, 19 Sep 2023 13:56:10 +0000 (UTC) Received: by kanga.kvack.org (Postfix) id 2A84E6B0533; Tue, 19 Sep 2023 09:56:02 -0400 (EDT) Received: by kanga.kvack.org (Postfix, from userid 40) id 230576B0534; Tue, 19 Sep 2023 09:56:02 -0400 (EDT) X-Delivered-To: int-list-linux-mm@kvack.org Received: by kanga.kvack.org (Postfix, from userid 63042) id 05AAB6B0535; Tue, 19 Sep 2023 09:56:01 -0400 (EDT) X-Delivered-To: linux-mm@kvack.org Received: from relay.hostedemail.com (smtprelay0014.hostedemail.com [216.40.44.14]) by kanga.kvack.org (Postfix) with ESMTP id E3BC36B0533 for ; Tue, 19 Sep 2023 09:56:01 -0400 (EDT) Received: from smtpin23.hostedemail.com (a10.router.float.18 [10.200.18.1]) by unirelay06.hostedemail.com (Postfix) with ESMTP id ADA43B3AB4 for ; Tue, 19 Sep 2023 13:56:01 +0000 (UTC) X-FDA: 81253495722.23.224B11F Received: from mailout1.w1.samsung.com (mailout1.w1.samsung.com [210.118.77.11]) by imf09.hostedemail.com (Postfix) with ESMTP id B66EE14000A for ; Tue, 19 Sep 2023 13:55:58 +0000 (UTC) Authentication-Results: imf09.hostedemail.com; dkim=pass header.d=samsung.com header.s=mail20170921 header.b=nuctfpPH; spf=pass (imf09.hostedemail.com: domain of da.gomez@samsung.com designates 210.118.77.11 as permitted sender) smtp.mailfrom=da.gomez@samsung.com; dmarc=pass (policy=none) header.from=samsung.com ARC-Seal: i=1; s=arc-20220608; d=hostedemail.com; t=1695131759; a=rsa-sha256; cv=none; b=BPk8SictT8+xpcyNyh90qdV7LZGH5XhVjZnYegGwXY9RmYkAFQSKR8fPYsknedA1x7yMAZ /jAKkNzGau4stFxzFjPhOEilHZHlVZrnCeUWhw/hFHWV9VKU5m58AZLpp38llBn/UPvvF9 ahuCcecZypgdlFpO4bDT9ipQyHZ9hbI= ARC-Authentication-Results: i=1; imf09.hostedemail.com; dkim=pass header.d=samsung.com header.s=mail20170921 header.b=nuctfpPH; spf=pass (imf09.hostedemail.com: domain of da.gomez@samsung.com designates 210.118.77.11 as permitted sender) smtp.mailfrom=da.gomez@samsung.com; dmarc=pass (policy=none) header.from=samsung.com ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=hostedemail.com; s=arc-20220608; t=1695131759; h=from:from:sender:reply-to:subject:subject:date:date: message-id:message-id:to:to:cc:cc:mime-version:mime-version: content-type:content-type: content-transfer-encoding:content-transfer-encoding: in-reply-to:in-reply-to:references:references:dkim-signature; bh=JKaS2+/EEMvNdZiV+zve2PZ9hmK4b8LE7tpeT3AMyFo=; b=aa9QjBzNRPi5sToxd2z3kESw0IIJCoMMROwHjuP/FRq6QqXoSX+xrmumbiXO0OAp58Jkrz bfkFef1LFLU0bduaD4k1FaxC6eRwZUhGhIszlxHWrW9DqzDC1+iYOq6VnWuh5e+ViZZWOM Bc+BYvPLyXzqnph0o7xZh8FhygLHeN8= Received: from eucas1p2.samsung.com (unknown [182.198.249.207]) by mailout1.w1.samsung.com (KnoxPortal) with ESMTP id 20230919135557euoutp01ae26dc16ee624b8a46ba35fa13cfcaf8~GUVoYXb-c0199901999euoutp015 for ; Tue, 19 Sep 2023 13:55:57 +0000 (GMT) DKIM-Filter: OpenDKIM Filter v2.11.0 mailout1.w1.samsung.com 20230919135557euoutp01ae26dc16ee624b8a46ba35fa13cfcaf8~GUVoYXb-c0199901999euoutp015 DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=samsung.com; s=mail20170921; t=1695131757; bh=JKaS2+/EEMvNdZiV+zve2PZ9hmK4b8LE7tpeT3AMyFo=; h=From:To:CC:Subject:Date:In-Reply-To:References:From; b=nuctfpPHnSMC7WQhH3HQtlmt0BgRwRmZt8mRpYkEnpagsqTg5Xc3QzV+nXnDwFb6D lxURGbKYETqe7oCM8Mgw+BtSPsANdW2I0sC1Kk9USuK7719XgBh986WUD/UyXmzVnP 2kwtvOkEHXQIb3zoZiL0vNOSS/gXZ178cpLtEY6M= Received: from eusmges1new.samsung.com (unknown [203.254.199.242]) by eucas1p2.samsung.com (KnoxPortal) with ESMTP id 20230919135556eucas1p25528f9f56de359f45b1e7b88035e664a~GUVn702Uc1835018350eucas1p2T; Tue, 19 Sep 2023 13:55:56 +0000 (GMT) Received: from eucas1p2.samsung.com ( [182.198.249.207]) by eusmges1new.samsung.com (EUCPMTA) with SMTP id FE.D1.42423.C68A9056; Tue, 19 Sep 2023 14:55:56 +0100 (BST) Received: from eusmtrp2.samsung.com (unknown [182.198.249.139]) by eucas1p1.samsung.com (KnoxPortal) with ESMTPA id 20230919135556eucas1p19920c52d4af0809499eac6bbf4466117~GUVnbnZpF0405804058eucas1p16; Tue, 19 Sep 2023 13:55:56 +0000 (GMT) Received: from eusmgms2.samsung.com (unknown [182.198.249.180]) by eusmtrp2.samsung.com (KnoxPortal) with ESMTP id 20230919135556eusmtrp27b41cacf6044fd9f4708968fe6ddd715~GUVna-NnB2324623246eusmtrp2O; Tue, 19 Sep 2023 13:55:56 +0000 (GMT) X-AuditID: cbfec7f2-a51ff7000002a5b7-8a-6509a86cdb59 Received: from eusmtip1.samsung.com ( [203.254.199.221]) by eusmgms2.samsung.com (EUCPMTA) with SMTP id 38.55.14344.B68A9056; Tue, 19 Sep 2023 14:55:55 +0100 (BST) Received: from CAMSVWEXC01.scsc.local (unknown [106.1.227.71]) by eusmtip1.samsung.com (KnoxPortal) with ESMTPA id 20230919135555eusmtip1dc621754b1f2f7018cbadae30e65482e~GUVnMaPw61888818888eusmtip14; Tue, 19 Sep 2023 13:55:55 +0000 (GMT) Received: from CAMSVWEXC02.scsc.local (2002:6a01:e348::6a01:e348) by CAMSVWEXC01.scsc.local (2002:6a01:e347::6a01:e347) with Microsoft SMTP Server (TLS) id 15.0.1497.2; Tue, 19 Sep 2023 14:55:55 +0100 Received: from CAMSVWEXC02.scsc.local ([::1]) by CAMSVWEXC02.scsc.local ([fe80::3c08:6c51:fa0a:6384%13]) with mapi id 15.00.1497.012; Tue, 19 Sep 2023 14:55:55 +0100 From: Daniel Gomez To: "minchan@kernel.org" , "senozhatsky@chromium.org" , "axboe@kernel.dk" , "djwong@kernel.org" , "willy@infradead.org" , "hughd@google.com" , "akpm@linux-foundation.org" , "mcgrof@kernel.org" , "linux-kernel@vger.kernel.org" , "linux-block@vger.kernel.org" , "linux-xfs@vger.kernel.org" , "linux-fsdevel@vger.kernel.org" , "linux-mm@kvack.org" CC: "gost.dev@samsung.com" , Pankaj Raghav , Daniel Gomez Subject: [PATCH v2 6/6] shmem: add large folios support to the write path Thread-Topic: [PATCH v2 6/6] shmem: add large folios support to the write path Thread-Index: AQHZ6wEBFE+DTgi9xUu5khR159FX3A== Date: Tue, 19 Sep 2023 13:55:54 +0000 Message-ID: <20230919135536.2165715-7-da.gomez@samsung.com> In-Reply-To: <20230919135536.2165715-1-da.gomez@samsung.com> Accept-Language: en-US, en-GB Content-Language: en-US X-MS-Has-Attach: X-MS-TNEF-Correlator: x-ms-exchange-messagesentrepresentingtype: 1 x-ms-exchange-transport-fromentityheader: Hosted x-originating-ip: [106.110.32.103] MIME-Version: 1.0 X-Brightmail-Tracker: H4sIAAAAAAAAA+NgFjrNKsWRmVeSWpSXmKPExsWy7djP87o5KzhTDfZtVLSYs34Nm8Xqu/1s Fpef8Fk8/dTHYrH3lrbFnr0nWSwu75rDZnFvzX9Wi11/drBb3JjwlNFi2df37Ba7Ny5is/j9 Yw6bA6/H7IaLLB4LNpV6bF6h5XH5bKnHplWdbB6bPk1i9zgx4zeLx+dNcgEcUVw2Kak5mWWp Rfp2CVwZk1c8Yyx4pVxx5/YElgbG2zJdjJwcEgImEk8ebmACsYUEVjBKtNzJ6GLkArK/MEr8 vbWdHcL5zChx5t17dpiOgye+MUEkljNKzLv4nAWuauGkL8wQzhlGic4X06GclYwS1xf9Aetn E9CU2HdyE9hgEYHZrBKHF3cwgiSYBeok1jybxQJiCwt4Sqw/ehWsQUQgQKKt/woThK0n8XfZ STCbRUBV4t2/hWC9vALWEj8nHgKLcwrYSNyY28QGYjMKyEo8WvmLHWK+uMStJ/OZIJ4QlFg0 ew8zhC0m8W/XQzYIW0fi7PUnjBC2gcTWpftYIGwliT8dC6Hu1JO4MXUKG4StLbFs4WtmiBsE JU7OfAIOCwmBf5wS02auhxrkItE79yHUMmGJV8e3QENSRuL/zvlMExi1ZyG5bxaSHbOQ7JiF ZMcCRpZVjOKppcW56anFhnmp5XrFibnFpXnpesn5uZsYgent9L/jn3Ywzn31Ue8QIxMH4yFG CQ5mJRHemYZsqUK8KYmVValF+fFFpTmpxYcYpTlYlMR5tW1PJgsJpCeWpGanphakFsFkmTg4 pRqYRBJnbv1x2ka78IHfHt81nFbfLUzndT1+Z6woZfPpbRLDtLuPOrPuZLEaKd3a1xn7YNqd L9ErLp0Us3GYUbVT7tk1b9WvNzmXcDI/jOvzLJT+dVy+Nm4O/1qXacdW5RgEfOXJXGy18Qer 70UV/QbRCMVCpgKPRiHR1ITMs1fZJvsdfTTFYm2JfU/a3xJdu/yasPn/559hWxm5Z44y82rL p++ktLrKnaR1losyc6r2n/i/2eqgWGrpNCufqpdack35shEx92JeCOduaNzscXPdpLY1N0ob 79lcXZthatOwoL70oJBdo5DO5amrK9ZN23nsmM4vw8yORUUpuRLN8XNXZlQ3r73lb9N5uCY1 yV6JpTgj0VCLuag4EQB80Q5y3gMAAA== X-Brightmail-Tracker: H4sIAAAAAAAAA+NgFjrOKsWRmVeSWpSXmKPExsVy+t/xu7rZKzhTDfZusbGYs34Nm8Xqu/1s Fpef8Fk8/dTHYrH3lrbFnr0nWSwu75rDZnFvzX9Wi11/drBb3JjwlNFi2df37Ba7Ny5is/j9 Yw6bA6/H7IaLLB4LNpV6bF6h5XH5bKnHplWdbB6bPk1i9zgx4zeLx+dNcgEcUXo2RfmlJakK GfnFJbZK0YYWRnqGlhZ6RiaWeobG5rFWRqZK+nY2Kak5mWWpRfp2CXoZk1c8Yyx4pVxx5/YE lgbG2zJdjJwcEgImEgdPfGPqYuTiEBJYyijx8/AHFoiEjMTGL1dZIWxhiT/Xutggij4yShw4 f4cFwjnDKHFj9X12CGclo8TZvVvAWtgENCX2ndwElhARmM0qcXhxByNIglmgTmLNs1lgO4QF PCXWH73KDmKLCPhJ/Hx0kxXC1pP4u+wkE4jNIqAq8e7fQrBeXgFriZ8TD4HFhYDsVwtXg9Vz CthI3JjbxAZiMwrISjxa+YsdYpe4xK0n85kgfhCQWLLnPDOELSrx8vE/qN90JM5ef8IIYRtI bF26D+p/JYk/HQuhbtaTuDF1ChuErS2xbOFrZoh7BCVOznzCMoFRehaSdbOQtMxC0jILScsC RpZVjCKppcW56bnFRnrFibnFpXnpesn5uZsYgSlq27GfW3Ywrnz1Ue8QIxMH4yFGCQ5mJRHe mYZsqUK8KYmVValF+fFFpTmpxYcYTYFhNJFZSjQ5H5gk80riDc0MTA1NzCwNTC3NjJXEeT0L OhKFBNITS1KzU1MLUotg+pg4OKUamKY0sD3SbStZePl74E6P/u7P+56FFDDFCxd47pzP/ShM Z+nfY7pee44yWPTXM79S5dqiW/k/zsxoj45P6Nvlro1b6+R/bJghk/RNzEM29q31l5kc+X+U XZMn/mv72vVL6b2ziQKfqd9uXdHnexZ0GVcc36+56+l8lZ6D+ikM+5WDlPgWWL1alVV+mCPe 5YLJqw33uqyLNBS0vqxZ2aR0sJWz9f0XPU/H/fJCU/UXRPjWtVWF7Vi19c0nRg/9T6kZF38H G97/emt20tnNG+rNZt50C9rokSYts90pS0/c/k1mUL/OYbOVF7TnKPZt1fyVZj9vysTvWzIf XtxfUfbmaKl8c/Dygk2Gm/WyowyvKLEUZyQaajEXFScCAEql5LjaAwAA X-CMS-MailID: 20230919135556eucas1p19920c52d4af0809499eac6bbf4466117 X-Msg-Generator: CA X-RootMTR: 20230919135556eucas1p19920c52d4af0809499eac6bbf4466117 X-EPHeader: CA CMS-TYPE: 201P X-CMS-RootMailID: 20230919135556eucas1p19920c52d4af0809499eac6bbf4466117 References: <20230919135536.2165715-1-da.gomez@samsung.com> X-Rspamd-Server: rspam08 X-Rspamd-Queue-Id: B66EE14000A X-Stat-Signature: j57eqghd5a753nyzqe66tzcif7e5srhr X-Rspam-User: X-HE-Tag: 1695131758-778579 X-HE-Meta: U2FsdGVkX18NjqfxKMb/F1nekzZlcXbSH8qLfYkbRxGvcUrt6Fjr092W4AmV2E8KGJkBNv2PBKaAF1x77Uu1ecgWqOCis9yheeY4PzUoA7cP9GU1mal5pfggMBMe1akkBg740xTdMAiBeaFcbAY82RD5zOsURuVWQc3Hj/tDwzHz/HfvVVTfwldvNZa1gUSEy44NSsAsTvg5Vwc7/2YD80Sdg/s66zSbp1CP0vOqSQ2CmxWwtAAMxvdrTG6DPZjZ0gNI6W7jakSj0XZJyzxL2KET2Yr158sphymNCVWVlqHaT42K3UWXBVi6AJ+Npv+CvcnJssvRmWBooHBGOm6+TwKQzB8euS71EPQTPsrWtPXqLlZ03kHZk7h/BTBlSZwenh8zo/JS0zaEDDLoknsaTLmJEtc7wKUV99+mK1mpUX0WXSOtsSZ3cf5KyXL5YooYsmK76/1XRBGvahGGwyzCIZh48tkuwy0k27Z5U8vEDwR9SQN5rIMx+/WFO8AlpvX0FT4vyKyt7UCWFx5+ZrjVPFMKGEzpK+5lvy4iSOPj4Hd2Rx1xw7NVY7DFZI8F5BOJlW+jEo6bY1ISv8ruQ+xlhn3JgEpthx4VkWBIcNWcRw8Hom3Qwxj5dHXxob6nnUwFI0d0myjinzb5RIVAKU1u1uH5vBpt2umlPfYK8w+fl7tTFPkTs65ZJJwe7xhfmOrl8AegF1a6r0ITKge0hDw0HSu8gPB/RX7cPRR+XU0LEBMQmeDvPTKuArD3BXyC0IlM5Q6peeRHYptIr3mEJFuJjwQ7Y9+Z7XU8scxQ3DKk5fUJEXIZd7bTxLy8lFQ+k09Shyq7N7+uLws6yfTU8TMvXZ6SB90840baF2ctCICI4FUk0V5pDjwSfIKpKEWJy9/G2ZKN2wC+rMSiVYDXzVioliuhIl97pyudNCQJsenkXyW1e4lnI82E1tkSAZZXtLsVWHBiE8MrCf3BMjXtA+5 Yb+5/QdO FANWXK0DbIWOCdNI86zilT8DGWdwu6D9TFrdLw+q+cGnO+xJ101u4XD69NhlfP3Hg7BCtNpBikffS6iQqnJVgTlPxab2nd/H0Md+RLvUf072PBdYwV8+zapcy2WxbGYCCM2aBjB5A6RqMIWnRaDXIO/SsIrhhRCj9pDkG X-Bogosity: Ham, tests=bogofilter, spamicity=0.000000, version=1.2.4 Sender: owner-linux-mm@kvack.org Precedence: bulk X-Loop: owner-majordomo@kvack.org List-ID: Add large folio support for shmem write path matching the same high order preference mechanism used for iomap buffered IO path as used in __filemap_get_folio() with a difference on the max order permitted (being PMD_ORDER-1) to respect the huge mount option when large folio is supported. Use the __folio_get_max_order to get a hint for the order of the folio based on file size which takes care of the mapping requirements. Swap does not support high order folios for now, so make it order 0 in case swap is enabled. Signed-off-by: Daniel Gomez --- mm/shmem.c | 66 ++++++++++++++++++++++++++++++++++++++++++------------ 1 file changed, 52 insertions(+), 14 deletions(-) diff --git a/mm/shmem.c b/mm/shmem.c index 38aafa0b0845..96c74c96c0d9 100644 --- a/mm/shmem.c +++ b/mm/shmem.c @@ -95,6 +95,9 @@ static struct vfsmount *shm_mnt; /* Symlink up to this size is kmalloc'ed instead of using a swappable page */ #define SHORT_SYMLINK_LEN 128 +/* Like MAX_PAGECACHE_ORDER but respecting huge option */ +#define MAX_SHMEM_ORDER HPAGE_PMD_ORDER - 1 + /* * shmem_fallocate communicates with shmem_fault or shmem_writepage via * inode->i_private (with i_rwsem making sure that it has only one user at @@ -1680,26 +1683,58 @@ static struct folio *shmem_alloc_folio(gfp_t gfp, return folio; } +/** + * shmem_mapping_size_order - Get maximum folio order for the given file size. + * @mapping: Target address_space. + * @index: The page index. + * @size: The suggested size of the folio to create. + * + * This returns a high order for folios (when supported) based on the file size + * which the mapping currently allows at the given index. The index is relevant + * due to alignment considerations the mapping might have. The returned order + * may be less than the size passed. + * + * Like __filemap_get_folio order calculation. + * + * Return: The order. + */ +static inline unsigned int +shmem_mapping_size_order(struct address_space *mapping, pgoff_t index, + size_t size, struct shmem_sb_info *sbinfo) +{ + unsigned int order = ilog2(size); + + if ((order <= PAGE_SHIFT) || + (!mapping_large_folio_support(mapping) || !sbinfo->noswap)) + return 0; + else + order = order - PAGE_SHIFT; + + /* If we're not aligned, allocate a smaller folio */ + if (index & ((1UL << order) - 1)) + order = __ffs(index); + + order = min_t(size_t, order, MAX_SHMEM_ORDER); + + /* Order-1 not supported due to THP dependency */ + return (order == 1) ? 0 : order; +} + static struct folio *shmem_alloc_and_acct_folio(gfp_t gfp, struct inode *inode, - pgoff_t index, bool huge, unsigned int *order) + pgoff_t index, unsigned int order) { struct shmem_inode_info *info = SHMEM_I(inode); struct folio *folio; - int nr; - int err; - - if (!IS_ENABLED(CONFIG_TRANSPARENT_HUGEPAGE)) - huge = false; - nr = huge ? HPAGE_PMD_NR : 1U << *order; + int nr = 1U << order; + int err = shmem_inode_acct_block(inode, nr); - err = shmem_inode_acct_block(inode, nr); if (err) goto failed; - if (huge) + if (order == HPAGE_PMD_ORDER) folio = shmem_alloc_hugefolio(gfp, info, index); else - folio = shmem_alloc_folio(gfp, info, index, *order); + folio = shmem_alloc_folio(gfp, info, index, order); if (folio) { __folio_set_locked(folio); __folio_set_swapbacked(folio); @@ -2030,18 +2065,19 @@ static int shmem_get_folio_gfp(struct inode *inode, pgoff_t index, return 0; } + order = shmem_mapping_size_order(inode->i_mapping, index, len, sbinfo); + if (!shmem_is_huge(inode, index, false, vma ? vma->vm_mm : NULL, vma ? vma->vm_flags : 0)) goto alloc_nohuge; huge_gfp = vma_thp_gfp_mask(vma); huge_gfp = limit_gfp_mask(huge_gfp, gfp); - folio = shmem_alloc_and_acct_folio(huge_gfp, inode, index, true, - &order); + folio = shmem_alloc_and_acct_folio(huge_gfp, inode, index, + HPAGE_PMD_ORDER); if (IS_ERR(folio)) { alloc_nohuge: - folio = shmem_alloc_and_acct_folio(gfp, inode, index, false, - &order); + folio = shmem_alloc_and_acct_folio(gfp, inode, index, order); } if (IS_ERR(folio)) { int retry = 5; @@ -2145,6 +2181,8 @@ static int shmem_get_folio_gfp(struct inode *inode, pgoff_t index, if (folio_test_large(folio)) { folio_unlock(folio); folio_put(folio); + if (--order == 1) + order = 0; goto alloc_nohuge; } unlock: