From patchwork Fri Jun 28 08:44:09 2024 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Chenliang Li X-Patchwork-Id: 13716838 Received: from mailout2.samsung.com (mailout2.samsung.com [203.254.224.25]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (No client certificate requested) by smtp.subspace.kernel.org (Postfix) with ESMTPS id 692CA14A85 for ; Sat, 29 Jun 2024 07:26:29 +0000 (UTC) Authentication-Results: smtp.subspace.kernel.org; arc=none smtp.client-ip=203.254.224.25 ARC-Seal: i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1719645992; cv=none; b=thoUK+R0Fxi0doY3LEvRcPPfzDgiWzQVCFAAQzLpPLUd4m2AIiVt/YPdTByQ0GpunASd/n1BGVpeXsU5waEbKnOhcEtMjtLaDdnk5Y/AVj1W/4yPii2QEv86ih0tGQ5hkwEgPqpXuGDkTYV872GN/i30USDKk8Za2aD3npcncc8= ARC-Message-Signature: i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1719645992; c=relaxed/simple; bh=dGg0F+GmbdFkSnWgqSGvP96ZO2YYn8/LGV24URAlSIM=; h=From:To:Cc:Subject:Date:Message-Id:In-Reply-To:MIME-Version: Content-Type:References; b=FIbh85Agx2DfsC9rH/JK2mM8zmMjLSqUh1hsn0Ji90jR7Oww+Mfgc2JhUglzW+jd2rWquFanCuIfuEFuI71o/Vqr7/6bzJG7GVZTNa8Q42Y1+EvqZhi2fkIixo/KwB0wZe6M+nUN53A5sgH7EKnpeTP7LcBiomkdqe+vRCNQOo0= ARC-Authentication-Results: i=1; smtp.subspace.kernel.org; dmarc=pass (p=none dis=none) header.from=samsung.com; spf=pass smtp.mailfrom=samsung.com; dkim=pass (1024-bit key) header.d=samsung.com header.i=@samsung.com header.b=cKAPOgaJ; arc=none smtp.client-ip=203.254.224.25 Authentication-Results: smtp.subspace.kernel.org; dmarc=pass (p=none dis=none) header.from=samsung.com Authentication-Results: smtp.subspace.kernel.org; spf=pass smtp.mailfrom=samsung.com Authentication-Results: smtp.subspace.kernel.org; dkim=pass (1024-bit key) header.d=samsung.com header.i=@samsung.com header.b="cKAPOgaJ" Received: from epcas5p2.samsung.com (unknown [182.195.41.40]) by mailout2.samsung.com (KnoxPortal) with ESMTP id 20240629072622epoutp02fc43712273710bc7a26349500d401906~daOjn0aPE0164801648epoutp02u for ; Sat, 29 Jun 2024 07:26:22 +0000 (GMT) DKIM-Filter: OpenDKIM Filter v2.11.0 mailout2.samsung.com 20240629072622epoutp02fc43712273710bc7a26349500d401906~daOjn0aPE0164801648epoutp02u DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=samsung.com; s=mail20170921; t=1719645982; bh=H44s/aUHCxaZote+pk8rKt/p0gDo70B4ZL3ZIS2Q3E0=; h=From:To:Cc:Subject:Date:In-Reply-To:References:From; b=cKAPOgaJI4lrtp4pGp98Wp1Q+u3ejMUPap9Ylk6FOFId0ZLrT5tZGpWnG/AOG80e8 8eSpbSItthYo7BpXtbgLs3+P84VPllgVnflleXF0z/7JjsPe0RF/HE/N0c+m15LxJc 5uY94WnfErHK7xf+hu/Hpd+SHooGk+g+MBArgv7k= Received: from epsnrtp1.localdomain (unknown [182.195.42.162]) by epcas5p2.samsung.com (KnoxPortal) with ESMTP id 20240629072620epcas5p22928b65361791025e64baf8ffc7cc2f5~daOiZN5m_0452704527epcas5p2N; Sat, 29 Jun 2024 07:26:20 +0000 (GMT) Received: from epsmgec5p1new.samsung.com (unknown [182.195.38.175]) by epsnrtp1.localdomain (Postfix) with ESMTP id 4WB3kz5HC3z4x9Pr; Sat, 29 Jun 2024 07:26:19 +0000 (GMT) Received: from epcas5p1.samsung.com ( [182.195.41.39]) by epsmgec5p1new.samsung.com (Symantec Messaging Gateway) with SMTP id 3D.48.07307.B17BF766; Sat, 29 Jun 2024 16:26:19 +0900 (KST) Received: from epsmtrp2.samsung.com (unknown [182.195.40.14]) by epcas5p3.samsung.com (KnoxPortal) with ESMTPA id 20240628084420epcas5p32f49e7c977695d20bcef7734eb2e38b4~dHpWLh3VQ1548615486epcas5p3v; Fri, 28 Jun 2024 08:44:20 +0000 (GMT) Received: from epsmgms1p1new.samsung.com (unknown [182.195.42.41]) by epsmtrp2.samsung.com (KnoxPortal) with ESMTP id 20240628084420epsmtrp2e4e1c20d50a418c5e2c5f03bfb02c143~dHpWKvrZn0935009350epsmtrp2c; Fri, 28 Jun 2024 08:44:20 +0000 (GMT) X-AuditID: b6c32a44-3f1fa70000011c8b-f9-667fb71babd2 Received: from epsmtip1.samsung.com ( [182.195.34.30]) by epsmgms1p1new.samsung.com (Symantec Messaging Gateway) with SMTP id 4F.04.29940.4E77E766; Fri, 28 Jun 2024 17:44:20 +0900 (KST) Received: from lcl-Standard-PC-i440FX-PIIX-1996.. (unknown [109.105.118.124]) by epsmtip1.samsung.com (KnoxPortal) with ESMTPA id 20240628084419epsmtip1090c41b2907317a7d61a46f553a83772~dHpU6dPxu0054500545epsmtip1u; Fri, 28 Jun 2024 08:44:18 +0000 (GMT) From: Chenliang Li To: axboe@kernel.dk, asml.silence@gmail.com Cc: io-uring@vger.kernel.org, peiwei.li@samsung.com, joshi.k@samsung.com, kundan.kumar@samsung.com, anuj20.g@samsung.com, gost.dev@samsung.com, Chenliang Li Subject: [PATCH v5 1/3] io_uring/rsrc: add hugepage fixed buffer coalesce helpers Date: Fri, 28 Jun 2024 16:44:09 +0800 Message-Id: <20240628084411.2371-2-cliang01.li@samsung.com> X-Mailer: git-send-email 2.34.1 In-Reply-To: <20240628084411.2371-1-cliang01.li@samsung.com> Precedence: bulk X-Mailing-List: io-uring@vger.kernel.org List-Id: List-Subscribe: List-Unsubscribe: MIME-Version: 1.0 X-Brightmail-Tracker: H4sIAAAAAAAAA+NgFprFJsWRmVeSWpSXmKPExsWy7bCmuq709vo0g23TVSyaJvxltpizahuj xeq7/WwWp/8+ZrG4eWAnk8W71nMsFkf/v2Wz+NV9l9Fi65evrBbP9nJanJ3wgdWB22PnrLvs HpfPlnr0bVnF6PF5k1wAS1S2TUZqYkpqkUJqXnJ+SmZeuq2Sd3C8c7ypmYGhrqGlhbmSQl5i bqqtkotPgK5bZg7QTUoKZYk5pUChgMTiYiV9O5ui/NKSVIWM/OISW6XUgpScApMCveLE3OLS vHS9vNQSK0MDAyNToMKE7Iz762oLZipVfLlR1MB4X7qLkZNDQsBEYt3/7+xdjFwcQgK7GSWe 3nrJDOF8YpT4vqkJyvnGKLHv6l82mJbFx6YwQST2MkpM3dcKVdXEJPHg+kEmkCo2AR2J3yt+ sYDYIgLaEq8fTwWzmQV2MUosPCcFYgsLBEusXrYPqJmDg0VAVeL1XjWQMK+AtcSZ9x8YIZbJ S+w/eJYZxOYUsJG4NH0WO0SNoMTJmU+gRspLNG+dDXaDhMBfdonnWy5ANbtIrFp3jQXCFpZ4 dXwLO4QtJfH53V42kL0SAsUSy9bJQfS2MEq8fzcHqtda4t+VPSwgNcwCmhLrd+lDhGUlpp5a xwSxl0+i9/cTJog4r8SOeTC2qsSFg9ugVklLrJ2wlRlilYfEogNqkKDqZ5S4vfwp6wRGhVlI 3pmF5J1ZCJsXMDKvYpRMLSjOTU9NNi0wzEsth0dxcn7uJkZwItVy2cF4Y/4/vUOMTByMhxgl OJiVRHj5M+vShHhTEiurUovy44tKc1KLDzGaAoN7IrOUaHI+MJXnlcQbmlgamJiZmZlYGpsZ Konzvm6dmyIkkJ5YkpqdmlqQWgTTx8TBKdXAdPOtzdVbcnNdz8UeWtiRKqc0NU6288MCsz6v 0KY9xtE3i6ffOjYx5NMhBbOf963kd+XE9khFGE88HHNq+v3HLpJds+1EJn/a+/novlWyrje+ 59p+bTAqv3H9QrXj/QlPusOa75/789voRVKglmZkaHXHsqKyZs7/LnIVkc3spUH/tdjsKuvq g/IEchfrrhCSP+tRKikR9uRv1FbvI6lJPa9ucew99/+1zaE1P6TZri0TbL9XVMneoFsceGP6 UnmJo5/v2M5kPftfOOHX1ZSDWitPPJPsFhIsNbq3niGntbw7rvCBxIu/THMnnKgys//AWrdl m2XunQDhr1MWas/YJbdp2owlyls1t042X/OjWomlOCPRUIu5qDgRAGxKyeYtBAAA X-Brightmail-Tracker: H4sIAAAAAAAAA+NgFjrPLMWRmVeSWpSXmKPExsWy7bCSnO6T8ro0g6PHeS2aJvxltpizahuj xeq7/WwWp/8+ZrG4eWAnk8W71nMsFkf/v2Wz+NV9l9Fi65evrBbP9nJanJ3wgdWB22PnrLvs HpfPlnr0bVnF6PF5k1wASxSXTUpqTmZZapG+XQJXxv11tQUzlSq+3ChqYLwv3cXIySEhYCKx +NgUpi5GLg4hgd2MElM/3WOCSEhLdBxqZYewhSVW/nsOZgsJNDBJXFoWAmKzCehI/F7xi6WL kYNDREBXovGuAsgcZoFDjBLNG5oZQWqEBQIl5l2awQ5SwyKgKvF6rxpImFfAWuLM+w+MEOPl JfYfPMsMYnMK2Ehcmj4LrFwIqObD3UiIckGJkzOfsIDYzEDlzVtnM09gFJiFJDULSWoBI9Mq RsnUguLc9NxiwwLDvNRyveLE3OLSvHS95PzcTYzgMNfS3MG4fdUHvUOMTByMQCdzMCuJ8PJn 1qUJ8aYkVlalFuXHF5XmpBYfYpTmYFES5xV/0ZsiJJCeWJKanZpakFoEk2Xi4JRqYJpi/fT6 j/Q365f3rItlZlo8eXrFzNKTF/lLU048mmWycc/pTZkRTdYKC6exGqZ/1eVR+yLTf87pu8Pl yZPUDN7d74yvCFcv9F474/3uO4LfMu2u2nrMX5y1O2/GqrCzEvEdX4uLIjS7dHSe/NP9e/Gq UgJjv5LJr1c+6jIqL3fynxN9e6h43q9fn1csYdAoF+8N43ltq3VfWNJZdK6Gi/QSnfBrBXMb v14+E7hD+e6Rzz5fLPgncvHlpqeV8Kw38eLY/nVhrMxfk0WpK57setyUs/mTQerkZWtz88/m 5m9UeBn0x/fd9I1RGZtv+qcn1fSvrCvbH//daenfrlmuc42TZTyqX1w3+xmUHtTlbaHEUpyR aKjFXFScCACgrTN04gIAAA== X-CMS-MailID: 20240628084420epcas5p32f49e7c977695d20bcef7734eb2e38b4 X-Msg-Generator: CA X-Sendblock-Type: REQ_APPROVE CMS-TYPE: 105P DLP-Filter: Pass X-CFilter-Loop: Reflected X-CMS-RootMailID: 20240628084420epcas5p32f49e7c977695d20bcef7734eb2e38b4 References: <20240628084411.2371-1-cliang01.li@samsung.com> Introduce helper functions to check and coalesce hugepage-backed fixed buffers. The coalescing optimizes both time and space consumption caused by mapping and storing multi-hugepage fixed buffers. Currently we only have single-hugepage buffer coalescing, so add support for multi-hugepage fixed buffer coalescing. A coalescable multi-hugepage buffer should fully cover its folios (except potentially the first and last one), and these folios should have the same size. These requirements are for easier processing later, also we need same size'd chunks in io_import_fixed for fast iov_iter adjust. Signed-off-by: Chenliang Li --- io_uring/rsrc.c | 87 +++++++++++++++++++++++++++++++++++++++++++++++++ io_uring/rsrc.h | 9 +++++ 2 files changed, 96 insertions(+) diff --git a/io_uring/rsrc.c b/io_uring/rsrc.c index 60c00144471a..c88ce8c38515 100644 --- a/io_uring/rsrc.c +++ b/io_uring/rsrc.c @@ -849,6 +849,93 @@ static int io_buffer_account_pin(struct io_ring_ctx *ctx, struct page **pages, return ret; } +static bool io_do_coalesce_buffer(struct page ***pages, int *nr_pages, + struct io_imu_folio_data *data, int nr_folios) +{ + struct page **page_array = *pages, **new_array = NULL; + int nr_pages_left = *nr_pages, i, j; + + /* Store head pages only*/ + new_array = kvmalloc_array(nr_folios, sizeof(struct page *), + GFP_KERNEL); + if (!new_array) + return false; + + new_array[0] = page_array[0]; + /* + * The pages are bound to the folio, it doesn't + * actually unpin them but drops all but one reference, + * which is usually put down by io_buffer_unmap(). + * Note, needs a better helper. + */ + if (data->nr_pages_head > 1) + unpin_user_pages(&page_array[1], data->nr_pages_head - 1); + + j = data->nr_pages_head; + nr_pages_left -= data->nr_pages_head; + for (i = 1; i < nr_folios; i++) { + unsigned int nr_unpin; + + new_array[i] = page_array[j]; + nr_unpin = min_t(unsigned int, nr_pages_left - 1, + data->nr_pages_mid - 1); + if (nr_unpin) + unpin_user_pages(&page_array[j+1], nr_unpin); + j += data->nr_pages_mid; + nr_pages_left -= data->nr_pages_mid; + } + kvfree(page_array); + *pages = new_array; + *nr_pages = nr_folios; + return true; +} + +static bool io_try_coalesce_buffer(struct page ***pages, int *nr_pages, + struct io_imu_folio_data *data) +{ + struct page **page_array = *pages; + struct folio *folio = page_folio(page_array[0]); + unsigned int count = 1, nr_folios = 1; + int i; + + if (*nr_pages <= 1) + return false; + + data->nr_pages_mid = folio_nr_pages(folio); + if (data->nr_pages_mid == 1) + return false; + + data->folio_shift = folio_shift(folio); + data->folio_size = folio_size(folio); + /* + * Check if pages are contiguous inside a folio, and all folios have + * the same page count except for the head and tail. + */ + for (i = 1; i < *nr_pages; i++) { + if (page_folio(page_array[i]) == folio && + page_array[i] == page_array[i-1] + 1) { + count++; + continue; + } + + if (nr_folios == 1) + data->nr_pages_head = count; + else if (count != data->nr_pages_mid) + return false; + + folio = page_folio(page_array[i]); + if (folio_size(folio) != data->folio_size) + return false; + + count = 1; + nr_folios++; + } + if (nr_folios == 1) + data->nr_pages_head = count; + + return io_do_coalesce_buffer(pages, nr_pages, data, nr_folios); +} + static int io_sqe_buffer_register(struct io_ring_ctx *ctx, struct iovec *iov, struct io_mapped_ubuf **pimu, struct page **last_hpage) diff --git a/io_uring/rsrc.h b/io_uring/rsrc.h index c032ca3436ca..cc66323535f6 100644 --- a/io_uring/rsrc.h +++ b/io_uring/rsrc.h @@ -50,6 +50,15 @@ struct io_mapped_ubuf { struct bio_vec bvec[] __counted_by(nr_bvecs); }; +struct io_imu_folio_data { + /* Head folio can be partially included in the fixed buf */ + unsigned int nr_pages_head; + /* For non-head/tail folios, has to be fully included */ + unsigned int nr_pages_mid; + unsigned int folio_shift; + size_t folio_size; +}; + void io_rsrc_node_ref_zero(struct io_rsrc_node *node); void io_rsrc_node_destroy(struct io_ring_ctx *ctx, struct io_rsrc_node *ref_node); struct io_rsrc_node *io_rsrc_node_alloc(struct io_ring_ctx *ctx); From patchwork Fri Jun 28 08:44:10 2024 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Chenliang Li X-Patchwork-Id: 13716837 Received: from mailout4.samsung.com (mailout4.samsung.com [203.254.224.34]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (No client certificate requested) by smtp.subspace.kernel.org (Postfix) with ESMTPS id 10E4D14A85 for ; Sat, 29 Jun 2024 07:26:25 +0000 (UTC) Authentication-Results: smtp.subspace.kernel.org; arc=none smtp.client-ip=203.254.224.34 ARC-Seal: i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1719645988; cv=none; b=pG3KtZhhcC1plcb3HGXkb0AmPq2rBS6d7Addq8FMyon2oPDGNQJLKLH7hBnN+A3WPw//xgNlVLcbY/bhXr+aMpqb/IVA9X2oHKIZHPTOayQEenxQk4dtX2t1lSGp+LYtnw/knS+ItgzMpRYvG1T2xHOqpra3p7YchmEs87YYQHQ= ARC-Message-Signature: i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1719645988; c=relaxed/simple; bh=0ckTLWkCqkNEr+z4jilUV3nA6GJHPHIrOJVJOMuANPY=; h=From:To:Cc:Subject:Date:Message-Id:In-Reply-To:MIME-Version: Content-Type:References; b=HWK0v5HQ/XemkQaD1bib4YWGUirCGTz760nR+y7+9WDicKd2WJCX0+frcb93Or+mmcwrJrZLjez+fox7by/kMXwa5kuhMkMQSChzN0Il4T7DNzojoFBAXlsm3apqik9SRYjryht2kVHiAA2WikQ2YjLAgWct1TIWcsdEiAAECk8= ARC-Authentication-Results: i=1; smtp.subspace.kernel.org; dmarc=pass (p=none dis=none) header.from=samsung.com; spf=pass smtp.mailfrom=samsung.com; dkim=pass (1024-bit key) header.d=samsung.com header.i=@samsung.com header.b=o94HnSxG; arc=none smtp.client-ip=203.254.224.34 Authentication-Results: smtp.subspace.kernel.org; dmarc=pass (p=none dis=none) header.from=samsung.com Authentication-Results: smtp.subspace.kernel.org; spf=pass smtp.mailfrom=samsung.com Authentication-Results: smtp.subspace.kernel.org; dkim=pass (1024-bit key) header.d=samsung.com header.i=@samsung.com header.b="o94HnSxG" Received: from epcas5p3.samsung.com (unknown [182.195.41.41]) by mailout4.samsung.com (KnoxPortal) with ESMTP id 20240629072624epoutp041ced77381fba8c7a9f94e35fd15b7bbb~daOlSoU2l0908809088epoutp04H for ; Sat, 29 Jun 2024 07:26:24 +0000 (GMT) DKIM-Filter: OpenDKIM Filter v2.11.0 mailout4.samsung.com 20240629072624epoutp041ced77381fba8c7a9f94e35fd15b7bbb~daOlSoU2l0908809088epoutp04H DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=samsung.com; s=mail20170921; t=1719645984; bh=oOYRZfHDM6ImX8DPxmphYo3scgdYkl/adrdNjXjCqB4=; h=From:To:Cc:Subject:Date:In-Reply-To:References:From; b=o94HnSxGJLifudEF+fitfV1P1br25jyjI9vXrWiRplJpAnzTNDO2AKOWDOo3qK56x hSeq6YTFjRi5wVOBgYClQRz3t6L163eF3I0r5xjduGkWx2kxK01u/Q0IY0+esZct2Q 26nStdhuFlB29Od5+N5J0KZTizGr+RHbxT7AHI64= Received: from epsnrtp1.localdomain (unknown [182.195.42.162]) by epcas5p4.samsung.com (KnoxPortal) with ESMTP id 20240629072623epcas5p44911431171797b09da3862ca34f212a8~daOk6JhTE0540905409epcas5p4G; Sat, 29 Jun 2024 07:26:23 +0000 (GMT) Received: from epsmges5p1new.samsung.com (unknown [182.195.38.178]) by epsnrtp1.localdomain (Postfix) with ESMTP id 4WB3l22pSrz4x9Pr; Sat, 29 Jun 2024 07:26:22 +0000 (GMT) Received: from epcas5p3.samsung.com ( [182.195.41.41]) by epsmges5p1new.samsung.com (Symantec Messaging Gateway) with SMTP id E6.26.11095.E17BF766; Sat, 29 Jun 2024 16:26:22 +0900 (KST) Received: from epsmtrp2.samsung.com (unknown [182.195.40.14]) by epcas5p3.samsung.com (KnoxPortal) with ESMTPA id 20240628084422epcas5p3b5d4c93e5fa30069c703bcead1fa0033~dHpX8nE2t2959029590epcas5p3s; Fri, 28 Jun 2024 08:44:22 +0000 (GMT) Received: from epsmgmc1p1new.samsung.com (unknown [182.195.42.40]) by epsmtrp2.samsung.com (KnoxPortal) with ESMTP id 20240628084422epsmtrp2309cefad43065bc8ff113ec24b9a4321~dHpX76ukC0935009350epsmtrp2j; Fri, 28 Jun 2024 08:44:22 +0000 (GMT) X-AuditID: b6c32a49-3c3ff70000012b57-92-667fb71ed5fd Received: from epsmtip1.samsung.com ( [182.195.34.30]) by epsmgmc1p1new.samsung.com (Symantec Messaging Gateway) with SMTP id 3E.3C.07412.6E77E766; Fri, 28 Jun 2024 17:44:22 +0900 (KST) Received: from lcl-Standard-PC-i440FX-PIIX-1996.. (unknown [109.105.118.124]) by epsmtip1.samsung.com (KnoxPortal) with ESMTPA id 20240628084420epsmtip19dd98dd4d415d5da6ef86f2d544e362e~dHpWtb3Y20069900699epsmtip1x; Fri, 28 Jun 2024 08:44:20 +0000 (GMT) From: Chenliang Li To: axboe@kernel.dk, asml.silence@gmail.com Cc: io-uring@vger.kernel.org, peiwei.li@samsung.com, joshi.k@samsung.com, kundan.kumar@samsung.com, anuj20.g@samsung.com, gost.dev@samsung.com, Chenliang Li Subject: [PATCH v5 2/3] io_uring/rsrc: store folio shift and mask into imu Date: Fri, 28 Jun 2024 16:44:10 +0800 Message-Id: <20240628084411.2371-3-cliang01.li@samsung.com> X-Mailer: git-send-email 2.34.1 In-Reply-To: <20240628084411.2371-1-cliang01.li@samsung.com> Precedence: bulk X-Mailing-List: io-uring@vger.kernel.org List-Id: List-Subscribe: List-Unsubscribe: MIME-Version: 1.0 X-Brightmail-Tracker: H4sIAAAAAAAAA+NgFprLJsWRmVeSWpSXmKPExsWy7bCmpq7c9vo0g9+fRSyaJvxltpizahuj xeq7/WwWp/8+ZrG4eWAnk8W71nMsFkf/v2Wz+NV9l9Fi65evrBbP9nJanJ3wgdWB22PnrLvs HpfPlnr0bVnF6PF5k1wAS1S2TUZqYkpqkUJqXnJ+SmZeuq2Sd3C8c7ypmYGhrqGlhbmSQl5i bqqtkotPgK5bZg7QTUoKZYk5pUChgMTiYiV9O5ui/NKSVIWM/OISW6XUgpScApMCveLE3OLS vHS9vNQSK0MDAyNToMKE7IyWLU9ZCppFK6ZOX8XSwDhXoIuRk0NCwETi97UOti5GLg4hgd2M ElN7HjJCOJ8YJSZe/cUC4XxjlHi0uIERpmVFyzwwW0hgL6PE6hX5EEVNTBLTzn5jAUmwCehI /F7xC8wWEdCWeP14KpjNLLCLUWLhOSkQW1jAS2J7w0k2EJtFQFXi6o92JhCbV8Ba4ueja2wQ y+Ql9h88ywxicwrYSFyaPosdokZQ4uTMJ1Az5SWat85mBjlCQuAvu8Sl2++ZIJpdJE6+mcEK YQtLvDq+hR3ClpL4/G4v0AIOILtYYtk6OYjeFkaJ9+/mQH1pLfHvyh4WkBpmAU2J9bv0IcKy ElNPrWOC2Msn0fv7CdQqXokd82BsVYkLB7dBrZKWWDthKzOE7SHx49dVaFj3M0r0rX7MNoFR YRaSf2Yh+WcWwuoFjMyrGCVTC4pz01OLTQsM81LL4bGcnJ+7iRGcTrU8dzDeffBB7xAjEwfj IUYJDmYlEV7+zLo0Id6UxMqq1KL8+KLSnNTiQ4ymwACfyCwlmpwPTOh5JfGGJpYGJmZmZiaW xmaGSuK8r1vnpggJpCeWpGanphakFsH0MXFwSjUwZSVPnbVgyXmf2HudVxd9sXmrl7EwdWnz u5ZWL/t+t+8WYd3cfCYCU3wv/f+8bFWjzs7TShnxcW5Z81unSZssOPUuRfpyd9n3zkeyS75p eGSeKNqusn3R1Wr2d38ja/eX3y4Qcl55Y5eg/Wm7IJbqyHuHt25Mzol+m7Ml8K7M4eTdf9gO mDlLJTdvSHJI9xffN3fHga1HN83tevz3Fu8819ANSZ+7ODw/CO7UupmrfH1F8sJDgu+enBf6 xb5SI6Lb/hcj1/Rit7zs0JP5zw5sLUyoVjDcMSOr65l8yLmlz5uWnH3vzPj+lKxCzYf3b7XW tAc+3mVjNKU76aI6Q2X73Zkmc7fsvbJv/hMbgQMffZRYijMSDbWYi4oTAYHiGrEwBAAA X-Brightmail-Tracker: H4sIAAAAAAAAA+NgFrrMLMWRmVeSWpSXmKPExsWy7bCSnO6z8ro0g8blXBZNE/4yW8xZtY3R YvXdfjaL038fs1jcPLCTyeJd6zkWi6P/37JZ/Oq+y2ix9ctXVotnezktzk74wOrA7bFz1l12 j8tnSz36tqxi9Pi8SS6AJYrLJiU1J7MstUjfLoEro2XLU5aCZtGKqdNXsTQwzhXoYuTkkBAw kVjRMo+xi5GLQ0hgN6PEsxvr2SAS0hIdh1rZIWxhiZX/nrNDFDUwSbS+m8AIkmAT0JH4veIX SxcjB4eIgK5E410FkBpmgUOMEs0bmsFqhAW8JLY3nAQbyiKgKnH1RzsTiM0rYC3x89E1qGXy EvsPnmUGsTkFbCQuTZ/FDjJTCKjmw91IiHJBiZMzn7CA2MxA5c1bZzNPYBSYhSQ1C0lqASPT KkbJ1ILi3PTcZMMCw7zUcr3ixNzi0rx0veT83E2M4HDX0tjBeG/+P71DjEwcjEA3czArifDy Z9alCfGmJFZWpRblxxeV5qQWH2KU5mBREuc1nDE7RUggPbEkNTs1tSC1CCbLxMEp1cDk9O/Z /u+7l127ebfCXvGKcqvR3xNeT6am7ljhHOXLITbDVSm1beOebzkydbLV96skDyvJmuok9S4W l0rpvtaxus574/ccI4cvL4+fN3+gqsTyREl3pnuv2IrKd8qzH/FsY/00c61JwEKJwvz1ue7f 6zfO3JPh/uboLmljrQ1as3Za77j8TP/p1XMrX+tU5M16Xjih9u/s/uKzl3oDH736nB2zuZLX b5rsunMbO9LkO1Q715kvUlGT+9+u82baDE6j/ELHxQunWbk+cj0/62yVYKq89Sme/228l60f Lbe8+HG3oIDTueyu7JW/NpuGJ73hnVbVycayLi/3/qkAjUPM+xUPPOz9YtzV1Lpmwc12JZbi jERDLeai4kQADDFmQ+YCAAA= X-CMS-MailID: 20240628084422epcas5p3b5d4c93e5fa30069c703bcead1fa0033 X-Msg-Generator: CA X-Sendblock-Type: REQ_APPROVE CMS-TYPE: 105P DLP-Filter: Pass X-CFilter-Loop: Reflected X-CMS-RootMailID: 20240628084422epcas5p3b5d4c93e5fa30069c703bcead1fa0033 References: <20240628084411.2371-1-cliang01.li@samsung.com> Store the folio shift and folio mask into imu struct and use it in iov_iter adjust, as we will have non PAGE_SIZE'd chunks if a multi-hugepage buffer get coalesced. Signed-off-by: Chenliang Li Reviewed-by: Anuj Gupta --- io_uring/rsrc.c | 15 ++++++--------- io_uring/rsrc.h | 2 ++ 2 files changed, 8 insertions(+), 9 deletions(-) diff --git a/io_uring/rsrc.c b/io_uring/rsrc.c index c88ce8c38515..3198cf854db1 100644 --- a/io_uring/rsrc.c +++ b/io_uring/rsrc.c @@ -1002,6 +1002,8 @@ static int io_sqe_buffer_register(struct io_ring_ctx *ctx, struct iovec *iov, imu->ubuf = (unsigned long) iov->iov_base; imu->ubuf_end = imu->ubuf + iov->iov_len; imu->nr_bvecs = nr_pages; + imu->folio_shift = PAGE_SHIFT; + imu->folio_mask = PAGE_MASK; *pimu = imu; ret = 0; @@ -1118,23 +1120,18 @@ int io_import_fixed(int ddir, struct iov_iter *iter, * we know that: * * 1) it's a BVEC iter, we set it up - * 2) all bvecs are PAGE_SIZE in size, except potentially the + * 2) all bvecs are the same in size, except potentially the * first and last bvec * * So just find our index, and adjust the iterator afterwards. * If the offset is within the first bvec (or the whole first * bvec, just use iov_iter_advance(). This makes it easier * since we can just skip the first segment, which may not - * be PAGE_SIZE aligned. + * be folio_size aligned. */ const struct bio_vec *bvec = imu->bvec; if (offset < bvec->bv_len) { - /* - * Note, huge pages buffers consists of one large - * bvec entry and should always go this way. The other - * branch doesn't expect non PAGE_SIZE'd chunks. - */ iter->bvec = bvec; iter->nr_segs = bvec->bv_len; iter->count -= offset; @@ -1144,12 +1141,12 @@ int io_import_fixed(int ddir, struct iov_iter *iter, /* skip first vec */ offset -= bvec->bv_len; - seg_skip = 1 + (offset >> PAGE_SHIFT); + seg_skip = 1 + (offset >> imu->folio_shift); iter->bvec = bvec + seg_skip; iter->nr_segs -= seg_skip; iter->count -= bvec->bv_len + offset; - iter->iov_offset = offset & ~PAGE_MASK; + iter->iov_offset = offset & ~imu->folio_mask; } } diff --git a/io_uring/rsrc.h b/io_uring/rsrc.h index cc66323535f6..b02fc0ef59fe 100644 --- a/io_uring/rsrc.h +++ b/io_uring/rsrc.h @@ -46,7 +46,9 @@ struct io_mapped_ubuf { u64 ubuf; u64 ubuf_end; unsigned int nr_bvecs; + unsigned int folio_shift; unsigned long acct_pages; + unsigned long folio_mask; struct bio_vec bvec[] __counted_by(nr_bvecs); }; From patchwork Fri Jun 28 08:44:11 2024 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Chenliang Li X-Patchwork-Id: 13716839 Received: from mailout1.samsung.com (mailout1.samsung.com [203.254.224.24]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (No client certificate requested) by smtp.subspace.kernel.org (Postfix) with ESMTPS id 7C92E14A85 for ; Sat, 29 Jun 2024 07:26:34 +0000 (UTC) Authentication-Results: smtp.subspace.kernel.org; arc=none smtp.client-ip=203.254.224.24 ARC-Seal: i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1719645996; cv=none; b=eB58bs8g2s6loUijpPYPK724qX6AbohOKW7blx3obIy9Uq9dO5fc4ZROw4PlBG0/4Wld8WyeE+mDsEMq3BT2CpKR+mBTwIcZ6slMNuKLUPT0/QjMhlxFfL5vNSFaF/F4vNBhrbS8dfWxTgbdXaPIblvxlPbtLIjEIhgDjOgygtY= ARC-Message-Signature: i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1719645996; c=relaxed/simple; bh=olUa9NNU5ieHcq+Y0HijNMgWRehRC83XOiz7F25J15s=; h=From:To:Cc:Subject:Date:Message-Id:In-Reply-To:MIME-Version: Content-Type:References; b=eAkwVb2KAvh/GYuKUaWUukgW1xMSg6wTsG1GRfjSVYLiA2BoHzIBYqGcalsO5eT6hCLvirM0vO3+We5EMmOYYVvrSMtPUcaAGDsx9ljSMVYku9b1N4zsJ7CRn/P7Gfqz32wqakcuX6MbZvygNrNbgo922BRvwp8ylh2j2O4Svxc= ARC-Authentication-Results: i=1; smtp.subspace.kernel.org; dmarc=pass (p=none dis=none) header.from=samsung.com; spf=pass smtp.mailfrom=samsung.com; dkim=pass (1024-bit key) header.d=samsung.com header.i=@samsung.com header.b=bsXHeIjx; arc=none smtp.client-ip=203.254.224.24 Authentication-Results: smtp.subspace.kernel.org; dmarc=pass (p=none dis=none) header.from=samsung.com Authentication-Results: smtp.subspace.kernel.org; spf=pass smtp.mailfrom=samsung.com Authentication-Results: smtp.subspace.kernel.org; dkim=pass (1024-bit key) header.d=samsung.com header.i=@samsung.com header.b="bsXHeIjx" Received: from epcas5p1.samsung.com (unknown [182.195.41.39]) by mailout1.samsung.com (KnoxPortal) with ESMTP id 20240629072627epoutp01532a3b1cc3466a99d61867fbc9c771d1~daOoQbekt0894708947epoutp01Q for ; Sat, 29 Jun 2024 07:26:27 +0000 (GMT) DKIM-Filter: OpenDKIM Filter v2.11.0 mailout1.samsung.com 20240629072627epoutp01532a3b1cc3466a99d61867fbc9c771d1~daOoQbekt0894708947epoutp01Q DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=samsung.com; s=mail20170921; t=1719645987; bh=6/v7IO359KNVg7LZfAqkCs7n+e4c0tDwe9IRKktQPCo=; h=From:To:Cc:Subject:Date:In-Reply-To:References:From; b=bsXHeIjxQBf88QiLovNMRTIaEpwJD7PjbU93PYX0aEeTkamGomhmoOaP6mE0Rle5f 1GdSYZpO23ehgb/SCTNmV3bZXAu+TaoPb63f9w2lCEpyetGr2BLqie+2pXuWjMYWPj tXD87LHMKLSQY+K2OYVA0Mxyc5W31ERMgyvgWOiE= Received: from epsnrtp4.localdomain (unknown [182.195.42.165]) by epcas5p4.samsung.com (KnoxPortal) with ESMTP id 20240629072626epcas5p4a1e05ea0923d97a61242039d34df095d~daOn6P7IE1644816448epcas5p4P; Sat, 29 Jun 2024 07:26:26 +0000 (GMT) Received: from epsmges5p2new.samsung.com (unknown [182.195.38.177]) by epsnrtp4.localdomain (Postfix) with ESMTP id 4WB3l54YYlz4x9Q1; Sat, 29 Jun 2024 07:26:25 +0000 (GMT) Received: from epcas5p3.samsung.com ( [182.195.41.41]) by epsmges5p2new.samsung.com (Symantec Messaging Gateway) with SMTP id D0.18.09989.127BF766; Sat, 29 Jun 2024 16:26:25 +0900 (KST) Received: from epsmtrp2.samsung.com (unknown [182.195.40.14]) by epcas5p3.samsung.com (KnoxPortal) with ESMTPA id 20240628084424epcas5p3c34ec2fb8fb45752ef6a11447812ae0d~dHpZveEF62959029590epcas5p3v; Fri, 28 Jun 2024 08:44:24 +0000 (GMT) Received: from epsmgms1p1new.samsung.com (unknown [182.195.42.41]) by epsmtrp2.samsung.com (KnoxPortal) with ESMTP id 20240628084424epsmtrp2ac106bb238054bd8e9b8034ab64773cc~dHpZux7Co0972309723epsmtrp2C; Fri, 28 Jun 2024 08:44:24 +0000 (GMT) X-AuditID: b6c32a4a-e57f970000002705-d2-667fb7214541 Received: from epsmtip1.samsung.com ( [182.195.34.30]) by epsmgms1p1new.samsung.com (Symantec Messaging Gateway) with SMTP id D3.14.29940.8E77E766; Fri, 28 Jun 2024 17:44:24 +0900 (KST) Received: from lcl-Standard-PC-i440FX-PIIX-1996.. (unknown [109.105.118.124]) by epsmtip1.samsung.com (KnoxPortal) with ESMTPA id 20240628084423epsmtip1b007ffc2cfce9051c28fe004bba20491~dHpYs2f-Q0054500545epsmtip1y; Fri, 28 Jun 2024 08:44:22 +0000 (GMT) From: Chenliang Li To: axboe@kernel.dk, asml.silence@gmail.com Cc: io-uring@vger.kernel.org, peiwei.li@samsung.com, joshi.k@samsung.com, kundan.kumar@samsung.com, anuj20.g@samsung.com, gost.dev@samsung.com, Chenliang Li Subject: [PATCH v5 3/3] io_uring/rsrc: enable multi-hugepage buffer coalescing Date: Fri, 28 Jun 2024 16:44:11 +0800 Message-Id: <20240628084411.2371-4-cliang01.li@samsung.com> X-Mailer: git-send-email 2.34.1 In-Reply-To: <20240628084411.2371-1-cliang01.li@samsung.com> Precedence: bulk X-Mailing-List: io-uring@vger.kernel.org List-Id: List-Subscribe: List-Unsubscribe: MIME-Version: 1.0 X-Brightmail-Tracker: H4sIAAAAAAAAA+NgFprLJsWRmVeSWpSXmKPExsWy7bCmpq7i9vo0g/X/5CyaJvxltpizahuj xeq7/WwWp/8+ZrG4eWAnk8W71nMsFkf/v2Wz+NV9l9Fi65evrBbP9nJanJ3wgdWB22PnrLvs HpfPlnr0bVnF6PF5k1wAS1S2TUZqYkpqkUJqXnJ+SmZeuq2Sd3C8c7ypmYGhrqGlhbmSQl5i bqqtkotPgK5bZg7QTUoKZYk5pUChgMTiYiV9O5ui/NKSVIWM/OISW6XUgpScApMCveLE3OLS vHS9vNQSK0MDAyNToMKE7IxNE2YxF/wVqZj9+DJrA+MsgS5GTg4JAROJDw/+snQxcnEICexm lLiwdAsbhPOJUWL+ytPMEM43RokJ8++xwrS03+1mArGFBPYyStw6ZARR1MQkcbJtJVgRm4CO xO8Vv1hAbBEBbYnXj6eC2cwCuxglFp6TArGFBQIknjWcZwexWQRUJW7vPwtm8wpYSzz4tZIZ Ypm8xP6DZ8FsTgEbiUvTZ0HVCEqcnPkEaqa8RPPW2WCXSgg0ckg82/6ZDaLZReL2wgeMELaw xKvjW9ghbCmJz+/2AtVwANnFEsvWyUH0tjBKvH83B6reWuLflT0sIDXMApoS63fpQ4RlJaae WscEsZdPovf3EyaIOK/EjnkwtqrEhYPboFZJS6ydsBXqFw+Jza+aGSGB1c8oseTUL8YJjAqz kPwzC8k/sxBWL2BkXsUomVpQnJueWmxaYJSXWg6P5eT83E2M4HSq5bWD8eGDD3qHGJk4GA8x SnAwK4nw8mfWpQnxpiRWVqUW5ccXleakFh9iNAUG+ERmKdHkfGBCzyuJNzSxNDAxMzMzsTQ2 M1QS533dOjdFSCA9sSQ1OzW1ILUIpo+Jg1OqgWme0BrOMwdVzjwUrnSofXP5eYfGne7WLwJ7 zv6fJpXGK5E3WXjLB6/EnKyzRcHGy6ZFbtNbovdIaN/PhnxFw2Xn/B1EfjAd/rOvYeKT4yd+ BHT+fHvw4KLddtE7697ovA5rvGbAG3Fj4wnnmM2f8i5En7Y/utoyYvGnczeVhbZsM/xlfDhW 7FmUW+fM6Ik9fbe9jR9ps61g2bb5WeTPZ69ZL3Tc+iRVErnYWqstO2Ot0UvXs3rTfUQz7nmq ydj0Jm7Ztu/QtvM8k4NuHFg8fXGWvfqeQL8DP6rW5u5M6vY8ILLKKu7M2WVbL7WoqCu/WKLg Zud/mPmR+nrVaptlgjKHq+sd+jN8vrTX71CWk9unxFKckWioxVxUnAgA7tPECjAEAAA= X-Brightmail-Tracker: H4sIAAAAAAAAA+NgFrrMLMWRmVeSWpSXmKPExsWy7bCSnO6L8ro0g/PXOSyaJvxltpizahuj xeq7/WwWp/8+ZrG4eWAnk8W71nMsFkf/v2Wz+NV9l9Fi65evrBbP9nJanJ3wgdWB22PnrLvs HpfPlnr0bVnF6PF5k1wASxSXTUpqTmZZapG+XQJXxqYJs5gL/opUzH58mbWBcZZAFyMnh4SA iUT73W6mLkYuDiGB3YwSc18eYINISEt0HGplh7CFJVb+e84OUdTAJDHv6yNmkASbgI7E7xW/ WLoYOThEBHQlGu8qgNQwCxxilGje0MwIUiMs4CdxoW8eE4jNIqAqcXv/WbChvALWEg9+rWSG WCAvsf/gWTCbU8BG4tL0WewgM4WAaj7cjYQoF5Q4OfMJC4jNDFTevHU28wRGgVlIUrOQpBYw Mq1ilEwtKM5Nzy02LDDMSy3XK07MLS7NS9dLzs/dxAgOdy3NHYzbV33QO8TIxMEIdDMHs5II L39mXZoQb0piZVVqUX58UWlOavEhRmkOFiVxXvEXvSlCAumJJanZqakFqUUwWSYOTqkGpsIf Veee2Res+JG58dOL5cy7tdZkLy5xbj++kXkNf8IxJeNZtiq+2+/MmqHRVvThz61ZCXPK3KNP b7v+JqlYXSEi+MKP87GKibsZ7sbxzhI5OYV3s11HYsn+ad945C5nnvs6Zc9SJsmIZp413VNk fy3fK5j2Mv+pbrfLoeg77d/5rDbFzjGx4JpXwX03I/pK2yGHSVlcr3aeturmaD8dJPnrghRD zLOvCes1jz5g3Vw1//Dc2nYB898+XcInPxsVCTYqZly71/W7IM/v5boi64MhrbNmbby2//+n tq3vvu0zn7D45aPlf4qvcqkylJ/L5r/vJfZ05u0LuvWscqfiip6KNW2fLLjE4/TkfJ/THxYp sRRnJBpqMRcVJwIA2rwVxeYCAAA= X-CMS-MailID: 20240628084424epcas5p3c34ec2fb8fb45752ef6a11447812ae0d X-Msg-Generator: CA X-Sendblock-Type: REQ_APPROVE CMS-TYPE: 105P DLP-Filter: Pass X-CFilter-Loop: Reflected X-CMS-RootMailID: 20240628084424epcas5p3c34ec2fb8fb45752ef6a11447812ae0d References: <20240628084411.2371-1-cliang01.li@samsung.com> Modify io_sqe_buffer_register to enable the coalescing for multi-hugepage fixed buffers. Signed-off-by: Chenliang Li --- io_uring/rsrc.c | 47 ++++++++++++++++------------------------------- 1 file changed, 16 insertions(+), 31 deletions(-) diff --git a/io_uring/rsrc.c b/io_uring/rsrc.c index 3198cf854db1..790ed3c1bcc8 100644 --- a/io_uring/rsrc.c +++ b/io_uring/rsrc.c @@ -945,7 +945,8 @@ static int io_sqe_buffer_register(struct io_ring_ctx *ctx, struct iovec *iov, unsigned long off; size_t size; int ret, nr_pages, i; - struct folio *folio = NULL; + struct io_imu_folio_data data; + bool coalesced; *pimu = (struct io_mapped_ubuf *)&dummy_ubuf; if (!iov->iov_base) @@ -960,31 +961,8 @@ static int io_sqe_buffer_register(struct io_ring_ctx *ctx, struct iovec *iov, goto done; } - /* If it's a huge page, try to coalesce them into a single bvec entry */ - if (nr_pages > 1) { - folio = page_folio(pages[0]); - for (i = 1; i < nr_pages; i++) { - /* - * Pages must be consecutive and on the same folio for - * this to work - */ - if (page_folio(pages[i]) != folio || - pages[i] != pages[i - 1] + 1) { - folio = NULL; - break; - } - } - if (folio) { - /* - * The pages are bound to the folio, it doesn't - * actually unpin them but drops all but one reference, - * which is usually put down by io_buffer_unmap(). - * Note, needs a better helper. - */ - unpin_user_pages(&pages[1], nr_pages - 1); - nr_pages = 1; - } - } + /* If it's huge page(s), try to coalesce them into fewer bvec entries */ + coalesced = io_try_coalesce_buffer(&pages, &nr_pages, &data); imu = kvmalloc(struct_size(imu, bvec, nr_pages), GFP_KERNEL); if (!imu) @@ -1004,17 +982,24 @@ static int io_sqe_buffer_register(struct io_ring_ctx *ctx, struct iovec *iov, imu->nr_bvecs = nr_pages; imu->folio_shift = PAGE_SHIFT; imu->folio_mask = PAGE_MASK; + if (coalesced) { + imu->folio_shift = data.folio_shift; + imu->folio_mask = ~((1UL << data.folio_shift) - 1); + } *pimu = imu; ret = 0; - if (folio) { - bvec_set_page(&imu->bvec[0], pages[0], size, off); - goto done; - } for (i = 0; i < nr_pages; i++) { size_t vec_len; - vec_len = min_t(size_t, size, PAGE_SIZE - off); + if (coalesced) { + size_t seg_size = i ? data.folio_size : + PAGE_SIZE * data.nr_pages_head; + + vec_len = min_t(size_t, size, seg_size - off); + } else { + vec_len = min_t(size_t, size, PAGE_SIZE - off); + } bvec_set_page(&imu->bvec[i], pages[i], vec_len, off); off = 0; size -= vec_len;