From patchwork Tue May 14 07:54:41 2024 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Chenliang Li X-Patchwork-Id: 13663847 Received: from mailout1.samsung.com (mailout1.samsung.com [203.254.224.24]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (No client certificate requested) by smtp.subspace.kernel.org (Postfix) with ESMTPS id 02863125A9 for ; Tue, 14 May 2024 07:57:31 +0000 (UTC) Authentication-Results: smtp.subspace.kernel.org; arc=none smtp.client-ip=203.254.224.24 ARC-Seal: i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1715673454; cv=none; b=dJSLxDjd96Y3LMtIoDu2MiYZtnN9MoDXpNpbwQHJfNI+Dp/htW+pE/+hTUJwxB+rnZayYZgj90AS4S/rg1IZKJd+/jDpSBCkkxIsVXcg6CHfrH3GbyWSsTDK/NSY2xqBlhF7B3xesprpLFe07QDOcojpQmdTu+AqWvtGcBACkjk= ARC-Message-Signature: i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1715673454; c=relaxed/simple; bh=GgB79gShNLQoKk+gewD6WVXUhfoRMFUqTI993fcouUM=; h=From:To:Cc:Subject:Date:Message-Id:In-Reply-To:MIME-Version: Content-Type:References; b=ZlGJX6R/heMdawzonxHQN+VKfXUatRY8xt+Dy+BQhc20fV9gPiGcF5ncgG7kz1FNplZDO6cjOD15YZkL6FWiDtPN8V920o/uOFSh39BYzcGA/GJ0xXQyC4GJ/op70zyVVJGxd6ny8ThuBdRh576N+IRA9M8gc80D2VILZjMYUMs= ARC-Authentication-Results: i=1; smtp.subspace.kernel.org; dmarc=pass (p=none dis=none) header.from=samsung.com; spf=pass smtp.mailfrom=samsung.com; dkim=pass (1024-bit key) header.d=samsung.com header.i=@samsung.com header.b=faID5NVQ; arc=none smtp.client-ip=203.254.224.24 Authentication-Results: smtp.subspace.kernel.org; dmarc=pass (p=none dis=none) header.from=samsung.com Authentication-Results: smtp.subspace.kernel.org; spf=pass smtp.mailfrom=samsung.com Authentication-Results: smtp.subspace.kernel.org; dkim=pass (1024-bit key) header.d=samsung.com header.i=@samsung.com header.b="faID5NVQ" Received: from epcas5p3.samsung.com (unknown [182.195.41.41]) by mailout1.samsung.com (KnoxPortal) with ESMTP id 20240514075728epoutp0114586bcf4ab64ad7c398ed08b3e68b08~PS_lb22y61756217562epoutp01p for ; Tue, 14 May 2024 07:57:28 +0000 (GMT) DKIM-Filter: OpenDKIM Filter v2.11.0 mailout1.samsung.com 20240514075728epoutp0114586bcf4ab64ad7c398ed08b3e68b08~PS_lb22y61756217562epoutp01p DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=samsung.com; s=mail20170921; t=1715673448; bh=luky4jv5sJLLkurx2YzjEwBLcK+91Q/SXM6pUFX2CzU=; h=From:To:Cc:Subject:Date:In-Reply-To:References:From; b=faID5NVQXC5L4sKV0XgFDVte5/tNCHCdAnXo0fFrQAiO6/9tNhyGdLLge20+ikYO2 JtI70+FRbpth5dQ3MzSsAZkGNBbzulY4LwFsCn+DpSaPrJTDP+nFU+FzlDsEH1uPDb Nl6FXff4jOEEPCkP9Mb0H9+lc9h3YUnVHaByS2zw= Received: from epsnrtp4.localdomain (unknown [182.195.42.165]) by epcas5p3.samsung.com (KnoxPortal) with ESMTP id 20240514075728epcas5p3c089ac80a41cadc715105eb9918be4f7~PS_k7Biw11959419594epcas5p3M; Tue, 14 May 2024 07:57:28 +0000 (GMT) Received: from epsmges5p1new.samsung.com (unknown [182.195.38.178]) by epsnrtp4.localdomain (Postfix) with ESMTP id 4Vdpc624ytz4x9QF; Tue, 14 May 2024 07:57:26 +0000 (GMT) Received: from epcas5p4.samsung.com ( [182.195.41.42]) by epsmges5p1new.samsung.com (Symantec Messaging Gateway) with SMTP id 53.7E.09666.66913466; Tue, 14 May 2024 16:57:26 +0900 (KST) Received: from epsmtrp1.samsung.com (unknown [182.195.40.13]) by epcas5p1.samsung.com (KnoxPortal) with ESMTPA id 20240514075457epcas5p10f02f1746f957df91353724ec859664f~PS8YdLRbi2658726587epcas5p1v; Tue, 14 May 2024 07:54:57 +0000 (GMT) Received: from epsmgms1p2new.samsung.com (unknown [182.195.42.42]) by epsmtrp1.samsung.com (KnoxPortal) with ESMTP id 20240514075457epsmtrp1c0a35af7cd0ac40b6171404cf6b7e852~PS8YcZ3wx0538105381epsmtrp1D; Tue, 14 May 2024 07:54:57 +0000 (GMT) X-AuditID: b6c32a49-cefff700000025c2-b5-664319666521 Received: from epsmtip2.samsung.com ( [182.195.34.31]) by epsmgms1p2new.samsung.com (Symantec Messaging Gateway) with SMTP id 61.D2.08390.1D813466; Tue, 14 May 2024 16:54:57 +0900 (KST) Received: from testpc118124.samsungds.net (unknown [109.105.118.124]) by epsmtip2.samsung.com (KnoxPortal) with ESMTPA id 20240514075456epsmtip2012dda6b7119e958e40e1700be8db43f~PS8XYIV0j0412004120epsmtip2R; Tue, 14 May 2024 07:54:56 +0000 (GMT) From: Chenliang Li To: axboe@kernel.dk, asml.silence@gmail.com Cc: io-uring@vger.kernel.org, peiwei.li@samsung.com, joshi.k@samsung.com, kundan.kumar@samsung.com, anuj20.g@samsung.com, gost.dev@samsung.com, Chenliang Li Subject: [PATCH v4 1/4] io_uring/rsrc: add hugepage buffer coalesce helpers Date: Tue, 14 May 2024 15:54:41 +0800 Message-Id: <20240514075444.590910-2-cliang01.li@samsung.com> X-Mailer: git-send-email 2.34.1 In-Reply-To: <20240514075444.590910-1-cliang01.li@samsung.com> Precedence: bulk X-Mailing-List: io-uring@vger.kernel.org List-Id: List-Subscribe: List-Unsubscribe: MIME-Version: 1.0 X-Brightmail-Tracker: H4sIAAAAAAAAA+NgFprDJsWRmVeSWpSXmKPExsWy7bCmlm6apHOawZPZPBZNE/4yW8xZtY3R YvXdfjaL038fs1jcPLCTyeJd6zkWi6P/37JZ/Oq+y2ix9ctXVotnezktzk74wOrA7bFz1l12 j8tnSz36tqxi9Pi8SS6AJSrbJiM1MSW1SCE1Lzk/JTMv3VbJOzjeOd7UzMBQ19DSwlxJIS8x N9VWycUnQNctMwfoJiWFssScUqBQQGJxsZK+nU1RfmlJqkJGfnGJrVJqQUpOgUmBXnFibnFp XrpeXmqJlaGBgZEpUGFCdsa86TtYC47KVVxdeY21gfGYRBcjJ4eEgInE6S0TmUFsIYHdjBJT Ztd2MXIB2Z8YJR48ucYG4XxjlPj27z07TMeH05NYIRJ7GSX+/F/KDuH8YpTY8PgbE0gVm4CO xO8Vv1hAbBEBbYnXj6eC2cwCuxglFp6TArGFBbwl5k+6wQZiswioSpz+dAhsA6+ArcTf7sVM ENvkJfYfPAt0HwcHp4CdxKkZLhAlghInZz6BGikv0bx1NjPIDRICP9klplw+BHWpi0Rz7x82 CFtY4tXxLVBxKYmX/W3sIDMlBIollq2Tg+htYZR4/24OI0SNtcS/K3tYQGqYBTQl1u/ShwjL Skw9tY4JYi+fRO/vJ1Bn8krsmAdjq0pcOLgNapW0xNoJW5khbA+Jn/vvsEDCaiKjxL+Tz5kn MCrMQvLPLCT/zEJYvYCReRWjZGpBcW56arFpgWFeajk8kpPzczcxgpOplucOxrsPPugdYmTi YDzEKMHBrCTC61BonybEm5JYWZValB9fVJqTWnyI0RQY3hOZpUST84HpPK8k3tDE0sDEzMzM xNLYzFBJnPd169wUIYH0xJLU7NTUgtQimD4mDk6pBibHmzeKjgecfHcgyOmMpfMME/UzK3cq sfX2/2+9lZFRdHXe42kCG30u/yxOvBc/67nWg1XXS7qLhbQ6limuzZtxPvTVjbfKEwM/Srzf 2/Y0/XqxT8YHLh/J+SmT2G8mJbFN+sZm/q/z5BHNlwc3vF3FWHr9QNoq28MVRsleZXeuP3s5 7WiCz5Nf9050MgRtu8tw1tj546313bkRddfNV/yW+H0r65jjD//UYj7nKUzNLmVOC/wPfLkm fnMql/HnfXefXbC95+WSeSw3dlb+5C3PAhcs1lQr3DHFiPHPSvc7eVuXz1rBYCPO8OjqM1HT 9OebjMWW5x+5qnZ+n/6VLUc2TZPy4BdpPzQpYXe+T/uyy0osxRmJhlrMRcWJAMQobpEvBAAA X-Brightmail-Tracker: H4sIAAAAAAAAA+NgFrrELMWRmVeSWpSXmKPExsWy7bCSvO5FCec0g2sveSyaJvxltpizahuj xeq7/WwWp/8+ZrG4eWAnk8W71nMsFkf/v2Wz+NV9l9Fi65evrBbP9nJanJ3wgdWB22PnrLvs HpfPlnr0bVnF6PF5k1wASxSXTUpqTmZZapG+XQJXxrzpO1gLjspVXF15jbWB8ZhEFyMnh4SA icSH05NYQWwhgd2MEs0ThSHi0hIdh1rZIWxhiZX/ngPZXEA1PxglJrz5xgySYBPQkfi94hdL FyMHh4iArkTjXQWQGmaBQ0BzNjQzgtQIC3hLzJ90gw3EZhFQlTj96RDYUF4BW4m/3YuZIBbI S+w/eJYZZA6ngJ3EqRkuEPfYSmz+cxyqXFDi5MwnLCA2M1B589bZzBMYBWYhSc1CklrAyLSK UTK1oDg3PbfYsMAoL7Vcrzgxt7g0L10vOT93EyM42LW0djDuWfVB7xAjEwcj0M0czEoivA6F 9mlCvCmJlVWpRfnxRaU5qcWHGKU5WJTEeb+97k0REkhPLEnNTk0tSC2CyTJxcEo1MCk8P7vZ R7k25fPbzUdkZdQX8au/LFsykXv7J9ONgbKHrnw9Ok897r3EzROhj2PUhT1v+TR3OEQy+T48 y3BJemVzl/IVnfoaU2ZdrTv1nOxSjnfl1n/976B6KZHv547m8/MEGF3ef2pd8cJdsONE/430 hIInzz5uNJrS5hd/9KTk0y+vfpVE9OzivJTxMdv63PWLbhcXLg3SvxVf1yS6V8HlpNVuxR2T RTSkuLu2RS368dZrUdKXLzWbxXMXqj+Rb3tslu7L/kPtdNemwhseM5jFFr4pvinxp9v73C65 lrhnWuJvn+5/uVTikdmlo6JCASYurbvqqpkMQyRCJBaq1zdcPqDeritc78vw4mFJlRJLcUai oRZzUXEiAOh6ta3lAgAA X-CMS-MailID: 20240514075457epcas5p10f02f1746f957df91353724ec859664f X-Msg-Generator: CA X-Sendblock-Type: REQ_APPROVE CMS-TYPE: 105P DLP-Filter: Pass X-CFilter-Loop: Reflected X-CMS-RootMailID: 20240514075457epcas5p10f02f1746f957df91353724ec859664f References: <20240514075444.590910-1-cliang01.li@samsung.com> Introduce helper functions to check whether a buffer can be coalesced or not, and gather folio data for later use. The coalescing optimizes time and space consumption caused by mapping and storing multi-hugepage fixed buffers. A coalescable multi-hugepage buffer should fully cover its folios (except potentially the first and last one), and these folios should have the same size. These requirements are for easier later process, also we need same size'd chunks in io_import_fixed for fast iov_iter adjust. Signed-off-by: Chenliang Li Reviewed-by: Anuj Gupta --- io_uring/rsrc.c | 78 +++++++++++++++++++++++++++++++++++++++++++++++++ io_uring/rsrc.h | 10 +++++++ 2 files changed, 88 insertions(+) diff --git a/io_uring/rsrc.c b/io_uring/rsrc.c index 65417c9553b1..d08224c0c5b0 100644 --- a/io_uring/rsrc.c +++ b/io_uring/rsrc.c @@ -871,6 +871,84 @@ static int io_buffer_account_pin(struct io_ring_ctx *ctx, struct page **pages, return ret; } +static bool __io_sqe_buffer_try_coalesce(struct page **pages, int nr_pages, + struct io_imu_folio_data *data) +{ + struct folio *folio = page_folio(pages[0]); + unsigned int count = 1; + int i; + + data->nr_pages_mid = folio_nr_pages(folio); + if (data->nr_pages_mid == 1) + return false; + + data->folio_shift = folio_shift(folio); + data->folio_size = folio_size(folio); + data->nr_folios = 1; + /* + * Check if pages are contiguous inside a folio, and all folios have + * the same page count except for the head and tail. + */ + for (i = 1; i < nr_pages; i++) { + if (page_folio(pages[i]) == folio && + pages[i] == pages[i-1] + 1) { + count++; + continue; + } + + if (data->nr_folios == 1) + data->nr_pages_head = count; + else if (count != data->nr_pages_mid) + return false; + + folio = page_folio(pages[i]); + if (folio_size(folio) != data->folio_size) + return false; + + count = 1; + data->nr_folios++; + } + if (data->nr_folios == 1) + data->nr_pages_head = count; + + return true; +} + +static bool io_sqe_buffer_try_coalesce(struct page **pages, int nr_pages, + struct io_imu_folio_data *data) +{ + int i, j; + + if (nr_pages <= 1 || + !__io_sqe_buffer_try_coalesce(pages, nr_pages, data)) + return false; + + /* + * The pages are bound to the folio, it doesn't + * actually unpin them but drops all but one reference, + * which is usually put down by io_buffer_unmap(). + * Note, needs a better helper. + */ + if (data->nr_pages_head > 1) + unpin_user_pages(&pages[1], data->nr_pages_head - 1); + + j = data->nr_pages_head; + nr_pages -= data->nr_pages_head; + for (i = 1; i < data->nr_folios; i++) { + unsigned int nr_unpin; + + nr_unpin = min_t(unsigned int, nr_pages - 1, + data->nr_pages_mid - 1); + if (nr_unpin == 0) + break; + unpin_user_pages(&pages[j+1], nr_unpin); + j += data->nr_pages_mid; + nr_pages -= data->nr_pages_mid; + } + + return true; +} + static int io_sqe_buffer_register(struct io_ring_ctx *ctx, struct iovec *iov, struct io_mapped_ubuf **pimu, struct page **last_hpage) diff --git a/io_uring/rsrc.h b/io_uring/rsrc.h index c032ca3436ca..b2a9d66b76dd 100644 --- a/io_uring/rsrc.h +++ b/io_uring/rsrc.h @@ -50,6 +50,16 @@ struct io_mapped_ubuf { struct bio_vec bvec[] __counted_by(nr_bvecs); }; +struct io_imu_folio_data { + /* Head folio can be partially included in the fixed buf */ + unsigned int nr_pages_head; + /* For non-head/tail folios, has to be fully included */ + unsigned int nr_pages_mid; + unsigned int nr_folios; + unsigned int folio_shift; + size_t folio_size; +}; + void io_rsrc_node_ref_zero(struct io_rsrc_node *node); void io_rsrc_node_destroy(struct io_ring_ctx *ctx, struct io_rsrc_node *ref_node); struct io_rsrc_node *io_rsrc_node_alloc(struct io_ring_ctx *ctx); From patchwork Tue May 14 07:54:42 2024 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Chenliang Li X-Patchwork-Id: 13663844 Received: from mailout2.samsung.com (mailout2.samsung.com [203.254.224.25]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (No client certificate requested) by smtp.subspace.kernel.org (Postfix) with ESMTPS id 9D99C15E81 for ; Tue, 14 May 2024 07:55:30 +0000 (UTC) Authentication-Results: smtp.subspace.kernel.org; arc=none smtp.client-ip=203.254.224.25 ARC-Seal: i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1715673333; cv=none; b=P1Ge9/g8dJnfOzR2rfy3BuqNYDYp+zmMe0ZuPpOAJFJ9kbgcbvlCu/jAsJUjlmc/GzQ9jOuqo9iWcHocoxkhG/Z8G9QZoLsnmjvQnUshU79CAYoPPXH797dqXPvc/LjnTDf7UgamH5fH84QoSfhEevJxlI0tD/hroX9vFP3iHD4= ARC-Message-Signature: i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1715673333; c=relaxed/simple; bh=v98Mo+MpYBU0qzaAkf44WS1u6iYjnUbvkJq6cTR5f6I=; h=From:To:Cc:Subject:Date:Message-Id:In-Reply-To:MIME-Version: Content-Type:References; b=Tfv/YQd7NIdHXXMxtr+R4GRiyxAx4LM2AQPBQhrMkEoBmMi8/E74Xpl5lkrPCRDM0+kphZ0/Q2S+caPwZSrLFZkrFCe/RdjVMulGhbntTaNx3Adeuu1OhnsFg9nBmOoaWb7eGqiS7itk4bRgPQ16SbUKlLCfVc09D612PBwdzZs= ARC-Authentication-Results: i=1; smtp.subspace.kernel.org; dmarc=pass (p=none dis=none) header.from=samsung.com; spf=pass smtp.mailfrom=samsung.com; dkim=pass (1024-bit key) header.d=samsung.com header.i=@samsung.com header.b=uORn+RaH; arc=none smtp.client-ip=203.254.224.25 Authentication-Results: smtp.subspace.kernel.org; dmarc=pass (p=none dis=none) header.from=samsung.com Authentication-Results: smtp.subspace.kernel.org; spf=pass smtp.mailfrom=samsung.com Authentication-Results: smtp.subspace.kernel.org; dkim=pass (1024-bit key) header.d=samsung.com header.i=@samsung.com header.b="uORn+RaH" Received: from epcas5p1.samsung.com (unknown [182.195.41.39]) by mailout2.samsung.com (KnoxPortal) with ESMTP id 20240514075528epoutp021b5d9cbc36297b007e630ac7d08d997a~PS81ojYMc0977909779epoutp02C for ; Tue, 14 May 2024 07:55:28 +0000 (GMT) DKIM-Filter: OpenDKIM Filter v2.11.0 mailout2.samsung.com 20240514075528epoutp021b5d9cbc36297b007e630ac7d08d997a~PS81ojYMc0977909779epoutp02C DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=samsung.com; s=mail20170921; t=1715673328; bh=9J3qY+gr/EQFxqtdizhx5LUqyyO9iMd1OdFM5PunXLE=; h=From:To:Cc:Subject:Date:In-Reply-To:References:From; b=uORn+RaHFc7szSgjY9e//btRQiqCYpDrAoRZYtIK31AvMlD2UeFhJM/217YcynmNN ILqqZZWAphsHdyTtcpbKX7IMwQaI8+LbAARh5O2KliGFRnBZfo7MgOWwNpvu/JPpyw 85U4ZHjtcTGQJFAki90xRx0YWMhPNBTPmpc1hq1s= Received: from epsnrtp3.localdomain (unknown [182.195.42.164]) by epcas5p3.samsung.com (KnoxPortal) with ESMTP id 20240514075528epcas5p37f4625815093feb52c2e374775a2e29d~PS81UL8Hl1697316973epcas5p37; Tue, 14 May 2024 07:55:28 +0000 (GMT) Received: from epsmges5p2new.samsung.com (unknown [182.195.38.180]) by epsnrtp3.localdomain (Postfix) with ESMTP id 4VdpYm2rDvz4x9Q3; Tue, 14 May 2024 07:55:24 +0000 (GMT) Received: from epcas5p1.samsung.com ( [182.195.41.39]) by epsmges5p2new.samsung.com (Symantec Messaging Gateway) with SMTP id 25.60.09688.CE813466; Tue, 14 May 2024 16:55:24 +0900 (KST) Received: from epsmtrp1.samsung.com (unknown [182.195.40.13]) by epcas5p2.samsung.com (KnoxPortal) with ESMTPA id 20240514075459epcas5p2275b4c26f16bcfcea200e97fc75c2a14~PS8aLx8PN2941229412epcas5p2T; Tue, 14 May 2024 07:54:59 +0000 (GMT) Received: from epsmgmcp1.samsung.com (unknown [182.195.42.82]) by epsmtrp1.samsung.com (KnoxPortal) with ESMTP id 20240514075459epsmtrp11afbd5a5a33b6e0e9aef108e7fbb025e~PS8aLFGfD0539705397epsmtrp1D; Tue, 14 May 2024 07:54:59 +0000 (GMT) X-AuditID: b6c32a4a-837fa700000025d8-d6-664318ecba3d Received: from epsmtip2.samsung.com ( [182.195.34.31]) by epsmgmcp1.samsung.com (Symantec Messaging Gateway) with SMTP id CA.21.19234.3D813466; Tue, 14 May 2024 16:54:59 +0900 (KST) Received: from testpc118124.samsungds.net (unknown [109.105.118.124]) by epsmtip2.samsung.com (KnoxPortal) with ESMTPA id 20240514075458epsmtip239b56869a032fb1900d326b2babb20ab~PS8ZHSVVf0426704267epsmtip2F; Tue, 14 May 2024 07:54:57 +0000 (GMT) From: Chenliang Li To: axboe@kernel.dk, asml.silence@gmail.com Cc: io-uring@vger.kernel.org, peiwei.li@samsung.com, joshi.k@samsung.com, kundan.kumar@samsung.com, anuj20.g@samsung.com, gost.dev@samsung.com, Chenliang Li Subject: [PATCH v4 2/4] io_uring/rsrc: store folio shift and mask into imu Date: Tue, 14 May 2024 15:54:42 +0800 Message-Id: <20240514075444.590910-3-cliang01.li@samsung.com> X-Mailer: git-send-email 2.34.1 In-Reply-To: <20240514075444.590910-1-cliang01.li@samsung.com> Precedence: bulk X-Mailing-List: io-uring@vger.kernel.org List-Id: List-Subscribe: List-Unsubscribe: MIME-Version: 1.0 X-Brightmail-Tracker: H4sIAAAAAAAAA+NgFprDJsWRmVeSWpSXmKPExsWy7bCmuu4bCec0g7c/RSyaJvxltpizahuj xeq7/WwWp/8+ZrG4eWAnk8W71nMsFkf/v2Wz+NV9l9Fi65evrBbP9nJanJ3wgdWB22PnrLvs HpfPlnr0bVnF6PF5k1wAS1S2TUZqYkpqkUJqXnJ+SmZeuq2Sd3C8c7ypmYGhrqGlhbmSQl5i bqqtkotPgK5bZg7QTUoKZYk5pUChgMTiYiV9O5ui/NKSVIWM/OISW6XUgpScApMCveLE3OLS vHS9vNQSK0MDAyNToMKE7IxT61cwFzzjqei/1sjYwDibq4uRk0NCwERi69b/TF2MXBxCArsZ JZ71n2eDcD4xShyfOo8Vzjl16xczTMv65nZ2iMRORonuGbuZQBJCAr8YJbbMMACx2QR0JH6v +MUCYosIaEu8fjwVzGYW2MUosfCcFIgtLOAlce/sYlYQm0VAVeL7rNtgC3gFbCVO3r3DCLFM XmL/wbNAcQ4OTgE7iVMzXCBKBCVOznwCNVJeonnrbGaQeyQEvrJLnL42kwmi10Xi+Z8+Fghb WOLV8S3sELaUxMv+NnaQmRICxRLL1slB9LYwSrx/Nwdqr7XEvyt7WEBqmAU0Jdbv0ocIy0pM PbWOCWIvn0Tv7ydQq3gldsyDsVUlLhzcBrVKWmLthK3QcPOQaO64wQIJt4mMEksXXGabwKgw C8k/s5D8Mwth9QJG5lWMkqkFxbnpqcWmBUZ5qeXwSE7Oz93ECE6mWl47GB8++KB3iJGJg/EQ owQHs5IIr0OhfZoQb0piZVVqUX58UWlOavEhRlNgeE9klhJNzgem87ySeEMTSwMTMzMzE0tj M0Mlcd7XrXNThATSE0tSs1NTC1KLYPqYODilGpjCJuQxqIkbs8w9pWPZFef6s2t/3VFGBd59 EdHvlhye75vCkaCw5EW9u3lXVb9rQt32186yf3kkz+3/8VHjeteJe3WHwnTPNTw08Lia/Oaq 4lWWVNETZ9R5pk1XUDN0yiia8vq6hYfU/53OqvfP/T26cPIeDT6xORMmdlyycFgb4Gf191tz U/66PZN/y7XdCff8MamlM7qh1j1DvfieHG/q6fNCMbuXR+5vtL/HKTj5TtKujenvDPba8Kzm rFJ03pfEveeIm/eF4Gpdk6WVPPMLdrxe95HTVGzS4gOlpy+2c3y4JJW+PPP9Jfvpmx8lZ5yS OMejdTe6vTr1mexmXd7zItwXZN/cqnP1VZrXK6DEUpyRaKjFXFScCAD4XtX1LwQAAA== X-Brightmail-Tracker: H4sIAAAAAAAAA+NgFjrPLMWRmVeSWpSXmKPExsWy7bCSvO5lCec0g+vHmS2aJvxltpizahuj xeq7/WwWp/8+ZrG4eWAnk8W71nMsFkf/v2Wz+NV9l9Fi65evrBbP9nJanJ3wgdWB22PnrLvs HpfPlnr0bVnF6PF5k1wASxSXTUpqTmZZapG+XQJXxqn1K5gLnvFU9F9rZGxgnM3VxcjJISFg IrG+uZ29i5GLQ0hgO6PE86UtjBAJaYmOQ63sELawxMp/z6GKfjBKnOmexwySYBPQkfi94hdL FyMHh4iArkTjXQWQGmaBQ4wSzRuawQYJC3hJ3Du7mBXEZhFQlfg+6zZYL6+ArcTJu3eglslL 7D94lhlkDqeAncSpGS4gYSGgks1/jrNDlAtKnJz5hAXEZgYqb946m3kCo8AsJKlZSFILGJlW MYqmFhTnpucmFxjqFSfmFpfmpesl5+duYgSHuVbQDsZl6//qHWJk4mAEupeDWUmE16HQPk2I NyWxsiq1KD++qDQntfgQozQHi5I4r3JOZ4qQQHpiSWp2ampBahFMlomDU6qBqUnqka37yS3i P41XlAg3dr4xuPGntu3L/Kk+ky5K7Z72RMrwp+2S1ed4HV0+iEZy9vBWNjvXaG19dLh/iZHp 2YkXuXS9Vk+646jcn3E8bPUFnanztwdkulvUNzx+8tFOoD5grVwJ2xLp6edKtmtyR616+WrO ncMHO94yb3z3umfhvpmFFXs8fIs+LzzNe/5f+tTjdVUNTYtPWU3PWCe4M70gKeig7I7A8g06 0zOOhC4WbPqy08tZV31q/5bHloH+GY9WcRTeXCMdeOODzuJq+WvyZx2XMjaY3PijcFxC/YWi h6e7alqLi8b7tqrvZyb+KdN6b5y5UNhoVtq/TaEy7WY7nm7efMn0GvtNeQPDDCWW4oxEQy3m ouJEAFGVSRjiAgAA X-CMS-MailID: 20240514075459epcas5p2275b4c26f16bcfcea200e97fc75c2a14 X-Msg-Generator: CA X-Sendblock-Type: REQ_APPROVE CMS-TYPE: 105P DLP-Filter: Pass X-CFilter-Loop: Reflected X-CMS-RootMailID: 20240514075459epcas5p2275b4c26f16bcfcea200e97fc75c2a14 References: <20240514075444.590910-1-cliang01.li@samsung.com> Store the folio shift and folio mask into imu struct and use it in iov_iter adjust, as we will have non PAGE_SIZE'd chunks if a multi-hugepage buffer get coalesced. Signed-off-by: Chenliang Li Reviewed-by: Anuj Gupta --- io_uring/rsrc.c | 6 ++++-- io_uring/rsrc.h | 2 ++ 2 files changed, 6 insertions(+), 2 deletions(-) diff --git a/io_uring/rsrc.c b/io_uring/rsrc.c index d08224c0c5b0..578d382ca9bc 100644 --- a/io_uring/rsrc.c +++ b/io_uring/rsrc.c @@ -1015,6 +1015,8 @@ static int io_sqe_buffer_register(struct io_ring_ctx *ctx, struct iovec *iov, imu->ubuf = (unsigned long) iov->iov_base; imu->ubuf_end = imu->ubuf + iov->iov_len; imu->nr_bvecs = nr_pages; + imu->folio_shift = PAGE_SHIFT; + imu->folio_mask = PAGE_MASK; *pimu = imu; ret = 0; @@ -1153,12 +1155,12 @@ int io_import_fixed(int ddir, struct iov_iter *iter, /* skip first vec */ offset -= bvec->bv_len; - seg_skip = 1 + (offset >> PAGE_SHIFT); + seg_skip = 1 + (offset >> imu->folio_shift); iter->bvec = bvec + seg_skip; iter->nr_segs -= seg_skip; iter->count -= bvec->bv_len + offset; - iter->iov_offset = offset & ~PAGE_MASK; + iter->iov_offset = offset & ~imu->folio_mask; } } diff --git a/io_uring/rsrc.h b/io_uring/rsrc.h index b2a9d66b76dd..93da02e652bc 100644 --- a/io_uring/rsrc.h +++ b/io_uring/rsrc.h @@ -46,7 +46,9 @@ struct io_mapped_ubuf { u64 ubuf; u64 ubuf_end; unsigned int nr_bvecs; + unsigned int folio_shift; unsigned long acct_pages; + unsigned long folio_mask; struct bio_vec bvec[] __counted_by(nr_bvecs); }; From patchwork Tue May 14 07:54:43 2024 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Chenliang Li X-Patchwork-Id: 13663845 Received: from mailout4.samsung.com (mailout4.samsung.com [203.254.224.34]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (No client certificate requested) by smtp.subspace.kernel.org (Postfix) with ESMTPS id 530F917573 for ; Tue, 14 May 2024 07:55:31 +0000 (UTC) Authentication-Results: smtp.subspace.kernel.org; arc=none smtp.client-ip=203.254.224.34 ARC-Seal: i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1715673334; cv=none; b=BDQusGr7Ygf0nRiHVSvkbPif3lodm+64KQpBNGXq0dd/lBwHcJNx7PpbfHuYL0dfUs7TKoVRoyQGNiG7cgPG+iolN9CXfpEXyET73uCtKtBniagMaa2RNt7D4O0Zb0xrYj3dxT3S5CCTQYmMRnhkiuLEmqFvzUOc7Q8xidnf/Kk= ARC-Message-Signature: i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1715673334; c=relaxed/simple; bh=L04S/T0HduAOk5jic8D4zPDYPUsOlWlq6z5J2Sfh0tI=; h=From:To:Cc:Subject:Date:Message-Id:In-Reply-To:MIME-Version: Content-Type:References; b=EekQ/2+EpluMUMo3nzrTODW92Yk2UhgQ73WvzfGW8/GbldljthS5AQHn7Q+7oACJTYCBsjMt0J/o0PTwEwZfxIgRvLdoJZATFY5EsCOCETI3WosG/S5RV12dBDk61l/zrKrMqE0dDLaB0wI2shhMoUnKhS2c3psMt0CnvpT4osg= ARC-Authentication-Results: i=1; smtp.subspace.kernel.org; dmarc=pass (p=none dis=none) header.from=samsung.com; spf=pass smtp.mailfrom=samsung.com; dkim=pass (1024-bit key) header.d=samsung.com header.i=@samsung.com header.b=plwAyrfs; arc=none smtp.client-ip=203.254.224.34 Authentication-Results: smtp.subspace.kernel.org; dmarc=pass (p=none dis=none) header.from=samsung.com Authentication-Results: smtp.subspace.kernel.org; spf=pass smtp.mailfrom=samsung.com Authentication-Results: smtp.subspace.kernel.org; dkim=pass (1024-bit key) header.d=samsung.com header.i=@samsung.com header.b="plwAyrfs" Received: from epcas5p2.samsung.com (unknown [182.195.41.40]) by mailout4.samsung.com (KnoxPortal) with ESMTP id 20240514075528epoutp047337c9e057c5977de4488060663a1514~PS815ssjR0538505385epoutp04v for ; Tue, 14 May 2024 07:55:28 +0000 (GMT) DKIM-Filter: OpenDKIM Filter v2.11.0 mailout4.samsung.com 20240514075528epoutp047337c9e057c5977de4488060663a1514~PS815ssjR0538505385epoutp04v DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=samsung.com; s=mail20170921; t=1715673328; bh=oXAeKS3l9SfB9JAX17XgdEuqjbyD8WlhE++s0l1R6lk=; h=From:To:Cc:Subject:Date:In-Reply-To:References:From; b=plwAyrfsKEEDwwJJ6JSbiVTUi/3oKlCf1RzVFIFqnJftTpWZhy/7uVEC4pz9ovXlQ gTbjJctXTsTc7/q786WvplQLV8VybHeK3PUQ37Itn8T/4kHGooMBi2jRgZjwiq0aZt U15MRoNp6dKLXKTPYFO3gkYW0qx7mP9x3oC9G8qg= Received: from epsnrtp1.localdomain (unknown [182.195.42.162]) by epcas5p2.samsung.com (KnoxPortal) with ESMTP id 20240514075528epcas5p24d2be71090b67c4a5e9fb751d6e09afd~PS81eAVwv0343203432epcas5p2q; Tue, 14 May 2024 07:55:28 +0000 (GMT) Received: from epsmgec5p1new.samsung.com (unknown [182.195.38.177]) by epsnrtp1.localdomain (Postfix) with ESMTP id 4VdpYp5LMfz4x9Q5; Tue, 14 May 2024 07:55:26 +0000 (GMT) Received: from epcas5p1.samsung.com ( [182.195.41.39]) by epsmgec5p1new.samsung.com (Symantec Messaging Gateway) with SMTP id 00.89.08600.EE813466; Tue, 14 May 2024 16:55:26 +0900 (KST) Received: from epsmtrp1.samsung.com (unknown [182.195.40.13]) by epcas5p1.samsung.com (KnoxPortal) with ESMTPA id 20240514075500epcas5p1e638b1ae84727b3669ff6b780cd1cb23~PS8buNdDm3182431824epcas5p1U; Tue, 14 May 2024 07:55:00 +0000 (GMT) Received: from epsmgmc1p1new.samsung.com (unknown [182.195.42.40]) by epsmtrp1.samsung.com (KnoxPortal) with ESMTP id 20240514075500epsmtrp16c874b7d5d0c1bce4dd02e1fc2ff1120~PS8bs6F5c0539705397epsmtrp1P; Tue, 14 May 2024 07:55:00 +0000 (GMT) X-AuditID: b6c32a44-6c3ff70000002198-7d-664318ee5c16 Received: from epsmtip2.samsung.com ( [182.195.34.31]) by epsmgmc1p1new.samsung.com (Symantec Messaging Gateway) with SMTP id 2C.98.09238.4D813466; Tue, 14 May 2024 16:55:00 +0900 (KST) Received: from testpc118124.samsungds.net (unknown [109.105.118.124]) by epsmtip2.samsung.com (KnoxPortal) with ESMTPA id 20240514075459epsmtip2ec94584c30746edf7a971605fabf9618~PS8arXGLb0428404284epsmtip2G; Tue, 14 May 2024 07:54:59 +0000 (GMT) From: Chenliang Li To: axboe@kernel.dk, asml.silence@gmail.com Cc: io-uring@vger.kernel.org, peiwei.li@samsung.com, joshi.k@samsung.com, kundan.kumar@samsung.com, anuj20.g@samsung.com, gost.dev@samsung.com, Chenliang Li Subject: [PATCH v4 3/4] io_uring/rsrc: add init and account functions for coalesced imus Date: Tue, 14 May 2024 15:54:43 +0800 Message-Id: <20240514075444.590910-4-cliang01.li@samsung.com> X-Mailer: git-send-email 2.34.1 In-Reply-To: <20240514075444.590910-1-cliang01.li@samsung.com> Precedence: bulk X-Mailing-List: io-uring@vger.kernel.org List-Id: List-Subscribe: List-Unsubscribe: MIME-Version: 1.0 X-Brightmail-Tracker: H4sIAAAAAAAAA+NgFprNJsWRmVeSWpSXmKPExsWy7bCmuu47Cec0gwddWhZNE/4yW8xZtY3R YvXdfjaL038fs1jcPLCTyeJd6zkWi6P/37JZ/Oq+y2ix9ctXVotnezktzk74wOrA7bFz1l12 j8tnSz36tqxi9Pi8SS6AJSrbJiM1MSW1SCE1Lzk/JTMv3VbJOzjeOd7UzMBQ19DSwlxJIS8x N9VWycUnQNctMwfoJiWFssScUqBQQGJxsZK+nU1RfmlJqkJGfnGJrVJqQUpOgUmBXnFibnFp XrpeXmqJlaGBgZEpUGFCdsbphwvYCp5LVaz9spG9gXGOaBcjJ4eEgIlE4+d9zF2MXBxCArsZ Je5Om8cG4XxilNg8/wYThPONUWLdrh9sMC1rT85kgUjsZZS4PG0NK0hCSOAXo8SU7UwgNpuA jsTvFb9YQGwRAW2J14+ngtnMArsYJRaekwKxhQWiJPbc3snexcjBwSKgKjHtKB9ImFfAVuLM iwlMELvkJfYfPMsMUsIpYCdxaoYLRImgxMmZT6Amyks0b50N9oGEwF92iYs/XjJD9LpIzPk/ hxXCFpZ4dXwLO4QtJfGyvw1srYRAscSydXIQvS2MEu/fzWGEqLGW+HdlDwtIDbOApsT6XfoQ YVmJqafWMUHs5ZPo/f0E6kxeiR3zYGxViQsHt0GtkpZYO2Er1DkeEl39jayQYJsIDLYV15gm MCrMQvLPLCT/zEJYvYCReRWjZGpBcW56arJpgWFeajk8jpPzczcxglOplssOxhvz/+kdYmTi YDzEKMHBrCTC61BonybEm5JYWZValB9fVJqTWnyI0RQY3BOZpUST84HJPK8k3tDE0sDEzMzM xNLYzFBJnPd169wUIYH0xJLU7NTUgtQimD4mDk6pBqbcZRvDNBjt3hm/m9xw5JnLknSnq9fY vjFfjSmvM1eYx9p6LHPmDz6rldN7SqqFDG2WqKmcLlX2P3Aj8Tf3Z49nJssmfuNMDYjcO89E UOAX//NjZ8P8P5pzbO5ok9xdpLv81F9TzYSZmdJXGPn4VsQ/UL35f/J/Ju3vj3/0SwmyLM9l Pfbs+fE1D+1zv/3+bsf+7kf4Rd7wIxqJ3HPa1u+eq1invZ7n/oob6dNi4xkrjd40z7DOCf20 2q1U7Un81pWi3NVsn7gddlr9b2Npf9KxzeVe7z4Xz5M3gqcHBU5d8bJJxKp97aWTXZN2hHwV /rfpinvALu4n6jPjtCsM657MLZjhKfFtXqVcWwnHFGclluKMREMt5qLiRABOqAyuLgQAAA== X-Brightmail-Tracker: H4sIAAAAAAAAA+NgFrrILMWRmVeSWpSXmKPExsWy7bCSvO4VCec0gx0T9CyaJvxltpizahuj xeq7/WwWp/8+ZrG4eWAnk8W71nMsFkf/v2Wz+NV9l9Fi65evrBbP9nJanJ3wgdWB22PnrLvs HpfPlnr0bVnF6PF5k1wASxSXTUpqTmZZapG+XQJXxumHC9gKnktVrP2ykb2BcY5oFyMnh4SA icTakzNZuhi5OIQEdjNKXHl2gx0iIS3RcagVyhaWWPnvOZgtJPCDUeL2bX8Qm01AR+L3il9A zRwcIgK6Eo13FUDmMAscYpRo3tDMCFIjLBAh8eD3KkaQGhYBVYlpR/lAwrwCthJnXkxgghgv L7H/4FlmkBJOATuJUzNcIDbZSmz+c5wdolxQ4uTMJywgNjNQefPW2cwTGAVmIUnNQpJawMi0 ilEytaA4Nz032bDAMC+1XK84Mbe4NC9dLzk/dxMjONS1NHYw3pv/T+8QIxMHI9DJHMxKIrwO hfZpQrwpiZVVqUX58UWlOanFhxilOViUxHkNZ8xOERJITyxJzU5NLUgtgskycXBKNTA92rUy v36dZkDrd7lX/C9PTH4nJnnjhYFmMmO0L0shs5fitJvth15Wcnc1HXmYfWl+1+nyY4s9o3uf xz2/YLSrgV9u28bwVZ8+mLZ2M4f61PbNE7U/X/Zje9uxjCkiatM4TV4Jy1ltiOV7wnXs+xcR r88Cfm1ykWd3sL5Yc+j8lD125WLPph1eb7F4K5tubvLZv0YJh0Xm8zSd+2a5eUWeW9HE9IAj /p7rTexTFPdv/eRlcH7XvxfboozyGyYVmBkrvOV4YFckcve+JuuzP1dFuXIbmKLapr/gztaI WCvNoJaWcXrO5uJbq/etbtna2lFU3ZtR+XH5xI1b0ycuWH5X9cu8t4tj1JbX7KyvebBeiaU4 I9FQi7moOBEAn+UGu+QCAAA= X-CMS-MailID: 20240514075500epcas5p1e638b1ae84727b3669ff6b780cd1cb23 X-Msg-Generator: CA X-Sendblock-Type: REQ_APPROVE CMS-TYPE: 105P DLP-Filter: Pass X-CFilter-Loop: Reflected X-CMS-RootMailID: 20240514075500epcas5p1e638b1ae84727b3669ff6b780cd1cb23 References: <20240514075444.590910-1-cliang01.li@samsung.com> Introduce two functions to separate the coalesced imu alloc and accounting path from the original one. This helps to keep the original code path clean. Signed-off-by: Chenliang Li --- io_uring/rsrc.c | 89 +++++++++++++++++++++++++++++++++++++++++++++++++ 1 file changed, 89 insertions(+) diff --git a/io_uring/rsrc.c b/io_uring/rsrc.c index 578d382ca9bc..53fac5f27bbf 100644 --- a/io_uring/rsrc.c +++ b/io_uring/rsrc.c @@ -871,6 +871,45 @@ static int io_buffer_account_pin(struct io_ring_ctx *ctx, struct page **pages, return ret; } +static int io_coalesced_buffer_account_pin(struct io_ring_ctx *ctx, + struct page **pages, + struct io_mapped_ubuf *imu, + struct page **last_hpage, + struct io_imu_folio_data *data) +{ + int i, j, ret; + + imu->acct_pages = 0; + j = 0; + for (i = 0; i < data->nr_folios; i++) { + struct page *hpage = pages[j]; + + if (hpage == *last_hpage) + continue; + *last_hpage = hpage; + /* + * Already checked the page array in try coalesce, + * so pass in nr_pages=0 here to waive that. + */ + if (headpage_already_acct(ctx, pages, 0, hpage)) + continue; + imu->acct_pages += data->nr_pages_mid; + if (i) + j += data->nr_pages_mid; + else + j = data->nr_pages_head; + } + + if (!imu->acct_pages) + return 0; + + ret = io_account_mem(ctx, imu->acct_pages); + if (!ret) + return 0; + imu->acct_pages = 0; + return ret; +} + static bool __io_sqe_buffer_try_coalesce(struct page **pages, int nr_pages, struct io_imu_folio_data *data) { @@ -949,6 +988,56 @@ static bool io_sqe_buffer_try_coalesce(struct page **pages, int nr_pages, return true; } +static int io_coalesced_imu_alloc(struct io_ring_ctx *ctx, struct iovec *iov, + struct io_mapped_ubuf **pimu, + struct page **last_hpage, struct page **pages, + struct io_imu_folio_data *data) +{ + struct io_mapped_ubuf *imu = NULL; + unsigned long off; + size_t size, vec_len; + int ret, i, j; + + ret = -ENOMEM; + imu = kvmalloc(struct_size(imu, bvec, data->nr_folios), GFP_KERNEL); + if (!imu) + return ret; + + ret = io_coalesced_buffer_account_pin(ctx, pages, imu, last_hpage, + data); + if (ret) { + unpin_user_page(pages[0]); + j = data->nr_pages_head; + for (i = 1; i < data->nr_folios; i++) { + unpin_user_page(pages[j]); + j += data->nr_pages_mid; + } + return ret; + } + off = (unsigned long) iov->iov_base & ~PAGE_MASK; + size = iov->iov_len; + /* store original address for later verification */ + imu->ubuf = (unsigned long) iov->iov_base; + imu->ubuf_end = imu->ubuf + iov->iov_len; + imu->nr_bvecs = data->nr_folios; + imu->folio_shift = data->folio_shift; + imu->folio_mask = ~((1UL << data->folio_shift) - 1); + *pimu = imu; + ret = 0; + + vec_len = min_t(size_t, size, PAGE_SIZE * data->nr_pages_head - off); + bvec_set_page(&imu->bvec[0], pages[0], vec_len, off); + size -= vec_len; + j = data->nr_pages_head; + for (i = 1; i < data->nr_folios; i++) { + vec_len = min_t(size_t, size, data->folio_size); + bvec_set_page(&imu->bvec[i], pages[j], vec_len, 0); + size -= vec_len; + j += data->nr_pages_mid; + } + return ret; +} + static int io_sqe_buffer_register(struct io_ring_ctx *ctx, struct iovec *iov, struct io_mapped_ubuf **pimu, struct page **last_hpage) From patchwork Tue May 14 07:54:44 2024 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Chenliang Li X-Patchwork-Id: 13663846 Received: from mailout3.samsung.com (mailout3.samsung.com [203.254.224.33]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (No client certificate requested) by smtp.subspace.kernel.org (Postfix) with ESMTPS id EE9B314A8F for ; Tue, 14 May 2024 07:55:40 +0000 (UTC) Authentication-Results: smtp.subspace.kernel.org; arc=none smtp.client-ip=203.254.224.33 ARC-Seal: i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1715673343; cv=none; b=PeKCdub4YGNq3hU+o8dz/ECpg/OzCx0US8bLZ9FxwuCB+yfj6Q4vOed8N5pY/dYNd1jtR+NFfYeeEMXEQcyT3/NtsEmMHih25cAlfPhqFxgZKs+NpPXZgkP+yF6jMEP+dreN8x5Nazz3xizaeZCUrvj2+RuJtWnlTvyLx9JwMgY= ARC-Message-Signature: i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1715673343; c=relaxed/simple; bh=/pquvSSb5hWMFri5I9lPidd/KPmZagrm0CwHv1irRLU=; h=From:To:Cc:Subject:Date:Message-Id:In-Reply-To:MIME-Version: Content-Type:References; b=tG6W4B8ja4x2TKa2+g7DjikcY03k6UIQOHZFDWBcZppHxO+OAGOmZFIUSPSx6e4apE5S5PP577M7IIoXGLqS7tZbLG6qYnZT3hIU93u8d3FqMYUqkBzZxOxJHLzfe1zGsl23UfPjvZbwqNX0wFdDhiFtbtx0wS2rzs7yo/RRL8E= ARC-Authentication-Results: i=1; smtp.subspace.kernel.org; dmarc=pass (p=none dis=none) header.from=samsung.com; spf=pass smtp.mailfrom=samsung.com; dkim=pass (1024-bit key) header.d=samsung.com header.i=@samsung.com header.b=imRHklgL; arc=none smtp.client-ip=203.254.224.33 Authentication-Results: smtp.subspace.kernel.org; dmarc=pass (p=none dis=none) header.from=samsung.com Authentication-Results: smtp.subspace.kernel.org; spf=pass smtp.mailfrom=samsung.com Authentication-Results: smtp.subspace.kernel.org; dkim=pass (1024-bit key) header.d=samsung.com header.i=@samsung.com header.b="imRHklgL" Received: from epcas5p3.samsung.com (unknown [182.195.41.41]) by mailout3.samsung.com (KnoxPortal) with ESMTP id 20240514075532epoutp0307e40085a83805a8130a844e0dba8030~PS85PoSeY1861718617epoutp03B for ; Tue, 14 May 2024 07:55:32 +0000 (GMT) DKIM-Filter: OpenDKIM Filter v2.11.0 mailout3.samsung.com 20240514075532epoutp0307e40085a83805a8130a844e0dba8030~PS85PoSeY1861718617epoutp03B DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=samsung.com; s=mail20170921; t=1715673332; bh=qprTvAGOCcQg9gi5Hb8YrQlfhdJvrh5dgetkxuBjKRI=; h=From:To:Cc:Subject:Date:In-Reply-To:References:From; b=imRHklgL9wqav+9xhhTsV+5w2OHBwbMtk0x7GVPMmmN+C3kdmP2GJxApxaYL/3BjO weGqnkobxGjNIaN8HQsbnScAnktlGcF3kucSt/AQY6LZP1kschbiRTdzEpfOaUBBj2 i6XA3E//qelpJQ9v5VNNhsZF20GYo0Rj3jFDjL5M= Received: from epsnrtp4.localdomain (unknown [182.195.42.165]) by epcas5p1.samsung.com (KnoxPortal) with ESMTP id 20240514075531epcas5p10a4b7b3c752d14ba6f6a48a366c6d028~PS84lIaTr3203932039epcas5p1H; Tue, 14 May 2024 07:55:31 +0000 (GMT) Received: from epsmgec5p1-new.samsung.com (unknown [182.195.38.177]) by epsnrtp4.localdomain (Postfix) with ESMTP id 4VdpYs272mz4x9QF; Tue, 14 May 2024 07:55:29 +0000 (GMT) Received: from epcas5p1.samsung.com ( [182.195.41.39]) by epsmgec5p1-new.samsung.com (Symantec Messaging Gateway) with SMTP id E4.E4.19431.1F813466; Tue, 14 May 2024 16:55:29 +0900 (KST) Received: from epsmtrp1.samsung.com (unknown [182.195.40.13]) by epcas5p1.samsung.com (KnoxPortal) with ESMTPA id 20240514075502epcas5p10be6bef71d284a110277575d6008563d~PS8dWW0gd3182031820epcas5p1f; Tue, 14 May 2024 07:55:02 +0000 (GMT) Received: from epsmgmcp1.samsung.com (unknown [182.195.42.82]) by epsmtrp1.samsung.com (KnoxPortal) with ESMTP id 20240514075502epsmtrp1ca786515251d91188ae854a15651b675~PS8dVb15A0538105381epsmtrp1w; Tue, 14 May 2024 07:55:02 +0000 (GMT) X-AuditID: b6c32a50-ccbff70000004be7-4c-664318f1572c Received: from epsmtip2.samsung.com ( [182.195.34.31]) by epsmgmcp1.samsung.com (Symantec Messaging Gateway) with SMTP id 8F.21.19234.6D813466; Tue, 14 May 2024 16:55:02 +0900 (KST) Received: from testpc118124.samsungds.net (unknown [109.105.118.124]) by epsmtip2.samsung.com (KnoxPortal) with ESMTPA id 20240514075501epsmtip2625cd05c8015d1fdac432ea955422ac3~PS8cQ21lN0438604386epsmtip2F; Tue, 14 May 2024 07:55:01 +0000 (GMT) From: Chenliang Li To: axboe@kernel.dk, asml.silence@gmail.com Cc: io-uring@vger.kernel.org, peiwei.li@samsung.com, joshi.k@samsung.com, kundan.kumar@samsung.com, anuj20.g@samsung.com, gost.dev@samsung.com, Chenliang Li Subject: [PATCH v4 4/4] io_uring/rsrc: enable multi-hugepage buffer coalescing Date: Tue, 14 May 2024 15:54:44 +0800 Message-Id: <20240514075444.590910-5-cliang01.li@samsung.com> X-Mailer: git-send-email 2.34.1 In-Reply-To: <20240514075444.590910-1-cliang01.li@samsung.com> Precedence: bulk X-Mailing-List: io-uring@vger.kernel.org List-Id: List-Subscribe: List-Unsubscribe: MIME-Version: 1.0 X-Brightmail-Tracker: H4sIAAAAAAAAA+NgFprLJsWRmVeSWpSXmKPExsWy7bCmuu5HCec0g/1/uC2aJvxltpizahuj xeq7/WwWp/8+ZrG4eWAnk8W71nMsFkf/v2Wz+NV9l9Fi65evrBbP9nJanJ3wgdWB22PnrLvs HpfPlnr0bVnF6PF5k1wAS1S2TUZqYkpqkUJqXnJ+SmZeuq2Sd3C8c7ypmYGhrqGlhbmSQl5i bqqtkotPgK5bZg7QTUoKZYk5pUChgMTiYiV9O5ui/NKSVIWM/OISW6XUgpScApMCveLE3OLS vHS9vNQSK0MDAyNToMKE7IzpR/+wFTySqDiyby1bA+MtoS5GTg4JAROJ05MmsnQxcnEICexh lLh5bSY7hPOJUeLZ1VuscM6rkyeYYVo2rngDVbWTUeJn10tGCOcXo8SP7x8YQarYBHQkfq/4 xQJiiwhoS7x+PBXMZhbYxSix8JwUiC0sECBx70YTWJxFQFVi+6oOJhCbV8BW4vGDL6wQ2+Ql 9h88C7SZg4NTwE7i1AwXiBJBiZMzn0CNlJdo3job6riv7BJrp6VD2C4S/x9/gYoLS7w6voUd wpaSeNnfxg4yUkKgWGLZOjmQ8yUEWhgl3r+bwwhRYy3x78oeFpAaZgFNifW79CHCshJTT61j gljLJ9H7+wkTRJxXYsc8GFtV4sLBbVCrpCXWTtgKdYKHxOYuWIBOBIbbsy0sExgVZiF5ZxaS d2YhrF7AyLyKUSq1oDg3PTXZtMBQNy+1HB7Lyfm5mxjB6VQrYAfj6g1/9Q4xMnEwHmKU4GBW EuF1KLRPE+JNSaysSi3Kjy8qzUktPsRoCgzvicxSosn5wISeVxJvaGJpYGJmZmZiaWxmqCTO +7p1boqQQHpiSWp2ampBahFMHxMHp1QDU0+G86IZ7g0cR4vOhe3uqPm2y3rrBomuPLew5De5 ydKPE9fPVtpZ/VWj42zT1BRJpi6RJlmfO38v1BRpu3PcKGq2ee7xzGfTgitvljpZbPI3Yvv1 TlnF9HLizsTj7z+r9E6otdqkLb9U8U3QdqntX4oO6XrbnM5yOKIgyP3seYXzw9OJO06FrvZ9 t3Wrbht36sTU6PPHVZUr1/2fWWDyP4jvsa7glb0dja8ObrwTpnsqpof5u2W87b1LGodV03Pj V1zX60l+FvSum/n0tS1XnK0bDv+qY72V2+elmPRqqbt1QeQraYPgiVybti++u/fNiiBf9n/v 5k3OfPpYmzXs95fvn5asDD1WsumwndXGcCWW4oxEQy3mouJEAOQjrMwwBAAA X-Brightmail-Tracker: H4sIAAAAAAAAA+NgFjrPLMWRmVeSWpSXmKPExsWy7bCSvO41Cec0g0md0hZNE/4yW8xZtY3R YvXdfjaL038fs1jcPLCTyeJd6zkWi6P/37JZ/Oq+y2ix9ctXVotnezktzk74wOrA7bFz1l12 j8tnSz36tqxi9Pi8SS6AJYrLJiU1J7MstUjfLoErY/rRP2wFjyQqjuxby9bAeEuoi5GTQ0LA RGLjijfsXYxcHEIC2xklHm3fxQSRkJboONTKDmELS6z89xyq6AejRO+2Y8wgCTYBHYnfK36x dDFycIgI6Eo03lUAqWEWOMQo0byhmRGkRljAT+LX1jlg9SwCqhLbV3WALeAVsJV4/OALK8QC eYn9B88yg8zhFLCTODXDBSQsBFSy+c9xdohyQYmTM5+wgNjMQOXNW2czT2AUmIUkNQtJagEj 0ypG0dSC4tz03OQCQ73ixNzi0rx0veT83E2M4DDXCtrBuGz9X71DjEwcjED3cjArifA6FNqn CfGmJFZWpRblxxeV5qQWH2KU5mBREudVzulMERJITyxJzU5NLUgtgskycXBKNTBFB32XMMq7 kfAzJm1TPZdIl6TAiu3bg+o8DHOEvjjMZD+74GbiIrmWW9fXeDacO6xudZ5h+7dvX9b3Hlbq 0z3rKK59VPBXtmPJ870MXga+N+4sWbgjpEPWaeXWjBwVnu4rSguF95+uX5jTrHtBolRjQ8zF TU/X7TW96zdrvrXZqSweSf74iTvX3o0T0Vt7L/HBZP/NH22fH8kPuiUtKskpzd3GcKzsfHLY OomFK6VDguPYkuNC/OdI1u38zRixe45TybH9/Hav5a8syr4lYFYfoSFcH5TEtSi67sV2ERut vXrfPy19G69ftMvGcq73ttdXg9iZy9hLtuW8vOYhlXJ1uY920cYiz+B82QImeSWW4oxEQy3m ouJEAEi01KfiAgAA X-CMS-MailID: 20240514075502epcas5p10be6bef71d284a110277575d6008563d X-Msg-Generator: CA X-Sendblock-Type: REQ_APPROVE CMS-TYPE: 105P DLP-Filter: Pass X-CFilter-Loop: Reflected X-CMS-RootMailID: 20240514075502epcas5p10be6bef71d284a110277575d6008563d References: <20240514075444.590910-1-cliang01.li@samsung.com> Modify the original buffer registration and enable the coalescing for buffers with more than one hugepages. Signed-off-by: Chenliang Li Reviewed-by: Anuj Gupta --- io_uring/rsrc.c | 44 ++++++++------------------------------------ 1 file changed, 8 insertions(+), 36 deletions(-) diff --git a/io_uring/rsrc.c b/io_uring/rsrc.c index 53fac5f27bbf..5e5c1d6f3501 100644 --- a/io_uring/rsrc.c +++ b/io_uring/rsrc.c @@ -1047,7 +1047,7 @@ static int io_sqe_buffer_register(struct io_ring_ctx *ctx, struct iovec *iov, unsigned long off; size_t size; int ret, nr_pages, i; - struct folio *folio = NULL; + struct io_imu_folio_data data; *pimu = (struct io_mapped_ubuf *)&dummy_ubuf; if (!iov->iov_base) @@ -1062,30 +1062,11 @@ static int io_sqe_buffer_register(struct io_ring_ctx *ctx, struct iovec *iov, goto done; } - /* If it's a huge page, try to coalesce them into a single bvec entry */ - if (nr_pages > 1) { - folio = page_folio(pages[0]); - for (i = 1; i < nr_pages; i++) { - /* - * Pages must be consecutive and on the same folio for - * this to work - */ - if (page_folio(pages[i]) != folio || - pages[i] != pages[i - 1] + 1) { - folio = NULL; - break; - } - } - if (folio) { - /* - * The pages are bound to the folio, it doesn't - * actually unpin them but drops all but one reference, - * which is usually put down by io_buffer_unmap(). - * Note, needs a better helper. - */ - unpin_user_pages(&pages[1], nr_pages - 1); - nr_pages = 1; - } + /* If it's huge page(s), try to coalesce them into fewer bvec entries */ + if (io_sqe_buffer_try_coalesce(pages, nr_pages, &data)) { + ret = io_coalesced_imu_alloc(ctx, iov, pimu, last_hpage, + pages, &data); + goto done; } imu = kvmalloc(struct_size(imu, bvec, nr_pages), GFP_KERNEL); @@ -1109,10 +1090,6 @@ static int io_sqe_buffer_register(struct io_ring_ctx *ctx, struct iovec *iov, *pimu = imu; ret = 0; - if (folio) { - bvec_set_page(&imu->bvec[0], pages[0], size, off); - goto done; - } for (i = 0; i < nr_pages; i++) { size_t vec_len; @@ -1218,23 +1195,18 @@ int io_import_fixed(int ddir, struct iov_iter *iter, * we know that: * * 1) it's a BVEC iter, we set it up - * 2) all bvecs are PAGE_SIZE in size, except potentially the + * 2) all bvecs are the same in size, except potentially the * first and last bvec * * So just find our index, and adjust the iterator afterwards. * If the offset is within the first bvec (or the whole first * bvec, just use iov_iter_advance(). This makes it easier * since we can just skip the first segment, which may not - * be PAGE_SIZE aligned. + * be folio_size aligned. */ const struct bio_vec *bvec = imu->bvec; if (offset < bvec->bv_len) { - /* - * Note, huge pages buffers consists of one large - * bvec entry and should always go this way. The other - * branch doesn't expect non PAGE_SIZE'd chunks. - */ iter->bvec = bvec; iter->nr_segs = bvec->bv_len; iter->count -= offset;