From patchwork Sun Aug 7 18:36:04 2022 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Kanchan Joshi X-Patchwork-Id: 12938241 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from vger.kernel.org (vger.kernel.org [23.128.96.18]) by smtp.lore.kernel.org (Postfix) with ESMTP id C423BC25B0C for ; Sun, 7 Aug 2022 18:45:56 +0000 (UTC) Received: (majordomo@vger.kernel.org) by vger.kernel.org via listexpand id S235497AbiHGSp4 (ORCPT ); Sun, 7 Aug 2022 14:45:56 -0400 Received: from lindbergh.monkeyblade.net ([23.128.96.19]:38356 "EHLO lindbergh.monkeyblade.net" rhost-flags-OK-OK-OK-OK) by vger.kernel.org with ESMTP id S232582AbiHGSpz (ORCPT ); Sun, 7 Aug 2022 14:45:55 -0400 Received: from mailout3.samsung.com (mailout3.samsung.com [203.254.224.33]) by lindbergh.monkeyblade.net (Postfix) with ESMTPS id 777EE958D for ; Sun, 7 Aug 2022 11:45:53 -0700 (PDT) Received: from epcas5p1.samsung.com (unknown [182.195.41.39]) by mailout3.samsung.com (KnoxPortal) with ESMTP id 20220807184548epoutp035eb2013d367aa7f15e20c0a7a5585b69~JJGO1CqYI2485424854epoutp03O for ; Sun, 7 Aug 2022 18:45:48 +0000 (GMT) DKIM-Filter: OpenDKIM Filter v2.11.0 mailout3.samsung.com 20220807184548epoutp035eb2013d367aa7f15e20c0a7a5585b69~JJGO1CqYI2485424854epoutp03O DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=samsung.com; s=mail20170921; t=1659897948; bh=lOaVuemJcvzVqvhmC9wH8jBWzJSDY06o4HSKqGiM8oo=; h=From:To:Cc:Subject:Date:In-Reply-To:References:From; b=TlTffWhms4f0hMSZu5S5Bfkd6r080Cm9EK5GrnYCNJR0xj+q7Y6123uoRSw0fCJ8z o4ACgrgS5Ar5+SQnIoPo61zNkmQz4Ny0/cc/gZd62vM4YW7T+zt1sO3jLq48b7IP/C TXCbSkyxHIhFiGL4r6L173sHr6cPGyJKlFw5bZfs= Received: from epsnrtp3.localdomain (unknown [182.195.42.164]) by epcas5p4.samsung.com (KnoxPortal) with ESMTP id 20220807184547epcas5p48b2150fea628f5cbe39ab0f8af669c1c~JJGOLdDhN2500925009epcas5p4_; Sun, 7 Aug 2022 18:45:47 +0000 (GMT) Received: from epsmges5p2new.samsung.com (unknown [182.195.38.182]) by epsnrtp3.localdomain (Postfix) with ESMTP id 4M17ZK1drvz4x9Pr; Sun, 7 Aug 2022 18:45:45 +0000 (GMT) Received: from epcas5p3.samsung.com ( [182.195.41.41]) by epsmges5p2new.samsung.com (Symantec Messaging Gateway) with SMTP id 52.E2.42669.95800F26; Mon, 8 Aug 2022 03:45:45 +0900 (KST) Received: from epsmtrp1.samsung.com (unknown [182.195.40.13]) by epcas5p1.samsung.com (KnoxPortal) with ESMTPA id 20220807184544epcas5p19f676581e0fdf555fa1d0a83906f2fc7~JJGLU98HN2851528515epcas5p1k; Sun, 7 Aug 2022 18:45:44 +0000 (GMT) Received: from epsmgms1p1new.samsung.com (unknown [182.195.42.41]) by epsmtrp1.samsung.com (KnoxPortal) with ESMTP id 20220807184544epsmtrp1511305150dd7048a48ee916a1029a9da~JJGLTgwnS0046100461epsmtrp1D; Sun, 7 Aug 2022 18:45:44 +0000 (GMT) X-AuditID: b6c32a4a-b3bff7000001a6ad-85-62f00859fd47 Received: from epsmtip2.samsung.com ( [182.195.34.31]) by epsmgms1p1new.samsung.com (Symantec Messaging Gateway) with SMTP id 33.35.08905.85800F26; Mon, 8 Aug 2022 03:45:44 +0900 (KST) Received: from localhost.localdomain (unknown [107.110.206.5]) by epsmtip2.samsung.com (KnoxPortal) with ESMTPA id 20220807184543epsmtip26d826f46aa4317493d357010c81cda5d~JJGKIR7wE1262212622epsmtip2a; Sun, 7 Aug 2022 18:45:43 +0000 (GMT) From: Kanchan Joshi To: axboe@kernel.dk, hch@lst.de Cc: io-uring@vger.kernel.org, linux-nvme@lists.infradead.org, linux-block@vger.kernel.org, ming.lei@redhat.com, gost.dev@samsung.com, Kanchan Joshi Subject: [PATCH for-next v2 1/4] fs: add file_operations->uring_cmd_iopoll Date: Mon, 8 Aug 2022 00:06:04 +0530 Message-Id: <20220807183607.352351-2-joshi.k@samsung.com> X-Mailer: git-send-email 2.25.1 In-Reply-To: <20220807183607.352351-1-joshi.k@samsung.com> MIME-Version: 1.0 X-Brightmail-Tracker: H4sIAAAAAAAAA+NgFupik+LIzCtJLcpLzFFi42LZdlhTUzeS40OSwbJ3bBar7/azWdw8sJPJ YuXqo0wW71rPsVgc/f+WzWLvLW2L+cueslscmtzM5MDhcflsqcfmJfUeu282sHm833eVzaNv yypGj8+b5ALYorJtMlITU1KLFFLzkvNTMvPSbZW8g+Od403NDAx1DS0tzJUU8hJzU22VXHwC dN0yc4CuUVIoS8wpBQoFJBYXK+nb2RTll5akKmTkF5fYKqUWpOQUmBToFSfmFpfmpevlpZZY GRoYGJkCFSZkZ2w//Zi9YC9rxfVbJ9kaGI+xdDFyckgImEg0Nm9n7GLk4hAS2M0ocWz7GxYI 5xOjxPuub+wQzjdGifkXTjDCtHT+nwvVspdRYuaHJjYI5zOQc+ElUIaDg01AU+LC5FKQBhEB eYkvt9eCjWUWWMsocXrvFyaQhLCAl0T3ie/MIDaLgKrE230TWUFsXgFLic+7+lghtslLzLz0 nR1kJqeAlUTXFmWIEkGJkzOfgP3ADFTSvHU2M8h8CYGP7BIXbyyA6nWR2HzrIhuELSzx6vgW dghbSuJlfxuUnSxxaeY5Jgi7ROLxnoNQtr1E66l+ZpC9zEC/rN+lD7GLT6L39xMmkLCEAK9E R5sQRLWixL1JT6G2iks8nLEEyvaQOL5jJzR4ehklmmdMZJ/AKD8LyQuzkLwwC2HbAkbmVYyS qQXFuempxaYFRnmp5fCITc7P3cQITpRaXjsYHz74oHeIkYmD8RCjBAezkgjvkbXvk4R4UxIr q1KL8uOLSnNSiw8xmgKDeCKzlGhyPjBV55XEG5pYGpiYmZmZWBqbGSqJ83pd3ZQkJJCeWJKa nZpakFoE08fEwSnVwMTcJ7XC+3xM/snKvPbHV1NlU2S1pnE1HTJkml9S/JO/9bbp3NqKVYly /KoTWgT2MBlNTbA03Mm4Rvv9/ZazDEHP11yPDDNovp0vKfJ8507xV01snZbbPkdoaM5ZXZOv ldjvKuC+cvvW9B/Jj2Pui6iwXs3oq2JudPSPjfATXzV5MyfL3qT481Gx916c2W+5cPJrgcAo 7Y1yZtcz+gvtrx+0LLPUKDk+h/twI5PcM96HbMwlZd9UOD0clHjFs3W/5x3ivvGfofekV5lI F+sjFeXN37hKw9N3K6resmk2KXsl/frViQ+Vz6UkvhdquN6xj46xdWy1+HSfY2q+861DqcU5 a5+GJ07lXZSgyKHEUpyRaKjFXFScCACMslZzHQQAAA== X-Brightmail-Tracker: H4sIAAAAAAAAA+NgFjrLLMWRmVeSWpSXmKPExsWy7bCSvG4Ex4ckg0VzZC1W3+1ns7h5YCeT xcrVR5ks3rWeY7E4+v8tm8XeW9oW85c9Zbc4NLmZyYHD4/LZUo/NS+o9dt9sYPN4v+8qm0ff llWMHp83yQWwRXHZpKTmZJalFunbJXBlbD/9mL1gL2vF9Vsn2RoYj7F0MXJySAiYSHT+n8vY xcjFISSwm1Hi8vqVUAlxieZrP9ghbGGJlf+es0MUfWSUOL78BpDDwcEmoClxYXIpSI2IgKLE xo9NYIOYBTYzSnw6fYwZJCEs4CXRfeI7mM0ioCrxdt9EVhCbV8BS4vOuPlaIBfISMy99B5vJ KWAl0bVFGSQsBFRy9kILM0S5oMTJmU/AbmMGKm/eOpt5AqPALCSpWUhSCxiZVjFKphYU56bn FhsWGOallusVJ+YWl+al6yXn525iBIe4luYOxu2rPugdYmTiYDzEKMHBrCTCe2Tt+yQh3pTE yqrUovz4otKc1OJDjNIcLErivBe6TsYLCaQnlqRmp6YWpBbBZJk4OKUamLyKG8WX8uuGrA2p mBPybuPq81ecg+b/ym1euvXxEmsFX/8zhi6XHPTLXYN/nF7kvqzpzQlNznZf1Uk6t7OS52c1 Pug8zCBfVnIyWe0BE/Py3+1fb/D6n85YePYk/0dNv6QfsuI/5XalqfXuD85J26xzKcwo0jFL bs3Tgg82zVvLl/8wrn91O+ni/k0BdzorDpTEcLCxKbOrFKZ2mcS2nZltHrTyfaXgLVVrl5Zf FT1ixf8+y3cfeDDzl9jGFfMEnD2T7y+f/5Ph5zSFJ5ov/kszzWNUi/gYP6EjdPJD+dD/LYU/ C5M2rO1/MuF7XkroruVTUnI/bFBaGX5++6ZfMq9jd0msLhMP3p3XOD3QXImlOCPRUIu5qDgR AODKZNHgAgAA X-CMS-MailID: 20220807184544epcas5p19f676581e0fdf555fa1d0a83906f2fc7 X-Msg-Generator: CA CMS-TYPE: 105P DLP-Filter: Pass X-CFilter-Loop: Reflected X-CMS-RootMailID: 20220807184544epcas5p19f676581e0fdf555fa1d0a83906f2fc7 References: <20220807183607.352351-1-joshi.k@samsung.com> Precedence: bulk List-ID: X-Mailing-List: io-uring@vger.kernel.org io_uring will trigger this to do completion polling on uring-cmd operations. Signed-off-by: Kanchan Joshi --- include/linux/fs.h | 1 + 1 file changed, 1 insertion(+) diff --git a/include/linux/fs.h b/include/linux/fs.h index 9f131e559d05..449941f99f50 100644 --- a/include/linux/fs.h +++ b/include/linux/fs.h @@ -2134,6 +2134,7 @@ struct file_operations { loff_t len, unsigned int remap_flags); int (*fadvise)(struct file *, loff_t, loff_t, int); int (*uring_cmd)(struct io_uring_cmd *ioucmd, unsigned int issue_flags); + int (*uring_cmd_iopoll)(struct io_uring_cmd *ioucmd); } __randomize_layout; struct inode_operations { From patchwork Sun Aug 7 18:36:05 2022 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Kanchan Joshi X-Patchwork-Id: 12938240 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from vger.kernel.org (vger.kernel.org [23.128.96.18]) by smtp.lore.kernel.org (Postfix) with ESMTP id 34DD4C19F2A for ; Sun, 7 Aug 2022 18:45:56 +0000 (UTC) Received: (majordomo@vger.kernel.org) by vger.kernel.org via listexpand id S235220AbiHGSpz (ORCPT ); Sun, 7 Aug 2022 14:45:55 -0400 Received: from lindbergh.monkeyblade.net ([23.128.96.19]:38338 "EHLO lindbergh.monkeyblade.net" rhost-flags-OK-OK-OK-OK) by vger.kernel.org with ESMTP id S235549AbiHGSpx (ORCPT ); Sun, 7 Aug 2022 14:45:53 -0400 Received: from mailout1.samsung.com (mailout1.samsung.com [203.254.224.24]) by lindbergh.monkeyblade.net (Postfix) with ESMTPS id 894139583 for ; Sun, 7 Aug 2022 11:45:52 -0700 (PDT) Received: from epcas5p3.samsung.com (unknown [182.195.41.41]) by mailout1.samsung.com (KnoxPortal) with ESMTP id 20220807184550epoutp01b2d39380aef0a295a4ea0212787e9abf~JJGRNuzn10485904859epoutp01R for ; Sun, 7 Aug 2022 18:45:50 +0000 (GMT) DKIM-Filter: OpenDKIM Filter v2.11.0 mailout1.samsung.com 20220807184550epoutp01b2d39380aef0a295a4ea0212787e9abf~JJGRNuzn10485904859epoutp01R DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=samsung.com; s=mail20170921; t=1659897950; bh=rdPsAxxvz7Zcyw0iMWU8XUYENn+3wN4ROlvI+GUBGXg=; h=From:To:Cc:Subject:Date:In-Reply-To:References:From; b=O5OF2pEM38hQGpCorhvSl4ZX/mO15mMOBlITyZJ8SiHrsFtYPefdlOXbGNZkbF3Zt a0Mqw24vRWPt+wZhCLdcguPhnPl5BbbBJHnr9wlZwPdMzgRvVpfj/hvPHy6iD8K5P/ xyhrRedrxOIB9e6UAdnvosfQ5c8UFcAvVkrIJswA= Received: from epsnrtp3.localdomain (unknown [182.195.42.164]) by epcas5p4.samsung.com (KnoxPortal) with ESMTP id 20220807184550epcas5p4d540192b1a2f33bcbadff32fb5ffee6c~JJGQzjwWU2500925009epcas5p4B; Sun, 7 Aug 2022 18:45:50 +0000 (GMT) Received: from epsmges5p1new.samsung.com (unknown [182.195.38.182]) by epsnrtp3.localdomain (Postfix) with ESMTP id 4M17ZN0HLTz4x9Pr; Sun, 7 Aug 2022 18:45:48 +0000 (GMT) Received: from epcas5p1.samsung.com ( [182.195.41.39]) by epsmges5p1new.samsung.com (Symantec Messaging Gateway) with SMTP id 70.5F.09662.B5800F26; Mon, 8 Aug 2022 03:45:47 +0900 (KST) Received: from epsmtrp2.samsung.com (unknown [182.195.40.14]) by epcas5p2.samsung.com (KnoxPortal) with ESMTPA id 20220807184547epcas5p23b4ef30467d65d1b81632e7c514fc192~JJGONytug2712827128epcas5p2F; Sun, 7 Aug 2022 18:45:47 +0000 (GMT) Received: from epsmgms1p2.samsung.com (unknown [182.195.42.42]) by epsmtrp2.samsung.com (KnoxPortal) with ESMTP id 20220807184547epsmtrp277cfcbfc92f490e7349f90ccdf4e0506~JJGOM8vrX0248002480epsmtrp2p; Sun, 7 Aug 2022 18:45:47 +0000 (GMT) X-AuditID: b6c32a49-86fff700000025be-d0-62f0085bdc63 Received: from epsmtip2.samsung.com ( [182.195.34.31]) by epsmgms1p2.samsung.com (Symantec Messaging Gateway) with SMTP id AE.79.08802.B5800F26; Mon, 8 Aug 2022 03:45:47 +0900 (KST) Received: from localhost.localdomain (unknown [107.110.206.5]) by epsmtip2.samsung.com (KnoxPortal) with ESMTPA id 20220807184546epsmtip23141023eab4008b1a5913e80bc88c964~JJGM56XMp2084620846epsmtip2h; Sun, 7 Aug 2022 18:45:46 +0000 (GMT) From: Kanchan Joshi To: axboe@kernel.dk, hch@lst.de Cc: io-uring@vger.kernel.org, linux-nvme@lists.infradead.org, linux-block@vger.kernel.org, ming.lei@redhat.com, gost.dev@samsung.com, Kanchan Joshi , Pankaj Raghav Subject: [PATCH for-next v2 2/4] io_uring: add iopoll infrastructure for io_uring_cmd Date: Mon, 8 Aug 2022 00:06:05 +0530 Message-Id: <20220807183607.352351-3-joshi.k@samsung.com> X-Mailer: git-send-email 2.25.1 In-Reply-To: <20220807183607.352351-1-joshi.k@samsung.com> MIME-Version: 1.0 X-Brightmail-Tracker: H4sIAAAAAAAAA+NgFupjk+LIzCtJLcpLzFFi42LZdlhTXTea40OSwYTj1har7/azWdw8sJPJ YuXqo0wW71rPsVgc/f+WzWLvLW2L+cueslscmtzMZPF5aQu7A6fH5bOlHpuX1HvsvtnA5vF+ 31U2j74tqxg9Pm+SC2CLyrbJSE1MSS1SSM1Lzk/JzEu3VfIOjneONzUzMNQ1tLQwV1LIS8xN tVVy8QnQdcvMATpJSaEsMacUKBSQWFyspG9nU5RfWpKqkJFfXGKrlFqQklNgUqBXnJhbXJqX rpeXWmJlaGBgZApUmJCdcfKUecE1uYq9FzawNDD+kehi5OSQEDCR+PLqMVsXIxeHkMBuRokp UxZAOZ8YJS4/OM4M4XxmlOhfeI4FpuXEpXfsEIldjBK3D25ngau6e/gfUxcjBwebgKbEhcml IA0iAvISX26vBathFrjMKNHWOYUZJCEsEC7xoLWVEcRmEVCVOLuriQnE5hWwlJhx6RUjxDZ5 iZmXvrODzOQUsJLo2qIMUSIocXLmE7CDmIFKmrfOBrtUQuAvu0TfmxdMEL0uEk8a/7NC2MIS r45vYYewpSQ+v9vLBmEnS1yaeQ6qvkTi8Z6DULa9ROupfmaQvcxAv6zfpQ+xi0+i9/cTsBcl BHglOtqEIKoVJe5Negq1SVzi4YwlULaHxNTJq6CB2MsocWTLDLYJjPKzkLwwC8kLsxC2LWBk XsUomVpQnJueWmxaYJiXWg6P1+T83E2M4ISp5bmD8e6DD3qHGJk4GA8xSnAwK4nwHln7PkmI NyWxsiq1KD++qDQntfgQoykwiCcyS4km5wNTdl5JvKGJpYGJmZmZiaWxmaGSOK/X1U1JQgLp iSWp2ampBalFMH1MHJxSDUx+8381veHVq+F+srnq5cZH26Zncqz/6WZzwqX06VvNKufcDZM3 JO2ZNVV+ot5Lo3brWz9f33r4dV387fLjh5y8ZqvYPNfJPZVf1bdJx7LlAW9tnvS7gwkrTbh9 W47vtHuzmOEY181puqmnLhy4oZ37Pvoio8Ceyu1JffUeEzO9NCeekCxy3/PqwY6S9IjjUo2V snlX5lpqaDPXWjkY1hodPrg8vq5nB0slyzZXi2LVrevK5wuy/Kx51nixWFK4d3JM6M8ih7q0 7WfbZ4QzlEtvs5W6FBd2l/Xq/9TDGxxdVkmkOCy+2sDQPtlqzspZZ5SeMAparZ2Rs7e13+Sv nvWtT5Uilt8T3H6ZSh9c8kuJpTgj0VCLuag4EQBBAUSDIQQAAA== X-Brightmail-Tracker: H4sIAAAAAAAAA+NgFrrALMWRmVeSWpSXmKPExsWy7bCSvG40x4ckg3WflS1W3+1ns7h5YCeT xcrVR5ks3rWeY7E4+v8tm8XeW9oW85c9Zbc4NLmZyeLz0hZ2B06Py2dLPTYvqffYfbOBzeP9 vqtsHn1bVjF6fN4kF8AWxWWTkpqTWZZapG+XwJVx8pR5wTW5ir0XNrA0MP6R6GLk5JAQMJE4 cekdO4gtJLCDUeLzRiuIuLhE87Uf7BC2sMTKf8+BbC6gmo+MEnd/LWbuYuTgYBPQlLgwuRSk RkRAUWLjxyZGkBpmgduMEnOnn2IBSQgLhEp8uLSbFcRmEVCVOLuriQnE5hWwlJhx6RUjxAJ5 iZmXvrODzOQUsJLo2qIMcY+lxNkLLcwQ5YISJ2c+ARvJDFTevHU28wRGgVlIUrOQpBYwMq1i lEwtKM5Nzy02LDDKSy3XK07MLS7NS9dLzs/dxAgOdC2tHYx7Vn3QO8TIxMF4iFGCg1lJhPfI 2vdJQrwpiZVVqUX58UWlOanFhxilOViUxHkvdJ2MFxJITyxJzU5NLUgtgskycXBKNTCxBlXM Xf2t8BbHl+635x/fvPDv1SX+xTv8y1g9YhV+rXjqYDWRoczO4t+hvNjrZ3k7b+mxKW6V39GZ nKrm57oz+GaCk5K5U9VUP23l7tjJezJLJ+X+2mKwIZRpgUXx5KQ11QEMxx24Xyj2rApOXMHx SmG+y/+bfmWq57teqcavPl7u/S/4U/o91rU7d0cLBQh1vzC8/Pm7EOMm15istF9qzzmPn9i+ 6c4X8/8/dHaEfWosjjzS3T/hFMtX27PHWn5XT+9az8/B9y7KQKXq6Fav9mt3eDlZzQut1jYd vOMZF1Mu/Wl7RcjkCK4bEez+4jPTmQPmbvm38JScNFfJlaTTLjPfpvrqdtwt3XjoiqcSS3FG oqEWc1FxIgBIO8Fl4wIAAA== X-CMS-MailID: 20220807184547epcas5p23b4ef30467d65d1b81632e7c514fc192 X-Msg-Generator: CA CMS-TYPE: 105P DLP-Filter: Pass X-CFilter-Loop: Reflected X-CMS-RootMailID: 20220807184547epcas5p23b4ef30467d65d1b81632e7c514fc192 References: <20220807183607.352351-1-joshi.k@samsung.com> Precedence: bulk List-ID: X-Mailing-List: io-uring@vger.kernel.org Put this up in the same way as iopoll is done for regular read/write IO. Make place for storing a cookie into struct io_uring_cmd on its submission. Perform the completion using the ->uring_cmd_iopoll handler. Signed-off-by: Kanchan Joshi Signed-off-by: Pankaj Raghav --- include/linux/io_uring.h | 8 ++++++-- io_uring/io_uring.c | 6 ++++++ io_uring/opdef.c | 1 + io_uring/rw.c | 8 +++++++- io_uring/uring_cmd.c | 11 +++++++++-- 5 files changed, 29 insertions(+), 5 deletions(-) diff --git a/include/linux/io_uring.h b/include/linux/io_uring.h index 4a2f6cc5a492..58676c0a398f 100644 --- a/include/linux/io_uring.h +++ b/include/linux/io_uring.h @@ -20,8 +20,12 @@ enum io_uring_cmd_flags { struct io_uring_cmd { struct file *file; const void *cmd; - /* callback to defer completions to task context */ - void (*task_work_cb)(struct io_uring_cmd *cmd); + union { + /* callback to defer completions to task context */ + void (*task_work_cb)(struct io_uring_cmd *cmd); + /* used for polled completion */ + void *cookie; + }; u32 cmd_op; u32 pad; u8 pdu[32]; /* available inline for free use */ diff --git a/io_uring/io_uring.c b/io_uring/io_uring.c index b54218da075c..48a430a86b50 100644 --- a/io_uring/io_uring.c +++ b/io_uring/io_uring.c @@ -1296,6 +1296,12 @@ static int io_iopoll_check(struct io_ring_ctx *ctx, long min) wq_list_empty(&ctx->iopoll_list)) break; } + + if (task_work_pending(current)) { + mutex_unlock(&ctx->uring_lock); + io_run_task_work(); + mutex_lock(&ctx->uring_lock); + } ret = io_do_iopoll(ctx, !min); if (ret < 0) break; diff --git a/io_uring/opdef.c b/io_uring/opdef.c index 72dd2b2d8a9d..9a0df19306fe 100644 --- a/io_uring/opdef.c +++ b/io_uring/opdef.c @@ -466,6 +466,7 @@ const struct io_op_def io_op_defs[] = { .needs_file = 1, .plug = 1, .name = "URING_CMD", + .iopoll = 1, .async_size = uring_cmd_pdu_size(1), .prep = io_uring_cmd_prep, .issue = io_uring_cmd, diff --git a/io_uring/rw.c b/io_uring/rw.c index 2b784795103c..1a4fb8a44b9a 100644 --- a/io_uring/rw.c +++ b/io_uring/rw.c @@ -1005,7 +1005,13 @@ int io_do_iopoll(struct io_ring_ctx *ctx, bool force_nonspin) if (READ_ONCE(req->iopoll_completed)) break; - ret = rw->kiocb.ki_filp->f_op->iopoll(&rw->kiocb, &iob, poll_flags); + if (req->opcode == IORING_OP_URING_CMD) { + struct io_uring_cmd *ioucmd = (struct io_uring_cmd *)rw; + + ret = req->file->f_op->uring_cmd_iopoll(ioucmd); + } else + ret = rw->kiocb.ki_filp->f_op->iopoll(&rw->kiocb, &iob, + poll_flags); if (unlikely(ret < 0)) return ret; else if (ret) diff --git a/io_uring/uring_cmd.c b/io_uring/uring_cmd.c index 0a421ed51e7e..5cc339fba8b8 100644 --- a/io_uring/uring_cmd.c +++ b/io_uring/uring_cmd.c @@ -49,7 +49,11 @@ void io_uring_cmd_done(struct io_uring_cmd *ioucmd, ssize_t ret, ssize_t res2) io_req_set_res(req, 0, ret); if (req->ctx->flags & IORING_SETUP_CQE32) io_req_set_cqe32_extra(req, res2, 0); - __io_req_complete(req, 0); + if (req->ctx->flags & IORING_SETUP_IOPOLL) + /* order with io_iopoll_req_issued() checking ->iopoll_completed */ + smp_store_release(&req->iopoll_completed, 1); + else + __io_req_complete(req, 0); } EXPORT_SYMBOL_GPL(io_uring_cmd_done); @@ -89,8 +93,11 @@ int io_uring_cmd(struct io_kiocb *req, unsigned int issue_flags) issue_flags |= IO_URING_F_SQE128; if (ctx->flags & IORING_SETUP_CQE32) issue_flags |= IO_URING_F_CQE32; - if (ctx->flags & IORING_SETUP_IOPOLL) + if (ctx->flags & IORING_SETUP_IOPOLL) { issue_flags |= IO_URING_F_IOPOLL; + req->iopoll_completed = 0; + WRITE_ONCE(ioucmd->cookie, NULL); + } if (req_has_async_data(req)) ioucmd->cmd = req->async_data; From patchwork Sun Aug 7 18:36:06 2022 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Kanchan Joshi X-Patchwork-Id: 12938242 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from vger.kernel.org (vger.kernel.org [23.128.96.18]) by smtp.lore.kernel.org (Postfix) with ESMTP id EA14CC19F2A for ; Sun, 7 Aug 2022 18:46:02 +0000 (UTC) Received: (majordomo@vger.kernel.org) by vger.kernel.org via listexpand id S235578AbiHGSqC (ORCPT ); Sun, 7 Aug 2022 14:46:02 -0400 Received: from lindbergh.monkeyblade.net ([23.128.96.19]:38570 "EHLO lindbergh.monkeyblade.net" rhost-flags-OK-OK-OK-OK) by vger.kernel.org with ESMTP id S232582AbiHGSqB (ORCPT ); Sun, 7 Aug 2022 14:46:01 -0400 Received: from mailout4.samsung.com (mailout4.samsung.com [203.254.224.34]) by lindbergh.monkeyblade.net (Postfix) with ESMTPS id AFA5495A6 for ; Sun, 7 Aug 2022 11:45:59 -0700 (PDT) Received: from epcas5p4.samsung.com (unknown [182.195.41.42]) by mailout4.samsung.com (KnoxPortal) with ESMTP id 20220807184555epoutp04fbd8de61aebd4fa8e2998ad9d43dc666~JJGVYtBUy0667206672epoutp04D for ; Sun, 7 Aug 2022 18:45:55 +0000 (GMT) DKIM-Filter: OpenDKIM Filter v2.11.0 mailout4.samsung.com 20220807184555epoutp04fbd8de61aebd4fa8e2998ad9d43dc666~JJGVYtBUy0667206672epoutp04D DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=samsung.com; s=mail20170921; t=1659897955; bh=5IRAjs6+m4OB4xTk3tclAQOLtoFiORzCHerBHOWaHZk=; h=From:To:Cc:Subject:Date:In-Reply-To:References:From; b=ck44F3rH65TOzqMZS4aLVEqAld5i8rOQV9/akI5/vlUnFfsgr+5heT4XTfbsfhDM2 eza7D2qj65IbWPBh9qWaGrk2PBcB/cNlGP+HzZ7pAcoNhmyeYSnpuAV7TWcCT6GUnn RQHnWX7KOc9U9/ZWJfTs6XWs+pTgu5oeFeh1GVZk= Received: from epsnrtp2.localdomain (unknown [182.195.42.163]) by epcas5p4.samsung.com (KnoxPortal) with ESMTP id 20220807184554epcas5p42c3108dc74a0f9949527b5a43fd3266d~JJGUXhwe62500825008epcas5p4O; Sun, 7 Aug 2022 18:45:54 +0000 (GMT) Received: from epsmges5p1new.samsung.com (unknown [182.195.38.177]) by epsnrtp2.localdomain (Postfix) with ESMTP id 4M17ZR4czRz4x9Pt; Sun, 7 Aug 2022 18:45:51 +0000 (GMT) Received: from epcas5p2.samsung.com ( [182.195.41.40]) by epsmges5p1new.samsung.com (Symantec Messaging Gateway) with SMTP id 11.5F.09662.F5800F26; Mon, 8 Aug 2022 03:45:51 +0900 (KST) Received: from epsmtrp2.samsung.com (unknown [182.195.40.14]) by epcas5p3.samsung.com (KnoxPortal) with ESMTPA id 20220807184551epcas5p3b85421505f9c28d31492163f69c59d69~JJGRkfS2c1284912849epcas5p3H; Sun, 7 Aug 2022 18:45:51 +0000 (GMT) Received: from epsmgms1p2.samsung.com (unknown [182.195.42.42]) by epsmtrp2.samsung.com (KnoxPortal) with ESMTP id 20220807184551epsmtrp2e02088ce37748614fff16b228adce981~JJGRjq0gh0248002480epsmtrp2q; Sun, 7 Aug 2022 18:45:51 +0000 (GMT) X-AuditID: b6c32a49-885ff700000025be-d5-62f0085fd424 Received: from epsmtip2.samsung.com ( [182.195.34.31]) by epsmgms1p2.samsung.com (Symantec Messaging Gateway) with SMTP id 4F.79.08802.F5800F26; Mon, 8 Aug 2022 03:45:51 +0900 (KST) Received: from localhost.localdomain (unknown [107.110.206.5]) by epsmtip2.samsung.com (KnoxPortal) with ESMTPA id 20220807184549epsmtip2280c98ff52b9b36411fbf9e829075728~JJGQW0uI22085320853epsmtip2k; Sun, 7 Aug 2022 18:45:49 +0000 (GMT) From: Kanchan Joshi To: axboe@kernel.dk, hch@lst.de Cc: io-uring@vger.kernel.org, linux-nvme@lists.infradead.org, linux-block@vger.kernel.org, ming.lei@redhat.com, gost.dev@samsung.com, Kanchan Joshi Subject: [PATCH for-next v2 3/4] block: export blk_rq_is_poll Date: Mon, 8 Aug 2022 00:06:06 +0530 Message-Id: <20220807183607.352351-4-joshi.k@samsung.com> X-Mailer: git-send-email 2.25.1 In-Reply-To: <20220807183607.352351-1-joshi.k@samsung.com> MIME-Version: 1.0 X-Brightmail-Tracker: H4sIAAAAAAAAA+NgFupkk+LIzCtJLcpLzFFi42LZdlhTQzee40OSwZXdyhar7/azWdw8sJPJ YuXqo0wW71rPsVgc/f+WzWLvLW2L+cueslscmtzM5MDhcflsqcfmJfUeu282sHm833eVzaNv yypGj8+b5ALYorJtMlITU1KLFFLzkvNTMvPSbZW8g+Od403NDAx1DS0tzJUU8hJzU22VXHwC dN0yc4CuUVIoS8wpBQoFJBYXK+nb2RTll5akKmTkF5fYKqUWpOQUmBToFSfmFpfmpevlpZZY GRoYGJkCFSZkZ8x9c4+1YBJ3xaaeI6wNjEs5uxg5OSQETCRmdPxl7GLk4hAS2M0o8XLFTRYI 5xOjxJcXDVCZz4wS9xq3ssO0zGrawQaR2MUo0TDtBztc1fOZL4EcDg42AU2JC5NLQRpEBOQl vtxeCzaWWWAto8TpvV+YQBLCArYSd26uZAGpZxFQldj/kAskzCtgKbFv6XM2iGXyEjMvfQcb ySlgJdG1RRmiRFDi5MwnLCA2M1BJ89bZzCDjJQR+sku8Wn+OEaLXRWLW86usELawxKvjW6Ae kJL4/G4v1PxkiUszzzFB2CUSj/cchLLtJVpP9TOD7GUGemX9Ln2IXXwSvb+fMIGEJQR4JTra hCCqFSXuTXoKtUlc4uGMJVC2h8Te3rXMkNDpZZTomXOEeQKj/CwkL8xC8sIshG0LGJlXMUqm FhTnpqcWmxYY5qWWw+M1OT93EyM4TWp57mC8++CD3iFGJg7GQ4wSHMxKIrxH1r5PEuJNSays Si3Kjy8qzUktPsRoCgzhicxSosn5wESdVxJvaGJpYGJmZmZiaWxmqCTO63V1U5KQQHpiSWp2 ampBahFMHxMHp1QD0/RbTPpFsQ+UuFPTp9zgLb0R/5tPouv32ky2TcY2zoG9MXJx8/p+PjD7 +1TXz8b2+rasdR1BrowXb6Wf7YupePro2esZS6Jaf9rtet1t8DjlspDMV43aVSWdgXcuzjy5 OWLRqcMrLZYfnHbF9nZ2lVfDRqvQdN+CkAVPpN6/KY828OLJ085pvvrGY++7JWaSf23+XUj0 qmHgvbJw75Lj5mtu/3279Eu9/xrNsx3h9r9N/iqsss7/eV+EV+zaNGeGOvH9OQ/Zfm7pXXEy pdb6R4aT1iPH4/r/Jlz5n7zOUOiHp9ic/c67au9OzNAJ/fdu734fs3X/5T/OWXXsiMJzZb7d Lq1dab02jzZuCPtz7pkSS3FGoqEWc1FxIgABKitPHAQAAA== X-Brightmail-Tracker: H4sIAAAAAAAAA+NgFjrDLMWRmVeSWpSXmKPExsWy7bCSvG48x4ckg9+zuSxW3+1ns7h5YCeT xcrVR5ks3rWeY7E4+v8tm8XeW9oW85c9Zbc4NLmZyYHD4/LZUo/NS+o9dt9sYPN4v+8qm0ff llWMHp83yQWwRXHZpKTmZJalFunbJXBlzH1zj7VgEnfFpp4jrA2MSzm7GDk5JARMJGY17WDr YuTiEBLYwSjx5+MRJoiEuETztR/sELawxMp/z9khij4ySjzbNJO5i5GDg01AU+LC5FKQGhEB RYmNH5sYQWqYBTYzSnw6fYwZJCEsYCtx5+ZKFpB6FgFVif0PuUDCvAKWEvuWPmeDmC8vMfPS d3aQEk4BK4muLcogYSGgkrMXWpghygUlTs58wgJiMwOVN2+dzTyBUWAWktQsJKkFjEyrGCVT C4pz03OLDQuM8lLL9YoTc4tL89L1kvNzNzGCA1xLawfjnlUf9A4xMnEwHmKU4GBWEuE9svZ9 khBvSmJlVWpRfnxRaU5q8SFGaQ4WJXHeC10n44UE0hNLUrNTUwtSi2CyTBycUg1Mkltva8jP 4zoasuZoXWvzvvYZfx/orDLKMjp8uGNzCZ9fqvub1zm/d0QyF0bJ9Ojp8ryNqalqvRqtc+Ta XfOz7hFNOybEzTR4eOPLIr5dO3c/tNbwNt+Ylp5Xo+K1WKHSb694CeuRgvct2+6u89x7wP5d fofbE+cfD1wb2Xc1rq7VEjG/u3nZQ7W9Ep8u3JymV37h5KW0lRITjve8/LK5zcXFbPe071dZ atSvP8/NPp3792WSSZNXhFDiLj6+tOlf66W1CyapFr/c33L0hqiUDvvvOxP8ec8cnp0+07df ryBqIZtwgs0szZT8p8sX7f19wWD5/3vSws4T3CyVnn385j11jpXmtLt+tS6XXA9PUWIpzkg0 1GIuKk4EAB/KuPHfAgAA X-CMS-MailID: 20220807184551epcas5p3b85421505f9c28d31492163f69c59d69 X-Msg-Generator: CA CMS-TYPE: 105P DLP-Filter: Pass X-CFilter-Loop: Reflected X-CMS-RootMailID: 20220807184551epcas5p3b85421505f9c28d31492163f69c59d69 References: <20220807183607.352351-1-joshi.k@samsung.com> Precedence: bulk List-ID: X-Mailing-List: io-uring@vger.kernel.org This is being done as preparation to support iopoll for nvme passthrough Signed-off-by: Kanchan Joshi --- block/blk-mq.c | 3 ++- include/linux/blk-mq.h | 1 + 2 files changed, 3 insertions(+), 1 deletion(-) diff --git a/block/blk-mq.c b/block/blk-mq.c index 5ee62b95f3e5..de42f7237bad 100644 --- a/block/blk-mq.c +++ b/block/blk-mq.c @@ -1233,7 +1233,7 @@ static void blk_end_sync_rq(struct request *rq, blk_status_t ret) complete(&wait->done); } -static bool blk_rq_is_poll(struct request *rq) +bool blk_rq_is_poll(struct request *rq) { if (!rq->mq_hctx) return false; @@ -1243,6 +1243,7 @@ static bool blk_rq_is_poll(struct request *rq) return false; return true; } +EXPORT_SYMBOL_GPL(blk_rq_is_poll); static void blk_rq_poll_completion(struct request *rq, struct completion *wait) { diff --git a/include/linux/blk-mq.h b/include/linux/blk-mq.h index effee1dc715a..8f841caaa4cb 100644 --- a/include/linux/blk-mq.h +++ b/include/linux/blk-mq.h @@ -981,6 +981,7 @@ int blk_rq_map_kern(struct request_queue *, struct request *, void *, int blk_rq_append_bio(struct request *rq, struct bio *bio); void blk_execute_rq_nowait(struct request *rq, bool at_head); blk_status_t blk_execute_rq(struct request *rq, bool at_head); +bool blk_rq_is_poll(struct request *rq); struct req_iterator { struct bvec_iter iter; From patchwork Sun Aug 7 18:36:07 2022 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Kanchan Joshi X-Patchwork-Id: 12938243 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from vger.kernel.org (vger.kernel.org [23.128.96.18]) by smtp.lore.kernel.org (Postfix) with ESMTP id CFA5AC25B0C for ; Sun, 7 Aug 2022 18:46:03 +0000 (UTC) Received: (majordomo@vger.kernel.org) by vger.kernel.org via listexpand id S235428AbiHGSqD (ORCPT ); Sun, 7 Aug 2022 14:46:03 -0400 Received: from lindbergh.monkeyblade.net ([23.128.96.19]:38582 "EHLO lindbergh.monkeyblade.net" rhost-flags-OK-OK-OK-OK) by vger.kernel.org with ESMTP id S235555AbiHGSqB (ORCPT ); Sun, 7 Aug 2022 14:46:01 -0400 Received: from mailout3.samsung.com (mailout3.samsung.com [203.254.224.33]) by lindbergh.monkeyblade.net (Postfix) with ESMTPS id 0CC2E95A7 for ; Sun, 7 Aug 2022 11:46:00 -0700 (PDT) Received: from epcas5p3.samsung.com (unknown [182.195.41.41]) by mailout3.samsung.com (KnoxPortal) with ESMTP id 20220807184558epoutp03a27528084de455a1816a7eea7e8509ea~JJGYRXiSw1899318993epoutp03U for ; Sun, 7 Aug 2022 18:45:58 +0000 (GMT) DKIM-Filter: OpenDKIM Filter v2.11.0 mailout3.samsung.com 20220807184558epoutp03a27528084de455a1816a7eea7e8509ea~JJGYRXiSw1899318993epoutp03U DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=samsung.com; s=mail20170921; t=1659897958; bh=ZYYFfaS0RNTkCcJc/ZH77FA4ETZSamAKY1PfxlySMeM=; h=From:To:Cc:Subject:Date:In-Reply-To:References:From; b=O/W2ioYF427vlSxF0K4QiYbuCriVeHQC0bhmOhrk8Ah2TBEGv6LAqCEYWUAICTP7M kp2hYskMaUBcOoYycxozTZveWrsMFxvn41/dEJdusZGzZwBDA+qUfJMTUBn4klXAsK dzExGU9e289Zks18oYOCKt9fcKkgJGO7SldM1hC0= Received: from epsnrtp2.localdomain (unknown [182.195.42.163]) by epcas5p3.samsung.com (KnoxPortal) with ESMTP id 20220807184557epcas5p3d4342cd5e8cebb3c2446b2ae77628a24~JJGX0KLpg2250522505epcas5p3D; Sun, 7 Aug 2022 18:45:57 +0000 (GMT) Received: from epsmges5p3new.samsung.com (unknown [182.195.38.180]) by epsnrtp2.localdomain (Postfix) with ESMTP id 4M17ZW4tdVz4x9Pt; Sun, 7 Aug 2022 18:45:55 +0000 (GMT) Received: from epcas5p2.samsung.com ( [182.195.41.40]) by epsmges5p3new.samsung.com (Symantec Messaging Gateway) with SMTP id C2.EA.09639.36800F26; Mon, 8 Aug 2022 03:45:55 +0900 (KST) Received: from epsmtrp1.samsung.com (unknown [182.195.40.13]) by epcas5p4.samsung.com (KnoxPortal) with ESMTPA id 20220807184555epcas5p4b7f5018c52d150150c32458fe3c21986~JJGVYxlpT2500925009epcas5p4G; Sun, 7 Aug 2022 18:45:55 +0000 (GMT) Received: from epsmgms1p1new.samsung.com (unknown [182.195.42.41]) by epsmtrp1.samsung.com (KnoxPortal) with ESMTP id 20220807184555epsmtrp1cb8e4d7b205ea346f05472292172790a~JJGVXsl273253032530epsmtrp1M; Sun, 7 Aug 2022 18:45:55 +0000 (GMT) X-AuditID: b6c32a4b-e83ff700000025a7-84-62f00863e574 Received: from epsmtip2.samsung.com ( [182.195.34.31]) by epsmgms1p1new.samsung.com (Symantec Messaging Gateway) with SMTP id E3.35.08905.36800F26; Mon, 8 Aug 2022 03:45:55 +0900 (KST) Received: from localhost.localdomain (unknown [107.110.206.5]) by epsmtip2.samsung.com (KnoxPortal) with ESMTPA id 20220807184553epsmtip21359e35c8d27aac4c0e3d33d6b2b9644~JJGT-5q6L2084620846epsmtip2i; Sun, 7 Aug 2022 18:45:53 +0000 (GMT) From: Kanchan Joshi To: axboe@kernel.dk, hch@lst.de Cc: io-uring@vger.kernel.org, linux-nvme@lists.infradead.org, linux-block@vger.kernel.org, ming.lei@redhat.com, gost.dev@samsung.com, Kanchan Joshi , Anuj Gupta Subject: [PATCH for-next v2 4/4] nvme: wire up async polling for io passthrough commands Date: Mon, 8 Aug 2022 00:06:07 +0530 Message-Id: <20220807183607.352351-5-joshi.k@samsung.com> X-Mailer: git-send-email 2.25.1 In-Reply-To: <20220807183607.352351-1-joshi.k@samsung.com> MIME-Version: 1.0 X-Brightmail-Tracker: H4sIAAAAAAAAA+NgFuphk+LIzCtJLcpLzFFi42LZdlhTQzeZ40OSQdMJZYumCX+ZLVbf7Wez uHlgJ5PFytVHmSzetZ5jsTj6/y2bxd5b2hbzlz1ltzg0uZnJgdPj8tlSj81L6j1232xg83i/ 7yqbR9+WVYwenzfJBbBFZdtkpCampBYppOYl56dk5qXbKnkHxzvHm5oZGOoaWlqYKynkJeam 2iq5+AToumXmAJ2kpFCWmFMKFApILC5W0rezKcovLUlVyMgvLrFVSi1IySkwKdArTswtLs1L 18tLLbEyNDAwMgUqTMjOOLT6HFPBdJOKufuusjQwntbuYuTgkBAwkfhzm7OLkYtDSGA3o8SB X1dYIZxPjBI/ph1khnC+MUq0nekBynCCdWx71wpVtZdRYsPLtSwQzmdGiV37DrKAzGUT0JS4 MLkUpEFEQF7iy22IGmaBC4wS93beBKsRFoiSOLkiEKSGRUBV4uKtbUwgNq+ApcTWCatZIJbJ S8y89J0dpJxTwEqia4syRImgxMmZT8BKmIFKmrfOBjtUQuAvu8Tvl4uhDnWRaP59jBnCFpZ4 dXwLO4QtJfGyvw3KTpa4NPMcE4RdIvF4z0Eo216i9VQ/M8heZqBX1u/Sh9jFJ9H7+wkTJOR4 JTrahCCqFSXuTXoKtVVc4uGMJawQJR4S9+7XQAKnl1Hi0L1O1gmM8rOQfDALyQezEJYtYGRe xSiZWlCcm55abFpgnJdaDo/V5PzcTYzgZKnlvYPx0YMPeocYmTgYDzFKcDArifAeWfs+SYg3 JbGyKrUoP76oNCe1+BCjKTCEJzJLiSbnA9N1Xkm8oYmlgYmZmZmJpbGZoZI4r9fVTUlCAumJ JanZqakFqUUwfUwcnFINTPePHXqlLedjXvr0osKHc39+VPIIWR42Y7vqEhQns8fIbf3VW/rn SkvzNVXsL87+LGe/dEX14qX3U4t+M7y78OAa87n5wWH55x7GpWq9eSFwI+DA1SXfs7ZJxgWr 6VSelRaq/Vt2Lcdx2j0h/ohoN6O8kuwuJZ1LqstyLjy3bI1ptuPft9C6dMFzdr6JlfnC/7qt k/ivOQofaSma+Ohmt06nuaNJhOhLybqihYanuSKbN01R1rwaarvaWHTj/H69PzdYitdwXn9p 31FW3S6ydNbsiCN3/uQrmC3uVpn7a07TxNp/rKlbzifqfHO+p/FzvZfmrltqi6WNxTqFHpne 7eha9nR7kbn9nJ+lD5NblFiKMxINtZiLihMBX+3JpR8EAAA= X-Brightmail-Tracker: H4sIAAAAAAAAA+NgFrrILMWRmVeSWpSXmKPExsWy7bCSvG4yx4ckg/WX+CyaJvxltlh9t5/N 4uaBnUwWK1cfZbJ413qOxeLo/7dsFntvaVvMX/aU3eLQ5GYmB06Py2dLPTYvqffYfbOBzeP9 vqtsHn1bVjF6fN4kF8AWxWWTkpqTWZZapG+XwJVxaPU5poLpJhVz911laWA8rd3FyMkhIWAi se1dK2sXIxeHkMBuRokz7zezQSTEJZqv/WCHsIUlVv57DmYLCXxklHi+SL2LkYODTUBT4sLk UpCwiICixMaPTYwgc5gFbjBK7OudwgSSEBaIkDi5YwsriM0ioCpx8dY2sDivgKXE1gmrWSDm y0vMvPSdHWQmp4CVRNcWZYhVlhJnL7QwQ5QLSpyc+QSsnBmovHnrbOYJjAKzkKRmIUktYGRa xSiZWlCcm55bbFhgmJdarlecmFtcmpeul5yfu4kRHOpamjsYt6/6oHeIkYmD8RCjBAezkgjv kbXvk4R4UxIrq1KL8uOLSnNSiw8xSnOwKInzXug6GS8kkJ5YkpqdmlqQWgSTZeLglGpgKmyp K3OQlmPafijugnlJKZNMy/azTDcPmR8Q/mUteo5lyvUVP7U0NjHfXsq9bm/K5eq1bfypvP9l dN5oOa3Yam35+LfVvPSylfLCKyTWPrpzQnSb6oWmrJ15c95qh30NKf9/uyWuaMYc5svac39n yN02uM+k2HBtyfRTta5vfz1g5zc9sGNZaOrFi7/d9zEeW8a4/NiRf+wtk5TSH4XPyjs3Ly6K +VBECc90jskJ0xV3lV0IPPVNT1Zs/oV/frMsGk9UvC+a9fjfxpTXrcaL/tdrijjEphSIJJ+/ WM7W91usimWBuNVq39PHP3O1nkpU+v+15N1uhxOTZ+8qcFxnpKBYxnvKMNR0j64fh+3uHCWW 4oxEQy3mouJEAB/EOp/kAgAA X-CMS-MailID: 20220807184555epcas5p4b7f5018c52d150150c32458fe3c21986 X-Msg-Generator: CA CMS-TYPE: 105P DLP-Filter: Pass X-CFilter-Loop: Reflected X-CMS-RootMailID: 20220807184555epcas5p4b7f5018c52d150150c32458fe3c21986 References: <20220807183607.352351-1-joshi.k@samsung.com> Precedence: bulk List-ID: X-Mailing-List: io-uring@vger.kernel.org Store a cookie during submission, and use that to implement completion-polling inside the ->uring_cmd_iopoll handler. This handler makes use of existing bio poll facility. Signed-off-by: Kanchan Joshi Signed-off-by: Anuj Gupta --- drivers/nvme/host/core.c | 1 + drivers/nvme/host/ioctl.c | 73 ++++++++++++++++++++++++++++++++--- drivers/nvme/host/multipath.c | 1 + drivers/nvme/host/nvme.h | 2 + 4 files changed, 72 insertions(+), 5 deletions(-) diff --git a/drivers/nvme/host/core.c b/drivers/nvme/host/core.c index 2429b11eb9a8..77b6c2882afd 100644 --- a/drivers/nvme/host/core.c +++ b/drivers/nvme/host/core.c @@ -3976,6 +3976,7 @@ static const struct file_operations nvme_ns_chr_fops = { .unlocked_ioctl = nvme_ns_chr_ioctl, .compat_ioctl = compat_ptr_ioctl, .uring_cmd = nvme_ns_chr_uring_cmd, + .uring_cmd_iopoll = nvme_ns_chr_uring_cmd_iopoll, }; static int nvme_add_ns_cdev(struct nvme_ns *ns) diff --git a/drivers/nvme/host/ioctl.c b/drivers/nvme/host/ioctl.c index 27614bee7380..7756b439a688 100644 --- a/drivers/nvme/host/ioctl.c +++ b/drivers/nvme/host/ioctl.c @@ -391,11 +391,19 @@ static void nvme_uring_cmd_end_io(struct request *req, blk_status_t err) struct nvme_uring_cmd_pdu *pdu = nvme_uring_cmd_pdu(ioucmd); /* extract bio before reusing the same field for request */ struct bio *bio = pdu->bio; + void *cookie = READ_ONCE(ioucmd->cookie); pdu->req = req; req->bio = bio; - /* this takes care of moving rest of completion-work to task context */ - io_uring_cmd_complete_in_task(ioucmd, nvme_uring_task_cb); + + /* + * For iopoll, complete it directly. + * Otherwise, move the completion to task work. + */ + if (cookie != NULL && blk_rq_is_poll(req)) + nvme_uring_task_cb(ioucmd); + else + io_uring_cmd_complete_in_task(ioucmd, nvme_uring_task_cb); } static int nvme_uring_cmd_io(struct nvme_ctrl *ctrl, struct nvme_ns *ns, @@ -445,7 +453,10 @@ static int nvme_uring_cmd_io(struct nvme_ctrl *ctrl, struct nvme_ns *ns, rq_flags = REQ_NOWAIT; blk_flags = BLK_MQ_REQ_NOWAIT; } + if (issue_flags & IO_URING_F_IOPOLL) + rq_flags |= REQ_POLLED; +retry: req = nvme_alloc_user_request(q, &c, nvme_to_user_ptr(d.addr), d.data_len, nvme_to_user_ptr(d.metadata), d.metadata_len, 0, &meta, d.timeout_ms ? @@ -456,6 +467,17 @@ static int nvme_uring_cmd_io(struct nvme_ctrl *ctrl, struct nvme_ns *ns, req->end_io = nvme_uring_cmd_end_io; req->end_io_data = ioucmd; + if (issue_flags & IO_URING_F_IOPOLL && rq_flags & REQ_POLLED) { + if (unlikely(!req->bio)) { + /* we can't poll this, so alloc regular req instead */ + blk_mq_free_request(req); + rq_flags &= ~REQ_POLLED; + goto retry; + } else { + WRITE_ONCE(ioucmd->cookie, req->bio); + req->bio->bi_opf |= REQ_POLLED; + } + } /* to free bio on completion, as req->bio will be null at that time */ pdu->bio = req->bio; pdu->meta = meta; @@ -559,9 +581,6 @@ long nvme_ns_chr_ioctl(struct file *file, unsigned int cmd, unsigned long arg) static int nvme_uring_cmd_checks(unsigned int issue_flags) { - /* IOPOLL not supported yet */ - if (issue_flags & IO_URING_F_IOPOLL) - return -EOPNOTSUPP; /* NVMe passthrough requires big SQE/CQE support */ if ((issue_flags & (IO_URING_F_SQE128|IO_URING_F_CQE32)) != @@ -604,6 +623,23 @@ int nvme_ns_chr_uring_cmd(struct io_uring_cmd *ioucmd, unsigned int issue_flags) return nvme_ns_uring_cmd(ns, ioucmd, issue_flags); } +int nvme_ns_chr_uring_cmd_iopoll(struct io_uring_cmd *ioucmd) +{ + struct bio *bio; + int ret = 0; + struct nvme_ns *ns; + struct request_queue *q; + + rcu_read_lock(); + bio = READ_ONCE(ioucmd->cookie); + ns = container_of(file_inode(ioucmd->file)->i_cdev, + struct nvme_ns, cdev); + q = ns->queue; + if (test_bit(QUEUE_FLAG_POLL, &q->queue_flags) && bio && bio->bi_bdev) + ret = bio_poll(bio, NULL, 0); + rcu_read_unlock(); + return ret; +} #ifdef CONFIG_NVME_MULTIPATH static int nvme_ns_head_ctrl_ioctl(struct nvme_ns *ns, unsigned int cmd, void __user *argp, struct nvme_ns_head *head, int srcu_idx) @@ -685,6 +721,29 @@ int nvme_ns_head_chr_uring_cmd(struct io_uring_cmd *ioucmd, srcu_read_unlock(&head->srcu, srcu_idx); return ret; } + +int nvme_ns_head_chr_uring_cmd_iopoll(struct io_uring_cmd *ioucmd) +{ + struct cdev *cdev = file_inode(ioucmd->file)->i_cdev; + struct nvme_ns_head *head = container_of(cdev, struct nvme_ns_head, cdev); + int srcu_idx = srcu_read_lock(&head->srcu); + struct nvme_ns *ns = nvme_find_path(head); + struct bio *bio; + int ret = 0; + struct request_queue *q; + + if (ns) { + rcu_read_lock(); + bio = READ_ONCE(ioucmd->cookie); + q = ns->queue; + if (test_bit(QUEUE_FLAG_POLL, &q->queue_flags) && bio + && bio->bi_bdev) + ret = bio_poll(bio, NULL, 0); + rcu_read_unlock(); + } + srcu_read_unlock(&head->srcu, srcu_idx); + return ret; +} #endif /* CONFIG_NVME_MULTIPATH */ int nvme_dev_uring_cmd(struct io_uring_cmd *ioucmd, unsigned int issue_flags) @@ -692,6 +751,10 @@ int nvme_dev_uring_cmd(struct io_uring_cmd *ioucmd, unsigned int issue_flags) struct nvme_ctrl *ctrl = ioucmd->file->private_data; int ret; + /* IOPOLL not supported yet */ + if (issue_flags & IO_URING_F_IOPOLL) + return -EOPNOTSUPP; + ret = nvme_uring_cmd_checks(issue_flags); if (ret) return ret; diff --git a/drivers/nvme/host/multipath.c b/drivers/nvme/host/multipath.c index 6ef497c75a16..00f2f81e20fa 100644 --- a/drivers/nvme/host/multipath.c +++ b/drivers/nvme/host/multipath.c @@ -439,6 +439,7 @@ static const struct file_operations nvme_ns_head_chr_fops = { .unlocked_ioctl = nvme_ns_head_chr_ioctl, .compat_ioctl = compat_ptr_ioctl, .uring_cmd = nvme_ns_head_chr_uring_cmd, + .uring_cmd_iopoll = nvme_ns_head_chr_uring_cmd_iopoll, }; static int nvme_add_ns_head_cdev(struct nvme_ns_head *head) diff --git a/drivers/nvme/host/nvme.h b/drivers/nvme/host/nvme.h index bdc0ff7ed9ab..3f2d3dda6e6c 100644 --- a/drivers/nvme/host/nvme.h +++ b/drivers/nvme/host/nvme.h @@ -821,6 +821,8 @@ long nvme_ns_head_chr_ioctl(struct file *file, unsigned int cmd, unsigned long arg); long nvme_dev_ioctl(struct file *file, unsigned int cmd, unsigned long arg); +int nvme_ns_chr_uring_cmd_iopoll(struct io_uring_cmd *ioucmd); +int nvme_ns_head_chr_uring_cmd_iopoll(struct io_uring_cmd *ioucmd); int nvme_ns_chr_uring_cmd(struct io_uring_cmd *ioucmd, unsigned int issue_flags); int nvme_ns_head_chr_uring_cmd(struct io_uring_cmd *ioucmd,