| Message ID | fc43826d510dc75de83d81161ca03e2688515686.1682701588.git.asml.silence@gmail.com (mailing list archive) |
|---|---|
| State | New |
| Headers | show |
| Series | Rethinking splice | expand |
On Sun, Apr 30, 2023 at 10:35:29AM +0100, Pavel Begunkov wrote: > There are several problems with splice requests, aka IORING_OP_SPLICE: > 1) They are always executed by a worker thread, which is a slow path, > as we don't have any reliable way to execute it NOWAIT. > 2) It can't easily poll for data, as there are 2 files it operates on. > It would either need to track what file to poll or poll both of them, > in both cases it'll be a mess and add lot of overhead. > 3) It has to have pipes in the middle, which adds overhead and is not > great from the uapi design perspective when it goes for io_uring > requests. > 4) We want to operate with spliced data as with a normal buffer, i.e. > write / send / etc. data as normally while it's zerocopy. > > It can partially be solved, but the root cause is a suboptimal for > io_uring design of IORING_OP_SPLICE. Introduce a new request type > called IORING_OP_GET_BUF, inspired by splice(2) as well as other > proposals like fused requests. The main idea is to use io_uring's > registered buffers as the middle man instead of pipes. Once a buffer > is fetched / spliced from a file using a new fops callback > ->iou_get_buf, it's installed as a registered buffers and can be used > by all operations supporting the feature. > > Once the userspace releases the buffer, io_uring will wait for all > requests using the buffer to complete and then use a file provided > callback ->release() to return the buffer back. It operates on the In the commit of "io_uring: add an example for buf-get op", I don't see any code to release the buffer, can you explain it in details about how to release the buffer in userspace? And add it in your example? Here I guess the ->release() is called in the following code path: io_buffer_unmap io_rsrc_buf_put io_rsrc_put_work io_rsrc_node_ref_zero io_put_rsrc_node If it is true, what is counter-pair code for io_put_rsrc_node()? So far, only see io_req_set_rsrc_node() is called from io_file_get_fixed(), is it needed for consumer OP of the buffer? Also io_buffer_unmap() is called after io_rsrc_node's reference drops to zero, which means ->release() isn't called after all its consumer(s) are done given io_rsrc_node is shared by in-flight requests. If it is true, this way will increase buffer lifetime a lot. ublk zero copy needs to call ->release() immediately after all consumers are done, because the ublk disk request won't be completed until the buffer is released(the buffer actually belongs to ublk block request). Also the usage in liburing example needs two extra syscall(io_uring_enter) for handling one IO, not take into account the "release OP". IMO, this way makes application more complicated, also might perform worse: 1) for ublk zero copy, the original IO just needs one OP, but now it takes three OPs, so application has to take coroutine for applying 3 stages batch submission(GET_BUF, IO, release buffer) since IO_LINK can't or not suggested to be used. In case of low QD, batch size is reduced much, and performance may hurt because IOs/syscall is 1/3 of fused command. 2) GET_BUF OP is separated from the consumer OP, this way may cause more cache miss, and I know this way is for avoiding IO_LINK. I'd understand the approach first before using it to implement ublk zero copy and comparing its performance with fused command. Thanks, Ming
On Tue, May 02, 2023 at 10:57:30PM +0800, Ming Lei wrote: > On Sun, Apr 30, 2023 at 10:35:29AM +0100, Pavel Begunkov wrote: > > There are several problems with splice requests, aka IORING_OP_SPLICE: > > 1) They are always executed by a worker thread, which is a slow path, > > as we don't have any reliable way to execute it NOWAIT. > > 2) It can't easily poll for data, as there are 2 files it operates on. > > It would either need to track what file to poll or poll both of them, > > in both cases it'll be a mess and add lot of overhead. > > 3) It has to have pipes in the middle, which adds overhead and is not > > great from the uapi design perspective when it goes for io_uring > > requests. > > 4) We want to operate with spliced data as with a normal buffer, i.e. > > write / send / etc. data as normally while it's zerocopy. > > > > It can partially be solved, but the root cause is a suboptimal for > > io_uring design of IORING_OP_SPLICE. Introduce a new request type > > called IORING_OP_GET_BUF, inspired by splice(2) as well as other > > proposals like fused requests. The main idea is to use io_uring's > > registered buffers as the middle man instead of pipes. Once a buffer > > is fetched / spliced from a file using a new fops callback > > ->iou_get_buf, it's installed as a registered buffers and can be used > > by all operations supporting the feature. > > > > Once the userspace releases the buffer, io_uring will wait for all > > requests using the buffer to complete and then use a file provided > > callback ->release() to return the buffer back. It operates on the > > In the commit of "io_uring: add an example for buf-get op", I don't see > any code to release the buffer, can you explain it in details about how > to release the buffer in userspace? And add it in your example? > > Here I guess the ->release() is called in the following code path: > > io_buffer_unmap > io_rsrc_buf_put > io_rsrc_put_work > io_rsrc_node_ref_zero > io_put_rsrc_node > > If it is true, what is counter-pair code for io_put_rsrc_node()? It is io_req_set_rsrc_node() called from io_prep_rw() and io_send_zc_prep(). So only fs IO and net zc supports it, however the fixed buffer is used in ->prep(), that means this way can't work when the rw/net_send_zc OP follows IORING_OP_GET_BUF by IO_LINK. This way is one big defect because IOSQE_IO_LINK does simplify application a lot. Thanks, Ming
On 5/2/23 15:57, Ming Lei wrote: > On Sun, Apr 30, 2023 at 10:35:29AM +0100, Pavel Begunkov wrote: >> There are several problems with splice requests, aka IORING_OP_SPLICE: >> 1) They are always executed by a worker thread, which is a slow path, >> as we don't have any reliable way to execute it NOWAIT. >> 2) It can't easily poll for data, as there are 2 files it operates on. >> It would either need to track what file to poll or poll both of them, >> in both cases it'll be a mess and add lot of overhead. >> 3) It has to have pipes in the middle, which adds overhead and is not >> great from the uapi design perspective when it goes for io_uring >> requests. >> 4) We want to operate with spliced data as with a normal buffer, i.e. >> write / send / etc. data as normally while it's zerocopy. >> >> It can partially be solved, but the root cause is a suboptimal for >> io_uring design of IORING_OP_SPLICE. Introduce a new request type >> called IORING_OP_GET_BUF, inspired by splice(2) as well as other >> proposals like fused requests. The main idea is to use io_uring's >> registered buffers as the middle man instead of pipes. Once a buffer >> is fetched / spliced from a file using a new fops callback >> ->iou_get_buf, it's installed as a registered buffers and can be used >> by all operations supporting the feature. >> >> Once the userspace releases the buffer, io_uring will wait for all >> requests using the buffer to complete and then use a file provided >> callback ->release() to return the buffer back. It operates on the > > In the commit of "io_uring: add an example for buf-get op", I don't see > any code to release the buffer, can you explain it in details about how > to release the buffer in userspace? And add it in your example? Sure, we need to add buf updates via request. Particularly, in this RFC, the removal from the table was happening in io_install_buffer() by one of the test-only patches, the "remove previous entry on update" style as it's with files. Then it's released with the last ref put, either on removal with a request like: io_free_batch_list() io_req_put_rsrc_locked() ... > Here I guess the ->release() is called in the following code path: > > io_buffer_unmap > io_rsrc_buf_put > io_rsrc_put_work > io_rsrc_node_ref_zero > io_put_rsrc_node > > If it is true, what is counter-pair code for io_put_rsrc_node()? > So far, only see io_req_set_rsrc_node() is called from > io_file_get_fixed(), is it needed for consumer OP of the buffer? > > Also io_buffer_unmap() is called after io_rsrc_node's reference drops > to zero, which means ->release() isn't called after all its consumer(s) > are done given io_rsrc_node is shared by in-flight requests. If it is > true, this way will increase buffer lifetime a lot. That's true. It's not a new downside, so might make more sense to do counting per rsrc (file, buffer), which is not so bad for now, but would be a bit concerning if we grow the number of rsrc types. > ublk zero copy needs to call ->release() immediately after all > consumers are done, because the ublk disk request won't be completed > until the buffer is released(the buffer actually belongs to ublk block request). > > Also the usage in liburing example needs two extra syscall(io_uring_enter) for > handling one IO, not take into account the "release OP". IMO, this way makes Something is amiss here. It's 3 requests, which means 3 syscalls if you send requests separately (each step can be batch more requests), or 1 syscall if you link them together. There is an example using links for 2 requests in the test case. > application more complicated, also might perform worse: > > 1) for ublk zero copy, the original IO just needs one OP, but now it > takes three OPs, so application has to take coroutine for applying Perhaps, you mean two requests for fused, IORING_OP_FUSED_CMD + IO request, vs three for IORING_OP_GET_BUF. There might be some sort of auto-remove on use, making it two requests, but that seems a bit ugly. > 3 stages batch submission(GET_BUF, IO, release buffer) since IO_LINK can't > or not suggested to be used. In case of low QD, batch size is reduced much, > and performance may hurt because IOs/syscall is 1/3 of fused command. I'm not a big fan of links for their inflexibility, but it can be used. The point is rather it's better not to be the only way to use the feature as we may need to stop in the middle, return control to the userspace and let it handle errors, do data processing and so on. The latter may need a partial memcpy() into the userspace, e.g. copy a handful bytes of headers to decide what to do with the rest of data. I deem fused cmds to be a variant of linking, so it's rather with it you link 2 requests vs optionally linking 3 with this patchset. > 2) GET_BUF OP is separated from the consumer OP, this way may cause > more cache miss, and I know this way is for avoiding IO_LINK. > > I'd understand the approach first before using it to implement ublk zero copy > and comparing its performance with fused command.
On Wed, May 03, 2023 at 03:54:02PM +0100, Pavel Begunkov wrote: > On 5/2/23 15:57, Ming Lei wrote: > > On Sun, Apr 30, 2023 at 10:35:29AM +0100, Pavel Begunkov wrote: > > > There are several problems with splice requests, aka IORING_OP_SPLICE: > > > 1) They are always executed by a worker thread, which is a slow path, > > > as we don't have any reliable way to execute it NOWAIT. > > > 2) It can't easily poll for data, as there are 2 files it operates on. > > > It would either need to track what file to poll or poll both of them, > > > in both cases it'll be a mess and add lot of overhead. > > > 3) It has to have pipes in the middle, which adds overhead and is not > > > great from the uapi design perspective when it goes for io_uring > > > requests. > > > 4) We want to operate with spliced data as with a normal buffer, i.e. > > > write / send / etc. data as normally while it's zerocopy. > > > > > > It can partially be solved, but the root cause is a suboptimal for > > > io_uring design of IORING_OP_SPLICE. Introduce a new request type > > > called IORING_OP_GET_BUF, inspired by splice(2) as well as other > > > proposals like fused requests. The main idea is to use io_uring's > > > registered buffers as the middle man instead of pipes. Once a buffer > > > is fetched / spliced from a file using a new fops callback > > > ->iou_get_buf, it's installed as a registered buffers and can be used > > > by all operations supporting the feature. > > > > > > Once the userspace releases the buffer, io_uring will wait for all > > > requests using the buffer to complete and then use a file provided > > > callback ->release() to return the buffer back. It operates on the > > > > In the commit of "io_uring: add an example for buf-get op", I don't see > > any code to release the buffer, can you explain it in details about how > > to release the buffer in userspace? And add it in your example? > > Sure, we need to add buf updates via request. I guess it can't be buf update, here we need to release buf. At least ublk needs to release the buffer explicitly after all consumers are done with the buffer registered by IORING_OP_GET_BUF, when there may not be any new buffer to be provided, and the buffer is per-IO for each io request from /dev/ublkbN. > > Particularly, in this RFC, the removal from the table was happening > in io_install_buffer() by one of the test-only patches, the "remove > previous entry on update" style as it's with files. Then it's > released with the last ref put, either on removal with a request > like: > > io_free_batch_list() > io_req_put_rsrc_locked() > ... > > > Here I guess the ->release() is called in the following code path: > > > > io_buffer_unmap > > io_rsrc_buf_put > > io_rsrc_put_work > > io_rsrc_node_ref_zero > > io_put_rsrc_node > > > > If it is true, what is counter-pair code for io_put_rsrc_node()? > > So far, only see io_req_set_rsrc_node() is called from > > io_file_get_fixed(), is it needed for consumer OP of the buffer? > > > > Also io_buffer_unmap() is called after io_rsrc_node's reference drops > > to zero, which means ->release() isn't called after all its consumer(s) > > are done given io_rsrc_node is shared by in-flight requests. If it is > > true, this way will increase buffer lifetime a lot. > > That's true. It's not a new downside, so might make more sense > to do counting per rsrc (file, buffer), which is not so bad for > now, but would be a bit concerning if we grow the number of rsrc > types. It may not be one deal for current fixed file user, which isn't usually updated in fast path. But IORING_OP_GET_BUF is supposed to be for fast path, this delay release is really one problem. Cause if there is any new buffer provided & consumed, old buffer can't be released any more. And this way can't be used in ublk zero copy, let me share the ublk model a bit: 1) one batch ublk blk io requests(/dev/ublkbN) are coming, and batch size is often QD of workload on /dev/ublkbN, and batch size could be 1 or more. 2) ublk driver notifies the blk io requests via io_uring command completion 3) ublk server starts to handle this batch of io requests, by calling IORING_OP_GET_BUF on /dev/ublkcN & normal OPs(rw, net recv/send, ...) for each request in this batch 4) new batch of ublk blk io requests can come when handling the previous batch of io requests, then the old buffer release is delayed until new batch of ublk io requests are completed. > > > ublk zero copy needs to call ->release() immediately after all > > consumers are done, because the ublk disk request won't be completed > > until the buffer is released(the buffer actually belongs to ublk block request). > > > > Also the usage in liburing example needs two extra syscall(io_uring_enter) for > > handling one IO, not take into account the "release OP". IMO, this way makes > > Something is amiss here. It's 3 requests, which means 3 syscalls > if you send requests separately (each step can be batch more > requests), or 1 syscall if you link them together. There is an > example using links for 2 requests in the test case. See the final comment about link support. > > > application more complicated, also might perform worse: > > > > 1) for ublk zero copy, the original IO just needs one OP, but now it > > takes three OPs, so application has to take coroutine for applying > > Perhaps, you mean two requests for fused, IORING_OP_FUSED_CMD + IO > request, vs three for IORING_OP_GET_BUF. There might be some sort of > auto-remove on use, making it two requests, but that seems a bit ugly. The most important part is that IORING_OP_GET_BUF adds two extra wait: 1) one io_uring_enter() is required after submitting IORING_OP_GET_BUF 2) another io_uring_enter() is needed after submitting buffer consumer OPs, before calling buffer release OP(not added yet in your patchset) The two waits not only causes application more complicated, but also hurts performance, cause IOs/syscall is reduced. People loves io_uring because it is really async io model, such as, all kinds of IO can be submitted in single context, and wait in single io_uring_enter(). > > > 3 stages batch submission(GET_BUF, IO, release buffer) since IO_LINK can't > > or not suggested to be used. In case of low QD, batch size is reduced much, > > and performance may hurt because IOs/syscall is 1/3 of fused command. > > I'm not a big fan of links for their inflexibility, but it can be > used. The point is rather it's better not to be the only way to > use the feature as we may need to stop in the middle, return LINK always support to stop in the middle, right? > control to the userspace and let it handle errors, do data processing > and so on. The latter may need a partial memcpy() into the userspace, > e.g. copy a handful bytes of headers to decide what to do with the > rest of data. At least this patchset doesn't work with IO_LINK, please see my previous reply because the current rw/net zc retrieves fixed buffer in ->prep(). Yeah, the problem can be addressed by moving the buffer retrieving into ->issue(). But my concern is more about easy use and performance: 1) with io_link, the extra two waits(after IORING_OP_GET_BUF and before IORING_OP_GET_BUF) required can be saved, then application doesn't need coroutine or similar trick for avoiding the extra wait handling. 2) but if three OPs uses the buffer registered by IORING_OP_GET_BUF, the three OPs have to be linked one by one, then all three have to be be submitted one after the previous one is completed, performance is hurt a lot. > > I deem fused cmds to be a variant of linking, so it's rather with > it you link 2 requests vs optionally linking 3 with this patchset. IMO, one extra 64byte touching may slow things a little, but I think it won't be big deal, what really matters is that easy use of the interface and performance. Fused command solves both: allow to submit the consumer OPs concurrently; avoid multiple wait point and make application easier to implement For example: 1) fused command based application: (every IO command takes 1 syscall usually) for each reapped CQE: - if CQE is for io command from /dev/ublkc: - handle it by allocating/queuing three SQEs: - one SQE is for primary command, - the 2nd SQE is for READ to file 1 - the 3rd SQE is for READ to file 2 - if CQE is for primary command or normal OP: - just check if the whole fused command is completed - if yes, notify that the current io command is completed (the current io request from /dev/ublkbN will be completed in ublk driver, and this io command is queued for incoming request) - if any err, handle the err: retry or terminate, similar with handling for 'yes' - otherwise, just return io_uring_enter(); //single wait point 2) IORING_OP_GET_BUF based application(every IO command needs two syscall) for each reapped CQE: - if CQE is for io command from /dev/ublkc: - handle it by: queuing IORING_OP_GET_BUF recoding this io needs to be hanlded io_uring_enter() //wait until all IORING_OP_GET_BUF are done for each IO recorded in above, queue two new RW OPs(one for file1, one for file2) io_uring_enter() //wait until the above two OPs are done for each reapped CQE: release the buffer registered by IORING_OP_GET_BUF Or coroutine based approach: (every io command needs two syscalls) for each reaaped CQE: - if CQE is for command from /dev/ublkc: - queue IORING_OP_GET_BUF - corouine wait: until IORING_OP_GET_BUF is done - queue two RW OPs with the registered buffer - coroutine wait: until all above OP is done - release buffer registered by IORING_OP_GET_BUF - otherwise: - call related co routine wakeup handler for advancing in above co routine wait point io_uring_enter(); coroutine based approach may look clean and easier to implement, but it depends on language support, and still needs extra care, also stackless coroutine isn't easy to use. Thanks, Ming
On 5/4/23 03:06, Ming Lei wrote: > On Wed, May 03, 2023 at 03:54:02PM +0100, Pavel Begunkov wrote: >> On 5/2/23 15:57, Ming Lei wrote: >>> On Sun, Apr 30, 2023 at 10:35:29AM +0100, Pavel Begunkov wrote: >>>> There are several problems with splice requests, aka IORING_OP_SPLICE: >>>> 1) They are always executed by a worker thread, which is a slow path, >>>> as we don't have any reliable way to execute it NOWAIT. >>>> 2) It can't easily poll for data, as there are 2 files it operates on. >>>> It would either need to track what file to poll or poll both of them, >>>> in both cases it'll be a mess and add lot of overhead. >>>> 3) It has to have pipes in the middle, which adds overhead and is not >>>> great from the uapi design perspective when it goes for io_uring >>>> requests. >>>> 4) We want to operate with spliced data as with a normal buffer, i.e. >>>> write / send / etc. data as normally while it's zerocopy. >>>> >>>> It can partially be solved, but the root cause is a suboptimal for >>>> io_uring design of IORING_OP_SPLICE. Introduce a new request type >>>> called IORING_OP_GET_BUF, inspired by splice(2) as well as other >>>> proposals like fused requests. The main idea is to use io_uring's >>>> registered buffers as the middle man instead of pipes. Once a buffer >>>> is fetched / spliced from a file using a new fops callback >>>> ->iou_get_buf, it's installed as a registered buffers and can be used >>>> by all operations supporting the feature. >>>> >>>> Once the userspace releases the buffer, io_uring will wait for all >>>> requests using the buffer to complete and then use a file provided >>>> callback ->release() to return the buffer back. It operates on the >>> >>> In the commit of "io_uring: add an example for buf-get op", I don't see >>> any code to release the buffer, can you explain it in details about how >>> to release the buffer in userspace? And add it in your example? >> >> Sure, we need to add buf updates via request. > > I guess it can't be buf update, here we need to release buf. At least > ublk needs to release the buffer explicitly after all consumers are done > with the buffer registered by IORING_OP_GET_BUF, when there may not be > any new buffer to be provided, and the buffer is per-IO for each io > request from /dev/ublkbN. By "updates" I usually mean removals as well, just like IORING_REGISTER_BUFFERS_UPDATE with iov_base == 0 would remove the buffer. >> Particularly, in this RFC, the removal from the table was happening >> in io_install_buffer() by one of the test-only patches, the "remove >> previous entry on update" style as it's with files. Then it's >> released with the last ref put, either on removal with a request >> like: >> >> io_free_batch_list() >> io_req_put_rsrc_locked() >> ... >> >>> Here I guess the ->release() is called in the following code path: >>> >>> io_buffer_unmap >>> io_rsrc_buf_put >>> io_rsrc_put_work >>> io_rsrc_node_ref_zero >>> io_put_rsrc_node >>> >>> If it is true, what is counter-pair code for io_put_rsrc_node()? >>> So far, only see io_req_set_rsrc_node() is called from >>> io_file_get_fixed(), is it needed for consumer OP of the buffer? >>> >>> Also io_buffer_unmap() is called after io_rsrc_node's reference drops >>> to zero, which means ->release() isn't called after all its consumer(s) >>> are done given io_rsrc_node is shared by in-flight requests. If it is >>> true, this way will increase buffer lifetime a lot. >> >> That's true. It's not a new downside, so might make more sense >> to do counting per rsrc (file, buffer), which is not so bad for >> now, but would be a bit concerning if we grow the number of rsrc >> types. > > It may not be one deal for current fixed file user, which isn't usually > updated in fast path. With opening/accepting right into io_uring skipping normal file tables and cases like open->read->close, it's rather somewhat of medium hotness, definitely not a slow path. And the problem affects it as well > But IORING_OP_GET_BUF is supposed to be for fast path, this delay > release is really one problem. Cause if there is any new buffer provided & > consumed, old buffer can't be released any more. And this way can't > be used in ublk zero copy, let me share the ublk model a bit: > > 1) one batch ublk blk io requests(/dev/ublkbN) are coming, and batch size is > often QD of workload on /dev/ublkbN, and batch size could be 1 or more. > > 2) ublk driver notifies the blk io requests via io_uring command > completion > > 3) ublk server starts to handle this batch of io requests, by calling > IORING_OP_GET_BUF on /dev/ublkcN & normal OPs(rw, net recv/send, ...) > for each request in this batch > > 4) new batch of ublk blk io requests can come when handling the previous > batch of io requests, then the old buffer release is delayed until new > batch of ublk io requests are completed. > >> >>> ublk zero copy needs to call ->release() immediately after all >>> consumers are done, because the ublk disk request won't be completed >>> until the buffer is released(the buffer actually belongs to ublk block request). >>> >>> Also the usage in liburing example needs two extra syscall(io_uring_enter) for >>> handling one IO, not take into account the "release OP". IMO, this way makes >> >> Something is amiss here. It's 3 requests, which means 3 syscalls >> if you send requests separately (each step can be batch more >> requests), or 1 syscall if you link them together. There is an >> example using links for 2 requests in the test case. > > See the final comment about link support. > >> >>> application more complicated, also might perform worse: >>> >>> 1) for ublk zero copy, the original IO just needs one OP, but now it >>> takes three OPs, so application has to take coroutine for applying >> >> Perhaps, you mean two requests for fused, IORING_OP_FUSED_CMD + IO >> request, vs three for IORING_OP_GET_BUF. There might be some sort of >> auto-remove on use, making it two requests, but that seems a bit ugly. > > The most important part is that IORING_OP_GET_BUF adds two extra wait: > > 1) one io_uring_enter() is required after submitting IORING_OP_GET_BUF > > 2) another io_uring_enter() is needed after submitting buffer consumer > OPs, before calling buffer release OP(not added yet in your patchset) > > The two waits not only causes application more complicated, but also > hurts performance, cause IOs/syscall is reduced. People loves io_uring > because it is really async io model, such as, all kinds of IO can be > submitted in single context, and wait in single io_uring_enter(). *Pseudo code* N = get_free_buffer_slot(); sqe1 = prep_getbuf_sqe(buf_idx = N); sqe1->flags |= F_LINK; sqe2 = prep_write_sqe(buf_idx = N); sqe2->flags |= F_LINK; sqe3 = prep_release_buf_sqe(buf_idx = N); sqe3->flags |= F_LINK; submit_and_wait(nr_wait=3); That should work, there is only 1 syscall. We can also play with SKIP_CQE_SUCCESS and/or HARDLINK to fold 3 cqes into 1. >>> 3 stages batch submission(GET_BUF, IO, release buffer) since IO_LINK can't >>> or not suggested to be used. In case of low QD, batch size is reduced much, >>> and performance may hurt because IOs/syscall is 1/3 of fused command. >> >> I'm not a big fan of links for their inflexibility, but it can be >> used. The point is rather it's better not to be the only way to >> use the feature as we may need to stop in the middle, return > > LINK always support to stop in the middle, right? Normal links will stop execution on "error", which is usually cqe->res < 0, but read/write will also break the link on short IO, and the short IO behaviour of send/recv will depend on MSG_WAITALL. IOSQE_IO_HARDLINK will continue executing linked requests even with prior errors. >> control to the userspace and let it handle errors, do data processing >> and so on. The latter may need a partial memcpy() into the userspace, >> e.g. copy a handful bytes of headers to decide what to do with the >> rest of data. > > At least this patchset doesn't work with IO_LINK, please see my previous > reply because the current rw/net zc retrieves fixed buffer in ->prep(). > > Yeah, the problem can be addressed by moving the buffer retrieving into > ->issue(). Right, one of the test patches does exactly that (fwiw, not tested seriously), and as it was previously done for files. It won't be a big problem. > But my concern is more about easy use and performance: > > 1) with io_link, the extra two waits(after IORING_OP_GET_BUF and before IORING_OP_GET_BUF) > required can be saved, then application doesn't need coroutine or > similar trick for avoiding the extra wait handling. It shouldn't need that, see above. > 2) but if three OPs uses the buffer registered by IORING_OP_GET_BUF, the three > OPs have to be linked one by one, then all three have to be be submitted one > after the previous one is completed, performance is hurt a lot. In general, it sounds like a generic feature and should be as such. But I agree with the sentiment, that's exactly why I was saying that they're not perfectly flexible and don't cover all the cases, but the same can be said for the 1->N kinds of dependency. And even with a completely configurable graphs of requests there will be questions like "how to take result from one request and give it to another with a random meaning, i.e. as a buffer/fd/size/ index/whatnot", and there are more complicated examples. I don't see any good scalable way for that without programmability. It appeared before that it was faster to return back to the userspace than using BPF, might be worth to test it again, especially with DEFER_TASKRUN and recent optimisations. >> I deem fused cmds to be a variant of linking, so it's rather with >> it you link 2 requests vs optionally linking 3 with this patchset. > > IMO, one extra 64byte touching may slow things a little, but I think it Fwiw, the overhead comes from all io_uring hops a request goes through, e.g. init, ->prep, ->issue, free, etc. I'd love to see it getting even slimmer, and there are definitely spots that can be optimised. > won't be big deal, what really matters is that easy use of the interface > and performance. Which should also be balanced with flexibility and not only. > Fused command solves both: allow to submit the consumer OPs concurrently; > avoid multiple wait point and make application easier to implement > > For example: > > 1) fused command based application: (every IO command takes 1 syscall usually) > > for each reapped CQE: > - if CQE is for io command from /dev/ublkc: > - handle it by allocating/queuing three SQEs: > - one SQE is for primary command, > - the 2nd SQE is for READ to file 1 > - the 3rd SQE is for READ to file 2 > - if CQE is for primary command or normal OP: > - just check if the whole fused command is completed > - if yes, notify that the current io command is > completed (the current io request from /dev/ublkbN will > be completed in ublk driver, and this io command is > queued for incoming request) > - if any err, handle the err: retry or terminate, > similar with handling for 'yes' > - otherwise, just return > io_uring_enter(); //single wait point > > 2) IORING_OP_GET_BUF based application(every IO command needs two syscall) > > for each reapped CQE: > - if CQE is for io command from /dev/ublkc: > - handle it by: > queuing IORING_OP_GET_BUF > recoding this io needs to be hanlded > > io_uring_enter() //wait until all IORING_OP_GET_BUF are done > > for each IO recorded in above, queue two new RW OPs(one for file1, one for file2) > > io_uring_enter() //wait until the above two OPs are done > > for each reapped CQE: > release the buffer registered by IORING_OP_GET_BUF > > Or coroutine based approach: (every io command needs two syscalls) > > for each reaaped CQE: > - if CQE is for command from /dev/ublkc: > - queue IORING_OP_GET_BUF > - corouine wait: until IORING_OP_GET_BUF is done > - queue two RW OPs with the registered buffer > - coroutine wait: until all above OP is done > - release buffer registered by IORING_OP_GET_BUF > - otherwise: > - call related co routine wakeup handler for advancing in > above co routine wait point > io_uring_enter(); > > coroutine based approach may look clean and easier to implement, but > it depends on language support, and still needs extra care, also stackless > coroutine isn't easy to use.
On Mon, May 08, 2023 at 03:30:55AM +0100, Pavel Begunkov wrote: > On 5/4/23 03:06, Ming Lei wrote: > > On Wed, May 03, 2023 at 03:54:02PM +0100, Pavel Begunkov wrote: > > > On 5/2/23 15:57, Ming Lei wrote: > > > > On Sun, Apr 30, 2023 at 10:35:29AM +0100, Pavel Begunkov wrote: > > > > > There are several problems with splice requests, aka IORING_OP_SPLICE: > > > > > 1) They are always executed by a worker thread, which is a slow path, > > > > > as we don't have any reliable way to execute it NOWAIT. > > > > > 2) It can't easily poll for data, as there are 2 files it operates on. > > > > > It would either need to track what file to poll or poll both of them, > > > > > in both cases it'll be a mess and add lot of overhead. > > > > > 3) It has to have pipes in the middle, which adds overhead and is not > > > > > great from the uapi design perspective when it goes for io_uring > > > > > requests. > > > > > 4) We want to operate with spliced data as with a normal buffer, i.e. > > > > > write / send / etc. data as normally while it's zerocopy. > > > > > > > > > > It can partially be solved, but the root cause is a suboptimal for > > > > > io_uring design of IORING_OP_SPLICE. Introduce a new request type > > > > > called IORING_OP_GET_BUF, inspired by splice(2) as well as other > > > > > proposals like fused requests. The main idea is to use io_uring's > > > > > registered buffers as the middle man instead of pipes. Once a buffer > > > > > is fetched / spliced from a file using a new fops callback > > > > > ->iou_get_buf, it's installed as a registered buffers and can be used > > > > > by all operations supporting the feature. > > > > > > > > > > Once the userspace releases the buffer, io_uring will wait for all > > > > > requests using the buffer to complete and then use a file provided > > > > > callback ->release() to return the buffer back. It operates on the > > > > > > > > In the commit of "io_uring: add an example for buf-get op", I don't see > > > > any code to release the buffer, can you explain it in details about how > > > > to release the buffer in userspace? And add it in your example? > > > > > > Sure, we need to add buf updates via request. > > > > I guess it can't be buf update, here we need to release buf. At least > > ublk needs to release the buffer explicitly after all consumers are done > > with the buffer registered by IORING_OP_GET_BUF, when there may not be > > any new buffer to be provided, and the buffer is per-IO for each io > > request from /dev/ublkbN. > > By "updates" I usually mean removals as well, just like > IORING_REGISTER_BUFFERS_UPDATE with iov_base == 0 would remove > the buffer. OK. > > > > > Particularly, in this RFC, the removal from the table was happening > > > in io_install_buffer() by one of the test-only patches, the "remove > > > previous entry on update" style as it's with files. Then it's > > > released with the last ref put, either on removal with a request > > > like: This way can't work for ublk, we definitely need to release the buffer immediately after it is consumed, otherwise the ublk block request won't be completed. > > > > > > io_free_batch_list() > > > io_req_put_rsrc_locked() > > > ... > > > > > > > Here I guess the ->release() is called in the following code path: > > > > > > > > io_buffer_unmap > > > > io_rsrc_buf_put > > > > io_rsrc_put_work > > > > io_rsrc_node_ref_zero > > > > io_put_rsrc_node > > > > > > > > If it is true, what is counter-pair code for io_put_rsrc_node()? > > > > So far, only see io_req_set_rsrc_node() is called from > > > > io_file_get_fixed(), is it needed for consumer OP of the buffer? > > > > > > > > Also io_buffer_unmap() is called after io_rsrc_node's reference drops > > > > to zero, which means ->release() isn't called after all its consumer(s) > > > > are done given io_rsrc_node is shared by in-flight requests. If it is > > > > true, this way will increase buffer lifetime a lot. > > > > > > That's true. It's not a new downside, so might make more sense > > > to do counting per rsrc (file, buffer), which is not so bad for > > > now, but would be a bit concerning if we grow the number of rsrc > > > types. > > > > It may not be one deal for current fixed file user, which isn't usually > > updated in fast path. > > With opening/accepting right into io_uring skipping normal file > tables and cases like open->read->close, it's rather somewhat of > medium hotness, definitely not a slow path. And the problem affects > it as well OK, then it becomes more rationale to remove the delayed release by using io_rsrc_node, I am wondering why not add reference counter to the buffer which needs to be used in such fast path? > > But IORING_OP_GET_BUF is supposed to be for fast path, this delay > > release is really one problem. Cause if there is any new buffer provided & > > consumed, old buffer can't be released any more. And this way can't > > be used in ublk zero copy, let me share the ublk model a bit: > > > > 1) one batch ublk blk io requests(/dev/ublkbN) are coming, and batch size is > > often QD of workload on /dev/ublkbN, and batch size could be 1 or more. > > > > 2) ublk driver notifies the blk io requests via io_uring command > > completion > > > > 3) ublk server starts to handle this batch of io requests, by calling > > IORING_OP_GET_BUF on /dev/ublkcN & normal OPs(rw, net recv/send, ...) > > for each request in this batch > > > > 4) new batch of ublk blk io requests can come when handling the previous > > batch of io requests, then the old buffer release is delayed until new > > batch of ublk io requests are completed. > > > > > > > > > ublk zero copy needs to call ->release() immediately after all > > > > consumers are done, because the ublk disk request won't be completed > > > > until the buffer is released(the buffer actually belongs to ublk block request). > > > > > > > > Also the usage in liburing example needs two extra syscall(io_uring_enter) for > > > > handling one IO, not take into account the "release OP". IMO, this way makes > > > > > > Something is amiss here. It's 3 requests, which means 3 syscalls > > > if you send requests separately (each step can be batch more > > > requests), or 1 syscall if you link them together. There is an > > > example using links for 2 requests in the test case. > > > > See the final comment about link support. > > > > > > > > > application more complicated, also might perform worse: > > > > > > > > 1) for ublk zero copy, the original IO just needs one OP, but now it > > > > takes three OPs, so application has to take coroutine for applying > > > > > > Perhaps, you mean two requests for fused, IORING_OP_FUSED_CMD + IO > > > request, vs three for IORING_OP_GET_BUF. There might be some sort of > > > auto-remove on use, making it two requests, but that seems a bit ugly. > > > > The most important part is that IORING_OP_GET_BUF adds two extra wait: > > > > 1) one io_uring_enter() is required after submitting IORING_OP_GET_BUF > > > > 2) another io_uring_enter() is needed after submitting buffer consumer > > OPs, before calling buffer release OP(not added yet in your patchset) > > > > The two waits not only causes application more complicated, but also > > hurts performance, cause IOs/syscall is reduced. People loves io_uring > > because it is really async io model, such as, all kinds of IO can be > > submitted in single context, and wait in single io_uring_enter(). > > *Pseudo code* > N = get_free_buffer_slot(); > sqe1 = prep_getbuf_sqe(buf_idx = N); > sqe1->flags |= F_LINK; > > sqe2 = prep_write_sqe(buf_idx = N); > sqe2->flags |= F_LINK; > > sqe3 = prep_release_buf_sqe(buf_idx = N); > sqe3->flags |= F_LINK; The above link shouldn't be needed. > > submit_and_wait(nr_wait=3); > > > That should work, there is only 1 syscall. We can also > play with SKIP_CQE_SUCCESS and/or HARDLINK to fold 3 cqes > into 1. Yeah, this way does simplify application and is more efficient since more OPs can be issued via single syscall, but another problem is raised, if there are multiple prep_write_sqe() in this chain and all consume this same buffer, all WRITEs have to be issued after the previous one is completed, this way is slow, and no such problem in fused command. > > > > > 3 stages batch submission(GET_BUF, IO, release buffer) since IO_LINK can't > > > > or not suggested to be used. In case of low QD, batch size is reduced much, > > > > and performance may hurt because IOs/syscall is 1/3 of fused command. > > > > > > I'm not a big fan of links for their inflexibility, but it can be > > > used. The point is rather it's better not to be the only way to > > > use the feature as we may need to stop in the middle, return > > > > LINK always support to stop in the middle, right? > > Normal links will stop execution on "error", which is usually > cqe->res < 0, but read/write will also break the link on short > IO, and the short IO behaviour of send/recv will depend on > MSG_WAITALL. IOSQE_IO_HARDLINK will continue executing linked > requests even with prior errors. I think ublk server is fine with current handling of link break. > > > > control to the userspace and let it handle errors, do data processing > > > and so on. The latter may need a partial memcpy() into the userspace, > > > e.g. copy a handful bytes of headers to decide what to do with the > > > rest of data. > > > > At least this patchset doesn't work with IO_LINK, please see my previous > > reply because the current rw/net zc retrieves fixed buffer in ->prep(). > > > > Yeah, the problem can be addressed by moving the buffer retrieving into > > ->issue(). > > Right, one of the test patches does exactly that (fwiw, not tested > seriously), and as it was previously done for files. It won't be > a big problem. OK, looks you must forget to include the patch in this series. > > > But my concern is more about easy use and performance: > > > > 1) with io_link, the extra two waits(after IORING_OP_GET_BUF and before IORING_OP_GET_BUF) > > required can be saved, then application doesn't need coroutine or > > similar trick for avoiding the extra wait handling. > > It shouldn't need that, see above. > > > 2) but if three OPs uses the buffer registered by IORING_OP_GET_BUF, the three > > OPs have to be linked one by one, then all three have to be be submitted one > > after the previous one is completed, performance is hurt a lot. > > In general, it sounds like a generic feature and should be as such. > But > > I agree with the sentiment, that's exactly why I was saying that > they're not perfectly flexible and don't cover all the cases, but > the same can be said for the 1->N kinds of dependency. And even > with a completely configurable graphs of requests there will be > questions like "how to take result from one request and give it > to another with a random meaning, i.e. as a buffer/fd/size/ > index/whatnot", and there are more complicated examples. > > I don't see any good scalable way for that without programmability. > It appeared before that it was faster to return back to the userspace > than using BPF, might be worth to test it again, especially with > DEFER_TASKRUN and recent optimisations. I don't think result from one request need to be retrieved for the following request, such as by taking your previous code: *Pseudo code* N = get_free_buffer_slot(); sqe1 = prep_getbuf_sqe(buf_idx = N); sqe1->flags |= F_LINK; sqe2 = prep_write_sqe(buf_idx = N); sqe2->flags |= F_LINK; sqe3= prep_write_sqe(buf_idx = N); sqe3->flags |= F_LINK; sqe4 = prep_release_buf_sqe(buf_idx = N); sqe4->flags |= F_LINK; After sqe1 is done, we just need to support to submit sqe2/sqe3/sqe4 concurrently, however prep_release_buf_sqe need to support async buf release. > > > > > I deem fused cmds to be a variant of linking, so it's rather with > > > it you link 2 requests vs optionally linking 3 with this patchset. > > > > IMO, one extra 64byte touching may slow things a little, but I think it > > Fwiw, the overhead comes from all io_uring hops a request goes > through, e.g. init, ->prep, ->issue, free, etc. I'd love to > see it getting even slimmer, and there are definitely spots > that can be optimised. > > > won't be big deal, what really matters is that easy use of the interface > > and performance. > > Which should also be balanced with flexibility and not only. That is why I prefer to fused command which can solve the problem in one easy way without big core code change(include interface), then generic core code still can keep its flexibility. thanks, Ming
diff --git a/include/linux/fs.h b/include/linux/fs.h index 475d88640d3d..a2528a39571f 100644 --- a/include/linux/fs.h +++ b/include/linux/fs.h @@ -1753,6 +1753,7 @@ struct dir_context { struct iov_iter; struct io_uring_cmd; +struct iou_get_buf_info; struct file_operations { struct module *owner; @@ -1798,6 +1799,7 @@ struct file_operations { int (*uring_cmd)(struct io_uring_cmd *ioucmd, unsigned int issue_flags); int (*uring_cmd_iopoll)(struct io_uring_cmd *, struct io_comp_batch *, unsigned int poll_flags); + int (*iou_get_buf)(struct file *file, struct iou_get_buf_info *); } __randomize_layout; struct inode_operations { diff --git a/include/linux/io_uring.h b/include/linux/io_uring.h index e0e7df5beefc..9564db555bab 100644 --- a/include/linux/io_uring.h +++ b/include/linux/io_uring.h @@ -30,6 +30,17 @@ struct iou_buf_desc { void *private; }; +enum { + IOU_GET_BUF_F_NOWAIT = 1, +}; + +struct iou_get_buf_info { + loff_t off; + size_t len; + unsigned flags; + struct iou_buf_desc *desc; +}; + struct io_uring_cmd { struct file *file; const void *cmd; diff --git a/include/uapi/linux/io_uring.h b/include/uapi/linux/io_uring.h index 0716cb17e436..b244215d03ad 100644 --- a/include/uapi/linux/io_uring.h +++ b/include/uapi/linux/io_uring.h @@ -223,6 +223,7 @@ enum io_uring_op { IORING_OP_URING_CMD, IORING_OP_SEND_ZC, IORING_OP_SENDMSG_ZC, + IORING_OP_GET_BUF, /* this goes last, obviously */ IORING_OP_LAST, diff --git a/io_uring/opdef.c b/io_uring/opdef.c index cca7c5b55208..d3b7144c685a 100644 --- a/io_uring/opdef.c +++ b/io_uring/opdef.c @@ -428,6 +428,13 @@ const struct io_issue_def io_issue_defs[] = { .prep = io_eopnotsupp_prep, #endif }, + [IORING_OP_GET_BUF] = { + .needs_file = 1, + .unbound_nonreg_file = 1, + .pollin = 1, + .prep = io_get_buf_prep, + .issue = io_get_buf, + }, }; @@ -648,6 +655,10 @@ const struct io_cold_def io_cold_defs[] = { .fail = io_sendrecv_fail, #endif }, + [IORING_OP_GET_BUF] = { + .name = "IORING_OP_GET_BUF", + .cleanup = io_get_buf_cleanup, + }, }; const char *io_uring_get_opcode(u8 opcode) diff --git a/io_uring/rsrc.c b/io_uring/rsrc.c index db4286b42dce..bdcd417bca87 100644 --- a/io_uring/rsrc.c +++ b/io_uring/rsrc.c @@ -89,7 +89,7 @@ static void io_put_reg_buf(struct io_ring_ctx *ctx, struct io_mapped_ubuf *imu) kvfree(imu); } -static struct io_mapped_ubuf *io_alloc_reg_buf(struct io_ring_ctx *ctx, +struct io_mapped_ubuf *io_alloc_reg_buf(struct io_ring_ctx *ctx, int nr_bvecs) { struct io_cache_entry *entry; diff --git a/io_uring/rsrc.h b/io_uring/rsrc.h index aba95bdd060e..6aaf7acb60c5 100644 --- a/io_uring/rsrc.h +++ b/io_uring/rsrc.h @@ -177,4 +177,6 @@ static inline void io_reg_buf_release(struct io_mapped_ubuf *imu) imu->desc.release(&imu->desc); } +struct io_mapped_ubuf *io_alloc_reg_buf(struct io_ring_ctx *ctx, int nr_bvecs); + #endif diff --git a/io_uring/splice.c b/io_uring/splice.c index 2a4bbb719531..3d50334caec5 100644 --- a/io_uring/splice.c +++ b/io_uring/splice.c @@ -13,6 +13,7 @@ #include "io_uring.h" #include "splice.h" +#include "rsrc.h" struct io_splice { struct file *file_out; @@ -119,3 +120,92 @@ int io_splice(struct io_kiocb *req, unsigned int issue_flags) io_req_set_res(req, ret, 0); return IOU_OK; } + +struct io_get_buf { + struct file *file; + struct io_mapped_ubuf *imu; + int max_pages; + loff_t off; + u64 len; +}; + +void io_get_buf_cleanup(struct io_kiocb *req) +{ + struct io_get_buf *gb = io_kiocb_to_cmd(req, struct io_get_buf); + struct io_mapped_ubuf *imu = gb->imu; + + if (!imu) + return; + if (imu->desc.nr_bvecs && !WARN_ON_ONCE(!imu->desc.release)) + io_reg_buf_release(imu); + + kvfree(imu); + gb->imu = NULL; +} + +int io_get_buf_prep(struct io_kiocb *req, const struct io_uring_sqe *sqe) +{ + struct io_get_buf *gb = io_kiocb_to_cmd(req, struct io_get_buf); + struct io_mapped_ubuf *imu; + int nr_pages; + + if (unlikely(sqe->splice_flags || sqe->splice_fd_in || sqe->ioprio || + sqe->addr || sqe->addr3)) + return -EINVAL; + + req->buf_index = READ_ONCE(sqe->buf_index); + gb->len = READ_ONCE(sqe->len); + gb->off = READ_ONCE(sqe->off); + nr_pages = (gb->len >> PAGE_SHIFT) + 2; + gb->max_pages = nr_pages; + + gb->imu = imu = io_alloc_reg_buf(req->ctx, nr_pages); + if (!imu) + return -ENOMEM; + imu->desc.nr_bvecs = 0; + req->flags |= REQ_F_NEED_CLEANUP; + return 0; +} + +int io_get_buf(struct io_kiocb *req, unsigned int issue_flags) +{ + struct io_get_buf *gb = io_kiocb_to_cmd(req, struct io_get_buf); + struct io_mapped_ubuf *imu = gb->imu; + struct iou_get_buf_info bi; + int ret, err; + + bi.off = gb->off; + bi.len = gb->len; + bi.flags = (issue_flags & IO_URING_F_NONBLOCK) ? IOU_GET_BUF_F_NOWAIT : 0; + bi.desc = &imu->desc; + + if (!gb->file->f_op->iou_get_buf) + return -ENOTSUPP; + ret = gb->file->f_op->iou_get_buf(gb->file, &bi); + if (ret < 0) { + if (ret == -EAGAIN && (issue_flags & IO_URING_F_NONBLOCK)) + return -EAGAIN; + goto done; + } + + imu->ubuf = 0; + imu->ubuf_end = ret; + imu->dir_mask = 1U << ITER_SOURCE; + imu->acct_pages = 0; + + io_ring_submit_lock(req->ctx, issue_flags); + err = io_install_buffer(req->ctx, imu, req->buf_index); + io_ring_submit_unlock(req->ctx, issue_flags); + if (unlikely(err)) { + ret = err; + goto done; + } + + gb->imu = NULL; + req->flags &= ~REQ_F_NEED_CLEANUP; +done: + if (ret != gb->len) + req_set_fail(req); + io_req_set_res(req, ret, 0); + return IOU_OK; +} diff --git a/io_uring/splice.h b/io_uring/splice.h index 542f94168ad3..2b923fc2bbf1 100644 --- a/io_uring/splice.h +++ b/io_uring/splice.h @@ -5,3 +5,7 @@ int io_tee(struct io_kiocb *req, unsigned int issue_flags); int io_splice_prep(struct io_kiocb *req, const struct io_uring_sqe *sqe); int io_splice(struct io_kiocb *req, unsigned int issue_flags); + +int io_get_buf_prep(struct io_kiocb *req, const struct io_uring_sqe *sqe); +int io_get_buf(struct io_kiocb *req, unsigned int issue_flags); +void io_get_buf_cleanup(struct io_kiocb *req);
There are several problems with splice requests, aka IORING_OP_SPLICE: 1) They are always executed by a worker thread, which is a slow path, as we don't have any reliable way to execute it NOWAIT. 2) It can't easily poll for data, as there are 2 files it operates on. It would either need to track what file to poll or poll both of them, in both cases it'll be a mess and add lot of overhead. 3) It has to have pipes in the middle, which adds overhead and is not great from the uapi design perspective when it goes for io_uring requests. 4) We want to operate with spliced data as with a normal buffer, i.e. write / send / etc. data as normally while it's zerocopy. It can partially be solved, but the root cause is a suboptimal for io_uring design of IORING_OP_SPLICE. Introduce a new request type called IORING_OP_GET_BUF, inspired by splice(2) as well as other proposals like fused requests. The main idea is to use io_uring's registered buffers as the middle man instead of pipes. Once a buffer is fetched / spliced from a file using a new fops callback ->iou_get_buf, it's installed as a registered buffers and can be used by all operations supporting the feature. Once the userspace releases the buffer, io_uring will wait for all requests using the buffer to complete and then use a file provided callback ->release() to return the buffer back. It operates on the level of the entire buffer instead of individual pages like it's with splice(2). As it was noted by the fused cmd work from where it came, this approach should be more flexible and efficient, and also leaves the space for more optimisations like custom caching or avoiding page refcounting altogether. Signed-off-by: Pavel Begunkov <asml.silence@gmail.com> --- include/linux/fs.h | 2 + include/linux/io_uring.h | 11 +++++ include/uapi/linux/io_uring.h | 1 + io_uring/opdef.c | 11 +++++ io_uring/rsrc.c | 2 +- io_uring/rsrc.h | 2 + io_uring/splice.c | 90 +++++++++++++++++++++++++++++++++++ io_uring/splice.h | 4 ++ 8 files changed, 122 insertions(+), 1 deletion(-)