From patchwork Fri Feb 18 19:57:34 2022 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Stefan Roesch X-Patchwork-Id: 12751875 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from vger.kernel.org (vger.kernel.org [23.128.96.18]) by smtp.lore.kernel.org (Postfix) with ESMTP id 424E6C4321E for ; Fri, 18 Feb 2022 19:58:10 +0000 (UTC) Received: (majordomo@vger.kernel.org) by vger.kernel.org via listexpand id S238670AbiBRT6Z (ORCPT ); Fri, 18 Feb 2022 14:58:25 -0500 Received: from mxb-00190b01.gslb.pphosted.com ([23.128.96.19]:44664 "EHLO lindbergh.monkeyblade.net" rhost-flags-OK-OK-OK-OK) by vger.kernel.org with ESMTP id S238593AbiBRT6W (ORCPT ); Fri, 18 Feb 2022 14:58:22 -0500 Received: from mx0a-00082601.pphosted.com (mx0b-00082601.pphosted.com [67.231.153.30]) by lindbergh.monkeyblade.net (Postfix) with ESMTPS id 58C3B1FA70 for ; Fri, 18 Feb 2022 11:58:04 -0800 (PST) Received: from pps.filterd (m0001303.ppops.net [127.0.0.1]) by m0001303.ppops.net (8.16.1.2/8.16.1.2) with ESMTP id 21IHUXXY014023 for ; Fri, 18 Feb 2022 11:58:03 -0800 DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=fb.com; h=from : to : cc : subject : date : message-id : in-reply-to : references : mime-version : content-transfer-encoding : content-type; s=facebook; bh=alXhLW7aurhHczIm3IzCj5kJidR1rbjDw8mIZ0YRvcw=; b=DHGSK/R4EqPc0NUjFBF6FAlhOwu1u6CCBsgoYKFDVyxj7y1ffyuQjXXki8V2OvUu+UpO Paw4O9E47g2vnIaOn0JhgQajgdWgDYsjOeJR0MPvFrGZJlirSJGnAc05A+dUpd+w1MO2 NwxyI7Cjqnr+lCbbdmzn03vfQ96O0nrVwW8= Received: from maileast.thefacebook.com ([163.114.130.16]) by m0001303.ppops.net (PPS) with ESMTPS id 3eafwrh1a3-2 (version=TLSv1.2 cipher=ECDHE-RSA-AES128-GCM-SHA256 bits=128 verify=NOT) for ; Fri, 18 Feb 2022 11:58:03 -0800 Received: from twshared6457.05.ash9.facebook.com (2620:10d:c0a8:1b::d) by mail.thefacebook.com (2620:10d:c0a8:82::c) with Microsoft SMTP Server (version=TLS1_2, cipher=TLS_ECDHE_RSA_WITH_AES_128_GCM_SHA256) id 15.1.2308.21; Fri, 18 Feb 2022 11:58:02 -0800 Received: by devvm225.atn0.facebook.com (Postfix, from userid 425415) id B5AC3AEB660D; Fri, 18 Feb 2022 11:57:50 -0800 (PST) From: Stefan Roesch To: , , , CC: Subject: [PATCH v2 08/13] io_uring: add support for async buffered writes Date: Fri, 18 Feb 2022 11:57:34 -0800 Message-ID: <20220218195739.585044-9-shr@fb.com> X-Mailer: git-send-email 2.30.2 In-Reply-To: <20220218195739.585044-1-shr@fb.com> References: <20220218195739.585044-1-shr@fb.com> MIME-Version: 1.0 X-FB-Internal: Safe X-Proofpoint-ORIG-GUID: Gc3PEEJ1LScDBv0T8ODLcG9wp7jvHSD_ X-Proofpoint-GUID: Gc3PEEJ1LScDBv0T8ODLcG9wp7jvHSD_ X-Proofpoint-Virus-Version: vendor=baseguard engine=ICAP:2.0.205,Aquarius:18.0.816,Hydra:6.0.425,FMLib:17.11.62.513 definitions=2022-02-18_09,2022-02-18_01,2021-12-02_01 X-Proofpoint-Spam-Details: rule=fb_outbound_notspam policy=fb_outbound score=0 phishscore=0 adultscore=0 lowpriorityscore=0 malwarescore=0 spamscore=0 impostorscore=0 clxscore=1015 suspectscore=0 priorityscore=1501 mlxscore=0 mlxlogscore=962 bulkscore=0 classifier=spam adjust=0 reason=mlx scancount=1 engine=8.12.0-2201110000 definitions=main-2202180122 X-FB-Internal: deliver Precedence: bulk List-ID: X-Mailing-List: linux-block@vger.kernel.org This enables the async buffered writes for block devices in io_uring. Buffered writes are enabled for blocks that are already in the page cache or can be acquired with noio. It is possible that a write request cannot be completely fullfilled (short write). In that case the request is punted and sent to the io workers to be completed. Before submitting the request to the io workers, the request is updated with how much has already been written. Signed-off-by: Stefan Roesch --- fs/io_uring.c | 29 ++++++++++++++++++++++++----- 1 file changed, 24 insertions(+), 5 deletions(-) diff --git a/fs/io_uring.c b/fs/io_uring.c index 77b9c7e4793b..52bd88908afd 100644 --- a/fs/io_uring.c +++ b/fs/io_uring.c @@ -3615,7 +3615,7 @@ static inline int io_iter_do_read(struct io_kiocb *req, struct iov_iter *iter) return -EINVAL; } -static bool need_read_all(struct io_kiocb *req) +static bool need_complete_io(struct io_kiocb *req) { return req->flags & REQ_F_ISREG || S_ISBLK(file_inode(req->file)->i_mode); @@ -3679,7 +3679,7 @@ static int io_read(struct io_kiocb *req, unsigned int issue_flags) } else if (ret == -EIOCBQUEUED) { goto out_free; } else if (ret == req->result || ret <= 0 || !force_nonblock || - (req->flags & REQ_F_NOWAIT) || !need_read_all(req)) { + (req->flags & REQ_F_NOWAIT) || !need_complete_io(req)) { /* read all, failed, already did sync or don't want to retry */ goto done; } @@ -3777,9 +3777,10 @@ static int io_write(struct io_kiocb *req, unsigned int issue_flags) if (unlikely(!io_file_supports_nowait(req))) goto copy_iov; - /* file path doesn't support NOWAIT for non-direct_IO */ - if (force_nonblock && !(kiocb->ki_flags & IOCB_DIRECT) && - (req->flags & REQ_F_ISREG)) + /* File path supports NOWAIT for non-direct_IO only for block devices. */ + if (!(kiocb->ki_flags & IOCB_DIRECT) && + !(kiocb->ki_filp->f_mode & FMODE_BUF_WASYNC) && + (req->flags & REQ_F_ISREG)) goto copy_iov; kiocb->ki_flags |= IOCB_NOWAIT; @@ -3831,6 +3832,24 @@ static int io_write(struct io_kiocb *req, unsigned int issue_flags) /* IOPOLL retry should happen for io-wq threads */ if (ret2 == -EAGAIN && (req->ctx->flags & IORING_SETUP_IOPOLL)) goto copy_iov; + + if (ret2 != req->result && ret2 >= 0 && need_complete_io(req)) { + struct io_async_rw *rw; + + /* This is a partial write. The file pos has already been + * updated, setup the async struct to complete the request + * in the worker. Also update bytes_done to account for + * the bytes already written. + */ + iov_iter_save_state(&s->iter, &s->iter_state); + ret = io_setup_async_rw(req, iovec, s, true); + + rw = req->async_data; + if (rw) + rw->bytes_done += ret2; + + return ret ? ret : -EAGAIN; + } done: kiocb_done(req, ret2, issue_flags); } else {