From patchwork Fri Mar 14 02:11:41 2025 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Ming Lei X-Patchwork-Id: 14016146 Received: from us-smtp-delivery-124.mimecast.com (us-smtp-delivery-124.mimecast.com [170.10.129.124]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (No client certificate requested) by smtp.subspace.kernel.org (Postfix) with ESMTPS id 48AA617588 for ; Fri, 14 Mar 2025 02:12:07 +0000 (UTC) Authentication-Results: smtp.subspace.kernel.org; arc=none smtp.client-ip=170.10.129.124 ARC-Seal: i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1741918329; cv=none; b=LoZgsqDHBDX1rpRhYtHH0AjO6I6tAwZIWGn4xFDg4+IEmv10tQzR72yGuaHgJal1bnIPaVA75iXo5YFHK28nMTOFOn4D+NC3NIerEJSjvEGFj0f11AEl4DBO9UwbopQglSTFlDqGIjrctDKwaVQz4px7ESsbSTchXfYZeu0o+1M= ARC-Message-Signature: i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1741918329; c=relaxed/simple; bh=XBrTJOi6RUf+zfX7H5Ssv7FsOXcGn3VYi6NQzhzSg8g=; h=From:To:Cc:Subject:Date:Message-ID:In-Reply-To:References: MIME-Version; b=Yey6gQQ5N1GRjjIX9w//Fu92Ye6ZuuSQlUelJ7QeJo6qkxH1OiTe+DMkdRpuVJ0ulA3P5OWuVD66NPGTsjD6T/tgPZiEAflPb6j4NMHCXfAfW81Jy94Lji9UeD8EV3plJaUlpjjNXNbk5yo7oSmdtLX0UTTGG1xNuhgt8QcG1yE= ARC-Authentication-Results: i=1; smtp.subspace.kernel.org; dmarc=pass (p=none dis=none) header.from=redhat.com; spf=pass smtp.mailfrom=redhat.com; dkim=pass (1024-bit key) header.d=redhat.com header.i=@redhat.com header.b=bFieyX45; arc=none smtp.client-ip=170.10.129.124 Authentication-Results: smtp.subspace.kernel.org; dmarc=pass (p=none dis=none) header.from=redhat.com Authentication-Results: smtp.subspace.kernel.org; spf=pass smtp.mailfrom=redhat.com Authentication-Results: smtp.subspace.kernel.org; dkim=pass (1024-bit key) header.d=redhat.com header.i=@redhat.com header.b="bFieyX45" DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=redhat.com; s=mimecast20190719; t=1741918327; h=from:from:reply-to:subject:subject:date:date:message-id:message-id: to:to:cc:cc:mime-version:mime-version: content-transfer-encoding:content-transfer-encoding: in-reply-to:in-reply-to:references:references; bh=ZJM7fhnzSVkAI0uB0b+VZz7idxBiXe+D5iz1Ay+75YE=; b=bFieyX45mPrqQC6Kz0U4VapoYlFFwCLQPhzZWEqvbcAVUaiHMWQp21c/GaaIHACrD0M5Hg 0IFgGm0rra9qyAq8LH8cbxDBmcy0NmwxsPaootQOL+DwI2PlepcPoW4HOReaqKBeQV+8Nh u6cj5wyk1xa2l26xd4mqTE23gwO9wh4= Received: from mx-prod-mc-08.mail-002.prod.us-west-2.aws.redhat.com (ec2-35-165-154-97.us-west-2.compute.amazonaws.com [35.165.154.97]) by relay.mimecast.com with ESMTP with STARTTLS (version=TLSv1.3, cipher=TLS_AES_256_GCM_SHA384) id us-mta-685-_S5p_J8vPNq2zUOrYratmA-1; Thu, 13 Mar 2025 22:12:05 -0400 X-MC-Unique: _S5p_J8vPNq2zUOrYratmA-1 X-Mimecast-MFC-AGG-ID: _S5p_J8vPNq2zUOrYratmA_1741918324 Received: from mx-prod-int-06.mail-002.prod.us-west-2.aws.redhat.com (mx-prod-int-06.mail-002.prod.us-west-2.aws.redhat.com [10.30.177.93]) (using TLSv1.3 with cipher TLS_AES_256_GCM_SHA384 (256/256 bits) key-exchange X25519 server-signature RSA-PSS (2048 bits) server-digest SHA256) (No client certificate requested) by mx-prod-mc-08.mail-002.prod.us-west-2.aws.redhat.com (Postfix) with ESMTPS id 5BA86180AB1C; Fri, 14 Mar 2025 02:12:03 +0000 (UTC) Received: from localhost (unknown [10.72.120.27]) by mx-prod-int-06.mail-002.prod.us-west-2.aws.redhat.com (Postfix) with ESMTP id B1ACF1828A87; Fri, 14 Mar 2025 02:12:00 +0000 (UTC) From: Ming Lei To: Jens Axboe , linux-block@vger.kernel.org Cc: Christoph Hellwig , Jooyung Han , Mike Snitzer , zkabelac@redhat.com, dm-devel@lists.linux.dev, Alasdair Kergon , Ming Lei Subject: [PATCH V2 1/5] loop: simplify do_req_filebacked() Date: Fri, 14 Mar 2025 10:11:41 +0800 Message-ID: <20250314021148.3081954-2-ming.lei@redhat.com> In-Reply-To: <20250314021148.3081954-1-ming.lei@redhat.com> References: <20250314021148.3081954-1-ming.lei@redhat.com> Precedence: bulk X-Mailing-List: dm-devel@lists.linux.dev List-Id: List-Subscribe: List-Unsubscribe: MIME-Version: 1.0 X-Scanned-By: MIMEDefang 3.4.1 on 10.30.177.93 lo_rw_aio() is only called for READ/WRITE operation, which can be figured out from request directly, so remove 'rw' parameter from lo_rw_aio(), meantime rename the local variable as 'dir' which makes the check more readable in lo_rw_aio(). Meantime add lo_rw_simple() so that do_req_filebacked() can be simplified in the following way: ``` if (cmd->use_aio) return lo_rw_aio(lo, cmd, pos); else return lo_rw_simple(lo, rq, pos); ``` Signed-off-by: Ming Lei Reviewed-by: Christoph Hellwig --- drivers/block/loop.c | 23 +++++++++++++---------- 1 file changed, 13 insertions(+), 10 deletions(-) diff --git a/drivers/block/loop.c b/drivers/block/loop.c index 657bf53decf3..554f6cefe8f6 100644 --- a/drivers/block/loop.c +++ b/drivers/block/loop.c @@ -285,6 +285,13 @@ static int lo_read_simple(struct loop_device *lo, struct request *rq, return 0; } +static int lo_rw_simple(struct loop_device *lo, struct request *rq, loff_t pos) +{ + if (req_op(rq) == REQ_OP_READ) + return lo_read_simple(lo, rq, pos); + return lo_write_simple(lo, rq, pos); +} + static void loop_clear_limits(struct loop_device *lo, int mode) { struct queue_limits lim = queue_limits_start_update(lo->lo_queue); @@ -400,13 +407,13 @@ static void lo_rw_aio_complete(struct kiocb *iocb, long ret) lo_rw_aio_do_completion(cmd); } -static int lo_rw_aio(struct loop_device *lo, struct loop_cmd *cmd, - loff_t pos, int rw) +static int lo_rw_aio(struct loop_device *lo, struct loop_cmd *cmd, loff_t pos) { struct iov_iter iter; struct req_iterator rq_iter; struct bio_vec *bvec; struct request *rq = blk_mq_rq_from_pdu(cmd); + int dir = (req_op(rq) == REQ_OP_READ) ? ITER_DEST : ITER_SOURCE; struct bio *bio = rq->bio; struct file *file = lo->lo_backing_file; struct bio_vec tmp; @@ -448,7 +455,7 @@ static int lo_rw_aio(struct loop_device *lo, struct loop_cmd *cmd, } atomic_set(&cmd->ref, 2); - iov_iter_bvec(&iter, rw, bvec, nr_bvec, blk_rq_bytes(rq)); + iov_iter_bvec(&iter, dir, bvec, nr_bvec, blk_rq_bytes(rq)); iter.iov_offset = offset; cmd->iocb.ki_pos = pos; @@ -457,7 +464,7 @@ static int lo_rw_aio(struct loop_device *lo, struct loop_cmd *cmd, cmd->iocb.ki_flags = IOCB_DIRECT; cmd->iocb.ki_ioprio = IOPRIO_PRIO_VALUE(IOPRIO_CLASS_NONE, 0); - if (rw == ITER_SOURCE) + if (dir == ITER_SOURCE) ret = file->f_op->write_iter(&cmd->iocb, &iter); else ret = file->f_op->read_iter(&cmd->iocb, &iter); @@ -498,15 +505,11 @@ static int do_req_filebacked(struct loop_device *lo, struct request *rq) case REQ_OP_DISCARD: return lo_fallocate(lo, rq, pos, FALLOC_FL_PUNCH_HOLE); case REQ_OP_WRITE: - if (cmd->use_aio) - return lo_rw_aio(lo, cmd, pos, ITER_SOURCE); - else - return lo_write_simple(lo, rq, pos); case REQ_OP_READ: if (cmd->use_aio) - return lo_rw_aio(lo, cmd, pos, ITER_DEST); + return lo_rw_aio(lo, cmd, pos); else - return lo_read_simple(lo, rq, pos); + return lo_rw_simple(lo, rq, pos); default: WARN_ON_ONCE(1); return -EIO; From patchwork Fri Mar 14 02:11:42 2025 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Ming Lei X-Patchwork-Id: 14016147 Received: from us-smtp-delivery-124.mimecast.com (us-smtp-delivery-124.mimecast.com [170.10.133.124]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (No client certificate requested) by smtp.subspace.kernel.org (Postfix) with ESMTPS id 850374400 for ; Fri, 14 Mar 2025 02:12:13 +0000 (UTC) Authentication-Results: smtp.subspace.kernel.org; arc=none smtp.client-ip=170.10.133.124 ARC-Seal: i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1741918335; cv=none; b=WB2essEnWP7UPEbkuhHFbtfF85HaGf9bA/51c/n/jfeYaxYMdNFVumZdTtEwlQYgxTSi4F+JhYmGXdjSmOBU/nBVyDs6HB6x3fATkh6TQ9ohI+KCkumagre5ZvQ588rchB3cXbE5CMKu2gEs9rWmoUHWuabnyeaqOjfhKjnEXnw= ARC-Message-Signature: i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1741918335; c=relaxed/simple; bh=DQTQ7+mQKC1JDnfE4frjrkcu3cb21sZ3cRWymGnDmY0=; h=From:To:Cc:Subject:Date:Message-ID:In-Reply-To:References: MIME-Version; b=ln1B0u4Tn1uJ4tlV6mSXnDxbFuHFqAhUIfItwlRotYnfNjs3lHQwxIXafe6cazGE6aFBz+GIBHnx7QbpcKtV9fEuSYhG8qeW7mMEsdjbEnA1W77/Zi8MwSnU+kBxNdvOjpuUjJjfcn/zzLVyggvD2pM3NQl1ISqOC/AIAe8qseM= ARC-Authentication-Results: i=1; smtp.subspace.kernel.org; dmarc=pass (p=none dis=none) header.from=redhat.com; spf=pass smtp.mailfrom=redhat.com; dkim=pass (1024-bit key) header.d=redhat.com header.i=@redhat.com header.b=PonVOxSa; arc=none smtp.client-ip=170.10.133.124 Authentication-Results: smtp.subspace.kernel.org; dmarc=pass (p=none dis=none) header.from=redhat.com Authentication-Results: smtp.subspace.kernel.org; spf=pass smtp.mailfrom=redhat.com Authentication-Results: smtp.subspace.kernel.org; dkim=pass (1024-bit key) header.d=redhat.com header.i=@redhat.com header.b="PonVOxSa" DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=redhat.com; s=mimecast20190719; t=1741918332; h=from:from:reply-to:subject:subject:date:date:message-id:message-id: to:to:cc:cc:mime-version:mime-version: content-transfer-encoding:content-transfer-encoding: in-reply-to:in-reply-to:references:references; bh=fgZoRfYizXi+vBT3gu5hmzqhybm+4j1jc5xO3FSiy8s=; b=PonVOxSaFWxIi8ONl54c9EO7yJcviHUbEWBcuh33UzOkC7TruwvaA7CUGf3yG0Znd9apoX 7BYi3yQ8nU8lxgKeqRv0vnJfB9ezkaZ9UjZP023BrF8E7dKt/+GEEggo8eL9WQ0bcOhKjd Kcag3nhPXvMQd1x5I5aRknHg0KiTrtE= Received: from mx-prod-mc-02.mail-002.prod.us-west-2.aws.redhat.com (ec2-54-186-198-63.us-west-2.compute.amazonaws.com [54.186.198.63]) by relay.mimecast.com with ESMTP with STARTTLS (version=TLSv1.3, cipher=TLS_AES_256_GCM_SHA384) id us-mta-62-JYE5q5slPLSu9QFQgTLWFA-1; Thu, 13 Mar 2025 22:12:09 -0400 X-MC-Unique: JYE5q5slPLSu9QFQgTLWFA-1 X-Mimecast-MFC-AGG-ID: JYE5q5slPLSu9QFQgTLWFA_1741918327 Received: from mx-prod-int-04.mail-002.prod.us-west-2.aws.redhat.com (mx-prod-int-04.mail-002.prod.us-west-2.aws.redhat.com [10.30.177.40]) (using TLSv1.3 with cipher TLS_AES_256_GCM_SHA384 (256/256 bits) key-exchange X25519 server-signature RSA-PSS (2048 bits) server-digest SHA256) (No client certificate requested) by mx-prod-mc-02.mail-002.prod.us-west-2.aws.redhat.com (Postfix) with ESMTPS id CBADE195609E; Fri, 14 Mar 2025 02:12:07 +0000 (UTC) Received: from localhost (unknown [10.72.120.27]) by mx-prod-int-04.mail-002.prod.us-west-2.aws.redhat.com (Postfix) with ESMTP id 7ABA619560AB; Fri, 14 Mar 2025 02:12:06 +0000 (UTC) From: Ming Lei To: Jens Axboe , linux-block@vger.kernel.org Cc: Christoph Hellwig , Jooyung Han , Mike Snitzer , zkabelac@redhat.com, dm-devel@lists.linux.dev, Alasdair Kergon , Ming Lei Subject: [PATCH V2 2/5] loop: cleanup lo_rw_aio() Date: Fri, 14 Mar 2025 10:11:42 +0800 Message-ID: <20250314021148.3081954-3-ming.lei@redhat.com> In-Reply-To: <20250314021148.3081954-1-ming.lei@redhat.com> References: <20250314021148.3081954-1-ming.lei@redhat.com> Precedence: bulk X-Mailing-List: dm-devel@lists.linux.dev List-Id: List-Subscribe: List-Unsubscribe: MIME-Version: 1.0 X-Scanned-By: MIMEDefang 3.0 on 10.30.177.40 Cleanup lo_rw_aio() a bit by refactoring it into three parts: - lo_cmd_nr_bvec(), for calculating how many bvecs in this request - lo_rw_aio_prep(), for preparing loop command, which need to be called once - lo_submit_rw_aio(), for submitting this lo command, which can be called multiple times Prepare for trying to handle loop command by NOWAIT read/write IO first. Signed-off-by: Ming Lei Reviewed-by: Christoph Hellwig --- drivers/block/loop.c | 81 ++++++++++++++++++++++++++++++-------------- 1 file changed, 56 insertions(+), 25 deletions(-) diff --git a/drivers/block/loop.c b/drivers/block/loop.c index 554f6cefe8f6..3f921d9d6de6 100644 --- a/drivers/block/loop.c +++ b/drivers/block/loop.c @@ -394,7 +394,6 @@ static void lo_rw_aio_do_completion(struct loop_cmd *cmd) if (!atomic_dec_and_test(&cmd->ref)) return; kfree(cmd->bvec); - cmd->bvec = NULL; if (likely(!blk_should_fake_timeout(rq->q))) blk_mq_complete_request(rq); } @@ -407,24 +406,29 @@ static void lo_rw_aio_complete(struct kiocb *iocb, long ret) lo_rw_aio_do_completion(cmd); } -static int lo_rw_aio(struct loop_device *lo, struct loop_cmd *cmd, loff_t pos) +static inline unsigned lo_cmd_nr_bvec(struct loop_cmd *cmd) { - struct iov_iter iter; struct req_iterator rq_iter; - struct bio_vec *bvec; struct request *rq = blk_mq_rq_from_pdu(cmd); - int dir = (req_op(rq) == REQ_OP_READ) ? ITER_DEST : ITER_SOURCE; - struct bio *bio = rq->bio; - struct file *file = lo->lo_backing_file; struct bio_vec tmp; - unsigned int offset; int nr_bvec = 0; - int ret; rq_for_each_bvec(tmp, rq, rq_iter) nr_bvec++; + return nr_bvec; +} + +static int lo_rw_aio_prep(struct loop_device *lo, struct loop_cmd *cmd, + unsigned nr_bvec) +{ + struct request *rq = blk_mq_rq_from_pdu(cmd); + loff_t pos = ((loff_t) blk_rq_pos(rq) << 9) + lo->lo_offset; + if (rq->bio != rq->biotail) { + struct req_iterator rq_iter; + struct bio_vec *bvec; + struct bio_vec tmp; bvec = kmalloc_array(nr_bvec, sizeof(struct bio_vec), GFP_NOIO); @@ -442,35 +446,62 @@ static int lo_rw_aio(struct loop_device *lo, struct loop_cmd *cmd, loff_t pos) *bvec = tmp; bvec++; } - bvec = cmd->bvec; - offset = 0; } else { + cmd->bvec = NULL; + } + cmd->iocb.ki_pos = pos; + cmd->iocb.ki_filp = lo->lo_backing_file; + cmd->iocb.ki_complete = lo_rw_aio_complete; + cmd->iocb.ki_flags = IOCB_DIRECT; + cmd->iocb.ki_ioprio = IOPRIO_PRIO_VALUE(IOPRIO_CLASS_NONE, 0); + + return 0; +} + +static int lo_submit_rw_aio(struct loop_device *lo, struct loop_cmd *cmd, + int nr_bvec) +{ + struct request *rq = blk_mq_rq_from_pdu(cmd); + int dir = (req_op(rq) == REQ_OP_READ) ? ITER_DEST : ITER_SOURCE; + struct file *file = lo->lo_backing_file; + struct iov_iter iter; + int ret; + + if (cmd->bvec) { + iov_iter_bvec(&iter, dir, cmd->bvec, nr_bvec, blk_rq_bytes(rq)); + iter.iov_offset = 0; + } else { + struct bio *bio = rq->bio; + struct bio_vec *bvec = __bvec_iter_bvec(bio->bi_io_vec, + bio->bi_iter); + /* * Same here, this bio may be started from the middle of the * 'bvec' because of bio splitting, so offset from the bvec * must be passed to iov iterator */ - offset = bio->bi_iter.bi_bvec_done; - bvec = __bvec_iter_bvec(bio->bi_io_vec, bio->bi_iter); + iov_iter_bvec(&iter, dir, bvec, nr_bvec, blk_rq_bytes(rq)); + iter.iov_offset = bio->bi_iter.bi_bvec_done; } - atomic_set(&cmd->ref, 2); - - iov_iter_bvec(&iter, dir, bvec, nr_bvec, blk_rq_bytes(rq)); - iter.iov_offset = offset; - - cmd->iocb.ki_pos = pos; - cmd->iocb.ki_filp = file; - cmd->iocb.ki_complete = lo_rw_aio_complete; - cmd->iocb.ki_flags = IOCB_DIRECT; - cmd->iocb.ki_ioprio = IOPRIO_PRIO_VALUE(IOPRIO_CLASS_NONE, 0); + atomic_set(&cmd->ref, 2); if (dir == ITER_SOURCE) ret = file->f_op->write_iter(&cmd->iocb, &iter); else ret = file->f_op->read_iter(&cmd->iocb, &iter); - lo_rw_aio_do_completion(cmd); + return ret; +} + + +static int lo_rw_aio(struct loop_device *lo, struct loop_cmd *cmd) +{ + unsigned int nr_bvec = lo_cmd_nr_bvec(cmd); + int ret = lo_rw_aio_prep(lo, cmd, nr_bvec); + + if (ret >= 0) + ret = lo_submit_rw_aio(lo, cmd, nr_bvec); if (ret != -EIOCBQUEUED) lo_rw_aio_complete(&cmd->iocb, ret); return 0; @@ -507,7 +538,7 @@ static int do_req_filebacked(struct loop_device *lo, struct request *rq) case REQ_OP_WRITE: case REQ_OP_READ: if (cmd->use_aio) - return lo_rw_aio(lo, cmd, pos); + return lo_rw_aio(lo, cmd); else return lo_rw_simple(lo, rq, pos); default: From patchwork Fri Mar 14 02:11:43 2025 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Ming Lei X-Patchwork-Id: 14016148 Received: from us-smtp-delivery-124.mimecast.com (us-smtp-delivery-124.mimecast.com [170.10.129.124]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (No client certificate requested) by smtp.subspace.kernel.org (Postfix) with ESMTPS id 4A16F282F0 for ; Fri, 14 Mar 2025 02:12:18 +0000 (UTC) Authentication-Results: smtp.subspace.kernel.org; arc=none smtp.client-ip=170.10.129.124 ARC-Seal: i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1741918339; cv=none; b=LbyFMLbhKS7zxNzlZSb70MI8DFRTA1g4dyQArZC3WaPde4FcbzQhxZF12fuu744ii3Vsnh4fIPQ08ITQpMTHtfZCbfVoVV3GVQEh8Q/9RhtB5SyFAjLU6ZK0BBeM5XZzbOq55LrLHq7qJTzx8zU+wgwTSYov2giP/OPyqS3cmdE= ARC-Message-Signature: i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1741918339; c=relaxed/simple; bh=8WMy0x/Qcscwy3LSRbN3fJBynt3EOpafTNWPsiBQF80=; h=From:To:Cc:Subject:Date:Message-ID:In-Reply-To:References: MIME-Version; b=dEEhGPBi/+JvyT+JMQ/1PR+EAGoqpzU1cMmDqwfzZ2NNlzpGLqZiluB/goqTXT8Xeup83tIGg4rnJRjHebuGmmzX2P4r1di2fV4CtlPMy0VY99sWVYZz8zuQ1bHe+GOBL9s4k+XkLZOGdh6qYPG4OsvnDkl+zWFWaCtk+wdw8ro= ARC-Authentication-Results: i=1; smtp.subspace.kernel.org; dmarc=pass (p=none dis=none) header.from=redhat.com; spf=pass smtp.mailfrom=redhat.com; dkim=pass (1024-bit key) header.d=redhat.com header.i=@redhat.com header.b=cUJPo/mt; arc=none smtp.client-ip=170.10.129.124 Authentication-Results: smtp.subspace.kernel.org; dmarc=pass (p=none dis=none) header.from=redhat.com Authentication-Results: smtp.subspace.kernel.org; spf=pass smtp.mailfrom=redhat.com Authentication-Results: smtp.subspace.kernel.org; dkim=pass (1024-bit key) header.d=redhat.com header.i=@redhat.com header.b="cUJPo/mt" DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=redhat.com; s=mimecast20190719; t=1741918337; h=from:from:reply-to:subject:subject:date:date:message-id:message-id: to:to:cc:cc:mime-version:mime-version: content-transfer-encoding:content-transfer-encoding: in-reply-to:in-reply-to:references:references; bh=R38aCB8X9sUjKnKzkBvPR6oMsrsyCkjm3yYm+OCxTo0=; b=cUJPo/mtfbXKCBg0QFEbNYwXgsGAS+5dUMr7AEAgbjtXtdWmd5/pfNd9ChEci/tpf0tV5M J+fu42Pspxf4tq14ZVQo8DWTpjftWbYk6rJdCpKij8j03cWLhghtvMesAe6ZJWtSFbV2XQ 7M6KWBNkRyP/MiDnu/EqFbkS+Bky+yU= Received: from mx-prod-mc-06.mail-002.prod.us-west-2.aws.redhat.com (ec2-35-165-154-97.us-west-2.compute.amazonaws.com [35.165.154.97]) by relay.mimecast.com with ESMTP with STARTTLS (version=TLSv1.3, cipher=TLS_AES_256_GCM_SHA384) id us-mta-621-OdVHmkqlMnWHInv3HRuZPg-1; Thu, 13 Mar 2025 22:12:13 -0400 X-MC-Unique: OdVHmkqlMnWHInv3HRuZPg-1 X-Mimecast-MFC-AGG-ID: OdVHmkqlMnWHInv3HRuZPg_1741918332 Received: from mx-prod-int-02.mail-002.prod.us-west-2.aws.redhat.com (mx-prod-int-02.mail-002.prod.us-west-2.aws.redhat.com [10.30.177.15]) (using TLSv1.3 with cipher TLS_AES_256_GCM_SHA384 (256/256 bits) key-exchange X25519 server-signature RSA-PSS (2048 bits) server-digest SHA256) (No client certificate requested) by mx-prod-mc-06.mail-002.prod.us-west-2.aws.redhat.com (Postfix) with ESMTPS id B52281800257; Fri, 14 Mar 2025 02:12:12 +0000 (UTC) Received: from localhost (unknown [10.72.120.27]) by mx-prod-int-02.mail-002.prod.us-west-2.aws.redhat.com (Postfix) with ESMTP id 4CA7719373D7; Fri, 14 Mar 2025 02:12:10 +0000 (UTC) From: Ming Lei To: Jens Axboe , linux-block@vger.kernel.org Cc: Christoph Hellwig , Jooyung Han , Mike Snitzer , zkabelac@redhat.com, dm-devel@lists.linux.dev, Alasdair Kergon , Ming Lei Subject: [PATCH V2 3/5] loop: move command blkcg/memcg initialization into loop_queue_work Date: Fri, 14 Mar 2025 10:11:43 +0800 Message-ID: <20250314021148.3081954-4-ming.lei@redhat.com> In-Reply-To: <20250314021148.3081954-1-ming.lei@redhat.com> References: <20250314021148.3081954-1-ming.lei@redhat.com> Precedence: bulk X-Mailing-List: dm-devel@lists.linux.dev List-Id: List-Subscribe: List-Unsubscribe: MIME-Version: 1.0 X-Scanned-By: MIMEDefang 3.0 on 10.30.177.15 Move loop command blkcg/memcg initialization into loop_queue_work, and prepare for supporting to handle loop io command by IOCB_NOWAIT. Signed-off-by: Ming Lei Reviewed-by: Christoph Hellwig --- drivers/block/loop.c | 32 +++++++++++++++++--------------- 1 file changed, 17 insertions(+), 15 deletions(-) diff --git a/drivers/block/loop.c b/drivers/block/loop.c index 3f921d9d6de6..81f6ba9bc911 100644 --- a/drivers/block/loop.c +++ b/drivers/block/loop.c @@ -878,11 +878,28 @@ static inline int queue_on_root_worker(struct cgroup_subsys_state *css) static void loop_queue_work(struct loop_device *lo, struct loop_cmd *cmd) { + struct request __maybe_unused *rq = blk_mq_rq_from_pdu(cmd); struct rb_node **node, *parent = NULL; struct loop_worker *cur_worker, *worker = NULL; struct work_struct *work; struct list_head *cmd_list; + /* always use the first bio's css */ + cmd->blkcg_css = NULL; + cmd->memcg_css = NULL; +#ifdef CONFIG_BLK_CGROUP + if (rq->bio) { + cmd->blkcg_css = bio_blkcg_css(rq->bio); +#ifdef CONFIG_MEMCG + if (cmd->blkcg_css) { + cmd->memcg_css = + cgroup_get_e_css(cmd->blkcg_css->cgroup, + &memory_cgrp_subsys); + } +#endif + } +#endif + spin_lock_irq(&lo->lo_work_lock); if (queue_on_root_worker(cmd->blkcg_css)) @@ -1920,21 +1937,6 @@ static blk_status_t loop_queue_rq(struct blk_mq_hw_ctx *hctx, break; } - /* always use the first bio's css */ - cmd->blkcg_css = NULL; - cmd->memcg_css = NULL; -#ifdef CONFIG_BLK_CGROUP - if (rq->bio) { - cmd->blkcg_css = bio_blkcg_css(rq->bio); -#ifdef CONFIG_MEMCG - if (cmd->blkcg_css) { - cmd->memcg_css = - cgroup_get_e_css(cmd->blkcg_css->cgroup, - &memory_cgrp_subsys); - } -#endif - } -#endif loop_queue_work(lo, cmd); return BLK_STS_OK; From patchwork Fri Mar 14 02:11:44 2025 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Ming Lei X-Patchwork-Id: 14016149 Received: from us-smtp-delivery-124.mimecast.com (us-smtp-delivery-124.mimecast.com [170.10.133.124]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (No client certificate requested) by smtp.subspace.kernel.org (Postfix) with ESMTPS id 73C2717588 for ; Fri, 14 Mar 2025 02:12:24 +0000 (UTC) Authentication-Results: smtp.subspace.kernel.org; arc=none smtp.client-ip=170.10.133.124 ARC-Seal: i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1741918346; cv=none; b=EeJYDFMraTODjFUZkxTyK9Du/jazGCRrA91zKZ2gnoK1CABU9e4dJpkNmGLNt2IGaYaGQh98xL6UcVwDHvUsTEjNST8f1BjGPG9iSvOvXfrrc7CG1BA3Kw+izEK7T9nC3RSBQ48DQJukK9VyyPRZfdKF1eW9uODHPGDu9SEhn+8= ARC-Message-Signature: i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1741918346; c=relaxed/simple; bh=gHm1+9Fr63yZCGuLdER1Pl9STie7jx/U5SsqUMMDIVw=; h=From:To:Cc:Subject:Date:Message-ID:In-Reply-To:References: MIME-Version; b=ca0PbfcIXvHwIPD9GEW38ZMSy0VuYB2XBPdzP0yC+eGVZ2PAiUvtpdqDCSlSaUmpCapKBCOAK5d3qbYFZZRIHwgfL6fVACRwYUzUzqAEb6z48qm0yP+tP14du+QEYuBEkuCDzEdesKRr8qFt78NAeYNaJNrhXoIaH4YNH8XG908= ARC-Authentication-Results: i=1; smtp.subspace.kernel.org; dmarc=pass (p=none dis=none) header.from=redhat.com; spf=pass smtp.mailfrom=redhat.com; dkim=pass (1024-bit key) header.d=redhat.com header.i=@redhat.com header.b=UMvdJHOF; arc=none smtp.client-ip=170.10.133.124 Authentication-Results: smtp.subspace.kernel.org; dmarc=pass (p=none dis=none) header.from=redhat.com Authentication-Results: smtp.subspace.kernel.org; spf=pass smtp.mailfrom=redhat.com Authentication-Results: smtp.subspace.kernel.org; dkim=pass (1024-bit key) header.d=redhat.com header.i=@redhat.com header.b="UMvdJHOF" DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=redhat.com; s=mimecast20190719; t=1741918343; h=from:from:reply-to:subject:subject:date:date:message-id:message-id: to:to:cc:cc:mime-version:mime-version: content-transfer-encoding:content-transfer-encoding: in-reply-to:in-reply-to:references:references; bh=AjF5jQkybIwVeLwRpnIgYis0nR2Ur+KxbDfyjyny6iM=; b=UMvdJHOFJr1Si5fWbETcLYQH5yNYlyPKBmC5qaIoK1FSBAqP1JZfhwuU/FQJ+7Uhlmwveh LaHOH37NftBoAVJwe7VIyqRtCx+ZBlHHBhycQu4mKqtQJAXfuXxN0cBFqG7ehEH7MA8mJz QVfnIo94kaQGf5x9EUt7zAi8QCHfWug= Received: from mx-prod-mc-01.mail-002.prod.us-west-2.aws.redhat.com (ec2-54-186-198-63.us-west-2.compute.amazonaws.com [54.186.198.63]) by relay.mimecast.com with ESMTP with STARTTLS (version=TLSv1.3, cipher=TLS_AES_256_GCM_SHA384) id us-mta-556-B7LZZ9wwM3mlBvjxq2r0kA-1; Thu, 13 Mar 2025 22:12:18 -0400 X-MC-Unique: B7LZZ9wwM3mlBvjxq2r0kA-1 X-Mimecast-MFC-AGG-ID: B7LZZ9wwM3mlBvjxq2r0kA_1741918336 Received: from mx-prod-int-02.mail-002.prod.us-west-2.aws.redhat.com (mx-prod-int-02.mail-002.prod.us-west-2.aws.redhat.com [10.30.177.15]) (using TLSv1.3 with cipher TLS_AES_256_GCM_SHA384 (256/256 bits) key-exchange X25519 server-signature RSA-PSS (2048 bits) server-digest SHA256) (No client certificate requested) by mx-prod-mc-01.mail-002.prod.us-west-2.aws.redhat.com (Postfix) with ESMTPS id A8F4B1956055; Fri, 14 Mar 2025 02:12:16 +0000 (UTC) Received: from localhost (unknown [10.72.120.27]) by mx-prod-int-02.mail-002.prod.us-west-2.aws.redhat.com (Postfix) with ESMTP id BAC1319373D7; Fri, 14 Mar 2025 02:12:15 +0000 (UTC) From: Ming Lei To: Jens Axboe , linux-block@vger.kernel.org Cc: Christoph Hellwig , Jooyung Han , Mike Snitzer , zkabelac@redhat.com, dm-devel@lists.linux.dev, Alasdair Kergon , Ming Lei Subject: [PATCH V2 4/5] loop: try to handle loop aio command via NOWAIT IO first Date: Fri, 14 Mar 2025 10:11:44 +0800 Message-ID: <20250314021148.3081954-5-ming.lei@redhat.com> In-Reply-To: <20250314021148.3081954-1-ming.lei@redhat.com> References: <20250314021148.3081954-1-ming.lei@redhat.com> Precedence: bulk X-Mailing-List: dm-devel@lists.linux.dev List-Id: List-Subscribe: List-Unsubscribe: MIME-Version: 1.0 X-Scanned-By: MIMEDefang 3.0 on 10.30.177.15 Try to handle loop aio command via NOWAIT IO first, then we can avoid to queue the aio command into workqueue. This is usually one big win in case that FS block mapping is stable, Mikulas verified [1] that this way improves IO perf by close to 5X in 12jobs sequential read/write test. Fallback to workqueue in case of -EAGAIN. This way may bring a little cost from the 1st retry, but when running the following write test over loop/sparse_file, the actual effect on randwrite is obvious: ``` truncate -s 4G 1.img #1.img is created on XFS/virtio-scsi losetup -f 1.img --direct-io=on fio --direct=1 --bs=4k --runtime=40 --time_based --numjobs=1 --ioengine=libaio \ --iodepth=16 --group_reporting=1 --filename=/dev/loop0 -name=job --rw=$RW ``` - RW=randwrite: obvious IOPS drop observed - RW=write: a little drop(%5 - 10%) This perf drop on randwrite over sparse file will be addressed in the following patch. BLK_MQ_F_BLOCKING has to be set for calling into .read_iter() or .write_iter() which might sleep even though it is NOWAIT, and the only effect is that rcu read lock is replaced with srcu read lock. Link: https://lore.kernel.org/linux-block/a8e5c76a-231f-07d1-a394-847de930f638@redhat.com/ [1] Signed-off-by: Ming Lei --- drivers/block/loop.c | 51 ++++++++++++++++++++++++++++++++++++++++---- 1 file changed, 47 insertions(+), 4 deletions(-) diff --git a/drivers/block/loop.c b/drivers/block/loop.c index 81f6ba9bc911..542b1fe938a7 100644 --- a/drivers/block/loop.c +++ b/drivers/block/loop.c @@ -92,6 +92,8 @@ struct loop_cmd { #define LOOP_IDLE_WORKER_TIMEOUT (60 * HZ) #define LOOP_DEFAULT_HW_Q_DEPTH 128 +static void loop_queue_work(struct loop_device *lo, struct loop_cmd *cmd); + static DEFINE_IDR(loop_index_idr); static DEFINE_MUTEX(loop_ctl_mutex); static DEFINE_MUTEX(loop_validate_mutex); @@ -393,6 +395,15 @@ static void lo_rw_aio_do_completion(struct loop_cmd *cmd) if (!atomic_dec_and_test(&cmd->ref)) return; + + /* -EAGAIN could be returned from bdev's ->ki_complete */ + if (cmd->ret == -EAGAIN) { + struct loop_device *lo = rq->q->queuedata; + + loop_queue_work(lo, cmd); + return; + } + kfree(cmd->bvec); if (likely(!blk_should_fake_timeout(rq->q))) blk_mq_complete_request(rq); @@ -498,15 +509,38 @@ static int lo_submit_rw_aio(struct loop_device *lo, struct loop_cmd *cmd, static int lo_rw_aio(struct loop_device *lo, struct loop_cmd *cmd) { unsigned int nr_bvec = lo_cmd_nr_bvec(cmd); - int ret = lo_rw_aio_prep(lo, cmd, nr_bvec); + int ret; - if (ret >= 0) - ret = lo_submit_rw_aio(lo, cmd, nr_bvec); + /* + * This command is prepared, and we have tried IOCB_NOWAIT, but got + * -EAGAIN, so clear it now + */ + cmd->iocb.ki_flags &= ~IOCB_NOWAIT; + ret = lo_submit_rw_aio(lo, cmd, nr_bvec); if (ret != -EIOCBQUEUED) lo_rw_aio_complete(&cmd->iocb, ret); return 0; } +static blk_status_t lo_rw_aio_nowait(struct loop_device *lo, + struct loop_cmd *cmd) +{ + unsigned int nr_bvec = lo_cmd_nr_bvec(cmd); + int ret = lo_rw_aio_prep(lo, cmd, nr_bvec); + + if (unlikely(ret < 0)) + return BLK_STS_IOERR; + + cmd->iocb.ki_flags |= IOCB_NOWAIT; + ret = lo_submit_rw_aio(lo, cmd, nr_bvec); + if (ret == -EAGAIN) + return BLK_STS_AGAIN; + + if (ret != -EIOCBQUEUED) + lo_rw_aio_complete(&cmd->iocb, ret); + return BLK_STS_OK; +} + static int do_req_filebacked(struct loop_device *lo, struct request *rq) { struct loop_cmd *cmd = blk_mq_rq_to_pdu(rq); @@ -1937,6 +1971,14 @@ static blk_status_t loop_queue_rq(struct blk_mq_hw_ctx *hctx, break; } + if (cmd->use_aio) { + blk_status_t res = lo_rw_aio_nowait(lo, cmd); + + if (res != BLK_STS_AGAIN) + return res; + /* fallback to workqueue for handling aio */ + } + loop_queue_work(lo, cmd); return BLK_STS_OK; @@ -2087,7 +2129,8 @@ static int loop_add(int i) lo->tag_set.queue_depth = hw_queue_depth; lo->tag_set.numa_node = NUMA_NO_NODE; lo->tag_set.cmd_size = sizeof(struct loop_cmd); - lo->tag_set.flags = BLK_MQ_F_STACKING | BLK_MQ_F_NO_SCHED_BY_DEFAULT; + lo->tag_set.flags = BLK_MQ_F_STACKING | BLK_MQ_F_NO_SCHED_BY_DEFAULT | + BLK_MQ_F_BLOCKING; lo->tag_set.driver_data = lo; err = blk_mq_alloc_tag_set(&lo->tag_set); From patchwork Fri Mar 14 02:11:45 2025 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Ming Lei X-Patchwork-Id: 14016150 Received: from us-smtp-delivery-124.mimecast.com (us-smtp-delivery-124.mimecast.com [170.10.129.124]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (No client certificate requested) by smtp.subspace.kernel.org (Postfix) with ESMTPS id E84982AF19 for ; Fri, 14 Mar 2025 02:12:24 +0000 (UTC) Authentication-Results: smtp.subspace.kernel.org; arc=none smtp.client-ip=170.10.129.124 ARC-Seal: i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1741918346; cv=none; b=mToWxM3aHZp9083RhCH/QhjTBPyo45y7rK2dpLBx8VuVLQ+5Lf7HxU9/f5p8+Wp3VYZnOHi3SPnsK4qEaQVqhQ1c4CpnLubLEjODvrW+CuG/WOje52nYWpjVtdQiXEaztPs4kbyDAplPBMd4dkIVzHfTJ1H2Cw/yX9wpUb95I/Y= ARC-Message-Signature: i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1741918346; c=relaxed/simple; bh=menII101TGojYykHQ6b+MfZFAVmDh/is6WcbZSElUuI=; h=From:To:Cc:Subject:Date:Message-ID:In-Reply-To:References: MIME-Version; b=nj1ZThvkI6gghlC/xrJSe9ZJ8pdMFz/UWUis0a6298ts9BJ1K0BnyyDts60/2zuhQGhIGEDjoAfdFhmveDOp9UfP7ndsWXcoCqEIE8mjzD3f6YBWXJzBHwqKpgpyVS0n5TupeMT63r01G1au5GKKqI8VpIObOy4WAL2l9gZHjrY= ARC-Authentication-Results: i=1; smtp.subspace.kernel.org; dmarc=pass (p=none dis=none) header.from=redhat.com; spf=pass smtp.mailfrom=redhat.com; dkim=pass (1024-bit key) header.d=redhat.com header.i=@redhat.com header.b=fJ9CSXCp; arc=none smtp.client-ip=170.10.129.124 Authentication-Results: smtp.subspace.kernel.org; dmarc=pass (p=none dis=none) header.from=redhat.com Authentication-Results: smtp.subspace.kernel.org; spf=pass smtp.mailfrom=redhat.com Authentication-Results: smtp.subspace.kernel.org; dkim=pass (1024-bit key) header.d=redhat.com header.i=@redhat.com header.b="fJ9CSXCp" DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=redhat.com; s=mimecast20190719; t=1741918344; h=from:from:reply-to:subject:subject:date:date:message-id:message-id: to:to:cc:cc:mime-version:mime-version: content-transfer-encoding:content-transfer-encoding: in-reply-to:in-reply-to:references:references; bh=wWBFjWQhhDx9tAlh9GEaUJkOGUMPicNbteHnVS/NWDs=; b=fJ9CSXCpTMz7W3nLmTAsdcy+8JPUGePMKBNTM4mzjucnl3kxn2GuV4ASUzjQJSJ+xpCeNU kLYCImF9ngso6tVGBuQ/ZLmHTFW+4O6DxV8keguBAk4wgMARiHjW68o3PRyQ6m8JYqxLum p06s/lSbQEBSIfp70fYpiIFuqc+xweo= Received: from mx-prod-mc-06.mail-002.prod.us-west-2.aws.redhat.com (ec2-35-165-154-97.us-west-2.compute.amazonaws.com [35.165.154.97]) by relay.mimecast.com with ESMTP with STARTTLS (version=TLSv1.3, cipher=TLS_AES_256_GCM_SHA384) id us-mta-27-So7hWVESM3q-NwN3DmOVSA-1; Thu, 13 Mar 2025 22:12:22 -0400 X-MC-Unique: So7hWVESM3q-NwN3DmOVSA-1 X-Mimecast-MFC-AGG-ID: So7hWVESM3q-NwN3DmOVSA_1741918341 Received: from mx-prod-int-01.mail-002.prod.us-west-2.aws.redhat.com (mx-prod-int-01.mail-002.prod.us-west-2.aws.redhat.com [10.30.177.4]) (using TLSv1.3 with cipher TLS_AES_256_GCM_SHA384 (256/256 bits) key-exchange X25519 server-signature RSA-PSS (2048 bits) server-digest SHA256) (No client certificate requested) by mx-prod-mc-06.mail-002.prod.us-west-2.aws.redhat.com (Postfix) with ESMTPS id 3A10D180034D; Fri, 14 Mar 2025 02:12:21 +0000 (UTC) Received: from localhost (unknown [10.72.120.27]) by mx-prod-int-01.mail-002.prod.us-west-2.aws.redhat.com (Postfix) with ESMTP id C851B3003770; Fri, 14 Mar 2025 02:12:19 +0000 (UTC) From: Ming Lei To: Jens Axboe , linux-block@vger.kernel.org Cc: Christoph Hellwig , Jooyung Han , Mike Snitzer , zkabelac@redhat.com, dm-devel@lists.linux.dev, Alasdair Kergon , Ming Lei Subject: [PATCH V2 5/5] loop: add hint for handling aio via IOCB_NOWAIT Date: Fri, 14 Mar 2025 10:11:45 +0800 Message-ID: <20250314021148.3081954-6-ming.lei@redhat.com> In-Reply-To: <20250314021148.3081954-1-ming.lei@redhat.com> References: <20250314021148.3081954-1-ming.lei@redhat.com> Precedence: bulk X-Mailing-List: dm-devel@lists.linux.dev List-Id: List-Subscribe: List-Unsubscribe: MIME-Version: 1.0 X-Scanned-By: MIMEDefang 3.4.1 on 10.30.177.4 Add hint for using IOCB_NOWAIT to handle loop aio command for avoiding to cause write(especially randwrite) perf regression on sparse file. Try IOCB_NOWAIT in the following situations: - backing file is block device - READ aio command - there isn't queued aio non-NOWAIT WRITE, since retry of NOWAIT won't cause contention on WRITE and non-NOWAIT WRITE often implies exclusive lock. With this simple policy, perf regression of randwrte/write on sparse backing file is fixed. Meantime this way addresses perf problem[1] in case of stable FS block mapping via NOWAIT. Link: https://lore.kernel.org/dm-devel/7d6ae2c9-df8e-50d0-7ad6-b787cb3cfab4@redhat.com/ Signed-off-by: Ming Lei --- drivers/block/loop.c | 61 ++++++++++++++++++++++++++++++++++++++++++++ 1 file changed, 61 insertions(+) diff --git a/drivers/block/loop.c b/drivers/block/loop.c index 542b1fe938a7..5bf14549cf8e 100644 --- a/drivers/block/loop.c +++ b/drivers/block/loop.c @@ -70,6 +70,7 @@ struct loop_device { struct rb_root worker_tree; struct timer_list timer; bool sysfs_inited; + unsigned queued_wait_write; struct request_queue *lo_queue; struct blk_mq_tag_set tag_set; @@ -522,6 +523,30 @@ static int lo_rw_aio(struct loop_device *lo, struct loop_cmd *cmd) return 0; } +static inline bool lo_aio_need_try_nowait(struct loop_device *lo, + struct loop_cmd *cmd) +{ + struct file *file = lo->lo_backing_file; + struct inode *inode = file->f_mapping->host; + struct request *rq = blk_mq_rq_from_pdu(cmd); + + /* NOWAIT works fine for backing block device */ + if (S_ISBLK(inode->i_mode)) + return true; + + /* NOWAIT is supposed to be fine for READ */ + if (req_op(rq) == REQ_OP_READ) + return true; + + /* + * If there is any queued non-NOWAIT async WRITE , don't try new + * NOWAIT WRITE for avoiding contention + * + * Here we focus on handling stable FS block mapping via NOWAIT + */ + return READ_ONCE(lo->queued_wait_write) == 0; +} + static blk_status_t lo_rw_aio_nowait(struct loop_device *lo, struct loop_cmd *cmd) { @@ -531,6 +556,9 @@ static blk_status_t lo_rw_aio_nowait(struct loop_device *lo, if (unlikely(ret < 0)) return BLK_STS_IOERR; + if (!lo_aio_need_try_nowait(lo, cmd)) + return BLK_STS_AGAIN; + cmd->iocb.ki_flags |= IOCB_NOWAIT; ret = lo_submit_rw_aio(lo, cmd, nr_bvec); if (ret == -EAGAIN) @@ -821,12 +849,18 @@ static ssize_t loop_attr_dio_show(struct loop_device *lo, char *buf) return sysfs_emit(buf, "%s\n", dio ? "1" : "0"); } +static ssize_t loop_attr_nr_wait_write_show(struct loop_device *lo, char *buf) +{ + return sysfs_emit(buf, "%u\n", lo->queued_wait_write); +} + LOOP_ATTR_RO(backing_file); LOOP_ATTR_RO(offset); LOOP_ATTR_RO(sizelimit); LOOP_ATTR_RO(autoclear); LOOP_ATTR_RO(partscan); LOOP_ATTR_RO(dio); +LOOP_ATTR_RO(nr_wait_write); static struct attribute *loop_attrs[] = { &loop_attr_backing_file.attr, @@ -835,6 +869,7 @@ static struct attribute *loop_attrs[] = { &loop_attr_autoclear.attr, &loop_attr_partscan.attr, &loop_attr_dio.attr, + &loop_attr_nr_wait_write.attr, NULL, }; @@ -910,6 +945,30 @@ static inline int queue_on_root_worker(struct cgroup_subsys_state *css) } #endif +static inline void loop_inc_wait_write(struct loop_device *lo, struct loop_cmd *cmd) +{ + lockdep_assert_held(&lo->lo_mutex); + + if (cmd->use_aio){ + struct request *rq = blk_mq_rq_from_pdu(cmd); + + if (req_op(rq) == REQ_OP_WRITE) + lo->queued_wait_write += 1; + } +} + +static inline void loop_dec_wait_write(struct loop_device *lo, struct loop_cmd *cmd) +{ + lockdep_assert_held(&lo->lo_mutex); + + if (cmd->use_aio){ + struct request *rq = blk_mq_rq_from_pdu(cmd); + + if (req_op(rq) == REQ_OP_WRITE) + lo->queued_wait_write -= 1; + } +} + static void loop_queue_work(struct loop_device *lo, struct loop_cmd *cmd) { struct request __maybe_unused *rq = blk_mq_rq_from_pdu(cmd); @@ -992,6 +1051,7 @@ static void loop_queue_work(struct loop_device *lo, struct loop_cmd *cmd) work = &lo->rootcg_work; cmd_list = &lo->rootcg_cmd_list; } + loop_inc_wait_write(lo, cmd); list_add_tail(&cmd->list_entry, cmd_list); queue_work(lo->workqueue, work); spin_unlock_irq(&lo->lo_work_lock); @@ -2051,6 +2111,7 @@ static void loop_process_work(struct loop_worker *worker, cond_resched(); spin_lock_irq(&lo->lo_work_lock); + loop_dec_wait_write(lo, cmd); } /*