From patchwork Thu Jun 18 14:43:41 2020 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Jens Axboe X-Patchwork-Id: 11612317 Return-Path: Received: from mail.kernel.org (pdx-korg-mail-1.web.codeaurora.org [172.30.200.123]) by pdx-korg-patchwork-2.web.codeaurora.org (Postfix) with ESMTP id BB7C1913 for ; Thu, 18 Jun 2020 14:44:06 +0000 (UTC) Received: from kanga.kvack.org (kanga.kvack.org [205.233.56.17]) by mail.kernel.org (Postfix) with ESMTP id 88FC120890 for ; Thu, 18 Jun 2020 14:44:06 +0000 (UTC) Authentication-Results: mail.kernel.org; dkim=fail reason="signature verification failed" (2048-bit key) header.d=kernel-dk.20150623.gappssmtp.com header.i=@kernel-dk.20150623.gappssmtp.com header.b="XjGUImD9" DMARC-Filter: OpenDMARC Filter v1.3.2 mail.kernel.org 88FC120890 Authentication-Results: mail.kernel.org; dmarc=none (p=none dis=none) header.from=kernel.dk Authentication-Results: mail.kernel.org; spf=pass smtp.mailfrom=owner-linux-mm@kvack.org Received: by kanga.kvack.org (Postfix) id 3AB238D0012; Thu, 18 Jun 2020 10:44:03 -0400 (EDT) Delivered-To: linux-mm-outgoing@kvack.org Received: by kanga.kvack.org (Postfix, from userid 40) id 2A6858D0018; Thu, 18 Jun 2020 10:44:03 -0400 (EDT) X-Original-To: int-list-linux-mm@kvack.org X-Delivered-To: int-list-linux-mm@kvack.org Received: by kanga.kvack.org (Postfix, from userid 63042) id 11F458D0019; Thu, 18 Jun 2020 10:44:03 -0400 (EDT) X-Original-To: linux-mm@kvack.org X-Delivered-To: linux-mm@kvack.org Received: from forelay.hostedemail.com (smtprelay0253.hostedemail.com [216.40.44.253]) by kanga.kvack.org (Postfix) with ESMTP id E9CB68D0018 for ; Thu, 18 Jun 2020 10:44:02 -0400 (EDT) Received: from smtpin21.hostedemail.com (10.5.19.251.rfc1918.com [10.5.19.251]) by forelay02.hostedemail.com (Postfix) with ESMTP id A505C6CA7 for ; Thu, 18 Jun 2020 14:44:02 +0000 (UTC) X-FDA: 76942602324.21.idea97_0504bd326e11 Received: from filter.hostedemail.com (10.5.16.251.rfc1918.com [10.5.16.251]) by smtpin21.hostedemail.com (Postfix) with ESMTP id 73451180442CD for ; Thu, 18 Jun 2020 14:44:02 +0000 (UTC) X-Spam-Summary: 2,0,0,26e7873f6e276b3f,d41d8cd98f00b204,axboe@kernel.dk,,RULES_HIT:41:355:379:541:800:960:988:989:1260:1311:1314:1345:1359:1431:1437:1515:1534:1541:1711:1730:1747:1777:1792:2194:2198:2199:2200:2393:2559:2562:3138:3139:3140:3141:3142:3352:3865:3867:3871:3874:4250:4321:4605:5007:6261:6653:7903:10004:11026:11473:11658:11914:12043:12296:12438:12517:12519:12555:12895:12986:13069:13153:13228:13255:13311:13357:13894:14096:14181:14384:14394:14721:21080:21444:21451:21627:21990:30046:30054,0,RBL:209.85.216.65:@kernel.dk:.lbl8.mailshell.net-66.100.201.201 62.2.0.100,CacheIP:none,Bayesian:0.5,0.5,0.5,Netcheck:none,DomainCache:0,MSF:not bulk,SPF:fp,MSBL:0,DNSBL:neutral,Custom_rules:0:0:0,LFtime:23,LUA_SUMMARY:none X-HE-Tag: idea97_0504bd326e11 X-Filterd-Recvd-Size: 4695 Received: from mail-pj1-f65.google.com (mail-pj1-f65.google.com [209.85.216.65]) by imf13.hostedemail.com (Postfix) with ESMTP for ; Thu, 18 Jun 2020 14:44:01 +0000 (UTC) Received: by mail-pj1-f65.google.com with SMTP id jz3so2649876pjb.0 for ; Thu, 18 Jun 2020 07:44:01 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=kernel-dk.20150623.gappssmtp.com; s=20150623; h=from:to:cc:subject:date:message-id:in-reply-to:references:reply-to :mime-version:content-transfer-encoding; bh=AMBC//GM9TUmKqc/Ee7MUTX4MVxXaivvBIm8XTKWtjI=; b=XjGUImD9rS9VeH3ATYR5UN46KsG/kGd5i8ObOVULAJ3bWbAHLUKz+Nt/bzRQeCC7k+ q4pmWPc7rGRsnlJnmLarBBAj7QbxKCn+OtJBti2v/itk1DKcuvDFfV26CpsOgPwwVj46 oKdSZT+E6O3bgsFXk2UysYjSDRy6zj4yUK78Rv1eqtIwgcAQ5nn/tpiH68Y8GD4/iHUw 3O1/2ihPs06whdoc10VZDpvDnnO0QVgM3L+4vMq/X6KTuGPCEjHx9ydMZTrOPjVK9Bkv Qkf+MhMg3TkdzkCd8QX51qCa7+tmj/0j1O1H3Kj4Zv1sGjsxfa6mxtY9/zGOSIjiER3+ piRw== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20161025; h=x-gm-message-state:from:to:cc:subject:date:message-id:in-reply-to :references:reply-to:mime-version:content-transfer-encoding; bh=AMBC//GM9TUmKqc/Ee7MUTX4MVxXaivvBIm8XTKWtjI=; b=g3m9aYgusAYeFsf5SIOoliQuQDHTG+qxc/Ix69i4IUb+h2dNOc0Q6qrhuCWIqFlDXX B12PKOKmC+4tI5UqlHW26UMiHOMBhgoDlEPX9zgaBzj/FgvhwgJaYhDEqrKJSVDYnqdB lslnbEQovbdXGZ0fUE+jW7XBf1/ngGx8fOLaqZrx2L3KZH/l5YR/D/v+CrXWI3zaFtZ3 nI3MlWT0MG11txnSt5nH11I+6rvMpoxeCnZUg50BaXzgg7Mj2WsG1oFw69iwL3R0HKt2 yN37fWqY7vX6jBE80XILcAPta/hax+zzLfywjM3AfMH4MRgXubao0QMuf72MivfcInC6 VrHA== X-Gm-Message-State: AOAM532rPJRx/uM6WYFoCRpd/j9lC+pul90YUKwCsQC7+uJnTIVEzT3T SnLoPJlz4ldwwuTcIFhkb30hHg== X-Google-Smtp-Source: ABdhPJxPg9RpVdrGQ4Z6ww3Iq/Db4vGVv54cHwt8FNsBk1NOsUw3hrbUnoL/Xc/AmyjDRE+S7btjoQ== X-Received: by 2002:a17:90b:3004:: with SMTP id hg4mr5050010pjb.208.1592491440947; Thu, 18 Jun 2020 07:44:00 -0700 (PDT) Received: from x1.localdomain ([65.144.74.34]) by smtp.gmail.com with ESMTPSA id g9sm3127197pfm.151.2020.06.18.07.44.00 (version=TLS1_3 cipher=TLS_AES_256_GCM_SHA384 bits=256/256); Thu, 18 Jun 2020 07:44:00 -0700 (PDT) From: Jens Axboe To: io-uring@vger.kernel.org Cc: linux-fsdevel@vger.kernel.org, linux-kernel@vger.kernel.org, linux-mm@kvack.org, akpm@linux-foundation.org, Jens Axboe Subject: [PATCH 01/15] block: provide plug based way of signaling forced no-wait semantics Date: Thu, 18 Jun 2020 08:43:41 -0600 Message-Id: <20200618144355.17324-2-axboe@kernel.dk> X-Mailer: git-send-email 2.27.0 In-Reply-To: <20200618144355.17324-1-axboe@kernel.dk> References: <20200618144355.17324-1-axboe@kernel.dk> Reply-To: "[PATCHSET v7 0/15]"@smtpin21.hostedemail.com, Add@smtpin21.hostedemail.com, support@smtpin21.hostedemail.com, for@smtpin21.hostedemail.com, async@smtpin21.hostedemail.com, buffered@smtpin21.hostedemail.com, reads@smtpin21.hostedemail.com MIME-Version: 1.0 X-Rspamd-Queue-Id: 73451180442CD X-Spamd-Result: default: False [0.00 / 100.00] X-Rspamd-Server: rspam03 X-Bogosity: Ham, tests=bogofilter, spamicity=0.000000, version=1.2.4 Sender: owner-linux-mm@kvack.org Precedence: bulk X-Loop: owner-majordomo@kvack.org List-ID: Provide a way for the caller to specify that IO should be marked with REQ_NOWAIT to avoid blocking on allocation. Signed-off-by: Jens Axboe --- block/blk-core.c | 6 ++++++ include/linux/blkdev.h | 1 + 2 files changed, 7 insertions(+) diff --git a/block/blk-core.c b/block/blk-core.c index 03252af8c82c..62a4904db921 100644 --- a/block/blk-core.c +++ b/block/blk-core.c @@ -958,6 +958,7 @@ generic_make_request_checks(struct bio *bio) struct request_queue *q; int nr_sectors = bio_sectors(bio); blk_status_t status = BLK_STS_IOERR; + struct blk_plug *plug; char b[BDEVNAME_SIZE]; might_sleep(); @@ -971,6 +972,10 @@ generic_make_request_checks(struct bio *bio) goto end_io; } + plug = blk_mq_plug(q, bio); + if (plug && plug->nowait) + bio->bi_opf |= REQ_NOWAIT; + /* * For a REQ_NOWAIT based request, return -EOPNOTSUPP * if queue is not a request based queue. @@ -1800,6 +1805,7 @@ void blk_start_plug(struct blk_plug *plug) INIT_LIST_HEAD(&plug->cb_list); plug->rq_count = 0; plug->multiple_queues = false; + plug->nowait = false; /* * Store ordering should not be needed here, since a potential diff --git a/include/linux/blkdev.h b/include/linux/blkdev.h index 8fd900998b4e..6e067dca94cf 100644 --- a/include/linux/blkdev.h +++ b/include/linux/blkdev.h @@ -1189,6 +1189,7 @@ struct blk_plug { struct list_head cb_list; /* md requires an unplug callback */ unsigned short rq_count; bool multiple_queues; + bool nowait; }; #define BLK_MAX_REQUEST_COUNT 16 #define BLK_PLUG_FLUSH_SIZE (128 * 1024) From patchwork Thu Jun 18 14:43:42 2020 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Jens Axboe X-Patchwork-Id: 11612319 Return-Path: Received: from mail.kernel.org (pdx-korg-mail-1.web.codeaurora.org [172.30.200.123]) by pdx-korg-patchwork-2.web.codeaurora.org (Postfix) with ESMTP id 0560D159A for ; Thu, 18 Jun 2020 14:44:09 +0000 (UTC) Received: from kanga.kvack.org (kanga.kvack.org [205.233.56.17]) by mail.kernel.org (Postfix) with ESMTP id C724C20CC7 for ; Thu, 18 Jun 2020 14:44:08 +0000 (UTC) Authentication-Results: mail.kernel.org; dkim=fail reason="signature verification failed" (2048-bit key) header.d=kernel-dk.20150623.gappssmtp.com header.i=@kernel-dk.20150623.gappssmtp.com header.b="yzp2+WvV" DMARC-Filter: OpenDMARC Filter v1.3.2 mail.kernel.org C724C20CC7 Authentication-Results: mail.kernel.org; dmarc=none (p=none dis=none) header.from=kernel.dk Authentication-Results: mail.kernel.org; spf=pass smtp.mailfrom=owner-linux-mm@kvack.org Received: by kanga.kvack.org (Postfix) id C44E28D0019; Thu, 18 Jun 2020 10:44:04 -0400 (EDT) Delivered-To: linux-mm-outgoing@kvack.org Received: by kanga.kvack.org (Postfix, from userid 40) id BCCB98D0018; Thu, 18 Jun 2020 10:44:04 -0400 (EDT) X-Original-To: int-list-linux-mm@kvack.org X-Delivered-To: int-list-linux-mm@kvack.org Received: by kanga.kvack.org (Postfix, from userid 63042) id ABCAD8D0019; Thu, 18 Jun 2020 10:44:04 -0400 (EDT) X-Original-To: linux-mm@kvack.org X-Delivered-To: linux-mm@kvack.org Received: from forelay.hostedemail.com (smtprelay0229.hostedemail.com [216.40.44.229]) by kanga.kvack.org (Postfix) with ESMTP id 9348C8D0018 for ; Thu, 18 Jun 2020 10:44:04 -0400 (EDT) Received: from smtpin17.hostedemail.com (10.5.19.251.rfc1918.com [10.5.19.251]) by forelay01.hostedemail.com (Postfix) with ESMTP id 4E7D8180AD807 for ; Thu, 18 Jun 2020 14:44:04 +0000 (UTC) X-FDA: 76942602408.17.rice66_37128b526e11 Received: from filter.hostedemail.com (10.5.16.251.rfc1918.com [10.5.16.251]) by smtpin17.hostedemail.com (Postfix) with ESMTP id 172CC180D0184 for ; Thu, 18 Jun 2020 14:44:04 +0000 (UTC) X-Spam-Summary: 2,0,0,c413f10283dccbb3,d41d8cd98f00b204,axboe@kernel.dk,,RULES_HIT:41:69:355:379:541:560:800:960:973:988:989:1260:1311:1314:1345:1359:1431:1437:1515:1535:1542:1711:1730:1747:1777:1792:2393:2559:2562:2693:2892:2902:3138:3139:3140:3141:3142:3353:3865:3867:3868:4250:5007:6261:6653:10004:11026:11473:11658:11914:12043:12296:12438:12517:12519:12555:12895:13894:14096:14181:14394:14721:14877:21080:21444:21451:21627:21796:21939:21990:30034:30036:30045:30054:30075,0,RBL:209.85.216.67:@kernel.dk:.lbl8.mailshell.net-62.2.0.100 66.100.201.201,CacheIP:none,Bayesian:0.5,0.5,0.5,Netcheck:none,DomainCache:0,MSF:not bulk,SPF:fp,MSBL:0,DNSBL:neutral,Custom_rules:0:0:0,LFtime:23,LUA_SUMMARY:none X-HE-Tag: rice66_37128b526e11 X-Filterd-Recvd-Size: 5162 Received: from mail-pj1-f67.google.com (mail-pj1-f67.google.com [209.85.216.67]) by imf14.hostedemail.com (Postfix) with ESMTP for ; Thu, 18 Jun 2020 14:44:03 +0000 (UTC) Received: by mail-pj1-f67.google.com with SMTP id b7so3467104pju.0 for ; Thu, 18 Jun 2020 07:44:03 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=kernel-dk.20150623.gappssmtp.com; s=20150623; h=from:to:cc:subject:date:message-id:in-reply-to:references:reply-to :mime-version:content-transfer-encoding; bh=aZaDFCszZJWwvw+YU15551W6DS/tp5Q0zZ/e2mzcPKg=; b=yzp2+WvVYDYAjxYEzGcL7PkavHxDy3cpr9qzW1IWcotTTR9f4VhesgN5dhZI94+PsU D9XrCxqsfO+3+KrzN5/i59EGbZklLXhJ8EEfRkcKjS+RSy9l3JJAAb3DNGy49QESvyup y+54KKaMpJCnfzL+F+KC8Nq7RiZ+0vIr9IiDsumf++0aVkCJenAnaKTtA3JP5ygXQhYB 2HbmH52rlEtjDxPlJY6OwsA0PBg77wgg7/2VGxIMl44vnzaycw7+hUNxlA6Vq9Gzvp1p BDytFguC2D8/8IQpvQ3NiTjxRjsGSXn+74N6mA5zAWSn0EOSGif2RsltVaUOkZ/y9GnI oyUw== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20161025; h=x-gm-message-state:from:to:cc:subject:date:message-id:in-reply-to :references:reply-to:mime-version:content-transfer-encoding; bh=aZaDFCszZJWwvw+YU15551W6DS/tp5Q0zZ/e2mzcPKg=; b=W3cbmsdgIDkNXojFT16dfO/wfDJL8x3/UzRyObO7kc9poBtElDiV1vWcSadVdbweEi pUJw2u94nONUuNf5uHr4sMhzxgGgNJWftGF7wIQ2AHX3VLGxVxxalMrxiymM6L9Q48Mb SVzjMvMZvKrvpV5FL2p1O8qd8ocERdKGyp9gwktC/TeZ8aW34D925/rTcxQwsYB7Jy+d MR5kmQw2A8WWAp9ZBsLHDMWO9I2+cOBBHQrsvDCm2FoLWsl6PcrwV8z6ufnWZxmyekm0 eHgYuduao53hBMmPItVVNdd2ghEpvMTH3cevVXeyUKBTTXvqKud7R2DmGaQOBfZ+SsQO /Thw== X-Gm-Message-State: AOAM530iIYUeSB/lrBeMZaKOtQYN3M2r6y2qyH6toMweCN5D4/rLPtAD kDvYHJjaXIH3WcYuPr8gE8asXw== X-Google-Smtp-Source: ABdhPJwcpBuWJToBprbE+kBnwBY9JxdUaCcM4SRk7uPRoXVsDbkErSY+i9Q0PI2mKLg9/3NhPQrRsQ== X-Received: by 2002:a17:90b:915:: with SMTP id bo21mr4702163pjb.52.1592491442581; Thu, 18 Jun 2020 07:44:02 -0700 (PDT) Received: from x1.localdomain ([65.144.74.34]) by smtp.gmail.com with ESMTPSA id g9sm3127197pfm.151.2020.06.18.07.44.01 (version=TLS1_3 cipher=TLS_AES_256_GCM_SHA384 bits=256/256); Thu, 18 Jun 2020 07:44:01 -0700 (PDT) From: Jens Axboe To: io-uring@vger.kernel.org Cc: linux-fsdevel@vger.kernel.org, linux-kernel@vger.kernel.org, linux-mm@kvack.org, akpm@linux-foundation.org, Jens Axboe Subject: [PATCH 02/15] io_uring: always plug for any number of IOs Date: Thu, 18 Jun 2020 08:43:42 -0600 Message-Id: <20200618144355.17324-3-axboe@kernel.dk> X-Mailer: git-send-email 2.27.0 In-Reply-To: <20200618144355.17324-1-axboe@kernel.dk> References: <20200618144355.17324-1-axboe@kernel.dk> Reply-To: "[PATCHSET v7 0/15]"@smtpin17.hostedemail.com, Add@smtpin17.hostedemail.com, support@smtpin17.hostedemail.com, for@smtpin17.hostedemail.com, async@smtpin17.hostedemail.com, buffered@smtpin17.hostedemail.com, reads@smtpin17.hostedemail.com MIME-Version: 1.0 X-Rspamd-Queue-Id: 172CC180D0184 X-Spamd-Result: default: False [0.00 / 100.00] X-Rspamd-Server: rspam03 X-Bogosity: Ham, tests=bogofilter, spamicity=0.000000, version=1.2.4 Sender: owner-linux-mm@kvack.org Precedence: bulk X-Loop: owner-majordomo@kvack.org List-ID: Currently we only plug if we're doing more than two request. We're going to be relying on always having the plug there to pass down information, so plug unconditionally. Signed-off-by: Jens Axboe --- fs/io_uring.c | 15 +++++---------- 1 file changed, 5 insertions(+), 10 deletions(-) diff --git a/fs/io_uring.c b/fs/io_uring.c index b14a8e6a0e15..ca78dd7c79da 100644 --- a/fs/io_uring.c +++ b/fs/io_uring.c @@ -676,7 +676,6 @@ struct io_kiocb { }; }; -#define IO_PLUG_THRESHOLD 2 #define IO_IOPOLL_BATCH 8 struct io_submit_state { @@ -5914,7 +5913,7 @@ static int io_init_req(struct io_ring_ctx *ctx, struct io_kiocb *req, static int io_submit_sqes(struct io_ring_ctx *ctx, unsigned int nr, struct file *ring_file, int ring_fd) { - struct io_submit_state state, *statep = NULL; + struct io_submit_state state; struct io_kiocb *link = NULL; int i, submitted = 0; @@ -5931,10 +5930,7 @@ static int io_submit_sqes(struct io_ring_ctx *ctx, unsigned int nr, if (!percpu_ref_tryget_many(&ctx->refs, nr)) return -EAGAIN; - if (nr > IO_PLUG_THRESHOLD) { - io_submit_state_start(&state, nr); - statep = &state; - } + io_submit_state_start(&state, nr); ctx->ring_fd = ring_fd; ctx->ring_file = ring_file; @@ -5949,14 +5945,14 @@ static int io_submit_sqes(struct io_ring_ctx *ctx, unsigned int nr, io_consume_sqe(ctx); break; } - req = io_alloc_req(ctx, statep); + req = io_alloc_req(ctx, &state); if (unlikely(!req)) { if (!submitted) submitted = -EAGAIN; break; } - err = io_init_req(ctx, req, sqe, statep); + err = io_init_req(ctx, req, sqe, &state); io_consume_sqe(ctx); /* will complete beyond this point, count as submitted */ submitted++; @@ -5982,8 +5978,7 @@ static int io_submit_sqes(struct io_ring_ctx *ctx, unsigned int nr, } if (link) io_queue_link_head(link); - if (statep) - io_submit_state_end(&state); + io_submit_state_end(&state); /* Commit SQ ring head once we've consumed and submitted all SQEs */ io_commit_sqring(ctx); From patchwork Thu Jun 18 14:43:43 2020 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Jens Axboe X-Patchwork-Id: 11612321 Return-Path: Received: from mail.kernel.org (pdx-korg-mail-1.web.codeaurora.org [172.30.200.123]) by pdx-korg-patchwork-2.web.codeaurora.org (Postfix) with ESMTP id 2558B14E3 for ; Thu, 18 Jun 2020 14:44:11 +0000 (UTC) Received: from kanga.kvack.org (kanga.kvack.org [205.233.56.17]) by mail.kernel.org (Postfix) with ESMTP id E62EC20897 for ; Thu, 18 Jun 2020 14:44:10 +0000 (UTC) Authentication-Results: mail.kernel.org; dkim=fail reason="signature verification failed" (2048-bit key) header.d=kernel-dk.20150623.gappssmtp.com header.i=@kernel-dk.20150623.gappssmtp.com header.b="uL0fwmVf" DMARC-Filter: OpenDMARC Filter v1.3.2 mail.kernel.org E62EC20897 Authentication-Results: mail.kernel.org; dmarc=none (p=none dis=none) header.from=kernel.dk Authentication-Results: mail.kernel.org; spf=pass smtp.mailfrom=owner-linux-mm@kvack.org Received: by kanga.kvack.org (Postfix) id 3E96E8D001C; Thu, 18 Jun 2020 10:44:07 -0400 (EDT) Delivered-To: linux-mm-outgoing@kvack.org Received: by kanga.kvack.org (Postfix, from userid 40) id 234C38D0018; Thu, 18 Jun 2020 10:44:07 -0400 (EDT) X-Original-To: int-list-linux-mm@kvack.org X-Delivered-To: int-list-linux-mm@kvack.org Received: by kanga.kvack.org (Postfix, from userid 63042) id 0F96A8D001B; Thu, 18 Jun 2020 10:44:07 -0400 (EDT) X-Original-To: linux-mm@kvack.org X-Delivered-To: linux-mm@kvack.org Received: from forelay.hostedemail.com (smtprelay0080.hostedemail.com [216.40.44.80]) by kanga.kvack.org (Postfix) with ESMTP id EB8BB8D0018 for ; Thu, 18 Jun 2020 10:44:06 -0400 (EDT) Received: from smtpin04.hostedemail.com (10.5.19.251.rfc1918.com [10.5.19.251]) by forelay04.hostedemail.com (Postfix) with ESMTP id A53F89F5D6 for ; Thu, 18 Jun 2020 14:44:06 +0000 (UTC) X-FDA: 76942602492.04.paint36_2807cd626e11 Received: from filter.hostedemail.com (10.5.16.251.rfc1918.com [10.5.16.251]) by smtpin04.hostedemail.com (Postfix) with ESMTP id 727FD80A4EBE for ; Thu, 18 Jun 2020 14:44:05 +0000 (UTC) X-Spam-Summary: 2,0,0,97ba831710f59da8,d41d8cd98f00b204,axboe@kernel.dk,,RULES_HIT:41:355:379:541:800:960:988:989:1260:1311:1314:1345:1359:1431:1437:1515:1535:1542:1711:1730:1747:1777:1792:2194:2199:2393:2559:2562:2692:2903:3138:3139:3140:3141:3142:3354:3865:3866:3867:3868:3870:3871:3872:3874:4250:4321:5007:6117:6119:6121:6261:6653:10004:11026:11658:11914:12043:12291:12296:12438:12517:12519:12555:12895:12986:13894:14096:14181:14394:14721:14877:21080:21212:21433:21444:21627:21990:30045:30054:30070,0,RBL:209.85.214.196:@kernel.dk:.lbl8.mailshell.net-66.100.201.201 62.2.0.100,CacheIP:none,Bayesian:0.5,0.5,0.5,Netcheck:none,DomainCache:0,MSF:not bulk,SPF:fp,MSBL:0,DNSBL:neutral,Custom_rules:0:0:0,LFtime:25,LUA_SUMMARY:none X-HE-Tag: paint36_2807cd626e11 X-Filterd-Recvd-Size: 5626 Received: from mail-pl1-f196.google.com (mail-pl1-f196.google.com [209.85.214.196]) by imf12.hostedemail.com (Postfix) with ESMTP for ; Thu, 18 Jun 2020 14:44:05 +0000 (UTC) Received: by mail-pl1-f196.google.com with SMTP id y18so2525132plr.4 for ; Thu, 18 Jun 2020 07:44:04 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=kernel-dk.20150623.gappssmtp.com; s=20150623; h=from:to:cc:subject:date:message-id:in-reply-to:references:reply-to :mime-version:content-transfer-encoding; bh=HUMIpAnpDXTm7durxzsxq78uIppsf6NP0VS0tAu1naU=; b=uL0fwmVf7za35bF7WmhIeuLwTRYPYomNz3qpWAcNPJ4MG8GTRnh40j/G/FSQGZeqHn yQkV1IezTLpFjD50TYMgv1HY78G4RgY498zLaJMuA+l8xEee6zj/2I82z6xH0BShFzod Jw88wuY/pILeFUtVbhY9sbTrVoWY7AC3u3FJAQooqqxLaL58sSt1DjQd+DiakJiej111 3QKNAaz9kpVK0JkkwmB/tU2PK7EUI1l3/XQW2Hhdy1LlQkxSISsyAWXCpJiSM6HpaavS QcI6gfKDo0Ou2qvSuNEV0Pw2sksh+dDzkS5v9GpxMofkwta1H+6MYdHHM9azIlc5CFhg lSog== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20161025; h=x-gm-message-state:from:to:cc:subject:date:message-id:in-reply-to :references:reply-to:mime-version:content-transfer-encoding; bh=HUMIpAnpDXTm7durxzsxq78uIppsf6NP0VS0tAu1naU=; b=BUQR4dYp5HPjW9KWQD6FlnhmFkGu+MllMV/KRpqjirxLkkPxs8EKbJb7165NgUT35u PiUX5aD3FOpD9v2jnfY7IwfHDGT8HeLVpdLnDXCefyIoLhR5xfUIazIFueqrVbL49x08 D3TVqiBG1QY1AsUpRCyWlpDiVu4ebNR37VBKh42b3/xryrp6qw/HFwDt4LVYOAFZn6sb upHU1r9t4eFl0LbB7WwevugeKo82CptUZh10/RnFqJLH/a6u98iPrSkuvplojJ8upk9H XDXhcEDe/qCJEoVoGaofLoVWRW2Z9cMx/u0irUgVQuNExRtSKLYtPQ7ECEgE5RZLmzui KmFA== X-Gm-Message-State: AOAM531+OLoqh+ur9FGIaswWQmoX/DmZe0JazPt5buPFBtpHnL2Uub7u 9Liskhtnxqj2qUoQNSV6N9taBA== X-Google-Smtp-Source: ABdhPJzURLyFzMozy7yIPWEvSTKNlwHInRbt5PeVmy/Fa+Ew58olXOewrYlOoVFbj5T5wAt30tH4HQ== X-Received: by 2002:a17:90a:de1:: with SMTP id 88mr4809422pjv.124.1592491443796; Thu, 18 Jun 2020 07:44:03 -0700 (PDT) Received: from x1.localdomain ([65.144.74.34]) by smtp.gmail.com with ESMTPSA id g9sm3127197pfm.151.2020.06.18.07.44.02 (version=TLS1_3 cipher=TLS_AES_256_GCM_SHA384 bits=256/256); Thu, 18 Jun 2020 07:44:03 -0700 (PDT) From: Jens Axboe To: io-uring@vger.kernel.org Cc: linux-fsdevel@vger.kernel.org, linux-kernel@vger.kernel.org, linux-mm@kvack.org, akpm@linux-foundation.org, Jens Axboe Subject: [PATCH 03/15] io_uring: catch -EIO from buffered issue request failure Date: Thu, 18 Jun 2020 08:43:43 -0600 Message-Id: <20200618144355.17324-4-axboe@kernel.dk> X-Mailer: git-send-email 2.27.0 In-Reply-To: <20200618144355.17324-1-axboe@kernel.dk> References: <20200618144355.17324-1-axboe@kernel.dk> Reply-To: "[PATCHSET v7 0/15]"@smtpin04.hostedemail.com, Add@smtpin04.hostedemail.com, support@smtpin04.hostedemail.com, for@smtpin04.hostedemail.com, async@smtpin04.hostedemail.com, buffered@smtpin04.hostedemail.com, reads@smtpin04.hostedemail.com MIME-Version: 1.0 X-Rspamd-Queue-Id: 727FD80A4EBE X-Spamd-Result: default: False [0.00 / 100.00] X-Rspamd-Server: rspam01 X-Bogosity: Ham, tests=bogofilter, spamicity=0.000000, version=1.2.4 Sender: owner-linux-mm@kvack.org Precedence: bulk X-Loop: owner-majordomo@kvack.org List-ID: -EIO bubbles up like -EAGAIN if we fail to allocate a request at the lower level. Play it safe and treat it like -EAGAIN in terms of sync retry, to avoid passing back an errant -EIO. Catch some of these early for block based file, as non-mq devices generally do not support NOWAIT. That saves us some overhead by not first trying, then retrying from async context. We can go straight to async punt instead. Signed-off-by: Jens Axboe --- fs/io_uring.c | 28 +++++++++++++++++++++++----- 1 file changed, 23 insertions(+), 5 deletions(-) diff --git a/fs/io_uring.c b/fs/io_uring.c index ca78dd7c79da..2e257c5a1866 100644 --- a/fs/io_uring.c +++ b/fs/io_uring.c @@ -2088,6 +2088,15 @@ static struct file *__io_file_get(struct io_submit_state *state, int fd) return state->file; } +static bool io_bdev_nowait(struct block_device *bdev) +{ +#ifdef CONFIG_BLOCK + return !bdev || queue_is_mq(bdev_get_queue(bdev)); +#else + return true; +#endif +} + /* * If we tracked the file through the SCM inflight mechanism, we could support * any file. For now, just ensure that anything potentially problematic is done @@ -2097,10 +2106,19 @@ static bool io_file_supports_async(struct file *file, int rw) { umode_t mode = file_inode(file)->i_mode; - if (S_ISBLK(mode) || S_ISCHR(mode) || S_ISSOCK(mode)) - return true; - if (S_ISREG(mode) && file->f_op != &io_uring_fops) + if (S_ISBLK(mode)) { + if (io_bdev_nowait(file->f_inode->i_bdev)) + return true; + return false; + } + if (S_ISCHR(mode) || S_ISSOCK(mode)) return true; + if (S_ISREG(mode)) { + if (io_bdev_nowait(file->f_inode->i_sb->s_bdev) && + file->f_op != &io_uring_fops) + return true; + return false; + } /* any ->read/write should understand O_NONBLOCK */ if (file->f_flags & O_NONBLOCK) @@ -2650,7 +2668,7 @@ static int io_read(struct io_kiocb *req, bool force_nonblock) iov_count = iov_iter_count(&iter); ret = rw_verify_area(READ, req->file, &kiocb->ki_pos, iov_count); if (!ret) { - ssize_t ret2; + ssize_t ret2 = 0; if (req->file->f_op->read_iter) ret2 = call_read_iter(req->file, kiocb, &iter); @@ -2658,7 +2676,7 @@ static int io_read(struct io_kiocb *req, bool force_nonblock) ret2 = loop_rw_iter(READ, req->file, kiocb, &iter); /* Catch -EAGAIN return for forced non-blocking submission */ - if (!force_nonblock || ret2 != -EAGAIN) { + if (!force_nonblock || (ret2 != -EAGAIN && ret2 != -EIO)) { kiocb_done(kiocb, ret2); } else { copy_iov: From patchwork Thu Jun 18 14:43:44 2020 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Jens Axboe X-Patchwork-Id: 11612323 Return-Path: Received: from mail.kernel.org (pdx-korg-mail-1.web.codeaurora.org [172.30.200.123]) by pdx-korg-patchwork-2.web.codeaurora.org (Postfix) with ESMTP id 8DD74913 for ; Thu, 18 Jun 2020 14:44:13 +0000 (UTC) Received: from kanga.kvack.org (kanga.kvack.org [205.233.56.17]) by mail.kernel.org (Postfix) with ESMTP id 49FAD20888 for ; Thu, 18 Jun 2020 14:44:13 +0000 (UTC) Authentication-Results: mail.kernel.org; dkim=fail reason="signature verification failed" (2048-bit key) header.d=kernel-dk.20150623.gappssmtp.com header.i=@kernel-dk.20150623.gappssmtp.com header.b="veVE78EJ" DMARC-Filter: OpenDMARC Filter v1.3.2 mail.kernel.org 49FAD20888 Authentication-Results: mail.kernel.org; dmarc=none (p=none dis=none) header.from=kernel.dk Authentication-Results: mail.kernel.org; spf=pass smtp.mailfrom=owner-linux-mm@kvack.org Received: by kanga.kvack.org (Postfix) id 7D7C88D001B; Thu, 18 Jun 2020 10:44:07 -0400 (EDT) Delivered-To: linux-mm-outgoing@kvack.org Received: by kanga.kvack.org (Postfix, from userid 40) id 7888F8D0018; Thu, 18 Jun 2020 10:44:07 -0400 (EDT) X-Original-To: int-list-linux-mm@kvack.org X-Delivered-To: int-list-linux-mm@kvack.org Received: by kanga.kvack.org (Postfix, from userid 63042) id 607288D001D; Thu, 18 Jun 2020 10:44:07 -0400 (EDT) X-Original-To: linux-mm@kvack.org X-Delivered-To: linux-mm@kvack.org Received: from forelay.hostedemail.com (smtprelay0066.hostedemail.com [216.40.44.66]) by kanga.kvack.org (Postfix) with ESMTP id 30CE68D001B for ; Thu, 18 Jun 2020 10:44:07 -0400 (EDT) Received: from smtpin28.hostedemail.com (10.5.19.251.rfc1918.com [10.5.19.251]) by forelay03.hostedemail.com (Postfix) with ESMTP id DF8F88248047 for ; Thu, 18 Jun 2020 14:44:06 +0000 (UTC) X-FDA: 76942602492.28.wing00_1c03db226e11 Received: from filter.hostedemail.com (10.5.16.251.rfc1918.com [10.5.16.251]) by smtpin28.hostedemail.com (Postfix) with ESMTP id B88B09F874 for ; Thu, 18 Jun 2020 14:44:06 +0000 (UTC) X-Spam-Summary: 2,0,0,33cf218b73021b71,d41d8cd98f00b204,axboe@kernel.dk,,RULES_HIT:1:2:41:355:379:541:800:960:966:973:988:989:1260:1311:1314:1345:1359:1431:1437:1515:1605:1730:1747:1777:1792:2196:2199:2393:2559:2562:2693:3138:3139:3140:3141:3142:3865:3866:3867:3868:3870:3871:3872:3874:4050:4385:4605:5007:6119:6261:6653:8603:9592:10004:11026:11473:11658:11914:12043:12291:12296:12438:12517:12519:12555:12683:12895:12986:13161:13229:13255:13894:14394:21080:21212:21324:21444:21611:21627:21740:21990:30045:30054,0,RBL:209.85.210.193:@kernel.dk:.lbl8.mailshell.net-66.100.201.201 62.2.0.100,CacheIP:none,Bayesian:0.5,0.5,0.5,Netcheck:none,DomainCache:0,MSF:not bulk,SPF:fp,MSBL:0,DNSBL:neutral,Custom_rules:0:0:0,LFtime:24,LUA_SUMMARY:none X-HE-Tag: wing00_1c03db226e11 X-Filterd-Recvd-Size: 10157 Received: from mail-pf1-f193.google.com (mail-pf1-f193.google.com [209.85.210.193]) by imf36.hostedemail.com (Postfix) with ESMTP for ; Thu, 18 Jun 2020 14:44:06 +0000 (UTC) Received: by mail-pf1-f193.google.com with SMTP id 64so2869598pfv.11 for ; Thu, 18 Jun 2020 07:44:05 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=kernel-dk.20150623.gappssmtp.com; s=20150623; h=from:to:cc:subject:date:message-id:in-reply-to:references:reply-to :mime-version:content-transfer-encoding; bh=tVJM5ezTdcIMG+aKDW1osETe1l5d87INy1YG0FUdQcM=; b=veVE78EJYXVACUi1db6AgzNJIzh1hHgYOEDwJ41A29iyHkJ0tRhmtt2MLXV57P+t9T huAegqWFdBcpK4nimSirWfSC0b9sPD6lnc/YrwOmnptgxWF7i9cd+CEONozpAruie+nd wPqh4hcqGMRiDgDp72UNJ9uJRJ3Tm6xYgQAXv7jwxFhQBMol00k7MbnVC0E69rPw6FIM 7UKG7C5tnxjAntploc2dKWqJuB+isxJpInUjNRMdqtbp8QrgmRcJRGjgnIdC2OqwERSf xgxD3JLAUc+AU3m+cc8OrVO8os02InMtoRgHoCVgMKC/eW5Z+iV6e3xMpQFkEs3n9M84 vhKg== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20161025; h=x-gm-message-state:from:to:cc:subject:date:message-id:in-reply-to :references:reply-to:mime-version:content-transfer-encoding; bh=tVJM5ezTdcIMG+aKDW1osETe1l5d87INy1YG0FUdQcM=; b=V6SkJd0srkIN365R2b375g9h40828T311dvwg85romOPXDfc2dnkJQyn/oR6mNmZsC 6BTdZKKrERgQKhvWH+Ro2fVSiTUwhlZ2kO2O0T4D9WS9dErim87XssnU+CebzGcvzJto PlbC708K12o4xq4Wq7sMsIvPga4VklQ7/Lw10eCQOrvlANRJkeihzxsBm04oam3uSBgw sB4s9LaqKg4SSdtbYfkEVOdDrHiANGfmWyj0Ngu07AzAooxm3W0Bq22bzV8uqSiDoSY6 +BQGcTreMwx7OmaESnkkHQwHeRPJnXEyzpWYzACyhHZIfiBKKVxlZJ2YZjtydwN35gWf YolA== X-Gm-Message-State: AOAM530ZNX6x3RKpvy5cJWrbMT3x8ingb0HfyR8fEIJVCmP5feojZflf 15EUh9A8vqK9yuBITdWgrufL8Q== X-Google-Smtp-Source: ABdhPJybozkuJOyPuS+aF/bntPw1nY8EdCmOZ0hRiMVA9zO7UZT6xbjz8eMsCaZwMXp2bvsabGcJ4A== X-Received: by 2002:aa7:9537:: with SMTP id c23mr3848900pfp.149.1592491444963; Thu, 18 Jun 2020 07:44:04 -0700 (PDT) Received: from x1.localdomain ([65.144.74.34]) by smtp.gmail.com with ESMTPSA id g9sm3127197pfm.151.2020.06.18.07.44.03 (version=TLS1_3 cipher=TLS_AES_256_GCM_SHA384 bits=256/256); Thu, 18 Jun 2020 07:44:04 -0700 (PDT) From: Jens Axboe To: io-uring@vger.kernel.org Cc: linux-fsdevel@vger.kernel.org, linux-kernel@vger.kernel.org, linux-mm@kvack.org, akpm@linux-foundation.org, Jens Axboe Subject: [PATCH 04/15] io_uring: re-issue block requests that failed because of resources Date: Thu, 18 Jun 2020 08:43:44 -0600 Message-Id: <20200618144355.17324-5-axboe@kernel.dk> X-Mailer: git-send-email 2.27.0 In-Reply-To: <20200618144355.17324-1-axboe@kernel.dk> References: <20200618144355.17324-1-axboe@kernel.dk> Reply-To: "[PATCHSET v7 0/15]"@smtpin28.hostedemail.com, Add@smtpin28.hostedemail.com, support@smtpin28.hostedemail.com, for@smtpin28.hostedemail.com, async@smtpin28.hostedemail.com, buffered@smtpin28.hostedemail.com, reads@smtpin28.hostedemail.com MIME-Version: 1.0 X-Rspamd-Queue-Id: B88B09F874 X-Spamd-Result: default: False [0.00 / 100.00] X-Rspamd-Server: rspam02 X-Bogosity: Ham, tests=bogofilter, spamicity=0.000000, version=1.2.4 Sender: owner-linux-mm@kvack.org Precedence: bulk X-Loop: owner-majordomo@kvack.org List-ID: Mark the plug with nowait == true, which will cause requests to avoid blocking on request allocation. If they do, we catch them and reissue them from a task_work based handler. Normally we can catch -EAGAIN directly, but the hard case is for split requests. As an example, the application issues a 512KB request. The block core will split this into 128KB if that's the max size for the device. The first request issues just fine, but we run into -EAGAIN for some latter splits for the same request. As the bio is split, we don't get to see the -EAGAIN until one of the actual reads complete, and hence we cannot handle it inline as part of submission. This does potentially cause re-reads of parts of the range, as the whole request is reissued. There's currently no better way to handle this. Signed-off-by: Jens Axboe --- fs/io_uring.c | 148 ++++++++++++++++++++++++++++++++++++++++++-------- 1 file changed, 124 insertions(+), 24 deletions(-) diff --git a/fs/io_uring.c b/fs/io_uring.c index 2e257c5a1866..40413fb9d07b 100644 --- a/fs/io_uring.c +++ b/fs/io_uring.c @@ -900,6 +900,13 @@ static int io_file_get(struct io_submit_state *state, struct io_kiocb *req, static void __io_queue_sqe(struct io_kiocb *req, const struct io_uring_sqe *sqe); +static ssize_t io_import_iovec(int rw, struct io_kiocb *req, + struct iovec **iovec, struct iov_iter *iter, + bool needs_lock); +static int io_setup_async_rw(struct io_kiocb *req, ssize_t io_size, + struct iovec *iovec, struct iovec *fast_iov, + struct iov_iter *iter); + static struct kmem_cache *req_cachep; static const struct file_operations io_uring_fops; @@ -1978,12 +1985,115 @@ static void io_complete_rw_common(struct kiocb *kiocb, long res) __io_cqring_add_event(req, res, cflags); } +static void io_sq_thread_drop_mm(struct io_ring_ctx *ctx) +{ + struct mm_struct *mm = current->mm; + + if (mm) { + kthread_unuse_mm(mm); + mmput(mm); + } +} + +static int io_sq_thread_acquire_mm(struct io_ring_ctx *ctx, + struct io_kiocb *req) +{ + if (io_op_defs[req->opcode].needs_mm && !current->mm) { + if (unlikely(!mmget_not_zero(ctx->sqo_mm))) + return -EFAULT; + kthread_use_mm(ctx->sqo_mm); + } + + return 0; +} + +#ifdef CONFIG_BLOCK +static bool io_resubmit_prep(struct io_kiocb *req, int error) +{ + struct iovec inline_vecs[UIO_FASTIOV], *iovec = inline_vecs; + ssize_t ret = -ECANCELED; + struct iov_iter iter; + int rw; + + if (error) { + ret = error; + goto end_req; + } + + switch (req->opcode) { + case IORING_OP_READV: + case IORING_OP_READ_FIXED: + case IORING_OP_READ: + rw = READ; + break; + case IORING_OP_WRITEV: + case IORING_OP_WRITE_FIXED: + case IORING_OP_WRITE: + rw = WRITE; + break; + default: + printk_once(KERN_WARNING "io_uring: bad opcode in resubmit %d\n", + req->opcode); + goto end_req; + } + + ret = io_import_iovec(rw, req, &iovec, &iter, false); + if (ret < 0) + goto end_req; + ret = io_setup_async_rw(req, ret, iovec, inline_vecs, &iter); + if (!ret) + return true; + kfree(iovec); +end_req: + io_cqring_add_event(req, ret); + req_set_fail_links(req); + io_put_req(req); + return false; +} + +static void io_rw_resubmit(struct callback_head *cb) +{ + struct io_kiocb *req = container_of(cb, struct io_kiocb, task_work); + struct io_ring_ctx *ctx = req->ctx; + int err; + + __set_current_state(TASK_RUNNING); + + err = io_sq_thread_acquire_mm(ctx, req); + + if (io_resubmit_prep(req, err)) { + refcount_inc(&req->refs); + io_queue_async_work(req); + } +} +#endif + +static bool io_rw_reissue(struct io_kiocb *req, long res) +{ +#ifdef CONFIG_BLOCK + struct task_struct *tsk; + int ret; + + if ((res != -EAGAIN && res != -EOPNOTSUPP) || io_wq_current_is_worker()) + return false; + + tsk = req->task; + init_task_work(&req->task_work, io_rw_resubmit); + ret = task_work_add(tsk, &req->task_work, true); + if (!ret) + return true; +#endif + return false; +} + static void io_complete_rw(struct kiocb *kiocb, long res, long res2) { struct io_kiocb *req = container_of(kiocb, struct io_kiocb, rw.kiocb); - io_complete_rw_common(kiocb, res); - io_put_req(req); + if (!io_rw_reissue(req, res)) { + io_complete_rw_common(kiocb, res); + io_put_req(req); + } } static void io_complete_rw_iopoll(struct kiocb *kiocb, long res, long res2) @@ -2169,6 +2279,9 @@ static int io_prep_rw(struct io_kiocb *req, const struct io_uring_sqe *sqe, if (kiocb->ki_flags & IOCB_NOWAIT) req->flags |= REQ_F_NOWAIT; + if (kiocb->ki_flags & IOCB_DIRECT) + io_get_req_task(req); + if (force_nonblock) kiocb->ki_flags |= IOCB_NOWAIT; @@ -2668,6 +2781,7 @@ static int io_read(struct io_kiocb *req, bool force_nonblock) iov_count = iov_iter_count(&iter); ret = rw_verify_area(READ, req->file, &kiocb->ki_pos, iov_count); if (!ret) { + unsigned long nr_segs = iter.nr_segs; ssize_t ret2 = 0; if (req->file->f_op->read_iter) @@ -2679,6 +2793,8 @@ static int io_read(struct io_kiocb *req, bool force_nonblock) if (!force_nonblock || (ret2 != -EAGAIN && ret2 != -EIO)) { kiocb_done(kiocb, ret2); } else { + iter.count = iov_count; + iter.nr_segs = nr_segs; copy_iov: ret = io_setup_async_rw(req, io_size, iovec, inline_vecs, &iter); @@ -2765,6 +2881,7 @@ static int io_write(struct io_kiocb *req, bool force_nonblock) iov_count = iov_iter_count(&iter); ret = rw_verify_area(WRITE, req->file, &kiocb->ki_pos, iov_count); if (!ret) { + unsigned long nr_segs = iter.nr_segs; ssize_t ret2; /* @@ -2802,6 +2919,8 @@ static int io_write(struct io_kiocb *req, bool force_nonblock) if (!force_nonblock || ret2 != -EAGAIN) { kiocb_done(kiocb, ret2); } else { + iter.count = iov_count; + iter.nr_segs = nr_segs; copy_iov: ret = io_setup_async_rw(req, io_size, iovec, inline_vecs, &iter); @@ -4282,28 +4401,6 @@ static void io_async_queue_proc(struct file *file, struct wait_queue_head *head, __io_queue_proc(&pt->req->apoll->poll, pt, head); } -static void io_sq_thread_drop_mm(struct io_ring_ctx *ctx) -{ - struct mm_struct *mm = current->mm; - - if (mm) { - kthread_unuse_mm(mm); - mmput(mm); - } -} - -static int io_sq_thread_acquire_mm(struct io_ring_ctx *ctx, - struct io_kiocb *req) -{ - if (io_op_defs[req->opcode].needs_mm && !current->mm) { - if (unlikely(!mmget_not_zero(ctx->sqo_mm))) - return -EFAULT; - kthread_use_mm(ctx->sqo_mm); - } - - return 0; -} - static void io_async_task_func(struct callback_head *cb) { struct io_kiocb *req = container_of(cb, struct io_kiocb, task_work); @@ -5814,6 +5911,9 @@ static void io_submit_state_start(struct io_submit_state *state, unsigned int max_ios) { blk_start_plug(&state->plug); +#ifdef CONFIG_BLOCK + state->plug.nowait = true; +#endif state->free_reqs = 0; state->file = NULL; state->ios_left = max_ios; From patchwork Thu Jun 18 14:43:45 2020 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Jens Axboe X-Patchwork-Id: 11612327 Return-Path: Received: from mail.kernel.org (pdx-korg-mail-1.web.codeaurora.org [172.30.200.123]) by pdx-korg-patchwork-2.web.codeaurora.org (Postfix) with ESMTP id 1A13B159A for ; Thu, 18 Jun 2020 14:44:16 +0000 (UTC) Received: from kanga.kvack.org (kanga.kvack.org [205.233.56.17]) by mail.kernel.org (Postfix) with ESMTP id D219020897 for ; Thu, 18 Jun 2020 14:44:15 +0000 (UTC) Authentication-Results: mail.kernel.org; dkim=fail reason="signature verification failed" (2048-bit key) header.d=kernel-dk.20150623.gappssmtp.com header.i=@kernel-dk.20150623.gappssmtp.com header.b="TF887Y4c" DMARC-Filter: OpenDMARC Filter v1.3.2 mail.kernel.org D219020897 Authentication-Results: mail.kernel.org; dmarc=none (p=none dis=none) header.from=kernel.dk Authentication-Results: mail.kernel.org; spf=pass smtp.mailfrom=owner-linux-mm@kvack.org Received: by kanga.kvack.org (Postfix) id 56BBC8D001D; Thu, 18 Jun 2020 10:44:08 -0400 (EDT) Delivered-To: linux-mm-outgoing@kvack.org Received: by kanga.kvack.org (Postfix, from userid 40) id 4A4488D0018; Thu, 18 Jun 2020 10:44:08 -0400 (EDT) X-Original-To: int-list-linux-mm@kvack.org X-Delivered-To: int-list-linux-mm@kvack.org Received: by kanga.kvack.org (Postfix, from userid 63042) id 36A5B8D001D; Thu, 18 Jun 2020 10:44:08 -0400 (EDT) X-Original-To: linux-mm@kvack.org X-Delivered-To: linux-mm@kvack.org Received: from forelay.hostedemail.com (smtprelay0076.hostedemail.com [216.40.44.76]) by kanga.kvack.org (Postfix) with ESMTP id 09B3F8D0018 for ; Thu, 18 Jun 2020 10:44:08 -0400 (EDT) Received: from smtpin03.hostedemail.com (10.5.19.251.rfc1918.com [10.5.19.251]) by forelay04.hostedemail.com (Postfix) with ESMTP id C3F0B9F868 for ; Thu, 18 Jun 2020 14:44:07 +0000 (UTC) X-FDA: 76942602534.03.sofa90_230222326e11 Received: from filter.hostedemail.com (10.5.16.251.rfc1918.com [10.5.16.251]) by smtpin03.hostedemail.com (Postfix) with ESMTP id 9222828A4EC for ; Thu, 18 Jun 2020 14:44:07 +0000 (UTC) X-Spam-Summary: 2,0,0,87ee68db881394de,d41d8cd98f00b204,axboe@kernel.dk,,RULES_HIT:41:355:379:541:800:960:973:988:989:1260:1311:1314:1345:1359:1431:1437:1515:1534:1539:1568:1711:1714:1730:1747:1777:1792:2194:2198:2199:2200:2393:2559:2562:3138:3139:3140:3141:3142:3865:3867:3870:3871:3872:4321:5007:6261:6653:7875:10004:11026:11658:11914:12043:12296:12438:12517:12519:12555:12895:12986:13069:13161:13229:13311:13357:13894:14181:14384:14394:14721:21080:21324:21444:21451:21627:30054,0,RBL:209.85.216.65:@kernel.dk:.lbl8.mailshell.net-66.100.201.201 62.2.0.100,CacheIP:none,Bayesian:0.5,0.5,0.5,Netcheck:none,DomainCache:0,MSF:not bulk,SPF:fp,MSBL:0,DNSBL:neutral,Custom_rules:0:0:0,LFtime:24,LUA_SUMMARY:none X-HE-Tag: sofa90_230222326e11 X-Filterd-Recvd-Size: 3752 Received: from mail-pj1-f65.google.com (mail-pj1-f65.google.com [209.85.216.65]) by imf28.hostedemail.com (Postfix) with ESMTP for ; Thu, 18 Jun 2020 14:44:07 +0000 (UTC) Received: by mail-pj1-f65.google.com with SMTP id jz3so2649985pjb.0 for ; Thu, 18 Jun 2020 07:44:06 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=kernel-dk.20150623.gappssmtp.com; s=20150623; h=from:to:cc:subject:date:message-id:in-reply-to:references:reply-to :mime-version:content-transfer-encoding; bh=kQSBvSrh7q7DQwM3udW0mgxqiTGah8xT/7K+LzTiHfI=; b=TF887Y4cr//mvXhTI5c5VEIRfZElOkM2eJr4igs7tYzM3I8rEROoL5414jXEsJNQ1d 6jCCrNh5fQhPQ07otOWMKTLuwned62gpZzcQSnGz+AtcAsFIRf8kVJYMZrFYkSHJEstI EF+Ln2Q+pUkq3tXurup9X6YD2LWffkOldc8tPFPnSONqddrmM/DgTfaNjc16nR3q5d4t hPjBvp3o4ycBlrYr8QUDl2U7jqa+BlqD3TbE19T3/QnC95yw4ECXM/85GcpiBdM2DbqP 8OM2osLFz5ebGolhbCSR09DpxT7wJ8dhnp+kF1wWM32/wq3hn9RugfvhthZyfIx3RxdO JS+Q== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20161025; h=x-gm-message-state:from:to:cc:subject:date:message-id:in-reply-to :references:reply-to:mime-version:content-transfer-encoding; bh=kQSBvSrh7q7DQwM3udW0mgxqiTGah8xT/7K+LzTiHfI=; b=MUoDMyMzs7oNZAeaVyzgP5O84TPoWSIlPHfiHv00vTLUjn1TP8t/NwidOZZjn/ss5g VQJ/PW9kqoDcCYjS5RNQpFrJSEeLWSM7ZShtOTrmOgJhpv7YKTmn6A75PVjl5/vbRaR6 cnysdE/J9i1F525t2CGZULKCKSfiAvrcX6WvsQyyyghrdsGOPigP4mhrztuscO/Cdf9T wXe05mvVaSAcEQ3/xwsw7XoRexkGtxza4zBBzR9yifqeTHSbYYRL6EYiIUGzE8L7t/rM YWTJiKEXNCuQF2mguj2qutzlsRQrWYHQfalvGxxyTTX0CSyPIDgvo3N08vkPOaQxzIXu JqOA== X-Gm-Message-State: AOAM533eRFUX04rLcIljR6ZZztnDvvgZnJzG3v2DMJcVPAO8vIVFMwSz RTSRfP7n2qaDwhRHqUaQ2vkkjg== X-Google-Smtp-Source: ABdhPJxFiInSxSM7iPuZuYhM5oN9cIihl9Wh6hIowfIxeUrII9w3vL4Xc8G06h5ebNdfyunElMZGlg== X-Received: by 2002:a17:902:eed1:: with SMTP id h17mr4012771plb.172.1592491446303; Thu, 18 Jun 2020 07:44:06 -0700 (PDT) Received: from x1.localdomain ([65.144.74.34]) by smtp.gmail.com with ESMTPSA id g9sm3127197pfm.151.2020.06.18.07.44.05 (version=TLS1_3 cipher=TLS_AES_256_GCM_SHA384 bits=256/256); Thu, 18 Jun 2020 07:44:05 -0700 (PDT) From: Jens Axboe To: io-uring@vger.kernel.org Cc: linux-fsdevel@vger.kernel.org, linux-kernel@vger.kernel.org, linux-mm@kvack.org, akpm@linux-foundation.org, Jens Axboe , Johannes Weiner Subject: [PATCH 05/15] mm: allow read-ahead with IOCB_NOWAIT set Date: Thu, 18 Jun 2020 08:43:45 -0600 Message-Id: <20200618144355.17324-6-axboe@kernel.dk> X-Mailer: git-send-email 2.27.0 In-Reply-To: <20200618144355.17324-1-axboe@kernel.dk> References: <20200618144355.17324-1-axboe@kernel.dk> Reply-To: "[PATCHSET v7 0/15]"@smtpin03.hostedemail.com, Add@smtpin03.hostedemail.com, support@smtpin03.hostedemail.com, for@smtpin03.hostedemail.com, async@smtpin03.hostedemail.com, buffered@smtpin03.hostedemail.com, reads@smtpin03.hostedemail.com MIME-Version: 1.0 X-Rspamd-Queue-Id: 9222828A4EC X-Spamd-Result: default: False [0.00 / 100.00] X-Rspamd-Server: rspam04 X-Bogosity: Ham, tests=bogofilter, spamicity=0.000000, version=1.2.4 Sender: owner-linux-mm@kvack.org Precedence: bulk X-Loop: owner-majordomo@kvack.org List-ID: The read-ahead shouldn't block, so allow it to be done even if IOCB_NOWAIT is set in the kiocb. Acked-by: Johannes Weiner Signed-off-by: Jens Axboe --- mm/filemap.c | 2 -- 1 file changed, 2 deletions(-) diff --git a/mm/filemap.c b/mm/filemap.c index f0ae9a6308cb..3378d4fca883 100644 --- a/mm/filemap.c +++ b/mm/filemap.c @@ -2028,8 +2028,6 @@ ssize_t generic_file_buffered_read(struct kiocb *iocb, page = find_get_page(mapping, index); if (!page) { - if (iocb->ki_flags & IOCB_NOWAIT) - goto would_block; page_cache_sync_readahead(mapping, ra, filp, index, last_index - index); From patchwork Thu Jun 18 14:43:46 2020 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Jens Axboe X-Patchwork-Id: 11612329 Return-Path: Received: from mail.kernel.org (pdx-korg-mail-1.web.codeaurora.org [172.30.200.123]) by pdx-korg-patchwork-2.web.codeaurora.org (Postfix) with ESMTP id EB127913 for ; Thu, 18 Jun 2020 14:44:18 +0000 (UTC) Received: from kanga.kvack.org (kanga.kvack.org [205.233.56.17]) by mail.kernel.org (Postfix) with ESMTP id B80B320897 for ; Thu, 18 Jun 2020 14:44:18 +0000 (UTC) Authentication-Results: mail.kernel.org; dkim=fail reason="signature verification failed" (2048-bit key) header.d=kernel-dk.20150623.gappssmtp.com header.i=@kernel-dk.20150623.gappssmtp.com header.b="YeULpx9I" DMARC-Filter: OpenDMARC Filter v1.3.2 mail.kernel.org B80B320897 Authentication-Results: mail.kernel.org; dmarc=none (p=none dis=none) header.from=kernel.dk Authentication-Results: mail.kernel.org; spf=pass smtp.mailfrom=owner-linux-mm@kvack.org Received: by kanga.kvack.org (Postfix) id 9889F8D001E; Thu, 18 Jun 2020 10:44:09 -0400 (EDT) Delivered-To: linux-mm-outgoing@kvack.org Received: by kanga.kvack.org (Postfix, from userid 40) id 935C08D0018; Thu, 18 Jun 2020 10:44:09 -0400 (EDT) X-Original-To: int-list-linux-mm@kvack.org X-Delivered-To: int-list-linux-mm@kvack.org Received: by kanga.kvack.org (Postfix, from userid 63042) id 84A528D001E; Thu, 18 Jun 2020 10:44:09 -0400 (EDT) X-Original-To: linux-mm@kvack.org X-Delivered-To: linux-mm@kvack.org Received: from forelay.hostedemail.com (smtprelay0096.hostedemail.com [216.40.44.96]) by kanga.kvack.org (Postfix) with ESMTP id 682A28D0018 for ; Thu, 18 Jun 2020 10:44:09 -0400 (EDT) Received: from smtpin19.hostedemail.com (10.5.19.251.rfc1918.com [10.5.19.251]) by forelay02.hostedemail.com (Postfix) with ESMTP id 26E431495E1 for ; Thu, 18 Jun 2020 14:44:09 +0000 (UTC) X-FDA: 76942602618.19.feast50_3317cb626e11 Received: from filter.hostedemail.com (10.5.16.251.rfc1918.com [10.5.16.251]) by smtpin19.hostedemail.com (Postfix) with ESMTP id EDCCF1AD1B2 for ; Thu, 18 Jun 2020 14:44:08 +0000 (UTC) X-Spam-Summary: 2,0,0,915ae43395add51e,d41d8cd98f00b204,axboe@kernel.dk,,RULES_HIT:41:69:355:379:541:800:960:968:973:988:989:1260:1311:1314:1345:1359:1431:1437:1515:1535:1543:1711:1730:1747:1777:1792:2194:2199:2393:2559:2562:3138:3139:3140:3141:3142:3355:3865:3866:3867:3868:3870:3871:3872:3874:4117:4321:4605:5007:6261:6653:7903:9592:10004:11026:11658:11914:12043:12114:12291:12294:12296:12438:12517:12519:12555:12683:12895:13894:14096:14110:14181:14394:14721:21080:21212:21324:21444:21451:21627:21990:30054,0,RBL:209.85.214.196:@kernel.dk:.lbl8.mailshell.net-62.2.0.100 66.100.201.201,CacheIP:none,Bayesian:0.5,0.5,0.5,Netcheck:none,DomainCache:0,MSF:not bulk,SPF:fp,MSBL:0,DNSBL:neutral,Custom_rules:0:0:0,LFtime:24,LUA_SUMMARY:none X-HE-Tag: feast50_3317cb626e11 X-Filterd-Recvd-Size: 6414 Received: from mail-pl1-f196.google.com (mail-pl1-f196.google.com [209.85.214.196]) by imf20.hostedemail.com (Postfix) with ESMTP for ; Thu, 18 Jun 2020 14:44:08 +0000 (UTC) Received: by mail-pl1-f196.google.com with SMTP id y17so2516374plb.8 for ; Thu, 18 Jun 2020 07:44:08 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=kernel-dk.20150623.gappssmtp.com; s=20150623; h=from:to:cc:subject:date:message-id:in-reply-to:references:reply-to :mime-version:content-transfer-encoding; bh=L8y2vzO4eFciFziI46a/ttSbeGYaIdwkqPPuYZk9V/Y=; b=YeULpx9IMVYlet06hXXtF3Dyovfb6OWV7znBhPslocYv61JEKGOzA86Uf7KlkwQUxA 5veTkQm0U3FiHje/G2qBCvINtR+5H6+EjnG+6Sf5DBiwhwEgiK3ZySkgoWKeRlArsl6Z KII1sM40sgw0BBBEt/0RNUdVdIscLTYaAk11C3WKb8/EKEdqvKbfubfLIO/VBQ2Xx9WL CEYBtGHqlJYku+rfQdHayepPiSv6ME4oZxtjXE8I56CTLxFCqrzXuobRRnwbzTHobf+i uKAVIMjqM417ECasSBzSLduZSGdbDtTFAHLkWJ12coSXUTQ7pSi5y5D2Y4Jzync6tV9q xrBQ== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20161025; h=x-gm-message-state:from:to:cc:subject:date:message-id:in-reply-to :references:reply-to:mime-version:content-transfer-encoding; bh=L8y2vzO4eFciFziI46a/ttSbeGYaIdwkqPPuYZk9V/Y=; b=BHQKhkZrIzAe6uh93DDItATFnXgB2gs0SbVq9eDgk1ThEnJyuw+RMhBifuFHVBoUto 9Y8eKZve+atlByca69m4MhuXA6+RVFZKRHCOkBxGvURuNFEV2wcW5qoDPrJZwLThp+VU kSwH1q8MJ2OKgKCNs5GdrK2xl9/2Uz8EkYPunC4972imDhjYMNIAy6s/7cvAYzBtM9O6 hC1In8hJKgLhJqZJnlt3L2ej+UzmlzqKmIN/lZfs2898m3abvHsotmh1YXIchxvOyndm ogdvzkavlHTxOE57GMSvd1OCFZ4ettYpnA/l4xuxUOAkA7QSOrNEcViiq5nzWnMmf8cW MQWA== X-Gm-Message-State: AOAM532IyxtQT+/op3/JJzUmASDyslej5UMBtO6AXQXOd2w6etFgfQcc keS5wItKBkAb15gm9j16SRO7Ww== X-Google-Smtp-Source: ABdhPJwx9um0vmY0DnYUG9kEzc1v1xMH5bH+K7HVKhmpcv+yl+Hyx+jkxHpVYBs3czBGcr7qfQ+u8g== X-Received: by 2002:a17:902:8690:: with SMTP id g16mr4021455plo.257.1592491447688; Thu, 18 Jun 2020 07:44:07 -0700 (PDT) Received: from x1.localdomain ([65.144.74.34]) by smtp.gmail.com with ESMTPSA id g9sm3127197pfm.151.2020.06.18.07.44.06 (version=TLS1_3 cipher=TLS_AES_256_GCM_SHA384 bits=256/256); Thu, 18 Jun 2020 07:44:06 -0700 (PDT) From: Jens Axboe To: io-uring@vger.kernel.org Cc: linux-fsdevel@vger.kernel.org, linux-kernel@vger.kernel.org, linux-mm@kvack.org, akpm@linux-foundation.org, Jens Axboe , Johannes Weiner Subject: [PATCH 06/15] mm: abstract out wake_page_match() from wake_page_function() Date: Thu, 18 Jun 2020 08:43:46 -0600 Message-Id: <20200618144355.17324-7-axboe@kernel.dk> X-Mailer: git-send-email 2.27.0 In-Reply-To: <20200618144355.17324-1-axboe@kernel.dk> References: <20200618144355.17324-1-axboe@kernel.dk> Reply-To: "[PATCHSET v7 0/15]"@smtpin19.hostedemail.com, Add@smtpin19.hostedemail.com, support@smtpin19.hostedemail.com, for@smtpin19.hostedemail.com, async@smtpin19.hostedemail.com, buffered@smtpin19.hostedemail.com, reads@smtpin19.hostedemail.com MIME-Version: 1.0 X-Rspamd-Queue-Id: EDCCF1AD1B2 X-Spamd-Result: default: False [0.00 / 100.00] X-Rspamd-Server: rspam03 X-Bogosity: Ham, tests=bogofilter, spamicity=0.000000, version=1.2.4 Sender: owner-linux-mm@kvack.org Precedence: bulk X-Loop: owner-majordomo@kvack.org List-ID: No functional changes in this patch, just in preparation for allowing more callers. Acked-by: Johannes Weiner Signed-off-by: Jens Axboe --- include/linux/pagemap.h | 37 +++++++++++++++++++++++++++++++++++++ mm/filemap.c | 35 ++++------------------------------- 2 files changed, 41 insertions(+), 31 deletions(-) diff --git a/include/linux/pagemap.h b/include/linux/pagemap.h index cf2468da68e9..2f18221bb5c8 100644 --- a/include/linux/pagemap.h +++ b/include/linux/pagemap.h @@ -496,6 +496,43 @@ static inline pgoff_t linear_page_index(struct vm_area_struct *vma, return pgoff; } +/* This has the same layout as wait_bit_key - see fs/cachefiles/rdwr.c */ +struct wait_page_key { + struct page *page; + int bit_nr; + int page_match; +}; + +struct wait_page_queue { + struct page *page; + int bit_nr; + wait_queue_entry_t wait; +}; + +static inline int wake_page_match(struct wait_page_queue *wait_page, + struct wait_page_key *key) +{ + if (wait_page->page != key->page) + return 0; + key->page_match = 1; + + if (wait_page->bit_nr != key->bit_nr) + return 0; + + /* + * Stop walking if it's locked. + * Is this safe if put_and_wait_on_page_locked() is in use? + * Yes: the waker must hold a reference to this page, and if PG_locked + * has now already been set by another task, that task must also hold + * a reference to the *same usage* of this page; so there is no need + * to walk on to wake even the put_and_wait_on_page_locked() callers. + */ + if (test_bit(key->bit_nr, &key->page->flags)) + return -1; + + return 1; +} + extern void __lock_page(struct page *page); extern int __lock_page_killable(struct page *page); extern int __lock_page_or_retry(struct page *page, struct mm_struct *mm, diff --git a/mm/filemap.c b/mm/filemap.c index 3378d4fca883..c3175dbd8fba 100644 --- a/mm/filemap.c +++ b/mm/filemap.c @@ -987,43 +987,16 @@ void __init pagecache_init(void) page_writeback_init(); } -/* This has the same layout as wait_bit_key - see fs/cachefiles/rdwr.c */ -struct wait_page_key { - struct page *page; - int bit_nr; - int page_match; -}; - -struct wait_page_queue { - struct page *page; - int bit_nr; - wait_queue_entry_t wait; -}; - static int wake_page_function(wait_queue_entry_t *wait, unsigned mode, int sync, void *arg) { struct wait_page_key *key = arg; struct wait_page_queue *wait_page = container_of(wait, struct wait_page_queue, wait); + int ret; - if (wait_page->page != key->page) - return 0; - key->page_match = 1; - - if (wait_page->bit_nr != key->bit_nr) - return 0; - - /* - * Stop walking if it's locked. - * Is this safe if put_and_wait_on_page_locked() is in use? - * Yes: the waker must hold a reference to this page, and if PG_locked - * has now already been set by another task, that task must also hold - * a reference to the *same usage* of this page; so there is no need - * to walk on to wake even the put_and_wait_on_page_locked() callers. - */ - if (test_bit(key->bit_nr, &key->page->flags)) - return -1; - + ret = wake_page_match(wait_page, key); + if (ret != 1) + return ret; return autoremove_wake_function(wait, mode, sync, key); } From patchwork Thu Jun 18 14:43:47 2020 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Jens Axboe X-Patchwork-Id: 11612333 Return-Path: Received: from mail.kernel.org (pdx-korg-mail-1.web.codeaurora.org [172.30.200.123]) by pdx-korg-patchwork-2.web.codeaurora.org (Postfix) with ESMTP id C26EC14E3 for ; Thu, 18 Jun 2020 14:44:21 +0000 (UTC) Received: from kanga.kvack.org (kanga.kvack.org [205.233.56.17]) by mail.kernel.org (Postfix) with ESMTP id 822EC20890 for ; Thu, 18 Jun 2020 14:44:21 +0000 (UTC) Authentication-Results: mail.kernel.org; dkim=fail reason="signature verification failed" (2048-bit key) header.d=kernel-dk.20150623.gappssmtp.com header.i=@kernel-dk.20150623.gappssmtp.com header.b="nPiXsRi6" DMARC-Filter: OpenDMARC Filter v1.3.2 mail.kernel.org 822EC20890 Authentication-Results: mail.kernel.org; dmarc=none (p=none dis=none) header.from=kernel.dk Authentication-Results: mail.kernel.org; spf=pass smtp.mailfrom=owner-linux-mm@kvack.org Received: by kanga.kvack.org (Postfix) id 344E48D001F; Thu, 18 Jun 2020 10:44:11 -0400 (EDT) Delivered-To: linux-mm-outgoing@kvack.org Received: by kanga.kvack.org (Postfix, from userid 40) id 2A4EB8D0018; Thu, 18 Jun 2020 10:44:11 -0400 (EDT) X-Original-To: int-list-linux-mm@kvack.org X-Delivered-To: int-list-linux-mm@kvack.org Received: by kanga.kvack.org (Postfix, from userid 63042) id 11F2D8D001F; Thu, 18 Jun 2020 10:44:11 -0400 (EDT) X-Original-To: linux-mm@kvack.org X-Delivered-To: linux-mm@kvack.org Received: from forelay.hostedemail.com (smtprelay0067.hostedemail.com [216.40.44.67]) by kanga.kvack.org (Postfix) with ESMTP id EC83D8D0018 for ; Thu, 18 Jun 2020 10:44:10 -0400 (EDT) Received: from smtpin02.hostedemail.com (10.5.19.251.rfc1918.com [10.5.19.251]) by forelay03.hostedemail.com (Postfix) with ESMTP id A910C8248047 for ; Thu, 18 Jun 2020 14:44:10 +0000 (UTC) X-FDA: 76942602660.02.death26_5701cd026e11 Received: from filter.hostedemail.com (10.5.16.251.rfc1918.com [10.5.16.251]) by smtpin02.hostedemail.com (Postfix) with ESMTP id 8E3D99F84D for ; Thu, 18 Jun 2020 14:44:10 +0000 (UTC) X-Spam-Summary: 2,0,0,6a022a5122f1126f,d41d8cd98f00b204,axboe@kernel.dk,,RULES_HIT:2:41:355:379:541:800:960:968:973:988:989:1260:1311:1314:1345:1359:1431:1437:1515:1535:1605:1606:1730:1747:1777:1792:2194:2199:2393:2553:2559:2562:2914:3138:3139:3140:3141:3142:3865:3867:3868:3870:3871:3872:3874:4119:4250:4321:4605:5007:6261:6653:7875:8957:10004:11026:11473:11658:11914:12043:12291:12296:12438:12517:12519:12555:12683:12895:12986:13138:13231:13894:14096:14394:21067:21080:21324:21444:21451:21627:21796:21990:30036:30054:30090,0,RBL:209.85.210.194:@kernel.dk:.lbl8.mailshell.net-66.100.201.201 62.2.0.100,CacheIP:none,Bayesian:0.5,0.5,0.5,Netcheck:none,DomainCache:0,MSF:not bulk,SPF:fp,MSBL:0,DNSBL:neutral,Custom_rules:0:0:0,LFtime:24,LUA_SUMMARY:none X-HE-Tag: death26_5701cd026e11 X-Filterd-Recvd-Size: 8384 Received: from mail-pf1-f194.google.com (mail-pf1-f194.google.com [209.85.210.194]) by imf08.hostedemail.com (Postfix) with ESMTP for ; Thu, 18 Jun 2020 14:44:10 +0000 (UTC) Received: by mail-pf1-f194.google.com with SMTP id z63so2895972pfb.1 for ; Thu, 18 Jun 2020 07:44:09 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=kernel-dk.20150623.gappssmtp.com; s=20150623; h=from:to:cc:subject:date:message-id:in-reply-to:references:reply-to :mime-version:content-transfer-encoding; bh=ugefse5kKTqXQ4eIaIDMHVyZsS3OmAuYWslFQFrKGAg=; b=nPiXsRi6BRt5JXEn23LLT0VlfowWc9FFrkRNPpQKyLz9yJSlB/8/wAL82qAGNedSkF cFv9Mr/dmOS/jD0ExP7JdeFsVG0cpLkPOn+Mv46eMO76ZpXYm8G+eaVtyJaSY5qfTAid FGe99XxOvLrha9M1k68N9nt6NIVNYnuM6FbUXYkyHG4WXFzvEDR8h5yR+ORXQIzt6rOY nlAC2fElI7Ax5Ja40mz8CQ3+pneQvvbUW7ssoIRcg++V8szsXNlw7tYb2iFTM3zhNZ0R BV6S7vWTAo03tso3K4liAsMATX93QS+pb1jxyiHuCxXSSpR0IvQ9l4/kjEcYZ79DaA9F dzDQ== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20161025; h=x-gm-message-state:from:to:cc:subject:date:message-id:in-reply-to :references:reply-to:mime-version:content-transfer-encoding; bh=ugefse5kKTqXQ4eIaIDMHVyZsS3OmAuYWslFQFrKGAg=; b=qx08KQBj5I0aRVuWcf6/PI7En4zmfkccDSz1bec3nGGjRo6wEyPgstT0s+bxJnl7y9 OtmvxVhIo+FxHbpzXuBy3L2vwC5PTv5PV3F7ufjub1r0EGurgYMvJZBhOyZrYIYsAA04 jw9hjU1WaXwN+DcgrqIYweJ9IhkOf4M+q7vqGyNdiJQIWaFgYYszYieVfZGzPH/xdzPE Uj9T+gIqxUZJ5TU/r3cruSZxL5OtPcjaY4jOmV9NRVUMlEttIdFQxCpQkNmQ3cbr6g9S qgexN+cxdcUlT771MbXu5OKmqXxx5bPbWUjJv8ouYucyQfOPQtCRxHkl132qFqs9jCIw 46zA== X-Gm-Message-State: AOAM532EmE5bCYbpoccDCRUX/5Ud9q/B8KUFlFdBpskeydFMN/DmOXcT A6yRtZzlZHIxNbh5uKwKH+zJBg== X-Google-Smtp-Source: ABdhPJzwVrGYYhzPh7B/cl4K86rqVhqHG34HLw3paGpNlY/2JChA4DR4YVjWC/vOkk6B5Jf1Y+/RRQ== X-Received: by 2002:a63:d858:: with SMTP id k24mr3368619pgj.288.1592491448987; Thu, 18 Jun 2020 07:44:08 -0700 (PDT) Received: from x1.localdomain ([65.144.74.34]) by smtp.gmail.com with ESMTPSA id g9sm3127197pfm.151.2020.06.18.07.44.07 (version=TLS1_3 cipher=TLS_AES_256_GCM_SHA384 bits=256/256); Thu, 18 Jun 2020 07:44:08 -0700 (PDT) From: Jens Axboe To: io-uring@vger.kernel.org Cc: linux-fsdevel@vger.kernel.org, linux-kernel@vger.kernel.org, linux-mm@kvack.org, akpm@linux-foundation.org, Jens Axboe , Johannes Weiner Subject: [PATCH 07/15] mm: add support for async page locking Date: Thu, 18 Jun 2020 08:43:47 -0600 Message-Id: <20200618144355.17324-8-axboe@kernel.dk> X-Mailer: git-send-email 2.27.0 In-Reply-To: <20200618144355.17324-1-axboe@kernel.dk> References: <20200618144355.17324-1-axboe@kernel.dk> Reply-To: "[PATCHSET v7 0/15]"@smtpin02.hostedemail.com, Add@smtpin02.hostedemail.com, support@smtpin02.hostedemail.com, for@smtpin02.hostedemail.com, async@smtpin02.hostedemail.com, buffered@smtpin02.hostedemail.com, reads@smtpin02.hostedemail.com MIME-Version: 1.0 X-Rspamd-Queue-Id: 8E3D99F84D X-Spamd-Result: default: False [0.00 / 100.00] X-Rspamd-Server: rspam04 X-Bogosity: Ham, tests=bogofilter, spamicity=0.000000, version=1.2.4 Sender: owner-linux-mm@kvack.org Precedence: bulk X-Loop: owner-majordomo@kvack.org List-ID: Normally waiting for a page to become unlocked, or locking the page, requires waiting for IO to complete. Add support for lock_page_async() and wait_on_page_locked_async(), which are callback based instead. This allows a caller to get notified when a page becomes unlocked, rather than wait for it. We add a new iocb field, ki_waitq, to pass in the necessary data for this to happen. We can unionize this with ki_cookie, since that is only used for polled IO. Polled IO can never co-exist with async callbacks, as it is (by definition) polled completions. struct wait_page_key is made public, and we define struct wait_page_async as the interface between the caller and the core. Acked-by: Johannes Weiner Signed-off-by: Jens Axboe --- include/linux/fs.h | 7 ++++++- include/linux/pagemap.h | 17 ++++++++++++++++ mm/filemap.c | 45 ++++++++++++++++++++++++++++++++++++++++- 3 files changed, 67 insertions(+), 2 deletions(-) diff --git a/include/linux/fs.h b/include/linux/fs.h index 6c4ab4dc1cd7..6ac919b40596 100644 --- a/include/linux/fs.h +++ b/include/linux/fs.h @@ -315,6 +315,8 @@ enum rw_hint { #define IOCB_SYNC (1 << 5) #define IOCB_WRITE (1 << 6) #define IOCB_NOWAIT (1 << 7) +/* iocb->ki_waitq is valid */ +#define IOCB_WAITQ (1 << 8) struct kiocb { struct file *ki_filp; @@ -328,7 +330,10 @@ struct kiocb { int ki_flags; u16 ki_hint; u16 ki_ioprio; /* See linux/ioprio.h */ - unsigned int ki_cookie; /* for ->iopoll */ + union { + unsigned int ki_cookie; /* for ->iopoll */ + struct wait_page_queue *ki_waitq; /* for async buffered IO */ + }; randomized_struct_fields_end }; diff --git a/include/linux/pagemap.h b/include/linux/pagemap.h index 2f18221bb5c8..e053e1d9a4d7 100644 --- a/include/linux/pagemap.h +++ b/include/linux/pagemap.h @@ -535,6 +535,7 @@ static inline int wake_page_match(struct wait_page_queue *wait_page, extern void __lock_page(struct page *page); extern int __lock_page_killable(struct page *page); +extern int __lock_page_async(struct page *page, struct wait_page_queue *wait); extern int __lock_page_or_retry(struct page *page, struct mm_struct *mm, unsigned int flags); extern void unlock_page(struct page *page); @@ -571,6 +572,22 @@ static inline int lock_page_killable(struct page *page) return 0; } +/* + * lock_page_async - Lock the page, unless this would block. If the page + * is already locked, then queue a callback when the page becomes unlocked. + * This callback can then retry the operation. + * + * Returns 0 if the page is locked successfully, or -EIOCBQUEUED if the page + * was already locked and the callback defined in 'wait' was queued. + */ +static inline int lock_page_async(struct page *page, + struct wait_page_queue *wait) +{ + if (!trylock_page(page)) + return __lock_page_async(page, wait); + return 0; +} + /* * lock_page_or_retry - Lock the page, unless this would block and the * caller indicated that it can handle a retry. diff --git a/mm/filemap.c b/mm/filemap.c index c3175dbd8fba..e8aaf43bee9f 100644 --- a/mm/filemap.c +++ b/mm/filemap.c @@ -1180,6 +1180,36 @@ int wait_on_page_bit_killable(struct page *page, int bit_nr) } EXPORT_SYMBOL(wait_on_page_bit_killable); +static int __wait_on_page_locked_async(struct page *page, + struct wait_page_queue *wait, bool set) +{ + struct wait_queue_head *q = page_waitqueue(page); + int ret = 0; + + wait->page = page; + wait->bit_nr = PG_locked; + + spin_lock_irq(&q->lock); + __add_wait_queue_entry_tail(q, &wait->wait); + SetPageWaiters(page); + if (set) + ret = !trylock_page(page); + else + ret = PageLocked(page); + /* + * If we were succesful now, we know we're still on the + * waitqueue as we're still under the lock. This means it's + * safe to remove and return success, we know the callback + * isn't going to trigger. + */ + if (!ret) + __remove_wait_queue(q, &wait->wait); + else + ret = -EIOCBQUEUED; + spin_unlock_irq(&q->lock); + return ret; +} + /** * put_and_wait_on_page_locked - Drop a reference and wait for it to be unlocked * @page: The page to wait for. @@ -1342,6 +1372,11 @@ int __lock_page_killable(struct page *__page) } EXPORT_SYMBOL_GPL(__lock_page_killable); +int __lock_page_async(struct page *page, struct wait_page_queue *wait) +{ + return __wait_on_page_locked_async(page, wait, true); +} + /* * Return values: * 1 - page is locked; mmap_lock is still held. @@ -2131,6 +2166,11 @@ ssize_t generic_file_buffered_read(struct kiocb *iocb, } readpage: + if (iocb->ki_flags & IOCB_NOWAIT) { + unlock_page(page); + put_page(page); + goto would_block; + } /* * A previous I/O error may have been due to temporary * failures, eg. multipath errors. @@ -2150,7 +2190,10 @@ ssize_t generic_file_buffered_read(struct kiocb *iocb, } if (!PageUptodate(page)) { - error = lock_page_killable(page); + if (iocb->ki_flags & IOCB_WAITQ) + error = lock_page_async(page, iocb->ki_waitq); + else + error = lock_page_killable(page); if (unlikely(error)) goto readpage_error; if (!PageUptodate(page)) { From patchwork Thu Jun 18 14:43:48 2020 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Jens Axboe X-Patchwork-Id: 11612335 Return-Path: Received: from mail.kernel.org (pdx-korg-mail-1.web.codeaurora.org [172.30.200.123]) by pdx-korg-patchwork-2.web.codeaurora.org (Postfix) with ESMTP id 7A2DD913 for ; Thu, 18 Jun 2020 14:44:24 +0000 (UTC) Received: from kanga.kvack.org (kanga.kvack.org [205.233.56.17]) by mail.kernel.org (Postfix) with ESMTP id 4744520888 for ; Thu, 18 Jun 2020 14:44:24 +0000 (UTC) Authentication-Results: mail.kernel.org; dkim=fail reason="signature verification failed" (2048-bit key) header.d=kernel-dk.20150623.gappssmtp.com header.i=@kernel-dk.20150623.gappssmtp.com header.b="hm30VxEN" DMARC-Filter: OpenDMARC Filter v1.3.2 mail.kernel.org 4744520888 Authentication-Results: mail.kernel.org; dmarc=none (p=none dis=none) header.from=kernel.dk Authentication-Results: mail.kernel.org; spf=pass smtp.mailfrom=owner-linux-mm@kvack.org Received: by kanga.kvack.org (Postfix) id 6C5538D0020; Thu, 18 Jun 2020 10:44:12 -0400 (EDT) Delivered-To: linux-mm-outgoing@kvack.org Received: by kanga.kvack.org (Postfix, from userid 40) id 6A11E8D0018; Thu, 18 Jun 2020 10:44:12 -0400 (EDT) X-Original-To: int-list-linux-mm@kvack.org X-Delivered-To: int-list-linux-mm@kvack.org Received: by kanga.kvack.org (Postfix, from userid 63042) id 4EF528D0020; Thu, 18 Jun 2020 10:44:12 -0400 (EDT) X-Original-To: linux-mm@kvack.org X-Delivered-To: linux-mm@kvack.org Received: from forelay.hostedemail.com (smtprelay0080.hostedemail.com [216.40.44.80]) by kanga.kvack.org (Postfix) with ESMTP id 3564D8D0018 for ; Thu, 18 Jun 2020 10:44:12 -0400 (EDT) Received: from smtpin11.hostedemail.com (10.5.19.251.rfc1918.com [10.5.19.251]) by forelay05.hostedemail.com (Postfix) with ESMTP id F04C1181AC9BF for ; Thu, 18 Jun 2020 14:44:11 +0000 (UTC) X-FDA: 76942602702.11.leaf97_0d0704c26e11 Received: from filter.hostedemail.com (10.5.16.251.rfc1918.com [10.5.16.251]) by smtpin11.hostedemail.com (Postfix) with ESMTP id B2E03180F8B86 for ; Thu, 18 Jun 2020 14:44:11 +0000 (UTC) X-Spam-Summary: 2,0,0,8d0be7c9a9af935f,d41d8cd98f00b204,axboe@kernel.dk,,RULES_HIT:41:69:355:379:541:800:960:973:988:989:1260:1311:1314:1345:1359:1431:1437:1515:1535:1542:1711:1730:1747:1777:1792:2194:2198:2199:2200:2393:2559:2562:2689:2901:3138:3139:3140:3141:3142:3354:3865:3867:3868:3870:3871:3872:3874:4250:4321:5007:6119:6261:6653:7875:7903:8957:9592:10004:11026:11658:11914:12043:12114:12291:12296:12438:12517:12519:12555:12683:12895:13161:13229:13894:14096:14181:14394:14721:21080:21324:21444:21451:21627:21990:30054,0,RBL:209.85.215.195:@kernel.dk:.lbl8.mailshell.net-66.100.201.201 62.2.0.100,CacheIP:none,Bayesian:0.5,0.5,0.5,Netcheck:none,DomainCache:0,MSF:not bulk,SPF:fp,MSBL:0,DNSBL:neutral,Custom_rules:0:0:0,LFtime:23,LUA_SUMMARY:none X-HE-Tag: leaf97_0d0704c26e11 X-Filterd-Recvd-Size: 5905 Received: from mail-pg1-f195.google.com (mail-pg1-f195.google.com [209.85.215.195]) by imf45.hostedemail.com (Postfix) with ESMTP for ; Thu, 18 Jun 2020 14:44:11 +0000 (UTC) Received: by mail-pg1-f195.google.com with SMTP id v14so3016739pgl.1 for ; Thu, 18 Jun 2020 07:44:11 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=kernel-dk.20150623.gappssmtp.com; s=20150623; h=from:to:cc:subject:date:message-id:in-reply-to:references:reply-to :mime-version:content-transfer-encoding; bh=UWm+xgAJblrksR6J1tnnSoleBzj4zdmtzBmjubOyR4o=; b=hm30VxENfZId4iIBXYzmK7NVLqG1L452OFVsVzBNxCcZsqrbma8q5o0CvvJElY7+8p Vs3+SDv61eWDJjcgEcNeNVdRAryioKuqAY0VZ5KUIBPne3UmEh/fpO8LsZt7zLrRhbVF r+xLfZ3r0wJVLTx1JAQn8WOQ4kpy0vI/zExZIzz2SQwYuNPeL12OnD2ggCtzoLOv29Ic dmRtYj8kmpp8on0fqufKHzxL4rQ9UdLZGMuQ6WCXYxQiYiLwh6uMWp6HgQzG+k87uazV SyI82aOrSqK4C/Ej6hlCmtBJKK+kxn+LVz6hW8Too2R1mZqI9Blhbsn1DdgKctf5HYDC n8lA== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20161025; h=x-gm-message-state:from:to:cc:subject:date:message-id:in-reply-to :references:reply-to:mime-version:content-transfer-encoding; bh=UWm+xgAJblrksR6J1tnnSoleBzj4zdmtzBmjubOyR4o=; b=ucuxuIG1cNu6WpcRSKCY3eUFdT82qqrVpNk7tTjP0xyu5nvyretR4vqHUYzAt05H3H mKhu+ybNcNNW77XLX7nf++ZPLYTaH4wV7PsBtjRmeXxsJWm1RAYKAPn/X5DjaB2KYZFs uwND+PF2AxlR6y8HwFjvNP/ZlbhRJy38L+m9Sq67zfyEI9txCBXXV3qOSDglZqLvDu+s YEyJ4Ntn4hw4o4Qnm60bIKz5bhPEEx1/Mrp1RRVbRbUnWX1lPW7LgHLZ+Z9woXABasAF tgqdjsD3sXW99Y5rtRUFyMS+XAvEV1SEqWgf9iocpvpj3eUy3LoZIeW2XfPWoXLQaTaN J5sQ== X-Gm-Message-State: AOAM532zkeXzSOne4Snab7zDYmD0GKgQHeN0xHG9O591DhbV07lQCFEY Q9Nfsr3KxNzdTqdAPDU+winZhw== X-Google-Smtp-Source: ABdhPJw0BKfA7CBOC1lO1/wFQlqefNyxHA+f5iR+FuKvtS2xAVdcBGbcEewx+yzDvjoPb4DPsIu+/g== X-Received: by 2002:a63:931b:: with SMTP id b27mr3478644pge.135.1592491450319; Thu, 18 Jun 2020 07:44:10 -0700 (PDT) Received: from x1.localdomain ([65.144.74.34]) by smtp.gmail.com with ESMTPSA id g9sm3127197pfm.151.2020.06.18.07.44.09 (version=TLS1_3 cipher=TLS_AES_256_GCM_SHA384 bits=256/256); Thu, 18 Jun 2020 07:44:09 -0700 (PDT) From: Jens Axboe To: io-uring@vger.kernel.org Cc: linux-fsdevel@vger.kernel.org, linux-kernel@vger.kernel.org, linux-mm@kvack.org, akpm@linux-foundation.org, Jens Axboe , Johannes Weiner Subject: [PATCH 08/15] mm: support async buffered reads in generic_file_buffered_read() Date: Thu, 18 Jun 2020 08:43:48 -0600 Message-Id: <20200618144355.17324-9-axboe@kernel.dk> X-Mailer: git-send-email 2.27.0 In-Reply-To: <20200618144355.17324-1-axboe@kernel.dk> References: <20200618144355.17324-1-axboe@kernel.dk> Reply-To: "[PATCHSET v7 0/15]"@smtpin11.hostedemail.com, Add@smtpin11.hostedemail.com, support@smtpin11.hostedemail.com, for@smtpin11.hostedemail.com, async@smtpin11.hostedemail.com, buffered@smtpin11.hostedemail.com, reads@smtpin11.hostedemail.com MIME-Version: 1.0 X-Rspamd-Queue-Id: B2E03180F8B86 X-Spamd-Result: default: False [0.00 / 100.00] X-Rspamd-Server: rspam02 X-Bogosity: Ham, tests=bogofilter, spamicity=0.000000, version=1.2.4 Sender: owner-linux-mm@kvack.org Precedence: bulk X-Loop: owner-majordomo@kvack.org List-ID: Use the async page locking infrastructure, if IOCB_WAITQ is set in the passed in iocb. The caller must expect an -EIOCBQUEUED return value, which means that IO is started but not done yet. This is similar to how O_DIRECT signals the same operation. Once the callback is received by the caller for IO completion, the caller must retry the operation. Acked-by: Johannes Weiner Signed-off-by: Jens Axboe --- mm/filemap.c | 38 +++++++++++++++++++++++++++----------- 1 file changed, 27 insertions(+), 11 deletions(-) diff --git a/mm/filemap.c b/mm/filemap.c index e8aaf43bee9f..a5b1fa8f7ce4 100644 --- a/mm/filemap.c +++ b/mm/filemap.c @@ -1210,6 +1210,14 @@ static int __wait_on_page_locked_async(struct page *page, return ret; } +static int wait_on_page_locked_async(struct page *page, + struct wait_page_queue *wait) +{ + if (!PageLocked(page)) + return 0; + return __wait_on_page_locked_async(compound_head(page), wait, false); +} + /** * put_and_wait_on_page_locked - Drop a reference and wait for it to be unlocked * @page: The page to wait for. @@ -2049,17 +2057,25 @@ ssize_t generic_file_buffered_read(struct kiocb *iocb, index, last_index - index); } if (!PageUptodate(page)) { - if (iocb->ki_flags & IOCB_NOWAIT) { - put_page(page); - goto would_block; - } - /* * See comment in do_read_cache_page on why * wait_on_page_locked is used to avoid unnecessarily * serialisations and why it's safe. */ - error = wait_on_page_locked_killable(page); + if (iocb->ki_flags & IOCB_WAITQ) { + if (written) { + put_page(page); + goto out; + } + error = wait_on_page_locked_async(page, + iocb->ki_waitq); + } else { + if (iocb->ki_flags & IOCB_NOWAIT) { + put_page(page); + goto would_block; + } + error = wait_on_page_locked_killable(page); + } if (unlikely(error)) goto readpage_error; if (PageUptodate(page)) @@ -2147,7 +2163,10 @@ ssize_t generic_file_buffered_read(struct kiocb *iocb, page_not_up_to_date: /* Get exclusive access to the page ... */ - error = lock_page_killable(page); + if (iocb->ki_flags & IOCB_WAITQ) + error = lock_page_async(page, iocb->ki_waitq); + else + error = lock_page_killable(page); if (unlikely(error)) goto readpage_error; @@ -2190,10 +2209,7 @@ ssize_t generic_file_buffered_read(struct kiocb *iocb, } if (!PageUptodate(page)) { - if (iocb->ki_flags & IOCB_WAITQ) - error = lock_page_async(page, iocb->ki_waitq); - else - error = lock_page_killable(page); + error = lock_page_killable(page); if (unlikely(error)) goto readpage_error; if (!PageUptodate(page)) { From patchwork Thu Jun 18 14:43:49 2020 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Jens Axboe X-Patchwork-Id: 11612337 Return-Path: Received: from mail.kernel.org (pdx-korg-mail-1.web.codeaurora.org [172.30.200.123]) by pdx-korg-patchwork-2.web.codeaurora.org (Postfix) with ESMTP id C6FF4913 for ; Thu, 18 Jun 2020 14:44:26 +0000 (UTC) Received: from kanga.kvack.org (kanga.kvack.org [205.233.56.17]) by mail.kernel.org (Postfix) with ESMTP id 93A8520888 for ; Thu, 18 Jun 2020 14:44:26 +0000 (UTC) Authentication-Results: mail.kernel.org; dkim=fail reason="signature verification failed" (2048-bit key) header.d=kernel-dk.20150623.gappssmtp.com header.i=@kernel-dk.20150623.gappssmtp.com header.b="e7NOJWPg" DMARC-Filter: OpenDMARC Filter v1.3.2 mail.kernel.org 93A8520888 Authentication-Results: mail.kernel.org; dmarc=none (p=none dis=none) header.from=kernel.dk Authentication-Results: mail.kernel.org; spf=pass smtp.mailfrom=owner-linux-mm@kvack.org Received: by kanga.kvack.org (Postfix) id EB5798D0021; Thu, 18 Jun 2020 10:44:13 -0400 (EDT) Delivered-To: linux-mm-outgoing@kvack.org Received: by kanga.kvack.org (Postfix, from userid 40) id E66E18D0018; Thu, 18 Jun 2020 10:44:13 -0400 (EDT) X-Original-To: int-list-linux-mm@kvack.org X-Delivered-To: int-list-linux-mm@kvack.org Received: by kanga.kvack.org (Postfix, from userid 63042) id D06478D0021; Thu, 18 Jun 2020 10:44:13 -0400 (EDT) X-Original-To: linux-mm@kvack.org X-Delivered-To: linux-mm@kvack.org Received: from forelay.hostedemail.com (smtprelay0089.hostedemail.com [216.40.44.89]) by kanga.kvack.org (Postfix) with ESMTP id B3B7F8D0018 for ; Thu, 18 Jun 2020 10:44:13 -0400 (EDT) Received: from smtpin12.hostedemail.com (10.5.19.251.rfc1918.com [10.5.19.251]) by forelay04.hostedemail.com (Postfix) with ESMTP id 55A09FD4A0 for ; Thu, 18 Jun 2020 14:44:13 +0000 (UTC) X-FDA: 76942602786.12.point55_611305c26e11 Received: from filter.hostedemail.com (10.5.16.251.rfc1918.com [10.5.16.251]) by smtpin12.hostedemail.com (Postfix) with ESMTP id 13E491825F9EB for ; Thu, 18 Jun 2020 14:44:12 +0000 (UTC) X-Spam-Summary: 2,0,0,ea23097fabbc553f,d41d8cd98f00b204,axboe@kernel.dk,,RULES_HIT:41:355:379:541:800:960:973:988:989:1260:1311:1314:1345:1359:1431:1437:1515:1534:1540:1711:1714:1730:1747:1777:1792:2393:2559:2562:3138:3139:3140:3141:3142:3351:3622:3865:3866:3867:3868:3874:4321:5007:6261:6653:10004:11026:11473:11658:11914:12296:12438:12517:12519:12555:12895:12986:13069:13311:13357:13894:14096:14181:14384:14394:14721:21080:21444:21450:21451:21627:30054:30070,0,RBL:209.85.216.68:@kernel.dk:.lbl8.mailshell.net-66.100.201.201 62.2.0.100,CacheIP:none,Bayesian:0.5,0.5,0.5,Netcheck:none,DomainCache:0,MSF:not bulk,SPF:fp,MSBL:0,DNSBL:neutral,Custom_rules:0:0:0,LFtime:23,LUA_SUMMARY:none X-HE-Tag: point55_611305c26e11 X-Filterd-Recvd-Size: 3816 Received: from mail-pj1-f68.google.com (mail-pj1-f68.google.com [209.85.216.68]) by imf46.hostedemail.com (Postfix) with ESMTP for ; Thu, 18 Jun 2020 14:44:12 +0000 (UTC) Received: by mail-pj1-f68.google.com with SMTP id k2so2644845pjs.2 for ; Thu, 18 Jun 2020 07:44:12 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=kernel-dk.20150623.gappssmtp.com; s=20150623; h=from:to:cc:subject:date:message-id:in-reply-to:references:reply-to :mime-version:content-transfer-encoding; bh=SizPTgUtV3iXJFQF9neCUoo2HH9dTA+Tsyik+Qs6Rtw=; b=e7NOJWPgpDilY8hAgu/E3UR0koeaR+7iSH91nRdwNwiag9SfhIjXt+DofFu1kfxmld sonG2ndDZmPtvlzTnmdVx6QC5xI0mKU+4Aw8UB4nmkxFWzScSGoM7yga5MuFL/2PZJnh Pi2uLHZ7iG/+7QjsmbzNkUTAyheEYTR5mB2REZBiiFdN8gvdYfMgHGneVhhGnbnz7xqE NBqAPOV/8kOoplTYc3iFVcW/Ztt8ElL2AgGcFfzG4+02vqZh/PMroO9cQupkyyW6unos 0oclD5apnWHqq9iEooOIUY0aKLpEh6E4q1f5yO/zY9SvBTrL+8900x9yRlsIPiwznjZd LoCA== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20161025; h=x-gm-message-state:from:to:cc:subject:date:message-id:in-reply-to :references:reply-to:mime-version:content-transfer-encoding; bh=SizPTgUtV3iXJFQF9neCUoo2HH9dTA+Tsyik+Qs6Rtw=; b=BZraXfT5VTlnre07EkvyuzbnNmcJ6LpF2gvoqaUeXt0aUJhmCsVM5holTPylw4FqSz 5LNwYmwtKqwBlwNM2M4rvOfr3NgeqfTpDgHu/+G4GhWhthuG+ePSQkWB1hM10ulXmdhg 2iADOsGT7cDQYOb/IEma7hxQnz2lFPvwDt++mrBOUYtyj17QadhRkLoiWA6jRG6zqMs2 OY5jDBuOCRA8VUoPTSHgaZUg8UAveyIWONPHh2k1YIG8af3OGDBjwPdIJLdbnqCOyHEO h3KPfjWsui1NS7Zq49Z73E2djVjY7EhHN4XzFthGLtmDBTXXTzZK3SKLNKRiPPNdq8Ss EBmw== X-Gm-Message-State: AOAM531l9pk86BOyhEjuK/vMfRLth4T5K46S3iJZxOUCaM89xAnvrfcA C4TI8jtzCTFEZaPDeSzECEP3lQ== X-Google-Smtp-Source: ABdhPJytALdF6/WXpXXo/EALA/GzyJrmCsd6yIDilF8kXfMl19bKdv+RXhHgzV2NYKcaqUbkTdgmNQ== X-Received: by 2002:a17:90a:b013:: with SMTP id x19mr4762711pjq.229.1592491451503; Thu, 18 Jun 2020 07:44:11 -0700 (PDT) Received: from x1.localdomain ([65.144.74.34]) by smtp.gmail.com with ESMTPSA id g9sm3127197pfm.151.2020.06.18.07.44.10 (version=TLS1_3 cipher=TLS_AES_256_GCM_SHA384 bits=256/256); Thu, 18 Jun 2020 07:44:11 -0700 (PDT) From: Jens Axboe To: io-uring@vger.kernel.org Cc: linux-fsdevel@vger.kernel.org, linux-kernel@vger.kernel.org, linux-mm@kvack.org, akpm@linux-foundation.org, Jens Axboe Subject: [PATCH 09/15] fs: add FMODE_BUF_RASYNC Date: Thu, 18 Jun 2020 08:43:49 -0600 Message-Id: <20200618144355.17324-10-axboe@kernel.dk> X-Mailer: git-send-email 2.27.0 In-Reply-To: <20200618144355.17324-1-axboe@kernel.dk> References: <20200618144355.17324-1-axboe@kernel.dk> Reply-To: "[PATCHSET v7 0/15]"@smtpin12.hostedemail.com, Add@smtpin12.hostedemail.com, support@smtpin12.hostedemail.com, for@smtpin12.hostedemail.com, async@smtpin12.hostedemail.com, buffered@smtpin12.hostedemail.com, reads@smtpin12.hostedemail.com MIME-Version: 1.0 X-Rspamd-Queue-Id: 13E491825F9EB X-Spamd-Result: default: False [0.00 / 100.00] X-Rspamd-Server: rspam01 X-Bogosity: Ham, tests=bogofilter, spamicity=0.000000, version=1.2.4 Sender: owner-linux-mm@kvack.org Precedence: bulk X-Loop: owner-majordomo@kvack.org List-ID: If set, this indicates that the file system supports IOCB_WAITQ for buffered reads. Signed-off-by: Jens Axboe --- include/linux/fs.h | 3 +++ 1 file changed, 3 insertions(+) diff --git a/include/linux/fs.h b/include/linux/fs.h index 6ac919b40596..3f9de90e0266 100644 --- a/include/linux/fs.h +++ b/include/linux/fs.h @@ -175,6 +175,9 @@ typedef int (dio_iodone_t)(struct kiocb *iocb, loff_t offset, /* File does not contribute to nr_files count */ #define FMODE_NOACCOUNT ((__force fmode_t)0x20000000) +/* File supports async buffered reads */ +#define FMODE_BUF_RASYNC ((__force fmode_t)0x40000000) + /* * Flag for rw_copy_check_uvector and compat_rw_copy_check_uvector * that indicates that they should check the contents of the iovec are From patchwork Thu Jun 18 14:43:50 2020 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Jens Axboe X-Patchwork-Id: 11612339 Return-Path: Received: from mail.kernel.org (pdx-korg-mail-1.web.codeaurora.org [172.30.200.123]) by pdx-korg-patchwork-2.web.codeaurora.org (Postfix) with ESMTP id 84BC114E3 for ; Thu, 18 Jun 2020 14:44:29 +0000 (UTC) Received: from kanga.kvack.org (kanga.kvack.org [205.233.56.17]) by mail.kernel.org (Postfix) with ESMTP id 51B8320888 for ; Thu, 18 Jun 2020 14:44:29 +0000 (UTC) Authentication-Results: mail.kernel.org; dkim=fail reason="signature verification failed" (2048-bit key) header.d=kernel-dk.20150623.gappssmtp.com header.i=@kernel-dk.20150623.gappssmtp.com header.b="ynTJluJ6" DMARC-Filter: OpenDMARC Filter v1.3.2 mail.kernel.org 51B8320888 Authentication-Results: mail.kernel.org; dmarc=none (p=none dis=none) header.from=kernel.dk Authentication-Results: mail.kernel.org; spf=pass smtp.mailfrom=owner-linux-mm@kvack.org Received: by kanga.kvack.org (Postfix) id F2A348D0022; Thu, 18 Jun 2020 10:44:14 -0400 (EDT) Delivered-To: linux-mm-outgoing@kvack.org Received: by kanga.kvack.org (Postfix, from userid 40) id EE2D68D0018; Thu, 18 Jun 2020 10:44:14 -0400 (EDT) X-Original-To: int-list-linux-mm@kvack.org X-Delivered-To: int-list-linux-mm@kvack.org Received: by kanga.kvack.org (Postfix, from userid 63042) id D07CA8D0022; Thu, 18 Jun 2020 10:44:14 -0400 (EDT) X-Original-To: linux-mm@kvack.org X-Delivered-To: linux-mm@kvack.org Received: from forelay.hostedemail.com (smtprelay0227.hostedemail.com [216.40.44.227]) by kanga.kvack.org (Postfix) with ESMTP id AF8C18D0018 for ; Thu, 18 Jun 2020 10:44:14 -0400 (EDT) Received: from smtpin19.hostedemail.com (10.5.19.251.rfc1918.com [10.5.19.251]) by forelay01.hostedemail.com (Postfix) with ESMTP id 72B6F180AD807 for ; Thu, 18 Jun 2020 14:44:14 +0000 (UTC) X-FDA: 76942602828.19.month93_0d1517b26e11 Received: from filter.hostedemail.com (10.5.16.251.rfc1918.com [10.5.16.251]) by smtpin19.hostedemail.com (Postfix) with ESMTP id 2B6EE1AD1B0 for ; Thu, 18 Jun 2020 14:44:14 +0000 (UTC) X-Spam-Summary: 2,0,0,31200f0bded0b032,d41d8cd98f00b204,axboe@kernel.dk,,RULES_HIT:41:355:379:541:800:960:988:989:1260:1311:1314:1345:1359:1431:1515:1534:1538:1567:1711:1714:1730:1747:1777:1792:2393:2559:2562:3138:3139:3140:3141:3142:3876:3877:4321:5007:6114:6261:6642:6653:10004:11026:11473:11658:11914:12517:12519:12555:12895:12986:13069:13311:13357:13894:14181:14384:14394:14721:21080:21444:21627:30054,0,RBL:209.85.210.194:@kernel.dk:.lbl8.mailshell.net-62.2.0.100 66.100.201.201,CacheIP:none,Bayesian:0.5,0.5,0.5,Netcheck:none,DomainCache:0,MSF:not bulk,SPF:fp,MSBL:0,DNSBL:neutral,Custom_rules:0:0:0,LFtime:22,LUA_SUMMARY:none X-HE-Tag: month93_0d1517b26e11 X-Filterd-Recvd-Size: 3590 Received: from mail-pf1-f194.google.com (mail-pf1-f194.google.com [209.85.210.194]) by imf01.hostedemail.com (Postfix) with ESMTP for ; Thu, 18 Jun 2020 14:44:13 +0000 (UTC) Received: by mail-pf1-f194.google.com with SMTP id 64so2869770pfv.11 for ; Thu, 18 Jun 2020 07:44:13 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=kernel-dk.20150623.gappssmtp.com; s=20150623; h=from:to:cc:subject:date:message-id:in-reply-to:references:reply-to :mime-version:content-transfer-encoding; bh=lmn4EHxlosWk2LKyZRbh3uHZDbpZvbZJJ5zRlTFAsKo=; b=ynTJluJ6Q99yEvMTa6y10W+u5+Z89pdb0A+GuJkEPIQuhyxpm60zb8aV1UgTzM49Cg 7mYXB2z8mtKRwGfC7oa8zmXN0NeqZcMk0nfEZAsWCQC7qS69O7EfAy7bvV4ntFSDUnKd s1HYlH3UUvMSO05JqooeCBbZY0ZRIjrWWqeXC7QjivnX5Jt0UXn8Nku5Kkav004uevwG 8PUXkLfmKjS6G8DCuicClsmZmNXcIm7PbrbS7/OqZObQQHmkmS+ttktTCCcqchSw4mb3 huoY15kQX2Ms3nHgl6XAnHvSp6STkjvZO6hPs2x+GSzE549wVTux9mlLhmmuwEge7kyC Whmg== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20161025; h=x-gm-message-state:from:to:cc:subject:date:message-id:in-reply-to :references:reply-to:mime-version:content-transfer-encoding; bh=lmn4EHxlosWk2LKyZRbh3uHZDbpZvbZJJ5zRlTFAsKo=; b=LICf0uSViI7Dgz+dayVJB7rdzQY6wTvRFZqS5UPY+aUhLY+G3Y1XjaqHwp9/JrvieQ q08chl8YIk9X84A4ztY/6rAwmTDa0+z6Ex8GjZ348aj03J2dVMjCYD0FtCu0wXAE8xeA gnMc6p6sISzD/c1bbpUZCw9amoj02UR1UH2JeQ/RYzRnynHf4ivqb/tBn7ZiuWJBeZlV N5qsOSE2Q7B/AX0/8O8OzfOOvlogjPJjd6uhjslETdg8GecWbd4ZEOu4nhXekOz72+X8 2AOfs+LkQ1TmcszTCIZURqH+5moxfgIcLF7c+GnGNRlTXo4JZ019NvT+TdReTzXxoY6B xn6A== X-Gm-Message-State: AOAM530d0vAzSjeigk0EsyUnU+ueweHlrpgJhJVD1gistYjZCiAlw5P8 FhHSMV46+NN+c3LCDBWEe7KUDQ== X-Google-Smtp-Source: ABdhPJzBNjEb2EomZLOu+f8/v3tw76mKzCcpM8kJeYEd8Lb2xxqO66ZxbvFvynHQ19qXPAAGDtGMUA== X-Received: by 2002:a63:1e60:: with SMTP id p32mr3518283pgm.172.1592491452832; Thu, 18 Jun 2020 07:44:12 -0700 (PDT) Received: from x1.localdomain ([65.144.74.34]) by smtp.gmail.com with ESMTPSA id g9sm3127197pfm.151.2020.06.18.07.44.11 (version=TLS1_3 cipher=TLS_AES_256_GCM_SHA384 bits=256/256); Thu, 18 Jun 2020 07:44:12 -0700 (PDT) From: Jens Axboe To: io-uring@vger.kernel.org Cc: linux-fsdevel@vger.kernel.org, linux-kernel@vger.kernel.org, linux-mm@kvack.org, akpm@linux-foundation.org, Jens Axboe Subject: [PATCH 10/15] block: flag block devices as supporting IOCB_WAITQ Date: Thu, 18 Jun 2020 08:43:50 -0600 Message-Id: <20200618144355.17324-11-axboe@kernel.dk> X-Mailer: git-send-email 2.27.0 In-Reply-To: <20200618144355.17324-1-axboe@kernel.dk> References: <20200618144355.17324-1-axboe@kernel.dk> Reply-To: "[PATCHSET v7 0/15]"@smtpin19.hostedemail.com, Add@smtpin19.hostedemail.com, support@smtpin19.hostedemail.com, for@smtpin19.hostedemail.com, async@smtpin19.hostedemail.com, buffered@smtpin19.hostedemail.com, reads@smtpin19.hostedemail.com MIME-Version: 1.0 X-Rspamd-Queue-Id: 2B6EE1AD1B0 X-Spamd-Result: default: False [0.00 / 100.00] X-Rspamd-Server: rspam02 X-Bogosity: Ham, tests=bogofilter, spamicity=0.000000, version=1.2.4 Sender: owner-linux-mm@kvack.org Precedence: bulk X-Loop: owner-majordomo@kvack.org List-ID: Signed-off-by: Jens Axboe --- fs/block_dev.c | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/fs/block_dev.c b/fs/block_dev.c index 47860e589388..54720c90dad0 100644 --- a/fs/block_dev.c +++ b/fs/block_dev.c @@ -1850,7 +1850,7 @@ static int blkdev_open(struct inode * inode, struct file * filp) */ filp->f_flags |= O_LARGEFILE; - filp->f_mode |= FMODE_NOWAIT; + filp->f_mode |= FMODE_NOWAIT | FMODE_BUF_RASYNC; if (filp->f_flags & O_NDELAY) filp->f_mode |= FMODE_NDELAY; From patchwork Thu Jun 18 14:43:51 2020 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Jens Axboe X-Patchwork-Id: 11612343 Return-Path: Received: from mail.kernel.org (pdx-korg-mail-1.web.codeaurora.org [172.30.200.123]) by pdx-korg-patchwork-2.web.codeaurora.org (Postfix) with ESMTP id EDD1C159A for ; Thu, 18 Jun 2020 14:44:31 +0000 (UTC) Received: from kanga.kvack.org (kanga.kvack.org [205.233.56.17]) by mail.kernel.org (Postfix) with ESMTP id BA66C20888 for ; Thu, 18 Jun 2020 14:44:31 +0000 (UTC) Authentication-Results: mail.kernel.org; dkim=fail reason="signature verification failed" (2048-bit key) header.d=kernel-dk.20150623.gappssmtp.com header.i=@kernel-dk.20150623.gappssmtp.com header.b="HvZc/ixP" DMARC-Filter: OpenDMARC Filter v1.3.2 mail.kernel.org BA66C20888 Authentication-Results: mail.kernel.org; dmarc=none (p=none dis=none) header.from=kernel.dk Authentication-Results: mail.kernel.org; spf=pass smtp.mailfrom=owner-linux-mm@kvack.org Received: by kanga.kvack.org (Postfix) id 567ED8D0023; Thu, 18 Jun 2020 10:44:16 -0400 (EDT) Delivered-To: linux-mm-outgoing@kvack.org Received: by kanga.kvack.org (Postfix, from userid 40) id 4F09B8D0018; Thu, 18 Jun 2020 10:44:16 -0400 (EDT) X-Original-To: int-list-linux-mm@kvack.org X-Delivered-To: int-list-linux-mm@kvack.org Received: by kanga.kvack.org (Postfix, from userid 63042) id 3B8738D0023; Thu, 18 Jun 2020 10:44:16 -0400 (EDT) X-Original-To: linux-mm@kvack.org X-Delivered-To: linux-mm@kvack.org Received: from forelay.hostedemail.com (smtprelay0177.hostedemail.com [216.40.44.177]) by kanga.kvack.org (Postfix) with ESMTP id 1A03B8D0018 for ; Thu, 18 Jun 2020 10:44:16 -0400 (EDT) Received: from smtpin02.hostedemail.com (10.5.19.251.rfc1918.com [10.5.19.251]) by forelay03.hostedemail.com (Postfix) with ESMTP id D22888248047 for ; Thu, 18 Jun 2020 14:44:15 +0000 (UTC) X-FDA: 76942602870.02.cat76_04037e826e11 Received: from filter.hostedemail.com (10.5.16.251.rfc1918.com [10.5.16.251]) by smtpin02.hostedemail.com (Postfix) with ESMTP id 92BEE149603 for ; Thu, 18 Jun 2020 14:44:15 +0000 (UTC) X-Spam-Summary: 10,1,0,0c3d61f682791087,d41d8cd98f00b204,axboe@kernel.dk,,RULES_HIT:41:355:379:541:800:960:973:988:989:1260:1311:1314:1345:1359:1431:1515:1534:1539:1567:1711:1714:1730:1747:1777:1792:2194:2199:2393:2559:2562:3138:3139:3140:3141:3142:3876:3877:4321:5007:6114:6119:6261:6642:6653:10004:11026:11473:11658:11914:12043:12438:12517:12519:12555:12895:13069:13311:13357:13523:13524:13894:14181:14384:14394:14721:21080:21444:21451:21627:21990:30054:30064,0,RBL:209.85.215.196:@kernel.dk:.lbl8.mailshell.net-66.100.201.201 62.2.0.100,CacheIP:none,Bayesian:0.5,0.5,0.5,Netcheck:none,DomainCache:0,MSF:not bulk,SPF:fp,MSBL:0,DNSBL:neutral,Custom_rules:0:0:0,LFtime:24,LUA_SUMMARY:none X-HE-Tag: cat76_04037e826e11 X-Filterd-Recvd-Size: 3723 Received: from mail-pg1-f196.google.com (mail-pg1-f196.google.com [209.85.215.196]) by imf43.hostedemail.com (Postfix) with ESMTP for ; Thu, 18 Jun 2020 14:44:14 +0000 (UTC) Received: by mail-pg1-f196.google.com with SMTP id l63so2983314pge.12 for ; Thu, 18 Jun 2020 07:44:14 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=kernel-dk.20150623.gappssmtp.com; s=20150623; h=from:to:cc:subject:date:message-id:in-reply-to:references:reply-to :mime-version:content-transfer-encoding; bh=KEZHLow75x1WGZauUbfw6ciLmnTzXmpQXtEtlinmygw=; b=HvZc/ixPovm/kRhvl4SJUiwM/PVM+gQUGpw4gn7GL1Ek48GAAcFBxFJBO4YBWLb0Y9 Oiqk0i8f6vQ1Em168554rqM+AcEyk7rObxLac2g8tH1Pd6I83C8/2mPcrM6EC5HB+yuI ZQq+xXWBQ17rDR3i6if4Yr3L8h/fWDRW9Sjj870yRZqr2dNo7/szXK+hje9neRqzDGj1 k+P4aQd8v6FJFSOymCFraV22X6dCtjZ03CNMccoXsgYKxxTrdgIEeGkBWmuwX9eMXQVf 6oKU1HJcojpsQ/gOVihItBajLbiifqA4r48bx/wgZ3Vs2f3Le4J1aEEGs7KW+RdevsH2 /nIA== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20161025; h=x-gm-message-state:from:to:cc:subject:date:message-id:in-reply-to :references:reply-to:mime-version:content-transfer-encoding; bh=KEZHLow75x1WGZauUbfw6ciLmnTzXmpQXtEtlinmygw=; b=iQaoyqejFCJspjHhAM1oPumRKI+AnhheBjtFVVz+3bhpGlXjXJhj+ZVCkQ9cx953Zv z95ZzwvDLyg0nGVac/6CmTfq6qiJz/NB6RmpfMeh4DlonrfVwR0C+DhF+lng49ZI5ivi tokfm3BBLlb/tFzjuJ0WOi6riAzJKeyoeNH4naExOQhKRh8BbLqx5s7o9apLzF9Fq5cs W51w0kNvqNpNIhwSVycnh8oSngW4Dp5cQXmj2fms+PSf3XE9GMwChWQTbaHvdmwDd27Z GsEvPkV+lM6sdo8yX/NxNQMBhgj2LQKl8gScD/j+riqmK1Gibor8Mz6ezNsr4c6HUMpa 1R1w== X-Gm-Message-State: AOAM531UCamcwN1O2ELt0kTPddwM0gDCI2ZghwHjijHx8kA5CTKqBz/F a2VGdogw4oLF+Yw1F2+dZ0TFPPPuukN95A== X-Google-Smtp-Source: ABdhPJybJIEFYk2ErRp3fvA+q2tyQLM9BD6LGpYjoii03ztTUTqqX4O2JwO+0Ij8wpxOd+bSneCDiQ== X-Received: by 2002:a63:205b:: with SMTP id r27mr3555018pgm.326.1592491454092; Thu, 18 Jun 2020 07:44:14 -0700 (PDT) Received: from x1.localdomain ([65.144.74.34]) by smtp.gmail.com with ESMTPSA id g9sm3127197pfm.151.2020.06.18.07.44.13 (version=TLS1_3 cipher=TLS_AES_256_GCM_SHA384 bits=256/256); Thu, 18 Jun 2020 07:44:13 -0700 (PDT) From: Jens Axboe To: io-uring@vger.kernel.org Cc: linux-fsdevel@vger.kernel.org, linux-kernel@vger.kernel.org, linux-mm@kvack.org, akpm@linux-foundation.org, Jens Axboe , "Darrick J . Wong" Subject: [PATCH 11/15] xfs: flag files as supporting buffered async reads Date: Thu, 18 Jun 2020 08:43:51 -0600 Message-Id: <20200618144355.17324-12-axboe@kernel.dk> X-Mailer: git-send-email 2.27.0 In-Reply-To: <20200618144355.17324-1-axboe@kernel.dk> References: <20200618144355.17324-1-axboe@kernel.dk> Reply-To: "[PATCHSET v7 0/15]"@smtpin02.hostedemail.com, Add@smtpin02.hostedemail.com, support@smtpin02.hostedemail.com, for@smtpin02.hostedemail.com, async@smtpin02.hostedemail.com, buffered@smtpin02.hostedemail.com, reads@smtpin02.hostedemail.com MIME-Version: 1.0 X-Rspamd-Queue-Id: 92BEE149603 X-Spamd-Result: default: False [0.00 / 100.00] X-Rspamd-Server: rspam02 X-Bogosity: Ham, tests=bogofilter, spamicity=0.000000, version=1.2.4 Sender: owner-linux-mm@kvack.org Precedence: bulk X-Loop: owner-majordomo@kvack.org List-ID: XFS uses generic_file_read_iter(), which already supports this. Acked-by: Darrick J. Wong Signed-off-by: Jens Axboe --- fs/xfs/xfs_file.c | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/fs/xfs/xfs_file.c b/fs/xfs/xfs_file.c index 00db81eac80d..fdbff4860d61 100644 --- a/fs/xfs/xfs_file.c +++ b/fs/xfs/xfs_file.c @@ -1080,7 +1080,7 @@ xfs_file_open( return -EFBIG; if (XFS_FORCED_SHUTDOWN(XFS_M(inode->i_sb))) return -EIO; - file->f_mode |= FMODE_NOWAIT; + file->f_mode |= FMODE_NOWAIT | FMODE_BUF_RASYNC; return 0; } From patchwork Thu Jun 18 14:43:52 2020 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Jens Axboe X-Patchwork-Id: 11612345 Return-Path: Received: from mail.kernel.org (pdx-korg-mail-1.web.codeaurora.org [172.30.200.123]) by pdx-korg-patchwork-2.web.codeaurora.org (Postfix) with ESMTP id 22ADB14E3 for ; Thu, 18 Jun 2020 14:44:35 +0000 (UTC) Received: from kanga.kvack.org (kanga.kvack.org [205.233.56.17]) by mail.kernel.org (Postfix) with ESMTP id D89B520888 for ; Thu, 18 Jun 2020 14:44:34 +0000 (UTC) Authentication-Results: mail.kernel.org; dkim=fail reason="signature verification failed" (2048-bit key) header.d=kernel-dk.20150623.gappssmtp.com header.i=@kernel-dk.20150623.gappssmtp.com header.b="JsQWJAr0" DMARC-Filter: OpenDMARC Filter v1.3.2 mail.kernel.org D89B520888 Authentication-Results: mail.kernel.org; dmarc=none (p=none dis=none) header.from=kernel.dk Authentication-Results: mail.kernel.org; spf=pass smtp.mailfrom=owner-linux-mm@kvack.org Received: by kanga.kvack.org (Postfix) id 678588D0024; Thu, 18 Jun 2020 10:44:18 -0400 (EDT) Delivered-To: linux-mm-outgoing@kvack.org Received: by kanga.kvack.org (Postfix, from userid 40) id 62AB88D0018; Thu, 18 Jun 2020 10:44:18 -0400 (EDT) X-Original-To: int-list-linux-mm@kvack.org X-Delivered-To: int-list-linux-mm@kvack.org Received: by kanga.kvack.org (Postfix, from userid 63042) id 517288D0024; Thu, 18 Jun 2020 10:44:18 -0400 (EDT) X-Original-To: linux-mm@kvack.org X-Delivered-To: linux-mm@kvack.org Received: from forelay.hostedemail.com (smtprelay0099.hostedemail.com [216.40.44.99]) by kanga.kvack.org (Postfix) with ESMTP id 344728D0018 for ; Thu, 18 Jun 2020 10:44:18 -0400 (EDT) Received: from smtpin04.hostedemail.com (10.5.19.251.rfc1918.com [10.5.19.251]) by forelay02.hostedemail.com (Postfix) with ESMTP id E2B91AA25D for ; Thu, 18 Jun 2020 14:44:17 +0000 (UTC) X-FDA: 76942602954.04.drink10_48169ec26e11 Received: from filter.hostedemail.com (10.5.16.251.rfc1918.com [10.5.16.251]) by smtpin04.hostedemail.com (Postfix) with ESMTP id B369680A5DEA for ; Thu, 18 Jun 2020 14:44:17 +0000 (UTC) X-Spam-Summary: 2,0,0,a6dc5c12b1bf3ba5,d41d8cd98f00b204,axboe@kernel.dk,,RULES_HIT:41:355:379:541:800:960:973:988:989:1260:1311:1314:1345:1359:1431:1515:1534:1539:1568:1711:1714:1730:1747:1777:1792:2194:2199:2393:2559:2562:3138:3139:3140:3141:3142:3876:3877:5007:6114:6261:6642:6653:10004:11026:11473:11658:11914:12438:12517:12519:12555:12895:13069:13311:13357:13894:14181:14384:14394:14721:21080:21444:21627:21990:30054:30064,0,RBL:209.85.215.195:@kernel.dk:.lbl8.mailshell.net-66.100.201.201 62.2.0.100,CacheIP:none,Bayesian:0.5,0.5,0.5,Netcheck:none,DomainCache:0,MSF:not bulk,SPF:fp,MSBL:0,DNSBL:neutral,Custom_rules:0:0:0,LFtime:24,LUA_SUMMARY:none X-HE-Tag: drink10_48169ec26e11 X-Filterd-Recvd-Size: 3754 Received: from mail-pg1-f195.google.com (mail-pg1-f195.google.com [209.85.215.195]) by imf31.hostedemail.com (Postfix) with ESMTP for ; Thu, 18 Jun 2020 14:44:17 +0000 (UTC) Received: by mail-pg1-f195.google.com with SMTP id l24so3005635pgb.5 for ; Thu, 18 Jun 2020 07:44:17 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=kernel-dk.20150623.gappssmtp.com; s=20150623; h=from:to:cc:subject:date:message-id:in-reply-to:references:reply-to :mime-version:content-transfer-encoding; bh=qn8Ub7OYxVXKid0SneeEv6VlK8roF4pAhw3PBRQGk58=; b=JsQWJAr0eX5i4/3VH0JUUcbrX1hvyikPQulQivfaSDiMps3VzdLvSQq6dC4JKqWrJw KNxWMk6jyBDtv6zAbNsi8EXWu/yzVZs9PhGYrbDhl/fbSE8G4wRID9uPFVW7kqxk71SV 0TMZ2olNJApcg9LZpxFyKjUmCe4HMfr+UszmCR8CNyND2jnymuGfLJggvQZrlKsGiP5H olUIpqHBAvyxiNTFfvQ13/71tmHFNQyLLqdFLI0ERuAdJTxyxlimD9W8/4OLmmmYJWbC NDlNdD05mAC1WxkMhISVU5ejMHN0Tp6Wbi9rsSKqKY3u8ZOVaCzS4KBi9ffc4A0l9I/a Yiuw== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20161025; h=x-gm-message-state:from:to:cc:subject:date:message-id:in-reply-to :references:reply-to:mime-version:content-transfer-encoding; bh=qn8Ub7OYxVXKid0SneeEv6VlK8roF4pAhw3PBRQGk58=; b=U1apirY2GuXfrXctGYwqcp3fzXm8G4Q6yXtykbkhu05daLT7e81OkQLqSxr2u9G9f/ ADHa7zYaU1/pD6LyMFEFn511dJ5SNTinUJVrLKNaXWHPuxuEOlc83T4ocSo18GSfxwVu vJvFB7tvNXGsMwYVO/PkVcsEEdzb4z3eH83v+DcCz26DDu9eEx67a1v+C10rnoKqNt8L g5hn+iaZrsMX/21WsQwApZcMxVbh1Sw+grSO6l4+okAv1JKfK1kZuetEzsknNhGeg0eX MUB2EttaM8LgQvYYIBKWzmw+O/ZINh8umlim/giwFpmFspUmlbEh1IidO+p0j8D9hk+O Ugmw== X-Gm-Message-State: AOAM5323RsekcOeQVcR4CLumx8PeJZtwg2cFuke6xuML7ymouQT/9Ve/ SjCS4vzXNs/Jz5ISWUczK6/rZA== X-Google-Smtp-Source: ABdhPJxqRDix9IaoTIWmuLrGVEzpKy6hFma+ZgZz9Se6n9okXrRAwv8gfsXZq8bmQZ3+Gotx4PSctg== X-Received: by 2002:a65:6119:: with SMTP id z25mr3404328pgu.52.1592491456371; Thu, 18 Jun 2020 07:44:16 -0700 (PDT) Received: from x1.localdomain ([65.144.74.34]) by smtp.gmail.com with ESMTPSA id g9sm3127197pfm.151.2020.06.18.07.44.14 (version=TLS1_3 cipher=TLS_AES_256_GCM_SHA384 bits=256/256); Thu, 18 Jun 2020 07:44:14 -0700 (PDT) From: Jens Axboe To: io-uring@vger.kernel.org Cc: linux-fsdevel@vger.kernel.org, linux-kernel@vger.kernel.org, linux-mm@kvack.org, akpm@linux-foundation.org, Jens Axboe , Chris Mason Subject: [PATCH 12/15] btrfs: flag files as supporting buffered async reads Date: Thu, 18 Jun 2020 08:43:52 -0600 Message-Id: <20200618144355.17324-13-axboe@kernel.dk> X-Mailer: git-send-email 2.27.0 In-Reply-To: <20200618144355.17324-1-axboe@kernel.dk> References: <20200618144355.17324-1-axboe@kernel.dk> Reply-To: "[PATCHSET v7 0/15]"@smtpin04.hostedemail.com, Add@smtpin04.hostedemail.com, support@smtpin04.hostedemail.com, for@smtpin04.hostedemail.com, async@smtpin04.hostedemail.com, buffered@smtpin04.hostedemail.com, reads@smtpin04.hostedemail.com MIME-Version: 1.0 X-Rspamd-Queue-Id: B369680A5DEA X-Spamd-Result: default: False [0.00 / 100.00] X-Rspamd-Server: rspam01 X-Bogosity: Ham, tests=bogofilter, spamicity=0.000000, version=1.2.4 Sender: owner-linux-mm@kvack.org Precedence: bulk X-Loop: owner-majordomo@kvack.org List-ID: btrfs uses generic_file_read_iter(), which already supports this. Acked-by: Chris Mason Signed-off-by: Jens Axboe --- fs/btrfs/file.c | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/fs/btrfs/file.c b/fs/btrfs/file.c index 2c14312b05e8..234a418eb6da 100644 --- a/fs/btrfs/file.c +++ b/fs/btrfs/file.c @@ -3472,7 +3472,7 @@ static loff_t btrfs_file_llseek(struct file *file, loff_t offset, int whence) static int btrfs_file_open(struct inode *inode, struct file *filp) { - filp->f_mode |= FMODE_NOWAIT; + filp->f_mode |= FMODE_NOWAIT | FMODE_BUF_RASYNC; return generic_file_open(inode, filp); } From patchwork Thu Jun 18 14:43:53 2020 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Jens Axboe X-Patchwork-Id: 11612347 Return-Path: Received: from mail.kernel.org (pdx-korg-mail-1.web.codeaurora.org [172.30.200.123]) by pdx-korg-patchwork-2.web.codeaurora.org (Postfix) with ESMTP id 181A8913 for ; Thu, 18 Jun 2020 14:44:38 +0000 (UTC) Received: from kanga.kvack.org (kanga.kvack.org [205.233.56.17]) by mail.kernel.org (Postfix) with ESMTP id D84FF20888 for ; Thu, 18 Jun 2020 14:44:37 +0000 (UTC) Authentication-Results: mail.kernel.org; dkim=fail reason="signature verification failed" (2048-bit key) header.d=kernel-dk.20150623.gappssmtp.com header.i=@kernel-dk.20150623.gappssmtp.com header.b="CmsR4pnk" DMARC-Filter: OpenDMARC Filter v1.3.2 mail.kernel.org D84FF20888 Authentication-Results: mail.kernel.org; dmarc=none (p=none dis=none) header.from=kernel.dk Authentication-Results: mail.kernel.org; spf=pass smtp.mailfrom=owner-linux-mm@kvack.org Received: by kanga.kvack.org (Postfix) id 95F2C8D0025; Thu, 18 Jun 2020 10:44:19 -0400 (EDT) Delivered-To: linux-mm-outgoing@kvack.org Received: by kanga.kvack.org (Postfix, from userid 40) id 90FE28D0018; Thu, 18 Jun 2020 10:44:19 -0400 (EDT) X-Original-To: int-list-linux-mm@kvack.org X-Delivered-To: int-list-linux-mm@kvack.org Received: by kanga.kvack.org (Postfix, from userid 63042) id 7AEC78D0025; Thu, 18 Jun 2020 10:44:19 -0400 (EDT) X-Original-To: linux-mm@kvack.org X-Delivered-To: linux-mm@kvack.org Received: from forelay.hostedemail.com (smtprelay0176.hostedemail.com [216.40.44.176]) by kanga.kvack.org (Postfix) with ESMTP id 5DB438D0018 for ; Thu, 18 Jun 2020 10:44:19 -0400 (EDT) Received: from smtpin15.hostedemail.com (10.5.19.251.rfc1918.com [10.5.19.251]) by forelay04.hostedemail.com (Postfix) with ESMTP id 2229714961D for ; Thu, 18 Jun 2020 14:44:19 +0000 (UTC) X-FDA: 76942603038.15.thing31_221076326e11 Received: from filter.hostedemail.com (10.5.16.251.rfc1918.com [10.5.16.251]) by smtpin15.hostedemail.com (Postfix) with ESMTP id DEBD71814B0C1 for ; Thu, 18 Jun 2020 14:44:18 +0000 (UTC) X-Spam-Summary: 2,0,0,a15a4744718f31cb,d41d8cd98f00b204,axboe@kernel.dk,,RULES_HIT:41:355:379:541:800:960:988:989:1260:1311:1314:1345:1359:1431:1515:1534:1539:1567:1711:1714:1730:1747:1777:1792:2194:2199:2393:2559:2562:3138:3139:3140:3141:3142:3876:3877:4321:5007:6114:6261:6642:6653:10004:11026:11473:11658:11914:12438:12517:12519:12555:12895:13069:13311:13357:13894:14181:14384:14394:14721:21080:21444:21451:21627:21990:30054,0,RBL:209.85.215.195:@kernel.dk:.lbl8.mailshell.net-62.2.0.100 66.100.201.201,CacheIP:none,Bayesian:0.5,0.5,0.5,Netcheck:none,DomainCache:0,MSF:not bulk,SPF:fp,MSBL:0,DNSBL:neutral,Custom_rules:0:0:0,LFtime:25,LUA_SUMMARY:none X-HE-Tag: thing31_221076326e11 X-Filterd-Recvd-Size: 3688 Received: from mail-pg1-f195.google.com (mail-pg1-f195.google.com [209.85.215.195]) by imf17.hostedemail.com (Postfix) with ESMTP for ; Thu, 18 Jun 2020 14:44:18 +0000 (UTC) Received: by mail-pg1-f195.google.com with SMTP id d4so3006169pgk.4 for ; Thu, 18 Jun 2020 07:44:18 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=kernel-dk.20150623.gappssmtp.com; s=20150623; h=from:to:cc:subject:date:message-id:in-reply-to:references:reply-to :mime-version:content-transfer-encoding; bh=agiR+Byeck2s775I8LSKf8tm0BQVQsWoO+u3QsmIwcE=; b=CmsR4pnk1FMjkbrTWikiw1ru36jrxC1tfShgKpQNL5fwTTR48LJzuB0WFeslN0A8R7 yVhdczhT7rCHZGNSPImdCvg2GFHYjPZq3Lo8LN55YdVfIqbcMnyIHIfCWRTp7nIlWoCP 8mIA0QOEOodS5hrggItuRz50XlDx1FRg2U51ti0K2K6RLhv04afTVM85LjEQHy1K3put aih63C1nllRb0Y93OtUHEJJE5MygBtOKi6TQwE2EGgx2382rGAnQZJ70Af+bxcWW78if p90wfGKQkPEoYPs0wXfDt35iVSEZ+5uCYlZohmz4//abZsAjr1a8x2PskDZN8j4J5CQD CYxQ== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20161025; h=x-gm-message-state:from:to:cc:subject:date:message-id:in-reply-to :references:reply-to:mime-version:content-transfer-encoding; bh=agiR+Byeck2s775I8LSKf8tm0BQVQsWoO+u3QsmIwcE=; b=kZZ23R+Jx2UJ1Z0fXCamZai/7fQ3OoxENRtJ9rAc5LJM0j+KTu88QMQNOJRcFoNB73 4ZFqd53M3+EZhg2rJTXH36hu9M1agnye7vnhNAYat/Sy1fiGLgiN2dBSW++3Jn6tiap3 v6q6r9QUamcWDGQviS0Yo+4FhbGaxCEP5wtSs5vsr5rQ3xY7f+nTAozwGyeSttfH1EpY ud21wwOTcPPv6rxwhPQW53t+ZHZVqLK0gAGoJhzGAAIh8liTxN4hYyXXFUVgQTyXKQIN zlQkvNuMtEXyKQnx14NvDobxlXY1boMKLGTQ3EFoDlPJzkiymU8lEaBEQxbON7OWfX2D PLVg== X-Gm-Message-State: AOAM532YYRSAPrNtNUxRvUNsS/3hI8ePB8BtoJb6qPgoANFOe7WXVMZz f9rTwe8QgTzMbuMsT9Ebt5/RUk7+9WQ5Iw== X-Google-Smtp-Source: ABdhPJyNr3+kHnVN8fM6oIP5PSOO4gsldaD6y/PdYSuuK7cpSwrVAwO5ubxT1Qc0fzuBScSDRUwL4w== X-Received: by 2002:a65:6558:: with SMTP id a24mr1850398pgw.110.1592491457531; Thu, 18 Jun 2020 07:44:17 -0700 (PDT) Received: from x1.localdomain ([65.144.74.34]) by smtp.gmail.com with ESMTPSA id g9sm3127197pfm.151.2020.06.18.07.44.16 (version=TLS1_3 cipher=TLS_AES_256_GCM_SHA384 bits=256/256); Thu, 18 Jun 2020 07:44:17 -0700 (PDT) From: Jens Axboe To: io-uring@vger.kernel.org Cc: linux-fsdevel@vger.kernel.org, linux-kernel@vger.kernel.org, linux-mm@kvack.org, akpm@linux-foundation.org, Jens Axboe , Theodore Ts'o Subject: [PATCH 13/15] ext4: flag as supporting buffered async reads Date: Thu, 18 Jun 2020 08:43:53 -0600 Message-Id: <20200618144355.17324-14-axboe@kernel.dk> X-Mailer: git-send-email 2.27.0 In-Reply-To: <20200618144355.17324-1-axboe@kernel.dk> References: <20200618144355.17324-1-axboe@kernel.dk> Reply-To: "[PATCHSET v7 0/15]"@smtpin15.hostedemail.com, Add@smtpin15.hostedemail.com, support@smtpin15.hostedemail.com, for@smtpin15.hostedemail.com, async@smtpin15.hostedemail.com, buffered@smtpin15.hostedemail.com, reads@smtpin15.hostedemail.com MIME-Version: 1.0 X-Rspamd-Queue-Id: DEBD71814B0C1 X-Spamd-Result: default: False [0.00 / 100.00] X-Rspamd-Server: rspam04 X-Bogosity: Ham, tests=bogofilter, spamicity=0.000000, version=1.2.4 Sender: owner-linux-mm@kvack.org Precedence: bulk X-Loop: owner-majordomo@kvack.org List-ID: ext4 uses generic_file_read_iter(), which already supports this. Cc: Theodore Ts'o Signed-off-by: Jens Axboe --- fs/ext4/file.c | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/fs/ext4/file.c b/fs/ext4/file.c index 2a01e31a032c..1e827410e9e1 100644 --- a/fs/ext4/file.c +++ b/fs/ext4/file.c @@ -839,7 +839,7 @@ static int ext4_file_open(struct inode * inode, struct file * filp) return ret; } - filp->f_mode |= FMODE_NOWAIT; + filp->f_mode |= FMODE_NOWAIT | FMODE_BUF_RASYNC; return dquot_file_open(inode, filp); } From patchwork Thu Jun 18 14:43:54 2020 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Jens Axboe X-Patchwork-Id: 11612351 Return-Path: Received: from mail.kernel.org (pdx-korg-mail-1.web.codeaurora.org [172.30.200.123]) by pdx-korg-patchwork-2.web.codeaurora.org (Postfix) with ESMTP id 4FD05913 for ; Thu, 18 Jun 2020 14:44:41 +0000 (UTC) Received: from kanga.kvack.org (kanga.kvack.org [205.233.56.17]) by mail.kernel.org (Postfix) with ESMTP id 1CA2920888 for ; Thu, 18 Jun 2020 14:44:41 +0000 (UTC) Authentication-Results: mail.kernel.org; dkim=fail reason="signature verification failed" (2048-bit key) header.d=kernel-dk.20150623.gappssmtp.com header.i=@kernel-dk.20150623.gappssmtp.com header.b="bUiZ2Fxa" DMARC-Filter: OpenDMARC Filter v1.3.2 mail.kernel.org 1CA2920888 Authentication-Results: mail.kernel.org; dmarc=none (p=none dis=none) header.from=kernel.dk Authentication-Results: mail.kernel.org; spf=pass smtp.mailfrom=owner-linux-mm@kvack.org Received: by kanga.kvack.org (Postfix) id CBD1A8D0026; Thu, 18 Jun 2020 10:44:20 -0400 (EDT) Delivered-To: linux-mm-outgoing@kvack.org Received: by kanga.kvack.org (Postfix, from userid 40) id C1C1F8D0018; Thu, 18 Jun 2020 10:44:20 -0400 (EDT) X-Original-To: int-list-linux-mm@kvack.org X-Delivered-To: int-list-linux-mm@kvack.org Received: by kanga.kvack.org (Postfix, from userid 63042) id B31688D0026; Thu, 18 Jun 2020 10:44:20 -0400 (EDT) X-Original-To: linux-mm@kvack.org X-Delivered-To: linux-mm@kvack.org Received: from forelay.hostedemail.com (smtprelay0191.hostedemail.com [216.40.44.191]) by kanga.kvack.org (Postfix) with ESMTP id 8C9A38D0018 for ; Thu, 18 Jun 2020 10:44:20 -0400 (EDT) Received: from smtpin02.hostedemail.com (10.5.19.251.rfc1918.com [10.5.19.251]) by forelay04.hostedemail.com (Postfix) with ESMTP id 449951498EF for ; Thu, 18 Jun 2020 14:44:20 +0000 (UTC) X-FDA: 76942603080.02.glue33_0303ab226e11 Received: from filter.hostedemail.com (10.5.16.251.rfc1918.com [10.5.16.251]) by smtpin02.hostedemail.com (Postfix) with ESMTP id 15D9DAA249 for ; Thu, 18 Jun 2020 14:44:20 +0000 (UTC) X-Spam-Summary: 2,0,0,adfaff542ccc240b,d41d8cd98f00b204,axboe@kernel.dk,,RULES_HIT:41:355:379:541:800:960:973:988:989:1260:1311:1314:1345:1359:1431:1437:1515:1534:1541:1711:1730:1747:1777:1792:2393:2559:2562:3138:3139:3140:3141:3142:3352:3865:3867:3868:3870:3872:3874:5007:6261:6653:10004:11026:11473:11658:11914:12043:12114:12291:12438:12517:12519:12555:12895:13069:13311:13357:13894:14096:14181:14384:14394:14721:21080:21324:21444:21451:21627:21990:30054,0,RBL:209.85.215.195:@kernel.dk:.lbl8.mailshell.net-62.2.0.100 66.100.201.201,CacheIP:none,Bayesian:0.5,0.5,0.5,Netcheck:none,DomainCache:0,MSF:not bulk,SPF:fp,MSBL:0,DNSBL:neutral,Custom_rules:0:0:0,LFtime:24,LUA_SUMMARY:none X-HE-Tag: glue33_0303ab226e11 X-Filterd-Recvd-Size: 4461 Received: from mail-pg1-f195.google.com (mail-pg1-f195.google.com [209.85.215.195]) by imf18.hostedemail.com (Postfix) with ESMTP for ; Thu, 18 Jun 2020 14:44:19 +0000 (UTC) Received: by mail-pg1-f195.google.com with SMTP id h10so2988001pgq.10 for ; Thu, 18 Jun 2020 07:44:19 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=kernel-dk.20150623.gappssmtp.com; s=20150623; h=from:to:cc:subject:date:message-id:in-reply-to:references:reply-to :mime-version:content-transfer-encoding; bh=qG+PVAwYXEH3Rowu54+BkyZS7fslCJIulLcAGKfzRB8=; b=bUiZ2FxaCsZtyTAaJBoFcJMrX/CUga9qC0v2B0H5JcW44TVjLNQskpGK86a5Zb631M p0I2rj2UBPJFfNusO9HGLjnYw/iM43LqQ928bwfe2TjZbjF/4Hg7nfa5uPlMNvTn3vO1 apcy+p2q8hky9Ls8gdUEdvgAwIA3LtH/8apm463k0TATUTGX0gwMdBhkDB52mw3XPrHh GrIGx+AghUZ13e4QtMHcv9dXOv4lssAk0Hv512Bg1on9er9ClFRT03k4kQjM4RRxOSPW 6hvmoIIT3WZGyFLsTEshOvym+xTAnpq2veuMMOcqepVgDlq37cK5gOQh2kGyUaORNsIr A/uw== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20161025; h=x-gm-message-state:from:to:cc:subject:date:message-id:in-reply-to :references:reply-to:mime-version:content-transfer-encoding; bh=qG+PVAwYXEH3Rowu54+BkyZS7fslCJIulLcAGKfzRB8=; b=YP/iEqaCDNIYDvm3PdD0jau8ypc6LYK18jzohQMQJvsD+OUaqE7x2oSIBgMs75Jtkd Oi6Qek8zmI2M16aJdoi4x7NbC0b5wRTa0xr9F8rNGYBKqL/brM8G4TiLgIj9eAP2tByb xRZB+jXugAEI4Ah5+AuGNO8w1LVFxFGtHxmcTP8T36OxmYqSb36njysO3SPlBC9uQ8+u PzSkzRXHiuZxLnE8scyiNWCFKfT8AuwpmlW9VYvpkjlBmiWhnAJthBmtRffe9mZ3ssDS bD0w74Q9ISjj7cjLrVp4aIf6h6TcpYJxp4TaDmnL2Dth5DdFK9aEgXkzXEC4nsFh7Zs+ fryQ== X-Gm-Message-State: AOAM533QFS2dY/gqLQkVo2LJa2pL4tfmVHu2vMHlSqXNnvaLA2NaSTAg FvZ8HLWbl6OlVy38O0MfrlgtJw== X-Google-Smtp-Source: ABdhPJzzGfgUvN38nQlE0ltPql9r24L3d5fAFgRA+DRt5giXUwPYRi3iuK4YpxPwlcYuqcOquHT1Ag== X-Received: by 2002:a62:7e95:: with SMTP id z143mr3898104pfc.108.1592491458734; Thu, 18 Jun 2020 07:44:18 -0700 (PDT) Received: from x1.localdomain ([65.144.74.34]) by smtp.gmail.com with ESMTPSA id g9sm3127197pfm.151.2020.06.18.07.44.17 (version=TLS1_3 cipher=TLS_AES_256_GCM_SHA384 bits=256/256); Thu, 18 Jun 2020 07:44:18 -0700 (PDT) From: Jens Axboe To: io-uring@vger.kernel.org Cc: linux-fsdevel@vger.kernel.org, linux-kernel@vger.kernel.org, linux-mm@kvack.org, akpm@linux-foundation.org, Jens Axboe , Johannes Weiner Subject: [PATCH 14/15] mm: add kiocb_wait_page_queue_init() helper Date: Thu, 18 Jun 2020 08:43:54 -0600 Message-Id: <20200618144355.17324-15-axboe@kernel.dk> X-Mailer: git-send-email 2.27.0 In-Reply-To: <20200618144355.17324-1-axboe@kernel.dk> References: <20200618144355.17324-1-axboe@kernel.dk> Reply-To: "[PATCHSET v7 0/15]"@smtpin02.hostedemail.com, Add@smtpin02.hostedemail.com, support@smtpin02.hostedemail.com, for@smtpin02.hostedemail.com, async@smtpin02.hostedemail.com, buffered@smtpin02.hostedemail.com, reads@smtpin02.hostedemail.com MIME-Version: 1.0 X-Rspamd-Queue-Id: 15D9DAA249 X-Spamd-Result: default: False [0.00 / 100.00] X-Rspamd-Server: rspam03 X-Bogosity: Ham, tests=bogofilter, spamicity=0.000000, version=1.2.4 Sender: owner-linux-mm@kvack.org Precedence: bulk X-Loop: owner-majordomo@kvack.org List-ID: Checks if the file supports it, and initializes the values that we need. Caller passes in 'data' pointer, if any, and the callback function to be used. Acked-by: Johannes Weiner Signed-off-by: Jens Axboe --- include/linux/pagemap.h | 21 +++++++++++++++++++++ 1 file changed, 21 insertions(+) diff --git a/include/linux/pagemap.h b/include/linux/pagemap.h index e053e1d9a4d7..7386bc67cc5a 100644 --- a/include/linux/pagemap.h +++ b/include/linux/pagemap.h @@ -533,6 +533,27 @@ static inline int wake_page_match(struct wait_page_queue *wait_page, return 1; } +static inline int kiocb_wait_page_queue_init(struct kiocb *kiocb, + struct wait_page_queue *wait, + wait_queue_func_t func, + void *data) +{ + /* Can't support async wakeup with polled IO */ + if (kiocb->ki_flags & IOCB_HIPRI) + return -EINVAL; + if (kiocb->ki_filp->f_mode & FMODE_BUF_RASYNC) { + wait->wait.func = func; + wait->wait.private = data; + wait->wait.flags = 0; + INIT_LIST_HEAD(&wait->wait.entry); + kiocb->ki_flags |= IOCB_WAITQ; + kiocb->ki_waitq = wait; + return 0; + } + + return -EOPNOTSUPP; +} + extern void __lock_page(struct page *page); extern int __lock_page_killable(struct page *page); extern int __lock_page_async(struct page *page, struct wait_page_queue *wait); From patchwork Thu Jun 18 14:43:55 2020 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Jens Axboe X-Patchwork-Id: 11612353 Return-Path: Received: from mail.kernel.org (pdx-korg-mail-1.web.codeaurora.org [172.30.200.123]) by pdx-korg-patchwork-2.web.codeaurora.org (Postfix) with ESMTP id 81D7B14E3 for ; Thu, 18 Jun 2020 14:44:44 +0000 (UTC) Received: from kanga.kvack.org (kanga.kvack.org [205.233.56.17]) by mail.kernel.org (Postfix) with ESMTP id 407CE20888 for ; Thu, 18 Jun 2020 14:44:44 +0000 (UTC) Authentication-Results: mail.kernel.org; dkim=fail reason="signature verification failed" (2048-bit key) header.d=kernel-dk.20150623.gappssmtp.com header.i=@kernel-dk.20150623.gappssmtp.com header.b="RfkCMfGN" DMARC-Filter: OpenDMARC Filter v1.3.2 mail.kernel.org 407CE20888 Authentication-Results: mail.kernel.org; dmarc=none (p=none dis=none) header.from=kernel.dk Authentication-Results: mail.kernel.org; spf=pass smtp.mailfrom=owner-linux-mm@kvack.org Received: by kanga.kvack.org (Postfix) id D7C6D8D0027; Thu, 18 Jun 2020 10:44:21 -0400 (EDT) Delivered-To: linux-mm-outgoing@kvack.org Received: by kanga.kvack.org (Postfix, from userid 40) id D2D708D0018; Thu, 18 Jun 2020 10:44:21 -0400 (EDT) X-Original-To: int-list-linux-mm@kvack.org X-Delivered-To: int-list-linux-mm@kvack.org Received: by kanga.kvack.org (Postfix, from userid 63042) id C43278D0027; Thu, 18 Jun 2020 10:44:21 -0400 (EDT) X-Original-To: linux-mm@kvack.org X-Delivered-To: linux-mm@kvack.org Received: from forelay.hostedemail.com (smtprelay0110.hostedemail.com [216.40.44.110]) by kanga.kvack.org (Postfix) with ESMTP id A25778D0018 for ; Thu, 18 Jun 2020 10:44:21 -0400 (EDT) Received: from smtpin08.hostedemail.com (10.5.19.251.rfc1918.com [10.5.19.251]) by forelay02.hostedemail.com (Postfix) with ESMTP id 687001498FF for ; Thu, 18 Jun 2020 14:44:21 +0000 (UTC) X-FDA: 76942603122.08.screw77_4f0dd7326e11 Received: from filter.hostedemail.com (10.5.16.251.rfc1918.com [10.5.16.251]) by smtpin08.hostedemail.com (Postfix) with ESMTP id 38B63180C95B5 for ; Thu, 18 Jun 2020 14:44:21 +0000 (UTC) X-Spam-Summary: 2,0,0,692872ca61be9aca,d41d8cd98f00b204,axboe@kernel.dk,,RULES_HIT:2:41:355:379:541:800:960:966:973:988:989:1260:1311:1314:1345:1359:1431:1434:1437:1515:1535:1605:1606:1730:1747:1777:1792:2196:2199:2393:2559:2562:3138:3139:3140:3141:3142:3865:3867:3868:3870:3871:3872:4120:4321:4385:4398:5007:6119:6261:6653:7903:8603:10004:11026:11473:11658:11914:12043:12291:12296:12438:12517:12519:12555:12683:12895:12986:13161:13229:13894:14394:21080:21324:21444:21451:21627:21740:21990:30054:30070,0,RBL:209.85.214.193:@kernel.dk:.lbl8.mailshell.net-66.100.201.201 62.2.0.100,CacheIP:none,Bayesian:0.5,0.5,0.5,Netcheck:none,DomainCache:0,MSF:not bulk,SPF:fp,MSBL:0,DNSBL:neutral,Custom_rules:0:0:0,LFtime:22,LUA_SUMMARY:none X-HE-Tag: screw77_4f0dd7326e11 X-Filterd-Recvd-Size: 9002 Received: from mail-pl1-f193.google.com (mail-pl1-f193.google.com [209.85.214.193]) by imf28.hostedemail.com (Postfix) with ESMTP for ; Thu, 18 Jun 2020 14:44:20 +0000 (UTC) Received: by mail-pl1-f193.google.com with SMTP id g12so2514327pll.10 for ; Thu, 18 Jun 2020 07:44:20 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=kernel-dk.20150623.gappssmtp.com; s=20150623; h=from:to:cc:subject:date:message-id:in-reply-to:references:reply-to :mime-version:content-transfer-encoding; bh=hItvrdMXW3x4/4yokWms4yIKM/BxkkcMkxbyn/WrUe0=; b=RfkCMfGNHVXXALcOZPZ2jdMi+X2j7BbBbajuLvtbMQgxMG/R/SN9Lmnx9NaM1whfmk FoFat2Ffm7ozF2Am9lqvkz0jsEoF+28OalJ8pxWgXrDzHjm9PzijBHVdQeoN953xZdDY tf/dvYtXZA04B8ODzL5vUKx87iaBZufPMtd6kedtozYO9BMoAcWeY9LWTHO0Ida6RpP3 x1eoT/nTeXvRj4v/5yKnIft1RpIV2avzGEzDdkbF3szc11YLa/Lb2tqMcnSEDn0GFTjr Fg3FMcJwXY/ZGtqbb19AHc5aPFxfUxJG8rl9k5i052NZMKop0iiA8EZgxFfSHlvxzSqN C9BA== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20161025; h=x-gm-message-state:from:to:cc:subject:date:message-id:in-reply-to :references:reply-to:mime-version:content-transfer-encoding; bh=hItvrdMXW3x4/4yokWms4yIKM/BxkkcMkxbyn/WrUe0=; b=b5reuceDOeTZLnjzZYs3i6rp3xDDNERjmE/NTncBPXhcgvGMblhtLfVwVzD1fb5tiZ UO9MW8ARRQZLbO6Z2se4oi963HP97u1KVk4ORsFRofaSX2rCmeQwf5KvyhFt5yjRhXUV AjA70FhcheNBLvdbGSM17xQ2hlGIH6lHnX+iISpubs9M1sH1aYQ/YnZsBwBfex7fcJwt LPOmyLis4EysBIWywpbg4YJQJC2GMiN8AgwM+hwZNlp8sXSqQY9qVfmvn8sb8h1R4TAX Qp8zzrSarqSSvL+xafve/gyepodhn6PoRgpAwsKnsdUt5LWQWbSMki8tHGuPHQgA+bXu 62FA== X-Gm-Message-State: AOAM530/V9wlBnF1dOELqfn2eZ5IWPz5aRB6iYDaL/vPO+FFBj+mHzfM 7pJ1VXxbhBFvaQLx14oek9aeEg== X-Google-Smtp-Source: ABdhPJxS5EFTVcmIXFEiggPIpu8uYctGJadye2v8tGqmJWPLNu+hH9ccfCYUZJj+BdxVYR30oM/QCw== X-Received: by 2002:a17:902:7c8f:: with SMTP id y15mr3899856pll.95.1592491459962; Thu, 18 Jun 2020 07:44:19 -0700 (PDT) Received: from x1.localdomain ([65.144.74.34]) by smtp.gmail.com with ESMTPSA id g9sm3127197pfm.151.2020.06.18.07.44.18 (version=TLS1_3 cipher=TLS_AES_256_GCM_SHA384 bits=256/256); Thu, 18 Jun 2020 07:44:19 -0700 (PDT) From: Jens Axboe To: io-uring@vger.kernel.org Cc: linux-fsdevel@vger.kernel.org, linux-kernel@vger.kernel.org, linux-mm@kvack.org, akpm@linux-foundation.org, Jens Axboe Subject: [PATCH 15/15] io_uring: support true async buffered reads, if file provides it Date: Thu, 18 Jun 2020 08:43:55 -0600 Message-Id: <20200618144355.17324-16-axboe@kernel.dk> X-Mailer: git-send-email 2.27.0 In-Reply-To: <20200618144355.17324-1-axboe@kernel.dk> References: <20200618144355.17324-1-axboe@kernel.dk> Reply-To: "[PATCHSET v7 0/15]"@smtpin08.hostedemail.com, Add@smtpin08.hostedemail.com, support@smtpin08.hostedemail.com, for@smtpin08.hostedemail.com, async@smtpin08.hostedemail.com, buffered@smtpin08.hostedemail.com, reads@smtpin08.hostedemail.com MIME-Version: 1.0 X-Rspamd-Queue-Id: 38B63180C95B5 X-Spamd-Result: default: False [0.00 / 100.00] X-Rspamd-Server: rspam05 X-Bogosity: Ham, tests=bogofilter, spamicity=0.000000, version=1.2.4 Sender: owner-linux-mm@kvack.org Precedence: bulk X-Loop: owner-majordomo@kvack.org List-ID: If the file is flagged with FMODE_BUF_RASYNC, then we don't have to punt the buffered read to an io-wq worker. Instead we can rely on page unlocking callbacks to support retry based async IO. This is a lot more efficient than doing async thread offload. The retry is done similarly to how we handle poll based retry. From the unlock callback, we simply queue the retry to a task_work based handler. Signed-off-by: Jens Axboe --- fs/io_uring.c | 145 +++++++++++++++++++++++++++++++++++++++++++++++--- 1 file changed, 137 insertions(+), 8 deletions(-) diff --git a/fs/io_uring.c b/fs/io_uring.c index 40413fb9d07b..94282be1c413 100644 --- a/fs/io_uring.c +++ b/fs/io_uring.c @@ -78,6 +78,7 @@ #include #include #include +#include #define CREATE_TRACE_POINTS #include @@ -503,6 +504,8 @@ struct io_async_rw { struct iovec *iov; ssize_t nr_segs; ssize_t size; + struct wait_page_queue wpq; + struct callback_head task_work; }; struct io_async_ctx { @@ -2750,6 +2753,126 @@ static int io_read_prep(struct io_kiocb *req, const struct io_uring_sqe *sqe, return 0; } +static void __io_async_buf_error(struct io_kiocb *req, int error) +{ + struct io_ring_ctx *ctx = req->ctx; + + spin_lock_irq(&ctx->completion_lock); + io_cqring_fill_event(req, error); + io_commit_cqring(ctx); + spin_unlock_irq(&ctx->completion_lock); + + io_cqring_ev_posted(ctx); + req_set_fail_links(req); + io_double_put_req(req); +} + +static void io_async_buf_cancel(struct callback_head *cb) +{ + struct io_async_rw *rw; + struct io_kiocb *req; + + rw = container_of(cb, struct io_async_rw, task_work); + req = rw->wpq.wait.private; + __io_async_buf_error(req, -ECANCELED); +} + +static void io_async_buf_retry(struct callback_head *cb) +{ + struct io_async_rw *rw; + struct io_ring_ctx *ctx; + struct io_kiocb *req; + + rw = container_of(cb, struct io_async_rw, task_work); + req = rw->wpq.wait.private; + ctx = req->ctx; + + __set_current_state(TASK_RUNNING); + if (!io_sq_thread_acquire_mm(ctx, req)) { + mutex_lock(&ctx->uring_lock); + __io_queue_sqe(req, NULL); + mutex_unlock(&ctx->uring_lock); + } else { + __io_async_buf_error(req, -EFAULT); + } +} + +static int io_async_buf_func(struct wait_queue_entry *wait, unsigned mode, + int sync, void *arg) +{ + struct wait_page_queue *wpq; + struct io_kiocb *req = wait->private; + struct io_async_rw *rw = &req->io->rw; + struct wait_page_key *key = arg; + struct task_struct *tsk; + int ret; + + wpq = container_of(wait, struct wait_page_queue, wait); + + ret = wake_page_match(wpq, key); + if (ret != 1) + return ret; + + list_del_init(&wait->entry); + + init_task_work(&rw->task_work, io_async_buf_retry); + /* submit ref gets dropped, acquire a new one */ + refcount_inc(&req->refs); + tsk = req->task; + ret = task_work_add(tsk, &rw->task_work, true); + if (unlikely(ret)) { + /* queue just for cancelation */ + init_task_work(&rw->task_work, io_async_buf_cancel); + tsk = io_wq_get_task(req->ctx->io_wq); + task_work_add(tsk, &rw->task_work, true); + } + wake_up_process(tsk); + return 1; +} + +static bool io_rw_should_retry(struct io_kiocb *req) +{ + struct kiocb *kiocb = &req->rw.kiocb; + int ret; + + /* never retry for NOWAIT, we just complete with -EAGAIN */ + if (req->flags & REQ_F_NOWAIT) + return false; + + /* already tried, or we're doing O_DIRECT */ + if (kiocb->ki_flags & (IOCB_DIRECT | IOCB_WAITQ)) + return false; + /* + * just use poll if we can, and don't attempt if the fs doesn't + * support callback based unlocks + */ + if (file_can_poll(req->file) || !(req->file->f_mode & FMODE_BUF_RASYNC)) + return false; + + /* + * If request type doesn't require req->io to defer in general, + * we need to allocate it here + */ + if (!req->io && __io_alloc_async_ctx(req)) + return false; + + ret = kiocb_wait_page_queue_init(kiocb, &req->io->rw.wpq, + io_async_buf_func, req); + if (!ret) { + io_get_req_task(req); + return true; + } + + return false; +} + +static int io_iter_do_read(struct io_kiocb *req, struct iov_iter *iter) +{ + if (req->file->f_op->read_iter) + return call_read_iter(req->file, &req->rw.kiocb, iter); + return loop_rw_iter(READ, req->file, &req->rw.kiocb, iter); +} + static int io_read(struct io_kiocb *req, bool force_nonblock) { struct iovec inline_vecs[UIO_FASTIOV], *iovec = inline_vecs; @@ -2784,10 +2907,7 @@ static int io_read(struct io_kiocb *req, bool force_nonblock) unsigned long nr_segs = iter.nr_segs; ssize_t ret2 = 0; - if (req->file->f_op->read_iter) - ret2 = call_read_iter(req->file, kiocb, &iter); - else - ret2 = loop_rw_iter(READ, req->file, kiocb, &iter); + ret2 = io_iter_do_read(req, &iter); /* Catch -EAGAIN return for forced non-blocking submission */ if (!force_nonblock || (ret2 != -EAGAIN && ret2 != -EIO)) { @@ -2799,17 +2919,26 @@ static int io_read(struct io_kiocb *req, bool force_nonblock) ret = io_setup_async_rw(req, io_size, iovec, inline_vecs, &iter); if (ret) - goto out_free; + goto out; /* any defer here is final, must blocking retry */ if (!(req->flags & REQ_F_NOWAIT) && !file_can_poll(req->file)) req->flags |= REQ_F_MUST_PUNT; + /* if we can retry, do so with the callbacks armed */ + if (io_rw_should_retry(req)) { + ret2 = io_iter_do_read(req, &iter); + if (ret2 == -EIOCBQUEUED) { + goto out; + } else if (ret2 != -EAGAIN) { + kiocb_done(kiocb, ret2); + goto out; + } + } + kiocb->ki_flags &= ~IOCB_WAITQ; return -EAGAIN; } } -out_free: - kfree(iovec); - req->flags &= ~REQ_F_NEED_CLEANUP; +out: return ret; }