From patchwork Wed Oct 27 12:21:07 2021 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Pavel Begunkov X-Patchwork-Id: 12587189 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from mail.kernel.org (mail.kernel.org [198.145.29.99]) by smtp.lore.kernel.org (Postfix) with ESMTP id 436ADC433F5 for ; Wed, 27 Oct 2021 12:21:36 +0000 (UTC) Received: from vger.kernel.org (vger.kernel.org [23.128.96.18]) by mail.kernel.org (Postfix) with ESMTP id 1F16B610A0 for ; Wed, 27 Oct 2021 12:21:36 +0000 (UTC) Received: (majordomo@vger.kernel.org) by vger.kernel.org via listexpand id S241838AbhJ0MYA (ORCPT ); Wed, 27 Oct 2021 08:24:00 -0400 Received: from lindbergh.monkeyblade.net ([23.128.96.19]:42546 "EHLO lindbergh.monkeyblade.net" rhost-flags-OK-OK-OK-OK) by vger.kernel.org with ESMTP id S237238AbhJ0MX7 (ORCPT ); Wed, 27 Oct 2021 08:23:59 -0400 Received: from mail-wm1-x32f.google.com (mail-wm1-x32f.google.com [IPv6:2a00:1450:4864:20::32f]) by lindbergh.monkeyblade.net (Postfix) with ESMTPS id 9FEC2C061570 for ; Wed, 27 Oct 2021 05:21:33 -0700 (PDT) Received: by mail-wm1-x32f.google.com with SMTP id z200so1603360wmc.1 for ; Wed, 27 Oct 2021 05:21:33 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=20210112; h=from:to:cc:subject:date:message-id:in-reply-to:references :mime-version:content-transfer-encoding; bh=9pO6D7cQueBQZGbkr1DMJ0+9BSyr2Nbj9PUyzZfSxnI=; b=OCLk8CQIy/e+HNbJ+Q+QrX6pG+KIUBehRuSWhE7qba2odsxedyAzlk3RsR31eRNHQp /QxVxRMy/L5cnYHXxOB4mkzDI+teeSWar7KkcPepyN33pAGAZRjpy31L8JNXbBtDvypb bWKNZJLg4ApmxO58FmnSn1yEOcq9MBR8zAL2TgW5K4I/LsSANNlm8S12tZ8/xly2Aroj ZhsYleaU03TJCGvkFOP4ild8Yx1mB+vutUYfTdRVCUkJ59ct4ukcTMLmxwlNA01rUtqK eMjpUJ+IZMdPNEa6QatiwfPDi4RCuvDv3S4HkbEi3OpEHywK9ss9f5l5gTtJ9Mw/H6Sz pMuQ== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20210112; h=x-gm-message-state:from:to:cc:subject:date:message-id:in-reply-to :references:mime-version:content-transfer-encoding; bh=9pO6D7cQueBQZGbkr1DMJ0+9BSyr2Nbj9PUyzZfSxnI=; b=cJamwRZymWU0c496oHC8CzXhrSRs5o2r5dgBJ3Q+leZI92AOJudLHvY5P7XqBBXD5i rD3tTyzyRAjTK0VfNqTsLCAwLg/Hsh3MQ1ndIaCvsacFRx16JwmRCfNuCLBBaLNiNm2G ZI5VMme5KwWfZTGwCs5uU7SLIVtX/2z5YZBpc8J1ZgOMD37DI25uOsZp6tp3grGbkI8V 5iPTf4MoaxkbKxcFGUkgxsw4gHhjUB83MJ8PUhVrGMJ2k9w5cqFHhlAg5cDbQASIwr6G Fd2Q1RwenEzSk4wZkomJu9DDwcCxfwzUWR6fgQw6Ho4xH1Enn7onLCIGm/yCqGGO/YxS c/FA== X-Gm-Message-State: AOAM530/yQWi7PSIPvZL98I/22LQ5lIRftRNYUw40oL5Jiu8It+PbFZB nPzvBaNcBWPaMD+UxEqf9wIDw2vp5F4= X-Google-Smtp-Source: ABdhPJxrDpBoH5jMd6l0sb3Fg/vx1DcK4+p2inPwE+kxy55aYB8IEpPlyvPYhzd1Ki9fqvh0r/juaA== X-Received: by 2002:a7b:cc07:: with SMTP id f7mr4351487wmh.106.1635337292093; Wed, 27 Oct 2021 05:21:32 -0700 (PDT) Received: from 127.0.0.1localhost ([148.252.132.100]) by smtp.gmail.com with ESMTPSA id d8sm22738807wrv.80.2021.10.27.05.21.31 (version=TLS1_3 cipher=TLS_AES_256_GCM_SHA384 bits=256/256); Wed, 27 Oct 2021 05:21:31 -0700 (PDT) From: Pavel Begunkov To: linux-block@vger.kernel.org Cc: Jens Axboe , Christoph Hellwig , Pavel Begunkov Subject: [PATCH v2 1/4] block: avoid extra iter advance with async iocb Date: Wed, 27 Oct 2021 13:21:07 +0100 Message-Id: X-Mailer: git-send-email 2.33.1 In-Reply-To: References: MIME-Version: 1.0 Precedence: bulk List-ID: X-Mailing-List: linux-block@vger.kernel.org Nobody cares about iov iterators state if we return -EIOCBQUEUED, so as the we now have __blkdev_direct_IO_async(), which gets pages only once, we can skip expensive iov_iter_advance(). It's around 1-2% of all CPU spent. Signed-off-by: Pavel Begunkov --- block/bio.c | 2 +- block/fops.c | 20 +++++++++++++++----- include/linux/bio.h | 1 + 3 files changed, 17 insertions(+), 6 deletions(-) diff --git a/block/bio.c b/block/bio.c index ead1f8a9ff5e..15ab0d6d1c06 100644 --- a/block/bio.c +++ b/block/bio.c @@ -1046,7 +1046,7 @@ void __bio_release_pages(struct bio *bio, bool mark_dirty) } EXPORT_SYMBOL_GPL(__bio_release_pages); -static void bio_iov_bvec_set(struct bio *bio, struct iov_iter *iter) +void bio_iov_bvec_set(struct bio *bio, struct iov_iter *iter) { size_t size = iov_iter_count(iter); diff --git a/block/fops.c b/block/fops.c index a7b328296912..092e5079e827 100644 --- a/block/fops.c +++ b/block/fops.c @@ -352,11 +352,21 @@ static ssize_t __blkdev_direct_IO_async(struct kiocb *iocb, bio->bi_end_io = blkdev_bio_end_io_async; bio->bi_ioprio = iocb->ki_ioprio; - ret = bio_iov_iter_get_pages(bio, iter); - if (unlikely(ret)) { - bio->bi_status = BLK_STS_IOERR; - bio_endio(bio); - return ret; + if (iov_iter_is_bvec(iter)) { + /* + * Users don't rely on the iterator being in any particular + * state for async I/O returning -EIOCBQUEUED, hence we can + * avoid expensive iov_iter_advance(). Bypass + * bio_iov_iter_get_pages() and set the bvec directly. + */ + bio_iov_bvec_set(bio, iter); + } else { + ret = bio_iov_iter_get_pages(bio, iter); + if (unlikely(ret)) { + bio->bi_status = BLK_STS_IOERR; + bio_endio(bio); + return ret; + } } dio->size = bio->bi_iter.bi_size; diff --git a/include/linux/bio.h b/include/linux/bio.h index c88700d1bdc3..fe6bdfbbef66 100644 --- a/include/linux/bio.h +++ b/include/linux/bio.h @@ -417,6 +417,7 @@ int bio_add_zone_append_page(struct bio *bio, struct page *page, void __bio_add_page(struct bio *bio, struct page *page, unsigned int len, unsigned int off); int bio_iov_iter_get_pages(struct bio *bio, struct iov_iter *iter); +void bio_iov_bvec_set(struct bio *bio, struct iov_iter *iter); void __bio_release_pages(struct bio *bio, bool mark_dirty); extern void bio_set_pages_dirty(struct bio *bio); extern void bio_check_pages_dirty(struct bio *bio); From patchwork Wed Oct 27 12:21:08 2021 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Pavel Begunkov X-Patchwork-Id: 12587191 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from mail.kernel.org (mail.kernel.org [198.145.29.99]) by smtp.lore.kernel.org (Postfix) with ESMTP id 90D2EC433EF for ; Wed, 27 Oct 2021 12:21:36 +0000 (UTC) Received: from vger.kernel.org (vger.kernel.org [23.128.96.18]) by mail.kernel.org (Postfix) with ESMTP id 6F9FE610A6 for ; Wed, 27 Oct 2021 12:21:36 +0000 (UTC) Received: (majordomo@vger.kernel.org) by vger.kernel.org via listexpand id S237238AbhJ0MYA (ORCPT ); Wed, 27 Oct 2021 08:24:00 -0400 Received: from lindbergh.monkeyblade.net ([23.128.96.19]:42552 "EHLO lindbergh.monkeyblade.net" rhost-flags-OK-OK-OK-OK) by vger.kernel.org with ESMTP id S240313AbhJ0MYA (ORCPT ); Wed, 27 Oct 2021 08:24:00 -0400 Received: from mail-wr1-x42e.google.com (mail-wr1-x42e.google.com [IPv6:2a00:1450:4864:20::42e]) by lindbergh.monkeyblade.net (Postfix) with ESMTPS id BA757C061570 for ; Wed, 27 Oct 2021 05:21:34 -0700 (PDT) Received: by mail-wr1-x42e.google.com with SMTP id s14so925710wrb.3 for ; Wed, 27 Oct 2021 05:21:34 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=20210112; h=from:to:cc:subject:date:message-id:in-reply-to:references :mime-version:content-transfer-encoding; bh=UPnbATQy+wkyVbwyTCqUjjmvDxWAZyXRB8e8V3FSQR4=; b=Eq8gLHngtAEw27/j1DMrB9cPKubv6I0u542tGa9lv0LjMv56w+Njps4mG3yjtTlH77 Skdcx9FKIDjv4ma1K8ys8xbv4So8V7mD8YpqEwxv0BkYOXw0GXQO9xHdhlJxwK1gMS9H ZkyV+KVOYYqft34WdKZecAeKb76+XkfFV55xjyXcE09xeZUaOzYBLeo9nIo04bdG9wzi vZMkq/EcapSDOhguK6bh0O4idrhNq7fEvjwSQcPw1ZjCIQgKv/pom8XzMd7QgqzYlAQV HHj5o6IfE/JhKsYQSVW8oKp/QZWMQyHGeanQsBh+5rRYO7e2NgTneUqU33ToSNtgaNJt cuOQ== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20210112; h=x-gm-message-state:from:to:cc:subject:date:message-id:in-reply-to :references:mime-version:content-transfer-encoding; bh=UPnbATQy+wkyVbwyTCqUjjmvDxWAZyXRB8e8V3FSQR4=; b=B+becj9ASj1WAMqJxia+GFLRmqVYCSSmv0XmNzcn6DG8GxWgdGYMOaGY8grqoYgxTZ YoMxVzz4UGNZUPv2lgInV4kj/Jlnaj7S/zjtsEdTu0cfP8lZxCU4iSkBI3qxX4R1Odwu CJU2M7r+SWH9vavo6M0X+e3pHG8nkgYyAnqiRtN6kEFQfLtCKLrCIP3xLnoygQoRJI78 Jifoa3BtXlF86lM+oZe+2Yb1Hat7rRjUXUr8wylrNXZA+xq7MQpxmDv9bZtJgu5fQ8DQ B769icxPLEMXFF/S1mEsWgAIcxe/T+O8r4nj3051KfAWZrCASyEydRlM98Ir8lrNmphv L11w== X-Gm-Message-State: AOAM532GWm1O79F+hDt6foe/Jqta/ddQQKnwkvaEGKCO52kJIhtVo0eA hWGOyprydggMeO/rSMNldWtmbGBccvU= X-Google-Smtp-Source: ABdhPJysJdU2TgdkrcqPkhMRWpTne6odbsLAPe3mEJa+2ChzNVNV14tgGxKmUzxVaEPIJs7FwGNRHA== X-Received: by 2002:adf:d4c9:: with SMTP id w9mr34368763wrk.372.1635337293127; Wed, 27 Oct 2021 05:21:33 -0700 (PDT) Received: from 127.0.0.1localhost ([148.252.132.100]) by smtp.gmail.com with ESMTPSA id d8sm22738807wrv.80.2021.10.27.05.21.32 (version=TLS1_3 cipher=TLS_AES_256_GCM_SHA384 bits=256/256); Wed, 27 Oct 2021 05:21:32 -0700 (PDT) From: Pavel Begunkov To: linux-block@vger.kernel.org Cc: Jens Axboe , Christoph Hellwig , Pavel Begunkov Subject: [PATCH v2 2/4] block: kill unused polling bits in __blkdev_direct_IO() Date: Wed, 27 Oct 2021 13:21:08 +0100 Message-Id: X-Mailer: git-send-email 2.33.1 In-Reply-To: References: MIME-Version: 1.0 Precedence: bulk List-ID: X-Mailing-List: linux-block@vger.kernel.org With addition of __blkdev_direct_IO_async(), __blkdev_direct_IO() now serves only multio-bio I/O, which we don't poll. Now we can remove anything related to I/O polling from it. Signed-off-by: Pavel Begunkov --- block/fops.c | 20 +++----------------- 1 file changed, 3 insertions(+), 17 deletions(-) diff --git a/block/fops.c b/block/fops.c index 092e5079e827..983e993c9a4b 100644 --- a/block/fops.c +++ b/block/fops.c @@ -190,7 +190,6 @@ static ssize_t __blkdev_direct_IO(struct kiocb *iocb, struct iov_iter *iter, struct blk_plug plug; struct blkdev_dio *dio; struct bio *bio; - bool do_poll = (iocb->ki_flags & IOCB_HIPRI); bool is_read = (iov_iter_rw(iter) == READ), is_sync; loff_t pos = iocb->ki_pos; int ret = 0; @@ -216,12 +215,7 @@ static ssize_t __blkdev_direct_IO(struct kiocb *iocb, struct iov_iter *iter, if (is_read && iter_is_iovec(iter)) dio->flags |= DIO_SHOULD_DIRTY; - /* - * Don't plug for HIPRI/polled IO, as those should go straight - * to issue - */ - if (!(iocb->ki_flags & IOCB_HIPRI)) - blk_start_plug(&plug); + blk_start_plug(&plug); for (;;) { bio_set_dev(bio, bdev); @@ -254,11 +248,7 @@ static ssize_t __blkdev_direct_IO(struct kiocb *iocb, struct iov_iter *iter, nr_pages = bio_iov_vecs_to_alloc(iter, BIO_MAX_VECS); if (!nr_pages) { - if (do_poll) - bio_set_polled(bio, iocb); submit_bio(bio); - if (do_poll) - WRITE_ONCE(iocb->private, bio); break; } if (!(dio->flags & DIO_MULTI_BIO)) { @@ -271,7 +261,6 @@ static ssize_t __blkdev_direct_IO(struct kiocb *iocb, struct iov_iter *iter, bio_get(bio); dio->flags |= DIO_MULTI_BIO; atomic_set(&dio->ref, 2); - do_poll = false; } else { atomic_inc(&dio->ref); } @@ -280,8 +269,7 @@ static ssize_t __blkdev_direct_IO(struct kiocb *iocb, struct iov_iter *iter, bio = bio_alloc(GFP_KERNEL, nr_pages); } - if (!(iocb->ki_flags & IOCB_HIPRI)) - blk_finish_plug(&plug); + blk_finish_plug(&plug); if (!is_sync) return -EIOCBQUEUED; @@ -290,9 +278,7 @@ static ssize_t __blkdev_direct_IO(struct kiocb *iocb, struct iov_iter *iter, set_current_state(TASK_UNINTERRUPTIBLE); if (!READ_ONCE(dio->waiter)) break; - - if (!do_poll || !bio_poll(bio, NULL, 0)) - blk_io_schedule(); + blk_io_schedule(); } __set_current_state(TASK_RUNNING); From patchwork Wed Oct 27 12:21:09 2021 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Pavel Begunkov X-Patchwork-Id: 12587193 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from mail.kernel.org (mail.kernel.org [198.145.29.99]) by smtp.lore.kernel.org (Postfix) with ESMTP id 8CF76C4332F for ; Wed, 27 Oct 2021 12:21:37 +0000 (UTC) Received: from vger.kernel.org (vger.kernel.org [23.128.96.18]) by mail.kernel.org (Postfix) with ESMTP id 76B89610A6 for ; Wed, 27 Oct 2021 12:21:37 +0000 (UTC) Received: (majordomo@vger.kernel.org) by vger.kernel.org via listexpand id S240328AbhJ0MYB (ORCPT ); Wed, 27 Oct 2021 08:24:01 -0400 Received: from lindbergh.monkeyblade.net ([23.128.96.19]:42562 "EHLO lindbergh.monkeyblade.net" rhost-flags-OK-OK-OK-OK) by vger.kernel.org with ESMTP id S240313AbhJ0MYB (ORCPT ); Wed, 27 Oct 2021 08:24:01 -0400 Received: from mail-wm1-x32b.google.com (mail-wm1-x32b.google.com [IPv6:2a00:1450:4864:20::32b]) by lindbergh.monkeyblade.net (Postfix) with ESMTPS id E1A18C061570 for ; Wed, 27 Oct 2021 05:21:35 -0700 (PDT) Received: by mail-wm1-x32b.google.com with SMTP id 82-20020a1c0055000000b0032ccc3ad5c1so4158848wma.2 for ; Wed, 27 Oct 2021 05:21:35 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=20210112; h=from:to:cc:subject:date:message-id:in-reply-to:references :mime-version:content-transfer-encoding; bh=i1POAIRR4spiafMLsX1khFGBr5yXWQ77ysbzkKIjVLc=; b=fdp73yMe3kw33V8U8vrulbIl30IAPFHtJSD4v7QH1MvjeoqOgFsuXyildrUhBXYoCu uTuR1tdFRpqO+osaZq1Y9ZXnSpBHQvwdF8ji2hiyJ5ESeuyt3GTEYoB6DifUGbXO0so3 Gh3W4k4Dv4KOM21BZfuap225WxnX0VJpf2noXYSbzKukFHzRJqTUqdYCLdjk5jyyDi/7 HehRdTQR3OC6Q0eAYMXAdgCURJLWQUHkpxnIS1KDZ47pAMrlMbYc/oW6KAmjnx5sjwle XIp2va8aSqa0dZB0h3yxVYLTgoA6uiG6yTo25U2YrmbayiBEmeet9mJwMv0vWEzE+eGz zWIw== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20210112; h=x-gm-message-state:from:to:cc:subject:date:message-id:in-reply-to :references:mime-version:content-transfer-encoding; bh=i1POAIRR4spiafMLsX1khFGBr5yXWQ77ysbzkKIjVLc=; b=YPvNrxFgiqHal7G4dKJcT2sg5lHEEiLn+K3JOUxfB9hm1aLi+JMbVzcnc5MZ179MO0 xp1BE/6YsM+N+GVtCUtHXdxOb6CCyA0la0JyGoK0fwyGCpsHbRjT6alCKSl2yJQRh6CF 6emhTY1z7vWy+ol4vWYqhhhPRFitzLQBrlc9VpacwzoDSvp6VTHAnLeyJHFkNmpwDL3c qPJf6r28zr7BeyOCZ9UHyJcjsToXn2eUdGLti1DgBVsQeEiLBsGXhhxcH26mzKbIfTzl /9BQptKmVjESQxqSOvMU5lEmUIt2u6LldIGTQJtEqOGmeKAktt4uDwUTRBrrDIQo0cch p7+g== X-Gm-Message-State: AOAM531NBmYgCOKzl6lislBfzKidJFn+6a1SrYL1yLFY06kiO0Lh2tGy 1UdIshNvLmOSxm7gy0AnkT6L5SDD/oM= X-Google-Smtp-Source: ABdhPJy/OZXpPcRhwE0eECTSZ8Ec8xZzqh3S1vnp4MjPUjtHm3z9i8Dp4IHUyGEJroF9E/sxvZxPxA== X-Received: by 2002:a7b:c4c1:: with SMTP id g1mr5335703wmk.2.1635337294366; Wed, 27 Oct 2021 05:21:34 -0700 (PDT) Received: from 127.0.0.1localhost ([148.252.132.100]) by smtp.gmail.com with ESMTPSA id d8sm22738807wrv.80.2021.10.27.05.21.33 (version=TLS1_3 cipher=TLS_AES_256_GCM_SHA384 bits=256/256); Wed, 27 Oct 2021 05:21:34 -0700 (PDT) From: Pavel Begunkov To: linux-block@vger.kernel.org Cc: Jens Axboe , Christoph Hellwig , Pavel Begunkov Subject: [PATCH v2 3/4] block: kill DIO_MULTI_BIO Date: Wed, 27 Oct 2021 13:21:09 +0100 Message-Id: <88eb488aae9ed4852a30f3a7132f296f56e43b80.1635337135.git.asml.silence@gmail.com> X-Mailer: git-send-email 2.33.1 In-Reply-To: References: MIME-Version: 1.0 Precedence: bulk List-ID: X-Mailing-List: linux-block@vger.kernel.org Now __blkdev_direct_IO() serves only multi-bio I/O, thus remove not used anymore single bio refcounting optimisations. Signed-off-by: Pavel Begunkov --- block/fops.c | 33 ++++++++++++--------------------- 1 file changed, 12 insertions(+), 21 deletions(-) diff --git a/block/fops.c b/block/fops.c index 983e993c9a4b..8594852bd344 100644 --- a/block/fops.c +++ b/block/fops.c @@ -124,9 +124,8 @@ static ssize_t __blkdev_direct_IO_simple(struct kiocb *iocb, } enum { - DIO_MULTI_BIO = 1, - DIO_SHOULD_DIRTY = 2, - DIO_IS_SYNC = 4, + DIO_SHOULD_DIRTY = 1, + DIO_IS_SYNC = 2, }; struct blkdev_dio { @@ -150,7 +149,7 @@ static void blkdev_bio_end_io(struct bio *bio) if (bio->bi_status && !dio->bio.bi_status) dio->bio.bi_status = bio->bi_status; - if (!(dio->flags & DIO_MULTI_BIO) || atomic_dec_and_test(&dio->ref)) { + if (atomic_dec_and_test(&dio->ref)) { if (!(dio->flags & DIO_IS_SYNC)) { struct kiocb *iocb = dio->iocb; ssize_t ret; @@ -165,8 +164,7 @@ static void blkdev_bio_end_io(struct bio *bio) } dio->iocb->ki_complete(iocb, ret, 0); - if (dio->flags & DIO_MULTI_BIO) - bio_put(&dio->bio); + bio_put(&dio->bio); } else { struct task_struct *waiter = dio->waiter; @@ -201,11 +199,17 @@ static ssize_t __blkdev_direct_IO(struct kiocb *iocb, struct iov_iter *iter, bio = bio_alloc_kiocb(iocb, nr_pages, &blkdev_dio_pool); dio = container_of(bio, struct blkdev_dio, bio); + atomic_set(&dio->ref, 1); + /* + * Grab an extra reference to ensure the dio structure which is embedded + * into the first bio stays around. + */ + bio_get(bio); + is_sync = is_sync_kiocb(iocb); if (is_sync) { dio->flags = DIO_IS_SYNC; dio->waiter = current; - bio_get(bio); } else { dio->flags = 0; dio->iocb = iocb; @@ -251,20 +255,7 @@ static ssize_t __blkdev_direct_IO(struct kiocb *iocb, struct iov_iter *iter, submit_bio(bio); break; } - if (!(dio->flags & DIO_MULTI_BIO)) { - /* - * AIO needs an extra reference to ensure the dio - * structure which is embedded into the first bio - * stays around. - */ - if (!is_sync) - bio_get(bio); - dio->flags |= DIO_MULTI_BIO; - atomic_set(&dio->ref, 2); - } else { - atomic_inc(&dio->ref); - } - + atomic_inc(&dio->ref); submit_bio(bio); bio = bio_alloc(GFP_KERNEL, nr_pages); } From patchwork Wed Oct 27 12:21:10 2021 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Pavel Begunkov X-Patchwork-Id: 12587195 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from mail.kernel.org (mail.kernel.org [198.145.29.99]) by smtp.lore.kernel.org (Postfix) with ESMTP id 0191AC433EF for ; Wed, 27 Oct 2021 12:21:39 +0000 (UTC) Received: from vger.kernel.org (vger.kernel.org [23.128.96.18]) by mail.kernel.org (Postfix) with ESMTP id DEAA8610A4 for ; Wed, 27 Oct 2021 12:21:38 +0000 (UTC) Received: (majordomo@vger.kernel.org) by vger.kernel.org via listexpand id S241844AbhJ0MYD (ORCPT ); Wed, 27 Oct 2021 08:24:03 -0400 Received: from lindbergh.monkeyblade.net ([23.128.96.19]:42570 "EHLO lindbergh.monkeyblade.net" rhost-flags-OK-OK-OK-OK) by vger.kernel.org with ESMTP id S241846AbhJ0MYC (ORCPT ); Wed, 27 Oct 2021 08:24:02 -0400 Received: from mail-wm1-x332.google.com (mail-wm1-x332.google.com [IPv6:2a00:1450:4864:20::332]) by lindbergh.monkeyblade.net (Postfix) with ESMTPS id 51E99C061745 for ; Wed, 27 Oct 2021 05:21:37 -0700 (PDT) Received: by mail-wm1-x332.google.com with SMTP id b194-20020a1c1bcb000000b0032cd7b47853so1048086wmb.5 for ; Wed, 27 Oct 2021 05:21:37 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=20210112; h=from:to:cc:subject:date:message-id:in-reply-to:references :mime-version:content-transfer-encoding; bh=baeDNJqUEMF5KJZcKkua2oxmhlA7b2HKEDmgofMMaIM=; b=bth39+snAD/GmR9yoPqP4F3fXDWBqoKWAlSRbtcNsfQypa84DkK9KyQlzw1OUFGe+k m5K4wuW/X69N7k8xpgMs1tkisssvVHBM0oFDfbTyaFvzor+c3lYhk8+olJlPNjEAYPAf sR4id8Eq47g6UN+jCf7lO3EsmwktodNdqO9IqyRcg57b3XjYqOb1XA6SmRn/lVhUzrmI U0LAc95nxHrMBEHyRWr2PoHEh5qyHjJxL2gJS5v5SZvHLvMsQFJm15RIN9/Ziu3q9or1 ZNQibjsW/Weoau+myXj0hQLjFzX1JtzYbcFhtONVM7JeoPAjD55EGRRPYwPE0bGKc9Rn O7TQ== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20210112; h=x-gm-message-state:from:to:cc:subject:date:message-id:in-reply-to :references:mime-version:content-transfer-encoding; bh=baeDNJqUEMF5KJZcKkua2oxmhlA7b2HKEDmgofMMaIM=; b=r3A5UR3O1ncSQSOGz/HKpmL3XK9UtKxHFk5L3Wv2bJU646I6tkLc9KI/1AF1bWDOPP fbLfH7HSrUkr6lwSW0Gs24pXlsSBlWU8jMExgBoiz0JHd7xO6trZ9GGQVxSRiYK6NO3Y gF28YEj0t9TKa3hCZr/Ojqj3lU/CUgFGG5pPGPso4sh+Sc1w0zSXDfCikFTLbAABckLg OLMmz2dBo5doDYARhpcUavMp3YwFDv/b+ECEMd272iOmWlulbsBMK29fxWcEFomP/sim 2QLXN72ZkgKAVYkAirf4StulkgdSG4S7gFw4ds5qMsqzLGL1vwVIQxPI+Wf7an/V8YqW qIZA== X-Gm-Message-State: AOAM5313uEDiqaV8uuHqamd041DKQU9Q8a9oxVR2G923H4q+VLfUaLZA BwnKbIsLpv6pe+YdGWUjJS3nJl3IdRY= X-Google-Smtp-Source: ABdhPJz/gm7PC31LP3/ZNiPmmpHp4pDEAGKld6urmybxe6d2o1D3t2wAn4d9/Bu7ygTAI/ODM48Xwg== X-Received: by 2002:a1c:e90a:: with SMTP id q10mr5491753wmc.108.1635337295740; Wed, 27 Oct 2021 05:21:35 -0700 (PDT) Received: from 127.0.0.1localhost ([148.252.132.100]) by smtp.gmail.com with ESMTPSA id d8sm22738807wrv.80.2021.10.27.05.21.34 (version=TLS1_3 cipher=TLS_AES_256_GCM_SHA384 bits=256/256); Wed, 27 Oct 2021 05:21:35 -0700 (PDT) From: Pavel Begunkov To: linux-block@vger.kernel.org Cc: Jens Axboe , Christoph Hellwig , Pavel Begunkov Subject: [PATCH v2 4/4] block: add async version of bio_set_polled Date: Wed, 27 Oct 2021 13:21:10 +0100 Message-Id: <8fa137885164a5d05fadcff4c3521da8d5a83d00.1635337135.git.asml.silence@gmail.com> X-Mailer: git-send-email 2.33.1 In-Reply-To: References: MIME-Version: 1.0 Precedence: bulk List-ID: X-Mailing-List: linux-block@vger.kernel.org If we know that a iocb is async we can optimise bio_set_polled() a bit, add a new helper bio_set_polled_async(). Signed-off-by: Pavel Begunkov --- block/fops.c | 7 +++---- 1 file changed, 3 insertions(+), 4 deletions(-) diff --git a/block/fops.c b/block/fops.c index 8594852bd344..a2f492e50782 100644 --- a/block/fops.c +++ b/block/fops.c @@ -358,14 +358,13 @@ static ssize_t __blkdev_direct_IO_async(struct kiocb *iocb, task_io_account_write(bio->bi_iter.bi_size); } - if (iocb->ki_flags & IOCB_NOWAIT) - bio->bi_opf |= REQ_NOWAIT; - if (iocb->ki_flags & IOCB_HIPRI) { - bio_set_polled(bio, iocb); + bio->bi_opf |= REQ_POLLED | REQ_NOWAIT; submit_bio(bio); WRITE_ONCE(iocb->private, bio); } else { + if (iocb->ki_flags & IOCB_NOWAIT) + bio->bi_opf |= REQ_NOWAIT; submit_bio(bio); } return -EIOCBQUEUED;