From patchwork Wed Dec 15 16:30:07 2021 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Jens Axboe X-Patchwork-Id: 12678735 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from vger.kernel.org (vger.kernel.org [23.128.96.18]) by smtp.lore.kernel.org (Postfix) with ESMTP id 025E2C433FE for ; Wed, 15 Dec 2021 16:30:14 +0000 (UTC) Received: (majordomo@vger.kernel.org) by vger.kernel.org via listexpand id S244820AbhLOQaO (ORCPT ); Wed, 15 Dec 2021 11:30:14 -0500 Received: from lindbergh.monkeyblade.net ([23.128.96.19]:50916 "EHLO lindbergh.monkeyblade.net" rhost-flags-OK-OK-OK-OK) by vger.kernel.org with ESMTP id S244814AbhLOQaN (ORCPT ); Wed, 15 Dec 2021 11:30:13 -0500 Received: from mail-io1-xd2e.google.com (mail-io1-xd2e.google.com [IPv6:2607:f8b0:4864:20::d2e]) by lindbergh.monkeyblade.net (Postfix) with ESMTPS id 20E31C061574 for ; Wed, 15 Dec 2021 08:30:13 -0800 (PST) Received: by mail-io1-xd2e.google.com with SMTP id p65so31154063iof.3 for ; Wed, 15 Dec 2021 08:30:13 -0800 (PST) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=kernel-dk.20210112.gappssmtp.com; s=20210112; h=from:to:cc:subject:date:message-id:in-reply-to:references :mime-version:content-transfer-encoding; bh=b7apqfi26aowPUwnOpzYHc1ZM0VpWPqRuNaIQ/bur6s=; b=M7fxNqT76tXtnlBF9BZ+hX0bQpwmKUUAYMyyhfSQuPH2vQx87Kk4wHbAwwQIwiVxaw bS6mo6SrIM9PzRP6J94eN8oBlRrScY5x1i3KdB4wfJhHgc9V1vZEzSN1WMOEgc0DUe8x q1WNRrnPaN9HI/s9cYVYlmd9BbTWrPRCudgAkVpsON3i2dUnkP2PSWFlaVA8CpmuBbuJ LSWVi4qM2Q+XfMp9vsz6mbF3eqB+4WW3pXnPomyUh8AB4rK1WR9qG432wA2msx3DK5+J WsUFoKqQV0QUzIIe3WkNGD79cLOhHfU8qfw9AZpFWNFmTu2uA1SGHFN9JKVmhN+1H5Su pIcw== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20210112; h=x-gm-message-state:from:to:cc:subject:date:message-id:in-reply-to :references:mime-version:content-transfer-encoding; bh=b7apqfi26aowPUwnOpzYHc1ZM0VpWPqRuNaIQ/bur6s=; b=qjD9X/1A6EVaA/1XOrV7cf9YKJ4s+sZotgT7ZCpqg1EgRkuP/h9hA1X9bWY206X7Gp xkabzrdS4qldpE+vxFFR2TgVcNvCBPiLYYSkbdCjOOCpntAfn5gr1ngsj2j92uMnZ+/7 7VOjB6y7h8n1b58uTqnJpAeFhv8Mgi7oO6cBRDEOuKDkUw3LxMIsslnq+laCKVzz4tX5 aMdSg0LZCmPV8W2YqJOhAdtpC5N7Ds7keoO8VuVaeO4Vct34Ni5nejXHrqwnyQHK6f86 k43orIJ/Dvpc1CEO1VYujl6p7yvpOuv+Y4gAnqKX9GVGpwLzgvSco+sf76Xw/z1wgdhk z2Vw== X-Gm-Message-State: AOAM531nkUJ/NxMmxXXNGMzUyegguIYCwPNo/RM0IfTrZLJzx3l5xkz3 J86MpvplV8rRvIDKECA0F0Oa1iAJ0hbpfQ== X-Google-Smtp-Source: ABdhPJzECfbN3WUtuMm815oEq2LR3vcHNBbqikX5HpWMbeKbddSzGiHReUS5RkbHtkiWcmMDpgdVbw== X-Received: by 2002:a05:6638:2720:: with SMTP id m32mr6313805jav.298.1639585812467; Wed, 15 Dec 2021 08:30:12 -0800 (PST) Received: from x1.localdomain ([207.135.234.126]) by smtp.gmail.com with ESMTPSA id d12sm1338528ilg.85.2021.12.15.08.30.11 (version=TLS1_3 cipher=TLS_AES_256_GCM_SHA384 bits=256/256); Wed, 15 Dec 2021 08:30:11 -0800 (PST) From: Jens Axboe To: io-uring@vger.kernel.org, linux-block@vger.kernel.org Cc: Jens Axboe Subject: [PATCH 1/3] block: add completion handler for fast path Date: Wed, 15 Dec 2021 09:30:07 -0700 Message-Id: <20211215163009.15269-2-axboe@kernel.dk> X-Mailer: git-send-email 2.34.1 In-Reply-To: <20211215163009.15269-1-axboe@kernel.dk> References: <20211215163009.15269-1-axboe@kernel.dk> MIME-Version: 1.0 Precedence: bulk List-ID: X-Mailing-List: linux-block@vger.kernel.org The batched completions only deal with non-partial requests anyway, and it doesn't deal with any requests that have errors. Add a completion handler that assumes it's a full request and that it's all being ended successfully. Signed-off-by: Jens Axboe Reviewed-by: Christoph Hellwig --- block/blk-mq.c | 43 ++++++++++++++++++++++++++++++++++++++++++- 1 file changed, 42 insertions(+), 1 deletion(-) diff --git a/block/blk-mq.c b/block/blk-mq.c index f24394cb2004..efa9b93b4ddb 100644 --- a/block/blk-mq.c +++ b/block/blk-mq.c @@ -709,6 +709,47 @@ static void blk_print_req_error(struct request *req, blk_status_t status) IOPRIO_PRIO_CLASS(req->ioprio)); } +/* + * Fully end IO on a request. Does not support partial completions, or + * errors. + */ +static void blk_complete_request(struct request *req) +{ + const bool is_flush = (req->rq_flags & RQF_FLUSH_SEQ) != 0; + int total_bytes = blk_rq_bytes(req); + struct bio *bio = req->bio; + + trace_block_rq_complete(req, BLK_STS_OK, total_bytes); + + if (!bio) + return; + +#ifdef CONFIG_BLK_DEV_INTEGRITY + if (blk_integrity_rq(req) && req_op(req) == REQ_OP_READ) + req->q->integrity.profile->complete_fn(req, total_bytes); +#endif + + blk_account_io_completion(req, total_bytes); + + do { + struct bio *next = bio->bi_next; + + /* Completion has already been traced */ + bio_clear_flag(bio, BIO_TRACE_COMPLETION); + if (!is_flush) + bio_endio(bio); + bio = next; + } while (bio); + + /* + * Reset counters so that the request stacking driver + * can find how many bytes remain in the request + * later. + */ + req->bio = NULL; + req->__data_len = 0; +} + /** * blk_update_request - Complete multiple bytes without completing the request * @req: the request being processed @@ -922,7 +963,7 @@ void blk_mq_end_request_batch(struct io_comp_batch *iob) prefetch(rq->bio); prefetch(rq->rq_next); - blk_update_request(rq, BLK_STS_OK, blk_rq_bytes(rq)); + blk_complete_request(rq); if (iob->need_ts) __blk_mq_end_request_acct(rq, now); From patchwork Wed Dec 15 16:30:08 2021 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Jens Axboe X-Patchwork-Id: 12678737 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from vger.kernel.org (vger.kernel.org [23.128.96.18]) by smtp.lore.kernel.org (Postfix) with ESMTP id 963E1C43219 for ; Wed, 15 Dec 2021 16:30:15 +0000 (UTC) Received: (majordomo@vger.kernel.org) by vger.kernel.org via listexpand id S244815AbhLOQaO (ORCPT ); Wed, 15 Dec 2021 11:30:14 -0500 Received: from lindbergh.monkeyblade.net ([23.128.96.19]:50924 "EHLO lindbergh.monkeyblade.net" rhost-flags-OK-OK-OK-OK) by vger.kernel.org with ESMTP id S244821AbhLOQaO (ORCPT ); Wed, 15 Dec 2021 11:30:14 -0500 Received: from mail-io1-xd2d.google.com (mail-io1-xd2d.google.com [IPv6:2607:f8b0:4864:20::d2d]) by lindbergh.monkeyblade.net (Postfix) with ESMTPS id D1508C061574 for ; Wed, 15 Dec 2021 08:30:13 -0800 (PST) Received: by mail-io1-xd2d.google.com with SMTP id x6so30857987iol.13 for ; Wed, 15 Dec 2021 08:30:13 -0800 (PST) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=kernel-dk.20210112.gappssmtp.com; s=20210112; h=from:to:cc:subject:date:message-id:in-reply-to:references :mime-version:content-transfer-encoding; bh=3EhyZ30nQFeKraMAxV9eYGygtfaD2lNomjgGVObfnqo=; b=i1Rievg6JWKC/XjQucj2spa/b4JmJfps1+Q+3ILsR7qU7AGYqySbH9npuPjw3rQFe8 RmJuZwZvosMMYwQeHraYsbJ7zGzpToSmPsvnVSu7cpkmoDUt5YRCzvtnmGg9W7uRrvd5 2ije8/BAvqD7Ehtsd2rPWpZteDnlFsgm1cn0Un3OhukgbCnFU+xaxHMxVNq7YxJSJ2BD Tu/j/ifw+SqTPC815zZQU3d9abtkYeAniiOUUPVcTqs+VfYcz6i+tR+v2XInKGA85Yfz h7r4zPGUd9eXkV010TRC5sil2HRSdH2XoEkZoMT/kvssCOzA+lC1T+ms78lj0lOx2WJs xsUA== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20210112; h=x-gm-message-state:from:to:cc:subject:date:message-id:in-reply-to :references:mime-version:content-transfer-encoding; bh=3EhyZ30nQFeKraMAxV9eYGygtfaD2lNomjgGVObfnqo=; b=Pyzwo6MFCqEmCjbTi7sFPzhiegar7eB24J2BAoUcE4HgreFRyJYOp5tU4P4qgVgLkE tqVLRfBRI845JwMQbKxiO0MySyWibAk++mbjsfu230R5/bnoWnZEmmSeUaim4BK3NC2r c3dCya+XvZpIGiT+NWkWG2PzzdeJru6YjrRmccJXVnbr0lgVr8ouM3E3VXhb6DFroyqV oNtlfUZe3OIegjop2ntQuG1UhzAKdq4A04wa6N6lXEixWa1XOeA79t+WqTahlBwlaFDB a0LntLgIWRSLx79LGffwWj6JJnHPaOIy2ZJf4VeaU/AO20F+Ufhmfcorh/46mdHEU/BN J8NQ== X-Gm-Message-State: AOAM533zFikDFF4RYKTCfj1DYUH7kfHJFTH+lTkQSAsMpujZf+DIhpxV dnCSmfBIpRBTe635V4Rc76j/TZpbsM+Vcg== X-Google-Smtp-Source: ABdhPJx6SLR+dfLopaZsnwX17G5Pf4gk6MZ5Jpuo95ukiTN4MAqDiTeYKZA7t07zW6NYRE91YZbNLg== X-Received: by 2002:a05:6602:2dd0:: with SMTP id l16mr7101823iow.198.1639585813187; Wed, 15 Dec 2021 08:30:13 -0800 (PST) Received: from x1.localdomain ([207.135.234.126]) by smtp.gmail.com with ESMTPSA id d12sm1338528ilg.85.2021.12.15.08.30.12 (version=TLS1_3 cipher=TLS_AES_256_GCM_SHA384 bits=256/256); Wed, 15 Dec 2021 08:30:12 -0800 (PST) From: Jens Axboe To: io-uring@vger.kernel.org, linux-block@vger.kernel.org Cc: Jens Axboe Subject: [PATCH 2/3] block: use singly linked list for bio cache Date: Wed, 15 Dec 2021 09:30:08 -0700 Message-Id: <20211215163009.15269-3-axboe@kernel.dk> X-Mailer: git-send-email 2.34.1 In-Reply-To: <20211215163009.15269-1-axboe@kernel.dk> References: <20211215163009.15269-1-axboe@kernel.dk> MIME-Version: 1.0 Precedence: bulk List-ID: X-Mailing-List: linux-block@vger.kernel.org Pointless to maintain a head/tail for the list, as we never need to access the tail. Entries are always LIFO for cache hotness reasons. Signed-off-by: Jens Axboe Reviewed-by: Christoph Hellwig --- block/bio.c | 14 +++++++++----- 1 file changed, 9 insertions(+), 5 deletions(-) diff --git a/block/bio.c b/block/bio.c index d9d8e1143edc..a76a3134625a 100644 --- a/block/bio.c +++ b/block/bio.c @@ -26,7 +26,7 @@ #include "blk-rq-qos.h" struct bio_alloc_cache { - struct bio_list free_list; + struct bio *free_list; unsigned int nr; }; @@ -630,7 +630,9 @@ static void bio_alloc_cache_prune(struct bio_alloc_cache *cache, unsigned int i = 0; struct bio *bio; - while ((bio = bio_list_pop(&cache->free_list)) != NULL) { + while (cache->free_list) { + bio = cache->free_list; + cache->free_list = bio->bi_next; cache->nr--; bio_free(bio); if (++i == nr) @@ -689,7 +691,8 @@ void bio_put(struct bio *bio) bio_uninit(bio); cache = per_cpu_ptr(bio->bi_pool->cache, get_cpu()); - bio_list_add_head(&cache->free_list, bio); + bio->bi_next = cache->free_list; + cache->free_list = bio; if (++cache->nr > ALLOC_CACHE_MAX + ALLOC_CACHE_SLACK) bio_alloc_cache_prune(cache, ALLOC_CACHE_SLACK); put_cpu(); @@ -1700,8 +1703,9 @@ struct bio *bio_alloc_kiocb(struct kiocb *kiocb, unsigned short nr_vecs, return bio_alloc_bioset(GFP_KERNEL, nr_vecs, bs); cache = per_cpu_ptr(bs->cache, get_cpu()); - bio = bio_list_pop(&cache->free_list); - if (bio) { + if (cache->free_list) { + bio = cache->free_list; + cache->free_list = bio->bi_next; cache->nr--; put_cpu(); bio_init(bio, nr_vecs ? bio->bi_inline_vecs : NULL, nr_vecs); From patchwork Wed Dec 15 16:30:09 2021 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Jens Axboe X-Patchwork-Id: 12678739 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from vger.kernel.org (vger.kernel.org [23.128.96.18]) by smtp.lore.kernel.org (Postfix) with ESMTP id 3992EC4321E for ; Wed, 15 Dec 2021 16:30:16 +0000 (UTC) Received: (majordomo@vger.kernel.org) by vger.kernel.org via listexpand id S244812AbhLOQaP (ORCPT ); Wed, 15 Dec 2021 11:30:15 -0500 Received: from lindbergh.monkeyblade.net ([23.128.96.19]:50930 "EHLO lindbergh.monkeyblade.net" rhost-flags-OK-OK-OK-OK) by vger.kernel.org with ESMTP id S244823AbhLOQaO (ORCPT ); Wed, 15 Dec 2021 11:30:14 -0500 Received: from mail-io1-xd30.google.com (mail-io1-xd30.google.com [IPv6:2607:f8b0:4864:20::d30]) by lindbergh.monkeyblade.net (Postfix) with ESMTPS id A5513C061574 for ; Wed, 15 Dec 2021 08:30:14 -0800 (PST) Received: by mail-io1-xd30.google.com with SMTP id y16so31098279ioc.8 for ; Wed, 15 Dec 2021 08:30:14 -0800 (PST) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=kernel-dk.20210112.gappssmtp.com; s=20210112; h=from:to:cc:subject:date:message-id:in-reply-to:references :mime-version:content-transfer-encoding; bh=HW4tADXnsX1kVJEjzQB5GXTaWIqCwvQGXd7qCGVjzhI=; b=O85KNpfujxQLOdxylblhrkc8zk3n1koN9OsNgJlPq7LevN961C02fpa+6XpLLIW3hC y3dVMn81+0K4ZXpGOhC1XMbq5z+qb44VVhOJQRiZbl48MsH88hc+lV8lzxaFW984eMub W9FgnZpQjouEBrSZd0fJnCzHzcNXu50LYM5WTUYbu3GCghIJzvNXT+hXvHKZNxX2c29r uVszepdFCY/llkmKkskC77t7W62qBChEqQPpYC2djcrq2805dhWILmPsJyBoZOPU/7Ih xWlfDSOurSkunSbdizmeIBEpdy2qh8XrJ5fEgl3WnPp7yEeiaDNFTrEsO99cOXDJS2GQ tiNA== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20210112; h=x-gm-message-state:from:to:cc:subject:date:message-id:in-reply-to :references:mime-version:content-transfer-encoding; bh=HW4tADXnsX1kVJEjzQB5GXTaWIqCwvQGXd7qCGVjzhI=; b=aPLef21QukMdyK7GUgCNGaHOP8u0h12wV3u+zqSYT9fBQtxIDDVWejxSOYHDzQbBPZ 2bddMnb+PU9tfACaTWSrrcW9Zh8tsTZQrbkPSDrOINVCtFvMamXiEASXUbKxseP0aiZ0 /tLGWzd84GlPIEHHhwd/PtpGP8Xjy7zQnv1NW1Sg8EiFrvCCavbtkkP3ibDsJ3POffVe QFMqWFDq6OM7rZi/AIWn6lCsgVId+RCDYYYoVXGYWx4BeA/q+6cucgP68+cbD95QGgF+ cNoI/y1i5PGBtyTCA8yDkNty53Xha8vFzPHDj8ow77+2w1UcOkvKWkdrvnK81h1cbC9v W2dg== X-Gm-Message-State: AOAM532tGXhFc7KHbQlLq6ItgDvxfFUdBZdsUM4hwus7HeckGOwWUlF7 3vgea3mYAs5NGolNnP4xS67bzxeO39MPPw== X-Google-Smtp-Source: ABdhPJzyYNhwu8xstNuEkNMl45gwmrENShy1czSS1j1NmthJoo05l66107FZ5MPIvwfc4RnA9vXMeA== X-Received: by 2002:a05:6638:1408:: with SMTP id k8mr7045204jad.301.1639585814032; Wed, 15 Dec 2021 08:30:14 -0800 (PST) Received: from x1.localdomain ([207.135.234.126]) by smtp.gmail.com with ESMTPSA id d12sm1338528ilg.85.2021.12.15.08.30.13 (version=TLS1_3 cipher=TLS_AES_256_GCM_SHA384 bits=256/256); Wed, 15 Dec 2021 08:30:13 -0800 (PST) From: Jens Axboe To: io-uring@vger.kernel.org, linux-block@vger.kernel.org Cc: Jens Axboe Subject: [PATCH 3/3] block: enable bio allocation cache for IRQ driven IO Date: Wed, 15 Dec 2021 09:30:09 -0700 Message-Id: <20211215163009.15269-4-axboe@kernel.dk> X-Mailer: git-send-email 2.34.1 In-Reply-To: <20211215163009.15269-1-axboe@kernel.dk> References: <20211215163009.15269-1-axboe@kernel.dk> MIME-Version: 1.0 Precedence: bulk List-ID: X-Mailing-List: linux-block@vger.kernel.org We currently cannot use the bio recycling allocation cache for IRQ driven IO, as the cache isn't IRQ safe (by design). Add a way for the completion side to pass back a bio that needs freeing, so we can do it from the io_uring side. io_uring completions always run in task context. This is good for about a 13% improvement in IRQ driven IO, taking us from around 6.3M/core to 7.1M/core IOPS. Signed-off-by: Jens Axboe --- block/fops.c | 11 ++++++++--- fs/io_uring.c | 6 ++++++ include/linux/fs.h | 4 ++++ 3 files changed, 18 insertions(+), 3 deletions(-) diff --git a/block/fops.c b/block/fops.c index bcf866b07edc..c7794d42be85 100644 --- a/block/fops.c +++ b/block/fops.c @@ -296,14 +296,19 @@ static void blkdev_bio_end_io_async(struct bio *bio) ret = blk_status_to_errno(bio->bi_status); } - iocb->ki_complete(iocb, ret); - if (dio->flags & DIO_SHOULD_DIRTY) { bio_check_pages_dirty(bio); } else { bio_release_pages(bio, false); - bio_put(bio); + if (iocb->ki_flags & IOCB_BIO_PASSBACK) { + iocb->ki_flags |= IOCB_PRIV_IS_BIO; + iocb->private = bio; + } else { + bio_put(bio); + } } + + iocb->ki_complete(iocb, ret); } static ssize_t __blkdev_direct_IO_async(struct kiocb *iocb, diff --git a/fs/io_uring.c b/fs/io_uring.c index 1a647a6a5add..b0302c0407e6 100644 --- a/fs/io_uring.c +++ b/fs/io_uring.c @@ -2828,6 +2828,11 @@ static void io_req_task_complete(struct io_kiocb *req, bool *locked) unsigned int cflags = io_put_kbuf(req); int res = req->result; +#ifdef CONFIG_BLOCK + if (req->rw.kiocb.ki_flags & IOCB_PRIV_IS_BIO) + bio_put(req->rw.kiocb.private); +#endif + if (*locked) { io_req_complete_state(req, res, cflags); io_req_add_compl_list(req); @@ -3024,6 +3029,7 @@ static int io_prep_rw(struct io_kiocb *req, const struct io_uring_sqe *sqe) } else { if (kiocb->ki_flags & IOCB_HIPRI) return -EINVAL; + kiocb->ki_flags |= IOCB_ALLOC_CACHE | IOCB_BIO_PASSBACK; kiocb->ki_complete = io_complete_rw; } diff --git a/include/linux/fs.h b/include/linux/fs.h index 6b8dc1a78df6..bf7a76dfdc29 100644 --- a/include/linux/fs.h +++ b/include/linux/fs.h @@ -322,6 +322,10 @@ enum rw_hint { #define IOCB_NOIO (1 << 20) /* can use bio alloc cache */ #define IOCB_ALLOC_CACHE (1 << 21) +/* iocb supports bio passback */ +#define IOCB_BIO_PASSBACK (1 << 22) +/* iocb->private holds bio to put */ +#define IOCB_PRIV_IS_BIO (1 << 23) struct kiocb { struct file *ki_filp;