From patchwork Tue Oct 19 21:24:10 2021 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Pavel Begunkov X-Patchwork-Id: 12571029 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from mail.kernel.org (mail.kernel.org [198.145.29.99]) by smtp.lore.kernel.org (Postfix) with ESMTP id E1462C433FE for ; Tue, 19 Oct 2021 21:24:36 +0000 (UTC) Received: from vger.kernel.org (vger.kernel.org [23.128.96.18]) by mail.kernel.org (Postfix) with ESMTP id C307861355 for ; Tue, 19 Oct 2021 21:24:36 +0000 (UTC) Received: (majordomo@vger.kernel.org) by vger.kernel.org via listexpand id S229519AbhJSV0t (ORCPT ); Tue, 19 Oct 2021 17:26:49 -0400 Received: from lindbergh.monkeyblade.net ([23.128.96.19]:46964 "EHLO lindbergh.monkeyblade.net" rhost-flags-OK-OK-OK-OK) by vger.kernel.org with ESMTP id S229483AbhJSV0s (ORCPT ); Tue, 19 Oct 2021 17:26:48 -0400 Received: from mail-wm1-x329.google.com (mail-wm1-x329.google.com [IPv6:2a00:1450:4864:20::329]) by lindbergh.monkeyblade.net (Postfix) with ESMTPS id 345B2C06161C for ; Tue, 19 Oct 2021 14:24:35 -0700 (PDT) Received: by mail-wm1-x329.google.com with SMTP id g2so14544803wme.4 for ; Tue, 19 Oct 2021 14:24:35 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=20210112; h=from:to:cc:subject:date:message-id:in-reply-to:references :mime-version:content-transfer-encoding; bh=eqByaVwNH1eDBgQxrWe4ZD3BlTfVAS6fttIbCtAyDOI=; b=pIUX/q7WhHPLBIVr/tMHIGuHsyHHwYs/7Mon24mXi6hc/vTdvM1INgZ2wVbY4DCLZX BUZuGmHjn6r2D8ginDbyM7T7vMGVnmWc8RY+itLzOUgERJmppvIDuA6Dyt5vKDHkpAfq 4MFf7fsHCzhzh7dV3FYNaw1VdWpjh/d8Hn23u2j3Zs6UFi9EhCQM4YN49hVLary+iziS s9T9kP/21AOw8zUNHi8VX4DxzDt6IRALGKWpIeEpdmwJami6pPzFqxjeDadbw/B1+qPo 3I7duSi+euPj5dgMweTzNcYAu/Bixv/Xx8S/mC/t93oY+whXiGktu56o5d9Gw/MR886G 6L0w== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20210112; h=x-gm-message-state:from:to:cc:subject:date:message-id:in-reply-to :references:mime-version:content-transfer-encoding; bh=eqByaVwNH1eDBgQxrWe4ZD3BlTfVAS6fttIbCtAyDOI=; b=14StiLRRbMI0vVRAmUN9vZmBtplE7f1EG7UViodU5qSg0dR1MASUCO0jYtOCYIDV/t wN2ZVcYMWdA3OdZahdTAq57N8XEc22z74QOFYnyOFAKsLlDdWU1plpsAIrzLq5pmodY6 3v2pnprBFVaqi0hPR35vNqWC0YXpu0zR5aKDQjXeNjUE32f3rrOdwYIiuCQwhbChAO7c x2flIqd+tJraCvFJYEU+9Do/FoyvSSzI68DnKOqp5H4vK++Uvm2jVhZ5hOVuCYx4sg9t TGp3TIXn5wloRYw7CqWZq++XXwLCllzZmu8sNbByt7UF/bSNzqPo+VIYE/j5wqyuEOOI Ss4w== X-Gm-Message-State: AOAM533XCeCPQTi/hEddRyS6WIioZ0Zx+akDTrXyWy9bVDqCOibewjrr hMlU7waVLqNX4luraEK1GQHR8bnkW2Roxg== X-Google-Smtp-Source: ABdhPJyR5+Ls5k9SMipR1qjzRAYTS9yAQr3bSKFPGNj1gknME7g7J81CJh/5hCRwW5Wld3DTdEtjqQ== X-Received: by 2002:a1c:2246:: with SMTP id i67mr9185501wmi.72.1634678673622; Tue, 19 Oct 2021 14:24:33 -0700 (PDT) Received: from 127.0.0.1localhost ([185.69.145.194]) by smtp.gmail.com with ESMTPSA id m14sm216020wms.25.2021.10.19.14.24.32 (version=TLS1_3 cipher=TLS_AES_256_GCM_SHA384 bits=256/256); Tue, 19 Oct 2021 14:24:33 -0700 (PDT) From: Pavel Begunkov To: linux-block@vger.kernel.org Cc: Jens Axboe , Pavel Begunkov Subject: [PATCH 01/16] block: turn macro helpers into inline functions Date: Tue, 19 Oct 2021 22:24:10 +0100 Message-Id: X-Mailer: git-send-email 2.33.1 In-Reply-To: References: MIME-Version: 1.0 Precedence: bulk List-ID: X-Mailing-List: linux-block@vger.kernel.org Replace bio_set_dev() with an identical inline helper and move it further to fix a dependency problem with bio_associate_blkg(). Do the same for bio_copy_dev(). Signed-off-by: Pavel Begunkov Reviewed-by: Christoph Hellwig --- include/linux/bio.h | 32 ++++++++++++++++---------------- 1 file changed, 16 insertions(+), 16 deletions(-) diff --git a/include/linux/bio.h b/include/linux/bio.h index 9538f20ffaa5..b12453d7b8a8 100644 --- a/include/linux/bio.h +++ b/include/linux/bio.h @@ -430,22 +430,6 @@ void zero_fill_bio(struct bio *bio); extern const char *bio_devname(struct bio *bio, char *buffer); -#define bio_set_dev(bio, bdev) \ -do { \ - bio_clear_flag(bio, BIO_REMAPPED); \ - if ((bio)->bi_bdev != (bdev)) \ - bio_clear_flag(bio, BIO_THROTTLED); \ - (bio)->bi_bdev = (bdev); \ - bio_associate_blkg(bio); \ -} while (0) - -#define bio_copy_dev(dst, src) \ -do { \ - bio_clear_flag(dst, BIO_REMAPPED); \ - (dst)->bi_bdev = (src)->bi_bdev; \ - bio_clone_blkg_association(dst, src); \ -} while (0) - #define bio_dev(bio) \ disk_devt((bio)->bi_bdev->bd_disk) @@ -463,6 +447,22 @@ static inline void bio_clone_blkg_association(struct bio *dst, struct bio *src) { } #endif /* CONFIG_BLK_CGROUP */ +static inline void bio_set_dev(struct bio *bio, struct block_device *bdev) +{ + bio_clear_flag(bio, BIO_REMAPPED); + if (bio->bi_bdev != bdev) + bio_clear_flag(bio, BIO_THROTTLED); + bio->bi_bdev = bdev; + bio_associate_blkg(bio); +} + +static inline void bio_copy_dev(struct bio *dst, struct bio *src) +{ + bio_clear_flag(dst, BIO_REMAPPED); + dst->bi_bdev = src->bi_bdev; + bio_clone_blkg_association(dst, src); +} + /* * BIO list management for use by remapping drivers (e.g. DM or MD) and loop. * From patchwork Tue Oct 19 21:24:11 2021 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Pavel Begunkov X-Patchwork-Id: 12571031 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from mail.kernel.org (mail.kernel.org [198.145.29.99]) by smtp.lore.kernel.org (Postfix) with ESMTP id B1697C433EF for ; Tue, 19 Oct 2021 21:24:37 +0000 (UTC) Received: from vger.kernel.org (vger.kernel.org [23.128.96.18]) by mail.kernel.org (Postfix) with ESMTP id 99B4E61355 for ; Tue, 19 Oct 2021 21:24:37 +0000 (UTC) Received: (majordomo@vger.kernel.org) by vger.kernel.org via listexpand id S229521AbhJSV0t (ORCPT ); Tue, 19 Oct 2021 17:26:49 -0400 Received: from lindbergh.monkeyblade.net ([23.128.96.19]:46970 "EHLO lindbergh.monkeyblade.net" rhost-flags-OK-OK-OK-OK) by vger.kernel.org with ESMTP id S229483AbhJSV0t (ORCPT ); Tue, 19 Oct 2021 17:26:49 -0400 Received: from mail-wm1-x32b.google.com (mail-wm1-x32b.google.com [IPv6:2a00:1450:4864:20::32b]) by lindbergh.monkeyblade.net (Postfix) with ESMTPS id 09DC4C06161C for ; Tue, 19 Oct 2021 14:24:36 -0700 (PDT) Received: by mail-wm1-x32b.google.com with SMTP id z77-20020a1c7e50000000b0030db7b70b6bso5679703wmc.1 for ; Tue, 19 Oct 2021 14:24:35 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=20210112; h=from:to:cc:subject:date:message-id:in-reply-to:references :mime-version:content-transfer-encoding; bh=fKkwHTUtYVuEoE7mZDvi41g9g5mXV7xtQqtX4zIAEZI=; b=FAry/GyM2e0kOTlythIvfEzRGo5aBPHNRgviKt1Yq0XNem8hxmIQ8VI31HcPsdrX5L Cufw6u+VsIAdKQEsYy85tehnAi/kofU06V08AKEqFUIB83FXXF1f5gzoYuzm02+k0WWl 5wiqJb/VAtVxnMcMNTk1NrcmMWfuBllbU65LT+VekelDzwAeGE8chXdW54Dk6muqCbwM 7F5VOzE5OpBx1/ApfEKR84qfpTorw+8LyjZJOq96qYP7Snrjeu6TtgFF1IYhV8EnltPU BCipQgskPHuRnjETbJH1DX+Qg5ioen9xjTeSB3I0ih7FB6kHv7CsrrhLHS+Bm1q+cV0p 9o+A== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20210112; h=x-gm-message-state:from:to:cc:subject:date:message-id:in-reply-to :references:mime-version:content-transfer-encoding; bh=fKkwHTUtYVuEoE7mZDvi41g9g5mXV7xtQqtX4zIAEZI=; b=y8zdPhFojS1c9XLPba9+5Hk6/idS4Qo16+cX7DdEXlNXaOTjACe1waj+6YfogBXm2M ogsMAe9gIkBxwmtMay+kTcInGRwMHZDra2PlgprCO3Lv1dGRzYnv3Qjx0Wgw/yEIgihu zl9p10BElK+v2uoiwtS+U53whFXlQe5cVmXH1kYpyMbqLDYJVc/1GC5+nMPl/tnMGesf KEadpwhj8bYK/bcmJGRcgKeXu0wGkshQYtV7PG8Z0C75/x5pMhUAiSkYi7CIuvVEWeeq pY0Ox40qW5l64cuZSB69ye3u1KgYpRERmlx8meQUeAZSZk7xPXknQeQNCZPWGFrZA2TQ B0eg== X-Gm-Message-State: AOAM532MSNd9ktVWR0O4yNWw3vSa0CgmiZKemefBY49DWjuyTI9qXTB4 EyHF0NPgX+aiAYMXqvu/6PLvECfGz4KIAA== X-Google-Smtp-Source: ABdhPJzviAd2Y25S1F7DtSVWZYvZmDsOz+2UX1aNvqQ7SOEMvGZ9+Ietqsv1UKsnYWLE6U4/ZsbDJQ== X-Received: by 2002:a05:600c:1912:: with SMTP id j18mr4584513wmq.117.1634678674446; Tue, 19 Oct 2021 14:24:34 -0700 (PDT) Received: from 127.0.0.1localhost ([185.69.145.194]) by smtp.gmail.com with ESMTPSA id m14sm216020wms.25.2021.10.19.14.24.33 (version=TLS1_3 cipher=TLS_AES_256_GCM_SHA384 bits=256/256); Tue, 19 Oct 2021 14:24:34 -0700 (PDT) From: Pavel Begunkov To: linux-block@vger.kernel.org Cc: Jens Axboe , Pavel Begunkov Subject: [PATCH 02/16] block: convert leftovers to bdev_get_queue Date: Tue, 19 Oct 2021 22:24:11 +0100 Message-Id: <654f1cba1fe9c321cb87943ee33a21c7ea3d8e65.1634676157.git.asml.silence@gmail.com> X-Mailer: git-send-email 2.33.1 In-Reply-To: References: MIME-Version: 1.0 Precedence: bulk List-ID: X-Mailing-List: linux-block@vger.kernel.org Convert bdev->bd_disk->queue to bdev_get_queue(), which is faster. Apparently, there are a few such spots in block that got lost during rebases. Signed-off-by: Pavel Begunkov Reviewed-by: Chaitanya Kulkarni --- block/blk-core.c | 2 +- block/blk-merge.c | 2 +- 2 files changed, 2 insertions(+), 2 deletions(-) diff --git a/block/blk-core.c b/block/blk-core.c index e6ad5b51d0c3..c1ba34777c6d 100644 --- a/block/blk-core.c +++ b/block/blk-core.c @@ -1080,7 +1080,7 @@ EXPORT_SYMBOL(submit_bio); */ int bio_poll(struct bio *bio, struct io_comp_batch *iob, unsigned int flags) { - struct request_queue *q = bio->bi_bdev->bd_disk->queue; + struct request_queue *q = bdev_get_queue(bio->bi_bdev); blk_qc_t cookie = READ_ONCE(bio->bi_cookie); int ret; diff --git a/block/blk-merge.c b/block/blk-merge.c index 3e6fa449caff..df69f4bb7717 100644 --- a/block/blk-merge.c +++ b/block/blk-merge.c @@ -383,7 +383,7 @@ void __blk_queue_split(struct request_queue *q, struct bio **bio, */ void blk_queue_split(struct bio **bio) { - struct request_queue *q = (*bio)->bi_bdev->bd_disk->queue; + struct request_queue *q = bdev_get_queue((*bio)->bi_bdev); unsigned int nr_segs; if (blk_may_split(q, *bio)) From patchwork Tue Oct 19 21:24:12 2021 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Pavel Begunkov X-Patchwork-Id: 12571033 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from mail.kernel.org (mail.kernel.org [198.145.29.99]) by smtp.lore.kernel.org (Postfix) with ESMTP id 125B5C43217 for ; Tue, 19 Oct 2021 21:24:39 +0000 (UTC) Received: from vger.kernel.org (vger.kernel.org [23.128.96.18]) by mail.kernel.org (Postfix) with ESMTP id EF25A60F46 for ; Tue, 19 Oct 2021 21:24:38 +0000 (UTC) Received: (majordomo@vger.kernel.org) by vger.kernel.org via listexpand id S229691AbhJSV0v (ORCPT ); Tue, 19 Oct 2021 17:26:51 -0400 Received: from lindbergh.monkeyblade.net ([23.128.96.19]:46982 "EHLO lindbergh.monkeyblade.net" rhost-flags-OK-OK-OK-OK) by vger.kernel.org with ESMTP id S229603AbhJSV0u (ORCPT ); Tue, 19 Oct 2021 17:26:50 -0400 Received: from mail-wm1-x336.google.com (mail-wm1-x336.google.com [IPv6:2a00:1450:4864:20::336]) by lindbergh.monkeyblade.net (Postfix) with ESMTPS id E28E2C06161C for ; Tue, 19 Oct 2021 14:24:36 -0700 (PDT) Received: by mail-wm1-x336.google.com with SMTP id o24so7117480wms.0 for ; Tue, 19 Oct 2021 14:24:36 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=20210112; h=from:to:cc:subject:date:message-id:in-reply-to:references :mime-version:content-transfer-encoding; bh=3jjrhInz/WqiiR8qYl275+vzOFnsIrXYK8DzELi3dlQ=; b=DT2vc44OEuKXkbYsltEQNdNRkVYPVatKlMLdY4zbJrHdG1EI2qEDFGKLzUjunmJpEB f79AFBWrYNm3yNt13p3/xoSqndq5cmwOiUsvCvayDIr8F+hQJZqW0BUvesupUoM/ivRu 75pn8PEjKTRsvFx6unl+lSFWdtsg/uNnBsTAIbTSSaU4lnSssz2IirAl5WQhCa7+N3PD l/JoCAx/eyOdhYOJ/mC7nqhBykf/51Eykvk7PoEmsaomf69tMmuFsCOhaD7afquLCxIw Xwv3+OUpLc3Z8IOIcsgvvN4jxIgSRsJkrpdZwuZcm04aU8GyeBCfJUs19MyQPp37gXh/ QraQ== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20210112; h=x-gm-message-state:from:to:cc:subject:date:message-id:in-reply-to :references:mime-version:content-transfer-encoding; bh=3jjrhInz/WqiiR8qYl275+vzOFnsIrXYK8DzELi3dlQ=; b=kuODyP/p8LnHbmadjfYIUoZYHVogMY8d1uTwJnAk8uMWrN/JZE/csO++LymaBC1Ymt MmdHMeps7HAzapplBBMWSm9jmT9ReajcF+GPsnmleHdJPwnFp84Xu0XDxjVwV78qOM0r Y+sGvl0T2wM7ie52niSl2ryBYsN5BJ0mQG7gheMhjf23LuNgF3yB+UJ5BTUNdg0f82Jl jafgOuJ6z76jxxKK9FzbhcgkR9CN0Dm569CCUVt7RNfmqRN9BLZoBHaRmzUfSg8DiEBq Jdz3APLxCh0RXwM9fEKhhL/oU0FKVG/8pJL2IOWEJBh7ZdU53TWzmrBRwaDHlsrEK17u tvPg== X-Gm-Message-State: AOAM532d/3F9yXLJucq9fr60S8EQn8zLQdm7tlTdPAyNI6t8VWgFVSw/ HTHqce9kZebY5ED+7O0igEdDldf9P5ML3A== X-Google-Smtp-Source: ABdhPJzC+fC2rwyVRzuti65Sk9EkYYqpF4wmhCOLRiyrwX8n1+0mp2wg9cPmWoyDfFaVcqUQS6rbsA== X-Received: by 2002:a05:600c:1d1a:: with SMTP id l26mr8817741wms.98.1634678675233; Tue, 19 Oct 2021 14:24:35 -0700 (PDT) Received: from 127.0.0.1localhost ([185.69.145.194]) by smtp.gmail.com with ESMTPSA id m14sm216020wms.25.2021.10.19.14.24.34 (version=TLS1_3 cipher=TLS_AES_256_GCM_SHA384 bits=256/256); Tue, 19 Oct 2021 14:24:34 -0700 (PDT) From: Pavel Begunkov To: linux-block@vger.kernel.org Cc: Jens Axboe , Pavel Begunkov Subject: [PATCH 03/16] block: optimise req_bio_endio() Date: Tue, 19 Oct 2021 22:24:12 +0100 Message-Id: <8061fa49096e1a0e44849aa204a0a1ae57c4423a.1634676157.git.asml.silence@gmail.com> X-Mailer: git-send-email 2.33.1 In-Reply-To: References: MIME-Version: 1.0 Precedence: bulk List-ID: X-Mailing-List: linux-block@vger.kernel.org First, get rid of an extra branch and chain error checks. Also reshuffle it with bio_advance(), so it goes closer to the final check, with that the compiler loads rq->rq_flags only once, and also doesn't reload bio->bi_iter.bi_size if bio_advance() didn't actually advanced the iter. Signed-off-by: Pavel Begunkov Reviewed-by: Christoph Hellwig --- block/blk-mq.c | 16 +++++++--------- 1 file changed, 7 insertions(+), 9 deletions(-) diff --git a/block/blk-mq.c b/block/blk-mq.c index 3481a8712234..bab1fccda6ca 100644 --- a/block/blk-mq.c +++ b/block/blk-mq.c @@ -617,25 +617,23 @@ void blk_mq_free_plug_rqs(struct blk_plug *plug) static void req_bio_endio(struct request *rq, struct bio *bio, unsigned int nbytes, blk_status_t error) { - if (error) + if (unlikely(error)) { bio->bi_status = error; - - if (unlikely(rq->rq_flags & RQF_QUIET)) - bio_set_flag(bio, BIO_QUIET); - - bio_advance(bio, nbytes); - - if (req_op(rq) == REQ_OP_ZONE_APPEND && error == BLK_STS_OK) { + } else if (req_op(rq) == REQ_OP_ZONE_APPEND) { /* * Partial zone append completions cannot be supported as the * BIO fragments may end up not being written sequentially. */ - if (bio->bi_iter.bi_size) + if (bio->bi_iter.bi_size == nbytes) bio->bi_status = BLK_STS_IOERR; else bio->bi_iter.bi_sector = rq->__sector; } + bio_advance(bio, nbytes); + + if (unlikely(rq->rq_flags & RQF_QUIET)) + bio_set_flag(bio, BIO_QUIET); /* don't actually finish bio if it's part of flush sequence */ if (bio->bi_iter.bi_size == 0 && !(rq->rq_flags & RQF_FLUSH_SEQ)) bio_endio(bio); From patchwork Tue Oct 19 21:24:13 2021 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Pavel Begunkov X-Patchwork-Id: 12571035 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from mail.kernel.org (mail.kernel.org [198.145.29.99]) by smtp.lore.kernel.org (Postfix) with ESMTP id D8987C433F5 for ; Tue, 19 Oct 2021 21:24:39 +0000 (UTC) Received: from vger.kernel.org (vger.kernel.org [23.128.96.18]) by mail.kernel.org (Postfix) with ESMTP id C368661355 for ; Tue, 19 Oct 2021 21:24:39 +0000 (UTC) Received: (majordomo@vger.kernel.org) by vger.kernel.org via listexpand id S229483AbhJSV0v (ORCPT ); Tue, 19 Oct 2021 17:26:51 -0400 Received: from lindbergh.monkeyblade.net ([23.128.96.19]:46986 "EHLO lindbergh.monkeyblade.net" rhost-flags-OK-OK-OK-OK) by vger.kernel.org with ESMTP id S229686AbhJSV0v (ORCPT ); Tue, 19 Oct 2021 17:26:51 -0400 Received: from mail-wm1-x332.google.com (mail-wm1-x332.google.com [IPv6:2a00:1450:4864:20::332]) by lindbergh.monkeyblade.net (Postfix) with ESMTPS id AC0CDC061746 for ; Tue, 19 Oct 2021 14:24:37 -0700 (PDT) Received: by mail-wm1-x332.google.com with SMTP id p21so14494355wmq.1 for ; Tue, 19 Oct 2021 14:24:37 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=20210112; h=from:to:cc:subject:date:message-id:in-reply-to:references :mime-version:content-transfer-encoding; bh=a+s1e1YWVl/iKcwmsT/qfM381a1sl+3GwTljlw0r+aQ=; b=gl71a5b9lIzPndqoWCOF9I/ihOVM9i/ZlmLPjO6oyG6hcOKuC5+HJUCJAwsFr0DIko WEUbKK9hRqzlhN0oCcVujNBxe3H/CZhk536J+kWbINmziYYGTud6Nw0dG8Gcc2Ov8P8B MaTVRU1BoDxh/qLataqs3+SKV6f5zt8gy06jQ4OGzXYvEijWG+8S/z4WCucVusiGWwfG XJ84kPGYxQbm+fsX/DuXKRqGPAYHrpY7IXmjgUxVF3n1e/kMwiRZRqmmCTtEl9ZEn6h/ ywDK6dL8RkjUl7xmLjcL+U6hoQBhql74p033vur85PrwV4jPBKE9baaiDNYYgDH+LZlv KKkA== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20210112; h=x-gm-message-state:from:to:cc:subject:date:message-id:in-reply-to :references:mime-version:content-transfer-encoding; bh=a+s1e1YWVl/iKcwmsT/qfM381a1sl+3GwTljlw0r+aQ=; b=psd4a/j4bGVhPbI7w2kwQIpNztl5LRZ4whGK6cmf9FYDiVQzEwvZ9dHE2Pe/UHyamS eV+/RyI5bxavZw1+Kmvdibe89X0BqC11ImXCkJ4dM8GDo2w3VT1A3vLK13waGSZvFlt9 d6wRDOKa1hyY+/d00CxaYFG7poelrLuzcXErZDkQdaPM4Oqs3BgBI+KhaHiW/Gv2bXYi 1NneGKGJjZ+wvURB7+jljLMen3T96LXxKm9apcB5tjIAh0locgdP9R1sUTEUy1UWJhu7 /AC1w86jvOkSAoCu2qSo3pczUGuIA4oe227cerC4NqHLErRsZ8KfGilDZHzkMYLtSWxE 48/w== X-Gm-Message-State: AOAM530moquR60CpA7kaCUknF08YZ6BeELZf5IeXAmlWsMa93Y2FIP5w CVv/jiz7qfnDAfAIdgEzvRD611oZR49WgA== X-Google-Smtp-Source: ABdhPJyn66wHvg3KF1ol/deW1sla//ZiNN0kIjkC1cZH1D/IxZbxAM6H5IxaXvWH4b4C1uRYA4nJMA== X-Received: by 2002:a1c:2785:: with SMTP id n127mr8661714wmn.155.1634678676074; Tue, 19 Oct 2021 14:24:36 -0700 (PDT) Received: from 127.0.0.1localhost ([185.69.145.194]) by smtp.gmail.com with ESMTPSA id m14sm216020wms.25.2021.10.19.14.24.35 (version=TLS1_3 cipher=TLS_AES_256_GCM_SHA384 bits=256/256); Tue, 19 Oct 2021 14:24:35 -0700 (PDT) From: Pavel Begunkov To: linux-block@vger.kernel.org Cc: Jens Axboe , Pavel Begunkov Subject: [PATCH 04/16] block: don't bloat enter_queue with percpu_ref Date: Tue, 19 Oct 2021 22:24:13 +0100 Message-Id: <49bff6b10644a6c789414bf72452edb7d54c132f.1634676157.git.asml.silence@gmail.com> X-Mailer: git-send-email 2.33.1 In-Reply-To: References: MIME-Version: 1.0 Precedence: bulk List-ID: X-Mailing-List: linux-block@vger.kernel.org percpu_ref_put() are inlined for performance and bloat the binary, we don't care about the fail case of blk_try_enter_queue(), so we can replace it with a call to blk_queue_exit(). Signed-off-by: Pavel Begunkov Reviewed-by: Chaitanya Kulkarni Reviewed-by: Christoph Hellwig --- block/blk-core.c | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/block/blk-core.c b/block/blk-core.c index c1ba34777c6d..88752e51d2b6 100644 --- a/block/blk-core.c +++ b/block/blk-core.c @@ -404,7 +404,7 @@ static bool blk_try_enter_queue(struct request_queue *q, bool pm) return true; fail_put: - percpu_ref_put(&q->q_usage_counter); + blk_queue_exit(q); fail: rcu_read_unlock(); return false; From patchwork Tue Oct 19 21:24:14 2021 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Pavel Begunkov X-Patchwork-Id: 12571037 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from mail.kernel.org (mail.kernel.org [198.145.29.99]) by smtp.lore.kernel.org (Postfix) with ESMTP id 520A4C4332F for ; Tue, 19 Oct 2021 21:24:40 +0000 (UTC) Received: from vger.kernel.org (vger.kernel.org [23.128.96.18]) by mail.kernel.org (Postfix) with ESMTP id 3902861373 for ; Tue, 19 Oct 2021 21:24:40 +0000 (UTC) Received: (majordomo@vger.kernel.org) by vger.kernel.org via listexpand id S229741AbhJSV0w (ORCPT ); Tue, 19 Oct 2021 17:26:52 -0400 Received: from lindbergh.monkeyblade.net ([23.128.96.19]:46992 "EHLO lindbergh.monkeyblade.net" rhost-flags-OK-OK-OK-OK) by vger.kernel.org with ESMTP id S229686AbhJSV0w (ORCPT ); Tue, 19 Oct 2021 17:26:52 -0400 Received: from mail-wm1-x32e.google.com (mail-wm1-x32e.google.com [IPv6:2a00:1450:4864:20::32e]) by lindbergh.monkeyblade.net (Postfix) with ESMTPS id 7E089C06161C for ; Tue, 19 Oct 2021 14:24:38 -0700 (PDT) Received: by mail-wm1-x32e.google.com with SMTP id 63-20020a1c0042000000b0030d60716239so5646162wma.4 for ; Tue, 19 Oct 2021 14:24:38 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=20210112; h=from:to:cc:subject:date:message-id:in-reply-to:references :mime-version:content-transfer-encoding; bh=ICDlpdHaMs6WzUPBmP4eN3qDpmvcW1LE/BZ0WZUx5Rg=; b=mlGdcTPb6zhfu1AmwUQbQIpdzrtm2cVcE8QINvTIAhGqvKQZT7xKXuB3XUz8HhJ+Lg nY2lKUeeH7L8D+mJImfbX0n99B70a77IAnCvEJEyMop/TiwDqsXbmn3P8YJWs3CDW5fS DM52TPnaGj41AHca3rFktM1U9M7gnmnOu5AEO/A+k7mGk4mnQIkYRWqqJ2qo0wf5gYvJ xehs2uC1HRDFRq6a/PHhw2O35CFkY9tQsYJwISm42MrvI5fr8NPKpnFwlWC6w8NJYA70 19dKsRGhZHR6uvIsYK/cDEjQ8Y0nTA3YL6kRNGAegkSp21Drav3EL9fYXOE/rvY3COUC Dikg== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20210112; h=x-gm-message-state:from:to:cc:subject:date:message-id:in-reply-to :references:mime-version:content-transfer-encoding; bh=ICDlpdHaMs6WzUPBmP4eN3qDpmvcW1LE/BZ0WZUx5Rg=; b=HIdtF/zNgZH/Xw0H2YNHucoTdKrHSWUyIRmVM/TLoE8KXKpIY5tVOpjISMy4Aei9T/ Ag7fzPTp87mxtLu12AiHydmt3J3joF6iBNP0F8lATfa1BKSEi0P45zJXPE8pbndzm1ip 0sJfjdLJSfPyzJ2kY7t2flzDotCaMQYOCRRaxmt+Psqx6rpCzH23H5A5W4ZFkpXZUi3w xXHrX18XybicVwh++S8bl/G7EkwcH28TA8JeZDNGLzYOrgo33c058kvdCi/j8htKA+6q NGGIV2sJD4/pUfsUCBI3GApBt6G+9Pk3ZMLqqkLoY/MNnJsI7QeB2ksRnWNv1Pp/s7gH duoA== X-Gm-Message-State: AOAM530kcBJNdfeV/YVC0xBIF95LWmnyGXG8bkf1mMUujmG12cFQ66Og EQXtZeOTYRbSi82rpyAwpQ0Ersz0Zoxksg== X-Google-Smtp-Source: ABdhPJzBLAQtr2uRuy1FIBeGI2kIOoSTtYmUwBbtU5uOs2+d8CVEPIcgveU9td/VHIde1txTpTdWXw== X-Received: by 2002:a05:600c:209:: with SMTP id 9mr8796730wmi.42.1634678676855; Tue, 19 Oct 2021 14:24:36 -0700 (PDT) Received: from 127.0.0.1localhost ([185.69.145.194]) by smtp.gmail.com with ESMTPSA id m14sm216020wms.25.2021.10.19.14.24.36 (version=TLS1_3 cipher=TLS_AES_256_GCM_SHA384 bits=256/256); Tue, 19 Oct 2021 14:24:36 -0700 (PDT) From: Pavel Begunkov To: linux-block@vger.kernel.org Cc: Jens Axboe , Pavel Begunkov Subject: [PATCH 05/16] block: inline a part of bio_release_pages() Date: Tue, 19 Oct 2021 22:24:14 +0100 Message-Id: X-Mailer: git-send-email 2.33.1 In-Reply-To: References: MIME-Version: 1.0 Precedence: bulk List-ID: X-Mailing-List: linux-block@vger.kernel.org Inline BIO_NO_PAGE_REF check of bio_release_pages() to avoid function call. Signed-off-by: Pavel Begunkov --- block/bio.c | 7 ++----- include/linux/bio.h | 8 +++++++- 2 files changed, 9 insertions(+), 6 deletions(-) diff --git a/block/bio.c b/block/bio.c index 4f397ba47db5..46a87c72d2b4 100644 --- a/block/bio.c +++ b/block/bio.c @@ -1033,21 +1033,18 @@ int bio_add_page(struct bio *bio, struct page *page, } EXPORT_SYMBOL(bio_add_page); -void bio_release_pages(struct bio *bio, bool mark_dirty) +void __bio_release_pages(struct bio *bio, bool mark_dirty) { struct bvec_iter_all iter_all; struct bio_vec *bvec; - if (bio_flagged(bio, BIO_NO_PAGE_REF)) - return; - bio_for_each_segment_all(bvec, bio, iter_all) { if (mark_dirty && !PageCompound(bvec->bv_page)) set_page_dirty_lock(bvec->bv_page); put_page(bvec->bv_page); } } -EXPORT_SYMBOL_GPL(bio_release_pages); +EXPORT_SYMBOL_GPL(__bio_release_pages); static void __bio_iov_bvec_set(struct bio *bio, struct iov_iter *iter) { diff --git a/include/linux/bio.h b/include/linux/bio.h index b12453d7b8a8..c88700d1bdc3 100644 --- a/include/linux/bio.h +++ b/include/linux/bio.h @@ -417,7 +417,7 @@ int bio_add_zone_append_page(struct bio *bio, struct page *page, void __bio_add_page(struct bio *bio, struct page *page, unsigned int len, unsigned int off); int bio_iov_iter_get_pages(struct bio *bio, struct iov_iter *iter); -void bio_release_pages(struct bio *bio, bool mark_dirty); +void __bio_release_pages(struct bio *bio, bool mark_dirty); extern void bio_set_pages_dirty(struct bio *bio); extern void bio_check_pages_dirty(struct bio *bio); @@ -428,6 +428,12 @@ extern void bio_free_pages(struct bio *bio); void guard_bio_eod(struct bio *bio); void zero_fill_bio(struct bio *bio); +static inline void bio_release_pages(struct bio *bio, bool mark_dirty) +{ + if (!bio_flagged(bio, BIO_NO_PAGE_REF)) + __bio_release_pages(bio, mark_dirty); +} + extern const char *bio_devname(struct bio *bio, char *buffer); #define bio_dev(bio) \ From patchwork Tue Oct 19 21:24:15 2021 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Pavel Begunkov X-Patchwork-Id: 12571039 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from mail.kernel.org (mail.kernel.org [198.145.29.99]) by smtp.lore.kernel.org (Postfix) with ESMTP id 1BC16C433FE for ; Tue, 19 Oct 2021 21:24:41 +0000 (UTC) Received: from vger.kernel.org (vger.kernel.org [23.128.96.18]) by mail.kernel.org (Postfix) with ESMTP id 097A46135E for ; Tue, 19 Oct 2021 21:24:41 +0000 (UTC) Received: (majordomo@vger.kernel.org) by vger.kernel.org via listexpand id S229779AbhJSV0x (ORCPT ); Tue, 19 Oct 2021 17:26:53 -0400 Received: from lindbergh.monkeyblade.net ([23.128.96.19]:46996 "EHLO lindbergh.monkeyblade.net" rhost-flags-OK-OK-OK-OK) by vger.kernel.org with ESMTP id S229686AbhJSV0w (ORCPT ); Tue, 19 Oct 2021 17:26:52 -0400 Received: from mail-wm1-x32a.google.com (mail-wm1-x32a.google.com [IPv6:2a00:1450:4864:20::32a]) by lindbergh.monkeyblade.net (Postfix) with ESMTPS id 607E1C06161C for ; Tue, 19 Oct 2021 14:24:39 -0700 (PDT) Received: by mail-wm1-x32a.google.com with SMTP id m42so14503458wms.2 for ; Tue, 19 Oct 2021 14:24:39 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=20210112; h=from:to:cc:subject:date:message-id:in-reply-to:references :mime-version:content-transfer-encoding; bh=OmS2BrUPNwMr2Y8HxTQBRF6ZuyMtaRKKrMwww/yn8fo=; b=BQnydNyPc66of3LhwBEG7A9vhE00L2Uq1MS3CleGgMJ1xoeXsjHoSyLQS+EqT5MRdt zR+SoHV0ugMBsnXhKbpfswnah+k0//RC+SP8q++c1p04TX0GQtoMJHgmL2fZwJyh2ctK iLsTj3TNXFhEqytd2/3CI6+F68Jbty8/OZa17a5Lp39H8tAzOKYK6mv1gOuL8sDOgeWk yXw/ucpZ6lAfsNnY1EAOMjO7pF76hkhYYNBkD2OPEyV5ZJgpdg3Jvgk0aCvBUIwsUsSl K9PhYosMdMQlnCdW7ZWmWhQivB5A7dIJ6e3WnVQ16CIOZhZ6UWqoSY8lLnWbLsGqpl+r f2rg== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20210112; h=x-gm-message-state:from:to:cc:subject:date:message-id:in-reply-to :references:mime-version:content-transfer-encoding; bh=OmS2BrUPNwMr2Y8HxTQBRF6ZuyMtaRKKrMwww/yn8fo=; b=ZbwOuLSO3Y0cfe7CGGJsSFPtY7saxpAwKgcIC9KGoiGP+MnvDkS8UXbLsUX8b4igL+ 2C1otfmYi/uUyddWvAMvtnwMBc3fHh113xK5YRsHxYSPB4ZMvvFWwTDVdCWW6LAX0yxn uYjPcq/yC0E1y1bKoeQsW931LIeLrVNTlrCNEjKPsgtohRfFKbL/ZhtccywOqB7qbfu9 bHaqXnKqUmTH5IEUMC0o/l/cNjMdq9kL2yFw6tW6reLScfl+bfHk9Kps+ilqpuP2XXyg ytE58+YsJrdjOfoIa/3fazxI7LaZG0fGxVDro3O41IA3vEUpoY6E96XdO0R4qpExvyoq 9FgQ== X-Gm-Message-State: AOAM531ZJVXtB95MIT7ETcxhQHoxQMe+0SudQ4aaYFYwAMlBN9MfDgG2 rhI9C0iP4GkROhjibkrDQM9YRux2il9NIg== X-Google-Smtp-Source: ABdhPJyq7NDN5PohfTveUeopo8wyJfjgP2SRZO4OZXII9CwbN0WmxrkH/zvB9acXTq1/JskO4Qj0xA== X-Received: by 2002:a1c:750b:: with SMTP id o11mr8988542wmc.5.1634678677785; Tue, 19 Oct 2021 14:24:37 -0700 (PDT) Received: from 127.0.0.1localhost ([185.69.145.194]) by smtp.gmail.com with ESMTPSA id m14sm216020wms.25.2021.10.19.14.24.37 (version=TLS1_3 cipher=TLS_AES_256_GCM_SHA384 bits=256/256); Tue, 19 Oct 2021 14:24:37 -0700 (PDT) From: Pavel Begunkov To: linux-block@vger.kernel.org Cc: Jens Axboe , Pavel Begunkov Subject: [PATCH 06/16] block: clean up blk_mq_submit_bio() merging Date: Tue, 19 Oct 2021 22:24:15 +0100 Message-Id: <4772d0d2111972ed5db4bc667e68e7416f809b57.1634676157.git.asml.silence@gmail.com> X-Mailer: git-send-email 2.33.1 In-Reply-To: References: MIME-Version: 1.0 Precedence: bulk List-ID: X-Mailing-List: linux-block@vger.kernel.org Combine blk_mq_sched_bio_merge() and blk_attempt_plug_merge() under a common if, so we don't check it twice. Also honor bio_mergeable() for blk_mq_sched_bio_merge(). Signed-off-by: Pavel Begunkov --- block/blk-mq-sched.c | 2 +- block/blk-mq-sched.h | 12 +----------- block/blk-mq.c | 13 +++++++------ 3 files changed, 9 insertions(+), 18 deletions(-) diff --git a/block/blk-mq-sched.c b/block/blk-mq-sched.c index e85b7556b096..5b259fdea794 100644 --- a/block/blk-mq-sched.c +++ b/block/blk-mq-sched.c @@ -361,7 +361,7 @@ void blk_mq_sched_dispatch_requests(struct blk_mq_hw_ctx *hctx) } } -bool __blk_mq_sched_bio_merge(struct request_queue *q, struct bio *bio, +bool blk_mq_sched_bio_merge(struct request_queue *q, struct bio *bio, unsigned int nr_segs) { struct elevator_queue *e = q->elevator; diff --git a/block/blk-mq-sched.h b/block/blk-mq-sched.h index 98836106b25f..25d1034952b6 100644 --- a/block/blk-mq-sched.h +++ b/block/blk-mq-sched.h @@ -12,7 +12,7 @@ void blk_mq_sched_assign_ioc(struct request *rq); bool blk_mq_sched_try_merge(struct request_queue *q, struct bio *bio, unsigned int nr_segs, struct request **merged_request); -bool __blk_mq_sched_bio_merge(struct request_queue *q, struct bio *bio, +bool blk_mq_sched_bio_merge(struct request_queue *q, struct bio *bio, unsigned int nr_segs); bool blk_mq_sched_try_insert_merge(struct request_queue *q, struct request *rq, struct list_head *free); @@ -42,16 +42,6 @@ static inline bool bio_mergeable(struct bio *bio) return !(bio->bi_opf & REQ_NOMERGE_FLAGS); } -static inline bool -blk_mq_sched_bio_merge(struct request_queue *q, struct bio *bio, - unsigned int nr_segs) -{ - if (blk_queue_nomerges(q) || !bio_mergeable(bio)) - return false; - - return __blk_mq_sched_bio_merge(q, bio, nr_segs); -} - static inline bool blk_mq_sched_allow_merge(struct request_queue *q, struct request *rq, struct bio *bio) diff --git a/block/blk-mq.c b/block/blk-mq.c index bab1fccda6ca..218bfaa98591 100644 --- a/block/blk-mq.c +++ b/block/blk-mq.c @@ -2482,12 +2482,13 @@ void blk_mq_submit_bio(struct bio *bio) if (!bio_integrity_prep(bio)) goto queue_exit; - if (!is_flush_fua && !blk_queue_nomerges(q) && - blk_attempt_plug_merge(q, bio, nr_segs, &same_queue_rq)) - goto queue_exit; - - if (blk_mq_sched_bio_merge(q, bio, nr_segs)) - goto queue_exit; + if (!blk_queue_nomerges(q) && bio_mergeable(bio)) { + if (!is_flush_fua && + blk_attempt_plug_merge(q, bio, nr_segs, &same_queue_rq)) + goto queue_exit; + if (blk_mq_sched_bio_merge(q, bio, nr_segs)) + goto queue_exit; + } rq_qos_throttle(q, bio); From patchwork Tue Oct 19 21:24:16 2021 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Pavel Begunkov X-Patchwork-Id: 12571041 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from mail.kernel.org (mail.kernel.org [198.145.29.99]) by smtp.lore.kernel.org (Postfix) with ESMTP id DE53FC433F5 for ; Tue, 19 Oct 2021 21:24:41 +0000 (UTC) Received: from vger.kernel.org (vger.kernel.org [23.128.96.18]) by mail.kernel.org (Postfix) with ESMTP id C580F6135E for ; Tue, 19 Oct 2021 21:24:41 +0000 (UTC) Received: (majordomo@vger.kernel.org) by vger.kernel.org via listexpand id S229602AbhJSV0y (ORCPT ); Tue, 19 Oct 2021 17:26:54 -0400 Received: from lindbergh.monkeyblade.net ([23.128.96.19]:47008 "EHLO lindbergh.monkeyblade.net" rhost-flags-OK-OK-OK-OK) by vger.kernel.org with ESMTP id S229686AbhJSV0x (ORCPT ); Tue, 19 Oct 2021 17:26:53 -0400 Received: from mail-wm1-x32f.google.com (mail-wm1-x32f.google.com [IPv6:2a00:1450:4864:20::32f]) by lindbergh.monkeyblade.net (Postfix) with ESMTPS id 403D3C06161C for ; Tue, 19 Oct 2021 14:24:40 -0700 (PDT) Received: by mail-wm1-x32f.google.com with SMTP id n40-20020a05600c3ba800b0030da2439b21so5702512wms.0 for ; Tue, 19 Oct 2021 14:24:40 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=20210112; h=from:to:cc:subject:date:message-id:in-reply-to:references :mime-version:content-transfer-encoding; bh=IAXLMKPgeBuYZroDLDZ6OvkIHJ1cntN48XfYtV9uC1M=; b=j2iELNjZLdpH2n/Ap/lzwkX1uRhJf6xbx4JVzMX3qoVHCRcwgoqRR6NNxL5ay8xE/z cj02VUt7zAFEGBE7W6uxDPX6r5mokDoBLVrrye1MXsM71xKlWpFe1hZG68Qexu25V41E s2U1q8bdEgW4Tx3akfaBxNzXiwvs7IE6NhJMtk+IiGn+SCFHeOrzf1NBO/mZGX7y3Psy dbkfzuMIcZ82vDPdjOY2JScsJmIuzWmy4kwm+GhT0P50vMNbAwELfvg/jpMCxF987RIk F5NkkzQywdslOFGeYOn6zOalZXr3e9IYkV/Xz3+TZ7coglVS/ZjaIszxZqAl2+R6u915 cYcA== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20210112; h=x-gm-message-state:from:to:cc:subject:date:message-id:in-reply-to :references:mime-version:content-transfer-encoding; bh=IAXLMKPgeBuYZroDLDZ6OvkIHJ1cntN48XfYtV9uC1M=; b=GqrWzpX57RmuOpBtLdvB+ATvk/oZ750euyeWlb7/9iQQEXIu5GNbhLrKjsq4rM/EUs 6KUDGL6rDYPo42fmWaKaEpGQ1CTDSVTrrasLj7PjLacVC0Yoyi7MD7fkXPmFj+ytwW3/ BmdKCZrphwCfWc06z7bM2P5fz5Q0DzQm4dUexd29SrNbuojrZx9EI4C9DhNiR6PVsZ6U X90zVOcyPQxtepN6M0YUJZQTsQLKRSBWDdgKnXEgir8DmJ0+2b0s6oFtQ2jD3dEAJFo/ mLbVwYgGqUa6zQ7fkUejdSZoYlaCq50HE+BmyvJrIz5uvZklX8hUfQESSTraRLmP2Qwa LJvg== X-Gm-Message-State: AOAM531mEJ4wXH7FokSoU+/YaiesfVwoOcFx9mfymmAUG6yHiYk3H1C3 MWFob4XJpx4RWUgCHd9QLwYW6uesmM+XVw== X-Google-Smtp-Source: ABdhPJzkyU08n71DJAvXVtIuhVXtRdzsdhRK/Y+yF1ZsZ9SzFJVH7weQtCQJ3gqBMy/iRY2J9wp5lw== X-Received: by 2002:a05:600c:4f92:: with SMTP id n18mr8903342wmq.22.1634678678585; Tue, 19 Oct 2021 14:24:38 -0700 (PDT) Received: from 127.0.0.1localhost ([185.69.145.194]) by smtp.gmail.com with ESMTPSA id m14sm216020wms.25.2021.10.19.14.24.37 (version=TLS1_3 cipher=TLS_AES_256_GCM_SHA384 bits=256/256); Tue, 19 Oct 2021 14:24:38 -0700 (PDT) From: Pavel Begunkov To: linux-block@vger.kernel.org Cc: Jens Axboe , Pavel Begunkov Subject: [PATCH 07/16] blocK: move plug flush functions to blk-mq.c Date: Tue, 19 Oct 2021 22:24:16 +0100 Message-Id: X-Mailer: git-send-email 2.33.1 In-Reply-To: References: MIME-Version: 1.0 Precedence: bulk List-ID: X-Mailing-List: linux-block@vger.kernel.org Flushing is tightly coupled with blk-mq and almost all blk_flush_plug_list() callees are in blk-mq.c. So move the whole thing there, so the compiler is able to apply more optimisations and inline. Signed-off-by: Pavel Begunkov Reviewed-by: Chaitanya Kulkarni --- block/blk-core.c | 27 --------------------------- block/blk-mq.c | 33 +++++++++++++++++++++++++++++---- block/blk-mq.h | 1 - include/linux/blk-mq.h | 2 -- 4 files changed, 29 insertions(+), 34 deletions(-) diff --git a/block/blk-core.c b/block/blk-core.c index 88752e51d2b6..52019b8a1487 100644 --- a/block/blk-core.c +++ b/block/blk-core.c @@ -1595,23 +1595,6 @@ void blk_start_plug(struct blk_plug *plug) } EXPORT_SYMBOL(blk_start_plug); -static void flush_plug_callbacks(struct blk_plug *plug, bool from_schedule) -{ - LIST_HEAD(callbacks); - - while (!list_empty(&plug->cb_list)) { - list_splice_init(&plug->cb_list, &callbacks); - - while (!list_empty(&callbacks)) { - struct blk_plug_cb *cb = list_first_entry(&callbacks, - struct blk_plug_cb, - list); - list_del(&cb->list); - cb->callback(cb, from_schedule); - } - } -} - struct blk_plug_cb *blk_check_plugged(blk_plug_cb_fn unplug, void *data, int size) { @@ -1637,16 +1620,6 @@ struct blk_plug_cb *blk_check_plugged(blk_plug_cb_fn unplug, void *data, } EXPORT_SYMBOL(blk_check_plugged); -void blk_flush_plug_list(struct blk_plug *plug, bool from_schedule) -{ - flush_plug_callbacks(plug, from_schedule); - - if (!rq_list_empty(plug->mq_list)) - blk_mq_flush_plug_list(plug, from_schedule); - if (unlikely(!from_schedule && plug->cached_rq)) - blk_mq_free_plug_rqs(plug); -} - /** * blk_finish_plug - mark the end of a batch of submitted I/O * @plug: The &struct blk_plug passed to blk_start_plug() diff --git a/block/blk-mq.c b/block/blk-mq.c index 218bfaa98591..6bdbaa838030 100644 --- a/block/blk-mq.c +++ b/block/blk-mq.c @@ -604,7 +604,7 @@ void blk_mq_free_request(struct request *rq) } EXPORT_SYMBOL_GPL(blk_mq_free_request); -void blk_mq_free_plug_rqs(struct blk_plug *plug) +static void blk_mq_free_plug_rqs(struct blk_plug *plug) { struct request *rq; @@ -2199,15 +2199,13 @@ static void blk_mq_plug_issue_direct(struct blk_plug *plug, bool from_schedule) blk_mq_commit_rqs(hctx, &queued, from_schedule); } -void blk_mq_flush_plug_list(struct blk_plug *plug, bool from_schedule) +static void blk_mq_flush_plug_list(struct blk_plug *plug, bool from_schedule) { struct blk_mq_hw_ctx *this_hctx; struct blk_mq_ctx *this_ctx; unsigned int depth; LIST_HEAD(list); - if (rq_list_empty(plug->mq_list)) - return; plug->rq_count = 0; if (!plug->multiple_queues && !plug->has_elevator) { @@ -2249,6 +2247,33 @@ void blk_mq_flush_plug_list(struct blk_plug *plug, bool from_schedule) } } +static void flush_plug_callbacks(struct blk_plug *plug, bool from_schedule) +{ + LIST_HEAD(callbacks); + + while (!list_empty(&plug->cb_list)) { + list_splice_init(&plug->cb_list, &callbacks); + + while (!list_empty(&callbacks)) { + struct blk_plug_cb *cb = list_first_entry(&callbacks, + struct blk_plug_cb, + list); + list_del(&cb->list); + cb->callback(cb, from_schedule); + } + } +} + +void blk_flush_plug_list(struct blk_plug *plug, bool from_schedule) +{ + flush_plug_callbacks(plug, from_schedule); + + if (!rq_list_empty(plug->mq_list)) + blk_mq_flush_plug_list(plug, from_schedule); + if (unlikely(!from_schedule && plug->cached_rq)) + blk_mq_free_plug_rqs(plug); +} + static void blk_mq_bio_to_request(struct request *rq, struct bio *bio, unsigned int nr_segs) { diff --git a/block/blk-mq.h b/block/blk-mq.h index ebf67f4d4f2e..bab40688e59b 100644 --- a/block/blk-mq.h +++ b/block/blk-mq.h @@ -121,7 +121,6 @@ extern int __blk_mq_register_dev(struct device *dev, struct request_queue *q); extern int blk_mq_sysfs_register(struct request_queue *q); extern void blk_mq_sysfs_unregister(struct request_queue *q); extern void blk_mq_hctx_kobj_init(struct blk_mq_hw_ctx *hctx); -void blk_mq_free_plug_rqs(struct blk_plug *plug); void blk_mq_release(struct request_queue *q); diff --git a/include/linux/blk-mq.h b/include/linux/blk-mq.h index 6cf35de151a9..e13780236550 100644 --- a/include/linux/blk-mq.h +++ b/include/linux/blk-mq.h @@ -656,8 +656,6 @@ int blk_mq_alloc_sq_tag_set(struct blk_mq_tag_set *set, unsigned int set_flags); void blk_mq_free_tag_set(struct blk_mq_tag_set *set); -void blk_mq_flush_plug_list(struct blk_plug *plug, bool from_schedule); - void blk_mq_free_request(struct request *rq); bool blk_mq_queue_inflight(struct request_queue *q); From patchwork Tue Oct 19 21:24:17 2021 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Pavel Begunkov X-Patchwork-Id: 12571043 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from mail.kernel.org (mail.kernel.org [198.145.29.99]) by smtp.lore.kernel.org (Postfix) with ESMTP id A9DBFC433FE for ; Tue, 19 Oct 2021 21:24:42 +0000 (UTC) Received: from vger.kernel.org (vger.kernel.org [23.128.96.18]) by mail.kernel.org (Postfix) with ESMTP id 923F26134F for ; Tue, 19 Oct 2021 21:24:42 +0000 (UTC) Received: (majordomo@vger.kernel.org) by vger.kernel.org via listexpand id S229791AbhJSV0y (ORCPT ); Tue, 19 Oct 2021 17:26:54 -0400 Received: from lindbergh.monkeyblade.net ([23.128.96.19]:47014 "EHLO lindbergh.monkeyblade.net" rhost-flags-OK-OK-OK-OK) by vger.kernel.org with ESMTP id S229784AbhJSV0y (ORCPT ); Tue, 19 Oct 2021 17:26:54 -0400 Received: from mail-wm1-x329.google.com (mail-wm1-x329.google.com [IPv6:2a00:1450:4864:20::329]) by lindbergh.monkeyblade.net (Postfix) with ESMTPS id 09023C06161C for ; Tue, 19 Oct 2021 14:24:41 -0700 (PDT) Received: by mail-wm1-x329.google.com with SMTP id l38-20020a05600c1d2600b0030d80c3667aso5616185wms.5 for ; Tue, 19 Oct 2021 14:24:40 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=20210112; h=from:to:cc:subject:date:message-id:in-reply-to:references :mime-version:content-transfer-encoding; bh=AVoXQBnS9I34e/jR0MATyoXPGgXTqfW0R5gKnmJ28c0=; b=XrVskauqhoBAXT/T/RytFGyrUPaWOw7mlti8hRCrySHipyffxLgwaA63mGLxIYvoLb 4uJF/Z1b3TqGoej2siTPl7/hM/z0po7B7O9RpL/F0OnDN6Yh12aU9ezsQ1jXEPLf3/fh A/49AD9NspNlwLadbj7cHBuRVVjYYoS8TRT0EndDxs/kymI9vsIU54kmmWDJtq5IDWf/ DwkTK7YoovJj+84HfIUgHg0TPd/poiSOC56QY17UvAJSVWanDAHOCpInZh6vAmnRKK2A toSSeFcThcmqvHAwEKbgoVdJDf3lBfGhbly0UAw2eE1Ij8kVSHR2dlJSAjCvwZ+LOTDb lu8Q== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20210112; h=x-gm-message-state:from:to:cc:subject:date:message-id:in-reply-to :references:mime-version:content-transfer-encoding; bh=AVoXQBnS9I34e/jR0MATyoXPGgXTqfW0R5gKnmJ28c0=; b=UbqsPj97pUuJ1V6c1hYRsktvH6uw/OOQCyNCQXIfyD6rygS1wjIfiZ05JpNkCgIvQp LJzly21GWCeRN9lvqXdWTNSjpraOScfS/Z95KU8OwjJDIn/OtEKaZOKg5e735SA/f9Mr xXJIw30SErtm5BL7myQjjvpFcuc/RL0HEN0lLx3Hi+bVJ1GTwQu2xWu76GH/qs/2EO3I KtUNgV7jUunaMWDoJBbGUVn17x/I6OphovRhgLEI+Hn9Vl2HpPyh94K1k/b9RniU0i4Q w2xllWOXxxnMnT6PuIVtDsEcsQmiJGn/rJdsyLyqOrWQY9ENOiHzeyaudUhL09+jAVfW GDZQ== X-Gm-Message-State: AOAM533u7xCLtsFbgpg4mL8C3IBnctJtqiFnRYtSyHWjSpbufN0t7A/V 6ub0DcBtkqlDZMIVmz+MMOFK0Y7JMvdgEw== X-Google-Smtp-Source: ABdhPJwGOx2wSeVTV+foACs0d8Ka1MNGkRtYQWZDVLlbaAIH2jaoXj2VnPoRinzEw+KuoAYAmo1LDQ== X-Received: by 2002:a1c:1b89:: with SMTP id b131mr8835880wmb.71.1634678679421; Tue, 19 Oct 2021 14:24:39 -0700 (PDT) Received: from 127.0.0.1localhost ([185.69.145.194]) by smtp.gmail.com with ESMTPSA id m14sm216020wms.25.2021.10.19.14.24.38 (version=TLS1_3 cipher=TLS_AES_256_GCM_SHA384 bits=256/256); Tue, 19 Oct 2021 14:24:39 -0700 (PDT) From: Pavel Begunkov To: linux-block@vger.kernel.org Cc: Jens Axboe , Pavel Begunkov Subject: [PATCH 08/16] block: optimise blk_flush_plug_list Date: Tue, 19 Oct 2021 22:24:17 +0100 Message-Id: X-Mailer: git-send-email 2.33.1 In-Reply-To: References: MIME-Version: 1.0 Precedence: bulk List-ID: X-Mailing-List: linux-block@vger.kernel.org First, don't init a callback list if there are no plug callbacks. Also, replace internals of the function with do-while. Signed-off-by: Pavel Begunkov --- block/blk-mq.c | 12 +++++++----- 1 file changed, 7 insertions(+), 5 deletions(-) diff --git a/block/blk-mq.c b/block/blk-mq.c index 6bdbaa838030..6627ea76f7c6 100644 --- a/block/blk-mq.c +++ b/block/blk-mq.c @@ -2251,22 +2251,24 @@ static void flush_plug_callbacks(struct blk_plug *plug, bool from_schedule) { LIST_HEAD(callbacks); - while (!list_empty(&plug->cb_list)) { + do { list_splice_init(&plug->cb_list, &callbacks); - while (!list_empty(&callbacks)) { + do { struct blk_plug_cb *cb = list_first_entry(&callbacks, struct blk_plug_cb, list); + list_del(&cb->list); cb->callback(cb, from_schedule); - } - } + } while (!list_empty(&callbacks)); + } while (!list_empty(&plug->cb_list)); } void blk_flush_plug_list(struct blk_plug *plug, bool from_schedule) { - flush_plug_callbacks(plug, from_schedule); + if (!list_empty(&plug->cb_list)) + flush_plug_callbacks(plug, from_schedule); if (!rq_list_empty(plug->mq_list)) blk_mq_flush_plug_list(plug, from_schedule); From patchwork Tue Oct 19 21:24:18 2021 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Pavel Begunkov X-Patchwork-Id: 12571045 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from mail.kernel.org (mail.kernel.org [198.145.29.99]) by smtp.lore.kernel.org (Postfix) with ESMTP id D7AEFC433F5 for ; Tue, 19 Oct 2021 21:24:43 +0000 (UTC) Received: from vger.kernel.org (vger.kernel.org [23.128.96.18]) by mail.kernel.org (Postfix) with ESMTP id BB2D260F46 for ; Tue, 19 Oct 2021 21:24:43 +0000 (UTC) Received: (majordomo@vger.kernel.org) by vger.kernel.org via listexpand id S229809AbhJSV0z (ORCPT ); Tue, 19 Oct 2021 17:26:55 -0400 Received: from lindbergh.monkeyblade.net ([23.128.96.19]:47018 "EHLO lindbergh.monkeyblade.net" rhost-flags-OK-OK-OK-OK) by vger.kernel.org with ESMTP id S229784AbhJSV0z (ORCPT ); Tue, 19 Oct 2021 17:26:55 -0400 Received: from mail-wm1-x32c.google.com (mail-wm1-x32c.google.com [IPv6:2a00:1450:4864:20::32c]) by lindbergh.monkeyblade.net (Postfix) with ESMTPS id C676DC06161C for ; Tue, 19 Oct 2021 14:24:41 -0700 (PDT) Received: by mail-wm1-x32c.google.com with SMTP id d198-20020a1c1dcf000000b00322f53b9b89so5680196wmd.0 for ; Tue, 19 Oct 2021 14:24:41 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=20210112; h=from:to:cc:subject:date:message-id:in-reply-to:references :mime-version:content-transfer-encoding; bh=5dMfaSBYPSk+bqOWtfqC72fmmxN8NldhpDFxApBTDmM=; b=K2Yer8ogJ5WrVrbn5OtlNfXf6zwQ3t+Yi4+2+A+djLkLl6bFe2pFNq751DSxahRLpx xlNDFY7yGHnZuzNx0SJlCY1m5dJ0bOs7KYgF9H5YI6lOJLVFToDl0qPIrtEKjcvAJh/E F7RhzgjI9VOQj6u0DqCXBk6r8V/ZCIJjJNSECMNUOAjAy0mErcxaMScmeoDXucpbXsZr W4/R2CFlCQVadArfw//uv5qAeOhOJnomaQn6jCW7K3yfSN/Gl5DuUMZOzJU/9tkgFKjk 2lvz1KIN2ydgG0A8f+DmoKDgfTof8X7ftBx0qP83c/EAwY9mRXdsYrwzbUcLilcaEUwN 4TEw== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20210112; h=x-gm-message-state:from:to:cc:subject:date:message-id:in-reply-to :references:mime-version:content-transfer-encoding; bh=5dMfaSBYPSk+bqOWtfqC72fmmxN8NldhpDFxApBTDmM=; b=GdrW/+DT5ecUMldzN9NGN40DSjjRjmANFoQJpdQa2WWcO0TEAs9OpCi5/bn2IOiT77 ytEcxqBCktFS88tP6Q4rOU03S5od+nB0NEPKEJRCRIc2qPaHnFPQj7itzoOyC1AmILNG y2CnU2oa5RpG9avw1gXfhxAVkXI7AUElfMxcUaKh4IRvJJJeh462rpnxu43pbLj0K07W IKna8VSzxGsrpCZm/9+sMYlvgaWd1GXrjPscdeG6D/oU6FwqbBQW7OJuZM5s9dLu0gYw nAlaxg0OQQJx7eAKBHAhqFg9GSGWgy2lgByKwimUDLUDnY+fvN/rfevHs8MZsF8rl53x rsWw== X-Gm-Message-State: AOAM531TfpdZqFmFsBQQSiP/ncqcmptNBbWrzmyVUjkwUfni3WA0rPN6 nwG4uiGFRyL/HrDt/TRJw3CB//lIKnN0Eg== X-Google-Smtp-Source: ABdhPJyCySowCV0MDOyoTBOPNDEsgQ4Dh1IAcIJlF6HfqnHljq+MXfcdKjRso+Kkdr5vLO6hcHpQAw== X-Received: by 2002:a1c:9ad4:: with SMTP id c203mr8944155wme.41.1634678680236; Tue, 19 Oct 2021 14:24:40 -0700 (PDT) Received: from 127.0.0.1localhost ([185.69.145.194]) by smtp.gmail.com with ESMTPSA id m14sm216020wms.25.2021.10.19.14.24.39 (version=TLS1_3 cipher=TLS_AES_256_GCM_SHA384 bits=256/256); Tue, 19 Oct 2021 14:24:39 -0700 (PDT) From: Pavel Begunkov To: linux-block@vger.kernel.org Cc: Jens Axboe , Pavel Begunkov Subject: [PATCH 09/16] block: optimise boundary blkdev_read_iter's checks Date: Tue, 19 Oct 2021 22:24:18 +0100 Message-Id: X-Mailer: git-send-email 2.33.1 In-Reply-To: References: MIME-Version: 1.0 Precedence: bulk List-ID: X-Mailing-List: linux-block@vger.kernel.org Combine pos and len checks and mark unlikely. Also, don't reexpand if it's not truncated. Signed-off-by: Pavel Begunkov Reviewed-by: Christoph Hellwig --- block/fops.c | 19 +++++++++++-------- 1 file changed, 11 insertions(+), 8 deletions(-) diff --git a/block/fops.c b/block/fops.c index 21d25ee0e4bf..8f733c919ed1 100644 --- a/block/fops.c +++ b/block/fops.c @@ -503,17 +503,20 @@ static ssize_t blkdev_read_iter(struct kiocb *iocb, struct iov_iter *to) size_t shorted = 0; ssize_t ret; - if (pos >= size) - return 0; - - size -= pos; - if (iov_iter_count(to) > size) { - shorted = iov_iter_count(to) - size; - iov_iter_truncate(to, size); + if (unlikely(pos + iov_iter_count(to) > size)) { + if (pos >= size) + return 0; + size -= pos; + if (iov_iter_count(to) > size) { + shorted = iov_iter_count(to) - size; + iov_iter_truncate(to, size); + } } ret = generic_file_read_iter(iocb, to); - iov_iter_reexpand(to, iov_iter_count(to) + shorted); + + if (unlikely(shorted)) + iov_iter_reexpand(to, iov_iter_count(to) + shorted); return ret; } From patchwork Tue Oct 19 21:24:19 2021 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Pavel Begunkov X-Patchwork-Id: 12571047 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from mail.kernel.org (mail.kernel.org [198.145.29.99]) by smtp.lore.kernel.org (Postfix) with ESMTP id C37F4C433F5 for ; Tue, 19 Oct 2021 21:24:45 +0000 (UTC) Received: from vger.kernel.org (vger.kernel.org [23.128.96.18]) by mail.kernel.org (Postfix) with ESMTP id ABAF360F46 for ; Tue, 19 Oct 2021 21:24:45 +0000 (UTC) Received: (majordomo@vger.kernel.org) by vger.kernel.org via listexpand id S229845AbhJSV05 (ORCPT ); Tue, 19 Oct 2021 17:26:57 -0400 Received: from lindbergh.monkeyblade.net ([23.128.96.19]:47026 "EHLO lindbergh.monkeyblade.net" rhost-flags-OK-OK-OK-OK) by vger.kernel.org with ESMTP id S229817AbhJSV04 (ORCPT ); Tue, 19 Oct 2021 17:26:56 -0400 Received: from mail-wm1-x32f.google.com (mail-wm1-x32f.google.com [IPv6:2a00:1450:4864:20::32f]) by lindbergh.monkeyblade.net (Postfix) with ESMTPS id A4029C06161C for ; Tue, 19 Oct 2021 14:24:42 -0700 (PDT) Received: by mail-wm1-x32f.google.com with SMTP id v127so14528226wme.5 for ; Tue, 19 Oct 2021 14:24:42 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=20210112; h=from:to:cc:subject:date:message-id:in-reply-to:references :mime-version:content-transfer-encoding; bh=jm6YiXYaSKf1Zqt9LVojCcnO0QnTkC8Ut2wu+PHwNzo=; b=N5333q8jkDCaio8OP6MbDVBhyPmfg1CXghnBRdZ/UP/9ijugLHMfFnVaOzgpeihel8 TYpu6EFsHP/npokzhzqiPmmNUUIEi2XqMwgLmk01+vNbWEQK3ikA85wrCP3TYy1FM6sk 5HWLu6M85n6U/HQCQMd25iAoKjmk62qUglI38kJzikBObIZBLb+MJZc/KIu4VlojzISO odObSjLTVoMv4rI3jpYG/0wO59RtAAfqzCmdpnkOJJf/Of7Xa+jplOFovHnMJtxWlmWL MpjiekbVYoCNpGww0mTlWG0Z2PiVqOtXd6s0LoSK7UjWvTsZ1RPMUkmZmUBPuUlXNzKS 2qig== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20210112; h=x-gm-message-state:from:to:cc:subject:date:message-id:in-reply-to :references:mime-version:content-transfer-encoding; bh=jm6YiXYaSKf1Zqt9LVojCcnO0QnTkC8Ut2wu+PHwNzo=; b=noVwGzZaf2Adr1hLWBGS7PhbkNMbLX2rHoz42F/0ZKcqxgsZLHo6lr59+sD215Lnos ZMvyl7OaJGOGyVu/n05jDqfYYH3IEZp9vl4/4DnBNHXQ5m12tTWz6otmP1zNa3JVOvV2 jD12mQqmCZEIut4IDKqzg8HBhlOE1F22p8KsFalt0g0YqCtaprboGLlxofvRdQ7L8586 OjMBySHzZpmLn1RFKNIGsrg5Ke24apPKXg7IAfh4//mKp75nPzetTjYnDt2/fVU/Da+v qGiXox7Hkl8N9D8+uZesz0tRUBG2Lp571pAl8fcYK0+myA3ui/kzPOeuK0okvlgnm1gp vB2w== X-Gm-Message-State: AOAM5327WjtJ20VIw5g47gTZl5Vl0dDJrO6eAoGt9yMccGejeGCLba8W baqQjcaNPGdpRRpa08vg9OaHbkPz9PE9tA== X-Google-Smtp-Source: ABdhPJyyoBc5eMgSOhDaT3ui6bWSu+Pjv942dGMhfA3WyvDAwEQqRyJ/3L7JvaUWWjQmRE4hNJNaGQ== X-Received: by 2002:a05:6000:10c7:: with SMTP id b7mr46705416wrx.36.1634678680977; Tue, 19 Oct 2021 14:24:40 -0700 (PDT) Received: from 127.0.0.1localhost ([185.69.145.194]) by smtp.gmail.com with ESMTPSA id m14sm216020wms.25.2021.10.19.14.24.40 (version=TLS1_3 cipher=TLS_AES_256_GCM_SHA384 bits=256/256); Tue, 19 Oct 2021 14:24:40 -0700 (PDT) From: Pavel Begunkov To: linux-block@vger.kernel.org Cc: Jens Axboe , Pavel Begunkov Subject: [PATCH 10/16] block: optimise blkdev_bio_end_io() Date: Tue, 19 Oct 2021 22:24:19 +0100 Message-Id: <8e6003932f65ecd9ada5d6296c6eb9d1b946a1eb.1634676157.git.asml.silence@gmail.com> X-Mailer: git-send-email 2.33.1 In-Reply-To: References: MIME-Version: 1.0 Precedence: bulk List-ID: X-Mailing-List: linux-block@vger.kernel.org Save dio->flags in a variable, so it doesn't reload it a bunch of times. Also use cached in a var iocb for the same reason. Signed-off-by: Pavel Begunkov --- block/fops.c | 12 ++++++------ 1 file changed, 6 insertions(+), 6 deletions(-) diff --git a/block/fops.c b/block/fops.c index 8f733c919ed1..8e3790faafb8 100644 --- a/block/fops.c +++ b/block/fops.c @@ -145,13 +145,13 @@ static struct bio_set blkdev_dio_pool; static void blkdev_bio_end_io(struct bio *bio) { struct blkdev_dio *dio = bio->bi_private; - bool should_dirty = dio->flags & DIO_SHOULD_DIRTY; + unsigned int flags = dio->flags; if (bio->bi_status && !dio->bio.bi_status) dio->bio.bi_status = bio->bi_status; - if (!(dio->flags & DIO_MULTI_BIO) || atomic_dec_and_test(&dio->ref)) { - if (!(dio->flags & DIO_IS_SYNC)) { + if (!(flags & DIO_MULTI_BIO) || atomic_dec_and_test(&dio->ref)) { + if (!(flags & DIO_IS_SYNC)) { struct kiocb *iocb = dio->iocb; ssize_t ret; @@ -164,8 +164,8 @@ static void blkdev_bio_end_io(struct bio *bio) ret = blk_status_to_errno(dio->bio.bi_status); } - dio->iocb->ki_complete(iocb, ret, 0); - if (dio->flags & DIO_MULTI_BIO) + iocb->ki_complete(iocb, ret, 0); + if (flags & DIO_MULTI_BIO) bio_put(&dio->bio); } else { struct task_struct *waiter = dio->waiter; @@ -175,7 +175,7 @@ static void blkdev_bio_end_io(struct bio *bio) } } - if (should_dirty) { + if (flags & DIO_SHOULD_DIRTY) { bio_check_pages_dirty(bio); } else { bio_release_pages(bio, false); From patchwork Tue Oct 19 21:24:20 2021 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Pavel Begunkov X-Patchwork-Id: 12571049 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from mail.kernel.org (mail.kernel.org [198.145.29.99]) by smtp.lore.kernel.org (Postfix) with ESMTP id 255B4C433EF for ; Tue, 19 Oct 2021 21:24:46 +0000 (UTC) Received: from vger.kernel.org (vger.kernel.org [23.128.96.18]) by mail.kernel.org (Postfix) with ESMTP id 105686135E for ; Tue, 19 Oct 2021 21:24:46 +0000 (UTC) Received: (majordomo@vger.kernel.org) by vger.kernel.org via listexpand id S229790AbhJSV06 (ORCPT ); Tue, 19 Oct 2021 17:26:58 -0400 Received: from lindbergh.monkeyblade.net ([23.128.96.19]:47030 "EHLO lindbergh.monkeyblade.net" rhost-flags-OK-OK-OK-OK) by vger.kernel.org with ESMTP id S229836AbhJSV04 (ORCPT ); Tue, 19 Oct 2021 17:26:56 -0400 Received: from mail-wm1-x32c.google.com (mail-wm1-x32c.google.com [IPv6:2a00:1450:4864:20::32c]) by lindbergh.monkeyblade.net (Postfix) with ESMTPS id 7D925C06161C for ; Tue, 19 Oct 2021 14:24:43 -0700 (PDT) Received: by mail-wm1-x32c.google.com with SMTP id n7-20020a05600c4f8700b00323023159e1so5671127wmq.2 for ; Tue, 19 Oct 2021 14:24:43 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=20210112; h=from:to:cc:subject:date:message-id:in-reply-to:references :mime-version:content-transfer-encoding; bh=7qTPRZgyDT08UP4tOI5cRqE6rGaKmGvoqqyH8NygZgg=; b=lpqqBwa7G0fG0NqFCwGTYPWdceC7YgZi5jV2go14jTyYKpnmrEFDAqjWl+gkkWla3W MMOMom7q/oCiAp4KpqtrQSBChgfy1TkCSMqQdrlouL8s/41fEaPtp4eYnN2y9rivH/tk QwKOmow0rvGCyhj+DCGo901i45zwAwJcZhWc6i+9f3QlwFDe48ceyRg5f5b/X8sTpwIG bRyvVC/0K5lysWRyBtyD35Tb5IXnvyBlLzuuvNDbLhEmpx4YCvBThW6G9jnfDExl1FQr 4Nz8GvICECOVNcQW/gsmvJpDNB33yWNQzfj1MU1bKDL7+BHOAbuZNlS7dsuSmJl4xwfv yBPw== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20210112; h=x-gm-message-state:from:to:cc:subject:date:message-id:in-reply-to :references:mime-version:content-transfer-encoding; bh=7qTPRZgyDT08UP4tOI5cRqE6rGaKmGvoqqyH8NygZgg=; b=RYiQrLfrgZhxsdY06NLADSO7OjKNOPNCjMq1iVRylmEiWECwLfjLo3CHQvqesQWbVX 2XFpga7MjHFyIEsfbl0jCsOjvcveiBLKjlfHrdSdYbwN3iH+D/smLvu9oIGSv32ni0GM D+orwaqpOBYEno2sUTPiQ/h67mi4NHUee9RAC6NBULw2L4q+o0XWw2O/qcujI6ejDByj e19p1nQlRKP5pJgEiMbal/R/elEMa/ESytUtczImwP61rNAGs2LGHK1tTBCfI/RBmxbT 9TMobxQUykke032oONXonST0x/BJbFmLhBpxFKbfYn366FgXiSj2YAEDh7IXPvU7coIN /riA== X-Gm-Message-State: AOAM531F/9K2N2DnDoM+LUNGsjrPPNi6+PAwlD0vRdkefnxG/zX7IfwG 8UyE2NSindL1osboNWsO/e+3zZn4tS3thw== X-Google-Smtp-Source: ABdhPJy8cTCeXpHb3uLNKBnb9FL/J7NIFnKvj2rlu32fIjM9A7bQVLO+2gyxsPxXkR0o6qLBqsoccQ== X-Received: by 2002:adf:a78a:: with SMTP id j10mr46784944wrc.231.1634678681879; Tue, 19 Oct 2021 14:24:41 -0700 (PDT) Received: from 127.0.0.1localhost ([185.69.145.194]) by smtp.gmail.com with ESMTPSA id m14sm216020wms.25.2021.10.19.14.24.41 (version=TLS1_3 cipher=TLS_AES_256_GCM_SHA384 bits=256/256); Tue, 19 Oct 2021 14:24:41 -0700 (PDT) From: Pavel Begunkov To: linux-block@vger.kernel.org Cc: Jens Axboe , Pavel Begunkov Subject: [PATCH 11/16] block: add optimised version bio_set_dev() Date: Tue, 19 Oct 2021 22:24:20 +0100 Message-Id: <3c908cb74959c631995341111a7ce116487da5c5.1634676157.git.asml.silence@gmail.com> X-Mailer: git-send-email 2.33.1 In-Reply-To: References: MIME-Version: 1.0 Precedence: bulk List-ID: X-Mailing-List: linux-block@vger.kernel.org If a bio was just allocated its flags should be zero and there is no need to clear them. Add __bio_set_dev(), which is faster and doesn't care about clering flags. Signed-off-by: Pavel Begunkov Reviewed-by: Chaitanya Kulkarni --- block/fops.c | 4 ++-- include/linux/bio.h | 10 ++++++++-- 2 files changed, 10 insertions(+), 4 deletions(-) diff --git a/block/fops.c b/block/fops.c index 8e3790faafb8..7cf98db0595a 100644 --- a/block/fops.c +++ b/block/fops.c @@ -75,7 +75,7 @@ static ssize_t __blkdev_direct_IO_simple(struct kiocb *iocb, } bio_init(&bio, vecs, nr_pages); - bio_set_dev(&bio, bdev); + __bio_set_dev(&bio, bdev); bio.bi_iter.bi_sector = pos >> 9; bio.bi_write_hint = iocb->ki_hint; bio.bi_private = current; @@ -224,7 +224,7 @@ static ssize_t __blkdev_direct_IO(struct kiocb *iocb, struct iov_iter *iter, blk_start_plug(&plug); for (;;) { - bio_set_dev(bio, bdev); + __bio_set_dev(bio, bdev); bio->bi_iter.bi_sector = pos >> 9; bio->bi_write_hint = iocb->ki_hint; bio->bi_private = dio; diff --git a/include/linux/bio.h b/include/linux/bio.h index c88700d1bdc3..0ab4fa2c89c3 100644 --- a/include/linux/bio.h +++ b/include/linux/bio.h @@ -453,13 +453,19 @@ static inline void bio_clone_blkg_association(struct bio *dst, struct bio *src) { } #endif /* CONFIG_BLK_CGROUP */ +/* can be used only with freshly allocated bios */ +static inline void __bio_set_dev(struct bio *bio, struct block_device *bdev) +{ + bio->bi_bdev = bdev; + bio_associate_blkg(bio); +} + static inline void bio_set_dev(struct bio *bio, struct block_device *bdev) { bio_clear_flag(bio, BIO_REMAPPED); if (bio->bi_bdev != bdev) bio_clear_flag(bio, BIO_THROTTLED); - bio->bi_bdev = bdev; - bio_associate_blkg(bio); + __bio_set_dev(bio, bdev); } static inline void bio_copy_dev(struct bio *dst, struct bio *src) From patchwork Tue Oct 19 21:24:21 2021 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Pavel Begunkov X-Patchwork-Id: 12571051 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from mail.kernel.org (mail.kernel.org [198.145.29.99]) by smtp.lore.kernel.org (Postfix) with ESMTP id 06861C4332F for ; Tue, 19 Oct 2021 21:24:47 +0000 (UTC) Received: from vger.kernel.org (vger.kernel.org [23.128.96.18]) by mail.kernel.org (Postfix) with ESMTP id E3B7960F46 for ; Tue, 19 Oct 2021 21:24:46 +0000 (UTC) Received: (majordomo@vger.kernel.org) by vger.kernel.org via listexpand id S229877AbhJSV06 (ORCPT ); Tue, 19 Oct 2021 17:26:58 -0400 Received: from lindbergh.monkeyblade.net ([23.128.96.19]:47032 "EHLO lindbergh.monkeyblade.net" rhost-flags-OK-OK-OK-OK) by vger.kernel.org with ESMTP id S229864AbhJSV05 (ORCPT ); Tue, 19 Oct 2021 17:26:57 -0400 Received: from mail-wm1-x329.google.com (mail-wm1-x329.google.com [IPv6:2a00:1450:4864:20::329]) by lindbergh.monkeyblade.net (Postfix) with ESMTPS id 3E5FDC061765 for ; Tue, 19 Oct 2021 14:24:44 -0700 (PDT) Received: by mail-wm1-x329.google.com with SMTP id v127so14528363wme.5 for ; Tue, 19 Oct 2021 14:24:44 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=20210112; h=from:to:cc:subject:date:message-id:in-reply-to:references :mime-version:content-transfer-encoding; bh=vie2hOMzoHrUFDMkHnt8EnTGZllVCb0xIPYnVaf/naE=; b=i3/a1ktXlvw55dUOyFaY/u7NnHi4/+QvRBHaEzXnaqhRuJKyVvTHReCOkEB1aBVKm2 yg3AvlQPibQVv49cZSgljLKjf40T9cXXv/B2GBQu140R4iYrR6QLr8UMAUjoebHo35M0 lx4gsV9iygWSO7nPoySrOHD1Nlzq4vUTn/gZAop/bTa8zHG2QqKd0Nr3kOE9hYqggsH6 07qZsM0h0X20Ht3uob0cIGmWmBozzfZ2Djr/N6HC9i3NyGV4AyvQsnyiic8UzkEOKBNp f+iuAu64R0m+VZZ+lN8c52B+VnjkkVzZfTSPZjbYtEdEbNnLme61qmiHdEXYY9aXVZ+K QBYg== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20210112; h=x-gm-message-state:from:to:cc:subject:date:message-id:in-reply-to :references:mime-version:content-transfer-encoding; bh=vie2hOMzoHrUFDMkHnt8EnTGZllVCb0xIPYnVaf/naE=; b=LHtLOL409voHEu9SNBLjdCzB9yvTcdGOTEzu4hVENKq7kgQ584zUG71t2GjMmLrFZI 5vo1a7Ls3v2VhoO+X7/TVEFnluccrPCvEPB3i6suEWUaTHFBI7P7adb9jp02EOsaYRpY 6S/jhONqVuNzhoabmm+Ny0wRDUKZoMWJ29I4BEVEVPhwNVMw7AjHqB0CrsyPmPJkv8ck ogyZR8heU73ddn2cf5h2jo6PrBtt91xwq3blpCLFoAuMm1+6zw0vp9McBLcLX3DJQvYR zU9fjudBHyshtfxqgt2NBJhzK8TT4pOuhtDwI+MBGOtDO2sudfPyFOg3krfE1aaGiHMc AWIQ== X-Gm-Message-State: AOAM533mNPpT6wH0PlzWxhgSLoWkOzuVks+wcymxb/lqO0mU5ifz8I1O C4J+ALFAXk9UE/fIGl80wrFZU52dhP1c+g== X-Google-Smtp-Source: ABdhPJxkKlaQYf001wojZduWXNhNowYiUAA8WR6L2haNf+030WwY1ziKT/SQiULrWsr+Ns/VBvt82Q== X-Received: by 2002:a1c:7918:: with SMTP id l24mr8708473wme.137.1634678682629; Tue, 19 Oct 2021 14:24:42 -0700 (PDT) Received: from 127.0.0.1localhost ([185.69.145.194]) by smtp.gmail.com with ESMTPSA id m14sm216020wms.25.2021.10.19.14.24.41 (version=TLS1_3 cipher=TLS_AES_256_GCM_SHA384 bits=256/256); Tue, 19 Oct 2021 14:24:42 -0700 (PDT) From: Pavel Begunkov To: linux-block@vger.kernel.org Cc: Jens Axboe , Pavel Begunkov Subject: [PATCH 12/16] block: add single bio async direct IO helper Date: Tue, 19 Oct 2021 22:24:21 +0100 Message-Id: X-Mailer: git-send-email 2.33.1 In-Reply-To: References: MIME-Version: 1.0 Precedence: bulk List-ID: X-Mailing-List: linux-block@vger.kernel.org As with __blkdev_direct_IO_simple(), we can implement direct IO more efficiently if there is only one bio. Add __blkdev_direct_IO_async() and blkdev_bio_end_io_async(). This patch brings me from 4.45-4.5 MIOPS with nullblk to 4.7+. Signed-off-by: Pavel Begunkov --- block/fops.c | 90 ++++++++++++++++++++++++++++++++++++++++++++++++++-- 1 file changed, 87 insertions(+), 3 deletions(-) diff --git a/block/fops.c b/block/fops.c index 7cf98db0595a..0f1332374756 100644 --- a/block/fops.c +++ b/block/fops.c @@ -305,6 +305,88 @@ static ssize_t __blkdev_direct_IO(struct kiocb *iocb, struct iov_iter *iter, return ret; } +static void blkdev_bio_end_io_async(struct bio *bio) +{ + struct blkdev_dio *dio = container_of(bio, struct blkdev_dio, bio); + struct kiocb *iocb = dio->iocb; + ssize_t ret; + + if (likely(!bio->bi_status)) { + ret = dio->size; + iocb->ki_pos += ret; + } else { + ret = blk_status_to_errno(bio->bi_status); + } + + iocb->ki_complete(iocb, ret, 0); + + if (dio->flags & DIO_SHOULD_DIRTY) { + bio_check_pages_dirty(bio); + } else { + bio_release_pages(bio, false); + bio_put(bio); + } +} + +static ssize_t __blkdev_direct_IO_async(struct kiocb *iocb, + struct iov_iter *iter, + unsigned int nr_pages) +{ + struct block_device *bdev = iocb->ki_filp->private_data; + struct blkdev_dio *dio; + struct bio *bio; + loff_t pos = iocb->ki_pos; + int ret = 0; + + if ((pos | iov_iter_alignment(iter)) & + (bdev_logical_block_size(bdev) - 1)) + return -EINVAL; + + bio = bio_alloc_kiocb(iocb, nr_pages, &blkdev_dio_pool); + dio = container_of(bio, struct blkdev_dio, bio); + __bio_set_dev(bio, bdev); + bio->bi_iter.bi_sector = pos >> 9; + bio->bi_write_hint = iocb->ki_hint; + bio->bi_end_io = blkdev_bio_end_io_async; + bio->bi_ioprio = iocb->ki_ioprio; + dio->flags = 0; + dio->iocb = iocb; + + ret = bio_iov_iter_get_pages(bio, iter); + if (unlikely(ret)) { + bio->bi_status = BLK_STS_IOERR; + bio_endio(bio); + return BLK_STS_IOERR; + } + dio->size = bio->bi_iter.bi_size; + + if (iov_iter_rw(iter) == READ) { + bio->bi_opf = REQ_OP_READ; + if (iter_is_iovec(iter)) { + dio->flags |= DIO_SHOULD_DIRTY; + bio_set_pages_dirty(bio); + } + } else { + bio->bi_opf = dio_bio_write_op(iocb); + task_io_account_write(bio->bi_iter.bi_size); + } + + if (iocb->ki_flags & IOCB_NOWAIT) + bio->bi_opf |= REQ_NOWAIT; + /* + * Don't plug for HIPRI/polled IO, as those should go straight + * to issue + */ + if (iocb->ki_flags & IOCB_HIPRI) { + bio_set_polled(bio, iocb); + submit_bio(bio); + WRITE_ONCE(iocb->private, bio); + } else { + submit_bio(bio); + } + return -EIOCBQUEUED; +} + static ssize_t blkdev_direct_IO(struct kiocb *iocb, struct iov_iter *iter) { unsigned int nr_pages; @@ -313,9 +395,11 @@ static ssize_t blkdev_direct_IO(struct kiocb *iocb, struct iov_iter *iter) return 0; nr_pages = bio_iov_vecs_to_alloc(iter, BIO_MAX_VECS + 1); - if (is_sync_kiocb(iocb) && nr_pages <= BIO_MAX_VECS) - return __blkdev_direct_IO_simple(iocb, iter, nr_pages); - + if (likely(nr_pages <= BIO_MAX_VECS)) { + if (is_sync_kiocb(iocb)) + return __blkdev_direct_IO_simple(iocb, iter, nr_pages); + return __blkdev_direct_IO_async(iocb, iter, nr_pages); + } return __blkdev_direct_IO(iocb, iter, bio_max_segs(nr_pages)); } From patchwork Tue Oct 19 21:24:22 2021 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Pavel Begunkov X-Patchwork-Id: 12571053 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from mail.kernel.org (mail.kernel.org [198.145.29.99]) by smtp.lore.kernel.org (Postfix) with ESMTP id 6CF4BC43217 for ; Tue, 19 Oct 2021 21:24:47 +0000 (UTC) Received: from vger.kernel.org (vger.kernel.org [23.128.96.18]) by mail.kernel.org (Postfix) with ESMTP id 5AF936136A for ; Tue, 19 Oct 2021 21:24:47 +0000 (UTC) Received: (majordomo@vger.kernel.org) by vger.kernel.org via listexpand id S229801AbhJSV07 (ORCPT ); Tue, 19 Oct 2021 17:26:59 -0400 Received: from lindbergh.monkeyblade.net ([23.128.96.19]:47046 "EHLO lindbergh.monkeyblade.net" rhost-flags-OK-OK-OK-OK) by vger.kernel.org with ESMTP id S229823AbhJSV06 (ORCPT ); Tue, 19 Oct 2021 17:26:58 -0400 Received: from mail-wm1-x331.google.com (mail-wm1-x331.google.com [IPv6:2a00:1450:4864:20::331]) by lindbergh.monkeyblade.net (Postfix) with ESMTPS id 0C18AC06161C for ; Tue, 19 Oct 2021 14:24:44 -0700 (PDT) Received: by mail-wm1-x331.google.com with SMTP id g39so10393320wmp.3 for ; Tue, 19 Oct 2021 14:24:44 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=20210112; h=from:to:cc:subject:date:message-id:in-reply-to:references :mime-version:content-transfer-encoding; bh=BAAEHR7unTmV3lAE/H9zviB08MujT1fS/ufxdxgv4h8=; b=jS4Ny2KcMg5TTsdHrVrjpQsbCJYujbQRK7VWCVU9Z2x2Le+rpboNkMGPxFD/bdPTu0 eb2ZFYncBA5VehpLjpMEqD6jgzlv/SluF18hmrj/TzlfkoKcyWxdduE7ZlYiL7x9Q7XE DvEYAJUIQ4yKXiu8i/qBbKRVu9m9eY+Aa/i3yQRN0l0Yzb/hIa/4oJdewduUNqkTMMsk MZjm4azcsZUgmRUCURmb3y3T2ewXeYvY1bT+skdPm5sWKB6dmz5hkFbYMMhSROe+h1wC uEGUYuZFJcPlBgx3XY4/Ugn9X0/YCG7lZdz5wcJgrOZB6R2w3pkNY+UPRChvHJ6Irovh 7AoQ== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20210112; h=x-gm-message-state:from:to:cc:subject:date:message-id:in-reply-to :references:mime-version:content-transfer-encoding; bh=BAAEHR7unTmV3lAE/H9zviB08MujT1fS/ufxdxgv4h8=; b=TEtzBMjLu/HwxbjBDzwK0OGfljMWZi3IKmvgk4W98TgeupPxCw1MdpO8yEMZmbA/f5 hWDErNf+tVJP4XfzUyTlLCNaklsqQFpDqDtyqFLG4JXgb42f6YfwZPleqcC6KjaqTkFk s/Jm2RxsqbweLe/B8Ux5G/OmkxDm1tXtdeBo4KDMDFrIgTQJieAd+91rzjX6dwWxdGLl TRds4y7E0EuIs/E/xgZxhG7tRvYuZFjTmSTmzw3V/2knp8VwRW54Yct6w06Rt+3VFIjf qKz4RgjcKWktqM1Srb6UhVuHezB2RPTRRL6VL0TOiQySQ8Iz3eTemG2sMYcGGXs+/KLj QnMg== X-Gm-Message-State: AOAM531pcPNaKH8l9dmTAvjwmTd/OcbUfEQRJzj8+43Okzht5H4RpRDk K+s7val/mn8+0l+futdCXBSG7MwqHFBvCQ== X-Google-Smtp-Source: ABdhPJytfxRGDCCGKdewe7Vi6WWDSlCavAi5rb0Vz9G9NCfat5PYXJupGhmZfIWoBDnDb2os108dBA== X-Received: by 2002:a5d:6442:: with SMTP id d2mr3543633wrw.356.1634678683415; Tue, 19 Oct 2021 14:24:43 -0700 (PDT) Received: from 127.0.0.1localhost ([185.69.145.194]) by smtp.gmail.com with ESMTPSA id m14sm216020wms.25.2021.10.19.14.24.42 (version=TLS1_3 cipher=TLS_AES_256_GCM_SHA384 bits=256/256); Tue, 19 Oct 2021 14:24:43 -0700 (PDT) From: Pavel Begunkov To: linux-block@vger.kernel.org Cc: Jens Axboe , Pavel Begunkov Subject: [PATCH 13/16] block: add async version of bio_set_polled Date: Tue, 19 Oct 2021 22:24:22 +0100 Message-Id: <673fc6ca8f2e761586d21c709348642113f13f86.1634676157.git.asml.silence@gmail.com> X-Mailer: git-send-email 2.33.1 In-Reply-To: References: MIME-Version: 1.0 Precedence: bulk List-ID: X-Mailing-List: linux-block@vger.kernel.org If we know that a iocb is async we can optimise bio_set_polled() a bit, add a new helper bio_set_polled_async(). Signed-off-by: Pavel Begunkov --- block/fops.c | 6 +++--- include/linux/bio.h | 5 +++++ 2 files changed, 8 insertions(+), 3 deletions(-) diff --git a/block/fops.c b/block/fops.c index 0f1332374756..ee27ffbdd018 100644 --- a/block/fops.c +++ b/block/fops.c @@ -371,17 +371,17 @@ static ssize_t __blkdev_direct_IO_async(struct kiocb *iocb, task_io_account_write(bio->bi_iter.bi_size); } - if (iocb->ki_flags & IOCB_NOWAIT) - bio->bi_opf |= REQ_NOWAIT; /* * Don't plug for HIPRI/polled IO, as those should go straight * to issue */ if (iocb->ki_flags & IOCB_HIPRI) { - bio_set_polled(bio, iocb); + bio_set_polled_async(bio, iocb); submit_bio(bio); WRITE_ONCE(iocb->private, bio); } else { + if (iocb->ki_flags & IOCB_NOWAIT) + bio->bi_opf |= REQ_NOWAIT; submit_bio(bio); } return -EIOCBQUEUED; diff --git a/include/linux/bio.h b/include/linux/bio.h index 0ab4fa2c89c3..4043e0774b89 100644 --- a/include/linux/bio.h +++ b/include/linux/bio.h @@ -743,6 +743,11 @@ static inline void bio_set_polled(struct bio *bio, struct kiocb *kiocb) bio->bi_opf |= REQ_NOWAIT; } +static inline void bio_set_polled_async(struct bio *bio, struct kiocb *kiocb) +{ + bio->bi_opf |= REQ_POLLED | REQ_NOWAIT; +} + struct bio *blk_next_bio(struct bio *bio, unsigned int nr_pages, gfp_t gfp); #endif /* __LINUX_BIO_H */ From patchwork Tue Oct 19 21:24:23 2021 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Pavel Begunkov X-Patchwork-Id: 12571055 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from mail.kernel.org (mail.kernel.org [198.145.29.99]) by smtp.lore.kernel.org (Postfix) with ESMTP id BA202C433EF for ; Tue, 19 Oct 2021 21:24:49 +0000 (UTC) Received: from vger.kernel.org (vger.kernel.org [23.128.96.18]) by mail.kernel.org (Postfix) with ESMTP id A85E96134F for ; Tue, 19 Oct 2021 21:24:49 +0000 (UTC) Received: (majordomo@vger.kernel.org) by vger.kernel.org via listexpand id S229784AbhJSV1C (ORCPT ); Tue, 19 Oct 2021 17:27:02 -0400 Received: from lindbergh.monkeyblade.net ([23.128.96.19]:47032 "EHLO lindbergh.monkeyblade.net" rhost-flags-OK-OK-OK-OK) by vger.kernel.org with ESMTP id S229828AbhJSV07 (ORCPT ); Tue, 19 Oct 2021 17:26:59 -0400 Received: from mail-wm1-x32c.google.com (mail-wm1-x32c.google.com [IPv6:2a00:1450:4864:20::32c]) by lindbergh.monkeyblade.net (Postfix) with ESMTPS id B7155C061765 for ; Tue, 19 Oct 2021 14:24:45 -0700 (PDT) Received: by mail-wm1-x32c.google.com with SMTP id 67-20020a1c1946000000b0030d4c90fa87so5636597wmz.2 for ; Tue, 19 Oct 2021 14:24:45 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=20210112; h=from:to:cc:subject:date:message-id:in-reply-to:references :mime-version:content-transfer-encoding; bh=N3Wj0OHFHwFZGJdqptazKyvXEmGreuwbNc6lbiDxxRU=; b=aPSgmPYCWHFgO3iYQgOC2MuksSqldQpWfH2pCWAyPYvf6gXzgyyLcCoa8LENPz7ORs Ms65vmGi/S1/b6ZWxiT3qhAZNY17iDs/dbWPSH4thn19FFjb5fycLId2xImN+ogVJGTP OyJa4NP5mteiv1mHJf/5+Vg+XZPVVpzN0b3+jKaJUltinzr3gP1hKmvMrtBAu/9W56/v TdqxXVaeODupV1AXZ3c7ztvJ9iptFUS2NzFMD6CWZx+ZpYpdtXp7dalwcAL2umK8AOOp WwM83p2PzU3/GbyzZpJ8L0voYR2KFLO3vOzjLZV5kcMRqXI4d7pkuu7mHXuGLoEQuqdA vx2g== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20210112; h=x-gm-message-state:from:to:cc:subject:date:message-id:in-reply-to :references:mime-version:content-transfer-encoding; bh=N3Wj0OHFHwFZGJdqptazKyvXEmGreuwbNc6lbiDxxRU=; b=Qszv6x/F1R/shv+ub/1QxMFX3De9CHiFhk0f7qvgIP3+H0O1W/bQ9DD06oArP/x4wx rLszBN3mY2Idhaja8fgPPlkcGv4aJ4WfAzLpU3UneJ8ETodW5dJz14/vGWZ8MW5RT8T9 IFjqSUqjfiW9Wrkozljd/5dy8TqF6q9r7KJgPXe84NPf/D/tp9WN6VWwvyhuKfs2virb bBy+e6jI/1LPcsxPcOYmue3vyDt6ae8GtxPezNPnNhqM2vmVKx/qg8x1R7H8AYZTajPA qlRHWUM4oLwtr449SbGQFiETU12rWwnUEprBO2FHN/XwmmTw/CNwgiwkjuPQZ18GUR14 xdwQ== X-Gm-Message-State: AOAM533snoWMu6flXM1SRaX9P/EnkOsxDW7n+vcbjN/2si7Gvib21RYz 2TvEWYtmfv8uB9zz9PveCGtov/O/c8HcPA== X-Google-Smtp-Source: ABdhPJzR9YRjEn8X9GpkUWyMa7jgdn1M9EEkj3JAnIjDjEsVA2ZMkfxvIh2hfF7RcZwZw7dYVQ30Lw== X-Received: by 2002:a7b:c351:: with SMTP id l17mr9029233wmj.120.1634678684146; Tue, 19 Oct 2021 14:24:44 -0700 (PDT) Received: from 127.0.0.1localhost ([185.69.145.194]) by smtp.gmail.com with ESMTPSA id m14sm216020wms.25.2021.10.19.14.24.43 (version=TLS1_3 cipher=TLS_AES_256_GCM_SHA384 bits=256/256); Tue, 19 Oct 2021 14:24:43 -0700 (PDT) From: Pavel Begunkov To: linux-block@vger.kernel.org Cc: Jens Axboe , Pavel Begunkov Subject: [PATCH 14/16] block: skip advance when async and not needed Date: Tue, 19 Oct 2021 22:24:23 +0100 Message-Id: <48fd2fe9d0367620ceda34b79857892841f18668.1634676157.git.asml.silence@gmail.com> X-Mailer: git-send-email 2.33.1 In-Reply-To: References: MIME-Version: 1.0 Precedence: bulk List-ID: X-Mailing-List: linux-block@vger.kernel.org Nobody cares about iov iterators state if we return -EIOCBQUEUED, so as the we now have __blkdev_direct_IO_async(), which gets pages only once, we can skip expensive iov_iter_advance(). It's around 1-2% of all CPU spent. Signed-off-by: Pavel Begunkov --- block/bio.c | 13 ++++++++----- block/fops.c | 2 +- include/linux/bio.h | 9 ++++++++- 3 files changed, 17 insertions(+), 7 deletions(-) diff --git a/block/bio.c b/block/bio.c index 46a87c72d2b4..0ed836e98734 100644 --- a/block/bio.c +++ b/block/bio.c @@ -1058,10 +1058,12 @@ static void __bio_iov_bvec_set(struct bio *bio, struct iov_iter *iter) bio_set_flag(bio, BIO_CLONED); } -static int bio_iov_bvec_set(struct bio *bio, struct iov_iter *iter) +static int bio_iov_bvec_set(struct bio *bio, struct iov_iter *iter, + bool hint_skip_advance) { __bio_iov_bvec_set(bio, iter); - iov_iter_advance(iter, iter->count); + if (!hint_skip_advance) + iov_iter_advance(iter, iter->count); return 0; } @@ -1212,14 +1214,15 @@ static int __bio_iov_append_get_pages(struct bio *bio, struct iov_iter *iter) * It's intended for direct IO, so doesn't do PSI tracking, the caller is * responsible for setting BIO_WORKINGSET if necessary. */ -int bio_iov_iter_get_pages(struct bio *bio, struct iov_iter *iter) +int bio_iov_iter_get_pages_hint(struct bio *bio, struct iov_iter *iter, + bool hint_skip_advance) { int ret = 0; if (iov_iter_is_bvec(iter)) { if (bio_op(bio) == REQ_OP_ZONE_APPEND) return bio_iov_bvec_set_append(bio, iter); - return bio_iov_bvec_set(bio, iter); + return bio_iov_bvec_set(bio, iter, hint_skip_advance); } do { @@ -1233,7 +1236,7 @@ int bio_iov_iter_get_pages(struct bio *bio, struct iov_iter *iter) bio_clear_flag(bio, BIO_WORKINGSET); return bio->bi_vcnt ? 0 : ret; } -EXPORT_SYMBOL_GPL(bio_iov_iter_get_pages); +EXPORT_SYMBOL_GPL(bio_iov_iter_get_pages_hint); static void submit_bio_wait_endio(struct bio *bio) { diff --git a/block/fops.c b/block/fops.c index ee27ffbdd018..d4c770c5085b 100644 --- a/block/fops.c +++ b/block/fops.c @@ -352,7 +352,7 @@ static ssize_t __blkdev_direct_IO_async(struct kiocb *iocb, dio->flags = 0; dio->iocb = iocb; - ret = bio_iov_iter_get_pages(bio, iter); + ret = bio_iov_iter_get_pages_hint(bio, iter, true); if (unlikely(ret)) { bio->bi_status = BLK_STS_IOERR; bio_endio(bio); diff --git a/include/linux/bio.h b/include/linux/bio.h index 4043e0774b89..51413fe33720 100644 --- a/include/linux/bio.h +++ b/include/linux/bio.h @@ -416,7 +416,8 @@ int bio_add_zone_append_page(struct bio *bio, struct page *page, unsigned int len, unsigned int offset); void __bio_add_page(struct bio *bio, struct page *page, unsigned int len, unsigned int off); -int bio_iov_iter_get_pages(struct bio *bio, struct iov_iter *iter); +int bio_iov_iter_get_pages_hint(struct bio *bio, struct iov_iter *iter, + bool hint_skip_advance); void __bio_release_pages(struct bio *bio, bool mark_dirty); extern void bio_set_pages_dirty(struct bio *bio); extern void bio_check_pages_dirty(struct bio *bio); @@ -428,6 +429,12 @@ extern void bio_free_pages(struct bio *bio); void guard_bio_eod(struct bio *bio); void zero_fill_bio(struct bio *bio); +static inline int bio_iov_iter_get_pages(struct bio *bio, + struct iov_iter *iter) +{ + return bio_iov_iter_get_pages_hint(bio, iter, false); +} + static inline void bio_release_pages(struct bio *bio, bool mark_dirty) { if (!bio_flagged(bio, BIO_NO_PAGE_REF)) From patchwork Tue Oct 19 21:24:24 2021 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Pavel Begunkov X-Patchwork-Id: 12571057 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from mail.kernel.org (mail.kernel.org [198.145.29.99]) by smtp.lore.kernel.org (Postfix) with ESMTP id 85527C433FE for ; Tue, 19 Oct 2021 21:24:51 +0000 (UTC) Received: from vger.kernel.org (vger.kernel.org [23.128.96.18]) by mail.kernel.org (Postfix) with ESMTP id 6D4C360F46 for ; Tue, 19 Oct 2021 21:24:51 +0000 (UTC) Received: (majordomo@vger.kernel.org) by vger.kernel.org via listexpand id S229881AbhJSV1D (ORCPT ); Tue, 19 Oct 2021 17:27:03 -0400 Received: from lindbergh.monkeyblade.net ([23.128.96.19]:47066 "EHLO lindbergh.monkeyblade.net" rhost-flags-OK-OK-OK-OK) by vger.kernel.org with ESMTP id S229890AbhJSV1A (ORCPT ); Tue, 19 Oct 2021 17:27:00 -0400 Received: from mail-wm1-x335.google.com (mail-wm1-x335.google.com [IPv6:2a00:1450:4864:20::335]) by lindbergh.monkeyblade.net (Postfix) with ESMTPS id 98366C061769 for ; Tue, 19 Oct 2021 14:24:46 -0700 (PDT) Received: by mail-wm1-x335.google.com with SMTP id 63-20020a1c0042000000b0030d60716239so5646918wma.4 for ; Tue, 19 Oct 2021 14:24:46 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=20210112; h=from:to:cc:subject:date:message-id:in-reply-to:references :mime-version:content-transfer-encoding; bh=MLzdRAYlFpXAuyX/ejQgU4JIAricFq29j5NgCWLoWFQ=; b=jjJCSe2kc5Nxnbqf80nrfoRN1PFPXQLedsRyVRDlsuJ7lU1+l+EFjyO59Y7DjbDFil pGziT2Bw9/pPEdhVL6U+ZlGC55Nd9cmLNNJBoRh+hxYcsnfgVruOJPL2LSstHJJYJMOg 8dknfy+mSbQvnCeJqtSoeWUteCp/iFLY980E8KGFhCVwhA2xY8yoA1O2n4t51MzJe43X vPnJnkPC9xdO2tO35FYu1KgJ2vs4tfF1xxsj+8s1vgxaB896+WU1Ta80h68CdUrOdwmK atHfE7mJidQ5EE6cl0OrAmQLjw+uxFF086tTE+GgFWsg+h8YJIyQj3idZim7/k40yhgL TS2w== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20210112; h=x-gm-message-state:from:to:cc:subject:date:message-id:in-reply-to :references:mime-version:content-transfer-encoding; bh=MLzdRAYlFpXAuyX/ejQgU4JIAricFq29j5NgCWLoWFQ=; b=zDZFwWvVBZj7FdZpH9UgFyPsU98O/uGAOjdON3YoeITZM9dHRIqwKUdNrMCTlY5JNI DqO4Q/PSYXyIRua0vzWVsrhdKp/pk9+TNeMAIPU1BBpmzs1Wvfj5ibAgnf4wxk0f+6wD nZlBQ/MQQozowMyGwnrL+vcIB6C1+5Le22pu3k22qyO9Tp/lSSoNMCUhW1VchFUfWI9y cDQYwxXw4sVFBoZtUNaTOu0BOAMAbkuAI5mOuTqT18vkhU8dI06XG70LF/vULhNEvuec 5wKAZpfp2hwNty3E7FxF/cMlhMncuVnm7epY2qBeC6m9C3D2njpQYFKZKmRCgZ/gl2Bo A0Fg== X-Gm-Message-State: AOAM531+p+Uum46c8Nj+lMSCxlQ99FFx5but9sjnH4vS/PFlVaYMqvtZ mpbPMUioZCgWGlVCKm1GgroWIdWbVuZuNw== X-Google-Smtp-Source: ABdhPJzOMbQsmR3Nnhi+3GI8AsYibqfLuk3p3LpTSDiMuZJ+1+G/t4Veplwijaqpo5VJJMzf6yG4HA== X-Received: by 2002:a05:6000:1541:: with SMTP id 1mr45665720wry.273.1634678685041; Tue, 19 Oct 2021 14:24:45 -0700 (PDT) Received: from 127.0.0.1localhost ([185.69.145.194]) by smtp.gmail.com with ESMTPSA id m14sm216020wms.25.2021.10.19.14.24.44 (version=TLS1_3 cipher=TLS_AES_256_GCM_SHA384 bits=256/256); Tue, 19 Oct 2021 14:24:44 -0700 (PDT) From: Pavel Begunkov To: linux-block@vger.kernel.org Cc: Jens Axboe , Pavel Begunkov Subject: [PATCH 15/16] block: optimise blk_may_split for normal rw Date: Tue, 19 Oct 2021 22:24:24 +0100 Message-Id: <9ded7cf6a3af7e6e577d12a835a385657da4a69e.1634676157.git.asml.silence@gmail.com> X-Mailer: git-send-email 2.33.1 In-Reply-To: References: MIME-Version: 1.0 Precedence: bulk List-ID: X-Mailing-List: linux-block@vger.kernel.org Read/write/flush are the most common operations, optimise switch in blk_may_split() for these cases. All three added conditions are compiled into a single comparison as the corresponding REQ_OP_* take 0-2. Signed-off-by: Pavel Begunkov --- block/blk.h | 20 ++++++++++++-------- 1 file changed, 12 insertions(+), 8 deletions(-) diff --git a/block/blk.h b/block/blk.h index 6a039e6c7d07..0bf00e96e1f0 100644 --- a/block/blk.h +++ b/block/blk.h @@ -269,14 +269,18 @@ ssize_t part_timeout_store(struct device *, struct device_attribute *, static inline bool blk_may_split(struct request_queue *q, struct bio *bio) { - switch (bio_op(bio)) { - case REQ_OP_DISCARD: - case REQ_OP_SECURE_ERASE: - case REQ_OP_WRITE_ZEROES: - case REQ_OP_WRITE_SAME: - return true; /* non-trivial splitting decisions */ - default: - break; + unsigned int op = bio_op(bio); + + if (op != REQ_OP_READ && op != REQ_OP_WRITE && op != REQ_OP_FLUSH) { + switch (op) { + case REQ_OP_DISCARD: + case REQ_OP_SECURE_ERASE: + case REQ_OP_WRITE_ZEROES: + case REQ_OP_WRITE_SAME: + return true; /* non-trivial splitting decisions */ + default: + break; + } } /* From patchwork Tue Oct 19 21:24:25 2021 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Pavel Begunkov X-Patchwork-Id: 12571059 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from mail.kernel.org (mail.kernel.org [198.145.29.99]) by smtp.lore.kernel.org (Postfix) with ESMTP id 9B278C433EF for ; Tue, 19 Oct 2021 21:24:53 +0000 (UTC) Received: from vger.kernel.org (vger.kernel.org [23.128.96.18]) by mail.kernel.org (Postfix) with ESMTP id 852A66135E for ; Tue, 19 Oct 2021 21:24:53 +0000 (UTC) Received: (majordomo@vger.kernel.org) by vger.kernel.org via listexpand id S229889AbhJSV1E (ORCPT ); Tue, 19 Oct 2021 17:27:04 -0400 Received: from lindbergh.monkeyblade.net ([23.128.96.19]:47084 "EHLO lindbergh.monkeyblade.net" rhost-flags-OK-OK-OK-OK) by vger.kernel.org with ESMTP id S229899AbhJSV1B (ORCPT ); Tue, 19 Oct 2021 17:27:01 -0400 Received: from mail-wm1-x329.google.com (mail-wm1-x329.google.com [IPv6:2a00:1450:4864:20::329]) by lindbergh.monkeyblade.net (Postfix) with ESMTPS id 86916C06176D for ; Tue, 19 Oct 2021 14:24:47 -0700 (PDT) Received: by mail-wm1-x329.google.com with SMTP id 63-20020a1c0042000000b0030d60716239so5646990wma.4 for ; Tue, 19 Oct 2021 14:24:47 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=20210112; h=from:to:cc:subject:date:message-id:in-reply-to:references :mime-version:content-transfer-encoding; bh=+EgGdcRessLiI1FhX2RaNc12KdVxW/zeIH1xYRNQfwU=; b=XH5xoa5wD9VRDzEjzmk2g+2N0zkPJThsMygbKK77vzMjOMGOCiUEEldIbKSEjEvtWG V8WLjxJJoQTCHWsdAJd9f7iJSfpRQo3MWHxkOhOdq5MBkdSiiuufDsVsr+rbtjvmDFhL SvEvNNdMhJs10mCdKRPtpdtXPrt8ie2rebLXNsQxD2TTw0Bd5itpgxYQTU6UPwxBW6mr 6EhydPnQBmDpaNpH0dpAeCU0J2nNPnr86/yxWK0ROs0+60I7sHSIxheAwSwlpEfVgcOm 4Zs2c1Xebb2ICVB+1FAe4S0NMM4hCUF0sB2hJ1qJjsGk4PNzAkloN66r64Ow2WlBWl24 mOPw== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20210112; h=x-gm-message-state:from:to:cc:subject:date:message-id:in-reply-to :references:mime-version:content-transfer-encoding; bh=+EgGdcRessLiI1FhX2RaNc12KdVxW/zeIH1xYRNQfwU=; b=Mv3EtpFN7s0K4C+e8aVydEN702Y6cGgTdbueCbQ2EKyQeO44g1xchLqjFYsvdUY4EC F409iIeqAxu8237ogJz7sZBz2uLi9KmwovLQkHlGrYLInfXwGZI9KoQF9Kq0RSbcZzwY nslRR49ih8NGZ+r091UptzWrPRAfe1VrhiJY4sDcbdpl/Nk1RcEkI2Vbl+G+0fDLDTfo 1KCmIbc8b4IrxDypW8h6U7wFHuJPrXN+aDBiWA6hsa0CdUhazAtFFdkfKVSZjqprFCeb AQ1U/N3bDaodNbJ6ORhjoqW90O8j1z4IWxBvnDY1eG4zgikH670mz2obiIkGDrPupwtY ZklA== X-Gm-Message-State: AOAM5308JThlE4mdRtUwfR6RqcvpopBUKYT8w+wdx6u/OcCy54ZpwysY EtVqAMTB3lC5Ykk7cmjumZkBRtrQ9VGlgw== X-Google-Smtp-Source: ABdhPJyTAwlnxG+msw3uUQnKbsSzM6FifGXac6J07zBoq0xL1pdzh7GGtlG6zSjYcyE6PihvEpKHfQ== X-Received: by 2002:a05:600c:154f:: with SMTP id f15mr9312837wmg.195.1634678685981; Tue, 19 Oct 2021 14:24:45 -0700 (PDT) Received: from 127.0.0.1localhost ([185.69.145.194]) by smtp.gmail.com with ESMTPSA id m14sm216020wms.25.2021.10.19.14.24.45 (version=TLS1_3 cipher=TLS_AES_256_GCM_SHA384 bits=256/256); Tue, 19 Oct 2021 14:24:45 -0700 (PDT) From: Pavel Begunkov To: linux-block@vger.kernel.org Cc: Jens Axboe , Pavel Begunkov Subject: [PATCH 16/16] block: optimise submit_bio_checks for normal rw Date: Tue, 19 Oct 2021 22:24:25 +0100 Message-Id: X-Mailer: git-send-email 2.33.1 In-Reply-To: References: MIME-Version: 1.0 Precedence: bulk List-ID: X-Mailing-List: linux-block@vger.kernel.org Optimise the switch in submit_bio_checks() for reads, writes and flushes. REQ_OP_READ/WRITE/FLUSH take numbers from 0 to 2, so the added checks are compiled into a single condition: if (op <= REQ_OP_FLUSH) {} else { switch() ... }; Signed-off-by: Pavel Begunkov --- block/blk-core.c | 74 +++++++++++++++++++++++++----------------------- 1 file changed, 39 insertions(+), 35 deletions(-) diff --git a/block/blk-core.c b/block/blk-core.c index 52019b8a1487..7ba8f53a8340 100644 --- a/block/blk-core.c +++ b/block/blk-core.c @@ -776,6 +776,7 @@ static noinline_for_stack bool submit_bio_checks(struct bio *bio) struct request_queue *q = bdev_get_queue(bdev); blk_status_t status = BLK_STS_IOERR; struct blk_plug *plug; + unsigned op; might_sleep(); @@ -817,41 +818,44 @@ static noinline_for_stack bool submit_bio_checks(struct bio *bio) if (!test_bit(QUEUE_FLAG_POLL, &q->queue_flags)) bio_clear_polled(bio); - switch (bio_op(bio)) { - case REQ_OP_DISCARD: - if (!blk_queue_discard(q)) - goto not_supported; - break; - case REQ_OP_SECURE_ERASE: - if (!blk_queue_secure_erase(q)) - goto not_supported; - break; - case REQ_OP_WRITE_SAME: - if (!q->limits.max_write_same_sectors) - goto not_supported; - break; - case REQ_OP_ZONE_APPEND: - status = blk_check_zone_append(q, bio); - if (status != BLK_STS_OK) - goto end_io; - break; - case REQ_OP_ZONE_RESET: - case REQ_OP_ZONE_OPEN: - case REQ_OP_ZONE_CLOSE: - case REQ_OP_ZONE_FINISH: - if (!blk_queue_is_zoned(q)) - goto not_supported; - break; - case REQ_OP_ZONE_RESET_ALL: - if (!blk_queue_is_zoned(q) || !blk_queue_zone_resetall(q)) - goto not_supported; - break; - case REQ_OP_WRITE_ZEROES: - if (!q->limits.max_write_zeroes_sectors) - goto not_supported; - break; - default: - break; + op = bio_op(bio); + if (op != REQ_OP_READ && op != REQ_OP_WRITE && op != REQ_OP_FLUSH) { + switch (op) { + case REQ_OP_DISCARD: + if (!blk_queue_discard(q)) + goto not_supported; + break; + case REQ_OP_SECURE_ERASE: + if (!blk_queue_secure_erase(q)) + goto not_supported; + break; + case REQ_OP_WRITE_SAME: + if (!q->limits.max_write_same_sectors) + goto not_supported; + break; + case REQ_OP_ZONE_APPEND: + status = blk_check_zone_append(q, bio); + if (status != BLK_STS_OK) + goto end_io; + break; + case REQ_OP_ZONE_RESET: + case REQ_OP_ZONE_OPEN: + case REQ_OP_ZONE_CLOSE: + case REQ_OP_ZONE_FINISH: + if (!blk_queue_is_zoned(q)) + goto not_supported; + break; + case REQ_OP_ZONE_RESET_ALL: + if (!blk_queue_is_zoned(q) || !blk_queue_zone_resetall(q)) + goto not_supported; + break; + case REQ_OP_WRITE_ZEROES: + if (!q->limits.max_write_zeroes_sectors) + goto not_supported; + break; + default: + break; + } } /*