From patchwork Sat Nov 10 15:13:16 2018 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Jens Axboe X-Patchwork-Id: 10677247 Return-Path: Received: from mail.wl.linuxfoundation.org (pdx-wl-mail.web.codeaurora.org [172.30.200.125]) by pdx-korg-patchwork-2.web.codeaurora.org (Postfix) with ESMTP id 04E90139B for ; Sat, 10 Nov 2018 15:13:33 +0000 (UTC) Received: from mail.wl.linuxfoundation.org (localhost [127.0.0.1]) by mail.wl.linuxfoundation.org (Postfix) with ESMTP id E8B99287D2 for ; Sat, 10 Nov 2018 15:13:32 +0000 (UTC) Received: by mail.wl.linuxfoundation.org (Postfix, from userid 486) id DD0032900D; Sat, 10 Nov 2018 15:13:32 +0000 (UTC) X-Spam-Checker-Version: SpamAssassin 3.3.1 (2010-03-16) on pdx-wl-mail.web.codeaurora.org X-Spam-Level: X-Spam-Status: No, score=-7.9 required=2.0 tests=BAYES_00,DKIM_SIGNED, DKIM_VALID,MAILING_LIST_MULTI,RCVD_IN_DNSWL_HI autolearn=ham version=3.3.1 Received: from vger.kernel.org (vger.kernel.org [209.132.180.67]) by mail.wl.linuxfoundation.org (Postfix) with ESMTP id 4F272287D2 for ; Sat, 10 Nov 2018 15:13:32 +0000 (UTC) Received: (majordomo@vger.kernel.org) by vger.kernel.org via listexpand id S1726335AbeKKA6u (ORCPT ); Sat, 10 Nov 2018 19:58:50 -0500 Received: from mail-pl1-f193.google.com ([209.85.214.193]:41348 "EHLO mail-pl1-f193.google.com" rhost-flags-OK-OK-OK-OK) by vger.kernel.org with ESMTP id S1726188AbeKKA6u (ORCPT ); Sat, 10 Nov 2018 19:58:50 -0500 Received: by mail-pl1-f193.google.com with SMTP id p16-v6so2259056plr.8 for ; Sat, 10 Nov 2018 07:13:31 -0800 (PST) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=kernel-dk.20150623.gappssmtp.com; s=20150623; h=from:to:cc:subject:date:message-id:in-reply-to:references; bh=gngy6ndCqerfPaGct02L9pdutncRCYm4yRPgtxDKoxs=; b=ucKpTIo+/7vlC9wWO8Mi0D/6qZA2L0nbNvv7ClGPOLJLYvKQXFf04cLfC5mfuMsTIp oLxK/c+yRiQJf4ooX8sWHzaXnty/jMkefKGGceOTgLq2xNu2vvLVREq+J9kKa6+WPTFW MoBrKCPwrMV5g4ZIR1ngzwu8QsE9+MWcIZCNCZqxFpXslQLS8ENVjvCwZhE09CwBCztW wVxQaLFCD11KUnmldGCJdfYf0xsrULhQ2IIMQ3t2fxI2t2Wl05yPT4azU261KXbgUadX yWlUyhlqIEsdB3WchZID2o1U1Q439ZScoxSEnMdkACC30feGLGOaI0eT22kX3RUMv4HQ uGEw== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20161025; h=x-gm-message-state:from:to:cc:subject:date:message-id:in-reply-to :references; bh=gngy6ndCqerfPaGct02L9pdutncRCYm4yRPgtxDKoxs=; b=tJfWRtf7sOYEqRY3at6FDBOJn8if5RXtj/pZJNNmvwVQKrrHvqwx1SFz6FiNlTDMCE +4mB8X+yjV+CMhpRX748IYwvTLSkYuRqGAV0YOamEmrdFATYkTfPt0NeHkWWRzuKecvo YzWy6DeaXGGau1jOjzLBVgdOe9p5xmJ3TlbTeo5fZrT0uxfQwTE85l3VYFp30R0dCMSt pJAdm3mCCXf/oNrDr+3+5UtVfWoNu/pn6SO6xLleOlG5zpa1movg2Kthvu14nV8M2otR 4Q1qPgpNAMWdQbZyyRp24c4yjVU/E5SVoTY7u/7IM8r3VdCroUTcbHrb7gEgcQlbsq/I 9+jg== X-Gm-Message-State: AGRZ1gI3UCLbDYu/lOFasbxxRzmwytrkWoZVPPqU+b+aYlYFxsMA5bK6 rAxnUJlhwkCbf96oN7kziUQUmZv7xn4= X-Google-Smtp-Source: AJdET5c9VyueCJ2zJTv+y62D7OppY4a/8+x48Vccbxs8JkUEUH3hZzx4sj5rO2qP7LnsQgNmjKvg7g== X-Received: by 2002:a17:902:f097:: with SMTP id go23mr12792663plb.328.1541862810651; Sat, 10 Nov 2018 07:13:30 -0800 (PST) Received: from x1.localdomain (66.29.188.166.static.utbb.net. [66.29.188.166]) by smtp.gmail.com with ESMTPSA id d68-v6sm11178615pfa.80.2018.11.10.07.13.28 (version=TLS1_2 cipher=ECDHE-RSA-AES128-GCM-SHA256 bits=128/128); Sat, 10 Nov 2018 07:13:29 -0800 (PST) From: Jens Axboe To: linux-block@vger.kernel.org Cc: Jens Axboe , Josef Bacik Subject: [PATCH 5/6] blk-rq-qos: inline check for q->rq_qos functions Date: Sat, 10 Nov 2018 08:13:16 -0700 Message-Id: <20181110151317.3813-6-axboe@kernel.dk> X-Mailer: git-send-email 2.17.1 In-Reply-To: <20181110151317.3813-1-axboe@kernel.dk> References: <20181110151317.3813-1-axboe@kernel.dk> Sender: linux-block-owner@vger.kernel.org Precedence: bulk List-ID: X-Mailing-List: linux-block@vger.kernel.org X-Virus-Scanned: ClamAV using ClamSMTP Put the short code in the fast path, where we don't have any functions attached to the queue. This minimizes the impact on the hot path in the core code. Cleanup duplicated code by having a macro setup both the inline check and the actual functions. Cc: Josef Bacik Signed-off-by: Jens Axboe --- block/blk-rq-qos.c | 90 +++++++++++++--------------------------------- block/blk-rq-qos.h | 35 ++++++++++++++---- 2 files changed, 52 insertions(+), 73 deletions(-) diff --git a/block/blk-rq-qos.c b/block/blk-rq-qos.c index 0005dfd568dd..266c9e111475 100644 --- a/block/blk-rq-qos.c +++ b/block/blk-rq-qos.c @@ -27,76 +27,34 @@ bool rq_wait_inc_below(struct rq_wait *rq_wait, unsigned int limit) return atomic_inc_below(&rq_wait->inflight, limit); } -void rq_qos_cleanup(struct request_queue *q, struct bio *bio) -{ - struct rq_qos *rqos; - - for (rqos = q->rq_qos; rqos; rqos = rqos->next) { - if (rqos->ops->cleanup) - rqos->ops->cleanup(rqos, bio); - } -} - -void rq_qos_done(struct request_queue *q, struct request *rq) -{ - struct rq_qos *rqos; - - for (rqos = q->rq_qos; rqos; rqos = rqos->next) { - if (rqos->ops->done) - rqos->ops->done(rqos, rq); - } -} - -void rq_qos_issue(struct request_queue *q, struct request *rq) -{ - struct rq_qos *rqos; - - for(rqos = q->rq_qos; rqos; rqos = rqos->next) { - if (rqos->ops->issue) - rqos->ops->issue(rqos, rq); - } -} - -void rq_qos_requeue(struct request_queue *q, struct request *rq) -{ - struct rq_qos *rqos; - - for(rqos = q->rq_qos; rqos; rqos = rqos->next) { - if (rqos->ops->requeue) - rqos->ops->requeue(rqos, rq); - } +#define __RQ_QOS_FUNC_ONE(__OP, type) \ +void __rq_qos_##__OP(struct rq_qos *rqos, type arg) \ +{ \ + do { \ + if ((rqos)->ops->__OP) \ + (rqos)->ops->__OP((rqos), arg); \ + (rqos) = (rqos)->next; \ + } while (rqos); \ } -void rq_qos_throttle(struct request_queue *q, struct bio *bio, - spinlock_t *lock) -{ - struct rq_qos *rqos; - - for(rqos = q->rq_qos; rqos; rqos = rqos->next) { - if (rqos->ops->throttle) - rqos->ops->throttle(rqos, bio, lock); - } +__RQ_QOS_FUNC_ONE(cleanup, struct bio *); +__RQ_QOS_FUNC_ONE(done, struct request *); +__RQ_QOS_FUNC_ONE(issue, struct request *); +__RQ_QOS_FUNC_ONE(requeue, struct request *); +__RQ_QOS_FUNC_ONE(done_bio, struct bio *); + +#define __RQ_QOS_FUNC_TWO(__OP, type1, type2) \ +void __rq_qos_##__OP(struct rq_qos *rqos, type1 arg1, type2 arg2) \ +{ \ + do { \ + if ((rqos)->ops->__OP) \ + (rqos)->ops->__OP((rqos), arg1, arg2); \ + (rqos) = (rqos)->next; \ + } while (rqos); \ } -void rq_qos_track(struct request_queue *q, struct request *rq, struct bio *bio) -{ - struct rq_qos *rqos; - - for(rqos = q->rq_qos; rqos; rqos = rqos->next) { - if (rqos->ops->track) - rqos->ops->track(rqos, rq, bio); - } -} - -void rq_qos_done_bio(struct request_queue *q, struct bio *bio) -{ - struct rq_qos *rqos; - - for(rqos = q->rq_qos; rqos; rqos = rqos->next) { - if (rqos->ops->done_bio) - rqos->ops->done_bio(rqos, bio); - } -} +__RQ_QOS_FUNC_TWO(throttle, struct bio *, spinlock_t *); +__RQ_QOS_FUNC_TWO(track, struct request *, struct bio *); /* * Return true, if we can't increase the depth further by scaling diff --git a/block/blk-rq-qos.h b/block/blk-rq-qos.h index 32b02efbfa66..50558a6ea248 100644 --- a/block/blk-rq-qos.h +++ b/block/blk-rq-qos.h @@ -98,12 +98,33 @@ void rq_depth_scale_up(struct rq_depth *rqd); void rq_depth_scale_down(struct rq_depth *rqd, bool hard_throttle); bool rq_depth_calc_max_depth(struct rq_depth *rqd); -void rq_qos_cleanup(struct request_queue *, struct bio *); -void rq_qos_done(struct request_queue *, struct request *); -void rq_qos_issue(struct request_queue *, struct request *); -void rq_qos_requeue(struct request_queue *, struct request *); -void rq_qos_done_bio(struct request_queue *q, struct bio *bio); -void rq_qos_throttle(struct request_queue *, struct bio *, spinlock_t *); -void rq_qos_track(struct request_queue *q, struct request *, struct bio *); +#define RQ_QOS_FUNC_ONE(__OP, type) \ +void __rq_qos_##__OP(struct rq_qos *rqos, type arg); \ +static inline void rq_qos_##__OP(struct request_queue *q, type arg) \ +{ \ + if ((q)->rq_qos) \ + __rq_qos_##__OP((q)->rq_qos, arg); \ +} + +#define RQ_QOS_FUNC_TWO(__OP, type1, type2) \ +void __rq_qos_##__OP(struct rq_qos *rqos, type1 arg1, type2 arg2); \ +static inline void rq_qos_##__OP(struct request_queue *q, type1 arg1, \ + type2 arg2) \ +{ \ + if ((q)->rq_qos) \ + __rq_qos_##__OP((q)->rq_qos, arg1, arg2); \ +} + +RQ_QOS_FUNC_ONE(cleanup, struct bio *); +RQ_QOS_FUNC_ONE(done, struct request *); +RQ_QOS_FUNC_ONE(issue, struct request *); +RQ_QOS_FUNC_ONE(requeue, struct request *); +RQ_QOS_FUNC_ONE(done_bio, struct bio *); +RQ_QOS_FUNC_TWO(throttle, struct bio *, spinlock_t *); +RQ_QOS_FUNC_TWO(track, struct request *, struct bio *); +#undef RQ_QOS_FUNC_ONE +#undef RQ_QOS_FUNC_TWO + void rq_qos_exit(struct request_queue *); + #endif