From patchwork Wed Jan 5 17:05:15 2022 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Keith Busch X-Patchwork-Id: 12704529 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from vger.kernel.org (vger.kernel.org [23.128.96.18]) by smtp.lore.kernel.org (Postfix) with ESMTP id 00BC7C433EF for ; Wed, 5 Jan 2022 17:05:27 +0000 (UTC) Received: (majordomo@vger.kernel.org) by vger.kernel.org via listexpand id S242089AbiAERF1 (ORCPT ); Wed, 5 Jan 2022 12:05:27 -0500 Received: from lindbergh.monkeyblade.net ([23.128.96.19]:49750 "EHLO lindbergh.monkeyblade.net" rhost-flags-OK-OK-OK-OK) by vger.kernel.org with ESMTP id S242091AbiAERF1 (ORCPT ); Wed, 5 Jan 2022 12:05:27 -0500 Received: from dfw.source.kernel.org (dfw.source.kernel.org [IPv6:2604:1380:4641:c500::1]) by lindbergh.monkeyblade.net (Postfix) with ESMTPS id D90C8C061245 for ; Wed, 5 Jan 2022 09:05:26 -0800 (PST) Received: from smtp.kernel.org (relay.kernel.org [52.25.139.140]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (No client certificate requested) by dfw.source.kernel.org (Postfix) with ESMTPS id 78E5C61831 for ; Wed, 5 Jan 2022 17:05:26 +0000 (UTC) Received: by smtp.kernel.org (Postfix) with ESMTPSA id 507CCC36AF4; Wed, 5 Jan 2022 17:05:25 +0000 (UTC) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/simple; d=kernel.org; s=k20201202; t=1641402325; bh=Qlt2OE4vBZMSSLU8g7PiBjCMLnPFQzuT5v6qcILkaNQ=; h=From:To:Cc:Subject:Date:In-Reply-To:References:From; b=jZdJtwlt9qiDvxF0WoV0FjauqJqwUXn2qhPWAJBTzo66PosVGsYA5nqyH9Huvr3oc 5BbFVgnVKBG4uoiZRC2pDt6ojcffQ53ALnokZng84gQ/eRzihCWyBW7XaqhQC7xUt1 YwvwNkrBV4ZfR+IC/XWAuG0LlP6Sa72/M1Wvc8yWtyzsb4f7ReRb/zTBY/Lhd9/xsT YWRjqcwtTMR7sUMo7TzUrWLFJE9vUQiQJ+zVmprwiNWTjTo9Ft7a/cTqoHLyoqlpvm 3JM7i5uYSPIkaX45DF8OCGGtcs40fH/iqHDimgU5O86qL6vblUNBI9+a03Q9dSaL6e Cus5R4uOxvtnw== From: Keith Busch To: linux-nvme@lists.infradead.org, linux-block@vger.kernel.org, axboe@kernel.dk Cc: hch@lst.de, sagi@grimberg.me, mgurtovoy@nvidia.com, Keith Busch Subject: [PATCHv3 1/4] block: move rq_list macros to blk-mq.h Date: Wed, 5 Jan 2022 09:05:15 -0800 Message-Id: <20220105170518.3181469-2-kbusch@kernel.org> X-Mailer: git-send-email 2.25.4 In-Reply-To: <20220105170518.3181469-1-kbusch@kernel.org> References: <20220105170518.3181469-1-kbusch@kernel.org> MIME-Version: 1.0 Precedence: bulk List-ID: X-Mailing-List: linux-block@vger.kernel.org Move the request list macros to the header file that defines that struct they operate on. Signed-off-by: Keith Busch Reviewed-by: Christoph Hellwig --- fs/io_uring.c | 2 +- include/linux/blk-mq.h | 29 +++++++++++++++++++++++++++++ include/linux/blkdev.h | 29 ----------------------------- 3 files changed, 30 insertions(+), 30 deletions(-) diff --git a/fs/io_uring.c b/fs/io_uring.c index 90002bb3fdf4..33f72b3b302c 100644 --- a/fs/io_uring.c +++ b/fs/io_uring.c @@ -57,7 +57,7 @@ #include #include #include -#include +#include #include #include #include diff --git a/include/linux/blk-mq.h b/include/linux/blk-mq.h index 550996cf419c..bf64b94cd64e 100644 --- a/include/linux/blk-mq.h +++ b/include/linux/blk-mq.h @@ -216,6 +216,35 @@ static inline unsigned short req_get_ioprio(struct request *req) #define rq_dma_dir(rq) \ (op_is_write(req_op(rq)) ? DMA_TO_DEVICE : DMA_FROM_DEVICE) +#define rq_list_add(listptr, rq) do { \ + (rq)->rq_next = *(listptr); \ + *(listptr) = rq; \ +} while (0) + +#define rq_list_pop(listptr) \ +({ \ + struct request *__req = NULL; \ + if ((listptr) && *(listptr)) { \ + __req = *(listptr); \ + *(listptr) = __req->rq_next; \ + } \ + __req; \ +}) + +#define rq_list_peek(listptr) \ +({ \ + struct request *__req = NULL; \ + if ((listptr) && *(listptr)) \ + __req = *(listptr); \ + __req; \ +}) + +#define rq_list_for_each(listptr, pos) \ + for (pos = rq_list_peek((listptr)); pos; pos = rq_list_next(pos)) + +#define rq_list_next(rq) (rq)->rq_next +#define rq_list_empty(list) ((list) == (struct request *) NULL) + enum blk_eh_timer_return { BLK_EH_DONE, /* drivers has completed the command */ BLK_EH_RESET_TIMER, /* reset timer and try again */ diff --git a/include/linux/blkdev.h b/include/linux/blkdev.h index 22746b2d6825..9c95df26fc26 100644 --- a/include/linux/blkdev.h +++ b/include/linux/blkdev.h @@ -1339,33 +1339,4 @@ struct io_comp_batch { #define DEFINE_IO_COMP_BATCH(name) struct io_comp_batch name = { } -#define rq_list_add(listptr, rq) do { \ - (rq)->rq_next = *(listptr); \ - *(listptr) = rq; \ -} while (0) - -#define rq_list_pop(listptr) \ -({ \ - struct request *__req = NULL; \ - if ((listptr) && *(listptr)) { \ - __req = *(listptr); \ - *(listptr) = __req->rq_next; \ - } \ - __req; \ -}) - -#define rq_list_peek(listptr) \ -({ \ - struct request *__req = NULL; \ - if ((listptr) && *(listptr)) \ - __req = *(listptr); \ - __req; \ -}) - -#define rq_list_for_each(listptr, pos) \ - for (pos = rq_list_peek((listptr)); pos; pos = rq_list_next(pos)) - -#define rq_list_next(rq) (rq)->rq_next -#define rq_list_empty(list) ((list) == (struct request *) NULL) - #endif /* _LINUX_BLKDEV_H */ From patchwork Wed Jan 5 17:05:16 2022 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Keith Busch X-Patchwork-Id: 12704531 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from vger.kernel.org (vger.kernel.org [23.128.96.18]) by smtp.lore.kernel.org (Postfix) with ESMTP id 21F2DC433EF for ; Wed, 5 Jan 2022 17:05:30 +0000 (UTC) Received: (majordomo@vger.kernel.org) by vger.kernel.org via listexpand id S242133AbiAERF3 (ORCPT ); Wed, 5 Jan 2022 12:05:29 -0500 Received: from ams.source.kernel.org ([145.40.68.75]:49710 "EHLO ams.source.kernel.org" rhost-flags-OK-OK-OK-OK) by vger.kernel.org with ESMTP id S242091AbiAERF3 (ORCPT ); Wed, 5 Jan 2022 12:05:29 -0500 Received: from smtp.kernel.org (relay.kernel.org [52.25.139.140]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (No client certificate requested) by ams.source.kernel.org (Postfix) with ESMTPS id CBD3AB81BBA for ; Wed, 5 Jan 2022 17:05:27 +0000 (UTC) Received: by smtp.kernel.org (Postfix) with ESMTPSA id 25B78C36AEF; Wed, 5 Jan 2022 17:05:26 +0000 (UTC) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/simple; d=kernel.org; s=k20201202; t=1641402326; bh=CXd24YzqYAqII94cHDbP9cfZzq/4lctBKXJnnDewKUw=; h=From:To:Cc:Subject:Date:In-Reply-To:References:From; b=Z3Yqw0D/U8orvCZylc01mAIK11/iuESuY1/8HoM/FNFQahnOl81G4543EUIqehYiG MK5dOgVdR9Egq58TI387PTAPMKZ7KShtDta7NmCewkwxdZv6lFpcVUMCG5VDkHXWLv V3xfFfB4asAJQKFHRMysMlVcrOW66ljGUdI93ygGhP1BAx+FBWVWXIT8d4rkkjJI1V c3EB0nIHTrhaic4ui+EFyYqr09gd2k9snC+0DWZsPGNzP1VfuvPKpKgJJBNF0mmR1n yA+6ZMNrlLSlgYQZttEoSgmPMG1kz7AK5oni9vkziCq5duC7AvG7Sm+37hoS1OsM6L J9WRcoD1eNKqA== From: Keith Busch To: linux-nvme@lists.infradead.org, linux-block@vger.kernel.org, axboe@kernel.dk Cc: hch@lst.de, sagi@grimberg.me, mgurtovoy@nvidia.com, Keith Busch Subject: [PATCHv3 2/4] block: introduce rq_list_for_each_safe macro Date: Wed, 5 Jan 2022 09:05:16 -0800 Message-Id: <20220105170518.3181469-3-kbusch@kernel.org> X-Mailer: git-send-email 2.25.4 In-Reply-To: <20220105170518.3181469-1-kbusch@kernel.org> References: <20220105170518.3181469-1-kbusch@kernel.org> MIME-Version: 1.0 Precedence: bulk List-ID: X-Mailing-List: linux-block@vger.kernel.org While iterating a list, a particular request may need to be removed for special handling. Provide an iterator that can safely handle that. Reviewed-by: Christoph Hellwig Signed-off-by: Keith Busch --- include/linux/blk-mq.h | 4 ++++ 1 file changed, 4 insertions(+) diff --git a/include/linux/blk-mq.h b/include/linux/blk-mq.h index bf64b94cd64e..1467f0fa2142 100644 --- a/include/linux/blk-mq.h +++ b/include/linux/blk-mq.h @@ -242,6 +242,10 @@ static inline unsigned short req_get_ioprio(struct request *req) #define rq_list_for_each(listptr, pos) \ for (pos = rq_list_peek((listptr)); pos; pos = rq_list_next(pos)) +#define rq_list_for_each_safe(listptr, pos, nxt) \ + for (pos = rq_list_peek((listptr)), nxt = rq_list_next(pos); \ + pos; pos = nxt, nxt = pos ? rq_list_next(pos) : NULL) + #define rq_list_next(rq) (rq)->rq_next #define rq_list_empty(list) ((list) == (struct request *) NULL) From patchwork Wed Jan 5 17:05:17 2022 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Keith Busch X-Patchwork-Id: 12704530 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from vger.kernel.org (vger.kernel.org [23.128.96.18]) by smtp.lore.kernel.org (Postfix) with ESMTP id CD9B7C433FE for ; Wed, 5 Jan 2022 17:05:29 +0000 (UTC) Received: (majordomo@vger.kernel.org) by vger.kernel.org via listexpand id S242135AbiAERF3 (ORCPT ); Wed, 5 Jan 2022 12:05:29 -0500 Received: from lindbergh.monkeyblade.net ([23.128.96.19]:49752 "EHLO lindbergh.monkeyblade.net" rhost-flags-OK-OK-OK-OK) by vger.kernel.org with ESMTP id S242133AbiAERF2 (ORCPT ); Wed, 5 Jan 2022 12:05:28 -0500 Received: from dfw.source.kernel.org (dfw.source.kernel.org [IPv6:2604:1380:4641:c500::1]) by lindbergh.monkeyblade.net (Postfix) with ESMTPS id 7A3A0C061245 for ; Wed, 5 Jan 2022 09:05:28 -0800 (PST) Received: from smtp.kernel.org (relay.kernel.org [52.25.139.140]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (No client certificate requested) by dfw.source.kernel.org (Postfix) with ESMTPS id 07CB061830 for ; Wed, 5 Jan 2022 17:05:28 +0000 (UTC) Received: by smtp.kernel.org (Postfix) with ESMTPSA id D44DEC36AE3; Wed, 5 Jan 2022 17:05:26 +0000 (UTC) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/simple; d=kernel.org; s=k20201202; t=1641402327; bh=+XQTZk3ubgunqyEYSBvBH7FNv+2vNL2x2STiFK+7xVM=; h=From:To:Cc:Subject:Date:In-Reply-To:References:From; b=TUv91FY+XbaScxPCnY65ap951HrSwbcsxBme3YMO8BrxR0go5m4laqctjvQeRqvCP v22/mim3vpwzf7ZuZ4nPF5He5t4PeGpYNHTRWpRMbDqy5722UF34rLl5oi8h6ucDyu cyXtzzNlHW0D8XTsmBsVYdJ+r3EnVr6LaC6s8eJPLTOI156scT+kTm/gSYjQbbg3z5 N+AQN1ANxE1buEO2g/HRtmz39KBTsYc+/tlYdUTU894jhhMSrvHb640CVKdxx7ZPtK xeptnBP+pmKp7wqgZQ6/a/wc/TKrpj/CCAKDJtG3z4W+Qz+fCym3dBOCVXTDLPVzEM tajuVOJg90VtA== From: Keith Busch To: linux-nvme@lists.infradead.org, linux-block@vger.kernel.org, axboe@kernel.dk Cc: hch@lst.de, sagi@grimberg.me, mgurtovoy@nvidia.com, Keith Busch Subject: [PATCHv3 3/4] block: introduce rq_list_move Date: Wed, 5 Jan 2022 09:05:17 -0800 Message-Id: <20220105170518.3181469-4-kbusch@kernel.org> X-Mailer: git-send-email 2.25.4 In-Reply-To: <20220105170518.3181469-1-kbusch@kernel.org> References: <20220105170518.3181469-1-kbusch@kernel.org> MIME-Version: 1.0 Precedence: bulk List-ID: X-Mailing-List: linux-block@vger.kernel.org When iterating a list, a particular request may need to be moved for special handling. Provide a helper function to achieve that so drivers don't need to reimplement rqlist manipulation. Signed-off-by: Keith Busch Reviewed-by: Christoph Hellwig --- include/linux/blk-mq.h | 17 +++++++++++++++++ 1 file changed, 17 insertions(+) diff --git a/include/linux/blk-mq.h b/include/linux/blk-mq.h index 1467f0fa2142..f40a05ecca4a 100644 --- a/include/linux/blk-mq.h +++ b/include/linux/blk-mq.h @@ -249,6 +249,23 @@ static inline unsigned short req_get_ioprio(struct request *req) #define rq_list_next(rq) (rq)->rq_next #define rq_list_empty(list) ((list) == (struct request *) NULL) +/** + * rq_list_move() - move a struct request from one list to another + * @src: The source list @rq is currently in + * @dst: The destination list that @rq will be appended to + * @rq: The request to move + * @prev: The request preceding @rq in @src (NULL if @rq is the head) + */ +static void inline rq_list_move(struct request **src, struct request **dst, + struct request *rq, struct request *prev) +{ + if (prev) + prev->rq_next = rq->rq_next; + else + *src = rq->rq_next; + rq_list_add(dst, rq); +} + enum blk_eh_timer_return { BLK_EH_DONE, /* drivers has completed the command */ BLK_EH_RESET_TIMER, /* reset timer and try again */ From patchwork Wed Jan 5 17:05:18 2022 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Keith Busch X-Patchwork-Id: 12704532 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from vger.kernel.org (vger.kernel.org [23.128.96.18]) by smtp.lore.kernel.org (Postfix) with ESMTP id 0C4EEC433EF for ; Wed, 5 Jan 2022 17:05:33 +0000 (UTC) Received: (majordomo@vger.kernel.org) by vger.kernel.org via listexpand id S242147AbiAERFb (ORCPT ); Wed, 5 Jan 2022 12:05:31 -0500 Received: from lindbergh.monkeyblade.net ([23.128.96.19]:49770 "EHLO lindbergh.monkeyblade.net" rhost-flags-OK-OK-OK-OK) by vger.kernel.org with ESMTP id S242142AbiAERFa (ORCPT ); Wed, 5 Jan 2022 12:05:30 -0500 Received: from ams.source.kernel.org (ams.source.kernel.org [IPv6:2604:1380:4601:e00::1]) by lindbergh.monkeyblade.net (Postfix) with ESMTPS id A1424C061201 for ; Wed, 5 Jan 2022 09:05:30 -0800 (PST) Received: from smtp.kernel.org (relay.kernel.org [52.25.139.140]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (No client certificate requested) by ams.source.kernel.org (Postfix) with ESMTPS id 5F198B81CBB for ; Wed, 5 Jan 2022 17:05:29 +0000 (UTC) Received: by smtp.kernel.org (Postfix) with ESMTPSA id 9A599C36AF2; Wed, 5 Jan 2022 17:05:27 +0000 (UTC) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/simple; d=kernel.org; s=k20201202; t=1641402328; bh=Z8VHzQF+RrX+Y/7r/9niEAi8EOiYU9sVNurudNt+iRo=; h=From:To:Cc:Subject:Date:In-Reply-To:References:From; b=Y8AhOdMLmHyhNy8IQuJbHY46fft0TAdIoqgsmH9RaOvmOQ8LDMNWbMY5+/oMO8VUz W8q9tcHXBR0HDoGhtFOd60NbBImDGgPCJPMwLzYEPq83sfAAl0eoXklTtOTaX/FIV6 fMdLUH82VgPUkeQZxEc94XCNuG5ykPKSVF7RaczZj2KT58dMuRfDxCNKaL/+CPDa1g EcDPw7vYs2fANJO4kF7LhK3biXHXajInSEmJiLyM7Eob/z8/3fK2Wv7g+1yDKbdSMA 4tzVU39y9kwS6a8yzPVcGaDkds/5uJL/OqUY2WN1lpUOJUpYbENsoaLTKh4J4u/jwb vBBjwxY4sxfJw== From: Keith Busch To: linux-nvme@lists.infradead.org, linux-block@vger.kernel.org, axboe@kernel.dk Cc: hch@lst.de, sagi@grimberg.me, mgurtovoy@nvidia.com, Keith Busch Subject: [PATCHv3 4/4] nvme-pci: fix queue_rqs list splitting Date: Wed, 5 Jan 2022 09:05:18 -0800 Message-Id: <20220105170518.3181469-5-kbusch@kernel.org> X-Mailer: git-send-email 2.25.4 In-Reply-To: <20220105170518.3181469-1-kbusch@kernel.org> References: <20220105170518.3181469-1-kbusch@kernel.org> MIME-Version: 1.0 Precedence: bulk List-ID: X-Mailing-List: linux-block@vger.kernel.org If command prep fails, current handling will orphan subsequent requests in the list. Consider a simple example: rqlist = [ 1 -> 2 ] When prep for request '1' fails, it will be appended to the 'requeue_list', leaving request '2' disconnected from the original rqlist and no longer tracked. Meanwhile, rqlist is still pointing to the failed request '1' and will attempt to submit the unprepped command. Fix this by updating the rqlist accordingly using the request list helper functions. Fixes: d62cbcf62f2f ("nvme: add support for mq_ops->queue_rqs()") Signed-off-by: Keith Busch Reviewed-by: Christoph Hellwig --- drivers/nvme/host/pci.c | 28 ++++++++++++++-------------- 1 file changed, 14 insertions(+), 14 deletions(-) diff --git a/drivers/nvme/host/pci.c b/drivers/nvme/host/pci.c index 50deb8b69c40..d8585df2c2fd 100644 --- a/drivers/nvme/host/pci.c +++ b/drivers/nvme/host/pci.c @@ -999,30 +999,30 @@ static bool nvme_prep_rq_batch(struct nvme_queue *nvmeq, struct request *req) static void nvme_queue_rqs(struct request **rqlist) { - struct request *req = rq_list_peek(rqlist), *prev = NULL; + struct request *req, *next, *prev = NULL; struct request *requeue_list = NULL; - do { + rq_list_for_each_safe(rqlist, req, next) { struct nvme_queue *nvmeq = req->mq_hctx->driver_data; if (!nvme_prep_rq_batch(nvmeq, req)) { /* detach 'req' and add to remainder list */ - if (prev) - prev->rq_next = req->rq_next; - rq_list_add(&requeue_list, req); - } else { - prev = req; + rq_list_move(rqlist, &requeue_list, req, prev); + + req = prev; + if (!req) + continue; } - req = rq_list_next(req); - if (!req || (prev && req->mq_hctx != prev->mq_hctx)) { + if (!next || req->mq_hctx != next->mq_hctx) { /* detach rest of list, and submit */ - if (prev) - prev->rq_next = NULL; + req->rq_next = NULL; nvme_submit_cmds(nvmeq, rqlist); - *rqlist = req; - } - } while (req); + *rqlist = next; + prev = NULL; + } else + prev = req; + } *rqlist = requeue_list; }