From patchwork Mon Feb 27 16:14:52 2017 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Omar Sandoval X-Patchwork-Id: 9593547 Return-Path: Received: from mail.wl.linuxfoundation.org (pdx-wl-mail.web.codeaurora.org [172.30.200.125]) by pdx-korg-patchwork.web.codeaurora.org (Postfix) with ESMTP id 0384460471 for ; Mon, 27 Feb 2017 16:15:48 +0000 (UTC) Received: from mail.wl.linuxfoundation.org (localhost [127.0.0.1]) by mail.wl.linuxfoundation.org (Postfix) with ESMTP id E928B2838E for ; Mon, 27 Feb 2017 16:15:47 +0000 (UTC) Received: by mail.wl.linuxfoundation.org (Postfix, from userid 486) id DE1EE28490; Mon, 27 Feb 2017 16:15:47 +0000 (UTC) X-Spam-Checker-Version: SpamAssassin 3.3.1 (2010-03-16) on pdx-wl-mail.web.codeaurora.org X-Spam-Level: X-Spam-Status: No, score=-6.4 required=2.0 tests=BAYES_00,DKIM_SIGNED, DKIM_VALID, RCVD_IN_DNSWL_HI, RCVD_IN_SORBS_SPAM autolearn=ham version=3.3.1 Received: from vger.kernel.org (vger.kernel.org [209.132.180.67]) by mail.wl.linuxfoundation.org (Postfix) with ESMTP id 5DCDE2838E for ; Mon, 27 Feb 2017 16:15:47 +0000 (UTC) Received: (majordomo@vger.kernel.org) by vger.kernel.org via listexpand id S1751367AbdB0QPf (ORCPT ); Mon, 27 Feb 2017 11:15:35 -0500 Received: from mail-pg0-f50.google.com ([74.125.83.50]:35360 "EHLO mail-pg0-f50.google.com" rhost-flags-OK-OK-OK-OK) by vger.kernel.org with ESMTP id S1751251AbdB0QPd (ORCPT ); Mon, 27 Feb 2017 11:15:33 -0500 Received: by mail-pg0-f50.google.com with SMTP id b129so46356601pgc.2 for ; Mon, 27 Feb 2017 08:15:33 -0800 (PST) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=osandov-com.20150623.gappssmtp.com; s=20150623; h=date:from:to:cc:subject:message-id:references:mime-version :content-disposition:in-reply-to:user-agent; bh=xatLDX6/mppjbn8Zk2YJv5ctqFvQyW+kAbzNk2opfUo=; b=0J8gF1dzAPRTxNp/JL/DkuEi+zynXae5gVwiAtAy00DIGG7rG0j9KwmNPkNYGg4Rnh P3YynCzAnW8YgzeGTz3Og/g3ST0Gu4ujpiAwzKOvKpWOtLfI29FNl9fbEDTipOA1jRKE kfjV8YjXOBVvsbVKFdjInQWOcJdZo4nvvEEtvZWxGDtUdhzmWj7sAIWFENttQ+nfmEFi BSZxAduHPcDiil7Pnsevy4wj7KSqaCdaOZ8zUcpCUHkMIcL1H0Rne3+vXS7Bf0Bm7tPr Pd3KCpIZ9DQOnzOEOSqcJstMm35TJbpt1RPkPNxDqfvJdBtho4jZhj3RC5+CdGTFH1x+ d1UA== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20161025; h=x-gm-message-state:date:from:to:cc:subject:message-id:references :mime-version:content-disposition:in-reply-to:user-agent; bh=xatLDX6/mppjbn8Zk2YJv5ctqFvQyW+kAbzNk2opfUo=; b=GthPjsi91st0Y6DDbqpMvr0AY3zQKGQYOS0KtyBwxFsSumphhw7HObkifiTcVqpeTG BUQR3iueJFykN2an70Xr5/3MOtO8AlBPxYpg1U9MM6Y9AxLBxbH9qWnGtCSAr/dLKHax 1xbtoiTpi7P+p+6EV8cV3+TzrwvuUZpjKNDCNnhI2LjsSnL/LQSzuPRPatAG8DvjFN0Y Sli4zc1j/tHgn3u6Iby2uPyr7PvQscAFxC5i7ZKrQfF1USeII2G01G3b9l8TOZgdmIqQ mBvE5yxgDVqbC9paCbcisJ9Zx35hvcmjFt46yUecrKIXo+Jhh75E74K284CX4ZCpeQdy Fthg== X-Gm-Message-State: AMke39mktXoNYthVQ+GW/f3dpcO+SePoib4dFcBGmTTjoAebnQiGjUZbZSAAfZBtw4yfGm0y X-Received: by 10.98.62.82 with SMTP id l79mr21463355pfa.164.1488212132828; Mon, 27 Feb 2017 08:15:32 -0800 (PST) Received: from vader ([2620:10d:c090:180::93d5]) by smtp.gmail.com with ESMTPSA id y21sm31712263pgh.52.2017.02.27.08.15.31 (version=TLS1_2 cipher=ECDHE-RSA-AES128-GCM-SHA256 bits=128/128); Mon, 27 Feb 2017 08:15:32 -0800 (PST) Date: Mon, 27 Feb 2017 08:14:52 -0800 From: Omar Sandoval To: Sagi Grimberg Cc: Jens Axboe , linux-block@vger.kernel.org, linux-nvme@lists.infradead.org Subject: Re: [PATCH 1/2] blk-mq-sched: Allocate sched reserved tags as specified in the original queue tagset Message-ID: <20170227161452.GB10715@vader> References: <1488209781-1084-1-git-send-email-sagi@grimberg.me> <20170227154939.GA10715@vader> <4829b70c-53fd-f622-310e-0a30e4ede6fa@kernel.dk> MIME-Version: 1.0 Content-Disposition: inline In-Reply-To: User-Agent: Mutt/1.8.0 (2017-02-23) Sender: linux-block-owner@vger.kernel.org Precedence: bulk List-ID: X-Mailing-List: linux-block@vger.kernel.org X-Virus-Scanned: ClamAV using ClamSMTP On Mon, Feb 27, 2017 at 06:10:01PM +0200, Sagi Grimberg wrote: > > > > Hm, this may fix the crash, but I'm not sure it'll work as intended. > > > When we allocate the request, we'll get a reserved scheduler tag, but > > > then when we go to dispatch the request and call > > > blk_mq_get_driver_tag(), we'll be competing with all of the normal > > > requests for a regular driver tag. So maybe on top of this we should add > > > the BLK_MQ_REQ_RESERVED flag to the allocation attempt in > > > blk_mq_get_driver_tag() if the scheduler tag is reserved? I'm hazy on > > > what we expect from reserved tags, so feel free to call me crazy. > > > > Yeah good point, we need to carry it through. Reserved tags exist > > because drivers often need a request/tag for error handling. If all > > tags currently are used up for regular IO that is stuck, you need > > a reserved tag for error handling to guarantee progress. > > > > So Sagi's patch does take it half the way there, but get_driver_tag > > really needs to know about this as well, or we will just get stuck > > there as well. Two solutions, I can think of: > > > > 1) Check the tag value in get_driver_tag, add BLK_MQ_REQ_RESERVED > > when allocating a driver tag if above X. > > 2) Add an RQF_SOMETHING_RESERVED. Add BLK_MQ_REQ_RESERVED in > > get_driver_tag if that is set. > > > > Comments? Option 1 looks simple enough that I don't think it warrants a new request flag (compile tested only): > Can't we just not go through the scheduler for reserved tags? Obviously > there is no point in scheduling them... That sounds nice, since I'd be worried about the scheduler also needing to be aware of the reserved status lest it also get the reserved request stuck behind some normal requests. But, we special case flush in this way, and it has been a huge pain. diff --git a/block/blk-mq.c b/block/blk-mq.c index 9e6b064e5339..87590f7d4f80 100644 --- a/block/blk-mq.c +++ b/block/blk-mq.c @@ -852,6 +852,9 @@ bool blk_mq_get_driver_tag(struct request *rq, struct blk_mq_hw_ctx **hctx, return true; } + if (rq->internal_tag < data.hctx->sched_tags->nr_reserved_tags) + data.flags |= BLK_MQ_REQ_RESERVED; + rq->tag = blk_mq_get_tag(&data); if (rq->tag >= 0) { if (blk_mq_tag_busy(data.hctx)) {