From patchwork Mon Dec 18 21:13:39 2023 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Bart Van Assche X-Patchwork-Id: 13497594 Received: from mail-pf1-f178.google.com (mail-pf1-f178.google.com [209.85.210.178]) (using TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 (128/128 bits)) (No client certificate requested) by smtp.subspace.kernel.org (Postfix) with ESMTPS id 1840AA2D for ; Mon, 18 Dec 2023 21:14:14 +0000 (UTC) Authentication-Results: smtp.subspace.kernel.org; dmarc=fail (p=none dis=none) header.from=acm.org Authentication-Results: smtp.subspace.kernel.org; spf=pass smtp.mailfrom=gmail.com Received: by mail-pf1-f178.google.com with SMTP id d2e1a72fcca58-6d8923fe26bso743089b3a.0 for ; Mon, 18 Dec 2023 13:14:14 -0800 (PST) X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20230601; t=1702934054; x=1703538854; h=content-transfer-encoding:mime-version:references:in-reply-to :message-id:date:subject:cc:to:from:x-gm-message-state:from:to:cc :subject:date:message-id:reply-to; bh=uaCtVjkh9mRRgN4leLEKJEwOg3CZP8s0yWPlO2ojdSI=; b=BVdKN6ZrVUbE7MLfMVvf9Ls+GhGW90n2brXojQo3OGfV/bELhb9rNb7uPLKfZqJUhX u/nfgRQ0hQ6yaq0qTdFHZApU7bPtD8XIYkbytRTGnBEm5NUZVfHAaoz2CI8d0/eyE2lI tCllfkI+4Xcy9R2niWKAmdazJYrIASGlS68+E3f49Jxeq5KmjeWitHkAEZBod7IKYcRn A5VUpkBvFA+66jxodhFOPCeIpO70VwQjFnCdenm/zulv4LOqFTCw0YA3/bTPnkw2Mft2 yoX78TQkLDxou19sr3jlzfzSDDciewz14fADoszMf/2/2Ba3gExE8FpFkWG1N2DW1S93 idfQ== X-Gm-Message-State: AOJu0YyxWK0NDrBdLNg6vHB9S4kBL2W9ihLI05ZWXNODk/Z6d09BvFMq IMFEIOXGidu/7sL++Ps+jGrgb4af1Jc= X-Google-Smtp-Source: AGHT+IG8jDpf0zG23U72An4NnkSfdATr8q7DlKUbZAs/e1/ZdUDhvBXVd1N2pn9sfyk/zFIVfW5dVQ== X-Received: by 2002:aa7:8881:0:b0:6cd:faa6:fc36 with SMTP id z1-20020aa78881000000b006cdfaa6fc36mr10511758pfe.30.1702934054259; Mon, 18 Dec 2023 13:14:14 -0800 (PST) Received: from bvanassche-linux.mtv.corp.google.com ([2620:0:1000:8411:e67:7ba6:36a9:8cd5]) by smtp.gmail.com with ESMTPSA id c17-20020aa78811000000b006c320b9897fsm3371497pfo.126.2023.12.18.13.14.12 (version=TLS1_3 cipher=TLS_AES_256_GCM_SHA384 bits=256/256); Mon, 18 Dec 2023 13:14:13 -0800 (PST) From: Bart Van Assche To: Jens Axboe Cc: linux-block@vger.kernel.org, Damien Le Moal , Christoph Hellwig , Bart Van Assche Subject: [PATCH v2 1/4] block/mq-deadline: Rename dd_rq_ioclass() and change its return type Date: Mon, 18 Dec 2023 13:13:39 -0800 Message-ID: <20231218211342.2179689-2-bvanassche@acm.org> X-Mailer: git-send-email 2.43.0.472.g3155946c3a-goog In-Reply-To: <20231218211342.2179689-1-bvanassche@acm.org> References: <20231218211342.2179689-1-bvanassche@acm.org> Precedence: bulk X-Mailing-List: linux-block@vger.kernel.org List-Id: List-Subscribe: List-Unsubscribe: MIME-Version: 1.0 All dd_rq_ioclass() callers convert the return value into type enum dd_prio. Move this conversion into dd_rq_ioclass() and rename this function. Move the definition of this function. Introduce an additional caller. No functionality is changed by this patch. Cc: Damien Le Moal Cc: Christoph Hellwig Signed-off-by: Bart Van Assche --- block/mq-deadline.c | 35 ++++++++++++++--------------------- 1 file changed, 14 insertions(+), 21 deletions(-) diff --git a/block/mq-deadline.c b/block/mq-deadline.c index f958e79277b8..8e5f71775cf1 100644 --- a/block/mq-deadline.c +++ b/block/mq-deadline.c @@ -119,15 +119,6 @@ deadline_rb_root(struct dd_per_prio *per_prio, struct request *rq) return &per_prio->sort_list[rq_data_dir(rq)]; } -/* - * Returns the I/O priority class (IOPRIO_CLASS_*) that has been assigned to a - * request. - */ -static u8 dd_rq_ioclass(struct request *rq) -{ - return IOPRIO_PRIO_CLASS(req_get_ioprio(rq)); -} - /* * get the request before `rq' in sector-sorted order */ @@ -190,6 +181,15 @@ static inline struct request *deadline_from_pos(struct dd_per_prio *per_prio, return res; } +/* + * Returns the I/O priority class (IOPRIO_CLASS_*) that has been assigned to a + * request. + */ +static enum dd_prio dd_rq_ioprio(struct request *rq) +{ + return ioprio_class_to_prio[IOPRIO_PRIO_CLASS(req_get_ioprio(rq))]; +} + static void deadline_add_rq_rb(struct dd_per_prio *per_prio, struct request *rq) { @@ -228,8 +228,7 @@ static void dd_request_merged(struct request_queue *q, struct request *req, enum elv_merge type) { struct deadline_data *dd = q->elevator->elevator_data; - const u8 ioprio_class = dd_rq_ioclass(req); - const enum dd_prio prio = ioprio_class_to_prio[ioprio_class]; + const enum dd_prio prio = dd_rq_ioprio(req); struct dd_per_prio *per_prio = &dd->per_prio[prio]; /* @@ -248,8 +247,7 @@ static void dd_merged_requests(struct request_queue *q, struct request *req, struct request *next) { struct deadline_data *dd = q->elevator->elevator_data; - const u8 ioprio_class = dd_rq_ioclass(next); - const enum dd_prio prio = ioprio_class_to_prio[ioprio_class]; + const enum dd_prio prio = dd_rq_ioprio(next); lockdep_assert_held(&dd->lock); @@ -447,7 +445,6 @@ static struct request *__dd_dispatch_request(struct deadline_data *dd, struct request *rq, *next_rq; enum dd_data_dir data_dir; enum dd_prio prio; - u8 ioprio_class; lockdep_assert_held(&dd->lock); @@ -545,8 +542,7 @@ static struct request *__dd_dispatch_request(struct deadline_data *dd, dd->batching++; deadline_move_request(dd, per_prio, rq); done: - ioprio_class = dd_rq_ioclass(rq); - prio = ioprio_class_to_prio[ioprio_class]; + prio = dd_rq_ioprio(rq); dd->per_prio[prio].latest_pos[data_dir] = blk_rq_pos(rq); dd->per_prio[prio].stats.dispatched++; /* @@ -798,8 +794,6 @@ static void dd_insert_request(struct blk_mq_hw_ctx *hctx, struct request *rq, struct request_queue *q = hctx->queue; struct deadline_data *dd = q->elevator->elevator_data; const enum dd_data_dir data_dir = rq_data_dir(rq); - u16 ioprio = req_get_ioprio(rq); - u8 ioprio_class = IOPRIO_PRIO_CLASS(ioprio); struct dd_per_prio *per_prio; enum dd_prio prio; @@ -811,7 +805,7 @@ static void dd_insert_request(struct blk_mq_hw_ctx *hctx, struct request *rq, */ blk_req_zone_write_unlock(rq); - prio = ioprio_class_to_prio[ioprio_class]; + prio = dd_rq_ioprio(rq); per_prio = &dd->per_prio[prio]; if (!rq->elv.priv[0]) { per_prio->stats.inserted++; @@ -920,8 +914,7 @@ static void dd_finish_request(struct request *rq) { struct request_queue *q = rq->q; struct deadline_data *dd = q->elevator->elevator_data; - const u8 ioprio_class = dd_rq_ioclass(rq); - const enum dd_prio prio = ioprio_class_to_prio[ioprio_class]; + const enum dd_prio prio = dd_rq_ioprio(rq); struct dd_per_prio *per_prio = &dd->per_prio[prio]; /* From patchwork Mon Dec 18 21:13:40 2023 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Bart Van Assche X-Patchwork-Id: 13497595 Received: from mail-oo1-f45.google.com (mail-oo1-f45.google.com [209.85.161.45]) (using TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 (128/128 bits)) (No client certificate requested) by smtp.subspace.kernel.org (Postfix) with ESMTPS id A46D073492 for ; Mon, 18 Dec 2023 21:14:16 +0000 (UTC) Authentication-Results: smtp.subspace.kernel.org; dmarc=fail (p=none dis=none) header.from=acm.org Authentication-Results: smtp.subspace.kernel.org; spf=pass smtp.mailfrom=gmail.com Received: by mail-oo1-f45.google.com with SMTP id 006d021491bc7-58e256505f7so2523629eaf.3 for ; Mon, 18 Dec 2023 13:14:16 -0800 (PST) X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20230601; t=1702934055; x=1703538855; h=content-transfer-encoding:mime-version:references:in-reply-to :message-id:date:subject:cc:to:from:x-gm-message-state:from:to:cc :subject:date:message-id:reply-to; bh=tBp0e9bv3GavBOwkFpku03+RC2SEUPQU/wnkINj2HC8=; b=sqZij17VFvfLz8RaA/gW5q4Dcj5jQ1emVGe90o4ceBdjLP7JCAwZbKrfevZ+MbwG+j qZ0ivDA0rT/TPD4AtXNvmuEP6CLlhibSNXUYx2B3lAP+6vIEsTvK7KaieFyPjziqZzeF a7v2lACV9deXT9gGiRcM3A1YiqdR1/bEGN2Tb1Q5Pmlv59bErj8tXa8ULvTvj9LSDg23 lxxl6L1yXzTNV2+pgAsySr/vre9iRoCVfd6XBsOIgU0BsYfW1QVJ+iVvvpyujhWBqYFd rVH69eiIqCWdvxR3/SvFE3TrJVIKPku+0GBQCiOvKMpf6g8EDLdxs9Z/id8Zmq5zS86k MCOg== X-Gm-Message-State: AOJu0YwV3J1yQyEBZ4qd5otguEGO1nwP7qZMV2ZSXWup3dboAHpl0Kha wHNsBJcQbsWxSl6ElfKred4ANGXm7Ac= X-Google-Smtp-Source: AGHT+IGn56NCoHBR49wPFLdylUi9WnvFRbQqHvwVWty2plVgQd7ipWfVvwOHiT+NMkvqGnX5myE80g== X-Received: by 2002:a05:6359:4114:b0:170:bfb9:fb41 with SMTP id kh20-20020a056359411400b00170bfb9fb41mr11746604rwc.28.1702934055514; Mon, 18 Dec 2023 13:14:15 -0800 (PST) Received: from bvanassche-linux.mtv.corp.google.com ([2620:0:1000:8411:e67:7ba6:36a9:8cd5]) by smtp.gmail.com with ESMTPSA id c17-20020aa78811000000b006c320b9897fsm3371497pfo.126.2023.12.18.13.14.14 (version=TLS1_3 cipher=TLS_AES_256_GCM_SHA384 bits=256/256); Mon, 18 Dec 2023 13:14:14 -0800 (PST) From: Bart Van Assche To: Jens Axboe Cc: linux-block@vger.kernel.org, Damien Le Moal , Christoph Hellwig , Bart Van Assche Subject: [PATCH v2 2/4] block/mq-deadline: Introduce dd_bio_ioclass() Date: Mon, 18 Dec 2023 13:13:40 -0800 Message-ID: <20231218211342.2179689-3-bvanassche@acm.org> X-Mailer: git-send-email 2.43.0.472.g3155946c3a-goog In-Reply-To: <20231218211342.2179689-1-bvanassche@acm.org> References: <20231218211342.2179689-1-bvanassche@acm.org> Precedence: bulk X-Mailing-List: linux-block@vger.kernel.org List-Id: List-Subscribe: List-Unsubscribe: MIME-Version: 1.0 Prepare for ignoring the I/O priority of certain requests. Cc: Damien Le Moal Cc: Christoph Hellwig Signed-off-by: Bart Van Assche --- block/mq-deadline.c | 8 ++++++-- 1 file changed, 6 insertions(+), 2 deletions(-) diff --git a/block/mq-deadline.c b/block/mq-deadline.c index 8e5f71775cf1..a5341a37c840 100644 --- a/block/mq-deadline.c +++ b/block/mq-deadline.c @@ -190,6 +190,11 @@ static enum dd_prio dd_rq_ioprio(struct request *rq) return ioprio_class_to_prio[IOPRIO_PRIO_CLASS(req_get_ioprio(rq))]; } +static enum dd_prio dd_bio_ioprio(struct bio *bio) +{ + return ioprio_class_to_prio[IOPRIO_PRIO_CLASS(bio->bi_ioprio)]; +} + static void deadline_add_rq_rb(struct dd_per_prio *per_prio, struct request *rq) { @@ -740,8 +745,7 @@ static int dd_request_merge(struct request_queue *q, struct request **rq, struct bio *bio) { struct deadline_data *dd = q->elevator->elevator_data; - const u8 ioprio_class = IOPRIO_PRIO_CLASS(bio->bi_ioprio); - const enum dd_prio prio = ioprio_class_to_prio[ioprio_class]; + const enum dd_prio prio = dd_bio_ioprio(bio); struct dd_per_prio *per_prio = &dd->per_prio[prio]; sector_t sector = bio_end_sector(bio); struct request *__rq; From patchwork Mon Dec 18 21:13:41 2023 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Bart Van Assche X-Patchwork-Id: 13497596 Received: from mail-pf1-f173.google.com (mail-pf1-f173.google.com [209.85.210.173]) (using TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 (128/128 bits)) (No client certificate requested) by smtp.subspace.kernel.org (Postfix) with ESMTPS id B60EDA2D for ; Mon, 18 Dec 2023 21:14:17 +0000 (UTC) Authentication-Results: smtp.subspace.kernel.org; dmarc=fail (p=none dis=none) header.from=acm.org Authentication-Results: smtp.subspace.kernel.org; spf=pass smtp.mailfrom=gmail.com Received: by mail-pf1-f173.google.com with SMTP id d2e1a72fcca58-6d7750e2265so841228b3a.3 for ; Mon, 18 Dec 2023 13:14:17 -0800 (PST) X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20230601; t=1702934057; x=1703538857; h=content-transfer-encoding:mime-version:references:in-reply-to :message-id:date:subject:cc:to:from:x-gm-message-state:from:to:cc :subject:date:message-id:reply-to; bh=dAiSiu05NpJmzIVrTZabB/0AwAkLIp2fczQUUurdWuE=; b=q5I4hmpmNHWCTGhkrv5lMPYmyiA6va1l035Ts9/z2d5VTBYQdpYf9AQ/hBrp2iD/T+ rEJ3da1YcNnlofjyOxWmU6FN0/+aV5i3KDZzSnWzdCRJ4p/J2a9/PvvDz42/RqACLp8T wNuv28jz5NTNYQ3u185IB3jwFKsHEoCceI9S/1RfNQsG5SGxV0YtqSK95briCWm8GYey TEyUALkbMOz0yiZrd07jcHOq5wzf5344Ut1hYODxreb6GTSCIAFj6FP1rTa01cJ3Qatt uFFSManK1uiGaPqGMsk6Eq+UGsd1cIYuKv1aMFW2v+WXOQRSGopjtsP41NQwykwCr4AV 0XGg== X-Gm-Message-State: AOJu0YxVFTIj4Rru1Qc8kxBzsJxyQJnllHeKLcScagT7NfCgqlE3Qjus eCjAOXVxV/VjuLsrGjWLnhk= X-Google-Smtp-Source: AGHT+IHp79gIuF2cWlzENCYnCUG+u/H/06KhKIERsNXNHmasD6M6NrPnJIzA5iiTr1GuEyq9Sw8Tnw== X-Received: by 2002:a05:6a00:8c4:b0:6ce:3bcd:61df with SMTP id s4-20020a056a0008c400b006ce3bcd61dfmr9947045pfu.31.1702934056874; Mon, 18 Dec 2023 13:14:16 -0800 (PST) Received: from bvanassche-linux.mtv.corp.google.com ([2620:0:1000:8411:e67:7ba6:36a9:8cd5]) by smtp.gmail.com with ESMTPSA id c17-20020aa78811000000b006c320b9897fsm3371497pfo.126.2023.12.18.13.14.15 (version=TLS1_3 cipher=TLS_AES_256_GCM_SHA384 bits=256/256); Mon, 18 Dec 2023 13:14:16 -0800 (PST) From: Bart Van Assche To: Jens Axboe Cc: linux-block@vger.kernel.org, Damien Le Moal , Christoph Hellwig , Bart Van Assche Subject: [PATCH v2 3/4] block/mq-deadline: Introduce deadline_first_rq_past_pos() Date: Mon, 18 Dec 2023 13:13:41 -0800 Message-ID: <20231218211342.2179689-4-bvanassche@acm.org> X-Mailer: git-send-email 2.43.0.472.g3155946c3a-goog In-Reply-To: <20231218211342.2179689-1-bvanassche@acm.org> References: <20231218211342.2179689-1-bvanassche@acm.org> Precedence: bulk X-Mailing-List: linux-block@vger.kernel.org List-Id: List-Subscribe: List-Unsubscribe: MIME-Version: 1.0 Prepare for introducing a second caller. Cc: Damien Le Moal Cc: Christoph Hellwig Signed-off-by: Bart Van Assche --- block/mq-deadline.c | 35 ++++++++++++++++++++++++----------- 1 file changed, 24 insertions(+), 11 deletions(-) diff --git a/block/mq-deadline.c b/block/mq-deadline.c index a5341a37c840..c0f92cc729ca 100644 --- a/block/mq-deadline.c +++ b/block/mq-deadline.c @@ -147,6 +147,28 @@ deadline_latter_request(struct request *rq) return NULL; } +/* + * Return the first request with the given priority, data direction and for + * which blk_rq_pos() >= @pos. + */ +static struct request *deadline_first_rq_past_pos(struct dd_per_prio *per_prio, + enum dd_data_dir data_dir, sector_t pos) +{ + struct rb_node *node = per_prio->sort_list[data_dir].rb_node; + struct request *rq, *res = NULL; + + while (node) { + rq = rb_entry_rq(node); + if (blk_rq_pos(rq) >= pos) { + res = rq; + node = node->rb_left; + } else { + node = node->rb_right; + } + } + return res; +} + /* * Return the first request for which blk_rq_pos() >= @pos. For zoned devices, * return the first request after the start of the zone containing @pos. @@ -155,7 +177,7 @@ static inline struct request *deadline_from_pos(struct dd_per_prio *per_prio, enum dd_data_dir data_dir, sector_t pos) { struct rb_node *node = per_prio->sort_list[data_dir].rb_node; - struct request *rq, *res = NULL; + struct request *rq; if (!node) return NULL; @@ -169,16 +191,7 @@ static inline struct request *deadline_from_pos(struct dd_per_prio *per_prio, if (blk_rq_is_seq_zoned_write(rq)) pos = round_down(pos, rq->q->limits.chunk_sectors); - while (node) { - rq = rb_entry_rq(node); - if (blk_rq_pos(rq) >= pos) { - res = rq; - node = node->rb_left; - } else { - node = node->rb_right; - } - } - return res; + return deadline_first_rq_past_pos(per_prio, data_dir, pos); } /* From patchwork Mon Dec 18 21:13:42 2023 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Bart Van Assche X-Patchwork-Id: 13497597 Received: from mail-oo1-f42.google.com (mail-oo1-f42.google.com [209.85.161.42]) (using TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 (128/128 bits)) (No client certificate requested) by smtp.subspace.kernel.org (Postfix) with ESMTPS id 855587408D for ; Mon, 18 Dec 2023 21:14:19 +0000 (UTC) Authentication-Results: smtp.subspace.kernel.org; dmarc=fail (p=none dis=none) header.from=acm.org Authentication-Results: smtp.subspace.kernel.org; spf=pass smtp.mailfrom=gmail.com Received: by mail-oo1-f42.google.com with SMTP id 006d021491bc7-5913922ab10so2294808eaf.1 for ; Mon, 18 Dec 2023 13:14:19 -0800 (PST) X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20230601; t=1702934058; x=1703538858; h=content-transfer-encoding:mime-version:references:in-reply-to :message-id:date:subject:cc:to:from:x-gm-message-state:from:to:cc :subject:date:message-id:reply-to; bh=a/IgCh6hFZVSZ9VmGW8j+wPURodjNiyEfdjlu8brBDE=; b=Mt167EEA6bz4BK9Al/69LNz5JyX/EjoMeeVixoiPnsZ4sk341yXjajDRLfUwaazVAd Rd/y8Z+FiOikvklew6Cyt2gu0bnhIKHZXzwjFDUSgFM1maQM46Dl87LpJgURcqTFjYVa gFtBptosztD1uwsGmwPE496FQeo912atwgKqUqvPF2IOMD14eZmM9Fjc15SORmaZNfq2 QvK7wqL2yWxiTzAyAvxqlevGe51h3KsgzPtc7Sy+p7pXBNTrHG+TvFxtUj6VEl/S68zq AYsaCT6S6s3h4IOL/Cylj4Z7ZyUyTo9VoMUze7niXSwM+jHR+FkvDgnAmHac1CBs7Mg7 xRiQ== X-Gm-Message-State: AOJu0YzkYNQUcV6TwJt5Om7UQat27XDBC0NxMx0eFfN0QZvdNjGzyP8Q 6J/SR2/VTDqKR2qc8caFooc= X-Google-Smtp-Source: AGHT+IHa+HNPAbjd2hPI55BcDUh9mC2onuwErgKNN5Tl/YdMKWNCDDV0Qw2eTbUZO5ziawaTa7F1jw== X-Received: by 2002:a05:6359:2c43:b0:170:2c52:2b4d with SMTP id qv3-20020a0563592c4300b001702c522b4dmr8767596rwb.19.1702934058280; Mon, 18 Dec 2023 13:14:18 -0800 (PST) Received: from bvanassche-linux.mtv.corp.google.com ([2620:0:1000:8411:e67:7ba6:36a9:8cd5]) by smtp.gmail.com with ESMTPSA id c17-20020aa78811000000b006c320b9897fsm3371497pfo.126.2023.12.18.13.14.17 (version=TLS1_3 cipher=TLS_AES_256_GCM_SHA384 bits=256/256); Mon, 18 Dec 2023 13:14:17 -0800 (PST) From: Bart Van Assche To: Jens Axboe Cc: linux-block@vger.kernel.org, Damien Le Moal , Christoph Hellwig , Bart Van Assche Subject: [PATCH v2 4/4] block/mq-deadline: Prevent zoned write reordering due to I/O prioritization Date: Mon, 18 Dec 2023 13:13:42 -0800 Message-ID: <20231218211342.2179689-5-bvanassche@acm.org> X-Mailer: git-send-email 2.43.0.472.g3155946c3a-goog In-Reply-To: <20231218211342.2179689-1-bvanassche@acm.org> References: <20231218211342.2179689-1-bvanassche@acm.org> Precedence: bulk X-Mailing-List: linux-block@vger.kernel.org List-Id: List-Subscribe: List-Unsubscribe: MIME-Version: 1.0 Assigning I/O priorities with the ioprio cgroup policy may cause different I/O priorities to be assigned to write requests for the same zone. Prevent that this causes unaligned write errors by adding zoned writes for the same zone in the same priority queue as prior zoned writes. Cc: Damien Le Moal Cc: Christoph Hellwig Signed-off-by: Bart Van Assche --- block/Kconfig.iosched | 5 ++ block/mq-deadline.c | 81 +++++++++++++++--- block/mq-deadline_test.c | 175 +++++++++++++++++++++++++++++++++++++++ include/linux/blk-mq.h | 17 ++++ 4 files changed, 268 insertions(+), 10 deletions(-) create mode 100644 block/mq-deadline_test.c diff --git a/block/Kconfig.iosched b/block/Kconfig.iosched index 27f11320b8d1..3e2fcbdbac65 100644 --- a/block/Kconfig.iosched +++ b/block/Kconfig.iosched @@ -7,6 +7,11 @@ config MQ_IOSCHED_DEADLINE help MQ version of the deadline IO scheduler. +config MQ_IOSCHED_DEADLINE_TEST + tristate "MQ deadline unit tests" if !KUNIT_ALL_TESTS + depends on MQ_IOSCHED_DEADLINE && KUNIT + default KUNIT_ALL_TESTS + config MQ_IOSCHED_KYBER tristate "Kyber I/O scheduler" default y diff --git a/block/mq-deadline.c b/block/mq-deadline.c index c0f92cc729ca..d1d54cac4c37 100644 --- a/block/mq-deadline.c +++ b/block/mq-deadline.c @@ -46,6 +46,8 @@ enum dd_data_dir { enum { DD_DIR_COUNT = 2 }; enum dd_prio { + DD_INVALID_PRIO = -1, + DD_PRIO_MIN = 0, DD_RT_PRIO = 0, DD_BE_PRIO = 1, DD_IDLE_PRIO = 2, @@ -113,6 +115,12 @@ static const enum dd_prio ioprio_class_to_prio[] = { [IOPRIO_CLASS_IDLE] = DD_IDLE_PRIO, }; +static const u8 prio_to_ioprio_class[] = { + [DD_RT_PRIO] = IOPRIO_CLASS_RT, + [DD_BE_PRIO] = IOPRIO_CLASS_BE, + [DD_IDLE_PRIO] = IOPRIO_CLASS_IDLE, +}; + static inline struct rb_root * deadline_rb_root(struct dd_per_prio *per_prio, struct request *rq) { @@ -194,18 +202,67 @@ static inline struct request *deadline_from_pos(struct dd_per_prio *per_prio, return deadline_first_rq_past_pos(per_prio, data_dir, pos); } +/* + * If any sequential write requests are pending for the zone containing @pos, + * return the I/O priority for these write requests. + */ +static enum dd_prio dd_zone_prio(struct deadline_data *dd, + struct block_device *bdev, sector_t pos) +{ +#ifdef CONFIG_BLK_DEV_ZONED + struct gendisk *disk = bdev->bd_disk; + const unsigned int zno = disk_zone_no(disk, pos); + enum dd_prio prio; + + pos -= bdev_offset_from_zone_start(bdev, pos); + for (prio = DD_PRIO_MIN; prio <= DD_PRIO_MAX; prio++) { + struct dd_per_prio *per_prio = &dd->per_prio[prio]; + struct request *rq; + + rq = deadline_first_rq_past_pos(per_prio, DD_WRITE, pos); + while (rq && blk_rq_zone_no(rq) == zno) { + struct rb_node *node; + + if (blk_rq_is_seq_zoned_write(rq)) + return prio; + node = rb_next(&rq->rb_node); + if (!node) + break; + rq = rb_entry_rq(node); + } + } +#endif + return DD_INVALID_PRIO; +} + /* * Returns the I/O priority class (IOPRIO_CLASS_*) that has been assigned to a * request. */ -static enum dd_prio dd_rq_ioprio(struct request *rq) +static enum dd_prio dd_rq_ioprio(struct deadline_data *dd, struct request *rq) { - return ioprio_class_to_prio[IOPRIO_PRIO_CLASS(req_get_ioprio(rq))]; + enum dd_prio prio; + + if (!blk_rq_is_seq_zoned_write(rq) || !rq->bio) + return ioprio_class_to_prio[IOPRIO_PRIO_CLASS( + req_get_ioprio(rq))]; + prio = dd_zone_prio(dd, rq->q->disk->part0, blk_rq_pos(rq)); + if (prio == DD_INVALID_PRIO) + return ioprio_class_to_prio[IOPRIO_PRIO_CLASS( + req_get_ioprio(rq))]; + return prio; } -static enum dd_prio dd_bio_ioprio(struct bio *bio) +static enum dd_prio dd_bio_ioprio(struct deadline_data *dd, struct bio *bio) { - return ioprio_class_to_prio[IOPRIO_PRIO_CLASS(bio->bi_ioprio)]; + enum dd_prio prio; + + if (!blk_bio_is_seq_zoned_write(bio)) + return ioprio_class_to_prio[IOPRIO_PRIO_CLASS(bio->bi_ioprio)]; + prio = dd_zone_prio(dd, bio->bi_bdev, bio->bi_iter.bi_sector); + if (prio == DD_INVALID_PRIO) + return ioprio_class_to_prio[IOPRIO_PRIO_CLASS(bio->bi_ioprio)]; + return prio; } static void @@ -246,7 +303,7 @@ static void dd_request_merged(struct request_queue *q, struct request *req, enum elv_merge type) { struct deadline_data *dd = q->elevator->elevator_data; - const enum dd_prio prio = dd_rq_ioprio(req); + const enum dd_prio prio = dd_rq_ioprio(dd, req); struct dd_per_prio *per_prio = &dd->per_prio[prio]; /* @@ -265,7 +322,7 @@ static void dd_merged_requests(struct request_queue *q, struct request *req, struct request *next) { struct deadline_data *dd = q->elevator->elevator_data; - const enum dd_prio prio = dd_rq_ioprio(next); + const enum dd_prio prio = dd_rq_ioprio(dd, next); lockdep_assert_held(&dd->lock); @@ -560,7 +617,7 @@ static struct request *__dd_dispatch_request(struct deadline_data *dd, dd->batching++; deadline_move_request(dd, per_prio, rq); done: - prio = dd_rq_ioprio(rq); + prio = dd_rq_ioprio(dd, rq); dd->per_prio[prio].latest_pos[data_dir] = blk_rq_pos(rq); dd->per_prio[prio].stats.dispatched++; /* @@ -758,7 +815,7 @@ static int dd_request_merge(struct request_queue *q, struct request **rq, struct bio *bio) { struct deadline_data *dd = q->elevator->elevator_data; - const enum dd_prio prio = dd_bio_ioprio(bio); + const enum dd_prio prio = dd_bio_ioprio(dd, bio); struct dd_per_prio *per_prio = &dd->per_prio[prio]; sector_t sector = bio_end_sector(bio); struct request *__rq; @@ -822,7 +879,7 @@ static void dd_insert_request(struct blk_mq_hw_ctx *hctx, struct request *rq, */ blk_req_zone_write_unlock(rq); - prio = dd_rq_ioprio(rq); + prio = dd_rq_ioprio(dd, rq); per_prio = &dd->per_prio[prio]; if (!rq->elv.priv[0]) { per_prio->stats.inserted++; @@ -931,7 +988,7 @@ static void dd_finish_request(struct request *rq) { struct request_queue *q = rq->q; struct deadline_data *dd = q->elevator->elevator_data; - const enum dd_prio prio = dd_rq_ioprio(rq); + const enum dd_prio prio = dd_rq_ioprio(dd, rq); struct dd_per_prio *per_prio = &dd->per_prio[prio]; /* @@ -1282,6 +1339,10 @@ static struct elevator_type mq_deadline = { }; MODULE_ALIAS("mq-deadline-iosched"); +#ifdef CONFIG_MQ_IOSCHED_DEADLINE_TEST +#include "mq-deadline_test.c" +#endif + static int __init deadline_init(void) { return elv_register(&mq_deadline); diff --git a/block/mq-deadline_test.c b/block/mq-deadline_test.c new file mode 100644 index 000000000000..72bec6fd5f7a --- /dev/null +++ b/block/mq-deadline_test.c @@ -0,0 +1,175 @@ +// SPDX-License-Identifier: GPL-2.0-only +/* + * Copyright 2023 Google LLC + */ +#include +#include + +static void test_ioprio(struct kunit *test) +{ + static struct block_device bdev; + static struct gendisk disk = { .part0 = &bdev }; + static struct request_queue queue = { .disk = &disk }; + static struct blk_mq_hw_ctx hctx = { .queue = &queue }; + static struct bio bio1 = { .bi_bdev = &bdev, + .bi_opf = REQ_OP_WRITE, + .bi_ioprio = IOPRIO_CLASS_IDLE + << IOPRIO_CLASS_SHIFT }; + static struct request rq1 = { .q = &queue, + .cmd_flags = REQ_OP_WRITE, + .__sector = 1, + .__data_len = 1, + .bio = &bio1, + .mq_hctx = &hctx, + .ioprio = IOPRIO_CLASS_IDLE + << IOPRIO_CLASS_SHIFT }; + static struct bio bio2 = { .bi_bdev = &bdev, + .bi_opf = REQ_OP_WRITE, + .bi_ioprio = IOPRIO_CLASS_BE + << IOPRIO_CLASS_SHIFT }; + static struct request rq2 = { .q = &queue, + .cmd_flags = REQ_OP_WRITE, + .__sector = 3, + .__data_len = 1, + .bio = &bio2, + .mq_hctx = &hctx, + .ioprio = IOPRIO_CLASS_BE + << IOPRIO_CLASS_SHIFT }; + static struct bio bio3 = { .bi_bdev = &bdev, + .bi_opf = REQ_OP_WRITE, + .bi_ioprio = IOPRIO_CLASS_RT + << IOPRIO_CLASS_SHIFT }; + static struct request rq3 = { .q = &queue, + .cmd_flags = REQ_OP_WRITE, + .__sector = 5, + .__data_len = 1, + .bio = &bio3, + .mq_hctx = &hctx, + .ioprio = IOPRIO_CLASS_RT + << IOPRIO_CLASS_SHIFT }; + struct request *rq; + static LIST_HEAD(rq_list); + + bdev.bd_disk = &disk; + bdev.bd_queue = &queue; + disk.queue = &queue; + + dd_init_sched(&queue, &mq_deadline); + dd_prepare_request(&rq1); + dd_prepare_request(&rq2); + dd_prepare_request(&rq3); + list_add_tail(&rq1.queuelist, &rq_list); + list_add_tail(&rq2.queuelist, &rq_list); + list_add_tail(&rq3.queuelist, &rq_list); + dd_insert_requests(&hctx, &rq_list, false); + rq = dd_dispatch_request(&hctx); + KUNIT_EXPECT_PTR_EQ(test, rq, &rq3); + dd_finish_request(rq); + rq = dd_dispatch_request(&hctx); + KUNIT_EXPECT_PTR_EQ(test, rq, &rq2); + dd_finish_request(rq); + rq = dd_dispatch_request(&hctx); + KUNIT_EXPECT_PTR_EQ(test, rq, &rq1); + dd_finish_request(rq); + dd_exit_sched(queue.elevator); +} + +/* + * Test that the write order is preserved if a higher I/O priority is assigned + * to higher LBAs. This test fails if dd_zone_prio() always returns + * DD_INVALID_PRIO. + */ +static void test_zone_prio(struct kunit *test) +{ + static struct block_device bdev; + static unsigned long seq_zones_wlock[1]; + static struct gendisk disk = { .conv_zones_bitmap = NULL, + .seq_zones_wlock = seq_zones_wlock, + .part0 = &bdev }; + static struct request_queue queue = { + .disk = &disk, + .limits = { .zoned = BLK_ZONED_HM, .chunk_sectors = 16 } + }; + static struct blk_mq_hw_ctx hctx = { .queue = &queue }; + static struct bio bio1 = { .bi_bdev = &bdev, + .bi_opf = REQ_OP_WRITE, + .bi_ioprio = IOPRIO_CLASS_IDLE + << IOPRIO_CLASS_SHIFT }; + static struct request rq1 = { .q = &queue, + .cmd_flags = REQ_OP_WRITE, + .__sector = 1, + .__data_len = 1, + .bio = &bio1, + .mq_hctx = &hctx, + .ioprio = IOPRIO_CLASS_IDLE + << IOPRIO_CLASS_SHIFT }; + static struct bio bio2 = { .bi_bdev = &bdev, + .bi_opf = REQ_OP_WRITE, + .bi_ioprio = IOPRIO_CLASS_BE + << IOPRIO_CLASS_SHIFT }; + static struct request rq2 = { .q = &queue, + .cmd_flags = REQ_OP_WRITE, + .__sector = 3, + .__data_len = 1, + .bio = &bio2, + .mq_hctx = &hctx, + .ioprio = IOPRIO_CLASS_BE + << IOPRIO_CLASS_SHIFT }; + static struct bio bio3 = { .bi_bdev = &bdev, + .bi_opf = REQ_OP_WRITE, + .bi_ioprio = IOPRIO_CLASS_RT + << IOPRIO_CLASS_SHIFT }; + static struct request rq3 = { .q = &queue, + .cmd_flags = REQ_OP_WRITE, + .__sector = 5, + .__data_len = 1, + .bio = &bio3, + .mq_hctx = &hctx, + .ioprio = IOPRIO_CLASS_RT + << IOPRIO_CLASS_SHIFT }; + struct request *rq; + static LIST_HEAD(rq_list); + + bdev.bd_disk = &disk; + bdev.bd_queue = &queue; + disk.queue = &queue; + + KUNIT_EXPECT_TRUE(test, blk_rq_is_seq_zoned_write(&rq1)); + KUNIT_EXPECT_TRUE(test, blk_rq_is_seq_zoned_write(&rq2)); + KUNIT_EXPECT_TRUE(test, blk_rq_is_seq_zoned_write(&rq3)); + + dd_init_sched(&queue, &mq_deadline); + dd_prepare_request(&rq1); + dd_prepare_request(&rq2); + dd_prepare_request(&rq3); + list_add_tail(&rq1.queuelist, &rq_list); + list_add_tail(&rq2.queuelist, &rq_list); + list_add_tail(&rq3.queuelist, &rq_list); + dd_insert_requests(&hctx, &rq_list, false); + rq = dd_dispatch_request(&hctx); + KUNIT_EXPECT_PTR_EQ(test, rq, &rq1); + dd_finish_request(rq); + rq = dd_dispatch_request(&hctx); + KUNIT_EXPECT_PTR_EQ(test, rq, &rq2); + dd_finish_request(rq); + rq = dd_dispatch_request(&hctx); + KUNIT_EXPECT_PTR_EQ(test, rq, &rq3); + dd_finish_request(rq); + dd_exit_sched(queue.elevator); +} + +static struct kunit_case mq_deadline_test_cases[] = { + KUNIT_CASE(test_ioprio), + KUNIT_CASE(test_zone_prio), + {} +}; + +static struct kunit_suite mq_deadline_test_suite = { + .name = "mq-deadline", + .test_cases = mq_deadline_test_cases, +}; +kunit_test_suite(mq_deadline_test_suite); + +MODULE_DESCRIPTION("mq-deadline unit tests"); +MODULE_AUTHOR("Bart Van Assche"); +MODULE_LICENSE("GPL"); diff --git a/include/linux/blk-mq.h b/include/linux/blk-mq.h index 1ab3081c82ed..e7fa81170b7c 100644 --- a/include/linux/blk-mq.h +++ b/include/linux/blk-mq.h @@ -1149,6 +1149,18 @@ static inline unsigned int blk_rq_zone_no(struct request *rq) return disk_zone_no(rq->q->disk, blk_rq_pos(rq)); } +/** + * blk_bio_is_seq_zoned_write() - Check if @bio requires write serialization. + * @bio: Bio to examine. + * + * Note: REQ_OP_ZONE_APPEND bios do not require serialization. + */ +static inline bool blk_bio_is_seq_zoned_write(struct bio *bio) +{ + return disk_zone_is_seq(bio->bi_bdev->bd_disk, bio->bi_iter.bi_sector) && + op_needs_zoned_write_locking(bio_op(bio)); +} + static inline unsigned int blk_rq_zone_is_seq(struct request *rq) { return disk_zone_is_seq(rq->q->disk, blk_rq_pos(rq)); @@ -1196,6 +1208,11 @@ static inline bool blk_req_can_dispatch_to_zone(struct request *rq) return !blk_req_zone_is_write_locked(rq); } #else /* CONFIG_BLK_DEV_ZONED */ +static inline bool blk_bio_is_seq_zoned_write(struct bio *bio) +{ + return false; +} + static inline bool blk_rq_is_seq_zoned_write(struct request *rq) { return false;