From patchwork Tue Jan 23 17:34:13 2024 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Jens Axboe X-Patchwork-Id: 13527917 Received: from mail-io1-f54.google.com (mail-io1-f54.google.com [209.85.166.54]) (using TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 (128/128 bits)) (No client certificate requested) by smtp.subspace.kernel.org (Postfix) with ESMTPS id 586BA80047 for ; Tue, 23 Jan 2024 17:40:27 +0000 (UTC) Authentication-Results: smtp.subspace.kernel.org; arc=none smtp.client-ip=209.85.166.54 ARC-Seal: i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1706031629; cv=none; b=aNCPLHPZ6fZRUetPfb0em50XSEw6Mcx2dREIL7Fe8X615Pvx1w2tiVXBftK6UvkSR7e5qmBXb1SR0cHXb0ytdk3uJOTbPkHtVFKDME2PMEHFSj3QVMkHzlC8sXM9RsW28n44SeDH111qJ2WmQoYgPI9+7K28QPtHYXz1mQ0mby8= ARC-Message-Signature: i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1706031629; c=relaxed/simple; bh=YRdOC/K4Kw35Qsd/uJLzZT5xSXTClNa+8FtHHnmdBoc=; h=From:To:Cc:Subject:Date:Message-ID:In-Reply-To:References: MIME-Version; b=EK3D1kPWcJui0mmuBi5erG8X+LVybN6l8xomraFbQcXzhXdKXzNZ/IYhKImG74+FFh3BHXZKrbyatg1laU0+Lo7+2SNGoruRAv/JFPbGunSCpY39MDGA7SDLMR+UymMcC41okxf3dIgq04jK0r2K5xJiS1+Sg4hsZm9vVkyCpbo= ARC-Authentication-Results: i=1; smtp.subspace.kernel.org; dmarc=none (p=none dis=none) header.from=kernel.dk; spf=pass smtp.mailfrom=kernel.dk; dkim=pass (2048-bit key) header.d=kernel-dk.20230601.gappssmtp.com header.i=@kernel-dk.20230601.gappssmtp.com header.b=SWsq30ly; arc=none smtp.client-ip=209.85.166.54 Authentication-Results: smtp.subspace.kernel.org; dmarc=none (p=none dis=none) header.from=kernel.dk Authentication-Results: smtp.subspace.kernel.org; spf=pass smtp.mailfrom=kernel.dk Authentication-Results: smtp.subspace.kernel.org; dkim=pass (2048-bit key) header.d=kernel-dk.20230601.gappssmtp.com header.i=@kernel-dk.20230601.gappssmtp.com header.b="SWsq30ly" Received: by mail-io1-f54.google.com with SMTP id ca18e2360f4ac-7bed82030faso59739539f.1 for ; Tue, 23 Jan 2024 09:40:27 -0800 (PST) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=kernel-dk.20230601.gappssmtp.com; s=20230601; t=1706031626; x=1706636426; darn=vger.kernel.org; h=content-transfer-encoding:mime-version:references:in-reply-to :message-id:date:subject:cc:to:from:from:to:cc:subject:date :message-id:reply-to; bh=qMRDiHz7VKi7s1ybtAQ77dKIgD14HxBezBGAiAO6Bmg=; b=SWsq30lyCkOFIFdeBm+jzihZEhkJUaYTtxed1Y5g6m+woiTQz2/HpYE/3aqn83QMzl d1AP5KnbNHkKvp71CkAyNDNET8gUdSj1T7W3hPtIrhVF1SgHMr5m3R5DVA53K3prF4+K 5l2Fx0qw/xZbJRaFFw2VJRjqL0DiJ9KHPhsPluAx+MzhPkz8qHIBKU5MawBTItWmVeiw SUSaCAdFgVUNmRWfbgqcM6pN5SKz0rF/rTfzRg3MkwLKO3IemfS2YqClgaktHMu2kD8E 0U+B5wm6F/5RgzQtnU5eHmQaIby7Z1j7AW7iPXklKKsWjnUCV0dRpbgrKBrBWmL1Dpig spxg== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20230601; t=1706031626; x=1706636426; h=content-transfer-encoding:mime-version:references:in-reply-to :message-id:date:subject:cc:to:from:x-gm-message-state:from:to:cc :subject:date:message-id:reply-to; bh=qMRDiHz7VKi7s1ybtAQ77dKIgD14HxBezBGAiAO6Bmg=; b=JVT00Myi+4fZD9wfQX0hJgVd8Nc1AlU0WRTNjoEMFyOgdXBzKH1UWm2c8ej05gYgei qjbvw6VaNtSnZSLzZMs/k4QNPbceuW7wvAQFHjt7JMozGuMBi9D9b1ZgYRoV4q9VMFHf 55hkTXgSE823Yfx5gvc69T1pwIeJwiyXH7WDmALgQtegCvAUM9tAZFfY4U5RnFsA2+ZD 1WnHdBQxUvm3I+X3FkOOfaM8lCn285e7G7vyJo5jcdIe02Al7Hcu2DQQgvsEJHsUNpcy fit64jGDPuQF88ecXGo1jRUrG4ehgr3pgpzupW0ECMn2YBrQ4rjNqquneNFeYk42ZIc7 FTjg== X-Gm-Message-State: AOJu0YyMsARQXesodJUtgrA0JfnxBaivfk2YLe8mOfnOBq9kdSbcaHj5 bwsXqRnut0E20W170xFLX7ugn89ZeiKUy1NjvDMQAKNIF/kLF0yEGpspVxrqW0EohtbF3B/Gbaz LVX0= X-Google-Smtp-Source: AGHT+IHyhdGzb5/8J+LIUTuQGHx614+1BWdAD+PE9rIUyz9ezbew8pbKZfK6ahAukBNmkALOmS94oQ== X-Received: by 2002:a5d:8942:0:b0:7bf:3651:4c6d with SMTP id b2-20020a5d8942000000b007bf36514c6dmr10357780iot.2.1706031626089; Tue, 23 Jan 2024 09:40:26 -0800 (PST) Received: from localhost.localdomain ([96.43.243.2]) by smtp.gmail.com with ESMTPSA id l15-20020a6b700f000000b007bf4b9fa2e6sm4647700ioc.47.2024.01.23.09.40.24 (version=TLS1_3 cipher=TLS_AES_256_GCM_SHA384 bits=256/256); Tue, 23 Jan 2024 09:40:24 -0800 (PST) From: Jens Axboe To: linux-block@vger.kernel.org Cc: Jens Axboe , Bart Van Assche Subject: [PATCH 1/8] block/mq-deadline: pass in queue directly to dd_insert_request() Date: Tue, 23 Jan 2024 10:34:13 -0700 Message-ID: <20240123174021.1967461-2-axboe@kernel.dk> X-Mailer: git-send-email 2.43.0 In-Reply-To: <20240123174021.1967461-1-axboe@kernel.dk> References: <20240123174021.1967461-1-axboe@kernel.dk> Precedence: bulk X-Mailing-List: linux-block@vger.kernel.org List-Id: List-Subscribe: List-Unsubscribe: MIME-Version: 1.0 The hardware queue isn't relevant, deadline only operates on the queue itself. Pass in the queue directly rather than the hardware queue, as that more clearly explains what is being operated on. Reviewed-by: Bart Van Assche Signed-off-by: Jens Axboe Reviewed-by: Johannes Thumshirn --- block/mq-deadline.c | 5 ++--- 1 file changed, 2 insertions(+), 3 deletions(-) diff --git a/block/mq-deadline.c b/block/mq-deadline.c index f958e79277b8..9b7563e9d638 100644 --- a/block/mq-deadline.c +++ b/block/mq-deadline.c @@ -792,10 +792,9 @@ static bool dd_bio_merge(struct request_queue *q, struct bio *bio, /* * add rq to rbtree and fifo */ -static void dd_insert_request(struct blk_mq_hw_ctx *hctx, struct request *rq, +static void dd_insert_request(struct request_queue *q, struct request *rq, blk_insert_t flags, struct list_head *free) { - struct request_queue *q = hctx->queue; struct deadline_data *dd = q->elevator->elevator_data; const enum dd_data_dir data_dir = rq_data_dir(rq); u16 ioprio = req_get_ioprio(rq); @@ -875,7 +874,7 @@ static void dd_insert_requests(struct blk_mq_hw_ctx *hctx, rq = list_first_entry(list, struct request, queuelist); list_del_init(&rq->queuelist); - dd_insert_request(hctx, rq, flags, &free); + dd_insert_request(q, rq, flags, &free); } spin_unlock(&dd->lock); From patchwork Tue Jan 23 17:34:14 2024 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Jens Axboe X-Patchwork-Id: 13527918 Received: from mail-io1-f42.google.com (mail-io1-f42.google.com [209.85.166.42]) (using TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 (128/128 bits)) (No client certificate requested) by smtp.subspace.kernel.org (Postfix) with ESMTPS id 579E180039 for ; Tue, 23 Jan 2024 17:40:29 +0000 (UTC) Authentication-Results: smtp.subspace.kernel.org; arc=none smtp.client-ip=209.85.166.42 ARC-Seal: i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1706031631; cv=none; b=o06StOyzVpbgHceZJqrIZhuQWWzx3xnJZV6E51F+oE1ekj4/3wfNEME0sM11GtN1MO0Jc78NBUhnLcbunYYfuyOpjm4qg2YrYbbauLN2b1bLoo/yALsNY06W2Odm/GdNpNLGj7KdrRjcadGHtizi4YKnEQZ05rqH1NLTUK2aNLA= ARC-Message-Signature: i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1706031631; c=relaxed/simple; bh=HKjrUm09tWm2y8NxmKTbCZTTthQcNTD7sCcoEi7D7q8=; h=From:To:Cc:Subject:Date:Message-ID:In-Reply-To:References: MIME-Version; b=kB3J2nJwYTfeAwA3snQFyA7cMuuWbFI/h+UWHpllXOLFq34Q8OHiPTNDt1jyw9AaYAYpDYSRwLhpQTbMxatOVy2A+TAneeb3PCjeviWm7UIQ/0dYnIc3xsYDr4MSLv9KTfbJt/7IRTU84uNM/5zUACYH01Loei4L1ukDfP+1/do= ARC-Authentication-Results: i=1; smtp.subspace.kernel.org; dmarc=none (p=none dis=none) header.from=kernel.dk; spf=pass smtp.mailfrom=kernel.dk; dkim=pass (2048-bit key) header.d=kernel-dk.20230601.gappssmtp.com header.i=@kernel-dk.20230601.gappssmtp.com header.b=UoixmkBv; arc=none smtp.client-ip=209.85.166.42 Authentication-Results: smtp.subspace.kernel.org; dmarc=none (p=none dis=none) header.from=kernel.dk Authentication-Results: smtp.subspace.kernel.org; spf=pass smtp.mailfrom=kernel.dk Authentication-Results: smtp.subspace.kernel.org; dkim=pass (2048-bit key) header.d=kernel-dk.20230601.gappssmtp.com header.i=@kernel-dk.20230601.gappssmtp.com header.b="UoixmkBv" Received: by mail-io1-f42.google.com with SMTP id ca18e2360f4ac-7bb06f56fe9so51895939f.0 for ; Tue, 23 Jan 2024 09:40:29 -0800 (PST) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=kernel-dk.20230601.gappssmtp.com; s=20230601; t=1706031628; x=1706636428; darn=vger.kernel.org; h=content-transfer-encoding:mime-version:references:in-reply-to :message-id:date:subject:cc:to:from:from:to:cc:subject:date :message-id:reply-to; bh=DPFAUZtHFunfe8V5aVWd0ro+6nJcJ7bLRkCM9nE4cp8=; b=UoixmkBvrw6aWutFPz5smkSMw8vHw+pT5u4GPeXJEi9YDia4Z2CohRCoMwKkUUBBDD Ht3tzLZJ3YUfx8Nspo7BqzhbVJ9+uS7QN0YmuXWwZLrBcJTnzZ+PuGss3BoUsu09V/LA kRJqt4NVm1NUdfK9D1ANMtQRkjX3bQ/DtZc41QSikMuGy3WJayPMNjAmP8yWZqB3pcDr 05x9zHUwBGzRD2qjOoKhvweD3szfkOnoOWwx4IrWu9CkDo0l0KrBnr28Q7+anCpn7ZA3 8gQ1i9JMgMotcrAPfajKT+zHnGfoShTzRCKmRw+aiS2N0tvvTUKQpoD7sDGk4lSm3XjL wJBw== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20230601; t=1706031628; x=1706636428; h=content-transfer-encoding:mime-version:references:in-reply-to :message-id:date:subject:cc:to:from:x-gm-message-state:from:to:cc :subject:date:message-id:reply-to; bh=DPFAUZtHFunfe8V5aVWd0ro+6nJcJ7bLRkCM9nE4cp8=; b=DN+cp0HxNWbsUKEfoVMoXqeE1ZMu9aA3Gba5w0pIKZPkYn7N6AZNffa7wD0fdJCLPx /4ZnCmHFiuBDObSAAQwukHOJGLbl99mBHBaEKHYRJ3ME34UYSIF7hi9LhSrgD4NCZfhw a6fHdTruol7E2GmeelSoU1jr2rV5xwGBbV3EOWJRLc665o5zDCR/bfp9g8z7XKOcYf5I +s7da+nlIwKJDUtY+T8BM9Km9wT7B51A+7z1e2Lnnad3b7WAUdn1nzeMbjulamOKTcw6 NfIZcE01gUMzgcRy1ytfJUnzBhEoBf3b9ym3fexcxwmOmjbEY99747cB6oXcO6AR8C35 xEtQ== X-Gm-Message-State: AOJu0Yww46hZ0Hob5+s2Ql4ApuqkQT2X27Qz5Vc9VmwqqhXviA5aLukJ Up8RnxAdy2WT6DhnhHeCer/b8OjJCD54f7tvgp8sAgZmxZBQFjwUcYeiYBKdtxnC/68mzcsK/LQ nO5M= X-Google-Smtp-Source: AGHT+IFVg2nRmNdwqYXo4ThEpQ+hr3WQeAXgGMqtk9FoTK7TK/ofcrUdXLLLfivod0DtcY+FlM8Dwg== X-Received: by 2002:a05:6602:123b:b0:7be:edbc:629f with SMTP id z27-20020a056602123b00b007beedbc629fmr10390570iot.0.1706031627890; Tue, 23 Jan 2024 09:40:27 -0800 (PST) Received: from localhost.localdomain ([96.43.243.2]) by smtp.gmail.com with ESMTPSA id l15-20020a6b700f000000b007bf4b9fa2e6sm4647700ioc.47.2024.01.23.09.40.26 (version=TLS1_3 cipher=TLS_AES_256_GCM_SHA384 bits=256/256); Tue, 23 Jan 2024 09:40:26 -0800 (PST) From: Jens Axboe To: linux-block@vger.kernel.org Cc: Jens Axboe Subject: [PATCH 2/8] block/mq-deadline: serialize request dispatching Date: Tue, 23 Jan 2024 10:34:14 -0700 Message-ID: <20240123174021.1967461-3-axboe@kernel.dk> X-Mailer: git-send-email 2.43.0 In-Reply-To: <20240123174021.1967461-1-axboe@kernel.dk> References: <20240123174021.1967461-1-axboe@kernel.dk> Precedence: bulk X-Mailing-List: linux-block@vger.kernel.org List-Id: List-Subscribe: List-Unsubscribe: MIME-Version: 1.0 If we're entering request dispatch but someone else is already dispatching, then just skip this dispatch. We know IO is inflight and this will trigger another dispatch event for any completion. This will potentially cause slightly lower queue depth for contended cases, but those are slowed down anyway and this should not cause an issue. By itself, this patch doesn't help a whole lot, as the dispatch lock contention reduction is just eating up by the same dd->lock now seeing increased insertion contention. But it's required work to be able to reduce the lock contention in general. Signed-off-by: Jens Axboe Reviewed-by: Bart Van Assche Reviewed-by: Johannes Thumshirn --- block/mq-deadline.c | 31 ++++++++++++++++++++++++++----- 1 file changed, 26 insertions(+), 5 deletions(-) diff --git a/block/mq-deadline.c b/block/mq-deadline.c index 9b7563e9d638..79bc3b6784b3 100644 --- a/block/mq-deadline.c +++ b/block/mq-deadline.c @@ -79,10 +79,20 @@ struct dd_per_prio { struct io_stats_per_prio stats; }; +enum { + DD_DISPATCHING = 0, +}; + struct deadline_data { /* * run time data */ + struct { + spinlock_t lock; + spinlock_t zone_lock; + } ____cacheline_aligned_in_smp; + + unsigned long run_state; struct dd_per_prio per_prio[DD_PRIO_COUNT]; @@ -100,9 +110,6 @@ struct deadline_data { int front_merges; u32 async_depth; int prio_aging_expire; - - spinlock_t lock; - spinlock_t zone_lock; }; /* Maps an I/O priority class to a deadline scheduler priority. */ @@ -600,6 +607,18 @@ static struct request *dd_dispatch_request(struct blk_mq_hw_ctx *hctx) struct request *rq; enum dd_prio prio; + /* + * If someone else is already dispatching, skip this one. This will + * defer the next dispatch event to when something completes, and could + * potentially lower the queue depth for contended cases. + * + * See the logic in blk_mq_do_dispatch_sched(), which loops and + * retries if nothing is dispatched. + */ + if (test_bit(DD_DISPATCHING, &dd->run_state) || + test_and_set_bit_lock(DD_DISPATCHING, &dd->run_state)) + return NULL; + spin_lock(&dd->lock); rq = dd_dispatch_prio_aged_requests(dd, now); if (rq) @@ -616,6 +635,7 @@ static struct request *dd_dispatch_request(struct blk_mq_hw_ctx *hctx) } unlock: + clear_bit_unlock(DD_DISPATCHING, &dd->run_state); spin_unlock(&dd->lock); return rq; @@ -706,6 +726,9 @@ static int dd_init_sched(struct request_queue *q, struct elevator_type *e) eq->elevator_data = dd; + spin_lock_init(&dd->lock); + spin_lock_init(&dd->zone_lock); + for (prio = 0; prio <= DD_PRIO_MAX; prio++) { struct dd_per_prio *per_prio = &dd->per_prio[prio]; @@ -722,8 +745,6 @@ static int dd_init_sched(struct request_queue *q, struct elevator_type *e) dd->last_dir = DD_WRITE; dd->fifo_batch = fifo_batch; dd->prio_aging_expire = prio_aging_expire; - spin_lock_init(&dd->lock); - spin_lock_init(&dd->zone_lock); /* We dispatch from request queue wide instead of hw queue */ blk_queue_flag_set(QUEUE_FLAG_SQ_SCHED, q); From patchwork Tue Jan 23 17:34:15 2024 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Jens Axboe X-Patchwork-Id: 13527919 Received: from mail-io1-f51.google.com (mail-io1-f51.google.com [209.85.166.51]) (using TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 (128/128 bits)) (No client certificate requested) by smtp.subspace.kernel.org (Postfix) with ESMTPS id 2EDD780039 for ; Tue, 23 Jan 2024 17:40:30 +0000 (UTC) Authentication-Results: smtp.subspace.kernel.org; arc=none smtp.client-ip=209.85.166.51 ARC-Seal: i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1706031633; cv=none; b=IQc1xGzLleBjHChgretX1WDv0pB2II/WLXu4oOy313syA0OM3Plpf9wYFGfx3ZbFwffILAuZKMRUGth4w2W7sSDxxiYJ/mbuI2+iYdN/NkT4yUZWsZJAsdhMR+BNFqJubSAgqugCI6Nd+WhO14YwafwO6pAtG1MUhDtAZP3+OZ0= ARC-Message-Signature: i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1706031633; c=relaxed/simple; bh=Tl6mGZbZRuoiBadYjH325snqUfqroPfSghWqjGW9KLA=; h=From:To:Cc:Subject:Date:Message-ID:In-Reply-To:References: MIME-Version; b=EIlGE3Bu4FZaEmrTQRoKrzUJefbFlTi2uJw6Lkdw/vn7CJNRBwtgf/bqk928LiPBr2Ukj0AemKEamwm650vx7uP+1iW2yj6PF7khlVVJX8dJ0Sf2t1PJWcAW5oH59QY5+qW0gb6jire+AMRWxIAJKEe5xZp/Ent3kIcTAQRjuzE= ARC-Authentication-Results: i=1; smtp.subspace.kernel.org; dmarc=none (p=none dis=none) header.from=kernel.dk; spf=pass smtp.mailfrom=kernel.dk; dkim=pass (2048-bit key) header.d=kernel-dk.20230601.gappssmtp.com header.i=@kernel-dk.20230601.gappssmtp.com header.b=x5B+dyZH; arc=none smtp.client-ip=209.85.166.51 Authentication-Results: smtp.subspace.kernel.org; dmarc=none (p=none dis=none) header.from=kernel.dk Authentication-Results: smtp.subspace.kernel.org; spf=pass smtp.mailfrom=kernel.dk Authentication-Results: smtp.subspace.kernel.org; dkim=pass (2048-bit key) header.d=kernel-dk.20230601.gappssmtp.com header.i=@kernel-dk.20230601.gappssmtp.com header.b="x5B+dyZH" Received: by mail-io1-f51.google.com with SMTP id ca18e2360f4ac-7beeeb1ba87so61743339f.0 for ; Tue, 23 Jan 2024 09:40:30 -0800 (PST) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=kernel-dk.20230601.gappssmtp.com; s=20230601; t=1706031630; x=1706636430; darn=vger.kernel.org; h=content-transfer-encoding:mime-version:references:in-reply-to :message-id:date:subject:cc:to:from:from:to:cc:subject:date :message-id:reply-to; bh=U9WFpwLwL00WE/4klKg8cmhjNxww+jJcfBSHYS0F1rQ=; b=x5B+dyZHHG9y4Q3AcunSGUYCw3HUrFdXHIpWI97YQO6LHiIEdjYpeYG66W7/G2lb1Y nlPk0rdsYU42+LSUfO8wd/baKWlrWvNmnsrFSoSVyl7l0u14Ovyoveot9INwGinV8L4g U1qJZD3rfWb0QRRoNIHoJn/wZVukJuSG0QpKNPZ4rL+lfbuEGe9KeSAEkjI+/7TlQNRT IRyRPyAVlA3dmZgsKAqnqkrbwtieDqHIeTpLPy3Qt7Gp8yrUHQoU3BGYFnwxVv218lSO iar3aJEU2FmyI6qQxHZIOeebPFXYMwokmKEE8HdkSrnhoGZvmFZYMqNPUyRUHxjIj5Fa iW1A== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20230601; t=1706031630; x=1706636430; h=content-transfer-encoding:mime-version:references:in-reply-to :message-id:date:subject:cc:to:from:x-gm-message-state:from:to:cc :subject:date:message-id:reply-to; bh=U9WFpwLwL00WE/4klKg8cmhjNxww+jJcfBSHYS0F1rQ=; b=wv91Z/VmLiGsp5bRZzqmIlhe5gCoODr9ZDMnhgyYZ64AwhJyfotY6YKC51kzuS5jML hgPise5Zvlvjgv94Nc1TWMqza+RurY03tLiyMymigqlhLyP1wb1L28tBuZNd1pzF4rQY uSSQGkKHBpwKg2hY0gdafaQldWDLtdniT8QP8qlycHZ6tbR1W/B3eQ/rWk3l/yXf6p+o oAxHx7sXw7UUc4hIvtIXKR8ZmlY4nWNYDk4Roz3prOeoibffo3TnXY4jbG39y+TcZgPH oudoqqqxTJ6m0vDTmTBgfp73nVtAiQN/RXuKwMQc14AEeQ8pvA1Ozg5bnSbGCokn5sum ujsw== X-Gm-Message-State: AOJu0YzupFmlRaMLr162Z5ezC4MJ2EaqLN/Ti3qGSe02TnbT18jTi8XD EjRVp9O0LhsVOSCcvwglSjeUaYR+Ivc495k4M70AbO8Rnf5ZuCwdRQtiyHa2WxqNcM3mglHBfSq q4UU= X-Google-Smtp-Source: AGHT+IF+wuBuYw+1lcVuH4lifKJFYDhSKr8AqqbOkvb5vR1nP/qK4JZjjUmCAA62nyMk+olY9wK3AQ== X-Received: by 2002:a05:6602:4145:b0:7bf:60bc:7f1e with SMTP id bv5-20020a056602414500b007bf60bc7f1emr11477190iob.1.1706031629792; Tue, 23 Jan 2024 09:40:29 -0800 (PST) Received: from localhost.localdomain ([96.43.243.2]) by smtp.gmail.com with ESMTPSA id l15-20020a6b700f000000b007bf4b9fa2e6sm4647700ioc.47.2024.01.23.09.40.27 (version=TLS1_3 cipher=TLS_AES_256_GCM_SHA384 bits=256/256); Tue, 23 Jan 2024 09:40:28 -0800 (PST) From: Jens Axboe To: linux-block@vger.kernel.org Cc: Jens Axboe Subject: [PATCH 3/8] block/mq-deadline: skip expensive merge lookups if contended Date: Tue, 23 Jan 2024 10:34:15 -0700 Message-ID: <20240123174021.1967461-4-axboe@kernel.dk> X-Mailer: git-send-email 2.43.0 In-Reply-To: <20240123174021.1967461-1-axboe@kernel.dk> References: <20240123174021.1967461-1-axboe@kernel.dk> Precedence: bulk X-Mailing-List: linux-block@vger.kernel.org List-Id: List-Subscribe: List-Unsubscribe: MIME-Version: 1.0 We do several stages of merging in the block layer - the most likely one to work is also the cheap one, merging direct in the per-task plug when IO is submitted. Getting merges outside of that is a lot less likely, but IO schedulers may still maintain internal data structures to facilitate merge lookups outside of the plug. Make mq-deadline skip expensive merge lookups if the queue lock is already contended. The likelihood of getting a merge here is not very high, hence it should not be a problem skipping the attempt in the also unlikely event that the queue is already contended. Perf diff shows the difference between a random read/write workload with 4 threads doing IO, with expensive merges turned on and off: 25.00% +61.94% [kernel.kallsyms] [k] queued_spin_lock_slowpath where we almost quadruple the lock contention by attempting these expensive merges. Signed-off-by: Jens Axboe Reviewed-by: Johannes Thumshirn --- block/mq-deadline.c | 14 +++++++++++++- 1 file changed, 13 insertions(+), 1 deletion(-) diff --git a/block/mq-deadline.c b/block/mq-deadline.c index 79bc3b6784b3..740b94f36cac 100644 --- a/block/mq-deadline.c +++ b/block/mq-deadline.c @@ -800,7 +800,19 @@ static bool dd_bio_merge(struct request_queue *q, struct bio *bio, struct request *free = NULL; bool ret; - spin_lock(&dd->lock); + /* + * bio merging is called for every bio queued, and it's very easy + * to run into contention because of that. If we fail getting + * the dd lock, just skip this merge attempt. For related IO, the + * plug will be the successful merging point. If we get here, we + * already failed doing the obvious merge. Chances of actually + * getting a merge off this path is a lot slimmer, so skipping an + * occassional lookup that will most likely not succeed anyway should + * not be a problem. + */ + if (!spin_trylock(&dd->lock)) + return false; + ret = blk_mq_sched_try_merge(q, bio, nr_segs, &free); spin_unlock(&dd->lock); From patchwork Tue Jan 23 17:34:16 2024 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Jens Axboe X-Patchwork-Id: 13527920 Received: from mail-io1-f53.google.com (mail-io1-f53.google.com [209.85.166.53]) (using TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 (128/128 bits)) (No client certificate requested) by smtp.subspace.kernel.org (Postfix) with ESMTPS id F3AD180058 for ; Tue, 23 Jan 2024 17:40:32 +0000 (UTC) Authentication-Results: smtp.subspace.kernel.org; arc=none smtp.client-ip=209.85.166.53 ARC-Seal: i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1706031634; cv=none; b=fJciO4EJ8Akvb2G+COxlnU0sC0FsWwvwYEaSSFgpt1fA5nWqNTJkgPzeikB5N7v9gArRqjbpdOPuK5bTvzjR9vBGxbG3q+hg2qk+0NHcvYRUbJPD3chnRZVefu/bZacNSioyM2SbJ6/ORf7L/JYR631e3c4YpC3Twf63RibAnqg= ARC-Message-Signature: i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1706031634; c=relaxed/simple; bh=rctKEqZgFH8GBDRhKSDbQ7fv+wiyAwfvfZ94Npf4OAs=; h=From:To:Cc:Subject:Date:Message-ID:In-Reply-To:References: MIME-Version; b=uIthPjehk6vYC2Bx34RUk0aN2lQ5NKHsyDz2AISOHygNoWg7CAR3iuOAW2Aj9XSPXtvUQ/a3+ZWShBY0YttYmXfNDYQk38q7P1rb8rf/aT8No2RD5khYvF+nElD3j6xNKQvOB7bfE8o7NL9E2DuvnhTIJGNwFbz9bH/JUasO7gs= ARC-Authentication-Results: i=1; smtp.subspace.kernel.org; dmarc=none (p=none dis=none) header.from=kernel.dk; spf=pass smtp.mailfrom=kernel.dk; dkim=pass (2048-bit key) header.d=kernel-dk.20230601.gappssmtp.com header.i=@kernel-dk.20230601.gappssmtp.com header.b=2+30bjJR; arc=none smtp.client-ip=209.85.166.53 Authentication-Results: smtp.subspace.kernel.org; dmarc=none (p=none dis=none) header.from=kernel.dk Authentication-Results: smtp.subspace.kernel.org; spf=pass smtp.mailfrom=kernel.dk Authentication-Results: smtp.subspace.kernel.org; dkim=pass (2048-bit key) header.d=kernel-dk.20230601.gappssmtp.com header.i=@kernel-dk.20230601.gappssmtp.com header.b="2+30bjJR" Received: by mail-io1-f53.google.com with SMTP id ca18e2360f4ac-7bed82030faso59740839f.1 for ; Tue, 23 Jan 2024 09:40:32 -0800 (PST) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=kernel-dk.20230601.gappssmtp.com; s=20230601; t=1706031631; x=1706636431; darn=vger.kernel.org; h=content-transfer-encoding:mime-version:references:in-reply-to :message-id:date:subject:cc:to:from:from:to:cc:subject:date :message-id:reply-to; bh=ldNwZctVsTHz0OYw14xBNqcgq5YNaiT4939FOB1Vvs0=; b=2+30bjJRpxRuA4zLMEtu9qpEUebuXTv6tgLB8ncTL6gr/IMf1GuvlwRudNoqz4pPi7 ImM/KBfIbRH7Se/V7dBBTwsleVTMdzwtROiTNur77QZsKSeXVfpYIbkrgsuvDRCZ1LEt P9dqOgO9i/whlKsl2ecMxOp2wvHYhIbkyujNF7hSklC3CshfsjWss2X5QGE0w8Iz9JOd a3OyCGFObsYy3k9I8Q/oDksPQrYCO468rRGafdf4kv6Bs/AELuNt1HiAXTbCilfWAJRv vb5F9syvzzhgc2IatDbvYnAimm5ukz6Hqlh90nN8SgzXuyjcII4bUX3rguStJQg11NJM LWlw== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20230601; t=1706031631; x=1706636431; h=content-transfer-encoding:mime-version:references:in-reply-to :message-id:date:subject:cc:to:from:x-gm-message-state:from:to:cc :subject:date:message-id:reply-to; bh=ldNwZctVsTHz0OYw14xBNqcgq5YNaiT4939FOB1Vvs0=; b=Eielfz6Ve1X7AUXb0nUyQ+rX8kqwdezsX41CHx+LsmQ6mv5DwRP+pqrM56cIKip/cJ Z1d+cXDuZcK9Ck+iRY1bzwMgLMiQzDOdt3bg6sqmJb/FTnt3tg3dIx103AOKETcT0GR3 U/Mf8lEXmP0Dt2EDsDogt9FBEYpV3xhpPy24PYrnzjg7XJLTTEHTGDdslyuX6/Rn7TfK k5ACmmEeL0Si44h5PiY6sKiLWlfNNi++o+JuFiKzlnCjiMXhMM2+q1KRD7YEPYv3qBew OfHjCdzAXEVklF0pkjCtiQpsfsLMc0JLxVYvok80d78XVw+uoJHlaIWWa2eAexTC4XNs T26g== X-Gm-Message-State: AOJu0Yzc8/2dGo5qVwAafJMLndnS3P2b4uYOam/CeLDLyD1qBdRQt5Xv Uoe22cLwRLRVdT2g+6N5/BccquvZqEqWyHL7cCRpNQlNfgqy/p0e6MHjcTRfylO3129YvbTKCac pr8Q= X-Google-Smtp-Source: AGHT+IHpNVN4x1bY43D8YgsNqUcNDgK1WZ0EdCGBNJoa/dJX2yTMYgJUKMcfP0Z3S43eSaHDPce5SA== X-Received: by 2002:a05:6602:4145:b0:7bf:60bc:7f1e with SMTP id bv5-20020a056602414500b007bf60bc7f1emr11477265iob.1.1706031631637; Tue, 23 Jan 2024 09:40:31 -0800 (PST) Received: from localhost.localdomain ([96.43.243.2]) by smtp.gmail.com with ESMTPSA id l15-20020a6b700f000000b007bf4b9fa2e6sm4647700ioc.47.2024.01.23.09.40.29 (version=TLS1_3 cipher=TLS_AES_256_GCM_SHA384 bits=256/256); Tue, 23 Jan 2024 09:40:30 -0800 (PST) From: Jens Axboe To: linux-block@vger.kernel.org Cc: Bart Van Assche , Jens Axboe Subject: [PATCH 4/8] block/mq-deadline: use separate insertion lists Date: Tue, 23 Jan 2024 10:34:16 -0700 Message-ID: <20240123174021.1967461-5-axboe@kernel.dk> X-Mailer: git-send-email 2.43.0 In-Reply-To: <20240123174021.1967461-1-axboe@kernel.dk> References: <20240123174021.1967461-1-axboe@kernel.dk> Precedence: bulk X-Mailing-List: linux-block@vger.kernel.org List-Id: List-Subscribe: List-Unsubscribe: MIME-Version: 1.0 From: Bart Van Assche Reduce lock contention on dd->lock by calling dd_insert_request() from inside the dispatch callback instead of from the insert callback. This patch is inspired by a patch from Jens. With the previous dispatch and merge optimization, this drastically reduces contention for a sample cases of 32 threads doing IO to devices. The test case looks as follows: fio --bs=512 --group_reporting=1 --gtod_reduce=1 --invalidate=1 \ --ioengine=io_uring --norandommap --runtime=60 --rw=randread \ --thread --time_based=1 --buffered=0 --fixedbufs=1 --numjobs=32 \ --iodepth=4 --iodepth_batch_submit=4 --iodepth_batch_complete=4 \ --name=scaletest --filename=/dev/$DEV Before: Device IOPS sys contention diff ==================================================== null_blk 879K 89% 93.6% nvme0n1 901K 86% 94.5% and after this and the previous two patches: Device IOPS sys contention diff ==================================================== null_blk 2867K 11.1% ~6.0% +226% nvme0n1 3162K 9.9% ~5.0% +250% which basically eliminates all of the lock contention, it's down to more normal levels. The throughput increases show that nicely, with more than a 200% improvement for both cases. Signed-off-by: Bart Van Assche [axboe: expand commit message with more details and perf results] Signed-off-by: Jens Axboe Reviewed-by: Bart Van Assche Reviewed-by: Johannes Thumshirn --- block/mq-deadline.c | 66 ++++++++++++++++++++++++++++++++++++--------- 1 file changed, 53 insertions(+), 13 deletions(-) diff --git a/block/mq-deadline.c b/block/mq-deadline.c index 740b94f36cac..1b0de4fc3958 100644 --- a/block/mq-deadline.c +++ b/block/mq-deadline.c @@ -89,11 +89,15 @@ struct deadline_data { */ struct { spinlock_t lock; + spinlock_t insert_lock; spinlock_t zone_lock; } ____cacheline_aligned_in_smp; unsigned long run_state; + struct list_head at_head; + struct list_head at_tail; + struct dd_per_prio per_prio[DD_PRIO_COUNT]; /* Data direction of latest dispatched request. */ @@ -120,6 +124,9 @@ static const enum dd_prio ioprio_class_to_prio[] = { [IOPRIO_CLASS_IDLE] = DD_IDLE_PRIO, }; +static void dd_insert_request(struct request_queue *q, struct request *rq, + blk_insert_t flags, struct list_head *free); + static inline struct rb_root * deadline_rb_root(struct dd_per_prio *per_prio, struct request *rq) { @@ -592,6 +599,33 @@ static struct request *dd_dispatch_prio_aged_requests(struct deadline_data *dd, return NULL; } +static void __dd_do_insert(struct request_queue *q, blk_insert_t flags, + struct list_head *list, struct list_head *free) +{ + while (!list_empty(list)) { + struct request *rq; + + rq = list_first_entry(list, struct request, queuelist); + list_del_init(&rq->queuelist); + dd_insert_request(q, rq, flags, free); + } +} + +static void dd_do_insert(struct request_queue *q, struct list_head *free) +{ + struct deadline_data *dd = q->elevator->elevator_data; + LIST_HEAD(at_head); + LIST_HEAD(at_tail); + + spin_lock(&dd->insert_lock); + list_splice_init(&dd->at_head, &at_head); + list_splice_init(&dd->at_tail, &at_tail); + spin_unlock(&dd->insert_lock); + + __dd_do_insert(q, BLK_MQ_INSERT_AT_HEAD, &at_head, free); + __dd_do_insert(q, 0, &at_tail, free); +} + /* * Called from blk_mq_run_hw_queue() -> __blk_mq_sched_dispatch_requests(). * @@ -602,10 +636,12 @@ static struct request *dd_dispatch_prio_aged_requests(struct deadline_data *dd, */ static struct request *dd_dispatch_request(struct blk_mq_hw_ctx *hctx) { - struct deadline_data *dd = hctx->queue->elevator->elevator_data; + struct request_queue *q = hctx->queue; + struct deadline_data *dd = q->elevator->elevator_data; const unsigned long now = jiffies; struct request *rq; enum dd_prio prio; + LIST_HEAD(free); /* * If someone else is already dispatching, skip this one. This will @@ -620,6 +656,7 @@ static struct request *dd_dispatch_request(struct blk_mq_hw_ctx *hctx) return NULL; spin_lock(&dd->lock); + dd_do_insert(q, &free); rq = dd_dispatch_prio_aged_requests(dd, now); if (rq) goto unlock; @@ -638,6 +675,7 @@ static struct request *dd_dispatch_request(struct blk_mq_hw_ctx *hctx) clear_bit_unlock(DD_DISPATCHING, &dd->run_state); spin_unlock(&dd->lock); + blk_mq_free_requests(&free); return rq; } @@ -727,8 +765,12 @@ static int dd_init_sched(struct request_queue *q, struct elevator_type *e) eq->elevator_data = dd; spin_lock_init(&dd->lock); + spin_lock_init(&dd->insert_lock); spin_lock_init(&dd->zone_lock); + INIT_LIST_HEAD(&dd->at_head); + INIT_LIST_HEAD(&dd->at_tail); + for (prio = 0; prio <= DD_PRIO_MAX; prio++) { struct dd_per_prio *per_prio = &dd->per_prio[prio]; @@ -899,19 +941,13 @@ static void dd_insert_requests(struct blk_mq_hw_ctx *hctx, { struct request_queue *q = hctx->queue; struct deadline_data *dd = q->elevator->elevator_data; - LIST_HEAD(free); - - spin_lock(&dd->lock); - while (!list_empty(list)) { - struct request *rq; - rq = list_first_entry(list, struct request, queuelist); - list_del_init(&rq->queuelist); - dd_insert_request(q, rq, flags, &free); - } - spin_unlock(&dd->lock); - - blk_mq_free_requests(&free); + spin_lock(&dd->insert_lock); + if (flags & BLK_MQ_INSERT_AT_HEAD) + list_splice_init(list, &dd->at_head); + else + list_splice_init(list, &dd->at_tail); + spin_unlock(&dd->insert_lock); } /* Callback from inside blk_mq_rq_ctx_init(). */ @@ -990,6 +1026,10 @@ static bool dd_has_work(struct blk_mq_hw_ctx *hctx) struct deadline_data *dd = hctx->queue->elevator->elevator_data; enum dd_prio prio; + if (!list_empty_careful(&dd->at_head) || + !list_empty_careful(&dd->at_tail)) + return true; + for (prio = 0; prio <= DD_PRIO_MAX; prio++) if (dd_has_work_for_prio(&dd->per_prio[prio])) return true; From patchwork Tue Jan 23 17:34:17 2024 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Jens Axboe X-Patchwork-Id: 13527921 Received: from mail-io1-f53.google.com (mail-io1-f53.google.com [209.85.166.53]) (using TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 (128/128 bits)) (No client certificate requested) by smtp.subspace.kernel.org (Postfix) with ESMTPS id 9F8068005D for ; Tue, 23 Jan 2024 17:40:34 +0000 (UTC) Authentication-Results: smtp.subspace.kernel.org; arc=none smtp.client-ip=209.85.166.53 ARC-Seal: i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1706031636; cv=none; b=Sf0yTNPJnQaRoQRR1UxsrxzEqMlMjUMKmWXfp/oaBq8fIeFOr0+ZjpEI48XSsrW+9sDtcb+2cWJVjTSb0ub1jMwU/DzTP60Wg113hn5IQPuVbZmXXkjGBVp4XuQb6Swy/0EZ77EFuZhzcZ7t8ixdBtn16iAGvsiiD2+8KVKWXZc= ARC-Message-Signature: i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1706031636; c=relaxed/simple; bh=bmcTErcUzGsFmwKKq2XNbsNIv5eloCUHW5cDrCRoPtw=; h=From:To:Cc:Subject:Date:Message-ID:In-Reply-To:References: MIME-Version; b=cU6B3KC90R5dfJnUd3B8r6Hvb3V222THw/bNCErCeHWAfVbJfs2Vvww3dQxWqpJOhcY7A2BrEveUmy120j86MZyclT4SIVy09k/sp0Zjwof++VP1iHVTDBl5Dzt4nB9fj1LHBkxBDgsVGPX9FSs+mIldyGCIikpHgfI2DO1nYPM= ARC-Authentication-Results: i=1; smtp.subspace.kernel.org; dmarc=none (p=none dis=none) header.from=kernel.dk; spf=pass smtp.mailfrom=kernel.dk; dkim=pass (2048-bit key) header.d=kernel-dk.20230601.gappssmtp.com header.i=@kernel-dk.20230601.gappssmtp.com header.b=3HbjR2jR; arc=none smtp.client-ip=209.85.166.53 Authentication-Results: smtp.subspace.kernel.org; dmarc=none (p=none dis=none) header.from=kernel.dk Authentication-Results: smtp.subspace.kernel.org; spf=pass smtp.mailfrom=kernel.dk Authentication-Results: smtp.subspace.kernel.org; dkim=pass (2048-bit key) header.d=kernel-dk.20230601.gappssmtp.com header.i=@kernel-dk.20230601.gappssmtp.com header.b="3HbjR2jR" Received: by mail-io1-f53.google.com with SMTP id ca18e2360f4ac-7bee9f626caso62181639f.0 for ; Tue, 23 Jan 2024 09:40:34 -0800 (PST) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=kernel-dk.20230601.gappssmtp.com; s=20230601; t=1706031633; x=1706636433; darn=vger.kernel.org; h=content-transfer-encoding:mime-version:references:in-reply-to :message-id:date:subject:cc:to:from:from:to:cc:subject:date :message-id:reply-to; bh=/QSidYI5kedHrpCI5ufZ8srx64p2MxR2IVg8u5MrqJU=; b=3HbjR2jRUSKldTYH625VK9lm+DRbU2usk/WSfoagOFw8cai2BOwLWoBfawM9lBAzmU gO+JOm3q1SL7t6DIFqEhiCw72EGK6PI6GHURbrzyvLmD/xqzZqCYPKp7pmqYuSjPxuKb Jpgz/xgdGzwK2hAiq04t9rSuljcNO4yFc6ruWeY3+9THiIsmCm8cPUG0X78v4zOxKFCW yuGbU1UaPVis85fKh9pX+RRRGdVoM5fDuI+qecBc+a3VmiGKtbE5PfDbMdpGwCUMK67l mOxxEx2J4+S/M3wgP4UG5nDUGSVt7tkIuhWO4dcEjU5AuwC7vBhMFU+7acWh7Luu3PkX 3d3g== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20230601; t=1706031633; x=1706636433; h=content-transfer-encoding:mime-version:references:in-reply-to :message-id:date:subject:cc:to:from:x-gm-message-state:from:to:cc :subject:date:message-id:reply-to; bh=/QSidYI5kedHrpCI5ufZ8srx64p2MxR2IVg8u5MrqJU=; b=qthNdhhQN0qiGhahLo1djtC+Qp+dlKC0ycUVHHj8cAbR88ZjwpTYm2c7Zf0ibheuM/ cKI+Q8Tyjy6IqF+S/COrM/lmoUZq8wtCufRhOCMto8YZETl1+3SWW/NPVHOlAYM4hTrq JEk4BaVo3+dfCeIfYcbDPrq7xHx86XBcD8NjaTidc5kQw+UXQsU/1tgw5wN8InwIxujj C1ScmNecpPyWTsMyCuUMv2CxuhWH0/6NUnOgu7qnJbzkTwYP7IaKK3JHwBi2xVBSVwnF /5aReuWm3wZ+GNgZZepjknFhzb2X40LXGLzXXdATYNCklCpmdQrYMCYK9fntloW1Acms lMOQ== X-Gm-Message-State: AOJu0YzTQmLS4FL2Iq+WrL+EeDwlU+Cwk8UE7/qKvA7bf1WisS1WZUVw NhOz/pRGoAWEw0+NV31iAmfHmgEBnGuiomfwUxrE9JdC2fWzqNiZpEx3TAmH0BoJoZDJXsa822+ wPzA= X-Google-Smtp-Source: AGHT+IEYpRc5/NOC5zFIUuxGkWellz09UjEjBd3qKwLOIv2vXVYafmnl7j7u+fWpxTmjo80yNx6pEA== X-Received: by 2002:a05:6602:e04:b0:7bf:f20:2c78 with SMTP id gp4-20020a0566020e0400b007bf0f202c78mr9491462iob.1.1706031633444; Tue, 23 Jan 2024 09:40:33 -0800 (PST) Received: from localhost.localdomain ([96.43.243.2]) by smtp.gmail.com with ESMTPSA id l15-20020a6b700f000000b007bf4b9fa2e6sm4647700ioc.47.2024.01.23.09.40.31 (version=TLS1_3 cipher=TLS_AES_256_GCM_SHA384 bits=256/256); Tue, 23 Jan 2024 09:40:32 -0800 (PST) From: Jens Axboe To: linux-block@vger.kernel.org Cc: Jens Axboe Subject: [PATCH 5/8] block/bfq: pass in queue directly to bfq_insert_request() Date: Tue, 23 Jan 2024 10:34:17 -0700 Message-ID: <20240123174021.1967461-6-axboe@kernel.dk> X-Mailer: git-send-email 2.43.0 In-Reply-To: <20240123174021.1967461-1-axboe@kernel.dk> References: <20240123174021.1967461-1-axboe@kernel.dk> Precedence: bulk X-Mailing-List: linux-block@vger.kernel.org List-Id: List-Subscribe: List-Unsubscribe: MIME-Version: 1.0 The hardware queue isn't relevant, bfq only operates on the queue itself. Pass in the queue directly rather than the hardware queue, as that more clearly explains what is being operated on. Signed-off-by: Jens Axboe Reviewed-by: Bart Van Assche Reviewed-by: Johannes Thumshirn --- block/bfq-iosched.c | 5 ++--- 1 file changed, 2 insertions(+), 3 deletions(-) diff --git a/block/bfq-iosched.c b/block/bfq-iosched.c index 3cce6de464a7..7d08442474ec 100644 --- a/block/bfq-iosched.c +++ b/block/bfq-iosched.c @@ -6236,10 +6236,9 @@ static inline void bfq_update_insert_stats(struct request_queue *q, static struct bfq_queue *bfq_init_rq(struct request *rq); -static void bfq_insert_request(struct blk_mq_hw_ctx *hctx, struct request *rq, +static void bfq_insert_request(struct request_queue *q, struct request *rq, blk_insert_t flags) { - struct request_queue *q = hctx->queue; struct bfq_data *bfqd = q->elevator->elevator_data; struct bfq_queue *bfqq; bool idle_timer_disabled = false; @@ -6301,7 +6300,7 @@ static void bfq_insert_requests(struct blk_mq_hw_ctx *hctx, rq = list_first_entry(list, struct request, queuelist); list_del_init(&rq->queuelist); - bfq_insert_request(hctx, rq, flags); + bfq_insert_request(hctx->queue, rq, flags); } } From patchwork Tue Jan 23 17:34:18 2024 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Jens Axboe X-Patchwork-Id: 13527923 Received: from mail-io1-f54.google.com (mail-io1-f54.google.com [209.85.166.54]) (using TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 (128/128 bits)) (No client certificate requested) by smtp.subspace.kernel.org (Postfix) with ESMTPS id A06FF811EA for ; Tue, 23 Jan 2024 17:40:37 +0000 (UTC) Authentication-Results: smtp.subspace.kernel.org; arc=none smtp.client-ip=209.85.166.54 ARC-Seal: i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1706031639; cv=none; b=qkDfhkG9D3kcxTP/aP43UBjwxuJ8EEONwKgkY2TBS6R9gZkdmIbYEtYIkJIdP3MObF4DmW+eBV5GY2ADR2nBMo6vEWp0AK3ZdqoDiMFd2Lm6P8fdElcl4XlAMG4zIYV8fSB5ejNxHykTjJe60rsKBpbrBbMvIxPsyN8D/INYN+I= ARC-Message-Signature: i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1706031639; c=relaxed/simple; bh=CyIXnGLP8A2y0N34/5wGJIe6MOdy+ZpqA6rH5AtvyPM=; h=From:To:Cc:Subject:Date:Message-ID:In-Reply-To:References: MIME-Version; b=oe4FST8U2AZEvMHGcRrm7ooIt/98y8oP6+DUrmmpQJ8UaI888MhlAYjF1AhYPpwmRPYP1AOsrmCiMsRj35G2N1Z+YjUBZ6l+WY/gVWHTUEOMr+qEWJkoN2zMOh1NxE7zcG5tRcoNKPDeUkNSmoTbFfArUjf3h8Oucuz1uyQbNQE= ARC-Authentication-Results: i=1; smtp.subspace.kernel.org; dmarc=none (p=none dis=none) header.from=kernel.dk; spf=pass smtp.mailfrom=kernel.dk; dkim=pass (2048-bit key) header.d=kernel-dk.20230601.gappssmtp.com header.i=@kernel-dk.20230601.gappssmtp.com header.b=L8FqDbDv; arc=none smtp.client-ip=209.85.166.54 Authentication-Results: smtp.subspace.kernel.org; dmarc=none (p=none dis=none) header.from=kernel.dk Authentication-Results: smtp.subspace.kernel.org; spf=pass smtp.mailfrom=kernel.dk Authentication-Results: smtp.subspace.kernel.org; dkim=pass (2048-bit key) header.d=kernel-dk.20230601.gappssmtp.com header.i=@kernel-dk.20230601.gappssmtp.com header.b="L8FqDbDv" Received: by mail-io1-f54.google.com with SMTP id ca18e2360f4ac-7bb06f56fe9so51897539f.0 for ; Tue, 23 Jan 2024 09:40:37 -0800 (PST) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=kernel-dk.20230601.gappssmtp.com; s=20230601; t=1706031636; x=1706636436; darn=vger.kernel.org; h=content-transfer-encoding:mime-version:references:in-reply-to :message-id:date:subject:cc:to:from:from:to:cc:subject:date :message-id:reply-to; bh=8t1KcTyX6xNZnJSO32AqKQbAhtDMC9k+fof8yKQwd4E=; b=L8FqDbDvWcoiiRHMzOm3OVg1/ma590HBAqpEtiHLmr33VNmcN4L7aTvoszmtYlCXue 16o+4pg1KtUUFuMhNsw99xNKIfFkGp5ejvVF6xlH9iDZMKhrHC8XrsMsAjhLPzCApNpg COz5/zOVB2JLRIc88p+coUObPp6xJNX4hXV8P1rzexUer12ej9SN3uT2QTvaGsAXNZs8 zXiSzS97ib6Ob18qW6g1//Wt1Og8N5jSsD4gSMst0zVZVTMFK7MkZLnB5GlaB3X7xOIj jOWoqsMf94dgURYoXtC1zCJvWXwWUX3N0OAd8rKhpu1JXXyMV70DMRcA9hn/gpDXejjw trXQ== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20230601; t=1706031636; x=1706636436; h=content-transfer-encoding:mime-version:references:in-reply-to :message-id:date:subject:cc:to:from:x-gm-message-state:from:to:cc :subject:date:message-id:reply-to; bh=8t1KcTyX6xNZnJSO32AqKQbAhtDMC9k+fof8yKQwd4E=; b=fh97nI2NsnguojizinutbNLtAm6rzIj5sM6JWrj+eSwOM9KnW36eKQx9AZwPYgGKPI XHq1lBKe51jamC//lZwkJS34ILJoEmBBV0OsbWX2rd8jbjrcY3+d65ng4YHaiHeG6iBi TwdUDT3Oq1On6ghKh020y3dCaa8VOKvGeTmps49x0bl2TDG+053ip7UZARlT79o5THEi SyrZ8ZA/1jgkGX1xasegjI3Gu2qxXCC4dwPCUs1ynmu9C1IqOWoYA3IyEeo/vI3X74+W R1zzx4duz1kOje1DZ59luOkYMR/ztAnstAy2dl109CrTKqVdJGbZNbBavO3380nbUWAr bb0g== X-Gm-Message-State: AOJu0Yw9SDWUxj+qoq3Z1dFP01MW9SdbHLKI4D2TBGMcGM5u9m33fgH9 NJoi0uj6PUaPR4qky3IInwmb03Ib8aWiXjGNZRaZFTgoW2wYY7mL9TIOqoFuTD3LGBORfIbpedy AtaI= X-Google-Smtp-Source: AGHT+IHGj6oyke1phBHLOCKfyYkqGZEgsHn0v/qIKHCb4ZRaSvF40RX54etbhQ23RvjlunauNLvLsg== X-Received: by 2002:a05:6602:123b:b0:7be:edbc:629f with SMTP id z27-20020a056602123b00b007beedbc629fmr10390915iot.0.1706031635143; Tue, 23 Jan 2024 09:40:35 -0800 (PST) Received: from localhost.localdomain ([96.43.243.2]) by smtp.gmail.com with ESMTPSA id l15-20020a6b700f000000b007bf4b9fa2e6sm4647700ioc.47.2024.01.23.09.40.33 (version=TLS1_3 cipher=TLS_AES_256_GCM_SHA384 bits=256/256); Tue, 23 Jan 2024 09:40:33 -0800 (PST) From: Jens Axboe To: linux-block@vger.kernel.org Cc: Jens Axboe Subject: [PATCH 6/8] block/bfq: serialize request dispatching Date: Tue, 23 Jan 2024 10:34:18 -0700 Message-ID: <20240123174021.1967461-7-axboe@kernel.dk> X-Mailer: git-send-email 2.43.0 In-Reply-To: <20240123174021.1967461-1-axboe@kernel.dk> References: <20240123174021.1967461-1-axboe@kernel.dk> Precedence: bulk X-Mailing-List: linux-block@vger.kernel.org List-Id: List-Subscribe: List-Unsubscribe: MIME-Version: 1.0 If we're entering request dispatch but someone else is already dispatching, then just skip this dispatch. We know IO is inflight and this will trigger another dispatch event for any completion. This will potentially cause slightly lower queue depth for contended cases, but those are slowed down anyway and this should not cause an issue. By itself, this patch doesn't help a whole lot, as the dispatch lock contention reduction is just eating up by the same dd->lock now seeing increased insertion contention. But it's required work to be able to reduce the lock contention in general. Signed-off-by: Jens Axboe --- block/bfq-iosched.c | 17 +++++++++++++++-- block/bfq-iosched.h | 12 ++++++++++-- 2 files changed, 25 insertions(+), 4 deletions(-) diff --git a/block/bfq-iosched.c b/block/bfq-iosched.c index 7d08442474ec..5ef4a4eba572 100644 --- a/block/bfq-iosched.c +++ b/block/bfq-iosched.c @@ -5304,6 +5304,18 @@ static struct request *bfq_dispatch_request(struct blk_mq_hw_ctx *hctx) struct bfq_queue *in_serv_queue; bool waiting_rq, idle_timer_disabled = false; + /* + * If someone else is already dispatching, skip this one. This will + * defer the next dispatch event to when something completes, and could + * potentially lower the queue depth for contended cases. + * + * See the logic in blk_mq_do_dispatch_sched(), which loops and + * retries if nothing is dispatched. + */ + if (test_bit(BFQ_DISPATCHING, &bfqd->run_state) || + test_and_set_bit_lock(BFQ_DISPATCHING, &bfqd->run_state)) + return NULL; + spin_lock_irq(&bfqd->lock); in_serv_queue = bfqd->in_service_queue; @@ -5315,6 +5327,7 @@ static struct request *bfq_dispatch_request(struct blk_mq_hw_ctx *hctx) waiting_rq && !bfq_bfqq_wait_request(in_serv_queue); } + clear_bit_unlock(BFQ_DISPATCHING, &bfqd->run_state); spin_unlock_irq(&bfqd->lock); bfq_update_dispatch_stats(hctx->queue, rq, idle_timer_disabled ? in_serv_queue : NULL, @@ -7210,6 +7223,8 @@ static int bfq_init_queue(struct request_queue *q, struct elevator_type *e) q->elevator = eq; spin_unlock_irq(&q->queue_lock); + spin_lock_init(&bfqd->lock); + /* * Our fallback bfqq if bfq_find_alloc_queue() runs into OOM issues. * Grab a permanent reference to it, so that the normal code flow @@ -7328,8 +7343,6 @@ static int bfq_init_queue(struct request_queue *q, struct elevator_type *e) /* see comments on the definition of next field inside bfq_data */ bfqd->actuator_load_threshold = 4; - spin_lock_init(&bfqd->lock); - /* * The invocation of the next bfq_create_group_hierarchy * function is the head of a chain of function calls diff --git a/block/bfq-iosched.h b/block/bfq-iosched.h index 467e8cfc41a2..56ff69f22163 100644 --- a/block/bfq-iosched.h +++ b/block/bfq-iosched.h @@ -504,12 +504,22 @@ struct bfq_io_cq { unsigned int requests; /* Number of requests this process has in flight */ }; +enum { + BFQ_DISPATCHING = 0, +}; + /** * struct bfq_data - per-device data structure. * * All the fields are protected by @lock. */ struct bfq_data { + struct { + spinlock_t lock; + } ____cacheline_aligned_in_smp; + + unsigned long run_state; + /* device request queue */ struct request_queue *queue; /* dispatch queue */ @@ -795,8 +805,6 @@ struct bfq_data { /* fallback dummy bfqq for extreme OOM conditions */ struct bfq_queue oom_bfqq; - spinlock_t lock; - /* * bic associated with the task issuing current bio for * merging. This and the next field are used as a support to From patchwork Tue Jan 23 17:34:19 2024 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Jens Axboe X-Patchwork-Id: 13527922 Received: from mail-io1-f51.google.com (mail-io1-f51.google.com [209.85.166.51]) (using TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 (128/128 bits)) (No client certificate requested) by smtp.subspace.kernel.org (Postfix) with ESMTPS id 342598003C for ; Tue, 23 Jan 2024 17:40:37 +0000 (UTC) Authentication-Results: smtp.subspace.kernel.org; arc=none smtp.client-ip=209.85.166.51 ARC-Seal: i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1706031638; cv=none; b=hOKtEtk6nOasIj0NxFPmbcv2qyFOB5ieYpbrc2y9p2sKYC3LXhu1tglHNPO/s2oJucoQopf9sumoQuHNsPP+n9GcH9xECNyNt7vkgH1VzpuLd4cdNa5n+JbrcuhWxN8l5JgUPCuAcqb8aoPkexmVBnOV/HOb6ayMm5zD1YGpiPY= ARC-Message-Signature: i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1706031638; c=relaxed/simple; bh=FJNWVgzbabsKcp7psepexXCA4FsLqAEPlAXmmOysrV0=; h=From:To:Cc:Subject:Date:Message-ID:In-Reply-To:References: MIME-Version; b=aQjYZ2CcaCw3b0hOQG3gL7edGNIqX/G6fccCyAhbcVUvuZE9w66HJC9qnKWPUlitzejxVXmYHEXzu/W+6I/vHPPc8S82OL+avQicPL6J8G8f7Bcv9Ku4FOHAgZIxHAedbAuHNX4WnJoIQkZLNsW3xI2ZiNH+nS4XZ8hofhsRoZ4= ARC-Authentication-Results: i=1; smtp.subspace.kernel.org; dmarc=none (p=none dis=none) header.from=kernel.dk; spf=pass smtp.mailfrom=kernel.dk; dkim=pass (2048-bit key) header.d=kernel-dk.20230601.gappssmtp.com header.i=@kernel-dk.20230601.gappssmtp.com header.b=vhGEk/C4; arc=none smtp.client-ip=209.85.166.51 Authentication-Results: smtp.subspace.kernel.org; dmarc=none (p=none dis=none) header.from=kernel.dk Authentication-Results: smtp.subspace.kernel.org; spf=pass smtp.mailfrom=kernel.dk Authentication-Results: smtp.subspace.kernel.org; dkim=pass (2048-bit key) header.d=kernel-dk.20230601.gappssmtp.com header.i=@kernel-dk.20230601.gappssmtp.com header.b="vhGEk/C4" Received: by mail-io1-f51.google.com with SMTP id ca18e2360f4ac-7bb06f56fe9so51897239f.0 for ; Tue, 23 Jan 2024 09:40:36 -0800 (PST) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=kernel-dk.20230601.gappssmtp.com; s=20230601; t=1706031636; x=1706636436; darn=vger.kernel.org; h=content-transfer-encoding:mime-version:references:in-reply-to :message-id:date:subject:cc:to:from:from:to:cc:subject:date :message-id:reply-to; bh=AFwFEWjrm/jIbIn3f2kPlUx+1yILSk4z8zlrLMpPL5g=; b=vhGEk/C4/+UOZI9z1tGvjS1wqm2vto1enjyEVD4KkFab7fL23EAtV0LT6uCRkLp4YK epjR1jGFAh914iFsB5H7sQTALHbBm/eZPsQqBbdj5JoWbw/S6nvT+n5BknkHKmztaKkE wAsWhFB0ziALez70fmiDe/za3WMCvKwkRIRxFUJKG2xmMIHGynCJd+Es1yajZ3eGhfcz W6vMHbgdfm7MBFOq65fL6lovz70B2Mdhqleo1GAGSWO4B9cmhfoNj5vUan5fcqFAqjla G+T8u7SNM6VVN6M45jmxyOFUKZlVngwKQcF0Uh/jJsIpne2QHWsU80wZXDVYBLrwJ/jF i+rw== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20230601; t=1706031636; x=1706636436; h=content-transfer-encoding:mime-version:references:in-reply-to :message-id:date:subject:cc:to:from:x-gm-message-state:from:to:cc :subject:date:message-id:reply-to; bh=AFwFEWjrm/jIbIn3f2kPlUx+1yILSk4z8zlrLMpPL5g=; b=j8XKkDPWoOxcPiwsgMV2awwaE6db9S3SCSOdK4n8iIkm4yLBAB5fClNDG3TOlcuAwI NWv4TwzGKKCOdjN588fKk2Yxjcbi+Dc1kkfpVcL3KaUAGfi0lazBIFmlCqRbaofKOkj+ wPOibdXqbejmjBLpNf0iSRf7WPkWfSqwnimvCy11na1QC7F9pfxNdmFY+W5I8Kpn4MOB CkSGdfnZr/lVJLi7a0REZc3RhbaAP5m7zUVCuzz5UjRpn2jmuUUdqs3Um5yWcDyl2Hml 0Yshk4BPZgg0XeRbMaPGyhmE2QKhF+DcwqTb1zeQFYLK+2ccmDTsLV5oYA52o2mgAWZb I16A== X-Gm-Message-State: AOJu0YwHJvDKLgoGSTPp8BITdb/RtHEFPLGiDcoga1EHSZOtW1jYXvRY LZzYfdo124VLPa6d+pHKaI95SR/kjE62A74h+cdao7tNE0+VByQe2hWq7UxtjfR5PvCcm7DBGdf sV7U= X-Google-Smtp-Source: AGHT+IFWXJ0VDUPEqcYzpxQvcGlKbTJJzCCNtKNHfiOSeOZJBd9RrTAzAsZkPKy/vYB8xDZ6CbtCAw== X-Received: by 2002:a6b:4f09:0:b0:7be:e080:6869 with SMTP id d9-20020a6b4f09000000b007bee0806869mr10098460iob.1.1706031636037; Tue, 23 Jan 2024 09:40:36 -0800 (PST) Received: from localhost.localdomain ([96.43.243.2]) by smtp.gmail.com with ESMTPSA id l15-20020a6b700f000000b007bf4b9fa2e6sm4647700ioc.47.2024.01.23.09.40.35 (version=TLS1_3 cipher=TLS_AES_256_GCM_SHA384 bits=256/256); Tue, 23 Jan 2024 09:40:35 -0800 (PST) From: Jens Axboe To: linux-block@vger.kernel.org Cc: Jens Axboe Subject: [PATCH 7/8] block/bfq: skip expensive merge lookups if contended Date: Tue, 23 Jan 2024 10:34:19 -0700 Message-ID: <20240123174021.1967461-8-axboe@kernel.dk> X-Mailer: git-send-email 2.43.0 In-Reply-To: <20240123174021.1967461-1-axboe@kernel.dk> References: <20240123174021.1967461-1-axboe@kernel.dk> Precedence: bulk X-Mailing-List: linux-block@vger.kernel.org List-Id: List-Subscribe: List-Unsubscribe: MIME-Version: 1.0 We do several stages of merging in the block layer - the most likely one to work is also the cheap one, merging direct in the per-task plug when IO is submitted. Getting merges outside of that is a lot less likely, but IO schedulers may still maintain internal data structures to facilitate merge lookups outside of the plug. Make BFQ skip expensive merge lookups if the queue lock or bfqd lock is already contended. The likelihood of getting a merge here is not very high, hence it should not be a problem skipping the attempt in the also unlikely event that either the queue or bfqd are already contended. Perf diff shows the difference between a random read/write workload with 4 threads doing IO, with expensive merges turned on and off: 31.70% +54.80% [kernel.kallsyms] [k] queued_spin_lock_slowpath where we almost tripple the lock contention (~32% -> ~87%) by attempting these expensive merges, and performance drops from 1630K to 1050K IOPS. At the same time, sys time drops from 37% to 14%. Signed-off-by: Jens Axboe --- block/bfq-iosched.c | 30 ++++++++++++++++++++++++++++-- 1 file changed, 28 insertions(+), 2 deletions(-) diff --git a/block/bfq-iosched.c b/block/bfq-iosched.c index 5ef4a4eba572..ea16a0c53082 100644 --- a/block/bfq-iosched.c +++ b/block/bfq-iosched.c @@ -467,6 +467,21 @@ static struct bfq_io_cq *bfq_bic_lookup(struct request_queue *q) return icq; } +static struct bfq_io_cq *bfq_bic_try_lookup(struct request_queue *q) +{ + if (!current->io_context) + return NULL; + if (spin_trylock_irq(&q->queue_lock)) { + struct bfq_io_cq *icq; + + icq = icq_to_bic(ioc_lookup_icq(q)); + spin_unlock_irq(&q->queue_lock); + return icq; + } + + return NULL; +} + /* * Scheduler run of queue, if there are requests pending and no one in the * driver that will restart queueing. @@ -2454,10 +2469,21 @@ static bool bfq_bio_merge(struct request_queue *q, struct bio *bio, * returned by bfq_bic_lookup does not go away before * bfqd->lock is taken. */ - struct bfq_io_cq *bic = bfq_bic_lookup(q); + struct bfq_io_cq *bic = bfq_bic_try_lookup(q); bool ret; - spin_lock_irq(&bfqd->lock); + /* + * bio merging is called for every bio queued, and it's very easy + * to run into contention because of that. If we fail getting + * the dd lock, just skip this merge attempt. For related IO, the + * plug will be the successful merging point. If we get here, we + * already failed doing the obvious merge. Chances of actually + * getting a merge off this path is a lot slimmer, so skipping an + * occassional lookup that will most likely not succeed anyway should + * not be a problem. + */ + if (!spin_trylock_irq(&bfqd->lock)) + return false; if (bic) { /* From patchwork Tue Jan 23 17:34:20 2024 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Jens Axboe X-Patchwork-Id: 13527924 Received: from mail-io1-f52.google.com (mail-io1-f52.google.com [209.85.166.52]) (using TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 (128/128 bits)) (No client certificate requested) by smtp.subspace.kernel.org (Postfix) with ESMTPS id ECAE5811EC for ; Tue, 23 Jan 2024 17:40:38 +0000 (UTC) Authentication-Results: smtp.subspace.kernel.org; arc=none smtp.client-ip=209.85.166.52 ARC-Seal: i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1706031640; cv=none; b=jvEs7BxkbKMKxoPYanbKfP5WY8jVZzFHiaAvjzONms5DZEZXCsH9KKJ7N2y50anybLkNlAvVjCHlsLoLCHm2Jr7WIAmYznSL1tasgkQkj+GO4+mmxuOZB118sef8nbJbL1nDl/jF2hZThc4wiijWCahuOnrbxUkAT/B+2ze1UXg= ARC-Message-Signature: i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1706031640; c=relaxed/simple; bh=KnWGHMmacZniTWp9tv0oQ44d65rhPNIr9+G1kWjEXEU=; h=From:To:Cc:Subject:Date:Message-ID:In-Reply-To:References: MIME-Version; b=okj9wsLZjkikXcMtVUMe60lRwiXNr1UJSmnDniu/Sjwz2CdWdleGS/GjsXUwcNFMcxPTAraCCUfFf2ugMENt5J+XYkBAlXKRWEeD7PQSV/Ow+W7f88rpSXu65JpfZjNl/LiWv0s/BFI9EbOVS/0SW+raOg+bcbvtlHE5koyAndA= ARC-Authentication-Results: i=1; smtp.subspace.kernel.org; dmarc=none (p=none dis=none) header.from=kernel.dk; spf=pass smtp.mailfrom=kernel.dk; dkim=pass (2048-bit key) header.d=kernel-dk.20230601.gappssmtp.com header.i=@kernel-dk.20230601.gappssmtp.com header.b=VjLJtHFq; arc=none smtp.client-ip=209.85.166.52 Authentication-Results: smtp.subspace.kernel.org; dmarc=none (p=none dis=none) header.from=kernel.dk Authentication-Results: smtp.subspace.kernel.org; spf=pass smtp.mailfrom=kernel.dk Authentication-Results: smtp.subspace.kernel.org; dkim=pass (2048-bit key) header.d=kernel-dk.20230601.gappssmtp.com header.i=@kernel-dk.20230601.gappssmtp.com header.b="VjLJtHFq" Received: by mail-io1-f52.google.com with SMTP id ca18e2360f4ac-7beeeb1ba87so61745939f.0 for ; Tue, 23 Jan 2024 09:40:38 -0800 (PST) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=kernel-dk.20230601.gappssmtp.com; s=20230601; t=1706031637; x=1706636437; darn=vger.kernel.org; h=content-transfer-encoding:mime-version:references:in-reply-to :message-id:date:subject:cc:to:from:from:to:cc:subject:date :message-id:reply-to; bh=7j1Y7sJYWn7DCvoraweTkCWNWnVnygffY6V8hmKwE+o=; b=VjLJtHFq5CQiGfCkcxn5dmfPyQEDIg38Hu61iPI/MAIuz0MvT0V387uEGwhWyVTvEF +KOtklZMZYTb6Z4d5pm1E0s8ItT2ykImKsLDST8Z82TMGFctoT5wh9tev2xiZGllHmcQ Vr9YcN7bR74gRnW0Ee7XwNyczuFwu6pxV3kcRTXYTEDoTw3xEo13iekkP4DQxA3dNM6U ehLSB5OfJhZ77yOkSlO8XzURyCPjmT+hhGujqSRI4FnpkuPOgkQFvesg6w+YlExPk6Px y6AxtJUO6LKgZhs2mV0J9iMJWoCdQu7P9NJkfkdbbJP82ipx9LptwdZv+b9UDn5oCTfI 7ECg== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20230601; t=1706031637; x=1706636437; h=content-transfer-encoding:mime-version:references:in-reply-to :message-id:date:subject:cc:to:from:x-gm-message-state:from:to:cc :subject:date:message-id:reply-to; bh=7j1Y7sJYWn7DCvoraweTkCWNWnVnygffY6V8hmKwE+o=; b=uTpvk6B01UCEplgJ1GtLEuDoZ7rUb5QjYEYEJnmw5VC/FFlkpyeTL/pGGsSKY7JQNA xJKjg8fJPDwqTboLrKTnqzdErPsNpMlAkdDDOslmTYPKYEhOOwhehbecaeejRMA0YdkT tOc9uRWeEeSRle1G9/oPRFP44m37+CNh9WO23NZ7aSL42S620SQdI4NwcT2rndaTPBGO hCjXdKTvRyZqGKUwr78ZiMuKlRvOfMysfsvy+IPNyPACf60mapxVA6YNIHGdEUtIRChn LZyUnTZ40NpF1VmJbenfsIHIy344lWm47OfCpYNCmrM3p7c/f2/kv+EsxiiXZCMIvS/h n1sQ== X-Gm-Message-State: AOJu0YyjvSlZxhWZQBuIGC2FdickaENrgvqvpQrO3B3jHDVNNSu7T9IM CTMV0+y3u/NHLvS+kRnFXJcs1ZkY/GSB1lBbqurS0nE64FEg4fr6L678/uAPLzPuYXC78FEMnMR SDCM= X-Google-Smtp-Source: AGHT+IHYZHy/EwBSLYqzI5TS06nzAFHB5QyfsbaL2tfvs+h9qPZOzfTiozr52kzBTQVo7h6QUAzuwg== X-Received: by 2002:a5d:8942:0:b0:7bf:3651:4c6d with SMTP id b2-20020a5d8942000000b007bf36514c6dmr10358314iot.2.1706031637621; Tue, 23 Jan 2024 09:40:37 -0800 (PST) Received: from localhost.localdomain ([96.43.243.2]) by smtp.gmail.com with ESMTPSA id l15-20020a6b700f000000b007bf4b9fa2e6sm4647700ioc.47.2024.01.23.09.40.36 (version=TLS1_3 cipher=TLS_AES_256_GCM_SHA384 bits=256/256); Tue, 23 Jan 2024 09:40:36 -0800 (PST) From: Jens Axboe To: linux-block@vger.kernel.org Cc: Jens Axboe Subject: [PATCH 8/8] block/bfq: use separate insertion lists Date: Tue, 23 Jan 2024 10:34:20 -0700 Message-ID: <20240123174021.1967461-9-axboe@kernel.dk> X-Mailer: git-send-email 2.43.0 In-Reply-To: <20240123174021.1967461-1-axboe@kernel.dk> References: <20240123174021.1967461-1-axboe@kernel.dk> Precedence: bulk X-Mailing-List: linux-block@vger.kernel.org List-Id: List-Subscribe: List-Unsubscribe: MIME-Version: 1.0 Based on the similar patch for mq-deadline, this uses separate insertion lists so we can defer touching dd->lock until dispatch time. This improves the following fio workload: fio --bs=512 --group_reporting=1 --gtod_reduce=1 --invalidate=1 \ --ioengine=io_uring --norandommap --runtime=60 --rw=randread \ --thread --time_based=1 --buffered=0 --fixedbufs=1 --numjobs=32 \ --iodepth=4 --iodepth_batch_submit=4 --iodepth_batch_complete=4 \ --name=/dev/nvme0n1 --filename=/dev/nvme0n1 from: /dev/nvme0n1: (groupid=0, jobs=32): err= 0: pid=1113: Fri Jan 19 20:59:26 2024 read: IOPS=567k, BW=277MiB/s (290MB/s)(1820MiB/6575msec) bw ( KiB/s): min=274824, max=291156, per=100.00%, avg=283930.08, stdev=143.01, samples=416 iops : min=549648, max=582312, avg=567860.31, stdev=286.01, samples=416 cpu : usr=0.18%, sys=86.04%, ctx=866079, majf=0, minf=0 IO depths : 1=0.0%, 2=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=3728344,0,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=4 with 96% lock contention and 86% sys time, to: /dev/nvme0n1: (groupid=0, jobs=32): err= 0: pid=8922: Sat Jan 20 11:16:20 2024 read: IOPS=1550k, BW=757MiB/s (794MB/s)(19.6GiB/26471msec) bw ( KiB/s): min=754668, max=848896, per=100.00%, avg=775459.33, stdev=624.43, samples=1664 iops : min=1509336, max=1697793, avg=1550918.83, stdev=1248.87, samples=1664 cpu : usr=1.34%, sys=14.49%, ctx=9950560, majf=0, minf=0 IO depths : 1=0.0%, 2=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=41042924,0,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=4 with ~30% lock contention and 14.5% sys time, by applying the lessons learnt with scaling mq-deadline. Patch needs to be split, but it: Signed-off-by: Jens Axboe --- block/bfq-iosched.c | 70 ++++++++++++++++++++++++++++++++++----------- block/bfq-iosched.h | 4 +++ 2 files changed, 58 insertions(+), 16 deletions(-) diff --git a/block/bfq-iosched.c b/block/bfq-iosched.c index ea16a0c53082..9bd57baa4b0b 100644 --- a/block/bfq-iosched.c +++ b/block/bfq-iosched.c @@ -5174,6 +5174,10 @@ static bool bfq_has_work(struct blk_mq_hw_ctx *hctx) { struct bfq_data *bfqd = hctx->queue->elevator->elevator_data; + if (!list_empty_careful(&bfqd->at_head) || + !list_empty_careful(&bfqd->at_tail)) + return true; + /* * Avoiding lock: a race on bfqd->queued should cause at * most a call to dispatch for nothing @@ -5323,12 +5327,44 @@ static inline void bfq_update_dispatch_stats(struct request_queue *q, bool idle_timer_disabled) {} #endif /* CONFIG_BFQ_CGROUP_DEBUG */ +static void bfq_insert_request(struct request_queue *q, struct request *rq, + blk_insert_t flags, struct list_head *free); + +static void __bfq_do_insert(struct request_queue *q, blk_insert_t flags, + struct list_head *list, struct list_head *free) +{ + while (!list_empty(list)) { + struct request *rq; + + rq = list_first_entry(list, struct request, queuelist); + list_del_init(&rq->queuelist); + bfq_insert_request(q, rq, flags, free); + } +} + +static void bfq_do_insert(struct request_queue *q, struct list_head *free) +{ + struct bfq_data *bfqd = q->elevator->elevator_data; + LIST_HEAD(at_head); + LIST_HEAD(at_tail); + + spin_lock(&bfqd->insert_lock); + list_splice_init(&bfqd->at_head, &at_head); + list_splice_init(&bfqd->at_tail, &at_tail); + spin_unlock(&bfqd->insert_lock); + + __bfq_do_insert(q, BLK_MQ_INSERT_AT_HEAD, &at_head, free); + __bfq_do_insert(q, 0, &at_tail, free); +} + static struct request *bfq_dispatch_request(struct blk_mq_hw_ctx *hctx) { - struct bfq_data *bfqd = hctx->queue->elevator->elevator_data; + struct request_queue *q = hctx->queue; + struct bfq_data *bfqd = q->elevator->elevator_data; struct request *rq; struct bfq_queue *in_serv_queue; bool waiting_rq, idle_timer_disabled = false; + LIST_HEAD(free); /* * If someone else is already dispatching, skip this one. This will @@ -5344,6 +5380,8 @@ static struct request *bfq_dispatch_request(struct blk_mq_hw_ctx *hctx) spin_lock_irq(&bfqd->lock); + bfq_do_insert(hctx->queue, &free); + in_serv_queue = bfqd->in_service_queue; waiting_rq = in_serv_queue && bfq_bfqq_wait_request(in_serv_queue); @@ -5355,6 +5393,7 @@ static struct request *bfq_dispatch_request(struct blk_mq_hw_ctx *hctx) clear_bit_unlock(BFQ_DISPATCHING, &bfqd->run_state); spin_unlock_irq(&bfqd->lock); + blk_mq_free_requests(&free); bfq_update_dispatch_stats(hctx->queue, rq, idle_timer_disabled ? in_serv_queue : NULL, idle_timer_disabled); @@ -6276,25 +6315,20 @@ static inline void bfq_update_insert_stats(struct request_queue *q, static struct bfq_queue *bfq_init_rq(struct request *rq); static void bfq_insert_request(struct request_queue *q, struct request *rq, - blk_insert_t flags) + blk_insert_t flags, struct list_head *free) { struct bfq_data *bfqd = q->elevator->elevator_data; struct bfq_queue *bfqq; bool idle_timer_disabled = false; blk_opf_t cmd_flags; - LIST_HEAD(free); #ifdef CONFIG_BFQ_GROUP_IOSCHED if (!cgroup_subsys_on_dfl(io_cgrp_subsys) && rq->bio) bfqg_stats_update_legacy_io(q, rq); #endif - spin_lock_irq(&bfqd->lock); bfqq = bfq_init_rq(rq); - if (blk_mq_sched_try_insert_merge(q, rq, &free)) { - spin_unlock_irq(&bfqd->lock); - blk_mq_free_requests(&free); + if (blk_mq_sched_try_insert_merge(q, rq, free)) return; - } trace_block_rq_insert(rq); @@ -6324,8 +6358,6 @@ static void bfq_insert_request(struct request_queue *q, struct request *rq, * merge). */ cmd_flags = rq->cmd_flags; - spin_unlock_irq(&bfqd->lock); - bfq_update_insert_stats(q, bfqq, idle_timer_disabled, cmd_flags); } @@ -6334,13 +6366,15 @@ static void bfq_insert_requests(struct blk_mq_hw_ctx *hctx, struct list_head *list, blk_insert_t flags) { - while (!list_empty(list)) { - struct request *rq; + struct request_queue *q = hctx->queue; + struct bfq_data *bfqd = q->elevator->elevator_data; - rq = list_first_entry(list, struct request, queuelist); - list_del_init(&rq->queuelist); - bfq_insert_request(hctx->queue, rq, flags); - } + spin_lock_irq(&bfqd->insert_lock); + if (flags & BLK_MQ_INSERT_AT_HEAD) + list_splice_init(list, &bfqd->at_head); + else + list_splice_init(list, &bfqd->at_tail); + spin_unlock_irq(&bfqd->insert_lock); } static void bfq_update_hw_tag(struct bfq_data *bfqd) @@ -7250,6 +7284,10 @@ static int bfq_init_queue(struct request_queue *q, struct elevator_type *e) spin_unlock_irq(&q->queue_lock); spin_lock_init(&bfqd->lock); + spin_lock_init(&bfqd->insert_lock); + + INIT_LIST_HEAD(&bfqd->at_head); + INIT_LIST_HEAD(&bfqd->at_tail); /* * Our fallback bfqq if bfq_find_alloc_queue() runs into OOM issues. diff --git a/block/bfq-iosched.h b/block/bfq-iosched.h index 56ff69f22163..f44f5d4ec2f4 100644 --- a/block/bfq-iosched.h +++ b/block/bfq-iosched.h @@ -516,10 +516,14 @@ enum { struct bfq_data { struct { spinlock_t lock; + spinlock_t insert_lock; } ____cacheline_aligned_in_smp; unsigned long run_state; + struct list_head at_head; + struct list_head at_tail; + /* device request queue */ struct request_queue *queue; /* dispatch queue */